[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-huggingface-huggingface-community-evals":3,"mdc-tmxg3d-key":38,"related-org-huggingface-huggingface-community-evals":1802,"related-repo-huggingface-huggingface-community-evals":1975},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":27,"repoUrl":28,"updatedAt":29,"license":30,"forks":31,"topics":32,"repo":33,"sourceUrl":36,"mdContent":37},"huggingface-community-evals","run local evaluations for Hugging Face models","Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM \u002F Transformers \u002F accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"huggingface","Hugging Face","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fhuggingface.png",[12,15,18,21,24],{"name":9,"slug":13,"type":14},"hugging-face","tag",{"name":16,"slug":17,"type":14},"LLM","llm",{"name":19,"slug":20,"type":14},"Deep Learning","deep-learning",{"name":22,"slug":23,"type":14},"Evals","evals",{"name":25,"slug":26,"type":14},"Testing","testing",10861,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fskills","2026-04-06T18:25:44.004748",null,721,[],{"repoUrl":28,"stars":27,"forks":31,"topics":34,"description":35},[],"Give your agents the power of the Hugging Face ecosystem","https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fhuggingface-community-evals","---\nname: huggingface-community-evals\ndescription: Run evaluations for Hugging Face Hub models using inspect-ai and lighteval on local hardware. Use for backend selection, local GPU evals, and choosing between vLLM \u002F Transformers \u002F accelerate. Not for HF Jobs orchestration, model-card PRs, .eval_results publication, or community-evals automation.\n---\n\n# Overview\n\nThis skill is for **running evaluations against models on the Hugging Face Hub on local hardware**.\n\nIt covers:\n- `inspect-ai` with local inference\n- `lighteval` with local inference\n- choosing between `vllm`, Hugging Face Transformers, and `accelerate`\n- smoke tests, task selection, and backend fallback strategy\n\nIt does **not** cover:\n- Hugging Face Jobs orchestration\n- model-card or `model-index` edits\n- README table extraction\n- Artificial Analysis imports\n- `.eval_results` generation or publishing\n- PR creation or community-evals automation\n\nIf the user wants to **run the same eval remotely on Hugging Face Jobs**, hand off to the `hugging-face-jobs` skill and pass it one of the local scripts in this skill.\n\nIf the user wants to **publish results into the community evals workflow**, stop after generating the evaluation run and hand off that publishing step to `~\u002Fcode\u002Fcommunity-evals`.\n\n> All paths below are relative to the directory containing this `SKILL.md`.\n\n# When To Use Which Script\n\n| Use case | Script |\n|---|---|\n| Local `inspect-ai` eval on a Hub model via inference providers | `scripts\u002Finspect_eval_uv.py` |\n| Local GPU eval with `inspect-ai` using `vllm` or Transformers | `scripts\u002Finspect_vllm_uv.py` |\n| Local GPU eval with `lighteval` using `vllm` or `accelerate` | `scripts\u002Flighteval_vllm_uv.py` |\n| Extra command patterns | `examples\u002FUSAGE_EXAMPLES.md` |\n\n# Prerequisites\n\n- Prefer `uv run` for local execution.\n- Set `HF_TOKEN` for gated\u002Fprivate models.\n- For local GPU runs, verify GPU access before starting:\n\n```bash\nuv --version\nprintenv HF_TOKEN >\u002Fdev\u002Fnull\nnvidia-smi\n```\n\nIf `nvidia-smi` is unavailable, either:\n- use `scripts\u002Finspect_eval_uv.py` for lighter provider-backed evaluation, or\n- hand off to the `hugging-face-jobs` skill if the user wants remote compute.\n\n# Core Workflow\n\n1. Choose the evaluation framework.\n   - Use `inspect-ai` when you want explicit task control and inspect-native flows.\n   - Use `lighteval` when the benchmark is naturally expressed as a lighteval task string, especially leaderboard-style tasks.\n2. Choose the inference backend.\n   - Prefer `vllm` for throughput on supported architectures.\n   - Use Hugging Face Transformers (`--backend hf`) or `accelerate` as compatibility fallbacks.\n3. Start with a smoke test.\n   - `inspect-ai`: add `--limit 10` or similar.\n   - `lighteval`: add `--max-samples 10`.\n4. Scale up only after the smoke test passes.\n5. If the user wants remote execution, hand off to `hugging-face-jobs` with the same script + args.\n\n# Quick Start\n\n## Option A: inspect-ai with local inference providers path\n\nBest when the model is already supported by Hugging Face Inference Providers and you want the lowest local setup overhead.\n\n```bash\nuv run scripts\u002Finspect_eval_uv.py \\\n  --model meta-llama\u002FLlama-3.2-1B \\\n  --task mmlu \\\n  --limit 20\n```\n\nUse this path when:\n- you want a quick local smoke test\n- you do not need direct GPU control\n- the task already exists in `inspect-evals`\n\n## Option B: inspect-ai on Local GPU\n\nBest when you need to load the Hub model directly, use `vllm`, or fall back to Transformers for unsupported architectures.\n\nLocal GPU:\n\n```bash\nuv run scripts\u002Finspect_vllm_uv.py \\\n  --model meta-llama\u002FLlama-3.2-1B \\\n  --task gsm8k \\\n  --limit 20\n```\n\nTransformers fallback:\n\n```bash\nuv run scripts\u002Finspect_vllm_uv.py \\\n  --model microsoft\u002Fphi-2 \\\n  --task mmlu \\\n  --backend hf \\\n  --trust-remote-code \\\n  --limit 20\n```\n\n## Option C: lighteval on Local GPU\n\nBest when the task is naturally expressed as a `lighteval` task string, especially Open LLM Leaderboard style benchmarks.\n\nLocal GPU:\n\n```bash\nuv run scripts\u002Flighteval_vllm_uv.py \\\n  --model meta-llama\u002FLlama-3.2-3B-Instruct \\\n  --tasks \"leaderboard|mmlu|5,leaderboard|gsm8k|5\" \\\n  --max-samples 20 \\\n  --use-chat-template\n```\n\n`accelerate` fallback:\n\n```bash\nuv run scripts\u002Flighteval_vllm_uv.py \\\n  --model microsoft\u002Fphi-2 \\\n  --tasks \"leaderboard|mmlu|5\" \\\n  --backend accelerate \\\n  --trust-remote-code \\\n  --max-samples 20\n```\n\n# Remote Execution Boundary\n\nThis skill intentionally stops at **local execution and backend selection**.\n\nIf the user wants to:\n- run these scripts on Hugging Face Jobs\n- pick remote hardware\n- pass secrets to remote jobs\n- schedule recurring runs\n- inspect \u002F cancel \u002F monitor jobs\n\nthen switch to the **`hugging-face-jobs`** skill and pass it one of these scripts plus the chosen arguments.\n\n# Task Selection\n\n`inspect-ai` examples:\n- `mmlu`\n- `gsm8k`\n- `hellaswag`\n- `arc_challenge`\n- `truthfulqa`\n- `winogrande`\n- `humaneval`\n\n`lighteval` task strings use `suite|task|num_fewshot`:\n- `leaderboard|mmlu|5`\n- `leaderboard|gsm8k|5`\n- `leaderboard|arc_challenge|25`\n- `lighteval|hellaswag|0`\n\nMultiple `lighteval` tasks can be comma-separated in `--tasks`.\n\n# Backend Selection\n\n- Prefer `inspect_vllm_uv.py --backend vllm` for fast GPU inference on supported architectures.\n- Use `inspect_vllm_uv.py --backend hf` when `vllm` does not support the model.\n- Prefer `lighteval_vllm_uv.py --backend vllm` for throughput on supported models.\n- Use `lighteval_vllm_uv.py --backend accelerate` as the compatibility fallback.\n- Use `inspect_eval_uv.py` when Inference Providers already cover the model and you do not need direct GPU control.\n\n# Hardware Guidance\n\n| Model size | Suggested local hardware |\n|---|---|\n| `\u003C 3B` | consumer GPU \u002F Apple Silicon \u002F small dev GPU |\n| `3B - 13B` | stronger local GPU |\n| `13B+` | high-memory local GPU or hand off to `hugging-face-jobs` |\n\nFor smoke tests, prefer cheaper local runs plus `--limit` or `--max-samples`.\n\n# Troubleshooting\n\n- CUDA or vLLM OOM:\n  - reduce `--batch-size`\n  - reduce `--gpu-memory-utilization`\n  - switch to a smaller model for the smoke test\n  - if necessary, hand off to `hugging-face-jobs`\n- Model unsupported by `vllm`:\n  - switch to `--backend hf` for `inspect-ai`\n  - switch to `--backend accelerate` for `lighteval`\n- Gated\u002Fprivate repo access fails:\n  - verify `HF_TOKEN`\n- Custom model code required:\n  - add `--trust-remote-code`\n\n# Examples\n\nSee:\n- `examples\u002FUSAGE_EXAMPLES.md` for local command patterns\n- `scripts\u002Finspect_eval_uv.py`\n- `scripts\u002Finspect_vllm_uv.py`\n- `scripts\u002Flighteval_vllm_uv.py`\n",{"data":39,"body":40},{"name":4,"description":6},{"type":41,"children":42},"root",[43,52,66,71,122,134,181,201,219,235,241,375,381,415,478,491,518,524,657,663,670,675,754,759,783,789,801,806,875,880,979,985,997,1001,1096,1106,1209,1215,1226,1231,1259,1274,1280,1290,1356,1374,1412,1431,1437,1507,1513,1591,1610,1616,1748,1754,1759,1796],{"type":44,"tag":45,"props":46,"children":48},"element","h1",{"id":47},"overview",[49],{"type":50,"value":51},"text","Overview",{"type":44,"tag":53,"props":54,"children":55},"p",{},[56,58,64],{"type":50,"value":57},"This skill is for ",{"type":44,"tag":59,"props":60,"children":61},"strong",{},[62],{"type":50,"value":63},"running evaluations against models on the Hugging Face Hub on local hardware",{"type":50,"value":65},".",{"type":44,"tag":53,"props":67,"children":68},{},[69],{"type":50,"value":70},"It covers:",{"type":44,"tag":72,"props":73,"children":74},"ul",{},[75,88,98,117],{"type":44,"tag":76,"props":77,"children":78},"li",{},[79,86],{"type":44,"tag":80,"props":81,"children":83},"code",{"className":82},[],[84],{"type":50,"value":85},"inspect-ai",{"type":50,"value":87}," with local inference",{"type":44,"tag":76,"props":89,"children":90},{},[91,97],{"type":44,"tag":80,"props":92,"children":94},{"className":93},[],[95],{"type":50,"value":96},"lighteval",{"type":50,"value":87},{"type":44,"tag":76,"props":99,"children":100},{},[101,103,109,111],{"type":50,"value":102},"choosing between ",{"type":44,"tag":80,"props":104,"children":106},{"className":105},[],[107],{"type":50,"value":108},"vllm",{"type":50,"value":110},", Hugging Face Transformers, and ",{"type":44,"tag":80,"props":112,"children":114},{"className":113},[],[115],{"type":50,"value":116},"accelerate",{"type":44,"tag":76,"props":118,"children":119},{},[120],{"type":50,"value":121},"smoke tests, task selection, and backend fallback strategy",{"type":44,"tag":53,"props":123,"children":124},{},[125,127,132],{"type":50,"value":126},"It does ",{"type":44,"tag":59,"props":128,"children":129},{},[130],{"type":50,"value":131},"not",{"type":50,"value":133}," cover:",{"type":44,"tag":72,"props":135,"children":136},{},[137,142,155,160,165,176],{"type":44,"tag":76,"props":138,"children":139},{},[140],{"type":50,"value":141},"Hugging Face Jobs orchestration",{"type":44,"tag":76,"props":143,"children":144},{},[145,147,153],{"type":50,"value":146},"model-card or ",{"type":44,"tag":80,"props":148,"children":150},{"className":149},[],[151],{"type":50,"value":152},"model-index",{"type":50,"value":154}," edits",{"type":44,"tag":76,"props":156,"children":157},{},[158],{"type":50,"value":159},"README table extraction",{"type":44,"tag":76,"props":161,"children":162},{},[163],{"type":50,"value":164},"Artificial Analysis imports",{"type":44,"tag":76,"props":166,"children":167},{},[168,174],{"type":44,"tag":80,"props":169,"children":171},{"className":170},[],[172],{"type":50,"value":173},".eval_results",{"type":50,"value":175}," generation or publishing",{"type":44,"tag":76,"props":177,"children":178},{},[179],{"type":50,"value":180},"PR creation or community-evals automation",{"type":44,"tag":53,"props":182,"children":183},{},[184,186,191,193,199],{"type":50,"value":185},"If the user wants to ",{"type":44,"tag":59,"props":187,"children":188},{},[189],{"type":50,"value":190},"run the same eval remotely on Hugging Face Jobs",{"type":50,"value":192},", hand off to the ",{"type":44,"tag":80,"props":194,"children":196},{"className":195},[],[197],{"type":50,"value":198},"hugging-face-jobs",{"type":50,"value":200}," skill and pass it one of the local scripts in this skill.",{"type":44,"tag":53,"props":202,"children":203},{},[204,205,210,212,218],{"type":50,"value":185},{"type":44,"tag":59,"props":206,"children":207},{},[208],{"type":50,"value":209},"publish results into the community evals workflow",{"type":50,"value":211},", stop after generating the evaluation run and hand off that publishing step to ",{"type":44,"tag":80,"props":213,"children":215},{"className":214},[],[216],{"type":50,"value":217},"~\u002Fcode\u002Fcommunity-evals",{"type":50,"value":65},{"type":44,"tag":220,"props":221,"children":222},"blockquote",{},[223],{"type":44,"tag":53,"props":224,"children":225},{},[226,228,234],{"type":50,"value":227},"All paths below are relative to the directory containing this ",{"type":44,"tag":80,"props":229,"children":231},{"className":230},[],[232],{"type":50,"value":233},"SKILL.md",{"type":50,"value":65},{"type":44,"tag":45,"props":236,"children":238},{"id":237},"when-to-use-which-script",[239],{"type":50,"value":240},"When To Use Which Script",{"type":44,"tag":242,"props":243,"children":244},"table",{},[245,264],{"type":44,"tag":246,"props":247,"children":248},"thead",{},[249],{"type":44,"tag":250,"props":251,"children":252},"tr",{},[253,259],{"type":44,"tag":254,"props":255,"children":256},"th",{},[257],{"type":50,"value":258},"Use case",{"type":44,"tag":254,"props":260,"children":261},{},[262],{"type":50,"value":263},"Script",{"type":44,"tag":265,"props":266,"children":267},"tbody",{},[268,293,324,358],{"type":44,"tag":250,"props":269,"children":270},{},[271,284],{"type":44,"tag":272,"props":273,"children":274},"td",{},[275,277,282],{"type":50,"value":276},"Local ",{"type":44,"tag":80,"props":278,"children":280},{"className":279},[],[281],{"type":50,"value":85},{"type":50,"value":283}," eval on a Hub model via inference providers",{"type":44,"tag":272,"props":285,"children":286},{},[287],{"type":44,"tag":80,"props":288,"children":290},{"className":289},[],[291],{"type":50,"value":292},"scripts\u002Finspect_eval_uv.py",{"type":44,"tag":250,"props":294,"children":295},{},[296,315],{"type":44,"tag":272,"props":297,"children":298},{},[299,301,306,308,313],{"type":50,"value":300},"Local GPU eval with ",{"type":44,"tag":80,"props":302,"children":304},{"className":303},[],[305],{"type":50,"value":85},{"type":50,"value":307}," using ",{"type":44,"tag":80,"props":309,"children":311},{"className":310},[],[312],{"type":50,"value":108},{"type":50,"value":314}," or Transformers",{"type":44,"tag":272,"props":316,"children":317},{},[318],{"type":44,"tag":80,"props":319,"children":321},{"className":320},[],[322],{"type":50,"value":323},"scripts\u002Finspect_vllm_uv.py",{"type":44,"tag":250,"props":325,"children":326},{},[327,349],{"type":44,"tag":272,"props":328,"children":329},{},[330,331,336,337,342,344],{"type":50,"value":300},{"type":44,"tag":80,"props":332,"children":334},{"className":333},[],[335],{"type":50,"value":96},{"type":50,"value":307},{"type":44,"tag":80,"props":338,"children":340},{"className":339},[],[341],{"type":50,"value":108},{"type":50,"value":343}," or ",{"type":44,"tag":80,"props":345,"children":347},{"className":346},[],[348],{"type":50,"value":116},{"type":44,"tag":272,"props":350,"children":351},{},[352],{"type":44,"tag":80,"props":353,"children":355},{"className":354},[],[356],{"type":50,"value":357},"scripts\u002Flighteval_vllm_uv.py",{"type":44,"tag":250,"props":359,"children":360},{},[361,366],{"type":44,"tag":272,"props":362,"children":363},{},[364],{"type":50,"value":365},"Extra command patterns",{"type":44,"tag":272,"props":367,"children":368},{},[369],{"type":44,"tag":80,"props":370,"children":372},{"className":371},[],[373],{"type":50,"value":374},"examples\u002FUSAGE_EXAMPLES.md",{"type":44,"tag":45,"props":376,"children":378},{"id":377},"prerequisites",[379],{"type":50,"value":380},"Prerequisites",{"type":44,"tag":72,"props":382,"children":383},{},[384,397,410],{"type":44,"tag":76,"props":385,"children":386},{},[387,389,395],{"type":50,"value":388},"Prefer ",{"type":44,"tag":80,"props":390,"children":392},{"className":391},[],[393],{"type":50,"value":394},"uv run",{"type":50,"value":396}," for local execution.",{"type":44,"tag":76,"props":398,"children":399},{},[400,402,408],{"type":50,"value":401},"Set ",{"type":44,"tag":80,"props":403,"children":405},{"className":404},[],[406],{"type":50,"value":407},"HF_TOKEN",{"type":50,"value":409}," for gated\u002Fprivate models.",{"type":44,"tag":76,"props":411,"children":412},{},[413],{"type":50,"value":414},"For local GPU runs, verify GPU access before starting:",{"type":44,"tag":416,"props":417,"children":422},"pre",{"className":418,"code":419,"language":420,"meta":421,"style":421},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","uv --version\nprintenv HF_TOKEN >\u002Fdev\u002Fnull\nnvidia-smi\n","bash","",[423],{"type":44,"tag":80,"props":424,"children":425},{"__ignoreMap":421},[426,444,469],{"type":44,"tag":427,"props":428,"children":431},"span",{"class":429,"line":430},"line",1,[432,438],{"type":44,"tag":427,"props":433,"children":435},{"style":434},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[436],{"type":50,"value":437},"uv",{"type":44,"tag":427,"props":439,"children":441},{"style":440},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[442],{"type":50,"value":443}," --version\n",{"type":44,"tag":427,"props":445,"children":447},{"class":429,"line":446},2,[448,453,458,464],{"type":44,"tag":427,"props":449,"children":450},{"style":434},[451],{"type":50,"value":452},"printenv",{"type":44,"tag":427,"props":454,"children":455},{"style":440},[456],{"type":50,"value":457}," HF_TOKEN",{"type":44,"tag":427,"props":459,"children":461},{"style":460},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[462],{"type":50,"value":463}," >",{"type":44,"tag":427,"props":465,"children":466},{"style":440},[467],{"type":50,"value":468},"\u002Fdev\u002Fnull\n",{"type":44,"tag":427,"props":470,"children":472},{"class":429,"line":471},3,[473],{"type":44,"tag":427,"props":474,"children":475},{"style":434},[476],{"type":50,"value":477},"nvidia-smi\n",{"type":44,"tag":53,"props":479,"children":480},{},[481,483,489],{"type":50,"value":482},"If ",{"type":44,"tag":80,"props":484,"children":486},{"className":485},[],[487],{"type":50,"value":488},"nvidia-smi",{"type":50,"value":490}," is unavailable, either:",{"type":44,"tag":72,"props":492,"children":493},{},[494,506],{"type":44,"tag":76,"props":495,"children":496},{},[497,499,504],{"type":50,"value":498},"use ",{"type":44,"tag":80,"props":500,"children":502},{"className":501},[],[503],{"type":50,"value":292},{"type":50,"value":505}," for lighter provider-backed evaluation, or",{"type":44,"tag":76,"props":507,"children":508},{},[509,511,516],{"type":50,"value":510},"hand off to the ",{"type":44,"tag":80,"props":512,"children":514},{"className":513},[],[515],{"type":50,"value":198},{"type":50,"value":517}," skill if the user wants remote compute.",{"type":44,"tag":45,"props":519,"children":521},{"id":520},"core-workflow",[522],{"type":50,"value":523},"Core Workflow",{"type":44,"tag":525,"props":526,"children":527},"ol",{},[528,559,598,640,645],{"type":44,"tag":76,"props":529,"children":530},{},[531,533],{"type":50,"value":532},"Choose the evaluation framework.\n",{"type":44,"tag":72,"props":534,"children":535},{},[536,548],{"type":44,"tag":76,"props":537,"children":538},{},[539,541,546],{"type":50,"value":540},"Use ",{"type":44,"tag":80,"props":542,"children":544},{"className":543},[],[545],{"type":50,"value":85},{"type":50,"value":547}," when you want explicit task control and inspect-native flows.",{"type":44,"tag":76,"props":549,"children":550},{},[551,552,557],{"type":50,"value":540},{"type":44,"tag":80,"props":553,"children":555},{"className":554},[],[556],{"type":50,"value":96},{"type":50,"value":558}," when the benchmark is naturally expressed as a lighteval task string, especially leaderboard-style tasks.",{"type":44,"tag":76,"props":560,"children":561},{},[562,564],{"type":50,"value":563},"Choose the inference backend.\n",{"type":44,"tag":72,"props":565,"children":566},{},[567,578],{"type":44,"tag":76,"props":568,"children":569},{},[570,571,576],{"type":50,"value":388},{"type":44,"tag":80,"props":572,"children":574},{"className":573},[],[575],{"type":50,"value":108},{"type":50,"value":577}," for throughput on supported architectures.",{"type":44,"tag":76,"props":579,"children":580},{},[581,583,589,591,596],{"type":50,"value":582},"Use Hugging Face Transformers (",{"type":44,"tag":80,"props":584,"children":586},{"className":585},[],[587],{"type":50,"value":588},"--backend hf",{"type":50,"value":590},") or ",{"type":44,"tag":80,"props":592,"children":594},{"className":593},[],[595],{"type":50,"value":116},{"type":50,"value":597}," as compatibility fallbacks.",{"type":44,"tag":76,"props":599,"children":600},{},[601,603],{"type":50,"value":602},"Start with a smoke test.\n",{"type":44,"tag":72,"props":604,"children":605},{},[606,624],{"type":44,"tag":76,"props":607,"children":608},{},[609,614,616,622],{"type":44,"tag":80,"props":610,"children":612},{"className":611},[],[613],{"type":50,"value":85},{"type":50,"value":615},": add ",{"type":44,"tag":80,"props":617,"children":619},{"className":618},[],[620],{"type":50,"value":621},"--limit 10",{"type":50,"value":623}," or similar.",{"type":44,"tag":76,"props":625,"children":626},{},[627,632,633,639],{"type":44,"tag":80,"props":628,"children":630},{"className":629},[],[631],{"type":50,"value":96},{"type":50,"value":615},{"type":44,"tag":80,"props":634,"children":636},{"className":635},[],[637],{"type":50,"value":638},"--max-samples 10",{"type":50,"value":65},{"type":44,"tag":76,"props":641,"children":642},{},[643],{"type":50,"value":644},"Scale up only after the smoke test passes.",{"type":44,"tag":76,"props":646,"children":647},{},[648,650,655],{"type":50,"value":649},"If the user wants remote execution, hand off to ",{"type":44,"tag":80,"props":651,"children":653},{"className":652},[],[654],{"type":50,"value":198},{"type":50,"value":656}," with the same script + args.",{"type":44,"tag":45,"props":658,"children":660},{"id":659},"quick-start",[661],{"type":50,"value":662},"Quick Start",{"type":44,"tag":664,"props":665,"children":667},"h2",{"id":666},"option-a-inspect-ai-with-local-inference-providers-path",[668],{"type":50,"value":669},"Option A: inspect-ai with local inference providers path",{"type":44,"tag":53,"props":671,"children":672},{},[673],{"type":50,"value":674},"Best when the model is already supported by Hugging Face Inference Providers and you want the lowest local setup overhead.",{"type":44,"tag":416,"props":676,"children":678},{"className":418,"code":677,"language":420,"meta":421,"style":421},"uv run scripts\u002Finspect_eval_uv.py \\\n  --model meta-llama\u002FLlama-3.2-1B \\\n  --task mmlu \\\n  --limit 20\n",[679],{"type":44,"tag":80,"props":680,"children":681},{"__ignoreMap":421},[682,705,722,739],{"type":44,"tag":427,"props":683,"children":684},{"class":429,"line":430},[685,689,694,699],{"type":44,"tag":427,"props":686,"children":687},{"style":434},[688],{"type":50,"value":437},{"type":44,"tag":427,"props":690,"children":691},{"style":440},[692],{"type":50,"value":693}," run",{"type":44,"tag":427,"props":695,"children":696},{"style":440},[697],{"type":50,"value":698}," scripts\u002Finspect_eval_uv.py",{"type":44,"tag":427,"props":700,"children":702},{"style":701},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[703],{"type":50,"value":704}," \\\n",{"type":44,"tag":427,"props":706,"children":707},{"class":429,"line":446},[708,713,718],{"type":44,"tag":427,"props":709,"children":710},{"style":440},[711],{"type":50,"value":712},"  --model",{"type":44,"tag":427,"props":714,"children":715},{"style":440},[716],{"type":50,"value":717}," meta-llama\u002FLlama-3.2-1B",{"type":44,"tag":427,"props":719,"children":720},{"style":701},[721],{"type":50,"value":704},{"type":44,"tag":427,"props":723,"children":724},{"class":429,"line":471},[725,730,735],{"type":44,"tag":427,"props":726,"children":727},{"style":440},[728],{"type":50,"value":729},"  --task",{"type":44,"tag":427,"props":731,"children":732},{"style":440},[733],{"type":50,"value":734}," mmlu",{"type":44,"tag":427,"props":736,"children":737},{"style":701},[738],{"type":50,"value":704},{"type":44,"tag":427,"props":740,"children":742},{"class":429,"line":741},4,[743,748],{"type":44,"tag":427,"props":744,"children":745},{"style":440},[746],{"type":50,"value":747},"  --limit",{"type":44,"tag":427,"props":749,"children":751},{"style":750},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[752],{"type":50,"value":753}," 20\n",{"type":44,"tag":53,"props":755,"children":756},{},[757],{"type":50,"value":758},"Use this path when:",{"type":44,"tag":72,"props":760,"children":761},{},[762,767,772],{"type":44,"tag":76,"props":763,"children":764},{},[765],{"type":50,"value":766},"you want a quick local smoke test",{"type":44,"tag":76,"props":768,"children":769},{},[770],{"type":50,"value":771},"you do not need direct GPU control",{"type":44,"tag":76,"props":773,"children":774},{},[775,777],{"type":50,"value":776},"the task already exists in ",{"type":44,"tag":80,"props":778,"children":780},{"className":779},[],[781],{"type":50,"value":782},"inspect-evals",{"type":44,"tag":664,"props":784,"children":786},{"id":785},"option-b-inspect-ai-on-local-gpu",[787],{"type":50,"value":788},"Option B: inspect-ai on Local GPU",{"type":44,"tag":53,"props":790,"children":791},{},[792,794,799],{"type":50,"value":793},"Best when you need to load the Hub model directly, use ",{"type":44,"tag":80,"props":795,"children":797},{"className":796},[],[798],{"type":50,"value":108},{"type":50,"value":800},", or fall back to Transformers for unsupported architectures.",{"type":44,"tag":53,"props":802,"children":803},{},[804],{"type":50,"value":805},"Local GPU:",{"type":44,"tag":416,"props":807,"children":809},{"className":418,"code":808,"language":420,"meta":421,"style":421},"uv run scripts\u002Finspect_vllm_uv.py \\\n  --model meta-llama\u002FLlama-3.2-1B \\\n  --task gsm8k \\\n  --limit 20\n",[810],{"type":44,"tag":80,"props":811,"children":812},{"__ignoreMap":421},[813,833,848,864],{"type":44,"tag":427,"props":814,"children":815},{"class":429,"line":430},[816,820,824,829],{"type":44,"tag":427,"props":817,"children":818},{"style":434},[819],{"type":50,"value":437},{"type":44,"tag":427,"props":821,"children":822},{"style":440},[823],{"type":50,"value":693},{"type":44,"tag":427,"props":825,"children":826},{"style":440},[827],{"type":50,"value":828}," scripts\u002Finspect_vllm_uv.py",{"type":44,"tag":427,"props":830,"children":831},{"style":701},[832],{"type":50,"value":704},{"type":44,"tag":427,"props":834,"children":835},{"class":429,"line":446},[836,840,844],{"type":44,"tag":427,"props":837,"children":838},{"style":440},[839],{"type":50,"value":712},{"type":44,"tag":427,"props":841,"children":842},{"style":440},[843],{"type":50,"value":717},{"type":44,"tag":427,"props":845,"children":846},{"style":701},[847],{"type":50,"value":704},{"type":44,"tag":427,"props":849,"children":850},{"class":429,"line":471},[851,855,860],{"type":44,"tag":427,"props":852,"children":853},{"style":440},[854],{"type":50,"value":729},{"type":44,"tag":427,"props":856,"children":857},{"style":440},[858],{"type":50,"value":859}," gsm8k",{"type":44,"tag":427,"props":861,"children":862},{"style":701},[863],{"type":50,"value":704},{"type":44,"tag":427,"props":865,"children":866},{"class":429,"line":741},[867,871],{"type":44,"tag":427,"props":868,"children":869},{"style":440},[870],{"type":50,"value":747},{"type":44,"tag":427,"props":872,"children":873},{"style":750},[874],{"type":50,"value":753},{"type":44,"tag":53,"props":876,"children":877},{},[878],{"type":50,"value":879},"Transformers fallback:",{"type":44,"tag":416,"props":881,"children":883},{"className":418,"code":882,"language":420,"meta":421,"style":421},"uv run scripts\u002Finspect_vllm_uv.py \\\n  --model microsoft\u002Fphi-2 \\\n  --task mmlu \\\n  --backend hf \\\n  --trust-remote-code \\\n  --limit 20\n",[884],{"type":44,"tag":80,"props":885,"children":886},{"__ignoreMap":421},[887,906,922,937,954,967],{"type":44,"tag":427,"props":888,"children":889},{"class":429,"line":430},[890,894,898,902],{"type":44,"tag":427,"props":891,"children":892},{"style":434},[893],{"type":50,"value":437},{"type":44,"tag":427,"props":895,"children":896},{"style":440},[897],{"type":50,"value":693},{"type":44,"tag":427,"props":899,"children":900},{"style":440},[901],{"type":50,"value":828},{"type":44,"tag":427,"props":903,"children":904},{"style":701},[905],{"type":50,"value":704},{"type":44,"tag":427,"props":907,"children":908},{"class":429,"line":446},[909,913,918],{"type":44,"tag":427,"props":910,"children":911},{"style":440},[912],{"type":50,"value":712},{"type":44,"tag":427,"props":914,"children":915},{"style":440},[916],{"type":50,"value":917}," microsoft\u002Fphi-2",{"type":44,"tag":427,"props":919,"children":920},{"style":701},[921],{"type":50,"value":704},{"type":44,"tag":427,"props":923,"children":924},{"class":429,"line":471},[925,929,933],{"type":44,"tag":427,"props":926,"children":927},{"style":440},[928],{"type":50,"value":729},{"type":44,"tag":427,"props":930,"children":931},{"style":440},[932],{"type":50,"value":734},{"type":44,"tag":427,"props":934,"children":935},{"style":701},[936],{"type":50,"value":704},{"type":44,"tag":427,"props":938,"children":939},{"class":429,"line":741},[940,945,950],{"type":44,"tag":427,"props":941,"children":942},{"style":440},[943],{"type":50,"value":944},"  --backend",{"type":44,"tag":427,"props":946,"children":947},{"style":440},[948],{"type":50,"value":949}," hf",{"type":44,"tag":427,"props":951,"children":952},{"style":701},[953],{"type":50,"value":704},{"type":44,"tag":427,"props":955,"children":957},{"class":429,"line":956},5,[958,963],{"type":44,"tag":427,"props":959,"children":960},{"style":440},[961],{"type":50,"value":962},"  --trust-remote-code",{"type":44,"tag":427,"props":964,"children":965},{"style":701},[966],{"type":50,"value":704},{"type":44,"tag":427,"props":968,"children":970},{"class":429,"line":969},6,[971,975],{"type":44,"tag":427,"props":972,"children":973},{"style":440},[974],{"type":50,"value":747},{"type":44,"tag":427,"props":976,"children":977},{"style":750},[978],{"type":50,"value":753},{"type":44,"tag":664,"props":980,"children":982},{"id":981},"option-c-lighteval-on-local-gpu",[983],{"type":50,"value":984},"Option C: lighteval on Local GPU",{"type":44,"tag":53,"props":986,"children":987},{},[988,990,995],{"type":50,"value":989},"Best when the task is naturally expressed as a ",{"type":44,"tag":80,"props":991,"children":993},{"className":992},[],[994],{"type":50,"value":96},{"type":50,"value":996}," task string, especially Open LLM Leaderboard style benchmarks.",{"type":44,"tag":53,"props":998,"children":999},{},[1000],{"type":50,"value":805},{"type":44,"tag":416,"props":1002,"children":1004},{"className":418,"code":1003,"language":420,"meta":421,"style":421},"uv run scripts\u002Flighteval_vllm_uv.py \\\n  --model meta-llama\u002FLlama-3.2-3B-Instruct \\\n  --tasks \"leaderboard|mmlu|5,leaderboard|gsm8k|5\" \\\n  --max-samples 20 \\\n  --use-chat-template\n",[1005],{"type":44,"tag":80,"props":1006,"children":1007},{"__ignoreMap":421},[1008,1028,1044,1071,1088],{"type":44,"tag":427,"props":1009,"children":1010},{"class":429,"line":430},[1011,1015,1019,1024],{"type":44,"tag":427,"props":1012,"children":1013},{"style":434},[1014],{"type":50,"value":437},{"type":44,"tag":427,"props":1016,"children":1017},{"style":440},[1018],{"type":50,"value":693},{"type":44,"tag":427,"props":1020,"children":1021},{"style":440},[1022],{"type":50,"value":1023}," scripts\u002Flighteval_vllm_uv.py",{"type":44,"tag":427,"props":1025,"children":1026},{"style":701},[1027],{"type":50,"value":704},{"type":44,"tag":427,"props":1029,"children":1030},{"class":429,"line":446},[1031,1035,1040],{"type":44,"tag":427,"props":1032,"children":1033},{"style":440},[1034],{"type":50,"value":712},{"type":44,"tag":427,"props":1036,"children":1037},{"style":440},[1038],{"type":50,"value":1039}," meta-llama\u002FLlama-3.2-3B-Instruct",{"type":44,"tag":427,"props":1041,"children":1042},{"style":701},[1043],{"type":50,"value":704},{"type":44,"tag":427,"props":1045,"children":1046},{"class":429,"line":471},[1047,1052,1057,1062,1067],{"type":44,"tag":427,"props":1048,"children":1049},{"style":440},[1050],{"type":50,"value":1051},"  --tasks",{"type":44,"tag":427,"props":1053,"children":1054},{"style":460},[1055],{"type":50,"value":1056}," \"",{"type":44,"tag":427,"props":1058,"children":1059},{"style":440},[1060],{"type":50,"value":1061},"leaderboard|mmlu|5,leaderboard|gsm8k|5",{"type":44,"tag":427,"props":1063,"children":1064},{"style":460},[1065],{"type":50,"value":1066},"\"",{"type":44,"tag":427,"props":1068,"children":1069},{"style":701},[1070],{"type":50,"value":704},{"type":44,"tag":427,"props":1072,"children":1073},{"class":429,"line":741},[1074,1079,1084],{"type":44,"tag":427,"props":1075,"children":1076},{"style":440},[1077],{"type":50,"value":1078},"  --max-samples",{"type":44,"tag":427,"props":1080,"children":1081},{"style":750},[1082],{"type":50,"value":1083}," 20",{"type":44,"tag":427,"props":1085,"children":1086},{"style":701},[1087],{"type":50,"value":704},{"type":44,"tag":427,"props":1089,"children":1090},{"class":429,"line":956},[1091],{"type":44,"tag":427,"props":1092,"children":1093},{"style":440},[1094],{"type":50,"value":1095},"  --use-chat-template\n",{"type":44,"tag":53,"props":1097,"children":1098},{},[1099,1104],{"type":44,"tag":80,"props":1100,"children":1102},{"className":1101},[],[1103],{"type":50,"value":116},{"type":50,"value":1105}," fallback:",{"type":44,"tag":416,"props":1107,"children":1109},{"className":418,"code":1108,"language":420,"meta":421,"style":421},"uv run scripts\u002Flighteval_vllm_uv.py \\\n  --model microsoft\u002Fphi-2 \\\n  --tasks \"leaderboard|mmlu|5\" \\\n  --backend accelerate \\\n  --trust-remote-code \\\n  --max-samples 20\n",[1110],{"type":44,"tag":80,"props":1111,"children":1112},{"__ignoreMap":421},[1113,1132,1147,1171,1187,1198],{"type":44,"tag":427,"props":1114,"children":1115},{"class":429,"line":430},[1116,1120,1124,1128],{"type":44,"tag":427,"props":1117,"children":1118},{"style":434},[1119],{"type":50,"value":437},{"type":44,"tag":427,"props":1121,"children":1122},{"style":440},[1123],{"type":50,"value":693},{"type":44,"tag":427,"props":1125,"children":1126},{"style":440},[1127],{"type":50,"value":1023},{"type":44,"tag":427,"props":1129,"children":1130},{"style":701},[1131],{"type":50,"value":704},{"type":44,"tag":427,"props":1133,"children":1134},{"class":429,"line":446},[1135,1139,1143],{"type":44,"tag":427,"props":1136,"children":1137},{"style":440},[1138],{"type":50,"value":712},{"type":44,"tag":427,"props":1140,"children":1141},{"style":440},[1142],{"type":50,"value":917},{"type":44,"tag":427,"props":1144,"children":1145},{"style":701},[1146],{"type":50,"value":704},{"type":44,"tag":427,"props":1148,"children":1149},{"class":429,"line":471},[1150,1154,1158,1163,1167],{"type":44,"tag":427,"props":1151,"children":1152},{"style":440},[1153],{"type":50,"value":1051},{"type":44,"tag":427,"props":1155,"children":1156},{"style":460},[1157],{"type":50,"value":1056},{"type":44,"tag":427,"props":1159,"children":1160},{"style":440},[1161],{"type":50,"value":1162},"leaderboard|mmlu|5",{"type":44,"tag":427,"props":1164,"children":1165},{"style":460},[1166],{"type":50,"value":1066},{"type":44,"tag":427,"props":1168,"children":1169},{"style":701},[1170],{"type":50,"value":704},{"type":44,"tag":427,"props":1172,"children":1173},{"class":429,"line":741},[1174,1178,1183],{"type":44,"tag":427,"props":1175,"children":1176},{"style":440},[1177],{"type":50,"value":944},{"type":44,"tag":427,"props":1179,"children":1180},{"style":440},[1181],{"type":50,"value":1182}," accelerate",{"type":44,"tag":427,"props":1184,"children":1185},{"style":701},[1186],{"type":50,"value":704},{"type":44,"tag":427,"props":1188,"children":1189},{"class":429,"line":956},[1190,1194],{"type":44,"tag":427,"props":1191,"children":1192},{"style":440},[1193],{"type":50,"value":962},{"type":44,"tag":427,"props":1195,"children":1196},{"style":701},[1197],{"type":50,"value":704},{"type":44,"tag":427,"props":1199,"children":1200},{"class":429,"line":969},[1201,1205],{"type":44,"tag":427,"props":1202,"children":1203},{"style":440},[1204],{"type":50,"value":1078},{"type":44,"tag":427,"props":1206,"children":1207},{"style":750},[1208],{"type":50,"value":753},{"type":44,"tag":45,"props":1210,"children":1212},{"id":1211},"remote-execution-boundary",[1213],{"type":50,"value":1214},"Remote Execution Boundary",{"type":44,"tag":53,"props":1216,"children":1217},{},[1218,1220,1225],{"type":50,"value":1219},"This skill intentionally stops at ",{"type":44,"tag":59,"props":1221,"children":1222},{},[1223],{"type":50,"value":1224},"local execution and backend selection",{"type":50,"value":65},{"type":44,"tag":53,"props":1227,"children":1228},{},[1229],{"type":50,"value":1230},"If the user wants to:",{"type":44,"tag":72,"props":1232,"children":1233},{},[1234,1239,1244,1249,1254],{"type":44,"tag":76,"props":1235,"children":1236},{},[1237],{"type":50,"value":1238},"run these scripts on Hugging Face Jobs",{"type":44,"tag":76,"props":1240,"children":1241},{},[1242],{"type":50,"value":1243},"pick remote hardware",{"type":44,"tag":76,"props":1245,"children":1246},{},[1247],{"type":50,"value":1248},"pass secrets to remote jobs",{"type":44,"tag":76,"props":1250,"children":1251},{},[1252],{"type":50,"value":1253},"schedule recurring runs",{"type":44,"tag":76,"props":1255,"children":1256},{},[1257],{"type":50,"value":1258},"inspect \u002F cancel \u002F monitor jobs",{"type":44,"tag":53,"props":1260,"children":1261},{},[1262,1264,1272],{"type":50,"value":1263},"then switch to the ",{"type":44,"tag":59,"props":1265,"children":1266},{},[1267],{"type":44,"tag":80,"props":1268,"children":1270},{"className":1269},[],[1271],{"type":50,"value":198},{"type":50,"value":1273}," skill and pass it one of these scripts plus the chosen arguments.",{"type":44,"tag":45,"props":1275,"children":1277},{"id":1276},"task-selection",[1278],{"type":50,"value":1279},"Task Selection",{"type":44,"tag":53,"props":1281,"children":1282},{},[1283,1288],{"type":44,"tag":80,"props":1284,"children":1286},{"className":1285},[],[1287],{"type":50,"value":85},{"type":50,"value":1289}," examples:",{"type":44,"tag":72,"props":1291,"children":1292},{},[1293,1302,1311,1320,1329,1338,1347],{"type":44,"tag":76,"props":1294,"children":1295},{},[1296],{"type":44,"tag":80,"props":1297,"children":1299},{"className":1298},[],[1300],{"type":50,"value":1301},"mmlu",{"type":44,"tag":76,"props":1303,"children":1304},{},[1305],{"type":44,"tag":80,"props":1306,"children":1308},{"className":1307},[],[1309],{"type":50,"value":1310},"gsm8k",{"type":44,"tag":76,"props":1312,"children":1313},{},[1314],{"type":44,"tag":80,"props":1315,"children":1317},{"className":1316},[],[1318],{"type":50,"value":1319},"hellaswag",{"type":44,"tag":76,"props":1321,"children":1322},{},[1323],{"type":44,"tag":80,"props":1324,"children":1326},{"className":1325},[],[1327],{"type":50,"value":1328},"arc_challenge",{"type":44,"tag":76,"props":1330,"children":1331},{},[1332],{"type":44,"tag":80,"props":1333,"children":1335},{"className":1334},[],[1336],{"type":50,"value":1337},"truthfulqa",{"type":44,"tag":76,"props":1339,"children":1340},{},[1341],{"type":44,"tag":80,"props":1342,"children":1344},{"className":1343},[],[1345],{"type":50,"value":1346},"winogrande",{"type":44,"tag":76,"props":1348,"children":1349},{},[1350],{"type":44,"tag":80,"props":1351,"children":1353},{"className":1352},[],[1354],{"type":50,"value":1355},"humaneval",{"type":44,"tag":53,"props":1357,"children":1358},{},[1359,1364,1366,1372],{"type":44,"tag":80,"props":1360,"children":1362},{"className":1361},[],[1363],{"type":50,"value":96},{"type":50,"value":1365}," task strings use ",{"type":44,"tag":80,"props":1367,"children":1369},{"className":1368},[],[1370],{"type":50,"value":1371},"suite|task|num_fewshot",{"type":50,"value":1373},":",{"type":44,"tag":72,"props":1375,"children":1376},{},[1377,1385,1394,1403],{"type":44,"tag":76,"props":1378,"children":1379},{},[1380],{"type":44,"tag":80,"props":1381,"children":1383},{"className":1382},[],[1384],{"type":50,"value":1162},{"type":44,"tag":76,"props":1386,"children":1387},{},[1388],{"type":44,"tag":80,"props":1389,"children":1391},{"className":1390},[],[1392],{"type":50,"value":1393},"leaderboard|gsm8k|5",{"type":44,"tag":76,"props":1395,"children":1396},{},[1397],{"type":44,"tag":80,"props":1398,"children":1400},{"className":1399},[],[1401],{"type":50,"value":1402},"leaderboard|arc_challenge|25",{"type":44,"tag":76,"props":1404,"children":1405},{},[1406],{"type":44,"tag":80,"props":1407,"children":1409},{"className":1408},[],[1410],{"type":50,"value":1411},"lighteval|hellaswag|0",{"type":44,"tag":53,"props":1413,"children":1414},{},[1415,1417,1422,1424,1430],{"type":50,"value":1416},"Multiple ",{"type":44,"tag":80,"props":1418,"children":1420},{"className":1419},[],[1421],{"type":50,"value":96},{"type":50,"value":1423}," tasks can be comma-separated in ",{"type":44,"tag":80,"props":1425,"children":1427},{"className":1426},[],[1428],{"type":50,"value":1429},"--tasks",{"type":50,"value":65},{"type":44,"tag":45,"props":1432,"children":1434},{"id":1433},"backend-selection",[1435],{"type":50,"value":1436},"Backend Selection",{"type":44,"tag":72,"props":1438,"children":1439},{},[1440,1452,1471,1483,1495],{"type":44,"tag":76,"props":1441,"children":1442},{},[1443,1444,1450],{"type":50,"value":388},{"type":44,"tag":80,"props":1445,"children":1447},{"className":1446},[],[1448],{"type":50,"value":1449},"inspect_vllm_uv.py --backend vllm",{"type":50,"value":1451}," for fast GPU inference on supported architectures.",{"type":44,"tag":76,"props":1453,"children":1454},{},[1455,1456,1462,1464,1469],{"type":50,"value":540},{"type":44,"tag":80,"props":1457,"children":1459},{"className":1458},[],[1460],{"type":50,"value":1461},"inspect_vllm_uv.py --backend hf",{"type":50,"value":1463}," when ",{"type":44,"tag":80,"props":1465,"children":1467},{"className":1466},[],[1468],{"type":50,"value":108},{"type":50,"value":1470}," does not support the model.",{"type":44,"tag":76,"props":1472,"children":1473},{},[1474,1475,1481],{"type":50,"value":388},{"type":44,"tag":80,"props":1476,"children":1478},{"className":1477},[],[1479],{"type":50,"value":1480},"lighteval_vllm_uv.py --backend vllm",{"type":50,"value":1482}," for throughput on supported models.",{"type":44,"tag":76,"props":1484,"children":1485},{},[1486,1487,1493],{"type":50,"value":540},{"type":44,"tag":80,"props":1488,"children":1490},{"className":1489},[],[1491],{"type":50,"value":1492},"lighteval_vllm_uv.py --backend accelerate",{"type":50,"value":1494}," as the compatibility fallback.",{"type":44,"tag":76,"props":1496,"children":1497},{},[1498,1499,1505],{"type":50,"value":540},{"type":44,"tag":80,"props":1500,"children":1502},{"className":1501},[],[1503],{"type":50,"value":1504},"inspect_eval_uv.py",{"type":50,"value":1506}," when Inference Providers already cover the model and you do not need direct GPU control.",{"type":44,"tag":45,"props":1508,"children":1510},{"id":1509},"hardware-guidance",[1511],{"type":50,"value":1512},"Hardware Guidance",{"type":44,"tag":242,"props":1514,"children":1515},{},[1516,1532],{"type":44,"tag":246,"props":1517,"children":1518},{},[1519],{"type":44,"tag":250,"props":1520,"children":1521},{},[1522,1527],{"type":44,"tag":254,"props":1523,"children":1524},{},[1525],{"type":50,"value":1526},"Model size",{"type":44,"tag":254,"props":1528,"children":1529},{},[1530],{"type":50,"value":1531},"Suggested local hardware",{"type":44,"tag":265,"props":1533,"children":1534},{},[1535,1552,1569],{"type":44,"tag":250,"props":1536,"children":1537},{},[1538,1547],{"type":44,"tag":272,"props":1539,"children":1540},{},[1541],{"type":44,"tag":80,"props":1542,"children":1544},{"className":1543},[],[1545],{"type":50,"value":1546},"\u003C 3B",{"type":44,"tag":272,"props":1548,"children":1549},{},[1550],{"type":50,"value":1551},"consumer GPU \u002F Apple Silicon \u002F small dev GPU",{"type":44,"tag":250,"props":1553,"children":1554},{},[1555,1564],{"type":44,"tag":272,"props":1556,"children":1557},{},[1558],{"type":44,"tag":80,"props":1559,"children":1561},{"className":1560},[],[1562],{"type":50,"value":1563},"3B - 13B",{"type":44,"tag":272,"props":1565,"children":1566},{},[1567],{"type":50,"value":1568},"stronger local GPU",{"type":44,"tag":250,"props":1570,"children":1571},{},[1572,1581],{"type":44,"tag":272,"props":1573,"children":1574},{},[1575],{"type":44,"tag":80,"props":1576,"children":1578},{"className":1577},[],[1579],{"type":50,"value":1580},"13B+",{"type":44,"tag":272,"props":1582,"children":1583},{},[1584,1586],{"type":50,"value":1585},"high-memory local GPU or hand off to ",{"type":44,"tag":80,"props":1587,"children":1589},{"className":1588},[],[1590],{"type":50,"value":198},{"type":44,"tag":53,"props":1592,"children":1593},{},[1594,1596,1602,1603,1609],{"type":50,"value":1595},"For smoke tests, prefer cheaper local runs plus ",{"type":44,"tag":80,"props":1597,"children":1599},{"className":1598},[],[1600],{"type":50,"value":1601},"--limit",{"type":50,"value":343},{"type":44,"tag":80,"props":1604,"children":1606},{"className":1605},[],[1607],{"type":50,"value":1608},"--max-samples",{"type":50,"value":65},{"type":44,"tag":45,"props":1611,"children":1613},{"id":1612},"troubleshooting",[1614],{"type":50,"value":1615},"Troubleshooting",{"type":44,"tag":72,"props":1617,"children":1618},{},[1619,1663,1711,1729],{"type":44,"tag":76,"props":1620,"children":1621},{},[1622,1624],{"type":50,"value":1623},"CUDA or vLLM OOM:\n",{"type":44,"tag":72,"props":1625,"children":1626},{},[1627,1638,1648,1653],{"type":44,"tag":76,"props":1628,"children":1629},{},[1630,1632],{"type":50,"value":1631},"reduce ",{"type":44,"tag":80,"props":1633,"children":1635},{"className":1634},[],[1636],{"type":50,"value":1637},"--batch-size",{"type":44,"tag":76,"props":1639,"children":1640},{},[1641,1642],{"type":50,"value":1631},{"type":44,"tag":80,"props":1643,"children":1645},{"className":1644},[],[1646],{"type":50,"value":1647},"--gpu-memory-utilization",{"type":44,"tag":76,"props":1649,"children":1650},{},[1651],{"type":50,"value":1652},"switch to a smaller model for the smoke test",{"type":44,"tag":76,"props":1654,"children":1655},{},[1656,1658],{"type":50,"value":1657},"if necessary, hand off to ",{"type":44,"tag":80,"props":1659,"children":1661},{"className":1660},[],[1662],{"type":50,"value":198},{"type":44,"tag":76,"props":1664,"children":1665},{},[1666,1668,1673,1675],{"type":50,"value":1667},"Model unsupported by ",{"type":44,"tag":80,"props":1669,"children":1671},{"className":1670},[],[1672],{"type":50,"value":108},{"type":50,"value":1674},":\n",{"type":44,"tag":72,"props":1676,"children":1677},{},[1678,1695],{"type":44,"tag":76,"props":1679,"children":1680},{},[1681,1683,1688,1690],{"type":50,"value":1682},"switch to ",{"type":44,"tag":80,"props":1684,"children":1686},{"className":1685},[],[1687],{"type":50,"value":588},{"type":50,"value":1689}," for ",{"type":44,"tag":80,"props":1691,"children":1693},{"className":1692},[],[1694],{"type":50,"value":85},{"type":44,"tag":76,"props":1696,"children":1697},{},[1698,1699,1705,1706],{"type":50,"value":1682},{"type":44,"tag":80,"props":1700,"children":1702},{"className":1701},[],[1703],{"type":50,"value":1704},"--backend accelerate",{"type":50,"value":1689},{"type":44,"tag":80,"props":1707,"children":1709},{"className":1708},[],[1710],{"type":50,"value":96},{"type":44,"tag":76,"props":1712,"children":1713},{},[1714,1716],{"type":50,"value":1715},"Gated\u002Fprivate repo access fails:\n",{"type":44,"tag":72,"props":1717,"children":1718},{},[1719],{"type":44,"tag":76,"props":1720,"children":1721},{},[1722,1724],{"type":50,"value":1723},"verify ",{"type":44,"tag":80,"props":1725,"children":1727},{"className":1726},[],[1728],{"type":50,"value":407},{"type":44,"tag":76,"props":1730,"children":1731},{},[1732,1734],{"type":50,"value":1733},"Custom model code required:\n",{"type":44,"tag":72,"props":1735,"children":1736},{},[1737],{"type":44,"tag":76,"props":1738,"children":1739},{},[1740,1742],{"type":50,"value":1741},"add ",{"type":44,"tag":80,"props":1743,"children":1745},{"className":1744},[],[1746],{"type":50,"value":1747},"--trust-remote-code",{"type":44,"tag":45,"props":1749,"children":1751},{"id":1750},"examples",[1752],{"type":50,"value":1753},"Examples",{"type":44,"tag":53,"props":1755,"children":1756},{},[1757],{"type":50,"value":1758},"See:",{"type":44,"tag":72,"props":1760,"children":1761},{},[1762,1772,1780,1788],{"type":44,"tag":76,"props":1763,"children":1764},{},[1765,1770],{"type":44,"tag":80,"props":1766,"children":1768},{"className":1767},[],[1769],{"type":50,"value":374},{"type":50,"value":1771}," for local command patterns",{"type":44,"tag":76,"props":1773,"children":1774},{},[1775],{"type":44,"tag":80,"props":1776,"children":1778},{"className":1777},[],[1779],{"type":50,"value":292},{"type":44,"tag":76,"props":1781,"children":1782},{},[1783],{"type":44,"tag":80,"props":1784,"children":1786},{"className":1785},[],[1787],{"type":50,"value":323},{"type":44,"tag":76,"props":1789,"children":1790},{},[1791],{"type":44,"tag":80,"props":1792,"children":1794},{"className":1793},[],[1795],{"type":50,"value":357},{"type":44,"tag":1797,"props":1798,"children":1799},"style",{},[1800],{"type":50,"value":1801},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1803,"total":1974},[1804,1822,1838,1853,1867,1880,1893,1908,1922,1932,1945,1959],{"slug":1805,"name":1805,"fn":1806,"description":1807,"org":1808,"tags":1809,"stars":1819,"repoUrl":1820,"updatedAt":1821},"train-sentence-transformers","train sentence-transformers models","Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder; dense or static embedding model; for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker; pair scoring for two-stage retrieval \u002F pair classification), and `SparseEncoder` (SPLADE, sparse embedding model; for learned-sparse retrieval). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, Matryoshka, and Hugging Face Hub publishing. Use for any sentence-transformers training task.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1810,1811,1812,1813,1816],{"name":19,"slug":20,"type":14},{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"name":1814,"slug":1815,"type":14},"Python","python",{"name":1817,"slug":1818,"type":14},"Search","search",18914,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fsentence-transformers","2026-05-08T05:09:16.820066",{"slug":1823,"name":1823,"fn":1824,"description":1825,"org":1826,"tags":1827,"stars":1835,"repoUrl":1836,"updatedAt":1837},"trl-training","train and fine-tune LLMs with TRL","Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1828,1831,1832,1833,1834],{"name":1829,"slug":1830,"type":14},"AI Infrastructure","ai-infrastructure",{"name":19,"slug":20,"type":14},{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"name":1814,"slug":1815,"type":14},18850,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl","2026-04-06T18:25:32.746828",{"slug":1839,"name":1839,"fn":1840,"description":1841,"org":1842,"tags":1843,"stars":27,"repoUrl":28,"updatedAt":1852},"hf-cli","manage Hugging Face Hub resources via CLI","Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing models, datasets, spaces, buckets, repos, papers, jobs, and more on the Hugging Face Hub. Use when: handling authentication; managing local cache; managing Hugging Face Buckets; running or scheduling jobs on Hugging Face infrastructure; managing Hugging Face repos; discussions and pull requests; browsing models, datasets and spaces; reading, searching, or browsing academic papers; managing collections; querying datasets; configuring spaces; setting up webhooks; or deploying and managing HF Inference Endpoints. Make sure to use this skill whenever the user mentions 'hf', 'huggingface', 'Hugging Face', 'huggingface-cli', or 'hugging face cli', or wants to do anything related to the Hugging Face ecosystem and to AI and ML in general. Also use for cloud storage needs like training checkpoints, data pipelines, or agent traces. Use even if the user doesn't explicitly ask for a CLI command. Replaces the deprecated `huggingface-cli`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1844,1847,1850,1851],{"name":1845,"slug":1846,"type":14},"CLI","cli",{"name":1848,"slug":1849,"type":14},"Datasets","datasets",{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},"2026-04-06T18:25:34.020855",{"slug":1854,"name":1854,"fn":1855,"description":1856,"org":1857,"tags":1858,"stars":27,"repoUrl":28,"updatedAt":1866},"hf-cloud-aws-context-discovery","discover local AWS environment context","Discover the user's local AWS context (active profile, region, account ID, caller identity) at the start of any AWS task. Use this skill before any other AWS work — deploying to SageMaker, creating resources, calling AWS APIs, or anything that touches an AWS account. Use it especially when the user has not specified a region or profile explicitly, when they say things like \"use my AWS account\", \"deploy to AWS\", \"use my profile\", or when about to make any AWS CLI or SDK call. Never guess the region or account ID — always use this skill to read it from the local configuration first.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1859,1862,1863],{"name":1860,"slug":1861,"type":14},"AWS","aws",{"name":1845,"slug":1846,"type":14},{"name":1864,"slug":1865,"type":14},"Configuration","configuration","2026-07-08T05:55:33.716099",{"slug":1868,"name":1868,"fn":1869,"description":1870,"org":1871,"tags":1872,"stars":27,"repoUrl":28,"updatedAt":1879},"hf-cloud-python-env-setup","set up Python environments for AWS","Set up an isolated Python environment for SageMaker \u002F AWS work, with the right Python version and current boto3. Use this skill whenever Python code will be executed for a SageMaker deployment, training job, or any AWS automation — including when about to run `pip install`, when about to invoke `boto3`, when creating or activating a virtualenv, or when the user asks to \"set up the environment\". Never use system Python and never `pip install` into it. Always isolate. This skill prevents the most common failure modes: wrong Python version, dependency conflicts, and stale SDKs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1873,1874,1875,1878],{"name":1829,"slug":1830,"type":14},{"name":1860,"slug":1861,"type":14},{"name":1876,"slug":1877,"type":14},"Engineering","engineering",{"name":1814,"slug":1815,"type":14},"2026-07-08T05:55:32.505017",{"slug":1881,"name":1881,"fn":1882,"description":1883,"org":1884,"tags":1885,"stars":27,"repoUrl":28,"updatedAt":1892},"hf-cloud-sagemaker-deployment-planner","plan model deployments to Amazon SageMaker","Plan and coordinate the deployment of a model to Amazon SageMaker AI. Use this skill whenever the user wants to deploy, host, serve, or expose a model on SageMaker or AWS — including phrases like \"deploy a model\", \"host this LLM on AWS\", \"serve this embedding model\", \"deploy a reranker\", \"deploy a text-to-image \u002F diffusion model\", \"host this for async inference\", \"create an endpoint\", \"serve my fine-tuned model\", or any request that involves making a model available for inference on AWS. Use this even when the user is vague (e.g. \"I just want to get this running on AWS, you figure it out\"). Works for text-generation LLMs, embedding models, rerankers, classifiers, text-to-image \u002F diffusion models — picks the right serving stack and chooses between real-time and async inference. This is the entry-point skill for SageMaker deployment work — it asks clarifying questions, picks a deployment pathway, and coordinates the other deployment skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1886,1887,1888,1891],{"name":1829,"slug":1830,"type":14},{"name":1860,"slug":1861,"type":14},{"name":1889,"slug":1890,"type":14},"Deployment","deployment",{"name":9,"slug":13,"type":14},"2026-07-08T05:55:37.387689",{"slug":1894,"name":1894,"fn":1895,"description":1896,"org":1897,"tags":1898,"stars":27,"repoUrl":28,"updatedAt":1907},"hf-cloud-sagemaker-iam-preflight","configure SageMaker IAM roles","Ensure a usable SageMaker execution role exists before deploying or training. Use this skill whenever about to create a SageMaker endpoint, model, training job, or any resource that requires an execution role. Use it especially when the user has not provided a role ARN explicitly, when scripts are about to call `iam:CreateRole`, or when an AccessDenied error mentions an IAM action. Never blindly call `iam:CreateRole` — always check for existing roles first. This skill prevents the most common SageMaker deployment failure: trying to create IAM resources from an SSO principal that has no IAM write permissions.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1899,1900,1901,1904],{"name":1829,"slug":1830,"type":14},{"name":1860,"slug":1861,"type":14},{"name":1902,"slug":1903,"type":14},"Permissions","permissions",{"name":1905,"slug":1906,"type":14},"Security","security","2026-07-08T05:55:34.948771",{"slug":1909,"name":1909,"fn":1910,"description":1911,"org":1912,"tags":1913,"stars":27,"repoUrl":28,"updatedAt":1921},"hf-cloud-sagemaker-production-defaults","create production-ready SageMaker endpoints","Create a SageMaker endpoint (real-time or async) with autoscaling, CloudWatch alarms, and tagging enabled by default. Use this skill whenever about to create a SageMaker endpoint, write deployment code that calls `create_endpoint`, or finalize a deployment after the image URI and IAM role are known. Provides deploy.py for real-time endpoints and deploy_async.py for async endpoints (with genuine scale-to-zero support). This is the last step in the SageMaker deployment workflow. Never generate a bare `create_endpoint` call without these defaults — endpoints without autoscaling or alarms are demos, not deployments.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1914,1915,1916,1917,1918],{"name":1829,"slug":1830,"type":14},{"name":1860,"slug":1861,"type":14},{"name":1889,"slug":1890,"type":14},{"name":9,"slug":13,"type":14},{"name":1919,"slug":1920,"type":14},"Monitoring","monitoring","2026-07-08T05:55:38.664702",{"slug":1923,"name":1923,"fn":1924,"description":1925,"org":1926,"tags":1927,"stars":27,"repoUrl":28,"updatedAt":1931},"hf-cloud-serving-image-selection","select SageMaker serving containers","Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMaker endpoint and an image URI needs to be chosen — including when the user says \"deploy this LLM\", \"host this HuggingFace model\", \"serve this fine-tuned model\", \"deploy this embedding model\", \"host a reranker\", \"serve a sentence-transformers model\", or when about to hardcode any container URI in deployment code. HuggingFace-curated Deep Learning Containers are ALWAYS preferred: HuggingFace vLLM (LLMs and generative rerankers), HuggingFace vLLM-Omni (multimodal), TEI (embeddings\u002Fcross-encoder rerankers), HF Inference Toolkit (other transformers). Generic images (AWS vLLM, DJL-LMI, SGLang) are used only when no HuggingFace image is compatible — never merely because they carry a newer version. Never hardcode a container URI from memory and never default to TGI. Prevents stale-image failures and wrong-region URIs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1928,1929,1930],{"name":1829,"slug":1830,"type":14},{"name":1860,"slug":1861,"type":14},{"name":1889,"slug":1890,"type":14},"2026-07-08T05:55:36.173465",{"slug":1933,"name":1933,"fn":1934,"description":1935,"org":1936,"tags":1937,"stars":27,"repoUrl":28,"updatedAt":1944},"hf-mcp","access Hugging Face Hub via MCP","Use Hugging Face Hub via MCP server tools. Search models, datasets, Spaces, papers. Get repo details, fetch documentation, run compute jobs, and use Gradio Spaces as AI tools. Available when connected to the HF MCP server.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1938,1939,1940,1941],{"name":1848,"slug":1849,"type":14},{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"name":1942,"slug":1943,"type":14},"MCP","mcp","2026-04-06T18:25:50.364185",{"slug":1946,"name":1946,"fn":1947,"description":1948,"org":1949,"tags":1950,"stars":27,"repoUrl":28,"updatedAt":1958},"hf-mem","estimate memory for Hugging Face models","Hugging Face CLI to estimate the required memory to load Safetensors or GGUF model weights for inference from the Hugging Face Hub",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1951,1952,1953,1954,1955],{"name":1829,"slug":1830,"type":14},{"name":1845,"slug":1846,"type":14},{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"name":1956,"slug":1957,"type":14},"Performance","performance","2026-06-13T07:23:57.101435",{"slug":1960,"name":1960,"fn":1961,"description":1962,"org":1963,"tags":1964,"stars":27,"repoUrl":28,"updatedAt":1973},"huggingface-best","find and compare Hugging Face models","Use when the user asks about finding the best, top, or recommended model for a task, wants to know what AI model to use, or wants to compare models by benchmark scores. Triggers on: \"best model for X\", \"what model should I use for\", \"top models for [task]\", \"which model runs on my laptop\u002Fmachine\u002Fdevice\", \"recommend a model for\", \"what LLM should I use for\", \"compare models for\", \"what's state of the art for\", or any question about choosing an AI model for a specific use case. Always use this skill when the user wants model recommendations or comparisons, even if they don't explicitly mention HuggingFace or benchmarks.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1965,1968,1969,1970],{"name":1966,"slug":1967,"type":14},"Analytics","analytics",{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"name":1971,"slug":1972,"type":14},"Research","research","2026-04-24T05:09:45.870658",37,{"items":1976,"total":2025},[1977,1984,1990,1997,2004,2011,2019],{"slug":1839,"name":1839,"fn":1840,"description":1841,"org":1978,"tags":1979,"stars":27,"repoUrl":28,"updatedAt":1852},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1980,1981,1982,1983],{"name":1845,"slug":1846,"type":14},{"name":1848,"slug":1849,"type":14},{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"slug":1854,"name":1854,"fn":1855,"description":1856,"org":1985,"tags":1986,"stars":27,"repoUrl":28,"updatedAt":1866},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1987,1988,1989],{"name":1860,"slug":1861,"type":14},{"name":1845,"slug":1846,"type":14},{"name":1864,"slug":1865,"type":14},{"slug":1868,"name":1868,"fn":1869,"description":1870,"org":1991,"tags":1992,"stars":27,"repoUrl":28,"updatedAt":1879},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[1993,1994,1995,1996],{"name":1829,"slug":1830,"type":14},{"name":1860,"slug":1861,"type":14},{"name":1876,"slug":1877,"type":14},{"name":1814,"slug":1815,"type":14},{"slug":1881,"name":1881,"fn":1882,"description":1883,"org":1998,"tags":1999,"stars":27,"repoUrl":28,"updatedAt":1892},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2000,2001,2002,2003],{"name":1829,"slug":1830,"type":14},{"name":1860,"slug":1861,"type":14},{"name":1889,"slug":1890,"type":14},{"name":9,"slug":13,"type":14},{"slug":1894,"name":1894,"fn":1895,"description":1896,"org":2005,"tags":2006,"stars":27,"repoUrl":28,"updatedAt":1907},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2007,2008,2009,2010],{"name":1829,"slug":1830,"type":14},{"name":1860,"slug":1861,"type":14},{"name":1902,"slug":1903,"type":14},{"name":1905,"slug":1906,"type":14},{"slug":1909,"name":1909,"fn":1910,"description":1911,"org":2012,"tags":2013,"stars":27,"repoUrl":28,"updatedAt":1921},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2014,2015,2016,2017,2018],{"name":1829,"slug":1830,"type":14},{"name":1860,"slug":1861,"type":14},{"name":1889,"slug":1890,"type":14},{"name":9,"slug":13,"type":14},{"name":1919,"slug":1920,"type":14},{"slug":1923,"name":1923,"fn":1924,"description":1925,"org":2020,"tags":2021,"stars":27,"repoUrl":28,"updatedAt":1931},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2022,2023,2024],{"name":1829,"slug":1830,"type":14},{"name":1860,"slug":1861,"type":14},{"name":1889,"slug":1890,"type":14},24]