[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-huggingface-hf-cloud-serving-image-selection":3,"mdc-81gjc1-key":33,"related-org-huggingface-hf-cloud-serving-image-selection":2616,"related-repo-huggingface-hf-cloud-serving-image-selection":2782},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":22,"repoUrl":23,"updatedAt":24,"license":25,"forks":26,"topics":27,"repo":28,"sourceUrl":31,"mdContent":32},"hf-cloud-serving-image-selection","select SageMaker serving containers","Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMaker endpoint and an image URI needs to be chosen — including when the user says \"deploy this LLM\", \"host this HuggingFace model\", \"serve this fine-tuned model\", \"deploy this embedding model\", \"host a reranker\", \"serve a sentence-transformers model\", or when about to hardcode any container URI in deployment code. HuggingFace-curated Deep Learning Containers are ALWAYS preferred: HuggingFace vLLM (LLMs and generative rerankers), HuggingFace vLLM-Omni (multimodal), TEI (embeddings\u002Fcross-encoder rerankers), HF Inference Toolkit (other transformers). Generic images (AWS vLLM, DJL-LMI, SGLang) are used only when no HuggingFace image is compatible — never merely because they carry a newer version. Never hardcode a container URI from memory and never default to TGI. Prevents stale-image failures and wrong-region URIs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"huggingface","Hugging Face","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fhuggingface.png",[12,16,19],{"name":13,"slug":14,"type":15},"Deployment","deployment","tag",{"name":17,"slug":18,"type":15},"AI Infrastructure","ai-infrastructure",{"name":20,"slug":21,"type":15},"AWS","aws",10861,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fskills","2026-07-08T05:55:36.173465",null,721,[],{"repoUrl":23,"stars":22,"forks":26,"topics":29,"description":30},[],"Give your agents the power of the Hugging Face ecosystem","https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fhf-cloud-serving-image-selection","---\nname: hf-cloud-serving-image-selection\ndescription: 'Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMaker endpoint and an image URI needs to be chosen — including when the user says \"deploy this LLM\", \"host this HuggingFace model\", \"serve this fine-tuned model\", \"deploy this embedding model\", \"host a reranker\", \"serve a sentence-transformers model\", or when about to hardcode any container URI in deployment code. HuggingFace-curated Deep Learning Containers are ALWAYS preferred: HuggingFace vLLM (LLMs and generative rerankers), HuggingFace vLLM-Omni (multimodal), TEI (embeddings\u002Fcross-encoder rerankers), HF Inference Toolkit (other transformers). Generic images (AWS vLLM, DJL-LMI, SGLang) are used only when no HuggingFace image is compatible — never merely because they carry a newer version. Never hardcode a container URI from memory and never default to TGI. Prevents stale-image failures and wrong-region URIs.'\n---\n\n# Serving Image Selection\n\nThe serving container is the single thing most likely to break a SageMaker deployment that \"looked correct on paper\". Wrong container, stale tag, or the wrong AMI — all produce the same opaque `Failed to pass health check` error.\n\n## Rule zero: HuggingFace images always win\n\nWhen both a HuggingFace-curated family (`huggingface-vllm`, `huggingface-vllm-omni`, `huggingface-sglang`, `tei`, `huggingface-pytorch-inference`) and a generic family (`vllm`, `vllm-omni`, `sglang`, `djl-inference`) can serve the model, **the HuggingFace one is mandatory, not preferred**. The only valid reasons to use a generic image:\n\n1. **Verified incompatibility** — the model needs an architecture\u002Fmodality\u002Ffeature no available HuggingFace tag supports, confirmed against the catalog (not assumed).\n2. **No HuggingFace tag exists in the target region** and mirroring is not an option.\n3. **The HuggingFace image is in \"Known-broken images\"** below.\n\nA **newer version number on the generic repo is not a reason**. The AWS `vllm` repo often publishes a higher vLLM version than `huggingface-vllm`; an older-but-compatible `huggingface-vllm` tag still wins. \"Latest vLLM\" is not a requirement anyone stated — compatibility with the model is. If you fall back, record in the deployment log which of the three reasons applied.\n\n## Where image URIs come from\n\n**Primary source: AWS's official Deep Learning Containers catalog.**\n\nURL: https:\u002F\u002Faws.github.io\u002Fdeep-learning-containers\u002Freference\u002Favailable_images\u002F\n\nThis page is AWS-maintained and lists every image family with example URIs, tags, CUDA versions, Python versions, and platform (SageMaker vs EC2\u002FECS\u002FEKS). When picking a URI for a deployment, **read it from this page directly** — copy the example URL, substitute `\u003Cregion>` with the user's region, and pass it to `deploy.py --image-uri`.\n\nThe example URLs use `763104351884` as the account ID for most regions. A few regions use different accounts (e.g. `eu-south-1` uses `692866216735`). Check the [Region Availability page](https:\u002F\u002Faws.github.io\u002Fdeep-learning-containers\u002Freference\u002Fregion_availability\u002F) when in doubt.\n\n**Exception: none currently.** Every image family used by this workflow is now on the AWS catalog page (TEI was added in late 2026). If you encounter a new family that isn't there, mirror it via `mirror_image.py` and pass the resulting URI directly.\n\n## Quick decision\n\n| Model | Container family | How to get the URI |\n|---|---|---|\n| HuggingFace text-generation LLM (Llama, Qwen, Mistral, etc.) | **HuggingFace vLLM** | AWS catalog → \"HuggingFace vLLM Inference\" (ECR repo `huggingface-vllm`) |\n| Same as above, multimodal | **HuggingFace vLLM-Omni** | AWS catalog → \"HuggingFace vLLM-Omni Inference\" (ECR repo `huggingface-vllm-omni`) |\n| HuggingFace embeddings | TEI | AWS catalog → \"HuggingFace Text Embeddings Inference\" |\n| Encoder \u002F cross-encoder rerankers (BERT-family `*ForSequenceClassification`) | TEI | Same as embeddings |\n| **Generative rerankers** (causal-LM, e.g. Qwen3-Reranker) | HuggingFace vLLM | Same as text-generation LLMs — **not TEI**, see \"Rerankers: TEI or vLLM?\" |\n| Text-to-image \u002F diffusion (Stable Diffusion, FLUX) | DJL Inference | AWS catalog → \"DJL Inference\" — **not** HF Inference Toolkit, see \"Known-broken images\" |\n| HuggingFace classifiers, NER, QA, summarization | HF Inference Toolkit (CPU) | AWS catalog → \"HuggingFace PyTorch Inference\"; GPU tags currently broken — see \"Known-broken images\" |\n| User specifically wants SGLang | HuggingFace SGLang | AWS catalog → \"HuggingFace SGLang Inference\" |\n| No compatible `huggingface-vllm` tag (verified incompatibility or region gap — see \"Rule zero\") | vLLM (AWS) | AWS catalog → \"vLLM\" section — fallback only, never for version freshness |\n| User specifically wants DJL-LMI | DJL Inference | AWS catalog → \"DJL Inference\" |\n| Amazon Nova | SageMaker JumpStart | Use JumpStart, not raw endpoint creation |\n| Custom inference code | BYOC | User provides URI |\n\n**HuggingFace-curated DLCs are mandatory when one is compatible (see \"Rule zero\").** `huggingface-vllm` is layered directly on the AWS vLLM DLC — **identical `SM_VLLM_*` env contract and the same cu130 AMI rule** — and adds current `transformers`, current `huggingface_hub` + `hf_xet` (avoids the XET-CDN 403 download failures older images hit), and HF performance defaults. It is also what SageMaker SDK v3 auto-routes to. The AWS `vllm` image is a compatibility escape hatch only; it usually shows a higher vLLM version than `huggingface-vllm`, and that is not a reason to pick it.\n\n**Do not use TGI.** Text Generation Inference is archived. Models released after the archive (Qwen3 most famously) fail ping health checks on TGI. Use vLLM instead. (The SageMaker SDK v3 agrees: since [PR #5960](https:\u002F\u002Fgithub.com\u002Faws\u002Fsagemaker-python-sdk\u002Fpull\u002F5960), June 2026, its `ModelBuilder` auto-routes `text-generation` to the HuggingFace vLLM DLC and multimodal tasks to HuggingFace vLLM-Omni.)\n\nFull reasoning for each family in `references\u002Fmodel-to-image.md`.\n\n## Rerankers: TEI or vLLM?\n\n\"Reranker\" covers two very different architectures, and picking wrong wastes a full endpoint-creation cycle (~20 min) before TEI rejects the model:\n\n- **Encoder cross-encoders** (BAAI\u002Fbge-reranker-*, mixedbread, most `sentence-transformers` rerankers) — BERT-family models with a classification head. `config.json` has `architectures: [..ForSequenceClassification]` on a TEI-supported encoder type. → **TEI**.\n- **Generative rerankers** (Qwen\u002FQwen3-Reranker-*, and similar causal-LM judges) — decoder LLMs that score relevance via the logprob of a yes\u002Fno token. `config.json` has `architectures: [..ForCausalLM]`. → **HuggingFace vLLM**, deployed exactly like a text-generation LLM. TEI will load the architecture then reject the `classifier` model type (Qwen3 support in TEI is *embeddings-only*). Invocation pattern (raw completions API, `max_tokens=1`, logprobs scoring) is in `hf-cloud-sagemaker-production-defaults`.\n\n**Preflight before creating any resources** — one HTTP GET settles it:\n\n```bash\ncurl -s https:\u002F\u002Fhuggingface.co\u002F\u003Cmodel-id>\u002Fraw\u002Fmain\u002Fconfig.json\n# \"architectures\": [\"Qwen3ForCausalLM\"]              → vLLM\n# \"architectures\": [\"XLMRobertaForSequenceClassification\"] → TEI\n```\n\nFor TEI also confirm the *(architecture, task)* pair: an architecture appearing in TEI's supported list means embeddings support, not necessarily classification\u002Freranking support.\n\nHeads-up: SageMaker SDK v3 (PR #5960) routes the `text-ranking` task to TEI **unconditionally** — correct for cross-encoders, wrong for generative rerankers. Don't treat the SDK's routing as evidence that TEI can serve a given reranker.\n\n## Workflow\n\nFor every family: **read the URI from the AWS catalog page**.\n\n1. Open https:\u002F\u002Faws.github.io\u002Fdeep-learning-containers\u002Freference\u002Favailable_images\u002F\n2. Find the section for the right family (e.g. \"HuggingFace vLLM Inference\" for HuggingFace LLMs, \"HuggingFace Text Embeddings Inference\" for embeddings)\n3. Pick the newest row marked `SageMaker` for the platform column — newest **within that family**. Do not switch to another family's section because it lists a higher engine version (see \"Rule zero\")\n4. Substitute `\u003Cregion>` with the user's region (from `hf-cloud-aws-context-discovery`)\n5. For vLLM: also check the AMI requirement (see \"vLLM AMI requirement\" below)\n6. Pass the URI to `deploy.py --image-uri` (real-time) or `deploy_async.py --image-uri` (async)\n\n### TEI: pick the right variant\n\nThe TEI catalog row lists two URIs — GPU (`tei` repo) and CPU (`tei-cpu` repo). Pick based on the instance type:\n\n- `ml.g*`, `ml.p*`, `ml.inf*` → GPU variant\n- `ml.c*`, `ml.m*`, `ml.t*` → CPU variant\n\nMixing them fails: CPU image on a GPU instance wastes hardware, GPU image on a CPU instance fails to start.\n\n**Note on the TEI account ID**: the catalog page shows `683313688378` as the example account, but TEI is published from a different account namespace than the main AWS DLCs and the per-region account IDs vary. If `683313688378.dkr.ecr.\u003Cregion>.amazonaws.com\u002Ftei:...` returns an ECR pull error for a region other than us-east-1, check the [Region Availability page](https:\u002F\u002Faws.github.io\u002Fdeep-learning-containers\u002Freference\u002Fregion_availability\u002F) for the correct account ID for that region.\n\n## vLLM AMI requirement\n\nvLLM DLC images with **CUDA 13 or higher** (current default: `cu130`) require setting `InferenceAmiVersion=al2-ami-sagemaker-inference-gpu-3-1` on the ProductionVariant. This applies equally to `huggingface-vllm` and `huggingface-vllm-omni` (layered on the same cu130 base) and to the AWS `vllm` repo. Without it the container dies on startup with no CloudWatch logs ever created. The failure looks identical to many other things (account-level issues, quota, networking) and routinely sends people down wrong diagnostic paths.\n\nLookup table:\n\n| Tag contains | InferenceAmiVersion to pass |\n|---|---|\n| `cu130` (or higher) | `al2-ami-sagemaker-inference-gpu-3-1` |\n| `cu129` or lower | (omit the flag; default AMI works) |\n\nRule of thumb: if the vLLM tag you picked contains `cu130` or later, pass `--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1` to `deploy.py`. If a future CUDA version (cu140+) needs a different AMI, add a row to the table when AWS publishes the new image.\n\nThis is a vLLM-specific concern. TEI and HF Inference Toolkit images don't need an AMI override.\n\n## Configuring the vLLM DLCs (HuggingFace vLLM and AWS vLLM)\n\nBoth images share the same contract: configuration as environment variables on the SageMaker model definition, `SM_VLLM_*` mapped to vLLM CLI flags. The `huggingface-vllm` entrypoint additionally auto-detects the model when `SM_VLLM_MODEL` is unset — from `\u002Fopt\u002Fml\u002Fmodel` if artifacts are mounted, else from `HF_MODEL_ID` — but setting `SM_VLLM_MODEL` explicitly works on both and is what our examples use.\n\n### Required for every HuggingFace LLM deployment\n\n| Env var | Purpose | Notes |\n|---|---|---|\n| `SM_VLLM_MODEL` | HF model ID (e.g. `Qwen\u002FQwen3-0.6B`) or `\u002Fopt\u002Fml\u002Fmodel` if loading from S3 | — |\n| `SM_VLLM_HOST` | **Must be `0.0.0.0`** | Otherwise vLLM binds localhost only, ping fails, container dies before logs. Top cause of mystery failures with this image. |\n| `SM_VLLM_TRUST_REMOTE_CODE` | `true` for Qwen and several recent architectures | Set unconditionally — downside negligible, upside is the model loads. |\n| `HUGGING_FACE_HUB_TOKEN` | HF token | Required for gated models. |\n\n### Tuning (optional)\n\n| Env var | Purpose |\n|---|---|\n| `SM_VLLM_MAX_MODEL_LEN` | Max sequence length — set this; defaults can be wrong for fine-tunes |\n| `SM_VLLM_GPU_MEMORY_UTILIZATION` | Float 0.0–1.0, ~0.9 reasonable |\n| `SM_VLLM_TENSOR_PARALLEL_SIZE` | GPU count for multi-GPU instances |\n| `SM_VLLM_DTYPE` | `auto`, `bfloat16`, `float16` |\n\nAny vLLM CLI flag works — uppercase, replace dashes with underscores, prepend `SM_VLLM_`.\n\n## Configuring TEI\n\nSimpler env contract than vLLM:\n\n| Env var | Purpose | Required |\n|---|---|---|\n| `HF_MODEL_ID` | HF model ID (e.g. `BAAI\u002Fbge-large-en-v1.5`) or `\u002Fopt\u002Fml\u002Fmodel` | Yes |\n| `HF_TOKEN` | HF auth token | Only for gated models |\n| `MAX_BATCH_TOKENS` | Max tokens per batch (default 16384) | No |\n| `MAX_CLIENT_BATCH_SIZE` | Max requests per client batch (default 32) | No |\n\nNo host-binding to configure, no trust-remote-code flag. The architectures TEI supports (BERT, CamemBERT, RoBERTa, XLM-RoBERTa, NomicBert, JinaBert, JinaCodeBert, Mistral, Qwen2\u002F3, Gemma2\u002F3, ModernBert) are baked into the image.\n\n## CUDA \u002F instance compatibility\n\nCritical and easy to get wrong:\n\n| CUDA in image tag | Default AMI | With `al2-ami-sagemaker-inference-gpu-3-1` |\n|---|---|---|\n| cu124 \u002F cu128 | g5, g6, p5 all work | (not needed) |\n| cu129 | g6, p5; g5 fails (driver mismatch → CannotStartContainerError) | expected to fix g5 (unverified) |\n| cu130+ | fails everywhere — AMI flag is mandatory | g5, g6, p5 all work (cu130-on-g5 verified June 2026) |\n\nThe driver comes from the host AMI, not the instance family — so passing the gpu-3-1 AMI (which vLLM cu130 images require anyway) also makes `ml.g5.*` viable for cu129+ images.\n\n## VPC \u002F NAT gateway problem\n\nSageMaker endpoints inside a VPC **without** a NAT gateway can't pull from `public.ecr.aws`. The deployment fails with an image-pull error that doesn't mention \"VPC\" or \"egress\".\n\nFor images on AWS's regional ECR (everything in the catalog): SageMaker reaches them through built-in routing, no NAT needed. Use the regional URI pattern (`\u003Caccount>.dkr.ecr.\u003Cregion>.amazonaws.com\u002F...`), not the `public.ecr.aws\u002F...` pattern.\n\nFor images requiring `public.ecr.aws` access (less common): mirror to a private ECR repo in your account with `scripts\u002Fmirror_image.py` (cross-platform; needs Docker + the `aws` CLI). Run it from the shell where the AWS CLI works.\n\n```bash\n# macOS \u002F Linux\nPRIVATE_URI=$(python3 scripts\u002Fmirror_image.py \\\n    public.ecr.aws\u002Fdeep-learning-containers\u002Fvllm:\u003Ctag> \\\n    vllm-mirror)\n```\n\n```powershell\n# Windows (PowerShell) — capture stdout into a variable\n$PRIVATE_URI = python scripts\\mirror_image.py `\n    public.ecr.aws\u002Fdeep-learning-containers\u002Fvllm:\u003Ctag> vllm-mirror\n```\n\n## When the catalog page won't render, is stale, or is wrong\n\n**The page won't render \u002F fetch returns junk**: the catalog page is JavaScript-heavy and some fetch tools get an empty shell. Fallbacks, in order:\n\n1. **The catalog's source data on GitHub** — the page is generated from one YAML file per version, listing exact tags, CUDA, and Python versions. List a family's files, then fetch the newest:\n   ```bash\n   curl -s https:\u002F\u002Fapi.github.com\u002Frepos\u002Faws\u002Fdeep-learning-containers\u002Fcontents\u002Fdocs\u002Fsrc\u002Fdata\u002Fhuggingface-vllm\n   curl -s https:\u002F\u002Fraw.githubusercontent.com\u002Faws\u002Fdeep-learning-containers\u002Fmain\u002Fdocs\u002Fsrc\u002Fdata\u002Fhuggingface-vllm\u002F0.21.0-gpu-sagemaker.yml\n   ```\n   Directory names match ECR repos (`huggingface-vllm`, `huggingface-vllm-omni`, `huggingface-tei`, `vllm`, `djl-inference`, ...).\n2. **Query ECR directly** for current tags in the target region (works with credentials that can read the DLC registry; if it returns AccessDenied, use the YAML files):\n   ```bash\n   aws ecr describe-images --registry-id 763104351884 --repository-name huggingface-vllm \\\n       --region \u003Cregion> --query 'sort_by(imageDetails,&imagePushedAt)[-5:].imageTags' --output json\n   ```\n3. [Release notes on the DLC GitHub repo](https:\u002F\u002Fgithub.com\u002Faws\u002Fdeep-learning-containers\u002Freleases).\n\n**A tag was just released and isn't on the page yet**: rare; AWS updates the page on each release. Check the release notes above.\n\n**An architecture you need isn't supported by the listed image yet**: for TEI specifically, you can mirror the upstream image from GHCR (`ghcr.io\u002Fhuggingface\u002Ftext-embeddings-inference:\u003Cversion>`) into private ECR and pass the resulting URI directly to `deploy.py --image-uri`. Same `mirror_image.py` script.\n\n## Known-broken images (last checked July 2026)\n\n| Image | Defect | Use instead |\n|---|---|---|\n| `huggingface-pytorch-inference` **GPU** tags — all recent ones tested (PT 2.3–2.6, cu121\u002Fcu124, transformers 4.48–5.5.3) | `ImportError: libtorch_cuda.so: undefined symbol: ncclCommResume` at `import torch`. The NCCL bundled in the image is older than what torch links against — a packaging defect *inside the container*, on g5 **and** g6, regardless of AMI, model, or inference code. The MMS Java front-end keeps answering `\u002Fping`, so the endpoint can reach InService while the Python worker crash-loops and serves nothing. | DJL Inference (bundles its own complete CUDA\u002FNCCL stack) or BYOC. CPU tags are unaffected. |\n\nRe-check when AWS publishes new `huggingface-pytorch-inference` GPU tags — remove the row once a fixed image is confirmed.\n\n**General fallback rule**: when an HF DLC fails with CUDA\u002FNCCL linker errors, switch to DJL Inference rather than iterating over sibling tags — the defect class is per-repo, not per-tag (three different tags were tried for the case above; all broken).\n\n**Related HF Hub gotcha**: older DLCs can fail model download with `403 Forbidden` from HF's XET CDN (their bundled `huggingface_hub` predates XET auth). Set `HF_HUB_ENABLE_HF_TRANSFER=0` to force the standard download path, or pre-stage weights in S3.\n\n## Hub download time at first boot\n\nLoading the model from HF Hub happens *inside the container after the endpoint starts* — expect **5–15+ minutes** before InService even for small models, longer for multi-GB ones. A slow first boot is not a failure; don't tear down or re-diagnose before the deploy script's 30-minute wait expires.\n\nFor production or repeated deployments, pre-stage the weights in S3 and pass `--model-s3-uri` to `deploy.py` (the model then loads from `\u002Fopt\u002Fml\u002Fmodel`) — faster, immune to Hub rate limits\u002Foutages, and no `HUGGING_FACE_HUB_TOKEN` needed at runtime.\n",{"data":34,"body":35},{"name":4,"description":6},{"type":36,"children":37},"root",[38,47,62,69,148,183,216,222,230,243,271,309,327,333,631,701,736,748,754,759,862,872,952,964,984,990,1001,1083,1090,1110,1163,1168,1201,1207,1256,1261,1324,1352,1357,1363,1413,1419,1563,1569,1675,1687,1693,1698,1820,1825,1831,1836,1921,1934,1940,1960,1981,2008,2095,2128,2134,2144,2357,2367,2399,2405,2497,2509,2519,2552,2558,2577,2610],{"type":39,"tag":40,"props":41,"children":43},"element","h1",{"id":42},"serving-image-selection",[44],{"type":45,"value":46},"text","Serving Image Selection",{"type":39,"tag":48,"props":49,"children":50},"p",{},[51,53,60],{"type":45,"value":52},"The serving container is the single thing most likely to break a SageMaker deployment that \"looked correct on paper\". Wrong container, stale tag, or the wrong AMI — all produce the same opaque ",{"type":39,"tag":54,"props":55,"children":57},"code",{"className":56},[],[58],{"type":45,"value":59},"Failed to pass health check",{"type":45,"value":61}," error.",{"type":39,"tag":63,"props":64,"children":66},"h2",{"id":65},"rule-zero-huggingface-images-always-win",[67],{"type":45,"value":68},"Rule zero: HuggingFace images always win",{"type":39,"tag":48,"props":70,"children":71},{},[72,74,80,82,88,89,95,96,102,103,109,111,117,118,124,125,131,132,138,140,146],{"type":45,"value":73},"When both a HuggingFace-curated family (",{"type":39,"tag":54,"props":75,"children":77},{"className":76},[],[78],{"type":45,"value":79},"huggingface-vllm",{"type":45,"value":81},", ",{"type":39,"tag":54,"props":83,"children":85},{"className":84},[],[86],{"type":45,"value":87},"huggingface-vllm-omni",{"type":45,"value":81},{"type":39,"tag":54,"props":90,"children":92},{"className":91},[],[93],{"type":45,"value":94},"huggingface-sglang",{"type":45,"value":81},{"type":39,"tag":54,"props":97,"children":99},{"className":98},[],[100],{"type":45,"value":101},"tei",{"type":45,"value":81},{"type":39,"tag":54,"props":104,"children":106},{"className":105},[],[107],{"type":45,"value":108},"huggingface-pytorch-inference",{"type":45,"value":110},") and a generic family (",{"type":39,"tag":54,"props":112,"children":114},{"className":113},[],[115],{"type":45,"value":116},"vllm",{"type":45,"value":81},{"type":39,"tag":54,"props":119,"children":121},{"className":120},[],[122],{"type":45,"value":123},"vllm-omni",{"type":45,"value":81},{"type":39,"tag":54,"props":126,"children":128},{"className":127},[],[129],{"type":45,"value":130},"sglang",{"type":45,"value":81},{"type":39,"tag":54,"props":133,"children":135},{"className":134},[],[136],{"type":45,"value":137},"djl-inference",{"type":45,"value":139},") can serve the model, ",{"type":39,"tag":141,"props":142,"children":143},"strong",{},[144],{"type":45,"value":145},"the HuggingFace one is mandatory, not preferred",{"type":45,"value":147},". The only valid reasons to use a generic image:",{"type":39,"tag":149,"props":150,"children":151},"ol",{},[152,163,173],{"type":39,"tag":153,"props":154,"children":155},"li",{},[156,161],{"type":39,"tag":141,"props":157,"children":158},{},[159],{"type":45,"value":160},"Verified incompatibility",{"type":45,"value":162}," — the model needs an architecture\u002Fmodality\u002Ffeature no available HuggingFace tag supports, confirmed against the catalog (not assumed).",{"type":39,"tag":153,"props":164,"children":165},{},[166,171],{"type":39,"tag":141,"props":167,"children":168},{},[169],{"type":45,"value":170},"No HuggingFace tag exists in the target region",{"type":45,"value":172}," and mirroring is not an option.",{"type":39,"tag":153,"props":174,"children":175},{},[176,181],{"type":39,"tag":141,"props":177,"children":178},{},[179],{"type":45,"value":180},"The HuggingFace image is in \"Known-broken images\"",{"type":45,"value":182}," below.",{"type":39,"tag":48,"props":184,"children":185},{},[186,188,193,195,200,202,207,209,214],{"type":45,"value":187},"A ",{"type":39,"tag":141,"props":189,"children":190},{},[191],{"type":45,"value":192},"newer version number on the generic repo is not a reason",{"type":45,"value":194},". The AWS ",{"type":39,"tag":54,"props":196,"children":198},{"className":197},[],[199],{"type":45,"value":116},{"type":45,"value":201}," repo often publishes a higher vLLM version than ",{"type":39,"tag":54,"props":203,"children":205},{"className":204},[],[206],{"type":45,"value":79},{"type":45,"value":208},"; an older-but-compatible ",{"type":39,"tag":54,"props":210,"children":212},{"className":211},[],[213],{"type":45,"value":79},{"type":45,"value":215}," tag still wins. \"Latest vLLM\" is not a requirement anyone stated — compatibility with the model is. If you fall back, record in the deployment log which of the three reasons applied.",{"type":39,"tag":63,"props":217,"children":219},{"id":218},"where-image-uris-come-from",[220],{"type":45,"value":221},"Where image URIs come from",{"type":39,"tag":48,"props":223,"children":224},{},[225],{"type":39,"tag":141,"props":226,"children":227},{},[228],{"type":45,"value":229},"Primary source: AWS's official Deep Learning Containers catalog.",{"type":39,"tag":48,"props":231,"children":232},{},[233,235],{"type":45,"value":234},"URL: ",{"type":39,"tag":236,"props":237,"children":241},"a",{"href":238,"rel":239},"https:\u002F\u002Faws.github.io\u002Fdeep-learning-containers\u002Freference\u002Favailable_images\u002F",[240],"nofollow",[242],{"type":45,"value":238},{"type":39,"tag":48,"props":244,"children":245},{},[246,248,253,255,261,263,269],{"type":45,"value":247},"This page is AWS-maintained and lists every image family with example URIs, tags, CUDA versions, Python versions, and platform (SageMaker vs EC2\u002FECS\u002FEKS). When picking a URI for a deployment, ",{"type":39,"tag":141,"props":249,"children":250},{},[251],{"type":45,"value":252},"read it from this page directly",{"type":45,"value":254}," — copy the example URL, substitute ",{"type":39,"tag":54,"props":256,"children":258},{"className":257},[],[259],{"type":45,"value":260},"\u003Cregion>",{"type":45,"value":262}," with the user's region, and pass it to ",{"type":39,"tag":54,"props":264,"children":266},{"className":265},[],[267],{"type":45,"value":268},"deploy.py --image-uri",{"type":45,"value":270},".",{"type":39,"tag":48,"props":272,"children":273},{},[274,276,282,284,290,292,298,300,307],{"type":45,"value":275},"The example URLs use ",{"type":39,"tag":54,"props":277,"children":279},{"className":278},[],[280],{"type":45,"value":281},"763104351884",{"type":45,"value":283}," as the account ID for most regions. A few regions use different accounts (e.g. ",{"type":39,"tag":54,"props":285,"children":287},{"className":286},[],[288],{"type":45,"value":289},"eu-south-1",{"type":45,"value":291}," uses ",{"type":39,"tag":54,"props":293,"children":295},{"className":294},[],[296],{"type":45,"value":297},"692866216735",{"type":45,"value":299},"). Check the ",{"type":39,"tag":236,"props":301,"children":304},{"href":302,"rel":303},"https:\u002F\u002Faws.github.io\u002Fdeep-learning-containers\u002Freference\u002Fregion_availability\u002F",[240],[305],{"type":45,"value":306},"Region Availability page",{"type":45,"value":308}," when in doubt.",{"type":39,"tag":48,"props":310,"children":311},{},[312,317,319,325],{"type":39,"tag":141,"props":313,"children":314},{},[315],{"type":45,"value":316},"Exception: none currently.",{"type":45,"value":318}," Every image family used by this workflow is now on the AWS catalog page (TEI was added in late 2026). If you encounter a new family that isn't there, mirror it via ",{"type":39,"tag":54,"props":320,"children":322},{"className":321},[],[323],{"type":45,"value":324},"mirror_image.py",{"type":45,"value":326}," and pass the resulting URI directly.",{"type":39,"tag":63,"props":328,"children":330},{"id":329},"quick-decision",[331],{"type":45,"value":332},"Quick decision",{"type":39,"tag":334,"props":335,"children":336},"table",{},[337,361],{"type":39,"tag":338,"props":339,"children":340},"thead",{},[341],{"type":39,"tag":342,"props":343,"children":344},"tr",{},[345,351,356],{"type":39,"tag":346,"props":347,"children":348},"th",{},[349],{"type":45,"value":350},"Model",{"type":39,"tag":346,"props":352,"children":353},{},[354],{"type":45,"value":355},"Container family",{"type":39,"tag":346,"props":357,"children":358},{},[359],{"type":45,"value":360},"How to get the URI",{"type":39,"tag":362,"props":363,"children":364},"tbody",{},[365,394,421,439,463,492,517,535,553,578,595,613],{"type":39,"tag":342,"props":366,"children":367},{},[368,374,382],{"type":39,"tag":369,"props":370,"children":371},"td",{},[372],{"type":45,"value":373},"HuggingFace text-generation LLM (Llama, Qwen, Mistral, etc.)",{"type":39,"tag":369,"props":375,"children":376},{},[377],{"type":39,"tag":141,"props":378,"children":379},{},[380],{"type":45,"value":381},"HuggingFace vLLM",{"type":39,"tag":369,"props":383,"children":384},{},[385,387,392],{"type":45,"value":386},"AWS catalog → \"HuggingFace vLLM Inference\" (ECR repo ",{"type":39,"tag":54,"props":388,"children":390},{"className":389},[],[391],{"type":45,"value":79},{"type":45,"value":393},")",{"type":39,"tag":342,"props":395,"children":396},{},[397,402,410],{"type":39,"tag":369,"props":398,"children":399},{},[400],{"type":45,"value":401},"Same as above, multimodal",{"type":39,"tag":369,"props":403,"children":404},{},[405],{"type":39,"tag":141,"props":406,"children":407},{},[408],{"type":45,"value":409},"HuggingFace vLLM-Omni",{"type":39,"tag":369,"props":411,"children":412},{},[413,415,420],{"type":45,"value":414},"AWS catalog → \"HuggingFace vLLM-Omni Inference\" (ECR repo ",{"type":39,"tag":54,"props":416,"children":418},{"className":417},[],[419],{"type":45,"value":87},{"type":45,"value":393},{"type":39,"tag":342,"props":422,"children":423},{},[424,429,434],{"type":39,"tag":369,"props":425,"children":426},{},[427],{"type":45,"value":428},"HuggingFace embeddings",{"type":39,"tag":369,"props":430,"children":431},{},[432],{"type":45,"value":433},"TEI",{"type":39,"tag":369,"props":435,"children":436},{},[437],{"type":45,"value":438},"AWS catalog → \"HuggingFace Text Embeddings Inference\"",{"type":39,"tag":342,"props":440,"children":441},{},[442,454,458],{"type":39,"tag":369,"props":443,"children":444},{},[445,447,453],{"type":45,"value":446},"Encoder \u002F cross-encoder rerankers (BERT-family ",{"type":39,"tag":54,"props":448,"children":450},{"className":449},[],[451],{"type":45,"value":452},"*ForSequenceClassification",{"type":45,"value":393},{"type":39,"tag":369,"props":455,"children":456},{},[457],{"type":45,"value":433},{"type":39,"tag":369,"props":459,"children":460},{},[461],{"type":45,"value":462},"Same as embeddings",{"type":39,"tag":342,"props":464,"children":465},{},[466,476,480],{"type":39,"tag":369,"props":467,"children":468},{},[469,474],{"type":39,"tag":141,"props":470,"children":471},{},[472],{"type":45,"value":473},"Generative rerankers",{"type":45,"value":475}," (causal-LM, e.g. Qwen3-Reranker)",{"type":39,"tag":369,"props":477,"children":478},{},[479],{"type":45,"value":381},{"type":39,"tag":369,"props":481,"children":482},{},[483,485,490],{"type":45,"value":484},"Same as text-generation LLMs — ",{"type":39,"tag":141,"props":486,"children":487},{},[488],{"type":45,"value":489},"not TEI",{"type":45,"value":491},", see \"Rerankers: TEI or vLLM?\"",{"type":39,"tag":342,"props":493,"children":494},{},[495,500,505],{"type":39,"tag":369,"props":496,"children":497},{},[498],{"type":45,"value":499},"Text-to-image \u002F diffusion (Stable Diffusion, FLUX)",{"type":39,"tag":369,"props":501,"children":502},{},[503],{"type":45,"value":504},"DJL Inference",{"type":39,"tag":369,"props":506,"children":507},{},[508,510,515],{"type":45,"value":509},"AWS catalog → \"DJL Inference\" — ",{"type":39,"tag":141,"props":511,"children":512},{},[513],{"type":45,"value":514},"not",{"type":45,"value":516}," HF Inference Toolkit, see \"Known-broken images\"",{"type":39,"tag":342,"props":518,"children":519},{},[520,525,530],{"type":39,"tag":369,"props":521,"children":522},{},[523],{"type":45,"value":524},"HuggingFace classifiers, NER, QA, summarization",{"type":39,"tag":369,"props":526,"children":527},{},[528],{"type":45,"value":529},"HF Inference Toolkit (CPU)",{"type":39,"tag":369,"props":531,"children":532},{},[533],{"type":45,"value":534},"AWS catalog → \"HuggingFace PyTorch Inference\"; GPU tags currently broken — see \"Known-broken images\"",{"type":39,"tag":342,"props":536,"children":537},{},[538,543,548],{"type":39,"tag":369,"props":539,"children":540},{},[541],{"type":45,"value":542},"User specifically wants SGLang",{"type":39,"tag":369,"props":544,"children":545},{},[546],{"type":45,"value":547},"HuggingFace SGLang",{"type":39,"tag":369,"props":549,"children":550},{},[551],{"type":45,"value":552},"AWS catalog → \"HuggingFace SGLang Inference\"",{"type":39,"tag":342,"props":554,"children":555},{},[556,568,573],{"type":39,"tag":369,"props":557,"children":558},{},[559,561,566],{"type":45,"value":560},"No compatible ",{"type":39,"tag":54,"props":562,"children":564},{"className":563},[],[565],{"type":45,"value":79},{"type":45,"value":567}," tag (verified incompatibility or region gap — see \"Rule zero\")",{"type":39,"tag":369,"props":569,"children":570},{},[571],{"type":45,"value":572},"vLLM (AWS)",{"type":39,"tag":369,"props":574,"children":575},{},[576],{"type":45,"value":577},"AWS catalog → \"vLLM\" section — fallback only, never for version freshness",{"type":39,"tag":342,"props":579,"children":580},{},[581,586,590],{"type":39,"tag":369,"props":582,"children":583},{},[584],{"type":45,"value":585},"User specifically wants DJL-LMI",{"type":39,"tag":369,"props":587,"children":588},{},[589],{"type":45,"value":504},{"type":39,"tag":369,"props":591,"children":592},{},[593],{"type":45,"value":594},"AWS catalog → \"DJL Inference\"",{"type":39,"tag":342,"props":596,"children":597},{},[598,603,608],{"type":39,"tag":369,"props":599,"children":600},{},[601],{"type":45,"value":602},"Amazon Nova",{"type":39,"tag":369,"props":604,"children":605},{},[606],{"type":45,"value":607},"SageMaker JumpStart",{"type":39,"tag":369,"props":609,"children":610},{},[611],{"type":45,"value":612},"Use JumpStart, not raw endpoint creation",{"type":39,"tag":342,"props":614,"children":615},{},[616,621,626],{"type":39,"tag":369,"props":617,"children":618},{},[619],{"type":45,"value":620},"Custom inference code",{"type":39,"tag":369,"props":622,"children":623},{},[624],{"type":45,"value":625},"BYOC",{"type":39,"tag":369,"props":627,"children":628},{},[629],{"type":45,"value":630},"User provides URI",{"type":39,"tag":48,"props":632,"children":633},{},[634,639,641,646,648,661,663,669,671,677,679,685,687,692,694,699],{"type":39,"tag":141,"props":635,"children":636},{},[637],{"type":45,"value":638},"HuggingFace-curated DLCs are mandatory when one is compatible (see \"Rule zero\").",{"type":45,"value":640}," ",{"type":39,"tag":54,"props":642,"children":644},{"className":643},[],[645],{"type":45,"value":79},{"type":45,"value":647}," is layered directly on the AWS vLLM DLC — ",{"type":39,"tag":141,"props":649,"children":650},{},[651,653,659],{"type":45,"value":652},"identical ",{"type":39,"tag":54,"props":654,"children":656},{"className":655},[],[657],{"type":45,"value":658},"SM_VLLM_*",{"type":45,"value":660}," env contract and the same cu130 AMI rule",{"type":45,"value":662}," — and adds current ",{"type":39,"tag":54,"props":664,"children":666},{"className":665},[],[667],{"type":45,"value":668},"transformers",{"type":45,"value":670},", current ",{"type":39,"tag":54,"props":672,"children":674},{"className":673},[],[675],{"type":45,"value":676},"huggingface_hub",{"type":45,"value":678}," + ",{"type":39,"tag":54,"props":680,"children":682},{"className":681},[],[683],{"type":45,"value":684},"hf_xet",{"type":45,"value":686}," (avoids the XET-CDN 403 download failures older images hit), and HF performance defaults. It is also what SageMaker SDK v3 auto-routes to. The AWS ",{"type":39,"tag":54,"props":688,"children":690},{"className":689},[],[691],{"type":45,"value":116},{"type":45,"value":693}," image is a compatibility escape hatch only; it usually shows a higher vLLM version than ",{"type":39,"tag":54,"props":695,"children":697},{"className":696},[],[698],{"type":45,"value":79},{"type":45,"value":700},", and that is not a reason to pick it.",{"type":39,"tag":48,"props":702,"children":703},{},[704,709,711,718,720,726,728,734],{"type":39,"tag":141,"props":705,"children":706},{},[707],{"type":45,"value":708},"Do not use TGI.",{"type":45,"value":710}," Text Generation Inference is archived. Models released after the archive (Qwen3 most famously) fail ping health checks on TGI. Use vLLM instead. (The SageMaker SDK v3 agrees: since ",{"type":39,"tag":236,"props":712,"children":715},{"href":713,"rel":714},"https:\u002F\u002Fgithub.com\u002Faws\u002Fsagemaker-python-sdk\u002Fpull\u002F5960",[240],[716],{"type":45,"value":717},"PR #5960",{"type":45,"value":719},", June 2026, its ",{"type":39,"tag":54,"props":721,"children":723},{"className":722},[],[724],{"type":45,"value":725},"ModelBuilder",{"type":45,"value":727}," auto-routes ",{"type":39,"tag":54,"props":729,"children":731},{"className":730},[],[732],{"type":45,"value":733},"text-generation",{"type":45,"value":735}," to the HuggingFace vLLM DLC and multimodal tasks to HuggingFace vLLM-Omni.)",{"type":39,"tag":48,"props":737,"children":738},{},[739,741,747],{"type":45,"value":740},"Full reasoning for each family in ",{"type":39,"tag":54,"props":742,"children":744},{"className":743},[],[745],{"type":45,"value":746},"references\u002Fmodel-to-image.md",{"type":45,"value":270},{"type":39,"tag":63,"props":749,"children":751},{"id":750},"rerankers-tei-or-vllm",[752],{"type":45,"value":753},"Rerankers: TEI or vLLM?",{"type":39,"tag":48,"props":755,"children":756},{},[757],{"type":45,"value":758},"\"Reranker\" covers two very different architectures, and picking wrong wastes a full endpoint-creation cycle (~20 min) before TEI rejects the model:",{"type":39,"tag":760,"props":761,"children":762},"ul",{},[763,802],{"type":39,"tag":153,"props":764,"children":765},{},[766,771,773,779,781,787,789,795,797,801],{"type":39,"tag":141,"props":767,"children":768},{},[769],{"type":45,"value":770},"Encoder cross-encoders",{"type":45,"value":772}," (BAAI\u002Fbge-reranker-*, mixedbread, most ",{"type":39,"tag":54,"props":774,"children":776},{"className":775},[],[777],{"type":45,"value":778},"sentence-transformers",{"type":45,"value":780}," rerankers) — BERT-family models with a classification head. ",{"type":39,"tag":54,"props":782,"children":784},{"className":783},[],[785],{"type":45,"value":786},"config.json",{"type":45,"value":788}," has ",{"type":39,"tag":54,"props":790,"children":792},{"className":791},[],[793],{"type":45,"value":794},"architectures: [..ForSequenceClassification]",{"type":45,"value":796}," on a TEI-supported encoder type. → ",{"type":39,"tag":141,"props":798,"children":799},{},[800],{"type":45,"value":433},{"type":45,"value":270},{"type":39,"tag":153,"props":803,"children":804},{},[805,809,811,816,817,823,825,829,831,837,839,845,847,853,855,861],{"type":39,"tag":141,"props":806,"children":807},{},[808],{"type":45,"value":473},{"type":45,"value":810}," (Qwen\u002FQwen3-Reranker-*, and similar causal-LM judges) — decoder LLMs that score relevance via the logprob of a yes\u002Fno token. ",{"type":39,"tag":54,"props":812,"children":814},{"className":813},[],[815],{"type":45,"value":786},{"type":45,"value":788},{"type":39,"tag":54,"props":818,"children":820},{"className":819},[],[821],{"type":45,"value":822},"architectures: [..ForCausalLM]",{"type":45,"value":824},". → ",{"type":39,"tag":141,"props":826,"children":827},{},[828],{"type":45,"value":381},{"type":45,"value":830},", deployed exactly like a text-generation LLM. TEI will load the architecture then reject the ",{"type":39,"tag":54,"props":832,"children":834},{"className":833},[],[835],{"type":45,"value":836},"classifier",{"type":45,"value":838}," model type (Qwen3 support in TEI is ",{"type":39,"tag":840,"props":841,"children":842},"em",{},[843],{"type":45,"value":844},"embeddings-only",{"type":45,"value":846},"). Invocation pattern (raw completions API, ",{"type":39,"tag":54,"props":848,"children":850},{"className":849},[],[851],{"type":45,"value":852},"max_tokens=1",{"type":45,"value":854},", logprobs scoring) is in ",{"type":39,"tag":54,"props":856,"children":858},{"className":857},[],[859],{"type":45,"value":860},"hf-cloud-sagemaker-production-defaults",{"type":45,"value":270},{"type":39,"tag":48,"props":863,"children":864},{},[865,870],{"type":39,"tag":141,"props":866,"children":867},{},[868],{"type":45,"value":869},"Preflight before creating any resources",{"type":45,"value":871}," — one HTTP GET settles it:",{"type":39,"tag":873,"props":874,"children":879},"pre",{"className":875,"code":876,"language":877,"meta":878,"style":878},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","curl -s https:\u002F\u002Fhuggingface.co\u002F\u003Cmodel-id>\u002Fraw\u002Fmain\u002Fconfig.json\n# \"architectures\": [\"Qwen3ForCausalLM\"]              → vLLM\n# \"architectures\": [\"XLMRobertaForSequenceClassification\"] → TEI\n","bash","",[880],{"type":39,"tag":54,"props":881,"children":882},{"__ignoreMap":878},[883,933,943],{"type":39,"tag":884,"props":885,"children":888},"span",{"class":886,"line":887},"line",1,[889,895,901,906,912,917,923,928],{"type":39,"tag":884,"props":890,"children":892},{"style":891},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[893],{"type":45,"value":894},"curl",{"type":39,"tag":884,"props":896,"children":898},{"style":897},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[899],{"type":45,"value":900}," -s",{"type":39,"tag":884,"props":902,"children":903},{"style":897},[904],{"type":45,"value":905}," https:\u002F\u002Fhuggingface.co\u002F",{"type":39,"tag":884,"props":907,"children":909},{"style":908},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[910],{"type":45,"value":911},"\u003C",{"type":39,"tag":884,"props":913,"children":914},{"style":897},[915],{"type":45,"value":916},"model-i",{"type":39,"tag":884,"props":918,"children":920},{"style":919},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[921],{"type":45,"value":922},"d",{"type":39,"tag":884,"props":924,"children":925},{"style":908},[926],{"type":45,"value":927},">",{"type":39,"tag":884,"props":929,"children":930},{"style":897},[931],{"type":45,"value":932},"\u002Fraw\u002Fmain\u002Fconfig.json\n",{"type":39,"tag":884,"props":934,"children":936},{"class":886,"line":935},2,[937],{"type":39,"tag":884,"props":938,"children":940},{"style":939},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[941],{"type":45,"value":942},"# \"architectures\": [\"Qwen3ForCausalLM\"]              → vLLM\n",{"type":39,"tag":884,"props":944,"children":946},{"class":886,"line":945},3,[947],{"type":39,"tag":884,"props":948,"children":949},{"style":939},[950],{"type":45,"value":951},"# \"architectures\": [\"XLMRobertaForSequenceClassification\"] → TEI\n",{"type":39,"tag":48,"props":953,"children":954},{},[955,957,962],{"type":45,"value":956},"For TEI also confirm the ",{"type":39,"tag":840,"props":958,"children":959},{},[960],{"type":45,"value":961},"(architecture, task)",{"type":45,"value":963}," pair: an architecture appearing in TEI's supported list means embeddings support, not necessarily classification\u002Freranking support.",{"type":39,"tag":48,"props":965,"children":966},{},[967,969,975,977,982],{"type":45,"value":968},"Heads-up: SageMaker SDK v3 (PR #5960) routes the ",{"type":39,"tag":54,"props":970,"children":972},{"className":971},[],[973],{"type":45,"value":974},"text-ranking",{"type":45,"value":976}," task to TEI ",{"type":39,"tag":141,"props":978,"children":979},{},[980],{"type":45,"value":981},"unconditionally",{"type":45,"value":983}," — correct for cross-encoders, wrong for generative rerankers. Don't treat the SDK's routing as evidence that TEI can serve a given reranker.",{"type":39,"tag":63,"props":985,"children":987},{"id":986},"workflow",[988],{"type":45,"value":989},"Workflow",{"type":39,"tag":48,"props":991,"children":992},{},[993,995,1000],{"type":45,"value":994},"For every family: ",{"type":39,"tag":141,"props":996,"children":997},{},[998],{"type":45,"value":999},"read the URI from the AWS catalog page",{"type":45,"value":270},{"type":39,"tag":149,"props":1002,"children":1003},{},[1004,1014,1019,1039,1058,1063],{"type":39,"tag":153,"props":1005,"children":1006},{},[1007,1009],{"type":45,"value":1008},"Open ",{"type":39,"tag":236,"props":1010,"children":1012},{"href":238,"rel":1011},[240],[1013],{"type":45,"value":238},{"type":39,"tag":153,"props":1015,"children":1016},{},[1017],{"type":45,"value":1018},"Find the section for the right family (e.g. \"HuggingFace vLLM Inference\" for HuggingFace LLMs, \"HuggingFace Text Embeddings Inference\" for embeddings)",{"type":39,"tag":153,"props":1020,"children":1021},{},[1022,1024,1030,1032,1037],{"type":45,"value":1023},"Pick the newest row marked ",{"type":39,"tag":54,"props":1025,"children":1027},{"className":1026},[],[1028],{"type":45,"value":1029},"SageMaker",{"type":45,"value":1031}," for the platform column — newest ",{"type":39,"tag":141,"props":1033,"children":1034},{},[1035],{"type":45,"value":1036},"within that family",{"type":45,"value":1038},". Do not switch to another family's section because it lists a higher engine version (see \"Rule zero\")",{"type":39,"tag":153,"props":1040,"children":1041},{},[1042,1044,1049,1051,1057],{"type":45,"value":1043},"Substitute ",{"type":39,"tag":54,"props":1045,"children":1047},{"className":1046},[],[1048],{"type":45,"value":260},{"type":45,"value":1050}," with the user's region (from ",{"type":39,"tag":54,"props":1052,"children":1054},{"className":1053},[],[1055],{"type":45,"value":1056},"hf-cloud-aws-context-discovery",{"type":45,"value":393},{"type":39,"tag":153,"props":1059,"children":1060},{},[1061],{"type":45,"value":1062},"For vLLM: also check the AMI requirement (see \"vLLM AMI requirement\" below)",{"type":39,"tag":153,"props":1064,"children":1065},{},[1066,1068,1073,1075,1081],{"type":45,"value":1067},"Pass the URI to ",{"type":39,"tag":54,"props":1069,"children":1071},{"className":1070},[],[1072],{"type":45,"value":268},{"type":45,"value":1074}," (real-time) or ",{"type":39,"tag":54,"props":1076,"children":1078},{"className":1077},[],[1079],{"type":45,"value":1080},"deploy_async.py --image-uri",{"type":45,"value":1082}," (async)",{"type":39,"tag":1084,"props":1085,"children":1087},"h3",{"id":1086},"tei-pick-the-right-variant",[1088],{"type":45,"value":1089},"TEI: pick the right variant",{"type":39,"tag":48,"props":1091,"children":1092},{},[1093,1095,1100,1102,1108],{"type":45,"value":1094},"The TEI catalog row lists two URIs — GPU (",{"type":39,"tag":54,"props":1096,"children":1098},{"className":1097},[],[1099],{"type":45,"value":101},{"type":45,"value":1101}," repo) and CPU (",{"type":39,"tag":54,"props":1103,"children":1105},{"className":1104},[],[1106],{"type":45,"value":1107},"tei-cpu",{"type":45,"value":1109}," repo). Pick based on the instance type:",{"type":39,"tag":760,"props":1111,"children":1112},{},[1113,1138],{"type":39,"tag":153,"props":1114,"children":1115},{},[1116,1122,1123,1129,1130,1136],{"type":39,"tag":54,"props":1117,"children":1119},{"className":1118},[],[1120],{"type":45,"value":1121},"ml.g*",{"type":45,"value":81},{"type":39,"tag":54,"props":1124,"children":1126},{"className":1125},[],[1127],{"type":45,"value":1128},"ml.p*",{"type":45,"value":81},{"type":39,"tag":54,"props":1131,"children":1133},{"className":1132},[],[1134],{"type":45,"value":1135},"ml.inf*",{"type":45,"value":1137}," → GPU variant",{"type":39,"tag":153,"props":1139,"children":1140},{},[1141,1147,1148,1154,1155,1161],{"type":39,"tag":54,"props":1142,"children":1144},{"className":1143},[],[1145],{"type":45,"value":1146},"ml.c*",{"type":45,"value":81},{"type":39,"tag":54,"props":1149,"children":1151},{"className":1150},[],[1152],{"type":45,"value":1153},"ml.m*",{"type":45,"value":81},{"type":39,"tag":54,"props":1156,"children":1158},{"className":1157},[],[1159],{"type":45,"value":1160},"ml.t*",{"type":45,"value":1162}," → CPU variant",{"type":39,"tag":48,"props":1164,"children":1165},{},[1166],{"type":45,"value":1167},"Mixing them fails: CPU image on a GPU instance wastes hardware, GPU image on a CPU instance fails to start.",{"type":39,"tag":48,"props":1169,"children":1170},{},[1171,1176,1178,1184,1186,1192,1194,1199],{"type":39,"tag":141,"props":1172,"children":1173},{},[1174],{"type":45,"value":1175},"Note on the TEI account ID",{"type":45,"value":1177},": the catalog page shows ",{"type":39,"tag":54,"props":1179,"children":1181},{"className":1180},[],[1182],{"type":45,"value":1183},"683313688378",{"type":45,"value":1185}," as the example account, but TEI is published from a different account namespace than the main AWS DLCs and the per-region account IDs vary. If ",{"type":39,"tag":54,"props":1187,"children":1189},{"className":1188},[],[1190],{"type":45,"value":1191},"683313688378.dkr.ecr.\u003Cregion>.amazonaws.com\u002Ftei:...",{"type":45,"value":1193}," returns an ECR pull error for a region other than us-east-1, check the ",{"type":39,"tag":236,"props":1195,"children":1197},{"href":302,"rel":1196},[240],[1198],{"type":45,"value":306},{"type":45,"value":1200}," for the correct account ID for that region.",{"type":39,"tag":63,"props":1202,"children":1204},{"id":1203},"vllm-ami-requirement",[1205],{"type":45,"value":1206},"vLLM AMI requirement",{"type":39,"tag":48,"props":1208,"children":1209},{},[1210,1212,1217,1219,1225,1227,1233,1235,1240,1242,1247,1249,1254],{"type":45,"value":1211},"vLLM DLC images with ",{"type":39,"tag":141,"props":1213,"children":1214},{},[1215],{"type":45,"value":1216},"CUDA 13 or higher",{"type":45,"value":1218}," (current default: ",{"type":39,"tag":54,"props":1220,"children":1222},{"className":1221},[],[1223],{"type":45,"value":1224},"cu130",{"type":45,"value":1226},") require setting ",{"type":39,"tag":54,"props":1228,"children":1230},{"className":1229},[],[1231],{"type":45,"value":1232},"InferenceAmiVersion=al2-ami-sagemaker-inference-gpu-3-1",{"type":45,"value":1234}," on the ProductionVariant. This applies equally to ",{"type":39,"tag":54,"props":1236,"children":1238},{"className":1237},[],[1239],{"type":45,"value":79},{"type":45,"value":1241}," and ",{"type":39,"tag":54,"props":1243,"children":1245},{"className":1244},[],[1246],{"type":45,"value":87},{"type":45,"value":1248}," (layered on the same cu130 base) and to the AWS ",{"type":39,"tag":54,"props":1250,"children":1252},{"className":1251},[],[1253],{"type":45,"value":116},{"type":45,"value":1255}," repo. Without it the container dies on startup with no CloudWatch logs ever created. The failure looks identical to many other things (account-level issues, quota, networking) and routinely sends people down wrong diagnostic paths.",{"type":39,"tag":48,"props":1257,"children":1258},{},[1259],{"type":45,"value":1260},"Lookup table:",{"type":39,"tag":334,"props":1262,"children":1263},{},[1264,1280],{"type":39,"tag":338,"props":1265,"children":1266},{},[1267],{"type":39,"tag":342,"props":1268,"children":1269},{},[1270,1275],{"type":39,"tag":346,"props":1271,"children":1272},{},[1273],{"type":45,"value":1274},"Tag contains",{"type":39,"tag":346,"props":1276,"children":1277},{},[1278],{"type":45,"value":1279},"InferenceAmiVersion to pass",{"type":39,"tag":362,"props":1281,"children":1282},{},[1283,1305],{"type":39,"tag":342,"props":1284,"children":1285},{},[1286,1296],{"type":39,"tag":369,"props":1287,"children":1288},{},[1289,1294],{"type":39,"tag":54,"props":1290,"children":1292},{"className":1291},[],[1293],{"type":45,"value":1224},{"type":45,"value":1295}," (or higher)",{"type":39,"tag":369,"props":1297,"children":1298},{},[1299],{"type":39,"tag":54,"props":1300,"children":1302},{"className":1301},[],[1303],{"type":45,"value":1304},"al2-ami-sagemaker-inference-gpu-3-1",{"type":39,"tag":342,"props":1306,"children":1307},{},[1308,1319],{"type":39,"tag":369,"props":1309,"children":1310},{},[1311,1317],{"type":39,"tag":54,"props":1312,"children":1314},{"className":1313},[],[1315],{"type":45,"value":1316},"cu129",{"type":45,"value":1318}," or lower",{"type":39,"tag":369,"props":1320,"children":1321},{},[1322],{"type":45,"value":1323},"(omit the flag; default AMI works)",{"type":39,"tag":48,"props":1325,"children":1326},{},[1327,1329,1334,1336,1342,1344,1350],{"type":45,"value":1328},"Rule of thumb: if the vLLM tag you picked contains ",{"type":39,"tag":54,"props":1330,"children":1332},{"className":1331},[],[1333],{"type":45,"value":1224},{"type":45,"value":1335}," or later, pass ",{"type":39,"tag":54,"props":1337,"children":1339},{"className":1338},[],[1340],{"type":45,"value":1341},"--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1",{"type":45,"value":1343}," to ",{"type":39,"tag":54,"props":1345,"children":1347},{"className":1346},[],[1348],{"type":45,"value":1349},"deploy.py",{"type":45,"value":1351},". If a future CUDA version (cu140+) needs a different AMI, add a row to the table when AWS publishes the new image.",{"type":39,"tag":48,"props":1353,"children":1354},{},[1355],{"type":45,"value":1356},"This is a vLLM-specific concern. TEI and HF Inference Toolkit images don't need an AMI override.",{"type":39,"tag":63,"props":1358,"children":1360},{"id":1359},"configuring-the-vllm-dlcs-huggingface-vllm-and-aws-vllm",[1361],{"type":45,"value":1362},"Configuring the vLLM DLCs (HuggingFace vLLM and AWS vLLM)",{"type":39,"tag":48,"props":1364,"children":1365},{},[1366,1368,1373,1375,1380,1382,1388,1390,1396,1398,1404,1406,1411],{"type":45,"value":1367},"Both images share the same contract: configuration as environment variables on the SageMaker model definition, ",{"type":39,"tag":54,"props":1369,"children":1371},{"className":1370},[],[1372],{"type":45,"value":658},{"type":45,"value":1374}," mapped to vLLM CLI flags. The ",{"type":39,"tag":54,"props":1376,"children":1378},{"className":1377},[],[1379],{"type":45,"value":79},{"type":45,"value":1381}," entrypoint additionally auto-detects the model when ",{"type":39,"tag":54,"props":1383,"children":1385},{"className":1384},[],[1386],{"type":45,"value":1387},"SM_VLLM_MODEL",{"type":45,"value":1389}," is unset — from ",{"type":39,"tag":54,"props":1391,"children":1393},{"className":1392},[],[1394],{"type":45,"value":1395},"\u002Fopt\u002Fml\u002Fmodel",{"type":45,"value":1397}," if artifacts are mounted, else from ",{"type":39,"tag":54,"props":1399,"children":1401},{"className":1400},[],[1402],{"type":45,"value":1403},"HF_MODEL_ID",{"type":45,"value":1405}," — but setting ",{"type":39,"tag":54,"props":1407,"children":1409},{"className":1408},[],[1410],{"type":45,"value":1387},{"type":45,"value":1412}," explicitly works on both and is what our examples use.",{"type":39,"tag":1084,"props":1414,"children":1416},{"id":1415},"required-for-every-huggingface-llm-deployment",[1417],{"type":45,"value":1418},"Required for every HuggingFace LLM deployment",{"type":39,"tag":334,"props":1420,"children":1421},{},[1422,1443],{"type":39,"tag":338,"props":1423,"children":1424},{},[1425],{"type":39,"tag":342,"props":1426,"children":1427},{},[1428,1433,1438],{"type":39,"tag":346,"props":1429,"children":1430},{},[1431],{"type":45,"value":1432},"Env var",{"type":39,"tag":346,"props":1434,"children":1435},{},[1436],{"type":45,"value":1437},"Purpose",{"type":39,"tag":346,"props":1439,"children":1440},{},[1441],{"type":45,"value":1442},"Notes",{"type":39,"tag":362,"props":1444,"children":1445},{},[1446,1482,1513,1541],{"type":39,"tag":342,"props":1447,"children":1448},{},[1449,1457,1477],{"type":39,"tag":369,"props":1450,"children":1451},{},[1452],{"type":39,"tag":54,"props":1453,"children":1455},{"className":1454},[],[1456],{"type":45,"value":1387},{"type":39,"tag":369,"props":1458,"children":1459},{},[1460,1462,1468,1470,1475],{"type":45,"value":1461},"HF model ID (e.g. ",{"type":39,"tag":54,"props":1463,"children":1465},{"className":1464},[],[1466],{"type":45,"value":1467},"Qwen\u002FQwen3-0.6B",{"type":45,"value":1469},") or ",{"type":39,"tag":54,"props":1471,"children":1473},{"className":1472},[],[1474],{"type":45,"value":1395},{"type":45,"value":1476}," if loading from S3",{"type":39,"tag":369,"props":1478,"children":1479},{},[1480],{"type":45,"value":1481},"—",{"type":39,"tag":342,"props":1483,"children":1484},{},[1485,1494,1508],{"type":39,"tag":369,"props":1486,"children":1487},{},[1488],{"type":39,"tag":54,"props":1489,"children":1491},{"className":1490},[],[1492],{"type":45,"value":1493},"SM_VLLM_HOST",{"type":39,"tag":369,"props":1495,"children":1496},{},[1497],{"type":39,"tag":141,"props":1498,"children":1499},{},[1500,1502],{"type":45,"value":1501},"Must be ",{"type":39,"tag":54,"props":1503,"children":1505},{"className":1504},[],[1506],{"type":45,"value":1507},"0.0.0.0",{"type":39,"tag":369,"props":1509,"children":1510},{},[1511],{"type":45,"value":1512},"Otherwise vLLM binds localhost only, ping fails, container dies before logs. Top cause of mystery failures with this image.",{"type":39,"tag":342,"props":1514,"children":1515},{},[1516,1525,1536],{"type":39,"tag":369,"props":1517,"children":1518},{},[1519],{"type":39,"tag":54,"props":1520,"children":1522},{"className":1521},[],[1523],{"type":45,"value":1524},"SM_VLLM_TRUST_REMOTE_CODE",{"type":39,"tag":369,"props":1526,"children":1527},{},[1528,1534],{"type":39,"tag":54,"props":1529,"children":1531},{"className":1530},[],[1532],{"type":45,"value":1533},"true",{"type":45,"value":1535}," for Qwen and several recent architectures",{"type":39,"tag":369,"props":1537,"children":1538},{},[1539],{"type":45,"value":1540},"Set unconditionally — downside negligible, upside is the model loads.",{"type":39,"tag":342,"props":1542,"children":1543},{},[1544,1553,1558],{"type":39,"tag":369,"props":1545,"children":1546},{},[1547],{"type":39,"tag":54,"props":1548,"children":1550},{"className":1549},[],[1551],{"type":45,"value":1552},"HUGGING_FACE_HUB_TOKEN",{"type":39,"tag":369,"props":1554,"children":1555},{},[1556],{"type":45,"value":1557},"HF token",{"type":39,"tag":369,"props":1559,"children":1560},{},[1561],{"type":45,"value":1562},"Required for gated models.",{"type":39,"tag":1084,"props":1564,"children":1566},{"id":1565},"tuning-optional",[1567],{"type":45,"value":1568},"Tuning (optional)",{"type":39,"tag":334,"props":1570,"children":1571},{},[1572,1586],{"type":39,"tag":338,"props":1573,"children":1574},{},[1575],{"type":39,"tag":342,"props":1576,"children":1577},{},[1578,1582],{"type":39,"tag":346,"props":1579,"children":1580},{},[1581],{"type":45,"value":1432},{"type":39,"tag":346,"props":1583,"children":1584},{},[1585],{"type":45,"value":1437},{"type":39,"tag":362,"props":1587,"children":1588},{},[1589,1606,1623,1640],{"type":39,"tag":342,"props":1590,"children":1591},{},[1592,1601],{"type":39,"tag":369,"props":1593,"children":1594},{},[1595],{"type":39,"tag":54,"props":1596,"children":1598},{"className":1597},[],[1599],{"type":45,"value":1600},"SM_VLLM_MAX_MODEL_LEN",{"type":39,"tag":369,"props":1602,"children":1603},{},[1604],{"type":45,"value":1605},"Max sequence length — set this; defaults can be wrong for fine-tunes",{"type":39,"tag":342,"props":1607,"children":1608},{},[1609,1618],{"type":39,"tag":369,"props":1610,"children":1611},{},[1612],{"type":39,"tag":54,"props":1613,"children":1615},{"className":1614},[],[1616],{"type":45,"value":1617},"SM_VLLM_GPU_MEMORY_UTILIZATION",{"type":39,"tag":369,"props":1619,"children":1620},{},[1621],{"type":45,"value":1622},"Float 0.0–1.0, ~0.9 reasonable",{"type":39,"tag":342,"props":1624,"children":1625},{},[1626,1635],{"type":39,"tag":369,"props":1627,"children":1628},{},[1629],{"type":39,"tag":54,"props":1630,"children":1632},{"className":1631},[],[1633],{"type":45,"value":1634},"SM_VLLM_TENSOR_PARALLEL_SIZE",{"type":39,"tag":369,"props":1636,"children":1637},{},[1638],{"type":45,"value":1639},"GPU count for multi-GPU instances",{"type":39,"tag":342,"props":1641,"children":1642},{},[1643,1652],{"type":39,"tag":369,"props":1644,"children":1645},{},[1646],{"type":39,"tag":54,"props":1647,"children":1649},{"className":1648},[],[1650],{"type":45,"value":1651},"SM_VLLM_DTYPE",{"type":39,"tag":369,"props":1653,"children":1654},{},[1655,1661,1662,1668,1669],{"type":39,"tag":54,"props":1656,"children":1658},{"className":1657},[],[1659],{"type":45,"value":1660},"auto",{"type":45,"value":81},{"type":39,"tag":54,"props":1663,"children":1665},{"className":1664},[],[1666],{"type":45,"value":1667},"bfloat16",{"type":45,"value":81},{"type":39,"tag":54,"props":1670,"children":1672},{"className":1671},[],[1673],{"type":45,"value":1674},"float16",{"type":39,"tag":48,"props":1676,"children":1677},{},[1678,1680,1686],{"type":45,"value":1679},"Any vLLM CLI flag works — uppercase, replace dashes with underscores, prepend ",{"type":39,"tag":54,"props":1681,"children":1683},{"className":1682},[],[1684],{"type":45,"value":1685},"SM_VLLM_",{"type":45,"value":270},{"type":39,"tag":63,"props":1688,"children":1690},{"id":1689},"configuring-tei",[1691],{"type":45,"value":1692},"Configuring TEI",{"type":39,"tag":48,"props":1694,"children":1695},{},[1696],{"type":45,"value":1697},"Simpler env contract than vLLM:",{"type":39,"tag":334,"props":1699,"children":1700},{},[1701,1720],{"type":39,"tag":338,"props":1702,"children":1703},{},[1704],{"type":39,"tag":342,"props":1705,"children":1706},{},[1707,1711,1715],{"type":39,"tag":346,"props":1708,"children":1709},{},[1710],{"type":45,"value":1432},{"type":39,"tag":346,"props":1712,"children":1713},{},[1714],{"type":45,"value":1437},{"type":39,"tag":346,"props":1716,"children":1717},{},[1718],{"type":45,"value":1719},"Required",{"type":39,"tag":362,"props":1721,"children":1722},{},[1723,1755,1777,1799],{"type":39,"tag":342,"props":1724,"children":1725},{},[1726,1734,1750],{"type":39,"tag":369,"props":1727,"children":1728},{},[1729],{"type":39,"tag":54,"props":1730,"children":1732},{"className":1731},[],[1733],{"type":45,"value":1403},{"type":39,"tag":369,"props":1735,"children":1736},{},[1737,1738,1744,1745],{"type":45,"value":1461},{"type":39,"tag":54,"props":1739,"children":1741},{"className":1740},[],[1742],{"type":45,"value":1743},"BAAI\u002Fbge-large-en-v1.5",{"type":45,"value":1469},{"type":39,"tag":54,"props":1746,"children":1748},{"className":1747},[],[1749],{"type":45,"value":1395},{"type":39,"tag":369,"props":1751,"children":1752},{},[1753],{"type":45,"value":1754},"Yes",{"type":39,"tag":342,"props":1756,"children":1757},{},[1758,1767,1772],{"type":39,"tag":369,"props":1759,"children":1760},{},[1761],{"type":39,"tag":54,"props":1762,"children":1764},{"className":1763},[],[1765],{"type":45,"value":1766},"HF_TOKEN",{"type":39,"tag":369,"props":1768,"children":1769},{},[1770],{"type":45,"value":1771},"HF auth token",{"type":39,"tag":369,"props":1773,"children":1774},{},[1775],{"type":45,"value":1776},"Only for gated models",{"type":39,"tag":342,"props":1778,"children":1779},{},[1780,1789,1794],{"type":39,"tag":369,"props":1781,"children":1782},{},[1783],{"type":39,"tag":54,"props":1784,"children":1786},{"className":1785},[],[1787],{"type":45,"value":1788},"MAX_BATCH_TOKENS",{"type":39,"tag":369,"props":1790,"children":1791},{},[1792],{"type":45,"value":1793},"Max tokens per batch (default 16384)",{"type":39,"tag":369,"props":1795,"children":1796},{},[1797],{"type":45,"value":1798},"No",{"type":39,"tag":342,"props":1800,"children":1801},{},[1802,1811,1816],{"type":39,"tag":369,"props":1803,"children":1804},{},[1805],{"type":39,"tag":54,"props":1806,"children":1808},{"className":1807},[],[1809],{"type":45,"value":1810},"MAX_CLIENT_BATCH_SIZE",{"type":39,"tag":369,"props":1812,"children":1813},{},[1814],{"type":45,"value":1815},"Max requests per client batch (default 32)",{"type":39,"tag":369,"props":1817,"children":1818},{},[1819],{"type":45,"value":1798},{"type":39,"tag":48,"props":1821,"children":1822},{},[1823],{"type":45,"value":1824},"No host-binding to configure, no trust-remote-code flag. The architectures TEI supports (BERT, CamemBERT, RoBERTa, XLM-RoBERTa, NomicBert, JinaBert, JinaCodeBert, Mistral, Qwen2\u002F3, Gemma2\u002F3, ModernBert) are baked into the image.",{"type":39,"tag":63,"props":1826,"children":1828},{"id":1827},"cuda-instance-compatibility",[1829],{"type":45,"value":1830},"CUDA \u002F instance compatibility",{"type":39,"tag":48,"props":1832,"children":1833},{},[1834],{"type":45,"value":1835},"Critical and easy to get wrong:",{"type":39,"tag":334,"props":1837,"children":1838},{},[1839,1865],{"type":39,"tag":338,"props":1840,"children":1841},{},[1842],{"type":39,"tag":342,"props":1843,"children":1844},{},[1845,1850,1855],{"type":39,"tag":346,"props":1846,"children":1847},{},[1848],{"type":45,"value":1849},"CUDA in image tag",{"type":39,"tag":346,"props":1851,"children":1852},{},[1853],{"type":45,"value":1854},"Default AMI",{"type":39,"tag":346,"props":1856,"children":1857},{},[1858,1860],{"type":45,"value":1859},"With ",{"type":39,"tag":54,"props":1861,"children":1863},{"className":1862},[],[1864],{"type":45,"value":1304},{"type":39,"tag":362,"props":1866,"children":1867},{},[1868,1886,1903],{"type":39,"tag":342,"props":1869,"children":1870},{},[1871,1876,1881],{"type":39,"tag":369,"props":1872,"children":1873},{},[1874],{"type":45,"value":1875},"cu124 \u002F cu128",{"type":39,"tag":369,"props":1877,"children":1878},{},[1879],{"type":45,"value":1880},"g5, g6, p5 all work",{"type":39,"tag":369,"props":1882,"children":1883},{},[1884],{"type":45,"value":1885},"(not needed)",{"type":39,"tag":342,"props":1887,"children":1888},{},[1889,1893,1898],{"type":39,"tag":369,"props":1890,"children":1891},{},[1892],{"type":45,"value":1316},{"type":39,"tag":369,"props":1894,"children":1895},{},[1896],{"type":45,"value":1897},"g6, p5; g5 fails (driver mismatch → CannotStartContainerError)",{"type":39,"tag":369,"props":1899,"children":1900},{},[1901],{"type":45,"value":1902},"expected to fix g5 (unverified)",{"type":39,"tag":342,"props":1904,"children":1905},{},[1906,1911,1916],{"type":39,"tag":369,"props":1907,"children":1908},{},[1909],{"type":45,"value":1910},"cu130+",{"type":39,"tag":369,"props":1912,"children":1913},{},[1914],{"type":45,"value":1915},"fails everywhere — AMI flag is mandatory",{"type":39,"tag":369,"props":1917,"children":1918},{},[1919],{"type":45,"value":1920},"g5, g6, p5 all work (cu130-on-g5 verified June 2026)",{"type":39,"tag":48,"props":1922,"children":1923},{},[1924,1926,1932],{"type":45,"value":1925},"The driver comes from the host AMI, not the instance family — so passing the gpu-3-1 AMI (which vLLM cu130 images require anyway) also makes ",{"type":39,"tag":54,"props":1927,"children":1929},{"className":1928},[],[1930],{"type":45,"value":1931},"ml.g5.*",{"type":45,"value":1933}," viable for cu129+ images.",{"type":39,"tag":63,"props":1935,"children":1937},{"id":1936},"vpc-nat-gateway-problem",[1938],{"type":45,"value":1939},"VPC \u002F NAT gateway problem",{"type":39,"tag":48,"props":1941,"children":1942},{},[1943,1945,1950,1952,1958],{"type":45,"value":1944},"SageMaker endpoints inside a VPC ",{"type":39,"tag":141,"props":1946,"children":1947},{},[1948],{"type":45,"value":1949},"without",{"type":45,"value":1951}," a NAT gateway can't pull from ",{"type":39,"tag":54,"props":1953,"children":1955},{"className":1954},[],[1956],{"type":45,"value":1957},"public.ecr.aws",{"type":45,"value":1959},". The deployment fails with an image-pull error that doesn't mention \"VPC\" or \"egress\".",{"type":39,"tag":48,"props":1961,"children":1962},{},[1963,1965,1971,1973,1979],{"type":45,"value":1964},"For images on AWS's regional ECR (everything in the catalog): SageMaker reaches them through built-in routing, no NAT needed. Use the regional URI pattern (",{"type":39,"tag":54,"props":1966,"children":1968},{"className":1967},[],[1969],{"type":45,"value":1970},"\u003Caccount>.dkr.ecr.\u003Cregion>.amazonaws.com\u002F...",{"type":45,"value":1972},"), not the ",{"type":39,"tag":54,"props":1974,"children":1976},{"className":1975},[],[1977],{"type":45,"value":1978},"public.ecr.aws\u002F...",{"type":45,"value":1980}," pattern.",{"type":39,"tag":48,"props":1982,"children":1983},{},[1984,1986,1991,1993,1999,2001,2006],{"type":45,"value":1985},"For images requiring ",{"type":39,"tag":54,"props":1987,"children":1989},{"className":1988},[],[1990],{"type":45,"value":1957},{"type":45,"value":1992}," access (less common): mirror to a private ECR repo in your account with ",{"type":39,"tag":54,"props":1994,"children":1996},{"className":1995},[],[1997],{"type":45,"value":1998},"scripts\u002Fmirror_image.py",{"type":45,"value":2000}," (cross-platform; needs Docker + the ",{"type":39,"tag":54,"props":2002,"children":2004},{"className":2003},[],[2005],{"type":45,"value":21},{"type":45,"value":2007}," CLI). Run it from the shell where the AWS CLI works.",{"type":39,"tag":873,"props":2009,"children":2011},{"className":875,"code":2010,"language":877,"meta":878,"style":878},"# macOS \u002F Linux\nPRIVATE_URI=$(python3 scripts\u002Fmirror_image.py \\\n    public.ecr.aws\u002Fdeep-learning-containers\u002Fvllm:\u003Ctag> \\\n    vllm-mirror)\n",[2012],{"type":39,"tag":54,"props":2013,"children":2014},{"__ignoreMap":878},[2015,2023,2051,2081],{"type":39,"tag":884,"props":2016,"children":2017},{"class":886,"line":887},[2018],{"type":39,"tag":884,"props":2019,"children":2020},{"style":939},[2021],{"type":45,"value":2022},"# macOS \u002F Linux\n",{"type":39,"tag":884,"props":2024,"children":2025},{"class":886,"line":935},[2026,2031,2036,2041,2046],{"type":39,"tag":884,"props":2027,"children":2028},{"style":919},[2029],{"type":45,"value":2030},"PRIVATE_URI",{"type":39,"tag":884,"props":2032,"children":2033},{"style":908},[2034],{"type":45,"value":2035},"=$(",{"type":39,"tag":884,"props":2037,"children":2038},{"style":891},[2039],{"type":45,"value":2040},"python3",{"type":39,"tag":884,"props":2042,"children":2043},{"style":897},[2044],{"type":45,"value":2045}," scripts\u002Fmirror_image.py",{"type":39,"tag":884,"props":2047,"children":2048},{"style":919},[2049],{"type":45,"value":2050}," \\\n",{"type":39,"tag":884,"props":2052,"children":2053},{"class":886,"line":945},[2054,2059,2063,2068,2073,2077],{"type":39,"tag":884,"props":2055,"children":2056},{"style":897},[2057],{"type":45,"value":2058},"    public.ecr.aws\u002Fdeep-learning-containers\u002Fvllm:",{"type":39,"tag":884,"props":2060,"children":2061},{"style":908},[2062],{"type":45,"value":911},{"type":39,"tag":884,"props":2064,"children":2065},{"style":897},[2066],{"type":45,"value":2067},"ta",{"type":39,"tag":884,"props":2069,"children":2070},{"style":919},[2071],{"type":45,"value":2072},"g",{"type":39,"tag":884,"props":2074,"children":2075},{"style":908},[2076],{"type":45,"value":927},{"type":39,"tag":884,"props":2078,"children":2079},{"style":919},[2080],{"type":45,"value":2050},{"type":39,"tag":884,"props":2082,"children":2084},{"class":886,"line":2083},4,[2085,2090],{"type":39,"tag":884,"props":2086,"children":2087},{"style":897},[2088],{"type":45,"value":2089},"    vllm-mirror",{"type":39,"tag":884,"props":2091,"children":2092},{"style":908},[2093],{"type":45,"value":2094},")\n",{"type":39,"tag":873,"props":2096,"children":2100},{"className":2097,"code":2098,"language":2099,"meta":878,"style":878},"language-powershell shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Windows (PowerShell) — capture stdout into a variable\n$PRIVATE_URI = python scripts\\mirror_image.py `\n    public.ecr.aws\u002Fdeep-learning-containers\u002Fvllm:\u003Ctag> vllm-mirror\n","powershell",[2101],{"type":39,"tag":54,"props":2102,"children":2103},{"__ignoreMap":878},[2104,2112,2120],{"type":39,"tag":884,"props":2105,"children":2106},{"class":886,"line":887},[2107],{"type":39,"tag":884,"props":2108,"children":2109},{},[2110],{"type":45,"value":2111},"# Windows (PowerShell) — capture stdout into a variable\n",{"type":39,"tag":884,"props":2113,"children":2114},{"class":886,"line":935},[2115],{"type":39,"tag":884,"props":2116,"children":2117},{},[2118],{"type":45,"value":2119},"$PRIVATE_URI = python scripts\\mirror_image.py `\n",{"type":39,"tag":884,"props":2121,"children":2122},{"class":886,"line":945},[2123],{"type":39,"tag":884,"props":2124,"children":2125},{},[2126],{"type":45,"value":2127},"    public.ecr.aws\u002Fdeep-learning-containers\u002Fvllm:\u003Ctag> vllm-mirror\n",{"type":39,"tag":63,"props":2129,"children":2131},{"id":2130},"when-the-catalog-page-wont-render-is-stale-or-is-wrong",[2132],{"type":45,"value":2133},"When the catalog page won't render, is stale, or is wrong",{"type":39,"tag":48,"props":2135,"children":2136},{},[2137,2142],{"type":39,"tag":141,"props":2138,"children":2139},{},[2140],{"type":45,"value":2141},"The page won't render \u002F fetch returns junk",{"type":45,"value":2143},": the catalog page is JavaScript-heavy and some fetch tools get an empty shell. Fallbacks, in order:",{"type":39,"tag":149,"props":2145,"children":2146},{},[2147,2230,2346],{"type":39,"tag":153,"props":2148,"children":2149},{},[2150,2155,2157,2196,2198,2203,2204,2209,2210,2216,2217,2222,2223,2228],{"type":39,"tag":141,"props":2151,"children":2152},{},[2153],{"type":45,"value":2154},"The catalog's source data on GitHub",{"type":45,"value":2156}," — the page is generated from one YAML file per version, listing exact tags, CUDA, and Python versions. List a family's files, then fetch the newest:\n",{"type":39,"tag":873,"props":2158,"children":2160},{"className":875,"code":2159,"language":877,"meta":878,"style":878},"curl -s https:\u002F\u002Fapi.github.com\u002Frepos\u002Faws\u002Fdeep-learning-containers\u002Fcontents\u002Fdocs\u002Fsrc\u002Fdata\u002Fhuggingface-vllm\ncurl -s https:\u002F\u002Fraw.githubusercontent.com\u002Faws\u002Fdeep-learning-containers\u002Fmain\u002Fdocs\u002Fsrc\u002Fdata\u002Fhuggingface-vllm\u002F0.21.0-gpu-sagemaker.yml\n",[2161],{"type":39,"tag":54,"props":2162,"children":2163},{"__ignoreMap":878},[2164,2180],{"type":39,"tag":884,"props":2165,"children":2166},{"class":886,"line":887},[2167,2171,2175],{"type":39,"tag":884,"props":2168,"children":2169},{"style":891},[2170],{"type":45,"value":894},{"type":39,"tag":884,"props":2172,"children":2173},{"style":897},[2174],{"type":45,"value":900},{"type":39,"tag":884,"props":2176,"children":2177},{"style":897},[2178],{"type":45,"value":2179}," https:\u002F\u002Fapi.github.com\u002Frepos\u002Faws\u002Fdeep-learning-containers\u002Fcontents\u002Fdocs\u002Fsrc\u002Fdata\u002Fhuggingface-vllm\n",{"type":39,"tag":884,"props":2181,"children":2182},{"class":886,"line":935},[2183,2187,2191],{"type":39,"tag":884,"props":2184,"children":2185},{"style":891},[2186],{"type":45,"value":894},{"type":39,"tag":884,"props":2188,"children":2189},{"style":897},[2190],{"type":45,"value":900},{"type":39,"tag":884,"props":2192,"children":2193},{"style":897},[2194],{"type":45,"value":2195}," https:\u002F\u002Fraw.githubusercontent.com\u002Faws\u002Fdeep-learning-containers\u002Fmain\u002Fdocs\u002Fsrc\u002Fdata\u002Fhuggingface-vllm\u002F0.21.0-gpu-sagemaker.yml\n",{"type":45,"value":2197},"\nDirectory names match ECR repos (",{"type":39,"tag":54,"props":2199,"children":2201},{"className":2200},[],[2202],{"type":45,"value":79},{"type":45,"value":81},{"type":39,"tag":54,"props":2205,"children":2207},{"className":2206},[],[2208],{"type":45,"value":87},{"type":45,"value":81},{"type":39,"tag":54,"props":2211,"children":2213},{"className":2212},[],[2214],{"type":45,"value":2215},"huggingface-tei",{"type":45,"value":81},{"type":39,"tag":54,"props":2218,"children":2220},{"className":2219},[],[2221],{"type":45,"value":116},{"type":45,"value":81},{"type":39,"tag":54,"props":2224,"children":2226},{"className":2225},[],[2227],{"type":45,"value":137},{"type":45,"value":2229},", ...).",{"type":39,"tag":153,"props":2231,"children":2232},{},[2233,2238,2240],{"type":39,"tag":141,"props":2234,"children":2235},{},[2236],{"type":45,"value":2237},"Query ECR directly",{"type":45,"value":2239}," for current tags in the target region (works with credentials that can read the DLC registry; if it returns AccessDenied, use the YAML files):\n",{"type":39,"tag":873,"props":2241,"children":2243},{"className":875,"code":2242,"language":877,"meta":878,"style":878},"aws ecr describe-images --registry-id 763104351884 --repository-name huggingface-vllm \\\n    --region \u003Cregion> --query 'sort_by(imageDetails,&imagePushedAt)[-5:].imageTags' --output json\n",[2244],{"type":39,"tag":54,"props":2245,"children":2246},{"__ignoreMap":878},[2247,2289],{"type":39,"tag":884,"props":2248,"children":2249},{"class":886,"line":887},[2250,2254,2259,2264,2269,2275,2280,2285],{"type":39,"tag":884,"props":2251,"children":2252},{"style":891},[2253],{"type":45,"value":21},{"type":39,"tag":884,"props":2255,"children":2256},{"style":897},[2257],{"type":45,"value":2258}," ecr",{"type":39,"tag":884,"props":2260,"children":2261},{"style":897},[2262],{"type":45,"value":2263}," describe-images",{"type":39,"tag":884,"props":2265,"children":2266},{"style":897},[2267],{"type":45,"value":2268}," --registry-id",{"type":39,"tag":884,"props":2270,"children":2272},{"style":2271},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[2273],{"type":45,"value":2274}," 763104351884",{"type":39,"tag":884,"props":2276,"children":2277},{"style":897},[2278],{"type":45,"value":2279}," --repository-name",{"type":39,"tag":884,"props":2281,"children":2282},{"style":897},[2283],{"type":45,"value":2284}," huggingface-vllm",{"type":39,"tag":884,"props":2286,"children":2287},{"style":919},[2288],{"type":45,"value":2050},{"type":39,"tag":884,"props":2290,"children":2291},{"class":886,"line":935},[2292,2297,2302,2307,2312,2316,2321,2326,2331,2336,2341],{"type":39,"tag":884,"props":2293,"children":2294},{"style":897},[2295],{"type":45,"value":2296},"    --region",{"type":39,"tag":884,"props":2298,"children":2299},{"style":908},[2300],{"type":45,"value":2301}," \u003C",{"type":39,"tag":884,"props":2303,"children":2304},{"style":897},[2305],{"type":45,"value":2306},"regio",{"type":39,"tag":884,"props":2308,"children":2309},{"style":919},[2310],{"type":45,"value":2311},"n",{"type":39,"tag":884,"props":2313,"children":2314},{"style":908},[2315],{"type":45,"value":927},{"type":39,"tag":884,"props":2317,"children":2318},{"style":897},[2319],{"type":45,"value":2320}," --query",{"type":39,"tag":884,"props":2322,"children":2323},{"style":908},[2324],{"type":45,"value":2325}," '",{"type":39,"tag":884,"props":2327,"children":2328},{"style":897},[2329],{"type":45,"value":2330},"sort_by(imageDetails,&imagePushedAt)[-5:].imageTags",{"type":39,"tag":884,"props":2332,"children":2333},{"style":908},[2334],{"type":45,"value":2335},"'",{"type":39,"tag":884,"props":2337,"children":2338},{"style":897},[2339],{"type":45,"value":2340}," --output",{"type":39,"tag":884,"props":2342,"children":2343},{"style":897},[2344],{"type":45,"value":2345}," json\n",{"type":39,"tag":153,"props":2347,"children":2348},{},[2349,2356],{"type":39,"tag":236,"props":2350,"children":2353},{"href":2351,"rel":2352},"https:\u002F\u002Fgithub.com\u002Faws\u002Fdeep-learning-containers\u002Freleases",[240],[2354],{"type":45,"value":2355},"Release notes on the DLC GitHub repo",{"type":45,"value":270},{"type":39,"tag":48,"props":2358,"children":2359},{},[2360,2365],{"type":39,"tag":141,"props":2361,"children":2362},{},[2363],{"type":45,"value":2364},"A tag was just released and isn't on the page yet",{"type":45,"value":2366},": rare; AWS updates the page on each release. Check the release notes above.",{"type":39,"tag":48,"props":2368,"children":2369},{},[2370,2375,2377,2383,2385,2390,2392,2397],{"type":39,"tag":141,"props":2371,"children":2372},{},[2373],{"type":45,"value":2374},"An architecture you need isn't supported by the listed image yet",{"type":45,"value":2376},": for TEI specifically, you can mirror the upstream image from GHCR (",{"type":39,"tag":54,"props":2378,"children":2380},{"className":2379},[],[2381],{"type":45,"value":2382},"ghcr.io\u002Fhuggingface\u002Ftext-embeddings-inference:\u003Cversion>",{"type":45,"value":2384},") into private ECR and pass the resulting URI directly to ",{"type":39,"tag":54,"props":2386,"children":2388},{"className":2387},[],[2389],{"type":45,"value":268},{"type":45,"value":2391},". Same ",{"type":39,"tag":54,"props":2393,"children":2395},{"className":2394},[],[2396],{"type":45,"value":324},{"type":45,"value":2398}," script.",{"type":39,"tag":63,"props":2400,"children":2402},{"id":2401},"known-broken-images-last-checked-july-2026",[2403],{"type":45,"value":2404},"Known-broken images (last checked July 2026)",{"type":39,"tag":334,"props":2406,"children":2407},{},[2408,2429],{"type":39,"tag":338,"props":2409,"children":2410},{},[2411],{"type":39,"tag":342,"props":2412,"children":2413},{},[2414,2419,2424],{"type":39,"tag":346,"props":2415,"children":2416},{},[2417],{"type":45,"value":2418},"Image",{"type":39,"tag":346,"props":2420,"children":2421},{},[2422],{"type":45,"value":2423},"Defect",{"type":39,"tag":346,"props":2425,"children":2426},{},[2427],{"type":45,"value":2428},"Use instead",{"type":39,"tag":362,"props":2430,"children":2431},{},[2432],{"type":39,"tag":342,"props":2433,"children":2434},{},[2435,2451,2492],{"type":39,"tag":369,"props":2436,"children":2437},{},[2438,2443,2444,2449],{"type":39,"tag":54,"props":2439,"children":2441},{"className":2440},[],[2442],{"type":45,"value":108},{"type":45,"value":640},{"type":39,"tag":141,"props":2445,"children":2446},{},[2447],{"type":45,"value":2448},"GPU",{"type":45,"value":2450}," tags — all recent ones tested (PT 2.3–2.6, cu121\u002Fcu124, transformers 4.48–5.5.3)",{"type":39,"tag":369,"props":2452,"children":2453},{},[2454,2460,2462,2468,2470,2475,2477,2482,2484,2490],{"type":39,"tag":54,"props":2455,"children":2457},{"className":2456},[],[2458],{"type":45,"value":2459},"ImportError: libtorch_cuda.so: undefined symbol: ncclCommResume",{"type":45,"value":2461}," at ",{"type":39,"tag":54,"props":2463,"children":2465},{"className":2464},[],[2466],{"type":45,"value":2467},"import torch",{"type":45,"value":2469},". The NCCL bundled in the image is older than what torch links against — a packaging defect ",{"type":39,"tag":840,"props":2471,"children":2472},{},[2473],{"type":45,"value":2474},"inside the container",{"type":45,"value":2476},", on g5 ",{"type":39,"tag":141,"props":2478,"children":2479},{},[2480],{"type":45,"value":2481},"and",{"type":45,"value":2483}," g6, regardless of AMI, model, or inference code. The MMS Java front-end keeps answering ",{"type":39,"tag":54,"props":2485,"children":2487},{"className":2486},[],[2488],{"type":45,"value":2489},"\u002Fping",{"type":45,"value":2491},", so the endpoint can reach InService while the Python worker crash-loops and serves nothing.",{"type":39,"tag":369,"props":2493,"children":2494},{},[2495],{"type":45,"value":2496},"DJL Inference (bundles its own complete CUDA\u002FNCCL stack) or BYOC. CPU tags are unaffected.",{"type":39,"tag":48,"props":2498,"children":2499},{},[2500,2502,2507],{"type":45,"value":2501},"Re-check when AWS publishes new ",{"type":39,"tag":54,"props":2503,"children":2505},{"className":2504},[],[2506],{"type":45,"value":108},{"type":45,"value":2508}," GPU tags — remove the row once a fixed image is confirmed.",{"type":39,"tag":48,"props":2510,"children":2511},{},[2512,2517],{"type":39,"tag":141,"props":2513,"children":2514},{},[2515],{"type":45,"value":2516},"General fallback rule",{"type":45,"value":2518},": when an HF DLC fails with CUDA\u002FNCCL linker errors, switch to DJL Inference rather than iterating over sibling tags — the defect class is per-repo, not per-tag (three different tags were tried for the case above; all broken).",{"type":39,"tag":48,"props":2520,"children":2521},{},[2522,2527,2529,2535,2537,2542,2544,2550],{"type":39,"tag":141,"props":2523,"children":2524},{},[2525],{"type":45,"value":2526},"Related HF Hub gotcha",{"type":45,"value":2528},": older DLCs can fail model download with ",{"type":39,"tag":54,"props":2530,"children":2532},{"className":2531},[],[2533],{"type":45,"value":2534},"403 Forbidden",{"type":45,"value":2536}," from HF's XET CDN (their bundled ",{"type":39,"tag":54,"props":2538,"children":2540},{"className":2539},[],[2541],{"type":45,"value":676},{"type":45,"value":2543}," predates XET auth). Set ",{"type":39,"tag":54,"props":2545,"children":2547},{"className":2546},[],[2548],{"type":45,"value":2549},"HF_HUB_ENABLE_HF_TRANSFER=0",{"type":45,"value":2551}," to force the standard download path, or pre-stage weights in S3.",{"type":39,"tag":63,"props":2553,"children":2555},{"id":2554},"hub-download-time-at-first-boot",[2556],{"type":45,"value":2557},"Hub download time at first boot",{"type":39,"tag":48,"props":2559,"children":2560},{},[2561,2563,2568,2570,2575],{"type":45,"value":2562},"Loading the model from HF Hub happens ",{"type":39,"tag":840,"props":2564,"children":2565},{},[2566],{"type":45,"value":2567},"inside the container after the endpoint starts",{"type":45,"value":2569}," — expect ",{"type":39,"tag":141,"props":2571,"children":2572},{},[2573],{"type":45,"value":2574},"5–15+ minutes",{"type":45,"value":2576}," before InService even for small models, longer for multi-GB ones. A slow first boot is not a failure; don't tear down or re-diagnose before the deploy script's 30-minute wait expires.",{"type":39,"tag":48,"props":2578,"children":2579},{},[2580,2582,2588,2589,2594,2596,2601,2603,2608],{"type":45,"value":2581},"For production or repeated deployments, pre-stage the weights in S3 and pass ",{"type":39,"tag":54,"props":2583,"children":2585},{"className":2584},[],[2586],{"type":45,"value":2587},"--model-s3-uri",{"type":45,"value":1343},{"type":39,"tag":54,"props":2590,"children":2592},{"className":2591},[],[2593],{"type":45,"value":1349},{"type":45,"value":2595}," (the model then loads from ",{"type":39,"tag":54,"props":2597,"children":2599},{"className":2598},[],[2600],{"type":45,"value":1395},{"type":45,"value":2602},") — faster, immune to Hub rate limits\u002Foutages, and no ",{"type":39,"tag":54,"props":2604,"children":2606},{"className":2605},[],[2607],{"type":45,"value":1552},{"type":45,"value":2609}," needed at runtime.",{"type":39,"tag":2611,"props":2612,"children":2613},"style",{},[2614],{"type":45,"value":2615},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2617,"total":2781},[2618,2641,2655,2670,2681,2694,2705,2720,2733,2739,2752,2766],{"slug":2619,"name":2619,"fn":2620,"description":2621,"org":2622,"tags":2623,"stars":2638,"repoUrl":2639,"updatedAt":2640},"train-sentence-transformers","train sentence-transformers models","Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder; dense or static embedding model; for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker; pair scoring for two-stage retrieval \u002F pair classification), and `SparseEncoder` (SPLADE, sparse embedding model; for learned-sparse retrieval). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, Matryoshka, and Hugging Face Hub publishing. Use for any sentence-transformers training task.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2624,2627,2629,2632,2635],{"name":2625,"slug":2626,"type":15},"Deep Learning","deep-learning",{"name":9,"slug":2628,"type":15},"hugging-face",{"name":2630,"slug":2631,"type":15},"LLM","llm",{"name":2633,"slug":2634,"type":15},"Python","python",{"name":2636,"slug":2637,"type":15},"Search","search",18914,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fsentence-transformers","2026-05-08T05:09:16.820066",{"slug":2642,"name":2642,"fn":2643,"description":2644,"org":2645,"tags":2646,"stars":2652,"repoUrl":2653,"updatedAt":2654},"trl-training","train and fine-tune LLMs with TRL","Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2647,2648,2649,2650,2651],{"name":17,"slug":18,"type":15},{"name":2625,"slug":2626,"type":15},{"name":9,"slug":2628,"type":15},{"name":2630,"slug":2631,"type":15},{"name":2633,"slug":2634,"type":15},18850,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl","2026-04-06T18:25:32.746828",{"slug":2656,"name":2656,"fn":2657,"description":2658,"org":2659,"tags":2660,"stars":22,"repoUrl":23,"updatedAt":2669},"hf-cli","manage Hugging Face Hub resources via CLI","Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing models, datasets, spaces, buckets, repos, papers, jobs, and more on the Hugging Face Hub. Use when: handling authentication; managing local cache; managing Hugging Face Buckets; running or scheduling jobs on Hugging Face infrastructure; managing Hugging Face repos; discussions and pull requests; browsing models, datasets and spaces; reading, searching, or browsing academic papers; managing collections; querying datasets; configuring spaces; setting up webhooks; or deploying and managing HF Inference Endpoints. Make sure to use this skill whenever the user mentions 'hf', 'huggingface', 'Hugging Face', 'huggingface-cli', or 'hugging face cli', or wants to do anything related to the Hugging Face ecosystem and to AI and ML in general. Also use for cloud storage needs like training checkpoints, data pipelines, or agent traces. Use even if the user doesn't explicitly ask for a CLI command. Replaces the deprecated `huggingface-cli`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2661,2664,2667,2668],{"name":2662,"slug":2663,"type":15},"CLI","cli",{"name":2665,"slug":2666,"type":15},"Datasets","datasets",{"name":9,"slug":2628,"type":15},{"name":2630,"slug":2631,"type":15},"2026-04-06T18:25:34.020855",{"slug":1056,"name":1056,"fn":2671,"description":2672,"org":2673,"tags":2674,"stars":22,"repoUrl":23,"updatedAt":2680},"discover local AWS environment context","Discover the user's local AWS context (active profile, region, account ID, caller identity) at the start of any AWS task. Use this skill before any other AWS work — deploying to SageMaker, creating resources, calling AWS APIs, or anything that touches an AWS account. Use it especially when the user has not specified a region or profile explicitly, when they say things like \"use my AWS account\", \"deploy to AWS\", \"use my profile\", or when about to make any AWS CLI or SDK call. Never guess the region or account ID — always use this skill to read it from the local configuration first.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2675,2676,2677],{"name":20,"slug":21,"type":15},{"name":2662,"slug":2663,"type":15},{"name":2678,"slug":2679,"type":15},"Configuration","configuration","2026-07-08T05:55:33.716099",{"slug":2682,"name":2682,"fn":2683,"description":2684,"org":2685,"tags":2686,"stars":22,"repoUrl":23,"updatedAt":2693},"hf-cloud-python-env-setup","set up Python environments for AWS","Set up an isolated Python environment for SageMaker \u002F AWS work, with the right Python version and current boto3. Use this skill whenever Python code will be executed for a SageMaker deployment, training job, or any AWS automation — including when about to run `pip install`, when about to invoke `boto3`, when creating or activating a virtualenv, or when the user asks to \"set up the environment\". Never use system Python and never `pip install` into it. Always isolate. This skill prevents the most common failure modes: wrong Python version, dependency conflicts, and stale SDKs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2687,2688,2689,2692],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":2690,"slug":2691,"type":15},"Engineering","engineering",{"name":2633,"slug":2634,"type":15},"2026-07-08T05:55:32.505017",{"slug":2695,"name":2695,"fn":2696,"description":2697,"org":2698,"tags":2699,"stars":22,"repoUrl":23,"updatedAt":2704},"hf-cloud-sagemaker-deployment-planner","plan model deployments to Amazon SageMaker","Plan and coordinate the deployment of a model to Amazon SageMaker AI. Use this skill whenever the user wants to deploy, host, serve, or expose a model on SageMaker or AWS — including phrases like \"deploy a model\", \"host this LLM on AWS\", \"serve this embedding model\", \"deploy a reranker\", \"deploy a text-to-image \u002F diffusion model\", \"host this for async inference\", \"create an endpoint\", \"serve my fine-tuned model\", or any request that involves making a model available for inference on AWS. Use this even when the user is vague (e.g. \"I just want to get this running on AWS, you figure it out\"). Works for text-generation LLMs, embedding models, rerankers, classifiers, text-to-image \u002F diffusion models — picks the right serving stack and chooses between real-time and async inference. This is the entry-point skill for SageMaker deployment work — it asks clarifying questions, picks a deployment pathway, and coordinates the other deployment skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2700,2701,2702,2703],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":2628,"type":15},"2026-07-08T05:55:37.387689",{"slug":2706,"name":2706,"fn":2707,"description":2708,"org":2709,"tags":2710,"stars":22,"repoUrl":23,"updatedAt":2719},"hf-cloud-sagemaker-iam-preflight","configure SageMaker IAM roles","Ensure a usable SageMaker execution role exists before deploying or training. Use this skill whenever about to create a SageMaker endpoint, model, training job, or any resource that requires an execution role. Use it especially when the user has not provided a role ARN explicitly, when scripts are about to call `iam:CreateRole`, or when an AccessDenied error mentions an IAM action. Never blindly call `iam:CreateRole` — always check for existing roles first. This skill prevents the most common SageMaker deployment failure: trying to create IAM resources from an SSO principal that has no IAM write permissions.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2711,2712,2713,2716],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":2714,"slug":2715,"type":15},"Permissions","permissions",{"name":2717,"slug":2718,"type":15},"Security","security","2026-07-08T05:55:34.948771",{"slug":860,"name":860,"fn":2721,"description":2722,"org":2723,"tags":2724,"stars":22,"repoUrl":23,"updatedAt":2732},"create production-ready SageMaker endpoints","Create a SageMaker endpoint (real-time or async) with autoscaling, CloudWatch alarms, and tagging enabled by default. Use this skill whenever about to create a SageMaker endpoint, write deployment code that calls `create_endpoint`, or finalize a deployment after the image URI and IAM role are known. Provides deploy.py for real-time endpoints and deploy_async.py for async endpoints (with genuine scale-to-zero support). This is the last step in the SageMaker deployment workflow. Never generate a bare `create_endpoint` call without these defaults — endpoints without autoscaling or alarms are demos, not deployments.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2725,2726,2727,2728,2729],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":2628,"type":15},{"name":2730,"slug":2731,"type":15},"Monitoring","monitoring","2026-07-08T05:55:38.664702",{"slug":4,"name":4,"fn":5,"description":6,"org":2734,"tags":2735,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2736,2737,2738],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},{"slug":2740,"name":2740,"fn":2741,"description":2742,"org":2743,"tags":2744,"stars":22,"repoUrl":23,"updatedAt":2751},"hf-mcp","access Hugging Face Hub via MCP","Use Hugging Face Hub via MCP server tools. Search models, datasets, Spaces, papers. Get repo details, fetch documentation, run compute jobs, and use Gradio Spaces as AI tools. Available when connected to the HF MCP server.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2745,2746,2747,2748],{"name":2665,"slug":2666,"type":15},{"name":9,"slug":2628,"type":15},{"name":2630,"slug":2631,"type":15},{"name":2749,"slug":2750,"type":15},"MCP","mcp","2026-04-06T18:25:50.364185",{"slug":2753,"name":2753,"fn":2754,"description":2755,"org":2756,"tags":2757,"stars":22,"repoUrl":23,"updatedAt":2765},"hf-mem","estimate memory for Hugging Face models","Hugging Face CLI to estimate the required memory to load Safetensors or GGUF model weights for inference from the Hugging Face Hub",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2758,2759,2760,2761,2762],{"name":17,"slug":18,"type":15},{"name":2662,"slug":2663,"type":15},{"name":9,"slug":2628,"type":15},{"name":2630,"slug":2631,"type":15},{"name":2763,"slug":2764,"type":15},"Performance","performance","2026-06-13T07:23:57.101435",{"slug":2767,"name":2767,"fn":2768,"description":2769,"org":2770,"tags":2771,"stars":22,"repoUrl":23,"updatedAt":2780},"huggingface-best","find and compare Hugging Face models","Use when the user asks about finding the best, top, or recommended model for a task, wants to know what AI model to use, or wants to compare models by benchmark scores. Triggers on: \"best model for X\", \"what model should I use for\", \"top models for [task]\", \"which model runs on my laptop\u002Fmachine\u002Fdevice\", \"recommend a model for\", \"what LLM should I use for\", \"compare models for\", \"what's state of the art for\", or any question about choosing an AI model for a specific use case. Always use this skill when the user wants model recommendations or comparisons, even if they don't explicitly mention HuggingFace or benchmarks.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2772,2775,2776,2777],{"name":2773,"slug":2774,"type":15},"Analytics","analytics",{"name":9,"slug":2628,"type":15},{"name":2630,"slug":2631,"type":15},{"name":2778,"slug":2779,"type":15},"Research","research","2026-04-24T05:09:45.870658",37,{"items":2783,"total":2832},[2784,2791,2797,2804,2811,2818,2826],{"slug":2656,"name":2656,"fn":2657,"description":2658,"org":2785,"tags":2786,"stars":22,"repoUrl":23,"updatedAt":2669},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2787,2788,2789,2790],{"name":2662,"slug":2663,"type":15},{"name":2665,"slug":2666,"type":15},{"name":9,"slug":2628,"type":15},{"name":2630,"slug":2631,"type":15},{"slug":1056,"name":1056,"fn":2671,"description":2672,"org":2792,"tags":2793,"stars":22,"repoUrl":23,"updatedAt":2680},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2794,2795,2796],{"name":20,"slug":21,"type":15},{"name":2662,"slug":2663,"type":15},{"name":2678,"slug":2679,"type":15},{"slug":2682,"name":2682,"fn":2683,"description":2684,"org":2798,"tags":2799,"stars":22,"repoUrl":23,"updatedAt":2693},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2800,2801,2802,2803],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":2690,"slug":2691,"type":15},{"name":2633,"slug":2634,"type":15},{"slug":2695,"name":2695,"fn":2696,"description":2697,"org":2805,"tags":2806,"stars":22,"repoUrl":23,"updatedAt":2704},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2807,2808,2809,2810],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":2628,"type":15},{"slug":2706,"name":2706,"fn":2707,"description":2708,"org":2812,"tags":2813,"stars":22,"repoUrl":23,"updatedAt":2719},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2814,2815,2816,2817],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":2714,"slug":2715,"type":15},{"name":2717,"slug":2718,"type":15},{"slug":860,"name":860,"fn":2721,"description":2722,"org":2819,"tags":2820,"stars":22,"repoUrl":23,"updatedAt":2732},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2821,2822,2823,2824,2825],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":2628,"type":15},{"name":2730,"slug":2731,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":2827,"tags":2828,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2829,2830,2831],{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},24]