[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-huggingface-hf-cloud-sagemaker-production-defaults":3,"mdc--bfpcor-key":38,"related-org-huggingface-hf-cloud-sagemaker-production-defaults":3401,"related-repo-huggingface-hf-cloud-sagemaker-production-defaults":3561},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":27,"repoUrl":28,"updatedAt":29,"license":30,"forks":31,"topics":32,"repo":33,"sourceUrl":36,"mdContent":37},"hf-cloud-sagemaker-production-defaults","create production-ready SageMaker endpoints","Create a SageMaker endpoint (real-time or async) with autoscaling, CloudWatch alarms, and tagging enabled by default. Use this skill whenever about to create a SageMaker endpoint, write deployment code that calls `create_endpoint`, or finalize a deployment after the image URI and IAM role are known. Provides deploy.py for real-time endpoints and deploy_async.py for async endpoints (with genuine scale-to-zero support). This is the last step in the SageMaker deployment workflow. Never generate a bare `create_endpoint` call without these defaults — endpoints without autoscaling or alarms are demos, not deployments.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"huggingface","Hugging Face","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fhuggingface.png",[12,15,18,21,24],{"name":9,"slug":13,"type":14},"hugging-face","tag",{"name":16,"slug":17,"type":14},"Monitoring","monitoring",{"name":19,"slug":20,"type":14},"Deployment","deployment",{"name":22,"slug":23,"type":14},"AI Infrastructure","ai-infrastructure",{"name":25,"slug":26,"type":14},"AWS","aws",10861,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fskills","2026-07-08T05:55:38.664702",null,721,[],{"repoUrl":28,"stars":27,"forks":31,"topics":34,"description":35},[],"Give your agents the power of the Hugging Face ecosystem","https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fhf-cloud-sagemaker-production-defaults","---\nname: hf-cloud-sagemaker-production-defaults\ndescription: 'Create a SageMaker endpoint (real-time or async) with autoscaling, CloudWatch alarms, and tagging enabled by default. Use this skill whenever about to create a SageMaker endpoint, write deployment code that calls `create_endpoint`, or finalize a deployment after the image URI and IAM role are known. Provides deploy.py for real-time endpoints and deploy_async.py for async endpoints (with genuine scale-to-zero support). This is the last step in the SageMaker deployment workflow. Never generate a bare `create_endpoint` call without these defaults — endpoints without autoscaling or alarms are demos, not deployments.'\n---\n\n# SageMaker Production Defaults\n\nThe difference between a demo endpoint and one you can leave running is: it scales with traffic, it tells you when it breaks, and you can debug it later. This skill makes those three the default rather than optional extras.\n\nBy the time this skill runs, the planner has chosen a real-time endpoint, IAM has a usable role, and image-selection has resolved a container URI + AMI version. This skill turns those into an actual deployment.\n\n## What gets created\n\nFor every endpoint, the skill creates these as a unit:\n\n1. **SageMaker Model** — image + env vars + execution role + S3 artifacts\n2. **Endpoint config** — instance type, initial count, optional data capture\n3. **Endpoint** — the real-time endpoint serving inference\n4. **Autoscaling target + policy** — target tracking on invocations per instance\n5. **CloudWatch alarms** — latency, errors, platform overhead\n\nData capture (logging requests\u002Fresponses to S3) is **off by default** — useful for debugging but creates ongoing S3 costs the user didn't necessarily ask for. Enable with `--enable-data-capture`.\n\nAll resources get a consistent tag set including `CreatedBy=agentic-deploy-skills` for later cleanup.\n\nDefaults and reasoning in `references\u002Fdeployment-template.md`.\n\n## Running the deployment\n\nFor a text-generation LLM (vLLM):\n\n```bash\npython scripts\u002Fdeploy.py \\\n    --model-name qwen3-medical \\\n    --image-uri \"$IMAGE_URI\" \\\n    --inference-ami-version \"$AMI\" \\\n    --role-arn \"$ROLE_ARN\" \\\n    --instance-type ml.g5.xlarge \\\n    --region \"$REGION\" \\\n    --env SM_VLLM_MODEL=Qwen\u002FQwen3-0.6B \\\n    --env SM_VLLM_HOST=0.0.0.0 \\\n    --env SM_VLLM_TRUST_REMOTE_CODE=true \\\n    --env SM_VLLM_MAX_MODEL_LEN=4096\n```\n\nFor an embedding model (TEI, often on CPU):\n\n```bash\npython scripts\u002Fdeploy.py \\\n    --model-name bge-large-embeddings \\\n    --image-uri \"$IMAGE_URI\" \\\n    --role-arn \"$ROLE_ARN\" \\\n    --instance-type ml.c6i.2xlarge \\\n    --region \"$REGION\" \\\n    --env HF_MODEL_ID=BAAI\u002Fbge-large-en-v1.5\n```\n\nNote: TEI deployments **do not** need `--inference-ami-version`. That flag is vLLM-specific. TEI env vars are also simpler (`HF_MODEL_ID` instead of `SM_VLLM_*`, no host or trust-remote-code to configure).\n\nWhere each value comes from:\n\n| Parameter | Source |\n|---|---|\n| `--image-uri` | `hf-cloud-serving-image-selection` — agent reads from the AWS DLC catalog page |\n| `--inference-ami-version` | `hf-cloud-serving-image-selection` — required for vLLM tags containing cu130+ |\n| `--role-arn` | `hf-cloud-sagemaker-iam-preflight` (`check_role.py`) |\n| `--region` | `hf-cloud-aws-context-discovery` |\n| `--instance-type` | User input or planner recommendation |\n| `--env` | Model-specific; see `hf-cloud-serving-image-selection` for required `SM_VLLM_*` vars |\n| `--model-s3-uri` | Optional — S3 path to model artifacts; omit if loading from HF Hub |\n\nThe script creates resources in order with error handling, waits for `InService` (up to 30 min), surfaces failure reasons, registers autoscaling and alarms, and prints a summary including the teardown command. Outputs a JSON blob on stdout with endpoint\u002Fconfig\u002Fmodel names for downstream scripting.\n\nThe scripts ship with this skill. If the installed copy is missing the `scripts\u002F` directory (some harnesses copy only SKILL.md on install), fetch them from the source repo rather than re-implementing them from this description.\n\n**Cold-start expectation**: when the model loads from HF Hub, the download happens inside the container after the endpoint starts — 5–15+ minutes to InService is normal, not a failure. `deploy.py` waits 30 minutes; if you write custom wait code, don't time out at 15. Pre-staging weights in S3 (`--model-s3-uri`) cuts this and removes the Hub dependency.\n\n## InService is not success — smoke-test before declaring victory\n\n`InService` only means the container answered `\u002Fping`. In MMS-based containers (HF Inference Toolkit) the Java front-end answers pings even while the Python worker crash-loops — an endpoint can be InService and serve nothing. Two checks, always:\n\n1. **One real invocation.**\n   - Real-time: `invoke_endpoint.py` (below) with a minimal payload; require an HTTP 200 with a sane body.\n   - Async: upload one input to S3, call `invoke-endpoint-async`, poll the output URI for a few minutes (see \"Invoking async endpoints\"). A result object = success; an object at the failure URI, or nothing appearing, = broken.\n2. **Scan the endpoint logs for worker-crash markers** — catches the crash-loop case even when the smoke request merely times out:\n\n   ```bash\n   aws logs filter-log-events \\\n       --log-group-name \u002Faws\u002Fsagemaker\u002FEndpoints\u002F\u003Cendpoint-name> \\\n       --filter-pattern '?\"Worker died\" ?\"Load model failed\" ?\"ImportError\"' \\\n       --region \u003Cregion> --max-items 5\n   ```\n\nOnly report the deployment complete after both pass. If the log scan hits, surface the actual traceback from CloudWatch — not the InService status.\n\n## Testing a real-time endpoint\n\nOnce the endpoint is `InService`, test it with the bundled helper. It is cross-platform and **BOM-safe** — use it instead of hand-writing a payload file and calling `invoke-endpoint` directly:\n\n```bash\n# macOS \u002F Linux\npython3 scripts\u002Finvoke_endpoint.py \\\n    --endpoint-name \u003Cendpoint-name> \\\n    --payload '{\"inputs\": \"Hello\"}' \\\n    --region \"$REGION\"\n```\n\n```powershell\n# Windows (PowerShell)\npython scripts\\invoke_endpoint.py `\n    --endpoint-name \u003Cendpoint-name> `\n    --payload-file payload.json `\n    --region $REGION\n```\n\nIt accepts either `--payload '\u003Cjson>'` (inline) or `--payload-file \u003Cpath>`, validates JSON, writes the request body as plain UTF-8, invokes the endpoint, and prints the response body to stdout.\n\n### The UTF-8 BOM gotcha (Windows)\n\nIf you write the request payload yourself on Windows, **do not** use `Set-Content -Encoding UTF8` — depending on the PowerShell version it prepends a UTF-8 byte-order mark (BOM). SageMaker's JSON parser rejects a BOM with a 400 `ModelError`:\n\n```\nUnexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)\n```\n\nThis is **not** a model, endpoint-health, or image problem — only the file encoding of the request body. `invoke_endpoint.py` avoids it entirely (it even strips a BOM from a `--payload-file` that already has one). If you must call the CLI directly, write the body as BOM-free UTF-8:\n\n```powershell\n# BOM-free UTF-8 — use this\n[System.IO.File]::WriteAllText((Resolve-Path \"payload.json\"), $json, [System.Text.UTF8Encoding]::new($false))\n\naws sagemaker-runtime invoke-endpoint `\n    --endpoint-name \u003Cendpoint-name> `\n    --content-type application\u002Fjson `\n    --body fileb:\u002F\u002Fpayload.json `\n    --region $REGION `\n    response.json\n```\n\n**Fallback:** if any invocation fails with `Unexpected UTF-8 BOM`, rewrite the payload as BOM-free UTF-8 (or re-run via `invoke_endpoint.py`) and retry once before treating the endpoint or model as broken.\n\n### Invoking a generative reranker (vLLM)\n\nGenerative rerankers (Qwen3-Reranker etc. — routed to the HuggingFace vLLM DLC by `hf-cloud-serving-image-selection`) are causal LMs scored by their first generated token, not chat models. Use the **completions API with a raw `prompt`**, not the messages\u002Fchat API: chat templating does not reliably honor `chat_template_kwargs` such as `{\"enable_thinking\": false}`, and a wrong template silently returns near-identical scores for every query–document pair instead of erroring.\n\nPayload shape (Qwen3-Reranker's expected format — substitute `{query}` \u002F `{document}`):\n\n```json\n{\n  \"prompt\": \"\u003C|im_start|>system\\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \\\"yes\\\" or \\\"no\\\".\u003C|im_end|>\\n\u003C|im_start|>user\\n\u003CInstruct>: Given a web search query, retrieve relevant passages that answer the query\\n\u003CQuery>: {query}\\n\u003CDocument>: {document}\u003C|im_end|>\\n\u003C|im_start|>assistant\\n\u003Cthink>\\n\\n\u003C\u002Fthink>\\n\\n\",\n  \"max_tokens\": 1,\n  \"temperature\": 0,\n  \"logprobs\": 20\n}\n```\n\nThe trailing `\u003C|im_start|>assistant\\n\u003Cthink>\\n\\n\u003C\u002Fthink>\\n\\n` suffix is load-bearing: it pre-fills an empty thinking block so the first generated token is the yes\u002Fno judgment. Score from the returned logprobs: `P(\"yes\") \u002F (P(\"yes\") + P(\"no\"))`. Sanity check the endpoint with one relevant pair (expect >0.9) and one irrelevant pair (expect \u003C0.05) — near-identical scores across pairs mean the prompt template is wrong, not that the model is broken.\n\nThe same rule generalizes: for any thinking-mode model where the prompt must be byte-exact, prefer the raw completions API over chat.\n\n### Picking the image URI\n\nThe agent reads the image URI from AWS's [Deep Learning Containers catalog](https:\u002F\u002Faws.github.io\u002Fdeep-learning-containers\u002Freference\u002Favailable_images\u002F) — pick the row that matches the model family (HuggingFace vLLM for LLMs, TEI for embeddings, etc.), substitute `\u003Cregion>` with the deployment region, and pass to `deploy.py --image-uri`.\n\nFor vLLM images specifically (both `huggingface-vllm` and the AWS `vllm` fallback), also check the tag's CUDA version:\n\n```bash\n# Example: HuggingFace vLLM 0.21.0 from the catalog\nIMAGE_URI=\"763104351884.dkr.ecr.eu-west-1.amazonaws.com\u002Fhuggingface-vllm:0.21.0-transformers5.8.1-gpu-py312-cu130-ubuntu22.04\"\n\n# cu130 tag → must pass --inference-ami-version\npython deploy.py --image-uri \"$IMAGE_URI\" \\\n    --inference-ami-version al2-ami-sagemaker-inference-gpu-3-1 \\\n    ...\n```\n\nFor tags with `cu129` or lower, omit `--inference-ami-version`. See `hf-cloud-serving-image-selection` for the full vLLM AMI lookup table and the env-var requirements for each image family.\n\n## Async inference deployments\n\nFor long-running inferences (>60s), large payloads, or workloads that are bursty\u002Fsparse enough to benefit from scale-to-zero, use `deploy_async.py` instead of `deploy.py`. Async genuinely supports `MinCapacity=0` — real-time autoscaling can't.\n\n```bash\npython scripts\u002Fdeploy_async.py \\\n    --model-name flux-text-to-image \\\n    --image-uri \"$IMAGE_URI\" \\\n    --role-arn \"$ROLE_ARN\" \\\n    --instance-type ml.g5.2xlarge \\\n    --region \"$REGION\" \\\n    --output-s3-uri s3:\u002F\u002Fmy-bucket\u002Fasync-output\u002F \\\n    --env HF_MODEL_ID=black-forest-labs\u002FFLUX.1-dev\n```\n\nRequired extras over `deploy.py`:\n- `--output-s3-uri` — where async results land (results are not returned synchronously)\n\nOptional async-specific flags:\n- `--failure-s3-uri` — separate path for failed invocations\n- `--success-sns-topic`, `--error-sns-topic` — get notified when async results are ready or fail\n- `--min-capacity 0` (the default) — scale to zero between batches\n- `--backlog-per-instance-target N` — target queue depth per instance (default 5)\n- `--max-concurrent-invocations-per-instance N` — default 4\n\n### How scale-to-zero works\n\nThe async script registers **two** autoscaling policies on the variant:\n\n1. **Target-tracking** on `ApproximateBacklogSizePerInstance` — handles ongoing scaling between min and max\n2. **Step-scaling** triggered by a `HasBacklogWithoutCapacity` CloudWatch alarm — handles `0→1` wake-from-zero\n\nBoth are needed. Target-tracking alone cannot transition from zero (it can't divide by zero instances), so without the step policy the endpoint comes up, scales to zero after the first batch, and never wakes again. The script wires this up automatically.\n\n### Async alarms\n\nThe script creates three CloudWatch alarms:\n- `ApproximateBacklogSize > 50` — queue is building faster than capacity can drain it\n- `InvocationsFailed > 5` — repeated processing failures\n- `HasBacklogWithoutCapacity` — drives the wake-from-zero policy (not a notification alarm; its action is the step-scaling policy, not the SNS topic)\n\nIf you pass `--sns-alarm-topic \u003Carn>`, the first two notify on that topic. The wake alarm always points at the step policy.\n\n### Invoking async endpoints\n\nAsync endpoints aren't called synchronously. You upload the input to S3, call `invoke-endpoint-async` with the S3 input location, and SageMaker writes the result to your `--output-s3-uri` when done:\n\n```bash\n# Upload your input first\naws s3 cp input.json s3:\u002F\u002Fmy-input-bucket\u002Fjob1\u002Finput.json\n\n# Invoke\naws sagemaker-runtime invoke-endpoint-async \\\n    --endpoint-name \u003Cendpoint-name> \\\n    --input-location s3:\u002F\u002Fmy-input-bucket\u002Fjob1\u002Finput.json \\\n    --content-type application\u002Fjson \\\n    --region \u003Cregion>\n\n# Poll for the result at your output URI\naws s3 cp s3:\u002F\u002Fmy-bucket\u002Fasync-output\u002F\u003Cinference-id>.out result.json\n```\n\nThe same UTF-8 BOM caveat applies to the `input.json` you upload (see \"The UTF-8 BOM gotcha\" above) — if you build it on Windows, write it as BOM-free UTF-8 or the container's JSON parser will reject it.\n\nTeardown works the same as real-time: `python3 scripts\u002Fteardown.py \u003Cendpoint-name>` (the teardown script discovers policies and alarms by name prefix, so it handles both deployment modes).\n\n## Defaults at a glance\n\n| Setting | Default | Override |\n|---|---|---|\n| Initial instance count | 1 | `--initial-instance-count` |\n| Autoscaling min \u002F max | 1 \u002F 4 | `--min-capacity`, `--max-capacity` |\n| Autoscaling target | 20 invocations\u002Fmin\u002Finstance | `--target-invocations-per-instance` |\n| Data capture | disabled (opt-in) | `--enable-data-capture` |\n| CloudWatch alarms | 3 alarms | `--no-alarms` |\n| SNS notification | none (alarms created but won't notify) | `--sns-alarm-topic \u003Carn>` |\n| Environment tag | `dev` | `--environment` |\n| InferenceAmiVersion | none (SageMaker default) | `--inference-ami-version` (REQUIRED for vLLM CUDA 13+) |\n\nNot defaulted (user-specific input needed): VPC config, KMS key, multi-variant, async inference.\n\n### Autoscaling target — tune by model type\n\nThe default `--target-invocations-per-instance 20` is conservative and tuned for LLM workloads where each request takes 1–5 seconds. For embedding deployments (TEI), each request is much faster (typically \u003C100ms on CPU, \u003C20ms on GPU), so a single instance can handle far more throughput. **For embedding deployments, raise the target to 100–500** depending on instance and model size. The default of 20 will trigger autoscaling far too aggressively for embeddings and waste money.\n\nA rule of thumb: target value ≈ 60 \u002F (typical request latency in seconds). LLM at 3s latency → target 20. Embedding at 100ms → target 600. Generative rerankers sit in between — they generate a single token per request, so ~40–100 is a reasonable target.\n\n## Data capture + IAM gotcha\n\nIf the user enables data capture, the execution role needs S3 write access to the capture prefix. The default URI (`s3:\u002F\u002Fsagemaker-\u003Cregion>-\u003Caccount>\u002F\u003Cendpoint>\u002Fdata-capture\u002F`) is typically a different bucket than the model artifact bucket. If `hf-cloud-sagemaker-iam-preflight` scoped the inline policy narrowly to just the model bucket, capture writes fail silently — endpoint keeps serving but no data appears.\n\nIf the user reports \"data capture isn't showing up\", check the role's S3 access. Either widen the inline policy or pass `--data-capture-s3-uri` pointing to a bucket the role can write.\n\n## Teardown\n\n```bash\npython3 scripts\u002Fteardown.py \u003Cendpoint-name> \u003Cregion>   # macOS \u002F Linux\npython  scripts\\teardown.py \u003Cendpoint-name> \u003Cregion>   # Windows\n```\n\nDeletes in safe order: alarms → autoscaling → endpoint (stops billing) → endpoint config → model. Idempotent.\n\nDoes **not** delete: the IAM execution role (might be shared), data capture S3 objects (user might want to keep), SNS topic, original model artifacts.\n\nAlways tell the user about the teardown command after the deployment summary. Users forget; endpoints accrue cost.\n\n## When the deployment fails\n\n**`CannotStartContainerError` + no CloudWatch logs ever created** — the InferenceAmiVersion problem. If the image tag contains `cu130` or later and you didn't pass `--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1`, this is the cause. See `hf-cloud-serving-image-selection`. Do NOT chase images, IAM roles, env vars, or instance types — the failure signature is identical for many other things but the cause here is the AMI.\n\n**\"Failed to pass ping health check\"** — the container *did* start and produced logs, but `\u002Fping` isn't responding. Check CloudWatch at `\u002Faws\u002Fsagemaker\u002FEndpoints\u002F\u003Cendpoint-name>`. Usually: wrong image for model architecture, missing HF token, or OOM.\n\n**\"Container failed to start\" (with logs present)** — entrypoint ran, then exited. Check CloudWatch. Common: missing required env vars (`SM_VLLM_MODEL`, `SM_VLLM_HOST`, `SM_VLLM_TRUST_REMOTE_CODE`), wrong `ModelDataUrl` format, unreadable model artifacts.\n\n**`ResourceLimitExceeded`** — no quota for the instance type in this region. Request increase or pick a different type (the planner should have checked quotas up front — see `hf-cloud-sagemaker-deployment-planner`).\n\n**`ImportError: libtorch_cuda.so: undefined symbol: ncclCommResume` in CloudWatch logs** — known packaging defect in `huggingface-pytorch-inference` GPU images (see \"Known-broken images\" in `hf-cloud-serving-image-selection`). Inside the container, so no env var, AMI, instance type, or sibling tag fixes it. Switch to DJL Inference.\n\n**InService, but invocations time out \u002F async outputs never appear** — dead Python worker behind a live MMS front-end. Run the log scan from \"InService is not success\" above; the traceback in CloudWatch is the real error.\n\n**`403 Forbidden` downloading weights from HF Hub during startup** — the container's bundled `huggingface_hub` predates HF's XET CDN auth. Add `--env HF_HUB_ENABLE_HF_TRANSFER=0`, or pre-stage the weights in S3. Note: this can *mask* a deeper failure (the worker may still crash after the download succeeds) — re-check logs after fixing it.\n\n**Diagnostic rule**: when failures look identical across multiple configurations (different images, roles, instance types) and **no logs are ever produced**, the cause is almost always below the container — host AMI, networking, account-level — not the deployment config. Stop iterating on config; check the AMI version and account state.\n\nDon't retry blindly. The script prints the specific `FailureReason` from `describe-endpoint` — fix the root cause before retrying.\n",{"data":39,"body":40},{"name":4,"description":6},{"type":41,"children":42},"root",[43,52,58,63,70,75,131,152,165,177,183,188,447,452,587,623,628,817,830,843,868,874,892,1071,1076,1082,1109,1215,1264,1285,1292,1319,1329,1356,1435,1460,1466,1507,1528,1790,1811,1816,1822,1853,1874,1987,2014,2020,2047,2200,2211,2225,2230,2296,2302,2314,2361,2366,2372,2377,2412,2425,2431,2450,2676,2689,2702,2708,2920,2925,2931,2951,2956,2962,2982,2995,3001,3116,3121,3132,3137,3143,3182,3215,3255,3277,3308,3318,3357,3374,3395],{"type":44,"tag":45,"props":46,"children":48},"element","h1",{"id":47},"sagemaker-production-defaults",[49],{"type":50,"value":51},"text","SageMaker Production Defaults",{"type":44,"tag":53,"props":54,"children":55},"p",{},[56],{"type":50,"value":57},"The difference between a demo endpoint and one you can leave running is: it scales with traffic, it tells you when it breaks, and you can debug it later. This skill makes those three the default rather than optional extras.",{"type":44,"tag":53,"props":59,"children":60},{},[61],{"type":50,"value":62},"By the time this skill runs, the planner has chosen a real-time endpoint, IAM has a usable role, and image-selection has resolved a container URI + AMI version. This skill turns those into an actual deployment.",{"type":44,"tag":64,"props":65,"children":67},"h2",{"id":66},"what-gets-created",[68],{"type":50,"value":69},"What gets created",{"type":44,"tag":53,"props":71,"children":72},{},[73],{"type":50,"value":74},"For every endpoint, the skill creates these as a unit:",{"type":44,"tag":76,"props":77,"children":78},"ol",{},[79,91,101,111,121],{"type":44,"tag":80,"props":81,"children":82},"li",{},[83,89],{"type":44,"tag":84,"props":85,"children":86},"strong",{},[87],{"type":50,"value":88},"SageMaker Model",{"type":50,"value":90}," — image + env vars + execution role + S3 artifacts",{"type":44,"tag":80,"props":92,"children":93},{},[94,99],{"type":44,"tag":84,"props":95,"children":96},{},[97],{"type":50,"value":98},"Endpoint config",{"type":50,"value":100}," — instance type, initial count, optional data capture",{"type":44,"tag":80,"props":102,"children":103},{},[104,109],{"type":44,"tag":84,"props":105,"children":106},{},[107],{"type":50,"value":108},"Endpoint",{"type":50,"value":110}," — the real-time endpoint serving inference",{"type":44,"tag":80,"props":112,"children":113},{},[114,119],{"type":44,"tag":84,"props":115,"children":116},{},[117],{"type":50,"value":118},"Autoscaling target + policy",{"type":50,"value":120}," — target tracking on invocations per instance",{"type":44,"tag":80,"props":122,"children":123},{},[124,129],{"type":44,"tag":84,"props":125,"children":126},{},[127],{"type":50,"value":128},"CloudWatch alarms",{"type":50,"value":130}," — latency, errors, platform overhead",{"type":44,"tag":53,"props":132,"children":133},{},[134,136,141,143,150],{"type":50,"value":135},"Data capture (logging requests\u002Fresponses to S3) is ",{"type":44,"tag":84,"props":137,"children":138},{},[139],{"type":50,"value":140},"off by default",{"type":50,"value":142}," — useful for debugging but creates ongoing S3 costs the user didn't necessarily ask for. Enable with ",{"type":44,"tag":144,"props":145,"children":147},"code",{"className":146},[],[148],{"type":50,"value":149},"--enable-data-capture",{"type":50,"value":151},".",{"type":44,"tag":53,"props":153,"children":154},{},[155,157,163],{"type":50,"value":156},"All resources get a consistent tag set including ",{"type":44,"tag":144,"props":158,"children":160},{"className":159},[],[161],{"type":50,"value":162},"CreatedBy=agentic-deploy-skills",{"type":50,"value":164}," for later cleanup.",{"type":44,"tag":53,"props":166,"children":167},{},[168,170,176],{"type":50,"value":169},"Defaults and reasoning in ",{"type":44,"tag":144,"props":171,"children":173},{"className":172},[],[174],{"type":50,"value":175},"references\u002Fdeployment-template.md",{"type":50,"value":151},{"type":44,"tag":64,"props":178,"children":180},{"id":179},"running-the-deployment",[181],{"type":50,"value":182},"Running the deployment",{"type":44,"tag":53,"props":184,"children":185},{},[186],{"type":50,"value":187},"For a text-generation LLM (vLLM):",{"type":44,"tag":189,"props":190,"children":195},"pre",{"className":191,"code":192,"language":193,"meta":194,"style":194},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","python scripts\u002Fdeploy.py \\\n    --model-name qwen3-medical \\\n    --image-uri \"$IMAGE_URI\" \\\n    --inference-ami-version \"$AMI\" \\\n    --role-arn \"$ROLE_ARN\" \\\n    --instance-type ml.g5.xlarge \\\n    --region \"$REGION\" \\\n    --env SM_VLLM_MODEL=Qwen\u002FQwen3-0.6B \\\n    --env SM_VLLM_HOST=0.0.0.0 \\\n    --env SM_VLLM_TRUST_REMOTE_CODE=true \\\n    --env SM_VLLM_MAX_MODEL_LEN=4096\n","bash","",[196],{"type":44,"tag":144,"props":197,"children":198},{"__ignoreMap":194},[199,223,241,270,296,322,340,366,384,407,429],{"type":44,"tag":200,"props":201,"children":204},"span",{"class":202,"line":203},"line",1,[205,211,217],{"type":44,"tag":200,"props":206,"children":208},{"style":207},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[209],{"type":50,"value":210},"python",{"type":44,"tag":200,"props":212,"children":214},{"style":213},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[215],{"type":50,"value":216}," scripts\u002Fdeploy.py",{"type":44,"tag":200,"props":218,"children":220},{"style":219},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[221],{"type":50,"value":222}," \\\n",{"type":44,"tag":200,"props":224,"children":226},{"class":202,"line":225},2,[227,232,237],{"type":44,"tag":200,"props":228,"children":229},{"style":213},[230],{"type":50,"value":231},"    --model-name",{"type":44,"tag":200,"props":233,"children":234},{"style":213},[235],{"type":50,"value":236}," qwen3-medical",{"type":44,"tag":200,"props":238,"children":239},{"style":219},[240],{"type":50,"value":222},{"type":44,"tag":200,"props":242,"children":244},{"class":202,"line":243},3,[245,250,256,261,266],{"type":44,"tag":200,"props":246,"children":247},{"style":213},[248],{"type":50,"value":249},"    --image-uri",{"type":44,"tag":200,"props":251,"children":253},{"style":252},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[254],{"type":50,"value":255}," \"",{"type":44,"tag":200,"props":257,"children":258},{"style":219},[259],{"type":50,"value":260},"$IMAGE_URI",{"type":44,"tag":200,"props":262,"children":263},{"style":252},[264],{"type":50,"value":265},"\"",{"type":44,"tag":200,"props":267,"children":268},{"style":219},[269],{"type":50,"value":222},{"type":44,"tag":200,"props":271,"children":273},{"class":202,"line":272},4,[274,279,283,288,292],{"type":44,"tag":200,"props":275,"children":276},{"style":213},[277],{"type":50,"value":278},"    --inference-ami-version",{"type":44,"tag":200,"props":280,"children":281},{"style":252},[282],{"type":50,"value":255},{"type":44,"tag":200,"props":284,"children":285},{"style":219},[286],{"type":50,"value":287},"$AMI",{"type":44,"tag":200,"props":289,"children":290},{"style":252},[291],{"type":50,"value":265},{"type":44,"tag":200,"props":293,"children":294},{"style":219},[295],{"type":50,"value":222},{"type":44,"tag":200,"props":297,"children":299},{"class":202,"line":298},5,[300,305,309,314,318],{"type":44,"tag":200,"props":301,"children":302},{"style":213},[303],{"type":50,"value":304},"    --role-arn",{"type":44,"tag":200,"props":306,"children":307},{"style":252},[308],{"type":50,"value":255},{"type":44,"tag":200,"props":310,"children":311},{"style":219},[312],{"type":50,"value":313},"$ROLE_ARN",{"type":44,"tag":200,"props":315,"children":316},{"style":252},[317],{"type":50,"value":265},{"type":44,"tag":200,"props":319,"children":320},{"style":219},[321],{"type":50,"value":222},{"type":44,"tag":200,"props":323,"children":325},{"class":202,"line":324},6,[326,331,336],{"type":44,"tag":200,"props":327,"children":328},{"style":213},[329],{"type":50,"value":330},"    --instance-type",{"type":44,"tag":200,"props":332,"children":333},{"style":213},[334],{"type":50,"value":335}," ml.g5.xlarge",{"type":44,"tag":200,"props":337,"children":338},{"style":219},[339],{"type":50,"value":222},{"type":44,"tag":200,"props":341,"children":343},{"class":202,"line":342},7,[344,349,353,358,362],{"type":44,"tag":200,"props":345,"children":346},{"style":213},[347],{"type":50,"value":348},"    --region",{"type":44,"tag":200,"props":350,"children":351},{"style":252},[352],{"type":50,"value":255},{"type":44,"tag":200,"props":354,"children":355},{"style":219},[356],{"type":50,"value":357},"$REGION",{"type":44,"tag":200,"props":359,"children":360},{"style":252},[361],{"type":50,"value":265},{"type":44,"tag":200,"props":363,"children":364},{"style":219},[365],{"type":50,"value":222},{"type":44,"tag":200,"props":367,"children":369},{"class":202,"line":368},8,[370,375,380],{"type":44,"tag":200,"props":371,"children":372},{"style":213},[373],{"type":50,"value":374},"    --env",{"type":44,"tag":200,"props":376,"children":377},{"style":213},[378],{"type":50,"value":379}," SM_VLLM_MODEL=Qwen\u002FQwen3-0.6B",{"type":44,"tag":200,"props":381,"children":382},{"style":219},[383],{"type":50,"value":222},{"type":44,"tag":200,"props":385,"children":387},{"class":202,"line":386},9,[388,392,397,403],{"type":44,"tag":200,"props":389,"children":390},{"style":213},[391],{"type":50,"value":374},{"type":44,"tag":200,"props":393,"children":394},{"style":213},[395],{"type":50,"value":396}," SM_VLLM_HOST=",{"type":44,"tag":200,"props":398,"children":400},{"style":399},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[401],{"type":50,"value":402},"0.0.0.0",{"type":44,"tag":200,"props":404,"children":405},{"style":219},[406],{"type":50,"value":222},{"type":44,"tag":200,"props":408,"children":410},{"class":202,"line":409},10,[411,415,420,425],{"type":44,"tag":200,"props":412,"children":413},{"style":213},[414],{"type":50,"value":374},{"type":44,"tag":200,"props":416,"children":417},{"style":213},[418],{"type":50,"value":419}," SM_VLLM_TRUST_REMOTE_CODE=",{"type":44,"tag":200,"props":421,"children":422},{"style":252},[423],{"type":50,"value":424},"true",{"type":44,"tag":200,"props":426,"children":427},{"style":219},[428],{"type":50,"value":222},{"type":44,"tag":200,"props":430,"children":432},{"class":202,"line":431},11,[433,437,442],{"type":44,"tag":200,"props":434,"children":435},{"style":213},[436],{"type":50,"value":374},{"type":44,"tag":200,"props":438,"children":439},{"style":213},[440],{"type":50,"value":441}," SM_VLLM_MAX_MODEL_LEN=",{"type":44,"tag":200,"props":443,"children":444},{"style":399},[445],{"type":50,"value":446},"4096\n",{"type":44,"tag":53,"props":448,"children":449},{},[450],{"type":50,"value":451},"For an embedding model (TEI, often on CPU):",{"type":44,"tag":189,"props":453,"children":455},{"className":191,"code":454,"language":193,"meta":194,"style":194},"python scripts\u002Fdeploy.py \\\n    --model-name bge-large-embeddings \\\n    --image-uri \"$IMAGE_URI\" \\\n    --role-arn \"$ROLE_ARN\" \\\n    --instance-type ml.c6i.2xlarge \\\n    --region \"$REGION\" \\\n    --env HF_MODEL_ID=BAAI\u002Fbge-large-en-v1.5\n",[456],{"type":44,"tag":144,"props":457,"children":458},{"__ignoreMap":194},[459,474,490,513,536,552,575],{"type":44,"tag":200,"props":460,"children":461},{"class":202,"line":203},[462,466,470],{"type":44,"tag":200,"props":463,"children":464},{"style":207},[465],{"type":50,"value":210},{"type":44,"tag":200,"props":467,"children":468},{"style":213},[469],{"type":50,"value":216},{"type":44,"tag":200,"props":471,"children":472},{"style":219},[473],{"type":50,"value":222},{"type":44,"tag":200,"props":475,"children":476},{"class":202,"line":225},[477,481,486],{"type":44,"tag":200,"props":478,"children":479},{"style":213},[480],{"type":50,"value":231},{"type":44,"tag":200,"props":482,"children":483},{"style":213},[484],{"type":50,"value":485}," bge-large-embeddings",{"type":44,"tag":200,"props":487,"children":488},{"style":219},[489],{"type":50,"value":222},{"type":44,"tag":200,"props":491,"children":492},{"class":202,"line":243},[493,497,501,505,509],{"type":44,"tag":200,"props":494,"children":495},{"style":213},[496],{"type":50,"value":249},{"type":44,"tag":200,"props":498,"children":499},{"style":252},[500],{"type":50,"value":255},{"type":44,"tag":200,"props":502,"children":503},{"style":219},[504],{"type":50,"value":260},{"type":44,"tag":200,"props":506,"children":507},{"style":252},[508],{"type":50,"value":265},{"type":44,"tag":200,"props":510,"children":511},{"style":219},[512],{"type":50,"value":222},{"type":44,"tag":200,"props":514,"children":515},{"class":202,"line":272},[516,520,524,528,532],{"type":44,"tag":200,"props":517,"children":518},{"style":213},[519],{"type":50,"value":304},{"type":44,"tag":200,"props":521,"children":522},{"style":252},[523],{"type":50,"value":255},{"type":44,"tag":200,"props":525,"children":526},{"style":219},[527],{"type":50,"value":313},{"type":44,"tag":200,"props":529,"children":530},{"style":252},[531],{"type":50,"value":265},{"type":44,"tag":200,"props":533,"children":534},{"style":219},[535],{"type":50,"value":222},{"type":44,"tag":200,"props":537,"children":538},{"class":202,"line":298},[539,543,548],{"type":44,"tag":200,"props":540,"children":541},{"style":213},[542],{"type":50,"value":330},{"type":44,"tag":200,"props":544,"children":545},{"style":213},[546],{"type":50,"value":547}," ml.c6i.2xlarge",{"type":44,"tag":200,"props":549,"children":550},{"style":219},[551],{"type":50,"value":222},{"type":44,"tag":200,"props":553,"children":554},{"class":202,"line":324},[555,559,563,567,571],{"type":44,"tag":200,"props":556,"children":557},{"style":213},[558],{"type":50,"value":348},{"type":44,"tag":200,"props":560,"children":561},{"style":252},[562],{"type":50,"value":255},{"type":44,"tag":200,"props":564,"children":565},{"style":219},[566],{"type":50,"value":357},{"type":44,"tag":200,"props":568,"children":569},{"style":252},[570],{"type":50,"value":265},{"type":44,"tag":200,"props":572,"children":573},{"style":219},[574],{"type":50,"value":222},{"type":44,"tag":200,"props":576,"children":577},{"class":202,"line":342},[578,582],{"type":44,"tag":200,"props":579,"children":580},{"style":213},[581],{"type":50,"value":374},{"type":44,"tag":200,"props":583,"children":584},{"style":213},[585],{"type":50,"value":586}," HF_MODEL_ID=BAAI\u002Fbge-large-en-v1.5\n",{"type":44,"tag":53,"props":588,"children":589},{},[590,592,597,599,605,607,613,615,621],{"type":50,"value":591},"Note: TEI deployments ",{"type":44,"tag":84,"props":593,"children":594},{},[595],{"type":50,"value":596},"do not",{"type":50,"value":598}," need ",{"type":44,"tag":144,"props":600,"children":602},{"className":601},[],[603],{"type":50,"value":604},"--inference-ami-version",{"type":50,"value":606},". That flag is vLLM-specific. TEI env vars are also simpler (",{"type":44,"tag":144,"props":608,"children":610},{"className":609},[],[611],{"type":50,"value":612},"HF_MODEL_ID",{"type":50,"value":614}," instead of ",{"type":44,"tag":144,"props":616,"children":618},{"className":617},[],[619],{"type":50,"value":620},"SM_VLLM_*",{"type":50,"value":622},", no host or trust-remote-code to configure).",{"type":44,"tag":53,"props":624,"children":625},{},[626],{"type":50,"value":627},"Where each value comes from:",{"type":44,"tag":629,"props":630,"children":631},"table",{},[632,651],{"type":44,"tag":633,"props":634,"children":635},"thead",{},[636],{"type":44,"tag":637,"props":638,"children":639},"tr",{},[640,646],{"type":44,"tag":641,"props":642,"children":643},"th",{},[644],{"type":50,"value":645},"Parameter",{"type":44,"tag":641,"props":647,"children":648},{},[649],{"type":50,"value":650},"Source",{"type":44,"tag":652,"props":653,"children":654},"tbody",{},[655,679,700,731,752,769,800],{"type":44,"tag":637,"props":656,"children":657},{},[658,668],{"type":44,"tag":659,"props":660,"children":661},"td",{},[662],{"type":44,"tag":144,"props":663,"children":665},{"className":664},[],[666],{"type":50,"value":667},"--image-uri",{"type":44,"tag":659,"props":669,"children":670},{},[671,677],{"type":44,"tag":144,"props":672,"children":674},{"className":673},[],[675],{"type":50,"value":676},"hf-cloud-serving-image-selection",{"type":50,"value":678}," — agent reads from the AWS DLC catalog page",{"type":44,"tag":637,"props":680,"children":681},{},[682,690],{"type":44,"tag":659,"props":683,"children":684},{},[685],{"type":44,"tag":144,"props":686,"children":688},{"className":687},[],[689],{"type":50,"value":604},{"type":44,"tag":659,"props":691,"children":692},{},[693,698],{"type":44,"tag":144,"props":694,"children":696},{"className":695},[],[697],{"type":50,"value":676},{"type":50,"value":699}," — required for vLLM tags containing cu130+",{"type":44,"tag":637,"props":701,"children":702},{},[703,712],{"type":44,"tag":659,"props":704,"children":705},{},[706],{"type":44,"tag":144,"props":707,"children":709},{"className":708},[],[710],{"type":50,"value":711},"--role-arn",{"type":44,"tag":659,"props":713,"children":714},{},[715,721,723,729],{"type":44,"tag":144,"props":716,"children":718},{"className":717},[],[719],{"type":50,"value":720},"hf-cloud-sagemaker-iam-preflight",{"type":50,"value":722}," (",{"type":44,"tag":144,"props":724,"children":726},{"className":725},[],[727],{"type":50,"value":728},"check_role.py",{"type":50,"value":730},")",{"type":44,"tag":637,"props":732,"children":733},{},[734,743],{"type":44,"tag":659,"props":735,"children":736},{},[737],{"type":44,"tag":144,"props":738,"children":740},{"className":739},[],[741],{"type":50,"value":742},"--region",{"type":44,"tag":659,"props":744,"children":745},{},[746],{"type":44,"tag":144,"props":747,"children":749},{"className":748},[],[750],{"type":50,"value":751},"hf-cloud-aws-context-discovery",{"type":44,"tag":637,"props":753,"children":754},{},[755,764],{"type":44,"tag":659,"props":756,"children":757},{},[758],{"type":44,"tag":144,"props":759,"children":761},{"className":760},[],[762],{"type":50,"value":763},"--instance-type",{"type":44,"tag":659,"props":765,"children":766},{},[767],{"type":50,"value":768},"User input or planner recommendation",{"type":44,"tag":637,"props":770,"children":771},{},[772,781],{"type":44,"tag":659,"props":773,"children":774},{},[775],{"type":44,"tag":144,"props":776,"children":778},{"className":777},[],[779],{"type":50,"value":780},"--env",{"type":44,"tag":659,"props":782,"children":783},{},[784,786,791,793,798],{"type":50,"value":785},"Model-specific; see ",{"type":44,"tag":144,"props":787,"children":789},{"className":788},[],[790],{"type":50,"value":676},{"type":50,"value":792}," for required ",{"type":44,"tag":144,"props":794,"children":796},{"className":795},[],[797],{"type":50,"value":620},{"type":50,"value":799}," vars",{"type":44,"tag":637,"props":801,"children":802},{},[803,812],{"type":44,"tag":659,"props":804,"children":805},{},[806],{"type":44,"tag":144,"props":807,"children":809},{"className":808},[],[810],{"type":50,"value":811},"--model-s3-uri",{"type":44,"tag":659,"props":813,"children":814},{},[815],{"type":50,"value":816},"Optional — S3 path to model artifacts; omit if loading from HF Hub",{"type":44,"tag":53,"props":818,"children":819},{},[820,822,828],{"type":50,"value":821},"The script creates resources in order with error handling, waits for ",{"type":44,"tag":144,"props":823,"children":825},{"className":824},[],[826],{"type":50,"value":827},"InService",{"type":50,"value":829}," (up to 30 min), surfaces failure reasons, registers autoscaling and alarms, and prints a summary including the teardown command. Outputs a JSON blob on stdout with endpoint\u002Fconfig\u002Fmodel names for downstream scripting.",{"type":44,"tag":53,"props":831,"children":832},{},[833,835,841],{"type":50,"value":834},"The scripts ship with this skill. If the installed copy is missing the ",{"type":44,"tag":144,"props":836,"children":838},{"className":837},[],[839],{"type":50,"value":840},"scripts\u002F",{"type":50,"value":842}," directory (some harnesses copy only SKILL.md on install), fetch them from the source repo rather than re-implementing them from this description.",{"type":44,"tag":53,"props":844,"children":845},{},[846,851,853,859,861,866],{"type":44,"tag":84,"props":847,"children":848},{},[849],{"type":50,"value":850},"Cold-start expectation",{"type":50,"value":852},": when the model loads from HF Hub, the download happens inside the container after the endpoint starts — 5–15+ minutes to InService is normal, not a failure. ",{"type":44,"tag":144,"props":854,"children":856},{"className":855},[],[857],{"type":50,"value":858},"deploy.py",{"type":50,"value":860}," waits 30 minutes; if you write custom wait code, don't time out at 15. Pre-staging weights in S3 (",{"type":44,"tag":144,"props":862,"children":864},{"className":863},[],[865],{"type":50,"value":811},{"type":50,"value":867},") cuts this and removes the Hub dependency.",{"type":44,"tag":64,"props":869,"children":871},{"id":870},"inservice-is-not-success-smoke-test-before-declaring-victory",[872],{"type":50,"value":873},"InService is not success — smoke-test before declaring victory",{"type":44,"tag":53,"props":875,"children":876},{},[877,882,884,890],{"type":44,"tag":144,"props":878,"children":880},{"className":879},[],[881],{"type":50,"value":827},{"type":50,"value":883}," only means the container answered ",{"type":44,"tag":144,"props":885,"children":887},{"className":886},[],[888],{"type":50,"value":889},"\u002Fping",{"type":50,"value":891},". In MMS-based containers (HF Inference Toolkit) the Java front-end answers pings even while the Python worker crash-loops — an endpoint can be InService and serve nothing. Two checks, always:",{"type":44,"tag":76,"props":893,"children":894},{},[895,933],{"type":44,"tag":80,"props":896,"children":897},{},[898,903],{"type":44,"tag":84,"props":899,"children":900},{},[901],{"type":50,"value":902},"One real invocation.",{"type":44,"tag":904,"props":905,"children":906},"ul",{},[907,920],{"type":44,"tag":80,"props":908,"children":909},{},[910,912,918],{"type":50,"value":911},"Real-time: ",{"type":44,"tag":144,"props":913,"children":915},{"className":914},[],[916],{"type":50,"value":917},"invoke_endpoint.py",{"type":50,"value":919}," (below) with a minimal payload; require an HTTP 200 with a sane body.",{"type":44,"tag":80,"props":921,"children":922},{},[923,925,931],{"type":50,"value":924},"Async: upload one input to S3, call ",{"type":44,"tag":144,"props":926,"children":928},{"className":927},[],[929],{"type":50,"value":930},"invoke-endpoint-async",{"type":50,"value":932},", poll the output URI for a few minutes (see \"Invoking async endpoints\"). A result object = success; an object at the failure URI, or nothing appearing, = broken.",{"type":44,"tag":80,"props":934,"children":935},{},[936,941,943],{"type":44,"tag":84,"props":937,"children":938},{},[939],{"type":50,"value":940},"Scan the endpoint logs for worker-crash markers",{"type":50,"value":942}," — catches the crash-loop case even when the smoke request merely times out:",{"type":44,"tag":189,"props":944,"children":946},{"className":191,"code":945,"language":193,"meta":194,"style":194},"aws logs filter-log-events \\\n    --log-group-name \u002Faws\u002Fsagemaker\u002FEndpoints\u002F\u003Cendpoint-name> \\\n    --filter-pattern '?\"Worker died\" ?\"Load model failed\" ?\"ImportError\"' \\\n    --region \u003Cregion> --max-items 5\n",[947],{"type":44,"tag":144,"props":948,"children":949},{"__ignoreMap":194},[950,971,1008,1035],{"type":44,"tag":200,"props":951,"children":952},{"class":202,"line":203},[953,957,962,967],{"type":44,"tag":200,"props":954,"children":955},{"style":207},[956],{"type":50,"value":26},{"type":44,"tag":200,"props":958,"children":959},{"style":213},[960],{"type":50,"value":961}," logs",{"type":44,"tag":200,"props":963,"children":964},{"style":213},[965],{"type":50,"value":966}," filter-log-events",{"type":44,"tag":200,"props":968,"children":969},{"style":219},[970],{"type":50,"value":222},{"type":44,"tag":200,"props":972,"children":973},{"class":202,"line":225},[974,979,984,989,994,999,1004],{"type":44,"tag":200,"props":975,"children":976},{"style":213},[977],{"type":50,"value":978},"    --log-group-name",{"type":44,"tag":200,"props":980,"children":981},{"style":213},[982],{"type":50,"value":983}," \u002Faws\u002Fsagemaker\u002FEndpoints\u002F",{"type":44,"tag":200,"props":985,"children":986},{"style":252},[987],{"type":50,"value":988},"\u003C",{"type":44,"tag":200,"props":990,"children":991},{"style":213},[992],{"type":50,"value":993},"endpoint-nam",{"type":44,"tag":200,"props":995,"children":996},{"style":219},[997],{"type":50,"value":998},"e",{"type":44,"tag":200,"props":1000,"children":1001},{"style":252},[1002],{"type":50,"value":1003},">",{"type":44,"tag":200,"props":1005,"children":1006},{"style":219},[1007],{"type":50,"value":222},{"type":44,"tag":200,"props":1009,"children":1010},{"class":202,"line":243},[1011,1016,1021,1026,1031],{"type":44,"tag":200,"props":1012,"children":1013},{"style":213},[1014],{"type":50,"value":1015},"    --filter-pattern",{"type":44,"tag":200,"props":1017,"children":1018},{"style":252},[1019],{"type":50,"value":1020}," '",{"type":44,"tag":200,"props":1022,"children":1023},{"style":213},[1024],{"type":50,"value":1025},"?\"Worker died\" ?\"Load model failed\" ?\"ImportError\"",{"type":44,"tag":200,"props":1027,"children":1028},{"style":252},[1029],{"type":50,"value":1030},"'",{"type":44,"tag":200,"props":1032,"children":1033},{"style":219},[1034],{"type":50,"value":222},{"type":44,"tag":200,"props":1036,"children":1037},{"class":202,"line":272},[1038,1042,1047,1052,1057,1061,1066],{"type":44,"tag":200,"props":1039,"children":1040},{"style":213},[1041],{"type":50,"value":348},{"type":44,"tag":200,"props":1043,"children":1044},{"style":252},[1045],{"type":50,"value":1046}," \u003C",{"type":44,"tag":200,"props":1048,"children":1049},{"style":213},[1050],{"type":50,"value":1051},"regio",{"type":44,"tag":200,"props":1053,"children":1054},{"style":219},[1055],{"type":50,"value":1056},"n",{"type":44,"tag":200,"props":1058,"children":1059},{"style":252},[1060],{"type":50,"value":1003},{"type":44,"tag":200,"props":1062,"children":1063},{"style":213},[1064],{"type":50,"value":1065}," --max-items",{"type":44,"tag":200,"props":1067,"children":1068},{"style":399},[1069],{"type":50,"value":1070}," 5\n",{"type":44,"tag":53,"props":1072,"children":1073},{},[1074],{"type":50,"value":1075},"Only report the deployment complete after both pass. If the log scan hits, surface the actual traceback from CloudWatch — not the InService status.",{"type":44,"tag":64,"props":1077,"children":1079},{"id":1078},"testing-a-real-time-endpoint",[1080],{"type":50,"value":1081},"Testing a real-time endpoint",{"type":44,"tag":53,"props":1083,"children":1084},{},[1085,1087,1092,1094,1099,1101,1107],{"type":50,"value":1086},"Once the endpoint is ",{"type":44,"tag":144,"props":1088,"children":1090},{"className":1089},[],[1091],{"type":50,"value":827},{"type":50,"value":1093},", test it with the bundled helper. It is cross-platform and ",{"type":44,"tag":84,"props":1095,"children":1096},{},[1097],{"type":50,"value":1098},"BOM-safe",{"type":50,"value":1100}," — use it instead of hand-writing a payload file and calling ",{"type":44,"tag":144,"props":1102,"children":1104},{"className":1103},[],[1105],{"type":50,"value":1106},"invoke-endpoint",{"type":50,"value":1108}," directly:",{"type":44,"tag":189,"props":1110,"children":1112},{"className":191,"code":1111,"language":193,"meta":194,"style":194},"# macOS \u002F Linux\npython3 scripts\u002Finvoke_endpoint.py \\\n    --endpoint-name \u003Cendpoint-name> \\\n    --payload '{\"inputs\": \"Hello\"}' \\\n    --region \"$REGION\"\n",[1113],{"type":44,"tag":144,"props":1114,"children":1115},{"__ignoreMap":194},[1116,1125,1142,1170,1195],{"type":44,"tag":200,"props":1117,"children":1118},{"class":202,"line":203},[1119],{"type":44,"tag":200,"props":1120,"children":1122},{"style":1121},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1123],{"type":50,"value":1124},"# macOS \u002F Linux\n",{"type":44,"tag":200,"props":1126,"children":1127},{"class":202,"line":225},[1128,1133,1138],{"type":44,"tag":200,"props":1129,"children":1130},{"style":207},[1131],{"type":50,"value":1132},"python3",{"type":44,"tag":200,"props":1134,"children":1135},{"style":213},[1136],{"type":50,"value":1137}," scripts\u002Finvoke_endpoint.py",{"type":44,"tag":200,"props":1139,"children":1140},{"style":219},[1141],{"type":50,"value":222},{"type":44,"tag":200,"props":1143,"children":1144},{"class":202,"line":243},[1145,1150,1154,1158,1162,1166],{"type":44,"tag":200,"props":1146,"children":1147},{"style":213},[1148],{"type":50,"value":1149},"    --endpoint-name",{"type":44,"tag":200,"props":1151,"children":1152},{"style":252},[1153],{"type":50,"value":1046},{"type":44,"tag":200,"props":1155,"children":1156},{"style":213},[1157],{"type":50,"value":993},{"type":44,"tag":200,"props":1159,"children":1160},{"style":219},[1161],{"type":50,"value":998},{"type":44,"tag":200,"props":1163,"children":1164},{"style":252},[1165],{"type":50,"value":1003},{"type":44,"tag":200,"props":1167,"children":1168},{"style":219},[1169],{"type":50,"value":222},{"type":44,"tag":200,"props":1171,"children":1172},{"class":202,"line":272},[1173,1178,1182,1187,1191],{"type":44,"tag":200,"props":1174,"children":1175},{"style":213},[1176],{"type":50,"value":1177},"    --payload",{"type":44,"tag":200,"props":1179,"children":1180},{"style":252},[1181],{"type":50,"value":1020},{"type":44,"tag":200,"props":1183,"children":1184},{"style":213},[1185],{"type":50,"value":1186},"{\"inputs\": \"Hello\"}",{"type":44,"tag":200,"props":1188,"children":1189},{"style":252},[1190],{"type":50,"value":1030},{"type":44,"tag":200,"props":1192,"children":1193},{"style":219},[1194],{"type":50,"value":222},{"type":44,"tag":200,"props":1196,"children":1197},{"class":202,"line":298},[1198,1202,1206,1210],{"type":44,"tag":200,"props":1199,"children":1200},{"style":213},[1201],{"type":50,"value":348},{"type":44,"tag":200,"props":1203,"children":1204},{"style":252},[1205],{"type":50,"value":255},{"type":44,"tag":200,"props":1207,"children":1208},{"style":219},[1209],{"type":50,"value":357},{"type":44,"tag":200,"props":1211,"children":1212},{"style":252},[1213],{"type":50,"value":1214},"\"\n",{"type":44,"tag":189,"props":1216,"children":1220},{"className":1217,"code":1218,"language":1219,"meta":194,"style":194},"language-powershell shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Windows (PowerShell)\npython scripts\\invoke_endpoint.py `\n    --endpoint-name \u003Cendpoint-name> `\n    --payload-file payload.json `\n    --region $REGION\n","powershell",[1221],{"type":44,"tag":144,"props":1222,"children":1223},{"__ignoreMap":194},[1224,1232,1240,1248,1256],{"type":44,"tag":200,"props":1225,"children":1226},{"class":202,"line":203},[1227],{"type":44,"tag":200,"props":1228,"children":1229},{},[1230],{"type":50,"value":1231},"# Windows (PowerShell)\n",{"type":44,"tag":200,"props":1233,"children":1234},{"class":202,"line":225},[1235],{"type":44,"tag":200,"props":1236,"children":1237},{},[1238],{"type":50,"value":1239},"python scripts\\invoke_endpoint.py `\n",{"type":44,"tag":200,"props":1241,"children":1242},{"class":202,"line":243},[1243],{"type":44,"tag":200,"props":1244,"children":1245},{},[1246],{"type":50,"value":1247},"    --endpoint-name \u003Cendpoint-name> `\n",{"type":44,"tag":200,"props":1249,"children":1250},{"class":202,"line":272},[1251],{"type":44,"tag":200,"props":1252,"children":1253},{},[1254],{"type":50,"value":1255},"    --payload-file payload.json `\n",{"type":44,"tag":200,"props":1257,"children":1258},{"class":202,"line":298},[1259],{"type":44,"tag":200,"props":1260,"children":1261},{},[1262],{"type":50,"value":1263},"    --region $REGION\n",{"type":44,"tag":53,"props":1265,"children":1266},{},[1267,1269,1275,1277,1283],{"type":50,"value":1268},"It accepts either ",{"type":44,"tag":144,"props":1270,"children":1272},{"className":1271},[],[1273],{"type":50,"value":1274},"--payload '\u003Cjson>'",{"type":50,"value":1276}," (inline) or ",{"type":44,"tag":144,"props":1278,"children":1280},{"className":1279},[],[1281],{"type":50,"value":1282},"--payload-file \u003Cpath>",{"type":50,"value":1284},", validates JSON, writes the request body as plain UTF-8, invokes the endpoint, and prints the response body to stdout.",{"type":44,"tag":1286,"props":1287,"children":1289},"h3",{"id":1288},"the-utf-8-bom-gotcha-windows",[1290],{"type":50,"value":1291},"The UTF-8 BOM gotcha (Windows)",{"type":44,"tag":53,"props":1293,"children":1294},{},[1295,1297,1301,1303,1309,1311,1317],{"type":50,"value":1296},"If you write the request payload yourself on Windows, ",{"type":44,"tag":84,"props":1298,"children":1299},{},[1300],{"type":50,"value":596},{"type":50,"value":1302}," use ",{"type":44,"tag":144,"props":1304,"children":1306},{"className":1305},[],[1307],{"type":50,"value":1308},"Set-Content -Encoding UTF8",{"type":50,"value":1310}," — depending on the PowerShell version it prepends a UTF-8 byte-order mark (BOM). SageMaker's JSON parser rejects a BOM with a 400 ",{"type":44,"tag":144,"props":1312,"children":1314},{"className":1313},[],[1315],{"type":50,"value":1316},"ModelError",{"type":50,"value":1318},":",{"type":44,"tag":189,"props":1320,"children":1324},{"className":1321,"code":1323,"language":50},[1322],"language-text","Unexpected UTF-8 BOM (decode using utf-8-sig): line 1 column 1 (char 0)\n",[1325],{"type":44,"tag":144,"props":1326,"children":1327},{"__ignoreMap":194},[1328],{"type":50,"value":1323},{"type":44,"tag":53,"props":1330,"children":1331},{},[1332,1334,1339,1341,1346,1348,1354],{"type":50,"value":1333},"This is ",{"type":44,"tag":84,"props":1335,"children":1336},{},[1337],{"type":50,"value":1338},"not",{"type":50,"value":1340}," a model, endpoint-health, or image problem — only the file encoding of the request body. ",{"type":44,"tag":144,"props":1342,"children":1344},{"className":1343},[],[1345],{"type":50,"value":917},{"type":50,"value":1347}," avoids it entirely (it even strips a BOM from a ",{"type":44,"tag":144,"props":1349,"children":1351},{"className":1350},[],[1352],{"type":50,"value":1353},"--payload-file",{"type":50,"value":1355}," that already has one). If you must call the CLI directly, write the body as BOM-free UTF-8:",{"type":44,"tag":189,"props":1357,"children":1359},{"className":1217,"code":1358,"language":1219,"meta":194,"style":194},"# BOM-free UTF-8 — use this\n[System.IO.File]::WriteAllText((Resolve-Path \"payload.json\"), $json, [System.Text.UTF8Encoding]::new($false))\n\naws sagemaker-runtime invoke-endpoint `\n    --endpoint-name \u003Cendpoint-name> `\n    --content-type application\u002Fjson `\n    --body fileb:\u002F\u002Fpayload.json `\n    --region $REGION `\n    response.json\n",[1360],{"type":44,"tag":144,"props":1361,"children":1362},{"__ignoreMap":194},[1363,1371,1379,1388,1396,1403,1411,1419,1427],{"type":44,"tag":200,"props":1364,"children":1365},{"class":202,"line":203},[1366],{"type":44,"tag":200,"props":1367,"children":1368},{},[1369],{"type":50,"value":1370},"# BOM-free UTF-8 — use this\n",{"type":44,"tag":200,"props":1372,"children":1373},{"class":202,"line":225},[1374],{"type":44,"tag":200,"props":1375,"children":1376},{},[1377],{"type":50,"value":1378},"[System.IO.File]::WriteAllText((Resolve-Path \"payload.json\"), $json, [System.Text.UTF8Encoding]::new($false))\n",{"type":44,"tag":200,"props":1380,"children":1381},{"class":202,"line":243},[1382],{"type":44,"tag":200,"props":1383,"children":1385},{"emptyLinePlaceholder":1384},true,[1386],{"type":50,"value":1387},"\n",{"type":44,"tag":200,"props":1389,"children":1390},{"class":202,"line":272},[1391],{"type":44,"tag":200,"props":1392,"children":1393},{},[1394],{"type":50,"value":1395},"aws sagemaker-runtime invoke-endpoint `\n",{"type":44,"tag":200,"props":1397,"children":1398},{"class":202,"line":298},[1399],{"type":44,"tag":200,"props":1400,"children":1401},{},[1402],{"type":50,"value":1247},{"type":44,"tag":200,"props":1404,"children":1405},{"class":202,"line":324},[1406],{"type":44,"tag":200,"props":1407,"children":1408},{},[1409],{"type":50,"value":1410},"    --content-type application\u002Fjson `\n",{"type":44,"tag":200,"props":1412,"children":1413},{"class":202,"line":342},[1414],{"type":44,"tag":200,"props":1415,"children":1416},{},[1417],{"type":50,"value":1418},"    --body fileb:\u002F\u002Fpayload.json `\n",{"type":44,"tag":200,"props":1420,"children":1421},{"class":202,"line":368},[1422],{"type":44,"tag":200,"props":1423,"children":1424},{},[1425],{"type":50,"value":1426},"    --region $REGION `\n",{"type":44,"tag":200,"props":1428,"children":1429},{"class":202,"line":386},[1430],{"type":44,"tag":200,"props":1431,"children":1432},{},[1433],{"type":50,"value":1434},"    response.json\n",{"type":44,"tag":53,"props":1436,"children":1437},{},[1438,1443,1445,1451,1453,1458],{"type":44,"tag":84,"props":1439,"children":1440},{},[1441],{"type":50,"value":1442},"Fallback:",{"type":50,"value":1444}," if any invocation fails with ",{"type":44,"tag":144,"props":1446,"children":1448},{"className":1447},[],[1449],{"type":50,"value":1450},"Unexpected UTF-8 BOM",{"type":50,"value":1452},", rewrite the payload as BOM-free UTF-8 (or re-run via ",{"type":44,"tag":144,"props":1454,"children":1456},{"className":1455},[],[1457],{"type":50,"value":917},{"type":50,"value":1459},") and retry once before treating the endpoint or model as broken.",{"type":44,"tag":1286,"props":1461,"children":1463},{"id":1462},"invoking-a-generative-reranker-vllm",[1464],{"type":50,"value":1465},"Invoking a generative reranker (vLLM)",{"type":44,"tag":53,"props":1467,"children":1468},{},[1469,1471,1476,1478,1489,1491,1497,1499,1505],{"type":50,"value":1470},"Generative rerankers (Qwen3-Reranker etc. — routed to the HuggingFace vLLM DLC by ",{"type":44,"tag":144,"props":1472,"children":1474},{"className":1473},[],[1475],{"type":50,"value":676},{"type":50,"value":1477},") are causal LMs scored by their first generated token, not chat models. Use the ",{"type":44,"tag":84,"props":1479,"children":1480},{},[1481,1483],{"type":50,"value":1482},"completions API with a raw ",{"type":44,"tag":144,"props":1484,"children":1486},{"className":1485},[],[1487],{"type":50,"value":1488},"prompt",{"type":50,"value":1490},", not the messages\u002Fchat API: chat templating does not reliably honor ",{"type":44,"tag":144,"props":1492,"children":1494},{"className":1493},[],[1495],{"type":50,"value":1496},"chat_template_kwargs",{"type":50,"value":1498}," such as ",{"type":44,"tag":144,"props":1500,"children":1502},{"className":1501},[],[1503],{"type":50,"value":1504},"{\"enable_thinking\": false}",{"type":50,"value":1506},", and a wrong template silently returns near-identical scores for every query–document pair instead of erroring.",{"type":44,"tag":53,"props":1508,"children":1509},{},[1510,1512,1518,1520,1526],{"type":50,"value":1511},"Payload shape (Qwen3-Reranker's expected format — substitute ",{"type":44,"tag":144,"props":1513,"children":1515},{"className":1514},[],[1516],{"type":50,"value":1517},"{query}",{"type":50,"value":1519}," \u002F ",{"type":44,"tag":144,"props":1521,"children":1523},{"className":1522},[],[1524],{"type":50,"value":1525},"{document}",{"type":50,"value":1527},"):",{"type":44,"tag":189,"props":1529,"children":1533},{"className":1530,"code":1531,"language":1532,"meta":194,"style":194},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","{\n  \"prompt\": \"\u003C|im_start|>system\\nJudge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be \\\"yes\\\" or \\\"no\\\".\u003C|im_end|>\\n\u003C|im_start|>user\\n\u003CInstruct>: Given a web search query, retrieve relevant passages that answer the query\\n\u003CQuery>: {query}\\n\u003CDocument>: {document}\u003C|im_end|>\\n\u003C|im_start|>assistant\\n\u003Cthink>\\n\\n\u003C\u002Fthink>\\n\\n\",\n  \"max_tokens\": 1,\n  \"temperature\": 0,\n  \"logprobs\": 20\n}\n","json",[1534],{"type":44,"tag":144,"props":1535,"children":1536},{"__ignoreMap":194},[1537,1545,1699,1728,1757,1782],{"type":44,"tag":200,"props":1538,"children":1539},{"class":202,"line":203},[1540],{"type":44,"tag":200,"props":1541,"children":1542},{"style":252},[1543],{"type":50,"value":1544},"{\n",{"type":44,"tag":200,"props":1546,"children":1547},{"class":202,"line":225},[1548,1553,1558,1562,1566,1570,1575,1580,1585,1590,1595,1599,1604,1608,1613,1617,1622,1626,1631,1635,1640,1644,1649,1653,1658,1662,1667,1671,1676,1681,1686,1690,1694],{"type":44,"tag":200,"props":1549,"children":1550},{"style":252},[1551],{"type":50,"value":1552},"  \"",{"type":44,"tag":200,"props":1554,"children":1556},{"style":1555},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1557],{"type":50,"value":1488},{"type":44,"tag":200,"props":1559,"children":1560},{"style":252},[1561],{"type":50,"value":265},{"type":44,"tag":200,"props":1563,"children":1564},{"style":252},[1565],{"type":50,"value":1318},{"type":44,"tag":200,"props":1567,"children":1568},{"style":252},[1569],{"type":50,"value":255},{"type":44,"tag":200,"props":1571,"children":1572},{"style":213},[1573],{"type":50,"value":1574},"\u003C|im_start|>system",{"type":44,"tag":200,"props":1576,"children":1577},{"style":219},[1578],{"type":50,"value":1579},"\\n",{"type":44,"tag":200,"props":1581,"children":1582},{"style":213},[1583],{"type":50,"value":1584},"Judge whether the Document meets the requirements based on the Query and the Instruct provided. Note that the answer can only be ",{"type":44,"tag":200,"props":1586,"children":1587},{"style":219},[1588],{"type":50,"value":1589},"\\\"",{"type":44,"tag":200,"props":1591,"children":1592},{"style":213},[1593],{"type":50,"value":1594},"yes",{"type":44,"tag":200,"props":1596,"children":1597},{"style":219},[1598],{"type":50,"value":1589},{"type":44,"tag":200,"props":1600,"children":1601},{"style":213},[1602],{"type":50,"value":1603}," or ",{"type":44,"tag":200,"props":1605,"children":1606},{"style":219},[1607],{"type":50,"value":1589},{"type":44,"tag":200,"props":1609,"children":1610},{"style":213},[1611],{"type":50,"value":1612},"no",{"type":44,"tag":200,"props":1614,"children":1615},{"style":219},[1616],{"type":50,"value":1589},{"type":44,"tag":200,"props":1618,"children":1619},{"style":213},[1620],{"type":50,"value":1621},".\u003C|im_end|>",{"type":44,"tag":200,"props":1623,"children":1624},{"style":219},[1625],{"type":50,"value":1579},{"type":44,"tag":200,"props":1627,"children":1628},{"style":213},[1629],{"type":50,"value":1630},"\u003C|im_start|>user",{"type":44,"tag":200,"props":1632,"children":1633},{"style":219},[1634],{"type":50,"value":1579},{"type":44,"tag":200,"props":1636,"children":1637},{"style":213},[1638],{"type":50,"value":1639},"\u003CInstruct>: Given a web search query, retrieve relevant passages that answer the query",{"type":44,"tag":200,"props":1641,"children":1642},{"style":219},[1643],{"type":50,"value":1579},{"type":44,"tag":200,"props":1645,"children":1646},{"style":213},[1647],{"type":50,"value":1648},"\u003CQuery>: {query}",{"type":44,"tag":200,"props":1650,"children":1651},{"style":219},[1652],{"type":50,"value":1579},{"type":44,"tag":200,"props":1654,"children":1655},{"style":213},[1656],{"type":50,"value":1657},"\u003CDocument>: {document}\u003C|im_end|>",{"type":44,"tag":200,"props":1659,"children":1660},{"style":219},[1661],{"type":50,"value":1579},{"type":44,"tag":200,"props":1663,"children":1664},{"style":213},[1665],{"type":50,"value":1666},"\u003C|im_start|>assistant",{"type":44,"tag":200,"props":1668,"children":1669},{"style":219},[1670],{"type":50,"value":1579},{"type":44,"tag":200,"props":1672,"children":1673},{"style":213},[1674],{"type":50,"value":1675},"\u003Cthink>",{"type":44,"tag":200,"props":1677,"children":1678},{"style":219},[1679],{"type":50,"value":1680},"\\n\\n",{"type":44,"tag":200,"props":1682,"children":1683},{"style":213},[1684],{"type":50,"value":1685},"\u003C\u002Fthink>",{"type":44,"tag":200,"props":1687,"children":1688},{"style":219},[1689],{"type":50,"value":1680},{"type":44,"tag":200,"props":1691,"children":1692},{"style":252},[1693],{"type":50,"value":265},{"type":44,"tag":200,"props":1695,"children":1696},{"style":252},[1697],{"type":50,"value":1698},",\n",{"type":44,"tag":200,"props":1700,"children":1701},{"class":202,"line":243},[1702,1706,1711,1715,1719,1724],{"type":44,"tag":200,"props":1703,"children":1704},{"style":252},[1705],{"type":50,"value":1552},{"type":44,"tag":200,"props":1707,"children":1708},{"style":1555},[1709],{"type":50,"value":1710},"max_tokens",{"type":44,"tag":200,"props":1712,"children":1713},{"style":252},[1714],{"type":50,"value":265},{"type":44,"tag":200,"props":1716,"children":1717},{"style":252},[1718],{"type":50,"value":1318},{"type":44,"tag":200,"props":1720,"children":1721},{"style":399},[1722],{"type":50,"value":1723}," 1",{"type":44,"tag":200,"props":1725,"children":1726},{"style":252},[1727],{"type":50,"value":1698},{"type":44,"tag":200,"props":1729,"children":1730},{"class":202,"line":272},[1731,1735,1740,1744,1748,1753],{"type":44,"tag":200,"props":1732,"children":1733},{"style":252},[1734],{"type":50,"value":1552},{"type":44,"tag":200,"props":1736,"children":1737},{"style":1555},[1738],{"type":50,"value":1739},"temperature",{"type":44,"tag":200,"props":1741,"children":1742},{"style":252},[1743],{"type":50,"value":265},{"type":44,"tag":200,"props":1745,"children":1746},{"style":252},[1747],{"type":50,"value":1318},{"type":44,"tag":200,"props":1749,"children":1750},{"style":399},[1751],{"type":50,"value":1752}," 0",{"type":44,"tag":200,"props":1754,"children":1755},{"style":252},[1756],{"type":50,"value":1698},{"type":44,"tag":200,"props":1758,"children":1759},{"class":202,"line":298},[1760,1764,1769,1773,1777],{"type":44,"tag":200,"props":1761,"children":1762},{"style":252},[1763],{"type":50,"value":1552},{"type":44,"tag":200,"props":1765,"children":1766},{"style":1555},[1767],{"type":50,"value":1768},"logprobs",{"type":44,"tag":200,"props":1770,"children":1771},{"style":252},[1772],{"type":50,"value":265},{"type":44,"tag":200,"props":1774,"children":1775},{"style":252},[1776],{"type":50,"value":1318},{"type":44,"tag":200,"props":1778,"children":1779},{"style":399},[1780],{"type":50,"value":1781}," 20\n",{"type":44,"tag":200,"props":1783,"children":1784},{"class":202,"line":324},[1785],{"type":44,"tag":200,"props":1786,"children":1787},{"style":252},[1788],{"type":50,"value":1789},"}\n",{"type":44,"tag":53,"props":1791,"children":1792},{},[1793,1795,1801,1803,1809],{"type":50,"value":1794},"The trailing ",{"type":44,"tag":144,"props":1796,"children":1798},{"className":1797},[],[1799],{"type":50,"value":1800},"\u003C|im_start|>assistant\\n\u003Cthink>\\n\\n\u003C\u002Fthink>\\n\\n",{"type":50,"value":1802}," suffix is load-bearing: it pre-fills an empty thinking block so the first generated token is the yes\u002Fno judgment. Score from the returned logprobs: ",{"type":44,"tag":144,"props":1804,"children":1806},{"className":1805},[],[1807],{"type":50,"value":1808},"P(\"yes\") \u002F (P(\"yes\") + P(\"no\"))",{"type":50,"value":1810},". Sanity check the endpoint with one relevant pair (expect >0.9) and one irrelevant pair (expect \u003C0.05) — near-identical scores across pairs mean the prompt template is wrong, not that the model is broken.",{"type":44,"tag":53,"props":1812,"children":1813},{},[1814],{"type":50,"value":1815},"The same rule generalizes: for any thinking-mode model where the prompt must be byte-exact, prefer the raw completions API over chat.",{"type":44,"tag":1286,"props":1817,"children":1819},{"id":1818},"picking-the-image-uri",[1820],{"type":50,"value":1821},"Picking the image URI",{"type":44,"tag":53,"props":1823,"children":1824},{},[1825,1827,1836,1838,1844,1846,1852],{"type":50,"value":1826},"The agent reads the image URI from AWS's ",{"type":44,"tag":1828,"props":1829,"children":1833},"a",{"href":1830,"rel":1831},"https:\u002F\u002Faws.github.io\u002Fdeep-learning-containers\u002Freference\u002Favailable_images\u002F",[1832],"nofollow",[1834],{"type":50,"value":1835},"Deep Learning Containers catalog",{"type":50,"value":1837}," — pick the row that matches the model family (HuggingFace vLLM for LLMs, TEI for embeddings, etc.), substitute ",{"type":44,"tag":144,"props":1839,"children":1841},{"className":1840},[],[1842],{"type":50,"value":1843},"\u003Cregion>",{"type":50,"value":1845}," with the deployment region, and pass to ",{"type":44,"tag":144,"props":1847,"children":1849},{"className":1848},[],[1850],{"type":50,"value":1851},"deploy.py --image-uri",{"type":50,"value":151},{"type":44,"tag":53,"props":1854,"children":1855},{},[1856,1858,1864,1866,1872],{"type":50,"value":1857},"For vLLM images specifically (both ",{"type":44,"tag":144,"props":1859,"children":1861},{"className":1860},[],[1862],{"type":50,"value":1863},"huggingface-vllm",{"type":50,"value":1865}," and the AWS ",{"type":44,"tag":144,"props":1867,"children":1869},{"className":1868},[],[1870],{"type":50,"value":1871},"vllm",{"type":50,"value":1873}," fallback), also check the tag's CUDA version:",{"type":44,"tag":189,"props":1875,"children":1877},{"className":191,"code":1876,"language":193,"meta":194,"style":194},"# Example: HuggingFace vLLM 0.21.0 from the catalog\nIMAGE_URI=\"763104351884.dkr.ecr.eu-west-1.amazonaws.com\u002Fhuggingface-vllm:0.21.0-transformers5.8.1-gpu-py312-cu130-ubuntu22.04\"\n\n# cu130 tag → must pass --inference-ami-version\npython deploy.py --image-uri \"$IMAGE_URI\" \\\n    --inference-ami-version al2-ami-sagemaker-inference-gpu-3-1 \\\n    ...\n",[1878],{"type":44,"tag":144,"props":1879,"children":1880},{"__ignoreMap":194},[1881,1889,1915,1922,1930,1963,1979],{"type":44,"tag":200,"props":1882,"children":1883},{"class":202,"line":203},[1884],{"type":44,"tag":200,"props":1885,"children":1886},{"style":1121},[1887],{"type":50,"value":1888},"# Example: HuggingFace vLLM 0.21.0 from the catalog\n",{"type":44,"tag":200,"props":1890,"children":1891},{"class":202,"line":225},[1892,1897,1902,1906,1911],{"type":44,"tag":200,"props":1893,"children":1894},{"style":219},[1895],{"type":50,"value":1896},"IMAGE_URI",{"type":44,"tag":200,"props":1898,"children":1899},{"style":252},[1900],{"type":50,"value":1901},"=",{"type":44,"tag":200,"props":1903,"children":1904},{"style":252},[1905],{"type":50,"value":265},{"type":44,"tag":200,"props":1907,"children":1908},{"style":213},[1909],{"type":50,"value":1910},"763104351884.dkr.ecr.eu-west-1.amazonaws.com\u002Fhuggingface-vllm:0.21.0-transformers5.8.1-gpu-py312-cu130-ubuntu22.04",{"type":44,"tag":200,"props":1912,"children":1913},{"style":252},[1914],{"type":50,"value":1214},{"type":44,"tag":200,"props":1916,"children":1917},{"class":202,"line":243},[1918],{"type":44,"tag":200,"props":1919,"children":1920},{"emptyLinePlaceholder":1384},[1921],{"type":50,"value":1387},{"type":44,"tag":200,"props":1923,"children":1924},{"class":202,"line":272},[1925],{"type":44,"tag":200,"props":1926,"children":1927},{"style":1121},[1928],{"type":50,"value":1929},"# cu130 tag → must pass --inference-ami-version\n",{"type":44,"tag":200,"props":1931,"children":1932},{"class":202,"line":298},[1933,1937,1942,1947,1951,1955,1959],{"type":44,"tag":200,"props":1934,"children":1935},{"style":207},[1936],{"type":50,"value":210},{"type":44,"tag":200,"props":1938,"children":1939},{"style":213},[1940],{"type":50,"value":1941}," deploy.py",{"type":44,"tag":200,"props":1943,"children":1944},{"style":213},[1945],{"type":50,"value":1946}," --image-uri",{"type":44,"tag":200,"props":1948,"children":1949},{"style":252},[1950],{"type":50,"value":255},{"type":44,"tag":200,"props":1952,"children":1953},{"style":219},[1954],{"type":50,"value":260},{"type":44,"tag":200,"props":1956,"children":1957},{"style":252},[1958],{"type":50,"value":265},{"type":44,"tag":200,"props":1960,"children":1961},{"style":219},[1962],{"type":50,"value":222},{"type":44,"tag":200,"props":1964,"children":1965},{"class":202,"line":324},[1966,1970,1975],{"type":44,"tag":200,"props":1967,"children":1968},{"style":213},[1969],{"type":50,"value":278},{"type":44,"tag":200,"props":1971,"children":1972},{"style":213},[1973],{"type":50,"value":1974}," al2-ami-sagemaker-inference-gpu-3-1",{"type":44,"tag":200,"props":1976,"children":1977},{"style":219},[1978],{"type":50,"value":222},{"type":44,"tag":200,"props":1980,"children":1981},{"class":202,"line":342},[1982],{"type":44,"tag":200,"props":1983,"children":1984},{"style":213},[1985],{"type":50,"value":1986},"    ...\n",{"type":44,"tag":53,"props":1988,"children":1989},{},[1990,1992,1998,2000,2005,2007,2012],{"type":50,"value":1991},"For tags with ",{"type":44,"tag":144,"props":1993,"children":1995},{"className":1994},[],[1996],{"type":50,"value":1997},"cu129",{"type":50,"value":1999}," or lower, omit ",{"type":44,"tag":144,"props":2001,"children":2003},{"className":2002},[],[2004],{"type":50,"value":604},{"type":50,"value":2006},". See ",{"type":44,"tag":144,"props":2008,"children":2010},{"className":2009},[],[2011],{"type":50,"value":676},{"type":50,"value":2013}," for the full vLLM AMI lookup table and the env-var requirements for each image family.",{"type":44,"tag":64,"props":2015,"children":2017},{"id":2016},"async-inference-deployments",[2018],{"type":50,"value":2019},"Async inference deployments",{"type":44,"tag":53,"props":2021,"children":2022},{},[2023,2025,2031,2032,2037,2039,2045],{"type":50,"value":2024},"For long-running inferences (>60s), large payloads, or workloads that are bursty\u002Fsparse enough to benefit from scale-to-zero, use ",{"type":44,"tag":144,"props":2026,"children":2028},{"className":2027},[],[2029],{"type":50,"value":2030},"deploy_async.py",{"type":50,"value":614},{"type":44,"tag":144,"props":2033,"children":2035},{"className":2034},[],[2036],{"type":50,"value":858},{"type":50,"value":2038},". Async genuinely supports ",{"type":44,"tag":144,"props":2040,"children":2042},{"className":2041},[],[2043],{"type":50,"value":2044},"MinCapacity=0",{"type":50,"value":2046}," — real-time autoscaling can't.",{"type":44,"tag":189,"props":2048,"children":2050},{"className":191,"code":2049,"language":193,"meta":194,"style":194},"python scripts\u002Fdeploy_async.py \\\n    --model-name flux-text-to-image \\\n    --image-uri \"$IMAGE_URI\" \\\n    --role-arn \"$ROLE_ARN\" \\\n    --instance-type ml.g5.2xlarge \\\n    --region \"$REGION\" \\\n    --output-s3-uri s3:\u002F\u002Fmy-bucket\u002Fasync-output\u002F \\\n    --env HF_MODEL_ID=black-forest-labs\u002FFLUX.1-dev\n",[2051],{"type":44,"tag":144,"props":2052,"children":2053},{"__ignoreMap":194},[2054,2070,2086,2109,2132,2148,2171,2188],{"type":44,"tag":200,"props":2055,"children":2056},{"class":202,"line":203},[2057,2061,2066],{"type":44,"tag":200,"props":2058,"children":2059},{"style":207},[2060],{"type":50,"value":210},{"type":44,"tag":200,"props":2062,"children":2063},{"style":213},[2064],{"type":50,"value":2065}," scripts\u002Fdeploy_async.py",{"type":44,"tag":200,"props":2067,"children":2068},{"style":219},[2069],{"type":50,"value":222},{"type":44,"tag":200,"props":2071,"children":2072},{"class":202,"line":225},[2073,2077,2082],{"type":44,"tag":200,"props":2074,"children":2075},{"style":213},[2076],{"type":50,"value":231},{"type":44,"tag":200,"props":2078,"children":2079},{"style":213},[2080],{"type":50,"value":2081}," flux-text-to-image",{"type":44,"tag":200,"props":2083,"children":2084},{"style":219},[2085],{"type":50,"value":222},{"type":44,"tag":200,"props":2087,"children":2088},{"class":202,"line":243},[2089,2093,2097,2101,2105],{"type":44,"tag":200,"props":2090,"children":2091},{"style":213},[2092],{"type":50,"value":249},{"type":44,"tag":200,"props":2094,"children":2095},{"style":252},[2096],{"type":50,"value":255},{"type":44,"tag":200,"props":2098,"children":2099},{"style":219},[2100],{"type":50,"value":260},{"type":44,"tag":200,"props":2102,"children":2103},{"style":252},[2104],{"type":50,"value":265},{"type":44,"tag":200,"props":2106,"children":2107},{"style":219},[2108],{"type":50,"value":222},{"type":44,"tag":200,"props":2110,"children":2111},{"class":202,"line":272},[2112,2116,2120,2124,2128],{"type":44,"tag":200,"props":2113,"children":2114},{"style":213},[2115],{"type":50,"value":304},{"type":44,"tag":200,"props":2117,"children":2118},{"style":252},[2119],{"type":50,"value":255},{"type":44,"tag":200,"props":2121,"children":2122},{"style":219},[2123],{"type":50,"value":313},{"type":44,"tag":200,"props":2125,"children":2126},{"style":252},[2127],{"type":50,"value":265},{"type":44,"tag":200,"props":2129,"children":2130},{"style":219},[2131],{"type":50,"value":222},{"type":44,"tag":200,"props":2133,"children":2134},{"class":202,"line":298},[2135,2139,2144],{"type":44,"tag":200,"props":2136,"children":2137},{"style":213},[2138],{"type":50,"value":330},{"type":44,"tag":200,"props":2140,"children":2141},{"style":213},[2142],{"type":50,"value":2143}," ml.g5.2xlarge",{"type":44,"tag":200,"props":2145,"children":2146},{"style":219},[2147],{"type":50,"value":222},{"type":44,"tag":200,"props":2149,"children":2150},{"class":202,"line":324},[2151,2155,2159,2163,2167],{"type":44,"tag":200,"props":2152,"children":2153},{"style":213},[2154],{"type":50,"value":348},{"type":44,"tag":200,"props":2156,"children":2157},{"style":252},[2158],{"type":50,"value":255},{"type":44,"tag":200,"props":2160,"children":2161},{"style":219},[2162],{"type":50,"value":357},{"type":44,"tag":200,"props":2164,"children":2165},{"style":252},[2166],{"type":50,"value":265},{"type":44,"tag":200,"props":2168,"children":2169},{"style":219},[2170],{"type":50,"value":222},{"type":44,"tag":200,"props":2172,"children":2173},{"class":202,"line":342},[2174,2179,2184],{"type":44,"tag":200,"props":2175,"children":2176},{"style":213},[2177],{"type":50,"value":2178},"    --output-s3-uri",{"type":44,"tag":200,"props":2180,"children":2181},{"style":213},[2182],{"type":50,"value":2183}," s3:\u002F\u002Fmy-bucket\u002Fasync-output\u002F",{"type":44,"tag":200,"props":2185,"children":2186},{"style":219},[2187],{"type":50,"value":222},{"type":44,"tag":200,"props":2189,"children":2190},{"class":202,"line":368},[2191,2195],{"type":44,"tag":200,"props":2192,"children":2193},{"style":213},[2194],{"type":50,"value":374},{"type":44,"tag":200,"props":2196,"children":2197},{"style":213},[2198],{"type":50,"value":2199}," HF_MODEL_ID=black-forest-labs\u002FFLUX.1-dev\n",{"type":44,"tag":53,"props":2201,"children":2202},{},[2203,2205,2210],{"type":50,"value":2204},"Required extras over ",{"type":44,"tag":144,"props":2206,"children":2208},{"className":2207},[],[2209],{"type":50,"value":858},{"type":50,"value":1318},{"type":44,"tag":904,"props":2212,"children":2213},{},[2214],{"type":44,"tag":80,"props":2215,"children":2216},{},[2217,2223],{"type":44,"tag":144,"props":2218,"children":2220},{"className":2219},[],[2221],{"type":50,"value":2222},"--output-s3-uri",{"type":50,"value":2224}," — where async results land (results are not returned synchronously)",{"type":44,"tag":53,"props":2226,"children":2227},{},[2228],{"type":50,"value":2229},"Optional async-specific flags:",{"type":44,"tag":904,"props":2231,"children":2232},{},[2233,2244,2263,2274,2285],{"type":44,"tag":80,"props":2234,"children":2235},{},[2236,2242],{"type":44,"tag":144,"props":2237,"children":2239},{"className":2238},[],[2240],{"type":50,"value":2241},"--failure-s3-uri",{"type":50,"value":2243}," — separate path for failed invocations",{"type":44,"tag":80,"props":2245,"children":2246},{},[2247,2253,2255,2261],{"type":44,"tag":144,"props":2248,"children":2250},{"className":2249},[],[2251],{"type":50,"value":2252},"--success-sns-topic",{"type":50,"value":2254},", ",{"type":44,"tag":144,"props":2256,"children":2258},{"className":2257},[],[2259],{"type":50,"value":2260},"--error-sns-topic",{"type":50,"value":2262}," — get notified when async results are ready or fail",{"type":44,"tag":80,"props":2264,"children":2265},{},[2266,2272],{"type":44,"tag":144,"props":2267,"children":2269},{"className":2268},[],[2270],{"type":50,"value":2271},"--min-capacity 0",{"type":50,"value":2273}," (the default) — scale to zero between batches",{"type":44,"tag":80,"props":2275,"children":2276},{},[2277,2283],{"type":44,"tag":144,"props":2278,"children":2280},{"className":2279},[],[2281],{"type":50,"value":2282},"--backlog-per-instance-target N",{"type":50,"value":2284}," — target queue depth per instance (default 5)",{"type":44,"tag":80,"props":2286,"children":2287},{},[2288,2294],{"type":44,"tag":144,"props":2289,"children":2291},{"className":2290},[],[2292],{"type":50,"value":2293},"--max-concurrent-invocations-per-instance N",{"type":50,"value":2295}," — default 4",{"type":44,"tag":1286,"props":2297,"children":2299},{"id":2298},"how-scale-to-zero-works",[2300],{"type":50,"value":2301},"How scale-to-zero works",{"type":44,"tag":53,"props":2303,"children":2304},{},[2305,2307,2312],{"type":50,"value":2306},"The async script registers ",{"type":44,"tag":84,"props":2308,"children":2309},{},[2310],{"type":50,"value":2311},"two",{"type":50,"value":2313}," autoscaling policies on the variant:",{"type":44,"tag":76,"props":2315,"children":2316},{},[2317,2335],{"type":44,"tag":80,"props":2318,"children":2319},{},[2320,2325,2327,2333],{"type":44,"tag":84,"props":2321,"children":2322},{},[2323],{"type":50,"value":2324},"Target-tracking",{"type":50,"value":2326}," on ",{"type":44,"tag":144,"props":2328,"children":2330},{"className":2329},[],[2331],{"type":50,"value":2332},"ApproximateBacklogSizePerInstance",{"type":50,"value":2334}," — handles ongoing scaling between min and max",{"type":44,"tag":80,"props":2336,"children":2337},{},[2338,2343,2345,2351,2353,2359],{"type":44,"tag":84,"props":2339,"children":2340},{},[2341],{"type":50,"value":2342},"Step-scaling",{"type":50,"value":2344}," triggered by a ",{"type":44,"tag":144,"props":2346,"children":2348},{"className":2347},[],[2349],{"type":50,"value":2350},"HasBacklogWithoutCapacity",{"type":50,"value":2352}," CloudWatch alarm — handles ",{"type":44,"tag":144,"props":2354,"children":2356},{"className":2355},[],[2357],{"type":50,"value":2358},"0→1",{"type":50,"value":2360}," wake-from-zero",{"type":44,"tag":53,"props":2362,"children":2363},{},[2364],{"type":50,"value":2365},"Both are needed. Target-tracking alone cannot transition from zero (it can't divide by zero instances), so without the step policy the endpoint comes up, scales to zero after the first batch, and never wakes again. The script wires this up automatically.",{"type":44,"tag":1286,"props":2367,"children":2369},{"id":2368},"async-alarms",[2370],{"type":50,"value":2371},"Async alarms",{"type":44,"tag":53,"props":2373,"children":2374},{},[2375],{"type":50,"value":2376},"The script creates three CloudWatch alarms:",{"type":44,"tag":904,"props":2378,"children":2379},{},[2380,2391,2402],{"type":44,"tag":80,"props":2381,"children":2382},{},[2383,2389],{"type":44,"tag":144,"props":2384,"children":2386},{"className":2385},[],[2387],{"type":50,"value":2388},"ApproximateBacklogSize > 50",{"type":50,"value":2390}," — queue is building faster than capacity can drain it",{"type":44,"tag":80,"props":2392,"children":2393},{},[2394,2400],{"type":44,"tag":144,"props":2395,"children":2397},{"className":2396},[],[2398],{"type":50,"value":2399},"InvocationsFailed > 5",{"type":50,"value":2401}," — repeated processing failures",{"type":44,"tag":80,"props":2403,"children":2404},{},[2405,2410],{"type":44,"tag":144,"props":2406,"children":2408},{"className":2407},[],[2409],{"type":50,"value":2350},{"type":50,"value":2411}," — drives the wake-from-zero policy (not a notification alarm; its action is the step-scaling policy, not the SNS topic)",{"type":44,"tag":53,"props":2413,"children":2414},{},[2415,2417,2423],{"type":50,"value":2416},"If you pass ",{"type":44,"tag":144,"props":2418,"children":2420},{"className":2419},[],[2421],{"type":50,"value":2422},"--sns-alarm-topic \u003Carn>",{"type":50,"value":2424},", the first two notify on that topic. The wake alarm always points at the step policy.",{"type":44,"tag":1286,"props":2426,"children":2428},{"id":2427},"invoking-async-endpoints",[2429],{"type":50,"value":2430},"Invoking async endpoints",{"type":44,"tag":53,"props":2432,"children":2433},{},[2434,2436,2441,2443,2448],{"type":50,"value":2435},"Async endpoints aren't called synchronously. You upload the input to S3, call ",{"type":44,"tag":144,"props":2437,"children":2439},{"className":2438},[],[2440],{"type":50,"value":930},{"type":50,"value":2442}," with the S3 input location, and SageMaker writes the result to your ",{"type":44,"tag":144,"props":2444,"children":2446},{"className":2445},[],[2447],{"type":50,"value":2222},{"type":50,"value":2449}," when done:",{"type":44,"tag":189,"props":2451,"children":2453},{"className":191,"code":2452,"language":193,"meta":194,"style":194},"# Upload your input first\naws s3 cp input.json s3:\u002F\u002Fmy-input-bucket\u002Fjob1\u002Finput.json\n\n# Invoke\naws sagemaker-runtime invoke-endpoint-async \\\n    --endpoint-name \u003Cendpoint-name> \\\n    --input-location s3:\u002F\u002Fmy-input-bucket\u002Fjob1\u002Finput.json \\\n    --content-type application\u002Fjson \\\n    --region \u003Cregion>\n\n# Poll for the result at your output URI\naws s3 cp s3:\u002F\u002Fmy-bucket\u002Fasync-output\u002F\u003Cinference-id>.out result.json\n",[2454],{"type":44,"tag":144,"props":2455,"children":2456},{"__ignoreMap":194},[2457,2465,2492,2499,2507,2528,2555,2572,2589,2613,2620,2628],{"type":44,"tag":200,"props":2458,"children":2459},{"class":202,"line":203},[2460],{"type":44,"tag":200,"props":2461,"children":2462},{"style":1121},[2463],{"type":50,"value":2464},"# Upload your input first\n",{"type":44,"tag":200,"props":2466,"children":2467},{"class":202,"line":225},[2468,2472,2477,2482,2487],{"type":44,"tag":200,"props":2469,"children":2470},{"style":207},[2471],{"type":50,"value":26},{"type":44,"tag":200,"props":2473,"children":2474},{"style":213},[2475],{"type":50,"value":2476}," s3",{"type":44,"tag":200,"props":2478,"children":2479},{"style":213},[2480],{"type":50,"value":2481}," cp",{"type":44,"tag":200,"props":2483,"children":2484},{"style":213},[2485],{"type":50,"value":2486}," input.json",{"type":44,"tag":200,"props":2488,"children":2489},{"style":213},[2490],{"type":50,"value":2491}," s3:\u002F\u002Fmy-input-bucket\u002Fjob1\u002Finput.json\n",{"type":44,"tag":200,"props":2493,"children":2494},{"class":202,"line":243},[2495],{"type":44,"tag":200,"props":2496,"children":2497},{"emptyLinePlaceholder":1384},[2498],{"type":50,"value":1387},{"type":44,"tag":200,"props":2500,"children":2501},{"class":202,"line":272},[2502],{"type":44,"tag":200,"props":2503,"children":2504},{"style":1121},[2505],{"type":50,"value":2506},"# Invoke\n",{"type":44,"tag":200,"props":2508,"children":2509},{"class":202,"line":298},[2510,2514,2519,2524],{"type":44,"tag":200,"props":2511,"children":2512},{"style":207},[2513],{"type":50,"value":26},{"type":44,"tag":200,"props":2515,"children":2516},{"style":213},[2517],{"type":50,"value":2518}," sagemaker-runtime",{"type":44,"tag":200,"props":2520,"children":2521},{"style":213},[2522],{"type":50,"value":2523}," invoke-endpoint-async",{"type":44,"tag":200,"props":2525,"children":2526},{"style":219},[2527],{"type":50,"value":222},{"type":44,"tag":200,"props":2529,"children":2530},{"class":202,"line":324},[2531,2535,2539,2543,2547,2551],{"type":44,"tag":200,"props":2532,"children":2533},{"style":213},[2534],{"type":50,"value":1149},{"type":44,"tag":200,"props":2536,"children":2537},{"style":252},[2538],{"type":50,"value":1046},{"type":44,"tag":200,"props":2540,"children":2541},{"style":213},[2542],{"type":50,"value":993},{"type":44,"tag":200,"props":2544,"children":2545},{"style":219},[2546],{"type":50,"value":998},{"type":44,"tag":200,"props":2548,"children":2549},{"style":252},[2550],{"type":50,"value":1003},{"type":44,"tag":200,"props":2552,"children":2553},{"style":219},[2554],{"type":50,"value":222},{"type":44,"tag":200,"props":2556,"children":2557},{"class":202,"line":342},[2558,2563,2568],{"type":44,"tag":200,"props":2559,"children":2560},{"style":213},[2561],{"type":50,"value":2562},"    --input-location",{"type":44,"tag":200,"props":2564,"children":2565},{"style":213},[2566],{"type":50,"value":2567}," s3:\u002F\u002Fmy-input-bucket\u002Fjob1\u002Finput.json",{"type":44,"tag":200,"props":2569,"children":2570},{"style":219},[2571],{"type":50,"value":222},{"type":44,"tag":200,"props":2573,"children":2574},{"class":202,"line":368},[2575,2580,2585],{"type":44,"tag":200,"props":2576,"children":2577},{"style":213},[2578],{"type":50,"value":2579},"    --content-type",{"type":44,"tag":200,"props":2581,"children":2582},{"style":213},[2583],{"type":50,"value":2584}," application\u002Fjson",{"type":44,"tag":200,"props":2586,"children":2587},{"style":219},[2588],{"type":50,"value":222},{"type":44,"tag":200,"props":2590,"children":2591},{"class":202,"line":386},[2592,2596,2600,2604,2608],{"type":44,"tag":200,"props":2593,"children":2594},{"style":213},[2595],{"type":50,"value":348},{"type":44,"tag":200,"props":2597,"children":2598},{"style":252},[2599],{"type":50,"value":1046},{"type":44,"tag":200,"props":2601,"children":2602},{"style":213},[2603],{"type":50,"value":1051},{"type":44,"tag":200,"props":2605,"children":2606},{"style":219},[2607],{"type":50,"value":1056},{"type":44,"tag":200,"props":2609,"children":2610},{"style":252},[2611],{"type":50,"value":2612},">\n",{"type":44,"tag":200,"props":2614,"children":2615},{"class":202,"line":409},[2616],{"type":44,"tag":200,"props":2617,"children":2618},{"emptyLinePlaceholder":1384},[2619],{"type":50,"value":1387},{"type":44,"tag":200,"props":2621,"children":2622},{"class":202,"line":431},[2623],{"type":44,"tag":200,"props":2624,"children":2625},{"style":1121},[2626],{"type":50,"value":2627},"# Poll for the result at your output URI\n",{"type":44,"tag":200,"props":2629,"children":2631},{"class":202,"line":2630},12,[2632,2636,2640,2644,2648,2652,2657,2662,2666,2671],{"type":44,"tag":200,"props":2633,"children":2634},{"style":207},[2635],{"type":50,"value":26},{"type":44,"tag":200,"props":2637,"children":2638},{"style":213},[2639],{"type":50,"value":2476},{"type":44,"tag":200,"props":2641,"children":2642},{"style":213},[2643],{"type":50,"value":2481},{"type":44,"tag":200,"props":2645,"children":2646},{"style":213},[2647],{"type":50,"value":2183},{"type":44,"tag":200,"props":2649,"children":2650},{"style":252},[2651],{"type":50,"value":988},{"type":44,"tag":200,"props":2653,"children":2654},{"style":213},[2655],{"type":50,"value":2656},"inference-i",{"type":44,"tag":200,"props":2658,"children":2659},{"style":219},[2660],{"type":50,"value":2661},"d",{"type":44,"tag":200,"props":2663,"children":2664},{"style":252},[2665],{"type":50,"value":1003},{"type":44,"tag":200,"props":2667,"children":2668},{"style":213},[2669],{"type":50,"value":2670},".out",{"type":44,"tag":200,"props":2672,"children":2673},{"style":213},[2674],{"type":50,"value":2675}," result.json\n",{"type":44,"tag":53,"props":2677,"children":2678},{},[2679,2681,2687],{"type":50,"value":2680},"The same UTF-8 BOM caveat applies to the ",{"type":44,"tag":144,"props":2682,"children":2684},{"className":2683},[],[2685],{"type":50,"value":2686},"input.json",{"type":50,"value":2688}," you upload (see \"The UTF-8 BOM gotcha\" above) — if you build it on Windows, write it as BOM-free UTF-8 or the container's JSON parser will reject it.",{"type":44,"tag":53,"props":2690,"children":2691},{},[2692,2694,2700],{"type":50,"value":2693},"Teardown works the same as real-time: ",{"type":44,"tag":144,"props":2695,"children":2697},{"className":2696},[],[2698],{"type":50,"value":2699},"python3 scripts\u002Fteardown.py \u003Cendpoint-name>",{"type":50,"value":2701}," (the teardown script discovers policies and alarms by name prefix, so it handles both deployment modes).",{"type":44,"tag":64,"props":2703,"children":2705},{"id":2704},"defaults-at-a-glance",[2706],{"type":50,"value":2707},"Defaults at a glance",{"type":44,"tag":629,"props":2709,"children":2710},{},[2711,2732],{"type":44,"tag":633,"props":2712,"children":2713},{},[2714],{"type":44,"tag":637,"props":2715,"children":2716},{},[2717,2722,2727],{"type":44,"tag":641,"props":2718,"children":2719},{},[2720],{"type":50,"value":2721},"Setting",{"type":44,"tag":641,"props":2723,"children":2724},{},[2725],{"type":50,"value":2726},"Default",{"type":44,"tag":641,"props":2728,"children":2729},{},[2730],{"type":50,"value":2731},"Override",{"type":44,"tag":652,"props":2733,"children":2734},{},[2735,2757,2786,2808,2829,2850,2871,2897],{"type":44,"tag":637,"props":2736,"children":2737},{},[2738,2743,2748],{"type":44,"tag":659,"props":2739,"children":2740},{},[2741],{"type":50,"value":2742},"Initial instance count",{"type":44,"tag":659,"props":2744,"children":2745},{},[2746],{"type":50,"value":2747},"1",{"type":44,"tag":659,"props":2749,"children":2750},{},[2751],{"type":44,"tag":144,"props":2752,"children":2754},{"className":2753},[],[2755],{"type":50,"value":2756},"--initial-instance-count",{"type":44,"tag":637,"props":2758,"children":2759},{},[2760,2765,2770],{"type":44,"tag":659,"props":2761,"children":2762},{},[2763],{"type":50,"value":2764},"Autoscaling min \u002F max",{"type":44,"tag":659,"props":2766,"children":2767},{},[2768],{"type":50,"value":2769},"1 \u002F 4",{"type":44,"tag":659,"props":2771,"children":2772},{},[2773,2779,2780],{"type":44,"tag":144,"props":2774,"children":2776},{"className":2775},[],[2777],{"type":50,"value":2778},"--min-capacity",{"type":50,"value":2254},{"type":44,"tag":144,"props":2781,"children":2783},{"className":2782},[],[2784],{"type":50,"value":2785},"--max-capacity",{"type":44,"tag":637,"props":2787,"children":2788},{},[2789,2794,2799],{"type":44,"tag":659,"props":2790,"children":2791},{},[2792],{"type":50,"value":2793},"Autoscaling target",{"type":44,"tag":659,"props":2795,"children":2796},{},[2797],{"type":50,"value":2798},"20 invocations\u002Fmin\u002Finstance",{"type":44,"tag":659,"props":2800,"children":2801},{},[2802],{"type":44,"tag":144,"props":2803,"children":2805},{"className":2804},[],[2806],{"type":50,"value":2807},"--target-invocations-per-instance",{"type":44,"tag":637,"props":2809,"children":2810},{},[2811,2816,2821],{"type":44,"tag":659,"props":2812,"children":2813},{},[2814],{"type":50,"value":2815},"Data capture",{"type":44,"tag":659,"props":2817,"children":2818},{},[2819],{"type":50,"value":2820},"disabled (opt-in)",{"type":44,"tag":659,"props":2822,"children":2823},{},[2824],{"type":44,"tag":144,"props":2825,"children":2827},{"className":2826},[],[2828],{"type":50,"value":149},{"type":44,"tag":637,"props":2830,"children":2831},{},[2832,2836,2841],{"type":44,"tag":659,"props":2833,"children":2834},{},[2835],{"type":50,"value":128},{"type":44,"tag":659,"props":2837,"children":2838},{},[2839],{"type":50,"value":2840},"3 alarms",{"type":44,"tag":659,"props":2842,"children":2843},{},[2844],{"type":44,"tag":144,"props":2845,"children":2847},{"className":2846},[],[2848],{"type":50,"value":2849},"--no-alarms",{"type":44,"tag":637,"props":2851,"children":2852},{},[2853,2858,2863],{"type":44,"tag":659,"props":2854,"children":2855},{},[2856],{"type":50,"value":2857},"SNS notification",{"type":44,"tag":659,"props":2859,"children":2860},{},[2861],{"type":50,"value":2862},"none (alarms created but won't notify)",{"type":44,"tag":659,"props":2864,"children":2865},{},[2866],{"type":44,"tag":144,"props":2867,"children":2869},{"className":2868},[],[2870],{"type":50,"value":2422},{"type":44,"tag":637,"props":2872,"children":2873},{},[2874,2879,2888],{"type":44,"tag":659,"props":2875,"children":2876},{},[2877],{"type":50,"value":2878},"Environment tag",{"type":44,"tag":659,"props":2880,"children":2881},{},[2882],{"type":44,"tag":144,"props":2883,"children":2885},{"className":2884},[],[2886],{"type":50,"value":2887},"dev",{"type":44,"tag":659,"props":2889,"children":2890},{},[2891],{"type":44,"tag":144,"props":2892,"children":2894},{"className":2893},[],[2895],{"type":50,"value":2896},"--environment",{"type":44,"tag":637,"props":2898,"children":2899},{},[2900,2905,2910],{"type":44,"tag":659,"props":2901,"children":2902},{},[2903],{"type":50,"value":2904},"InferenceAmiVersion",{"type":44,"tag":659,"props":2906,"children":2907},{},[2908],{"type":50,"value":2909},"none (SageMaker default)",{"type":44,"tag":659,"props":2911,"children":2912},{},[2913,2918],{"type":44,"tag":144,"props":2914,"children":2916},{"className":2915},[],[2917],{"type":50,"value":604},{"type":50,"value":2919}," (REQUIRED for vLLM CUDA 13+)",{"type":44,"tag":53,"props":2921,"children":2922},{},[2923],{"type":50,"value":2924},"Not defaulted (user-specific input needed): VPC config, KMS key, multi-variant, async inference.",{"type":44,"tag":1286,"props":2926,"children":2928},{"id":2927},"autoscaling-target-tune-by-model-type",[2929],{"type":50,"value":2930},"Autoscaling target — tune by model type",{"type":44,"tag":53,"props":2932,"children":2933},{},[2934,2936,2942,2944,2949],{"type":50,"value":2935},"The default ",{"type":44,"tag":144,"props":2937,"children":2939},{"className":2938},[],[2940],{"type":50,"value":2941},"--target-invocations-per-instance 20",{"type":50,"value":2943}," is conservative and tuned for LLM workloads where each request takes 1–5 seconds. For embedding deployments (TEI), each request is much faster (typically \u003C100ms on CPU, \u003C20ms on GPU), so a single instance can handle far more throughput. ",{"type":44,"tag":84,"props":2945,"children":2946},{},[2947],{"type":50,"value":2948},"For embedding deployments, raise the target to 100–500",{"type":50,"value":2950}," depending on instance and model size. The default of 20 will trigger autoscaling far too aggressively for embeddings and waste money.",{"type":44,"tag":53,"props":2952,"children":2953},{},[2954],{"type":50,"value":2955},"A rule of thumb: target value ≈ 60 \u002F (typical request latency in seconds). LLM at 3s latency → target 20. Embedding at 100ms → target 600. Generative rerankers sit in between — they generate a single token per request, so ~40–100 is a reasonable target.",{"type":44,"tag":64,"props":2957,"children":2959},{"id":2958},"data-capture-iam-gotcha",[2960],{"type":50,"value":2961},"Data capture + IAM gotcha",{"type":44,"tag":53,"props":2963,"children":2964},{},[2965,2967,2973,2975,2980],{"type":50,"value":2966},"If the user enables data capture, the execution role needs S3 write access to the capture prefix. The default URI (",{"type":44,"tag":144,"props":2968,"children":2970},{"className":2969},[],[2971],{"type":50,"value":2972},"s3:\u002F\u002Fsagemaker-\u003Cregion>-\u003Caccount>\u002F\u003Cendpoint>\u002Fdata-capture\u002F",{"type":50,"value":2974},") is typically a different bucket than the model artifact bucket. If ",{"type":44,"tag":144,"props":2976,"children":2978},{"className":2977},[],[2979],{"type":50,"value":720},{"type":50,"value":2981}," scoped the inline policy narrowly to just the model bucket, capture writes fail silently — endpoint keeps serving but no data appears.",{"type":44,"tag":53,"props":2983,"children":2984},{},[2985,2987,2993],{"type":50,"value":2986},"If the user reports \"data capture isn't showing up\", check the role's S3 access. Either widen the inline policy or pass ",{"type":44,"tag":144,"props":2988,"children":2990},{"className":2989},[],[2991],{"type":50,"value":2992},"--data-capture-s3-uri",{"type":50,"value":2994}," pointing to a bucket the role can write.",{"type":44,"tag":64,"props":2996,"children":2998},{"id":2997},"teardown",[2999],{"type":50,"value":3000},"Teardown",{"type":44,"tag":189,"props":3002,"children":3004},{"className":191,"code":3003,"language":193,"meta":194,"style":194},"python3 scripts\u002Fteardown.py \u003Cendpoint-name> \u003Cregion>   # macOS \u002F Linux\npython  scripts\\teardown.py \u003Cendpoint-name> \u003Cregion>   # Windows\n",[3005],{"type":44,"tag":144,"props":3006,"children":3007},{"__ignoreMap":194},[3008,3057],{"type":44,"tag":200,"props":3009,"children":3010},{"class":202,"line":203},[3011,3015,3020,3024,3028,3032,3036,3040,3044,3048,3052],{"type":44,"tag":200,"props":3012,"children":3013},{"style":207},[3014],{"type":50,"value":1132},{"type":44,"tag":200,"props":3016,"children":3017},{"style":213},[3018],{"type":50,"value":3019}," scripts\u002Fteardown.py",{"type":44,"tag":200,"props":3021,"children":3022},{"style":252},[3023],{"type":50,"value":1046},{"type":44,"tag":200,"props":3025,"children":3026},{"style":213},[3027],{"type":50,"value":993},{"type":44,"tag":200,"props":3029,"children":3030},{"style":219},[3031],{"type":50,"value":998},{"type":44,"tag":200,"props":3033,"children":3034},{"style":252},[3035],{"type":50,"value":1003},{"type":44,"tag":200,"props":3037,"children":3038},{"style":252},[3039],{"type":50,"value":1046},{"type":44,"tag":200,"props":3041,"children":3042},{"style":213},[3043],{"type":50,"value":1051},{"type":44,"tag":200,"props":3045,"children":3046},{"style":219},[3047],{"type":50,"value":1056},{"type":44,"tag":200,"props":3049,"children":3050},{"style":252},[3051],{"type":50,"value":1003},{"type":44,"tag":200,"props":3053,"children":3054},{"style":1121},[3055],{"type":50,"value":3056},"   # macOS \u002F Linux\n",{"type":44,"tag":200,"props":3058,"children":3059},{"class":202,"line":225},[3060,3064,3069,3074,3079,3083,3087,3091,3095,3099,3103,3107,3111],{"type":44,"tag":200,"props":3061,"children":3062},{"style":207},[3063],{"type":50,"value":210},{"type":44,"tag":200,"props":3065,"children":3066},{"style":213},[3067],{"type":50,"value":3068},"  scripts",{"type":44,"tag":200,"props":3070,"children":3071},{"style":219},[3072],{"type":50,"value":3073},"\\t",{"type":44,"tag":200,"props":3075,"children":3076},{"style":213},[3077],{"type":50,"value":3078},"eardown.py",{"type":44,"tag":200,"props":3080,"children":3081},{"style":252},[3082],{"type":50,"value":1046},{"type":44,"tag":200,"props":3084,"children":3085},{"style":213},[3086],{"type":50,"value":993},{"type":44,"tag":200,"props":3088,"children":3089},{"style":219},[3090],{"type":50,"value":998},{"type":44,"tag":200,"props":3092,"children":3093},{"style":252},[3094],{"type":50,"value":1003},{"type":44,"tag":200,"props":3096,"children":3097},{"style":252},[3098],{"type":50,"value":1046},{"type":44,"tag":200,"props":3100,"children":3101},{"style":213},[3102],{"type":50,"value":1051},{"type":44,"tag":200,"props":3104,"children":3105},{"style":219},[3106],{"type":50,"value":1056},{"type":44,"tag":200,"props":3108,"children":3109},{"style":252},[3110],{"type":50,"value":1003},{"type":44,"tag":200,"props":3112,"children":3113},{"style":1121},[3114],{"type":50,"value":3115},"   # Windows\n",{"type":44,"tag":53,"props":3117,"children":3118},{},[3119],{"type":50,"value":3120},"Deletes in safe order: alarms → autoscaling → endpoint (stops billing) → endpoint config → model. Idempotent.",{"type":44,"tag":53,"props":3122,"children":3123},{},[3124,3126,3130],{"type":50,"value":3125},"Does ",{"type":44,"tag":84,"props":3127,"children":3128},{},[3129],{"type":50,"value":1338},{"type":50,"value":3131}," delete: the IAM execution role (might be shared), data capture S3 objects (user might want to keep), SNS topic, original model artifacts.",{"type":44,"tag":53,"props":3133,"children":3134},{},[3135],{"type":50,"value":3136},"Always tell the user about the teardown command after the deployment summary. Users forget; endpoints accrue cost.",{"type":44,"tag":64,"props":3138,"children":3140},{"id":3139},"when-the-deployment-fails",[3141],{"type":50,"value":3142},"When the deployment fails",{"type":44,"tag":53,"props":3144,"children":3145},{},[3146,3157,3159,3165,3167,3173,3175,3180],{"type":44,"tag":84,"props":3147,"children":3148},{},[3149,3155],{"type":44,"tag":144,"props":3150,"children":3152},{"className":3151},[],[3153],{"type":50,"value":3154},"CannotStartContainerError",{"type":50,"value":3156}," + no CloudWatch logs ever created",{"type":50,"value":3158}," — the InferenceAmiVersion problem. If the image tag contains ",{"type":44,"tag":144,"props":3160,"children":3162},{"className":3161},[],[3163],{"type":50,"value":3164},"cu130",{"type":50,"value":3166}," or later and you didn't pass ",{"type":44,"tag":144,"props":3168,"children":3170},{"className":3169},[],[3171],{"type":50,"value":3172},"--inference-ami-version al2-ami-sagemaker-inference-gpu-3-1",{"type":50,"value":3174},", this is the cause. See ",{"type":44,"tag":144,"props":3176,"children":3178},{"className":3177},[],[3179],{"type":50,"value":676},{"type":50,"value":3181},". Do NOT chase images, IAM roles, env vars, or instance types — the failure signature is identical for many other things but the cause here is the AMI.",{"type":44,"tag":53,"props":3183,"children":3184},{},[3185,3190,3192,3198,3200,3205,3207,3213],{"type":44,"tag":84,"props":3186,"children":3187},{},[3188],{"type":50,"value":3189},"\"Failed to pass ping health check\"",{"type":50,"value":3191}," — the container ",{"type":44,"tag":3193,"props":3194,"children":3195},"em",{},[3196],{"type":50,"value":3197},"did",{"type":50,"value":3199}," start and produced logs, but ",{"type":44,"tag":144,"props":3201,"children":3203},{"className":3202},[],[3204],{"type":50,"value":889},{"type":50,"value":3206}," isn't responding. Check CloudWatch at ",{"type":44,"tag":144,"props":3208,"children":3210},{"className":3209},[],[3211],{"type":50,"value":3212},"\u002Faws\u002Fsagemaker\u002FEndpoints\u002F\u003Cendpoint-name>",{"type":50,"value":3214},". Usually: wrong image for model architecture, missing HF token, or OOM.",{"type":44,"tag":53,"props":3216,"children":3217},{},[3218,3223,3225,3231,3232,3238,3239,3245,3247,3253],{"type":44,"tag":84,"props":3219,"children":3220},{},[3221],{"type":50,"value":3222},"\"Container failed to start\" (with logs present)",{"type":50,"value":3224}," — entrypoint ran, then exited. Check CloudWatch. Common: missing required env vars (",{"type":44,"tag":144,"props":3226,"children":3228},{"className":3227},[],[3229],{"type":50,"value":3230},"SM_VLLM_MODEL",{"type":50,"value":2254},{"type":44,"tag":144,"props":3233,"children":3235},{"className":3234},[],[3236],{"type":50,"value":3237},"SM_VLLM_HOST",{"type":50,"value":2254},{"type":44,"tag":144,"props":3240,"children":3242},{"className":3241},[],[3243],{"type":50,"value":3244},"SM_VLLM_TRUST_REMOTE_CODE",{"type":50,"value":3246},"), wrong ",{"type":44,"tag":144,"props":3248,"children":3250},{"className":3249},[],[3251],{"type":50,"value":3252},"ModelDataUrl",{"type":50,"value":3254}," format, unreadable model artifacts.",{"type":44,"tag":53,"props":3256,"children":3257},{},[3258,3267,3269,3275],{"type":44,"tag":84,"props":3259,"children":3260},{},[3261],{"type":44,"tag":144,"props":3262,"children":3264},{"className":3263},[],[3265],{"type":50,"value":3266},"ResourceLimitExceeded",{"type":50,"value":3268}," — no quota for the instance type in this region. Request increase or pick a different type (the planner should have checked quotas up front — see ",{"type":44,"tag":144,"props":3270,"children":3272},{"className":3271},[],[3273],{"type":50,"value":3274},"hf-cloud-sagemaker-deployment-planner",{"type":50,"value":3276},").",{"type":44,"tag":53,"props":3278,"children":3279},{},[3280,3291,3293,3299,3301,3306],{"type":44,"tag":84,"props":3281,"children":3282},{},[3283,3289],{"type":44,"tag":144,"props":3284,"children":3286},{"className":3285},[],[3287],{"type":50,"value":3288},"ImportError: libtorch_cuda.so: undefined symbol: ncclCommResume",{"type":50,"value":3290}," in CloudWatch logs",{"type":50,"value":3292}," — known packaging defect in ",{"type":44,"tag":144,"props":3294,"children":3296},{"className":3295},[],[3297],{"type":50,"value":3298},"huggingface-pytorch-inference",{"type":50,"value":3300}," GPU images (see \"Known-broken images\" in ",{"type":44,"tag":144,"props":3302,"children":3304},{"className":3303},[],[3305],{"type":50,"value":676},{"type":50,"value":3307},"). Inside the container, so no env var, AMI, instance type, or sibling tag fixes it. Switch to DJL Inference.",{"type":44,"tag":53,"props":3309,"children":3310},{},[3311,3316],{"type":44,"tag":84,"props":3312,"children":3313},{},[3314],{"type":50,"value":3315},"InService, but invocations time out \u002F async outputs never appear",{"type":50,"value":3317}," — dead Python worker behind a live MMS front-end. Run the log scan from \"InService is not success\" above; the traceback in CloudWatch is the real error.",{"type":44,"tag":53,"props":3319,"children":3320},{},[3321,3332,3334,3340,3342,3348,3350,3355],{"type":44,"tag":84,"props":3322,"children":3323},{},[3324,3330],{"type":44,"tag":144,"props":3325,"children":3327},{"className":3326},[],[3328],{"type":50,"value":3329},"403 Forbidden",{"type":50,"value":3331}," downloading weights from HF Hub during startup",{"type":50,"value":3333}," — the container's bundled ",{"type":44,"tag":144,"props":3335,"children":3337},{"className":3336},[],[3338],{"type":50,"value":3339},"huggingface_hub",{"type":50,"value":3341}," predates HF's XET CDN auth. Add ",{"type":44,"tag":144,"props":3343,"children":3345},{"className":3344},[],[3346],{"type":50,"value":3347},"--env HF_HUB_ENABLE_HF_TRANSFER=0",{"type":50,"value":3349},", or pre-stage the weights in S3. Note: this can ",{"type":44,"tag":3193,"props":3351,"children":3352},{},[3353],{"type":50,"value":3354},"mask",{"type":50,"value":3356}," a deeper failure (the worker may still crash after the download succeeds) — re-check logs after fixing it.",{"type":44,"tag":53,"props":3358,"children":3359},{},[3360,3365,3367,3372],{"type":44,"tag":84,"props":3361,"children":3362},{},[3363],{"type":50,"value":3364},"Diagnostic rule",{"type":50,"value":3366},": when failures look identical across multiple configurations (different images, roles, instance types) and ",{"type":44,"tag":84,"props":3368,"children":3369},{},[3370],{"type":50,"value":3371},"no logs are ever produced",{"type":50,"value":3373},", the cause is almost always below the container — host AMI, networking, account-level — not the deployment config. Stop iterating on config; check the AMI version and account state.",{"type":44,"tag":53,"props":3375,"children":3376},{},[3377,3379,3385,3387,3393],{"type":50,"value":3378},"Don't retry blindly. The script prints the specific ",{"type":44,"tag":144,"props":3380,"children":3382},{"className":3381},[],[3383],{"type":50,"value":3384},"FailureReason",{"type":50,"value":3386}," from ",{"type":44,"tag":144,"props":3388,"children":3390},{"className":3389},[],[3391],{"type":50,"value":3392},"describe-endpoint",{"type":50,"value":3394}," — fix the root cause before retrying.",{"type":44,"tag":3396,"props":3397,"children":3398},"style",{},[3399],{"type":50,"value":3400},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3402,"total":3560},[3403,3424,3438,3453,3464,3477,3487,3501,3509,3518,3531,3545],{"slug":3404,"name":3404,"fn":3405,"description":3406,"org":3407,"tags":3408,"stars":3421,"repoUrl":3422,"updatedAt":3423},"train-sentence-transformers","train sentence-transformers models","Train or fine-tune sentence-transformers models across `SentenceTransformer` (bi-encoder; dense or static embedding model; for retrieval, similarity, clustering, classification, paraphrase mining, dedup, multimodal), `CrossEncoder` (reranker; pair scoring for two-stage retrieval \u002F pair classification), and `SparseEncoder` (SPLADE, sparse embedding model; for learned-sparse retrieval). Covers loss selection, hard-negative mining, evaluators, distillation, LoRA, Matryoshka, and Hugging Face Hub publishing. Use for any sentence-transformers training task.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3409,3412,3413,3416,3418],{"name":3410,"slug":3411,"type":14},"Deep Learning","deep-learning",{"name":9,"slug":13,"type":14},{"name":3414,"slug":3415,"type":14},"LLM","llm",{"name":3417,"slug":210,"type":14},"Python",{"name":3419,"slug":3420,"type":14},"Search","search",18914,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Fsentence-transformers","2026-05-08T05:09:16.820066",{"slug":3425,"name":3425,"fn":3426,"description":3427,"org":3428,"tags":3429,"stars":3435,"repoUrl":3436,"updatedAt":3437},"trl-training","train and fine-tune LLMs with TRL","Train and fine-tune transformer language models using TRL (Transformers Reinforcement Learning). Supports SFT, DPO, GRPO, KTO, RLOO and Reward Model training via CLI commands.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3430,3431,3432,3433,3434],{"name":22,"slug":23,"type":14},{"name":3410,"slug":3411,"type":14},{"name":9,"slug":13,"type":14},{"name":3414,"slug":3415,"type":14},{"name":3417,"slug":210,"type":14},18850,"https:\u002F\u002Fgithub.com\u002Fhuggingface\u002Ftrl","2026-04-06T18:25:32.746828",{"slug":3439,"name":3439,"fn":3440,"description":3441,"org":3442,"tags":3443,"stars":27,"repoUrl":28,"updatedAt":3452},"hf-cli","manage Hugging Face Hub resources via CLI","Hugging Face Hub CLI (`hf`) for downloading, uploading, and managing models, datasets, spaces, buckets, repos, papers, jobs, and more on the Hugging Face Hub. Use when: handling authentication; managing local cache; managing Hugging Face Buckets; running or scheduling jobs on Hugging Face infrastructure; managing Hugging Face repos; discussions and pull requests; browsing models, datasets and spaces; reading, searching, or browsing academic papers; managing collections; querying datasets; configuring spaces; setting up webhooks; or deploying and managing HF Inference Endpoints. Make sure to use this skill whenever the user mentions 'hf', 'huggingface', 'Hugging Face', 'huggingface-cli', or 'hugging face cli', or wants to do anything related to the Hugging Face ecosystem and to AI and ML in general. Also use for cloud storage needs like training checkpoints, data pipelines, or agent traces. Use even if the user doesn't explicitly ask for a CLI command. Replaces the deprecated `huggingface-cli`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3444,3447,3450,3451],{"name":3445,"slug":3446,"type":14},"CLI","cli",{"name":3448,"slug":3449,"type":14},"Datasets","datasets",{"name":9,"slug":13,"type":14},{"name":3414,"slug":3415,"type":14},"2026-04-06T18:25:34.020855",{"slug":751,"name":751,"fn":3454,"description":3455,"org":3456,"tags":3457,"stars":27,"repoUrl":28,"updatedAt":3463},"discover local AWS environment context","Discover the user's local AWS context (active profile, region, account ID, caller identity) at the start of any AWS task. Use this skill before any other AWS work — deploying to SageMaker, creating resources, calling AWS APIs, or anything that touches an AWS account. Use it especially when the user has not specified a region or profile explicitly, when they say things like \"use my AWS account\", \"deploy to AWS\", \"use my profile\", or when about to make any AWS CLI or SDK call. Never guess the region or account ID — always use this skill to read it from the local configuration first.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3458,3459,3460],{"name":25,"slug":26,"type":14},{"name":3445,"slug":3446,"type":14},{"name":3461,"slug":3462,"type":14},"Configuration","configuration","2026-07-08T05:55:33.716099",{"slug":3465,"name":3465,"fn":3466,"description":3467,"org":3468,"tags":3469,"stars":27,"repoUrl":28,"updatedAt":3476},"hf-cloud-python-env-setup","set up Python environments for AWS","Set up an isolated Python environment for SageMaker \u002F AWS work, with the right Python version and current boto3. Use this skill whenever Python code will be executed for a SageMaker deployment, training job, or any AWS automation — including when about to run `pip install`, when about to invoke `boto3`, when creating or activating a virtualenv, or when the user asks to \"set up the environment\". Never use system Python and never `pip install` into it. Always isolate. This skill prevents the most common failure modes: wrong Python version, dependency conflicts, and stale SDKs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3470,3471,3472,3475],{"name":22,"slug":23,"type":14},{"name":25,"slug":26,"type":14},{"name":3473,"slug":3474,"type":14},"Engineering","engineering",{"name":3417,"slug":210,"type":14},"2026-07-08T05:55:32.505017",{"slug":3274,"name":3274,"fn":3478,"description":3479,"org":3480,"tags":3481,"stars":27,"repoUrl":28,"updatedAt":3486},"plan model deployments to Amazon SageMaker","Plan and coordinate the deployment of a model to Amazon SageMaker AI. Use this skill whenever the user wants to deploy, host, serve, or expose a model on SageMaker or AWS — including phrases like \"deploy a model\", \"host this LLM on AWS\", \"serve this embedding model\", \"deploy a reranker\", \"deploy a text-to-image \u002F diffusion model\", \"host this for async inference\", \"create an endpoint\", \"serve my fine-tuned model\", or any request that involves making a model available for inference on AWS. Use this even when the user is vague (e.g. \"I just want to get this running on AWS, you figure it out\"). Works for text-generation LLMs, embedding models, rerankers, classifiers, text-to-image \u002F diffusion models — picks the right serving stack and chooses between real-time and async inference. This is the entry-point skill for SageMaker deployment work — it asks clarifying questions, picks a deployment pathway, and coordinates the other deployment skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3482,3483,3484,3485],{"name":22,"slug":23,"type":14},{"name":25,"slug":26,"type":14},{"name":19,"slug":20,"type":14},{"name":9,"slug":13,"type":14},"2026-07-08T05:55:37.387689",{"slug":720,"name":720,"fn":3488,"description":3489,"org":3490,"tags":3491,"stars":27,"repoUrl":28,"updatedAt":3500},"configure SageMaker IAM roles","Ensure a usable SageMaker execution role exists before deploying or training. Use this skill whenever about to create a SageMaker endpoint, model, training job, or any resource that requires an execution role. Use it especially when the user has not provided a role ARN explicitly, when scripts are about to call `iam:CreateRole`, or when an AccessDenied error mentions an IAM action. Never blindly call `iam:CreateRole` — always check for existing roles first. This skill prevents the most common SageMaker deployment failure: trying to create IAM resources from an SSO principal that has no IAM write permissions.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3492,3493,3494,3497],{"name":22,"slug":23,"type":14},{"name":25,"slug":26,"type":14},{"name":3495,"slug":3496,"type":14},"Permissions","permissions",{"name":3498,"slug":3499,"type":14},"Security","security","2026-07-08T05:55:34.948771",{"slug":4,"name":4,"fn":5,"description":6,"org":3502,"tags":3503,"stars":27,"repoUrl":28,"updatedAt":29},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3504,3505,3506,3507,3508],{"name":22,"slug":23,"type":14},{"name":25,"slug":26,"type":14},{"name":19,"slug":20,"type":14},{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"slug":676,"name":676,"fn":3510,"description":3511,"org":3512,"tags":3513,"stars":27,"repoUrl":28,"updatedAt":3517},"select SageMaker serving containers","Pick the right serving container for a SageMaker model deployment and find its current image URI. Use this skill whenever about to deploy a model to a SageMaker endpoint and an image URI needs to be chosen — including when the user says \"deploy this LLM\", \"host this HuggingFace model\", \"serve this fine-tuned model\", \"deploy this embedding model\", \"host a reranker\", \"serve a sentence-transformers model\", or when about to hardcode any container URI in deployment code. HuggingFace-curated Deep Learning Containers are ALWAYS preferred: HuggingFace vLLM (LLMs and generative rerankers), HuggingFace vLLM-Omni (multimodal), TEI (embeddings\u002Fcross-encoder rerankers), HF Inference Toolkit (other transformers). Generic images (AWS vLLM, DJL-LMI, SGLang) are used only when no HuggingFace image is compatible — never merely because they carry a newer version. Never hardcode a container URI from memory and never default to TGI. Prevents stale-image failures and wrong-region URIs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3514,3515,3516],{"name":22,"slug":23,"type":14},{"name":25,"slug":26,"type":14},{"name":19,"slug":20,"type":14},"2026-07-08T05:55:36.173465",{"slug":3519,"name":3519,"fn":3520,"description":3521,"org":3522,"tags":3523,"stars":27,"repoUrl":28,"updatedAt":3530},"hf-mcp","access Hugging Face Hub via MCP","Use Hugging Face Hub via MCP server tools. Search models, datasets, Spaces, papers. Get repo details, fetch documentation, run compute jobs, and use Gradio Spaces as AI tools. Available when connected to the HF MCP server.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3524,3525,3526,3527],{"name":3448,"slug":3449,"type":14},{"name":9,"slug":13,"type":14},{"name":3414,"slug":3415,"type":14},{"name":3528,"slug":3529,"type":14},"MCP","mcp","2026-04-06T18:25:50.364185",{"slug":3532,"name":3532,"fn":3533,"description":3534,"org":3535,"tags":3536,"stars":27,"repoUrl":28,"updatedAt":3544},"hf-mem","estimate memory for Hugging Face models","Hugging Face CLI to estimate the required memory to load Safetensors or GGUF model weights for inference from the Hugging Face Hub",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3537,3538,3539,3540,3541],{"name":22,"slug":23,"type":14},{"name":3445,"slug":3446,"type":14},{"name":9,"slug":13,"type":14},{"name":3414,"slug":3415,"type":14},{"name":3542,"slug":3543,"type":14},"Performance","performance","2026-06-13T07:23:57.101435",{"slug":3546,"name":3546,"fn":3547,"description":3548,"org":3549,"tags":3550,"stars":27,"repoUrl":28,"updatedAt":3559},"huggingface-best","find and compare Hugging Face models","Use when the user asks about finding the best, top, or recommended model for a task, wants to know what AI model to use, or wants to compare models by benchmark scores. Triggers on: \"best model for X\", \"what model should I use for\", \"top models for [task]\", \"which model runs on my laptop\u002Fmachine\u002Fdevice\", \"recommend a model for\", \"what LLM should I use for\", \"compare models for\", \"what's state of the art for\", or any question about choosing an AI model for a specific use case. Always use this skill when the user wants model recommendations or comparisons, even if they don't explicitly mention HuggingFace or benchmarks.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3551,3554,3555,3556],{"name":3552,"slug":3553,"type":14},"Analytics","analytics",{"name":9,"slug":13,"type":14},{"name":3414,"slug":3415,"type":14},{"name":3557,"slug":3558,"type":14},"Research","research","2026-04-24T05:09:45.870658",37,{"items":3562,"total":3611},[3563,3570,3576,3583,3590,3597,3605],{"slug":3439,"name":3439,"fn":3440,"description":3441,"org":3564,"tags":3565,"stars":27,"repoUrl":28,"updatedAt":3452},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3566,3567,3568,3569],{"name":3445,"slug":3446,"type":14},{"name":3448,"slug":3449,"type":14},{"name":9,"slug":13,"type":14},{"name":3414,"slug":3415,"type":14},{"slug":751,"name":751,"fn":3454,"description":3455,"org":3571,"tags":3572,"stars":27,"repoUrl":28,"updatedAt":3463},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3573,3574,3575],{"name":25,"slug":26,"type":14},{"name":3445,"slug":3446,"type":14},{"name":3461,"slug":3462,"type":14},{"slug":3465,"name":3465,"fn":3466,"description":3467,"org":3577,"tags":3578,"stars":27,"repoUrl":28,"updatedAt":3476},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3579,3580,3581,3582],{"name":22,"slug":23,"type":14},{"name":25,"slug":26,"type":14},{"name":3473,"slug":3474,"type":14},{"name":3417,"slug":210,"type":14},{"slug":3274,"name":3274,"fn":3478,"description":3479,"org":3584,"tags":3585,"stars":27,"repoUrl":28,"updatedAt":3486},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3586,3587,3588,3589],{"name":22,"slug":23,"type":14},{"name":25,"slug":26,"type":14},{"name":19,"slug":20,"type":14},{"name":9,"slug":13,"type":14},{"slug":720,"name":720,"fn":3488,"description":3489,"org":3591,"tags":3592,"stars":27,"repoUrl":28,"updatedAt":3500},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3593,3594,3595,3596],{"name":22,"slug":23,"type":14},{"name":25,"slug":26,"type":14},{"name":3495,"slug":3496,"type":14},{"name":3498,"slug":3499,"type":14},{"slug":4,"name":4,"fn":5,"description":6,"org":3598,"tags":3599,"stars":27,"repoUrl":28,"updatedAt":29},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3600,3601,3602,3603,3604],{"name":22,"slug":23,"type":14},{"name":25,"slug":26,"type":14},{"name":19,"slug":20,"type":14},{"name":9,"slug":13,"type":14},{"name":16,"slug":17,"type":14},{"slug":676,"name":676,"fn":3510,"description":3511,"org":3606,"tags":3607,"stars":27,"repoUrl":28,"updatedAt":3517},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3608,3609,3610],{"name":22,"slug":23,"type":14},{"name":25,"slug":26,"type":14},{"name":19,"slug":20,"type":14},24]