[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-deployment":3,"mdc-mvcvfh-key":33,"related-repo-nvidia-deployment":2764,"related-org-nvidia-deployment":2803},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":22,"repoUrl":23,"updatedAt":24,"license":25,"forks":26,"topics":27,"repo":28,"sourceUrl":31,"mdContent":32},"deployment","deploy LLM checkpoints as API endpoints","Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says \"deploy model\", \"serve model\", \"start vLLM server\", \"launch SGLang\", \"TRT-LLM deploy\", \"AutoDeploy\", \"benchmark throughput\", \"serve checkpoint\", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use ptq) or evaluating accuracy (use evaluation).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,21],{"name":13,"slug":14,"type":15},"LLM","llm","tag",{"name":17,"slug":18,"type":15},"vLLM","vllm",{"name":20,"slug":4,"type":15},"Deployment",{"name":9,"slug":8,"type":15},3227,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer","2026-08-31T09:18:55.144088","Apache-2.0",498,[],{"repoUrl":23,"stars":22,"forks":26,"topics":29,"description":30},[],"A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Ftree\u002FHEAD\u002Fplugins\u002Fmodelopt\u002Fskills\u002Fdeployment","---\nname: deployment\ndescription: Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says \"deploy model\", \"serve model\", \"start vLLM server\", \"launch SGLang\", \"TRT-LLM deploy\", \"AutoDeploy\", \"benchmark throughput\", \"serve checkpoint\", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use ptq) or evaluating accuracy (use evaluation).\nlicense: Apache-2.0\n---\n\n# Deployment Skill\n\nServe a model checkpoint as an OpenAI-compatible inference endpoint. Supports vLLM, SGLang, and TRT-LLM (including AutoDeploy).\n\n## Quick Start\n\nPrefer `$SKILL_DIR\u002Fscripts\u002Fdeploy.sh` for standard local deployments — it handles quant detection, health checks, and server lifecycle. Use the raw framework commands in Step 4 when you need flags the script doesn't support, or for remote deployment.\n\n```bash\n# Start vLLM server with a ModelOpt checkpoint\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" start --model .\u002Fqwen3-0.6b-fp8\n\n# Start with SGLang and tensor parallelism\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" start --model .\u002Fllama-70b-nvfp4 --framework sglang --tp 4\n\n# Start from HuggingFace hub\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" start --model nvidia\u002FLlama-3.1-8B-Instruct-FP8\n\n# Test the API\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" test\n\n# Check status\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" status\n\n# Stop\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" stop\n```\n\nThe script handles: GPU detection, quantization flag auto-detection (FP8 vs FP4), server lifecycle (start\u002Fstop\u002Frestart\u002Fstatus), health check polling, and API testing.\n\n## Decision Flow\n\n### 0. Check workspace (multi-user \u002F Slack bot)\n\nIf `MODELOPT_WORKSPACE_ROOT` is set, use the common skill's `workspace-management.md`. Before creating a new workspace, check the current session for existing model workspaces — especially if deploying a checkpoint from a prior PTQ run:\n\n```bash\nls \"$MODELOPT_WORKSPACE_ROOT\u002F\u003Csession_id>\u002F\" 2>\u002Fdev\u002Fnull\n```\n\nIf the user says \"deploy the model I just quantized\" or references a previous PTQ, find the matching workspace and `cd` into it. The checkpoint should be in that workspace's output directory.\n\n### 1. Identify the checkpoint\n\nDetermine what the user wants to deploy:\n\n- **Local quantized checkpoint** (from ptq skill or manual export): look for `hf_quant_config.json` in the directory. If coming from a prior PTQ run in the same workspace, check common output locations: `output\u002F`, `outputs\u002F`, `exported_model\u002F`, or the `--export_path` used in the PTQ command.\n- **HuggingFace model hub** (e.g., `nvidia\u002FLlama-3.1-8B-Instruct-FP8`): use directly\n- **Unquantized model**: deploy as-is (BF16) or suggest quantizing first with the ptq skill\n\n> **Note:** This skill expects HF-format checkpoints (from PTQ with `--export_fmt hf`). TRT-LLM format checkpoints should be deployed directly with TRT-LLM — see `references\u002Ftrtllm.md`.\n\nCheck the quantization format if applicable:\n\n```bash\ncat \u003Ccheckpoint_path>\u002Fhf_quant_config.json 2>\u002Fdev\u002Fnull || echo \"No hf_quant_config.json\"\n```\n\nIf not found, also check `config.json` for a `quantization_config` section with `quant_method: \"modelopt\"`. If neither exists, the checkpoint is unquantized.\n\n### 2. Choose the framework\n\nIf the user hasn't specified a framework, recommend based on this priority:\n\n| Situation | Recommended | Why |\n|-----------|-------------|-----|\n| General use | **vLLM** | Widest ecosystem, easy setup, OpenAI-compatible |\n| Best SGLang model support | **SGLang** | Strong DeepSeek\u002FLlama 4 support |\n| Maximum optimization | **TRT-LLM** | Best throughput via engine compilation |\n| Mixed-precision \u002F AutoQuant | **TRT-LLM AutoDeploy** | Only option for AutoQuant checkpoints |\n\nCheck the support matrix in `references\u002Fsupport-matrix.md` to confirm the model + format + framework combination is supported.\n\n### 3. Check the environment\n\nUse the common skill's `environment-setup.md` for GPU detection, local vs remote, and SLURM\u002FDocker\u002Fbare metal detection. After completing it you should know: GPU model\u002Fcount, local or remote, and execution environment.\n\nThen check the **deployment framework** is installed:\n\n```bash\npython -c \"import vllm; print(f'vLLM {vllm.__version__}')\" 2>\u002Fdev\u002Fnull || echo \"vLLM not installed\"\npython -c \"import sglang; print(f'SGLang {sglang.__version__}')\" 2>\u002Fdev\u002Fnull || echo \"SGLang not installed\"\npython -c \"import tensorrt_llm; print(f'TRT-LLM {tensorrt_llm.__version__}')\" 2>\u002Fdev\u002Fnull || echo \"TRT-LLM not installed\"\n```\n\nIf not installed, consult `references\u002Fsetup.md`.\n\n**GPU memory estimate** (to determine tensor parallelism):\n\n- BF16: `params × 2 bytes` (8B ≈ 16 GB)\n- FP8: `params × 1 byte` (8B ≈ 8 GB)\n- FP4: `params × 0.5 bytes` (8B ≈ 4 GB)\n- Add ~2-4 GB for KV cache and framework overhead\n\nIf the model exceeds single GPU memory, use tensor parallelism (`-tp \u003Cnum_gpus>`).\n\n### 4. Deploy\n\nRead the framework-specific reference for detailed instructions:\n\n| Framework | Reference file |\n|-----------|---------------|\n| vLLM | `references\u002Fvllm.md` |\n| SGLang | `references\u002Fsglang.md` |\n| TRT-LLM | `references\u002Ftrtllm.md` |\n\n**Quick-start commands** (for common cases):\n\n#### vLLM\n\n```bash\n# Serve as OpenAI-compatible endpoint\npython -m vllm.entrypoints.openai.api_server \\\n    --model \u003Ccheckpoint_path> \\\n    --quantization modelopt \\\n    --tensor-parallel-size \u003Cnum_gpus> \\\n    --host 0.0.0.0 --port 8000\n```\n\nFor NVFP4 checkpoints, use `--quantization modelopt_fp4`.\n\n> **NVFP4 on Blackwell B300\u002FGB300 (sm_103) needs a CUDA-13 image.** From v0.20.0\n> on, release tags are CUDA-13 unsuffixed (e.g. `vllm\u002Fvllm-openai:v0.26.0`) with\n> `-cu129` the CUDA-12 opt-out; v0.19.x and earlier were the other way round\n> (`-cu130` = CUDA 13), and `-cu130` no longer exists after v0.20.0. Don't trust\n> the tag name — select a tag reporting `CUDA_VERSION` >= 13 in the config blob\n> of your platform's child manifest (arm64 Grace\u002FGB300, amd64 x86);\n> `TORCH_CUDA_ARCH_LIST` differs between the two. A cu12 build has **no sm_103\n> FP4 kernel**, so vLLM loads the checkpoint then dies at engine init with `CUDA\n> error: no kernel image is available for execution on the device` (affects the\n> `flashinfer` and `cutlass` NVFP4 backends; `marlin` separately fails on\n> non-64-divisible layer dims).\n> Cross-check via\n> `recipes.vllm.ai\u002F\u003Corg>\u002F\u003Cmodel>?hardware=b300` (JS-rendered — fetch the raw\n> markdown at `github.com\u002Fvllm-project\u002Frecipes\u002Fblob\u002Fmain\u002F\u003Corg>\u002F\u003Cmodel>.md`). For\n> multimodal models on sm_103, also pass `--mm-encoder-attn-backend TRITON_ATTN`\n> (the default CuTe ViT flash-attn asserts \"Only SM 10.x and 11.x\").\n\n#### SGLang\n\n```bash\npython -m sglang.launch_server \\\n    --model-path \u003Ccheckpoint_path> \\\n    --quantization modelopt \\\n    --tp \u003Cnum_gpus> \\\n    --host 0.0.0.0 --port 8000\n```\n\nFor NVFP4 checkpoints, use `--quantization modelopt_fp4`.\n\n> **Cross-check SGLang launch flags via the SGLang cookbook** (the SGLang analog\n> of `recipes.vllm.ai`): `docs.sglang.io\u002Fcookbook\u002F\u003Ccategory>\u002F\u003Corg>\u002F\u003Cmodel>` (e.g.\n> `...\u002Fautoregressive\u002FDeepSeek\u002FDeepSeek-V4`) — authoritative for parallelism, MoE\n> backends, strategy flags, Docker image, and min version. Select the variant via\n> the URL fragment `#hw=...&variant=...&quant=...&strategy=...&nodes=...`. The\n> page is **JS-rendered** — fetch the raw markdown at\n> `raw.githubusercontent.com\u002Fsgl-project\u002Fsglang\u002Fmain\u002Fdocs_new\u002Fcookbook\u002F\u003Ccategory>\u002F\u003Corg>\u002F\u003Cmodel>.mdx`.\n> SM120 (RTX PRO 6000) needs the `lmsysorg\u002Fsglang:dev` nightly (`:latest` lacks\n> SM120). See `references\u002Fsglang.md` for the full backend\u002Fflag matrix.\n\n#### TRT-LLM (direct)\n\n```python\nfrom tensorrt_llm import LLM, SamplingParams\nllm = LLM(model=\"\u003Ccheckpoint_path>\")\noutputs = llm.generate([\"Hello, my name is\"], SamplingParams(temperature=0.8, top_p=0.95))\n```\n\n#### TRT-LLM AutoDeploy\n\nFor AutoQuant or mixed-precision checkpoints, see `references\u002Ftrtllm.md`.\n\n### 5. Verify the deployment\n\nAfter the server starts, verify it's healthy:\n\n```bash\n# Health check\ncurl -s http:\u002F\u002Flocalhost:8000\u002Fhealth\n\n# List models\ncurl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels | python -m json.tool\n\n# Test generation\ncurl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fcompletions \\\n    -H \"Content-Type: application\u002Fjson\" \\\n    -d '{\n        \"model\": \"\u003Cmodel_name>\",\n        \"prompt\": \"The capital of France is\",\n        \"max_tokens\": 32\n    }' | python -m json.tool\n```\n\nAll checks must pass before reporting success to the user.\n\n### 5b. Benchmark throughput\u002Flatency (optional)\n\nIf the user asks to benchmark, measure throughput\u002Flatency, or compare precisions,\nuse **AIPerf** (Apache-2.0, OpenAI-compatible client benchmark). See\n`references\u002Fbenchmarking.md` for install, the pre-benchmark coherence gate, the\n`aiperf profile` flags (notably `--extra-inputs ignore_eos:true`), suggested\ntoken shapes, and how to read `profile_export_aiperf.json`.\n\n### 6. Remote deployment (SSH\u002FSLURM)\n\nIf a cluster config exists (`~\u002F.config\u002Fmodelopt\u002Fclusters.yaml`, `.agents\u002Fclusters.yaml`, or `.claude\u002Fclusters.yaml`), or the user mentions running on a remote machine:\n\n0. **Check container registry auth** — before submitting any SLURM job with a container image, verify credentials exist on the cluster per the common skill's `slurm-setup.md` section 6. If credentials are missing for the image's registry, ask the user to fix auth or switch to an image on an authenticated registry (e.g., NGC). **Do not submit until auth is confirmed.**\n\n1. **Source remote utilities:** Load the common skill, then resolve\n   `remote_exec.sh` from that skill's root.\n\n   ```bash\n   source \"\u003Ccommon-skill-dir>\u002Fremote_exec.sh\"\n   remote_load_cluster\n   remote_check_ssh\n   remote_detect_env\n   ```\n\n2. **Sync the checkpoint** (only if it was produced locally):\n\n   If the checkpoint path is a remote\u002Fabsolute path (e.g., from a prior PTQ run on the cluster), skip sync — it's already there. Verify with `remote_run \"ls \u003Ccheckpoint_path>\u002Fconfig.json\"`. Only sync if the checkpoint is local:\n\n   ```bash\n   remote_sync_to \u003Clocal_checkpoint_path> \u003Csession_id>\u002F\u003Cmodel>\u002Fcheckpoints\u002F\n   ```\n\n3. **Deploy based on remote environment:**\n\n   - **SLURM** — see the common skill's `slurm-setup.md` for job script templates (container setup, account\u002Fpartition discovery). The server command inside the container is the same as Step 4 (e.g., `python -m vllm.entrypoints.openai.api_server --model \u003Cpath> --quantization modelopt`). After submitting, register the job and set up monitoring per the **monitor skill**. Get the node hostname from `squeue -j $JOBID -o %N`.\n\n   - **Bare metal \u002F Docker** — use `remote_run` to start the server directly:\n\n     ```bash\n     remote_run \"nohup python -m vllm.entrypoints.openai.api_server --model \u003Cpath> --port 8000 > deploy.log 2>&1 &\"\n     ```\n\n4. **Verify remotely:**\n\n   ```bash\n   remote_run \"curl -s http:\u002F\u002Flocalhost:8000\u002Fhealth\"\n   remote_run \"curl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels\"\n   ```\n\n5. **Report the endpoint** — include the remote hostname and port so the user can connect (e.g., `http:\u002F\u002F\u003Cnode_hostname>:8000`). For SLURM, note that the port is only reachable from within the cluster network.\n\nFor NEL-managed deployment (evaluation with self-deployment), use the evaluation skill instead — NEL handles SLURM container deployment, health checks, and teardown automatically.\n\n## Error Handling\n\n| Error | Cause | Fix |\n|-------|-------|-----|\n| `CUDA error: an illegal memory access` on an **NVFP4 MoE** (`trtllm_fused_moe_dev_kernel.cu`, `deepgemm`) | Fused-MoE FP4 kernel fault on long-context loads; engine dies, all requests 500 | Try `VLLM_USE_FLASHINFER_MOE_FP4=1` + `VLLM_FLASHINFER_MOE_BACKEND=throughput`. **Not always a fix** — on DeepSeek-V4 it moved the fault from TRT-LLM to DeepGEMM. It also makes quantized vs baseline not kernel-matched; record that when reporting deltas. |\n| `CUDA out of memory` | Model too large for GPU(s) | Increase `--tensor-parallel-size` or use a smaller model |\n| `quantization=\"modelopt\" not recognized` | vLLM\u002FSGLang version too old | Upgrade: vLLM >= 0.10.1, SGLang >= 0.4.10 |\n| `hf_quant_config.json not found` | Not a ModelOpt-exported checkpoint | Re-export with `export_hf_checkpoint()`, or remove `--quantization` flag |\n| `Connection refused` on health check | Server still starting | Wait 30-60s for large models; check logs for errors |\n| `modelopt_fp4 not supported` | Framework doesn't support FP4 for this model | Check support matrix in `references\u002Fsupport-matrix.md` |\n\n## Unsupported Models\n\nIf the model is not in the validated support matrix (`references\u002Fsupport-matrix.md`), deployment may fail due to weight key mismatches, missing architecture mappings, or quantized\u002Funquantized layer confusion. Read `references\u002Funsupported-models.md` for the iterative debug loop: **run → read error → diagnose → patch framework source → re-run**. For kernel-level issues, escalate to the framework team rather than attempting fixes.\n\n## Success Criteria\n\n1. Server process is running and healthy (`\u002Fhealth` returns 200)\n2. Model is listed at `\u002Fv1\u002Fmodels`\n3. Test generation produces coherent output\n4. Server URL and port are reported to the user\n5. If benchmarking was requested, throughput\u002Flatency numbers are reported\n",{"data":34,"body":35},{"name":4,"description":6,"license":25},{"type":36,"children":37},"root",[38,47,53,60,74,363,368,374,381,402,447,460,466,471,554,584,589,663,692,698,703,819,832,838,851,863,1031,1043,1053,1100,1113,1119,1124,1193,1203,1208,1343,1355,1478,1483,1600,1610,1693,1699,1731,1736,1747,1753,1758,1972,1977,1983,2026,2032,2060,2429,2434,2440,2677,2683,2710,2716,2758],{"type":39,"tag":40,"props":41,"children":43},"element","h1",{"id":42},"deployment-skill",[44],{"type":45,"value":46},"text","Deployment Skill",{"type":39,"tag":48,"props":49,"children":50},"p",{},[51],{"type":45,"value":52},"Serve a model checkpoint as an OpenAI-compatible inference endpoint. Supports vLLM, SGLang, and TRT-LLM (including AutoDeploy).",{"type":39,"tag":54,"props":55,"children":57},"h2",{"id":56},"quick-start",[58],{"type":45,"value":59},"Quick Start",{"type":39,"tag":48,"props":61,"children":62},{},[63,65,72],{"type":45,"value":64},"Prefer ",{"type":39,"tag":66,"props":67,"children":69},"code",{"className":68},[],[70],{"type":45,"value":71},"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh",{"type":45,"value":73}," for standard local deployments — it handles quant detection, health checks, and server lifecycle. Use the raw framework commands in Step 4 when you need flags the script doesn't support, or for remote deployment.",{"type":39,"tag":75,"props":76,"children":81},"pre",{"className":77,"code":78,"language":79,"meta":80,"style":80},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Start vLLM server with a ModelOpt checkpoint\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" start --model .\u002Fqwen3-0.6b-fp8\n\n# Start with SGLang and tensor parallelism\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" start --model .\u002Fllama-70b-nvfp4 --framework sglang --tp 4\n\n# Start from HuggingFace hub\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" start --model nvidia\u002FLlama-3.1-8B-Instruct-FP8\n\n# Test the API\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" test\n\n# Check status\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" status\n\n# Stop\n\"$SKILL_DIR\u002Fscripts\u002Fdeploy.sh\" stop\n","bash","",[82],{"type":39,"tag":66,"props":83,"children":84},{"__ignoreMap":80},[85,97,134,144,153,203,211,220,249,257,266,287,295,304,325,333,342],{"type":39,"tag":86,"props":87,"children":90},"span",{"class":88,"line":89},"line",1,[91],{"type":39,"tag":86,"props":92,"children":94},{"style":93},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[95],{"type":45,"value":96},"# Start vLLM server with a ModelOpt checkpoint\n",{"type":39,"tag":86,"props":98,"children":100},{"class":88,"line":99},2,[101,107,113,118,124,129],{"type":39,"tag":86,"props":102,"children":104},{"style":103},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[105],{"type":45,"value":106},"\"",{"type":39,"tag":86,"props":108,"children":110},{"style":109},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[111],{"type":45,"value":112},"$SKILL_DIR",{"type":39,"tag":86,"props":114,"children":115},{"style":103},[116],{"type":45,"value":117},"\u002Fscripts\u002Fdeploy.sh\"",{"type":39,"tag":86,"props":119,"children":121},{"style":120},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[122],{"type":45,"value":123}," start",{"type":39,"tag":86,"props":125,"children":126},{"style":120},[127],{"type":45,"value":128}," --model",{"type":39,"tag":86,"props":130,"children":131},{"style":120},[132],{"type":45,"value":133}," .\u002Fqwen3-0.6b-fp8\n",{"type":39,"tag":86,"props":135,"children":137},{"class":88,"line":136},3,[138],{"type":39,"tag":86,"props":139,"children":141},{"emptyLinePlaceholder":140},true,[142],{"type":45,"value":143},"\n",{"type":39,"tag":86,"props":145,"children":147},{"class":88,"line":146},4,[148],{"type":39,"tag":86,"props":149,"children":150},{"style":93},[151],{"type":45,"value":152},"# Start with SGLang and tensor parallelism\n",{"type":39,"tag":86,"props":154,"children":156},{"class":88,"line":155},5,[157,161,165,169,173,177,182,187,192,197],{"type":39,"tag":86,"props":158,"children":159},{"style":103},[160],{"type":45,"value":106},{"type":39,"tag":86,"props":162,"children":163},{"style":109},[164],{"type":45,"value":112},{"type":39,"tag":86,"props":166,"children":167},{"style":103},[168],{"type":45,"value":117},{"type":39,"tag":86,"props":170,"children":171},{"style":120},[172],{"type":45,"value":123},{"type":39,"tag":86,"props":174,"children":175},{"style":120},[176],{"type":45,"value":128},{"type":39,"tag":86,"props":178,"children":179},{"style":120},[180],{"type":45,"value":181}," .\u002Fllama-70b-nvfp4",{"type":39,"tag":86,"props":183,"children":184},{"style":120},[185],{"type":45,"value":186}," --framework",{"type":39,"tag":86,"props":188,"children":189},{"style":120},[190],{"type":45,"value":191}," sglang",{"type":39,"tag":86,"props":193,"children":194},{"style":120},[195],{"type":45,"value":196}," --tp",{"type":39,"tag":86,"props":198,"children":200},{"style":199},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[201],{"type":45,"value":202}," 4\n",{"type":39,"tag":86,"props":204,"children":206},{"class":88,"line":205},6,[207],{"type":39,"tag":86,"props":208,"children":209},{"emptyLinePlaceholder":140},[210],{"type":45,"value":143},{"type":39,"tag":86,"props":212,"children":214},{"class":88,"line":213},7,[215],{"type":39,"tag":86,"props":216,"children":217},{"style":93},[218],{"type":45,"value":219},"# Start from HuggingFace hub\n",{"type":39,"tag":86,"props":221,"children":223},{"class":88,"line":222},8,[224,228,232,236,240,244],{"type":39,"tag":86,"props":225,"children":226},{"style":103},[227],{"type":45,"value":106},{"type":39,"tag":86,"props":229,"children":230},{"style":109},[231],{"type":45,"value":112},{"type":39,"tag":86,"props":233,"children":234},{"style":103},[235],{"type":45,"value":117},{"type":39,"tag":86,"props":237,"children":238},{"style":120},[239],{"type":45,"value":123},{"type":39,"tag":86,"props":241,"children":242},{"style":120},[243],{"type":45,"value":128},{"type":39,"tag":86,"props":245,"children":246},{"style":120},[247],{"type":45,"value":248}," nvidia\u002FLlama-3.1-8B-Instruct-FP8\n",{"type":39,"tag":86,"props":250,"children":252},{"class":88,"line":251},9,[253],{"type":39,"tag":86,"props":254,"children":255},{"emptyLinePlaceholder":140},[256],{"type":45,"value":143},{"type":39,"tag":86,"props":258,"children":260},{"class":88,"line":259},10,[261],{"type":39,"tag":86,"props":262,"children":263},{"style":93},[264],{"type":45,"value":265},"# Test the API\n",{"type":39,"tag":86,"props":267,"children":269},{"class":88,"line":268},11,[270,274,278,282],{"type":39,"tag":86,"props":271,"children":272},{"style":103},[273],{"type":45,"value":106},{"type":39,"tag":86,"props":275,"children":276},{"style":109},[277],{"type":45,"value":112},{"type":39,"tag":86,"props":279,"children":280},{"style":103},[281],{"type":45,"value":117},{"type":39,"tag":86,"props":283,"children":284},{"style":120},[285],{"type":45,"value":286}," test\n",{"type":39,"tag":86,"props":288,"children":290},{"class":88,"line":289},12,[291],{"type":39,"tag":86,"props":292,"children":293},{"emptyLinePlaceholder":140},[294],{"type":45,"value":143},{"type":39,"tag":86,"props":296,"children":298},{"class":88,"line":297},13,[299],{"type":39,"tag":86,"props":300,"children":301},{"style":93},[302],{"type":45,"value":303},"# Check status\n",{"type":39,"tag":86,"props":305,"children":307},{"class":88,"line":306},14,[308,312,316,320],{"type":39,"tag":86,"props":309,"children":310},{"style":103},[311],{"type":45,"value":106},{"type":39,"tag":86,"props":313,"children":314},{"style":109},[315],{"type":45,"value":112},{"type":39,"tag":86,"props":317,"children":318},{"style":103},[319],{"type":45,"value":117},{"type":39,"tag":86,"props":321,"children":322},{"style":120},[323],{"type":45,"value":324}," status\n",{"type":39,"tag":86,"props":326,"children":328},{"class":88,"line":327},15,[329],{"type":39,"tag":86,"props":330,"children":331},{"emptyLinePlaceholder":140},[332],{"type":45,"value":143},{"type":39,"tag":86,"props":334,"children":336},{"class":88,"line":335},16,[337],{"type":39,"tag":86,"props":338,"children":339},{"style":93},[340],{"type":45,"value":341},"# Stop\n",{"type":39,"tag":86,"props":343,"children":345},{"class":88,"line":344},17,[346,350,354,358],{"type":39,"tag":86,"props":347,"children":348},{"style":103},[349],{"type":45,"value":106},{"type":39,"tag":86,"props":351,"children":352},{"style":109},[353],{"type":45,"value":112},{"type":39,"tag":86,"props":355,"children":356},{"style":103},[357],{"type":45,"value":117},{"type":39,"tag":86,"props":359,"children":360},{"style":120},[361],{"type":45,"value":362}," stop\n",{"type":39,"tag":48,"props":364,"children":365},{},[366],{"type":45,"value":367},"The script handles: GPU detection, quantization flag auto-detection (FP8 vs FP4), server lifecycle (start\u002Fstop\u002Frestart\u002Fstatus), health check polling, and API testing.",{"type":39,"tag":54,"props":369,"children":371},{"id":370},"decision-flow",[372],{"type":45,"value":373},"Decision Flow",{"type":39,"tag":375,"props":376,"children":378},"h3",{"id":377},"_0-check-workspace-multi-user-slack-bot",[379],{"type":45,"value":380},"0. Check workspace (multi-user \u002F Slack bot)",{"type":39,"tag":48,"props":382,"children":383},{},[384,386,392,394,400],{"type":45,"value":385},"If ",{"type":39,"tag":66,"props":387,"children":389},{"className":388},[],[390],{"type":45,"value":391},"MODELOPT_WORKSPACE_ROOT",{"type":45,"value":393}," is set, use the common skill's ",{"type":39,"tag":66,"props":395,"children":397},{"className":396},[],[398],{"type":45,"value":399},"workspace-management.md",{"type":45,"value":401},". Before creating a new workspace, check the current session for existing model workspaces — especially if deploying a checkpoint from a prior PTQ run:",{"type":39,"tag":75,"props":403,"children":405},{"className":77,"code":404,"language":79,"meta":80,"style":80},"ls \"$MODELOPT_WORKSPACE_ROOT\u002F\u003Csession_id>\u002F\" 2>\u002Fdev\u002Fnull\n",[406],{"type":39,"tag":66,"props":407,"children":408},{"__ignoreMap":80},[409],{"type":39,"tag":86,"props":410,"children":411},{"class":88,"line":89},[412,417,423,428,433,437,442],{"type":39,"tag":86,"props":413,"children":414},{"style":103},[415],{"type":45,"value":416},"ls",{"type":39,"tag":86,"props":418,"children":420},{"style":419},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[421],{"type":45,"value":422}," \"",{"type":39,"tag":86,"props":424,"children":425},{"style":109},[426],{"type":45,"value":427},"$MODELOPT_WORKSPACE_ROOT",{"type":39,"tag":86,"props":429,"children":430},{"style":120},[431],{"type":45,"value":432},"\u002F\u003Csession_id>\u002F",{"type":39,"tag":86,"props":434,"children":435},{"style":419},[436],{"type":45,"value":106},{"type":39,"tag":86,"props":438,"children":439},{"style":419},[440],{"type":45,"value":441}," 2>",{"type":39,"tag":86,"props":443,"children":444},{"style":120},[445],{"type":45,"value":446},"\u002Fdev\u002Fnull\n",{"type":39,"tag":48,"props":448,"children":449},{},[450,452,458],{"type":45,"value":451},"If the user says \"deploy the model I just quantized\" or references a previous PTQ, find the matching workspace and ",{"type":39,"tag":66,"props":453,"children":455},{"className":454},[],[456],{"type":45,"value":457},"cd",{"type":45,"value":459}," into it. The checkpoint should be in that workspace's output directory.",{"type":39,"tag":375,"props":461,"children":463},{"id":462},"_1-identify-the-checkpoint",[464],{"type":45,"value":465},"1. Identify the checkpoint",{"type":39,"tag":48,"props":467,"children":468},{},[469],{"type":45,"value":470},"Determine what the user wants to deploy:",{"type":39,"tag":472,"props":473,"children":474},"ul",{},[475,526,544],{"type":39,"tag":476,"props":477,"children":478},"li",{},[479,485,487,493,495,501,503,509,510,516,518,524],{"type":39,"tag":480,"props":481,"children":482},"strong",{},[483],{"type":45,"value":484},"Local quantized checkpoint",{"type":45,"value":486}," (from ptq skill or manual export): look for ",{"type":39,"tag":66,"props":488,"children":490},{"className":489},[],[491],{"type":45,"value":492},"hf_quant_config.json",{"type":45,"value":494}," in the directory. If coming from a prior PTQ run in the same workspace, check common output locations: ",{"type":39,"tag":66,"props":496,"children":498},{"className":497},[],[499],{"type":45,"value":500},"output\u002F",{"type":45,"value":502},", ",{"type":39,"tag":66,"props":504,"children":506},{"className":505},[],[507],{"type":45,"value":508},"outputs\u002F",{"type":45,"value":502},{"type":39,"tag":66,"props":511,"children":513},{"className":512},[],[514],{"type":45,"value":515},"exported_model\u002F",{"type":45,"value":517},", or the ",{"type":39,"tag":66,"props":519,"children":521},{"className":520},[],[522],{"type":45,"value":523},"--export_path",{"type":45,"value":525}," used in the PTQ command.",{"type":39,"tag":476,"props":527,"children":528},{},[529,534,536,542],{"type":39,"tag":480,"props":530,"children":531},{},[532],{"type":45,"value":533},"HuggingFace model hub",{"type":45,"value":535}," (e.g., ",{"type":39,"tag":66,"props":537,"children":539},{"className":538},[],[540],{"type":45,"value":541},"nvidia\u002FLlama-3.1-8B-Instruct-FP8",{"type":45,"value":543},"): use directly",{"type":39,"tag":476,"props":545,"children":546},{},[547,552],{"type":39,"tag":480,"props":548,"children":549},{},[550],{"type":45,"value":551},"Unquantized model",{"type":45,"value":553},": deploy as-is (BF16) or suggest quantizing first with the ptq skill",{"type":39,"tag":555,"props":556,"children":557},"blockquote",{},[558],{"type":39,"tag":48,"props":559,"children":560},{},[561,566,568,574,576,582],{"type":39,"tag":480,"props":562,"children":563},{},[564],{"type":45,"value":565},"Note:",{"type":45,"value":567}," This skill expects HF-format checkpoints (from PTQ with ",{"type":39,"tag":66,"props":569,"children":571},{"className":570},[],[572],{"type":45,"value":573},"--export_fmt hf",{"type":45,"value":575},"). TRT-LLM format checkpoints should be deployed directly with TRT-LLM — see ",{"type":39,"tag":66,"props":577,"children":579},{"className":578},[],[580],{"type":45,"value":581},"references\u002Ftrtllm.md",{"type":45,"value":583},".",{"type":39,"tag":48,"props":585,"children":586},{},[587],{"type":45,"value":588},"Check the quantization format if applicable:",{"type":39,"tag":75,"props":590,"children":592},{"className":77,"code":591,"language":79,"meta":80,"style":80},"cat \u003Ccheckpoint_path>\u002Fhf_quant_config.json 2>\u002Fdev\u002Fnull || echo \"No hf_quant_config.json\"\n",[593],{"type":39,"tag":66,"props":594,"children":595},{"__ignoreMap":80},[596],{"type":39,"tag":86,"props":597,"children":598},{"class":88,"line":89},[599,604,609,614,619,624,629,633,638,643,649,653,658],{"type":39,"tag":86,"props":600,"children":601},{"style":103},[602],{"type":45,"value":603},"cat",{"type":39,"tag":86,"props":605,"children":606},{"style":419},[607],{"type":45,"value":608}," \u003C",{"type":39,"tag":86,"props":610,"children":611},{"style":120},[612],{"type":45,"value":613},"checkpoint_pat",{"type":39,"tag":86,"props":615,"children":616},{"style":109},[617],{"type":45,"value":618},"h",{"type":39,"tag":86,"props":620,"children":621},{"style":419},[622],{"type":45,"value":623},">",{"type":39,"tag":86,"props":625,"children":626},{"style":120},[627],{"type":45,"value":628},"\u002Fhf_quant_config.json",{"type":39,"tag":86,"props":630,"children":631},{"style":419},[632],{"type":45,"value":441},{"type":39,"tag":86,"props":634,"children":635},{"style":120},[636],{"type":45,"value":637},"\u002Fdev\u002Fnull",{"type":39,"tag":86,"props":639,"children":640},{"style":419},[641],{"type":45,"value":642}," ||",{"type":39,"tag":86,"props":644,"children":646},{"style":645},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[647],{"type":45,"value":648}," echo",{"type":39,"tag":86,"props":650,"children":651},{"style":419},[652],{"type":45,"value":422},{"type":39,"tag":86,"props":654,"children":655},{"style":120},[656],{"type":45,"value":657},"No hf_quant_config.json",{"type":39,"tag":86,"props":659,"children":660},{"style":419},[661],{"type":45,"value":662},"\"\n",{"type":39,"tag":48,"props":664,"children":665},{},[666,668,674,676,682,684,690],{"type":45,"value":667},"If not found, also check ",{"type":39,"tag":66,"props":669,"children":671},{"className":670},[],[672],{"type":45,"value":673},"config.json",{"type":45,"value":675}," for a ",{"type":39,"tag":66,"props":677,"children":679},{"className":678},[],[680],{"type":45,"value":681},"quantization_config",{"type":45,"value":683}," section with ",{"type":39,"tag":66,"props":685,"children":687},{"className":686},[],[688],{"type":45,"value":689},"quant_method: \"modelopt\"",{"type":45,"value":691},". If neither exists, the checkpoint is unquantized.",{"type":39,"tag":375,"props":693,"children":695},{"id":694},"_2-choose-the-framework",[696],{"type":45,"value":697},"2. Choose the framework",{"type":39,"tag":48,"props":699,"children":700},{},[701],{"type":45,"value":702},"If the user hasn't specified a framework, recommend based on this priority:",{"type":39,"tag":704,"props":705,"children":706},"table",{},[707,731],{"type":39,"tag":708,"props":709,"children":710},"thead",{},[711],{"type":39,"tag":712,"props":713,"children":714},"tr",{},[715,721,726],{"type":39,"tag":716,"props":717,"children":718},"th",{},[719],{"type":45,"value":720},"Situation",{"type":39,"tag":716,"props":722,"children":723},{},[724],{"type":45,"value":725},"Recommended",{"type":39,"tag":716,"props":727,"children":728},{},[729],{"type":45,"value":730},"Why",{"type":39,"tag":732,"props":733,"children":734},"tbody",{},[735,756,777,798],{"type":39,"tag":712,"props":736,"children":737},{},[738,744,751],{"type":39,"tag":739,"props":740,"children":741},"td",{},[742],{"type":45,"value":743},"General use",{"type":39,"tag":739,"props":745,"children":746},{},[747],{"type":39,"tag":480,"props":748,"children":749},{},[750],{"type":45,"value":17},{"type":39,"tag":739,"props":752,"children":753},{},[754],{"type":45,"value":755},"Widest ecosystem, easy setup, OpenAI-compatible",{"type":39,"tag":712,"props":757,"children":758},{},[759,764,772],{"type":39,"tag":739,"props":760,"children":761},{},[762],{"type":45,"value":763},"Best SGLang model support",{"type":39,"tag":739,"props":765,"children":766},{},[767],{"type":39,"tag":480,"props":768,"children":769},{},[770],{"type":45,"value":771},"SGLang",{"type":39,"tag":739,"props":773,"children":774},{},[775],{"type":45,"value":776},"Strong DeepSeek\u002FLlama 4 support",{"type":39,"tag":712,"props":778,"children":779},{},[780,785,793],{"type":39,"tag":739,"props":781,"children":782},{},[783],{"type":45,"value":784},"Maximum optimization",{"type":39,"tag":739,"props":786,"children":787},{},[788],{"type":39,"tag":480,"props":789,"children":790},{},[791],{"type":45,"value":792},"TRT-LLM",{"type":39,"tag":739,"props":794,"children":795},{},[796],{"type":45,"value":797},"Best throughput via engine compilation",{"type":39,"tag":712,"props":799,"children":800},{},[801,806,814],{"type":39,"tag":739,"props":802,"children":803},{},[804],{"type":45,"value":805},"Mixed-precision \u002F AutoQuant",{"type":39,"tag":739,"props":807,"children":808},{},[809],{"type":39,"tag":480,"props":810,"children":811},{},[812],{"type":45,"value":813},"TRT-LLM AutoDeploy",{"type":39,"tag":739,"props":815,"children":816},{},[817],{"type":45,"value":818},"Only option for AutoQuant checkpoints",{"type":39,"tag":48,"props":820,"children":821},{},[822,824,830],{"type":45,"value":823},"Check the support matrix in ",{"type":39,"tag":66,"props":825,"children":827},{"className":826},[],[828],{"type":45,"value":829},"references\u002Fsupport-matrix.md",{"type":45,"value":831}," to confirm the model + format + framework combination is supported.",{"type":39,"tag":375,"props":833,"children":835},{"id":834},"_3-check-the-environment",[836],{"type":45,"value":837},"3. Check the environment",{"type":39,"tag":48,"props":839,"children":840},{},[841,843,849],{"type":45,"value":842},"Use the common skill's ",{"type":39,"tag":66,"props":844,"children":846},{"className":845},[],[847],{"type":45,"value":848},"environment-setup.md",{"type":45,"value":850}," for GPU detection, local vs remote, and SLURM\u002FDocker\u002Fbare metal detection. After completing it you should know: GPU model\u002Fcount, local or remote, and execution environment.",{"type":39,"tag":48,"props":852,"children":853},{},[854,856,861],{"type":45,"value":855},"Then check the ",{"type":39,"tag":480,"props":857,"children":858},{},[859],{"type":45,"value":860},"deployment framework",{"type":45,"value":862}," is installed:",{"type":39,"tag":75,"props":864,"children":866},{"className":77,"code":865,"language":79,"meta":80,"style":80},"python -c \"import vllm; print(f'vLLM {vllm.__version__}')\" 2>\u002Fdev\u002Fnull || echo \"vLLM not installed\"\npython -c \"import sglang; print(f'SGLang {sglang.__version__}')\" 2>\u002Fdev\u002Fnull || echo \"SGLang not installed\"\npython -c \"import tensorrt_llm; print(f'TRT-LLM {tensorrt_llm.__version__}')\" 2>\u002Fdev\u002Fnull || echo \"TRT-LLM not installed\"\n",[867],{"type":39,"tag":66,"props":868,"children":869},{"__ignoreMap":80},[870,925,978],{"type":39,"tag":86,"props":871,"children":872},{"class":88,"line":89},[873,878,883,887,892,896,900,904,908,912,916,921],{"type":39,"tag":86,"props":874,"children":875},{"style":103},[876],{"type":45,"value":877},"python",{"type":39,"tag":86,"props":879,"children":880},{"style":120},[881],{"type":45,"value":882}," -c",{"type":39,"tag":86,"props":884,"children":885},{"style":419},[886],{"type":45,"value":422},{"type":39,"tag":86,"props":888,"children":889},{"style":120},[890],{"type":45,"value":891},"import vllm; print(f'vLLM {vllm.__version__}')",{"type":39,"tag":86,"props":893,"children":894},{"style":419},[895],{"type":45,"value":106},{"type":39,"tag":86,"props":897,"children":898},{"style":419},[899],{"type":45,"value":441},{"type":39,"tag":86,"props":901,"children":902},{"style":120},[903],{"type":45,"value":637},{"type":39,"tag":86,"props":905,"children":906},{"style":419},[907],{"type":45,"value":642},{"type":39,"tag":86,"props":909,"children":910},{"style":645},[911],{"type":45,"value":648},{"type":39,"tag":86,"props":913,"children":914},{"style":419},[915],{"type":45,"value":422},{"type":39,"tag":86,"props":917,"children":918},{"style":120},[919],{"type":45,"value":920},"vLLM not installed",{"type":39,"tag":86,"props":922,"children":923},{"style":419},[924],{"type":45,"value":662},{"type":39,"tag":86,"props":926,"children":927},{"class":88,"line":99},[928,932,936,940,945,949,953,957,961,965,969,974],{"type":39,"tag":86,"props":929,"children":930},{"style":103},[931],{"type":45,"value":877},{"type":39,"tag":86,"props":933,"children":934},{"style":120},[935],{"type":45,"value":882},{"type":39,"tag":86,"props":937,"children":938},{"style":419},[939],{"type":45,"value":422},{"type":39,"tag":86,"props":941,"children":942},{"style":120},[943],{"type":45,"value":944},"import sglang; print(f'SGLang {sglang.__version__}')",{"type":39,"tag":86,"props":946,"children":947},{"style":419},[948],{"type":45,"value":106},{"type":39,"tag":86,"props":950,"children":951},{"style":419},[952],{"type":45,"value":441},{"type":39,"tag":86,"props":954,"children":955},{"style":120},[956],{"type":45,"value":637},{"type":39,"tag":86,"props":958,"children":959},{"style":419},[960],{"type":45,"value":642},{"type":39,"tag":86,"props":962,"children":963},{"style":645},[964],{"type":45,"value":648},{"type":39,"tag":86,"props":966,"children":967},{"style":419},[968],{"type":45,"value":422},{"type":39,"tag":86,"props":970,"children":971},{"style":120},[972],{"type":45,"value":973},"SGLang not installed",{"type":39,"tag":86,"props":975,"children":976},{"style":419},[977],{"type":45,"value":662},{"type":39,"tag":86,"props":979,"children":980},{"class":88,"line":136},[981,985,989,993,998,1002,1006,1010,1014,1018,1022,1027],{"type":39,"tag":86,"props":982,"children":983},{"style":103},[984],{"type":45,"value":877},{"type":39,"tag":86,"props":986,"children":987},{"style":120},[988],{"type":45,"value":882},{"type":39,"tag":86,"props":990,"children":991},{"style":419},[992],{"type":45,"value":422},{"type":39,"tag":86,"props":994,"children":995},{"style":120},[996],{"type":45,"value":997},"import tensorrt_llm; print(f'TRT-LLM {tensorrt_llm.__version__}')",{"type":39,"tag":86,"props":999,"children":1000},{"style":419},[1001],{"type":45,"value":106},{"type":39,"tag":86,"props":1003,"children":1004},{"style":419},[1005],{"type":45,"value":441},{"type":39,"tag":86,"props":1007,"children":1008},{"style":120},[1009],{"type":45,"value":637},{"type":39,"tag":86,"props":1011,"children":1012},{"style":419},[1013],{"type":45,"value":642},{"type":39,"tag":86,"props":1015,"children":1016},{"style":645},[1017],{"type":45,"value":648},{"type":39,"tag":86,"props":1019,"children":1020},{"style":419},[1021],{"type":45,"value":422},{"type":39,"tag":86,"props":1023,"children":1024},{"style":120},[1025],{"type":45,"value":1026},"TRT-LLM not installed",{"type":39,"tag":86,"props":1028,"children":1029},{"style":419},[1030],{"type":45,"value":662},{"type":39,"tag":48,"props":1032,"children":1033},{},[1034,1036,1042],{"type":45,"value":1035},"If not installed, consult ",{"type":39,"tag":66,"props":1037,"children":1039},{"className":1038},[],[1040],{"type":45,"value":1041},"references\u002Fsetup.md",{"type":45,"value":583},{"type":39,"tag":48,"props":1044,"children":1045},{},[1046,1051],{"type":39,"tag":480,"props":1047,"children":1048},{},[1049],{"type":45,"value":1050},"GPU memory estimate",{"type":45,"value":1052}," (to determine tensor parallelism):",{"type":39,"tag":472,"props":1054,"children":1055},{},[1056,1069,1082,1095],{"type":39,"tag":476,"props":1057,"children":1058},{},[1059,1061,1067],{"type":45,"value":1060},"BF16: ",{"type":39,"tag":66,"props":1062,"children":1064},{"className":1063},[],[1065],{"type":45,"value":1066},"params × 2 bytes",{"type":45,"value":1068}," (8B ≈ 16 GB)",{"type":39,"tag":476,"props":1070,"children":1071},{},[1072,1074,1080],{"type":45,"value":1073},"FP8: ",{"type":39,"tag":66,"props":1075,"children":1077},{"className":1076},[],[1078],{"type":45,"value":1079},"params × 1 byte",{"type":45,"value":1081}," (8B ≈ 8 GB)",{"type":39,"tag":476,"props":1083,"children":1084},{},[1085,1087,1093],{"type":45,"value":1086},"FP4: ",{"type":39,"tag":66,"props":1088,"children":1090},{"className":1089},[],[1091],{"type":45,"value":1092},"params × 0.5 bytes",{"type":45,"value":1094}," (8B ≈ 4 GB)",{"type":39,"tag":476,"props":1096,"children":1097},{},[1098],{"type":45,"value":1099},"Add ~2-4 GB for KV cache and framework overhead",{"type":39,"tag":48,"props":1101,"children":1102},{},[1103,1105,1111],{"type":45,"value":1104},"If the model exceeds single GPU memory, use tensor parallelism (",{"type":39,"tag":66,"props":1106,"children":1108},{"className":1107},[],[1109],{"type":45,"value":1110},"-tp \u003Cnum_gpus>",{"type":45,"value":1112},").",{"type":39,"tag":375,"props":1114,"children":1116},{"id":1115},"_4-deploy",[1117],{"type":45,"value":1118},"4. Deploy",{"type":39,"tag":48,"props":1120,"children":1121},{},[1122],{"type":45,"value":1123},"Read the framework-specific reference for detailed instructions:",{"type":39,"tag":704,"props":1125,"children":1126},{},[1127,1143],{"type":39,"tag":708,"props":1128,"children":1129},{},[1130],{"type":39,"tag":712,"props":1131,"children":1132},{},[1133,1138],{"type":39,"tag":716,"props":1134,"children":1135},{},[1136],{"type":45,"value":1137},"Framework",{"type":39,"tag":716,"props":1139,"children":1140},{},[1141],{"type":45,"value":1142},"Reference file",{"type":39,"tag":732,"props":1144,"children":1145},{},[1146,1162,1178],{"type":39,"tag":712,"props":1147,"children":1148},{},[1149,1153],{"type":39,"tag":739,"props":1150,"children":1151},{},[1152],{"type":45,"value":17},{"type":39,"tag":739,"props":1154,"children":1155},{},[1156],{"type":39,"tag":66,"props":1157,"children":1159},{"className":1158},[],[1160],{"type":45,"value":1161},"references\u002Fvllm.md",{"type":39,"tag":712,"props":1163,"children":1164},{},[1165,1169],{"type":39,"tag":739,"props":1166,"children":1167},{},[1168],{"type":45,"value":771},{"type":39,"tag":739,"props":1170,"children":1171},{},[1172],{"type":39,"tag":66,"props":1173,"children":1175},{"className":1174},[],[1176],{"type":45,"value":1177},"references\u002Fsglang.md",{"type":39,"tag":712,"props":1179,"children":1180},{},[1181,1185],{"type":39,"tag":739,"props":1182,"children":1183},{},[1184],{"type":45,"value":792},{"type":39,"tag":739,"props":1186,"children":1187},{},[1188],{"type":39,"tag":66,"props":1189,"children":1191},{"className":1190},[],[1192],{"type":45,"value":581},{"type":39,"tag":48,"props":1194,"children":1195},{},[1196,1201],{"type":39,"tag":480,"props":1197,"children":1198},{},[1199],{"type":45,"value":1200},"Quick-start commands",{"type":45,"value":1202}," (for common cases):",{"type":39,"tag":1204,"props":1205,"children":1206},"h4",{"id":18},[1207],{"type":45,"value":17},{"type":39,"tag":75,"props":1209,"children":1211},{"className":77,"code":1210,"language":79,"meta":80,"style":80},"# Serve as OpenAI-compatible endpoint\npython -m vllm.entrypoints.openai.api_server \\\n    --model \u003Ccheckpoint_path> \\\n    --quantization modelopt \\\n    --tensor-parallel-size \u003Cnum_gpus> \\\n    --host 0.0.0.0 --port 8000\n",[1212],{"type":39,"tag":66,"props":1213,"children":1214},{"__ignoreMap":80},[1215,1223,1245,1273,1290,1320],{"type":39,"tag":86,"props":1216,"children":1217},{"class":88,"line":89},[1218],{"type":39,"tag":86,"props":1219,"children":1220},{"style":93},[1221],{"type":45,"value":1222},"# Serve as OpenAI-compatible endpoint\n",{"type":39,"tag":86,"props":1224,"children":1225},{"class":88,"line":99},[1226,1230,1235,1240],{"type":39,"tag":86,"props":1227,"children":1228},{"style":103},[1229],{"type":45,"value":877},{"type":39,"tag":86,"props":1231,"children":1232},{"style":120},[1233],{"type":45,"value":1234}," -m",{"type":39,"tag":86,"props":1236,"children":1237},{"style":120},[1238],{"type":45,"value":1239}," vllm.entrypoints.openai.api_server",{"type":39,"tag":86,"props":1241,"children":1242},{"style":109},[1243],{"type":45,"value":1244}," \\\n",{"type":39,"tag":86,"props":1246,"children":1247},{"class":88,"line":136},[1248,1253,1257,1261,1265,1269],{"type":39,"tag":86,"props":1249,"children":1250},{"style":120},[1251],{"type":45,"value":1252},"    --model",{"type":39,"tag":86,"props":1254,"children":1255},{"style":419},[1256],{"type":45,"value":608},{"type":39,"tag":86,"props":1258,"children":1259},{"style":120},[1260],{"type":45,"value":613},{"type":39,"tag":86,"props":1262,"children":1263},{"style":109},[1264],{"type":45,"value":618},{"type":39,"tag":86,"props":1266,"children":1267},{"style":419},[1268],{"type":45,"value":623},{"type":39,"tag":86,"props":1270,"children":1271},{"style":109},[1272],{"type":45,"value":1244},{"type":39,"tag":86,"props":1274,"children":1275},{"class":88,"line":146},[1276,1281,1286],{"type":39,"tag":86,"props":1277,"children":1278},{"style":120},[1279],{"type":45,"value":1280},"    --quantization",{"type":39,"tag":86,"props":1282,"children":1283},{"style":120},[1284],{"type":45,"value":1285}," modelopt",{"type":39,"tag":86,"props":1287,"children":1288},{"style":109},[1289],{"type":45,"value":1244},{"type":39,"tag":86,"props":1291,"children":1292},{"class":88,"line":155},[1293,1298,1302,1307,1312,1316],{"type":39,"tag":86,"props":1294,"children":1295},{"style":120},[1296],{"type":45,"value":1297},"    --tensor-parallel-size",{"type":39,"tag":86,"props":1299,"children":1300},{"style":419},[1301],{"type":45,"value":608},{"type":39,"tag":86,"props":1303,"children":1304},{"style":120},[1305],{"type":45,"value":1306},"num_gpu",{"type":39,"tag":86,"props":1308,"children":1309},{"style":109},[1310],{"type":45,"value":1311},"s",{"type":39,"tag":86,"props":1313,"children":1314},{"style":419},[1315],{"type":45,"value":623},{"type":39,"tag":86,"props":1317,"children":1318},{"style":109},[1319],{"type":45,"value":1244},{"type":39,"tag":86,"props":1321,"children":1322},{"class":88,"line":205},[1323,1328,1333,1338],{"type":39,"tag":86,"props":1324,"children":1325},{"style":120},[1326],{"type":45,"value":1327},"    --host",{"type":39,"tag":86,"props":1329,"children":1330},{"style":199},[1331],{"type":45,"value":1332}," 0.0.0.0",{"type":39,"tag":86,"props":1334,"children":1335},{"style":120},[1336],{"type":45,"value":1337}," --port",{"type":39,"tag":86,"props":1339,"children":1340},{"style":199},[1341],{"type":45,"value":1342}," 8000\n",{"type":39,"tag":48,"props":1344,"children":1345},{},[1346,1348,1354],{"type":45,"value":1347},"For NVFP4 checkpoints, use ",{"type":39,"tag":66,"props":1349,"children":1351},{"className":1350},[],[1352],{"type":45,"value":1353},"--quantization modelopt_fp4",{"type":45,"value":583},{"type":39,"tag":555,"props":1356,"children":1357},{},[1358],{"type":39,"tag":48,"props":1359,"children":1360},{},[1361,1366,1368,1374,1376,1382,1384,1390,1392,1397,1399,1405,1407,1413,1415,1420,1422,1428,1430,1436,1438,1444,1446,1452,1454,1460,1462,1468,1470,1476],{"type":39,"tag":480,"props":1362,"children":1363},{},[1364],{"type":45,"value":1365},"NVFP4 on Blackwell B300\u002FGB300 (sm_103) needs a CUDA-13 image.",{"type":45,"value":1367}," From v0.20.0\non, release tags are CUDA-13 unsuffixed (e.g. ",{"type":39,"tag":66,"props":1369,"children":1371},{"className":1370},[],[1372],{"type":45,"value":1373},"vllm\u002Fvllm-openai:v0.26.0",{"type":45,"value":1375},") with\n",{"type":39,"tag":66,"props":1377,"children":1379},{"className":1378},[],[1380],{"type":45,"value":1381},"-cu129",{"type":45,"value":1383}," the CUDA-12 opt-out; v0.19.x and earlier were the other way round\n(",{"type":39,"tag":66,"props":1385,"children":1387},{"className":1386},[],[1388],{"type":45,"value":1389},"-cu130",{"type":45,"value":1391}," = CUDA 13), and ",{"type":39,"tag":66,"props":1393,"children":1395},{"className":1394},[],[1396],{"type":45,"value":1389},{"type":45,"value":1398}," no longer exists after v0.20.0. Don't trust\nthe tag name — select a tag reporting ",{"type":39,"tag":66,"props":1400,"children":1402},{"className":1401},[],[1403],{"type":45,"value":1404},"CUDA_VERSION",{"type":45,"value":1406}," >= 13 in the config blob\nof your platform's child manifest (arm64 Grace\u002FGB300, amd64 x86);\n",{"type":39,"tag":66,"props":1408,"children":1410},{"className":1409},[],[1411],{"type":45,"value":1412},"TORCH_CUDA_ARCH_LIST",{"type":45,"value":1414}," differs between the two. A cu12 build has ",{"type":39,"tag":480,"props":1416,"children":1417},{},[1418],{"type":45,"value":1419},"no sm_103\nFP4 kernel",{"type":45,"value":1421},", so vLLM loads the checkpoint then dies at engine init with ",{"type":39,"tag":66,"props":1423,"children":1425},{"className":1424},[],[1426],{"type":45,"value":1427},"CUDA error: no kernel image is available for execution on the device",{"type":45,"value":1429}," (affects the\n",{"type":39,"tag":66,"props":1431,"children":1433},{"className":1432},[],[1434],{"type":45,"value":1435},"flashinfer",{"type":45,"value":1437}," and ",{"type":39,"tag":66,"props":1439,"children":1441},{"className":1440},[],[1442],{"type":45,"value":1443},"cutlass",{"type":45,"value":1445}," NVFP4 backends; ",{"type":39,"tag":66,"props":1447,"children":1449},{"className":1448},[],[1450],{"type":45,"value":1451},"marlin",{"type":45,"value":1453}," separately fails on\nnon-64-divisible layer dims).\nCross-check via\n",{"type":39,"tag":66,"props":1455,"children":1457},{"className":1456},[],[1458],{"type":45,"value":1459},"recipes.vllm.ai\u002F\u003Corg>\u002F\u003Cmodel>?hardware=b300",{"type":45,"value":1461}," (JS-rendered — fetch the raw\nmarkdown at ",{"type":39,"tag":66,"props":1463,"children":1465},{"className":1464},[],[1466],{"type":45,"value":1467},"github.com\u002Fvllm-project\u002Frecipes\u002Fblob\u002Fmain\u002F\u003Corg>\u002F\u003Cmodel>.md",{"type":45,"value":1469},"). For\nmultimodal models on sm_103, also pass ",{"type":39,"tag":66,"props":1471,"children":1473},{"className":1472},[],[1474],{"type":45,"value":1475},"--mm-encoder-attn-backend TRITON_ATTN",{"type":45,"value":1477},"\n(the default CuTe ViT flash-attn asserts \"Only SM 10.x and 11.x\").",{"type":39,"tag":1204,"props":1479,"children":1481},{"id":1480},"sglang",[1482],{"type":45,"value":771},{"type":39,"tag":75,"props":1484,"children":1486},{"className":77,"code":1485,"language":79,"meta":80,"style":80},"python -m sglang.launch_server \\\n    --model-path \u003Ccheckpoint_path> \\\n    --quantization modelopt \\\n    --tp \u003Cnum_gpus> \\\n    --host 0.0.0.0 --port 8000\n",[1487],{"type":39,"tag":66,"props":1488,"children":1489},{"__ignoreMap":80},[1490,1510,1538,1553,1581],{"type":39,"tag":86,"props":1491,"children":1492},{"class":88,"line":89},[1493,1497,1501,1506],{"type":39,"tag":86,"props":1494,"children":1495},{"style":103},[1496],{"type":45,"value":877},{"type":39,"tag":86,"props":1498,"children":1499},{"style":120},[1500],{"type":45,"value":1234},{"type":39,"tag":86,"props":1502,"children":1503},{"style":120},[1504],{"type":45,"value":1505}," sglang.launch_server",{"type":39,"tag":86,"props":1507,"children":1508},{"style":109},[1509],{"type":45,"value":1244},{"type":39,"tag":86,"props":1511,"children":1512},{"class":88,"line":99},[1513,1518,1522,1526,1530,1534],{"type":39,"tag":86,"props":1514,"children":1515},{"style":120},[1516],{"type":45,"value":1517},"    --model-path",{"type":39,"tag":86,"props":1519,"children":1520},{"style":419},[1521],{"type":45,"value":608},{"type":39,"tag":86,"props":1523,"children":1524},{"style":120},[1525],{"type":45,"value":613},{"type":39,"tag":86,"props":1527,"children":1528},{"style":109},[1529],{"type":45,"value":618},{"type":39,"tag":86,"props":1531,"children":1532},{"style":419},[1533],{"type":45,"value":623},{"type":39,"tag":86,"props":1535,"children":1536},{"style":109},[1537],{"type":45,"value":1244},{"type":39,"tag":86,"props":1539,"children":1540},{"class":88,"line":136},[1541,1545,1549],{"type":39,"tag":86,"props":1542,"children":1543},{"style":120},[1544],{"type":45,"value":1280},{"type":39,"tag":86,"props":1546,"children":1547},{"style":120},[1548],{"type":45,"value":1285},{"type":39,"tag":86,"props":1550,"children":1551},{"style":109},[1552],{"type":45,"value":1244},{"type":39,"tag":86,"props":1554,"children":1555},{"class":88,"line":146},[1556,1561,1565,1569,1573,1577],{"type":39,"tag":86,"props":1557,"children":1558},{"style":120},[1559],{"type":45,"value":1560},"    --tp",{"type":39,"tag":86,"props":1562,"children":1563},{"style":419},[1564],{"type":45,"value":608},{"type":39,"tag":86,"props":1566,"children":1567},{"style":120},[1568],{"type":45,"value":1306},{"type":39,"tag":86,"props":1570,"children":1571},{"style":109},[1572],{"type":45,"value":1311},{"type":39,"tag":86,"props":1574,"children":1575},{"style":419},[1576],{"type":45,"value":623},{"type":39,"tag":86,"props":1578,"children":1579},{"style":109},[1580],{"type":45,"value":1244},{"type":39,"tag":86,"props":1582,"children":1583},{"class":88,"line":155},[1584,1588,1592,1596],{"type":39,"tag":86,"props":1585,"children":1586},{"style":120},[1587],{"type":45,"value":1327},{"type":39,"tag":86,"props":1589,"children":1590},{"style":199},[1591],{"type":45,"value":1332},{"type":39,"tag":86,"props":1593,"children":1594},{"style":120},[1595],{"type":45,"value":1337},{"type":39,"tag":86,"props":1597,"children":1598},{"style":199},[1599],{"type":45,"value":1342},{"type":39,"tag":48,"props":1601,"children":1602},{},[1603,1604,1609],{"type":45,"value":1347},{"type":39,"tag":66,"props":1605,"children":1607},{"className":1606},[],[1608],{"type":45,"value":1353},{"type":45,"value":583},{"type":39,"tag":555,"props":1611,"children":1612},{},[1613],{"type":39,"tag":48,"props":1614,"children":1615},{},[1616,1621,1623,1629,1631,1637,1639,1645,1647,1653,1655,1660,1662,1668,1670,1676,1678,1684,1686,1691],{"type":39,"tag":480,"props":1617,"children":1618},{},[1619],{"type":45,"value":1620},"Cross-check SGLang launch flags via the SGLang cookbook",{"type":45,"value":1622}," (the SGLang analog\nof ",{"type":39,"tag":66,"props":1624,"children":1626},{"className":1625},[],[1627],{"type":45,"value":1628},"recipes.vllm.ai",{"type":45,"value":1630},"): ",{"type":39,"tag":66,"props":1632,"children":1634},{"className":1633},[],[1635],{"type":45,"value":1636},"docs.sglang.io\u002Fcookbook\u002F\u003Ccategory>\u002F\u003Corg>\u002F\u003Cmodel>",{"type":45,"value":1638}," (e.g.\n",{"type":39,"tag":66,"props":1640,"children":1642},{"className":1641},[],[1643],{"type":45,"value":1644},"...\u002Fautoregressive\u002FDeepSeek\u002FDeepSeek-V4",{"type":45,"value":1646},") — authoritative for parallelism, MoE\nbackends, strategy flags, Docker image, and min version. Select the variant via\nthe URL fragment ",{"type":39,"tag":66,"props":1648,"children":1650},{"className":1649},[],[1651],{"type":45,"value":1652},"#hw=...&variant=...&quant=...&strategy=...&nodes=...",{"type":45,"value":1654},". The\npage is ",{"type":39,"tag":480,"props":1656,"children":1657},{},[1658],{"type":45,"value":1659},"JS-rendered",{"type":45,"value":1661}," — fetch the raw markdown at\n",{"type":39,"tag":66,"props":1663,"children":1665},{"className":1664},[],[1666],{"type":45,"value":1667},"raw.githubusercontent.com\u002Fsgl-project\u002Fsglang\u002Fmain\u002Fdocs_new\u002Fcookbook\u002F\u003Ccategory>\u002F\u003Corg>\u002F\u003Cmodel>.mdx",{"type":45,"value":1669},".\nSM120 (RTX PRO 6000) needs the ",{"type":39,"tag":66,"props":1671,"children":1673},{"className":1672},[],[1674],{"type":45,"value":1675},"lmsysorg\u002Fsglang:dev",{"type":45,"value":1677}," nightly (",{"type":39,"tag":66,"props":1679,"children":1681},{"className":1680},[],[1682],{"type":45,"value":1683},":latest",{"type":45,"value":1685}," lacks\nSM120). See ",{"type":39,"tag":66,"props":1687,"children":1689},{"className":1688},[],[1690],{"type":45,"value":1177},{"type":45,"value":1692}," for the full backend\u002Fflag matrix.",{"type":39,"tag":1204,"props":1694,"children":1696},{"id":1695},"trt-llm-direct",[1697],{"type":45,"value":1698},"TRT-LLM (direct)",{"type":39,"tag":75,"props":1700,"children":1703},{"className":1701,"code":1702,"language":877,"meta":80,"style":80},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from tensorrt_llm import LLM, SamplingParams\nllm = LLM(model=\"\u003Ccheckpoint_path>\")\noutputs = llm.generate([\"Hello, my name is\"], SamplingParams(temperature=0.8, top_p=0.95))\n",[1704],{"type":39,"tag":66,"props":1705,"children":1706},{"__ignoreMap":80},[1707,1715,1723],{"type":39,"tag":86,"props":1708,"children":1709},{"class":88,"line":89},[1710],{"type":39,"tag":86,"props":1711,"children":1712},{},[1713],{"type":45,"value":1714},"from tensorrt_llm import LLM, SamplingParams\n",{"type":39,"tag":86,"props":1716,"children":1717},{"class":88,"line":99},[1718],{"type":39,"tag":86,"props":1719,"children":1720},{},[1721],{"type":45,"value":1722},"llm = LLM(model=\"\u003Ccheckpoint_path>\")\n",{"type":39,"tag":86,"props":1724,"children":1725},{"class":88,"line":136},[1726],{"type":39,"tag":86,"props":1727,"children":1728},{},[1729],{"type":45,"value":1730},"outputs = llm.generate([\"Hello, my name is\"], SamplingParams(temperature=0.8, top_p=0.95))\n",{"type":39,"tag":1204,"props":1732,"children":1734},{"id":1733},"trt-llm-autodeploy",[1735],{"type":45,"value":813},{"type":39,"tag":48,"props":1737,"children":1738},{},[1739,1741,1746],{"type":45,"value":1740},"For AutoQuant or mixed-precision checkpoints, see ",{"type":39,"tag":66,"props":1742,"children":1744},{"className":1743},[],[1745],{"type":45,"value":581},{"type":45,"value":583},{"type":39,"tag":375,"props":1748,"children":1750},{"id":1749},"_5-verify-the-deployment",[1751],{"type":45,"value":1752},"5. Verify the deployment",{"type":39,"tag":48,"props":1754,"children":1755},{},[1756],{"type":45,"value":1757},"After the server starts, verify it's healthy:",{"type":39,"tag":75,"props":1759,"children":1761},{"className":77,"code":1760,"language":79,"meta":80,"style":80},"# Health check\ncurl -s http:\u002F\u002Flocalhost:8000\u002Fhealth\n\n# List models\ncurl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels | python -m json.tool\n\n# Test generation\ncurl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fcompletions \\\n    -H \"Content-Type: application\u002Fjson\" \\\n    -d '{\n        \"model\": \"\u003Cmodel_name>\",\n        \"prompt\": \"The capital of France is\",\n        \"max_tokens\": 32\n    }' | python -m json.tool\n",[1762],{"type":39,"tag":66,"props":1763,"children":1764},{"__ignoreMap":80},[1765,1773,1791,1798,1806,1841,1848,1856,1876,1901,1919,1927,1935,1943],{"type":39,"tag":86,"props":1766,"children":1767},{"class":88,"line":89},[1768],{"type":39,"tag":86,"props":1769,"children":1770},{"style":93},[1771],{"type":45,"value":1772},"# Health check\n",{"type":39,"tag":86,"props":1774,"children":1775},{"class":88,"line":99},[1776,1781,1786],{"type":39,"tag":86,"props":1777,"children":1778},{"style":103},[1779],{"type":45,"value":1780},"curl",{"type":39,"tag":86,"props":1782,"children":1783},{"style":120},[1784],{"type":45,"value":1785}," -s",{"type":39,"tag":86,"props":1787,"children":1788},{"style":120},[1789],{"type":45,"value":1790}," http:\u002F\u002Flocalhost:8000\u002Fhealth\n",{"type":39,"tag":86,"props":1792,"children":1793},{"class":88,"line":136},[1794],{"type":39,"tag":86,"props":1795,"children":1796},{"emptyLinePlaceholder":140},[1797],{"type":45,"value":143},{"type":39,"tag":86,"props":1799,"children":1800},{"class":88,"line":146},[1801],{"type":39,"tag":86,"props":1802,"children":1803},{"style":93},[1804],{"type":45,"value":1805},"# List models\n",{"type":39,"tag":86,"props":1807,"children":1808},{"class":88,"line":155},[1809,1813,1817,1822,1827,1832,1836],{"type":39,"tag":86,"props":1810,"children":1811},{"style":103},[1812],{"type":45,"value":1780},{"type":39,"tag":86,"props":1814,"children":1815},{"style":120},[1816],{"type":45,"value":1785},{"type":39,"tag":86,"props":1818,"children":1819},{"style":120},[1820],{"type":45,"value":1821}," http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels",{"type":39,"tag":86,"props":1823,"children":1824},{"style":419},[1825],{"type":45,"value":1826}," |",{"type":39,"tag":86,"props":1828,"children":1829},{"style":103},[1830],{"type":45,"value":1831}," python",{"type":39,"tag":86,"props":1833,"children":1834},{"style":120},[1835],{"type":45,"value":1234},{"type":39,"tag":86,"props":1837,"children":1838},{"style":120},[1839],{"type":45,"value":1840}," json.tool\n",{"type":39,"tag":86,"props":1842,"children":1843},{"class":88,"line":205},[1844],{"type":39,"tag":86,"props":1845,"children":1846},{"emptyLinePlaceholder":140},[1847],{"type":45,"value":143},{"type":39,"tag":86,"props":1849,"children":1850},{"class":88,"line":213},[1851],{"type":39,"tag":86,"props":1852,"children":1853},{"style":93},[1854],{"type":45,"value":1855},"# Test generation\n",{"type":39,"tag":86,"props":1857,"children":1858},{"class":88,"line":222},[1859,1863,1867,1872],{"type":39,"tag":86,"props":1860,"children":1861},{"style":103},[1862],{"type":45,"value":1780},{"type":39,"tag":86,"props":1864,"children":1865},{"style":120},[1866],{"type":45,"value":1785},{"type":39,"tag":86,"props":1868,"children":1869},{"style":120},[1870],{"type":45,"value":1871}," http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fcompletions",{"type":39,"tag":86,"props":1873,"children":1874},{"style":109},[1875],{"type":45,"value":1244},{"type":39,"tag":86,"props":1877,"children":1878},{"class":88,"line":251},[1879,1884,1888,1893,1897],{"type":39,"tag":86,"props":1880,"children":1881},{"style":120},[1882],{"type":45,"value":1883},"    -H",{"type":39,"tag":86,"props":1885,"children":1886},{"style":419},[1887],{"type":45,"value":422},{"type":39,"tag":86,"props":1889,"children":1890},{"style":120},[1891],{"type":45,"value":1892},"Content-Type: application\u002Fjson",{"type":39,"tag":86,"props":1894,"children":1895},{"style":419},[1896],{"type":45,"value":106},{"type":39,"tag":86,"props":1898,"children":1899},{"style":109},[1900],{"type":45,"value":1244},{"type":39,"tag":86,"props":1902,"children":1903},{"class":88,"line":259},[1904,1909,1914],{"type":39,"tag":86,"props":1905,"children":1906},{"style":120},[1907],{"type":45,"value":1908},"    -d",{"type":39,"tag":86,"props":1910,"children":1911},{"style":419},[1912],{"type":45,"value":1913}," '",{"type":39,"tag":86,"props":1915,"children":1916},{"style":120},[1917],{"type":45,"value":1918},"{\n",{"type":39,"tag":86,"props":1920,"children":1921},{"class":88,"line":268},[1922],{"type":39,"tag":86,"props":1923,"children":1924},{"style":120},[1925],{"type":45,"value":1926},"        \"model\": \"\u003Cmodel_name>\",\n",{"type":39,"tag":86,"props":1928,"children":1929},{"class":88,"line":289},[1930],{"type":39,"tag":86,"props":1931,"children":1932},{"style":120},[1933],{"type":45,"value":1934},"        \"prompt\": \"The capital of France is\",\n",{"type":39,"tag":86,"props":1936,"children":1937},{"class":88,"line":297},[1938],{"type":39,"tag":86,"props":1939,"children":1940},{"style":120},[1941],{"type":45,"value":1942},"        \"max_tokens\": 32\n",{"type":39,"tag":86,"props":1944,"children":1945},{"class":88,"line":306},[1946,1951,1956,1960,1964,1968],{"type":39,"tag":86,"props":1947,"children":1948},{"style":120},[1949],{"type":45,"value":1950},"    }",{"type":39,"tag":86,"props":1952,"children":1953},{"style":419},[1954],{"type":45,"value":1955},"'",{"type":39,"tag":86,"props":1957,"children":1958},{"style":419},[1959],{"type":45,"value":1826},{"type":39,"tag":86,"props":1961,"children":1962},{"style":103},[1963],{"type":45,"value":1831},{"type":39,"tag":86,"props":1965,"children":1966},{"style":120},[1967],{"type":45,"value":1234},{"type":39,"tag":86,"props":1969,"children":1970},{"style":120},[1971],{"type":45,"value":1840},{"type":39,"tag":48,"props":1973,"children":1974},{},[1975],{"type":45,"value":1976},"All checks must pass before reporting success to the user.",{"type":39,"tag":375,"props":1978,"children":1980},{"id":1979},"_5b-benchmark-throughputlatency-optional",[1981],{"type":45,"value":1982},"5b. Benchmark throughput\u002Flatency (optional)",{"type":39,"tag":48,"props":1984,"children":1985},{},[1986,1988,1993,1995,2001,2003,2009,2011,2017,2019,2025],{"type":45,"value":1987},"If the user asks to benchmark, measure throughput\u002Flatency, or compare precisions,\nuse ",{"type":39,"tag":480,"props":1989,"children":1990},{},[1991],{"type":45,"value":1992},"AIPerf",{"type":45,"value":1994}," (Apache-2.0, OpenAI-compatible client benchmark). See\n",{"type":39,"tag":66,"props":1996,"children":1998},{"className":1997},[],[1999],{"type":45,"value":2000},"references\u002Fbenchmarking.md",{"type":45,"value":2002}," for install, the pre-benchmark coherence gate, the\n",{"type":39,"tag":66,"props":2004,"children":2006},{"className":2005},[],[2007],{"type":45,"value":2008},"aiperf profile",{"type":45,"value":2010}," flags (notably ",{"type":39,"tag":66,"props":2012,"children":2014},{"className":2013},[],[2015],{"type":45,"value":2016},"--extra-inputs ignore_eos:true",{"type":45,"value":2018},"), suggested\ntoken shapes, and how to read ",{"type":39,"tag":66,"props":2020,"children":2022},{"className":2021},[],[2023],{"type":45,"value":2024},"profile_export_aiperf.json",{"type":45,"value":583},{"type":39,"tag":375,"props":2027,"children":2029},{"id":2028},"_6-remote-deployment-sshslurm",[2030],{"type":45,"value":2031},"6. Remote deployment (SSH\u002FSLURM)",{"type":39,"tag":48,"props":2033,"children":2034},{},[2035,2037,2043,2044,2050,2052,2058],{"type":45,"value":2036},"If a cluster config exists (",{"type":39,"tag":66,"props":2038,"children":2040},{"className":2039},[],[2041],{"type":45,"value":2042},"~\u002F.config\u002Fmodelopt\u002Fclusters.yaml",{"type":45,"value":502},{"type":39,"tag":66,"props":2045,"children":2047},{"className":2046},[],[2048],{"type":45,"value":2049},".agents\u002Fclusters.yaml",{"type":45,"value":2051},", or ",{"type":39,"tag":66,"props":2053,"children":2055},{"className":2054},[],[2056],{"type":45,"value":2057},".claude\u002Fclusters.yaml",{"type":45,"value":2059},"), or the user mentions running on a remote machine:",{"type":39,"tag":2061,"props":2062,"children":2064},"ol",{"start":2063},0,[2065,2088,2158,2261,2356,2411],{"type":39,"tag":476,"props":2066,"children":2067},{},[2068,2073,2075,2081,2083],{"type":39,"tag":480,"props":2069,"children":2070},{},[2071],{"type":45,"value":2072},"Check container registry auth",{"type":45,"value":2074}," — before submitting any SLURM job with a container image, verify credentials exist on the cluster per the common skill's ",{"type":39,"tag":66,"props":2076,"children":2078},{"className":2077},[],[2079],{"type":45,"value":2080},"slurm-setup.md",{"type":45,"value":2082}," section 6. If credentials are missing for the image's registry, ask the user to fix auth or switch to an image on an authenticated registry (e.g., NGC). ",{"type":39,"tag":480,"props":2084,"children":2085},{},[2086],{"type":45,"value":2087},"Do not submit until auth is confirmed.",{"type":39,"tag":476,"props":2089,"children":2090},{},[2091,2096,2098,2104,2106],{"type":39,"tag":480,"props":2092,"children":2093},{},[2094],{"type":45,"value":2095},"Source remote utilities:",{"type":45,"value":2097}," Load the common skill, then resolve\n",{"type":39,"tag":66,"props":2099,"children":2101},{"className":2100},[],[2102],{"type":45,"value":2103},"remote_exec.sh",{"type":45,"value":2105}," from that skill's root.",{"type":39,"tag":75,"props":2107,"children":2109},{"className":77,"code":2108,"language":79,"meta":80,"style":80},"source \"\u003Ccommon-skill-dir>\u002Fremote_exec.sh\"\nremote_load_cluster\nremote_check_ssh\nremote_detect_env\n",[2110],{"type":39,"tag":66,"props":2111,"children":2112},{"__ignoreMap":80},[2113,2134,2142,2150],{"type":39,"tag":86,"props":2114,"children":2115},{"class":88,"line":89},[2116,2121,2125,2130],{"type":39,"tag":86,"props":2117,"children":2118},{"style":645},[2119],{"type":45,"value":2120},"source",{"type":39,"tag":86,"props":2122,"children":2123},{"style":419},[2124],{"type":45,"value":422},{"type":39,"tag":86,"props":2126,"children":2127},{"style":120},[2128],{"type":45,"value":2129},"\u003Ccommon-skill-dir>\u002Fremote_exec.sh",{"type":39,"tag":86,"props":2131,"children":2132},{"style":419},[2133],{"type":45,"value":662},{"type":39,"tag":86,"props":2135,"children":2136},{"class":88,"line":99},[2137],{"type":39,"tag":86,"props":2138,"children":2139},{"style":103},[2140],{"type":45,"value":2141},"remote_load_cluster\n",{"type":39,"tag":86,"props":2143,"children":2144},{"class":88,"line":136},[2145],{"type":39,"tag":86,"props":2146,"children":2147},{"style":103},[2148],{"type":45,"value":2149},"remote_check_ssh\n",{"type":39,"tag":86,"props":2151,"children":2152},{"class":88,"line":146},[2153],{"type":39,"tag":86,"props":2154,"children":2155},{"style":103},[2156],{"type":45,"value":2157},"remote_detect_env\n",{"type":39,"tag":476,"props":2159,"children":2160},{},[2161,2166,2168,2172,2174,2180,2182],{"type":39,"tag":480,"props":2162,"children":2163},{},[2164],{"type":45,"value":2165},"Sync the checkpoint",{"type":45,"value":2167}," (only if it was produced locally):",{"type":39,"tag":2169,"props":2170,"children":2171},"br",{},[],{"type":45,"value":2173},"If the checkpoint path is a remote\u002Fabsolute path (e.g., from a prior PTQ run on the cluster), skip sync — it's already there. Verify with ",{"type":39,"tag":66,"props":2175,"children":2177},{"className":2176},[],[2178],{"type":45,"value":2179},"remote_run \"ls \u003Ccheckpoint_path>\u002Fconfig.json\"",{"type":45,"value":2181},". Only sync if the checkpoint is local:",{"type":39,"tag":75,"props":2183,"children":2185},{"className":77,"code":2184,"language":79,"meta":80,"style":80},"remote_sync_to \u003Clocal_checkpoint_path> \u003Csession_id>\u002F\u003Cmodel>\u002Fcheckpoints\u002F\n",[2186],{"type":39,"tag":66,"props":2187,"children":2188},{"__ignoreMap":80},[2189],{"type":39,"tag":86,"props":2190,"children":2191},{"class":88,"line":89},[2192,2197,2201,2206,2210,2214,2218,2223,2228,2232,2237,2242,2247,2252,2256],{"type":39,"tag":86,"props":2193,"children":2194},{"style":103},[2195],{"type":45,"value":2196},"remote_sync_to",{"type":39,"tag":86,"props":2198,"children":2199},{"style":419},[2200],{"type":45,"value":608},{"type":39,"tag":86,"props":2202,"children":2203},{"style":120},[2204],{"type":45,"value":2205},"local_checkpoint_pat",{"type":39,"tag":86,"props":2207,"children":2208},{"style":109},[2209],{"type":45,"value":618},{"type":39,"tag":86,"props":2211,"children":2212},{"style":419},[2213],{"type":45,"value":623},{"type":39,"tag":86,"props":2215,"children":2216},{"style":419},[2217],{"type":45,"value":608},{"type":39,"tag":86,"props":2219,"children":2220},{"style":120},[2221],{"type":45,"value":2222},"session_i",{"type":39,"tag":86,"props":2224,"children":2225},{"style":109},[2226],{"type":45,"value":2227},"d",{"type":39,"tag":86,"props":2229,"children":2230},{"style":419},[2231],{"type":45,"value":623},{"type":39,"tag":86,"props":2233,"children":2234},{"style":120},[2235],{"type":45,"value":2236},"\u002F",{"type":39,"tag":86,"props":2238,"children":2239},{"style":419},[2240],{"type":45,"value":2241},"\u003C",{"type":39,"tag":86,"props":2243,"children":2244},{"style":120},[2245],{"type":45,"value":2246},"mode",{"type":39,"tag":86,"props":2248,"children":2249},{"style":109},[2250],{"type":45,"value":2251},"l",{"type":39,"tag":86,"props":2253,"children":2254},{"style":419},[2255],{"type":45,"value":623},{"type":39,"tag":86,"props":2257,"children":2258},{"style":120},[2259],{"type":45,"value":2260},"\u002Fcheckpoints\u002F\n",{"type":39,"tag":476,"props":2262,"children":2263},{},[2264,2269],{"type":39,"tag":480,"props":2265,"children":2266},{},[2267],{"type":45,"value":2268},"Deploy based on remote environment:",{"type":39,"tag":472,"props":2270,"children":2271},{},[2272,2311],{"type":39,"tag":476,"props":2273,"children":2274},{},[2275,2280,2282,2287,2289,2295,2297,2302,2304,2310],{"type":39,"tag":480,"props":2276,"children":2277},{},[2278],{"type":45,"value":2279},"SLURM",{"type":45,"value":2281}," — see the common skill's ",{"type":39,"tag":66,"props":2283,"children":2285},{"className":2284},[],[2286],{"type":45,"value":2080},{"type":45,"value":2288}," for job script templates (container setup, account\u002Fpartition discovery). The server command inside the container is the same as Step 4 (e.g., ",{"type":39,"tag":66,"props":2290,"children":2292},{"className":2291},[],[2293],{"type":45,"value":2294},"python -m vllm.entrypoints.openai.api_server --model \u003Cpath> --quantization modelopt",{"type":45,"value":2296},"). After submitting, register the job and set up monitoring per the ",{"type":39,"tag":480,"props":2298,"children":2299},{},[2300],{"type":45,"value":2301},"monitor skill",{"type":45,"value":2303},". Get the node hostname from ",{"type":39,"tag":66,"props":2305,"children":2307},{"className":2306},[],[2308],{"type":45,"value":2309},"squeue -j $JOBID -o %N",{"type":45,"value":583},{"type":39,"tag":476,"props":2312,"children":2313},{},[2314,2319,2321,2327,2329],{"type":39,"tag":480,"props":2315,"children":2316},{},[2317],{"type":45,"value":2318},"Bare metal \u002F Docker",{"type":45,"value":2320}," — use ",{"type":39,"tag":66,"props":2322,"children":2324},{"className":2323},[],[2325],{"type":45,"value":2326},"remote_run",{"type":45,"value":2328}," to start the server directly:",{"type":39,"tag":75,"props":2330,"children":2332},{"className":77,"code":2331,"language":79,"meta":80,"style":80},"remote_run \"nohup python -m vllm.entrypoints.openai.api_server --model \u003Cpath> --port 8000 > deploy.log 2>&1 &\"\n",[2333],{"type":39,"tag":66,"props":2334,"children":2335},{"__ignoreMap":80},[2336],{"type":39,"tag":86,"props":2337,"children":2338},{"class":88,"line":89},[2339,2343,2347,2352],{"type":39,"tag":86,"props":2340,"children":2341},{"style":103},[2342],{"type":45,"value":2326},{"type":39,"tag":86,"props":2344,"children":2345},{"style":419},[2346],{"type":45,"value":422},{"type":39,"tag":86,"props":2348,"children":2349},{"style":120},[2350],{"type":45,"value":2351},"nohup python -m vllm.entrypoints.openai.api_server --model \u003Cpath> --port 8000 > deploy.log 2>&1 &",{"type":39,"tag":86,"props":2353,"children":2354},{"style":419},[2355],{"type":45,"value":662},{"type":39,"tag":476,"props":2357,"children":2358},{},[2359,2364],{"type":39,"tag":480,"props":2360,"children":2361},{},[2362],{"type":45,"value":2363},"Verify remotely:",{"type":39,"tag":75,"props":2365,"children":2367},{"className":77,"code":2366,"language":79,"meta":80,"style":80},"remote_run \"curl -s http:\u002F\u002Flocalhost:8000\u002Fhealth\"\nremote_run \"curl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels\"\n",[2368],{"type":39,"tag":66,"props":2369,"children":2370},{"__ignoreMap":80},[2371,2391],{"type":39,"tag":86,"props":2372,"children":2373},{"class":88,"line":89},[2374,2378,2382,2387],{"type":39,"tag":86,"props":2375,"children":2376},{"style":103},[2377],{"type":45,"value":2326},{"type":39,"tag":86,"props":2379,"children":2380},{"style":419},[2381],{"type":45,"value":422},{"type":39,"tag":86,"props":2383,"children":2384},{"style":120},[2385],{"type":45,"value":2386},"curl -s http:\u002F\u002Flocalhost:8000\u002Fhealth",{"type":39,"tag":86,"props":2388,"children":2389},{"style":419},[2390],{"type":45,"value":662},{"type":39,"tag":86,"props":2392,"children":2393},{"class":88,"line":99},[2394,2398,2402,2407],{"type":39,"tag":86,"props":2395,"children":2396},{"style":103},[2397],{"type":45,"value":2326},{"type":39,"tag":86,"props":2399,"children":2400},{"style":419},[2401],{"type":45,"value":422},{"type":39,"tag":86,"props":2403,"children":2404},{"style":120},[2405],{"type":45,"value":2406},"curl -s http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels",{"type":39,"tag":86,"props":2408,"children":2409},{"style":419},[2410],{"type":45,"value":662},{"type":39,"tag":476,"props":2412,"children":2413},{},[2414,2419,2421,2427],{"type":39,"tag":480,"props":2415,"children":2416},{},[2417],{"type":45,"value":2418},"Report the endpoint",{"type":45,"value":2420}," — include the remote hostname and port so the user can connect (e.g., ",{"type":39,"tag":66,"props":2422,"children":2424},{"className":2423},[],[2425],{"type":45,"value":2426},"http:\u002F\u002F\u003Cnode_hostname>:8000",{"type":45,"value":2428},"). For SLURM, note that the port is only reachable from within the cluster network.",{"type":39,"tag":48,"props":2430,"children":2431},{},[2432],{"type":45,"value":2433},"For NEL-managed deployment (evaluation with self-deployment), use the evaluation skill instead — NEL handles SLURM container deployment, health checks, and teardown automatically.",{"type":39,"tag":54,"props":2435,"children":2437},{"id":2436},"error-handling",[2438],{"type":45,"value":2439},"Error Handling",{"type":39,"tag":704,"props":2441,"children":2442},{},[2443,2464],{"type":39,"tag":708,"props":2444,"children":2445},{},[2446],{"type":39,"tag":712,"props":2447,"children":2448},{},[2449,2454,2459],{"type":39,"tag":716,"props":2450,"children":2451},{},[2452],{"type":45,"value":2453},"Error",{"type":39,"tag":716,"props":2455,"children":2456},{},[2457],{"type":45,"value":2458},"Cause",{"type":39,"tag":716,"props":2460,"children":2461},{},[2462],{"type":45,"value":2463},"Fix",{"type":39,"tag":732,"props":2465,"children":2466},{},[2467,2536,2566,2588,2626,2650],{"type":39,"tag":712,"props":2468,"children":2469},{},[2470,2503,2508],{"type":39,"tag":739,"props":2471,"children":2472},{},[2473,2479,2481,2486,2488,2494,2495,2501],{"type":39,"tag":66,"props":2474,"children":2476},{"className":2475},[],[2477],{"type":45,"value":2478},"CUDA error: an illegal memory access",{"type":45,"value":2480}," on an ",{"type":39,"tag":480,"props":2482,"children":2483},{},[2484],{"type":45,"value":2485},"NVFP4 MoE",{"type":45,"value":2487}," (",{"type":39,"tag":66,"props":2489,"children":2491},{"className":2490},[],[2492],{"type":45,"value":2493},"trtllm_fused_moe_dev_kernel.cu",{"type":45,"value":502},{"type":39,"tag":66,"props":2496,"children":2498},{"className":2497},[],[2499],{"type":45,"value":2500},"deepgemm",{"type":45,"value":2502},")",{"type":39,"tag":739,"props":2504,"children":2505},{},[2506],{"type":45,"value":2507},"Fused-MoE FP4 kernel fault on long-context loads; engine dies, all requests 500",{"type":39,"tag":739,"props":2509,"children":2510},{},[2511,2513,2519,2521,2527,2529,2534],{"type":45,"value":2512},"Try ",{"type":39,"tag":66,"props":2514,"children":2516},{"className":2515},[],[2517],{"type":45,"value":2518},"VLLM_USE_FLASHINFER_MOE_FP4=1",{"type":45,"value":2520}," + ",{"type":39,"tag":66,"props":2522,"children":2524},{"className":2523},[],[2525],{"type":45,"value":2526},"VLLM_FLASHINFER_MOE_BACKEND=throughput",{"type":45,"value":2528},". ",{"type":39,"tag":480,"props":2530,"children":2531},{},[2532],{"type":45,"value":2533},"Not always a fix",{"type":45,"value":2535}," — on DeepSeek-V4 it moved the fault from TRT-LLM to DeepGEMM. It also makes quantized vs baseline not kernel-matched; record that when reporting deltas.",{"type":39,"tag":712,"props":2537,"children":2538},{},[2539,2548,2553],{"type":39,"tag":739,"props":2540,"children":2541},{},[2542],{"type":39,"tag":66,"props":2543,"children":2545},{"className":2544},[],[2546],{"type":45,"value":2547},"CUDA out of memory",{"type":39,"tag":739,"props":2549,"children":2550},{},[2551],{"type":45,"value":2552},"Model too large for GPU(s)",{"type":39,"tag":739,"props":2554,"children":2555},{},[2556,2558,2564],{"type":45,"value":2557},"Increase ",{"type":39,"tag":66,"props":2559,"children":2561},{"className":2560},[],[2562],{"type":45,"value":2563},"--tensor-parallel-size",{"type":45,"value":2565}," or use a smaller model",{"type":39,"tag":712,"props":2567,"children":2568},{},[2569,2578,2583],{"type":39,"tag":739,"props":2570,"children":2571},{},[2572],{"type":39,"tag":66,"props":2573,"children":2575},{"className":2574},[],[2576],{"type":45,"value":2577},"quantization=\"modelopt\" not recognized",{"type":39,"tag":739,"props":2579,"children":2580},{},[2581],{"type":45,"value":2582},"vLLM\u002FSGLang version too old",{"type":39,"tag":739,"props":2584,"children":2585},{},[2586],{"type":45,"value":2587},"Upgrade: vLLM >= 0.10.1, SGLang >= 0.4.10",{"type":39,"tag":712,"props":2589,"children":2590},{},[2591,2600,2605],{"type":39,"tag":739,"props":2592,"children":2593},{},[2594],{"type":39,"tag":66,"props":2595,"children":2597},{"className":2596},[],[2598],{"type":45,"value":2599},"hf_quant_config.json not found",{"type":39,"tag":739,"props":2601,"children":2602},{},[2603],{"type":45,"value":2604},"Not a ModelOpt-exported checkpoint",{"type":39,"tag":739,"props":2606,"children":2607},{},[2608,2610,2616,2618,2624],{"type":45,"value":2609},"Re-export with ",{"type":39,"tag":66,"props":2611,"children":2613},{"className":2612},[],[2614],{"type":45,"value":2615},"export_hf_checkpoint()",{"type":45,"value":2617},", or remove ",{"type":39,"tag":66,"props":2619,"children":2621},{"className":2620},[],[2622],{"type":45,"value":2623},"--quantization",{"type":45,"value":2625}," flag",{"type":39,"tag":712,"props":2627,"children":2628},{},[2629,2640,2645],{"type":39,"tag":739,"props":2630,"children":2631},{},[2632,2638],{"type":39,"tag":66,"props":2633,"children":2635},{"className":2634},[],[2636],{"type":45,"value":2637},"Connection refused",{"type":45,"value":2639}," on health check",{"type":39,"tag":739,"props":2641,"children":2642},{},[2643],{"type":45,"value":2644},"Server still starting",{"type":39,"tag":739,"props":2646,"children":2647},{},[2648],{"type":45,"value":2649},"Wait 30-60s for large models; check logs for errors",{"type":39,"tag":712,"props":2651,"children":2652},{},[2653,2662,2667],{"type":39,"tag":739,"props":2654,"children":2655},{},[2656],{"type":39,"tag":66,"props":2657,"children":2659},{"className":2658},[],[2660],{"type":45,"value":2661},"modelopt_fp4 not supported",{"type":39,"tag":739,"props":2663,"children":2664},{},[2665],{"type":45,"value":2666},"Framework doesn't support FP4 for this model",{"type":39,"tag":739,"props":2668,"children":2669},{},[2670,2672],{"type":45,"value":2671},"Check support matrix in ",{"type":39,"tag":66,"props":2673,"children":2675},{"className":2674},[],[2676],{"type":45,"value":829},{"type":39,"tag":54,"props":2678,"children":2680},{"id":2679},"unsupported-models",[2681],{"type":45,"value":2682},"Unsupported Models",{"type":39,"tag":48,"props":2684,"children":2685},{},[2686,2688,2693,2695,2701,2703,2708],{"type":45,"value":2687},"If the model is not in the validated support matrix (",{"type":39,"tag":66,"props":2689,"children":2691},{"className":2690},[],[2692],{"type":45,"value":829},{"type":45,"value":2694},"), deployment may fail due to weight key mismatches, missing architecture mappings, or quantized\u002Funquantized layer confusion. Read ",{"type":39,"tag":66,"props":2696,"children":2698},{"className":2697},[],[2699],{"type":45,"value":2700},"references\u002Funsupported-models.md",{"type":45,"value":2702}," for the iterative debug loop: ",{"type":39,"tag":480,"props":2704,"children":2705},{},[2706],{"type":45,"value":2707},"run → read error → diagnose → patch framework source → re-run",{"type":45,"value":2709},". For kernel-level issues, escalate to the framework team rather than attempting fixes.",{"type":39,"tag":54,"props":2711,"children":2713},{"id":2712},"success-criteria",[2714],{"type":45,"value":2715},"Success Criteria",{"type":39,"tag":2061,"props":2717,"children":2718},{},[2719,2732,2743,2748,2753],{"type":39,"tag":476,"props":2720,"children":2721},{},[2722,2724,2730],{"type":45,"value":2723},"Server process is running and healthy (",{"type":39,"tag":66,"props":2725,"children":2727},{"className":2726},[],[2728],{"type":45,"value":2729},"\u002Fhealth",{"type":45,"value":2731}," returns 200)",{"type":39,"tag":476,"props":2733,"children":2734},{},[2735,2737],{"type":45,"value":2736},"Model is listed at ",{"type":39,"tag":66,"props":2738,"children":2740},{"className":2739},[],[2741],{"type":45,"value":2742},"\u002Fv1\u002Fmodels",{"type":39,"tag":476,"props":2744,"children":2745},{},[2746],{"type":45,"value":2747},"Test generation produces coherent output",{"type":39,"tag":476,"props":2749,"children":2750},{},[2751],{"type":45,"value":2752},"Server URL and port are reported to the user",{"type":39,"tag":476,"props":2754,"children":2755},{},[2756],{"type":45,"value":2757},"If benchmarking was requested, throughput\u002Flatency numbers are reported",{"type":39,"tag":2759,"props":2760,"children":2761},"style",{},[2762],{"type":45,"value":2763},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2765,"total":136},[2766,2781,2788],{"slug":2767,"name":2767,"fn":2768,"description":2769,"org":2770,"tags":2771,"stars":22,"repoUrl":23,"updatedAt":2780},"day0-release","automate quantized checkpoint releases","Deterministic end-to-end driver for day-0 quantized-checkpoint releases — chains PTQ → evaluation → comparison with enforced gates between stages (the evaluation stage deploys the checkpoint itself), and returns a publish decision (ACCEPT \u002F REGRESSION \u002F ANOMALOUS \u002F INFEASIBLE). Use when the user asks to \"release a model at day-0\", \"quantize and validate model X is within N% of baseline and tell me if it's publishable\", or \"run the full day-0 workflow\". Do NOT use for single-stage requests — quantizing only (use ptq), serving only (use deployment), evaluating only (use evaluation), or comparing two existing runs (use compare-results).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2772,2775,2778,2779],{"name":2773,"slug":2774,"type":15},"Automation","automation",{"name":2776,"slug":2777,"type":15},"CI\u002FCD","ci-cd",{"name":20,"slug":4,"type":15},{"name":9,"slug":8,"type":15},"2026-08-31T09:18:55.832925",{"slug":4,"name":4,"fn":5,"description":6,"org":2782,"tags":2783,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2784,2785,2786,2787],{"name":20,"slug":4,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":17,"slug":18,"type":15},{"slug":2789,"name":2789,"fn":2790,"description":2791,"org":2792,"tags":2793,"stars":22,"repoUrl":23,"updatedAt":2802},"evaluation","evaluate LLM accuracy with NeMo","Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on \"evaluate model\", \"benchmark accuracy\", \"run MMLU\", \"evaluate quantized model\", \"run nel\". Handles deployment, config generation, and evaluation execution. Not for quantizing models (use ptq), deploying\u002Fserving models (use deployment), or comparing completed baseline-vs-quantized results (use compare-results).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2794,2797,2800,2801],{"name":2795,"slug":2796,"type":15},"Benchmarking","benchmarking",{"name":2798,"slug":2799,"type":15},"Evals","evals",{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},"2026-08-31T09:19:35.777526",{"items":2804,"total":2956},[2805,2823,2838,2847,2859,2873,2886,2900,2913,2924,2938,2947],{"slug":2806,"name":2806,"fn":2807,"description":2808,"org":2809,"tags":2810,"stars":2820,"repoUrl":2821,"updatedAt":2822},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2811,2814,2817],{"name":2812,"slug":2813,"type":15},"Documentation","documentation",{"name":2815,"slug":2816,"type":15},"MCP","mcp",{"name":2818,"slug":2819,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-08-25T03:29:57.273192",{"slug":2824,"name":2824,"fn":2825,"description":2826,"org":2827,"tags":2828,"stars":2835,"repoUrl":2836,"updatedAt":2837},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2829,2832,2833],{"name":2830,"slug":2831,"type":15},"Containers","containers",{"name":20,"slug":4,"type":15},{"name":2834,"slug":877,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2839,"name":2839,"fn":2840,"description":2841,"org":2842,"tags":2843,"stars":2835,"repoUrl":2836,"updatedAt":2846},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2844,2845],{"name":2776,"slug":2777,"type":15},{"name":20,"slug":4,"type":15},"2026-07-14T05:25:59.97109",{"slug":2848,"name":2848,"fn":2849,"description":2850,"org":2851,"tags":2852,"stars":2835,"repoUrl":2836,"updatedAt":2858},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2853,2854,2855],{"name":2776,"slug":2777,"type":15},{"name":20,"slug":4,"type":15},{"name":2856,"slug":2857,"type":15},"GitHub","github","2026-08-28T14:38:16.959248",{"slug":2860,"name":2860,"fn":2861,"description":2862,"org":2863,"tags":2864,"stars":2835,"repoUrl":2836,"updatedAt":2872},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2865,2868,2869],{"name":2866,"slug":2867,"type":15},"Debugging","debugging",{"name":2856,"slug":2857,"type":15},{"name":2870,"slug":2871,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2874,"name":2874,"fn":2875,"description":2876,"org":2877,"tags":2878,"stars":2835,"repoUrl":2836,"updatedAt":2885},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2879,2882],{"name":2880,"slug":2881,"type":15},"Best Practices","best-practices",{"name":2883,"slug":2884,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2887,"name":2887,"fn":2888,"description":2889,"org":2890,"tags":2891,"stars":2835,"repoUrl":2836,"updatedAt":2899},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel, including the mechanical steps for transferring an existing pretrain_gpt.py launch script.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2892,2895,2898],{"name":2893,"slug":2894,"type":15},"Machine Learning","machine-learning",{"name":2896,"slug":2897,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-08-21T03:36:57.470256",{"slug":2901,"name":2901,"fn":2902,"description":2903,"org":2904,"tags":2905,"stars":2835,"repoUrl":2836,"updatedAt":2912},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2906,2909],{"name":2907,"slug":2908,"type":15},"QA","qa",{"name":2910,"slug":2911,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":2914,"name":2914,"fn":2915,"description":2916,"org":2917,"tags":2918,"stars":2835,"repoUrl":2836,"updatedAt":2923},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2919,2920],{"name":20,"slug":4,"type":15},{"name":2921,"slug":2922,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":2925,"name":2925,"fn":2926,"description":2927,"org":2928,"tags":2929,"stars":2835,"repoUrl":2836,"updatedAt":2937},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2930,2933,2934],{"name":2931,"slug":2932,"type":15},"Code Review","code-review",{"name":2856,"slug":2857,"type":15},{"name":2935,"slug":2936,"type":15},"Pull Requests","pull-requests","2026-08-25T03:29:16.211287",{"slug":2939,"name":2939,"fn":2940,"description":2941,"org":2942,"tags":2943,"stars":2835,"repoUrl":2836,"updatedAt":2946},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2944,2945],{"name":2907,"slug":2908,"type":15},{"name":2910,"slug":2911,"type":15},"2026-07-14T05:25:54.928983",{"slug":2948,"name":2948,"fn":2949,"description":2950,"org":2951,"tags":2952,"stars":2835,"repoUrl":2836,"updatedAt":2955},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2953,2954],{"name":2773,"slug":2774,"type":15},{"name":2776,"slug":2777,"type":15},"2026-07-30T05:29:03.275638",563]