[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-vllm-setup":3,"mdc-u483c6-key":34,"related-repo-nvidia-vllm-setup":909,"related-org-nvidia-vllm-setup":1013},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"vllm-setup","deploy vLLM inference servers","Deploy a vLLM inference server on an NVIDIA DGX Station GB300 with validated container, GPU targeting, and tuning parameters. Use when the user asks to serve a model with vLLM, start a vLLM endpoint, or set up OpenAI-compatible inference on DGX Station.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"LLM","llm","tag",{"name":17,"slug":18,"type":15},"Deployment","deployment",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"AI Infrastructure","ai-infrastructure",1144,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fdgx-spark-playbooks","2026-07-14T05:31:00.271483",null,249,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"Collection of step-by-step playbooks for setting up AI\u002FML workloads on NVIDIA DGX Spark devices with Blackwell architecture.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fdgx-spark-playbooks\u002Ftree\u002FHEAD\u002Fnvidia\u002Fstation-ai-skills\u002Fassets\u002Fskills\u002Fvllm-setup","---\nname: vllm-setup\ndescription: Deploy a vLLM inference server on an NVIDIA DGX Station GB300 with validated container, GPU targeting, and tuning parameters. Use when the user asks to serve a model with vLLM, start a vLLM endpoint, or set up OpenAI-compatible inference on DGX Station.\nmetadata:\n  publisher: nvidia\n  hardware: DGX Station GB300\n---\n\n# vLLM Setup on DGX Station\n\nDeploy a vLLM inference server on DGX Station with validated configuration.\n\n## Steps\n\n1. **Find the GB300 GPU index.** Run:\n   ```bash\n   nvidia-smi --query-gpu=index,name --format=csv,noheader\n   ```\n   Identify the device index for the GB300 (typically device 1). Use this index for `--gpus` below. Do NOT use `--gpus all` — mixed coherency will cause CUDA failures.\n\n2. **Ask the user which model to serve.** If they don't have a preference, suggest:\n   - `nvidia\u002FQwen3-235B-A22B-NVFP4` — large MoE model, fits in 279 GB HBM\n   - `meta-llama\u002FLlama-3.1-70B-Instruct` — solid general-purpose model\n   - `Qwen\u002FQwen3-8B` — small model for testing\n\n3. **Check if the user has an HF_TOKEN.** Many models require HuggingFace authentication. The token must be passed inline with `-e HF_TOKEN=\"...\"` — do not rely on shell export in background Docker tasks.\n\n4. **Deploy the container.** Use this validated configuration:\n\n   ```bash\n   docker pull nvcr.io\u002Fnvidia\u002Fvllm:26.01-py3\n\n   docker run -d \\\n     --name vllm-server \\\n     --gpus '\"device=\u003CGB300_INDEX>\"' \\\n     --ipc host \\\n     --ulimit memlock=-1 \\\n     --ulimit stack=67108864 \\\n     -p 8000:8000 \\\n     -e HF_TOKEN=\"\u003CTOKEN>\" \\\n     -v \"$HOME\u002F.cache\u002Fhuggingface\u002Fhub:\u002Froot\u002F.cache\u002Fhuggingface\u002Fhub\" \\\n     nvcr.io\u002Fnvidia\u002Fvllm:26.01-py3 \\\n     vllm serve \"\u003CMODEL>\" \\\n       --max-model-len 32768 \\\n       --gpu-memory-utilization 0.9\n   ```\n\n   **Container version:** Use `nvcr.io\u002Fnvidia\u002Fvllm:26.01-py3`. Do NOT use 25.10 — it has a FlashInfer buffer overflow on DGX Station.\n\n5. **Wait for the server to be ready.** Monitor logs:\n   ```bash\n   docker logs -f vllm-server\n   ```\n   Wait for the line indicating the server is listening on port 8000.\n\n6. **Test the server:**\n   ```bash\n   curl http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions \\\n     -H \"Content-Type: application\u002Fjson\" \\\n     -d '{\n       \"model\": \"\u003CMODEL>\",\n       \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n       \"max_tokens\": 64\n     }'\n   ```\n\n7. **Report the result** to the user, including:\n   - Model loaded and serving on port 8000\n   - GPU memory utilization\n   - How to stop: `docker stop vllm-server && docker rm vllm-server`\n\n## Tuning parameters\n\nAdjust these based on the user's workload:\n\n| Parameter | Default | Agent workloads | Throughput workloads |\n|-----------|---------|-----------------|---------------------|\n| `--max-model-len` | 32768 | 32768-65536 | 8192-16384 |\n| `--gpu-memory-utilization` | 0.9 | 0.85-0.90 | 0.90-0.92 |\n| `--enable-prefix-caching` | off | Enable (multi-turn reuse) | Enable |\n| `--max-num-seqs` | default | 4-16 (lower latency) | 32+ (higher throughput) |\n",{"data":35,"body":38},{"name":4,"description":6,"metadata":36},{"publisher":8,"hardware":37},"DGX Station GB300",{"type":39,"children":40},"root",[41,50,56,63,746,752,757,903],{"type":42,"tag":43,"props":44,"children":46},"element","h1",{"id":45},"vllm-setup-on-dgx-station",[47],{"type":48,"value":49},"text","vLLM Setup on DGX Station",{"type":42,"tag":51,"props":52,"children":53},"p",{},[54],{"type":48,"value":55},"Deploy a vLLM inference server on DGX Station with validated configuration.",{"type":42,"tag":57,"props":58,"children":60},"h2",{"id":59},"steps",[61],{"type":48,"value":62},"Steps",{"type":42,"tag":64,"props":65,"children":66},"ol",{},[67,136,183,201,557,601,712],{"type":42,"tag":68,"props":69,"children":70},"li",{},[71,77,79,114,118,120,126,128,134],{"type":42,"tag":72,"props":73,"children":74},"strong",{},[75],{"type":48,"value":76},"Find the GB300 GPU index.",{"type":48,"value":78}," Run:",{"type":42,"tag":80,"props":81,"children":86},"pre",{"className":82,"code":83,"language":84,"meta":85,"style":85},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","nvidia-smi --query-gpu=index,name --format=csv,noheader\n","bash","",[87],{"type":42,"tag":88,"props":89,"children":90},"code",{"__ignoreMap":85},[91],{"type":42,"tag":92,"props":93,"children":96},"span",{"class":94,"line":95},"line",1,[97,103,109],{"type":42,"tag":92,"props":98,"children":100},{"style":99},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[101],{"type":48,"value":102},"nvidia-smi",{"type":42,"tag":92,"props":104,"children":106},{"style":105},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[107],{"type":48,"value":108}," --query-gpu=index,name",{"type":42,"tag":92,"props":110,"children":111},{"style":105},[112],{"type":48,"value":113}," --format=csv,noheader\n",{"type":42,"tag":115,"props":116,"children":117},"br",{},[],{"type":48,"value":119},"Identify the device index for the GB300 (typically device 1). Use this index for ",{"type":42,"tag":88,"props":121,"children":123},{"className":122},[],[124],{"type":48,"value":125},"--gpus",{"type":48,"value":127}," below. Do NOT use ",{"type":42,"tag":88,"props":129,"children":131},{"className":130},[],[132],{"type":48,"value":133},"--gpus all",{"type":48,"value":135}," — mixed coherency will cause CUDA failures.",{"type":42,"tag":68,"props":137,"children":138},{},[139,144,146],{"type":42,"tag":72,"props":140,"children":141},{},[142],{"type":48,"value":143},"Ask the user which model to serve.",{"type":48,"value":145}," If they don't have a preference, suggest:",{"type":42,"tag":147,"props":148,"children":149},"ul",{},[150,161,172],{"type":42,"tag":68,"props":151,"children":152},{},[153,159],{"type":42,"tag":88,"props":154,"children":156},{"className":155},[],[157],{"type":48,"value":158},"nvidia\u002FQwen3-235B-A22B-NVFP4",{"type":48,"value":160}," — large MoE model, fits in 279 GB HBM",{"type":42,"tag":68,"props":162,"children":163},{},[164,170],{"type":42,"tag":88,"props":165,"children":167},{"className":166},[],[168],{"type":48,"value":169},"meta-llama\u002FLlama-3.1-70B-Instruct",{"type":48,"value":171}," — solid general-purpose model",{"type":42,"tag":68,"props":173,"children":174},{},[175,181],{"type":42,"tag":88,"props":176,"children":178},{"className":177},[],[179],{"type":48,"value":180},"Qwen\u002FQwen3-8B",{"type":48,"value":182}," — small model for testing",{"type":42,"tag":68,"props":184,"children":185},{},[186,191,193,199],{"type":42,"tag":72,"props":187,"children":188},{},[189],{"type":48,"value":190},"Check if the user has an HF_TOKEN.",{"type":48,"value":192}," Many models require HuggingFace authentication. The token must be passed inline with ",{"type":42,"tag":88,"props":194,"children":196},{"className":195},[],[197],{"type":48,"value":198},"-e HF_TOKEN=\"...\"",{"type":48,"value":200}," — do not rely on shell export in background Docker tasks.",{"type":42,"tag":68,"props":202,"children":203},{},[204,209,211,539,542,547,549,555],{"type":42,"tag":72,"props":205,"children":206},{},[207],{"type":48,"value":208},"Deploy the container.",{"type":48,"value":210}," Use this validated configuration:",{"type":42,"tag":80,"props":212,"children":214},{"className":82,"code":213,"language":84,"meta":85,"style":85},"docker pull nvcr.io\u002Fnvidia\u002Fvllm:26.01-py3\n\ndocker run -d \\\n  --name vllm-server \\\n  --gpus '\"device=\u003CGB300_INDEX>\"' \\\n  --ipc host \\\n  --ulimit memlock=-1 \\\n  --ulimit stack=67108864 \\\n  -p 8000:8000 \\\n  -e HF_TOKEN=\"\u003CTOKEN>\" \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface\u002Fhub:\u002Froot\u002F.cache\u002Fhuggingface\u002Fhub\" \\\n  nvcr.io\u002Fnvidia\u002Fvllm:26.01-py3 \\\n  vllm serve \"\u003CMODEL>\" \\\n    --max-model-len 32768 \\\n    --gpu-memory-utilization 0.9\n",[215],{"type":42,"tag":88,"props":216,"children":217},{"__ignoreMap":85},[218,236,246,270,288,317,335,359,381,399,431,463,476,507,525],{"type":42,"tag":92,"props":219,"children":220},{"class":94,"line":95},[221,226,231],{"type":42,"tag":92,"props":222,"children":223},{"style":99},[224],{"type":48,"value":225},"docker",{"type":42,"tag":92,"props":227,"children":228},{"style":105},[229],{"type":48,"value":230}," pull",{"type":42,"tag":92,"props":232,"children":233},{"style":105},[234],{"type":48,"value":235}," nvcr.io\u002Fnvidia\u002Fvllm:26.01-py3\n",{"type":42,"tag":92,"props":237,"children":239},{"class":94,"line":238},2,[240],{"type":42,"tag":92,"props":241,"children":243},{"emptyLinePlaceholder":242},true,[244],{"type":48,"value":245},"\n",{"type":42,"tag":92,"props":247,"children":249},{"class":94,"line":248},3,[250,254,259,264],{"type":42,"tag":92,"props":251,"children":252},{"style":99},[253],{"type":48,"value":225},{"type":42,"tag":92,"props":255,"children":256},{"style":105},[257],{"type":48,"value":258}," run",{"type":42,"tag":92,"props":260,"children":261},{"style":105},[262],{"type":48,"value":263}," -d",{"type":42,"tag":92,"props":265,"children":267},{"style":266},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[268],{"type":48,"value":269}," \\\n",{"type":42,"tag":92,"props":271,"children":273},{"class":94,"line":272},4,[274,279,284],{"type":42,"tag":92,"props":275,"children":276},{"style":105},[277],{"type":48,"value":278},"  --name",{"type":42,"tag":92,"props":280,"children":281},{"style":105},[282],{"type":48,"value":283}," vllm-server",{"type":42,"tag":92,"props":285,"children":286},{"style":266},[287],{"type":48,"value":269},{"type":42,"tag":92,"props":289,"children":291},{"class":94,"line":290},5,[292,297,303,308,313],{"type":42,"tag":92,"props":293,"children":294},{"style":105},[295],{"type":48,"value":296},"  --gpus",{"type":42,"tag":92,"props":298,"children":300},{"style":299},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[301],{"type":48,"value":302}," '",{"type":42,"tag":92,"props":304,"children":305},{"style":105},[306],{"type":48,"value":307},"\"device=\u003CGB300_INDEX>\"",{"type":42,"tag":92,"props":309,"children":310},{"style":299},[311],{"type":48,"value":312},"'",{"type":42,"tag":92,"props":314,"children":315},{"style":266},[316],{"type":48,"value":269},{"type":42,"tag":92,"props":318,"children":320},{"class":94,"line":319},6,[321,326,331],{"type":42,"tag":92,"props":322,"children":323},{"style":105},[324],{"type":48,"value":325},"  --ipc",{"type":42,"tag":92,"props":327,"children":328},{"style":105},[329],{"type":48,"value":330}," host",{"type":42,"tag":92,"props":332,"children":333},{"style":266},[334],{"type":48,"value":269},{"type":42,"tag":92,"props":336,"children":338},{"class":94,"line":337},7,[339,344,349,355],{"type":42,"tag":92,"props":340,"children":341},{"style":105},[342],{"type":48,"value":343},"  --ulimit",{"type":42,"tag":92,"props":345,"children":346},{"style":105},[347],{"type":48,"value":348}," memlock=",{"type":42,"tag":92,"props":350,"children":352},{"style":351},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[353],{"type":48,"value":354},"-1",{"type":42,"tag":92,"props":356,"children":357},{"style":266},[358],{"type":48,"value":269},{"type":42,"tag":92,"props":360,"children":362},{"class":94,"line":361},8,[363,367,372,377],{"type":42,"tag":92,"props":364,"children":365},{"style":105},[366],{"type":48,"value":343},{"type":42,"tag":92,"props":368,"children":369},{"style":105},[370],{"type":48,"value":371}," stack=",{"type":42,"tag":92,"props":373,"children":374},{"style":351},[375],{"type":48,"value":376},"67108864",{"type":42,"tag":92,"props":378,"children":379},{"style":266},[380],{"type":48,"value":269},{"type":42,"tag":92,"props":382,"children":384},{"class":94,"line":383},9,[385,390,395],{"type":42,"tag":92,"props":386,"children":387},{"style":105},[388],{"type":48,"value":389},"  -p",{"type":42,"tag":92,"props":391,"children":392},{"style":105},[393],{"type":48,"value":394}," 8000:8000",{"type":42,"tag":92,"props":396,"children":397},{"style":266},[398],{"type":48,"value":269},{"type":42,"tag":92,"props":400,"children":402},{"class":94,"line":401},10,[403,408,413,418,423,427],{"type":42,"tag":92,"props":404,"children":405},{"style":105},[406],{"type":48,"value":407},"  -e",{"type":42,"tag":92,"props":409,"children":410},{"style":105},[411],{"type":48,"value":412}," HF_TOKEN=",{"type":42,"tag":92,"props":414,"children":415},{"style":299},[416],{"type":48,"value":417},"\"",{"type":42,"tag":92,"props":419,"children":420},{"style":105},[421],{"type":48,"value":422},"\u003CTOKEN>",{"type":42,"tag":92,"props":424,"children":425},{"style":299},[426],{"type":48,"value":417},{"type":42,"tag":92,"props":428,"children":429},{"style":266},[430],{"type":48,"value":269},{"type":42,"tag":92,"props":432,"children":434},{"class":94,"line":433},11,[435,440,445,450,455,459],{"type":42,"tag":92,"props":436,"children":437},{"style":105},[438],{"type":48,"value":439},"  -v",{"type":42,"tag":92,"props":441,"children":442},{"style":299},[443],{"type":48,"value":444}," \"",{"type":42,"tag":92,"props":446,"children":447},{"style":266},[448],{"type":48,"value":449},"$HOME",{"type":42,"tag":92,"props":451,"children":452},{"style":105},[453],{"type":48,"value":454},"\u002F.cache\u002Fhuggingface\u002Fhub:\u002Froot\u002F.cache\u002Fhuggingface\u002Fhub",{"type":42,"tag":92,"props":456,"children":457},{"style":299},[458],{"type":48,"value":417},{"type":42,"tag":92,"props":460,"children":461},{"style":266},[462],{"type":48,"value":269},{"type":42,"tag":92,"props":464,"children":466},{"class":94,"line":465},12,[467,472],{"type":42,"tag":92,"props":468,"children":469},{"style":105},[470],{"type":48,"value":471},"  nvcr.io\u002Fnvidia\u002Fvllm:26.01-py3",{"type":42,"tag":92,"props":473,"children":474},{"style":266},[475],{"type":48,"value":269},{"type":42,"tag":92,"props":477,"children":479},{"class":94,"line":478},13,[480,485,490,494,499,503],{"type":42,"tag":92,"props":481,"children":482},{"style":105},[483],{"type":48,"value":484},"  vllm",{"type":42,"tag":92,"props":486,"children":487},{"style":105},[488],{"type":48,"value":489}," serve",{"type":42,"tag":92,"props":491,"children":492},{"style":299},[493],{"type":48,"value":444},{"type":42,"tag":92,"props":495,"children":496},{"style":105},[497],{"type":48,"value":498},"\u003CMODEL>",{"type":42,"tag":92,"props":500,"children":501},{"style":299},[502],{"type":48,"value":417},{"type":42,"tag":92,"props":504,"children":505},{"style":266},[506],{"type":48,"value":269},{"type":42,"tag":92,"props":508,"children":510},{"class":94,"line":509},14,[511,516,521],{"type":42,"tag":92,"props":512,"children":513},{"style":105},[514],{"type":48,"value":515},"    --max-model-len",{"type":42,"tag":92,"props":517,"children":518},{"style":351},[519],{"type":48,"value":520}," 32768",{"type":42,"tag":92,"props":522,"children":523},{"style":266},[524],{"type":48,"value":269},{"type":42,"tag":92,"props":526,"children":528},{"class":94,"line":527},15,[529,534],{"type":42,"tag":92,"props":530,"children":531},{"style":105},[532],{"type":48,"value":533},"    --gpu-memory-utilization",{"type":42,"tag":92,"props":535,"children":536},{"style":351},[537],{"type":48,"value":538}," 0.9\n",{"type":42,"tag":115,"props":540,"children":541},{},[],{"type":42,"tag":72,"props":543,"children":544},{},[545],{"type":48,"value":546},"Container version:",{"type":48,"value":548}," Use ",{"type":42,"tag":88,"props":550,"children":552},{"className":551},[],[553],{"type":48,"value":554},"nvcr.io\u002Fnvidia\u002Fvllm:26.01-py3",{"type":48,"value":556},". Do NOT use 25.10 — it has a FlashInfer buffer overflow on DGX Station.",{"type":42,"tag":68,"props":558,"children":559},{},[560,565,567,596,599],{"type":42,"tag":72,"props":561,"children":562},{},[563],{"type":48,"value":564},"Wait for the server to be ready.",{"type":48,"value":566}," Monitor logs:",{"type":42,"tag":80,"props":568,"children":570},{"className":82,"code":569,"language":84,"meta":85,"style":85},"docker logs -f vllm-server\n",[571],{"type":42,"tag":88,"props":572,"children":573},{"__ignoreMap":85},[574],{"type":42,"tag":92,"props":575,"children":576},{"class":94,"line":95},[577,581,586,591],{"type":42,"tag":92,"props":578,"children":579},{"style":99},[580],{"type":48,"value":225},{"type":42,"tag":92,"props":582,"children":583},{"style":105},[584],{"type":48,"value":585}," logs",{"type":42,"tag":92,"props":587,"children":588},{"style":105},[589],{"type":48,"value":590}," -f",{"type":42,"tag":92,"props":592,"children":593},{"style":105},[594],{"type":48,"value":595}," vllm-server\n",{"type":42,"tag":115,"props":597,"children":598},{},[],{"type":48,"value":600},"Wait for the line indicating the server is listening on port 8000.",{"type":42,"tag":68,"props":602,"children":603},{},[604,609],{"type":42,"tag":72,"props":605,"children":606},{},[607],{"type":48,"value":608},"Test the server:",{"type":42,"tag":80,"props":610,"children":612},{"className":82,"code":611,"language":84,"meta":85,"style":85},"curl http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions \\\n  -H \"Content-Type: application\u002Fjson\" \\\n  -d '{\n    \"model\": \"\u003CMODEL>\",\n    \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n    \"max_tokens\": 64\n  }'\n",[613],{"type":42,"tag":88,"props":614,"children":615},{"__ignoreMap":85},[616,633,658,675,683,691,699],{"type":42,"tag":92,"props":617,"children":618},{"class":94,"line":95},[619,624,629],{"type":42,"tag":92,"props":620,"children":621},{"style":99},[622],{"type":48,"value":623},"curl",{"type":42,"tag":92,"props":625,"children":626},{"style":105},[627],{"type":48,"value":628}," http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fchat\u002Fcompletions",{"type":42,"tag":92,"props":630,"children":631},{"style":266},[632],{"type":48,"value":269},{"type":42,"tag":92,"props":634,"children":635},{"class":94,"line":238},[636,641,645,650,654],{"type":42,"tag":92,"props":637,"children":638},{"style":105},[639],{"type":48,"value":640},"  -H",{"type":42,"tag":92,"props":642,"children":643},{"style":299},[644],{"type":48,"value":444},{"type":42,"tag":92,"props":646,"children":647},{"style":105},[648],{"type":48,"value":649},"Content-Type: application\u002Fjson",{"type":42,"tag":92,"props":651,"children":652},{"style":299},[653],{"type":48,"value":417},{"type":42,"tag":92,"props":655,"children":656},{"style":266},[657],{"type":48,"value":269},{"type":42,"tag":92,"props":659,"children":660},{"class":94,"line":248},[661,666,670],{"type":42,"tag":92,"props":662,"children":663},{"style":105},[664],{"type":48,"value":665},"  -d",{"type":42,"tag":92,"props":667,"children":668},{"style":299},[669],{"type":48,"value":302},{"type":42,"tag":92,"props":671,"children":672},{"style":105},[673],{"type":48,"value":674},"{\n",{"type":42,"tag":92,"props":676,"children":677},{"class":94,"line":272},[678],{"type":42,"tag":92,"props":679,"children":680},{"style":105},[681],{"type":48,"value":682},"    \"model\": \"\u003CMODEL>\",\n",{"type":42,"tag":92,"props":684,"children":685},{"class":94,"line":290},[686],{"type":42,"tag":92,"props":687,"children":688},{"style":105},[689],{"type":48,"value":690},"    \"messages\": [{\"role\": \"user\", \"content\": \"Hello\"}],\n",{"type":42,"tag":92,"props":692,"children":693},{"class":94,"line":319},[694],{"type":42,"tag":92,"props":695,"children":696},{"style":105},[697],{"type":48,"value":698},"    \"max_tokens\": 64\n",{"type":42,"tag":92,"props":700,"children":701},{"class":94,"line":337},[702,707],{"type":42,"tag":92,"props":703,"children":704},{"style":105},[705],{"type":48,"value":706},"  }",{"type":42,"tag":92,"props":708,"children":709},{"style":299},[710],{"type":48,"value":711},"'\n",{"type":42,"tag":68,"props":713,"children":714},{},[715,720,722],{"type":42,"tag":72,"props":716,"children":717},{},[718],{"type":48,"value":719},"Report the result",{"type":48,"value":721}," to the user, including:",{"type":42,"tag":147,"props":723,"children":724},{},[725,730,735],{"type":42,"tag":68,"props":726,"children":727},{},[728],{"type":48,"value":729},"Model loaded and serving on port 8000",{"type":42,"tag":68,"props":731,"children":732},{},[733],{"type":48,"value":734},"GPU memory utilization",{"type":42,"tag":68,"props":736,"children":737},{},[738,740],{"type":48,"value":739},"How to stop: ",{"type":42,"tag":88,"props":741,"children":743},{"className":742},[],[744],{"type":48,"value":745},"docker stop vllm-server && docker rm vllm-server",{"type":42,"tag":57,"props":747,"children":749},{"id":748},"tuning-parameters",[750],{"type":48,"value":751},"Tuning parameters",{"type":42,"tag":51,"props":753,"children":754},{},[755],{"type":48,"value":756},"Adjust these based on the user's workload:",{"type":42,"tag":758,"props":759,"children":760},"table",{},[761,790],{"type":42,"tag":762,"props":763,"children":764},"thead",{},[765],{"type":42,"tag":766,"props":767,"children":768},"tr",{},[769,775,780,785],{"type":42,"tag":770,"props":771,"children":772},"th",{},[773],{"type":48,"value":774},"Parameter",{"type":42,"tag":770,"props":776,"children":777},{},[778],{"type":48,"value":779},"Default",{"type":42,"tag":770,"props":781,"children":782},{},[783],{"type":48,"value":784},"Agent workloads",{"type":42,"tag":770,"props":786,"children":787},{},[788],{"type":48,"value":789},"Throughput workloads",{"type":42,"tag":791,"props":792,"children":793},"tbody",{},[794,822,849,876],{"type":42,"tag":766,"props":795,"children":796},{},[797,807,812,817],{"type":42,"tag":798,"props":799,"children":800},"td",{},[801],{"type":42,"tag":88,"props":802,"children":804},{"className":803},[],[805],{"type":48,"value":806},"--max-model-len",{"type":42,"tag":798,"props":808,"children":809},{},[810],{"type":48,"value":811},"32768",{"type":42,"tag":798,"props":813,"children":814},{},[815],{"type":48,"value":816},"32768-65536",{"type":42,"tag":798,"props":818,"children":819},{},[820],{"type":48,"value":821},"8192-16384",{"type":42,"tag":766,"props":823,"children":824},{},[825,834,839,844],{"type":42,"tag":798,"props":826,"children":827},{},[828],{"type":42,"tag":88,"props":829,"children":831},{"className":830},[],[832],{"type":48,"value":833},"--gpu-memory-utilization",{"type":42,"tag":798,"props":835,"children":836},{},[837],{"type":48,"value":838},"0.9",{"type":42,"tag":798,"props":840,"children":841},{},[842],{"type":48,"value":843},"0.85-0.90",{"type":42,"tag":798,"props":845,"children":846},{},[847],{"type":48,"value":848},"0.90-0.92",{"type":42,"tag":766,"props":850,"children":851},{},[852,861,866,871],{"type":42,"tag":798,"props":853,"children":854},{},[855],{"type":42,"tag":88,"props":856,"children":858},{"className":857},[],[859],{"type":48,"value":860},"--enable-prefix-caching",{"type":42,"tag":798,"props":862,"children":863},{},[864],{"type":48,"value":865},"off",{"type":42,"tag":798,"props":867,"children":868},{},[869],{"type":48,"value":870},"Enable (multi-turn reuse)",{"type":42,"tag":798,"props":872,"children":873},{},[874],{"type":48,"value":875},"Enable",{"type":42,"tag":766,"props":877,"children":878},{},[879,888,893,898],{"type":42,"tag":798,"props":880,"children":881},{},[882],{"type":42,"tag":88,"props":883,"children":885},{"className":884},[],[886],{"type":48,"value":887},"--max-num-seqs",{"type":42,"tag":798,"props":889,"children":890},{},[891],{"type":48,"value":892},"default",{"type":42,"tag":798,"props":894,"children":895},{},[896],{"type":48,"value":897},"4-16 (lower latency)",{"type":42,"tag":798,"props":899,"children":900},{},[901],{"type":48,"value":902},"32+ (higher throughput)",{"type":42,"tag":904,"props":905,"children":906},"style",{},[907],{"type":48,"value":908},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":910,"total":433},[911,931,944,959,974,985,1000],{"slug":912,"name":912,"fn":913,"description":914,"org":915,"tags":916,"stars":23,"repoUrl":24,"updatedAt":930},"analysis-methods","write Python analysis code for FHIR data","Teaches the analyst agent how to write correct, robust Python analysis code for FHIR clinical data using pandas, matplotlib, and scipy.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[917,920,923,926,927],{"name":918,"slug":919,"type":15},"Data Analysis","data-analysis",{"name":921,"slug":922,"type":15},"FHIR","fhir",{"name":924,"slug":925,"type":15},"Healthcare","healthcare",{"name":9,"slug":8,"type":15},{"name":928,"slug":929,"type":15},"Python","python","2026-07-14T05:35:59.037962",{"slug":932,"name":932,"fn":933,"description":934,"org":935,"tags":936,"stars":23,"repoUrl":24,"updatedAt":943},"case-summary","summarize clinical patient cases from FHIR","Prepare a complete clinical case summary for a patient from FHIR endpoints. Use when asked to summarize a patient, compile a case, or prepare for tumor board.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[937,938,939,940],{"name":921,"slug":922,"type":15},{"name":924,"slug":925,"type":15},{"name":9,"slug":8,"type":15},{"name":941,"slug":942,"type":15},"Summarization","summarization","2026-07-14T05:35:52.790528",{"slug":945,"name":945,"fn":946,"description":947,"org":948,"tags":949,"stars":23,"repoUrl":24,"updatedAt":958},"clinical-delegation","delegate clinical tasks to specialist agents","How to delegate clinical tasks to specialist agents. Always use sub-agent runtime with explicit agentId — never ACP. Never call FHIR via web_fetch.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[950,953,954,957],{"name":951,"slug":952,"type":15},"Agents","agents",{"name":924,"slug":925,"type":15},{"name":955,"slug":956,"type":15},"Multi-Agent","multi-agent",{"name":9,"slug":8,"type":15},"2026-07-14T05:35:55.294972",{"slug":960,"name":960,"fn":961,"description":962,"org":963,"tags":964,"stars":23,"repoUrl":24,"updatedAt":973},"clinical-knowledge","provide clinical reference and regulatory context","Teaches agents clinical reference ranges, condition codes, quality measure definitions, drug classifications, and regulatory context so they can flag abnormal values and identify care gaps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[965,968,969,970],{"name":966,"slug":967,"type":15},"Clinical Trials","clinical-trials",{"name":924,"slug":925,"type":15},{"name":9,"slug":8,"type":15},{"name":971,"slug":972,"type":15},"Regulatory Compliance","regulatory-compliance","2026-07-14T05:35:56.550833",{"slug":975,"name":975,"fn":976,"description":977,"org":978,"tags":979,"stars":23,"repoUrl":24,"updatedAt":984},"cohort-compare","analyze patient cohorts from FHIR endpoints","Analyze a cohort of patients from FHIR endpoints to find care gaps and patterns. Use when asked to compare patients, find quality gaps, or analyze a population.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[980,981,982,983],{"name":918,"slug":919,"type":15},{"name":921,"slug":922,"type":15},{"name":924,"slug":925,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:35:51.543095",{"slug":986,"name":986,"fn":987,"description":988,"org":989,"tags":990,"stars":23,"repoUrl":24,"updatedAt":999},"dgx-diagnose","diagnose NVIDIA DGX Station hardware issues","Diagnose common DGX Station GB300 issues — CUDA crashes, wrong-GPU targeting, vLLM\u002FSGLang container bugs, MIG state problems, NVLink\u002FFabric Manager errors, X\u002FVulkan failures, HuggingFace auth, and port conflicts. Use when the user reports a GPU error, inference server crash, MIG problem, or any unexplained DGX Station failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[991,992,995,996],{"name":21,"slug":22,"type":15},{"name":993,"slug":994,"type":15},"Debugging","debugging",{"name":9,"slug":8,"type":15},{"name":997,"slug":998,"type":15},"Observability","observability","2026-07-14T05:31:04.085598",{"slug":1001,"name":1001,"fn":1002,"description":1003,"org":1004,"tags":1005,"stars":23,"repoUrl":24,"updatedAt":1012},"fhir-basics","query and parse FHIR R4 API resources","Teaches agents how FHIR R4 APIs work, what resources are available, how to query them with search parameters, and how to correctly parse all response formats including component Observations.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1006,1009,1010,1011],{"name":1007,"slug":1008,"type":15},"API Development","api-development",{"name":921,"slug":922,"type":15},{"name":924,"slug":925,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:35:57.797226",{"items":1014,"total":1167},[1015,1033,1047,1058,1070,1082,1095,1109,1122,1133,1147,1156],{"slug":1016,"name":1016,"fn":1017,"description":1018,"org":1019,"tags":1020,"stars":1030,"repoUrl":1031,"updatedAt":1032},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1021,1024,1027],{"name":1022,"slug":1023,"type":15},"Documentation","documentation",{"name":1025,"slug":1026,"type":15},"MCP","mcp",{"name":1028,"slug":1029,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":1034,"name":1034,"fn":1035,"description":1036,"org":1037,"tags":1038,"stars":1044,"repoUrl":1045,"updatedAt":1046},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1039,1042,1043],{"name":1040,"slug":1041,"type":15},"Containers","containers",{"name":17,"slug":18,"type":15},{"name":928,"slug":929,"type":15},17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":1048,"name":1048,"fn":1049,"description":1050,"org":1051,"tags":1052,"stars":1044,"repoUrl":1045,"updatedAt":1057},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1053,1056],{"name":1054,"slug":1055,"type":15},"CI\u002FCD","ci-cd",{"name":17,"slug":18,"type":15},"2026-07-14T05:25:59.97109",{"slug":1059,"name":1059,"fn":1060,"description":1061,"org":1062,"tags":1063,"stars":1044,"repoUrl":1045,"updatedAt":1069},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1064,1065,1066],{"name":1054,"slug":1055,"type":15},{"name":17,"slug":18,"type":15},{"name":1067,"slug":1068,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":1071,"name":1071,"fn":1072,"description":1073,"org":1074,"tags":1075,"stars":1044,"repoUrl":1045,"updatedAt":1081},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1076,1077,1078],{"name":993,"slug":994,"type":15},{"name":1067,"slug":1068,"type":15},{"name":1079,"slug":1080,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1083,"name":1083,"fn":1084,"description":1085,"org":1086,"tags":1087,"stars":1044,"repoUrl":1045,"updatedAt":1094},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1088,1091],{"name":1089,"slug":1090,"type":15},"Best Practices","best-practices",{"name":1092,"slug":1093,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1096,"name":1096,"fn":1097,"description":1098,"org":1099,"tags":1100,"stars":1044,"repoUrl":1045,"updatedAt":1108},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1101,1104,1107],{"name":1102,"slug":1103,"type":15},"Machine Learning","machine-learning",{"name":1105,"slug":1106,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":1110,"name":1110,"fn":1111,"description":1112,"org":1113,"tags":1114,"stars":1044,"repoUrl":1045,"updatedAt":1121},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1115,1118],{"name":1116,"slug":1117,"type":15},"QA","qa",{"name":1119,"slug":1120,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":1123,"name":1123,"fn":1124,"description":1125,"org":1126,"tags":1127,"stars":1044,"repoUrl":1045,"updatedAt":1132},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1128,1129],{"name":17,"slug":18,"type":15},{"name":1130,"slug":1131,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":1134,"name":1134,"fn":1135,"description":1136,"org":1137,"tags":1138,"stars":1044,"repoUrl":1045,"updatedAt":1146},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1139,1142,1143],{"name":1140,"slug":1141,"type":15},"Code Review","code-review",{"name":1067,"slug":1068,"type":15},{"name":1144,"slug":1145,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":1148,"name":1148,"fn":1149,"description":1150,"org":1151,"tags":1152,"stars":1044,"repoUrl":1045,"updatedAt":1155},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1153,1154],{"name":1116,"slug":1117,"type":15},{"name":1119,"slug":1120,"type":15},"2026-07-14T05:25:54.928983",{"slug":1157,"name":1157,"fn":1158,"description":1159,"org":1160,"tags":1161,"stars":1044,"repoUrl":1045,"updatedAt":1166},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1162,1165],{"name":1163,"slug":1164,"type":15},"Automation","automation",{"name":1054,"slug":1055,"type":15},"2026-07-30T05:29:03.275638",496]