[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-jetson-llm-serve":3,"mdc--qlq2q5-key":31,"related-repo-nvidia-jetson-llm-serve":2125,"related-org-nvidia-jetson-llm-serve":2230},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":20,"repoUrl":21,"updatedAt":22,"license":23,"forks":24,"topics":25,"repo":26,"sourceUrl":29,"mdContent":30},"jetson-llm-serve","serve LLMs on Jetson devices","Stand up vLLM or SGLang serving on Jetson, using upstream vLLM on Thor and Orin JetPack 7.2+, and NVIDIA-AI-IOT vLLM on older Orin.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19],{"name":13,"slug":14,"type":15},"LLM","llm","tag",{"name":17,"slug":18,"type":15},"Hardware","hardware",{"name":9,"slug":8,"type":15},2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:26:31.409181","Apache-2.0",281,[],{"repoUrl":21,"stars":20,"forks":24,"topics":27,"description":28},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fjetson-llm-serve","---\nname: jetson-llm-serve\ndescription: Stand up vLLM or SGLang serving on Jetson, using upstream vLLM on Thor and Orin JetPack 7.2+, and NVIDIA-AI-IOT vLLM on older Orin.\nversion: 0.0.1\nlicense: \"Apache-2.0\"\nmetadata:\n  author: \"Jetson Team\"\n  tags: [jetson, llm, serving]\n  languages: [markdown]\n  data-classification: public\n---\n\n# Jetson LLM Serve\n\nEncodes the [Jetson AI Lab GenAI tutorial](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F): on Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`); on older Orin, pick the NVIDIA-AI-IOT prebuilt vLLM container; on Thor, use upstream vLLM 0.20+ or validated native vLLM 0.20+, and use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) when SGLang is requested. Set MAXN, make Hugging Face credentials\u002Fcache available, and launch an OpenAI-compatible server. Works for both LLMs and VLMs.\n\n## Purpose\n\nProvide a Jetson-appropriate serving recipe for an LLM or VLM using vLLM or SGLang, including runtime path, launch command, endpoint, and verification step.\n\n## When to use\n\n- \"Run \u002F serve \u002F host this model on a Jetson.\"\n- \"Start a vLLM server I can hit from Open WebUI \u002F my app.\"\n- After `jetson-inference-mem-tune` produced launch flags and the user wants to actually start the server.\n\nFor recipe-only questions, answer from this document without starting\ncontainers. Run live pre-flight checks only when the user asks you to check this\ndevice or execute the deployment.\n\n## Prerequisites\n\n- Run on the Jetson host or a shell with Docker access to the Jetson GPU runtime.\n- Know the target Jetson generation (`thor` or `orin`) and the model identifier or local checkpoint path.\n- Use `HF_TOKEN` only when the model is gated\u002Fprivate; public models should omit the token environment variable.\n- Use `jetson-inference-mem-tune` first when memory headroom or launch flags are uncertain.\n\n## Instructions\n\nFor recipe questions, provide a complete launch recipe instead of trying to call\n`jetson-llm-serve` as a tool. A complete answer includes:\n\n- The Jetson-appropriate runtime path: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) on Thor, NVIDIA-AI-IOT vLLM container on older Orin, or upstream vLLM 0.20+ on Orin JetPack 7.2 \u002F L4T r39+.\n- The model checkpoint \u002F Hugging Face repo the user named.\n- A `docker run` + server command sketch with `--host 0.0.0.0 --port 8000`.\n- The OpenAI-compatible endpoint: `http:\u002F\u002F\u003Cjetson-ip>:8000\u002Fv1`.\n- A verification step such as `curl http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels`.\n\nFor VLM questions, explicitly say the VLM uses the same vLLM serving flow as an\nLLM with a different vision-language checkpoint. Do not omit `vLLM` or the\nJetson container when answering VLM prompts.\n\n## Step 1 — Pick the runtime path (per Jetson family)\n\nUse upstream vLLM 0.20+ on Thor (`vllm\u002Fvllm-openai:latest`, or a validated native vLLM 0.20+ install). On Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`). On older Orin releases, use the **NVIDIA-AI-IOT prebuilt vLLM image** ([packages](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages)) because it ships the correct CUDA \u002F cuDNN \u002F TensorRT stack for that JetPack. Use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) on Thor when the user asks for SGLang, RAG, tool-use, or programmable serving; do not recommend native upstream SGLang on Orin unless a JetPack-matched release explicitly supports it.\n\n| Jetson family               | Runtime path                                      |\n|-----------------------------|---------------------------------------------------|\n| Thor (T5000, T4000)         | upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) |\n| AGX Orin \u002F Orin NX \u002F Nano   | Orin JetPack 7.2 \u002F L4T r39+: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`); older Orin: `ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin` |\n\nTo detect the silicon era for image tags:\n\n1. Source the detector so exports survive in your shell:\n   ```bash\n   . skills\u002Fjetson-diagnostic\u002Fscripts\u002Fdetect_jetson.sh\n   ```\n2. Check `JETSON_GENERATION` (`thor` or `orin`) and choose the matching runtime path from the table above.\n3. Use `JETSON_PRODUCT_LINE` for a finer bucket such as `thor-agx` or `orin-nano`; `JETSON_SKU` remains the legacy identifier.\n\nDo not use `bash skills\u002Fjetson-diagnostic\u002Fscripts\u002Fdetect_jetson.sh` when you need exported variables in the caller; running with `bash` uses a subshell.\n\n## Step 2 — Set MAXN power mode\n\n```bash\nsudo nvpmodel -m 0 && sudo jetson_clocks\n```\n\nSkip this only if the user explicitly asks for a power-constrained run; otherwise benchmark and serving numbers will be inconsistent.\n\n## Step 3 — Run the server\n\nOn Thor with vLLM, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) or a validated native vLLM 0.20+ install:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --ipc host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  vllm\u002Fvllm-openai:latest \\\n  vllm serve \u003Chf-repo-id> \\\n    --host 0.0.0.0 --port 8000 \\\n    --max-model-len 8192 \\\n    --gpu-memory-utilization 0.75 \\\n    --tensor-parallel-size 1\n```\n\nOn Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`). On older Orin releases, use the NVIDIA-AI-IOT container:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin \\\n  vllm serve \u003Chf-repo-id> \\\n    --host 0.0.0.0 --port 8000 \\\n    --max-model-len 4096 \\\n    --gpu-memory-utilization 0.85 \\\n    --tensor-parallel-size 1\n```\n\n`HF_TOKEN` is required only for gated\u002Fprivate Hugging Face models; omit the `-e HF_TOKEN=\"$HF_TOKEN\"` line for public models that do not need Hub authentication. Passing `HF_TOKEN` as an environment variable can expose it through Docker inspect output, process metadata, or logs on shared systems. Prefer the narrowest-scoped token possible, rotate\u002Frevoke it after shared-container use, and use a mounted credential file or Docker secret when the deployment environment supports that pattern.\n\nWait for `Application startup complete.` Server is on `http:\u002F\u002F0.0.0.0:8000\u002Fv1`.\n\nFor SGLang on Thor, use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`), which packages SGLang 0.5.5.post2 and lists Jetson Thor support. Do not judge Thor SGLang support from older prerelease SGLang results:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --ipc host --name sglang \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3 \\\n  python3 -m sglang.launch_server \\\n    --model-path \u003Chf-repo-id> \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    --mem-fraction-static 0.60 \\\n    --max-running-requests 8\n```\n\nUse SGLang when the user needs RAG\u002Ftool-use workflows, structured generation, or\nSGLang-specific scheduling. For plain high-throughput OpenAI-compatible serving,\nprefer vLLM unless the user asks for SGLang.\n\n### SKU-appropriate defaults\n\n| Knob                       | Orin Nano \u002F NX | AGX Orin \u002F Thor |\n|----------------------------|----------------|-----------------|\n| `--max-model-len`          | `4096`         | `8192`          |\n| `--gpu-memory-utilization` | `0.85`         | `0.85`          |\n| `--tensor-parallel-size`   | `1`            | `1`             |\n\nIf the server OOMs at startup, lower `--gpu-memory-utilization` by 0.05 and re-launch (or run `jetson-inference-mem-tune` for a workload-aware recommendation).\n\n## Quantization preferences (matters more than the runtime)\n\nFor vLLM and SGLang, choose checkpoint formats by Jetson family:\n\n| Jetson family | First choice | Acceptable fallback |\n|---------------|--------------|---------------------|\n| Thor | **NVFP4** when the model\u002Fruntime supports it | W4A16 |\n| Orin Nano \u002F NX | **W4A16** | AWQ or GPTQ 4-bit |\n| AGX Orin | **W4A16** | AWQ or GPTQ 4-bit |\n\nFor llama.cpp and `Ollama`, use GGUF model quantization names instead: recommend **INT4 \u002F Q4_K_M GGUF** on both Orin and Thor, and choose a smaller INT4 GGUF model if memory is tight. Do not call GGUF Q4_K_M a W4A16\u002FAWQ\u002FGPTQ model. NVFP4 is Thor-preferred and Thor-tuned for runtimes that support it.\n\n## VLM mode\n\nVLMs use the same flow as LLMs: same container, same `vllm serve` invocation, different vision-language checkpoint. The container handles image preprocessing. For a VLM-specific browser UI, use the [`live-vlm-webui`](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages) container; for a generic chat UI for either, use Open WebUI pointed at `http:\u002F\u002F\u003Cjetson-ip>:8000\u002Fv1`.\n\n## Do not fabricate device capacity\n\nDo not invent RAM totals, free-memory values, model sizes, JetPack versions, or\nSKU\u002Fvariant names when giving a serving recipe. If capacity matters, either run\nthe live pre-flight checks (when execution is allowed) or hand off to\n`jetson-inference-mem-tune` \u002F `jetson-memory-audit`. If live data is not\navailable, say the value is unknown and provide conservative defaults instead\nof quoting a made-up number.\n\n## Pre-flight checklist (the agent should verify before running Step 3)\n\n- [ ] On a Jetson (`\u002Fproc\u002Fdevice-tree\u002Fmodel` contains `NVIDIA Jetson`).\n- [ ] `nvpmodel -q` reports a recognized max-performance mode: `MAXN` or `MAXN_*` such as `MAXN_SUPER`. Wattage-named modes should be reported as warnings unless the user explicitly confirms they are the intended benchmark mode for that device.\n- [ ] On Thor, check whether MIG is enabled before launching (`nvidia-smi -L` and `nvidia-smi mig -lgi`). If MIG is enabled, warn that vLLM\u002FSGLang may see only a MIG slice or no CUDA device.\n- [ ] On Thor with MIG or display\u002Fcamera contention, inspect GPU users with `sudo lsof \u002Fdev\u002Fnvidia*`. Display managers, `Xorg`\u002FGNOME, or `nvargus-daemon` may hold GPU device files; do not stop services or change MIG mode unless the user explicitly approves.\n- [ ] No container named `vllm` already running (`docker ps --format '{{.Names}}'`); otherwise `docker rm -f vllm` first.\n- [ ] Docker exposes the NVIDIA runtime (`docker info | grep -i 'runtimes.*nvidia'`), or a GPU-enabled container can run `nvidia-smi`.\n- [ ] `~\u002F.cache\u002Fhuggingface` exists; `HF_TOKEN` is set if the model is gated.\n\n## Limitations\n\n- This skill provides serving commands and pre-flight checks; it does not benchmark the deployed server.\n- Container tags such as `latest` are mutable. For release or compliance deployments, pin a digest and record it with the deployment notes.\n- vLLM and SGLang memory limits still depend on model architecture, quantization, context length, and concurrent request count. Use `jetson-inference-mem-tune` when a command OOMs or memory headroom matters.\n- Thor vLLM requires upstream vLLM 0.20+ or newer. Older upstream vLLM images may not support Thor \u002F SM 11.0 correctly.\n- Thor SGLang should use NVIDIA SGLang 26.01 or newer release notes that explicitly list Jetson Thor support. NVIDIA SGLang 26.01 contains SGLang 0.5.5.post2.\n- On Thor, MIG, desktop display, or camera services can hide the full GPU from containers. This skill should detect and warn only; disabling MIG or stopping services such as `gdm3` or `nvargus-daemon` requires explicit user approval.\n- Host-native vLLM\u002FSGLang on Thor should be used only when that install is already validated on the target JetPack.\n\n## Hand off to\n\n- `jetson-llm-benchmark` to actually measure the deployed server.\n- `jetson-speculative-decoding` to add EAGLE-3 \u002F draft-model speculation by appending `--speculative-config '{...}'` to the `vllm serve` command above.\n- `jetson-inference-mem-tune` if the server OOMs or is memory-bound.\n\n## Source\n\n[Jetson AI Lab — Introduction to GenAI on Jetson: How to Run LLMs and VLMs](https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F) and [NVIDIA-AI-IOT GHCR packages](https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages).\n",{"data":32,"body":42},{"name":4,"description":6,"version":33,"license":23,"metadata":34},"0.0.1",{"author":35,"tags":36,"languages":39,"data-classification":41},"Jetson Team",[37,14,38],"jetson","serving",[40],"markdown","public",{"type":43,"children":44},"root",[45,53,87,94,99,105,133,138,144,197,203,215,287,300,306,349,429,434,531,551,557,605,610,616,628,896,908,1125,1150,1170,1182,1422,1427,1434,1549,1568,1574,1579,1666,1686,1692,1723,1729,1749,1755,1967,1973,2040,2046,2096,2102,2119],{"type":46,"tag":47,"props":48,"children":49},"element","h1",{"id":4},[50],{"type":51,"value":52},"text","Jetson LLM Serve",{"type":46,"tag":54,"props":55,"children":56},"p",{},[57,59,68,70,77,79,85],{"type":51,"value":58},"Encodes the ",{"type":46,"tag":60,"props":61,"children":65},"a",{"href":62,"rel":63},"https:\u002F\u002Fwww.jetson-ai-lab.com\u002Ftutorials\u002Fgenai-on-jetson-llms-vlms\u002F",[64],"nofollow",[66],{"type":51,"value":67},"Jetson AI Lab GenAI tutorial",{"type":51,"value":69},": on Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (",{"type":46,"tag":71,"props":72,"children":74},"code",{"className":73},[],[75],{"type":51,"value":76},"vllm\u002Fvllm-openai:latest",{"type":51,"value":78},"); on older Orin, pick the NVIDIA-AI-IOT prebuilt vLLM container; on Thor, use upstream vLLM 0.20+ or validated native vLLM 0.20+, and use NVIDIA SGLang 26.01 (",{"type":46,"tag":71,"props":80,"children":82},{"className":81},[],[83],{"type":51,"value":84},"nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3",{"type":51,"value":86},", SGLang 0.5.5.post2) when SGLang is requested. Set MAXN, make Hugging Face credentials\u002Fcache available, and launch an OpenAI-compatible server. Works for both LLMs and VLMs.",{"type":46,"tag":88,"props":89,"children":91},"h2",{"id":90},"purpose",[92],{"type":51,"value":93},"Purpose",{"type":46,"tag":54,"props":95,"children":96},{},[97],{"type":51,"value":98},"Provide a Jetson-appropriate serving recipe for an LLM or VLM using vLLM or SGLang, including runtime path, launch command, endpoint, and verification step.",{"type":46,"tag":88,"props":100,"children":102},{"id":101},"when-to-use",[103],{"type":51,"value":104},"When to use",{"type":46,"tag":106,"props":107,"children":108},"ul",{},[109,115,120],{"type":46,"tag":110,"props":111,"children":112},"li",{},[113],{"type":51,"value":114},"\"Run \u002F serve \u002F host this model on a Jetson.\"",{"type":46,"tag":110,"props":116,"children":117},{},[118],{"type":51,"value":119},"\"Start a vLLM server I can hit from Open WebUI \u002F my app.\"",{"type":46,"tag":110,"props":121,"children":122},{},[123,125,131],{"type":51,"value":124},"After ",{"type":46,"tag":71,"props":126,"children":128},{"className":127},[],[129],{"type":51,"value":130},"jetson-inference-mem-tune",{"type":51,"value":132}," produced launch flags and the user wants to actually start the server.",{"type":46,"tag":54,"props":134,"children":135},{},[136],{"type":51,"value":137},"For recipe-only questions, answer from this document without starting\ncontainers. Run live pre-flight checks only when the user asks you to check this\ndevice or execute the deployment.",{"type":46,"tag":88,"props":139,"children":141},{"id":140},"prerequisites",[142],{"type":51,"value":143},"Prerequisites",{"type":46,"tag":106,"props":145,"children":146},{},[147,152,173,186],{"type":46,"tag":110,"props":148,"children":149},{},[150],{"type":51,"value":151},"Run on the Jetson host or a shell with Docker access to the Jetson GPU runtime.",{"type":46,"tag":110,"props":153,"children":154},{},[155,157,163,165,171],{"type":51,"value":156},"Know the target Jetson generation (",{"type":46,"tag":71,"props":158,"children":160},{"className":159},[],[161],{"type":51,"value":162},"thor",{"type":51,"value":164}," or ",{"type":46,"tag":71,"props":166,"children":168},{"className":167},[],[169],{"type":51,"value":170},"orin",{"type":51,"value":172},") and the model identifier or local checkpoint path.",{"type":46,"tag":110,"props":174,"children":175},{},[176,178,184],{"type":51,"value":177},"Use ",{"type":46,"tag":71,"props":179,"children":181},{"className":180},[],[182],{"type":51,"value":183},"HF_TOKEN",{"type":51,"value":185}," only when the model is gated\u002Fprivate; public models should omit the token environment variable.",{"type":46,"tag":110,"props":187,"children":188},{},[189,190,195],{"type":51,"value":177},{"type":46,"tag":71,"props":191,"children":193},{"className":192},[],[194],{"type":51,"value":130},{"type":51,"value":196}," first when memory headroom or launch flags are uncertain.",{"type":46,"tag":88,"props":198,"children":200},{"id":199},"instructions",[201],{"type":51,"value":202},"Instructions",{"type":46,"tag":54,"props":204,"children":205},{},[206,208,213],{"type":51,"value":207},"For recipe questions, provide a complete launch recipe instead of trying to call\n",{"type":46,"tag":71,"props":209,"children":211},{"className":210},[],[212],{"type":51,"value":4},{"type":51,"value":214}," as a tool. A complete answer includes:",{"type":46,"tag":106,"props":216,"children":217},{},[218,237,242,263,275],{"type":46,"tag":110,"props":219,"children":220},{},[221,223,228,230,235],{"type":51,"value":222},"The Jetson-appropriate runtime path: upstream vLLM 0.20+ (",{"type":46,"tag":71,"props":224,"children":226},{"className":225},[],[227],{"type":51,"value":76},{"type":51,"value":229},") or NVIDIA SGLang 26.01 (",{"type":46,"tag":71,"props":231,"children":233},{"className":232},[],[234],{"type":51,"value":84},{"type":51,"value":236},", SGLang 0.5.5.post2) on Thor, NVIDIA-AI-IOT vLLM container on older Orin, or upstream vLLM 0.20+ on Orin JetPack 7.2 \u002F L4T r39+.",{"type":46,"tag":110,"props":238,"children":239},{},[240],{"type":51,"value":241},"The model checkpoint \u002F Hugging Face repo the user named.",{"type":46,"tag":110,"props":243,"children":244},{},[245,247,253,255,261],{"type":51,"value":246},"A ",{"type":46,"tag":71,"props":248,"children":250},{"className":249},[],[251],{"type":51,"value":252},"docker run",{"type":51,"value":254}," + server command sketch with ",{"type":46,"tag":71,"props":256,"children":258},{"className":257},[],[259],{"type":51,"value":260},"--host 0.0.0.0 --port 8000",{"type":51,"value":262},".",{"type":46,"tag":110,"props":264,"children":265},{},[266,268,274],{"type":51,"value":267},"The OpenAI-compatible endpoint: ",{"type":46,"tag":71,"props":269,"children":271},{"className":270},[],[272],{"type":51,"value":273},"http:\u002F\u002F\u003Cjetson-ip>:8000\u002Fv1",{"type":51,"value":262},{"type":46,"tag":110,"props":276,"children":277},{},[278,280,286],{"type":51,"value":279},"A verification step such as ",{"type":46,"tag":71,"props":281,"children":283},{"className":282},[],[284],{"type":51,"value":285},"curl http:\u002F\u002Flocalhost:8000\u002Fv1\u002Fmodels",{"type":51,"value":262},{"type":46,"tag":54,"props":288,"children":289},{},[290,292,298],{"type":51,"value":291},"For VLM questions, explicitly say the VLM uses the same vLLM serving flow as an\nLLM with a different vision-language checkpoint. Do not omit ",{"type":46,"tag":71,"props":293,"children":295},{"className":294},[],[296],{"type":51,"value":297},"vLLM",{"type":51,"value":299}," or the\nJetson container when answering VLM prompts.",{"type":46,"tag":88,"props":301,"children":303},{"id":302},"step-1-pick-the-runtime-path-per-jetson-family",[304],{"type":51,"value":305},"Step 1 — Pick the runtime path (per Jetson family)",{"type":46,"tag":54,"props":307,"children":308},{},[309,311,316,318,323,325,331,333,340,342,347],{"type":51,"value":310},"Use upstream vLLM 0.20+ on Thor (",{"type":46,"tag":71,"props":312,"children":314},{"className":313},[],[315],{"type":51,"value":76},{"type":51,"value":317},", or a validated native vLLM 0.20+ install). On Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (",{"type":46,"tag":71,"props":319,"children":321},{"className":320},[],[322],{"type":51,"value":76},{"type":51,"value":324},"). On older Orin releases, use the ",{"type":46,"tag":326,"props":327,"children":328},"strong",{},[329],{"type":51,"value":330},"NVIDIA-AI-IOT prebuilt vLLM image",{"type":51,"value":332}," (",{"type":46,"tag":60,"props":334,"children":337},{"href":335,"rel":336},"https:\u002F\u002Fgithub.com\u002Forgs\u002FNVIDIA-AI-IOT\u002Fpackages",[64],[338],{"type":51,"value":339},"packages",{"type":51,"value":341},") because it ships the correct CUDA \u002F cuDNN \u002F TensorRT stack for that JetPack. Use NVIDIA SGLang 26.01 (",{"type":46,"tag":71,"props":343,"children":345},{"className":344},[],[346],{"type":51,"value":84},{"type":51,"value":348},", SGLang 0.5.5.post2) on Thor when the user asks for SGLang, RAG, tool-use, or programmable serving; do not recommend native upstream SGLang on Orin unless a JetPack-matched release explicitly supports it.",{"type":46,"tag":350,"props":351,"children":352},"table",{},[353,372],{"type":46,"tag":354,"props":355,"children":356},"thead",{},[357],{"type":46,"tag":358,"props":359,"children":360},"tr",{},[361,367],{"type":46,"tag":362,"props":363,"children":364},"th",{},[365],{"type":51,"value":366},"Jetson family",{"type":46,"tag":362,"props":368,"children":369},{},[370],{"type":51,"value":371},"Runtime path",{"type":46,"tag":373,"props":374,"children":375},"tbody",{},[376,403],{"type":46,"tag":358,"props":377,"children":378},{},[379,385],{"type":46,"tag":380,"props":381,"children":382},"td",{},[383],{"type":51,"value":384},"Thor (T5000, T4000)",{"type":46,"tag":380,"props":386,"children":387},{},[388,390,395,396,401],{"type":51,"value":389},"upstream vLLM 0.20+ (",{"type":46,"tag":71,"props":391,"children":393},{"className":392},[],[394],{"type":51,"value":76},{"type":51,"value":229},{"type":46,"tag":71,"props":397,"children":399},{"className":398},[],[400],{"type":51,"value":84},{"type":51,"value":402},", SGLang 0.5.5.post2)",{"type":46,"tag":358,"props":404,"children":405},{},[406,411],{"type":46,"tag":380,"props":407,"children":408},{},[409],{"type":51,"value":410},"AGX Orin \u002F Orin NX \u002F Nano",{"type":46,"tag":380,"props":412,"children":413},{},[414,416,421,423],{"type":51,"value":415},"Orin JetPack 7.2 \u002F L4T r39+: upstream vLLM 0.20+ (",{"type":46,"tag":71,"props":417,"children":419},{"className":418},[],[420],{"type":51,"value":76},{"type":51,"value":422},"); older Orin: ",{"type":46,"tag":71,"props":424,"children":426},{"className":425},[],[427],{"type":51,"value":428},"ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin",{"type":46,"tag":54,"props":430,"children":431},{},[432],{"type":51,"value":433},"To detect the silicon era for image tags:",{"type":46,"tag":435,"props":436,"children":437},"ol",{},[438,471,496],{"type":46,"tag":110,"props":439,"children":440},{},[441,443],{"type":51,"value":442},"Source the detector so exports survive in your shell:\n",{"type":46,"tag":444,"props":445,"children":450},"pre",{"className":446,"code":447,"language":448,"meta":449,"style":449},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight",". skills\u002Fjetson-diagnostic\u002Fscripts\u002Fdetect_jetson.sh\n","bash","",[451],{"type":46,"tag":71,"props":452,"children":453},{"__ignoreMap":449},[454],{"type":46,"tag":455,"props":456,"children":459},"span",{"class":457,"line":458},"line",1,[460,465],{"type":46,"tag":455,"props":461,"children":463},{"style":462},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[464],{"type":51,"value":262},{"type":46,"tag":455,"props":466,"children":468},{"style":467},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[469],{"type":51,"value":470}," skills\u002Fjetson-diagnostic\u002Fscripts\u002Fdetect_jetson.sh\n",{"type":46,"tag":110,"props":472,"children":473},{},[474,476,482,483,488,489,494],{"type":51,"value":475},"Check ",{"type":46,"tag":71,"props":477,"children":479},{"className":478},[],[480],{"type":51,"value":481},"JETSON_GENERATION",{"type":51,"value":332},{"type":46,"tag":71,"props":484,"children":486},{"className":485},[],[487],{"type":51,"value":162},{"type":51,"value":164},{"type":46,"tag":71,"props":490,"children":492},{"className":491},[],[493],{"type":51,"value":170},{"type":51,"value":495},") and choose the matching runtime path from the table above.",{"type":46,"tag":110,"props":497,"children":498},{},[499,500,506,508,514,515,521,523,529],{"type":51,"value":177},{"type":46,"tag":71,"props":501,"children":503},{"className":502},[],[504],{"type":51,"value":505},"JETSON_PRODUCT_LINE",{"type":51,"value":507}," for a finer bucket such as ",{"type":46,"tag":71,"props":509,"children":511},{"className":510},[],[512],{"type":51,"value":513},"thor-agx",{"type":51,"value":164},{"type":46,"tag":71,"props":516,"children":518},{"className":517},[],[519],{"type":51,"value":520},"orin-nano",{"type":51,"value":522},"; ",{"type":46,"tag":71,"props":524,"children":526},{"className":525},[],[527],{"type":51,"value":528},"JETSON_SKU",{"type":51,"value":530}," remains the legacy identifier.",{"type":46,"tag":54,"props":532,"children":533},{},[534,536,542,544,549],{"type":51,"value":535},"Do not use ",{"type":46,"tag":71,"props":537,"children":539},{"className":538},[],[540],{"type":51,"value":541},"bash skills\u002Fjetson-diagnostic\u002Fscripts\u002Fdetect_jetson.sh",{"type":51,"value":543}," when you need exported variables in the caller; running with ",{"type":46,"tag":71,"props":545,"children":547},{"className":546},[],[548],{"type":51,"value":448},{"type":51,"value":550}," uses a subshell.",{"type":46,"tag":88,"props":552,"children":554},{"id":553},"step-2-set-maxn-power-mode",[555],{"type":51,"value":556},"Step 2 — Set MAXN power mode",{"type":46,"tag":444,"props":558,"children":560},{"className":446,"code":559,"language":448,"meta":449,"style":449},"sudo nvpmodel -m 0 && sudo jetson_clocks\n",[561],{"type":46,"tag":71,"props":562,"children":563},{"__ignoreMap":449},[564],{"type":46,"tag":455,"props":565,"children":566},{"class":457,"line":458},[567,573,578,583,589,595,600],{"type":46,"tag":455,"props":568,"children":570},{"style":569},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[571],{"type":51,"value":572},"sudo",{"type":46,"tag":455,"props":574,"children":575},{"style":467},[576],{"type":51,"value":577}," nvpmodel",{"type":46,"tag":455,"props":579,"children":580},{"style":467},[581],{"type":51,"value":582}," -m",{"type":46,"tag":455,"props":584,"children":586},{"style":585},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[587],{"type":51,"value":588}," 0",{"type":46,"tag":455,"props":590,"children":592},{"style":591},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[593],{"type":51,"value":594}," &&",{"type":46,"tag":455,"props":596,"children":597},{"style":569},[598],{"type":51,"value":599}," sudo",{"type":46,"tag":455,"props":601,"children":602},{"style":467},[603],{"type":51,"value":604}," jetson_clocks\n",{"type":46,"tag":54,"props":606,"children":607},{},[608],{"type":51,"value":609},"Skip this only if the user explicitly asks for a power-constrained run; otherwise benchmark and serving numbers will be inconsistent.",{"type":46,"tag":88,"props":611,"children":613},{"id":612},"step-3-run-the-server",[614],{"type":51,"value":615},"Step 3 — Run the server",{"type":46,"tag":54,"props":617,"children":618},{},[619,621,626],{"type":51,"value":620},"On Thor with vLLM, use upstream vLLM 0.20+ (",{"type":46,"tag":71,"props":622,"children":624},{"className":623},[],[625],{"type":51,"value":76},{"type":51,"value":627},") or a validated native vLLM 0.20+ install:",{"type":46,"tag":444,"props":629,"children":631},{"className":446,"code":630,"language":448,"meta":449,"style":449},"docker run --rm -it --runtime nvidia --network host --ipc host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  vllm\u002Fvllm-openai:latest \\\n  vllm serve \u003Chf-repo-id> \\\n    --host 0.0.0.0 --port 8000 \\\n    --max-model-len 8192 \\\n    --gpu-memory-utilization 0.75 \\\n    --tensor-parallel-size 1\n",[632],{"type":46,"tag":71,"props":633,"children":634},{"__ignoreMap":449},[635,703,736,767,780,818,846,864,882],{"type":46,"tag":455,"props":636,"children":637},{"class":457,"line":458},[638,643,648,653,658,663,668,673,678,683,687,692,697],{"type":46,"tag":455,"props":639,"children":640},{"style":569},[641],{"type":51,"value":642},"docker",{"type":46,"tag":455,"props":644,"children":645},{"style":467},[646],{"type":51,"value":647}," run",{"type":46,"tag":455,"props":649,"children":650},{"style":467},[651],{"type":51,"value":652}," --rm",{"type":46,"tag":455,"props":654,"children":655},{"style":467},[656],{"type":51,"value":657}," -it",{"type":46,"tag":455,"props":659,"children":660},{"style":467},[661],{"type":51,"value":662}," --runtime",{"type":46,"tag":455,"props":664,"children":665},{"style":467},[666],{"type":51,"value":667}," nvidia",{"type":46,"tag":455,"props":669,"children":670},{"style":467},[671],{"type":51,"value":672}," --network",{"type":46,"tag":455,"props":674,"children":675},{"style":467},[676],{"type":51,"value":677}," host",{"type":46,"tag":455,"props":679,"children":680},{"style":467},[681],{"type":51,"value":682}," --ipc",{"type":46,"tag":455,"props":684,"children":685},{"style":467},[686],{"type":51,"value":677},{"type":46,"tag":455,"props":688,"children":689},{"style":467},[690],{"type":51,"value":691}," --name",{"type":46,"tag":455,"props":693,"children":694},{"style":467},[695],{"type":51,"value":696}," vllm",{"type":46,"tag":455,"props":698,"children":700},{"style":699},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[701],{"type":51,"value":702}," \\\n",{"type":46,"tag":455,"props":704,"children":706},{"class":457,"line":705},2,[707,712,717,722,727,732],{"type":46,"tag":455,"props":708,"children":709},{"style":467},[710],{"type":51,"value":711},"  -v",{"type":46,"tag":455,"props":713,"children":714},{"style":591},[715],{"type":51,"value":716}," \"",{"type":46,"tag":455,"props":718,"children":719},{"style":699},[720],{"type":51,"value":721},"$HOME",{"type":46,"tag":455,"props":723,"children":724},{"style":467},[725],{"type":51,"value":726},"\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface",{"type":46,"tag":455,"props":728,"children":729},{"style":591},[730],{"type":51,"value":731},"\"",{"type":46,"tag":455,"props":733,"children":734},{"style":699},[735],{"type":51,"value":702},{"type":46,"tag":455,"props":737,"children":739},{"class":457,"line":738},3,[740,745,750,754,759,763],{"type":46,"tag":455,"props":741,"children":742},{"style":467},[743],{"type":51,"value":744},"  -e",{"type":46,"tag":455,"props":746,"children":747},{"style":467},[748],{"type":51,"value":749}," HF_TOKEN=",{"type":46,"tag":455,"props":751,"children":752},{"style":591},[753],{"type":51,"value":731},{"type":46,"tag":455,"props":755,"children":756},{"style":699},[757],{"type":51,"value":758},"$HF_TOKEN",{"type":46,"tag":455,"props":760,"children":761},{"style":591},[762],{"type":51,"value":731},{"type":46,"tag":455,"props":764,"children":765},{"style":699},[766],{"type":51,"value":702},{"type":46,"tag":455,"props":768,"children":770},{"class":457,"line":769},4,[771,776],{"type":46,"tag":455,"props":772,"children":773},{"style":467},[774],{"type":51,"value":775},"  vllm\u002Fvllm-openai:latest",{"type":46,"tag":455,"props":777,"children":778},{"style":699},[779],{"type":51,"value":702},{"type":46,"tag":455,"props":781,"children":783},{"class":457,"line":782},5,[784,789,794,799,804,809,814],{"type":46,"tag":455,"props":785,"children":786},{"style":467},[787],{"type":51,"value":788},"  vllm",{"type":46,"tag":455,"props":790,"children":791},{"style":467},[792],{"type":51,"value":793}," serve",{"type":46,"tag":455,"props":795,"children":796},{"style":591},[797],{"type":51,"value":798}," \u003C",{"type":46,"tag":455,"props":800,"children":801},{"style":467},[802],{"type":51,"value":803},"hf-repo-i",{"type":46,"tag":455,"props":805,"children":806},{"style":699},[807],{"type":51,"value":808},"d",{"type":46,"tag":455,"props":810,"children":811},{"style":591},[812],{"type":51,"value":813},">",{"type":46,"tag":455,"props":815,"children":816},{"style":699},[817],{"type":51,"value":702},{"type":46,"tag":455,"props":819,"children":821},{"class":457,"line":820},6,[822,827,832,837,842],{"type":46,"tag":455,"props":823,"children":824},{"style":467},[825],{"type":51,"value":826},"    --host",{"type":46,"tag":455,"props":828,"children":829},{"style":585},[830],{"type":51,"value":831}," 0.0.0.0",{"type":46,"tag":455,"props":833,"children":834},{"style":467},[835],{"type":51,"value":836}," --port",{"type":46,"tag":455,"props":838,"children":839},{"style":585},[840],{"type":51,"value":841}," 8000",{"type":46,"tag":455,"props":843,"children":844},{"style":699},[845],{"type":51,"value":702},{"type":46,"tag":455,"props":847,"children":849},{"class":457,"line":848},7,[850,855,860],{"type":46,"tag":455,"props":851,"children":852},{"style":467},[853],{"type":51,"value":854},"    --max-model-len",{"type":46,"tag":455,"props":856,"children":857},{"style":585},[858],{"type":51,"value":859}," 8192",{"type":46,"tag":455,"props":861,"children":862},{"style":699},[863],{"type":51,"value":702},{"type":46,"tag":455,"props":865,"children":867},{"class":457,"line":866},8,[868,873,878],{"type":46,"tag":455,"props":869,"children":870},{"style":467},[871],{"type":51,"value":872},"    --gpu-memory-utilization",{"type":46,"tag":455,"props":874,"children":875},{"style":585},[876],{"type":51,"value":877}," 0.75",{"type":46,"tag":455,"props":879,"children":880},{"style":699},[881],{"type":51,"value":702},{"type":46,"tag":455,"props":883,"children":885},{"class":457,"line":884},9,[886,891],{"type":46,"tag":455,"props":887,"children":888},{"style":467},[889],{"type":51,"value":890},"    --tensor-parallel-size",{"type":46,"tag":455,"props":892,"children":893},{"style":585},[894],{"type":51,"value":895}," 1\n",{"type":46,"tag":54,"props":897,"children":898},{},[899,901,906],{"type":51,"value":900},"On Orin JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (",{"type":46,"tag":71,"props":902,"children":904},{"className":903},[],[905],{"type":51,"value":76},{"type":51,"value":907},"). On older Orin releases, use the NVIDIA-AI-IOT container:",{"type":46,"tag":444,"props":909,"children":911},{"className":446,"code":910,"language":448,"meta":449,"style":449},"docker run --rm -it --runtime nvidia --network host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin \\\n  vllm serve \u003Chf-repo-id> \\\n    --host 0.0.0.0 --port 8000 \\\n    --max-model-len 4096 \\\n    --gpu-memory-utilization 0.85 \\\n    --tensor-parallel-size 1\n",[912],{"type":46,"tag":71,"props":913,"children":914},{"__ignoreMap":449},[915,962,989,1016,1028,1059,1082,1098,1114],{"type":46,"tag":455,"props":916,"children":917},{"class":457,"line":458},[918,922,926,930,934,938,942,946,950,954,958],{"type":46,"tag":455,"props":919,"children":920},{"style":569},[921],{"type":51,"value":642},{"type":46,"tag":455,"props":923,"children":924},{"style":467},[925],{"type":51,"value":647},{"type":46,"tag":455,"props":927,"children":928},{"style":467},[929],{"type":51,"value":652},{"type":46,"tag":455,"props":931,"children":932},{"style":467},[933],{"type":51,"value":657},{"type":46,"tag":455,"props":935,"children":936},{"style":467},[937],{"type":51,"value":662},{"type":46,"tag":455,"props":939,"children":940},{"style":467},[941],{"type":51,"value":667},{"type":46,"tag":455,"props":943,"children":944},{"style":467},[945],{"type":51,"value":672},{"type":46,"tag":455,"props":947,"children":948},{"style":467},[949],{"type":51,"value":677},{"type":46,"tag":455,"props":951,"children":952},{"style":467},[953],{"type":51,"value":691},{"type":46,"tag":455,"props":955,"children":956},{"style":467},[957],{"type":51,"value":696},{"type":46,"tag":455,"props":959,"children":960},{"style":699},[961],{"type":51,"value":702},{"type":46,"tag":455,"props":963,"children":964},{"class":457,"line":705},[965,969,973,977,981,985],{"type":46,"tag":455,"props":966,"children":967},{"style":467},[968],{"type":51,"value":711},{"type":46,"tag":455,"props":970,"children":971},{"style":591},[972],{"type":51,"value":716},{"type":46,"tag":455,"props":974,"children":975},{"style":699},[976],{"type":51,"value":721},{"type":46,"tag":455,"props":978,"children":979},{"style":467},[980],{"type":51,"value":726},{"type":46,"tag":455,"props":982,"children":983},{"style":591},[984],{"type":51,"value":731},{"type":46,"tag":455,"props":986,"children":987},{"style":699},[988],{"type":51,"value":702},{"type":46,"tag":455,"props":990,"children":991},{"class":457,"line":738},[992,996,1000,1004,1008,1012],{"type":46,"tag":455,"props":993,"children":994},{"style":467},[995],{"type":51,"value":744},{"type":46,"tag":455,"props":997,"children":998},{"style":467},[999],{"type":51,"value":749},{"type":46,"tag":455,"props":1001,"children":1002},{"style":591},[1003],{"type":51,"value":731},{"type":46,"tag":455,"props":1005,"children":1006},{"style":699},[1007],{"type":51,"value":758},{"type":46,"tag":455,"props":1009,"children":1010},{"style":591},[1011],{"type":51,"value":731},{"type":46,"tag":455,"props":1013,"children":1014},{"style":699},[1015],{"type":51,"value":702},{"type":46,"tag":455,"props":1017,"children":1018},{"class":457,"line":769},[1019,1024],{"type":46,"tag":455,"props":1020,"children":1021},{"style":467},[1022],{"type":51,"value":1023},"  ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin",{"type":46,"tag":455,"props":1025,"children":1026},{"style":699},[1027],{"type":51,"value":702},{"type":46,"tag":455,"props":1029,"children":1030},{"class":457,"line":782},[1031,1035,1039,1043,1047,1051,1055],{"type":46,"tag":455,"props":1032,"children":1033},{"style":467},[1034],{"type":51,"value":788},{"type":46,"tag":455,"props":1036,"children":1037},{"style":467},[1038],{"type":51,"value":793},{"type":46,"tag":455,"props":1040,"children":1041},{"style":591},[1042],{"type":51,"value":798},{"type":46,"tag":455,"props":1044,"children":1045},{"style":467},[1046],{"type":51,"value":803},{"type":46,"tag":455,"props":1048,"children":1049},{"style":699},[1050],{"type":51,"value":808},{"type":46,"tag":455,"props":1052,"children":1053},{"style":591},[1054],{"type":51,"value":813},{"type":46,"tag":455,"props":1056,"children":1057},{"style":699},[1058],{"type":51,"value":702},{"type":46,"tag":455,"props":1060,"children":1061},{"class":457,"line":820},[1062,1066,1070,1074,1078],{"type":46,"tag":455,"props":1063,"children":1064},{"style":467},[1065],{"type":51,"value":826},{"type":46,"tag":455,"props":1067,"children":1068},{"style":585},[1069],{"type":51,"value":831},{"type":46,"tag":455,"props":1071,"children":1072},{"style":467},[1073],{"type":51,"value":836},{"type":46,"tag":455,"props":1075,"children":1076},{"style":585},[1077],{"type":51,"value":841},{"type":46,"tag":455,"props":1079,"children":1080},{"style":699},[1081],{"type":51,"value":702},{"type":46,"tag":455,"props":1083,"children":1084},{"class":457,"line":848},[1085,1089,1094],{"type":46,"tag":455,"props":1086,"children":1087},{"style":467},[1088],{"type":51,"value":854},{"type":46,"tag":455,"props":1090,"children":1091},{"style":585},[1092],{"type":51,"value":1093}," 4096",{"type":46,"tag":455,"props":1095,"children":1096},{"style":699},[1097],{"type":51,"value":702},{"type":46,"tag":455,"props":1099,"children":1100},{"class":457,"line":866},[1101,1105,1110],{"type":46,"tag":455,"props":1102,"children":1103},{"style":467},[1104],{"type":51,"value":872},{"type":46,"tag":455,"props":1106,"children":1107},{"style":585},[1108],{"type":51,"value":1109}," 0.85",{"type":46,"tag":455,"props":1111,"children":1112},{"style":699},[1113],{"type":51,"value":702},{"type":46,"tag":455,"props":1115,"children":1116},{"class":457,"line":884},[1117,1121],{"type":46,"tag":455,"props":1118,"children":1119},{"style":467},[1120],{"type":51,"value":890},{"type":46,"tag":455,"props":1122,"children":1123},{"style":585},[1124],{"type":51,"value":895},{"type":46,"tag":54,"props":1126,"children":1127},{},[1128,1133,1135,1141,1143,1148],{"type":46,"tag":71,"props":1129,"children":1131},{"className":1130},[],[1132],{"type":51,"value":183},{"type":51,"value":1134}," is required only for gated\u002Fprivate Hugging Face models; omit the ",{"type":46,"tag":71,"props":1136,"children":1138},{"className":1137},[],[1139],{"type":51,"value":1140},"-e HF_TOKEN=\"$HF_TOKEN\"",{"type":51,"value":1142}," line for public models that do not need Hub authentication. Passing ",{"type":46,"tag":71,"props":1144,"children":1146},{"className":1145},[],[1147],{"type":51,"value":183},{"type":51,"value":1149}," as an environment variable can expose it through Docker inspect output, process metadata, or logs on shared systems. Prefer the narrowest-scoped token possible, rotate\u002Frevoke it after shared-container use, and use a mounted credential file or Docker secret when the deployment environment supports that pattern.",{"type":46,"tag":54,"props":1151,"children":1152},{},[1153,1155,1161,1163,1169],{"type":51,"value":1154},"Wait for ",{"type":46,"tag":71,"props":1156,"children":1158},{"className":1157},[],[1159],{"type":51,"value":1160},"Application startup complete.",{"type":51,"value":1162}," Server is on ",{"type":46,"tag":71,"props":1164,"children":1166},{"className":1165},[],[1167],{"type":51,"value":1168},"http:\u002F\u002F0.0.0.0:8000\u002Fv1",{"type":51,"value":262},{"type":46,"tag":54,"props":1171,"children":1172},{},[1173,1175,1180],{"type":51,"value":1174},"For SGLang on Thor, use NVIDIA SGLang 26.01 (",{"type":46,"tag":71,"props":1176,"children":1178},{"className":1177},[],[1179],{"type":51,"value":84},{"type":51,"value":1181},"), which packages SGLang 0.5.5.post2 and lists Jetson Thor support. Do not judge Thor SGLang support from older prerelease SGLang results:",{"type":46,"tag":444,"props":1183,"children":1185},{"className":446,"code":1184,"language":448,"meta":449,"style":449},"docker run --rm -it --runtime nvidia --network host --ipc host --name sglang \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3 \\\n  python3 -m sglang.launch_server \\\n    --model-path \u003Chf-repo-id> \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    --mem-fraction-static 0.60 \\\n    --max-running-requests 8\n",[1186],{"type":46,"tag":71,"props":1187,"children":1188},{"__ignoreMap":449},[1189,1245,1272,1299,1311,1332,1360,1375,1391,1408],{"type":46,"tag":455,"props":1190,"children":1191},{"class":457,"line":458},[1192,1196,1200,1204,1208,1212,1216,1220,1224,1228,1232,1236,1241],{"type":46,"tag":455,"props":1193,"children":1194},{"style":569},[1195],{"type":51,"value":642},{"type":46,"tag":455,"props":1197,"children":1198},{"style":467},[1199],{"type":51,"value":647},{"type":46,"tag":455,"props":1201,"children":1202},{"style":467},[1203],{"type":51,"value":652},{"type":46,"tag":455,"props":1205,"children":1206},{"style":467},[1207],{"type":51,"value":657},{"type":46,"tag":455,"props":1209,"children":1210},{"style":467},[1211],{"type":51,"value":662},{"type":46,"tag":455,"props":1213,"children":1214},{"style":467},[1215],{"type":51,"value":667},{"type":46,"tag":455,"props":1217,"children":1218},{"style":467},[1219],{"type":51,"value":672},{"type":46,"tag":455,"props":1221,"children":1222},{"style":467},[1223],{"type":51,"value":677},{"type":46,"tag":455,"props":1225,"children":1226},{"style":467},[1227],{"type":51,"value":682},{"type":46,"tag":455,"props":1229,"children":1230},{"style":467},[1231],{"type":51,"value":677},{"type":46,"tag":455,"props":1233,"children":1234},{"style":467},[1235],{"type":51,"value":691},{"type":46,"tag":455,"props":1237,"children":1238},{"style":467},[1239],{"type":51,"value":1240}," sglang",{"type":46,"tag":455,"props":1242,"children":1243},{"style":699},[1244],{"type":51,"value":702},{"type":46,"tag":455,"props":1246,"children":1247},{"class":457,"line":705},[1248,1252,1256,1260,1264,1268],{"type":46,"tag":455,"props":1249,"children":1250},{"style":467},[1251],{"type":51,"value":711},{"type":46,"tag":455,"props":1253,"children":1254},{"style":591},[1255],{"type":51,"value":716},{"type":46,"tag":455,"props":1257,"children":1258},{"style":699},[1259],{"type":51,"value":721},{"type":46,"tag":455,"props":1261,"children":1262},{"style":467},[1263],{"type":51,"value":726},{"type":46,"tag":455,"props":1265,"children":1266},{"style":591},[1267],{"type":51,"value":731},{"type":46,"tag":455,"props":1269,"children":1270},{"style":699},[1271],{"type":51,"value":702},{"type":46,"tag":455,"props":1273,"children":1274},{"class":457,"line":738},[1275,1279,1283,1287,1291,1295],{"type":46,"tag":455,"props":1276,"children":1277},{"style":467},[1278],{"type":51,"value":744},{"type":46,"tag":455,"props":1280,"children":1281},{"style":467},[1282],{"type":51,"value":749},{"type":46,"tag":455,"props":1284,"children":1285},{"style":591},[1286],{"type":51,"value":731},{"type":46,"tag":455,"props":1288,"children":1289},{"style":699},[1290],{"type":51,"value":758},{"type":46,"tag":455,"props":1292,"children":1293},{"style":591},[1294],{"type":51,"value":731},{"type":46,"tag":455,"props":1296,"children":1297},{"style":699},[1298],{"type":51,"value":702},{"type":46,"tag":455,"props":1300,"children":1301},{"class":457,"line":769},[1302,1307],{"type":46,"tag":455,"props":1303,"children":1304},{"style":467},[1305],{"type":51,"value":1306},"  nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3",{"type":46,"tag":455,"props":1308,"children":1309},{"style":699},[1310],{"type":51,"value":702},{"type":46,"tag":455,"props":1312,"children":1313},{"class":457,"line":782},[1314,1319,1323,1328],{"type":46,"tag":455,"props":1315,"children":1316},{"style":467},[1317],{"type":51,"value":1318},"  python3",{"type":46,"tag":455,"props":1320,"children":1321},{"style":467},[1322],{"type":51,"value":582},{"type":46,"tag":455,"props":1324,"children":1325},{"style":467},[1326],{"type":51,"value":1327}," sglang.launch_server",{"type":46,"tag":455,"props":1329,"children":1330},{"style":699},[1331],{"type":51,"value":702},{"type":46,"tag":455,"props":1333,"children":1334},{"class":457,"line":820},[1335,1340,1344,1348,1352,1356],{"type":46,"tag":455,"props":1336,"children":1337},{"style":467},[1338],{"type":51,"value":1339},"    --model-path",{"type":46,"tag":455,"props":1341,"children":1342},{"style":591},[1343],{"type":51,"value":798},{"type":46,"tag":455,"props":1345,"children":1346},{"style":467},[1347],{"type":51,"value":803},{"type":46,"tag":455,"props":1349,"children":1350},{"style":699},[1351],{"type":51,"value":808},{"type":46,"tag":455,"props":1353,"children":1354},{"style":591},[1355],{"type":51,"value":813},{"type":46,"tag":455,"props":1357,"children":1358},{"style":699},[1359],{"type":51,"value":702},{"type":46,"tag":455,"props":1361,"children":1362},{"class":457,"line":848},[1363,1367,1371],{"type":46,"tag":455,"props":1364,"children":1365},{"style":467},[1366],{"type":51,"value":826},{"type":46,"tag":455,"props":1368,"children":1369},{"style":585},[1370],{"type":51,"value":831},{"type":46,"tag":455,"props":1372,"children":1373},{"style":699},[1374],{"type":51,"value":702},{"type":46,"tag":455,"props":1376,"children":1377},{"class":457,"line":866},[1378,1383,1387],{"type":46,"tag":455,"props":1379,"children":1380},{"style":467},[1381],{"type":51,"value":1382},"    --port",{"type":46,"tag":455,"props":1384,"children":1385},{"style":585},[1386],{"type":51,"value":841},{"type":46,"tag":455,"props":1388,"children":1389},{"style":699},[1390],{"type":51,"value":702},{"type":46,"tag":455,"props":1392,"children":1393},{"class":457,"line":884},[1394,1399,1404],{"type":46,"tag":455,"props":1395,"children":1396},{"style":467},[1397],{"type":51,"value":1398},"    --mem-fraction-static",{"type":46,"tag":455,"props":1400,"children":1401},{"style":585},[1402],{"type":51,"value":1403}," 0.60",{"type":46,"tag":455,"props":1405,"children":1406},{"style":699},[1407],{"type":51,"value":702},{"type":46,"tag":455,"props":1409,"children":1411},{"class":457,"line":1410},10,[1412,1417],{"type":46,"tag":455,"props":1413,"children":1414},{"style":467},[1415],{"type":51,"value":1416},"    --max-running-requests",{"type":46,"tag":455,"props":1418,"children":1419},{"style":585},[1420],{"type":51,"value":1421}," 8\n",{"type":46,"tag":54,"props":1423,"children":1424},{},[1425],{"type":51,"value":1426},"Use SGLang when the user needs RAG\u002Ftool-use workflows, structured generation, or\nSGLang-specific scheduling. For plain high-throughput OpenAI-compatible serving,\nprefer vLLM unless the user asks for SGLang.",{"type":46,"tag":1428,"props":1429,"children":1431},"h3",{"id":1430},"sku-appropriate-defaults",[1432],{"type":51,"value":1433},"SKU-appropriate defaults",{"type":46,"tag":350,"props":1435,"children":1436},{},[1437,1458],{"type":46,"tag":354,"props":1438,"children":1439},{},[1440],{"type":46,"tag":358,"props":1441,"children":1442},{},[1443,1448,1453],{"type":46,"tag":362,"props":1444,"children":1445},{},[1446],{"type":51,"value":1447},"Knob",{"type":46,"tag":362,"props":1449,"children":1450},{},[1451],{"type":51,"value":1452},"Orin Nano \u002F NX",{"type":46,"tag":362,"props":1454,"children":1455},{},[1456],{"type":51,"value":1457},"AGX Orin \u002F Thor",{"type":46,"tag":373,"props":1459,"children":1460},{},[1461,1491,1520],{"type":46,"tag":358,"props":1462,"children":1463},{},[1464,1473,1482],{"type":46,"tag":380,"props":1465,"children":1466},{},[1467],{"type":46,"tag":71,"props":1468,"children":1470},{"className":1469},[],[1471],{"type":51,"value":1472},"--max-model-len",{"type":46,"tag":380,"props":1474,"children":1475},{},[1476],{"type":46,"tag":71,"props":1477,"children":1479},{"className":1478},[],[1480],{"type":51,"value":1481},"4096",{"type":46,"tag":380,"props":1483,"children":1484},{},[1485],{"type":46,"tag":71,"props":1486,"children":1488},{"className":1487},[],[1489],{"type":51,"value":1490},"8192",{"type":46,"tag":358,"props":1492,"children":1493},{},[1494,1503,1512],{"type":46,"tag":380,"props":1495,"children":1496},{},[1497],{"type":46,"tag":71,"props":1498,"children":1500},{"className":1499},[],[1501],{"type":51,"value":1502},"--gpu-memory-utilization",{"type":46,"tag":380,"props":1504,"children":1505},{},[1506],{"type":46,"tag":71,"props":1507,"children":1509},{"className":1508},[],[1510],{"type":51,"value":1511},"0.85",{"type":46,"tag":380,"props":1513,"children":1514},{},[1515],{"type":46,"tag":71,"props":1516,"children":1518},{"className":1517},[],[1519],{"type":51,"value":1511},{"type":46,"tag":358,"props":1521,"children":1522},{},[1523,1532,1541],{"type":46,"tag":380,"props":1524,"children":1525},{},[1526],{"type":46,"tag":71,"props":1527,"children":1529},{"className":1528},[],[1530],{"type":51,"value":1531},"--tensor-parallel-size",{"type":46,"tag":380,"props":1533,"children":1534},{},[1535],{"type":46,"tag":71,"props":1536,"children":1538},{"className":1537},[],[1539],{"type":51,"value":1540},"1",{"type":46,"tag":380,"props":1542,"children":1543},{},[1544],{"type":46,"tag":71,"props":1545,"children":1547},{"className":1546},[],[1548],{"type":51,"value":1540},{"type":46,"tag":54,"props":1550,"children":1551},{},[1552,1554,1559,1561,1566],{"type":51,"value":1553},"If the server OOMs at startup, lower ",{"type":46,"tag":71,"props":1555,"children":1557},{"className":1556},[],[1558],{"type":51,"value":1502},{"type":51,"value":1560}," by 0.05 and re-launch (or run ",{"type":46,"tag":71,"props":1562,"children":1564},{"className":1563},[],[1565],{"type":51,"value":130},{"type":51,"value":1567}," for a workload-aware recommendation).",{"type":46,"tag":88,"props":1569,"children":1571},{"id":1570},"quantization-preferences-matters-more-than-the-runtime",[1572],{"type":51,"value":1573},"Quantization preferences (matters more than the runtime)",{"type":46,"tag":54,"props":1575,"children":1576},{},[1577],{"type":51,"value":1578},"For vLLM and SGLang, choose checkpoint formats by Jetson family:",{"type":46,"tag":350,"props":1580,"children":1581},{},[1582,1602],{"type":46,"tag":354,"props":1583,"children":1584},{},[1585],{"type":46,"tag":358,"props":1586,"children":1587},{},[1588,1592,1597],{"type":46,"tag":362,"props":1589,"children":1590},{},[1591],{"type":51,"value":366},{"type":46,"tag":362,"props":1593,"children":1594},{},[1595],{"type":51,"value":1596},"First choice",{"type":46,"tag":362,"props":1598,"children":1599},{},[1600],{"type":51,"value":1601},"Acceptable fallback",{"type":46,"tag":373,"props":1603,"children":1604},{},[1605,1628,1647],{"type":46,"tag":358,"props":1606,"children":1607},{},[1608,1613,1623],{"type":46,"tag":380,"props":1609,"children":1610},{},[1611],{"type":51,"value":1612},"Thor",{"type":46,"tag":380,"props":1614,"children":1615},{},[1616,1621],{"type":46,"tag":326,"props":1617,"children":1618},{},[1619],{"type":51,"value":1620},"NVFP4",{"type":51,"value":1622}," when the model\u002Fruntime supports it",{"type":46,"tag":380,"props":1624,"children":1625},{},[1626],{"type":51,"value":1627},"W4A16",{"type":46,"tag":358,"props":1629,"children":1630},{},[1631,1635,1642],{"type":46,"tag":380,"props":1632,"children":1633},{},[1634],{"type":51,"value":1452},{"type":46,"tag":380,"props":1636,"children":1637},{},[1638],{"type":46,"tag":326,"props":1639,"children":1640},{},[1641],{"type":51,"value":1627},{"type":46,"tag":380,"props":1643,"children":1644},{},[1645],{"type":51,"value":1646},"AWQ or GPTQ 4-bit",{"type":46,"tag":358,"props":1648,"children":1649},{},[1650,1655,1662],{"type":46,"tag":380,"props":1651,"children":1652},{},[1653],{"type":51,"value":1654},"AGX Orin",{"type":46,"tag":380,"props":1656,"children":1657},{},[1658],{"type":46,"tag":326,"props":1659,"children":1660},{},[1661],{"type":51,"value":1627},{"type":46,"tag":380,"props":1663,"children":1664},{},[1665],{"type":51,"value":1646},{"type":46,"tag":54,"props":1667,"children":1668},{},[1669,1671,1677,1679,1684],{"type":51,"value":1670},"For llama.cpp and ",{"type":46,"tag":71,"props":1672,"children":1674},{"className":1673},[],[1675],{"type":51,"value":1676},"Ollama",{"type":51,"value":1678},", use GGUF model quantization names instead: recommend ",{"type":46,"tag":326,"props":1680,"children":1681},{},[1682],{"type":51,"value":1683},"INT4 \u002F Q4_K_M GGUF",{"type":51,"value":1685}," on both Orin and Thor, and choose a smaller INT4 GGUF model if memory is tight. Do not call GGUF Q4_K_M a W4A16\u002FAWQ\u002FGPTQ model. NVFP4 is Thor-preferred and Thor-tuned for runtimes that support it.",{"type":46,"tag":88,"props":1687,"children":1689},{"id":1688},"vlm-mode",[1690],{"type":51,"value":1691},"VLM mode",{"type":46,"tag":54,"props":1693,"children":1694},{},[1695,1697,1703,1705,1715,1717,1722],{"type":51,"value":1696},"VLMs use the same flow as LLMs: same container, same ",{"type":46,"tag":71,"props":1698,"children":1700},{"className":1699},[],[1701],{"type":51,"value":1702},"vllm serve",{"type":51,"value":1704}," invocation, different vision-language checkpoint. The container handles image preprocessing. For a VLM-specific browser UI, use the ",{"type":46,"tag":60,"props":1706,"children":1708},{"href":335,"rel":1707},[64],[1709],{"type":46,"tag":71,"props":1710,"children":1712},{"className":1711},[],[1713],{"type":51,"value":1714},"live-vlm-webui",{"type":51,"value":1716}," container; for a generic chat UI for either, use Open WebUI pointed at ",{"type":46,"tag":71,"props":1718,"children":1720},{"className":1719},[],[1721],{"type":51,"value":273},{"type":51,"value":262},{"type":46,"tag":88,"props":1724,"children":1726},{"id":1725},"do-not-fabricate-device-capacity",[1727],{"type":51,"value":1728},"Do not fabricate device capacity",{"type":46,"tag":54,"props":1730,"children":1731},{},[1732,1734,1739,1741,1747],{"type":51,"value":1733},"Do not invent RAM totals, free-memory values, model sizes, JetPack versions, or\nSKU\u002Fvariant names when giving a serving recipe. If capacity matters, either run\nthe live pre-flight checks (when execution is allowed) or hand off to\n",{"type":46,"tag":71,"props":1735,"children":1737},{"className":1736},[],[1738],{"type":51,"value":130},{"type":51,"value":1740}," \u002F ",{"type":46,"tag":71,"props":1742,"children":1744},{"className":1743},[],[1745],{"type":51,"value":1746},"jetson-memory-audit",{"type":51,"value":1748},". If live data is not\navailable, say the value is unknown and provide conservative defaults instead\nof quoting a made-up number.",{"type":46,"tag":88,"props":1750,"children":1752},{"id":1751},"pre-flight-checklist-the-agent-should-verify-before-running-step-3",[1753],{"type":51,"value":1754},"Pre-flight checklist (the agent should verify before running Step 3)",{"type":46,"tag":106,"props":1756,"children":1759},{"className":1757},[1758],"contains-task-list",[1760,1789,1829,1854,1887,1920,1944],{"type":46,"tag":110,"props":1761,"children":1764},{"className":1762},[1763],"task-list-item",[1765,1771,1773,1779,1781,1787],{"type":46,"tag":1766,"props":1767,"children":1770},"input",{"disabled":1768,"type":1769},true,"checkbox",[],{"type":51,"value":1772}," On a Jetson (",{"type":46,"tag":71,"props":1774,"children":1776},{"className":1775},[],[1777],{"type":51,"value":1778},"\u002Fproc\u002Fdevice-tree\u002Fmodel",{"type":51,"value":1780}," contains ",{"type":46,"tag":71,"props":1782,"children":1784},{"className":1783},[],[1785],{"type":51,"value":1786},"NVIDIA Jetson",{"type":51,"value":1788},").",{"type":46,"tag":110,"props":1790,"children":1792},{"className":1791},[1763],[1793,1796,1798,1804,1806,1812,1813,1819,1821,1827],{"type":46,"tag":1766,"props":1794,"children":1795},{"disabled":1768,"type":1769},[],{"type":51,"value":1797}," ",{"type":46,"tag":71,"props":1799,"children":1801},{"className":1800},[],[1802],{"type":51,"value":1803},"nvpmodel -q",{"type":51,"value":1805}," reports a recognized max-performance mode: ",{"type":46,"tag":71,"props":1807,"children":1809},{"className":1808},[],[1810],{"type":51,"value":1811},"MAXN",{"type":51,"value":164},{"type":46,"tag":71,"props":1814,"children":1816},{"className":1815},[],[1817],{"type":51,"value":1818},"MAXN_*",{"type":51,"value":1820}," such as ",{"type":46,"tag":71,"props":1822,"children":1824},{"className":1823},[],[1825],{"type":51,"value":1826},"MAXN_SUPER",{"type":51,"value":1828},". Wattage-named modes should be reported as warnings unless the user explicitly confirms they are the intended benchmark mode for that device.",{"type":46,"tag":110,"props":1830,"children":1832},{"className":1831},[1763],[1833,1836,1838,1844,1846,1852],{"type":46,"tag":1766,"props":1834,"children":1835},{"disabled":1768,"type":1769},[],{"type":51,"value":1837}," On Thor, check whether MIG is enabled before launching (",{"type":46,"tag":71,"props":1839,"children":1841},{"className":1840},[],[1842],{"type":51,"value":1843},"nvidia-smi -L",{"type":51,"value":1845}," and ",{"type":46,"tag":71,"props":1847,"children":1849},{"className":1848},[],[1850],{"type":51,"value":1851},"nvidia-smi mig -lgi",{"type":51,"value":1853},"). If MIG is enabled, warn that vLLM\u002FSGLang may see only a MIG slice or no CUDA device.",{"type":46,"tag":110,"props":1855,"children":1857},{"className":1856},[1763],[1858,1861,1863,1869,1871,1877,1879,1885],{"type":46,"tag":1766,"props":1859,"children":1860},{"disabled":1768,"type":1769},[],{"type":51,"value":1862}," On Thor with MIG or display\u002Fcamera contention, inspect GPU users with ",{"type":46,"tag":71,"props":1864,"children":1866},{"className":1865},[],[1867],{"type":51,"value":1868},"sudo lsof \u002Fdev\u002Fnvidia*",{"type":51,"value":1870},". Display managers, ",{"type":46,"tag":71,"props":1872,"children":1874},{"className":1873},[],[1875],{"type":51,"value":1876},"Xorg",{"type":51,"value":1878},"\u002FGNOME, or ",{"type":46,"tag":71,"props":1880,"children":1882},{"className":1881},[],[1883],{"type":51,"value":1884},"nvargus-daemon",{"type":51,"value":1886}," may hold GPU device files; do not stop services or change MIG mode unless the user explicitly approves.",{"type":46,"tag":110,"props":1888,"children":1890},{"className":1889},[1763],[1891,1894,1896,1902,1904,1910,1912,1918],{"type":46,"tag":1766,"props":1892,"children":1893},{"disabled":1768,"type":1769},[],{"type":51,"value":1895}," No container named ",{"type":46,"tag":71,"props":1897,"children":1899},{"className":1898},[],[1900],{"type":51,"value":1901},"vllm",{"type":51,"value":1903}," already running (",{"type":46,"tag":71,"props":1905,"children":1907},{"className":1906},[],[1908],{"type":51,"value":1909},"docker ps --format '{{.Names}}'",{"type":51,"value":1911},"); otherwise ",{"type":46,"tag":71,"props":1913,"children":1915},{"className":1914},[],[1916],{"type":51,"value":1917},"docker rm -f vllm",{"type":51,"value":1919}," first.",{"type":46,"tag":110,"props":1921,"children":1923},{"className":1922},[1763],[1924,1927,1929,1935,1937,1943],{"type":46,"tag":1766,"props":1925,"children":1926},{"disabled":1768,"type":1769},[],{"type":51,"value":1928}," Docker exposes the NVIDIA runtime (",{"type":46,"tag":71,"props":1930,"children":1932},{"className":1931},[],[1933],{"type":51,"value":1934},"docker info | grep -i 'runtimes.*nvidia'",{"type":51,"value":1936},"), or a GPU-enabled container can run ",{"type":46,"tag":71,"props":1938,"children":1940},{"className":1939},[],[1941],{"type":51,"value":1942},"nvidia-smi",{"type":51,"value":262},{"type":46,"tag":110,"props":1945,"children":1947},{"className":1946},[1763],[1948,1951,1952,1958,1960,1965],{"type":46,"tag":1766,"props":1949,"children":1950},{"disabled":1768,"type":1769},[],{"type":51,"value":1797},{"type":46,"tag":71,"props":1953,"children":1955},{"className":1954},[],[1956],{"type":51,"value":1957},"~\u002F.cache\u002Fhuggingface",{"type":51,"value":1959}," exists; ",{"type":46,"tag":71,"props":1961,"children":1963},{"className":1962},[],[1964],{"type":51,"value":183},{"type":51,"value":1966}," is set if the model is gated.",{"type":46,"tag":88,"props":1968,"children":1970},{"id":1969},"limitations",[1971],{"type":51,"value":1972},"Limitations",{"type":46,"tag":106,"props":1974,"children":1975},{},[1976,1981,1994,2006,2011,2016,2035],{"type":46,"tag":110,"props":1977,"children":1978},{},[1979],{"type":51,"value":1980},"This skill provides serving commands and pre-flight checks; it does not benchmark the deployed server.",{"type":46,"tag":110,"props":1982,"children":1983},{},[1984,1986,1992],{"type":51,"value":1985},"Container tags such as ",{"type":46,"tag":71,"props":1987,"children":1989},{"className":1988},[],[1990],{"type":51,"value":1991},"latest",{"type":51,"value":1993}," are mutable. For release or compliance deployments, pin a digest and record it with the deployment notes.",{"type":46,"tag":110,"props":1995,"children":1996},{},[1997,1999,2004],{"type":51,"value":1998},"vLLM and SGLang memory limits still depend on model architecture, quantization, context length, and concurrent request count. Use ",{"type":46,"tag":71,"props":2000,"children":2002},{"className":2001},[],[2003],{"type":51,"value":130},{"type":51,"value":2005}," when a command OOMs or memory headroom matters.",{"type":46,"tag":110,"props":2007,"children":2008},{},[2009],{"type":51,"value":2010},"Thor vLLM requires upstream vLLM 0.20+ or newer. Older upstream vLLM images may not support Thor \u002F SM 11.0 correctly.",{"type":46,"tag":110,"props":2012,"children":2013},{},[2014],{"type":51,"value":2015},"Thor SGLang should use NVIDIA SGLang 26.01 or newer release notes that explicitly list Jetson Thor support. NVIDIA SGLang 26.01 contains SGLang 0.5.5.post2.",{"type":46,"tag":110,"props":2017,"children":2018},{},[2019,2021,2027,2028,2033],{"type":51,"value":2020},"On Thor, MIG, desktop display, or camera services can hide the full GPU from containers. This skill should detect and warn only; disabling MIG or stopping services such as ",{"type":46,"tag":71,"props":2022,"children":2024},{"className":2023},[],[2025],{"type":51,"value":2026},"gdm3",{"type":51,"value":164},{"type":46,"tag":71,"props":2029,"children":2031},{"className":2030},[],[2032],{"type":51,"value":1884},{"type":51,"value":2034}," requires explicit user approval.",{"type":46,"tag":110,"props":2036,"children":2037},{},[2038],{"type":51,"value":2039},"Host-native vLLM\u002FSGLang on Thor should be used only when that install is already validated on the target JetPack.",{"type":46,"tag":88,"props":2041,"children":2043},{"id":2042},"hand-off-to",[2044],{"type":51,"value":2045},"Hand off to",{"type":46,"tag":106,"props":2047,"children":2048},{},[2049,2060,2086],{"type":46,"tag":110,"props":2050,"children":2051},{},[2052,2058],{"type":46,"tag":71,"props":2053,"children":2055},{"className":2054},[],[2056],{"type":51,"value":2057},"jetson-llm-benchmark",{"type":51,"value":2059}," to actually measure the deployed server.",{"type":46,"tag":110,"props":2061,"children":2062},{},[2063,2069,2071,2077,2079,2084],{"type":46,"tag":71,"props":2064,"children":2066},{"className":2065},[],[2067],{"type":51,"value":2068},"jetson-speculative-decoding",{"type":51,"value":2070}," to add EAGLE-3 \u002F draft-model speculation by appending ",{"type":46,"tag":71,"props":2072,"children":2074},{"className":2073},[],[2075],{"type":51,"value":2076},"--speculative-config '{...}'",{"type":51,"value":2078}," to the ",{"type":46,"tag":71,"props":2080,"children":2082},{"className":2081},[],[2083],{"type":51,"value":1702},{"type":51,"value":2085}," command above.",{"type":46,"tag":110,"props":2087,"children":2088},{},[2089,2094],{"type":46,"tag":71,"props":2090,"children":2092},{"className":2091},[],[2093],{"type":51,"value":130},{"type":51,"value":2095}," if the server OOMs or is memory-bound.",{"type":46,"tag":88,"props":2097,"children":2099},{"id":2098},"source",[2100],{"type":51,"value":2101},"Source",{"type":46,"tag":54,"props":2103,"children":2104},{},[2105,2111,2112,2118],{"type":46,"tag":60,"props":2106,"children":2108},{"href":62,"rel":2107},[64],[2109],{"type":51,"value":2110},"Jetson AI Lab — Introduction to GenAI on Jetson: How to Run LLMs and VLMs",{"type":51,"value":1845},{"type":46,"tag":60,"props":2113,"children":2115},{"href":335,"rel":2114},[64],[2116],{"type":51,"value":2117},"NVIDIA-AI-IOT GHCR packages",{"type":51,"value":262},{"type":46,"tag":2120,"props":2121,"children":2122},"style",{},[2123],{"type":51,"value":2124},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2126,"total":2229},[2127,2144,2158,2172,2184,2201,2215],{"slug":2128,"name":2128,"fn":2129,"description":2130,"org":2131,"tags":2132,"stars":20,"repoUrl":21,"updatedAt":2143},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2133,2136,2139,2140],{"name":2134,"slug":2135,"type":15},"Data Analysis","data-analysis",{"name":2137,"slug":2138,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":2141,"slug":2142,"type":15},"Performance","performance","2026-07-14T05:28:43.176466",{"slug":2145,"name":2145,"fn":2146,"description":2147,"org":2148,"tags":2149,"stars":20,"repoUrl":21,"updatedAt":2157},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2150,2153,2156],{"name":2151,"slug":2152,"type":15},"Deployment","deployment",{"name":2154,"slug":2155,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":2159,"name":2159,"fn":2160,"description":2161,"org":2162,"tags":2163,"stars":20,"repoUrl":21,"updatedAt":2171},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2164,2167,2168],{"name":2165,"slug":2166,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2169,"slug":2170,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":2173,"name":2173,"fn":2174,"description":2175,"org":2176,"tags":2177,"stars":20,"repoUrl":21,"updatedAt":2183},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2178,2179,2180],{"name":2134,"slug":2135,"type":15},{"name":9,"slug":8,"type":15},{"name":2181,"slug":2182,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":2185,"name":2185,"fn":2186,"description":2187,"org":2188,"tags":2189,"stars":20,"repoUrl":21,"updatedAt":2200},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2190,2193,2196,2197],{"name":2191,"slug":2192,"type":15},"Automation","automation",{"name":2194,"slug":2195,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":2198,"slug":2199,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":2202,"name":2202,"fn":2203,"description":2204,"org":2205,"tags":2206,"stars":20,"repoUrl":21,"updatedAt":2214},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2207,2208,2210,2211],{"name":2151,"slug":2152,"type":15},{"name":2209,"slug":642,"type":15},"Docker",{"name":9,"slug":8,"type":15},{"name":2212,"slug":2213,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":2216,"name":2216,"fn":2217,"description":2218,"org":2219,"tags":2220,"stars":20,"repoUrl":21,"updatedAt":2228},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2221,2222,2225],{"name":9,"slug":8,"type":15},{"name":2223,"slug":2224,"type":15},"Quantum Computing","quantum-computing",{"name":2226,"slug":2227,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":2231,"total":2382},[2232,2250,2266,2277,2289,2303,2316,2330,2341,2350,2364,2373],{"slug":2233,"name":2233,"fn":2234,"description":2235,"org":2236,"tags":2237,"stars":2247,"repoUrl":2248,"updatedAt":2249},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2238,2241,2244],{"name":2239,"slug":2240,"type":15},"Documentation","documentation",{"name":2242,"slug":2243,"type":15},"MCP","mcp",{"name":2245,"slug":2246,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2251,"name":2251,"fn":2252,"description":2253,"org":2254,"tags":2255,"stars":2263,"repoUrl":2264,"updatedAt":2265},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2256,2259,2260],{"name":2257,"slug":2258,"type":15},"Containers","containers",{"name":2151,"slug":2152,"type":15},{"name":2261,"slug":2262,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2267,"name":2267,"fn":2268,"description":2269,"org":2270,"tags":2271,"stars":2263,"repoUrl":2264,"updatedAt":2276},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2272,2275],{"name":2273,"slug":2274,"type":15},"CI\u002FCD","ci-cd",{"name":2151,"slug":2152,"type":15},"2026-07-14T05:25:59.97109",{"slug":2278,"name":2278,"fn":2279,"description":2280,"org":2281,"tags":2282,"stars":2263,"repoUrl":2264,"updatedAt":2288},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2283,2284,2285],{"name":2273,"slug":2274,"type":15},{"name":2151,"slug":2152,"type":15},{"name":2286,"slug":2287,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2290,"name":2290,"fn":2291,"description":2292,"org":2293,"tags":2294,"stars":2263,"repoUrl":2264,"updatedAt":2302},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2295,2298,2299],{"name":2296,"slug":2297,"type":15},"Debugging","debugging",{"name":2286,"slug":2287,"type":15},{"name":2300,"slug":2301,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2304,"name":2304,"fn":2305,"description":2306,"org":2307,"tags":2308,"stars":2263,"repoUrl":2264,"updatedAt":2315},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2309,2312],{"name":2310,"slug":2311,"type":15},"Best Practices","best-practices",{"name":2313,"slug":2314,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2317,"name":2317,"fn":2318,"description":2319,"org":2320,"tags":2321,"stars":2263,"repoUrl":2264,"updatedAt":2329},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2322,2325,2328],{"name":2323,"slug":2324,"type":15},"Machine Learning","machine-learning",{"name":2326,"slug":2327,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2331,"name":2331,"fn":2332,"description":2333,"org":2334,"tags":2335,"stars":2263,"repoUrl":2264,"updatedAt":2340},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2336,2339],{"name":2337,"slug":2338,"type":15},"QA","qa",{"name":2181,"slug":2182,"type":15},"2026-07-14T05:25:53.673039",{"slug":2342,"name":2342,"fn":2343,"description":2344,"org":2345,"tags":2346,"stars":2263,"repoUrl":2264,"updatedAt":2349},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2347,2348],{"name":2151,"slug":2152,"type":15},{"name":2154,"slug":2155,"type":15},"2026-07-14T05:25:49.362534",{"slug":2351,"name":2351,"fn":2352,"description":2353,"org":2354,"tags":2355,"stars":2263,"repoUrl":2264,"updatedAt":2363},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2356,2359,2360],{"name":2357,"slug":2358,"type":15},"Code Review","code-review",{"name":2286,"slug":2287,"type":15},{"name":2361,"slug":2362,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2365,"name":2365,"fn":2366,"description":2367,"org":2368,"tags":2369,"stars":2263,"repoUrl":2264,"updatedAt":2372},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2370,2371],{"name":2337,"slug":2338,"type":15},{"name":2181,"slug":2182,"type":15},"2026-07-14T05:25:54.928983",{"slug":2374,"name":2374,"fn":2375,"description":2376,"org":2377,"tags":2378,"stars":2263,"repoUrl":2264,"updatedAt":2381},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2379,2380],{"name":2191,"slug":2192,"type":15},{"name":2273,"slug":2274,"type":15},"2026-07-30T05:29:03.275638",496]