[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-jetson-inference-mem-tune":3,"mdc--i5dug4-key":34,"related-repo-nvidia-jetson-inference-mem-tune":2764,"related-org-nvidia-jetson-inference-mem-tune":2867},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"jetson-inference-mem-tune","optimize LLM memory usage on NVIDIA Jetson","Pick the serving stack and per-runtime memory flags (vLLM, SGLang, llama.cpp, TensorRT Edge-LLM) for an LLM\u002FVLM workload on any NVIDIA Jetson.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":17,"slug":18,"type":15},"LLM","llm",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Edge","edge",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:26:37.799148","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fjetson-inference-mem-tune","---\nname: jetson-inference-mem-tune\ndescription: Pick the serving stack and per-runtime memory flags (vLLM, SGLang, llama.cpp, TensorRT Edge-LLM) for an LLM\u002FVLM workload on any NVIDIA Jetson.\nversion: 0.0.1\nlicense: \"Apache-2.0\"\nmetadata:\n  author: \"Jetson Team\"\n  tags: [jetson, inference, memory]\n  languages: [python]\n  data-classification: public\n---\n\n# Jetson Inference Memory Tuning\n\nRecommends an inference runtime and the specific memory-related flags to pass to it, given the Jetson SKU\u002Fvariant and the user's workload. Does not include quantization recipe selection — that lives in the model-benchmarking skill — but it does point at the precision floor each runtime can serve efficiently.\n\n## Purpose\n\nTurn a live `jetson-memory-audit` snapshot into runtime and launch-flag recommendations for LLM\u002FVLM serving on Jetson. Use this when the user needs to fit a model, reduce OOM risk, or switch to a lower-memory serving stack.\n\n## When to use\n\n- \"Which serving stack should I use on Orin Nano 8 GB to run a 7B model?\"\n- \"vLLM is OOMing — what should `--gpu-memory-utilization` and `--max-model-len` be?\"\n- \"Same model, less memory — can I switch from vLLM to llama.cpp?\"\n- After `jetson-memory-audit` shows a model server is the top NvMap \u002F PSS consumer.\n\n## Prerequisites\n\n- Start with a current `jetson-memory-audit\u002Fscripts\u002Faudit.sh` JSON snapshot from the target Jetson.\n- Know the intended workload: `llm-server`, `vlm-server`, `embedding`, or `rag`.\n- If the user gives a desired free-memory target, pass it as `--target-mb`; otherwise let the script use SKU defaults.\n\n## Available Scripts\n\n| Script | Purpose | Arguments |\n|--------|---------|-----------|\n| `scripts\u002Frecommend.py` | Reads an audit JSON and emits runtime plus launch-flag recommendations. | `--audit PATH`, `--runtime`, `--workload`, `--target-mb`, `--human`. |\n\nIf your agent runtime supports `run_script`, invoke `run_script(\"scripts\u002Frecommend.py\", [\"--audit\", \"\u002Ftmp\u002Faudit.json\", \"--runtime\", \"auto\", \"--workload\", \"llm-server\"])` and summarize the returned JSON. Otherwise run it with `python3` from the repository root.\n\n## Instructions\n\n1. Run `jetson-memory-audit\u002Fscripts\u002Faudit.sh` to capture the device baseline.\n2. Run `scripts\u002Frecommend.py --audit \u002Ftmp\u002Faudit.json --runtime auto --workload llm-server --target-mb 6000` to get a JSON of runtime + flag recommendations.\n3. The agent presents the suggested runtime and the exact CLI flags. The user (or an outer agent) launches \u002F restarts the server with those flags.\n4. Re-run the audit to verify.\n\n## Expected workflow\n\nUse `scripts\u002Frecommend.py` for the specific prompt and answer from the JSON it emits. If direct execution is blocked, run it as `python3 {baseDir}\u002Fscripts\u002Frecommend.py ...`.\n\n- For vLLM OOM prompts, run with `--runtime vllm --workload llm-server` and include concrete `--gpu-memory-utilization=\u003C0.x>` and `--max-model-len=\u003Cnumber>` values from `launch_flags`.\n- For \"lowest memory\" or Orin Nano 8 GB prompts, run with `--runtime auto --workload llm-server`; prefer the runtime in the JSON and explicitly mention the GGUF \u002F 4-bit tradeoff when it selects `llama-cpp`.\n- For SGLang prompts, run with `--runtime sglang` and quote `--mem-fraction-static`, `--max-running-requests`, and any context\u002FKV-cache note.\n- For \"switch from vLLM to llama.cpp\" prompts, run with `--runtime llama-cpp` and quote `-ngl`, `-c`, and `--no-mmap`.\n\n## Limitations\n\n- Recommendations are only as fresh as the audit JSON. Re-run `jetson-memory-audit` after stopping services, changing power mode, or restarting model servers.\n- The script estimates memory pressure from SKU defaults and audit totals; model-specific KV-cache, quantization, and tokenizer behavior can still require benchmarking.\n- This skill emits flags only. It does not start, stop, or restart model servers.\n\n## Error handling\n\n- Exit `2`: the audit JSON could not be read, parsed, or did not contain valid numeric memory fields. Ask the user to rerun `jetson-memory-audit\u002Fscripts\u002Faudit.sh`.\n- Exit `3`: unsupported runtime or workload request. Re-run with one of the `--runtime` and `--workload` values listed in `scripts\u002Frecommend.py --help`.\n- Empty or missing `launch_flags`: do not invent fallback flags. Report the script failure and ask for a fresh audit or a supported runtime.\n\n## Output contract for `recommend.py`\n\n```json\n{\n  \"sku\": \"orin-nx\",\n  \"variant\": \"orin-nx-16gb\",\n  \"mem_total_gb\": 16,\n  \"runtime\": \"vllm\",\n  \"rationale\": \"Highest throughput at this memory budget given continuous batching + paged attention.\",\n  \"launch_flags\": [\n    \"--gpu-memory-utilization=0.55\",\n    \"--max-model-len=4096\",\n    \"--max-num-seqs=8\",\n    \"--enable-prefix-caching\"\n  ],\n  \"alternatives\": [\n    { \"runtime\": \"llama-cpp\", \"rationale\": \"Lower memory floor with GGUF Q4_K_M.\", \"launch_flags\": [\"-ngl 28\", \"-c 4096\", \"--no-mmap\"] }\n  ],\n  \"notes\": [\"Lower --gpu-memory-utilization further if you also run a small VLM alongside.\"]\n}\n```\n\n## Runtimes covered\n\n| Runtime              | Best for                                                  | Key memory knobs                                                                           | Preferred install path |\n|----------------------|-----------------------------------------------------------|--------------------------------------------------------------------------------------------|------------------------|\n| **llama.cpp**        | Tightest budget; GGUF; Orin Nano-class                    | `-ngl`, `-c`, `--mlock`, `--no-mmap`                                                       | `ghcr.io\u002Fnvidia-ai-iot\u002Fllama_cpp:latest-jetson-{orin,thor}` |\n| **vLLM**             | High-throughput serving with continuous batching          | `--gpu-memory-utilization`, `--max-model-len`, `--max-num-seqs`, `--enable-prefix-caching` | Thor and Orin JetPack 7.2 \u002F L4T r39+: upstream vLLM 0.20+ (`vllm\u002Fvllm-openai`) container or validated native vLLM 0.20+. Older Orin: NVIDIA-AI-IOT image |\n| **SGLang**           | Programmable workflows (RAG, tool use, structured output) | `--mem-fraction-static`, `--mem-fraction-dynamic`, `--max-running-requests`                | Thor: NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2). Orin: JetPack-matched environment |\n| **TensorRT Edge-LLM**| NVIDIA-tuned production serving                           | Build profile per SKU; paged-KV; KV reuse                                                  | Vendor docs for the target JetPack |\n\n> For Orin JetPack 7.2 \u002F L4T r39+, upstream vLLM 0.20+ is supported. For older Orin releases, prefer NVIDIA-AI-IOT prebuilt vLLM images where available because they ship the matching CUDA\u002FcuDNN\u002FTensorRT stack for JetPack. For Thor, prefer upstream vLLM 0.20+ (`vllm\u002Fvllm-openai`) or a validated native vLLM 0.20+ install; for SGLang use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`, SGLang 0.5.5.post2) or newer NVIDIA SGLang release notes that explicitly list Jetson Thor support. Do not force an Orin-specific Jetson container path on Thor, and do not assume native upstream SGLang support on Orin.\n\n## Quantization recommendations\n\nUse runtime-specific quantization names. vLLM and SGLang usually consume Hugging Face checkpoints such as W4A16, AWQ, GPTQ, FP16, or NVFP4. llama.cpp and Ollama consume GGUF models, so recommend INT4\u002FQ4_K_M-style GGUF instead.\n\n| Runtime family | Jetson family | First choice | Fallback |\n|----------------|---------------|--------------|----------|\n| vLLM \u002F SGLang | Thor | NVFP4 when the model\u002Fruntime supports it | W4A16 |\n| vLLM \u002F SGLang | Orin Nano \u002F NX | W4A16 | AWQ or GPTQ 4-bit |\n| vLLM \u002F SGLang | AGX Orin | W4A16 | AWQ or GPTQ 4-bit |\n| llama.cpp \u002F Ollama | Orin and Thor | GGUF INT4 \u002F Q4_K_M | Smaller INT4 GGUF model if memory is tight |\n\nDo not describe GGUF Q4_K_M as W4A16\u002FAWQ\u002FGPTQ. Do not compare Thor NVFP4 results with Orin W4A16 results unless the output includes a `quant` field.\n\n## Runtime command guidance\n\nUse `recommend.py` as the source of truth for memory knobs, then place its `launch_flags` into the matching serving command. Keep the command guidance in this skill instead of separate small reference files so agents ingest one complete instruction set.\n\nFor vLLM on Orin with JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`). On older Orin releases, use the NVIDIA-AI-IOT image:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin \\\n  vllm serve \u003Chf-model-id-or-local-path> \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    --gpu-memory-utilization 0.60 \\\n    --max-model-len 4096 \\\n    --max-num-seqs 8 \\\n    --enable-prefix-caching\n```\n\nFor vLLM on Thor, use upstream vLLM 0.20+ (`vllm\u002Fvllm-openai:latest`) unless host-native vLLM 0.20+ is already installed and validated:\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --ipc host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  vllm\u002Fvllm-openai:latest \\\n  vllm serve \u003Chf-model-id-or-local-path> \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    --gpu-memory-utilization 0.75 \\\n    --max-model-len 8192 \\\n    --max-num-seqs 32 \\\n    --enable-prefix-caching\n```\n\nThor vLLM note: do not judge Thor support from pre-0.20 vLLM results; upstream vLLM support starts at vLLM 0.20+.\n\nFor SGLang on Thor, use NVIDIA SGLang 26.01 (`nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3`). NVIDIA SGLang 26.01 contains SGLang `0.5.5.post2` and explicitly lists Jetson Thor support. Avoid judging Thor support from older prerelease SGLang results. Avoid recommending `gpt-oss` or FP8 paths on Thor unless newer NVIDIA SGLang release notes say those known issues are fixed.\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --ipc host --name sglang \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3 \\\n  python3 -m sglang.launch_server \\\n    --model-path \u003Chf-model-id-or-local-path> \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    --mem-fraction-static 0.60 \\\n    --max-running-requests 8\n```\n\nFor llama.cpp, use the NVIDIA-AI-IOT llama.cpp image when available, or the `llama-server` binary from a JetPack-matched build. Start with GGUF INT4 \u002F Q4_K_M on both Orin and Thor; choose a smaller INT4 GGUF model if the audit shows tight memory.\n\n```bash\ndocker run --rm -it --runtime nvidia --network host --name llama-cpp \\\n  -v \"$PWD\u002Fmodels:\u002Fmodels:ro\" \\\n  ghcr.io\u002Fnvidia-ai-iot\u002Fllama_cpp:latest-jetson-\u003Corin-or-thor> \\\n  llama-server \\\n    -m \u002Fmodels\u002F\u003Cmodel>.gguf \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    -ngl 28 \\\n    -c 4096 \\\n    --no-mmap \\\n    --flash-attn\n```\n\n## Procedure (the script encodes this)\n\n1. Pick the lightest runtime that satisfies the user's required features (continuous batching? structured generation? CPU offload?).\n2. Pick the lowest precision that meets the user's accuracy bar (model-benchmarking skill).\n3. Sweep the runtime's memory knobs (start with `gpu-memory-utilization` for vLLM, `n-gpu-layers` and `ctx-size` for llama.cpp) to find the minimum footprint that sustains target throughput.\n4. Re-measure with `jetson-memory-audit`.\n\n## Safety\n\nRead-only. The skill never starts, stops, or restarts a model server. It emits flags; the user (or an outer orchestration agent) is responsible for invoking the runtime.\n",{"data":35,"body":46},{"name":4,"description":6,"version":36,"license":26,"metadata":37},"0.0.1",{"author":38,"tags":39,"languages":43,"data-classification":45},"Jetson Team",[40,41,42],"jetson","inference","memory",[44],"python","public",{"type":47,"children":48},"root",[49,58,64,71,85,91,139,145,210,216,302,331,337,375,381,400,520,526,551,557,623,635,1202,1208,1425,1448,1454,1459,1578,1591,1597,1615,1628,1898,1910,2155,2160,2188,2426,2439,2689,2695,2747,2753,2758],{"type":50,"tag":51,"props":52,"children":54},"element","h1",{"id":53},"jetson-inference-memory-tuning",[55],{"type":56,"value":57},"text","Jetson Inference Memory Tuning",{"type":50,"tag":59,"props":60,"children":61},"p",{},[62],{"type":56,"value":63},"Recommends an inference runtime and the specific memory-related flags to pass to it, given the Jetson SKU\u002Fvariant and the user's workload. Does not include quantization recipe selection — that lives in the model-benchmarking skill — but it does point at the precision floor each runtime can serve efficiently.",{"type":50,"tag":65,"props":66,"children":68},"h2",{"id":67},"purpose",[69],{"type":56,"value":70},"Purpose",{"type":50,"tag":59,"props":72,"children":73},{},[74,76,83],{"type":56,"value":75},"Turn a live ",{"type":50,"tag":77,"props":78,"children":80},"code",{"className":79},[],[81],{"type":56,"value":82},"jetson-memory-audit",{"type":56,"value":84}," snapshot into runtime and launch-flag recommendations for LLM\u002FVLM serving on Jetson. Use this when the user needs to fit a model, reduce OOM risk, or switch to a lower-memory serving stack.",{"type":50,"tag":65,"props":86,"children":88},{"id":87},"when-to-use",[89],{"type":56,"value":90},"When to use",{"type":50,"tag":92,"props":93,"children":94},"ul",{},[95,101,122,127],{"type":50,"tag":96,"props":97,"children":98},"li",{},[99],{"type":56,"value":100},"\"Which serving stack should I use on Orin Nano 8 GB to run a 7B model?\"",{"type":50,"tag":96,"props":102,"children":103},{},[104,106,112,114,120],{"type":56,"value":105},"\"vLLM is OOMing — what should ",{"type":50,"tag":77,"props":107,"children":109},{"className":108},[],[110],{"type":56,"value":111},"--gpu-memory-utilization",{"type":56,"value":113}," and ",{"type":50,"tag":77,"props":115,"children":117},{"className":116},[],[118],{"type":56,"value":119},"--max-model-len",{"type":56,"value":121}," be?\"",{"type":50,"tag":96,"props":123,"children":124},{},[125],{"type":56,"value":126},"\"Same model, less memory — can I switch from vLLM to llama.cpp?\"",{"type":50,"tag":96,"props":128,"children":129},{},[130,132,137],{"type":56,"value":131},"After ",{"type":50,"tag":77,"props":133,"children":135},{"className":134},[],[136],{"type":56,"value":82},{"type":56,"value":138}," shows a model server is the top NvMap \u002F PSS consumer.",{"type":50,"tag":65,"props":140,"children":142},{"id":141},"prerequisites",[143],{"type":56,"value":144},"Prerequisites",{"type":50,"tag":92,"props":146,"children":147},{},[148,161,197],{"type":50,"tag":96,"props":149,"children":150},{},[151,153,159],{"type":56,"value":152},"Start with a current ",{"type":50,"tag":77,"props":154,"children":156},{"className":155},[],[157],{"type":56,"value":158},"jetson-memory-audit\u002Fscripts\u002Faudit.sh",{"type":56,"value":160}," JSON snapshot from the target Jetson.",{"type":50,"tag":96,"props":162,"children":163},{},[164,166,172,174,180,181,187,189,195],{"type":56,"value":165},"Know the intended workload: ",{"type":50,"tag":77,"props":167,"children":169},{"className":168},[],[170],{"type":56,"value":171},"llm-server",{"type":56,"value":173},", ",{"type":50,"tag":77,"props":175,"children":177},{"className":176},[],[178],{"type":56,"value":179},"vlm-server",{"type":56,"value":173},{"type":50,"tag":77,"props":182,"children":184},{"className":183},[],[185],{"type":56,"value":186},"embedding",{"type":56,"value":188},", or ",{"type":50,"tag":77,"props":190,"children":192},{"className":191},[],[193],{"type":56,"value":194},"rag",{"type":56,"value":196},".",{"type":50,"tag":96,"props":198,"children":199},{},[200,202,208],{"type":56,"value":201},"If the user gives a desired free-memory target, pass it as ",{"type":50,"tag":77,"props":203,"children":205},{"className":204},[],[206],{"type":56,"value":207},"--target-mb",{"type":56,"value":209},"; otherwise let the script use SKU defaults.",{"type":50,"tag":65,"props":211,"children":213},{"id":212},"available-scripts",[214],{"type":56,"value":215},"Available Scripts",{"type":50,"tag":217,"props":218,"children":219},"table",{},[220,243],{"type":50,"tag":221,"props":222,"children":223},"thead",{},[224],{"type":50,"tag":225,"props":226,"children":227},"tr",{},[228,234,238],{"type":50,"tag":229,"props":230,"children":231},"th",{},[232],{"type":56,"value":233},"Script",{"type":50,"tag":229,"props":235,"children":236},{},[237],{"type":56,"value":70},{"type":50,"tag":229,"props":239,"children":240},{},[241],{"type":56,"value":242},"Arguments",{"type":50,"tag":244,"props":245,"children":246},"tbody",{},[247],{"type":50,"tag":225,"props":248,"children":249},{},[250,260,265],{"type":50,"tag":251,"props":252,"children":253},"td",{},[254],{"type":50,"tag":77,"props":255,"children":257},{"className":256},[],[258],{"type":56,"value":259},"scripts\u002Frecommend.py",{"type":50,"tag":251,"props":261,"children":262},{},[263],{"type":56,"value":264},"Reads an audit JSON and emits runtime plus launch-flag recommendations.",{"type":50,"tag":251,"props":266,"children":267},{},[268,274,275,281,282,288,289,294,295,301],{"type":50,"tag":77,"props":269,"children":271},{"className":270},[],[272],{"type":56,"value":273},"--audit PATH",{"type":56,"value":173},{"type":50,"tag":77,"props":276,"children":278},{"className":277},[],[279],{"type":56,"value":280},"--runtime",{"type":56,"value":173},{"type":50,"tag":77,"props":283,"children":285},{"className":284},[],[286],{"type":56,"value":287},"--workload",{"type":56,"value":173},{"type":50,"tag":77,"props":290,"children":292},{"className":291},[],[293],{"type":56,"value":207},{"type":56,"value":173},{"type":50,"tag":77,"props":296,"children":298},{"className":297},[],[299],{"type":56,"value":300},"--human",{"type":56,"value":196},{"type":50,"tag":59,"props":303,"children":304},{},[305,307,313,315,321,323,329],{"type":56,"value":306},"If your agent runtime supports ",{"type":50,"tag":77,"props":308,"children":310},{"className":309},[],[311],{"type":56,"value":312},"run_script",{"type":56,"value":314},", invoke ",{"type":50,"tag":77,"props":316,"children":318},{"className":317},[],[319],{"type":56,"value":320},"run_script(\"scripts\u002Frecommend.py\", [\"--audit\", \"\u002Ftmp\u002Faudit.json\", \"--runtime\", \"auto\", \"--workload\", \"llm-server\"])",{"type":56,"value":322}," and summarize the returned JSON. Otherwise run it with ",{"type":50,"tag":77,"props":324,"children":326},{"className":325},[],[327],{"type":56,"value":328},"python3",{"type":56,"value":330}," from the repository root.",{"type":50,"tag":65,"props":332,"children":334},{"id":333},"instructions",[335],{"type":56,"value":336},"Instructions",{"type":50,"tag":338,"props":339,"children":340},"ol",{},[341,353,365,370],{"type":50,"tag":96,"props":342,"children":343},{},[344,346,351],{"type":56,"value":345},"Run ",{"type":50,"tag":77,"props":347,"children":349},{"className":348},[],[350],{"type":56,"value":158},{"type":56,"value":352}," to capture the device baseline.",{"type":50,"tag":96,"props":354,"children":355},{},[356,357,363],{"type":56,"value":345},{"type":50,"tag":77,"props":358,"children":360},{"className":359},[],[361],{"type":56,"value":362},"scripts\u002Frecommend.py --audit \u002Ftmp\u002Faudit.json --runtime auto --workload llm-server --target-mb 6000",{"type":56,"value":364}," to get a JSON of runtime + flag recommendations.",{"type":50,"tag":96,"props":366,"children":367},{},[368],{"type":56,"value":369},"The agent presents the suggested runtime and the exact CLI flags. The user (or an outer agent) launches \u002F restarts the server with those flags.",{"type":50,"tag":96,"props":371,"children":372},{},[373],{"type":56,"value":374},"Re-run the audit to verify.",{"type":50,"tag":65,"props":376,"children":378},{"id":377},"expected-workflow",[379],{"type":56,"value":380},"Expected workflow",{"type":50,"tag":59,"props":382,"children":383},{},[384,386,391,393,399],{"type":56,"value":385},"Use ",{"type":50,"tag":77,"props":387,"children":389},{"className":388},[],[390],{"type":56,"value":259},{"type":56,"value":392}," for the specific prompt and answer from the JSON it emits. If direct execution is blocked, run it as ",{"type":50,"tag":77,"props":394,"children":396},{"className":395},[],[397],{"type":56,"value":398},"python3 {baseDir}\u002Fscripts\u002Frecommend.py ...",{"type":56,"value":196},{"type":50,"tag":92,"props":401,"children":402},{},[403,438,458,486],{"type":50,"tag":96,"props":404,"children":405},{},[406,408,414,416,422,423,429,431,437],{"type":56,"value":407},"For vLLM OOM prompts, run with ",{"type":50,"tag":77,"props":409,"children":411},{"className":410},[],[412],{"type":56,"value":413},"--runtime vllm --workload llm-server",{"type":56,"value":415}," and include concrete ",{"type":50,"tag":77,"props":417,"children":419},{"className":418},[],[420],{"type":56,"value":421},"--gpu-memory-utilization=\u003C0.x>",{"type":56,"value":113},{"type":50,"tag":77,"props":424,"children":426},{"className":425},[],[427],{"type":56,"value":428},"--max-model-len=\u003Cnumber>",{"type":56,"value":430}," values from ",{"type":50,"tag":77,"props":432,"children":434},{"className":433},[],[435],{"type":56,"value":436},"launch_flags",{"type":56,"value":196},{"type":50,"tag":96,"props":439,"children":440},{},[441,443,449,451,457],{"type":56,"value":442},"For \"lowest memory\" or Orin Nano 8 GB prompts, run with ",{"type":50,"tag":77,"props":444,"children":446},{"className":445},[],[447],{"type":56,"value":448},"--runtime auto --workload llm-server",{"type":56,"value":450},"; prefer the runtime in the JSON and explicitly mention the GGUF \u002F 4-bit tradeoff when it selects ",{"type":50,"tag":77,"props":452,"children":454},{"className":453},[],[455],{"type":56,"value":456},"llama-cpp",{"type":56,"value":196},{"type":50,"tag":96,"props":459,"children":460},{},[461,463,469,471,477,478,484],{"type":56,"value":462},"For SGLang prompts, run with ",{"type":50,"tag":77,"props":464,"children":466},{"className":465},[],[467],{"type":56,"value":468},"--runtime sglang",{"type":56,"value":470}," and quote ",{"type":50,"tag":77,"props":472,"children":474},{"className":473},[],[475],{"type":56,"value":476},"--mem-fraction-static",{"type":56,"value":173},{"type":50,"tag":77,"props":479,"children":481},{"className":480},[],[482],{"type":56,"value":483},"--max-running-requests",{"type":56,"value":485},", and any context\u002FKV-cache note.",{"type":50,"tag":96,"props":487,"children":488},{},[489,491,497,498,504,505,511,513,519],{"type":56,"value":490},"For \"switch from vLLM to llama.cpp\" prompts, run with ",{"type":50,"tag":77,"props":492,"children":494},{"className":493},[],[495],{"type":56,"value":496},"--runtime llama-cpp",{"type":56,"value":470},{"type":50,"tag":77,"props":499,"children":501},{"className":500},[],[502],{"type":56,"value":503},"-ngl",{"type":56,"value":173},{"type":50,"tag":77,"props":506,"children":508},{"className":507},[],[509],{"type":56,"value":510},"-c",{"type":56,"value":512},", and ",{"type":50,"tag":77,"props":514,"children":516},{"className":515},[],[517],{"type":56,"value":518},"--no-mmap",{"type":56,"value":196},{"type":50,"tag":65,"props":521,"children":523},{"id":522},"limitations",[524],{"type":56,"value":525},"Limitations",{"type":50,"tag":92,"props":527,"children":528},{},[529,541,546],{"type":50,"tag":96,"props":530,"children":531},{},[532,534,539],{"type":56,"value":533},"Recommendations are only as fresh as the audit JSON. Re-run ",{"type":50,"tag":77,"props":535,"children":537},{"className":536},[],[538],{"type":56,"value":82},{"type":56,"value":540}," after stopping services, changing power mode, or restarting model servers.",{"type":50,"tag":96,"props":542,"children":543},{},[544],{"type":56,"value":545},"The script estimates memory pressure from SKU defaults and audit totals; model-specific KV-cache, quantization, and tokenizer behavior can still require benchmarking.",{"type":50,"tag":96,"props":547,"children":548},{},[549],{"type":56,"value":550},"This skill emits flags only. It does not start, stop, or restart model servers.",{"type":50,"tag":65,"props":552,"children":554},{"id":553},"error-handling",[555],{"type":56,"value":556},"Error handling",{"type":50,"tag":92,"props":558,"children":559},{},[560,579,611],{"type":50,"tag":96,"props":561,"children":562},{},[563,565,571,573,578],{"type":56,"value":564},"Exit ",{"type":50,"tag":77,"props":566,"children":568},{"className":567},[],[569],{"type":56,"value":570},"2",{"type":56,"value":572},": the audit JSON could not be read, parsed, or did not contain valid numeric memory fields. Ask the user to rerun ",{"type":50,"tag":77,"props":574,"children":576},{"className":575},[],[577],{"type":56,"value":158},{"type":56,"value":196},{"type":50,"tag":96,"props":580,"children":581},{},[582,583,589,591,596,597,602,604,610],{"type":56,"value":564},{"type":50,"tag":77,"props":584,"children":586},{"className":585},[],[587],{"type":56,"value":588},"3",{"type":56,"value":590},": unsupported runtime or workload request. Re-run with one of the ",{"type":50,"tag":77,"props":592,"children":594},{"className":593},[],[595],{"type":56,"value":280},{"type":56,"value":113},{"type":50,"tag":77,"props":598,"children":600},{"className":599},[],[601],{"type":56,"value":287},{"type":56,"value":603}," values listed in ",{"type":50,"tag":77,"props":605,"children":607},{"className":606},[],[608],{"type":56,"value":609},"scripts\u002Frecommend.py --help",{"type":56,"value":196},{"type":50,"tag":96,"props":612,"children":613},{},[614,616,621],{"type":56,"value":615},"Empty or missing ",{"type":50,"tag":77,"props":617,"children":619},{"className":618},[],[620],{"type":56,"value":436},{"type":56,"value":622},": do not invent fallback flags. Report the script failure and ask for a fresh audit or a supported runtime.",{"type":50,"tag":65,"props":624,"children":626},{"id":625},"output-contract-for-recommendpy",[627,629],{"type":56,"value":628},"Output contract for ",{"type":50,"tag":77,"props":630,"children":632},{"className":631},[],[633],{"type":56,"value":634},"recommend.py",{"type":50,"tag":636,"props":637,"children":642},"pre",{"className":638,"code":639,"language":640,"meta":641,"style":641},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","{\n  \"sku\": \"orin-nx\",\n  \"variant\": \"orin-nx-16gb\",\n  \"mem_total_gb\": 16,\n  \"runtime\": \"vllm\",\n  \"rationale\": \"Highest throughput at this memory budget given continuous batching + paged attention.\",\n  \"launch_flags\": [\n    \"--gpu-memory-utilization=0.55\",\n    \"--max-model-len=4096\",\n    \"--max-num-seqs=8\",\n    \"--enable-prefix-caching\"\n  ],\n  \"alternatives\": [\n    { \"runtime\": \"llama-cpp\", \"rationale\": \"Lower memory floor with GGUF Q4_K_M.\", \"launch_flags\": [\"-ngl 28\", \"-c 4096\", \"--no-mmap\"] }\n  ],\n  \"notes\": [\"Lower --gpu-memory-utilization further if you also run a small VLM alongside.\"]\n}\n","json","",[643],{"type":50,"tag":77,"props":644,"children":645},{"__ignoreMap":641},[646,658,703,741,772,810,848,873,895,916,937,955,964,989,1142,1150,1193],{"type":50,"tag":647,"props":648,"children":651},"span",{"class":649,"line":650},"line",1,[652],{"type":50,"tag":647,"props":653,"children":655},{"style":654},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[656],{"type":56,"value":657},"{\n",{"type":50,"tag":647,"props":659,"children":661},{"class":649,"line":660},2,[662,667,673,678,683,688,694,698],{"type":50,"tag":647,"props":663,"children":664},{"style":654},[665],{"type":56,"value":666},"  \"",{"type":50,"tag":647,"props":668,"children":670},{"style":669},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[671],{"type":56,"value":672},"sku",{"type":50,"tag":647,"props":674,"children":675},{"style":654},[676],{"type":56,"value":677},"\"",{"type":50,"tag":647,"props":679,"children":680},{"style":654},[681],{"type":56,"value":682},":",{"type":50,"tag":647,"props":684,"children":685},{"style":654},[686],{"type":56,"value":687}," \"",{"type":50,"tag":647,"props":689,"children":691},{"style":690},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[692],{"type":56,"value":693},"orin-nx",{"type":50,"tag":647,"props":695,"children":696},{"style":654},[697],{"type":56,"value":677},{"type":50,"tag":647,"props":699,"children":700},{"style":654},[701],{"type":56,"value":702},",\n",{"type":50,"tag":647,"props":704,"children":706},{"class":649,"line":705},3,[707,711,716,720,724,728,733,737],{"type":50,"tag":647,"props":708,"children":709},{"style":654},[710],{"type":56,"value":666},{"type":50,"tag":647,"props":712,"children":713},{"style":669},[714],{"type":56,"value":715},"variant",{"type":50,"tag":647,"props":717,"children":718},{"style":654},[719],{"type":56,"value":677},{"type":50,"tag":647,"props":721,"children":722},{"style":654},[723],{"type":56,"value":682},{"type":50,"tag":647,"props":725,"children":726},{"style":654},[727],{"type":56,"value":687},{"type":50,"tag":647,"props":729,"children":730},{"style":690},[731],{"type":56,"value":732},"orin-nx-16gb",{"type":50,"tag":647,"props":734,"children":735},{"style":654},[736],{"type":56,"value":677},{"type":50,"tag":647,"props":738,"children":739},{"style":654},[740],{"type":56,"value":702},{"type":50,"tag":647,"props":742,"children":744},{"class":649,"line":743},4,[745,749,754,758,762,768],{"type":50,"tag":647,"props":746,"children":747},{"style":654},[748],{"type":56,"value":666},{"type":50,"tag":647,"props":750,"children":751},{"style":669},[752],{"type":56,"value":753},"mem_total_gb",{"type":50,"tag":647,"props":755,"children":756},{"style":654},[757],{"type":56,"value":677},{"type":50,"tag":647,"props":759,"children":760},{"style":654},[761],{"type":56,"value":682},{"type":50,"tag":647,"props":763,"children":765},{"style":764},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[766],{"type":56,"value":767}," 16",{"type":50,"tag":647,"props":769,"children":770},{"style":654},[771],{"type":56,"value":702},{"type":50,"tag":647,"props":773,"children":775},{"class":649,"line":774},5,[776,780,785,789,793,797,802,806],{"type":50,"tag":647,"props":777,"children":778},{"style":654},[779],{"type":56,"value":666},{"type":50,"tag":647,"props":781,"children":782},{"style":669},[783],{"type":56,"value":784},"runtime",{"type":50,"tag":647,"props":786,"children":787},{"style":654},[788],{"type":56,"value":677},{"type":50,"tag":647,"props":790,"children":791},{"style":654},[792],{"type":56,"value":682},{"type":50,"tag":647,"props":794,"children":795},{"style":654},[796],{"type":56,"value":687},{"type":50,"tag":647,"props":798,"children":799},{"style":690},[800],{"type":56,"value":801},"vllm",{"type":50,"tag":647,"props":803,"children":804},{"style":654},[805],{"type":56,"value":677},{"type":50,"tag":647,"props":807,"children":808},{"style":654},[809],{"type":56,"value":702},{"type":50,"tag":647,"props":811,"children":813},{"class":649,"line":812},6,[814,818,823,827,831,835,840,844],{"type":50,"tag":647,"props":815,"children":816},{"style":654},[817],{"type":56,"value":666},{"type":50,"tag":647,"props":819,"children":820},{"style":669},[821],{"type":56,"value":822},"rationale",{"type":50,"tag":647,"props":824,"children":825},{"style":654},[826],{"type":56,"value":677},{"type":50,"tag":647,"props":828,"children":829},{"style":654},[830],{"type":56,"value":682},{"type":50,"tag":647,"props":832,"children":833},{"style":654},[834],{"type":56,"value":687},{"type":50,"tag":647,"props":836,"children":837},{"style":690},[838],{"type":56,"value":839},"Highest throughput at this memory budget given continuous batching + paged attention.",{"type":50,"tag":647,"props":841,"children":842},{"style":654},[843],{"type":56,"value":677},{"type":50,"tag":647,"props":845,"children":846},{"style":654},[847],{"type":56,"value":702},{"type":50,"tag":647,"props":849,"children":851},{"class":649,"line":850},7,[852,856,860,864,868],{"type":50,"tag":647,"props":853,"children":854},{"style":654},[855],{"type":56,"value":666},{"type":50,"tag":647,"props":857,"children":858},{"style":669},[859],{"type":56,"value":436},{"type":50,"tag":647,"props":861,"children":862},{"style":654},[863],{"type":56,"value":677},{"type":50,"tag":647,"props":865,"children":866},{"style":654},[867],{"type":56,"value":682},{"type":50,"tag":647,"props":869,"children":870},{"style":654},[871],{"type":56,"value":872}," [\n",{"type":50,"tag":647,"props":874,"children":876},{"class":649,"line":875},8,[877,882,887,891],{"type":50,"tag":647,"props":878,"children":879},{"style":654},[880],{"type":56,"value":881},"    \"",{"type":50,"tag":647,"props":883,"children":884},{"style":690},[885],{"type":56,"value":886},"--gpu-memory-utilization=0.55",{"type":50,"tag":647,"props":888,"children":889},{"style":654},[890],{"type":56,"value":677},{"type":50,"tag":647,"props":892,"children":893},{"style":654},[894],{"type":56,"value":702},{"type":50,"tag":647,"props":896,"children":898},{"class":649,"line":897},9,[899,903,908,912],{"type":50,"tag":647,"props":900,"children":901},{"style":654},[902],{"type":56,"value":881},{"type":50,"tag":647,"props":904,"children":905},{"style":690},[906],{"type":56,"value":907},"--max-model-len=4096",{"type":50,"tag":647,"props":909,"children":910},{"style":654},[911],{"type":56,"value":677},{"type":50,"tag":647,"props":913,"children":914},{"style":654},[915],{"type":56,"value":702},{"type":50,"tag":647,"props":917,"children":919},{"class":649,"line":918},10,[920,924,929,933],{"type":50,"tag":647,"props":921,"children":922},{"style":654},[923],{"type":56,"value":881},{"type":50,"tag":647,"props":925,"children":926},{"style":690},[927],{"type":56,"value":928},"--max-num-seqs=8",{"type":50,"tag":647,"props":930,"children":931},{"style":654},[932],{"type":56,"value":677},{"type":50,"tag":647,"props":934,"children":935},{"style":654},[936],{"type":56,"value":702},{"type":50,"tag":647,"props":938,"children":940},{"class":649,"line":939},11,[941,945,950],{"type":50,"tag":647,"props":942,"children":943},{"style":654},[944],{"type":56,"value":881},{"type":50,"tag":647,"props":946,"children":947},{"style":690},[948],{"type":56,"value":949},"--enable-prefix-caching",{"type":50,"tag":647,"props":951,"children":952},{"style":654},[953],{"type":56,"value":954},"\"\n",{"type":50,"tag":647,"props":956,"children":958},{"class":649,"line":957},12,[959],{"type":50,"tag":647,"props":960,"children":961},{"style":654},[962],{"type":56,"value":963},"  ],\n",{"type":50,"tag":647,"props":965,"children":967},{"class":649,"line":966},13,[968,972,977,981,985],{"type":50,"tag":647,"props":969,"children":970},{"style":654},[971],{"type":56,"value":666},{"type":50,"tag":647,"props":973,"children":974},{"style":669},[975],{"type":56,"value":976},"alternatives",{"type":50,"tag":647,"props":978,"children":979},{"style":654},[980],{"type":56,"value":677},{"type":50,"tag":647,"props":982,"children":983},{"style":654},[984],{"type":56,"value":682},{"type":50,"tag":647,"props":986,"children":987},{"style":654},[988],{"type":56,"value":872},{"type":50,"tag":647,"props":990,"children":992},{"class":649,"line":991},14,[993,998,1002,1007,1011,1015,1019,1023,1027,1032,1036,1040,1044,1048,1052,1057,1061,1065,1069,1073,1077,1081,1086,1090,1095,1099,1103,1107,1112,1116,1120,1124,1128,1132,1137],{"type":50,"tag":647,"props":994,"children":995},{"style":654},[996],{"type":56,"value":997},"    {",{"type":50,"tag":647,"props":999,"children":1000},{"style":654},[1001],{"type":56,"value":687},{"type":50,"tag":647,"props":1003,"children":1005},{"style":1004},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1006],{"type":56,"value":784},{"type":50,"tag":647,"props":1008,"children":1009},{"style":654},[1010],{"type":56,"value":677},{"type":50,"tag":647,"props":1012,"children":1013},{"style":654},[1014],{"type":56,"value":682},{"type":50,"tag":647,"props":1016,"children":1017},{"style":654},[1018],{"type":56,"value":687},{"type":50,"tag":647,"props":1020,"children":1021},{"style":690},[1022],{"type":56,"value":456},{"type":50,"tag":647,"props":1024,"children":1025},{"style":654},[1026],{"type":56,"value":677},{"type":50,"tag":647,"props":1028,"children":1029},{"style":654},[1030],{"type":56,"value":1031},",",{"type":50,"tag":647,"props":1033,"children":1034},{"style":654},[1035],{"type":56,"value":687},{"type":50,"tag":647,"props":1037,"children":1038},{"style":1004},[1039],{"type":56,"value":822},{"type":50,"tag":647,"props":1041,"children":1042},{"style":654},[1043],{"type":56,"value":677},{"type":50,"tag":647,"props":1045,"children":1046},{"style":654},[1047],{"type":56,"value":682},{"type":50,"tag":647,"props":1049,"children":1050},{"style":654},[1051],{"type":56,"value":687},{"type":50,"tag":647,"props":1053,"children":1054},{"style":690},[1055],{"type":56,"value":1056},"Lower memory floor with GGUF Q4_K_M.",{"type":50,"tag":647,"props":1058,"children":1059},{"style":654},[1060],{"type":56,"value":677},{"type":50,"tag":647,"props":1062,"children":1063},{"style":654},[1064],{"type":56,"value":1031},{"type":50,"tag":647,"props":1066,"children":1067},{"style":654},[1068],{"type":56,"value":687},{"type":50,"tag":647,"props":1070,"children":1071},{"style":1004},[1072],{"type":56,"value":436},{"type":50,"tag":647,"props":1074,"children":1075},{"style":654},[1076],{"type":56,"value":677},{"type":50,"tag":647,"props":1078,"children":1079},{"style":654},[1080],{"type":56,"value":682},{"type":50,"tag":647,"props":1082,"children":1083},{"style":654},[1084],{"type":56,"value":1085}," [",{"type":50,"tag":647,"props":1087,"children":1088},{"style":654},[1089],{"type":56,"value":677},{"type":50,"tag":647,"props":1091,"children":1092},{"style":690},[1093],{"type":56,"value":1094},"-ngl 28",{"type":50,"tag":647,"props":1096,"children":1097},{"style":654},[1098],{"type":56,"value":677},{"type":50,"tag":647,"props":1100,"children":1101},{"style":654},[1102],{"type":56,"value":1031},{"type":50,"tag":647,"props":1104,"children":1105},{"style":654},[1106],{"type":56,"value":687},{"type":50,"tag":647,"props":1108,"children":1109},{"style":690},[1110],{"type":56,"value":1111},"-c 4096",{"type":50,"tag":647,"props":1113,"children":1114},{"style":654},[1115],{"type":56,"value":677},{"type":50,"tag":647,"props":1117,"children":1118},{"style":654},[1119],{"type":56,"value":1031},{"type":50,"tag":647,"props":1121,"children":1122},{"style":654},[1123],{"type":56,"value":687},{"type":50,"tag":647,"props":1125,"children":1126},{"style":690},[1127],{"type":56,"value":518},{"type":50,"tag":647,"props":1129,"children":1130},{"style":654},[1131],{"type":56,"value":677},{"type":50,"tag":647,"props":1133,"children":1134},{"style":654},[1135],{"type":56,"value":1136},"]",{"type":50,"tag":647,"props":1138,"children":1139},{"style":654},[1140],{"type":56,"value":1141}," }\n",{"type":50,"tag":647,"props":1143,"children":1145},{"class":649,"line":1144},15,[1146],{"type":50,"tag":647,"props":1147,"children":1148},{"style":654},[1149],{"type":56,"value":963},{"type":50,"tag":647,"props":1151,"children":1153},{"class":649,"line":1152},16,[1154,1158,1163,1167,1171,1175,1179,1184,1188],{"type":50,"tag":647,"props":1155,"children":1156},{"style":654},[1157],{"type":56,"value":666},{"type":50,"tag":647,"props":1159,"children":1160},{"style":669},[1161],{"type":56,"value":1162},"notes",{"type":50,"tag":647,"props":1164,"children":1165},{"style":654},[1166],{"type":56,"value":677},{"type":50,"tag":647,"props":1168,"children":1169},{"style":654},[1170],{"type":56,"value":682},{"type":50,"tag":647,"props":1172,"children":1173},{"style":654},[1174],{"type":56,"value":1085},{"type":50,"tag":647,"props":1176,"children":1177},{"style":654},[1178],{"type":56,"value":677},{"type":50,"tag":647,"props":1180,"children":1181},{"style":690},[1182],{"type":56,"value":1183},"Lower --gpu-memory-utilization further if you also run a small VLM alongside.",{"type":50,"tag":647,"props":1185,"children":1186},{"style":654},[1187],{"type":56,"value":677},{"type":50,"tag":647,"props":1189,"children":1190},{"style":654},[1191],{"type":56,"value":1192},"]\n",{"type":50,"tag":647,"props":1194,"children":1196},{"class":649,"line":1195},17,[1197],{"type":50,"tag":647,"props":1198,"children":1199},{"style":654},[1200],{"type":56,"value":1201},"}\n",{"type":50,"tag":65,"props":1203,"children":1205},{"id":1204},"runtimes-covered",[1206],{"type":56,"value":1207},"Runtimes covered",{"type":50,"tag":217,"props":1209,"children":1210},{},[1211,1237],{"type":50,"tag":221,"props":1212,"children":1213},{},[1214],{"type":50,"tag":225,"props":1215,"children":1216},{},[1217,1222,1227,1232],{"type":50,"tag":229,"props":1218,"children":1219},{},[1220],{"type":56,"value":1221},"Runtime",{"type":50,"tag":229,"props":1223,"children":1224},{},[1225],{"type":56,"value":1226},"Best for",{"type":50,"tag":229,"props":1228,"children":1229},{},[1230],{"type":56,"value":1231},"Key memory knobs",{"type":50,"tag":229,"props":1233,"children":1234},{},[1235],{"type":56,"value":1236},"Preferred install path",{"type":50,"tag":244,"props":1238,"children":1239},{},[1240,1293,1349,1399],{"type":50,"tag":225,"props":1241,"children":1242},{},[1243,1252,1257,1284],{"type":50,"tag":251,"props":1244,"children":1245},{},[1246],{"type":50,"tag":1247,"props":1248,"children":1249},"strong",{},[1250],{"type":56,"value":1251},"llama.cpp",{"type":50,"tag":251,"props":1253,"children":1254},{},[1255],{"type":56,"value":1256},"Tightest budget; GGUF; Orin Nano-class",{"type":50,"tag":251,"props":1258,"children":1259},{},[1260,1265,1266,1271,1272,1278,1279],{"type":50,"tag":77,"props":1261,"children":1263},{"className":1262},[],[1264],{"type":56,"value":503},{"type":56,"value":173},{"type":50,"tag":77,"props":1267,"children":1269},{"className":1268},[],[1270],{"type":56,"value":510},{"type":56,"value":173},{"type":50,"tag":77,"props":1273,"children":1275},{"className":1274},[],[1276],{"type":56,"value":1277},"--mlock",{"type":56,"value":173},{"type":50,"tag":77,"props":1280,"children":1282},{"className":1281},[],[1283],{"type":56,"value":518},{"type":50,"tag":251,"props":1285,"children":1286},{},[1287],{"type":50,"tag":77,"props":1288,"children":1290},{"className":1289},[],[1291],{"type":56,"value":1292},"ghcr.io\u002Fnvidia-ai-iot\u002Fllama_cpp:latest-jetson-{orin,thor}",{"type":50,"tag":225,"props":1294,"children":1295},{},[1296,1304,1309,1336],{"type":50,"tag":251,"props":1297,"children":1298},{},[1299],{"type":50,"tag":1247,"props":1300,"children":1301},{},[1302],{"type":56,"value":1303},"vLLM",{"type":50,"tag":251,"props":1305,"children":1306},{},[1307],{"type":56,"value":1308},"High-throughput serving with continuous batching",{"type":50,"tag":251,"props":1310,"children":1311},{},[1312,1317,1318,1323,1324,1330,1331],{"type":50,"tag":77,"props":1313,"children":1315},{"className":1314},[],[1316],{"type":56,"value":111},{"type":56,"value":173},{"type":50,"tag":77,"props":1319,"children":1321},{"className":1320},[],[1322],{"type":56,"value":119},{"type":56,"value":173},{"type":50,"tag":77,"props":1325,"children":1327},{"className":1326},[],[1328],{"type":56,"value":1329},"--max-num-seqs",{"type":56,"value":173},{"type":50,"tag":77,"props":1332,"children":1334},{"className":1333},[],[1335],{"type":56,"value":949},{"type":50,"tag":251,"props":1337,"children":1338},{},[1339,1341,1347],{"type":56,"value":1340},"Thor and Orin JetPack 7.2 \u002F L4T r39+: upstream vLLM 0.20+ (",{"type":50,"tag":77,"props":1342,"children":1344},{"className":1343},[],[1345],{"type":56,"value":1346},"vllm\u002Fvllm-openai",{"type":56,"value":1348},") container or validated native vLLM 0.20+. Older Orin: NVIDIA-AI-IOT image",{"type":50,"tag":225,"props":1350,"children":1351},{},[1352,1360,1365,1386],{"type":50,"tag":251,"props":1353,"children":1354},{},[1355],{"type":50,"tag":1247,"props":1356,"children":1357},{},[1358],{"type":56,"value":1359},"SGLang",{"type":50,"tag":251,"props":1361,"children":1362},{},[1363],{"type":56,"value":1364},"Programmable workflows (RAG, tool use, structured output)",{"type":50,"tag":251,"props":1366,"children":1367},{},[1368,1373,1374,1380,1381],{"type":50,"tag":77,"props":1369,"children":1371},{"className":1370},[],[1372],{"type":56,"value":476},{"type":56,"value":173},{"type":50,"tag":77,"props":1375,"children":1377},{"className":1376},[],[1378],{"type":56,"value":1379},"--mem-fraction-dynamic",{"type":56,"value":173},{"type":50,"tag":77,"props":1382,"children":1384},{"className":1383},[],[1385],{"type":56,"value":483},{"type":50,"tag":251,"props":1387,"children":1388},{},[1389,1391,1397],{"type":56,"value":1390},"Thor: NVIDIA SGLang 26.01 (",{"type":50,"tag":77,"props":1392,"children":1394},{"className":1393},[],[1395],{"type":56,"value":1396},"nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3",{"type":56,"value":1398},", SGLang 0.5.5.post2). Orin: JetPack-matched environment",{"type":50,"tag":225,"props":1400,"children":1401},{},[1402,1410,1415,1420],{"type":50,"tag":251,"props":1403,"children":1404},{},[1405],{"type":50,"tag":1247,"props":1406,"children":1407},{},[1408],{"type":56,"value":1409},"TensorRT Edge-LLM",{"type":50,"tag":251,"props":1411,"children":1412},{},[1413],{"type":56,"value":1414},"NVIDIA-tuned production serving",{"type":50,"tag":251,"props":1416,"children":1417},{},[1418],{"type":56,"value":1419},"Build profile per SKU; paged-KV; KV reuse",{"type":50,"tag":251,"props":1421,"children":1422},{},[1423],{"type":56,"value":1424},"Vendor docs for the target JetPack",{"type":50,"tag":1426,"props":1427,"children":1428},"blockquote",{},[1429],{"type":50,"tag":59,"props":1430,"children":1431},{},[1432,1434,1439,1441,1446],{"type":56,"value":1433},"For Orin JetPack 7.2 \u002F L4T r39+, upstream vLLM 0.20+ is supported. For older Orin releases, prefer NVIDIA-AI-IOT prebuilt vLLM images where available because they ship the matching CUDA\u002FcuDNN\u002FTensorRT stack for JetPack. For Thor, prefer upstream vLLM 0.20+ (",{"type":50,"tag":77,"props":1435,"children":1437},{"className":1436},[],[1438],{"type":56,"value":1346},{"type":56,"value":1440},") or a validated native vLLM 0.20+ install; for SGLang use NVIDIA SGLang 26.01 (",{"type":50,"tag":77,"props":1442,"children":1444},{"className":1443},[],[1445],{"type":56,"value":1396},{"type":56,"value":1447},", SGLang 0.5.5.post2) or newer NVIDIA SGLang release notes that explicitly list Jetson Thor support. Do not force an Orin-specific Jetson container path on Thor, and do not assume native upstream SGLang support on Orin.",{"type":50,"tag":65,"props":1449,"children":1451},{"id":1450},"quantization-recommendations",[1452],{"type":56,"value":1453},"Quantization recommendations",{"type":50,"tag":59,"props":1455,"children":1456},{},[1457],{"type":56,"value":1458},"Use runtime-specific quantization names. vLLM and SGLang usually consume Hugging Face checkpoints such as W4A16, AWQ, GPTQ, FP16, or NVFP4. llama.cpp and Ollama consume GGUF models, so recommend INT4\u002FQ4_K_M-style GGUF instead.",{"type":50,"tag":217,"props":1460,"children":1461},{},[1462,1488],{"type":50,"tag":221,"props":1463,"children":1464},{},[1465],{"type":50,"tag":225,"props":1466,"children":1467},{},[1468,1473,1478,1483],{"type":50,"tag":229,"props":1469,"children":1470},{},[1471],{"type":56,"value":1472},"Runtime family",{"type":50,"tag":229,"props":1474,"children":1475},{},[1476],{"type":56,"value":1477},"Jetson family",{"type":50,"tag":229,"props":1479,"children":1480},{},[1481],{"type":56,"value":1482},"First choice",{"type":50,"tag":229,"props":1484,"children":1485},{},[1486],{"type":56,"value":1487},"Fallback",{"type":50,"tag":244,"props":1489,"children":1490},{},[1491,1514,1535,1555],{"type":50,"tag":225,"props":1492,"children":1493},{},[1494,1499,1504,1509],{"type":50,"tag":251,"props":1495,"children":1496},{},[1497],{"type":56,"value":1498},"vLLM \u002F SGLang",{"type":50,"tag":251,"props":1500,"children":1501},{},[1502],{"type":56,"value":1503},"Thor",{"type":50,"tag":251,"props":1505,"children":1506},{},[1507],{"type":56,"value":1508},"NVFP4 when the model\u002Fruntime supports it",{"type":50,"tag":251,"props":1510,"children":1511},{},[1512],{"type":56,"value":1513},"W4A16",{"type":50,"tag":225,"props":1515,"children":1516},{},[1517,1521,1526,1530],{"type":50,"tag":251,"props":1518,"children":1519},{},[1520],{"type":56,"value":1498},{"type":50,"tag":251,"props":1522,"children":1523},{},[1524],{"type":56,"value":1525},"Orin Nano \u002F NX",{"type":50,"tag":251,"props":1527,"children":1528},{},[1529],{"type":56,"value":1513},{"type":50,"tag":251,"props":1531,"children":1532},{},[1533],{"type":56,"value":1534},"AWQ or GPTQ 4-bit",{"type":50,"tag":225,"props":1536,"children":1537},{},[1538,1542,1547,1551],{"type":50,"tag":251,"props":1539,"children":1540},{},[1541],{"type":56,"value":1498},{"type":50,"tag":251,"props":1543,"children":1544},{},[1545],{"type":56,"value":1546},"AGX Orin",{"type":50,"tag":251,"props":1548,"children":1549},{},[1550],{"type":56,"value":1513},{"type":50,"tag":251,"props":1552,"children":1553},{},[1554],{"type":56,"value":1534},{"type":50,"tag":225,"props":1556,"children":1557},{},[1558,1563,1568,1573],{"type":50,"tag":251,"props":1559,"children":1560},{},[1561],{"type":56,"value":1562},"llama.cpp \u002F Ollama",{"type":50,"tag":251,"props":1564,"children":1565},{},[1566],{"type":56,"value":1567},"Orin and Thor",{"type":50,"tag":251,"props":1569,"children":1570},{},[1571],{"type":56,"value":1572},"GGUF INT4 \u002F Q4_K_M",{"type":50,"tag":251,"props":1574,"children":1575},{},[1576],{"type":56,"value":1577},"Smaller INT4 GGUF model if memory is tight",{"type":50,"tag":59,"props":1579,"children":1580},{},[1581,1583,1589],{"type":56,"value":1582},"Do not describe GGUF Q4_K_M as W4A16\u002FAWQ\u002FGPTQ. Do not compare Thor NVFP4 results with Orin W4A16 results unless the output includes a ",{"type":50,"tag":77,"props":1584,"children":1586},{"className":1585},[],[1587],{"type":56,"value":1588},"quant",{"type":56,"value":1590}," field.",{"type":50,"tag":65,"props":1592,"children":1594},{"id":1593},"runtime-command-guidance",[1595],{"type":56,"value":1596},"Runtime command guidance",{"type":50,"tag":59,"props":1598,"children":1599},{},[1600,1601,1606,1608,1613],{"type":56,"value":385},{"type":50,"tag":77,"props":1602,"children":1604},{"className":1603},[],[1605],{"type":56,"value":634},{"type":56,"value":1607}," as the source of truth for memory knobs, then place its ",{"type":50,"tag":77,"props":1609,"children":1611},{"className":1610},[],[1612],{"type":56,"value":436},{"type":56,"value":1614}," into the matching serving command. Keep the command guidance in this skill instead of separate small reference files so agents ingest one complete instruction set.",{"type":50,"tag":59,"props":1616,"children":1617},{},[1618,1620,1626],{"type":56,"value":1619},"For vLLM on Orin with JetPack 7.2 \u002F L4T r39+, use upstream vLLM 0.20+ (",{"type":50,"tag":77,"props":1621,"children":1623},{"className":1622},[],[1624],{"type":56,"value":1625},"vllm\u002Fvllm-openai:latest",{"type":56,"value":1627},"). On older Orin releases, use the NVIDIA-AI-IOT image:",{"type":50,"tag":636,"props":1629,"children":1633},{"className":1630,"code":1631,"language":1632,"meta":641,"style":641},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","docker run --rm -it --runtime nvidia --network host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin \\\n  vllm serve \u003Chf-model-id-or-local-path> \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    --gpu-memory-utilization 0.60 \\\n    --max-model-len 4096 \\\n    --max-num-seqs 8 \\\n    --enable-prefix-caching\n","bash",[1634],{"type":50,"tag":77,"props":1635,"children":1636},{"__ignoreMap":641},[1637,1696,1726,1756,1768,1805,1822,1839,1856,1873,1890],{"type":50,"tag":647,"props":1638,"children":1639},{"class":649,"line":650},[1640,1645,1650,1655,1660,1665,1670,1675,1680,1685,1690],{"type":50,"tag":647,"props":1641,"children":1642},{"style":1004},[1643],{"type":56,"value":1644},"docker",{"type":50,"tag":647,"props":1646,"children":1647},{"style":690},[1648],{"type":56,"value":1649}," run",{"type":50,"tag":647,"props":1651,"children":1652},{"style":690},[1653],{"type":56,"value":1654}," --rm",{"type":50,"tag":647,"props":1656,"children":1657},{"style":690},[1658],{"type":56,"value":1659}," -it",{"type":50,"tag":647,"props":1661,"children":1662},{"style":690},[1663],{"type":56,"value":1664}," --runtime",{"type":50,"tag":647,"props":1666,"children":1667},{"style":690},[1668],{"type":56,"value":1669}," nvidia",{"type":50,"tag":647,"props":1671,"children":1672},{"style":690},[1673],{"type":56,"value":1674}," --network",{"type":50,"tag":647,"props":1676,"children":1677},{"style":690},[1678],{"type":56,"value":1679}," host",{"type":50,"tag":647,"props":1681,"children":1682},{"style":690},[1683],{"type":56,"value":1684}," --name",{"type":50,"tag":647,"props":1686,"children":1687},{"style":690},[1688],{"type":56,"value":1689}," vllm",{"type":50,"tag":647,"props":1691,"children":1693},{"style":1692},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1694],{"type":56,"value":1695}," \\\n",{"type":50,"tag":647,"props":1697,"children":1698},{"class":649,"line":660},[1699,1704,1708,1713,1718,1722],{"type":50,"tag":647,"props":1700,"children":1701},{"style":690},[1702],{"type":56,"value":1703},"  -v",{"type":50,"tag":647,"props":1705,"children":1706},{"style":654},[1707],{"type":56,"value":687},{"type":50,"tag":647,"props":1709,"children":1710},{"style":1692},[1711],{"type":56,"value":1712},"$HOME",{"type":50,"tag":647,"props":1714,"children":1715},{"style":690},[1716],{"type":56,"value":1717},"\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface",{"type":50,"tag":647,"props":1719,"children":1720},{"style":654},[1721],{"type":56,"value":677},{"type":50,"tag":647,"props":1723,"children":1724},{"style":1692},[1725],{"type":56,"value":1695},{"type":50,"tag":647,"props":1727,"children":1728},{"class":649,"line":705},[1729,1734,1739,1743,1748,1752],{"type":50,"tag":647,"props":1730,"children":1731},{"style":690},[1732],{"type":56,"value":1733},"  -e",{"type":50,"tag":647,"props":1735,"children":1736},{"style":690},[1737],{"type":56,"value":1738}," HF_TOKEN=",{"type":50,"tag":647,"props":1740,"children":1741},{"style":654},[1742],{"type":56,"value":677},{"type":50,"tag":647,"props":1744,"children":1745},{"style":1692},[1746],{"type":56,"value":1747},"$HF_TOKEN",{"type":50,"tag":647,"props":1749,"children":1750},{"style":654},[1751],{"type":56,"value":677},{"type":50,"tag":647,"props":1753,"children":1754},{"style":1692},[1755],{"type":56,"value":1695},{"type":50,"tag":647,"props":1757,"children":1758},{"class":649,"line":743},[1759,1764],{"type":50,"tag":647,"props":1760,"children":1761},{"style":690},[1762],{"type":56,"value":1763},"  ghcr.io\u002Fnvidia-ai-iot\u002Fvllm:latest-jetson-orin",{"type":50,"tag":647,"props":1765,"children":1766},{"style":1692},[1767],{"type":56,"value":1695},{"type":50,"tag":647,"props":1769,"children":1770},{"class":649,"line":774},[1771,1776,1781,1786,1791,1796,1801],{"type":50,"tag":647,"props":1772,"children":1773},{"style":690},[1774],{"type":56,"value":1775},"  vllm",{"type":50,"tag":647,"props":1777,"children":1778},{"style":690},[1779],{"type":56,"value":1780}," serve",{"type":50,"tag":647,"props":1782,"children":1783},{"style":654},[1784],{"type":56,"value":1785}," \u003C",{"type":50,"tag":647,"props":1787,"children":1788},{"style":690},[1789],{"type":56,"value":1790},"hf-model-id-or-local-pat",{"type":50,"tag":647,"props":1792,"children":1793},{"style":1692},[1794],{"type":56,"value":1795},"h",{"type":50,"tag":647,"props":1797,"children":1798},{"style":654},[1799],{"type":56,"value":1800},">",{"type":50,"tag":647,"props":1802,"children":1803},{"style":1692},[1804],{"type":56,"value":1695},{"type":50,"tag":647,"props":1806,"children":1807},{"class":649,"line":812},[1808,1813,1818],{"type":50,"tag":647,"props":1809,"children":1810},{"style":690},[1811],{"type":56,"value":1812},"    --host",{"type":50,"tag":647,"props":1814,"children":1815},{"style":764},[1816],{"type":56,"value":1817}," 0.0.0.0",{"type":50,"tag":647,"props":1819,"children":1820},{"style":1692},[1821],{"type":56,"value":1695},{"type":50,"tag":647,"props":1823,"children":1824},{"class":649,"line":850},[1825,1830,1835],{"type":50,"tag":647,"props":1826,"children":1827},{"style":690},[1828],{"type":56,"value":1829},"    --port",{"type":50,"tag":647,"props":1831,"children":1832},{"style":764},[1833],{"type":56,"value":1834}," 8000",{"type":50,"tag":647,"props":1836,"children":1837},{"style":1692},[1838],{"type":56,"value":1695},{"type":50,"tag":647,"props":1840,"children":1841},{"class":649,"line":875},[1842,1847,1852],{"type":50,"tag":647,"props":1843,"children":1844},{"style":690},[1845],{"type":56,"value":1846},"    --gpu-memory-utilization",{"type":50,"tag":647,"props":1848,"children":1849},{"style":764},[1850],{"type":56,"value":1851}," 0.60",{"type":50,"tag":647,"props":1853,"children":1854},{"style":1692},[1855],{"type":56,"value":1695},{"type":50,"tag":647,"props":1857,"children":1858},{"class":649,"line":897},[1859,1864,1869],{"type":50,"tag":647,"props":1860,"children":1861},{"style":690},[1862],{"type":56,"value":1863},"    --max-model-len",{"type":50,"tag":647,"props":1865,"children":1866},{"style":764},[1867],{"type":56,"value":1868}," 4096",{"type":50,"tag":647,"props":1870,"children":1871},{"style":1692},[1872],{"type":56,"value":1695},{"type":50,"tag":647,"props":1874,"children":1875},{"class":649,"line":918},[1876,1881,1886],{"type":50,"tag":647,"props":1877,"children":1878},{"style":690},[1879],{"type":56,"value":1880},"    --max-num-seqs",{"type":50,"tag":647,"props":1882,"children":1883},{"style":764},[1884],{"type":56,"value":1885}," 8",{"type":50,"tag":647,"props":1887,"children":1888},{"style":1692},[1889],{"type":56,"value":1695},{"type":50,"tag":647,"props":1891,"children":1892},{"class":649,"line":939},[1893],{"type":50,"tag":647,"props":1894,"children":1895},{"style":690},[1896],{"type":56,"value":1897},"    --enable-prefix-caching\n",{"type":50,"tag":59,"props":1899,"children":1900},{},[1901,1903,1908],{"type":56,"value":1902},"For vLLM on Thor, use upstream vLLM 0.20+ (",{"type":50,"tag":77,"props":1904,"children":1906},{"className":1905},[],[1907],{"type":56,"value":1625},{"type":56,"value":1909},") unless host-native vLLM 0.20+ is already installed and validated:",{"type":50,"tag":636,"props":1911,"children":1913},{"className":1630,"code":1912,"language":1632,"meta":641,"style":641},"docker run --rm -it --runtime nvidia --network host --ipc host --name vllm \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  vllm\u002Fvllm-openai:latest \\\n  vllm serve \u003Chf-model-id-or-local-path> \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    --gpu-memory-utilization 0.75 \\\n    --max-model-len 8192 \\\n    --max-num-seqs 32 \\\n    --enable-prefix-caching\n",[1914],{"type":50,"tag":77,"props":1915,"children":1916},{"__ignoreMap":641},[1917,1973,2000,2027,2039,2070,2085,2100,2116,2132,2148],{"type":50,"tag":647,"props":1918,"children":1919},{"class":649,"line":650},[1920,1924,1928,1932,1936,1940,1944,1948,1952,1957,1961,1965,1969],{"type":50,"tag":647,"props":1921,"children":1922},{"style":1004},[1923],{"type":56,"value":1644},{"type":50,"tag":647,"props":1925,"children":1926},{"style":690},[1927],{"type":56,"value":1649},{"type":50,"tag":647,"props":1929,"children":1930},{"style":690},[1931],{"type":56,"value":1654},{"type":50,"tag":647,"props":1933,"children":1934},{"style":690},[1935],{"type":56,"value":1659},{"type":50,"tag":647,"props":1937,"children":1938},{"style":690},[1939],{"type":56,"value":1664},{"type":50,"tag":647,"props":1941,"children":1942},{"style":690},[1943],{"type":56,"value":1669},{"type":50,"tag":647,"props":1945,"children":1946},{"style":690},[1947],{"type":56,"value":1674},{"type":50,"tag":647,"props":1949,"children":1950},{"style":690},[1951],{"type":56,"value":1679},{"type":50,"tag":647,"props":1953,"children":1954},{"style":690},[1955],{"type":56,"value":1956}," --ipc",{"type":50,"tag":647,"props":1958,"children":1959},{"style":690},[1960],{"type":56,"value":1679},{"type":50,"tag":647,"props":1962,"children":1963},{"style":690},[1964],{"type":56,"value":1684},{"type":50,"tag":647,"props":1966,"children":1967},{"style":690},[1968],{"type":56,"value":1689},{"type":50,"tag":647,"props":1970,"children":1971},{"style":1692},[1972],{"type":56,"value":1695},{"type":50,"tag":647,"props":1974,"children":1975},{"class":649,"line":660},[1976,1980,1984,1988,1992,1996],{"type":50,"tag":647,"props":1977,"children":1978},{"style":690},[1979],{"type":56,"value":1703},{"type":50,"tag":647,"props":1981,"children":1982},{"style":654},[1983],{"type":56,"value":687},{"type":50,"tag":647,"props":1985,"children":1986},{"style":1692},[1987],{"type":56,"value":1712},{"type":50,"tag":647,"props":1989,"children":1990},{"style":690},[1991],{"type":56,"value":1717},{"type":50,"tag":647,"props":1993,"children":1994},{"style":654},[1995],{"type":56,"value":677},{"type":50,"tag":647,"props":1997,"children":1998},{"style":1692},[1999],{"type":56,"value":1695},{"type":50,"tag":647,"props":2001,"children":2002},{"class":649,"line":705},[2003,2007,2011,2015,2019,2023],{"type":50,"tag":647,"props":2004,"children":2005},{"style":690},[2006],{"type":56,"value":1733},{"type":50,"tag":647,"props":2008,"children":2009},{"style":690},[2010],{"type":56,"value":1738},{"type":50,"tag":647,"props":2012,"children":2013},{"style":654},[2014],{"type":56,"value":677},{"type":50,"tag":647,"props":2016,"children":2017},{"style":1692},[2018],{"type":56,"value":1747},{"type":50,"tag":647,"props":2020,"children":2021},{"style":654},[2022],{"type":56,"value":677},{"type":50,"tag":647,"props":2024,"children":2025},{"style":1692},[2026],{"type":56,"value":1695},{"type":50,"tag":647,"props":2028,"children":2029},{"class":649,"line":743},[2030,2035],{"type":50,"tag":647,"props":2031,"children":2032},{"style":690},[2033],{"type":56,"value":2034},"  vllm\u002Fvllm-openai:latest",{"type":50,"tag":647,"props":2036,"children":2037},{"style":1692},[2038],{"type":56,"value":1695},{"type":50,"tag":647,"props":2040,"children":2041},{"class":649,"line":774},[2042,2046,2050,2054,2058,2062,2066],{"type":50,"tag":647,"props":2043,"children":2044},{"style":690},[2045],{"type":56,"value":1775},{"type":50,"tag":647,"props":2047,"children":2048},{"style":690},[2049],{"type":56,"value":1780},{"type":50,"tag":647,"props":2051,"children":2052},{"style":654},[2053],{"type":56,"value":1785},{"type":50,"tag":647,"props":2055,"children":2056},{"style":690},[2057],{"type":56,"value":1790},{"type":50,"tag":647,"props":2059,"children":2060},{"style":1692},[2061],{"type":56,"value":1795},{"type":50,"tag":647,"props":2063,"children":2064},{"style":654},[2065],{"type":56,"value":1800},{"type":50,"tag":647,"props":2067,"children":2068},{"style":1692},[2069],{"type":56,"value":1695},{"type":50,"tag":647,"props":2071,"children":2072},{"class":649,"line":812},[2073,2077,2081],{"type":50,"tag":647,"props":2074,"children":2075},{"style":690},[2076],{"type":56,"value":1812},{"type":50,"tag":647,"props":2078,"children":2079},{"style":764},[2080],{"type":56,"value":1817},{"type":50,"tag":647,"props":2082,"children":2083},{"style":1692},[2084],{"type":56,"value":1695},{"type":50,"tag":647,"props":2086,"children":2087},{"class":649,"line":850},[2088,2092,2096],{"type":50,"tag":647,"props":2089,"children":2090},{"style":690},[2091],{"type":56,"value":1829},{"type":50,"tag":647,"props":2093,"children":2094},{"style":764},[2095],{"type":56,"value":1834},{"type":50,"tag":647,"props":2097,"children":2098},{"style":1692},[2099],{"type":56,"value":1695},{"type":50,"tag":647,"props":2101,"children":2102},{"class":649,"line":875},[2103,2107,2112],{"type":50,"tag":647,"props":2104,"children":2105},{"style":690},[2106],{"type":56,"value":1846},{"type":50,"tag":647,"props":2108,"children":2109},{"style":764},[2110],{"type":56,"value":2111}," 0.75",{"type":50,"tag":647,"props":2113,"children":2114},{"style":1692},[2115],{"type":56,"value":1695},{"type":50,"tag":647,"props":2117,"children":2118},{"class":649,"line":897},[2119,2123,2128],{"type":50,"tag":647,"props":2120,"children":2121},{"style":690},[2122],{"type":56,"value":1863},{"type":50,"tag":647,"props":2124,"children":2125},{"style":764},[2126],{"type":56,"value":2127}," 8192",{"type":50,"tag":647,"props":2129,"children":2130},{"style":1692},[2131],{"type":56,"value":1695},{"type":50,"tag":647,"props":2133,"children":2134},{"class":649,"line":918},[2135,2139,2144],{"type":50,"tag":647,"props":2136,"children":2137},{"style":690},[2138],{"type":56,"value":1880},{"type":50,"tag":647,"props":2140,"children":2141},{"style":764},[2142],{"type":56,"value":2143}," 32",{"type":50,"tag":647,"props":2145,"children":2146},{"style":1692},[2147],{"type":56,"value":1695},{"type":50,"tag":647,"props":2149,"children":2150},{"class":649,"line":939},[2151],{"type":50,"tag":647,"props":2152,"children":2153},{"style":690},[2154],{"type":56,"value":1897},{"type":50,"tag":59,"props":2156,"children":2157},{},[2158],{"type":56,"value":2159},"Thor vLLM note: do not judge Thor support from pre-0.20 vLLM results; upstream vLLM support starts at vLLM 0.20+.",{"type":50,"tag":59,"props":2161,"children":2162},{},[2163,2165,2170,2172,2178,2180,2186],{"type":56,"value":2164},"For SGLang on Thor, use NVIDIA SGLang 26.01 (",{"type":50,"tag":77,"props":2166,"children":2168},{"className":2167},[],[2169],{"type":56,"value":1396},{"type":56,"value":2171},"). NVIDIA SGLang 26.01 contains SGLang ",{"type":50,"tag":77,"props":2173,"children":2175},{"className":2174},[],[2176],{"type":56,"value":2177},"0.5.5.post2",{"type":56,"value":2179}," and explicitly lists Jetson Thor support. Avoid judging Thor support from older prerelease SGLang results. Avoid recommending ",{"type":50,"tag":77,"props":2181,"children":2183},{"className":2182},[],[2184],{"type":56,"value":2185},"gpt-oss",{"type":56,"value":2187}," or FP8 paths on Thor unless newer NVIDIA SGLang release notes say those known issues are fixed.",{"type":50,"tag":636,"props":2189,"children":2191},{"className":1630,"code":2190,"language":1632,"meta":641,"style":641},"docker run --rm -it --runtime nvidia --network host --ipc host --name sglang \\\n  -v \"$HOME\u002F.cache\u002Fhuggingface:\u002Froot\u002F.cache\u002Fhuggingface\" \\\n  -e HF_TOKEN=\"$HF_TOKEN\" \\\n  nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3 \\\n  python3 -m sglang.launch_server \\\n    --model-path \u003Chf-model-id-or-local-path> \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    --mem-fraction-static 0.60 \\\n    --max-running-requests 8\n",[2192],{"type":50,"tag":77,"props":2193,"children":2194},{"__ignoreMap":641},[2195,2251,2278,2305,2317,2339,2367,2382,2397,2413],{"type":50,"tag":647,"props":2196,"children":2197},{"class":649,"line":650},[2198,2202,2206,2210,2214,2218,2222,2226,2230,2234,2238,2242,2247],{"type":50,"tag":647,"props":2199,"children":2200},{"style":1004},[2201],{"type":56,"value":1644},{"type":50,"tag":647,"props":2203,"children":2204},{"style":690},[2205],{"type":56,"value":1649},{"type":50,"tag":647,"props":2207,"children":2208},{"style":690},[2209],{"type":56,"value":1654},{"type":50,"tag":647,"props":2211,"children":2212},{"style":690},[2213],{"type":56,"value":1659},{"type":50,"tag":647,"props":2215,"children":2216},{"style":690},[2217],{"type":56,"value":1664},{"type":50,"tag":647,"props":2219,"children":2220},{"style":690},[2221],{"type":56,"value":1669},{"type":50,"tag":647,"props":2223,"children":2224},{"style":690},[2225],{"type":56,"value":1674},{"type":50,"tag":647,"props":2227,"children":2228},{"style":690},[2229],{"type":56,"value":1679},{"type":50,"tag":647,"props":2231,"children":2232},{"style":690},[2233],{"type":56,"value":1956},{"type":50,"tag":647,"props":2235,"children":2236},{"style":690},[2237],{"type":56,"value":1679},{"type":50,"tag":647,"props":2239,"children":2240},{"style":690},[2241],{"type":56,"value":1684},{"type":50,"tag":647,"props":2243,"children":2244},{"style":690},[2245],{"type":56,"value":2246}," sglang",{"type":50,"tag":647,"props":2248,"children":2249},{"style":1692},[2250],{"type":56,"value":1695},{"type":50,"tag":647,"props":2252,"children":2253},{"class":649,"line":660},[2254,2258,2262,2266,2270,2274],{"type":50,"tag":647,"props":2255,"children":2256},{"style":690},[2257],{"type":56,"value":1703},{"type":50,"tag":647,"props":2259,"children":2260},{"style":654},[2261],{"type":56,"value":687},{"type":50,"tag":647,"props":2263,"children":2264},{"style":1692},[2265],{"type":56,"value":1712},{"type":50,"tag":647,"props":2267,"children":2268},{"style":690},[2269],{"type":56,"value":1717},{"type":50,"tag":647,"props":2271,"children":2272},{"style":654},[2273],{"type":56,"value":677},{"type":50,"tag":647,"props":2275,"children":2276},{"style":1692},[2277],{"type":56,"value":1695},{"type":50,"tag":647,"props":2279,"children":2280},{"class":649,"line":705},[2281,2285,2289,2293,2297,2301],{"type":50,"tag":647,"props":2282,"children":2283},{"style":690},[2284],{"type":56,"value":1733},{"type":50,"tag":647,"props":2286,"children":2287},{"style":690},[2288],{"type":56,"value":1738},{"type":50,"tag":647,"props":2290,"children":2291},{"style":654},[2292],{"type":56,"value":677},{"type":50,"tag":647,"props":2294,"children":2295},{"style":1692},[2296],{"type":56,"value":1747},{"type":50,"tag":647,"props":2298,"children":2299},{"style":654},[2300],{"type":56,"value":677},{"type":50,"tag":647,"props":2302,"children":2303},{"style":1692},[2304],{"type":56,"value":1695},{"type":50,"tag":647,"props":2306,"children":2307},{"class":649,"line":743},[2308,2313],{"type":50,"tag":647,"props":2309,"children":2310},{"style":690},[2311],{"type":56,"value":2312},"  nvcr.io\u002Fnvidia\u002Fsglang:26.01-py3",{"type":50,"tag":647,"props":2314,"children":2315},{"style":1692},[2316],{"type":56,"value":1695},{"type":50,"tag":647,"props":2318,"children":2319},{"class":649,"line":774},[2320,2325,2330,2335],{"type":50,"tag":647,"props":2321,"children":2322},{"style":690},[2323],{"type":56,"value":2324},"  python3",{"type":50,"tag":647,"props":2326,"children":2327},{"style":690},[2328],{"type":56,"value":2329}," -m",{"type":50,"tag":647,"props":2331,"children":2332},{"style":690},[2333],{"type":56,"value":2334}," sglang.launch_server",{"type":50,"tag":647,"props":2336,"children":2337},{"style":1692},[2338],{"type":56,"value":1695},{"type":50,"tag":647,"props":2340,"children":2341},{"class":649,"line":812},[2342,2347,2351,2355,2359,2363],{"type":50,"tag":647,"props":2343,"children":2344},{"style":690},[2345],{"type":56,"value":2346},"    --model-path",{"type":50,"tag":647,"props":2348,"children":2349},{"style":654},[2350],{"type":56,"value":1785},{"type":50,"tag":647,"props":2352,"children":2353},{"style":690},[2354],{"type":56,"value":1790},{"type":50,"tag":647,"props":2356,"children":2357},{"style":1692},[2358],{"type":56,"value":1795},{"type":50,"tag":647,"props":2360,"children":2361},{"style":654},[2362],{"type":56,"value":1800},{"type":50,"tag":647,"props":2364,"children":2365},{"style":1692},[2366],{"type":56,"value":1695},{"type":50,"tag":647,"props":2368,"children":2369},{"class":649,"line":850},[2370,2374,2378],{"type":50,"tag":647,"props":2371,"children":2372},{"style":690},[2373],{"type":56,"value":1812},{"type":50,"tag":647,"props":2375,"children":2376},{"style":764},[2377],{"type":56,"value":1817},{"type":50,"tag":647,"props":2379,"children":2380},{"style":1692},[2381],{"type":56,"value":1695},{"type":50,"tag":647,"props":2383,"children":2384},{"class":649,"line":875},[2385,2389,2393],{"type":50,"tag":647,"props":2386,"children":2387},{"style":690},[2388],{"type":56,"value":1829},{"type":50,"tag":647,"props":2390,"children":2391},{"style":764},[2392],{"type":56,"value":1834},{"type":50,"tag":647,"props":2394,"children":2395},{"style":1692},[2396],{"type":56,"value":1695},{"type":50,"tag":647,"props":2398,"children":2399},{"class":649,"line":897},[2400,2405,2409],{"type":50,"tag":647,"props":2401,"children":2402},{"style":690},[2403],{"type":56,"value":2404},"    --mem-fraction-static",{"type":50,"tag":647,"props":2406,"children":2407},{"style":764},[2408],{"type":56,"value":1851},{"type":50,"tag":647,"props":2410,"children":2411},{"style":1692},[2412],{"type":56,"value":1695},{"type":50,"tag":647,"props":2414,"children":2415},{"class":649,"line":918},[2416,2421],{"type":50,"tag":647,"props":2417,"children":2418},{"style":690},[2419],{"type":56,"value":2420},"    --max-running-requests",{"type":50,"tag":647,"props":2422,"children":2423},{"style":764},[2424],{"type":56,"value":2425}," 8\n",{"type":50,"tag":59,"props":2427,"children":2428},{},[2429,2431,2437],{"type":56,"value":2430},"For llama.cpp, use the NVIDIA-AI-IOT llama.cpp image when available, or the ",{"type":50,"tag":77,"props":2432,"children":2434},{"className":2433},[],[2435],{"type":56,"value":2436},"llama-server",{"type":56,"value":2438}," binary from a JetPack-matched build. Start with GGUF INT4 \u002F Q4_K_M on both Orin and Thor; choose a smaller INT4 GGUF model if the audit shows tight memory.",{"type":50,"tag":636,"props":2440,"children":2442},{"className":1630,"code":2441,"language":1632,"meta":641,"style":641},"docker run --rm -it --runtime nvidia --network host --name llama-cpp \\\n  -v \"$PWD\u002Fmodels:\u002Fmodels:ro\" \\\n  ghcr.io\u002Fnvidia-ai-iot\u002Fllama_cpp:latest-jetson-\u003Corin-or-thor> \\\n  llama-server \\\n    -m \u002Fmodels\u002F\u003Cmodel>.gguf \\\n    --host 0.0.0.0 \\\n    --port 8000 \\\n    -ngl 28 \\\n    -c 4096 \\\n    --no-mmap \\\n    --flash-attn\n",[2443],{"type":50,"tag":77,"props":2444,"children":2445},{"__ignoreMap":641},[2446,2494,2523,2554,2566,2606,2621,2636,2653,2669,2681],{"type":50,"tag":647,"props":2447,"children":2448},{"class":649,"line":650},[2449,2453,2457,2461,2465,2469,2473,2477,2481,2485,2490],{"type":50,"tag":647,"props":2450,"children":2451},{"style":1004},[2452],{"type":56,"value":1644},{"type":50,"tag":647,"props":2454,"children":2455},{"style":690},[2456],{"type":56,"value":1649},{"type":50,"tag":647,"props":2458,"children":2459},{"style":690},[2460],{"type":56,"value":1654},{"type":50,"tag":647,"props":2462,"children":2463},{"style":690},[2464],{"type":56,"value":1659},{"type":50,"tag":647,"props":2466,"children":2467},{"style":690},[2468],{"type":56,"value":1664},{"type":50,"tag":647,"props":2470,"children":2471},{"style":690},[2472],{"type":56,"value":1669},{"type":50,"tag":647,"props":2474,"children":2475},{"style":690},[2476],{"type":56,"value":1674},{"type":50,"tag":647,"props":2478,"children":2479},{"style":690},[2480],{"type":56,"value":1679},{"type":50,"tag":647,"props":2482,"children":2483},{"style":690},[2484],{"type":56,"value":1684},{"type":50,"tag":647,"props":2486,"children":2487},{"style":690},[2488],{"type":56,"value":2489}," llama-cpp",{"type":50,"tag":647,"props":2491,"children":2492},{"style":1692},[2493],{"type":56,"value":1695},{"type":50,"tag":647,"props":2495,"children":2496},{"class":649,"line":660},[2497,2501,2505,2510,2515,2519],{"type":50,"tag":647,"props":2498,"children":2499},{"style":690},[2500],{"type":56,"value":1703},{"type":50,"tag":647,"props":2502,"children":2503},{"style":654},[2504],{"type":56,"value":687},{"type":50,"tag":647,"props":2506,"children":2507},{"style":1692},[2508],{"type":56,"value":2509},"$PWD",{"type":50,"tag":647,"props":2511,"children":2512},{"style":690},[2513],{"type":56,"value":2514},"\u002Fmodels:\u002Fmodels:ro",{"type":50,"tag":647,"props":2516,"children":2517},{"style":654},[2518],{"type":56,"value":677},{"type":50,"tag":647,"props":2520,"children":2521},{"style":1692},[2522],{"type":56,"value":1695},{"type":50,"tag":647,"props":2524,"children":2525},{"class":649,"line":705},[2526,2531,2536,2541,2546,2550],{"type":50,"tag":647,"props":2527,"children":2528},{"style":690},[2529],{"type":56,"value":2530},"  ghcr.io\u002Fnvidia-ai-iot\u002Fllama_cpp:latest-jetson-",{"type":50,"tag":647,"props":2532,"children":2533},{"style":654},[2534],{"type":56,"value":2535},"\u003C",{"type":50,"tag":647,"props":2537,"children":2538},{"style":690},[2539],{"type":56,"value":2540},"orin-or-tho",{"type":50,"tag":647,"props":2542,"children":2543},{"style":1692},[2544],{"type":56,"value":2545},"r",{"type":50,"tag":647,"props":2547,"children":2548},{"style":654},[2549],{"type":56,"value":1800},{"type":50,"tag":647,"props":2551,"children":2552},{"style":1692},[2553],{"type":56,"value":1695},{"type":50,"tag":647,"props":2555,"children":2556},{"class":649,"line":743},[2557,2562],{"type":50,"tag":647,"props":2558,"children":2559},{"style":690},[2560],{"type":56,"value":2561},"  llama-server",{"type":50,"tag":647,"props":2563,"children":2564},{"style":1692},[2565],{"type":56,"value":1695},{"type":50,"tag":647,"props":2567,"children":2568},{"class":649,"line":774},[2569,2574,2579,2583,2588,2593,2597,2602],{"type":50,"tag":647,"props":2570,"children":2571},{"style":690},[2572],{"type":56,"value":2573},"    -m",{"type":50,"tag":647,"props":2575,"children":2576},{"style":690},[2577],{"type":56,"value":2578}," \u002Fmodels\u002F",{"type":50,"tag":647,"props":2580,"children":2581},{"style":654},[2582],{"type":56,"value":2535},{"type":50,"tag":647,"props":2584,"children":2585},{"style":690},[2586],{"type":56,"value":2587},"mode",{"type":50,"tag":647,"props":2589,"children":2590},{"style":1692},[2591],{"type":56,"value":2592},"l",{"type":50,"tag":647,"props":2594,"children":2595},{"style":654},[2596],{"type":56,"value":1800},{"type":50,"tag":647,"props":2598,"children":2599},{"style":690},[2600],{"type":56,"value":2601},".gguf",{"type":50,"tag":647,"props":2603,"children":2604},{"style":1692},[2605],{"type":56,"value":1695},{"type":50,"tag":647,"props":2607,"children":2608},{"class":649,"line":812},[2609,2613,2617],{"type":50,"tag":647,"props":2610,"children":2611},{"style":690},[2612],{"type":56,"value":1812},{"type":50,"tag":647,"props":2614,"children":2615},{"style":764},[2616],{"type":56,"value":1817},{"type":50,"tag":647,"props":2618,"children":2619},{"style":1692},[2620],{"type":56,"value":1695},{"type":50,"tag":647,"props":2622,"children":2623},{"class":649,"line":850},[2624,2628,2632],{"type":50,"tag":647,"props":2625,"children":2626},{"style":690},[2627],{"type":56,"value":1829},{"type":50,"tag":647,"props":2629,"children":2630},{"style":764},[2631],{"type":56,"value":1834},{"type":50,"tag":647,"props":2633,"children":2634},{"style":1692},[2635],{"type":56,"value":1695},{"type":50,"tag":647,"props":2637,"children":2638},{"class":649,"line":875},[2639,2644,2649],{"type":50,"tag":647,"props":2640,"children":2641},{"style":690},[2642],{"type":56,"value":2643},"    -ngl",{"type":50,"tag":647,"props":2645,"children":2646},{"style":764},[2647],{"type":56,"value":2648}," 28",{"type":50,"tag":647,"props":2650,"children":2651},{"style":1692},[2652],{"type":56,"value":1695},{"type":50,"tag":647,"props":2654,"children":2655},{"class":649,"line":897},[2656,2661,2665],{"type":50,"tag":647,"props":2657,"children":2658},{"style":690},[2659],{"type":56,"value":2660},"    -c",{"type":50,"tag":647,"props":2662,"children":2663},{"style":764},[2664],{"type":56,"value":1868},{"type":50,"tag":647,"props":2666,"children":2667},{"style":1692},[2668],{"type":56,"value":1695},{"type":50,"tag":647,"props":2670,"children":2671},{"class":649,"line":918},[2672,2677],{"type":50,"tag":647,"props":2673,"children":2674},{"style":690},[2675],{"type":56,"value":2676},"    --no-mmap",{"type":50,"tag":647,"props":2678,"children":2679},{"style":1692},[2680],{"type":56,"value":1695},{"type":50,"tag":647,"props":2682,"children":2683},{"class":649,"line":939},[2684],{"type":50,"tag":647,"props":2685,"children":2686},{"style":690},[2687],{"type":56,"value":2688},"    --flash-attn\n",{"type":50,"tag":65,"props":2690,"children":2692},{"id":2691},"procedure-the-script-encodes-this",[2693],{"type":56,"value":2694},"Procedure (the script encodes this)",{"type":50,"tag":338,"props":2696,"children":2697},{},[2698,2703,2708,2736],{"type":50,"tag":96,"props":2699,"children":2700},{},[2701],{"type":56,"value":2702},"Pick the lightest runtime that satisfies the user's required features (continuous batching? structured generation? CPU offload?).",{"type":50,"tag":96,"props":2704,"children":2705},{},[2706],{"type":56,"value":2707},"Pick the lowest precision that meets the user's accuracy bar (model-benchmarking skill).",{"type":50,"tag":96,"props":2709,"children":2710},{},[2711,2713,2719,2721,2727,2728,2734],{"type":56,"value":2712},"Sweep the runtime's memory knobs (start with ",{"type":50,"tag":77,"props":2714,"children":2716},{"className":2715},[],[2717],{"type":56,"value":2718},"gpu-memory-utilization",{"type":56,"value":2720}," for vLLM, ",{"type":50,"tag":77,"props":2722,"children":2724},{"className":2723},[],[2725],{"type":56,"value":2726},"n-gpu-layers",{"type":56,"value":113},{"type":50,"tag":77,"props":2729,"children":2731},{"className":2730},[],[2732],{"type":56,"value":2733},"ctx-size",{"type":56,"value":2735}," for llama.cpp) to find the minimum footprint that sustains target throughput.",{"type":50,"tag":96,"props":2737,"children":2738},{},[2739,2741,2746],{"type":56,"value":2740},"Re-measure with ",{"type":50,"tag":77,"props":2742,"children":2744},{"className":2743},[],[2745],{"type":56,"value":82},{"type":56,"value":196},{"type":50,"tag":65,"props":2748,"children":2750},{"id":2749},"safety",[2751],{"type":56,"value":2752},"Safety",{"type":50,"tag":59,"props":2754,"children":2755},{},[2756],{"type":56,"value":2757},"Read-only. The skill never starts, stops, or restarts a model server. It emits flags; the user (or an outer orchestration agent) is responsible for invoking the runtime.",{"type":50,"tag":2759,"props":2760,"children":2761},"style",{},[2762],{"type":56,"value":2763},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2765,"total":2866},[2766,2781,2795,2809,2821,2838,2852],{"slug":2767,"name":2767,"fn":2768,"description":2769,"org":2770,"tags":2771,"stars":23,"repoUrl":24,"updatedAt":2780},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2772,2775,2778,2779],{"name":2773,"slug":2774,"type":15},"Data Analysis","data-analysis",{"name":2776,"slug":2777,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:28:43.176466",{"slug":2782,"name":2782,"fn":2783,"description":2784,"org":2785,"tags":2786,"stars":23,"repoUrl":24,"updatedAt":2794},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2787,2790,2793],{"name":2788,"slug":2789,"type":15},"Deployment","deployment",{"name":2791,"slug":2792,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":2796,"name":2796,"fn":2797,"description":2798,"org":2799,"tags":2800,"stars":23,"repoUrl":24,"updatedAt":2808},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2801,2804,2805],{"name":2802,"slug":2803,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2806,"slug":2807,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":2810,"name":2810,"fn":2811,"description":2812,"org":2813,"tags":2814,"stars":23,"repoUrl":24,"updatedAt":2820},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2815,2816,2817],{"name":2773,"slug":2774,"type":15},{"name":9,"slug":8,"type":15},{"name":2818,"slug":2819,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":2822,"name":2822,"fn":2823,"description":2824,"org":2825,"tags":2826,"stars":23,"repoUrl":24,"updatedAt":2837},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2827,2830,2833,2834],{"name":2828,"slug":2829,"type":15},"Automation","automation",{"name":2831,"slug":2832,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":2835,"slug":2836,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":2839,"name":2839,"fn":2840,"description":2841,"org":2842,"tags":2843,"stars":23,"repoUrl":24,"updatedAt":2851},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2844,2845,2847,2848],{"name":2788,"slug":2789,"type":15},{"name":2846,"slug":1644,"type":15},"Docker",{"name":9,"slug":8,"type":15},{"name":2849,"slug":2850,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":2853,"name":2853,"fn":2854,"description":2855,"org":2856,"tags":2857,"stars":23,"repoUrl":24,"updatedAt":2865},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2858,2859,2862],{"name":9,"slug":8,"type":15},{"name":2860,"slug":2861,"type":15},"Quantum Computing","quantum-computing",{"name":2863,"slug":2864,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":2868,"total":3018},[2869,2887,2902,2913,2925,2939,2952,2966,2977,2986,3000,3009],{"slug":2870,"name":2870,"fn":2871,"description":2872,"org":2873,"tags":2874,"stars":2884,"repoUrl":2885,"updatedAt":2886},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2875,2878,2881],{"name":2876,"slug":2877,"type":15},"Documentation","documentation",{"name":2879,"slug":2880,"type":15},"MCP","mcp",{"name":2882,"slug":2883,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2888,"name":2888,"fn":2889,"description":2890,"org":2891,"tags":2892,"stars":2899,"repoUrl":2900,"updatedAt":2901},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2893,2896,2897],{"name":2894,"slug":2895,"type":15},"Containers","containers",{"name":2788,"slug":2789,"type":15},{"name":2898,"slug":44,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2903,"name":2903,"fn":2904,"description":2905,"org":2906,"tags":2907,"stars":2899,"repoUrl":2900,"updatedAt":2912},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2908,2911],{"name":2909,"slug":2910,"type":15},"CI\u002FCD","ci-cd",{"name":2788,"slug":2789,"type":15},"2026-07-14T05:25:59.97109",{"slug":2914,"name":2914,"fn":2915,"description":2916,"org":2917,"tags":2918,"stars":2899,"repoUrl":2900,"updatedAt":2924},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2919,2920,2921],{"name":2909,"slug":2910,"type":15},{"name":2788,"slug":2789,"type":15},{"name":2922,"slug":2923,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2926,"name":2926,"fn":2927,"description":2928,"org":2929,"tags":2930,"stars":2899,"repoUrl":2900,"updatedAt":2938},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2931,2934,2935],{"name":2932,"slug":2933,"type":15},"Debugging","debugging",{"name":2922,"slug":2923,"type":15},{"name":2936,"slug":2937,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2940,"name":2940,"fn":2941,"description":2942,"org":2943,"tags":2944,"stars":2899,"repoUrl":2900,"updatedAt":2951},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2945,2948],{"name":2946,"slug":2947,"type":15},"Best Practices","best-practices",{"name":2949,"slug":2950,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2953,"name":2953,"fn":2954,"description":2955,"org":2956,"tags":2957,"stars":2899,"repoUrl":2900,"updatedAt":2965},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2958,2961,2964],{"name":2959,"slug":2960,"type":15},"Machine Learning","machine-learning",{"name":2962,"slug":2963,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2967,"name":2967,"fn":2968,"description":2969,"org":2970,"tags":2971,"stars":2899,"repoUrl":2900,"updatedAt":2976},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2972,2975],{"name":2973,"slug":2974,"type":15},"QA","qa",{"name":2818,"slug":2819,"type":15},"2026-07-14T05:25:53.673039",{"slug":2978,"name":2978,"fn":2979,"description":2980,"org":2981,"tags":2982,"stars":2899,"repoUrl":2900,"updatedAt":2985},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2983,2984],{"name":2788,"slug":2789,"type":15},{"name":2791,"slug":2792,"type":15},"2026-07-14T05:25:49.362534",{"slug":2987,"name":2987,"fn":2988,"description":2989,"org":2990,"tags":2991,"stars":2899,"repoUrl":2900,"updatedAt":2999},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2992,2995,2996],{"name":2993,"slug":2994,"type":15},"Code Review","code-review",{"name":2922,"slug":2923,"type":15},{"name":2997,"slug":2998,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":3001,"name":3001,"fn":3002,"description":3003,"org":3004,"tags":3005,"stars":2899,"repoUrl":2900,"updatedAt":3008},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3006,3007],{"name":2973,"slug":2974,"type":15},{"name":2818,"slug":2819,"type":15},"2026-07-14T05:25:54.928983",{"slug":3010,"name":3010,"fn":3011,"description":3012,"org":3013,"tags":3014,"stars":2899,"repoUrl":2900,"updatedAt":3017},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3015,3016],{"name":2828,"slug":2829,"type":15},{"name":2909,"slug":2910,"type":15},"2026-07-30T05:29:03.275638",496]