[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-dgx-diagnose":3,"mdc--qws7vl-key":34,"related-repo-nvidia-dgx-diagnose":885,"related-org-nvidia-dgx-diagnose":981},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"dgx-diagnose","diagnose NVIDIA DGX Station hardware issues","Diagnose common DGX Station GB300 issues — CUDA crashes, wrong-GPU targeting, vLLM\u002FSGLang container bugs, MIG state problems, NVLink\u002FFabric Manager errors, X\u002FVulkan failures, HuggingFace auth, and port conflicts. Use when the user reports a GPU error, inference server crash, MIG problem, or any unexplained DGX Station failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,17,20],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":9,"slug":8,"type":15},{"name":18,"slug":19,"type":15},"AI Infrastructure","ai-infrastructure",{"name":21,"slug":22,"type":15},"Debugging","debugging",1144,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fdgx-spark-playbooks","2026-07-14T05:31:04.085598",null,249,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"Collection of step-by-step playbooks for setting up AI\u002FML workloads on NVIDIA DGX Spark devices with Blackwell architecture.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fdgx-spark-playbooks\u002Ftree\u002FHEAD\u002Fnvidia\u002Fstation-ai-skills\u002Fassets\u002Fskills\u002Fdgx-diagnose","---\nname: dgx-diagnose\ndescription: Diagnose common DGX Station GB300 issues — CUDA crashes, wrong-GPU targeting, vLLM\u002FSGLang container bugs, MIG state problems, NVLink\u002FFabric Manager errors, X\u002FVulkan failures, HuggingFace auth, and port conflicts. Use when the user reports a GPU error, inference server crash, MIG problem, or any unexplained DGX Station failure.\nmetadata:\n  publisher: nvidia\n  hardware: DGX Station GB300\n---\n\n# DGX Station Diagnostics\n\nDiagnose common DGX Station issues. Run through the checks below to identify the problem.\n\n## Step 1. Gather system state\n\nRun these commands and analyze the output:\n\n```bash\n# GPU status\nnvidia-smi\n\n# GPU device list with indices\nnvidia-smi --query-gpu=index,name,memory.used,memory.total --format=csv,noheader\n\n# Driver version\nnvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1\n\n# MIG state\nnvidia-smi -i 1 -q 2>\u002Fdev\u002Fnull | grep -i \"MIG Mode\" || echo \"Could not query MIG on device 1\"\n\n# Fabric Manager\nsystemctl is-active nvidia-fabricmanager\n\n# GPU processes\nsudo fuser -v \u002Fdev\u002Fnvidia* 2>\u002Fdev\u002Fnull || echo \"No GPU processes found\"\n\n# Docker containers using GPUs\ndocker ps --format \"table {{.Names}}\\t{{.Image}}\\t{{.Status}}\" 2>\u002Fdev\u002Fnull\n```\n\n## Step 2. Match symptoms to known issues\n\nBased on the gathered state and the user's reported problem, check for these known issues:\n\n### CUDA crashes with `--gpus all`\n**Cause:** Mixed coherency — GB300 (ATS) and RTX PRO (non-ATS) cannot share a CUDA context.\n**Fix:** Use `--gpus '\"device=N\"'` targeting only the GB300.\n\n### Model running on wrong GPU (RTX PRO instead of GB300)\n**Check:** The device index in the docker command vs actual GPU indices.\n**Fix:** Verify with `nvidia-smi --query-gpu=index,name --format=csv,noheader` and correct the `--gpus` flag.\n\n### vLLM crash \u002F FlashInfer buffer overflow\n**Check:** Container version — `docker inspect vllm-server | grep Image`\n**Fix:** Use `nvcr.io\u002Fnvidia\u002Fvllm:26.01-py3`. Version 25.10 has a known FlashInfer bug on DGX Station.\n\n### SGLang CUDA errors\n**Check:** Container tag — must be `cu130` for Blackwell SM103.\n**Fix:** Use `lmsysorg\u002Fsglang:latest-cu130`.\n\n### CUDA OOM despite 279 GB HBM\n**Check:** `--max-model-len` \u002F `--context-length` and memory utilization settings.\n**Fix:** Reduce context length or lower `--gpu-memory-utilization` \u002F `--mem-fraction-static`.\n\n### `nvidia-smi -mig 1` returns \"In use by another client\"\n**Check:** `sudo fuser -v \u002Fdev\u002Fnvidia*` — GPU processes must be stopped first.\n**Fix:** Stop all GPU workloads, then retry.\n\n### NVLink errors after disabling MIG\n**Check:** `systemctl is-active nvidia-fabricmanager`\n**Fix:** `sudo systemctl start nvidia-fabricmanager`\n\n### X server crash after nvidia-xconfig -a\n**Fix:** `sudo cp \u002Fetc\u002FX11\u002Fxorg.conf.nvidia-xconfig-original \u002Fetc\u002FX11\u002Fxorg.conf`\n\n### Vulkan VK_ERROR_INITIALIZATION_FAILED\n**Cause:** CUDA initialized before Vulkan, binding to GB300.\n**Fix:** Run CUDA and Vulkan workloads in separate processes. For Vulkan apps: `__GL_DeviceModalityPreference=2 .\u002Fyour_app`\n\n### HuggingFace 401 \u002F token errors\n**Fix:** Pass token inline: `-e HF_TOKEN=\"hf_...\"`. Don't rely on shell export for background Docker tasks.\n\n### Port already in use\n**Check:** `lsof -i :\u003CPORT>`\n**Fix:** Stop the conflicting process or use a different host port: `-p 8001:8000`.\n\n## Step 3. Report findings\n\nTell the user:\n1. What the issue is\n2. Why it happens (root cause)\n3. The specific command to fix it\n4. How to verify the fix worked\n",{"data":35,"body":38},{"name":4,"description":6,"metadata":36},{"publisher":8,"hardware":37},"DGX Station GB300",{"type":39,"children":40},"root",[41,50,56,63,68,466,472,477,490,516,522,554,560,588,594,624,630,675,687,709,715,740,746,760,766,787,793,810,816,843,849,854,879],{"type":42,"tag":43,"props":44,"children":46},"element","h1",{"id":45},"dgx-station-diagnostics",[47],{"type":48,"value":49},"text","DGX Station Diagnostics",{"type":42,"tag":51,"props":52,"children":53},"p",{},[54],{"type":48,"value":55},"Diagnose common DGX Station issues. Run through the checks below to identify the problem.",{"type":42,"tag":57,"props":58,"children":60},"h2",{"id":59},"step-1-gather-system-state",[61],{"type":48,"value":62},"Step 1. Gather system state",{"type":42,"tag":51,"props":64,"children":65},{},[66],{"type":48,"value":67},"Run these commands and analyze the output:",{"type":42,"tag":69,"props":70,"children":75},"pre",{"className":71,"code":72,"language":73,"meta":74,"style":74},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# GPU status\nnvidia-smi\n\n# GPU device list with indices\nnvidia-smi --query-gpu=index,name,memory.used,memory.total --format=csv,noheader\n\n# Driver version\nnvidia-smi --query-gpu=driver_version --format=csv,noheader | head -1\n\n# MIG state\nnvidia-smi -i 1 -q 2>\u002Fdev\u002Fnull | grep -i \"MIG Mode\" || echo \"Could not query MIG on device 1\"\n\n# Fabric Manager\nsystemctl is-active nvidia-fabricmanager\n\n# GPU processes\nsudo fuser -v \u002Fdev\u002Fnvidia* 2>\u002Fdev\u002Fnull || echo \"No GPU processes found\"\n\n# Docker containers using GPUs\ndocker ps --format \"table {{.Names}}\\t{{.Image}}\\t{{.Status}}\" 2>\u002Fdev\u002Fnull\n","bash","",[76],{"type":42,"tag":77,"props":78,"children":79},"code",{"__ignoreMap":74},[80,92,102,112,121,141,149,158,192,200,209,296,304,313,332,340,349,408,416,425],{"type":42,"tag":81,"props":82,"children":85},"span",{"class":83,"line":84},"line",1,[86],{"type":42,"tag":81,"props":87,"children":89},{"style":88},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[90],{"type":48,"value":91},"# GPU status\n",{"type":42,"tag":81,"props":93,"children":95},{"class":83,"line":94},2,[96],{"type":42,"tag":81,"props":97,"children":99},{"style":98},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[100],{"type":48,"value":101},"nvidia-smi\n",{"type":42,"tag":81,"props":103,"children":105},{"class":83,"line":104},3,[106],{"type":42,"tag":81,"props":107,"children":109},{"emptyLinePlaceholder":108},true,[110],{"type":48,"value":111},"\n",{"type":42,"tag":81,"props":113,"children":115},{"class":83,"line":114},4,[116],{"type":42,"tag":81,"props":117,"children":118},{"style":88},[119],{"type":48,"value":120},"# GPU device list with indices\n",{"type":42,"tag":81,"props":122,"children":124},{"class":83,"line":123},5,[125,130,136],{"type":42,"tag":81,"props":126,"children":127},{"style":98},[128],{"type":48,"value":129},"nvidia-smi",{"type":42,"tag":81,"props":131,"children":133},{"style":132},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[134],{"type":48,"value":135}," --query-gpu=index,name,memory.used,memory.total",{"type":42,"tag":81,"props":137,"children":138},{"style":132},[139],{"type":48,"value":140}," --format=csv,noheader\n",{"type":42,"tag":81,"props":142,"children":144},{"class":83,"line":143},6,[145],{"type":42,"tag":81,"props":146,"children":147},{"emptyLinePlaceholder":108},[148],{"type":48,"value":111},{"type":42,"tag":81,"props":150,"children":152},{"class":83,"line":151},7,[153],{"type":42,"tag":81,"props":154,"children":155},{"style":88},[156],{"type":48,"value":157},"# Driver version\n",{"type":42,"tag":81,"props":159,"children":161},{"class":83,"line":160},8,[162,166,171,176,182,187],{"type":42,"tag":81,"props":163,"children":164},{"style":98},[165],{"type":48,"value":129},{"type":42,"tag":81,"props":167,"children":168},{"style":132},[169],{"type":48,"value":170}," --query-gpu=driver_version",{"type":42,"tag":81,"props":172,"children":173},{"style":132},[174],{"type":48,"value":175}," --format=csv,noheader",{"type":42,"tag":81,"props":177,"children":179},{"style":178},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[180],{"type":48,"value":181}," |",{"type":42,"tag":81,"props":183,"children":184},{"style":98},[185],{"type":48,"value":186}," head",{"type":42,"tag":81,"props":188,"children":189},{"style":132},[190],{"type":48,"value":191}," -1\n",{"type":42,"tag":81,"props":193,"children":195},{"class":83,"line":194},9,[196],{"type":42,"tag":81,"props":197,"children":198},{"emptyLinePlaceholder":108},[199],{"type":48,"value":111},{"type":42,"tag":81,"props":201,"children":203},{"class":83,"line":202},10,[204],{"type":42,"tag":81,"props":205,"children":206},{"style":88},[207],{"type":48,"value":208},"# MIG state\n",{"type":42,"tag":81,"props":210,"children":212},{"class":83,"line":211},11,[213,217,222,228,233,238,243,247,252,256,261,266,271,276,282,286,291],{"type":42,"tag":81,"props":214,"children":215},{"style":98},[216],{"type":48,"value":129},{"type":42,"tag":81,"props":218,"children":219},{"style":132},[220],{"type":48,"value":221}," -i",{"type":42,"tag":81,"props":223,"children":225},{"style":224},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[226],{"type":48,"value":227}," 1",{"type":42,"tag":81,"props":229,"children":230},{"style":132},[231],{"type":48,"value":232}," -q",{"type":42,"tag":81,"props":234,"children":235},{"style":178},[236],{"type":48,"value":237}," 2>",{"type":42,"tag":81,"props":239,"children":240},{"style":132},[241],{"type":48,"value":242},"\u002Fdev\u002Fnull",{"type":42,"tag":81,"props":244,"children":245},{"style":178},[246],{"type":48,"value":181},{"type":42,"tag":81,"props":248,"children":249},{"style":98},[250],{"type":48,"value":251}," grep",{"type":42,"tag":81,"props":253,"children":254},{"style":132},[255],{"type":48,"value":221},{"type":42,"tag":81,"props":257,"children":258},{"style":178},[259],{"type":48,"value":260}," \"",{"type":42,"tag":81,"props":262,"children":263},{"style":132},[264],{"type":48,"value":265},"MIG Mode",{"type":42,"tag":81,"props":267,"children":268},{"style":178},[269],{"type":48,"value":270},"\"",{"type":42,"tag":81,"props":272,"children":273},{"style":178},[274],{"type":48,"value":275}," ||",{"type":42,"tag":81,"props":277,"children":279},{"style":278},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[280],{"type":48,"value":281}," echo",{"type":42,"tag":81,"props":283,"children":284},{"style":178},[285],{"type":48,"value":260},{"type":42,"tag":81,"props":287,"children":288},{"style":132},[289],{"type":48,"value":290},"Could not query MIG on device 1",{"type":42,"tag":81,"props":292,"children":293},{"style":178},[294],{"type":48,"value":295},"\"\n",{"type":42,"tag":81,"props":297,"children":299},{"class":83,"line":298},12,[300],{"type":42,"tag":81,"props":301,"children":302},{"emptyLinePlaceholder":108},[303],{"type":48,"value":111},{"type":42,"tag":81,"props":305,"children":307},{"class":83,"line":306},13,[308],{"type":42,"tag":81,"props":309,"children":310},{"style":88},[311],{"type":48,"value":312},"# Fabric Manager\n",{"type":42,"tag":81,"props":314,"children":316},{"class":83,"line":315},14,[317,322,327],{"type":42,"tag":81,"props":318,"children":319},{"style":98},[320],{"type":48,"value":321},"systemctl",{"type":42,"tag":81,"props":323,"children":324},{"style":132},[325],{"type":48,"value":326}," is-active",{"type":42,"tag":81,"props":328,"children":329},{"style":132},[330],{"type":48,"value":331}," nvidia-fabricmanager\n",{"type":42,"tag":81,"props":333,"children":335},{"class":83,"line":334},15,[336],{"type":42,"tag":81,"props":337,"children":338},{"emptyLinePlaceholder":108},[339],{"type":48,"value":111},{"type":42,"tag":81,"props":341,"children":343},{"class":83,"line":342},16,[344],{"type":42,"tag":81,"props":345,"children":346},{"style":88},[347],{"type":48,"value":348},"# GPU processes\n",{"type":42,"tag":81,"props":350,"children":352},{"class":83,"line":351},17,[353,358,363,368,373,379,383,387,391,395,399,404],{"type":42,"tag":81,"props":354,"children":355},{"style":98},[356],{"type":48,"value":357},"sudo",{"type":42,"tag":81,"props":359,"children":360},{"style":132},[361],{"type":48,"value":362}," fuser",{"type":42,"tag":81,"props":364,"children":365},{"style":132},[366],{"type":48,"value":367}," -v",{"type":42,"tag":81,"props":369,"children":370},{"style":132},[371],{"type":48,"value":372}," \u002Fdev\u002Fnvidia",{"type":42,"tag":81,"props":374,"children":376},{"style":375},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[377],{"type":48,"value":378},"*",{"type":42,"tag":81,"props":380,"children":381},{"style":178},[382],{"type":48,"value":237},{"type":42,"tag":81,"props":384,"children":385},{"style":132},[386],{"type":48,"value":242},{"type":42,"tag":81,"props":388,"children":389},{"style":178},[390],{"type":48,"value":275},{"type":42,"tag":81,"props":392,"children":393},{"style":278},[394],{"type":48,"value":281},{"type":42,"tag":81,"props":396,"children":397},{"style":178},[398],{"type":48,"value":260},{"type":42,"tag":81,"props":400,"children":401},{"style":132},[402],{"type":48,"value":403},"No GPU processes found",{"type":42,"tag":81,"props":405,"children":406},{"style":178},[407],{"type":48,"value":295},{"type":42,"tag":81,"props":409,"children":411},{"class":83,"line":410},18,[412],{"type":42,"tag":81,"props":413,"children":414},{"emptyLinePlaceholder":108},[415],{"type":48,"value":111},{"type":42,"tag":81,"props":417,"children":419},{"class":83,"line":418},19,[420],{"type":42,"tag":81,"props":421,"children":422},{"style":88},[423],{"type":48,"value":424},"# Docker containers using GPUs\n",{"type":42,"tag":81,"props":426,"children":428},{"class":83,"line":427},20,[429,434,439,444,448,453,457,461],{"type":42,"tag":81,"props":430,"children":431},{"style":98},[432],{"type":48,"value":433},"docker",{"type":42,"tag":81,"props":435,"children":436},{"style":132},[437],{"type":48,"value":438}," ps",{"type":42,"tag":81,"props":440,"children":441},{"style":132},[442],{"type":48,"value":443}," --format",{"type":42,"tag":81,"props":445,"children":446},{"style":178},[447],{"type":48,"value":260},{"type":42,"tag":81,"props":449,"children":450},{"style":132},[451],{"type":48,"value":452},"table {{.Names}}\\t{{.Image}}\\t{{.Status}}",{"type":42,"tag":81,"props":454,"children":455},{"style":178},[456],{"type":48,"value":270},{"type":42,"tag":81,"props":458,"children":459},{"style":178},[460],{"type":48,"value":237},{"type":42,"tag":81,"props":462,"children":463},{"style":132},[464],{"type":48,"value":465},"\u002Fdev\u002Fnull\n",{"type":42,"tag":57,"props":467,"children":469},{"id":468},"step-2-match-symptoms-to-known-issues",[470],{"type":48,"value":471},"Step 2. Match symptoms to known issues",{"type":42,"tag":51,"props":473,"children":474},{},[475],{"type":48,"value":476},"Based on the gathered state and the user's reported problem, check for these known issues:",{"type":42,"tag":478,"props":479,"children":481},"h3",{"id":480},"cuda-crashes-with-gpus-all",[482,484],{"type":48,"value":483},"CUDA crashes with ",{"type":42,"tag":77,"props":485,"children":487},{"className":486},[],[488],{"type":48,"value":489},"--gpus all",{"type":42,"tag":51,"props":491,"children":492},{},[493,499,501,506,508,514],{"type":42,"tag":494,"props":495,"children":496},"strong",{},[497],{"type":48,"value":498},"Cause:",{"type":48,"value":500}," Mixed coherency — GB300 (ATS) and RTX PRO (non-ATS) cannot share a CUDA context.\n",{"type":42,"tag":494,"props":502,"children":503},{},[504],{"type":48,"value":505},"Fix:",{"type":48,"value":507}," Use ",{"type":42,"tag":77,"props":509,"children":511},{"className":510},[],[512],{"type":48,"value":513},"--gpus '\"device=N\"'",{"type":48,"value":515}," targeting only the GB300.",{"type":42,"tag":478,"props":517,"children":519},{"id":518},"model-running-on-wrong-gpu-rtx-pro-instead-of-gb300",[520],{"type":48,"value":521},"Model running on wrong GPU (RTX PRO instead of GB300)",{"type":42,"tag":51,"props":523,"children":524},{},[525,530,532,536,538,544,546,552],{"type":42,"tag":494,"props":526,"children":527},{},[528],{"type":48,"value":529},"Check:",{"type":48,"value":531}," The device index in the docker command vs actual GPU indices.\n",{"type":42,"tag":494,"props":533,"children":534},{},[535],{"type":48,"value":505},{"type":48,"value":537}," Verify with ",{"type":42,"tag":77,"props":539,"children":541},{"className":540},[],[542],{"type":48,"value":543},"nvidia-smi --query-gpu=index,name --format=csv,noheader",{"type":48,"value":545}," and correct the ",{"type":42,"tag":77,"props":547,"children":549},{"className":548},[],[550],{"type":48,"value":551},"--gpus",{"type":48,"value":553}," flag.",{"type":42,"tag":478,"props":555,"children":557},{"id":556},"vllm-crash-flashinfer-buffer-overflow",[558],{"type":48,"value":559},"vLLM crash \u002F FlashInfer buffer overflow",{"type":42,"tag":51,"props":561,"children":562},{},[563,567,569,575,579,580,586],{"type":42,"tag":494,"props":564,"children":565},{},[566],{"type":48,"value":529},{"type":48,"value":568}," Container version — ",{"type":42,"tag":77,"props":570,"children":572},{"className":571},[],[573],{"type":48,"value":574},"docker inspect vllm-server | grep Image",{"type":42,"tag":494,"props":576,"children":577},{},[578],{"type":48,"value":505},{"type":48,"value":507},{"type":42,"tag":77,"props":581,"children":583},{"className":582},[],[584],{"type":48,"value":585},"nvcr.io\u002Fnvidia\u002Fvllm:26.01-py3",{"type":48,"value":587},". Version 25.10 has a known FlashInfer bug on DGX Station.",{"type":42,"tag":478,"props":589,"children":591},{"id":590},"sglang-cuda-errors",[592],{"type":48,"value":593},"SGLang CUDA errors",{"type":42,"tag":51,"props":595,"children":596},{},[597,601,603,609,611,615,616,622],{"type":42,"tag":494,"props":598,"children":599},{},[600],{"type":48,"value":529},{"type":48,"value":602}," Container tag — must be ",{"type":42,"tag":77,"props":604,"children":606},{"className":605},[],[607],{"type":48,"value":608},"cu130",{"type":48,"value":610}," for Blackwell SM103.\n",{"type":42,"tag":494,"props":612,"children":613},{},[614],{"type":48,"value":505},{"type":48,"value":507},{"type":42,"tag":77,"props":617,"children":619},{"className":618},[],[620],{"type":48,"value":621},"lmsysorg\u002Fsglang:latest-cu130",{"type":48,"value":623},".",{"type":42,"tag":478,"props":625,"children":627},{"id":626},"cuda-oom-despite-279-gb-hbm",[628],{"type":48,"value":629},"CUDA OOM despite 279 GB HBM",{"type":42,"tag":51,"props":631,"children":632},{},[633,637,639,645,647,653,655,659,661,667,668,674],{"type":42,"tag":494,"props":634,"children":635},{},[636],{"type":48,"value":529},{"type":48,"value":638}," ",{"type":42,"tag":77,"props":640,"children":642},{"className":641},[],[643],{"type":48,"value":644},"--max-model-len",{"type":48,"value":646}," \u002F ",{"type":42,"tag":77,"props":648,"children":650},{"className":649},[],[651],{"type":48,"value":652},"--context-length",{"type":48,"value":654}," and memory utilization settings.\n",{"type":42,"tag":494,"props":656,"children":657},{},[658],{"type":48,"value":505},{"type":48,"value":660}," Reduce context length or lower ",{"type":42,"tag":77,"props":662,"children":664},{"className":663},[],[665],{"type":48,"value":666},"--gpu-memory-utilization",{"type":48,"value":646},{"type":42,"tag":77,"props":669,"children":671},{"className":670},[],[672],{"type":48,"value":673},"--mem-fraction-static",{"type":48,"value":623},{"type":42,"tag":478,"props":676,"children":678},{"id":677},"nvidia-smi-mig-1-returns-in-use-by-another-client",[679,685],{"type":42,"tag":77,"props":680,"children":682},{"className":681},[],[683],{"type":48,"value":684},"nvidia-smi -mig 1",{"type":48,"value":686}," returns \"In use by another client\"",{"type":42,"tag":51,"props":688,"children":689},{},[690,694,695,701,703,707],{"type":42,"tag":494,"props":691,"children":692},{},[693],{"type":48,"value":529},{"type":48,"value":638},{"type":42,"tag":77,"props":696,"children":698},{"className":697},[],[699],{"type":48,"value":700},"sudo fuser -v \u002Fdev\u002Fnvidia*",{"type":48,"value":702}," — GPU processes must be stopped first.\n",{"type":42,"tag":494,"props":704,"children":705},{},[706],{"type":48,"value":505},{"type":48,"value":708}," Stop all GPU workloads, then retry.",{"type":42,"tag":478,"props":710,"children":712},{"id":711},"nvlink-errors-after-disabling-mig",[713],{"type":48,"value":714},"NVLink errors after disabling MIG",{"type":42,"tag":51,"props":716,"children":717},{},[718,722,723,729,733,734],{"type":42,"tag":494,"props":719,"children":720},{},[721],{"type":48,"value":529},{"type":48,"value":638},{"type":42,"tag":77,"props":724,"children":726},{"className":725},[],[727],{"type":48,"value":728},"systemctl is-active nvidia-fabricmanager",{"type":42,"tag":494,"props":730,"children":731},{},[732],{"type":48,"value":505},{"type":48,"value":638},{"type":42,"tag":77,"props":735,"children":737},{"className":736},[],[738],{"type":48,"value":739},"sudo systemctl start nvidia-fabricmanager",{"type":42,"tag":478,"props":741,"children":743},{"id":742},"x-server-crash-after-nvidia-xconfig-a",[744],{"type":48,"value":745},"X server crash after nvidia-xconfig -a",{"type":42,"tag":51,"props":747,"children":748},{},[749,753,754],{"type":42,"tag":494,"props":750,"children":751},{},[752],{"type":48,"value":505},{"type":48,"value":638},{"type":42,"tag":77,"props":755,"children":757},{"className":756},[],[758],{"type":48,"value":759},"sudo cp \u002Fetc\u002FX11\u002Fxorg.conf.nvidia-xconfig-original \u002Fetc\u002FX11\u002Fxorg.conf",{"type":42,"tag":478,"props":761,"children":763},{"id":762},"vulkan-vk_error_initialization_failed",[764],{"type":48,"value":765},"Vulkan VK_ERROR_INITIALIZATION_FAILED",{"type":42,"tag":51,"props":767,"children":768},{},[769,773,775,779,781],{"type":42,"tag":494,"props":770,"children":771},{},[772],{"type":48,"value":498},{"type":48,"value":774}," CUDA initialized before Vulkan, binding to GB300.\n",{"type":42,"tag":494,"props":776,"children":777},{},[778],{"type":48,"value":505},{"type":48,"value":780}," Run CUDA and Vulkan workloads in separate processes. For Vulkan apps: ",{"type":42,"tag":77,"props":782,"children":784},{"className":783},[],[785],{"type":48,"value":786},"__GL_DeviceModalityPreference=2 .\u002Fyour_app",{"type":42,"tag":478,"props":788,"children":790},{"id":789},"huggingface-401-token-errors",[791],{"type":48,"value":792},"HuggingFace 401 \u002F token errors",{"type":42,"tag":51,"props":794,"children":795},{},[796,800,802,808],{"type":42,"tag":494,"props":797,"children":798},{},[799],{"type":48,"value":505},{"type":48,"value":801}," Pass token inline: ",{"type":42,"tag":77,"props":803,"children":805},{"className":804},[],[806],{"type":48,"value":807},"-e HF_TOKEN=\"hf_...\"",{"type":48,"value":809},". Don't rely on shell export for background Docker tasks.",{"type":42,"tag":478,"props":811,"children":813},{"id":812},"port-already-in-use",[814],{"type":48,"value":815},"Port already in use",{"type":42,"tag":51,"props":817,"children":818},{},[819,823,824,830,834,836,842],{"type":42,"tag":494,"props":820,"children":821},{},[822],{"type":48,"value":529},{"type":48,"value":638},{"type":42,"tag":77,"props":825,"children":827},{"className":826},[],[828],{"type":48,"value":829},"lsof -i :\u003CPORT>",{"type":42,"tag":494,"props":831,"children":832},{},[833],{"type":48,"value":505},{"type":48,"value":835}," Stop the conflicting process or use a different host port: ",{"type":42,"tag":77,"props":837,"children":839},{"className":838},[],[840],{"type":48,"value":841},"-p 8001:8000",{"type":48,"value":623},{"type":42,"tag":57,"props":844,"children":846},{"id":845},"step-3-report-findings",[847],{"type":48,"value":848},"Step 3. Report findings",{"type":42,"tag":51,"props":850,"children":851},{},[852],{"type":48,"value":853},"Tell the user:",{"type":42,"tag":855,"props":856,"children":857},"ol",{},[858,864,869,874],{"type":42,"tag":859,"props":860,"children":861},"li",{},[862],{"type":48,"value":863},"What the issue is",{"type":42,"tag":859,"props":865,"children":866},{},[867],{"type":48,"value":868},"Why it happens (root cause)",{"type":42,"tag":859,"props":870,"children":871},{},[872],{"type":48,"value":873},"The specific command to fix it",{"type":42,"tag":859,"props":875,"children":876},{},[877],{"type":48,"value":878},"How to verify the fix worked",{"type":42,"tag":880,"props":881,"children":882},"style",{},[883],{"type":48,"value":884},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":886,"total":211},[887,907,920,935,950,961,968],{"slug":888,"name":888,"fn":889,"description":890,"org":891,"tags":892,"stars":23,"repoUrl":24,"updatedAt":906},"analysis-methods","write Python analysis code for FHIR data","Teaches the analyst agent how to write correct, robust Python analysis code for FHIR clinical data using pandas, matplotlib, and scipy.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[893,896,899,902,903],{"name":894,"slug":895,"type":15},"Data Analysis","data-analysis",{"name":897,"slug":898,"type":15},"FHIR","fhir",{"name":900,"slug":901,"type":15},"Healthcare","healthcare",{"name":9,"slug":8,"type":15},{"name":904,"slug":905,"type":15},"Python","python","2026-07-14T05:35:59.037962",{"slug":908,"name":908,"fn":909,"description":910,"org":911,"tags":912,"stars":23,"repoUrl":24,"updatedAt":919},"case-summary","summarize clinical patient cases from FHIR","Prepare a complete clinical case summary for a patient from FHIR endpoints. Use when asked to summarize a patient, compile a case, or prepare for tumor board.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[913,914,915,916],{"name":897,"slug":898,"type":15},{"name":900,"slug":901,"type":15},{"name":9,"slug":8,"type":15},{"name":917,"slug":918,"type":15},"Summarization","summarization","2026-07-14T05:35:52.790528",{"slug":921,"name":921,"fn":922,"description":923,"org":924,"tags":925,"stars":23,"repoUrl":24,"updatedAt":934},"clinical-delegation","delegate clinical tasks to specialist agents","How to delegate clinical tasks to specialist agents. Always use sub-agent runtime with explicit agentId — never ACP. Never call FHIR via web_fetch.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[926,929,930,933],{"name":927,"slug":928,"type":15},"Agents","agents",{"name":900,"slug":901,"type":15},{"name":931,"slug":932,"type":15},"Multi-Agent","multi-agent",{"name":9,"slug":8,"type":15},"2026-07-14T05:35:55.294972",{"slug":936,"name":936,"fn":937,"description":938,"org":939,"tags":940,"stars":23,"repoUrl":24,"updatedAt":949},"clinical-knowledge","provide clinical reference and regulatory context","Teaches agents clinical reference ranges, condition codes, quality measure definitions, drug classifications, and regulatory context so they can flag abnormal values and identify care gaps.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[941,944,945,946],{"name":942,"slug":943,"type":15},"Clinical Trials","clinical-trials",{"name":900,"slug":901,"type":15},{"name":9,"slug":8,"type":15},{"name":947,"slug":948,"type":15},"Regulatory Compliance","regulatory-compliance","2026-07-14T05:35:56.550833",{"slug":951,"name":951,"fn":952,"description":953,"org":954,"tags":955,"stars":23,"repoUrl":24,"updatedAt":960},"cohort-compare","analyze patient cohorts from FHIR endpoints","Analyze a cohort of patients from FHIR endpoints to find care gaps and patterns. Use when asked to compare patients, find quality gaps, or analyze a population.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[956,957,958,959],{"name":894,"slug":895,"type":15},{"name":897,"slug":898,"type":15},{"name":900,"slug":901,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:35:51.543095",{"slug":4,"name":4,"fn":5,"description":6,"org":962,"tags":963,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[964,965,966,967],{"name":18,"slug":19,"type":15},{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"slug":969,"name":969,"fn":970,"description":971,"org":972,"tags":973,"stars":23,"repoUrl":24,"updatedAt":980},"fhir-basics","query and parse FHIR R4 API resources","Teaches agents how FHIR R4 APIs work, what resources are available, how to query them with search parameters, and how to correctly parse all response formats including component Observations.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[974,977,978,979],{"name":975,"slug":976,"type":15},"API Development","api-development",{"name":897,"slug":898,"type":15},{"name":900,"slug":901,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:35:57.797226",{"items":982,"total":1137},[983,1001,1017,1028,1040,1052,1065,1079,1092,1103,1117,1126],{"slug":984,"name":984,"fn":985,"description":986,"org":987,"tags":988,"stars":998,"repoUrl":999,"updatedAt":1000},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[989,992,995],{"name":990,"slug":991,"type":15},"Documentation","documentation",{"name":993,"slug":994,"type":15},"MCP","mcp",{"name":996,"slug":997,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":1002,"name":1002,"fn":1003,"description":1004,"org":1005,"tags":1006,"stars":1014,"repoUrl":1015,"updatedAt":1016},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1007,1010,1013],{"name":1008,"slug":1009,"type":15},"Containers","containers",{"name":1011,"slug":1012,"type":15},"Deployment","deployment",{"name":904,"slug":905,"type":15},17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":1018,"name":1018,"fn":1019,"description":1020,"org":1021,"tags":1022,"stars":1014,"repoUrl":1015,"updatedAt":1027},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1023,1026],{"name":1024,"slug":1025,"type":15},"CI\u002FCD","ci-cd",{"name":1011,"slug":1012,"type":15},"2026-07-14T05:25:59.97109",{"slug":1029,"name":1029,"fn":1030,"description":1031,"org":1032,"tags":1033,"stars":1014,"repoUrl":1015,"updatedAt":1039},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1034,1035,1036],{"name":1024,"slug":1025,"type":15},{"name":1011,"slug":1012,"type":15},{"name":1037,"slug":1038,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":1041,"name":1041,"fn":1042,"description":1043,"org":1044,"tags":1045,"stars":1014,"repoUrl":1015,"updatedAt":1051},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1046,1047,1048],{"name":21,"slug":22,"type":15},{"name":1037,"slug":1038,"type":15},{"name":1049,"slug":1050,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1053,"name":1053,"fn":1054,"description":1055,"org":1056,"tags":1057,"stars":1014,"repoUrl":1015,"updatedAt":1064},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1058,1061],{"name":1059,"slug":1060,"type":15},"Best Practices","best-practices",{"name":1062,"slug":1063,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1066,"name":1066,"fn":1067,"description":1068,"org":1069,"tags":1070,"stars":1014,"repoUrl":1015,"updatedAt":1078},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1071,1074,1077],{"name":1072,"slug":1073,"type":15},"Machine Learning","machine-learning",{"name":1075,"slug":1076,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":1080,"name":1080,"fn":1081,"description":1082,"org":1083,"tags":1084,"stars":1014,"repoUrl":1015,"updatedAt":1091},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1085,1088],{"name":1086,"slug":1087,"type":15},"QA","qa",{"name":1089,"slug":1090,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":1093,"name":1093,"fn":1094,"description":1095,"org":1096,"tags":1097,"stars":1014,"repoUrl":1015,"updatedAt":1102},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1098,1099],{"name":1011,"slug":1012,"type":15},{"name":1100,"slug":1101,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":1104,"name":1104,"fn":1105,"description":1106,"org":1107,"tags":1108,"stars":1014,"repoUrl":1015,"updatedAt":1116},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1109,1112,1113],{"name":1110,"slug":1111,"type":15},"Code Review","code-review",{"name":1037,"slug":1038,"type":15},{"name":1114,"slug":1115,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":1118,"name":1118,"fn":1119,"description":1120,"org":1121,"tags":1122,"stars":1014,"repoUrl":1015,"updatedAt":1125},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1123,1124],{"name":1086,"slug":1087,"type":15},{"name":1089,"slug":1090,"type":15},"2026-07-14T05:25:54.928983",{"slug":1127,"name":1127,"fn":1128,"description":1129,"org":1130,"tags":1131,"stars":1014,"repoUrl":1015,"updatedAt":1136},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1132,1135],{"name":1133,"slug":1134,"type":15},"Automation","automation",{"name":1024,"slug":1025,"type":15},"2026-07-30T05:29:03.275638",496]