[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-dynamo-interconnect-check":3,"mdc--uuc62a-key":37,"related-repo-nvidia-dynamo-interconnect-check":1194,"related-org-nvidia-dynamo-interconnect-check":1294},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":32,"sourceUrl":35,"mdContent":36},"dynamo-interconnect-check","validate Dynamo interconnect readiness","Validate that a Dynamo deployment's NIXL\u002FUCX\u002FNCCL interconnect is ready for disaggregated serving over RDMA\u002FNVLink. Use after recipe-runner brings a deployment up (especially disagg\u002Fmulti-node) to confirm the KV transport is correct; use troubleshoot for diagnosing already-failed pods.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,22,23],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":17,"slug":18,"type":15},"Networking","networking",{"name":20,"slug":21,"type":15},"Deployment","deployment",{"name":9,"slug":8,"type":15},{"name":24,"slug":25,"type":15},"Infrastructure","infrastructure",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:30:05.688732","Apache-2.0",281,[],{"repoUrl":27,"stars":26,"forks":30,"topics":33,"description":34},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fdynamo-interconnect-check","---\nname: dynamo-interconnect-check\ndescription: Validate that a Dynamo deployment's NIXL\u002FUCX\u002FNCCL interconnect is ready for disaggregated serving over RDMA\u002FNVLink. Use after recipe-runner brings a deployment up (especially disagg\u002Fmulti-node) to confirm the KV transport is correct; use troubleshoot for diagnosing already-failed pods.\nlicense: Apache-2.0\nmetadata:\n  author: Dan Gil \u003Cdagil@nvidia.com>\n  tags:\n    - dynamo\n    - nixl\n    - rdma\n    - disagg\n    - validation\n---\n\n# Dynamo Interconnect Check\n\n\u003C!--\nSPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved.\nSPDX-License-Identifier: CC-BY-4.0\n-->\n\n## Purpose\n\nConfirm that the transport disaggregated serving depends on actually works. A\ndeployment can pass an endpoint smoke test while disagg is silently wrong: if\nNIXL\u002FUCX cannot reach the peer worker over RDMA or NVLink, KV transfer falls\nback to a slow or broken path. Catch that with read-only checks before trusting\na disagg deployment or its benchmark numbers.\n\nThis skill is read-only. It never mutates the cluster and never prints secrets.\n\n## Prerequisites\n\n- Python 3.10+ on the operator machine.\n- `kubectl exec` access to a worker pod in the target Dynamo deployment.\n- Read access to the recipe directory (`recipes\u002F\u003Cmodel>\u002F\u003Cframework>\u002F\u003Cmode>`).\n- For node-capability checks: tools like `ibstat`, `nvidia-smi`, `lsmod` available in the worker pod image (missing tools are reported as `skipped`, not failures).\n\n## When To Use\n\n- After `dynamo-recipe-runner` deploys a **disagg** or multi-node recipe.\n- Before reporting disagg throughput\u002Flatency, so numbers reflect the real\n  transport.\n- When agg works but disagg is slow, hangs, or returns wrong output and you\n  suspect the fabric rather than the model.\n\nFor diagnosing pods that are already crashing or unschedulable, use\n`dynamo-troubleshoot` first.\n\n## Instructions\n\n### 1. Check Transport Env Vars On The Recipe\n\n```bash\npython3 scripts\u002Fcheck_interconnect.py env recipes\u002F\u003Cmodel>\u002F\u003Cframework>\u002F\u003Cmode>\n```\n\nReports which NIXL\u002FUCX\u002FNCCL transport variables are set and flags\ndisagg-critical ones (e.g. `UCX_TLS`, `UCX_NET_DEVICES`, `NCCL_IB_HCA`) that are\nabsent. Missing here is only a warning — they may be baked into the image — so\nconfirm with the node and NIXL checks. See\n`references\u002Finterconnect-env-vars.md` for what each variable does.\n\n### 2. Check Node Capabilities\n\nLocally on a GPU node, or inside a running worker pod:\n\n```bash\npython3 scripts\u002Fcheck_interconnect.py node \\\n  --namespace \"${NAMESPACE}\" --pod \u003Cworker-pod>\n```\n\nProbes (read-only) for: InfiniBand devices and Active links, GPUDirect RDMA\n(`nvidia_peermem`), GDRCopy, and NVLink in the GPU topology. Missing tools are\nreported as `skipped`, not failures.\n\n### 3. Validate NIXL Reachability\n\n```bash\npython3 scripts\u002Fcheck_interconnect.py nixl \\\n  --namespace \"${NAMESPACE}\" --pod \u003Cworker-pod>\n```\n\nLooks for NIXL test tooling in the pod and surfaces the exact next step to run a\npairwise prefill↔decode transfer test. A full cross-pod transfer test requires\ntwo scheduled GPU pods on the fabric.\n\n## Available Scripts\n\n| Script | Purpose | Arguments |\n|---|---|---|\n| `scripts\u002Fcheck_interconnect.py env` | Inspect NIXL\u002FUCX\u002FNCCL env vars on a recipe | positional recipe path |\n| `scripts\u002Fcheck_interconnect.py node` | Probe InfiniBand, GPUDirect RDMA, GDRCopy, NVLink on a node or pod | `--namespace`, `--pod` |\n| `scripts\u002Fcheck_interconnect.py nixl` | Surface NIXL transfer-test readiness for a pod | `--namespace`, `--pod` |\n\nInvoke via the agentskills.io `run_script()` protocol:\n\n```python\nrun_script(\"scripts\u002Fcheck_interconnect.py\", args=[\"env\", \"recipes\u002Fqwen3-coder-480b\u002Fsglang\u002Fdisagg\"])\nrun_script(\"scripts\u002Fcheck_interconnect.py\", args=[\"node\", \"--namespace\", \"dynamo-demo\", \"--pod\", \"qwen-worker-0\"])\n```\n\n## Examples\n\nVerify a disagg recipe's transport env shape before deploy:\n\n```bash\npython3 scripts\u002Fcheck_interconnect.py env recipes\u002Fqwen3-coder-480b\u002Fsglang\u002Fdisagg\n```\n\nAfter deploy, validate a worker pod's fabric:\n\n```bash\npython3 scripts\u002Fcheck_interconnect.py node \\\n  --namespace dynamo-demo --pod qwen-worker-0\npython3 scripts\u002Fcheck_interconnect.py nixl \\\n  --namespace dynamo-demo --pod qwen-worker-0\n```\n\nEquivalent through the agent protocol:\n\n```python\nrun_script(\"scripts\u002Fcheck_interconnect.py\", args=[\"nixl\", \"--namespace\", \"dynamo-demo\", \"--pod\", \"qwen-worker-0\"])\n```\n\n## Output Contract\n\nEach check returns `ok` \u002F `warn` \u002F `fail` \u002F `skipped` with a one-line detail,\nplus a rolled-up verdict on disagg transport readiness. Report:\n\n- transport env vars present vs. disagg-critical ones missing\n- RDMA \u002F GPUDirect \u002F NVLink capability status\n- whether NIXL reachability was validated, and the next command if not\n- a clear statement of whether disagg can be trusted, or what to fix first\n\n## Limitations\n\n- Read-only fabric probe; does not run a full pairwise NIXL transfer (requires two scheduled GPU pods and the in-pod NIXL test tools).\n- `skipped` results for missing tools (`ibstat`, `nvidia-smi`, `lsmod`) are inconclusive, not a pass.\n- Env-var check inspects the recipe text; values injected at runtime via initContainers or operator-applied envs are not detected.\n- Single-node agg deployments do not exercise the transport — this skill is for disagg \u002F multi-node validation.\n\n## Troubleshooting\n\n| Symptom | Likely cause | Next step |\n|---|---|---|\n| `env` reports all critical vars missing | Vars baked into image or injected by operator | Run the `node` check inside the worker pod to verify actual env |\n| `node` reports no Active IB link | Fabric down or HCA not provisioned to the node | Contact cluster admin; verify `kubectl describe node` shows `nvidia.com\u002Fgpu` and IB labels |\n| `nvidia_peermem` missing | GPUDirect RDMA module not loaded | Ask cluster admin to load `nvidia-peermem`; without it, NIXL falls back to staged copies |\n| `nixl` finds no test tools | Worker image lacks NIXL test harness | Use a NIXL-enabled image or run the standalone transfer test from a debug pod |\n\n## Benchmark\n\nSee `BENCHMARK.md` for the NVCARPS-EVAL performance report (auto-generated by the NVSkills CI pipeline). To refresh, re-run `\u002Fnvskills-ci` on an upstream PR touching this skill.\n\n## References\n\n- `references\u002Finterconnect-env-vars.md` — NIXL\u002FUCX\u002FNCCL env var catalog and IB\n  capability checklist.\n- Use `scripts\u002Fcheck_interconnect.py` for all read-only checks.\n",{"data":38,"body":47},{"name":4,"description":6,"license":29,"metadata":39},{"author":40,"tags":41},"Dan Gil \u003Cdagil@nvidia.com>",[42,43,44,45,46],"dynamo","nixl","rdma","disagg","validation",{"type":48,"children":49},"root",[50,58,65,71,76,82,153,159,192,205,211,218,325,360,366,371,447,467,473,539,544,550,668,681,706,712,717,744,749,836,841,855,861,895,918,924,971,977,1129,1135,1156,1162,1188],{"type":51,"tag":52,"props":53,"children":54},"element","h1",{"id":4},[55],{"type":56,"value":57},"text","Dynamo Interconnect Check",{"type":51,"tag":59,"props":60,"children":62},"h2",{"id":61},"purpose",[63],{"type":56,"value":64},"Purpose",{"type":51,"tag":66,"props":67,"children":68},"p",{},[69],{"type":56,"value":70},"Confirm that the transport disaggregated serving depends on actually works. A\ndeployment can pass an endpoint smoke test while disagg is silently wrong: if\nNIXL\u002FUCX cannot reach the peer worker over RDMA or NVLink, KV transfer falls\nback to a slow or broken path. Catch that with read-only checks before trusting\na disagg deployment or its benchmark numbers.",{"type":51,"tag":66,"props":72,"children":73},{},[74],{"type":56,"value":75},"This skill is read-only. It never mutates the cluster and never prints secrets.",{"type":51,"tag":59,"props":77,"children":79},{"id":78},"prerequisites",[80],{"type":56,"value":81},"Prerequisites",{"type":51,"tag":83,"props":84,"children":85},"ul",{},[86,92,104,117],{"type":51,"tag":87,"props":88,"children":89},"li",{},[90],{"type":56,"value":91},"Python 3.10+ on the operator machine.",{"type":51,"tag":87,"props":93,"children":94},{},[95,102],{"type":51,"tag":96,"props":97,"children":99},"code",{"className":98},[],[100],{"type":56,"value":101},"kubectl exec",{"type":56,"value":103}," access to a worker pod in the target Dynamo deployment.",{"type":51,"tag":87,"props":105,"children":106},{},[107,109,115],{"type":56,"value":108},"Read access to the recipe directory (",{"type":51,"tag":96,"props":110,"children":112},{"className":111},[],[113],{"type":56,"value":114},"recipes\u002F\u003Cmodel>\u002F\u003Cframework>\u002F\u003Cmode>",{"type":56,"value":116},").",{"type":51,"tag":87,"props":118,"children":119},{},[120,122,128,130,136,137,143,145,151],{"type":56,"value":121},"For node-capability checks: tools like ",{"type":51,"tag":96,"props":123,"children":125},{"className":124},[],[126],{"type":56,"value":127},"ibstat",{"type":56,"value":129},", ",{"type":51,"tag":96,"props":131,"children":133},{"className":132},[],[134],{"type":56,"value":135},"nvidia-smi",{"type":56,"value":129},{"type":51,"tag":96,"props":138,"children":140},{"className":139},[],[141],{"type":56,"value":142},"lsmod",{"type":56,"value":144}," available in the worker pod image (missing tools are reported as ",{"type":51,"tag":96,"props":146,"children":148},{"className":147},[],[149],{"type":56,"value":150},"skipped",{"type":56,"value":152},", not failures).",{"type":51,"tag":59,"props":154,"children":156},{"id":155},"when-to-use",[157],{"type":56,"value":158},"When To Use",{"type":51,"tag":83,"props":160,"children":161},{},[162,182,187],{"type":51,"tag":87,"props":163,"children":164},{},[165,167,173,175,180],{"type":56,"value":166},"After ",{"type":51,"tag":96,"props":168,"children":170},{"className":169},[],[171],{"type":56,"value":172},"dynamo-recipe-runner",{"type":56,"value":174}," deploys a ",{"type":51,"tag":176,"props":177,"children":178},"strong",{},[179],{"type":56,"value":45},{"type":56,"value":181}," or multi-node recipe.",{"type":51,"tag":87,"props":183,"children":184},{},[185],{"type":56,"value":186},"Before reporting disagg throughput\u002Flatency, so numbers reflect the real\ntransport.",{"type":51,"tag":87,"props":188,"children":189},{},[190],{"type":56,"value":191},"When agg works but disagg is slow, hangs, or returns wrong output and you\nsuspect the fabric rather than the model.",{"type":51,"tag":66,"props":193,"children":194},{},[195,197,203],{"type":56,"value":196},"For diagnosing pods that are already crashing or unschedulable, use\n",{"type":51,"tag":96,"props":198,"children":200},{"className":199},[],[201],{"type":56,"value":202},"dynamo-troubleshoot",{"type":56,"value":204}," first.",{"type":51,"tag":59,"props":206,"children":208},{"id":207},"instructions",[209],{"type":56,"value":210},"Instructions",{"type":51,"tag":212,"props":213,"children":215},"h3",{"id":214},"_1-check-transport-env-vars-on-the-recipe",[216],{"type":56,"value":217},"1. Check Transport Env Vars On The Recipe",{"type":51,"tag":219,"props":220,"children":225},"pre",{"className":221,"code":222,"language":223,"meta":224,"style":224},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","python3 scripts\u002Fcheck_interconnect.py env recipes\u002F\u003Cmodel>\u002F\u003Cframework>\u002F\u003Cmode>\n","bash","",[226],{"type":51,"tag":96,"props":227,"children":228},{"__ignoreMap":224},[229],{"type":51,"tag":230,"props":231,"children":234},"span",{"class":232,"line":233},"line",1,[235,241,247,252,257,263,268,274,279,284,288,293,298,302,306,310,315,320],{"type":51,"tag":230,"props":236,"children":238},{"style":237},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[239],{"type":56,"value":240},"python3",{"type":51,"tag":230,"props":242,"children":244},{"style":243},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[245],{"type":56,"value":246}," scripts\u002Fcheck_interconnect.py",{"type":51,"tag":230,"props":248,"children":249},{"style":243},[250],{"type":56,"value":251}," env",{"type":51,"tag":230,"props":253,"children":254},{"style":243},[255],{"type":56,"value":256}," recipes\u002F",{"type":51,"tag":230,"props":258,"children":260},{"style":259},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[261],{"type":56,"value":262},"\u003C",{"type":51,"tag":230,"props":264,"children":265},{"style":243},[266],{"type":56,"value":267},"mode",{"type":51,"tag":230,"props":269,"children":271},{"style":270},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[272],{"type":56,"value":273},"l",{"type":51,"tag":230,"props":275,"children":276},{"style":259},[277],{"type":56,"value":278},">",{"type":51,"tag":230,"props":280,"children":281},{"style":243},[282],{"type":56,"value":283},"\u002F",{"type":51,"tag":230,"props":285,"children":286},{"style":259},[287],{"type":56,"value":262},{"type":51,"tag":230,"props":289,"children":290},{"style":243},[291],{"type":56,"value":292},"framewor",{"type":51,"tag":230,"props":294,"children":295},{"style":270},[296],{"type":56,"value":297},"k",{"type":51,"tag":230,"props":299,"children":300},{"style":259},[301],{"type":56,"value":278},{"type":51,"tag":230,"props":303,"children":304},{"style":243},[305],{"type":56,"value":283},{"type":51,"tag":230,"props":307,"children":308},{"style":259},[309],{"type":56,"value":262},{"type":51,"tag":230,"props":311,"children":312},{"style":243},[313],{"type":56,"value":314},"mod",{"type":51,"tag":230,"props":316,"children":317},{"style":270},[318],{"type":56,"value":319},"e",{"type":51,"tag":230,"props":321,"children":322},{"style":259},[323],{"type":56,"value":324},">\n",{"type":51,"tag":66,"props":326,"children":327},{},[328,330,336,337,343,344,350,352,358],{"type":56,"value":329},"Reports which NIXL\u002FUCX\u002FNCCL transport variables are set and flags\ndisagg-critical ones (e.g. ",{"type":51,"tag":96,"props":331,"children":333},{"className":332},[],[334],{"type":56,"value":335},"UCX_TLS",{"type":56,"value":129},{"type":51,"tag":96,"props":338,"children":340},{"className":339},[],[341],{"type":56,"value":342},"UCX_NET_DEVICES",{"type":56,"value":129},{"type":51,"tag":96,"props":345,"children":347},{"className":346},[],[348],{"type":56,"value":349},"NCCL_IB_HCA",{"type":56,"value":351},") that are\nabsent. Missing here is only a warning — they may be baked into the image — so\nconfirm with the node and NIXL checks. See\n",{"type":51,"tag":96,"props":353,"children":355},{"className":354},[],[356],{"type":56,"value":357},"references\u002Finterconnect-env-vars.md",{"type":56,"value":359}," for what each variable does.",{"type":51,"tag":212,"props":361,"children":363},{"id":362},"_2-check-node-capabilities",[364],{"type":56,"value":365},"2. Check Node Capabilities",{"type":51,"tag":66,"props":367,"children":368},{},[369],{"type":56,"value":370},"Locally on a GPU node, or inside a running worker pod:",{"type":51,"tag":219,"props":372,"children":374},{"className":221,"code":373,"language":223,"meta":224,"style":224},"python3 scripts\u002Fcheck_interconnect.py node \\\n  --namespace \"${NAMESPACE}\" --pod \u003Cworker-pod>\n",[375],{"type":51,"tag":96,"props":376,"children":377},{"__ignoreMap":224},[378,399],{"type":51,"tag":230,"props":379,"children":380},{"class":232,"line":233},[381,385,389,394],{"type":51,"tag":230,"props":382,"children":383},{"style":237},[384],{"type":56,"value":240},{"type":51,"tag":230,"props":386,"children":387},{"style":243},[388],{"type":56,"value":246},{"type":51,"tag":230,"props":390,"children":391},{"style":243},[392],{"type":56,"value":393}," node",{"type":51,"tag":230,"props":395,"children":396},{"style":270},[397],{"type":56,"value":398}," \\\n",{"type":51,"tag":230,"props":400,"children":402},{"class":232,"line":401},2,[403,408,413,418,423,428,433,438,443],{"type":51,"tag":230,"props":404,"children":405},{"style":243},[406],{"type":56,"value":407},"  --namespace",{"type":51,"tag":230,"props":409,"children":410},{"style":259},[411],{"type":56,"value":412}," \"${",{"type":51,"tag":230,"props":414,"children":415},{"style":270},[416],{"type":56,"value":417},"NAMESPACE",{"type":51,"tag":230,"props":419,"children":420},{"style":259},[421],{"type":56,"value":422},"}\"",{"type":51,"tag":230,"props":424,"children":425},{"style":243},[426],{"type":56,"value":427}," --pod",{"type":51,"tag":230,"props":429,"children":430},{"style":259},[431],{"type":56,"value":432}," \u003C",{"type":51,"tag":230,"props":434,"children":435},{"style":243},[436],{"type":56,"value":437},"worker-po",{"type":51,"tag":230,"props":439,"children":440},{"style":270},[441],{"type":56,"value":442},"d",{"type":51,"tag":230,"props":444,"children":445},{"style":259},[446],{"type":56,"value":324},{"type":51,"tag":66,"props":448,"children":449},{},[450,452,458,460,465],{"type":56,"value":451},"Probes (read-only) for: InfiniBand devices and Active links, GPUDirect RDMA\n(",{"type":51,"tag":96,"props":453,"children":455},{"className":454},[],[456],{"type":56,"value":457},"nvidia_peermem",{"type":56,"value":459},"), GDRCopy, and NVLink in the GPU topology. Missing tools are\nreported as ",{"type":51,"tag":96,"props":461,"children":463},{"className":462},[],[464],{"type":56,"value":150},{"type":56,"value":466},", not failures.",{"type":51,"tag":212,"props":468,"children":470},{"id":469},"_3-validate-nixl-reachability",[471],{"type":56,"value":472},"3. Validate NIXL Reachability",{"type":51,"tag":219,"props":474,"children":476},{"className":221,"code":475,"language":223,"meta":224,"style":224},"python3 scripts\u002Fcheck_interconnect.py nixl \\\n  --namespace \"${NAMESPACE}\" --pod \u003Cworker-pod>\n",[477],{"type":51,"tag":96,"props":478,"children":479},{"__ignoreMap":224},[480,500],{"type":51,"tag":230,"props":481,"children":482},{"class":232,"line":233},[483,487,491,496],{"type":51,"tag":230,"props":484,"children":485},{"style":237},[486],{"type":56,"value":240},{"type":51,"tag":230,"props":488,"children":489},{"style":243},[490],{"type":56,"value":246},{"type":51,"tag":230,"props":492,"children":493},{"style":243},[494],{"type":56,"value":495}," nixl",{"type":51,"tag":230,"props":497,"children":498},{"style":270},[499],{"type":56,"value":398},{"type":51,"tag":230,"props":501,"children":502},{"class":232,"line":401},[503,507,511,515,519,523,527,531,535],{"type":51,"tag":230,"props":504,"children":505},{"style":243},[506],{"type":56,"value":407},{"type":51,"tag":230,"props":508,"children":509},{"style":259},[510],{"type":56,"value":412},{"type":51,"tag":230,"props":512,"children":513},{"style":270},[514],{"type":56,"value":417},{"type":51,"tag":230,"props":516,"children":517},{"style":259},[518],{"type":56,"value":422},{"type":51,"tag":230,"props":520,"children":521},{"style":243},[522],{"type":56,"value":427},{"type":51,"tag":230,"props":524,"children":525},{"style":259},[526],{"type":56,"value":432},{"type":51,"tag":230,"props":528,"children":529},{"style":243},[530],{"type":56,"value":437},{"type":51,"tag":230,"props":532,"children":533},{"style":270},[534],{"type":56,"value":442},{"type":51,"tag":230,"props":536,"children":537},{"style":259},[538],{"type":56,"value":324},{"type":51,"tag":66,"props":540,"children":541},{},[542],{"type":56,"value":543},"Looks for NIXL test tooling in the pod and surfaces the exact next step to run a\npairwise prefill↔decode transfer test. A full cross-pod transfer test requires\ntwo scheduled GPU pods on the fabric.",{"type":51,"tag":59,"props":545,"children":547},{"id":546},"available-scripts",[548],{"type":56,"value":549},"Available Scripts",{"type":51,"tag":551,"props":552,"children":553},"table",{},[554,577],{"type":51,"tag":555,"props":556,"children":557},"thead",{},[558],{"type":51,"tag":559,"props":560,"children":561},"tr",{},[562,568,572],{"type":51,"tag":563,"props":564,"children":565},"th",{},[566],{"type":56,"value":567},"Script",{"type":51,"tag":563,"props":569,"children":570},{},[571],{"type":56,"value":64},{"type":51,"tag":563,"props":573,"children":574},{},[575],{"type":56,"value":576},"Arguments",{"type":51,"tag":578,"props":579,"children":580},"tbody",{},[581,604,637],{"type":51,"tag":559,"props":582,"children":583},{},[584,594,599],{"type":51,"tag":585,"props":586,"children":587},"td",{},[588],{"type":51,"tag":96,"props":589,"children":591},{"className":590},[],[592],{"type":56,"value":593},"scripts\u002Fcheck_interconnect.py env",{"type":51,"tag":585,"props":595,"children":596},{},[597],{"type":56,"value":598},"Inspect NIXL\u002FUCX\u002FNCCL env vars on a recipe",{"type":51,"tag":585,"props":600,"children":601},{},[602],{"type":56,"value":603},"positional recipe path",{"type":51,"tag":559,"props":605,"children":606},{},[607,616,621],{"type":51,"tag":585,"props":608,"children":609},{},[610],{"type":51,"tag":96,"props":611,"children":613},{"className":612},[],[614],{"type":56,"value":615},"scripts\u002Fcheck_interconnect.py node",{"type":51,"tag":585,"props":617,"children":618},{},[619],{"type":56,"value":620},"Probe InfiniBand, GPUDirect RDMA, GDRCopy, NVLink on a node or pod",{"type":51,"tag":585,"props":622,"children":623},{},[624,630,631],{"type":51,"tag":96,"props":625,"children":627},{"className":626},[],[628],{"type":56,"value":629},"--namespace",{"type":56,"value":129},{"type":51,"tag":96,"props":632,"children":634},{"className":633},[],[635],{"type":56,"value":636},"--pod",{"type":51,"tag":559,"props":638,"children":639},{},[640,649,654],{"type":51,"tag":585,"props":641,"children":642},{},[643],{"type":51,"tag":96,"props":644,"children":646},{"className":645},[],[647],{"type":56,"value":648},"scripts\u002Fcheck_interconnect.py nixl",{"type":51,"tag":585,"props":650,"children":651},{},[652],{"type":56,"value":653},"Surface NIXL transfer-test readiness for a pod",{"type":51,"tag":585,"props":655,"children":656},{},[657,662,663],{"type":51,"tag":96,"props":658,"children":660},{"className":659},[],[661],{"type":56,"value":629},{"type":56,"value":129},{"type":51,"tag":96,"props":664,"children":666},{"className":665},[],[667],{"type":56,"value":636},{"type":51,"tag":66,"props":669,"children":670},{},[671,673,679],{"type":56,"value":672},"Invoke via the agentskills.io ",{"type":51,"tag":96,"props":674,"children":676},{"className":675},[],[677],{"type":56,"value":678},"run_script()",{"type":56,"value":680}," protocol:",{"type":51,"tag":219,"props":682,"children":686},{"className":683,"code":684,"language":685,"meta":224,"style":224},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","run_script(\"scripts\u002Fcheck_interconnect.py\", args=[\"env\", \"recipes\u002Fqwen3-coder-480b\u002Fsglang\u002Fdisagg\"])\nrun_script(\"scripts\u002Fcheck_interconnect.py\", args=[\"node\", \"--namespace\", \"dynamo-demo\", \"--pod\", \"qwen-worker-0\"])\n","python",[687],{"type":51,"tag":96,"props":688,"children":689},{"__ignoreMap":224},[690,698],{"type":51,"tag":230,"props":691,"children":692},{"class":232,"line":233},[693],{"type":51,"tag":230,"props":694,"children":695},{},[696],{"type":56,"value":697},"run_script(\"scripts\u002Fcheck_interconnect.py\", args=[\"env\", \"recipes\u002Fqwen3-coder-480b\u002Fsglang\u002Fdisagg\"])\n",{"type":51,"tag":230,"props":699,"children":700},{"class":232,"line":401},[701],{"type":51,"tag":230,"props":702,"children":703},{},[704],{"type":56,"value":705},"run_script(\"scripts\u002Fcheck_interconnect.py\", args=[\"node\", \"--namespace\", \"dynamo-demo\", \"--pod\", \"qwen-worker-0\"])\n",{"type":51,"tag":59,"props":707,"children":709},{"id":708},"examples",[710],{"type":56,"value":711},"Examples",{"type":51,"tag":66,"props":713,"children":714},{},[715],{"type":56,"value":716},"Verify a disagg recipe's transport env shape before deploy:",{"type":51,"tag":219,"props":718,"children":720},{"className":221,"code":719,"language":223,"meta":224,"style":224},"python3 scripts\u002Fcheck_interconnect.py env recipes\u002Fqwen3-coder-480b\u002Fsglang\u002Fdisagg\n",[721],{"type":51,"tag":96,"props":722,"children":723},{"__ignoreMap":224},[724],{"type":51,"tag":230,"props":725,"children":726},{"class":232,"line":233},[727,731,735,739],{"type":51,"tag":230,"props":728,"children":729},{"style":237},[730],{"type":56,"value":240},{"type":51,"tag":230,"props":732,"children":733},{"style":243},[734],{"type":56,"value":246},{"type":51,"tag":230,"props":736,"children":737},{"style":243},[738],{"type":56,"value":251},{"type":51,"tag":230,"props":740,"children":741},{"style":243},[742],{"type":56,"value":743}," recipes\u002Fqwen3-coder-480b\u002Fsglang\u002Fdisagg\n",{"type":51,"tag":66,"props":745,"children":746},{},[747],{"type":56,"value":748},"After deploy, validate a worker pod's fabric:",{"type":51,"tag":219,"props":750,"children":752},{"className":221,"code":751,"language":223,"meta":224,"style":224},"python3 scripts\u002Fcheck_interconnect.py node \\\n  --namespace dynamo-demo --pod qwen-worker-0\npython3 scripts\u002Fcheck_interconnect.py nixl \\\n  --namespace dynamo-demo --pod qwen-worker-0\n",[753],{"type":51,"tag":96,"props":754,"children":755},{"__ignoreMap":224},[756,775,796,816],{"type":51,"tag":230,"props":757,"children":758},{"class":232,"line":233},[759,763,767,771],{"type":51,"tag":230,"props":760,"children":761},{"style":237},[762],{"type":56,"value":240},{"type":51,"tag":230,"props":764,"children":765},{"style":243},[766],{"type":56,"value":246},{"type":51,"tag":230,"props":768,"children":769},{"style":243},[770],{"type":56,"value":393},{"type":51,"tag":230,"props":772,"children":773},{"style":270},[774],{"type":56,"value":398},{"type":51,"tag":230,"props":776,"children":777},{"class":232,"line":401},[778,782,787,791],{"type":51,"tag":230,"props":779,"children":780},{"style":243},[781],{"type":56,"value":407},{"type":51,"tag":230,"props":783,"children":784},{"style":243},[785],{"type":56,"value":786}," dynamo-demo",{"type":51,"tag":230,"props":788,"children":789},{"style":243},[790],{"type":56,"value":427},{"type":51,"tag":230,"props":792,"children":793},{"style":243},[794],{"type":56,"value":795}," qwen-worker-0\n",{"type":51,"tag":230,"props":797,"children":799},{"class":232,"line":798},3,[800,804,808,812],{"type":51,"tag":230,"props":801,"children":802},{"style":237},[803],{"type":56,"value":240},{"type":51,"tag":230,"props":805,"children":806},{"style":243},[807],{"type":56,"value":246},{"type":51,"tag":230,"props":809,"children":810},{"style":243},[811],{"type":56,"value":495},{"type":51,"tag":230,"props":813,"children":814},{"style":270},[815],{"type":56,"value":398},{"type":51,"tag":230,"props":817,"children":819},{"class":232,"line":818},4,[820,824,828,832],{"type":51,"tag":230,"props":821,"children":822},{"style":243},[823],{"type":56,"value":407},{"type":51,"tag":230,"props":825,"children":826},{"style":243},[827],{"type":56,"value":786},{"type":51,"tag":230,"props":829,"children":830},{"style":243},[831],{"type":56,"value":427},{"type":51,"tag":230,"props":833,"children":834},{"style":243},[835],{"type":56,"value":795},{"type":51,"tag":66,"props":837,"children":838},{},[839],{"type":56,"value":840},"Equivalent through the agent protocol:",{"type":51,"tag":219,"props":842,"children":844},{"className":683,"code":843,"language":685,"meta":224,"style":224},"run_script(\"scripts\u002Fcheck_interconnect.py\", args=[\"nixl\", \"--namespace\", \"dynamo-demo\", \"--pod\", \"qwen-worker-0\"])\n",[845],{"type":51,"tag":96,"props":846,"children":847},{"__ignoreMap":224},[848],{"type":51,"tag":230,"props":849,"children":850},{"class":232,"line":233},[851],{"type":51,"tag":230,"props":852,"children":853},{},[854],{"type":56,"value":843},{"type":51,"tag":59,"props":856,"children":858},{"id":857},"output-contract",[859],{"type":56,"value":860},"Output Contract",{"type":51,"tag":66,"props":862,"children":863},{},[864,866,872,874,880,881,887,888,893],{"type":56,"value":865},"Each check returns ",{"type":51,"tag":96,"props":867,"children":869},{"className":868},[],[870],{"type":56,"value":871},"ok",{"type":56,"value":873}," \u002F ",{"type":51,"tag":96,"props":875,"children":877},{"className":876},[],[878],{"type":56,"value":879},"warn",{"type":56,"value":873},{"type":51,"tag":96,"props":882,"children":884},{"className":883},[],[885],{"type":56,"value":886},"fail",{"type":56,"value":873},{"type":51,"tag":96,"props":889,"children":891},{"className":890},[],[892],{"type":56,"value":150},{"type":56,"value":894}," with a one-line detail,\nplus a rolled-up verdict on disagg transport readiness. Report:",{"type":51,"tag":83,"props":896,"children":897},{},[898,903,908,913],{"type":51,"tag":87,"props":899,"children":900},{},[901],{"type":56,"value":902},"transport env vars present vs. disagg-critical ones missing",{"type":51,"tag":87,"props":904,"children":905},{},[906],{"type":56,"value":907},"RDMA \u002F GPUDirect \u002F NVLink capability status",{"type":51,"tag":87,"props":909,"children":910},{},[911],{"type":56,"value":912},"whether NIXL reachability was validated, and the next command if not",{"type":51,"tag":87,"props":914,"children":915},{},[916],{"type":56,"value":917},"a clear statement of whether disagg can be trusted, or what to fix first",{"type":51,"tag":59,"props":919,"children":921},{"id":920},"limitations",[922],{"type":56,"value":923},"Limitations",{"type":51,"tag":83,"props":925,"children":926},{},[927,932,961,966],{"type":51,"tag":87,"props":928,"children":929},{},[930],{"type":56,"value":931},"Read-only fabric probe; does not run a full pairwise NIXL transfer (requires two scheduled GPU pods and the in-pod NIXL test tools).",{"type":51,"tag":87,"props":933,"children":934},{},[935,940,942,947,948,953,954,959],{"type":51,"tag":96,"props":936,"children":938},{"className":937},[],[939],{"type":56,"value":150},{"type":56,"value":941}," results for missing tools (",{"type":51,"tag":96,"props":943,"children":945},{"className":944},[],[946],{"type":56,"value":127},{"type":56,"value":129},{"type":51,"tag":96,"props":949,"children":951},{"className":950},[],[952],{"type":56,"value":135},{"type":56,"value":129},{"type":51,"tag":96,"props":955,"children":957},{"className":956},[],[958],{"type":56,"value":142},{"type":56,"value":960},") are inconclusive, not a pass.",{"type":51,"tag":87,"props":962,"children":963},{},[964],{"type":56,"value":965},"Env-var check inspects the recipe text; values injected at runtime via initContainers or operator-applied envs are not detected.",{"type":51,"tag":87,"props":967,"children":968},{},[969],{"type":56,"value":970},"Single-node agg deployments do not exercise the transport — this skill is for disagg \u002F multi-node validation.",{"type":51,"tag":59,"props":972,"children":974},{"id":973},"troubleshooting",[975],{"type":56,"value":976},"Troubleshooting",{"type":51,"tag":551,"props":978,"children":979},{},[980,1001],{"type":51,"tag":555,"props":981,"children":982},{},[983],{"type":51,"tag":559,"props":984,"children":985},{},[986,991,996],{"type":51,"tag":563,"props":987,"children":988},{},[989],{"type":56,"value":990},"Symptom",{"type":51,"tag":563,"props":992,"children":993},{},[994],{"type":56,"value":995},"Likely cause",{"type":51,"tag":563,"props":997,"children":998},{},[999],{"type":56,"value":1000},"Next step",{"type":51,"tag":578,"props":1002,"children":1003},{},[1004,1036,1075,1106],{"type":51,"tag":559,"props":1005,"children":1006},{},[1007,1018,1023],{"type":51,"tag":585,"props":1008,"children":1009},{},[1010,1016],{"type":51,"tag":96,"props":1011,"children":1013},{"className":1012},[],[1014],{"type":56,"value":1015},"env",{"type":56,"value":1017}," reports all critical vars missing",{"type":51,"tag":585,"props":1019,"children":1020},{},[1021],{"type":56,"value":1022},"Vars baked into image or injected by operator",{"type":51,"tag":585,"props":1024,"children":1025},{},[1026,1028,1034],{"type":56,"value":1027},"Run the ",{"type":51,"tag":96,"props":1029,"children":1031},{"className":1030},[],[1032],{"type":56,"value":1033},"node",{"type":56,"value":1035}," check inside the worker pod to verify actual env",{"type":51,"tag":559,"props":1037,"children":1038},{},[1039,1049,1054],{"type":51,"tag":585,"props":1040,"children":1041},{},[1042,1047],{"type":51,"tag":96,"props":1043,"children":1045},{"className":1044},[],[1046],{"type":56,"value":1033},{"type":56,"value":1048}," reports no Active IB link",{"type":51,"tag":585,"props":1050,"children":1051},{},[1052],{"type":56,"value":1053},"Fabric down or HCA not provisioned to the node",{"type":51,"tag":585,"props":1055,"children":1056},{},[1057,1059,1065,1067,1073],{"type":56,"value":1058},"Contact cluster admin; verify ",{"type":51,"tag":96,"props":1060,"children":1062},{"className":1061},[],[1063],{"type":56,"value":1064},"kubectl describe node",{"type":56,"value":1066}," shows ",{"type":51,"tag":96,"props":1068,"children":1070},{"className":1069},[],[1071],{"type":56,"value":1072},"nvidia.com\u002Fgpu",{"type":56,"value":1074}," and IB labels",{"type":51,"tag":559,"props":1076,"children":1077},{},[1078,1088,1093],{"type":51,"tag":585,"props":1079,"children":1080},{},[1081,1086],{"type":51,"tag":96,"props":1082,"children":1084},{"className":1083},[],[1085],{"type":56,"value":457},{"type":56,"value":1087}," missing",{"type":51,"tag":585,"props":1089,"children":1090},{},[1091],{"type":56,"value":1092},"GPUDirect RDMA module not loaded",{"type":51,"tag":585,"props":1094,"children":1095},{},[1096,1098,1104],{"type":56,"value":1097},"Ask cluster admin to load ",{"type":51,"tag":96,"props":1099,"children":1101},{"className":1100},[],[1102],{"type":56,"value":1103},"nvidia-peermem",{"type":56,"value":1105},"; without it, NIXL falls back to staged copies",{"type":51,"tag":559,"props":1107,"children":1108},{},[1109,1119,1124],{"type":51,"tag":585,"props":1110,"children":1111},{},[1112,1117],{"type":51,"tag":96,"props":1113,"children":1115},{"className":1114},[],[1116],{"type":56,"value":43},{"type":56,"value":1118}," finds no test tools",{"type":51,"tag":585,"props":1120,"children":1121},{},[1122],{"type":56,"value":1123},"Worker image lacks NIXL test harness",{"type":51,"tag":585,"props":1125,"children":1126},{},[1127],{"type":56,"value":1128},"Use a NIXL-enabled image or run the standalone transfer test from a debug pod",{"type":51,"tag":59,"props":1130,"children":1132},{"id":1131},"benchmark",[1133],{"type":56,"value":1134},"Benchmark",{"type":51,"tag":66,"props":1136,"children":1137},{},[1138,1140,1146,1148,1154],{"type":56,"value":1139},"See ",{"type":51,"tag":96,"props":1141,"children":1143},{"className":1142},[],[1144],{"type":56,"value":1145},"BENCHMARK.md",{"type":56,"value":1147}," for the NVCARPS-EVAL performance report (auto-generated by the NVSkills CI pipeline). To refresh, re-run ",{"type":51,"tag":96,"props":1149,"children":1151},{"className":1150},[],[1152],{"type":56,"value":1153},"\u002Fnvskills-ci",{"type":56,"value":1155}," on an upstream PR touching this skill.",{"type":51,"tag":59,"props":1157,"children":1159},{"id":1158},"references",[1160],{"type":56,"value":1161},"References",{"type":51,"tag":83,"props":1163,"children":1164},{},[1165,1175],{"type":51,"tag":87,"props":1166,"children":1167},{},[1168,1173],{"type":51,"tag":96,"props":1169,"children":1171},{"className":1170},[],[1172],{"type":56,"value":357},{"type":56,"value":1174}," — NIXL\u002FUCX\u002FNCCL env var catalog and IB\ncapability checklist.",{"type":51,"tag":87,"props":1176,"children":1177},{},[1178,1180,1186],{"type":56,"value":1179},"Use ",{"type":51,"tag":96,"props":1181,"children":1183},{"className":1182},[],[1184],{"type":56,"value":1185},"scripts\u002Fcheck_interconnect.py",{"type":56,"value":1187}," for all read-only checks.",{"type":51,"tag":1189,"props":1190,"children":1191},"style",{},[1192],{"type":56,"value":1193},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1195,"total":1293},[1196,1211,1221,1235,1247,1264,1279],{"slug":1197,"name":1197,"fn":1198,"description":1199,"org":1200,"tags":1201,"stars":26,"repoUrl":27,"updatedAt":1210},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1202,1205,1208,1209],{"name":1203,"slug":1204,"type":15},"Data Analysis","data-analysis",{"name":1206,"slug":1207,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:28:43.176466",{"slug":1212,"name":1212,"fn":1213,"description":1214,"org":1215,"tags":1216,"stars":26,"repoUrl":27,"updatedAt":1220},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1217,1218,1219],{"name":20,"slug":21,"type":15},{"name":24,"slug":25,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":1222,"name":1222,"fn":1223,"description":1224,"org":1225,"tags":1226,"stars":26,"repoUrl":27,"updatedAt":1234},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1227,1230,1231],{"name":1228,"slug":1229,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":1232,"slug":1233,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":1236,"name":1236,"fn":1237,"description":1238,"org":1239,"tags":1240,"stars":26,"repoUrl":27,"updatedAt":1246},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1241,1242,1243],{"name":1203,"slug":1204,"type":15},{"name":9,"slug":8,"type":15},{"name":1244,"slug":1245,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":1248,"name":1248,"fn":1249,"description":1250,"org":1251,"tags":1252,"stars":26,"repoUrl":27,"updatedAt":1263},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1253,1256,1259,1260],{"name":1254,"slug":1255,"type":15},"Automation","automation",{"name":1257,"slug":1258,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":1261,"slug":1262,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":1265,"name":1265,"fn":1266,"description":1267,"org":1268,"tags":1269,"stars":26,"repoUrl":27,"updatedAt":1278},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1270,1271,1274,1275],{"name":20,"slug":21,"type":15},{"name":1272,"slug":1273,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":1276,"slug":1277,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":1280,"name":1280,"fn":1281,"description":1282,"org":1283,"tags":1284,"stars":26,"repoUrl":27,"updatedAt":1292},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1285,1286,1289],{"name":9,"slug":8,"type":15},{"name":1287,"slug":1288,"type":15},"Quantum Computing","quantum-computing",{"name":1290,"slug":1291,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":1295,"total":1445},[1296,1314,1329,1340,1352,1366,1379,1393,1404,1413,1427,1436],{"slug":1297,"name":1297,"fn":1298,"description":1299,"org":1300,"tags":1301,"stars":1311,"repoUrl":1312,"updatedAt":1313},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1302,1305,1308],{"name":1303,"slug":1304,"type":15},"Documentation","documentation",{"name":1306,"slug":1307,"type":15},"MCP","mcp",{"name":1309,"slug":1310,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":1315,"name":1315,"fn":1316,"description":1317,"org":1318,"tags":1319,"stars":1326,"repoUrl":1327,"updatedAt":1328},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1320,1323,1324],{"name":1321,"slug":1322,"type":15},"Containers","containers",{"name":20,"slug":21,"type":15},{"name":1325,"slug":685,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":1330,"name":1330,"fn":1331,"description":1332,"org":1333,"tags":1334,"stars":1326,"repoUrl":1327,"updatedAt":1339},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1335,1338],{"name":1336,"slug":1337,"type":15},"CI\u002FCD","ci-cd",{"name":20,"slug":21,"type":15},"2026-07-14T05:25:59.97109",{"slug":1341,"name":1341,"fn":1342,"description":1343,"org":1344,"tags":1345,"stars":1326,"repoUrl":1327,"updatedAt":1351},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1346,1347,1348],{"name":1336,"slug":1337,"type":15},{"name":20,"slug":21,"type":15},{"name":1349,"slug":1350,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":1353,"name":1353,"fn":1354,"description":1355,"org":1356,"tags":1357,"stars":1326,"repoUrl":1327,"updatedAt":1365},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1358,1361,1362],{"name":1359,"slug":1360,"type":15},"Debugging","debugging",{"name":1349,"slug":1350,"type":15},{"name":1363,"slug":1364,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1367,"name":1367,"fn":1368,"description":1369,"org":1370,"tags":1371,"stars":1326,"repoUrl":1327,"updatedAt":1378},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1372,1375],{"name":1373,"slug":1374,"type":15},"Best Practices","best-practices",{"name":1376,"slug":1377,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1380,"name":1380,"fn":1381,"description":1382,"org":1383,"tags":1384,"stars":1326,"repoUrl":1327,"updatedAt":1392},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1385,1388,1391],{"name":1386,"slug":1387,"type":15},"Machine Learning","machine-learning",{"name":1389,"slug":1390,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":1394,"name":1394,"fn":1395,"description":1396,"org":1397,"tags":1398,"stars":1326,"repoUrl":1327,"updatedAt":1403},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1399,1402],{"name":1400,"slug":1401,"type":15},"QA","qa",{"name":1244,"slug":1245,"type":15},"2026-07-14T05:25:53.673039",{"slug":1405,"name":1405,"fn":1406,"description":1407,"org":1408,"tags":1409,"stars":1326,"repoUrl":1327,"updatedAt":1412},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1410,1411],{"name":20,"slug":21,"type":15},{"name":24,"slug":25,"type":15},"2026-07-14T05:25:49.362534",{"slug":1414,"name":1414,"fn":1415,"description":1416,"org":1417,"tags":1418,"stars":1326,"repoUrl":1327,"updatedAt":1426},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1419,1422,1423],{"name":1420,"slug":1421,"type":15},"Code Review","code-review",{"name":1349,"slug":1350,"type":15},{"name":1424,"slug":1425,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":1428,"name":1428,"fn":1429,"description":1430,"org":1431,"tags":1432,"stars":1326,"repoUrl":1327,"updatedAt":1435},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1433,1434],{"name":1400,"slug":1401,"type":15},{"name":1244,"slug":1245,"type":15},"2026-07-14T05:25:54.928983",{"slug":1437,"name":1437,"fn":1438,"description":1439,"org":1440,"tags":1441,"stars":1326,"repoUrl":1327,"updatedAt":1444},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1442,1443],{"name":1254,"slug":1255,"type":15},{"name":1336,"slug":1337,"type":15},"2026-07-30T05:29:03.275638",496]