[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-k8s-launch-kit-troubleshoot":3,"mdc--yqem98-key":37,"related-repo-nvidia-k8s-launch-kit-troubleshoot":1266,"related-org-nvidia-k8s-launch-kit-troubleshoot":1361},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":32,"sourceUrl":35,"mdContent":36},"k8s-launch-kit-troubleshoot","troubleshoot NVIDIA Network Operator deployments","Use this skill when the user has problems with NVIDIA Network Operator on Kubernetes, or wants to analyze a sosreport diagnostic dump. Activate for: OFED driver crashes, SR-IOV pods failing, NicClusterPolicy errors, network operator pod issues, RDMA not working, NIC configuration failures, pods stuck in CrashLoopBackOff or ContainerCreating with network annotations, VF allocation issues, or when the user mentions 'troubleshoot', 'debug', 'sosreport', 'diagnose', or describes any NVIDIA networking failure -- even if they don't explicitly ask for troubleshooting.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20,23],{"name":13,"slug":14,"type":15},"Operations","operations","tag",{"name":17,"slug":18,"type":15},"Networking","networking",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Kubernetes","kubernetes",{"name":24,"slug":25,"type":15},"Debugging","debugging",14,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fk8s-launch-kit","2026-07-14T05:31:52.594088",null,4,[],{"repoUrl":27,"stars":26,"forks":30,"topics":33,"description":34},[],"K8s Launch Kit (l8k) is a CLI tool for deploying and managing NVIDIA cloud-native solutions on Kubernetes. The tool helps provide flexible deployment workflows for optimal network performance with SR-IOV, RDMA, and other networking technologies.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fk8s-launch-kit\u002Ftree\u002FHEAD\u002Fskills\u002Fk8s-launch-kit-troubleshoot","---\nname: k8s-launch-kit-troubleshoot\nversion: 1.1.0\ndescription: \"Use this skill when the user has problems with NVIDIA Network Operator on Kubernetes, or wants to analyze a sosreport diagnostic dump. Activate for: OFED driver crashes, SR-IOV pods failing, NicClusterPolicy errors, network operator pod issues, RDMA not working, NIC configuration failures, pods stuck in CrashLoopBackOff or ContainerCreating with network annotations, VF allocation issues, or when the user mentions 'troubleshoot', 'debug', 'sosreport', 'diagnose', or describes any NVIDIA networking failure -- even if they don't explicitly ask for troubleshooting.\"\nmetadata:\n  requires:\n    skills: [\"k8s-launch-kit-shared\"]\n---\n\n# l8k: Troubleshooting\n\n> **PREREQUISITE:** Read `..\u002Fk8s-launch-kit-shared\u002FSKILL.md` for install paths, global flags, and exit codes.\n\nDebug NVIDIA Network Operator issues on Kubernetes, with or without a sosreport.\n\n## l8k Troubleshooting Commands\n\n```bash\n# Collect a diagnostic dump from the cluster\nl8k sosreport [--kubeconfig \u003CPATH>] --output-dir .\u002Fsosreport\n```\n\n`l8k sosreport` gathers cluster state, CRDs, operator logs, and per-node NIC info into a structured directory for offline analysis. Use the diagnostic commands and triage workflow below to interpret the dump.\n\n## Diagnostic Commands\n\n```bash\n# NicClusterPolicy status (check state: ready vs notReady)\nkubectl get nicclusterpolicy -o yaml\n\n# Network operator pods\nkubectl get pods -n \u003Coperator-ns> -o wide\n\n# SR-IOV node states (VF allocation)\nkubectl get sriovnetworknodestates -A -o yaml\n\n# OFED driver pod logs\nkubectl logs -n \u003Coperator-ns> -l app=mofed-\u003Cos> --tail=100\n\n# NIC configuration daemon logs\nkubectl logs -n \u003Coperator-ns> -l app=nic-configuration-daemon --tail=100\n\n# Check for pods stuck on network resources\nkubectl get pods -A -o wide | grep -E 'ContainerCreating|Init'\n```\n\n## Common Failure Patterns\n\n| Symptom | Likely Cause | Fix |\n|---------|-------------|-----|\n| NicClusterPolicy `state: notReady` | OFED driver pods failing | Check mofed pod logs, verify kernel\u002Fdriver compatibility |\n| Pods stuck in `ContainerCreating` | VFs not allocated or SR-IOV policy not applied | Check `sriovnetworknodestates`, verify device plugin pods |\n| `CrashLoopBackOff` on mofed pods | Kernel module conflict | Check `thirdPartyRDMAModules`, enable `unloadThirdPartyRDMAModules` |\n| No VFs on node | SriovNetworkNodePolicy not matching | Verify `nodeSelector` labels match worker nodes |\n| RDMA not working | Missing RDMA device plugin or wrong resource name | Check `rdma-shared-dp` pods, verify resource annotations |\n| `l8k discover` daemon pods stuck (ImagePullBackOff \u002F Pending) | Bad image tag, missing pull secret, or no `feature.node.kubernetes.io\u002Fpci-15b3.present=true` nodes | Re-run with `--keep-namespace` then `kubectl describe pod -n nvidia-k8s-launch-kit`. Fix `networkOperator.componentVersion` \u002F pass `--image-pull-secrets` \u002F verify NFD is running. |\n| `l8k validate` \u002F `deploy` can't find Network Operator pods | Operator namespace mismatch | Verify `--network-operator-namespace` matches actual namespace (does NOT apply to `l8k discover` — it ignores the flag and uses its own `nvidia-k8s-launch-kit` namespace) |\n| IPPool not allocating | NV-IPAM subnet exhausted or misconfigured | Check `ippools` CR status, verify CIDR ranges |\n| `--for requires --node-selector` | `--for` was passed without `--node-selector` | Add `--node-selector key=val,…`. The synthesized clusterConfig has no live worker-node list; the selector identifies target nodes at apply time. |\n| `--for and --discover-cluster-config are mutually exclusive` | Both flags passed simultaneously | Pick one: `--for` skips discovery, `--discover-cluster-config` runs it. |\n| `unknown preset \"X\"; available: …` | `--for X` doesn't match any directory under `presets\u002F` | Run `l8k preset list` and re-run with one of those names. |\n| `preset has no capabilities block` | Preset YAML used by `--for` is missing `capabilities.nodes.{sriov,rdma,ib}` | Add the block to the preset's `topology.yaml`. Discovery-time overlay does not require it; only `--for` does. |\n| `unknown field \"productType\"` in YAML | Hand-authored config still uses the old key name | Rename `productType:` to `gpuType:` (the field was renamed). |\n\nFor detailed triage workflow, read `references\u002Ftroubleshooting-guide.md`.\n\n## sosreport Analysis\n\nIf the user has a pre-collected sosreport directory (from `l8k sosreport` or manual collection):\n\n```\nsosreport\u002F\n├── metadata\u002F          # Cluster info, node list\n├── crds\u002F              # NicClusterPolicy, SriovNetworkNodePolicy, IPPool, etc.\n├── operator\u002F          # Network operator pod logs\n├── nodes\u002F             # Per-node device info\n└── network\u002F           # Interface config, routing tables\n```\n\n### Triage Checklist\n\n1. Read `metadata\u002Fdiagnostic-summary.yaml` for overview\n2. Check pod health in `operator\u002Fpods.yaml`\n3. Inspect CRDs in `crds\u002F` for status fields\n4. Read operator logs in `operator\u002Flogs\u002F` for errors\n5. Check per-node NIC state in `nodes\u002F\u003Cnode>\u002F`\n\n## See Also\n\n- [k8s-launch-kit-shared](..\u002Fk8s-launch-kit-shared\u002FSKILL.md) — Exit codes and error structure\n- [k8s-launch-kit-discover](..\u002Fk8s-launch-kit-discover\u002FSKILL.md) — Re-discover to verify hardware state\n- `references\u002Ftroubleshooting-guide.md` — Detailed triage workflow\n",{"data":38,"body":44},{"name":4,"version":39,"description":6,"metadata":40},"1.1.0",{"requires":41},{"skills":42},[43],"k8s-launch-kit-shared",{"type":45,"children":46},"root",[47,56,81,86,93,174,185,191,568,574,1105,1118,1124,1136,1146,1153,1219,1225,1260],{"type":48,"tag":49,"props":50,"children":52},"element","h1",{"id":51},"l8k-troubleshooting",[53],{"type":54,"value":55},"text","l8k: Troubleshooting",{"type":48,"tag":57,"props":58,"children":59},"blockquote",{},[60],{"type":48,"tag":61,"props":62,"children":63},"p",{},[64,70,72,79],{"type":48,"tag":65,"props":66,"children":67},"strong",{},[68],{"type":54,"value":69},"PREREQUISITE:",{"type":54,"value":71}," Read ",{"type":48,"tag":73,"props":74,"children":76},"code",{"className":75},[],[77],{"type":54,"value":78},"..\u002Fk8s-launch-kit-shared\u002FSKILL.md",{"type":54,"value":80}," for install paths, global flags, and exit codes.",{"type":48,"tag":61,"props":82,"children":83},{},[84],{"type":54,"value":85},"Debug NVIDIA Network Operator issues on Kubernetes, with or without a sosreport.",{"type":48,"tag":87,"props":88,"children":90},"h2",{"id":89},"l8k-troubleshooting-commands",[91],{"type":54,"value":92},"l8k Troubleshooting Commands",{"type":48,"tag":94,"props":95,"children":100},"pre",{"className":96,"code":97,"language":98,"meta":99,"style":99},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Collect a diagnostic dump from the cluster\nl8k sosreport [--kubeconfig \u003CPATH>] --output-dir .\u002Fsosreport\n","bash","",[101],{"type":48,"tag":73,"props":102,"children":103},{"__ignoreMap":99},[104,116],{"type":48,"tag":105,"props":106,"children":109},"span",{"class":107,"line":108},"line",1,[110],{"type":48,"tag":105,"props":111,"children":113},{"style":112},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[114],{"type":54,"value":115},"# Collect a diagnostic dump from the cluster\n",{"type":48,"tag":105,"props":117,"children":119},{"class":107,"line":118},2,[120,126,132,138,144,149,154,159,164,169],{"type":48,"tag":105,"props":121,"children":123},{"style":122},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[124],{"type":54,"value":125},"l8k",{"type":48,"tag":105,"props":127,"children":129},{"style":128},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[130],{"type":54,"value":131}," sosreport",{"type":48,"tag":105,"props":133,"children":135},{"style":134},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[136],{"type":54,"value":137}," [--kubeconfig ",{"type":48,"tag":105,"props":139,"children":141},{"style":140},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[142],{"type":54,"value":143},"\u003C",{"type":48,"tag":105,"props":145,"children":146},{"style":128},[147],{"type":54,"value":148},"PAT",{"type":48,"tag":105,"props":150,"children":151},{"style":134},[152],{"type":54,"value":153},"H",{"type":48,"tag":105,"props":155,"children":156},{"style":140},[157],{"type":54,"value":158},">",{"type":48,"tag":105,"props":160,"children":161},{"style":128},[162],{"type":54,"value":163},"]",{"type":48,"tag":105,"props":165,"children":166},{"style":128},[167],{"type":54,"value":168}," --output-dir",{"type":48,"tag":105,"props":170,"children":171},{"style":128},[172],{"type":54,"value":173}," .\u002Fsosreport\n",{"type":48,"tag":61,"props":175,"children":176},{},[177,183],{"type":48,"tag":73,"props":178,"children":180},{"className":179},[],[181],{"type":54,"value":182},"l8k sosreport",{"type":54,"value":184}," gathers cluster state, CRDs, operator logs, and per-node NIC info into a structured directory for offline analysis. Use the diagnostic commands and triage workflow below to interpret the dump.",{"type":48,"tag":87,"props":186,"children":188},{"id":187},"diagnostic-commands",[189],{"type":54,"value":190},"Diagnostic Commands",{"type":48,"tag":94,"props":192,"children":194},{"className":96,"code":193,"language":98,"meta":99,"style":99},"# NicClusterPolicy status (check state: ready vs notReady)\nkubectl get nicclusterpolicy -o yaml\n\n# Network operator pods\nkubectl get pods -n \u003Coperator-ns> -o wide\n\n# SR-IOV node states (VF allocation)\nkubectl get sriovnetworknodestates -A -o yaml\n\n# OFED driver pod logs\nkubectl logs -n \u003Coperator-ns> -l app=mofed-\u003Cos> --tail=100\n\n# NIC configuration daemon logs\nkubectl logs -n \u003Coperator-ns> -l app=nic-configuration-daemon --tail=100\n\n# Check for pods stuck on network resources\nkubectl get pods -A -o wide | grep -E 'ContainerCreating|Init'\n",[195],{"type":48,"tag":73,"props":196,"children":197},{"__ignoreMap":99},[198,206,234,244,252,302,310,319,349,357,366,431,439,448,492,500,509],{"type":48,"tag":105,"props":199,"children":200},{"class":107,"line":108},[201],{"type":48,"tag":105,"props":202,"children":203},{"style":112},[204],{"type":54,"value":205},"# NicClusterPolicy status (check state: ready vs notReady)\n",{"type":48,"tag":105,"props":207,"children":208},{"class":107,"line":118},[209,214,219,224,229],{"type":48,"tag":105,"props":210,"children":211},{"style":122},[212],{"type":54,"value":213},"kubectl",{"type":48,"tag":105,"props":215,"children":216},{"style":128},[217],{"type":54,"value":218}," get",{"type":48,"tag":105,"props":220,"children":221},{"style":128},[222],{"type":54,"value":223}," nicclusterpolicy",{"type":48,"tag":105,"props":225,"children":226},{"style":128},[227],{"type":54,"value":228}," -o",{"type":48,"tag":105,"props":230,"children":231},{"style":128},[232],{"type":54,"value":233}," yaml\n",{"type":48,"tag":105,"props":235,"children":237},{"class":107,"line":236},3,[238],{"type":48,"tag":105,"props":239,"children":241},{"emptyLinePlaceholder":240},true,[242],{"type":54,"value":243},"\n",{"type":48,"tag":105,"props":245,"children":246},{"class":107,"line":30},[247],{"type":48,"tag":105,"props":248,"children":249},{"style":112},[250],{"type":54,"value":251},"# Network operator pods\n",{"type":48,"tag":105,"props":253,"children":255},{"class":107,"line":254},5,[256,260,264,269,274,279,284,289,293,297],{"type":48,"tag":105,"props":257,"children":258},{"style":122},[259],{"type":54,"value":213},{"type":48,"tag":105,"props":261,"children":262},{"style":128},[263],{"type":54,"value":218},{"type":48,"tag":105,"props":265,"children":266},{"style":128},[267],{"type":54,"value":268}," pods",{"type":48,"tag":105,"props":270,"children":271},{"style":128},[272],{"type":54,"value":273}," -n",{"type":48,"tag":105,"props":275,"children":276},{"style":140},[277],{"type":54,"value":278}," \u003C",{"type":48,"tag":105,"props":280,"children":281},{"style":128},[282],{"type":54,"value":283},"operator-n",{"type":48,"tag":105,"props":285,"children":286},{"style":134},[287],{"type":54,"value":288},"s",{"type":48,"tag":105,"props":290,"children":291},{"style":140},[292],{"type":54,"value":158},{"type":48,"tag":105,"props":294,"children":295},{"style":128},[296],{"type":54,"value":228},{"type":48,"tag":105,"props":298,"children":299},{"style":128},[300],{"type":54,"value":301}," wide\n",{"type":48,"tag":105,"props":303,"children":305},{"class":107,"line":304},6,[306],{"type":48,"tag":105,"props":307,"children":308},{"emptyLinePlaceholder":240},[309],{"type":54,"value":243},{"type":48,"tag":105,"props":311,"children":313},{"class":107,"line":312},7,[314],{"type":48,"tag":105,"props":315,"children":316},{"style":112},[317],{"type":54,"value":318},"# SR-IOV node states (VF allocation)\n",{"type":48,"tag":105,"props":320,"children":322},{"class":107,"line":321},8,[323,327,331,336,341,345],{"type":48,"tag":105,"props":324,"children":325},{"style":122},[326],{"type":54,"value":213},{"type":48,"tag":105,"props":328,"children":329},{"style":128},[330],{"type":54,"value":218},{"type":48,"tag":105,"props":332,"children":333},{"style":128},[334],{"type":54,"value":335}," sriovnetworknodestates",{"type":48,"tag":105,"props":337,"children":338},{"style":128},[339],{"type":54,"value":340}," -A",{"type":48,"tag":105,"props":342,"children":343},{"style":128},[344],{"type":54,"value":228},{"type":48,"tag":105,"props":346,"children":347},{"style":128},[348],{"type":54,"value":233},{"type":48,"tag":105,"props":350,"children":352},{"class":107,"line":351},9,[353],{"type":48,"tag":105,"props":354,"children":355},{"emptyLinePlaceholder":240},[356],{"type":54,"value":243},{"type":48,"tag":105,"props":358,"children":360},{"class":107,"line":359},10,[361],{"type":48,"tag":105,"props":362,"children":363},{"style":112},[364],{"type":54,"value":365},"# OFED driver pod logs\n",{"type":48,"tag":105,"props":367,"children":369},{"class":107,"line":368},11,[370,374,379,383,387,391,395,399,404,409,413,418,422,426],{"type":48,"tag":105,"props":371,"children":372},{"style":122},[373],{"type":54,"value":213},{"type":48,"tag":105,"props":375,"children":376},{"style":128},[377],{"type":54,"value":378}," logs",{"type":48,"tag":105,"props":380,"children":381},{"style":128},[382],{"type":54,"value":273},{"type":48,"tag":105,"props":384,"children":385},{"style":140},[386],{"type":54,"value":278},{"type":48,"tag":105,"props":388,"children":389},{"style":128},[390],{"type":54,"value":283},{"type":48,"tag":105,"props":392,"children":393},{"style":134},[394],{"type":54,"value":288},{"type":48,"tag":105,"props":396,"children":397},{"style":140},[398],{"type":54,"value":158},{"type":48,"tag":105,"props":400,"children":401},{"style":128},[402],{"type":54,"value":403}," -l",{"type":48,"tag":105,"props":405,"children":406},{"style":128},[407],{"type":54,"value":408}," app=mofed-",{"type":48,"tag":105,"props":410,"children":411},{"style":140},[412],{"type":54,"value":143},{"type":48,"tag":105,"props":414,"children":415},{"style":128},[416],{"type":54,"value":417},"o",{"type":48,"tag":105,"props":419,"children":420},{"style":134},[421],{"type":54,"value":288},{"type":48,"tag":105,"props":423,"children":424},{"style":140},[425],{"type":54,"value":158},{"type":48,"tag":105,"props":427,"children":428},{"style":128},[429],{"type":54,"value":430}," --tail=100\n",{"type":48,"tag":105,"props":432,"children":434},{"class":107,"line":433},12,[435],{"type":48,"tag":105,"props":436,"children":437},{"emptyLinePlaceholder":240},[438],{"type":54,"value":243},{"type":48,"tag":105,"props":440,"children":442},{"class":107,"line":441},13,[443],{"type":48,"tag":105,"props":444,"children":445},{"style":112},[446],{"type":54,"value":447},"# NIC configuration daemon logs\n",{"type":48,"tag":105,"props":449,"children":450},{"class":107,"line":26},[451,455,459,463,467,471,475,479,483,488],{"type":48,"tag":105,"props":452,"children":453},{"style":122},[454],{"type":54,"value":213},{"type":48,"tag":105,"props":456,"children":457},{"style":128},[458],{"type":54,"value":378},{"type":48,"tag":105,"props":460,"children":461},{"style":128},[462],{"type":54,"value":273},{"type":48,"tag":105,"props":464,"children":465},{"style":140},[466],{"type":54,"value":278},{"type":48,"tag":105,"props":468,"children":469},{"style":128},[470],{"type":54,"value":283},{"type":48,"tag":105,"props":472,"children":473},{"style":134},[474],{"type":54,"value":288},{"type":48,"tag":105,"props":476,"children":477},{"style":140},[478],{"type":54,"value":158},{"type":48,"tag":105,"props":480,"children":481},{"style":128},[482],{"type":54,"value":403},{"type":48,"tag":105,"props":484,"children":485},{"style":128},[486],{"type":54,"value":487}," app=nic-configuration-daemon",{"type":48,"tag":105,"props":489,"children":490},{"style":128},[491],{"type":54,"value":430},{"type":48,"tag":105,"props":493,"children":495},{"class":107,"line":494},15,[496],{"type":48,"tag":105,"props":497,"children":498},{"emptyLinePlaceholder":240},[499],{"type":54,"value":243},{"type":48,"tag":105,"props":501,"children":503},{"class":107,"line":502},16,[504],{"type":48,"tag":105,"props":505,"children":506},{"style":112},[507],{"type":54,"value":508},"# Check for pods stuck on network resources\n",{"type":48,"tag":105,"props":510,"children":512},{"class":107,"line":511},17,[513,517,521,525,529,533,538,543,548,553,558,563],{"type":48,"tag":105,"props":514,"children":515},{"style":122},[516],{"type":54,"value":213},{"type":48,"tag":105,"props":518,"children":519},{"style":128},[520],{"type":54,"value":218},{"type":48,"tag":105,"props":522,"children":523},{"style":128},[524],{"type":54,"value":268},{"type":48,"tag":105,"props":526,"children":527},{"style":128},[528],{"type":54,"value":340},{"type":48,"tag":105,"props":530,"children":531},{"style":128},[532],{"type":54,"value":228},{"type":48,"tag":105,"props":534,"children":535},{"style":128},[536],{"type":54,"value":537}," wide",{"type":48,"tag":105,"props":539,"children":540},{"style":140},[541],{"type":54,"value":542}," |",{"type":48,"tag":105,"props":544,"children":545},{"style":122},[546],{"type":54,"value":547}," grep",{"type":48,"tag":105,"props":549,"children":550},{"style":128},[551],{"type":54,"value":552}," -E",{"type":48,"tag":105,"props":554,"children":555},{"style":140},[556],{"type":54,"value":557}," '",{"type":48,"tag":105,"props":559,"children":560},{"style":128},[561],{"type":54,"value":562},"ContainerCreating|Init",{"type":48,"tag":105,"props":564,"children":565},{"style":140},[566],{"type":54,"value":567},"'\n",{"type":48,"tag":87,"props":569,"children":571},{"id":570},"common-failure-patterns",[572],{"type":54,"value":573},"Common Failure Patterns",{"type":48,"tag":575,"props":576,"children":577},"table",{},[578,602],{"type":48,"tag":579,"props":580,"children":581},"thead",{},[582],{"type":48,"tag":583,"props":584,"children":585},"tr",{},[586,592,597],{"type":48,"tag":587,"props":588,"children":589},"th",{},[590],{"type":54,"value":591},"Symptom",{"type":48,"tag":587,"props":593,"children":594},{},[595],{"type":54,"value":596},"Likely Cause",{"type":48,"tag":587,"props":598,"children":599},{},[600],{"type":54,"value":601},"Fix",{"type":48,"tag":603,"props":604,"children":605},"tbody",{},[606,631,663,700,726,751,815,869,894,936,973,1015,1065],{"type":48,"tag":583,"props":607,"children":608},{},[609,621,626],{"type":48,"tag":610,"props":611,"children":612},"td",{},[613,615],{"type":54,"value":614},"NicClusterPolicy ",{"type":48,"tag":73,"props":616,"children":618},{"className":617},[],[619],{"type":54,"value":620},"state: notReady",{"type":48,"tag":610,"props":622,"children":623},{},[624],{"type":54,"value":625},"OFED driver pods failing",{"type":48,"tag":610,"props":627,"children":628},{},[629],{"type":54,"value":630},"Check mofed pod logs, verify kernel\u002Fdriver compatibility",{"type":48,"tag":583,"props":632,"children":633},{},[634,645,650],{"type":48,"tag":610,"props":635,"children":636},{},[637,639],{"type":54,"value":638},"Pods stuck in ",{"type":48,"tag":73,"props":640,"children":642},{"className":641},[],[643],{"type":54,"value":644},"ContainerCreating",{"type":48,"tag":610,"props":646,"children":647},{},[648],{"type":54,"value":649},"VFs not allocated or SR-IOV policy not applied",{"type":48,"tag":610,"props":651,"children":652},{},[653,655,661],{"type":54,"value":654},"Check ",{"type":48,"tag":73,"props":656,"children":658},{"className":657},[],[659],{"type":54,"value":660},"sriovnetworknodestates",{"type":54,"value":662},", verify device plugin pods",{"type":48,"tag":583,"props":664,"children":665},{},[666,677,682],{"type":48,"tag":610,"props":667,"children":668},{},[669,675],{"type":48,"tag":73,"props":670,"children":672},{"className":671},[],[673],{"type":54,"value":674},"CrashLoopBackOff",{"type":54,"value":676}," on mofed pods",{"type":48,"tag":610,"props":678,"children":679},{},[680],{"type":54,"value":681},"Kernel module conflict",{"type":48,"tag":610,"props":683,"children":684},{},[685,686,692,694],{"type":54,"value":654},{"type":48,"tag":73,"props":687,"children":689},{"className":688},[],[690],{"type":54,"value":691},"thirdPartyRDMAModules",{"type":54,"value":693},", enable ",{"type":48,"tag":73,"props":695,"children":697},{"className":696},[],[698],{"type":54,"value":699},"unloadThirdPartyRDMAModules",{"type":48,"tag":583,"props":701,"children":702},{},[703,708,713],{"type":48,"tag":610,"props":704,"children":705},{},[706],{"type":54,"value":707},"No VFs on node",{"type":48,"tag":610,"props":709,"children":710},{},[711],{"type":54,"value":712},"SriovNetworkNodePolicy not matching",{"type":48,"tag":610,"props":714,"children":715},{},[716,718,724],{"type":54,"value":717},"Verify ",{"type":48,"tag":73,"props":719,"children":721},{"className":720},[],[722],{"type":54,"value":723},"nodeSelector",{"type":54,"value":725}," labels match worker nodes",{"type":48,"tag":583,"props":727,"children":728},{},[729,734,739],{"type":48,"tag":610,"props":730,"children":731},{},[732],{"type":54,"value":733},"RDMA not working",{"type":48,"tag":610,"props":735,"children":736},{},[737],{"type":54,"value":738},"Missing RDMA device plugin or wrong resource name",{"type":48,"tag":610,"props":740,"children":741},{},[742,743,749],{"type":54,"value":654},{"type":48,"tag":73,"props":744,"children":746},{"className":745},[],[747],{"type":54,"value":748},"rdma-shared-dp",{"type":54,"value":750}," pods, verify resource annotations",{"type":48,"tag":583,"props":752,"children":753},{},[754,765,778],{"type":48,"tag":610,"props":755,"children":756},{},[757,763],{"type":48,"tag":73,"props":758,"children":760},{"className":759},[],[761],{"type":54,"value":762},"l8k discover",{"type":54,"value":764}," daemon pods stuck (ImagePullBackOff \u002F Pending)",{"type":48,"tag":610,"props":766,"children":767},{},[768,770,776],{"type":54,"value":769},"Bad image tag, missing pull secret, or no ",{"type":48,"tag":73,"props":771,"children":773},{"className":772},[],[774],{"type":54,"value":775},"feature.node.kubernetes.io\u002Fpci-15b3.present=true",{"type":54,"value":777}," nodes",{"type":48,"tag":610,"props":779,"children":780},{},[781,783,789,791,797,799,805,807,813],{"type":54,"value":782},"Re-run with ",{"type":48,"tag":73,"props":784,"children":786},{"className":785},[],[787],{"type":54,"value":788},"--keep-namespace",{"type":54,"value":790}," then ",{"type":48,"tag":73,"props":792,"children":794},{"className":793},[],[795],{"type":54,"value":796},"kubectl describe pod -n nvidia-k8s-launch-kit",{"type":54,"value":798},". Fix ",{"type":48,"tag":73,"props":800,"children":802},{"className":801},[],[803],{"type":54,"value":804},"networkOperator.componentVersion",{"type":54,"value":806}," \u002F pass ",{"type":48,"tag":73,"props":808,"children":810},{"className":809},[],[811],{"type":54,"value":812},"--image-pull-secrets",{"type":54,"value":814}," \u002F verify NFD is running.",{"type":48,"tag":583,"props":816,"children":817},{},[818,837,842],{"type":48,"tag":610,"props":819,"children":820},{},[821,827,829,835],{"type":48,"tag":73,"props":822,"children":824},{"className":823},[],[825],{"type":54,"value":826},"l8k validate",{"type":54,"value":828}," \u002F ",{"type":48,"tag":73,"props":830,"children":832},{"className":831},[],[833],{"type":54,"value":834},"deploy",{"type":54,"value":836}," can't find Network Operator pods",{"type":48,"tag":610,"props":838,"children":839},{},[840],{"type":54,"value":841},"Operator namespace mismatch",{"type":48,"tag":610,"props":843,"children":844},{},[845,846,852,854,859,861,867],{"type":54,"value":717},{"type":48,"tag":73,"props":847,"children":849},{"className":848},[],[850],{"type":54,"value":851},"--network-operator-namespace",{"type":54,"value":853}," matches actual namespace (does NOT apply to ",{"type":48,"tag":73,"props":855,"children":857},{"className":856},[],[858],{"type":54,"value":762},{"type":54,"value":860}," — it ignores the flag and uses its own ",{"type":48,"tag":73,"props":862,"children":864},{"className":863},[],[865],{"type":54,"value":866},"nvidia-k8s-launch-kit",{"type":54,"value":868}," namespace)",{"type":48,"tag":583,"props":870,"children":871},{},[872,877,882],{"type":48,"tag":610,"props":873,"children":874},{},[875],{"type":54,"value":876},"IPPool not allocating",{"type":48,"tag":610,"props":878,"children":879},{},[880],{"type":54,"value":881},"NV-IPAM subnet exhausted or misconfigured",{"type":48,"tag":610,"props":883,"children":884},{},[885,886,892],{"type":54,"value":654},{"type":48,"tag":73,"props":887,"children":889},{"className":888},[],[890],{"type":54,"value":891},"ippools",{"type":54,"value":893}," CR status, verify CIDR ranges",{"type":48,"tag":583,"props":895,"children":896},{},[897,906,923],{"type":48,"tag":610,"props":898,"children":899},{},[900],{"type":48,"tag":73,"props":901,"children":903},{"className":902},[],[904],{"type":54,"value":905},"--for requires --node-selector",{"type":48,"tag":610,"props":907,"children":908},{},[909,915,917],{"type":48,"tag":73,"props":910,"children":912},{"className":911},[],[913],{"type":54,"value":914},"--for",{"type":54,"value":916}," was passed without ",{"type":48,"tag":73,"props":918,"children":920},{"className":919},[],[921],{"type":54,"value":922},"--node-selector",{"type":48,"tag":610,"props":924,"children":925},{},[926,928,934],{"type":54,"value":927},"Add ",{"type":48,"tag":73,"props":929,"children":931},{"className":930},[],[932],{"type":54,"value":933},"--node-selector key=val,…",{"type":54,"value":935},". The synthesized clusterConfig has no live worker-node list; the selector identifies target nodes at apply time.",{"type":48,"tag":583,"props":937,"children":938},{},[939,948,953],{"type":48,"tag":610,"props":940,"children":941},{},[942],{"type":48,"tag":73,"props":943,"children":945},{"className":944},[],[946],{"type":54,"value":947},"--for and --discover-cluster-config are mutually exclusive",{"type":48,"tag":610,"props":949,"children":950},{},[951],{"type":54,"value":952},"Both flags passed simultaneously",{"type":48,"tag":610,"props":954,"children":955},{},[956,958,963,965,971],{"type":54,"value":957},"Pick one: ",{"type":48,"tag":73,"props":959,"children":961},{"className":960},[],[962],{"type":54,"value":914},{"type":54,"value":964}," skips discovery, ",{"type":48,"tag":73,"props":966,"children":968},{"className":967},[],[969],{"type":54,"value":970},"--discover-cluster-config",{"type":54,"value":972}," runs it.",{"type":48,"tag":583,"props":974,"children":975},{},[976,985,1002],{"type":48,"tag":610,"props":977,"children":978},{},[979],{"type":48,"tag":73,"props":980,"children":982},{"className":981},[],[983],{"type":54,"value":984},"unknown preset \"X\"; available: …",{"type":48,"tag":610,"props":986,"children":987},{},[988,994,996],{"type":48,"tag":73,"props":989,"children":991},{"className":990},[],[992],{"type":54,"value":993},"--for X",{"type":54,"value":995}," doesn't match any directory under ",{"type":48,"tag":73,"props":997,"children":999},{"className":998},[],[1000],{"type":54,"value":1001},"presets\u002F",{"type":48,"tag":610,"props":1003,"children":1004},{},[1005,1007,1013],{"type":54,"value":1006},"Run ",{"type":48,"tag":73,"props":1008,"children":1010},{"className":1009},[],[1011],{"type":54,"value":1012},"l8k preset list",{"type":54,"value":1014}," and re-run with one of those names.",{"type":48,"tag":583,"props":1016,"children":1017},{},[1018,1027,1045],{"type":48,"tag":610,"props":1019,"children":1020},{},[1021],{"type":48,"tag":73,"props":1022,"children":1024},{"className":1023},[],[1025],{"type":54,"value":1026},"preset has no capabilities block",{"type":48,"tag":610,"props":1028,"children":1029},{},[1030,1032,1037,1039],{"type":54,"value":1031},"Preset YAML used by ",{"type":48,"tag":73,"props":1033,"children":1035},{"className":1034},[],[1036],{"type":54,"value":914},{"type":54,"value":1038}," is missing ",{"type":48,"tag":73,"props":1040,"children":1042},{"className":1041},[],[1043],{"type":54,"value":1044},"capabilities.nodes.{sriov,rdma,ib}",{"type":48,"tag":610,"props":1046,"children":1047},{},[1048,1050,1056,1058,1063],{"type":54,"value":1049},"Add the block to the preset's ",{"type":48,"tag":73,"props":1051,"children":1053},{"className":1052},[],[1054],{"type":54,"value":1055},"topology.yaml",{"type":54,"value":1057},". Discovery-time overlay does not require it; only ",{"type":48,"tag":73,"props":1059,"children":1061},{"className":1060},[],[1062],{"type":54,"value":914},{"type":54,"value":1064}," does.",{"type":48,"tag":583,"props":1066,"children":1067},{},[1068,1079,1084],{"type":48,"tag":610,"props":1069,"children":1070},{},[1071,1077],{"type":48,"tag":73,"props":1072,"children":1074},{"className":1073},[],[1075],{"type":54,"value":1076},"unknown field \"productType\"",{"type":54,"value":1078}," in YAML",{"type":48,"tag":610,"props":1080,"children":1081},{},[1082],{"type":54,"value":1083},"Hand-authored config still uses the old key name",{"type":48,"tag":610,"props":1085,"children":1086},{},[1087,1089,1095,1097,1103],{"type":54,"value":1088},"Rename ",{"type":48,"tag":73,"props":1090,"children":1092},{"className":1091},[],[1093],{"type":54,"value":1094},"productType:",{"type":54,"value":1096}," to ",{"type":48,"tag":73,"props":1098,"children":1100},{"className":1099},[],[1101],{"type":54,"value":1102},"gpuType:",{"type":54,"value":1104}," (the field was renamed).",{"type":48,"tag":61,"props":1106,"children":1107},{},[1108,1110,1116],{"type":54,"value":1109},"For detailed triage workflow, read ",{"type":48,"tag":73,"props":1111,"children":1113},{"className":1112},[],[1114],{"type":54,"value":1115},"references\u002Ftroubleshooting-guide.md",{"type":54,"value":1117},".",{"type":48,"tag":87,"props":1119,"children":1121},{"id":1120},"sosreport-analysis",[1122],{"type":54,"value":1123},"sosreport Analysis",{"type":48,"tag":61,"props":1125,"children":1126},{},[1127,1129,1134],{"type":54,"value":1128},"If the user has a pre-collected sosreport directory (from ",{"type":48,"tag":73,"props":1130,"children":1132},{"className":1131},[],[1133],{"type":54,"value":182},{"type":54,"value":1135}," or manual collection):",{"type":48,"tag":94,"props":1137,"children":1141},{"className":1138,"code":1140,"language":54},[1139],"language-text","sosreport\u002F\n├── metadata\u002F          # Cluster info, node list\n├── crds\u002F              # NicClusterPolicy, SriovNetworkNodePolicy, IPPool, etc.\n├── operator\u002F          # Network operator pod logs\n├── nodes\u002F             # Per-node device info\n└── network\u002F           # Interface config, routing tables\n",[1142],{"type":48,"tag":73,"props":1143,"children":1144},{"__ignoreMap":99},[1145],{"type":54,"value":1140},{"type":48,"tag":1147,"props":1148,"children":1150},"h3",{"id":1149},"triage-checklist",[1151],{"type":54,"value":1152},"Triage Checklist",{"type":48,"tag":1154,"props":1155,"children":1156},"ol",{},[1157,1171,1182,1195,1208],{"type":48,"tag":1158,"props":1159,"children":1160},"li",{},[1161,1163,1169],{"type":54,"value":1162},"Read ",{"type":48,"tag":73,"props":1164,"children":1166},{"className":1165},[],[1167],{"type":54,"value":1168},"metadata\u002Fdiagnostic-summary.yaml",{"type":54,"value":1170}," for overview",{"type":48,"tag":1158,"props":1172,"children":1173},{},[1174,1176],{"type":54,"value":1175},"Check pod health in ",{"type":48,"tag":73,"props":1177,"children":1179},{"className":1178},[],[1180],{"type":54,"value":1181},"operator\u002Fpods.yaml",{"type":48,"tag":1158,"props":1183,"children":1184},{},[1185,1187,1193],{"type":54,"value":1186},"Inspect CRDs in ",{"type":48,"tag":73,"props":1188,"children":1190},{"className":1189},[],[1191],{"type":54,"value":1192},"crds\u002F",{"type":54,"value":1194}," for status fields",{"type":48,"tag":1158,"props":1196,"children":1197},{},[1198,1200,1206],{"type":54,"value":1199},"Read operator logs in ",{"type":48,"tag":73,"props":1201,"children":1203},{"className":1202},[],[1204],{"type":54,"value":1205},"operator\u002Flogs\u002F",{"type":54,"value":1207}," for errors",{"type":48,"tag":1158,"props":1209,"children":1210},{},[1211,1213],{"type":54,"value":1212},"Check per-node NIC state in ",{"type":48,"tag":73,"props":1214,"children":1216},{"className":1215},[],[1217],{"type":54,"value":1218},"nodes\u002F\u003Cnode>\u002F",{"type":48,"tag":87,"props":1220,"children":1222},{"id":1221},"see-also",[1223],{"type":54,"value":1224},"See Also",{"type":48,"tag":1226,"props":1227,"children":1228},"ul",{},[1229,1239,1250],{"type":48,"tag":1158,"props":1230,"children":1231},{},[1232,1237],{"type":48,"tag":1233,"props":1234,"children":1235},"a",{"href":78},[1236],{"type":54,"value":43},{"type":54,"value":1238}," — Exit codes and error structure",{"type":48,"tag":1158,"props":1240,"children":1241},{},[1242,1248],{"type":48,"tag":1233,"props":1243,"children":1245},{"href":1244},"..\u002Fk8s-launch-kit-discover\u002FSKILL.md",[1246],{"type":54,"value":1247},"k8s-launch-kit-discover",{"type":54,"value":1249}," — Re-discover to verify hardware state",{"type":48,"tag":1158,"props":1251,"children":1252},{},[1253,1258],{"type":48,"tag":73,"props":1254,"children":1256},{"className":1255},[],[1257],{"type":54,"value":1115},{"type":54,"value":1259}," — Detailed triage workflow",{"type":48,"tag":1261,"props":1262,"children":1263},"style",{},[1264],{"type":54,"value":1265},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1267,"total":359},[1268,1283,1294,1307,1319,1333,1347],{"slug":1269,"name":1269,"fn":1270,"description":1271,"org":1272,"tags":1273,"stars":26,"repoUrl":27,"updatedAt":1282},"k8s-launch-kit-config","configure k8s-launch-kit clusters","Use this skill when the user needs help understanding, creating, or editing a k8s-launch-kit (l8k) configuration file (l8k-config.yaml or cluster-config.yaml). Activate for: config file questions, parameter tuning, subnet configuration, NV-IPAM setup, DOCA driver settings, maintenance concurrency, NIC configuration operator settings, changing MTU, VFs, resource names, or understanding what any config field does.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1274,1277,1280,1281],{"name":1275,"slug":1276,"type":15},"Configuration","configuration",{"name":1278,"slug":1279,"type":15},"Deployment","deployment",{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},"2026-07-30T05:29:15.229216",{"slug":1284,"name":1284,"fn":1285,"description":1286,"org":1287,"tags":1288,"stars":26,"repoUrl":27,"updatedAt":1293},"k8s-launch-kit-deploy","deploy NVIDIA networking manifests to Kubernetes","Use this skill when the user wants to deploy generated NVIDIA networking manifests to a Kubernetes cluster using k8s-launch-kit (l8k). Activate for: applying manifests, deploying to cluster, the `l8k deploy` subcommand or the legacy --deploy flag on `l8k generate`, applying generated files, or any mention of pushing l8k output to a live cluster. Even if the user just says 'apply these' or 'push to cluster' after generating manifests, use this skill.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1289,1290,1291,1292],{"name":1278,"slug":1279,"type":15},{"name":21,"slug":22,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:31:50.114923",{"slug":1247,"name":1247,"fn":1295,"description":1296,"org":1297,"tags":1298,"stars":26,"repoUrl":27,"updatedAt":1306},"discover Kubernetes cluster network hardware capabilities","Use this skill when the user wants to discover their Kubernetes cluster's network hardware capabilities using k8s-launch-kit (l8k). Activate for: cluster discovery, hardware detection, NIC detection, finding what GPUs or NICs are in a cluster, creating a cluster config file, or when the user says 'discover' in the context of l8k or NVIDIA networking.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1299,1302,1303,1304,1305],{"name":1300,"slug":1301,"type":15},"Hardware","hardware",{"name":21,"slug":22,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:31:58.825512",{"slug":1308,"name":1308,"fn":1309,"description":1310,"org":1311,"tags":1312,"stars":26,"repoUrl":27,"updatedAt":1318},"k8s-launch-kit-dryrun","preview NVIDIA networking deployment changes","Use this skill when the user wants to preview what k8s-launch-kit (l8k) would deploy without making changes, or wants to safely validate their configuration before applying. Activate for: dry-run, preview, validation, 'what would happen if', testing configurations, schema discovery, checking generated manifests, or any cautious pre-deployment step. Also use when the user asks 'is my config valid' or 'show me what would be created' -- even without mentioning dry-run explicitly.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1313,1314,1315,1316,1317],{"name":1275,"slug":1276,"type":15},{"name":1278,"slug":1279,"type":15},{"name":21,"slug":22,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:31:56.325698",{"slug":1320,"name":1320,"fn":1321,"description":1322,"org":1323,"tags":1324,"stars":26,"repoUrl":27,"updatedAt":1332},"k8s-launch-kit-generate","generate Kubernetes manifests for NVIDIA networking","Use this skill when the user wants to generate Kubernetes YAML manifests for NVIDIA networking deployment using k8s-launch-kit (l8k). Activate for: manifest generation, profile selection, choosing between SR-IOV\u002Fhost-device\u002FRDMA-shared\u002FIPoIB\u002FMacVLAN\u002FSpectrum-X, creating deployment files, or when the user asks 'which profile should I use' or needs help choosing a network configuration.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1325,1326,1327,1328,1329],{"name":1278,"slug":1279,"type":15},{"name":21,"slug":22,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":1330,"slug":1331,"type":15},"YAML","yaml","2026-07-14T05:31:51.352013",{"slug":1334,"name":1334,"fn":1335,"description":1336,"org":1337,"tags":1338,"stars":26,"repoUrl":27,"updatedAt":1346},"k8s-launch-kit-pipeline","run end-to-end NVIDIA networking deployment pipelines","Use this skill when the user wants to run the full k8s-launch-kit (l8k) pipeline end-to-end: discover cluster hardware, select a profile, generate manifests, and deploy them all in one command. Also activate for CI\u002FCD integration, automation pipelines, 'one-liner', 'complete workflow', or end-to-end NVIDIA networking deployment.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1339,1342,1343,1344,1345],{"name":1340,"slug":1341,"type":15},"Automation","automation",{"name":1278,"slug":1279,"type":15},{"name":21,"slug":22,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:31:55.07782",{"slug":43,"name":43,"fn":1348,"description":1349,"org":1350,"tags":1351,"stars":26,"repoUrl":27,"updatedAt":1360},"manage NVIDIA k8s-launch-kit CLI patterns","k8s-launch-kit (l8k) CLI: Shared patterns for binary location, global flags, output formatting, exit codes, and error handling. Read this before using any other k8s-launch-kit skill.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1352,1355,1358,1359],{"name":1353,"slug":1354,"type":15},"CLI","cli",{"name":1356,"slug":1357,"type":15},"Engineering","engineering",{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:32:01.309487",{"items":1362,"total":1515},[1363,1381,1397,1408,1420,1432,1445,1459,1472,1483,1497,1506],{"slug":1364,"name":1364,"fn":1365,"description":1366,"org":1367,"tags":1368,"stars":1378,"repoUrl":1379,"updatedAt":1380},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1369,1372,1375],{"name":1370,"slug":1371,"type":15},"Documentation","documentation",{"name":1373,"slug":1374,"type":15},"MCP","mcp",{"name":1376,"slug":1377,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":1382,"name":1382,"fn":1383,"description":1384,"org":1385,"tags":1386,"stars":1394,"repoUrl":1395,"updatedAt":1396},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1387,1390,1391],{"name":1388,"slug":1389,"type":15},"Containers","containers",{"name":1278,"slug":1279,"type":15},{"name":1392,"slug":1393,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":1398,"name":1398,"fn":1399,"description":1400,"org":1401,"tags":1402,"stars":1394,"repoUrl":1395,"updatedAt":1407},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1403,1406],{"name":1404,"slug":1405,"type":15},"CI\u002FCD","ci-cd",{"name":1278,"slug":1279,"type":15},"2026-07-14T05:25:59.97109",{"slug":1409,"name":1409,"fn":1410,"description":1411,"org":1412,"tags":1413,"stars":1394,"repoUrl":1395,"updatedAt":1419},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1414,1415,1416],{"name":1404,"slug":1405,"type":15},{"name":1278,"slug":1279,"type":15},{"name":1417,"slug":1418,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":1421,"name":1421,"fn":1422,"description":1423,"org":1424,"tags":1425,"stars":1394,"repoUrl":1395,"updatedAt":1431},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1426,1427,1428],{"name":24,"slug":25,"type":15},{"name":1417,"slug":1418,"type":15},{"name":1429,"slug":1430,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1433,"name":1433,"fn":1434,"description":1435,"org":1436,"tags":1437,"stars":1394,"repoUrl":1395,"updatedAt":1444},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1438,1441],{"name":1439,"slug":1440,"type":15},"Best Practices","best-practices",{"name":1442,"slug":1443,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1446,"name":1446,"fn":1447,"description":1448,"org":1449,"tags":1450,"stars":1394,"repoUrl":1395,"updatedAt":1458},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1451,1454,1457],{"name":1452,"slug":1453,"type":15},"Machine Learning","machine-learning",{"name":1455,"slug":1456,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":1460,"name":1460,"fn":1461,"description":1462,"org":1463,"tags":1464,"stars":1394,"repoUrl":1395,"updatedAt":1471},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1465,1468],{"name":1466,"slug":1467,"type":15},"QA","qa",{"name":1469,"slug":1470,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":1473,"name":1473,"fn":1474,"description":1475,"org":1476,"tags":1477,"stars":1394,"repoUrl":1395,"updatedAt":1482},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1478,1479],{"name":1278,"slug":1279,"type":15},{"name":1480,"slug":1481,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":1484,"name":1484,"fn":1485,"description":1486,"org":1487,"tags":1488,"stars":1394,"repoUrl":1395,"updatedAt":1496},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1489,1492,1493],{"name":1490,"slug":1491,"type":15},"Code Review","code-review",{"name":1417,"slug":1418,"type":15},{"name":1494,"slug":1495,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":1498,"name":1498,"fn":1499,"description":1500,"org":1501,"tags":1502,"stars":1394,"repoUrl":1395,"updatedAt":1505},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1503,1504],{"name":1466,"slug":1467,"type":15},{"name":1469,"slug":1470,"type":15},"2026-07-14T05:25:54.928983",{"slug":1507,"name":1507,"fn":1508,"description":1509,"org":1510,"tags":1511,"stars":1394,"repoUrl":1395,"updatedAt":1514},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1512,1513],{"name":1340,"slug":1341,"type":15},{"name":1404,"slug":1405,"type":15},"2026-07-30T05:29:03.275638",496]