[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nvshmem-troubleshoot-and-report-bugs":3,"mdc--sx3a68-key":39,"related-repo-nvidia-nvshmem-troubleshoot-and-report-bugs":938,"related-org-nvidia-nvshmem-troubleshoot-and-report-bugs":1029},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":34,"sourceUrl":37,"mdContent":38},"nvshmem-troubleshoot-and-report-bugs","diagnose and report NVSHMEM runtime issues","Diagnose NVSHMEM runtime failures and prepare bug reports for launch, initialization, crashes, hangs, correctness, transport, or topology issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"GPU","gpu","tag",{"name":17,"slug":18,"type":15},"Diagnostics","diagnostics",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Debugging","debugging",579,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnvshmem","2026-08-31T09:18:49.43654","Apache-2.0",104,[29,30,31,32,8,33],"communciations","cpp","cuda","deep-learning","python",{"repoUrl":24,"stars":23,"forks":27,"topics":35,"description":36},[29,30,31,32,8,33],"NVIDIA NVSHMEM is a parallel programming interface for NVIDIA GPUs based on OpenSHMEM. NVSHMEM can significantly reduce multi-process communication and coordination overheads by allowing programmers to perform one-sided communication from within CUDA kernels and on CUDA streams.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnvshmem\u002Ftree\u002FHEAD\u002Fskills\u002Fnvshmem-troubleshoot-and-report-bugs","---\nname: nvshmem-troubleshoot-and-report-bugs\ndescription: Diagnose NVSHMEM runtime failures and prepare bug reports for launch, initialization, crashes, hangs, correctness, transport, or topology issues.\nlicense: Apache-2.0\nmetadata:\n  version: \"1.0.0\"\n  author: NVIDIA NVSHMEM Team \u003Cnvshmem@nvidia.com>\n  tags:\n    - nvshmem\n    - troubleshooting\n    - bug\n---\n\n# NVSHMEM Troubleshooting and Bug Reporting\n\n## Purpose\n\nDiagnose NVSHMEM runtime failures from high-signal evidence. Match documented problems conservatively, verify version-sensitive claims with live official documentation, separate NVSHMEM defects from application or environment failures, and prepare a portable report when escalation is useful.\n\n## Prerequisites\n\n- Require no credentials, API keys, or privileged access.\n- Work from the user's exact launch command, complete output, and generated logs. Use direct read-only inspection only when the user supplies access and asks for it.\n- Keep `$nvshmem-docs` available to verify version-specific behavior and `$nvshmem-install` available for installation or build handoffs. When a sibling skill is not registered, read its `SKILL.md` directly.\n- Keep [troubleshooting-knowledge-base.md](references\u002Ftroubleshooting-knowledge-base.md) available for diagnosis and [bug-report-template.md](references\u002Fbug-report-template.md) available for escalation.\n\n## Inputs\n\n- **Required:** Obtain the exact command or complete batch-script launch stanza and complete stdout and stderr from every PE, including lines before the first error and any per-PE logs.\n- **Conditional:** Obtain output that confirms INFO-level NVSHMEM logging before diagnosing an in-process NVSHMEM failure. Apply the documented pre-start exception only when NVSHMEM demonstrably cannot emit logs.\n- **Optional:** Obtain the full NVSHMEM version, CUDA and driver versions, launcher and bootstrap, transport settings, PE\u002Fnode\u002FGPU counts, rank-to-GPU mapping, topology, smallest failing scale, and last known-good configuration only when they distinguish plausible causes.\n- **Source precedence:** Prefer raw artifacts from the current run, then explicit values in the current user prompt or invocation, then non-conflicting agent context, then focused follow-up answers. Report conflicts and prefer the current raw artifact unless the user confirms a correction.\n\n## Limitations\n\n- Do not confirm a root cause from paraphrased errors, incomplete PE output, a partial signature match, or timing changes alone.\n- Treat the bundled knowledge base as a routing snapshot, not current authority. Require live official documentation for version-specific claims.\n- Do not claim that a fixed-in release proves every earlier release is affected or that an upgrade is compatible without checking the collected environment.\n- Do not perform privileged platform changes or directly submit reports. Provide read-only checks, administrator questions, and paste-ready Markdown instead.\n\n## Instructions\n\n### Start With the Command and Output\n\nAsk first for both of the following when either is missing, then stop:\n\n1. The exact command or complete batch-script launch stanza, including environment settings, launcher options, container invocation, and module loads.\n2. The complete stdout and stderr from every PE, including lines before the first error and any per-PE log files.\n\nDo not begin with a broad environment questionnaire. Do not diagnose from a paraphrased error when raw output can be obtained.\n\n### Require INFO Logging\n\nDetermine whether the captured output contains `NVSHMEM_DEBUG=INFO` or clear INFO-level NVSHMEM records. The command line alone is not sufficient because a launcher may fail to propagate its environment. If the output does not confirm INFO logging, ask the user to repeat the same run with:\n\n```text\nNVSHMEM_DEBUG=INFO\nNVSHMEM_DEBUG_SUBSYS=ALL\nNVSHMEM_DEBUG_FILE=nvshmem-%h-%p.log\n```\n\nPreserve the executable, launcher, PE and node counts, binding, transport, container, and workload arguments. Put the variables where the selected launcher propagates them to every PE. For a simple direct launch, prefix them to the command. For `srun`, `mpirun`, `nvshmrun`, or another launcher, invoke `$nvshmem-docs` for the detected NVSHMEM version before giving launcher-specific syntax when propagation is uncertain.\n\nAsk for the repeated command, stdout and stderr, and every generated `nvshmem-\u003Chost>-\u003Cpid>.log`. Wait for that evidence before analyzing NVSHMEM behavior.\n\nApply one exception: diagnose immediately when the failure demonstrably occurs before NVSHMEM starts and cannot emit NVSHMEM logs, such as shell parsing, executable lookup, loader startup, or launcher admission failure. State why the debug rerun cannot add NVSHMEM evidence. Do not use the exception merely because an existing error looks familiar.\n\n### Build the Evidence Record\n\nAfter the logging gate, extract before asking for more information:\n\n- the first causal error or divergence on each affected PE, separate from cascading aborts;\n- the NVSHMEM version and any host\u002Fdevice library mismatch reported by INFO logs;\n- launcher and bootstrap, selected transports, relevant `NVSHMEM_*` settings, PE\u002Fnode\u002FGPU counts, and rank-to-GPU mapping;\n- whether the failure changes with one PE, one node, different GPU pairs, or multiple nodes;\n- the operation and phase: launch, initialization, allocation, communication, synchronization, collective, finalization, or process teardown.\n\nAsk only for missing facts that distinguish plausible causes. Never fabricate a version, topology, environment setting, or reproduction result.\n\n### Diagnose Conservatively\n\nRead [troubleshooting-knowledge-base.md](references\u002Ftroubleshooting-knowledge-base.md) for every diagnosis after the logging gate or pre-start exception. Use its topic index to select the relevant section, then search exact log signatures within that section before falling back to semantic matches.\n\nClassify the result as exactly one of:\n\n1. **Confirmed bundled knowledge-base match**: the signature and required conditions match an entry.\n2. **Version-specific known issue or later-release fix**: official release notes establish applicability to the detected version.\n3. **Likely NVSHMEM diagnosis not in the knowledge base**: evidence points into NVSHMEM, but no bundled or documented issue matches.\n4. **Likely external issue**: evidence points to the application, CUDA, launcher, loader, network, driver, scheduler, container, or platform configuration.\n\nUse class 1 whenever a bundled entry is confirmed, even when that entry assigns ownership to an external component; state ownership separately in `Diagnosis`. Use class 4 only when no bundled or versioned documented issue matches. For partial signature matches, use class 3 or 4 rather than calling the match confirmed. Treat a similar release-note symptom as a lead, not proof that the user has the same defect. A release note that says only that a bug was fixed establishes the fix release, not every affected earlier release. Describe an older installed version as predating a candidate fix unless official evidence explicitly establishes that it is affected or a controlled upgrade test confirms the match.\n\nInvoke `$nvshmem-docs` with the complete detected version before asserting environment-variable behavior, requirements, limitations, known issues, fixes, or upgrade guidance. Ask for the complete version when logs do not provide it and a version claim matters. When the sibling skill is not registered, read and follow `..\u002Fnvshmem-docs\u002FSKILL.md` directly.\n\nBefore recommending a target upgrade, verify that release's compatibility and known issues against the collected CUDA, driver, GPU, CPU, NCCL, bootstrap, and transport context. If that context is incomplete, present the release as a candidate test and name the missing compatibility checks instead of calling it suitable.\n\nSeparate:\n\n- **Observation**: what the command and logs directly show.\n- **Documentation evidence**: the matched FAQ, environment-variable entry, limitation, known issue, or fixed issue.\n- **Inference**: the most likely explanation and its confidence.\n- **Verification**: the smallest safe test that can confirm or reject the inference.\n\nIf no entry matches, say exactly: `No matching entry was found in the bundled NVSHMEM troubleshooting knowledge base.` Then provide a likely diagnosis, confidence, competing explanations, and targeted next checks.\n\nFor a likely external issue, identify the likely owner and give concrete next checks. Do not stop at “not NVSHMEM.” Route installation and build failures to `$nvshmem-install`; route performance-only regressions to an available NVSHMEM performance skill, or to `$nvshmem-docs` when none is available.\n\nDo not perform privileged platform changes. For IMEX, driver, fabric-manager, device-node, kernel-module, NIC, or scheduler configuration, provide read-only checks and tell the user what to ask the administrator to verify.\n\n### Present the Diagnosis\n\nReturn these headings:\n\n- `Evidence Collected`\n- `Diagnosis`\n- `Knowledge-Base Match`\n- `Version Applicability`\n- `Next Actions`\n\nInclude direct official source links for documentation-backed claims. Keep commands copyable and mark placeholders. Prefer one discriminating test at a time over a long generic checklist.\n\n### Prepare a Bug Report or Question\n\nOffer a report when the issue remains unresolved, is likely an NVSHMEM defect, or the user explicitly requests one. Do not routinely offer escalation for a resolved local configuration problem or a clearly external failure.\n\nWhen INFO-level NVSHMEM logs are absent, explicitly ask the user to reproduce the failure before support submission with:\n\n```text\nNVSHMEM_DEBUG=INFO\nNVSHMEM_DEBUG_SUBSYS=ALL\nNVSHMEM_DEBUG_FILE=nvshmem-%h-%p.log\n```\n\nPreserve the executable, launcher, PE and node counts, binding, transport, container, and workload arguments. Ask for the exact repeated command, complete stdout\u002Fstderr from every PE, and all generated log files. A draft may still be generated when the user asks to preserve the investigation, but label it incomplete and include this rerun as a required pre-submission step; do not present it as support-ready.\n\nWhen preparing one, read [bug-report-template.md](references\u002Fbug-report-template.md) and fill it from collected evidence. Mirror the `NVSHMEM issue or bug` GitHub form: start the title with `[Issue]:`, use its exact field headings, and select exactly one supported impact value. Preserve exact commands and the first causal error. Use `Not provided` for missing facts and list those gaps; never invent them. Remind the user to redact credentials, tokens, internal hostnames and IPs, and user-identifying data before sharing logs.\n\nReturn the Markdown in a fenced block so it can be pasted into GitHub, NVONLINE, or email. Do not require or directly submit through a connector.\n\n## Troubleshooting\n\nUse these patterns when the investigation stalls:\n\n| Error or signal | Likely cause | Solution |\n| --- | --- | --- |\n| No INFO records appear after setting debug variables | The launcher did not propagate the environment, or the process failed before NVSHMEM started | Verify propagation with version-matched launcher syntax. Use the pre-start exception only when the emitting component proves NVSHMEM did not start. |\n| Logs show only peer aborts or launcher cleanup | The visible lines are cascading consequences | Compare every PE log and use the earliest causal or divergent line, including preceding context. |\n| A knowledge-base signature matches but its required conditions do not | The match is only a hypothesis | Classify the issue as likely NVSHMEM or likely external, state the mismatch, and run one discriminating test. |\n| The NVSHMEM version is missing or live documentation is unavailable | Version applicability cannot be verified | Ask for the complete version and label version-sensitive guidance unverified; do not infer applicability from the snapshot. |\n| INFO logging changes or removes the failure | The failure may be timing-sensitive | Preserve both commands and outcomes, repeat enough trials to characterize frequency, and do not call the logging change a fix. |\n\n## Examples\n\n- **Missing evidence:** Given only “NVSHMEM hangs during init,” ask for the exact launch stanza and complete output from every PE, then stop.\n- **Missing INFO logs:** Given a complete command and non-INFO output from an in-process failure, request the same run with the three debug settings and all generated logs; preserve scale and arguments.\n- **Pre-start failure:** Given `error while loading shared libraries` before any PE enters NVSHMEM, state why INFO logging cannot help, identify the loader as the emitting component, and route an NVSHMEM-library installation problem to `$nvshmem-install`.\n- **Confirmed bundled match:** Given `No more teams available` plus matching team-usage conditions, classify it as a confirmed bundled match, verify `NVSHMEM_MAX_TEAMS` for the complete version, and propose one minimal rerun.\n- **No match:** State the required no-match sentence, separate observation from inference, give one discriminating test, and offer a filled report only if the issue remains unresolved or looks like an NVSHMEM defect.\n",{"data":40,"body":48},{"name":4,"description":6,"license":26,"metadata":41},{"version":42,"author":43,"tags":44},"1.0.0","NVIDIA NVSHMEM Team \u003Cnvshmem@nvidia.com>",[45,46,47],"nvshmem","troubleshooting","bug",{"type":49,"children":50},"root",[51,60,67,73,79,145,151,195,201,224,230,237,242,256,261,267,280,292,327,340,345,351,356,392,397,403,414,419,462,475,494,499,504,547,560,579,584,590,595,642,647,653,658,663,671,676,711,716,721,726,849,855],{"type":52,"tag":53,"props":54,"children":56},"element","h1",{"id":55},"nvshmem-troubleshooting-and-bug-reporting",[57],{"type":58,"value":59},"text","NVSHMEM Troubleshooting and Bug Reporting",{"type":52,"tag":61,"props":62,"children":64},"h2",{"id":63},"purpose",[65],{"type":58,"value":66},"Purpose",{"type":52,"tag":68,"props":69,"children":70},"p",{},[71],{"type":58,"value":72},"Diagnose NVSHMEM runtime failures from high-signal evidence. Match documented problems conservatively, verify version-sensitive claims with live official documentation, separate NVSHMEM defects from application or environment failures, and prepare a portable report when escalation is useful.",{"type":52,"tag":61,"props":74,"children":76},{"id":75},"prerequisites",[77],{"type":58,"value":78},"Prerequisites",{"type":52,"tag":80,"props":81,"children":82},"ul",{},[83,89,94,124],{"type":52,"tag":84,"props":85,"children":86},"li",{},[87],{"type":58,"value":88},"Require no credentials, API keys, or privileged access.",{"type":52,"tag":84,"props":90,"children":91},{},[92],{"type":58,"value":93},"Work from the user's exact launch command, complete output, and generated logs. Use direct read-only inspection only when the user supplies access and asks for it.",{"type":52,"tag":84,"props":95,"children":96},{},[97,99,106,108,114,116,122],{"type":58,"value":98},"Keep ",{"type":52,"tag":100,"props":101,"children":103},"code",{"className":102},[],[104],{"type":58,"value":105},"$nvshmem-docs",{"type":58,"value":107}," available to verify version-specific behavior and ",{"type":52,"tag":100,"props":109,"children":111},{"className":110},[],[112],{"type":58,"value":113},"$nvshmem-install",{"type":58,"value":115}," available for installation or build handoffs. When a sibling skill is not registered, read its ",{"type":52,"tag":100,"props":117,"children":119},{"className":118},[],[120],{"type":58,"value":121},"SKILL.md",{"type":58,"value":123}," directly.",{"type":52,"tag":84,"props":125,"children":126},{},[127,128,135,137,143],{"type":58,"value":98},{"type":52,"tag":129,"props":130,"children":132},"a",{"href":131},"references\u002Ftroubleshooting-knowledge-base.md",[133],{"type":58,"value":134},"troubleshooting-knowledge-base.md",{"type":58,"value":136}," available for diagnosis and ",{"type":52,"tag":129,"props":138,"children":140},{"href":139},"references\u002Fbug-report-template.md",[141],{"type":58,"value":142},"bug-report-template.md",{"type":58,"value":144}," available for escalation.",{"type":52,"tag":61,"props":146,"children":148},{"id":147},"inputs",[149],{"type":58,"value":150},"Inputs",{"type":52,"tag":80,"props":152,"children":153},{},[154,165,175,185],{"type":52,"tag":84,"props":155,"children":156},{},[157,163],{"type":52,"tag":158,"props":159,"children":160},"strong",{},[161],{"type":58,"value":162},"Required:",{"type":58,"value":164}," Obtain the exact command or complete batch-script launch stanza and complete stdout and stderr from every PE, including lines before the first error and any per-PE logs.",{"type":52,"tag":84,"props":166,"children":167},{},[168,173],{"type":52,"tag":158,"props":169,"children":170},{},[171],{"type":58,"value":172},"Conditional:",{"type":58,"value":174}," Obtain output that confirms INFO-level NVSHMEM logging before diagnosing an in-process NVSHMEM failure. Apply the documented pre-start exception only when NVSHMEM demonstrably cannot emit logs.",{"type":52,"tag":84,"props":176,"children":177},{},[178,183],{"type":52,"tag":158,"props":179,"children":180},{},[181],{"type":58,"value":182},"Optional:",{"type":58,"value":184}," Obtain the full NVSHMEM version, CUDA and driver versions, launcher and bootstrap, transport settings, PE\u002Fnode\u002FGPU counts, rank-to-GPU mapping, topology, smallest failing scale, and last known-good configuration only when they distinguish plausible causes.",{"type":52,"tag":84,"props":186,"children":187},{},[188,193],{"type":52,"tag":158,"props":189,"children":190},{},[191],{"type":58,"value":192},"Source precedence:",{"type":58,"value":194}," Prefer raw artifacts from the current run, then explicit values in the current user prompt or invocation, then non-conflicting agent context, then focused follow-up answers. Report conflicts and prefer the current raw artifact unless the user confirms a correction.",{"type":52,"tag":61,"props":196,"children":198},{"id":197},"limitations",[199],{"type":58,"value":200},"Limitations",{"type":52,"tag":80,"props":202,"children":203},{},[204,209,214,219],{"type":52,"tag":84,"props":205,"children":206},{},[207],{"type":58,"value":208},"Do not confirm a root cause from paraphrased errors, incomplete PE output, a partial signature match, or timing changes alone.",{"type":52,"tag":84,"props":210,"children":211},{},[212],{"type":58,"value":213},"Treat the bundled knowledge base as a routing snapshot, not current authority. Require live official documentation for version-specific claims.",{"type":52,"tag":84,"props":215,"children":216},{},[217],{"type":58,"value":218},"Do not claim that a fixed-in release proves every earlier release is affected or that an upgrade is compatible without checking the collected environment.",{"type":52,"tag":84,"props":220,"children":221},{},[222],{"type":58,"value":223},"Do not perform privileged platform changes or directly submit reports. Provide read-only checks, administrator questions, and paste-ready Markdown instead.",{"type":52,"tag":61,"props":225,"children":227},{"id":226},"instructions",[228],{"type":58,"value":229},"Instructions",{"type":52,"tag":231,"props":232,"children":234},"h3",{"id":233},"start-with-the-command-and-output",[235],{"type":58,"value":236},"Start With the Command and Output",{"type":52,"tag":68,"props":238,"children":239},{},[240],{"type":58,"value":241},"Ask first for both of the following when either is missing, then stop:",{"type":52,"tag":243,"props":244,"children":245},"ol",{},[246,251],{"type":52,"tag":84,"props":247,"children":248},{},[249],{"type":58,"value":250},"The exact command or complete batch-script launch stanza, including environment settings, launcher options, container invocation, and module loads.",{"type":52,"tag":84,"props":252,"children":253},{},[254],{"type":58,"value":255},"The complete stdout and stderr from every PE, including lines before the first error and any per-PE log files.",{"type":52,"tag":68,"props":257,"children":258},{},[259],{"type":58,"value":260},"Do not begin with a broad environment questionnaire. Do not diagnose from a paraphrased error when raw output can be obtained.",{"type":52,"tag":231,"props":262,"children":264},{"id":263},"require-info-logging",[265],{"type":58,"value":266},"Require INFO Logging",{"type":52,"tag":68,"props":268,"children":269},{},[270,272,278],{"type":58,"value":271},"Determine whether the captured output contains ",{"type":52,"tag":100,"props":273,"children":275},{"className":274},[],[276],{"type":58,"value":277},"NVSHMEM_DEBUG=INFO",{"type":58,"value":279}," or clear INFO-level NVSHMEM records. The command line alone is not sufficient because a launcher may fail to propagate its environment. If the output does not confirm INFO logging, ask the user to repeat the same run with:",{"type":52,"tag":281,"props":282,"children":287},"pre",{"className":283,"code":285,"language":58,"meta":286},[284],"language-text","NVSHMEM_DEBUG=INFO\nNVSHMEM_DEBUG_SUBSYS=ALL\nNVSHMEM_DEBUG_FILE=nvshmem-%h-%p.log\n","",[288],{"type":52,"tag":100,"props":289,"children":290},{"__ignoreMap":286},[291],{"type":58,"value":285},{"type":52,"tag":68,"props":293,"children":294},{},[295,297,303,305,311,312,318,320,325],{"type":58,"value":296},"Preserve the executable, launcher, PE and node counts, binding, transport, container, and workload arguments. Put the variables where the selected launcher propagates them to every PE. For a simple direct launch, prefix them to the command. For ",{"type":52,"tag":100,"props":298,"children":300},{"className":299},[],[301],{"type":58,"value":302},"srun",{"type":58,"value":304},", ",{"type":52,"tag":100,"props":306,"children":308},{"className":307},[],[309],{"type":58,"value":310},"mpirun",{"type":58,"value":304},{"type":52,"tag":100,"props":313,"children":315},{"className":314},[],[316],{"type":58,"value":317},"nvshmrun",{"type":58,"value":319},", or another launcher, invoke ",{"type":52,"tag":100,"props":321,"children":323},{"className":322},[],[324],{"type":58,"value":105},{"type":58,"value":326}," for the detected NVSHMEM version before giving launcher-specific syntax when propagation is uncertain.",{"type":52,"tag":68,"props":328,"children":329},{},[330,332,338],{"type":58,"value":331},"Ask for the repeated command, stdout and stderr, and every generated ",{"type":52,"tag":100,"props":333,"children":335},{"className":334},[],[336],{"type":58,"value":337},"nvshmem-\u003Chost>-\u003Cpid>.log",{"type":58,"value":339},". Wait for that evidence before analyzing NVSHMEM behavior.",{"type":52,"tag":68,"props":341,"children":342},{},[343],{"type":58,"value":344},"Apply one exception: diagnose immediately when the failure demonstrably occurs before NVSHMEM starts and cannot emit NVSHMEM logs, such as shell parsing, executable lookup, loader startup, or launcher admission failure. State why the debug rerun cannot add NVSHMEM evidence. Do not use the exception merely because an existing error looks familiar.",{"type":52,"tag":231,"props":346,"children":348},{"id":347},"build-the-evidence-record",[349],{"type":58,"value":350},"Build the Evidence Record",{"type":52,"tag":68,"props":352,"children":353},{},[354],{"type":58,"value":355},"After the logging gate, extract before asking for more information:",{"type":52,"tag":80,"props":357,"children":358},{},[359,364,369,382,387],{"type":52,"tag":84,"props":360,"children":361},{},[362],{"type":58,"value":363},"the first causal error or divergence on each affected PE, separate from cascading aborts;",{"type":52,"tag":84,"props":365,"children":366},{},[367],{"type":58,"value":368},"the NVSHMEM version and any host\u002Fdevice library mismatch reported by INFO logs;",{"type":52,"tag":84,"props":370,"children":371},{},[372,374,380],{"type":58,"value":373},"launcher and bootstrap, selected transports, relevant ",{"type":52,"tag":100,"props":375,"children":377},{"className":376},[],[378],{"type":58,"value":379},"NVSHMEM_*",{"type":58,"value":381}," settings, PE\u002Fnode\u002FGPU counts, and rank-to-GPU mapping;",{"type":52,"tag":84,"props":383,"children":384},{},[385],{"type":58,"value":386},"whether the failure changes with one PE, one node, different GPU pairs, or multiple nodes;",{"type":52,"tag":84,"props":388,"children":389},{},[390],{"type":58,"value":391},"the operation and phase: launch, initialization, allocation, communication, synchronization, collective, finalization, or process teardown.",{"type":52,"tag":68,"props":393,"children":394},{},[395],{"type":58,"value":396},"Ask only for missing facts that distinguish plausible causes. Never fabricate a version, topology, environment setting, or reproduction result.",{"type":52,"tag":231,"props":398,"children":400},{"id":399},"diagnose-conservatively",[401],{"type":58,"value":402},"Diagnose Conservatively",{"type":52,"tag":68,"props":404,"children":405},{},[406,408,412],{"type":58,"value":407},"Read ",{"type":52,"tag":129,"props":409,"children":410},{"href":131},[411],{"type":58,"value":134},{"type":58,"value":413}," for every diagnosis after the logging gate or pre-start exception. Use its topic index to select the relevant section, then search exact log signatures within that section before falling back to semantic matches.",{"type":52,"tag":68,"props":415,"children":416},{},[417],{"type":58,"value":418},"Classify the result as exactly one of:",{"type":52,"tag":243,"props":420,"children":421},{},[422,432,442,452],{"type":52,"tag":84,"props":423,"children":424},{},[425,430],{"type":52,"tag":158,"props":426,"children":427},{},[428],{"type":58,"value":429},"Confirmed bundled knowledge-base match",{"type":58,"value":431},": the signature and required conditions match an entry.",{"type":52,"tag":84,"props":433,"children":434},{},[435,440],{"type":52,"tag":158,"props":436,"children":437},{},[438],{"type":58,"value":439},"Version-specific known issue or later-release fix",{"type":58,"value":441},": official release notes establish applicability to the detected version.",{"type":52,"tag":84,"props":443,"children":444},{},[445,450],{"type":52,"tag":158,"props":446,"children":447},{},[448],{"type":58,"value":449},"Likely NVSHMEM diagnosis not in the knowledge base",{"type":58,"value":451},": evidence points into NVSHMEM, but no bundled or documented issue matches.",{"type":52,"tag":84,"props":453,"children":454},{},[455,460],{"type":52,"tag":158,"props":456,"children":457},{},[458],{"type":58,"value":459},"Likely external issue",{"type":58,"value":461},": evidence points to the application, CUDA, launcher, loader, network, driver, scheduler, container, or platform configuration.",{"type":52,"tag":68,"props":463,"children":464},{},[465,467,473],{"type":58,"value":466},"Use class 1 whenever a bundled entry is confirmed, even when that entry assigns ownership to an external component; state ownership separately in ",{"type":52,"tag":100,"props":468,"children":470},{"className":469},[],[471],{"type":58,"value":472},"Diagnosis",{"type":58,"value":474},". Use class 4 only when no bundled or versioned documented issue matches. For partial signature matches, use class 3 or 4 rather than calling the match confirmed. Treat a similar release-note symptom as a lead, not proof that the user has the same defect. A release note that says only that a bug was fixed establishes the fix release, not every affected earlier release. Describe an older installed version as predating a candidate fix unless official evidence explicitly establishes that it is affected or a controlled upgrade test confirms the match.",{"type":52,"tag":68,"props":476,"children":477},{},[478,480,485,487,493],{"type":58,"value":479},"Invoke ",{"type":52,"tag":100,"props":481,"children":483},{"className":482},[],[484],{"type":58,"value":105},{"type":58,"value":486}," with the complete detected version before asserting environment-variable behavior, requirements, limitations, known issues, fixes, or upgrade guidance. Ask for the complete version when logs do not provide it and a version claim matters. When the sibling skill is not registered, read and follow ",{"type":52,"tag":100,"props":488,"children":490},{"className":489},[],[491],{"type":58,"value":492},"..\u002Fnvshmem-docs\u002FSKILL.md",{"type":58,"value":123},{"type":52,"tag":68,"props":495,"children":496},{},[497],{"type":58,"value":498},"Before recommending a target upgrade, verify that release's compatibility and known issues against the collected CUDA, driver, GPU, CPU, NCCL, bootstrap, and transport context. If that context is incomplete, present the release as a candidate test and name the missing compatibility checks instead of calling it suitable.",{"type":52,"tag":68,"props":500,"children":501},{},[502],{"type":58,"value":503},"Separate:",{"type":52,"tag":80,"props":505,"children":506},{},[507,517,527,537],{"type":52,"tag":84,"props":508,"children":509},{},[510,515],{"type":52,"tag":158,"props":511,"children":512},{},[513],{"type":58,"value":514},"Observation",{"type":58,"value":516},": what the command and logs directly show.",{"type":52,"tag":84,"props":518,"children":519},{},[520,525],{"type":52,"tag":158,"props":521,"children":522},{},[523],{"type":58,"value":524},"Documentation evidence",{"type":58,"value":526},": the matched FAQ, environment-variable entry, limitation, known issue, or fixed issue.",{"type":52,"tag":84,"props":528,"children":529},{},[530,535],{"type":52,"tag":158,"props":531,"children":532},{},[533],{"type":58,"value":534},"Inference",{"type":58,"value":536},": the most likely explanation and its confidence.",{"type":52,"tag":84,"props":538,"children":539},{},[540,545],{"type":52,"tag":158,"props":541,"children":542},{},[543],{"type":58,"value":544},"Verification",{"type":58,"value":546},": the smallest safe test that can confirm or reject the inference.",{"type":52,"tag":68,"props":548,"children":549},{},[550,552,558],{"type":58,"value":551},"If no entry matches, say exactly: ",{"type":52,"tag":100,"props":553,"children":555},{"className":554},[],[556],{"type":58,"value":557},"No matching entry was found in the bundled NVSHMEM troubleshooting knowledge base.",{"type":58,"value":559}," Then provide a likely diagnosis, confidence, competing explanations, and targeted next checks.",{"type":52,"tag":68,"props":561,"children":562},{},[563,565,570,572,577],{"type":58,"value":564},"For a likely external issue, identify the likely owner and give concrete next checks. Do not stop at “not NVSHMEM.” Route installation and build failures to ",{"type":52,"tag":100,"props":566,"children":568},{"className":567},[],[569],{"type":58,"value":113},{"type":58,"value":571},"; route performance-only regressions to an available NVSHMEM performance skill, or to ",{"type":52,"tag":100,"props":573,"children":575},{"className":574},[],[576],{"type":58,"value":105},{"type":58,"value":578}," when none is available.",{"type":52,"tag":68,"props":580,"children":581},{},[582],{"type":58,"value":583},"Do not perform privileged platform changes. For IMEX, driver, fabric-manager, device-node, kernel-module, NIC, or scheduler configuration, provide read-only checks and tell the user what to ask the administrator to verify.",{"type":52,"tag":231,"props":585,"children":587},{"id":586},"present-the-diagnosis",[588],{"type":58,"value":589},"Present the Diagnosis",{"type":52,"tag":68,"props":591,"children":592},{},[593],{"type":58,"value":594},"Return these headings:",{"type":52,"tag":80,"props":596,"children":597},{},[598,607,615,624,633],{"type":52,"tag":84,"props":599,"children":600},{},[601],{"type":52,"tag":100,"props":602,"children":604},{"className":603},[],[605],{"type":58,"value":606},"Evidence Collected",{"type":52,"tag":84,"props":608,"children":609},{},[610],{"type":52,"tag":100,"props":611,"children":613},{"className":612},[],[614],{"type":58,"value":472},{"type":52,"tag":84,"props":616,"children":617},{},[618],{"type":52,"tag":100,"props":619,"children":621},{"className":620},[],[622],{"type":58,"value":623},"Knowledge-Base Match",{"type":52,"tag":84,"props":625,"children":626},{},[627],{"type":52,"tag":100,"props":628,"children":630},{"className":629},[],[631],{"type":58,"value":632},"Version Applicability",{"type":52,"tag":84,"props":634,"children":635},{},[636],{"type":52,"tag":100,"props":637,"children":639},{"className":638},[],[640],{"type":58,"value":641},"Next Actions",{"type":52,"tag":68,"props":643,"children":644},{},[645],{"type":58,"value":646},"Include direct official source links for documentation-backed claims. Keep commands copyable and mark placeholders. Prefer one discriminating test at a time over a long generic checklist.",{"type":52,"tag":231,"props":648,"children":650},{"id":649},"prepare-a-bug-report-or-question",[651],{"type":58,"value":652},"Prepare a Bug Report or Question",{"type":52,"tag":68,"props":654,"children":655},{},[656],{"type":58,"value":657},"Offer a report when the issue remains unresolved, is likely an NVSHMEM defect, or the user explicitly requests one. Do not routinely offer escalation for a resolved local configuration problem or a clearly external failure.",{"type":52,"tag":68,"props":659,"children":660},{},[661],{"type":58,"value":662},"When INFO-level NVSHMEM logs are absent, explicitly ask the user to reproduce the failure before support submission with:",{"type":52,"tag":281,"props":664,"children":666},{"className":665,"code":285,"language":58,"meta":286},[284],[667],{"type":52,"tag":100,"props":668,"children":669},{"__ignoreMap":286},[670],{"type":58,"value":285},{"type":52,"tag":68,"props":672,"children":673},{},[674],{"type":58,"value":675},"Preserve the executable, launcher, PE and node counts, binding, transport, container, and workload arguments. Ask for the exact repeated command, complete stdout\u002Fstderr from every PE, and all generated log files. A draft may still be generated when the user asks to preserve the investigation, but label it incomplete and include this rerun as a required pre-submission step; do not present it as support-ready.",{"type":52,"tag":68,"props":677,"children":678},{},[679,681,685,687,693,695,701,703,709],{"type":58,"value":680},"When preparing one, read ",{"type":52,"tag":129,"props":682,"children":683},{"href":139},[684],{"type":58,"value":142},{"type":58,"value":686}," and fill it from collected evidence. Mirror the ",{"type":52,"tag":100,"props":688,"children":690},{"className":689},[],[691],{"type":58,"value":692},"NVSHMEM issue or bug",{"type":58,"value":694}," GitHub form: start the title with ",{"type":52,"tag":100,"props":696,"children":698},{"className":697},[],[699],{"type":58,"value":700},"[Issue]:",{"type":58,"value":702},", use its exact field headings, and select exactly one supported impact value. Preserve exact commands and the first causal error. Use ",{"type":52,"tag":100,"props":704,"children":706},{"className":705},[],[707],{"type":58,"value":708},"Not provided",{"type":58,"value":710}," for missing facts and list those gaps; never invent them. Remind the user to redact credentials, tokens, internal hostnames and IPs, and user-identifying data before sharing logs.",{"type":52,"tag":68,"props":712,"children":713},{},[714],{"type":58,"value":715},"Return the Markdown in a fenced block so it can be pasted into GitHub, NVONLINE, or email. Do not require or directly submit through a connector.",{"type":52,"tag":61,"props":717,"children":718},{"id":46},[719],{"type":58,"value":720},"Troubleshooting",{"type":52,"tag":68,"props":722,"children":723},{},[724],{"type":58,"value":725},"Use these patterns when the investigation stalls:",{"type":52,"tag":727,"props":728,"children":729},"table",{},[730,754],{"type":52,"tag":731,"props":732,"children":733},"thead",{},[734],{"type":52,"tag":735,"props":736,"children":737},"tr",{},[738,744,749],{"type":52,"tag":739,"props":740,"children":741},"th",{},[742],{"type":58,"value":743},"Error or signal",{"type":52,"tag":739,"props":745,"children":746},{},[747],{"type":58,"value":748},"Likely cause",{"type":52,"tag":739,"props":750,"children":751},{},[752],{"type":58,"value":753},"Solution",{"type":52,"tag":755,"props":756,"children":757},"tbody",{},[758,777,795,813,831],{"type":52,"tag":735,"props":759,"children":760},{},[761,767,772],{"type":52,"tag":762,"props":763,"children":764},"td",{},[765],{"type":58,"value":766},"No INFO records appear after setting debug variables",{"type":52,"tag":762,"props":768,"children":769},{},[770],{"type":58,"value":771},"The launcher did not propagate the environment, or the process failed before NVSHMEM started",{"type":52,"tag":762,"props":773,"children":774},{},[775],{"type":58,"value":776},"Verify propagation with version-matched launcher syntax. Use the pre-start exception only when the emitting component proves NVSHMEM did not start.",{"type":52,"tag":735,"props":778,"children":779},{},[780,785,790],{"type":52,"tag":762,"props":781,"children":782},{},[783],{"type":58,"value":784},"Logs show only peer aborts or launcher cleanup",{"type":52,"tag":762,"props":786,"children":787},{},[788],{"type":58,"value":789},"The visible lines are cascading consequences",{"type":52,"tag":762,"props":791,"children":792},{},[793],{"type":58,"value":794},"Compare every PE log and use the earliest causal or divergent line, including preceding context.",{"type":52,"tag":735,"props":796,"children":797},{},[798,803,808],{"type":52,"tag":762,"props":799,"children":800},{},[801],{"type":58,"value":802},"A knowledge-base signature matches but its required conditions do not",{"type":52,"tag":762,"props":804,"children":805},{},[806],{"type":58,"value":807},"The match is only a hypothesis",{"type":52,"tag":762,"props":809,"children":810},{},[811],{"type":58,"value":812},"Classify the issue as likely NVSHMEM or likely external, state the mismatch, and run one discriminating test.",{"type":52,"tag":735,"props":814,"children":815},{},[816,821,826],{"type":52,"tag":762,"props":817,"children":818},{},[819],{"type":58,"value":820},"The NVSHMEM version is missing or live documentation is unavailable",{"type":52,"tag":762,"props":822,"children":823},{},[824],{"type":58,"value":825},"Version applicability cannot be verified",{"type":52,"tag":762,"props":827,"children":828},{},[829],{"type":58,"value":830},"Ask for the complete version and label version-sensitive guidance unverified; do not infer applicability from the snapshot.",{"type":52,"tag":735,"props":832,"children":833},{},[834,839,844],{"type":52,"tag":762,"props":835,"children":836},{},[837],{"type":58,"value":838},"INFO logging changes or removes the failure",{"type":52,"tag":762,"props":840,"children":841},{},[842],{"type":58,"value":843},"The failure may be timing-sensitive",{"type":52,"tag":762,"props":845,"children":846},{},[847],{"type":58,"value":848},"Preserve both commands and outcomes, repeat enough trials to characterize frequency, and do not call the logging change a fix.",{"type":52,"tag":61,"props":850,"children":852},{"id":851},"examples",[853],{"type":58,"value":854},"Examples",{"type":52,"tag":80,"props":856,"children":857},{},[858,868,878,903,928],{"type":52,"tag":84,"props":859,"children":860},{},[861,866],{"type":52,"tag":158,"props":862,"children":863},{},[864],{"type":58,"value":865},"Missing evidence:",{"type":58,"value":867}," Given only “NVSHMEM hangs during init,” ask for the exact launch stanza and complete output from every PE, then stop.",{"type":52,"tag":84,"props":869,"children":870},{},[871,876],{"type":52,"tag":158,"props":872,"children":873},{},[874],{"type":58,"value":875},"Missing INFO logs:",{"type":58,"value":877}," Given a complete command and non-INFO output from an in-process failure, request the same run with the three debug settings and all generated logs; preserve scale and arguments.",{"type":52,"tag":84,"props":879,"children":880},{},[881,886,888,894,896,901],{"type":52,"tag":158,"props":882,"children":883},{},[884],{"type":58,"value":885},"Pre-start failure:",{"type":58,"value":887}," Given ",{"type":52,"tag":100,"props":889,"children":891},{"className":890},[],[892],{"type":58,"value":893},"error while loading shared libraries",{"type":58,"value":895}," before any PE enters NVSHMEM, state why INFO logging cannot help, identify the loader as the emitting component, and route an NVSHMEM-library installation problem to ",{"type":52,"tag":100,"props":897,"children":899},{"className":898},[],[900],{"type":58,"value":113},{"type":58,"value":902},".",{"type":52,"tag":84,"props":904,"children":905},{},[906,911,912,918,920,926],{"type":52,"tag":158,"props":907,"children":908},{},[909],{"type":58,"value":910},"Confirmed bundled match:",{"type":58,"value":887},{"type":52,"tag":100,"props":913,"children":915},{"className":914},[],[916],{"type":58,"value":917},"No more teams available",{"type":58,"value":919}," plus matching team-usage conditions, classify it as a confirmed bundled match, verify ",{"type":52,"tag":100,"props":921,"children":923},{"className":922},[],[924],{"type":58,"value":925},"NVSHMEM_MAX_TEAMS",{"type":58,"value":927}," for the complete version, and propose one minimal rerun.",{"type":52,"tag":84,"props":929,"children":930},{},[931,936],{"type":52,"tag":158,"props":932,"children":933},{},[934],{"type":58,"value":935},"No match:",{"type":58,"value":937}," State the required no-match sentence, separate observation from inference, give one discriminating test, and offer a filled report only if the issue remains unresolved or looks like an NVSHMEM defect.",{"items":939,"total":1028},[940,954,968,982,994,1006,1018],{"slug":941,"name":941,"fn":942,"description":943,"org":944,"tags":945,"stars":23,"repoUrl":24,"updatedAt":953},"nvshmem-collect-performance-data","collect NVSHMEM performance metrics","Collect and package NVSHMEM put\u002Fget bandwidth, latency, and other perftest results with system and topology evidence for performance sanity checks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[946,949,950],{"name":947,"slug":948,"type":15},"Monitoring","monitoring",{"name":9,"slug":8,"type":15},{"name":951,"slug":952,"type":15},"Performance","performance","2026-08-31T09:19:00.917887",{"slug":955,"name":955,"fn":956,"description":957,"org":958,"tags":959,"stars":23,"repoUrl":24,"updatedAt":967},"nvshmem-configure-nic-pe-mapping","configure NVSHMEM NIC-to-PE mappings","Recommend NVSHMEM NIC-to-PE mappings and environment exports. Use for HCA selection, multi-NIC configuration, or topology-based mapping diagnostics.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[960,963,966],{"name":961,"slug":962,"type":15},"Configuration","configuration",{"name":964,"slug":965,"type":15},"Networking","networking",{"name":9,"slug":8,"type":15},"2026-08-31T09:19:06.10294",{"slug":969,"name":969,"fn":970,"description":971,"org":972,"tags":973,"stars":23,"repoUrl":24,"updatedAt":981},"nvshmem-docs","fetch official NVSHMEM documentation","Find version-aware official NVSHMEM and NVSHMEM4Py documentation for releases, installation, APIs, runtime settings, transports, containers, and troubleshooting.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[974,977,978],{"name":975,"slug":976,"type":15},"Documentation","documentation",{"name":9,"slug":8,"type":15},{"name":979,"slug":980,"type":15},"Reference","reference","2026-08-31T09:19:00.214841",{"slug":983,"name":983,"fn":984,"description":985,"org":986,"tags":987,"stars":23,"repoUrl":24,"updatedAt":993},"nvshmem-enable-tma","optimize NVSHMEM CUDA kernels for TMA","Prepare or review NVSHMEM CUDA kernels for TMA SMEM registration and direct-SMEM transfers. Do not use for unrelated CUDA tuning.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[988,991,992],{"name":989,"slug":990,"type":15},"Engineering","engineering",{"name":9,"slug":8,"type":15},{"name":951,"slug":952,"type":15},"2026-08-31T09:18:50.449676",{"slug":995,"name":995,"fn":996,"description":997,"org":998,"tags":999,"stars":23,"repoUrl":24,"updatedAt":1005},"nvshmem-get-started","onboard developers to NVSHMEM","Guide NVSHMEM beginners through fit assessment, mental models, first C\u002FC++ or Python NVSHMEM programs, compilation, launching, and next steps. Use for onboarding.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1000,1001,1002],{"name":975,"slug":976,"type":15},{"name":9,"slug":8,"type":15},{"name":1003,"slug":1004,"type":15},"Onboarding","onboarding","2026-08-31T09:18:55.48573",{"slug":1007,"name":1007,"fn":1008,"description":1009,"org":1010,"tags":1011,"stars":23,"repoUrl":24,"updatedAt":1017},"nvshmem-install","install and validate NVSHMEM","Plan and validate NVSHMEM and NVSHMEM4Py installations. Use for package, container, or source deployments.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1012,1015,1016],{"name":1013,"slug":1014,"type":15},"Deployment","deployment",{"name":989,"slug":990,"type":15},{"name":9,"slug":8,"type":15},"2026-08-31T09:18:50.105521",{"slug":1019,"name":1019,"fn":1020,"description":1021,"org":1022,"tags":1023,"stars":23,"repoUrl":24,"updatedAt":1027},"nvshmem-select-remote-transport","select NVSHMEM remote transport configurations","Select an NVSHMEM remote transport from target system and kernel evidence. Use for inter-node selection, compatibility checks, or configuration.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1024,1025,1026],{"name":961,"slug":962,"type":15},{"name":9,"slug":8,"type":15},{"name":951,"slug":952,"type":15},"2026-08-31T09:18:59.493722",9,{"items":1030,"total":1182},[1031,1047,1062,1073,1085,1097,1110,1124,1137,1148,1162,1171],{"slug":1032,"name":1032,"fn":1033,"description":1034,"org":1035,"tags":1036,"stars":1044,"repoUrl":1045,"updatedAt":1046},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1037,1038,1041],{"name":975,"slug":976,"type":15},{"name":1039,"slug":1040,"type":15},"MCP","mcp",{"name":1042,"slug":1043,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-08-25T03:29:57.273192",{"slug":1048,"name":1048,"fn":1049,"description":1050,"org":1051,"tags":1052,"stars":1059,"repoUrl":1060,"updatedAt":1061},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1053,1056,1057],{"name":1054,"slug":1055,"type":15},"Containers","containers",{"name":1013,"slug":1014,"type":15},{"name":1058,"slug":33,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":1063,"name":1063,"fn":1064,"description":1065,"org":1066,"tags":1067,"stars":1059,"repoUrl":1060,"updatedAt":1072},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1068,1071],{"name":1069,"slug":1070,"type":15},"CI\u002FCD","ci-cd",{"name":1013,"slug":1014,"type":15},"2026-07-14T05:25:59.97109",{"slug":1074,"name":1074,"fn":1075,"description":1076,"org":1077,"tags":1078,"stars":1059,"repoUrl":1060,"updatedAt":1084},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1079,1080,1081],{"name":1069,"slug":1070,"type":15},{"name":1013,"slug":1014,"type":15},{"name":1082,"slug":1083,"type":15},"GitHub","github","2026-08-28T14:38:16.959248",{"slug":1086,"name":1086,"fn":1087,"description":1088,"org":1089,"tags":1090,"stars":1059,"repoUrl":1060,"updatedAt":1096},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1091,1092,1093],{"name":21,"slug":22,"type":15},{"name":1082,"slug":1083,"type":15},{"name":1094,"slug":1095,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1098,"name":1098,"fn":1099,"description":1100,"org":1101,"tags":1102,"stars":1059,"repoUrl":1060,"updatedAt":1109},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1103,1106],{"name":1104,"slug":1105,"type":15},"Best Practices","best-practices",{"name":1107,"slug":1108,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1111,"name":1111,"fn":1112,"description":1113,"org":1114,"tags":1115,"stars":1059,"repoUrl":1060,"updatedAt":1123},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel, including the mechanical steps for transferring an existing pretrain_gpt.py launch script.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1116,1119,1122],{"name":1117,"slug":1118,"type":15},"Machine Learning","machine-learning",{"name":1120,"slug":1121,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-08-21T03:36:57.470256",{"slug":1125,"name":1125,"fn":1126,"description":1127,"org":1128,"tags":1129,"stars":1059,"repoUrl":1060,"updatedAt":1136},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1130,1133],{"name":1131,"slug":1132,"type":15},"QA","qa",{"name":1134,"slug":1135,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":1138,"name":1138,"fn":1139,"description":1140,"org":1141,"tags":1142,"stars":1059,"repoUrl":1060,"updatedAt":1147},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1143,1144],{"name":1013,"slug":1014,"type":15},{"name":1145,"slug":1146,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":1149,"name":1149,"fn":1150,"description":1151,"org":1152,"tags":1153,"stars":1059,"repoUrl":1060,"updatedAt":1161},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1154,1157,1158],{"name":1155,"slug":1156,"type":15},"Code Review","code-review",{"name":1082,"slug":1083,"type":15},{"name":1159,"slug":1160,"type":15},"Pull Requests","pull-requests","2026-08-25T03:29:16.211287",{"slug":1163,"name":1163,"fn":1164,"description":1165,"org":1166,"tags":1167,"stars":1059,"repoUrl":1060,"updatedAt":1170},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1168,1169],{"name":1131,"slug":1132,"type":15},{"name":1134,"slug":1135,"type":15},"2026-07-14T05:25:54.928983",{"slug":1172,"name":1172,"fn":1173,"description":1174,"org":1175,"tags":1176,"stars":1059,"repoUrl":1060,"updatedAt":1181},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1177,1180],{"name":1178,"slug":1179,"type":15},"Automation","automation",{"name":1069,"slug":1070,"type":15},"2026-07-30T05:29:03.275638",563]