[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-fr-analysis":3,"mdc-wd5mar-key":34,"related-org-nvidia-fr-analysis":976,"related-repo-nvidia-fr-analysis":1134},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"fr-analysis","analyze NCCL flight-recorder dumps","Analyze PyTorch NCCL flight-recorder (FR) dumps to identify collective operation hangs and isolate the responsible ranks using CollectiveAnalyzer. Use when a distributed training job hangs due to an NCCL collective timeout and FR dump files are available. Detects the wavefront process group where collectives diverge and returns the root-cause suspect ranks.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":17,"slug":18,"type":15},"Analysis","analysis",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Debugging","debugging",308,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnvidia-resiliency-ext","2026-07-14T05:35:13.703056",null,56,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"NVIDIA Resiliency Extension is a python package for framework developers and users to implement fault-tolerant features. It improves the effective training time by minimizing the downtime due to failures and interruptions.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnvidia-resiliency-ext\u002Ftree\u002FHEAD\u002Fsrc\u002Fnvidia_resiliency_ext\u002Fskills\u002Fnvrx-attr\u002Ffr-analysis","---\nname: fr-analysis\ndescription: >\n  Analyze PyTorch NCCL flight-recorder (FR) dumps to identify collective operation hangs and\n  isolate the responsible ranks using CollectiveAnalyzer. Use when a distributed training job\n  hangs due to an NCCL collective timeout and FR dump files are available. Detects the wavefront\n  process group where collectives diverge and returns the root-cause suspect ranks.\ncompatibility: Requires PyTorch NCCL FR dumps (TORCH_NCCL_TRACE_BUFFER_SIZE > 0 must be set during training). LLM_API_KEY and langchain-openai are required only when using --llm-analyze.\nmetadata:\n  entry-point: CollectiveAnalyzer\n  script: scripts\u002Ffr_attribution.py\n---\n\n# Skill: fr_analysis\n\nAnalyze PyTorch NCCL flight-recorder (FR) dumps to identify the collective operation hang\nand isolate the ranks responsible, using `CollectiveAnalyzer`.\n\n**Script:** [`scripts\u002Ffr_attribution.py`](.\u002Fscripts\u002Ffr_attribution.py) → `attribution\u002Ftrace_analyzer\u002Ffr_attribution.py`\n\n---\n\n## What it does\n\n1. Loads all FR dump files matching a glob pattern under `--fr-path`.\n2. Parses each dump into `Collective` records (op type, ranks, process group, timing, state).\n3. Groups collectives by process group and sequence ID across ranks to detect mismatches.\n4. Identifies the **wavefront** — the process group boundary where collectives diverge — and\n   returns the missing ranks at that boundary as the root-cause suspects.\n5. Optionally runs an LLM pass (`--llm-analyze`) over the structured findings for a\n   human-readable summary.\n\n---\n\n## CLI\n\n```bash\npython scripts\u002Ffr_attribution.py \\\n    --fr-path \u002Fpath\u002Fto\u002Ffr_dumps\u002F \\\n    [-p \"_dump_*\"] \\\n    [--verbose] \\\n    [--health-check] \\\n    [--llm-analyze] \\\n    [--model MODEL] \\\n    [--debug]\n```\n\n| Flag | Default | Description |\n|------|---------|-------------|\n| `--fr-path` | required | Path to a directory (or single file) containing FR dump files |\n| `--pattern`, `-p` | `_dump_*` | Glob pattern for dump files within `--fr-path` |\n| `--verbose`, `-v` | off | Print detailed per-rank collective tables |\n| `--health-check`, `-c` | off | Include node health check results in output |\n| `--llm-analyze`, `-l` | off | Pass structured findings to the LLM for a narrative summary |\n| `--model`, `-m` | `nvidia\u002Fnemotron-3-super-120b-a12b` | LLM model (only used with `--llm-analyze`) |\n| `--debug` | off | Convert binary trace files to JSON for inspection |\n\n---\n\n## Programmatic API\n\n```python\nfrom nvidia_resiliency_ext.attribution.trace_analyzer.fr_attribution import CollectiveAnalyzer\n\nanalyzer = CollectiveAnalyzer({\n    \"fr_path\": \"\u002Fpath\u002Fto\u002Ffr_dumps\u002F\",\n    \"pattern\": \"_dump_*\",\n    \"verbose\": False,\n    \"health_check\": False,\n    \"llm_analyze\": False,\n    \"model\": \"nvidia\u002Fnemotron-3-super-120b-a12b\",\n})\nresults = analyzer.run_sync({\n    \"fr_path\": \"\u002Fpath\u002Fto\u002Ffr_dumps\u002F\",\n})\n# results: tuple[FRAnalysisResult | str, AttributionState]\n```\n\n---\n\n## Output\n\nReturns `(result, AttributionState)` where `result` is the FR analysis table and describes:\n\n- The selected wavefront\u002Ffront process group\n- **Missing ranks** at that process group (root-cause suspects)\n- Per-rank collective status tables (when `--verbose`)\n- Node health summary (when `--health-check`)\n- LLM narrative (when `--llm-analyze`)\n\n`AttributionState.STOP` indicates the hang is unrecoverable; `CONTINUE` indicates the job\nmay be restartable after isolating the identified ranks.\n\n---\n\n## Dump file formats\n\n| Format | Notes |\n|--------|-------|\n| `_dump_*` files | PyTorch FR dump prefix pattern used by the feedback loop |\n| Binary pickle \u002F JSON payloads | Detected automatically; use `--debug` to convert binary traces to JSON |\n\nFR dumps are typically written to the directory specified by `TORCH_NCCL_DEBUG_INFO_TEMP_FILE`\nor triggered automatically on NCCL timeout.\n\n---\n\n## Prerequisites\n\n- FR dump files produced by PyTorch NCCL (set `TORCH_NCCL_TRACE_BUFFER_SIZE` > 0)\n- `LLM_API_KEY` required only when using `--llm-analyze`\n- `langchain-openai` required only when using `--llm-analyze`\n- `FR_DEBUG=1` env var enables verbose debug logging in the script\n",{"data":35,"body":40},{"name":4,"description":6,"compatibility":36,"metadata":37},"Requires PyTorch NCCL FR dumps (TORCH_NCCL_TRACE_BUFFER_SIZE > 0 must be set during training). LLM_API_KEY and langchain-openai are required only when using --llm-analyze.",{"entry-point":38,"script":39},"CollectiveAnalyzer","scripts\u002Ffr_attribution.py",{"type":41,"children":42},"root",[43,52,66,95,99,106,166,169,175,351,586,589,595,720,723,729,750,802,821,824,830,890,903,906,912,970],{"type":44,"tag":45,"props":46,"children":48},"element","h1",{"id":47},"skill-fr_analysis",[49],{"type":50,"value":51},"text","Skill: fr_analysis",{"type":44,"tag":53,"props":54,"children":55},"p",{},[56,58,64],{"type":50,"value":57},"Analyze PyTorch NCCL flight-recorder (FR) dumps to identify the collective operation hang\nand isolate the ranks responsible, using ",{"type":44,"tag":59,"props":60,"children":62},"code",{"className":61},[],[63],{"type":50,"value":38},{"type":50,"value":65},".",{"type":44,"tag":53,"props":67,"children":68},{},[69,75,77,87,89],{"type":44,"tag":70,"props":71,"children":72},"strong",{},[73],{"type":50,"value":74},"Script:",{"type":50,"value":76}," ",{"type":44,"tag":78,"props":79,"children":81},"a",{"href":80},".\u002Fscripts\u002Ffr_attribution.py",[82],{"type":44,"tag":59,"props":83,"children":85},{"className":84},[],[86],{"type":50,"value":39},{"type":50,"value":88}," → ",{"type":44,"tag":59,"props":90,"children":92},{"className":91},[],[93],{"type":50,"value":94},"attribution\u002Ftrace_analyzer\u002Ffr_attribution.py",{"type":44,"tag":96,"props":97,"children":98},"hr",{},[],{"type":44,"tag":100,"props":101,"children":103},"h2",{"id":102},"what-it-does",[104],{"type":50,"value":105},"What it does",{"type":44,"tag":107,"props":108,"children":109},"ol",{},[110,123,136,141,153],{"type":44,"tag":111,"props":112,"children":113},"li",{},[114,116,122],{"type":50,"value":115},"Loads all FR dump files matching a glob pattern under ",{"type":44,"tag":59,"props":117,"children":119},{"className":118},[],[120],{"type":50,"value":121},"--fr-path",{"type":50,"value":65},{"type":44,"tag":111,"props":124,"children":125},{},[126,128,134],{"type":50,"value":127},"Parses each dump into ",{"type":44,"tag":59,"props":129,"children":131},{"className":130},[],[132],{"type":50,"value":133},"Collective",{"type":50,"value":135}," records (op type, ranks, process group, timing, state).",{"type":44,"tag":111,"props":137,"children":138},{},[139],{"type":50,"value":140},"Groups collectives by process group and sequence ID across ranks to detect mismatches.",{"type":44,"tag":111,"props":142,"children":143},{},[144,146,151],{"type":50,"value":145},"Identifies the ",{"type":44,"tag":70,"props":147,"children":148},{},[149],{"type":50,"value":150},"wavefront",{"type":50,"value":152}," — the process group boundary where collectives diverge — and\nreturns the missing ranks at that boundary as the root-cause suspects.",{"type":44,"tag":111,"props":154,"children":155},{},[156,158,164],{"type":50,"value":157},"Optionally runs an LLM pass (",{"type":44,"tag":59,"props":159,"children":161},{"className":160},[],[162],{"type":50,"value":163},"--llm-analyze",{"type":50,"value":165},") over the structured findings for a\nhuman-readable summary.",{"type":44,"tag":96,"props":167,"children":168},{},[],{"type":44,"tag":100,"props":170,"children":172},{"id":171},"cli",[173],{"type":50,"value":174},"CLI",{"type":44,"tag":176,"props":177,"children":182},"pre",{"className":178,"code":179,"language":180,"meta":181,"style":181},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","python scripts\u002Ffr_attribution.py \\\n    --fr-path \u002Fpath\u002Fto\u002Ffr_dumps\u002F \\\n    [-p \"_dump_*\"] \\\n    [--verbose] \\\n    [--health-check] \\\n    [--llm-analyze] \\\n    [--model MODEL] \\\n    [--debug]\n","bash","",[183],{"type":44,"tag":59,"props":184,"children":185},{"__ignoreMap":181},[186,210,228,261,270,292,312,333],{"type":44,"tag":187,"props":188,"children":191},"span",{"class":189,"line":190},"line",1,[192,198,204],{"type":44,"tag":187,"props":193,"children":195},{"style":194},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[196],{"type":50,"value":197},"python",{"type":44,"tag":187,"props":199,"children":201},{"style":200},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[202],{"type":50,"value":203}," scripts\u002Ffr_attribution.py",{"type":44,"tag":187,"props":205,"children":207},{"style":206},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[208],{"type":50,"value":209}," \\\n",{"type":44,"tag":187,"props":211,"children":213},{"class":189,"line":212},2,[214,219,224],{"type":44,"tag":187,"props":215,"children":216},{"style":200},[217],{"type":50,"value":218},"    --fr-path",{"type":44,"tag":187,"props":220,"children":221},{"style":200},[222],{"type":50,"value":223}," \u002Fpath\u002Fto\u002Ffr_dumps\u002F",{"type":44,"tag":187,"props":225,"children":226},{"style":206},[227],{"type":50,"value":209},{"type":44,"tag":187,"props":229,"children":231},{"class":189,"line":230},3,[232,237,243,248,252,257],{"type":44,"tag":187,"props":233,"children":234},{"style":206},[235],{"type":50,"value":236},"    [-p ",{"type":44,"tag":187,"props":238,"children":240},{"style":239},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[241],{"type":50,"value":242},"\"",{"type":44,"tag":187,"props":244,"children":245},{"style":200},[246],{"type":50,"value":247},"_dump_*",{"type":44,"tag":187,"props":249,"children":250},{"style":239},[251],{"type":50,"value":242},{"type":44,"tag":187,"props":253,"children":254},{"style":200},[255],{"type":50,"value":256},"]",{"type":44,"tag":187,"props":258,"children":259},{"style":206},[260],{"type":50,"value":209},{"type":44,"tag":187,"props":262,"children":264},{"class":189,"line":263},4,[265],{"type":44,"tag":187,"props":266,"children":267},{"style":206},[268],{"type":50,"value":269},"    [--verbose] \\\n",{"type":44,"tag":187,"props":271,"children":273},{"class":189,"line":272},5,[274,279,284,288],{"type":44,"tag":187,"props":275,"children":276},{"style":239},[277],{"type":50,"value":278},"    [",{"type":44,"tag":187,"props":280,"children":281},{"style":206},[282],{"type":50,"value":283},"--health-check",{"type":44,"tag":187,"props":285,"children":286},{"style":239},[287],{"type":50,"value":256},{"type":44,"tag":187,"props":289,"children":290},{"style":206},[291],{"type":50,"value":209},{"type":44,"tag":187,"props":293,"children":295},{"class":189,"line":294},6,[296,300,304,308],{"type":44,"tag":187,"props":297,"children":298},{"style":239},[299],{"type":50,"value":278},{"type":44,"tag":187,"props":301,"children":302},{"style":206},[303],{"type":50,"value":163},{"type":44,"tag":187,"props":305,"children":306},{"style":239},[307],{"type":50,"value":256},{"type":44,"tag":187,"props":309,"children":310},{"style":206},[311],{"type":50,"value":209},{"type":44,"tag":187,"props":313,"children":315},{"class":189,"line":314},7,[316,320,325,329],{"type":44,"tag":187,"props":317,"children":318},{"style":239},[319],{"type":50,"value":278},{"type":44,"tag":187,"props":321,"children":322},{"style":206},[323],{"type":50,"value":324},"--model MODEL",{"type":44,"tag":187,"props":326,"children":327},{"style":239},[328],{"type":50,"value":256},{"type":44,"tag":187,"props":330,"children":331},{"style":206},[332],{"type":50,"value":209},{"type":44,"tag":187,"props":334,"children":336},{"class":189,"line":335},8,[337,341,346],{"type":44,"tag":187,"props":338,"children":339},{"style":239},[340],{"type":50,"value":278},{"type":44,"tag":187,"props":342,"children":343},{"style":206},[344],{"type":50,"value":345},"--debug",{"type":44,"tag":187,"props":347,"children":348},{"style":239},[349],{"type":50,"value":350},"]\n",{"type":44,"tag":352,"props":353,"children":354},"table",{},[355,379],{"type":44,"tag":356,"props":357,"children":358},"thead",{},[359],{"type":44,"tag":360,"props":361,"children":362},"tr",{},[363,369,374],{"type":44,"tag":364,"props":365,"children":366},"th",{},[367],{"type":50,"value":368},"Flag",{"type":44,"tag":364,"props":370,"children":371},{},[372],{"type":50,"value":373},"Default",{"type":44,"tag":364,"props":375,"children":376},{},[377],{"type":50,"value":378},"Description",{"type":44,"tag":380,"props":381,"children":382},"tbody",{},[383,405,443,472,499,526,566],{"type":44,"tag":360,"props":384,"children":385},{},[386,395,400],{"type":44,"tag":387,"props":388,"children":389},"td",{},[390],{"type":44,"tag":59,"props":391,"children":393},{"className":392},[],[394],{"type":50,"value":121},{"type":44,"tag":387,"props":396,"children":397},{},[398],{"type":50,"value":399},"required",{"type":44,"tag":387,"props":401,"children":402},{},[403],{"type":50,"value":404},"Path to a directory (or single file) containing FR dump files",{"type":44,"tag":360,"props":406,"children":407},{},[408,425,433],{"type":44,"tag":387,"props":409,"children":410},{},[411,417,419],{"type":44,"tag":59,"props":412,"children":414},{"className":413},[],[415],{"type":50,"value":416},"--pattern",{"type":50,"value":418},", ",{"type":44,"tag":59,"props":420,"children":422},{"className":421},[],[423],{"type":50,"value":424},"-p",{"type":44,"tag":387,"props":426,"children":427},{},[428],{"type":44,"tag":59,"props":429,"children":431},{"className":430},[],[432],{"type":50,"value":247},{"type":44,"tag":387,"props":434,"children":435},{},[436,438],{"type":50,"value":437},"Glob pattern for dump files within ",{"type":44,"tag":59,"props":439,"children":441},{"className":440},[],[442],{"type":50,"value":121},{"type":44,"tag":360,"props":444,"children":445},{},[446,462,467],{"type":44,"tag":387,"props":447,"children":448},{},[449,455,456],{"type":44,"tag":59,"props":450,"children":452},{"className":451},[],[453],{"type":50,"value":454},"--verbose",{"type":50,"value":418},{"type":44,"tag":59,"props":457,"children":459},{"className":458},[],[460],{"type":50,"value":461},"-v",{"type":44,"tag":387,"props":463,"children":464},{},[465],{"type":50,"value":466},"off",{"type":44,"tag":387,"props":468,"children":469},{},[470],{"type":50,"value":471},"Print detailed per-rank collective tables",{"type":44,"tag":360,"props":473,"children":474},{},[475,490,494],{"type":44,"tag":387,"props":476,"children":477},{},[478,483,484],{"type":44,"tag":59,"props":479,"children":481},{"className":480},[],[482],{"type":50,"value":283},{"type":50,"value":418},{"type":44,"tag":59,"props":485,"children":487},{"className":486},[],[488],{"type":50,"value":489},"-c",{"type":44,"tag":387,"props":491,"children":492},{},[493],{"type":50,"value":466},{"type":44,"tag":387,"props":495,"children":496},{},[497],{"type":50,"value":498},"Include node health check results in output",{"type":44,"tag":360,"props":500,"children":501},{},[502,517,521],{"type":44,"tag":387,"props":503,"children":504},{},[505,510,511],{"type":44,"tag":59,"props":506,"children":508},{"className":507},[],[509],{"type":50,"value":163},{"type":50,"value":418},{"type":44,"tag":59,"props":512,"children":514},{"className":513},[],[515],{"type":50,"value":516},"-l",{"type":44,"tag":387,"props":518,"children":519},{},[520],{"type":50,"value":466},{"type":44,"tag":387,"props":522,"children":523},{},[524],{"type":50,"value":525},"Pass structured findings to the LLM for a narrative summary",{"type":44,"tag":360,"props":527,"children":528},{},[529,545,554],{"type":44,"tag":387,"props":530,"children":531},{},[532,538,539],{"type":44,"tag":59,"props":533,"children":535},{"className":534},[],[536],{"type":50,"value":537},"--model",{"type":50,"value":418},{"type":44,"tag":59,"props":540,"children":542},{"className":541},[],[543],{"type":50,"value":544},"-m",{"type":44,"tag":387,"props":546,"children":547},{},[548],{"type":44,"tag":59,"props":549,"children":551},{"className":550},[],[552],{"type":50,"value":553},"nvidia\u002Fnemotron-3-super-120b-a12b",{"type":44,"tag":387,"props":555,"children":556},{},[557,559,564],{"type":50,"value":558},"LLM model (only used with ",{"type":44,"tag":59,"props":560,"children":562},{"className":561},[],[563],{"type":50,"value":163},{"type":50,"value":565},")",{"type":44,"tag":360,"props":567,"children":568},{},[569,577,581],{"type":44,"tag":387,"props":570,"children":571},{},[572],{"type":44,"tag":59,"props":573,"children":575},{"className":574},[],[576],{"type":50,"value":345},{"type":44,"tag":387,"props":578,"children":579},{},[580],{"type":50,"value":466},{"type":44,"tag":387,"props":582,"children":583},{},[584],{"type":50,"value":585},"Convert binary trace files to JSON for inspection",{"type":44,"tag":96,"props":587,"children":588},{},[],{"type":44,"tag":100,"props":590,"children":592},{"id":591},"programmatic-api",[593],{"type":50,"value":594},"Programmatic API",{"type":44,"tag":176,"props":596,"children":599},{"className":597,"code":598,"language":197,"meta":181,"style":181},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from nvidia_resiliency_ext.attribution.trace_analyzer.fr_attribution import CollectiveAnalyzer\n\nanalyzer = CollectiveAnalyzer({\n    \"fr_path\": \"\u002Fpath\u002Fto\u002Ffr_dumps\u002F\",\n    \"pattern\": \"_dump_*\",\n    \"verbose\": False,\n    \"health_check\": False,\n    \"llm_analyze\": False,\n    \"model\": \"nvidia\u002Fnemotron-3-super-120b-a12b\",\n})\nresults = analyzer.run_sync({\n    \"fr_path\": \"\u002Fpath\u002Fto\u002Ffr_dumps\u002F\",\n})\n# results: tuple[FRAnalysisResult | str, AttributionState]\n",[600],{"type":44,"tag":59,"props":601,"children":602},{"__ignoreMap":181},[603,611,620,628,636,644,652,660,668,677,686,695,703,711],{"type":44,"tag":187,"props":604,"children":605},{"class":189,"line":190},[606],{"type":44,"tag":187,"props":607,"children":608},{},[609],{"type":50,"value":610},"from nvidia_resiliency_ext.attribution.trace_analyzer.fr_attribution import CollectiveAnalyzer\n",{"type":44,"tag":187,"props":612,"children":613},{"class":189,"line":212},[614],{"type":44,"tag":187,"props":615,"children":617},{"emptyLinePlaceholder":616},true,[618],{"type":50,"value":619},"\n",{"type":44,"tag":187,"props":621,"children":622},{"class":189,"line":230},[623],{"type":44,"tag":187,"props":624,"children":625},{},[626],{"type":50,"value":627},"analyzer = CollectiveAnalyzer({\n",{"type":44,"tag":187,"props":629,"children":630},{"class":189,"line":263},[631],{"type":44,"tag":187,"props":632,"children":633},{},[634],{"type":50,"value":635},"    \"fr_path\": \"\u002Fpath\u002Fto\u002Ffr_dumps\u002F\",\n",{"type":44,"tag":187,"props":637,"children":638},{"class":189,"line":272},[639],{"type":44,"tag":187,"props":640,"children":641},{},[642],{"type":50,"value":643},"    \"pattern\": \"_dump_*\",\n",{"type":44,"tag":187,"props":645,"children":646},{"class":189,"line":294},[647],{"type":44,"tag":187,"props":648,"children":649},{},[650],{"type":50,"value":651},"    \"verbose\": False,\n",{"type":44,"tag":187,"props":653,"children":654},{"class":189,"line":314},[655],{"type":44,"tag":187,"props":656,"children":657},{},[658],{"type":50,"value":659},"    \"health_check\": False,\n",{"type":44,"tag":187,"props":661,"children":662},{"class":189,"line":335},[663],{"type":44,"tag":187,"props":664,"children":665},{},[666],{"type":50,"value":667},"    \"llm_analyze\": False,\n",{"type":44,"tag":187,"props":669,"children":671},{"class":189,"line":670},9,[672],{"type":44,"tag":187,"props":673,"children":674},{},[675],{"type":50,"value":676},"    \"model\": \"nvidia\u002Fnemotron-3-super-120b-a12b\",\n",{"type":44,"tag":187,"props":678,"children":680},{"class":189,"line":679},10,[681],{"type":44,"tag":187,"props":682,"children":683},{},[684],{"type":50,"value":685},"})\n",{"type":44,"tag":187,"props":687,"children":689},{"class":189,"line":688},11,[690],{"type":44,"tag":187,"props":691,"children":692},{},[693],{"type":50,"value":694},"results = analyzer.run_sync({\n",{"type":44,"tag":187,"props":696,"children":698},{"class":189,"line":697},12,[699],{"type":44,"tag":187,"props":700,"children":701},{},[702],{"type":50,"value":635},{"type":44,"tag":187,"props":704,"children":706},{"class":189,"line":705},13,[707],{"type":44,"tag":187,"props":708,"children":709},{},[710],{"type":50,"value":685},{"type":44,"tag":187,"props":712,"children":714},{"class":189,"line":713},14,[715],{"type":44,"tag":187,"props":716,"children":717},{},[718],{"type":50,"value":719},"# results: tuple[FRAnalysisResult | str, AttributionState]\n",{"type":44,"tag":96,"props":721,"children":722},{},[],{"type":44,"tag":100,"props":724,"children":726},{"id":725},"output",[727],{"type":50,"value":728},"Output",{"type":44,"tag":53,"props":730,"children":731},{},[732,734,740,742,748],{"type":50,"value":733},"Returns ",{"type":44,"tag":59,"props":735,"children":737},{"className":736},[],[738],{"type":50,"value":739},"(result, AttributionState)",{"type":50,"value":741}," where ",{"type":44,"tag":59,"props":743,"children":745},{"className":744},[],[746],{"type":50,"value":747},"result",{"type":50,"value":749}," is the FR analysis table and describes:",{"type":44,"tag":751,"props":752,"children":753},"ul",{},[754,759,769,780,791],{"type":44,"tag":111,"props":755,"children":756},{},[757],{"type":50,"value":758},"The selected wavefront\u002Ffront process group",{"type":44,"tag":111,"props":760,"children":761},{},[762,767],{"type":44,"tag":70,"props":763,"children":764},{},[765],{"type":50,"value":766},"Missing ranks",{"type":50,"value":768}," at that process group (root-cause suspects)",{"type":44,"tag":111,"props":770,"children":771},{},[772,774,779],{"type":50,"value":773},"Per-rank collective status tables (when ",{"type":44,"tag":59,"props":775,"children":777},{"className":776},[],[778],{"type":50,"value":454},{"type":50,"value":565},{"type":44,"tag":111,"props":781,"children":782},{},[783,785,790],{"type":50,"value":784},"Node health summary (when ",{"type":44,"tag":59,"props":786,"children":788},{"className":787},[],[789],{"type":50,"value":283},{"type":50,"value":565},{"type":44,"tag":111,"props":792,"children":793},{},[794,796,801],{"type":50,"value":795},"LLM narrative (when ",{"type":44,"tag":59,"props":797,"children":799},{"className":798},[],[800],{"type":50,"value":163},{"type":50,"value":565},{"type":44,"tag":53,"props":803,"children":804},{},[805,811,813,819],{"type":44,"tag":59,"props":806,"children":808},{"className":807},[],[809],{"type":50,"value":810},"AttributionState.STOP",{"type":50,"value":812}," indicates the hang is unrecoverable; ",{"type":44,"tag":59,"props":814,"children":816},{"className":815},[],[817],{"type":50,"value":818},"CONTINUE",{"type":50,"value":820}," indicates the job\nmay be restartable after isolating the identified ranks.",{"type":44,"tag":96,"props":822,"children":823},{},[],{"type":44,"tag":100,"props":825,"children":827},{"id":826},"dump-file-formats",[828],{"type":50,"value":829},"Dump file formats",{"type":44,"tag":352,"props":831,"children":832},{},[833,849],{"type":44,"tag":356,"props":834,"children":835},{},[836],{"type":44,"tag":360,"props":837,"children":838},{},[839,844],{"type":44,"tag":364,"props":840,"children":841},{},[842],{"type":50,"value":843},"Format",{"type":44,"tag":364,"props":845,"children":846},{},[847],{"type":50,"value":848},"Notes",{"type":44,"tag":380,"props":850,"children":851},{},[852,870],{"type":44,"tag":360,"props":853,"children":854},{},[855,865],{"type":44,"tag":387,"props":856,"children":857},{},[858,863],{"type":44,"tag":59,"props":859,"children":861},{"className":860},[],[862],{"type":50,"value":247},{"type":50,"value":864}," files",{"type":44,"tag":387,"props":866,"children":867},{},[868],{"type":50,"value":869},"PyTorch FR dump prefix pattern used by the feedback loop",{"type":44,"tag":360,"props":871,"children":872},{},[873,878],{"type":44,"tag":387,"props":874,"children":875},{},[876],{"type":50,"value":877},"Binary pickle \u002F JSON payloads",{"type":44,"tag":387,"props":879,"children":880},{},[881,883,888],{"type":50,"value":882},"Detected automatically; use ",{"type":44,"tag":59,"props":884,"children":886},{"className":885},[],[887],{"type":50,"value":345},{"type":50,"value":889}," to convert binary traces to JSON",{"type":44,"tag":53,"props":891,"children":892},{},[893,895,901],{"type":50,"value":894},"FR dumps are typically written to the directory specified by ",{"type":44,"tag":59,"props":896,"children":898},{"className":897},[],[899],{"type":50,"value":900},"TORCH_NCCL_DEBUG_INFO_TEMP_FILE",{"type":50,"value":902},"\nor triggered automatically on NCCL timeout.",{"type":44,"tag":96,"props":904,"children":905},{},[],{"type":44,"tag":100,"props":907,"children":909},{"id":908},"prerequisites",[910],{"type":50,"value":911},"Prerequisites",{"type":44,"tag":751,"props":913,"children":914},{},[915,928,944,959],{"type":44,"tag":111,"props":916,"children":917},{},[918,920,926],{"type":50,"value":919},"FR dump files produced by PyTorch NCCL (set ",{"type":44,"tag":59,"props":921,"children":923},{"className":922},[],[924],{"type":50,"value":925},"TORCH_NCCL_TRACE_BUFFER_SIZE",{"type":50,"value":927}," > 0)",{"type":44,"tag":111,"props":929,"children":930},{},[931,937,939],{"type":44,"tag":59,"props":932,"children":934},{"className":933},[],[935],{"type":50,"value":936},"LLM_API_KEY",{"type":50,"value":938}," required only when using ",{"type":44,"tag":59,"props":940,"children":942},{"className":941},[],[943],{"type":50,"value":163},{"type":44,"tag":111,"props":945,"children":946},{},[947,953,954],{"type":44,"tag":59,"props":948,"children":950},{"className":949},[],[951],{"type":50,"value":952},"langchain-openai",{"type":50,"value":938},{"type":44,"tag":59,"props":955,"children":957},{"className":956},[],[958],{"type":50,"value":163},{"type":44,"tag":111,"props":960,"children":961},{},[962,968],{"type":44,"tag":59,"props":963,"children":965},{"className":964},[],[966],{"type":50,"value":967},"FR_DEBUG=1",{"type":50,"value":969}," env var enables verbose debug logging in the script",{"type":44,"tag":971,"props":972,"children":973},"style",{},[974],{"type":50,"value":975},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":977,"total":1133},[978,996,1013,1024,1036,1048,1061,1075,1088,1099,1113,1122],{"slug":979,"name":979,"fn":980,"description":981,"org":982,"tags":983,"stars":993,"repoUrl":994,"updatedAt":995},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[984,987,990],{"name":985,"slug":986,"type":15},"Documentation","documentation",{"name":988,"slug":989,"type":15},"MCP","mcp",{"name":991,"slug":992,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":997,"name":997,"fn":998,"description":999,"org":1000,"tags":1001,"stars":1010,"repoUrl":1011,"updatedAt":1012},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1002,1005,1008],{"name":1003,"slug":1004,"type":15},"Containers","containers",{"name":1006,"slug":1007,"type":15},"Deployment","deployment",{"name":1009,"slug":197,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":1014,"name":1014,"fn":1015,"description":1016,"org":1017,"tags":1018,"stars":1010,"repoUrl":1011,"updatedAt":1023},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1019,1022],{"name":1020,"slug":1021,"type":15},"CI\u002FCD","ci-cd",{"name":1006,"slug":1007,"type":15},"2026-07-14T05:25:59.97109",{"slug":1025,"name":1025,"fn":1026,"description":1027,"org":1028,"tags":1029,"stars":1010,"repoUrl":1011,"updatedAt":1035},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1030,1031,1032],{"name":1020,"slug":1021,"type":15},{"name":1006,"slug":1007,"type":15},{"name":1033,"slug":1034,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":1037,"name":1037,"fn":1038,"description":1039,"org":1040,"tags":1041,"stars":1010,"repoUrl":1011,"updatedAt":1047},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1042,1043,1044],{"name":21,"slug":22,"type":15},{"name":1033,"slug":1034,"type":15},{"name":1045,"slug":1046,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1049,"name":1049,"fn":1050,"description":1051,"org":1052,"tags":1053,"stars":1010,"repoUrl":1011,"updatedAt":1060},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1054,1057],{"name":1055,"slug":1056,"type":15},"Best Practices","best-practices",{"name":1058,"slug":1059,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1062,"name":1062,"fn":1063,"description":1064,"org":1065,"tags":1066,"stars":1010,"repoUrl":1011,"updatedAt":1074},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1067,1070,1073],{"name":1068,"slug":1069,"type":15},"Machine Learning","machine-learning",{"name":1071,"slug":1072,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":1076,"name":1076,"fn":1077,"description":1078,"org":1079,"tags":1080,"stars":1010,"repoUrl":1011,"updatedAt":1087},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1081,1084],{"name":1082,"slug":1083,"type":15},"QA","qa",{"name":1085,"slug":1086,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":1089,"name":1089,"fn":1090,"description":1091,"org":1092,"tags":1093,"stars":1010,"repoUrl":1011,"updatedAt":1098},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1094,1095],{"name":1006,"slug":1007,"type":15},{"name":1096,"slug":1097,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":1100,"name":1100,"fn":1101,"description":1102,"org":1103,"tags":1104,"stars":1010,"repoUrl":1011,"updatedAt":1112},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1105,1108,1109],{"name":1106,"slug":1107,"type":15},"Code Review","code-review",{"name":1033,"slug":1034,"type":15},{"name":1110,"slug":1111,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":1114,"name":1114,"fn":1115,"description":1116,"org":1117,"tags":1118,"stars":1010,"repoUrl":1011,"updatedAt":1121},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1119,1120],{"name":1082,"slug":1083,"type":15},{"name":1085,"slug":1086,"type":15},"2026-07-14T05:25:54.928983",{"slug":1123,"name":1123,"fn":1124,"description":1125,"org":1126,"tags":1127,"stars":1010,"repoUrl":1011,"updatedAt":1132},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1128,1131],{"name":1129,"slug":1130,"type":15},"Automation","automation",{"name":1020,"slug":1021,"type":15},"2026-07-30T05:29:03.275638",496,{"items":1135,"total":263},[1136,1151,1158,1171],{"slug":1137,"name":1137,"fn":1138,"description":1139,"org":1140,"tags":1141,"stars":23,"repoUrl":24,"updatedAt":1150},"fault-injection-loop","run closed-loop fault injection benchmarks","Closed-loop fault injection and attribution accuracy benchmark. Draws from a prioritized pool of (fault_type, rank, iter, nodes) experiments and submits them 2 at a time via sbatch — waiting for each pair to finish before submitting the next — to bound filesystem load. GPU-related faults are front-loaded in the pool. After all jobs complete, runs \u002Flog-analysis and \u002Ffr-analysis on every experiment, scores attribution vs. ground truth, aggregates gaps, and iterates on attribution modules to close them.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1142,1145,1146,1149],{"name":1143,"slug":1144,"type":15},"Benchmarking","benchmarking",{"name":9,"slug":8,"type":15},{"name":1147,"slug":1148,"type":15},"Simulation","simulation",{"name":1085,"slug":1086,"type":15},"2026-07-14T05:35:12.409884",{"slug":4,"name":4,"fn":5,"description":6,"org":1152,"tags":1153,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1154,1155,1156,1157],{"name":17,"slug":18,"type":15},{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"slug":1159,"name":1159,"fn":1160,"description":1161,"org":1162,"tags":1163,"stars":23,"repoUrl":24,"updatedAt":1170},"log-analysis","analyze SLURM training job logs","Analyze a SLURM job log file for failure root-cause attribution and restart decisions using NVRxLogAnalyzer. Use when you have a SLURM training job log and need to determine why the job failed and whether it should be restarted. Performs per-cycle chunking, fast-path pattern matching, and LLM-based classification.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1164,1165,1166,1169],{"name":17,"slug":18,"type":15},{"name":21,"slug":22,"type":15},{"name":1167,"slug":1168,"type":15},"Logs","logs",{"name":9,"slug":8,"type":15},"2026-07-14T05:35:14.94051",{"slug":1172,"name":1172,"fn":1173,"description":1174,"org":1175,"tags":1176,"stars":23,"repoUrl":24,"updatedAt":1183},"nvrx-attr","analyze attribution quality for Megatron-LM","Orchestration layer over nvidia_resiliency_ext attribution modules. Provides log-analysis, fr-analysis, and a Megatron-LM-oriented fault-injection feedback loop for benchmarking attribution quality on SLURM workloads.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1177,1180,1181,1182],{"name":1178,"slug":1179,"type":15},"AI Infrastructure","ai-infrastructure",{"name":1143,"slug":1144,"type":15},{"name":1167,"slug":1168,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:36:19.02075"]