[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nemo-mbridge-perf-cuda-graphs":3,"mdc--rmzyj7-key":31,"related-repo-nvidia-nemo-mbridge-perf-cuda-graphs":2470,"related-org-nvidia-nemo-mbridge-perf-cuda-graphs":2574},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":20,"repoUrl":21,"updatedAt":22,"license":23,"forks":24,"topics":25,"repo":26,"sourceUrl":29,"mdContent":30},"nemo-mbridge-perf-cuda-graphs","validate CUDA graph capture in Megatron Bridge","Validate and use CUDA graph capture in Megatron Bridge, including local full-iteration graphs and Transformer Engine scoped graphs for attention, MLP, and MoE modules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,17],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":9,"slug":8,"type":15},{"name":18,"slug":19,"type":15},"Engineering","engineering",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:29:24.440456","Apache-2.0",281,[],{"repoUrl":21,"stars":20,"forks":24,"topics":27,"description":28},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fnemo-mbridge-perf-cuda-graphs","---\nname: nemo-mbridge-perf-cuda-graphs\ndescription: Validate and use CUDA graph capture in Megatron Bridge, including local full-iteration graphs and Transformer Engine scoped graphs for attention, MLP, and MoE modules.\nlicense: Apache-2.0\nwhen_to_use: Reducing host-driver overhead via CUDA graphs, or tracing a crash or regression to a CUDA graph config change; 'cuda_graph_impl', 'full iteration graph', 'TE scoped graph', 'graphed callables', 'CUDA graph capture'.\n---\n\n# CUDA Graphs\n\nStable documentation: @docs\u002Ftraining\u002Fcuda-graphs.md\nCard: @skills\u002Fnemo-mbridge-perf-cuda-graphs\u002Fcard.yaml\n\n\u003C!-- NVSkills CI refresh: 2026-06-15. No instruction changes. -->\n\n## What It Is\n\nCUDA graphs capture GPU operations once and replay them with minimal\nhost-driver overhead. Bridge supports two implementations:\n\n| `cuda_graph_impl` | Mechanism | Scope support |\n|---|---|---|\n| `\"local\"` | MCore `FullCudaGraphWrapper` wrapping entire fwd+bwd | `full_iteration` |\n| `\"transformer_engine\"` | TE `make_graphed_callables()` per layer | `attn`, `mlp`, `moe`, `moe_router`, `moe_preprocess`, `mamba` |\n\n## Quick Decision\n\nStart with TE-scoped graphs for most training workloads, then verify replay\ntiming against eager on the same dispatcher, layout, and container:\n\n- dense models: `attn`, then optionally `mlp`\n- dropless MoE: `attn moe_router moe_preprocess`\n- VLMs: the same dropless-MoE scope, but only after the real-data path is stable\n\nUse `local` + `full_iteration` only when you specifically want full-iteration\ncapture and can satisfy the tighter constraints.\n\nFor recompute-heavy workloads:\n\n- TE-scoped graphs pair naturally with selective recompute\n- full recompute usually pushes you toward `local` full-iteration graphs or away\n  from graphs entirely\n\nRelated docs:\n\n- @docs\u002Ftraining\u002Fcuda-graphs.md\n- @docs\u002Ftraining\u002Factivation-recomputation.md\n\n## Enablement\n\n### Local full-iteration graph\n\n```python\ncfg.model.cuda_graph_impl = \"local\"\ncfg.model.cuda_graph_scope = [\"full_iteration\"]\ncfg.model.cuda_graph_warmup_steps = 3\ncfg.model.use_te_rng_tracker = True\ncfg.rng.te_rng_tracker = True\ncfg.rerun_state_machine.check_for_nan_in_loss = False\ncfg.ddp.check_for_nan_in_grad = False\n```\n\n### TE scoped graph (dense model)\n\n```python\ncfg.model.cuda_graph_impl = \"transformer_engine\"\ncfg.model.cuda_graph_scope = [\"attn\"]           # or [\"attn\", \"mlp\"]\ncfg.model.cuda_graph_warmup_steps = 3\ncfg.model.use_te_rng_tracker = True\ncfg.rng.te_rng_tracker = True\n```\n\n### TE scoped graph (MoE model)\n\n```python\ncfg.model.cuda_graph_impl = \"transformer_engine\"\ncfg.model.cuda_graph_scope = [\"attn\", \"moe_router\", \"moe_preprocess\"]\ncfg.model.cuda_graph_warmup_steps = 3\ncfg.model.use_te_rng_tracker = True\ncfg.rng.te_rng_tracker = True\n```\n\n### Performance harness CLI\n\n```bash\nuv run python scripts\u002Fperformance\u002Frun_script.py \\\n  -m qwen \\\n  -mr qwen3_30b_a3b \\\n  --task pretrain \\\n  -g h100 \\\n  -c bf16 \\\n  -ng 16 \\\n  --cuda_graph_impl transformer_engine \\\n  --cuda_graph_scope attn,moe_router,moe_preprocess \\\n  ...\n```\n\nValid CLI values live in `scripts\u002Fperformance\u002Fargument_parser.py`:\n- `VALID_CUDA_GRAPH_IMPLS`: `[\"none\", \"local\", \"transformer_engine\"]`\n- `VALID_CUDA_GRAPH_SCOPES`: `[\"full_iteration\", \"attn\", \"mlp\", \"moe\", \"moe_router\", \"moe_preprocess\", \"mamba\"]`\n\nThe performance harness uses a comma-separated `--cuda_graph_scope` value and\nauto-enables `model.use_te_rng_tracker` plus `rng.te_rng_tracker` when\n`--cuda_graph_impl` is not `none`.\n\n### Required constraints\n\n- `use_te_rng_tracker = True` (enforced in `gpt_provider.py`)\n- `full_iteration` scope only with `cuda_graph_impl = \"local\"`\n- `full_iteration` scope requires `check_for_nan_in_loss = False`\n- Do not combine `moe` scope and `moe_router` scope\n- Tensor shapes must be static (fixed seq_length, fixed micro_batch_size)\n- MoE token-dropless routing limits graphable scope to dense modules\n- With `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`, set\n  `NCCL_GRAPH_REGISTER=0` (MCore enforces for local impl on arch \u003C sm_100;\n  TE impl asserts unconditionally)\n- CPU offloading is incompatible with CUDA graphs\n- `moe_preprocess` scope requires `moe_router` scope to also be set\n\n### Practical bring-up order\n\n1. Stabilize the eager run first.\n2. Fix sequence length and micro-batch size.\n3. Enable the narrowest useful graph scope.\n4. Confirm replay is active and memory is still acceptable.\n5. Compare eager against graph replay iterations after warmup and capture; do\n   not include the capture step in steady-state timing.\n6. Only then widen scope or combine with overlap features.\n\n## Code Anchors\n\n### Bridge config and validation\n\n```1524:1531:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py\n        # CUDA graph scope validation: check_for_nan_in_loss must be disabled with full_iteration graph\n        if self.model.cuda_graph_impl == \"local\" and CudaGraphScope.full_iteration in self.model.cuda_graph_scope:\n            assert not self.rerun_state_machine.check_for_nan_in_loss, (\n                \"check_for_nan_in_loss must be disabled when using full_iteration CUDA graph. \"\n                \"Set rerun_state_machine.check_for_nan_in_loss=False.\"\n            )\n        if self.model.cuda_graph_impl == \"none\":\n            self.model.cuda_graph_scope = []\n```\n\n### TE RNG tracker requirement\n\n```213:216:src\u002Fmegatron\u002Fbridge\u002Fmodels\u002Fgpt_provider.py\n        if self.cuda_graph_impl != \"none\":\n            assert getattr(self, \"use_te_rng_tracker\", False), (\n                \"Transformer engine's RNG tracker is required for cudagraphs, it can be \"\n                \"enabled with use_te_rng_tracker=True'.\"\n```\n\n### Graph creation and capture in training loop\n\n```231:255:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py\n    # Capture CUDA Graphs.\n    cuda_graph_helper = None\n    if model_config.cuda_graph_impl == \"transformer_engine\":\n        cuda_graph_helper = TECudaGraphHelper(...)\n    # ...\n    if config.model.cuda_graph_impl == \"local\" and CudaGraphScope.full_iteration in config.model.cuda_graph_scope:\n        forward_backward_func = FullCudaGraphWrapper(\n            forward_backward_func, cuda_graph_warmup_steps=config.model.cuda_graph_warmup_steps\n        )\n```\n\n### TE graph capture after warmup\n\n```338:350:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py\n        # Capture CUDA Graphs after warmup.\n        if (\n            model_config.cuda_graph_impl == \"transformer_engine\"\n            and cuda_graph_helper is not None\n            and not cuda_graph_helper.graphs_created()\n            and global_state.train_state.step - start_iteration == model_config.cuda_graph_warmup_steps\n        ):\n            if model_config.cuda_graph_warmup_steps > 0 and should_toggle_forward_pre_hook:\n                disable_forward_pre_hook(model, param_sync=False)\n            cuda_graph_helper.create_cudagraphs()\n            if model_config.cuda_graph_warmup_steps > 0 and should_toggle_forward_pre_hook:\n                enable_forward_pre_hook(model)\n                cuda_graph_helper.cuda_graph_set_manual_hooks()\n```\n\n### RNG initialization\n\n```199:206:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finitialize.py\n        _set_random_seed(\n            rng_config.seed,\n            rng_config.data_parallel_random_init,\n            rng_config.te_rng_tracker,\n            rng_config.inference_rng_tracker,\n            use_cudagraphable_rng=(model_config.cuda_graph_impl != \"none\"),\n            pg_collection=pg_collection,\n        )\n```\n\n### Delayed wgrad + CUDA graph interaction\n\n```522:555:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcomm_overlap.py\n            cuda_graph_scope = getattr(model_cfg, \"cuda_graph_scope\", []) or []\n            # ... scope parsing ...\n            if wgrad_in_graph_scope:\n                assert is_te_min_version(\"2.12.0\"), ...\n                assert model_cfg.gradient_accumulation_fusion, ...\n                if attn_scope_enabled:\n                    assert not model_cfg.add_bias_linear and not model_cfg.add_qkv_bias, ...\n```\n\n### Perf harness override helper\n\n```102:124:scripts\u002Fperformance\u002Futils\u002Foverrides.py\ndef _set_cuda_graph_overrides(\n    recipe, cuda_graph_impl=None, cuda_graph_scope=None\n):\n    # Sets impl, scope, and auto-enables te_rng_tracker\n```\n\n### Graph cleanup\n\n```1414:1441:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py\ndef _delete_cuda_graphs(cuda_graph_helper):\n    # Deletes FullCudaGraphWrapper and TE graph objects to free NCCL buffers\n```\n\n### MCore classes (in 3rdparty\u002FMegatron-LM)\n\n- `CudaGraphManager`: `megatron\u002Fcore\u002Ftransformer\u002Fcuda_graphs.py`\n- `TECudaGraphHelper`: `megatron\u002Fcore\u002Ftransformer\u002Fcuda_graphs.py`\n- `FullCudaGraphWrapper`: `megatron\u002Fcore\u002Ffull_cuda_graph.py`\n- `CudaGraphScope` enum: `megatron\u002Fcore\u002Ftransformer\u002Fenums.py`\n\n### Positive recipe anchors\n\n- `src\u002Fmegatron\u002Fbridge\u002Fperf_recipes\u002Fdeepseek\u002Fgb300\u002Fdeepseek_v3.py`\n- `src\u002Fmegatron\u002Fbridge\u002Fperf_recipes\u002Fqwen\u002Fgb300\u002Fqwen3_moe.py`\n- `src\u002Fmegatron\u002Fbridge\u002Fperf_recipes\u002Fgpt_oss\u002Fgb300\u002Fgpt_oss.py`\n\n### Tests\n\n| File | Coverage |\n|---|---|\n| `tests\u002Funit_tests\u002Ftraining\u002Ftest_config.py` | `full_iteration` NaN-check constraint |\n| `tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py` | `delay_wgrad` + CUDA graph interaction |\n| `tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_full_te_layer_autocast_spec.py` | TE autocast with CUDA graphs |\n| `tests\u002Ffunctional_tests\u002Ftest_groups\u002Frecipes\u002Ftest_llama_recipes_pretrain_cuda_graphs.py` | End-to-end local and TE graph smoke tests |\n| `tests\u002Funit_tests\u002Frecipes\u002Fkimi\u002Ftest_kimi_k2.py` | TE + CUDA graph recipe config |\n| `tests\u002Funit_tests\u002Frecipes\u002Fgpt\u002Ftest_gpt3_175b.py` | TE + CUDA graph recipe config |\n| `tests\u002Funit_tests\u002Frecipes\u002Fqwen_vl\u002Ftest_qwen25_vl_recipes.py` | VLM CUDA graph settings |\n\n## Pitfalls\n\n1. **TE RNG tracker is mandatory**: Setting `cuda_graph_impl` without\n   `use_te_rng_tracker=True` and `rng.te_rng_tracker=True` will assert\n   in the provider.\n\n2. **`full_iteration` requires NaN checks disabled**: The entire fwd+bwd is\n   captured, so loss-NaN checking cannot inspect intermediate values.\n\n3. **MoE scope restrictions**: `moe` scope and `moe_router` scope are\n   mutually exclusive. Token-dropless MoE can only graph `moe_router` and\n   `moe_preprocess`, not the full expert dispatch.\n\n4. **Memory overhead**: CUDA graphs pin all intermediate buffers for the\n   graph's lifetime (no memory reuse). TE scoped graphs add a few GB;\n   full-iteration graphs can increase peak memory by 1.5–2×. `PP > 1`\n   compounds overhead since each stage holds its own graph.\n\n5. **Delayed wgrad interaction**: When `delay_wgrad_compute=True` and\n   attention or MoE router is in `cuda_graph_scope`, additional constraints\n   apply: TE >= 2.12.0, `gradient_accumulation_fusion=True`, and no\n   attention bias.\n\n6. **Variable-length sequences break graphs**: Sequence lengths must be\n   constant across steps. Use padded packed sequences if packing is needed.\n\n7. **Graph cleanup is required**: CUDA graph objects hold NCCL buffer\n   references. Bridge handles this in `_delete_cuda_graphs()` at the end\n   of training, but early exits must call it explicitly.\n\n8. **Older GPU architectures**: On GPUs with compute capability \u003C 10.0\n   (pre-Blackwell), set `NCCL_GRAPH_REGISTER=0` when using\n   `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`. Enforced in MCore\n   `CudaGraphManager` (cuda_graphs.py:1428) and `TECudaGraphHelper`\n   (cuda_graphs.py:1697). The TE impl asserts unconditionally regardless\n   of arch.\n\n9. **CPU offloading incompatible**: CUDA graphs cannot be used with CPU\n   offloading. Enforced in MCore `transformer_config.py:1907`.\n\n10. **MoE recompute + moe_router scope**: MoE recompute is not supported\n    with `moe_router` CUDA graph scope when using `cuda_graph_impl =\n    \"transformer_engine\"`. Enforced in MCore `transformer_config.py:1977`.\n\n11. **Layer-level recompute requires `full_iteration` scope**: Using\n    `recompute_granularity=\"full\"` with `recompute_num_layers` (recompute N\n    whole transformer layers) is incompatible with TE-scoped graphs. MCore\n    calls this \"full\" granularity even though you're selecting how many\n    layers — the name refers to recomputing the full layer, not full model.\n    Any TE-scoped scope (`attn`, `mlp`, `moe_router`, etc.) will assert:\n    `AssertionError: full recompute is only supported with full iteration CUDA graph.`\n    This commonly hits FP8 configs that default to TE-scoped graphs (e.g.\n    `LLAMA3_70B_SFT_CONFIG_H100_FP8_CS_V1` uses `cuda_graph_impl=\n    \"transformer_engine\"`, `cuda_graph_scope=\"mlp\"`). Fix: use submodule\n    recompute (`recompute_granularity=\"selective\"` + `recompute_modules`),\n    disable CUDA graphs, or switch to `local` + `full_iteration`. Enforced\n    in MCore `transformer_config.py:2001-2005`. See also\n    @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md.\n\n12. **Benchmark numbers are workload-specific**: graph wins are usually real\n    when host overhead is visible, but the exact gain depends on batch shape,\n    PP depth, recompute, dispatcher backend, and whether the eager baseline was\n    already optimized.\n\n13. **A successful capture is not a speedup guarantee**: On 2026-05-18,\n    Qwen3 30B A3B H100 BF16 pretrain with the all-to-all dispatcher captured\n    TE-scoped `attn,moe_router,moe_preprocess` graphs successfully (`48`\n    graphable layers, about `6.9 s` capture time on rank 0), but replay\n    iterations 5-8 averaged `42.00 s` versus `41.36 s` for eager. Treat\n    scoped graphs as a bring-up candidate and validate on the target stack.\n\n## Verification\n\n### Unit tests\n\n```bash\nuv run python -m pytest \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_config.py -k \"cuda_graph\" \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py -k \"cuda_graph\" \\\n  tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_full_te_layer_autocast_spec.py -k \"cuda_graph\" -q\n```\n\n### Functional smoke test (requires GPU)\n\n```bash\nuv run python -m pytest \\\n  tests\u002Ffunctional_tests\u002Ftest_groups\u002Frecipes\u002Ftest_llama_recipes_pretrain_cuda_graphs.py -q\n```\n\n### Success criteria\n\n- Unit tests pass, covering config validation for both `local` and\n  `transformer_engine` implementations.\n- Functional test completes training steps with both CUDA graph\n  implementations.\n- No NCCL errors or illegal memory access in logs.\n",{"data":32,"body":34},{"name":4,"description":6,"license":23,"when_to_use":33},"Reducing host-driver overhead via CUDA graphs, or tracing a crash or regression to a CUDA graph config change; 'cuda_graph_impl', 'full iteration graph', 'TE scoped graph', 'graphed callables', 'CUDA graph capture'.",{"type":35,"children":36},"root",[37,46,52,59,64,206,212,217,255,275,280,300,305,318,324,331,407,413,457,463,506,512,700,713,749,794,800,925,931,965,971,977,1050,1056,1097,1103,1184,1190,1305,1311,1383,1389,1454,1460,1501,1507,1532,1538,1604,1610,1640,1646,1797,1803,2236,2242,2248,2374,2380,2426,2432,2464],{"type":38,"tag":39,"props":40,"children":42},"element","h1",{"id":41},"cuda-graphs",[43],{"type":44,"value":45},"text","CUDA Graphs",{"type":38,"tag":47,"props":48,"children":49},"p",{},[50],{"type":44,"value":51},"Stable documentation: @docs\u002Ftraining\u002Fcuda-graphs.md\nCard: @skills\u002Fnemo-mbridge-perf-cuda-graphs\u002Fcard.yaml",{"type":38,"tag":53,"props":54,"children":56},"h2",{"id":55},"what-it-is",[57],{"type":44,"value":58},"What It Is",{"type":38,"tag":47,"props":60,"children":61},{},[62],{"type":44,"value":63},"CUDA graphs capture GPU operations once and replay them with minimal\nhost-driver overhead. Bridge supports two implementations:",{"type":38,"tag":65,"props":66,"children":67},"table",{},[68,97],{"type":38,"tag":69,"props":70,"children":71},"thead",{},[72],{"type":38,"tag":73,"props":74,"children":75},"tr",{},[76,87,92],{"type":38,"tag":77,"props":78,"children":79},"th",{},[80],{"type":38,"tag":81,"props":82,"children":84},"code",{"className":83},[],[85],{"type":44,"value":86},"cuda_graph_impl",{"type":38,"tag":77,"props":88,"children":89},{},[90],{"type":44,"value":91},"Mechanism",{"type":38,"tag":77,"props":93,"children":94},{},[95],{"type":44,"value":96},"Scope support",{"type":38,"tag":98,"props":99,"children":100},"tbody",{},[101,136],{"type":38,"tag":73,"props":102,"children":103},{},[104,114,127],{"type":38,"tag":105,"props":106,"children":107},"td",{},[108],{"type":38,"tag":81,"props":109,"children":111},{"className":110},[],[112],{"type":44,"value":113},"\"local\"",{"type":38,"tag":105,"props":115,"children":116},{},[117,119,125],{"type":44,"value":118},"MCore ",{"type":38,"tag":81,"props":120,"children":122},{"className":121},[],[123],{"type":44,"value":124},"FullCudaGraphWrapper",{"type":44,"value":126}," wrapping entire fwd+bwd",{"type":38,"tag":105,"props":128,"children":129},{},[130],{"type":38,"tag":81,"props":131,"children":133},{"className":132},[],[134],{"type":44,"value":135},"full_iteration",{"type":38,"tag":73,"props":137,"children":138},{},[139,148,161],{"type":38,"tag":105,"props":140,"children":141},{},[142],{"type":38,"tag":81,"props":143,"children":145},{"className":144},[],[146],{"type":44,"value":147},"\"transformer_engine\"",{"type":38,"tag":105,"props":149,"children":150},{},[151,153,159],{"type":44,"value":152},"TE ",{"type":38,"tag":81,"props":154,"children":156},{"className":155},[],[157],{"type":44,"value":158},"make_graphed_callables()",{"type":44,"value":160}," per layer",{"type":38,"tag":105,"props":162,"children":163},{},[164,170,172,178,179,185,186,192,193,199,200],{"type":38,"tag":81,"props":165,"children":167},{"className":166},[],[168],{"type":44,"value":169},"attn",{"type":44,"value":171},", ",{"type":38,"tag":81,"props":173,"children":175},{"className":174},[],[176],{"type":44,"value":177},"mlp",{"type":44,"value":171},{"type":38,"tag":81,"props":180,"children":182},{"className":181},[],[183],{"type":44,"value":184},"moe",{"type":44,"value":171},{"type":38,"tag":81,"props":187,"children":189},{"className":188},[],[190],{"type":44,"value":191},"moe_router",{"type":44,"value":171},{"type":38,"tag":81,"props":194,"children":196},{"className":195},[],[197],{"type":44,"value":198},"moe_preprocess",{"type":44,"value":171},{"type":38,"tag":81,"props":201,"children":203},{"className":202},[],[204],{"type":44,"value":205},"mamba",{"type":38,"tag":53,"props":207,"children":209},{"id":208},"quick-decision",[210],{"type":44,"value":211},"Quick Decision",{"type":38,"tag":47,"props":213,"children":214},{},[215],{"type":44,"value":216},"Start with TE-scoped graphs for most training workloads, then verify replay\ntiming against eager on the same dispatcher, layout, and container:",{"type":38,"tag":218,"props":219,"children":220},"ul",{},[221,239,250],{"type":38,"tag":222,"props":223,"children":224},"li",{},[225,227,232,234],{"type":44,"value":226},"dense models: ",{"type":38,"tag":81,"props":228,"children":230},{"className":229},[],[231],{"type":44,"value":169},{"type":44,"value":233},", then optionally ",{"type":38,"tag":81,"props":235,"children":237},{"className":236},[],[238],{"type":44,"value":177},{"type":38,"tag":222,"props":240,"children":241},{},[242,244],{"type":44,"value":243},"dropless MoE: ",{"type":38,"tag":81,"props":245,"children":247},{"className":246},[],[248],{"type":44,"value":249},"attn moe_router moe_preprocess",{"type":38,"tag":222,"props":251,"children":252},{},[253],{"type":44,"value":254},"VLMs: the same dropless-MoE scope, but only after the real-data path is stable",{"type":38,"tag":47,"props":256,"children":257},{},[258,260,266,268,273],{"type":44,"value":259},"Use ",{"type":38,"tag":81,"props":261,"children":263},{"className":262},[],[264],{"type":44,"value":265},"local",{"type":44,"value":267}," + ",{"type":38,"tag":81,"props":269,"children":271},{"className":270},[],[272],{"type":44,"value":135},{"type":44,"value":274}," only when you specifically want full-iteration\ncapture and can satisfy the tighter constraints.",{"type":38,"tag":47,"props":276,"children":277},{},[278],{"type":44,"value":279},"For recompute-heavy workloads:",{"type":38,"tag":218,"props":281,"children":282},{},[283,288],{"type":38,"tag":222,"props":284,"children":285},{},[286],{"type":44,"value":287},"TE-scoped graphs pair naturally with selective recompute",{"type":38,"tag":222,"props":289,"children":290},{},[291,293,298],{"type":44,"value":292},"full recompute usually pushes you toward ",{"type":38,"tag":81,"props":294,"children":296},{"className":295},[],[297],{"type":44,"value":265},{"type":44,"value":299}," full-iteration graphs or away\nfrom graphs entirely",{"type":38,"tag":47,"props":301,"children":302},{},[303],{"type":44,"value":304},"Related docs:",{"type":38,"tag":218,"props":306,"children":307},{},[308,313],{"type":38,"tag":222,"props":309,"children":310},{},[311],{"type":44,"value":312},"@docs\u002Ftraining\u002Fcuda-graphs.md",{"type":38,"tag":222,"props":314,"children":315},{},[316],{"type":44,"value":317},"@docs\u002Ftraining\u002Factivation-recomputation.md",{"type":38,"tag":53,"props":319,"children":321},{"id":320},"enablement",[322],{"type":44,"value":323},"Enablement",{"type":38,"tag":325,"props":326,"children":328},"h3",{"id":327},"local-full-iteration-graph",[329],{"type":44,"value":330},"Local full-iteration graph",{"type":38,"tag":332,"props":333,"children":338},"pre",{"className":334,"code":335,"language":336,"meta":337,"style":337},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","cfg.model.cuda_graph_impl = \"local\"\ncfg.model.cuda_graph_scope = [\"full_iteration\"]\ncfg.model.cuda_graph_warmup_steps = 3\ncfg.model.use_te_rng_tracker = True\ncfg.rng.te_rng_tracker = True\ncfg.rerun_state_machine.check_for_nan_in_loss = False\ncfg.ddp.check_for_nan_in_grad = False\n","python","",[339],{"type":38,"tag":81,"props":340,"children":341},{"__ignoreMap":337},[342,353,362,371,380,389,398],{"type":38,"tag":343,"props":344,"children":347},"span",{"class":345,"line":346},"line",1,[348],{"type":38,"tag":343,"props":349,"children":350},{},[351],{"type":44,"value":352},"cfg.model.cuda_graph_impl = \"local\"\n",{"type":38,"tag":343,"props":354,"children":356},{"class":345,"line":355},2,[357],{"type":38,"tag":343,"props":358,"children":359},{},[360],{"type":44,"value":361},"cfg.model.cuda_graph_scope = [\"full_iteration\"]\n",{"type":38,"tag":343,"props":363,"children":365},{"class":345,"line":364},3,[366],{"type":38,"tag":343,"props":367,"children":368},{},[369],{"type":44,"value":370},"cfg.model.cuda_graph_warmup_steps = 3\n",{"type":38,"tag":343,"props":372,"children":374},{"class":345,"line":373},4,[375],{"type":38,"tag":343,"props":376,"children":377},{},[378],{"type":44,"value":379},"cfg.model.use_te_rng_tracker = True\n",{"type":38,"tag":343,"props":381,"children":383},{"class":345,"line":382},5,[384],{"type":38,"tag":343,"props":385,"children":386},{},[387],{"type":44,"value":388},"cfg.rng.te_rng_tracker = True\n",{"type":38,"tag":343,"props":390,"children":392},{"class":345,"line":391},6,[393],{"type":38,"tag":343,"props":394,"children":395},{},[396],{"type":44,"value":397},"cfg.rerun_state_machine.check_for_nan_in_loss = False\n",{"type":38,"tag":343,"props":399,"children":401},{"class":345,"line":400},7,[402],{"type":38,"tag":343,"props":403,"children":404},{},[405],{"type":44,"value":406},"cfg.ddp.check_for_nan_in_grad = False\n",{"type":38,"tag":325,"props":408,"children":410},{"id":409},"te-scoped-graph-dense-model",[411],{"type":44,"value":412},"TE scoped graph (dense model)",{"type":38,"tag":332,"props":414,"children":416},{"className":334,"code":415,"language":336,"meta":337,"style":337},"cfg.model.cuda_graph_impl = \"transformer_engine\"\ncfg.model.cuda_graph_scope = [\"attn\"]           # or [\"attn\", \"mlp\"]\ncfg.model.cuda_graph_warmup_steps = 3\ncfg.model.use_te_rng_tracker = True\ncfg.rng.te_rng_tracker = True\n",[417],{"type":38,"tag":81,"props":418,"children":419},{"__ignoreMap":337},[420,428,436,443,450],{"type":38,"tag":343,"props":421,"children":422},{"class":345,"line":346},[423],{"type":38,"tag":343,"props":424,"children":425},{},[426],{"type":44,"value":427},"cfg.model.cuda_graph_impl = \"transformer_engine\"\n",{"type":38,"tag":343,"props":429,"children":430},{"class":345,"line":355},[431],{"type":38,"tag":343,"props":432,"children":433},{},[434],{"type":44,"value":435},"cfg.model.cuda_graph_scope = [\"attn\"]           # or [\"attn\", \"mlp\"]\n",{"type":38,"tag":343,"props":437,"children":438},{"class":345,"line":364},[439],{"type":38,"tag":343,"props":440,"children":441},{},[442],{"type":44,"value":370},{"type":38,"tag":343,"props":444,"children":445},{"class":345,"line":373},[446],{"type":38,"tag":343,"props":447,"children":448},{},[449],{"type":44,"value":379},{"type":38,"tag":343,"props":451,"children":452},{"class":345,"line":382},[453],{"type":38,"tag":343,"props":454,"children":455},{},[456],{"type":44,"value":388},{"type":38,"tag":325,"props":458,"children":460},{"id":459},"te-scoped-graph-moe-model",[461],{"type":44,"value":462},"TE scoped graph (MoE model)",{"type":38,"tag":332,"props":464,"children":466},{"className":334,"code":465,"language":336,"meta":337,"style":337},"cfg.model.cuda_graph_impl = \"transformer_engine\"\ncfg.model.cuda_graph_scope = [\"attn\", \"moe_router\", \"moe_preprocess\"]\ncfg.model.cuda_graph_warmup_steps = 3\ncfg.model.use_te_rng_tracker = True\ncfg.rng.te_rng_tracker = True\n",[467],{"type":38,"tag":81,"props":468,"children":469},{"__ignoreMap":337},[470,477,485,492,499],{"type":38,"tag":343,"props":471,"children":472},{"class":345,"line":346},[473],{"type":38,"tag":343,"props":474,"children":475},{},[476],{"type":44,"value":427},{"type":38,"tag":343,"props":478,"children":479},{"class":345,"line":355},[480],{"type":38,"tag":343,"props":481,"children":482},{},[483],{"type":44,"value":484},"cfg.model.cuda_graph_scope = [\"attn\", \"moe_router\", \"moe_preprocess\"]\n",{"type":38,"tag":343,"props":486,"children":487},{"class":345,"line":364},[488],{"type":38,"tag":343,"props":489,"children":490},{},[491],{"type":44,"value":370},{"type":38,"tag":343,"props":493,"children":494},{"class":345,"line":373},[495],{"type":38,"tag":343,"props":496,"children":497},{},[498],{"type":44,"value":379},{"type":38,"tag":343,"props":500,"children":501},{"class":345,"line":382},[502],{"type":38,"tag":343,"props":503,"children":504},{},[505],{"type":44,"value":388},{"type":38,"tag":325,"props":507,"children":509},{"id":508},"performance-harness-cli",[510],{"type":44,"value":511},"Performance harness CLI",{"type":38,"tag":332,"props":513,"children":517},{"className":514,"code":515,"language":516,"meta":337,"style":337},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","uv run python scripts\u002Fperformance\u002Frun_script.py \\\n  -m qwen \\\n  -mr qwen3_30b_a3b \\\n  --task pretrain \\\n  -g h100 \\\n  -c bf16 \\\n  -ng 16 \\\n  --cuda_graph_impl transformer_engine \\\n  --cuda_graph_scope attn,moe_router,moe_preprocess \\\n  ...\n","bash",[518],{"type":38,"tag":81,"props":519,"children":520},{"__ignoreMap":337},[521,552,569,586,603,620,637,655,673,691],{"type":38,"tag":343,"props":522,"children":523},{"class":345,"line":346},[524,530,536,541,546],{"type":38,"tag":343,"props":525,"children":527},{"style":526},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[528],{"type":44,"value":529},"uv",{"type":38,"tag":343,"props":531,"children":533},{"style":532},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[534],{"type":44,"value":535}," run",{"type":38,"tag":343,"props":537,"children":538},{"style":532},[539],{"type":44,"value":540}," python",{"type":38,"tag":343,"props":542,"children":543},{"style":532},[544],{"type":44,"value":545}," scripts\u002Fperformance\u002Frun_script.py",{"type":38,"tag":343,"props":547,"children":549},{"style":548},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[550],{"type":44,"value":551}," \\\n",{"type":38,"tag":343,"props":553,"children":554},{"class":345,"line":355},[555,560,565],{"type":38,"tag":343,"props":556,"children":557},{"style":532},[558],{"type":44,"value":559},"  -m",{"type":38,"tag":343,"props":561,"children":562},{"style":532},[563],{"type":44,"value":564}," qwen",{"type":38,"tag":343,"props":566,"children":567},{"style":548},[568],{"type":44,"value":551},{"type":38,"tag":343,"props":570,"children":571},{"class":345,"line":364},[572,577,582],{"type":38,"tag":343,"props":573,"children":574},{"style":532},[575],{"type":44,"value":576},"  -mr",{"type":38,"tag":343,"props":578,"children":579},{"style":532},[580],{"type":44,"value":581}," qwen3_30b_a3b",{"type":38,"tag":343,"props":583,"children":584},{"style":548},[585],{"type":44,"value":551},{"type":38,"tag":343,"props":587,"children":588},{"class":345,"line":373},[589,594,599],{"type":38,"tag":343,"props":590,"children":591},{"style":532},[592],{"type":44,"value":593},"  --task",{"type":38,"tag":343,"props":595,"children":596},{"style":532},[597],{"type":44,"value":598}," pretrain",{"type":38,"tag":343,"props":600,"children":601},{"style":548},[602],{"type":44,"value":551},{"type":38,"tag":343,"props":604,"children":605},{"class":345,"line":382},[606,611,616],{"type":38,"tag":343,"props":607,"children":608},{"style":532},[609],{"type":44,"value":610},"  -g",{"type":38,"tag":343,"props":612,"children":613},{"style":532},[614],{"type":44,"value":615}," h100",{"type":38,"tag":343,"props":617,"children":618},{"style":548},[619],{"type":44,"value":551},{"type":38,"tag":343,"props":621,"children":622},{"class":345,"line":391},[623,628,633],{"type":38,"tag":343,"props":624,"children":625},{"style":532},[626],{"type":44,"value":627},"  -c",{"type":38,"tag":343,"props":629,"children":630},{"style":532},[631],{"type":44,"value":632}," bf16",{"type":38,"tag":343,"props":634,"children":635},{"style":548},[636],{"type":44,"value":551},{"type":38,"tag":343,"props":638,"children":639},{"class":345,"line":400},[640,645,651],{"type":38,"tag":343,"props":641,"children":642},{"style":532},[643],{"type":44,"value":644},"  -ng",{"type":38,"tag":343,"props":646,"children":648},{"style":647},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[649],{"type":44,"value":650}," 16",{"type":38,"tag":343,"props":652,"children":653},{"style":548},[654],{"type":44,"value":551},{"type":38,"tag":343,"props":656,"children":658},{"class":345,"line":657},8,[659,664,669],{"type":38,"tag":343,"props":660,"children":661},{"style":532},[662],{"type":44,"value":663},"  --cuda_graph_impl",{"type":38,"tag":343,"props":665,"children":666},{"style":532},[667],{"type":44,"value":668}," transformer_engine",{"type":38,"tag":343,"props":670,"children":671},{"style":548},[672],{"type":44,"value":551},{"type":38,"tag":343,"props":674,"children":676},{"class":345,"line":675},9,[677,682,687],{"type":38,"tag":343,"props":678,"children":679},{"style":532},[680],{"type":44,"value":681},"  --cuda_graph_scope",{"type":38,"tag":343,"props":683,"children":684},{"style":532},[685],{"type":44,"value":686}," attn,moe_router,moe_preprocess",{"type":38,"tag":343,"props":688,"children":689},{"style":548},[690],{"type":44,"value":551},{"type":38,"tag":343,"props":692,"children":694},{"class":345,"line":693},10,[695],{"type":38,"tag":343,"props":696,"children":697},{"style":532},[698],{"type":44,"value":699},"  ...\n",{"type":38,"tag":47,"props":701,"children":702},{},[703,705,711],{"type":44,"value":704},"Valid CLI values live in ",{"type":38,"tag":81,"props":706,"children":708},{"className":707},[],[709],{"type":44,"value":710},"scripts\u002Fperformance\u002Fargument_parser.py",{"type":44,"value":712},":",{"type":38,"tag":218,"props":714,"children":715},{},[716,733],{"type":38,"tag":222,"props":717,"children":718},{},[719,725,727],{"type":38,"tag":81,"props":720,"children":722},{"className":721},[],[723],{"type":44,"value":724},"VALID_CUDA_GRAPH_IMPLS",{"type":44,"value":726},": ",{"type":38,"tag":81,"props":728,"children":730},{"className":729},[],[731],{"type":44,"value":732},"[\"none\", \"local\", \"transformer_engine\"]",{"type":38,"tag":222,"props":734,"children":735},{},[736,742,743],{"type":38,"tag":81,"props":737,"children":739},{"className":738},[],[740],{"type":44,"value":741},"VALID_CUDA_GRAPH_SCOPES",{"type":44,"value":726},{"type":38,"tag":81,"props":744,"children":746},{"className":745},[],[747],{"type":44,"value":748},"[\"full_iteration\", \"attn\", \"mlp\", \"moe\", \"moe_router\", \"moe_preprocess\", \"mamba\"]",{"type":38,"tag":47,"props":750,"children":751},{},[752,754,760,762,768,770,776,778,784,786,792],{"type":44,"value":753},"The performance harness uses a comma-separated ",{"type":38,"tag":81,"props":755,"children":757},{"className":756},[],[758],{"type":44,"value":759},"--cuda_graph_scope",{"type":44,"value":761}," value and\nauto-enables ",{"type":38,"tag":81,"props":763,"children":765},{"className":764},[],[766],{"type":44,"value":767},"model.use_te_rng_tracker",{"type":44,"value":769}," plus ",{"type":38,"tag":81,"props":771,"children":773},{"className":772},[],[774],{"type":44,"value":775},"rng.te_rng_tracker",{"type":44,"value":777}," when\n",{"type":38,"tag":81,"props":779,"children":781},{"className":780},[],[782],{"type":44,"value":783},"--cuda_graph_impl",{"type":44,"value":785}," is not ",{"type":38,"tag":81,"props":787,"children":789},{"className":788},[],[790],{"type":44,"value":791},"none",{"type":44,"value":793},".",{"type":38,"tag":325,"props":795,"children":797},{"id":796},"required-constraints",[798],{"type":44,"value":799},"Required constraints",{"type":38,"tag":218,"props":801,"children":802},{},[803,822,838,854,873,878,883,904,909],{"type":38,"tag":222,"props":804,"children":805},{},[806,812,814,820],{"type":38,"tag":81,"props":807,"children":809},{"className":808},[],[810],{"type":44,"value":811},"use_te_rng_tracker = True",{"type":44,"value":813}," (enforced in ",{"type":38,"tag":81,"props":815,"children":817},{"className":816},[],[818],{"type":44,"value":819},"gpt_provider.py",{"type":44,"value":821},")",{"type":38,"tag":222,"props":823,"children":824},{},[825,830,832],{"type":38,"tag":81,"props":826,"children":828},{"className":827},[],[829],{"type":44,"value":135},{"type":44,"value":831}," scope only with ",{"type":38,"tag":81,"props":833,"children":835},{"className":834},[],[836],{"type":44,"value":837},"cuda_graph_impl = \"local\"",{"type":38,"tag":222,"props":839,"children":840},{},[841,846,848],{"type":38,"tag":81,"props":842,"children":844},{"className":843},[],[845],{"type":44,"value":135},{"type":44,"value":847}," scope requires ",{"type":38,"tag":81,"props":849,"children":851},{"className":850},[],[852],{"type":44,"value":853},"check_for_nan_in_loss = False",{"type":38,"tag":222,"props":855,"children":856},{},[857,859,864,866,871],{"type":44,"value":858},"Do not combine ",{"type":38,"tag":81,"props":860,"children":862},{"className":861},[],[863],{"type":44,"value":184},{"type":44,"value":865}," scope and ",{"type":38,"tag":81,"props":867,"children":869},{"className":868},[],[870],{"type":44,"value":191},{"type":44,"value":872}," scope",{"type":38,"tag":222,"props":874,"children":875},{},[876],{"type":44,"value":877},"Tensor shapes must be static (fixed seq_length, fixed micro_batch_size)",{"type":38,"tag":222,"props":879,"children":880},{},[881],{"type":44,"value":882},"MoE token-dropless routing limits graphable scope to dense modules",{"type":38,"tag":222,"props":884,"children":885},{},[886,888,894,896,902],{"type":44,"value":887},"With ",{"type":38,"tag":81,"props":889,"children":891},{"className":890},[],[892],{"type":44,"value":893},"PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True",{"type":44,"value":895},", set\n",{"type":38,"tag":81,"props":897,"children":899},{"className":898},[],[900],{"type":44,"value":901},"NCCL_GRAPH_REGISTER=0",{"type":44,"value":903}," (MCore enforces for local impl on arch \u003C sm_100;\nTE impl asserts unconditionally)",{"type":38,"tag":222,"props":905,"children":906},{},[907],{"type":44,"value":908},"CPU offloading is incompatible with CUDA graphs",{"type":38,"tag":222,"props":910,"children":911},{},[912,917,918,923],{"type":38,"tag":81,"props":913,"children":915},{"className":914},[],[916],{"type":44,"value":198},{"type":44,"value":847},{"type":38,"tag":81,"props":919,"children":921},{"className":920},[],[922],{"type":44,"value":191},{"type":44,"value":924}," scope to also be set",{"type":38,"tag":325,"props":926,"children":928},{"id":927},"practical-bring-up-order",[929],{"type":44,"value":930},"Practical bring-up order",{"type":38,"tag":932,"props":933,"children":934},"ol",{},[935,940,945,950,955,960],{"type":38,"tag":222,"props":936,"children":937},{},[938],{"type":44,"value":939},"Stabilize the eager run first.",{"type":38,"tag":222,"props":941,"children":942},{},[943],{"type":44,"value":944},"Fix sequence length and micro-batch size.",{"type":38,"tag":222,"props":946,"children":947},{},[948],{"type":44,"value":949},"Enable the narrowest useful graph scope.",{"type":38,"tag":222,"props":951,"children":952},{},[953],{"type":44,"value":954},"Confirm replay is active and memory is still acceptable.",{"type":38,"tag":222,"props":956,"children":957},{},[958],{"type":44,"value":959},"Compare eager against graph replay iterations after warmup and capture; do\nnot include the capture step in steady-state timing.",{"type":38,"tag":222,"props":961,"children":962},{},[963],{"type":44,"value":964},"Only then widen scope or combine with overlap features.",{"type":38,"tag":53,"props":966,"children":968},{"id":967},"code-anchors",[969],{"type":44,"value":970},"Code Anchors",{"type":38,"tag":325,"props":972,"children":974},{"id":973},"bridge-config-and-validation",[975],{"type":44,"value":976},"Bridge config and validation",{"type":38,"tag":332,"props":978,"children":982},{"className":979,"code":980,"language":981,"meta":337,"style":337},"language-1524:1531:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","        # CUDA graph scope validation: check_for_nan_in_loss must be disabled with full_iteration graph\n        if self.model.cuda_graph_impl == \"local\" and CudaGraphScope.full_iteration in self.model.cuda_graph_scope:\n            assert not self.rerun_state_machine.check_for_nan_in_loss, (\n                \"check_for_nan_in_loss must be disabled when using full_iteration CUDA graph. \"\n                \"Set rerun_state_machine.check_for_nan_in_loss=False.\"\n            )\n        if self.model.cuda_graph_impl == \"none\":\n            self.model.cuda_graph_scope = []\n","1524:1531:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fconfig.py",[983],{"type":38,"tag":81,"props":984,"children":985},{"__ignoreMap":337},[986,994,1002,1010,1018,1026,1034,1042],{"type":38,"tag":343,"props":987,"children":988},{"class":345,"line":346},[989],{"type":38,"tag":343,"props":990,"children":991},{},[992],{"type":44,"value":993},"        # CUDA graph scope validation: check_for_nan_in_loss must be disabled with full_iteration graph\n",{"type":38,"tag":343,"props":995,"children":996},{"class":345,"line":355},[997],{"type":38,"tag":343,"props":998,"children":999},{},[1000],{"type":44,"value":1001},"        if self.model.cuda_graph_impl == \"local\" and CudaGraphScope.full_iteration in self.model.cuda_graph_scope:\n",{"type":38,"tag":343,"props":1003,"children":1004},{"class":345,"line":364},[1005],{"type":38,"tag":343,"props":1006,"children":1007},{},[1008],{"type":44,"value":1009},"            assert not self.rerun_state_machine.check_for_nan_in_loss, (\n",{"type":38,"tag":343,"props":1011,"children":1012},{"class":345,"line":373},[1013],{"type":38,"tag":343,"props":1014,"children":1015},{},[1016],{"type":44,"value":1017},"                \"check_for_nan_in_loss must be disabled when using full_iteration CUDA graph. \"\n",{"type":38,"tag":343,"props":1019,"children":1020},{"class":345,"line":382},[1021],{"type":38,"tag":343,"props":1022,"children":1023},{},[1024],{"type":44,"value":1025},"                \"Set rerun_state_machine.check_for_nan_in_loss=False.\"\n",{"type":38,"tag":343,"props":1027,"children":1028},{"class":345,"line":391},[1029],{"type":38,"tag":343,"props":1030,"children":1031},{},[1032],{"type":44,"value":1033},"            )\n",{"type":38,"tag":343,"props":1035,"children":1036},{"class":345,"line":400},[1037],{"type":38,"tag":343,"props":1038,"children":1039},{},[1040],{"type":44,"value":1041},"        if self.model.cuda_graph_impl == \"none\":\n",{"type":38,"tag":343,"props":1043,"children":1044},{"class":345,"line":657},[1045],{"type":38,"tag":343,"props":1046,"children":1047},{},[1048],{"type":44,"value":1049},"            self.model.cuda_graph_scope = []\n",{"type":38,"tag":325,"props":1051,"children":1053},{"id":1052},"te-rng-tracker-requirement",[1054],{"type":44,"value":1055},"TE RNG tracker requirement",{"type":38,"tag":332,"props":1057,"children":1061},{"className":1058,"code":1059,"language":1060,"meta":337,"style":337},"language-213:216:src\u002Fmegatron\u002Fbridge\u002Fmodels\u002Fgpt_provider.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","        if self.cuda_graph_impl != \"none\":\n            assert getattr(self, \"use_te_rng_tracker\", False), (\n                \"Transformer engine's RNG tracker is required for cudagraphs, it can be \"\n                \"enabled with use_te_rng_tracker=True'.\"\n","213:216:src\u002Fmegatron\u002Fbridge\u002Fmodels\u002Fgpt_provider.py",[1062],{"type":38,"tag":81,"props":1063,"children":1064},{"__ignoreMap":337},[1065,1073,1081,1089],{"type":38,"tag":343,"props":1066,"children":1067},{"class":345,"line":346},[1068],{"type":38,"tag":343,"props":1069,"children":1070},{},[1071],{"type":44,"value":1072},"        if self.cuda_graph_impl != \"none\":\n",{"type":38,"tag":343,"props":1074,"children":1075},{"class":345,"line":355},[1076],{"type":38,"tag":343,"props":1077,"children":1078},{},[1079],{"type":44,"value":1080},"            assert getattr(self, \"use_te_rng_tracker\", False), (\n",{"type":38,"tag":343,"props":1082,"children":1083},{"class":345,"line":364},[1084],{"type":38,"tag":343,"props":1085,"children":1086},{},[1087],{"type":44,"value":1088},"                \"Transformer engine's RNG tracker is required for cudagraphs, it can be \"\n",{"type":38,"tag":343,"props":1090,"children":1091},{"class":345,"line":373},[1092],{"type":38,"tag":343,"props":1093,"children":1094},{},[1095],{"type":44,"value":1096},"                \"enabled with use_te_rng_tracker=True'.\"\n",{"type":38,"tag":325,"props":1098,"children":1100},{"id":1099},"graph-creation-and-capture-in-training-loop",[1101],{"type":44,"value":1102},"Graph creation and capture in training loop",{"type":38,"tag":332,"props":1104,"children":1108},{"className":1105,"code":1106,"language":1107,"meta":337,"style":337},"language-231:255:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","    # Capture CUDA Graphs.\n    cuda_graph_helper = None\n    if model_config.cuda_graph_impl == \"transformer_engine\":\n        cuda_graph_helper = TECudaGraphHelper(...)\n    # ...\n    if config.model.cuda_graph_impl == \"local\" and CudaGraphScope.full_iteration in config.model.cuda_graph_scope:\n        forward_backward_func = FullCudaGraphWrapper(\n            forward_backward_func, cuda_graph_warmup_steps=config.model.cuda_graph_warmup_steps\n        )\n","231:255:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py",[1109],{"type":38,"tag":81,"props":1110,"children":1111},{"__ignoreMap":337},[1112,1120,1128,1136,1144,1152,1160,1168,1176],{"type":38,"tag":343,"props":1113,"children":1114},{"class":345,"line":346},[1115],{"type":38,"tag":343,"props":1116,"children":1117},{},[1118],{"type":44,"value":1119},"    # Capture CUDA Graphs.\n",{"type":38,"tag":343,"props":1121,"children":1122},{"class":345,"line":355},[1123],{"type":38,"tag":343,"props":1124,"children":1125},{},[1126],{"type":44,"value":1127},"    cuda_graph_helper = None\n",{"type":38,"tag":343,"props":1129,"children":1130},{"class":345,"line":364},[1131],{"type":38,"tag":343,"props":1132,"children":1133},{},[1134],{"type":44,"value":1135},"    if model_config.cuda_graph_impl == \"transformer_engine\":\n",{"type":38,"tag":343,"props":1137,"children":1138},{"class":345,"line":373},[1139],{"type":38,"tag":343,"props":1140,"children":1141},{},[1142],{"type":44,"value":1143},"        cuda_graph_helper = TECudaGraphHelper(...)\n",{"type":38,"tag":343,"props":1145,"children":1146},{"class":345,"line":382},[1147],{"type":38,"tag":343,"props":1148,"children":1149},{},[1150],{"type":44,"value":1151},"    # ...\n",{"type":38,"tag":343,"props":1153,"children":1154},{"class":345,"line":391},[1155],{"type":38,"tag":343,"props":1156,"children":1157},{},[1158],{"type":44,"value":1159},"    if config.model.cuda_graph_impl == \"local\" and CudaGraphScope.full_iteration in config.model.cuda_graph_scope:\n",{"type":38,"tag":343,"props":1161,"children":1162},{"class":345,"line":400},[1163],{"type":38,"tag":343,"props":1164,"children":1165},{},[1166],{"type":44,"value":1167},"        forward_backward_func = FullCudaGraphWrapper(\n",{"type":38,"tag":343,"props":1169,"children":1170},{"class":345,"line":657},[1171],{"type":38,"tag":343,"props":1172,"children":1173},{},[1174],{"type":44,"value":1175},"            forward_backward_func, cuda_graph_warmup_steps=config.model.cuda_graph_warmup_steps\n",{"type":38,"tag":343,"props":1177,"children":1178},{"class":345,"line":675},[1179],{"type":38,"tag":343,"props":1180,"children":1181},{},[1182],{"type":44,"value":1183},"        )\n",{"type":38,"tag":325,"props":1185,"children":1187},{"id":1186},"te-graph-capture-after-warmup",[1188],{"type":44,"value":1189},"TE graph capture after warmup",{"type":38,"tag":332,"props":1191,"children":1195},{"className":1192,"code":1193,"language":1194,"meta":337,"style":337},"language-338:350:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","        # Capture CUDA Graphs after warmup.\n        if (\n            model_config.cuda_graph_impl == \"transformer_engine\"\n            and cuda_graph_helper is not None\n            and not cuda_graph_helper.graphs_created()\n            and global_state.train_state.step - start_iteration == model_config.cuda_graph_warmup_steps\n        ):\n            if model_config.cuda_graph_warmup_steps > 0 and should_toggle_forward_pre_hook:\n                disable_forward_pre_hook(model, param_sync=False)\n            cuda_graph_helper.create_cudagraphs()\n            if model_config.cuda_graph_warmup_steps > 0 and should_toggle_forward_pre_hook:\n                enable_forward_pre_hook(model)\n                cuda_graph_helper.cuda_graph_set_manual_hooks()\n","338:350:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py",[1196],{"type":38,"tag":81,"props":1197,"children":1198},{"__ignoreMap":337},[1199,1207,1215,1223,1231,1239,1247,1255,1263,1271,1279,1287,1296],{"type":38,"tag":343,"props":1200,"children":1201},{"class":345,"line":346},[1202],{"type":38,"tag":343,"props":1203,"children":1204},{},[1205],{"type":44,"value":1206},"        # Capture CUDA Graphs after warmup.\n",{"type":38,"tag":343,"props":1208,"children":1209},{"class":345,"line":355},[1210],{"type":38,"tag":343,"props":1211,"children":1212},{},[1213],{"type":44,"value":1214},"        if (\n",{"type":38,"tag":343,"props":1216,"children":1217},{"class":345,"line":364},[1218],{"type":38,"tag":343,"props":1219,"children":1220},{},[1221],{"type":44,"value":1222},"            model_config.cuda_graph_impl == \"transformer_engine\"\n",{"type":38,"tag":343,"props":1224,"children":1225},{"class":345,"line":373},[1226],{"type":38,"tag":343,"props":1227,"children":1228},{},[1229],{"type":44,"value":1230},"            and cuda_graph_helper is not None\n",{"type":38,"tag":343,"props":1232,"children":1233},{"class":345,"line":382},[1234],{"type":38,"tag":343,"props":1235,"children":1236},{},[1237],{"type":44,"value":1238},"            and not cuda_graph_helper.graphs_created()\n",{"type":38,"tag":343,"props":1240,"children":1241},{"class":345,"line":391},[1242],{"type":38,"tag":343,"props":1243,"children":1244},{},[1245],{"type":44,"value":1246},"            and global_state.train_state.step - start_iteration == model_config.cuda_graph_warmup_steps\n",{"type":38,"tag":343,"props":1248,"children":1249},{"class":345,"line":400},[1250],{"type":38,"tag":343,"props":1251,"children":1252},{},[1253],{"type":44,"value":1254},"        ):\n",{"type":38,"tag":343,"props":1256,"children":1257},{"class":345,"line":657},[1258],{"type":38,"tag":343,"props":1259,"children":1260},{},[1261],{"type":44,"value":1262},"            if model_config.cuda_graph_warmup_steps > 0 and should_toggle_forward_pre_hook:\n",{"type":38,"tag":343,"props":1264,"children":1265},{"class":345,"line":675},[1266],{"type":38,"tag":343,"props":1267,"children":1268},{},[1269],{"type":44,"value":1270},"                disable_forward_pre_hook(model, param_sync=False)\n",{"type":38,"tag":343,"props":1272,"children":1273},{"class":345,"line":693},[1274],{"type":38,"tag":343,"props":1275,"children":1276},{},[1277],{"type":44,"value":1278},"            cuda_graph_helper.create_cudagraphs()\n",{"type":38,"tag":343,"props":1280,"children":1282},{"class":345,"line":1281},11,[1283],{"type":38,"tag":343,"props":1284,"children":1285},{},[1286],{"type":44,"value":1262},{"type":38,"tag":343,"props":1288,"children":1290},{"class":345,"line":1289},12,[1291],{"type":38,"tag":343,"props":1292,"children":1293},{},[1294],{"type":44,"value":1295},"                enable_forward_pre_hook(model)\n",{"type":38,"tag":343,"props":1297,"children":1299},{"class":345,"line":1298},13,[1300],{"type":38,"tag":343,"props":1301,"children":1302},{},[1303],{"type":44,"value":1304},"                cuda_graph_helper.cuda_graph_set_manual_hooks()\n",{"type":38,"tag":325,"props":1306,"children":1308},{"id":1307},"rng-initialization",[1309],{"type":44,"value":1310},"RNG initialization",{"type":38,"tag":332,"props":1312,"children":1316},{"className":1313,"code":1314,"language":1315,"meta":337,"style":337},"language-199:206:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finitialize.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","        _set_random_seed(\n            rng_config.seed,\n            rng_config.data_parallel_random_init,\n            rng_config.te_rng_tracker,\n            rng_config.inference_rng_tracker,\n            use_cudagraphable_rng=(model_config.cuda_graph_impl != \"none\"),\n            pg_collection=pg_collection,\n        )\n","199:206:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Finitialize.py",[1317],{"type":38,"tag":81,"props":1318,"children":1319},{"__ignoreMap":337},[1320,1328,1336,1344,1352,1360,1368,1376],{"type":38,"tag":343,"props":1321,"children":1322},{"class":345,"line":346},[1323],{"type":38,"tag":343,"props":1324,"children":1325},{},[1326],{"type":44,"value":1327},"        _set_random_seed(\n",{"type":38,"tag":343,"props":1329,"children":1330},{"class":345,"line":355},[1331],{"type":38,"tag":343,"props":1332,"children":1333},{},[1334],{"type":44,"value":1335},"            rng_config.seed,\n",{"type":38,"tag":343,"props":1337,"children":1338},{"class":345,"line":364},[1339],{"type":38,"tag":343,"props":1340,"children":1341},{},[1342],{"type":44,"value":1343},"            rng_config.data_parallel_random_init,\n",{"type":38,"tag":343,"props":1345,"children":1346},{"class":345,"line":373},[1347],{"type":38,"tag":343,"props":1348,"children":1349},{},[1350],{"type":44,"value":1351},"            rng_config.te_rng_tracker,\n",{"type":38,"tag":343,"props":1353,"children":1354},{"class":345,"line":382},[1355],{"type":38,"tag":343,"props":1356,"children":1357},{},[1358],{"type":44,"value":1359},"            rng_config.inference_rng_tracker,\n",{"type":38,"tag":343,"props":1361,"children":1362},{"class":345,"line":391},[1363],{"type":38,"tag":343,"props":1364,"children":1365},{},[1366],{"type":44,"value":1367},"            use_cudagraphable_rng=(model_config.cuda_graph_impl != \"none\"),\n",{"type":38,"tag":343,"props":1369,"children":1370},{"class":345,"line":400},[1371],{"type":38,"tag":343,"props":1372,"children":1373},{},[1374],{"type":44,"value":1375},"            pg_collection=pg_collection,\n",{"type":38,"tag":343,"props":1377,"children":1378},{"class":345,"line":657},[1379],{"type":38,"tag":343,"props":1380,"children":1381},{},[1382],{"type":44,"value":1183},{"type":38,"tag":325,"props":1384,"children":1386},{"id":1385},"delayed-wgrad-cuda-graph-interaction",[1387],{"type":44,"value":1388},"Delayed wgrad + CUDA graph interaction",{"type":38,"tag":332,"props":1390,"children":1394},{"className":1391,"code":1392,"language":1393,"meta":337,"style":337},"language-522:555:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcomm_overlap.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","            cuda_graph_scope = getattr(model_cfg, \"cuda_graph_scope\", []) or []\n            # ... scope parsing ...\n            if wgrad_in_graph_scope:\n                assert is_te_min_version(\"2.12.0\"), ...\n                assert model_cfg.gradient_accumulation_fusion, ...\n                if attn_scope_enabled:\n                    assert not model_cfg.add_bias_linear and not model_cfg.add_qkv_bias, ...\n","522:555:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Fcomm_overlap.py",[1395],{"type":38,"tag":81,"props":1396,"children":1397},{"__ignoreMap":337},[1398,1406,1414,1422,1430,1438,1446],{"type":38,"tag":343,"props":1399,"children":1400},{"class":345,"line":346},[1401],{"type":38,"tag":343,"props":1402,"children":1403},{},[1404],{"type":44,"value":1405},"            cuda_graph_scope = getattr(model_cfg, \"cuda_graph_scope\", []) or []\n",{"type":38,"tag":343,"props":1407,"children":1408},{"class":345,"line":355},[1409],{"type":38,"tag":343,"props":1410,"children":1411},{},[1412],{"type":44,"value":1413},"            # ... scope parsing ...\n",{"type":38,"tag":343,"props":1415,"children":1416},{"class":345,"line":364},[1417],{"type":38,"tag":343,"props":1418,"children":1419},{},[1420],{"type":44,"value":1421},"            if wgrad_in_graph_scope:\n",{"type":38,"tag":343,"props":1423,"children":1424},{"class":345,"line":373},[1425],{"type":38,"tag":343,"props":1426,"children":1427},{},[1428],{"type":44,"value":1429},"                assert is_te_min_version(\"2.12.0\"), ...\n",{"type":38,"tag":343,"props":1431,"children":1432},{"class":345,"line":382},[1433],{"type":38,"tag":343,"props":1434,"children":1435},{},[1436],{"type":44,"value":1437},"                assert model_cfg.gradient_accumulation_fusion, ...\n",{"type":38,"tag":343,"props":1439,"children":1440},{"class":345,"line":391},[1441],{"type":38,"tag":343,"props":1442,"children":1443},{},[1444],{"type":44,"value":1445},"                if attn_scope_enabled:\n",{"type":38,"tag":343,"props":1447,"children":1448},{"class":345,"line":400},[1449],{"type":38,"tag":343,"props":1450,"children":1451},{},[1452],{"type":44,"value":1453},"                    assert not model_cfg.add_bias_linear and not model_cfg.add_qkv_bias, ...\n",{"type":38,"tag":325,"props":1455,"children":1457},{"id":1456},"perf-harness-override-helper",[1458],{"type":44,"value":1459},"Perf harness override helper",{"type":38,"tag":332,"props":1461,"children":1465},{"className":1462,"code":1463,"language":1464,"meta":337,"style":337},"language-102:124:scripts\u002Fperformance\u002Futils\u002Foverrides.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","def _set_cuda_graph_overrides(\n    recipe, cuda_graph_impl=None, cuda_graph_scope=None\n):\n    # Sets impl, scope, and auto-enables te_rng_tracker\n","102:124:scripts\u002Fperformance\u002Futils\u002Foverrides.py",[1466],{"type":38,"tag":81,"props":1467,"children":1468},{"__ignoreMap":337},[1469,1477,1485,1493],{"type":38,"tag":343,"props":1470,"children":1471},{"class":345,"line":346},[1472],{"type":38,"tag":343,"props":1473,"children":1474},{},[1475],{"type":44,"value":1476},"def _set_cuda_graph_overrides(\n",{"type":38,"tag":343,"props":1478,"children":1479},{"class":345,"line":355},[1480],{"type":38,"tag":343,"props":1481,"children":1482},{},[1483],{"type":44,"value":1484},"    recipe, cuda_graph_impl=None, cuda_graph_scope=None\n",{"type":38,"tag":343,"props":1486,"children":1487},{"class":345,"line":364},[1488],{"type":38,"tag":343,"props":1489,"children":1490},{},[1491],{"type":44,"value":1492},"):\n",{"type":38,"tag":343,"props":1494,"children":1495},{"class":345,"line":373},[1496],{"type":38,"tag":343,"props":1497,"children":1498},{},[1499],{"type":44,"value":1500},"    # Sets impl, scope, and auto-enables te_rng_tracker\n",{"type":38,"tag":325,"props":1502,"children":1504},{"id":1503},"graph-cleanup",[1505],{"type":44,"value":1506},"Graph cleanup",{"type":38,"tag":332,"props":1508,"children":1512},{"className":1509,"code":1510,"language":1511,"meta":337,"style":337},"language-1414:1441:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","def _delete_cuda_graphs(cuda_graph_helper):\n    # Deletes FullCudaGraphWrapper and TE graph objects to free NCCL buffers\n","1414:1441:src\u002Fmegatron\u002Fbridge\u002Ftraining\u002Ftrain.py",[1513],{"type":38,"tag":81,"props":1514,"children":1515},{"__ignoreMap":337},[1516,1524],{"type":38,"tag":343,"props":1517,"children":1518},{"class":345,"line":346},[1519],{"type":38,"tag":343,"props":1520,"children":1521},{},[1522],{"type":44,"value":1523},"def _delete_cuda_graphs(cuda_graph_helper):\n",{"type":38,"tag":343,"props":1525,"children":1526},{"class":345,"line":355},[1527],{"type":38,"tag":343,"props":1528,"children":1529},{},[1530],{"type":44,"value":1531},"    # Deletes FullCudaGraphWrapper and TE graph objects to free NCCL buffers\n",{"type":38,"tag":325,"props":1533,"children":1535},{"id":1534},"mcore-classes-in-3rdpartymegatron-lm",[1536],{"type":44,"value":1537},"MCore classes (in 3rdparty\u002FMegatron-LM)",{"type":38,"tag":218,"props":1539,"children":1540},{},[1541,1557,1572,1587],{"type":38,"tag":222,"props":1542,"children":1543},{},[1544,1550,1551],{"type":38,"tag":81,"props":1545,"children":1547},{"className":1546},[],[1548],{"type":44,"value":1549},"CudaGraphManager",{"type":44,"value":726},{"type":38,"tag":81,"props":1552,"children":1554},{"className":1553},[],[1555],{"type":44,"value":1556},"megatron\u002Fcore\u002Ftransformer\u002Fcuda_graphs.py",{"type":38,"tag":222,"props":1558,"children":1559},{},[1560,1566,1567],{"type":38,"tag":81,"props":1561,"children":1563},{"className":1562},[],[1564],{"type":44,"value":1565},"TECudaGraphHelper",{"type":44,"value":726},{"type":38,"tag":81,"props":1568,"children":1570},{"className":1569},[],[1571],{"type":44,"value":1556},{"type":38,"tag":222,"props":1573,"children":1574},{},[1575,1580,1581],{"type":38,"tag":81,"props":1576,"children":1578},{"className":1577},[],[1579],{"type":44,"value":124},{"type":44,"value":726},{"type":38,"tag":81,"props":1582,"children":1584},{"className":1583},[],[1585],{"type":44,"value":1586},"megatron\u002Fcore\u002Ffull_cuda_graph.py",{"type":38,"tag":222,"props":1588,"children":1589},{},[1590,1596,1598],{"type":38,"tag":81,"props":1591,"children":1593},{"className":1592},[],[1594],{"type":44,"value":1595},"CudaGraphScope",{"type":44,"value":1597}," enum: ",{"type":38,"tag":81,"props":1599,"children":1601},{"className":1600},[],[1602],{"type":44,"value":1603},"megatron\u002Fcore\u002Ftransformer\u002Fenums.py",{"type":38,"tag":325,"props":1605,"children":1607},{"id":1606},"positive-recipe-anchors",[1608],{"type":44,"value":1609},"Positive recipe anchors",{"type":38,"tag":218,"props":1611,"children":1612},{},[1613,1622,1631],{"type":38,"tag":222,"props":1614,"children":1615},{},[1616],{"type":38,"tag":81,"props":1617,"children":1619},{"className":1618},[],[1620],{"type":44,"value":1621},"src\u002Fmegatron\u002Fbridge\u002Fperf_recipes\u002Fdeepseek\u002Fgb300\u002Fdeepseek_v3.py",{"type":38,"tag":222,"props":1623,"children":1624},{},[1625],{"type":38,"tag":81,"props":1626,"children":1628},{"className":1627},[],[1629],{"type":44,"value":1630},"src\u002Fmegatron\u002Fbridge\u002Fperf_recipes\u002Fqwen\u002Fgb300\u002Fqwen3_moe.py",{"type":38,"tag":222,"props":1632,"children":1633},{},[1634],{"type":38,"tag":81,"props":1635,"children":1637},{"className":1636},[],[1638],{"type":44,"value":1639},"src\u002Fmegatron\u002Fbridge\u002Fperf_recipes\u002Fgpt_oss\u002Fgb300\u002Fgpt_oss.py",{"type":38,"tag":325,"props":1641,"children":1643},{"id":1642},"tests",[1644],{"type":44,"value":1645},"Tests",{"type":38,"tag":65,"props":1647,"children":1648},{},[1649,1665],{"type":38,"tag":69,"props":1650,"children":1651},{},[1652],{"type":38,"tag":73,"props":1653,"children":1654},{},[1655,1660],{"type":38,"tag":77,"props":1656,"children":1657},{},[1658],{"type":44,"value":1659},"File",{"type":38,"tag":77,"props":1661,"children":1662},{},[1663],{"type":44,"value":1664},"Coverage",{"type":38,"tag":98,"props":1666,"children":1667},{},[1668,1690,1713,1730,1747,1764,1780],{"type":38,"tag":73,"props":1669,"children":1670},{},[1671,1680],{"type":38,"tag":105,"props":1672,"children":1673},{},[1674],{"type":38,"tag":81,"props":1675,"children":1677},{"className":1676},[],[1678],{"type":44,"value":1679},"tests\u002Funit_tests\u002Ftraining\u002Ftest_config.py",{"type":38,"tag":105,"props":1681,"children":1682},{},[1683,1688],{"type":38,"tag":81,"props":1684,"children":1686},{"className":1685},[],[1687],{"type":44,"value":135},{"type":44,"value":1689}," NaN-check constraint",{"type":38,"tag":73,"props":1691,"children":1692},{},[1693,1702],{"type":38,"tag":105,"props":1694,"children":1695},{},[1696],{"type":38,"tag":81,"props":1697,"children":1699},{"className":1698},[],[1700],{"type":44,"value":1701},"tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py",{"type":38,"tag":105,"props":1703,"children":1704},{},[1705,1711],{"type":38,"tag":81,"props":1706,"children":1708},{"className":1707},[],[1709],{"type":44,"value":1710},"delay_wgrad",{"type":44,"value":1712}," + CUDA graph interaction",{"type":38,"tag":73,"props":1714,"children":1715},{},[1716,1725],{"type":38,"tag":105,"props":1717,"children":1718},{},[1719],{"type":38,"tag":81,"props":1720,"children":1722},{"className":1721},[],[1723],{"type":44,"value":1724},"tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_full_te_layer_autocast_spec.py",{"type":38,"tag":105,"props":1726,"children":1727},{},[1728],{"type":44,"value":1729},"TE autocast with CUDA graphs",{"type":38,"tag":73,"props":1731,"children":1732},{},[1733,1742],{"type":38,"tag":105,"props":1734,"children":1735},{},[1736],{"type":38,"tag":81,"props":1737,"children":1739},{"className":1738},[],[1740],{"type":44,"value":1741},"tests\u002Ffunctional_tests\u002Ftest_groups\u002Frecipes\u002Ftest_llama_recipes_pretrain_cuda_graphs.py",{"type":38,"tag":105,"props":1743,"children":1744},{},[1745],{"type":44,"value":1746},"End-to-end local and TE graph smoke tests",{"type":38,"tag":73,"props":1748,"children":1749},{},[1750,1759],{"type":38,"tag":105,"props":1751,"children":1752},{},[1753],{"type":38,"tag":81,"props":1754,"children":1756},{"className":1755},[],[1757],{"type":44,"value":1758},"tests\u002Funit_tests\u002Frecipes\u002Fkimi\u002Ftest_kimi_k2.py",{"type":38,"tag":105,"props":1760,"children":1761},{},[1762],{"type":44,"value":1763},"TE + CUDA graph recipe config",{"type":38,"tag":73,"props":1765,"children":1766},{},[1767,1776],{"type":38,"tag":105,"props":1768,"children":1769},{},[1770],{"type":38,"tag":81,"props":1771,"children":1773},{"className":1772},[],[1774],{"type":44,"value":1775},"tests\u002Funit_tests\u002Frecipes\u002Fgpt\u002Ftest_gpt3_175b.py",{"type":38,"tag":105,"props":1777,"children":1778},{},[1779],{"type":44,"value":1763},{"type":38,"tag":73,"props":1781,"children":1782},{},[1783,1792],{"type":38,"tag":105,"props":1784,"children":1785},{},[1786],{"type":38,"tag":81,"props":1787,"children":1789},{"className":1788},[],[1790],{"type":44,"value":1791},"tests\u002Funit_tests\u002Frecipes\u002Fqwen_vl\u002Ftest_qwen25_vl_recipes.py",{"type":38,"tag":105,"props":1793,"children":1794},{},[1795],{"type":44,"value":1796},"VLM CUDA graph settings",{"type":38,"tag":53,"props":1798,"children":1800},{"id":1799},"pitfalls",[1801],{"type":44,"value":1802},"Pitfalls",{"type":38,"tag":932,"props":1804,"children":1805},{},[1806,1840,1855,1891,1909,1943,1953,1971,2009,2026,2058,2176,2186],{"type":38,"tag":222,"props":1807,"children":1808},{},[1809,1815,1817,1822,1824,1830,1832,1838],{"type":38,"tag":1810,"props":1811,"children":1812},"strong",{},[1813],{"type":44,"value":1814},"TE RNG tracker is mandatory",{"type":44,"value":1816},": Setting ",{"type":38,"tag":81,"props":1818,"children":1820},{"className":1819},[],[1821],{"type":44,"value":86},{"type":44,"value":1823}," without\n",{"type":38,"tag":81,"props":1825,"children":1827},{"className":1826},[],[1828],{"type":44,"value":1829},"use_te_rng_tracker=True",{"type":44,"value":1831}," and ",{"type":38,"tag":81,"props":1833,"children":1835},{"className":1834},[],[1836],{"type":44,"value":1837},"rng.te_rng_tracker=True",{"type":44,"value":1839}," will assert\nin the provider.",{"type":38,"tag":222,"props":1841,"children":1842},{},[1843,1853],{"type":38,"tag":1810,"props":1844,"children":1845},{},[1846,1851],{"type":38,"tag":81,"props":1847,"children":1849},{"className":1848},[],[1850],{"type":44,"value":135},{"type":44,"value":1852}," requires NaN checks disabled",{"type":44,"value":1854},": The entire fwd+bwd is\ncaptured, so loss-NaN checking cannot inspect intermediate values.",{"type":38,"tag":222,"props":1856,"children":1857},{},[1858,1863,1864,1869,1870,1875,1877,1882,1884,1889],{"type":38,"tag":1810,"props":1859,"children":1860},{},[1861],{"type":44,"value":1862},"MoE scope restrictions",{"type":44,"value":726},{"type":38,"tag":81,"props":1865,"children":1867},{"className":1866},[],[1868],{"type":44,"value":184},{"type":44,"value":865},{"type":38,"tag":81,"props":1871,"children":1873},{"className":1872},[],[1874],{"type":44,"value":191},{"type":44,"value":1876}," scope are\nmutually exclusive. Token-dropless MoE can only graph ",{"type":38,"tag":81,"props":1878,"children":1880},{"className":1879},[],[1881],{"type":44,"value":191},{"type":44,"value":1883}," and\n",{"type":38,"tag":81,"props":1885,"children":1887},{"className":1886},[],[1888],{"type":44,"value":198},{"type":44,"value":1890},", not the full expert dispatch.",{"type":38,"tag":222,"props":1892,"children":1893},{},[1894,1899,1901,1907],{"type":38,"tag":1810,"props":1895,"children":1896},{},[1897],{"type":44,"value":1898},"Memory overhead",{"type":44,"value":1900},": CUDA graphs pin all intermediate buffers for the\ngraph's lifetime (no memory reuse). TE scoped graphs add a few GB;\nfull-iteration graphs can increase peak memory by 1.5–2×. ",{"type":38,"tag":81,"props":1902,"children":1904},{"className":1903},[],[1905],{"type":44,"value":1906},"PP > 1",{"type":44,"value":1908},"\ncompounds overhead since each stage holds its own graph.",{"type":38,"tag":222,"props":1910,"children":1911},{},[1912,1917,1919,1925,1927,1933,1935,1941],{"type":38,"tag":1810,"props":1913,"children":1914},{},[1915],{"type":44,"value":1916},"Delayed wgrad interaction",{"type":44,"value":1918},": When ",{"type":38,"tag":81,"props":1920,"children":1922},{"className":1921},[],[1923],{"type":44,"value":1924},"delay_wgrad_compute=True",{"type":44,"value":1926}," and\nattention or MoE router is in ",{"type":38,"tag":81,"props":1928,"children":1930},{"className":1929},[],[1931],{"type":44,"value":1932},"cuda_graph_scope",{"type":44,"value":1934},", additional constraints\napply: TE >= 2.12.0, ",{"type":38,"tag":81,"props":1936,"children":1938},{"className":1937},[],[1939],{"type":44,"value":1940},"gradient_accumulation_fusion=True",{"type":44,"value":1942},", and no\nattention bias.",{"type":38,"tag":222,"props":1944,"children":1945},{},[1946,1951],{"type":38,"tag":1810,"props":1947,"children":1948},{},[1949],{"type":44,"value":1950},"Variable-length sequences break graphs",{"type":44,"value":1952},": Sequence lengths must be\nconstant across steps. Use padded packed sequences if packing is needed.",{"type":38,"tag":222,"props":1954,"children":1955},{},[1956,1961,1963,1969],{"type":38,"tag":1810,"props":1957,"children":1958},{},[1959],{"type":44,"value":1960},"Graph cleanup is required",{"type":44,"value":1962},": CUDA graph objects hold NCCL buffer\nreferences. Bridge handles this in ",{"type":38,"tag":81,"props":1964,"children":1966},{"className":1965},[],[1967],{"type":44,"value":1968},"_delete_cuda_graphs()",{"type":44,"value":1970}," at the end\nof training, but early exits must call it explicitly.",{"type":38,"tag":222,"props":1972,"children":1973},{},[1974,1979,1981,1986,1988,1993,1995,2000,2002,2007],{"type":38,"tag":1810,"props":1975,"children":1976},{},[1977],{"type":44,"value":1978},"Older GPU architectures",{"type":44,"value":1980},": On GPUs with compute capability \u003C 10.0\n(pre-Blackwell), set ",{"type":38,"tag":81,"props":1982,"children":1984},{"className":1983},[],[1985],{"type":44,"value":901},{"type":44,"value":1987}," when using\n",{"type":38,"tag":81,"props":1989,"children":1991},{"className":1990},[],[1992],{"type":44,"value":893},{"type":44,"value":1994},". Enforced in MCore\n",{"type":38,"tag":81,"props":1996,"children":1998},{"className":1997},[],[1999],{"type":44,"value":1549},{"type":44,"value":2001}," (cuda_graphs.py:1428) and ",{"type":38,"tag":81,"props":2003,"children":2005},{"className":2004},[],[2006],{"type":44,"value":1565},{"type":44,"value":2008},"\n(cuda_graphs.py:1697). The TE impl asserts unconditionally regardless\nof arch.",{"type":38,"tag":222,"props":2010,"children":2011},{},[2012,2017,2019,2025],{"type":38,"tag":1810,"props":2013,"children":2014},{},[2015],{"type":44,"value":2016},"CPU offloading incompatible",{"type":44,"value":2018},": CUDA graphs cannot be used with CPU\noffloading. Enforced in MCore ",{"type":38,"tag":81,"props":2020,"children":2022},{"className":2021},[],[2023],{"type":44,"value":2024},"transformer_config.py:1907",{"type":44,"value":793},{"type":38,"tag":222,"props":2027,"children":2028},{},[2029,2034,2036,2041,2043,2049,2051,2057],{"type":38,"tag":1810,"props":2030,"children":2031},{},[2032],{"type":44,"value":2033},"MoE recompute + moe_router scope",{"type":44,"value":2035},": MoE recompute is not supported\nwith ",{"type":38,"tag":81,"props":2037,"children":2039},{"className":2038},[],[2040],{"type":44,"value":191},{"type":44,"value":2042}," CUDA graph scope when using ",{"type":38,"tag":81,"props":2044,"children":2046},{"className":2045},[],[2047],{"type":44,"value":2048},"cuda_graph_impl = \"transformer_engine\"",{"type":44,"value":2050},". Enforced in MCore ",{"type":38,"tag":81,"props":2052,"children":2054},{"className":2053},[],[2055],{"type":44,"value":2056},"transformer_config.py:1977",{"type":44,"value":793},{"type":38,"tag":222,"props":2059,"children":2060},{},[2061,2072,2074,2080,2082,2088,2090,2095,2096,2101,2102,2107,2109,2115,2117,2123,2125,2131,2132,2138,2140,2146,2147,2153,2155,2160,2161,2166,2168,2174],{"type":38,"tag":1810,"props":2062,"children":2063},{},[2064,2066,2071],{"type":44,"value":2065},"Layer-level recompute requires ",{"type":38,"tag":81,"props":2067,"children":2069},{"className":2068},[],[2070],{"type":44,"value":135},{"type":44,"value":872},{"type":44,"value":2073},": Using\n",{"type":38,"tag":81,"props":2075,"children":2077},{"className":2076},[],[2078],{"type":44,"value":2079},"recompute_granularity=\"full\"",{"type":44,"value":2081}," with ",{"type":38,"tag":81,"props":2083,"children":2085},{"className":2084},[],[2086],{"type":44,"value":2087},"recompute_num_layers",{"type":44,"value":2089}," (recompute N\nwhole transformer layers) is incompatible with TE-scoped graphs. MCore\ncalls this \"full\" granularity even though you're selecting how many\nlayers — the name refers to recomputing the full layer, not full model.\nAny TE-scoped scope (",{"type":38,"tag":81,"props":2091,"children":2093},{"className":2092},[],[2094],{"type":44,"value":169},{"type":44,"value":171},{"type":38,"tag":81,"props":2097,"children":2099},{"className":2098},[],[2100],{"type":44,"value":177},{"type":44,"value":171},{"type":38,"tag":81,"props":2103,"children":2105},{"className":2104},[],[2106],{"type":44,"value":191},{"type":44,"value":2108},", etc.) will assert:\n",{"type":38,"tag":81,"props":2110,"children":2112},{"className":2111},[],[2113],{"type":44,"value":2114},"AssertionError: full recompute is only supported with full iteration CUDA graph.",{"type":44,"value":2116},"\nThis commonly hits FP8 configs that default to TE-scoped graphs (e.g.\n",{"type":38,"tag":81,"props":2118,"children":2120},{"className":2119},[],[2121],{"type":44,"value":2122},"LLAMA3_70B_SFT_CONFIG_H100_FP8_CS_V1",{"type":44,"value":2124}," uses ",{"type":38,"tag":81,"props":2126,"children":2128},{"className":2127},[],[2129],{"type":44,"value":2130},"cuda_graph_impl= \"transformer_engine\"",{"type":44,"value":171},{"type":38,"tag":81,"props":2133,"children":2135},{"className":2134},[],[2136],{"type":44,"value":2137},"cuda_graph_scope=\"mlp\"",{"type":44,"value":2139},"). Fix: use submodule\nrecompute (",{"type":38,"tag":81,"props":2141,"children":2143},{"className":2142},[],[2144],{"type":44,"value":2145},"recompute_granularity=\"selective\"",{"type":44,"value":267},{"type":38,"tag":81,"props":2148,"children":2150},{"className":2149},[],[2151],{"type":44,"value":2152},"recompute_modules",{"type":44,"value":2154},"),\ndisable CUDA graphs, or switch to ",{"type":38,"tag":81,"props":2156,"children":2158},{"className":2157},[],[2159],{"type":44,"value":265},{"type":44,"value":267},{"type":38,"tag":81,"props":2162,"children":2164},{"className":2163},[],[2165],{"type":44,"value":135},{"type":44,"value":2167},". Enforced\nin MCore ",{"type":38,"tag":81,"props":2169,"children":2171},{"className":2170},[],[2172],{"type":44,"value":2173},"transformer_config.py:2001-2005",{"type":44,"value":2175},". See also\n@skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md.",{"type":38,"tag":222,"props":2177,"children":2178},{},[2179,2184],{"type":38,"tag":1810,"props":2180,"children":2181},{},[2182],{"type":44,"value":2183},"Benchmark numbers are workload-specific",{"type":44,"value":2185},": graph wins are usually real\nwhen host overhead is visible, but the exact gain depends on batch shape,\nPP depth, recompute, dispatcher backend, and whether the eager baseline was\nalready optimized.",{"type":38,"tag":222,"props":2187,"children":2188},{},[2189,2194,2196,2202,2204,2210,2212,2218,2220,2226,2228,2234],{"type":38,"tag":1810,"props":2190,"children":2191},{},[2192],{"type":44,"value":2193},"A successful capture is not a speedup guarantee",{"type":44,"value":2195},": On 2026-05-18,\nQwen3 30B A3B H100 BF16 pretrain with the all-to-all dispatcher captured\nTE-scoped ",{"type":38,"tag":81,"props":2197,"children":2199},{"className":2198},[],[2200],{"type":44,"value":2201},"attn,moe_router,moe_preprocess",{"type":44,"value":2203}," graphs successfully (",{"type":38,"tag":81,"props":2205,"children":2207},{"className":2206},[],[2208],{"type":44,"value":2209},"48",{"type":44,"value":2211},"\ngraphable layers, about ",{"type":38,"tag":81,"props":2213,"children":2215},{"className":2214},[],[2216],{"type":44,"value":2217},"6.9 s",{"type":44,"value":2219}," capture time on rank 0), but replay\niterations 5-8 averaged ",{"type":38,"tag":81,"props":2221,"children":2223},{"className":2222},[],[2224],{"type":44,"value":2225},"42.00 s",{"type":44,"value":2227}," versus ",{"type":38,"tag":81,"props":2229,"children":2231},{"className":2230},[],[2232],{"type":44,"value":2233},"41.36 s",{"type":44,"value":2235}," for eager. Treat\nscoped graphs as a bring-up candidate and validate on the target stack.",{"type":38,"tag":53,"props":2237,"children":2239},{"id":2238},"verification",[2240],{"type":44,"value":2241},"Verification",{"type":38,"tag":325,"props":2243,"children":2245},{"id":2244},"unit-tests",[2246],{"type":44,"value":2247},"Unit tests",{"type":38,"tag":332,"props":2249,"children":2251},{"className":514,"code":2250,"language":516,"meta":337,"style":337},"uv run python -m pytest \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_config.py -k \"cuda_graph\" \\\n  tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py -k \"cuda_graph\" \\\n  tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_full_te_layer_autocast_spec.py -k \"cuda_graph\" -q\n",[2252],{"type":38,"tag":81,"props":2253,"children":2254},{"__ignoreMap":337},[2255,2284,2317,2345],{"type":38,"tag":343,"props":2256,"children":2257},{"class":345,"line":346},[2258,2262,2266,2270,2275,2280],{"type":38,"tag":343,"props":2259,"children":2260},{"style":526},[2261],{"type":44,"value":529},{"type":38,"tag":343,"props":2263,"children":2264},{"style":532},[2265],{"type":44,"value":535},{"type":38,"tag":343,"props":2267,"children":2268},{"style":532},[2269],{"type":44,"value":540},{"type":38,"tag":343,"props":2271,"children":2272},{"style":532},[2273],{"type":44,"value":2274}," -m",{"type":38,"tag":343,"props":2276,"children":2277},{"style":532},[2278],{"type":44,"value":2279}," pytest",{"type":38,"tag":343,"props":2281,"children":2282},{"style":548},[2283],{"type":44,"value":551},{"type":38,"tag":343,"props":2285,"children":2286},{"class":345,"line":355},[2287,2292,2297,2303,2308,2313],{"type":38,"tag":343,"props":2288,"children":2289},{"style":532},[2290],{"type":44,"value":2291},"  tests\u002Funit_tests\u002Ftraining\u002Ftest_config.py",{"type":38,"tag":343,"props":2293,"children":2294},{"style":532},[2295],{"type":44,"value":2296}," -k",{"type":38,"tag":343,"props":2298,"children":2300},{"style":2299},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[2301],{"type":44,"value":2302}," \"",{"type":38,"tag":343,"props":2304,"children":2305},{"style":532},[2306],{"type":44,"value":2307},"cuda_graph",{"type":38,"tag":343,"props":2309,"children":2310},{"style":2299},[2311],{"type":44,"value":2312},"\"",{"type":38,"tag":343,"props":2314,"children":2315},{"style":548},[2316],{"type":44,"value":551},{"type":38,"tag":343,"props":2318,"children":2319},{"class":345,"line":364},[2320,2325,2329,2333,2337,2341],{"type":38,"tag":343,"props":2321,"children":2322},{"style":532},[2323],{"type":44,"value":2324},"  tests\u002Funit_tests\u002Ftraining\u002Ftest_comm_overlap.py",{"type":38,"tag":343,"props":2326,"children":2327},{"style":532},[2328],{"type":44,"value":2296},{"type":38,"tag":343,"props":2330,"children":2331},{"style":2299},[2332],{"type":44,"value":2302},{"type":38,"tag":343,"props":2334,"children":2335},{"style":532},[2336],{"type":44,"value":2307},{"type":38,"tag":343,"props":2338,"children":2339},{"style":2299},[2340],{"type":44,"value":2312},{"type":38,"tag":343,"props":2342,"children":2343},{"style":548},[2344],{"type":44,"value":551},{"type":38,"tag":343,"props":2346,"children":2347},{"class":345,"line":373},[2348,2353,2357,2361,2365,2369],{"type":38,"tag":343,"props":2349,"children":2350},{"style":532},[2351],{"type":44,"value":2352},"  tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_full_te_layer_autocast_spec.py",{"type":38,"tag":343,"props":2354,"children":2355},{"style":532},[2356],{"type":44,"value":2296},{"type":38,"tag":343,"props":2358,"children":2359},{"style":2299},[2360],{"type":44,"value":2302},{"type":38,"tag":343,"props":2362,"children":2363},{"style":532},[2364],{"type":44,"value":2307},{"type":38,"tag":343,"props":2366,"children":2367},{"style":2299},[2368],{"type":44,"value":2312},{"type":38,"tag":343,"props":2370,"children":2371},{"style":532},[2372],{"type":44,"value":2373}," -q\n",{"type":38,"tag":325,"props":2375,"children":2377},{"id":2376},"functional-smoke-test-requires-gpu",[2378],{"type":44,"value":2379},"Functional smoke test (requires GPU)",{"type":38,"tag":332,"props":2381,"children":2383},{"className":514,"code":2382,"language":516,"meta":337,"style":337},"uv run python -m pytest \\\n  tests\u002Ffunctional_tests\u002Ftest_groups\u002Frecipes\u002Ftest_llama_recipes_pretrain_cuda_graphs.py -q\n",[2384],{"type":38,"tag":81,"props":2385,"children":2386},{"__ignoreMap":337},[2387,2414],{"type":38,"tag":343,"props":2388,"children":2389},{"class":345,"line":346},[2390,2394,2398,2402,2406,2410],{"type":38,"tag":343,"props":2391,"children":2392},{"style":526},[2393],{"type":44,"value":529},{"type":38,"tag":343,"props":2395,"children":2396},{"style":532},[2397],{"type":44,"value":535},{"type":38,"tag":343,"props":2399,"children":2400},{"style":532},[2401],{"type":44,"value":540},{"type":38,"tag":343,"props":2403,"children":2404},{"style":532},[2405],{"type":44,"value":2274},{"type":38,"tag":343,"props":2407,"children":2408},{"style":532},[2409],{"type":44,"value":2279},{"type":38,"tag":343,"props":2411,"children":2412},{"style":548},[2413],{"type":44,"value":551},{"type":38,"tag":343,"props":2415,"children":2416},{"class":345,"line":355},[2417,2422],{"type":38,"tag":343,"props":2418,"children":2419},{"style":532},[2420],{"type":44,"value":2421},"  tests\u002Ffunctional_tests\u002Ftest_groups\u002Frecipes\u002Ftest_llama_recipes_pretrain_cuda_graphs.py",{"type":38,"tag":343,"props":2423,"children":2424},{"style":532},[2425],{"type":44,"value":2373},{"type":38,"tag":325,"props":2427,"children":2429},{"id":2428},"success-criteria",[2430],{"type":44,"value":2431},"Success criteria",{"type":38,"tag":218,"props":2433,"children":2434},{},[2435,2454,2459],{"type":38,"tag":222,"props":2436,"children":2437},{},[2438,2440,2445,2446,2452],{"type":44,"value":2439},"Unit tests pass, covering config validation for both ",{"type":38,"tag":81,"props":2441,"children":2443},{"className":2442},[],[2444],{"type":44,"value":265},{"type":44,"value":1883},{"type":38,"tag":81,"props":2447,"children":2449},{"className":2448},[],[2450],{"type":44,"value":2451},"transformer_engine",{"type":44,"value":2453}," implementations.",{"type":38,"tag":222,"props":2455,"children":2456},{},[2457],{"type":44,"value":2458},"Functional test completes training steps with both CUDA graph\nimplementations.",{"type":38,"tag":222,"props":2460,"children":2461},{},[2462],{"type":44,"value":2463},"No NCCL errors or illegal memory access in logs.",{"type":38,"tag":2465,"props":2466,"children":2467},"style",{},[2468],{"type":44,"value":2469},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2471,"total":2573},[2472,2487,2501,2515,2527,2544,2559],{"slug":2473,"name":2473,"fn":2474,"description":2475,"org":2476,"tags":2477,"stars":20,"repoUrl":21,"updatedAt":2486},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2478,2481,2484,2485],{"name":2479,"slug":2480,"type":15},"Data Analysis","data-analysis",{"name":2482,"slug":2483,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:28:43.176466",{"slug":2488,"name":2488,"fn":2489,"description":2490,"org":2491,"tags":2492,"stars":20,"repoUrl":21,"updatedAt":2500},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2493,2496,2499],{"name":2494,"slug":2495,"type":15},"Deployment","deployment",{"name":2497,"slug":2498,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":2502,"name":2502,"fn":2503,"description":2504,"org":2505,"tags":2506,"stars":20,"repoUrl":21,"updatedAt":2514},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2507,2510,2511],{"name":2508,"slug":2509,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2512,"slug":2513,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":2516,"name":2516,"fn":2517,"description":2518,"org":2519,"tags":2520,"stars":20,"repoUrl":21,"updatedAt":2526},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2521,2522,2523],{"name":2479,"slug":2480,"type":15},{"name":9,"slug":8,"type":15},{"name":2524,"slug":2525,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":2528,"name":2528,"fn":2529,"description":2530,"org":2531,"tags":2532,"stars":20,"repoUrl":21,"updatedAt":2543},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2533,2536,2539,2540],{"name":2534,"slug":2535,"type":15},"Automation","automation",{"name":2537,"slug":2538,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":2541,"slug":2542,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":2545,"name":2545,"fn":2546,"description":2547,"org":2548,"tags":2549,"stars":20,"repoUrl":21,"updatedAt":2558},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2550,2551,2554,2555],{"name":2494,"slug":2495,"type":15},{"name":2552,"slug":2553,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":2556,"slug":2557,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":2560,"name":2560,"fn":2561,"description":2562,"org":2563,"tags":2564,"stars":20,"repoUrl":21,"updatedAt":2572},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2565,2566,2569],{"name":9,"slug":8,"type":15},{"name":2567,"slug":2568,"type":15},"Quantum Computing","quantum-computing",{"name":2570,"slug":2571,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":2575,"total":2725},[2576,2594,2609,2620,2632,2646,2659,2673,2684,2693,2707,2716],{"slug":2577,"name":2577,"fn":2578,"description":2579,"org":2580,"tags":2581,"stars":2591,"repoUrl":2592,"updatedAt":2593},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2582,2585,2588],{"name":2583,"slug":2584,"type":15},"Documentation","documentation",{"name":2586,"slug":2587,"type":15},"MCP","mcp",{"name":2589,"slug":2590,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2595,"name":2595,"fn":2596,"description":2597,"org":2598,"tags":2599,"stars":2606,"repoUrl":2607,"updatedAt":2608},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2600,2603,2604],{"name":2601,"slug":2602,"type":15},"Containers","containers",{"name":2494,"slug":2495,"type":15},{"name":2605,"slug":336,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2610,"name":2610,"fn":2611,"description":2612,"org":2613,"tags":2614,"stars":2606,"repoUrl":2607,"updatedAt":2619},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2615,2618],{"name":2616,"slug":2617,"type":15},"CI\u002FCD","ci-cd",{"name":2494,"slug":2495,"type":15},"2026-07-14T05:25:59.97109",{"slug":2621,"name":2621,"fn":2622,"description":2623,"org":2624,"tags":2625,"stars":2606,"repoUrl":2607,"updatedAt":2631},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2626,2627,2628],{"name":2616,"slug":2617,"type":15},{"name":2494,"slug":2495,"type":15},{"name":2629,"slug":2630,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2633,"name":2633,"fn":2634,"description":2635,"org":2636,"tags":2637,"stars":2606,"repoUrl":2607,"updatedAt":2645},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2638,2641,2642],{"name":2639,"slug":2640,"type":15},"Debugging","debugging",{"name":2629,"slug":2630,"type":15},{"name":2643,"slug":2644,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2647,"name":2647,"fn":2648,"description":2649,"org":2650,"tags":2651,"stars":2606,"repoUrl":2607,"updatedAt":2658},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2652,2655],{"name":2653,"slug":2654,"type":15},"Best Practices","best-practices",{"name":2656,"slug":2657,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2660,"name":2660,"fn":2661,"description":2662,"org":2663,"tags":2664,"stars":2606,"repoUrl":2607,"updatedAt":2672},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2665,2668,2671],{"name":2666,"slug":2667,"type":15},"Machine Learning","machine-learning",{"name":2669,"slug":2670,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2674,"name":2674,"fn":2675,"description":2676,"org":2677,"tags":2678,"stars":2606,"repoUrl":2607,"updatedAt":2683},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2679,2682],{"name":2680,"slug":2681,"type":15},"QA","qa",{"name":2524,"slug":2525,"type":15},"2026-07-14T05:25:53.673039",{"slug":2685,"name":2685,"fn":2686,"description":2687,"org":2688,"tags":2689,"stars":2606,"repoUrl":2607,"updatedAt":2692},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2690,2691],{"name":2494,"slug":2495,"type":15},{"name":2497,"slug":2498,"type":15},"2026-07-14T05:25:49.362534",{"slug":2694,"name":2694,"fn":2695,"description":2696,"org":2697,"tags":2698,"stars":2606,"repoUrl":2607,"updatedAt":2706},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2699,2702,2703],{"name":2700,"slug":2701,"type":15},"Code Review","code-review",{"name":2629,"slug":2630,"type":15},{"name":2704,"slug":2705,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2708,"name":2708,"fn":2709,"description":2710,"org":2711,"tags":2712,"stars":2606,"repoUrl":2607,"updatedAt":2715},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2713,2714],{"name":2680,"slug":2681,"type":15},{"name":2524,"slug":2525,"type":15},"2026-07-14T05:25:54.928983",{"slug":2717,"name":2717,"fn":2718,"description":2719,"org":2720,"tags":2721,"stars":2606,"repoUrl":2607,"updatedAt":2724},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2722,2723],{"name":2534,"slug":2535,"type":15},{"name":2616,"slug":2617,"type":15},"2026-07-30T05:29:03.275638",496]