[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nemo-mbridge-perf-memory-tuning":3,"mdc-msjavo-key":30,"related-repo-nvidia-nemo-mbridge-perf-memory-tuning":2190,"related-org-nvidia-nemo-mbridge-perf-memory-tuning":2294},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":19,"repoUrl":20,"updatedAt":21,"license":22,"forks":23,"topics":24,"repo":25,"sourceUrl":28,"mdContent":29},"nemo-mbridge-perf-memory-tuning","optimize GPU memory for Megatron Bridge","Techniques for reducing peak GPU memory in Megatron Bridge — expandable segments, PEFT + SP input re-gather, parallelism resizing, activation recompute, CPU offloading constraints, and common OOM fixes.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":17,"slug":18,"type":15},"Deep Learning","deep-learning",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:29:04.182992","Apache-2.0",281,[],{"repoUrl":20,"stars":19,"forks":23,"topics":26,"description":27},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fnemo-mbridge-perf-memory-tuning","---\nname: nemo-mbridge-perf-memory-tuning\ndescription: Techniques for reducing peak GPU memory in Megatron Bridge — expandable segments, PEFT + SP input re-gather, parallelism resizing, activation recompute, CPU offloading constraints, and common OOM fixes.\nlicense: Apache-2.0\nwhen_to_use: GPU OOM errors, reducing peak memory, reducing LoRA or PEFT activation memory with sequence parallelism, or tracing an OOM regression to a specific commit or config change; 'out of memory', 'OOM', 'memory fragmentation', 'expandable_segments', 'reduce GPU memory', 'LoRA memory', 'PEFT memory', 'sequence_parallel_input_regather', 'PYTORCH_CUDA_ALLOC_CONF'.\n---\n\n# Memory Tuning\n\nStable docs: @docs\u002Fparallelisms.md\nCard: @skills\u002Fnemo-mbridge-perf-memory-tuning\u002Fcard.yaml\n\n## What It Is\n\nGPU OOM failures during training often stem from memory **fragmentation** rather\nthan raw capacity.  PyTorch's default CUDA allocator can leave unusable gaps\nbetween allocations.  The single most effective fix is:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nThis tells PyTorch to use expandable (non-fixed-size) memory segments, which\ndramatically reduces fragmentation and often eliminates borderline OOM without\nany model or parallelism changes.\n\nBeyond fragmentation, actual peak memory is determined by:\n\n- **Parameter + optimizer state memory** — controlled by TP, PP, DP sharding\n  (distributed optimizer, FSDP)\n- **Activation memory** — controlled by activation recompute, sequence length,\n  micro-batch size, and PEFT-specific retention of gathered inputs\n- **Temporary \u002F workspace memory** — CUDA kernels, NCCL buffers, CUDA graphs\n\nFor configuration planning, use the Bridge theoretical estimator before launching\nlarge jobs:\n\n```python\nfrom megatron.bridge.training.utils.theoretical_memory_utils import estimate_training_memory\n\nestimate = estimate_training_memory(cfg, num_microbatches=num_microbatches)\n```\n\nThe estimator reports the most-loaded GPU shard and separates dense\u002Fembedding,\nrouted MoE expert, and activation components. It does not include allocator\nfragmentation, CUDA\u002FNCCL workspace, CUDA graph buffers, token imbalance, or\ndispatcher workspace, so validate final configs with runtime memory metrics.\n\n## Quick Decision\n\nWhen a training run OOMs or is close to the memory limit:\n\n1. **Set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` first.** This fixes\n   fragmentation-induced OOM with zero performance cost. Most Slurm launch\n   templates already include it.\n2. **For LoRA with sequence parallelism, enable input re-gather**\n   (`LoRA(sequence_parallel_input_regather=True)`). This avoids retaining the\n   full gathered LoRA-A input in every eligible layer; it has no effect when SP\n   is disabled.\n3. **Add selective activation recompute** (`recompute_modules=[core_attn]`) if\n   not already enabled. See @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md.\n4. **Avoid increasing TP** as a memory fix — doubling TP dramatically increases\n   NVLink all-reduce volume and often kills throughput (-28% on Llama3 70B).\n5. **Avoid increasing PP at the cost of DP** — halving DP doubles gradient\n   accumulation steps and hurts throughput (~6%).\n6. Consider `mlp` recompute if still OOM. Saves ~3 GB but costs ~16% GPU\n   utilization on large dense models (Llama3 70B).\n7. CPU offloading is **blocked when PP > 1**.\n\n## Enablement\n\n### Expandable segments (recommended first step)\n\nSet in the job's environment before launching:\n\n```bash\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nIn Slurm scripts this is typically placed alongside other env vars:\n\n```bash\nexport CUDA_DEVICE_MAX_CONNECTIONS=1\nexport NVTE_ALLOW_NONDETERMINISTIC_ALGO=1\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n```\n\nNo model config changes needed. Zero throughput cost.\n\n### Parallelism resizing\n\nIf the model genuinely does not fit (not fragmentation), adjust parallelism:\n\n| Strategy | Memory effect | Throughput cost | Notes |\n|---|---|---|---|\n| Increase PP (keeping DP) | Fewer layers per stage | Moderate (~6% if DP halved) | Only if GPU count allows |\n| Increase TP | Fewer params per GPU | Severe (-28% on 70B) | Last resort |\n| Distributed optimizer | Shards optimizer state across DP ranks | ~1-2% | Recommended for large models |\n| FSDP | Shards params + grads + optimizer | Varies | See @skills\u002Fnemo-mbridge-perf-megatron-fsdp\u002FSKILL.md |\n\n### Activation recompute\n\nSee @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full details.\n\n### PEFT + sequence-parallel input re-gather\n\nFor `LoRA` training with sequence parallelism, eligible column-parallel\n`linear_qkv` and `linear_fc1` adapters consume a gathered LayerNorm output.\nBecause LoRA-A is trainable, the default path retains that full gathered input\nuntil backward for the LoRA-A weight gradient.\n\nEnable input re-gather when constructing the PEFT config:\n\n```python\nfrom megatron.bridge.peft.lora import LoRA\n\ncfg.peft = LoRA(\n    # Keep the recipe's existing LoRA settings here.\n    sequence_parallel_input_regather=True,\n)\n```\n\nWith this option, forward still materializes the full input temporarily for the\nLoRA-A GEMM, but MCore autograd retains only its sequence-local shard. Backward\nasynchronously gathers the full input again, overlaps the collective with\ndgrad when possible, computes the LoRA-A weight gradient, and then reuses the\ntemporary communication buffer.\n\nThis is a memory-for-communication tradeoff, not conventional activation\ncheckpointing: no LayerNorm, attention, MLP, or LoRA GEMM is rerun. Some\nthroughput degradation is expected, and the benefit grows with the amount of\neligible LoRA-A activation retained. The option has no effect when sequence\nparallelism is disabled.\n\n### CPU offloading\n\n```python\ncfg.model.cpu_offloading = True\n```\n\n**Incompatible with PP > 1.** Only usable when `pipeline_model_parallel_size = 1`.\n\n## A Note on VPP\n\nVirtual pipeline parallelism (VPP) is primarily a **throughput** optimization\nthat reduces pipeline bubble overhead by interleaving smaller model chunks. Its\neffect on peak memory is minimal — changing VPP does not meaningfully change\nthe total activation, parameter, or optimizer memory on a GPU.\n\nIn earlier experiments we incorrectly attributed an OOM fix to VPP tuning\n(VPP 5→10). The actual fix was `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True`\nwhich eliminated memory fragmentation. The VPP=10 run actually used slightly\n**more** peak memory (60.2 GB vs 58.8 GB) but did not OOM because expandable\nsegments prevented fragmentation.\n\nVPP should be tuned for pipeline bubble reduction (see @docs\u002Fparallelisms.md),\nnot as a memory fix.\n\n## Compatibility and Constraints\n\n- `expandable_segments:True` is incompatible with `--use-nccl-ub` (NCCL\n  user-buffer registration). See Megatron-FSDP docs.\n- When using CUDA graphs with `expandable_segments:True`, set\n  `NCCL_GRAPH_REGISTER=0` (required on pre-Blackwell GPUs, enforced by MCore\n  `CudaGraphManager`).\n- CPU offloading requires `pipeline_model_parallel_size = 1`.\n- Distributed optimizer requires `use_distributed_optimizer = True` in the\n  optimizer config.\n- `sequence_parallel_input_regather` applies only to eligible non-expert\n  column-parallel LoRA-A projections. Row-parallel adapters, expert adapters,\n  TP=1, CUDA graphs, CPU activation offload, and overlapping full-layer or\n  selective MLP activation recompute fall back to the existing path.\n\n## Measured Results\n\nLlama3 70B SFT on 32x H100 80GB, FP8 (Current Scaling):\n- Baseline: TP=4, PP=4, VPP=5, DP=2, MBS=1, GBS=32, seq_len=4096\n- Golden GPU utilization: 709.93 TFLOP\u002Fs\u002FGPU\n- Regression threshold: 5%\n\n### Strategy comparison: parallelism changes for memory reduction\n\n| Experiment | TP | PP | VPP | DP | TFLOP\u002Fs\u002FGPU | vs Golden | Peak Mem (GB) | Result |\n|---|---|---|---|---|---|---|---|---|\n| Baseline | 4 | 4 | 5 | 2 | ~704 | -0.8% | 58.8 | OOM (fragmentation) |\n| More PP | 4 | 8 | 5 | 1 | 668.0 | -5.9% | 53.2 | Borderline perf |\n| More TP | 8 | 4 | 5 | 1 | 508.7 | -28.4% | 50.2 | Severe regression |\n| Baseline + expandable_segments | 4 | 4 | 5 | 2 | ~704 | -0.8% | ~59 | **Passed** |\n\nKey takeaways:\n\n- **`expandable_segments:True` is the winner.** The baseline OOM was caused by\n  memory fragmentation, not insufficient capacity. Setting this env var\n  eliminated the OOM with zero throughput cost and no parallelism changes.\n- **PP=8 works for memory but loses DP** (2→1), meaning 32 gradient accumulation\n  steps per batch, which hurts throughput by ~6%.\n- **TP=8 is catastrophic** (-28%) because doubling TP increases all-reduce\n  communication volume proportionally across NVLink, and DP=1 means no\n  micro-batch overlap.\n\n### CPU offloading: blocked\n\n| Experiment | offload_layers | Result |\n|---|---|---|\n| Exp 4 | 2 | Incompatible (PP > 1) |\n| Exp 5 | 4 | Incompatible (PP > 1) |\n| Exp 6 | 6 | Incompatible (PP > 1) |\n\n`ValueError: Currently there is no support for Pipeline parallelism with CPU\noffloading.` This approach is blocked for any model using PP > 1.\n\n### Activation recompute: expensive alternative\n\nSelective activation recompute with `mlp` saved ~3 GB peak memory but cost\n~16% GPU utilization on this workload. See\n@skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full results.\n\n### LoRA + SP input re-gather\n\nReal-checkpoint H100 training with SQuAD showed lower peak memory in all tested\nconfigurations, with workload-dependent throughput cost:\n\n| Model\u002Fconfig | Baseline peak | Input re-gather peak | Memory saved | Throughput change |\n|---|---:|---:|---:|---:|\n| Qwen3-8B, TP2, seq 8192 | 47.545 GB | 42.814 GB | 4.731 GB (10.0%) | -6.74% |\n| Qwen3-30B-A3B, TP4\u002FEP4 | 29.890 GB | 28.321 GB | 1.569 GB (5.2%) | -2.89% |\n| GPT-OSS-120B, TP2\u002FEP8 | 52.185 GB | 51.371 GB | 0.814 GB (1.6%) | -0.34% |\n\nAll runs had finite losses with zero skipped or NaN iterations. Two-rank BF16\nand FP32 checks matched the baseline for outputs, input gradients, LoRA-A and\nLoRA-B gradients, and two-microbatch fused `main_grad` accumulation.\n\n## Code Anchors\n\n### LoRA sequence-parallel input re-gather\n\n```text\nsrc\u002Fmegatron\u002Fbridge\u002Fpeft\u002Flora.py\n    LoRA.sequence_parallel_input_regather\n\nsrc\u002Fmegatron\u002Fbridge\u002Fpeft\u002Futils.py\n    ParallelLinearAdapter._sequence_parallel_input_regather_eligibility()\n    ParallelLinearAdapter.forward()\n```\n\n### CPU offloading PP incompatibility (MCore)\n\n```1303:1306:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n        if self.cpu_offloading and self.pipeline_model_parallel_size > 1:\n            raise ValueError(\n                \"Currently there is no support for Pipeline parallelism with CPU offloading\"\n            )\n```\n\n### VPP config and layer divisibility validation (MCore)\n\n```1581:1592:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n            if pipeline_parallel_size and self.virtual_pipeline_model_parallel_size is not None:\n                num_layers_per_middle_pipeline_rank = num_layers \u002F\u002F pipeline_parallel_size\n                if (\n                    not num_layers_per_middle_pipeline_rank\n                    % self.virtual_pipeline_model_parallel_size\n                    == 0\n                ):\n                    raise ValueError(\n                        f\"number of layers on each middle pipeline rank:\"\n                        f\"{num_layers_per_middle_pipeline_rank} must be divisible by virtual\"\n                        f\"pipeline parallel degree {self.virtual_pipeline_model_parallel_size}\"\n                    )\n```\n\n### Parallelism docs on interleaved pipeline schedule\n\n```116:124:docs\u002Fparallelisms.md\nTo minimize the pipeline bubble, the computation on each GPU can be divided into multiple subsets of layers (referred to as model chunks), rather than a single contiguous block. Enable this by setting `virtual_pipeline_model_parallel_size`:\n\nmodel_config = GPTModelProvider(\n    pipeline_model_parallel_size=4,\n    virtual_pipeline_model_parallel_size=2,  # 2 model chunks per pipeline stage\n    # ... other model parameters\n)\n```\n\n## Failure Diagnosis\n\n| Symptom | Cause | Confirm | Fix |\n|---|---|---|---|\n| OOM on a single rank despite headroom on others | Memory fragmentation | check if `expandable_segments:True` is set | set `PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True` |\n| OOM with `expandable_segments` already set | Genuine capacity limit | check `nvidia-smi` for param\u002Foptimizer memory | increase PP, use distributed optimizer, or add recompute |\n| Estimated memory exceeds GPU capacity before launch | model state or activations genuinely too large | run `estimate_training_memory` and inspect the largest component | adjust PP\u002FTP\u002FCP\u002FEP, distributed optimizer, or recompute before launching |\n| LoRA + SP retains unexpectedly high activation memory | full gathered LoRA-A inputs are retained until backward | check whether `cfg.peft.sequence_parallel_input_regather` is enabled and the target is eligible | set `LoRA(sequence_parallel_input_regather=True)`; verify fallback constraints |\n| `ValueError: PP + CPU offloading` | using cpu_offloading with PP > 1 | check PP config | disable CPU offloading or set PP=1 |\n| `RuntimeError` with `--use-nccl-ub` + expandable segments | NCCL UB incompatible with expandable allocator | check env vars | remove `expandable_segments:True` or disable `--use-nccl-ub` |\n\n## Known Limitations\n\n- CPU offloading is blocked when PP > 1\n- Parallelism resizing (TP\u002FPP) often has significant throughput costs\n- The theoretical estimator is formula-based and does not replace runtime\n  profiling or CUDA memory reports\n- LoRA input re-gather does not cover row-parallel or expert adapters and may\n  have negligible benefit when few eligible LoRA-A activations dominate memory\n\n## Verification\n\nQuick check that `expandable_segments:True` is active:\n\n```python\nimport os\nassert \"expandable_segments:True\" in os.environ.get(\"PYTORCH_CUDA_ALLOC_CONF\", \"\")\n```\n\nFor Slurm jobs, verify the env var is exported before the training command\nin the launch script.\n\nFor LoRA + SP input re-gather, run the focused configuration tests and the real\ntwo-rank MCore backward-parity test:\n\n```bash\nuv run python -m pytest \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_utils.py -k \"sequence_parallel_input_regather\" \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_lora.py -k \"sequence_parallel_input_regather\"\n\nuv run python -m torch.distributed.run --nproc_per_node=2 -m pytest \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_lora_sp_input_regather_distributed.py\n```\n",{"data":31,"body":33},{"name":4,"description":6,"license":22,"when_to_use":32},"GPU OOM errors, reducing peak memory, reducing LoRA or PEFT activation memory with sequence parallelism, or tracing an OOM regression to a specific commit or config change; 'out of memory', 'OOM', 'memory fragmentation', 'expandable_segments', 'reduce GPU memory', 'LoRA memory', 'PEFT memory', 'sequence_parallel_input_regather', 'PYTORCH_CUDA_ALLOC_CONF'.",{"type":34,"children":35},"root",[36,45,51,58,71,112,117,122,157,162,198,203,209,214,317,323,330,335,360,365,433,438,444,449,579,585,590,596,625,630,687,692,697,703,717,734,740,752,771,776,782,867,873,878,896,902,1141,1146,1184,1190,1265,1276,1282,1294,1300,1305,1427,1440,1446,1452,1462,1468,1509,1515,1626,1632,1695,1701,1950,1956,1979,1985,1997,2020,2025,2030,2184],{"type":37,"tag":38,"props":39,"children":41},"element","h1",{"id":40},"memory-tuning",[42],{"type":43,"value":44},"text","Memory Tuning",{"type":37,"tag":46,"props":47,"children":48},"p",{},[49],{"type":43,"value":50},"Stable docs: @docs\u002Fparallelisms.md\nCard: @skills\u002Fnemo-mbridge-perf-memory-tuning\u002Fcard.yaml",{"type":37,"tag":52,"props":53,"children":55},"h2",{"id":54},"what-it-is",[56],{"type":43,"value":57},"What It Is",{"type":37,"tag":46,"props":59,"children":60},{},[61,63,69],{"type":43,"value":62},"GPU OOM failures during training often stem from memory ",{"type":37,"tag":64,"props":65,"children":66},"strong",{},[67],{"type":43,"value":68},"fragmentation",{"type":43,"value":70}," rather\nthan raw capacity.  PyTorch's default CUDA allocator can leave unusable gaps\nbetween allocations.  The single most effective fix is:",{"type":37,"tag":72,"props":73,"children":78},"pre",{"className":74,"code":75,"language":76,"meta":77,"style":77},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","export PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n","bash","",[79],{"type":37,"tag":80,"props":81,"children":82},"code",{"__ignoreMap":77},[83],{"type":37,"tag":84,"props":85,"children":88},"span",{"class":86,"line":87},"line",1,[89,95,101,107],{"type":37,"tag":84,"props":90,"children":92},{"style":91},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[93],{"type":43,"value":94},"export",{"type":37,"tag":84,"props":96,"children":98},{"style":97},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[99],{"type":43,"value":100}," PYTORCH_CUDA_ALLOC_CONF",{"type":37,"tag":84,"props":102,"children":104},{"style":103},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[105],{"type":43,"value":106},"=",{"type":37,"tag":84,"props":108,"children":109},{"style":97},[110],{"type":43,"value":111},"expandable_segments:True\n",{"type":37,"tag":46,"props":113,"children":114},{},[115],{"type":43,"value":116},"This tells PyTorch to use expandable (non-fixed-size) memory segments, which\ndramatically reduces fragmentation and often eliminates borderline OOM without\nany model or parallelism changes.",{"type":37,"tag":46,"props":118,"children":119},{},[120],{"type":43,"value":121},"Beyond fragmentation, actual peak memory is determined by:",{"type":37,"tag":123,"props":124,"children":125},"ul",{},[126,137,147],{"type":37,"tag":127,"props":128,"children":129},"li",{},[130,135],{"type":37,"tag":64,"props":131,"children":132},{},[133],{"type":43,"value":134},"Parameter + optimizer state memory",{"type":43,"value":136}," — controlled by TP, PP, DP sharding\n(distributed optimizer, FSDP)",{"type":37,"tag":127,"props":138,"children":139},{},[140,145],{"type":37,"tag":64,"props":141,"children":142},{},[143],{"type":43,"value":144},"Activation memory",{"type":43,"value":146}," — controlled by activation recompute, sequence length,\nmicro-batch size, and PEFT-specific retention of gathered inputs",{"type":37,"tag":127,"props":148,"children":149},{},[150,155],{"type":37,"tag":64,"props":151,"children":152},{},[153],{"type":43,"value":154},"Temporary \u002F workspace memory",{"type":43,"value":156}," — CUDA kernels, NCCL buffers, CUDA graphs",{"type":37,"tag":46,"props":158,"children":159},{},[160],{"type":43,"value":161},"For configuration planning, use the Bridge theoretical estimator before launching\nlarge jobs:",{"type":37,"tag":72,"props":163,"children":167},{"className":164,"code":165,"language":166,"meta":77,"style":77},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from megatron.bridge.training.utils.theoretical_memory_utils import estimate_training_memory\n\nestimate = estimate_training_memory(cfg, num_microbatches=num_microbatches)\n","python",[168],{"type":37,"tag":80,"props":169,"children":170},{"__ignoreMap":77},[171,179,189],{"type":37,"tag":84,"props":172,"children":173},{"class":86,"line":87},[174],{"type":37,"tag":84,"props":175,"children":176},{},[177],{"type":43,"value":178},"from megatron.bridge.training.utils.theoretical_memory_utils import estimate_training_memory\n",{"type":37,"tag":84,"props":180,"children":182},{"class":86,"line":181},2,[183],{"type":37,"tag":84,"props":184,"children":186},{"emptyLinePlaceholder":185},true,[187],{"type":43,"value":188},"\n",{"type":37,"tag":84,"props":190,"children":192},{"class":86,"line":191},3,[193],{"type":37,"tag":84,"props":194,"children":195},{},[196],{"type":43,"value":197},"estimate = estimate_training_memory(cfg, num_microbatches=num_microbatches)\n",{"type":37,"tag":46,"props":199,"children":200},{},[201],{"type":43,"value":202},"The estimator reports the most-loaded GPU shard and separates dense\u002Fembedding,\nrouted MoE expert, and activation components. It does not include allocator\nfragmentation, CUDA\u002FNCCL workspace, CUDA graph buffers, token imbalance, or\ndispatcher workspace, so validate final configs with runtime memory metrics.",{"type":37,"tag":52,"props":204,"children":206},{"id":205},"quick-decision",[207],{"type":43,"value":208},"Quick Decision",{"type":37,"tag":46,"props":210,"children":211},{},[212],{"type":43,"value":213},"When a training run OOMs or is close to the memory limit:",{"type":37,"tag":215,"props":216,"children":217},"ol",{},[218,236,254,272,282,292,305],{"type":37,"tag":127,"props":219,"children":220},{},[221,234],{"type":37,"tag":64,"props":222,"children":223},{},[224,226,232],{"type":43,"value":225},"Set ",{"type":37,"tag":80,"props":227,"children":229},{"className":228},[],[230],{"type":43,"value":231},"PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True",{"type":43,"value":233}," first.",{"type":43,"value":235}," This fixes\nfragmentation-induced OOM with zero performance cost. Most Slurm launch\ntemplates already include it.",{"type":37,"tag":127,"props":237,"children":238},{},[239,244,246,252],{"type":37,"tag":64,"props":240,"children":241},{},[242],{"type":43,"value":243},"For LoRA with sequence parallelism, enable input re-gather",{"type":43,"value":245},"\n(",{"type":37,"tag":80,"props":247,"children":249},{"className":248},[],[250],{"type":43,"value":251},"LoRA(sequence_parallel_input_regather=True)",{"type":43,"value":253},"). This avoids retaining the\nfull gathered LoRA-A input in every eligible layer; it has no effect when SP\nis disabled.",{"type":37,"tag":127,"props":255,"children":256},{},[257,262,264,270],{"type":37,"tag":64,"props":258,"children":259},{},[260],{"type":43,"value":261},"Add selective activation recompute",{"type":43,"value":263}," (",{"type":37,"tag":80,"props":265,"children":267},{"className":266},[],[268],{"type":43,"value":269},"recompute_modules=[core_attn]",{"type":43,"value":271},") if\nnot already enabled. See @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md.",{"type":37,"tag":127,"props":273,"children":274},{},[275,280],{"type":37,"tag":64,"props":276,"children":277},{},[278],{"type":43,"value":279},"Avoid increasing TP",{"type":43,"value":281}," as a memory fix — doubling TP dramatically increases\nNVLink all-reduce volume and often kills throughput (-28% on Llama3 70B).",{"type":37,"tag":127,"props":283,"children":284},{},[285,290],{"type":37,"tag":64,"props":286,"children":287},{},[288],{"type":43,"value":289},"Avoid increasing PP at the cost of DP",{"type":43,"value":291}," — halving DP doubles gradient\naccumulation steps and hurts throughput (~6%).",{"type":37,"tag":127,"props":293,"children":294},{},[295,297,303],{"type":43,"value":296},"Consider ",{"type":37,"tag":80,"props":298,"children":300},{"className":299},[],[301],{"type":43,"value":302},"mlp",{"type":43,"value":304}," recompute if still OOM. Saves ~3 GB but costs ~16% GPU\nutilization on large dense models (Llama3 70B).",{"type":37,"tag":127,"props":306,"children":307},{},[308,310,315],{"type":43,"value":309},"CPU offloading is ",{"type":37,"tag":64,"props":311,"children":312},{},[313],{"type":43,"value":314},"blocked when PP > 1",{"type":43,"value":316},".",{"type":37,"tag":52,"props":318,"children":320},{"id":319},"enablement",[321],{"type":43,"value":322},"Enablement",{"type":37,"tag":324,"props":325,"children":327},"h3",{"id":326},"expandable-segments-recommended-first-step",[328],{"type":43,"value":329},"Expandable segments (recommended first step)",{"type":37,"tag":46,"props":331,"children":332},{},[333],{"type":43,"value":334},"Set in the job's environment before launching:",{"type":37,"tag":72,"props":336,"children":337},{"className":74,"code":75,"language":76,"meta":77,"style":77},[338],{"type":37,"tag":80,"props":339,"children":340},{"__ignoreMap":77},[341],{"type":37,"tag":84,"props":342,"children":343},{"class":86,"line":87},[344,348,352,356],{"type":37,"tag":84,"props":345,"children":346},{"style":91},[347],{"type":43,"value":94},{"type":37,"tag":84,"props":349,"children":350},{"style":97},[351],{"type":43,"value":100},{"type":37,"tag":84,"props":353,"children":354},{"style":103},[355],{"type":43,"value":106},{"type":37,"tag":84,"props":357,"children":358},{"style":97},[359],{"type":43,"value":111},{"type":37,"tag":46,"props":361,"children":362},{},[363],{"type":43,"value":364},"In Slurm scripts this is typically placed alongside other env vars:",{"type":37,"tag":72,"props":366,"children":368},{"className":74,"code":367,"language":76,"meta":77,"style":77},"export CUDA_DEVICE_MAX_CONNECTIONS=1\nexport NVTE_ALLOW_NONDETERMINISTIC_ALGO=1\nexport PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True\n",[369],{"type":37,"tag":80,"props":370,"children":371},{"__ignoreMap":77},[372,394,414],{"type":37,"tag":84,"props":373,"children":374},{"class":86,"line":87},[375,379,384,388],{"type":37,"tag":84,"props":376,"children":377},{"style":91},[378],{"type":43,"value":94},{"type":37,"tag":84,"props":380,"children":381},{"style":97},[382],{"type":43,"value":383}," CUDA_DEVICE_MAX_CONNECTIONS",{"type":37,"tag":84,"props":385,"children":386},{"style":103},[387],{"type":43,"value":106},{"type":37,"tag":84,"props":389,"children":391},{"style":390},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[392],{"type":43,"value":393},"1\n",{"type":37,"tag":84,"props":395,"children":396},{"class":86,"line":181},[397,401,406,410],{"type":37,"tag":84,"props":398,"children":399},{"style":91},[400],{"type":43,"value":94},{"type":37,"tag":84,"props":402,"children":403},{"style":97},[404],{"type":43,"value":405}," NVTE_ALLOW_NONDETERMINISTIC_ALGO",{"type":37,"tag":84,"props":407,"children":408},{"style":103},[409],{"type":43,"value":106},{"type":37,"tag":84,"props":411,"children":412},{"style":390},[413],{"type":43,"value":393},{"type":37,"tag":84,"props":415,"children":416},{"class":86,"line":191},[417,421,425,429],{"type":37,"tag":84,"props":418,"children":419},{"style":91},[420],{"type":43,"value":94},{"type":37,"tag":84,"props":422,"children":423},{"style":97},[424],{"type":43,"value":100},{"type":37,"tag":84,"props":426,"children":427},{"style":103},[428],{"type":43,"value":106},{"type":37,"tag":84,"props":430,"children":431},{"style":97},[432],{"type":43,"value":111},{"type":37,"tag":46,"props":434,"children":435},{},[436],{"type":43,"value":437},"No model config changes needed. Zero throughput cost.",{"type":37,"tag":324,"props":439,"children":441},{"id":440},"parallelism-resizing",[442],{"type":43,"value":443},"Parallelism resizing",{"type":37,"tag":46,"props":445,"children":446},{},[447],{"type":43,"value":448},"If the model genuinely does not fit (not fragmentation), adjust parallelism:",{"type":37,"tag":450,"props":451,"children":452},"table",{},[453,482],{"type":37,"tag":454,"props":455,"children":456},"thead",{},[457],{"type":37,"tag":458,"props":459,"children":460},"tr",{},[461,467,472,477],{"type":37,"tag":462,"props":463,"children":464},"th",{},[465],{"type":43,"value":466},"Strategy",{"type":37,"tag":462,"props":468,"children":469},{},[470],{"type":43,"value":471},"Memory effect",{"type":37,"tag":462,"props":473,"children":474},{},[475],{"type":43,"value":476},"Throughput cost",{"type":37,"tag":462,"props":478,"children":479},{},[480],{"type":43,"value":481},"Notes",{"type":37,"tag":483,"props":484,"children":485},"tbody",{},[486,510,533,556],{"type":37,"tag":458,"props":487,"children":488},{},[489,495,500,505],{"type":37,"tag":490,"props":491,"children":492},"td",{},[493],{"type":43,"value":494},"Increase PP (keeping DP)",{"type":37,"tag":490,"props":496,"children":497},{},[498],{"type":43,"value":499},"Fewer layers per stage",{"type":37,"tag":490,"props":501,"children":502},{},[503],{"type":43,"value":504},"Moderate (~6% if DP halved)",{"type":37,"tag":490,"props":506,"children":507},{},[508],{"type":43,"value":509},"Only if GPU count allows",{"type":37,"tag":458,"props":511,"children":512},{},[513,518,523,528],{"type":37,"tag":490,"props":514,"children":515},{},[516],{"type":43,"value":517},"Increase TP",{"type":37,"tag":490,"props":519,"children":520},{},[521],{"type":43,"value":522},"Fewer params per GPU",{"type":37,"tag":490,"props":524,"children":525},{},[526],{"type":43,"value":527},"Severe (-28% on 70B)",{"type":37,"tag":490,"props":529,"children":530},{},[531],{"type":43,"value":532},"Last resort",{"type":37,"tag":458,"props":534,"children":535},{},[536,541,546,551],{"type":37,"tag":490,"props":537,"children":538},{},[539],{"type":43,"value":540},"Distributed optimizer",{"type":37,"tag":490,"props":542,"children":543},{},[544],{"type":43,"value":545},"Shards optimizer state across DP ranks",{"type":37,"tag":490,"props":547,"children":548},{},[549],{"type":43,"value":550},"~1-2%",{"type":37,"tag":490,"props":552,"children":553},{},[554],{"type":43,"value":555},"Recommended for large models",{"type":37,"tag":458,"props":557,"children":558},{},[559,564,569,574],{"type":37,"tag":490,"props":560,"children":561},{},[562],{"type":43,"value":563},"FSDP",{"type":37,"tag":490,"props":565,"children":566},{},[567],{"type":43,"value":568},"Shards params + grads + optimizer",{"type":37,"tag":490,"props":570,"children":571},{},[572],{"type":43,"value":573},"Varies",{"type":37,"tag":490,"props":575,"children":576},{},[577],{"type":43,"value":578},"See @skills\u002Fnemo-mbridge-perf-megatron-fsdp\u002FSKILL.md",{"type":37,"tag":324,"props":580,"children":582},{"id":581},"activation-recompute",[583],{"type":43,"value":584},"Activation recompute",{"type":37,"tag":46,"props":586,"children":587},{},[588],{"type":43,"value":589},"See @skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full details.",{"type":37,"tag":324,"props":591,"children":593},{"id":592},"peft-sequence-parallel-input-re-gather",[594],{"type":43,"value":595},"PEFT + sequence-parallel input re-gather",{"type":37,"tag":46,"props":597,"children":598},{},[599,601,607,609,615,617,623],{"type":43,"value":600},"For ",{"type":37,"tag":80,"props":602,"children":604},{"className":603},[],[605],{"type":43,"value":606},"LoRA",{"type":43,"value":608}," training with sequence parallelism, eligible column-parallel\n",{"type":37,"tag":80,"props":610,"children":612},{"className":611},[],[613],{"type":43,"value":614},"linear_qkv",{"type":43,"value":616}," and ",{"type":37,"tag":80,"props":618,"children":620},{"className":619},[],[621],{"type":43,"value":622},"linear_fc1",{"type":43,"value":624}," adapters consume a gathered LayerNorm output.\nBecause LoRA-A is trainable, the default path retains that full gathered input\nuntil backward for the LoRA-A weight gradient.",{"type":37,"tag":46,"props":626,"children":627},{},[628],{"type":43,"value":629},"Enable input re-gather when constructing the PEFT config:",{"type":37,"tag":72,"props":631,"children":633},{"className":164,"code":632,"language":166,"meta":77,"style":77},"from megatron.bridge.peft.lora import LoRA\n\ncfg.peft = LoRA(\n    # Keep the recipe's existing LoRA settings here.\n    sequence_parallel_input_regather=True,\n)\n",[634],{"type":37,"tag":80,"props":635,"children":636},{"__ignoreMap":77},[637,645,652,660,669,678],{"type":37,"tag":84,"props":638,"children":639},{"class":86,"line":87},[640],{"type":37,"tag":84,"props":641,"children":642},{},[643],{"type":43,"value":644},"from megatron.bridge.peft.lora import LoRA\n",{"type":37,"tag":84,"props":646,"children":647},{"class":86,"line":181},[648],{"type":37,"tag":84,"props":649,"children":650},{"emptyLinePlaceholder":185},[651],{"type":43,"value":188},{"type":37,"tag":84,"props":653,"children":654},{"class":86,"line":191},[655],{"type":37,"tag":84,"props":656,"children":657},{},[658],{"type":43,"value":659},"cfg.peft = LoRA(\n",{"type":37,"tag":84,"props":661,"children":663},{"class":86,"line":662},4,[664],{"type":37,"tag":84,"props":665,"children":666},{},[667],{"type":43,"value":668},"    # Keep the recipe's existing LoRA settings here.\n",{"type":37,"tag":84,"props":670,"children":672},{"class":86,"line":671},5,[673],{"type":37,"tag":84,"props":674,"children":675},{},[676],{"type":43,"value":677},"    sequence_parallel_input_regather=True,\n",{"type":37,"tag":84,"props":679,"children":681},{"class":86,"line":680},6,[682],{"type":37,"tag":84,"props":683,"children":684},{},[685],{"type":43,"value":686},")\n",{"type":37,"tag":46,"props":688,"children":689},{},[690],{"type":43,"value":691},"With this option, forward still materializes the full input temporarily for the\nLoRA-A GEMM, but MCore autograd retains only its sequence-local shard. Backward\nasynchronously gathers the full input again, overlaps the collective with\ndgrad when possible, computes the LoRA-A weight gradient, and then reuses the\ntemporary communication buffer.",{"type":37,"tag":46,"props":693,"children":694},{},[695],{"type":43,"value":696},"This is a memory-for-communication tradeoff, not conventional activation\ncheckpointing: no LayerNorm, attention, MLP, or LoRA GEMM is rerun. Some\nthroughput degradation is expected, and the benefit grows with the amount of\neligible LoRA-A activation retained. The option has no effect when sequence\nparallelism is disabled.",{"type":37,"tag":324,"props":698,"children":700},{"id":699},"cpu-offloading",[701],{"type":43,"value":702},"CPU offloading",{"type":37,"tag":72,"props":704,"children":706},{"className":164,"code":705,"language":166,"meta":77,"style":77},"cfg.model.cpu_offloading = True\n",[707],{"type":37,"tag":80,"props":708,"children":709},{"__ignoreMap":77},[710],{"type":37,"tag":84,"props":711,"children":712},{"class":86,"line":87},[713],{"type":37,"tag":84,"props":714,"children":715},{},[716],{"type":43,"value":705},{"type":37,"tag":46,"props":718,"children":719},{},[720,725,727,733],{"type":37,"tag":64,"props":721,"children":722},{},[723],{"type":43,"value":724},"Incompatible with PP > 1.",{"type":43,"value":726}," Only usable when ",{"type":37,"tag":80,"props":728,"children":730},{"className":729},[],[731],{"type":43,"value":732},"pipeline_model_parallel_size = 1",{"type":43,"value":316},{"type":37,"tag":52,"props":735,"children":737},{"id":736},"a-note-on-vpp",[738],{"type":43,"value":739},"A Note on VPP",{"type":37,"tag":46,"props":741,"children":742},{},[743,745,750],{"type":43,"value":744},"Virtual pipeline parallelism (VPP) is primarily a ",{"type":37,"tag":64,"props":746,"children":747},{},[748],{"type":43,"value":749},"throughput",{"type":43,"value":751}," optimization\nthat reduces pipeline bubble overhead by interleaving smaller model chunks. Its\neffect on peak memory is minimal — changing VPP does not meaningfully change\nthe total activation, parameter, or optimizer memory on a GPU.",{"type":37,"tag":46,"props":753,"children":754},{},[755,757,762,764,769],{"type":43,"value":756},"In earlier experiments we incorrectly attributed an OOM fix to VPP tuning\n(VPP 5→10). The actual fix was ",{"type":37,"tag":80,"props":758,"children":760},{"className":759},[],[761],{"type":43,"value":231},{"type":43,"value":763},"\nwhich eliminated memory fragmentation. The VPP=10 run actually used slightly\n",{"type":37,"tag":64,"props":765,"children":766},{},[767],{"type":43,"value":768},"more",{"type":43,"value":770}," peak memory (60.2 GB vs 58.8 GB) but did not OOM because expandable\nsegments prevented fragmentation.",{"type":37,"tag":46,"props":772,"children":773},{},[774],{"type":43,"value":775},"VPP should be tuned for pipeline bubble reduction (see @docs\u002Fparallelisms.md),\nnot as a memory fix.",{"type":37,"tag":52,"props":777,"children":779},{"id":778},"compatibility-and-constraints",[780],{"type":43,"value":781},"Compatibility and Constraints",{"type":37,"tag":123,"props":783,"children":784},{},[785,804,832,843,856],{"type":37,"tag":127,"props":786,"children":787},{},[788,794,796,802],{"type":37,"tag":80,"props":789,"children":791},{"className":790},[],[792],{"type":43,"value":793},"expandable_segments:True",{"type":43,"value":795}," is incompatible with ",{"type":37,"tag":80,"props":797,"children":799},{"className":798},[],[800],{"type":43,"value":801},"--use-nccl-ub",{"type":43,"value":803}," (NCCL\nuser-buffer registration). See Megatron-FSDP docs.",{"type":37,"tag":127,"props":805,"children":806},{},[807,809,814,816,822,824,830],{"type":43,"value":808},"When using CUDA graphs with ",{"type":37,"tag":80,"props":810,"children":812},{"className":811},[],[813],{"type":43,"value":793},{"type":43,"value":815},", set\n",{"type":37,"tag":80,"props":817,"children":819},{"className":818},[],[820],{"type":43,"value":821},"NCCL_GRAPH_REGISTER=0",{"type":43,"value":823}," (required on pre-Blackwell GPUs, enforced by MCore\n",{"type":37,"tag":80,"props":825,"children":827},{"className":826},[],[828],{"type":43,"value":829},"CudaGraphManager",{"type":43,"value":831},").",{"type":37,"tag":127,"props":833,"children":834},{},[835,837,842],{"type":43,"value":836},"CPU offloading requires ",{"type":37,"tag":80,"props":838,"children":840},{"className":839},[],[841],{"type":43,"value":732},{"type":43,"value":316},{"type":37,"tag":127,"props":844,"children":845},{},[846,848,854],{"type":43,"value":847},"Distributed optimizer requires ",{"type":37,"tag":80,"props":849,"children":851},{"className":850},[],[852],{"type":43,"value":853},"use_distributed_optimizer = True",{"type":43,"value":855}," in the\noptimizer config.",{"type":37,"tag":127,"props":857,"children":858},{},[859,865],{"type":37,"tag":80,"props":860,"children":862},{"className":861},[],[863],{"type":43,"value":864},"sequence_parallel_input_regather",{"type":43,"value":866}," applies only to eligible non-expert\ncolumn-parallel LoRA-A projections. Row-parallel adapters, expert adapters,\nTP=1, CUDA graphs, CPU activation offload, and overlapping full-layer or\nselective MLP activation recompute fall back to the existing path.",{"type":37,"tag":52,"props":868,"children":870},{"id":869},"measured-results",[871],{"type":43,"value":872},"Measured Results",{"type":37,"tag":46,"props":874,"children":875},{},[876],{"type":43,"value":877},"Llama3 70B SFT on 32x H100 80GB, FP8 (Current Scaling):",{"type":37,"tag":123,"props":879,"children":880},{},[881,886,891],{"type":37,"tag":127,"props":882,"children":883},{},[884],{"type":43,"value":885},"Baseline: TP=4, PP=4, VPP=5, DP=2, MBS=1, GBS=32, seq_len=4096",{"type":37,"tag":127,"props":887,"children":888},{},[889],{"type":43,"value":890},"Golden GPU utilization: 709.93 TFLOP\u002Fs\u002FGPU",{"type":37,"tag":127,"props":892,"children":893},{},[894],{"type":43,"value":895},"Regression threshold: 5%",{"type":37,"tag":324,"props":897,"children":899},{"id":898},"strategy-comparison-parallelism-changes-for-memory-reduction",[900],{"type":43,"value":901},"Strategy comparison: parallelism changes for memory reduction",{"type":37,"tag":450,"props":903,"children":904},{},[905,956],{"type":37,"tag":454,"props":906,"children":907},{},[908],{"type":37,"tag":458,"props":909,"children":910},{},[911,916,921,926,931,936,941,946,951],{"type":37,"tag":462,"props":912,"children":913},{},[914],{"type":43,"value":915},"Experiment",{"type":37,"tag":462,"props":917,"children":918},{},[919],{"type":43,"value":920},"TP",{"type":37,"tag":462,"props":922,"children":923},{},[924],{"type":43,"value":925},"PP",{"type":37,"tag":462,"props":927,"children":928},{},[929],{"type":43,"value":930},"VPP",{"type":37,"tag":462,"props":932,"children":933},{},[934],{"type":43,"value":935},"DP",{"type":37,"tag":462,"props":937,"children":938},{},[939],{"type":43,"value":940},"TFLOP\u002Fs\u002FGPU",{"type":37,"tag":462,"props":942,"children":943},{},[944],{"type":43,"value":945},"vs Golden",{"type":37,"tag":462,"props":947,"children":948},{},[949],{"type":43,"value":950},"Peak Mem (GB)",{"type":37,"tag":462,"props":952,"children":953},{},[954],{"type":43,"value":955},"Result",{"type":37,"tag":483,"props":957,"children":958},{},[959,1006,1052,1096],{"type":37,"tag":458,"props":960,"children":961},{},[962,967,972,976,981,986,991,996,1001],{"type":37,"tag":490,"props":963,"children":964},{},[965],{"type":43,"value":966},"Baseline",{"type":37,"tag":490,"props":968,"children":969},{},[970],{"type":43,"value":971},"4",{"type":37,"tag":490,"props":973,"children":974},{},[975],{"type":43,"value":971},{"type":37,"tag":490,"props":977,"children":978},{},[979],{"type":43,"value":980},"5",{"type":37,"tag":490,"props":982,"children":983},{},[984],{"type":43,"value":985},"2",{"type":37,"tag":490,"props":987,"children":988},{},[989],{"type":43,"value":990},"~704",{"type":37,"tag":490,"props":992,"children":993},{},[994],{"type":43,"value":995},"-0.8%",{"type":37,"tag":490,"props":997,"children":998},{},[999],{"type":43,"value":1000},"58.8",{"type":37,"tag":490,"props":1002,"children":1003},{},[1004],{"type":43,"value":1005},"OOM (fragmentation)",{"type":37,"tag":458,"props":1007,"children":1008},{},[1009,1014,1018,1023,1027,1032,1037,1042,1047],{"type":37,"tag":490,"props":1010,"children":1011},{},[1012],{"type":43,"value":1013},"More PP",{"type":37,"tag":490,"props":1015,"children":1016},{},[1017],{"type":43,"value":971},{"type":37,"tag":490,"props":1019,"children":1020},{},[1021],{"type":43,"value":1022},"8",{"type":37,"tag":490,"props":1024,"children":1025},{},[1026],{"type":43,"value":980},{"type":37,"tag":490,"props":1028,"children":1029},{},[1030],{"type":43,"value":1031},"1",{"type":37,"tag":490,"props":1033,"children":1034},{},[1035],{"type":43,"value":1036},"668.0",{"type":37,"tag":490,"props":1038,"children":1039},{},[1040],{"type":43,"value":1041},"-5.9%",{"type":37,"tag":490,"props":1043,"children":1044},{},[1045],{"type":43,"value":1046},"53.2",{"type":37,"tag":490,"props":1048,"children":1049},{},[1050],{"type":43,"value":1051},"Borderline perf",{"type":37,"tag":458,"props":1053,"children":1054},{},[1055,1060,1064,1068,1072,1076,1081,1086,1091],{"type":37,"tag":490,"props":1056,"children":1057},{},[1058],{"type":43,"value":1059},"More TP",{"type":37,"tag":490,"props":1061,"children":1062},{},[1063],{"type":43,"value":1022},{"type":37,"tag":490,"props":1065,"children":1066},{},[1067],{"type":43,"value":971},{"type":37,"tag":490,"props":1069,"children":1070},{},[1071],{"type":43,"value":980},{"type":37,"tag":490,"props":1073,"children":1074},{},[1075],{"type":43,"value":1031},{"type":37,"tag":490,"props":1077,"children":1078},{},[1079],{"type":43,"value":1080},"508.7",{"type":37,"tag":490,"props":1082,"children":1083},{},[1084],{"type":43,"value":1085},"-28.4%",{"type":37,"tag":490,"props":1087,"children":1088},{},[1089],{"type":43,"value":1090},"50.2",{"type":37,"tag":490,"props":1092,"children":1093},{},[1094],{"type":43,"value":1095},"Severe regression",{"type":37,"tag":458,"props":1097,"children":1098},{},[1099,1104,1108,1112,1116,1120,1124,1128,1133],{"type":37,"tag":490,"props":1100,"children":1101},{},[1102],{"type":43,"value":1103},"Baseline + expandable_segments",{"type":37,"tag":490,"props":1105,"children":1106},{},[1107],{"type":43,"value":971},{"type":37,"tag":490,"props":1109,"children":1110},{},[1111],{"type":43,"value":971},{"type":37,"tag":490,"props":1113,"children":1114},{},[1115],{"type":43,"value":980},{"type":37,"tag":490,"props":1117,"children":1118},{},[1119],{"type":43,"value":985},{"type":37,"tag":490,"props":1121,"children":1122},{},[1123],{"type":43,"value":990},{"type":37,"tag":490,"props":1125,"children":1126},{},[1127],{"type":43,"value":995},{"type":37,"tag":490,"props":1129,"children":1130},{},[1131],{"type":43,"value":1132},"~59",{"type":37,"tag":490,"props":1134,"children":1135},{},[1136],{"type":37,"tag":64,"props":1137,"children":1138},{},[1139],{"type":43,"value":1140},"Passed",{"type":37,"tag":46,"props":1142,"children":1143},{},[1144],{"type":43,"value":1145},"Key takeaways:",{"type":37,"tag":123,"props":1147,"children":1148},{},[1149,1164,1174],{"type":37,"tag":127,"props":1150,"children":1151},{},[1152,1162],{"type":37,"tag":64,"props":1153,"children":1154},{},[1155,1160],{"type":37,"tag":80,"props":1156,"children":1158},{"className":1157},[],[1159],{"type":43,"value":793},{"type":43,"value":1161}," is the winner.",{"type":43,"value":1163}," The baseline OOM was caused by\nmemory fragmentation, not insufficient capacity. Setting this env var\neliminated the OOM with zero throughput cost and no parallelism changes.",{"type":37,"tag":127,"props":1165,"children":1166},{},[1167,1172],{"type":37,"tag":64,"props":1168,"children":1169},{},[1170],{"type":43,"value":1171},"PP=8 works for memory but loses DP",{"type":43,"value":1173}," (2→1), meaning 32 gradient accumulation\nsteps per batch, which hurts throughput by ~6%.",{"type":37,"tag":127,"props":1175,"children":1176},{},[1177,1182],{"type":37,"tag":64,"props":1178,"children":1179},{},[1180],{"type":43,"value":1181},"TP=8 is catastrophic",{"type":43,"value":1183}," (-28%) because doubling TP increases all-reduce\ncommunication volume proportionally across NVLink, and DP=1 means no\nmicro-batch overlap.",{"type":37,"tag":324,"props":1185,"children":1187},{"id":1186},"cpu-offloading-blocked",[1188],{"type":43,"value":1189},"CPU offloading: blocked",{"type":37,"tag":450,"props":1191,"children":1192},{},[1193,1212],{"type":37,"tag":454,"props":1194,"children":1195},{},[1196],{"type":37,"tag":458,"props":1197,"children":1198},{},[1199,1203,1208],{"type":37,"tag":462,"props":1200,"children":1201},{},[1202],{"type":43,"value":915},{"type":37,"tag":462,"props":1204,"children":1205},{},[1206],{"type":43,"value":1207},"offload_layers",{"type":37,"tag":462,"props":1209,"children":1210},{},[1211],{"type":43,"value":955},{"type":37,"tag":483,"props":1213,"children":1214},{},[1215,1232,1248],{"type":37,"tag":458,"props":1216,"children":1217},{},[1218,1223,1227],{"type":37,"tag":490,"props":1219,"children":1220},{},[1221],{"type":43,"value":1222},"Exp 4",{"type":37,"tag":490,"props":1224,"children":1225},{},[1226],{"type":43,"value":985},{"type":37,"tag":490,"props":1228,"children":1229},{},[1230],{"type":43,"value":1231},"Incompatible (PP > 1)",{"type":37,"tag":458,"props":1233,"children":1234},{},[1235,1240,1244],{"type":37,"tag":490,"props":1236,"children":1237},{},[1238],{"type":43,"value":1239},"Exp 5",{"type":37,"tag":490,"props":1241,"children":1242},{},[1243],{"type":43,"value":971},{"type":37,"tag":490,"props":1245,"children":1246},{},[1247],{"type":43,"value":1231},{"type":37,"tag":458,"props":1249,"children":1250},{},[1251,1256,1261],{"type":37,"tag":490,"props":1252,"children":1253},{},[1254],{"type":43,"value":1255},"Exp 6",{"type":37,"tag":490,"props":1257,"children":1258},{},[1259],{"type":43,"value":1260},"6",{"type":37,"tag":490,"props":1262,"children":1263},{},[1264],{"type":43,"value":1231},{"type":37,"tag":46,"props":1266,"children":1267},{},[1268,1274],{"type":37,"tag":80,"props":1269,"children":1271},{"className":1270},[],[1272],{"type":43,"value":1273},"ValueError: Currently there is no support for Pipeline parallelism with CPU offloading.",{"type":43,"value":1275}," This approach is blocked for any model using PP > 1.",{"type":37,"tag":324,"props":1277,"children":1279},{"id":1278},"activation-recompute-expensive-alternative",[1280],{"type":43,"value":1281},"Activation recompute: expensive alternative",{"type":37,"tag":46,"props":1283,"children":1284},{},[1285,1287,1292],{"type":43,"value":1286},"Selective activation recompute with ",{"type":37,"tag":80,"props":1288,"children":1290},{"className":1289},[],[1291],{"type":43,"value":302},{"type":43,"value":1293}," saved ~3 GB peak memory but cost\n~16% GPU utilization on this workload. See\n@skills\u002Fnemo-mbridge-perf-activation-recompute\u002FSKILL.md for full results.",{"type":37,"tag":324,"props":1295,"children":1297},{"id":1296},"lora-sp-input-re-gather",[1298],{"type":43,"value":1299},"LoRA + SP input re-gather",{"type":37,"tag":46,"props":1301,"children":1302},{},[1303],{"type":43,"value":1304},"Real-checkpoint H100 training with SQuAD showed lower peak memory in all tested\nconfigurations, with workload-dependent throughput cost:",{"type":37,"tag":450,"props":1306,"children":1307},{},[1308,1340],{"type":37,"tag":454,"props":1309,"children":1310},{},[1311],{"type":37,"tag":458,"props":1312,"children":1313},{},[1314,1319,1325,1330,1335],{"type":37,"tag":462,"props":1315,"children":1316},{},[1317],{"type":43,"value":1318},"Model\u002Fconfig",{"type":37,"tag":462,"props":1320,"children":1322},{"align":1321},"right",[1323],{"type":43,"value":1324},"Baseline peak",{"type":37,"tag":462,"props":1326,"children":1327},{"align":1321},[1328],{"type":43,"value":1329},"Input re-gather peak",{"type":37,"tag":462,"props":1331,"children":1332},{"align":1321},[1333],{"type":43,"value":1334},"Memory saved",{"type":37,"tag":462,"props":1336,"children":1337},{"align":1321},[1338],{"type":43,"value":1339},"Throughput change",{"type":37,"tag":483,"props":1341,"children":1342},{},[1343,1371,1399],{"type":37,"tag":458,"props":1344,"children":1345},{},[1346,1351,1356,1361,1366],{"type":37,"tag":490,"props":1347,"children":1348},{},[1349],{"type":43,"value":1350},"Qwen3-8B, TP2, seq 8192",{"type":37,"tag":490,"props":1352,"children":1353},{"align":1321},[1354],{"type":43,"value":1355},"47.545 GB",{"type":37,"tag":490,"props":1357,"children":1358},{"align":1321},[1359],{"type":43,"value":1360},"42.814 GB",{"type":37,"tag":490,"props":1362,"children":1363},{"align":1321},[1364],{"type":43,"value":1365},"4.731 GB (10.0%)",{"type":37,"tag":490,"props":1367,"children":1368},{"align":1321},[1369],{"type":43,"value":1370},"-6.74%",{"type":37,"tag":458,"props":1372,"children":1373},{},[1374,1379,1384,1389,1394],{"type":37,"tag":490,"props":1375,"children":1376},{},[1377],{"type":43,"value":1378},"Qwen3-30B-A3B, TP4\u002FEP4",{"type":37,"tag":490,"props":1380,"children":1381},{"align":1321},[1382],{"type":43,"value":1383},"29.890 GB",{"type":37,"tag":490,"props":1385,"children":1386},{"align":1321},[1387],{"type":43,"value":1388},"28.321 GB",{"type":37,"tag":490,"props":1390,"children":1391},{"align":1321},[1392],{"type":43,"value":1393},"1.569 GB (5.2%)",{"type":37,"tag":490,"props":1395,"children":1396},{"align":1321},[1397],{"type":43,"value":1398},"-2.89%",{"type":37,"tag":458,"props":1400,"children":1401},{},[1402,1407,1412,1417,1422],{"type":37,"tag":490,"props":1403,"children":1404},{},[1405],{"type":43,"value":1406},"GPT-OSS-120B, TP2\u002FEP8",{"type":37,"tag":490,"props":1408,"children":1409},{"align":1321},[1410],{"type":43,"value":1411},"52.185 GB",{"type":37,"tag":490,"props":1413,"children":1414},{"align":1321},[1415],{"type":43,"value":1416},"51.371 GB",{"type":37,"tag":490,"props":1418,"children":1419},{"align":1321},[1420],{"type":43,"value":1421},"0.814 GB (1.6%)",{"type":37,"tag":490,"props":1423,"children":1424},{"align":1321},[1425],{"type":43,"value":1426},"-0.34%",{"type":37,"tag":46,"props":1428,"children":1429},{},[1430,1432,1438],{"type":43,"value":1431},"All runs had finite losses with zero skipped or NaN iterations. Two-rank BF16\nand FP32 checks matched the baseline for outputs, input gradients, LoRA-A and\nLoRA-B gradients, and two-microbatch fused ",{"type":37,"tag":80,"props":1433,"children":1435},{"className":1434},[],[1436],{"type":43,"value":1437},"main_grad",{"type":43,"value":1439}," accumulation.",{"type":37,"tag":52,"props":1441,"children":1443},{"id":1442},"code-anchors",[1444],{"type":43,"value":1445},"Code Anchors",{"type":37,"tag":324,"props":1447,"children":1449},{"id":1448},"lora-sequence-parallel-input-re-gather",[1450],{"type":43,"value":1451},"LoRA sequence-parallel input re-gather",{"type":37,"tag":72,"props":1453,"children":1457},{"className":1454,"code":1456,"language":43,"meta":77},[1455],"language-text","src\u002Fmegatron\u002Fbridge\u002Fpeft\u002Flora.py\n    LoRA.sequence_parallel_input_regather\n\nsrc\u002Fmegatron\u002Fbridge\u002Fpeft\u002Futils.py\n    ParallelLinearAdapter._sequence_parallel_input_regather_eligibility()\n    ParallelLinearAdapter.forward()\n",[1458],{"type":37,"tag":80,"props":1459,"children":1460},{"__ignoreMap":77},[1461],{"type":43,"value":1456},{"type":37,"tag":324,"props":1463,"children":1465},{"id":1464},"cpu-offloading-pp-incompatibility-mcore",[1466],{"type":43,"value":1467},"CPU offloading PP incompatibility (MCore)",{"type":37,"tag":72,"props":1469,"children":1473},{"className":1470,"code":1471,"language":1472,"meta":77,"style":77},"language-1303:1306:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","        if self.cpu_offloading and self.pipeline_model_parallel_size > 1:\n            raise ValueError(\n                \"Currently there is no support for Pipeline parallelism with CPU offloading\"\n            )\n","1303:1306:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py",[1474],{"type":37,"tag":80,"props":1475,"children":1476},{"__ignoreMap":77},[1477,1485,1493,1501],{"type":37,"tag":84,"props":1478,"children":1479},{"class":86,"line":87},[1480],{"type":37,"tag":84,"props":1481,"children":1482},{},[1483],{"type":43,"value":1484},"        if self.cpu_offloading and self.pipeline_model_parallel_size > 1:\n",{"type":37,"tag":84,"props":1486,"children":1487},{"class":86,"line":181},[1488],{"type":37,"tag":84,"props":1489,"children":1490},{},[1491],{"type":43,"value":1492},"            raise ValueError(\n",{"type":37,"tag":84,"props":1494,"children":1495},{"class":86,"line":191},[1496],{"type":37,"tag":84,"props":1497,"children":1498},{},[1499],{"type":43,"value":1500},"                \"Currently there is no support for Pipeline parallelism with CPU offloading\"\n",{"type":37,"tag":84,"props":1502,"children":1503},{"class":86,"line":662},[1504],{"type":37,"tag":84,"props":1505,"children":1506},{},[1507],{"type":43,"value":1508},"            )\n",{"type":37,"tag":324,"props":1510,"children":1512},{"id":1511},"vpp-config-and-layer-divisibility-validation-mcore",[1513],{"type":43,"value":1514},"VPP config and layer divisibility validation (MCore)",{"type":37,"tag":72,"props":1516,"children":1520},{"className":1517,"code":1518,"language":1519,"meta":77,"style":77},"language-1581:1592:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","            if pipeline_parallel_size and self.virtual_pipeline_model_parallel_size is not None:\n                num_layers_per_middle_pipeline_rank = num_layers \u002F\u002F pipeline_parallel_size\n                if (\n                    not num_layers_per_middle_pipeline_rank\n                    % self.virtual_pipeline_model_parallel_size\n                    == 0\n                ):\n                    raise ValueError(\n                        f\"number of layers on each middle pipeline rank:\"\n                        f\"{num_layers_per_middle_pipeline_rank} must be divisible by virtual\"\n                        f\"pipeline parallel degree {self.virtual_pipeline_model_parallel_size}\"\n                    )\n","1581:1592:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py",[1521],{"type":37,"tag":80,"props":1522,"children":1523},{"__ignoreMap":77},[1524,1532,1540,1548,1556,1564,1572,1581,1590,1599,1608,1617],{"type":37,"tag":84,"props":1525,"children":1526},{"class":86,"line":87},[1527],{"type":37,"tag":84,"props":1528,"children":1529},{},[1530],{"type":43,"value":1531},"            if pipeline_parallel_size and self.virtual_pipeline_model_parallel_size is not None:\n",{"type":37,"tag":84,"props":1533,"children":1534},{"class":86,"line":181},[1535],{"type":37,"tag":84,"props":1536,"children":1537},{},[1538],{"type":43,"value":1539},"                num_layers_per_middle_pipeline_rank = num_layers \u002F\u002F pipeline_parallel_size\n",{"type":37,"tag":84,"props":1541,"children":1542},{"class":86,"line":191},[1543],{"type":37,"tag":84,"props":1544,"children":1545},{},[1546],{"type":43,"value":1547},"                if (\n",{"type":37,"tag":84,"props":1549,"children":1550},{"class":86,"line":662},[1551],{"type":37,"tag":84,"props":1552,"children":1553},{},[1554],{"type":43,"value":1555},"                    not num_layers_per_middle_pipeline_rank\n",{"type":37,"tag":84,"props":1557,"children":1558},{"class":86,"line":671},[1559],{"type":37,"tag":84,"props":1560,"children":1561},{},[1562],{"type":43,"value":1563},"                    % self.virtual_pipeline_model_parallel_size\n",{"type":37,"tag":84,"props":1565,"children":1566},{"class":86,"line":680},[1567],{"type":37,"tag":84,"props":1568,"children":1569},{},[1570],{"type":43,"value":1571},"                    == 0\n",{"type":37,"tag":84,"props":1573,"children":1575},{"class":86,"line":1574},7,[1576],{"type":37,"tag":84,"props":1577,"children":1578},{},[1579],{"type":43,"value":1580},"                ):\n",{"type":37,"tag":84,"props":1582,"children":1584},{"class":86,"line":1583},8,[1585],{"type":37,"tag":84,"props":1586,"children":1587},{},[1588],{"type":43,"value":1589},"                    raise ValueError(\n",{"type":37,"tag":84,"props":1591,"children":1593},{"class":86,"line":1592},9,[1594],{"type":37,"tag":84,"props":1595,"children":1596},{},[1597],{"type":43,"value":1598},"                        f\"number of layers on each middle pipeline rank:\"\n",{"type":37,"tag":84,"props":1600,"children":1602},{"class":86,"line":1601},10,[1603],{"type":37,"tag":84,"props":1604,"children":1605},{},[1606],{"type":43,"value":1607},"                        f\"{num_layers_per_middle_pipeline_rank} must be divisible by virtual\"\n",{"type":37,"tag":84,"props":1609,"children":1611},{"class":86,"line":1610},11,[1612],{"type":37,"tag":84,"props":1613,"children":1614},{},[1615],{"type":43,"value":1616},"                        f\"pipeline parallel degree {self.virtual_pipeline_model_parallel_size}\"\n",{"type":37,"tag":84,"props":1618,"children":1620},{"class":86,"line":1619},12,[1621],{"type":37,"tag":84,"props":1622,"children":1623},{},[1624],{"type":43,"value":1625},"                    )\n",{"type":37,"tag":324,"props":1627,"children":1629},{"id":1628},"parallelism-docs-on-interleaved-pipeline-schedule",[1630],{"type":43,"value":1631},"Parallelism docs on interleaved pipeline schedule",{"type":37,"tag":72,"props":1633,"children":1637},{"className":1634,"code":1635,"language":1636,"meta":77,"style":77},"language-116:124:docs\u002Fparallelisms.md shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","To minimize the pipeline bubble, the computation on each GPU can be divided into multiple subsets of layers (referred to as model chunks), rather than a single contiguous block. Enable this by setting `virtual_pipeline_model_parallel_size`:\n\nmodel_config = GPTModelProvider(\n    pipeline_model_parallel_size=4,\n    virtual_pipeline_model_parallel_size=2,  # 2 model chunks per pipeline stage\n    # ... other model parameters\n)\n","116:124:docs\u002Fparallelisms.md",[1638],{"type":37,"tag":80,"props":1639,"children":1640},{"__ignoreMap":77},[1641,1649,1656,1664,1672,1680,1688],{"type":37,"tag":84,"props":1642,"children":1643},{"class":86,"line":87},[1644],{"type":37,"tag":84,"props":1645,"children":1646},{},[1647],{"type":43,"value":1648},"To minimize the pipeline bubble, the computation on each GPU can be divided into multiple subsets of layers (referred to as model chunks), rather than a single contiguous block. Enable this by setting `virtual_pipeline_model_parallel_size`:\n",{"type":37,"tag":84,"props":1650,"children":1651},{"class":86,"line":181},[1652],{"type":37,"tag":84,"props":1653,"children":1654},{"emptyLinePlaceholder":185},[1655],{"type":43,"value":188},{"type":37,"tag":84,"props":1657,"children":1658},{"class":86,"line":191},[1659],{"type":37,"tag":84,"props":1660,"children":1661},{},[1662],{"type":43,"value":1663},"model_config = GPTModelProvider(\n",{"type":37,"tag":84,"props":1665,"children":1666},{"class":86,"line":662},[1667],{"type":37,"tag":84,"props":1668,"children":1669},{},[1670],{"type":43,"value":1671},"    pipeline_model_parallel_size=4,\n",{"type":37,"tag":84,"props":1673,"children":1674},{"class":86,"line":671},[1675],{"type":37,"tag":84,"props":1676,"children":1677},{},[1678],{"type":43,"value":1679},"    virtual_pipeline_model_parallel_size=2,  # 2 model chunks per pipeline stage\n",{"type":37,"tag":84,"props":1681,"children":1682},{"class":86,"line":680},[1683],{"type":37,"tag":84,"props":1684,"children":1685},{},[1686],{"type":43,"value":1687},"    # ... other model parameters\n",{"type":37,"tag":84,"props":1689,"children":1690},{"class":86,"line":1574},[1691],{"type":37,"tag":84,"props":1692,"children":1693},{},[1694],{"type":43,"value":686},{"type":37,"tag":52,"props":1696,"children":1698},{"id":1697},"failure-diagnosis",[1699],{"type":43,"value":1700},"Failure Diagnosis",{"type":37,"tag":450,"props":1702,"children":1703},{},[1704,1730],{"type":37,"tag":454,"props":1705,"children":1706},{},[1707],{"type":37,"tag":458,"props":1708,"children":1709},{},[1710,1715,1720,1725],{"type":37,"tag":462,"props":1711,"children":1712},{},[1713],{"type":43,"value":1714},"Symptom",{"type":37,"tag":462,"props":1716,"children":1717},{},[1718],{"type":43,"value":1719},"Cause",{"type":37,"tag":462,"props":1721,"children":1722},{},[1723],{"type":43,"value":1724},"Confirm",{"type":37,"tag":462,"props":1726,"children":1727},{},[1728],{"type":43,"value":1729},"Fix",{"type":37,"tag":483,"props":1731,"children":1732},{},[1733,1768,1807,1838,1875,1902],{"type":37,"tag":458,"props":1734,"children":1735},{},[1736,1741,1746,1758],{"type":37,"tag":490,"props":1737,"children":1738},{},[1739],{"type":43,"value":1740},"OOM on a single rank despite headroom on others",{"type":37,"tag":490,"props":1742,"children":1743},{},[1744],{"type":43,"value":1745},"Memory fragmentation",{"type":37,"tag":490,"props":1747,"children":1748},{},[1749,1751,1756],{"type":43,"value":1750},"check if ",{"type":37,"tag":80,"props":1752,"children":1754},{"className":1753},[],[1755],{"type":43,"value":793},{"type":43,"value":1757}," is set",{"type":37,"tag":490,"props":1759,"children":1760},{},[1761,1763],{"type":43,"value":1762},"set ",{"type":37,"tag":80,"props":1764,"children":1766},{"className":1765},[],[1767],{"type":43,"value":231},{"type":37,"tag":458,"props":1769,"children":1770},{},[1771,1784,1789,1802],{"type":37,"tag":490,"props":1772,"children":1773},{},[1774,1776,1782],{"type":43,"value":1775},"OOM with ",{"type":37,"tag":80,"props":1777,"children":1779},{"className":1778},[],[1780],{"type":43,"value":1781},"expandable_segments",{"type":43,"value":1783}," already set",{"type":37,"tag":490,"props":1785,"children":1786},{},[1787],{"type":43,"value":1788},"Genuine capacity limit",{"type":37,"tag":490,"props":1790,"children":1791},{},[1792,1794,1800],{"type":43,"value":1793},"check ",{"type":37,"tag":80,"props":1795,"children":1797},{"className":1796},[],[1798],{"type":43,"value":1799},"nvidia-smi",{"type":43,"value":1801}," for param\u002Foptimizer memory",{"type":37,"tag":490,"props":1803,"children":1804},{},[1805],{"type":43,"value":1806},"increase PP, use distributed optimizer, or add recompute",{"type":37,"tag":458,"props":1808,"children":1809},{},[1810,1815,1820,1833],{"type":37,"tag":490,"props":1811,"children":1812},{},[1813],{"type":43,"value":1814},"Estimated memory exceeds GPU capacity before launch",{"type":37,"tag":490,"props":1816,"children":1817},{},[1818],{"type":43,"value":1819},"model state or activations genuinely too large",{"type":37,"tag":490,"props":1821,"children":1822},{},[1823,1825,1831],{"type":43,"value":1824},"run ",{"type":37,"tag":80,"props":1826,"children":1828},{"className":1827},[],[1829],{"type":43,"value":1830},"estimate_training_memory",{"type":43,"value":1832}," and inspect the largest component",{"type":37,"tag":490,"props":1834,"children":1835},{},[1836],{"type":43,"value":1837},"adjust PP\u002FTP\u002FCP\u002FEP, distributed optimizer, or recompute before launching",{"type":37,"tag":458,"props":1839,"children":1840},{},[1841,1846,1851,1864],{"type":37,"tag":490,"props":1842,"children":1843},{},[1844],{"type":43,"value":1845},"LoRA + SP retains unexpectedly high activation memory",{"type":37,"tag":490,"props":1847,"children":1848},{},[1849],{"type":43,"value":1850},"full gathered LoRA-A inputs are retained until backward",{"type":37,"tag":490,"props":1852,"children":1853},{},[1854,1856,1862],{"type":43,"value":1855},"check whether ",{"type":37,"tag":80,"props":1857,"children":1859},{"className":1858},[],[1860],{"type":43,"value":1861},"cfg.peft.sequence_parallel_input_regather",{"type":43,"value":1863}," is enabled and the target is eligible",{"type":37,"tag":490,"props":1865,"children":1866},{},[1867,1868,1873],{"type":43,"value":1762},{"type":37,"tag":80,"props":1869,"children":1871},{"className":1870},[],[1872],{"type":43,"value":251},{"type":43,"value":1874},"; verify fallback constraints",{"type":37,"tag":458,"props":1876,"children":1877},{},[1878,1887,1892,1897],{"type":37,"tag":490,"props":1879,"children":1880},{},[1881],{"type":37,"tag":80,"props":1882,"children":1884},{"className":1883},[],[1885],{"type":43,"value":1886},"ValueError: PP + CPU offloading",{"type":37,"tag":490,"props":1888,"children":1889},{},[1890],{"type":43,"value":1891},"using cpu_offloading with PP > 1",{"type":37,"tag":490,"props":1893,"children":1894},{},[1895],{"type":43,"value":1896},"check PP config",{"type":37,"tag":490,"props":1898,"children":1899},{},[1900],{"type":43,"value":1901},"disable CPU offloading or set PP=1",{"type":37,"tag":458,"props":1903,"children":1904},{},[1905,1923,1928,1933],{"type":37,"tag":490,"props":1906,"children":1907},{},[1908,1914,1916,1921],{"type":37,"tag":80,"props":1909,"children":1911},{"className":1910},[],[1912],{"type":43,"value":1913},"RuntimeError",{"type":43,"value":1915}," with ",{"type":37,"tag":80,"props":1917,"children":1919},{"className":1918},[],[1920],{"type":43,"value":801},{"type":43,"value":1922}," + expandable segments",{"type":37,"tag":490,"props":1924,"children":1925},{},[1926],{"type":43,"value":1927},"NCCL UB incompatible with expandable allocator",{"type":37,"tag":490,"props":1929,"children":1930},{},[1931],{"type":43,"value":1932},"check env vars",{"type":37,"tag":490,"props":1934,"children":1935},{},[1936,1938,1943,1945],{"type":43,"value":1937},"remove ",{"type":37,"tag":80,"props":1939,"children":1941},{"className":1940},[],[1942],{"type":43,"value":793},{"type":43,"value":1944}," or disable ",{"type":37,"tag":80,"props":1946,"children":1948},{"className":1947},[],[1949],{"type":43,"value":801},{"type":37,"tag":52,"props":1951,"children":1953},{"id":1952},"known-limitations",[1954],{"type":43,"value":1955},"Known Limitations",{"type":37,"tag":123,"props":1957,"children":1958},{},[1959,1964,1969,1974],{"type":37,"tag":127,"props":1960,"children":1961},{},[1962],{"type":43,"value":1963},"CPU offloading is blocked when PP > 1",{"type":37,"tag":127,"props":1965,"children":1966},{},[1967],{"type":43,"value":1968},"Parallelism resizing (TP\u002FPP) often has significant throughput costs",{"type":37,"tag":127,"props":1970,"children":1971},{},[1972],{"type":43,"value":1973},"The theoretical estimator is formula-based and does not replace runtime\nprofiling or CUDA memory reports",{"type":37,"tag":127,"props":1975,"children":1976},{},[1977],{"type":43,"value":1978},"LoRA input re-gather does not cover row-parallel or expert adapters and may\nhave negligible benefit when few eligible LoRA-A activations dominate memory",{"type":37,"tag":52,"props":1980,"children":1982},{"id":1981},"verification",[1983],{"type":43,"value":1984},"Verification",{"type":37,"tag":46,"props":1986,"children":1987},{},[1988,1990,1995],{"type":43,"value":1989},"Quick check that ",{"type":37,"tag":80,"props":1991,"children":1993},{"className":1992},[],[1994],{"type":43,"value":793},{"type":43,"value":1996}," is active:",{"type":37,"tag":72,"props":1998,"children":2000},{"className":164,"code":1999,"language":166,"meta":77,"style":77},"import os\nassert \"expandable_segments:True\" in os.environ.get(\"PYTORCH_CUDA_ALLOC_CONF\", \"\")\n",[2001],{"type":37,"tag":80,"props":2002,"children":2003},{"__ignoreMap":77},[2004,2012],{"type":37,"tag":84,"props":2005,"children":2006},{"class":86,"line":87},[2007],{"type":37,"tag":84,"props":2008,"children":2009},{},[2010],{"type":43,"value":2011},"import os\n",{"type":37,"tag":84,"props":2013,"children":2014},{"class":86,"line":181},[2015],{"type":37,"tag":84,"props":2016,"children":2017},{},[2018],{"type":43,"value":2019},"assert \"expandable_segments:True\" in os.environ.get(\"PYTORCH_CUDA_ALLOC_CONF\", \"\")\n",{"type":37,"tag":46,"props":2021,"children":2022},{},[2023],{"type":43,"value":2024},"For Slurm jobs, verify the env var is exported before the training command\nin the launch script.",{"type":37,"tag":46,"props":2026,"children":2027},{},[2028],{"type":43,"value":2029},"For LoRA + SP input re-gather, run the focused configuration tests and the real\ntwo-rank MCore backward-parity test:",{"type":37,"tag":72,"props":2031,"children":2033},{"className":74,"code":2032,"language":76,"meta":77,"style":77},"uv run python -m pytest \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_utils.py -k \"sequence_parallel_input_regather\" \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_lora.py -k \"sequence_parallel_input_regather\"\n\nuv run python -m torch.distributed.run --nproc_per_node=2 -m pytest \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_lora_sp_input_regather_distributed.py\n",[2034],{"type":37,"tag":80,"props":2035,"children":2036},{"__ignoreMap":77},[2037,2072,2103,2128,2135,2176],{"type":37,"tag":84,"props":2038,"children":2039},{"class":86,"line":87},[2040,2046,2052,2057,2062,2067],{"type":37,"tag":84,"props":2041,"children":2043},{"style":2042},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[2044],{"type":43,"value":2045},"uv",{"type":37,"tag":84,"props":2047,"children":2049},{"style":2048},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[2050],{"type":43,"value":2051}," run",{"type":37,"tag":84,"props":2053,"children":2054},{"style":2048},[2055],{"type":43,"value":2056}," python",{"type":37,"tag":84,"props":2058,"children":2059},{"style":2048},[2060],{"type":43,"value":2061}," -m",{"type":37,"tag":84,"props":2063,"children":2064},{"style":2048},[2065],{"type":43,"value":2066}," pytest",{"type":37,"tag":84,"props":2068,"children":2069},{"style":97},[2070],{"type":43,"value":2071}," \\\n",{"type":37,"tag":84,"props":2073,"children":2074},{"class":86,"line":181},[2075,2080,2085,2090,2094,2099],{"type":37,"tag":84,"props":2076,"children":2077},{"style":2048},[2078],{"type":43,"value":2079},"  tests\u002Funit_tests\u002Fpeft\u002Ftest_utils.py",{"type":37,"tag":84,"props":2081,"children":2082},{"style":2048},[2083],{"type":43,"value":2084}," -k",{"type":37,"tag":84,"props":2086,"children":2087},{"style":103},[2088],{"type":43,"value":2089}," \"",{"type":37,"tag":84,"props":2091,"children":2092},{"style":2048},[2093],{"type":43,"value":864},{"type":37,"tag":84,"props":2095,"children":2096},{"style":103},[2097],{"type":43,"value":2098},"\"",{"type":37,"tag":84,"props":2100,"children":2101},{"style":97},[2102],{"type":43,"value":2071},{"type":37,"tag":84,"props":2104,"children":2105},{"class":86,"line":191},[2106,2111,2115,2119,2123],{"type":37,"tag":84,"props":2107,"children":2108},{"style":2048},[2109],{"type":43,"value":2110},"  tests\u002Funit_tests\u002Fpeft\u002Ftest_lora.py",{"type":37,"tag":84,"props":2112,"children":2113},{"style":2048},[2114],{"type":43,"value":2084},{"type":37,"tag":84,"props":2116,"children":2117},{"style":103},[2118],{"type":43,"value":2089},{"type":37,"tag":84,"props":2120,"children":2121},{"style":2048},[2122],{"type":43,"value":864},{"type":37,"tag":84,"props":2124,"children":2125},{"style":103},[2126],{"type":43,"value":2127},"\"\n",{"type":37,"tag":84,"props":2129,"children":2130},{"class":86,"line":662},[2131],{"type":37,"tag":84,"props":2132,"children":2133},{"emptyLinePlaceholder":185},[2134],{"type":43,"value":188},{"type":37,"tag":84,"props":2136,"children":2137},{"class":86,"line":671},[2138,2142,2146,2150,2154,2159,2164,2168,2172],{"type":37,"tag":84,"props":2139,"children":2140},{"style":2042},[2141],{"type":43,"value":2045},{"type":37,"tag":84,"props":2143,"children":2144},{"style":2048},[2145],{"type":43,"value":2051},{"type":37,"tag":84,"props":2147,"children":2148},{"style":2048},[2149],{"type":43,"value":2056},{"type":37,"tag":84,"props":2151,"children":2152},{"style":2048},[2153],{"type":43,"value":2061},{"type":37,"tag":84,"props":2155,"children":2156},{"style":2048},[2157],{"type":43,"value":2158}," torch.distributed.run",{"type":37,"tag":84,"props":2160,"children":2161},{"style":2048},[2162],{"type":43,"value":2163}," --nproc_per_node=2",{"type":37,"tag":84,"props":2165,"children":2166},{"style":2048},[2167],{"type":43,"value":2061},{"type":37,"tag":84,"props":2169,"children":2170},{"style":2048},[2171],{"type":43,"value":2066},{"type":37,"tag":84,"props":2173,"children":2174},{"style":97},[2175],{"type":43,"value":2071},{"type":37,"tag":84,"props":2177,"children":2178},{"class":86,"line":680},[2179],{"type":37,"tag":84,"props":2180,"children":2181},{"style":2048},[2182],{"type":43,"value":2183},"  tests\u002Funit_tests\u002Fpeft\u002Ftest_lora_sp_input_regather_distributed.py\n",{"type":37,"tag":2185,"props":2186,"children":2187},"style",{},[2188],{"type":43,"value":2189},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2191,"total":2293},[2192,2207,2221,2235,2247,2264,2279],{"slug":2193,"name":2193,"fn":2194,"description":2195,"org":2196,"tags":2197,"stars":19,"repoUrl":20,"updatedAt":2206},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2198,2201,2204,2205],{"name":2199,"slug":2200,"type":15},"Data Analysis","data-analysis",{"name":2202,"slug":2203,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:28:43.176466",{"slug":2208,"name":2208,"fn":2209,"description":2210,"org":2211,"tags":2212,"stars":19,"repoUrl":20,"updatedAt":2220},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2213,2216,2219],{"name":2214,"slug":2215,"type":15},"Deployment","deployment",{"name":2217,"slug":2218,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":2222,"name":2222,"fn":2223,"description":2224,"org":2225,"tags":2226,"stars":19,"repoUrl":20,"updatedAt":2234},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2227,2230,2231],{"name":2228,"slug":2229,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2232,"slug":2233,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":2236,"name":2236,"fn":2237,"description":2238,"org":2239,"tags":2240,"stars":19,"repoUrl":20,"updatedAt":2246},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2241,2242,2243],{"name":2199,"slug":2200,"type":15},{"name":9,"slug":8,"type":15},{"name":2244,"slug":2245,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":2248,"name":2248,"fn":2249,"description":2250,"org":2251,"tags":2252,"stars":19,"repoUrl":20,"updatedAt":2263},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2253,2256,2259,2260],{"name":2254,"slug":2255,"type":15},"Automation","automation",{"name":2257,"slug":2258,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":2261,"slug":2262,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":2265,"name":2265,"fn":2266,"description":2267,"org":2268,"tags":2269,"stars":19,"repoUrl":20,"updatedAt":2278},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2270,2271,2274,2275],{"name":2214,"slug":2215,"type":15},{"name":2272,"slug":2273,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":2276,"slug":2277,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":2280,"name":2280,"fn":2281,"description":2282,"org":2283,"tags":2284,"stars":19,"repoUrl":20,"updatedAt":2292},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2285,2286,2289],{"name":9,"slug":8,"type":15},{"name":2287,"slug":2288,"type":15},"Quantum Computing","quantum-computing",{"name":2290,"slug":2291,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":2295,"total":2445},[2296,2314,2329,2340,2352,2366,2379,2393,2404,2413,2427,2436],{"slug":2297,"name":2297,"fn":2298,"description":2299,"org":2300,"tags":2301,"stars":2311,"repoUrl":2312,"updatedAt":2313},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2302,2305,2308],{"name":2303,"slug":2304,"type":15},"Documentation","documentation",{"name":2306,"slug":2307,"type":15},"MCP","mcp",{"name":2309,"slug":2310,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2315,"name":2315,"fn":2316,"description":2317,"org":2318,"tags":2319,"stars":2326,"repoUrl":2327,"updatedAt":2328},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2320,2323,2324],{"name":2321,"slug":2322,"type":15},"Containers","containers",{"name":2214,"slug":2215,"type":15},{"name":2325,"slug":166,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2330,"name":2330,"fn":2331,"description":2332,"org":2333,"tags":2334,"stars":2326,"repoUrl":2327,"updatedAt":2339},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2335,2338],{"name":2336,"slug":2337,"type":15},"CI\u002FCD","ci-cd",{"name":2214,"slug":2215,"type":15},"2026-07-14T05:25:59.97109",{"slug":2341,"name":2341,"fn":2342,"description":2343,"org":2344,"tags":2345,"stars":2326,"repoUrl":2327,"updatedAt":2351},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2346,2347,2348],{"name":2336,"slug":2337,"type":15},{"name":2214,"slug":2215,"type":15},{"name":2349,"slug":2350,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2353,"name":2353,"fn":2354,"description":2355,"org":2356,"tags":2357,"stars":2326,"repoUrl":2327,"updatedAt":2365},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2358,2361,2362],{"name":2359,"slug":2360,"type":15},"Debugging","debugging",{"name":2349,"slug":2350,"type":15},{"name":2363,"slug":2364,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2367,"name":2367,"fn":2368,"description":2369,"org":2370,"tags":2371,"stars":2326,"repoUrl":2327,"updatedAt":2378},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2372,2375],{"name":2373,"slug":2374,"type":15},"Best Practices","best-practices",{"name":2376,"slug":2377,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2380,"name":2380,"fn":2381,"description":2382,"org":2383,"tags":2384,"stars":2326,"repoUrl":2327,"updatedAt":2392},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2385,2388,2391],{"name":2386,"slug":2387,"type":15},"Machine Learning","machine-learning",{"name":2389,"slug":2390,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2394,"name":2394,"fn":2395,"description":2396,"org":2397,"tags":2398,"stars":2326,"repoUrl":2327,"updatedAt":2403},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2399,2402],{"name":2400,"slug":2401,"type":15},"QA","qa",{"name":2244,"slug":2245,"type":15},"2026-07-14T05:25:53.673039",{"slug":2405,"name":2405,"fn":2406,"description":2407,"org":2408,"tags":2409,"stars":2326,"repoUrl":2327,"updatedAt":2412},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2410,2411],{"name":2214,"slug":2215,"type":15},{"name":2217,"slug":2218,"type":15},"2026-07-14T05:25:49.362534",{"slug":2414,"name":2414,"fn":2415,"description":2416,"org":2417,"tags":2418,"stars":2326,"repoUrl":2327,"updatedAt":2426},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2419,2422,2423],{"name":2420,"slug":2421,"type":15},"Code Review","code-review",{"name":2349,"slug":2350,"type":15},{"name":2424,"slug":2425,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2428,"name":2428,"fn":2429,"description":2430,"org":2431,"tags":2432,"stars":2326,"repoUrl":2327,"updatedAt":2435},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2433,2434],{"name":2400,"slug":2401,"type":15},{"name":2244,"slug":2245,"type":15},"2026-07-14T05:25:54.928983",{"slug":2437,"name":2437,"fn":2438,"description":2439,"org":2440,"tags":2441,"stars":2326,"repoUrl":2327,"updatedAt":2444},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2442,2443],{"name":2254,"slug":2255,"type":15},{"name":2336,"slug":2337,"type":15},"2026-07-30T05:29:03.275638",496]