[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nemo-mbridge-perf-cpu-offloading":3,"mdc-ctckgy-key":31,"related-repo-nvidia-nemo-mbridge-perf-cpu-offloading":1874,"related-org-nvidia-nemo-mbridge-perf-cpu-offloading":1978},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":20,"repoUrl":21,"updatedAt":22,"license":23,"forks":24,"topics":25,"repo":26,"sourceUrl":29,"mdContent":30},"nemo-mbridge-perf-cpu-offloading","configure CPU offloading in Megatron-Bridge","Validate and use CPU offloading in Megatron Bridge, including layer-level activation offloading and fractional optimizer state offloading with HybridDeviceOptimizer.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,17],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":9,"slug":8,"type":15},{"name":18,"slug":19,"type":15},"Engineering","engineering",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:26:56.413374","Apache-2.0",281,[],{"repoUrl":21,"stars":20,"forks":24,"topics":27,"description":28},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fnemo-mbridge-perf-cpu-offloading","---\nname: nemo-mbridge-perf-cpu-offloading\ndescription: Validate and use CPU offloading in Megatron Bridge, including layer-level activation offloading and fractional optimizer state offloading with HybridDeviceOptimizer.\nlicense: Apache-2.0\nwhen_to_use: Enabling CPU offload to reduce GPU memory, or investigating a commit that changed CPU offloading config and caused OOM or a crash; 'cpu_offloading', 'optimizer_cpu_offload', 'optimizer_offload_fraction', 'HybridDeviceOptimizer', 'move optimizer to CPU'.\n---\n\n# CPU Offloading\n\n## References\n\n- Stable docs: @docs\u002Ftraining\u002Fcpu-offloading.md\n- Structured metadata: @skills\u002Fnemo-mbridge-perf-cpu-offloading\u002Fcard.yaml\n\n## What It Is\n\nTwo independent mechanisms to move data from GPU to CPU memory:\n\n| Mechanism | Config namespace | What gets offloaded | PP restriction |\n|---|---|---|---|\n| Activation offloading | `model.cpu_offloading*` | Activations (and optionally weights) per transformer layer | PP must be 1 |\n| Optimizer offloading | `optimizer.optimizer_cpu_offload` | Adam optimizer states (momentum + variance) via `HybridDeviceOptimizer` | None |\n\n## Quick Decision\n\n| Situation | Recommendation |\n|---|---|\n| Large MoE model (30B+), needs PP > 1 | Optimizer offloading — activation offloading is blocked by PP=1 |\n| Small\u002Fmedium model, PP=1 fits, activation memory dominates | Activation offloading |\n| Want tunable memory-speed tradeoff | Optimizer offloading with fractional `optimizer_offload_fraction` |\n| Throughput is top priority | Don't enable — offloading always adds overhead |\n| CUDA graphs are needed | Only optimizer offloading — activation offloading is incompatible |\n| Memory pressure is moderate | Optimizer offload at 25–50% fraction for best efficiency |\n\n## Enablement\n\n### Optimizer CPU offloading (recommended for large models)\n\n```python\ncfg.optimizer.optimizer_cpu_offload = True\ncfg.optimizer.optimizer_offload_fraction = 1.0\ncfg.optimizer.overlap_cpu_optimizer_d2h_h2d = True\n```\n\nCLI overrides:\n\n```bash\noptimizer.optimizer_cpu_offload=True \\\noptimizer.optimizer_offload_fraction=0.5 \\\noptimizer.overlap_cpu_optimizer_d2h_h2d=True\n```\n\n### Activation CPU offloading (small\u002Fmedium models only)\n\n```python\ncfg.model.cpu_offloading = True\ncfg.model.cpu_offloading_num_layers = 16\ncfg.model.cpu_offloading_activations = True\ncfg.model.cpu_offloading_weights = False\n\ncfg.model.pipeline_model_parallel_size = 1\ncfg.model.recompute_granularity = None\ncfg.model.cuda_graph_impl = \"none\"\n```\n\n## Config Parameter Reference\n\n### Optimizer offloading\n\n| Parameter | Default | Description |\n|-----------|---------|-------------|\n| `optimizer_cpu_offload` | `False` | Master switch |\n| `optimizer_offload_fraction` | `0.0` | Fraction of optimizer states on CPU (0.0–1.0) |\n| `overlap_cpu_optimizer_d2h_h2d` | `False` | Overlap GPU↔CPU transfers with compute |\n| `use_torch_optimizer_for_cpu_offload` | `False` | Use `torch.optim` instead of fused optimizer for CPU portion |\n\n### Activation offloading\n\n| Parameter | Default | Description |\n|-----------|---------|-------------|\n| `cpu_offloading` | `False` | Master switch |\n| `cpu_offloading_num_layers` | `0` | Number of transformer layers to offload (0 to num_layers-1) |\n| `cpu_offloading_activations` | `True` | Offload activations |\n| `cpu_offloading_weights` | `False` | Offload weights |\n| `cpu_offloading_double_buffering` | `False` | Double-buffer across layers while reloading |\n\n## Compatibility And Constraints\n\n### Activation offloading\n\n- `pipeline_model_parallel_size` must be 1\n- `recompute_granularity` must be `None`\n- Cannot combine with `fine_grained_activation_offloading`\n- Cannot combine with CUDA graphs\n- `cpu_offloading_num_layers` must be in `[0, num_layers-1)`\n\n### Optimizer offloading\n\n- Requires `use_distributed_optimizer = True` (default in most recipes)\n- No PP, recompute, or CUDA graph restrictions\n- `optimizer_offload_fraction` must be in `[0.0, 1.0]`\n\n### Practical: large MoE models\n\nActivation offloading is blocked for Qwen3-30B-A3B and similar large MoE\nmodels. The PP=1 constraint means each GPU holds all 48 layers; model\nweights + optimizer states alone (~70 GB) exceed H100 80 GB capacity.\n\n## Minimal Runnable Command\n\n```bash\nuv run python scripts\u002Ftraining\u002Frun_recipe.py \\\n  --recipe qwen3_30b_a3b_pretrain_config \\\n  optimizer.optimizer_cpu_offload=True \\\n  optimizer.optimizer_offload_fraction=0.5 \\\n  train.train_iters=20 \\\n  train.global_batch_size=8 \\\n  train.micro_batch_size=1\n```\n\n## Verification\n\n### Unit tests\n\n```bash\nuv run python -m pytest \\\n  tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_full_te_layer_autocast_spec.py -k \"cpu_offload\" \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_utils.py -k \"cpu_offload\" -q\n```\n\n### Success criteria\n\n- Config validation passes for the selected offloading mode\n- Training completes without OOM or NCCL errors\n- Loss matches the non-offloaded baseline (max delta \u003C 0.001)\n- Memory usage drops proportionally to offload fraction\n\n## Code Anchors\n\n### MCore activation offload constraints\n\n```1296:1310:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n        if self.cpu_offloading and (\n            self.cpu_offloading_num_layers \u003C 0 or self.cpu_offloading_num_layers >= self.num_layers\n        ):\n            raise ValueError(...)\n\n        if self.cpu_offloading and self.pipeline_model_parallel_size > 1:\n            raise ValueError(\n                \"Currently there is no support for Pipeline parallelism with CPU offloading\"\n            )\n\n        if self.cpu_offloading and self.recompute_granularity is not None:\n            raise ValueError(\n                \"CPU offloading does not work when activation recomputation is enabled\"\n            )\n```\n\n### MCore CUDA graph incompatibility\n\n```1943:1944:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n            if self.cpu_offloading:\n                raise ValueError(\"CUDA graphs not supported with CPU offloading.\")\n```\n\n### MCore fine-grained offloading mutual exclusion\n\n```1427:1430:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py\n        if self.fine_grained_activation_offloading:\n            assert (\n                not self.cpu_offloading\n            ), \"fine_grained_activation_offloading cannot be enabled with cpu_offloading.\"\n```\n\n### MCore HybridDeviceOptimizer instantiation\n\n```480:518:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Foptimizer\u002F__init__.py\n        if config.optimizer_cpu_offload:\n            # ... setup cpu\u002Fgpu optimizer classes ...\n            optimizer = HybridDeviceOptimizer(\n                param_groups,\n                offload_fraction=config.optimizer_offload_fraction,\n                cpu_optimizer_cls=cpu_optimizer_cls,\n                gpu_optimizer_cls=gpu_optimizer_cls,\n                overlap_cpu_optimizer_d2h_h2d=config.overlap_cpu_optimizer_d2h_h2d,\n                pin_cpu_grads=config.pin_cpu_grads,\n                pin_cpu_params=config.pin_cpu_params,\n            )\n```\n\n### Bridge CUDA graph guard\n\n```232:234:src\u002Fmegatron\u002Fbridge\u002Fmodels\u002Fgpt_full_te_layer_autocast_spec.py\n        assert not config.cpu_offloading and config.recompute_granularity is None, \"Cudagraphs not supported\"\n```\n\n### Bridge activation offloading in PEFT\n\n```621:631:src\u002Fmegatron\u002Fbridge\u002Fpeft\u002Futils.py\n        if self.config.cpu_offloading and self.config.cpu_offloading_activations:\n            x.activation_offloading = True\n        x, _ = self.linear_in(x)\n        x = self.activation(x)\n        if self.config.cpu_offloading and self.config.cpu_offloading_activations:\n            x.activation_offloading = True\n        x, _ = self.linear_out(x)\n```\n\n## Failure Diagnosis\n\n| Symptom | Likely Cause | How To Confirm | Fix |\n|---|---|---|---|\n| `Currently there is no support for Pipeline parallelism with CPU offloading` | Activation offload + PP > 1 | Check `pipeline_model_parallel_size` | Set PP=1 or use optimizer offloading |\n| `CPU offloading does not work when activation recomputation is enabled` | Activation offload + recompute | Check `recompute_granularity` | Set `recompute_granularity=null` |\n| `fine_grained_activation_offloading cannot be enabled with cpu_offloading` | Both offloading modes enabled | Check both flags | Use one or the other |\n| `CUDA graphs not supported with CPU offloading` | CUDA graphs + activation offload | Check `cuda_graph_impl` | Set `cuda_graph_impl=\"none\"` |\n| OOM with activation offloading | Model too large for PP=1 | Check allocated memory vs 80 GB | Use optimizer offloading with PP > 1 |\n| Extreme slowdown (>4x) | 100% optimizer offload, CPU Adam bottleneck | Compare iter time at different fractions | Reduce fraction or enable `overlap_cpu_optimizer_d2h_h2d` |\n| OOM at partial optimizer offload | Insufficient offload for this config | Check memory at different fractions | Increase fraction or add PP |\n\n## Known Limitations\n\n- Activation offloading requires PP=1, making it impractical for large models\n  (30B+ MoE) that need pipeline parallelism.\n- Optimizer offloading throughput penalty scales linearly (~1.9x at 25%,\n  ~4.2x at 100% for Qwen3-30B-A3B).\n- D2H\u002FH2D overlap provides only ~7% speedup because CPU Adam compute is\n  the dominant bottleneck.\n- `fine_grained_activation_offloading` is a separate module-level approach\n  that works with PP > 1 but cannot be combined with layer-level\n  `cpu_offloading`.\n",{"data":32,"body":34},{"name":4,"description":6,"license":23,"when_to_use":33},"Enabling CPU offload to reduce GPU memory, or investigating a commit that changed CPU offloading config and caused OOM or a crash; 'cpu_offloading', 'optimizer_cpu_offload', 'optimizer_offload_fraction', 'HybridDeviceOptimizer', 'move optimizer to CPU'.",{"type":35,"children":36},"root",[37,46,53,68,74,80,179,185,290,296,303,343,348,393,399,476,482,487,623,628,778,784,789,851,856,892,898,903,909,1037,1043,1049,1147,1153,1176,1182,1188,1311,1317,1342,1348,1389,1395,1491,1497,1513,1519,1582,1588,1827,1833,1868],{"type":38,"tag":39,"props":40,"children":42},"element","h1",{"id":41},"cpu-offloading",[43],{"type":44,"value":45},"text","CPU Offloading",{"type":38,"tag":47,"props":48,"children":50},"h2",{"id":49},"references",[51],{"type":44,"value":52},"References",{"type":38,"tag":54,"props":55,"children":56},"ul",{},[57,63],{"type":38,"tag":58,"props":59,"children":60},"li",{},[61],{"type":44,"value":62},"Stable docs: @docs\u002Ftraining\u002Fcpu-offloading.md",{"type":38,"tag":58,"props":64,"children":65},{},[66],{"type":44,"value":67},"Structured metadata: @skills\u002Fnemo-mbridge-perf-cpu-offloading\u002Fcard.yaml",{"type":38,"tag":47,"props":69,"children":71},{"id":70},"what-it-is",[72],{"type":44,"value":73},"What It Is",{"type":38,"tag":75,"props":76,"children":77},"p",{},[78],{"type":44,"value":79},"Two independent mechanisms to move data from GPU to CPU memory:",{"type":38,"tag":81,"props":82,"children":83},"table",{},[84,113],{"type":38,"tag":85,"props":86,"children":87},"thead",{},[88],{"type":38,"tag":89,"props":90,"children":91},"tr",{},[92,98,103,108],{"type":38,"tag":93,"props":94,"children":95},"th",{},[96],{"type":44,"value":97},"Mechanism",{"type":38,"tag":93,"props":99,"children":100},{},[101],{"type":44,"value":102},"Config namespace",{"type":38,"tag":93,"props":104,"children":105},{},[106],{"type":44,"value":107},"What gets offloaded",{"type":38,"tag":93,"props":109,"children":110},{},[111],{"type":44,"value":112},"PP restriction",{"type":38,"tag":114,"props":115,"children":116},"tbody",{},[117,146],{"type":38,"tag":89,"props":118,"children":119},{},[120,126,136,141],{"type":38,"tag":121,"props":122,"children":123},"td",{},[124],{"type":44,"value":125},"Activation offloading",{"type":38,"tag":121,"props":127,"children":128},{},[129],{"type":38,"tag":130,"props":131,"children":133},"code",{"className":132},[],[134],{"type":44,"value":135},"model.cpu_offloading*",{"type":38,"tag":121,"props":137,"children":138},{},[139],{"type":44,"value":140},"Activations (and optionally weights) per transformer layer",{"type":38,"tag":121,"props":142,"children":143},{},[144],{"type":44,"value":145},"PP must be 1",{"type":38,"tag":89,"props":147,"children":148},{},[149,154,163,174],{"type":38,"tag":121,"props":150,"children":151},{},[152],{"type":44,"value":153},"Optimizer offloading",{"type":38,"tag":121,"props":155,"children":156},{},[157],{"type":38,"tag":130,"props":158,"children":160},{"className":159},[],[161],{"type":44,"value":162},"optimizer.optimizer_cpu_offload",{"type":38,"tag":121,"props":164,"children":165},{},[166,168],{"type":44,"value":167},"Adam optimizer states (momentum + variance) via ",{"type":38,"tag":130,"props":169,"children":171},{"className":170},[],[172],{"type":44,"value":173},"HybridDeviceOptimizer",{"type":38,"tag":121,"props":175,"children":176},{},[177],{"type":44,"value":178},"None",{"type":38,"tag":47,"props":180,"children":182},{"id":181},"quick-decision",[183],{"type":44,"value":184},"Quick Decision",{"type":38,"tag":81,"props":186,"children":187},{},[188,204],{"type":38,"tag":85,"props":189,"children":190},{},[191],{"type":38,"tag":89,"props":192,"children":193},{},[194,199],{"type":38,"tag":93,"props":195,"children":196},{},[197],{"type":44,"value":198},"Situation",{"type":38,"tag":93,"props":200,"children":201},{},[202],{"type":44,"value":203},"Recommendation",{"type":38,"tag":114,"props":205,"children":206},{},[207,220,232,251,264,277],{"type":38,"tag":89,"props":208,"children":209},{},[210,215],{"type":38,"tag":121,"props":211,"children":212},{},[213],{"type":44,"value":214},"Large MoE model (30B+), needs PP > 1",{"type":38,"tag":121,"props":216,"children":217},{},[218],{"type":44,"value":219},"Optimizer offloading — activation offloading is blocked by PP=1",{"type":38,"tag":89,"props":221,"children":222},{},[223,228],{"type":38,"tag":121,"props":224,"children":225},{},[226],{"type":44,"value":227},"Small\u002Fmedium model, PP=1 fits, activation memory dominates",{"type":38,"tag":121,"props":229,"children":230},{},[231],{"type":44,"value":125},{"type":38,"tag":89,"props":233,"children":234},{},[235,240],{"type":38,"tag":121,"props":236,"children":237},{},[238],{"type":44,"value":239},"Want tunable memory-speed tradeoff",{"type":38,"tag":121,"props":241,"children":242},{},[243,245],{"type":44,"value":244},"Optimizer offloading with fractional ",{"type":38,"tag":130,"props":246,"children":248},{"className":247},[],[249],{"type":44,"value":250},"optimizer_offload_fraction",{"type":38,"tag":89,"props":252,"children":253},{},[254,259],{"type":38,"tag":121,"props":255,"children":256},{},[257],{"type":44,"value":258},"Throughput is top priority",{"type":38,"tag":121,"props":260,"children":261},{},[262],{"type":44,"value":263},"Don't enable — offloading always adds overhead",{"type":38,"tag":89,"props":265,"children":266},{},[267,272],{"type":38,"tag":121,"props":268,"children":269},{},[270],{"type":44,"value":271},"CUDA graphs are needed",{"type":38,"tag":121,"props":273,"children":274},{},[275],{"type":44,"value":276},"Only optimizer offloading — activation offloading is incompatible",{"type":38,"tag":89,"props":278,"children":279},{},[280,285],{"type":38,"tag":121,"props":281,"children":282},{},[283],{"type":44,"value":284},"Memory pressure is moderate",{"type":38,"tag":121,"props":286,"children":287},{},[288],{"type":44,"value":289},"Optimizer offload at 25–50% fraction for best efficiency",{"type":38,"tag":47,"props":291,"children":293},{"id":292},"enablement",[294],{"type":44,"value":295},"Enablement",{"type":38,"tag":297,"props":298,"children":300},"h3",{"id":299},"optimizer-cpu-offloading-recommended-for-large-models",[301],{"type":44,"value":302},"Optimizer CPU offloading (recommended for large models)",{"type":38,"tag":304,"props":305,"children":310},"pre",{"className":306,"code":307,"language":308,"meta":309,"style":309},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","cfg.optimizer.optimizer_cpu_offload = True\ncfg.optimizer.optimizer_offload_fraction = 1.0\ncfg.optimizer.overlap_cpu_optimizer_d2h_h2d = True\n","python","",[311],{"type":38,"tag":130,"props":312,"children":313},{"__ignoreMap":309},[314,325,334],{"type":38,"tag":315,"props":316,"children":319},"span",{"class":317,"line":318},"line",1,[320],{"type":38,"tag":315,"props":321,"children":322},{},[323],{"type":44,"value":324},"cfg.optimizer.optimizer_cpu_offload = True\n",{"type":38,"tag":315,"props":326,"children":328},{"class":317,"line":327},2,[329],{"type":38,"tag":315,"props":330,"children":331},{},[332],{"type":44,"value":333},"cfg.optimizer.optimizer_offload_fraction = 1.0\n",{"type":38,"tag":315,"props":335,"children":337},{"class":317,"line":336},3,[338],{"type":38,"tag":315,"props":339,"children":340},{},[341],{"type":44,"value":342},"cfg.optimizer.overlap_cpu_optimizer_d2h_h2d = True\n",{"type":38,"tag":75,"props":344,"children":345},{},[346],{"type":44,"value":347},"CLI overrides:",{"type":38,"tag":304,"props":349,"children":353},{"className":350,"code":351,"language":352,"meta":309,"style":309},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","optimizer.optimizer_cpu_offload=True \\\noptimizer.optimizer_offload_fraction=0.5 \\\noptimizer.overlap_cpu_optimizer_d2h_h2d=True\n","bash",[354],{"type":38,"tag":130,"props":355,"children":356},{"__ignoreMap":309},[357,377,385],{"type":38,"tag":315,"props":358,"children":359},{"class":317,"line":318},[360,365,371],{"type":38,"tag":315,"props":361,"children":363},{"style":362},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[364],{"type":44,"value":162},{"type":38,"tag":315,"props":366,"children":368},{"style":367},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[369],{"type":44,"value":370},"=True",{"type":38,"tag":315,"props":372,"children":374},{"style":373},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[375],{"type":44,"value":376}," \\\n",{"type":38,"tag":315,"props":378,"children":379},{"class":317,"line":327},[380],{"type":38,"tag":315,"props":381,"children":382},{"style":373},[383],{"type":44,"value":384},"optimizer.optimizer_offload_fraction=0.5 \\\n",{"type":38,"tag":315,"props":386,"children":387},{"class":317,"line":336},[388],{"type":38,"tag":315,"props":389,"children":390},{"style":373},[391],{"type":44,"value":392},"optimizer.overlap_cpu_optimizer_d2h_h2d=True\n",{"type":38,"tag":297,"props":394,"children":396},{"id":395},"activation-cpu-offloading-smallmedium-models-only",[397],{"type":44,"value":398},"Activation CPU offloading (small\u002Fmedium models only)",{"type":38,"tag":304,"props":400,"children":402},{"className":306,"code":401,"language":308,"meta":309,"style":309},"cfg.model.cpu_offloading = True\ncfg.model.cpu_offloading_num_layers = 16\ncfg.model.cpu_offloading_activations = True\ncfg.model.cpu_offloading_weights = False\n\ncfg.model.pipeline_model_parallel_size = 1\ncfg.model.recompute_granularity = None\ncfg.model.cuda_graph_impl = \"none\"\n",[403],{"type":38,"tag":130,"props":404,"children":405},{"__ignoreMap":309},[406,414,422,430,439,449,458,467],{"type":38,"tag":315,"props":407,"children":408},{"class":317,"line":318},[409],{"type":38,"tag":315,"props":410,"children":411},{},[412],{"type":44,"value":413},"cfg.model.cpu_offloading = True\n",{"type":38,"tag":315,"props":415,"children":416},{"class":317,"line":327},[417],{"type":38,"tag":315,"props":418,"children":419},{},[420],{"type":44,"value":421},"cfg.model.cpu_offloading_num_layers = 16\n",{"type":38,"tag":315,"props":423,"children":424},{"class":317,"line":336},[425],{"type":38,"tag":315,"props":426,"children":427},{},[428],{"type":44,"value":429},"cfg.model.cpu_offloading_activations = True\n",{"type":38,"tag":315,"props":431,"children":433},{"class":317,"line":432},4,[434],{"type":38,"tag":315,"props":435,"children":436},{},[437],{"type":44,"value":438},"cfg.model.cpu_offloading_weights = False\n",{"type":38,"tag":315,"props":440,"children":442},{"class":317,"line":441},5,[443],{"type":38,"tag":315,"props":444,"children":446},{"emptyLinePlaceholder":445},true,[447],{"type":44,"value":448},"\n",{"type":38,"tag":315,"props":450,"children":452},{"class":317,"line":451},6,[453],{"type":38,"tag":315,"props":454,"children":455},{},[456],{"type":44,"value":457},"cfg.model.pipeline_model_parallel_size = 1\n",{"type":38,"tag":315,"props":459,"children":461},{"class":317,"line":460},7,[462],{"type":38,"tag":315,"props":463,"children":464},{},[465],{"type":44,"value":466},"cfg.model.recompute_granularity = None\n",{"type":38,"tag":315,"props":468,"children":470},{"class":317,"line":469},8,[471],{"type":38,"tag":315,"props":472,"children":473},{},[474],{"type":44,"value":475},"cfg.model.cuda_graph_impl = \"none\"\n",{"type":38,"tag":47,"props":477,"children":479},{"id":478},"config-parameter-reference",[480],{"type":44,"value":481},"Config Parameter Reference",{"type":38,"tag":297,"props":483,"children":485},{"id":484},"optimizer-offloading",[486],{"type":44,"value":153},{"type":38,"tag":81,"props":488,"children":489},{},[490,511],{"type":38,"tag":85,"props":491,"children":492},{},[493],{"type":38,"tag":89,"props":494,"children":495},{},[496,501,506],{"type":38,"tag":93,"props":497,"children":498},{},[499],{"type":44,"value":500},"Parameter",{"type":38,"tag":93,"props":502,"children":503},{},[504],{"type":44,"value":505},"Default",{"type":38,"tag":93,"props":507,"children":508},{},[509],{"type":44,"value":510},"Description",{"type":38,"tag":114,"props":512,"children":513},{},[514,540,565,590],{"type":38,"tag":89,"props":515,"children":516},{},[517,526,535],{"type":38,"tag":121,"props":518,"children":519},{},[520],{"type":38,"tag":130,"props":521,"children":523},{"className":522},[],[524],{"type":44,"value":525},"optimizer_cpu_offload",{"type":38,"tag":121,"props":527,"children":528},{},[529],{"type":38,"tag":130,"props":530,"children":532},{"className":531},[],[533],{"type":44,"value":534},"False",{"type":38,"tag":121,"props":536,"children":537},{},[538],{"type":44,"value":539},"Master switch",{"type":38,"tag":89,"props":541,"children":542},{},[543,551,560],{"type":38,"tag":121,"props":544,"children":545},{},[546],{"type":38,"tag":130,"props":547,"children":549},{"className":548},[],[550],{"type":44,"value":250},{"type":38,"tag":121,"props":552,"children":553},{},[554],{"type":38,"tag":130,"props":555,"children":557},{"className":556},[],[558],{"type":44,"value":559},"0.0",{"type":38,"tag":121,"props":561,"children":562},{},[563],{"type":44,"value":564},"Fraction of optimizer states on CPU (0.0–1.0)",{"type":38,"tag":89,"props":566,"children":567},{},[568,577,585],{"type":38,"tag":121,"props":569,"children":570},{},[571],{"type":38,"tag":130,"props":572,"children":574},{"className":573},[],[575],{"type":44,"value":576},"overlap_cpu_optimizer_d2h_h2d",{"type":38,"tag":121,"props":578,"children":579},{},[580],{"type":38,"tag":130,"props":581,"children":583},{"className":582},[],[584],{"type":44,"value":534},{"type":38,"tag":121,"props":586,"children":587},{},[588],{"type":44,"value":589},"Overlap GPU↔CPU transfers with compute",{"type":38,"tag":89,"props":591,"children":592},{},[593,602,610],{"type":38,"tag":121,"props":594,"children":595},{},[596],{"type":38,"tag":130,"props":597,"children":599},{"className":598},[],[600],{"type":44,"value":601},"use_torch_optimizer_for_cpu_offload",{"type":38,"tag":121,"props":603,"children":604},{},[605],{"type":38,"tag":130,"props":606,"children":608},{"className":607},[],[609],{"type":44,"value":534},{"type":38,"tag":121,"props":611,"children":612},{},[613,615,621],{"type":44,"value":614},"Use ",{"type":38,"tag":130,"props":616,"children":618},{"className":617},[],[619],{"type":44,"value":620},"torch.optim",{"type":44,"value":622}," instead of fused optimizer for CPU portion",{"type":38,"tag":297,"props":624,"children":626},{"id":625},"activation-offloading",[627],{"type":44,"value":125},{"type":38,"tag":81,"props":629,"children":630},{},[631,649],{"type":38,"tag":85,"props":632,"children":633},{},[634],{"type":38,"tag":89,"props":635,"children":636},{},[637,641,645],{"type":38,"tag":93,"props":638,"children":639},{},[640],{"type":44,"value":500},{"type":38,"tag":93,"props":642,"children":643},{},[644],{"type":44,"value":505},{"type":38,"tag":93,"props":646,"children":647},{},[648],{"type":44,"value":510},{"type":38,"tag":114,"props":650,"children":651},{},[652,676,702,728,753],{"type":38,"tag":89,"props":653,"children":654},{},[655,664,672],{"type":38,"tag":121,"props":656,"children":657},{},[658],{"type":38,"tag":130,"props":659,"children":661},{"className":660},[],[662],{"type":44,"value":663},"cpu_offloading",{"type":38,"tag":121,"props":665,"children":666},{},[667],{"type":38,"tag":130,"props":668,"children":670},{"className":669},[],[671],{"type":44,"value":534},{"type":38,"tag":121,"props":673,"children":674},{},[675],{"type":44,"value":539},{"type":38,"tag":89,"props":677,"children":678},{},[679,688,697],{"type":38,"tag":121,"props":680,"children":681},{},[682],{"type":38,"tag":130,"props":683,"children":685},{"className":684},[],[686],{"type":44,"value":687},"cpu_offloading_num_layers",{"type":38,"tag":121,"props":689,"children":690},{},[691],{"type":38,"tag":130,"props":692,"children":694},{"className":693},[],[695],{"type":44,"value":696},"0",{"type":38,"tag":121,"props":698,"children":699},{},[700],{"type":44,"value":701},"Number of transformer layers to offload (0 to num_layers-1)",{"type":38,"tag":89,"props":703,"children":704},{},[705,714,723],{"type":38,"tag":121,"props":706,"children":707},{},[708],{"type":38,"tag":130,"props":709,"children":711},{"className":710},[],[712],{"type":44,"value":713},"cpu_offloading_activations",{"type":38,"tag":121,"props":715,"children":716},{},[717],{"type":38,"tag":130,"props":718,"children":720},{"className":719},[],[721],{"type":44,"value":722},"True",{"type":38,"tag":121,"props":724,"children":725},{},[726],{"type":44,"value":727},"Offload activations",{"type":38,"tag":89,"props":729,"children":730},{},[731,740,748],{"type":38,"tag":121,"props":732,"children":733},{},[734],{"type":38,"tag":130,"props":735,"children":737},{"className":736},[],[738],{"type":44,"value":739},"cpu_offloading_weights",{"type":38,"tag":121,"props":741,"children":742},{},[743],{"type":38,"tag":130,"props":744,"children":746},{"className":745},[],[747],{"type":44,"value":534},{"type":38,"tag":121,"props":749,"children":750},{},[751],{"type":44,"value":752},"Offload weights",{"type":38,"tag":89,"props":754,"children":755},{},[756,765,773],{"type":38,"tag":121,"props":757,"children":758},{},[759],{"type":38,"tag":130,"props":760,"children":762},{"className":761},[],[763],{"type":44,"value":764},"cpu_offloading_double_buffering",{"type":38,"tag":121,"props":766,"children":767},{},[768],{"type":38,"tag":130,"props":769,"children":771},{"className":770},[],[772],{"type":44,"value":534},{"type":38,"tag":121,"props":774,"children":775},{},[776],{"type":44,"value":777},"Double-buffer across layers while reloading",{"type":38,"tag":47,"props":779,"children":781},{"id":780},"compatibility-and-constraints",[782],{"type":44,"value":783},"Compatibility And Constraints",{"type":38,"tag":297,"props":785,"children":787},{"id":786},"activation-offloading-1",[788],{"type":44,"value":125},{"type":38,"tag":54,"props":790,"children":791},{},[792,803,819,830,835],{"type":38,"tag":58,"props":793,"children":794},{},[795,801],{"type":38,"tag":130,"props":796,"children":798},{"className":797},[],[799],{"type":44,"value":800},"pipeline_model_parallel_size",{"type":44,"value":802}," must be 1",{"type":38,"tag":58,"props":804,"children":805},{},[806,812,814],{"type":38,"tag":130,"props":807,"children":809},{"className":808},[],[810],{"type":44,"value":811},"recompute_granularity",{"type":44,"value":813}," must be ",{"type":38,"tag":130,"props":815,"children":817},{"className":816},[],[818],{"type":44,"value":178},{"type":38,"tag":58,"props":820,"children":821},{},[822,824],{"type":44,"value":823},"Cannot combine with ",{"type":38,"tag":130,"props":825,"children":827},{"className":826},[],[828],{"type":44,"value":829},"fine_grained_activation_offloading",{"type":38,"tag":58,"props":831,"children":832},{},[833],{"type":44,"value":834},"Cannot combine with CUDA graphs",{"type":38,"tag":58,"props":836,"children":837},{},[838,843,845],{"type":38,"tag":130,"props":839,"children":841},{"className":840},[],[842],{"type":44,"value":687},{"type":44,"value":844}," must be in ",{"type":38,"tag":130,"props":846,"children":848},{"className":847},[],[849],{"type":44,"value":850},"[0, num_layers-1)",{"type":38,"tag":297,"props":852,"children":854},{"id":853},"optimizer-offloading-1",[855],{"type":44,"value":153},{"type":38,"tag":54,"props":857,"children":858},{},[859,872,877],{"type":38,"tag":58,"props":860,"children":861},{},[862,864,870],{"type":44,"value":863},"Requires ",{"type":38,"tag":130,"props":865,"children":867},{"className":866},[],[868],{"type":44,"value":869},"use_distributed_optimizer = True",{"type":44,"value":871}," (default in most recipes)",{"type":38,"tag":58,"props":873,"children":874},{},[875],{"type":44,"value":876},"No PP, recompute, or CUDA graph restrictions",{"type":38,"tag":58,"props":878,"children":879},{},[880,885,886],{"type":38,"tag":130,"props":881,"children":883},{"className":882},[],[884],{"type":44,"value":250},{"type":44,"value":844},{"type":38,"tag":130,"props":887,"children":889},{"className":888},[],[890],{"type":44,"value":891},"[0.0, 1.0]",{"type":38,"tag":297,"props":893,"children":895},{"id":894},"practical-large-moe-models",[896],{"type":44,"value":897},"Practical: large MoE models",{"type":38,"tag":75,"props":899,"children":900},{},[901],{"type":44,"value":902},"Activation offloading is blocked for Qwen3-30B-A3B and similar large MoE\nmodels. The PP=1 constraint means each GPU holds all 48 layers; model\nweights + optimizer states alone (~70 GB) exceed H100 80 GB capacity.",{"type":38,"tag":47,"props":904,"children":906},{"id":905},"minimal-runnable-command",[907],{"type":44,"value":908},"Minimal Runnable Command",{"type":38,"tag":304,"props":910,"children":912},{"className":350,"code":911,"language":352,"meta":309,"style":309},"uv run python scripts\u002Ftraining\u002Frun_recipe.py \\\n  --recipe qwen3_30b_a3b_pretrain_config \\\n  optimizer.optimizer_cpu_offload=True \\\n  optimizer.optimizer_offload_fraction=0.5 \\\n  train.train_iters=20 \\\n  train.global_batch_size=8 \\\n  train.micro_batch_size=1\n",[913],{"type":38,"tag":130,"props":914,"children":915},{"__ignoreMap":309},[916,943,960,972,990,1007,1024],{"type":38,"tag":315,"props":917,"children":918},{"class":317,"line":318},[919,924,929,934,939],{"type":38,"tag":315,"props":920,"children":921},{"style":362},[922],{"type":44,"value":923},"uv",{"type":38,"tag":315,"props":925,"children":926},{"style":367},[927],{"type":44,"value":928}," run",{"type":38,"tag":315,"props":930,"children":931},{"style":367},[932],{"type":44,"value":933}," python",{"type":38,"tag":315,"props":935,"children":936},{"style":367},[937],{"type":44,"value":938}," scripts\u002Ftraining\u002Frun_recipe.py",{"type":38,"tag":315,"props":940,"children":941},{"style":373},[942],{"type":44,"value":376},{"type":38,"tag":315,"props":944,"children":945},{"class":317,"line":327},[946,951,956],{"type":38,"tag":315,"props":947,"children":948},{"style":367},[949],{"type":44,"value":950},"  --recipe",{"type":38,"tag":315,"props":952,"children":953},{"style":367},[954],{"type":44,"value":955}," qwen3_30b_a3b_pretrain_config",{"type":38,"tag":315,"props":957,"children":958},{"style":373},[959],{"type":44,"value":376},{"type":38,"tag":315,"props":961,"children":962},{"class":317,"line":336},[963,968],{"type":38,"tag":315,"props":964,"children":965},{"style":367},[966],{"type":44,"value":967},"  optimizer.optimizer_cpu_offload=True",{"type":38,"tag":315,"props":969,"children":970},{"style":373},[971],{"type":44,"value":376},{"type":38,"tag":315,"props":973,"children":974},{"class":317,"line":432},[975,980,986],{"type":38,"tag":315,"props":976,"children":977},{"style":367},[978],{"type":44,"value":979},"  optimizer.optimizer_offload_fraction=",{"type":38,"tag":315,"props":981,"children":983},{"style":982},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[984],{"type":44,"value":985},"0.5",{"type":38,"tag":315,"props":987,"children":988},{"style":373},[989],{"type":44,"value":376},{"type":38,"tag":315,"props":991,"children":992},{"class":317,"line":441},[993,998,1003],{"type":38,"tag":315,"props":994,"children":995},{"style":367},[996],{"type":44,"value":997},"  train.train_iters=",{"type":38,"tag":315,"props":999,"children":1000},{"style":982},[1001],{"type":44,"value":1002},"20",{"type":38,"tag":315,"props":1004,"children":1005},{"style":373},[1006],{"type":44,"value":376},{"type":38,"tag":315,"props":1008,"children":1009},{"class":317,"line":451},[1010,1015,1020],{"type":38,"tag":315,"props":1011,"children":1012},{"style":367},[1013],{"type":44,"value":1014},"  train.global_batch_size=",{"type":38,"tag":315,"props":1016,"children":1017},{"style":982},[1018],{"type":44,"value":1019},"8",{"type":38,"tag":315,"props":1021,"children":1022},{"style":373},[1023],{"type":44,"value":376},{"type":38,"tag":315,"props":1025,"children":1026},{"class":317,"line":460},[1027,1032],{"type":38,"tag":315,"props":1028,"children":1029},{"style":367},[1030],{"type":44,"value":1031},"  train.micro_batch_size=",{"type":38,"tag":315,"props":1033,"children":1034},{"style":982},[1035],{"type":44,"value":1036},"1\n",{"type":38,"tag":47,"props":1038,"children":1040},{"id":1039},"verification",[1041],{"type":44,"value":1042},"Verification",{"type":38,"tag":297,"props":1044,"children":1046},{"id":1045},"unit-tests",[1047],{"type":44,"value":1048},"Unit tests",{"type":38,"tag":304,"props":1050,"children":1052},{"className":350,"code":1051,"language":352,"meta":309,"style":309},"uv run python -m pytest \\\n  tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_full_te_layer_autocast_spec.py -k \"cpu_offload\" \\\n  tests\u002Funit_tests\u002Fpeft\u002Ftest_utils.py -k \"cpu_offload\" -q\n",[1053],{"type":38,"tag":130,"props":1054,"children":1055},{"__ignoreMap":309},[1056,1085,1118],{"type":38,"tag":315,"props":1057,"children":1058},{"class":317,"line":318},[1059,1063,1067,1071,1076,1081],{"type":38,"tag":315,"props":1060,"children":1061},{"style":362},[1062],{"type":44,"value":923},{"type":38,"tag":315,"props":1064,"children":1065},{"style":367},[1066],{"type":44,"value":928},{"type":38,"tag":315,"props":1068,"children":1069},{"style":367},[1070],{"type":44,"value":933},{"type":38,"tag":315,"props":1072,"children":1073},{"style":367},[1074],{"type":44,"value":1075}," -m",{"type":38,"tag":315,"props":1077,"children":1078},{"style":367},[1079],{"type":44,"value":1080}," pytest",{"type":38,"tag":315,"props":1082,"children":1083},{"style":373},[1084],{"type":44,"value":376},{"type":38,"tag":315,"props":1086,"children":1087},{"class":317,"line":327},[1088,1093,1098,1104,1109,1114],{"type":38,"tag":315,"props":1089,"children":1090},{"style":367},[1091],{"type":44,"value":1092},"  tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_full_te_layer_autocast_spec.py",{"type":38,"tag":315,"props":1094,"children":1095},{"style":367},[1096],{"type":44,"value":1097}," -k",{"type":38,"tag":315,"props":1099,"children":1101},{"style":1100},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1102],{"type":44,"value":1103}," \"",{"type":38,"tag":315,"props":1105,"children":1106},{"style":367},[1107],{"type":44,"value":1108},"cpu_offload",{"type":38,"tag":315,"props":1110,"children":1111},{"style":1100},[1112],{"type":44,"value":1113},"\"",{"type":38,"tag":315,"props":1115,"children":1116},{"style":373},[1117],{"type":44,"value":376},{"type":38,"tag":315,"props":1119,"children":1120},{"class":317,"line":336},[1121,1126,1130,1134,1138,1142],{"type":38,"tag":315,"props":1122,"children":1123},{"style":367},[1124],{"type":44,"value":1125},"  tests\u002Funit_tests\u002Fpeft\u002Ftest_utils.py",{"type":38,"tag":315,"props":1127,"children":1128},{"style":367},[1129],{"type":44,"value":1097},{"type":38,"tag":315,"props":1131,"children":1132},{"style":1100},[1133],{"type":44,"value":1103},{"type":38,"tag":315,"props":1135,"children":1136},{"style":367},[1137],{"type":44,"value":1108},{"type":38,"tag":315,"props":1139,"children":1140},{"style":1100},[1141],{"type":44,"value":1113},{"type":38,"tag":315,"props":1143,"children":1144},{"style":367},[1145],{"type":44,"value":1146}," -q\n",{"type":38,"tag":297,"props":1148,"children":1150},{"id":1149},"success-criteria",[1151],{"type":44,"value":1152},"Success criteria",{"type":38,"tag":54,"props":1154,"children":1155},{},[1156,1161,1166,1171],{"type":38,"tag":58,"props":1157,"children":1158},{},[1159],{"type":44,"value":1160},"Config validation passes for the selected offloading mode",{"type":38,"tag":58,"props":1162,"children":1163},{},[1164],{"type":44,"value":1165},"Training completes without OOM or NCCL errors",{"type":38,"tag":58,"props":1167,"children":1168},{},[1169],{"type":44,"value":1170},"Loss matches the non-offloaded baseline (max delta \u003C 0.001)",{"type":38,"tag":58,"props":1172,"children":1173},{},[1174],{"type":44,"value":1175},"Memory usage drops proportionally to offload fraction",{"type":38,"tag":47,"props":1177,"children":1179},{"id":1178},"code-anchors",[1180],{"type":44,"value":1181},"Code Anchors",{"type":38,"tag":297,"props":1183,"children":1185},{"id":1184},"mcore-activation-offload-constraints",[1186],{"type":44,"value":1187},"MCore activation offload constraints",{"type":38,"tag":304,"props":1189,"children":1193},{"className":1190,"code":1191,"language":1192,"meta":309,"style":309},"language-1296:1310:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","        if self.cpu_offloading and (\n            self.cpu_offloading_num_layers \u003C 0 or self.cpu_offloading_num_layers >= self.num_layers\n        ):\n            raise ValueError(...)\n\n        if self.cpu_offloading and self.pipeline_model_parallel_size > 1:\n            raise ValueError(\n                \"Currently there is no support for Pipeline parallelism with CPU offloading\"\n            )\n\n        if self.cpu_offloading and self.recompute_granularity is not None:\n            raise ValueError(\n                \"CPU offloading does not work when activation recomputation is enabled\"\n            )\n","1296:1310:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py",[1194],{"type":38,"tag":130,"props":1195,"children":1196},{"__ignoreMap":309},[1197,1205,1213,1221,1229,1236,1244,1252,1260,1269,1277,1286,1294,1303],{"type":38,"tag":315,"props":1198,"children":1199},{"class":317,"line":318},[1200],{"type":38,"tag":315,"props":1201,"children":1202},{},[1203],{"type":44,"value":1204},"        if self.cpu_offloading and (\n",{"type":38,"tag":315,"props":1206,"children":1207},{"class":317,"line":327},[1208],{"type":38,"tag":315,"props":1209,"children":1210},{},[1211],{"type":44,"value":1212},"            self.cpu_offloading_num_layers \u003C 0 or self.cpu_offloading_num_layers >= self.num_layers\n",{"type":38,"tag":315,"props":1214,"children":1215},{"class":317,"line":336},[1216],{"type":38,"tag":315,"props":1217,"children":1218},{},[1219],{"type":44,"value":1220},"        ):\n",{"type":38,"tag":315,"props":1222,"children":1223},{"class":317,"line":432},[1224],{"type":38,"tag":315,"props":1225,"children":1226},{},[1227],{"type":44,"value":1228},"            raise ValueError(...)\n",{"type":38,"tag":315,"props":1230,"children":1231},{"class":317,"line":441},[1232],{"type":38,"tag":315,"props":1233,"children":1234},{"emptyLinePlaceholder":445},[1235],{"type":44,"value":448},{"type":38,"tag":315,"props":1237,"children":1238},{"class":317,"line":451},[1239],{"type":38,"tag":315,"props":1240,"children":1241},{},[1242],{"type":44,"value":1243},"        if self.cpu_offloading and self.pipeline_model_parallel_size > 1:\n",{"type":38,"tag":315,"props":1245,"children":1246},{"class":317,"line":460},[1247],{"type":38,"tag":315,"props":1248,"children":1249},{},[1250],{"type":44,"value":1251},"            raise ValueError(\n",{"type":38,"tag":315,"props":1253,"children":1254},{"class":317,"line":469},[1255],{"type":38,"tag":315,"props":1256,"children":1257},{},[1258],{"type":44,"value":1259},"                \"Currently there is no support for Pipeline parallelism with CPU offloading\"\n",{"type":38,"tag":315,"props":1261,"children":1263},{"class":317,"line":1262},9,[1264],{"type":38,"tag":315,"props":1265,"children":1266},{},[1267],{"type":44,"value":1268},"            )\n",{"type":38,"tag":315,"props":1270,"children":1272},{"class":317,"line":1271},10,[1273],{"type":38,"tag":315,"props":1274,"children":1275},{"emptyLinePlaceholder":445},[1276],{"type":44,"value":448},{"type":38,"tag":315,"props":1278,"children":1280},{"class":317,"line":1279},11,[1281],{"type":38,"tag":315,"props":1282,"children":1283},{},[1284],{"type":44,"value":1285},"        if self.cpu_offloading and self.recompute_granularity is not None:\n",{"type":38,"tag":315,"props":1287,"children":1289},{"class":317,"line":1288},12,[1290],{"type":38,"tag":315,"props":1291,"children":1292},{},[1293],{"type":44,"value":1251},{"type":38,"tag":315,"props":1295,"children":1297},{"class":317,"line":1296},13,[1298],{"type":38,"tag":315,"props":1299,"children":1300},{},[1301],{"type":44,"value":1302},"                \"CPU offloading does not work when activation recomputation is enabled\"\n",{"type":38,"tag":315,"props":1304,"children":1306},{"class":317,"line":1305},14,[1307],{"type":38,"tag":315,"props":1308,"children":1309},{},[1310],{"type":44,"value":1268},{"type":38,"tag":297,"props":1312,"children":1314},{"id":1313},"mcore-cuda-graph-incompatibility",[1315],{"type":44,"value":1316},"MCore CUDA graph incompatibility",{"type":38,"tag":304,"props":1318,"children":1322},{"className":1319,"code":1320,"language":1321,"meta":309,"style":309},"language-1943:1944:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","            if self.cpu_offloading:\n                raise ValueError(\"CUDA graphs not supported with CPU offloading.\")\n","1943:1944:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py",[1323],{"type":38,"tag":130,"props":1324,"children":1325},{"__ignoreMap":309},[1326,1334],{"type":38,"tag":315,"props":1327,"children":1328},{"class":317,"line":318},[1329],{"type":38,"tag":315,"props":1330,"children":1331},{},[1332],{"type":44,"value":1333},"            if self.cpu_offloading:\n",{"type":38,"tag":315,"props":1335,"children":1336},{"class":317,"line":327},[1337],{"type":38,"tag":315,"props":1338,"children":1339},{},[1340],{"type":44,"value":1341},"                raise ValueError(\"CUDA graphs not supported with CPU offloading.\")\n",{"type":38,"tag":297,"props":1343,"children":1345},{"id":1344},"mcore-fine-grained-offloading-mutual-exclusion",[1346],{"type":44,"value":1347},"MCore fine-grained offloading mutual exclusion",{"type":38,"tag":304,"props":1349,"children":1353},{"className":1350,"code":1351,"language":1352,"meta":309,"style":309},"language-1427:1430:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","        if self.fine_grained_activation_offloading:\n            assert (\n                not self.cpu_offloading\n            ), \"fine_grained_activation_offloading cannot be enabled with cpu_offloading.\"\n","1427:1430:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Ftransformer\u002Ftransformer_config.py",[1354],{"type":38,"tag":130,"props":1355,"children":1356},{"__ignoreMap":309},[1357,1365,1373,1381],{"type":38,"tag":315,"props":1358,"children":1359},{"class":317,"line":318},[1360],{"type":38,"tag":315,"props":1361,"children":1362},{},[1363],{"type":44,"value":1364},"        if self.fine_grained_activation_offloading:\n",{"type":38,"tag":315,"props":1366,"children":1367},{"class":317,"line":327},[1368],{"type":38,"tag":315,"props":1369,"children":1370},{},[1371],{"type":44,"value":1372},"            assert (\n",{"type":38,"tag":315,"props":1374,"children":1375},{"class":317,"line":336},[1376],{"type":38,"tag":315,"props":1377,"children":1378},{},[1379],{"type":44,"value":1380},"                not self.cpu_offloading\n",{"type":38,"tag":315,"props":1382,"children":1383},{"class":317,"line":432},[1384],{"type":38,"tag":315,"props":1385,"children":1386},{},[1387],{"type":44,"value":1388},"            ), \"fine_grained_activation_offloading cannot be enabled with cpu_offloading.\"\n",{"type":38,"tag":297,"props":1390,"children":1392},{"id":1391},"mcore-hybriddeviceoptimizer-instantiation",[1393],{"type":44,"value":1394},"MCore HybridDeviceOptimizer instantiation",{"type":38,"tag":304,"props":1396,"children":1400},{"className":1397,"code":1398,"language":1399,"meta":309,"style":309},"language-480:518:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Foptimizer\u002F__init__.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","        if config.optimizer_cpu_offload:\n            # ... setup cpu\u002Fgpu optimizer classes ...\n            optimizer = HybridDeviceOptimizer(\n                param_groups,\n                offload_fraction=config.optimizer_offload_fraction,\n                cpu_optimizer_cls=cpu_optimizer_cls,\n                gpu_optimizer_cls=gpu_optimizer_cls,\n                overlap_cpu_optimizer_d2h_h2d=config.overlap_cpu_optimizer_d2h_h2d,\n                pin_cpu_grads=config.pin_cpu_grads,\n                pin_cpu_params=config.pin_cpu_params,\n            )\n","480:518:3rdparty\u002FMegatron-LM\u002Fmegatron\u002Fcore\u002Foptimizer\u002F__init__.py",[1401],{"type":38,"tag":130,"props":1402,"children":1403},{"__ignoreMap":309},[1404,1412,1420,1428,1436,1444,1452,1460,1468,1476,1484],{"type":38,"tag":315,"props":1405,"children":1406},{"class":317,"line":318},[1407],{"type":38,"tag":315,"props":1408,"children":1409},{},[1410],{"type":44,"value":1411},"        if config.optimizer_cpu_offload:\n",{"type":38,"tag":315,"props":1413,"children":1414},{"class":317,"line":327},[1415],{"type":38,"tag":315,"props":1416,"children":1417},{},[1418],{"type":44,"value":1419},"            # ... setup cpu\u002Fgpu optimizer classes ...\n",{"type":38,"tag":315,"props":1421,"children":1422},{"class":317,"line":336},[1423],{"type":38,"tag":315,"props":1424,"children":1425},{},[1426],{"type":44,"value":1427},"            optimizer = HybridDeviceOptimizer(\n",{"type":38,"tag":315,"props":1429,"children":1430},{"class":317,"line":432},[1431],{"type":38,"tag":315,"props":1432,"children":1433},{},[1434],{"type":44,"value":1435},"                param_groups,\n",{"type":38,"tag":315,"props":1437,"children":1438},{"class":317,"line":441},[1439],{"type":38,"tag":315,"props":1440,"children":1441},{},[1442],{"type":44,"value":1443},"                offload_fraction=config.optimizer_offload_fraction,\n",{"type":38,"tag":315,"props":1445,"children":1446},{"class":317,"line":451},[1447],{"type":38,"tag":315,"props":1448,"children":1449},{},[1450],{"type":44,"value":1451},"                cpu_optimizer_cls=cpu_optimizer_cls,\n",{"type":38,"tag":315,"props":1453,"children":1454},{"class":317,"line":460},[1455],{"type":38,"tag":315,"props":1456,"children":1457},{},[1458],{"type":44,"value":1459},"                gpu_optimizer_cls=gpu_optimizer_cls,\n",{"type":38,"tag":315,"props":1461,"children":1462},{"class":317,"line":469},[1463],{"type":38,"tag":315,"props":1464,"children":1465},{},[1466],{"type":44,"value":1467},"                overlap_cpu_optimizer_d2h_h2d=config.overlap_cpu_optimizer_d2h_h2d,\n",{"type":38,"tag":315,"props":1469,"children":1470},{"class":317,"line":1262},[1471],{"type":38,"tag":315,"props":1472,"children":1473},{},[1474],{"type":44,"value":1475},"                pin_cpu_grads=config.pin_cpu_grads,\n",{"type":38,"tag":315,"props":1477,"children":1478},{"class":317,"line":1271},[1479],{"type":38,"tag":315,"props":1480,"children":1481},{},[1482],{"type":44,"value":1483},"                pin_cpu_params=config.pin_cpu_params,\n",{"type":38,"tag":315,"props":1485,"children":1486},{"class":317,"line":1279},[1487],{"type":38,"tag":315,"props":1488,"children":1489},{},[1490],{"type":44,"value":1268},{"type":38,"tag":297,"props":1492,"children":1494},{"id":1493},"bridge-cuda-graph-guard",[1495],{"type":44,"value":1496},"Bridge CUDA graph guard",{"type":38,"tag":304,"props":1498,"children":1502},{"className":1499,"code":1500,"language":1501,"meta":309,"style":309},"language-232:234:src\u002Fmegatron\u002Fbridge\u002Fmodels\u002Fgpt_full_te_layer_autocast_spec.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","        assert not config.cpu_offloading and config.recompute_granularity is None, \"Cudagraphs not supported\"\n","232:234:src\u002Fmegatron\u002Fbridge\u002Fmodels\u002Fgpt_full_te_layer_autocast_spec.py",[1503],{"type":38,"tag":130,"props":1504,"children":1505},{"__ignoreMap":309},[1506],{"type":38,"tag":315,"props":1507,"children":1508},{"class":317,"line":318},[1509],{"type":38,"tag":315,"props":1510,"children":1511},{},[1512],{"type":44,"value":1500},{"type":38,"tag":297,"props":1514,"children":1516},{"id":1515},"bridge-activation-offloading-in-peft",[1517],{"type":44,"value":1518},"Bridge activation offloading in PEFT",{"type":38,"tag":304,"props":1520,"children":1524},{"className":1521,"code":1522,"language":1523,"meta":309,"style":309},"language-621:631:src\u002Fmegatron\u002Fbridge\u002Fpeft\u002Futils.py shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","        if self.config.cpu_offloading and self.config.cpu_offloading_activations:\n            x.activation_offloading = True\n        x, _ = self.linear_in(x)\n        x = self.activation(x)\n        if self.config.cpu_offloading and self.config.cpu_offloading_activations:\n            x.activation_offloading = True\n        x, _ = self.linear_out(x)\n","621:631:src\u002Fmegatron\u002Fbridge\u002Fpeft\u002Futils.py",[1525],{"type":38,"tag":130,"props":1526,"children":1527},{"__ignoreMap":309},[1528,1536,1544,1552,1560,1567,1574],{"type":38,"tag":315,"props":1529,"children":1530},{"class":317,"line":318},[1531],{"type":38,"tag":315,"props":1532,"children":1533},{},[1534],{"type":44,"value":1535},"        if self.config.cpu_offloading and self.config.cpu_offloading_activations:\n",{"type":38,"tag":315,"props":1537,"children":1538},{"class":317,"line":327},[1539],{"type":38,"tag":315,"props":1540,"children":1541},{},[1542],{"type":44,"value":1543},"            x.activation_offloading = True\n",{"type":38,"tag":315,"props":1545,"children":1546},{"class":317,"line":336},[1547],{"type":38,"tag":315,"props":1548,"children":1549},{},[1550],{"type":44,"value":1551},"        x, _ = self.linear_in(x)\n",{"type":38,"tag":315,"props":1553,"children":1554},{"class":317,"line":432},[1555],{"type":38,"tag":315,"props":1556,"children":1557},{},[1558],{"type":44,"value":1559},"        x = self.activation(x)\n",{"type":38,"tag":315,"props":1561,"children":1562},{"class":317,"line":441},[1563],{"type":38,"tag":315,"props":1564,"children":1565},{},[1566],{"type":44,"value":1535},{"type":38,"tag":315,"props":1568,"children":1569},{"class":317,"line":451},[1570],{"type":38,"tag":315,"props":1571,"children":1572},{},[1573],{"type":44,"value":1543},{"type":38,"tag":315,"props":1575,"children":1576},{"class":317,"line":460},[1577],{"type":38,"tag":315,"props":1578,"children":1579},{},[1580],{"type":44,"value":1581},"        x, _ = self.linear_out(x)\n",{"type":38,"tag":47,"props":1583,"children":1585},{"id":1584},"failure-diagnosis",[1586],{"type":44,"value":1587},"Failure Diagnosis",{"type":38,"tag":81,"props":1589,"children":1590},{},[1591,1617],{"type":38,"tag":85,"props":1592,"children":1593},{},[1594],{"type":38,"tag":89,"props":1595,"children":1596},{},[1597,1602,1607,1612],{"type":38,"tag":93,"props":1598,"children":1599},{},[1600],{"type":44,"value":1601},"Symptom",{"type":38,"tag":93,"props":1603,"children":1604},{},[1605],{"type":44,"value":1606},"Likely Cause",{"type":38,"tag":93,"props":1608,"children":1609},{},[1610],{"type":44,"value":1611},"How To Confirm",{"type":38,"tag":93,"props":1613,"children":1614},{},[1615],{"type":44,"value":1616},"Fix",{"type":38,"tag":114,"props":1618,"children":1619},{},[1620,1652,1689,1716,1753,1776,1804],{"type":38,"tag":89,"props":1621,"children":1622},{},[1623,1632,1637,1647],{"type":38,"tag":121,"props":1624,"children":1625},{},[1626],{"type":38,"tag":130,"props":1627,"children":1629},{"className":1628},[],[1630],{"type":44,"value":1631},"Currently there is no support for Pipeline parallelism with CPU offloading",{"type":38,"tag":121,"props":1633,"children":1634},{},[1635],{"type":44,"value":1636},"Activation offload + PP > 1",{"type":38,"tag":121,"props":1638,"children":1639},{},[1640,1642],{"type":44,"value":1641},"Check ",{"type":38,"tag":130,"props":1643,"children":1645},{"className":1644},[],[1646],{"type":44,"value":800},{"type":38,"tag":121,"props":1648,"children":1649},{},[1650],{"type":44,"value":1651},"Set PP=1 or use optimizer offloading",{"type":38,"tag":89,"props":1653,"children":1654},{},[1655,1664,1669,1678],{"type":38,"tag":121,"props":1656,"children":1657},{},[1658],{"type":38,"tag":130,"props":1659,"children":1661},{"className":1660},[],[1662],{"type":44,"value":1663},"CPU offloading does not work when activation recomputation is enabled",{"type":38,"tag":121,"props":1665,"children":1666},{},[1667],{"type":44,"value":1668},"Activation offload + recompute",{"type":38,"tag":121,"props":1670,"children":1671},{},[1672,1673],{"type":44,"value":1641},{"type":38,"tag":130,"props":1674,"children":1676},{"className":1675},[],[1677],{"type":44,"value":811},{"type":38,"tag":121,"props":1679,"children":1680},{},[1681,1683],{"type":44,"value":1682},"Set ",{"type":38,"tag":130,"props":1684,"children":1686},{"className":1685},[],[1687],{"type":44,"value":1688},"recompute_granularity=null",{"type":38,"tag":89,"props":1690,"children":1691},{},[1692,1701,1706,1711],{"type":38,"tag":121,"props":1693,"children":1694},{},[1695],{"type":38,"tag":130,"props":1696,"children":1698},{"className":1697},[],[1699],{"type":44,"value":1700},"fine_grained_activation_offloading cannot be enabled with cpu_offloading",{"type":38,"tag":121,"props":1702,"children":1703},{},[1704],{"type":44,"value":1705},"Both offloading modes enabled",{"type":38,"tag":121,"props":1707,"children":1708},{},[1709],{"type":44,"value":1710},"Check both flags",{"type":38,"tag":121,"props":1712,"children":1713},{},[1714],{"type":44,"value":1715},"Use one or the other",{"type":38,"tag":89,"props":1717,"children":1718},{},[1719,1728,1733,1743],{"type":38,"tag":121,"props":1720,"children":1721},{},[1722],{"type":38,"tag":130,"props":1723,"children":1725},{"className":1724},[],[1726],{"type":44,"value":1727},"CUDA graphs not supported with CPU offloading",{"type":38,"tag":121,"props":1729,"children":1730},{},[1731],{"type":44,"value":1732},"CUDA graphs + activation offload",{"type":38,"tag":121,"props":1734,"children":1735},{},[1736,1737],{"type":44,"value":1641},{"type":38,"tag":130,"props":1738,"children":1740},{"className":1739},[],[1741],{"type":44,"value":1742},"cuda_graph_impl",{"type":38,"tag":121,"props":1744,"children":1745},{},[1746,1747],{"type":44,"value":1682},{"type":38,"tag":130,"props":1748,"children":1750},{"className":1749},[],[1751],{"type":44,"value":1752},"cuda_graph_impl=\"none\"",{"type":38,"tag":89,"props":1754,"children":1755},{},[1756,1761,1766,1771],{"type":38,"tag":121,"props":1757,"children":1758},{},[1759],{"type":44,"value":1760},"OOM with activation offloading",{"type":38,"tag":121,"props":1762,"children":1763},{},[1764],{"type":44,"value":1765},"Model too large for PP=1",{"type":38,"tag":121,"props":1767,"children":1768},{},[1769],{"type":44,"value":1770},"Check allocated memory vs 80 GB",{"type":38,"tag":121,"props":1772,"children":1773},{},[1774],{"type":44,"value":1775},"Use optimizer offloading with PP > 1",{"type":38,"tag":89,"props":1777,"children":1778},{},[1779,1784,1789,1794],{"type":38,"tag":121,"props":1780,"children":1781},{},[1782],{"type":44,"value":1783},"Extreme slowdown (>4x)",{"type":38,"tag":121,"props":1785,"children":1786},{},[1787],{"type":44,"value":1788},"100% optimizer offload, CPU Adam bottleneck",{"type":38,"tag":121,"props":1790,"children":1791},{},[1792],{"type":44,"value":1793},"Compare iter time at different fractions",{"type":38,"tag":121,"props":1795,"children":1796},{},[1797,1799],{"type":44,"value":1798},"Reduce fraction or enable ",{"type":38,"tag":130,"props":1800,"children":1802},{"className":1801},[],[1803],{"type":44,"value":576},{"type":38,"tag":89,"props":1805,"children":1806},{},[1807,1812,1817,1822],{"type":38,"tag":121,"props":1808,"children":1809},{},[1810],{"type":44,"value":1811},"OOM at partial optimizer offload",{"type":38,"tag":121,"props":1813,"children":1814},{},[1815],{"type":44,"value":1816},"Insufficient offload for this config",{"type":38,"tag":121,"props":1818,"children":1819},{},[1820],{"type":44,"value":1821},"Check memory at different fractions",{"type":38,"tag":121,"props":1823,"children":1824},{},[1825],{"type":44,"value":1826},"Increase fraction or add PP",{"type":38,"tag":47,"props":1828,"children":1830},{"id":1829},"known-limitations",[1831],{"type":44,"value":1832},"Known Limitations",{"type":38,"tag":54,"props":1834,"children":1835},{},[1836,1841,1846,1851],{"type":38,"tag":58,"props":1837,"children":1838},{},[1839],{"type":44,"value":1840},"Activation offloading requires PP=1, making it impractical for large models\n(30B+ MoE) that need pipeline parallelism.",{"type":38,"tag":58,"props":1842,"children":1843},{},[1844],{"type":44,"value":1845},"Optimizer offloading throughput penalty scales linearly (~1.9x at 25%,\n~4.2x at 100% for Qwen3-30B-A3B).",{"type":38,"tag":58,"props":1847,"children":1848},{},[1849],{"type":44,"value":1850},"D2H\u002FH2D overlap provides only ~7% speedup because CPU Adam compute is\nthe dominant bottleneck.",{"type":38,"tag":58,"props":1852,"children":1853},{},[1854,1859,1861,1866],{"type":38,"tag":130,"props":1855,"children":1857},{"className":1856},[],[1858],{"type":44,"value":829},{"type":44,"value":1860}," is a separate module-level approach\nthat works with PP > 1 but cannot be combined with layer-level\n",{"type":38,"tag":130,"props":1862,"children":1864},{"className":1863},[],[1865],{"type":44,"value":663},{"type":44,"value":1867},".",{"type":38,"tag":1869,"props":1870,"children":1871},"style",{},[1872],{"type":44,"value":1873},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1875,"total":1977},[1876,1891,1905,1919,1931,1948,1963],{"slug":1877,"name":1877,"fn":1878,"description":1879,"org":1880,"tags":1881,"stars":20,"repoUrl":21,"updatedAt":1890},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1882,1885,1888,1889],{"name":1883,"slug":1884,"type":15},"Data Analysis","data-analysis",{"name":1886,"slug":1887,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:28:43.176466",{"slug":1892,"name":1892,"fn":1893,"description":1894,"org":1895,"tags":1896,"stars":20,"repoUrl":21,"updatedAt":1904},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1897,1900,1903],{"name":1898,"slug":1899,"type":15},"Deployment","deployment",{"name":1901,"slug":1902,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":1906,"name":1906,"fn":1907,"description":1908,"org":1909,"tags":1910,"stars":20,"repoUrl":21,"updatedAt":1918},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1911,1914,1915],{"name":1912,"slug":1913,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":1916,"slug":1917,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":1920,"name":1920,"fn":1921,"description":1922,"org":1923,"tags":1924,"stars":20,"repoUrl":21,"updatedAt":1930},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1925,1926,1927],{"name":1883,"slug":1884,"type":15},{"name":9,"slug":8,"type":15},{"name":1928,"slug":1929,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":1932,"name":1932,"fn":1933,"description":1934,"org":1935,"tags":1936,"stars":20,"repoUrl":21,"updatedAt":1947},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1937,1940,1943,1944],{"name":1938,"slug":1939,"type":15},"Automation","automation",{"name":1941,"slug":1942,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":1945,"slug":1946,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":1949,"name":1949,"fn":1950,"description":1951,"org":1952,"tags":1953,"stars":20,"repoUrl":21,"updatedAt":1962},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1954,1955,1958,1959],{"name":1898,"slug":1899,"type":15},{"name":1956,"slug":1957,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":1960,"slug":1961,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":1964,"name":1964,"fn":1965,"description":1966,"org":1967,"tags":1968,"stars":20,"repoUrl":21,"updatedAt":1976},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1969,1970,1973],{"name":9,"slug":8,"type":15},{"name":1971,"slug":1972,"type":15},"Quantum Computing","quantum-computing",{"name":1974,"slug":1975,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":1979,"total":2129},[1980,1998,2013,2024,2036,2050,2063,2077,2088,2097,2111,2120],{"slug":1981,"name":1981,"fn":1982,"description":1983,"org":1984,"tags":1985,"stars":1995,"repoUrl":1996,"updatedAt":1997},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1986,1989,1992],{"name":1987,"slug":1988,"type":15},"Documentation","documentation",{"name":1990,"slug":1991,"type":15},"MCP","mcp",{"name":1993,"slug":1994,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":1999,"name":1999,"fn":2000,"description":2001,"org":2002,"tags":2003,"stars":2010,"repoUrl":2011,"updatedAt":2012},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2004,2007,2008],{"name":2005,"slug":2006,"type":15},"Containers","containers",{"name":1898,"slug":1899,"type":15},{"name":2009,"slug":308,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2014,"name":2014,"fn":2015,"description":2016,"org":2017,"tags":2018,"stars":2010,"repoUrl":2011,"updatedAt":2023},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2019,2022],{"name":2020,"slug":2021,"type":15},"CI\u002FCD","ci-cd",{"name":1898,"slug":1899,"type":15},"2026-07-14T05:25:59.97109",{"slug":2025,"name":2025,"fn":2026,"description":2027,"org":2028,"tags":2029,"stars":2010,"repoUrl":2011,"updatedAt":2035},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2030,2031,2032],{"name":2020,"slug":2021,"type":15},{"name":1898,"slug":1899,"type":15},{"name":2033,"slug":2034,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2037,"name":2037,"fn":2038,"description":2039,"org":2040,"tags":2041,"stars":2010,"repoUrl":2011,"updatedAt":2049},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2042,2045,2046],{"name":2043,"slug":2044,"type":15},"Debugging","debugging",{"name":2033,"slug":2034,"type":15},{"name":2047,"slug":2048,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2051,"name":2051,"fn":2052,"description":2053,"org":2054,"tags":2055,"stars":2010,"repoUrl":2011,"updatedAt":2062},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2056,2059],{"name":2057,"slug":2058,"type":15},"Best Practices","best-practices",{"name":2060,"slug":2061,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2064,"name":2064,"fn":2065,"description":2066,"org":2067,"tags":2068,"stars":2010,"repoUrl":2011,"updatedAt":2076},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2069,2072,2075],{"name":2070,"slug":2071,"type":15},"Machine Learning","machine-learning",{"name":2073,"slug":2074,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2078,"name":2078,"fn":2079,"description":2080,"org":2081,"tags":2082,"stars":2010,"repoUrl":2011,"updatedAt":2087},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2083,2086],{"name":2084,"slug":2085,"type":15},"QA","qa",{"name":1928,"slug":1929,"type":15},"2026-07-14T05:25:53.673039",{"slug":2089,"name":2089,"fn":2090,"description":2091,"org":2092,"tags":2093,"stars":2010,"repoUrl":2011,"updatedAt":2096},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2094,2095],{"name":1898,"slug":1899,"type":15},{"name":1901,"slug":1902,"type":15},"2026-07-14T05:25:49.362534",{"slug":2098,"name":2098,"fn":2099,"description":2100,"org":2101,"tags":2102,"stars":2010,"repoUrl":2011,"updatedAt":2110},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2103,2106,2107],{"name":2104,"slug":2105,"type":15},"Code Review","code-review",{"name":2033,"slug":2034,"type":15},{"name":2108,"slug":2109,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2112,"name":2112,"fn":2113,"description":2114,"org":2115,"tags":2116,"stars":2010,"repoUrl":2011,"updatedAt":2119},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2117,2118],{"name":2084,"slug":2085,"type":15},{"name":1928,"slug":1929,"type":15},"2026-07-14T05:25:54.928983",{"slug":2121,"name":2121,"fn":2122,"description":2123,"org":2124,"tags":2125,"stars":2010,"repoUrl":2011,"updatedAt":2128},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2126,2127],{"name":1938,"slug":1939,"type":15},{"name":2020,"slug":2021,"type":15},"2026-07-30T05:29:03.275638",496]