[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nemo-mbridge-mlm-bridge-training":3,"mdc-woop0u-key":34,"related-repo-nvidia-nemo-mbridge-mlm-bridge-training":1879,"related-org-nvidia-nemo-mbridge-mlm-bridge-training":1985},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"nemo-mbridge-mlm-bridge-training","run Megatron-LM bridge training","Run Megatron-LM (MLM) and Megatron Bridge training with mock or real data. Covers correlation testing, available recipes, and multi-GPU examples.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"LLM","llm","tag",{"name":17,"slug":18,"type":15},"Deep Learning","deep-learning",{"name":20,"slug":21,"type":15},"Machine Learning","machine-learning",{"name":9,"slug":8,"type":15},2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-17T05:28:54.895155","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fnemo-mbridge-mlm-bridge-training","---\nname: nemo-mbridge-mlm-bridge-training\ndescription: Run Megatron-LM (MLM) and Megatron Bridge training with mock or real data. Covers correlation testing, available recipes, and multi-GPU examples.\nlicense: Apache-2.0\nwhen_to_use: Running training, comparing MLM vs Bridge loss curves, translating MLM CLI args to Bridge config, or investigating why loss curves diverged after a commit; 'how do I run training', 'MLM vs Bridge', 'correlation test'.\n---\n\n# MLM vs Bridge Training\n\nFor how they differ, the arg mapping tables, gotchas, and translation script, see:\n\n- @docs\u002Fmegatron-lm-to-megatron-bridge.md\n\n## First Answer Checklist\n\nFor MLM-vs-Bridge correlation questions, always name these items up front:\n\n1. Bridge recipe: `vanilla_gpt_pretrain_config`.\n2. Bridge entry point: `scripts\u002Ftraining\u002Frun_recipe.py`.\n3. MLM entry point: `3rdparty\u002FMegatron-LM\u002Fpretrain_gpt.py`.\n4. Launch wrapper for both: `uv run python -m torch.distributed.run`.\n5. Fresh-run cleanup: `rm -rf nemo_experiments` before the Bridge run.\n\nAlso state that MLM needs\n`PYTHONPATH=3rdparty\u002FMegatron-LM:$PYTHONPATH`, matched Bridge and MLM losses\nshould agree within BF16 rounding, and files under `3rdparty\u002FMegatron-LM\u002F`\nshould not be modified from this repo.\n\n## Correlation Testing\n\nUse `vanilla_gpt_pretrain_config` for loss-correlation testing. This recipe uses\nbare `GPTModelProvider` defaults (LayerNorm, GeLU, learned_absolute position\nembeddings, `vocab_size` inherited from tokenizer) — matching MLM\n`pretrain_gpt.py` defaults with no args.\n\n### MLM Correlation Run (2L\u002F256H, 1 GPU)\n\n```bash\nPYTHONPATH=3rdparty\u002FMegatron-LM:$PYTHONPATH \\\nuv run python -m torch.distributed.run --nproc_per_node=1 \\\n  3rdparty\u002FMegatron-LM\u002Fpretrain_gpt.py \\\n  --num-layers 2 --hidden-size 256 --num-attention-heads 4 \\\n  --ffn-hidden-size 1024 --seq-length 512 --max-position-embeddings 512 \\\n  --micro-batch-size 4 --global-batch-size 32 \\\n  --train-iters 10 --eval-iters 2 --eval-interval 10 \\\n  --mock-data --bf16 --use-mcore-models \\\n  --tokenizer-type NullTokenizer --vocab-size 32000 \\\n  --lr 3e-4 --min-lr 3e-5 --seed 1234 --log-interval 1\n```\n\n### Bridge Correlation Run (same config, 1 GPU)\n\n```bash\nrm -rf nemo_experiments && \\\nuv run python -m torch.distributed.run --nproc_per_node=1 \\\n  scripts\u002Ftraining\u002Frun_recipe.py \\\n  --recipe vanilla_gpt_pretrain_config \\\n  model.num_layers=2 model.hidden_size=256 \\\n  model.num_attention_heads=4 model.ffn_hidden_size=1024 \\\n  model.seq_length=512 dataset.seq_length=512 \\\n  train.train_iters=10 train.global_batch_size=32 train.micro_batch_size=4 \\\n  validation.eval_interval=10 validation.eval_iters=2 \\\n  optimizer.lr=3e-4 optimizer.min_lr=3e-5 \\\n  scheduler.lr_warmup_iters=1 scheduler.lr_decay_iters=10 \\\n  rng.seed=1234 logger.log_interval=1\n```\n\n### Verification\n\nWith matched parameters the LM losses should be nearly identical at each\niteration. Compare `lm loss` values from both logs — they should agree to\nwithin BF16 rounding.\n\n## Multi-GPU Examples\n\n### MLM 2-GPU with TP=2\n\n```bash\nPYTHONPATH=3rdparty\u002FMegatron-LM:$PYTHONPATH \\\nuv run python -m torch.distributed.run --nproc_per_node=2 \\\n  3rdparty\u002FMegatron-LM\u002Fpretrain_gpt.py \\\n  --tensor-model-parallel-size 2 --sequence-parallel \\\n  --num-layers 4 --hidden-size 256 --num-attention-heads 4 \\\n  --seq-length 1024 --max-position-embeddings 1024 \\\n  --micro-batch-size 2 --global-batch-size 16 \\\n  --train-iters 10 --eval-iters 2 --eval-interval 10 \\\n  --mock-data --bf16 --use-mcore-models \\\n  --tokenizer-type NullTokenizer --vocab-size 1024 \\\n  --lr 1e-4 --log-interval 1\n```\n\n### Bridge 2-GPU with TP=2\n\n```bash\nrm -rf nemo_experiments && \\\nuv run python -m torch.distributed.run --nproc_per_node=2 \\\n  scripts\u002Ftraining\u002Frun_recipe.py \\\n  --recipe vanilla_gpt_pretrain_config \\\n  model.tensor_model_parallel_size=2 model.sequence_parallel=true \\\n  model.num_layers=4 model.hidden_size=256 \\\n  model.num_attention_heads=4 model.ffn_hidden_size=1024 \\\n  model.seq_length=1024 dataset.seq_length=1024 \\\n  train.train_iters=10 train.global_batch_size=16 train.micro_batch_size=2 \\\n  validation.eval_interval=10 validation.eval_iters=2 \\\n  scheduler.lr_warmup_iters=2 scheduler.lr_decay_iters=10 \\\n  logger.log_interval=1\n```\n\n## Available Recipes\n\nCommon recipes (use with `--recipe`):\n\n- `vanilla_gpt_pretrain_config` — Minimal GPT (bare GPTModelProvider defaults,\n  ideal for correlation testing and custom configs)\n- `llama32_1b_pretrain_config` — Llama 3.2 1B (16L, 2048H, GBS=512, seq=8192)\n- `llama3_8b_pretrain_config` — Llama 3 8B\n- `qwen3_8b_pretrain_config` — Qwen3 8B\n- `deepseek_v2_lite_pretrain_config` — DeepSeek-V2-Lite 16B MoE\n\nSFT\u002FPEFT variants use `_sft_config` \u002F `_peft_config` suffix.\n\n## Megatron-Core Submodule\n\nFor what the submodule is and why two versions exist, see\n@docs\u002Fmegatron-lm-to-megatron-bridge.md.\n\n### Check current version\n\n```bash\n.\u002Fscripts\u002Fswitch_mcore.sh status\n```\n\n### Switch to dev for testing newer MCore features\n\n```bash\n.\u002Fscripts\u002Fswitch_mcore.sh dev\n\n# uv sync (without --locked) since lockfile is for main\nuv sync\n```\n\n### Switch back to main\n\n```bash\n.\u002Fscripts\u002Fswitch_mcore.sh main\n```\n\n### After pulling latest main\n\nWhen you pull the latest Bridge main branch, the submodule pointer may have\nbeen updated. Re-sync the submodule:\n\n```bash\ngit submodule update --init 3rdparty\u002FMegatron-LM\n```\n\n## Pitfalls\n\n1. **Always `rm -rf nemo_experiments`** before a fresh correlation run. Bridge\n   auto-resumes from stale checkpoints silently.\n\n2. **`uv run` required**: Always use `uv run python -m torch.distributed.run`\n   (not bare `torchrun` or `python`).\n\n3. **MLM PYTHONPATH**: Must include `3rdparty\u002FMegatron-LM` so `gpt_builders.py`\n   is importable.\n\n4. **Scheduler overrides**: When overriding `train.train_iters` to a small\n   value, also set `scheduler.lr_warmup_iters` and `scheduler.lr_decay_iters`\n   or you get an assertion error.\n\n5. **Use `dataset.seq_length`** in CLI overrides for both pretraining and fine-tuning datasets.\n\n6. **MoE OOM**: Large MoE models require full activation recomputation and\n   typically multi-node EP. TP does NOT reduce per-GPU expert memory.\n\n7. **`uv sync --locked` fails after switching to dev**: The lockfile is generated\n   against the main MCore commit. Use `uv sync` (without `--locked`) when on dev.\n",{"data":35,"body":37},{"name":4,"description":6,"license":26,"when_to_use":36},"Running training, comparing MLM vs Bridge loss curves, translating MLM CLI args to Bridge config, or investigating why loss curves diverged after a commit; 'how do I run training', 'MLM vs Bridge', 'correlation test'.",{"type":38,"children":39},"root",[40,49,55,65,72,77,144,165,171,207,214,546,552,857,863,876,882,888,1154,1160,1432,1438,1451,1508,1529,1535,1540,1546,1566,1572,1621,1627,1646,1652,1657,1692,1698,1873],{"type":41,"tag":42,"props":43,"children":45},"element","h1",{"id":44},"mlm-vs-bridge-training",[46],{"type":47,"value":48},"text","MLM vs Bridge Training",{"type":41,"tag":50,"props":51,"children":52},"p",{},[53],{"type":47,"value":54},"For how they differ, the arg mapping tables, gotchas, and translation script, see:",{"type":41,"tag":56,"props":57,"children":58},"ul",{},[59],{"type":41,"tag":60,"props":61,"children":62},"li",{},[63],{"type":47,"value":64},"@docs\u002Fmegatron-lm-to-megatron-bridge.md",{"type":41,"tag":66,"props":67,"children":69},"h2",{"id":68},"first-answer-checklist",[70],{"type":47,"value":71},"First Answer Checklist",{"type":41,"tag":50,"props":73,"children":74},{},[75],{"type":47,"value":76},"For MLM-vs-Bridge correlation questions, always name these items up front:",{"type":41,"tag":78,"props":79,"children":80},"ol",{},[81,95,107,119,131],{"type":41,"tag":60,"props":82,"children":83},{},[84,86,93],{"type":47,"value":85},"Bridge recipe: ",{"type":41,"tag":87,"props":88,"children":90},"code",{"className":89},[],[91],{"type":47,"value":92},"vanilla_gpt_pretrain_config",{"type":47,"value":94},".",{"type":41,"tag":60,"props":96,"children":97},{},[98,100,106],{"type":47,"value":99},"Bridge entry point: ",{"type":41,"tag":87,"props":101,"children":103},{"className":102},[],[104],{"type":47,"value":105},"scripts\u002Ftraining\u002Frun_recipe.py",{"type":47,"value":94},{"type":41,"tag":60,"props":108,"children":109},{},[110,112,118],{"type":47,"value":111},"MLM entry point: ",{"type":41,"tag":87,"props":113,"children":115},{"className":114},[],[116],{"type":47,"value":117},"3rdparty\u002FMegatron-LM\u002Fpretrain_gpt.py",{"type":47,"value":94},{"type":41,"tag":60,"props":120,"children":121},{},[122,124,130],{"type":47,"value":123},"Launch wrapper for both: ",{"type":41,"tag":87,"props":125,"children":127},{"className":126},[],[128],{"type":47,"value":129},"uv run python -m torch.distributed.run",{"type":47,"value":94},{"type":41,"tag":60,"props":132,"children":133},{},[134,136,142],{"type":47,"value":135},"Fresh-run cleanup: ",{"type":41,"tag":87,"props":137,"children":139},{"className":138},[],[140],{"type":47,"value":141},"rm -rf nemo_experiments",{"type":47,"value":143}," before the Bridge run.",{"type":41,"tag":50,"props":145,"children":146},{},[147,149,155,157,163],{"type":47,"value":148},"Also state that MLM needs\n",{"type":41,"tag":87,"props":150,"children":152},{"className":151},[],[153],{"type":47,"value":154},"PYTHONPATH=3rdparty\u002FMegatron-LM:$PYTHONPATH",{"type":47,"value":156},", matched Bridge and MLM losses\nshould agree within BF16 rounding, and files under ",{"type":41,"tag":87,"props":158,"children":160},{"className":159},[],[161],{"type":47,"value":162},"3rdparty\u002FMegatron-LM\u002F",{"type":47,"value":164},"\nshould not be modified from this repo.",{"type":41,"tag":66,"props":166,"children":168},{"id":167},"correlation-testing",[169],{"type":47,"value":170},"Correlation Testing",{"type":41,"tag":50,"props":172,"children":173},{},[174,176,181,183,189,191,197,199,205],{"type":47,"value":175},"Use ",{"type":41,"tag":87,"props":177,"children":179},{"className":178},[],[180],{"type":47,"value":92},{"type":47,"value":182}," for loss-correlation testing. This recipe uses\nbare ",{"type":41,"tag":87,"props":184,"children":186},{"className":185},[],[187],{"type":47,"value":188},"GPTModelProvider",{"type":47,"value":190}," defaults (LayerNorm, GeLU, learned_absolute position\nembeddings, ",{"type":41,"tag":87,"props":192,"children":194},{"className":193},[],[195],{"type":47,"value":196},"vocab_size",{"type":47,"value":198}," inherited from tokenizer) — matching MLM\n",{"type":41,"tag":87,"props":200,"children":202},{"className":201},[],[203],{"type":47,"value":204},"pretrain_gpt.py",{"type":47,"value":206}," defaults with no args.",{"type":41,"tag":208,"props":209,"children":211},"h3",{"id":210},"mlm-correlation-run-2l256h-1-gpu",[212],{"type":47,"value":213},"MLM Correlation Run (2L\u002F256H, 1 GPU)",{"type":41,"tag":215,"props":216,"children":221},"pre",{"className":217,"code":218,"language":219,"meta":220,"style":220},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","PYTHONPATH=3rdparty\u002FMegatron-LM:$PYTHONPATH \\\nuv run python -m torch.distributed.run --nproc_per_node=1 \\\n  3rdparty\u002FMegatron-LM\u002Fpretrain_gpt.py \\\n  --num-layers 2 --hidden-size 256 --num-attention-heads 4 \\\n  --ffn-hidden-size 1024 --seq-length 512 --max-position-embeddings 512 \\\n  --micro-batch-size 4 --global-batch-size 32 \\\n  --train-iters 10 --eval-iters 2 --eval-interval 10 \\\n  --mock-data --bf16 --use-mcore-models \\\n  --tokenizer-type NullTokenizer --vocab-size 32000 \\\n  --lr 3e-4 --min-lr 3e-5 --seed 1234 --log-interval 1\n","bash","",[222],{"type":41,"tag":87,"props":223,"children":224},{"__ignoreMap":220},[225,260,299,312,351,388,415,451,474,502],{"type":41,"tag":226,"props":227,"children":230},"span",{"class":228,"line":229},"line",1,[231,237,243,249,254],{"type":41,"tag":226,"props":232,"children":234},{"style":233},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[235],{"type":47,"value":236},"PYTHONPATH",{"type":41,"tag":226,"props":238,"children":240},{"style":239},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[241],{"type":47,"value":242},"=",{"type":41,"tag":226,"props":244,"children":246},{"style":245},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[247],{"type":47,"value":248},"3rdparty\u002FMegatron-LM:",{"type":41,"tag":226,"props":250,"children":251},{"style":233},[252],{"type":47,"value":253},"$PYTHONPATH ",{"type":41,"tag":226,"props":255,"children":257},{"style":256},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[258],{"type":47,"value":259},"\\\n",{"type":41,"tag":226,"props":261,"children":263},{"class":228,"line":262},2,[264,269,274,279,284,289,294],{"type":41,"tag":226,"props":265,"children":266},{"style":233},[267],{"type":47,"value":268},"uv ",{"type":41,"tag":226,"props":270,"children":271},{"style":245},[272],{"type":47,"value":273},"run",{"type":41,"tag":226,"props":275,"children":276},{"style":245},[277],{"type":47,"value":278}," python",{"type":41,"tag":226,"props":280,"children":281},{"style":245},[282],{"type":47,"value":283}," -m",{"type":41,"tag":226,"props":285,"children":286},{"style":245},[287],{"type":47,"value":288}," torch.distributed.run",{"type":41,"tag":226,"props":290,"children":291},{"style":245},[292],{"type":47,"value":293}," --nproc_per_node=1",{"type":41,"tag":226,"props":295,"children":296},{"style":233},[297],{"type":47,"value":298}," \\\n",{"type":41,"tag":226,"props":300,"children":302},{"class":228,"line":301},3,[303,308],{"type":41,"tag":226,"props":304,"children":305},{"style":245},[306],{"type":47,"value":307},"  3rdparty\u002FMegatron-LM\u002Fpretrain_gpt.py",{"type":41,"tag":226,"props":309,"children":310},{"style":233},[311],{"type":47,"value":298},{"type":41,"tag":226,"props":313,"children":315},{"class":228,"line":314},4,[316,321,327,332,337,342,347],{"type":41,"tag":226,"props":317,"children":318},{"style":245},[319],{"type":47,"value":320},"  --num-layers",{"type":41,"tag":226,"props":322,"children":324},{"style":323},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[325],{"type":47,"value":326}," 2",{"type":41,"tag":226,"props":328,"children":329},{"style":245},[330],{"type":47,"value":331}," --hidden-size",{"type":41,"tag":226,"props":333,"children":334},{"style":323},[335],{"type":47,"value":336}," 256",{"type":41,"tag":226,"props":338,"children":339},{"style":245},[340],{"type":47,"value":341}," --num-attention-heads",{"type":41,"tag":226,"props":343,"children":344},{"style":323},[345],{"type":47,"value":346}," 4",{"type":41,"tag":226,"props":348,"children":349},{"style":233},[350],{"type":47,"value":298},{"type":41,"tag":226,"props":352,"children":354},{"class":228,"line":353},5,[355,360,365,370,375,380,384],{"type":41,"tag":226,"props":356,"children":357},{"style":245},[358],{"type":47,"value":359},"  --ffn-hidden-size",{"type":41,"tag":226,"props":361,"children":362},{"style":323},[363],{"type":47,"value":364}," 1024",{"type":41,"tag":226,"props":366,"children":367},{"style":245},[368],{"type":47,"value":369}," --seq-length",{"type":41,"tag":226,"props":371,"children":372},{"style":323},[373],{"type":47,"value":374}," 512",{"type":41,"tag":226,"props":376,"children":377},{"style":245},[378],{"type":47,"value":379}," --max-position-embeddings",{"type":41,"tag":226,"props":381,"children":382},{"style":323},[383],{"type":47,"value":374},{"type":41,"tag":226,"props":385,"children":386},{"style":233},[387],{"type":47,"value":298},{"type":41,"tag":226,"props":389,"children":391},{"class":228,"line":390},6,[392,397,401,406,411],{"type":41,"tag":226,"props":393,"children":394},{"style":245},[395],{"type":47,"value":396},"  --micro-batch-size",{"type":41,"tag":226,"props":398,"children":399},{"style":323},[400],{"type":47,"value":346},{"type":41,"tag":226,"props":402,"children":403},{"style":245},[404],{"type":47,"value":405}," --global-batch-size",{"type":41,"tag":226,"props":407,"children":408},{"style":323},[409],{"type":47,"value":410}," 32",{"type":41,"tag":226,"props":412,"children":413},{"style":233},[414],{"type":47,"value":298},{"type":41,"tag":226,"props":416,"children":418},{"class":228,"line":417},7,[419,424,429,434,438,443,447],{"type":41,"tag":226,"props":420,"children":421},{"style":245},[422],{"type":47,"value":423},"  --train-iters",{"type":41,"tag":226,"props":425,"children":426},{"style":323},[427],{"type":47,"value":428}," 10",{"type":41,"tag":226,"props":430,"children":431},{"style":245},[432],{"type":47,"value":433}," --eval-iters",{"type":41,"tag":226,"props":435,"children":436},{"style":323},[437],{"type":47,"value":326},{"type":41,"tag":226,"props":439,"children":440},{"style":245},[441],{"type":47,"value":442}," --eval-interval",{"type":41,"tag":226,"props":444,"children":445},{"style":323},[446],{"type":47,"value":428},{"type":41,"tag":226,"props":448,"children":449},{"style":233},[450],{"type":47,"value":298},{"type":41,"tag":226,"props":452,"children":454},{"class":228,"line":453},8,[455,460,465,470],{"type":41,"tag":226,"props":456,"children":457},{"style":245},[458],{"type":47,"value":459},"  --mock-data",{"type":41,"tag":226,"props":461,"children":462},{"style":245},[463],{"type":47,"value":464}," --bf16",{"type":41,"tag":226,"props":466,"children":467},{"style":245},[468],{"type":47,"value":469}," --use-mcore-models",{"type":41,"tag":226,"props":471,"children":472},{"style":233},[473],{"type":47,"value":298},{"type":41,"tag":226,"props":475,"children":477},{"class":228,"line":476},9,[478,483,488,493,498],{"type":41,"tag":226,"props":479,"children":480},{"style":245},[481],{"type":47,"value":482},"  --tokenizer-type",{"type":41,"tag":226,"props":484,"children":485},{"style":245},[486],{"type":47,"value":487}," NullTokenizer",{"type":41,"tag":226,"props":489,"children":490},{"style":245},[491],{"type":47,"value":492}," --vocab-size",{"type":41,"tag":226,"props":494,"children":495},{"style":323},[496],{"type":47,"value":497}," 32000",{"type":41,"tag":226,"props":499,"children":500},{"style":233},[501],{"type":47,"value":298},{"type":41,"tag":226,"props":503,"children":505},{"class":228,"line":504},10,[506,511,516,521,526,531,536,541],{"type":41,"tag":226,"props":507,"children":508},{"style":245},[509],{"type":47,"value":510},"  --lr",{"type":41,"tag":226,"props":512,"children":513},{"style":245},[514],{"type":47,"value":515}," 3e-4",{"type":41,"tag":226,"props":517,"children":518},{"style":245},[519],{"type":47,"value":520}," --min-lr",{"type":41,"tag":226,"props":522,"children":523},{"style":245},[524],{"type":47,"value":525}," 3e-5",{"type":41,"tag":226,"props":527,"children":528},{"style":245},[529],{"type":47,"value":530}," --seed",{"type":41,"tag":226,"props":532,"children":533},{"style":323},[534],{"type":47,"value":535}," 1234",{"type":41,"tag":226,"props":537,"children":538},{"style":245},[539],{"type":47,"value":540}," --log-interval",{"type":41,"tag":226,"props":542,"children":543},{"style":323},[544],{"type":47,"value":545}," 1\n",{"type":41,"tag":208,"props":547,"children":549},{"id":548},"bridge-correlation-run-same-config-1-gpu",[550],{"type":47,"value":551},"Bridge Correlation Run (same config, 1 GPU)",{"type":41,"tag":215,"props":553,"children":555},{"className":217,"code":554,"language":219,"meta":220,"style":220},"rm -rf nemo_experiments && \\\nuv run python -m torch.distributed.run --nproc_per_node=1 \\\n  scripts\u002Ftraining\u002Frun_recipe.py \\\n  --recipe vanilla_gpt_pretrain_config \\\n  model.num_layers=2 model.hidden_size=256 \\\n  model.num_attention_heads=4 model.ffn_hidden_size=1024 \\\n  model.seq_length=512 dataset.seq_length=512 \\\n  train.train_iters=10 train.global_batch_size=32 train.micro_batch_size=4 \\\n  validation.eval_interval=10 validation.eval_iters=2 \\\n  optimizer.lr=3e-4 optimizer.min_lr=3e-5 \\\n  scheduler.lr_warmup_iters=1 scheduler.lr_decay_iters=10 \\\n  rng.seed=1234 logger.log_interval=1\n",[556],{"type":41,"tag":87,"props":557,"children":558},{"__ignoreMap":220},[559,586,619,631,648,675,702,728,764,789,806,833],{"type":41,"tag":226,"props":560,"children":561},{"class":228,"line":229},[562,567,572,577,582],{"type":41,"tag":226,"props":563,"children":564},{"style":256},[565],{"type":47,"value":566},"rm",{"type":41,"tag":226,"props":568,"children":569},{"style":245},[570],{"type":47,"value":571}," -rf",{"type":41,"tag":226,"props":573,"children":574},{"style":245},[575],{"type":47,"value":576}," nemo_experiments",{"type":41,"tag":226,"props":578,"children":579},{"style":239},[580],{"type":47,"value":581}," &&",{"type":41,"tag":226,"props":583,"children":584},{"style":233},[585],{"type":47,"value":298},{"type":41,"tag":226,"props":587,"children":588},{"class":228,"line":262},[589,594,599,603,607,611,615],{"type":41,"tag":226,"props":590,"children":591},{"style":256},[592],{"type":47,"value":593},"uv",{"type":41,"tag":226,"props":595,"children":596},{"style":245},[597],{"type":47,"value":598}," run",{"type":41,"tag":226,"props":600,"children":601},{"style":245},[602],{"type":47,"value":278},{"type":41,"tag":226,"props":604,"children":605},{"style":245},[606],{"type":47,"value":283},{"type":41,"tag":226,"props":608,"children":609},{"style":245},[610],{"type":47,"value":288},{"type":41,"tag":226,"props":612,"children":613},{"style":245},[614],{"type":47,"value":293},{"type":41,"tag":226,"props":616,"children":617},{"style":233},[618],{"type":47,"value":298},{"type":41,"tag":226,"props":620,"children":621},{"class":228,"line":301},[622,627],{"type":41,"tag":226,"props":623,"children":624},{"style":245},[625],{"type":47,"value":626},"  scripts\u002Ftraining\u002Frun_recipe.py",{"type":41,"tag":226,"props":628,"children":629},{"style":233},[630],{"type":47,"value":298},{"type":41,"tag":226,"props":632,"children":633},{"class":228,"line":314},[634,639,644],{"type":41,"tag":226,"props":635,"children":636},{"style":245},[637],{"type":47,"value":638},"  --recipe",{"type":41,"tag":226,"props":640,"children":641},{"style":245},[642],{"type":47,"value":643}," vanilla_gpt_pretrain_config",{"type":41,"tag":226,"props":645,"children":646},{"style":233},[647],{"type":47,"value":298},{"type":41,"tag":226,"props":649,"children":650},{"class":228,"line":353},[651,656,661,666,671],{"type":41,"tag":226,"props":652,"children":653},{"style":245},[654],{"type":47,"value":655},"  model.num_layers=",{"type":41,"tag":226,"props":657,"children":658},{"style":323},[659],{"type":47,"value":660},"2",{"type":41,"tag":226,"props":662,"children":663},{"style":245},[664],{"type":47,"value":665}," model.hidden_size=",{"type":41,"tag":226,"props":667,"children":668},{"style":323},[669],{"type":47,"value":670},"256",{"type":41,"tag":226,"props":672,"children":673},{"style":233},[674],{"type":47,"value":298},{"type":41,"tag":226,"props":676,"children":677},{"class":228,"line":390},[678,683,688,693,698],{"type":41,"tag":226,"props":679,"children":680},{"style":245},[681],{"type":47,"value":682},"  model.num_attention_heads=",{"type":41,"tag":226,"props":684,"children":685},{"style":323},[686],{"type":47,"value":687},"4",{"type":41,"tag":226,"props":689,"children":690},{"style":245},[691],{"type":47,"value":692}," model.ffn_hidden_size=",{"type":41,"tag":226,"props":694,"children":695},{"style":323},[696],{"type":47,"value":697},"1024",{"type":41,"tag":226,"props":699,"children":700},{"style":233},[701],{"type":47,"value":298},{"type":41,"tag":226,"props":703,"children":704},{"class":228,"line":417},[705,710,715,720,724],{"type":41,"tag":226,"props":706,"children":707},{"style":245},[708],{"type":47,"value":709},"  model.seq_length=",{"type":41,"tag":226,"props":711,"children":712},{"style":323},[713],{"type":47,"value":714},"512",{"type":41,"tag":226,"props":716,"children":717},{"style":245},[718],{"type":47,"value":719}," dataset.seq_length=",{"type":41,"tag":226,"props":721,"children":722},{"style":323},[723],{"type":47,"value":714},{"type":41,"tag":226,"props":725,"children":726},{"style":233},[727],{"type":47,"value":298},{"type":41,"tag":226,"props":729,"children":730},{"class":228,"line":453},[731,736,741,746,751,756,760],{"type":41,"tag":226,"props":732,"children":733},{"style":245},[734],{"type":47,"value":735},"  train.train_iters=",{"type":41,"tag":226,"props":737,"children":738},{"style":323},[739],{"type":47,"value":740},"10",{"type":41,"tag":226,"props":742,"children":743},{"style":245},[744],{"type":47,"value":745}," train.global_batch_size=",{"type":41,"tag":226,"props":747,"children":748},{"style":323},[749],{"type":47,"value":750},"32",{"type":41,"tag":226,"props":752,"children":753},{"style":245},[754],{"type":47,"value":755}," train.micro_batch_size=",{"type":41,"tag":226,"props":757,"children":758},{"style":323},[759],{"type":47,"value":687},{"type":41,"tag":226,"props":761,"children":762},{"style":233},[763],{"type":47,"value":298},{"type":41,"tag":226,"props":765,"children":766},{"class":228,"line":476},[767,772,776,781,785],{"type":41,"tag":226,"props":768,"children":769},{"style":245},[770],{"type":47,"value":771},"  validation.eval_interval=",{"type":41,"tag":226,"props":773,"children":774},{"style":323},[775],{"type":47,"value":740},{"type":41,"tag":226,"props":777,"children":778},{"style":245},[779],{"type":47,"value":780}," validation.eval_iters=",{"type":41,"tag":226,"props":782,"children":783},{"style":323},[784],{"type":47,"value":660},{"type":41,"tag":226,"props":786,"children":787},{"style":233},[788],{"type":47,"value":298},{"type":41,"tag":226,"props":790,"children":791},{"class":228,"line":504},[792,797,802],{"type":41,"tag":226,"props":793,"children":794},{"style":245},[795],{"type":47,"value":796},"  optimizer.lr=3e-4",{"type":41,"tag":226,"props":798,"children":799},{"style":245},[800],{"type":47,"value":801}," optimizer.min_lr=3e-5",{"type":41,"tag":226,"props":803,"children":804},{"style":233},[805],{"type":47,"value":298},{"type":41,"tag":226,"props":807,"children":809},{"class":228,"line":808},11,[810,815,820,825,829],{"type":41,"tag":226,"props":811,"children":812},{"style":245},[813],{"type":47,"value":814},"  scheduler.lr_warmup_iters=",{"type":41,"tag":226,"props":816,"children":817},{"style":323},[818],{"type":47,"value":819},"1",{"type":41,"tag":226,"props":821,"children":822},{"style":245},[823],{"type":47,"value":824}," scheduler.lr_decay_iters=",{"type":41,"tag":226,"props":826,"children":827},{"style":323},[828],{"type":47,"value":740},{"type":41,"tag":226,"props":830,"children":831},{"style":233},[832],{"type":47,"value":298},{"type":41,"tag":226,"props":834,"children":836},{"class":228,"line":835},12,[837,842,847,852],{"type":41,"tag":226,"props":838,"children":839},{"style":245},[840],{"type":47,"value":841},"  rng.seed=",{"type":41,"tag":226,"props":843,"children":844},{"style":323},[845],{"type":47,"value":846},"1234",{"type":41,"tag":226,"props":848,"children":849},{"style":245},[850],{"type":47,"value":851}," logger.log_interval=",{"type":41,"tag":226,"props":853,"children":854},{"style":323},[855],{"type":47,"value":856},"1\n",{"type":41,"tag":208,"props":858,"children":860},{"id":859},"verification",[861],{"type":47,"value":862},"Verification",{"type":41,"tag":50,"props":864,"children":865},{},[866,868,874],{"type":47,"value":867},"With matched parameters the LM losses should be nearly identical at each\niteration. Compare ",{"type":41,"tag":87,"props":869,"children":871},{"className":870},[],[872],{"type":47,"value":873},"lm loss",{"type":47,"value":875}," values from both logs — they should agree to\nwithin BF16 rounding.",{"type":41,"tag":66,"props":877,"children":879},{"id":878},"multi-gpu-examples",[880],{"type":47,"value":881},"Multi-GPU Examples",{"type":41,"tag":208,"props":883,"children":885},{"id":884},"mlm-2-gpu-with-tp2",[886],{"type":47,"value":887},"MLM 2-GPU with TP=2",{"type":41,"tag":215,"props":889,"children":891},{"className":217,"code":890,"language":219,"meta":220,"style":220},"PYTHONPATH=3rdparty\u002FMegatron-LM:$PYTHONPATH \\\nuv run python -m torch.distributed.run --nproc_per_node=2 \\\n  3rdparty\u002FMegatron-LM\u002Fpretrain_gpt.py \\\n  --tensor-model-parallel-size 2 --sequence-parallel \\\n  --num-layers 4 --hidden-size 256 --num-attention-heads 4 \\\n  --seq-length 1024 --max-position-embeddings 1024 \\\n  --micro-batch-size 2 --global-batch-size 16 \\\n  --train-iters 10 --eval-iters 2 --eval-interval 10 \\\n  --mock-data --bf16 --use-mcore-models \\\n  --tokenizer-type NullTokenizer --vocab-size 1024 \\\n  --lr 1e-4 --log-interval 1\n",[892],{"type":41,"tag":87,"props":893,"children":894},{"__ignoreMap":220},[895,918,950,961,982,1013,1037,1061,1092,1111,1134],{"type":41,"tag":226,"props":896,"children":897},{"class":228,"line":229},[898,902,906,910,914],{"type":41,"tag":226,"props":899,"children":900},{"style":233},[901],{"type":47,"value":236},{"type":41,"tag":226,"props":903,"children":904},{"style":239},[905],{"type":47,"value":242},{"type":41,"tag":226,"props":907,"children":908},{"style":245},[909],{"type":47,"value":248},{"type":41,"tag":226,"props":911,"children":912},{"style":233},[913],{"type":47,"value":253},{"type":41,"tag":226,"props":915,"children":916},{"style":256},[917],{"type":47,"value":259},{"type":41,"tag":226,"props":919,"children":920},{"class":228,"line":262},[921,925,929,933,937,941,946],{"type":41,"tag":226,"props":922,"children":923},{"style":233},[924],{"type":47,"value":268},{"type":41,"tag":226,"props":926,"children":927},{"style":245},[928],{"type":47,"value":273},{"type":41,"tag":226,"props":930,"children":931},{"style":245},[932],{"type":47,"value":278},{"type":41,"tag":226,"props":934,"children":935},{"style":245},[936],{"type":47,"value":283},{"type":41,"tag":226,"props":938,"children":939},{"style":245},[940],{"type":47,"value":288},{"type":41,"tag":226,"props":942,"children":943},{"style":245},[944],{"type":47,"value":945}," --nproc_per_node=2",{"type":41,"tag":226,"props":947,"children":948},{"style":233},[949],{"type":47,"value":298},{"type":41,"tag":226,"props":951,"children":952},{"class":228,"line":301},[953,957],{"type":41,"tag":226,"props":954,"children":955},{"style":245},[956],{"type":47,"value":307},{"type":41,"tag":226,"props":958,"children":959},{"style":233},[960],{"type":47,"value":298},{"type":41,"tag":226,"props":962,"children":963},{"class":228,"line":314},[964,969,973,978],{"type":41,"tag":226,"props":965,"children":966},{"style":245},[967],{"type":47,"value":968},"  --tensor-model-parallel-size",{"type":41,"tag":226,"props":970,"children":971},{"style":323},[972],{"type":47,"value":326},{"type":41,"tag":226,"props":974,"children":975},{"style":245},[976],{"type":47,"value":977}," --sequence-parallel",{"type":41,"tag":226,"props":979,"children":980},{"style":233},[981],{"type":47,"value":298},{"type":41,"tag":226,"props":983,"children":984},{"class":228,"line":353},[985,989,993,997,1001,1005,1009],{"type":41,"tag":226,"props":986,"children":987},{"style":245},[988],{"type":47,"value":320},{"type":41,"tag":226,"props":990,"children":991},{"style":323},[992],{"type":47,"value":346},{"type":41,"tag":226,"props":994,"children":995},{"style":245},[996],{"type":47,"value":331},{"type":41,"tag":226,"props":998,"children":999},{"style":323},[1000],{"type":47,"value":336},{"type":41,"tag":226,"props":1002,"children":1003},{"style":245},[1004],{"type":47,"value":341},{"type":41,"tag":226,"props":1006,"children":1007},{"style":323},[1008],{"type":47,"value":346},{"type":41,"tag":226,"props":1010,"children":1011},{"style":233},[1012],{"type":47,"value":298},{"type":41,"tag":226,"props":1014,"children":1015},{"class":228,"line":390},[1016,1021,1025,1029,1033],{"type":41,"tag":226,"props":1017,"children":1018},{"style":245},[1019],{"type":47,"value":1020},"  --seq-length",{"type":41,"tag":226,"props":1022,"children":1023},{"style":323},[1024],{"type":47,"value":364},{"type":41,"tag":226,"props":1026,"children":1027},{"style":245},[1028],{"type":47,"value":379},{"type":41,"tag":226,"props":1030,"children":1031},{"style":323},[1032],{"type":47,"value":364},{"type":41,"tag":226,"props":1034,"children":1035},{"style":233},[1036],{"type":47,"value":298},{"type":41,"tag":226,"props":1038,"children":1039},{"class":228,"line":417},[1040,1044,1048,1052,1057],{"type":41,"tag":226,"props":1041,"children":1042},{"style":245},[1043],{"type":47,"value":396},{"type":41,"tag":226,"props":1045,"children":1046},{"style":323},[1047],{"type":47,"value":326},{"type":41,"tag":226,"props":1049,"children":1050},{"style":245},[1051],{"type":47,"value":405},{"type":41,"tag":226,"props":1053,"children":1054},{"style":323},[1055],{"type":47,"value":1056}," 16",{"type":41,"tag":226,"props":1058,"children":1059},{"style":233},[1060],{"type":47,"value":298},{"type":41,"tag":226,"props":1062,"children":1063},{"class":228,"line":453},[1064,1068,1072,1076,1080,1084,1088],{"type":41,"tag":226,"props":1065,"children":1066},{"style":245},[1067],{"type":47,"value":423},{"type":41,"tag":226,"props":1069,"children":1070},{"style":323},[1071],{"type":47,"value":428},{"type":41,"tag":226,"props":1073,"children":1074},{"style":245},[1075],{"type":47,"value":433},{"type":41,"tag":226,"props":1077,"children":1078},{"style":323},[1079],{"type":47,"value":326},{"type":41,"tag":226,"props":1081,"children":1082},{"style":245},[1083],{"type":47,"value":442},{"type":41,"tag":226,"props":1085,"children":1086},{"style":323},[1087],{"type":47,"value":428},{"type":41,"tag":226,"props":1089,"children":1090},{"style":233},[1091],{"type":47,"value":298},{"type":41,"tag":226,"props":1093,"children":1094},{"class":228,"line":476},[1095,1099,1103,1107],{"type":41,"tag":226,"props":1096,"children":1097},{"style":245},[1098],{"type":47,"value":459},{"type":41,"tag":226,"props":1100,"children":1101},{"style":245},[1102],{"type":47,"value":464},{"type":41,"tag":226,"props":1104,"children":1105},{"style":245},[1106],{"type":47,"value":469},{"type":41,"tag":226,"props":1108,"children":1109},{"style":233},[1110],{"type":47,"value":298},{"type":41,"tag":226,"props":1112,"children":1113},{"class":228,"line":504},[1114,1118,1122,1126,1130],{"type":41,"tag":226,"props":1115,"children":1116},{"style":245},[1117],{"type":47,"value":482},{"type":41,"tag":226,"props":1119,"children":1120},{"style":245},[1121],{"type":47,"value":487},{"type":41,"tag":226,"props":1123,"children":1124},{"style":245},[1125],{"type":47,"value":492},{"type":41,"tag":226,"props":1127,"children":1128},{"style":323},[1129],{"type":47,"value":364},{"type":41,"tag":226,"props":1131,"children":1132},{"style":233},[1133],{"type":47,"value":298},{"type":41,"tag":226,"props":1135,"children":1136},{"class":228,"line":808},[1137,1141,1146,1150],{"type":41,"tag":226,"props":1138,"children":1139},{"style":245},[1140],{"type":47,"value":510},{"type":41,"tag":226,"props":1142,"children":1143},{"style":245},[1144],{"type":47,"value":1145}," 1e-4",{"type":41,"tag":226,"props":1147,"children":1148},{"style":245},[1149],{"type":47,"value":540},{"type":41,"tag":226,"props":1151,"children":1152},{"style":323},[1153],{"type":47,"value":545},{"type":41,"tag":208,"props":1155,"children":1157},{"id":1156},"bridge-2-gpu-with-tp2",[1158],{"type":47,"value":1159},"Bridge 2-GPU with TP=2",{"type":41,"tag":215,"props":1161,"children":1163},{"className":217,"code":1162,"language":219,"meta":220,"style":220},"rm -rf nemo_experiments && \\\nuv run python -m torch.distributed.run --nproc_per_node=2 \\\n  scripts\u002Ftraining\u002Frun_recipe.py \\\n  --recipe vanilla_gpt_pretrain_config \\\n  model.tensor_model_parallel_size=2 model.sequence_parallel=true \\\n  model.num_layers=4 model.hidden_size=256 \\\n  model.num_attention_heads=4 model.ffn_hidden_size=1024 \\\n  model.seq_length=1024 dataset.seq_length=1024 \\\n  train.train_iters=10 train.global_batch_size=16 train.micro_batch_size=2 \\\n  validation.eval_interval=10 validation.eval_iters=2 \\\n  scheduler.lr_warmup_iters=2 scheduler.lr_decay_iters=10 \\\n  logger.log_interval=1\n",[1164],{"type":41,"tag":87,"props":1165,"children":1166},{"__ignoreMap":220},[1167,1190,1221,1232,1247,1273,1296,1319,1342,1374,1397,1420],{"type":41,"tag":226,"props":1168,"children":1169},{"class":228,"line":229},[1170,1174,1178,1182,1186],{"type":41,"tag":226,"props":1171,"children":1172},{"style":256},[1173],{"type":47,"value":566},{"type":41,"tag":226,"props":1175,"children":1176},{"style":245},[1177],{"type":47,"value":571},{"type":41,"tag":226,"props":1179,"children":1180},{"style":245},[1181],{"type":47,"value":576},{"type":41,"tag":226,"props":1183,"children":1184},{"style":239},[1185],{"type":47,"value":581},{"type":41,"tag":226,"props":1187,"children":1188},{"style":233},[1189],{"type":47,"value":298},{"type":41,"tag":226,"props":1191,"children":1192},{"class":228,"line":262},[1193,1197,1201,1205,1209,1213,1217],{"type":41,"tag":226,"props":1194,"children":1195},{"style":256},[1196],{"type":47,"value":593},{"type":41,"tag":226,"props":1198,"children":1199},{"style":245},[1200],{"type":47,"value":598},{"type":41,"tag":226,"props":1202,"children":1203},{"style":245},[1204],{"type":47,"value":278},{"type":41,"tag":226,"props":1206,"children":1207},{"style":245},[1208],{"type":47,"value":283},{"type":41,"tag":226,"props":1210,"children":1211},{"style":245},[1212],{"type":47,"value":288},{"type":41,"tag":226,"props":1214,"children":1215},{"style":245},[1216],{"type":47,"value":945},{"type":41,"tag":226,"props":1218,"children":1219},{"style":233},[1220],{"type":47,"value":298},{"type":41,"tag":226,"props":1222,"children":1223},{"class":228,"line":301},[1224,1228],{"type":41,"tag":226,"props":1225,"children":1226},{"style":245},[1227],{"type":47,"value":626},{"type":41,"tag":226,"props":1229,"children":1230},{"style":233},[1231],{"type":47,"value":298},{"type":41,"tag":226,"props":1233,"children":1234},{"class":228,"line":314},[1235,1239,1243],{"type":41,"tag":226,"props":1236,"children":1237},{"style":245},[1238],{"type":47,"value":638},{"type":41,"tag":226,"props":1240,"children":1241},{"style":245},[1242],{"type":47,"value":643},{"type":41,"tag":226,"props":1244,"children":1245},{"style":233},[1246],{"type":47,"value":298},{"type":41,"tag":226,"props":1248,"children":1249},{"class":228,"line":353},[1250,1255,1259,1264,1269],{"type":41,"tag":226,"props":1251,"children":1252},{"style":245},[1253],{"type":47,"value":1254},"  model.tensor_model_parallel_size=",{"type":41,"tag":226,"props":1256,"children":1257},{"style":323},[1258],{"type":47,"value":660},{"type":41,"tag":226,"props":1260,"children":1261},{"style":245},[1262],{"type":47,"value":1263}," model.sequence_parallel=",{"type":41,"tag":226,"props":1265,"children":1266},{"style":239},[1267],{"type":47,"value":1268},"true",{"type":41,"tag":226,"props":1270,"children":1271},{"style":233},[1272],{"type":47,"value":298},{"type":41,"tag":226,"props":1274,"children":1275},{"class":228,"line":390},[1276,1280,1284,1288,1292],{"type":41,"tag":226,"props":1277,"children":1278},{"style":245},[1279],{"type":47,"value":655},{"type":41,"tag":226,"props":1281,"children":1282},{"style":323},[1283],{"type":47,"value":687},{"type":41,"tag":226,"props":1285,"children":1286},{"style":245},[1287],{"type":47,"value":665},{"type":41,"tag":226,"props":1289,"children":1290},{"style":323},[1291],{"type":47,"value":670},{"type":41,"tag":226,"props":1293,"children":1294},{"style":233},[1295],{"type":47,"value":298},{"type":41,"tag":226,"props":1297,"children":1298},{"class":228,"line":417},[1299,1303,1307,1311,1315],{"type":41,"tag":226,"props":1300,"children":1301},{"style":245},[1302],{"type":47,"value":682},{"type":41,"tag":226,"props":1304,"children":1305},{"style":323},[1306],{"type":47,"value":687},{"type":41,"tag":226,"props":1308,"children":1309},{"style":245},[1310],{"type":47,"value":692},{"type":41,"tag":226,"props":1312,"children":1313},{"style":323},[1314],{"type":47,"value":697},{"type":41,"tag":226,"props":1316,"children":1317},{"style":233},[1318],{"type":47,"value":298},{"type":41,"tag":226,"props":1320,"children":1321},{"class":228,"line":453},[1322,1326,1330,1334,1338],{"type":41,"tag":226,"props":1323,"children":1324},{"style":245},[1325],{"type":47,"value":709},{"type":41,"tag":226,"props":1327,"children":1328},{"style":323},[1329],{"type":47,"value":697},{"type":41,"tag":226,"props":1331,"children":1332},{"style":245},[1333],{"type":47,"value":719},{"type":41,"tag":226,"props":1335,"children":1336},{"style":323},[1337],{"type":47,"value":697},{"type":41,"tag":226,"props":1339,"children":1340},{"style":233},[1341],{"type":47,"value":298},{"type":41,"tag":226,"props":1343,"children":1344},{"class":228,"line":476},[1345,1349,1353,1357,1362,1366,1370],{"type":41,"tag":226,"props":1346,"children":1347},{"style":245},[1348],{"type":47,"value":735},{"type":41,"tag":226,"props":1350,"children":1351},{"style":323},[1352],{"type":47,"value":740},{"type":41,"tag":226,"props":1354,"children":1355},{"style":245},[1356],{"type":47,"value":745},{"type":41,"tag":226,"props":1358,"children":1359},{"style":323},[1360],{"type":47,"value":1361},"16",{"type":41,"tag":226,"props":1363,"children":1364},{"style":245},[1365],{"type":47,"value":755},{"type":41,"tag":226,"props":1367,"children":1368},{"style":323},[1369],{"type":47,"value":660},{"type":41,"tag":226,"props":1371,"children":1372},{"style":233},[1373],{"type":47,"value":298},{"type":41,"tag":226,"props":1375,"children":1376},{"class":228,"line":504},[1377,1381,1385,1389,1393],{"type":41,"tag":226,"props":1378,"children":1379},{"style":245},[1380],{"type":47,"value":771},{"type":41,"tag":226,"props":1382,"children":1383},{"style":323},[1384],{"type":47,"value":740},{"type":41,"tag":226,"props":1386,"children":1387},{"style":245},[1388],{"type":47,"value":780},{"type":41,"tag":226,"props":1390,"children":1391},{"style":323},[1392],{"type":47,"value":660},{"type":41,"tag":226,"props":1394,"children":1395},{"style":233},[1396],{"type":47,"value":298},{"type":41,"tag":226,"props":1398,"children":1399},{"class":228,"line":808},[1400,1404,1408,1412,1416],{"type":41,"tag":226,"props":1401,"children":1402},{"style":245},[1403],{"type":47,"value":814},{"type":41,"tag":226,"props":1405,"children":1406},{"style":323},[1407],{"type":47,"value":660},{"type":41,"tag":226,"props":1409,"children":1410},{"style":245},[1411],{"type":47,"value":824},{"type":41,"tag":226,"props":1413,"children":1414},{"style":323},[1415],{"type":47,"value":740},{"type":41,"tag":226,"props":1417,"children":1418},{"style":233},[1419],{"type":47,"value":298},{"type":41,"tag":226,"props":1421,"children":1422},{"class":228,"line":835},[1423,1428],{"type":41,"tag":226,"props":1424,"children":1425},{"style":245},[1426],{"type":47,"value":1427},"  logger.log_interval=",{"type":41,"tag":226,"props":1429,"children":1430},{"style":323},[1431],{"type":47,"value":856},{"type":41,"tag":66,"props":1433,"children":1435},{"id":1434},"available-recipes",[1436],{"type":47,"value":1437},"Available Recipes",{"type":41,"tag":50,"props":1439,"children":1440},{},[1441,1443,1449],{"type":47,"value":1442},"Common recipes (use with ",{"type":41,"tag":87,"props":1444,"children":1446},{"className":1445},[],[1447],{"type":47,"value":1448},"--recipe",{"type":47,"value":1450},"):",{"type":41,"tag":56,"props":1452,"children":1453},{},[1454,1464,1475,1486,1497],{"type":41,"tag":60,"props":1455,"children":1456},{},[1457,1462],{"type":41,"tag":87,"props":1458,"children":1460},{"className":1459},[],[1461],{"type":47,"value":92},{"type":47,"value":1463}," — Minimal GPT (bare GPTModelProvider defaults,\nideal for correlation testing and custom configs)",{"type":41,"tag":60,"props":1465,"children":1466},{},[1467,1473],{"type":41,"tag":87,"props":1468,"children":1470},{"className":1469},[],[1471],{"type":47,"value":1472},"llama32_1b_pretrain_config",{"type":47,"value":1474}," — Llama 3.2 1B (16L, 2048H, GBS=512, seq=8192)",{"type":41,"tag":60,"props":1476,"children":1477},{},[1478,1484],{"type":41,"tag":87,"props":1479,"children":1481},{"className":1480},[],[1482],{"type":47,"value":1483},"llama3_8b_pretrain_config",{"type":47,"value":1485}," — Llama 3 8B",{"type":41,"tag":60,"props":1487,"children":1488},{},[1489,1495],{"type":41,"tag":87,"props":1490,"children":1492},{"className":1491},[],[1493],{"type":47,"value":1494},"qwen3_8b_pretrain_config",{"type":47,"value":1496}," — Qwen3 8B",{"type":41,"tag":60,"props":1498,"children":1499},{},[1500,1506],{"type":41,"tag":87,"props":1501,"children":1503},{"className":1502},[],[1504],{"type":47,"value":1505},"deepseek_v2_lite_pretrain_config",{"type":47,"value":1507}," — DeepSeek-V2-Lite 16B MoE",{"type":41,"tag":50,"props":1509,"children":1510},{},[1511,1513,1519,1521,1527],{"type":47,"value":1512},"SFT\u002FPEFT variants use ",{"type":41,"tag":87,"props":1514,"children":1516},{"className":1515},[],[1517],{"type":47,"value":1518},"_sft_config",{"type":47,"value":1520}," \u002F ",{"type":41,"tag":87,"props":1522,"children":1524},{"className":1523},[],[1525],{"type":47,"value":1526},"_peft_config",{"type":47,"value":1528}," suffix.",{"type":41,"tag":66,"props":1530,"children":1532},{"id":1531},"megatron-core-submodule",[1533],{"type":47,"value":1534},"Megatron-Core Submodule",{"type":41,"tag":50,"props":1536,"children":1537},{},[1538],{"type":47,"value":1539},"For what the submodule is and why two versions exist, see\n@docs\u002Fmegatron-lm-to-megatron-bridge.md.",{"type":41,"tag":208,"props":1541,"children":1543},{"id":1542},"check-current-version",[1544],{"type":47,"value":1545},"Check current version",{"type":41,"tag":215,"props":1547,"children":1549},{"className":217,"code":1548,"language":219,"meta":220,"style":220},".\u002Fscripts\u002Fswitch_mcore.sh status\n",[1550],{"type":41,"tag":87,"props":1551,"children":1552},{"__ignoreMap":220},[1553],{"type":41,"tag":226,"props":1554,"children":1555},{"class":228,"line":229},[1556,1561],{"type":41,"tag":226,"props":1557,"children":1558},{"style":256},[1559],{"type":47,"value":1560},".\u002Fscripts\u002Fswitch_mcore.sh",{"type":41,"tag":226,"props":1562,"children":1563},{"style":245},[1564],{"type":47,"value":1565}," status\n",{"type":41,"tag":208,"props":1567,"children":1569},{"id":1568},"switch-to-dev-for-testing-newer-mcore-features",[1570],{"type":47,"value":1571},"Switch to dev for testing newer MCore features",{"type":41,"tag":215,"props":1573,"children":1575},{"className":217,"code":1574,"language":219,"meta":220,"style":220},".\u002Fscripts\u002Fswitch_mcore.sh dev\n\n# uv sync (without --locked) since lockfile is for main\nuv sync\n",[1576],{"type":41,"tag":87,"props":1577,"children":1578},{"__ignoreMap":220},[1579,1591,1600,1609],{"type":41,"tag":226,"props":1580,"children":1581},{"class":228,"line":229},[1582,1586],{"type":41,"tag":226,"props":1583,"children":1584},{"style":256},[1585],{"type":47,"value":1560},{"type":41,"tag":226,"props":1587,"children":1588},{"style":245},[1589],{"type":47,"value":1590}," dev\n",{"type":41,"tag":226,"props":1592,"children":1593},{"class":228,"line":262},[1594],{"type":41,"tag":226,"props":1595,"children":1597},{"emptyLinePlaceholder":1596},true,[1598],{"type":47,"value":1599},"\n",{"type":41,"tag":226,"props":1601,"children":1602},{"class":228,"line":301},[1603],{"type":41,"tag":226,"props":1604,"children":1606},{"style":1605},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1607],{"type":47,"value":1608},"# uv sync (without --locked) since lockfile is for main\n",{"type":41,"tag":226,"props":1610,"children":1611},{"class":228,"line":314},[1612,1616],{"type":41,"tag":226,"props":1613,"children":1614},{"style":256},[1615],{"type":47,"value":593},{"type":41,"tag":226,"props":1617,"children":1618},{"style":245},[1619],{"type":47,"value":1620}," sync\n",{"type":41,"tag":208,"props":1622,"children":1624},{"id":1623},"switch-back-to-main",[1625],{"type":47,"value":1626},"Switch back to main",{"type":41,"tag":215,"props":1628,"children":1630},{"className":217,"code":1629,"language":219,"meta":220,"style":220},".\u002Fscripts\u002Fswitch_mcore.sh main\n",[1631],{"type":41,"tag":87,"props":1632,"children":1633},{"__ignoreMap":220},[1634],{"type":41,"tag":226,"props":1635,"children":1636},{"class":228,"line":229},[1637,1641],{"type":41,"tag":226,"props":1638,"children":1639},{"style":256},[1640],{"type":47,"value":1560},{"type":41,"tag":226,"props":1642,"children":1643},{"style":245},[1644],{"type":47,"value":1645}," main\n",{"type":41,"tag":208,"props":1647,"children":1649},{"id":1648},"after-pulling-latest-main",[1650],{"type":47,"value":1651},"After pulling latest main",{"type":41,"tag":50,"props":1653,"children":1654},{},[1655],{"type":47,"value":1656},"When you pull the latest Bridge main branch, the submodule pointer may have\nbeen updated. Re-sync the submodule:",{"type":41,"tag":215,"props":1658,"children":1660},{"className":217,"code":1659,"language":219,"meta":220,"style":220},"git submodule update --init 3rdparty\u002FMegatron-LM\n",[1661],{"type":41,"tag":87,"props":1662,"children":1663},{"__ignoreMap":220},[1664],{"type":41,"tag":226,"props":1665,"children":1666},{"class":228,"line":229},[1667,1672,1677,1682,1687],{"type":41,"tag":226,"props":1668,"children":1669},{"style":256},[1670],{"type":47,"value":1671},"git",{"type":41,"tag":226,"props":1673,"children":1674},{"style":245},[1675],{"type":47,"value":1676}," submodule",{"type":41,"tag":226,"props":1678,"children":1679},{"style":245},[1680],{"type":47,"value":1681}," update",{"type":41,"tag":226,"props":1683,"children":1684},{"style":245},[1685],{"type":47,"value":1686}," --init",{"type":41,"tag":226,"props":1688,"children":1689},{"style":245},[1690],{"type":47,"value":1691}," 3rdparty\u002FMegatron-LM\n",{"type":41,"tag":66,"props":1693,"children":1695},{"id":1694},"pitfalls",[1696],{"type":47,"value":1697},"Pitfalls",{"type":41,"tag":78,"props":1699,"children":1700},{},[1701,1717,1756,1782,1816,1831,1841],{"type":41,"tag":60,"props":1702,"children":1703},{},[1704,1715],{"type":41,"tag":1705,"props":1706,"children":1707},"strong",{},[1708,1710],{"type":47,"value":1709},"Always ",{"type":41,"tag":87,"props":1711,"children":1713},{"className":1712},[],[1714],{"type":47,"value":141},{"type":47,"value":1716}," before a fresh correlation run. Bridge\nauto-resumes from stale checkpoints silently.",{"type":41,"tag":60,"props":1718,"children":1719},{},[1720,1731,1733,1738,1740,1746,1748,1754],{"type":41,"tag":1705,"props":1721,"children":1722},{},[1723,1729],{"type":41,"tag":87,"props":1724,"children":1726},{"className":1725},[],[1727],{"type":47,"value":1728},"uv run",{"type":47,"value":1730}," required",{"type":47,"value":1732},": Always use ",{"type":41,"tag":87,"props":1734,"children":1736},{"className":1735},[],[1737],{"type":47,"value":129},{"type":47,"value":1739},"\n(not bare ",{"type":41,"tag":87,"props":1741,"children":1743},{"className":1742},[],[1744],{"type":47,"value":1745},"torchrun",{"type":47,"value":1747}," or ",{"type":41,"tag":87,"props":1749,"children":1751},{"className":1750},[],[1752],{"type":47,"value":1753},"python",{"type":47,"value":1755},").",{"type":41,"tag":60,"props":1757,"children":1758},{},[1759,1764,1766,1772,1774,1780],{"type":41,"tag":1705,"props":1760,"children":1761},{},[1762],{"type":47,"value":1763},"MLM PYTHONPATH",{"type":47,"value":1765},": Must include ",{"type":41,"tag":87,"props":1767,"children":1769},{"className":1768},[],[1770],{"type":47,"value":1771},"3rdparty\u002FMegatron-LM",{"type":47,"value":1773}," so ",{"type":41,"tag":87,"props":1775,"children":1777},{"className":1776},[],[1778],{"type":47,"value":1779},"gpt_builders.py",{"type":47,"value":1781},"\nis importable.",{"type":41,"tag":60,"props":1783,"children":1784},{},[1785,1790,1792,1798,1800,1806,1808,1814],{"type":41,"tag":1705,"props":1786,"children":1787},{},[1788],{"type":47,"value":1789},"Scheduler overrides",{"type":47,"value":1791},": When overriding ",{"type":41,"tag":87,"props":1793,"children":1795},{"className":1794},[],[1796],{"type":47,"value":1797},"train.train_iters",{"type":47,"value":1799}," to a small\nvalue, also set ",{"type":41,"tag":87,"props":1801,"children":1803},{"className":1802},[],[1804],{"type":47,"value":1805},"scheduler.lr_warmup_iters",{"type":47,"value":1807}," and ",{"type":41,"tag":87,"props":1809,"children":1811},{"className":1810},[],[1812],{"type":47,"value":1813},"scheduler.lr_decay_iters",{"type":47,"value":1815},"\nor you get an assertion error.",{"type":41,"tag":60,"props":1817,"children":1818},{},[1819,1829],{"type":41,"tag":1705,"props":1820,"children":1821},{},[1822,1823],{"type":47,"value":175},{"type":41,"tag":87,"props":1824,"children":1826},{"className":1825},[],[1827],{"type":47,"value":1828},"dataset.seq_length",{"type":47,"value":1830}," in CLI overrides for both pretraining and fine-tuning datasets.",{"type":41,"tag":60,"props":1832,"children":1833},{},[1834,1839],{"type":41,"tag":1705,"props":1835,"children":1836},{},[1837],{"type":47,"value":1838},"MoE OOM",{"type":47,"value":1840},": Large MoE models require full activation recomputation and\ntypically multi-node EP. TP does NOT reduce per-GPU expert memory.",{"type":41,"tag":60,"props":1842,"children":1843},{},[1844,1855,1857,1863,1865,1871],{"type":41,"tag":1705,"props":1845,"children":1846},{},[1847,1853],{"type":41,"tag":87,"props":1848,"children":1850},{"className":1849},[],[1851],{"type":47,"value":1852},"uv sync --locked",{"type":47,"value":1854}," fails after switching to dev",{"type":47,"value":1856},": The lockfile is generated\nagainst the main MCore commit. Use ",{"type":41,"tag":87,"props":1858,"children":1860},{"className":1859},[],[1861],{"type":47,"value":1862},"uv sync",{"type":47,"value":1864}," (without ",{"type":41,"tag":87,"props":1866,"children":1868},{"className":1867},[],[1869],{"type":47,"value":1870},"--locked",{"type":47,"value":1872},") when on dev.",{"type":41,"tag":1874,"props":1875,"children":1876},"style",{},[1877],{"type":47,"value":1878},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1880,"total":1984},[1881,1898,1912,1926,1938,1955,1970],{"slug":1882,"name":1882,"fn":1883,"description":1884,"org":1885,"tags":1886,"stars":23,"repoUrl":24,"updatedAt":1897},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1887,1890,1893,1894],{"name":1888,"slug":1889,"type":15},"Data Analysis","data-analysis",{"name":1891,"slug":1892,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":1895,"slug":1896,"type":15},"Performance","performance","2026-07-14T05:28:43.176466",{"slug":1899,"name":1899,"fn":1900,"description":1901,"org":1902,"tags":1903,"stars":23,"repoUrl":24,"updatedAt":1911},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1904,1907,1910],{"name":1905,"slug":1906,"type":15},"Deployment","deployment",{"name":1908,"slug":1909,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":1913,"name":1913,"fn":1914,"description":1915,"org":1916,"tags":1917,"stars":23,"repoUrl":24,"updatedAt":1925},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1918,1921,1922],{"name":1919,"slug":1920,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":1923,"slug":1924,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":1927,"name":1927,"fn":1928,"description":1929,"org":1930,"tags":1931,"stars":23,"repoUrl":24,"updatedAt":1937},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1932,1933,1934],{"name":1888,"slug":1889,"type":15},{"name":9,"slug":8,"type":15},{"name":1935,"slug":1936,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":1939,"name":1939,"fn":1940,"description":1941,"org":1942,"tags":1943,"stars":23,"repoUrl":24,"updatedAt":1954},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1944,1947,1950,1951],{"name":1945,"slug":1946,"type":15},"Automation","automation",{"name":1948,"slug":1949,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":1952,"slug":1953,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":1956,"name":1956,"fn":1957,"description":1958,"org":1959,"tags":1960,"stars":23,"repoUrl":24,"updatedAt":1969},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1961,1962,1965,1966],{"name":1905,"slug":1906,"type":15},{"name":1963,"slug":1964,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":1967,"slug":1968,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":1971,"name":1971,"fn":1972,"description":1973,"org":1974,"tags":1975,"stars":23,"repoUrl":24,"updatedAt":1983},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1976,1977,1980],{"name":9,"slug":8,"type":15},{"name":1978,"slug":1979,"type":15},"Quantum Computing","quantum-computing",{"name":1981,"slug":1982,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":1986,"total":2134},[1987,2005,2020,2031,2043,2057,2070,2082,2093,2102,2116,2125],{"slug":1988,"name":1988,"fn":1989,"description":1990,"org":1991,"tags":1992,"stars":2002,"repoUrl":2003,"updatedAt":2004},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1993,1996,1999],{"name":1994,"slug":1995,"type":15},"Documentation","documentation",{"name":1997,"slug":1998,"type":15},"MCP","mcp",{"name":2000,"slug":2001,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2006,"name":2006,"fn":2007,"description":2008,"org":2009,"tags":2010,"stars":2017,"repoUrl":2018,"updatedAt":2019},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2011,2014,2015],{"name":2012,"slug":2013,"type":15},"Containers","containers",{"name":1905,"slug":1906,"type":15},{"name":2016,"slug":1753,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2021,"name":2021,"fn":2022,"description":2023,"org":2024,"tags":2025,"stars":2017,"repoUrl":2018,"updatedAt":2030},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2026,2029],{"name":2027,"slug":2028,"type":15},"CI\u002FCD","ci-cd",{"name":1905,"slug":1906,"type":15},"2026-07-14T05:25:59.97109",{"slug":2032,"name":2032,"fn":2033,"description":2034,"org":2035,"tags":2036,"stars":2017,"repoUrl":2018,"updatedAt":2042},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2037,2038,2039],{"name":2027,"slug":2028,"type":15},{"name":1905,"slug":1906,"type":15},{"name":2040,"slug":2041,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2044,"name":2044,"fn":2045,"description":2046,"org":2047,"tags":2048,"stars":2017,"repoUrl":2018,"updatedAt":2056},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2049,2052,2053],{"name":2050,"slug":2051,"type":15},"Debugging","debugging",{"name":2040,"slug":2041,"type":15},{"name":2054,"slug":2055,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2058,"name":2058,"fn":2059,"description":2060,"org":2061,"tags":2062,"stars":2017,"repoUrl":2018,"updatedAt":2069},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2063,2066],{"name":2064,"slug":2065,"type":15},"Best Practices","best-practices",{"name":2067,"slug":2068,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2071,"name":2071,"fn":2072,"description":2073,"org":2074,"tags":2075,"stars":2017,"repoUrl":2018,"updatedAt":2081},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2076,2077,2080],{"name":20,"slug":21,"type":15},{"name":2078,"slug":2079,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2083,"name":2083,"fn":2084,"description":2085,"org":2086,"tags":2087,"stars":2017,"repoUrl":2018,"updatedAt":2092},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2088,2091],{"name":2089,"slug":2090,"type":15},"QA","qa",{"name":1935,"slug":1936,"type":15},"2026-07-14T05:25:53.673039",{"slug":2094,"name":2094,"fn":2095,"description":2096,"org":2097,"tags":2098,"stars":2017,"repoUrl":2018,"updatedAt":2101},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2099,2100],{"name":1905,"slug":1906,"type":15},{"name":1908,"slug":1909,"type":15},"2026-07-14T05:25:49.362534",{"slug":2103,"name":2103,"fn":2104,"description":2105,"org":2106,"tags":2107,"stars":2017,"repoUrl":2018,"updatedAt":2115},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2108,2111,2112],{"name":2109,"slug":2110,"type":15},"Code Review","code-review",{"name":2040,"slug":2041,"type":15},{"name":2113,"slug":2114,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2117,"name":2117,"fn":2118,"description":2119,"org":2120,"tags":2121,"stars":2017,"repoUrl":2018,"updatedAt":2124},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2122,2123],{"name":2089,"slug":2090,"type":15},{"name":1935,"slug":1936,"type":15},"2026-07-14T05:25:54.928983",{"slug":2126,"name":2126,"fn":2127,"description":2128,"org":2129,"tags":2130,"stars":2017,"repoUrl":2018,"updatedAt":2133},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2131,2132],{"name":1945,"slug":1946,"type":15},{"name":2027,"slug":2028,"type":15},"2026-07-30T05:29:03.275638",496]