[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nemo-mbridge-perf-moe-vlm-training":3,"mdc--eqmlg4-key":34,"related-org-nvidia-nemo-mbridge-perf-moe-vlm-training":571,"related-repo-nvidia-nemo-mbridge-perf-moe-vlm-training":732},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"nemo-mbridge-perf-moe-vlm-training","train MoE multimodal models with Megatron Bridge","Practical guidance for training MoE VLMs in Megatron Bridge. Compares FSDP and 3D-parallel approaches, using rounded lessons from Qwen3-VL, Qwen3-Next, and other multimodal experiments.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":17,"slug":18,"type":15},"Deep Learning","deep-learning",{"name":20,"slug":21,"type":15},"Multimodal","multimodal",{"name":9,"slug":8,"type":15},2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:27:55.624362","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fnemo-mbridge-perf-moe-vlm-training","---\nname: nemo-mbridge-perf-moe-vlm-training\ndescription: Practical guidance for training MoE VLMs in Megatron Bridge. Compares FSDP and 3D-parallel approaches, using rounded lessons from Qwen3-VL, Qwen3-Next, and other multimodal experiments.\nlicense: Apache-2.0\nwhen_to_use: Training MoE VLMs, or investigating a commit that caused MoE VLM training failure or OOM; 'MoE VLM', 'multimodal MoE', 'Qwen3-VL training', 'FSDP vs 3D-parallel for VLM', 'MoE vision language model'.\n---\n\n# MoE VLM Training\n\nStable docs: @docs\u002Ftraining\u002Fmoe-optimization.md\nCard: @skills\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fcard.yaml\n\n## FSDP vs 3D Parallel\n\n| Approach | Strength | Best fit |\n|---|---|---|\n| FSDP | Simplest path to a working multimodal run | first bring-up, memory-first tuning, awkward PP boundaries |\n| 3D parallel | Higher ceiling after tuning | stable models with a clean PP layout and time for deeper sweeps |\n\nFor MoE VLMs, the practical workflow is usually:\n\n1. get the first reliable run with FSDP\n2. stabilize real-data input, recompute, and memory behavior\n3. move to 3D parallel only if the throughput headroom is worth the extra work\n\n## Rounded Findings From Recent VLM Runs\n\n### Qwen3-VL class models\n\nThe main patterns were consistent across the tracker:\n\n- FSDP on GB200-class systems can already reach healthy high-teens utilization\n  with a comparatively simple setup\n- B200 FSDP runs are viable, but more sensitive to recompute choice and frozen\n  vision settings\n- 3D parallel can recover to a similar or better operating point, but only after\n  tuning MBS, recompute, and the real vision path together\n\n### Real data vs mock data\n\nMock-data VLM runs are not trustworthy performance proxies. In the experiments,\nimage-free mock runs looked closer to \"roughly twice as fast\" than \"slightly\noptimistic\" when compared with real multimodal input.\n\nUse real or realistic image payloads before drawing any conclusion about VLM\nthroughput.\n\n### Smaller multimodal MoE runs\n\nThe smaller Qwen3.5-style multimodal experiments reinforce the same lessons:\n\n- HybridEP is a solid default on GB200\n- TE-scoped CUDA graphs help once the training loop is stable\n- larger MBS can pay off, but only if the vision encoder does not become the\n  next bottleneck\n\n## Decision Guide\n\n### Choose FSDP when\n\n- you are bringing up a new VLM for the first time\n- the model has awkward stage boundaries across embedding, vision, and decoder\n- memory fit matters more than absolute throughput\n- you may freeze the vision stack during decoder-focused tuning\n\n### Choose 3D parallel when\n\n- the model is already stable under FSDP\n- the PP layout is clear and repeatable\n- you can sweep MBS, recompute, and CUDA-graph scope together\n- the goal is best steady-state throughput, not easiest bring-up\n\n## Key Tuning Knobs\n\n1. **Freeze the vision stack when appropriate**: if the work is decoder-focused,\n   freezing the vision side often gives a small but real throughput gain and\n   reduces memory pressure.\n\n2. **Sweep MBS aggressively**: VLMs are more MBS-sensitive than text-only MoE\n   runs because the vision path changes the compute-to-overhead balance.\n\n3. **Prefer selective recompute once the model fits**: full recompute is a\n   useful bring-up tool, but selective recompute is usually the better steady\n   state.\n\n4. **Match CUDA-graph scope to the workload**: `attn moe_router moe_preprocess`\n   is the safer MoE default, while narrower scopes can still be useful for\n   controlled experiments.\n\n5. **Use ETP only when EP alone is insufficient**: it can unlock a layout, but\n   it also introduces more communication and more tuning surface.\n\n## Representative Config Families\n\n### FSDP-first GB200 path\n\n```text\nTP=1  CP=1  PP=1\nEP sized to the expert topology, often large\nDispatcher: HybridEP on GB200-class systems\nRecompute: start with full, then relax toward selective recompute\n```\n\n### 3D-parallel GB200 path\n\n```text\nTP=1  CP=1  PP=1 or modest PP\nEP and ETP sized to the expert topology\nDispatcher: HybridEP\nCUDA Graph: start narrow, then widen only after the real-data path is stable\n```\n\n## Compatibility\n\n| Feature | FSDP | 3D parallel |\n|---|---|---|\n| HybridEP on GB200 | strong default | strong default once topology is stable |\n| CUDA graphs | useful after bring-up | useful, but more scope-sensitive |\n| Freeze vision | natural fit | possible, but less often used as the headline perf path |\n| Selective recompute | recommended | recommended |\n\n## Pitfalls\n\n1. **Mock multimodal data is misleading**: it can make the decoder look much\n   healthier than the real end-to-end VLM path.\n\n2. **The vision encoder can dominate unexpectedly**: profile encoder, projector,\n   and decoder separately before attributing everything to the dispatcher.\n\n3. **Do not compare FSDP and 3D-parallel runs with different effective work**:\n   normalize by useful tokens and workload shape, not only by step time.\n\n4. **ETP is not free**: use it as a fit or topology tool, not as the default.\n\n5. **Recompute and CUDA-graph choices are coupled**: the setting that gets the\n   model to fit is often not the setting that gives the best steady-state speed.\n",{"data":35,"body":37},{"name":4,"description":6,"license":26,"when_to_use":36},"Training MoE VLMs, or investigating a commit that caused MoE VLM training failure or OOM; 'MoE VLM', 'multimodal MoE', 'Qwen3-VL training', 'FSDP vs 3D-parallel for VLM', 'MoE vision language model'.",{"type":38,"children":39},"root",[40,49,55,62,131,136,156,162,169,174,193,199,204,209,215,220,238,244,250,273,279,302,308,371,377,383,395,401,410,416,512,518],{"type":41,"tag":42,"props":43,"children":45},"element","h1",{"id":44},"moe-vlm-training",[46],{"type":47,"value":48},"text","MoE VLM Training",{"type":41,"tag":50,"props":51,"children":52},"p",{},[53],{"type":47,"value":54},"Stable docs: @docs\u002Ftraining\u002Fmoe-optimization.md\nCard: @skills\u002Fnemo-mbridge-perf-moe-vlm-training\u002Fcard.yaml",{"type":41,"tag":56,"props":57,"children":59},"h2",{"id":58},"fsdp-vs-3d-parallel",[60],{"type":47,"value":61},"FSDP vs 3D Parallel",{"type":41,"tag":63,"props":64,"children":65},"table",{},[66,90],{"type":41,"tag":67,"props":68,"children":69},"thead",{},[70],{"type":41,"tag":71,"props":72,"children":73},"tr",{},[74,80,85],{"type":41,"tag":75,"props":76,"children":77},"th",{},[78],{"type":47,"value":79},"Approach",{"type":41,"tag":75,"props":81,"children":82},{},[83],{"type":47,"value":84},"Strength",{"type":41,"tag":75,"props":86,"children":87},{},[88],{"type":47,"value":89},"Best fit",{"type":41,"tag":91,"props":92,"children":93},"tbody",{},[94,113],{"type":41,"tag":71,"props":95,"children":96},{},[97,103,108],{"type":41,"tag":98,"props":99,"children":100},"td",{},[101],{"type":47,"value":102},"FSDP",{"type":41,"tag":98,"props":104,"children":105},{},[106],{"type":47,"value":107},"Simplest path to a working multimodal run",{"type":41,"tag":98,"props":109,"children":110},{},[111],{"type":47,"value":112},"first bring-up, memory-first tuning, awkward PP boundaries",{"type":41,"tag":71,"props":114,"children":115},{},[116,121,126],{"type":41,"tag":98,"props":117,"children":118},{},[119],{"type":47,"value":120},"3D parallel",{"type":41,"tag":98,"props":122,"children":123},{},[124],{"type":47,"value":125},"Higher ceiling after tuning",{"type":41,"tag":98,"props":127,"children":128},{},[129],{"type":47,"value":130},"stable models with a clean PP layout and time for deeper sweeps",{"type":41,"tag":50,"props":132,"children":133},{},[134],{"type":47,"value":135},"For MoE VLMs, the practical workflow is usually:",{"type":41,"tag":137,"props":138,"children":139},"ol",{},[140,146,151],{"type":41,"tag":141,"props":142,"children":143},"li",{},[144],{"type":47,"value":145},"get the first reliable run with FSDP",{"type":41,"tag":141,"props":147,"children":148},{},[149],{"type":47,"value":150},"stabilize real-data input, recompute, and memory behavior",{"type":41,"tag":141,"props":152,"children":153},{},[154],{"type":47,"value":155},"move to 3D parallel only if the throughput headroom is worth the extra work",{"type":41,"tag":56,"props":157,"children":159},{"id":158},"rounded-findings-from-recent-vlm-runs",[160],{"type":47,"value":161},"Rounded Findings From Recent VLM Runs",{"type":41,"tag":163,"props":164,"children":166},"h3",{"id":165},"qwen3-vl-class-models",[167],{"type":47,"value":168},"Qwen3-VL class models",{"type":41,"tag":50,"props":170,"children":171},{},[172],{"type":47,"value":173},"The main patterns were consistent across the tracker:",{"type":41,"tag":175,"props":176,"children":177},"ul",{},[178,183,188],{"type":41,"tag":141,"props":179,"children":180},{},[181],{"type":47,"value":182},"FSDP on GB200-class systems can already reach healthy high-teens utilization\nwith a comparatively simple setup",{"type":41,"tag":141,"props":184,"children":185},{},[186],{"type":47,"value":187},"B200 FSDP runs are viable, but more sensitive to recompute choice and frozen\nvision settings",{"type":41,"tag":141,"props":189,"children":190},{},[191],{"type":47,"value":192},"3D parallel can recover to a similar or better operating point, but only after\ntuning MBS, recompute, and the real vision path together",{"type":41,"tag":163,"props":194,"children":196},{"id":195},"real-data-vs-mock-data",[197],{"type":47,"value":198},"Real data vs mock data",{"type":41,"tag":50,"props":200,"children":201},{},[202],{"type":47,"value":203},"Mock-data VLM runs are not trustworthy performance proxies. In the experiments,\nimage-free mock runs looked closer to \"roughly twice as fast\" than \"slightly\noptimistic\" when compared with real multimodal input.",{"type":41,"tag":50,"props":205,"children":206},{},[207],{"type":47,"value":208},"Use real or realistic image payloads before drawing any conclusion about VLM\nthroughput.",{"type":41,"tag":163,"props":210,"children":212},{"id":211},"smaller-multimodal-moe-runs",[213],{"type":47,"value":214},"Smaller multimodal MoE runs",{"type":41,"tag":50,"props":216,"children":217},{},[218],{"type":47,"value":219},"The smaller Qwen3.5-style multimodal experiments reinforce the same lessons:",{"type":41,"tag":175,"props":221,"children":222},{},[223,228,233],{"type":41,"tag":141,"props":224,"children":225},{},[226],{"type":47,"value":227},"HybridEP is a solid default on GB200",{"type":41,"tag":141,"props":229,"children":230},{},[231],{"type":47,"value":232},"TE-scoped CUDA graphs help once the training loop is stable",{"type":41,"tag":141,"props":234,"children":235},{},[236],{"type":47,"value":237},"larger MBS can pay off, but only if the vision encoder does not become the\nnext bottleneck",{"type":41,"tag":56,"props":239,"children":241},{"id":240},"decision-guide",[242],{"type":47,"value":243},"Decision Guide",{"type":41,"tag":163,"props":245,"children":247},{"id":246},"choose-fsdp-when",[248],{"type":47,"value":249},"Choose FSDP when",{"type":41,"tag":175,"props":251,"children":252},{},[253,258,263,268],{"type":41,"tag":141,"props":254,"children":255},{},[256],{"type":47,"value":257},"you are bringing up a new VLM for the first time",{"type":41,"tag":141,"props":259,"children":260},{},[261],{"type":47,"value":262},"the model has awkward stage boundaries across embedding, vision, and decoder",{"type":41,"tag":141,"props":264,"children":265},{},[266],{"type":47,"value":267},"memory fit matters more than absolute throughput",{"type":41,"tag":141,"props":269,"children":270},{},[271],{"type":47,"value":272},"you may freeze the vision stack during decoder-focused tuning",{"type":41,"tag":163,"props":274,"children":276},{"id":275},"choose-3d-parallel-when",[277],{"type":47,"value":278},"Choose 3D parallel when",{"type":41,"tag":175,"props":280,"children":281},{},[282,287,292,297],{"type":41,"tag":141,"props":283,"children":284},{},[285],{"type":47,"value":286},"the model is already stable under FSDP",{"type":41,"tag":141,"props":288,"children":289},{},[290],{"type":47,"value":291},"the PP layout is clear and repeatable",{"type":41,"tag":141,"props":293,"children":294},{},[295],{"type":47,"value":296},"you can sweep MBS, recompute, and CUDA-graph scope together",{"type":41,"tag":141,"props":298,"children":299},{},[300],{"type":47,"value":301},"the goal is best steady-state throughput, not easiest bring-up",{"type":41,"tag":56,"props":303,"children":305},{"id":304},"key-tuning-knobs",[306],{"type":47,"value":307},"Key Tuning Knobs",{"type":41,"tag":137,"props":309,"children":310},{},[311,322,332,342,361],{"type":41,"tag":141,"props":312,"children":313},{},[314,320],{"type":41,"tag":315,"props":316,"children":317},"strong",{},[318],{"type":47,"value":319},"Freeze the vision stack when appropriate",{"type":47,"value":321},": if the work is decoder-focused,\nfreezing the vision side often gives a small but real throughput gain and\nreduces memory pressure.",{"type":41,"tag":141,"props":323,"children":324},{},[325,330],{"type":41,"tag":315,"props":326,"children":327},{},[328],{"type":47,"value":329},"Sweep MBS aggressively",{"type":47,"value":331},": VLMs are more MBS-sensitive than text-only MoE\nruns because the vision path changes the compute-to-overhead balance.",{"type":41,"tag":141,"props":333,"children":334},{},[335,340],{"type":41,"tag":315,"props":336,"children":337},{},[338],{"type":47,"value":339},"Prefer selective recompute once the model fits",{"type":47,"value":341},": full recompute is a\nuseful bring-up tool, but selective recompute is usually the better steady\nstate.",{"type":41,"tag":141,"props":343,"children":344},{},[345,350,352,359],{"type":41,"tag":315,"props":346,"children":347},{},[348],{"type":47,"value":349},"Match CUDA-graph scope to the workload",{"type":47,"value":351},": ",{"type":41,"tag":353,"props":354,"children":356},"code",{"className":355},[],[357],{"type":47,"value":358},"attn moe_router moe_preprocess",{"type":47,"value":360},"\nis the safer MoE default, while narrower scopes can still be useful for\ncontrolled experiments.",{"type":41,"tag":141,"props":362,"children":363},{},[364,369],{"type":41,"tag":315,"props":365,"children":366},{},[367],{"type":47,"value":368},"Use ETP only when EP alone is insufficient",{"type":47,"value":370},": it can unlock a layout, but\nit also introduces more communication and more tuning surface.",{"type":41,"tag":56,"props":372,"children":374},{"id":373},"representative-config-families",[375],{"type":47,"value":376},"Representative Config Families",{"type":41,"tag":163,"props":378,"children":380},{"id":379},"fsdp-first-gb200-path",[381],{"type":47,"value":382},"FSDP-first GB200 path",{"type":41,"tag":384,"props":385,"children":390},"pre",{"className":386,"code":388,"language":47,"meta":389},[387],"language-text","TP=1  CP=1  PP=1\nEP sized to the expert topology, often large\nDispatcher: HybridEP on GB200-class systems\nRecompute: start with full, then relax toward selective recompute\n","",[391],{"type":41,"tag":353,"props":392,"children":393},{"__ignoreMap":389},[394],{"type":47,"value":388},{"type":41,"tag":163,"props":396,"children":398},{"id":397},"_3d-parallel-gb200-path",[399],{"type":47,"value":400},"3D-parallel GB200 path",{"type":41,"tag":384,"props":402,"children":405},{"className":403,"code":404,"language":47,"meta":389},[387],"TP=1  CP=1  PP=1 or modest PP\nEP and ETP sized to the expert topology\nDispatcher: HybridEP\nCUDA Graph: start narrow, then widen only after the real-data path is stable\n",[406],{"type":41,"tag":353,"props":407,"children":408},{"__ignoreMap":389},[409],{"type":47,"value":404},{"type":41,"tag":56,"props":411,"children":413},{"id":412},"compatibility",[414],{"type":47,"value":415},"Compatibility",{"type":41,"tag":63,"props":417,"children":418},{},[419,438],{"type":41,"tag":67,"props":420,"children":421},{},[422],{"type":41,"tag":71,"props":423,"children":424},{},[425,430,434],{"type":41,"tag":75,"props":426,"children":427},{},[428],{"type":47,"value":429},"Feature",{"type":41,"tag":75,"props":431,"children":432},{},[433],{"type":47,"value":102},{"type":41,"tag":75,"props":435,"children":436},{},[437],{"type":47,"value":120},{"type":41,"tag":91,"props":439,"children":440},{},[441,459,477,495],{"type":41,"tag":71,"props":442,"children":443},{},[444,449,454],{"type":41,"tag":98,"props":445,"children":446},{},[447],{"type":47,"value":448},"HybridEP on GB200",{"type":41,"tag":98,"props":450,"children":451},{},[452],{"type":47,"value":453},"strong default",{"type":41,"tag":98,"props":455,"children":456},{},[457],{"type":47,"value":458},"strong default once topology is stable",{"type":41,"tag":71,"props":460,"children":461},{},[462,467,472],{"type":41,"tag":98,"props":463,"children":464},{},[465],{"type":47,"value":466},"CUDA graphs",{"type":41,"tag":98,"props":468,"children":469},{},[470],{"type":47,"value":471},"useful after bring-up",{"type":41,"tag":98,"props":473,"children":474},{},[475],{"type":47,"value":476},"useful, but more scope-sensitive",{"type":41,"tag":71,"props":478,"children":479},{},[480,485,490],{"type":41,"tag":98,"props":481,"children":482},{},[483],{"type":47,"value":484},"Freeze vision",{"type":41,"tag":98,"props":486,"children":487},{},[488],{"type":47,"value":489},"natural fit",{"type":41,"tag":98,"props":491,"children":492},{},[493],{"type":47,"value":494},"possible, but less often used as the headline perf path",{"type":41,"tag":71,"props":496,"children":497},{},[498,503,508],{"type":41,"tag":98,"props":499,"children":500},{},[501],{"type":47,"value":502},"Selective recompute",{"type":41,"tag":98,"props":504,"children":505},{},[506],{"type":47,"value":507},"recommended",{"type":41,"tag":98,"props":509,"children":510},{},[511],{"type":47,"value":507},{"type":41,"tag":56,"props":513,"children":515},{"id":514},"pitfalls",[516],{"type":47,"value":517},"Pitfalls",{"type":41,"tag":137,"props":519,"children":520},{},[521,531,541,551,561],{"type":41,"tag":141,"props":522,"children":523},{},[524,529],{"type":41,"tag":315,"props":525,"children":526},{},[527],{"type":47,"value":528},"Mock multimodal data is misleading",{"type":47,"value":530},": it can make the decoder look much\nhealthier than the real end-to-end VLM path.",{"type":41,"tag":141,"props":532,"children":533},{},[534,539],{"type":41,"tag":315,"props":535,"children":536},{},[537],{"type":47,"value":538},"The vision encoder can dominate unexpectedly",{"type":47,"value":540},": profile encoder, projector,\nand decoder separately before attributing everything to the dispatcher.",{"type":41,"tag":141,"props":542,"children":543},{},[544,549],{"type":41,"tag":315,"props":545,"children":546},{},[547],{"type":47,"value":548},"Do not compare FSDP and 3D-parallel runs with different effective work",{"type":47,"value":550},":\nnormalize by useful tokens and workload shape, not only by step time.",{"type":41,"tag":141,"props":552,"children":553},{},[554,559],{"type":41,"tag":315,"props":555,"children":556},{},[557],{"type":47,"value":558},"ETP is not free",{"type":47,"value":560},": use it as a fit or topology tool, not as the default.",{"type":41,"tag":141,"props":562,"children":563},{},[564,569],{"type":41,"tag":315,"props":565,"children":566},{},[567],{"type":47,"value":568},"Recompute and CUDA-graph choices are coupled",{"type":47,"value":570},": the setting that gets the\nmodel to fit is often not the setting that gives the best steady-state speed.",{"items":572,"total":731},[573,591,609,620,632,646,659,673,686,697,711,720],{"slug":574,"name":574,"fn":575,"description":576,"org":577,"tags":578,"stars":588,"repoUrl":589,"updatedAt":590},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[579,582,585],{"name":580,"slug":581,"type":15},"Documentation","documentation",{"name":583,"slug":584,"type":15},"MCP","mcp",{"name":586,"slug":587,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":592,"name":592,"fn":593,"description":594,"org":595,"tags":596,"stars":606,"repoUrl":607,"updatedAt":608},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[597,600,603],{"name":598,"slug":599,"type":15},"Containers","containers",{"name":601,"slug":602,"type":15},"Deployment","deployment",{"name":604,"slug":605,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":610,"name":610,"fn":611,"description":612,"org":613,"tags":614,"stars":606,"repoUrl":607,"updatedAt":619},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[615,618],{"name":616,"slug":617,"type":15},"CI\u002FCD","ci-cd",{"name":601,"slug":602,"type":15},"2026-07-14T05:25:59.97109",{"slug":621,"name":621,"fn":622,"description":623,"org":624,"tags":625,"stars":606,"repoUrl":607,"updatedAt":631},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[626,627,628],{"name":616,"slug":617,"type":15},{"name":601,"slug":602,"type":15},{"name":629,"slug":630,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":633,"name":633,"fn":634,"description":635,"org":636,"tags":637,"stars":606,"repoUrl":607,"updatedAt":645},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[638,641,642],{"name":639,"slug":640,"type":15},"Debugging","debugging",{"name":629,"slug":630,"type":15},{"name":643,"slug":644,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":647,"name":647,"fn":648,"description":649,"org":650,"tags":651,"stars":606,"repoUrl":607,"updatedAt":658},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[652,655],{"name":653,"slug":654,"type":15},"Best Practices","best-practices",{"name":656,"slug":657,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":660,"name":660,"fn":661,"description":662,"org":663,"tags":664,"stars":606,"repoUrl":607,"updatedAt":672},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[665,668,671],{"name":666,"slug":667,"type":15},"Machine Learning","machine-learning",{"name":669,"slug":670,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":674,"name":674,"fn":675,"description":676,"org":677,"tags":678,"stars":606,"repoUrl":607,"updatedAt":685},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[679,682],{"name":680,"slug":681,"type":15},"QA","qa",{"name":683,"slug":684,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":687,"name":687,"fn":688,"description":689,"org":690,"tags":691,"stars":606,"repoUrl":607,"updatedAt":696},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[692,693],{"name":601,"slug":602,"type":15},{"name":694,"slug":695,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":698,"name":698,"fn":699,"description":700,"org":701,"tags":702,"stars":606,"repoUrl":607,"updatedAt":710},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[703,706,707],{"name":704,"slug":705,"type":15},"Code Review","code-review",{"name":629,"slug":630,"type":15},{"name":708,"slug":709,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":712,"name":712,"fn":713,"description":714,"org":715,"tags":716,"stars":606,"repoUrl":607,"updatedAt":719},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[717,718],{"name":680,"slug":681,"type":15},{"name":683,"slug":684,"type":15},"2026-07-14T05:25:54.928983",{"slug":721,"name":721,"fn":722,"description":723,"org":724,"tags":725,"stars":606,"repoUrl":607,"updatedAt":730},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[726,729],{"name":727,"slug":728,"type":15},"Automation","automation",{"name":616,"slug":617,"type":15},"2026-07-30T05:29:03.275638",496,{"items":733,"total":827},[734,749,759,773,783,798,813],{"slug":735,"name":735,"fn":736,"description":737,"org":738,"tags":739,"stars":23,"repoUrl":24,"updatedAt":748},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[740,743,746,747],{"name":741,"slug":742,"type":15},"Data Analysis","data-analysis",{"name":744,"slug":745,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:28:43.176466",{"slug":750,"name":750,"fn":751,"description":752,"org":753,"tags":754,"stars":23,"repoUrl":24,"updatedAt":758},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[755,756,757],{"name":601,"slug":602,"type":15},{"name":694,"slug":695,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":760,"name":760,"fn":761,"description":762,"org":763,"tags":764,"stars":23,"repoUrl":24,"updatedAt":772},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[765,768,769],{"name":766,"slug":767,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":770,"slug":771,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":774,"name":774,"fn":775,"description":776,"org":777,"tags":778,"stars":23,"repoUrl":24,"updatedAt":782},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[779,780,781],{"name":741,"slug":742,"type":15},{"name":9,"slug":8,"type":15},{"name":683,"slug":684,"type":15},"2026-07-17T05:29:03.913266",{"slug":784,"name":784,"fn":785,"description":786,"org":787,"tags":788,"stars":23,"repoUrl":24,"updatedAt":797},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[789,790,793,794],{"name":727,"slug":728,"type":15},{"name":791,"slug":792,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":795,"slug":796,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":799,"name":799,"fn":800,"description":801,"org":802,"tags":803,"stars":23,"repoUrl":24,"updatedAt":812},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[804,805,808,809],{"name":601,"slug":602,"type":15},{"name":806,"slug":807,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":810,"slug":811,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":814,"name":814,"fn":815,"description":816,"org":817,"tags":818,"stars":23,"repoUrl":24,"updatedAt":826},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[819,820,823],{"name":9,"slug":8,"type":15},{"name":821,"slug":822,"type":15},"Quantum Computing","quantum-computing",{"name":824,"slug":825,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305]