[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nemo-mbridge-perf-moe-optimization-workflow":3,"mdc--h3tebb-key":34,"related-repo-nvidia-nemo-mbridge-perf-moe-optimization-workflow":887,"related-org-nvidia-nemo-mbridge-perf-moe-optimization-workflow":991},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"nemo-mbridge-perf-moe-optimization-workflow","optimize MoE training in Megatron Bridge","Systematic workflow for MoE training optimization in Megatron Bridge, based on the Megatron-Core MoE paper. Covers the Three Walls framework, parallel folding, recompute strategy, dispatcher choice, and CUDA-graph bring-up.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":17,"slug":18,"type":15},"Deep Learning","deep-learning",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Engineering","engineering",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:27:00.12376","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fnemo-mbridge-perf-moe-optimization-workflow","---\nname: nemo-mbridge-perf-moe-optimization-workflow\ndescription: Systematic workflow for MoE training optimization in Megatron Bridge, based on the Megatron-Core MoE paper. Covers the Three Walls framework, parallel folding, recompute strategy, dispatcher choice, and CUDA-graph bring-up.\nlicense: Apache-2.0\nwhen_to_use: Full MoE throughput tuning sweep, or diagnosing a MoE throughput regression after a commit or config change; 'optimize MoE throughput', 'MoE perf tuning', 'Three Walls', 'memory wall', 'communication wall', 'compute wall'.\n---\n\n# MoE Training Optimization Workflow\n\nStable docs: @docs\u002Ftraining\u002Fmoe-optimization.md\nCard: @skills\u002Fnemo-mbridge-perf-moe-optimization-workflow\u002Fcard.yaml\nSource: [Scalable Training of MoE Models with Megatron Core](https:\u002F\u002Farxiv.org\u002Fabs\u002F2603.07685)\n\n## Quick Reference\n\nThink in terms of the paper's Three Walls:\n\n- memory wall\n- communication wall\n- compute and host-overhead wall\n\nMoE tuning is iterative. Fixing one wall usually exposes the next one, so the\nbest workflow is: fit first, scale second, profile third, then retune.\n\n## First Answer Checklist\n\nFor MoE optimization workflow prompts, present the response in this order:\n\n1. **Fit**: make the model memory-feasible first. Use the smallest model\n   parallelism that fits, prefer selective recompute before full recompute, add\n   offloading only after recompute and parallelism are insufficient, and use\n   `--fake-init-process-group` to sanity-check large layouts.\n2. **Scale**: maximize DP after the model fits, keep hot communication inside\n   the fastest interconnect, use PP plus VPP for multi-node scaling, prefer EP\n   over extra TP for expert layers, and add CP when long context makes attention\n   memory dominant.\n3. **Profile**: identify the dominant wall: memory, communication, host\n   overhead, or compute.\n4. **Retune**: change dispatcher, overlap, FP8 mode, CUDA graphs, or recompute\n   based on the profiled bottleneck.\n5. Include the exact Parallel Folding meshes: `Attention: TP x CP x DP x PP`\n   and `MoE: ETP x EP x EDP x PP`.\n6. Include the default mappings: `alltoall` for safe bring-up,\n   `flex` + `deepep` for H100\u002FB200-style systems, `flex` + `hybridep` for\n   GB200\u002FGB300\u002FNVL72 systems, Hopper to FP8 blockwise, Blackwell to MXFP8, and\n   dropless MoE TE-scoped CUDA graphs over `attn`, `moe_router`, and\n   `moe_preprocess`.\n\n## Phase 1: Make The Run Memory-Feasible\n\nStart with a configuration that fits reliably before chasing throughput.\n\nRecommended order:\n\n1. Use the smallest amount of model parallelism that still fits.\n2. Turn on selective recompute before falling back to full recompute.\n3. Add offloading only when recompute and parallelism are still insufficient.\n4. Use `--fake-init-process-group` to sanity-check large parallel layouts on a\n   single GPU before burning cluster time.\n\n### Recompute guidance\n\nPrefer selective recompute for MoE runs:\n\n- good first choices: `layernorm`, `core_attn`, `moe_act`, `mlp`, or\n  model-specific modules (`shared_experts`, `mla_up_proj`)\n- use full recompute only when the run still does not fit\n- revisit recompute after enabling CUDA graphs, because some graph scopes and\n  full recompute paths do not mix well\n\nAs a rule of thumb, fine-grained recompute often recovers most of the needed\nmemory while keeping throughput much closer to the non-recompute baseline than\nfull-layer recompute does.\n\n## Phase 2: Choose Parallelism For Scale\n\nPriority order:\n\n1. Maximize DP once the model fits.\n2. Keep the hot communication path inside the fast interconnect when possible.\n3. Use PP, plus VPP if needed, for multi-node scaling.\n4. Prefer EP over extra TP for expert layers.\n5. Add CP for long context once sequence length makes attention memory dominant.\n\n### Parallel Folding\n\nParallel Folding decouples attention and MoE parallelism so you do not have to\npick a single compromise layout:\n\n```text\nAttention: TP × CP × DP × PP\nMoE:       ETP × EP × EDP × PP\n```\n\nKey knobs:\n\n- `--expert-model-parallel-size`\n- `--expert-tensor-parallel-size`\n\nUse it when attention prefers some TP or CP, but expert layers benefit from a\nlarger EP degree than the dense layers can tolerate.\n\n## Phase 3: Profile The Dominant Bottleneck\n\n| Bottleneck | What it looks like | Primary fixes |\n|---|---|---|\n| Memory | Run fits only with aggressive full recompute or OOMs during warmup | selective recompute, FP8, offloading, better PP layout |\n| Communication | Nsight shows large all-to-all or collective blocks | DeepEP or HybridEP, EP overlap, DP\u002FTP overlap, better PP layout |\n| Host overhead | GPU gaps, launch-bound traces, Python overhead | CUDA graphs, `--manual-gc`, higher MBS, CPU affinity tuning |\n| Compute | Low SM utilization after comm and host issues are addressed | grouped GEMM, fusion work, FP8, dispatcher-specific kernel tuning |\n\n## Dispatcher And Overlap Guidance\n\nUse dispatcher choice as a bottleneck fix, not as the first tuning knob.\n\n- `moe_token_dispatcher_type=\"alltoall\"`: safest bring-up path, fine for\n  smaller EP sizes\n- `moe_token_dispatcher_type=\"flex\"` + `moe_flex_dispatcher_backend=\"deepep\"`:\n  strong default for H100 and B200 style deployments\n- `moe_token_dispatcher_type=\"flex\"` + `moe_flex_dispatcher_backend=\"hybridep\"`:\n  strongest starting point on GB200 or GB300 NVL72 systems\n\nIf the all-to-all path is visible in profiles, combine dispatcher tuning with:\n\n- `--overlap-moe-expert-parallel-comm`\n- `--overlap-grad-reduce`\n- `--tp-comm-overlap`\n\n## FP8 Recipe Quick Decision\n\n| Platform | Recommended starting recipe |\n|---|---|\n| Hopper | FP8 blockwise |\n| Blackwell | MXFP8 |\n| Blackwell, speed-first exploration | NVFP4 after the BF16 or FP8 path is stable |\n\nKeep the router in FP32. The largest wins usually come from expert GEMMs and\nother heavy matrix math, not from trying to quantize every small MoE component.\n\n## CUDA Graphs For MoE\n\nFor dropless MoE, start with partial TE-scoped graphs:\n\n- `attn`\n- `moe_router`\n- `moe_preprocess`\n\nThat path usually gives a meaningful step-time win while keeping the dynamic\nexpert work outside the graph. Expect a moderate speedup when launch overhead is\nvisible, but budget several extra GB of memory and verify that shapes remain\nstatic.\n\nUse full-iteration graphs only for graph-friendly workloads such as drop-and-pad\nor tightly controlled static-shape experiments.\n\nRelated references:\n\n- @skills\u002Fnemo-mbridge-perf-cuda-graphs\u002FSKILL.md\n- @docs\u002Ftraining\u002Fcuda-graphs.md\n- @docs\u002Ftraining\u002Factivation-recomputation.md\n\n## Pitfalls\n\n1. **Do not optimize in the wrong order**: fitting the model and selecting sane\n   parallelism matter more than micro-optimizations.\n\n2. **Platform changes the limiting wall**: H100-class runs often feel more\n   communication-bound, while GB200 or GB300 runs often expose CPU or launch\n   overhead earlier.\n\n3. **FP8 MFU can look misleadingly low**: compare absolute throughput as well as\n   MFU when switching precision modes.\n\n4. **CUDA graphs and recompute interact**: TE-scoped graphs are usually paired\n   with selective recompute, not blanket full recompute.\n\n5. **Parallel Folding is not optional at large scale**: once attention and expert\n   layers want clearly different layouts, a single shared TP or EP plan becomes\n   a tax on both.\n",{"data":35,"body":37},{"name":4,"description":6,"license":26,"when_to_use":36},"Full MoE throughput tuning sweep, or diagnosing a MoE throughput regression after a commit or config change; 'optimize MoE throughput', 'MoE perf tuning', 'Three Walls', 'memory wall', 'communication wall', 'compute wall'.",{"type":38,"children":39},"root",[40,49,64,71,76,96,101,107,112,253,259,264,269,299,306,311,373,378,384,389,417,423,428,440,445,466,471,477,590,596,601,650,655,685,691,752,757,763,768,795,800,805,810,828,834],{"type":41,"tag":42,"props":43,"children":45},"element","h1",{"id":44},"moe-training-optimization-workflow",[46],{"type":47,"value":48},"text","MoE Training Optimization Workflow",{"type":41,"tag":50,"props":51,"children":52},"p",{},[53,55],{"type":47,"value":54},"Stable docs: @docs\u002Ftraining\u002Fmoe-optimization.md\nCard: @skills\u002Fnemo-mbridge-perf-moe-optimization-workflow\u002Fcard.yaml\nSource: ",{"type":41,"tag":56,"props":57,"children":61},"a",{"href":58,"rel":59},"https:\u002F\u002Farxiv.org\u002Fabs\u002F2603.07685",[60],"nofollow",[62],{"type":47,"value":63},"Scalable Training of MoE Models with Megatron Core",{"type":41,"tag":65,"props":66,"children":68},"h2",{"id":67},"quick-reference",[69],{"type":47,"value":70},"Quick Reference",{"type":41,"tag":50,"props":72,"children":73},{},[74],{"type":47,"value":75},"Think in terms of the paper's Three Walls:",{"type":41,"tag":77,"props":78,"children":79},"ul",{},[80,86,91],{"type":41,"tag":81,"props":82,"children":83},"li",{},[84],{"type":47,"value":85},"memory wall",{"type":41,"tag":81,"props":87,"children":88},{},[89],{"type":47,"value":90},"communication wall",{"type":41,"tag":81,"props":92,"children":93},{},[94],{"type":47,"value":95},"compute and host-overhead wall",{"type":41,"tag":50,"props":97,"children":98},{},[99],{"type":47,"value":100},"MoE tuning is iterative. Fixing one wall usually exposes the next one, so the\nbest workflow is: fit first, scale second, profile third, then retune.",{"type":41,"tag":65,"props":102,"children":104},{"id":103},"first-answer-checklist",[105],{"type":47,"value":106},"First Answer Checklist",{"type":41,"tag":50,"props":108,"children":109},{},[110],{"type":47,"value":111},"For MoE optimization workflow prompts, present the response in this order:",{"type":41,"tag":113,"props":114,"children":115},"ol",{},[116,136,146,156,166,187],{"type":41,"tag":81,"props":117,"children":118},{},[119,125,127,134],{"type":41,"tag":120,"props":121,"children":122},"strong",{},[123],{"type":47,"value":124},"Fit",{"type":47,"value":126},": make the model memory-feasible first. Use the smallest model\nparallelism that fits, prefer selective recompute before full recompute, add\noffloading only after recompute and parallelism are insufficient, and use\n",{"type":41,"tag":128,"props":129,"children":131},"code",{"className":130},[],[132],{"type":47,"value":133},"--fake-init-process-group",{"type":47,"value":135}," to sanity-check large layouts.",{"type":41,"tag":81,"props":137,"children":138},{},[139,144],{"type":41,"tag":120,"props":140,"children":141},{},[142],{"type":47,"value":143},"Scale",{"type":47,"value":145},": maximize DP after the model fits, keep hot communication inside\nthe fastest interconnect, use PP plus VPP for multi-node scaling, prefer EP\nover extra TP for expert layers, and add CP when long context makes attention\nmemory dominant.",{"type":41,"tag":81,"props":147,"children":148},{},[149,154],{"type":41,"tag":120,"props":150,"children":151},{},[152],{"type":47,"value":153},"Profile",{"type":47,"value":155},": identify the dominant wall: memory, communication, host\noverhead, or compute.",{"type":41,"tag":81,"props":157,"children":158},{},[159,164],{"type":41,"tag":120,"props":160,"children":161},{},[162],{"type":47,"value":163},"Retune",{"type":47,"value":165},": change dispatcher, overlap, FP8 mode, CUDA graphs, or recompute\nbased on the profiled bottleneck.",{"type":41,"tag":81,"props":167,"children":168},{},[169,171,177,179,185],{"type":47,"value":170},"Include the exact Parallel Folding meshes: ",{"type":41,"tag":128,"props":172,"children":174},{"className":173},[],[175],{"type":47,"value":176},"Attention: TP x CP x DP x PP",{"type":47,"value":178},"\nand ",{"type":41,"tag":128,"props":180,"children":182},{"className":181},[],[183],{"type":47,"value":184},"MoE: ETP x EP x EDP x PP",{"type":47,"value":186},".",{"type":41,"tag":81,"props":188,"children":189},{},[190,192,198,200,206,208,214,216,221,222,228,230,236,238,244,246,252],{"type":47,"value":191},"Include the default mappings: ",{"type":41,"tag":128,"props":193,"children":195},{"className":194},[],[196],{"type":47,"value":197},"alltoall",{"type":47,"value":199}," for safe bring-up,\n",{"type":41,"tag":128,"props":201,"children":203},{"className":202},[],[204],{"type":47,"value":205},"flex",{"type":47,"value":207}," + ",{"type":41,"tag":128,"props":209,"children":211},{"className":210},[],[212],{"type":47,"value":213},"deepep",{"type":47,"value":215}," for H100\u002FB200-style systems, ",{"type":41,"tag":128,"props":217,"children":219},{"className":218},[],[220],{"type":47,"value":205},{"type":47,"value":207},{"type":41,"tag":128,"props":223,"children":225},{"className":224},[],[226],{"type":47,"value":227},"hybridep",{"type":47,"value":229}," for\nGB200\u002FGB300\u002FNVL72 systems, Hopper to FP8 blockwise, Blackwell to MXFP8, and\ndropless MoE TE-scoped CUDA graphs over ",{"type":41,"tag":128,"props":231,"children":233},{"className":232},[],[234],{"type":47,"value":235},"attn",{"type":47,"value":237},", ",{"type":41,"tag":128,"props":239,"children":241},{"className":240},[],[242],{"type":47,"value":243},"moe_router",{"type":47,"value":245},", and\n",{"type":41,"tag":128,"props":247,"children":249},{"className":248},[],[250],{"type":47,"value":251},"moe_preprocess",{"type":47,"value":186},{"type":41,"tag":65,"props":254,"children":256},{"id":255},"phase-1-make-the-run-memory-feasible",[257],{"type":47,"value":258},"Phase 1: Make The Run Memory-Feasible",{"type":41,"tag":50,"props":260,"children":261},{},[262],{"type":47,"value":263},"Start with a configuration that fits reliably before chasing throughput.",{"type":41,"tag":50,"props":265,"children":266},{},[267],{"type":47,"value":268},"Recommended order:",{"type":41,"tag":113,"props":270,"children":271},{},[272,277,282,287],{"type":41,"tag":81,"props":273,"children":274},{},[275],{"type":47,"value":276},"Use the smallest amount of model parallelism that still fits.",{"type":41,"tag":81,"props":278,"children":279},{},[280],{"type":47,"value":281},"Turn on selective recompute before falling back to full recompute.",{"type":41,"tag":81,"props":283,"children":284},{},[285],{"type":47,"value":286},"Add offloading only when recompute and parallelism are still insufficient.",{"type":41,"tag":81,"props":288,"children":289},{},[290,292,297],{"type":47,"value":291},"Use ",{"type":41,"tag":128,"props":293,"children":295},{"className":294},[],[296],{"type":47,"value":133},{"type":47,"value":298}," to sanity-check large parallel layouts on a\nsingle GPU before burning cluster time.",{"type":41,"tag":300,"props":301,"children":303},"h3",{"id":302},"recompute-guidance",[304],{"type":47,"value":305},"Recompute guidance",{"type":41,"tag":50,"props":307,"children":308},{},[309],{"type":47,"value":310},"Prefer selective recompute for MoE runs:",{"type":41,"tag":77,"props":312,"children":313},{},[314,363,368],{"type":41,"tag":81,"props":315,"children":316},{},[317,319,325,326,332,333,339,340,346,348,354,355,361],{"type":47,"value":318},"good first choices: ",{"type":41,"tag":128,"props":320,"children":322},{"className":321},[],[323],{"type":47,"value":324},"layernorm",{"type":47,"value":237},{"type":41,"tag":128,"props":327,"children":329},{"className":328},[],[330],{"type":47,"value":331},"core_attn",{"type":47,"value":237},{"type":41,"tag":128,"props":334,"children":336},{"className":335},[],[337],{"type":47,"value":338},"moe_act",{"type":47,"value":237},{"type":41,"tag":128,"props":341,"children":343},{"className":342},[],[344],{"type":47,"value":345},"mlp",{"type":47,"value":347},", or\nmodel-specific modules (",{"type":41,"tag":128,"props":349,"children":351},{"className":350},[],[352],{"type":47,"value":353},"shared_experts",{"type":47,"value":237},{"type":41,"tag":128,"props":356,"children":358},{"className":357},[],[359],{"type":47,"value":360},"mla_up_proj",{"type":47,"value":362},")",{"type":41,"tag":81,"props":364,"children":365},{},[366],{"type":47,"value":367},"use full recompute only when the run still does not fit",{"type":41,"tag":81,"props":369,"children":370},{},[371],{"type":47,"value":372},"revisit recompute after enabling CUDA graphs, because some graph scopes and\nfull recompute paths do not mix well",{"type":41,"tag":50,"props":374,"children":375},{},[376],{"type":47,"value":377},"As a rule of thumb, fine-grained recompute often recovers most of the needed\nmemory while keeping throughput much closer to the non-recompute baseline than\nfull-layer recompute does.",{"type":41,"tag":65,"props":379,"children":381},{"id":380},"phase-2-choose-parallelism-for-scale",[382],{"type":47,"value":383},"Phase 2: Choose Parallelism For Scale",{"type":41,"tag":50,"props":385,"children":386},{},[387],{"type":47,"value":388},"Priority order:",{"type":41,"tag":113,"props":390,"children":391},{},[392,397,402,407,412],{"type":41,"tag":81,"props":393,"children":394},{},[395],{"type":47,"value":396},"Maximize DP once the model fits.",{"type":41,"tag":81,"props":398,"children":399},{},[400],{"type":47,"value":401},"Keep the hot communication path inside the fast interconnect when possible.",{"type":41,"tag":81,"props":403,"children":404},{},[405],{"type":47,"value":406},"Use PP, plus VPP if needed, for multi-node scaling.",{"type":41,"tag":81,"props":408,"children":409},{},[410],{"type":47,"value":411},"Prefer EP over extra TP for expert layers.",{"type":41,"tag":81,"props":413,"children":414},{},[415],{"type":47,"value":416},"Add CP for long context once sequence length makes attention memory dominant.",{"type":41,"tag":300,"props":418,"children":420},{"id":419},"parallel-folding",[421],{"type":47,"value":422},"Parallel Folding",{"type":41,"tag":50,"props":424,"children":425},{},[426],{"type":47,"value":427},"Parallel Folding decouples attention and MoE parallelism so you do not have to\npick a single compromise layout:",{"type":41,"tag":429,"props":430,"children":435},"pre",{"className":431,"code":433,"language":47,"meta":434},[432],"language-text","Attention: TP × CP × DP × PP\nMoE:       ETP × EP × EDP × PP\n","",[436],{"type":41,"tag":128,"props":437,"children":438},{"__ignoreMap":434},[439],{"type":47,"value":433},{"type":41,"tag":50,"props":441,"children":442},{},[443],{"type":47,"value":444},"Key knobs:",{"type":41,"tag":77,"props":446,"children":447},{},[448,457],{"type":41,"tag":81,"props":449,"children":450},{},[451],{"type":41,"tag":128,"props":452,"children":454},{"className":453},[],[455],{"type":47,"value":456},"--expert-model-parallel-size",{"type":41,"tag":81,"props":458,"children":459},{},[460],{"type":41,"tag":128,"props":461,"children":463},{"className":462},[],[464],{"type":47,"value":465},"--expert-tensor-parallel-size",{"type":41,"tag":50,"props":467,"children":468},{},[469],{"type":47,"value":470},"Use it when attention prefers some TP or CP, but expert layers benefit from a\nlarger EP degree than the dense layers can tolerate.",{"type":41,"tag":65,"props":472,"children":474},{"id":473},"phase-3-profile-the-dominant-bottleneck",[475],{"type":47,"value":476},"Phase 3: Profile The Dominant Bottleneck",{"type":41,"tag":478,"props":479,"children":480},"table",{},[481,505],{"type":41,"tag":482,"props":483,"children":484},"thead",{},[485],{"type":41,"tag":486,"props":487,"children":488},"tr",{},[489,495,500],{"type":41,"tag":490,"props":491,"children":492},"th",{},[493],{"type":47,"value":494},"Bottleneck",{"type":41,"tag":490,"props":496,"children":497},{},[498],{"type":47,"value":499},"What it looks like",{"type":41,"tag":490,"props":501,"children":502},{},[503],{"type":47,"value":504},"Primary fixes",{"type":41,"tag":506,"props":507,"children":508},"tbody",{},[509,528,546,572],{"type":41,"tag":486,"props":510,"children":511},{},[512,518,523],{"type":41,"tag":513,"props":514,"children":515},"td",{},[516],{"type":47,"value":517},"Memory",{"type":41,"tag":513,"props":519,"children":520},{},[521],{"type":47,"value":522},"Run fits only with aggressive full recompute or OOMs during warmup",{"type":41,"tag":513,"props":524,"children":525},{},[526],{"type":47,"value":527},"selective recompute, FP8, offloading, better PP layout",{"type":41,"tag":486,"props":529,"children":530},{},[531,536,541],{"type":41,"tag":513,"props":532,"children":533},{},[534],{"type":47,"value":535},"Communication",{"type":41,"tag":513,"props":537,"children":538},{},[539],{"type":47,"value":540},"Nsight shows large all-to-all or collective blocks",{"type":41,"tag":513,"props":542,"children":543},{},[544],{"type":47,"value":545},"DeepEP or HybridEP, EP overlap, DP\u002FTP overlap, better PP layout",{"type":41,"tag":486,"props":547,"children":548},{},[549,554,559],{"type":41,"tag":513,"props":550,"children":551},{},[552],{"type":47,"value":553},"Host overhead",{"type":41,"tag":513,"props":555,"children":556},{},[557],{"type":47,"value":558},"GPU gaps, launch-bound traces, Python overhead",{"type":41,"tag":513,"props":560,"children":561},{},[562,564,570],{"type":47,"value":563},"CUDA graphs, ",{"type":41,"tag":128,"props":565,"children":567},{"className":566},[],[568],{"type":47,"value":569},"--manual-gc",{"type":47,"value":571},", higher MBS, CPU affinity tuning",{"type":41,"tag":486,"props":573,"children":574},{},[575,580,585],{"type":41,"tag":513,"props":576,"children":577},{},[578],{"type":47,"value":579},"Compute",{"type":41,"tag":513,"props":581,"children":582},{},[583],{"type":47,"value":584},"Low SM utilization after comm and host issues are addressed",{"type":41,"tag":513,"props":586,"children":587},{},[588],{"type":47,"value":589},"grouped GEMM, fusion work, FP8, dispatcher-specific kernel tuning",{"type":41,"tag":65,"props":591,"children":593},{"id":592},"dispatcher-and-overlap-guidance",[594],{"type":47,"value":595},"Dispatcher And Overlap Guidance",{"type":41,"tag":50,"props":597,"children":598},{},[599],{"type":47,"value":600},"Use dispatcher choice as a bottleneck fix, not as the first tuning knob.",{"type":41,"tag":77,"props":602,"children":603},{},[604,615,633],{"type":41,"tag":81,"props":605,"children":606},{},[607,613],{"type":41,"tag":128,"props":608,"children":610},{"className":609},[],[611],{"type":47,"value":612},"moe_token_dispatcher_type=\"alltoall\"",{"type":47,"value":614},": safest bring-up path, fine for\nsmaller EP sizes",{"type":41,"tag":81,"props":616,"children":617},{},[618,624,625,631],{"type":41,"tag":128,"props":619,"children":621},{"className":620},[],[622],{"type":47,"value":623},"moe_token_dispatcher_type=\"flex\"",{"type":47,"value":207},{"type":41,"tag":128,"props":626,"children":628},{"className":627},[],[629],{"type":47,"value":630},"moe_flex_dispatcher_backend=\"deepep\"",{"type":47,"value":632},":\nstrong default for H100 and B200 style deployments",{"type":41,"tag":81,"props":634,"children":635},{},[636,641,642,648],{"type":41,"tag":128,"props":637,"children":639},{"className":638},[],[640],{"type":47,"value":623},{"type":47,"value":207},{"type":41,"tag":128,"props":643,"children":645},{"className":644},[],[646],{"type":47,"value":647},"moe_flex_dispatcher_backend=\"hybridep\"",{"type":47,"value":649},":\nstrongest starting point on GB200 or GB300 NVL72 systems",{"type":41,"tag":50,"props":651,"children":652},{},[653],{"type":47,"value":654},"If the all-to-all path is visible in profiles, combine dispatcher tuning with:",{"type":41,"tag":77,"props":656,"children":657},{},[658,667,676],{"type":41,"tag":81,"props":659,"children":660},{},[661],{"type":41,"tag":128,"props":662,"children":664},{"className":663},[],[665],{"type":47,"value":666},"--overlap-moe-expert-parallel-comm",{"type":41,"tag":81,"props":668,"children":669},{},[670],{"type":41,"tag":128,"props":671,"children":673},{"className":672},[],[674],{"type":47,"value":675},"--overlap-grad-reduce",{"type":41,"tag":81,"props":677,"children":678},{},[679],{"type":41,"tag":128,"props":680,"children":682},{"className":681},[],[683],{"type":47,"value":684},"--tp-comm-overlap",{"type":41,"tag":65,"props":686,"children":688},{"id":687},"fp8-recipe-quick-decision",[689],{"type":47,"value":690},"FP8 Recipe Quick Decision",{"type":41,"tag":478,"props":692,"children":693},{},[694,710],{"type":41,"tag":482,"props":695,"children":696},{},[697],{"type":41,"tag":486,"props":698,"children":699},{},[700,705],{"type":41,"tag":490,"props":701,"children":702},{},[703],{"type":47,"value":704},"Platform",{"type":41,"tag":490,"props":706,"children":707},{},[708],{"type":47,"value":709},"Recommended starting recipe",{"type":41,"tag":506,"props":711,"children":712},{},[713,726,739],{"type":41,"tag":486,"props":714,"children":715},{},[716,721],{"type":41,"tag":513,"props":717,"children":718},{},[719],{"type":47,"value":720},"Hopper",{"type":41,"tag":513,"props":722,"children":723},{},[724],{"type":47,"value":725},"FP8 blockwise",{"type":41,"tag":486,"props":727,"children":728},{},[729,734],{"type":41,"tag":513,"props":730,"children":731},{},[732],{"type":47,"value":733},"Blackwell",{"type":41,"tag":513,"props":735,"children":736},{},[737],{"type":47,"value":738},"MXFP8",{"type":41,"tag":486,"props":740,"children":741},{},[742,747],{"type":41,"tag":513,"props":743,"children":744},{},[745],{"type":47,"value":746},"Blackwell, speed-first exploration",{"type":41,"tag":513,"props":748,"children":749},{},[750],{"type":47,"value":751},"NVFP4 after the BF16 or FP8 path is stable",{"type":41,"tag":50,"props":753,"children":754},{},[755],{"type":47,"value":756},"Keep the router in FP32. The largest wins usually come from expert GEMMs and\nother heavy matrix math, not from trying to quantize every small MoE component.",{"type":41,"tag":65,"props":758,"children":760},{"id":759},"cuda-graphs-for-moe",[761],{"type":47,"value":762},"CUDA Graphs For MoE",{"type":41,"tag":50,"props":764,"children":765},{},[766],{"type":47,"value":767},"For dropless MoE, start with partial TE-scoped graphs:",{"type":41,"tag":77,"props":769,"children":770},{},[771,779,787],{"type":41,"tag":81,"props":772,"children":773},{},[774],{"type":41,"tag":128,"props":775,"children":777},{"className":776},[],[778],{"type":47,"value":235},{"type":41,"tag":81,"props":780,"children":781},{},[782],{"type":41,"tag":128,"props":783,"children":785},{"className":784},[],[786],{"type":47,"value":243},{"type":41,"tag":81,"props":788,"children":789},{},[790],{"type":41,"tag":128,"props":791,"children":793},{"className":792},[],[794],{"type":47,"value":251},{"type":41,"tag":50,"props":796,"children":797},{},[798],{"type":47,"value":799},"That path usually gives a meaningful step-time win while keeping the dynamic\nexpert work outside the graph. Expect a moderate speedup when launch overhead is\nvisible, but budget several extra GB of memory and verify that shapes remain\nstatic.",{"type":41,"tag":50,"props":801,"children":802},{},[803],{"type":47,"value":804},"Use full-iteration graphs only for graph-friendly workloads such as drop-and-pad\nor tightly controlled static-shape experiments.",{"type":41,"tag":50,"props":806,"children":807},{},[808],{"type":47,"value":809},"Related references:",{"type":41,"tag":77,"props":811,"children":812},{},[813,818,823],{"type":41,"tag":81,"props":814,"children":815},{},[816],{"type":47,"value":817},"@skills\u002Fnemo-mbridge-perf-cuda-graphs\u002FSKILL.md",{"type":41,"tag":81,"props":819,"children":820},{},[821],{"type":47,"value":822},"@docs\u002Ftraining\u002Fcuda-graphs.md",{"type":41,"tag":81,"props":824,"children":825},{},[826],{"type":47,"value":827},"@docs\u002Ftraining\u002Factivation-recomputation.md",{"type":41,"tag":65,"props":829,"children":831},{"id":830},"pitfalls",[832],{"type":47,"value":833},"Pitfalls",{"type":41,"tag":113,"props":835,"children":836},{},[837,847,857,867,877],{"type":41,"tag":81,"props":838,"children":839},{},[840,845],{"type":41,"tag":120,"props":841,"children":842},{},[843],{"type":47,"value":844},"Do not optimize in the wrong order",{"type":47,"value":846},": fitting the model and selecting sane\nparallelism matter more than micro-optimizations.",{"type":41,"tag":81,"props":848,"children":849},{},[850,855],{"type":41,"tag":120,"props":851,"children":852},{},[853],{"type":47,"value":854},"Platform changes the limiting wall",{"type":47,"value":856},": H100-class runs often feel more\ncommunication-bound, while GB200 or GB300 runs often expose CPU or launch\noverhead earlier.",{"type":41,"tag":81,"props":858,"children":859},{},[860,865],{"type":41,"tag":120,"props":861,"children":862},{},[863],{"type":47,"value":864},"FP8 MFU can look misleadingly low",{"type":47,"value":866},": compare absolute throughput as well as\nMFU when switching precision modes.",{"type":41,"tag":81,"props":868,"children":869},{},[870,875],{"type":41,"tag":120,"props":871,"children":872},{},[873],{"type":47,"value":874},"CUDA graphs and recompute interact",{"type":47,"value":876},": TE-scoped graphs are usually paired\nwith selective recompute, not blanket full recompute.",{"type":41,"tag":81,"props":878,"children":879},{},[880,885],{"type":41,"tag":120,"props":881,"children":882},{},[883],{"type":47,"value":884},"Parallel Folding is not optional at large scale",{"type":47,"value":886},": once attention and expert\nlayers want clearly different layouts, a single shared TP or EP plan becomes\na tax on both.",{"items":888,"total":990},[889,904,918,932,944,961,976],{"slug":890,"name":890,"fn":891,"description":892,"org":893,"tags":894,"stars":23,"repoUrl":24,"updatedAt":903},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[895,898,901,902],{"name":896,"slug":897,"type":15},"Data Analysis","data-analysis",{"name":899,"slug":900,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:28:43.176466",{"slug":905,"name":905,"fn":906,"description":907,"org":908,"tags":909,"stars":23,"repoUrl":24,"updatedAt":917},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[910,913,916],{"name":911,"slug":912,"type":15},"Deployment","deployment",{"name":914,"slug":915,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":919,"name":919,"fn":920,"description":921,"org":922,"tags":923,"stars":23,"repoUrl":24,"updatedAt":931},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[924,927,928],{"name":925,"slug":926,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":929,"slug":930,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":933,"name":933,"fn":934,"description":935,"org":936,"tags":937,"stars":23,"repoUrl":24,"updatedAt":943},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[938,939,940],{"name":896,"slug":897,"type":15},{"name":9,"slug":8,"type":15},{"name":941,"slug":942,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":945,"name":945,"fn":946,"description":947,"org":948,"tags":949,"stars":23,"repoUrl":24,"updatedAt":960},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[950,953,956,957],{"name":951,"slug":952,"type":15},"Automation","automation",{"name":954,"slug":955,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":958,"slug":959,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":962,"name":962,"fn":963,"description":964,"org":965,"tags":966,"stars":23,"repoUrl":24,"updatedAt":975},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[967,968,971,972],{"name":911,"slug":912,"type":15},{"name":969,"slug":970,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":973,"slug":974,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":977,"name":977,"fn":978,"description":979,"org":980,"tags":981,"stars":23,"repoUrl":24,"updatedAt":989},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[982,983,986],{"name":9,"slug":8,"type":15},{"name":984,"slug":985,"type":15},"Quantum Computing","quantum-computing",{"name":987,"slug":988,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":992,"total":1143},[993,1011,1027,1038,1050,1064,1077,1091,1102,1111,1125,1134],{"slug":994,"name":994,"fn":995,"description":996,"org":997,"tags":998,"stars":1008,"repoUrl":1009,"updatedAt":1010},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[999,1002,1005],{"name":1000,"slug":1001,"type":15},"Documentation","documentation",{"name":1003,"slug":1004,"type":15},"MCP","mcp",{"name":1006,"slug":1007,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":1012,"name":1012,"fn":1013,"description":1014,"org":1015,"tags":1016,"stars":1024,"repoUrl":1025,"updatedAt":1026},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1017,1020,1021],{"name":1018,"slug":1019,"type":15},"Containers","containers",{"name":911,"slug":912,"type":15},{"name":1022,"slug":1023,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":1028,"name":1028,"fn":1029,"description":1030,"org":1031,"tags":1032,"stars":1024,"repoUrl":1025,"updatedAt":1037},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1033,1036],{"name":1034,"slug":1035,"type":15},"CI\u002FCD","ci-cd",{"name":911,"slug":912,"type":15},"2026-07-14T05:25:59.97109",{"slug":1039,"name":1039,"fn":1040,"description":1041,"org":1042,"tags":1043,"stars":1024,"repoUrl":1025,"updatedAt":1049},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1044,1045,1046],{"name":1034,"slug":1035,"type":15},{"name":911,"slug":912,"type":15},{"name":1047,"slug":1048,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":1051,"name":1051,"fn":1052,"description":1053,"org":1054,"tags":1055,"stars":1024,"repoUrl":1025,"updatedAt":1063},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1056,1059,1060],{"name":1057,"slug":1058,"type":15},"Debugging","debugging",{"name":1047,"slug":1048,"type":15},{"name":1061,"slug":1062,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1065,"name":1065,"fn":1066,"description":1067,"org":1068,"tags":1069,"stars":1024,"repoUrl":1025,"updatedAt":1076},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1070,1073],{"name":1071,"slug":1072,"type":15},"Best Practices","best-practices",{"name":1074,"slug":1075,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1078,"name":1078,"fn":1079,"description":1080,"org":1081,"tags":1082,"stars":1024,"repoUrl":1025,"updatedAt":1090},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1083,1086,1089],{"name":1084,"slug":1085,"type":15},"Machine Learning","machine-learning",{"name":1087,"slug":1088,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":1092,"name":1092,"fn":1093,"description":1094,"org":1095,"tags":1096,"stars":1024,"repoUrl":1025,"updatedAt":1101},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1097,1100],{"name":1098,"slug":1099,"type":15},"QA","qa",{"name":941,"slug":942,"type":15},"2026-07-14T05:25:53.673039",{"slug":1103,"name":1103,"fn":1104,"description":1105,"org":1106,"tags":1107,"stars":1024,"repoUrl":1025,"updatedAt":1110},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1108,1109],{"name":911,"slug":912,"type":15},{"name":914,"slug":915,"type":15},"2026-07-14T05:25:49.362534",{"slug":1112,"name":1112,"fn":1113,"description":1114,"org":1115,"tags":1116,"stars":1024,"repoUrl":1025,"updatedAt":1124},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1117,1120,1121],{"name":1118,"slug":1119,"type":15},"Code Review","code-review",{"name":1047,"slug":1048,"type":15},{"name":1122,"slug":1123,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":1126,"name":1126,"fn":1127,"description":1128,"org":1129,"tags":1130,"stars":1024,"repoUrl":1025,"updatedAt":1133},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1131,1132],{"name":1098,"slug":1099,"type":15},{"name":941,"slug":942,"type":15},"2026-07-14T05:25:54.928983",{"slug":1135,"name":1135,"fn":1136,"description":1137,"org":1138,"tags":1139,"stars":1024,"repoUrl":1025,"updatedAt":1142},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1140,1141],{"name":951,"slug":952,"type":15},{"name":1034,"slug":1035,"type":15},"2026-07-30T05:29:03.275638",496]