[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nemo-automodel-distributed-training":3,"mdc-xsfgsp-key":31,"related-repo-nvidia-nemo-automodel-distributed-training":4085,"related-org-nvidia-nemo-automodel-distributed-training":4191},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":20,"repoUrl":21,"updatedAt":22,"license":23,"forks":24,"topics":25,"repo":26,"sourceUrl":29,"mdContent":30},"nemo-automodel-distributed-training","configure distributed training in NeMo AutoModel","Guide for selecting and configuring distributed training strategies in NeMo AutoModel, including FSDP2, Megatron FSDP, DDP, and parallelism settings.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,17],{"name":13,"slug":14,"type":15},"Machine Learning","machine-learning","tag",{"name":9,"slug":8,"type":15},{"name":18,"slug":19,"type":15},"Engineering","engineering",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:29:25.700517","Apache-2.0",281,[],{"repoUrl":21,"stars":20,"forks":24,"topics":27,"description":28},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fnemo-automodel-distributed-training","---\nname: nemo-automodel-distributed-training\ndescription: Guide for selecting and configuring distributed training strategies in NeMo AutoModel, including FSDP2, Megatron FSDP, DDP, and parallelism settings.\nwhen_to_use: Adding or modifying distributed training strategies (FSDP2, HSDP, DDP), debugging multi-GPU or multi-node failures, configuring context or tensor parallelism, or tuning sharding settings.\nlicense: Apache-2.0\nmetadata:\n  author: NVIDIA\n  tags:\n    - nemo-automodel\n    - distributed-training\n---\n\n# Distributed Training in NeMo AutoModel\n\n## Purpose\n\nNeMo AutoModel uses PyTorch-native distributed training.\nAll parallelism is orchestrated through a single `MeshContext` object that\nholds device meshes, strategy configs, and axis names.\n\u003C!-- NVSkills signature refresh requested for AM-519. -->\n\n## Instructions\n\nFor conceptual distributed-training questions, answer directly from the quick\npatterns in this skill without inspecting the repository. Start with the\nstrategy choice, then list only the YAML fields and constraints relevant to the\nquestion.\n\nUse direct action verbs in the final answer: recommend the strategy, show the\nminimal YAML, state the sizing constraint, and name the unsupported strategies.\nDo not discuss model onboarding, recipes, Slurm, SkyPilot, or checkpointing\nunless the user asks.\n\n## Examples\n\n### TP plus PP for a large multi-node model\n\nRecommend `strategy: fsdp2`. Mention `tp_size`, `pp_size`, `cp_size`,\n`ep_size`, and the `pipeline` sub-config. State that `dp_size` is inferred from\n`world_size \u002F (tp_size * pp_size * cp_size)`.\n\n```yaml\ndistributed:\n  strategy: fsdp2\n  tp_size: 8\n  pp_size: 4\n  cp_size: 1\n  ep_size: 1\n  pipeline:\n    pp_schedule: interleaved1f1b\n    pp_microbatch_size: 1\n```\n\n### MoE expert parallelism\n\nRecommend `strategy: fsdp2` with `ep_size > 1`. Say this creates a separate\n`moe_mesh`; include the `moe` sub-config when relevant; state that `ep_size`\nmust divide `dp_size * cp_size`. Do not recommend `megatron_fsdp` or `ddp`.\n\n```yaml\ndistributed:\n  strategy: fsdp2\n  ep_size: 8\n  moe:\n    reshard_after_forward: false\n```\n\n### MegatronFSDP limitations\n\nSay no for pipeline parallelism, expert parallelism, and `sequence_parallel`.\nRecommend `fsdp2` for PP, EP, or `sequence_parallel`; mention that DDP is only\nsimple data parallelism.\n\n## Strategy Selection\n\nThree strategies are available, selected via the `distributed.strategy` YAML key:\n\n| Strategy | YAML value | Best for |\n|---|---|---|\n| FSDP2 | `fsdp2` | General use, recommended default. Supports TP, PP, CP, EP, HSDP. |\n| MegatronFSDP | `megatron_fsdp` | NVIDIA Megatron-style FSDP. No PP, no EP, no sequence_parallel. |\n| DDP | `ddp` | Simple data parallelism only. No TP, PP, CP, or EP. |\n\nDecision tree:\n\n- Single GPU: no distributed config needed (FSDP2Manager skips parallelization when world_size=1).\n- Multi-GPU single node: `fsdp2` (default). Use `ddp` only if you need the simplest possible setup.\n- Multi-node: `fsdp2` with appropriate TP\u002FPP sizing.\n- MoE models with expert parallelism: `fsdp2` with `ep_size > 1` (creates a separate `moe_mesh`).\n- Large models (70B+): `fsdp2` with PP + TP.\n- Long sequences (8K+): add CP (`cp_size > 1`).\n\nWhen answering strategy-selection questions, state the chosen `distributed.strategy`\nfirst, then enumerate the YAML fields the user must set.\n\nQuick TP + PP answer:\n\n- Use `strategy: fsdp2`; do not use `megatron_fsdp` when pipeline parallelism is required.\n- Set `tp_size` for tensor parallelism and `pp_size` for pipeline parallelism.\n- Add a `pipeline:` sub-config with `pp_schedule` and `pp_microbatch_size`.\n- Leave `dp_size` unset or `none`; it is inferred as `world_size \u002F (tp_size * pp_size * cp_size)`.\n- Keep TP inside a fast intra-node domain when possible, and use PP across model depth for 70B+ models.\n\nQuick MoE expert-parallel answer:\n\n- Start with `strategy: fsdp2` and `ep_size > 1`.\n- Include a `moe:` sub-config only when `ep_size > 1`; it maps to `MoEParallelizerConfig`.\n- Expect a separate `moe_mesh` for expert parallelism in addition to the main `device_mesh`.\n- Do not recommend `megatron_fsdp` or `ddp` for expert parallelism; `megatron_fsdp` has no EP support.\n- Before finishing an MoE EP answer, explicitly state that `ep_size` must divide `dp_size * cp_size` and that `megatron_fsdp` does not support EP, PP, or `sequence_parallel`.\n\n## YAML Config Structure\n\nThe `distributed` section in the recipe YAML maps directly to\n`parse_distributed_section()` in `recipes\u002F_dist_utils.py`:\n\n```yaml\ndistributed:\n  strategy: fsdp2           # fsdp2 | megatron_fsdp | ddp\n  dp_size: none             # auto-calculated from world_size \u002F (tp * pp * cp)\n  dp_replicate_size: none   # FSDP2-only, for HSDP\n  tp_size: 1\n  pp_size: 1\n  cp_size: 1\n  ep_size: 1\n\n  # Strategy-specific flags (forwarded to the strategy dataclass):\n  sequence_parallel: false\n  activation_checkpointing: false\n  defer_fsdp_grad_sync: true   # FSDP2 only\n\n  # Sub-configs (optional):\n  pipeline:\n    pp_schedule: 1f1b\n    pp_microbatch_size: 1\n    # ... see PipelineConfig fields\n\n  moe:\n    reshard_after_forward: false\n    # ... see MoEParallelizerConfig fields\n```\n\nThe `dp_size` is always inferred:\n\n```\ndp_size = world_size \u002F (tp_size * pp_size * cp_size)\n```\n\n## Infrastructure Flow\n\n```\ninitialize_distributed()                       [components\u002Fdistributed\u002Finit_utils.py]\n    -> initializes torch.distributed process group and returns DistInfo\nYAML distributed section + DistInfo.world_size\n    -> parse_distributed_section()          [recipes\u002F_dist_utils.py]\n    -> create_distributed_setup_from_config()              [recipes\u002F_dist_utils.py]\n        -> DistributedSetup.build()         [components\u002Fdistributed\u002Fconfig.py]\n    -> instantiate_infrastructure()         [_transformers\u002Finfrastructure.py]\n        -> _instantiate_distributed()       -> FSDP2Manager \u002F MegatronFSDPManager \u002F DDPManager\n        -> _instantiate_pipeline()          -> AutoPipeline (if pp_size > 1)\n        -> parallelize_fn                   -> MoE parallelizer (if ep_size > 1) or PP wrapper\n    -> apply_model_infrastructure()         [_transformers\u002Finfrastructure.py]\n        -> _shard_pp() or _shard_ep_fsdp()  (applies sharding to the model)\n```\n\n## FSDP2 Configuration\n\n### Basic FSDP2 (data parallelism only)\n\n```yaml\ndistributed:\n  strategy: fsdp2\n  tp_size: 1\n  cp_size: 1\n```\n\nThis auto-calculates `dp_size = world_size` and applies `fully_shard()` per\ntransformer block via DTensor-based sharding.\n\n### FSDP2 with Tensor Parallelism\n\nKeep TP within a single NVLink domain (typically one node):\n\n```yaml\ndistributed:\n  strategy: fsdp2\n  tp_size: 4        # 2, 4, or 8 -- must divide GPUs per node\n  sequence_parallel: true\n```\n\nThe TP plan is auto-selected based on the model type. Pass a custom plan via\nthe Python API if needed:\n\n```python\nconfig = FSDP2Config(sequence_parallel=True, tp_plan=my_custom_plan)\n```\n\n### FSDP2 with Pipeline Parallelism\n\n```yaml\ndistributed:\n  strategy: fsdp2\n  pp_size: 2\n  pipeline:\n    pp_schedule: interleaved1f1b   # 1f1b, gpipe, interleaved_1f1b, etc.\n    pp_microbatch_size: 4\n    scale_grads_in_schedule: false\n```\n\nThe model must have a `_pp_plan` attribute (set on the HF model class) for\n`AutoPipeline` to know how to split layers across stages. Models without\n`_pp_plan` are not compatible with PP.\n\n### FSDP2 with HSDP (Hybrid Sharded Data Parallel)\n\nIntra-node full sharding + inter-node replication via a 2D DeviceMesh:\n\n```yaml\ndistributed:\n  strategy: fsdp2\n  dp_replicate_size: 2   # must divide dp_size\n```\n\nConstraint: `dp_replicate_size \u003C dp_size` (pure replication with no sharding\nis not supported by FSDP2).\n\n### Activation Checkpointing\n\nTrades compute for memory by recomputing activations during backward:\n\n```yaml\ndistributed:\n  activation_checkpointing: true\n```\n\nThis is a model-build\u002Ftraining behavior flag, not mesh topology. Dense\nstrategies read it from the strategy config; EP\u002FMoE paths pass the recipe-level\nflag directly into model infrastructure.\n\n### Gradient Sync Deferral\n\nFSDP2 defers gradient sync to the final micro-batch by default for\ncommunication overlap:\n\n```yaml\ndistributed:\n  defer_fsdp_grad_sync: true   # default\n```\n\n### Mixed Precision\n\nFSDP2Config defaults to bfloat16 for all three precision knobs via\n`MixedPrecisionPolicy(param_dtype=bf16, reduce_dtype=bf16, output_dtype=bf16,\ncast_forward_inputs=True)`. Override via the Python API:\n\n```python\nfrom torch.distributed.fsdp import MixedPrecisionPolicy\nconfig = FSDP2Config(\n    mp_policy=MixedPrecisionPolicy(param_dtype=torch.float16, reduce_dtype=torch.float32),\n)\n```\n\n## Pipeline Parallelism\n\n### Requirements\n\n1. Model class must define `_pp_plan` (a dict mapping module FQNs to stages).\n2. `pp_size > 1` in the distributed section.\n3. A `pipeline` sub-config with schedule and microbatch size.\n\n### Supported schedules\n\nDefined in `PipelineConfig.pp_schedule`:\n\n- `1f1b` (one-forward-one-backward, default)\n- `gpipe`\n- `interleaved_1f1b` \u002F `interleaved1f1b`\n- `looped_bfs`\n- `dfs`\n- `v_schedule`\n- `zero_bubble`\n\n### Example (8B model on 8 GPUs, PP=2 + DP=4)\n\n```yaml\ndistributed:\n  strategy: fsdp2\n  pp_size: 2\n\n  pipeline:\n    pp_schedule: interleaved1f1b\n    pp_microbatch_size: 4\n    scale_grads_in_schedule: false\n\ncheckpoint:\n  model_save_format: safetensors\n  save_consolidated: final\n```\n\n### How it works\n\n`AutoPipeline.build()` calls `pipeline_model()` which splits the model into\nstages using the model's `_pp_plan`, creates `PipelineStage` objects, and\nbuilds the schedule. During training, `schedule.step()` drives forward and\nbackward through the pipeline.\n\n## Context Parallelism\n\nUse CP for long sequences (8K+). CP shards Q\u002FK\u002FV on the sequence dimension\nas DTensors.\n\n### Config\n\n```yaml\ndistributed:\n  strategy: fsdp2\n  cp_size: 2   # or 4, 8\n```\n\n### Requirements\n\n- SDPA (Flash Attention or Efficient Attention backend) or Transformer Engine\n  attention. SDPBackend.MATH is not compatible with DTensor.\n- Attention masks are automatically stripped; `is_causal=True` is set via\n  forward pre-hooks registered by `attach_context_parallel_hooks()`.\n\n### How it works\n\n1. After model sharding, `apply_model_infrastructure()` calls\n   `attach_context_parallel_hooks()` on each model part (for non-TE models).\n2. At each training step, `make_cp_batch_and_ctx()` creates a CP context\n   manager that shards the batch along the sequence dimension and sets up\n   `context_parallel()` from `torch.distributed.tensor.experimental`.\n3. For TE attention models, `make_cp_batch_for_te()` uses THD format and\n   TE's `thd_get_partitioned_indices` for sharding.\n\n### CP with Sequence Packing\n\nCP works with packed sequences. The `packed_sequence_size` must be divisible\nby `cp_size`. When using TE, chunks are sharded per-chunk via\n`_shard_thd_chunk_for_te()`.\n\n## Sequence Packing\n\nPacking multiple sequences into a single training sample for efficiency.\n\n### Config\n\n```yaml\npacked_sequence:\n  packed_sequence_size: 4096   # 0 = disabled\n\nstep_scheduler:\n  local_batch_size: 1          # must be 1 for packed sequences\n```\n\nWhen `packed_sequence_size > 0`, the dataset collator packs sequences up to\nthat length. `local_batch_size` must be 1 because each \"sample\" is already a\npacked batch.\n\n## MoE Distributed Training\n\n### Expert Parallelism\n\nSet `ep_size > 1` to distribute experts across GPUs. This creates a separate\n`moe_mesh` alongside the main `device_mesh`:\n\n```yaml\ndistributed:\n  strategy: fsdp2\n  ep_size: 8\n  activation_checkpointing: true\n```\n\nThe `moe_mesh` shape is `(pp_size, ep_shard_size, ep_size)` with dimension\nnames `(\"pp\", \"ep_shard\", \"ep\")`.\n\nConstraint: `dp_cp_size` (= `dp_size * cp_size`) must be divisible by\n`ep_size`.\n\n### MoE sub-config\n\n```yaml\ndistributed:\n  strategy: fsdp2\n  ep_size: 8\n  activation_checkpointing: true\n\n  moe:\n    reshard_after_forward: false\n    ignore_router_for_ac: false\n    wrap_outer_model: true\n```\n\nThe `moe` sub-section maps to `MoEParallelizerConfig` and is only\ninstantiated when `ep_size > 1`.\n\n### Full MoE example (Qwen3-30B-A3B on 8 GPUs)\n\n```yaml\ndistributed:\n  strategy: fsdp2\n  tp_size: 1\n  cp_size: 1\n  pp_size: 1\n  ep_size: 8\n  sequence_parallel: false\n  activation_checkpointing: true\n```\n\n### MegatronFSDP limitations\n\nDespite its name, `megatron_fsdp` does **not** support expert parallelism\n(`ep_size > 1`), pipeline parallelism (`pp_size > 1`), or\n`sequence_parallel`. Use `fsdp2` for these features.\n\n## Parallelism Sizing Guidelines\n\n### Dense models\n\n| Model size | TP | PP | CP | Strategy |\n|---|---|---|---|---|\n| \u003C 3B | 1 | 1 | 1 | FSDP2 (DP only) |\n| 3-13B | 2-4 | 1 | 1 | FSDP2 + TP |\n| 13-70B | 4-8 | 2-4 | 1 | FSDP2 + TP + PP |\n| 70B+ | 8 | 4-8 | 1 | FSDP2 + TP + PP |\n| Any + long seq (8K+) | as above | as above | 2-8 | add CP |\n\n### MoE models\n\nMoE models need less TP than dense models of similar total parameter count\nbecause only a fraction of parameters are active per token. EP is the primary\nscaling dimension:\n\n| Model | TP | PP | EP | Notes |\n|---|---|---|---|---|\n| Small MoE (\u003C10B total) | 1 | 1 | 8 | EP only |\n| Medium MoE (10-30B total) | 1-2 | 1 | 8 | small TP for shared layers |\n| Large MoE (100B+ total) | 1-2 | 4+ | 8-64 | PP for depth, EP for experts |\n\n### Hardware topology rules\n\n- TP must stay within a single NVLink domain (one node, typically 8 GPUs).\n- Use PP or DP for cross-node scaling.\n- TP across InfiniBand degrades throughput severely.\n\n## Programmatic API (from_pretrained \u002F from_config)\n\nWhen not using YAML recipes, configure distributed training via Python:\n\n```python\nfrom nemo_automodel.components.distributed import (\n    DistributedSetup,\n    FSDP2Config,\n    ParallelismSizes,\n    initialize_distributed,\n)\n\ndist_env = initialize_distributed(\"nccl\")\ndistributed_setup = DistributedSetup.build(\n    strategy=FSDP2Config(sequence_parallel=True),\n    parallelism_sizes=ParallelismSizes(tp_size=2),\n    activation_checkpointing=True,\n    world_size=dist_env.world_size,\n)\n```\n\nOr pass directly to `from_pretrained`:\n\n```python\nfrom nemo_automodel import NeMoAutoModelForCausalLM\n\nmodel = NeMoAutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.2-1B\",\n    distributed_setup=distributed_setup,\n)\n```\n\n## Code Anchors\n\nStrategy config dataclasses:\n\n```\ncomponents\u002Fdistributed\u002Fconfig.py\n    FSDP2Config       -- sequence_parallel, tp_plan, mp_policy, offload_policy,\n                         activation_checkpointing, defer_fsdp_grad_sync\n    MegatronFSDPConfig -- zero_dp_strategy, overlap_grad_reduce, overlap_param_gather, etc.\n    DDPConfig          -- activation_checkpointing only\n```\n\nMeshContext (single source of truth for parallelism):\n\n```\ncomponents\u002Fdistributed\u002Fmesh.py\n    MeshContext  -- device_mesh, moe_mesh\n                    Properties: tp_size, pp_size, cp_size, ep_size, dp_size, dp_replicate_size\n    MeshAxisName -- PP, DP, DP_REPLICATE, DP_SHARD, DP_SHARD_CP, DP_CP, CP, TP, EP, EP_SHARD\n```\n\nMesh context and raw mesh creation:\n\n```\ncomponents\u002Fdistributed\u002Fconfig.py\n    DistributedSetup.build()      -- builds MeshContext from strategy + parallelism\ncomponents\u002Fdistributed\u002Fmesh_utils.py\n    _create_device_meshes()       -- routes to FSDP2\u002FMegatronFSDP\u002FDDP raw mesh creation\n    _create_fsdp2_device_mesh()   -- shape (pp, dp_replicate, dp_shard, cp, tp) + flattened submeshes\n    _create_megatron_fsdp_device_mesh() -- shape (dp, cp, tp)\n```\n\nDistributed managers:\n\n```\ncomponents\u002Fdistributed\u002Ffsdp2.py          -- FSDP2Manager.parallelize()\ncomponents\u002Fdistributed\u002Fmegatron_fsdp.py  -- MegatronFSDPManager.parallelize()\ncomponents\u002Fdistributed\u002Fddp.py            -- DDPManager\n```\n\nPipeline parallelism:\n\n```\ncomponents\u002Fdistributed\u002Fpipelining\u002Fconfig.py        -- PipelineConfig dataclass\ncomponents\u002Fdistributed\u002Fpipelining\u002Fautopipeline.py  -- AutoPipeline orchestrator\ncomponents\u002Fdistributed\u002Fpipelining\u002Ffunctional.py    -- pipeline_model(), schedule creation\ncomponents\u002Fdistributed\u002Fpipelining\u002Fhf_utils.py      -- HF model validation for PP\n```\n\nContext parallelism:\n\n```\ncomponents\u002Fdistributed\u002Fcp_utils.py\n    make_cp_batch_and_ctx()            -- creates CP context manager + shards batch\n    create_context_parallel_ctx()      -- wraps torch.distributed.tensor.experimental.context_parallel\n    attach_context_parallel_hooks()    -- strips attention_mask, sets is_causal=True\n    make_cp_batch_for_te()             -- TE-specific CP batch sharding (THD format)\n```\n\nInfrastructure orchestration:\n\n```\n_transformers\u002Finfrastructure.py\n    instantiate_infrastructure()    -- config objects -> runtime objects\n    apply_model_infrastructure()    -- applies sharding, PEFT, checkpoints to model\n    _shard_pp()                     -- pipeline parallel path\n    _shard_ep_fsdp()                -- EP + FSDP path (non-PP)\n```\n\nYAML parsing:\n\n```\nrecipes\u002F_dist_utils.py\n    parse_distributed_section()  -- YAML dict -> typed configs + sizes\n    create_distributed_setup_from_config()  -- recipe adapter: parse + create DistributedSetup; does not init process group\n```\n\nMoE config:\n\n```\ncomponents\u002Fdistributed\u002Fconfig.py\n    MoEParallelizerConfig  -- reshard_after_forward, ignore_router_for_ac, wrap_outer_model, etc.\ncomponents\u002Fmoe\u002Fconfig.py\n    MoEConfig              -- n_routed_experts, n_activated_experts, score_func, etc.\n```\n\n## Pitfalls\n\n1. **TP across nodes destroys throughput.** Always keep TP within a single\n   NVLink domain. Use PP or DP for cross-node scaling.\n\n2. **PP requires `_pp_plan` on the model class.** Not all HF models have this.\n   Check `validate_hf_model_for_pipeline_support()` before enabling PP.\n\n3. **PP bubbles reduce GPU utilization.** Use interleaved schedules\n   (`interleaved_1f1b`) and smaller microbatches to reduce bubble time.\n\n4. **FSDP2 requires DTensor-aware state dict saving.** Use `safetensors` with\n   `save_consolidated: final` for final HF export, or `save_consolidated: false`\n   plus the generated `model\u002Fconsolidate.sh` helper for offline export.\n\n5. **CP requires compatible attention.** SDPA (Flash Attention or Efficient\n   Attention) or TE attention only. `SDPBackend.MATH` is not compatible with\n   DTensor.\n\n6. **MoE EP size must evenly divide `dp_size * cp_size`.** The device mesh\n   creation asserts `dp_cp_size % ep_size == 0`.\n\n7. **MegatronFSDP is more limited than FSDP2.** It does not support PP\n   (`pp_size > 1`), EP (`ep_size > 1`), or `sequence_parallel`. The\n   `MeshContext` validation raises on these combinations.\n\n8. **DDP supports nothing beyond data parallelism.** No TP, PP, CP, EP, or\n   HSDP. Validation raises on any of these.\n\n9. **Activation checkpointing increases compute.** It saves memory by\n   recomputing activations during backward, but adds ~30% compute overhead.\n\n10. **Mixed precision policy must match model expectations.** The default\n    bfloat16 policy works for most models. FP16 models may need a custom\n    `MixedPrecisionPolicy`.\n\n11. **`packed_sequence_size` must be divisible by `cp_size`** when using CP\n    with packed sequences.\n\n12. **`dp_replicate_size` is FSDP2-only.** Passing it with `megatron_fsdp`\n    or `ddp` raises a `ValueError`.\n\n## Verification\n\nRun the smallest recipe that exercises the requested strategy. Success means\nexit code 0, finite loss, no NCCL timeout, and log output matching the expected\nTP\u002FPP\u002FCP\u002FEP sizes.\n",{"data":32,"body":38},{"name":4,"description":6,"when_to_use":33,"license":23,"metadata":34},"Adding or modifying distributed training strategies (FSDP2, HSDP, DDP), debugging multi-GPU or multi-node failures, configuring context or tensor parallelism, or tuning sharding settings.",{"author":9,"tags":35},[36,37],"nemo-automodel","distributed-training",{"type":39,"children":40},"root",[41,50,57,72,78,83,88,94,101,169,338,344,409,487,493,521,527,540,636,641,731,743,748,848,853,976,982,1009,1343,1354,1364,1370,1379,1385,1391,1454,1475,1481,1486,1556,1561,1577,1583,1695,1723,1729,1734,1788,1801,1807,1812,1845,1850,1856,1861,1899,1905,1918,1957,1963,1969,2008,2014,2026,2102,2108,2272,2278,2320,2326,2331,2337,2390,2395,2423,2428,2500,2506,2533,2539,2544,2549,2631,2652,2658,2664,2688,2751,2777,2802,2808,2936,2960,2966,3089,3094,3142,3148,3154,3320,3326,3331,3444,3450,3468,3474,3479,3595,3607,3660,3666,3671,3680,3685,3694,3699,3708,3713,3722,3727,3736,3741,3750,3755,3764,3769,3778,3783,3792,3798,4068,4074,4079],{"type":42,"tag":43,"props":44,"children":46},"element","h1",{"id":45},"distributed-training-in-nemo-automodel",[47],{"type":48,"value":49},"text","Distributed Training in NeMo AutoModel",{"type":42,"tag":51,"props":52,"children":54},"h2",{"id":53},"purpose",[55],{"type":48,"value":56},"Purpose",{"type":42,"tag":58,"props":59,"children":60},"p",{},[61,63,70],{"type":48,"value":62},"NeMo AutoModel uses PyTorch-native distributed training.\nAll parallelism is orchestrated through a single ",{"type":42,"tag":64,"props":65,"children":67},"code",{"className":66},[],[68],{"type":48,"value":69},"MeshContext",{"type":48,"value":71}," object that\nholds device meshes, strategy configs, and axis names.",{"type":42,"tag":51,"props":73,"children":75},{"id":74},"instructions",[76],{"type":48,"value":77},"Instructions",{"type":42,"tag":58,"props":79,"children":80},{},[81],{"type":48,"value":82},"For conceptual distributed-training questions, answer directly from the quick\npatterns in this skill without inspecting the repository. Start with the\nstrategy choice, then list only the YAML fields and constraints relevant to the\nquestion.",{"type":42,"tag":58,"props":84,"children":85},{},[86],{"type":48,"value":87},"Use direct action verbs in the final answer: recommend the strategy, show the\nminimal YAML, state the sizing constraint, and name the unsupported strategies.\nDo not discuss model onboarding, recipes, Slurm, SkyPilot, or checkpointing\nunless the user asks.",{"type":42,"tag":51,"props":89,"children":91},{"id":90},"examples",[92],{"type":48,"value":93},"Examples",{"type":42,"tag":95,"props":96,"children":98},"h3",{"id":97},"tp-plus-pp-for-a-large-multi-node-model",[99],{"type":48,"value":100},"TP plus PP for a large multi-node model",{"type":42,"tag":58,"props":102,"children":103},{},[104,106,112,114,120,122,128,129,135,137,143,145,151,153,159,161,167],{"type":48,"value":105},"Recommend ",{"type":42,"tag":64,"props":107,"children":109},{"className":108},[],[110],{"type":48,"value":111},"strategy: fsdp2",{"type":48,"value":113},". Mention ",{"type":42,"tag":64,"props":115,"children":117},{"className":116},[],[118],{"type":48,"value":119},"tp_size",{"type":48,"value":121},", ",{"type":42,"tag":64,"props":123,"children":125},{"className":124},[],[126],{"type":48,"value":127},"pp_size",{"type":48,"value":121},{"type":42,"tag":64,"props":130,"children":132},{"className":131},[],[133],{"type":48,"value":134},"cp_size",{"type":48,"value":136},",\n",{"type":42,"tag":64,"props":138,"children":140},{"className":139},[],[141],{"type":48,"value":142},"ep_size",{"type":48,"value":144},", and the ",{"type":42,"tag":64,"props":146,"children":148},{"className":147},[],[149],{"type":48,"value":150},"pipeline",{"type":48,"value":152}," sub-config. State that ",{"type":42,"tag":64,"props":154,"children":156},{"className":155},[],[157],{"type":48,"value":158},"dp_size",{"type":48,"value":160}," is inferred from\n",{"type":42,"tag":64,"props":162,"children":164},{"className":163},[],[165],{"type":48,"value":166},"world_size \u002F (tp_size * pp_size * cp_size)",{"type":48,"value":168},".",{"type":42,"tag":170,"props":171,"children":176},"pre",{"className":172,"code":173,"language":174,"meta":175,"style":175},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","distributed:\n  strategy: fsdp2\n  tp_size: 8\n  pp_size: 4\n  cp_size: 1\n  ep_size: 1\n  pipeline:\n    pp_schedule: interleaved1f1b\n    pp_microbatch_size: 1\n","yaml","",[177],{"type":42,"tag":64,"props":178,"children":179},{"__ignoreMap":175},[180,198,218,237,255,273,290,303,321],{"type":42,"tag":181,"props":182,"children":185},"span",{"class":183,"line":184},"line",1,[186,192],{"type":42,"tag":181,"props":187,"children":189},{"style":188},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[190],{"type":48,"value":191},"distributed",{"type":42,"tag":181,"props":193,"children":195},{"style":194},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[196],{"type":48,"value":197},":\n",{"type":42,"tag":181,"props":199,"children":201},{"class":183,"line":200},2,[202,207,212],{"type":42,"tag":181,"props":203,"children":204},{"style":188},[205],{"type":48,"value":206},"  strategy",{"type":42,"tag":181,"props":208,"children":209},{"style":194},[210],{"type":48,"value":211},":",{"type":42,"tag":181,"props":213,"children":215},{"style":214},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[216],{"type":48,"value":217}," fsdp2\n",{"type":42,"tag":181,"props":219,"children":221},{"class":183,"line":220},3,[222,227,231],{"type":42,"tag":181,"props":223,"children":224},{"style":188},[225],{"type":48,"value":226},"  tp_size",{"type":42,"tag":181,"props":228,"children":229},{"style":194},[230],{"type":48,"value":211},{"type":42,"tag":181,"props":232,"children":234},{"style":233},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[235],{"type":48,"value":236}," 8\n",{"type":42,"tag":181,"props":238,"children":240},{"class":183,"line":239},4,[241,246,250],{"type":42,"tag":181,"props":242,"children":243},{"style":188},[244],{"type":48,"value":245},"  pp_size",{"type":42,"tag":181,"props":247,"children":248},{"style":194},[249],{"type":48,"value":211},{"type":42,"tag":181,"props":251,"children":252},{"style":233},[253],{"type":48,"value":254}," 4\n",{"type":42,"tag":181,"props":256,"children":258},{"class":183,"line":257},5,[259,264,268],{"type":42,"tag":181,"props":260,"children":261},{"style":188},[262],{"type":48,"value":263},"  cp_size",{"type":42,"tag":181,"props":265,"children":266},{"style":194},[267],{"type":48,"value":211},{"type":42,"tag":181,"props":269,"children":270},{"style":233},[271],{"type":48,"value":272}," 1\n",{"type":42,"tag":181,"props":274,"children":276},{"class":183,"line":275},6,[277,282,286],{"type":42,"tag":181,"props":278,"children":279},{"style":188},[280],{"type":48,"value":281},"  ep_size",{"type":42,"tag":181,"props":283,"children":284},{"style":194},[285],{"type":48,"value":211},{"type":42,"tag":181,"props":287,"children":288},{"style":233},[289],{"type":48,"value":272},{"type":42,"tag":181,"props":291,"children":293},{"class":183,"line":292},7,[294,299],{"type":42,"tag":181,"props":295,"children":296},{"style":188},[297],{"type":48,"value":298},"  pipeline",{"type":42,"tag":181,"props":300,"children":301},{"style":194},[302],{"type":48,"value":197},{"type":42,"tag":181,"props":304,"children":306},{"class":183,"line":305},8,[307,312,316],{"type":42,"tag":181,"props":308,"children":309},{"style":188},[310],{"type":48,"value":311},"    pp_schedule",{"type":42,"tag":181,"props":313,"children":314},{"style":194},[315],{"type":48,"value":211},{"type":42,"tag":181,"props":317,"children":318},{"style":214},[319],{"type":48,"value":320}," interleaved1f1b\n",{"type":42,"tag":181,"props":322,"children":324},{"class":183,"line":323},9,[325,330,334],{"type":42,"tag":181,"props":326,"children":327},{"style":188},[328],{"type":48,"value":329},"    pp_microbatch_size",{"type":42,"tag":181,"props":331,"children":332},{"style":194},[333],{"type":48,"value":211},{"type":42,"tag":181,"props":335,"children":336},{"style":233},[337],{"type":48,"value":272},{"type":42,"tag":95,"props":339,"children":341},{"id":340},"moe-expert-parallelism",[342],{"type":48,"value":343},"MoE expert parallelism",{"type":42,"tag":58,"props":345,"children":346},{},[347,348,353,355,361,363,369,371,377,379,384,386,392,394,400,402,408],{"type":48,"value":105},{"type":42,"tag":64,"props":349,"children":351},{"className":350},[],[352],{"type":48,"value":111},{"type":48,"value":354}," with ",{"type":42,"tag":64,"props":356,"children":358},{"className":357},[],[359],{"type":48,"value":360},"ep_size > 1",{"type":48,"value":362},". Say this creates a separate\n",{"type":42,"tag":64,"props":364,"children":366},{"className":365},[],[367],{"type":48,"value":368},"moe_mesh",{"type":48,"value":370},"; include the ",{"type":42,"tag":64,"props":372,"children":374},{"className":373},[],[375],{"type":48,"value":376},"moe",{"type":48,"value":378}," sub-config when relevant; state that ",{"type":42,"tag":64,"props":380,"children":382},{"className":381},[],[383],{"type":48,"value":142},{"type":48,"value":385},"\nmust divide ",{"type":42,"tag":64,"props":387,"children":389},{"className":388},[],[390],{"type":48,"value":391},"dp_size * cp_size",{"type":48,"value":393},". Do not recommend ",{"type":42,"tag":64,"props":395,"children":397},{"className":396},[],[398],{"type":48,"value":399},"megatron_fsdp",{"type":48,"value":401}," or ",{"type":42,"tag":64,"props":403,"children":405},{"className":404},[],[406],{"type":48,"value":407},"ddp",{"type":48,"value":168},{"type":42,"tag":170,"props":410,"children":412},{"className":172,"code":411,"language":174,"meta":175,"style":175},"distributed:\n  strategy: fsdp2\n  ep_size: 8\n  moe:\n    reshard_after_forward: false\n",[413],{"type":42,"tag":64,"props":414,"children":415},{"__ignoreMap":175},[416,427,442,457,469],{"type":42,"tag":181,"props":417,"children":418},{"class":183,"line":184},[419,423],{"type":42,"tag":181,"props":420,"children":421},{"style":188},[422],{"type":48,"value":191},{"type":42,"tag":181,"props":424,"children":425},{"style":194},[426],{"type":48,"value":197},{"type":42,"tag":181,"props":428,"children":429},{"class":183,"line":200},[430,434,438],{"type":42,"tag":181,"props":431,"children":432},{"style":188},[433],{"type":48,"value":206},{"type":42,"tag":181,"props":435,"children":436},{"style":194},[437],{"type":48,"value":211},{"type":42,"tag":181,"props":439,"children":440},{"style":214},[441],{"type":48,"value":217},{"type":42,"tag":181,"props":443,"children":444},{"class":183,"line":220},[445,449,453],{"type":42,"tag":181,"props":446,"children":447},{"style":188},[448],{"type":48,"value":281},{"type":42,"tag":181,"props":450,"children":451},{"style":194},[452],{"type":48,"value":211},{"type":42,"tag":181,"props":454,"children":455},{"style":233},[456],{"type":48,"value":236},{"type":42,"tag":181,"props":458,"children":459},{"class":183,"line":239},[460,465],{"type":42,"tag":181,"props":461,"children":462},{"style":188},[463],{"type":48,"value":464},"  moe",{"type":42,"tag":181,"props":466,"children":467},{"style":194},[468],{"type":48,"value":197},{"type":42,"tag":181,"props":470,"children":471},{"class":183,"line":257},[472,477,481],{"type":42,"tag":181,"props":473,"children":474},{"style":188},[475],{"type":48,"value":476},"    reshard_after_forward",{"type":42,"tag":181,"props":478,"children":479},{"style":194},[480],{"type":48,"value":211},{"type":42,"tag":181,"props":482,"children":484},{"style":483},"--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC",[485],{"type":48,"value":486}," false\n",{"type":42,"tag":95,"props":488,"children":490},{"id":489},"megatronfsdp-limitations",[491],{"type":48,"value":492},"MegatronFSDP limitations",{"type":42,"tag":58,"props":494,"children":495},{},[496,498,504,506,512,514,519],{"type":48,"value":497},"Say no for pipeline parallelism, expert parallelism, and ",{"type":42,"tag":64,"props":499,"children":501},{"className":500},[],[502],{"type":48,"value":503},"sequence_parallel",{"type":48,"value":505},".\nRecommend ",{"type":42,"tag":64,"props":507,"children":509},{"className":508},[],[510],{"type":48,"value":511},"fsdp2",{"type":48,"value":513}," for PP, EP, or ",{"type":42,"tag":64,"props":515,"children":517},{"className":516},[],[518],{"type":48,"value":503},{"type":48,"value":520},"; mention that DDP is only\nsimple data parallelism.",{"type":42,"tag":51,"props":522,"children":524},{"id":523},"strategy-selection",[525],{"type":48,"value":526},"Strategy Selection",{"type":42,"tag":58,"props":528,"children":529},{},[530,532,538],{"type":48,"value":531},"Three strategies are available, selected via the ",{"type":42,"tag":64,"props":533,"children":535},{"className":534},[],[536],{"type":48,"value":537},"distributed.strategy",{"type":48,"value":539}," YAML key:",{"type":42,"tag":541,"props":542,"children":543},"table",{},[544,568],{"type":42,"tag":545,"props":546,"children":547},"thead",{},[548],{"type":42,"tag":549,"props":550,"children":551},"tr",{},[552,558,563],{"type":42,"tag":553,"props":554,"children":555},"th",{},[556],{"type":48,"value":557},"Strategy",{"type":42,"tag":553,"props":559,"children":560},{},[561],{"type":48,"value":562},"YAML value",{"type":42,"tag":553,"props":564,"children":565},{},[566],{"type":48,"value":567},"Best for",{"type":42,"tag":569,"props":570,"children":571},"tbody",{},[572,594,615],{"type":42,"tag":549,"props":573,"children":574},{},[575,581,589],{"type":42,"tag":576,"props":577,"children":578},"td",{},[579],{"type":48,"value":580},"FSDP2",{"type":42,"tag":576,"props":582,"children":583},{},[584],{"type":42,"tag":64,"props":585,"children":587},{"className":586},[],[588],{"type":48,"value":511},{"type":42,"tag":576,"props":590,"children":591},{},[592],{"type":48,"value":593},"General use, recommended default. Supports TP, PP, CP, EP, HSDP.",{"type":42,"tag":549,"props":595,"children":596},{},[597,602,610],{"type":42,"tag":576,"props":598,"children":599},{},[600],{"type":48,"value":601},"MegatronFSDP",{"type":42,"tag":576,"props":603,"children":604},{},[605],{"type":42,"tag":64,"props":606,"children":608},{"className":607},[],[609],{"type":48,"value":399},{"type":42,"tag":576,"props":611,"children":612},{},[613],{"type":48,"value":614},"NVIDIA Megatron-style FSDP. No PP, no EP, no sequence_parallel.",{"type":42,"tag":549,"props":616,"children":617},{},[618,623,631],{"type":42,"tag":576,"props":619,"children":620},{},[621],{"type":48,"value":622},"DDP",{"type":42,"tag":576,"props":624,"children":625},{},[626],{"type":42,"tag":64,"props":627,"children":629},{"className":628},[],[630],{"type":48,"value":407},{"type":42,"tag":576,"props":632,"children":633},{},[634],{"type":48,"value":635},"Simple data parallelism only. No TP, PP, CP, or EP.",{"type":42,"tag":58,"props":637,"children":638},{},[639],{"type":48,"value":640},"Decision tree:",{"type":42,"tag":642,"props":643,"children":644},"ul",{},[645,651,670,682,707,719],{"type":42,"tag":646,"props":647,"children":648},"li",{},[649],{"type":48,"value":650},"Single GPU: no distributed config needed (FSDP2Manager skips parallelization when world_size=1).",{"type":42,"tag":646,"props":652,"children":653},{},[654,656,661,663,668],{"type":48,"value":655},"Multi-GPU single node: ",{"type":42,"tag":64,"props":657,"children":659},{"className":658},[],[660],{"type":48,"value":511},{"type":48,"value":662}," (default). Use ",{"type":42,"tag":64,"props":664,"children":666},{"className":665},[],[667],{"type":48,"value":407},{"type":48,"value":669}," only if you need the simplest possible setup.",{"type":42,"tag":646,"props":671,"children":672},{},[673,675,680],{"type":48,"value":674},"Multi-node: ",{"type":42,"tag":64,"props":676,"children":678},{"className":677},[],[679],{"type":48,"value":511},{"type":48,"value":681}," with appropriate TP\u002FPP sizing.",{"type":42,"tag":646,"props":683,"children":684},{},[685,687,692,693,698,700,705],{"type":48,"value":686},"MoE models with expert parallelism: ",{"type":42,"tag":64,"props":688,"children":690},{"className":689},[],[691],{"type":48,"value":511},{"type":48,"value":354},{"type":42,"tag":64,"props":694,"children":696},{"className":695},[],[697],{"type":48,"value":360},{"type":48,"value":699}," (creates a separate ",{"type":42,"tag":64,"props":701,"children":703},{"className":702},[],[704],{"type":48,"value":368},{"type":48,"value":706},").",{"type":42,"tag":646,"props":708,"children":709},{},[710,712,717],{"type":48,"value":711},"Large models (70B+): ",{"type":42,"tag":64,"props":713,"children":715},{"className":714},[],[716],{"type":48,"value":511},{"type":48,"value":718}," with PP + TP.",{"type":42,"tag":646,"props":720,"children":721},{},[722,724,730],{"type":48,"value":723},"Long sequences (8K+): add CP (",{"type":42,"tag":64,"props":725,"children":727},{"className":726},[],[728],{"type":48,"value":729},"cp_size > 1",{"type":48,"value":706},{"type":42,"tag":58,"props":732,"children":733},{},[734,736,741],{"type":48,"value":735},"When answering strategy-selection questions, state the chosen ",{"type":42,"tag":64,"props":737,"children":739},{"className":738},[],[740],{"type":48,"value":537},{"type":48,"value":742},"\nfirst, then enumerate the YAML fields the user must set.",{"type":42,"tag":58,"props":744,"children":745},{},[746],{"type":48,"value":747},"Quick TP + PP answer:",{"type":42,"tag":642,"props":749,"children":750},{},[751,770,789,817,843],{"type":42,"tag":646,"props":752,"children":753},{},[754,756,761,763,768],{"type":48,"value":755},"Use ",{"type":42,"tag":64,"props":757,"children":759},{"className":758},[],[760],{"type":48,"value":111},{"type":48,"value":762},"; do not use ",{"type":42,"tag":64,"props":764,"children":766},{"className":765},[],[767],{"type":48,"value":399},{"type":48,"value":769}," when pipeline parallelism is required.",{"type":42,"tag":646,"props":771,"children":772},{},[773,775,780,782,787],{"type":48,"value":774},"Set ",{"type":42,"tag":64,"props":776,"children":778},{"className":777},[],[779],{"type":48,"value":119},{"type":48,"value":781}," for tensor parallelism and ",{"type":42,"tag":64,"props":783,"children":785},{"className":784},[],[786],{"type":48,"value":127},{"type":48,"value":788}," for pipeline parallelism.",{"type":42,"tag":646,"props":790,"children":791},{},[792,794,800,802,808,810,816],{"type":48,"value":793},"Add a ",{"type":42,"tag":64,"props":795,"children":797},{"className":796},[],[798],{"type":48,"value":799},"pipeline:",{"type":48,"value":801}," sub-config with ",{"type":42,"tag":64,"props":803,"children":805},{"className":804},[],[806],{"type":48,"value":807},"pp_schedule",{"type":48,"value":809}," and ",{"type":42,"tag":64,"props":811,"children":813},{"className":812},[],[814],{"type":48,"value":815},"pp_microbatch_size",{"type":48,"value":168},{"type":42,"tag":646,"props":818,"children":819},{},[820,822,827,829,835,837,842],{"type":48,"value":821},"Leave ",{"type":42,"tag":64,"props":823,"children":825},{"className":824},[],[826],{"type":48,"value":158},{"type":48,"value":828}," unset or ",{"type":42,"tag":64,"props":830,"children":832},{"className":831},[],[833],{"type":48,"value":834},"none",{"type":48,"value":836},"; it is inferred as ",{"type":42,"tag":64,"props":838,"children":840},{"className":839},[],[841],{"type":48,"value":166},{"type":48,"value":168},{"type":42,"tag":646,"props":844,"children":845},{},[846],{"type":48,"value":847},"Keep TP inside a fast intra-node domain when possible, and use PP across model depth for 70B+ models.",{"type":42,"tag":58,"props":849,"children":850},{},[851],{"type":48,"value":852},"Quick MoE expert-parallel answer:",{"type":42,"tag":642,"props":854,"children":855},{},[856,873,900,919,944],{"type":42,"tag":646,"props":857,"children":858},{},[859,861,866,867,872],{"type":48,"value":860},"Start with ",{"type":42,"tag":64,"props":862,"children":864},{"className":863},[],[865],{"type":48,"value":111},{"type":48,"value":809},{"type":42,"tag":64,"props":868,"children":870},{"className":869},[],[871],{"type":48,"value":360},{"type":48,"value":168},{"type":42,"tag":646,"props":874,"children":875},{},[876,878,884,886,891,893,899],{"type":48,"value":877},"Include a ",{"type":42,"tag":64,"props":879,"children":881},{"className":880},[],[882],{"type":48,"value":883},"moe:",{"type":48,"value":885}," sub-config only when ",{"type":42,"tag":64,"props":887,"children":889},{"className":888},[],[890],{"type":48,"value":360},{"type":48,"value":892},"; it maps to ",{"type":42,"tag":64,"props":894,"children":896},{"className":895},[],[897],{"type":48,"value":898},"MoEParallelizerConfig",{"type":48,"value":168},{"type":42,"tag":646,"props":901,"children":902},{},[903,905,910,912,918],{"type":48,"value":904},"Expect a separate ",{"type":42,"tag":64,"props":906,"children":908},{"className":907},[],[909],{"type":48,"value":368},{"type":48,"value":911}," for expert parallelism in addition to the main ",{"type":42,"tag":64,"props":913,"children":915},{"className":914},[],[916],{"type":48,"value":917},"device_mesh",{"type":48,"value":168},{"type":42,"tag":646,"props":920,"children":921},{},[922,924,929,930,935,937,942],{"type":48,"value":923},"Do not recommend ",{"type":42,"tag":64,"props":925,"children":927},{"className":926},[],[928],{"type":48,"value":399},{"type":48,"value":401},{"type":42,"tag":64,"props":931,"children":933},{"className":932},[],[934],{"type":48,"value":407},{"type":48,"value":936}," for expert parallelism; ",{"type":42,"tag":64,"props":938,"children":940},{"className":939},[],[941],{"type":48,"value":399},{"type":48,"value":943}," has no EP support.",{"type":42,"tag":646,"props":945,"children":946},{},[947,949,954,956,961,963,968,970,975],{"type":48,"value":948},"Before finishing an MoE EP answer, explicitly state that ",{"type":42,"tag":64,"props":950,"children":952},{"className":951},[],[953],{"type":48,"value":142},{"type":48,"value":955}," must divide ",{"type":42,"tag":64,"props":957,"children":959},{"className":958},[],[960],{"type":48,"value":391},{"type":48,"value":962}," and that ",{"type":42,"tag":64,"props":964,"children":966},{"className":965},[],[967],{"type":48,"value":399},{"type":48,"value":969}," does not support EP, PP, or ",{"type":42,"tag":64,"props":971,"children":973},{"className":972},[],[974],{"type":48,"value":503},{"type":48,"value":168},{"type":42,"tag":51,"props":977,"children":979},{"id":978},"yaml-config-structure",[980],{"type":48,"value":981},"YAML Config Structure",{"type":42,"tag":58,"props":983,"children":984},{},[985,987,992,994,1000,1002,1008],{"type":48,"value":986},"The ",{"type":42,"tag":64,"props":988,"children":990},{"className":989},[],[991],{"type":48,"value":191},{"type":48,"value":993}," section in the recipe YAML maps directly to\n",{"type":42,"tag":64,"props":995,"children":997},{"className":996},[],[998],{"type":48,"value":999},"parse_distributed_section()",{"type":48,"value":1001}," in ",{"type":42,"tag":64,"props":1003,"children":1005},{"className":1004},[],[1006],{"type":48,"value":1007},"recipes\u002F_dist_utils.py",{"type":48,"value":211},{"type":42,"tag":170,"props":1010,"children":1012},{"className":172,"code":1011,"language":174,"meta":175,"style":175},"distributed:\n  strategy: fsdp2           # fsdp2 | megatron_fsdp | ddp\n  dp_size: none             # auto-calculated from world_size \u002F (tp * pp * cp)\n  dp_replicate_size: none   # FSDP2-only, for HSDP\n  tp_size: 1\n  pp_size: 1\n  cp_size: 1\n  ep_size: 1\n\n  # Strategy-specific flags (forwarded to the strategy dataclass):\n  sequence_parallel: false\n  activation_checkpointing: false\n  defer_fsdp_grad_sync: true   # FSDP2 only\n\n  # Sub-configs (optional):\n  pipeline:\n    pp_schedule: 1f1b\n    pp_microbatch_size: 1\n    # ... see PipelineConfig fields\n\n  moe:\n    reshard_after_forward: false\n    # ... see MoEParallelizerConfig fields\n",[1013],{"type":42,"tag":64,"props":1014,"children":1015},{"__ignoreMap":175},[1016,1027,1049,1071,1092,1107,1122,1137,1152,1161,1170,1187,1204,1227,1235,1244,1256,1273,1289,1298,1306,1318,1334],{"type":42,"tag":181,"props":1017,"children":1018},{"class":183,"line":184},[1019,1023],{"type":42,"tag":181,"props":1020,"children":1021},{"style":188},[1022],{"type":48,"value":191},{"type":42,"tag":181,"props":1024,"children":1025},{"style":194},[1026],{"type":48,"value":197},{"type":42,"tag":181,"props":1028,"children":1029},{"class":183,"line":200},[1030,1034,1038,1043],{"type":42,"tag":181,"props":1031,"children":1032},{"style":188},[1033],{"type":48,"value":206},{"type":42,"tag":181,"props":1035,"children":1036},{"style":194},[1037],{"type":48,"value":211},{"type":42,"tag":181,"props":1039,"children":1040},{"style":214},[1041],{"type":48,"value":1042}," fsdp2",{"type":42,"tag":181,"props":1044,"children":1046},{"style":1045},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1047],{"type":48,"value":1048},"           # fsdp2 | megatron_fsdp | ddp\n",{"type":42,"tag":181,"props":1050,"children":1051},{"class":183,"line":220},[1052,1057,1061,1066],{"type":42,"tag":181,"props":1053,"children":1054},{"style":188},[1055],{"type":48,"value":1056},"  dp_size",{"type":42,"tag":181,"props":1058,"children":1059},{"style":194},[1060],{"type":48,"value":211},{"type":42,"tag":181,"props":1062,"children":1063},{"style":214},[1064],{"type":48,"value":1065}," none",{"type":42,"tag":181,"props":1067,"children":1068},{"style":1045},[1069],{"type":48,"value":1070},"             # auto-calculated from world_size \u002F (tp * pp * cp)\n",{"type":42,"tag":181,"props":1072,"children":1073},{"class":183,"line":239},[1074,1079,1083,1087],{"type":42,"tag":181,"props":1075,"children":1076},{"style":188},[1077],{"type":48,"value":1078},"  dp_replicate_size",{"type":42,"tag":181,"props":1080,"children":1081},{"style":194},[1082],{"type":48,"value":211},{"type":42,"tag":181,"props":1084,"children":1085},{"style":214},[1086],{"type":48,"value":1065},{"type":42,"tag":181,"props":1088,"children":1089},{"style":1045},[1090],{"type":48,"value":1091},"   # FSDP2-only, for HSDP\n",{"type":42,"tag":181,"props":1093,"children":1094},{"class":183,"line":257},[1095,1099,1103],{"type":42,"tag":181,"props":1096,"children":1097},{"style":188},[1098],{"type":48,"value":226},{"type":42,"tag":181,"props":1100,"children":1101},{"style":194},[1102],{"type":48,"value":211},{"type":42,"tag":181,"props":1104,"children":1105},{"style":233},[1106],{"type":48,"value":272},{"type":42,"tag":181,"props":1108,"children":1109},{"class":183,"line":275},[1110,1114,1118],{"type":42,"tag":181,"props":1111,"children":1112},{"style":188},[1113],{"type":48,"value":245},{"type":42,"tag":181,"props":1115,"children":1116},{"style":194},[1117],{"type":48,"value":211},{"type":42,"tag":181,"props":1119,"children":1120},{"style":233},[1121],{"type":48,"value":272},{"type":42,"tag":181,"props":1123,"children":1124},{"class":183,"line":292},[1125,1129,1133],{"type":42,"tag":181,"props":1126,"children":1127},{"style":188},[1128],{"type":48,"value":263},{"type":42,"tag":181,"props":1130,"children":1131},{"style":194},[1132],{"type":48,"value":211},{"type":42,"tag":181,"props":1134,"children":1135},{"style":233},[1136],{"type":48,"value":272},{"type":42,"tag":181,"props":1138,"children":1139},{"class":183,"line":305},[1140,1144,1148],{"type":42,"tag":181,"props":1141,"children":1142},{"style":188},[1143],{"type":48,"value":281},{"type":42,"tag":181,"props":1145,"children":1146},{"style":194},[1147],{"type":48,"value":211},{"type":42,"tag":181,"props":1149,"children":1150},{"style":233},[1151],{"type":48,"value":272},{"type":42,"tag":181,"props":1153,"children":1154},{"class":183,"line":323},[1155],{"type":42,"tag":181,"props":1156,"children":1158},{"emptyLinePlaceholder":1157},true,[1159],{"type":48,"value":1160},"\n",{"type":42,"tag":181,"props":1162,"children":1164},{"class":183,"line":1163},10,[1165],{"type":42,"tag":181,"props":1166,"children":1167},{"style":1045},[1168],{"type":48,"value":1169},"  # Strategy-specific flags (forwarded to the strategy dataclass):\n",{"type":42,"tag":181,"props":1171,"children":1173},{"class":183,"line":1172},11,[1174,1179,1183],{"type":42,"tag":181,"props":1175,"children":1176},{"style":188},[1177],{"type":48,"value":1178},"  sequence_parallel",{"type":42,"tag":181,"props":1180,"children":1181},{"style":194},[1182],{"type":48,"value":211},{"type":42,"tag":181,"props":1184,"children":1185},{"style":483},[1186],{"type":48,"value":486},{"type":42,"tag":181,"props":1188,"children":1190},{"class":183,"line":1189},12,[1191,1196,1200],{"type":42,"tag":181,"props":1192,"children":1193},{"style":188},[1194],{"type":48,"value":1195},"  activation_checkpointing",{"type":42,"tag":181,"props":1197,"children":1198},{"style":194},[1199],{"type":48,"value":211},{"type":42,"tag":181,"props":1201,"children":1202},{"style":483},[1203],{"type":48,"value":486},{"type":42,"tag":181,"props":1205,"children":1207},{"class":183,"line":1206},13,[1208,1213,1217,1222],{"type":42,"tag":181,"props":1209,"children":1210},{"style":188},[1211],{"type":48,"value":1212},"  defer_fsdp_grad_sync",{"type":42,"tag":181,"props":1214,"children":1215},{"style":194},[1216],{"type":48,"value":211},{"type":42,"tag":181,"props":1218,"children":1219},{"style":483},[1220],{"type":48,"value":1221}," true",{"type":42,"tag":181,"props":1223,"children":1224},{"style":1045},[1225],{"type":48,"value":1226},"   # FSDP2 only\n",{"type":42,"tag":181,"props":1228,"children":1230},{"class":183,"line":1229},14,[1231],{"type":42,"tag":181,"props":1232,"children":1233},{"emptyLinePlaceholder":1157},[1234],{"type":48,"value":1160},{"type":42,"tag":181,"props":1236,"children":1238},{"class":183,"line":1237},15,[1239],{"type":42,"tag":181,"props":1240,"children":1241},{"style":1045},[1242],{"type":48,"value":1243},"  # Sub-configs (optional):\n",{"type":42,"tag":181,"props":1245,"children":1247},{"class":183,"line":1246},16,[1248,1252],{"type":42,"tag":181,"props":1249,"children":1250},{"style":188},[1251],{"type":48,"value":298},{"type":42,"tag":181,"props":1253,"children":1254},{"style":194},[1255],{"type":48,"value":197},{"type":42,"tag":181,"props":1257,"children":1259},{"class":183,"line":1258},17,[1260,1264,1268],{"type":42,"tag":181,"props":1261,"children":1262},{"style":188},[1263],{"type":48,"value":311},{"type":42,"tag":181,"props":1265,"children":1266},{"style":194},[1267],{"type":48,"value":211},{"type":42,"tag":181,"props":1269,"children":1270},{"style":214},[1271],{"type":48,"value":1272}," 1f1b\n",{"type":42,"tag":181,"props":1274,"children":1276},{"class":183,"line":1275},18,[1277,1281,1285],{"type":42,"tag":181,"props":1278,"children":1279},{"style":188},[1280],{"type":48,"value":329},{"type":42,"tag":181,"props":1282,"children":1283},{"style":194},[1284],{"type":48,"value":211},{"type":42,"tag":181,"props":1286,"children":1287},{"style":233},[1288],{"type":48,"value":272},{"type":42,"tag":181,"props":1290,"children":1292},{"class":183,"line":1291},19,[1293],{"type":42,"tag":181,"props":1294,"children":1295},{"style":1045},[1296],{"type":48,"value":1297},"    # ... see PipelineConfig fields\n",{"type":42,"tag":181,"props":1299,"children":1301},{"class":183,"line":1300},20,[1302],{"type":42,"tag":181,"props":1303,"children":1304},{"emptyLinePlaceholder":1157},[1305],{"type":48,"value":1160},{"type":42,"tag":181,"props":1307,"children":1309},{"class":183,"line":1308},21,[1310,1314],{"type":42,"tag":181,"props":1311,"children":1312},{"style":188},[1313],{"type":48,"value":464},{"type":42,"tag":181,"props":1315,"children":1316},{"style":194},[1317],{"type":48,"value":197},{"type":42,"tag":181,"props":1319,"children":1321},{"class":183,"line":1320},22,[1322,1326,1330],{"type":42,"tag":181,"props":1323,"children":1324},{"style":188},[1325],{"type":48,"value":476},{"type":42,"tag":181,"props":1327,"children":1328},{"style":194},[1329],{"type":48,"value":211},{"type":42,"tag":181,"props":1331,"children":1332},{"style":483},[1333],{"type":48,"value":486},{"type":42,"tag":181,"props":1335,"children":1337},{"class":183,"line":1336},23,[1338],{"type":42,"tag":181,"props":1339,"children":1340},{"style":1045},[1341],{"type":48,"value":1342},"    # ... see MoEParallelizerConfig fields\n",{"type":42,"tag":58,"props":1344,"children":1345},{},[1346,1347,1352],{"type":48,"value":986},{"type":42,"tag":64,"props":1348,"children":1350},{"className":1349},[],[1351],{"type":48,"value":158},{"type":48,"value":1353}," is always inferred:",{"type":42,"tag":170,"props":1355,"children":1359},{"className":1356,"code":1358,"language":48},[1357],"language-text","dp_size = world_size \u002F (tp_size * pp_size * cp_size)\n",[1360],{"type":42,"tag":64,"props":1361,"children":1362},{"__ignoreMap":175},[1363],{"type":48,"value":1358},{"type":42,"tag":51,"props":1365,"children":1367},{"id":1366},"infrastructure-flow",[1368],{"type":48,"value":1369},"Infrastructure Flow",{"type":42,"tag":170,"props":1371,"children":1374},{"className":1372,"code":1373,"language":48},[1357],"initialize_distributed()                       [components\u002Fdistributed\u002Finit_utils.py]\n    -> initializes torch.distributed process group and returns DistInfo\nYAML distributed section + DistInfo.world_size\n    -> parse_distributed_section()          [recipes\u002F_dist_utils.py]\n    -> create_distributed_setup_from_config()              [recipes\u002F_dist_utils.py]\n        -> DistributedSetup.build()         [components\u002Fdistributed\u002Fconfig.py]\n    -> instantiate_infrastructure()         [_transformers\u002Finfrastructure.py]\n        -> _instantiate_distributed()       -> FSDP2Manager \u002F MegatronFSDPManager \u002F DDPManager\n        -> _instantiate_pipeline()          -> AutoPipeline (if pp_size > 1)\n        -> parallelize_fn                   -> MoE parallelizer (if ep_size > 1) or PP wrapper\n    -> apply_model_infrastructure()         [_transformers\u002Finfrastructure.py]\n        -> _shard_pp() or _shard_ep_fsdp()  (applies sharding to the model)\n",[1375],{"type":42,"tag":64,"props":1376,"children":1377},{"__ignoreMap":175},[1378],{"type":48,"value":1373},{"type":42,"tag":51,"props":1380,"children":1382},{"id":1381},"fsdp2-configuration",[1383],{"type":48,"value":1384},"FSDP2 Configuration",{"type":42,"tag":95,"props":1386,"children":1388},{"id":1387},"basic-fsdp2-data-parallelism-only",[1389],{"type":48,"value":1390},"Basic FSDP2 (data parallelism only)",{"type":42,"tag":170,"props":1392,"children":1394},{"className":172,"code":1393,"language":174,"meta":175,"style":175},"distributed:\n  strategy: fsdp2\n  tp_size: 1\n  cp_size: 1\n",[1395],{"type":42,"tag":64,"props":1396,"children":1397},{"__ignoreMap":175},[1398,1409,1424,1439],{"type":42,"tag":181,"props":1399,"children":1400},{"class":183,"line":184},[1401,1405],{"type":42,"tag":181,"props":1402,"children":1403},{"style":188},[1404],{"type":48,"value":191},{"type":42,"tag":181,"props":1406,"children":1407},{"style":194},[1408],{"type":48,"value":197},{"type":42,"tag":181,"props":1410,"children":1411},{"class":183,"line":200},[1412,1416,1420],{"type":42,"tag":181,"props":1413,"children":1414},{"style":188},[1415],{"type":48,"value":206},{"type":42,"tag":181,"props":1417,"children":1418},{"style":194},[1419],{"type":48,"value":211},{"type":42,"tag":181,"props":1421,"children":1422},{"style":214},[1423],{"type":48,"value":217},{"type":42,"tag":181,"props":1425,"children":1426},{"class":183,"line":220},[1427,1431,1435],{"type":42,"tag":181,"props":1428,"children":1429},{"style":188},[1430],{"type":48,"value":226},{"type":42,"tag":181,"props":1432,"children":1433},{"style":194},[1434],{"type":48,"value":211},{"type":42,"tag":181,"props":1436,"children":1437},{"style":233},[1438],{"type":48,"value":272},{"type":42,"tag":181,"props":1440,"children":1441},{"class":183,"line":239},[1442,1446,1450],{"type":42,"tag":181,"props":1443,"children":1444},{"style":188},[1445],{"type":48,"value":263},{"type":42,"tag":181,"props":1447,"children":1448},{"style":194},[1449],{"type":48,"value":211},{"type":42,"tag":181,"props":1451,"children":1452},{"style":233},[1453],{"type":48,"value":272},{"type":42,"tag":58,"props":1455,"children":1456},{},[1457,1459,1465,1467,1473],{"type":48,"value":1458},"This auto-calculates ",{"type":42,"tag":64,"props":1460,"children":1462},{"className":1461},[],[1463],{"type":48,"value":1464},"dp_size = world_size",{"type":48,"value":1466}," and applies ",{"type":42,"tag":64,"props":1468,"children":1470},{"className":1469},[],[1471],{"type":48,"value":1472},"fully_shard()",{"type":48,"value":1474}," per\ntransformer block via DTensor-based sharding.",{"type":42,"tag":95,"props":1476,"children":1478},{"id":1477},"fsdp2-with-tensor-parallelism",[1479],{"type":48,"value":1480},"FSDP2 with Tensor Parallelism",{"type":42,"tag":58,"props":1482,"children":1483},{},[1484],{"type":48,"value":1485},"Keep TP within a single NVLink domain (typically one node):",{"type":42,"tag":170,"props":1487,"children":1489},{"className":172,"code":1488,"language":174,"meta":175,"style":175},"distributed:\n  strategy: fsdp2\n  tp_size: 4        # 2, 4, or 8 -- must divide GPUs per node\n  sequence_parallel: true\n",[1490],{"type":42,"tag":64,"props":1491,"children":1492},{"__ignoreMap":175},[1493,1504,1519,1540],{"type":42,"tag":181,"props":1494,"children":1495},{"class":183,"line":184},[1496,1500],{"type":42,"tag":181,"props":1497,"children":1498},{"style":188},[1499],{"type":48,"value":191},{"type":42,"tag":181,"props":1501,"children":1502},{"style":194},[1503],{"type":48,"value":197},{"type":42,"tag":181,"props":1505,"children":1506},{"class":183,"line":200},[1507,1511,1515],{"type":42,"tag":181,"props":1508,"children":1509},{"style":188},[1510],{"type":48,"value":206},{"type":42,"tag":181,"props":1512,"children":1513},{"style":194},[1514],{"type":48,"value":211},{"type":42,"tag":181,"props":1516,"children":1517},{"style":214},[1518],{"type":48,"value":217},{"type":42,"tag":181,"props":1520,"children":1521},{"class":183,"line":220},[1522,1526,1530,1535],{"type":42,"tag":181,"props":1523,"children":1524},{"style":188},[1525],{"type":48,"value":226},{"type":42,"tag":181,"props":1527,"children":1528},{"style":194},[1529],{"type":48,"value":211},{"type":42,"tag":181,"props":1531,"children":1532},{"style":233},[1533],{"type":48,"value":1534}," 4",{"type":42,"tag":181,"props":1536,"children":1537},{"style":1045},[1538],{"type":48,"value":1539},"        # 2, 4, or 8 -- must divide GPUs per node\n",{"type":42,"tag":181,"props":1541,"children":1542},{"class":183,"line":239},[1543,1547,1551],{"type":42,"tag":181,"props":1544,"children":1545},{"style":188},[1546],{"type":48,"value":1178},{"type":42,"tag":181,"props":1548,"children":1549},{"style":194},[1550],{"type":48,"value":211},{"type":42,"tag":181,"props":1552,"children":1553},{"style":483},[1554],{"type":48,"value":1555}," true\n",{"type":42,"tag":58,"props":1557,"children":1558},{},[1559],{"type":48,"value":1560},"The TP plan is auto-selected based on the model type. Pass a custom plan via\nthe Python API if needed:",{"type":42,"tag":170,"props":1562,"children":1566},{"className":1563,"code":1564,"language":1565,"meta":175,"style":175},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","config = FSDP2Config(sequence_parallel=True, tp_plan=my_custom_plan)\n","python",[1567],{"type":42,"tag":64,"props":1568,"children":1569},{"__ignoreMap":175},[1570],{"type":42,"tag":181,"props":1571,"children":1572},{"class":183,"line":184},[1573],{"type":42,"tag":181,"props":1574,"children":1575},{},[1576],{"type":48,"value":1564},{"type":42,"tag":95,"props":1578,"children":1580},{"id":1579},"fsdp2-with-pipeline-parallelism",[1581],{"type":48,"value":1582},"FSDP2 with Pipeline Parallelism",{"type":42,"tag":170,"props":1584,"children":1586},{"className":172,"code":1585,"language":174,"meta":175,"style":175},"distributed:\n  strategy: fsdp2\n  pp_size: 2\n  pipeline:\n    pp_schedule: interleaved1f1b   # 1f1b, gpipe, interleaved_1f1b, etc.\n    pp_microbatch_size: 4\n    scale_grads_in_schedule: false\n",[1587],{"type":42,"tag":64,"props":1588,"children":1589},{"__ignoreMap":175},[1590,1601,1616,1632,1643,1664,1679],{"type":42,"tag":181,"props":1591,"children":1592},{"class":183,"line":184},[1593,1597],{"type":42,"tag":181,"props":1594,"children":1595},{"style":188},[1596],{"type":48,"value":191},{"type":42,"tag":181,"props":1598,"children":1599},{"style":194},[1600],{"type":48,"value":197},{"type":42,"tag":181,"props":1602,"children":1603},{"class":183,"line":200},[1604,1608,1612],{"type":42,"tag":181,"props":1605,"children":1606},{"style":188},[1607],{"type":48,"value":206},{"type":42,"tag":181,"props":1609,"children":1610},{"style":194},[1611],{"type":48,"value":211},{"type":42,"tag":181,"props":1613,"children":1614},{"style":214},[1615],{"type":48,"value":217},{"type":42,"tag":181,"props":1617,"children":1618},{"class":183,"line":220},[1619,1623,1627],{"type":42,"tag":181,"props":1620,"children":1621},{"style":188},[1622],{"type":48,"value":245},{"type":42,"tag":181,"props":1624,"children":1625},{"style":194},[1626],{"type":48,"value":211},{"type":42,"tag":181,"props":1628,"children":1629},{"style":233},[1630],{"type":48,"value":1631}," 2\n",{"type":42,"tag":181,"props":1633,"children":1634},{"class":183,"line":239},[1635,1639],{"type":42,"tag":181,"props":1636,"children":1637},{"style":188},[1638],{"type":48,"value":298},{"type":42,"tag":181,"props":1640,"children":1641},{"style":194},[1642],{"type":48,"value":197},{"type":42,"tag":181,"props":1644,"children":1645},{"class":183,"line":257},[1646,1650,1654,1659],{"type":42,"tag":181,"props":1647,"children":1648},{"style":188},[1649],{"type":48,"value":311},{"type":42,"tag":181,"props":1651,"children":1652},{"style":194},[1653],{"type":48,"value":211},{"type":42,"tag":181,"props":1655,"children":1656},{"style":214},[1657],{"type":48,"value":1658}," interleaved1f1b",{"type":42,"tag":181,"props":1660,"children":1661},{"style":1045},[1662],{"type":48,"value":1663},"   # 1f1b, gpipe, interleaved_1f1b, etc.\n",{"type":42,"tag":181,"props":1665,"children":1666},{"class":183,"line":275},[1667,1671,1675],{"type":42,"tag":181,"props":1668,"children":1669},{"style":188},[1670],{"type":48,"value":329},{"type":42,"tag":181,"props":1672,"children":1673},{"style":194},[1674],{"type":48,"value":211},{"type":42,"tag":181,"props":1676,"children":1677},{"style":233},[1678],{"type":48,"value":254},{"type":42,"tag":181,"props":1680,"children":1681},{"class":183,"line":292},[1682,1687,1691],{"type":42,"tag":181,"props":1683,"children":1684},{"style":188},[1685],{"type":48,"value":1686},"    scale_grads_in_schedule",{"type":42,"tag":181,"props":1688,"children":1689},{"style":194},[1690],{"type":48,"value":211},{"type":42,"tag":181,"props":1692,"children":1693},{"style":483},[1694],{"type":48,"value":486},{"type":42,"tag":58,"props":1696,"children":1697},{},[1698,1700,1706,1708,1714,1716,1721],{"type":48,"value":1699},"The model must have a ",{"type":42,"tag":64,"props":1701,"children":1703},{"className":1702},[],[1704],{"type":48,"value":1705},"_pp_plan",{"type":48,"value":1707}," attribute (set on the HF model class) for\n",{"type":42,"tag":64,"props":1709,"children":1711},{"className":1710},[],[1712],{"type":48,"value":1713},"AutoPipeline",{"type":48,"value":1715}," to know how to split layers across stages. Models without\n",{"type":42,"tag":64,"props":1717,"children":1719},{"className":1718},[],[1720],{"type":48,"value":1705},{"type":48,"value":1722}," are not compatible with PP.",{"type":42,"tag":95,"props":1724,"children":1726},{"id":1725},"fsdp2-with-hsdp-hybrid-sharded-data-parallel",[1727],{"type":48,"value":1728},"FSDP2 with HSDP (Hybrid Sharded Data Parallel)",{"type":42,"tag":58,"props":1730,"children":1731},{},[1732],{"type":48,"value":1733},"Intra-node full sharding + inter-node replication via a 2D DeviceMesh:",{"type":42,"tag":170,"props":1735,"children":1737},{"className":172,"code":1736,"language":174,"meta":175,"style":175},"distributed:\n  strategy: fsdp2\n  dp_replicate_size: 2   # must divide dp_size\n",[1738],{"type":42,"tag":64,"props":1739,"children":1740},{"__ignoreMap":175},[1741,1752,1767],{"type":42,"tag":181,"props":1742,"children":1743},{"class":183,"line":184},[1744,1748],{"type":42,"tag":181,"props":1745,"children":1746},{"style":188},[1747],{"type":48,"value":191},{"type":42,"tag":181,"props":1749,"children":1750},{"style":194},[1751],{"type":48,"value":197},{"type":42,"tag":181,"props":1753,"children":1754},{"class":183,"line":200},[1755,1759,1763],{"type":42,"tag":181,"props":1756,"children":1757},{"style":188},[1758],{"type":48,"value":206},{"type":42,"tag":181,"props":1760,"children":1761},{"style":194},[1762],{"type":48,"value":211},{"type":42,"tag":181,"props":1764,"children":1765},{"style":214},[1766],{"type":48,"value":217},{"type":42,"tag":181,"props":1768,"children":1769},{"class":183,"line":220},[1770,1774,1778,1783],{"type":42,"tag":181,"props":1771,"children":1772},{"style":188},[1773],{"type":48,"value":1078},{"type":42,"tag":181,"props":1775,"children":1776},{"style":194},[1777],{"type":48,"value":211},{"type":42,"tag":181,"props":1779,"children":1780},{"style":233},[1781],{"type":48,"value":1782}," 2",{"type":42,"tag":181,"props":1784,"children":1785},{"style":1045},[1786],{"type":48,"value":1787},"   # must divide dp_size\n",{"type":42,"tag":58,"props":1789,"children":1790},{},[1791,1793,1799],{"type":48,"value":1792},"Constraint: ",{"type":42,"tag":64,"props":1794,"children":1796},{"className":1795},[],[1797],{"type":48,"value":1798},"dp_replicate_size \u003C dp_size",{"type":48,"value":1800}," (pure replication with no sharding\nis not supported by FSDP2).",{"type":42,"tag":95,"props":1802,"children":1804},{"id":1803},"activation-checkpointing",[1805],{"type":48,"value":1806},"Activation Checkpointing",{"type":42,"tag":58,"props":1808,"children":1809},{},[1810],{"type":48,"value":1811},"Trades compute for memory by recomputing activations during backward:",{"type":42,"tag":170,"props":1813,"children":1815},{"className":172,"code":1814,"language":174,"meta":175,"style":175},"distributed:\n  activation_checkpointing: true\n",[1816],{"type":42,"tag":64,"props":1817,"children":1818},{"__ignoreMap":175},[1819,1830],{"type":42,"tag":181,"props":1820,"children":1821},{"class":183,"line":184},[1822,1826],{"type":42,"tag":181,"props":1823,"children":1824},{"style":188},[1825],{"type":48,"value":191},{"type":42,"tag":181,"props":1827,"children":1828},{"style":194},[1829],{"type":48,"value":197},{"type":42,"tag":181,"props":1831,"children":1832},{"class":183,"line":200},[1833,1837,1841],{"type":42,"tag":181,"props":1834,"children":1835},{"style":188},[1836],{"type":48,"value":1195},{"type":42,"tag":181,"props":1838,"children":1839},{"style":194},[1840],{"type":48,"value":211},{"type":42,"tag":181,"props":1842,"children":1843},{"style":483},[1844],{"type":48,"value":1555},{"type":42,"tag":58,"props":1846,"children":1847},{},[1848],{"type":48,"value":1849},"This is a model-build\u002Ftraining behavior flag, not mesh topology. Dense\nstrategies read it from the strategy config; EP\u002FMoE paths pass the recipe-level\nflag directly into model infrastructure.",{"type":42,"tag":95,"props":1851,"children":1853},{"id":1852},"gradient-sync-deferral",[1854],{"type":48,"value":1855},"Gradient Sync Deferral",{"type":42,"tag":58,"props":1857,"children":1858},{},[1859],{"type":48,"value":1860},"FSDP2 defers gradient sync to the final micro-batch by default for\ncommunication overlap:",{"type":42,"tag":170,"props":1862,"children":1864},{"className":172,"code":1863,"language":174,"meta":175,"style":175},"distributed:\n  defer_fsdp_grad_sync: true   # default\n",[1865],{"type":42,"tag":64,"props":1866,"children":1867},{"__ignoreMap":175},[1868,1879],{"type":42,"tag":181,"props":1869,"children":1870},{"class":183,"line":184},[1871,1875],{"type":42,"tag":181,"props":1872,"children":1873},{"style":188},[1874],{"type":48,"value":191},{"type":42,"tag":181,"props":1876,"children":1877},{"style":194},[1878],{"type":48,"value":197},{"type":42,"tag":181,"props":1880,"children":1881},{"class":183,"line":200},[1882,1886,1890,1894],{"type":42,"tag":181,"props":1883,"children":1884},{"style":188},[1885],{"type":48,"value":1212},{"type":42,"tag":181,"props":1887,"children":1888},{"style":194},[1889],{"type":48,"value":211},{"type":42,"tag":181,"props":1891,"children":1892},{"style":483},[1893],{"type":48,"value":1221},{"type":42,"tag":181,"props":1895,"children":1896},{"style":1045},[1897],{"type":48,"value":1898},"   # default\n",{"type":42,"tag":95,"props":1900,"children":1902},{"id":1901},"mixed-precision",[1903],{"type":48,"value":1904},"Mixed Precision",{"type":42,"tag":58,"props":1906,"children":1907},{},[1908,1910,1916],{"type":48,"value":1909},"FSDP2Config defaults to bfloat16 for all three precision knobs via\n",{"type":42,"tag":64,"props":1911,"children":1913},{"className":1912},[],[1914],{"type":48,"value":1915},"MixedPrecisionPolicy(param_dtype=bf16, reduce_dtype=bf16, output_dtype=bf16, cast_forward_inputs=True)",{"type":48,"value":1917},". Override via the Python API:",{"type":42,"tag":170,"props":1919,"children":1921},{"className":1563,"code":1920,"language":1565,"meta":175,"style":175},"from torch.distributed.fsdp import MixedPrecisionPolicy\nconfig = FSDP2Config(\n    mp_policy=MixedPrecisionPolicy(param_dtype=torch.float16, reduce_dtype=torch.float32),\n)\n",[1922],{"type":42,"tag":64,"props":1923,"children":1924},{"__ignoreMap":175},[1925,1933,1941,1949],{"type":42,"tag":181,"props":1926,"children":1927},{"class":183,"line":184},[1928],{"type":42,"tag":181,"props":1929,"children":1930},{},[1931],{"type":48,"value":1932},"from torch.distributed.fsdp import MixedPrecisionPolicy\n",{"type":42,"tag":181,"props":1934,"children":1935},{"class":183,"line":200},[1936],{"type":42,"tag":181,"props":1937,"children":1938},{},[1939],{"type":48,"value":1940},"config = FSDP2Config(\n",{"type":42,"tag":181,"props":1942,"children":1943},{"class":183,"line":220},[1944],{"type":42,"tag":181,"props":1945,"children":1946},{},[1947],{"type":48,"value":1948},"    mp_policy=MixedPrecisionPolicy(param_dtype=torch.float16, reduce_dtype=torch.float32),\n",{"type":42,"tag":181,"props":1950,"children":1951},{"class":183,"line":239},[1952],{"type":42,"tag":181,"props":1953,"children":1954},{},[1955],{"type":48,"value":1956},")\n",{"type":42,"tag":51,"props":1958,"children":1960},{"id":1959},"pipeline-parallelism",[1961],{"type":48,"value":1962},"Pipeline Parallelism",{"type":42,"tag":95,"props":1964,"children":1966},{"id":1965},"requirements",[1967],{"type":48,"value":1968},"Requirements",{"type":42,"tag":1970,"props":1971,"children":1972},"ol",{},[1973,1985,1996],{"type":42,"tag":646,"props":1974,"children":1975},{},[1976,1978,1983],{"type":48,"value":1977},"Model class must define ",{"type":42,"tag":64,"props":1979,"children":1981},{"className":1980},[],[1982],{"type":48,"value":1705},{"type":48,"value":1984}," (a dict mapping module FQNs to stages).",{"type":42,"tag":646,"props":1986,"children":1987},{},[1988,1994],{"type":42,"tag":64,"props":1989,"children":1991},{"className":1990},[],[1992],{"type":48,"value":1993},"pp_size > 1",{"type":48,"value":1995}," in the distributed section.",{"type":42,"tag":646,"props":1997,"children":1998},{},[1999,2001,2006],{"type":48,"value":2000},"A ",{"type":42,"tag":64,"props":2002,"children":2004},{"className":2003},[],[2005],{"type":48,"value":150},{"type":48,"value":2007}," sub-config with schedule and microbatch size.",{"type":42,"tag":95,"props":2009,"children":2011},{"id":2010},"supported-schedules",[2012],{"type":48,"value":2013},"Supported schedules",{"type":42,"tag":58,"props":2015,"children":2016},{},[2017,2019,2025],{"type":48,"value":2018},"Defined in ",{"type":42,"tag":64,"props":2020,"children":2022},{"className":2021},[],[2023],{"type":48,"value":2024},"PipelineConfig.pp_schedule",{"type":48,"value":211},{"type":42,"tag":642,"props":2027,"children":2028},{},[2029,2040,2049,2066,2075,2084,2093],{"type":42,"tag":646,"props":2030,"children":2031},{},[2032,2038],{"type":42,"tag":64,"props":2033,"children":2035},{"className":2034},[],[2036],{"type":48,"value":2037},"1f1b",{"type":48,"value":2039}," (one-forward-one-backward, default)",{"type":42,"tag":646,"props":2041,"children":2042},{},[2043],{"type":42,"tag":64,"props":2044,"children":2046},{"className":2045},[],[2047],{"type":48,"value":2048},"gpipe",{"type":42,"tag":646,"props":2050,"children":2051},{},[2052,2058,2060],{"type":42,"tag":64,"props":2053,"children":2055},{"className":2054},[],[2056],{"type":48,"value":2057},"interleaved_1f1b",{"type":48,"value":2059}," \u002F ",{"type":42,"tag":64,"props":2061,"children":2063},{"className":2062},[],[2064],{"type":48,"value":2065},"interleaved1f1b",{"type":42,"tag":646,"props":2067,"children":2068},{},[2069],{"type":42,"tag":64,"props":2070,"children":2072},{"className":2071},[],[2073],{"type":48,"value":2074},"looped_bfs",{"type":42,"tag":646,"props":2076,"children":2077},{},[2078],{"type":42,"tag":64,"props":2079,"children":2081},{"className":2080},[],[2082],{"type":48,"value":2083},"dfs",{"type":42,"tag":646,"props":2085,"children":2086},{},[2087],{"type":42,"tag":64,"props":2088,"children":2090},{"className":2089},[],[2091],{"type":48,"value":2092},"v_schedule",{"type":42,"tag":646,"props":2094,"children":2095},{},[2096],{"type":42,"tag":64,"props":2097,"children":2099},{"className":2098},[],[2100],{"type":48,"value":2101},"zero_bubble",{"type":42,"tag":95,"props":2103,"children":2105},{"id":2104},"example-8b-model-on-8-gpus-pp2-dp4",[2106],{"type":48,"value":2107},"Example (8B model on 8 GPUs, PP=2 + DP=4)",{"type":42,"tag":170,"props":2109,"children":2111},{"className":172,"code":2110,"language":174,"meta":175,"style":175},"distributed:\n  strategy: fsdp2\n  pp_size: 2\n\n  pipeline:\n    pp_schedule: interleaved1f1b\n    pp_microbatch_size: 4\n    scale_grads_in_schedule: false\n\ncheckpoint:\n  model_save_format: safetensors\n  save_consolidated: final\n",[2112],{"type":42,"tag":64,"props":2113,"children":2114},{"__ignoreMap":175},[2115,2126,2141,2156,2163,2174,2189,2204,2219,2226,2238,2255],{"type":42,"tag":181,"props":2116,"children":2117},{"class":183,"line":184},[2118,2122],{"type":42,"tag":181,"props":2119,"children":2120},{"style":188},[2121],{"type":48,"value":191},{"type":42,"tag":181,"props":2123,"children":2124},{"style":194},[2125],{"type":48,"value":197},{"type":42,"tag":181,"props":2127,"children":2128},{"class":183,"line":200},[2129,2133,2137],{"type":42,"tag":181,"props":2130,"children":2131},{"style":188},[2132],{"type":48,"value":206},{"type":42,"tag":181,"props":2134,"children":2135},{"style":194},[2136],{"type":48,"value":211},{"type":42,"tag":181,"props":2138,"children":2139},{"style":214},[2140],{"type":48,"value":217},{"type":42,"tag":181,"props":2142,"children":2143},{"class":183,"line":220},[2144,2148,2152],{"type":42,"tag":181,"props":2145,"children":2146},{"style":188},[2147],{"type":48,"value":245},{"type":42,"tag":181,"props":2149,"children":2150},{"style":194},[2151],{"type":48,"value":211},{"type":42,"tag":181,"props":2153,"children":2154},{"style":233},[2155],{"type":48,"value":1631},{"type":42,"tag":181,"props":2157,"children":2158},{"class":183,"line":239},[2159],{"type":42,"tag":181,"props":2160,"children":2161},{"emptyLinePlaceholder":1157},[2162],{"type":48,"value":1160},{"type":42,"tag":181,"props":2164,"children":2165},{"class":183,"line":257},[2166,2170],{"type":42,"tag":181,"props":2167,"children":2168},{"style":188},[2169],{"type":48,"value":298},{"type":42,"tag":181,"props":2171,"children":2172},{"style":194},[2173],{"type":48,"value":197},{"type":42,"tag":181,"props":2175,"children":2176},{"class":183,"line":275},[2177,2181,2185],{"type":42,"tag":181,"props":2178,"children":2179},{"style":188},[2180],{"type":48,"value":311},{"type":42,"tag":181,"props":2182,"children":2183},{"style":194},[2184],{"type":48,"value":211},{"type":42,"tag":181,"props":2186,"children":2187},{"style":214},[2188],{"type":48,"value":320},{"type":42,"tag":181,"props":2190,"children":2191},{"class":183,"line":292},[2192,2196,2200],{"type":42,"tag":181,"props":2193,"children":2194},{"style":188},[2195],{"type":48,"value":329},{"type":42,"tag":181,"props":2197,"children":2198},{"style":194},[2199],{"type":48,"value":211},{"type":42,"tag":181,"props":2201,"children":2202},{"style":233},[2203],{"type":48,"value":254},{"type":42,"tag":181,"props":2205,"children":2206},{"class":183,"line":305},[2207,2211,2215],{"type":42,"tag":181,"props":2208,"children":2209},{"style":188},[2210],{"type":48,"value":1686},{"type":42,"tag":181,"props":2212,"children":2213},{"style":194},[2214],{"type":48,"value":211},{"type":42,"tag":181,"props":2216,"children":2217},{"style":483},[2218],{"type":48,"value":486},{"type":42,"tag":181,"props":2220,"children":2221},{"class":183,"line":323},[2222],{"type":42,"tag":181,"props":2223,"children":2224},{"emptyLinePlaceholder":1157},[2225],{"type":48,"value":1160},{"type":42,"tag":181,"props":2227,"children":2228},{"class":183,"line":1163},[2229,2234],{"type":42,"tag":181,"props":2230,"children":2231},{"style":188},[2232],{"type":48,"value":2233},"checkpoint",{"type":42,"tag":181,"props":2235,"children":2236},{"style":194},[2237],{"type":48,"value":197},{"type":42,"tag":181,"props":2239,"children":2240},{"class":183,"line":1172},[2241,2246,2250],{"type":42,"tag":181,"props":2242,"children":2243},{"style":188},[2244],{"type":48,"value":2245},"  model_save_format",{"type":42,"tag":181,"props":2247,"children":2248},{"style":194},[2249],{"type":48,"value":211},{"type":42,"tag":181,"props":2251,"children":2252},{"style":214},[2253],{"type":48,"value":2254}," safetensors\n",{"type":42,"tag":181,"props":2256,"children":2257},{"class":183,"line":1189},[2258,2263,2267],{"type":42,"tag":181,"props":2259,"children":2260},{"style":188},[2261],{"type":48,"value":2262},"  save_consolidated",{"type":42,"tag":181,"props":2264,"children":2265},{"style":194},[2266],{"type":48,"value":211},{"type":42,"tag":181,"props":2268,"children":2269},{"style":214},[2270],{"type":48,"value":2271}," final\n",{"type":42,"tag":95,"props":2273,"children":2275},{"id":2274},"how-it-works",[2276],{"type":48,"value":2277},"How it works",{"type":42,"tag":58,"props":2279,"children":2280},{},[2281,2287,2289,2295,2297,2302,2304,2310,2312,2318],{"type":42,"tag":64,"props":2282,"children":2284},{"className":2283},[],[2285],{"type":48,"value":2286},"AutoPipeline.build()",{"type":48,"value":2288}," calls ",{"type":42,"tag":64,"props":2290,"children":2292},{"className":2291},[],[2293],{"type":48,"value":2294},"pipeline_model()",{"type":48,"value":2296}," which splits the model into\nstages using the model's ",{"type":42,"tag":64,"props":2298,"children":2300},{"className":2299},[],[2301],{"type":48,"value":1705},{"type":48,"value":2303},", creates ",{"type":42,"tag":64,"props":2305,"children":2307},{"className":2306},[],[2308],{"type":48,"value":2309},"PipelineStage",{"type":48,"value":2311}," objects, and\nbuilds the schedule. During training, ",{"type":42,"tag":64,"props":2313,"children":2315},{"className":2314},[],[2316],{"type":48,"value":2317},"schedule.step()",{"type":48,"value":2319}," drives forward and\nbackward through the pipeline.",{"type":42,"tag":51,"props":2321,"children":2323},{"id":2322},"context-parallelism",[2324],{"type":48,"value":2325},"Context Parallelism",{"type":42,"tag":58,"props":2327,"children":2328},{},[2329],{"type":48,"value":2330},"Use CP for long sequences (8K+). CP shards Q\u002FK\u002FV on the sequence dimension\nas DTensors.",{"type":42,"tag":95,"props":2332,"children":2334},{"id":2333},"config",[2335],{"type":48,"value":2336},"Config",{"type":42,"tag":170,"props":2338,"children":2340},{"className":172,"code":2339,"language":174,"meta":175,"style":175},"distributed:\n  strategy: fsdp2\n  cp_size: 2   # or 4, 8\n",[2341],{"type":42,"tag":64,"props":2342,"children":2343},{"__ignoreMap":175},[2344,2355,2370],{"type":42,"tag":181,"props":2345,"children":2346},{"class":183,"line":184},[2347,2351],{"type":42,"tag":181,"props":2348,"children":2349},{"style":188},[2350],{"type":48,"value":191},{"type":42,"tag":181,"props":2352,"children":2353},{"style":194},[2354],{"type":48,"value":197},{"type":42,"tag":181,"props":2356,"children":2357},{"class":183,"line":200},[2358,2362,2366],{"type":42,"tag":181,"props":2359,"children":2360},{"style":188},[2361],{"type":48,"value":206},{"type":42,"tag":181,"props":2363,"children":2364},{"style":194},[2365],{"type":48,"value":211},{"type":42,"tag":181,"props":2367,"children":2368},{"style":214},[2369],{"type":48,"value":217},{"type":42,"tag":181,"props":2371,"children":2372},{"class":183,"line":220},[2373,2377,2381,2385],{"type":42,"tag":181,"props":2374,"children":2375},{"style":188},[2376],{"type":48,"value":263},{"type":42,"tag":181,"props":2378,"children":2379},{"style":194},[2380],{"type":48,"value":211},{"type":42,"tag":181,"props":2382,"children":2383},{"style":233},[2384],{"type":48,"value":1782},{"type":42,"tag":181,"props":2386,"children":2387},{"style":1045},[2388],{"type":48,"value":2389},"   # or 4, 8\n",{"type":42,"tag":95,"props":2391,"children":2393},{"id":2392},"requirements-1",[2394],{"type":48,"value":1968},{"type":42,"tag":642,"props":2396,"children":2397},{},[2398,2403],{"type":42,"tag":646,"props":2399,"children":2400},{},[2401],{"type":48,"value":2402},"SDPA (Flash Attention or Efficient Attention backend) or Transformer Engine\nattention. SDPBackend.MATH is not compatible with DTensor.",{"type":42,"tag":646,"props":2404,"children":2405},{},[2406,2408,2414,2416,2422],{"type":48,"value":2407},"Attention masks are automatically stripped; ",{"type":42,"tag":64,"props":2409,"children":2411},{"className":2410},[],[2412],{"type":48,"value":2413},"is_causal=True",{"type":48,"value":2415}," is set via\nforward pre-hooks registered by ",{"type":42,"tag":64,"props":2417,"children":2419},{"className":2418},[],[2420],{"type":48,"value":2421},"attach_context_parallel_hooks()",{"type":48,"value":168},{"type":42,"tag":95,"props":2424,"children":2426},{"id":2425},"how-it-works-1",[2427],{"type":48,"value":2277},{"type":42,"tag":1970,"props":2429,"children":2430},{},[2431,2451,2479],{"type":42,"tag":646,"props":2432,"children":2433},{},[2434,2436,2442,2444,2449],{"type":48,"value":2435},"After model sharding, ",{"type":42,"tag":64,"props":2437,"children":2439},{"className":2438},[],[2440],{"type":48,"value":2441},"apply_model_infrastructure()",{"type":48,"value":2443}," calls\n",{"type":42,"tag":64,"props":2445,"children":2447},{"className":2446},[],[2448],{"type":48,"value":2421},{"type":48,"value":2450}," on each model part (for non-TE models).",{"type":42,"tag":646,"props":2452,"children":2453},{},[2454,2456,2462,2464,2470,2472,2478],{"type":48,"value":2455},"At each training step, ",{"type":42,"tag":64,"props":2457,"children":2459},{"className":2458},[],[2460],{"type":48,"value":2461},"make_cp_batch_and_ctx()",{"type":48,"value":2463}," creates a CP context\nmanager that shards the batch along the sequence dimension and sets up\n",{"type":42,"tag":64,"props":2465,"children":2467},{"className":2466},[],[2468],{"type":48,"value":2469},"context_parallel()",{"type":48,"value":2471}," from ",{"type":42,"tag":64,"props":2473,"children":2475},{"className":2474},[],[2476],{"type":48,"value":2477},"torch.distributed.tensor.experimental",{"type":48,"value":168},{"type":42,"tag":646,"props":2480,"children":2481},{},[2482,2484,2490,2492,2498],{"type":48,"value":2483},"For TE attention models, ",{"type":42,"tag":64,"props":2485,"children":2487},{"className":2486},[],[2488],{"type":48,"value":2489},"make_cp_batch_for_te()",{"type":48,"value":2491}," uses THD format and\nTE's ",{"type":42,"tag":64,"props":2493,"children":2495},{"className":2494},[],[2496],{"type":48,"value":2497},"thd_get_partitioned_indices",{"type":48,"value":2499}," for sharding.",{"type":42,"tag":95,"props":2501,"children":2503},{"id":2502},"cp-with-sequence-packing",[2504],{"type":48,"value":2505},"CP with Sequence Packing",{"type":42,"tag":58,"props":2507,"children":2508},{},[2509,2511,2517,2519,2524,2526,2532],{"type":48,"value":2510},"CP works with packed sequences. The ",{"type":42,"tag":64,"props":2512,"children":2514},{"className":2513},[],[2515],{"type":48,"value":2516},"packed_sequence_size",{"type":48,"value":2518}," must be divisible\nby ",{"type":42,"tag":64,"props":2520,"children":2522},{"className":2521},[],[2523],{"type":48,"value":134},{"type":48,"value":2525},". When using TE, chunks are sharded per-chunk via\n",{"type":42,"tag":64,"props":2527,"children":2529},{"className":2528},[],[2530],{"type":48,"value":2531},"_shard_thd_chunk_for_te()",{"type":48,"value":168},{"type":42,"tag":51,"props":2534,"children":2536},{"id":2535},"sequence-packing",[2537],{"type":48,"value":2538},"Sequence Packing",{"type":42,"tag":58,"props":2540,"children":2541},{},[2542],{"type":48,"value":2543},"Packing multiple sequences into a single training sample for efficiency.",{"type":42,"tag":95,"props":2545,"children":2547},{"id":2546},"config-1",[2548],{"type":48,"value":2336},{"type":42,"tag":170,"props":2550,"children":2552},{"className":172,"code":2551,"language":174,"meta":175,"style":175},"packed_sequence:\n  packed_sequence_size: 4096   # 0 = disabled\n\nstep_scheduler:\n  local_batch_size: 1          # must be 1 for packed sequences\n",[2553],{"type":42,"tag":64,"props":2554,"children":2555},{"__ignoreMap":175},[2556,2568,2590,2597,2609],{"type":42,"tag":181,"props":2557,"children":2558},{"class":183,"line":184},[2559,2564],{"type":42,"tag":181,"props":2560,"children":2561},{"style":188},[2562],{"type":48,"value":2563},"packed_sequence",{"type":42,"tag":181,"props":2565,"children":2566},{"style":194},[2567],{"type":48,"value":197},{"type":42,"tag":181,"props":2569,"children":2570},{"class":183,"line":200},[2571,2576,2580,2585],{"type":42,"tag":181,"props":2572,"children":2573},{"style":188},[2574],{"type":48,"value":2575},"  packed_sequence_size",{"type":42,"tag":181,"props":2577,"children":2578},{"style":194},[2579],{"type":48,"value":211},{"type":42,"tag":181,"props":2581,"children":2582},{"style":233},[2583],{"type":48,"value":2584}," 4096",{"type":42,"tag":181,"props":2586,"children":2587},{"style":1045},[2588],{"type":48,"value":2589},"   # 0 = disabled\n",{"type":42,"tag":181,"props":2591,"children":2592},{"class":183,"line":220},[2593],{"type":42,"tag":181,"props":2594,"children":2595},{"emptyLinePlaceholder":1157},[2596],{"type":48,"value":1160},{"type":42,"tag":181,"props":2598,"children":2599},{"class":183,"line":239},[2600,2605],{"type":42,"tag":181,"props":2601,"children":2602},{"style":188},[2603],{"type":48,"value":2604},"step_scheduler",{"type":42,"tag":181,"props":2606,"children":2607},{"style":194},[2608],{"type":48,"value":197},{"type":42,"tag":181,"props":2610,"children":2611},{"class":183,"line":257},[2612,2617,2621,2626],{"type":42,"tag":181,"props":2613,"children":2614},{"style":188},[2615],{"type":48,"value":2616},"  local_batch_size",{"type":42,"tag":181,"props":2618,"children":2619},{"style":194},[2620],{"type":48,"value":211},{"type":42,"tag":181,"props":2622,"children":2623},{"style":233},[2624],{"type":48,"value":2625}," 1",{"type":42,"tag":181,"props":2627,"children":2628},{"style":1045},[2629],{"type":48,"value":2630},"          # must be 1 for packed sequences\n",{"type":42,"tag":58,"props":2632,"children":2633},{},[2634,2636,2642,2644,2650],{"type":48,"value":2635},"When ",{"type":42,"tag":64,"props":2637,"children":2639},{"className":2638},[],[2640],{"type":48,"value":2641},"packed_sequence_size > 0",{"type":48,"value":2643},", the dataset collator packs sequences up to\nthat length. ",{"type":42,"tag":64,"props":2645,"children":2647},{"className":2646},[],[2648],{"type":48,"value":2649},"local_batch_size",{"type":48,"value":2651}," must be 1 because each \"sample\" is already a\npacked batch.",{"type":42,"tag":51,"props":2653,"children":2655},{"id":2654},"moe-distributed-training",[2656],{"type":48,"value":2657},"MoE Distributed Training",{"type":42,"tag":95,"props":2659,"children":2661},{"id":2660},"expert-parallelism",[2662],{"type":48,"value":2663},"Expert Parallelism",{"type":42,"tag":58,"props":2665,"children":2666},{},[2667,2668,2673,2675,2680,2682,2687],{"type":48,"value":774},{"type":42,"tag":64,"props":2669,"children":2671},{"className":2670},[],[2672],{"type":48,"value":360},{"type":48,"value":2674}," to distribute experts across GPUs. This creates a separate\n",{"type":42,"tag":64,"props":2676,"children":2678},{"className":2677},[],[2679],{"type":48,"value":368},{"type":48,"value":2681}," alongside the main ",{"type":42,"tag":64,"props":2683,"children":2685},{"className":2684},[],[2686],{"type":48,"value":917},{"type":48,"value":211},{"type":42,"tag":170,"props":2689,"children":2691},{"className":172,"code":2690,"language":174,"meta":175,"style":175},"distributed:\n  strategy: fsdp2\n  ep_size: 8\n  activation_checkpointing: true\n",[2692],{"type":42,"tag":64,"props":2693,"children":2694},{"__ignoreMap":175},[2695,2706,2721,2736],{"type":42,"tag":181,"props":2696,"children":2697},{"class":183,"line":184},[2698,2702],{"type":42,"tag":181,"props":2699,"children":2700},{"style":188},[2701],{"type":48,"value":191},{"type":42,"tag":181,"props":2703,"children":2704},{"style":194},[2705],{"type":48,"value":197},{"type":42,"tag":181,"props":2707,"children":2708},{"class":183,"line":200},[2709,2713,2717],{"type":42,"tag":181,"props":2710,"children":2711},{"style":188},[2712],{"type":48,"value":206},{"type":42,"tag":181,"props":2714,"children":2715},{"style":194},[2716],{"type":48,"value":211},{"type":42,"tag":181,"props":2718,"children":2719},{"style":214},[2720],{"type":48,"value":217},{"type":42,"tag":181,"props":2722,"children":2723},{"class":183,"line":220},[2724,2728,2732],{"type":42,"tag":181,"props":2725,"children":2726},{"style":188},[2727],{"type":48,"value":281},{"type":42,"tag":181,"props":2729,"children":2730},{"style":194},[2731],{"type":48,"value":211},{"type":42,"tag":181,"props":2733,"children":2734},{"style":233},[2735],{"type":48,"value":236},{"type":42,"tag":181,"props":2737,"children":2738},{"class":183,"line":239},[2739,2743,2747],{"type":42,"tag":181,"props":2740,"children":2741},{"style":188},[2742],{"type":48,"value":1195},{"type":42,"tag":181,"props":2744,"children":2745},{"style":194},[2746],{"type":48,"value":211},{"type":42,"tag":181,"props":2748,"children":2749},{"style":483},[2750],{"type":48,"value":1555},{"type":42,"tag":58,"props":2752,"children":2753},{},[2754,2755,2760,2762,2768,2770,2776],{"type":48,"value":986},{"type":42,"tag":64,"props":2756,"children":2758},{"className":2757},[],[2759],{"type":48,"value":368},{"type":48,"value":2761}," shape is ",{"type":42,"tag":64,"props":2763,"children":2765},{"className":2764},[],[2766],{"type":48,"value":2767},"(pp_size, ep_shard_size, ep_size)",{"type":48,"value":2769}," with dimension\nnames ",{"type":42,"tag":64,"props":2771,"children":2773},{"className":2772},[],[2774],{"type":48,"value":2775},"(\"pp\", \"ep_shard\", \"ep\")",{"type":48,"value":168},{"type":42,"tag":58,"props":2778,"children":2779},{},[2780,2781,2787,2789,2794,2796,2801],{"type":48,"value":1792},{"type":42,"tag":64,"props":2782,"children":2784},{"className":2783},[],[2785],{"type":48,"value":2786},"dp_cp_size",{"type":48,"value":2788}," (= ",{"type":42,"tag":64,"props":2790,"children":2792},{"className":2791},[],[2793],{"type":48,"value":391},{"type":48,"value":2795},") must be divisible by\n",{"type":42,"tag":64,"props":2797,"children":2799},{"className":2798},[],[2800],{"type":48,"value":142},{"type":48,"value":168},{"type":42,"tag":95,"props":2803,"children":2805},{"id":2804},"moe-sub-config",[2806],{"type":48,"value":2807},"MoE sub-config",{"type":42,"tag":170,"props":2809,"children":2811},{"className":172,"code":2810,"language":174,"meta":175,"style":175},"distributed:\n  strategy: fsdp2\n  ep_size: 8\n  activation_checkpointing: true\n\n  moe:\n    reshard_after_forward: false\n    ignore_router_for_ac: false\n    wrap_outer_model: true\n",[2812],{"type":42,"tag":64,"props":2813,"children":2814},{"__ignoreMap":175},[2815,2826,2841,2856,2871,2878,2889,2904,2920],{"type":42,"tag":181,"props":2816,"children":2817},{"class":183,"line":184},[2818,2822],{"type":42,"tag":181,"props":2819,"children":2820},{"style":188},[2821],{"type":48,"value":191},{"type":42,"tag":181,"props":2823,"children":2824},{"style":194},[2825],{"type":48,"value":197},{"type":42,"tag":181,"props":2827,"children":2828},{"class":183,"line":200},[2829,2833,2837],{"type":42,"tag":181,"props":2830,"children":2831},{"style":188},[2832],{"type":48,"value":206},{"type":42,"tag":181,"props":2834,"children":2835},{"style":194},[2836],{"type":48,"value":211},{"type":42,"tag":181,"props":2838,"children":2839},{"style":214},[2840],{"type":48,"value":217},{"type":42,"tag":181,"props":2842,"children":2843},{"class":183,"line":220},[2844,2848,2852],{"type":42,"tag":181,"props":2845,"children":2846},{"style":188},[2847],{"type":48,"value":281},{"type":42,"tag":181,"props":2849,"children":2850},{"style":194},[2851],{"type":48,"value":211},{"type":42,"tag":181,"props":2853,"children":2854},{"style":233},[2855],{"type":48,"value":236},{"type":42,"tag":181,"props":2857,"children":2858},{"class":183,"line":239},[2859,2863,2867],{"type":42,"tag":181,"props":2860,"children":2861},{"style":188},[2862],{"type":48,"value":1195},{"type":42,"tag":181,"props":2864,"children":2865},{"style":194},[2866],{"type":48,"value":211},{"type":42,"tag":181,"props":2868,"children":2869},{"style":483},[2870],{"type":48,"value":1555},{"type":42,"tag":181,"props":2872,"children":2873},{"class":183,"line":257},[2874],{"type":42,"tag":181,"props":2875,"children":2876},{"emptyLinePlaceholder":1157},[2877],{"type":48,"value":1160},{"type":42,"tag":181,"props":2879,"children":2880},{"class":183,"line":275},[2881,2885],{"type":42,"tag":181,"props":2882,"children":2883},{"style":188},[2884],{"type":48,"value":464},{"type":42,"tag":181,"props":2886,"children":2887},{"style":194},[2888],{"type":48,"value":197},{"type":42,"tag":181,"props":2890,"children":2891},{"class":183,"line":292},[2892,2896,2900],{"type":42,"tag":181,"props":2893,"children":2894},{"style":188},[2895],{"type":48,"value":476},{"type":42,"tag":181,"props":2897,"children":2898},{"style":194},[2899],{"type":48,"value":211},{"type":42,"tag":181,"props":2901,"children":2902},{"style":483},[2903],{"type":48,"value":486},{"type":42,"tag":181,"props":2905,"children":2906},{"class":183,"line":305},[2907,2912,2916],{"type":42,"tag":181,"props":2908,"children":2909},{"style":188},[2910],{"type":48,"value":2911},"    ignore_router_for_ac",{"type":42,"tag":181,"props":2913,"children":2914},{"style":194},[2915],{"type":48,"value":211},{"type":42,"tag":181,"props":2917,"children":2918},{"style":483},[2919],{"type":48,"value":486},{"type":42,"tag":181,"props":2921,"children":2922},{"class":183,"line":323},[2923,2928,2932],{"type":42,"tag":181,"props":2924,"children":2925},{"style":188},[2926],{"type":48,"value":2927},"    wrap_outer_model",{"type":42,"tag":181,"props":2929,"children":2930},{"style":194},[2931],{"type":48,"value":211},{"type":42,"tag":181,"props":2933,"children":2934},{"style":483},[2935],{"type":48,"value":1555},{"type":42,"tag":58,"props":2937,"children":2938},{},[2939,2940,2945,2947,2952,2954,2959],{"type":48,"value":986},{"type":42,"tag":64,"props":2941,"children":2943},{"className":2942},[],[2944],{"type":48,"value":376},{"type":48,"value":2946}," sub-section maps to ",{"type":42,"tag":64,"props":2948,"children":2950},{"className":2949},[],[2951],{"type":48,"value":898},{"type":48,"value":2953}," and is only\ninstantiated when ",{"type":42,"tag":64,"props":2955,"children":2957},{"className":2956},[],[2958],{"type":48,"value":360},{"type":48,"value":168},{"type":42,"tag":95,"props":2961,"children":2963},{"id":2962},"full-moe-example-qwen3-30b-a3b-on-8-gpus",[2964],{"type":48,"value":2965},"Full MoE example (Qwen3-30B-A3B on 8 GPUs)",{"type":42,"tag":170,"props":2967,"children":2969},{"className":172,"code":2968,"language":174,"meta":175,"style":175},"distributed:\n  strategy: fsdp2\n  tp_size: 1\n  cp_size: 1\n  pp_size: 1\n  ep_size: 8\n  sequence_parallel: false\n  activation_checkpointing: true\n",[2970],{"type":42,"tag":64,"props":2971,"children":2972},{"__ignoreMap":175},[2973,2984,2999,3014,3029,3044,3059,3074],{"type":42,"tag":181,"props":2974,"children":2975},{"class":183,"line":184},[2976,2980],{"type":42,"tag":181,"props":2977,"children":2978},{"style":188},[2979],{"type":48,"value":191},{"type":42,"tag":181,"props":2981,"children":2982},{"style":194},[2983],{"type":48,"value":197},{"type":42,"tag":181,"props":2985,"children":2986},{"class":183,"line":200},[2987,2991,2995],{"type":42,"tag":181,"props":2988,"children":2989},{"style":188},[2990],{"type":48,"value":206},{"type":42,"tag":181,"props":2992,"children":2993},{"style":194},[2994],{"type":48,"value":211},{"type":42,"tag":181,"props":2996,"children":2997},{"style":214},[2998],{"type":48,"value":217},{"type":42,"tag":181,"props":3000,"children":3001},{"class":183,"line":220},[3002,3006,3010],{"type":42,"tag":181,"props":3003,"children":3004},{"style":188},[3005],{"type":48,"value":226},{"type":42,"tag":181,"props":3007,"children":3008},{"style":194},[3009],{"type":48,"value":211},{"type":42,"tag":181,"props":3011,"children":3012},{"style":233},[3013],{"type":48,"value":272},{"type":42,"tag":181,"props":3015,"children":3016},{"class":183,"line":239},[3017,3021,3025],{"type":42,"tag":181,"props":3018,"children":3019},{"style":188},[3020],{"type":48,"value":263},{"type":42,"tag":181,"props":3022,"children":3023},{"style":194},[3024],{"type":48,"value":211},{"type":42,"tag":181,"props":3026,"children":3027},{"style":233},[3028],{"type":48,"value":272},{"type":42,"tag":181,"props":3030,"children":3031},{"class":183,"line":257},[3032,3036,3040],{"type":42,"tag":181,"props":3033,"children":3034},{"style":188},[3035],{"type":48,"value":245},{"type":42,"tag":181,"props":3037,"children":3038},{"style":194},[3039],{"type":48,"value":211},{"type":42,"tag":181,"props":3041,"children":3042},{"style":233},[3043],{"type":48,"value":272},{"type":42,"tag":181,"props":3045,"children":3046},{"class":183,"line":275},[3047,3051,3055],{"type":42,"tag":181,"props":3048,"children":3049},{"style":188},[3050],{"type":48,"value":281},{"type":42,"tag":181,"props":3052,"children":3053},{"style":194},[3054],{"type":48,"value":211},{"type":42,"tag":181,"props":3056,"children":3057},{"style":233},[3058],{"type":48,"value":236},{"type":42,"tag":181,"props":3060,"children":3061},{"class":183,"line":292},[3062,3066,3070],{"type":42,"tag":181,"props":3063,"children":3064},{"style":188},[3065],{"type":48,"value":1178},{"type":42,"tag":181,"props":3067,"children":3068},{"style":194},[3069],{"type":48,"value":211},{"type":42,"tag":181,"props":3071,"children":3072},{"style":483},[3073],{"type":48,"value":486},{"type":42,"tag":181,"props":3075,"children":3076},{"class":183,"line":305},[3077,3081,3085],{"type":42,"tag":181,"props":3078,"children":3079},{"style":188},[3080],{"type":48,"value":1195},{"type":42,"tag":181,"props":3082,"children":3083},{"style":194},[3084],{"type":48,"value":211},{"type":42,"tag":181,"props":3086,"children":3087},{"style":483},[3088],{"type":48,"value":1555},{"type":42,"tag":95,"props":3090,"children":3092},{"id":3091},"megatronfsdp-limitations-1",[3093],{"type":48,"value":492},{"type":42,"tag":58,"props":3095,"children":3096},{},[3097,3099,3104,3106,3112,3114,3119,3121,3126,3128,3133,3135,3140],{"type":48,"value":3098},"Despite its name, ",{"type":42,"tag":64,"props":3100,"children":3102},{"className":3101},[],[3103],{"type":48,"value":399},{"type":48,"value":3105}," does ",{"type":42,"tag":3107,"props":3108,"children":3109},"strong",{},[3110],{"type":48,"value":3111},"not",{"type":48,"value":3113}," support expert parallelism\n(",{"type":42,"tag":64,"props":3115,"children":3117},{"className":3116},[],[3118],{"type":48,"value":360},{"type":48,"value":3120},"), pipeline parallelism (",{"type":42,"tag":64,"props":3122,"children":3124},{"className":3123},[],[3125],{"type":48,"value":1993},{"type":48,"value":3127},"), or\n",{"type":42,"tag":64,"props":3129,"children":3131},{"className":3130},[],[3132],{"type":48,"value":503},{"type":48,"value":3134},". Use ",{"type":42,"tag":64,"props":3136,"children":3138},{"className":3137},[],[3139],{"type":48,"value":511},{"type":48,"value":3141}," for these features.",{"type":42,"tag":51,"props":3143,"children":3145},{"id":3144},"parallelism-sizing-guidelines",[3146],{"type":48,"value":3147},"Parallelism Sizing Guidelines",{"type":42,"tag":95,"props":3149,"children":3151},{"id":3150},"dense-models",[3152],{"type":48,"value":3153},"Dense models",{"type":42,"tag":541,"props":3155,"children":3156},{},[3157,3187],{"type":42,"tag":545,"props":3158,"children":3159},{},[3160],{"type":42,"tag":549,"props":3161,"children":3162},{},[3163,3168,3173,3178,3183],{"type":42,"tag":553,"props":3164,"children":3165},{},[3166],{"type":48,"value":3167},"Model size",{"type":42,"tag":553,"props":3169,"children":3170},{},[3171],{"type":48,"value":3172},"TP",{"type":42,"tag":553,"props":3174,"children":3175},{},[3176],{"type":48,"value":3177},"PP",{"type":42,"tag":553,"props":3179,"children":3180},{},[3181],{"type":48,"value":3182},"CP",{"type":42,"tag":553,"props":3184,"children":3185},{},[3186],{"type":48,"value":557},{"type":42,"tag":569,"props":3188,"children":3189},{},[3190,3216,3242,3268,3293],{"type":42,"tag":549,"props":3191,"children":3192},{},[3193,3198,3203,3207,3211],{"type":42,"tag":576,"props":3194,"children":3195},{},[3196],{"type":48,"value":3197},"\u003C 3B",{"type":42,"tag":576,"props":3199,"children":3200},{},[3201],{"type":48,"value":3202},"1",{"type":42,"tag":576,"props":3204,"children":3205},{},[3206],{"type":48,"value":3202},{"type":42,"tag":576,"props":3208,"children":3209},{},[3210],{"type":48,"value":3202},{"type":42,"tag":576,"props":3212,"children":3213},{},[3214],{"type":48,"value":3215},"FSDP2 (DP only)",{"type":42,"tag":549,"props":3217,"children":3218},{},[3219,3224,3229,3233,3237],{"type":42,"tag":576,"props":3220,"children":3221},{},[3222],{"type":48,"value":3223},"3-13B",{"type":42,"tag":576,"props":3225,"children":3226},{},[3227],{"type":48,"value":3228},"2-4",{"type":42,"tag":576,"props":3230,"children":3231},{},[3232],{"type":48,"value":3202},{"type":42,"tag":576,"props":3234,"children":3235},{},[3236],{"type":48,"value":3202},{"type":42,"tag":576,"props":3238,"children":3239},{},[3240],{"type":48,"value":3241},"FSDP2 + TP",{"type":42,"tag":549,"props":3243,"children":3244},{},[3245,3250,3255,3259,3263],{"type":42,"tag":576,"props":3246,"children":3247},{},[3248],{"type":48,"value":3249},"13-70B",{"type":42,"tag":576,"props":3251,"children":3252},{},[3253],{"type":48,"value":3254},"4-8",{"type":42,"tag":576,"props":3256,"children":3257},{},[3258],{"type":48,"value":3228},{"type":42,"tag":576,"props":3260,"children":3261},{},[3262],{"type":48,"value":3202},{"type":42,"tag":576,"props":3264,"children":3265},{},[3266],{"type":48,"value":3267},"FSDP2 + TP + PP",{"type":42,"tag":549,"props":3269,"children":3270},{},[3271,3276,3281,3285,3289],{"type":42,"tag":576,"props":3272,"children":3273},{},[3274],{"type":48,"value":3275},"70B+",{"type":42,"tag":576,"props":3277,"children":3278},{},[3279],{"type":48,"value":3280},"8",{"type":42,"tag":576,"props":3282,"children":3283},{},[3284],{"type":48,"value":3254},{"type":42,"tag":576,"props":3286,"children":3287},{},[3288],{"type":48,"value":3202},{"type":42,"tag":576,"props":3290,"children":3291},{},[3292],{"type":48,"value":3267},{"type":42,"tag":549,"props":3294,"children":3295},{},[3296,3301,3306,3310,3315],{"type":42,"tag":576,"props":3297,"children":3298},{},[3299],{"type":48,"value":3300},"Any + long seq (8K+)",{"type":42,"tag":576,"props":3302,"children":3303},{},[3304],{"type":48,"value":3305},"as above",{"type":42,"tag":576,"props":3307,"children":3308},{},[3309],{"type":48,"value":3305},{"type":42,"tag":576,"props":3311,"children":3312},{},[3313],{"type":48,"value":3314},"2-8",{"type":42,"tag":576,"props":3316,"children":3317},{},[3318],{"type":48,"value":3319},"add CP",{"type":42,"tag":95,"props":3321,"children":3323},{"id":3322},"moe-models",[3324],{"type":48,"value":3325},"MoE models",{"type":42,"tag":58,"props":3327,"children":3328},{},[3329],{"type":48,"value":3330},"MoE models need less TP than dense models of similar total parameter count\nbecause only a fraction of parameters are active per token. EP is the primary\nscaling dimension:",{"type":42,"tag":541,"props":3332,"children":3333},{},[3334,3363],{"type":42,"tag":545,"props":3335,"children":3336},{},[3337],{"type":42,"tag":549,"props":3338,"children":3339},{},[3340,3345,3349,3353,3358],{"type":42,"tag":553,"props":3341,"children":3342},{},[3343],{"type":48,"value":3344},"Model",{"type":42,"tag":553,"props":3346,"children":3347},{},[3348],{"type":48,"value":3172},{"type":42,"tag":553,"props":3350,"children":3351},{},[3352],{"type":48,"value":3177},{"type":42,"tag":553,"props":3354,"children":3355},{},[3356],{"type":48,"value":3357},"EP",{"type":42,"tag":553,"props":3359,"children":3360},{},[3361],{"type":48,"value":3362},"Notes",{"type":42,"tag":569,"props":3364,"children":3365},{},[3366,3391,3417],{"type":42,"tag":549,"props":3367,"children":3368},{},[3369,3374,3378,3382,3386],{"type":42,"tag":576,"props":3370,"children":3371},{},[3372],{"type":48,"value":3373},"Small MoE (\u003C10B total)",{"type":42,"tag":576,"props":3375,"children":3376},{},[3377],{"type":48,"value":3202},{"type":42,"tag":576,"props":3379,"children":3380},{},[3381],{"type":48,"value":3202},{"type":42,"tag":576,"props":3383,"children":3384},{},[3385],{"type":48,"value":3280},{"type":42,"tag":576,"props":3387,"children":3388},{},[3389],{"type":48,"value":3390},"EP only",{"type":42,"tag":549,"props":3392,"children":3393},{},[3394,3399,3404,3408,3412],{"type":42,"tag":576,"props":3395,"children":3396},{},[3397],{"type":48,"value":3398},"Medium MoE (10-30B total)",{"type":42,"tag":576,"props":3400,"children":3401},{},[3402],{"type":48,"value":3403},"1-2",{"type":42,"tag":576,"props":3405,"children":3406},{},[3407],{"type":48,"value":3202},{"type":42,"tag":576,"props":3409,"children":3410},{},[3411],{"type":48,"value":3280},{"type":42,"tag":576,"props":3413,"children":3414},{},[3415],{"type":48,"value":3416},"small TP for shared layers",{"type":42,"tag":549,"props":3418,"children":3419},{},[3420,3425,3429,3434,3439],{"type":42,"tag":576,"props":3421,"children":3422},{},[3423],{"type":48,"value":3424},"Large MoE (100B+ total)",{"type":42,"tag":576,"props":3426,"children":3427},{},[3428],{"type":48,"value":3403},{"type":42,"tag":576,"props":3430,"children":3431},{},[3432],{"type":48,"value":3433},"4+",{"type":42,"tag":576,"props":3435,"children":3436},{},[3437],{"type":48,"value":3438},"8-64",{"type":42,"tag":576,"props":3440,"children":3441},{},[3442],{"type":48,"value":3443},"PP for depth, EP for experts",{"type":42,"tag":95,"props":3445,"children":3447},{"id":3446},"hardware-topology-rules",[3448],{"type":48,"value":3449},"Hardware topology rules",{"type":42,"tag":642,"props":3451,"children":3452},{},[3453,3458,3463],{"type":42,"tag":646,"props":3454,"children":3455},{},[3456],{"type":48,"value":3457},"TP must stay within a single NVLink domain (one node, typically 8 GPUs).",{"type":42,"tag":646,"props":3459,"children":3460},{},[3461],{"type":48,"value":3462},"Use PP or DP for cross-node scaling.",{"type":42,"tag":646,"props":3464,"children":3465},{},[3466],{"type":48,"value":3467},"TP across InfiniBand degrades throughput severely.",{"type":42,"tag":51,"props":3469,"children":3471},{"id":3470},"programmatic-api-from_pretrained-from_config",[3472],{"type":48,"value":3473},"Programmatic API (from_pretrained \u002F from_config)",{"type":42,"tag":58,"props":3475,"children":3476},{},[3477],{"type":48,"value":3478},"When not using YAML recipes, configure distributed training via Python:",{"type":42,"tag":170,"props":3480,"children":3482},{"className":1563,"code":3481,"language":1565,"meta":175,"style":175},"from nemo_automodel.components.distributed import (\n    DistributedSetup,\n    FSDP2Config,\n    ParallelismSizes,\n    initialize_distributed,\n)\n\ndist_env = initialize_distributed(\"nccl\")\ndistributed_setup = DistributedSetup.build(\n    strategy=FSDP2Config(sequence_parallel=True),\n    parallelism_sizes=ParallelismSizes(tp_size=2),\n    activation_checkpointing=True,\n    world_size=dist_env.world_size,\n)\n",[3483],{"type":42,"tag":64,"props":3484,"children":3485},{"__ignoreMap":175},[3486,3494,3502,3510,3518,3526,3533,3540,3548,3556,3564,3572,3580,3588],{"type":42,"tag":181,"props":3487,"children":3488},{"class":183,"line":184},[3489],{"type":42,"tag":181,"props":3490,"children":3491},{},[3492],{"type":48,"value":3493},"from nemo_automodel.components.distributed import (\n",{"type":42,"tag":181,"props":3495,"children":3496},{"class":183,"line":200},[3497],{"type":42,"tag":181,"props":3498,"children":3499},{},[3500],{"type":48,"value":3501},"    DistributedSetup,\n",{"type":42,"tag":181,"props":3503,"children":3504},{"class":183,"line":220},[3505],{"type":42,"tag":181,"props":3506,"children":3507},{},[3508],{"type":48,"value":3509},"    FSDP2Config,\n",{"type":42,"tag":181,"props":3511,"children":3512},{"class":183,"line":239},[3513],{"type":42,"tag":181,"props":3514,"children":3515},{},[3516],{"type":48,"value":3517},"    ParallelismSizes,\n",{"type":42,"tag":181,"props":3519,"children":3520},{"class":183,"line":257},[3521],{"type":42,"tag":181,"props":3522,"children":3523},{},[3524],{"type":48,"value":3525},"    initialize_distributed,\n",{"type":42,"tag":181,"props":3527,"children":3528},{"class":183,"line":275},[3529],{"type":42,"tag":181,"props":3530,"children":3531},{},[3532],{"type":48,"value":1956},{"type":42,"tag":181,"props":3534,"children":3535},{"class":183,"line":292},[3536],{"type":42,"tag":181,"props":3537,"children":3538},{"emptyLinePlaceholder":1157},[3539],{"type":48,"value":1160},{"type":42,"tag":181,"props":3541,"children":3542},{"class":183,"line":305},[3543],{"type":42,"tag":181,"props":3544,"children":3545},{},[3546],{"type":48,"value":3547},"dist_env = initialize_distributed(\"nccl\")\n",{"type":42,"tag":181,"props":3549,"children":3550},{"class":183,"line":323},[3551],{"type":42,"tag":181,"props":3552,"children":3553},{},[3554],{"type":48,"value":3555},"distributed_setup = DistributedSetup.build(\n",{"type":42,"tag":181,"props":3557,"children":3558},{"class":183,"line":1163},[3559],{"type":42,"tag":181,"props":3560,"children":3561},{},[3562],{"type":48,"value":3563},"    strategy=FSDP2Config(sequence_parallel=True),\n",{"type":42,"tag":181,"props":3565,"children":3566},{"class":183,"line":1172},[3567],{"type":42,"tag":181,"props":3568,"children":3569},{},[3570],{"type":48,"value":3571},"    parallelism_sizes=ParallelismSizes(tp_size=2),\n",{"type":42,"tag":181,"props":3573,"children":3574},{"class":183,"line":1189},[3575],{"type":42,"tag":181,"props":3576,"children":3577},{},[3578],{"type":48,"value":3579},"    activation_checkpointing=True,\n",{"type":42,"tag":181,"props":3581,"children":3582},{"class":183,"line":1206},[3583],{"type":42,"tag":181,"props":3584,"children":3585},{},[3586],{"type":48,"value":3587},"    world_size=dist_env.world_size,\n",{"type":42,"tag":181,"props":3589,"children":3590},{"class":183,"line":1229},[3591],{"type":42,"tag":181,"props":3592,"children":3593},{},[3594],{"type":48,"value":1956},{"type":42,"tag":58,"props":3596,"children":3597},{},[3598,3600,3606],{"type":48,"value":3599},"Or pass directly to ",{"type":42,"tag":64,"props":3601,"children":3603},{"className":3602},[],[3604],{"type":48,"value":3605},"from_pretrained",{"type":48,"value":211},{"type":42,"tag":170,"props":3608,"children":3610},{"className":1563,"code":3609,"language":1565,"meta":175,"style":175},"from nemo_automodel import NeMoAutoModelForCausalLM\n\nmodel = NeMoAutoModelForCausalLM.from_pretrained(\n    \"meta-llama\u002FLlama-3.2-1B\",\n    distributed_setup=distributed_setup,\n)\n",[3611],{"type":42,"tag":64,"props":3612,"children":3613},{"__ignoreMap":175},[3614,3622,3629,3637,3645,3653],{"type":42,"tag":181,"props":3615,"children":3616},{"class":183,"line":184},[3617],{"type":42,"tag":181,"props":3618,"children":3619},{},[3620],{"type":48,"value":3621},"from nemo_automodel import NeMoAutoModelForCausalLM\n",{"type":42,"tag":181,"props":3623,"children":3624},{"class":183,"line":200},[3625],{"type":42,"tag":181,"props":3626,"children":3627},{"emptyLinePlaceholder":1157},[3628],{"type":48,"value":1160},{"type":42,"tag":181,"props":3630,"children":3631},{"class":183,"line":220},[3632],{"type":42,"tag":181,"props":3633,"children":3634},{},[3635],{"type":48,"value":3636},"model = NeMoAutoModelForCausalLM.from_pretrained(\n",{"type":42,"tag":181,"props":3638,"children":3639},{"class":183,"line":239},[3640],{"type":42,"tag":181,"props":3641,"children":3642},{},[3643],{"type":48,"value":3644},"    \"meta-llama\u002FLlama-3.2-1B\",\n",{"type":42,"tag":181,"props":3646,"children":3647},{"class":183,"line":257},[3648],{"type":42,"tag":181,"props":3649,"children":3650},{},[3651],{"type":48,"value":3652},"    distributed_setup=distributed_setup,\n",{"type":42,"tag":181,"props":3654,"children":3655},{"class":183,"line":275},[3656],{"type":42,"tag":181,"props":3657,"children":3658},{},[3659],{"type":48,"value":1956},{"type":42,"tag":51,"props":3661,"children":3663},{"id":3662},"code-anchors",[3664],{"type":48,"value":3665},"Code Anchors",{"type":42,"tag":58,"props":3667,"children":3668},{},[3669],{"type":48,"value":3670},"Strategy config dataclasses:",{"type":42,"tag":170,"props":3672,"children":3675},{"className":3673,"code":3674,"language":48},[1357],"components\u002Fdistributed\u002Fconfig.py\n    FSDP2Config       -- sequence_parallel, tp_plan, mp_policy, offload_policy,\n                         activation_checkpointing, defer_fsdp_grad_sync\n    MegatronFSDPConfig -- zero_dp_strategy, overlap_grad_reduce, overlap_param_gather, etc.\n    DDPConfig          -- activation_checkpointing only\n",[3676],{"type":42,"tag":64,"props":3677,"children":3678},{"__ignoreMap":175},[3679],{"type":48,"value":3674},{"type":42,"tag":58,"props":3681,"children":3682},{},[3683],{"type":48,"value":3684},"MeshContext (single source of truth for parallelism):",{"type":42,"tag":170,"props":3686,"children":3689},{"className":3687,"code":3688,"language":48},[1357],"components\u002Fdistributed\u002Fmesh.py\n    MeshContext  -- device_mesh, moe_mesh\n                    Properties: tp_size, pp_size, cp_size, ep_size, dp_size, dp_replicate_size\n    MeshAxisName -- PP, DP, DP_REPLICATE, DP_SHARD, DP_SHARD_CP, DP_CP, CP, TP, EP, EP_SHARD\n",[3690],{"type":42,"tag":64,"props":3691,"children":3692},{"__ignoreMap":175},[3693],{"type":48,"value":3688},{"type":42,"tag":58,"props":3695,"children":3696},{},[3697],{"type":48,"value":3698},"Mesh context and raw mesh creation:",{"type":42,"tag":170,"props":3700,"children":3703},{"className":3701,"code":3702,"language":48},[1357],"components\u002Fdistributed\u002Fconfig.py\n    DistributedSetup.build()      -- builds MeshContext from strategy + parallelism\ncomponents\u002Fdistributed\u002Fmesh_utils.py\n    _create_device_meshes()       -- routes to FSDP2\u002FMegatronFSDP\u002FDDP raw mesh creation\n    _create_fsdp2_device_mesh()   -- shape (pp, dp_replicate, dp_shard, cp, tp) + flattened submeshes\n    _create_megatron_fsdp_device_mesh() -- shape (dp, cp, tp)\n",[3704],{"type":42,"tag":64,"props":3705,"children":3706},{"__ignoreMap":175},[3707],{"type":48,"value":3702},{"type":42,"tag":58,"props":3709,"children":3710},{},[3711],{"type":48,"value":3712},"Distributed managers:",{"type":42,"tag":170,"props":3714,"children":3717},{"className":3715,"code":3716,"language":48},[1357],"components\u002Fdistributed\u002Ffsdp2.py          -- FSDP2Manager.parallelize()\ncomponents\u002Fdistributed\u002Fmegatron_fsdp.py  -- MegatronFSDPManager.parallelize()\ncomponents\u002Fdistributed\u002Fddp.py            -- DDPManager\n",[3718],{"type":42,"tag":64,"props":3719,"children":3720},{"__ignoreMap":175},[3721],{"type":48,"value":3716},{"type":42,"tag":58,"props":3723,"children":3724},{},[3725],{"type":48,"value":3726},"Pipeline parallelism:",{"type":42,"tag":170,"props":3728,"children":3731},{"className":3729,"code":3730,"language":48},[1357],"components\u002Fdistributed\u002Fpipelining\u002Fconfig.py        -- PipelineConfig dataclass\ncomponents\u002Fdistributed\u002Fpipelining\u002Fautopipeline.py  -- AutoPipeline orchestrator\ncomponents\u002Fdistributed\u002Fpipelining\u002Ffunctional.py    -- pipeline_model(), schedule creation\ncomponents\u002Fdistributed\u002Fpipelining\u002Fhf_utils.py      -- HF model validation for PP\n",[3732],{"type":42,"tag":64,"props":3733,"children":3734},{"__ignoreMap":175},[3735],{"type":48,"value":3730},{"type":42,"tag":58,"props":3737,"children":3738},{},[3739],{"type":48,"value":3740},"Context parallelism:",{"type":42,"tag":170,"props":3742,"children":3745},{"className":3743,"code":3744,"language":48},[1357],"components\u002Fdistributed\u002Fcp_utils.py\n    make_cp_batch_and_ctx()            -- creates CP context manager + shards batch\n    create_context_parallel_ctx()      -- wraps torch.distributed.tensor.experimental.context_parallel\n    attach_context_parallel_hooks()    -- strips attention_mask, sets is_causal=True\n    make_cp_batch_for_te()             -- TE-specific CP batch sharding (THD format)\n",[3746],{"type":42,"tag":64,"props":3747,"children":3748},{"__ignoreMap":175},[3749],{"type":48,"value":3744},{"type":42,"tag":58,"props":3751,"children":3752},{},[3753],{"type":48,"value":3754},"Infrastructure orchestration:",{"type":42,"tag":170,"props":3756,"children":3759},{"className":3757,"code":3758,"language":48},[1357],"_transformers\u002Finfrastructure.py\n    instantiate_infrastructure()    -- config objects -> runtime objects\n    apply_model_infrastructure()    -- applies sharding, PEFT, checkpoints to model\n    _shard_pp()                     -- pipeline parallel path\n    _shard_ep_fsdp()                -- EP + FSDP path (non-PP)\n",[3760],{"type":42,"tag":64,"props":3761,"children":3762},{"__ignoreMap":175},[3763],{"type":48,"value":3758},{"type":42,"tag":58,"props":3765,"children":3766},{},[3767],{"type":48,"value":3768},"YAML parsing:",{"type":42,"tag":170,"props":3770,"children":3773},{"className":3771,"code":3772,"language":48},[1357],"recipes\u002F_dist_utils.py\n    parse_distributed_section()  -- YAML dict -> typed configs + sizes\n    create_distributed_setup_from_config()  -- recipe adapter: parse + create DistributedSetup; does not init process group\n",[3774],{"type":42,"tag":64,"props":3775,"children":3776},{"__ignoreMap":175},[3777],{"type":48,"value":3772},{"type":42,"tag":58,"props":3779,"children":3780},{},[3781],{"type":48,"value":3782},"MoE config:",{"type":42,"tag":170,"props":3784,"children":3787},{"className":3785,"code":3786,"language":48},[1357],"components\u002Fdistributed\u002Fconfig.py\n    MoEParallelizerConfig  -- reshard_after_forward, ignore_router_for_ac, wrap_outer_model, etc.\ncomponents\u002Fmoe\u002Fconfig.py\n    MoEConfig              -- n_routed_experts, n_activated_experts, score_func, etc.\n",[3788],{"type":42,"tag":64,"props":3789,"children":3790},{"__ignoreMap":175},[3791],{"type":48,"value":3786},{"type":42,"tag":51,"props":3793,"children":3795},{"id":3794},"pitfalls",[3796],{"type":48,"value":3797},"Pitfalls",{"type":42,"tag":1970,"props":3799,"children":3800},{},[3801,3811,3836,3853,3895,3913,3936,3974,3984,3994,4011,4031],{"type":42,"tag":646,"props":3802,"children":3803},{},[3804,3809],{"type":42,"tag":3107,"props":3805,"children":3806},{},[3807],{"type":48,"value":3808},"TP across nodes destroys throughput.",{"type":48,"value":3810}," Always keep TP within a single\nNVLink domain. Use PP or DP for cross-node scaling.",{"type":42,"tag":646,"props":3812,"children":3813},{},[3814,3826,3828,3834],{"type":42,"tag":3107,"props":3815,"children":3816},{},[3817,3819,3824],{"type":48,"value":3818},"PP requires ",{"type":42,"tag":64,"props":3820,"children":3822},{"className":3821},[],[3823],{"type":48,"value":1705},{"type":48,"value":3825}," on the model class.",{"type":48,"value":3827}," Not all HF models have this.\nCheck ",{"type":42,"tag":64,"props":3829,"children":3831},{"className":3830},[],[3832],{"type":48,"value":3833},"validate_hf_model_for_pipeline_support()",{"type":48,"value":3835}," before enabling PP.",{"type":42,"tag":646,"props":3837,"children":3838},{},[3839,3844,3846,3851],{"type":42,"tag":3107,"props":3840,"children":3841},{},[3842],{"type":48,"value":3843},"PP bubbles reduce GPU utilization.",{"type":48,"value":3845}," Use interleaved schedules\n(",{"type":42,"tag":64,"props":3847,"children":3849},{"className":3848},[],[3850],{"type":48,"value":2057},{"type":48,"value":3852},") and smaller microbatches to reduce bubble time.",{"type":42,"tag":646,"props":3854,"children":3855},{},[3856,3861,3863,3869,3871,3877,3879,3885,3887,3893],{"type":42,"tag":3107,"props":3857,"children":3858},{},[3859],{"type":48,"value":3860},"FSDP2 requires DTensor-aware state dict saving.",{"type":48,"value":3862}," Use ",{"type":42,"tag":64,"props":3864,"children":3866},{"className":3865},[],[3867],{"type":48,"value":3868},"safetensors",{"type":48,"value":3870}," with\n",{"type":42,"tag":64,"props":3872,"children":3874},{"className":3873},[],[3875],{"type":48,"value":3876},"save_consolidated: final",{"type":48,"value":3878}," for final HF export, or ",{"type":42,"tag":64,"props":3880,"children":3882},{"className":3881},[],[3883],{"type":48,"value":3884},"save_consolidated: false",{"type":48,"value":3886},"\nplus the generated ",{"type":42,"tag":64,"props":3888,"children":3890},{"className":3889},[],[3891],{"type":48,"value":3892},"model\u002Fconsolidate.sh",{"type":48,"value":3894}," helper for offline export.",{"type":42,"tag":646,"props":3896,"children":3897},{},[3898,3903,3905,3911],{"type":42,"tag":3107,"props":3899,"children":3900},{},[3901],{"type":48,"value":3902},"CP requires compatible attention.",{"type":48,"value":3904}," SDPA (Flash Attention or Efficient\nAttention) or TE attention only. ",{"type":42,"tag":64,"props":3906,"children":3908},{"className":3907},[],[3909],{"type":48,"value":3910},"SDPBackend.MATH",{"type":48,"value":3912}," is not compatible with\nDTensor.",{"type":42,"tag":646,"props":3914,"children":3915},{},[3916,3927,3929,3935],{"type":42,"tag":3107,"props":3917,"children":3918},{},[3919,3921,3926],{"type":48,"value":3920},"MoE EP size must evenly divide ",{"type":42,"tag":64,"props":3922,"children":3924},{"className":3923},[],[3925],{"type":48,"value":391},{"type":48,"value":168},{"type":48,"value":3928}," The device mesh\ncreation asserts ",{"type":42,"tag":64,"props":3930,"children":3932},{"className":3931},[],[3933],{"type":48,"value":3934},"dp_cp_size % ep_size == 0",{"type":48,"value":168},{"type":42,"tag":646,"props":3937,"children":3938},{},[3939,3944,3946,3951,3953,3958,3960,3965,3967,3972],{"type":42,"tag":3107,"props":3940,"children":3941},{},[3942],{"type":48,"value":3943},"MegatronFSDP is more limited than FSDP2.",{"type":48,"value":3945}," It does not support PP\n(",{"type":42,"tag":64,"props":3947,"children":3949},{"className":3948},[],[3950],{"type":48,"value":1993},{"type":48,"value":3952},"), EP (",{"type":42,"tag":64,"props":3954,"children":3956},{"className":3955},[],[3957],{"type":48,"value":360},{"type":48,"value":3959},"), or ",{"type":42,"tag":64,"props":3961,"children":3963},{"className":3962},[],[3964],{"type":48,"value":503},{"type":48,"value":3966},". The\n",{"type":42,"tag":64,"props":3968,"children":3970},{"className":3969},[],[3971],{"type":48,"value":69},{"type":48,"value":3973}," validation raises on these combinations.",{"type":42,"tag":646,"props":3975,"children":3976},{},[3977,3982],{"type":42,"tag":3107,"props":3978,"children":3979},{},[3980],{"type":48,"value":3981},"DDP supports nothing beyond data parallelism.",{"type":48,"value":3983}," No TP, PP, CP, EP, or\nHSDP. Validation raises on any of these.",{"type":42,"tag":646,"props":3985,"children":3986},{},[3987,3992],{"type":42,"tag":3107,"props":3988,"children":3989},{},[3990],{"type":48,"value":3991},"Activation checkpointing increases compute.",{"type":48,"value":3993}," It saves memory by\nrecomputing activations during backward, but adds ~30% compute overhead.",{"type":42,"tag":646,"props":3995,"children":3996},{},[3997,4002,4004,4010],{"type":42,"tag":3107,"props":3998,"children":3999},{},[4000],{"type":48,"value":4001},"Mixed precision policy must match model expectations.",{"type":48,"value":4003}," The default\nbfloat16 policy works for most models. FP16 models may need a custom\n",{"type":42,"tag":64,"props":4005,"children":4007},{"className":4006},[],[4008],{"type":48,"value":4009},"MixedPrecisionPolicy",{"type":48,"value":168},{"type":42,"tag":646,"props":4012,"children":4013},{},[4014,4029],{"type":42,"tag":3107,"props":4015,"children":4016},{},[4017,4022,4024],{"type":42,"tag":64,"props":4018,"children":4020},{"className":4019},[],[4021],{"type":48,"value":2516},{"type":48,"value":4023}," must be divisible by ",{"type":42,"tag":64,"props":4025,"children":4027},{"className":4026},[],[4028],{"type":48,"value":134},{"type":48,"value":4030}," when using CP\nwith packed sequences.",{"type":42,"tag":646,"props":4032,"children":4033},{},[4034,4045,4047,4052,4054,4059,4061,4067],{"type":42,"tag":3107,"props":4035,"children":4036},{},[4037,4043],{"type":42,"tag":64,"props":4038,"children":4040},{"className":4039},[],[4041],{"type":48,"value":4042},"dp_replicate_size",{"type":48,"value":4044}," is FSDP2-only.",{"type":48,"value":4046}," Passing it with ",{"type":42,"tag":64,"props":4048,"children":4050},{"className":4049},[],[4051],{"type":48,"value":399},{"type":48,"value":4053},"\nor ",{"type":42,"tag":64,"props":4055,"children":4057},{"className":4056},[],[4058],{"type":48,"value":407},{"type":48,"value":4060}," raises a ",{"type":42,"tag":64,"props":4062,"children":4064},{"className":4063},[],[4065],{"type":48,"value":4066},"ValueError",{"type":48,"value":168},{"type":42,"tag":51,"props":4069,"children":4071},{"id":4070},"verification",[4072],{"type":48,"value":4073},"Verification",{"type":42,"tag":58,"props":4075,"children":4076},{},[4077],{"type":48,"value":4078},"Run the smallest recipe that exercises the requested strategy. Success means\nexit code 0, finite loss, no NCCL timeout, and log output matching the expected\nTP\u002FPP\u002FCP\u002FEP sizes.",{"type":42,"tag":4080,"props":4081,"children":4082},"style",{},[4083],{"type":48,"value":4084},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":4086,"total":4190},[4087,4104,4118,4132,4144,4161,4176],{"slug":4088,"name":4088,"fn":4089,"description":4090,"org":4091,"tags":4092,"stars":20,"repoUrl":21,"updatedAt":4103},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4093,4096,4099,4100],{"name":4094,"slug":4095,"type":15},"Data Analysis","data-analysis",{"name":4097,"slug":4098,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":4101,"slug":4102,"type":15},"Performance","performance","2026-07-14T05:28:43.176466",{"slug":4105,"name":4105,"fn":4106,"description":4107,"org":4108,"tags":4109,"stars":20,"repoUrl":21,"updatedAt":4117},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4110,4113,4116],{"name":4111,"slug":4112,"type":15},"Deployment","deployment",{"name":4114,"slug":4115,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":4119,"name":4119,"fn":4120,"description":4121,"org":4122,"tags":4123,"stars":20,"repoUrl":21,"updatedAt":4131},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4124,4127,4128],{"name":4125,"slug":4126,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":4129,"slug":4130,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":4133,"name":4133,"fn":4134,"description":4135,"org":4136,"tags":4137,"stars":20,"repoUrl":21,"updatedAt":4143},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4138,4139,4140],{"name":4094,"slug":4095,"type":15},{"name":9,"slug":8,"type":15},{"name":4141,"slug":4142,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":4145,"name":4145,"fn":4146,"description":4147,"org":4148,"tags":4149,"stars":20,"repoUrl":21,"updatedAt":4160},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4150,4153,4156,4157],{"name":4151,"slug":4152,"type":15},"Automation","automation",{"name":4154,"slug":4155,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":4158,"slug":4159,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":4162,"name":4162,"fn":4163,"description":4164,"org":4165,"tags":4166,"stars":20,"repoUrl":21,"updatedAt":4175},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4167,4168,4171,4172],{"name":4111,"slug":4112,"type":15},{"name":4169,"slug":4170,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":4173,"slug":4174,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":4177,"name":4177,"fn":4178,"description":4179,"org":4180,"tags":4181,"stars":20,"repoUrl":21,"updatedAt":4189},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4182,4183,4186],{"name":9,"slug":8,"type":15},{"name":4184,"slug":4185,"type":15},"Quantum Computing","quantum-computing",{"name":4187,"slug":4188,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":4192,"total":4340},[4193,4211,4226,4237,4249,4263,4276,4288,4299,4308,4322,4331],{"slug":4194,"name":4194,"fn":4195,"description":4196,"org":4197,"tags":4198,"stars":4208,"repoUrl":4209,"updatedAt":4210},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4199,4202,4205],{"name":4200,"slug":4201,"type":15},"Documentation","documentation",{"name":4203,"slug":4204,"type":15},"MCP","mcp",{"name":4206,"slug":4207,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":4212,"name":4212,"fn":4213,"description":4214,"org":4215,"tags":4216,"stars":4223,"repoUrl":4224,"updatedAt":4225},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4217,4220,4221],{"name":4218,"slug":4219,"type":15},"Containers","containers",{"name":4111,"slug":4112,"type":15},{"name":4222,"slug":1565,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":4227,"name":4227,"fn":4228,"description":4229,"org":4230,"tags":4231,"stars":4223,"repoUrl":4224,"updatedAt":4236},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4232,4235],{"name":4233,"slug":4234,"type":15},"CI\u002FCD","ci-cd",{"name":4111,"slug":4112,"type":15},"2026-07-14T05:25:59.97109",{"slug":4238,"name":4238,"fn":4239,"description":4240,"org":4241,"tags":4242,"stars":4223,"repoUrl":4224,"updatedAt":4248},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4243,4244,4245],{"name":4233,"slug":4234,"type":15},{"name":4111,"slug":4112,"type":15},{"name":4246,"slug":4247,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":4250,"name":4250,"fn":4251,"description":4252,"org":4253,"tags":4254,"stars":4223,"repoUrl":4224,"updatedAt":4262},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4255,4258,4259],{"name":4256,"slug":4257,"type":15},"Debugging","debugging",{"name":4246,"slug":4247,"type":15},{"name":4260,"slug":4261,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":4264,"name":4264,"fn":4265,"description":4266,"org":4267,"tags":4268,"stars":4223,"repoUrl":4224,"updatedAt":4275},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4269,4272],{"name":4270,"slug":4271,"type":15},"Best Practices","best-practices",{"name":4273,"slug":4274,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":4277,"name":4277,"fn":4278,"description":4279,"org":4280,"tags":4281,"stars":4223,"repoUrl":4224,"updatedAt":4287},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4282,4283,4286],{"name":13,"slug":14,"type":15},{"name":4284,"slug":4285,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":4289,"name":4289,"fn":4290,"description":4291,"org":4292,"tags":4293,"stars":4223,"repoUrl":4224,"updatedAt":4298},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4294,4297],{"name":4295,"slug":4296,"type":15},"QA","qa",{"name":4141,"slug":4142,"type":15},"2026-07-14T05:25:53.673039",{"slug":4300,"name":4300,"fn":4301,"description":4302,"org":4303,"tags":4304,"stars":4223,"repoUrl":4224,"updatedAt":4307},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4305,4306],{"name":4111,"slug":4112,"type":15},{"name":4114,"slug":4115,"type":15},"2026-07-14T05:25:49.362534",{"slug":4309,"name":4309,"fn":4310,"description":4311,"org":4312,"tags":4313,"stars":4223,"repoUrl":4224,"updatedAt":4321},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4314,4317,4318],{"name":4315,"slug":4316,"type":15},"Code Review","code-review",{"name":4246,"slug":4247,"type":15},{"name":4319,"slug":4320,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":4323,"name":4323,"fn":4324,"description":4325,"org":4326,"tags":4327,"stars":4223,"repoUrl":4224,"updatedAt":4330},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4328,4329],{"name":4295,"slug":4296,"type":15},{"name":4141,"slug":4142,"type":15},"2026-07-14T05:25:54.928983",{"slug":4332,"name":4332,"fn":4333,"description":4334,"org":4335,"tags":4336,"stars":4223,"repoUrl":4224,"updatedAt":4339},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4337,4338],{"name":4151,"slug":4152,"type":15},{"name":4233,"slug":4234,"type":15},"2026-07-30T05:29:03.275638",496]