[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-mcore-run-on-slurm":3,"mdc--1hi3vu-key":33,"related-org-nvidia-mcore-run-on-slurm":1564,"related-repo-nvidia-mcore-run-on-slurm":1713},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":19,"repoUrl":20,"updatedAt":21,"license":22,"forks":23,"topics":24,"repo":28,"sourceUrl":31,"mdContent":32},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16],{"name":13,"slug":14,"type":15},"Deployment","deployment","tag",{"name":17,"slug":18,"type":15},"Infrastructure","infrastructure",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-14T05:25:49.362534","Apache-2.0",4230,[25,26,27],"large-language-models","model-para","transformers",{"repoUrl":20,"stars":19,"forks":23,"topics":29,"description":30},[25,26,27],"Ongoing research training transformer models at scale","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM\u002Ftree\u002FHEAD\u002Fskills\u002Fmcore-run-on-slurm","---\nname: mcore-run-on-slurm\ndescription: How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.\nlicense: Apache-2.0\nwhen_to_use: Submitting a SLURM job; writing or debugging an sbatch script; configuring multi-node distributed training; setting MASTER_ADDR \u002F MASTER_PORT \u002F WORLD_SIZE; diagnosing a SLURM job failure; 'how do I run on the cluster', 'sbatch', 'multi-node training'.\nmetadata:\n  author: Oliver Koenig \u003Cokoenig@nvidia.com>\n---\n\n# Run Megatron-LM on SLURM\n\n## Answer-First Constants\n\nFor text-only SLURM setup questions, answer with these constants before the\nfull script:\n\n- Submit from a shared worktree path visible to every node; `cd` there in the\n  script before launching training.\n- Use one `srun` task per node and launch workers with\n  `uv run python -m torch.distributed.run`, not bare `torchrun`.\n- Set `MASTER_ADDR` from\n  `scontrol show hostnames \"$SLURM_JOB_NODELIST\" | head -n1`, set `MASTER_PORT`,\n  `NNODES=${SLURM_NNODES}`, `GPUS_PER_NODE=\u003CGPUS_PER_NODE>`, and\n  `WORLD_SIZE=$((NNODES * GPUS_PER_NODE))`.\n- Pass `--nnodes`, `--nproc-per-node`, `--node-rank`, `--master-addr`, and\n  `--master-port` to `torch.distributed.run`.\n- `CUDA_DEVICE_MAX_CONNECTIONS`: pre-Blackwell Hopper\u002FAmpere with TP>1 or CP>1\n  and non-FSDP uses `1`; Blackwell\u002FGB200 does not need it; Torch-FSDP2 or\n  Megatron-FSDP must not use `1`; `overlap_moe_expert_parallel_comm` uses `32`.\n\n## Prerequisites\n\n- A SLURM cluster login with submission rights to a GPU partition.\n- Megatron-LM checked out on a filesystem visible to all nodes in the allocation (NFS, Lustre, or similar). All nodes must reach the same paths for code, data, checkpoints, and output.\n- `uv` installed; run `uv sync --extra training --extra dev` (or `--extra lts`) on the worktree once before submission so the `.venv` is materialized and visible to every node.\n\n## Minimal sbatch script\n\nSave as `run_megatron.slurm` in the worktree:\n\n```bash\n#!\u002Fbin\u002Fbash\n#SBATCH --job-name=megatron\n#SBATCH --account=\u003CSLURM_ACCOUNT>\n#SBATCH --partition=\u003CSLURM_PARTITION>\n#SBATCH --nodes=\u003CNODES>\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=\u003CGPUS_PER_NODE>\n#SBATCH --time=\u003CHH:MM:SS>\n#SBATCH --output=logs\u002F%x-%j.out\n#SBATCH --error=logs\u002F%x-%j.err\n\nset -euo pipefail\ncd \u003CMEGATRON_WORKTREE>\n\nexport MASTER_ADDR=$(scontrol show hostnames \"$SLURM_JOB_NODELIST\" | head -n1)\nexport MASTER_PORT=${MASTER_PORT:-29500}\nexport NNODES=${SLURM_NNODES}\nexport GPUS_PER_NODE=\u003CGPUS_PER_NODE>\nexport WORLD_SIZE=$((NNODES * GPUS_PER_NODE))\n\n# Set CUDA_DEVICE_MAX_CONNECTIONS only when your configuration requires it\n# (see the section below). Example for pre-Blackwell with TP>1 or CP>1\n# (non-FSDP):\n#   export CUDA_DEVICE_MAX_CONNECTIONS=1\n\nsrun --ntasks=${NNODES} --ntasks-per-node=1 bash -c '\n  # NODE_RANK comes from SLURM_NODEID with one task per node.\n  NODE_RANK=${SLURM_NODEID}\n  uv run python -m torch.distributed.run \\\n    --nnodes='\"${NNODES}\"' \\\n    --nproc-per-node='\"${GPUS_PER_NODE}\"' \\\n    --node-rank=${NODE_RANK} \\\n    --master-addr='\"${MASTER_ADDR}\"' \\\n    --master-port='\"${MASTER_PORT}\"' \\\n    pretrain_gpt.py \\\n      \u003CMEGATRON_ARGS>\n'\n```\n\nSubmit:\n\n```bash\nmkdir -p logs && JOB_ID=$(sbatch --parsable run_megatron.slurm)\necho \"Submitted ${JOB_ID}\"\n```\n\n## Multi-node rules\n\n- Submit from the worktree you intend to run, or `cd` to it in the script. All nodes must reach the same path on a shared filesystem (NFS, Lustre, or similar) — node-local paths will not be visible to peer ranks.\n- Use one `torchrun` worker group across all nodes; do not start independent single-node jobs.\n- `--nproc-per-node` should equal the number of visible GPUs per node.\n- Write checkpoints, tensorboard data, and structured logs to shared storage.\n\n## CUDA_DEVICE_MAX_CONNECTIONS\n\nThe right value depends on your hardware and parallelism mode. Do not export it unconditionally:\n\n- **Pre-Blackwell (Hopper, Ampere) with TP>1 or CP>1, non-FSDP:** set to `1`. The relevant code path asserts on this — you will get an assertion error if it is not `1`, not a silent deadlock.\n- **Blackwell:** not required; setting it has no effect.\n- **Torch-FSDP2 or Megatron-FSDP:** must NOT be `1`. Leave the env var unset, or set it to a value greater than `1`.\n- **`overlap_moe_expert_parallel_comm` enabled:** set to `32`.\n\nSet it explicitly in the sbatch script when your configuration calls for it.\n\n## Containers\n\nMany sites run Megatron-LM inside a container (enroot\u002Fpyxis on some clusters, singularity on others). If you do, the uv-managed `.venv` must live on a path that is visible from inside the container, and the container image must provide the CUDA \u002F NCCL \u002F torch versions the repo expects (see `docker\u002F.ngc_version.dev` and `.ngc_version.lts`). The skeleton above stays the same; wrap the `srun` invocation with your scheduler's container flags (`--container-image=…`, `--container-mounts=…`, etc.).\n\n## Monitor and collect\n\n```bash\nsqueue -j \"$JOB_ID\" -o \"%.10i %.8T %.10M %.6D %R\"\nsacct -j \"$JOB_ID\" --format=JobID,State,ExitCode,Elapsed\nscancel \"$JOB_ID\"\n```\n\nIf your training script writes a result artifact (a JSON metrics file from rank 0, a final checkpoint, etc.), poll for the artifact rather than waiting only on `squeue` state. Useful output usually appears before SLURM marks the job complete, and polling on the artifact lets you cancel the job as soon as it lands instead of holding the allocation until the timeout.\n\n## Failure diagnosis\n\nScan stderr from every rank, not just rank 0. The earliest non-NCCL Python traceback is usually the root cause; later NCCL timeouts on other ranks are downstream symptoms of the first crash.\n\nClassify quickly:\n\n- **OOM**: record rank, phase (forward \u002F backward \u002F optimizer), batch size, sequence length, parallelism (TP\u002FDP\u002FCP\u002FPP), and peak memory before adjusting.\n- **Shape \u002F divisibility error**: check `WORLD_SIZE = TP × DP × CP × PP` and head-count divisibility (`num_attention_heads % TP == 0`).\n- **Import error**: wrong worktree, missing `uv sync`, or stale `PYTHONPATH`. Confirm `cd \u003CMEGATRON_WORKTREE>` before launch.\n- **NCCL failure** with no Python traceback: verify allocation, port reachability, `MASTER_ADDR` resolution, and command consistency across ranks.\n\n## Common pitfalls\n\n- Forgetting `uv sync` before the first submission. If the venv is missing, every job rebuilds it from inside `srun`, costing minutes per job.\n- Writing logs to a node-local path that disappears at job exit. Always write to the shared filesystem.\n- Setting `CUDA_DEVICE_MAX_CONNECTIONS=1` blindly. The right value depends on hardware and parallelism mode (see the dedicated section above). Setting it to `1` with FSDP causes a different problem; on Blackwell it has no effect; on pre-Blackwell with TP>1 or CP>1 (non-FSDP) the code asserts, it does not deadlock.\n- Running bare `torchrun` instead of `uv run python -m torch.distributed.run`. Bare `torchrun` may dispatch through a python interpreter that does not see venv packages, depending on how the venv is set up.\n",{"data":34,"body":38},{"name":4,"description":6,"license":22,"when_to_use":35,"metadata":36},"Submitting a SLURM job; writing or debugging an sbatch script; configuring multi-node distributed training; setting MASTER_ADDR \u002F MASTER_PORT \u002F WORLD_SIZE; diagnosing a SLURM job failure; 'how do I run on the cluster', 'sbatch', 'multi-node training'.",{"author":37},"Oliver Koenig \u003Cokoenig@nvidia.com>",{"type":39,"children":40},"root",[41,50,57,63,252,258,306,312,325,961,966,1055,1061,1102,1107,1112,1193,1198,1204,1254,1260,1361,1373,1379,1384,1389,1479,1485,1558],{"type":42,"tag":43,"props":44,"children":46},"element","h1",{"id":45},"run-megatron-lm-on-slurm",[47],{"type":48,"value":49},"text","Run Megatron-LM on SLURM",{"type":42,"tag":51,"props":52,"children":54},"h2",{"id":53},"answer-first-constants",[55],{"type":48,"value":56},"Answer-First Constants",{"type":42,"tag":58,"props":59,"children":60},"p",{},[61],{"type":48,"value":62},"For text-only SLURM setup questions, answer with these constants before the\nfull script:",{"type":42,"tag":64,"props":65,"children":66},"ul",{},[67,82,111,163,211],{"type":42,"tag":68,"props":69,"children":70},"li",{},[71,73,80],{"type":48,"value":72},"Submit from a shared worktree path visible to every node; ",{"type":42,"tag":74,"props":75,"children":77},"code",{"className":76},[],[78],{"type":48,"value":79},"cd",{"type":48,"value":81}," there in the\nscript before launching training.",{"type":42,"tag":68,"props":83,"children":84},{},[85,87,93,95,101,103,109],{"type":48,"value":86},"Use one ",{"type":42,"tag":74,"props":88,"children":90},{"className":89},[],[91],{"type":48,"value":92},"srun",{"type":48,"value":94}," task per node and launch workers with\n",{"type":42,"tag":74,"props":96,"children":98},{"className":97},[],[99],{"type":48,"value":100},"uv run python -m torch.distributed.run",{"type":48,"value":102},", not bare ",{"type":42,"tag":74,"props":104,"children":106},{"className":105},[],[107],{"type":48,"value":108},"torchrun",{"type":48,"value":110},".",{"type":42,"tag":68,"props":112,"children":113},{},[114,116,122,124,130,132,138,140,146,148,154,156,162],{"type":48,"value":115},"Set ",{"type":42,"tag":74,"props":117,"children":119},{"className":118},[],[120],{"type":48,"value":121},"MASTER_ADDR",{"type":48,"value":123}," from\n",{"type":42,"tag":74,"props":125,"children":127},{"className":126},[],[128],{"type":48,"value":129},"scontrol show hostnames \"$SLURM_JOB_NODELIST\" | head -n1",{"type":48,"value":131},", set ",{"type":42,"tag":74,"props":133,"children":135},{"className":134},[],[136],{"type":48,"value":137},"MASTER_PORT",{"type":48,"value":139},",\n",{"type":42,"tag":74,"props":141,"children":143},{"className":142},[],[144],{"type":48,"value":145},"NNODES=${SLURM_NNODES}",{"type":48,"value":147},", ",{"type":42,"tag":74,"props":149,"children":151},{"className":150},[],[152],{"type":48,"value":153},"GPUS_PER_NODE=\u003CGPUS_PER_NODE>",{"type":48,"value":155},", and\n",{"type":42,"tag":74,"props":157,"children":159},{"className":158},[],[160],{"type":48,"value":161},"WORLD_SIZE=$((NNODES * GPUS_PER_NODE))",{"type":48,"value":110},{"type":42,"tag":68,"props":164,"children":165},{},[166,168,174,175,181,182,188,189,195,196,202,204,210],{"type":48,"value":167},"Pass ",{"type":42,"tag":74,"props":169,"children":171},{"className":170},[],[172],{"type":48,"value":173},"--nnodes",{"type":48,"value":147},{"type":42,"tag":74,"props":176,"children":178},{"className":177},[],[179],{"type":48,"value":180},"--nproc-per-node",{"type":48,"value":147},{"type":42,"tag":74,"props":183,"children":185},{"className":184},[],[186],{"type":48,"value":187},"--node-rank",{"type":48,"value":147},{"type":42,"tag":74,"props":190,"children":192},{"className":191},[],[193],{"type":48,"value":194},"--master-addr",{"type":48,"value":155},{"type":42,"tag":74,"props":197,"children":199},{"className":198},[],[200],{"type":48,"value":201},"--master-port",{"type":48,"value":203}," to ",{"type":42,"tag":74,"props":205,"children":207},{"className":206},[],[208],{"type":48,"value":209},"torch.distributed.run",{"type":48,"value":110},{"type":42,"tag":68,"props":212,"children":213},{},[214,220,222,228,230,235,237,243,245,251],{"type":42,"tag":74,"props":215,"children":217},{"className":216},[],[218],{"type":48,"value":219},"CUDA_DEVICE_MAX_CONNECTIONS",{"type":48,"value":221},": pre-Blackwell Hopper\u002FAmpere with TP>1 or CP>1\nand non-FSDP uses ",{"type":42,"tag":74,"props":223,"children":225},{"className":224},[],[226],{"type":48,"value":227},"1",{"type":48,"value":229},"; Blackwell\u002FGB200 does not need it; Torch-FSDP2 or\nMegatron-FSDP must not use ",{"type":42,"tag":74,"props":231,"children":233},{"className":232},[],[234],{"type":48,"value":227},{"type":48,"value":236},"; ",{"type":42,"tag":74,"props":238,"children":240},{"className":239},[],[241],{"type":48,"value":242},"overlap_moe_expert_parallel_comm",{"type":48,"value":244}," uses ",{"type":42,"tag":74,"props":246,"children":248},{"className":247},[],[249],{"type":48,"value":250},"32",{"type":48,"value":110},{"type":42,"tag":51,"props":253,"children":255},{"id":254},"prerequisites",[256],{"type":48,"value":257},"Prerequisites",{"type":42,"tag":64,"props":259,"children":260},{},[261,266,271],{"type":42,"tag":68,"props":262,"children":263},{},[264],{"type":48,"value":265},"A SLURM cluster login with submission rights to a GPU partition.",{"type":42,"tag":68,"props":267,"children":268},{},[269],{"type":48,"value":270},"Megatron-LM checked out on a filesystem visible to all nodes in the allocation (NFS, Lustre, or similar). All nodes must reach the same paths for code, data, checkpoints, and output.",{"type":42,"tag":68,"props":272,"children":273},{},[274,280,282,288,290,296,298,304],{"type":42,"tag":74,"props":275,"children":277},{"className":276},[],[278],{"type":48,"value":279},"uv",{"type":48,"value":281}," installed; run ",{"type":42,"tag":74,"props":283,"children":285},{"className":284},[],[286],{"type":48,"value":287},"uv sync --extra training --extra dev",{"type":48,"value":289}," (or ",{"type":42,"tag":74,"props":291,"children":293},{"className":292},[],[294],{"type":48,"value":295},"--extra lts",{"type":48,"value":297},") on the worktree once before submission so the ",{"type":42,"tag":74,"props":299,"children":301},{"className":300},[],[302],{"type":48,"value":303},".venv",{"type":48,"value":305}," is materialized and visible to every node.",{"type":42,"tag":51,"props":307,"children":309},{"id":308},"minimal-sbatch-script",[310],{"type":48,"value":311},"Minimal sbatch script",{"type":42,"tag":58,"props":313,"children":314},{},[315,317,323],{"type":48,"value":316},"Save as ",{"type":42,"tag":74,"props":318,"children":320},{"className":319},[],[321],{"type":48,"value":322},"run_megatron.slurm",{"type":48,"value":324}," in the worktree:",{"type":42,"tag":326,"props":327,"children":332},"pre",{"className":328,"code":329,"language":330,"meta":331,"style":331},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","#!\u002Fbin\u002Fbash\n#SBATCH --job-name=megatron\n#SBATCH --account=\u003CSLURM_ACCOUNT>\n#SBATCH --partition=\u003CSLURM_PARTITION>\n#SBATCH --nodes=\u003CNODES>\n#SBATCH --ntasks-per-node=1\n#SBATCH --gpus-per-node=\u003CGPUS_PER_NODE>\n#SBATCH --time=\u003CHH:MM:SS>\n#SBATCH --output=logs\u002F%x-%j.out\n#SBATCH --error=logs\u002F%x-%j.err\n\nset -euo pipefail\ncd \u003CMEGATRON_WORKTREE>\n\nexport MASTER_ADDR=$(scontrol show hostnames \"$SLURM_JOB_NODELIST\" | head -n1)\nexport MASTER_PORT=${MASTER_PORT:-29500}\nexport NNODES=${SLURM_NNODES}\nexport GPUS_PER_NODE=\u003CGPUS_PER_NODE>\nexport WORLD_SIZE=$((NNODES * GPUS_PER_NODE))\n\n# Set CUDA_DEVICE_MAX_CONNECTIONS only when your configuration requires it\n# (see the section below). Example for pre-Blackwell with TP>1 or CP>1\n# (non-FSDP):\n#   export CUDA_DEVICE_MAX_CONNECTIONS=1\n\nsrun --ntasks=${NNODES} --ntasks-per-node=1 bash -c '\n  # NODE_RANK comes from SLURM_NODEID with one task per node.\n  NODE_RANK=${SLURM_NODEID}\n  uv run python -m torch.distributed.run \\\n    --nnodes='\"${NNODES}\"' \\\n    --nproc-per-node='\"${GPUS_PER_NODE}\"' \\\n    --node-rank=${NODE_RANK} \\\n    --master-addr='\"${MASTER_ADDR}\"' \\\n    --master-port='\"${MASTER_PORT}\"' \\\n    pretrain_gpt.py \\\n      \u003CMEGATRON_ARGS>\n'\n","bash","",[333],{"type":42,"tag":74,"props":334,"children":335},{"__ignoreMap":331},[336,348,357,366,375,384,393,402,411,420,429,439,460,490,498,569,606,632,659,696,704,713,722,731,740,748,795,804,813,822,850,875,884,909,934,943,952],{"type":42,"tag":337,"props":338,"children":341},"span",{"class":339,"line":340},"line",1,[342],{"type":42,"tag":337,"props":343,"children":345},{"style":344},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[346],{"type":48,"value":347},"#!\u002Fbin\u002Fbash\n",{"type":42,"tag":337,"props":349,"children":351},{"class":339,"line":350},2,[352],{"type":42,"tag":337,"props":353,"children":354},{"style":344},[355],{"type":48,"value":356},"#SBATCH --job-name=megatron\n",{"type":42,"tag":337,"props":358,"children":360},{"class":339,"line":359},3,[361],{"type":42,"tag":337,"props":362,"children":363},{"style":344},[364],{"type":48,"value":365},"#SBATCH --account=\u003CSLURM_ACCOUNT>\n",{"type":42,"tag":337,"props":367,"children":369},{"class":339,"line":368},4,[370],{"type":42,"tag":337,"props":371,"children":372},{"style":344},[373],{"type":48,"value":374},"#SBATCH --partition=\u003CSLURM_PARTITION>\n",{"type":42,"tag":337,"props":376,"children":378},{"class":339,"line":377},5,[379],{"type":42,"tag":337,"props":380,"children":381},{"style":344},[382],{"type":48,"value":383},"#SBATCH --nodes=\u003CNODES>\n",{"type":42,"tag":337,"props":385,"children":387},{"class":339,"line":386},6,[388],{"type":42,"tag":337,"props":389,"children":390},{"style":344},[391],{"type":48,"value":392},"#SBATCH --ntasks-per-node=1\n",{"type":42,"tag":337,"props":394,"children":396},{"class":339,"line":395},7,[397],{"type":42,"tag":337,"props":398,"children":399},{"style":344},[400],{"type":48,"value":401},"#SBATCH --gpus-per-node=\u003CGPUS_PER_NODE>\n",{"type":42,"tag":337,"props":403,"children":405},{"class":339,"line":404},8,[406],{"type":42,"tag":337,"props":407,"children":408},{"style":344},[409],{"type":48,"value":410},"#SBATCH --time=\u003CHH:MM:SS>\n",{"type":42,"tag":337,"props":412,"children":414},{"class":339,"line":413},9,[415],{"type":42,"tag":337,"props":416,"children":417},{"style":344},[418],{"type":48,"value":419},"#SBATCH --output=logs\u002F%x-%j.out\n",{"type":42,"tag":337,"props":421,"children":423},{"class":339,"line":422},10,[424],{"type":42,"tag":337,"props":425,"children":426},{"style":344},[427],{"type":48,"value":428},"#SBATCH --error=logs\u002F%x-%j.err\n",{"type":42,"tag":337,"props":430,"children":432},{"class":339,"line":431},11,[433],{"type":42,"tag":337,"props":434,"children":436},{"emptyLinePlaceholder":435},true,[437],{"type":48,"value":438},"\n",{"type":42,"tag":337,"props":440,"children":442},{"class":339,"line":441},12,[443,449,455],{"type":42,"tag":337,"props":444,"children":446},{"style":445},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[447],{"type":48,"value":448},"set",{"type":42,"tag":337,"props":450,"children":452},{"style":451},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[453],{"type":48,"value":454}," -euo",{"type":42,"tag":337,"props":456,"children":457},{"style":451},[458],{"type":48,"value":459}," pipefail\n",{"type":42,"tag":337,"props":461,"children":463},{"class":339,"line":462},13,[464,468,474,479,485],{"type":42,"tag":337,"props":465,"children":466},{"style":445},[467],{"type":48,"value":79},{"type":42,"tag":337,"props":469,"children":471},{"style":470},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[472],{"type":48,"value":473}," \u003C",{"type":42,"tag":337,"props":475,"children":476},{"style":451},[477],{"type":48,"value":478},"MEGATRON_WORKTRE",{"type":42,"tag":337,"props":480,"children":482},{"style":481},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[483],{"type":48,"value":484},"E",{"type":42,"tag":337,"props":486,"children":487},{"style":470},[488],{"type":48,"value":489},">\n",{"type":42,"tag":337,"props":491,"children":493},{"class":339,"line":492},14,[494],{"type":42,"tag":337,"props":495,"children":496},{"emptyLinePlaceholder":435},[497],{"type":48,"value":438},{"type":42,"tag":337,"props":499,"children":501},{"class":339,"line":500},15,[502,508,513,518,524,529,534,539,544,549,554,559,564],{"type":42,"tag":337,"props":503,"children":505},{"style":504},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[506],{"type":48,"value":507},"export",{"type":42,"tag":337,"props":509,"children":510},{"style":481},[511],{"type":48,"value":512}," MASTER_ADDR",{"type":42,"tag":337,"props":514,"children":515},{"style":470},[516],{"type":48,"value":517},"=$(",{"type":42,"tag":337,"props":519,"children":521},{"style":520},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[522],{"type":48,"value":523},"scontrol",{"type":42,"tag":337,"props":525,"children":526},{"style":451},[527],{"type":48,"value":528}," show",{"type":42,"tag":337,"props":530,"children":531},{"style":451},[532],{"type":48,"value":533}," hostnames",{"type":42,"tag":337,"props":535,"children":536},{"style":470},[537],{"type":48,"value":538}," \"",{"type":42,"tag":337,"props":540,"children":541},{"style":481},[542],{"type":48,"value":543},"$SLURM_JOB_NODELIST",{"type":42,"tag":337,"props":545,"children":546},{"style":470},[547],{"type":48,"value":548},"\"",{"type":42,"tag":337,"props":550,"children":551},{"style":470},[552],{"type":48,"value":553}," |",{"type":42,"tag":337,"props":555,"children":556},{"style":520},[557],{"type":48,"value":558}," head",{"type":42,"tag":337,"props":560,"children":561},{"style":451},[562],{"type":48,"value":563}," -n1",{"type":42,"tag":337,"props":565,"children":566},{"style":470},[567],{"type":48,"value":568},")\n",{"type":42,"tag":337,"props":570,"children":572},{"class":339,"line":571},16,[573,577,582,587,591,596,601],{"type":42,"tag":337,"props":574,"children":575},{"style":504},[576],{"type":48,"value":507},{"type":42,"tag":337,"props":578,"children":579},{"style":481},[580],{"type":48,"value":581}," MASTER_PORT",{"type":42,"tag":337,"props":583,"children":584},{"style":470},[585],{"type":48,"value":586},"=${",{"type":42,"tag":337,"props":588,"children":589},{"style":481},[590],{"type":48,"value":137},{"type":42,"tag":337,"props":592,"children":593},{"style":470},[594],{"type":48,"value":595},":-",{"type":42,"tag":337,"props":597,"children":598},{"style":481},[599],{"type":48,"value":600},"29500",{"type":42,"tag":337,"props":602,"children":603},{"style":470},[604],{"type":48,"value":605},"}\n",{"type":42,"tag":337,"props":607,"children":609},{"class":339,"line":608},17,[610,614,619,623,628],{"type":42,"tag":337,"props":611,"children":612},{"style":504},[613],{"type":48,"value":507},{"type":42,"tag":337,"props":615,"children":616},{"style":481},[617],{"type":48,"value":618}," NNODES",{"type":42,"tag":337,"props":620,"children":621},{"style":470},[622],{"type":48,"value":586},{"type":42,"tag":337,"props":624,"children":625},{"style":481},[626],{"type":48,"value":627},"SLURM_NNODES",{"type":42,"tag":337,"props":629,"children":630},{"style":470},[631],{"type":48,"value":605},{"type":42,"tag":337,"props":633,"children":635},{"class":339,"line":634},18,[636,640,645,650,655],{"type":42,"tag":337,"props":637,"children":638},{"style":504},[639],{"type":48,"value":507},{"type":42,"tag":337,"props":641,"children":642},{"style":481},[643],{"type":48,"value":644}," GPUS_PER_NODE",{"type":42,"tag":337,"props":646,"children":647},{"style":470},[648],{"type":48,"value":649},"=\u003C",{"type":42,"tag":337,"props":651,"children":652},{"style":481},[653],{"type":48,"value":654},"GPUS_PER_NODE",{"type":42,"tag":337,"props":656,"children":657},{"style":470},[658],{"type":48,"value":489},{"type":42,"tag":337,"props":660,"children":662},{"class":339,"line":661},19,[663,667,672,677,682,687,691],{"type":42,"tag":337,"props":664,"children":665},{"style":504},[666],{"type":48,"value":507},{"type":42,"tag":337,"props":668,"children":669},{"style":481},[670],{"type":48,"value":671}," WORLD_SIZE",{"type":42,"tag":337,"props":673,"children":674},{"style":470},[675],{"type":48,"value":676},"=$((",{"type":42,"tag":337,"props":678,"children":679},{"style":520},[680],{"type":48,"value":681},"NNODES",{"type":42,"tag":337,"props":683,"children":684},{"style":481},[685],{"type":48,"value":686}," *",{"type":42,"tag":337,"props":688,"children":689},{"style":451},[690],{"type":48,"value":644},{"type":42,"tag":337,"props":692,"children":693},{"style":470},[694],{"type":48,"value":695},"))\n",{"type":42,"tag":337,"props":697,"children":699},{"class":339,"line":698},20,[700],{"type":42,"tag":337,"props":701,"children":702},{"emptyLinePlaceholder":435},[703],{"type":48,"value":438},{"type":42,"tag":337,"props":705,"children":707},{"class":339,"line":706},21,[708],{"type":42,"tag":337,"props":709,"children":710},{"style":344},[711],{"type":48,"value":712},"# Set CUDA_DEVICE_MAX_CONNECTIONS only when your configuration requires it\n",{"type":42,"tag":337,"props":714,"children":716},{"class":339,"line":715},22,[717],{"type":42,"tag":337,"props":718,"children":719},{"style":344},[720],{"type":48,"value":721},"# (see the section below). Example for pre-Blackwell with TP>1 or CP>1\n",{"type":42,"tag":337,"props":723,"children":725},{"class":339,"line":724},23,[726],{"type":42,"tag":337,"props":727,"children":728},{"style":344},[729],{"type":48,"value":730},"# (non-FSDP):\n",{"type":42,"tag":337,"props":732,"children":734},{"class":339,"line":733},24,[735],{"type":42,"tag":337,"props":736,"children":737},{"style":344},[738],{"type":48,"value":739},"#   export CUDA_DEVICE_MAX_CONNECTIONS=1\n",{"type":42,"tag":337,"props":741,"children":743},{"class":339,"line":742},25,[744],{"type":42,"tag":337,"props":745,"children":746},{"emptyLinePlaceholder":435},[747],{"type":48,"value":438},{"type":42,"tag":337,"props":749,"children":751},{"class":339,"line":750},26,[752,756,761,766,770,775,780,785,790],{"type":42,"tag":337,"props":753,"children":754},{"style":520},[755],{"type":48,"value":92},{"type":42,"tag":337,"props":757,"children":758},{"style":451},[759],{"type":48,"value":760}," --ntasks=",{"type":42,"tag":337,"props":762,"children":763},{"style":470},[764],{"type":48,"value":765},"${",{"type":42,"tag":337,"props":767,"children":768},{"style":481},[769],{"type":48,"value":681},{"type":42,"tag":337,"props":771,"children":772},{"style":470},[773],{"type":48,"value":774},"}",{"type":42,"tag":337,"props":776,"children":777},{"style":451},[778],{"type":48,"value":779}," --ntasks-per-node=1",{"type":42,"tag":337,"props":781,"children":782},{"style":451},[783],{"type":48,"value":784}," bash",{"type":42,"tag":337,"props":786,"children":787},{"style":451},[788],{"type":48,"value":789}," -c",{"type":42,"tag":337,"props":791,"children":792},{"style":470},[793],{"type":48,"value":794}," '\n",{"type":42,"tag":337,"props":796,"children":798},{"class":339,"line":797},27,[799],{"type":42,"tag":337,"props":800,"children":801},{"style":451},[802],{"type":48,"value":803},"  # NODE_RANK comes from SLURM_NODEID with one task per node.\n",{"type":42,"tag":337,"props":805,"children":807},{"class":339,"line":806},28,[808],{"type":42,"tag":337,"props":809,"children":810},{"style":451},[811],{"type":48,"value":812},"  NODE_RANK=${SLURM_NODEID}\n",{"type":42,"tag":337,"props":814,"children":816},{"class":339,"line":815},29,[817],{"type":42,"tag":337,"props":818,"children":819},{"style":451},[820],{"type":48,"value":821},"  uv run python -m torch.distributed.run \\\n",{"type":42,"tag":337,"props":823,"children":825},{"class":339,"line":824},30,[826,831,836,840,845],{"type":42,"tag":337,"props":827,"children":828},{"style":451},[829],{"type":48,"value":830},"    --nnodes=",{"type":42,"tag":337,"props":832,"children":833},{"style":470},[834],{"type":48,"value":835},"'\"${",{"type":42,"tag":337,"props":837,"children":838},{"style":481},[839],{"type":48,"value":681},{"type":42,"tag":337,"props":841,"children":842},{"style":470},[843],{"type":48,"value":844},"}\"'",{"type":42,"tag":337,"props":846,"children":847},{"style":451},[848],{"type":48,"value":849}," \\\n",{"type":42,"tag":337,"props":851,"children":853},{"class":339,"line":852},31,[854,859,863,867,871],{"type":42,"tag":337,"props":855,"children":856},{"style":451},[857],{"type":48,"value":858},"    --nproc-per-node=",{"type":42,"tag":337,"props":860,"children":861},{"style":470},[862],{"type":48,"value":835},{"type":42,"tag":337,"props":864,"children":865},{"style":481},[866],{"type":48,"value":654},{"type":42,"tag":337,"props":868,"children":869},{"style":470},[870],{"type":48,"value":844},{"type":42,"tag":337,"props":872,"children":873},{"style":451},[874],{"type":48,"value":849},{"type":42,"tag":337,"props":876,"children":878},{"class":339,"line":877},32,[879],{"type":42,"tag":337,"props":880,"children":881},{"style":451},[882],{"type":48,"value":883},"    --node-rank=${NODE_RANK} \\\n",{"type":42,"tag":337,"props":885,"children":887},{"class":339,"line":886},33,[888,893,897,901,905],{"type":42,"tag":337,"props":889,"children":890},{"style":451},[891],{"type":48,"value":892},"    --master-addr=",{"type":42,"tag":337,"props":894,"children":895},{"style":470},[896],{"type":48,"value":835},{"type":42,"tag":337,"props":898,"children":899},{"style":481},[900],{"type":48,"value":121},{"type":42,"tag":337,"props":902,"children":903},{"style":470},[904],{"type":48,"value":844},{"type":42,"tag":337,"props":906,"children":907},{"style":451},[908],{"type":48,"value":849},{"type":42,"tag":337,"props":910,"children":912},{"class":339,"line":911},34,[913,918,922,926,930],{"type":42,"tag":337,"props":914,"children":915},{"style":451},[916],{"type":48,"value":917},"    --master-port=",{"type":42,"tag":337,"props":919,"children":920},{"style":470},[921],{"type":48,"value":835},{"type":42,"tag":337,"props":923,"children":924},{"style":481},[925],{"type":48,"value":137},{"type":42,"tag":337,"props":927,"children":928},{"style":470},[929],{"type":48,"value":844},{"type":42,"tag":337,"props":931,"children":932},{"style":451},[933],{"type":48,"value":849},{"type":42,"tag":337,"props":935,"children":937},{"class":339,"line":936},35,[938],{"type":42,"tag":337,"props":939,"children":940},{"style":451},[941],{"type":48,"value":942},"    pretrain_gpt.py \\\n",{"type":42,"tag":337,"props":944,"children":946},{"class":339,"line":945},36,[947],{"type":42,"tag":337,"props":948,"children":949},{"style":451},[950],{"type":48,"value":951},"      \u003CMEGATRON_ARGS>\n",{"type":42,"tag":337,"props":953,"children":955},{"class":339,"line":954},37,[956],{"type":42,"tag":337,"props":957,"children":958},{"style":470},[959],{"type":48,"value":960},"'\n",{"type":42,"tag":58,"props":962,"children":963},{},[964],{"type":48,"value":965},"Submit:",{"type":42,"tag":326,"props":967,"children":969},{"className":328,"code":968,"language":330,"meta":331,"style":331},"mkdir -p logs && JOB_ID=$(sbatch --parsable run_megatron.slurm)\necho \"Submitted ${JOB_ID}\"\n",[970],{"type":42,"tag":74,"props":971,"children":972},{"__ignoreMap":331},[973,1024],{"type":42,"tag":337,"props":974,"children":975},{"class":339,"line":340},[976,981,986,991,996,1001,1005,1010,1015,1020],{"type":42,"tag":337,"props":977,"children":978},{"style":520},[979],{"type":48,"value":980},"mkdir",{"type":42,"tag":337,"props":982,"children":983},{"style":451},[984],{"type":48,"value":985}," -p",{"type":42,"tag":337,"props":987,"children":988},{"style":451},[989],{"type":48,"value":990}," logs",{"type":42,"tag":337,"props":992,"children":993},{"style":470},[994],{"type":48,"value":995}," &&",{"type":42,"tag":337,"props":997,"children":998},{"style":481},[999],{"type":48,"value":1000}," JOB_ID",{"type":42,"tag":337,"props":1002,"children":1003},{"style":470},[1004],{"type":48,"value":517},{"type":42,"tag":337,"props":1006,"children":1007},{"style":520},[1008],{"type":48,"value":1009},"sbatch",{"type":42,"tag":337,"props":1011,"children":1012},{"style":451},[1013],{"type":48,"value":1014}," --parsable",{"type":42,"tag":337,"props":1016,"children":1017},{"style":451},[1018],{"type":48,"value":1019}," run_megatron.slurm",{"type":42,"tag":337,"props":1021,"children":1022},{"style":470},[1023],{"type":48,"value":568},{"type":42,"tag":337,"props":1025,"children":1026},{"class":339,"line":350},[1027,1032,1036,1041,1045,1050],{"type":42,"tag":337,"props":1028,"children":1029},{"style":445},[1030],{"type":48,"value":1031},"echo",{"type":42,"tag":337,"props":1033,"children":1034},{"style":470},[1035],{"type":48,"value":538},{"type":42,"tag":337,"props":1037,"children":1038},{"style":451},[1039],{"type":48,"value":1040},"Submitted ",{"type":42,"tag":337,"props":1042,"children":1043},{"style":470},[1044],{"type":48,"value":765},{"type":42,"tag":337,"props":1046,"children":1047},{"style":481},[1048],{"type":48,"value":1049},"JOB_ID",{"type":42,"tag":337,"props":1051,"children":1052},{"style":470},[1053],{"type":48,"value":1054},"}\"\n",{"type":42,"tag":51,"props":1056,"children":1058},{"id":1057},"multi-node-rules",[1059],{"type":48,"value":1060},"Multi-node rules",{"type":42,"tag":64,"props":1062,"children":1063},{},[1064,1076,1087,1097],{"type":42,"tag":68,"props":1065,"children":1066},{},[1067,1069,1074],{"type":48,"value":1068},"Submit from the worktree you intend to run, or ",{"type":42,"tag":74,"props":1070,"children":1072},{"className":1071},[],[1073],{"type":48,"value":79},{"type":48,"value":1075}," to it in the script. All nodes must reach the same path on a shared filesystem (NFS, Lustre, or similar) — node-local paths will not be visible to peer ranks.",{"type":42,"tag":68,"props":1077,"children":1078},{},[1079,1080,1085],{"type":48,"value":86},{"type":42,"tag":74,"props":1081,"children":1083},{"className":1082},[],[1084],{"type":48,"value":108},{"type":48,"value":1086}," worker group across all nodes; do not start independent single-node jobs.",{"type":42,"tag":68,"props":1088,"children":1089},{},[1090,1095],{"type":42,"tag":74,"props":1091,"children":1093},{"className":1092},[],[1094],{"type":48,"value":180},{"type":48,"value":1096}," should equal the number of visible GPUs per node.",{"type":42,"tag":68,"props":1098,"children":1099},{},[1100],{"type":48,"value":1101},"Write checkpoints, tensorboard data, and structured logs to shared storage.",{"type":42,"tag":51,"props":1103,"children":1105},{"id":1104},"cuda_device_max_connections",[1106],{"type":48,"value":219},{"type":42,"tag":58,"props":1108,"children":1109},{},[1110],{"type":48,"value":1111},"The right value depends on your hardware and parallelism mode. Do not export it unconditionally:",{"type":42,"tag":64,"props":1113,"children":1114},{},[1115,1140,1150,1173],{"type":42,"tag":68,"props":1116,"children":1117},{},[1118,1124,1126,1131,1133,1138],{"type":42,"tag":1119,"props":1120,"children":1121},"strong",{},[1122],{"type":48,"value":1123},"Pre-Blackwell (Hopper, Ampere) with TP>1 or CP>1, non-FSDP:",{"type":48,"value":1125}," set to ",{"type":42,"tag":74,"props":1127,"children":1129},{"className":1128},[],[1130],{"type":48,"value":227},{"type":48,"value":1132},". The relevant code path asserts on this — you will get an assertion error if it is not ",{"type":42,"tag":74,"props":1134,"children":1136},{"className":1135},[],[1137],{"type":48,"value":227},{"type":48,"value":1139},", not a silent deadlock.",{"type":42,"tag":68,"props":1141,"children":1142},{},[1143,1148],{"type":42,"tag":1119,"props":1144,"children":1145},{},[1146],{"type":48,"value":1147},"Blackwell:",{"type":48,"value":1149}," not required; setting it has no effect.",{"type":42,"tag":68,"props":1151,"children":1152},{},[1153,1158,1160,1165,1167,1172],{"type":42,"tag":1119,"props":1154,"children":1155},{},[1156],{"type":48,"value":1157},"Torch-FSDP2 or Megatron-FSDP:",{"type":48,"value":1159}," must NOT be ",{"type":42,"tag":74,"props":1161,"children":1163},{"className":1162},[],[1164],{"type":48,"value":227},{"type":48,"value":1166},". Leave the env var unset, or set it to a value greater than ",{"type":42,"tag":74,"props":1168,"children":1170},{"className":1169},[],[1171],{"type":48,"value":227},{"type":48,"value":110},{"type":42,"tag":68,"props":1174,"children":1175},{},[1176,1186,1187,1192],{"type":42,"tag":1119,"props":1177,"children":1178},{},[1179,1184],{"type":42,"tag":74,"props":1180,"children":1182},{"className":1181},[],[1183],{"type":48,"value":242},{"type":48,"value":1185}," enabled:",{"type":48,"value":1125},{"type":42,"tag":74,"props":1188,"children":1190},{"className":1189},[],[1191],{"type":48,"value":250},{"type":48,"value":110},{"type":42,"tag":58,"props":1194,"children":1195},{},[1196],{"type":48,"value":1197},"Set it explicitly in the sbatch script when your configuration calls for it.",{"type":42,"tag":51,"props":1199,"children":1201},{"id":1200},"containers",[1202],{"type":48,"value":1203},"Containers",{"type":42,"tag":58,"props":1205,"children":1206},{},[1207,1209,1214,1216,1222,1224,1230,1232,1237,1239,1245,1246,1252],{"type":48,"value":1208},"Many sites run Megatron-LM inside a container (enroot\u002Fpyxis on some clusters, singularity on others). If you do, the uv-managed ",{"type":42,"tag":74,"props":1210,"children":1212},{"className":1211},[],[1213],{"type":48,"value":303},{"type":48,"value":1215}," must live on a path that is visible from inside the container, and the container image must provide the CUDA \u002F NCCL \u002F torch versions the repo expects (see ",{"type":42,"tag":74,"props":1217,"children":1219},{"className":1218},[],[1220],{"type":48,"value":1221},"docker\u002F.ngc_version.dev",{"type":48,"value":1223}," and ",{"type":42,"tag":74,"props":1225,"children":1227},{"className":1226},[],[1228],{"type":48,"value":1229},".ngc_version.lts",{"type":48,"value":1231},"). The skeleton above stays the same; wrap the ",{"type":42,"tag":74,"props":1233,"children":1235},{"className":1234},[],[1236],{"type":48,"value":92},{"type":48,"value":1238}," invocation with your scheduler's container flags (",{"type":42,"tag":74,"props":1240,"children":1242},{"className":1241},[],[1243],{"type":48,"value":1244},"--container-image=…",{"type":48,"value":147},{"type":42,"tag":74,"props":1247,"children":1249},{"className":1248},[],[1250],{"type":48,"value":1251},"--container-mounts=…",{"type":48,"value":1253},", etc.).",{"type":42,"tag":51,"props":1255,"children":1257},{"id":1256},"monitor-and-collect",[1258],{"type":48,"value":1259},"Monitor and collect",{"type":42,"tag":326,"props":1261,"children":1263},{"className":328,"code":1262,"language":330,"meta":331,"style":331},"squeue -j \"$JOB_ID\" -o \"%.10i %.8T %.10M %.6D %R\"\nsacct -j \"$JOB_ID\" --format=JobID,State,ExitCode,Elapsed\nscancel \"$JOB_ID\"\n",[1264],{"type":42,"tag":74,"props":1265,"children":1266},{"__ignoreMap":331},[1267,1312,1341],{"type":42,"tag":337,"props":1268,"children":1269},{"class":339,"line":340},[1270,1275,1280,1284,1289,1293,1298,1302,1307],{"type":42,"tag":337,"props":1271,"children":1272},{"style":520},[1273],{"type":48,"value":1274},"squeue",{"type":42,"tag":337,"props":1276,"children":1277},{"style":451},[1278],{"type":48,"value":1279}," -j",{"type":42,"tag":337,"props":1281,"children":1282},{"style":470},[1283],{"type":48,"value":538},{"type":42,"tag":337,"props":1285,"children":1286},{"style":481},[1287],{"type":48,"value":1288},"$JOB_ID",{"type":42,"tag":337,"props":1290,"children":1291},{"style":470},[1292],{"type":48,"value":548},{"type":42,"tag":337,"props":1294,"children":1295},{"style":451},[1296],{"type":48,"value":1297}," -o",{"type":42,"tag":337,"props":1299,"children":1300},{"style":470},[1301],{"type":48,"value":538},{"type":42,"tag":337,"props":1303,"children":1304},{"style":451},[1305],{"type":48,"value":1306},"%.10i %.8T %.10M %.6D %R",{"type":42,"tag":337,"props":1308,"children":1309},{"style":470},[1310],{"type":48,"value":1311},"\"\n",{"type":42,"tag":337,"props":1313,"children":1314},{"class":339,"line":350},[1315,1320,1324,1328,1332,1336],{"type":42,"tag":337,"props":1316,"children":1317},{"style":520},[1318],{"type":48,"value":1319},"sacct",{"type":42,"tag":337,"props":1321,"children":1322},{"style":451},[1323],{"type":48,"value":1279},{"type":42,"tag":337,"props":1325,"children":1326},{"style":470},[1327],{"type":48,"value":538},{"type":42,"tag":337,"props":1329,"children":1330},{"style":481},[1331],{"type":48,"value":1288},{"type":42,"tag":337,"props":1333,"children":1334},{"style":470},[1335],{"type":48,"value":548},{"type":42,"tag":337,"props":1337,"children":1338},{"style":451},[1339],{"type":48,"value":1340}," --format=JobID,State,ExitCode,Elapsed\n",{"type":42,"tag":337,"props":1342,"children":1343},{"class":339,"line":359},[1344,1349,1353,1357],{"type":42,"tag":337,"props":1345,"children":1346},{"style":520},[1347],{"type":48,"value":1348},"scancel",{"type":42,"tag":337,"props":1350,"children":1351},{"style":470},[1352],{"type":48,"value":538},{"type":42,"tag":337,"props":1354,"children":1355},{"style":481},[1356],{"type":48,"value":1288},{"type":42,"tag":337,"props":1358,"children":1359},{"style":470},[1360],{"type":48,"value":1311},{"type":42,"tag":58,"props":1362,"children":1363},{},[1364,1366,1371],{"type":48,"value":1365},"If your training script writes a result artifact (a JSON metrics file from rank 0, a final checkpoint, etc.), poll for the artifact rather than waiting only on ",{"type":42,"tag":74,"props":1367,"children":1369},{"className":1368},[],[1370],{"type":48,"value":1274},{"type":48,"value":1372}," state. Useful output usually appears before SLURM marks the job complete, and polling on the artifact lets you cancel the job as soon as it lands instead of holding the allocation until the timeout.",{"type":42,"tag":51,"props":1374,"children":1376},{"id":1375},"failure-diagnosis",[1377],{"type":48,"value":1378},"Failure diagnosis",{"type":42,"tag":58,"props":1380,"children":1381},{},[1382],{"type":48,"value":1383},"Scan stderr from every rank, not just rank 0. The earliest non-NCCL Python traceback is usually the root cause; later NCCL timeouts on other ranks are downstream symptoms of the first crash.",{"type":42,"tag":58,"props":1385,"children":1386},{},[1387],{"type":48,"value":1388},"Classify quickly:",{"type":42,"tag":64,"props":1390,"children":1391},{},[1392,1402,1428,1462],{"type":42,"tag":68,"props":1393,"children":1394},{},[1395,1400],{"type":42,"tag":1119,"props":1396,"children":1397},{},[1398],{"type":48,"value":1399},"OOM",{"type":48,"value":1401},": record rank, phase (forward \u002F backward \u002F optimizer), batch size, sequence length, parallelism (TP\u002FDP\u002FCP\u002FPP), and peak memory before adjusting.",{"type":42,"tag":68,"props":1403,"children":1404},{},[1405,1410,1412,1418,1420,1426],{"type":42,"tag":1119,"props":1406,"children":1407},{},[1408],{"type":48,"value":1409},"Shape \u002F divisibility error",{"type":48,"value":1411},": check ",{"type":42,"tag":74,"props":1413,"children":1415},{"className":1414},[],[1416],{"type":48,"value":1417},"WORLD_SIZE = TP × DP × CP × PP",{"type":48,"value":1419}," and head-count divisibility (",{"type":42,"tag":74,"props":1421,"children":1423},{"className":1422},[],[1424],{"type":48,"value":1425},"num_attention_heads % TP == 0",{"type":48,"value":1427},").",{"type":42,"tag":68,"props":1429,"children":1430},{},[1431,1436,1438,1444,1446,1452,1454,1460],{"type":42,"tag":1119,"props":1432,"children":1433},{},[1434],{"type":48,"value":1435},"Import error",{"type":48,"value":1437},": wrong worktree, missing ",{"type":42,"tag":74,"props":1439,"children":1441},{"className":1440},[],[1442],{"type":48,"value":1443},"uv sync",{"type":48,"value":1445},", or stale ",{"type":42,"tag":74,"props":1447,"children":1449},{"className":1448},[],[1450],{"type":48,"value":1451},"PYTHONPATH",{"type":48,"value":1453},". Confirm ",{"type":42,"tag":74,"props":1455,"children":1457},{"className":1456},[],[1458],{"type":48,"value":1459},"cd \u003CMEGATRON_WORKTREE>",{"type":48,"value":1461}," before launch.",{"type":42,"tag":68,"props":1463,"children":1464},{},[1465,1470,1472,1477],{"type":42,"tag":1119,"props":1466,"children":1467},{},[1468],{"type":48,"value":1469},"NCCL failure",{"type":48,"value":1471}," with no Python traceback: verify allocation, port reachability, ",{"type":42,"tag":74,"props":1473,"children":1475},{"className":1474},[],[1476],{"type":48,"value":121},{"type":48,"value":1478}," resolution, and command consistency across ranks.",{"type":42,"tag":51,"props":1480,"children":1482},{"id":1481},"common-pitfalls",[1483],{"type":48,"value":1484},"Common pitfalls",{"type":42,"tag":64,"props":1486,"children":1487},{},[1488,1507,1512,1532],{"type":42,"tag":68,"props":1489,"children":1490},{},[1491,1493,1498,1500,1505],{"type":48,"value":1492},"Forgetting ",{"type":42,"tag":74,"props":1494,"children":1496},{"className":1495},[],[1497],{"type":48,"value":1443},{"type":48,"value":1499}," before the first submission. If the venv is missing, every job rebuilds it from inside ",{"type":42,"tag":74,"props":1501,"children":1503},{"className":1502},[],[1504],{"type":48,"value":92},{"type":48,"value":1506},", costing minutes per job.",{"type":42,"tag":68,"props":1508,"children":1509},{},[1510],{"type":48,"value":1511},"Writing logs to a node-local path that disappears at job exit. Always write to the shared filesystem.",{"type":42,"tag":68,"props":1513,"children":1514},{},[1515,1517,1523,1525,1530],{"type":48,"value":1516},"Setting ",{"type":42,"tag":74,"props":1518,"children":1520},{"className":1519},[],[1521],{"type":48,"value":1522},"CUDA_DEVICE_MAX_CONNECTIONS=1",{"type":48,"value":1524}," blindly. The right value depends on hardware and parallelism mode (see the dedicated section above). Setting it to ",{"type":42,"tag":74,"props":1526,"children":1528},{"className":1527},[],[1529],{"type":48,"value":227},{"type":48,"value":1531}," with FSDP causes a different problem; on Blackwell it has no effect; on pre-Blackwell with TP>1 or CP>1 (non-FSDP) the code asserts, it does not deadlock.",{"type":42,"tag":68,"props":1533,"children":1534},{},[1535,1537,1542,1544,1549,1551,1556],{"type":48,"value":1536},"Running bare ",{"type":42,"tag":74,"props":1538,"children":1540},{"className":1539},[],[1541],{"type":48,"value":108},{"type":48,"value":1543}," instead of ",{"type":42,"tag":74,"props":1545,"children":1547},{"className":1546},[],[1548],{"type":48,"value":100},{"type":48,"value":1550},". Bare ",{"type":42,"tag":74,"props":1552,"children":1554},{"className":1553},[],[1555],{"type":48,"value":108},{"type":48,"value":1557}," may dispatch through a python interpreter that does not see venv packages, depending on how the venv is set up.",{"type":42,"tag":1559,"props":1560,"children":1561},"style",{},[1562],{"type":48,"value":1563},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1565,"total":1712},[1566,1584,1596,1607,1619,1633,1646,1660,1673,1678,1692,1701],{"slug":1567,"name":1567,"fn":1568,"description":1569,"org":1570,"tags":1571,"stars":1581,"repoUrl":1582,"updatedAt":1583},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1572,1575,1578],{"name":1573,"slug":1574,"type":15},"Documentation","documentation",{"name":1576,"slug":1577,"type":15},"MCP","mcp",{"name":1579,"slug":1580,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":1585,"name":1585,"fn":1586,"description":1587,"org":1588,"tags":1589,"stars":19,"repoUrl":20,"updatedAt":1595},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1590,1591,1592],{"name":1203,"slug":1200,"type":15},{"name":13,"slug":14,"type":15},{"name":1593,"slug":1594,"type":15},"Python","python","2026-07-27T06:06:11.249662",{"slug":1597,"name":1597,"fn":1598,"description":1599,"org":1600,"tags":1601,"stars":19,"repoUrl":20,"updatedAt":1606},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1602,1605],{"name":1603,"slug":1604,"type":15},"CI\u002FCD","ci-cd",{"name":13,"slug":14,"type":15},"2026-07-14T05:25:59.97109",{"slug":1608,"name":1608,"fn":1609,"description":1610,"org":1611,"tags":1612,"stars":19,"repoUrl":20,"updatedAt":1618},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1613,1614,1615],{"name":1603,"slug":1604,"type":15},{"name":13,"slug":14,"type":15},{"name":1616,"slug":1617,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":1620,"name":1620,"fn":1621,"description":1622,"org":1623,"tags":1624,"stars":19,"repoUrl":20,"updatedAt":1632},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1625,1628,1629],{"name":1626,"slug":1627,"type":15},"Debugging","debugging",{"name":1616,"slug":1617,"type":15},{"name":1630,"slug":1631,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1634,"name":1634,"fn":1635,"description":1636,"org":1637,"tags":1638,"stars":19,"repoUrl":20,"updatedAt":1645},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1639,1642],{"name":1640,"slug":1641,"type":15},"Best Practices","best-practices",{"name":1643,"slug":1644,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1647,"name":1647,"fn":1648,"description":1649,"org":1650,"tags":1651,"stars":19,"repoUrl":20,"updatedAt":1659},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1652,1655,1658],{"name":1653,"slug":1654,"type":15},"Machine Learning","machine-learning",{"name":1656,"slug":1657,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":1661,"name":1661,"fn":1662,"description":1663,"org":1664,"tags":1665,"stars":19,"repoUrl":20,"updatedAt":1672},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1666,1669],{"name":1667,"slug":1668,"type":15},"QA","qa",{"name":1670,"slug":1671,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":4,"name":4,"fn":5,"description":6,"org":1674,"tags":1675,"stars":19,"repoUrl":20,"updatedAt":21},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1676,1677],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"slug":1679,"name":1679,"fn":1680,"description":1681,"org":1682,"tags":1683,"stars":19,"repoUrl":20,"updatedAt":1691},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1684,1687,1688],{"name":1685,"slug":1686,"type":15},"Code Review","code-review",{"name":1616,"slug":1617,"type":15},{"name":1689,"slug":1690,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":1693,"name":1693,"fn":1694,"description":1695,"org":1696,"tags":1697,"stars":19,"repoUrl":20,"updatedAt":1700},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1698,1699],{"name":1667,"slug":1668,"type":15},{"name":1670,"slug":1671,"type":15},"2026-07-14T05:25:54.928983",{"slug":1702,"name":1702,"fn":1703,"description":1704,"org":1705,"tags":1706,"stars":19,"repoUrl":20,"updatedAt":1711},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1707,1710],{"name":1708,"slug":1709,"type":15},"Automation","automation",{"name":1603,"slug":1604,"type":15},"2026-07-30T05:29:03.275638",496,{"items":1714,"total":462},[1715,1721,1726,1732,1738,1743,1749],{"slug":1585,"name":1585,"fn":1586,"description":1587,"org":1716,"tags":1717,"stars":19,"repoUrl":20,"updatedAt":1595},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1718,1719,1720],{"name":1203,"slug":1200,"type":15},{"name":13,"slug":14,"type":15},{"name":1593,"slug":1594,"type":15},{"slug":1597,"name":1597,"fn":1598,"description":1599,"org":1722,"tags":1723,"stars":19,"repoUrl":20,"updatedAt":1606},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1724,1725],{"name":1603,"slug":1604,"type":15},{"name":13,"slug":14,"type":15},{"slug":1608,"name":1608,"fn":1609,"description":1610,"org":1727,"tags":1728,"stars":19,"repoUrl":20,"updatedAt":1618},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1729,1730,1731],{"name":1603,"slug":1604,"type":15},{"name":13,"slug":14,"type":15},{"name":1616,"slug":1617,"type":15},{"slug":1620,"name":1620,"fn":1621,"description":1622,"org":1733,"tags":1734,"stars":19,"repoUrl":20,"updatedAt":1632},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1735,1736,1737],{"name":1626,"slug":1627,"type":15},{"name":1616,"slug":1617,"type":15},{"name":1630,"slug":1631,"type":15},{"slug":1634,"name":1634,"fn":1635,"description":1636,"org":1739,"tags":1740,"stars":19,"repoUrl":20,"updatedAt":1645},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1741,1742],{"name":1640,"slug":1641,"type":15},{"name":1643,"slug":1644,"type":15},{"slug":1647,"name":1647,"fn":1648,"description":1649,"org":1744,"tags":1745,"stars":19,"repoUrl":20,"updatedAt":1659},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1746,1747,1748],{"name":1653,"slug":1654,"type":15},{"name":1656,"slug":1657,"type":15},{"name":9,"slug":8,"type":15},{"slug":1661,"name":1661,"fn":1662,"description":1663,"org":1750,"tags":1751,"stars":19,"repoUrl":20,"updatedAt":1672},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1752,1753],{"name":1667,"slug":1668,"type":15},{"name":1670,"slug":1671,"type":15}]