[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nemo-mbridge-multi-node-slurm":3,"mdc-yv6atg-key":34,"related-repo-nvidia-nemo-mbridge-multi-node-slurm":3997,"related-org-nvidia-nemo-mbridge-multi-node-slurm":4099},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"nemo-mbridge-multi-node-slurm","configure and debug multi-node Slurm jobs","Convert single-node scripts to multi-node Slurm sbatch jobs and debug common multi-node failures. Covers srun-native vs uv run torch.distributed approaches, container setup, NCCL timeouts, OOM sizing for MoE models, and interactive allocation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":17,"slug":18,"type":15},"Deployment","deployment",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Engineering","engineering",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:28:56.724056","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fnemo-mbridge-multi-node-slurm","---\nname: nemo-mbridge-multi-node-slurm\ndescription: Convert single-node scripts to multi-node Slurm sbatch jobs and debug common multi-node failures. Covers srun-native vs uv run torch.distributed approaches, container setup, NCCL timeouts, OOM sizing for MoE models, and interactive allocation.\nlicense: Apache-2.0\nwhen_to_use: Writing or converting Slurm sbatch scripts, scaling to multiple nodes, debugging NCCL\u002Flaunch failures, or investigating a commit that caused multi-node training failures; 'run on multiple nodes', 'sbatch script', 'NCCL timeout', 'multi-node OOM'.\n---\n\n# Multi-Node Slurm\n\nConvert single-node `uv run python -m torch.distributed.run` commands into multi-node Slurm sbatch scripts with Enroot container support, and debug common multi-node failures.\n\n## First Answer Checklist\n\nWhen converting or debugging Bridge multi-node jobs, answer in this order:\n\n1. Prefer the **srun-native** launch shape for Bridge scripts that reach\n   `initialize.py`: `#SBATCH --ntasks-per-node=8` and a direct `srun ... uv run\n   python \u003Cscript> ...` launch. Do not wrap these jobs in\n   `python -m torch.distributed.run`.\n2. State that Bridge derives `RANK`, `WORLD_SIZE`, `LOCAL_RANK`,\n   `MASTER_ADDR`, and `MASTER_PORT` from SLURM variables during\n   `initialize.py` distributed init.\n3. Require shared paths and matching container mounts for the repo, data, logs,\n   `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME`.\n4. For NCCL timeout reports, do these first-log checks before speculating:\n   - grep for real errors while filtering warning\u002Fframe noise\n   - inspect `Failures:` to find the first failed rank and node\n   - grep for `ncclUniqueId`, `timeout`, or `crash on rank 0`\n\n## Two Approaches: srun-native vs uv run torch.distributed\n\n| Approach | `ntasks-per-node` | Process spawning | Best for |\n|---|---|---|---|\n| **srun-native** (preferred) | 8 | Slurm spawns 8 tasks\u002Fnode | Conversion, inference, Bridge scripts |\n| **uv run torch.distributed** (legacy) | 1 | `uv run python -m torch.distributed.run` spawns 8 procs\u002Fnode | MLM pretrain_gpt.py |\n\n**Prefer srun-native** — simpler, avoids shell escaping issues with TRAIN_CMD. Megatron Bridge auto-derives `RANK`, `WORLD_SIZE`, `LOCAL_RANK`, `MASTER_ADDR`, `MASTER_PORT` from SLURM env vars (`SLURM_PROCID`, `SLURM_NTASKS`, `SLURM_LOCALID`, `SLURM_NODELIST`) via `common_utils.py` helpers called during `initialize.py` distributed init, so you never need to set them manually.\n\n## Cluster Environment\n\nUse a shared filesystem for the repository, data, logs, `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME`. `NEMO_HOME` must not use the container-local default (`\u002Froot\u002F.cache\u002Fnemo`) for multi-node SFT\u002FPEFT jobs, because packed-sequence data prepared on node 0 must be visible to the other nodes.\n\nKeep credentials out of sbatch templates and logs. Provide `HF_TOKEN`, `GH_TOKEN`, and `WANDB_API_KEY` through the scheduler environment or a restricted secrets file, and never hardcode token values in the script body. For copy-paste environment and sbatch templates, read `references\u002Ftemplates.md`.\n\n### Log Directory\n\n```text\n\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_\u003Csuffix>\n```\n\n## srun-native Approach (Preferred)\n\nSlurm spawns all processes directly. No `torch.distributed.run`, no TRAIN_CMD escaping.\n\n### SBATCH Headers\n\n```bash\n#SBATCH --job-name=\u003Cmodel>-\u003Ctask>\n#SBATCH --nodes=\u003CNNODES>\n#SBATCH --ntasks-per-node=8          # Slurm spawns 8 tasks per node\n#SBATCH --gpus-per-node=8\n#SBATCH --time=00:30:00\n#SBATCH --account=\u003CYOUR_ACCOUNT>\n#SBATCH --partition=batch\n#SBATCH --output=\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_%j.log\n#SBATCH --exclusive\n```\n\n### Build and Launch\n\nUse a two-phase `srun` pattern: first run a single-process `uv sync` to populate the shared cache, then launch the full multi-node job. The full copy-paste version lives in `references\u002Ftemplates.md`.\n\n### srun-native Key Points\n\n- Phase 1 runs `uv sync` once on a single node\u002Fprocess, building all wheels into the shared cache on Lustre\n- Phase 2's `uv sync` is a fast no-op (everything is cached) — safe to run on all ranks without sleep guards\n- `initialize.py` + `common_utils.py` auto-set `RANK`, `WORLD_SIZE`, `LOCAL_RANK`, `MASTER_ADDR`, `MASTER_PORT` from SLURM env vars\n- Env vars like `HF_TOKEN`, `HF_HOME`, `UV_CACHE_DIR` exported at sbatch level are inherited by srun tasks\n- Reference: `examples\u002Fmodels\u002Fglm\u002Fglm_45v\u002Fslurm_sft.sh`, `examples\u002Fmodels\u002Fminimax\u002Fminimax_m2\u002Fslurm_conversion.sh`\n\n---\n\n## uv run torch.distributed Approach (Legacy)\n\nUse when the script requires `torch.distributed.run` (e.g., MLM pretrain_gpt.py) or when Bridge's `initialize.py` is not in the call path.\n\n### 1. Add SBATCH Headers\n\n```bash\n#SBATCH --job-name=\u003Cmodel>-\u003Cframework>\n#SBATCH --nodes=\u003CNNODES>\n#SBATCH --ntasks-per-node=1          # ALWAYS 1 — torchrun handles per-node spawning\n#SBATCH --gpus-per-node=8\n#SBATCH --time=00:30:00\n#SBATCH --account=\u003CYOUR_ACCOUNT>\n#SBATCH --partition=batch\n#SBATCH --output=\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_%j.log\n#SBATCH --exclusive\n```\n\n**Critical**: `--ntasks-per-node=1`, NOT 8. `uv run python -m torch.distributed.run --nproc_per_node=8` spawns 8 processes per node. Using `ntasks-per-node=8` causes EADDRINUSE port collisions (8 tasks x 8 procs = 64 per node).\n\n### 2. Convert to Multi-Node\n\nReplace single-node:\n\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=8 \\\n  \u003Cscript> \u003Cargs>\n```\n\nWith multi-node (inside `TRAIN_CMD` string):\n\n```bash\nuv run python -m torch.distributed.run \\\n  --nproc_per_node=8 \\\n  --nnodes=\\${SLURM_JOB_NUM_NODES} \\\n  --node_rank=\\${SLURM_NODEID} \\\n  \u003Cscript> \u003Cargs>\n```\n\n`MASTER_ADDR` and `MASTER_PORT` are auto-derived from SLURM env vars by `initialize.py` \u002F `common_utils.py` — no need to set them.\n\n### 3. Wrap in TRAIN_CMD + two-phase srun\n\nUse the same two-phase pattern: first a single-process srun to warm the uv cache, then the full run.\n\nSet runtime variables inside the container, but do not inject token values into a long `bash -c` string. Export credentials through the scheduler or source a restricted secrets file before the job starts. Keep `HF_HOME`, `UV_CACHE_DIR`, and `NEMO_HOME` on shared storage.\n\n### 4. Launch (two-phase)\n\nUse the two-phase launch template in `references\u002Ftemplates.md`, keeping `#SBATCH --ntasks-per-node=1` for this legacy approach.\n\n### 5. (Optional) Add Loss Extraction Footer\n\n```bash\necho \"======================================\"\necho \"Done. Losses:\"\necho \"======================================\"\ngrep -E \"iteration\\s+\" \"$LOGDIR\u002F\u003Cprefix>_${SLURM_JOB_ID}.log\" | grep -iE \"lm loss|reduced_train_loss\" | head -25\n```\n\n---\n\n## Interactive GPU Allocation (`salloc` + `srun`)\n\nFor ad-hoc testing (inference, conversion debugging), always follow these 3 steps:\n\n### Step 1: Allocate the node\n\n```bash\nsalloc --account \u003CYOUR_ACCOUNT> -N 1 \\\n  -J \u003CYOUR_ACCOUNT>-debug \\\n  -p interactive --gpus-per-node=8 -t 240\n```\n\n### Step 2: Launch container shell\n\n```bash\nsrun --mpi=pmix --no-kill \\\n  --container-image $CONTAINER_IMAGE \\\n  --container-mounts $CONTAINER_MOUNTS \\\n  --account \u003CYOUR_ACCOUNT> -N 1 \\\n  -J \u003CYOUR_ACCOUNT>-debug \\\n  --no-container-mount-home --gpus-per-node=8 \\\n  -p interactive --pty bash\n```\n\n### Step 3: Set up environment inside container\n\n```bash\nexport GH_TOKEN=\u003CYOUR_GITHUB_TOKEN>\nwandb login \u003CYOUR_WANDB_KEY>\nexport HF_TOKEN=\u003CYOUR_HF_TOKEN>\nexport HF_HOME=\u003CSHARED_FS>\u002FHF_HOME\nexport UV_CACHE_DIR=\"\u003CSHARED_FS>\u002Fuv_cache\"\nexport NEMO_HOME=\"\u003CSHARED_FS>\u002Fcache\u002Fnemo\"\nuv sync\n```\n\nThen run commands with `uv run` (uses the synced virtualenv):\n\n```bash\nuv run python -m torch.distributed.run --nproc_per_node=8 \\\n  examples\u002Fconversion\u002Fhf_to_megatron_generate_text.py \\\n  --hf_model_path \u003Corg>\u002F\u003Cmodel> --prompt \"What is AI?\" --max_new_tokens 50 --ep 8\n```\n\n**Pitfalls with interactive allocation:**\n\n| Error | Cause | Fix |\n|---|---|---|\n| `Cannot find GPU specification` | Missing `--gpus-per-node` | Always include `--gpus-per-node=8` in both `salloc` and `srun` |\n| `invalid partition specified: pool0` | Wrong partition name | Use `interactive` for interactive, `batch` for sbatch. Check: `sinfo --summarize` |\n| `Invalid account or account\u002Fpartition combination` | Partition not available for account | Check combos: `sacctmgr -nP show assoc where user=$USER format=account,partition` |\n| `Unable to create step for job... Requested node configuration is not available` | `-w \u003Cnode>` conflicts with allocation | Remove `-w` flag — HF cache is on shared filesystem, accessible from any node |\n| `uv: command not found` inside container | Container doesn't have `uv` pre-installed | Use a container with `uv` pre-installed, or `pip install uv` |\n| `No space left on device` during `uv` or `pip` | Container's `\u002Froot\u002F.cache\u002F` is full | Redirect: `export UV_CACHE_DIR=\u003CSHARED_FS>\u002Fuv_cache` |\n| `ModuleNotFoundError: No module named 'megatron.core.activations'` | Container's pre-installed megatron-core conflicts with local `3rdparty\u002FMegatron-LM` | Install local: `pip install -e 3rdparty\u002FMegatron-LM --no-deps --no-build-isolation` |\n\n---\n\n## Debugging Multi-Node Failures\n\n### Quick Diagnosis\n\nCheck the log for these patterns (in order):\n\n```bash\n# 1. Find the actual error (filter noise)\ngrep -a 'Error\\|OOM\\|CUDA out of memory\\|FAILED\\|Killed' job.log \\\n  | grep -v 'UserWarning\\|AllocatorConfig\\|transformer_engine\\|frame\\|srun: error'\n\n# 2. Check which rank crashed first\ngrep -a 'Failures:' -A 20 job.log | head -25\n\n# 3. Check for NCCL timeout\ngrep -a 'ncclUniqueId\\|timeout\\|crash on rank 0' job.log | head -5\n```\n\n### Debugging Checklist\n\nWhen a multi-node job fails:\n\n1. **Check exit code**: 1 = Python error, 9 = OOM killed, 143 = SIGTERM (timeout or cascade)\n2. **Find first failure**: Which task\u002Fnode crashed first? Others get SIGTERM (143) as cascade\n3. **grep the actual error**: Filter out UserWarnings, NCCL frame dumps\n4. **Check rank 0 specifically**: Most save\u002Fexport errors happen on rank 0\n5. **Verify EP sizing**: For MoE models, ensure `num_experts \u002F EP` fits in GPU memory with headroom\n6. **Try interactive first**: Use `salloc -N 2 -p interactive` to iterate faster than sbatch queue\n\n### NCCL Timeout at `dist.barrier()` — \"crash on rank 0\"\n\n**Symptom**: All ranks on node 2+ show:\n```text\n[rank8] is setting up NCCL communicator and retrieving ncclUniqueId from [0]\n... wait timeout after 600000ms\nThis may indicate a possible application crash on rank 0\n```\n\n**Root causes** (check in order):\n\n| Cause | How to verify | Fix |\n|---|---|---|\n| `save_artifacts` hangs on rank 0 | Error is in `save_hf_weights` → `dist.barrier()` | Increase timeout: `init_process_group(\"nccl\", timeout=timedelta(minutes=60))` |\n| `ImportError` in custom model code | `grep ImportError job.log` | Catch `ImportError` in `save_artifacts` (see below) |\n| Rank 0 OOM during export | `grep 'OutOfMemory' job.log` | Increase EP or nodes |\n| Network issue between nodes | Error only on cross-node ranks | Check `sinfo`, try different nodes |\n\n**The `save_artifacts` problem**: When `trust_remote_code=True`, rank 0 runs `save_artifacts()` (downloads tokenizer, config, custom modeling code) while all other ranks skip directly to `dist.barrier()`. If `save_artifacts` is slow or crashes, other ranks timeout.\n\n**Fix for ImportError in save_artifacts** (`hf_pretrained\u002Fbase.py`):\n```python\n# Change:\nexcept OSError:\n    pass\n# To:\nexcept (OSError, ImportError):\n    pass\n```\n\n### OOM for MoE Models\n\n**Symptom**: `torch.OutOfMemoryError: CUDA out of memory` during model loading or forward pass.\n\n**Key insight**: TP does NOT reduce expert memory. Only EP splits experts across GPUs.\n\n**Sizing formula**:\n```text\nexperts_per_gpu = num_experts \u002F EP\nexpert_memory_gb ≈ experts_per_gpu * expert_params * 2 \u002F 1e9  (bf16)\ntotal_per_gpu ≈ expert_memory_gb + attention_memory_gb + kv_cache_gb\n```\n\n**MiniMax-M2 example** (256 experts, ~230GB fp8 → ~460GB bf16):\n\n| Config | Nodes | GPUs | Experts\u002FGPU | Result |\n|---|---|---|---|---|\n| TP=2, EP=4 | 1 | 8 | 64 | OOM (too many experts) |\n| TP=2, EP=8 | 2 | 16 | 32 | Works for roundtrip (weight-only), OOM for inference |\n| TP=1, EP=16 | 2 | 16 | 16 | Works for inference |\n| TP=2, EP=32 | 8 | 64 | 8 | Comfortable for training |\n\n**Rules of thumb**:\n- Roundtrip (weight-only): can use more experts per GPU (~60GB model params OK)\n- Inference (forward pass + KV cache): needs headroom (~40GB model params max)\n- Training (activations + optimizer): needs even more headroom (~30GB model params max)\n\n### `ModuleNotFoundError: No module named 'megatron.core.tensor_parallel'`\n\n**Cause**: Container's pre-installed megatron-core conflicts with local `3rdparty\u002FMegatron-LM`.\n\n**Fix**: Add `uv sync` before running:\n```bash\nCMD=\"if [ \\\"\\$SLURM_LOCALID\\\" -eq 0 ]; then uv sync; else sleep 10; fi && \"\nCMD=\"${CMD}uv run --no-sync python \u003Cscript> \u003Cargs>\"\n```\n\n### FP8 Weight Mismatch in Roundtrip\n\n**Symptom**: Roundtrip completes but shows ❌ for all expert weights and raises `ValueError: Weight mismatch detected`.\n\n**Cause**: Original HF weights are FP8, Megatron stores in BF16. Exported weights are BF16. Comparison against original FP8 exceeds `atol=1e-1`.\n\n**This is expected for FP8 models.** The conversion is correct; the comparison tolerance is insufficient for the FP8→BF16 precision gap.\n\n### `WORLD_SIZE` Not Set with srun\n\n**Symptom**: Script exits with \"must be launched with torchrun\".\n\n**Cause**: Scripts check `os.environ.get(\"WORLD_SIZE\")` which torchrun sets but srun doesn't.\n\n**Fix**: Also check `SLURM_NTASKS`:\n```python\nif os.environ.get(\"WORLD_SIZE\") is None and os.environ.get(\"SLURM_NTASKS\") is None:\n    sys.exit(1)\n```\n\nBridge's `common_utils.py` helpers (called by `initialize.py`) populate env vars from SLURM:\n```python\nif \"RANK\" not in os.environ:\n    os.environ[\"RANK\"] = str(get_rank_safe())          # uses SLURM_PROCID\nif \"WORLD_SIZE\" not in os.environ:\n    os.environ[\"WORLD_SIZE\"] = str(get_world_size_safe())  # uses SLURM_NTASKS\nif \"MASTER_ADDR\" not in os.environ:\n    os.environ[\"MASTER_ADDR\"] = get_master_addr_safe()     # parses SLURM_NODELIST\nif \"MASTER_PORT\" not in os.environ:\n    os.environ[\"MASTER_PORT\"] = str(get_master_port_safe()) # derives from SLURM_JOB_ID\n```\n\n---\n\n## Key Gotchas\n\n1. **Two-phase srun for `uv sync`**: Run a single-process srun first to warm the cache, then the full multi-node srun. The second `uv sync` is a fast no-op since everything is already cached on the shared filesystem.\n\n2. **`--no-container-mount-home`** is an `srun` flag, NOT an `#SBATCH` directive.\n\n3. **Escaping inside TRAIN_CMD**: Since `TRAIN_CMD` is a double-quoted string, escape inner `$` for Slurm variables that must expand at runtime (not sbatch time):\n   - `\\${SLURM_PROCID}`, `\\${SLURM_JOB_NUM_NODES}`, `\\${SLURM_NODEID}`\n   - Host-side variables like `$GH_TOKEN`, `$LOGDIR`, `$WORKDIR` expand at sbatch time — no escaping needed.\n\n4. **Bridge `rm -rf nemo_experiments`**: Add before training to avoid stale checkpoint auto-resume.\n\n5. **MLM needs PYTHONPATH**: For pretrain_gpt.py scripts, add inside TRAIN_CMD:\n   ```bash\n   PYTHONPATH=${WORKDIR}\u002F3rdparty\u002FMegatron-LM:\\${PYTHONPATH:-} \\\n   ```\n\n6. **Node count heuristic**: Total GPUs = `NNODES * 8`. Must satisfy: `TP * PP * EP * DP >= total_GPUs` where `DP = total_GPUs \u002F (TP * PP * EP)`.\n\n7. **`NEMO_HOME` on shared filesystem for multi-node SFT**: The default nemo cache (`\u002Froot\u002F.cache\u002Fnemo`)\n   is container-local. Multi-node SFT with packed sequences prepares `.npy` files on one node\n   that are invisible to others. Set `export NEMO_HOME=\u003CSHARED_FS>\u002Fcache\u002Fnemo` so packed data\n   is shared. Without this, ranks on other nodes fail with `TypeError: 'NoneType' object is not an iterator`.\n\n## Full Templates and Command Bodies\n\nFor copyable sbatch scaffolding and Bridge\u002FMLM-specific `TRAIN_CMD` bodies, read\n[references\u002Ftemplates.md](references\u002Ftemplates.md).\n",{"data":35,"body":37},{"name":4,"description":6,"license":26,"when_to_use":36},"Writing or converting Slurm sbatch scripts, scaling to multiple nodes, debugging NCCL\u002Flaunch failures, or investigating a commit that caused multi-node training failures; 'run on multiple nodes', 'sbatch script', 'NCCL timeout', 'multi-node OOM'.",{"type":38,"children":39},"root",[40,49,64,71,76,256,262,364,449,455,494,528,535,547,553,566,572,665,671,698,704,821,825,831,850,856,928,961,967,972,1064,1077,1210,1241,1247,1252,1284,1290,1310,1316,1493,1496,1516,1521,1527,1640,1646,1804,1810,2002,2015,2153,2161,2472,2475,2481,2487,2492,2696,2702,2707,2786,2800,2810,2819,2829,2987,3034,3052,3108,3114,3130,3140,3150,3159,3169,3310,3319,3337,3347,3362,3378,3462,3468,3484,3500,3510,3521,3530,3547,3562,3585,3604,3675,3678,3684,3967,3973,3991],{"type":41,"tag":42,"props":43,"children":45},"element","h1",{"id":44},"multi-node-slurm",[46],{"type":47,"value":48},"text","Multi-Node Slurm",{"type":41,"tag":50,"props":51,"children":52},"p",{},[53,55,62],{"type":47,"value":54},"Convert single-node ",{"type":41,"tag":56,"props":57,"children":59},"code",{"className":58},[],[60],{"type":47,"value":61},"uv run python -m torch.distributed.run",{"type":47,"value":63}," commands into multi-node Slurm sbatch scripts with Enroot container support, and debug common multi-node failures.",{"type":41,"tag":65,"props":66,"children":68},"h2",{"id":67},"first-answer-checklist",[69],{"type":47,"value":70},"First Answer Checklist",{"type":41,"tag":50,"props":72,"children":73},{},[74],{"type":47,"value":75},"When converting or debugging Bridge multi-node jobs, answer in this order:",{"type":41,"tag":77,"props":78,"children":79},"ol",{},[80,126,177,203],{"type":41,"tag":81,"props":82,"children":83},"li",{},[84,86,92,94,100,102,108,110,116,118,124],{"type":47,"value":85},"Prefer the ",{"type":41,"tag":87,"props":88,"children":89},"strong",{},[90],{"type":47,"value":91},"srun-native",{"type":47,"value":93}," launch shape for Bridge scripts that reach\n",{"type":41,"tag":56,"props":95,"children":97},{"className":96},[],[98],{"type":47,"value":99},"initialize.py",{"type":47,"value":101},": ",{"type":41,"tag":56,"props":103,"children":105},{"className":104},[],[106],{"type":47,"value":107},"#SBATCH --ntasks-per-node=8",{"type":47,"value":109}," and a direct ",{"type":41,"tag":56,"props":111,"children":113},{"className":112},[],[114],{"type":47,"value":115},"srun ... uv run python \u003Cscript> ...",{"type":47,"value":117}," launch. Do not wrap these jobs in\n",{"type":41,"tag":56,"props":119,"children":121},{"className":120},[],[122],{"type":47,"value":123},"python -m torch.distributed.run",{"type":47,"value":125},".",{"type":41,"tag":81,"props":127,"children":128},{},[129,131,137,139,145,146,152,154,160,162,168,170,175],{"type":47,"value":130},"State that Bridge derives ",{"type":41,"tag":56,"props":132,"children":134},{"className":133},[],[135],{"type":47,"value":136},"RANK",{"type":47,"value":138},", ",{"type":41,"tag":56,"props":140,"children":142},{"className":141},[],[143],{"type":47,"value":144},"WORLD_SIZE",{"type":47,"value":138},{"type":41,"tag":56,"props":147,"children":149},{"className":148},[],[150],{"type":47,"value":151},"LOCAL_RANK",{"type":47,"value":153},",\n",{"type":41,"tag":56,"props":155,"children":157},{"className":156},[],[158],{"type":47,"value":159},"MASTER_ADDR",{"type":47,"value":161},", and ",{"type":41,"tag":56,"props":163,"children":165},{"className":164},[],[166],{"type":47,"value":167},"MASTER_PORT",{"type":47,"value":169}," from SLURM variables during\n",{"type":41,"tag":56,"props":171,"children":173},{"className":172},[],[174],{"type":47,"value":99},{"type":47,"value":176}," distributed init.",{"type":41,"tag":81,"props":178,"children":179},{},[180,182,188,189,195,196,202],{"type":47,"value":181},"Require shared paths and matching container mounts for the repo, data, logs,\n",{"type":41,"tag":56,"props":183,"children":185},{"className":184},[],[186],{"type":47,"value":187},"HF_HOME",{"type":47,"value":138},{"type":41,"tag":56,"props":190,"children":192},{"className":191},[],[193],{"type":47,"value":194},"UV_CACHE_DIR",{"type":47,"value":161},{"type":41,"tag":56,"props":197,"children":199},{"className":198},[],[200],{"type":47,"value":201},"NEMO_HOME",{"type":47,"value":125},{"type":41,"tag":81,"props":204,"children":205},{},[206,208],{"type":47,"value":207},"For NCCL timeout reports, do these first-log checks before speculating:\n",{"type":41,"tag":209,"props":210,"children":211},"ul",{},[212,217,230],{"type":41,"tag":81,"props":213,"children":214},{},[215],{"type":47,"value":216},"grep for real errors while filtering warning\u002Fframe noise",{"type":41,"tag":81,"props":218,"children":219},{},[220,222,228],{"type":47,"value":221},"inspect ",{"type":41,"tag":56,"props":223,"children":225},{"className":224},[],[226],{"type":47,"value":227},"Failures:",{"type":47,"value":229}," to find the first failed rank and node",{"type":41,"tag":81,"props":231,"children":232},{},[233,235,241,242,248,250],{"type":47,"value":234},"grep for ",{"type":41,"tag":56,"props":236,"children":238},{"className":237},[],[239],{"type":47,"value":240},"ncclUniqueId",{"type":47,"value":138},{"type":41,"tag":56,"props":243,"children":245},{"className":244},[],[246],{"type":47,"value":247},"timeout",{"type":47,"value":249},", or ",{"type":41,"tag":56,"props":251,"children":253},{"className":252},[],[254],{"type":47,"value":255},"crash on rank 0",{"type":41,"tag":65,"props":257,"children":259},{"id":258},"two-approaches-srun-native-vs-uv-run-torchdistributed",[260],{"type":47,"value":261},"Two Approaches: srun-native vs uv run torch.distributed",{"type":41,"tag":263,"props":264,"children":265},"table",{},[266,299],{"type":41,"tag":267,"props":268,"children":269},"thead",{},[270],{"type":41,"tag":271,"props":272,"children":273},"tr",{},[274,280,289,294],{"type":41,"tag":275,"props":276,"children":277},"th",{},[278],{"type":47,"value":279},"Approach",{"type":41,"tag":275,"props":281,"children":282},{},[283],{"type":41,"tag":56,"props":284,"children":286},{"className":285},[],[287],{"type":47,"value":288},"ntasks-per-node",{"type":41,"tag":275,"props":290,"children":291},{},[292],{"type":47,"value":293},"Process spawning",{"type":41,"tag":275,"props":295,"children":296},{},[297],{"type":47,"value":298},"Best for",{"type":41,"tag":300,"props":301,"children":302},"tbody",{},[303,331],{"type":41,"tag":271,"props":304,"children":305},{},[306,316,321,326],{"type":41,"tag":307,"props":308,"children":309},"td",{},[310,314],{"type":41,"tag":87,"props":311,"children":312},{},[313],{"type":47,"value":91},{"type":47,"value":315}," (preferred)",{"type":41,"tag":307,"props":317,"children":318},{},[319],{"type":47,"value":320},"8",{"type":41,"tag":307,"props":322,"children":323},{},[324],{"type":47,"value":325},"Slurm spawns 8 tasks\u002Fnode",{"type":41,"tag":307,"props":327,"children":328},{},[329],{"type":47,"value":330},"Conversion, inference, Bridge scripts",{"type":41,"tag":271,"props":332,"children":333},{},[334,344,349,359],{"type":41,"tag":307,"props":335,"children":336},{},[337,342],{"type":41,"tag":87,"props":338,"children":339},{},[340],{"type":47,"value":341},"uv run torch.distributed",{"type":47,"value":343}," (legacy)",{"type":41,"tag":307,"props":345,"children":346},{},[347],{"type":47,"value":348},"1",{"type":41,"tag":307,"props":350,"children":351},{},[352,357],{"type":41,"tag":56,"props":353,"children":355},{"className":354},[],[356],{"type":47,"value":61},{"type":47,"value":358}," spawns 8 procs\u002Fnode",{"type":41,"tag":307,"props":360,"children":361},{},[362],{"type":47,"value":363},"MLM pretrain_gpt.py",{"type":41,"tag":50,"props":365,"children":366},{},[367,372,374,379,380,385,386,391,392,397,398,403,405,411,412,418,419,425,426,432,434,440,442,447],{"type":41,"tag":87,"props":368,"children":369},{},[370],{"type":47,"value":371},"Prefer srun-native",{"type":47,"value":373}," — simpler, avoids shell escaping issues with TRAIN_CMD. Megatron Bridge auto-derives ",{"type":41,"tag":56,"props":375,"children":377},{"className":376},[],[378],{"type":47,"value":136},{"type":47,"value":138},{"type":41,"tag":56,"props":381,"children":383},{"className":382},[],[384],{"type":47,"value":144},{"type":47,"value":138},{"type":41,"tag":56,"props":387,"children":389},{"className":388},[],[390],{"type":47,"value":151},{"type":47,"value":138},{"type":41,"tag":56,"props":393,"children":395},{"className":394},[],[396],{"type":47,"value":159},{"type":47,"value":138},{"type":41,"tag":56,"props":399,"children":401},{"className":400},[],[402],{"type":47,"value":167},{"type":47,"value":404}," from SLURM env vars (",{"type":41,"tag":56,"props":406,"children":408},{"className":407},[],[409],{"type":47,"value":410},"SLURM_PROCID",{"type":47,"value":138},{"type":41,"tag":56,"props":413,"children":415},{"className":414},[],[416],{"type":47,"value":417},"SLURM_NTASKS",{"type":47,"value":138},{"type":41,"tag":56,"props":420,"children":422},{"className":421},[],[423],{"type":47,"value":424},"SLURM_LOCALID",{"type":47,"value":138},{"type":41,"tag":56,"props":427,"children":429},{"className":428},[],[430],{"type":47,"value":431},"SLURM_NODELIST",{"type":47,"value":433},") via ",{"type":41,"tag":56,"props":435,"children":437},{"className":436},[],[438],{"type":47,"value":439},"common_utils.py",{"type":47,"value":441}," helpers called during ",{"type":41,"tag":56,"props":443,"children":445},{"className":444},[],[446],{"type":47,"value":99},{"type":47,"value":448}," distributed init, so you never need to set them manually.",{"type":41,"tag":65,"props":450,"children":452},{"id":451},"cluster-environment",[453],{"type":47,"value":454},"Cluster Environment",{"type":41,"tag":50,"props":456,"children":457},{},[458,460,465,466,471,472,477,479,484,486,492],{"type":47,"value":459},"Use a shared filesystem for the repository, data, logs, ",{"type":41,"tag":56,"props":461,"children":463},{"className":462},[],[464],{"type":47,"value":187},{"type":47,"value":138},{"type":41,"tag":56,"props":467,"children":469},{"className":468},[],[470],{"type":47,"value":194},{"type":47,"value":161},{"type":41,"tag":56,"props":473,"children":475},{"className":474},[],[476],{"type":47,"value":201},{"type":47,"value":478},". ",{"type":41,"tag":56,"props":480,"children":482},{"className":481},[],[483],{"type":47,"value":201},{"type":47,"value":485}," must not use the container-local default (",{"type":41,"tag":56,"props":487,"children":489},{"className":488},[],[490],{"type":47,"value":491},"\u002Froot\u002F.cache\u002Fnemo",{"type":47,"value":493},") for multi-node SFT\u002FPEFT jobs, because packed-sequence data prepared on node 0 must be visible to the other nodes.",{"type":41,"tag":50,"props":495,"children":496},{},[497,499,505,506,512,513,519,521,527],{"type":47,"value":498},"Keep credentials out of sbatch templates and logs. Provide ",{"type":41,"tag":56,"props":500,"children":502},{"className":501},[],[503],{"type":47,"value":504},"HF_TOKEN",{"type":47,"value":138},{"type":41,"tag":56,"props":507,"children":509},{"className":508},[],[510],{"type":47,"value":511},"GH_TOKEN",{"type":47,"value":161},{"type":41,"tag":56,"props":514,"children":516},{"className":515},[],[517],{"type":47,"value":518},"WANDB_API_KEY",{"type":47,"value":520}," through the scheduler environment or a restricted secrets file, and never hardcode token values in the script body. For copy-paste environment and sbatch templates, read ",{"type":41,"tag":56,"props":522,"children":524},{"className":523},[],[525],{"type":47,"value":526},"references\u002Ftemplates.md",{"type":47,"value":125},{"type":41,"tag":529,"props":530,"children":532},"h3",{"id":531},"log-directory",[533],{"type":47,"value":534},"Log Directory",{"type":41,"tag":536,"props":537,"children":542},"pre",{"className":538,"code":540,"language":47,"meta":541},[539],"language-text","\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_\u003Csuffix>\n","",[543],{"type":41,"tag":56,"props":544,"children":545},{"__ignoreMap":541},[546],{"type":47,"value":540},{"type":41,"tag":65,"props":548,"children":550},{"id":549},"srun-native-approach-preferred",[551],{"type":47,"value":552},"srun-native Approach (Preferred)",{"type":41,"tag":50,"props":554,"children":555},{},[556,558,564],{"type":47,"value":557},"Slurm spawns all processes directly. No ",{"type":41,"tag":56,"props":559,"children":561},{"className":560},[],[562],{"type":47,"value":563},"torch.distributed.run",{"type":47,"value":565},", no TRAIN_CMD escaping.",{"type":41,"tag":529,"props":567,"children":569},{"id":568},"sbatch-headers",[570],{"type":47,"value":571},"SBATCH Headers",{"type":41,"tag":536,"props":573,"children":577},{"className":574,"code":575,"language":576,"meta":541,"style":541},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","#SBATCH --job-name=\u003Cmodel>-\u003Ctask>\n#SBATCH --nodes=\u003CNNODES>\n#SBATCH --ntasks-per-node=8          # Slurm spawns 8 tasks per node\n#SBATCH --gpus-per-node=8\n#SBATCH --time=00:30:00\n#SBATCH --account=\u003CYOUR_ACCOUNT>\n#SBATCH --partition=batch\n#SBATCH --output=\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_%j.log\n#SBATCH --exclusive\n","bash",[578],{"type":41,"tag":56,"props":579,"children":580},{"__ignoreMap":541},[581,593,602,611,620,629,638,647,656],{"type":41,"tag":582,"props":583,"children":586},"span",{"class":584,"line":585},"line",1,[587],{"type":41,"tag":582,"props":588,"children":590},{"style":589},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[591],{"type":47,"value":592},"#SBATCH --job-name=\u003Cmodel>-\u003Ctask>\n",{"type":41,"tag":582,"props":594,"children":596},{"class":584,"line":595},2,[597],{"type":41,"tag":582,"props":598,"children":599},{"style":589},[600],{"type":47,"value":601},"#SBATCH --nodes=\u003CNNODES>\n",{"type":41,"tag":582,"props":603,"children":605},{"class":584,"line":604},3,[606],{"type":41,"tag":582,"props":607,"children":608},{"style":589},[609],{"type":47,"value":610},"#SBATCH --ntasks-per-node=8          # Slurm spawns 8 tasks per node\n",{"type":41,"tag":582,"props":612,"children":614},{"class":584,"line":613},4,[615],{"type":41,"tag":582,"props":616,"children":617},{"style":589},[618],{"type":47,"value":619},"#SBATCH --gpus-per-node=8\n",{"type":41,"tag":582,"props":621,"children":623},{"class":584,"line":622},5,[624],{"type":41,"tag":582,"props":625,"children":626},{"style":589},[627],{"type":47,"value":628},"#SBATCH --time=00:30:00\n",{"type":41,"tag":582,"props":630,"children":632},{"class":584,"line":631},6,[633],{"type":41,"tag":582,"props":634,"children":635},{"style":589},[636],{"type":47,"value":637},"#SBATCH --account=\u003CYOUR_ACCOUNT>\n",{"type":41,"tag":582,"props":639,"children":641},{"class":584,"line":640},7,[642],{"type":41,"tag":582,"props":643,"children":644},{"style":589},[645],{"type":47,"value":646},"#SBATCH --partition=batch\n",{"type":41,"tag":582,"props":648,"children":650},{"class":584,"line":649},8,[651],{"type":41,"tag":582,"props":652,"children":653},{"style":589},[654],{"type":47,"value":655},"#SBATCH --output=\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_%j.log\n",{"type":41,"tag":582,"props":657,"children":659},{"class":584,"line":658},9,[660],{"type":41,"tag":582,"props":661,"children":662},{"style":589},[663],{"type":47,"value":664},"#SBATCH --exclusive\n",{"type":41,"tag":529,"props":666,"children":668},{"id":667},"build-and-launch",[669],{"type":47,"value":670},"Build and Launch",{"type":41,"tag":50,"props":672,"children":673},{},[674,676,682,684,690,692,697],{"type":47,"value":675},"Use a two-phase ",{"type":41,"tag":56,"props":677,"children":679},{"className":678},[],[680],{"type":47,"value":681},"srun",{"type":47,"value":683}," pattern: first run a single-process ",{"type":41,"tag":56,"props":685,"children":687},{"className":686},[],[688],{"type":47,"value":689},"uv sync",{"type":47,"value":691}," to populate the shared cache, then launch the full multi-node job. The full copy-paste version lives in ",{"type":41,"tag":56,"props":693,"children":695},{"className":694},[],[696],{"type":47,"value":526},{"type":47,"value":125},{"type":41,"tag":529,"props":699,"children":701},{"id":700},"srun-native-key-points",[702],{"type":47,"value":703},"srun-native Key Points",{"type":41,"tag":209,"props":705,"children":706},{},[707,719,731,779,803],{"type":41,"tag":81,"props":708,"children":709},{},[710,712,717],{"type":47,"value":711},"Phase 1 runs ",{"type":41,"tag":56,"props":713,"children":715},{"className":714},[],[716],{"type":47,"value":689},{"type":47,"value":718}," once on a single node\u002Fprocess, building all wheels into the shared cache on Lustre",{"type":41,"tag":81,"props":720,"children":721},{},[722,724,729],{"type":47,"value":723},"Phase 2's ",{"type":41,"tag":56,"props":725,"children":727},{"className":726},[],[728],{"type":47,"value":689},{"type":47,"value":730}," is a fast no-op (everything is cached) — safe to run on all ranks without sleep guards",{"type":41,"tag":81,"props":732,"children":733},{},[734,739,741,746,748,753,754,759,760,765,766,771,772,777],{"type":41,"tag":56,"props":735,"children":737},{"className":736},[],[738],{"type":47,"value":99},{"type":47,"value":740}," + ",{"type":41,"tag":56,"props":742,"children":744},{"className":743},[],[745],{"type":47,"value":439},{"type":47,"value":747}," auto-set ",{"type":41,"tag":56,"props":749,"children":751},{"className":750},[],[752],{"type":47,"value":136},{"type":47,"value":138},{"type":41,"tag":56,"props":755,"children":757},{"className":756},[],[758],{"type":47,"value":144},{"type":47,"value":138},{"type":41,"tag":56,"props":761,"children":763},{"className":762},[],[764],{"type":47,"value":151},{"type":47,"value":138},{"type":41,"tag":56,"props":767,"children":769},{"className":768},[],[770],{"type":47,"value":159},{"type":47,"value":138},{"type":41,"tag":56,"props":773,"children":775},{"className":774},[],[776],{"type":47,"value":167},{"type":47,"value":778}," from SLURM env vars",{"type":41,"tag":81,"props":780,"children":781},{},[782,784,789,790,795,796,801],{"type":47,"value":783},"Env vars like ",{"type":41,"tag":56,"props":785,"children":787},{"className":786},[],[788],{"type":47,"value":504},{"type":47,"value":138},{"type":41,"tag":56,"props":791,"children":793},{"className":792},[],[794],{"type":47,"value":187},{"type":47,"value":138},{"type":41,"tag":56,"props":797,"children":799},{"className":798},[],[800],{"type":47,"value":194},{"type":47,"value":802}," exported at sbatch level are inherited by srun tasks",{"type":41,"tag":81,"props":804,"children":805},{},[806,808,814,815],{"type":47,"value":807},"Reference: ",{"type":41,"tag":56,"props":809,"children":811},{"className":810},[],[812],{"type":47,"value":813},"examples\u002Fmodels\u002Fglm\u002Fglm_45v\u002Fslurm_sft.sh",{"type":47,"value":138},{"type":41,"tag":56,"props":816,"children":818},{"className":817},[],[819],{"type":47,"value":820},"examples\u002Fmodels\u002Fminimax\u002Fminimax_m2\u002Fslurm_conversion.sh",{"type":41,"tag":822,"props":823,"children":824},"hr",{},[],{"type":41,"tag":65,"props":826,"children":828},{"id":827},"uv-run-torchdistributed-approach-legacy",[829],{"type":47,"value":830},"uv run torch.distributed Approach (Legacy)",{"type":41,"tag":50,"props":832,"children":833},{},[834,836,841,843,848],{"type":47,"value":835},"Use when the script requires ",{"type":41,"tag":56,"props":837,"children":839},{"className":838},[],[840],{"type":47,"value":563},{"type":47,"value":842}," (e.g., MLM pretrain_gpt.py) or when Bridge's ",{"type":41,"tag":56,"props":844,"children":846},{"className":845},[],[847],{"type":47,"value":99},{"type":47,"value":849}," is not in the call path.",{"type":41,"tag":529,"props":851,"children":853},{"id":852},"_1-add-sbatch-headers",[854],{"type":47,"value":855},"1. Add SBATCH Headers",{"type":41,"tag":536,"props":857,"children":859},{"className":574,"code":858,"language":576,"meta":541,"style":541},"#SBATCH --job-name=\u003Cmodel>-\u003Cframework>\n#SBATCH --nodes=\u003CNNODES>\n#SBATCH --ntasks-per-node=1          # ALWAYS 1 — torchrun handles per-node spawning\n#SBATCH --gpus-per-node=8\n#SBATCH --time=00:30:00\n#SBATCH --account=\u003CYOUR_ACCOUNT>\n#SBATCH --partition=batch\n#SBATCH --output=\u003CSHARED_FS>\u002Flogs\u002F\u003Cjob_name>_%j.log\n#SBATCH --exclusive\n",[860],{"type":41,"tag":56,"props":861,"children":862},{"__ignoreMap":541},[863,871,878,886,893,900,907,914,921],{"type":41,"tag":582,"props":864,"children":865},{"class":584,"line":585},[866],{"type":41,"tag":582,"props":867,"children":868},{"style":589},[869],{"type":47,"value":870},"#SBATCH --job-name=\u003Cmodel>-\u003Cframework>\n",{"type":41,"tag":582,"props":872,"children":873},{"class":584,"line":595},[874],{"type":41,"tag":582,"props":875,"children":876},{"style":589},[877],{"type":47,"value":601},{"type":41,"tag":582,"props":879,"children":880},{"class":584,"line":604},[881],{"type":41,"tag":582,"props":882,"children":883},{"style":589},[884],{"type":47,"value":885},"#SBATCH --ntasks-per-node=1          # ALWAYS 1 — torchrun handles per-node spawning\n",{"type":41,"tag":582,"props":887,"children":888},{"class":584,"line":613},[889],{"type":41,"tag":582,"props":890,"children":891},{"style":589},[892],{"type":47,"value":619},{"type":41,"tag":582,"props":894,"children":895},{"class":584,"line":622},[896],{"type":41,"tag":582,"props":897,"children":898},{"style":589},[899],{"type":47,"value":628},{"type":41,"tag":582,"props":901,"children":902},{"class":584,"line":631},[903],{"type":41,"tag":582,"props":904,"children":905},{"style":589},[906],{"type":47,"value":637},{"type":41,"tag":582,"props":908,"children":909},{"class":584,"line":640},[910],{"type":41,"tag":582,"props":911,"children":912},{"style":589},[913],{"type":47,"value":646},{"type":41,"tag":582,"props":915,"children":916},{"class":584,"line":649},[917],{"type":41,"tag":582,"props":918,"children":919},{"style":589},[920],{"type":47,"value":655},{"type":41,"tag":582,"props":922,"children":923},{"class":584,"line":658},[924],{"type":41,"tag":582,"props":925,"children":926},{"style":589},[927],{"type":47,"value":664},{"type":41,"tag":50,"props":929,"children":930},{},[931,936,937,943,945,951,953,959],{"type":41,"tag":87,"props":932,"children":933},{},[934],{"type":47,"value":935},"Critical",{"type":47,"value":101},{"type":41,"tag":56,"props":938,"children":940},{"className":939},[],[941],{"type":47,"value":942},"--ntasks-per-node=1",{"type":47,"value":944},", NOT 8. ",{"type":41,"tag":56,"props":946,"children":948},{"className":947},[],[949],{"type":47,"value":950},"uv run python -m torch.distributed.run --nproc_per_node=8",{"type":47,"value":952}," spawns 8 processes per node. Using ",{"type":41,"tag":56,"props":954,"children":956},{"className":955},[],[957],{"type":47,"value":958},"ntasks-per-node=8",{"type":47,"value":960}," causes EADDRINUSE port collisions (8 tasks x 8 procs = 64 per node).",{"type":41,"tag":529,"props":962,"children":964},{"id":963},"_2-convert-to-multi-node",[965],{"type":47,"value":966},"2. Convert to Multi-Node",{"type":41,"tag":50,"props":968,"children":969},{},[970],{"type":47,"value":971},"Replace single-node:",{"type":41,"tag":536,"props":973,"children":975},{"className":574,"code":974,"language":576,"meta":541,"style":541},"uv run python -m torch.distributed.run --nproc_per_node=8 \\\n  \u003Cscript> \u003Cargs>\n",[976],{"type":41,"tag":56,"props":977,"children":978},{"__ignoreMap":541},[979,1020],{"type":41,"tag":582,"props":980,"children":981},{"class":584,"line":585},[982,988,994,999,1004,1009,1014],{"type":41,"tag":582,"props":983,"children":985},{"style":984},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[986],{"type":47,"value":987},"uv",{"type":41,"tag":582,"props":989,"children":991},{"style":990},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[992],{"type":47,"value":993}," run",{"type":41,"tag":582,"props":995,"children":996},{"style":990},[997],{"type":47,"value":998}," python",{"type":41,"tag":582,"props":1000,"children":1001},{"style":990},[1002],{"type":47,"value":1003}," -m",{"type":41,"tag":582,"props":1005,"children":1006},{"style":990},[1007],{"type":47,"value":1008}," torch.distributed.run",{"type":41,"tag":582,"props":1010,"children":1011},{"style":990},[1012],{"type":47,"value":1013}," --nproc_per_node=8",{"type":41,"tag":582,"props":1015,"children":1017},{"style":1016},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1018],{"type":47,"value":1019}," \\\n",{"type":41,"tag":582,"props":1021,"children":1022},{"class":584,"line":595},[1023,1029,1034,1039,1044,1049,1054,1059],{"type":41,"tag":582,"props":1024,"children":1026},{"style":1025},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1027],{"type":47,"value":1028},"  \u003C",{"type":41,"tag":582,"props":1030,"children":1031},{"style":990},[1032],{"type":47,"value":1033},"scrip",{"type":41,"tag":582,"props":1035,"children":1036},{"style":1016},[1037],{"type":47,"value":1038},"t",{"type":41,"tag":582,"props":1040,"children":1041},{"style":1025},[1042],{"type":47,"value":1043},">",{"type":41,"tag":582,"props":1045,"children":1046},{"style":1025},[1047],{"type":47,"value":1048}," \u003C",{"type":41,"tag":582,"props":1050,"children":1051},{"style":990},[1052],{"type":47,"value":1053},"arg",{"type":41,"tag":582,"props":1055,"children":1056},{"style":1016},[1057],{"type":47,"value":1058},"s",{"type":41,"tag":582,"props":1060,"children":1061},{"style":1025},[1062],{"type":47,"value":1063},">\n",{"type":41,"tag":50,"props":1065,"children":1066},{},[1067,1069,1075],{"type":47,"value":1068},"With multi-node (inside ",{"type":41,"tag":56,"props":1070,"children":1072},{"className":1071},[],[1073],{"type":47,"value":1074},"TRAIN_CMD",{"type":47,"value":1076}," string):",{"type":41,"tag":536,"props":1078,"children":1080},{"className":574,"code":1079,"language":576,"meta":541,"style":541},"uv run python -m torch.distributed.run \\\n  --nproc_per_node=8 \\\n  --nnodes=\\${SLURM_JOB_NUM_NODES} \\\n  --node_rank=\\${SLURM_NODEID} \\\n  \u003Cscript> \u003Cargs>\n",[1081],{"type":41,"tag":56,"props":1082,"children":1083},{"__ignoreMap":541},[1084,1111,1123,1150,1175],{"type":41,"tag":582,"props":1085,"children":1086},{"class":584,"line":585},[1087,1091,1095,1099,1103,1107],{"type":41,"tag":582,"props":1088,"children":1089},{"style":984},[1090],{"type":47,"value":987},{"type":41,"tag":582,"props":1092,"children":1093},{"style":990},[1094],{"type":47,"value":993},{"type":41,"tag":582,"props":1096,"children":1097},{"style":990},[1098],{"type":47,"value":998},{"type":41,"tag":582,"props":1100,"children":1101},{"style":990},[1102],{"type":47,"value":1003},{"type":41,"tag":582,"props":1104,"children":1105},{"style":990},[1106],{"type":47,"value":1008},{"type":41,"tag":582,"props":1108,"children":1109},{"style":1016},[1110],{"type":47,"value":1019},{"type":41,"tag":582,"props":1112,"children":1113},{"class":584,"line":595},[1114,1119],{"type":41,"tag":582,"props":1115,"children":1116},{"style":990},[1117],{"type":47,"value":1118},"  --nproc_per_node=8",{"type":41,"tag":582,"props":1120,"children":1121},{"style":1016},[1122],{"type":47,"value":1019},{"type":41,"tag":582,"props":1124,"children":1125},{"class":584,"line":604},[1126,1131,1136,1141,1146],{"type":41,"tag":582,"props":1127,"children":1128},{"style":990},[1129],{"type":47,"value":1130},"  --nnodes=",{"type":41,"tag":582,"props":1132,"children":1133},{"style":1016},[1134],{"type":47,"value":1135},"\\$",{"type":41,"tag":582,"props":1137,"children":1138},{"style":1016},[1139],{"type":47,"value":1140},"{",{"type":41,"tag":582,"props":1142,"children":1143},{"style":984},[1144],{"type":47,"value":1145},"SLURM_JOB_NUM_NODES}",{"type":41,"tag":582,"props":1147,"children":1148},{"style":1016},[1149],{"type":47,"value":1019},{"type":41,"tag":582,"props":1151,"children":1152},{"class":584,"line":613},[1153,1158,1162,1166,1171],{"type":41,"tag":582,"props":1154,"children":1155},{"style":990},[1156],{"type":47,"value":1157},"  --node_rank=",{"type":41,"tag":582,"props":1159,"children":1160},{"style":1016},[1161],{"type":47,"value":1135},{"type":41,"tag":582,"props":1163,"children":1164},{"style":1016},[1165],{"type":47,"value":1140},{"type":41,"tag":582,"props":1167,"children":1168},{"style":984},[1169],{"type":47,"value":1170},"SLURM_NODEID}",{"type":41,"tag":582,"props":1172,"children":1173},{"style":1016},[1174],{"type":47,"value":1019},{"type":41,"tag":582,"props":1176,"children":1177},{"class":584,"line":622},[1178,1182,1186,1190,1194,1198,1202,1206],{"type":41,"tag":582,"props":1179,"children":1180},{"style":1025},[1181],{"type":47,"value":1028},{"type":41,"tag":582,"props":1183,"children":1184},{"style":990},[1185],{"type":47,"value":1033},{"type":41,"tag":582,"props":1187,"children":1188},{"style":1016},[1189],{"type":47,"value":1038},{"type":41,"tag":582,"props":1191,"children":1192},{"style":1025},[1193],{"type":47,"value":1043},{"type":41,"tag":582,"props":1195,"children":1196},{"style":1025},[1197],{"type":47,"value":1048},{"type":41,"tag":582,"props":1199,"children":1200},{"style":990},[1201],{"type":47,"value":1053},{"type":41,"tag":582,"props":1203,"children":1204},{"style":1016},[1205],{"type":47,"value":1058},{"type":41,"tag":582,"props":1207,"children":1208},{"style":1025},[1209],{"type":47,"value":1063},{"type":41,"tag":50,"props":1211,"children":1212},{},[1213,1218,1220,1225,1227,1232,1234,1239],{"type":41,"tag":56,"props":1214,"children":1216},{"className":1215},[],[1217],{"type":47,"value":159},{"type":47,"value":1219}," and ",{"type":41,"tag":56,"props":1221,"children":1223},{"className":1222},[],[1224],{"type":47,"value":167},{"type":47,"value":1226}," are auto-derived from SLURM env vars by ",{"type":41,"tag":56,"props":1228,"children":1230},{"className":1229},[],[1231],{"type":47,"value":99},{"type":47,"value":1233}," \u002F ",{"type":41,"tag":56,"props":1235,"children":1237},{"className":1236},[],[1238],{"type":47,"value":439},{"type":47,"value":1240}," — no need to set them.",{"type":41,"tag":529,"props":1242,"children":1244},{"id":1243},"_3-wrap-in-train_cmd-two-phase-srun",[1245],{"type":47,"value":1246},"3. Wrap in TRAIN_CMD + two-phase srun",{"type":41,"tag":50,"props":1248,"children":1249},{},[1250],{"type":47,"value":1251},"Use the same two-phase pattern: first a single-process srun to warm the uv cache, then the full run.",{"type":41,"tag":50,"props":1253,"children":1254},{},[1255,1257,1263,1265,1270,1271,1276,1277,1282],{"type":47,"value":1256},"Set runtime variables inside the container, but do not inject token values into a long ",{"type":41,"tag":56,"props":1258,"children":1260},{"className":1259},[],[1261],{"type":47,"value":1262},"bash -c",{"type":47,"value":1264}," string. Export credentials through the scheduler or source a restricted secrets file before the job starts. Keep ",{"type":41,"tag":56,"props":1266,"children":1268},{"className":1267},[],[1269],{"type":47,"value":187},{"type":47,"value":138},{"type":41,"tag":56,"props":1272,"children":1274},{"className":1273},[],[1275],{"type":47,"value":194},{"type":47,"value":161},{"type":41,"tag":56,"props":1278,"children":1280},{"className":1279},[],[1281],{"type":47,"value":201},{"type":47,"value":1283}," on shared storage.",{"type":41,"tag":529,"props":1285,"children":1287},{"id":1286},"_4-launch-two-phase",[1288],{"type":47,"value":1289},"4. Launch (two-phase)",{"type":41,"tag":50,"props":1291,"children":1292},{},[1293,1295,1300,1302,1308],{"type":47,"value":1294},"Use the two-phase launch template in ",{"type":41,"tag":56,"props":1296,"children":1298},{"className":1297},[],[1299],{"type":47,"value":526},{"type":47,"value":1301},", keeping ",{"type":41,"tag":56,"props":1303,"children":1305},{"className":1304},[],[1306],{"type":47,"value":1307},"#SBATCH --ntasks-per-node=1",{"type":47,"value":1309}," for this legacy approach.",{"type":41,"tag":529,"props":1311,"children":1313},{"id":1312},"_5-optional-add-loss-extraction-footer",[1314],{"type":47,"value":1315},"5. (Optional) Add Loss Extraction Footer",{"type":41,"tag":536,"props":1317,"children":1319},{"className":574,"code":1318,"language":576,"meta":541,"style":541},"echo \"======================================\"\necho \"Done. Losses:\"\necho \"======================================\"\ngrep -E \"iteration\\s+\" \"$LOGDIR\u002F\u003Cprefix>_${SLURM_JOB_ID}.log\" | grep -iE \"lm loss|reduced_train_loss\" | head -25\n",[1320],{"type":41,"tag":56,"props":1321,"children":1322},{"__ignoreMap":541},[1323,1347,1367,1386],{"type":41,"tag":582,"props":1324,"children":1325},{"class":584,"line":585},[1326,1332,1337,1342],{"type":41,"tag":582,"props":1327,"children":1329},{"style":1328},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[1330],{"type":47,"value":1331},"echo",{"type":41,"tag":582,"props":1333,"children":1334},{"style":1025},[1335],{"type":47,"value":1336}," \"",{"type":41,"tag":582,"props":1338,"children":1339},{"style":990},[1340],{"type":47,"value":1341},"======================================",{"type":41,"tag":582,"props":1343,"children":1344},{"style":1025},[1345],{"type":47,"value":1346},"\"\n",{"type":41,"tag":582,"props":1348,"children":1349},{"class":584,"line":595},[1350,1354,1358,1363],{"type":41,"tag":582,"props":1351,"children":1352},{"style":1328},[1353],{"type":47,"value":1331},{"type":41,"tag":582,"props":1355,"children":1356},{"style":1025},[1357],{"type":47,"value":1336},{"type":41,"tag":582,"props":1359,"children":1360},{"style":990},[1361],{"type":47,"value":1362},"Done. Losses:",{"type":41,"tag":582,"props":1364,"children":1365},{"style":1025},[1366],{"type":47,"value":1346},{"type":41,"tag":582,"props":1368,"children":1369},{"class":584,"line":604},[1370,1374,1378,1382],{"type":41,"tag":582,"props":1371,"children":1372},{"style":1328},[1373],{"type":47,"value":1331},{"type":41,"tag":582,"props":1375,"children":1376},{"style":1025},[1377],{"type":47,"value":1336},{"type":41,"tag":582,"props":1379,"children":1380},{"style":990},[1381],{"type":47,"value":1341},{"type":41,"tag":582,"props":1383,"children":1384},{"style":1025},[1385],{"type":47,"value":1346},{"type":41,"tag":582,"props":1387,"children":1388},{"class":584,"line":613},[1389,1394,1399,1403,1408,1413,1417,1422,1427,1432,1437,1442,1447,1451,1456,1461,1466,1470,1475,1479,1483,1488],{"type":41,"tag":582,"props":1390,"children":1391},{"style":984},[1392],{"type":47,"value":1393},"grep",{"type":41,"tag":582,"props":1395,"children":1396},{"style":990},[1397],{"type":47,"value":1398}," -E",{"type":41,"tag":582,"props":1400,"children":1401},{"style":1025},[1402],{"type":47,"value":1336},{"type":41,"tag":582,"props":1404,"children":1405},{"style":990},[1406],{"type":47,"value":1407},"iteration\\s+",{"type":41,"tag":582,"props":1409,"children":1410},{"style":1025},[1411],{"type":47,"value":1412},"\"",{"type":41,"tag":582,"props":1414,"children":1415},{"style":1025},[1416],{"type":47,"value":1336},{"type":41,"tag":582,"props":1418,"children":1419},{"style":1016},[1420],{"type":47,"value":1421},"$LOGDIR",{"type":41,"tag":582,"props":1423,"children":1424},{"style":990},[1425],{"type":47,"value":1426},"\u002F\u003Cprefix>_",{"type":41,"tag":582,"props":1428,"children":1429},{"style":1025},[1430],{"type":47,"value":1431},"${",{"type":41,"tag":582,"props":1433,"children":1434},{"style":1016},[1435],{"type":47,"value":1436},"SLURM_JOB_ID",{"type":41,"tag":582,"props":1438,"children":1439},{"style":1025},[1440],{"type":47,"value":1441},"}",{"type":41,"tag":582,"props":1443,"children":1444},{"style":990},[1445],{"type":47,"value":1446},".log",{"type":41,"tag":582,"props":1448,"children":1449},{"style":1025},[1450],{"type":47,"value":1412},{"type":41,"tag":582,"props":1452,"children":1453},{"style":1025},[1454],{"type":47,"value":1455}," |",{"type":41,"tag":582,"props":1457,"children":1458},{"style":984},[1459],{"type":47,"value":1460}," grep",{"type":41,"tag":582,"props":1462,"children":1463},{"style":990},[1464],{"type":47,"value":1465}," -iE",{"type":41,"tag":582,"props":1467,"children":1468},{"style":1025},[1469],{"type":47,"value":1336},{"type":41,"tag":582,"props":1471,"children":1472},{"style":990},[1473],{"type":47,"value":1474},"lm loss|reduced_train_loss",{"type":41,"tag":582,"props":1476,"children":1477},{"style":1025},[1478],{"type":47,"value":1412},{"type":41,"tag":582,"props":1480,"children":1481},{"style":1025},[1482],{"type":47,"value":1455},{"type":41,"tag":582,"props":1484,"children":1485},{"style":984},[1486],{"type":47,"value":1487}," head",{"type":41,"tag":582,"props":1489,"children":1490},{"style":990},[1491],{"type":47,"value":1492}," -25\n",{"type":41,"tag":822,"props":1494,"children":1495},{},[],{"type":41,"tag":65,"props":1497,"children":1499},{"id":1498},"interactive-gpu-allocation-salloc-srun",[1500,1502,1508,1509,1514],{"type":47,"value":1501},"Interactive GPU Allocation (",{"type":41,"tag":56,"props":1503,"children":1505},{"className":1504},[],[1506],{"type":47,"value":1507},"salloc",{"type":47,"value":740},{"type":41,"tag":56,"props":1510,"children":1512},{"className":1511},[],[1513],{"type":47,"value":681},{"type":47,"value":1515},")",{"type":41,"tag":50,"props":1517,"children":1518},{},[1519],{"type":47,"value":1520},"For ad-hoc testing (inference, conversion debugging), always follow these 3 steps:",{"type":41,"tag":529,"props":1522,"children":1524},{"id":1523},"step-1-allocate-the-node",[1525],{"type":47,"value":1526},"Step 1: Allocate the node",{"type":41,"tag":536,"props":1528,"children":1530},{"className":574,"code":1529,"language":576,"meta":541,"style":541},"salloc --account \u003CYOUR_ACCOUNT> -N 1 \\\n  -J \u003CYOUR_ACCOUNT>-debug \\\n  -p interactive --gpus-per-node=8 -t 240\n",[1531],{"type":41,"tag":56,"props":1532,"children":1533},{"__ignoreMap":541},[1534,1579,1612],{"type":41,"tag":582,"props":1535,"children":1536},{"class":584,"line":585},[1537,1541,1546,1550,1555,1560,1564,1569,1575],{"type":41,"tag":582,"props":1538,"children":1539},{"style":984},[1540],{"type":47,"value":1507},{"type":41,"tag":582,"props":1542,"children":1543},{"style":990},[1544],{"type":47,"value":1545}," --account",{"type":41,"tag":582,"props":1547,"children":1548},{"style":1025},[1549],{"type":47,"value":1048},{"type":41,"tag":582,"props":1551,"children":1552},{"style":990},[1553],{"type":47,"value":1554},"YOUR_ACCOUN",{"type":41,"tag":582,"props":1556,"children":1557},{"style":1016},[1558],{"type":47,"value":1559},"T",{"type":41,"tag":582,"props":1561,"children":1562},{"style":1025},[1563],{"type":47,"value":1043},{"type":41,"tag":582,"props":1565,"children":1566},{"style":990},[1567],{"type":47,"value":1568}," -N",{"type":41,"tag":582,"props":1570,"children":1572},{"style":1571},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1573],{"type":47,"value":1574}," 1",{"type":41,"tag":582,"props":1576,"children":1577},{"style":1016},[1578],{"type":47,"value":1019},{"type":41,"tag":582,"props":1580,"children":1581},{"class":584,"line":595},[1582,1587,1591,1595,1599,1603,1608],{"type":41,"tag":582,"props":1583,"children":1584},{"style":990},[1585],{"type":47,"value":1586},"  -J",{"type":41,"tag":582,"props":1588,"children":1589},{"style":1025},[1590],{"type":47,"value":1048},{"type":41,"tag":582,"props":1592,"children":1593},{"style":990},[1594],{"type":47,"value":1554},{"type":41,"tag":582,"props":1596,"children":1597},{"style":1016},[1598],{"type":47,"value":1559},{"type":41,"tag":582,"props":1600,"children":1601},{"style":1025},[1602],{"type":47,"value":1043},{"type":41,"tag":582,"props":1604,"children":1605},{"style":990},[1606],{"type":47,"value":1607},"-debug",{"type":41,"tag":582,"props":1609,"children":1610},{"style":1016},[1611],{"type":47,"value":1019},{"type":41,"tag":582,"props":1613,"children":1614},{"class":584,"line":604},[1615,1620,1625,1630,1635],{"type":41,"tag":582,"props":1616,"children":1617},{"style":990},[1618],{"type":47,"value":1619},"  -p",{"type":41,"tag":582,"props":1621,"children":1622},{"style":990},[1623],{"type":47,"value":1624}," interactive",{"type":41,"tag":582,"props":1626,"children":1627},{"style":990},[1628],{"type":47,"value":1629}," --gpus-per-node=8",{"type":41,"tag":582,"props":1631,"children":1632},{"style":990},[1633],{"type":47,"value":1634}," -t",{"type":41,"tag":582,"props":1636,"children":1637},{"style":1571},[1638],{"type":47,"value":1639}," 240\n",{"type":41,"tag":529,"props":1641,"children":1643},{"id":1642},"step-2-launch-container-shell",[1644],{"type":47,"value":1645},"Step 2: Launch container shell",{"type":41,"tag":536,"props":1647,"children":1649},{"className":574,"code":1648,"language":576,"meta":541,"style":541},"srun --mpi=pmix --no-kill \\\n  --container-image $CONTAINER_IMAGE \\\n  --container-mounts $CONTAINER_MOUNTS \\\n  --account \u003CYOUR_ACCOUNT> -N 1 \\\n  -J \u003CYOUR_ACCOUNT>-debug \\\n  --no-container-mount-home --gpus-per-node=8 \\\n  -p interactive --pty bash\n",[1650],{"type":41,"tag":56,"props":1651,"children":1652},{"__ignoreMap":541},[1653,1674,1687,1700,1736,1767,1783],{"type":41,"tag":582,"props":1654,"children":1655},{"class":584,"line":585},[1656,1660,1665,1670],{"type":41,"tag":582,"props":1657,"children":1658},{"style":984},[1659],{"type":47,"value":681},{"type":41,"tag":582,"props":1661,"children":1662},{"style":990},[1663],{"type":47,"value":1664}," --mpi=pmix",{"type":41,"tag":582,"props":1666,"children":1667},{"style":990},[1668],{"type":47,"value":1669}," --no-kill",{"type":41,"tag":582,"props":1671,"children":1672},{"style":1016},[1673],{"type":47,"value":1019},{"type":41,"tag":582,"props":1675,"children":1676},{"class":584,"line":595},[1677,1682],{"type":41,"tag":582,"props":1678,"children":1679},{"style":990},[1680],{"type":47,"value":1681},"  --container-image",{"type":41,"tag":582,"props":1683,"children":1684},{"style":1016},[1685],{"type":47,"value":1686}," $CONTAINER_IMAGE \\\n",{"type":41,"tag":582,"props":1688,"children":1689},{"class":584,"line":604},[1690,1695],{"type":41,"tag":582,"props":1691,"children":1692},{"style":990},[1693],{"type":47,"value":1694},"  --container-mounts",{"type":41,"tag":582,"props":1696,"children":1697},{"style":1016},[1698],{"type":47,"value":1699}," $CONTAINER_MOUNTS \\\n",{"type":41,"tag":582,"props":1701,"children":1702},{"class":584,"line":613},[1703,1708,1712,1716,1720,1724,1728,1732],{"type":41,"tag":582,"props":1704,"children":1705},{"style":990},[1706],{"type":47,"value":1707},"  --account",{"type":41,"tag":582,"props":1709,"children":1710},{"style":1025},[1711],{"type":47,"value":1048},{"type":41,"tag":582,"props":1713,"children":1714},{"style":990},[1715],{"type":47,"value":1554},{"type":41,"tag":582,"props":1717,"children":1718},{"style":1016},[1719],{"type":47,"value":1559},{"type":41,"tag":582,"props":1721,"children":1722},{"style":1025},[1723],{"type":47,"value":1043},{"type":41,"tag":582,"props":1725,"children":1726},{"style":990},[1727],{"type":47,"value":1568},{"type":41,"tag":582,"props":1729,"children":1730},{"style":1571},[1731],{"type":47,"value":1574},{"type":41,"tag":582,"props":1733,"children":1734},{"style":1016},[1735],{"type":47,"value":1019},{"type":41,"tag":582,"props":1737,"children":1738},{"class":584,"line":622},[1739,1743,1747,1751,1755,1759,1763],{"type":41,"tag":582,"props":1740,"children":1741},{"style":990},[1742],{"type":47,"value":1586},{"type":41,"tag":582,"props":1744,"children":1745},{"style":1025},[1746],{"type":47,"value":1048},{"type":41,"tag":582,"props":1748,"children":1749},{"style":990},[1750],{"type":47,"value":1554},{"type":41,"tag":582,"props":1752,"children":1753},{"style":1016},[1754],{"type":47,"value":1559},{"type":41,"tag":582,"props":1756,"children":1757},{"style":1025},[1758],{"type":47,"value":1043},{"type":41,"tag":582,"props":1760,"children":1761},{"style":990},[1762],{"type":47,"value":1607},{"type":41,"tag":582,"props":1764,"children":1765},{"style":1016},[1766],{"type":47,"value":1019},{"type":41,"tag":582,"props":1768,"children":1769},{"class":584,"line":631},[1770,1775,1779],{"type":41,"tag":582,"props":1771,"children":1772},{"style":990},[1773],{"type":47,"value":1774},"  --no-container-mount-home",{"type":41,"tag":582,"props":1776,"children":1777},{"style":990},[1778],{"type":47,"value":1629},{"type":41,"tag":582,"props":1780,"children":1781},{"style":1016},[1782],{"type":47,"value":1019},{"type":41,"tag":582,"props":1784,"children":1785},{"class":584,"line":640},[1786,1790,1794,1799],{"type":41,"tag":582,"props":1787,"children":1788},{"style":990},[1789],{"type":47,"value":1619},{"type":41,"tag":582,"props":1791,"children":1792},{"style":990},[1793],{"type":47,"value":1624},{"type":41,"tag":582,"props":1795,"children":1796},{"style":990},[1797],{"type":47,"value":1798}," --pty",{"type":41,"tag":582,"props":1800,"children":1801},{"style":990},[1802],{"type":47,"value":1803}," bash\n",{"type":41,"tag":529,"props":1805,"children":1807},{"id":1806},"step-3-set-up-environment-inside-container",[1808],{"type":47,"value":1809},"Step 3: Set up environment inside container",{"type":41,"tag":536,"props":1811,"children":1813},{"className":574,"code":1812,"language":576,"meta":541,"style":541},"export GH_TOKEN=\u003CYOUR_GITHUB_TOKEN>\nwandb login \u003CYOUR_WANDB_KEY>\nexport HF_TOKEN=\u003CYOUR_HF_TOKEN>\nexport HF_HOME=\u003CSHARED_FS>\u002FHF_HOME\nexport UV_CACHE_DIR=\"\u003CSHARED_FS>\u002Fuv_cache\"\nexport NEMO_HOME=\"\u003CSHARED_FS>\u002Fcache\u002Fnemo\"\nuv sync\n",[1814],{"type":41,"tag":56,"props":1815,"children":1816},{"__ignoreMap":541},[1817,1845,1876,1901,1931,1961,1990],{"type":41,"tag":582,"props":1818,"children":1819},{"class":584,"line":585},[1820,1826,1831,1836,1841],{"type":41,"tag":582,"props":1821,"children":1823},{"style":1822},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1824],{"type":47,"value":1825},"export",{"type":41,"tag":582,"props":1827,"children":1828},{"style":1016},[1829],{"type":47,"value":1830}," GH_TOKEN",{"type":41,"tag":582,"props":1832,"children":1833},{"style":1025},[1834],{"type":47,"value":1835},"=\u003C",{"type":41,"tag":582,"props":1837,"children":1838},{"style":1016},[1839],{"type":47,"value":1840},"YOUR_GITHUB_TOKEN",{"type":41,"tag":582,"props":1842,"children":1843},{"style":1025},[1844],{"type":47,"value":1063},{"type":41,"tag":582,"props":1846,"children":1847},{"class":584,"line":595},[1848,1853,1858,1862,1867,1872],{"type":41,"tag":582,"props":1849,"children":1850},{"style":984},[1851],{"type":47,"value":1852},"wandb",{"type":41,"tag":582,"props":1854,"children":1855},{"style":990},[1856],{"type":47,"value":1857}," login",{"type":41,"tag":582,"props":1859,"children":1860},{"style":1025},[1861],{"type":47,"value":1048},{"type":41,"tag":582,"props":1863,"children":1864},{"style":990},[1865],{"type":47,"value":1866},"YOUR_WANDB_KE",{"type":41,"tag":582,"props":1868,"children":1869},{"style":1016},[1870],{"type":47,"value":1871},"Y",{"type":41,"tag":582,"props":1873,"children":1874},{"style":1025},[1875],{"type":47,"value":1063},{"type":41,"tag":582,"props":1877,"children":1878},{"class":584,"line":604},[1879,1883,1888,1892,1897],{"type":41,"tag":582,"props":1880,"children":1881},{"style":1822},[1882],{"type":47,"value":1825},{"type":41,"tag":582,"props":1884,"children":1885},{"style":1016},[1886],{"type":47,"value":1887}," HF_TOKEN",{"type":41,"tag":582,"props":1889,"children":1890},{"style":1025},[1891],{"type":47,"value":1835},{"type":41,"tag":582,"props":1893,"children":1894},{"style":1016},[1895],{"type":47,"value":1896},"YOUR_HF_TOKEN",{"type":41,"tag":582,"props":1898,"children":1899},{"style":1025},[1900],{"type":47,"value":1063},{"type":41,"tag":582,"props":1902,"children":1903},{"class":584,"line":613},[1904,1908,1913,1917,1922,1926],{"type":41,"tag":582,"props":1905,"children":1906},{"style":1822},[1907],{"type":47,"value":1825},{"type":41,"tag":582,"props":1909,"children":1910},{"style":1016},[1911],{"type":47,"value":1912}," HF_HOME",{"type":41,"tag":582,"props":1914,"children":1915},{"style":1025},[1916],{"type":47,"value":1835},{"type":41,"tag":582,"props":1918,"children":1919},{"style":1016},[1920],{"type":47,"value":1921},"SHARED_FS",{"type":41,"tag":582,"props":1923,"children":1924},{"style":1025},[1925],{"type":47,"value":1043},{"type":41,"tag":582,"props":1927,"children":1928},{"style":1016},[1929],{"type":47,"value":1930},"\u002FHF_HOME\n",{"type":41,"tag":582,"props":1932,"children":1933},{"class":584,"line":622},[1934,1938,1943,1948,1952,1957],{"type":41,"tag":582,"props":1935,"children":1936},{"style":1822},[1937],{"type":47,"value":1825},{"type":41,"tag":582,"props":1939,"children":1940},{"style":1016},[1941],{"type":47,"value":1942}," UV_CACHE_DIR",{"type":41,"tag":582,"props":1944,"children":1945},{"style":1025},[1946],{"type":47,"value":1947},"=",{"type":41,"tag":582,"props":1949,"children":1950},{"style":1025},[1951],{"type":47,"value":1412},{"type":41,"tag":582,"props":1953,"children":1954},{"style":990},[1955],{"type":47,"value":1956},"\u003CSHARED_FS>\u002Fuv_cache",{"type":41,"tag":582,"props":1958,"children":1959},{"style":1025},[1960],{"type":47,"value":1346},{"type":41,"tag":582,"props":1962,"children":1963},{"class":584,"line":631},[1964,1968,1973,1977,1981,1986],{"type":41,"tag":582,"props":1965,"children":1966},{"style":1822},[1967],{"type":47,"value":1825},{"type":41,"tag":582,"props":1969,"children":1970},{"style":1016},[1971],{"type":47,"value":1972}," NEMO_HOME",{"type":41,"tag":582,"props":1974,"children":1975},{"style":1025},[1976],{"type":47,"value":1947},{"type":41,"tag":582,"props":1978,"children":1979},{"style":1025},[1980],{"type":47,"value":1412},{"type":41,"tag":582,"props":1982,"children":1983},{"style":990},[1984],{"type":47,"value":1985},"\u003CSHARED_FS>\u002Fcache\u002Fnemo",{"type":41,"tag":582,"props":1987,"children":1988},{"style":1025},[1989],{"type":47,"value":1346},{"type":41,"tag":582,"props":1991,"children":1992},{"class":584,"line":640},[1993,1997],{"type":41,"tag":582,"props":1994,"children":1995},{"style":984},[1996],{"type":47,"value":987},{"type":41,"tag":582,"props":1998,"children":1999},{"style":990},[2000],{"type":47,"value":2001}," sync\n",{"type":41,"tag":50,"props":2003,"children":2004},{},[2005,2007,2013],{"type":47,"value":2006},"Then run commands with ",{"type":41,"tag":56,"props":2008,"children":2010},{"className":2009},[],[2011],{"type":47,"value":2012},"uv run",{"type":47,"value":2014}," (uses the synced virtualenv):",{"type":41,"tag":536,"props":2016,"children":2018},{"className":574,"code":2017,"language":576,"meta":541,"style":541},"uv run python -m torch.distributed.run --nproc_per_node=8 \\\n  examples\u002Fconversion\u002Fhf_to_megatron_generate_text.py \\\n  --hf_model_path \u003Corg>\u002F\u003Cmodel> --prompt \"What is AI?\" --max_new_tokens 50 --ep 8\n",[2019],{"type":41,"tag":56,"props":2020,"children":2021},{"__ignoreMap":541},[2022,2053,2065],{"type":41,"tag":582,"props":2023,"children":2024},{"class":584,"line":585},[2025,2029,2033,2037,2041,2045,2049],{"type":41,"tag":582,"props":2026,"children":2027},{"style":984},[2028],{"type":47,"value":987},{"type":41,"tag":582,"props":2030,"children":2031},{"style":990},[2032],{"type":47,"value":993},{"type":41,"tag":582,"props":2034,"children":2035},{"style":990},[2036],{"type":47,"value":998},{"type":41,"tag":582,"props":2038,"children":2039},{"style":990},[2040],{"type":47,"value":1003},{"type":41,"tag":582,"props":2042,"children":2043},{"style":990},[2044],{"type":47,"value":1008},{"type":41,"tag":582,"props":2046,"children":2047},{"style":990},[2048],{"type":47,"value":1013},{"type":41,"tag":582,"props":2050,"children":2051},{"style":1016},[2052],{"type":47,"value":1019},{"type":41,"tag":582,"props":2054,"children":2055},{"class":584,"line":595},[2056,2061],{"type":41,"tag":582,"props":2057,"children":2058},{"style":990},[2059],{"type":47,"value":2060},"  examples\u002Fconversion\u002Fhf_to_megatron_generate_text.py",{"type":41,"tag":582,"props":2062,"children":2063},{"style":1016},[2064],{"type":47,"value":1019},{"type":41,"tag":582,"props":2066,"children":2067},{"class":584,"line":604},[2068,2073,2077,2082,2087,2091,2096,2101,2106,2111,2115,2120,2124,2129,2133,2138,2143,2148],{"type":41,"tag":582,"props":2069,"children":2070},{"style":990},[2071],{"type":47,"value":2072},"  --hf_model_path",{"type":41,"tag":582,"props":2074,"children":2075},{"style":1025},[2076],{"type":47,"value":1048},{"type":41,"tag":582,"props":2078,"children":2079},{"style":990},[2080],{"type":47,"value":2081},"or",{"type":41,"tag":582,"props":2083,"children":2084},{"style":1016},[2085],{"type":47,"value":2086},"g",{"type":41,"tag":582,"props":2088,"children":2089},{"style":1025},[2090],{"type":47,"value":1043},{"type":41,"tag":582,"props":2092,"children":2093},{"style":990},[2094],{"type":47,"value":2095},"\u002F",{"type":41,"tag":582,"props":2097,"children":2098},{"style":1025},[2099],{"type":47,"value":2100},"\u003C",{"type":41,"tag":582,"props":2102,"children":2103},{"style":990},[2104],{"type":47,"value":2105},"mode",{"type":41,"tag":582,"props":2107,"children":2108},{"style":1016},[2109],{"type":47,"value":2110},"l",{"type":41,"tag":582,"props":2112,"children":2113},{"style":1025},[2114],{"type":47,"value":1043},{"type":41,"tag":582,"props":2116,"children":2117},{"style":990},[2118],{"type":47,"value":2119}," --prompt",{"type":41,"tag":582,"props":2121,"children":2122},{"style":1025},[2123],{"type":47,"value":1336},{"type":41,"tag":582,"props":2125,"children":2126},{"style":990},[2127],{"type":47,"value":2128},"What is AI?",{"type":41,"tag":582,"props":2130,"children":2131},{"style":1025},[2132],{"type":47,"value":1412},{"type":41,"tag":582,"props":2134,"children":2135},{"style":990},[2136],{"type":47,"value":2137}," --max_new_tokens",{"type":41,"tag":582,"props":2139,"children":2140},{"style":1571},[2141],{"type":47,"value":2142}," 50",{"type":41,"tag":582,"props":2144,"children":2145},{"style":990},[2146],{"type":47,"value":2147}," --ep",{"type":41,"tag":582,"props":2149,"children":2150},{"style":1571},[2151],{"type":47,"value":2152}," 8\n",{"type":41,"tag":50,"props":2154,"children":2155},{},[2156],{"type":41,"tag":87,"props":2157,"children":2158},{},[2159],{"type":47,"value":2160},"Pitfalls with interactive allocation:",{"type":41,"tag":263,"props":2162,"children":2163},{},[2164,2185],{"type":41,"tag":267,"props":2165,"children":2166},{},[2167],{"type":41,"tag":271,"props":2168,"children":2169},{},[2170,2175,2180],{"type":41,"tag":275,"props":2171,"children":2172},{},[2173],{"type":47,"value":2174},"Error",{"type":41,"tag":275,"props":2176,"children":2177},{},[2178],{"type":47,"value":2179},"Cause",{"type":41,"tag":275,"props":2181,"children":2182},{},[2183],{"type":47,"value":2184},"Fix",{"type":41,"tag":300,"props":2186,"children":2187},{},[2188,2235,2279,2307,2343,2387,2438],{"type":41,"tag":271,"props":2189,"children":2190},{},[2191,2200,2211],{"type":41,"tag":307,"props":2192,"children":2193},{},[2194],{"type":41,"tag":56,"props":2195,"children":2197},{"className":2196},[],[2198],{"type":47,"value":2199},"Cannot find GPU specification",{"type":41,"tag":307,"props":2201,"children":2202},{},[2203,2205],{"type":47,"value":2204},"Missing ",{"type":41,"tag":56,"props":2206,"children":2208},{"className":2207},[],[2209],{"type":47,"value":2210},"--gpus-per-node",{"type":41,"tag":307,"props":2212,"children":2213},{},[2214,2216,2222,2224,2229,2230],{"type":47,"value":2215},"Always include ",{"type":41,"tag":56,"props":2217,"children":2219},{"className":2218},[],[2220],{"type":47,"value":2221},"--gpus-per-node=8",{"type":47,"value":2223}," in both ",{"type":41,"tag":56,"props":2225,"children":2227},{"className":2226},[],[2228],{"type":47,"value":1507},{"type":47,"value":1219},{"type":41,"tag":56,"props":2231,"children":2233},{"className":2232},[],[2234],{"type":47,"value":681},{"type":41,"tag":271,"props":2236,"children":2237},{},[2238,2247,2252],{"type":41,"tag":307,"props":2239,"children":2240},{},[2241],{"type":41,"tag":56,"props":2242,"children":2244},{"className":2243},[],[2245],{"type":47,"value":2246},"invalid partition specified: pool0",{"type":41,"tag":307,"props":2248,"children":2249},{},[2250],{"type":47,"value":2251},"Wrong partition name",{"type":41,"tag":307,"props":2253,"children":2254},{},[2255,2257,2263,2265,2271,2273],{"type":47,"value":2256},"Use ",{"type":41,"tag":56,"props":2258,"children":2260},{"className":2259},[],[2261],{"type":47,"value":2262},"interactive",{"type":47,"value":2264}," for interactive, ",{"type":41,"tag":56,"props":2266,"children":2268},{"className":2267},[],[2269],{"type":47,"value":2270},"batch",{"type":47,"value":2272}," for sbatch. Check: ",{"type":41,"tag":56,"props":2274,"children":2276},{"className":2275},[],[2277],{"type":47,"value":2278},"sinfo --summarize",{"type":41,"tag":271,"props":2280,"children":2281},{},[2282,2291,2296],{"type":41,"tag":307,"props":2283,"children":2284},{},[2285],{"type":41,"tag":56,"props":2286,"children":2288},{"className":2287},[],[2289],{"type":47,"value":2290},"Invalid account or account\u002Fpartition combination",{"type":41,"tag":307,"props":2292,"children":2293},{},[2294],{"type":47,"value":2295},"Partition not available for account",{"type":41,"tag":307,"props":2297,"children":2298},{},[2299,2301],{"type":47,"value":2300},"Check combos: ",{"type":41,"tag":56,"props":2302,"children":2304},{"className":2303},[],[2305],{"type":47,"value":2306},"sacctmgr -nP show assoc where user=$USER format=account,partition",{"type":41,"tag":271,"props":2308,"children":2309},{},[2310,2319,2330],{"type":41,"tag":307,"props":2311,"children":2312},{},[2313],{"type":41,"tag":56,"props":2314,"children":2316},{"className":2315},[],[2317],{"type":47,"value":2318},"Unable to create step for job... Requested node configuration is not available",{"type":41,"tag":307,"props":2320,"children":2321},{},[2322,2328],{"type":41,"tag":56,"props":2323,"children":2325},{"className":2324},[],[2326],{"type":47,"value":2327},"-w \u003Cnode>",{"type":47,"value":2329}," conflicts with allocation",{"type":41,"tag":307,"props":2331,"children":2332},{},[2333,2335,2341],{"type":47,"value":2334},"Remove ",{"type":41,"tag":56,"props":2336,"children":2338},{"className":2337},[],[2339],{"type":47,"value":2340},"-w",{"type":47,"value":2342}," flag — HF cache is on shared filesystem, accessible from any node",{"type":41,"tag":271,"props":2344,"children":2345},{},[2346,2357,2369],{"type":41,"tag":307,"props":2347,"children":2348},{},[2349,2355],{"type":41,"tag":56,"props":2350,"children":2352},{"className":2351},[],[2353],{"type":47,"value":2354},"uv: command not found",{"type":47,"value":2356}," inside container",{"type":41,"tag":307,"props":2358,"children":2359},{},[2360,2362,2367],{"type":47,"value":2361},"Container doesn't have ",{"type":41,"tag":56,"props":2363,"children":2365},{"className":2364},[],[2366],{"type":47,"value":987},{"type":47,"value":2368}," pre-installed",{"type":41,"tag":307,"props":2370,"children":2371},{},[2372,2374,2379,2381],{"type":47,"value":2373},"Use a container with ",{"type":41,"tag":56,"props":2375,"children":2377},{"className":2376},[],[2378],{"type":47,"value":987},{"type":47,"value":2380}," pre-installed, or ",{"type":41,"tag":56,"props":2382,"children":2384},{"className":2383},[],[2385],{"type":47,"value":2386},"pip install uv",{"type":41,"tag":271,"props":2388,"children":2389},{},[2390,2414,2427],{"type":41,"tag":307,"props":2391,"children":2392},{},[2393,2399,2401,2406,2408],{"type":41,"tag":56,"props":2394,"children":2396},{"className":2395},[],[2397],{"type":47,"value":2398},"No space left on device",{"type":47,"value":2400}," during ",{"type":41,"tag":56,"props":2402,"children":2404},{"className":2403},[],[2405],{"type":47,"value":987},{"type":47,"value":2407}," or ",{"type":41,"tag":56,"props":2409,"children":2411},{"className":2410},[],[2412],{"type":47,"value":2413},"pip",{"type":41,"tag":307,"props":2415,"children":2416},{},[2417,2419,2425],{"type":47,"value":2418},"Container's ",{"type":41,"tag":56,"props":2420,"children":2422},{"className":2421},[],[2423],{"type":47,"value":2424},"\u002Froot\u002F.cache\u002F",{"type":47,"value":2426}," is full",{"type":41,"tag":307,"props":2428,"children":2429},{},[2430,2432],{"type":47,"value":2431},"Redirect: ",{"type":41,"tag":56,"props":2433,"children":2435},{"className":2434},[],[2436],{"type":47,"value":2437},"export UV_CACHE_DIR=\u003CSHARED_FS>\u002Fuv_cache",{"type":41,"tag":271,"props":2439,"children":2440},{},[2441,2450,2461],{"type":41,"tag":307,"props":2442,"children":2443},{},[2444],{"type":41,"tag":56,"props":2445,"children":2447},{"className":2446},[],[2448],{"type":47,"value":2449},"ModuleNotFoundError: No module named 'megatron.core.activations'",{"type":41,"tag":307,"props":2451,"children":2452},{},[2453,2455],{"type":47,"value":2454},"Container's pre-installed megatron-core conflicts with local ",{"type":41,"tag":56,"props":2456,"children":2458},{"className":2457},[],[2459],{"type":47,"value":2460},"3rdparty\u002FMegatron-LM",{"type":41,"tag":307,"props":2462,"children":2463},{},[2464,2466],{"type":47,"value":2465},"Install local: ",{"type":41,"tag":56,"props":2467,"children":2469},{"className":2468},[],[2470],{"type":47,"value":2471},"pip install -e 3rdparty\u002FMegatron-LM --no-deps --no-build-isolation",{"type":41,"tag":822,"props":2473,"children":2474},{},[],{"type":41,"tag":65,"props":2476,"children":2478},{"id":2477},"debugging-multi-node-failures",[2479],{"type":47,"value":2480},"Debugging Multi-Node Failures",{"type":41,"tag":529,"props":2482,"children":2484},{"id":2483},"quick-diagnosis",[2485],{"type":47,"value":2486},"Quick Diagnosis",{"type":41,"tag":50,"props":2488,"children":2489},{},[2490],{"type":47,"value":2491},"Check the log for these patterns (in order):",{"type":41,"tag":536,"props":2493,"children":2495},{"className":574,"code":2494,"language":576,"meta":541,"style":541},"# 1. Find the actual error (filter noise)\ngrep -a 'Error\\|OOM\\|CUDA out of memory\\|FAILED\\|Killed' job.log \\\n  | grep -v 'UserWarning\\|AllocatorConfig\\|transformer_engine\\|frame\\|srun: error'\n\n# 2. Check which rank crashed first\ngrep -a 'Failures:' -A 20 job.log | head -25\n\n# 3. Check for NCCL timeout\ngrep -a 'ncclUniqueId\\|timeout\\|crash on rank 0' job.log | head -5\n",[2496],{"type":41,"tag":56,"props":2497,"children":2498},{"__ignoreMap":541},[2499,2507,2543,2574,2583,2591,2640,2647,2655],{"type":41,"tag":582,"props":2500,"children":2501},{"class":584,"line":585},[2502],{"type":41,"tag":582,"props":2503,"children":2504},{"style":589},[2505],{"type":47,"value":2506},"# 1. Find the actual error (filter noise)\n",{"type":41,"tag":582,"props":2508,"children":2509},{"class":584,"line":595},[2510,2514,2519,2524,2529,2534,2539],{"type":41,"tag":582,"props":2511,"children":2512},{"style":984},[2513],{"type":47,"value":1393},{"type":41,"tag":582,"props":2515,"children":2516},{"style":990},[2517],{"type":47,"value":2518}," -a",{"type":41,"tag":582,"props":2520,"children":2521},{"style":1025},[2522],{"type":47,"value":2523}," '",{"type":41,"tag":582,"props":2525,"children":2526},{"style":990},[2527],{"type":47,"value":2528},"Error\\|OOM\\|CUDA out of memory\\|FAILED\\|Killed",{"type":41,"tag":582,"props":2530,"children":2531},{"style":1025},[2532],{"type":47,"value":2533},"'",{"type":41,"tag":582,"props":2535,"children":2536},{"style":990},[2537],{"type":47,"value":2538}," job.log",{"type":41,"tag":582,"props":2540,"children":2541},{"style":1016},[2542],{"type":47,"value":1019},{"type":41,"tag":582,"props":2544,"children":2545},{"class":584,"line":604},[2546,2551,2555,2560,2564,2569],{"type":41,"tag":582,"props":2547,"children":2548},{"style":1025},[2549],{"type":47,"value":2550},"  |",{"type":41,"tag":582,"props":2552,"children":2553},{"style":984},[2554],{"type":47,"value":1460},{"type":41,"tag":582,"props":2556,"children":2557},{"style":990},[2558],{"type":47,"value":2559}," -v",{"type":41,"tag":582,"props":2561,"children":2562},{"style":1025},[2563],{"type":47,"value":2523},{"type":41,"tag":582,"props":2565,"children":2566},{"style":990},[2567],{"type":47,"value":2568},"UserWarning\\|AllocatorConfig\\|transformer_engine\\|frame\\|srun: error",{"type":41,"tag":582,"props":2570,"children":2571},{"style":1025},[2572],{"type":47,"value":2573},"'\n",{"type":41,"tag":582,"props":2575,"children":2576},{"class":584,"line":613},[2577],{"type":41,"tag":582,"props":2578,"children":2580},{"emptyLinePlaceholder":2579},true,[2581],{"type":47,"value":2582},"\n",{"type":41,"tag":582,"props":2584,"children":2585},{"class":584,"line":622},[2586],{"type":41,"tag":582,"props":2587,"children":2588},{"style":589},[2589],{"type":47,"value":2590},"# 2. Check which rank crashed first\n",{"type":41,"tag":582,"props":2592,"children":2593},{"class":584,"line":631},[2594,2598,2602,2606,2610,2614,2619,2624,2628,2632,2636],{"type":41,"tag":582,"props":2595,"children":2596},{"style":984},[2597],{"type":47,"value":1393},{"type":41,"tag":582,"props":2599,"children":2600},{"style":990},[2601],{"type":47,"value":2518},{"type":41,"tag":582,"props":2603,"children":2604},{"style":1025},[2605],{"type":47,"value":2523},{"type":41,"tag":582,"props":2607,"children":2608},{"style":990},[2609],{"type":47,"value":227},{"type":41,"tag":582,"props":2611,"children":2612},{"style":1025},[2613],{"type":47,"value":2533},{"type":41,"tag":582,"props":2615,"children":2616},{"style":990},[2617],{"type":47,"value":2618}," -A",{"type":41,"tag":582,"props":2620,"children":2621},{"style":1571},[2622],{"type":47,"value":2623}," 20",{"type":41,"tag":582,"props":2625,"children":2626},{"style":990},[2627],{"type":47,"value":2538},{"type":41,"tag":582,"props":2629,"children":2630},{"style":1025},[2631],{"type":47,"value":1455},{"type":41,"tag":582,"props":2633,"children":2634},{"style":984},[2635],{"type":47,"value":1487},{"type":41,"tag":582,"props":2637,"children":2638},{"style":990},[2639],{"type":47,"value":1492},{"type":41,"tag":582,"props":2641,"children":2642},{"class":584,"line":640},[2643],{"type":41,"tag":582,"props":2644,"children":2645},{"emptyLinePlaceholder":2579},[2646],{"type":47,"value":2582},{"type":41,"tag":582,"props":2648,"children":2649},{"class":584,"line":649},[2650],{"type":41,"tag":582,"props":2651,"children":2652},{"style":589},[2653],{"type":47,"value":2654},"# 3. Check for NCCL timeout\n",{"type":41,"tag":582,"props":2656,"children":2657},{"class":584,"line":658},[2658,2662,2666,2670,2675,2679,2683,2687,2691],{"type":41,"tag":582,"props":2659,"children":2660},{"style":984},[2661],{"type":47,"value":1393},{"type":41,"tag":582,"props":2663,"children":2664},{"style":990},[2665],{"type":47,"value":2518},{"type":41,"tag":582,"props":2667,"children":2668},{"style":1025},[2669],{"type":47,"value":2523},{"type":41,"tag":582,"props":2671,"children":2672},{"style":990},[2673],{"type":47,"value":2674},"ncclUniqueId\\|timeout\\|crash on rank 0",{"type":41,"tag":582,"props":2676,"children":2677},{"style":1025},[2678],{"type":47,"value":2533},{"type":41,"tag":582,"props":2680,"children":2681},{"style":990},[2682],{"type":47,"value":2538},{"type":41,"tag":582,"props":2684,"children":2685},{"style":1025},[2686],{"type":47,"value":1455},{"type":41,"tag":582,"props":2688,"children":2689},{"style":984},[2690],{"type":47,"value":1487},{"type":41,"tag":582,"props":2692,"children":2693},{"style":990},[2694],{"type":47,"value":2695}," -5\n",{"type":41,"tag":529,"props":2697,"children":2699},{"id":2698},"debugging-checklist",[2700],{"type":47,"value":2701},"Debugging Checklist",{"type":41,"tag":50,"props":2703,"children":2704},{},[2705],{"type":47,"value":2706},"When a multi-node job fails:",{"type":41,"tag":77,"props":2708,"children":2709},{},[2710,2720,2730,2740,2750,2768],{"type":41,"tag":81,"props":2711,"children":2712},{},[2713,2718],{"type":41,"tag":87,"props":2714,"children":2715},{},[2716],{"type":47,"value":2717},"Check exit code",{"type":47,"value":2719},": 1 = Python error, 9 = OOM killed, 143 = SIGTERM (timeout or cascade)",{"type":41,"tag":81,"props":2721,"children":2722},{},[2723,2728],{"type":41,"tag":87,"props":2724,"children":2725},{},[2726],{"type":47,"value":2727},"Find first failure",{"type":47,"value":2729},": Which task\u002Fnode crashed first? Others get SIGTERM (143) as cascade",{"type":41,"tag":81,"props":2731,"children":2732},{},[2733,2738],{"type":41,"tag":87,"props":2734,"children":2735},{},[2736],{"type":47,"value":2737},"grep the actual error",{"type":47,"value":2739},": Filter out UserWarnings, NCCL frame dumps",{"type":41,"tag":81,"props":2741,"children":2742},{},[2743,2748],{"type":41,"tag":87,"props":2744,"children":2745},{},[2746],{"type":47,"value":2747},"Check rank 0 specifically",{"type":47,"value":2749},": Most save\u002Fexport errors happen on rank 0",{"type":41,"tag":81,"props":2751,"children":2752},{},[2753,2758,2760,2766],{"type":41,"tag":87,"props":2754,"children":2755},{},[2756],{"type":47,"value":2757},"Verify EP sizing",{"type":47,"value":2759},": For MoE models, ensure ",{"type":41,"tag":56,"props":2761,"children":2763},{"className":2762},[],[2764],{"type":47,"value":2765},"num_experts \u002F EP",{"type":47,"value":2767}," fits in GPU memory with headroom",{"type":41,"tag":81,"props":2769,"children":2770},{},[2771,2776,2778,2784],{"type":41,"tag":87,"props":2772,"children":2773},{},[2774],{"type":47,"value":2775},"Try interactive first",{"type":47,"value":2777},": Use ",{"type":41,"tag":56,"props":2779,"children":2781},{"className":2780},[],[2782],{"type":47,"value":2783},"salloc -N 2 -p interactive",{"type":47,"value":2785}," to iterate faster than sbatch queue",{"type":41,"tag":529,"props":2787,"children":2789},{"id":2788},"nccl-timeout-at-distbarrier-crash-on-rank-0",[2790,2792,2798],{"type":47,"value":2791},"NCCL Timeout at ",{"type":41,"tag":56,"props":2793,"children":2795},{"className":2794},[],[2796],{"type":47,"value":2797},"dist.barrier()",{"type":47,"value":2799}," — \"crash on rank 0\"",{"type":41,"tag":50,"props":2801,"children":2802},{},[2803,2808],{"type":41,"tag":87,"props":2804,"children":2805},{},[2806],{"type":47,"value":2807},"Symptom",{"type":47,"value":2809},": All ranks on node 2+ show:",{"type":41,"tag":536,"props":2811,"children":2814},{"className":2812,"code":2813,"language":47,"meta":541},[539],"[rank8] is setting up NCCL communicator and retrieving ncclUniqueId from [0]\n... wait timeout after 600000ms\nThis may indicate a possible application crash on rank 0\n",[2815],{"type":41,"tag":56,"props":2816,"children":2817},{"__ignoreMap":541},[2818],{"type":47,"value":2813},{"type":41,"tag":50,"props":2820,"children":2821},{},[2822,2827],{"type":41,"tag":87,"props":2823,"children":2824},{},[2825],{"type":47,"value":2826},"Root causes",{"type":47,"value":2828}," (check in order):",{"type":41,"tag":263,"props":2830,"children":2831},{},[2832,2851],{"type":41,"tag":267,"props":2833,"children":2834},{},[2835],{"type":41,"tag":271,"props":2836,"children":2837},{},[2838,2842,2847],{"type":41,"tag":275,"props":2839,"children":2840},{},[2841],{"type":47,"value":2179},{"type":41,"tag":275,"props":2843,"children":2844},{},[2845],{"type":47,"value":2846},"How to verify",{"type":41,"tag":275,"props":2848,"children":2849},{},[2850],{"type":47,"value":2184},{"type":41,"tag":300,"props":2852,"children":2853},{},[2854,2897,2939,2961],{"type":41,"tag":271,"props":2855,"children":2856},{},[2857,2868,2886],{"type":41,"tag":307,"props":2858,"children":2859},{},[2860,2866],{"type":41,"tag":56,"props":2861,"children":2863},{"className":2862},[],[2864],{"type":47,"value":2865},"save_artifacts",{"type":47,"value":2867}," hangs on rank 0",{"type":41,"tag":307,"props":2869,"children":2870},{},[2871,2873,2879,2881],{"type":47,"value":2872},"Error is in ",{"type":41,"tag":56,"props":2874,"children":2876},{"className":2875},[],[2877],{"type":47,"value":2878},"save_hf_weights",{"type":47,"value":2880}," → ",{"type":41,"tag":56,"props":2882,"children":2884},{"className":2883},[],[2885],{"type":47,"value":2797},{"type":41,"tag":307,"props":2887,"children":2888},{},[2889,2891],{"type":47,"value":2890},"Increase timeout: ",{"type":41,"tag":56,"props":2892,"children":2894},{"className":2893},[],[2895],{"type":47,"value":2896},"init_process_group(\"nccl\", timeout=timedelta(minutes=60))",{"type":41,"tag":271,"props":2898,"children":2899},{},[2900,2911,2920],{"type":41,"tag":307,"props":2901,"children":2902},{},[2903,2909],{"type":41,"tag":56,"props":2904,"children":2906},{"className":2905},[],[2907],{"type":47,"value":2908},"ImportError",{"type":47,"value":2910}," in custom model code",{"type":41,"tag":307,"props":2912,"children":2913},{},[2914],{"type":41,"tag":56,"props":2915,"children":2917},{"className":2916},[],[2918],{"type":47,"value":2919},"grep ImportError job.log",{"type":41,"tag":307,"props":2921,"children":2922},{},[2923,2925,2930,2932,2937],{"type":47,"value":2924},"Catch ",{"type":41,"tag":56,"props":2926,"children":2928},{"className":2927},[],[2929],{"type":47,"value":2908},{"type":47,"value":2931}," in ",{"type":41,"tag":56,"props":2933,"children":2935},{"className":2934},[],[2936],{"type":47,"value":2865},{"type":47,"value":2938}," (see below)",{"type":41,"tag":271,"props":2940,"children":2941},{},[2942,2947,2956],{"type":41,"tag":307,"props":2943,"children":2944},{},[2945],{"type":47,"value":2946},"Rank 0 OOM during export",{"type":41,"tag":307,"props":2948,"children":2949},{},[2950],{"type":41,"tag":56,"props":2951,"children":2953},{"className":2952},[],[2954],{"type":47,"value":2955},"grep 'OutOfMemory' job.log",{"type":41,"tag":307,"props":2957,"children":2958},{},[2959],{"type":47,"value":2960},"Increase EP or nodes",{"type":41,"tag":271,"props":2962,"children":2963},{},[2964,2969,2974],{"type":41,"tag":307,"props":2965,"children":2966},{},[2967],{"type":47,"value":2968},"Network issue between nodes",{"type":41,"tag":307,"props":2970,"children":2971},{},[2972],{"type":47,"value":2973},"Error only on cross-node ranks",{"type":41,"tag":307,"props":2975,"children":2976},{},[2977,2979,2985],{"type":47,"value":2978},"Check ",{"type":41,"tag":56,"props":2980,"children":2982},{"className":2981},[],[2983],{"type":47,"value":2984},"sinfo",{"type":47,"value":2986},", try different nodes",{"type":41,"tag":50,"props":2988,"children":2989},{},[2990,3002,3004,3010,3012,3018,3020,3025,3027,3032],{"type":41,"tag":87,"props":2991,"children":2992},{},[2993,2995,3000],{"type":47,"value":2994},"The ",{"type":41,"tag":56,"props":2996,"children":2998},{"className":2997},[],[2999],{"type":47,"value":2865},{"type":47,"value":3001}," problem",{"type":47,"value":3003},": When ",{"type":41,"tag":56,"props":3005,"children":3007},{"className":3006},[],[3008],{"type":47,"value":3009},"trust_remote_code=True",{"type":47,"value":3011},", rank 0 runs ",{"type":41,"tag":56,"props":3013,"children":3015},{"className":3014},[],[3016],{"type":47,"value":3017},"save_artifacts()",{"type":47,"value":3019}," (downloads tokenizer, config, custom modeling code) while all other ranks skip directly to ",{"type":41,"tag":56,"props":3021,"children":3023},{"className":3022},[],[3024],{"type":47,"value":2797},{"type":47,"value":3026},". If ",{"type":41,"tag":56,"props":3028,"children":3030},{"className":3029},[],[3031],{"type":47,"value":2865},{"type":47,"value":3033}," is slow or crashes, other ranks timeout.",{"type":41,"tag":50,"props":3035,"children":3036},{},[3037,3042,3044,3050],{"type":41,"tag":87,"props":3038,"children":3039},{},[3040],{"type":47,"value":3041},"Fix for ImportError in save_artifacts",{"type":47,"value":3043}," (",{"type":41,"tag":56,"props":3045,"children":3047},{"className":3046},[],[3048],{"type":47,"value":3049},"hf_pretrained\u002Fbase.py",{"type":47,"value":3051},"):",{"type":41,"tag":536,"props":3053,"children":3057},{"className":3054,"code":3055,"language":3056,"meta":541,"style":541},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Change:\nexcept OSError:\n    pass\n# To:\nexcept (OSError, ImportError):\n    pass\n","python",[3058],{"type":41,"tag":56,"props":3059,"children":3060},{"__ignoreMap":541},[3061,3069,3077,3085,3093,3101],{"type":41,"tag":582,"props":3062,"children":3063},{"class":584,"line":585},[3064],{"type":41,"tag":582,"props":3065,"children":3066},{},[3067],{"type":47,"value":3068},"# Change:\n",{"type":41,"tag":582,"props":3070,"children":3071},{"class":584,"line":595},[3072],{"type":41,"tag":582,"props":3073,"children":3074},{},[3075],{"type":47,"value":3076},"except OSError:\n",{"type":41,"tag":582,"props":3078,"children":3079},{"class":584,"line":604},[3080],{"type":41,"tag":582,"props":3081,"children":3082},{},[3083],{"type":47,"value":3084},"    pass\n",{"type":41,"tag":582,"props":3086,"children":3087},{"class":584,"line":613},[3088],{"type":41,"tag":582,"props":3089,"children":3090},{},[3091],{"type":47,"value":3092},"# To:\n",{"type":41,"tag":582,"props":3094,"children":3095},{"class":584,"line":622},[3096],{"type":41,"tag":582,"props":3097,"children":3098},{},[3099],{"type":47,"value":3100},"except (OSError, ImportError):\n",{"type":41,"tag":582,"props":3102,"children":3103},{"class":584,"line":631},[3104],{"type":41,"tag":582,"props":3105,"children":3106},{},[3107],{"type":47,"value":3084},{"type":41,"tag":529,"props":3109,"children":3111},{"id":3110},"oom-for-moe-models",[3112],{"type":47,"value":3113},"OOM for MoE Models",{"type":41,"tag":50,"props":3115,"children":3116},{},[3117,3121,3122,3128],{"type":41,"tag":87,"props":3118,"children":3119},{},[3120],{"type":47,"value":2807},{"type":47,"value":101},{"type":41,"tag":56,"props":3123,"children":3125},{"className":3124},[],[3126],{"type":47,"value":3127},"torch.OutOfMemoryError: CUDA out of memory",{"type":47,"value":3129}," during model loading or forward pass.",{"type":41,"tag":50,"props":3131,"children":3132},{},[3133,3138],{"type":41,"tag":87,"props":3134,"children":3135},{},[3136],{"type":47,"value":3137},"Key insight",{"type":47,"value":3139},": TP does NOT reduce expert memory. Only EP splits experts across GPUs.",{"type":41,"tag":50,"props":3141,"children":3142},{},[3143,3148],{"type":41,"tag":87,"props":3144,"children":3145},{},[3146],{"type":47,"value":3147},"Sizing formula",{"type":47,"value":3149},":",{"type":41,"tag":536,"props":3151,"children":3154},{"className":3152,"code":3153,"language":47,"meta":541},[539],"experts_per_gpu = num_experts \u002F EP\nexpert_memory_gb ≈ experts_per_gpu * expert_params * 2 \u002F 1e9  (bf16)\ntotal_per_gpu ≈ expert_memory_gb + attention_memory_gb + kv_cache_gb\n",[3155],{"type":41,"tag":56,"props":3156,"children":3157},{"__ignoreMap":541},[3158],{"type":47,"value":3153},{"type":41,"tag":50,"props":3160,"children":3161},{},[3162,3167],{"type":41,"tag":87,"props":3163,"children":3164},{},[3165],{"type":47,"value":3166},"MiniMax-M2 example",{"type":47,"value":3168}," (256 experts, ~230GB fp8 → ~460GB bf16):",{"type":41,"tag":263,"props":3170,"children":3171},{},[3172,3203],{"type":41,"tag":267,"props":3173,"children":3174},{},[3175],{"type":41,"tag":271,"props":3176,"children":3177},{},[3178,3183,3188,3193,3198],{"type":41,"tag":275,"props":3179,"children":3180},{},[3181],{"type":47,"value":3182},"Config",{"type":41,"tag":275,"props":3184,"children":3185},{},[3186],{"type":47,"value":3187},"Nodes",{"type":41,"tag":275,"props":3189,"children":3190},{},[3191],{"type":47,"value":3192},"GPUs",{"type":41,"tag":275,"props":3194,"children":3195},{},[3196],{"type":47,"value":3197},"Experts\u002FGPU",{"type":41,"tag":275,"props":3199,"children":3200},{},[3201],{"type":47,"value":3202},"Result",{"type":41,"tag":300,"props":3204,"children":3205},{},[3206,3232,3260,3285],{"type":41,"tag":271,"props":3207,"children":3208},{},[3209,3214,3218,3222,3227],{"type":41,"tag":307,"props":3210,"children":3211},{},[3212],{"type":47,"value":3213},"TP=2, EP=4",{"type":41,"tag":307,"props":3215,"children":3216},{},[3217],{"type":47,"value":348},{"type":41,"tag":307,"props":3219,"children":3220},{},[3221],{"type":47,"value":320},{"type":41,"tag":307,"props":3223,"children":3224},{},[3225],{"type":47,"value":3226},"64",{"type":41,"tag":307,"props":3228,"children":3229},{},[3230],{"type":47,"value":3231},"OOM (too many experts)",{"type":41,"tag":271,"props":3233,"children":3234},{},[3235,3240,3245,3250,3255],{"type":41,"tag":307,"props":3236,"children":3237},{},[3238],{"type":47,"value":3239},"TP=2, EP=8",{"type":41,"tag":307,"props":3241,"children":3242},{},[3243],{"type":47,"value":3244},"2",{"type":41,"tag":307,"props":3246,"children":3247},{},[3248],{"type":47,"value":3249},"16",{"type":41,"tag":307,"props":3251,"children":3252},{},[3253],{"type":47,"value":3254},"32",{"type":41,"tag":307,"props":3256,"children":3257},{},[3258],{"type":47,"value":3259},"Works for roundtrip (weight-only), OOM for inference",{"type":41,"tag":271,"props":3261,"children":3262},{},[3263,3268,3272,3276,3280],{"type":41,"tag":307,"props":3264,"children":3265},{},[3266],{"type":47,"value":3267},"TP=1, EP=16",{"type":41,"tag":307,"props":3269,"children":3270},{},[3271],{"type":47,"value":3244},{"type":41,"tag":307,"props":3273,"children":3274},{},[3275],{"type":47,"value":3249},{"type":41,"tag":307,"props":3277,"children":3278},{},[3279],{"type":47,"value":3249},{"type":41,"tag":307,"props":3281,"children":3282},{},[3283],{"type":47,"value":3284},"Works for inference",{"type":41,"tag":271,"props":3286,"children":3287},{},[3288,3293,3297,3301,3305],{"type":41,"tag":307,"props":3289,"children":3290},{},[3291],{"type":47,"value":3292},"TP=2, EP=32",{"type":41,"tag":307,"props":3294,"children":3295},{},[3296],{"type":47,"value":320},{"type":41,"tag":307,"props":3298,"children":3299},{},[3300],{"type":47,"value":3226},{"type":41,"tag":307,"props":3302,"children":3303},{},[3304],{"type":47,"value":320},{"type":41,"tag":307,"props":3306,"children":3307},{},[3308],{"type":47,"value":3309},"Comfortable for training",{"type":41,"tag":50,"props":3311,"children":3312},{},[3313,3318],{"type":41,"tag":87,"props":3314,"children":3315},{},[3316],{"type":47,"value":3317},"Rules of thumb",{"type":47,"value":3149},{"type":41,"tag":209,"props":3320,"children":3321},{},[3322,3327,3332],{"type":41,"tag":81,"props":3323,"children":3324},{},[3325],{"type":47,"value":3326},"Roundtrip (weight-only): can use more experts per GPU (~60GB model params OK)",{"type":41,"tag":81,"props":3328,"children":3329},{},[3330],{"type":47,"value":3331},"Inference (forward pass + KV cache): needs headroom (~40GB model params max)",{"type":41,"tag":81,"props":3333,"children":3334},{},[3335],{"type":47,"value":3336},"Training (activations + optimizer): needs even more headroom (~30GB model params max)",{"type":41,"tag":529,"props":3338,"children":3340},{"id":3339},"modulenotfounderror-no-module-named-megatroncoretensor_parallel",[3341],{"type":41,"tag":56,"props":3342,"children":3344},{"className":3343},[],[3345],{"type":47,"value":3346},"ModuleNotFoundError: No module named 'megatron.core.tensor_parallel'",{"type":41,"tag":50,"props":3348,"children":3349},{},[3350,3354,3356,3361],{"type":41,"tag":87,"props":3351,"children":3352},{},[3353],{"type":47,"value":2179},{"type":47,"value":3355},": Container's pre-installed megatron-core conflicts with local ",{"type":41,"tag":56,"props":3357,"children":3359},{"className":3358},[],[3360],{"type":47,"value":2460},{"type":47,"value":125},{"type":41,"tag":50,"props":3363,"children":3364},{},[3365,3369,3371,3376],{"type":41,"tag":87,"props":3366,"children":3367},{},[3368],{"type":47,"value":2184},{"type":47,"value":3370},": Add ",{"type":41,"tag":56,"props":3372,"children":3374},{"className":3373},[],[3375],{"type":47,"value":689},{"type":47,"value":3377}," before running:",{"type":41,"tag":536,"props":3379,"children":3381},{"className":574,"code":3380,"language":576,"meta":541,"style":541},"CMD=\"if [ \\\"\\$SLURM_LOCALID\\\" -eq 0 ]; then uv sync; else sleep 10; fi && \"\nCMD=\"${CMD}uv run --no-sync python \u003Cscript> \u003Cargs>\"\n",[3382],{"type":41,"tag":56,"props":3383,"children":3384},{"__ignoreMap":541},[3385,3429],{"type":41,"tag":582,"props":3386,"children":3387},{"class":584,"line":585},[3388,3393,3397,3401,3406,3411,3415,3420,3425],{"type":41,"tag":582,"props":3389,"children":3390},{"style":1016},[3391],{"type":47,"value":3392},"CMD",{"type":41,"tag":582,"props":3394,"children":3395},{"style":1025},[3396],{"type":47,"value":1947},{"type":41,"tag":582,"props":3398,"children":3399},{"style":1025},[3400],{"type":47,"value":1412},{"type":41,"tag":582,"props":3402,"children":3403},{"style":990},[3404],{"type":47,"value":3405},"if [ ",{"type":41,"tag":582,"props":3407,"children":3408},{"style":1016},[3409],{"type":47,"value":3410},"\\\"\\$",{"type":41,"tag":582,"props":3412,"children":3413},{"style":990},[3414],{"type":47,"value":424},{"type":41,"tag":582,"props":3416,"children":3417},{"style":1016},[3418],{"type":47,"value":3419},"\\\"",{"type":41,"tag":582,"props":3421,"children":3422},{"style":990},[3423],{"type":47,"value":3424}," -eq 0 ]; then uv sync; else sleep 10; fi && ",{"type":41,"tag":582,"props":3426,"children":3427},{"style":1025},[3428],{"type":47,"value":1346},{"type":41,"tag":582,"props":3430,"children":3431},{"class":584,"line":595},[3432,3436,3440,3445,3449,3453,3458],{"type":41,"tag":582,"props":3433,"children":3434},{"style":1016},[3435],{"type":47,"value":3392},{"type":41,"tag":582,"props":3437,"children":3438},{"style":1025},[3439],{"type":47,"value":1947},{"type":41,"tag":582,"props":3441,"children":3442},{"style":1025},[3443],{"type":47,"value":3444},"\"${",{"type":41,"tag":582,"props":3446,"children":3447},{"style":1016},[3448],{"type":47,"value":3392},{"type":41,"tag":582,"props":3450,"children":3451},{"style":1025},[3452],{"type":47,"value":1441},{"type":41,"tag":582,"props":3454,"children":3455},{"style":990},[3456],{"type":47,"value":3457},"uv run --no-sync python \u003Cscript> \u003Cargs>",{"type":41,"tag":582,"props":3459,"children":3460},{"style":1025},[3461],{"type":47,"value":1346},{"type":41,"tag":529,"props":3463,"children":3465},{"id":3464},"fp8-weight-mismatch-in-roundtrip",[3466],{"type":47,"value":3467},"FP8 Weight Mismatch in Roundtrip",{"type":41,"tag":50,"props":3469,"children":3470},{},[3471,3475,3477,3483],{"type":41,"tag":87,"props":3472,"children":3473},{},[3474],{"type":47,"value":2807},{"type":47,"value":3476},": Roundtrip completes but shows ❌ for all expert weights and raises ",{"type":41,"tag":56,"props":3478,"children":3480},{"className":3479},[],[3481],{"type":47,"value":3482},"ValueError: Weight mismatch detected",{"type":47,"value":125},{"type":41,"tag":50,"props":3485,"children":3486},{},[3487,3491,3493,3499],{"type":41,"tag":87,"props":3488,"children":3489},{},[3490],{"type":47,"value":2179},{"type":47,"value":3492},": Original HF weights are FP8, Megatron stores in BF16. Exported weights are BF16. Comparison against original FP8 exceeds ",{"type":41,"tag":56,"props":3494,"children":3496},{"className":3495},[],[3497],{"type":47,"value":3498},"atol=1e-1",{"type":47,"value":125},{"type":41,"tag":50,"props":3501,"children":3502},{},[3503,3508],{"type":41,"tag":87,"props":3504,"children":3505},{},[3506],{"type":47,"value":3507},"This is expected for FP8 models.",{"type":47,"value":3509}," The conversion is correct; the comparison tolerance is insufficient for the FP8→BF16 precision gap.",{"type":41,"tag":529,"props":3511,"children":3513},{"id":3512},"world_size-not-set-with-srun",[3514,3519],{"type":41,"tag":56,"props":3515,"children":3517},{"className":3516},[],[3518],{"type":47,"value":144},{"type":47,"value":3520}," Not Set with srun",{"type":41,"tag":50,"props":3522,"children":3523},{},[3524,3528],{"type":41,"tag":87,"props":3525,"children":3526},{},[3527],{"type":47,"value":2807},{"type":47,"value":3529},": Script exits with \"must be launched with torchrun\".",{"type":41,"tag":50,"props":3531,"children":3532},{},[3533,3537,3539,3545],{"type":41,"tag":87,"props":3534,"children":3535},{},[3536],{"type":47,"value":2179},{"type":47,"value":3538},": Scripts check ",{"type":41,"tag":56,"props":3540,"children":3542},{"className":3541},[],[3543],{"type":47,"value":3544},"os.environ.get(\"WORLD_SIZE\")",{"type":47,"value":3546}," which torchrun sets but srun doesn't.",{"type":41,"tag":50,"props":3548,"children":3549},{},[3550,3554,3556,3561],{"type":41,"tag":87,"props":3551,"children":3552},{},[3553],{"type":47,"value":2184},{"type":47,"value":3555},": Also check ",{"type":41,"tag":56,"props":3557,"children":3559},{"className":3558},[],[3560],{"type":47,"value":417},{"type":47,"value":3149},{"type":41,"tag":536,"props":3563,"children":3565},{"className":3054,"code":3564,"language":3056,"meta":541,"style":541},"if os.environ.get(\"WORLD_SIZE\") is None and os.environ.get(\"SLURM_NTASKS\") is None:\n    sys.exit(1)\n",[3566],{"type":41,"tag":56,"props":3567,"children":3568},{"__ignoreMap":541},[3569,3577],{"type":41,"tag":582,"props":3570,"children":3571},{"class":584,"line":585},[3572],{"type":41,"tag":582,"props":3573,"children":3574},{},[3575],{"type":47,"value":3576},"if os.environ.get(\"WORLD_SIZE\") is None and os.environ.get(\"SLURM_NTASKS\") is None:\n",{"type":41,"tag":582,"props":3578,"children":3579},{"class":584,"line":595},[3580],{"type":41,"tag":582,"props":3581,"children":3582},{},[3583],{"type":47,"value":3584},"    sys.exit(1)\n",{"type":41,"tag":50,"props":3586,"children":3587},{},[3588,3590,3595,3597,3602],{"type":47,"value":3589},"Bridge's ",{"type":41,"tag":56,"props":3591,"children":3593},{"className":3592},[],[3594],{"type":47,"value":439},{"type":47,"value":3596}," helpers (called by ",{"type":41,"tag":56,"props":3598,"children":3600},{"className":3599},[],[3601],{"type":47,"value":99},{"type":47,"value":3603},") populate env vars from SLURM:",{"type":41,"tag":536,"props":3605,"children":3607},{"className":3054,"code":3606,"language":3056,"meta":541,"style":541},"if \"RANK\" not in os.environ:\n    os.environ[\"RANK\"] = str(get_rank_safe())          # uses SLURM_PROCID\nif \"WORLD_SIZE\" not in os.environ:\n    os.environ[\"WORLD_SIZE\"] = str(get_world_size_safe())  # uses SLURM_NTASKS\nif \"MASTER_ADDR\" not in os.environ:\n    os.environ[\"MASTER_ADDR\"] = get_master_addr_safe()     # parses SLURM_NODELIST\nif \"MASTER_PORT\" not in os.environ:\n    os.environ[\"MASTER_PORT\"] = str(get_master_port_safe()) # derives from SLURM_JOB_ID\n",[3608],{"type":41,"tag":56,"props":3609,"children":3610},{"__ignoreMap":541},[3611,3619,3627,3635,3643,3651,3659,3667],{"type":41,"tag":582,"props":3612,"children":3613},{"class":584,"line":585},[3614],{"type":41,"tag":582,"props":3615,"children":3616},{},[3617],{"type":47,"value":3618},"if \"RANK\" not in os.environ:\n",{"type":41,"tag":582,"props":3620,"children":3621},{"class":584,"line":595},[3622],{"type":41,"tag":582,"props":3623,"children":3624},{},[3625],{"type":47,"value":3626},"    os.environ[\"RANK\"] = str(get_rank_safe())          # uses SLURM_PROCID\n",{"type":41,"tag":582,"props":3628,"children":3629},{"class":584,"line":604},[3630],{"type":41,"tag":582,"props":3631,"children":3632},{},[3633],{"type":47,"value":3634},"if \"WORLD_SIZE\" not in os.environ:\n",{"type":41,"tag":582,"props":3636,"children":3637},{"class":584,"line":613},[3638],{"type":41,"tag":582,"props":3639,"children":3640},{},[3641],{"type":47,"value":3642},"    os.environ[\"WORLD_SIZE\"] = str(get_world_size_safe())  # uses SLURM_NTASKS\n",{"type":41,"tag":582,"props":3644,"children":3645},{"class":584,"line":622},[3646],{"type":41,"tag":582,"props":3647,"children":3648},{},[3649],{"type":47,"value":3650},"if \"MASTER_ADDR\" not in os.environ:\n",{"type":41,"tag":582,"props":3652,"children":3653},{"class":584,"line":631},[3654],{"type":41,"tag":582,"props":3655,"children":3656},{},[3657],{"type":47,"value":3658},"    os.environ[\"MASTER_ADDR\"] = get_master_addr_safe()     # parses SLURM_NODELIST\n",{"type":41,"tag":582,"props":3660,"children":3661},{"class":584,"line":640},[3662],{"type":41,"tag":582,"props":3663,"children":3664},{},[3665],{"type":47,"value":3666},"if \"MASTER_PORT\" not in os.environ:\n",{"type":41,"tag":582,"props":3668,"children":3669},{"class":584,"line":649},[3670],{"type":41,"tag":582,"props":3671,"children":3672},{},[3673],{"type":47,"value":3674},"    os.environ[\"MASTER_PORT\"] = str(get_master_port_safe()) # derives from SLURM_JOB_ID\n",{"type":41,"tag":822,"props":3676,"children":3677},{},[],{"type":41,"tag":65,"props":3679,"children":3681},{"id":3680},"key-gotchas",[3682],{"type":47,"value":3683},"Key Gotchas",{"type":41,"tag":77,"props":3685,"children":3686},{},[3687,3709,3738,3815,3831,3889,3922],{"type":41,"tag":81,"props":3688,"children":3689},{},[3690,3700,3702,3707],{"type":41,"tag":87,"props":3691,"children":3692},{},[3693,3695],{"type":47,"value":3694},"Two-phase srun for ",{"type":41,"tag":56,"props":3696,"children":3698},{"className":3697},[],[3699],{"type":47,"value":689},{"type":47,"value":3701},": Run a single-process srun first to warm the cache, then the full multi-node srun. The second ",{"type":41,"tag":56,"props":3703,"children":3705},{"className":3704},[],[3706],{"type":47,"value":689},{"type":47,"value":3708}," is a fast no-op since everything is already cached on the shared filesystem.",{"type":41,"tag":81,"props":3710,"children":3711},{},[3712,3721,3723,3728,3730,3736],{"type":41,"tag":87,"props":3713,"children":3714},{},[3715],{"type":41,"tag":56,"props":3716,"children":3718},{"className":3717},[],[3719],{"type":47,"value":3720},"--no-container-mount-home",{"type":47,"value":3722}," is an ",{"type":41,"tag":56,"props":3724,"children":3726},{"className":3725},[],[3727],{"type":47,"value":681},{"type":47,"value":3729}," flag, NOT an ",{"type":41,"tag":56,"props":3731,"children":3733},{"className":3732},[],[3734],{"type":47,"value":3735},"#SBATCH",{"type":47,"value":3737}," directive.",{"type":41,"tag":81,"props":3739,"children":3740},{},[3741,3746,3748,3753,3755,3761,3763],{"type":41,"tag":87,"props":3742,"children":3743},{},[3744],{"type":47,"value":3745},"Escaping inside TRAIN_CMD",{"type":47,"value":3747},": Since ",{"type":41,"tag":56,"props":3749,"children":3751},{"className":3750},[],[3752],{"type":47,"value":1074},{"type":47,"value":3754}," is a double-quoted string, escape inner ",{"type":41,"tag":56,"props":3756,"children":3758},{"className":3757},[],[3759],{"type":47,"value":3760},"$",{"type":47,"value":3762}," for Slurm variables that must expand at runtime (not sbatch time):",{"type":41,"tag":209,"props":3764,"children":3765},{},[3766,3789],{"type":41,"tag":81,"props":3767,"children":3768},{},[3769,3775,3776,3782,3783],{"type":41,"tag":56,"props":3770,"children":3772},{"className":3771},[],[3773],{"type":47,"value":3774},"\\${SLURM_PROCID}",{"type":47,"value":138},{"type":41,"tag":56,"props":3777,"children":3779},{"className":3778},[],[3780],{"type":47,"value":3781},"\\${SLURM_JOB_NUM_NODES}",{"type":47,"value":138},{"type":41,"tag":56,"props":3784,"children":3786},{"className":3785},[],[3787],{"type":47,"value":3788},"\\${SLURM_NODEID}",{"type":41,"tag":81,"props":3790,"children":3791},{},[3792,3794,3800,3801,3806,3807,3813],{"type":47,"value":3793},"Host-side variables like ",{"type":41,"tag":56,"props":3795,"children":3797},{"className":3796},[],[3798],{"type":47,"value":3799},"$GH_TOKEN",{"type":47,"value":138},{"type":41,"tag":56,"props":3802,"children":3804},{"className":3803},[],[3805],{"type":47,"value":1421},{"type":47,"value":138},{"type":41,"tag":56,"props":3808,"children":3810},{"className":3809},[],[3811],{"type":47,"value":3812},"$WORKDIR",{"type":47,"value":3814}," expand at sbatch time — no escaping needed.",{"type":41,"tag":81,"props":3816,"children":3817},{},[3818,3829],{"type":41,"tag":87,"props":3819,"children":3820},{},[3821,3823],{"type":47,"value":3822},"Bridge ",{"type":41,"tag":56,"props":3824,"children":3826},{"className":3825},[],[3827],{"type":47,"value":3828},"rm -rf nemo_experiments",{"type":47,"value":3830},": Add before training to avoid stale checkpoint auto-resume.",{"type":41,"tag":81,"props":3832,"children":3833},{},[3834,3839,3841],{"type":41,"tag":87,"props":3835,"children":3836},{},[3837],{"type":47,"value":3838},"MLM needs PYTHONPATH",{"type":47,"value":3840},": For pretrain_gpt.py scripts, add inside TRAIN_CMD:",{"type":41,"tag":536,"props":3842,"children":3844},{"className":574,"code":3843,"language":576,"meta":541,"style":541},"PYTHONPATH=${WORKDIR}\u002F3rdparty\u002FMegatron-LM:\\${PYTHONPATH:-} \\\n",[3845],{"type":41,"tag":56,"props":3846,"children":3847},{"__ignoreMap":541},[3848],{"type":41,"tag":582,"props":3849,"children":3850},{"class":584,"line":585},[3851,3856,3861,3866,3870,3875,3880,3885],{"type":41,"tag":582,"props":3852,"children":3853},{"style":1016},[3854],{"type":47,"value":3855},"PYTHONPATH",{"type":41,"tag":582,"props":3857,"children":3858},{"style":1025},[3859],{"type":47,"value":3860},"=${",{"type":41,"tag":582,"props":3862,"children":3863},{"style":1016},[3864],{"type":47,"value":3865},"WORKDIR",{"type":41,"tag":582,"props":3867,"children":3868},{"style":1025},[3869],{"type":47,"value":1441},{"type":41,"tag":582,"props":3871,"children":3872},{"style":990},[3873],{"type":47,"value":3874},"\u002F3rdparty\u002FMegatron-LM:",{"type":41,"tag":582,"props":3876,"children":3877},{"style":1016},[3878],{"type":47,"value":3879},"\\${",{"type":41,"tag":582,"props":3881,"children":3882},{"style":984},[3883],{"type":47,"value":3884},"PYTHONPATH:-}",{"type":41,"tag":582,"props":3886,"children":3887},{"style":1016},[3888],{"type":47,"value":1019},{"type":41,"tag":81,"props":3890,"children":3891},{},[3892,3897,3899,3905,3907,3913,3915,3921],{"type":41,"tag":87,"props":3893,"children":3894},{},[3895],{"type":47,"value":3896},"Node count heuristic",{"type":47,"value":3898},": Total GPUs = ",{"type":41,"tag":56,"props":3900,"children":3902},{"className":3901},[],[3903],{"type":47,"value":3904},"NNODES * 8",{"type":47,"value":3906},". Must satisfy: ",{"type":41,"tag":56,"props":3908,"children":3910},{"className":3909},[],[3911],{"type":47,"value":3912},"TP * PP * EP * DP >= total_GPUs",{"type":47,"value":3914}," where ",{"type":41,"tag":56,"props":3916,"children":3918},{"className":3917},[],[3919],{"type":47,"value":3920},"DP = total_GPUs \u002F (TP * PP * EP)",{"type":47,"value":125},{"type":41,"tag":81,"props":3923,"children":3924},{},[3925,3935,3937,3942,3944,3950,3952,3958,3960,3966],{"type":41,"tag":87,"props":3926,"children":3927},{},[3928,3933],{"type":41,"tag":56,"props":3929,"children":3931},{"className":3930},[],[3932],{"type":47,"value":201},{"type":47,"value":3934}," on shared filesystem for multi-node SFT",{"type":47,"value":3936},": The default nemo cache (",{"type":41,"tag":56,"props":3938,"children":3940},{"className":3939},[],[3941],{"type":47,"value":491},{"type":47,"value":3943},")\nis container-local. Multi-node SFT with packed sequences prepares ",{"type":41,"tag":56,"props":3945,"children":3947},{"className":3946},[],[3948],{"type":47,"value":3949},".npy",{"type":47,"value":3951}," files on one node\nthat are invisible to others. Set ",{"type":41,"tag":56,"props":3953,"children":3955},{"className":3954},[],[3956],{"type":47,"value":3957},"export NEMO_HOME=\u003CSHARED_FS>\u002Fcache\u002Fnemo",{"type":47,"value":3959}," so packed data\nis shared. Without this, ranks on other nodes fail with ",{"type":41,"tag":56,"props":3961,"children":3963},{"className":3962},[],[3964],{"type":47,"value":3965},"TypeError: 'NoneType' object is not an iterator",{"type":47,"value":125},{"type":41,"tag":65,"props":3968,"children":3970},{"id":3969},"full-templates-and-command-bodies",[3971],{"type":47,"value":3972},"Full Templates and Command Bodies",{"type":41,"tag":50,"props":3974,"children":3975},{},[3976,3978,3983,3985,3990],{"type":47,"value":3977},"For copyable sbatch scaffolding and Bridge\u002FMLM-specific ",{"type":41,"tag":56,"props":3979,"children":3981},{"className":3980},[],[3982],{"type":47,"value":1074},{"type":47,"value":3984}," bodies, read\n",{"type":41,"tag":3986,"props":3987,"children":3988},"a",{"href":526},[3989],{"type":47,"value":526},{"type":47,"value":125},{"type":41,"tag":3992,"props":3993,"children":3994},"style",{},[3995],{"type":47,"value":3996},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3998,"total":4098},[3999,4014,4026,4040,4052,4069,4084],{"slug":4000,"name":4000,"fn":4001,"description":4002,"org":4003,"tags":4004,"stars":23,"repoUrl":24,"updatedAt":4013},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4005,4008,4011,4012],{"name":4006,"slug":4007,"type":15},"Data Analysis","data-analysis",{"name":4009,"slug":4010,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-14T05:28:43.176466",{"slug":4015,"name":4015,"fn":4016,"description":4017,"org":4018,"tags":4019,"stars":23,"repoUrl":24,"updatedAt":4025},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4020,4021,4024],{"name":17,"slug":18,"type":15},{"name":4022,"slug":4023,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":4027,"name":4027,"fn":4028,"description":4029,"org":4030,"tags":4031,"stars":23,"repoUrl":24,"updatedAt":4039},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4032,4035,4036],{"name":4033,"slug":4034,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":4037,"slug":4038,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":4041,"name":4041,"fn":4042,"description":4043,"org":4044,"tags":4045,"stars":23,"repoUrl":24,"updatedAt":4051},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4046,4047,4048],{"name":4006,"slug":4007,"type":15},{"name":9,"slug":8,"type":15},{"name":4049,"slug":4050,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":4053,"name":4053,"fn":4054,"description":4055,"org":4056,"tags":4057,"stars":23,"repoUrl":24,"updatedAt":4068},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4058,4061,4064,4065],{"name":4059,"slug":4060,"type":15},"Automation","automation",{"name":4062,"slug":4063,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":4066,"slug":4067,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":4070,"name":4070,"fn":4071,"description":4072,"org":4073,"tags":4074,"stars":23,"repoUrl":24,"updatedAt":4083},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4075,4076,4079,4080],{"name":17,"slug":18,"type":15},{"name":4077,"slug":4078,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":4081,"slug":4082,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":4085,"name":4085,"fn":4086,"description":4087,"org":4088,"tags":4089,"stars":23,"repoUrl":24,"updatedAt":4097},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4090,4091,4094],{"name":9,"slug":8,"type":15},{"name":4092,"slug":4093,"type":15},"Quantum Computing","quantum-computing",{"name":4095,"slug":4096,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":4100,"total":4250},[4101,4119,4134,4145,4157,4171,4184,4198,4209,4218,4232,4241],{"slug":4102,"name":4102,"fn":4103,"description":4104,"org":4105,"tags":4106,"stars":4116,"repoUrl":4117,"updatedAt":4118},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4107,4110,4113],{"name":4108,"slug":4109,"type":15},"Documentation","documentation",{"name":4111,"slug":4112,"type":15},"MCP","mcp",{"name":4114,"slug":4115,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":4120,"name":4120,"fn":4121,"description":4122,"org":4123,"tags":4124,"stars":4131,"repoUrl":4132,"updatedAt":4133},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4125,4128,4129],{"name":4126,"slug":4127,"type":15},"Containers","containers",{"name":17,"slug":18,"type":15},{"name":4130,"slug":3056,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":4135,"name":4135,"fn":4136,"description":4137,"org":4138,"tags":4139,"stars":4131,"repoUrl":4132,"updatedAt":4144},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4140,4143],{"name":4141,"slug":4142,"type":15},"CI\u002FCD","ci-cd",{"name":17,"slug":18,"type":15},"2026-07-14T05:25:59.97109",{"slug":4146,"name":4146,"fn":4147,"description":4148,"org":4149,"tags":4150,"stars":4131,"repoUrl":4132,"updatedAt":4156},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4151,4152,4153],{"name":4141,"slug":4142,"type":15},{"name":17,"slug":18,"type":15},{"name":4154,"slug":4155,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":4158,"name":4158,"fn":4159,"description":4160,"org":4161,"tags":4162,"stars":4131,"repoUrl":4132,"updatedAt":4170},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4163,4166,4167],{"name":4164,"slug":4165,"type":15},"Debugging","debugging",{"name":4154,"slug":4155,"type":15},{"name":4168,"slug":4169,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":4172,"name":4172,"fn":4173,"description":4174,"org":4175,"tags":4176,"stars":4131,"repoUrl":4132,"updatedAt":4183},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4177,4180],{"name":4178,"slug":4179,"type":15},"Best Practices","best-practices",{"name":4181,"slug":4182,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":4185,"name":4185,"fn":4186,"description":4187,"org":4188,"tags":4189,"stars":4131,"repoUrl":4132,"updatedAt":4197},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4190,4193,4196],{"name":4191,"slug":4192,"type":15},"Machine Learning","machine-learning",{"name":4194,"slug":4195,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":4199,"name":4199,"fn":4200,"description":4201,"org":4202,"tags":4203,"stars":4131,"repoUrl":4132,"updatedAt":4208},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4204,4207],{"name":4205,"slug":4206,"type":15},"QA","qa",{"name":4049,"slug":4050,"type":15},"2026-07-14T05:25:53.673039",{"slug":4210,"name":4210,"fn":4211,"description":4212,"org":4213,"tags":4214,"stars":4131,"repoUrl":4132,"updatedAt":4217},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4215,4216],{"name":17,"slug":18,"type":15},{"name":4022,"slug":4023,"type":15},"2026-07-14T05:25:49.362534",{"slug":4219,"name":4219,"fn":4220,"description":4221,"org":4222,"tags":4223,"stars":4131,"repoUrl":4132,"updatedAt":4231},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4224,4227,4228],{"name":4225,"slug":4226,"type":15},"Code Review","code-review",{"name":4154,"slug":4155,"type":15},{"name":4229,"slug":4230,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":4233,"name":4233,"fn":4234,"description":4235,"org":4236,"tags":4237,"stars":4131,"repoUrl":4132,"updatedAt":4240},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4238,4239],{"name":4205,"slug":4206,"type":15},{"name":4049,"slug":4050,"type":15},"2026-07-14T05:25:54.928983",{"slug":4242,"name":4242,"fn":4243,"description":4244,"org":4245,"tags":4246,"stars":4131,"repoUrl":4132,"updatedAt":4249},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4247,4248],{"name":4059,"slug":4060,"type":15},{"name":4141,"slug":4142,"type":15},"2026-07-30T05:29:03.275638",496]