[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-mcore-build-and-dependency":3,"mdc-hmkoc7-key":36,"related-org-nvidia-mcore-build-and-dependency":2570,"related-repo-nvidia-mcore-build-and-dependency":2719},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":22,"repoUrl":23,"updatedAt":24,"license":25,"forks":26,"topics":27,"repo":31,"sourceUrl":34,"mdContent":35},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19],{"name":13,"slug":14,"type":15},"Containers","containers","tag",{"name":17,"slug":18,"type":15},"Deployment","deployment",{"name":20,"slug":21,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662","Apache-2.0",4230,[28,29,30],"large-language-models","model-para","transformers",{"repoUrl":23,"stars":22,"forks":26,"topics":32,"description":33},[28,29,30],"Ongoing research training transformer models at scale","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM\u002Ftree\u002FHEAD\u002Fskills\u002Fmcore-build-and-dependency","---\nname: mcore-build-and-dependency\ndescription: Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.\nlicense: Apache-2.0\nwhen_to_use: Adding, removing, or updating a dependency; editing pyproject.toml or uv.lock; uv.lock merge conflict; setting up a dev environment; pulling or building the CI container; container build errors; uv errors; 'how do I install', 'uv sync fails', 'ModuleNotFoundError'.\nmetadata:\n  author: Oliver Koenig \u003Cokoenig@nvidia.com>\n---\n\n# Build & Dependency Guide\n\nThe core principle: **build and develop inside containers** — the CI container\nships the correct CUDA toolkit, PyTorch build, and pre-compiled native extensions\n(TransformerEngine, DeepEP, …) that cannot be reproduced on a bare host.\n\n## Answer-First Constants\n\nFor text-only dependency or container questions, give these repo-specific facts\nup front before the longer workflow:\n\n- Run dependency work inside the Megatron-LM CI container, not on the host.\n- The container venv is `\u002Fopt\u002Fvenv`, already on `PATH`.\n- Default `dev` uses `docker\u002F.ngc_version.dev` and the `dev` uv group; `lts`\n  uses `docker\u002F.ngc_version.lts` and the `lts` uv group. The `container::lts`\n  PR label selects the LTS path; otherwise CI uses `dev`.\n- **`lts` is opt-in only when the user explicitly asks for it.** It is the older\n  long-term-support base, not a routine second lane — never attach\n  `container::lts`, build the LTS image, or run the `lts` uv group on your own\n  initiative, not even for a container or dependency change.\n- Install commands inside the container: `uv sync --locked --group dev --group test`,\n  `uv sync --locked --only-group linting`, or\n  `uv sync --locked --group lts --group test`.\n- Dependency edits use `uv add \u003Cpackage>` followed by `uv lock`, both inside\n  the container.\n- `docker\u002FDockerfile.ci.dev` has `main` and `jet` stages. The `jet` stage needs\n  an internal secret; local\u002Fpublic builds should pass `--target main`.\n\n---\n\n## Why Containers\n\nMegatron-LM depends on CUDA, NCCL, PyTorch with GPU support, TransformerEngine,\nand optional components like ModelOpt and DeepEP. Installing these on a bare host\nis fragile and hard to reproduce. The project ships Dockerfiles that pin every\ndependency.\n\n**Use the container as your development environment.** This guarantees:\n\n- Identical CUDA \u002F NCCL \u002F cuDNN versions across all developers and CI.\n- `uv.lock` resolves the same way locally and in CI.\n- GPU-dependent operations (training, testing) work out of the box.\n\n---\n\n## dev vs lts\n\nTwo image variants exist, each with its own Dockerfile, selected by the\n`container::lts` PR label. The defining difference is the **base container**:\n`dev` tracks the latest NGC PyTorch release, while `lts` (\"long-term support\")\npins the previous, still-supported NGC PyTorch\u002FCUDA release. `container::lts`\nexists to verify a change still works on that older base — the dependency\ndifferences below follow from it, they are not the point.\n\n| Variant | Base image pin | Dockerfile | Where deps live | When used |\n|---------|---------------|------------|-----------------|-----------|\n| **`dev`** | `docker\u002F.ngc_version.dev` (latest NGC release) | `docker\u002FDockerfile.ci.dev` | `pyproject.toml` `dev` extra (uv-resolved) | Default — CI, local development, most PRs |\n| **`lts`** | `docker\u002F.ngc_version.lts` (older long-term-support release) | `docker\u002FDockerfile.ci.lts` | `docker\u002Flts\u002Frequirements.txt` (pinned, sourced from main's `uv.lock` at AUT-479) | Backward-compat lane — verify the change still runs on the older NGC base; extras not carried on it (ModelOpt, the CUDA-13 TransformerEngine build) are dropped |\n\n> LTS deps used to live in `[project.optional-dependencies].lts` in\n> `pyproject.toml`. They were moved into `docker\u002Flts\u002Frequirements.txt` so\n> `pyproject.toml` can host meaningful module-level extras without colliding\n> with the LTS pin set. To bump an LTS dependency, edit the version in\n> `docker\u002Flts\u002Frequirements.txt` and rebuild `docker\u002FDockerfile.ci.lts`.\n\n**Use `dev` for everything. `lts` is off-limits unless the user explicitly asks\nfor it.** CI runs `dev` by default, and that is the only variant you touch on\nyour own initiative. Treat `container::lts` as a high barrier, not a fallback: do\n**not** attach the label, build `docker\u002FDockerfile.ci.lts`, or run the `lts` uv\ngroup unless the user has explicitly requested LTS validation — not even for a\ncontainer or dependency change. When they do ask, `container::lts` verifies the\nchange still works on the older long-term-support PyTorch\u002FCUDA base that LTS\nusers run. The `@pytest.mark.flaky_in_dev` marker skips tests in the `dev`\nenvironment; `@pytest.mark.flaky` skips them in `lts`.\n\n---\n\n## Step 1 — Acquire an Image\n\n**Option A — NVIDIA-internal: pull a CI-built image**\n\n> ⚠️ Requires access to the internal GitLab instance.\n> See @tools\u002Ftrigger_internal_ci.md for setup (adding the git remote, obtaining a token).\n\nThe internal GitLab CI publishes images to its container registry.\nDerive the registry host from your configured `gitlab` remote — the same\nhost you use for `trigger_internal_ci.py`:\n\n```bash\n# Derive host from your 'gitlab' remote:\nGITLAB_HOST=$(git remote get-url gitlab | sed 's\u002F.*@\\(.*\\):.*\u002F\\1\u002F')\n\ndocker pull ${GITLAB_HOST}\u002Fadlr\u002Fmegatron-lm\u002Fmcore_ci_dev:main\n```\n\n**Option B — Build from scratch (works for everyone)**\n\n> ⚠️ `Dockerfile.ci.dev` has two stages: `main` and `jet`. The `jet` stage\n> requires an internal build secret and will fail without it. Always pass\n> `--target main` to stop at the public stage.\n\n```bash\n# dev image (default)\ndocker build \\\n  --target main \\\n  --build-arg FROM_IMAGE_NAME=$(cat docker\u002F.ngc_version.dev) \\\n  --build-arg IMAGE_TYPE=dev \\\n  -f docker\u002FDockerfile.ci.dev \\\n  -t megatron-lm:local .\n\n# lts image (uses a dedicated Dockerfile; no IMAGE_TYPE arg)\ndocker build \\\n  --target main \\\n  --build-arg FROM_IMAGE_NAME=$(cat docker\u002F.ngc_version.lts) \\\n  -f docker\u002FDockerfile.ci.lts \\\n  -t megatron-lm:local-lts .\n```\n\nWhich image variant is used is controlled by the PR label `container::lts`;\nabsent that label, `dev` is used.\n\n---\n\n## Step 2 — Launch the Container\n\n**Option A — Local Docker runtime**\n\n```bash\ndocker run --rm --gpus all \\\n  -v $(pwd):\u002Fworkspace \\\n  -w \u002Fworkspace \\\n  megatron-lm:local \\\n  bash -c \"\u003Cyour command>\"\n```\n\n**Option B — Slurm cluster (for those without a local Docker runtime)**\n\nNVIDIA clusters typically use [Pyxis](https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fpyxis) +\n[enroot](https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fenroot). Request an interactive session:\n\n```bash\nsrun \\\n  --nodes=1 --gpus-per-node=8 \\\n  --container-image megatron-lm:local \\\n  --container-mounts $(pwd):\u002Fworkspace \\\n  --container-workdir \u002Fworkspace \\\n  --pty bash\n```\n\nFor clusters that require a `.sqsh` archive first:\n\n```bash\nenroot import -o megatron-lm.sqsh dockerd:\u002F\u002Fmegatron-lm:local\nsrun \\\n  --nodes=1 --gpus-per-node=8 \\\n  --container-image $(pwd)\u002Fmegatron-lm.sqsh \\\n  --container-mounts $(pwd):\u002Fworkspace \\\n  --container-workdir \u002Fworkspace \\\n  --pty bash\n```\n\n---\n\n## Dependency Management\n\nDependencies are declared in `pyproject.toml`. The venv lives at `\u002Fopt\u002Fvenv`\ninside the container (already on `PATH`).\n\n> **All `uv` operations must be run inside the container.**\n> Never run `uv sync` \u002F `uv pip install` on the host.\n\n### uv Dependency Groups\n\n| Group | Purpose |\n|-------|---------|\n| `training` | Runtime training extras |\n| `dev` | Full dev environment (TransformerEngine, ModelOpt, …) |\n| `test` | pytest, coverage, nemo-run |\n| `linting` | ruff, black, isort, pylint |\n| `build` | Cython, pybind11, nvidia-mathdx |\n\n> The previous `lts` extra has been emptied. LTS deps are pinned in\n> `docker\u002Flts\u002Frequirements.txt` rather than `pyproject.toml`. Do not add new\n> packages under `[project.optional-dependencies].lts`.\n\nInstall commands (inside the container):\n\n```bash\n# Full dev + test environment\nuv sync --locked --group dev --group test\n\n# Linting only\nuv sync --locked --only-group linting\n```\n\nThe LTS environment is reproduced by building `docker\u002FDockerfile.ci.lts`\nend-to-end; there is no `uv sync`-only equivalent because the LTS deps no\nlonger live in `pyproject.toml`. The LTS top-level pin set is in\n`docker\u002Flts\u002Frequirements.txt`; bump versions there and rebuild the image.\n\nSeveral dependencies are sourced directly from git (TransformerEngine, nemo-run,\nFlashMLA, Emerging-Optimizers, nvidia-resiliency-ext). The locked `uv.lock` file\npins exact revisions; update it with `uv lock` when changing `pyproject.toml`.\n\n### Adding a New Dependency\n\nFollow this three-step workflow:\n\n1. **Acquire a container image** — see [Step 1](#step-1--acquire-an-image) above.\n2. **Launch the container interactively** — see [Step 2](#step-2--launch-the-container) above.\n3. **Update the lock file inside the container**, then commit it:\n\n   ```bash\n   # Inside the container:\n   uv add \u003Cpackage>          # adds to pyproject.toml and resolves\n   uv lock                   # regenerates uv.lock\n   # Exit the container, then on the host:\n   git add pyproject.toml uv.lock\n   git commit -S -s -m \"build: add \u003Cpackage> dependency\"\n   ```\n\n### Resolving a merge conflict in uv.lock\n\n`uv.lock` is machine-generated; never resolve conflicts manually. Instead:\n\n```bash\ngit checkout origin\u002Fmain -- uv.lock   # take main's version as the base\n# then inside the container:\nuv lock                               # re-resolve on top of your pyproject.toml changes\n```\n\n---\n\n## Common Pitfalls\n\n| Problem | Cause | Fix |\n|---------|-------|-----|\n| `uv sync --locked` fails | Dependency conflict or stale `uv.lock` | Re-run `uv lock` inside the container and commit updated lock |\n| `ModuleNotFoundError` after pip install | pip installed outside the uv-managed venv | Use `uv add` and `uv sync`, never bare `pip install` |\n| `uv: command not found` inside container | Wrong container image | Use the `megatron-lm` image built from `Dockerfile.ci.dev` |\n| `No space left on device` during uv ops | Cache fills container's `\u002Froot\u002F.cache\u002F` | Mount a host cache dir via `-v $HOME\u002F.cache\u002Fuv:\u002Froot\u002F.cache\u002Fuv` |\n| `docker build` fails with secret-related error | `Dockerfile.ci.dev` has a `jet` stage that requires an internal secret | Add `--target main` to stop before the `jet` stage |\n| `access forbidden` when pulling | Registry URL includes an explicit port (e.g. `:5005`) | Use `${GITLAB_HOST}\u002Fadlr\u002F...` with no port — the sed extracts the hostname only |\n",{"data":37,"body":41},{"name":4,"description":6,"license":25,"when_to_use":38,"metadata":39},"Adding, removing, or updating a dependency; editing pyproject.toml or uv.lock; uv.lock merge conflict; setting up a dev environment; pulling or building the CI container; container build errors; uv errors; 'how do I install', 'uv sync fails', 'ModuleNotFoundError'.",{"author":40},"Oliver Koenig \u003Cokoenig@nvidia.com>",{"type":42,"children":43},"root",[44,53,67,74,79,294,298,304,309,319,343,346,352,392,546,597,692,695,701,709,717,738,872,880,923,1179,1198,1201,1207,1215,1342,1350,1375,1484,1497,1638,1641,1647,1673,1710,1717,1823,1858,1863,1954,1987,2012,2018,2023,2209,2215,2225,2288,2291,2297,2564],{"type":45,"tag":46,"props":47,"children":49},"element","h1",{"id":48},"build-dependency-guide",[50],{"type":51,"value":52},"text","Build & Dependency Guide",{"type":45,"tag":54,"props":55,"children":56},"p",{},[57,59,65],{"type":51,"value":58},"The core principle: ",{"type":45,"tag":60,"props":61,"children":62},"strong",{},[63],{"type":51,"value":64},"build and develop inside containers",{"type":51,"value":66}," — the CI container\nships the correct CUDA toolkit, PyTorch build, and pre-compiled native extensions\n(TransformerEngine, DeepEP, …) that cannot be reproduced on a bare host.",{"type":45,"tag":68,"props":69,"children":71},"h2",{"id":70},"answer-first-constants",[72],{"type":51,"value":73},"Answer-First Constants",{"type":45,"tag":54,"props":75,"children":76},{},[77],{"type":51,"value":78},"For text-only dependency or container questions, give these repo-specific facts\nup front before the longer workflow:",{"type":45,"tag":80,"props":81,"children":82},"ul",{},[83,89,111,175,204,232,253],{"type":45,"tag":84,"props":85,"children":86},"li",{},[87],{"type":51,"value":88},"Run dependency work inside the Megatron-LM CI container, not on the host.",{"type":45,"tag":84,"props":90,"children":91},{},[92,94,101,103,109],{"type":51,"value":93},"The container venv is ",{"type":45,"tag":95,"props":96,"children":98},"code",{"className":97},[],[99],{"type":51,"value":100},"\u002Fopt\u002Fvenv",{"type":51,"value":102},", already on ",{"type":45,"tag":95,"props":104,"children":106},{"className":105},[],[107],{"type":51,"value":108},"PATH",{"type":51,"value":110},".",{"type":45,"tag":84,"props":112,"children":113},{},[114,116,122,124,130,132,137,139,145,147,153,154,159,161,167,169,174],{"type":51,"value":115},"Default ",{"type":45,"tag":95,"props":117,"children":119},{"className":118},[],[120],{"type":51,"value":121},"dev",{"type":51,"value":123}," uses ",{"type":45,"tag":95,"props":125,"children":127},{"className":126},[],[128],{"type":51,"value":129},"docker\u002F.ngc_version.dev",{"type":51,"value":131}," and the ",{"type":45,"tag":95,"props":133,"children":135},{"className":134},[],[136],{"type":51,"value":121},{"type":51,"value":138}," uv group; ",{"type":45,"tag":95,"props":140,"children":142},{"className":141},[],[143],{"type":51,"value":144},"lts",{"type":51,"value":146},"\nuses ",{"type":45,"tag":95,"props":148,"children":150},{"className":149},[],[151],{"type":51,"value":152},"docker\u002F.ngc_version.lts",{"type":51,"value":131},{"type":45,"tag":95,"props":155,"children":157},{"className":156},[],[158],{"type":51,"value":144},{"type":51,"value":160}," uv group. The ",{"type":45,"tag":95,"props":162,"children":164},{"className":163},[],[165],{"type":51,"value":166},"container::lts",{"type":51,"value":168},"\nPR label selects the LTS path; otherwise CI uses ",{"type":45,"tag":95,"props":170,"children":172},{"className":171},[],[173],{"type":51,"value":121},{"type":51,"value":110},{"type":45,"tag":84,"props":176,"children":177},{},[178,188,190,195,197,202],{"type":45,"tag":60,"props":179,"children":180},{},[181,186],{"type":45,"tag":95,"props":182,"children":184},{"className":183},[],[185],{"type":51,"value":144},{"type":51,"value":187}," is opt-in only when the user explicitly asks for it.",{"type":51,"value":189}," It is the older\nlong-term-support base, not a routine second lane — never attach\n",{"type":45,"tag":95,"props":191,"children":193},{"className":192},[],[194],{"type":51,"value":166},{"type":51,"value":196},", build the LTS image, or run the ",{"type":45,"tag":95,"props":198,"children":200},{"className":199},[],[201],{"type":51,"value":144},{"type":51,"value":203}," uv group on your own\ninitiative, not even for a container or dependency change.",{"type":45,"tag":84,"props":205,"children":206},{},[207,209,215,217,223,225,231],{"type":51,"value":208},"Install commands inside the container: ",{"type":45,"tag":95,"props":210,"children":212},{"className":211},[],[213],{"type":51,"value":214},"uv sync --locked --group dev --group test",{"type":51,"value":216},",\n",{"type":45,"tag":95,"props":218,"children":220},{"className":219},[],[221],{"type":51,"value":222},"uv sync --locked --only-group linting",{"type":51,"value":224},", or\n",{"type":45,"tag":95,"props":226,"children":228},{"className":227},[],[229],{"type":51,"value":230},"uv sync --locked --group lts --group test",{"type":51,"value":110},{"type":45,"tag":84,"props":233,"children":234},{},[235,237,243,245,251],{"type":51,"value":236},"Dependency edits use ",{"type":45,"tag":95,"props":238,"children":240},{"className":239},[],[241],{"type":51,"value":242},"uv add \u003Cpackage>",{"type":51,"value":244}," followed by ",{"type":45,"tag":95,"props":246,"children":248},{"className":247},[],[249],{"type":51,"value":250},"uv lock",{"type":51,"value":252},", both inside\nthe container.",{"type":45,"tag":84,"props":254,"children":255},{},[256,262,264,270,272,278,280,285,287,293],{"type":45,"tag":95,"props":257,"children":259},{"className":258},[],[260],{"type":51,"value":261},"docker\u002FDockerfile.ci.dev",{"type":51,"value":263}," has ",{"type":45,"tag":95,"props":265,"children":267},{"className":266},[],[268],{"type":51,"value":269},"main",{"type":51,"value":271}," and ",{"type":45,"tag":95,"props":273,"children":275},{"className":274},[],[276],{"type":51,"value":277},"jet",{"type":51,"value":279}," stages. The ",{"type":45,"tag":95,"props":281,"children":283},{"className":282},[],[284],{"type":51,"value":277},{"type":51,"value":286}," stage needs\nan internal secret; local\u002Fpublic builds should pass ",{"type":45,"tag":95,"props":288,"children":290},{"className":289},[],[291],{"type":51,"value":292},"--target main",{"type":51,"value":110},{"type":45,"tag":295,"props":296,"children":297},"hr",{},[],{"type":45,"tag":68,"props":299,"children":301},{"id":300},"why-containers",[302],{"type":51,"value":303},"Why Containers",{"type":45,"tag":54,"props":305,"children":306},{},[307],{"type":51,"value":308},"Megatron-LM depends on CUDA, NCCL, PyTorch with GPU support, TransformerEngine,\nand optional components like ModelOpt and DeepEP. Installing these on a bare host\nis fragile and hard to reproduce. The project ships Dockerfiles that pin every\ndependency.",{"type":45,"tag":54,"props":310,"children":311},{},[312,317],{"type":45,"tag":60,"props":313,"children":314},{},[315],{"type":51,"value":316},"Use the container as your development environment.",{"type":51,"value":318}," This guarantees:",{"type":45,"tag":80,"props":320,"children":321},{},[322,327,338],{"type":45,"tag":84,"props":323,"children":324},{},[325],{"type":51,"value":326},"Identical CUDA \u002F NCCL \u002F cuDNN versions across all developers and CI.",{"type":45,"tag":84,"props":328,"children":329},{},[330,336],{"type":45,"tag":95,"props":331,"children":333},{"className":332},[],[334],{"type":51,"value":335},"uv.lock",{"type":51,"value":337}," resolves the same way locally and in CI.",{"type":45,"tag":84,"props":339,"children":340},{},[341],{"type":51,"value":342},"GPU-dependent operations (training, testing) work out of the box.",{"type":45,"tag":295,"props":344,"children":345},{},[],{"type":45,"tag":68,"props":347,"children":349},{"id":348},"dev-vs-lts",[350],{"type":51,"value":351},"dev vs lts",{"type":45,"tag":54,"props":353,"children":354},{},[355,357,362,364,369,371,376,378,383,385,390],{"type":51,"value":356},"Two image variants exist, each with its own Dockerfile, selected by the\n",{"type":45,"tag":95,"props":358,"children":360},{"className":359},[],[361],{"type":51,"value":166},{"type":51,"value":363}," PR label. The defining difference is the ",{"type":45,"tag":60,"props":365,"children":366},{},[367],{"type":51,"value":368},"base container",{"type":51,"value":370},":\n",{"type":45,"tag":95,"props":372,"children":374},{"className":373},[],[375],{"type":51,"value":121},{"type":51,"value":377}," tracks the latest NGC PyTorch release, while ",{"type":45,"tag":95,"props":379,"children":381},{"className":380},[],[382],{"type":51,"value":144},{"type":51,"value":384}," (\"long-term support\")\npins the previous, still-supported NGC PyTorch\u002FCUDA release. ",{"type":45,"tag":95,"props":386,"children":388},{"className":387},[],[389],{"type":51,"value":166},{"type":51,"value":391},"\nexists to verify a change still works on that older base — the dependency\ndifferences below follow from it, they are not the point.",{"type":45,"tag":393,"props":394,"children":395},"table",{},[396,430],{"type":45,"tag":397,"props":398,"children":399},"thead",{},[400],{"type":45,"tag":401,"props":402,"children":403},"tr",{},[404,410,415,420,425],{"type":45,"tag":405,"props":406,"children":407},"th",{},[408],{"type":51,"value":409},"Variant",{"type":45,"tag":405,"props":411,"children":412},{},[413],{"type":51,"value":414},"Base image pin",{"type":45,"tag":405,"props":416,"children":417},{},[418],{"type":51,"value":419},"Dockerfile",{"type":45,"tag":405,"props":421,"children":422},{},[423],{"type":51,"value":424},"Where deps live",{"type":45,"tag":405,"props":426,"children":427},{},[428],{"type":51,"value":429},"When used",{"type":45,"tag":431,"props":432,"children":433},"tbody",{},[434,490],{"type":45,"tag":401,"props":435,"children":436},{},[437,449,459,467,485],{"type":45,"tag":438,"props":439,"children":440},"td",{},[441],{"type":45,"tag":60,"props":442,"children":443},{},[444],{"type":45,"tag":95,"props":445,"children":447},{"className":446},[],[448],{"type":51,"value":121},{"type":45,"tag":438,"props":450,"children":451},{},[452,457],{"type":45,"tag":95,"props":453,"children":455},{"className":454},[],[456],{"type":51,"value":129},{"type":51,"value":458}," (latest NGC release)",{"type":45,"tag":438,"props":460,"children":461},{},[462],{"type":45,"tag":95,"props":463,"children":465},{"className":464},[],[466],{"type":51,"value":261},{"type":45,"tag":438,"props":468,"children":469},{},[470,476,478,483],{"type":45,"tag":95,"props":471,"children":473},{"className":472},[],[474],{"type":51,"value":475},"pyproject.toml",{"type":51,"value":477}," ",{"type":45,"tag":95,"props":479,"children":481},{"className":480},[],[482],{"type":51,"value":121},{"type":51,"value":484}," extra (uv-resolved)",{"type":45,"tag":438,"props":486,"children":487},{},[488],{"type":51,"value":489},"Default — CI, local development, most PRs",{"type":45,"tag":401,"props":491,"children":492},{},[493,504,514,523,541],{"type":45,"tag":438,"props":494,"children":495},{},[496],{"type":45,"tag":60,"props":497,"children":498},{},[499],{"type":45,"tag":95,"props":500,"children":502},{"className":501},[],[503],{"type":51,"value":144},{"type":45,"tag":438,"props":505,"children":506},{},[507,512],{"type":45,"tag":95,"props":508,"children":510},{"className":509},[],[511],{"type":51,"value":152},{"type":51,"value":513}," (older long-term-support release)",{"type":45,"tag":438,"props":515,"children":516},{},[517],{"type":45,"tag":95,"props":518,"children":520},{"className":519},[],[521],{"type":51,"value":522},"docker\u002FDockerfile.ci.lts",{"type":45,"tag":438,"props":524,"children":525},{},[526,532,534,539],{"type":45,"tag":95,"props":527,"children":529},{"className":528},[],[530],{"type":51,"value":531},"docker\u002Flts\u002Frequirements.txt",{"type":51,"value":533}," (pinned, sourced from main's ",{"type":45,"tag":95,"props":535,"children":537},{"className":536},[],[538],{"type":51,"value":335},{"type":51,"value":540}," at AUT-479)",{"type":45,"tag":438,"props":542,"children":543},{},[544],{"type":51,"value":545},"Backward-compat lane — verify the change still runs on the older NGC base; extras not carried on it (ModelOpt, the CUDA-13 TransformerEngine build) are dropped",{"type":45,"tag":547,"props":548,"children":549},"blockquote",{},[550],{"type":45,"tag":54,"props":551,"children":552},{},[553,555,561,563,568,570,575,577,582,584,589,591,596],{"type":51,"value":554},"LTS deps used to live in ",{"type":45,"tag":95,"props":556,"children":558},{"className":557},[],[559],{"type":51,"value":560},"[project.optional-dependencies].lts",{"type":51,"value":562}," in\n",{"type":45,"tag":95,"props":564,"children":566},{"className":565},[],[567],{"type":51,"value":475},{"type":51,"value":569},". They were moved into ",{"type":45,"tag":95,"props":571,"children":573},{"className":572},[],[574],{"type":51,"value":531},{"type":51,"value":576}," so\n",{"type":45,"tag":95,"props":578,"children":580},{"className":579},[],[581],{"type":51,"value":475},{"type":51,"value":583}," can host meaningful module-level extras without colliding\nwith the LTS pin set. To bump an LTS dependency, edit the version in\n",{"type":45,"tag":95,"props":585,"children":587},{"className":586},[],[588],{"type":51,"value":531},{"type":51,"value":590}," and rebuild ",{"type":45,"tag":95,"props":592,"children":594},{"className":593},[],[595],{"type":51,"value":522},{"type":51,"value":110},{"type":45,"tag":54,"props":598,"children":599},{},[600,619,621,626,628,633,635,640,642,647,649,654,656,661,663,669,671,676,678,684,686,691],{"type":45,"tag":60,"props":601,"children":602},{},[603,605,610,612,617],{"type":51,"value":604},"Use ",{"type":45,"tag":95,"props":606,"children":608},{"className":607},[],[609],{"type":51,"value":121},{"type":51,"value":611}," for everything. ",{"type":45,"tag":95,"props":613,"children":615},{"className":614},[],[616],{"type":51,"value":144},{"type":51,"value":618}," is off-limits unless the user explicitly asks\nfor it.",{"type":51,"value":620}," CI runs ",{"type":45,"tag":95,"props":622,"children":624},{"className":623},[],[625],{"type":51,"value":121},{"type":51,"value":627}," by default, and that is the only variant you touch on\nyour own initiative. Treat ",{"type":45,"tag":95,"props":629,"children":631},{"className":630},[],[632],{"type":51,"value":166},{"type":51,"value":634}," as a high barrier, not a fallback: do\n",{"type":45,"tag":60,"props":636,"children":637},{},[638],{"type":51,"value":639},"not",{"type":51,"value":641}," attach the label, build ",{"type":45,"tag":95,"props":643,"children":645},{"className":644},[],[646],{"type":51,"value":522},{"type":51,"value":648},", or run the ",{"type":45,"tag":95,"props":650,"children":652},{"className":651},[],[653],{"type":51,"value":144},{"type":51,"value":655}," uv\ngroup unless the user has explicitly requested LTS validation — not even for a\ncontainer or dependency change. When they do ask, ",{"type":45,"tag":95,"props":657,"children":659},{"className":658},[],[660],{"type":51,"value":166},{"type":51,"value":662}," verifies the\nchange still works on the older long-term-support PyTorch\u002FCUDA base that LTS\nusers run. The ",{"type":45,"tag":95,"props":664,"children":666},{"className":665},[],[667],{"type":51,"value":668},"@pytest.mark.flaky_in_dev",{"type":51,"value":670}," marker skips tests in the ",{"type":45,"tag":95,"props":672,"children":674},{"className":673},[],[675],{"type":51,"value":121},{"type":51,"value":677},"\nenvironment; ",{"type":45,"tag":95,"props":679,"children":681},{"className":680},[],[682],{"type":51,"value":683},"@pytest.mark.flaky",{"type":51,"value":685}," skips them in ",{"type":45,"tag":95,"props":687,"children":689},{"className":688},[],[690],{"type":51,"value":144},{"type":51,"value":110},{"type":45,"tag":295,"props":693,"children":694},{},[],{"type":45,"tag":68,"props":696,"children":698},{"id":697},"step-1-acquire-an-image",[699],{"type":51,"value":700},"Step 1 — Acquire an Image",{"type":45,"tag":54,"props":702,"children":703},{},[704],{"type":45,"tag":60,"props":705,"children":706},{},[707],{"type":51,"value":708},"Option A — NVIDIA-internal: pull a CI-built image",{"type":45,"tag":547,"props":710,"children":711},{},[712],{"type":45,"tag":54,"props":713,"children":714},{},[715],{"type":51,"value":716},"⚠️ Requires access to the internal GitLab instance.\nSee @tools\u002Ftrigger_internal_ci.md for setup (adding the git remote, obtaining a token).",{"type":45,"tag":54,"props":718,"children":719},{},[720,722,728,730,736],{"type":51,"value":721},"The internal GitLab CI publishes images to its container registry.\nDerive the registry host from your configured ",{"type":45,"tag":95,"props":723,"children":725},{"className":724},[],[726],{"type":51,"value":727},"gitlab",{"type":51,"value":729}," remote — the same\nhost you use for ",{"type":45,"tag":95,"props":731,"children":733},{"className":732},[],[734],{"type":51,"value":735},"trigger_internal_ci.py",{"type":51,"value":737},":",{"type":45,"tag":739,"props":740,"children":745},"pre",{"className":741,"code":742,"language":743,"meta":744,"style":744},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Derive host from your 'gitlab' remote:\nGITLAB_HOST=$(git remote get-url gitlab | sed 's\u002F.*@\\(.*\\):.*\u002F\\1\u002F')\n\ndocker pull ${GITLAB_HOST}\u002Fadlr\u002Fmegatron-lm\u002Fmcore_ci_dev:main\n","bash","",[746],{"type":45,"tag":95,"props":747,"children":748},{"__ignoreMap":744},[749,761,829,839],{"type":45,"tag":750,"props":751,"children":754},"span",{"class":752,"line":753},"line",1,[755],{"type":45,"tag":750,"props":756,"children":758},{"style":757},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[759],{"type":51,"value":760},"# Derive host from your 'gitlab' remote:\n",{"type":45,"tag":750,"props":762,"children":764},{"class":752,"line":763},2,[765,771,777,783,789,794,799,804,809,814,819,824],{"type":45,"tag":750,"props":766,"children":768},{"style":767},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[769],{"type":51,"value":770},"GITLAB_HOST",{"type":45,"tag":750,"props":772,"children":774},{"style":773},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[775],{"type":51,"value":776},"=$(",{"type":45,"tag":750,"props":778,"children":780},{"style":779},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[781],{"type":51,"value":782},"git",{"type":45,"tag":750,"props":784,"children":786},{"style":785},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[787],{"type":51,"value":788}," remote",{"type":45,"tag":750,"props":790,"children":791},{"style":785},[792],{"type":51,"value":793}," get-url",{"type":45,"tag":750,"props":795,"children":796},{"style":785},[797],{"type":51,"value":798}," gitlab",{"type":45,"tag":750,"props":800,"children":801},{"style":773},[802],{"type":51,"value":803}," |",{"type":45,"tag":750,"props":805,"children":806},{"style":779},[807],{"type":51,"value":808}," sed",{"type":45,"tag":750,"props":810,"children":811},{"style":773},[812],{"type":51,"value":813}," '",{"type":45,"tag":750,"props":815,"children":816},{"style":785},[817],{"type":51,"value":818},"s\u002F.*@\\(.*\\):.*\u002F\\1\u002F",{"type":45,"tag":750,"props":820,"children":821},{"style":773},[822],{"type":51,"value":823},"'",{"type":45,"tag":750,"props":825,"children":826},{"style":773},[827],{"type":51,"value":828},")\n",{"type":45,"tag":750,"props":830,"children":832},{"class":752,"line":831},3,[833],{"type":45,"tag":750,"props":834,"children":836},{"emptyLinePlaceholder":835},true,[837],{"type":51,"value":838},"\n",{"type":45,"tag":750,"props":840,"children":842},{"class":752,"line":841},4,[843,848,853,858,862,867],{"type":45,"tag":750,"props":844,"children":845},{"style":779},[846],{"type":51,"value":847},"docker",{"type":45,"tag":750,"props":849,"children":850},{"style":785},[851],{"type":51,"value":852}," pull",{"type":45,"tag":750,"props":854,"children":855},{"style":773},[856],{"type":51,"value":857}," ${",{"type":45,"tag":750,"props":859,"children":860},{"style":767},[861],{"type":51,"value":770},{"type":45,"tag":750,"props":863,"children":864},{"style":773},[865],{"type":51,"value":866},"}",{"type":45,"tag":750,"props":868,"children":869},{"style":785},[870],{"type":51,"value":871},"\u002Fadlr\u002Fmegatron-lm\u002Fmcore_ci_dev:main\n",{"type":45,"tag":54,"props":873,"children":874},{},[875],{"type":45,"tag":60,"props":876,"children":877},{},[878],{"type":51,"value":879},"Option B — Build from scratch (works for everyone)",{"type":45,"tag":547,"props":881,"children":882},{},[883],{"type":45,"tag":54,"props":884,"children":885},{},[886,888,894,896,901,902,907,909,914,916,921],{"type":51,"value":887},"⚠️ ",{"type":45,"tag":95,"props":889,"children":891},{"className":890},[],[892],{"type":51,"value":893},"Dockerfile.ci.dev",{"type":51,"value":895}," has two stages: ",{"type":45,"tag":95,"props":897,"children":899},{"className":898},[],[900],{"type":51,"value":269},{"type":51,"value":271},{"type":45,"tag":95,"props":903,"children":905},{"className":904},[],[906],{"type":51,"value":277},{"type":51,"value":908},". The ",{"type":45,"tag":95,"props":910,"children":912},{"className":911},[],[913],{"type":51,"value":277},{"type":51,"value":915}," stage\nrequires an internal build secret and will fail without it. Always pass\n",{"type":45,"tag":95,"props":917,"children":919},{"className":918},[],[920],{"type":51,"value":292},{"type":51,"value":922}," to stop at the public stage.",{"type":45,"tag":739,"props":924,"children":926},{"className":741,"code":925,"language":743,"meta":744,"style":744},"# dev image (default)\ndocker build \\\n  --target main \\\n  --build-arg FROM_IMAGE_NAME=$(cat docker\u002F.ngc_version.dev) \\\n  --build-arg IMAGE_TYPE=dev \\\n  -f docker\u002FDockerfile.ci.dev \\\n  -t megatron-lm:local .\n\n# lts image (uses a dedicated Dockerfile; no IMAGE_TYPE arg)\ndocker build \\\n  --target main \\\n  --build-arg FROM_IMAGE_NAME=$(cat docker\u002F.ngc_version.lts) \\\n  -f docker\u002FDockerfile.ci.lts \\\n  -t megatron-lm:local-lts .\n",[927],{"type":45,"tag":95,"props":928,"children":929},{"__ignoreMap":744},[930,938,955,972,1009,1026,1044,1063,1071,1080,1096,1112,1145,1162],{"type":45,"tag":750,"props":931,"children":932},{"class":752,"line":753},[933],{"type":45,"tag":750,"props":934,"children":935},{"style":757},[936],{"type":51,"value":937},"# dev image (default)\n",{"type":45,"tag":750,"props":939,"children":940},{"class":752,"line":763},[941,945,950],{"type":45,"tag":750,"props":942,"children":943},{"style":779},[944],{"type":51,"value":847},{"type":45,"tag":750,"props":946,"children":947},{"style":785},[948],{"type":51,"value":949}," build",{"type":45,"tag":750,"props":951,"children":952},{"style":767},[953],{"type":51,"value":954}," \\\n",{"type":45,"tag":750,"props":956,"children":957},{"class":752,"line":831},[958,963,968],{"type":45,"tag":750,"props":959,"children":960},{"style":785},[961],{"type":51,"value":962},"  --target",{"type":45,"tag":750,"props":964,"children":965},{"style":785},[966],{"type":51,"value":967}," main",{"type":45,"tag":750,"props":969,"children":970},{"style":767},[971],{"type":51,"value":954},{"type":45,"tag":750,"props":973,"children":974},{"class":752,"line":841},[975,980,985,990,995,1000,1005],{"type":45,"tag":750,"props":976,"children":977},{"style":785},[978],{"type":51,"value":979},"  --build-arg",{"type":45,"tag":750,"props":981,"children":982},{"style":785},[983],{"type":51,"value":984}," FROM_IMAGE_NAME=",{"type":45,"tag":750,"props":986,"children":987},{"style":773},[988],{"type":51,"value":989},"$(",{"type":45,"tag":750,"props":991,"children":992},{"style":779},[993],{"type":51,"value":994},"cat",{"type":45,"tag":750,"props":996,"children":997},{"style":785},[998],{"type":51,"value":999}," docker\u002F.ngc_version.dev",{"type":45,"tag":750,"props":1001,"children":1002},{"style":773},[1003],{"type":51,"value":1004},")",{"type":45,"tag":750,"props":1006,"children":1007},{"style":767},[1008],{"type":51,"value":954},{"type":45,"tag":750,"props":1010,"children":1012},{"class":752,"line":1011},5,[1013,1017,1022],{"type":45,"tag":750,"props":1014,"children":1015},{"style":785},[1016],{"type":51,"value":979},{"type":45,"tag":750,"props":1018,"children":1019},{"style":785},[1020],{"type":51,"value":1021}," IMAGE_TYPE=dev",{"type":45,"tag":750,"props":1023,"children":1024},{"style":767},[1025],{"type":51,"value":954},{"type":45,"tag":750,"props":1027,"children":1029},{"class":752,"line":1028},6,[1030,1035,1040],{"type":45,"tag":750,"props":1031,"children":1032},{"style":785},[1033],{"type":51,"value":1034},"  -f",{"type":45,"tag":750,"props":1036,"children":1037},{"style":785},[1038],{"type":51,"value":1039}," docker\u002FDockerfile.ci.dev",{"type":45,"tag":750,"props":1041,"children":1042},{"style":767},[1043],{"type":51,"value":954},{"type":45,"tag":750,"props":1045,"children":1047},{"class":752,"line":1046},7,[1048,1053,1058],{"type":45,"tag":750,"props":1049,"children":1050},{"style":785},[1051],{"type":51,"value":1052},"  -t",{"type":45,"tag":750,"props":1054,"children":1055},{"style":785},[1056],{"type":51,"value":1057}," megatron-lm:local",{"type":45,"tag":750,"props":1059,"children":1060},{"style":785},[1061],{"type":51,"value":1062}," .\n",{"type":45,"tag":750,"props":1064,"children":1066},{"class":752,"line":1065},8,[1067],{"type":45,"tag":750,"props":1068,"children":1069},{"emptyLinePlaceholder":835},[1070],{"type":51,"value":838},{"type":45,"tag":750,"props":1072,"children":1074},{"class":752,"line":1073},9,[1075],{"type":45,"tag":750,"props":1076,"children":1077},{"style":757},[1078],{"type":51,"value":1079},"# lts image (uses a dedicated Dockerfile; no IMAGE_TYPE arg)\n",{"type":45,"tag":750,"props":1081,"children":1083},{"class":752,"line":1082},10,[1084,1088,1092],{"type":45,"tag":750,"props":1085,"children":1086},{"style":779},[1087],{"type":51,"value":847},{"type":45,"tag":750,"props":1089,"children":1090},{"style":785},[1091],{"type":51,"value":949},{"type":45,"tag":750,"props":1093,"children":1094},{"style":767},[1095],{"type":51,"value":954},{"type":45,"tag":750,"props":1097,"children":1099},{"class":752,"line":1098},11,[1100,1104,1108],{"type":45,"tag":750,"props":1101,"children":1102},{"style":785},[1103],{"type":51,"value":962},{"type":45,"tag":750,"props":1105,"children":1106},{"style":785},[1107],{"type":51,"value":967},{"type":45,"tag":750,"props":1109,"children":1110},{"style":767},[1111],{"type":51,"value":954},{"type":45,"tag":750,"props":1113,"children":1115},{"class":752,"line":1114},12,[1116,1120,1124,1128,1132,1137,1141],{"type":45,"tag":750,"props":1117,"children":1118},{"style":785},[1119],{"type":51,"value":979},{"type":45,"tag":750,"props":1121,"children":1122},{"style":785},[1123],{"type":51,"value":984},{"type":45,"tag":750,"props":1125,"children":1126},{"style":773},[1127],{"type":51,"value":989},{"type":45,"tag":750,"props":1129,"children":1130},{"style":779},[1131],{"type":51,"value":994},{"type":45,"tag":750,"props":1133,"children":1134},{"style":785},[1135],{"type":51,"value":1136}," docker\u002F.ngc_version.lts",{"type":45,"tag":750,"props":1138,"children":1139},{"style":773},[1140],{"type":51,"value":1004},{"type":45,"tag":750,"props":1142,"children":1143},{"style":767},[1144],{"type":51,"value":954},{"type":45,"tag":750,"props":1146,"children":1148},{"class":752,"line":1147},13,[1149,1153,1158],{"type":45,"tag":750,"props":1150,"children":1151},{"style":785},[1152],{"type":51,"value":1034},{"type":45,"tag":750,"props":1154,"children":1155},{"style":785},[1156],{"type":51,"value":1157}," docker\u002FDockerfile.ci.lts",{"type":45,"tag":750,"props":1159,"children":1160},{"style":767},[1161],{"type":51,"value":954},{"type":45,"tag":750,"props":1163,"children":1165},{"class":752,"line":1164},14,[1166,1170,1175],{"type":45,"tag":750,"props":1167,"children":1168},{"style":785},[1169],{"type":51,"value":1052},{"type":45,"tag":750,"props":1171,"children":1172},{"style":785},[1173],{"type":51,"value":1174}," megatron-lm:local-lts",{"type":45,"tag":750,"props":1176,"children":1177},{"style":785},[1178],{"type":51,"value":1062},{"type":45,"tag":54,"props":1180,"children":1181},{},[1182,1184,1189,1191,1196],{"type":51,"value":1183},"Which image variant is used is controlled by the PR label ",{"type":45,"tag":95,"props":1185,"children":1187},{"className":1186},[],[1188],{"type":51,"value":166},{"type":51,"value":1190},";\nabsent that label, ",{"type":45,"tag":95,"props":1192,"children":1194},{"className":1193},[],[1195],{"type":51,"value":121},{"type":51,"value":1197}," is used.",{"type":45,"tag":295,"props":1199,"children":1200},{},[],{"type":45,"tag":68,"props":1202,"children":1204},{"id":1203},"step-2-launch-the-container",[1205],{"type":51,"value":1206},"Step 2 — Launch the Container",{"type":45,"tag":54,"props":1208,"children":1209},{},[1210],{"type":45,"tag":60,"props":1211,"children":1212},{},[1213],{"type":51,"value":1214},"Option A — Local Docker runtime",{"type":45,"tag":739,"props":1216,"children":1218},{"className":741,"code":1217,"language":743,"meta":744,"style":744},"docker run --rm --gpus all \\\n  -v $(pwd):\u002Fworkspace \\\n  -w \u002Fworkspace \\\n  megatron-lm:local \\\n  bash -c \"\u003Cyour command>\"\n",[1219],{"type":45,"tag":95,"props":1220,"children":1221},{"__ignoreMap":744},[1222,1253,1285,1302,1314],{"type":45,"tag":750,"props":1223,"children":1224},{"class":752,"line":753},[1225,1229,1234,1239,1244,1249],{"type":45,"tag":750,"props":1226,"children":1227},{"style":779},[1228],{"type":51,"value":847},{"type":45,"tag":750,"props":1230,"children":1231},{"style":785},[1232],{"type":51,"value":1233}," run",{"type":45,"tag":750,"props":1235,"children":1236},{"style":785},[1237],{"type":51,"value":1238}," --rm",{"type":45,"tag":750,"props":1240,"children":1241},{"style":785},[1242],{"type":51,"value":1243}," --gpus",{"type":45,"tag":750,"props":1245,"children":1246},{"style":785},[1247],{"type":51,"value":1248}," all",{"type":45,"tag":750,"props":1250,"children":1251},{"style":767},[1252],{"type":51,"value":954},{"type":45,"tag":750,"props":1254,"children":1255},{"class":752,"line":763},[1256,1261,1266,1272,1276,1281],{"type":45,"tag":750,"props":1257,"children":1258},{"style":785},[1259],{"type":51,"value":1260},"  -v",{"type":45,"tag":750,"props":1262,"children":1263},{"style":773},[1264],{"type":51,"value":1265}," $(",{"type":45,"tag":750,"props":1267,"children":1269},{"style":1268},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[1270],{"type":51,"value":1271},"pwd",{"type":45,"tag":750,"props":1273,"children":1274},{"style":773},[1275],{"type":51,"value":1004},{"type":45,"tag":750,"props":1277,"children":1278},{"style":785},[1279],{"type":51,"value":1280},":\u002Fworkspace",{"type":45,"tag":750,"props":1282,"children":1283},{"style":767},[1284],{"type":51,"value":954},{"type":45,"tag":750,"props":1286,"children":1287},{"class":752,"line":831},[1288,1293,1298],{"type":45,"tag":750,"props":1289,"children":1290},{"style":785},[1291],{"type":51,"value":1292},"  -w",{"type":45,"tag":750,"props":1294,"children":1295},{"style":785},[1296],{"type":51,"value":1297}," \u002Fworkspace",{"type":45,"tag":750,"props":1299,"children":1300},{"style":767},[1301],{"type":51,"value":954},{"type":45,"tag":750,"props":1303,"children":1304},{"class":752,"line":841},[1305,1310],{"type":45,"tag":750,"props":1306,"children":1307},{"style":785},[1308],{"type":51,"value":1309},"  megatron-lm:local",{"type":45,"tag":750,"props":1311,"children":1312},{"style":767},[1313],{"type":51,"value":954},{"type":45,"tag":750,"props":1315,"children":1316},{"class":752,"line":1011},[1317,1322,1327,1332,1337],{"type":45,"tag":750,"props":1318,"children":1319},{"style":785},[1320],{"type":51,"value":1321},"  bash",{"type":45,"tag":750,"props":1323,"children":1324},{"style":785},[1325],{"type":51,"value":1326}," -c",{"type":45,"tag":750,"props":1328,"children":1329},{"style":773},[1330],{"type":51,"value":1331}," \"",{"type":45,"tag":750,"props":1333,"children":1334},{"style":785},[1335],{"type":51,"value":1336},"\u003Cyour command>",{"type":45,"tag":750,"props":1338,"children":1339},{"style":773},[1340],{"type":51,"value":1341},"\"\n",{"type":45,"tag":54,"props":1343,"children":1344},{},[1345],{"type":45,"tag":60,"props":1346,"children":1347},{},[1348],{"type":51,"value":1349},"Option B — Slurm cluster (for those without a local Docker runtime)",{"type":45,"tag":54,"props":1351,"children":1352},{},[1353,1355,1364,1366,1373],{"type":51,"value":1354},"NVIDIA clusters typically use ",{"type":45,"tag":1356,"props":1357,"children":1361},"a",{"href":1358,"rel":1359},"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fpyxis",[1360],"nofollow",[1362],{"type":51,"value":1363},"Pyxis",{"type":51,"value":1365}," +\n",{"type":45,"tag":1356,"props":1367,"children":1370},{"href":1368,"rel":1369},"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fenroot",[1360],[1371],{"type":51,"value":1372},"enroot",{"type":51,"value":1374},". Request an interactive session:",{"type":45,"tag":739,"props":1376,"children":1378},{"className":741,"code":1377,"language":743,"meta":744,"style":744},"srun \\\n  --nodes=1 --gpus-per-node=8 \\\n  --container-image megatron-lm:local \\\n  --container-mounts $(pwd):\u002Fworkspace \\\n  --container-workdir \u002Fworkspace \\\n  --pty bash\n",[1379],{"type":45,"tag":95,"props":1380,"children":1381},{"__ignoreMap":744},[1382,1394,1411,1427,1455,1471],{"type":45,"tag":750,"props":1383,"children":1384},{"class":752,"line":753},[1385,1390],{"type":45,"tag":750,"props":1386,"children":1387},{"style":779},[1388],{"type":51,"value":1389},"srun",{"type":45,"tag":750,"props":1391,"children":1392},{"style":767},[1393],{"type":51,"value":954},{"type":45,"tag":750,"props":1395,"children":1396},{"class":752,"line":763},[1397,1402,1407],{"type":45,"tag":750,"props":1398,"children":1399},{"style":785},[1400],{"type":51,"value":1401},"  --nodes=1",{"type":45,"tag":750,"props":1403,"children":1404},{"style":785},[1405],{"type":51,"value":1406}," --gpus-per-node=8",{"type":45,"tag":750,"props":1408,"children":1409},{"style":767},[1410],{"type":51,"value":954},{"type":45,"tag":750,"props":1412,"children":1413},{"class":752,"line":831},[1414,1419,1423],{"type":45,"tag":750,"props":1415,"children":1416},{"style":785},[1417],{"type":51,"value":1418},"  --container-image",{"type":45,"tag":750,"props":1420,"children":1421},{"style":785},[1422],{"type":51,"value":1057},{"type":45,"tag":750,"props":1424,"children":1425},{"style":767},[1426],{"type":51,"value":954},{"type":45,"tag":750,"props":1428,"children":1429},{"class":752,"line":841},[1430,1435,1439,1443,1447,1451],{"type":45,"tag":750,"props":1431,"children":1432},{"style":785},[1433],{"type":51,"value":1434},"  --container-mounts",{"type":45,"tag":750,"props":1436,"children":1437},{"style":773},[1438],{"type":51,"value":1265},{"type":45,"tag":750,"props":1440,"children":1441},{"style":1268},[1442],{"type":51,"value":1271},{"type":45,"tag":750,"props":1444,"children":1445},{"style":773},[1446],{"type":51,"value":1004},{"type":45,"tag":750,"props":1448,"children":1449},{"style":785},[1450],{"type":51,"value":1280},{"type":45,"tag":750,"props":1452,"children":1453},{"style":767},[1454],{"type":51,"value":954},{"type":45,"tag":750,"props":1456,"children":1457},{"class":752,"line":1011},[1458,1463,1467],{"type":45,"tag":750,"props":1459,"children":1460},{"style":785},[1461],{"type":51,"value":1462},"  --container-workdir",{"type":45,"tag":750,"props":1464,"children":1465},{"style":785},[1466],{"type":51,"value":1297},{"type":45,"tag":750,"props":1468,"children":1469},{"style":767},[1470],{"type":51,"value":954},{"type":45,"tag":750,"props":1472,"children":1473},{"class":752,"line":1028},[1474,1479],{"type":45,"tag":750,"props":1475,"children":1476},{"style":785},[1477],{"type":51,"value":1478},"  --pty",{"type":45,"tag":750,"props":1480,"children":1481},{"style":785},[1482],{"type":51,"value":1483}," bash\n",{"type":45,"tag":54,"props":1485,"children":1486},{},[1487,1489,1495],{"type":51,"value":1488},"For clusters that require a ",{"type":45,"tag":95,"props":1490,"children":1492},{"className":1491},[],[1493],{"type":51,"value":1494},".sqsh",{"type":51,"value":1496}," archive first:",{"type":45,"tag":739,"props":1498,"children":1500},{"className":741,"code":1499,"language":743,"meta":744,"style":744},"enroot import -o megatron-lm.sqsh dockerd:\u002F\u002Fmegatron-lm:local\nsrun \\\n  --nodes=1 --gpus-per-node=8 \\\n  --container-image $(pwd)\u002Fmegatron-lm.sqsh \\\n  --container-mounts $(pwd):\u002Fworkspace \\\n  --container-workdir \u002Fworkspace \\\n  --pty bash\n",[1501],{"type":45,"tag":95,"props":1502,"children":1503},{"__ignoreMap":744},[1504,1531,1542,1557,1585,1612,1627],{"type":45,"tag":750,"props":1505,"children":1506},{"class":752,"line":753},[1507,1511,1516,1521,1526],{"type":45,"tag":750,"props":1508,"children":1509},{"style":779},[1510],{"type":51,"value":1372},{"type":45,"tag":750,"props":1512,"children":1513},{"style":785},[1514],{"type":51,"value":1515}," import",{"type":45,"tag":750,"props":1517,"children":1518},{"style":785},[1519],{"type":51,"value":1520}," -o",{"type":45,"tag":750,"props":1522,"children":1523},{"style":785},[1524],{"type":51,"value":1525}," megatron-lm.sqsh",{"type":45,"tag":750,"props":1527,"children":1528},{"style":785},[1529],{"type":51,"value":1530}," dockerd:\u002F\u002Fmegatron-lm:local\n",{"type":45,"tag":750,"props":1532,"children":1533},{"class":752,"line":763},[1534,1538],{"type":45,"tag":750,"props":1535,"children":1536},{"style":779},[1537],{"type":51,"value":1389},{"type":45,"tag":750,"props":1539,"children":1540},{"style":767},[1541],{"type":51,"value":954},{"type":45,"tag":750,"props":1543,"children":1544},{"class":752,"line":831},[1545,1549,1553],{"type":45,"tag":750,"props":1546,"children":1547},{"style":785},[1548],{"type":51,"value":1401},{"type":45,"tag":750,"props":1550,"children":1551},{"style":785},[1552],{"type":51,"value":1406},{"type":45,"tag":750,"props":1554,"children":1555},{"style":767},[1556],{"type":51,"value":954},{"type":45,"tag":750,"props":1558,"children":1559},{"class":752,"line":841},[1560,1564,1568,1572,1576,1581],{"type":45,"tag":750,"props":1561,"children":1562},{"style":785},[1563],{"type":51,"value":1418},{"type":45,"tag":750,"props":1565,"children":1566},{"style":773},[1567],{"type":51,"value":1265},{"type":45,"tag":750,"props":1569,"children":1570},{"style":1268},[1571],{"type":51,"value":1271},{"type":45,"tag":750,"props":1573,"children":1574},{"style":773},[1575],{"type":51,"value":1004},{"type":45,"tag":750,"props":1577,"children":1578},{"style":785},[1579],{"type":51,"value":1580},"\u002Fmegatron-lm.sqsh",{"type":45,"tag":750,"props":1582,"children":1583},{"style":767},[1584],{"type":51,"value":954},{"type":45,"tag":750,"props":1586,"children":1587},{"class":752,"line":1011},[1588,1592,1596,1600,1604,1608],{"type":45,"tag":750,"props":1589,"children":1590},{"style":785},[1591],{"type":51,"value":1434},{"type":45,"tag":750,"props":1593,"children":1594},{"style":773},[1595],{"type":51,"value":1265},{"type":45,"tag":750,"props":1597,"children":1598},{"style":1268},[1599],{"type":51,"value":1271},{"type":45,"tag":750,"props":1601,"children":1602},{"style":773},[1603],{"type":51,"value":1004},{"type":45,"tag":750,"props":1605,"children":1606},{"style":785},[1607],{"type":51,"value":1280},{"type":45,"tag":750,"props":1609,"children":1610},{"style":767},[1611],{"type":51,"value":954},{"type":45,"tag":750,"props":1613,"children":1614},{"class":752,"line":1028},[1615,1619,1623],{"type":45,"tag":750,"props":1616,"children":1617},{"style":785},[1618],{"type":51,"value":1462},{"type":45,"tag":750,"props":1620,"children":1621},{"style":785},[1622],{"type":51,"value":1297},{"type":45,"tag":750,"props":1624,"children":1625},{"style":767},[1626],{"type":51,"value":954},{"type":45,"tag":750,"props":1628,"children":1629},{"class":752,"line":1046},[1630,1634],{"type":45,"tag":750,"props":1631,"children":1632},{"style":785},[1633],{"type":51,"value":1478},{"type":45,"tag":750,"props":1635,"children":1636},{"style":785},[1637],{"type":51,"value":1483},{"type":45,"tag":295,"props":1639,"children":1640},{},[],{"type":45,"tag":68,"props":1642,"children":1644},{"id":1643},"dependency-management",[1645],{"type":51,"value":1646},"Dependency Management",{"type":45,"tag":54,"props":1648,"children":1649},{},[1650,1652,1657,1659,1664,1666,1671],{"type":51,"value":1651},"Dependencies are declared in ",{"type":45,"tag":95,"props":1653,"children":1655},{"className":1654},[],[1656],{"type":51,"value":475},{"type":51,"value":1658},". The venv lives at ",{"type":45,"tag":95,"props":1660,"children":1662},{"className":1661},[],[1663],{"type":51,"value":100},{"type":51,"value":1665},"\ninside the container (already on ",{"type":45,"tag":95,"props":1667,"children":1669},{"className":1668},[],[1670],{"type":51,"value":108},{"type":51,"value":1672},").",{"type":45,"tag":547,"props":1674,"children":1675},{},[1676],{"type":45,"tag":54,"props":1677,"children":1678},{},[1679,1692,1694,1700,1702,1708],{"type":45,"tag":60,"props":1680,"children":1681},{},[1682,1684,1690],{"type":51,"value":1683},"All ",{"type":45,"tag":95,"props":1685,"children":1687},{"className":1686},[],[1688],{"type":51,"value":1689},"uv",{"type":51,"value":1691}," operations must be run inside the container.",{"type":51,"value":1693},"\nNever run ",{"type":45,"tag":95,"props":1695,"children":1697},{"className":1696},[],[1698],{"type":51,"value":1699},"uv sync",{"type":51,"value":1701}," \u002F ",{"type":45,"tag":95,"props":1703,"children":1705},{"className":1704},[],[1706],{"type":51,"value":1707},"uv pip install",{"type":51,"value":1709}," on the host.",{"type":45,"tag":1711,"props":1712,"children":1714},"h3",{"id":1713},"uv-dependency-groups",[1715],{"type":51,"value":1716},"uv Dependency Groups",{"type":45,"tag":393,"props":1718,"children":1719},{},[1720,1736],{"type":45,"tag":397,"props":1721,"children":1722},{},[1723],{"type":45,"tag":401,"props":1724,"children":1725},{},[1726,1731],{"type":45,"tag":405,"props":1727,"children":1728},{},[1729],{"type":51,"value":1730},"Group",{"type":45,"tag":405,"props":1732,"children":1733},{},[1734],{"type":51,"value":1735},"Purpose",{"type":45,"tag":431,"props":1737,"children":1738},{},[1739,1756,1772,1789,1806],{"type":45,"tag":401,"props":1740,"children":1741},{},[1742,1751],{"type":45,"tag":438,"props":1743,"children":1744},{},[1745],{"type":45,"tag":95,"props":1746,"children":1748},{"className":1747},[],[1749],{"type":51,"value":1750},"training",{"type":45,"tag":438,"props":1752,"children":1753},{},[1754],{"type":51,"value":1755},"Runtime training extras",{"type":45,"tag":401,"props":1757,"children":1758},{},[1759,1767],{"type":45,"tag":438,"props":1760,"children":1761},{},[1762],{"type":45,"tag":95,"props":1763,"children":1765},{"className":1764},[],[1766],{"type":51,"value":121},{"type":45,"tag":438,"props":1768,"children":1769},{},[1770],{"type":51,"value":1771},"Full dev environment (TransformerEngine, ModelOpt, …)",{"type":45,"tag":401,"props":1773,"children":1774},{},[1775,1784],{"type":45,"tag":438,"props":1776,"children":1777},{},[1778],{"type":45,"tag":95,"props":1779,"children":1781},{"className":1780},[],[1782],{"type":51,"value":1783},"test",{"type":45,"tag":438,"props":1785,"children":1786},{},[1787],{"type":51,"value":1788},"pytest, coverage, nemo-run",{"type":45,"tag":401,"props":1790,"children":1791},{},[1792,1801],{"type":45,"tag":438,"props":1793,"children":1794},{},[1795],{"type":45,"tag":95,"props":1796,"children":1798},{"className":1797},[],[1799],{"type":51,"value":1800},"linting",{"type":45,"tag":438,"props":1802,"children":1803},{},[1804],{"type":51,"value":1805},"ruff, black, isort, pylint",{"type":45,"tag":401,"props":1807,"children":1808},{},[1809,1818],{"type":45,"tag":438,"props":1810,"children":1811},{},[1812],{"type":45,"tag":95,"props":1813,"children":1815},{"className":1814},[],[1816],{"type":51,"value":1817},"build",{"type":45,"tag":438,"props":1819,"children":1820},{},[1821],{"type":51,"value":1822},"Cython, pybind11, nvidia-mathdx",{"type":45,"tag":547,"props":1824,"children":1825},{},[1826],{"type":45,"tag":54,"props":1827,"children":1828},{},[1829,1831,1836,1838,1843,1845,1850,1852,1857],{"type":51,"value":1830},"The previous ",{"type":45,"tag":95,"props":1832,"children":1834},{"className":1833},[],[1835],{"type":51,"value":144},{"type":51,"value":1837}," extra has been emptied. LTS deps are pinned in\n",{"type":45,"tag":95,"props":1839,"children":1841},{"className":1840},[],[1842],{"type":51,"value":531},{"type":51,"value":1844}," rather than ",{"type":45,"tag":95,"props":1846,"children":1848},{"className":1847},[],[1849],{"type":51,"value":475},{"type":51,"value":1851},". Do not add new\npackages under ",{"type":45,"tag":95,"props":1853,"children":1855},{"className":1854},[],[1856],{"type":51,"value":560},{"type":51,"value":110},{"type":45,"tag":54,"props":1859,"children":1860},{},[1861],{"type":51,"value":1862},"Install commands (inside the container):",{"type":45,"tag":739,"props":1864,"children":1866},{"className":741,"code":1865,"language":743,"meta":744,"style":744},"# Full dev + test environment\nuv sync --locked --group dev --group test\n\n# Linting only\nuv sync --locked --only-group linting\n",[1867],{"type":45,"tag":95,"props":1868,"children":1869},{"__ignoreMap":744},[1870,1878,1914,1921,1929],{"type":45,"tag":750,"props":1871,"children":1872},{"class":752,"line":753},[1873],{"type":45,"tag":750,"props":1874,"children":1875},{"style":757},[1876],{"type":51,"value":1877},"# Full dev + test environment\n",{"type":45,"tag":750,"props":1879,"children":1880},{"class":752,"line":763},[1881,1885,1890,1895,1900,1905,1909],{"type":45,"tag":750,"props":1882,"children":1883},{"style":779},[1884],{"type":51,"value":1689},{"type":45,"tag":750,"props":1886,"children":1887},{"style":785},[1888],{"type":51,"value":1889}," sync",{"type":45,"tag":750,"props":1891,"children":1892},{"style":785},[1893],{"type":51,"value":1894}," --locked",{"type":45,"tag":750,"props":1896,"children":1897},{"style":785},[1898],{"type":51,"value":1899}," --group",{"type":45,"tag":750,"props":1901,"children":1902},{"style":785},[1903],{"type":51,"value":1904}," dev",{"type":45,"tag":750,"props":1906,"children":1907},{"style":785},[1908],{"type":51,"value":1899},{"type":45,"tag":750,"props":1910,"children":1911},{"style":785},[1912],{"type":51,"value":1913}," test\n",{"type":45,"tag":750,"props":1915,"children":1916},{"class":752,"line":831},[1917],{"type":45,"tag":750,"props":1918,"children":1919},{"emptyLinePlaceholder":835},[1920],{"type":51,"value":838},{"type":45,"tag":750,"props":1922,"children":1923},{"class":752,"line":841},[1924],{"type":45,"tag":750,"props":1925,"children":1926},{"style":757},[1927],{"type":51,"value":1928},"# Linting only\n",{"type":45,"tag":750,"props":1930,"children":1931},{"class":752,"line":1011},[1932,1936,1940,1944,1949],{"type":45,"tag":750,"props":1933,"children":1934},{"style":779},[1935],{"type":51,"value":1689},{"type":45,"tag":750,"props":1937,"children":1938},{"style":785},[1939],{"type":51,"value":1889},{"type":45,"tag":750,"props":1941,"children":1942},{"style":785},[1943],{"type":51,"value":1894},{"type":45,"tag":750,"props":1945,"children":1946},{"style":785},[1947],{"type":51,"value":1948}," --only-group",{"type":45,"tag":750,"props":1950,"children":1951},{"style":785},[1952],{"type":51,"value":1953}," linting\n",{"type":45,"tag":54,"props":1955,"children":1956},{},[1957,1959,1964,1966,1971,1973,1978,1980,1985],{"type":51,"value":1958},"The LTS environment is reproduced by building ",{"type":45,"tag":95,"props":1960,"children":1962},{"className":1961},[],[1963],{"type":51,"value":522},{"type":51,"value":1965},"\nend-to-end; there is no ",{"type":45,"tag":95,"props":1967,"children":1969},{"className":1968},[],[1970],{"type":51,"value":1699},{"type":51,"value":1972},"-only equivalent because the LTS deps no\nlonger live in ",{"type":45,"tag":95,"props":1974,"children":1976},{"className":1975},[],[1977],{"type":51,"value":475},{"type":51,"value":1979},". The LTS top-level pin set is in\n",{"type":45,"tag":95,"props":1981,"children":1983},{"className":1982},[],[1984],{"type":51,"value":531},{"type":51,"value":1986},"; bump versions there and rebuild the image.",{"type":45,"tag":54,"props":1988,"children":1989},{},[1990,1992,1997,1999,2004,2006,2011],{"type":51,"value":1991},"Several dependencies are sourced directly from git (TransformerEngine, nemo-run,\nFlashMLA, Emerging-Optimizers, nvidia-resiliency-ext). The locked ",{"type":45,"tag":95,"props":1993,"children":1995},{"className":1994},[],[1996],{"type":51,"value":335},{"type":51,"value":1998}," file\npins exact revisions; update it with ",{"type":45,"tag":95,"props":2000,"children":2002},{"className":2001},[],[2003],{"type":51,"value":250},{"type":51,"value":2005}," when changing ",{"type":45,"tag":95,"props":2007,"children":2009},{"className":2008},[],[2010],{"type":51,"value":475},{"type":51,"value":110},{"type":45,"tag":1711,"props":2013,"children":2015},{"id":2014},"adding-a-new-dependency",[2016],{"type":51,"value":2017},"Adding a New Dependency",{"type":45,"tag":54,"props":2019,"children":2020},{},[2021],{"type":51,"value":2022},"Follow this three-step workflow:",{"type":45,"tag":2024,"props":2025,"children":2026},"ol",{},[2027,2045,2061],{"type":45,"tag":84,"props":2028,"children":2029},{},[2030,2035,2037,2043],{"type":45,"tag":60,"props":2031,"children":2032},{},[2033],{"type":51,"value":2034},"Acquire a container image",{"type":51,"value":2036}," — see ",{"type":45,"tag":1356,"props":2038,"children":2040},{"href":2039},"#step-1--acquire-an-image",[2041],{"type":51,"value":2042},"Step 1",{"type":51,"value":2044}," above.",{"type":45,"tag":84,"props":2046,"children":2047},{},[2048,2053,2054,2060],{"type":45,"tag":60,"props":2049,"children":2050},{},[2051],{"type":51,"value":2052},"Launch the container interactively",{"type":51,"value":2036},{"type":45,"tag":1356,"props":2055,"children":2057},{"href":2056},"#step-2--launch-the-container",[2058],{"type":51,"value":2059},"Step 2",{"type":51,"value":2044},{"type":45,"tag":84,"props":2062,"children":2063},{},[2064,2069,2071],{"type":45,"tag":60,"props":2065,"children":2066},{},[2067],{"type":51,"value":2068},"Update the lock file inside the container",{"type":51,"value":2070},", then commit it:",{"type":45,"tag":739,"props":2072,"children":2074},{"className":741,"code":2073,"language":743,"meta":744,"style":744},"# Inside the container:\nuv add \u003Cpackage>          # adds to pyproject.toml and resolves\nuv lock                   # regenerates uv.lock\n# Exit the container, then on the host:\ngit add pyproject.toml uv.lock\ngit commit -S -s -m \"build: add \u003Cpackage> dependency\"\n",[2075],{"type":45,"tag":95,"props":2076,"children":2077},{"__ignoreMap":744},[2078,2086,2123,2140,2148,2169],{"type":45,"tag":750,"props":2079,"children":2080},{"class":752,"line":753},[2081],{"type":45,"tag":750,"props":2082,"children":2083},{"style":757},[2084],{"type":51,"value":2085},"# Inside the container:\n",{"type":45,"tag":750,"props":2087,"children":2088},{"class":752,"line":763},[2089,2093,2098,2103,2108,2113,2118],{"type":45,"tag":750,"props":2090,"children":2091},{"style":779},[2092],{"type":51,"value":1689},{"type":45,"tag":750,"props":2094,"children":2095},{"style":785},[2096],{"type":51,"value":2097}," add",{"type":45,"tag":750,"props":2099,"children":2100},{"style":773},[2101],{"type":51,"value":2102}," \u003C",{"type":45,"tag":750,"props":2104,"children":2105},{"style":785},[2106],{"type":51,"value":2107},"packag",{"type":45,"tag":750,"props":2109,"children":2110},{"style":767},[2111],{"type":51,"value":2112},"e",{"type":45,"tag":750,"props":2114,"children":2115},{"style":773},[2116],{"type":51,"value":2117},">",{"type":45,"tag":750,"props":2119,"children":2120},{"style":757},[2121],{"type":51,"value":2122},"          # adds to pyproject.toml and resolves\n",{"type":45,"tag":750,"props":2124,"children":2125},{"class":752,"line":831},[2126,2130,2135],{"type":45,"tag":750,"props":2127,"children":2128},{"style":779},[2129],{"type":51,"value":1689},{"type":45,"tag":750,"props":2131,"children":2132},{"style":785},[2133],{"type":51,"value":2134}," lock",{"type":45,"tag":750,"props":2136,"children":2137},{"style":757},[2138],{"type":51,"value":2139},"                   # regenerates uv.lock\n",{"type":45,"tag":750,"props":2141,"children":2142},{"class":752,"line":841},[2143],{"type":45,"tag":750,"props":2144,"children":2145},{"style":757},[2146],{"type":51,"value":2147},"# Exit the container, then on the host:\n",{"type":45,"tag":750,"props":2149,"children":2150},{"class":752,"line":1011},[2151,2155,2159,2164],{"type":45,"tag":750,"props":2152,"children":2153},{"style":779},[2154],{"type":51,"value":782},{"type":45,"tag":750,"props":2156,"children":2157},{"style":785},[2158],{"type":51,"value":2097},{"type":45,"tag":750,"props":2160,"children":2161},{"style":785},[2162],{"type":51,"value":2163}," pyproject.toml",{"type":45,"tag":750,"props":2165,"children":2166},{"style":785},[2167],{"type":51,"value":2168}," uv.lock\n",{"type":45,"tag":750,"props":2170,"children":2171},{"class":752,"line":1028},[2172,2176,2181,2186,2191,2196,2200,2205],{"type":45,"tag":750,"props":2173,"children":2174},{"style":779},[2175],{"type":51,"value":782},{"type":45,"tag":750,"props":2177,"children":2178},{"style":785},[2179],{"type":51,"value":2180}," commit",{"type":45,"tag":750,"props":2182,"children":2183},{"style":785},[2184],{"type":51,"value":2185}," -S",{"type":45,"tag":750,"props":2187,"children":2188},{"style":785},[2189],{"type":51,"value":2190}," -s",{"type":45,"tag":750,"props":2192,"children":2193},{"style":785},[2194],{"type":51,"value":2195}," -m",{"type":45,"tag":750,"props":2197,"children":2198},{"style":773},[2199],{"type":51,"value":1331},{"type":45,"tag":750,"props":2201,"children":2202},{"style":785},[2203],{"type":51,"value":2204},"build: add \u003Cpackage> dependency",{"type":45,"tag":750,"props":2206,"children":2207},{"style":773},[2208],{"type":51,"value":1341},{"type":45,"tag":1711,"props":2210,"children":2212},{"id":2211},"resolving-a-merge-conflict-in-uvlock",[2213],{"type":51,"value":2214},"Resolving a merge conflict in uv.lock",{"type":45,"tag":54,"props":2216,"children":2217},{},[2218,2223],{"type":45,"tag":95,"props":2219,"children":2221},{"className":2220},[],[2222],{"type":51,"value":335},{"type":51,"value":2224}," is machine-generated; never resolve conflicts manually. Instead:",{"type":45,"tag":739,"props":2226,"children":2228},{"className":741,"code":2227,"language":743,"meta":744,"style":744},"git checkout origin\u002Fmain -- uv.lock   # take main's version as the base\n# then inside the container:\nuv lock                               # re-resolve on top of your pyproject.toml changes\n",[2229],{"type":45,"tag":95,"props":2230,"children":2231},{"__ignoreMap":744},[2232,2264,2272],{"type":45,"tag":750,"props":2233,"children":2234},{"class":752,"line":753},[2235,2239,2244,2249,2254,2259],{"type":45,"tag":750,"props":2236,"children":2237},{"style":779},[2238],{"type":51,"value":782},{"type":45,"tag":750,"props":2240,"children":2241},{"style":785},[2242],{"type":51,"value":2243}," checkout",{"type":45,"tag":750,"props":2245,"children":2246},{"style":785},[2247],{"type":51,"value":2248}," origin\u002Fmain",{"type":45,"tag":750,"props":2250,"children":2251},{"style":785},[2252],{"type":51,"value":2253}," --",{"type":45,"tag":750,"props":2255,"children":2256},{"style":785},[2257],{"type":51,"value":2258}," uv.lock",{"type":45,"tag":750,"props":2260,"children":2261},{"style":757},[2262],{"type":51,"value":2263},"   # take main's version as the base\n",{"type":45,"tag":750,"props":2265,"children":2266},{"class":752,"line":763},[2267],{"type":45,"tag":750,"props":2268,"children":2269},{"style":757},[2270],{"type":51,"value":2271},"# then inside the container:\n",{"type":45,"tag":750,"props":2273,"children":2274},{"class":752,"line":831},[2275,2279,2283],{"type":45,"tag":750,"props":2276,"children":2277},{"style":779},[2278],{"type":51,"value":1689},{"type":45,"tag":750,"props":2280,"children":2281},{"style":785},[2282],{"type":51,"value":2134},{"type":45,"tag":750,"props":2284,"children":2285},{"style":757},[2286],{"type":51,"value":2287},"                               # re-resolve on top of your pyproject.toml changes\n",{"type":45,"tag":295,"props":2289,"children":2290},{},[],{"type":45,"tag":68,"props":2292,"children":2294},{"id":2293},"common-pitfalls",[2295],{"type":51,"value":2296},"Common Pitfalls",{"type":45,"tag":393,"props":2298,"children":2299},{},[2300,2321],{"type":45,"tag":397,"props":2301,"children":2302},{},[2303],{"type":45,"tag":401,"props":2304,"children":2305},{},[2306,2311,2316],{"type":45,"tag":405,"props":2307,"children":2308},{},[2309],{"type":51,"value":2310},"Problem",{"type":45,"tag":405,"props":2312,"children":2313},{},[2314],{"type":51,"value":2315},"Cause",{"type":45,"tag":405,"props":2317,"children":2318},{},[2319],{"type":51,"value":2320},"Fix",{"type":45,"tag":431,"props":2322,"children":2323},{},[2324,2360,2403,2440,2476,2526],{"type":45,"tag":401,"props":2325,"children":2326},{},[2327,2338,2348],{"type":45,"tag":438,"props":2328,"children":2329},{},[2330,2336],{"type":45,"tag":95,"props":2331,"children":2333},{"className":2332},[],[2334],{"type":51,"value":2335},"uv sync --locked",{"type":51,"value":2337}," fails",{"type":45,"tag":438,"props":2339,"children":2340},{},[2341,2343],{"type":51,"value":2342},"Dependency conflict or stale ",{"type":45,"tag":95,"props":2344,"children":2346},{"className":2345},[],[2347],{"type":51,"value":335},{"type":45,"tag":438,"props":2349,"children":2350},{},[2351,2353,2358],{"type":51,"value":2352},"Re-run ",{"type":45,"tag":95,"props":2354,"children":2356},{"className":2355},[],[2357],{"type":51,"value":250},{"type":51,"value":2359}," inside the container and commit updated lock",{"type":45,"tag":401,"props":2361,"children":2362},{},[2363,2374,2379],{"type":45,"tag":438,"props":2364,"children":2365},{},[2366,2372],{"type":45,"tag":95,"props":2367,"children":2369},{"className":2368},[],[2370],{"type":51,"value":2371},"ModuleNotFoundError",{"type":51,"value":2373}," after pip install",{"type":45,"tag":438,"props":2375,"children":2376},{},[2377],{"type":51,"value":2378},"pip installed outside the uv-managed venv",{"type":45,"tag":438,"props":2380,"children":2381},{},[2382,2383,2389,2390,2395,2397],{"type":51,"value":604},{"type":45,"tag":95,"props":2384,"children":2386},{"className":2385},[],[2387],{"type":51,"value":2388},"uv add",{"type":51,"value":271},{"type":45,"tag":95,"props":2391,"children":2393},{"className":2392},[],[2394],{"type":51,"value":1699},{"type":51,"value":2396},", never bare ",{"type":45,"tag":95,"props":2398,"children":2400},{"className":2399},[],[2401],{"type":51,"value":2402},"pip install",{"type":45,"tag":401,"props":2404,"children":2405},{},[2406,2417,2422],{"type":45,"tag":438,"props":2407,"children":2408},{},[2409,2415],{"type":45,"tag":95,"props":2410,"children":2412},{"className":2411},[],[2413],{"type":51,"value":2414},"uv: command not found",{"type":51,"value":2416}," inside container",{"type":45,"tag":438,"props":2418,"children":2419},{},[2420],{"type":51,"value":2421},"Wrong container image",{"type":45,"tag":438,"props":2423,"children":2424},{},[2425,2427,2433,2435],{"type":51,"value":2426},"Use the ",{"type":45,"tag":95,"props":2428,"children":2430},{"className":2429},[],[2431],{"type":51,"value":2432},"megatron-lm",{"type":51,"value":2434}," image built from ",{"type":45,"tag":95,"props":2436,"children":2438},{"className":2437},[],[2439],{"type":51,"value":893},{"type":45,"tag":401,"props":2441,"children":2442},{},[2443,2454,2465],{"type":45,"tag":438,"props":2444,"children":2445},{},[2446,2452],{"type":45,"tag":95,"props":2447,"children":2449},{"className":2448},[],[2450],{"type":51,"value":2451},"No space left on device",{"type":51,"value":2453}," during uv ops",{"type":45,"tag":438,"props":2455,"children":2456},{},[2457,2459],{"type":51,"value":2458},"Cache fills container's ",{"type":45,"tag":95,"props":2460,"children":2462},{"className":2461},[],[2463],{"type":51,"value":2464},"\u002Froot\u002F.cache\u002F",{"type":45,"tag":438,"props":2466,"children":2467},{},[2468,2470],{"type":51,"value":2469},"Mount a host cache dir via ",{"type":45,"tag":95,"props":2471,"children":2473},{"className":2472},[],[2474],{"type":51,"value":2475},"-v $HOME\u002F.cache\u002Fuv:\u002Froot\u002F.cache\u002Fuv",{"type":45,"tag":401,"props":2477,"children":2478},{},[2479,2490,2507],{"type":45,"tag":438,"props":2480,"children":2481},{},[2482,2488],{"type":45,"tag":95,"props":2483,"children":2485},{"className":2484},[],[2486],{"type":51,"value":2487},"docker build",{"type":51,"value":2489}," fails with secret-related error",{"type":45,"tag":438,"props":2491,"children":2492},{},[2493,2498,2500,2505],{"type":45,"tag":95,"props":2494,"children":2496},{"className":2495},[],[2497],{"type":51,"value":893},{"type":51,"value":2499}," has a ",{"type":45,"tag":95,"props":2501,"children":2503},{"className":2502},[],[2504],{"type":51,"value":277},{"type":51,"value":2506}," stage that requires an internal secret",{"type":45,"tag":438,"props":2508,"children":2509},{},[2510,2512,2517,2519,2524],{"type":51,"value":2511},"Add ",{"type":45,"tag":95,"props":2513,"children":2515},{"className":2514},[],[2516],{"type":51,"value":292},{"type":51,"value":2518}," to stop before the ",{"type":45,"tag":95,"props":2520,"children":2522},{"className":2521},[],[2523],{"type":51,"value":277},{"type":51,"value":2525}," stage",{"type":45,"tag":401,"props":2527,"children":2528},{},[2529,2540,2552],{"type":45,"tag":438,"props":2530,"children":2531},{},[2532,2538],{"type":45,"tag":95,"props":2533,"children":2535},{"className":2534},[],[2536],{"type":51,"value":2537},"access forbidden",{"type":51,"value":2539}," when pulling",{"type":45,"tag":438,"props":2541,"children":2542},{},[2543,2545,2551],{"type":51,"value":2544},"Registry URL includes an explicit port (e.g. ",{"type":45,"tag":95,"props":2546,"children":2548},{"className":2547},[],[2549],{"type":51,"value":2550},":5005",{"type":51,"value":1004},{"type":45,"tag":438,"props":2553,"children":2554},{},[2555,2556,2562],{"type":51,"value":604},{"type":45,"tag":95,"props":2557,"children":2559},{"className":2558},[],[2560],{"type":51,"value":2561},"${GITLAB_HOST}\u002Fadlr\u002F...",{"type":51,"value":2563}," with no port — the sed extracts the hostname only",{"type":45,"tag":2565,"props":2566,"children":2567},"style",{},[2568],{"type":51,"value":2569},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2571,"total":2718},[2572,2590,2596,2607,2619,2633,2646,2660,2673,2684,2698,2707],{"slug":2573,"name":2573,"fn":2574,"description":2575,"org":2576,"tags":2577,"stars":2587,"repoUrl":2588,"updatedAt":2589},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2578,2581,2584],{"name":2579,"slug":2580,"type":15},"Documentation","documentation",{"name":2582,"slug":2583,"type":15},"MCP","mcp",{"name":2585,"slug":2586,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":4,"name":4,"fn":5,"description":6,"org":2591,"tags":2592,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2593,2594,2595],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"slug":2597,"name":2597,"fn":2598,"description":2599,"org":2600,"tags":2601,"stars":22,"repoUrl":23,"updatedAt":2606},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2602,2605],{"name":2603,"slug":2604,"type":15},"CI\u002FCD","ci-cd",{"name":17,"slug":18,"type":15},"2026-07-14T05:25:59.97109",{"slug":2608,"name":2608,"fn":2609,"description":2610,"org":2611,"tags":2612,"stars":22,"repoUrl":23,"updatedAt":2618},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2613,2614,2615],{"name":2603,"slug":2604,"type":15},{"name":17,"slug":18,"type":15},{"name":2616,"slug":2617,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2620,"name":2620,"fn":2621,"description":2622,"org":2623,"tags":2624,"stars":22,"repoUrl":23,"updatedAt":2632},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2625,2628,2629],{"name":2626,"slug":2627,"type":15},"Debugging","debugging",{"name":2616,"slug":2617,"type":15},{"name":2630,"slug":2631,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2634,"name":2634,"fn":2635,"description":2636,"org":2637,"tags":2638,"stars":22,"repoUrl":23,"updatedAt":2645},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2639,2642],{"name":2640,"slug":2641,"type":15},"Best Practices","best-practices",{"name":2643,"slug":2644,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2647,"name":2647,"fn":2648,"description":2649,"org":2650,"tags":2651,"stars":22,"repoUrl":23,"updatedAt":2659},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2652,2655,2658],{"name":2653,"slug":2654,"type":15},"Machine Learning","machine-learning",{"name":2656,"slug":2657,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2661,"name":2661,"fn":2662,"description":2663,"org":2664,"tags":2665,"stars":22,"repoUrl":23,"updatedAt":2672},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2666,2669],{"name":2667,"slug":2668,"type":15},"QA","qa",{"name":2670,"slug":2671,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":2674,"name":2674,"fn":2675,"description":2676,"org":2677,"tags":2678,"stars":22,"repoUrl":23,"updatedAt":2683},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2679,2680],{"name":17,"slug":18,"type":15},{"name":2681,"slug":2682,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":2685,"name":2685,"fn":2686,"description":2687,"org":2688,"tags":2689,"stars":22,"repoUrl":23,"updatedAt":2697},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2690,2693,2694],{"name":2691,"slug":2692,"type":15},"Code Review","code-review",{"name":2616,"slug":2617,"type":15},{"name":2695,"slug":2696,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2699,"name":2699,"fn":2700,"description":2701,"org":2702,"tags":2703,"stars":22,"repoUrl":23,"updatedAt":2706},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2704,2705],{"name":2667,"slug":2668,"type":15},{"name":2670,"slug":2671,"type":15},"2026-07-14T05:25:54.928983",{"slug":2708,"name":2708,"fn":2709,"description":2710,"org":2711,"tags":2712,"stars":22,"repoUrl":23,"updatedAt":2717},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2713,2716],{"name":2714,"slug":2715,"type":15},"Automation","automation",{"name":2603,"slug":2604,"type":15},"2026-07-30T05:29:03.275638",496,{"items":2720,"total":1147},[2721,2727,2732,2738,2744,2749,2755],{"slug":4,"name":4,"fn":5,"description":6,"org":2722,"tags":2723,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2724,2725,2726],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"name":20,"slug":21,"type":15},{"slug":2597,"name":2597,"fn":2598,"description":2599,"org":2728,"tags":2729,"stars":22,"repoUrl":23,"updatedAt":2606},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2730,2731],{"name":2603,"slug":2604,"type":15},{"name":17,"slug":18,"type":15},{"slug":2608,"name":2608,"fn":2609,"description":2610,"org":2733,"tags":2734,"stars":22,"repoUrl":23,"updatedAt":2618},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2735,2736,2737],{"name":2603,"slug":2604,"type":15},{"name":17,"slug":18,"type":15},{"name":2616,"slug":2617,"type":15},{"slug":2620,"name":2620,"fn":2621,"description":2622,"org":2739,"tags":2740,"stars":22,"repoUrl":23,"updatedAt":2632},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2741,2742,2743],{"name":2626,"slug":2627,"type":15},{"name":2616,"slug":2617,"type":15},{"name":2630,"slug":2631,"type":15},{"slug":2634,"name":2634,"fn":2635,"description":2636,"org":2745,"tags":2746,"stars":22,"repoUrl":23,"updatedAt":2645},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2747,2748],{"name":2640,"slug":2641,"type":15},{"name":2643,"slug":2644,"type":15},{"slug":2647,"name":2647,"fn":2648,"description":2649,"org":2750,"tags":2751,"stars":22,"repoUrl":23,"updatedAt":2659},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2752,2753,2754],{"name":2653,"slug":2654,"type":15},{"name":2656,"slug":2657,"type":15},{"name":9,"slug":8,"type":15},{"slug":2661,"name":2661,"fn":2662,"description":2663,"org":2756,"tags":2757,"stars":22,"repoUrl":23,"updatedAt":2672},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2758,2759],{"name":2667,"slug":2668,"type":15},{"name":2670,"slug":2671,"type":15}]