[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-mcore-bump-base-image":3,"mdc-xtsgg3-key":33,"related-repo-nvidia-mcore-bump-base-image":1869,"related-org-nvidia-mcore-bump-base-image":1957},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":19,"repoUrl":20,"updatedAt":21,"license":22,"forks":23,"topics":24,"repo":28,"sourceUrl":31,"mdContent":32},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16],{"name":13,"slug":14,"type":15},"Deployment","deployment","tag",{"name":17,"slug":18,"type":15},"CI\u002FCD","ci-cd",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-14T05:25:59.97109","Apache-2.0",4230,[25,26,27],"large-language-models","model-para","transformers",{"repoUrl":20,"stars":19,"forks":23,"topics":29,"description":30},[25,26,27],"Ongoing research training transformer models at scale","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM\u002Ftree\u002FHEAD\u002Fskills\u002Fmcore-bump-base-image","---\nname: mcore-bump-base-image\ndescription: Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs #4611 and #4688.\nlicense: Apache-2.0\nwhen_to_use: User wants to upgrade the PyTorch container (e.g. \"bump base image to 26.04\"); CI is failing after a previous bump because the GitLab pin was missed; functional tests are failing with `lm loss` \u002F `num-zeros` \u002F `iteration-time` drift right after a container bump; a functional test hangs, times out, or OOMs after a bump; the user mentions `.ngc_version.dev`, `nvcr.io\u002Fnvidia\u002Fpytorch`, \"container base image\", or \"Update Docker image version\".\nmetadata:\n  author: Oliver Koenig \u003Cokoenig@nvidia.com>\n---\n\n# Bump the PyTorch base image\n\nEnd-to-end workflow for moving Megatron-LM's CI to a newer `nvcr.io\u002Fnvidia\u002Fpytorch:\u003CYY.MM>-py3` container. The most common failure mode is forgetting that **GitHub CI and GitLab CI have separate pins** — a bump that only touches the former lands green, then breaks GitLab CI on `main` and forces an immediate follow-up PR. Always update both in the same PR.\n\n## Answer-First Pattern: dev Pin Sync\n\nFor a dev-only base-image bump, lead with the synchronization rule:\n\n- `docker\u002F.ngc_version.dev` is only the GitHub\u002Flocal Dockerfile pin.\n- GitLab CI has separate hardcoded `BASE_IMAGE` rows in\n  `.gitlab\u002Fstages\u002F01.build.yml`; update both `IMAGE_TYPE: dev` rows, one\n  `PLATFORM: amd64` and one `PLATFORM: arm64`.\n- Leave `docker\u002F.ngc_version.lts` and all `IMAGE_TYPE: lts` rows unchanged\n  unless the user explicitly asks for an LTS bump.\n- Verify before review with `cat docker\u002F.ngc_version.dev` plus\n  `rg -n '^\\s*BASE_IMAGE: nvcr\\.io\u002Fnvidia\u002Fpytorch:' .gitlab\u002Fstages\u002F01.build.yml | rg -B1 'IMAGE_TYPE: dev' | rg 'BASE_IMAGE'`.\n\n## Inputs to gather from the user\n\n1. **Target tag**, e.g. `26.04-py3`. NVIDIA NGC PyTorch containers are released as `nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`.\n2. **Scope** — usually `dev` only. The `lts` pin (`docker\u002F.ngc_version.lts`, plus the `FILE: Dockerfile.ci.lts` rows in GitLab) is bumped on a different cadence; only touch it if the user explicitly asks.\n3. **Workflow run ID** (optional but typical) — after the first CI run, the user will provide a GitHub Actions run ID for golden-value refresh.\n\n## Workflow\n\n```\n- [ ] Step 1: Update the GitHub CI pin (docker\u002F.ngc_version.dev)\n- [ ] Step 2: Update the GitLab CI pin (.gitlab\u002Fstages\u002F01.build.yml)\n- [ ] Step 3: Open the PR with the `Run functional tests` label\n- [ ] Step 4: Re-run failing tests via `\u002Fok to test \u003Ccommit-sha>`\n- [ ] Step 5: For golden-value drift → refresh with the `update-golden-values` skill\n- [ ] Step 6: For hangs \u002F real regressions → mark tests `mr-broken` and file tracking issues\n- [ ] Step 7: Verify both pins are in sync before merging\n```\n\n### Step 1 — GitHub CI pin\n\n`docker\u002F.ngc_version.dev` is a single-line file consumed by `docker\u002FDockerfile.ci.dev` (via `FROM_IMAGE_NAME=$(cat docker\u002F.ngc_version.dev)`). Overwrite it:\n\n```bash\necho 'nvcr.io\u002Fnvidia\u002Fpytorch:\u003CYY.MM>-py3' > docker\u002F.ngc_version.dev\n```\n\nThe file has no trailing newline historically; preserving or adding one is fine — the build args treat the value as `$(cat ...)`. Do **not** touch `docker\u002F.ngc_version.lts` unless bumping LTS too.\n\n### Step 2 — GitLab CI pin\n\nGitLab CI does **not** read `docker\u002F.ngc_version.dev`. It hardcodes `BASE_IMAGE` in a `parallel: matrix:` block. Update the two `IMAGE_TYPE: dev` rows (one per platform):\n\n```yaml\n# .gitlab\u002Fstages\u002F01.build.yml — under test:pre_build_image -> parallel.matrix\n- IMAGE: CI_MCORE_DEV_IMAGE\n  FILE: Dockerfile.ci.dev\n  IMAGE_TYPE: dev\n  BASE_IMAGE: nvcr.io\u002Fnvidia\u002Fpytorch:\u003CYY.MM>-py3   # amd64 row\n  PLATFORM: amd64\n- IMAGE: CI_MCORE_DEV_IMAGE\n  FILE: Dockerfile.ci.dev\n  IMAGE_TYPE: dev\n  BASE_IMAGE: nvcr.io\u002Fnvidia\u002Fpytorch:\u003CYY.MM>-py3   # arm64 row\n  PLATFORM: arm64\n```\n\nLeave the `FILE: Dockerfile.ci.lts` rows alone. Quick sanity check before commit:\n\n```bash\nrg -n '^\\s*BASE_IMAGE: nvcr\\.io\u002Fnvidia\u002Fpytorch:' .gitlab\u002Fstages\u002F01.build.yml\n# expect:  lts pin × 2 unchanged, dev pin × 2 == new tag\n```\n\n### Step 3 — Open the PR\n\n- Title convention: `chore: Update Docker image version to \u003CYY.MM>-py3` (see #4611).\n- **Apply the `Run functional tests` label** before the first push. This unlocks the full functional matrix on the PR; without it the bump only runs the standard GH PR checks and you'll miss the drift.\n- Push as draft first if you're still iterating; the bot will auto-draft otherwise.\n\n### Step 4 — Re-running CI on a new commit\n\nFor PRs from forks (the typical contributor case), each new commit needs an explicit `\u002Fok to test \u003Ccommit-sha>` PR comment to authorize NVIDIA runners (see the `copy-pr-bot` flow in #4611). One comment per commit. If `copy-pr-bot` reports \"had a problem deploying to test\", just push another commit (or re-issue the comment after the next push); the deploy is per-commit, not per-comment.\n\n### Step 5 — Golden-value drift\n\nContainer bumps shift CUDA \u002F cuBLAS \u002F cuDNN \u002F kernel autotuning, which moves `lm loss`, `num-zeros`, `iteration-time`, and `mem-*` metrics on a large fraction of functional tests. This is **expected** and is not a correctness regression — refresh the golden values rather than chasing each test.\n\nHand off to the `update-golden-values` skill with:\n\n- `--source github`\n- `--pipeline-id \u003CWORKFLOW_RUN_ID>` from the failing CI run\n- `--only-failing` (refresh just the trajectories that drifted)\n\nPR #4611 refreshed **78 golden-value files** across `dev_dgx_h100` and `dev_dgx_gb200` for GPT \u002F MoE \u002F MIMO \u002F hybrid suites in a single pass via this exact flow. The per-metric relative-difference summary the skill produces is the recommended PR description blurb — reviewers expect to see it.\n\n### Step 6 — Real regressions: mark broken, don't block the bump\n\nA small number of tests will genuinely break (hangs, OOM, real numerical regressions). Don't gate the base-image bump on fixing them — that conflates two changes. Instead:\n\n1. **File a GitHub issue** describing the failure mode and linking the failing CI run.\n2. **Flip the test's scope to the `-broken` variant** in the recipe YAML under `tests\u002Ftest_utils\u002Frecipes\u002F\u003Carch>\u002F`, with an inline comment that references the issue. Pattern:\n\n   ```yaml\n   - test_case: [hybrid_dynamic_inference_tp1_ep8_nanov3_chunked_prefill]\n     products:\n       - environment: [dev]\n         # Broken: hangs on repeat iter 3, exceeds 1h job limit — see issue #\u003CN>.\n         scope: [mr-broken, mr-github-broken]      # was: [mr, mr-github]\n         platforms: [dgx_h100]\n   ```\n\n   Scope mapping (replace, don't append):\n\n   | Before        | After                |\n   | ------------- | -------------------- |\n   | `mr`          | `mr-broken`          |\n   | `mr-github`   | `mr-github-broken`   |\n   | `nightly`     | `nightly-broken`     |\n\n   The recipe still runs in the `-broken` scope, but failures stop blocking PR merges.\n\n### Step 7 — Sync check before merging\n\nThe single biggest failure mode of this workflow is shipping #4611 without #4688. Before you ask for the merge, confirm both pins resolve to the same tag:\n\n```bash\necho -n \"ngc_version.dev: \" && cat docker\u002F.ngc_version.dev\necho\necho \"gitlab dev rows:\"\nrg -n '^\\s*BASE_IMAGE: nvcr\\.io\u002Fnvidia\u002Fpytorch:' .gitlab\u002Fstages\u002F01.build.yml \\\n  | rg -B1 'IMAGE_TYPE: dev' \\\n  | rg 'BASE_IMAGE'\n```\n\nAll three lines should show `nvcr.io\u002Fnvidia\u002Fpytorch:\u003CYY.MM>-py3`. If they don't, fix it before merge — otherwise GitLab CI keeps building on the old container and the next person hits the same trap.\n\n## File-touch cheat sheet\n\n| Path                                                                                          | Edit                                                                                   |\n| --------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |\n| `docker\u002F.ngc_version.dev`                                                                     | Overwrite with new `nvcr.io\u002Fnvidia\u002Fpytorch:\u003CYY.MM>-py3`                                |\n| `.gitlab\u002Fstages\u002F01.build.yml`                                                                 | Update both `IMAGE_TYPE: dev` `BASE_IMAGE:` rows (amd64 + arm64)                       |\n| `tests\u002Ffunctional_tests\u002Ftest_cases\u002F**\u002Fgolden_values_dev_dgx_{h100,gb200}.json`                | Refresh via the `update-golden-values` skill                                           |\n| `tests\u002Ftest_utils\u002Frecipes\u002F\u003Carch>\u002F\u003Csuite>.yaml`                                                | Flip drifting \u002F hanging cases to `mr-broken` \u002F `mr-github-broken` with an issue link    |\n| `docker\u002F.ngc_version.lts`, `.gitlab\u002Fstages\u002F01.build.yml` `FILE: Dockerfile.ci.lts` rows       | **Skip unless explicitly bumping LTS.** LTS has its own release cadence and its own Dockerfile (`docker\u002FDockerfile.ci.lts`); LTS Python deps are pinned in `docker\u002Flts\u002Frequirements.txt`. |\n\n## Gotchas\n\n- **GitHub vs GitLab pins are independent.** `docker\u002F.ngc_version.dev` only drives GitHub CI's local container build via `Dockerfile.ci.dev`. GitLab CI has its own hardcoded `BASE_IMAGE:` matrix in `.gitlab\u002Fstages\u002F01.build.yml`. PR #4688 existed solely because #4611 forgot the second one — don't repeat this.\n- **Don't bump LTS along with dev.** The `FILE: Dockerfile.ci.lts` rows, `docker\u002FDockerfile.ci.lts`, `docker\u002Flts\u002Frequirements.txt`, and `docker\u002F.ngc_version.lts` are stability-pinned for the `container::lts` label path. Bump them in a dedicated PR with its own LTS validation. LTS Python deps are pinned in `docker\u002Flts\u002Frequirements.txt` (not in `pyproject.toml`) — edit that file when an LTS dependency needs to move.\n- **Don't fix golden-value drift by hand.** Use `tests\u002Ftest_utils\u002Fpython_scripts\u002Fdownload_golden_values.py` via the `update-golden-values` skill. Hand-editing the JSONs invites diff noise and relative-difference regressions on subsequent bumps.\n- **`mr-broken` is a real scope, not a comment marker.** It keeps the recipe wired into the matrix (so it stays discoverable and runnable on demand) without gating merges. Don't delete the test case from the recipe.\n- **`\u002Fok to test` is per-commit.** A new force-push or fixup commit needs a fresh `\u002Fok to test \u003Csha>` comment to re-trigger NVIDIA-runner CI on a fork PR.\n- **Don't merge until the GitLab pin matches.** Use the Step 7 grep before requesting review.\n\n## Related skills\n\n- [update-golden-values](..\u002Fupdate-golden-values\u002FSKILL.md) — call this as soon as the first post-bump CI run finishes and you have a workflow run ID with failing golden checks. Produces the per-metric relative-difference summary you paste into the PR description.\n- [build-and-dependency](..\u002Fbuild-and-dependency\u002FSKILL.md) — for verifying the new image builds locally before opening the PR (`docker build --target main --build-arg FROM_IMAGE_NAME=$(cat docker\u002F.ngc_version.dev) ...`).\n- [cicd](..\u002Fcicd\u002FSKILL.md) — for the PR scope-label semantics (`Run functional tests`, `complexity::*`) and the `copy-pr-bot` flow.\n",{"data":34,"body":38},{"name":4,"description":6,"license":22,"when_to_use":35,"metadata":36},"User wants to upgrade the PyTorch container (e.g. \"bump base image to 26.04\"); CI is failing after a previous bump because the GitLab pin was missed; functional tests are failing with `lm loss` \u002F `num-zeros` \u002F `iteration-time` drift right after a container bump; a functional test hangs, times out, or OOMs after a bump; the user mentions `.ngc_version.dev`, `nvcr.io\u002Fnvidia\u002Fpytorch`, \"container base image\", or \"Update Docker image version\".",{"author":37},"Oliver Koenig \u003Cokoenig@nvidia.com>",{"type":39,"children":40},"root",[41,50,81,88,93,195,201,281,287,299,306,332,379,406,412,452,662,674,721,727,766,772,800,806,849,862,896,924,930,935,1236,1242,1247,1415,1427,1433,1612,1618,1792,1798,1863],{"type":42,"tag":43,"props":44,"children":46},"element","h1",{"id":45},"bump-the-pytorch-base-image",[47],{"type":48,"value":49},"text","Bump the PyTorch base image",{"type":42,"tag":51,"props":52,"children":53},"p",{},[54,56,63,65,71,73,79],{"type":48,"value":55},"End-to-end workflow for moving Megatron-LM's CI to a newer ",{"type":42,"tag":57,"props":58,"children":60},"code",{"className":59},[],[61],{"type":48,"value":62},"nvcr.io\u002Fnvidia\u002Fpytorch:\u003CYY.MM>-py3",{"type":48,"value":64}," container. The most common failure mode is forgetting that ",{"type":42,"tag":66,"props":67,"children":68},"strong",{},[69],{"type":48,"value":70},"GitHub CI and GitLab CI have separate pins",{"type":48,"value":72}," — a bump that only touches the former lands green, then breaks GitLab CI on ",{"type":42,"tag":57,"props":74,"children":76},{"className":75},[],[77],{"type":48,"value":78},"main",{"type":48,"value":80}," and forces an immediate follow-up PR. Always update both in the same PR.",{"type":42,"tag":82,"props":83,"children":85},"h2",{"id":84},"answer-first-pattern-dev-pin-sync",[86],{"type":48,"value":87},"Answer-First Pattern: dev Pin Sync",{"type":42,"tag":51,"props":89,"children":90},{},[91],{"type":48,"value":92},"For a dev-only base-image bump, lead with the synchronization rule:",{"type":42,"tag":94,"props":95,"children":96},"ul",{},[97,109,154,175],{"type":42,"tag":98,"props":99,"children":100},"li",{},[101,107],{"type":42,"tag":57,"props":102,"children":104},{"className":103},[],[105],{"type":48,"value":106},"docker\u002F.ngc_version.dev",{"type":48,"value":108}," is only the GitHub\u002Flocal Dockerfile pin.",{"type":42,"tag":98,"props":110,"children":111},{},[112,114,120,122,128,130,136,138,144,146,152],{"type":48,"value":113},"GitLab CI has separate hardcoded ",{"type":42,"tag":57,"props":115,"children":117},{"className":116},[],[118],{"type":48,"value":119},"BASE_IMAGE",{"type":48,"value":121}," rows in\n",{"type":42,"tag":57,"props":123,"children":125},{"className":124},[],[126],{"type":48,"value":127},".gitlab\u002Fstages\u002F01.build.yml",{"type":48,"value":129},"; update both ",{"type":42,"tag":57,"props":131,"children":133},{"className":132},[],[134],{"type":48,"value":135},"IMAGE_TYPE: dev",{"type":48,"value":137}," rows, one\n",{"type":42,"tag":57,"props":139,"children":141},{"className":140},[],[142],{"type":48,"value":143},"PLATFORM: amd64",{"type":48,"value":145}," and one ",{"type":42,"tag":57,"props":147,"children":149},{"className":148},[],[150],{"type":48,"value":151},"PLATFORM: arm64",{"type":48,"value":153},".",{"type":42,"tag":98,"props":155,"children":156},{},[157,159,165,167,173],{"type":48,"value":158},"Leave ",{"type":42,"tag":57,"props":160,"children":162},{"className":161},[],[163],{"type":48,"value":164},"docker\u002F.ngc_version.lts",{"type":48,"value":166}," and all ",{"type":42,"tag":57,"props":168,"children":170},{"className":169},[],[171],{"type":48,"value":172},"IMAGE_TYPE: lts",{"type":48,"value":174}," rows unchanged\nunless the user explicitly asks for an LTS bump.",{"type":42,"tag":98,"props":176,"children":177},{},[178,180,186,188,194],{"type":48,"value":179},"Verify before review with ",{"type":42,"tag":57,"props":181,"children":183},{"className":182},[],[184],{"type":48,"value":185},"cat docker\u002F.ngc_version.dev",{"type":48,"value":187}," plus\n",{"type":42,"tag":57,"props":189,"children":191},{"className":190},[],[192],{"type":48,"value":193},"rg -n '^\\s*BASE_IMAGE: nvcr\\.io\u002Fnvidia\u002Fpytorch:' .gitlab\u002Fstages\u002F01.build.yml | rg -B1 'IMAGE_TYPE: dev' | rg 'BASE_IMAGE'",{"type":48,"value":153},{"type":42,"tag":82,"props":196,"children":198},{"id":197},"inputs-to-gather-from-the-user",[199],{"type":48,"value":200},"Inputs to gather from the user",{"type":42,"tag":202,"props":203,"children":204},"ol",{},[205,230,271],{"type":42,"tag":98,"props":206,"children":207},{},[208,213,215,221,223,229],{"type":42,"tag":66,"props":209,"children":210},{},[211],{"type":48,"value":212},"Target tag",{"type":48,"value":214},", e.g. ",{"type":42,"tag":57,"props":216,"children":218},{"className":217},[],[219],{"type":48,"value":220},"26.04-py3",{"type":48,"value":222},". NVIDIA NGC PyTorch containers are released as ",{"type":42,"tag":57,"props":224,"children":226},{"className":225},[],[227],{"type":48,"value":228},"nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3",{"type":48,"value":153},{"type":42,"tag":98,"props":231,"children":232},{},[233,238,240,246,248,254,256,261,263,269],{"type":42,"tag":66,"props":234,"children":235},{},[236],{"type":48,"value":237},"Scope",{"type":48,"value":239}," — usually ",{"type":42,"tag":57,"props":241,"children":243},{"className":242},[],[244],{"type":48,"value":245},"dev",{"type":48,"value":247}," only. The ",{"type":42,"tag":57,"props":249,"children":251},{"className":250},[],[252],{"type":48,"value":253},"lts",{"type":48,"value":255}," pin (",{"type":42,"tag":57,"props":257,"children":259},{"className":258},[],[260],{"type":48,"value":164},{"type":48,"value":262},", plus the ",{"type":42,"tag":57,"props":264,"children":266},{"className":265},[],[267],{"type":48,"value":268},"FILE: Dockerfile.ci.lts",{"type":48,"value":270}," rows in GitLab) is bumped on a different cadence; only touch it if the user explicitly asks.",{"type":42,"tag":98,"props":272,"children":273},{},[274,279],{"type":42,"tag":66,"props":275,"children":276},{},[277],{"type":48,"value":278},"Workflow run ID",{"type":48,"value":280}," (optional but typical) — after the first CI run, the user will provide a GitHub Actions run ID for golden-value refresh.",{"type":42,"tag":82,"props":282,"children":284},{"id":283},"workflow",[285],{"type":48,"value":286},"Workflow",{"type":42,"tag":288,"props":289,"children":293},"pre",{"className":290,"code":292,"language":48},[291],"language-text","- [ ] Step 1: Update the GitHub CI pin (docker\u002F.ngc_version.dev)\n- [ ] Step 2: Update the GitLab CI pin (.gitlab\u002Fstages\u002F01.build.yml)\n- [ ] Step 3: Open the PR with the `Run functional tests` label\n- [ ] Step 4: Re-run failing tests via `\u002Fok to test \u003Ccommit-sha>`\n- [ ] Step 5: For golden-value drift → refresh with the `update-golden-values` skill\n- [ ] Step 6: For hangs \u002F real regressions → mark tests `mr-broken` and file tracking issues\n- [ ] Step 7: Verify both pins are in sync before merging\n",[294],{"type":42,"tag":57,"props":295,"children":297},{"__ignoreMap":296},"",[298],{"type":48,"value":292},{"type":42,"tag":300,"props":301,"children":303},"h3",{"id":302},"step-1-github-ci-pin",[304],{"type":48,"value":305},"Step 1 — GitHub CI pin",{"type":42,"tag":51,"props":307,"children":308},{},[309,314,316,322,324,330],{"type":42,"tag":57,"props":310,"children":312},{"className":311},[],[313],{"type":48,"value":106},{"type":48,"value":315}," is a single-line file consumed by ",{"type":42,"tag":57,"props":317,"children":319},{"className":318},[],[320],{"type":48,"value":321},"docker\u002FDockerfile.ci.dev",{"type":48,"value":323}," (via ",{"type":42,"tag":57,"props":325,"children":327},{"className":326},[],[328],{"type":48,"value":329},"FROM_IMAGE_NAME=$(cat docker\u002F.ngc_version.dev)",{"type":48,"value":331},"). Overwrite it:",{"type":42,"tag":288,"props":333,"children":337},{"className":334,"code":335,"language":336,"meta":296,"style":296},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","echo 'nvcr.io\u002Fnvidia\u002Fpytorch:\u003CYY.MM>-py3' > docker\u002F.ngc_version.dev\n","bash",[338],{"type":42,"tag":57,"props":339,"children":340},{"__ignoreMap":296},[341],{"type":42,"tag":342,"props":343,"children":346},"span",{"class":344,"line":345},"line",1,[347,353,359,364,369,374],{"type":42,"tag":342,"props":348,"children":350},{"style":349},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[351],{"type":48,"value":352},"echo",{"type":42,"tag":342,"props":354,"children":356},{"style":355},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[357],{"type":48,"value":358}," '",{"type":42,"tag":342,"props":360,"children":362},{"style":361},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[363],{"type":48,"value":62},{"type":42,"tag":342,"props":365,"children":366},{"style":355},[367],{"type":48,"value":368},"'",{"type":42,"tag":342,"props":370,"children":371},{"style":355},[372],{"type":48,"value":373}," >",{"type":42,"tag":342,"props":375,"children":376},{"style":361},[377],{"type":48,"value":378}," docker\u002F.ngc_version.dev\n",{"type":42,"tag":51,"props":380,"children":381},{},[382,384,390,392,397,399,404],{"type":48,"value":383},"The file has no trailing newline historically; preserving or adding one is fine — the build args treat the value as ",{"type":42,"tag":57,"props":385,"children":387},{"className":386},[],[388],{"type":48,"value":389},"$(cat ...)",{"type":48,"value":391},". Do ",{"type":42,"tag":66,"props":393,"children":394},{},[395],{"type":48,"value":396},"not",{"type":48,"value":398}," touch ",{"type":42,"tag":57,"props":400,"children":402},{"className":401},[],[403],{"type":48,"value":164},{"type":48,"value":405}," unless bumping LTS too.",{"type":42,"tag":300,"props":407,"children":409},{"id":408},"step-2-gitlab-ci-pin",[410],{"type":48,"value":411},"Step 2 — GitLab CI pin",{"type":42,"tag":51,"props":413,"children":414},{},[415,417,421,423,428,430,435,437,443,445,450],{"type":48,"value":416},"GitLab CI does ",{"type":42,"tag":66,"props":418,"children":419},{},[420],{"type":48,"value":396},{"type":48,"value":422}," read ",{"type":42,"tag":57,"props":424,"children":426},{"className":425},[],[427],{"type":48,"value":106},{"type":48,"value":429},". It hardcodes ",{"type":42,"tag":57,"props":431,"children":433},{"className":432},[],[434],{"type":48,"value":119},{"type":48,"value":436}," in a ",{"type":42,"tag":57,"props":438,"children":440},{"className":439},[],[441],{"type":48,"value":442},"parallel: matrix:",{"type":48,"value":444}," block. Update the two ",{"type":42,"tag":57,"props":446,"children":448},{"className":447},[],[449],{"type":48,"value":135},{"type":48,"value":451}," rows (one per platform):",{"type":42,"tag":288,"props":453,"children":457},{"className":454,"code":455,"language":456,"meta":296,"style":296},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# .gitlab\u002Fstages\u002F01.build.yml — under test:pre_build_image -> parallel.matrix\n- IMAGE: CI_MCORE_DEV_IMAGE\n  FILE: Dockerfile.ci.dev\n  IMAGE_TYPE: dev\n  BASE_IMAGE: nvcr.io\u002Fnvidia\u002Fpytorch:\u003CYY.MM>-py3   # amd64 row\n  PLATFORM: amd64\n- IMAGE: CI_MCORE_DEV_IMAGE\n  FILE: Dockerfile.ci.dev\n  IMAGE_TYPE: dev\n  BASE_IMAGE: nvcr.io\u002Fnvidia\u002Fpytorch:\u003CYY.MM>-py3   # arm64 row\n  PLATFORM: arm64\n","yaml",[458],{"type":42,"tag":57,"props":459,"children":460},{"__ignoreMap":296},[461,470,495,513,531,554,572,592,608,624,645],{"type":42,"tag":342,"props":462,"children":463},{"class":344,"line":345},[464],{"type":42,"tag":342,"props":465,"children":467},{"style":466},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[468],{"type":48,"value":469},"# .gitlab\u002Fstages\u002F01.build.yml — under test:pre_build_image -> parallel.matrix\n",{"type":42,"tag":342,"props":471,"children":473},{"class":344,"line":472},2,[474,479,485,490],{"type":42,"tag":342,"props":475,"children":476},{"style":355},[477],{"type":48,"value":478},"-",{"type":42,"tag":342,"props":480,"children":482},{"style":481},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[483],{"type":48,"value":484}," IMAGE",{"type":42,"tag":342,"props":486,"children":487},{"style":355},[488],{"type":48,"value":489},":",{"type":42,"tag":342,"props":491,"children":492},{"style":361},[493],{"type":48,"value":494}," CI_MCORE_DEV_IMAGE\n",{"type":42,"tag":342,"props":496,"children":498},{"class":344,"line":497},3,[499,504,508],{"type":42,"tag":342,"props":500,"children":501},{"style":481},[502],{"type":48,"value":503},"  FILE",{"type":42,"tag":342,"props":505,"children":506},{"style":355},[507],{"type":48,"value":489},{"type":42,"tag":342,"props":509,"children":510},{"style":361},[511],{"type":48,"value":512}," Dockerfile.ci.dev\n",{"type":42,"tag":342,"props":514,"children":516},{"class":344,"line":515},4,[517,522,526],{"type":42,"tag":342,"props":518,"children":519},{"style":481},[520],{"type":48,"value":521},"  IMAGE_TYPE",{"type":42,"tag":342,"props":523,"children":524},{"style":355},[525],{"type":48,"value":489},{"type":42,"tag":342,"props":527,"children":528},{"style":361},[529],{"type":48,"value":530}," dev\n",{"type":42,"tag":342,"props":532,"children":534},{"class":344,"line":533},5,[535,540,544,549],{"type":42,"tag":342,"props":536,"children":537},{"style":481},[538],{"type":48,"value":539},"  BASE_IMAGE",{"type":42,"tag":342,"props":541,"children":542},{"style":355},[543],{"type":48,"value":489},{"type":42,"tag":342,"props":545,"children":546},{"style":361},[547],{"type":48,"value":548}," nvcr.io\u002Fnvidia\u002Fpytorch:\u003CYY.MM>-py3",{"type":42,"tag":342,"props":550,"children":551},{"style":466},[552],{"type":48,"value":553},"   # amd64 row\n",{"type":42,"tag":342,"props":555,"children":557},{"class":344,"line":556},6,[558,563,567],{"type":42,"tag":342,"props":559,"children":560},{"style":481},[561],{"type":48,"value":562},"  PLATFORM",{"type":42,"tag":342,"props":564,"children":565},{"style":355},[566],{"type":48,"value":489},{"type":42,"tag":342,"props":568,"children":569},{"style":361},[570],{"type":48,"value":571}," amd64\n",{"type":42,"tag":342,"props":573,"children":575},{"class":344,"line":574},7,[576,580,584,588],{"type":42,"tag":342,"props":577,"children":578},{"style":355},[579],{"type":48,"value":478},{"type":42,"tag":342,"props":581,"children":582},{"style":481},[583],{"type":48,"value":484},{"type":42,"tag":342,"props":585,"children":586},{"style":355},[587],{"type":48,"value":489},{"type":42,"tag":342,"props":589,"children":590},{"style":361},[591],{"type":48,"value":494},{"type":42,"tag":342,"props":593,"children":595},{"class":344,"line":594},8,[596,600,604],{"type":42,"tag":342,"props":597,"children":598},{"style":481},[599],{"type":48,"value":503},{"type":42,"tag":342,"props":601,"children":602},{"style":355},[603],{"type":48,"value":489},{"type":42,"tag":342,"props":605,"children":606},{"style":361},[607],{"type":48,"value":512},{"type":42,"tag":342,"props":609,"children":611},{"class":344,"line":610},9,[612,616,620],{"type":42,"tag":342,"props":613,"children":614},{"style":481},[615],{"type":48,"value":521},{"type":42,"tag":342,"props":617,"children":618},{"style":355},[619],{"type":48,"value":489},{"type":42,"tag":342,"props":621,"children":622},{"style":361},[623],{"type":48,"value":530},{"type":42,"tag":342,"props":625,"children":627},{"class":344,"line":626},10,[628,632,636,640],{"type":42,"tag":342,"props":629,"children":630},{"style":481},[631],{"type":48,"value":539},{"type":42,"tag":342,"props":633,"children":634},{"style":355},[635],{"type":48,"value":489},{"type":42,"tag":342,"props":637,"children":638},{"style":361},[639],{"type":48,"value":548},{"type":42,"tag":342,"props":641,"children":642},{"style":466},[643],{"type":48,"value":644},"   # arm64 row\n",{"type":42,"tag":342,"props":646,"children":648},{"class":344,"line":647},11,[649,653,657],{"type":42,"tag":342,"props":650,"children":651},{"style":481},[652],{"type":48,"value":562},{"type":42,"tag":342,"props":654,"children":655},{"style":355},[656],{"type":48,"value":489},{"type":42,"tag":342,"props":658,"children":659},{"style":361},[660],{"type":48,"value":661}," arm64\n",{"type":42,"tag":51,"props":663,"children":664},{},[665,667,672],{"type":48,"value":666},"Leave the ",{"type":42,"tag":57,"props":668,"children":670},{"className":669},[],[671],{"type":48,"value":268},{"type":48,"value":673}," rows alone. Quick sanity check before commit:",{"type":42,"tag":288,"props":675,"children":677},{"className":334,"code":676,"language":336,"meta":296,"style":296},"rg -n '^\\s*BASE_IMAGE: nvcr\\.io\u002Fnvidia\u002Fpytorch:' .gitlab\u002Fstages\u002F01.build.yml\n# expect:  lts pin × 2 unchanged, dev pin × 2 == new tag\n",[678],{"type":42,"tag":57,"props":679,"children":680},{"__ignoreMap":296},[681,713],{"type":42,"tag":342,"props":682,"children":683},{"class":344,"line":345},[684,690,695,699,704,708],{"type":42,"tag":342,"props":685,"children":687},{"style":686},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[688],{"type":48,"value":689},"rg",{"type":42,"tag":342,"props":691,"children":692},{"style":361},[693],{"type":48,"value":694}," -n",{"type":42,"tag":342,"props":696,"children":697},{"style":355},[698],{"type":48,"value":358},{"type":42,"tag":342,"props":700,"children":701},{"style":361},[702],{"type":48,"value":703},"^\\s*BASE_IMAGE: nvcr\\.io\u002Fnvidia\u002Fpytorch:",{"type":42,"tag":342,"props":705,"children":706},{"style":355},[707],{"type":48,"value":368},{"type":42,"tag":342,"props":709,"children":710},{"style":361},[711],{"type":48,"value":712}," .gitlab\u002Fstages\u002F01.build.yml\n",{"type":42,"tag":342,"props":714,"children":715},{"class":344,"line":472},[716],{"type":42,"tag":342,"props":717,"children":718},{"style":466},[719],{"type":48,"value":720},"# expect:  lts pin × 2 unchanged, dev pin × 2 == new tag\n",{"type":42,"tag":300,"props":722,"children":724},{"id":723},"step-3-open-the-pr",[725],{"type":48,"value":726},"Step 3 — Open the PR",{"type":42,"tag":94,"props":728,"children":729},{},[730,743,761],{"type":42,"tag":98,"props":731,"children":732},{},[733,735,741],{"type":48,"value":734},"Title convention: ",{"type":42,"tag":57,"props":736,"children":738},{"className":737},[],[739],{"type":48,"value":740},"chore: Update Docker image version to \u003CYY.MM>-py3",{"type":48,"value":742}," (see #4611).",{"type":42,"tag":98,"props":744,"children":745},{},[746,759],{"type":42,"tag":66,"props":747,"children":748},{},[749,751,757],{"type":48,"value":750},"Apply the ",{"type":42,"tag":57,"props":752,"children":754},{"className":753},[],[755],{"type":48,"value":756},"Run functional tests",{"type":48,"value":758}," label",{"type":48,"value":760}," before the first push. This unlocks the full functional matrix on the PR; without it the bump only runs the standard GH PR checks and you'll miss the drift.",{"type":42,"tag":98,"props":762,"children":763},{},[764],{"type":48,"value":765},"Push as draft first if you're still iterating; the bot will auto-draft otherwise.",{"type":42,"tag":300,"props":767,"children":769},{"id":768},"step-4-re-running-ci-on-a-new-commit",[770],{"type":48,"value":771},"Step 4 — Re-running CI on a new commit",{"type":42,"tag":51,"props":773,"children":774},{},[775,777,783,785,791,793,798],{"type":48,"value":776},"For PRs from forks (the typical contributor case), each new commit needs an explicit ",{"type":42,"tag":57,"props":778,"children":780},{"className":779},[],[781],{"type":48,"value":782},"\u002Fok to test \u003Ccommit-sha>",{"type":48,"value":784}," PR comment to authorize NVIDIA runners (see the ",{"type":42,"tag":57,"props":786,"children":788},{"className":787},[],[789],{"type":48,"value":790},"copy-pr-bot",{"type":48,"value":792}," flow in #4611). One comment per commit. If ",{"type":42,"tag":57,"props":794,"children":796},{"className":795},[],[797],{"type":48,"value":790},{"type":48,"value":799}," reports \"had a problem deploying to test\", just push another commit (or re-issue the comment after the next push); the deploy is per-commit, not per-comment.",{"type":42,"tag":300,"props":801,"children":803},{"id":802},"step-5-golden-value-drift",[804],{"type":48,"value":805},"Step 5 — Golden-value drift",{"type":42,"tag":51,"props":807,"children":808},{},[809,811,817,819,825,826,832,834,840,842,847],{"type":48,"value":810},"Container bumps shift CUDA \u002F cuBLAS \u002F cuDNN \u002F kernel autotuning, which moves ",{"type":42,"tag":57,"props":812,"children":814},{"className":813},[],[815],{"type":48,"value":816},"lm loss",{"type":48,"value":818},", ",{"type":42,"tag":57,"props":820,"children":822},{"className":821},[],[823],{"type":48,"value":824},"num-zeros",{"type":48,"value":818},{"type":42,"tag":57,"props":827,"children":829},{"className":828},[],[830],{"type":48,"value":831},"iteration-time",{"type":48,"value":833},", and ",{"type":42,"tag":57,"props":835,"children":837},{"className":836},[],[838],{"type":48,"value":839},"mem-*",{"type":48,"value":841}," metrics on a large fraction of functional tests. This is ",{"type":42,"tag":66,"props":843,"children":844},{},[845],{"type":48,"value":846},"expected",{"type":48,"value":848}," and is not a correctness regression — refresh the golden values rather than chasing each test.",{"type":42,"tag":51,"props":850,"children":851},{},[852,854,860],{"type":48,"value":853},"Hand off to the ",{"type":42,"tag":57,"props":855,"children":857},{"className":856},[],[858],{"type":48,"value":859},"update-golden-values",{"type":48,"value":861}," skill with:",{"type":42,"tag":94,"props":863,"children":864},{},[865,874,885],{"type":42,"tag":98,"props":866,"children":867},{},[868],{"type":42,"tag":57,"props":869,"children":871},{"className":870},[],[872],{"type":48,"value":873},"--source github",{"type":42,"tag":98,"props":875,"children":876},{},[877,883],{"type":42,"tag":57,"props":878,"children":880},{"className":879},[],[881],{"type":48,"value":882},"--pipeline-id \u003CWORKFLOW_RUN_ID>",{"type":48,"value":884}," from the failing CI run",{"type":42,"tag":98,"props":886,"children":887},{},[888,894],{"type":42,"tag":57,"props":889,"children":891},{"className":890},[],[892],{"type":48,"value":893},"--only-failing",{"type":48,"value":895}," (refresh just the trajectories that drifted)",{"type":42,"tag":51,"props":897,"children":898},{},[899,901,906,908,914,916,922],{"type":48,"value":900},"PR #4611 refreshed ",{"type":42,"tag":66,"props":902,"children":903},{},[904],{"type":48,"value":905},"78 golden-value files",{"type":48,"value":907}," across ",{"type":42,"tag":57,"props":909,"children":911},{"className":910},[],[912],{"type":48,"value":913},"dev_dgx_h100",{"type":48,"value":915}," and ",{"type":42,"tag":57,"props":917,"children":919},{"className":918},[],[920],{"type":48,"value":921},"dev_dgx_gb200",{"type":48,"value":923}," for GPT \u002F MoE \u002F MIMO \u002F hybrid suites in a single pass via this exact flow. The per-metric relative-difference summary the skill produces is the recommended PR description blurb — reviewers expect to see it.",{"type":42,"tag":300,"props":925,"children":927},{"id":926},"step-6-real-regressions-mark-broken-dont-block-the-bump",[928],{"type":48,"value":929},"Step 6 — Real regressions: mark broken, don't block the bump",{"type":42,"tag":51,"props":931,"children":932},{},[933],{"type":48,"value":934},"A small number of tests will genuinely break (hangs, OOM, real numerical regressions). Don't gate the base-image bump on fixing them — that conflates two changes. Instead:",{"type":42,"tag":202,"props":936,"children":937},{},[938,948],{"type":42,"tag":98,"props":939,"children":940},{},[941,946],{"type":42,"tag":66,"props":942,"children":943},{},[944],{"type":48,"value":945},"File a GitHub issue",{"type":48,"value":947}," describing the failure mode and linking the failing CI run.",{"type":42,"tag":98,"props":949,"children":950},{},[951,964,966,972,974,1128,1132,1134,1224,1227,1229,1234],{"type":42,"tag":66,"props":952,"children":953},{},[954,956,962],{"type":48,"value":955},"Flip the test's scope to the ",{"type":42,"tag":57,"props":957,"children":959},{"className":958},[],[960],{"type":48,"value":961},"-broken",{"type":48,"value":963}," variant",{"type":48,"value":965}," in the recipe YAML under ",{"type":42,"tag":57,"props":967,"children":969},{"className":968},[],[970],{"type":48,"value":971},"tests\u002Ftest_utils\u002Frecipes\u002F\u003Carch>\u002F",{"type":48,"value":973},", with an inline comment that references the issue. Pattern:",{"type":42,"tag":288,"props":975,"children":977},{"className":454,"code":976,"language":456,"meta":296,"style":296},"- test_case: [hybrid_dynamic_inference_tp1_ep8_nanov3_chunked_prefill]\n  products:\n    - environment: [dev]\n      # Broken: hangs on repeat iter 3, exceeds 1h job limit — see issue #\u003CN>.\n      scope: [mr-broken, mr-github-broken]      # was: [mr, mr-github]\n      platforms: [dgx_h100]\n",[978],{"type":42,"tag":57,"props":979,"children":980},{"__ignoreMap":296},[981,1012,1025,1054,1062,1103],{"type":42,"tag":342,"props":982,"children":983},{"class":344,"line":345},[984,988,993,997,1002,1007],{"type":42,"tag":342,"props":985,"children":986},{"style":355},[987],{"type":48,"value":478},{"type":42,"tag":342,"props":989,"children":990},{"style":481},[991],{"type":48,"value":992}," test_case",{"type":42,"tag":342,"props":994,"children":995},{"style":355},[996],{"type":48,"value":489},{"type":42,"tag":342,"props":998,"children":999},{"style":355},[1000],{"type":48,"value":1001}," [",{"type":42,"tag":342,"props":1003,"children":1004},{"style":361},[1005],{"type":48,"value":1006},"hybrid_dynamic_inference_tp1_ep8_nanov3_chunked_prefill",{"type":42,"tag":342,"props":1008,"children":1009},{"style":355},[1010],{"type":48,"value":1011},"]\n",{"type":42,"tag":342,"props":1013,"children":1014},{"class":344,"line":472},[1015,1020],{"type":42,"tag":342,"props":1016,"children":1017},{"style":481},[1018],{"type":48,"value":1019},"  products",{"type":42,"tag":342,"props":1021,"children":1022},{"style":355},[1023],{"type":48,"value":1024},":\n",{"type":42,"tag":342,"props":1026,"children":1027},{"class":344,"line":497},[1028,1033,1038,1042,1046,1050],{"type":42,"tag":342,"props":1029,"children":1030},{"style":355},[1031],{"type":48,"value":1032},"    -",{"type":42,"tag":342,"props":1034,"children":1035},{"style":481},[1036],{"type":48,"value":1037}," environment",{"type":42,"tag":342,"props":1039,"children":1040},{"style":355},[1041],{"type":48,"value":489},{"type":42,"tag":342,"props":1043,"children":1044},{"style":355},[1045],{"type":48,"value":1001},{"type":42,"tag":342,"props":1047,"children":1048},{"style":361},[1049],{"type":48,"value":245},{"type":42,"tag":342,"props":1051,"children":1052},{"style":355},[1053],{"type":48,"value":1011},{"type":42,"tag":342,"props":1055,"children":1056},{"class":344,"line":515},[1057],{"type":42,"tag":342,"props":1058,"children":1059},{"style":466},[1060],{"type":48,"value":1061},"      # Broken: hangs on repeat iter 3, exceeds 1h job limit — see issue #\u003CN>.\n",{"type":42,"tag":342,"props":1063,"children":1064},{"class":344,"line":533},[1065,1070,1074,1078,1083,1088,1093,1098],{"type":42,"tag":342,"props":1066,"children":1067},{"style":481},[1068],{"type":48,"value":1069},"      scope",{"type":42,"tag":342,"props":1071,"children":1072},{"style":355},[1073],{"type":48,"value":489},{"type":42,"tag":342,"props":1075,"children":1076},{"style":355},[1077],{"type":48,"value":1001},{"type":42,"tag":342,"props":1079,"children":1080},{"style":361},[1081],{"type":48,"value":1082},"mr-broken",{"type":42,"tag":342,"props":1084,"children":1085},{"style":355},[1086],{"type":48,"value":1087},",",{"type":42,"tag":342,"props":1089,"children":1090},{"style":361},[1091],{"type":48,"value":1092}," mr-github-broken",{"type":42,"tag":342,"props":1094,"children":1095},{"style":355},[1096],{"type":48,"value":1097},"]",{"type":42,"tag":342,"props":1099,"children":1100},{"style":466},[1101],{"type":48,"value":1102},"      # was: [mr, mr-github]\n",{"type":42,"tag":342,"props":1104,"children":1105},{"class":344,"line":556},[1106,1111,1115,1119,1124],{"type":42,"tag":342,"props":1107,"children":1108},{"style":481},[1109],{"type":48,"value":1110},"      platforms",{"type":42,"tag":342,"props":1112,"children":1113},{"style":355},[1114],{"type":48,"value":489},{"type":42,"tag":342,"props":1116,"children":1117},{"style":355},[1118],{"type":48,"value":1001},{"type":42,"tag":342,"props":1120,"children":1121},{"style":361},[1122],{"type":48,"value":1123},"dgx_h100",{"type":42,"tag":342,"props":1125,"children":1126},{"style":355},[1127],{"type":48,"value":1011},{"type":42,"tag":1129,"props":1130,"children":1131},"br",{},[],{"type":48,"value":1133},"Scope mapping (replace, don't append):",{"type":42,"tag":1135,"props":1136,"children":1137},"table",{},[1138,1157],{"type":42,"tag":1139,"props":1140,"children":1141},"thead",{},[1142],{"type":42,"tag":1143,"props":1144,"children":1145},"tr",{},[1146,1152],{"type":42,"tag":1147,"props":1148,"children":1149},"th",{},[1150],{"type":48,"value":1151},"Before",{"type":42,"tag":1147,"props":1153,"children":1154},{},[1155],{"type":48,"value":1156},"After",{"type":42,"tag":1158,"props":1159,"children":1160},"tbody",{},[1161,1182,1203],{"type":42,"tag":1143,"props":1162,"children":1163},{},[1164,1174],{"type":42,"tag":1165,"props":1166,"children":1167},"td",{},[1168],{"type":42,"tag":57,"props":1169,"children":1171},{"className":1170},[],[1172],{"type":48,"value":1173},"mr",{"type":42,"tag":1165,"props":1175,"children":1176},{},[1177],{"type":42,"tag":57,"props":1178,"children":1180},{"className":1179},[],[1181],{"type":48,"value":1082},{"type":42,"tag":1143,"props":1183,"children":1184},{},[1185,1194],{"type":42,"tag":1165,"props":1186,"children":1187},{},[1188],{"type":42,"tag":57,"props":1189,"children":1191},{"className":1190},[],[1192],{"type":48,"value":1193},"mr-github",{"type":42,"tag":1165,"props":1195,"children":1196},{},[1197],{"type":42,"tag":57,"props":1198,"children":1200},{"className":1199},[],[1201],{"type":48,"value":1202},"mr-github-broken",{"type":42,"tag":1143,"props":1204,"children":1205},{},[1206,1215],{"type":42,"tag":1165,"props":1207,"children":1208},{},[1209],{"type":42,"tag":57,"props":1210,"children":1212},{"className":1211},[],[1213],{"type":48,"value":1214},"nightly",{"type":42,"tag":1165,"props":1216,"children":1217},{},[1218],{"type":42,"tag":57,"props":1219,"children":1221},{"className":1220},[],[1222],{"type":48,"value":1223},"nightly-broken",{"type":42,"tag":1129,"props":1225,"children":1226},{},[],{"type":48,"value":1228},"The recipe still runs in the ",{"type":42,"tag":57,"props":1230,"children":1232},{"className":1231},[],[1233],{"type":48,"value":961},{"type":48,"value":1235}," scope, but failures stop blocking PR merges.",{"type":42,"tag":300,"props":1237,"children":1239},{"id":1238},"step-7-sync-check-before-merging",[1240],{"type":48,"value":1241},"Step 7 — Sync check before merging",{"type":42,"tag":51,"props":1243,"children":1244},{},[1245],{"type":48,"value":1246},"The single biggest failure mode of this workflow is shipping #4611 without #4688. Before you ask for the merge, confirm both pins resolve to the same tag:",{"type":42,"tag":288,"props":1248,"children":1250},{"className":334,"code":1249,"language":336,"meta":296,"style":296},"echo -n \"ngc_version.dev: \" && cat docker\u002F.ngc_version.dev\necho\necho \"gitlab dev rows:\"\nrg -n '^\\s*BASE_IMAGE: nvcr\\.io\u002Fnvidia\u002Fpytorch:' .gitlab\u002Fstages\u002F01.build.yml \\\n  | rg -B1 'IMAGE_TYPE: dev' \\\n  | rg 'BASE_IMAGE'\n",[1251],{"type":42,"tag":57,"props":1252,"children":1253},{"__ignoreMap":296},[1254,1294,1302,1323,1357,1391],{"type":42,"tag":342,"props":1255,"children":1256},{"class":344,"line":345},[1257,1261,1265,1270,1275,1280,1285,1290],{"type":42,"tag":342,"props":1258,"children":1259},{"style":349},[1260],{"type":48,"value":352},{"type":42,"tag":342,"props":1262,"children":1263},{"style":361},[1264],{"type":48,"value":694},{"type":42,"tag":342,"props":1266,"children":1267},{"style":355},[1268],{"type":48,"value":1269}," \"",{"type":42,"tag":342,"props":1271,"children":1272},{"style":361},[1273],{"type":48,"value":1274},"ngc_version.dev: ",{"type":42,"tag":342,"props":1276,"children":1277},{"style":355},[1278],{"type":48,"value":1279},"\"",{"type":42,"tag":342,"props":1281,"children":1282},{"style":355},[1283],{"type":48,"value":1284}," &&",{"type":42,"tag":342,"props":1286,"children":1287},{"style":686},[1288],{"type":48,"value":1289}," cat",{"type":42,"tag":342,"props":1291,"children":1292},{"style":361},[1293],{"type":48,"value":378},{"type":42,"tag":342,"props":1295,"children":1296},{"class":344,"line":472},[1297],{"type":42,"tag":342,"props":1298,"children":1299},{"style":349},[1300],{"type":48,"value":1301},"echo\n",{"type":42,"tag":342,"props":1303,"children":1304},{"class":344,"line":497},[1305,1309,1313,1318],{"type":42,"tag":342,"props":1306,"children":1307},{"style":349},[1308],{"type":48,"value":352},{"type":42,"tag":342,"props":1310,"children":1311},{"style":355},[1312],{"type":48,"value":1269},{"type":42,"tag":342,"props":1314,"children":1315},{"style":361},[1316],{"type":48,"value":1317},"gitlab dev rows:",{"type":42,"tag":342,"props":1319,"children":1320},{"style":355},[1321],{"type":48,"value":1322},"\"\n",{"type":42,"tag":342,"props":1324,"children":1325},{"class":344,"line":515},[1326,1330,1334,1338,1342,1346,1351],{"type":42,"tag":342,"props":1327,"children":1328},{"style":686},[1329],{"type":48,"value":689},{"type":42,"tag":342,"props":1331,"children":1332},{"style":361},[1333],{"type":48,"value":694},{"type":42,"tag":342,"props":1335,"children":1336},{"style":355},[1337],{"type":48,"value":358},{"type":42,"tag":342,"props":1339,"children":1340},{"style":361},[1341],{"type":48,"value":703},{"type":42,"tag":342,"props":1343,"children":1344},{"style":355},[1345],{"type":48,"value":368},{"type":42,"tag":342,"props":1347,"children":1348},{"style":361},[1349],{"type":48,"value":1350}," .gitlab\u002Fstages\u002F01.build.yml",{"type":42,"tag":342,"props":1352,"children":1354},{"style":1353},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1355],{"type":48,"value":1356}," \\\n",{"type":42,"tag":342,"props":1358,"children":1359},{"class":344,"line":533},[1360,1365,1370,1375,1379,1383,1387],{"type":42,"tag":342,"props":1361,"children":1362},{"style":355},[1363],{"type":48,"value":1364},"  |",{"type":42,"tag":342,"props":1366,"children":1367},{"style":686},[1368],{"type":48,"value":1369}," rg",{"type":42,"tag":342,"props":1371,"children":1372},{"style":361},[1373],{"type":48,"value":1374}," -B1",{"type":42,"tag":342,"props":1376,"children":1377},{"style":355},[1378],{"type":48,"value":358},{"type":42,"tag":342,"props":1380,"children":1381},{"style":361},[1382],{"type":48,"value":135},{"type":42,"tag":342,"props":1384,"children":1385},{"style":355},[1386],{"type":48,"value":368},{"type":42,"tag":342,"props":1388,"children":1389},{"style":1353},[1390],{"type":48,"value":1356},{"type":42,"tag":342,"props":1392,"children":1393},{"class":344,"line":556},[1394,1398,1402,1406,1410],{"type":42,"tag":342,"props":1395,"children":1396},{"style":355},[1397],{"type":48,"value":1364},{"type":42,"tag":342,"props":1399,"children":1400},{"style":686},[1401],{"type":48,"value":1369},{"type":42,"tag":342,"props":1403,"children":1404},{"style":355},[1405],{"type":48,"value":358},{"type":42,"tag":342,"props":1407,"children":1408},{"style":361},[1409],{"type":48,"value":119},{"type":42,"tag":342,"props":1411,"children":1412},{"style":355},[1413],{"type":48,"value":1414},"'\n",{"type":42,"tag":51,"props":1416,"children":1417},{},[1418,1420,1425],{"type":48,"value":1419},"All three lines should show ",{"type":42,"tag":57,"props":1421,"children":1423},{"className":1422},[],[1424],{"type":48,"value":62},{"type":48,"value":1426},". If they don't, fix it before merge — otherwise GitLab CI keeps building on the old container and the next person hits the same trap.",{"type":42,"tag":82,"props":1428,"children":1430},{"id":1429},"file-touch-cheat-sheet",[1431],{"type":48,"value":1432},"File-touch cheat sheet",{"type":42,"tag":1135,"props":1434,"children":1435},{},[1436,1452],{"type":42,"tag":1139,"props":1437,"children":1438},{},[1439],{"type":42,"tag":1143,"props":1440,"children":1441},{},[1442,1447],{"type":42,"tag":1147,"props":1443,"children":1444},{},[1445],{"type":48,"value":1446},"Path",{"type":42,"tag":1147,"props":1448,"children":1449},{},[1450],{"type":48,"value":1451},"Edit",{"type":42,"tag":1158,"props":1453,"children":1454},{},[1455,1476,1507,1531,1562],{"type":42,"tag":1143,"props":1456,"children":1457},{},[1458,1466],{"type":42,"tag":1165,"props":1459,"children":1460},{},[1461],{"type":42,"tag":57,"props":1462,"children":1464},{"className":1463},[],[1465],{"type":48,"value":106},{"type":42,"tag":1165,"props":1467,"children":1468},{},[1469,1471],{"type":48,"value":1470},"Overwrite with new ",{"type":42,"tag":57,"props":1472,"children":1474},{"className":1473},[],[1475],{"type":48,"value":62},{"type":42,"tag":1143,"props":1477,"children":1478},{},[1479,1487],{"type":42,"tag":1165,"props":1480,"children":1481},{},[1482],{"type":42,"tag":57,"props":1483,"children":1485},{"className":1484},[],[1486],{"type":48,"value":127},{"type":42,"tag":1165,"props":1488,"children":1489},{},[1490,1492,1497,1499,1505],{"type":48,"value":1491},"Update both ",{"type":42,"tag":57,"props":1493,"children":1495},{"className":1494},[],[1496],{"type":48,"value":135},{"type":48,"value":1498}," ",{"type":42,"tag":57,"props":1500,"children":1502},{"className":1501},[],[1503],{"type":48,"value":1504},"BASE_IMAGE:",{"type":48,"value":1506}," rows (amd64 + arm64)",{"type":42,"tag":1143,"props":1508,"children":1509},{},[1510,1519],{"type":42,"tag":1165,"props":1511,"children":1512},{},[1513],{"type":42,"tag":57,"props":1514,"children":1516},{"className":1515},[],[1517],{"type":48,"value":1518},"tests\u002Ffunctional_tests\u002Ftest_cases\u002F**\u002Fgolden_values_dev_dgx_{h100,gb200}.json",{"type":42,"tag":1165,"props":1520,"children":1521},{},[1522,1524,1529],{"type":48,"value":1523},"Refresh via the ",{"type":42,"tag":57,"props":1525,"children":1527},{"className":1526},[],[1528],{"type":48,"value":859},{"type":48,"value":1530}," skill",{"type":42,"tag":1143,"props":1532,"children":1533},{},[1534,1543],{"type":42,"tag":1165,"props":1535,"children":1536},{},[1537],{"type":42,"tag":57,"props":1538,"children":1540},{"className":1539},[],[1541],{"type":48,"value":1542},"tests\u002Ftest_utils\u002Frecipes\u002F\u003Carch>\u002F\u003Csuite>.yaml",{"type":42,"tag":1165,"props":1544,"children":1545},{},[1546,1548,1553,1555,1560],{"type":48,"value":1547},"Flip drifting \u002F hanging cases to ",{"type":42,"tag":57,"props":1549,"children":1551},{"className":1550},[],[1552],{"type":48,"value":1082},{"type":48,"value":1554}," \u002F ",{"type":42,"tag":57,"props":1556,"children":1558},{"className":1557},[],[1559],{"type":48,"value":1202},{"type":48,"value":1561}," with an issue link",{"type":42,"tag":1143,"props":1563,"children":1564},{},[1565,1587],{"type":42,"tag":1165,"props":1566,"children":1567},{},[1568,1573,1574,1579,1580,1585],{"type":42,"tag":57,"props":1569,"children":1571},{"className":1570},[],[1572],{"type":48,"value":164},{"type":48,"value":818},{"type":42,"tag":57,"props":1575,"children":1577},{"className":1576},[],[1578],{"type":48,"value":127},{"type":48,"value":1498},{"type":42,"tag":57,"props":1581,"children":1583},{"className":1582},[],[1584],{"type":48,"value":268},{"type":48,"value":1586}," rows",{"type":42,"tag":1165,"props":1588,"children":1589},{},[1590,1595,1597,1603,1605,1611],{"type":42,"tag":66,"props":1591,"children":1592},{},[1593],{"type":48,"value":1594},"Skip unless explicitly bumping LTS.",{"type":48,"value":1596}," LTS has its own release cadence and its own Dockerfile (",{"type":42,"tag":57,"props":1598,"children":1600},{"className":1599},[],[1601],{"type":48,"value":1602},"docker\u002FDockerfile.ci.lts",{"type":48,"value":1604},"); LTS Python deps are pinned in ",{"type":42,"tag":57,"props":1606,"children":1608},{"className":1607},[],[1609],{"type":48,"value":1610},"docker\u002Flts\u002Frequirements.txt",{"type":48,"value":153},{"type":42,"tag":82,"props":1613,"children":1615},{"id":1614},"gotchas",[1616],{"type":48,"value":1617},"Gotchas",{"type":42,"tag":94,"props":1619,"children":1620},{},[1621,1659,1718,1743,1758,1782],{"type":42,"tag":98,"props":1622,"children":1623},{},[1624,1629,1630,1635,1637,1643,1645,1650,1652,1657],{"type":42,"tag":66,"props":1625,"children":1626},{},[1627],{"type":48,"value":1628},"GitHub vs GitLab pins are independent.",{"type":48,"value":1498},{"type":42,"tag":57,"props":1631,"children":1633},{"className":1632},[],[1634],{"type":48,"value":106},{"type":48,"value":1636}," only drives GitHub CI's local container build via ",{"type":42,"tag":57,"props":1638,"children":1640},{"className":1639},[],[1641],{"type":48,"value":1642},"Dockerfile.ci.dev",{"type":48,"value":1644},". GitLab CI has its own hardcoded ",{"type":42,"tag":57,"props":1646,"children":1648},{"className":1647},[],[1649],{"type":48,"value":1504},{"type":48,"value":1651}," matrix in ",{"type":42,"tag":57,"props":1653,"children":1655},{"className":1654},[],[1656],{"type":48,"value":127},{"type":48,"value":1658},". PR #4688 existed solely because #4611 forgot the second one — don't repeat this.",{"type":42,"tag":98,"props":1660,"children":1661},{},[1662,1667,1669,1674,1676,1681,1682,1687,1688,1693,1695,1701,1703,1708,1710,1716],{"type":42,"tag":66,"props":1663,"children":1664},{},[1665],{"type":48,"value":1666},"Don't bump LTS along with dev.",{"type":48,"value":1668}," The ",{"type":42,"tag":57,"props":1670,"children":1672},{"className":1671},[],[1673],{"type":48,"value":268},{"type":48,"value":1675}," rows, ",{"type":42,"tag":57,"props":1677,"children":1679},{"className":1678},[],[1680],{"type":48,"value":1602},{"type":48,"value":818},{"type":42,"tag":57,"props":1683,"children":1685},{"className":1684},[],[1686],{"type":48,"value":1610},{"type":48,"value":833},{"type":42,"tag":57,"props":1689,"children":1691},{"className":1690},[],[1692],{"type":48,"value":164},{"type":48,"value":1694}," are stability-pinned for the ",{"type":42,"tag":57,"props":1696,"children":1698},{"className":1697},[],[1699],{"type":48,"value":1700},"container::lts",{"type":48,"value":1702}," label path. Bump them in a dedicated PR with its own LTS validation. LTS Python deps are pinned in ",{"type":42,"tag":57,"props":1704,"children":1706},{"className":1705},[],[1707],{"type":48,"value":1610},{"type":48,"value":1709}," (not in ",{"type":42,"tag":57,"props":1711,"children":1713},{"className":1712},[],[1714],{"type":48,"value":1715},"pyproject.toml",{"type":48,"value":1717},") — edit that file when an LTS dependency needs to move.",{"type":42,"tag":98,"props":1719,"children":1720},{},[1721,1726,1728,1734,1736,1741],{"type":42,"tag":66,"props":1722,"children":1723},{},[1724],{"type":48,"value":1725},"Don't fix golden-value drift by hand.",{"type":48,"value":1727}," Use ",{"type":42,"tag":57,"props":1729,"children":1731},{"className":1730},[],[1732],{"type":48,"value":1733},"tests\u002Ftest_utils\u002Fpython_scripts\u002Fdownload_golden_values.py",{"type":48,"value":1735}," via the ",{"type":42,"tag":57,"props":1737,"children":1739},{"className":1738},[],[1740],{"type":48,"value":859},{"type":48,"value":1742}," skill. Hand-editing the JSONs invites diff noise and relative-difference regressions on subsequent bumps.",{"type":42,"tag":98,"props":1744,"children":1745},{},[1746,1756],{"type":42,"tag":66,"props":1747,"children":1748},{},[1749,1754],{"type":42,"tag":57,"props":1750,"children":1752},{"className":1751},[],[1753],{"type":48,"value":1082},{"type":48,"value":1755}," is a real scope, not a comment marker.",{"type":48,"value":1757}," It keeps the recipe wired into the matrix (so it stays discoverable and runnable on demand) without gating merges. Don't delete the test case from the recipe.",{"type":42,"tag":98,"props":1759,"children":1760},{},[1761,1772,1774,1780],{"type":42,"tag":66,"props":1762,"children":1763},{},[1764,1770],{"type":42,"tag":57,"props":1765,"children":1767},{"className":1766},[],[1768],{"type":48,"value":1769},"\u002Fok to test",{"type":48,"value":1771}," is per-commit.",{"type":48,"value":1773}," A new force-push or fixup commit needs a fresh ",{"type":42,"tag":57,"props":1775,"children":1777},{"className":1776},[],[1778],{"type":48,"value":1779},"\u002Fok to test \u003Csha>",{"type":48,"value":1781}," comment to re-trigger NVIDIA-runner CI on a fork PR.",{"type":42,"tag":98,"props":1783,"children":1784},{},[1785,1790],{"type":42,"tag":66,"props":1786,"children":1787},{},[1788],{"type":48,"value":1789},"Don't merge until the GitLab pin matches.",{"type":48,"value":1791}," Use the Step 7 grep before requesting review.",{"type":42,"tag":82,"props":1793,"children":1795},{"id":1794},"related-skills",[1796],{"type":48,"value":1797},"Related skills",{"type":42,"tag":94,"props":1799,"children":1800},{},[1801,1812,1831],{"type":42,"tag":98,"props":1802,"children":1803},{},[1804,1810],{"type":42,"tag":1805,"props":1806,"children":1808},"a",{"href":1807},"..\u002Fupdate-golden-values\u002FSKILL.md",[1809],{"type":48,"value":859},{"type":48,"value":1811}," — call this as soon as the first post-bump CI run finishes and you have a workflow run ID with failing golden checks. Produces the per-metric relative-difference summary you paste into the PR description.",{"type":42,"tag":98,"props":1813,"children":1814},{},[1815,1821,1823,1829],{"type":42,"tag":1805,"props":1816,"children":1818},{"href":1817},"..\u002Fbuild-and-dependency\u002FSKILL.md",[1819],{"type":48,"value":1820},"build-and-dependency",{"type":48,"value":1822}," — for verifying the new image builds locally before opening the PR (",{"type":42,"tag":57,"props":1824,"children":1826},{"className":1825},[],[1827],{"type":48,"value":1828},"docker build --target main --build-arg FROM_IMAGE_NAME=$(cat docker\u002F.ngc_version.dev) ...",{"type":48,"value":1830},").",{"type":42,"tag":98,"props":1832,"children":1833},{},[1834,1840,1842,1847,1848,1854,1856,1861],{"type":42,"tag":1805,"props":1835,"children":1837},{"href":1836},"..\u002Fcicd\u002FSKILL.md",[1838],{"type":48,"value":1839},"cicd",{"type":48,"value":1841}," — for the PR scope-label semantics (",{"type":42,"tag":57,"props":1843,"children":1845},{"className":1844},[],[1846],{"type":48,"value":756},{"type":48,"value":818},{"type":42,"tag":57,"props":1849,"children":1851},{"className":1850},[],[1852],{"type":48,"value":1853},"complexity::*",{"type":48,"value":1855},") and the ",{"type":42,"tag":57,"props":1857,"children":1859},{"className":1858},[],[1860],{"type":48,"value":790},{"type":48,"value":1862}," flow.",{"type":42,"tag":1864,"props":1865,"children":1866},"style",{},[1867],{"type":48,"value":1868},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1870,"total":1956},[1871,1885,1890,1902,1916,1929,1943],{"slug":1872,"name":1872,"fn":1873,"description":1874,"org":1875,"tags":1876,"stars":19,"repoUrl":20,"updatedAt":1884},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1877,1880,1881],{"name":1878,"slug":1879,"type":15},"Containers","containers",{"name":13,"slug":14,"type":15},{"name":1882,"slug":1883,"type":15},"Python","python","2026-07-27T06:06:11.249662",{"slug":4,"name":4,"fn":5,"description":6,"org":1886,"tags":1887,"stars":19,"repoUrl":20,"updatedAt":21},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1888,1889],{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":1891,"name":1891,"fn":1892,"description":1893,"org":1894,"tags":1895,"stars":19,"repoUrl":20,"updatedAt":1901},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1896,1897,1898],{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":1899,"slug":1900,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":1903,"name":1903,"fn":1904,"description":1905,"org":1906,"tags":1907,"stars":19,"repoUrl":20,"updatedAt":1915},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1908,1911,1912],{"name":1909,"slug":1910,"type":15},"Debugging","debugging",{"name":1899,"slug":1900,"type":15},{"name":1913,"slug":1914,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1917,"name":1917,"fn":1918,"description":1919,"org":1920,"tags":1921,"stars":19,"repoUrl":20,"updatedAt":1928},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1922,1925],{"name":1923,"slug":1924,"type":15},"Best Practices","best-practices",{"name":1926,"slug":1927,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1930,"name":1930,"fn":1931,"description":1932,"org":1933,"tags":1934,"stars":19,"repoUrl":20,"updatedAt":1942},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1935,1938,1941],{"name":1936,"slug":1937,"type":15},"Machine Learning","machine-learning",{"name":1939,"slug":1940,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":1944,"name":1944,"fn":1945,"description":1946,"org":1947,"tags":1948,"stars":19,"repoUrl":20,"updatedAt":1955},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1949,1952],{"name":1950,"slug":1951,"type":15},"QA","qa",{"name":1953,"slug":1954,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",13,{"items":1958,"total":2061},[1959,1977,1983,1988,1994,2000,2005,2011,2016,2027,2041,2050],{"slug":1960,"name":1960,"fn":1961,"description":1962,"org":1963,"tags":1964,"stars":1974,"repoUrl":1975,"updatedAt":1976},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1965,1968,1971],{"name":1966,"slug":1967,"type":15},"Documentation","documentation",{"name":1969,"slug":1970,"type":15},"MCP","mcp",{"name":1972,"slug":1973,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":1872,"name":1872,"fn":1873,"description":1874,"org":1978,"tags":1979,"stars":19,"repoUrl":20,"updatedAt":1884},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1980,1981,1982],{"name":1878,"slug":1879,"type":15},{"name":13,"slug":14,"type":15},{"name":1882,"slug":1883,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":1984,"tags":1985,"stars":19,"repoUrl":20,"updatedAt":21},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1986,1987],{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":1891,"name":1891,"fn":1892,"description":1893,"org":1989,"tags":1990,"stars":19,"repoUrl":20,"updatedAt":1901},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1991,1992,1993],{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":1899,"slug":1900,"type":15},{"slug":1903,"name":1903,"fn":1904,"description":1905,"org":1995,"tags":1996,"stars":19,"repoUrl":20,"updatedAt":1915},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1997,1998,1999],{"name":1909,"slug":1910,"type":15},{"name":1899,"slug":1900,"type":15},{"name":1913,"slug":1914,"type":15},{"slug":1917,"name":1917,"fn":1918,"description":1919,"org":2001,"tags":2002,"stars":19,"repoUrl":20,"updatedAt":1928},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2003,2004],{"name":1923,"slug":1924,"type":15},{"name":1926,"slug":1927,"type":15},{"slug":1930,"name":1930,"fn":1931,"description":1932,"org":2006,"tags":2007,"stars":19,"repoUrl":20,"updatedAt":1942},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2008,2009,2010],{"name":1936,"slug":1937,"type":15},{"name":1939,"slug":1940,"type":15},{"name":9,"slug":8,"type":15},{"slug":1944,"name":1944,"fn":1945,"description":1946,"org":2012,"tags":2013,"stars":19,"repoUrl":20,"updatedAt":1955},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2014,2015],{"name":1950,"slug":1951,"type":15},{"name":1953,"slug":1954,"type":15},{"slug":2017,"name":2017,"fn":2018,"description":2019,"org":2020,"tags":2021,"stars":19,"repoUrl":20,"updatedAt":2026},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2022,2023],{"name":13,"slug":14,"type":15},{"name":2024,"slug":2025,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":2028,"name":2028,"fn":2029,"description":2030,"org":2031,"tags":2032,"stars":19,"repoUrl":20,"updatedAt":2040},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2033,2036,2037],{"name":2034,"slug":2035,"type":15},"Code Review","code-review",{"name":1899,"slug":1900,"type":15},{"name":2038,"slug":2039,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2042,"name":2042,"fn":2043,"description":2044,"org":2045,"tags":2046,"stars":19,"repoUrl":20,"updatedAt":2049},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2047,2048],{"name":1950,"slug":1951,"type":15},{"name":1953,"slug":1954,"type":15},"2026-07-14T05:25:54.928983",{"slug":2051,"name":2051,"fn":2052,"description":2053,"org":2054,"tags":2055,"stars":19,"repoUrl":20,"updatedAt":2060},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2056,2059],{"name":2057,"slug":2058,"type":15},"Automation","automation",{"name":17,"slug":18,"type":15},"2026-07-30T05:29:03.275638",496]