[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-mcore-cicd":3,"mdc--oamd5w-key":36,"related-repo-nvidia-mcore-cicd":1984,"related-org-nvidia-mcore-cicd":2068},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":22,"repoUrl":23,"updatedAt":24,"license":25,"forks":26,"topics":27,"repo":31,"sourceUrl":34,"mdContent":35},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19],{"name":13,"slug":14,"type":15},"GitHub","github","tag",{"name":17,"slug":18,"type":15},"Deployment","deployment",{"name":20,"slug":21,"type":15},"CI\u002FCD","ci-cd",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:12.278222","Apache-2.0",4230,[28,29,30],"large-language-models","model-para","transformers",{"repoUrl":23,"stars":22,"forks":26,"topics":32,"description":33},[28,29,30],"Ongoing research training transformer models at scale","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM\u002Ftree\u002FHEAD\u002Fskills\u002Fmcore-cicd","---\nname: mcore-cicd\ndescription: CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.\nlicense: Apache-2.0\nwhen_to_use: Investigating a CI failure; understanding the pipeline structure; which CI label to attach; triggering internal GitLab CI; 'CI is red', 'how do I trigger CI', 'PR labels', 'where are the logs', 'pull-request branch'.\nmetadata:\n  author: Oliver Koenig \u003Cokoenig@nvidia.com>\n---\n\n# CI\u002FCD Guide\n\n---\n\n## Answer-First CI Facts\n\nFor PR-label or trigger questions, lead with the exact values:\n\n- No label: `scope=mr-github-slim`, `n_repeat=5`, `lightweight=false`.\n- `Run tests`: `scope=mr-github`, `n_repeat=1`, `lightweight=true`.\n- `Run functional tests`: `scope=mr-github`, `n_repeat=5`, `lightweight=false`.\n- `container::lts` only switches the container image path to LTS and combines\n  with any scope label. **Opt-in only — attach it solely when the user\n  explicitly asks for LTS validation; never add it on your own initiative, even\n  for a container or dependency change.**\n- `Run MBridge tests` additionally triggers the MBridge L1 suite.\n- ⚠️ **WARNING — destructive remote write.** `tools\u002Ftrigger_internal_ci.py`\n  **force-pushes the current branch** to the internal GitLab remote as\n  `pull-request\u002F\u003Cbranch>`. Always run with `--dry-run` first and confirm the\n  destination ref before invoking it without the flag. Never run against a\n  shared or protected branch — only target your own pull-request branch.\n  Safe preflight: `python tools\u002Ftrigger_internal_ci.py --gitlab-origin gitlab --dry-run`.\n  Add the optional `--functional-test-*` flags only after the dry-run output\n  matches the intended destination.\n\n---\n\n## CI Pipeline Structure\n\nThe main workflow is `.github\u002Fworkflows\u002Fcicd-main.yml`. It triggers on pushes\nto branches matching `pull-request\u002F[0-9]+` and `deploy-release\u002F*`, on merge\ngroups, on a daily schedule, and on manual dispatch.\n\n```text\nis-not-external-contributor\n  └─ pre-flight\n       └─ configure          # determines scope, container tag, n_repeat\n            ├─ linting\n            ├─ cicd-container-build\n            │    ├─ cicd-parse-unit-tests → cicd-unit-tests-latest\n            │    ├─ cicd-parse-integration-tests-h100 → cicd-integration-tests-latest-h100\n            │    └─ cicd-parse-integration-tests-gb200 → cicd-integration-tests-latest-gb200 (maintainers only)\n            └─ Nemo_CICD_Test  # final pass\u002Ffail gate\n```\n\nImages are pushed to:\n\n- AWS ECR: `766267172432.dkr.ecr.us-east-1.amazonaws.com\u002F…`\n- GCP Artifact Registry: `us-east4-docker.pkg.dev\u002Fnv-projdgxchipp-20260113193621\u002Fmegatron-lm\u002F…`\n\n---\n\n## CI Test Scope Labels\n\nThe CI pipeline reads PR labels to decide test scope, n_repeat, and container image.\n\n**Decision tree (first match wins):**\n\n| Condition | `scope` | `n_repeat` | `lightweight` | Notes |\n|-----------|---------|-----------|---------------|-------|\n| Merge group | `mr-github` | 1 | false | Automatic, no label needed |\n| Label: **`Run tests`** | `mr-github` | 1 | **true** | Trains 4 steps, no golden-value compare |\n| Label: **`Run functional tests`** | `mr-github` | 5 | **false** | Trains 100 steps, golden-value compare |\n| _(no label)_ | `mr-github-slim` | 5 | false | Slim subset only |\n\n**Orthogonal image label:**\n\n| Label | Effect |\n|-------|--------|\n| **`container::lts`** | Build on the older long-term-support NGC PyTorch base instead of `dev`'s latest — a backward-compat check, not a different test set (combinable with any scope label) |\n| **`Run MBridge tests`** | Also triggers the MBridge L1 test suite |\n\n### Which label to attach when opening a PR\n\n| Changed paths \u002F nature of change | Label to attach |\n|----------------------------------|-----------------|\n| Docs only (`docs\u002F`, `*.md`, docstrings) | _(none)_ |\n| CI\u002Ftooling only (`.github\u002F`, `tools\u002F`, `Makefile`) | _(none)_ |\n| Test files only (`tests\u002F`) — existing tests, no new golden values | `Run tests` |\n| **New test cases added** (no golden values exist yet) | `Run functional tests` |\n| **Re-enabling a disabled test** (scope `-broken` → active) | `Run functional tests` |\n| Non-numerical library code (logging, error handling, CLI flags, refactors) | `Run tests` |\n| Could affect training numerics (model arch, attention, optimizer, distributed, MoE routing) | `Run functional tests` |\n| Container or dependency changes (`docker\u002F`, `pyproject.toml`, `uv.lock`) | `Run tests` (add `container::lts` **only if the user explicitly asks** to validate LTS) |\n| Touches MBridge integration | add `Run MBridge tests` |\n\n**Rule of thumb:** default to `Run tests`. Always use `Run functional tests` when the PR adds new test cases (golden values must be generated) or when the change could plausibly shift loss curves.\n\n---\n\n## Triggering Internal CI\n\nUse `tools\u002Ftrigger_internal_ci.py` after the internal GitLab remote and\n`GITLAB_TOKEN` are configured; see @tools\u002Ftrigger_internal_ci.md for setup\ndetails. First run a dry run and verify the destination ref:\n\n```bash\npython tools\u002Ftrigger_internal_ci.py --gitlab-origin gitlab --dry-run\n```\n\nThe script force-pushes the current branch to `pull-request\u002F\u003Cbranch>` before\ntriggering the pipeline. Only target your own pull-request branch, never a shared\nor protected branch. Add optional `--functional-test-*` flags only after the\ndry-run output matches the intended destination.\n\n---\n\n## CI Failure Investigation\n\nCI branches always follow the pattern `pull-request\u002F\u003Cnumber>`.\n\n### Locating the PR from a CI Branch\n\n```bash\n# Extract PR number from the current branch\nPR_NUMBER=$(git rev-parse --abbrev-ref HEAD | grep -oP '(?\u003C=pull-request\u002F)\\d+')\n\n# Fetch the PR metadata (title, labels, author, base branch)\ngh pr view \"$PR_NUMBER\" --repo NVIDIA\u002FMegatron-LM\n\n# Show the changeset for that PR\ngh pr diff \"$PR_NUMBER\" --repo NVIDIA\u002FMegatron-LM\n```\n\n### Reading CI Job Logs\n\n```bash\n# List recent workflow runs for the PR\ngh run list --repo NVIDIA\u002FMegatron-LM --branch \"pull-request\u002F$PR_NUMBER\"\n\n# Stream failing job output\ngh run view \u003Crun-id> --repo NVIDIA\u002FMegatron-LM --log-failed\n```\n\nFull per-rank logs are **not** in the runner stdout. They are uploaded as\nGitHub artifacts named `logs-\u003Ctest_case>-\u003Crun_id>-\u003Cuuid>`.\n\n```bash\n# 1. Find artifact name\ngh run view \u003Crun-id> --repo NVIDIA\u002FMegatron-LM --json artifacts \\\n  --jq '.artifacts[].name'\n\n# 2. Download the artifact zip\ngh run download \u003Crun-id> --repo NVIDIA\u002FMegatron-LM \\\n  --name \"logs-\u003Cartifact-name>\" -D .\u002Fci-logs\n\n# 3. Locate which rank logs contain errors\ngrep -r -l \"ERROR\\|Traceback\\|FAILED\\|fatal\" .\u002Fci-logs\u002F\n\n# 4. Log files can exceed 10 000 lines — never read a full log at once.\nwc -l .\u002Fci-logs\u002F\u003Ctest>\u002F\u003Cattempt>\u002Fattempt_0\u002F\u003Crank>\u002Fstderr.log\nsed -n '1,200p' .\u002Fci-logs\u002F...\u002Fstderr.log   # read in chunks\n```\n\n### Identifying Failure Root Cause\n\n1. **Linting failure** — re-run `tools\u002Fautoformat.sh` locally; the diff shows exactly what needs to change.\n2. **Container build failure** — inspect the `cicd-container-build` job log.\n3. **Unit test failure** — the failing bucket is in the `cicd-unit-tests-latest` job matrix.\n4. **Functional test failure** — look at the `cicd-integration-tests-*` job. Start with `stdout.log` for rank 0.\n5. **Flaky test** — the runner retries automatically up to 3 times. If all retries exhausted and the pattern matches a known transient (NCCL, ECC, segfault), it is infrastructure noise.\n\n### Correlating a Failure with the PR Changeset\n\n```bash\n# Find unit tests that cover a changed source file\ngrep -r \"from megatron.core.transformer.attention\" tests\u002Funit_tests\u002F -l\n\n# Check CODEOWNERS for reviewer assignment\ncat .github\u002FCODEOWNERS | grep \"\u003Cchanged-path>\"\n```\n",{"data":37,"body":41},{"name":4,"description":6,"license":25,"when_to_use":38,"metadata":39},"Investigating a CI failure; understanding the pipeline structure; which CI label to attach; triggering internal GitLab CI; 'CI is red', 'how do I trigger CI', 'PR labels', 'where are the logs', 'pull-request branch'.",{"author":40},"Oliver Koenig \u003Cokoenig@nvidia.com>",{"type":42,"children":43},"root",[44,53,57,64,70,249,252,258,287,299,304,329,332,338,343,351,553,561,629,636,902,926,929,935,955,997,1016,1019,1025,1037,1043,1247,1253,1380,1399,1774,1780,1874,1880,1978],{"type":45,"tag":46,"props":47,"children":49},"element","h1",{"id":48},"cicd-guide",[50],{"type":51,"value":52},"text","CI\u002FCD Guide",{"type":45,"tag":54,"props":55,"children":56},"hr",{},[],{"type":45,"tag":58,"props":59,"children":61},"h2",{"id":60},"answer-first-ci-facts",[62],{"type":51,"value":63},"Answer-First CI Facts",{"type":45,"tag":65,"props":66,"children":67},"p",{},[68],{"type":51,"value":69},"For PR-label or trigger questions, lead with the exact values:",{"type":45,"tag":71,"props":72,"children":73},"ul",{},[74,104,136,164,181,192],{"type":45,"tag":75,"props":76,"children":77},"li",{},[78,80,87,89,95,96,102],{"type":51,"value":79},"No label: ",{"type":45,"tag":81,"props":82,"children":84},"code",{"className":83},[],[85],{"type":51,"value":86},"scope=mr-github-slim",{"type":51,"value":88},", ",{"type":45,"tag":81,"props":90,"children":92},{"className":91},[],[93],{"type":51,"value":94},"n_repeat=5",{"type":51,"value":88},{"type":45,"tag":81,"props":97,"children":99},{"className":98},[],[100],{"type":51,"value":101},"lightweight=false",{"type":51,"value":103},".",{"type":45,"tag":75,"props":105,"children":106},{},[107,113,115,121,122,128,129,135],{"type":45,"tag":81,"props":108,"children":110},{"className":109},[],[111],{"type":51,"value":112},"Run tests",{"type":51,"value":114},": ",{"type":45,"tag":81,"props":116,"children":118},{"className":117},[],[119],{"type":51,"value":120},"scope=mr-github",{"type":51,"value":88},{"type":45,"tag":81,"props":123,"children":125},{"className":124},[],[126],{"type":51,"value":127},"n_repeat=1",{"type":51,"value":88},{"type":45,"tag":81,"props":130,"children":132},{"className":131},[],[133],{"type":51,"value":134},"lightweight=true",{"type":51,"value":103},{"type":45,"tag":75,"props":137,"children":138},{},[139,145,146,151,152,157,158,163],{"type":45,"tag":81,"props":140,"children":142},{"className":141},[],[143],{"type":51,"value":144},"Run functional tests",{"type":51,"value":114},{"type":45,"tag":81,"props":147,"children":149},{"className":148},[],[150],{"type":51,"value":120},{"type":51,"value":88},{"type":45,"tag":81,"props":153,"children":155},{"className":154},[],[156],{"type":51,"value":94},{"type":51,"value":88},{"type":45,"tag":81,"props":159,"children":161},{"className":160},[],[162],{"type":51,"value":101},{"type":51,"value":103},{"type":45,"tag":75,"props":165,"children":166},{},[167,173,175],{"type":45,"tag":81,"props":168,"children":170},{"className":169},[],[171],{"type":51,"value":172},"container::lts",{"type":51,"value":174}," only switches the container image path to LTS and combines\nwith any scope label. ",{"type":45,"tag":176,"props":177,"children":178},"strong",{},[179],{"type":51,"value":180},"Opt-in only — attach it solely when the user\nexplicitly asks for LTS validation; never add it on your own initiative, even\nfor a container or dependency change.",{"type":45,"tag":75,"props":182,"children":183},{},[184,190],{"type":45,"tag":81,"props":185,"children":187},{"className":186},[],[188],{"type":51,"value":189},"Run MBridge tests",{"type":51,"value":191}," additionally triggers the MBridge L1 suite.",{"type":45,"tag":75,"props":193,"children":194},{},[195,197,202,204,210,215,217,223,225,231,233,239,241,247],{"type":51,"value":196},"⚠️ ",{"type":45,"tag":176,"props":198,"children":199},{},[200],{"type":51,"value":201},"WARNING — destructive remote write.",{"type":51,"value":203}," ",{"type":45,"tag":81,"props":205,"children":207},{"className":206},[],[208],{"type":51,"value":209},"tools\u002Ftrigger_internal_ci.py",{"type":45,"tag":176,"props":211,"children":212},{},[213],{"type":51,"value":214},"force-pushes the current branch",{"type":51,"value":216}," to the internal GitLab remote as\n",{"type":45,"tag":81,"props":218,"children":220},{"className":219},[],[221],{"type":51,"value":222},"pull-request\u002F\u003Cbranch>",{"type":51,"value":224},". Always run with ",{"type":45,"tag":81,"props":226,"children":228},{"className":227},[],[229],{"type":51,"value":230},"--dry-run",{"type":51,"value":232}," first and confirm the\ndestination ref before invoking it without the flag. Never run against a\nshared or protected branch — only target your own pull-request branch.\nSafe preflight: ",{"type":45,"tag":81,"props":234,"children":236},{"className":235},[],[237],{"type":51,"value":238},"python tools\u002Ftrigger_internal_ci.py --gitlab-origin gitlab --dry-run",{"type":51,"value":240},".\nAdd the optional ",{"type":45,"tag":81,"props":242,"children":244},{"className":243},[],[245],{"type":51,"value":246},"--functional-test-*",{"type":51,"value":248}," flags only after the dry-run output\nmatches the intended destination.",{"type":45,"tag":54,"props":250,"children":251},{},[],{"type":45,"tag":58,"props":253,"children":255},{"id":254},"ci-pipeline-structure",[256],{"type":51,"value":257},"CI Pipeline Structure",{"type":45,"tag":65,"props":259,"children":260},{},[261,263,269,271,277,279,285],{"type":51,"value":262},"The main workflow is ",{"type":45,"tag":81,"props":264,"children":266},{"className":265},[],[267],{"type":51,"value":268},".github\u002Fworkflows\u002Fcicd-main.yml",{"type":51,"value":270},". It triggers on pushes\nto branches matching ",{"type":45,"tag":81,"props":272,"children":274},{"className":273},[],[275],{"type":51,"value":276},"pull-request\u002F[0-9]+",{"type":51,"value":278}," and ",{"type":45,"tag":81,"props":280,"children":282},{"className":281},[],[283],{"type":51,"value":284},"deploy-release\u002F*",{"type":51,"value":286},", on merge\ngroups, on a daily schedule, and on manual dispatch.",{"type":45,"tag":288,"props":289,"children":294},"pre",{"className":290,"code":292,"language":51,"meta":293},[291],"language-text","is-not-external-contributor\n  └─ pre-flight\n       └─ configure          # determines scope, container tag, n_repeat\n            ├─ linting\n            ├─ cicd-container-build\n            │    ├─ cicd-parse-unit-tests → cicd-unit-tests-latest\n            │    ├─ cicd-parse-integration-tests-h100 → cicd-integration-tests-latest-h100\n            │    └─ cicd-parse-integration-tests-gb200 → cicd-integration-tests-latest-gb200 (maintainers only)\n            └─ Nemo_CICD_Test  # final pass\u002Ffail gate\n","",[295],{"type":45,"tag":81,"props":296,"children":297},{"__ignoreMap":293},[298],{"type":51,"value":292},{"type":45,"tag":65,"props":300,"children":301},{},[302],{"type":51,"value":303},"Images are pushed to:",{"type":45,"tag":71,"props":305,"children":306},{},[307,318],{"type":45,"tag":75,"props":308,"children":309},{},[310,312],{"type":51,"value":311},"AWS ECR: ",{"type":45,"tag":81,"props":313,"children":315},{"className":314},[],[316],{"type":51,"value":317},"766267172432.dkr.ecr.us-east-1.amazonaws.com\u002F…",{"type":45,"tag":75,"props":319,"children":320},{},[321,323],{"type":51,"value":322},"GCP Artifact Registry: ",{"type":45,"tag":81,"props":324,"children":326},{"className":325},[],[327],{"type":51,"value":328},"us-east4-docker.pkg.dev\u002Fnv-projdgxchipp-20260113193621\u002Fmegatron-lm\u002F…",{"type":45,"tag":54,"props":330,"children":331},{},[],{"type":45,"tag":58,"props":333,"children":335},{"id":334},"ci-test-scope-labels",[336],{"type":51,"value":337},"CI Test Scope Labels",{"type":45,"tag":65,"props":339,"children":340},{},[341],{"type":51,"value":342},"The CI pipeline reads PR labels to decide test scope, n_repeat, and container image.",{"type":45,"tag":65,"props":344,"children":345},{},[346],{"type":45,"tag":176,"props":347,"children":348},{},[349],{"type":51,"value":350},"Decision tree (first match wins):",{"type":45,"tag":352,"props":353,"children":354},"table",{},[355,401],{"type":45,"tag":356,"props":357,"children":358},"thead",{},[359],{"type":45,"tag":360,"props":361,"children":362},"tr",{},[363,369,378,387,396],{"type":45,"tag":364,"props":365,"children":366},"th",{},[367],{"type":51,"value":368},"Condition",{"type":45,"tag":364,"props":370,"children":371},{},[372],{"type":45,"tag":81,"props":373,"children":375},{"className":374},[],[376],{"type":51,"value":377},"scope",{"type":45,"tag":364,"props":379,"children":380},{},[381],{"type":45,"tag":81,"props":382,"children":384},{"className":383},[],[385],{"type":51,"value":386},"n_repeat",{"type":45,"tag":364,"props":388,"children":389},{},[390],{"type":45,"tag":81,"props":391,"children":393},{"className":392},[],[394],{"type":51,"value":395},"lightweight",{"type":45,"tag":364,"props":397,"children":398},{},[399],{"type":51,"value":400},"Notes",{"type":45,"tag":402,"props":403,"children":404},"tbody",{},[405,438,479,519],{"type":45,"tag":360,"props":406,"children":407},{},[408,414,423,428,433],{"type":45,"tag":409,"props":410,"children":411},"td",{},[412],{"type":51,"value":413},"Merge group",{"type":45,"tag":409,"props":415,"children":416},{},[417],{"type":45,"tag":81,"props":418,"children":420},{"className":419},[],[421],{"type":51,"value":422},"mr-github",{"type":45,"tag":409,"props":424,"children":425},{},[426],{"type":51,"value":427},"1",{"type":45,"tag":409,"props":429,"children":430},{},[431],{"type":51,"value":432},"false",{"type":45,"tag":409,"props":434,"children":435},{},[436],{"type":51,"value":437},"Automatic, no label needed",{"type":45,"tag":360,"props":439,"children":440},{},[441,454,462,466,474],{"type":45,"tag":409,"props":442,"children":443},{},[444,446],{"type":51,"value":445},"Label: ",{"type":45,"tag":176,"props":447,"children":448},{},[449],{"type":45,"tag":81,"props":450,"children":452},{"className":451},[],[453],{"type":51,"value":112},{"type":45,"tag":409,"props":455,"children":456},{},[457],{"type":45,"tag":81,"props":458,"children":460},{"className":459},[],[461],{"type":51,"value":422},{"type":45,"tag":409,"props":463,"children":464},{},[465],{"type":51,"value":427},{"type":45,"tag":409,"props":467,"children":468},{},[469],{"type":45,"tag":176,"props":470,"children":471},{},[472],{"type":51,"value":473},"true",{"type":45,"tag":409,"props":475,"children":476},{},[477],{"type":51,"value":478},"Trains 4 steps, no golden-value compare",{"type":45,"tag":360,"props":480,"children":481},{},[482,494,502,507,514],{"type":45,"tag":409,"props":483,"children":484},{},[485,486],{"type":51,"value":445},{"type":45,"tag":176,"props":487,"children":488},{},[489],{"type":45,"tag":81,"props":490,"children":492},{"className":491},[],[493],{"type":51,"value":144},{"type":45,"tag":409,"props":495,"children":496},{},[497],{"type":45,"tag":81,"props":498,"children":500},{"className":499},[],[501],{"type":51,"value":422},{"type":45,"tag":409,"props":503,"children":504},{},[505],{"type":51,"value":506},"5",{"type":45,"tag":409,"props":508,"children":509},{},[510],{"type":45,"tag":176,"props":511,"children":512},{},[513],{"type":51,"value":432},{"type":45,"tag":409,"props":515,"children":516},{},[517],{"type":51,"value":518},"Trains 100 steps, golden-value compare",{"type":45,"tag":360,"props":520,"children":521},{},[522,531,540,544,548],{"type":45,"tag":409,"props":523,"children":524},{},[525],{"type":45,"tag":526,"props":527,"children":528},"em",{},[529],{"type":51,"value":530},"(no label)",{"type":45,"tag":409,"props":532,"children":533},{},[534],{"type":45,"tag":81,"props":535,"children":537},{"className":536},[],[538],{"type":51,"value":539},"mr-github-slim",{"type":45,"tag":409,"props":541,"children":542},{},[543],{"type":51,"value":506},{"type":45,"tag":409,"props":545,"children":546},{},[547],{"type":51,"value":432},{"type":45,"tag":409,"props":549,"children":550},{},[551],{"type":51,"value":552},"Slim subset only",{"type":45,"tag":65,"props":554,"children":555},{},[556],{"type":45,"tag":176,"props":557,"children":558},{},[559],{"type":51,"value":560},"Orthogonal image label:",{"type":45,"tag":352,"props":562,"children":563},{},[564,580],{"type":45,"tag":356,"props":565,"children":566},{},[567],{"type":45,"tag":360,"props":568,"children":569},{},[570,575],{"type":45,"tag":364,"props":571,"children":572},{},[573],{"type":51,"value":574},"Label",{"type":45,"tag":364,"props":576,"children":577},{},[578],{"type":51,"value":579},"Effect",{"type":45,"tag":402,"props":581,"children":582},{},[583,610],{"type":45,"tag":360,"props":584,"children":585},{},[586,597],{"type":45,"tag":409,"props":587,"children":588},{},[589],{"type":45,"tag":176,"props":590,"children":591},{},[592],{"type":45,"tag":81,"props":593,"children":595},{"className":594},[],[596],{"type":51,"value":172},{"type":45,"tag":409,"props":598,"children":599},{},[600,602,608],{"type":51,"value":601},"Build on the older long-term-support NGC PyTorch base instead of ",{"type":45,"tag":81,"props":603,"children":605},{"className":604},[],[606],{"type":51,"value":607},"dev",{"type":51,"value":609},"'s latest — a backward-compat check, not a different test set (combinable with any scope label)",{"type":45,"tag":360,"props":611,"children":612},{},[613,624],{"type":45,"tag":409,"props":614,"children":615},{},[616],{"type":45,"tag":176,"props":617,"children":618},{},[619],{"type":45,"tag":81,"props":620,"children":622},{"className":621},[],[623],{"type":51,"value":189},{"type":45,"tag":409,"props":625,"children":626},{},[627],{"type":51,"value":628},"Also triggers the MBridge L1 test suite",{"type":45,"tag":630,"props":631,"children":633},"h3",{"id":632},"which-label-to-attach-when-opening-a-pr",[634],{"type":51,"value":635},"Which label to attach when opening a PR",{"type":45,"tag":352,"props":637,"children":638},{},[639,655],{"type":45,"tag":356,"props":640,"children":641},{},[642],{"type":45,"tag":360,"props":643,"children":644},{},[645,650],{"type":45,"tag":364,"props":646,"children":647},{},[648],{"type":51,"value":649},"Changed paths \u002F nature of change",{"type":45,"tag":364,"props":651,"children":652},{},[653],{"type":51,"value":654},"Label to attach",{"type":45,"tag":402,"props":656,"children":657},{},[658,689,726,750,771,800,816,832,884],{"type":45,"tag":360,"props":659,"children":660},{},[661,681],{"type":45,"tag":409,"props":662,"children":663},{},[664,666,672,673,679],{"type":51,"value":665},"Docs only (",{"type":45,"tag":81,"props":667,"children":669},{"className":668},[],[670],{"type":51,"value":671},"docs\u002F",{"type":51,"value":88},{"type":45,"tag":81,"props":674,"children":676},{"className":675},[],[677],{"type":51,"value":678},"*.md",{"type":51,"value":680},", docstrings)",{"type":45,"tag":409,"props":682,"children":683},{},[684],{"type":45,"tag":526,"props":685,"children":686},{},[687],{"type":51,"value":688},"(none)",{"type":45,"tag":360,"props":690,"children":691},{},[692,719],{"type":45,"tag":409,"props":693,"children":694},{},[695,697,703,704,710,711,717],{"type":51,"value":696},"CI\u002Ftooling only (",{"type":45,"tag":81,"props":698,"children":700},{"className":699},[],[701],{"type":51,"value":702},".github\u002F",{"type":51,"value":88},{"type":45,"tag":81,"props":705,"children":707},{"className":706},[],[708],{"type":51,"value":709},"tools\u002F",{"type":51,"value":88},{"type":45,"tag":81,"props":712,"children":714},{"className":713},[],[715],{"type":51,"value":716},"Makefile",{"type":51,"value":718},")",{"type":45,"tag":409,"props":720,"children":721},{},[722],{"type":45,"tag":526,"props":723,"children":724},{},[725],{"type":51,"value":688},{"type":45,"tag":360,"props":727,"children":728},{},[729,742],{"type":45,"tag":409,"props":730,"children":731},{},[732,734,740],{"type":51,"value":733},"Test files only (",{"type":45,"tag":81,"props":735,"children":737},{"className":736},[],[738],{"type":51,"value":739},"tests\u002F",{"type":51,"value":741},") — existing tests, no new golden values",{"type":45,"tag":409,"props":743,"children":744},{},[745],{"type":45,"tag":81,"props":746,"children":748},{"className":747},[],[749],{"type":51,"value":112},{"type":45,"tag":360,"props":751,"children":752},{},[753,763],{"type":45,"tag":409,"props":754,"children":755},{},[756,761],{"type":45,"tag":176,"props":757,"children":758},{},[759],{"type":51,"value":760},"New test cases added",{"type":51,"value":762}," (no golden values exist yet)",{"type":45,"tag":409,"props":764,"children":765},{},[766],{"type":45,"tag":81,"props":767,"children":769},{"className":768},[],[770],{"type":51,"value":144},{"type":45,"tag":360,"props":772,"children":773},{},[774,792],{"type":45,"tag":409,"props":775,"children":776},{},[777,782,784,790],{"type":45,"tag":176,"props":778,"children":779},{},[780],{"type":51,"value":781},"Re-enabling a disabled test",{"type":51,"value":783}," (scope ",{"type":45,"tag":81,"props":785,"children":787},{"className":786},[],[788],{"type":51,"value":789},"-broken",{"type":51,"value":791}," → active)",{"type":45,"tag":409,"props":793,"children":794},{},[795],{"type":45,"tag":81,"props":796,"children":798},{"className":797},[],[799],{"type":51,"value":144},{"type":45,"tag":360,"props":801,"children":802},{},[803,808],{"type":45,"tag":409,"props":804,"children":805},{},[806],{"type":51,"value":807},"Non-numerical library code (logging, error handling, CLI flags, refactors)",{"type":45,"tag":409,"props":809,"children":810},{},[811],{"type":45,"tag":81,"props":812,"children":814},{"className":813},[],[815],{"type":51,"value":112},{"type":45,"tag":360,"props":817,"children":818},{},[819,824],{"type":45,"tag":409,"props":820,"children":821},{},[822],{"type":51,"value":823},"Could affect training numerics (model arch, attention, optimizer, distributed, MoE routing)",{"type":45,"tag":409,"props":825,"children":826},{},[827],{"type":45,"tag":81,"props":828,"children":830},{"className":829},[],[831],{"type":51,"value":144},{"type":45,"tag":360,"props":833,"children":834},{},[835,861],{"type":45,"tag":409,"props":836,"children":837},{},[838,840,846,847,853,854,860],{"type":51,"value":839},"Container or dependency changes (",{"type":45,"tag":81,"props":841,"children":843},{"className":842},[],[844],{"type":51,"value":845},"docker\u002F",{"type":51,"value":88},{"type":45,"tag":81,"props":848,"children":850},{"className":849},[],[851],{"type":51,"value":852},"pyproject.toml",{"type":51,"value":88},{"type":45,"tag":81,"props":855,"children":857},{"className":856},[],[858],{"type":51,"value":859},"uv.lock",{"type":51,"value":718},{"type":45,"tag":409,"props":862,"children":863},{},[864,869,871,876,877,882],{"type":45,"tag":81,"props":865,"children":867},{"className":866},[],[868],{"type":51,"value":112},{"type":51,"value":870}," (add ",{"type":45,"tag":81,"props":872,"children":874},{"className":873},[],[875],{"type":51,"value":172},{"type":51,"value":203},{"type":45,"tag":176,"props":878,"children":879},{},[880],{"type":51,"value":881},"only if the user explicitly asks",{"type":51,"value":883}," to validate LTS)",{"type":45,"tag":360,"props":885,"children":886},{},[887,892],{"type":45,"tag":409,"props":888,"children":889},{},[890],{"type":51,"value":891},"Touches MBridge integration",{"type":45,"tag":409,"props":893,"children":894},{},[895,897],{"type":51,"value":896},"add ",{"type":45,"tag":81,"props":898,"children":900},{"className":899},[],[901],{"type":51,"value":189},{"type":45,"tag":65,"props":903,"children":904},{},[905,910,912,917,919,924],{"type":45,"tag":176,"props":906,"children":907},{},[908],{"type":51,"value":909},"Rule of thumb:",{"type":51,"value":911}," default to ",{"type":45,"tag":81,"props":913,"children":915},{"className":914},[],[916],{"type":51,"value":112},{"type":51,"value":918},". Always use ",{"type":45,"tag":81,"props":920,"children":922},{"className":921},[],[923],{"type":51,"value":144},{"type":51,"value":925}," when the PR adds new test cases (golden values must be generated) or when the change could plausibly shift loss curves.",{"type":45,"tag":54,"props":927,"children":928},{},[],{"type":45,"tag":58,"props":930,"children":932},{"id":931},"triggering-internal-ci",[933],{"type":51,"value":934},"Triggering Internal CI",{"type":45,"tag":65,"props":936,"children":937},{},[938,940,945,947,953],{"type":51,"value":939},"Use ",{"type":45,"tag":81,"props":941,"children":943},{"className":942},[],[944],{"type":51,"value":209},{"type":51,"value":946}," after the internal GitLab remote and\n",{"type":45,"tag":81,"props":948,"children":950},{"className":949},[],[951],{"type":51,"value":952},"GITLAB_TOKEN",{"type":51,"value":954}," are configured; see @tools\u002Ftrigger_internal_ci.md for setup\ndetails. First run a dry run and verify the destination ref:",{"type":45,"tag":288,"props":956,"children":960},{"className":957,"code":958,"language":959,"meta":293,"style":293},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","python tools\u002Ftrigger_internal_ci.py --gitlab-origin gitlab --dry-run\n","bash",[961],{"type":45,"tag":81,"props":962,"children":963},{"__ignoreMap":293},[964],{"type":45,"tag":965,"props":966,"children":969},"span",{"class":967,"line":968},"line",1,[970,976,982,987,992],{"type":45,"tag":965,"props":971,"children":973},{"style":972},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[974],{"type":51,"value":975},"python",{"type":45,"tag":965,"props":977,"children":979},{"style":978},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[980],{"type":51,"value":981}," tools\u002Ftrigger_internal_ci.py",{"type":45,"tag":965,"props":983,"children":984},{"style":978},[985],{"type":51,"value":986}," --gitlab-origin",{"type":45,"tag":965,"props":988,"children":989},{"style":978},[990],{"type":51,"value":991}," gitlab",{"type":45,"tag":965,"props":993,"children":994},{"style":978},[995],{"type":51,"value":996}," --dry-run\n",{"type":45,"tag":65,"props":998,"children":999},{},[1000,1002,1007,1009,1014],{"type":51,"value":1001},"The script force-pushes the current branch to ",{"type":45,"tag":81,"props":1003,"children":1005},{"className":1004},[],[1006],{"type":51,"value":222},{"type":51,"value":1008}," before\ntriggering the pipeline. Only target your own pull-request branch, never a shared\nor protected branch. Add optional ",{"type":45,"tag":81,"props":1010,"children":1012},{"className":1011},[],[1013],{"type":51,"value":246},{"type":51,"value":1015}," flags only after the\ndry-run output matches the intended destination.",{"type":45,"tag":54,"props":1017,"children":1018},{},[],{"type":45,"tag":58,"props":1020,"children":1022},{"id":1021},"ci-failure-investigation",[1023],{"type":51,"value":1024},"CI Failure Investigation",{"type":45,"tag":65,"props":1026,"children":1027},{},[1028,1030,1036],{"type":51,"value":1029},"CI branches always follow the pattern ",{"type":45,"tag":81,"props":1031,"children":1033},{"className":1032},[],[1034],{"type":51,"value":1035},"pull-request\u002F\u003Cnumber>",{"type":51,"value":103},{"type":45,"tag":630,"props":1038,"children":1040},{"id":1039},"locating-the-pr-from-a-ci-branch",[1041],{"type":51,"value":1042},"Locating the PR from a CI Branch",{"type":45,"tag":288,"props":1044,"children":1046},{"className":957,"code":1045,"language":959,"meta":293,"style":293},"# Extract PR number from the current branch\nPR_NUMBER=$(git rev-parse --abbrev-ref HEAD | grep -oP '(?\u003C=pull-request\u002F)\\d+')\n\n# Fetch the PR metadata (title, labels, author, base branch)\ngh pr view \"$PR_NUMBER\" --repo NVIDIA\u002FMegatron-LM\n\n# Show the changeset for that PR\ngh pr diff \"$PR_NUMBER\" --repo NVIDIA\u002FMegatron-LM\n",[1047],{"type":45,"tag":81,"props":1048,"children":1049},{"__ignoreMap":293},[1050,1059,1130,1140,1149,1193,1201,1210],{"type":45,"tag":965,"props":1051,"children":1052},{"class":967,"line":968},[1053],{"type":45,"tag":965,"props":1054,"children":1056},{"style":1055},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1057],{"type":51,"value":1058},"# Extract PR number from the current branch\n",{"type":45,"tag":965,"props":1060,"children":1062},{"class":967,"line":1061},2,[1063,1069,1075,1080,1085,1090,1095,1100,1105,1110,1115,1120,1125],{"type":45,"tag":965,"props":1064,"children":1066},{"style":1065},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1067],{"type":51,"value":1068},"PR_NUMBER",{"type":45,"tag":965,"props":1070,"children":1072},{"style":1071},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1073],{"type":51,"value":1074},"=$(",{"type":45,"tag":965,"props":1076,"children":1077},{"style":972},[1078],{"type":51,"value":1079},"git",{"type":45,"tag":965,"props":1081,"children":1082},{"style":978},[1083],{"type":51,"value":1084}," rev-parse",{"type":45,"tag":965,"props":1086,"children":1087},{"style":978},[1088],{"type":51,"value":1089}," --abbrev-ref",{"type":45,"tag":965,"props":1091,"children":1092},{"style":978},[1093],{"type":51,"value":1094}," HEAD",{"type":45,"tag":965,"props":1096,"children":1097},{"style":1071},[1098],{"type":51,"value":1099}," |",{"type":45,"tag":965,"props":1101,"children":1102},{"style":972},[1103],{"type":51,"value":1104}," grep",{"type":45,"tag":965,"props":1106,"children":1107},{"style":978},[1108],{"type":51,"value":1109}," -oP",{"type":45,"tag":965,"props":1111,"children":1112},{"style":1071},[1113],{"type":51,"value":1114}," '",{"type":45,"tag":965,"props":1116,"children":1117},{"style":978},[1118],{"type":51,"value":1119},"(?\u003C=pull-request\u002F)\\d+",{"type":45,"tag":965,"props":1121,"children":1122},{"style":1071},[1123],{"type":51,"value":1124},"'",{"type":45,"tag":965,"props":1126,"children":1127},{"style":1071},[1128],{"type":51,"value":1129},")\n",{"type":45,"tag":965,"props":1131,"children":1133},{"class":967,"line":1132},3,[1134],{"type":45,"tag":965,"props":1135,"children":1137},{"emptyLinePlaceholder":1136},true,[1138],{"type":51,"value":1139},"\n",{"type":45,"tag":965,"props":1141,"children":1143},{"class":967,"line":1142},4,[1144],{"type":45,"tag":965,"props":1145,"children":1146},{"style":1055},[1147],{"type":51,"value":1148},"# Fetch the PR metadata (title, labels, author, base branch)\n",{"type":45,"tag":965,"props":1150,"children":1152},{"class":967,"line":1151},5,[1153,1158,1163,1168,1173,1178,1183,1188],{"type":45,"tag":965,"props":1154,"children":1155},{"style":972},[1156],{"type":51,"value":1157},"gh",{"type":45,"tag":965,"props":1159,"children":1160},{"style":978},[1161],{"type":51,"value":1162}," pr",{"type":45,"tag":965,"props":1164,"children":1165},{"style":978},[1166],{"type":51,"value":1167}," view",{"type":45,"tag":965,"props":1169,"children":1170},{"style":1071},[1171],{"type":51,"value":1172}," \"",{"type":45,"tag":965,"props":1174,"children":1175},{"style":1065},[1176],{"type":51,"value":1177},"$PR_NUMBER",{"type":45,"tag":965,"props":1179,"children":1180},{"style":1071},[1181],{"type":51,"value":1182},"\"",{"type":45,"tag":965,"props":1184,"children":1185},{"style":978},[1186],{"type":51,"value":1187}," --repo",{"type":45,"tag":965,"props":1189,"children":1190},{"style":978},[1191],{"type":51,"value":1192}," NVIDIA\u002FMegatron-LM\n",{"type":45,"tag":965,"props":1194,"children":1196},{"class":967,"line":1195},6,[1197],{"type":45,"tag":965,"props":1198,"children":1199},{"emptyLinePlaceholder":1136},[1200],{"type":51,"value":1139},{"type":45,"tag":965,"props":1202,"children":1204},{"class":967,"line":1203},7,[1205],{"type":45,"tag":965,"props":1206,"children":1207},{"style":1055},[1208],{"type":51,"value":1209},"# Show the changeset for that PR\n",{"type":45,"tag":965,"props":1211,"children":1213},{"class":967,"line":1212},8,[1214,1218,1222,1227,1231,1235,1239,1243],{"type":45,"tag":965,"props":1215,"children":1216},{"style":972},[1217],{"type":51,"value":1157},{"type":45,"tag":965,"props":1219,"children":1220},{"style":978},[1221],{"type":51,"value":1162},{"type":45,"tag":965,"props":1223,"children":1224},{"style":978},[1225],{"type":51,"value":1226}," diff",{"type":45,"tag":965,"props":1228,"children":1229},{"style":1071},[1230],{"type":51,"value":1172},{"type":45,"tag":965,"props":1232,"children":1233},{"style":1065},[1234],{"type":51,"value":1177},{"type":45,"tag":965,"props":1236,"children":1237},{"style":1071},[1238],{"type":51,"value":1182},{"type":45,"tag":965,"props":1240,"children":1241},{"style":978},[1242],{"type":51,"value":1187},{"type":45,"tag":965,"props":1244,"children":1245},{"style":978},[1246],{"type":51,"value":1192},{"type":45,"tag":630,"props":1248,"children":1250},{"id":1249},"reading-ci-job-logs",[1251],{"type":51,"value":1252},"Reading CI Job Logs",{"type":45,"tag":288,"props":1254,"children":1256},{"className":957,"code":1255,"language":959,"meta":293,"style":293},"# List recent workflow runs for the PR\ngh run list --repo NVIDIA\u002FMegatron-LM --branch \"pull-request\u002F$PR_NUMBER\"\n\n# Stream failing job output\ngh run view \u003Crun-id> --repo NVIDIA\u002FMegatron-LM --log-failed\n",[1257],{"type":45,"tag":81,"props":1258,"children":1259},{"__ignoreMap":293},[1260,1268,1317,1324,1332],{"type":45,"tag":965,"props":1261,"children":1262},{"class":967,"line":968},[1263],{"type":45,"tag":965,"props":1264,"children":1265},{"style":1055},[1266],{"type":51,"value":1267},"# List recent workflow runs for the PR\n",{"type":45,"tag":965,"props":1269,"children":1270},{"class":967,"line":1061},[1271,1275,1280,1285,1289,1294,1299,1303,1308,1312],{"type":45,"tag":965,"props":1272,"children":1273},{"style":972},[1274],{"type":51,"value":1157},{"type":45,"tag":965,"props":1276,"children":1277},{"style":978},[1278],{"type":51,"value":1279}," run",{"type":45,"tag":965,"props":1281,"children":1282},{"style":978},[1283],{"type":51,"value":1284}," list",{"type":45,"tag":965,"props":1286,"children":1287},{"style":978},[1288],{"type":51,"value":1187},{"type":45,"tag":965,"props":1290,"children":1291},{"style":978},[1292],{"type":51,"value":1293}," NVIDIA\u002FMegatron-LM",{"type":45,"tag":965,"props":1295,"children":1296},{"style":978},[1297],{"type":51,"value":1298}," --branch",{"type":45,"tag":965,"props":1300,"children":1301},{"style":1071},[1302],{"type":51,"value":1172},{"type":45,"tag":965,"props":1304,"children":1305},{"style":978},[1306],{"type":51,"value":1307},"pull-request\u002F",{"type":45,"tag":965,"props":1309,"children":1310},{"style":1065},[1311],{"type":51,"value":1177},{"type":45,"tag":965,"props":1313,"children":1314},{"style":1071},[1315],{"type":51,"value":1316},"\"\n",{"type":45,"tag":965,"props":1318,"children":1319},{"class":967,"line":1132},[1320],{"type":45,"tag":965,"props":1321,"children":1322},{"emptyLinePlaceholder":1136},[1323],{"type":51,"value":1139},{"type":45,"tag":965,"props":1325,"children":1326},{"class":967,"line":1142},[1327],{"type":45,"tag":965,"props":1328,"children":1329},{"style":1055},[1330],{"type":51,"value":1331},"# Stream failing job output\n",{"type":45,"tag":965,"props":1333,"children":1334},{"class":967,"line":1151},[1335,1339,1343,1347,1352,1357,1362,1367,1371,1375],{"type":45,"tag":965,"props":1336,"children":1337},{"style":972},[1338],{"type":51,"value":1157},{"type":45,"tag":965,"props":1340,"children":1341},{"style":978},[1342],{"type":51,"value":1279},{"type":45,"tag":965,"props":1344,"children":1345},{"style":978},[1346],{"type":51,"value":1167},{"type":45,"tag":965,"props":1348,"children":1349},{"style":1071},[1350],{"type":51,"value":1351}," \u003C",{"type":45,"tag":965,"props":1353,"children":1354},{"style":978},[1355],{"type":51,"value":1356},"run-i",{"type":45,"tag":965,"props":1358,"children":1359},{"style":1065},[1360],{"type":51,"value":1361},"d",{"type":45,"tag":965,"props":1363,"children":1364},{"style":1071},[1365],{"type":51,"value":1366},">",{"type":45,"tag":965,"props":1368,"children":1369},{"style":978},[1370],{"type":51,"value":1187},{"type":45,"tag":965,"props":1372,"children":1373},{"style":978},[1374],{"type":51,"value":1293},{"type":45,"tag":965,"props":1376,"children":1377},{"style":978},[1378],{"type":51,"value":1379}," --log-failed\n",{"type":45,"tag":65,"props":1381,"children":1382},{},[1383,1385,1390,1392,1398],{"type":51,"value":1384},"Full per-rank logs are ",{"type":45,"tag":176,"props":1386,"children":1387},{},[1388],{"type":51,"value":1389},"not",{"type":51,"value":1391}," in the runner stdout. They are uploaded as\nGitHub artifacts named ",{"type":45,"tag":81,"props":1393,"children":1395},{"className":1394},[],[1396],{"type":51,"value":1397},"logs-\u003Ctest_case>-\u003Crun_id>-\u003Cuuid>",{"type":51,"value":103},{"type":45,"tag":288,"props":1400,"children":1402},{"className":957,"code":1401,"language":959,"meta":293,"style":293},"# 1. Find artifact name\ngh run view \u003Crun-id> --repo NVIDIA\u002FMegatron-LM --json artifacts \\\n  --jq '.artifacts[].name'\n\n# 2. Download the artifact zip\ngh run download \u003Crun-id> --repo NVIDIA\u002FMegatron-LM \\\n  --name \"logs-\u003Cartifact-name>\" -D .\u002Fci-logs\n\n# 3. Locate which rank logs contain errors\ngrep -r -l \"ERROR\\|Traceback\\|FAILED\\|fatal\" .\u002Fci-logs\u002F\n\n# 4. Log files can exceed 10 000 lines — never read a full log at once.\nwc -l .\u002Fci-logs\u002F\u003Ctest>\u002F\u003Cattempt>\u002Fattempt_0\u002F\u003Crank>\u002Fstderr.log\nsed -n '1,200p' .\u002Fci-logs\u002F...\u002Fstderr.log   # read in chunks\n",[1403],{"type":45,"tag":81,"props":1404,"children":1405},{"__ignoreMap":293},[1406,1414,1468,1490,1497,1505,1549,1580,1587,1596,1633,1641,1650,1737],{"type":45,"tag":965,"props":1407,"children":1408},{"class":967,"line":968},[1409],{"type":45,"tag":965,"props":1410,"children":1411},{"style":1055},[1412],{"type":51,"value":1413},"# 1. Find artifact name\n",{"type":45,"tag":965,"props":1415,"children":1416},{"class":967,"line":1061},[1417,1421,1425,1429,1433,1437,1441,1445,1449,1453,1458,1463],{"type":45,"tag":965,"props":1418,"children":1419},{"style":972},[1420],{"type":51,"value":1157},{"type":45,"tag":965,"props":1422,"children":1423},{"style":978},[1424],{"type":51,"value":1279},{"type":45,"tag":965,"props":1426,"children":1427},{"style":978},[1428],{"type":51,"value":1167},{"type":45,"tag":965,"props":1430,"children":1431},{"style":1071},[1432],{"type":51,"value":1351},{"type":45,"tag":965,"props":1434,"children":1435},{"style":978},[1436],{"type":51,"value":1356},{"type":45,"tag":965,"props":1438,"children":1439},{"style":1065},[1440],{"type":51,"value":1361},{"type":45,"tag":965,"props":1442,"children":1443},{"style":1071},[1444],{"type":51,"value":1366},{"type":45,"tag":965,"props":1446,"children":1447},{"style":978},[1448],{"type":51,"value":1187},{"type":45,"tag":965,"props":1450,"children":1451},{"style":978},[1452],{"type":51,"value":1293},{"type":45,"tag":965,"props":1454,"children":1455},{"style":978},[1456],{"type":51,"value":1457}," --json",{"type":45,"tag":965,"props":1459,"children":1460},{"style":978},[1461],{"type":51,"value":1462}," artifacts",{"type":45,"tag":965,"props":1464,"children":1465},{"style":1065},[1466],{"type":51,"value":1467}," \\\n",{"type":45,"tag":965,"props":1469,"children":1470},{"class":967,"line":1132},[1471,1476,1480,1485],{"type":45,"tag":965,"props":1472,"children":1473},{"style":978},[1474],{"type":51,"value":1475},"  --jq",{"type":45,"tag":965,"props":1477,"children":1478},{"style":1071},[1479],{"type":51,"value":1114},{"type":45,"tag":965,"props":1481,"children":1482},{"style":978},[1483],{"type":51,"value":1484},".artifacts[].name",{"type":45,"tag":965,"props":1486,"children":1487},{"style":1071},[1488],{"type":51,"value":1489},"'\n",{"type":45,"tag":965,"props":1491,"children":1492},{"class":967,"line":1142},[1493],{"type":45,"tag":965,"props":1494,"children":1495},{"emptyLinePlaceholder":1136},[1496],{"type":51,"value":1139},{"type":45,"tag":965,"props":1498,"children":1499},{"class":967,"line":1151},[1500],{"type":45,"tag":965,"props":1501,"children":1502},{"style":1055},[1503],{"type":51,"value":1504},"# 2. Download the artifact zip\n",{"type":45,"tag":965,"props":1506,"children":1507},{"class":967,"line":1195},[1508,1512,1516,1521,1525,1529,1533,1537,1541,1545],{"type":45,"tag":965,"props":1509,"children":1510},{"style":972},[1511],{"type":51,"value":1157},{"type":45,"tag":965,"props":1513,"children":1514},{"style":978},[1515],{"type":51,"value":1279},{"type":45,"tag":965,"props":1517,"children":1518},{"style":978},[1519],{"type":51,"value":1520}," download",{"type":45,"tag":965,"props":1522,"children":1523},{"style":1071},[1524],{"type":51,"value":1351},{"type":45,"tag":965,"props":1526,"children":1527},{"style":978},[1528],{"type":51,"value":1356},{"type":45,"tag":965,"props":1530,"children":1531},{"style":1065},[1532],{"type":51,"value":1361},{"type":45,"tag":965,"props":1534,"children":1535},{"style":1071},[1536],{"type":51,"value":1366},{"type":45,"tag":965,"props":1538,"children":1539},{"style":978},[1540],{"type":51,"value":1187},{"type":45,"tag":965,"props":1542,"children":1543},{"style":978},[1544],{"type":51,"value":1293},{"type":45,"tag":965,"props":1546,"children":1547},{"style":1065},[1548],{"type":51,"value":1467},{"type":45,"tag":965,"props":1550,"children":1551},{"class":967,"line":1203},[1552,1557,1561,1566,1570,1575],{"type":45,"tag":965,"props":1553,"children":1554},{"style":978},[1555],{"type":51,"value":1556},"  --name",{"type":45,"tag":965,"props":1558,"children":1559},{"style":1071},[1560],{"type":51,"value":1172},{"type":45,"tag":965,"props":1562,"children":1563},{"style":978},[1564],{"type":51,"value":1565},"logs-\u003Cartifact-name>",{"type":45,"tag":965,"props":1567,"children":1568},{"style":1071},[1569],{"type":51,"value":1182},{"type":45,"tag":965,"props":1571,"children":1572},{"style":978},[1573],{"type":51,"value":1574}," -D",{"type":45,"tag":965,"props":1576,"children":1577},{"style":978},[1578],{"type":51,"value":1579}," .\u002Fci-logs\n",{"type":45,"tag":965,"props":1581,"children":1582},{"class":967,"line":1212},[1583],{"type":45,"tag":965,"props":1584,"children":1585},{"emptyLinePlaceholder":1136},[1586],{"type":51,"value":1139},{"type":45,"tag":965,"props":1588,"children":1590},{"class":967,"line":1589},9,[1591],{"type":45,"tag":965,"props":1592,"children":1593},{"style":1055},[1594],{"type":51,"value":1595},"# 3. Locate which rank logs contain errors\n",{"type":45,"tag":965,"props":1597,"children":1599},{"class":967,"line":1598},10,[1600,1605,1610,1615,1619,1624,1628],{"type":45,"tag":965,"props":1601,"children":1602},{"style":972},[1603],{"type":51,"value":1604},"grep",{"type":45,"tag":965,"props":1606,"children":1607},{"style":978},[1608],{"type":51,"value":1609}," -r",{"type":45,"tag":965,"props":1611,"children":1612},{"style":978},[1613],{"type":51,"value":1614}," -l",{"type":45,"tag":965,"props":1616,"children":1617},{"style":1071},[1618],{"type":51,"value":1172},{"type":45,"tag":965,"props":1620,"children":1621},{"style":978},[1622],{"type":51,"value":1623},"ERROR\\|Traceback\\|FAILED\\|fatal",{"type":45,"tag":965,"props":1625,"children":1626},{"style":1071},[1627],{"type":51,"value":1182},{"type":45,"tag":965,"props":1629,"children":1630},{"style":978},[1631],{"type":51,"value":1632}," .\u002Fci-logs\u002F\n",{"type":45,"tag":965,"props":1634,"children":1636},{"class":967,"line":1635},11,[1637],{"type":45,"tag":965,"props":1638,"children":1639},{"emptyLinePlaceholder":1136},[1640],{"type":51,"value":1139},{"type":45,"tag":965,"props":1642,"children":1644},{"class":967,"line":1643},12,[1645],{"type":45,"tag":965,"props":1646,"children":1647},{"style":1055},[1648],{"type":51,"value":1649},"# 4. Log files can exceed 10 000 lines — never read a full log at once.\n",{"type":45,"tag":965,"props":1651,"children":1653},{"class":967,"line":1652},13,[1654,1659,1663,1668,1673,1678,1683,1687,1692,1696,1701,1705,1709,1714,1718,1723,1728,1732],{"type":45,"tag":965,"props":1655,"children":1656},{"style":972},[1657],{"type":51,"value":1658},"wc",{"type":45,"tag":965,"props":1660,"children":1661},{"style":978},[1662],{"type":51,"value":1614},{"type":45,"tag":965,"props":1664,"children":1665},{"style":978},[1666],{"type":51,"value":1667}," .\u002Fci-logs\u002F",{"type":45,"tag":965,"props":1669,"children":1670},{"style":1071},[1671],{"type":51,"value":1672},"\u003C",{"type":45,"tag":965,"props":1674,"children":1675},{"style":978},[1676],{"type":51,"value":1677},"tes",{"type":45,"tag":965,"props":1679,"children":1680},{"style":1065},[1681],{"type":51,"value":1682},"t",{"type":45,"tag":965,"props":1684,"children":1685},{"style":1071},[1686],{"type":51,"value":1366},{"type":45,"tag":965,"props":1688,"children":1689},{"style":978},[1690],{"type":51,"value":1691},"\u002F",{"type":45,"tag":965,"props":1693,"children":1694},{"style":1071},[1695],{"type":51,"value":1672},{"type":45,"tag":965,"props":1697,"children":1698},{"style":978},[1699],{"type":51,"value":1700},"attemp",{"type":45,"tag":965,"props":1702,"children":1703},{"style":1065},[1704],{"type":51,"value":1682},{"type":45,"tag":965,"props":1706,"children":1707},{"style":1071},[1708],{"type":51,"value":1366},{"type":45,"tag":965,"props":1710,"children":1711},{"style":978},[1712],{"type":51,"value":1713},"\u002Fattempt_0\u002F",{"type":45,"tag":965,"props":1715,"children":1716},{"style":1071},[1717],{"type":51,"value":1672},{"type":45,"tag":965,"props":1719,"children":1720},{"style":978},[1721],{"type":51,"value":1722},"ran",{"type":45,"tag":965,"props":1724,"children":1725},{"style":1065},[1726],{"type":51,"value":1727},"k",{"type":45,"tag":965,"props":1729,"children":1730},{"style":1071},[1731],{"type":51,"value":1366},{"type":45,"tag":965,"props":1733,"children":1734},{"style":978},[1735],{"type":51,"value":1736},"\u002Fstderr.log\n",{"type":45,"tag":965,"props":1738,"children":1740},{"class":967,"line":1739},14,[1741,1746,1751,1755,1760,1764,1769],{"type":45,"tag":965,"props":1742,"children":1743},{"style":972},[1744],{"type":51,"value":1745},"sed",{"type":45,"tag":965,"props":1747,"children":1748},{"style":978},[1749],{"type":51,"value":1750}," -n",{"type":45,"tag":965,"props":1752,"children":1753},{"style":1071},[1754],{"type":51,"value":1114},{"type":45,"tag":965,"props":1756,"children":1757},{"style":978},[1758],{"type":51,"value":1759},"1,200p",{"type":45,"tag":965,"props":1761,"children":1762},{"style":1071},[1763],{"type":51,"value":1124},{"type":45,"tag":965,"props":1765,"children":1766},{"style":978},[1767],{"type":51,"value":1768}," .\u002Fci-logs\u002F...\u002Fstderr.log",{"type":45,"tag":965,"props":1770,"children":1771},{"style":1055},[1772],{"type":51,"value":1773},"   # read in chunks\n",{"type":45,"tag":630,"props":1775,"children":1777},{"id":1776},"identifying-failure-root-cause",[1778],{"type":51,"value":1779},"Identifying Failure Root Cause",{"type":45,"tag":1781,"props":1782,"children":1783},"ol",{},[1784,1802,1820,1838,1864],{"type":45,"tag":75,"props":1785,"children":1786},{},[1787,1792,1794,1800],{"type":45,"tag":176,"props":1788,"children":1789},{},[1790],{"type":51,"value":1791},"Linting failure",{"type":51,"value":1793}," — re-run ",{"type":45,"tag":81,"props":1795,"children":1797},{"className":1796},[],[1798],{"type":51,"value":1799},"tools\u002Fautoformat.sh",{"type":51,"value":1801}," locally; the diff shows exactly what needs to change.",{"type":45,"tag":75,"props":1803,"children":1804},{},[1805,1810,1812,1818],{"type":45,"tag":176,"props":1806,"children":1807},{},[1808],{"type":51,"value":1809},"Container build failure",{"type":51,"value":1811}," — inspect the ",{"type":45,"tag":81,"props":1813,"children":1815},{"className":1814},[],[1816],{"type":51,"value":1817},"cicd-container-build",{"type":51,"value":1819}," job log.",{"type":45,"tag":75,"props":1821,"children":1822},{},[1823,1828,1830,1836],{"type":45,"tag":176,"props":1824,"children":1825},{},[1826],{"type":51,"value":1827},"Unit test failure",{"type":51,"value":1829}," — the failing bucket is in the ",{"type":45,"tag":81,"props":1831,"children":1833},{"className":1832},[],[1834],{"type":51,"value":1835},"cicd-unit-tests-latest",{"type":51,"value":1837}," job matrix.",{"type":45,"tag":75,"props":1839,"children":1840},{},[1841,1846,1848,1854,1856,1862],{"type":45,"tag":176,"props":1842,"children":1843},{},[1844],{"type":51,"value":1845},"Functional test failure",{"type":51,"value":1847}," — look at the ",{"type":45,"tag":81,"props":1849,"children":1851},{"className":1850},[],[1852],{"type":51,"value":1853},"cicd-integration-tests-*",{"type":51,"value":1855}," job. Start with ",{"type":45,"tag":81,"props":1857,"children":1859},{"className":1858},[],[1860],{"type":51,"value":1861},"stdout.log",{"type":51,"value":1863}," for rank 0.",{"type":45,"tag":75,"props":1865,"children":1866},{},[1867,1872],{"type":45,"tag":176,"props":1868,"children":1869},{},[1870],{"type":51,"value":1871},"Flaky test",{"type":51,"value":1873}," — the runner retries automatically up to 3 times. If all retries exhausted and the pattern matches a known transient (NCCL, ECC, segfault), it is infrastructure noise.",{"type":45,"tag":630,"props":1875,"children":1877},{"id":1876},"correlating-a-failure-with-the-pr-changeset",[1878],{"type":51,"value":1879},"Correlating a Failure with the PR Changeset",{"type":45,"tag":288,"props":1881,"children":1883},{"className":957,"code":1882,"language":959,"meta":293,"style":293},"# Find unit tests that cover a changed source file\ngrep -r \"from megatron.core.transformer.attention\" tests\u002Funit_tests\u002F -l\n\n# Check CODEOWNERS for reviewer assignment\ncat .github\u002FCODEOWNERS | grep \"\u003Cchanged-path>\"\n",[1884],{"type":45,"tag":81,"props":1885,"children":1886},{"__ignoreMap":293},[1887,1895,1929,1936,1944],{"type":45,"tag":965,"props":1888,"children":1889},{"class":967,"line":968},[1890],{"type":45,"tag":965,"props":1891,"children":1892},{"style":1055},[1893],{"type":51,"value":1894},"# Find unit tests that cover a changed source file\n",{"type":45,"tag":965,"props":1896,"children":1897},{"class":967,"line":1061},[1898,1902,1906,1910,1915,1919,1924],{"type":45,"tag":965,"props":1899,"children":1900},{"style":972},[1901],{"type":51,"value":1604},{"type":45,"tag":965,"props":1903,"children":1904},{"style":978},[1905],{"type":51,"value":1609},{"type":45,"tag":965,"props":1907,"children":1908},{"style":1071},[1909],{"type":51,"value":1172},{"type":45,"tag":965,"props":1911,"children":1912},{"style":978},[1913],{"type":51,"value":1914},"from megatron.core.transformer.attention",{"type":45,"tag":965,"props":1916,"children":1917},{"style":1071},[1918],{"type":51,"value":1182},{"type":45,"tag":965,"props":1920,"children":1921},{"style":978},[1922],{"type":51,"value":1923}," tests\u002Funit_tests\u002F",{"type":45,"tag":965,"props":1925,"children":1926},{"style":978},[1927],{"type":51,"value":1928}," -l\n",{"type":45,"tag":965,"props":1930,"children":1931},{"class":967,"line":1132},[1932],{"type":45,"tag":965,"props":1933,"children":1934},{"emptyLinePlaceholder":1136},[1935],{"type":51,"value":1139},{"type":45,"tag":965,"props":1937,"children":1938},{"class":967,"line":1142},[1939],{"type":45,"tag":965,"props":1940,"children":1941},{"style":1055},[1942],{"type":51,"value":1943},"# Check CODEOWNERS for reviewer assignment\n",{"type":45,"tag":965,"props":1945,"children":1946},{"class":967,"line":1151},[1947,1952,1957,1961,1965,1969,1974],{"type":45,"tag":965,"props":1948,"children":1949},{"style":972},[1950],{"type":51,"value":1951},"cat",{"type":45,"tag":965,"props":1953,"children":1954},{"style":978},[1955],{"type":51,"value":1956}," .github\u002FCODEOWNERS",{"type":45,"tag":965,"props":1958,"children":1959},{"style":1071},[1960],{"type":51,"value":1099},{"type":45,"tag":965,"props":1962,"children":1963},{"style":972},[1964],{"type":51,"value":1104},{"type":45,"tag":965,"props":1966,"children":1967},{"style":1071},[1968],{"type":51,"value":1172},{"type":45,"tag":965,"props":1970,"children":1971},{"style":978},[1972],{"type":51,"value":1973},"\u003Cchanged-path>",{"type":45,"tag":965,"props":1975,"children":1976},{"style":1071},[1977],{"type":51,"value":1316},{"type":45,"tag":1979,"props":1980,"children":1981},"style",{},[1982],{"type":51,"value":1983},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1985,"total":1652},[1986,1999,2008,2014,2028,2041,2055],{"slug":1987,"name":1987,"fn":1988,"description":1989,"org":1990,"tags":1991,"stars":22,"repoUrl":23,"updatedAt":1998},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1992,1995,1996],{"name":1993,"slug":1994,"type":15},"Containers","containers",{"name":17,"slug":18,"type":15},{"name":1997,"slug":975,"type":15},"Python","2026-07-27T06:06:11.249662",{"slug":2000,"name":2000,"fn":2001,"description":2002,"org":2003,"tags":2004,"stars":22,"repoUrl":23,"updatedAt":2007},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2005,2006],{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},"2026-07-14T05:25:59.97109",{"slug":4,"name":4,"fn":5,"description":6,"org":2009,"tags":2010,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2011,2012,2013],{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":2015,"name":2015,"fn":2016,"description":2017,"org":2018,"tags":2019,"stars":22,"repoUrl":23,"updatedAt":2027},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2020,2023,2024],{"name":2021,"slug":2022,"type":15},"Debugging","debugging",{"name":13,"slug":14,"type":15},{"name":2025,"slug":2026,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2029,"name":2029,"fn":2030,"description":2031,"org":2032,"tags":2033,"stars":22,"repoUrl":23,"updatedAt":2040},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2034,2037],{"name":2035,"slug":2036,"type":15},"Best Practices","best-practices",{"name":2038,"slug":2039,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2042,"name":2042,"fn":2043,"description":2044,"org":2045,"tags":2046,"stars":22,"repoUrl":23,"updatedAt":2054},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2047,2050,2053],{"name":2048,"slug":2049,"type":15},"Machine Learning","machine-learning",{"name":2051,"slug":2052,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2056,"name":2056,"fn":2057,"description":2058,"org":2059,"tags":2060,"stars":22,"repoUrl":23,"updatedAt":2067},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2061,2064],{"name":2062,"slug":2063,"type":15},"QA","qa",{"name":2065,"slug":2066,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"items":2069,"total":2172},[2070,2088,2094,2099,2105,2111,2116,2122,2127,2138,2152,2161],{"slug":2071,"name":2071,"fn":2072,"description":2073,"org":2074,"tags":2075,"stars":2085,"repoUrl":2086,"updatedAt":2087},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2076,2079,2082],{"name":2077,"slug":2078,"type":15},"Documentation","documentation",{"name":2080,"slug":2081,"type":15},"MCP","mcp",{"name":2083,"slug":2084,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":1987,"name":1987,"fn":1988,"description":1989,"org":2089,"tags":2090,"stars":22,"repoUrl":23,"updatedAt":1998},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2091,2092,2093],{"name":1993,"slug":1994,"type":15},{"name":17,"slug":18,"type":15},{"name":1997,"slug":975,"type":15},{"slug":2000,"name":2000,"fn":2001,"description":2002,"org":2095,"tags":2096,"stars":22,"repoUrl":23,"updatedAt":2007},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2097,2098],{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":2100,"tags":2101,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2102,2103,2104],{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":2015,"name":2015,"fn":2016,"description":2017,"org":2106,"tags":2107,"stars":22,"repoUrl":23,"updatedAt":2027},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2108,2109,2110],{"name":2021,"slug":2022,"type":15},{"name":13,"slug":14,"type":15},{"name":2025,"slug":2026,"type":15},{"slug":2029,"name":2029,"fn":2030,"description":2031,"org":2112,"tags":2113,"stars":22,"repoUrl":23,"updatedAt":2040},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2114,2115],{"name":2035,"slug":2036,"type":15},{"name":2038,"slug":2039,"type":15},{"slug":2042,"name":2042,"fn":2043,"description":2044,"org":2117,"tags":2118,"stars":22,"repoUrl":23,"updatedAt":2054},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2119,2120,2121],{"name":2048,"slug":2049,"type":15},{"name":2051,"slug":2052,"type":15},{"name":9,"slug":8,"type":15},{"slug":2056,"name":2056,"fn":2057,"description":2058,"org":2123,"tags":2124,"stars":22,"repoUrl":23,"updatedAt":2067},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2125,2126],{"name":2062,"slug":2063,"type":15},{"name":2065,"slug":2066,"type":15},{"slug":2128,"name":2128,"fn":2129,"description":2130,"org":2131,"tags":2132,"stars":22,"repoUrl":23,"updatedAt":2137},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2133,2134],{"name":17,"slug":18,"type":15},{"name":2135,"slug":2136,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":2139,"name":2139,"fn":2140,"description":2141,"org":2142,"tags":2143,"stars":22,"repoUrl":23,"updatedAt":2151},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2144,2147,2148],{"name":2145,"slug":2146,"type":15},"Code Review","code-review",{"name":13,"slug":14,"type":15},{"name":2149,"slug":2150,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2153,"name":2153,"fn":2154,"description":2155,"org":2156,"tags":2157,"stars":22,"repoUrl":23,"updatedAt":2160},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2158,2159],{"name":2062,"slug":2063,"type":15},{"name":2065,"slug":2066,"type":15},"2026-07-14T05:25:54.928983",{"slug":2162,"name":2162,"fn":2163,"description":2164,"org":2165,"tags":2166,"stars":22,"repoUrl":23,"updatedAt":2171},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2167,2170],{"name":2168,"slug":2169,"type":15},"Automation","automation",{"name":20,"slug":21,"type":15},"2026-07-30T05:29:03.275638",496]