[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-mcore-testing":3,"mdc-omb5ks-key":33,"related-org-nvidia-mcore-testing":2045,"related-repo-nvidia-mcore-testing":2195},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":19,"repoUrl":20,"updatedAt":21,"license":22,"forks":23,"topics":24,"repo":28,"sourceUrl":31,"mdContent":32},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16],{"name":13,"slug":14,"type":15},"QA","qa","tag",{"name":17,"slug":18,"type":15},"Testing","testing",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-14T05:25:54.928983","Apache-2.0",4230,[25,26,27],"large-language-models","model-para","transformers",{"repoUrl":20,"stars":19,"forks":23,"topics":29,"description":30},[25,26,27],"Ongoing research training transformer models at scale","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM\u002Ftree\u002FHEAD\u002Fskills\u002Fmcore-testing","---\nname: mcore-testing\ndescription: Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.\nlicense: Apache-2.0\nwhen_to_use: Adding or running a unit or functional test; understanding the test layout; writing a recipe YAML; downloading or updating golden values; reproducing a test failure locally; 'how do I add a test', 'run unit tests', 'pytest fails', 'test layout', 'golden values', 'recipe YAML', 'marker filter'.\nmetadata:\n  author: Oliver Koenig \u003Cokoenig@nvidia.com>\n---\n\n# Testing Guide\n\n---\n\n## Answer-First Testing Facts\n\nFor questions about disabling tests without deleting them:\n\n- Functional recipe entries stay in YAML; disable by suffixing scope with\n  `-broken`, for example `scope: [mr-github]` -> `scope: [mr-github-broken]`.\n- Unit-test skips use pytest markers instead: `@pytest.mark.flaky_in_dev` skips\n  in the default dev environment, and `@pytest.mark.flaky` skips in LTS.\n- Do not delete the test case or recipe entry when the goal is discoverability\n  and easy re-enable.\n\n---\n\n## Test Layout\n\n```text\ntests\u002F\n├── unit_tests\u002F          # pytest, 1 node × 8 GPUs, torch.distributed runner\n├── functional_tests\u002F    # end-to-end shell + training scripts\n│   └── test_cases\u002F\n│       └── {model}\u002F{test_case}\u002F\n│           ├── model_config.yaml          # training args\n│           └── golden_values_{env}_{platform}.json\n└── test_utils\u002F\n    ├── recipes\u002F\n    │   ├── h100\u002F        # YAML recipes for H100 jobs\n    │   └── gb200\u002F       # YAML recipes for GB200 jobs\n    └── python_scripts\u002F  # helpers (recipe_parser, golden-value download, …)\n```\n\n---\n\n## How Tests Execute\n\nThe GitHub Actions runner invokes `launch_nemo_run_workload.py`, which uses\n**nemo-run** to launch a `DockerExecutor` container. The repo is bind-mounted\nat `\u002Fopt\u002Fmegatron-lm`; training data is mounted at `\u002Fmnt\u002Fartifacts`.\n\n**Unit tests** are dispatched through `torch.distributed.run`:\n\n- Ranks 0 and 3 are tee-d to stdout; all other ranks write only to log files.\n- Per-rank log files land at `{assets_dir}\u002Flogs\u002F1\u002F` and are uploaded as a\n  GitHub artifact after the run.\n\n**Functional tests** are driven by\n`tests\u002Ffunctional_tests\u002Fshell_test_utils\u002Frun_ci_test.sh`. Only rank 0 runs the\npytest validation step; training output from all ranks is uploaded as an artifact.\n\n**Flaky-failure auto-retry**: `launch_nemo_run_workload.py` retries up to\n**3 times** for known transient patterns (NCCL timeout, ECC error, segfault,\nHuggingFace connectivity, …) before declaring a genuine failure.\n\n---\n\n## Recipe YAML Structure\n\nRecipes live in `tests\u002Ftest_utils\u002Frecipes\u002F` and are parsed by\n`tests\u002Ftest_utils\u002Fpython_scripts\u002Frecipe_parser.py`. Each file expands a\ncartesian `products` block into individual workload specs:\n\n```yaml\ntype: basic\nformat_version: 1\nmaintainers: [mcore]\nloggers: [stdout]\nspec:\n  name: \"{test_case}_{environment}_{platforms}\"\n  model: gpt              # maps to tests\u002Ffunctional_tests\u002Ftest_cases\u002F{model}\u002F\n  build: mcore-pyt-{environment}\n  nodes: 1\n  gpus: 8\n  n_repeat: 5\n  platforms: dgx_h100\n  time_limit: 1800\n  script_setup: |\n    ...\n  script: |-\n    bash tests\u002Ffunctional_tests\u002Fshell_test_utils\u002Frun_ci_test.sh ...\nproducts:\n  - test_case: [my_test]\n    products:\n      - environment: [dev, lts]\n        scope: [mr-github]\n        platforms: [dgx_h100]\n```\n\nKey runtime placeholders: `{assets_dir}`, `{artifacts_dir}`, `{test_case}`,\n`{environment}`, `{platforms}`, `{n_repeat}`.\n\n### Disabling a Test Without Deleting It\n\nTo temporarily disable a test case in a recipe YAML, suffix its `scope` value\nwith `-broken` — **do not delete the entry**:\n\n```yaml\n# before (test runs in CI)\nscope: [mr-github]\n\n# after (test is skipped; entry preserved for easy re-enable)\nscope: [mr-github-broken]\n```\n\n---\n\n## Running Unit Tests Locally\n\nAll unit tests initialize a `torch.distributed` group, so every invocation\nrequires GPU access and must go through `torch.distributed.run`:\n\n```bash\n# Full suite\nuv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \\\n  tests\u002Funit_tests\n\n# Single file\nuv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \\\n  tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_model.py\n\n# Single test\nuv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \\\n  tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_model.py::TestGPTModel::test_constructor\n\n# Filter by name substring\nuv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \\\n  tests\u002Funit_tests -k optimizer\n```\n\n### Marker filters\n\n```bash\n# Exclude flaky tests during development\nuv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \\\n  tests\u002Funit_tests -m \"not flaky and not flaky_in_dev\"\n\n# Include experimental tests\nuv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \\\n  tests\u002Funit_tests --experimental\n```\n\n### CI parity\n\nUse `tests\u002Funit_tests\u002Frun_ci_test.sh` to reproduce a CI bucket failure exactly.\nFor ad-hoc runs, prefer the direct `torch.distributed.run` invocations above.\n\n### Gotchas\n\n- `pyproject.toml` sets `addopts = --durations=15 -s -rA` — stdout is not\n  captured (`-s`), so ranks interleave during multi-rank runs. Override with\n  `--capture=fd` when debugging a specific rank.\n- `tests\u002Funit_tests\u002Fconftest.py` looks for test data under `\u002Fopt\u002Fdata` and\n  attempts a download if missing. Supply it manually or skip data-dependent\n  tests when running outside the canonical container.\n\n---\n\n## Adding a Unit Test\n\n1. Create `tests\u002Funit_tests\u002F\u003Ccategory>\u002Ftest_\u003Cname>.py`.\n2. Use fixtures from `tests\u002Funit_tests\u002Fconftest.py`.\n3. Apply markers as needed:\n   - `@pytest.mark.internal` — skipped on `legacy` tag\n   - `@pytest.mark.flaky_in_dev` — skipped in `dev` environment (CI default; use this to disable a flaky test without blocking the standard pipeline)\n   - `@pytest.mark.flaky` — skipped in `lts` environment\n   - `@pytest.mark.experimental` — `latest` tag only\n4. Verify locally (see Running Unit Tests Locally above).\n5. If the test needs a dedicated CI bucket, add an entry to\n   `tests\u002Ftest_utils\u002Frecipes\u002Fh100\u002Funit-tests.yaml`.\n\n---\n\n## Adding a Functional \u002F Integration Test\n\n1. Create `tests\u002Ffunctional_tests\u002Ftest_cases\u002F\u003Cmodel>\u002F\u003Ctest_name>\u002F`.\n2. Write `model_config.yaml` with `MODEL_ARGS`, `ENV_VARS`, and `TEST_TYPE`.\n3. Add a YAML recipe under `tests\u002Ftest_utils\u002Frecipes\u002Fh100\u002F` (and `gb200\u002F` if\n   needed). Required fields: `scope`, `environment`, `platform`, `n_repeat`,\n   `time_limit`.\n4. Push the PR, add the label **\"Run functional tests\"** to trigger a full run.\n5. After a successful run, download golden values:\n\n   ```bash\n   python tests\u002Ftest_utils\u002Fpython_scripts\u002Fdownload_golden_values.py \\\n     --source github --pipeline-id \u003Crun-id>\n   ```\n\n6. Commit the downloaded golden values.\n\n---\n\n## Common Pitfalls\n\n| Problem | Cause | Fix |\n|---------|-------|-----|\n| Test passes locally but fails in CI | Different environment or data path | Check `DATA_PATH`, `DATA_CACHE_PATH`, and the `environment` tag (`dev` vs `lts`) |\n| Golden value mismatch after a code change | Numerical regression | Download new golden values via `download_golden_values.py` after a clean run |\n| `cicd-integration-tests-gb200` not triggered | GB200 jobs require maintainer status | Ask a maintainer to trigger, or add the `Run functional tests` label |\n",{"data":34,"body":38},{"name":4,"description":6,"license":22,"when_to_use":35,"metadata":36},"Adding or running a unit or functional test; understanding the test layout; writing a recipe YAML; downloading or updating golden values; reproducing a test failure locally; 'how do I add a test', 'run unit tests', 'pytest fails', 'test layout', 'golden values', 'recipe YAML', 'marker filter'.",{"author":37},"Oliver Koenig \u003Cokoenig@nvidia.com>",{"type":39,"children":40},"root",[41,50,54,61,67,128,131,137,149,152,158,202,220,241,259,283,286,292,321,809,858,865,891,970,973,979,998,1302,1308,1468,1474,1494,1500,1557,1560,1566,1688,1691,1697,1885,1888,1894,2039],{"type":42,"tag":43,"props":44,"children":46},"element","h1",{"id":45},"testing-guide",[47],{"type":48,"value":49},"text","Testing Guide",{"type":42,"tag":51,"props":52,"children":53},"hr",{},[],{"type":42,"tag":55,"props":56,"children":58},"h2",{"id":57},"answer-first-testing-facts",[59],{"type":48,"value":60},"Answer-First Testing Facts",{"type":42,"tag":62,"props":63,"children":64},"p",{},[65],{"type":48,"value":66},"For questions about disabling tests without deleting them:",{"type":42,"tag":68,"props":69,"children":70},"ul",{},[71,102,123],{"type":42,"tag":72,"props":73,"children":74},"li",{},[75,77,84,86,92,94,100],{"type":48,"value":76},"Functional recipe entries stay in YAML; disable by suffixing scope with\n",{"type":42,"tag":78,"props":79,"children":81},"code",{"className":80},[],[82],{"type":48,"value":83},"-broken",{"type":48,"value":85},", for example ",{"type":42,"tag":78,"props":87,"children":89},{"className":88},[],[90],{"type":48,"value":91},"scope: [mr-github]",{"type":48,"value":93}," -> ",{"type":42,"tag":78,"props":95,"children":97},{"className":96},[],[98],{"type":48,"value":99},"scope: [mr-github-broken]",{"type":48,"value":101},".",{"type":42,"tag":72,"props":103,"children":104},{},[105,107,113,115,121],{"type":48,"value":106},"Unit-test skips use pytest markers instead: ",{"type":42,"tag":78,"props":108,"children":110},{"className":109},[],[111],{"type":48,"value":112},"@pytest.mark.flaky_in_dev",{"type":48,"value":114}," skips\nin the default dev environment, and ",{"type":42,"tag":78,"props":116,"children":118},{"className":117},[],[119],{"type":48,"value":120},"@pytest.mark.flaky",{"type":48,"value":122}," skips in LTS.",{"type":42,"tag":72,"props":124,"children":125},{},[126],{"type":48,"value":127},"Do not delete the test case or recipe entry when the goal is discoverability\nand easy re-enable.",{"type":42,"tag":51,"props":129,"children":130},{},[],{"type":42,"tag":55,"props":132,"children":134},{"id":133},"test-layout",[135],{"type":48,"value":136},"Test Layout",{"type":42,"tag":138,"props":139,"children":144},"pre",{"className":140,"code":142,"language":48,"meta":143},[141],"language-text","tests\u002F\n├── unit_tests\u002F          # pytest, 1 node × 8 GPUs, torch.distributed runner\n├── functional_tests\u002F    # end-to-end shell + training scripts\n│   └── test_cases\u002F\n│       └── {model}\u002F{test_case}\u002F\n│           ├── model_config.yaml          # training args\n│           └── golden_values_{env}_{platform}.json\n└── test_utils\u002F\n    ├── recipes\u002F\n    │   ├── h100\u002F        # YAML recipes for H100 jobs\n    │   └── gb200\u002F       # YAML recipes for GB200 jobs\n    └── python_scripts\u002F  # helpers (recipe_parser, golden-value download, …)\n","",[145],{"type":42,"tag":78,"props":146,"children":147},{"__ignoreMap":143},[148],{"type":48,"value":142},{"type":42,"tag":51,"props":150,"children":151},{},[],{"type":42,"tag":55,"props":153,"children":155},{"id":154},"how-tests-execute",[156],{"type":48,"value":157},"How Tests Execute",{"type":42,"tag":62,"props":159,"children":160},{},[161,163,169,171,177,179,185,187,193,195,201],{"type":48,"value":162},"The GitHub Actions runner invokes ",{"type":42,"tag":78,"props":164,"children":166},{"className":165},[],[167],{"type":48,"value":168},"launch_nemo_run_workload.py",{"type":48,"value":170},", which uses\n",{"type":42,"tag":172,"props":173,"children":174},"strong",{},[175],{"type":48,"value":176},"nemo-run",{"type":48,"value":178}," to launch a ",{"type":42,"tag":78,"props":180,"children":182},{"className":181},[],[183],{"type":48,"value":184},"DockerExecutor",{"type":48,"value":186}," container. The repo is bind-mounted\nat ",{"type":42,"tag":78,"props":188,"children":190},{"className":189},[],[191],{"type":48,"value":192},"\u002Fopt\u002Fmegatron-lm",{"type":48,"value":194},"; training data is mounted at ",{"type":42,"tag":78,"props":196,"children":198},{"className":197},[],[199],{"type":48,"value":200},"\u002Fmnt\u002Fartifacts",{"type":48,"value":101},{"type":42,"tag":62,"props":203,"children":204},{},[205,210,212,218],{"type":42,"tag":172,"props":206,"children":207},{},[208],{"type":48,"value":209},"Unit tests",{"type":48,"value":211}," are dispatched through ",{"type":42,"tag":78,"props":213,"children":215},{"className":214},[],[216],{"type":48,"value":217},"torch.distributed.run",{"type":48,"value":219},":",{"type":42,"tag":68,"props":221,"children":222},{},[223,228],{"type":42,"tag":72,"props":224,"children":225},{},[226],{"type":48,"value":227},"Ranks 0 and 3 are tee-d to stdout; all other ranks write only to log files.",{"type":42,"tag":72,"props":229,"children":230},{},[231,233,239],{"type":48,"value":232},"Per-rank log files land at ",{"type":42,"tag":78,"props":234,"children":236},{"className":235},[],[237],{"type":48,"value":238},"{assets_dir}\u002Flogs\u002F1\u002F",{"type":48,"value":240}," and are uploaded as a\nGitHub artifact after the run.",{"type":42,"tag":62,"props":242,"children":243},{},[244,249,251,257],{"type":42,"tag":172,"props":245,"children":246},{},[247],{"type":48,"value":248},"Functional tests",{"type":48,"value":250}," are driven by\n",{"type":42,"tag":78,"props":252,"children":254},{"className":253},[],[255],{"type":48,"value":256},"tests\u002Ffunctional_tests\u002Fshell_test_utils\u002Frun_ci_test.sh",{"type":48,"value":258},". Only rank 0 runs the\npytest validation step; training output from all ranks is uploaded as an artifact.",{"type":42,"tag":62,"props":260,"children":261},{},[262,267,269,274,276,281],{"type":42,"tag":172,"props":263,"children":264},{},[265],{"type":48,"value":266},"Flaky-failure auto-retry",{"type":48,"value":268},": ",{"type":42,"tag":78,"props":270,"children":272},{"className":271},[],[273],{"type":48,"value":168},{"type":48,"value":275}," retries up to\n",{"type":42,"tag":172,"props":277,"children":278},{},[279],{"type":48,"value":280},"3 times",{"type":48,"value":282}," for known transient patterns (NCCL timeout, ECC error, segfault,\nHuggingFace connectivity, …) before declaring a genuine failure.",{"type":42,"tag":51,"props":284,"children":285},{},[],{"type":42,"tag":55,"props":287,"children":289},{"id":288},"recipe-yaml-structure",[290],{"type":48,"value":291},"Recipe YAML Structure",{"type":42,"tag":62,"props":293,"children":294},{},[295,297,303,305,311,313,319],{"type":48,"value":296},"Recipes live in ",{"type":42,"tag":78,"props":298,"children":300},{"className":299},[],[301],{"type":48,"value":302},"tests\u002Ftest_utils\u002Frecipes\u002F",{"type":48,"value":304}," and are parsed by\n",{"type":42,"tag":78,"props":306,"children":308},{"className":307},[],[309],{"type":48,"value":310},"tests\u002Ftest_utils\u002Fpython_scripts\u002Frecipe_parser.py",{"type":48,"value":312},". Each file expands a\ncartesian ",{"type":42,"tag":78,"props":314,"children":316},{"className":315},[],[317],{"type":48,"value":318},"products",{"type":48,"value":320}," block into individual workload specs:",{"type":42,"tag":138,"props":322,"children":326},{"className":323,"code":324,"language":325,"meta":143,"style":143},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","type: basic\nformat_version: 1\nmaintainers: [mcore]\nloggers: [stdout]\nspec:\n  name: \"{test_case}_{environment}_{platforms}\"\n  model: gpt              # maps to tests\u002Ffunctional_tests\u002Ftest_cases\u002F{model}\u002F\n  build: mcore-pyt-{environment}\n  nodes: 1\n  gpus: 8\n  n_repeat: 5\n  platforms: dgx_h100\n  time_limit: 1800\n  script_setup: |\n    ...\n  script: |-\n    bash tests\u002Ffunctional_tests\u002Fshell_test_utils\u002Frun_ci_test.sh ...\nproducts:\n  - test_case: [my_test]\n    products:\n      - environment: [dev, lts]\n        scope: [mr-github]\n        platforms: [dgx_h100]\n","yaml",[327],{"type":42,"tag":78,"props":328,"children":329},{"__ignoreMap":143},[330,353,372,400,426,440,468,492,510,527,545,563,581,599,618,627,651,660,672,703,716,757,783],{"type":42,"tag":331,"props":332,"children":335},"span",{"class":333,"line":334},"line",1,[336,342,347],{"type":42,"tag":331,"props":337,"children":339},{"style":338},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[340],{"type":48,"value":341},"type",{"type":42,"tag":331,"props":343,"children":345},{"style":344},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[346],{"type":48,"value":219},{"type":42,"tag":331,"props":348,"children":350},{"style":349},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[351],{"type":48,"value":352}," basic\n",{"type":42,"tag":331,"props":354,"children":356},{"class":333,"line":355},2,[357,362,366],{"type":42,"tag":331,"props":358,"children":359},{"style":338},[360],{"type":48,"value":361},"format_version",{"type":42,"tag":331,"props":363,"children":364},{"style":344},[365],{"type":48,"value":219},{"type":42,"tag":331,"props":367,"children":369},{"style":368},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[370],{"type":48,"value":371}," 1\n",{"type":42,"tag":331,"props":373,"children":375},{"class":333,"line":374},3,[376,381,385,390,395],{"type":42,"tag":331,"props":377,"children":378},{"style":338},[379],{"type":48,"value":380},"maintainers",{"type":42,"tag":331,"props":382,"children":383},{"style":344},[384],{"type":48,"value":219},{"type":42,"tag":331,"props":386,"children":387},{"style":344},[388],{"type":48,"value":389}," [",{"type":42,"tag":331,"props":391,"children":392},{"style":349},[393],{"type":48,"value":394},"mcore",{"type":42,"tag":331,"props":396,"children":397},{"style":344},[398],{"type":48,"value":399},"]\n",{"type":42,"tag":331,"props":401,"children":403},{"class":333,"line":402},4,[404,409,413,417,422],{"type":42,"tag":331,"props":405,"children":406},{"style":338},[407],{"type":48,"value":408},"loggers",{"type":42,"tag":331,"props":410,"children":411},{"style":344},[412],{"type":48,"value":219},{"type":42,"tag":331,"props":414,"children":415},{"style":344},[416],{"type":48,"value":389},{"type":42,"tag":331,"props":418,"children":419},{"style":349},[420],{"type":48,"value":421},"stdout",{"type":42,"tag":331,"props":423,"children":424},{"style":344},[425],{"type":48,"value":399},{"type":42,"tag":331,"props":427,"children":429},{"class":333,"line":428},5,[430,435],{"type":42,"tag":331,"props":431,"children":432},{"style":338},[433],{"type":48,"value":434},"spec",{"type":42,"tag":331,"props":436,"children":437},{"style":344},[438],{"type":48,"value":439},":\n",{"type":42,"tag":331,"props":441,"children":443},{"class":333,"line":442},6,[444,449,453,458,463],{"type":42,"tag":331,"props":445,"children":446},{"style":338},[447],{"type":48,"value":448},"  name",{"type":42,"tag":331,"props":450,"children":451},{"style":344},[452],{"type":48,"value":219},{"type":42,"tag":331,"props":454,"children":455},{"style":344},[456],{"type":48,"value":457}," \"",{"type":42,"tag":331,"props":459,"children":460},{"style":349},[461],{"type":48,"value":462},"{test_case}_{environment}_{platforms}",{"type":42,"tag":331,"props":464,"children":465},{"style":344},[466],{"type":48,"value":467},"\"\n",{"type":42,"tag":331,"props":469,"children":471},{"class":333,"line":470},7,[472,477,481,486],{"type":42,"tag":331,"props":473,"children":474},{"style":338},[475],{"type":48,"value":476},"  model",{"type":42,"tag":331,"props":478,"children":479},{"style":344},[480],{"type":48,"value":219},{"type":42,"tag":331,"props":482,"children":483},{"style":349},[484],{"type":48,"value":485}," gpt",{"type":42,"tag":331,"props":487,"children":489},{"style":488},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[490],{"type":48,"value":491},"              # maps to tests\u002Ffunctional_tests\u002Ftest_cases\u002F{model}\u002F\n",{"type":42,"tag":331,"props":493,"children":495},{"class":333,"line":494},8,[496,501,505],{"type":42,"tag":331,"props":497,"children":498},{"style":338},[499],{"type":48,"value":500},"  build",{"type":42,"tag":331,"props":502,"children":503},{"style":344},[504],{"type":48,"value":219},{"type":42,"tag":331,"props":506,"children":507},{"style":349},[508],{"type":48,"value":509}," mcore-pyt-{environment}\n",{"type":42,"tag":331,"props":511,"children":513},{"class":333,"line":512},9,[514,519,523],{"type":42,"tag":331,"props":515,"children":516},{"style":338},[517],{"type":48,"value":518},"  nodes",{"type":42,"tag":331,"props":520,"children":521},{"style":344},[522],{"type":48,"value":219},{"type":42,"tag":331,"props":524,"children":525},{"style":368},[526],{"type":48,"value":371},{"type":42,"tag":331,"props":528,"children":530},{"class":333,"line":529},10,[531,536,540],{"type":42,"tag":331,"props":532,"children":533},{"style":338},[534],{"type":48,"value":535},"  gpus",{"type":42,"tag":331,"props":537,"children":538},{"style":344},[539],{"type":48,"value":219},{"type":42,"tag":331,"props":541,"children":542},{"style":368},[543],{"type":48,"value":544}," 8\n",{"type":42,"tag":331,"props":546,"children":548},{"class":333,"line":547},11,[549,554,558],{"type":42,"tag":331,"props":550,"children":551},{"style":338},[552],{"type":48,"value":553},"  n_repeat",{"type":42,"tag":331,"props":555,"children":556},{"style":344},[557],{"type":48,"value":219},{"type":42,"tag":331,"props":559,"children":560},{"style":368},[561],{"type":48,"value":562}," 5\n",{"type":42,"tag":331,"props":564,"children":566},{"class":333,"line":565},12,[567,572,576],{"type":42,"tag":331,"props":568,"children":569},{"style":338},[570],{"type":48,"value":571},"  platforms",{"type":42,"tag":331,"props":573,"children":574},{"style":344},[575],{"type":48,"value":219},{"type":42,"tag":331,"props":577,"children":578},{"style":349},[579],{"type":48,"value":580}," dgx_h100\n",{"type":42,"tag":331,"props":582,"children":584},{"class":333,"line":583},13,[585,590,594],{"type":42,"tag":331,"props":586,"children":587},{"style":338},[588],{"type":48,"value":589},"  time_limit",{"type":42,"tag":331,"props":591,"children":592},{"style":344},[593],{"type":48,"value":219},{"type":42,"tag":331,"props":595,"children":596},{"style":368},[597],{"type":48,"value":598}," 1800\n",{"type":42,"tag":331,"props":600,"children":602},{"class":333,"line":601},14,[603,608,612],{"type":42,"tag":331,"props":604,"children":605},{"style":338},[606],{"type":48,"value":607},"  script_setup",{"type":42,"tag":331,"props":609,"children":610},{"style":344},[611],{"type":48,"value":219},{"type":42,"tag":331,"props":613,"children":615},{"style":614},"--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic",[616],{"type":48,"value":617}," |\n",{"type":42,"tag":331,"props":619,"children":621},{"class":333,"line":620},15,[622],{"type":42,"tag":331,"props":623,"children":624},{"style":349},[625],{"type":48,"value":626},"    ...\n",{"type":42,"tag":331,"props":628,"children":630},{"class":333,"line":629},16,[631,636,640,645],{"type":42,"tag":331,"props":632,"children":633},{"style":338},[634],{"type":48,"value":635},"  script",{"type":42,"tag":331,"props":637,"children":638},{"style":344},[639],{"type":48,"value":219},{"type":42,"tag":331,"props":641,"children":642},{"style":614},[643],{"type":48,"value":644}," |",{"type":42,"tag":331,"props":646,"children":648},{"style":647},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[649],{"type":48,"value":650},"-\n",{"type":42,"tag":331,"props":652,"children":654},{"class":333,"line":653},17,[655],{"type":42,"tag":331,"props":656,"children":657},{"style":349},[658],{"type":48,"value":659},"    bash tests\u002Ffunctional_tests\u002Fshell_test_utils\u002Frun_ci_test.sh ...\n",{"type":42,"tag":331,"props":661,"children":663},{"class":333,"line":662},18,[664,668],{"type":42,"tag":331,"props":665,"children":666},{"style":338},[667],{"type":48,"value":318},{"type":42,"tag":331,"props":669,"children":670},{"style":344},[671],{"type":48,"value":439},{"type":42,"tag":331,"props":673,"children":675},{"class":333,"line":674},19,[676,681,686,690,694,699],{"type":42,"tag":331,"props":677,"children":678},{"style":344},[679],{"type":48,"value":680},"  -",{"type":42,"tag":331,"props":682,"children":683},{"style":338},[684],{"type":48,"value":685}," test_case",{"type":42,"tag":331,"props":687,"children":688},{"style":344},[689],{"type":48,"value":219},{"type":42,"tag":331,"props":691,"children":692},{"style":344},[693],{"type":48,"value":389},{"type":42,"tag":331,"props":695,"children":696},{"style":349},[697],{"type":48,"value":698},"my_test",{"type":42,"tag":331,"props":700,"children":701},{"style":344},[702],{"type":48,"value":399},{"type":42,"tag":331,"props":704,"children":706},{"class":333,"line":705},20,[707,712],{"type":42,"tag":331,"props":708,"children":709},{"style":338},[710],{"type":48,"value":711},"    products",{"type":42,"tag":331,"props":713,"children":714},{"style":344},[715],{"type":48,"value":439},{"type":42,"tag":331,"props":717,"children":719},{"class":333,"line":718},21,[720,725,730,734,738,743,748,753],{"type":42,"tag":331,"props":721,"children":722},{"style":344},[723],{"type":48,"value":724},"      -",{"type":42,"tag":331,"props":726,"children":727},{"style":338},[728],{"type":48,"value":729}," environment",{"type":42,"tag":331,"props":731,"children":732},{"style":344},[733],{"type":48,"value":219},{"type":42,"tag":331,"props":735,"children":736},{"style":344},[737],{"type":48,"value":389},{"type":42,"tag":331,"props":739,"children":740},{"style":349},[741],{"type":48,"value":742},"dev",{"type":42,"tag":331,"props":744,"children":745},{"style":344},[746],{"type":48,"value":747},",",{"type":42,"tag":331,"props":749,"children":750},{"style":349},[751],{"type":48,"value":752}," lts",{"type":42,"tag":331,"props":754,"children":755},{"style":344},[756],{"type":48,"value":399},{"type":42,"tag":331,"props":758,"children":760},{"class":333,"line":759},22,[761,766,770,774,779],{"type":42,"tag":331,"props":762,"children":763},{"style":338},[764],{"type":48,"value":765},"        scope",{"type":42,"tag":331,"props":767,"children":768},{"style":344},[769],{"type":48,"value":219},{"type":42,"tag":331,"props":771,"children":772},{"style":344},[773],{"type":48,"value":389},{"type":42,"tag":331,"props":775,"children":776},{"style":349},[777],{"type":48,"value":778},"mr-github",{"type":42,"tag":331,"props":780,"children":781},{"style":344},[782],{"type":48,"value":399},{"type":42,"tag":331,"props":784,"children":786},{"class":333,"line":785},23,[787,792,796,800,805],{"type":42,"tag":331,"props":788,"children":789},{"style":338},[790],{"type":48,"value":791},"        platforms",{"type":42,"tag":331,"props":793,"children":794},{"style":344},[795],{"type":48,"value":219},{"type":42,"tag":331,"props":797,"children":798},{"style":344},[799],{"type":48,"value":389},{"type":42,"tag":331,"props":801,"children":802},{"style":349},[803],{"type":48,"value":804},"dgx_h100",{"type":42,"tag":331,"props":806,"children":807},{"style":344},[808],{"type":48,"value":399},{"type":42,"tag":62,"props":810,"children":811},{},[812,814,820,822,828,829,835,837,843,844,850,851,857],{"type":48,"value":813},"Key runtime placeholders: ",{"type":42,"tag":78,"props":815,"children":817},{"className":816},[],[818],{"type":48,"value":819},"{assets_dir}",{"type":48,"value":821},", ",{"type":42,"tag":78,"props":823,"children":825},{"className":824},[],[826],{"type":48,"value":827},"{artifacts_dir}",{"type":48,"value":821},{"type":42,"tag":78,"props":830,"children":832},{"className":831},[],[833],{"type":48,"value":834},"{test_case}",{"type":48,"value":836},",\n",{"type":42,"tag":78,"props":838,"children":840},{"className":839},[],[841],{"type":48,"value":842},"{environment}",{"type":48,"value":821},{"type":42,"tag":78,"props":845,"children":847},{"className":846},[],[848],{"type":48,"value":849},"{platforms}",{"type":48,"value":821},{"type":42,"tag":78,"props":852,"children":854},{"className":853},[],[855],{"type":48,"value":856},"{n_repeat}",{"type":48,"value":101},{"type":42,"tag":859,"props":860,"children":862},"h3",{"id":861},"disabling-a-test-without-deleting-it",[863],{"type":48,"value":864},"Disabling a Test Without Deleting It",{"type":42,"tag":62,"props":866,"children":867},{},[868,870,876,878,883,885,890],{"type":48,"value":869},"To temporarily disable a test case in a recipe YAML, suffix its ",{"type":42,"tag":78,"props":871,"children":873},{"className":872},[],[874],{"type":48,"value":875},"scope",{"type":48,"value":877}," value\nwith ",{"type":42,"tag":78,"props":879,"children":881},{"className":880},[],[882],{"type":48,"value":83},{"type":48,"value":884}," — ",{"type":42,"tag":172,"props":886,"children":887},{},[888],{"type":48,"value":889},"do not delete the entry",{"type":48,"value":219},{"type":42,"tag":138,"props":892,"children":894},{"className":323,"code":893,"language":325,"meta":143,"style":143},"# before (test runs in CI)\nscope: [mr-github]\n\n# after (test is skipped; entry preserved for easy re-enable)\nscope: [mr-github-broken]\n",[895],{"type":42,"tag":78,"props":896,"children":897},{"__ignoreMap":143},[898,906,929,938,946],{"type":42,"tag":331,"props":899,"children":900},{"class":333,"line":334},[901],{"type":42,"tag":331,"props":902,"children":903},{"style":488},[904],{"type":48,"value":905},"# before (test runs in CI)\n",{"type":42,"tag":331,"props":907,"children":908},{"class":333,"line":355},[909,913,917,921,925],{"type":42,"tag":331,"props":910,"children":911},{"style":338},[912],{"type":48,"value":875},{"type":42,"tag":331,"props":914,"children":915},{"style":344},[916],{"type":48,"value":219},{"type":42,"tag":331,"props":918,"children":919},{"style":344},[920],{"type":48,"value":389},{"type":42,"tag":331,"props":922,"children":923},{"style":349},[924],{"type":48,"value":778},{"type":42,"tag":331,"props":926,"children":927},{"style":344},[928],{"type":48,"value":399},{"type":42,"tag":331,"props":930,"children":931},{"class":333,"line":374},[932],{"type":42,"tag":331,"props":933,"children":935},{"emptyLinePlaceholder":934},true,[936],{"type":48,"value":937},"\n",{"type":42,"tag":331,"props":939,"children":940},{"class":333,"line":402},[941],{"type":42,"tag":331,"props":942,"children":943},{"style":488},[944],{"type":48,"value":945},"# after (test is skipped; entry preserved for easy re-enable)\n",{"type":42,"tag":331,"props":947,"children":948},{"class":333,"line":428},[949,953,957,961,966],{"type":42,"tag":331,"props":950,"children":951},{"style":338},[952],{"type":48,"value":875},{"type":42,"tag":331,"props":954,"children":955},{"style":344},[956],{"type":48,"value":219},{"type":42,"tag":331,"props":958,"children":959},{"style":344},[960],{"type":48,"value":389},{"type":42,"tag":331,"props":962,"children":963},{"style":349},[964],{"type":48,"value":965},"mr-github-broken",{"type":42,"tag":331,"props":967,"children":968},{"style":344},[969],{"type":48,"value":399},{"type":42,"tag":51,"props":971,"children":972},{},[],{"type":42,"tag":55,"props":974,"children":976},{"id":975},"running-unit-tests-locally",[977],{"type":48,"value":978},"Running Unit Tests Locally",{"type":42,"tag":62,"props":980,"children":981},{},[982,984,990,992,997],{"type":48,"value":983},"All unit tests initialize a ",{"type":42,"tag":78,"props":985,"children":987},{"className":986},[],[988],{"type":48,"value":989},"torch.distributed",{"type":48,"value":991}," group, so every invocation\nrequires GPU access and must go through ",{"type":42,"tag":78,"props":993,"children":995},{"className":994},[],[996],{"type":48,"value":217},{"type":48,"value":219},{"type":42,"tag":138,"props":999,"children":1003},{"className":1000,"code":1001,"language":1002,"meta":143,"style":143},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Full suite\nuv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \\\n  tests\u002Funit_tests\n\n# Single file\nuv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \\\n  tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_model.py\n\n# Single test\nuv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \\\n  tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_model.py::TestGPTModel::test_constructor\n\n# Filter by name substring\nuv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \\\n  tests\u002Funit_tests -k optimizer\n","bash",[1004],{"type":42,"tag":78,"props":1005,"children":1006},{"__ignoreMap":143},[1007,1015,1074,1082,1089,1097,1144,1152,1159,1167,1214,1222,1229,1237,1284],{"type":42,"tag":331,"props":1008,"children":1009},{"class":333,"line":334},[1010],{"type":42,"tag":331,"props":1011,"children":1012},{"style":488},[1013],{"type":48,"value":1014},"# Full suite\n",{"type":42,"tag":331,"props":1016,"children":1017},{"class":333,"line":355},[1018,1024,1029,1034,1039,1044,1049,1054,1058,1063,1068],{"type":42,"tag":331,"props":1019,"children":1021},{"style":1020},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1022],{"type":48,"value":1023},"uv",{"type":42,"tag":331,"props":1025,"children":1026},{"style":349},[1027],{"type":48,"value":1028}," run",{"type":42,"tag":331,"props":1030,"children":1031},{"style":349},[1032],{"type":48,"value":1033}," python",{"type":42,"tag":331,"props":1035,"children":1036},{"style":349},[1037],{"type":48,"value":1038}," -m",{"type":42,"tag":331,"props":1040,"children":1041},{"style":349},[1042],{"type":48,"value":1043}," torch.distributed.run",{"type":42,"tag":331,"props":1045,"children":1046},{"style":349},[1047],{"type":48,"value":1048}," --nproc-per-node",{"type":42,"tag":331,"props":1050,"children":1051},{"style":368},[1052],{"type":48,"value":1053}," 8",{"type":42,"tag":331,"props":1055,"children":1056},{"style":349},[1057],{"type":48,"value":1038},{"type":42,"tag":331,"props":1059,"children":1060},{"style":349},[1061],{"type":48,"value":1062}," pytest",{"type":42,"tag":331,"props":1064,"children":1065},{"style":349},[1066],{"type":48,"value":1067}," -q",{"type":42,"tag":331,"props":1069,"children":1071},{"style":1070},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1072],{"type":48,"value":1073}," \\\n",{"type":42,"tag":331,"props":1075,"children":1076},{"class":333,"line":374},[1077],{"type":42,"tag":331,"props":1078,"children":1079},{"style":349},[1080],{"type":48,"value":1081},"  tests\u002Funit_tests\n",{"type":42,"tag":331,"props":1083,"children":1084},{"class":333,"line":402},[1085],{"type":42,"tag":331,"props":1086,"children":1087},{"emptyLinePlaceholder":934},[1088],{"type":48,"value":937},{"type":42,"tag":331,"props":1090,"children":1091},{"class":333,"line":428},[1092],{"type":42,"tag":331,"props":1093,"children":1094},{"style":488},[1095],{"type":48,"value":1096},"# Single file\n",{"type":42,"tag":331,"props":1098,"children":1099},{"class":333,"line":442},[1100,1104,1108,1112,1116,1120,1124,1128,1132,1136,1140],{"type":42,"tag":331,"props":1101,"children":1102},{"style":1020},[1103],{"type":48,"value":1023},{"type":42,"tag":331,"props":1105,"children":1106},{"style":349},[1107],{"type":48,"value":1028},{"type":42,"tag":331,"props":1109,"children":1110},{"style":349},[1111],{"type":48,"value":1033},{"type":42,"tag":331,"props":1113,"children":1114},{"style":349},[1115],{"type":48,"value":1038},{"type":42,"tag":331,"props":1117,"children":1118},{"style":349},[1119],{"type":48,"value":1043},{"type":42,"tag":331,"props":1121,"children":1122},{"style":349},[1123],{"type":48,"value":1048},{"type":42,"tag":331,"props":1125,"children":1126},{"style":368},[1127],{"type":48,"value":1053},{"type":42,"tag":331,"props":1129,"children":1130},{"style":349},[1131],{"type":48,"value":1038},{"type":42,"tag":331,"props":1133,"children":1134},{"style":349},[1135],{"type":48,"value":1062},{"type":42,"tag":331,"props":1137,"children":1138},{"style":349},[1139],{"type":48,"value":1067},{"type":42,"tag":331,"props":1141,"children":1142},{"style":1070},[1143],{"type":48,"value":1073},{"type":42,"tag":331,"props":1145,"children":1146},{"class":333,"line":470},[1147],{"type":42,"tag":331,"props":1148,"children":1149},{"style":349},[1150],{"type":48,"value":1151},"  tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_model.py\n",{"type":42,"tag":331,"props":1153,"children":1154},{"class":333,"line":494},[1155],{"type":42,"tag":331,"props":1156,"children":1157},{"emptyLinePlaceholder":934},[1158],{"type":48,"value":937},{"type":42,"tag":331,"props":1160,"children":1161},{"class":333,"line":512},[1162],{"type":42,"tag":331,"props":1163,"children":1164},{"style":488},[1165],{"type":48,"value":1166},"# Single test\n",{"type":42,"tag":331,"props":1168,"children":1169},{"class":333,"line":529},[1170,1174,1178,1182,1186,1190,1194,1198,1202,1206,1210],{"type":42,"tag":331,"props":1171,"children":1172},{"style":1020},[1173],{"type":48,"value":1023},{"type":42,"tag":331,"props":1175,"children":1176},{"style":349},[1177],{"type":48,"value":1028},{"type":42,"tag":331,"props":1179,"children":1180},{"style":349},[1181],{"type":48,"value":1033},{"type":42,"tag":331,"props":1183,"children":1184},{"style":349},[1185],{"type":48,"value":1038},{"type":42,"tag":331,"props":1187,"children":1188},{"style":349},[1189],{"type":48,"value":1043},{"type":42,"tag":331,"props":1191,"children":1192},{"style":349},[1193],{"type":48,"value":1048},{"type":42,"tag":331,"props":1195,"children":1196},{"style":368},[1197],{"type":48,"value":1053},{"type":42,"tag":331,"props":1199,"children":1200},{"style":349},[1201],{"type":48,"value":1038},{"type":42,"tag":331,"props":1203,"children":1204},{"style":349},[1205],{"type":48,"value":1062},{"type":42,"tag":331,"props":1207,"children":1208},{"style":349},[1209],{"type":48,"value":1067},{"type":42,"tag":331,"props":1211,"children":1212},{"style":1070},[1213],{"type":48,"value":1073},{"type":42,"tag":331,"props":1215,"children":1216},{"class":333,"line":547},[1217],{"type":42,"tag":331,"props":1218,"children":1219},{"style":349},[1220],{"type":48,"value":1221},"  tests\u002Funit_tests\u002Fmodels\u002Ftest_gpt_model.py::TestGPTModel::test_constructor\n",{"type":42,"tag":331,"props":1223,"children":1224},{"class":333,"line":565},[1225],{"type":42,"tag":331,"props":1226,"children":1227},{"emptyLinePlaceholder":934},[1228],{"type":48,"value":937},{"type":42,"tag":331,"props":1230,"children":1231},{"class":333,"line":583},[1232],{"type":42,"tag":331,"props":1233,"children":1234},{"style":488},[1235],{"type":48,"value":1236},"# Filter by name substring\n",{"type":42,"tag":331,"props":1238,"children":1239},{"class":333,"line":601},[1240,1244,1248,1252,1256,1260,1264,1268,1272,1276,1280],{"type":42,"tag":331,"props":1241,"children":1242},{"style":1020},[1243],{"type":48,"value":1023},{"type":42,"tag":331,"props":1245,"children":1246},{"style":349},[1247],{"type":48,"value":1028},{"type":42,"tag":331,"props":1249,"children":1250},{"style":349},[1251],{"type":48,"value":1033},{"type":42,"tag":331,"props":1253,"children":1254},{"style":349},[1255],{"type":48,"value":1038},{"type":42,"tag":331,"props":1257,"children":1258},{"style":349},[1259],{"type":48,"value":1043},{"type":42,"tag":331,"props":1261,"children":1262},{"style":349},[1263],{"type":48,"value":1048},{"type":42,"tag":331,"props":1265,"children":1266},{"style":368},[1267],{"type":48,"value":1053},{"type":42,"tag":331,"props":1269,"children":1270},{"style":349},[1271],{"type":48,"value":1038},{"type":42,"tag":331,"props":1273,"children":1274},{"style":349},[1275],{"type":48,"value":1062},{"type":42,"tag":331,"props":1277,"children":1278},{"style":349},[1279],{"type":48,"value":1067},{"type":42,"tag":331,"props":1281,"children":1282},{"style":1070},[1283],{"type":48,"value":1073},{"type":42,"tag":331,"props":1285,"children":1286},{"class":333,"line":620},[1287,1292,1297],{"type":42,"tag":331,"props":1288,"children":1289},{"style":349},[1290],{"type":48,"value":1291},"  tests\u002Funit_tests",{"type":42,"tag":331,"props":1293,"children":1294},{"style":349},[1295],{"type":48,"value":1296}," -k",{"type":42,"tag":331,"props":1298,"children":1299},{"style":349},[1300],{"type":48,"value":1301}," optimizer\n",{"type":42,"tag":859,"props":1303,"children":1305},{"id":1304},"marker-filters",[1306],{"type":48,"value":1307},"Marker filters",{"type":42,"tag":138,"props":1309,"children":1311},{"className":1000,"code":1310,"language":1002,"meta":143,"style":143},"# Exclude flaky tests during development\nuv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \\\n  tests\u002Funit_tests -m \"not flaky and not flaky_in_dev\"\n\n# Include experimental tests\nuv run python -m torch.distributed.run --nproc-per-node 8 -m pytest -q \\\n  tests\u002Funit_tests --experimental\n",[1312],{"type":42,"tag":78,"props":1313,"children":1314},{"__ignoreMap":143},[1315,1323,1370,1394,1401,1409,1456],{"type":42,"tag":331,"props":1316,"children":1317},{"class":333,"line":334},[1318],{"type":42,"tag":331,"props":1319,"children":1320},{"style":488},[1321],{"type":48,"value":1322},"# Exclude flaky tests during development\n",{"type":42,"tag":331,"props":1324,"children":1325},{"class":333,"line":355},[1326,1330,1334,1338,1342,1346,1350,1354,1358,1362,1366],{"type":42,"tag":331,"props":1327,"children":1328},{"style":1020},[1329],{"type":48,"value":1023},{"type":42,"tag":331,"props":1331,"children":1332},{"style":349},[1333],{"type":48,"value":1028},{"type":42,"tag":331,"props":1335,"children":1336},{"style":349},[1337],{"type":48,"value":1033},{"type":42,"tag":331,"props":1339,"children":1340},{"style":349},[1341],{"type":48,"value":1038},{"type":42,"tag":331,"props":1343,"children":1344},{"style":349},[1345],{"type":48,"value":1043},{"type":42,"tag":331,"props":1347,"children":1348},{"style":349},[1349],{"type":48,"value":1048},{"type":42,"tag":331,"props":1351,"children":1352},{"style":368},[1353],{"type":48,"value":1053},{"type":42,"tag":331,"props":1355,"children":1356},{"style":349},[1357],{"type":48,"value":1038},{"type":42,"tag":331,"props":1359,"children":1360},{"style":349},[1361],{"type":48,"value":1062},{"type":42,"tag":331,"props":1363,"children":1364},{"style":349},[1365],{"type":48,"value":1067},{"type":42,"tag":331,"props":1367,"children":1368},{"style":1070},[1369],{"type":48,"value":1073},{"type":42,"tag":331,"props":1371,"children":1372},{"class":333,"line":374},[1373,1377,1381,1385,1390],{"type":42,"tag":331,"props":1374,"children":1375},{"style":349},[1376],{"type":48,"value":1291},{"type":42,"tag":331,"props":1378,"children":1379},{"style":349},[1380],{"type":48,"value":1038},{"type":42,"tag":331,"props":1382,"children":1383},{"style":344},[1384],{"type":48,"value":457},{"type":42,"tag":331,"props":1386,"children":1387},{"style":349},[1388],{"type":48,"value":1389},"not flaky and not flaky_in_dev",{"type":42,"tag":331,"props":1391,"children":1392},{"style":344},[1393],{"type":48,"value":467},{"type":42,"tag":331,"props":1395,"children":1396},{"class":333,"line":402},[1397],{"type":42,"tag":331,"props":1398,"children":1399},{"emptyLinePlaceholder":934},[1400],{"type":48,"value":937},{"type":42,"tag":331,"props":1402,"children":1403},{"class":333,"line":428},[1404],{"type":42,"tag":331,"props":1405,"children":1406},{"style":488},[1407],{"type":48,"value":1408},"# Include experimental tests\n",{"type":42,"tag":331,"props":1410,"children":1411},{"class":333,"line":442},[1412,1416,1420,1424,1428,1432,1436,1440,1444,1448,1452],{"type":42,"tag":331,"props":1413,"children":1414},{"style":1020},[1415],{"type":48,"value":1023},{"type":42,"tag":331,"props":1417,"children":1418},{"style":349},[1419],{"type":48,"value":1028},{"type":42,"tag":331,"props":1421,"children":1422},{"style":349},[1423],{"type":48,"value":1033},{"type":42,"tag":331,"props":1425,"children":1426},{"style":349},[1427],{"type":48,"value":1038},{"type":42,"tag":331,"props":1429,"children":1430},{"style":349},[1431],{"type":48,"value":1043},{"type":42,"tag":331,"props":1433,"children":1434},{"style":349},[1435],{"type":48,"value":1048},{"type":42,"tag":331,"props":1437,"children":1438},{"style":368},[1439],{"type":48,"value":1053},{"type":42,"tag":331,"props":1441,"children":1442},{"style":349},[1443],{"type":48,"value":1038},{"type":42,"tag":331,"props":1445,"children":1446},{"style":349},[1447],{"type":48,"value":1062},{"type":42,"tag":331,"props":1449,"children":1450},{"style":349},[1451],{"type":48,"value":1067},{"type":42,"tag":331,"props":1453,"children":1454},{"style":1070},[1455],{"type":48,"value":1073},{"type":42,"tag":331,"props":1457,"children":1458},{"class":333,"line":470},[1459,1463],{"type":42,"tag":331,"props":1460,"children":1461},{"style":349},[1462],{"type":48,"value":1291},{"type":42,"tag":331,"props":1464,"children":1465},{"style":349},[1466],{"type":48,"value":1467}," --experimental\n",{"type":42,"tag":859,"props":1469,"children":1471},{"id":1470},"ci-parity",[1472],{"type":48,"value":1473},"CI parity",{"type":42,"tag":62,"props":1475,"children":1476},{},[1477,1479,1485,1487,1492],{"type":48,"value":1478},"Use ",{"type":42,"tag":78,"props":1480,"children":1482},{"className":1481},[],[1483],{"type":48,"value":1484},"tests\u002Funit_tests\u002Frun_ci_test.sh",{"type":48,"value":1486}," to reproduce a CI bucket failure exactly.\nFor ad-hoc runs, prefer the direct ",{"type":42,"tag":78,"props":1488,"children":1490},{"className":1489},[],[1491],{"type":48,"value":217},{"type":48,"value":1493}," invocations above.",{"type":42,"tag":859,"props":1495,"children":1497},{"id":1496},"gotchas",[1498],{"type":48,"value":1499},"Gotchas",{"type":42,"tag":68,"props":1501,"children":1502},{},[1503,1538],{"type":42,"tag":72,"props":1504,"children":1505},{},[1506,1512,1514,1520,1522,1528,1530,1536],{"type":42,"tag":78,"props":1507,"children":1509},{"className":1508},[],[1510],{"type":48,"value":1511},"pyproject.toml",{"type":48,"value":1513}," sets ",{"type":42,"tag":78,"props":1515,"children":1517},{"className":1516},[],[1518],{"type":48,"value":1519},"addopts = --durations=15 -s -rA",{"type":48,"value":1521}," — stdout is not\ncaptured (",{"type":42,"tag":78,"props":1523,"children":1525},{"className":1524},[],[1526],{"type":48,"value":1527},"-s",{"type":48,"value":1529},"), so ranks interleave during multi-rank runs. Override with\n",{"type":42,"tag":78,"props":1531,"children":1533},{"className":1532},[],[1534],{"type":48,"value":1535},"--capture=fd",{"type":48,"value":1537}," when debugging a specific rank.",{"type":42,"tag":72,"props":1539,"children":1540},{},[1541,1547,1549,1555],{"type":42,"tag":78,"props":1542,"children":1544},{"className":1543},[],[1545],{"type":48,"value":1546},"tests\u002Funit_tests\u002Fconftest.py",{"type":48,"value":1548}," looks for test data under ",{"type":42,"tag":78,"props":1550,"children":1552},{"className":1551},[],[1553],{"type":48,"value":1554},"\u002Fopt\u002Fdata",{"type":48,"value":1556}," and\nattempts a download if missing. Supply it manually or skip data-dependent\ntests when running outside the canonical container.",{"type":42,"tag":51,"props":1558,"children":1559},{},[],{"type":42,"tag":55,"props":1561,"children":1563},{"id":1562},"adding-a-unit-test",[1564],{"type":48,"value":1565},"Adding a Unit Test",{"type":42,"tag":1567,"props":1568,"children":1569},"ol",{},[1570,1582,1593,1671,1676],{"type":42,"tag":72,"props":1571,"children":1572},{},[1573,1575,1581],{"type":48,"value":1574},"Create ",{"type":42,"tag":78,"props":1576,"children":1578},{"className":1577},[],[1579],{"type":48,"value":1580},"tests\u002Funit_tests\u002F\u003Ccategory>\u002Ftest_\u003Cname>.py",{"type":48,"value":101},{"type":42,"tag":72,"props":1583,"children":1584},{},[1585,1587,1592],{"type":48,"value":1586},"Use fixtures from ",{"type":42,"tag":78,"props":1588,"children":1590},{"className":1589},[],[1591],{"type":48,"value":1546},{"type":48,"value":101},{"type":42,"tag":72,"props":1594,"children":1595},{},[1596,1598],{"type":48,"value":1597},"Apply markers as needed:\n",{"type":42,"tag":68,"props":1599,"children":1600},{},[1601,1620,1637,1653],{"type":42,"tag":72,"props":1602,"children":1603},{},[1604,1610,1612,1618],{"type":42,"tag":78,"props":1605,"children":1607},{"className":1606},[],[1608],{"type":48,"value":1609},"@pytest.mark.internal",{"type":48,"value":1611}," — skipped on ",{"type":42,"tag":78,"props":1613,"children":1615},{"className":1614},[],[1616],{"type":48,"value":1617},"legacy",{"type":48,"value":1619}," tag",{"type":42,"tag":72,"props":1621,"children":1622},{},[1623,1628,1630,1635],{"type":42,"tag":78,"props":1624,"children":1626},{"className":1625},[],[1627],{"type":48,"value":112},{"type":48,"value":1629}," — skipped in ",{"type":42,"tag":78,"props":1631,"children":1633},{"className":1632},[],[1634],{"type":48,"value":742},{"type":48,"value":1636}," environment (CI default; use this to disable a flaky test without blocking the standard pipeline)",{"type":42,"tag":72,"props":1638,"children":1639},{},[1640,1645,1646,1652],{"type":42,"tag":78,"props":1641,"children":1643},{"className":1642},[],[1644],{"type":48,"value":120},{"type":48,"value":1629},{"type":42,"tag":78,"props":1647,"children":1649},{"className":1648},[],[1650],{"type":48,"value":1651},"lts",{"type":48,"value":729},{"type":42,"tag":72,"props":1654,"children":1655},{},[1656,1662,1663,1669],{"type":42,"tag":78,"props":1657,"children":1659},{"className":1658},[],[1660],{"type":48,"value":1661},"@pytest.mark.experimental",{"type":48,"value":884},{"type":42,"tag":78,"props":1664,"children":1666},{"className":1665},[],[1667],{"type":48,"value":1668},"latest",{"type":48,"value":1670}," tag only",{"type":42,"tag":72,"props":1672,"children":1673},{},[1674],{"type":48,"value":1675},"Verify locally (see Running Unit Tests Locally above).",{"type":42,"tag":72,"props":1677,"children":1678},{},[1679,1681,1687],{"type":48,"value":1680},"If the test needs a dedicated CI bucket, add an entry to\n",{"type":42,"tag":78,"props":1682,"children":1684},{"className":1683},[],[1685],{"type":48,"value":1686},"tests\u002Ftest_utils\u002Frecipes\u002Fh100\u002Funit-tests.yaml",{"type":48,"value":101},{"type":42,"tag":51,"props":1689,"children":1690},{},[],{"type":42,"tag":55,"props":1692,"children":1694},{"id":1693},"adding-a-functional-integration-test",[1695],{"type":48,"value":1696},"Adding a Functional \u002F Integration Test",{"type":42,"tag":1567,"props":1698,"children":1699},{},[1700,1711,1746,1801,1813,1880],{"type":42,"tag":72,"props":1701,"children":1702},{},[1703,1704,1710],{"type":48,"value":1574},{"type":42,"tag":78,"props":1705,"children":1707},{"className":1706},[],[1708],{"type":48,"value":1709},"tests\u002Ffunctional_tests\u002Ftest_cases\u002F\u003Cmodel>\u002F\u003Ctest_name>\u002F",{"type":48,"value":101},{"type":42,"tag":72,"props":1712,"children":1713},{},[1714,1716,1722,1724,1730,1731,1737,1739,1745],{"type":48,"value":1715},"Write ",{"type":42,"tag":78,"props":1717,"children":1719},{"className":1718},[],[1720],{"type":48,"value":1721},"model_config.yaml",{"type":48,"value":1723}," with ",{"type":42,"tag":78,"props":1725,"children":1727},{"className":1726},[],[1728],{"type":48,"value":1729},"MODEL_ARGS",{"type":48,"value":821},{"type":42,"tag":78,"props":1732,"children":1734},{"className":1733},[],[1735],{"type":48,"value":1736},"ENV_VARS",{"type":48,"value":1738},", and ",{"type":42,"tag":78,"props":1740,"children":1742},{"className":1741},[],[1743],{"type":48,"value":1744},"TEST_TYPE",{"type":48,"value":101},{"type":42,"tag":72,"props":1747,"children":1748},{},[1749,1751,1757,1759,1765,1767,1772,1773,1779,1780,1786,1787,1793,1794,1800],{"type":48,"value":1750},"Add a YAML recipe under ",{"type":42,"tag":78,"props":1752,"children":1754},{"className":1753},[],[1755],{"type":48,"value":1756},"tests\u002Ftest_utils\u002Frecipes\u002Fh100\u002F",{"type":48,"value":1758}," (and ",{"type":42,"tag":78,"props":1760,"children":1762},{"className":1761},[],[1763],{"type":48,"value":1764},"gb200\u002F",{"type":48,"value":1766}," if\nneeded). Required fields: ",{"type":42,"tag":78,"props":1768,"children":1770},{"className":1769},[],[1771],{"type":48,"value":875},{"type":48,"value":821},{"type":42,"tag":78,"props":1774,"children":1776},{"className":1775},[],[1777],{"type":48,"value":1778},"environment",{"type":48,"value":821},{"type":42,"tag":78,"props":1781,"children":1783},{"className":1782},[],[1784],{"type":48,"value":1785},"platform",{"type":48,"value":821},{"type":42,"tag":78,"props":1788,"children":1790},{"className":1789},[],[1791],{"type":48,"value":1792},"n_repeat",{"type":48,"value":836},{"type":42,"tag":78,"props":1795,"children":1797},{"className":1796},[],[1798],{"type":48,"value":1799},"time_limit",{"type":48,"value":101},{"type":42,"tag":72,"props":1802,"children":1803},{},[1804,1806,1811],{"type":48,"value":1805},"Push the PR, add the label ",{"type":42,"tag":172,"props":1807,"children":1808},{},[1809],{"type":48,"value":1810},"\"Run functional tests\"",{"type":48,"value":1812}," to trigger a full run.",{"type":42,"tag":72,"props":1814,"children":1815},{},[1816,1818],{"type":48,"value":1817},"After a successful run, download golden values:",{"type":42,"tag":138,"props":1819,"children":1821},{"className":1000,"code":1820,"language":1002,"meta":143,"style":143},"python tests\u002Ftest_utils\u002Fpython_scripts\u002Fdownload_golden_values.py \\\n  --source github --pipeline-id \u003Crun-id>\n",[1822],{"type":42,"tag":78,"props":1823,"children":1824},{"__ignoreMap":143},[1825,1842],{"type":42,"tag":331,"props":1826,"children":1827},{"class":333,"line":334},[1828,1833,1838],{"type":42,"tag":331,"props":1829,"children":1830},{"style":1020},[1831],{"type":48,"value":1832},"python",{"type":42,"tag":331,"props":1834,"children":1835},{"style":349},[1836],{"type":48,"value":1837}," tests\u002Ftest_utils\u002Fpython_scripts\u002Fdownload_golden_values.py",{"type":42,"tag":331,"props":1839,"children":1840},{"style":1070},[1841],{"type":48,"value":1073},{"type":42,"tag":331,"props":1843,"children":1844},{"class":333,"line":355},[1845,1850,1855,1860,1865,1870,1875],{"type":42,"tag":331,"props":1846,"children":1847},{"style":349},[1848],{"type":48,"value":1849},"  --source",{"type":42,"tag":331,"props":1851,"children":1852},{"style":349},[1853],{"type":48,"value":1854}," github",{"type":42,"tag":331,"props":1856,"children":1857},{"style":349},[1858],{"type":48,"value":1859}," --pipeline-id",{"type":42,"tag":331,"props":1861,"children":1862},{"style":344},[1863],{"type":48,"value":1864}," \u003C",{"type":42,"tag":331,"props":1866,"children":1867},{"style":349},[1868],{"type":48,"value":1869},"run-i",{"type":42,"tag":331,"props":1871,"children":1872},{"style":1070},[1873],{"type":48,"value":1874},"d",{"type":42,"tag":331,"props":1876,"children":1877},{"style":344},[1878],{"type":48,"value":1879},">\n",{"type":42,"tag":72,"props":1881,"children":1882},{},[1883],{"type":48,"value":1884},"Commit the downloaded golden values.",{"type":42,"tag":51,"props":1886,"children":1887},{},[],{"type":42,"tag":55,"props":1889,"children":1891},{"id":1890},"common-pitfalls",[1892],{"type":48,"value":1893},"Common Pitfalls",{"type":42,"tag":1895,"props":1896,"children":1897},"table",{},[1898,1922],{"type":42,"tag":1899,"props":1900,"children":1901},"thead",{},[1902],{"type":42,"tag":1903,"props":1904,"children":1905},"tr",{},[1906,1912,1917],{"type":42,"tag":1907,"props":1908,"children":1909},"th",{},[1910],{"type":48,"value":1911},"Problem",{"type":42,"tag":1907,"props":1913,"children":1914},{},[1915],{"type":48,"value":1916},"Cause",{"type":42,"tag":1907,"props":1918,"children":1919},{},[1920],{"type":48,"value":1921},"Fix",{"type":42,"tag":1923,"props":1924,"children":1925},"tbody",{},[1926,1981,2007],{"type":42,"tag":1903,"props":1927,"children":1928},{},[1929,1935,1940],{"type":42,"tag":1930,"props":1931,"children":1932},"td",{},[1933],{"type":48,"value":1934},"Test passes locally but fails in CI",{"type":42,"tag":1930,"props":1936,"children":1937},{},[1938],{"type":48,"value":1939},"Different environment or data path",{"type":42,"tag":1930,"props":1941,"children":1942},{},[1943,1945,1951,1952,1958,1960,1965,1967,1972,1974,1979],{"type":48,"value":1944},"Check ",{"type":42,"tag":78,"props":1946,"children":1948},{"className":1947},[],[1949],{"type":48,"value":1950},"DATA_PATH",{"type":48,"value":821},{"type":42,"tag":78,"props":1953,"children":1955},{"className":1954},[],[1956],{"type":48,"value":1957},"DATA_CACHE_PATH",{"type":48,"value":1959},", and the ",{"type":42,"tag":78,"props":1961,"children":1963},{"className":1962},[],[1964],{"type":48,"value":1778},{"type":48,"value":1966}," tag (",{"type":42,"tag":78,"props":1968,"children":1970},{"className":1969},[],[1971],{"type":48,"value":742},{"type":48,"value":1973}," vs ",{"type":42,"tag":78,"props":1975,"children":1977},{"className":1976},[],[1978],{"type":48,"value":1651},{"type":48,"value":1980},")",{"type":42,"tag":1903,"props":1982,"children":1983},{},[1984,1989,1994],{"type":42,"tag":1930,"props":1985,"children":1986},{},[1987],{"type":48,"value":1988},"Golden value mismatch after a code change",{"type":42,"tag":1930,"props":1990,"children":1991},{},[1992],{"type":48,"value":1993},"Numerical regression",{"type":42,"tag":1930,"props":1995,"children":1996},{},[1997,1999,2005],{"type":48,"value":1998},"Download new golden values via ",{"type":42,"tag":78,"props":2000,"children":2002},{"className":2001},[],[2003],{"type":48,"value":2004},"download_golden_values.py",{"type":48,"value":2006}," after a clean run",{"type":42,"tag":1903,"props":2008,"children":2009},{},[2010,2021,2026],{"type":42,"tag":1930,"props":2011,"children":2012},{},[2013,2019],{"type":42,"tag":78,"props":2014,"children":2016},{"className":2015},[],[2017],{"type":48,"value":2018},"cicd-integration-tests-gb200",{"type":48,"value":2020}," not triggered",{"type":42,"tag":1930,"props":2022,"children":2023},{},[2024],{"type":48,"value":2025},"GB200 jobs require maintainer status",{"type":42,"tag":1930,"props":2027,"children":2028},{},[2029,2031,2037],{"type":48,"value":2030},"Ask a maintainer to trigger, or add the ",{"type":42,"tag":78,"props":2032,"children":2034},{"className":2033},[],[2035],{"type":48,"value":2036},"Run functional tests",{"type":48,"value":2038}," label",{"type":42,"tag":2040,"props":2041,"children":2042},"style",{},[2043],{"type":48,"value":2044},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2046,"total":2194},[2047,2065,2080,2091,2103,2117,2130,2144,2153,2164,2178,2183],{"slug":2048,"name":2048,"fn":2049,"description":2050,"org":2051,"tags":2052,"stars":2062,"repoUrl":2063,"updatedAt":2064},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2053,2056,2059],{"name":2054,"slug":2055,"type":15},"Documentation","documentation",{"name":2057,"slug":2058,"type":15},"MCP","mcp",{"name":2060,"slug":2061,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2066,"name":2066,"fn":2067,"description":2068,"org":2069,"tags":2070,"stars":19,"repoUrl":20,"updatedAt":2079},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2071,2074,2077],{"name":2072,"slug":2073,"type":15},"Containers","containers",{"name":2075,"slug":2076,"type":15},"Deployment","deployment",{"name":2078,"slug":1832,"type":15},"Python","2026-07-27T06:06:11.249662",{"slug":2081,"name":2081,"fn":2082,"description":2083,"org":2084,"tags":2085,"stars":19,"repoUrl":20,"updatedAt":2090},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2086,2089],{"name":2087,"slug":2088,"type":15},"CI\u002FCD","ci-cd",{"name":2075,"slug":2076,"type":15},"2026-07-14T05:25:59.97109",{"slug":2092,"name":2092,"fn":2093,"description":2094,"org":2095,"tags":2096,"stars":19,"repoUrl":20,"updatedAt":2102},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2097,2098,2099],{"name":2087,"slug":2088,"type":15},{"name":2075,"slug":2076,"type":15},{"name":2100,"slug":2101,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2104,"name":2104,"fn":2105,"description":2106,"org":2107,"tags":2108,"stars":19,"repoUrl":20,"updatedAt":2116},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2109,2112,2113],{"name":2110,"slug":2111,"type":15},"Debugging","debugging",{"name":2100,"slug":2101,"type":15},{"name":2114,"slug":2115,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2118,"name":2118,"fn":2119,"description":2120,"org":2121,"tags":2122,"stars":19,"repoUrl":20,"updatedAt":2129},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2123,2126],{"name":2124,"slug":2125,"type":15},"Best Practices","best-practices",{"name":2127,"slug":2128,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2131,"name":2131,"fn":2132,"description":2133,"org":2134,"tags":2135,"stars":19,"repoUrl":20,"updatedAt":2143},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2136,2139,2142],{"name":2137,"slug":2138,"type":15},"Machine Learning","machine-learning",{"name":2140,"slug":2141,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2145,"name":2145,"fn":2146,"description":2147,"org":2148,"tags":2149,"stars":19,"repoUrl":20,"updatedAt":2152},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2150,2151],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},"2026-07-14T05:25:53.673039",{"slug":2154,"name":2154,"fn":2155,"description":2156,"org":2157,"tags":2158,"stars":19,"repoUrl":20,"updatedAt":2163},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2159,2160],{"name":2075,"slug":2076,"type":15},{"name":2161,"slug":2162,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":2165,"name":2165,"fn":2166,"description":2167,"org":2168,"tags":2169,"stars":19,"repoUrl":20,"updatedAt":2177},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2170,2173,2174],{"name":2171,"slug":2172,"type":15},"Code Review","code-review",{"name":2100,"slug":2101,"type":15},{"name":2175,"slug":2176,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":4,"name":4,"fn":5,"description":6,"org":2179,"tags":2180,"stars":19,"repoUrl":20,"updatedAt":21},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2181,2182],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"slug":2184,"name":2184,"fn":2185,"description":2186,"org":2187,"tags":2188,"stars":19,"repoUrl":20,"updatedAt":2193},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2189,2192],{"name":2190,"slug":2191,"type":15},"Automation","automation",{"name":2087,"slug":2088,"type":15},"2026-07-30T05:29:03.275638",496,{"items":2196,"total":583},[2197,2203,2208,2214,2220,2225,2231],{"slug":2066,"name":2066,"fn":2067,"description":2068,"org":2198,"tags":2199,"stars":19,"repoUrl":20,"updatedAt":2079},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2200,2201,2202],{"name":2072,"slug":2073,"type":15},{"name":2075,"slug":2076,"type":15},{"name":2078,"slug":1832,"type":15},{"slug":2081,"name":2081,"fn":2082,"description":2083,"org":2204,"tags":2205,"stars":19,"repoUrl":20,"updatedAt":2090},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2206,2207],{"name":2087,"slug":2088,"type":15},{"name":2075,"slug":2076,"type":15},{"slug":2092,"name":2092,"fn":2093,"description":2094,"org":2209,"tags":2210,"stars":19,"repoUrl":20,"updatedAt":2102},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2211,2212,2213],{"name":2087,"slug":2088,"type":15},{"name":2075,"slug":2076,"type":15},{"name":2100,"slug":2101,"type":15},{"slug":2104,"name":2104,"fn":2105,"description":2106,"org":2215,"tags":2216,"stars":19,"repoUrl":20,"updatedAt":2116},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2217,2218,2219],{"name":2110,"slug":2111,"type":15},{"name":2100,"slug":2101,"type":15},{"name":2114,"slug":2115,"type":15},{"slug":2118,"name":2118,"fn":2119,"description":2120,"org":2221,"tags":2222,"stars":19,"repoUrl":20,"updatedAt":2129},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2223,2224],{"name":2124,"slug":2125,"type":15},{"name":2127,"slug":2128,"type":15},{"slug":2131,"name":2131,"fn":2132,"description":2133,"org":2226,"tags":2227,"stars":19,"repoUrl":20,"updatedAt":2143},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2228,2229,2230],{"name":2137,"slug":2138,"type":15},{"name":2140,"slug":2141,"type":15},{"name":9,"slug":8,"type":15},{"slug":2145,"name":2145,"fn":2146,"description":2147,"org":2232,"tags":2233,"stars":19,"repoUrl":20,"updatedAt":2152},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2234,2235],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15}]