[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nemo-automodel-recipe-development":3,"mdc--qjnokp-key":34,"related-repo-nvidia-nemo-automodel-recipe-development":2505,"related-org-nvidia-nemo-automodel-recipe-development":2611},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"nemo-automodel-recipe-development","develop NeMo AutoModel training recipes","Create and modify NeMo AutoModel training and evaluation recipes, including YAML structure, builders, and execution flow.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"Configuration","configuration","tag",{"name":17,"slug":18,"type":15},"Machine Learning","machine-learning",{"name":20,"slug":21,"type":15},"YAML","yaml",{"name":9,"slug":8,"type":15},2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:27:41.609632","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fnemo-automodel-recipe-development","---\nname: nemo-automodel-recipe-development\ndescription: Create and modify NeMo AutoModel training and evaluation recipes, including YAML structure, builders, and execution flow.\nwhen_to_use: Creating or modifying training, SFT, or eval recipes, adding new YAML config fields, debugging recipe construction or trainer issues, or understanding the recipe execution flow.\nlicense: Apache-2.0\nmetadata:\n  author: NVIDIA\n  tags:\n    - nemo-automodel\n    - recipe-development\n---\n\n# NeMo AutoModel Recipe Development\n\u003C!-- NVSkills signature refresh requested for AM-519. -->\n\n## Instructions\n\nFor recipe questions, answer with the smallest complete path to action:\n\n1. Name the relevant recipe file or YAML section.\n2. List the builder functions or config keys involved.\n3. Include a minimal YAML or command example when the question asks how to\n   configure something.\n4. End with a local validation command or tiny CPU-compatible test.\n\nFor conceptual recipe questions, answer from this skill without inspecting the\nrepository or loading other AutoModel skills unless the user asks you to edit\nfiles. Keep the response focused on recipe YAML, builders, CLI routing, tests,\nand local validation.\n\nUse these compact answer patterns for common questions:\n\n- New finetuning recipe variant: start from the closest file under\n  `nemo_automodel\u002Frecipes\u002F`, update the model, dataset or dataloader,\n  optimizer, loss, LR scheduler, step scheduler, and checkpoint builders,\n  register a CLI route only if adding a command or domain alias, add example\n  YAML under `examples\u002F`, then add a tiny CPU-compatible unit test and run\n  `automodel finetune llm -c \u003Cconfig.yaml>`.\n- `_target_` fields: describe `_target_` as the fully qualified Python callable,\n  explain that sibling keys become keyword arguments, show optimizer and dataset\n  examples, and mention nested CLI overrides such as `--optimizer.lr`.\n- Validation and checkpointing: name `step_scheduler.val_check_interval`,\n  `step_scheduler.checkpoint_interval`, `validation_dataset`,\n  `restore_from.path`, and consolidated safetensors; include the minimal YAML\n  snippet from this skill.\n\nFor validation and checkpointing, always name:\n\n- `step_scheduler.val_check_interval` for validation cadence.\n- `step_scheduler.checkpoint_interval` for save cadence.\n- `validation_dataset` as the validation dataloader source.\n- `restore_from.path` for resume.\n- Consolidated safetensors as the default checkpoint format for HF ecosystem\n  compatibility.\n\n## Routing Boundary\n\nUse this skill for recipe construction and execution-flow questions: YAML\nstructure, `_target_` callables, builder functions, validation datasets,\ncheckpoint configuration, CLI route registration, and recipe-specific tests.\n\nDo not use this skill for standalone distributed strategy selection, cluster\nlauncher configuration, or model architecture onboarding unless the user is\nasking how those choices appear inside an AutoModel recipe YAML.\n\n## Recipe Architecture\n\n### Execution Flow\n\n```\nCLI (automodel finetune llm -c config.yaml)\n  -> app.py parses command + domain + config\n    -> recipe script (e.g. train_ft.py) main(config_path)\n      -> Recipe class .setup() builds all components\n        -> .run_train_validation_loop() executes training\n```\n\n### Recipe Class\n\nRecipes inherit from `BaseRecipe` and implement two methods:\n\n- `setup()` -- builds model, optimizer, dataloader, loss, LR scheduler, step scheduler, and checkpoint config via builder functions.\n- `run_train_validation_loop()` -- executes the training and validation loop.\n\n### Builder Pattern\n\nAll components are constructed through dedicated builder functions:\n\n- `build_model()` -- instantiates the model from config\n- `build_optimizer()` -- creates optimizer (AdamW, etc.)\n- `build_dataloader()` -- sets up train and validation dataloaders\n- `build_loss_module()` -- creates the loss function\n- `build_lr_scheduler()` -- creates the learning rate scheduler\n- `build_step_scheduler()` -- creates the step scheduler controlling training progression\n- `CheckpointingConfig` -- configures checkpointing (built directly from the YAML `checkpoint:` block via `RecipeConfig.checkpoint`)\n\n### Infrastructure Application Order\n\nComponents are applied in this strict order after building:\n\n1. PEFT (LoRA, etc.)\n2. FP8 quantization\n3. QAT (quantization-aware training)\n4. Checkpoint load \u002F restore\n5. Parameter freezing\n6. Sharding (FSDP2, Megatron-FSDP, DDP)\n7. Device placement\n8. `torch.compile`\n9. Context parallelism hooks\n\n## YAML Config Anatomy\n\nA complete recipe config follows this structure:\n\n```yaml\nstep_scheduler:\n  max_steps: 1000\n  num_epochs: 1\n  grad_accumulation_steps: 4\n  val_check_interval: 100\n  checkpoint_interval: 500\n  log_interval: 10\n\ndist_env:\n  master_addr: localhost\n  master_port: 29500\n\nrng:\n  seed: 42\n\nmodel:\n  _target_: nemo_automodel.models.llm.NemotronHForCausalLM\n  name_or_path: meta-llama\u002FLlama-3.2-1B\n  # additional model kwargs passed to the constructor\n\ncompile:\n  enabled: false\n  backend: inductor\n\nclip_grad_norm:\n  max_norm: 1.0\n\ndistributed:\n  strategy: fsdp2       # fsdp2 | megatron_fsdp | ddp\n  dp_size: auto\n  tp_size: 1\n  cp_size: 1\n\nloss_fn:\n  _target_: torch.nn.CrossEntropyLoss\n\ndataset:\n  _target_: nemo_automodel.datasets.squad.SquadDataset\n  tokenizer_name_or_path: meta-llama\u002FLlama-3.2-1B\n  max_seq_length: 2048\n\nvalidation_dataset:\n  _target_: nemo_automodel.datasets.squad.SquadDataset\n  split: validation\n\npacked_sequence:\n  enabled: false\n\ndataloader:\n  batch_size: 4\n  num_workers: 4\n  pin_memory: true\n\noptimizer:\n  _target_: torch.optim.AdamW\n  lr: 2.0e-5\n  weight_decay: 0.01\n\nlr_scheduler:\n  _target_: nemo_automodel.schedulers.CosineAnnealingWarmup\n  warmup_steps: 50\n  min_lr: 1.0e-6\n```\n\n### The `_target_` Pattern\n\nThe `_target_` key specifies a fully qualified Python callable. All remaining keys in that section are passed as keyword arguments:\n\n```yaml\noptimizer:\n  _target_: torch.optim.AdamW   # callable\n  lr: 2.0e-5                    # kwarg\n  weight_decay: 0.01            # kwarg\n```\n\nThis is equivalent to: `torch.optim.AdamW(lr=2e-5, weight_decay=0.01)`.\n\n### CLI Overrides\n\nAny config value can be overridden from the command line:\n\n```bash\nautomodel finetune llm -c config.yaml \\\n  --optimizer.lr 1e-4 \\\n  --step_scheduler.max_steps 500 \\\n  --distributed.tp_size 2\n```\n\n## Examples\n\nValidation and checkpointing:\n\n```yaml\nstep_scheduler:\n  val_check_interval: 100\n  checkpoint_interval: 500\n\nvalidation_dataset:\n  _target_: nemo_automodel.datasets.squad.SquadDataset\n  split: validation\n\nrestore_from:\n  path: \u002Fcheckpoints\u002Fstep-500\n```\n\n## Domain-Specific Notes\n\n### LLM\n\n- `nemo_automodel\u002Frecipes\u002Fllm\u002Ftrain_ft.py` handles both finetuning and pretraining. The distinction is in the config (dataset, learning rate, etc.).\n- `nemo_automodel\u002Frecipes\u002Fllm\u002Fkd.py` implements knowledge distillation with a teacher and student model.\n- `nemo_automodel\u002Frecipes\u002Fllm\u002Fbenchmark.py` runs throughput and latency benchmarks.\n\n### VLM\n\n- Uses `NeMoAutoModelForImageTextToText` instead of causal LM classes.\n- Config includes a `processor` section instead of a standalone tokenizer.\n- Recipe lives in `nemo_automodel\u002Frecipes\u002Fvlm\u002Ffinetune.py`.\n\n### Diffusion\n\n- Uses `NeMoAutoDiffusionPipeline`.\n- Requires a `parallel_scheme` dict in config to define parallelism.\n- Only supports DDP and FSDP2 strategies (no Megatron-FSDP).\n- Recipe lives in `nemo_automodel\u002Frecipes\u002Fdiffusion\u002Ftrain.py`.\n\n### Retrieval\n\n- Two encoder patterns:\n  - **Bi-encoder** (`nemo_automodel\u002Frecipes\u002Fretrieval\u002Ftrain_bi_encoder.py`): separate query and document encoders, contrastive loss.\n  - **Cross-encoder** (`nemo_automodel\u002Frecipes\u002Fretrieval\u002Ftrain_cross_encoder.py`): joint encoding, classification head.\n- Hard negative mining: `nemo_automodel\u002Frecipes\u002Fretrieval\u002Fmine_hard_negatives.py`.\n\n## Training Loop Details\n\nThe training loop follows this structure per epoch:\n\n```\nfor epoch in range(num_epochs):\n    for batch_idx in range(batches_per_epoch):\n        # --- gradient accumulation inner loop ---\n        for micro_batch in micro_batches:\n            if pipeline_parallel:\n                schedule.step(micro_batch)    # PP schedule\n            else:\n                loss = model(micro_batch)     # direct forward\n                loss.backward()\n\n        # --- optimizer step ---\n        scale_grads_and_clip_grad_norm(model, max_norm)\n        optimizer.step()\n        lr_scheduler.step()\n        optimizer.zero_grad()\n\n        # --- logging ---\n        MetricsSample(step, epoch, loss, grad_norm, lr, mem, tps, mfu)\n\n        # --- validation (at configured intervals) ---\n        if step % val_check_interval == 0:\n            run_validation()\n\n        # --- checkpoint (at configured intervals) ---\n        if step % checkpoint_interval == 0:\n            save_checkpoint()\n```\n\n### StepScheduler\n\nControls all training progression: total epochs, total steps, gradient accumulation steps, validation interval, checkpoint interval, and logging interval.\n\n### Gradient Clipping\n\nApplied via `scale_grads_and_clip_grad_norm()` after the backward pass and before the optimizer step. Controlled by `clip_grad_norm.max_norm` in config.\n\n### Context Parallelism\n\nWhen `cp_size > 1`, batches are split across the context-parallel group using `make_cp_batch_and_ctx()`. This must happen before the forward pass.\n\n### MetricsSample\n\nEach training step produces a `MetricsSample` with fields:\n\n- `step` -- global step count\n- `epoch` -- current epoch\n- `loss` -- training loss\n- `grad_norm` -- gradient norm after clipping\n- `lr` -- current learning rate\n- `mem` -- GPU memory usage\n- `tps` -- tokens per second\n- `mfu` -- model FLOPS utilization\n\n## Validation & Checkpointing\n\n### Validation\n\n- Runs at intervals defined by `step_scheduler.val_check_interval`.\n- Uses the validation dataloader built from `validation_dataset` config.\n- Model is set to eval mode; gradients are disabled.\n\n### Checkpointing\n\n- Default format: consolidated safetensors for easy deployment on HF ecosystem (always prefer this over DCP).\n- Checkpoint interval controlled by `step_scheduler.checkpoint_interval`.\n- Resume training via the `restore_from` config key pointing to a checkpoint directory.\n\n```yaml\nrestore_from:\n  path: \u002Fcheckpoints\u002Fstep-500\n```\n\n## Pitfalls\n\n| Problem | Cause | Fix |\n|---|---|---|\n| Silent config errors | Typo in `_target_` value | The class path must be a valid, importable Python callable. Double-check the module path and class name. |\n| Training crashes at first step | `global_batch_size` not divisible by `local_batch_size * dp_size * grad_accumulation_steps` | Ensure the batch size math is consistent across all dimensions. |\n| New recipe not accessible via CLI | Missing CLI command alias registration | Register the new route in the CLI app so `automodel \u003Ccommand> \u003Cdomain>` resolves correctly. |\n| Shape mismatch at forward pass | Dataset collate function output does not match model input signature | Verify that the collate function returns tensors with the keys and shapes the model expects. |\n| OOM during validation | Validation batch size too large or gradients not disabled | Wrap validation in `torch.no_grad()` and consider a smaller validation batch size. |\n| Checkpoint restore fails | Mismatched model architecture between checkpoint and config | Ensure the model config matches the checkpoint exactly (layer count, hidden dim, vocab size). |\n",{"data":35,"body":41},{"name":4,"description":6,"when_to_use":36,"license":26,"metadata":37},"Creating or modifying training, SFT, or eval recipes, adding new YAML config fields, debugging recipe construction or trainer issues, or understanding the recipe execution flow.",{"author":9,"tags":38},[39,40],"nemo-automodel","recipe-development",{"type":42,"children":43},"root",[44,52,59,65,90,95,100,195,200,248,254,266,271,277,284,296,302,315,340,346,351,447,453,458,510,516,521,1439,1452,1463,1544,1556,1562,1567,1658,1664,1669,1801,1807,1813,1849,1855,1896,1902,1945,1951,2010,2016,2021,2030,2036,2041,2047,2068,2074,2095,2101,2113,2204,2210,2216,2247,2253,2284,2317,2323,2499],{"type":45,"tag":46,"props":47,"children":48},"element","h1",{"id":4},[49],{"type":50,"value":51},"text","NeMo AutoModel Recipe Development",{"type":45,"tag":53,"props":54,"children":56},"h2",{"id":55},"instructions",[57],{"type":50,"value":58},"Instructions",{"type":45,"tag":60,"props":61,"children":62},"p",{},[63],{"type":50,"value":64},"For recipe questions, answer with the smallest complete path to action:",{"type":45,"tag":66,"props":67,"children":68},"ol",{},[69,75,80,85],{"type":45,"tag":70,"props":71,"children":72},"li",{},[73],{"type":50,"value":74},"Name the relevant recipe file or YAML section.",{"type":45,"tag":70,"props":76,"children":77},{},[78],{"type":50,"value":79},"List the builder functions or config keys involved.",{"type":45,"tag":70,"props":81,"children":82},{},[83],{"type":50,"value":84},"Include a minimal YAML or command example when the question asks how to\nconfigure something.",{"type":45,"tag":70,"props":86,"children":87},{},[88],{"type":50,"value":89},"End with a local validation command or tiny CPU-compatible test.",{"type":45,"tag":60,"props":91,"children":92},{},[93],{"type":50,"value":94},"For conceptual recipe questions, answer from this skill without inspecting the\nrepository or loading other AutoModel skills unless the user asks you to edit\nfiles. Keep the response focused on recipe YAML, builders, CLI routing, tests,\nand local validation.",{"type":45,"tag":60,"props":96,"children":97},{},[98],{"type":50,"value":99},"Use these compact answer patterns for common questions:",{"type":45,"tag":101,"props":102,"children":103},"ul",{},[104,134,159],{"type":45,"tag":70,"props":105,"children":106},{},[107,109,116,118,124,126,132],{"type":50,"value":108},"New finetuning recipe variant: start from the closest file under\n",{"type":45,"tag":110,"props":111,"children":113},"code",{"className":112},[],[114],{"type":50,"value":115},"nemo_automodel\u002Frecipes\u002F",{"type":50,"value":117},", update the model, dataset or dataloader,\noptimizer, loss, LR scheduler, step scheduler, and checkpoint builders,\nregister a CLI route only if adding a command or domain alias, add example\nYAML under ",{"type":45,"tag":110,"props":119,"children":121},{"className":120},[],[122],{"type":50,"value":123},"examples\u002F",{"type":50,"value":125},", then add a tiny CPU-compatible unit test and run\n",{"type":45,"tag":110,"props":127,"children":129},{"className":128},[],[130],{"type":50,"value":131},"automodel finetune llm -c \u003Cconfig.yaml>",{"type":50,"value":133},".",{"type":45,"tag":70,"props":135,"children":136},{},[137,143,145,150,152,158],{"type":45,"tag":110,"props":138,"children":140},{"className":139},[],[141],{"type":50,"value":142},"_target_",{"type":50,"value":144}," fields: describe ",{"type":45,"tag":110,"props":146,"children":148},{"className":147},[],[149],{"type":50,"value":142},{"type":50,"value":151}," as the fully qualified Python callable,\nexplain that sibling keys become keyword arguments, show optimizer and dataset\nexamples, and mention nested CLI overrides such as ",{"type":45,"tag":110,"props":153,"children":155},{"className":154},[],[156],{"type":50,"value":157},"--optimizer.lr",{"type":50,"value":133},{"type":45,"tag":70,"props":160,"children":161},{},[162,164,170,172,178,180,186,187,193],{"type":50,"value":163},"Validation and checkpointing: name ",{"type":45,"tag":110,"props":165,"children":167},{"className":166},[],[168],{"type":50,"value":169},"step_scheduler.val_check_interval",{"type":50,"value":171},",\n",{"type":45,"tag":110,"props":173,"children":175},{"className":174},[],[176],{"type":50,"value":177},"step_scheduler.checkpoint_interval",{"type":50,"value":179},", ",{"type":45,"tag":110,"props":181,"children":183},{"className":182},[],[184],{"type":50,"value":185},"validation_dataset",{"type":50,"value":171},{"type":45,"tag":110,"props":188,"children":190},{"className":189},[],[191],{"type":50,"value":192},"restore_from.path",{"type":50,"value":194},", and consolidated safetensors; include the minimal YAML\nsnippet from this skill.",{"type":45,"tag":60,"props":196,"children":197},{},[198],{"type":50,"value":199},"For validation and checkpointing, always name:",{"type":45,"tag":101,"props":201,"children":202},{},[203,213,223,233,243],{"type":45,"tag":70,"props":204,"children":205},{},[206,211],{"type":45,"tag":110,"props":207,"children":209},{"className":208},[],[210],{"type":50,"value":169},{"type":50,"value":212}," for validation cadence.",{"type":45,"tag":70,"props":214,"children":215},{},[216,221],{"type":45,"tag":110,"props":217,"children":219},{"className":218},[],[220],{"type":50,"value":177},{"type":50,"value":222}," for save cadence.",{"type":45,"tag":70,"props":224,"children":225},{},[226,231],{"type":45,"tag":110,"props":227,"children":229},{"className":228},[],[230],{"type":50,"value":185},{"type":50,"value":232}," as the validation dataloader source.",{"type":45,"tag":70,"props":234,"children":235},{},[236,241],{"type":45,"tag":110,"props":237,"children":239},{"className":238},[],[240],{"type":50,"value":192},{"type":50,"value":242}," for resume.",{"type":45,"tag":70,"props":244,"children":245},{},[246],{"type":50,"value":247},"Consolidated safetensors as the default checkpoint format for HF ecosystem\ncompatibility.",{"type":45,"tag":53,"props":249,"children":251},{"id":250},"routing-boundary",[252],{"type":50,"value":253},"Routing Boundary",{"type":45,"tag":60,"props":255,"children":256},{},[257,259,264],{"type":50,"value":258},"Use this skill for recipe construction and execution-flow questions: YAML\nstructure, ",{"type":45,"tag":110,"props":260,"children":262},{"className":261},[],[263],{"type":50,"value":142},{"type":50,"value":265}," callables, builder functions, validation datasets,\ncheckpoint configuration, CLI route registration, and recipe-specific tests.",{"type":45,"tag":60,"props":267,"children":268},{},[269],{"type":50,"value":270},"Do not use this skill for standalone distributed strategy selection, cluster\nlauncher configuration, or model architecture onboarding unless the user is\nasking how those choices appear inside an AutoModel recipe YAML.",{"type":45,"tag":53,"props":272,"children":274},{"id":273},"recipe-architecture",[275],{"type":50,"value":276},"Recipe Architecture",{"type":45,"tag":278,"props":279,"children":281},"h3",{"id":280},"execution-flow",[282],{"type":50,"value":283},"Execution Flow",{"type":45,"tag":285,"props":286,"children":290},"pre",{"className":287,"code":289,"language":50},[288],"language-text","CLI (automodel finetune llm -c config.yaml)\n  -> app.py parses command + domain + config\n    -> recipe script (e.g. train_ft.py) main(config_path)\n      -> Recipe class .setup() builds all components\n        -> .run_train_validation_loop() executes training\n",[291],{"type":45,"tag":110,"props":292,"children":294},{"__ignoreMap":293},"",[295],{"type":50,"value":289},{"type":45,"tag":278,"props":297,"children":299},{"id":298},"recipe-class",[300],{"type":50,"value":301},"Recipe Class",{"type":45,"tag":60,"props":303,"children":304},{},[305,307,313],{"type":50,"value":306},"Recipes inherit from ",{"type":45,"tag":110,"props":308,"children":310},{"className":309},[],[311],{"type":50,"value":312},"BaseRecipe",{"type":50,"value":314}," and implement two methods:",{"type":45,"tag":101,"props":316,"children":317},{},[318,329],{"type":45,"tag":70,"props":319,"children":320},{},[321,327],{"type":45,"tag":110,"props":322,"children":324},{"className":323},[],[325],{"type":50,"value":326},"setup()",{"type":50,"value":328}," -- builds model, optimizer, dataloader, loss, LR scheduler, step scheduler, and checkpoint config via builder functions.",{"type":45,"tag":70,"props":330,"children":331},{},[332,338],{"type":45,"tag":110,"props":333,"children":335},{"className":334},[],[336],{"type":50,"value":337},"run_train_validation_loop()",{"type":50,"value":339}," -- executes the training and validation loop.",{"type":45,"tag":278,"props":341,"children":343},{"id":342},"builder-pattern",[344],{"type":50,"value":345},"Builder Pattern",{"type":45,"tag":60,"props":347,"children":348},{},[349],{"type":50,"value":350},"All components are constructed through dedicated builder functions:",{"type":45,"tag":101,"props":352,"children":353},{},[354,365,376,387,398,409,420],{"type":45,"tag":70,"props":355,"children":356},{},[357,363],{"type":45,"tag":110,"props":358,"children":360},{"className":359},[],[361],{"type":50,"value":362},"build_model()",{"type":50,"value":364}," -- instantiates the model from config",{"type":45,"tag":70,"props":366,"children":367},{},[368,374],{"type":45,"tag":110,"props":369,"children":371},{"className":370},[],[372],{"type":50,"value":373},"build_optimizer()",{"type":50,"value":375}," -- creates optimizer (AdamW, etc.)",{"type":45,"tag":70,"props":377,"children":378},{},[379,385],{"type":45,"tag":110,"props":380,"children":382},{"className":381},[],[383],{"type":50,"value":384},"build_dataloader()",{"type":50,"value":386}," -- sets up train and validation dataloaders",{"type":45,"tag":70,"props":388,"children":389},{},[390,396],{"type":45,"tag":110,"props":391,"children":393},{"className":392},[],[394],{"type":50,"value":395},"build_loss_module()",{"type":50,"value":397}," -- creates the loss function",{"type":45,"tag":70,"props":399,"children":400},{},[401,407],{"type":45,"tag":110,"props":402,"children":404},{"className":403},[],[405],{"type":50,"value":406},"build_lr_scheduler()",{"type":50,"value":408}," -- creates the learning rate scheduler",{"type":45,"tag":70,"props":410,"children":411},{},[412,418],{"type":45,"tag":110,"props":413,"children":415},{"className":414},[],[416],{"type":50,"value":417},"build_step_scheduler()",{"type":50,"value":419}," -- creates the step scheduler controlling training progression",{"type":45,"tag":70,"props":421,"children":422},{},[423,429,431,437,439,445],{"type":45,"tag":110,"props":424,"children":426},{"className":425},[],[427],{"type":50,"value":428},"CheckpointingConfig",{"type":50,"value":430}," -- configures checkpointing (built directly from the YAML ",{"type":45,"tag":110,"props":432,"children":434},{"className":433},[],[435],{"type":50,"value":436},"checkpoint:",{"type":50,"value":438}," block via ",{"type":45,"tag":110,"props":440,"children":442},{"className":441},[],[443],{"type":50,"value":444},"RecipeConfig.checkpoint",{"type":50,"value":446},")",{"type":45,"tag":278,"props":448,"children":450},{"id":449},"infrastructure-application-order",[451],{"type":50,"value":452},"Infrastructure Application Order",{"type":45,"tag":60,"props":454,"children":455},{},[456],{"type":50,"value":457},"Components are applied in this strict order after building:",{"type":45,"tag":66,"props":459,"children":460},{},[461,466,471,476,481,486,491,496,505],{"type":45,"tag":70,"props":462,"children":463},{},[464],{"type":50,"value":465},"PEFT (LoRA, etc.)",{"type":45,"tag":70,"props":467,"children":468},{},[469],{"type":50,"value":470},"FP8 quantization",{"type":45,"tag":70,"props":472,"children":473},{},[474],{"type":50,"value":475},"QAT (quantization-aware training)",{"type":45,"tag":70,"props":477,"children":478},{},[479],{"type":50,"value":480},"Checkpoint load \u002F restore",{"type":45,"tag":70,"props":482,"children":483},{},[484],{"type":50,"value":485},"Parameter freezing",{"type":45,"tag":70,"props":487,"children":488},{},[489],{"type":50,"value":490},"Sharding (FSDP2, Megatron-FSDP, DDP)",{"type":45,"tag":70,"props":492,"children":493},{},[494],{"type":50,"value":495},"Device placement",{"type":45,"tag":70,"props":497,"children":498},{},[499],{"type":45,"tag":110,"props":500,"children":502},{"className":501},[],[503],{"type":50,"value":504},"torch.compile",{"type":45,"tag":70,"props":506,"children":507},{},[508],{"type":50,"value":509},"Context parallelism hooks",{"type":45,"tag":53,"props":511,"children":513},{"id":512},"yaml-config-anatomy",[514],{"type":50,"value":515},"YAML Config Anatomy",{"type":45,"tag":60,"props":517,"children":518},{},[519],{"type":50,"value":520},"A complete recipe config follows this structure:",{"type":45,"tag":285,"props":522,"children":525},{"className":523,"code":524,"language":21,"meta":293,"style":293},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","step_scheduler:\n  max_steps: 1000\n  num_epochs: 1\n  grad_accumulation_steps: 4\n  val_check_interval: 100\n  checkpoint_interval: 500\n  log_interval: 10\n\ndist_env:\n  master_addr: localhost\n  master_port: 29500\n\nrng:\n  seed: 42\n\nmodel:\n  _target_: nemo_automodel.models.llm.NemotronHForCausalLM\n  name_or_path: meta-llama\u002FLlama-3.2-1B\n  # additional model kwargs passed to the constructor\n\ncompile:\n  enabled: false\n  backend: inductor\n\nclip_grad_norm:\n  max_norm: 1.0\n\ndistributed:\n  strategy: fsdp2       # fsdp2 | megatron_fsdp | ddp\n  dp_size: auto\n  tp_size: 1\n  cp_size: 1\n\nloss_fn:\n  _target_: torch.nn.CrossEntropyLoss\n\ndataset:\n  _target_: nemo_automodel.datasets.squad.SquadDataset\n  tokenizer_name_or_path: meta-llama\u002FLlama-3.2-1B\n  max_seq_length: 2048\n\nvalidation_dataset:\n  _target_: nemo_automodel.datasets.squad.SquadDataset\n  split: validation\n\npacked_sequence:\n  enabled: false\n\ndataloader:\n  batch_size: 4\n  num_workers: 4\n  pin_memory: true\n\noptimizer:\n  _target_: torch.optim.AdamW\n  lr: 2.0e-5\n  weight_decay: 0.01\n\nlr_scheduler:\n  _target_: nemo_automodel.schedulers.CosineAnnealingWarmup\n  warmup_steps: 50\n  min_lr: 1.0e-6\n",[526],{"type":45,"tag":110,"props":527,"children":528},{"__ignoreMap":293},[529,547,567,585,603,621,639,657,667,680,699,717,725,738,756,764,777,795,813,823,831,844,863,881,889,902,920,928,941,964,982,999,1016,1024,1037,1054,1062,1075,1092,1109,1127,1135,1147,1163,1181,1189,1202,1218,1226,1239,1256,1273,1291,1299,1312,1329,1347,1365,1373,1386,1403,1421],{"type":45,"tag":530,"props":531,"children":534},"span",{"class":532,"line":533},"line",1,[535,541],{"type":45,"tag":530,"props":536,"children":538},{"style":537},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[539],{"type":50,"value":540},"step_scheduler",{"type":45,"tag":530,"props":542,"children":544},{"style":543},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[545],{"type":50,"value":546},":\n",{"type":45,"tag":530,"props":548,"children":550},{"class":532,"line":549},2,[551,556,561],{"type":45,"tag":530,"props":552,"children":553},{"style":537},[554],{"type":50,"value":555},"  max_steps",{"type":45,"tag":530,"props":557,"children":558},{"style":543},[559],{"type":50,"value":560},":",{"type":45,"tag":530,"props":562,"children":564},{"style":563},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[565],{"type":50,"value":566}," 1000\n",{"type":45,"tag":530,"props":568,"children":570},{"class":532,"line":569},3,[571,576,580],{"type":45,"tag":530,"props":572,"children":573},{"style":537},[574],{"type":50,"value":575},"  num_epochs",{"type":45,"tag":530,"props":577,"children":578},{"style":543},[579],{"type":50,"value":560},{"type":45,"tag":530,"props":581,"children":582},{"style":563},[583],{"type":50,"value":584}," 1\n",{"type":45,"tag":530,"props":586,"children":588},{"class":532,"line":587},4,[589,594,598],{"type":45,"tag":530,"props":590,"children":591},{"style":537},[592],{"type":50,"value":593},"  grad_accumulation_steps",{"type":45,"tag":530,"props":595,"children":596},{"style":543},[597],{"type":50,"value":560},{"type":45,"tag":530,"props":599,"children":600},{"style":563},[601],{"type":50,"value":602}," 4\n",{"type":45,"tag":530,"props":604,"children":606},{"class":532,"line":605},5,[607,612,616],{"type":45,"tag":530,"props":608,"children":609},{"style":537},[610],{"type":50,"value":611},"  val_check_interval",{"type":45,"tag":530,"props":613,"children":614},{"style":543},[615],{"type":50,"value":560},{"type":45,"tag":530,"props":617,"children":618},{"style":563},[619],{"type":50,"value":620}," 100\n",{"type":45,"tag":530,"props":622,"children":624},{"class":532,"line":623},6,[625,630,634],{"type":45,"tag":530,"props":626,"children":627},{"style":537},[628],{"type":50,"value":629},"  checkpoint_interval",{"type":45,"tag":530,"props":631,"children":632},{"style":543},[633],{"type":50,"value":560},{"type":45,"tag":530,"props":635,"children":636},{"style":563},[637],{"type":50,"value":638}," 500\n",{"type":45,"tag":530,"props":640,"children":642},{"class":532,"line":641},7,[643,648,652],{"type":45,"tag":530,"props":644,"children":645},{"style":537},[646],{"type":50,"value":647},"  log_interval",{"type":45,"tag":530,"props":649,"children":650},{"style":543},[651],{"type":50,"value":560},{"type":45,"tag":530,"props":653,"children":654},{"style":563},[655],{"type":50,"value":656}," 10\n",{"type":45,"tag":530,"props":658,"children":660},{"class":532,"line":659},8,[661],{"type":45,"tag":530,"props":662,"children":664},{"emptyLinePlaceholder":663},true,[665],{"type":50,"value":666},"\n",{"type":45,"tag":530,"props":668,"children":670},{"class":532,"line":669},9,[671,676],{"type":45,"tag":530,"props":672,"children":673},{"style":537},[674],{"type":50,"value":675},"dist_env",{"type":45,"tag":530,"props":677,"children":678},{"style":543},[679],{"type":50,"value":546},{"type":45,"tag":530,"props":681,"children":683},{"class":532,"line":682},10,[684,689,693],{"type":45,"tag":530,"props":685,"children":686},{"style":537},[687],{"type":50,"value":688},"  master_addr",{"type":45,"tag":530,"props":690,"children":691},{"style":543},[692],{"type":50,"value":560},{"type":45,"tag":530,"props":694,"children":696},{"style":695},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[697],{"type":50,"value":698}," localhost\n",{"type":45,"tag":530,"props":700,"children":702},{"class":532,"line":701},11,[703,708,712],{"type":45,"tag":530,"props":704,"children":705},{"style":537},[706],{"type":50,"value":707},"  master_port",{"type":45,"tag":530,"props":709,"children":710},{"style":543},[711],{"type":50,"value":560},{"type":45,"tag":530,"props":713,"children":714},{"style":563},[715],{"type":50,"value":716}," 29500\n",{"type":45,"tag":530,"props":718,"children":720},{"class":532,"line":719},12,[721],{"type":45,"tag":530,"props":722,"children":723},{"emptyLinePlaceholder":663},[724],{"type":50,"value":666},{"type":45,"tag":530,"props":726,"children":728},{"class":532,"line":727},13,[729,734],{"type":45,"tag":530,"props":730,"children":731},{"style":537},[732],{"type":50,"value":733},"rng",{"type":45,"tag":530,"props":735,"children":736},{"style":543},[737],{"type":50,"value":546},{"type":45,"tag":530,"props":739,"children":741},{"class":532,"line":740},14,[742,747,751],{"type":45,"tag":530,"props":743,"children":744},{"style":537},[745],{"type":50,"value":746},"  seed",{"type":45,"tag":530,"props":748,"children":749},{"style":543},[750],{"type":50,"value":560},{"type":45,"tag":530,"props":752,"children":753},{"style":563},[754],{"type":50,"value":755}," 42\n",{"type":45,"tag":530,"props":757,"children":759},{"class":532,"line":758},15,[760],{"type":45,"tag":530,"props":761,"children":762},{"emptyLinePlaceholder":663},[763],{"type":50,"value":666},{"type":45,"tag":530,"props":765,"children":767},{"class":532,"line":766},16,[768,773],{"type":45,"tag":530,"props":769,"children":770},{"style":537},[771],{"type":50,"value":772},"model",{"type":45,"tag":530,"props":774,"children":775},{"style":543},[776],{"type":50,"value":546},{"type":45,"tag":530,"props":778,"children":780},{"class":532,"line":779},17,[781,786,790],{"type":45,"tag":530,"props":782,"children":783},{"style":537},[784],{"type":50,"value":785},"  _target_",{"type":45,"tag":530,"props":787,"children":788},{"style":543},[789],{"type":50,"value":560},{"type":45,"tag":530,"props":791,"children":792},{"style":695},[793],{"type":50,"value":794}," nemo_automodel.models.llm.NemotronHForCausalLM\n",{"type":45,"tag":530,"props":796,"children":798},{"class":532,"line":797},18,[799,804,808],{"type":45,"tag":530,"props":800,"children":801},{"style":537},[802],{"type":50,"value":803},"  name_or_path",{"type":45,"tag":530,"props":805,"children":806},{"style":543},[807],{"type":50,"value":560},{"type":45,"tag":530,"props":809,"children":810},{"style":695},[811],{"type":50,"value":812}," meta-llama\u002FLlama-3.2-1B\n",{"type":45,"tag":530,"props":814,"children":816},{"class":532,"line":815},19,[817],{"type":45,"tag":530,"props":818,"children":820},{"style":819},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[821],{"type":50,"value":822},"  # additional model kwargs passed to the constructor\n",{"type":45,"tag":530,"props":824,"children":826},{"class":532,"line":825},20,[827],{"type":45,"tag":530,"props":828,"children":829},{"emptyLinePlaceholder":663},[830],{"type":50,"value":666},{"type":45,"tag":530,"props":832,"children":834},{"class":532,"line":833},21,[835,840],{"type":45,"tag":530,"props":836,"children":837},{"style":537},[838],{"type":50,"value":839},"compile",{"type":45,"tag":530,"props":841,"children":842},{"style":543},[843],{"type":50,"value":546},{"type":45,"tag":530,"props":845,"children":847},{"class":532,"line":846},22,[848,853,857],{"type":45,"tag":530,"props":849,"children":850},{"style":537},[851],{"type":50,"value":852},"  enabled",{"type":45,"tag":530,"props":854,"children":855},{"style":543},[856],{"type":50,"value":560},{"type":45,"tag":530,"props":858,"children":860},{"style":859},"--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC",[861],{"type":50,"value":862}," false\n",{"type":45,"tag":530,"props":864,"children":866},{"class":532,"line":865},23,[867,872,876],{"type":45,"tag":530,"props":868,"children":869},{"style":537},[870],{"type":50,"value":871},"  backend",{"type":45,"tag":530,"props":873,"children":874},{"style":543},[875],{"type":50,"value":560},{"type":45,"tag":530,"props":877,"children":878},{"style":695},[879],{"type":50,"value":880}," inductor\n",{"type":45,"tag":530,"props":882,"children":884},{"class":532,"line":883},24,[885],{"type":45,"tag":530,"props":886,"children":887},{"emptyLinePlaceholder":663},[888],{"type":50,"value":666},{"type":45,"tag":530,"props":890,"children":892},{"class":532,"line":891},25,[893,898],{"type":45,"tag":530,"props":894,"children":895},{"style":537},[896],{"type":50,"value":897},"clip_grad_norm",{"type":45,"tag":530,"props":899,"children":900},{"style":543},[901],{"type":50,"value":546},{"type":45,"tag":530,"props":903,"children":905},{"class":532,"line":904},26,[906,911,915],{"type":45,"tag":530,"props":907,"children":908},{"style":537},[909],{"type":50,"value":910},"  max_norm",{"type":45,"tag":530,"props":912,"children":913},{"style":543},[914],{"type":50,"value":560},{"type":45,"tag":530,"props":916,"children":917},{"style":563},[918],{"type":50,"value":919}," 1.0\n",{"type":45,"tag":530,"props":921,"children":923},{"class":532,"line":922},27,[924],{"type":45,"tag":530,"props":925,"children":926},{"emptyLinePlaceholder":663},[927],{"type":50,"value":666},{"type":45,"tag":530,"props":929,"children":931},{"class":532,"line":930},28,[932,937],{"type":45,"tag":530,"props":933,"children":934},{"style":537},[935],{"type":50,"value":936},"distributed",{"type":45,"tag":530,"props":938,"children":939},{"style":543},[940],{"type":50,"value":546},{"type":45,"tag":530,"props":942,"children":944},{"class":532,"line":943},29,[945,950,954,959],{"type":45,"tag":530,"props":946,"children":947},{"style":537},[948],{"type":50,"value":949},"  strategy",{"type":45,"tag":530,"props":951,"children":952},{"style":543},[953],{"type":50,"value":560},{"type":45,"tag":530,"props":955,"children":956},{"style":695},[957],{"type":50,"value":958}," fsdp2",{"type":45,"tag":530,"props":960,"children":961},{"style":819},[962],{"type":50,"value":963},"       # fsdp2 | megatron_fsdp | ddp\n",{"type":45,"tag":530,"props":965,"children":967},{"class":532,"line":966},30,[968,973,977],{"type":45,"tag":530,"props":969,"children":970},{"style":537},[971],{"type":50,"value":972},"  dp_size",{"type":45,"tag":530,"props":974,"children":975},{"style":543},[976],{"type":50,"value":560},{"type":45,"tag":530,"props":978,"children":979},{"style":695},[980],{"type":50,"value":981}," auto\n",{"type":45,"tag":530,"props":983,"children":985},{"class":532,"line":984},31,[986,991,995],{"type":45,"tag":530,"props":987,"children":988},{"style":537},[989],{"type":50,"value":990},"  tp_size",{"type":45,"tag":530,"props":992,"children":993},{"style":543},[994],{"type":50,"value":560},{"type":45,"tag":530,"props":996,"children":997},{"style":563},[998],{"type":50,"value":584},{"type":45,"tag":530,"props":1000,"children":1002},{"class":532,"line":1001},32,[1003,1008,1012],{"type":45,"tag":530,"props":1004,"children":1005},{"style":537},[1006],{"type":50,"value":1007},"  cp_size",{"type":45,"tag":530,"props":1009,"children":1010},{"style":543},[1011],{"type":50,"value":560},{"type":45,"tag":530,"props":1013,"children":1014},{"style":563},[1015],{"type":50,"value":584},{"type":45,"tag":530,"props":1017,"children":1019},{"class":532,"line":1018},33,[1020],{"type":45,"tag":530,"props":1021,"children":1022},{"emptyLinePlaceholder":663},[1023],{"type":50,"value":666},{"type":45,"tag":530,"props":1025,"children":1027},{"class":532,"line":1026},34,[1028,1033],{"type":45,"tag":530,"props":1029,"children":1030},{"style":537},[1031],{"type":50,"value":1032},"loss_fn",{"type":45,"tag":530,"props":1034,"children":1035},{"style":543},[1036],{"type":50,"value":546},{"type":45,"tag":530,"props":1038,"children":1040},{"class":532,"line":1039},35,[1041,1045,1049],{"type":45,"tag":530,"props":1042,"children":1043},{"style":537},[1044],{"type":50,"value":785},{"type":45,"tag":530,"props":1046,"children":1047},{"style":543},[1048],{"type":50,"value":560},{"type":45,"tag":530,"props":1050,"children":1051},{"style":695},[1052],{"type":50,"value":1053}," torch.nn.CrossEntropyLoss\n",{"type":45,"tag":530,"props":1055,"children":1057},{"class":532,"line":1056},36,[1058],{"type":45,"tag":530,"props":1059,"children":1060},{"emptyLinePlaceholder":663},[1061],{"type":50,"value":666},{"type":45,"tag":530,"props":1063,"children":1065},{"class":532,"line":1064},37,[1066,1071],{"type":45,"tag":530,"props":1067,"children":1068},{"style":537},[1069],{"type":50,"value":1070},"dataset",{"type":45,"tag":530,"props":1072,"children":1073},{"style":543},[1074],{"type":50,"value":546},{"type":45,"tag":530,"props":1076,"children":1078},{"class":532,"line":1077},38,[1079,1083,1087],{"type":45,"tag":530,"props":1080,"children":1081},{"style":537},[1082],{"type":50,"value":785},{"type":45,"tag":530,"props":1084,"children":1085},{"style":543},[1086],{"type":50,"value":560},{"type":45,"tag":530,"props":1088,"children":1089},{"style":695},[1090],{"type":50,"value":1091}," nemo_automodel.datasets.squad.SquadDataset\n",{"type":45,"tag":530,"props":1093,"children":1095},{"class":532,"line":1094},39,[1096,1101,1105],{"type":45,"tag":530,"props":1097,"children":1098},{"style":537},[1099],{"type":50,"value":1100},"  tokenizer_name_or_path",{"type":45,"tag":530,"props":1102,"children":1103},{"style":543},[1104],{"type":50,"value":560},{"type":45,"tag":530,"props":1106,"children":1107},{"style":695},[1108],{"type":50,"value":812},{"type":45,"tag":530,"props":1110,"children":1112},{"class":532,"line":1111},40,[1113,1118,1122],{"type":45,"tag":530,"props":1114,"children":1115},{"style":537},[1116],{"type":50,"value":1117},"  max_seq_length",{"type":45,"tag":530,"props":1119,"children":1120},{"style":543},[1121],{"type":50,"value":560},{"type":45,"tag":530,"props":1123,"children":1124},{"style":563},[1125],{"type":50,"value":1126}," 2048\n",{"type":45,"tag":530,"props":1128,"children":1130},{"class":532,"line":1129},41,[1131],{"type":45,"tag":530,"props":1132,"children":1133},{"emptyLinePlaceholder":663},[1134],{"type":50,"value":666},{"type":45,"tag":530,"props":1136,"children":1138},{"class":532,"line":1137},42,[1139,1143],{"type":45,"tag":530,"props":1140,"children":1141},{"style":537},[1142],{"type":50,"value":185},{"type":45,"tag":530,"props":1144,"children":1145},{"style":543},[1146],{"type":50,"value":546},{"type":45,"tag":530,"props":1148,"children":1150},{"class":532,"line":1149},43,[1151,1155,1159],{"type":45,"tag":530,"props":1152,"children":1153},{"style":537},[1154],{"type":50,"value":785},{"type":45,"tag":530,"props":1156,"children":1157},{"style":543},[1158],{"type":50,"value":560},{"type":45,"tag":530,"props":1160,"children":1161},{"style":695},[1162],{"type":50,"value":1091},{"type":45,"tag":530,"props":1164,"children":1166},{"class":532,"line":1165},44,[1167,1172,1176],{"type":45,"tag":530,"props":1168,"children":1169},{"style":537},[1170],{"type":50,"value":1171},"  split",{"type":45,"tag":530,"props":1173,"children":1174},{"style":543},[1175],{"type":50,"value":560},{"type":45,"tag":530,"props":1177,"children":1178},{"style":695},[1179],{"type":50,"value":1180}," validation\n",{"type":45,"tag":530,"props":1182,"children":1184},{"class":532,"line":1183},45,[1185],{"type":45,"tag":530,"props":1186,"children":1187},{"emptyLinePlaceholder":663},[1188],{"type":50,"value":666},{"type":45,"tag":530,"props":1190,"children":1192},{"class":532,"line":1191},46,[1193,1198],{"type":45,"tag":530,"props":1194,"children":1195},{"style":537},[1196],{"type":50,"value":1197},"packed_sequence",{"type":45,"tag":530,"props":1199,"children":1200},{"style":543},[1201],{"type":50,"value":546},{"type":45,"tag":530,"props":1203,"children":1205},{"class":532,"line":1204},47,[1206,1210,1214],{"type":45,"tag":530,"props":1207,"children":1208},{"style":537},[1209],{"type":50,"value":852},{"type":45,"tag":530,"props":1211,"children":1212},{"style":543},[1213],{"type":50,"value":560},{"type":45,"tag":530,"props":1215,"children":1216},{"style":859},[1217],{"type":50,"value":862},{"type":45,"tag":530,"props":1219,"children":1221},{"class":532,"line":1220},48,[1222],{"type":45,"tag":530,"props":1223,"children":1224},{"emptyLinePlaceholder":663},[1225],{"type":50,"value":666},{"type":45,"tag":530,"props":1227,"children":1229},{"class":532,"line":1228},49,[1230,1235],{"type":45,"tag":530,"props":1231,"children":1232},{"style":537},[1233],{"type":50,"value":1234},"dataloader",{"type":45,"tag":530,"props":1236,"children":1237},{"style":543},[1238],{"type":50,"value":546},{"type":45,"tag":530,"props":1240,"children":1242},{"class":532,"line":1241},50,[1243,1248,1252],{"type":45,"tag":530,"props":1244,"children":1245},{"style":537},[1246],{"type":50,"value":1247},"  batch_size",{"type":45,"tag":530,"props":1249,"children":1250},{"style":543},[1251],{"type":50,"value":560},{"type":45,"tag":530,"props":1253,"children":1254},{"style":563},[1255],{"type":50,"value":602},{"type":45,"tag":530,"props":1257,"children":1259},{"class":532,"line":1258},51,[1260,1265,1269],{"type":45,"tag":530,"props":1261,"children":1262},{"style":537},[1263],{"type":50,"value":1264},"  num_workers",{"type":45,"tag":530,"props":1266,"children":1267},{"style":543},[1268],{"type":50,"value":560},{"type":45,"tag":530,"props":1270,"children":1271},{"style":563},[1272],{"type":50,"value":602},{"type":45,"tag":530,"props":1274,"children":1276},{"class":532,"line":1275},52,[1277,1282,1286],{"type":45,"tag":530,"props":1278,"children":1279},{"style":537},[1280],{"type":50,"value":1281},"  pin_memory",{"type":45,"tag":530,"props":1283,"children":1284},{"style":543},[1285],{"type":50,"value":560},{"type":45,"tag":530,"props":1287,"children":1288},{"style":859},[1289],{"type":50,"value":1290}," true\n",{"type":45,"tag":530,"props":1292,"children":1294},{"class":532,"line":1293},53,[1295],{"type":45,"tag":530,"props":1296,"children":1297},{"emptyLinePlaceholder":663},[1298],{"type":50,"value":666},{"type":45,"tag":530,"props":1300,"children":1302},{"class":532,"line":1301},54,[1303,1308],{"type":45,"tag":530,"props":1304,"children":1305},{"style":537},[1306],{"type":50,"value":1307},"optimizer",{"type":45,"tag":530,"props":1309,"children":1310},{"style":543},[1311],{"type":50,"value":546},{"type":45,"tag":530,"props":1313,"children":1315},{"class":532,"line":1314},55,[1316,1320,1324],{"type":45,"tag":530,"props":1317,"children":1318},{"style":537},[1319],{"type":50,"value":785},{"type":45,"tag":530,"props":1321,"children":1322},{"style":543},[1323],{"type":50,"value":560},{"type":45,"tag":530,"props":1325,"children":1326},{"style":695},[1327],{"type":50,"value":1328}," torch.optim.AdamW\n",{"type":45,"tag":530,"props":1330,"children":1332},{"class":532,"line":1331},56,[1333,1338,1342],{"type":45,"tag":530,"props":1334,"children":1335},{"style":537},[1336],{"type":50,"value":1337},"  lr",{"type":45,"tag":530,"props":1339,"children":1340},{"style":543},[1341],{"type":50,"value":560},{"type":45,"tag":530,"props":1343,"children":1344},{"style":563},[1345],{"type":50,"value":1346}," 2.0e-5\n",{"type":45,"tag":530,"props":1348,"children":1350},{"class":532,"line":1349},57,[1351,1356,1360],{"type":45,"tag":530,"props":1352,"children":1353},{"style":537},[1354],{"type":50,"value":1355},"  weight_decay",{"type":45,"tag":530,"props":1357,"children":1358},{"style":543},[1359],{"type":50,"value":560},{"type":45,"tag":530,"props":1361,"children":1362},{"style":563},[1363],{"type":50,"value":1364}," 0.01\n",{"type":45,"tag":530,"props":1366,"children":1368},{"class":532,"line":1367},58,[1369],{"type":45,"tag":530,"props":1370,"children":1371},{"emptyLinePlaceholder":663},[1372],{"type":50,"value":666},{"type":45,"tag":530,"props":1374,"children":1376},{"class":532,"line":1375},59,[1377,1382],{"type":45,"tag":530,"props":1378,"children":1379},{"style":537},[1380],{"type":50,"value":1381},"lr_scheduler",{"type":45,"tag":530,"props":1383,"children":1384},{"style":543},[1385],{"type":50,"value":546},{"type":45,"tag":530,"props":1387,"children":1389},{"class":532,"line":1388},60,[1390,1394,1398],{"type":45,"tag":530,"props":1391,"children":1392},{"style":537},[1393],{"type":50,"value":785},{"type":45,"tag":530,"props":1395,"children":1396},{"style":543},[1397],{"type":50,"value":560},{"type":45,"tag":530,"props":1399,"children":1400},{"style":695},[1401],{"type":50,"value":1402}," nemo_automodel.schedulers.CosineAnnealingWarmup\n",{"type":45,"tag":530,"props":1404,"children":1406},{"class":532,"line":1405},61,[1407,1412,1416],{"type":45,"tag":530,"props":1408,"children":1409},{"style":537},[1410],{"type":50,"value":1411},"  warmup_steps",{"type":45,"tag":530,"props":1413,"children":1414},{"style":543},[1415],{"type":50,"value":560},{"type":45,"tag":530,"props":1417,"children":1418},{"style":563},[1419],{"type":50,"value":1420}," 50\n",{"type":45,"tag":530,"props":1422,"children":1424},{"class":532,"line":1423},62,[1425,1430,1434],{"type":45,"tag":530,"props":1426,"children":1427},{"style":537},[1428],{"type":50,"value":1429},"  min_lr",{"type":45,"tag":530,"props":1431,"children":1432},{"style":543},[1433],{"type":50,"value":560},{"type":45,"tag":530,"props":1435,"children":1436},{"style":563},[1437],{"type":50,"value":1438}," 1.0e-6\n",{"type":45,"tag":278,"props":1440,"children":1442},{"id":1441},"the-_target_-pattern",[1443,1445,1450],{"type":50,"value":1444},"The ",{"type":45,"tag":110,"props":1446,"children":1448},{"className":1447},[],[1449],{"type":50,"value":142},{"type":50,"value":1451}," Pattern",{"type":45,"tag":60,"props":1453,"children":1454},{},[1455,1456,1461],{"type":50,"value":1444},{"type":45,"tag":110,"props":1457,"children":1459},{"className":1458},[],[1460],{"type":50,"value":142},{"type":50,"value":1462}," key specifies a fully qualified Python callable. All remaining keys in that section are passed as keyword arguments:",{"type":45,"tag":285,"props":1464,"children":1466},{"className":523,"code":1465,"language":21,"meta":293,"style":293},"optimizer:\n  _target_: torch.optim.AdamW   # callable\n  lr: 2.0e-5                    # kwarg\n  weight_decay: 0.01            # kwarg\n",[1467],{"type":45,"tag":110,"props":1468,"children":1469},{"__ignoreMap":293},[1470,1481,1502,1523],{"type":45,"tag":530,"props":1471,"children":1472},{"class":532,"line":533},[1473,1477],{"type":45,"tag":530,"props":1474,"children":1475},{"style":537},[1476],{"type":50,"value":1307},{"type":45,"tag":530,"props":1478,"children":1479},{"style":543},[1480],{"type":50,"value":546},{"type":45,"tag":530,"props":1482,"children":1483},{"class":532,"line":549},[1484,1488,1492,1497],{"type":45,"tag":530,"props":1485,"children":1486},{"style":537},[1487],{"type":50,"value":785},{"type":45,"tag":530,"props":1489,"children":1490},{"style":543},[1491],{"type":50,"value":560},{"type":45,"tag":530,"props":1493,"children":1494},{"style":695},[1495],{"type":50,"value":1496}," torch.optim.AdamW",{"type":45,"tag":530,"props":1498,"children":1499},{"style":819},[1500],{"type":50,"value":1501},"   # callable\n",{"type":45,"tag":530,"props":1503,"children":1504},{"class":532,"line":569},[1505,1509,1513,1518],{"type":45,"tag":530,"props":1506,"children":1507},{"style":537},[1508],{"type":50,"value":1337},{"type":45,"tag":530,"props":1510,"children":1511},{"style":543},[1512],{"type":50,"value":560},{"type":45,"tag":530,"props":1514,"children":1515},{"style":563},[1516],{"type":50,"value":1517}," 2.0e-5",{"type":45,"tag":530,"props":1519,"children":1520},{"style":819},[1521],{"type":50,"value":1522},"                    # kwarg\n",{"type":45,"tag":530,"props":1524,"children":1525},{"class":532,"line":587},[1526,1530,1534,1539],{"type":45,"tag":530,"props":1527,"children":1528},{"style":537},[1529],{"type":50,"value":1355},{"type":45,"tag":530,"props":1531,"children":1532},{"style":543},[1533],{"type":50,"value":560},{"type":45,"tag":530,"props":1535,"children":1536},{"style":563},[1537],{"type":50,"value":1538}," 0.01",{"type":45,"tag":530,"props":1540,"children":1541},{"style":819},[1542],{"type":50,"value":1543},"            # kwarg\n",{"type":45,"tag":60,"props":1545,"children":1546},{},[1547,1549,1555],{"type":50,"value":1548},"This is equivalent to: ",{"type":45,"tag":110,"props":1550,"children":1552},{"className":1551},[],[1553],{"type":50,"value":1554},"torch.optim.AdamW(lr=2e-5, weight_decay=0.01)",{"type":50,"value":133},{"type":45,"tag":278,"props":1557,"children":1559},{"id":1558},"cli-overrides",[1560],{"type":50,"value":1561},"CLI Overrides",{"type":45,"tag":60,"props":1563,"children":1564},{},[1565],{"type":50,"value":1566},"Any config value can be overridden from the command line:",{"type":45,"tag":285,"props":1568,"children":1572},{"className":1569,"code":1570,"language":1571,"meta":293,"style":293},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","automodel finetune llm -c config.yaml \\\n  --optimizer.lr 1e-4 \\\n  --step_scheduler.max_steps 500 \\\n  --distributed.tp_size 2\n","bash",[1573],{"type":45,"tag":110,"props":1574,"children":1575},{"__ignoreMap":293},[1576,1611,1628,1645],{"type":45,"tag":530,"props":1577,"children":1578},{"class":532,"line":533},[1579,1585,1590,1595,1600,1605],{"type":45,"tag":530,"props":1580,"children":1582},{"style":1581},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1583],{"type":50,"value":1584},"automodel",{"type":45,"tag":530,"props":1586,"children":1587},{"style":695},[1588],{"type":50,"value":1589}," finetune",{"type":45,"tag":530,"props":1591,"children":1592},{"style":695},[1593],{"type":50,"value":1594}," llm",{"type":45,"tag":530,"props":1596,"children":1597},{"style":695},[1598],{"type":50,"value":1599}," -c",{"type":45,"tag":530,"props":1601,"children":1602},{"style":695},[1603],{"type":50,"value":1604}," config.yaml",{"type":45,"tag":530,"props":1606,"children":1608},{"style":1607},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1609],{"type":50,"value":1610}," \\\n",{"type":45,"tag":530,"props":1612,"children":1613},{"class":532,"line":549},[1614,1619,1624],{"type":45,"tag":530,"props":1615,"children":1616},{"style":695},[1617],{"type":50,"value":1618},"  --optimizer.lr",{"type":45,"tag":530,"props":1620,"children":1621},{"style":695},[1622],{"type":50,"value":1623}," 1e-4",{"type":45,"tag":530,"props":1625,"children":1626},{"style":1607},[1627],{"type":50,"value":1610},{"type":45,"tag":530,"props":1629,"children":1630},{"class":532,"line":569},[1631,1636,1641],{"type":45,"tag":530,"props":1632,"children":1633},{"style":695},[1634],{"type":50,"value":1635},"  --step_scheduler.max_steps",{"type":45,"tag":530,"props":1637,"children":1638},{"style":563},[1639],{"type":50,"value":1640}," 500",{"type":45,"tag":530,"props":1642,"children":1643},{"style":1607},[1644],{"type":50,"value":1610},{"type":45,"tag":530,"props":1646,"children":1647},{"class":532,"line":587},[1648,1653],{"type":45,"tag":530,"props":1649,"children":1650},{"style":695},[1651],{"type":50,"value":1652},"  --distributed.tp_size",{"type":45,"tag":530,"props":1654,"children":1655},{"style":563},[1656],{"type":50,"value":1657}," 2\n",{"type":45,"tag":53,"props":1659,"children":1661},{"id":1660},"examples",[1662],{"type":50,"value":1663},"Examples",{"type":45,"tag":60,"props":1665,"children":1666},{},[1667],{"type":50,"value":1668},"Validation and checkpointing:",{"type":45,"tag":285,"props":1670,"children":1672},{"className":523,"code":1671,"language":21,"meta":293,"style":293},"step_scheduler:\n  val_check_interval: 100\n  checkpoint_interval: 500\n\nvalidation_dataset:\n  _target_: nemo_automodel.datasets.squad.SquadDataset\n  split: validation\n\nrestore_from:\n  path: \u002Fcheckpoints\u002Fstep-500\n",[1673],{"type":45,"tag":110,"props":1674,"children":1675},{"__ignoreMap":293},[1676,1687,1702,1717,1724,1735,1750,1765,1772,1784],{"type":45,"tag":530,"props":1677,"children":1678},{"class":532,"line":533},[1679,1683],{"type":45,"tag":530,"props":1680,"children":1681},{"style":537},[1682],{"type":50,"value":540},{"type":45,"tag":530,"props":1684,"children":1685},{"style":543},[1686],{"type":50,"value":546},{"type":45,"tag":530,"props":1688,"children":1689},{"class":532,"line":549},[1690,1694,1698],{"type":45,"tag":530,"props":1691,"children":1692},{"style":537},[1693],{"type":50,"value":611},{"type":45,"tag":530,"props":1695,"children":1696},{"style":543},[1697],{"type":50,"value":560},{"type":45,"tag":530,"props":1699,"children":1700},{"style":563},[1701],{"type":50,"value":620},{"type":45,"tag":530,"props":1703,"children":1704},{"class":532,"line":569},[1705,1709,1713],{"type":45,"tag":530,"props":1706,"children":1707},{"style":537},[1708],{"type":50,"value":629},{"type":45,"tag":530,"props":1710,"children":1711},{"style":543},[1712],{"type":50,"value":560},{"type":45,"tag":530,"props":1714,"children":1715},{"style":563},[1716],{"type":50,"value":638},{"type":45,"tag":530,"props":1718,"children":1719},{"class":532,"line":587},[1720],{"type":45,"tag":530,"props":1721,"children":1722},{"emptyLinePlaceholder":663},[1723],{"type":50,"value":666},{"type":45,"tag":530,"props":1725,"children":1726},{"class":532,"line":605},[1727,1731],{"type":45,"tag":530,"props":1728,"children":1729},{"style":537},[1730],{"type":50,"value":185},{"type":45,"tag":530,"props":1732,"children":1733},{"style":543},[1734],{"type":50,"value":546},{"type":45,"tag":530,"props":1736,"children":1737},{"class":532,"line":623},[1738,1742,1746],{"type":45,"tag":530,"props":1739,"children":1740},{"style":537},[1741],{"type":50,"value":785},{"type":45,"tag":530,"props":1743,"children":1744},{"style":543},[1745],{"type":50,"value":560},{"type":45,"tag":530,"props":1747,"children":1748},{"style":695},[1749],{"type":50,"value":1091},{"type":45,"tag":530,"props":1751,"children":1752},{"class":532,"line":641},[1753,1757,1761],{"type":45,"tag":530,"props":1754,"children":1755},{"style":537},[1756],{"type":50,"value":1171},{"type":45,"tag":530,"props":1758,"children":1759},{"style":543},[1760],{"type":50,"value":560},{"type":45,"tag":530,"props":1762,"children":1763},{"style":695},[1764],{"type":50,"value":1180},{"type":45,"tag":530,"props":1766,"children":1767},{"class":532,"line":659},[1768],{"type":45,"tag":530,"props":1769,"children":1770},{"emptyLinePlaceholder":663},[1771],{"type":50,"value":666},{"type":45,"tag":530,"props":1773,"children":1774},{"class":532,"line":669},[1775,1780],{"type":45,"tag":530,"props":1776,"children":1777},{"style":537},[1778],{"type":50,"value":1779},"restore_from",{"type":45,"tag":530,"props":1781,"children":1782},{"style":543},[1783],{"type":50,"value":546},{"type":45,"tag":530,"props":1785,"children":1786},{"class":532,"line":682},[1787,1792,1796],{"type":45,"tag":530,"props":1788,"children":1789},{"style":537},[1790],{"type":50,"value":1791},"  path",{"type":45,"tag":530,"props":1793,"children":1794},{"style":543},[1795],{"type":50,"value":560},{"type":45,"tag":530,"props":1797,"children":1798},{"style":695},[1799],{"type":50,"value":1800}," \u002Fcheckpoints\u002Fstep-500\n",{"type":45,"tag":53,"props":1802,"children":1804},{"id":1803},"domain-specific-notes",[1805],{"type":50,"value":1806},"Domain-Specific Notes",{"type":45,"tag":278,"props":1808,"children":1810},{"id":1809},"llm",[1811],{"type":50,"value":1812},"LLM",{"type":45,"tag":101,"props":1814,"children":1815},{},[1816,1827,1838],{"type":45,"tag":70,"props":1817,"children":1818},{},[1819,1825],{"type":45,"tag":110,"props":1820,"children":1822},{"className":1821},[],[1823],{"type":50,"value":1824},"nemo_automodel\u002Frecipes\u002Fllm\u002Ftrain_ft.py",{"type":50,"value":1826}," handles both finetuning and pretraining. The distinction is in the config (dataset, learning rate, etc.).",{"type":45,"tag":70,"props":1828,"children":1829},{},[1830,1836],{"type":45,"tag":110,"props":1831,"children":1833},{"className":1832},[],[1834],{"type":50,"value":1835},"nemo_automodel\u002Frecipes\u002Fllm\u002Fkd.py",{"type":50,"value":1837}," implements knowledge distillation with a teacher and student model.",{"type":45,"tag":70,"props":1839,"children":1840},{},[1841,1847],{"type":45,"tag":110,"props":1842,"children":1844},{"className":1843},[],[1845],{"type":50,"value":1846},"nemo_automodel\u002Frecipes\u002Fllm\u002Fbenchmark.py",{"type":50,"value":1848}," runs throughput and latency benchmarks.",{"type":45,"tag":278,"props":1850,"children":1852},{"id":1851},"vlm",[1853],{"type":50,"value":1854},"VLM",{"type":45,"tag":101,"props":1856,"children":1857},{},[1858,1871,1884],{"type":45,"tag":70,"props":1859,"children":1860},{},[1861,1863,1869],{"type":50,"value":1862},"Uses ",{"type":45,"tag":110,"props":1864,"children":1866},{"className":1865},[],[1867],{"type":50,"value":1868},"NeMoAutoModelForImageTextToText",{"type":50,"value":1870}," instead of causal LM classes.",{"type":45,"tag":70,"props":1872,"children":1873},{},[1874,1876,1882],{"type":50,"value":1875},"Config includes a ",{"type":45,"tag":110,"props":1877,"children":1879},{"className":1878},[],[1880],{"type":50,"value":1881},"processor",{"type":50,"value":1883}," section instead of a standalone tokenizer.",{"type":45,"tag":70,"props":1885,"children":1886},{},[1887,1889,1895],{"type":50,"value":1888},"Recipe lives in ",{"type":45,"tag":110,"props":1890,"children":1892},{"className":1891},[],[1893],{"type":50,"value":1894},"nemo_automodel\u002Frecipes\u002Fvlm\u002Ffinetune.py",{"type":50,"value":133},{"type":45,"tag":278,"props":1897,"children":1899},{"id":1898},"diffusion",[1900],{"type":50,"value":1901},"Diffusion",{"type":45,"tag":101,"props":1903,"children":1904},{},[1905,1916,1929,1934],{"type":45,"tag":70,"props":1906,"children":1907},{},[1908,1909,1915],{"type":50,"value":1862},{"type":45,"tag":110,"props":1910,"children":1912},{"className":1911},[],[1913],{"type":50,"value":1914},"NeMoAutoDiffusionPipeline",{"type":50,"value":133},{"type":45,"tag":70,"props":1917,"children":1918},{},[1919,1921,1927],{"type":50,"value":1920},"Requires a ",{"type":45,"tag":110,"props":1922,"children":1924},{"className":1923},[],[1925],{"type":50,"value":1926},"parallel_scheme",{"type":50,"value":1928}," dict in config to define parallelism.",{"type":45,"tag":70,"props":1930,"children":1931},{},[1932],{"type":50,"value":1933},"Only supports DDP and FSDP2 strategies (no Megatron-FSDP).",{"type":45,"tag":70,"props":1935,"children":1936},{},[1937,1938,1944],{"type":50,"value":1888},{"type":45,"tag":110,"props":1939,"children":1941},{"className":1940},[],[1942],{"type":50,"value":1943},"nemo_automodel\u002Frecipes\u002Fdiffusion\u002Ftrain.py",{"type":50,"value":133},{"type":45,"tag":278,"props":1946,"children":1948},{"id":1947},"retrieval",[1949],{"type":50,"value":1950},"Retrieval",{"type":45,"tag":101,"props":1952,"children":1953},{},[1954,1998],{"type":45,"tag":70,"props":1955,"children":1956},{},[1957,1959],{"type":50,"value":1958},"Two encoder patterns:\n",{"type":45,"tag":101,"props":1960,"children":1961},{},[1962,1981],{"type":45,"tag":70,"props":1963,"children":1964},{},[1965,1971,1973,1979],{"type":45,"tag":1966,"props":1967,"children":1968},"strong",{},[1969],{"type":50,"value":1970},"Bi-encoder",{"type":50,"value":1972}," (",{"type":45,"tag":110,"props":1974,"children":1976},{"className":1975},[],[1977],{"type":50,"value":1978},"nemo_automodel\u002Frecipes\u002Fretrieval\u002Ftrain_bi_encoder.py",{"type":50,"value":1980},"): separate query and document encoders, contrastive loss.",{"type":45,"tag":70,"props":1982,"children":1983},{},[1984,1989,1990,1996],{"type":45,"tag":1966,"props":1985,"children":1986},{},[1987],{"type":50,"value":1988},"Cross-encoder",{"type":50,"value":1972},{"type":45,"tag":110,"props":1991,"children":1993},{"className":1992},[],[1994],{"type":50,"value":1995},"nemo_automodel\u002Frecipes\u002Fretrieval\u002Ftrain_cross_encoder.py",{"type":50,"value":1997},"): joint encoding, classification head.",{"type":45,"tag":70,"props":1999,"children":2000},{},[2001,2003,2009],{"type":50,"value":2002},"Hard negative mining: ",{"type":45,"tag":110,"props":2004,"children":2006},{"className":2005},[],[2007],{"type":50,"value":2008},"nemo_automodel\u002Frecipes\u002Fretrieval\u002Fmine_hard_negatives.py",{"type":50,"value":133},{"type":45,"tag":53,"props":2011,"children":2013},{"id":2012},"training-loop-details",[2014],{"type":50,"value":2015},"Training Loop Details",{"type":45,"tag":60,"props":2017,"children":2018},{},[2019],{"type":50,"value":2020},"The training loop follows this structure per epoch:",{"type":45,"tag":285,"props":2022,"children":2025},{"className":2023,"code":2024,"language":50},[288],"for epoch in range(num_epochs):\n    for batch_idx in range(batches_per_epoch):\n        # --- gradient accumulation inner loop ---\n        for micro_batch in micro_batches:\n            if pipeline_parallel:\n                schedule.step(micro_batch)    # PP schedule\n            else:\n                loss = model(micro_batch)     # direct forward\n                loss.backward()\n\n        # --- optimizer step ---\n        scale_grads_and_clip_grad_norm(model, max_norm)\n        optimizer.step()\n        lr_scheduler.step()\n        optimizer.zero_grad()\n\n        # --- logging ---\n        MetricsSample(step, epoch, loss, grad_norm, lr, mem, tps, mfu)\n\n        # --- validation (at configured intervals) ---\n        if step % val_check_interval == 0:\n            run_validation()\n\n        # --- checkpoint (at configured intervals) ---\n        if step % checkpoint_interval == 0:\n            save_checkpoint()\n",[2026],{"type":45,"tag":110,"props":2027,"children":2028},{"__ignoreMap":293},[2029],{"type":50,"value":2024},{"type":45,"tag":278,"props":2031,"children":2033},{"id":2032},"stepscheduler",[2034],{"type":50,"value":2035},"StepScheduler",{"type":45,"tag":60,"props":2037,"children":2038},{},[2039],{"type":50,"value":2040},"Controls all training progression: total epochs, total steps, gradient accumulation steps, validation interval, checkpoint interval, and logging interval.",{"type":45,"tag":278,"props":2042,"children":2044},{"id":2043},"gradient-clipping",[2045],{"type":50,"value":2046},"Gradient Clipping",{"type":45,"tag":60,"props":2048,"children":2049},{},[2050,2052,2058,2060,2066],{"type":50,"value":2051},"Applied via ",{"type":45,"tag":110,"props":2053,"children":2055},{"className":2054},[],[2056],{"type":50,"value":2057},"scale_grads_and_clip_grad_norm()",{"type":50,"value":2059}," after the backward pass and before the optimizer step. Controlled by ",{"type":45,"tag":110,"props":2061,"children":2063},{"className":2062},[],[2064],{"type":50,"value":2065},"clip_grad_norm.max_norm",{"type":50,"value":2067}," in config.",{"type":45,"tag":278,"props":2069,"children":2071},{"id":2070},"context-parallelism",[2072],{"type":50,"value":2073},"Context Parallelism",{"type":45,"tag":60,"props":2075,"children":2076},{},[2077,2079,2085,2087,2093],{"type":50,"value":2078},"When ",{"type":45,"tag":110,"props":2080,"children":2082},{"className":2081},[],[2083],{"type":50,"value":2084},"cp_size > 1",{"type":50,"value":2086},", batches are split across the context-parallel group using ",{"type":45,"tag":110,"props":2088,"children":2090},{"className":2089},[],[2091],{"type":50,"value":2092},"make_cp_batch_and_ctx()",{"type":50,"value":2094},". This must happen before the forward pass.",{"type":45,"tag":278,"props":2096,"children":2098},{"id":2097},"metricssample",[2099],{"type":50,"value":2100},"MetricsSample",{"type":45,"tag":60,"props":2102,"children":2103},{},[2104,2106,2111],{"type":50,"value":2105},"Each training step produces a ",{"type":45,"tag":110,"props":2107,"children":2109},{"className":2108},[],[2110],{"type":50,"value":2100},{"type":50,"value":2112}," with fields:",{"type":45,"tag":101,"props":2114,"children":2115},{},[2116,2127,2138,2149,2160,2171,2182,2193],{"type":45,"tag":70,"props":2117,"children":2118},{},[2119,2125],{"type":45,"tag":110,"props":2120,"children":2122},{"className":2121},[],[2123],{"type":50,"value":2124},"step",{"type":50,"value":2126}," -- global step count",{"type":45,"tag":70,"props":2128,"children":2129},{},[2130,2136],{"type":45,"tag":110,"props":2131,"children":2133},{"className":2132},[],[2134],{"type":50,"value":2135},"epoch",{"type":50,"value":2137}," -- current epoch",{"type":45,"tag":70,"props":2139,"children":2140},{},[2141,2147],{"type":45,"tag":110,"props":2142,"children":2144},{"className":2143},[],[2145],{"type":50,"value":2146},"loss",{"type":50,"value":2148}," -- training loss",{"type":45,"tag":70,"props":2150,"children":2151},{},[2152,2158],{"type":45,"tag":110,"props":2153,"children":2155},{"className":2154},[],[2156],{"type":50,"value":2157},"grad_norm",{"type":50,"value":2159}," -- gradient norm after clipping",{"type":45,"tag":70,"props":2161,"children":2162},{},[2163,2169],{"type":45,"tag":110,"props":2164,"children":2166},{"className":2165},[],[2167],{"type":50,"value":2168},"lr",{"type":50,"value":2170}," -- current learning rate",{"type":45,"tag":70,"props":2172,"children":2173},{},[2174,2180],{"type":45,"tag":110,"props":2175,"children":2177},{"className":2176},[],[2178],{"type":50,"value":2179},"mem",{"type":50,"value":2181}," -- GPU memory usage",{"type":45,"tag":70,"props":2183,"children":2184},{},[2185,2191],{"type":45,"tag":110,"props":2186,"children":2188},{"className":2187},[],[2189],{"type":50,"value":2190},"tps",{"type":50,"value":2192}," -- tokens per second",{"type":45,"tag":70,"props":2194,"children":2195},{},[2196,2202],{"type":45,"tag":110,"props":2197,"children":2199},{"className":2198},[],[2200],{"type":50,"value":2201},"mfu",{"type":50,"value":2203}," -- model FLOPS utilization",{"type":45,"tag":53,"props":2205,"children":2207},{"id":2206},"validation-checkpointing",[2208],{"type":50,"value":2209},"Validation & Checkpointing",{"type":45,"tag":278,"props":2211,"children":2213},{"id":2212},"validation",[2214],{"type":50,"value":2215},"Validation",{"type":45,"tag":101,"props":2217,"children":2218},{},[2219,2230,2242],{"type":45,"tag":70,"props":2220,"children":2221},{},[2222,2224,2229],{"type":50,"value":2223},"Runs at intervals defined by ",{"type":45,"tag":110,"props":2225,"children":2227},{"className":2226},[],[2228],{"type":50,"value":169},{"type":50,"value":133},{"type":45,"tag":70,"props":2231,"children":2232},{},[2233,2235,2240],{"type":50,"value":2234},"Uses the validation dataloader built from ",{"type":45,"tag":110,"props":2236,"children":2238},{"className":2237},[],[2239],{"type":50,"value":185},{"type":50,"value":2241}," config.",{"type":45,"tag":70,"props":2243,"children":2244},{},[2245],{"type":50,"value":2246},"Model is set to eval mode; gradients are disabled.",{"type":45,"tag":278,"props":2248,"children":2250},{"id":2249},"checkpointing",[2251],{"type":50,"value":2252},"Checkpointing",{"type":45,"tag":101,"props":2254,"children":2255},{},[2256,2261,2272],{"type":45,"tag":70,"props":2257,"children":2258},{},[2259],{"type":50,"value":2260},"Default format: consolidated safetensors for easy deployment on HF ecosystem (always prefer this over DCP).",{"type":45,"tag":70,"props":2262,"children":2263},{},[2264,2266,2271],{"type":50,"value":2265},"Checkpoint interval controlled by ",{"type":45,"tag":110,"props":2267,"children":2269},{"className":2268},[],[2270],{"type":50,"value":177},{"type":50,"value":133},{"type":45,"tag":70,"props":2273,"children":2274},{},[2275,2277,2282],{"type":50,"value":2276},"Resume training via the ",{"type":45,"tag":110,"props":2278,"children":2280},{"className":2279},[],[2281],{"type":50,"value":1779},{"type":50,"value":2283}," config key pointing to a checkpoint directory.",{"type":45,"tag":285,"props":2285,"children":2287},{"className":523,"code":2286,"language":21,"meta":293,"style":293},"restore_from:\n  path: \u002Fcheckpoints\u002Fstep-500\n",[2288],{"type":45,"tag":110,"props":2289,"children":2290},{"__ignoreMap":293},[2291,2302],{"type":45,"tag":530,"props":2292,"children":2293},{"class":532,"line":533},[2294,2298],{"type":45,"tag":530,"props":2295,"children":2296},{"style":537},[2297],{"type":50,"value":1779},{"type":45,"tag":530,"props":2299,"children":2300},{"style":543},[2301],{"type":50,"value":546},{"type":45,"tag":530,"props":2303,"children":2304},{"class":532,"line":549},[2305,2309,2313],{"type":45,"tag":530,"props":2306,"children":2307},{"style":537},[2308],{"type":50,"value":1791},{"type":45,"tag":530,"props":2310,"children":2311},{"style":543},[2312],{"type":50,"value":560},{"type":45,"tag":530,"props":2314,"children":2315},{"style":695},[2316],{"type":50,"value":1800},{"type":45,"tag":53,"props":2318,"children":2320},{"id":2319},"pitfalls",[2321],{"type":50,"value":2322},"Pitfalls",{"type":45,"tag":2324,"props":2325,"children":2326},"table",{},[2327,2351],{"type":45,"tag":2328,"props":2329,"children":2330},"thead",{},[2331],{"type":45,"tag":2332,"props":2333,"children":2334},"tr",{},[2335,2341,2346],{"type":45,"tag":2336,"props":2337,"children":2338},"th",{},[2339],{"type":50,"value":2340},"Problem",{"type":45,"tag":2336,"props":2342,"children":2343},{},[2344],{"type":50,"value":2345},"Cause",{"type":45,"tag":2336,"props":2347,"children":2348},{},[2349],{"type":50,"value":2350},"Fix",{"type":45,"tag":2352,"props":2353,"children":2354},"tbody",{},[2355,2381,2411,2437,2455,2481],{"type":45,"tag":2332,"props":2356,"children":2357},{},[2358,2364,2376],{"type":45,"tag":2359,"props":2360,"children":2361},"td",{},[2362],{"type":50,"value":2363},"Silent config errors",{"type":45,"tag":2359,"props":2365,"children":2366},{},[2367,2369,2374],{"type":50,"value":2368},"Typo in ",{"type":45,"tag":110,"props":2370,"children":2372},{"className":2371},[],[2373],{"type":50,"value":142},{"type":50,"value":2375}," value",{"type":45,"tag":2359,"props":2377,"children":2378},{},[2379],{"type":50,"value":2380},"The class path must be a valid, importable Python callable. Double-check the module path and class name.",{"type":45,"tag":2332,"props":2382,"children":2383},{},[2384,2389,2406],{"type":45,"tag":2359,"props":2385,"children":2386},{},[2387],{"type":50,"value":2388},"Training crashes at first step",{"type":45,"tag":2359,"props":2390,"children":2391},{},[2392,2398,2400],{"type":45,"tag":110,"props":2393,"children":2395},{"className":2394},[],[2396],{"type":50,"value":2397},"global_batch_size",{"type":50,"value":2399}," not divisible by ",{"type":45,"tag":110,"props":2401,"children":2403},{"className":2402},[],[2404],{"type":50,"value":2405},"local_batch_size * dp_size * grad_accumulation_steps",{"type":45,"tag":2359,"props":2407,"children":2408},{},[2409],{"type":50,"value":2410},"Ensure the batch size math is consistent across all dimensions.",{"type":45,"tag":2332,"props":2412,"children":2413},{},[2414,2419,2424],{"type":45,"tag":2359,"props":2415,"children":2416},{},[2417],{"type":50,"value":2418},"New recipe not accessible via CLI",{"type":45,"tag":2359,"props":2420,"children":2421},{},[2422],{"type":50,"value":2423},"Missing CLI command alias registration",{"type":45,"tag":2359,"props":2425,"children":2426},{},[2427,2429,2435],{"type":50,"value":2428},"Register the new route in the CLI app so ",{"type":45,"tag":110,"props":2430,"children":2432},{"className":2431},[],[2433],{"type":50,"value":2434},"automodel \u003Ccommand> \u003Cdomain>",{"type":50,"value":2436}," resolves correctly.",{"type":45,"tag":2332,"props":2438,"children":2439},{},[2440,2445,2450],{"type":45,"tag":2359,"props":2441,"children":2442},{},[2443],{"type":50,"value":2444},"Shape mismatch at forward pass",{"type":45,"tag":2359,"props":2446,"children":2447},{},[2448],{"type":50,"value":2449},"Dataset collate function output does not match model input signature",{"type":45,"tag":2359,"props":2451,"children":2452},{},[2453],{"type":50,"value":2454},"Verify that the collate function returns tensors with the keys and shapes the model expects.",{"type":45,"tag":2332,"props":2456,"children":2457},{},[2458,2463,2468],{"type":45,"tag":2359,"props":2459,"children":2460},{},[2461],{"type":50,"value":2462},"OOM during validation",{"type":45,"tag":2359,"props":2464,"children":2465},{},[2466],{"type":50,"value":2467},"Validation batch size too large or gradients not disabled",{"type":45,"tag":2359,"props":2469,"children":2470},{},[2471,2473,2479],{"type":50,"value":2472},"Wrap validation in ",{"type":45,"tag":110,"props":2474,"children":2476},{"className":2475},[],[2477],{"type":50,"value":2478},"torch.no_grad()",{"type":50,"value":2480}," and consider a smaller validation batch size.",{"type":45,"tag":2332,"props":2482,"children":2483},{},[2484,2489,2494],{"type":45,"tag":2359,"props":2485,"children":2486},{},[2487],{"type":50,"value":2488},"Checkpoint restore fails",{"type":45,"tag":2359,"props":2490,"children":2491},{},[2492],{"type":50,"value":2493},"Mismatched model architecture between checkpoint and config",{"type":45,"tag":2359,"props":2495,"children":2496},{},[2497],{"type":50,"value":2498},"Ensure the model config matches the checkpoint exactly (layer count, hidden dim, vocab size).",{"type":45,"tag":2500,"props":2501,"children":2502},"style",{},[2503],{"type":50,"value":2504},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2506,"total":2610},[2507,2524,2538,2552,2564,2581,2596],{"slug":2508,"name":2508,"fn":2509,"description":2510,"org":2511,"tags":2512,"stars":23,"repoUrl":24,"updatedAt":2523},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2513,2516,2519,2520],{"name":2514,"slug":2515,"type":15},"Data Analysis","data-analysis",{"name":2517,"slug":2518,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":2521,"slug":2522,"type":15},"Performance","performance","2026-07-14T05:28:43.176466",{"slug":2525,"name":2525,"fn":2526,"description":2527,"org":2528,"tags":2529,"stars":23,"repoUrl":24,"updatedAt":2537},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2530,2533,2536],{"name":2531,"slug":2532,"type":15},"Deployment","deployment",{"name":2534,"slug":2535,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":2539,"name":2539,"fn":2540,"description":2541,"org":2542,"tags":2543,"stars":23,"repoUrl":24,"updatedAt":2551},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2544,2547,2548],{"name":2545,"slug":2546,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2549,"slug":2550,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":2553,"name":2553,"fn":2554,"description":2555,"org":2556,"tags":2557,"stars":23,"repoUrl":24,"updatedAt":2563},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2558,2559,2560],{"name":2514,"slug":2515,"type":15},{"name":9,"slug":8,"type":15},{"name":2561,"slug":2562,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":2565,"name":2565,"fn":2566,"description":2567,"org":2568,"tags":2569,"stars":23,"repoUrl":24,"updatedAt":2580},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2570,2573,2576,2577],{"name":2571,"slug":2572,"type":15},"Automation","automation",{"name":2574,"slug":2575,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":2578,"slug":2579,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":2582,"name":2582,"fn":2583,"description":2584,"org":2585,"tags":2586,"stars":23,"repoUrl":24,"updatedAt":2595},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2587,2588,2591,2592],{"name":2531,"slug":2532,"type":15},{"name":2589,"slug":2590,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":2593,"slug":2594,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":2597,"name":2597,"fn":2598,"description":2599,"org":2600,"tags":2601,"stars":23,"repoUrl":24,"updatedAt":2609},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2602,2603,2606],{"name":9,"slug":8,"type":15},{"name":2604,"slug":2605,"type":15},"Quantum Computing","quantum-computing",{"name":2607,"slug":2608,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":2612,"total":2761},[2613,2631,2647,2658,2670,2684,2697,2709,2720,2729,2743,2752],{"slug":2614,"name":2614,"fn":2615,"description":2616,"org":2617,"tags":2618,"stars":2628,"repoUrl":2629,"updatedAt":2630},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2619,2622,2625],{"name":2620,"slug":2621,"type":15},"Documentation","documentation",{"name":2623,"slug":2624,"type":15},"MCP","mcp",{"name":2626,"slug":2627,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2632,"name":2632,"fn":2633,"description":2634,"org":2635,"tags":2636,"stars":2644,"repoUrl":2645,"updatedAt":2646},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2637,2640,2641],{"name":2638,"slug":2639,"type":15},"Containers","containers",{"name":2531,"slug":2532,"type":15},{"name":2642,"slug":2643,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2648,"name":2648,"fn":2649,"description":2650,"org":2651,"tags":2652,"stars":2644,"repoUrl":2645,"updatedAt":2657},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2653,2656],{"name":2654,"slug":2655,"type":15},"CI\u002FCD","ci-cd",{"name":2531,"slug":2532,"type":15},"2026-07-14T05:25:59.97109",{"slug":2659,"name":2659,"fn":2660,"description":2661,"org":2662,"tags":2663,"stars":2644,"repoUrl":2645,"updatedAt":2669},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2664,2665,2666],{"name":2654,"slug":2655,"type":15},{"name":2531,"slug":2532,"type":15},{"name":2667,"slug":2668,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2671,"name":2671,"fn":2672,"description":2673,"org":2674,"tags":2675,"stars":2644,"repoUrl":2645,"updatedAt":2683},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2676,2679,2680],{"name":2677,"slug":2678,"type":15},"Debugging","debugging",{"name":2667,"slug":2668,"type":15},{"name":2681,"slug":2682,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2685,"name":2685,"fn":2686,"description":2687,"org":2688,"tags":2689,"stars":2644,"repoUrl":2645,"updatedAt":2696},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2690,2693],{"name":2691,"slug":2692,"type":15},"Best Practices","best-practices",{"name":2694,"slug":2695,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2698,"name":2698,"fn":2699,"description":2700,"org":2701,"tags":2702,"stars":2644,"repoUrl":2645,"updatedAt":2708},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2703,2704,2707],{"name":17,"slug":18,"type":15},{"name":2705,"slug":2706,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2710,"name":2710,"fn":2711,"description":2712,"org":2713,"tags":2714,"stars":2644,"repoUrl":2645,"updatedAt":2719},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2715,2718],{"name":2716,"slug":2717,"type":15},"QA","qa",{"name":2561,"slug":2562,"type":15},"2026-07-14T05:25:53.673039",{"slug":2721,"name":2721,"fn":2722,"description":2723,"org":2724,"tags":2725,"stars":2644,"repoUrl":2645,"updatedAt":2728},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2726,2727],{"name":2531,"slug":2532,"type":15},{"name":2534,"slug":2535,"type":15},"2026-07-14T05:25:49.362534",{"slug":2730,"name":2730,"fn":2731,"description":2732,"org":2733,"tags":2734,"stars":2644,"repoUrl":2645,"updatedAt":2742},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2735,2738,2739],{"name":2736,"slug":2737,"type":15},"Code Review","code-review",{"name":2667,"slug":2668,"type":15},{"name":2740,"slug":2741,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2744,"name":2744,"fn":2745,"description":2746,"org":2747,"tags":2748,"stars":2644,"repoUrl":2645,"updatedAt":2751},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2749,2750],{"name":2716,"slug":2717,"type":15},{"name":2561,"slug":2562,"type":15},"2026-07-14T05:25:54.928983",{"slug":2753,"name":2753,"fn":2754,"description":2755,"org":2756,"tags":2757,"stars":2644,"repoUrl":2645,"updatedAt":2760},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2758,2759],{"name":2571,"slug":2572,"type":15},{"name":2654,"slug":2655,"type":15},"2026-07-30T05:29:03.275638",496]