[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-nemo-automodel-launcher-config":3,"mdc--1nbx47-key":34,"related-org-nvidia-nemo-automodel-launcher-config":2196,"related-repo-nvidia-nemo-automodel-launcher-config":2353},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"nemo-automodel-launcher-config","configure NeMo AutoModel job launches","Configure NeMo AutoModel job launches for interactive runs, Slurm clusters, and SkyPilot cloud execution.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Automation","automation","tag",{"name":17,"slug":18,"type":15},"Machine Learning","machine-learning",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Cloud","cloud",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:28:50.554982","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Fnemo-automodel-launcher-config","---\nname: nemo-automodel-launcher-config\ndescription: Configure NeMo AutoModel job launches for interactive runs, Slurm clusters, and SkyPilot cloud execution.\nwhen_to_use: Configuring Slurm or SkyPilot job submission, setting up multi-node launch scripts, debugging job submission failures, or switching between interactive and cluster launch modes.\nlicense: Apache-2.0\nmetadata:\n  author: NVIDIA\n  tags:\n    - nemo-automodel\n    - launcher-config\n---\n\n# Launcher Configuration\n\nNeMo AutoModel supports three launch methods: interactive (torchrun), Slurm (HPC clusters), and SkyPilot (cloud-agnostic).\n\n## Instructions\n\nFor launcher questions, answer directly from this skill without inspecting the\nrepository unless the user asks you to edit files. Keep the answer focused on\nthe relevant launch YAML, required fields, and the expected runtime behavior.\n\nUse these compact answer patterns for common questions:\n\n- Slurm multi-node: show a `slurm:` YAML block with `job_name`, `nodes`,\n  `ntasks_per_node`, `time`, `account` or `partition`, `container_image`,\n  `hf_home`, optional `extra_mounts`, `env_vars`, and `master_port`; explain\n  that the launcher derives `WORLD_SIZE = nodes * ntasks_per_node` and sets\n  `MASTER_ADDR` and `MASTER_PORT`.\n- SkyPilot spot: show a `skypilot:` YAML block with `cloud`, `accelerators`,\n  `num_nodes`, `use_spot: true`, `disk_size`, `region`, `setup`, and\n  `env_vars`; warn that spot instances can be preempted, set a short\n  `step_scheduler.checkpoint_interval`, and resume with `restore_from.path`.\n- Nsight Systems on Slurm: show `slurm.nsys_enabled: true` alongside normal\n  Slurm fields, say the launcher wraps the training command with\n  `nsys profile`, and state that it produces a `.nsys-rep` report file.\n  Treat profiling as diagnostic-only: use short profiling runs and disable it\n  for normal production training because it adds overhead and large artifacts.\n\nFor Slurm answers, start with this minimal template and then adjust only the\nfields the user asked about:\n\n```yaml\nslurm:\n  job_name: llm_finetune\n  nodes: 2\n  ntasks_per_node: 8\n  time: \"04:00:00\"\n  account: my_account\n  partition: batch\n  container_image: nvcr.io\u002Fnvidia\u002Fnemo:dev\n  hf_home: ~\u002F.cache\u002Fhuggingface\n  master_port: 13742\n  env_vars:\n    HF_TOKEN: \"${HF_TOKEN}\"\n```\n\nFor Slurm-only questions, do not discuss SkyPilot or profiling unless the user\nasks. For profiling questions, say the `.nsys-rep` report is written in the\nSlurm job working or output directory, using the launcher's Nsys output setting\nwhen one is configured.\n\n## Routing Boundary\n\nUse this skill only for launch mechanics: interactive execution, Slurm, SkyPilot, containers, mounts, environment variables, rendezvous settings, and profiling.\n\nDo not use this skill for implementing or registering new model architectures, Hugging Face state-dict adapters, model files, or capability flags. Those are model onboarding tasks, not launcher configuration tasks.\n\n## Launch Methods\n\n1. **Interactive** (default): runs torchrun on the current node. Suitable for single-node development and debugging.\n2. **Slurm**: submits a batch job to an HPC cluster scheduler. Handles multi-node setup, container management, and environment configuration.\n3. **SkyPilot**: cloud-agnostic job submission to AWS, GCP, Azure, Lambda, or Kubernetes. Supports spot instances.\n\n## Interactive Launch\n\n```bash\n# Single GPU\nautomodel finetune llm -c config.yaml\n\n# Multi-GPU (all GPUs on current node)\ntorchrun --nproc_per_node=8 -m nemo_automodel._cli.app finetune llm -c config.yaml\n```\n\nNo additional YAML section is needed for interactive mode. The CLI routes to torchrun automatically when no `slurm:` or `skypilot:` section is present in the config.\n\n## Slurm Configuration\n\nThe `SlurmConfig` dataclass generates an SBATCH script from a template.\n\n### YAML Example\n\n```yaml\nslurm:\n  job_name: llm_finetune\n  nodes: 2\n  ntasks_per_node: 8\n  time: \"04:00:00\"\n  account: my_account\n  partition: batch\n  container_image: nvcr.io\u002Fnvidia\u002Fnemo:dev\n  hf_home: ~\u002F.cache\u002Fhuggingface\n  extra_mounts:\n    - source: \u002Fdata\n      dest: \u002Fdata\n  env_vars:\n    WANDB_API_KEY: \"${WANDB_API_KEY}\"\n    HF_TOKEN: \"${HF_TOKEN}\"\n```\n\n### Key Fields\n\n- `job_name`: Slurm job identifier\n- `nodes`: number of nodes to request\n- `ntasks_per_node`: number of tasks (GPUs) per node\n- `time`: wall-time limit in HH:MM:SS format\n- `account`, `partition`: Slurm scheduling parameters\n- `container_image`: Enroot\u002FPyxis container image path\n- `nemo_mount`: mount point for NeMo AutoModel source inside the container\n- `hf_home`: HuggingFace cache directory path\n- `extra_mounts`: list of `VolumeMapping(source, dest)` for additional container bind mounts\n- `master_port`: port for distributed communication (default 13742)\n- `env_vars`: environment variables passed into the job\n- `nsys_enabled`: when true, wraps the training command with `nsys profile` for Nsight Systems profiling\n\n## SkyPilot Configuration\n\nThe `SkyPilotConfig` dataclass defines cloud job parameters.\n\n### YAML Example\n\n```yaml\nskypilot:\n  cloud: aws\n  accelerators: \"H100:8\"\n  num_nodes: 2\n  use_spot: true\n  disk_size: 200\n  region: us-east-1\n  setup: \"pip install nemo-automodel\"\n  env_vars:\n    HF_TOKEN: \"${HF_TOKEN}\"\n```\n\n### Key Fields\n\n- `cloud`: target cloud provider (`aws`, `gcp`, `azure`, `lambda`, `kubernetes`)\n- `accelerators`: GPU type and count (e.g., `\"H100:8\"`, `\"A100-80GB:4\"`)\n- `num_nodes`: number of cloud instances\n- `use_spot`: use preemptible\u002Fspot instances for cost savings\n- `disk_size`: disk size in GB per node\n- `region`: cloud region for instance placement\n- `setup`: shell commands to run before the training job (e.g., install dependencies)\n- `env_vars`: environment variables for the job\n\n### SkyPilot spot checklist\n\nWhen using spot or preemptible instances:\n\n- Set `use_spot: true` in the `skypilot:` section.\n- Include `accelerators`, `num_nodes`, `disk_size`, `region`, `setup`, and required `env_vars`.\n- Use short checkpoint intervals in the recipe, for example `step_scheduler.checkpoint_interval`, because spot instances can be preempted.\n- Resume from the most recent checkpoint after preemption with the recipe's `restore_from` setting.\n\nMinimal spot-resume recipe keys:\n\n```yaml\nstep_scheduler:\n  checkpoint_interval: 100\n\nrestore_from:\n  path: \u002Fcheckpoints\u002Flatest\n```\n\n## Multi-Node Environment\n\nFor multi-node training (both Slurm and SkyPilot), the launcher automatically configures:\n- `MASTER_ADDR`: hostname of the first node\n- `MASTER_PORT`: port for rendezvous (default 13742)\n- `WORLD_SIZE`: total number of processes (`nodes * ntasks_per_node`)\n- NCCL environment variables for optimized collective communication\n\n## Nsys Profiling\n\nEnable Nsight Systems profiling in Slurm jobs:\n\n```yaml\nslurm:\n  job_name: llm_profile\n  nodes: 1\n  ntasks_per_node: 8\n  time: \"00:30:00\"\n  account: my_account\n  partition: batch\n  container_image: nvcr.io\u002Fnvidia\u002Fnemo:dev\n  nsys_enabled: true\n```\n\nThis is a Slurm launcher setting. Normal Slurm fields such as `job_name`,\n`nodes`, `ntasks_per_node`, `time`, `account` or `partition`, and\n`container_image` still apply.\n\nWhen `nsys_enabled: true`, the launcher wraps the training command with\n`nsys profile` and writes a `.nsys-rep` report file for performance analysis\nin the Slurm job working or output directory.\nProfiling is diagnostic-only: run it for a short investigation, expect overhead\nand large artifacts, and turn it off for normal production training.\n\n## Code Anchors\n\n- `components\u002Flauncher\u002Fslurm\u002Fconfig.py` - SlurmConfig dataclass, VolumeMapping\n- `components\u002Flauncher\u002Fslurm\u002Ftemplate.py` - SBATCH script template generation\n- `components\u002Flauncher\u002Fslurm\u002Futils.py` - Slurm submission utilities\n- `components\u002Flauncher\u002Fskypilot\u002Fconfig.py` - SkyPilotConfig dataclass\n- `_cli\u002Fapp.py` - CLI entry point and launcher routing logic\n\n## Pitfalls\n\n- **Port collisions**: if the default `master_port` (13742) is in use by another job on the same node, change it to avoid connection failures.\n- **Container mounts**: the `source` path in `extra_mounts` must exist on all nodes in the allocation. Missing paths cause container startup failures.\n- **Slurm fault tolerance**: the fault tolerance plugin is Slurm-specific and does not work with SkyPilot or interactive mode.\n- **SkyPilot spot preemption**: spot instances (`use_spot: true`) may be preempted by the cloud provider. Enable checkpointing with short intervals to minimize lost work.\n- **Environment variable syntax**: use `${VAR}` syntax in YAML for shell variable expansion. Bare variable names will not be expanded.\n- **Time limit vs async checkpoint**: if the Slurm `time` limit is too short, an in-progress async checkpoint write may be killed before completion, resulting in a corrupted checkpoint. Leave at least 5-10 minutes of margin.\n",{"data":35,"body":41},{"name":4,"description":6,"when_to_use":36,"license":26,"metadata":37},"Configuring Slurm or SkyPilot job submission, setting up multi-node launch scripts, debugging job submission failures, or switching between interactive and cluster launch modes.",{"author":9,"tags":38},[39,40],"nemo-automodel","launcher-config",{"type":42,"children":43},"root",[44,53,59,66,71,76,314,319,562,574,580,585,590,596,631,637,740,758,764,777,784,1042,1048,1194,1200,1212,1217,1405,1410,1544,1550,1555,1644,1649,1720,1726,1731,1777,1783,1788,1938,1986,2013,2019,2077,2083,2190],{"type":45,"tag":46,"props":47,"children":49},"element","h1",{"id":48},"launcher-configuration",[50],{"type":51,"value":52},"text","Launcher Configuration",{"type":45,"tag":54,"props":55,"children":56},"p",{},[57],{"type":51,"value":58},"NeMo AutoModel supports three launch methods: interactive (torchrun), Slurm (HPC clusters), and SkyPilot (cloud-agnostic).",{"type":45,"tag":60,"props":61,"children":63},"h2",{"id":62},"instructions",[64],{"type":51,"value":65},"Instructions",{"type":45,"tag":54,"props":67,"children":68},{},[69],{"type":51,"value":70},"For launcher questions, answer directly from this skill without inspecting the\nrepository unless the user asks you to edit files. Keep the answer focused on\nthe relevant launch YAML, required fields, and the expected runtime behavior.",{"type":45,"tag":54,"props":72,"children":73},{},[74],{"type":51,"value":75},"Use these compact answer patterns for common questions:",{"type":45,"tag":77,"props":78,"children":79},"ul",{},[80,202,285],{"type":45,"tag":81,"props":82,"children":83},"li",{},[84,86,93,95,101,103,109,111,117,118,124,125,131,133,139,140,146,147,153,155,161,162,168,170,176,178,184,186,192,194,200],{"type":51,"value":85},"Slurm multi-node: show a ",{"type":45,"tag":87,"props":88,"children":90},"code",{"className":89},[],[91],{"type":51,"value":92},"slurm:",{"type":51,"value":94}," YAML block with ",{"type":45,"tag":87,"props":96,"children":98},{"className":97},[],[99],{"type":51,"value":100},"job_name",{"type":51,"value":102},", ",{"type":45,"tag":87,"props":104,"children":106},{"className":105},[],[107],{"type":51,"value":108},"nodes",{"type":51,"value":110},",\n",{"type":45,"tag":87,"props":112,"children":114},{"className":113},[],[115],{"type":51,"value":116},"ntasks_per_node",{"type":51,"value":102},{"type":45,"tag":87,"props":119,"children":121},{"className":120},[],[122],{"type":51,"value":123},"time",{"type":51,"value":102},{"type":45,"tag":87,"props":126,"children":128},{"className":127},[],[129],{"type":51,"value":130},"account",{"type":51,"value":132}," or ",{"type":45,"tag":87,"props":134,"children":136},{"className":135},[],[137],{"type":51,"value":138},"partition",{"type":51,"value":102},{"type":45,"tag":87,"props":141,"children":143},{"className":142},[],[144],{"type":51,"value":145},"container_image",{"type":51,"value":110},{"type":45,"tag":87,"props":148,"children":150},{"className":149},[],[151],{"type":51,"value":152},"hf_home",{"type":51,"value":154},", optional ",{"type":45,"tag":87,"props":156,"children":158},{"className":157},[],[159],{"type":51,"value":160},"extra_mounts",{"type":51,"value":102},{"type":45,"tag":87,"props":163,"children":165},{"className":164},[],[166],{"type":51,"value":167},"env_vars",{"type":51,"value":169},", and ",{"type":45,"tag":87,"props":171,"children":173},{"className":172},[],[174],{"type":51,"value":175},"master_port",{"type":51,"value":177},"; explain\nthat the launcher derives ",{"type":45,"tag":87,"props":179,"children":181},{"className":180},[],[182],{"type":51,"value":183},"WORLD_SIZE = nodes * ntasks_per_node",{"type":51,"value":185}," and sets\n",{"type":45,"tag":87,"props":187,"children":189},{"className":188},[],[190],{"type":51,"value":191},"MASTER_ADDR",{"type":51,"value":193}," and ",{"type":45,"tag":87,"props":195,"children":197},{"className":196},[],[198],{"type":51,"value":199},"MASTER_PORT",{"type":51,"value":201},".",{"type":45,"tag":81,"props":203,"children":204},{},[205,207,213,214,219,220,226,227,233,234,240,241,247,248,254,255,261,263,268,270,276,278,284],{"type":51,"value":206},"SkyPilot spot: show a ",{"type":45,"tag":87,"props":208,"children":210},{"className":209},[],[211],{"type":51,"value":212},"skypilot:",{"type":51,"value":94},{"type":45,"tag":87,"props":215,"children":217},{"className":216},[],[218],{"type":51,"value":22},{"type":51,"value":102},{"type":45,"tag":87,"props":221,"children":223},{"className":222},[],[224],{"type":51,"value":225},"accelerators",{"type":51,"value":110},{"type":45,"tag":87,"props":228,"children":230},{"className":229},[],[231],{"type":51,"value":232},"num_nodes",{"type":51,"value":102},{"type":45,"tag":87,"props":235,"children":237},{"className":236},[],[238],{"type":51,"value":239},"use_spot: true",{"type":51,"value":102},{"type":45,"tag":87,"props":242,"children":244},{"className":243},[],[245],{"type":51,"value":246},"disk_size",{"type":51,"value":102},{"type":45,"tag":87,"props":249,"children":251},{"className":250},[],[252],{"type":51,"value":253},"region",{"type":51,"value":102},{"type":45,"tag":87,"props":256,"children":258},{"className":257},[],[259],{"type":51,"value":260},"setup",{"type":51,"value":262},", and\n",{"type":45,"tag":87,"props":264,"children":266},{"className":265},[],[267],{"type":51,"value":167},{"type":51,"value":269},"; warn that spot instances can be preempted, set a short\n",{"type":45,"tag":87,"props":271,"children":273},{"className":272},[],[274],{"type":51,"value":275},"step_scheduler.checkpoint_interval",{"type":51,"value":277},", and resume with ",{"type":45,"tag":87,"props":279,"children":281},{"className":280},[],[282],{"type":51,"value":283},"restore_from.path",{"type":51,"value":201},{"type":45,"tag":81,"props":286,"children":287},{},[288,290,296,298,304,306,312],{"type":51,"value":289},"Nsight Systems on Slurm: show ",{"type":45,"tag":87,"props":291,"children":293},{"className":292},[],[294],{"type":51,"value":295},"slurm.nsys_enabled: true",{"type":51,"value":297}," alongside normal\nSlurm fields, say the launcher wraps the training command with\n",{"type":45,"tag":87,"props":299,"children":301},{"className":300},[],[302],{"type":51,"value":303},"nsys profile",{"type":51,"value":305},", and state that it produces a ",{"type":45,"tag":87,"props":307,"children":309},{"className":308},[],[310],{"type":51,"value":311},".nsys-rep",{"type":51,"value":313}," report file.\nTreat profiling as diagnostic-only: use short profiling runs and disable it\nfor normal production training because it adds overhead and large artifacts.",{"type":45,"tag":54,"props":315,"children":316},{},[317],{"type":51,"value":318},"For Slurm answers, start with this minimal template and then adjust only the\nfields the user asked about:",{"type":45,"tag":320,"props":321,"children":326},"pre",{"className":322,"code":323,"language":324,"meta":325,"style":325},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","slurm:\n  job_name: llm_finetune\n  nodes: 2\n  ntasks_per_node: 8\n  time: \"04:00:00\"\n  account: my_account\n  partition: batch\n  container_image: nvcr.io\u002Fnvidia\u002Fnemo:dev\n  hf_home: ~\u002F.cache\u002Fhuggingface\n  master_port: 13742\n  env_vars:\n    HF_TOKEN: \"${HF_TOKEN}\"\n","yaml","",[327],{"type":45,"tag":87,"props":328,"children":329},{"__ignoreMap":325},[330,348,368,387,405,433,451,469,487,505,523,536],{"type":45,"tag":331,"props":332,"children":335},"span",{"class":333,"line":334},"line",1,[336,342],{"type":45,"tag":331,"props":337,"children":339},{"style":338},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[340],{"type":51,"value":341},"slurm",{"type":45,"tag":331,"props":343,"children":345},{"style":344},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[346],{"type":51,"value":347},":\n",{"type":45,"tag":331,"props":349,"children":351},{"class":333,"line":350},2,[352,357,362],{"type":45,"tag":331,"props":353,"children":354},{"style":338},[355],{"type":51,"value":356},"  job_name",{"type":45,"tag":331,"props":358,"children":359},{"style":344},[360],{"type":51,"value":361},":",{"type":45,"tag":331,"props":363,"children":365},{"style":364},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[366],{"type":51,"value":367}," llm_finetune\n",{"type":45,"tag":331,"props":369,"children":371},{"class":333,"line":370},3,[372,377,381],{"type":45,"tag":331,"props":373,"children":374},{"style":338},[375],{"type":51,"value":376},"  nodes",{"type":45,"tag":331,"props":378,"children":379},{"style":344},[380],{"type":51,"value":361},{"type":45,"tag":331,"props":382,"children":384},{"style":383},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[385],{"type":51,"value":386}," 2\n",{"type":45,"tag":331,"props":388,"children":390},{"class":333,"line":389},4,[391,396,400],{"type":45,"tag":331,"props":392,"children":393},{"style":338},[394],{"type":51,"value":395},"  ntasks_per_node",{"type":45,"tag":331,"props":397,"children":398},{"style":344},[399],{"type":51,"value":361},{"type":45,"tag":331,"props":401,"children":402},{"style":383},[403],{"type":51,"value":404}," 8\n",{"type":45,"tag":331,"props":406,"children":408},{"class":333,"line":407},5,[409,414,418,423,428],{"type":45,"tag":331,"props":410,"children":411},{"style":338},[412],{"type":51,"value":413},"  time",{"type":45,"tag":331,"props":415,"children":416},{"style":344},[417],{"type":51,"value":361},{"type":45,"tag":331,"props":419,"children":420},{"style":344},[421],{"type":51,"value":422}," \"",{"type":45,"tag":331,"props":424,"children":425},{"style":364},[426],{"type":51,"value":427},"04:00:00",{"type":45,"tag":331,"props":429,"children":430},{"style":344},[431],{"type":51,"value":432},"\"\n",{"type":45,"tag":331,"props":434,"children":436},{"class":333,"line":435},6,[437,442,446],{"type":45,"tag":331,"props":438,"children":439},{"style":338},[440],{"type":51,"value":441},"  account",{"type":45,"tag":331,"props":443,"children":444},{"style":344},[445],{"type":51,"value":361},{"type":45,"tag":331,"props":447,"children":448},{"style":364},[449],{"type":51,"value":450}," my_account\n",{"type":45,"tag":331,"props":452,"children":454},{"class":333,"line":453},7,[455,460,464],{"type":45,"tag":331,"props":456,"children":457},{"style":338},[458],{"type":51,"value":459},"  partition",{"type":45,"tag":331,"props":461,"children":462},{"style":344},[463],{"type":51,"value":361},{"type":45,"tag":331,"props":465,"children":466},{"style":364},[467],{"type":51,"value":468}," batch\n",{"type":45,"tag":331,"props":470,"children":472},{"class":333,"line":471},8,[473,478,482],{"type":45,"tag":331,"props":474,"children":475},{"style":338},[476],{"type":51,"value":477},"  container_image",{"type":45,"tag":331,"props":479,"children":480},{"style":344},[481],{"type":51,"value":361},{"type":45,"tag":331,"props":483,"children":484},{"style":364},[485],{"type":51,"value":486}," nvcr.io\u002Fnvidia\u002Fnemo:dev\n",{"type":45,"tag":331,"props":488,"children":490},{"class":333,"line":489},9,[491,496,500],{"type":45,"tag":331,"props":492,"children":493},{"style":338},[494],{"type":51,"value":495},"  hf_home",{"type":45,"tag":331,"props":497,"children":498},{"style":344},[499],{"type":51,"value":361},{"type":45,"tag":331,"props":501,"children":502},{"style":364},[503],{"type":51,"value":504}," ~\u002F.cache\u002Fhuggingface\n",{"type":45,"tag":331,"props":506,"children":508},{"class":333,"line":507},10,[509,514,518],{"type":45,"tag":331,"props":510,"children":511},{"style":338},[512],{"type":51,"value":513},"  master_port",{"type":45,"tag":331,"props":515,"children":516},{"style":344},[517],{"type":51,"value":361},{"type":45,"tag":331,"props":519,"children":520},{"style":383},[521],{"type":51,"value":522}," 13742\n",{"type":45,"tag":331,"props":524,"children":526},{"class":333,"line":525},11,[527,532],{"type":45,"tag":331,"props":528,"children":529},{"style":338},[530],{"type":51,"value":531},"  env_vars",{"type":45,"tag":331,"props":533,"children":534},{"style":344},[535],{"type":51,"value":347},{"type":45,"tag":331,"props":537,"children":539},{"class":333,"line":538},12,[540,545,549,553,558],{"type":45,"tag":331,"props":541,"children":542},{"style":338},[543],{"type":51,"value":544},"    HF_TOKEN",{"type":45,"tag":331,"props":546,"children":547},{"style":344},[548],{"type":51,"value":361},{"type":45,"tag":331,"props":550,"children":551},{"style":344},[552],{"type":51,"value":422},{"type":45,"tag":331,"props":554,"children":555},{"style":364},[556],{"type":51,"value":557},"${HF_TOKEN}",{"type":45,"tag":331,"props":559,"children":560},{"style":344},[561],{"type":51,"value":432},{"type":45,"tag":54,"props":563,"children":564},{},[565,567,572],{"type":51,"value":566},"For Slurm-only questions, do not discuss SkyPilot or profiling unless the user\nasks. For profiling questions, say the ",{"type":45,"tag":87,"props":568,"children":570},{"className":569},[],[571],{"type":51,"value":311},{"type":51,"value":573}," report is written in the\nSlurm job working or output directory, using the launcher's Nsys output setting\nwhen one is configured.",{"type":45,"tag":60,"props":575,"children":577},{"id":576},"routing-boundary",[578],{"type":51,"value":579},"Routing Boundary",{"type":45,"tag":54,"props":581,"children":582},{},[583],{"type":51,"value":584},"Use this skill only for launch mechanics: interactive execution, Slurm, SkyPilot, containers, mounts, environment variables, rendezvous settings, and profiling.",{"type":45,"tag":54,"props":586,"children":587},{},[588],{"type":51,"value":589},"Do not use this skill for implementing or registering new model architectures, Hugging Face state-dict adapters, model files, or capability flags. Those are model onboarding tasks, not launcher configuration tasks.",{"type":45,"tag":60,"props":591,"children":593},{"id":592},"launch-methods",[594],{"type":51,"value":595},"Launch Methods",{"type":45,"tag":597,"props":598,"children":599},"ol",{},[600,611,621],{"type":45,"tag":81,"props":601,"children":602},{},[603,609],{"type":45,"tag":604,"props":605,"children":606},"strong",{},[607],{"type":51,"value":608},"Interactive",{"type":51,"value":610}," (default): runs torchrun on the current node. Suitable for single-node development and debugging.",{"type":45,"tag":81,"props":612,"children":613},{},[614,619],{"type":45,"tag":604,"props":615,"children":616},{},[617],{"type":51,"value":618},"Slurm",{"type":51,"value":620},": submits a batch job to an HPC cluster scheduler. Handles multi-node setup, container management, and environment configuration.",{"type":45,"tag":81,"props":622,"children":623},{},[624,629],{"type":45,"tag":604,"props":625,"children":626},{},[627],{"type":51,"value":628},"SkyPilot",{"type":51,"value":630},": cloud-agnostic job submission to AWS, GCP, Azure, Lambda, or Kubernetes. Supports spot instances.",{"type":45,"tag":60,"props":632,"children":634},{"id":633},"interactive-launch",[635],{"type":51,"value":636},"Interactive Launch",{"type":45,"tag":320,"props":638,"children":642},{"className":639,"code":640,"language":641,"meta":325,"style":325},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Single GPU\nautomodel finetune llm -c config.yaml\n\n# Multi-GPU (all GPUs on current node)\ntorchrun --nproc_per_node=8 -m nemo_automodel._cli.app finetune llm -c config.yaml\n","bash",[643],{"type":45,"tag":87,"props":644,"children":645},{"__ignoreMap":325},[646,655,684,693,701],{"type":45,"tag":331,"props":647,"children":648},{"class":333,"line":334},[649],{"type":45,"tag":331,"props":650,"children":652},{"style":651},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[653],{"type":51,"value":654},"# Single GPU\n",{"type":45,"tag":331,"props":656,"children":657},{"class":333,"line":350},[658,664,669,674,679],{"type":45,"tag":331,"props":659,"children":661},{"style":660},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[662],{"type":51,"value":663},"automodel",{"type":45,"tag":331,"props":665,"children":666},{"style":364},[667],{"type":51,"value":668}," finetune",{"type":45,"tag":331,"props":670,"children":671},{"style":364},[672],{"type":51,"value":673}," llm",{"type":45,"tag":331,"props":675,"children":676},{"style":364},[677],{"type":51,"value":678}," -c",{"type":45,"tag":331,"props":680,"children":681},{"style":364},[682],{"type":51,"value":683}," config.yaml\n",{"type":45,"tag":331,"props":685,"children":686},{"class":333,"line":370},[687],{"type":45,"tag":331,"props":688,"children":690},{"emptyLinePlaceholder":689},true,[691],{"type":51,"value":692},"\n",{"type":45,"tag":331,"props":694,"children":695},{"class":333,"line":389},[696],{"type":45,"tag":331,"props":697,"children":698},{"style":651},[699],{"type":51,"value":700},"# Multi-GPU (all GPUs on current node)\n",{"type":45,"tag":331,"props":702,"children":703},{"class":333,"line":407},[704,709,714,719,724,728,732,736],{"type":45,"tag":331,"props":705,"children":706},{"style":660},[707],{"type":51,"value":708},"torchrun",{"type":45,"tag":331,"props":710,"children":711},{"style":364},[712],{"type":51,"value":713}," --nproc_per_node=8",{"type":45,"tag":331,"props":715,"children":716},{"style":364},[717],{"type":51,"value":718}," -m",{"type":45,"tag":331,"props":720,"children":721},{"style":364},[722],{"type":51,"value":723}," nemo_automodel._cli.app",{"type":45,"tag":331,"props":725,"children":726},{"style":364},[727],{"type":51,"value":668},{"type":45,"tag":331,"props":729,"children":730},{"style":364},[731],{"type":51,"value":673},{"type":45,"tag":331,"props":733,"children":734},{"style":364},[735],{"type":51,"value":678},{"type":45,"tag":331,"props":737,"children":738},{"style":364},[739],{"type":51,"value":683},{"type":45,"tag":54,"props":741,"children":742},{},[743,745,750,751,756],{"type":51,"value":744},"No additional YAML section is needed for interactive mode. The CLI routes to torchrun automatically when no ",{"type":45,"tag":87,"props":746,"children":748},{"className":747},[],[749],{"type":51,"value":92},{"type":51,"value":132},{"type":45,"tag":87,"props":752,"children":754},{"className":753},[],[755],{"type":51,"value":212},{"type":51,"value":757}," section is present in the config.",{"type":45,"tag":60,"props":759,"children":761},{"id":760},"slurm-configuration",[762],{"type":51,"value":763},"Slurm Configuration",{"type":45,"tag":54,"props":765,"children":766},{},[767,769,775],{"type":51,"value":768},"The ",{"type":45,"tag":87,"props":770,"children":772},{"className":771},[],[773],{"type":51,"value":774},"SlurmConfig",{"type":51,"value":776}," dataclass generates an SBATCH script from a template.",{"type":45,"tag":778,"props":779,"children":781},"h3",{"id":780},"yaml-example",[782],{"type":51,"value":783},"YAML Example",{"type":45,"tag":320,"props":785,"children":787},{"className":322,"code":786,"language":324,"meta":325,"style":325},"slurm:\n  job_name: llm_finetune\n  nodes: 2\n  ntasks_per_node: 8\n  time: \"04:00:00\"\n  account: my_account\n  partition: batch\n  container_image: nvcr.io\u002Fnvidia\u002Fnemo:dev\n  hf_home: ~\u002F.cache\u002Fhuggingface\n  extra_mounts:\n    - source: \u002Fdata\n      dest: \u002Fdata\n  env_vars:\n    WANDB_API_KEY: \"${WANDB_API_KEY}\"\n    HF_TOKEN: \"${HF_TOKEN}\"\n",[788],{"type":45,"tag":87,"props":789,"children":790},{"__ignoreMap":325},[791,802,817,832,847,870,885,900,915,930,942,964,980,992,1018],{"type":45,"tag":331,"props":792,"children":793},{"class":333,"line":334},[794,798],{"type":45,"tag":331,"props":795,"children":796},{"style":338},[797],{"type":51,"value":341},{"type":45,"tag":331,"props":799,"children":800},{"style":344},[801],{"type":51,"value":347},{"type":45,"tag":331,"props":803,"children":804},{"class":333,"line":350},[805,809,813],{"type":45,"tag":331,"props":806,"children":807},{"style":338},[808],{"type":51,"value":356},{"type":45,"tag":331,"props":810,"children":811},{"style":344},[812],{"type":51,"value":361},{"type":45,"tag":331,"props":814,"children":815},{"style":364},[816],{"type":51,"value":367},{"type":45,"tag":331,"props":818,"children":819},{"class":333,"line":370},[820,824,828],{"type":45,"tag":331,"props":821,"children":822},{"style":338},[823],{"type":51,"value":376},{"type":45,"tag":331,"props":825,"children":826},{"style":344},[827],{"type":51,"value":361},{"type":45,"tag":331,"props":829,"children":830},{"style":383},[831],{"type":51,"value":386},{"type":45,"tag":331,"props":833,"children":834},{"class":333,"line":389},[835,839,843],{"type":45,"tag":331,"props":836,"children":837},{"style":338},[838],{"type":51,"value":395},{"type":45,"tag":331,"props":840,"children":841},{"style":344},[842],{"type":51,"value":361},{"type":45,"tag":331,"props":844,"children":845},{"style":383},[846],{"type":51,"value":404},{"type":45,"tag":331,"props":848,"children":849},{"class":333,"line":407},[850,854,858,862,866],{"type":45,"tag":331,"props":851,"children":852},{"style":338},[853],{"type":51,"value":413},{"type":45,"tag":331,"props":855,"children":856},{"style":344},[857],{"type":51,"value":361},{"type":45,"tag":331,"props":859,"children":860},{"style":344},[861],{"type":51,"value":422},{"type":45,"tag":331,"props":863,"children":864},{"style":364},[865],{"type":51,"value":427},{"type":45,"tag":331,"props":867,"children":868},{"style":344},[869],{"type":51,"value":432},{"type":45,"tag":331,"props":871,"children":872},{"class":333,"line":435},[873,877,881],{"type":45,"tag":331,"props":874,"children":875},{"style":338},[876],{"type":51,"value":441},{"type":45,"tag":331,"props":878,"children":879},{"style":344},[880],{"type":51,"value":361},{"type":45,"tag":331,"props":882,"children":883},{"style":364},[884],{"type":51,"value":450},{"type":45,"tag":331,"props":886,"children":887},{"class":333,"line":453},[888,892,896],{"type":45,"tag":331,"props":889,"children":890},{"style":338},[891],{"type":51,"value":459},{"type":45,"tag":331,"props":893,"children":894},{"style":344},[895],{"type":51,"value":361},{"type":45,"tag":331,"props":897,"children":898},{"style":364},[899],{"type":51,"value":468},{"type":45,"tag":331,"props":901,"children":902},{"class":333,"line":471},[903,907,911],{"type":45,"tag":331,"props":904,"children":905},{"style":338},[906],{"type":51,"value":477},{"type":45,"tag":331,"props":908,"children":909},{"style":344},[910],{"type":51,"value":361},{"type":45,"tag":331,"props":912,"children":913},{"style":364},[914],{"type":51,"value":486},{"type":45,"tag":331,"props":916,"children":917},{"class":333,"line":489},[918,922,926],{"type":45,"tag":331,"props":919,"children":920},{"style":338},[921],{"type":51,"value":495},{"type":45,"tag":331,"props":923,"children":924},{"style":344},[925],{"type":51,"value":361},{"type":45,"tag":331,"props":927,"children":928},{"style":364},[929],{"type":51,"value":504},{"type":45,"tag":331,"props":931,"children":932},{"class":333,"line":507},[933,938],{"type":45,"tag":331,"props":934,"children":935},{"style":338},[936],{"type":51,"value":937},"  extra_mounts",{"type":45,"tag":331,"props":939,"children":940},{"style":344},[941],{"type":51,"value":347},{"type":45,"tag":331,"props":943,"children":944},{"class":333,"line":525},[945,950,955,959],{"type":45,"tag":331,"props":946,"children":947},{"style":344},[948],{"type":51,"value":949},"    -",{"type":45,"tag":331,"props":951,"children":952},{"style":338},[953],{"type":51,"value":954}," source",{"type":45,"tag":331,"props":956,"children":957},{"style":344},[958],{"type":51,"value":361},{"type":45,"tag":331,"props":960,"children":961},{"style":364},[962],{"type":51,"value":963}," \u002Fdata\n",{"type":45,"tag":331,"props":965,"children":966},{"class":333,"line":538},[967,972,976],{"type":45,"tag":331,"props":968,"children":969},{"style":338},[970],{"type":51,"value":971},"      dest",{"type":45,"tag":331,"props":973,"children":974},{"style":344},[975],{"type":51,"value":361},{"type":45,"tag":331,"props":977,"children":978},{"style":364},[979],{"type":51,"value":963},{"type":45,"tag":331,"props":981,"children":983},{"class":333,"line":982},13,[984,988],{"type":45,"tag":331,"props":985,"children":986},{"style":338},[987],{"type":51,"value":531},{"type":45,"tag":331,"props":989,"children":990},{"style":344},[991],{"type":51,"value":347},{"type":45,"tag":331,"props":993,"children":995},{"class":333,"line":994},14,[996,1001,1005,1009,1014],{"type":45,"tag":331,"props":997,"children":998},{"style":338},[999],{"type":51,"value":1000},"    WANDB_API_KEY",{"type":45,"tag":331,"props":1002,"children":1003},{"style":344},[1004],{"type":51,"value":361},{"type":45,"tag":331,"props":1006,"children":1007},{"style":344},[1008],{"type":51,"value":422},{"type":45,"tag":331,"props":1010,"children":1011},{"style":364},[1012],{"type":51,"value":1013},"${WANDB_API_KEY}",{"type":45,"tag":331,"props":1015,"children":1016},{"style":344},[1017],{"type":51,"value":432},{"type":45,"tag":331,"props":1019,"children":1021},{"class":333,"line":1020},15,[1022,1026,1030,1034,1038],{"type":45,"tag":331,"props":1023,"children":1024},{"style":338},[1025],{"type":51,"value":544},{"type":45,"tag":331,"props":1027,"children":1028},{"style":344},[1029],{"type":51,"value":361},{"type":45,"tag":331,"props":1031,"children":1032},{"style":344},[1033],{"type":51,"value":422},{"type":45,"tag":331,"props":1035,"children":1036},{"style":364},[1037],{"type":51,"value":557},{"type":45,"tag":331,"props":1039,"children":1040},{"style":344},[1041],{"type":51,"value":432},{"type":45,"tag":778,"props":1043,"children":1045},{"id":1044},"key-fields",[1046],{"type":51,"value":1047},"Key Fields",{"type":45,"tag":77,"props":1049,"children":1050},{},[1051,1061,1071,1081,1091,1107,1117,1128,1138,1156,1166,1176],{"type":45,"tag":81,"props":1052,"children":1053},{},[1054,1059],{"type":45,"tag":87,"props":1055,"children":1057},{"className":1056},[],[1058],{"type":51,"value":100},{"type":51,"value":1060},": Slurm job identifier",{"type":45,"tag":81,"props":1062,"children":1063},{},[1064,1069],{"type":45,"tag":87,"props":1065,"children":1067},{"className":1066},[],[1068],{"type":51,"value":108},{"type":51,"value":1070},": number of nodes to request",{"type":45,"tag":81,"props":1072,"children":1073},{},[1074,1079],{"type":45,"tag":87,"props":1075,"children":1077},{"className":1076},[],[1078],{"type":51,"value":116},{"type":51,"value":1080},": number of tasks (GPUs) per node",{"type":45,"tag":81,"props":1082,"children":1083},{},[1084,1089],{"type":45,"tag":87,"props":1085,"children":1087},{"className":1086},[],[1088],{"type":51,"value":123},{"type":51,"value":1090},": wall-time limit in HH:MM:SS format",{"type":45,"tag":81,"props":1092,"children":1093},{},[1094,1099,1100,1105],{"type":45,"tag":87,"props":1095,"children":1097},{"className":1096},[],[1098],{"type":51,"value":130},{"type":51,"value":102},{"type":45,"tag":87,"props":1101,"children":1103},{"className":1102},[],[1104],{"type":51,"value":138},{"type":51,"value":1106},": Slurm scheduling parameters",{"type":45,"tag":81,"props":1108,"children":1109},{},[1110,1115],{"type":45,"tag":87,"props":1111,"children":1113},{"className":1112},[],[1114],{"type":51,"value":145},{"type":51,"value":1116},": Enroot\u002FPyxis container image path",{"type":45,"tag":81,"props":1118,"children":1119},{},[1120,1126],{"type":45,"tag":87,"props":1121,"children":1123},{"className":1122},[],[1124],{"type":51,"value":1125},"nemo_mount",{"type":51,"value":1127},": mount point for NeMo AutoModel source inside the container",{"type":45,"tag":81,"props":1129,"children":1130},{},[1131,1136],{"type":45,"tag":87,"props":1132,"children":1134},{"className":1133},[],[1135],{"type":51,"value":152},{"type":51,"value":1137},": HuggingFace cache directory path",{"type":45,"tag":81,"props":1139,"children":1140},{},[1141,1146,1148,1154],{"type":45,"tag":87,"props":1142,"children":1144},{"className":1143},[],[1145],{"type":51,"value":160},{"type":51,"value":1147},": list of ",{"type":45,"tag":87,"props":1149,"children":1151},{"className":1150},[],[1152],{"type":51,"value":1153},"VolumeMapping(source, dest)",{"type":51,"value":1155}," for additional container bind mounts",{"type":45,"tag":81,"props":1157,"children":1158},{},[1159,1164],{"type":45,"tag":87,"props":1160,"children":1162},{"className":1161},[],[1163],{"type":51,"value":175},{"type":51,"value":1165},": port for distributed communication (default 13742)",{"type":45,"tag":81,"props":1167,"children":1168},{},[1169,1174],{"type":45,"tag":87,"props":1170,"children":1172},{"className":1171},[],[1173],{"type":51,"value":167},{"type":51,"value":1175},": environment variables passed into the job",{"type":45,"tag":81,"props":1177,"children":1178},{},[1179,1185,1187,1192],{"type":45,"tag":87,"props":1180,"children":1182},{"className":1181},[],[1183],{"type":51,"value":1184},"nsys_enabled",{"type":51,"value":1186},": when true, wraps the training command with ",{"type":45,"tag":87,"props":1188,"children":1190},{"className":1189},[],[1191],{"type":51,"value":303},{"type":51,"value":1193}," for Nsight Systems profiling",{"type":45,"tag":60,"props":1195,"children":1197},{"id":1196},"skypilot-configuration",[1198],{"type":51,"value":1199},"SkyPilot Configuration",{"type":45,"tag":54,"props":1201,"children":1202},{},[1203,1204,1210],{"type":51,"value":768},{"type":45,"tag":87,"props":1205,"children":1207},{"className":1206},[],[1208],{"type":51,"value":1209},"SkyPilotConfig",{"type":51,"value":1211}," dataclass defines cloud job parameters.",{"type":45,"tag":778,"props":1213,"children":1215},{"id":1214},"yaml-example-1",[1216],{"type":51,"value":783},{"type":45,"tag":320,"props":1218,"children":1220},{"className":322,"code":1219,"language":324,"meta":325,"style":325},"skypilot:\n  cloud: aws\n  accelerators: \"H100:8\"\n  num_nodes: 2\n  use_spot: true\n  disk_size: 200\n  region: us-east-1\n  setup: \"pip install nemo-automodel\"\n  env_vars:\n    HF_TOKEN: \"${HF_TOKEN}\"\n",[1221],{"type":45,"tag":87,"props":1222,"children":1223},{"__ignoreMap":325},[1224,1236,1253,1278,1294,1312,1329,1346,1371,1382],{"type":45,"tag":331,"props":1225,"children":1226},{"class":333,"line":334},[1227,1232],{"type":45,"tag":331,"props":1228,"children":1229},{"style":338},[1230],{"type":51,"value":1231},"skypilot",{"type":45,"tag":331,"props":1233,"children":1234},{"style":344},[1235],{"type":51,"value":347},{"type":45,"tag":331,"props":1237,"children":1238},{"class":333,"line":350},[1239,1244,1248],{"type":45,"tag":331,"props":1240,"children":1241},{"style":338},[1242],{"type":51,"value":1243},"  cloud",{"type":45,"tag":331,"props":1245,"children":1246},{"style":344},[1247],{"type":51,"value":361},{"type":45,"tag":331,"props":1249,"children":1250},{"style":364},[1251],{"type":51,"value":1252}," aws\n",{"type":45,"tag":331,"props":1254,"children":1255},{"class":333,"line":370},[1256,1261,1265,1269,1274],{"type":45,"tag":331,"props":1257,"children":1258},{"style":338},[1259],{"type":51,"value":1260},"  accelerators",{"type":45,"tag":331,"props":1262,"children":1263},{"style":344},[1264],{"type":51,"value":361},{"type":45,"tag":331,"props":1266,"children":1267},{"style":344},[1268],{"type":51,"value":422},{"type":45,"tag":331,"props":1270,"children":1271},{"style":364},[1272],{"type":51,"value":1273},"H100:8",{"type":45,"tag":331,"props":1275,"children":1276},{"style":344},[1277],{"type":51,"value":432},{"type":45,"tag":331,"props":1279,"children":1280},{"class":333,"line":389},[1281,1286,1290],{"type":45,"tag":331,"props":1282,"children":1283},{"style":338},[1284],{"type":51,"value":1285},"  num_nodes",{"type":45,"tag":331,"props":1287,"children":1288},{"style":344},[1289],{"type":51,"value":361},{"type":45,"tag":331,"props":1291,"children":1292},{"style":383},[1293],{"type":51,"value":386},{"type":45,"tag":331,"props":1295,"children":1296},{"class":333,"line":407},[1297,1302,1306],{"type":45,"tag":331,"props":1298,"children":1299},{"style":338},[1300],{"type":51,"value":1301},"  use_spot",{"type":45,"tag":331,"props":1303,"children":1304},{"style":344},[1305],{"type":51,"value":361},{"type":45,"tag":331,"props":1307,"children":1309},{"style":1308},"--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC",[1310],{"type":51,"value":1311}," true\n",{"type":45,"tag":331,"props":1313,"children":1314},{"class":333,"line":435},[1315,1320,1324],{"type":45,"tag":331,"props":1316,"children":1317},{"style":338},[1318],{"type":51,"value":1319},"  disk_size",{"type":45,"tag":331,"props":1321,"children":1322},{"style":344},[1323],{"type":51,"value":361},{"type":45,"tag":331,"props":1325,"children":1326},{"style":383},[1327],{"type":51,"value":1328}," 200\n",{"type":45,"tag":331,"props":1330,"children":1331},{"class":333,"line":453},[1332,1337,1341],{"type":45,"tag":331,"props":1333,"children":1334},{"style":338},[1335],{"type":51,"value":1336},"  region",{"type":45,"tag":331,"props":1338,"children":1339},{"style":344},[1340],{"type":51,"value":361},{"type":45,"tag":331,"props":1342,"children":1343},{"style":364},[1344],{"type":51,"value":1345}," us-east-1\n",{"type":45,"tag":331,"props":1347,"children":1348},{"class":333,"line":471},[1349,1354,1358,1362,1367],{"type":45,"tag":331,"props":1350,"children":1351},{"style":338},[1352],{"type":51,"value":1353},"  setup",{"type":45,"tag":331,"props":1355,"children":1356},{"style":344},[1357],{"type":51,"value":361},{"type":45,"tag":331,"props":1359,"children":1360},{"style":344},[1361],{"type":51,"value":422},{"type":45,"tag":331,"props":1363,"children":1364},{"style":364},[1365],{"type":51,"value":1366},"pip install nemo-automodel",{"type":45,"tag":331,"props":1368,"children":1369},{"style":344},[1370],{"type":51,"value":432},{"type":45,"tag":331,"props":1372,"children":1373},{"class":333,"line":489},[1374,1378],{"type":45,"tag":331,"props":1375,"children":1376},{"style":338},[1377],{"type":51,"value":531},{"type":45,"tag":331,"props":1379,"children":1380},{"style":344},[1381],{"type":51,"value":347},{"type":45,"tag":331,"props":1383,"children":1384},{"class":333,"line":507},[1385,1389,1393,1397,1401],{"type":45,"tag":331,"props":1386,"children":1387},{"style":338},[1388],{"type":51,"value":544},{"type":45,"tag":331,"props":1390,"children":1391},{"style":344},[1392],{"type":51,"value":361},{"type":45,"tag":331,"props":1394,"children":1395},{"style":344},[1396],{"type":51,"value":422},{"type":45,"tag":331,"props":1398,"children":1399},{"style":364},[1400],{"type":51,"value":557},{"type":45,"tag":331,"props":1402,"children":1403},{"style":344},[1404],{"type":51,"value":432},{"type":45,"tag":778,"props":1406,"children":1408},{"id":1407},"key-fields-1",[1409],{"type":51,"value":1047},{"type":45,"tag":77,"props":1411,"children":1412},{},[1413,1459,1483,1493,1504,1514,1524,1534],{"type":45,"tag":81,"props":1414,"children":1415},{},[1416,1421,1423,1429,1430,1436,1437,1443,1444,1450,1451,1457],{"type":45,"tag":87,"props":1417,"children":1419},{"className":1418},[],[1420],{"type":51,"value":22},{"type":51,"value":1422},": target cloud provider (",{"type":45,"tag":87,"props":1424,"children":1426},{"className":1425},[],[1427],{"type":51,"value":1428},"aws",{"type":51,"value":102},{"type":45,"tag":87,"props":1431,"children":1433},{"className":1432},[],[1434],{"type":51,"value":1435},"gcp",{"type":51,"value":102},{"type":45,"tag":87,"props":1438,"children":1440},{"className":1439},[],[1441],{"type":51,"value":1442},"azure",{"type":51,"value":102},{"type":45,"tag":87,"props":1445,"children":1447},{"className":1446},[],[1448],{"type":51,"value":1449},"lambda",{"type":51,"value":102},{"type":45,"tag":87,"props":1452,"children":1454},{"className":1453},[],[1455],{"type":51,"value":1456},"kubernetes",{"type":51,"value":1458},")",{"type":45,"tag":81,"props":1460,"children":1461},{},[1462,1467,1469,1475,1476,1482],{"type":45,"tag":87,"props":1463,"children":1465},{"className":1464},[],[1466],{"type":51,"value":225},{"type":51,"value":1468},": GPU type and count (e.g., ",{"type":45,"tag":87,"props":1470,"children":1472},{"className":1471},[],[1473],{"type":51,"value":1474},"\"H100:8\"",{"type":51,"value":102},{"type":45,"tag":87,"props":1477,"children":1479},{"className":1478},[],[1480],{"type":51,"value":1481},"\"A100-80GB:4\"",{"type":51,"value":1458},{"type":45,"tag":81,"props":1484,"children":1485},{},[1486,1491],{"type":45,"tag":87,"props":1487,"children":1489},{"className":1488},[],[1490],{"type":51,"value":232},{"type":51,"value":1492},": number of cloud instances",{"type":45,"tag":81,"props":1494,"children":1495},{},[1496,1502],{"type":45,"tag":87,"props":1497,"children":1499},{"className":1498},[],[1500],{"type":51,"value":1501},"use_spot",{"type":51,"value":1503},": use preemptible\u002Fspot instances for cost savings",{"type":45,"tag":81,"props":1505,"children":1506},{},[1507,1512],{"type":45,"tag":87,"props":1508,"children":1510},{"className":1509},[],[1511],{"type":51,"value":246},{"type":51,"value":1513},": disk size in GB per node",{"type":45,"tag":81,"props":1515,"children":1516},{},[1517,1522],{"type":45,"tag":87,"props":1518,"children":1520},{"className":1519},[],[1521],{"type":51,"value":253},{"type":51,"value":1523},": cloud region for instance placement",{"type":45,"tag":81,"props":1525,"children":1526},{},[1527,1532],{"type":45,"tag":87,"props":1528,"children":1530},{"className":1529},[],[1531],{"type":51,"value":260},{"type":51,"value":1533},": shell commands to run before the training job (e.g., install dependencies)",{"type":45,"tag":81,"props":1535,"children":1536},{},[1537,1542],{"type":45,"tag":87,"props":1538,"children":1540},{"className":1539},[],[1541],{"type":51,"value":167},{"type":51,"value":1543},": environment variables for the job",{"type":45,"tag":778,"props":1545,"children":1547},{"id":1546},"skypilot-spot-checklist",[1548],{"type":51,"value":1549},"SkyPilot spot checklist",{"type":45,"tag":54,"props":1551,"children":1552},{},[1553],{"type":51,"value":1554},"When using spot or preemptible instances:",{"type":45,"tag":77,"props":1556,"children":1557},{},[1558,1577,1619,1631],{"type":45,"tag":81,"props":1559,"children":1560},{},[1561,1563,1568,1570,1575],{"type":51,"value":1562},"Set ",{"type":45,"tag":87,"props":1564,"children":1566},{"className":1565},[],[1567],{"type":51,"value":239},{"type":51,"value":1569}," in the ",{"type":45,"tag":87,"props":1571,"children":1573},{"className":1572},[],[1574],{"type":51,"value":212},{"type":51,"value":1576}," section.",{"type":45,"tag":81,"props":1578,"children":1579},{},[1580,1582,1587,1588,1593,1594,1599,1600,1605,1606,1611,1613,1618],{"type":51,"value":1581},"Include ",{"type":45,"tag":87,"props":1583,"children":1585},{"className":1584},[],[1586],{"type":51,"value":225},{"type":51,"value":102},{"type":45,"tag":87,"props":1589,"children":1591},{"className":1590},[],[1592],{"type":51,"value":232},{"type":51,"value":102},{"type":45,"tag":87,"props":1595,"children":1597},{"className":1596},[],[1598],{"type":51,"value":246},{"type":51,"value":102},{"type":45,"tag":87,"props":1601,"children":1603},{"className":1602},[],[1604],{"type":51,"value":253},{"type":51,"value":102},{"type":45,"tag":87,"props":1607,"children":1609},{"className":1608},[],[1610],{"type":51,"value":260},{"type":51,"value":1612},", and required ",{"type":45,"tag":87,"props":1614,"children":1616},{"className":1615},[],[1617],{"type":51,"value":167},{"type":51,"value":201},{"type":45,"tag":81,"props":1620,"children":1621},{},[1622,1624,1629],{"type":51,"value":1623},"Use short checkpoint intervals in the recipe, for example ",{"type":45,"tag":87,"props":1625,"children":1627},{"className":1626},[],[1628],{"type":51,"value":275},{"type":51,"value":1630},", because spot instances can be preempted.",{"type":45,"tag":81,"props":1632,"children":1633},{},[1634,1636,1642],{"type":51,"value":1635},"Resume from the most recent checkpoint after preemption with the recipe's ",{"type":45,"tag":87,"props":1637,"children":1639},{"className":1638},[],[1640],{"type":51,"value":1641},"restore_from",{"type":51,"value":1643}," setting.",{"type":45,"tag":54,"props":1645,"children":1646},{},[1647],{"type":51,"value":1648},"Minimal spot-resume recipe keys:",{"type":45,"tag":320,"props":1650,"children":1652},{"className":322,"code":1651,"language":324,"meta":325,"style":325},"step_scheduler:\n  checkpoint_interval: 100\n\nrestore_from:\n  path: \u002Fcheckpoints\u002Flatest\n",[1653],{"type":45,"tag":87,"props":1654,"children":1655},{"__ignoreMap":325},[1656,1668,1685,1692,1703],{"type":45,"tag":331,"props":1657,"children":1658},{"class":333,"line":334},[1659,1664],{"type":45,"tag":331,"props":1660,"children":1661},{"style":338},[1662],{"type":51,"value":1663},"step_scheduler",{"type":45,"tag":331,"props":1665,"children":1666},{"style":344},[1667],{"type":51,"value":347},{"type":45,"tag":331,"props":1669,"children":1670},{"class":333,"line":350},[1671,1676,1680],{"type":45,"tag":331,"props":1672,"children":1673},{"style":338},[1674],{"type":51,"value":1675},"  checkpoint_interval",{"type":45,"tag":331,"props":1677,"children":1678},{"style":344},[1679],{"type":51,"value":361},{"type":45,"tag":331,"props":1681,"children":1682},{"style":383},[1683],{"type":51,"value":1684}," 100\n",{"type":45,"tag":331,"props":1686,"children":1687},{"class":333,"line":370},[1688],{"type":45,"tag":331,"props":1689,"children":1690},{"emptyLinePlaceholder":689},[1691],{"type":51,"value":692},{"type":45,"tag":331,"props":1693,"children":1694},{"class":333,"line":389},[1695,1699],{"type":45,"tag":331,"props":1696,"children":1697},{"style":338},[1698],{"type":51,"value":1641},{"type":45,"tag":331,"props":1700,"children":1701},{"style":344},[1702],{"type":51,"value":347},{"type":45,"tag":331,"props":1704,"children":1705},{"class":333,"line":407},[1706,1711,1715],{"type":45,"tag":331,"props":1707,"children":1708},{"style":338},[1709],{"type":51,"value":1710},"  path",{"type":45,"tag":331,"props":1712,"children":1713},{"style":344},[1714],{"type":51,"value":361},{"type":45,"tag":331,"props":1716,"children":1717},{"style":364},[1718],{"type":51,"value":1719}," \u002Fcheckpoints\u002Flatest\n",{"type":45,"tag":60,"props":1721,"children":1723},{"id":1722},"multi-node-environment",[1724],{"type":51,"value":1725},"Multi-Node Environment",{"type":45,"tag":54,"props":1727,"children":1728},{},[1729],{"type":51,"value":1730},"For multi-node training (both Slurm and SkyPilot), the launcher automatically configures:",{"type":45,"tag":77,"props":1732,"children":1733},{},[1734,1744,1754,1772],{"type":45,"tag":81,"props":1735,"children":1736},{},[1737,1742],{"type":45,"tag":87,"props":1738,"children":1740},{"className":1739},[],[1741],{"type":51,"value":191},{"type":51,"value":1743},": hostname of the first node",{"type":45,"tag":81,"props":1745,"children":1746},{},[1747,1752],{"type":45,"tag":87,"props":1748,"children":1750},{"className":1749},[],[1751],{"type":51,"value":199},{"type":51,"value":1753},": port for rendezvous (default 13742)",{"type":45,"tag":81,"props":1755,"children":1756},{},[1757,1763,1765,1771],{"type":45,"tag":87,"props":1758,"children":1760},{"className":1759},[],[1761],{"type":51,"value":1762},"WORLD_SIZE",{"type":51,"value":1764},": total number of processes (",{"type":45,"tag":87,"props":1766,"children":1768},{"className":1767},[],[1769],{"type":51,"value":1770},"nodes * ntasks_per_node",{"type":51,"value":1458},{"type":45,"tag":81,"props":1773,"children":1774},{},[1775],{"type":51,"value":1776},"NCCL environment variables for optimized collective communication",{"type":45,"tag":60,"props":1778,"children":1780},{"id":1779},"nsys-profiling",[1781],{"type":51,"value":1782},"Nsys Profiling",{"type":45,"tag":54,"props":1784,"children":1785},{},[1786],{"type":51,"value":1787},"Enable Nsight Systems profiling in Slurm jobs:",{"type":45,"tag":320,"props":1789,"children":1791},{"className":322,"code":1790,"language":324,"meta":325,"style":325},"slurm:\n  job_name: llm_profile\n  nodes: 1\n  ntasks_per_node: 8\n  time: \"00:30:00\"\n  account: my_account\n  partition: batch\n  container_image: nvcr.io\u002Fnvidia\u002Fnemo:dev\n  nsys_enabled: true\n",[1792],{"type":45,"tag":87,"props":1793,"children":1794},{"__ignoreMap":325},[1795,1806,1822,1838,1853,1877,1892,1907,1922],{"type":45,"tag":331,"props":1796,"children":1797},{"class":333,"line":334},[1798,1802],{"type":45,"tag":331,"props":1799,"children":1800},{"style":338},[1801],{"type":51,"value":341},{"type":45,"tag":331,"props":1803,"children":1804},{"style":344},[1805],{"type":51,"value":347},{"type":45,"tag":331,"props":1807,"children":1808},{"class":333,"line":350},[1809,1813,1817],{"type":45,"tag":331,"props":1810,"children":1811},{"style":338},[1812],{"type":51,"value":356},{"type":45,"tag":331,"props":1814,"children":1815},{"style":344},[1816],{"type":51,"value":361},{"type":45,"tag":331,"props":1818,"children":1819},{"style":364},[1820],{"type":51,"value":1821}," llm_profile\n",{"type":45,"tag":331,"props":1823,"children":1824},{"class":333,"line":370},[1825,1829,1833],{"type":45,"tag":331,"props":1826,"children":1827},{"style":338},[1828],{"type":51,"value":376},{"type":45,"tag":331,"props":1830,"children":1831},{"style":344},[1832],{"type":51,"value":361},{"type":45,"tag":331,"props":1834,"children":1835},{"style":383},[1836],{"type":51,"value":1837}," 1\n",{"type":45,"tag":331,"props":1839,"children":1840},{"class":333,"line":389},[1841,1845,1849],{"type":45,"tag":331,"props":1842,"children":1843},{"style":338},[1844],{"type":51,"value":395},{"type":45,"tag":331,"props":1846,"children":1847},{"style":344},[1848],{"type":51,"value":361},{"type":45,"tag":331,"props":1850,"children":1851},{"style":383},[1852],{"type":51,"value":404},{"type":45,"tag":331,"props":1854,"children":1855},{"class":333,"line":407},[1856,1860,1864,1868,1873],{"type":45,"tag":331,"props":1857,"children":1858},{"style":338},[1859],{"type":51,"value":413},{"type":45,"tag":331,"props":1861,"children":1862},{"style":344},[1863],{"type":51,"value":361},{"type":45,"tag":331,"props":1865,"children":1866},{"style":344},[1867],{"type":51,"value":422},{"type":45,"tag":331,"props":1869,"children":1870},{"style":364},[1871],{"type":51,"value":1872},"00:30:00",{"type":45,"tag":331,"props":1874,"children":1875},{"style":344},[1876],{"type":51,"value":432},{"type":45,"tag":331,"props":1878,"children":1879},{"class":333,"line":435},[1880,1884,1888],{"type":45,"tag":331,"props":1881,"children":1882},{"style":338},[1883],{"type":51,"value":441},{"type":45,"tag":331,"props":1885,"children":1886},{"style":344},[1887],{"type":51,"value":361},{"type":45,"tag":331,"props":1889,"children":1890},{"style":364},[1891],{"type":51,"value":450},{"type":45,"tag":331,"props":1893,"children":1894},{"class":333,"line":453},[1895,1899,1903],{"type":45,"tag":331,"props":1896,"children":1897},{"style":338},[1898],{"type":51,"value":459},{"type":45,"tag":331,"props":1900,"children":1901},{"style":344},[1902],{"type":51,"value":361},{"type":45,"tag":331,"props":1904,"children":1905},{"style":364},[1906],{"type":51,"value":468},{"type":45,"tag":331,"props":1908,"children":1909},{"class":333,"line":471},[1910,1914,1918],{"type":45,"tag":331,"props":1911,"children":1912},{"style":338},[1913],{"type":51,"value":477},{"type":45,"tag":331,"props":1915,"children":1916},{"style":344},[1917],{"type":51,"value":361},{"type":45,"tag":331,"props":1919,"children":1920},{"style":364},[1921],{"type":51,"value":486},{"type":45,"tag":331,"props":1923,"children":1924},{"class":333,"line":489},[1925,1930,1934],{"type":45,"tag":331,"props":1926,"children":1927},{"style":338},[1928],{"type":51,"value":1929},"  nsys_enabled",{"type":45,"tag":331,"props":1931,"children":1932},{"style":344},[1933],{"type":51,"value":361},{"type":45,"tag":331,"props":1935,"children":1936},{"style":1308},[1937],{"type":51,"value":1311},{"type":45,"tag":54,"props":1939,"children":1940},{},[1941,1943,1948,1949,1954,1955,1960,1961,1966,1967,1972,1973,1978,1979,1984],{"type":51,"value":1942},"This is a Slurm launcher setting. Normal Slurm fields such as ",{"type":45,"tag":87,"props":1944,"children":1946},{"className":1945},[],[1947],{"type":51,"value":100},{"type":51,"value":110},{"type":45,"tag":87,"props":1950,"children":1952},{"className":1951},[],[1953],{"type":51,"value":108},{"type":51,"value":102},{"type":45,"tag":87,"props":1956,"children":1958},{"className":1957},[],[1959],{"type":51,"value":116},{"type":51,"value":102},{"type":45,"tag":87,"props":1962,"children":1964},{"className":1963},[],[1965],{"type":51,"value":123},{"type":51,"value":102},{"type":45,"tag":87,"props":1968,"children":1970},{"className":1969},[],[1971],{"type":51,"value":130},{"type":51,"value":132},{"type":45,"tag":87,"props":1974,"children":1976},{"className":1975},[],[1977],{"type":51,"value":138},{"type":51,"value":262},{"type":45,"tag":87,"props":1980,"children":1982},{"className":1981},[],[1983],{"type":51,"value":145},{"type":51,"value":1985}," still apply.",{"type":45,"tag":54,"props":1987,"children":1988},{},[1989,1991,1997,1999,2004,2006,2011],{"type":51,"value":1990},"When ",{"type":45,"tag":87,"props":1992,"children":1994},{"className":1993},[],[1995],{"type":51,"value":1996},"nsys_enabled: true",{"type":51,"value":1998},", the launcher wraps the training command with\n",{"type":45,"tag":87,"props":2000,"children":2002},{"className":2001},[],[2003],{"type":51,"value":303},{"type":51,"value":2005}," and writes a ",{"type":45,"tag":87,"props":2007,"children":2009},{"className":2008},[],[2010],{"type":51,"value":311},{"type":51,"value":2012}," report file for performance analysis\nin the Slurm job working or output directory.\nProfiling is diagnostic-only: run it for a short investigation, expect overhead\nand large artifacts, and turn it off for normal production training.",{"type":45,"tag":60,"props":2014,"children":2016},{"id":2015},"code-anchors",[2017],{"type":51,"value":2018},"Code Anchors",{"type":45,"tag":77,"props":2020,"children":2021},{},[2022,2033,2044,2055,2066],{"type":45,"tag":81,"props":2023,"children":2024},{},[2025,2031],{"type":45,"tag":87,"props":2026,"children":2028},{"className":2027},[],[2029],{"type":51,"value":2030},"components\u002Flauncher\u002Fslurm\u002Fconfig.py",{"type":51,"value":2032}," - SlurmConfig dataclass, VolumeMapping",{"type":45,"tag":81,"props":2034,"children":2035},{},[2036,2042],{"type":45,"tag":87,"props":2037,"children":2039},{"className":2038},[],[2040],{"type":51,"value":2041},"components\u002Flauncher\u002Fslurm\u002Ftemplate.py",{"type":51,"value":2043}," - SBATCH script template generation",{"type":45,"tag":81,"props":2045,"children":2046},{},[2047,2053],{"type":45,"tag":87,"props":2048,"children":2050},{"className":2049},[],[2051],{"type":51,"value":2052},"components\u002Flauncher\u002Fslurm\u002Futils.py",{"type":51,"value":2054}," - Slurm submission utilities",{"type":45,"tag":81,"props":2056,"children":2057},{},[2058,2064],{"type":45,"tag":87,"props":2059,"children":2061},{"className":2060},[],[2062],{"type":51,"value":2063},"components\u002Flauncher\u002Fskypilot\u002Fconfig.py",{"type":51,"value":2065}," - SkyPilotConfig dataclass",{"type":45,"tag":81,"props":2067,"children":2068},{},[2069,2075],{"type":45,"tag":87,"props":2070,"children":2072},{"className":2071},[],[2073],{"type":51,"value":2074},"_cli\u002Fapp.py",{"type":51,"value":2076}," - CLI entry point and launcher routing logic",{"type":45,"tag":60,"props":2078,"children":2080},{"id":2079},"pitfalls",[2081],{"type":51,"value":2082},"Pitfalls",{"type":45,"tag":77,"props":2084,"children":2085},{},[2086,2103,2128,2138,2155,2173],{"type":45,"tag":81,"props":2087,"children":2088},{},[2089,2094,2096,2101],{"type":45,"tag":604,"props":2090,"children":2091},{},[2092],{"type":51,"value":2093},"Port collisions",{"type":51,"value":2095},": if the default ",{"type":45,"tag":87,"props":2097,"children":2099},{"className":2098},[],[2100],{"type":51,"value":175},{"type":51,"value":2102}," (13742) is in use by another job on the same node, change it to avoid connection failures.",{"type":45,"tag":81,"props":2104,"children":2105},{},[2106,2111,2113,2119,2121,2126],{"type":45,"tag":604,"props":2107,"children":2108},{},[2109],{"type":51,"value":2110},"Container mounts",{"type":51,"value":2112},": the ",{"type":45,"tag":87,"props":2114,"children":2116},{"className":2115},[],[2117],{"type":51,"value":2118},"source",{"type":51,"value":2120}," path in ",{"type":45,"tag":87,"props":2122,"children":2124},{"className":2123},[],[2125],{"type":51,"value":160},{"type":51,"value":2127}," must exist on all nodes in the allocation. Missing paths cause container startup failures.",{"type":45,"tag":81,"props":2129,"children":2130},{},[2131,2136],{"type":45,"tag":604,"props":2132,"children":2133},{},[2134],{"type":51,"value":2135},"Slurm fault tolerance",{"type":51,"value":2137},": the fault tolerance plugin is Slurm-specific and does not work with SkyPilot or interactive mode.",{"type":45,"tag":81,"props":2139,"children":2140},{},[2141,2146,2148,2153],{"type":45,"tag":604,"props":2142,"children":2143},{},[2144],{"type":51,"value":2145},"SkyPilot spot preemption",{"type":51,"value":2147},": spot instances (",{"type":45,"tag":87,"props":2149,"children":2151},{"className":2150},[],[2152],{"type":51,"value":239},{"type":51,"value":2154},") may be preempted by the cloud provider. Enable checkpointing with short intervals to minimize lost work.",{"type":45,"tag":81,"props":2156,"children":2157},{},[2158,2163,2165,2171],{"type":45,"tag":604,"props":2159,"children":2160},{},[2161],{"type":51,"value":2162},"Environment variable syntax",{"type":51,"value":2164},": use ",{"type":45,"tag":87,"props":2166,"children":2168},{"className":2167},[],[2169],{"type":51,"value":2170},"${VAR}",{"type":51,"value":2172}," syntax in YAML for shell variable expansion. Bare variable names will not be expanded.",{"type":45,"tag":81,"props":2174,"children":2175},{},[2176,2181,2183,2188],{"type":45,"tag":604,"props":2177,"children":2178},{},[2179],{"type":51,"value":2180},"Time limit vs async checkpoint",{"type":51,"value":2182},": if the Slurm ",{"type":45,"tag":87,"props":2184,"children":2186},{"className":2185},[],[2187],{"type":51,"value":123},{"type":51,"value":2189}," limit is too short, an in-progress async checkpoint write may be killed before completion, resulting in a corrupted checkpoint. Leave at least 5-10 minutes of margin.",{"type":45,"tag":2191,"props":2192,"children":2193},"style",{},[2194],{"type":51,"value":2195},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2197,"total":2352},[2198,2216,2234,2245,2257,2271,2284,2296,2309,2320,2334,2343],{"slug":2199,"name":2199,"fn":2200,"description":2201,"org":2202,"tags":2203,"stars":2213,"repoUrl":2214,"updatedAt":2215},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2204,2207,2210],{"name":2205,"slug":2206,"type":15},"Documentation","documentation",{"name":2208,"slug":2209,"type":15},"MCP","mcp",{"name":2211,"slug":2212,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2217,"name":2217,"fn":2218,"description":2219,"org":2220,"tags":2221,"stars":2231,"repoUrl":2232,"updatedAt":2233},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2222,2225,2228],{"name":2223,"slug":2224,"type":15},"Containers","containers",{"name":2226,"slug":2227,"type":15},"Deployment","deployment",{"name":2229,"slug":2230,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2235,"name":2235,"fn":2236,"description":2237,"org":2238,"tags":2239,"stars":2231,"repoUrl":2232,"updatedAt":2244},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2240,2243],{"name":2241,"slug":2242,"type":15},"CI\u002FCD","ci-cd",{"name":2226,"slug":2227,"type":15},"2026-07-14T05:25:59.97109",{"slug":2246,"name":2246,"fn":2247,"description":2248,"org":2249,"tags":2250,"stars":2231,"repoUrl":2232,"updatedAt":2256},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2251,2252,2253],{"name":2241,"slug":2242,"type":15},{"name":2226,"slug":2227,"type":15},{"name":2254,"slug":2255,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2258,"name":2258,"fn":2259,"description":2260,"org":2261,"tags":2262,"stars":2231,"repoUrl":2232,"updatedAt":2270},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2263,2266,2267],{"name":2264,"slug":2265,"type":15},"Debugging","debugging",{"name":2254,"slug":2255,"type":15},{"name":2268,"slug":2269,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2272,"name":2272,"fn":2273,"description":2274,"org":2275,"tags":2276,"stars":2231,"repoUrl":2232,"updatedAt":2283},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2277,2280],{"name":2278,"slug":2279,"type":15},"Best Practices","best-practices",{"name":2281,"slug":2282,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2285,"name":2285,"fn":2286,"description":2287,"org":2288,"tags":2289,"stars":2231,"repoUrl":2232,"updatedAt":2295},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2290,2291,2294],{"name":17,"slug":18,"type":15},{"name":2292,"slug":2293,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2297,"name":2297,"fn":2298,"description":2299,"org":2300,"tags":2301,"stars":2231,"repoUrl":2232,"updatedAt":2308},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2302,2305],{"name":2303,"slug":2304,"type":15},"QA","qa",{"name":2306,"slug":2307,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":2310,"name":2310,"fn":2311,"description":2312,"org":2313,"tags":2314,"stars":2231,"repoUrl":2232,"updatedAt":2319},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2315,2316],{"name":2226,"slug":2227,"type":15},{"name":2317,"slug":2318,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":2321,"name":2321,"fn":2322,"description":2323,"org":2324,"tags":2325,"stars":2231,"repoUrl":2232,"updatedAt":2333},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2326,2329,2330],{"name":2327,"slug":2328,"type":15},"Code Review","code-review",{"name":2254,"slug":2255,"type":15},{"name":2331,"slug":2332,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2335,"name":2335,"fn":2336,"description":2337,"org":2338,"tags":2339,"stars":2231,"repoUrl":2232,"updatedAt":2342},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2340,2341],{"name":2303,"slug":2304,"type":15},{"name":2306,"slug":2307,"type":15},"2026-07-14T05:25:54.928983",{"slug":2344,"name":2344,"fn":2345,"description":2346,"org":2347,"tags":2348,"stars":2231,"repoUrl":2232,"updatedAt":2351},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2349,2350],{"name":13,"slug":14,"type":15},{"name":2241,"slug":2242,"type":15},"2026-07-30T05:29:03.275638",496,{"items":2354,"total":2450},[2355,2372,2382,2396,2406,2421,2436],{"slug":2356,"name":2356,"fn":2357,"description":2358,"org":2359,"tags":2360,"stars":23,"repoUrl":24,"updatedAt":2371},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2361,2364,2367,2368],{"name":2362,"slug":2363,"type":15},"Data Analysis","data-analysis",{"name":2365,"slug":2366,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":2369,"slug":2370,"type":15},"Performance","performance","2026-07-14T05:28:43.176466",{"slug":2373,"name":2373,"fn":2374,"description":2375,"org":2376,"tags":2377,"stars":23,"repoUrl":24,"updatedAt":2381},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2378,2379,2380],{"name":2226,"slug":2227,"type":15},{"name":2317,"slug":2318,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":2383,"name":2383,"fn":2384,"description":2385,"org":2386,"tags":2387,"stars":23,"repoUrl":24,"updatedAt":2395},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2388,2391,2392],{"name":2389,"slug":2390,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":2393,"slug":2394,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":2397,"name":2397,"fn":2398,"description":2399,"org":2400,"tags":2401,"stars":23,"repoUrl":24,"updatedAt":2405},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2402,2403,2404],{"name":2362,"slug":2363,"type":15},{"name":9,"slug":8,"type":15},{"name":2306,"slug":2307,"type":15},"2026-07-17T05:29:03.913266",{"slug":2407,"name":2407,"fn":2408,"description":2409,"org":2410,"tags":2411,"stars":23,"repoUrl":24,"updatedAt":2420},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2412,2413,2416,2417],{"name":13,"slug":14,"type":15},{"name":2414,"slug":2415,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":2418,"slug":2419,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":2422,"name":2422,"fn":2423,"description":2424,"org":2425,"tags":2426,"stars":23,"repoUrl":24,"updatedAt":2435},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2427,2428,2431,2432],{"name":2226,"slug":2227,"type":15},{"name":2429,"slug":2430,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":2433,"slug":2434,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":2437,"name":2437,"fn":2438,"description":2439,"org":2440,"tags":2441,"stars":23,"repoUrl":24,"updatedAt":2449},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2442,2443,2446],{"name":9,"slug":8,"type":15},{"name":2444,"slug":2445,"type":15},"Quantum Computing","quantum-computing",{"name":2447,"slug":2448,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305]