[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-tao-run-on-slurm":3,"mdc-ioaatb-key":34,"related-org-nvidia-tao-run-on-slurm":1405,"related-repo-nvidia-tao-run-on-slurm":1562},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"tao-run-on-slurm","run TAO training jobs on SLURM clusters","Remote SLURM GPU cluster execution over SSH with sbatch\u002Fsrun, Pyxis\u002FEnroot containers, and Lustre-backed results. Use when running TAO training\u002Feval\u002Finference jobs on an on-prem or DGX SLURM cluster. Trigger phrases include \"run on SLURM\", \"submit sbatch\", \"DGX SLURM cluster\", \"Pyxis\u002FEnroot container\", \"Lustre dataset\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Containers","containers","tag",{"name":17,"slug":18,"type":15},"Deployment","deployment",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Engineering","engineering",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:28:33.280994","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Ftao-run-on-slurm","---\nname: tao-run-on-slurm\ndescription: Remote SLURM GPU cluster execution over SSH with sbatch\u002Fsrun, Pyxis\u002FEnroot containers, and Lustre-backed\n  results. Use when running TAO training\u002Feval\u002Finference jobs on an on-prem or DGX SLURM cluster. Trigger phrases include\n  \"run on SLURM\", \"submit sbatch\", \"DGX SLURM cluster\", \"Pyxis\u002FEnroot container\", \"Lustre dataset\".\nlicense: Apache-2.0\ncompatibility: Requires SSH access to a SLURM login node (passwordless via key auth) and SLURM_USER + SLURM_HOSTNAME env vars.\n  The TAO SDK with the slurm extra (pip install 'nvidia-tao-sdk[slurm]') is needed only if you want Job handles, S3 I\u002FO wrapping,\n  or run-folder durability via ActionWorkflow.\nmetadata:\n  author: NVIDIA Corporation\n  version: \"0.1.0\"\nallowed-tools: Read Bash\ntags:\n- platform\n- slurm\n---\n\n# SLURM\n\nRemote GPU compute platform for clusters managed by SLURM. Jobs are submitted\nfrom the TAO service or SDK host to a login node over SSH, staged on a shared\nfilesystem, submitted with `sbatch`, and executed with `srun` container support.\n\n## When to use\n\nUse SLURM when the user has access to a managed GPU cluster, shared Lustre\nstorage, and scheduler-owned GPU allocation. Do not use SLURM for local files\nthat exist only on the agent machine; data and outputs must be reachable from\nthe cluster.\n\n## Preflight + SSH\n\nConfirm `SLURM_USER` and `SLURM_HOSTNAME` are exported and passwordless SSH to a\nlogin host works (`ssh -o BatchMode=yes`). Optionally install the TAO SDK\nwrapper for Job handles + S3 wrapping (`nvidia-tao-sdk[slurm]`, on public PyPI).\nFor private `nvcr.io` images, install `~\u002F.config\u002Fenroot\u002F.credentials` on the\ncluster once per (cluster, user): Pyxis\u002FEnroot does not read `NGC_KEY` from the\njob env, and without persistent credentials, auth-gated pulls fail with \"Could\nnot process JSON input\" at job startup. Install it via the `printf | ssh`\nheredoc so the `NGC_KEY` value never lands in shell history, intermediate files,\nor chat output; never `cat`\u002F`echo` the value.\n\nIf a preflight check fails, the agent prompts the user to authorize the\ninstall\u002Ffix via Bash. Pip-installable Python requirements are the exception:\ninstall them automatically, then rerun preflight.\n\nSee `references\u002Fslurm-ssh-credentials.md` for the full preflight script, the\nenroot-credentials heredoc, prerequisite key setup (keypair, `ssh-copy-id`,\n`known_hosts`, container key mounts, 2FA handling), and the SSH failure\nremediation prompt.\n\n## Storage\n\nUse shared-filesystem URIs, not local or `file:\u002F\u002F` paths; `tao-core` rejects\nlocal\u002Ffile paths for remote backends.\n\n- `lustre:\u002F\u002F\u002Fabsolute\u002Fpath` for user-provided datasets on Lustre.\n- `slurm:\u002F\u002F` paths may appear in microservices metadata and are converted to\n  Lustre paths before the container starts.\n\nAccept either dataset roots (model skills map them to required files) or direct\nspec-key paths. After SSH succeeds and before generating scripts, `test -e` each\nrequired dataset path from the login host; if it fails, stop and ask for\ncorrected paths or staged data rather than producing scripts that fail in the\nfirst training job. See `references\u002Fslurm-ssh-credentials.md` for root vs.\ndirect-spec modes, backend details, and the results-dir default.\n\n## Container execution\n\n`tao-core` runs TAO containers through Pyxis\u002FEnroot:\n\n1. Stage compact JSON files for specs, environment, and cloud metadata under\n   `\u003Cjob_dir>\u002Fspecs`, `\u003Cjob_dir>\u002Fenv`, and `\u003Cjob_dir>\u002Fmeta`.\n2. Optionally convert the Docker image to a cached SQSH image with\n   `srun -n1 -p \u003Cconversion_partition> enroot import`.\n3. Write an sbatch script under `\u003Cjob_dir>\u002Fsbatch\u002Fjob_\u003Cjob_id>.sbatch`.\n4. Submit `sbatch --export=ALL \u003Cscript>`.\n5. Run the container with `srun --container-image=\u003Cimage> --container-mounts=\u002Flustre`.\n\nAccepted image formats: `\u002Fpath\u002Fto\u002Fimage.sqsh`, `registry#image:tag`,\n`docker:\u002F\u002Fregistry#image:tag`, and ordinary `registry\u002Fimage:tag` (converted to\nPyxis form when needed). SQSH conversion is cached by image name; for `:latest`\nimages the cached SQSH is reused unless `force_reconvert_latest` is enabled.\n\n## Monitoring and cancellation\n\n- Scheduler status comes from the stored SLURM job id via `squeue`\u002F`sacct`;\n  TAO terminal status comes from `status.json` in the shared results folder.\n- While chat monitoring is enabled, keep polling at the requested interval for\n  any non-terminal job (`PENDING`, `RUNNING`, or otherwise). Do not stop after a\n  fixed elapsed time such as 30 minutes; long queue waits are normal on shared\n  GPU partitions.\n- Do not send a final response for a non-terminal SLURM job when chat\n  monitoring is enabled. A final response is a detach action; use it only if the\n  user asked to detach\u002Fstop or the job reached terminal state.\n- Logs are read over SSH from\n  `\u003Cjob_dir>\u002Fslurm-logs\u002F\u003Cslurm_job_name>-\u003Cslurm_job_id>\u002Fmain.out` and `.err`.\n- Cancel by looking up `backend_details.slurm_metadata.slurm_job_id` and running\n  `scancel \u003Cslurm_job_id>` over SSH. Treat missing or already terminated jobs as\n  successful cancellation.\n\nStatus mapping:\n\n- `PENDING` -> `Pending`\n- `RUNNING` or `COMPLETING` -> `Running`\n- `COMPLETED` -> check `status.json`\n- `FAILED`, `BOOT_FAIL`, `DEADLINE`, `OUT_OF_MEMORY`, `NODE_FAIL` -> retry if\n  logs match retriable infrastructure patterns, otherwise `Error`\n- `CANCELLED`, `PREEMPTED`, `REVOKED` -> `Canceled`\n- `TIMEOUT` -> `Error`\n- `SUSPENDED`, `STOPPED` -> `Paused`\n\n## Required inputs\n\nAsk for these in the SLURM intake; see `references\u002Fslurm-ssh-credentials.md`\nfor the full credential list, microservices schema keys, and defaults.\n\n- **SLURM_USER** (required): SSH username for the login node.\n- **SLURM_HOSTNAME** (required): Comma-separated login hostnames for failover.\n- **SLURM_PARTITION** (required): Partition list for GPU submission. Packaged\n  default `polar,polar3,polar4,grizzly`, treated as 4-hour queues.\n- **SSH_KEY_PATH** (preferred, expected before launch): private key for\n  non-interactive public-key auth. Ask for this first in remediation; prefer it\n  over the `SSH_AUTH_SOCK` agent-socket fallback.\n- **SLURM_BASE_RESULTS_DIR** (optional): base shared-filesystem path; default\n  `\u002Flustre\u002Ffsw\u002Fportfolios\u002Fedgeai\u002Fusers\u002F\u003Cyour-dir>` (your per-user Lustre dir).\n- **SLURM_ACCOUNT** (usually required by site policy): account for `#SBATCH --account`.\n\nDo not ask for `SLURM_ACCOUNT` or `SLURM_BASE_RESULTS_DIR` in the initial\nintake unless the user says their site requires an account, wants a custom\nresults root, or the workflow cannot proceed without overriding defaults.\n\n## Resource defaults\n\nDefaults from `tao-core`:\n\n- `num_nodes`: 1\n- `num_gpus`: 4\n- `max_num_gpus_per_node`: 8\n- `cpus_per_task`: 16\n- `time_hours`: 4\n- `timeout_hours`: 3.8\n- `max_time_hours`: 4\n- `container_mounts`: `\u002Flustre`\n- `use_requeue`: true\n- `use_sqsh`: true\n\nWhen generating launchers or wrapper scripts for SLURM, set the wall-time\ndefaults explicitly from the packaged platform resource defaults:\n\n```bash\nexport SLURM_TIME_HOURS=\"${SLURM_TIME_HOURS:-4}\"\nexport SLURM_TIMEOUT_HOURS=\"${SLURM_TIMEOUT_HOURS:-3.8}\"\n```\n\nDo not default to 12 hours on SLURM. If the user supplies a longer\n`SLURM_TIME_HOURS`, verify that the selected partition supports it before\nsubmitting. For the packaged default partition list\n`polar,polar3,polar4,grizzly`, reject requests above 4 hours and ask for a\ndifferent partition only if the user actually wants a longer wall time.\n\nWhen `num_gpus` is greater than or equal to `max_num_gpus_per_node`, the\nhandler treats the request as exclusive per node and computes additional nodes\nfrom total GPU count when necessary.\n\n## Multi-node, SDK, and retries\n\nFor multi-node jobs (`num_nodes > 1`), the SDK builds the sbatch directives and\nexports the PyTorch-distributed rendezvous env vars automatically: `WORLD_SIZE`,\n`NUM_GPU_PER_NODE`, `NODE_RANK`, `MASTER_ADDR`, and `MASTER_PORT` (29500).\nTAO entrypoints read `WORLD_SIZE` + `NUM_GPU_PER_NODE` and build torchrun\ninternally. Cosmos-RL has special multi-node role handling for controller,\npolicy, and rollout workers.\n\n**Use Lustre, not S3, for SLURM job inputs.** The GPU allocation starts the\nmoment the job is dispatched, so a long `s3:\u002F\u002F` download at the top of the\nscript burns the allocation, can get the job killed for GPU-idle, and is billed\neither way. Stage training data on the shared filesystem first and reference it\nas `lustre:\u002F\u002F\u002F...`. S3\u002FHF\u002FNGC pre-fetch is fine for small auxiliary inputs\n(checkpoints, configs), not training datasets. K8s\u002FBrev do not share this\nscheduler-idle constraint.\n\nAuto-retry of infrastructure failures (`NODE_FAIL`, `BOOT_FAIL`, NCCL transport\ntimeouts, CUDA driver init failures, GPU\u002FIB link-down, OOM-killer node reaping,\nXid errors) is automatic in the SDK, with a stable user-facing `Job.id` across\nretries. Plain training failures surface immediately so a broken spec does not\nconsume the retry budget. `#SBATCH --requeue` is enabled by default via\n`SLURM_USE_REQUEUE=true`.\n\nSee `references\u002Fslurm-container-execution.md` for the full multi-node\nenv-var\u002Fsbatch directive detail and table, cluster requirements, the optional\nTAO SDK path (`SlurmSDK`, `build_entrypoint`, `ActionWorkflow`) with code, the\nLustre-not-S3 rule in full, and the failure-mode checklist;\n`references\u002Fslurm-execution-sdk.md` covers the `MAX_JOB_RETRIES` retry budget.\nWhen the SDK is in scope, read `tao-skill-bank:tao-run-platform` for the\n`SlurmSDK` kwarg reference.\n\n## References\n\n- `references\u002Fslurm-ssh-credentials.md` — preflight script, SSH\u002Fkey setup,\n  enroot credentials, full credential list, backend details, storage rules,\n  SSH remediation prompt.\n- `references\u002Fslurm-container-execution.md` — container execution steps,\n  monitoring, status mapping, cancellation, multi-node detail, SDK use,\n  Lustre-not-S3, auto-retry, failure modes.\n- `references\u002Fslurm-preflight-storage.md` — extended preflight\u002Fstorage notes.\n- `references\u002Fslurm-execution-sdk.md` — extended execution\u002FSDK notes.\n- `references\u002Fdetailed-guide.md` — navigation map for the split references.\n",{"data":35,"body":44},{"name":4,"description":6,"license":26,"compatibility":36,"metadata":37,"allowed-tools":40,"tags":41},"Requires SSH access to a SLURM login node (passwordless via key auth) and SLURM_USER + SLURM_HOSTNAME env vars. The TAO SDK with the slurm extra (pip install 'nvidia-tao-sdk[slurm]') is needed only if you want Job handles, S3 I\u002FO wrapping, or run-folder durability via ActionWorkflow.",{"author":38,"version":39},"NVIDIA Corporation","0.1.0","Read Bash",[42,43],"platform","slurm",{"type":45,"children":46},"root",[47,55,78,85,90,96,188,193,222,228,249,276,296,302,312,393,444,450,546,551,722,728,740,833,851,857,869,985,990,1099,1118,1137,1143,1206,1232,1273,1338,1344,1399],{"type":48,"tag":49,"props":50,"children":51},"element","h1",{"id":43},[52],{"type":53,"value":54},"text","SLURM",{"type":48,"tag":56,"props":57,"children":58},"p",{},[59,61,68,70,76],{"type":53,"value":60},"Remote GPU compute platform for clusters managed by SLURM. Jobs are submitted\nfrom the TAO service or SDK host to a login node over SSH, staged on a shared\nfilesystem, submitted with ",{"type":48,"tag":62,"props":63,"children":65},"code",{"className":64},[],[66],{"type":53,"value":67},"sbatch",{"type":53,"value":69},", and executed with ",{"type":48,"tag":62,"props":71,"children":73},{"className":72},[],[74],{"type":53,"value":75},"srun",{"type":53,"value":77}," container support.",{"type":48,"tag":79,"props":80,"children":82},"h2",{"id":81},"when-to-use",[83],{"type":53,"value":84},"When to use",{"type":48,"tag":56,"props":86,"children":87},{},[88],{"type":53,"value":89},"Use SLURM when the user has access to a managed GPU cluster, shared Lustre\nstorage, and scheduler-owned GPU allocation. Do not use SLURM for local files\nthat exist only on the agent machine; data and outputs must be reachable from\nthe cluster.",{"type":48,"tag":79,"props":91,"children":93},{"id":92},"preflight-ssh",[94],{"type":53,"value":95},"Preflight + SSH",{"type":48,"tag":56,"props":97,"children":98},{},[99,101,107,109,115,117,123,125,131,133,139,141,147,149,155,157,163,165,170,172,178,180,186],{"type":53,"value":100},"Confirm ",{"type":48,"tag":62,"props":102,"children":104},{"className":103},[],[105],{"type":53,"value":106},"SLURM_USER",{"type":53,"value":108}," and ",{"type":48,"tag":62,"props":110,"children":112},{"className":111},[],[113],{"type":53,"value":114},"SLURM_HOSTNAME",{"type":53,"value":116}," are exported and passwordless SSH to a\nlogin host works (",{"type":48,"tag":62,"props":118,"children":120},{"className":119},[],[121],{"type":53,"value":122},"ssh -o BatchMode=yes",{"type":53,"value":124},"). Optionally install the TAO SDK\nwrapper for Job handles + S3 wrapping (",{"type":48,"tag":62,"props":126,"children":128},{"className":127},[],[129],{"type":53,"value":130},"nvidia-tao-sdk[slurm]",{"type":53,"value":132},", on public PyPI).\nFor private ",{"type":48,"tag":62,"props":134,"children":136},{"className":135},[],[137],{"type":53,"value":138},"nvcr.io",{"type":53,"value":140}," images, install ",{"type":48,"tag":62,"props":142,"children":144},{"className":143},[],[145],{"type":53,"value":146},"~\u002F.config\u002Fenroot\u002F.credentials",{"type":53,"value":148}," on the\ncluster once per (cluster, user): Pyxis\u002FEnroot does not read ",{"type":48,"tag":62,"props":150,"children":152},{"className":151},[],[153],{"type":53,"value":154},"NGC_KEY",{"type":53,"value":156}," from the\njob env, and without persistent credentials, auth-gated pulls fail with \"Could\nnot process JSON input\" at job startup. Install it via the ",{"type":48,"tag":62,"props":158,"children":160},{"className":159},[],[161],{"type":53,"value":162},"printf | ssh",{"type":53,"value":164},"\nheredoc so the ",{"type":48,"tag":62,"props":166,"children":168},{"className":167},[],[169],{"type":53,"value":154},{"type":53,"value":171}," value never lands in shell history, intermediate files,\nor chat output; never ",{"type":48,"tag":62,"props":173,"children":175},{"className":174},[],[176],{"type":53,"value":177},"cat",{"type":53,"value":179},"\u002F",{"type":48,"tag":62,"props":181,"children":183},{"className":182},[],[184],{"type":53,"value":185},"echo",{"type":53,"value":187}," the value.",{"type":48,"tag":56,"props":189,"children":190},{},[191],{"type":53,"value":192},"If a preflight check fails, the agent prompts the user to authorize the\ninstall\u002Ffix via Bash. Pip-installable Python requirements are the exception:\ninstall them automatically, then rerun preflight.",{"type":48,"tag":56,"props":194,"children":195},{},[196,198,204,206,212,214,220],{"type":53,"value":197},"See ",{"type":48,"tag":62,"props":199,"children":201},{"className":200},[],[202],{"type":53,"value":203},"references\u002Fslurm-ssh-credentials.md",{"type":53,"value":205}," for the full preflight script, the\nenroot-credentials heredoc, prerequisite key setup (keypair, ",{"type":48,"tag":62,"props":207,"children":209},{"className":208},[],[210],{"type":53,"value":211},"ssh-copy-id",{"type":53,"value":213},",\n",{"type":48,"tag":62,"props":215,"children":217},{"className":216},[],[218],{"type":53,"value":219},"known_hosts",{"type":53,"value":221},", container key mounts, 2FA handling), and the SSH failure\nremediation prompt.",{"type":48,"tag":79,"props":223,"children":225},{"id":224},"storage",[226],{"type":53,"value":227},"Storage",{"type":48,"tag":56,"props":229,"children":230},{},[231,233,239,241,247],{"type":53,"value":232},"Use shared-filesystem URIs, not local or ",{"type":48,"tag":62,"props":234,"children":236},{"className":235},[],[237],{"type":53,"value":238},"file:\u002F\u002F",{"type":53,"value":240}," paths; ",{"type":48,"tag":62,"props":242,"children":244},{"className":243},[],[245],{"type":53,"value":246},"tao-core",{"type":53,"value":248}," rejects\nlocal\u002Ffile paths for remote backends.",{"type":48,"tag":250,"props":251,"children":252},"ul",{},[253,265],{"type":48,"tag":254,"props":255,"children":256},"li",{},[257,263],{"type":48,"tag":62,"props":258,"children":260},{"className":259},[],[261],{"type":53,"value":262},"lustre:\u002F\u002F\u002Fabsolute\u002Fpath",{"type":53,"value":264}," for user-provided datasets on Lustre.",{"type":48,"tag":254,"props":266,"children":267},{},[268,274],{"type":48,"tag":62,"props":269,"children":271},{"className":270},[],[272],{"type":53,"value":273},"slurm:\u002F\u002F",{"type":53,"value":275}," paths may appear in microservices metadata and are converted to\nLustre paths before the container starts.",{"type":48,"tag":56,"props":277,"children":278},{},[279,281,287,289,294],{"type":53,"value":280},"Accept either dataset roots (model skills map them to required files) or direct\nspec-key paths. After SSH succeeds and before generating scripts, ",{"type":48,"tag":62,"props":282,"children":284},{"className":283},[],[285],{"type":53,"value":286},"test -e",{"type":53,"value":288}," each\nrequired dataset path from the login host; if it fails, stop and ask for\ncorrected paths or staged data rather than producing scripts that fail in the\nfirst training job. See ",{"type":48,"tag":62,"props":290,"children":292},{"className":291},[],[293],{"type":53,"value":203},{"type":53,"value":295}," for root vs.\ndirect-spec modes, backend details, and the results-dir default.",{"type":48,"tag":79,"props":297,"children":299},{"id":298},"container-execution",[300],{"type":53,"value":301},"Container execution",{"type":48,"tag":56,"props":303,"children":304},{},[305,310],{"type":48,"tag":62,"props":306,"children":308},{"className":307},[],[309],{"type":53,"value":246},{"type":53,"value":311}," runs TAO containers through Pyxis\u002FEnroot:",{"type":48,"tag":313,"props":314,"children":315},"ol",{},[316,345,357,369,381],{"type":48,"tag":254,"props":317,"children":318},{},[319,321,327,329,335,337,343],{"type":53,"value":320},"Stage compact JSON files for specs, environment, and cloud metadata under\n",{"type":48,"tag":62,"props":322,"children":324},{"className":323},[],[325],{"type":53,"value":326},"\u003Cjob_dir>\u002Fspecs",{"type":53,"value":328},", ",{"type":48,"tag":62,"props":330,"children":332},{"className":331},[],[333],{"type":53,"value":334},"\u003Cjob_dir>\u002Fenv",{"type":53,"value":336},", and ",{"type":48,"tag":62,"props":338,"children":340},{"className":339},[],[341],{"type":53,"value":342},"\u003Cjob_dir>\u002Fmeta",{"type":53,"value":344},".",{"type":48,"tag":254,"props":346,"children":347},{},[348,350,356],{"type":53,"value":349},"Optionally convert the Docker image to a cached SQSH image with\n",{"type":48,"tag":62,"props":351,"children":353},{"className":352},[],[354],{"type":53,"value":355},"srun -n1 -p \u003Cconversion_partition> enroot import",{"type":53,"value":344},{"type":48,"tag":254,"props":358,"children":359},{},[360,362,368],{"type":53,"value":361},"Write an sbatch script under ",{"type":48,"tag":62,"props":363,"children":365},{"className":364},[],[366],{"type":53,"value":367},"\u003Cjob_dir>\u002Fsbatch\u002Fjob_\u003Cjob_id>.sbatch",{"type":53,"value":344},{"type":48,"tag":254,"props":370,"children":371},{},[372,374,380],{"type":53,"value":373},"Submit ",{"type":48,"tag":62,"props":375,"children":377},{"className":376},[],[378],{"type":53,"value":379},"sbatch --export=ALL \u003Cscript>",{"type":53,"value":344},{"type":48,"tag":254,"props":382,"children":383},{},[384,386,392],{"type":53,"value":385},"Run the container with ",{"type":48,"tag":62,"props":387,"children":389},{"className":388},[],[390],{"type":53,"value":391},"srun --container-image=\u003Cimage> --container-mounts=\u002Flustre",{"type":53,"value":344},{"type":48,"tag":56,"props":394,"children":395},{},[396,398,404,405,411,412,418,420,426,428,434,436,442],{"type":53,"value":397},"Accepted image formats: ",{"type":48,"tag":62,"props":399,"children":401},{"className":400},[],[402],{"type":53,"value":403},"\u002Fpath\u002Fto\u002Fimage.sqsh",{"type":53,"value":328},{"type":48,"tag":62,"props":406,"children":408},{"className":407},[],[409],{"type":53,"value":410},"registry#image:tag",{"type":53,"value":213},{"type":48,"tag":62,"props":413,"children":415},{"className":414},[],[416],{"type":53,"value":417},"docker:\u002F\u002Fregistry#image:tag",{"type":53,"value":419},", and ordinary ",{"type":48,"tag":62,"props":421,"children":423},{"className":422},[],[424],{"type":53,"value":425},"registry\u002Fimage:tag",{"type":53,"value":427}," (converted to\nPyxis form when needed). SQSH conversion is cached by image name; for ",{"type":48,"tag":62,"props":429,"children":431},{"className":430},[],[432],{"type":53,"value":433},":latest",{"type":53,"value":435},"\nimages the cached SQSH is reused unless ",{"type":48,"tag":62,"props":437,"children":439},{"className":438},[],[440],{"type":53,"value":441},"force_reconvert_latest",{"type":53,"value":443}," is enabled.",{"type":48,"tag":79,"props":445,"children":447},{"id":446},"monitoring-and-cancellation",[448],{"type":53,"value":449},"Monitoring and cancellation",{"type":48,"tag":250,"props":451,"children":452},{},[453,481,501,506,525],{"type":48,"tag":254,"props":454,"children":455},{},[456,458,464,465,471,473,479],{"type":53,"value":457},"Scheduler status comes from the stored SLURM job id via ",{"type":48,"tag":62,"props":459,"children":461},{"className":460},[],[462],{"type":53,"value":463},"squeue",{"type":53,"value":179},{"type":48,"tag":62,"props":466,"children":468},{"className":467},[],[469],{"type":53,"value":470},"sacct",{"type":53,"value":472},";\nTAO terminal status comes from ",{"type":48,"tag":62,"props":474,"children":476},{"className":475},[],[477],{"type":53,"value":478},"status.json",{"type":53,"value":480}," in the shared results folder.",{"type":48,"tag":254,"props":482,"children":483},{},[484,486,492,493,499],{"type":53,"value":485},"While chat monitoring is enabled, keep polling at the requested interval for\nany non-terminal job (",{"type":48,"tag":62,"props":487,"children":489},{"className":488},[],[490],{"type":53,"value":491},"PENDING",{"type":53,"value":328},{"type":48,"tag":62,"props":494,"children":496},{"className":495},[],[497],{"type":53,"value":498},"RUNNING",{"type":53,"value":500},", or otherwise). Do not stop after a\nfixed elapsed time such as 30 minutes; long queue waits are normal on shared\nGPU partitions.",{"type":48,"tag":254,"props":502,"children":503},{},[504],{"type":53,"value":505},"Do not send a final response for a non-terminal SLURM job when chat\nmonitoring is enabled. A final response is a detach action; use it only if the\nuser asked to detach\u002Fstop or the job reached terminal state.",{"type":48,"tag":254,"props":507,"children":508},{},[509,511,517,518,524],{"type":53,"value":510},"Logs are read over SSH from\n",{"type":48,"tag":62,"props":512,"children":514},{"className":513},[],[515],{"type":53,"value":516},"\u003Cjob_dir>\u002Fslurm-logs\u002F\u003Cslurm_job_name>-\u003Cslurm_job_id>\u002Fmain.out",{"type":53,"value":108},{"type":48,"tag":62,"props":519,"children":521},{"className":520},[],[522],{"type":53,"value":523},".err",{"type":53,"value":344},{"type":48,"tag":254,"props":526,"children":527},{},[528,530,536,538,544],{"type":53,"value":529},"Cancel by looking up ",{"type":48,"tag":62,"props":531,"children":533},{"className":532},[],[534],{"type":53,"value":535},"backend_details.slurm_metadata.slurm_job_id",{"type":53,"value":537}," and running\n",{"type":48,"tag":62,"props":539,"children":541},{"className":540},[],[542],{"type":53,"value":543},"scancel \u003Cslurm_job_id>",{"type":53,"value":545}," over SSH. Treat missing or already terminated jobs as\nsuccessful cancellation.",{"type":48,"tag":56,"props":547,"children":548},{},[549],{"type":53,"value":550},"Status mapping:",{"type":48,"tag":250,"props":552,"children":553},{},[554,570,593,609,654,684,699],{"type":48,"tag":254,"props":555,"children":556},{},[557,562,564],{"type":48,"tag":62,"props":558,"children":560},{"className":559},[],[561],{"type":53,"value":491},{"type":53,"value":563}," -> ",{"type":48,"tag":62,"props":565,"children":567},{"className":566},[],[568],{"type":53,"value":569},"Pending",{"type":48,"tag":254,"props":571,"children":572},{},[573,578,580,586,587],{"type":48,"tag":62,"props":574,"children":576},{"className":575},[],[577],{"type":53,"value":498},{"type":53,"value":579}," or ",{"type":48,"tag":62,"props":581,"children":583},{"className":582},[],[584],{"type":53,"value":585},"COMPLETING",{"type":53,"value":563},{"type":48,"tag":62,"props":588,"children":590},{"className":589},[],[591],{"type":53,"value":592},"Running",{"type":48,"tag":254,"props":594,"children":595},{},[596,602,604],{"type":48,"tag":62,"props":597,"children":599},{"className":598},[],[600],{"type":53,"value":601},"COMPLETED",{"type":53,"value":603}," -> check ",{"type":48,"tag":62,"props":605,"children":607},{"className":606},[],[608],{"type":53,"value":478},{"type":48,"tag":254,"props":610,"children":611},{},[612,618,619,625,626,632,633,639,640,646,648],{"type":48,"tag":62,"props":613,"children":615},{"className":614},[],[616],{"type":53,"value":617},"FAILED",{"type":53,"value":328},{"type":48,"tag":62,"props":620,"children":622},{"className":621},[],[623],{"type":53,"value":624},"BOOT_FAIL",{"type":53,"value":328},{"type":48,"tag":62,"props":627,"children":629},{"className":628},[],[630],{"type":53,"value":631},"DEADLINE",{"type":53,"value":328},{"type":48,"tag":62,"props":634,"children":636},{"className":635},[],[637],{"type":53,"value":638},"OUT_OF_MEMORY",{"type":53,"value":328},{"type":48,"tag":62,"props":641,"children":643},{"className":642},[],[644],{"type":53,"value":645},"NODE_FAIL",{"type":53,"value":647}," -> retry if\nlogs match retriable infrastructure patterns, otherwise ",{"type":48,"tag":62,"props":649,"children":651},{"className":650},[],[652],{"type":53,"value":653},"Error",{"type":48,"tag":254,"props":655,"children":656},{},[657,663,664,670,671,677,678],{"type":48,"tag":62,"props":658,"children":660},{"className":659},[],[661],{"type":53,"value":662},"CANCELLED",{"type":53,"value":328},{"type":48,"tag":62,"props":665,"children":667},{"className":666},[],[668],{"type":53,"value":669},"PREEMPTED",{"type":53,"value":328},{"type":48,"tag":62,"props":672,"children":674},{"className":673},[],[675],{"type":53,"value":676},"REVOKED",{"type":53,"value":563},{"type":48,"tag":62,"props":679,"children":681},{"className":680},[],[682],{"type":53,"value":683},"Canceled",{"type":48,"tag":254,"props":685,"children":686},{},[687,693,694],{"type":48,"tag":62,"props":688,"children":690},{"className":689},[],[691],{"type":53,"value":692},"TIMEOUT",{"type":53,"value":563},{"type":48,"tag":62,"props":695,"children":697},{"className":696},[],[698],{"type":53,"value":653},{"type":48,"tag":254,"props":700,"children":701},{},[702,708,709,715,716],{"type":48,"tag":62,"props":703,"children":705},{"className":704},[],[706],{"type":53,"value":707},"SUSPENDED",{"type":53,"value":328},{"type":48,"tag":62,"props":710,"children":712},{"className":711},[],[713],{"type":53,"value":714},"STOPPED",{"type":53,"value":563},{"type":48,"tag":62,"props":717,"children":719},{"className":718},[],[720],{"type":53,"value":721},"Paused",{"type":48,"tag":79,"props":723,"children":725},{"id":724},"required-inputs",[726],{"type":53,"value":727},"Required inputs",{"type":48,"tag":56,"props":729,"children":730},{},[731,733,738],{"type":53,"value":732},"Ask for these in the SLURM intake; see ",{"type":48,"tag":62,"props":734,"children":736},{"className":735},[],[737],{"type":53,"value":203},{"type":53,"value":739},"\nfor the full credential list, microservices schema keys, and defaults.",{"type":48,"tag":250,"props":741,"children":742},{},[743,753,762,780,798,816],{"type":48,"tag":254,"props":744,"children":745},{},[746,751],{"type":48,"tag":747,"props":748,"children":749},"strong",{},[750],{"type":53,"value":106},{"type":53,"value":752}," (required): SSH username for the login node.",{"type":48,"tag":254,"props":754,"children":755},{},[756,760],{"type":48,"tag":747,"props":757,"children":758},{},[759],{"type":53,"value":114},{"type":53,"value":761}," (required): Comma-separated login hostnames for failover.",{"type":48,"tag":254,"props":763,"children":764},{},[765,770,772,778],{"type":48,"tag":747,"props":766,"children":767},{},[768],{"type":53,"value":769},"SLURM_PARTITION",{"type":53,"value":771}," (required): Partition list for GPU submission. Packaged\ndefault ",{"type":48,"tag":62,"props":773,"children":775},{"className":774},[],[776],{"type":53,"value":777},"polar,polar3,polar4,grizzly",{"type":53,"value":779},", treated as 4-hour queues.",{"type":48,"tag":254,"props":781,"children":782},{},[783,788,790,796],{"type":48,"tag":747,"props":784,"children":785},{},[786],{"type":53,"value":787},"SSH_KEY_PATH",{"type":53,"value":789}," (preferred, expected before launch): private key for\nnon-interactive public-key auth. Ask for this first in remediation; prefer it\nover the ",{"type":48,"tag":62,"props":791,"children":793},{"className":792},[],[794],{"type":53,"value":795},"SSH_AUTH_SOCK",{"type":53,"value":797}," agent-socket fallback.",{"type":48,"tag":254,"props":799,"children":800},{},[801,806,808,814],{"type":48,"tag":747,"props":802,"children":803},{},[804],{"type":53,"value":805},"SLURM_BASE_RESULTS_DIR",{"type":53,"value":807}," (optional): base shared-filesystem path; default\n",{"type":48,"tag":62,"props":809,"children":811},{"className":810},[],[812],{"type":53,"value":813},"\u002Flustre\u002Ffsw\u002Fportfolios\u002Fedgeai\u002Fusers\u002F\u003Cyour-dir>",{"type":53,"value":815}," (your per-user Lustre dir).",{"type":48,"tag":254,"props":817,"children":818},{},[819,824,826,832],{"type":48,"tag":747,"props":820,"children":821},{},[822],{"type":53,"value":823},"SLURM_ACCOUNT",{"type":53,"value":825}," (usually required by site policy): account for ",{"type":48,"tag":62,"props":827,"children":829},{"className":828},[],[830],{"type":53,"value":831},"#SBATCH --account",{"type":53,"value":344},{"type":48,"tag":56,"props":834,"children":835},{},[836,838,843,844,849],{"type":53,"value":837},"Do not ask for ",{"type":48,"tag":62,"props":839,"children":841},{"className":840},[],[842],{"type":53,"value":823},{"type":53,"value":579},{"type":48,"tag":62,"props":845,"children":847},{"className":846},[],[848],{"type":53,"value":805},{"type":53,"value":850}," in the initial\nintake unless the user says their site requires an account, wants a custom\nresults root, or the workflow cannot proceed without overriding defaults.",{"type":48,"tag":79,"props":852,"children":854},{"id":853},"resource-defaults",[855],{"type":53,"value":856},"Resource defaults",{"type":48,"tag":56,"props":858,"children":859},{},[860,862,867],{"type":53,"value":861},"Defaults from ",{"type":48,"tag":62,"props":863,"children":865},{"className":864},[],[866],{"type":53,"value":246},{"type":53,"value":868},":",{"type":48,"tag":250,"props":870,"children":871},{},[872,883,894,905,916,926,937,947,964,975],{"type":48,"tag":254,"props":873,"children":874},{},[875,881],{"type":48,"tag":62,"props":876,"children":878},{"className":877},[],[879],{"type":53,"value":880},"num_nodes",{"type":53,"value":882},": 1",{"type":48,"tag":254,"props":884,"children":885},{},[886,892],{"type":48,"tag":62,"props":887,"children":889},{"className":888},[],[890],{"type":53,"value":891},"num_gpus",{"type":53,"value":893},": 4",{"type":48,"tag":254,"props":895,"children":896},{},[897,903],{"type":48,"tag":62,"props":898,"children":900},{"className":899},[],[901],{"type":53,"value":902},"max_num_gpus_per_node",{"type":53,"value":904},": 8",{"type":48,"tag":254,"props":906,"children":907},{},[908,914],{"type":48,"tag":62,"props":909,"children":911},{"className":910},[],[912],{"type":53,"value":913},"cpus_per_task",{"type":53,"value":915},": 16",{"type":48,"tag":254,"props":917,"children":918},{},[919,925],{"type":48,"tag":62,"props":920,"children":922},{"className":921},[],[923],{"type":53,"value":924},"time_hours",{"type":53,"value":893},{"type":48,"tag":254,"props":927,"children":928},{},[929,935],{"type":48,"tag":62,"props":930,"children":932},{"className":931},[],[933],{"type":53,"value":934},"timeout_hours",{"type":53,"value":936},": 3.8",{"type":48,"tag":254,"props":938,"children":939},{},[940,946],{"type":48,"tag":62,"props":941,"children":943},{"className":942},[],[944],{"type":53,"value":945},"max_time_hours",{"type":53,"value":893},{"type":48,"tag":254,"props":948,"children":949},{},[950,956,958],{"type":48,"tag":62,"props":951,"children":953},{"className":952},[],[954],{"type":53,"value":955},"container_mounts",{"type":53,"value":957},": ",{"type":48,"tag":62,"props":959,"children":961},{"className":960},[],[962],{"type":53,"value":963},"\u002Flustre",{"type":48,"tag":254,"props":965,"children":966},{},[967,973],{"type":48,"tag":62,"props":968,"children":970},{"className":969},[],[971],{"type":53,"value":972},"use_requeue",{"type":53,"value":974},": true",{"type":48,"tag":254,"props":976,"children":977},{},[978,984],{"type":48,"tag":62,"props":979,"children":981},{"className":980},[],[982],{"type":53,"value":983},"use_sqsh",{"type":53,"value":974},{"type":48,"tag":56,"props":986,"children":987},{},[988],{"type":53,"value":989},"When generating launchers or wrapper scripts for SLURM, set the wall-time\ndefaults explicitly from the packaged platform resource defaults:",{"type":48,"tag":991,"props":992,"children":997},"pre",{"className":993,"code":994,"language":995,"meta":996,"style":996},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","export SLURM_TIME_HOURS=\"${SLURM_TIME_HOURS:-4}\"\nexport SLURM_TIMEOUT_HOURS=\"${SLURM_TIMEOUT_HOURS:-3.8}\"\n","bash","",[998],{"type":48,"tag":62,"props":999,"children":1000},{"__ignoreMap":996},[1001,1050],{"type":48,"tag":1002,"props":1003,"children":1006},"span",{"class":1004,"line":1005},"line",1,[1007,1013,1019,1025,1030,1035,1040,1045],{"type":48,"tag":1002,"props":1008,"children":1010},{"style":1009},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1011],{"type":53,"value":1012},"export",{"type":48,"tag":1002,"props":1014,"children":1016},{"style":1015},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1017],{"type":53,"value":1018}," SLURM_TIME_HOURS",{"type":48,"tag":1002,"props":1020,"children":1022},{"style":1021},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1023],{"type":53,"value":1024},"=",{"type":48,"tag":1002,"props":1026,"children":1027},{"style":1021},[1028],{"type":53,"value":1029},"\"${",{"type":48,"tag":1002,"props":1031,"children":1032},{"style":1015},[1033],{"type":53,"value":1034},"SLURM_TIME_HOURS",{"type":48,"tag":1002,"props":1036,"children":1037},{"style":1021},[1038],{"type":53,"value":1039},":-",{"type":48,"tag":1002,"props":1041,"children":1042},{"style":1015},[1043],{"type":53,"value":1044},"4",{"type":48,"tag":1002,"props":1046,"children":1047},{"style":1021},[1048],{"type":53,"value":1049},"}\"\n",{"type":48,"tag":1002,"props":1051,"children":1053},{"class":1004,"line":1052},2,[1054,1058,1063,1067,1071,1076,1080,1085,1090,1095],{"type":48,"tag":1002,"props":1055,"children":1056},{"style":1009},[1057],{"type":53,"value":1012},{"type":48,"tag":1002,"props":1059,"children":1060},{"style":1015},[1061],{"type":53,"value":1062}," SLURM_TIMEOUT_HOURS",{"type":48,"tag":1002,"props":1064,"children":1065},{"style":1021},[1066],{"type":53,"value":1024},{"type":48,"tag":1002,"props":1068,"children":1069},{"style":1021},[1070],{"type":53,"value":1029},{"type":48,"tag":1002,"props":1072,"children":1073},{"style":1015},[1074],{"type":53,"value":1075},"SLURM_TIMEOUT_HOURS",{"type":48,"tag":1002,"props":1077,"children":1078},{"style":1021},[1079],{"type":53,"value":1039},{"type":48,"tag":1002,"props":1081,"children":1082},{"style":1015},[1083],{"type":53,"value":1084},"3",{"type":48,"tag":1002,"props":1086,"children":1088},{"style":1087},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1089],{"type":53,"value":344},{"type":48,"tag":1002,"props":1091,"children":1092},{"style":1015},[1093],{"type":53,"value":1094},"8",{"type":48,"tag":1002,"props":1096,"children":1097},{"style":1021},[1098],{"type":53,"value":1049},{"type":48,"tag":56,"props":1100,"children":1101},{},[1102,1104,1109,1111,1116],{"type":53,"value":1103},"Do not default to 12 hours on SLURM. If the user supplies a longer\n",{"type":48,"tag":62,"props":1105,"children":1107},{"className":1106},[],[1108],{"type":53,"value":1034},{"type":53,"value":1110},", verify that the selected partition supports it before\nsubmitting. For the packaged default partition list\n",{"type":48,"tag":62,"props":1112,"children":1114},{"className":1113},[],[1115],{"type":53,"value":777},{"type":53,"value":1117},", reject requests above 4 hours and ask for a\ndifferent partition only if the user actually wants a longer wall time.",{"type":48,"tag":56,"props":1119,"children":1120},{},[1121,1123,1128,1130,1135],{"type":53,"value":1122},"When ",{"type":48,"tag":62,"props":1124,"children":1126},{"className":1125},[],[1127],{"type":53,"value":891},{"type":53,"value":1129}," is greater than or equal to ",{"type":48,"tag":62,"props":1131,"children":1133},{"className":1132},[],[1134],{"type":53,"value":902},{"type":53,"value":1136},", the\nhandler treats the request as exclusive per node and computes additional nodes\nfrom total GPU count when necessary.",{"type":48,"tag":79,"props":1138,"children":1140},{"id":1139},"multi-node-sdk-and-retries",[1141],{"type":53,"value":1142},"Multi-node, SDK, and retries",{"type":48,"tag":56,"props":1144,"children":1145},{},[1146,1148,1154,1156,1162,1163,1169,1170,1176,1177,1183,1184,1190,1192,1197,1199,1204],{"type":53,"value":1147},"For multi-node jobs (",{"type":48,"tag":62,"props":1149,"children":1151},{"className":1150},[],[1152],{"type":53,"value":1153},"num_nodes > 1",{"type":53,"value":1155},"), the SDK builds the sbatch directives and\nexports the PyTorch-distributed rendezvous env vars automatically: ",{"type":48,"tag":62,"props":1157,"children":1159},{"className":1158},[],[1160],{"type":53,"value":1161},"WORLD_SIZE",{"type":53,"value":213},{"type":48,"tag":62,"props":1164,"children":1166},{"className":1165},[],[1167],{"type":53,"value":1168},"NUM_GPU_PER_NODE",{"type":53,"value":328},{"type":48,"tag":62,"props":1171,"children":1173},{"className":1172},[],[1174],{"type":53,"value":1175},"NODE_RANK",{"type":53,"value":328},{"type":48,"tag":62,"props":1178,"children":1180},{"className":1179},[],[1181],{"type":53,"value":1182},"MASTER_ADDR",{"type":53,"value":336},{"type":48,"tag":62,"props":1185,"children":1187},{"className":1186},[],[1188],{"type":53,"value":1189},"MASTER_PORT",{"type":53,"value":1191}," (29500).\nTAO entrypoints read ",{"type":48,"tag":62,"props":1193,"children":1195},{"className":1194},[],[1196],{"type":53,"value":1161},{"type":53,"value":1198}," + ",{"type":48,"tag":62,"props":1200,"children":1202},{"className":1201},[],[1203],{"type":53,"value":1168},{"type":53,"value":1205}," and build torchrun\ninternally. Cosmos-RL has special multi-node role handling for controller,\npolicy, and rollout workers.",{"type":48,"tag":56,"props":1207,"children":1208},{},[1209,1214,1216,1222,1224,1230],{"type":48,"tag":747,"props":1210,"children":1211},{},[1212],{"type":53,"value":1213},"Use Lustre, not S3, for SLURM job inputs.",{"type":53,"value":1215}," The GPU allocation starts the\nmoment the job is dispatched, so a long ",{"type":48,"tag":62,"props":1217,"children":1219},{"className":1218},[],[1220],{"type":53,"value":1221},"s3:\u002F\u002F",{"type":53,"value":1223}," download at the top of the\nscript burns the allocation, can get the job killed for GPU-idle, and is billed\neither way. Stage training data on the shared filesystem first and reference it\nas ",{"type":48,"tag":62,"props":1225,"children":1227},{"className":1226},[],[1228],{"type":53,"value":1229},"lustre:\u002F\u002F\u002F...",{"type":53,"value":1231},". S3\u002FHF\u002FNGC pre-fetch is fine for small auxiliary inputs\n(checkpoints, configs), not training datasets. K8s\u002FBrev do not share this\nscheduler-idle constraint.",{"type":48,"tag":56,"props":1233,"children":1234},{},[1235,1237,1242,1243,1248,1250,1256,1258,1264,1266,1272],{"type":53,"value":1236},"Auto-retry of infrastructure failures (",{"type":48,"tag":62,"props":1238,"children":1240},{"className":1239},[],[1241],{"type":53,"value":645},{"type":53,"value":328},{"type":48,"tag":62,"props":1244,"children":1246},{"className":1245},[],[1247],{"type":53,"value":624},{"type":53,"value":1249},", NCCL transport\ntimeouts, CUDA driver init failures, GPU\u002FIB link-down, OOM-killer node reaping,\nXid errors) is automatic in the SDK, with a stable user-facing ",{"type":48,"tag":62,"props":1251,"children":1253},{"className":1252},[],[1254],{"type":53,"value":1255},"Job.id",{"type":53,"value":1257}," across\nretries. Plain training failures surface immediately so a broken spec does not\nconsume the retry budget. ",{"type":48,"tag":62,"props":1259,"children":1261},{"className":1260},[],[1262],{"type":53,"value":1263},"#SBATCH --requeue",{"type":53,"value":1265}," is enabled by default via\n",{"type":48,"tag":62,"props":1267,"children":1269},{"className":1268},[],[1270],{"type":53,"value":1271},"SLURM_USE_REQUEUE=true",{"type":53,"value":344},{"type":48,"tag":56,"props":1274,"children":1275},{},[1276,1277,1283,1285,1291,1292,1298,1299,1305,1307,1313,1315,1321,1323,1329,1331,1336],{"type":53,"value":197},{"type":48,"tag":62,"props":1278,"children":1280},{"className":1279},[],[1281],{"type":53,"value":1282},"references\u002Fslurm-container-execution.md",{"type":53,"value":1284}," for the full multi-node\nenv-var\u002Fsbatch directive detail and table, cluster requirements, the optional\nTAO SDK path (",{"type":48,"tag":62,"props":1286,"children":1288},{"className":1287},[],[1289],{"type":53,"value":1290},"SlurmSDK",{"type":53,"value":328},{"type":48,"tag":62,"props":1293,"children":1295},{"className":1294},[],[1296],{"type":53,"value":1297},"build_entrypoint",{"type":53,"value":328},{"type":48,"tag":62,"props":1300,"children":1302},{"className":1301},[],[1303],{"type":53,"value":1304},"ActionWorkflow",{"type":53,"value":1306},") with code, the\nLustre-not-S3 rule in full, and the failure-mode checklist;\n",{"type":48,"tag":62,"props":1308,"children":1310},{"className":1309},[],[1311],{"type":53,"value":1312},"references\u002Fslurm-execution-sdk.md",{"type":53,"value":1314}," covers the ",{"type":48,"tag":62,"props":1316,"children":1318},{"className":1317},[],[1319],{"type":53,"value":1320},"MAX_JOB_RETRIES",{"type":53,"value":1322}," retry budget.\nWhen the SDK is in scope, read ",{"type":48,"tag":62,"props":1324,"children":1326},{"className":1325},[],[1327],{"type":53,"value":1328},"tao-skill-bank:tao-run-platform",{"type":53,"value":1330}," for the\n",{"type":48,"tag":62,"props":1332,"children":1334},{"className":1333},[],[1335],{"type":53,"value":1290},{"type":53,"value":1337}," kwarg reference.",{"type":48,"tag":79,"props":1339,"children":1341},{"id":1340},"references",[1342],{"type":53,"value":1343},"References",{"type":48,"tag":250,"props":1345,"children":1346},{},[1347,1357,1367,1378,1388],{"type":48,"tag":254,"props":1348,"children":1349},{},[1350,1355],{"type":48,"tag":62,"props":1351,"children":1353},{"className":1352},[],[1354],{"type":53,"value":203},{"type":53,"value":1356}," — preflight script, SSH\u002Fkey setup,\nenroot credentials, full credential list, backend details, storage rules,\nSSH remediation prompt.",{"type":48,"tag":254,"props":1358,"children":1359},{},[1360,1365],{"type":48,"tag":62,"props":1361,"children":1363},{"className":1362},[],[1364],{"type":53,"value":1282},{"type":53,"value":1366}," — container execution steps,\nmonitoring, status mapping, cancellation, multi-node detail, SDK use,\nLustre-not-S3, auto-retry, failure modes.",{"type":48,"tag":254,"props":1368,"children":1369},{},[1370,1376],{"type":48,"tag":62,"props":1371,"children":1373},{"className":1372},[],[1374],{"type":53,"value":1375},"references\u002Fslurm-preflight-storage.md",{"type":53,"value":1377}," — extended preflight\u002Fstorage notes.",{"type":48,"tag":254,"props":1379,"children":1380},{},[1381,1386],{"type":48,"tag":62,"props":1382,"children":1384},{"className":1383},[],[1385],{"type":53,"value":1312},{"type":53,"value":1387}," — extended execution\u002FSDK notes.",{"type":48,"tag":254,"props":1389,"children":1390},{},[1391,1397],{"type":48,"tag":62,"props":1392,"children":1394},{"className":1393},[],[1395],{"type":53,"value":1396},"references\u002Fdetailed-guide.md",{"type":53,"value":1398}," — navigation map for the split references.",{"type":48,"tag":1400,"props":1401,"children":1402},"style",{},[1403],{"type":53,"value":1404},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1406,"total":1561},[1407,1425,1439,1450,1462,1476,1489,1503,1516,1527,1541,1550],{"slug":1408,"name":1408,"fn":1409,"description":1410,"org":1411,"tags":1412,"stars":1422,"repoUrl":1423,"updatedAt":1424},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1413,1416,1419],{"name":1414,"slug":1415,"type":15},"Documentation","documentation",{"name":1417,"slug":1418,"type":15},"MCP","mcp",{"name":1420,"slug":1421,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":1426,"name":1426,"fn":1427,"description":1428,"org":1429,"tags":1430,"stars":1436,"repoUrl":1437,"updatedAt":1438},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1431,1432,1433],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"name":1434,"slug":1435,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":1440,"name":1440,"fn":1441,"description":1442,"org":1443,"tags":1444,"stars":1436,"repoUrl":1437,"updatedAt":1449},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1445,1448],{"name":1446,"slug":1447,"type":15},"CI\u002FCD","ci-cd",{"name":17,"slug":18,"type":15},"2026-07-14T05:25:59.97109",{"slug":1451,"name":1451,"fn":1452,"description":1453,"org":1454,"tags":1455,"stars":1436,"repoUrl":1437,"updatedAt":1461},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1456,1457,1458],{"name":1446,"slug":1447,"type":15},{"name":17,"slug":18,"type":15},{"name":1459,"slug":1460,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":1463,"name":1463,"fn":1464,"description":1465,"org":1466,"tags":1467,"stars":1436,"repoUrl":1437,"updatedAt":1475},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1468,1471,1472],{"name":1469,"slug":1470,"type":15},"Debugging","debugging",{"name":1459,"slug":1460,"type":15},{"name":1473,"slug":1474,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":1477,"name":1477,"fn":1478,"description":1479,"org":1480,"tags":1481,"stars":1436,"repoUrl":1437,"updatedAt":1488},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1482,1485],{"name":1483,"slug":1484,"type":15},"Best Practices","best-practices",{"name":1486,"slug":1487,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":1490,"name":1490,"fn":1491,"description":1492,"org":1493,"tags":1494,"stars":1436,"repoUrl":1437,"updatedAt":1502},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1495,1498,1501],{"name":1496,"slug":1497,"type":15},"Machine Learning","machine-learning",{"name":1499,"slug":1500,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":1504,"name":1504,"fn":1505,"description":1506,"org":1507,"tags":1508,"stars":1436,"repoUrl":1437,"updatedAt":1515},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1509,1512],{"name":1510,"slug":1511,"type":15},"QA","qa",{"name":1513,"slug":1514,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":1517,"name":1517,"fn":1518,"description":1519,"org":1520,"tags":1521,"stars":1436,"repoUrl":1437,"updatedAt":1526},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1522,1523],{"name":17,"slug":18,"type":15},{"name":1524,"slug":1525,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":1528,"name":1528,"fn":1529,"description":1530,"org":1531,"tags":1532,"stars":1436,"repoUrl":1437,"updatedAt":1540},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1533,1536,1537],{"name":1534,"slug":1535,"type":15},"Code Review","code-review",{"name":1459,"slug":1460,"type":15},{"name":1538,"slug":1539,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":1542,"name":1542,"fn":1543,"description":1544,"org":1545,"tags":1546,"stars":1436,"repoUrl":1437,"updatedAt":1549},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1547,1548],{"name":1510,"slug":1511,"type":15},{"name":1513,"slug":1514,"type":15},"2026-07-14T05:25:54.928983",{"slug":1551,"name":1551,"fn":1552,"description":1553,"org":1554,"tags":1555,"stars":1436,"repoUrl":1437,"updatedAt":1560},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1556,1559],{"name":1557,"slug":1558,"type":15},"Automation","automation",{"name":1446,"slug":1447,"type":15},"2026-07-30T05:29:03.275638",496,{"items":1563,"total":1659},[1564,1581,1591,1605,1615,1630,1645],{"slug":1565,"name":1565,"fn":1566,"description":1567,"org":1568,"tags":1569,"stars":23,"repoUrl":24,"updatedAt":1580},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1570,1573,1576,1577],{"name":1571,"slug":1572,"type":15},"Data Analysis","data-analysis",{"name":1574,"slug":1575,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":1578,"slug":1579,"type":15},"Performance","performance","2026-07-14T05:28:43.176466",{"slug":1582,"name":1582,"fn":1583,"description":1584,"org":1585,"tags":1586,"stars":23,"repoUrl":24,"updatedAt":1590},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1587,1588,1589],{"name":17,"slug":18,"type":15},{"name":1524,"slug":1525,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":1592,"name":1592,"fn":1593,"description":1594,"org":1595,"tags":1596,"stars":23,"repoUrl":24,"updatedAt":1604},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1597,1600,1601],{"name":1598,"slug":1599,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":1602,"slug":1603,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":1606,"name":1606,"fn":1607,"description":1608,"org":1609,"tags":1610,"stars":23,"repoUrl":24,"updatedAt":1614},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1611,1612,1613],{"name":1571,"slug":1572,"type":15},{"name":9,"slug":8,"type":15},{"name":1513,"slug":1514,"type":15},"2026-07-17T05:29:03.913266",{"slug":1616,"name":1616,"fn":1617,"description":1618,"org":1619,"tags":1620,"stars":23,"repoUrl":24,"updatedAt":1629},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1621,1622,1625,1626],{"name":1557,"slug":1558,"type":15},{"name":1623,"slug":1624,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":1627,"slug":1628,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":1631,"name":1631,"fn":1632,"description":1633,"org":1634,"tags":1635,"stars":23,"repoUrl":24,"updatedAt":1644},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1636,1637,1640,1641],{"name":17,"slug":18,"type":15},{"name":1638,"slug":1639,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":1642,"slug":1643,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":1646,"name":1646,"fn":1647,"description":1648,"org":1649,"tags":1650,"stars":23,"repoUrl":24,"updatedAt":1658},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1651,1652,1655],{"name":9,"slug":8,"type":15},{"name":1653,"slug":1654,"type":15},"Quantum Computing","quantum-computing",{"name":1656,"slug":1657,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305]