[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-tao-run-on-kubernetes":3,"mdc-68m5vy-key":34,"related-org-nvidia-tao-run-on-kubernetes":3226,"related-repo-nvidia-tao-run-on-kubernetes":3382},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"tao-run-on-kubernetes","execute TAO jobs on Kubernetes","Kubernetes execution platform — submits TAO container jobs as single-pod k8s Jobs with NVIDIA GPU scheduling. Use when running on EKS \u002F GKE \u002F AKS \u002F on-prem clusters with the NVIDIA GPU Operator installed, or when integrating TAO into an existing k8s-native ML platform.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Automation","automation","tag",{"name":17,"slug":18,"type":15},"Deployment","deployment",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Kubernetes","kubernetes",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:28:34.519037","Apache-2.0",281,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Ftao-run-on-kubernetes","---\nname: tao-run-on-kubernetes\ndescription: Kubernetes execution platform — submits TAO container jobs as single-pod k8s Jobs with NVIDIA GPU scheduling.\n  Use when running on EKS \u002F GKE \u002F AKS \u002F on-prem clusters with the NVIDIA GPU Operator installed, or when integrating TAO\n  into an existing k8s-native ML platform.\nlicense: Apache-2.0\ncompatibility: Requires GPU worker nodes with NVIDIA driver branch 580, CUDA Toolkit 13.0, and NVIDIA Container Toolkit 1.19.0; the nvidia-tao-sdk Python package with the kubernetes extra (pip install 'nvidia-tao-sdk[kubernetes]'); an authenticated cluster; and the NVIDIA GPU Operator or device plugin.\nmetadata:\n  author: NVIDIA Corporation\n  version: \"0.1.0\"\nallowed-tools: Read Bash\ntags:\n- kubernetes\n- k8s\n- gpu\n- compute\n- container\n---\n\n# Kubernetes\n\nSubmits TAO container jobs as Kubernetes Jobs. Works on any cluster reachable via kubeconfig (EKS \u002F GKE \u002F AKS \u002F on-prem) or in-cluster service account (when the SDK runs inside a pod).\n\nSingle-pod by default; opt into multi-node distributed training via `num_nodes > 1` (uses Indexed Job + headless Service, see [Multi-node training](#multi-node-training-distributed) below).\n\n## Preflight\n\nFour checks: GPU host runtime ready, SDK installed, cluster reachable, GPU\nOperator\u002Fdevice plugin present.\n\n```bash\n# 0. GPU node host runtime.\n# Run this on each self-managed GPU worker node or in the node image build.\n# Set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1 only when using managed GPU nodes whose\n# driver\u002Ftoolkit lifecycle is owned by the cloud provider or GPU Operator policy.\nif [ \"${TAO_K8S_SKIP_NODE_RUNTIME_CHECK:-0}\" != \"1\" ]; then\n  TAO_SKILL_BANK_ROOT=\"${TAO_SKILL_BANK_ROOT:-$PWD}\"\n  SETUP_SCRIPT=\"${TAO_SKILL_BANK_ROOT}\u002Fplatform\u002Ftao-setup-nvidia-gpu-host\u002Fscripts\u002Fsetup-nvidia-gpu-host.sh\"\n\n  bash \"$SETUP_SCRIPT\" --backend kubernetes --check-only || {\n    echo \"MISSING: TAO Kubernetes GPU node runtime is not ready.\"\n    echo \"For self-managed GPU nodes, run after user approval:\"\n    echo \"  bash \\\"$SETUP_SCRIPT\\\" --backend kubernetes --install --yes\"\n    echo \"For managed clusters, verify the node image\u002FGPU Operator policy installs driver 580 and toolkit 1.19.0, then set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1.\"\n    exit 1\n  }\nfi\n\n# 1. SDK + kubernetes extra installed.\n# nvidia-tao-sdk is on public PyPI; pin lives in versions.yaml (wheels.tao_sdk_kubernetes).\nPIN=$(\"${TAO_SKILL_BANK_PATH:?}\u002Fscripts\u002Fresolve_versions_key.py\" wheels.tao_sdk_kubernetes)\npython -c \"import tao_sdk\" 2>\u002Fdev\u002Fnull || {\n  echo \"Installing missing Python requirement: $PIN\"\n  python -m pip install \"$PIN\"\n}\npython -c \"import kubernetes\" 2>\u002Fdev\u002Fnull || {\n  echo \"Installing missing Python requirement: $PIN\"\n  python -m pip install \"$PIN\"\n}\npython -c \"import tao_sdk, kubernetes\"\n\n# 2. Cluster reachable (kubeconfig OR in-cluster service account)\npython -c \"from kubernetes import config; config.load_kube_config()\" 2>\u002Fdev\u002Fnull || \\\n  python -c \"from kubernetes import config; config.load_incluster_config()\" 2>\u002Fdev\u002Fnull || {\n    echo \"MISSING: no kubeconfig at ~\u002F.kube\u002Fconfig and not running in a pod.\"\n    echo \"Configure kubectl (e.g., 'aws eks update-kubeconfig --name my-cluster') or set \\$KUBECONFIG.\"\n    exit 1\n  }\n\n# 3. NVIDIA GPU Operator present (soft check — warn if kubectl available, don't fail)\nif command -v kubectl >\u002Fdev\u002Fnull 2>&1; then\n  gpu=$(kubectl get nodes -o jsonpath='{range .items[*]}{.status.allocatable.nvidia\\.com\u002Fgpu}{\"\\n\"}{end}' 2>\u002Fdev\u002Fnull | grep -v '^$' | head -1)\n  if [ -z \"$gpu\" ] || [ \"$gpu\" = \"0\" ]; then\n    echo \"WARN: no nvidia.com\u002Fgpu allocatable on this cluster.\"\n    echo \"Install the NVIDIA GPU Operator before submitting GPU jobs:\"\n    echo \"  https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002Flatest\u002Fgetting-started.html\"\n  fi\nfi\n```\n\nThe GPU node runtime check is mandatory for self-managed nodes. For managed\nclusters where the client is not running on a GPU worker, verify the provider\nnode image or GPU Operator policy and set `TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1`\ninstead of running the installer on the client. The final GPU capacity check is\na warning rather than a hard fail — `kubectl` isn't always installed. The SDK\ndoes a hard guard inside\n`KubernetesSDK.create_job()` that uses the kubernetes Python client to verify\nGPU capacity before submitting.\n\n## Credentials & configuration\n\n- **Kubeconfig** (one of):\n  - `~\u002F.kube\u002Fconfig` — default discovery path\n  - `$KUBECONFIG` — alternate path\n  - In-cluster service account — used when running inside a pod (no kubeconfig needed)\n- **TAO_K8S_NAMESPACE** (optional): default namespace for Job submission. Defaults to `default`.\n- **TAO_K8S_CONTEXT** (optional): kubeconfig context name to switch clusters.\n- **NGC_KEY** (optional): for nvcr.io image pulls. If you've pre-created an image-pull secret in the target namespace, pass its name to `create_job` via the `image_pull_secret` argument.\n- **ACCESS_KEY \u002F SECRET_KEY \u002F S3_BUCKET_NAME \u002F S3_ENDPOINT_URL** (optional): for S3 dataset I\u002FO via the SDK's `inputs`\u002F`outputs` script_runner wrapping.\n\nDo not ask for Brev or SLURM credentials for Kubernetes runs. Ask for\nS3 credentials only when the selected workflow uses `s3:\u002F\u002F` inputs or outputs,\nand ask for model-specific credentials such as `HF_TOKEN` only when the selected\nmodel requires them. Before launch, verify the selected namespace can create\nJobs, dataset\u002Fresult paths are visible from the pod, and PVC\u002Fmounted filesystem\npaths are proven to be mounted into the job container; an agent-host local path\nis not sufficient proof.\n\n## SDK API\n\nK8s is SDK-only — there is no `kubectl`-only launch path. Read\n`tao-skill-bank:tao-run-platform` before drafting `create_job` calls; it covers\n`build_entrypoint`, the shared kwarg contract, monitoring, and `ActionWorkflow`.\n\n```python\nfrom tao_sdk.platforms.kubernetes import KubernetesSDK\n\nsdk = KubernetesSDK()  # auto-detects auth\njob = sdk.create_job(\n    image='nvcr.io\u002Fnvidia\u002Ftao\u002Ftao-toolkit:6.26.3-pyt',\n    command='dino train -e \u002Ftmp\u002Fspec.yaml',\n    gpu_count=1,\n    env_vars={'NGC_KEY': os.environ['NGC_KEY']},\n    inputs={'\u002Fdata\u002Ftrain.json': 's3:\u002F\u002Fbucket\u002Fcoco\u002Ftrain.json'},\n    outputs=['\u002Fresults\u002F'],\n    namespace='tao-jobs',                       # optional override\n    image_pull_secret='ngc-pull-secret',         # optional, pre-created\n    node_selector={'gpu-type': 'h100'},          # optional\n)\n```\n\nThe SDK constructs a `V1Job` with:\n- `spec.template.spec.containers[0]`: the requested image and `command=[\"\u002Fbin\u002Fbash\", \"-c\", \u003Ccommand>]`.\n- `resources.limits[\"nvidia.com\u002Fgpu\"]: \u003Cgpu_count>` — schedules onto GPU nodes via the NVIDIA Device Plugin \u002F GPU Operator.\n- `env_vars` flowed through, plus auto-injected S3\u002FNGC\u002FHF credentials for `script_runner`.\n- `restart_policy=Never` and `backoff_limit=0` — failures surface to the user instead of silently retrying.\n- `ttl_seconds_after_finished=3600` — Job auto-cleans 1 hour after terminal state.\n\n## Status & monitoring\n\n```python\nstatus = sdk.get_job_status(job.id)\n# status.status ∈ {\"Pending\", \"Running\", \"Complete\", \"Error\", \"Canceled\", \"Unknown\"}\n\nlogs = sdk.get_job_logs(job.id, tail=200)  # concatenates logs from all pods of the Job\n\n# For stuck-Pending jobs — replica diagnostics:\nfor r in sdk.get_job_replicas(job.id):\n    issue = r[\"status\"].get(\"readiness_issue\")\n    if issue:\n        print(issue[\"reason\"], issue[\"message\"])\n        # e.g. \"ImagePullBackOff\" \u002F \"Back-off pulling image...\"\n        # e.g. \"Pending\"           \u002F \"0\u002F3 nodes available: 3 Insufficient nvidia.com\u002Fgpu\"\n\n# On failure:\nanalysis = sdk.get_failure_analysis(job.id)\n# {\"err_class\": \"ERR_PROGRAM\" | \"ERR_INFRA\",\n#  \"suggestion\": \"Container OOM-killed. Reduce batch size...\",\n#  \"job_failure_by_node_event\": [{\"node_event_name\": \"OOMKilled\", ...}]}\n```\n\n## Cancel & cleanup\n\n```python\nsdk.cancel_job(job.id)  # delete_namespaced_job with propagation_policy=\"Foreground\"\n```\n\n`ttl_seconds_after_finished=3600` means completed Jobs auto-delete after 1h. To cancel an in-flight Job, `cancel_job` deletes it and its pods immediately.\n\n## GPU Operator dependency\n\nThe SDK refuses to submit GPU jobs to a cluster with no `nvidia.com\u002Fgpu` allocatable. For self-managed clusters, first run the `tao-setup-nvidia-gpu-host` install action on every GPU worker node or bake the same package set into the node image:\n\n```bash\nbash skills\u002Fplatform\u002Ftao-setup-nvidia-gpu-host\u002Fscripts\u002Fsetup-nvidia-gpu-host.sh --backend kubernetes --install --yes\n```\n\nThen install the NVIDIA GPU Operator or device plugin:\n\n```bash\nhelm repo add nvidia https:\u002F\u002Fhelm.ngc.nvidia.com\u002Fnvidia\nhelm repo update\nhelm install --wait gpu-operator -n gpu-operator --create-namespace nvidia\u002Fgpu-operator\n```\n\nFull guide: https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002Flatest\u002Fgetting-started.html\n\n## Multi-node training (distributed)\n\nPass `num_nodes > 1` to `create_job()` to run distributed training across N pods. The SDK provisions:\n\n1. A **headless Service** named after the Job (selector: `job-name=\u003Cjob-name>`, `clusterIP: None`, `publishNotReadyAddresses: true` so pods can rendezvous before they're all Ready).\n2. An **Indexed Job** with `parallelism = completions = num_nodes`, `completionMode: Indexed`. Each pod gets `JOB_COMPLETION_INDEX` injected by k8s automatically (= the node rank).\n3. A **command wrapper** that exports the rendezvous env vars before invoking the user command. Two naming conventions are exported simultaneously:\n\n   | Env var | Value | Read by |\n   |---|---|---|\n   | `WORLD_SIZE` | `num_nodes` | TAO PyTorch container's `nvidia_tao_pytorch\u002Fcore\u002Fentrypoint.py` (uses this to mean *node count*, even though PyTorch's own convention is *total processes*) |\n   | `NUM_GPU_PER_NODE` | `gpu_count` | TAO PyTorch container's entrypoint |\n   | `NNODES` | `num_nodes` | `torchrun` and PyTorch-standard rendezvous |\n   | `NPROC_PER_NODE` | `gpu_count` | `torchrun` |\n   | `NODE_RANK` | `$JOB_COMPLETION_INDEX` | both |\n   | `MASTER_ADDR` | `\u003Cjob-name>-0.\u003Cjob-name>` (pod-0's DNS) | both |\n   | `MASTER_PORT` | `29500` | both (TAO's default) |\n\n   Both naming conventions are set so TAO entrypoints (`dino train`, etc.) and raw `torchrun` commands work without modification.\n\n```python\njob = sdk.create_job(\n    image='nvcr.io\u002Fnvidia\u002Ftao\u002Ftao-toolkit:6.26.3-pyt',\n    command='dino train -e \u002Ftmp\u002Fspec.yaml',  # TAO entrypoint reads spec.train.num_nodes; env vars are wired by the container\n    gpu_count=8,           # GPUs per node\n    num_nodes=4,           # 4 × 8 = 32 GPUs total\n    inputs={'\u002Fdata\u002Ftrain.json': 's3:\u002F\u002Fbucket\u002Fcoco\u002Ftrain.json'},\n    outputs=['\u002Fresults\u002F'],\n)\n```\n\nFor raw `torchrun`-based commands (non-TAO containers):\n\n```python\njob = sdk.create_job(\n    image='nvcr.io\u002Fnvidia\u002Fpytorch:25.08-py3',\n    command='torchrun --nnodes=$NNODES --nproc-per-node=$NPROC_PER_NODE --node-rank=$NODE_RANK '\n            '--master-addr=$MASTER_ADDR --master-port=$MASTER_PORT train.py',\n    gpu_count=8,\n    num_nodes=4,\n)\n```\n\nThe capacity check sums across nodes: `gpu_count × num_nodes` ≤ cluster's allocatable `nvidia.com\u002Fgpu`.\n\n### Cluster requirements for multi-node\n\n- **k8s 1.28+** is required for stable pod hostnames in Indexed Jobs (the `PodIndexLabel` feature). On older clusters the `MASTER_ADDR=\u003Cjob>-0.\u003Csvc>` DNS lookup fails. Verify with `kubectl version`.\n- **Pod-to-pod networking** must be open on port 29500 (PyTorch default; configurable via `MASTER_PORT` env var). Most CNIs (Calico, Cilium, AWS VPC CNI) allow this by default; restrictive NetworkPolicies must be relaxed.\n- **NCCL** in the container talks GPU-to-GPU; if the cluster has multi-NIC nodes or RDMA, set `NCCL_SOCKET_IFNAME` \u002F `NCCL_IB_HCA` via `env_vars`.\n\n### Reference reading\n\n- Kubernetes Indexed Job: \u003Chttps:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fcontrollers\u002Fjob\u002F#completion-mode>\n- Indexed Job for batch ML: \u003Chttps:\u002F\u002Fkubernetes.io\u002Fblog\u002F2022\u002F06\u002F01\u002Findexed-jobs-mpi\u002F>\n- PyTorch distributed (env-var rendezvous): \u003Chttps:\u002F\u002Fpytorch.org\u002Fdocs\u002Fstable\u002Felastic\u002Frun.html>\n- NCCL networking tuning (NCCL_SOCKET_IFNAME, NCCL_IB_HCA): \u003Chttps:\u002F\u002Fdocs.nvidia.com\u002Fdeeplearning\u002Fnccl\u002Fuser-guide\u002Fdocs\u002Fenv.html>\n\n### Kubernetes operator alternatives\n\nFor more sophisticated topologies (gang scheduling, PyTorch elastic \u002F fault-tolerant training, MPI \u002F Horovod, RDMA setup), reach for an operator instead of plain Indexed Job:\n\n- **MPI Operator** — \u003Chttps:\u002F\u002Fgithub.com\u002Fkubeflow\u002Fmpi-operator> — for MPI \u002F Horovod workloads.\n- **Kubeflow Training Operator** (`PyTorchJob`, `TFJob`) — \u003Chttps:\u002F\u002Fwww.kubeflow.org\u002Fdocs\u002Fcomponents\u002Ftraining\u002F> — for elastic PyTorch training with built-in restart logic.\n- **Volcano** — \u003Chttps:\u002F\u002Fvolcano.sh\u002F> — gang scheduling, queues, fair-share. Useful in shared multi-tenant clusters.\n- **Kueue** — \u003Chttps:\u002F\u002Fkueue.sigs.k8s.io\u002F> — quota \u002F queue layer on top of any of the above.\n\nThe TAO SDK's Indexed Job path is intentionally simple and dependency-free; if you need elastic restart or gang scheduling, layer one of these on top and submit jobs through the operator's CRD instead.\n\n## Common error patterns\n\n**`No nvidia.com\u002Fgpu resources allocatable on the cluster`** — the GPU Operator (or NVIDIA Device Plugin) isn't installed. Install per the link above; verify with `kubectl get nodes -o jsonpath='{.items[*].status.allocatable}'`.\n\n**`ImagePullBackOff` \u002F `ErrImagePull`** — the cluster can't pull the image. For nvcr.io: pre-create an image-pull secret in the namespace and pass its name via the `image_pull_secret` argument:\n```bash\nkubectl create secret docker-registry ngc-pull-secret \\\n  --docker-server=nvcr.io \\\n  --docker-username='$oauthtoken' \\\n  --docker-password=$NGC_KEY -n tao-jobs\n```\n\n**Pod stays `Pending` forever** — `get_job_replicas(job_id)` will show the readiness_issue. Common causes: insufficient GPU capacity (`Insufficient nvidia.com\u002Fgpu`), no node matches `node_selector`, missing image-pull secret, or PVC mount failure.\n\n**`OOMKilled` (exit 137)** — container exceeded memory. Reduce batch size, lower max_length, or add a memory request\u002Flimit and target a larger node.\n\n**`CredentialError: Could not authenticate to a Kubernetes cluster`** — neither kubeconfig nor in-cluster auth worked. Run `kubectl get nodes` to verify your config, or set `$KUBECONFIG` to the right path.\n\n## What this skill does NOT support (yet)\n\n- **Elastic \u002F fault-tolerant training.** Indexed Job has `backoff_limit=0` — failures fail the whole training run. For elastic restart (e.g., resume from checkpoint after a node death), use Kubeflow's `PyTorchJob` operator instead.\n- **Gang scheduling.** Indexed Job pods are scheduled independently — no all-or-nothing. Multi-node training will *partially* start if only some pods can be scheduled (rank-0 will hang waiting for peers). For all-or-nothing scheduling on shared clusters, use Volcano or Kueue.\n- **MPI \u002F Horovod.** Use the MPI Operator. The Indexed Job path here is PyTorch-distributed-shaped (env-var rendezvous on `MASTER_ADDR:MASTER_PORT`).\n- **Persistent volumes for shared storage.** S3 only via the script_runner. PVC support is a follow-up.\n- **Auto-creating image-pull secrets from `$NGC_KEY`.** You pre-create the secret in the target namespace and pass the name. K8s namespace conventions vary widely, so we keep secret creation explicit.\n",{"data":35,"body":46},{"name":4,"description":6,"license":26,"compatibility":36,"metadata":37,"allowed-tools":40,"tags":41},"Requires GPU worker nodes with NVIDIA driver branch 580, CUDA Toolkit 13.0, and NVIDIA Container Toolkit 1.19.0; the nvidia-tao-sdk Python package with the kubernetes extra (pip install 'nvidia-tao-sdk[kubernetes]'); an authenticated cluster; and the NVIDIA GPU Operator or device plugin.",{"author":38,"version":39},"NVIDIA Corporation","0.1.0","Read Bash",[22,42,43,44,45],"k8s","gpu","compute","container",{"type":47,"children":48},"root",[49,56,62,85,92,97,1299,1327,1333,1459,1480,1486,1528,1646,1659,1739,1745,1893,1899,1913,1931,1937,1958,1995,2000,2091,2103,2109,2129,2481,2547,2559,2620,2639,2646,2731,2737,2784,2790,2795,2883,2888,2894,2915,2943,3040,3081,3097,3126,3132,3220],{"type":50,"tag":51,"props":52,"children":53},"element","h1",{"id":22},[54],{"type":55,"value":21},"text",{"type":50,"tag":57,"props":58,"children":59},"p",{},[60],{"type":55,"value":61},"Submits TAO container jobs as Kubernetes Jobs. Works on any cluster reachable via kubeconfig (EKS \u002F GKE \u002F AKS \u002F on-prem) or in-cluster service account (when the SDK runs inside a pod).",{"type":50,"tag":57,"props":63,"children":64},{},[65,67,74,76,83],{"type":55,"value":66},"Single-pod by default; opt into multi-node distributed training via ",{"type":50,"tag":68,"props":69,"children":71},"code",{"className":70},[],[72],{"type":55,"value":73},"num_nodes > 1",{"type":55,"value":75}," (uses Indexed Job + headless Service, see ",{"type":50,"tag":77,"props":78,"children":80},"a",{"href":79},"#multi-node-training-distributed",[81],{"type":55,"value":82},"Multi-node training",{"type":55,"value":84}," below).",{"type":50,"tag":86,"props":87,"children":89},"h2",{"id":88},"preflight",[90],{"type":55,"value":91},"Preflight",{"type":50,"tag":57,"props":93,"children":94},{},[95],{"type":55,"value":96},"Four checks: GPU host runtime ready, SDK installed, cluster reachable, GPU\nOperator\u002Fdevice plugin present.",{"type":50,"tag":98,"props":99,"children":104},"pre",{"className":100,"code":101,"language":102,"meta":103,"style":103},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# 0. GPU node host runtime.\n# Run this on each self-managed GPU worker node or in the node image build.\n# Set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1 only when using managed GPU nodes whose\n# driver\u002Ftoolkit lifecycle is owned by the cloud provider or GPU Operator policy.\nif [ \"${TAO_K8S_SKIP_NODE_RUNTIME_CHECK:-0}\" != \"1\" ]; then\n  TAO_SKILL_BANK_ROOT=\"${TAO_SKILL_BANK_ROOT:-$PWD}\"\n  SETUP_SCRIPT=\"${TAO_SKILL_BANK_ROOT}\u002Fplatform\u002Ftao-setup-nvidia-gpu-host\u002Fscripts\u002Fsetup-nvidia-gpu-host.sh\"\n\n  bash \"$SETUP_SCRIPT\" --backend kubernetes --check-only || {\n    echo \"MISSING: TAO Kubernetes GPU node runtime is not ready.\"\n    echo \"For self-managed GPU nodes, run after user approval:\"\n    echo \"  bash \\\"$SETUP_SCRIPT\\\" --backend kubernetes --install --yes\"\n    echo \"For managed clusters, verify the node image\u002FGPU Operator policy installs driver 580 and toolkit 1.19.0, then set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1.\"\n    exit 1\n  }\nfi\n\n# 1. SDK + kubernetes extra installed.\n# nvidia-tao-sdk is on public PyPI; pin lives in versions.yaml (wheels.tao_sdk_kubernetes).\nPIN=$(\"${TAO_SKILL_BANK_PATH:?}\u002Fscripts\u002Fresolve_versions_key.py\" wheels.tao_sdk_kubernetes)\npython -c \"import tao_sdk\" 2>\u002Fdev\u002Fnull || {\n  echo \"Installing missing Python requirement: $PIN\"\n  python -m pip install \"$PIN\"\n}\npython -c \"import kubernetes\" 2>\u002Fdev\u002Fnull || {\n  echo \"Installing missing Python requirement: $PIN\"\n  python -m pip install \"$PIN\"\n}\npython -c \"import tao_sdk, kubernetes\"\n\n# 2. Cluster reachable (kubeconfig OR in-cluster service account)\npython -c \"from kubernetes import config; config.load_kube_config()\" 2>\u002Fdev\u002Fnull || \\\n  python -c \"from kubernetes import config; config.load_incluster_config()\" 2>\u002Fdev\u002Fnull || {\n    echo \"MISSING: no kubeconfig at ~\u002F.kube\u002Fconfig and not running in a pod.\"\n    echo \"Configure kubectl (e.g., 'aws eks update-kubeconfig --name my-cluster') or set \\$KUBECONFIG.\"\n    exit 1\n  }\n\n# 3. NVIDIA GPU Operator present (soft check — warn if kubectl available, don't fail)\nif command -v kubectl >\u002Fdev\u002Fnull 2>&1; then\n  gpu=$(kubectl get nodes -o jsonpath='{range .items[*]}{.status.allocatable.nvidia\\.com\u002Fgpu}{\"\\n\"}{end}' 2>\u002Fdev\u002Fnull | grep -v '^$' | head -1)\n  if [ -z \"$gpu\" ] || [ \"$gpu\" = \"0\" ]; then\n    echo \"WARN: no nvidia.com\u002Fgpu allocatable on this cluster.\"\n    echo \"Install the NVIDIA GPU Operator before submitting GPU jobs:\"\n    echo \"  https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002Flatest\u002Fgetting-started.html\"\n  fi\nfi\n","bash","",[105],{"type":50,"tag":68,"props":106,"children":107},{"__ignoreMap":103},[108,120,129,138,147,220,258,294,304,352,375,396,427,448,463,472,481,489,498,507,555,600,627,663,672,713,737,769,777,802,810,819,861,902,923,954,966,974,982,991,1032,1138,1219,1240,1261,1282,1291],{"type":50,"tag":109,"props":110,"children":113},"span",{"class":111,"line":112},"line",1,[114],{"type":50,"tag":109,"props":115,"children":117},{"style":116},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[118],{"type":55,"value":119},"# 0. GPU node host runtime.\n",{"type":50,"tag":109,"props":121,"children":123},{"class":111,"line":122},2,[124],{"type":50,"tag":109,"props":125,"children":126},{"style":116},[127],{"type":55,"value":128},"# Run this on each self-managed GPU worker node or in the node image build.\n",{"type":50,"tag":109,"props":130,"children":132},{"class":111,"line":131},3,[133],{"type":50,"tag":109,"props":134,"children":135},{"style":116},[136],{"type":55,"value":137},"# Set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1 only when using managed GPU nodes whose\n",{"type":50,"tag":109,"props":139,"children":141},{"class":111,"line":140},4,[142],{"type":50,"tag":109,"props":143,"children":144},{"style":116},[145],{"type":55,"value":146},"# driver\u002Ftoolkit lifecycle is owned by the cloud provider or GPU Operator policy.\n",{"type":50,"tag":109,"props":148,"children":150},{"class":111,"line":149},5,[151,157,163,168,174,179,184,189,194,199,205,210,215],{"type":50,"tag":109,"props":152,"children":154},{"style":153},"--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic",[155],{"type":55,"value":156},"if",{"type":50,"tag":109,"props":158,"children":160},{"style":159},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[161],{"type":55,"value":162}," [",{"type":50,"tag":109,"props":164,"children":165},{"style":159},[166],{"type":55,"value":167}," \"${",{"type":50,"tag":109,"props":169,"children":171},{"style":170},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[172],{"type":55,"value":173},"TAO_K8S_SKIP_NODE_RUNTIME_CHECK",{"type":50,"tag":109,"props":175,"children":176},{"style":159},[177],{"type":55,"value":178},":-",{"type":50,"tag":109,"props":180,"children":181},{"style":170},[182],{"type":55,"value":183},"0",{"type":50,"tag":109,"props":185,"children":186},{"style":159},[187],{"type":55,"value":188},"}\"",{"type":50,"tag":109,"props":190,"children":191},{"style":159},[192],{"type":55,"value":193}," !=",{"type":50,"tag":109,"props":195,"children":196},{"style":159},[197],{"type":55,"value":198}," \"",{"type":50,"tag":109,"props":200,"children":202},{"style":201},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[203],{"type":55,"value":204},"1",{"type":50,"tag":109,"props":206,"children":207},{"style":159},[208],{"type":55,"value":209},"\"",{"type":50,"tag":109,"props":211,"children":212},{"style":159},[213],{"type":55,"value":214}," ];",{"type":50,"tag":109,"props":216,"children":217},{"style":153},[218],{"type":55,"value":219}," then\n",{"type":50,"tag":109,"props":221,"children":223},{"class":111,"line":222},6,[224,229,234,239,244,248,253],{"type":50,"tag":109,"props":225,"children":226},{"style":170},[227],{"type":55,"value":228},"  TAO_SKILL_BANK_ROOT",{"type":50,"tag":109,"props":230,"children":231},{"style":159},[232],{"type":55,"value":233},"=",{"type":50,"tag":109,"props":235,"children":236},{"style":159},[237],{"type":55,"value":238},"\"${",{"type":50,"tag":109,"props":240,"children":241},{"style":170},[242],{"type":55,"value":243},"TAO_SKILL_BANK_ROOT",{"type":50,"tag":109,"props":245,"children":246},{"style":159},[247],{"type":55,"value":178},{"type":50,"tag":109,"props":249,"children":250},{"style":170},[251],{"type":55,"value":252},"$PWD",{"type":50,"tag":109,"props":254,"children":255},{"style":159},[256],{"type":55,"value":257},"}\"\n",{"type":50,"tag":109,"props":259,"children":261},{"class":111,"line":260},7,[262,267,271,275,279,284,289],{"type":50,"tag":109,"props":263,"children":264},{"style":170},[265],{"type":55,"value":266},"  SETUP_SCRIPT",{"type":50,"tag":109,"props":268,"children":269},{"style":159},[270],{"type":55,"value":233},{"type":50,"tag":109,"props":272,"children":273},{"style":159},[274],{"type":55,"value":238},{"type":50,"tag":109,"props":276,"children":277},{"style":170},[278],{"type":55,"value":243},{"type":50,"tag":109,"props":280,"children":281},{"style":159},[282],{"type":55,"value":283},"}",{"type":50,"tag":109,"props":285,"children":286},{"style":201},[287],{"type":55,"value":288},"\u002Fplatform\u002Ftao-setup-nvidia-gpu-host\u002Fscripts\u002Fsetup-nvidia-gpu-host.sh",{"type":50,"tag":109,"props":290,"children":291},{"style":159},[292],{"type":55,"value":293},"\"\n",{"type":50,"tag":109,"props":295,"children":297},{"class":111,"line":296},8,[298],{"type":50,"tag":109,"props":299,"children":301},{"emptyLinePlaceholder":300},true,[302],{"type":55,"value":303},"\n",{"type":50,"tag":109,"props":305,"children":307},{"class":111,"line":306},9,[308,314,318,323,327,332,337,342,347],{"type":50,"tag":109,"props":309,"children":311},{"style":310},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[312],{"type":55,"value":313},"  bash",{"type":50,"tag":109,"props":315,"children":316},{"style":159},[317],{"type":55,"value":198},{"type":50,"tag":109,"props":319,"children":320},{"style":170},[321],{"type":55,"value":322},"$SETUP_SCRIPT",{"type":50,"tag":109,"props":324,"children":325},{"style":159},[326],{"type":55,"value":209},{"type":50,"tag":109,"props":328,"children":329},{"style":201},[330],{"type":55,"value":331}," --backend",{"type":50,"tag":109,"props":333,"children":334},{"style":201},[335],{"type":55,"value":336}," kubernetes",{"type":50,"tag":109,"props":338,"children":339},{"style":201},[340],{"type":55,"value":341}," --check-only",{"type":50,"tag":109,"props":343,"children":344},{"style":159},[345],{"type":55,"value":346}," ||",{"type":50,"tag":109,"props":348,"children":349},{"style":159},[350],{"type":55,"value":351}," {\n",{"type":50,"tag":109,"props":353,"children":355},{"class":111,"line":354},10,[356,362,366,371],{"type":50,"tag":109,"props":357,"children":359},{"style":358},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[360],{"type":55,"value":361},"    echo",{"type":50,"tag":109,"props":363,"children":364},{"style":159},[365],{"type":55,"value":198},{"type":50,"tag":109,"props":367,"children":368},{"style":201},[369],{"type":55,"value":370},"MISSING: TAO Kubernetes GPU node runtime is not ready.",{"type":50,"tag":109,"props":372,"children":373},{"style":159},[374],{"type":55,"value":293},{"type":50,"tag":109,"props":376,"children":378},{"class":111,"line":377},11,[379,383,387,392],{"type":50,"tag":109,"props":380,"children":381},{"style":358},[382],{"type":55,"value":361},{"type":50,"tag":109,"props":384,"children":385},{"style":159},[386],{"type":55,"value":198},{"type":50,"tag":109,"props":388,"children":389},{"style":201},[390],{"type":55,"value":391},"For self-managed GPU nodes, run after user approval:",{"type":50,"tag":109,"props":393,"children":394},{"style":159},[395],{"type":55,"value":293},{"type":50,"tag":109,"props":397,"children":399},{"class":111,"line":398},12,[400,404,408,413,418,423],{"type":50,"tag":109,"props":401,"children":402},{"style":358},[403],{"type":55,"value":361},{"type":50,"tag":109,"props":405,"children":406},{"style":159},[407],{"type":55,"value":198},{"type":50,"tag":109,"props":409,"children":410},{"style":201},[411],{"type":55,"value":412},"  bash ",{"type":50,"tag":109,"props":414,"children":415},{"style":170},[416],{"type":55,"value":417},"\\\"$SETUP_SCRIPT\\\"",{"type":50,"tag":109,"props":419,"children":420},{"style":201},[421],{"type":55,"value":422}," --backend kubernetes --install --yes",{"type":50,"tag":109,"props":424,"children":425},{"style":159},[426],{"type":55,"value":293},{"type":50,"tag":109,"props":428,"children":430},{"class":111,"line":429},13,[431,435,439,444],{"type":50,"tag":109,"props":432,"children":433},{"style":358},[434],{"type":55,"value":361},{"type":50,"tag":109,"props":436,"children":437},{"style":159},[438],{"type":55,"value":198},{"type":50,"tag":109,"props":440,"children":441},{"style":201},[442],{"type":55,"value":443},"For managed clusters, verify the node image\u002FGPU Operator policy installs driver 580 and toolkit 1.19.0, then set TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1.",{"type":50,"tag":109,"props":445,"children":446},{"style":159},[447],{"type":55,"value":293},{"type":50,"tag":109,"props":449,"children":451},{"class":111,"line":450},14,[452,457],{"type":50,"tag":109,"props":453,"children":454},{"style":358},[455],{"type":55,"value":456},"    exit",{"type":50,"tag":109,"props":458,"children":460},{"style":459},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[461],{"type":55,"value":462}," 1\n",{"type":50,"tag":109,"props":464,"children":466},{"class":111,"line":465},15,[467],{"type":50,"tag":109,"props":468,"children":469},{"style":159},[470],{"type":55,"value":471},"  }\n",{"type":50,"tag":109,"props":473,"children":475},{"class":111,"line":474},16,[476],{"type":50,"tag":109,"props":477,"children":478},{"style":153},[479],{"type":55,"value":480},"fi\n",{"type":50,"tag":109,"props":482,"children":484},{"class":111,"line":483},17,[485],{"type":50,"tag":109,"props":486,"children":487},{"emptyLinePlaceholder":300},[488],{"type":55,"value":303},{"type":50,"tag":109,"props":490,"children":492},{"class":111,"line":491},18,[493],{"type":50,"tag":109,"props":494,"children":495},{"style":116},[496],{"type":55,"value":497},"# 1. SDK + kubernetes extra installed.\n",{"type":50,"tag":109,"props":499,"children":501},{"class":111,"line":500},19,[502],{"type":50,"tag":109,"props":503,"children":504},{"style":116},[505],{"type":55,"value":506},"# nvidia-tao-sdk is on public PyPI; pin lives in versions.yaml (wheels.tao_sdk_kubernetes).\n",{"type":50,"tag":109,"props":508,"children":510},{"class":111,"line":509},20,[511,516,521,525,530,535,540,545,550],{"type":50,"tag":109,"props":512,"children":513},{"style":170},[514],{"type":55,"value":515},"PIN",{"type":50,"tag":109,"props":517,"children":518},{"style":159},[519],{"type":55,"value":520},"=$(",{"type":50,"tag":109,"props":522,"children":523},{"style":310},[524],{"type":55,"value":209},{"type":50,"tag":109,"props":526,"children":527},{"style":159},[528],{"type":55,"value":529},"${",{"type":50,"tag":109,"props":531,"children":532},{"style":170},[533],{"type":55,"value":534},"TAO_SKILL_BANK_PATH",{"type":50,"tag":109,"props":536,"children":537},{"style":159},[538],{"type":55,"value":539},":?}",{"type":50,"tag":109,"props":541,"children":542},{"style":310},[543],{"type":55,"value":544},"\u002Fscripts\u002Fresolve_versions_key.py\"",{"type":50,"tag":109,"props":546,"children":547},{"style":201},[548],{"type":55,"value":549}," wheels.tao_sdk_kubernetes",{"type":50,"tag":109,"props":551,"children":552},{"style":159},[553],{"type":55,"value":554},")\n",{"type":50,"tag":109,"props":556,"children":558},{"class":111,"line":557},21,[559,564,569,573,578,582,587,592,596],{"type":50,"tag":109,"props":560,"children":561},{"style":310},[562],{"type":55,"value":563},"python",{"type":50,"tag":109,"props":565,"children":566},{"style":201},[567],{"type":55,"value":568}," -c",{"type":50,"tag":109,"props":570,"children":571},{"style":159},[572],{"type":55,"value":198},{"type":50,"tag":109,"props":574,"children":575},{"style":201},[576],{"type":55,"value":577},"import tao_sdk",{"type":50,"tag":109,"props":579,"children":580},{"style":159},[581],{"type":55,"value":209},{"type":50,"tag":109,"props":583,"children":584},{"style":159},[585],{"type":55,"value":586}," 2>",{"type":50,"tag":109,"props":588,"children":589},{"style":201},[590],{"type":55,"value":591},"\u002Fdev\u002Fnull",{"type":50,"tag":109,"props":593,"children":594},{"style":159},[595],{"type":55,"value":346},{"type":50,"tag":109,"props":597,"children":598},{"style":159},[599],{"type":55,"value":351},{"type":50,"tag":109,"props":601,"children":603},{"class":111,"line":602},22,[604,609,613,618,623],{"type":50,"tag":109,"props":605,"children":606},{"style":358},[607],{"type":55,"value":608},"  echo",{"type":50,"tag":109,"props":610,"children":611},{"style":159},[612],{"type":55,"value":198},{"type":50,"tag":109,"props":614,"children":615},{"style":201},[616],{"type":55,"value":617},"Installing missing Python requirement: ",{"type":50,"tag":109,"props":619,"children":620},{"style":170},[621],{"type":55,"value":622},"$PIN",{"type":50,"tag":109,"props":624,"children":625},{"style":159},[626],{"type":55,"value":293},{"type":50,"tag":109,"props":628,"children":630},{"class":111,"line":629},23,[631,636,641,646,651,655,659],{"type":50,"tag":109,"props":632,"children":633},{"style":310},[634],{"type":55,"value":635},"  python",{"type":50,"tag":109,"props":637,"children":638},{"style":201},[639],{"type":55,"value":640}," -m",{"type":50,"tag":109,"props":642,"children":643},{"style":201},[644],{"type":55,"value":645}," pip",{"type":50,"tag":109,"props":647,"children":648},{"style":201},[649],{"type":55,"value":650}," install",{"type":50,"tag":109,"props":652,"children":653},{"style":159},[654],{"type":55,"value":198},{"type":50,"tag":109,"props":656,"children":657},{"style":170},[658],{"type":55,"value":622},{"type":50,"tag":109,"props":660,"children":661},{"style":159},[662],{"type":55,"value":293},{"type":50,"tag":109,"props":664,"children":666},{"class":111,"line":665},24,[667],{"type":50,"tag":109,"props":668,"children":669},{"style":159},[670],{"type":55,"value":671},"}\n",{"type":50,"tag":109,"props":673,"children":675},{"class":111,"line":674},25,[676,680,684,688,693,697,701,705,709],{"type":50,"tag":109,"props":677,"children":678},{"style":310},[679],{"type":55,"value":563},{"type":50,"tag":109,"props":681,"children":682},{"style":201},[683],{"type":55,"value":568},{"type":50,"tag":109,"props":685,"children":686},{"style":159},[687],{"type":55,"value":198},{"type":50,"tag":109,"props":689,"children":690},{"style":201},[691],{"type":55,"value":692},"import kubernetes",{"type":50,"tag":109,"props":694,"children":695},{"style":159},[696],{"type":55,"value":209},{"type":50,"tag":109,"props":698,"children":699},{"style":159},[700],{"type":55,"value":586},{"type":50,"tag":109,"props":702,"children":703},{"style":201},[704],{"type":55,"value":591},{"type":50,"tag":109,"props":706,"children":707},{"style":159},[708],{"type":55,"value":346},{"type":50,"tag":109,"props":710,"children":711},{"style":159},[712],{"type":55,"value":351},{"type":50,"tag":109,"props":714,"children":716},{"class":111,"line":715},26,[717,721,725,729,733],{"type":50,"tag":109,"props":718,"children":719},{"style":358},[720],{"type":55,"value":608},{"type":50,"tag":109,"props":722,"children":723},{"style":159},[724],{"type":55,"value":198},{"type":50,"tag":109,"props":726,"children":727},{"style":201},[728],{"type":55,"value":617},{"type":50,"tag":109,"props":730,"children":731},{"style":170},[732],{"type":55,"value":622},{"type":50,"tag":109,"props":734,"children":735},{"style":159},[736],{"type":55,"value":293},{"type":50,"tag":109,"props":738,"children":740},{"class":111,"line":739},27,[741,745,749,753,757,761,765],{"type":50,"tag":109,"props":742,"children":743},{"style":310},[744],{"type":55,"value":635},{"type":50,"tag":109,"props":746,"children":747},{"style":201},[748],{"type":55,"value":640},{"type":50,"tag":109,"props":750,"children":751},{"style":201},[752],{"type":55,"value":645},{"type":50,"tag":109,"props":754,"children":755},{"style":201},[756],{"type":55,"value":650},{"type":50,"tag":109,"props":758,"children":759},{"style":159},[760],{"type":55,"value":198},{"type":50,"tag":109,"props":762,"children":763},{"style":170},[764],{"type":55,"value":622},{"type":50,"tag":109,"props":766,"children":767},{"style":159},[768],{"type":55,"value":293},{"type":50,"tag":109,"props":770,"children":772},{"class":111,"line":771},28,[773],{"type":50,"tag":109,"props":774,"children":775},{"style":159},[776],{"type":55,"value":671},{"type":50,"tag":109,"props":778,"children":780},{"class":111,"line":779},29,[781,785,789,793,798],{"type":50,"tag":109,"props":782,"children":783},{"style":310},[784],{"type":55,"value":563},{"type":50,"tag":109,"props":786,"children":787},{"style":201},[788],{"type":55,"value":568},{"type":50,"tag":109,"props":790,"children":791},{"style":159},[792],{"type":55,"value":198},{"type":50,"tag":109,"props":794,"children":795},{"style":201},[796],{"type":55,"value":797},"import tao_sdk, kubernetes",{"type":50,"tag":109,"props":799,"children":800},{"style":159},[801],{"type":55,"value":293},{"type":50,"tag":109,"props":803,"children":805},{"class":111,"line":804},30,[806],{"type":50,"tag":109,"props":807,"children":808},{"emptyLinePlaceholder":300},[809],{"type":55,"value":303},{"type":50,"tag":109,"props":811,"children":813},{"class":111,"line":812},31,[814],{"type":50,"tag":109,"props":815,"children":816},{"style":116},[817],{"type":55,"value":818},"# 2. Cluster reachable (kubeconfig OR in-cluster service account)\n",{"type":50,"tag":109,"props":820,"children":822},{"class":111,"line":821},32,[823,827,831,835,840,844,848,852,856],{"type":50,"tag":109,"props":824,"children":825},{"style":310},[826],{"type":55,"value":563},{"type":50,"tag":109,"props":828,"children":829},{"style":201},[830],{"type":55,"value":568},{"type":50,"tag":109,"props":832,"children":833},{"style":159},[834],{"type":55,"value":198},{"type":50,"tag":109,"props":836,"children":837},{"style":201},[838],{"type":55,"value":839},"from kubernetes import config; config.load_kube_config()",{"type":50,"tag":109,"props":841,"children":842},{"style":159},[843],{"type":55,"value":209},{"type":50,"tag":109,"props":845,"children":846},{"style":159},[847],{"type":55,"value":586},{"type":50,"tag":109,"props":849,"children":850},{"style":201},[851],{"type":55,"value":591},{"type":50,"tag":109,"props":853,"children":854},{"style":159},[855],{"type":55,"value":346},{"type":50,"tag":109,"props":857,"children":858},{"style":170},[859],{"type":55,"value":860}," \\\n",{"type":50,"tag":109,"props":862,"children":864},{"class":111,"line":863},33,[865,869,873,877,882,886,890,894,898],{"type":50,"tag":109,"props":866,"children":867},{"style":310},[868],{"type":55,"value":635},{"type":50,"tag":109,"props":870,"children":871},{"style":201},[872],{"type":55,"value":568},{"type":50,"tag":109,"props":874,"children":875},{"style":159},[876],{"type":55,"value":198},{"type":50,"tag":109,"props":878,"children":879},{"style":201},[880],{"type":55,"value":881},"from kubernetes import config; config.load_incluster_config()",{"type":50,"tag":109,"props":883,"children":884},{"style":159},[885],{"type":55,"value":209},{"type":50,"tag":109,"props":887,"children":888},{"style":159},[889],{"type":55,"value":586},{"type":50,"tag":109,"props":891,"children":892},{"style":201},[893],{"type":55,"value":591},{"type":50,"tag":109,"props":895,"children":896},{"style":159},[897],{"type":55,"value":346},{"type":50,"tag":109,"props":899,"children":900},{"style":159},[901],{"type":55,"value":351},{"type":50,"tag":109,"props":903,"children":905},{"class":111,"line":904},34,[906,910,914,919],{"type":50,"tag":109,"props":907,"children":908},{"style":358},[909],{"type":55,"value":361},{"type":50,"tag":109,"props":911,"children":912},{"style":159},[913],{"type":55,"value":198},{"type":50,"tag":109,"props":915,"children":916},{"style":201},[917],{"type":55,"value":918},"MISSING: no kubeconfig at ~\u002F.kube\u002Fconfig and not running in a pod.",{"type":50,"tag":109,"props":920,"children":921},{"style":159},[922],{"type":55,"value":293},{"type":50,"tag":109,"props":924,"children":926},{"class":111,"line":925},35,[927,931,935,940,945,950],{"type":50,"tag":109,"props":928,"children":929},{"style":358},[930],{"type":55,"value":361},{"type":50,"tag":109,"props":932,"children":933},{"style":159},[934],{"type":55,"value":198},{"type":50,"tag":109,"props":936,"children":937},{"style":201},[938],{"type":55,"value":939},"Configure kubectl (e.g., 'aws eks update-kubeconfig --name my-cluster') or set ",{"type":50,"tag":109,"props":941,"children":942},{"style":170},[943],{"type":55,"value":944},"\\$",{"type":50,"tag":109,"props":946,"children":947},{"style":201},[948],{"type":55,"value":949},"KUBECONFIG.",{"type":50,"tag":109,"props":951,"children":952},{"style":159},[953],{"type":55,"value":293},{"type":50,"tag":109,"props":955,"children":957},{"class":111,"line":956},36,[958,962],{"type":50,"tag":109,"props":959,"children":960},{"style":358},[961],{"type":55,"value":456},{"type":50,"tag":109,"props":963,"children":964},{"style":459},[965],{"type":55,"value":462},{"type":50,"tag":109,"props":967,"children":969},{"class":111,"line":968},37,[970],{"type":50,"tag":109,"props":971,"children":972},{"style":159},[973],{"type":55,"value":471},{"type":50,"tag":109,"props":975,"children":977},{"class":111,"line":976},38,[978],{"type":50,"tag":109,"props":979,"children":980},{"emptyLinePlaceholder":300},[981],{"type":55,"value":303},{"type":50,"tag":109,"props":983,"children":985},{"class":111,"line":984},39,[986],{"type":50,"tag":109,"props":987,"children":988},{"style":116},[989],{"type":55,"value":990},"# 3. NVIDIA GPU Operator present (soft check — warn if kubectl available, don't fail)\n",{"type":50,"tag":109,"props":992,"children":994},{"class":111,"line":993},40,[995,999,1004,1009,1014,1019,1023,1028],{"type":50,"tag":109,"props":996,"children":997},{"style":153},[998],{"type":55,"value":156},{"type":50,"tag":109,"props":1000,"children":1001},{"style":358},[1002],{"type":55,"value":1003}," command",{"type":50,"tag":109,"props":1005,"children":1006},{"style":201},[1007],{"type":55,"value":1008}," -v",{"type":50,"tag":109,"props":1010,"children":1011},{"style":201},[1012],{"type":55,"value":1013}," kubectl",{"type":50,"tag":109,"props":1015,"children":1016},{"style":159},[1017],{"type":55,"value":1018}," >",{"type":50,"tag":109,"props":1020,"children":1021},{"style":201},[1022],{"type":55,"value":591},{"type":50,"tag":109,"props":1024,"children":1025},{"style":159},[1026],{"type":55,"value":1027}," 2>&1;",{"type":50,"tag":109,"props":1029,"children":1030},{"style":153},[1031],{"type":55,"value":219},{"type":50,"tag":109,"props":1033,"children":1035},{"class":111,"line":1034},41,[1036,1041,1045,1050,1055,1060,1065,1070,1075,1080,1084,1088,1092,1097,1102,1106,1111,1116,1120,1124,1129,1134],{"type":50,"tag":109,"props":1037,"children":1038},{"style":170},[1039],{"type":55,"value":1040},"  gpu",{"type":50,"tag":109,"props":1042,"children":1043},{"style":159},[1044],{"type":55,"value":520},{"type":50,"tag":109,"props":1046,"children":1047},{"style":310},[1048],{"type":55,"value":1049},"kubectl",{"type":50,"tag":109,"props":1051,"children":1052},{"style":201},[1053],{"type":55,"value":1054}," get",{"type":50,"tag":109,"props":1056,"children":1057},{"style":201},[1058],{"type":55,"value":1059}," nodes",{"type":50,"tag":109,"props":1061,"children":1062},{"style":201},[1063],{"type":55,"value":1064}," -o",{"type":50,"tag":109,"props":1066,"children":1067},{"style":201},[1068],{"type":55,"value":1069}," jsonpath=",{"type":50,"tag":109,"props":1071,"children":1072},{"style":159},[1073],{"type":55,"value":1074},"'",{"type":50,"tag":109,"props":1076,"children":1077},{"style":201},[1078],{"type":55,"value":1079},"{range .items[*]}{.status.allocatable.nvidia\\.com\u002Fgpu}{\"\\n\"}{end}",{"type":50,"tag":109,"props":1081,"children":1082},{"style":159},[1083],{"type":55,"value":1074},{"type":50,"tag":109,"props":1085,"children":1086},{"style":159},[1087],{"type":55,"value":586},{"type":50,"tag":109,"props":1089,"children":1090},{"style":201},[1091],{"type":55,"value":591},{"type":50,"tag":109,"props":1093,"children":1094},{"style":159},[1095],{"type":55,"value":1096}," |",{"type":50,"tag":109,"props":1098,"children":1099},{"style":310},[1100],{"type":55,"value":1101}," grep",{"type":50,"tag":109,"props":1103,"children":1104},{"style":201},[1105],{"type":55,"value":1008},{"type":50,"tag":109,"props":1107,"children":1108},{"style":159},[1109],{"type":55,"value":1110}," '",{"type":50,"tag":109,"props":1112,"children":1113},{"style":201},[1114],{"type":55,"value":1115},"^$",{"type":50,"tag":109,"props":1117,"children":1118},{"style":159},[1119],{"type":55,"value":1074},{"type":50,"tag":109,"props":1121,"children":1122},{"style":159},[1123],{"type":55,"value":1096},{"type":50,"tag":109,"props":1125,"children":1126},{"style":310},[1127],{"type":55,"value":1128}," head",{"type":50,"tag":109,"props":1130,"children":1131},{"style":201},[1132],{"type":55,"value":1133}," -1",{"type":50,"tag":109,"props":1135,"children":1136},{"style":159},[1137],{"type":55,"value":554},{"type":50,"tag":109,"props":1139,"children":1141},{"class":111,"line":1140},42,[1142,1147,1151,1156,1160,1165,1169,1174,1178,1182,1186,1190,1194,1199,1203,1207,1211,1215],{"type":50,"tag":109,"props":1143,"children":1144},{"style":153},[1145],{"type":55,"value":1146},"  if",{"type":50,"tag":109,"props":1148,"children":1149},{"style":159},[1150],{"type":55,"value":162},{"type":50,"tag":109,"props":1152,"children":1153},{"style":159},[1154],{"type":55,"value":1155}," -z",{"type":50,"tag":109,"props":1157,"children":1158},{"style":159},[1159],{"type":55,"value":198},{"type":50,"tag":109,"props":1161,"children":1162},{"style":170},[1163],{"type":55,"value":1164},"$gpu",{"type":50,"tag":109,"props":1166,"children":1167},{"style":159},[1168],{"type":55,"value":209},{"type":50,"tag":109,"props":1170,"children":1171},{"style":159},[1172],{"type":55,"value":1173}," ]",{"type":50,"tag":109,"props":1175,"children":1176},{"style":159},[1177],{"type":55,"value":346},{"type":50,"tag":109,"props":1179,"children":1180},{"style":159},[1181],{"type":55,"value":162},{"type":50,"tag":109,"props":1183,"children":1184},{"style":159},[1185],{"type":55,"value":198},{"type":50,"tag":109,"props":1187,"children":1188},{"style":170},[1189],{"type":55,"value":1164},{"type":50,"tag":109,"props":1191,"children":1192},{"style":159},[1193],{"type":55,"value":209},{"type":50,"tag":109,"props":1195,"children":1196},{"style":159},[1197],{"type":55,"value":1198}," =",{"type":50,"tag":109,"props":1200,"children":1201},{"style":159},[1202],{"type":55,"value":198},{"type":50,"tag":109,"props":1204,"children":1205},{"style":201},[1206],{"type":55,"value":183},{"type":50,"tag":109,"props":1208,"children":1209},{"style":159},[1210],{"type":55,"value":209},{"type":50,"tag":109,"props":1212,"children":1213},{"style":159},[1214],{"type":55,"value":214},{"type":50,"tag":109,"props":1216,"children":1217},{"style":153},[1218],{"type":55,"value":219},{"type":50,"tag":109,"props":1220,"children":1222},{"class":111,"line":1221},43,[1223,1227,1231,1236],{"type":50,"tag":109,"props":1224,"children":1225},{"style":358},[1226],{"type":55,"value":361},{"type":50,"tag":109,"props":1228,"children":1229},{"style":159},[1230],{"type":55,"value":198},{"type":50,"tag":109,"props":1232,"children":1233},{"style":201},[1234],{"type":55,"value":1235},"WARN: no nvidia.com\u002Fgpu allocatable on this cluster.",{"type":50,"tag":109,"props":1237,"children":1238},{"style":159},[1239],{"type":55,"value":293},{"type":50,"tag":109,"props":1241,"children":1243},{"class":111,"line":1242},44,[1244,1248,1252,1257],{"type":50,"tag":109,"props":1245,"children":1246},{"style":358},[1247],{"type":55,"value":361},{"type":50,"tag":109,"props":1249,"children":1250},{"style":159},[1251],{"type":55,"value":198},{"type":50,"tag":109,"props":1253,"children":1254},{"style":201},[1255],{"type":55,"value":1256},"Install the NVIDIA GPU Operator before submitting GPU jobs:",{"type":50,"tag":109,"props":1258,"children":1259},{"style":159},[1260],{"type":55,"value":293},{"type":50,"tag":109,"props":1262,"children":1264},{"class":111,"line":1263},45,[1265,1269,1273,1278],{"type":50,"tag":109,"props":1266,"children":1267},{"style":358},[1268],{"type":55,"value":361},{"type":50,"tag":109,"props":1270,"children":1271},{"style":159},[1272],{"type":55,"value":198},{"type":50,"tag":109,"props":1274,"children":1275},{"style":201},[1276],{"type":55,"value":1277},"  https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002Flatest\u002Fgetting-started.html",{"type":50,"tag":109,"props":1279,"children":1280},{"style":159},[1281],{"type":55,"value":293},{"type":50,"tag":109,"props":1283,"children":1285},{"class":111,"line":1284},46,[1286],{"type":50,"tag":109,"props":1287,"children":1288},{"style":153},[1289],{"type":55,"value":1290},"  fi\n",{"type":50,"tag":109,"props":1292,"children":1294},{"class":111,"line":1293},47,[1295],{"type":50,"tag":109,"props":1296,"children":1297},{"style":153},[1298],{"type":55,"value":480},{"type":50,"tag":57,"props":1300,"children":1301},{},[1302,1304,1310,1312,1317,1319,1325],{"type":55,"value":1303},"The GPU node runtime check is mandatory for self-managed nodes. For managed\nclusters where the client is not running on a GPU worker, verify the provider\nnode image or GPU Operator policy and set ",{"type":50,"tag":68,"props":1305,"children":1307},{"className":1306},[],[1308],{"type":55,"value":1309},"TAO_K8S_SKIP_NODE_RUNTIME_CHECK=1",{"type":55,"value":1311},"\ninstead of running the installer on the client. The final GPU capacity check is\na warning rather than a hard fail — ",{"type":50,"tag":68,"props":1313,"children":1315},{"className":1314},[],[1316],{"type":55,"value":1049},{"type":55,"value":1318}," isn't always installed. The SDK\ndoes a hard guard inside\n",{"type":50,"tag":68,"props":1320,"children":1322},{"className":1321},[],[1323],{"type":55,"value":1324},"KubernetesSDK.create_job()",{"type":55,"value":1326}," that uses the kubernetes Python client to verify\nGPU capacity before submitting.",{"type":50,"tag":86,"props":1328,"children":1330},{"id":1329},"credentials-configuration",[1331],{"type":55,"value":1332},"Credentials & configuration",{"type":50,"tag":1334,"props":1335,"children":1336},"ul",{},[1337,1379,1397,1407,1433],{"type":50,"tag":1338,"props":1339,"children":1340},"li",{},[1341,1347,1349],{"type":50,"tag":1342,"props":1343,"children":1344},"strong",{},[1345],{"type":55,"value":1346},"Kubeconfig",{"type":55,"value":1348}," (one of):\n",{"type":50,"tag":1334,"props":1350,"children":1351},{},[1352,1363,1374],{"type":50,"tag":1338,"props":1353,"children":1354},{},[1355,1361],{"type":50,"tag":68,"props":1356,"children":1358},{"className":1357},[],[1359],{"type":55,"value":1360},"~\u002F.kube\u002Fconfig",{"type":55,"value":1362}," — default discovery path",{"type":50,"tag":1338,"props":1364,"children":1365},{},[1366,1372],{"type":50,"tag":68,"props":1367,"children":1369},{"className":1368},[],[1370],{"type":55,"value":1371},"$KUBECONFIG",{"type":55,"value":1373}," — alternate path",{"type":50,"tag":1338,"props":1375,"children":1376},{},[1377],{"type":55,"value":1378},"In-cluster service account — used when running inside a pod (no kubeconfig needed)",{"type":50,"tag":1338,"props":1380,"children":1381},{},[1382,1387,1389,1395],{"type":50,"tag":1342,"props":1383,"children":1384},{},[1385],{"type":55,"value":1386},"TAO_K8S_NAMESPACE",{"type":55,"value":1388}," (optional): default namespace for Job submission. Defaults to ",{"type":50,"tag":68,"props":1390,"children":1392},{"className":1391},[],[1393],{"type":55,"value":1394},"default",{"type":55,"value":1396},".",{"type":50,"tag":1338,"props":1398,"children":1399},{},[1400,1405],{"type":50,"tag":1342,"props":1401,"children":1402},{},[1403],{"type":55,"value":1404},"TAO_K8S_CONTEXT",{"type":55,"value":1406}," (optional): kubeconfig context name to switch clusters.",{"type":50,"tag":1338,"props":1408,"children":1409},{},[1410,1415,1417,1423,1425,1431],{"type":50,"tag":1342,"props":1411,"children":1412},{},[1413],{"type":55,"value":1414},"NGC_KEY",{"type":55,"value":1416}," (optional): for nvcr.io image pulls. If you've pre-created an image-pull secret in the target namespace, pass its name to ",{"type":50,"tag":68,"props":1418,"children":1420},{"className":1419},[],[1421],{"type":55,"value":1422},"create_job",{"type":55,"value":1424}," via the ",{"type":50,"tag":68,"props":1426,"children":1428},{"className":1427},[],[1429],{"type":55,"value":1430},"image_pull_secret",{"type":55,"value":1432}," argument.",{"type":50,"tag":1338,"props":1434,"children":1435},{},[1436,1441,1443,1449,1451,1457],{"type":50,"tag":1342,"props":1437,"children":1438},{},[1439],{"type":55,"value":1440},"ACCESS_KEY \u002F SECRET_KEY \u002F S3_BUCKET_NAME \u002F S3_ENDPOINT_URL",{"type":55,"value":1442}," (optional): for S3 dataset I\u002FO via the SDK's ",{"type":50,"tag":68,"props":1444,"children":1446},{"className":1445},[],[1447],{"type":55,"value":1448},"inputs",{"type":55,"value":1450},"\u002F",{"type":50,"tag":68,"props":1452,"children":1454},{"className":1453},[],[1455],{"type":55,"value":1456},"outputs",{"type":55,"value":1458}," script_runner wrapping.",{"type":50,"tag":57,"props":1460,"children":1461},{},[1462,1464,1470,1472,1478],{"type":55,"value":1463},"Do not ask for Brev or SLURM credentials for Kubernetes runs. Ask for\nS3 credentials only when the selected workflow uses ",{"type":50,"tag":68,"props":1465,"children":1467},{"className":1466},[],[1468],{"type":55,"value":1469},"s3:\u002F\u002F",{"type":55,"value":1471}," inputs or outputs,\nand ask for model-specific credentials such as ",{"type":50,"tag":68,"props":1473,"children":1475},{"className":1474},[],[1476],{"type":55,"value":1477},"HF_TOKEN",{"type":55,"value":1479}," only when the selected\nmodel requires them. Before launch, verify the selected namespace can create\nJobs, dataset\u002Fresult paths are visible from the pod, and PVC\u002Fmounted filesystem\npaths are proven to be mounted into the job container; an agent-host local path\nis not sufficient proof.",{"type":50,"tag":86,"props":1481,"children":1483},{"id":1482},"sdk-api",[1484],{"type":55,"value":1485},"SDK API",{"type":50,"tag":57,"props":1487,"children":1488},{},[1489,1491,1496,1498,1504,1506,1511,1513,1519,1521,1527],{"type":55,"value":1490},"K8s is SDK-only — there is no ",{"type":50,"tag":68,"props":1492,"children":1494},{"className":1493},[],[1495],{"type":55,"value":1049},{"type":55,"value":1497},"-only launch path. Read\n",{"type":50,"tag":68,"props":1499,"children":1501},{"className":1500},[],[1502],{"type":55,"value":1503},"tao-skill-bank:tao-run-platform",{"type":55,"value":1505}," before drafting ",{"type":50,"tag":68,"props":1507,"children":1509},{"className":1508},[],[1510],{"type":55,"value":1422},{"type":55,"value":1512}," calls; it covers\n",{"type":50,"tag":68,"props":1514,"children":1516},{"className":1515},[],[1517],{"type":55,"value":1518},"build_entrypoint",{"type":55,"value":1520},", the shared kwarg contract, monitoring, and ",{"type":50,"tag":68,"props":1522,"children":1524},{"className":1523},[],[1525],{"type":55,"value":1526},"ActionWorkflow",{"type":55,"value":1396},{"type":50,"tag":98,"props":1529,"children":1532},{"className":1530,"code":1531,"language":563,"meta":103,"style":103},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from tao_sdk.platforms.kubernetes import KubernetesSDK\n\nsdk = KubernetesSDK()  # auto-detects auth\njob = sdk.create_job(\n    image='nvcr.io\u002Fnvidia\u002Ftao\u002Ftao-toolkit:6.26.3-pyt',\n    command='dino train -e \u002Ftmp\u002Fspec.yaml',\n    gpu_count=1,\n    env_vars={'NGC_KEY': os.environ['NGC_KEY']},\n    inputs={'\u002Fdata\u002Ftrain.json': 's3:\u002F\u002Fbucket\u002Fcoco\u002Ftrain.json'},\n    outputs=['\u002Fresults\u002F'],\n    namespace='tao-jobs',                       # optional override\n    image_pull_secret='ngc-pull-secret',         # optional, pre-created\n    node_selector={'gpu-type': 'h100'},          # optional\n)\n",[1533],{"type":50,"tag":68,"props":1534,"children":1535},{"__ignoreMap":103},[1536,1544,1551,1559,1567,1575,1583,1591,1599,1607,1615,1623,1631,1639],{"type":50,"tag":109,"props":1537,"children":1538},{"class":111,"line":112},[1539],{"type":50,"tag":109,"props":1540,"children":1541},{},[1542],{"type":55,"value":1543},"from tao_sdk.platforms.kubernetes import KubernetesSDK\n",{"type":50,"tag":109,"props":1545,"children":1546},{"class":111,"line":122},[1547],{"type":50,"tag":109,"props":1548,"children":1549},{"emptyLinePlaceholder":300},[1550],{"type":55,"value":303},{"type":50,"tag":109,"props":1552,"children":1553},{"class":111,"line":131},[1554],{"type":50,"tag":109,"props":1555,"children":1556},{},[1557],{"type":55,"value":1558},"sdk = KubernetesSDK()  # auto-detects auth\n",{"type":50,"tag":109,"props":1560,"children":1561},{"class":111,"line":140},[1562],{"type":50,"tag":109,"props":1563,"children":1564},{},[1565],{"type":55,"value":1566},"job = sdk.create_job(\n",{"type":50,"tag":109,"props":1568,"children":1569},{"class":111,"line":149},[1570],{"type":50,"tag":109,"props":1571,"children":1572},{},[1573],{"type":55,"value":1574},"    image='nvcr.io\u002Fnvidia\u002Ftao\u002Ftao-toolkit:6.26.3-pyt',\n",{"type":50,"tag":109,"props":1576,"children":1577},{"class":111,"line":222},[1578],{"type":50,"tag":109,"props":1579,"children":1580},{},[1581],{"type":55,"value":1582},"    command='dino train -e \u002Ftmp\u002Fspec.yaml',\n",{"type":50,"tag":109,"props":1584,"children":1585},{"class":111,"line":260},[1586],{"type":50,"tag":109,"props":1587,"children":1588},{},[1589],{"type":55,"value":1590},"    gpu_count=1,\n",{"type":50,"tag":109,"props":1592,"children":1593},{"class":111,"line":296},[1594],{"type":50,"tag":109,"props":1595,"children":1596},{},[1597],{"type":55,"value":1598},"    env_vars={'NGC_KEY': os.environ['NGC_KEY']},\n",{"type":50,"tag":109,"props":1600,"children":1601},{"class":111,"line":306},[1602],{"type":50,"tag":109,"props":1603,"children":1604},{},[1605],{"type":55,"value":1606},"    inputs={'\u002Fdata\u002Ftrain.json': 's3:\u002F\u002Fbucket\u002Fcoco\u002Ftrain.json'},\n",{"type":50,"tag":109,"props":1608,"children":1609},{"class":111,"line":354},[1610],{"type":50,"tag":109,"props":1611,"children":1612},{},[1613],{"type":55,"value":1614},"    outputs=['\u002Fresults\u002F'],\n",{"type":50,"tag":109,"props":1616,"children":1617},{"class":111,"line":377},[1618],{"type":50,"tag":109,"props":1619,"children":1620},{},[1621],{"type":55,"value":1622},"    namespace='tao-jobs',                       # optional override\n",{"type":50,"tag":109,"props":1624,"children":1625},{"class":111,"line":398},[1626],{"type":50,"tag":109,"props":1627,"children":1628},{},[1629],{"type":55,"value":1630},"    image_pull_secret='ngc-pull-secret',         # optional, pre-created\n",{"type":50,"tag":109,"props":1632,"children":1633},{"class":111,"line":429},[1634],{"type":50,"tag":109,"props":1635,"children":1636},{},[1637],{"type":55,"value":1638},"    node_selector={'gpu-type': 'h100'},          # optional\n",{"type":50,"tag":109,"props":1640,"children":1641},{"class":111,"line":450},[1642],{"type":50,"tag":109,"props":1643,"children":1644},{},[1645],{"type":55,"value":554},{"type":50,"tag":57,"props":1647,"children":1648},{},[1649,1651,1657],{"type":55,"value":1650},"The SDK constructs a ",{"type":50,"tag":68,"props":1652,"children":1654},{"className":1653},[],[1655],{"type":55,"value":1656},"V1Job",{"type":55,"value":1658}," with:",{"type":50,"tag":1334,"props":1660,"children":1661},{},[1662,1680,1691,1709,1728],{"type":50,"tag":1338,"props":1663,"children":1664},{},[1665,1671,1673,1679],{"type":50,"tag":68,"props":1666,"children":1668},{"className":1667},[],[1669],{"type":55,"value":1670},"spec.template.spec.containers[0]",{"type":55,"value":1672},": the requested image and ",{"type":50,"tag":68,"props":1674,"children":1676},{"className":1675},[],[1677],{"type":55,"value":1678},"command=[\"\u002Fbin\u002Fbash\", \"-c\", \u003Ccommand>]",{"type":55,"value":1396},{"type":50,"tag":1338,"props":1681,"children":1682},{},[1683,1689],{"type":50,"tag":68,"props":1684,"children":1686},{"className":1685},[],[1687],{"type":55,"value":1688},"resources.limits[\"nvidia.com\u002Fgpu\"]: \u003Cgpu_count>",{"type":55,"value":1690}," — schedules onto GPU nodes via the NVIDIA Device Plugin \u002F GPU Operator.",{"type":50,"tag":1338,"props":1692,"children":1693},{},[1694,1700,1702,1708],{"type":50,"tag":68,"props":1695,"children":1697},{"className":1696},[],[1698],{"type":55,"value":1699},"env_vars",{"type":55,"value":1701}," flowed through, plus auto-injected S3\u002FNGC\u002FHF credentials for ",{"type":50,"tag":68,"props":1703,"children":1705},{"className":1704},[],[1706],{"type":55,"value":1707},"script_runner",{"type":55,"value":1396},{"type":50,"tag":1338,"props":1710,"children":1711},{},[1712,1718,1720,1726],{"type":50,"tag":68,"props":1713,"children":1715},{"className":1714},[],[1716],{"type":55,"value":1717},"restart_policy=Never",{"type":55,"value":1719}," and ",{"type":50,"tag":68,"props":1721,"children":1723},{"className":1722},[],[1724],{"type":55,"value":1725},"backoff_limit=0",{"type":55,"value":1727}," — failures surface to the user instead of silently retrying.",{"type":50,"tag":1338,"props":1729,"children":1730},{},[1731,1737],{"type":50,"tag":68,"props":1732,"children":1734},{"className":1733},[],[1735],{"type":55,"value":1736},"ttl_seconds_after_finished=3600",{"type":55,"value":1738}," — Job auto-cleans 1 hour after terminal state.",{"type":50,"tag":86,"props":1740,"children":1742},{"id":1741},"status-monitoring",[1743],{"type":55,"value":1744},"Status & monitoring",{"type":50,"tag":98,"props":1746,"children":1748},{"className":1530,"code":1747,"language":563,"meta":103,"style":103},"status = sdk.get_job_status(job.id)\n# status.status ∈ {\"Pending\", \"Running\", \"Complete\", \"Error\", \"Canceled\", \"Unknown\"}\n\nlogs = sdk.get_job_logs(job.id, tail=200)  # concatenates logs from all pods of the Job\n\n# For stuck-Pending jobs — replica diagnostics:\nfor r in sdk.get_job_replicas(job.id):\n    issue = r[\"status\"].get(\"readiness_issue\")\n    if issue:\n        print(issue[\"reason\"], issue[\"message\"])\n        # e.g. \"ImagePullBackOff\" \u002F \"Back-off pulling image...\"\n        # e.g. \"Pending\"           \u002F \"0\u002F3 nodes available: 3 Insufficient nvidia.com\u002Fgpu\"\n\n# On failure:\nanalysis = sdk.get_failure_analysis(job.id)\n# {\"err_class\": \"ERR_PROGRAM\" | \"ERR_INFRA\",\n#  \"suggestion\": \"Container OOM-killed. Reduce batch size...\",\n#  \"job_failure_by_node_event\": [{\"node_event_name\": \"OOMKilled\", ...}]}\n",[1749],{"type":50,"tag":68,"props":1750,"children":1751},{"__ignoreMap":103},[1752,1760,1768,1775,1783,1790,1798,1806,1814,1822,1830,1838,1846,1853,1861,1869,1877,1885],{"type":50,"tag":109,"props":1753,"children":1754},{"class":111,"line":112},[1755],{"type":50,"tag":109,"props":1756,"children":1757},{},[1758],{"type":55,"value":1759},"status = sdk.get_job_status(job.id)\n",{"type":50,"tag":109,"props":1761,"children":1762},{"class":111,"line":122},[1763],{"type":50,"tag":109,"props":1764,"children":1765},{},[1766],{"type":55,"value":1767},"# status.status ∈ {\"Pending\", \"Running\", \"Complete\", \"Error\", \"Canceled\", \"Unknown\"}\n",{"type":50,"tag":109,"props":1769,"children":1770},{"class":111,"line":131},[1771],{"type":50,"tag":109,"props":1772,"children":1773},{"emptyLinePlaceholder":300},[1774],{"type":55,"value":303},{"type":50,"tag":109,"props":1776,"children":1777},{"class":111,"line":140},[1778],{"type":50,"tag":109,"props":1779,"children":1780},{},[1781],{"type":55,"value":1782},"logs = sdk.get_job_logs(job.id, tail=200)  # concatenates logs from all pods of the Job\n",{"type":50,"tag":109,"props":1784,"children":1785},{"class":111,"line":149},[1786],{"type":50,"tag":109,"props":1787,"children":1788},{"emptyLinePlaceholder":300},[1789],{"type":55,"value":303},{"type":50,"tag":109,"props":1791,"children":1792},{"class":111,"line":222},[1793],{"type":50,"tag":109,"props":1794,"children":1795},{},[1796],{"type":55,"value":1797},"# For stuck-Pending jobs — replica diagnostics:\n",{"type":50,"tag":109,"props":1799,"children":1800},{"class":111,"line":260},[1801],{"type":50,"tag":109,"props":1802,"children":1803},{},[1804],{"type":55,"value":1805},"for r in sdk.get_job_replicas(job.id):\n",{"type":50,"tag":109,"props":1807,"children":1808},{"class":111,"line":296},[1809],{"type":50,"tag":109,"props":1810,"children":1811},{},[1812],{"type":55,"value":1813},"    issue = r[\"status\"].get(\"readiness_issue\")\n",{"type":50,"tag":109,"props":1815,"children":1816},{"class":111,"line":306},[1817],{"type":50,"tag":109,"props":1818,"children":1819},{},[1820],{"type":55,"value":1821},"    if issue:\n",{"type":50,"tag":109,"props":1823,"children":1824},{"class":111,"line":354},[1825],{"type":50,"tag":109,"props":1826,"children":1827},{},[1828],{"type":55,"value":1829},"        print(issue[\"reason\"], issue[\"message\"])\n",{"type":50,"tag":109,"props":1831,"children":1832},{"class":111,"line":377},[1833],{"type":50,"tag":109,"props":1834,"children":1835},{},[1836],{"type":55,"value":1837},"        # e.g. \"ImagePullBackOff\" \u002F \"Back-off pulling image...\"\n",{"type":50,"tag":109,"props":1839,"children":1840},{"class":111,"line":398},[1841],{"type":50,"tag":109,"props":1842,"children":1843},{},[1844],{"type":55,"value":1845},"        # e.g. \"Pending\"           \u002F \"0\u002F3 nodes available: 3 Insufficient nvidia.com\u002Fgpu\"\n",{"type":50,"tag":109,"props":1847,"children":1848},{"class":111,"line":429},[1849],{"type":50,"tag":109,"props":1850,"children":1851},{"emptyLinePlaceholder":300},[1852],{"type":55,"value":303},{"type":50,"tag":109,"props":1854,"children":1855},{"class":111,"line":450},[1856],{"type":50,"tag":109,"props":1857,"children":1858},{},[1859],{"type":55,"value":1860},"# On failure:\n",{"type":50,"tag":109,"props":1862,"children":1863},{"class":111,"line":465},[1864],{"type":50,"tag":109,"props":1865,"children":1866},{},[1867],{"type":55,"value":1868},"analysis = sdk.get_failure_analysis(job.id)\n",{"type":50,"tag":109,"props":1870,"children":1871},{"class":111,"line":474},[1872],{"type":50,"tag":109,"props":1873,"children":1874},{},[1875],{"type":55,"value":1876},"# {\"err_class\": \"ERR_PROGRAM\" | \"ERR_INFRA\",\n",{"type":50,"tag":109,"props":1878,"children":1879},{"class":111,"line":483},[1880],{"type":50,"tag":109,"props":1881,"children":1882},{},[1883],{"type":55,"value":1884},"#  \"suggestion\": \"Container OOM-killed. Reduce batch size...\",\n",{"type":50,"tag":109,"props":1886,"children":1887},{"class":111,"line":491},[1888],{"type":50,"tag":109,"props":1889,"children":1890},{},[1891],{"type":55,"value":1892},"#  \"job_failure_by_node_event\": [{\"node_event_name\": \"OOMKilled\", ...}]}\n",{"type":50,"tag":86,"props":1894,"children":1896},{"id":1895},"cancel-cleanup",[1897],{"type":55,"value":1898},"Cancel & cleanup",{"type":50,"tag":98,"props":1900,"children":1902},{"className":1530,"code":1901,"language":563,"meta":103,"style":103},"sdk.cancel_job(job.id)  # delete_namespaced_job with propagation_policy=\"Foreground\"\n",[1903],{"type":50,"tag":68,"props":1904,"children":1905},{"__ignoreMap":103},[1906],{"type":50,"tag":109,"props":1907,"children":1908},{"class":111,"line":112},[1909],{"type":50,"tag":109,"props":1910,"children":1911},{},[1912],{"type":55,"value":1901},{"type":50,"tag":57,"props":1914,"children":1915},{},[1916,1921,1923,1929],{"type":50,"tag":68,"props":1917,"children":1919},{"className":1918},[],[1920],{"type":55,"value":1736},{"type":55,"value":1922}," means completed Jobs auto-delete after 1h. To cancel an in-flight Job, ",{"type":50,"tag":68,"props":1924,"children":1926},{"className":1925},[],[1927],{"type":55,"value":1928},"cancel_job",{"type":55,"value":1930}," deletes it and its pods immediately.",{"type":50,"tag":86,"props":1932,"children":1934},{"id":1933},"gpu-operator-dependency",[1935],{"type":55,"value":1936},"GPU Operator dependency",{"type":50,"tag":57,"props":1938,"children":1939},{},[1940,1942,1948,1950,1956],{"type":55,"value":1941},"The SDK refuses to submit GPU jobs to a cluster with no ",{"type":50,"tag":68,"props":1943,"children":1945},{"className":1944},[],[1946],{"type":55,"value":1947},"nvidia.com\u002Fgpu",{"type":55,"value":1949}," allocatable. For self-managed clusters, first run the ",{"type":50,"tag":68,"props":1951,"children":1953},{"className":1952},[],[1954],{"type":55,"value":1955},"tao-setup-nvidia-gpu-host",{"type":55,"value":1957}," install action on every GPU worker node or bake the same package set into the node image:",{"type":50,"tag":98,"props":1959,"children":1961},{"className":100,"code":1960,"language":102,"meta":103,"style":103},"bash skills\u002Fplatform\u002Ftao-setup-nvidia-gpu-host\u002Fscripts\u002Fsetup-nvidia-gpu-host.sh --backend kubernetes --install --yes\n",[1962],{"type":50,"tag":68,"props":1963,"children":1964},{"__ignoreMap":103},[1965],{"type":50,"tag":109,"props":1966,"children":1967},{"class":111,"line":112},[1968,1972,1977,1981,1985,1990],{"type":50,"tag":109,"props":1969,"children":1970},{"style":310},[1971],{"type":55,"value":102},{"type":50,"tag":109,"props":1973,"children":1974},{"style":201},[1975],{"type":55,"value":1976}," skills\u002Fplatform\u002Ftao-setup-nvidia-gpu-host\u002Fscripts\u002Fsetup-nvidia-gpu-host.sh",{"type":50,"tag":109,"props":1978,"children":1979},{"style":201},[1980],{"type":55,"value":331},{"type":50,"tag":109,"props":1982,"children":1983},{"style":201},[1984],{"type":55,"value":336},{"type":50,"tag":109,"props":1986,"children":1987},{"style":201},[1988],{"type":55,"value":1989}," --install",{"type":50,"tag":109,"props":1991,"children":1992},{"style":201},[1993],{"type":55,"value":1994}," --yes\n",{"type":50,"tag":57,"props":1996,"children":1997},{},[1998],{"type":55,"value":1999},"Then install the NVIDIA GPU Operator or device plugin:",{"type":50,"tag":98,"props":2001,"children":2003},{"className":100,"code":2002,"language":102,"meta":103,"style":103},"helm repo add nvidia https:\u002F\u002Fhelm.ngc.nvidia.com\u002Fnvidia\nhelm repo update\nhelm install --wait gpu-operator -n gpu-operator --create-namespace nvidia\u002Fgpu-operator\n",[2004],{"type":50,"tag":68,"props":2005,"children":2006},{"__ignoreMap":103},[2007,2035,2051],{"type":50,"tag":109,"props":2008,"children":2009},{"class":111,"line":112},[2010,2015,2020,2025,2030],{"type":50,"tag":109,"props":2011,"children":2012},{"style":310},[2013],{"type":55,"value":2014},"helm",{"type":50,"tag":109,"props":2016,"children":2017},{"style":201},[2018],{"type":55,"value":2019}," repo",{"type":50,"tag":109,"props":2021,"children":2022},{"style":201},[2023],{"type":55,"value":2024}," add",{"type":50,"tag":109,"props":2026,"children":2027},{"style":201},[2028],{"type":55,"value":2029}," nvidia",{"type":50,"tag":109,"props":2031,"children":2032},{"style":201},[2033],{"type":55,"value":2034}," https:\u002F\u002Fhelm.ngc.nvidia.com\u002Fnvidia\n",{"type":50,"tag":109,"props":2036,"children":2037},{"class":111,"line":122},[2038,2042,2046],{"type":50,"tag":109,"props":2039,"children":2040},{"style":310},[2041],{"type":55,"value":2014},{"type":50,"tag":109,"props":2043,"children":2044},{"style":201},[2045],{"type":55,"value":2019},{"type":50,"tag":109,"props":2047,"children":2048},{"style":201},[2049],{"type":55,"value":2050}," update\n",{"type":50,"tag":109,"props":2052,"children":2053},{"class":111,"line":131},[2054,2058,2062,2067,2072,2077,2081,2086],{"type":50,"tag":109,"props":2055,"children":2056},{"style":310},[2057],{"type":55,"value":2014},{"type":50,"tag":109,"props":2059,"children":2060},{"style":201},[2061],{"type":55,"value":650},{"type":50,"tag":109,"props":2063,"children":2064},{"style":201},[2065],{"type":55,"value":2066}," --wait",{"type":50,"tag":109,"props":2068,"children":2069},{"style":201},[2070],{"type":55,"value":2071}," gpu-operator",{"type":50,"tag":109,"props":2073,"children":2074},{"style":201},[2075],{"type":55,"value":2076}," -n",{"type":50,"tag":109,"props":2078,"children":2079},{"style":201},[2080],{"type":55,"value":2071},{"type":50,"tag":109,"props":2082,"children":2083},{"style":201},[2084],{"type":55,"value":2085}," --create-namespace",{"type":50,"tag":109,"props":2087,"children":2088},{"style":201},[2089],{"type":55,"value":2090}," nvidia\u002Fgpu-operator\n",{"type":50,"tag":57,"props":2092,"children":2093},{},[2094,2096],{"type":55,"value":2095},"Full guide: ",{"type":50,"tag":77,"props":2097,"children":2101},{"href":2098,"rel":2099},"https:\u002F\u002Fdocs.nvidia.com\u002Fdatacenter\u002Fcloud-native\u002Fgpu-operator\u002Flatest\u002Fgetting-started.html",[2100],"nofollow",[2102],{"type":55,"value":2098},{"type":50,"tag":86,"props":2104,"children":2106},{"id":2105},"multi-node-training-distributed",[2107],{"type":55,"value":2108},"Multi-node training (distributed)",{"type":50,"tag":57,"props":2110,"children":2111},{},[2112,2114,2119,2121,2127],{"type":55,"value":2113},"Pass ",{"type":50,"tag":68,"props":2115,"children":2117},{"className":2116},[],[2118],{"type":55,"value":73},{"type":55,"value":2120}," to ",{"type":50,"tag":68,"props":2122,"children":2124},{"className":2123},[],[2125],{"type":55,"value":2126},"create_job()",{"type":55,"value":2128}," to run distributed training across N pods. The SDK provisions:",{"type":50,"tag":2130,"props":2131,"children":2132},"ol",{},[2133,2168,2203],{"type":50,"tag":1338,"props":2134,"children":2135},{},[2136,2138,2143,2145,2151,2153,2159,2160,2166],{"type":55,"value":2137},"A ",{"type":50,"tag":1342,"props":2139,"children":2140},{},[2141],{"type":55,"value":2142},"headless Service",{"type":55,"value":2144}," named after the Job (selector: ",{"type":50,"tag":68,"props":2146,"children":2148},{"className":2147},[],[2149],{"type":55,"value":2150},"job-name=\u003Cjob-name>",{"type":55,"value":2152},", ",{"type":50,"tag":68,"props":2154,"children":2156},{"className":2155},[],[2157],{"type":55,"value":2158},"clusterIP: None",{"type":55,"value":2152},{"type":50,"tag":68,"props":2161,"children":2163},{"className":2162},[],[2164],{"type":55,"value":2165},"publishNotReadyAddresses: true",{"type":55,"value":2167}," so pods can rendezvous before they're all Ready).",{"type":50,"tag":1338,"props":2169,"children":2170},{},[2171,2173,2178,2180,2186,2187,2193,2195,2201],{"type":55,"value":2172},"An ",{"type":50,"tag":1342,"props":2174,"children":2175},{},[2176],{"type":55,"value":2177},"Indexed Job",{"type":55,"value":2179}," with ",{"type":50,"tag":68,"props":2181,"children":2183},{"className":2182},[],[2184],{"type":55,"value":2185},"parallelism = completions = num_nodes",{"type":55,"value":2152},{"type":50,"tag":68,"props":2188,"children":2190},{"className":2189},[],[2191],{"type":55,"value":2192},"completionMode: Indexed",{"type":55,"value":2194},". Each pod gets ",{"type":50,"tag":68,"props":2196,"children":2198},{"className":2197},[],[2199],{"type":55,"value":2200},"JOB_COMPLETION_INDEX",{"type":55,"value":2202}," injected by k8s automatically (= the node rank).",{"type":50,"tag":1338,"props":2204,"children":2205},{},[2206,2207,2212,2214,2460,2464,2466,2472,2474,2479],{"type":55,"value":2137},{"type":50,"tag":1342,"props":2208,"children":2209},{},[2210],{"type":55,"value":2211},"command wrapper",{"type":55,"value":2213}," that exports the rendezvous env vars before invoking the user command. Two naming conventions are exported simultaneously:",{"type":50,"tag":2215,"props":2216,"children":2217},"table",{},[2218,2242],{"type":50,"tag":2219,"props":2220,"children":2221},"thead",{},[2222],{"type":50,"tag":2223,"props":2224,"children":2225},"tr",{},[2226,2232,2237],{"type":50,"tag":2227,"props":2228,"children":2229},"th",{},[2230],{"type":55,"value":2231},"Env var",{"type":50,"tag":2227,"props":2233,"children":2234},{},[2235],{"type":55,"value":2236},"Value",{"type":50,"tag":2227,"props":2238,"children":2239},{},[2240],{"type":55,"value":2241},"Read by",{"type":50,"tag":2243,"props":2244,"children":2245},"tbody",{},[2246,2296,2322,2353,2381,2407,2434],{"type":50,"tag":2223,"props":2247,"children":2248},{},[2249,2259,2268],{"type":50,"tag":2250,"props":2251,"children":2252},"td",{},[2253],{"type":50,"tag":68,"props":2254,"children":2256},{"className":2255},[],[2257],{"type":55,"value":2258},"WORLD_SIZE",{"type":50,"tag":2250,"props":2260,"children":2261},{},[2262],{"type":50,"tag":68,"props":2263,"children":2265},{"className":2264},[],[2266],{"type":55,"value":2267},"num_nodes",{"type":50,"tag":2250,"props":2269,"children":2270},{},[2271,2273,2279,2281,2287,2289,2294],{"type":55,"value":2272},"TAO PyTorch container's ",{"type":50,"tag":68,"props":2274,"children":2276},{"className":2275},[],[2277],{"type":55,"value":2278},"nvidia_tao_pytorch\u002Fcore\u002Fentrypoint.py",{"type":55,"value":2280}," (uses this to mean ",{"type":50,"tag":2282,"props":2283,"children":2284},"em",{},[2285],{"type":55,"value":2286},"node count",{"type":55,"value":2288},", even though PyTorch's own convention is ",{"type":50,"tag":2282,"props":2290,"children":2291},{},[2292],{"type":55,"value":2293},"total processes",{"type":55,"value":2295},")",{"type":50,"tag":2223,"props":2297,"children":2298},{},[2299,2308,2317],{"type":50,"tag":2250,"props":2300,"children":2301},{},[2302],{"type":50,"tag":68,"props":2303,"children":2305},{"className":2304},[],[2306],{"type":55,"value":2307},"NUM_GPU_PER_NODE",{"type":50,"tag":2250,"props":2309,"children":2310},{},[2311],{"type":50,"tag":68,"props":2312,"children":2314},{"className":2313},[],[2315],{"type":55,"value":2316},"gpu_count",{"type":50,"tag":2250,"props":2318,"children":2319},{},[2320],{"type":55,"value":2321},"TAO PyTorch container's entrypoint",{"type":50,"tag":2223,"props":2323,"children":2324},{},[2325,2334,2342],{"type":50,"tag":2250,"props":2326,"children":2327},{},[2328],{"type":50,"tag":68,"props":2329,"children":2331},{"className":2330},[],[2332],{"type":55,"value":2333},"NNODES",{"type":50,"tag":2250,"props":2335,"children":2336},{},[2337],{"type":50,"tag":68,"props":2338,"children":2340},{"className":2339},[],[2341],{"type":55,"value":2267},{"type":50,"tag":2250,"props":2343,"children":2344},{},[2345,2351],{"type":50,"tag":68,"props":2346,"children":2348},{"className":2347},[],[2349],{"type":55,"value":2350},"torchrun",{"type":55,"value":2352}," and PyTorch-standard rendezvous",{"type":50,"tag":2223,"props":2354,"children":2355},{},[2356,2365,2373],{"type":50,"tag":2250,"props":2357,"children":2358},{},[2359],{"type":50,"tag":68,"props":2360,"children":2362},{"className":2361},[],[2363],{"type":55,"value":2364},"NPROC_PER_NODE",{"type":50,"tag":2250,"props":2366,"children":2367},{},[2368],{"type":50,"tag":68,"props":2369,"children":2371},{"className":2370},[],[2372],{"type":55,"value":2316},{"type":50,"tag":2250,"props":2374,"children":2375},{},[2376],{"type":50,"tag":68,"props":2377,"children":2379},{"className":2378},[],[2380],{"type":55,"value":2350},{"type":50,"tag":2223,"props":2382,"children":2383},{},[2384,2393,2402],{"type":50,"tag":2250,"props":2385,"children":2386},{},[2387],{"type":50,"tag":68,"props":2388,"children":2390},{"className":2389},[],[2391],{"type":55,"value":2392},"NODE_RANK",{"type":50,"tag":2250,"props":2394,"children":2395},{},[2396],{"type":50,"tag":68,"props":2397,"children":2399},{"className":2398},[],[2400],{"type":55,"value":2401},"$JOB_COMPLETION_INDEX",{"type":50,"tag":2250,"props":2403,"children":2404},{},[2405],{"type":55,"value":2406},"both",{"type":50,"tag":2223,"props":2408,"children":2409},{},[2410,2419,2430],{"type":50,"tag":2250,"props":2411,"children":2412},{},[2413],{"type":50,"tag":68,"props":2414,"children":2416},{"className":2415},[],[2417],{"type":55,"value":2418},"MASTER_ADDR",{"type":50,"tag":2250,"props":2420,"children":2421},{},[2422,2428],{"type":50,"tag":68,"props":2423,"children":2425},{"className":2424},[],[2426],{"type":55,"value":2427},"\u003Cjob-name>-0.\u003Cjob-name>",{"type":55,"value":2429}," (pod-0's DNS)",{"type":50,"tag":2250,"props":2431,"children":2432},{},[2433],{"type":55,"value":2406},{"type":50,"tag":2223,"props":2435,"children":2436},{},[2437,2446,2455],{"type":50,"tag":2250,"props":2438,"children":2439},{},[2440],{"type":50,"tag":68,"props":2441,"children":2443},{"className":2442},[],[2444],{"type":55,"value":2445},"MASTER_PORT",{"type":50,"tag":2250,"props":2447,"children":2448},{},[2449],{"type":50,"tag":68,"props":2450,"children":2452},{"className":2451},[],[2453],{"type":55,"value":2454},"29500",{"type":50,"tag":2250,"props":2456,"children":2457},{},[2458],{"type":55,"value":2459},"both (TAO's default)",{"type":50,"tag":2461,"props":2462,"children":2463},"br",{},[],{"type":55,"value":2465},"Both naming conventions are set so TAO entrypoints (",{"type":50,"tag":68,"props":2467,"children":2469},{"className":2468},[],[2470],{"type":55,"value":2471},"dino train",{"type":55,"value":2473},", etc.) and raw ",{"type":50,"tag":68,"props":2475,"children":2477},{"className":2476},[],[2478],{"type":55,"value":2350},{"type":55,"value":2480}," commands work without modification.",{"type":50,"tag":98,"props":2482,"children":2484},{"className":1530,"code":2483,"language":563,"meta":103,"style":103},"job = sdk.create_job(\n    image='nvcr.io\u002Fnvidia\u002Ftao\u002Ftao-toolkit:6.26.3-pyt',\n    command='dino train -e \u002Ftmp\u002Fspec.yaml',  # TAO entrypoint reads spec.train.num_nodes; env vars are wired by the container\n    gpu_count=8,           # GPUs per node\n    num_nodes=4,           # 4 × 8 = 32 GPUs total\n    inputs={'\u002Fdata\u002Ftrain.json': 's3:\u002F\u002Fbucket\u002Fcoco\u002Ftrain.json'},\n    outputs=['\u002Fresults\u002F'],\n)\n",[2485],{"type":50,"tag":68,"props":2486,"children":2487},{"__ignoreMap":103},[2488,2495,2502,2510,2518,2526,2533,2540],{"type":50,"tag":109,"props":2489,"children":2490},{"class":111,"line":112},[2491],{"type":50,"tag":109,"props":2492,"children":2493},{},[2494],{"type":55,"value":1566},{"type":50,"tag":109,"props":2496,"children":2497},{"class":111,"line":122},[2498],{"type":50,"tag":109,"props":2499,"children":2500},{},[2501],{"type":55,"value":1574},{"type":50,"tag":109,"props":2503,"children":2504},{"class":111,"line":131},[2505],{"type":50,"tag":109,"props":2506,"children":2507},{},[2508],{"type":55,"value":2509},"    command='dino train -e \u002Ftmp\u002Fspec.yaml',  # TAO entrypoint reads spec.train.num_nodes; env vars are wired by the container\n",{"type":50,"tag":109,"props":2511,"children":2512},{"class":111,"line":140},[2513],{"type":50,"tag":109,"props":2514,"children":2515},{},[2516],{"type":55,"value":2517},"    gpu_count=8,           # GPUs per node\n",{"type":50,"tag":109,"props":2519,"children":2520},{"class":111,"line":149},[2521],{"type":50,"tag":109,"props":2522,"children":2523},{},[2524],{"type":55,"value":2525},"    num_nodes=4,           # 4 × 8 = 32 GPUs total\n",{"type":50,"tag":109,"props":2527,"children":2528},{"class":111,"line":222},[2529],{"type":50,"tag":109,"props":2530,"children":2531},{},[2532],{"type":55,"value":1606},{"type":50,"tag":109,"props":2534,"children":2535},{"class":111,"line":260},[2536],{"type":50,"tag":109,"props":2537,"children":2538},{},[2539],{"type":55,"value":1614},{"type":50,"tag":109,"props":2541,"children":2542},{"class":111,"line":296},[2543],{"type":50,"tag":109,"props":2544,"children":2545},{},[2546],{"type":55,"value":554},{"type":50,"tag":57,"props":2548,"children":2549},{},[2550,2552,2557],{"type":55,"value":2551},"For raw ",{"type":50,"tag":68,"props":2553,"children":2555},{"className":2554},[],[2556],{"type":55,"value":2350},{"type":55,"value":2558},"-based commands (non-TAO containers):",{"type":50,"tag":98,"props":2560,"children":2562},{"className":1530,"code":2561,"language":563,"meta":103,"style":103},"job = sdk.create_job(\n    image='nvcr.io\u002Fnvidia\u002Fpytorch:25.08-py3',\n    command='torchrun --nnodes=$NNODES --nproc-per-node=$NPROC_PER_NODE --node-rank=$NODE_RANK '\n            '--master-addr=$MASTER_ADDR --master-port=$MASTER_PORT train.py',\n    gpu_count=8,\n    num_nodes=4,\n)\n",[2563],{"type":50,"tag":68,"props":2564,"children":2565},{"__ignoreMap":103},[2566,2573,2581,2589,2597,2605,2613],{"type":50,"tag":109,"props":2567,"children":2568},{"class":111,"line":112},[2569],{"type":50,"tag":109,"props":2570,"children":2571},{},[2572],{"type":55,"value":1566},{"type":50,"tag":109,"props":2574,"children":2575},{"class":111,"line":122},[2576],{"type":50,"tag":109,"props":2577,"children":2578},{},[2579],{"type":55,"value":2580},"    image='nvcr.io\u002Fnvidia\u002Fpytorch:25.08-py3',\n",{"type":50,"tag":109,"props":2582,"children":2583},{"class":111,"line":131},[2584],{"type":50,"tag":109,"props":2585,"children":2586},{},[2587],{"type":55,"value":2588},"    command='torchrun --nnodes=$NNODES --nproc-per-node=$NPROC_PER_NODE --node-rank=$NODE_RANK '\n",{"type":50,"tag":109,"props":2590,"children":2591},{"class":111,"line":140},[2592],{"type":50,"tag":109,"props":2593,"children":2594},{},[2595],{"type":55,"value":2596},"            '--master-addr=$MASTER_ADDR --master-port=$MASTER_PORT train.py',\n",{"type":50,"tag":109,"props":2598,"children":2599},{"class":111,"line":149},[2600],{"type":50,"tag":109,"props":2601,"children":2602},{},[2603],{"type":55,"value":2604},"    gpu_count=8,\n",{"type":50,"tag":109,"props":2606,"children":2607},{"class":111,"line":222},[2608],{"type":50,"tag":109,"props":2609,"children":2610},{},[2611],{"type":55,"value":2612},"    num_nodes=4,\n",{"type":50,"tag":109,"props":2614,"children":2615},{"class":111,"line":260},[2616],{"type":50,"tag":109,"props":2617,"children":2618},{},[2619],{"type":55,"value":554},{"type":50,"tag":57,"props":2621,"children":2622},{},[2623,2625,2631,2633,2638],{"type":55,"value":2624},"The capacity check sums across nodes: ",{"type":50,"tag":68,"props":2626,"children":2628},{"className":2627},[],[2629],{"type":55,"value":2630},"gpu_count × num_nodes",{"type":55,"value":2632}," ≤ cluster's allocatable ",{"type":50,"tag":68,"props":2634,"children":2636},{"className":2635},[],[2637],{"type":55,"value":1947},{"type":55,"value":1396},{"type":50,"tag":2640,"props":2641,"children":2643},"h3",{"id":2642},"cluster-requirements-for-multi-node",[2644],{"type":55,"value":2645},"Cluster requirements for multi-node",{"type":50,"tag":1334,"props":2647,"children":2648},{},[2649,2682,2699],{"type":50,"tag":1338,"props":2650,"children":2651},{},[2652,2657,2659,2665,2667,2673,2675,2681],{"type":50,"tag":1342,"props":2653,"children":2654},{},[2655],{"type":55,"value":2656},"k8s 1.28+",{"type":55,"value":2658}," is required for stable pod hostnames in Indexed Jobs (the ",{"type":50,"tag":68,"props":2660,"children":2662},{"className":2661},[],[2663],{"type":55,"value":2664},"PodIndexLabel",{"type":55,"value":2666}," feature). On older clusters the ",{"type":50,"tag":68,"props":2668,"children":2670},{"className":2669},[],[2671],{"type":55,"value":2672},"MASTER_ADDR=\u003Cjob>-0.\u003Csvc>",{"type":55,"value":2674}," DNS lookup fails. Verify with ",{"type":50,"tag":68,"props":2676,"children":2678},{"className":2677},[],[2679],{"type":55,"value":2680},"kubectl version",{"type":55,"value":1396},{"type":50,"tag":1338,"props":2683,"children":2684},{},[2685,2690,2692,2697],{"type":50,"tag":1342,"props":2686,"children":2687},{},[2688],{"type":55,"value":2689},"Pod-to-pod networking",{"type":55,"value":2691}," must be open on port 29500 (PyTorch default; configurable via ",{"type":50,"tag":68,"props":2693,"children":2695},{"className":2694},[],[2696],{"type":55,"value":2445},{"type":55,"value":2698}," env var). Most CNIs (Calico, Cilium, AWS VPC CNI) allow this by default; restrictive NetworkPolicies must be relaxed.",{"type":50,"tag":1338,"props":2700,"children":2701},{},[2702,2707,2709,2715,2717,2723,2725,2730],{"type":50,"tag":1342,"props":2703,"children":2704},{},[2705],{"type":55,"value":2706},"NCCL",{"type":55,"value":2708}," in the container talks GPU-to-GPU; if the cluster has multi-NIC nodes or RDMA, set ",{"type":50,"tag":68,"props":2710,"children":2712},{"className":2711},[],[2713],{"type":55,"value":2714},"NCCL_SOCKET_IFNAME",{"type":55,"value":2716}," \u002F ",{"type":50,"tag":68,"props":2718,"children":2720},{"className":2719},[],[2721],{"type":55,"value":2722},"NCCL_IB_HCA",{"type":55,"value":2724}," via ",{"type":50,"tag":68,"props":2726,"children":2728},{"className":2727},[],[2729],{"type":55,"value":1699},{"type":55,"value":1396},{"type":50,"tag":2640,"props":2732,"children":2734},{"id":2733},"reference-reading",[2735],{"type":55,"value":2736},"Reference reading",{"type":50,"tag":1334,"props":2738,"children":2739},{},[2740,2751,2762,2773],{"type":50,"tag":1338,"props":2741,"children":2742},{},[2743,2745],{"type":55,"value":2744},"Kubernetes Indexed Job: ",{"type":50,"tag":77,"props":2746,"children":2749},{"href":2747,"rel":2748},"https:\u002F\u002Fkubernetes.io\u002Fdocs\u002Fconcepts\u002Fworkloads\u002Fcontrollers\u002Fjob\u002F#completion-mode",[2100],[2750],{"type":55,"value":2747},{"type":50,"tag":1338,"props":2752,"children":2753},{},[2754,2756],{"type":55,"value":2755},"Indexed Job for batch ML: ",{"type":50,"tag":77,"props":2757,"children":2760},{"href":2758,"rel":2759},"https:\u002F\u002Fkubernetes.io\u002Fblog\u002F2022\u002F06\u002F01\u002Findexed-jobs-mpi\u002F",[2100],[2761],{"type":55,"value":2758},{"type":50,"tag":1338,"props":2763,"children":2764},{},[2765,2767],{"type":55,"value":2766},"PyTorch distributed (env-var rendezvous): ",{"type":50,"tag":77,"props":2768,"children":2771},{"href":2769,"rel":2770},"https:\u002F\u002Fpytorch.org\u002Fdocs\u002Fstable\u002Felastic\u002Frun.html",[2100],[2772],{"type":55,"value":2769},{"type":50,"tag":1338,"props":2774,"children":2775},{},[2776,2778],{"type":55,"value":2777},"NCCL networking tuning (NCCL_SOCKET_IFNAME, NCCL_IB_HCA): ",{"type":50,"tag":77,"props":2779,"children":2782},{"href":2780,"rel":2781},"https:\u002F\u002Fdocs.nvidia.com\u002Fdeeplearning\u002Fnccl\u002Fuser-guide\u002Fdocs\u002Fenv.html",[2100],[2783],{"type":55,"value":2780},{"type":50,"tag":2640,"props":2785,"children":2787},{"id":2786},"kubernetes-operator-alternatives",[2788],{"type":55,"value":2789},"Kubernetes operator alternatives",{"type":50,"tag":57,"props":2791,"children":2792},{},[2793],{"type":55,"value":2794},"For more sophisticated topologies (gang scheduling, PyTorch elastic \u002F fault-tolerant training, MPI \u002F Horovod, RDMA setup), reach for an operator instead of plain Indexed Job:",{"type":50,"tag":1334,"props":2796,"children":2797},{},[2798,2816,2849,2866],{"type":50,"tag":1338,"props":2799,"children":2800},{},[2801,2806,2808,2814],{"type":50,"tag":1342,"props":2802,"children":2803},{},[2804],{"type":55,"value":2805},"MPI Operator",{"type":55,"value":2807}," — ",{"type":50,"tag":77,"props":2809,"children":2812},{"href":2810,"rel":2811},"https:\u002F\u002Fgithub.com\u002Fkubeflow\u002Fmpi-operator",[2100],[2813],{"type":55,"value":2810},{"type":55,"value":2815}," — for MPI \u002F Horovod workloads.",{"type":50,"tag":1338,"props":2817,"children":2818},{},[2819,2824,2826,2832,2833,2839,2841,2847],{"type":50,"tag":1342,"props":2820,"children":2821},{},[2822],{"type":55,"value":2823},"Kubeflow Training Operator",{"type":55,"value":2825}," (",{"type":50,"tag":68,"props":2827,"children":2829},{"className":2828},[],[2830],{"type":55,"value":2831},"PyTorchJob",{"type":55,"value":2152},{"type":50,"tag":68,"props":2834,"children":2836},{"className":2835},[],[2837],{"type":55,"value":2838},"TFJob",{"type":55,"value":2840},") — ",{"type":50,"tag":77,"props":2842,"children":2845},{"href":2843,"rel":2844},"https:\u002F\u002Fwww.kubeflow.org\u002Fdocs\u002Fcomponents\u002Ftraining\u002F",[2100],[2846],{"type":55,"value":2843},{"type":55,"value":2848}," — for elastic PyTorch training with built-in restart logic.",{"type":50,"tag":1338,"props":2850,"children":2851},{},[2852,2857,2858,2864],{"type":50,"tag":1342,"props":2853,"children":2854},{},[2855],{"type":55,"value":2856},"Volcano",{"type":55,"value":2807},{"type":50,"tag":77,"props":2859,"children":2862},{"href":2860,"rel":2861},"https:\u002F\u002Fvolcano.sh\u002F",[2100],[2863],{"type":55,"value":2860},{"type":55,"value":2865}," — gang scheduling, queues, fair-share. Useful in shared multi-tenant clusters.",{"type":50,"tag":1338,"props":2867,"children":2868},{},[2869,2874,2875,2881],{"type":50,"tag":1342,"props":2870,"children":2871},{},[2872],{"type":55,"value":2873},"Kueue",{"type":55,"value":2807},{"type":50,"tag":77,"props":2876,"children":2879},{"href":2877,"rel":2878},"https:\u002F\u002Fkueue.sigs.k8s.io\u002F",[2100],[2880],{"type":55,"value":2877},{"type":55,"value":2882}," — quota \u002F queue layer on top of any of the above.",{"type":50,"tag":57,"props":2884,"children":2885},{},[2886],{"type":55,"value":2887},"The TAO SDK's Indexed Job path is intentionally simple and dependency-free; if you need elastic restart or gang scheduling, layer one of these on top and submit jobs through the operator's CRD instead.",{"type":50,"tag":86,"props":2889,"children":2891},{"id":2890},"common-error-patterns",[2892],{"type":55,"value":2893},"Common error patterns",{"type":50,"tag":57,"props":2895,"children":2896},{},[2897,2906,2908,2914],{"type":50,"tag":1342,"props":2898,"children":2899},{},[2900],{"type":50,"tag":68,"props":2901,"children":2903},{"className":2902},[],[2904],{"type":55,"value":2905},"No nvidia.com\u002Fgpu resources allocatable on the cluster",{"type":55,"value":2907}," — the GPU Operator (or NVIDIA Device Plugin) isn't installed. Install per the link above; verify with ",{"type":50,"tag":68,"props":2909,"children":2911},{"className":2910},[],[2912],{"type":55,"value":2913},"kubectl get nodes -o jsonpath='{.items[*].status.allocatable}'",{"type":55,"value":1396},{"type":50,"tag":57,"props":2916,"children":2917},{},[2918,2934,2936,2941],{"type":50,"tag":1342,"props":2919,"children":2920},{},[2921,2927,2928],{"type":50,"tag":68,"props":2922,"children":2924},{"className":2923},[],[2925],{"type":55,"value":2926},"ImagePullBackOff",{"type":55,"value":2716},{"type":50,"tag":68,"props":2929,"children":2931},{"className":2930},[],[2932],{"type":55,"value":2933},"ErrImagePull",{"type":55,"value":2935}," — the cluster can't pull the image. For nvcr.io: pre-create an image-pull secret in the namespace and pass its name via the ",{"type":50,"tag":68,"props":2937,"children":2939},{"className":2938},[],[2940],{"type":55,"value":1430},{"type":55,"value":2942}," argument:",{"type":50,"tag":98,"props":2944,"children":2946},{"className":100,"code":2945,"language":102,"meta":103,"style":103},"kubectl create secret docker-registry ngc-pull-secret \\\n  --docker-server=nvcr.io \\\n  --docker-username='$oauthtoken' \\\n  --docker-password=$NGC_KEY -n tao-jobs\n",[2947],{"type":50,"tag":68,"props":2948,"children":2949},{"__ignoreMap":103},[2950,2981,2993,3018],{"type":50,"tag":109,"props":2951,"children":2952},{"class":111,"line":112},[2953,2957,2962,2967,2972,2977],{"type":50,"tag":109,"props":2954,"children":2955},{"style":310},[2956],{"type":55,"value":1049},{"type":50,"tag":109,"props":2958,"children":2959},{"style":201},[2960],{"type":55,"value":2961}," create",{"type":50,"tag":109,"props":2963,"children":2964},{"style":201},[2965],{"type":55,"value":2966}," secret",{"type":50,"tag":109,"props":2968,"children":2969},{"style":201},[2970],{"type":55,"value":2971}," docker-registry",{"type":50,"tag":109,"props":2973,"children":2974},{"style":201},[2975],{"type":55,"value":2976}," ngc-pull-secret",{"type":50,"tag":109,"props":2978,"children":2979},{"style":170},[2980],{"type":55,"value":860},{"type":50,"tag":109,"props":2982,"children":2983},{"class":111,"line":122},[2984,2989],{"type":50,"tag":109,"props":2985,"children":2986},{"style":201},[2987],{"type":55,"value":2988},"  --docker-server=nvcr.io",{"type":50,"tag":109,"props":2990,"children":2991},{"style":170},[2992],{"type":55,"value":860},{"type":50,"tag":109,"props":2994,"children":2995},{"class":111,"line":131},[2996,3001,3005,3010,3014],{"type":50,"tag":109,"props":2997,"children":2998},{"style":201},[2999],{"type":55,"value":3000},"  --docker-username=",{"type":50,"tag":109,"props":3002,"children":3003},{"style":159},[3004],{"type":55,"value":1074},{"type":50,"tag":109,"props":3006,"children":3007},{"style":201},[3008],{"type":55,"value":3009},"$oauthtoken",{"type":50,"tag":109,"props":3011,"children":3012},{"style":159},[3013],{"type":55,"value":1074},{"type":50,"tag":109,"props":3015,"children":3016},{"style":170},[3017],{"type":55,"value":860},{"type":50,"tag":109,"props":3019,"children":3020},{"class":111,"line":140},[3021,3026,3031,3035],{"type":50,"tag":109,"props":3022,"children":3023},{"style":201},[3024],{"type":55,"value":3025},"  --docker-password=",{"type":50,"tag":109,"props":3027,"children":3028},{"style":170},[3029],{"type":55,"value":3030},"$NGC_KEY",{"type":50,"tag":109,"props":3032,"children":3033},{"style":201},[3034],{"type":55,"value":2076},{"type":50,"tag":109,"props":3036,"children":3037},{"style":201},[3038],{"type":55,"value":3039}," tao-jobs\n",{"type":50,"tag":57,"props":3041,"children":3042},{},[3043,3056,3057,3063,3065,3071,3073,3079],{"type":50,"tag":1342,"props":3044,"children":3045},{},[3046,3048,3054],{"type":55,"value":3047},"Pod stays ",{"type":50,"tag":68,"props":3049,"children":3051},{"className":3050},[],[3052],{"type":55,"value":3053},"Pending",{"type":55,"value":3055}," forever",{"type":55,"value":2807},{"type":50,"tag":68,"props":3058,"children":3060},{"className":3059},[],[3061],{"type":55,"value":3062},"get_job_replicas(job_id)",{"type":55,"value":3064}," will show the readiness_issue. Common causes: insufficient GPU capacity (",{"type":50,"tag":68,"props":3066,"children":3068},{"className":3067},[],[3069],{"type":55,"value":3070},"Insufficient nvidia.com\u002Fgpu",{"type":55,"value":3072},"), no node matches ",{"type":50,"tag":68,"props":3074,"children":3076},{"className":3075},[],[3077],{"type":55,"value":3078},"node_selector",{"type":55,"value":3080},", missing image-pull secret, or PVC mount failure.",{"type":50,"tag":57,"props":3082,"children":3083},{},[3084,3095],{"type":50,"tag":1342,"props":3085,"children":3086},{},[3087,3093],{"type":50,"tag":68,"props":3088,"children":3090},{"className":3089},[],[3091],{"type":55,"value":3092},"OOMKilled",{"type":55,"value":3094}," (exit 137)",{"type":55,"value":3096}," — container exceeded memory. Reduce batch size, lower max_length, or add a memory request\u002Flimit and target a larger node.",{"type":50,"tag":57,"props":3098,"children":3099},{},[3100,3109,3111,3117,3119,3124],{"type":50,"tag":1342,"props":3101,"children":3102},{},[3103],{"type":50,"tag":68,"props":3104,"children":3106},{"className":3105},[],[3107],{"type":55,"value":3108},"CredentialError: Could not authenticate to a Kubernetes cluster",{"type":55,"value":3110}," — neither kubeconfig nor in-cluster auth worked. Run ",{"type":50,"tag":68,"props":3112,"children":3114},{"className":3113},[],[3115],{"type":55,"value":3116},"kubectl get nodes",{"type":55,"value":3118}," to verify your config, or set ",{"type":50,"tag":68,"props":3120,"children":3122},{"className":3121},[],[3123],{"type":55,"value":1371},{"type":55,"value":3125}," to the right path.",{"type":50,"tag":86,"props":3127,"children":3129},{"id":3128},"what-this-skill-does-not-support-yet",[3130],{"type":55,"value":3131},"What this skill does NOT support (yet)",{"type":50,"tag":1334,"props":3133,"children":3134},{},[3135,3159,3176,3194,3204],{"type":50,"tag":1338,"props":3136,"children":3137},{},[3138,3143,3145,3150,3152,3157],{"type":50,"tag":1342,"props":3139,"children":3140},{},[3141],{"type":55,"value":3142},"Elastic \u002F fault-tolerant training.",{"type":55,"value":3144}," Indexed Job has ",{"type":50,"tag":68,"props":3146,"children":3148},{"className":3147},[],[3149],{"type":55,"value":1725},{"type":55,"value":3151}," — failures fail the whole training run. For elastic restart (e.g., resume from checkpoint after a node death), use Kubeflow's ",{"type":50,"tag":68,"props":3153,"children":3155},{"className":3154},[],[3156],{"type":55,"value":2831},{"type":55,"value":3158}," operator instead.",{"type":50,"tag":1338,"props":3160,"children":3161},{},[3162,3167,3169,3174],{"type":50,"tag":1342,"props":3163,"children":3164},{},[3165],{"type":55,"value":3166},"Gang scheduling.",{"type":55,"value":3168}," Indexed Job pods are scheduled independently — no all-or-nothing. Multi-node training will ",{"type":50,"tag":2282,"props":3170,"children":3171},{},[3172],{"type":55,"value":3173},"partially",{"type":55,"value":3175}," start if only some pods can be scheduled (rank-0 will hang waiting for peers). For all-or-nothing scheduling on shared clusters, use Volcano or Kueue.",{"type":50,"tag":1338,"props":3177,"children":3178},{},[3179,3184,3186,3192],{"type":50,"tag":1342,"props":3180,"children":3181},{},[3182],{"type":55,"value":3183},"MPI \u002F Horovod.",{"type":55,"value":3185}," Use the MPI Operator. The Indexed Job path here is PyTorch-distributed-shaped (env-var rendezvous on ",{"type":50,"tag":68,"props":3187,"children":3189},{"className":3188},[],[3190],{"type":55,"value":3191},"MASTER_ADDR:MASTER_PORT",{"type":55,"value":3193},").",{"type":50,"tag":1338,"props":3195,"children":3196},{},[3197,3202],{"type":50,"tag":1342,"props":3198,"children":3199},{},[3200],{"type":55,"value":3201},"Persistent volumes for shared storage.",{"type":55,"value":3203}," S3 only via the script_runner. PVC support is a follow-up.",{"type":50,"tag":1338,"props":3205,"children":3206},{},[3207,3218],{"type":50,"tag":1342,"props":3208,"children":3209},{},[3210,3212,3217],{"type":55,"value":3211},"Auto-creating image-pull secrets from ",{"type":50,"tag":68,"props":3213,"children":3215},{"className":3214},[],[3216],{"type":55,"value":3030},{"type":55,"value":1396},{"type":55,"value":3219}," You pre-create the secret in the target namespace and pass the name. K8s namespace conventions vary widely, so we keep secret creation explicit.",{"type":50,"tag":3221,"props":3222,"children":3223},"style",{},[3224],{"type":55,"value":3225},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3227,"total":3381},[3228,3246,3261,3272,3284,3298,3311,3325,3338,3349,3363,3372],{"slug":3229,"name":3229,"fn":3230,"description":3231,"org":3232,"tags":3233,"stars":3243,"repoUrl":3244,"updatedAt":3245},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3234,3237,3240],{"name":3235,"slug":3236,"type":15},"Documentation","documentation",{"name":3238,"slug":3239,"type":15},"MCP","mcp",{"name":3241,"slug":3242,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":3247,"name":3247,"fn":3248,"description":3249,"org":3250,"tags":3251,"stars":3258,"repoUrl":3259,"updatedAt":3260},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3252,3255,3256],{"name":3253,"slug":3254,"type":15},"Containers","containers",{"name":17,"slug":18,"type":15},{"name":3257,"slug":563,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":3262,"name":3262,"fn":3263,"description":3264,"org":3265,"tags":3266,"stars":3258,"repoUrl":3259,"updatedAt":3271},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3267,3270],{"name":3268,"slug":3269,"type":15},"CI\u002FCD","ci-cd",{"name":17,"slug":18,"type":15},"2026-07-14T05:25:59.97109",{"slug":3273,"name":3273,"fn":3274,"description":3275,"org":3276,"tags":3277,"stars":3258,"repoUrl":3259,"updatedAt":3283},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3278,3279,3280],{"name":3268,"slug":3269,"type":15},{"name":17,"slug":18,"type":15},{"name":3281,"slug":3282,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":3285,"name":3285,"fn":3286,"description":3287,"org":3288,"tags":3289,"stars":3258,"repoUrl":3259,"updatedAt":3297},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3290,3293,3294],{"name":3291,"slug":3292,"type":15},"Debugging","debugging",{"name":3281,"slug":3282,"type":15},{"name":3295,"slug":3296,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":3299,"name":3299,"fn":3300,"description":3301,"org":3302,"tags":3303,"stars":3258,"repoUrl":3259,"updatedAt":3310},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3304,3307],{"name":3305,"slug":3306,"type":15},"Best Practices","best-practices",{"name":3308,"slug":3309,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":3312,"name":3312,"fn":3313,"description":3314,"org":3315,"tags":3316,"stars":3258,"repoUrl":3259,"updatedAt":3324},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3317,3320,3323],{"name":3318,"slug":3319,"type":15},"Machine Learning","machine-learning",{"name":3321,"slug":3322,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":3326,"name":3326,"fn":3327,"description":3328,"org":3329,"tags":3330,"stars":3258,"repoUrl":3259,"updatedAt":3337},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3331,3334],{"name":3332,"slug":3333,"type":15},"QA","qa",{"name":3335,"slug":3336,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":3339,"name":3339,"fn":3340,"description":3341,"org":3342,"tags":3343,"stars":3258,"repoUrl":3259,"updatedAt":3348},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3344,3345],{"name":17,"slug":18,"type":15},{"name":3346,"slug":3347,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":3350,"name":3350,"fn":3351,"description":3352,"org":3353,"tags":3354,"stars":3258,"repoUrl":3259,"updatedAt":3362},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3355,3358,3359],{"name":3356,"slug":3357,"type":15},"Code Review","code-review",{"name":3281,"slug":3282,"type":15},{"name":3360,"slug":3361,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":3364,"name":3364,"fn":3365,"description":3366,"org":3367,"tags":3368,"stars":3258,"repoUrl":3259,"updatedAt":3371},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3369,3370],{"name":3332,"slug":3333,"type":15},{"name":3335,"slug":3336,"type":15},"2026-07-14T05:25:54.928983",{"slug":3373,"name":3373,"fn":3374,"description":3375,"org":3376,"tags":3377,"stars":3258,"repoUrl":3259,"updatedAt":3380},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3378,3379],{"name":13,"slug":14,"type":15},{"name":3268,"slug":3269,"type":15},"2026-07-30T05:29:03.275638",496,{"items":3383,"total":3479},[3384,3401,3411,3425,3435,3450,3465],{"slug":3385,"name":3385,"fn":3386,"description":3387,"org":3388,"tags":3389,"stars":23,"repoUrl":24,"updatedAt":3400},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3390,3393,3396,3397],{"name":3391,"slug":3392,"type":15},"Data Analysis","data-analysis",{"name":3394,"slug":3395,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":3398,"slug":3399,"type":15},"Performance","performance","2026-07-14T05:28:43.176466",{"slug":3402,"name":3402,"fn":3403,"description":3404,"org":3405,"tags":3406,"stars":23,"repoUrl":24,"updatedAt":3410},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3407,3408,3409],{"name":17,"slug":18,"type":15},{"name":3346,"slug":3347,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":3412,"name":3412,"fn":3413,"description":3414,"org":3415,"tags":3416,"stars":23,"repoUrl":24,"updatedAt":3424},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3417,3420,3421],{"name":3418,"slug":3419,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":3422,"slug":3423,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":3426,"name":3426,"fn":3427,"description":3428,"org":3429,"tags":3430,"stars":23,"repoUrl":24,"updatedAt":3434},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3431,3432,3433],{"name":3391,"slug":3392,"type":15},{"name":9,"slug":8,"type":15},{"name":3335,"slug":3336,"type":15},"2026-07-17T05:29:03.913266",{"slug":3436,"name":3436,"fn":3437,"description":3438,"org":3439,"tags":3440,"stars":23,"repoUrl":24,"updatedAt":3449},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3441,3442,3445,3446],{"name":13,"slug":14,"type":15},{"name":3443,"slug":3444,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":3447,"slug":3448,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":3451,"name":3451,"fn":3452,"description":3453,"org":3454,"tags":3455,"stars":23,"repoUrl":24,"updatedAt":3464},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3456,3457,3460,3461],{"name":17,"slug":18,"type":15},{"name":3458,"slug":3459,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":3462,"slug":3463,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":3466,"name":3466,"fn":3467,"description":3468,"org":3469,"tags":3470,"stars":23,"repoUrl":24,"updatedAt":3478},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[3471,3472,3475],{"name":9,"slug":8,"type":15},{"name":3473,"slug":3474,"type":15},"Quantum Computing","quantum-computing",{"name":3476,"slug":3477,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305]