[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-azure-taugrid":3,"mdc-p2d591-key":38,"related-org-azure-taugrid":4266,"related-repo-azure-taugrid":4445},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":27,"repoUrl":28,"updatedAt":29,"license":30,"forks":31,"topics":32,"repo":33,"sourceUrl":36,"mdContent":37},"taugrid","run GPU and AI workloads on Kubernetes","Run GPU and AI workloads on Kubernetes with TauGrid — the tau CLI plus Kueue queueing, KubeRay\u002FRay orchestration, GPU node health, and observability. Use for distributed training, fine-tuning, hyperparameter sweeps, model serving\u002Finference, datasets, run lifecycle, and cluster\u002Fworkspace onboarding. Trigger on plain-language intent even with no product name: 'run my training script on the cluster', 'fine-tune a 7B model', 'my GPU job is stuck\u002Fpending\u002Fpreempted\u002FOOMKilled', 'deploy this model for inference', 'set up a queue for my team'. Also trigger on tau.yaml, run config, Kueue admission or quota, LocalQueue\u002FClusterQueue, RayJob, torchrun, Ray Train\u002FTune, TauWorkspace, taugrid-portal, or Stellar. Use it BEFORE writing YAML destined for tau run, and before suggesting kubectl for a Tau-managed workload.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"azure","Azure (Microsoft)","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fazure.png","Azure",[13,17,20,21,24],{"name":14,"slug":15,"type":16},"Observability","observability","tag",{"name":18,"slug":19,"type":16},"GPU","gpu",{"name":11,"slug":8,"type":16},{"name":22,"slug":23,"type":16},"AI Infrastructure","ai-infrastructure",{"name":25,"slug":26,"type":16},"Kubernetes","kubernetes",5,"https:\u002F\u002Fgithub.com\u002FAzure\u002Ftaugrid","2026-09-02T07:47:33.048936",null,0,[],{"repoUrl":28,"stars":27,"forks":31,"topics":34,"description":35},[],"Cloud-native AI infrastructure for teams to run, schedule, and monitor AI workloads on GPU-enabled Kubernetes clusters, from data preparation to distributed training, fine-tuning, and inference.","https:\u002F\u002Fgithub.com\u002FAzure\u002Ftaugrid\u002Ftree\u002FHEAD\u002Fskills\u002Ftaugrid","---\nname: taugrid\ndescription: \"Run GPU and AI workloads on Kubernetes with TauGrid — the tau CLI plus Kueue queueing, KubeRay\u002FRay orchestration, GPU node health, and observability. Use for distributed training, fine-tuning, hyperparameter sweeps, model serving\u002Finference, datasets, run lifecycle, and cluster\u002Fworkspace onboarding. Trigger on plain-language intent even with no product name: 'run my training script on the cluster', 'fine-tune a 7B model', 'my GPU job is stuck\u002Fpending\u002Fpreempted\u002FOOMKilled', 'deploy this model for inference', 'set up a queue for my team'. Also trigger on tau.yaml, run config, Kueue admission or quota, LocalQueue\u002FClusterQueue, RayJob, torchrun, Ray Train\u002FTune, TauWorkspace, taugrid-portal, or Stellar. Use it BEFORE writing YAML destined for tau run, and before suggesting kubectl for a Tau-managed workload.\"\n---\n\n# TauGrid and the tau CLI\n\nTauGrid makes it easier to run GPU workloads on Kubernetes — data preparation,\ndistributed training, fine-tuning, and inference. It brings together the `tau`\nCLI, workload queueing and admission with **Kueue**, Ray cluster orchestration\nwith **KubeRay**, node-level **GPU health monitoring**, and cluster\u002Fworkload\n**observability** into one stack, so platform teams get an integrated\nfoundation and researchers can stay in their code instead of Kubernetes\nplumbing.\n\n`tau` itself is a CLI, renderer, and observer — **not** a scheduler, operator,\nor cloud provisioner. It turns checked-in workload intent into Kubernetes Jobs\nor KubeRay RayJobs, submits them through Kueue, and gives one lifecycle surface\nfor the result.\n\nGetting that boundary right is what separates useful help from confidently\nwrong help. When a job is stuck, the answer is usually \"Kueue has no quota\" or\n\"the workspace is Degraded\" — not something you can fix by editing YAML.\n\n## Who is asking\n\nAdapt to the person, since the same symptom has different owners:\n\n- **Researcher** — authoring `tau.yaml`, submitting runs, reading status\u002Flogs,\n retrying, serving a model, pulling experiment evidence. They cannot fix\n quota, node health, or workspace policy; tell them who can.\n- **Platform operator** — preparing clusters, bootstrapping workspaces,\n queues, storage, and node health. See\n [references\u002Fplatform.md](references\u002Fplatform.md).\n- **Contributor** — changing this repository's Go code. See\n [Repository layout](#repository-layout-for-contributors) at the end.\n\nMost users have only the `tau` binary, not this source tree. Answer from\n`tau --help`, `tau run schema`, and this skill rather than pointing them at\nrepository paths they do not have.\n\n## Ground rules that prevent most mistakes\n\n**Verify against the installed binary, not memory.** The CLI surface changed in\nv0.5 and binaries in the wild are often stale:\n\n```bash\ntau --help # actual root commands\ntau run schema -o json # authoritative config schema\ntau run explain-config # field reference with statuses\n```\n\nIf a command you expect is missing, the binary is old — say so rather than\nwriting instructions against a surface the user does not have. They upgrade on\nLinux or macOS with\n`curl -fsSL https:\u002F\u002Fgithub.com\u002FAzure\u002Ftaugrid\u002Freleases\u002Flatest\u002Fdownload\u002Finstall.sh | sh`;\nthe installer verifies the matching binary against `SHA256SUMS`.\n\n**`tau` has exactly seven roots:** `cluster`, `workspace`, `run`, `serve`,\n`data`, `python`, `version`. Experiment tracking and the observability portal\nlive in a **separate binary**, `taugrid-portal` — if a user still has\n`tau experiment` or `tau portal`, they are on a pre-split build. Pre-v0.5 flat\nroots (`submit`, `finetune`, `status`, `logs`, `ray`, `exp`, `queue`, `model`,\n`dataset`, …) are deleted, not deprecated.\n\n**`tau run TARGET` is a positional argument, not a subcommand.** The single\neasiest thing to get wrong, since it reads identically to a real subcommand:\n\n- `tau run train` → `run` root with `TARGET=train`, resolving `tau\u002Ftrain.yaml`\n- `tau run status` → the real `status` subcommand\n\nSubcommands are exactly: `validate`, `schema`, `explain-config`, `list`,\n`status`, `logs`, `get`, `cancel`, `resume`, `history`. Anything else is a\ntarget name — which is why `tau run retry` appears to \"work\" while doing\nnothing useful. There is no retry subcommand; retry is config behavior.\n\n**Prefer `tau` over raw `kubectl` for Tau-managed workloads.**\n`tau run status` already merges Job\u002FRayJob state, Kueue admission, the startup\nphase tree, and pods into one ordered view. Reach for `kubectl` only to confirm\nsomething `tau run status` has already pointed at, and say why you're doing it.\n\n## Authoring a run config\n\nThe normal contract is a checked-in **direct run config** — hand-written YAML\nthat `tau run --config` reads. Minimal shape:\n\n```yaml\nname: train # run name; also derives the default checkpoint path\nengine: rayjob # job | rayjob\nentrypoint: train.py # resolved relative to the config file\n\nruntime:\n image: \u003Cpinned-image> # pin a tag or digest; never :latest\n\ncompute:\n workers: 2 # Ray worker pods\n gpus_per_worker: 8\n\nstorage:\n data_pvc: training-data # mounted at \u002Fdata\n output: \u002Fdata\u002Fcheckpoints\u002Fworkflows\u002Ftrain\n```\n\nTop-level groups: `name`, `engine`, `entrypoint`, `script`, `image`,\n`schema_version`, `runtime`, `compute`, `execution`, `policy`, `storage`,\n`resilience`, `profiler`, `experiment`, `metrics`, `run`, `workflow`. There is\nno `eval` group — evaluation is an ordinary run. Full field list in\n[references\u002Frun-config.md](references\u002Frun-config.md); `tau run schema -o json`\nis the final authority since it is generated from the implementation.\n\n**The schema is strict — unknown fields are hard errors.** A typo like\n`compute.gpu` fails with `field gpu not found in type runconfig.Compute`.\n\n**`validate` alone is not enough — always follow it with a client dry-run.**\n`validate` is offline schema checking; preset resolution and GPU-count\narithmetic happen at *render* time, so a config can validate clean and still be\nunrunnable:\n\n```bash\ntau run validate --config tau\u002Ftrain.yaml # schema only\ntau run train --dry-run=client # renders; catches the rest\n```\n\nTwo failures `validate` will pass:\n\n| Config | `validate` | `--dry-run=client` |\n|---|---|---|\n| No `policy.preset` or `policy.profile` | is valid | `policy.profile or policy.preset is required` |\n| `processes_per_node: 99` on an 8-GPU preset | is valid | `processes_per_node (99) exceeds profile GPU count (8)` |\n\nBoth are free and offline, so run both before telling a user a config is good.\nNote that dry-run resolves `entrypoint` on disk, so run it from the config's\ndirectory or the script will appear missing.\n\n`--config` takes an explicit path; a bare `TARGET` resolves `tau\u002FTARGET.yaml`.\nThey are not interchangeable.\n\n### Engine choice constrains everything else\n\n`engine: job` renders a `batch\u002Fv1` Job — one pod, or an Indexed Job with\ntorchrun. `engine: rayjob` renders a KubeRay RayJob (head + `compute.workers`\nworkers). Mixing the two vocabularies is the most common authoring failure:\n\n| Intent | Correct | Common mistake |\n|---|---|---|\n| Multi-node PyTorch DDP | `engine: job` + `launcher: torchrun` + `execution.nodes: N` | `execution.nodes` with `engine: rayjob` |\n| Ray Train distributed | `engine: rayjob` + `compute.workers: N` | `launcher: torchrun` with `engine: rayjob` |\n| GPUs per pod on a Job | `policy.preset` (the node shape) | `compute.gpus_per_worker` with `engine: job` |\n| Extra PVC mounts | `engine: job` + `storage.mounts` | `storage.mounts` with `engine: rayjob` |\n\n`compute`'s Ray-shaped fields — `workers`, `gpus_per_worker`, `runtime.pip`,\n`head_*`\u002F`worker_*` — are **rejected on `engine: job`**. For a Job, GPU count\ncomes from the resolved preset, and `execution.processes_per_node` is validated\nagainst it. \"8 GPUs per node\" feels like a `compute` concern but is a placement\none. `execution.launcher` is engine-scoped: `job` takes `python`\u002F`torchrun`,\n`ray` takes `ray-train`\u002F`ray-tune`.\n\nTau owns the distributed-training env vars. `MASTER_ADDR`, `MASTER_PORT`,\n`TAU_WORLD_SIZE`, `TAU_DIST_BACKEND`, `TAU_NUM_WORKERS`, and `NCCL_*` are\nrejected in `runtime.env` so rendezvous stays consistent with the rendered\ntopology. `execution.allow_nccl_override: true` unblocks only the `NCCL_*`\nones, for deliberate tuning.\n\n### Secrets\n\nNever put a secret value in a run config:\n\n- `runtime.env_secret` — `KEY: \"secret-name:key\"` → `valueFrom.secretKeyRef`.\n- `runtime.env_kv` — Azure Key Vault via Secrets Store CSI; all entries must\n resolve to one vault.\n\nClient dry-run redacts both while keeping the dependency shape visible.\n\n## Running and observing\n\n```bash\ntau run train # submit the checked-in target\ntau run status \u003Crun-name> --watch # startup phase tree, live\ntau run logs \u003Crun-name> # Ray driver output, or Job pod logs\ntau run get \u003Crun-name> # durable results recorded by storage.output\ntau run cancel \u003Crun-name> # delete workload; Kueue reclaims quota\ntau run list -n \u003Cnamespace> # Tau-managed Jobs and RayJobs\n```\n\nOn a repository's first cluster-backed `tau run`, Tau resolves credentials\nthrough the descriptor's access method: it either isolates an existing\nkubeconfig context or obtains AKS credentials with the user's Azure identity.\nThe dedicated kubeconfig avoids mutating their main one; it is **not** a\nresearcher-isolation boundary and should not be described as one.\n\n`tau run status` is the canonical lifecycle view. It walks an ordered phase\ntree — Submitted, Kueue admission, (RayCluster), pod scheduling, DRA\nallocation, image pull, init containers, container start, Ready, (RayJob\nstatus) — where each phase reports pending, active, done, warning, or skipped.\n\n**Read it top to bottom and stop at the first phase that is not `done`.** That\nphase is the layer to investigate; everything after it is downstream noise.\n\nThree distinctions that matter when interpreting it:\n\n- **Admitted ≠ scheduled.** Kueue reserved quota; no pod exists yet.\n- **Running ≠ progressing.** Containers started; the training loop may be hung.\n- **Completed ≠ evidence preserved.** Check that artifacts actually landed.\n\nA `skipped` DRA phase is normal when the workload requests GPUs through the\ndevice plugin rather than DRA. It is not a failure.\n\n## Diagnosing a stuck or failed run\n\nWork the layers in order. Jumping to `kubectl describe pod` when the real\nproblem is quota (layer 4) or a Degraded workspace (layer 2) wastes time and\nproduces a misdiagnosis.\n\n| # | Layer | Command | Owner if it fails |\n|---|---|---|---|\n| 1 | Repo\u002Fconnection resolution and access | `tau workspace connection` (`--offline` for local configuration only) | Researcher (descriptor) \u002F platform (access) |\n| 2 | TauWorkspace readiness | `tau workspace status \u003Cname>` | Platform operator |\n| 3 | Config validation and render | `tau run validate --config \u003Cpath>` | Researcher |\n| 4 | Kueue admission and quota | `tau run status \u003Crun>` (admission phase) | Queue owner |\n| 5 | Scheduling, DRA, image pull, init | `tau run status \u003Crun> --watch` | Platform or researcher |\n| 6 | GPU\u002Fnode\u002Ftopology health | `tau cluster validate nodes` \u002F `... topology` | Node-pool operator |\n| 7 | Runtime progress and evidence | `tau run logs \u003Crun>` + `taugrid-portal experiment status \u003Cname>` | Researcher \u002F platform |\n| 8 | Recovery | see retry\u002Fresume below | — |\n\nLayers 1–3 are offline and cost nothing, so run them first when a symptom is\nambiguous. Full per-layer detail, including the operator-only `kubectl`\ncommands for each, is in\n[references\u002Ftroubleshooting.md](references\u002Ftroubleshooting.md).\n\nTwo guardrails worth stating to users directly:\n\n- Do not add namespace, queue, kubeconfig, or cloud credentials to a project\n config to work around a platform readiness failure. Those are workspace\n concerns, the submission gate is enforced server-side, and there is no\n client-side bypass.\n- Do not resubmit blindly after a failure. Locate the first failed layer, then\n choose retry or resume deliberately.\n\n## Retry and resume\n\nAutomatic retry is configuration, not a command:\n\n```yaml\nresilience:\n max_retries: 2\n retry_on: [\"Preempted\", \"Evicted\"] # default; OOMKilled is opt-in\n backoff_initial: 30s\n backoff_max: 5m\n checkpoint_path: \u002Fdata\u002Fcheckpoints\u002Ffinetunes\u002F\u003Cname>\n```\n\nWhen `max_retries > 0` and no dry-run is set, `tau run` waits for terminal\nstate, classifies the failure, checks it against `retry_on`, applies bounded\nexponential backoff, injects the checkpoint path and attempt number, then\ndeletes and resubmits. If the reason is not in `retry_on`, Tau exits with an\nerror naming it — unexpected failures surface instead of looping.\n\n`OOMKilled` is deliberately excluded by default: the same config with the same\nmemory usually reproduces the same OOM, so retrying without changing anything\njust spends queue time to fail identically. Suggest adding it only after\n`compute` memory has been raised. `Unknown` is never retryable in either path.\n\nManual resume:\n\n```bash\ntau run resume \u003Crun-name> --config tau\u002Ftrain.yaml # --config is required\n```\n\nResume needs a durable checkpoint under `\u002Fdata` — a workload that only wrote to\nnode-local scratch has nothing to resume from, since that state does not\nsurvive workload deletion. If the original failure was `OOMKilled`, resume\nrequires `--force`.\n\n## Serving a model\n\n`tau serve` renders a KubeRay RayService (default) or a plain Deployment. A\nservice is not a run — it does not inherit run lifecycle commands.\n\n```bash\ntau serve deploy \u003Cname> --kind=rayservice --profile \u003Cp> --image \u003Cpinned> \\\n --import-path serve:app --checkpoint \u003Cpath> --checkpoint-pvc \u003Cpvc> \\\n --namespace \u003Cns> --context \u003Cctx> --dry-run=client\ntau serve status \u003Cname> --kind=rayservice -n \u003Cns>\ntau serve scale \u003Cname> --kind=deployment --replicas 3 -n \u003Cns>\ntau serve delete \u003Cname> --kind=rayservice -n \u003Cns>\n```\n\n`--checkpoint` mounts the PVC at `\u002Fdata`, resolves relative paths under\n`\u002Fdata\u002Fcheckpoints`, and sets `TAU_MODEL_PATH`; the app still owns loading.\nDirect `scale` works only for `--kind=deployment` — a RayService's Serve config\nis a serialized field, so redeploy or set `--min-replicas`\u002F`--max-replicas` at\ncreation. `--from-finetune`\u002F`--from-model` read cluster metadata, so they\ncannot use client dry-run. Serving does not activate\n`tau\u002Fworkspace.connection.yaml` — pass namespace and context explicitly.\n\n## Datasets and models\n\n```bash\ntau data dataset list|show|ref|verify # curated dataset registry\ntau data model list|show|best|alias # durable checkpoints and aliases\n```\n\nThe registry is a catalog, not a data plane: records point at storage accessed\nby workload identity, and are immutable once registered — only aliases move.\nReference a dataset from a run through its resolved path on the mounted PVC.\n\n## Evidence: experiments and artifacts\n\nRun evidence lives in the **`taugrid-portal`** binary, not `tau`:\n\n```bash\ntaugrid-portal experiment search # find indexed runs\ntaugrid-portal experiment stellar \u003Crun-name> # local dashboard (-o html|json|tui)\ntaugrid-portal experiment open \u003Crun-name> # serve and open in a browser\ntaugrid-portal experiment status \u003Cname> # durable lifecycle record\ntaugrid-portal portal serve # unified observability portal\n```\n\nThe local expstore is the authoritative packet; ADX\u002FKusto and the hosted\nStellar UI are optional scalar projections. So \"the dashboard is empty\" is a\nprojection problem, not necessarily lost data.\n\nAdd discovery metadata so runs group correctly:\n\n```yaml\nname: \u003Crun-name>\nscript: train.py\ncompute:\n gpus: 0\nexperiment:\n project: \u003Cproject>\n name: \u003Cthe set of runs being compared>\n group: \u003Cnamed subset>\n```\n\n`experiment` has no `question` field — that is an expstore\u002FStellar concept.\nThe research question a set of runs answers is carried by `experiment.name`.\n\n## Platform operator work\n\nOperator commands need cluster-admin-level access and are deliberately not\nresearcher-facing. `tau cluster validate nodes` creates privileged pods.\n\n```bash\ntau cluster install --version \u003Cversion> --values \u003Cfile>\ntau cluster validate nodes --gpu-class \u003Cc> --min-healthy \u003Cn>\ntau cluster validate topology --preset \u003Cpreset> # or --cluster-queue (default taugrid-cq)\ntau cluster uninstall --yes # Helm-owned resources only\n\ntau workspace check \u003Cname> # exits non-zero unless Ready\ntau workspace status \u003Cname> -o json\n```\n\n`cluster install` and `cluster uninstall` are Helm-only. Platform owners create\n`TauWorkspace` resources through reviewed Helm\u002FGitOps\u002FIaC. TauGrid 0.1 does not\ncreate or mutate StorageClasses, PVCs, PVs, CSI configuration, cloud storage,\ncredentials, or Secrets.\n\nWorkspace `Ready` requires `RBACReady` and `QueueReady` true with no drift.\nTauGrid 0.1 has no `StorageReady` condition on `TauWorkspace` or `TauCluster`,\nso **a `Ready` workspace can still have a missing PVC**. Verify the\nplatform-managed claim before handing off. `WorkloadIdentityReady` is also\ndiagnostic only. Degraded-condition recovery and the install\u002Funinstall contract\nare in [references\u002Fplatform.md](references\u002Fplatform.md).\n\n## Boundaries — what to tell users Tau will not do\n\nTau owns resolution, validation, rendering, submission, and lifecycle. It does\n**not** own:\n\n- Azure\u002FAKS provisioning, node pools, or cloud RBAC\n- Kubernetes scheduling, or Kueue quota and admission decisions\n- Ray Train, PyTorch, or model-framework behavior\n- Project data preparation, model code, or serving application semantics\n\nWhen a request falls outside these — \"make Tau create my GPU node pool\", \"have\nTau raise my quota\" — name the system that actually owns it and who to ask.\nGuessing a plausible-sounding flag is worse than saying so.\n\n## Contributing to this repository\n\nModule layout, CI guards, the `tau.azure.com\u002F*` label contract, and how to\nverify a change: [references\u002Fcontributing.md](references\u002Fcontributing.md).\nUsers of the `tau` binary do not need it.\n",{"data":39,"body":40},{"name":4,"description":6},{"type":41,"children":42},"root",[43,52,95,112,117,124,129,188,216,222,232,321,341,501,517,570,652,696,702,722,971,1112,1137,1165,1228,1240,1342,1354,1381,1388,1423,1604,1727,1799,1805,1810,1849,1854,1860,2088,2107,2117,2134,2139,2172,2185,2191,2204,2473,2491,2496,2509,2515,2520,2666,2701,2727,2732,2784,2811,2817,2828,3216,3304,3310,3423,3428,3434,3455,3615,3620,3625,3754,3779,3785,3797,4078,4105,4182,4188,4199,4222,4227,4233,4260],{"type":44,"tag":45,"props":46,"children":48},"element","h1",{"id":47},"taugrid-and-the-tau-cli",[49],{"type":50,"value":51},"text","TauGrid and the tau CLI",{"type":44,"tag":53,"props":54,"children":55},"p",{},[56,58,65,67,73,75,80,82,87,89,93],{"type":50,"value":57},"TauGrid makes it easier to run GPU workloads on Kubernetes — data preparation,\ndistributed training, fine-tuning, and inference. It brings together the ",{"type":44,"tag":59,"props":60,"children":62},"code",{"className":61},[],[63],{"type":50,"value":64},"tau",{"type":50,"value":66},"\nCLI, workload queueing and admission with ",{"type":44,"tag":68,"props":69,"children":70},"strong",{},[71],{"type":50,"value":72},"Kueue",{"type":50,"value":74},", Ray cluster orchestration\nwith ",{"type":44,"tag":68,"props":76,"children":77},{},[78],{"type":50,"value":79},"KubeRay",{"type":50,"value":81},", node-level ",{"type":44,"tag":68,"props":83,"children":84},{},[85],{"type":50,"value":86},"GPU health monitoring",{"type":50,"value":88},", and cluster\u002Fworkload\n",{"type":44,"tag":68,"props":90,"children":91},{},[92],{"type":50,"value":15},{"type":50,"value":94}," into one stack, so platform teams get an integrated\nfoundation and researchers can stay in their code instead of Kubernetes\nplumbing.",{"type":44,"tag":53,"props":96,"children":97},{},[98,103,105,110],{"type":44,"tag":59,"props":99,"children":101},{"className":100},[],[102],{"type":50,"value":64},{"type":50,"value":104}," itself is a CLI, renderer, and observer — ",{"type":44,"tag":68,"props":106,"children":107},{},[108],{"type":50,"value":109},"not",{"type":50,"value":111}," a scheduler, operator,\nor cloud provisioner. It turns checked-in workload intent into Kubernetes Jobs\nor KubeRay RayJobs, submits them through Kueue, and gives one lifecycle surface\nfor the result.",{"type":44,"tag":53,"props":113,"children":114},{},[115],{"type":50,"value":116},"Getting that boundary right is what separates useful help from confidently\nwrong help. When a job is stuck, the answer is usually \"Kueue has no quota\" or\n\"the workspace is Degraded\" — not something you can fix by editing YAML.",{"type":44,"tag":118,"props":119,"children":121},"h2",{"id":120},"who-is-asking",[122],{"type":50,"value":123},"Who is asking",{"type":44,"tag":53,"props":125,"children":126},{},[127],{"type":50,"value":128},"Adapt to the person, since the same symptom has different owners:",{"type":44,"tag":130,"props":131,"children":132},"ul",{},[133,152,170],{"type":44,"tag":134,"props":135,"children":136},"li",{},[137,142,144,150],{"type":44,"tag":68,"props":138,"children":139},{},[140],{"type":50,"value":141},"Researcher",{"type":50,"value":143}," — authoring ",{"type":44,"tag":59,"props":145,"children":147},{"className":146},[],[148],{"type":50,"value":149},"tau.yaml",{"type":50,"value":151},", submitting runs, reading status\u002Flogs,\nretrying, serving a model, pulling experiment evidence. They cannot fix\nquota, node health, or workspace policy; tell them who can.",{"type":44,"tag":134,"props":153,"children":154},{},[155,160,162,168],{"type":44,"tag":68,"props":156,"children":157},{},[158],{"type":50,"value":159},"Platform operator",{"type":50,"value":161}," — preparing clusters, bootstrapping workspaces,\nqueues, storage, and node health. See\n",{"type":44,"tag":163,"props":164,"children":166},"a",{"href":165},"references\u002Fplatform.md",[167],{"type":50,"value":165},{"type":50,"value":169},".",{"type":44,"tag":134,"props":171,"children":172},{},[173,178,180,186],{"type":44,"tag":68,"props":174,"children":175},{},[176],{"type":50,"value":177},"Contributor",{"type":50,"value":179}," — changing this repository's Go code. See\n",{"type":44,"tag":163,"props":181,"children":183},{"href":182},"#repository-layout-for-contributors",[184],{"type":50,"value":185},"Repository layout",{"type":50,"value":187}," at the end.",{"type":44,"tag":53,"props":189,"children":190},{},[191,193,198,200,206,208,214],{"type":50,"value":192},"Most users have only the ",{"type":44,"tag":59,"props":194,"children":196},{"className":195},[],[197],{"type":50,"value":64},{"type":50,"value":199}," binary, not this source tree. Answer from\n",{"type":44,"tag":59,"props":201,"children":203},{"className":202},[],[204],{"type":50,"value":205},"tau --help",{"type":50,"value":207},", ",{"type":44,"tag":59,"props":209,"children":211},{"className":210},[],[212],{"type":50,"value":213},"tau run schema",{"type":50,"value":215},", and this skill rather than pointing them at\nrepository paths they do not have.",{"type":44,"tag":118,"props":217,"children":219},{"id":218},"ground-rules-that-prevent-most-mistakes",[220],{"type":50,"value":221},"Ground rules that prevent most mistakes",{"type":44,"tag":53,"props":223,"children":224},{},[225,230],{"type":44,"tag":68,"props":226,"children":227},{},[228],{"type":50,"value":229},"Verify against the installed binary, not memory.",{"type":50,"value":231}," The CLI surface changed in\nv0.5 and binaries in the wild are often stale:",{"type":44,"tag":233,"props":234,"children":239},"pre",{"className":235,"code":236,"language":237,"meta":238,"style":238},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","tau --help # actual root commands\ntau run schema -o json # authoritative config schema\ntau run explain-config # field reference with statuses\n","bash","",[240],{"type":44,"tag":59,"props":241,"children":242},{"__ignoreMap":238},[243,266,299],{"type":44,"tag":244,"props":245,"children":248},"span",{"class":246,"line":247},"line",1,[249,254,260],{"type":44,"tag":244,"props":250,"children":252},{"style":251},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[253],{"type":50,"value":64},{"type":44,"tag":244,"props":255,"children":257},{"style":256},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[258],{"type":50,"value":259}," --help",{"type":44,"tag":244,"props":261,"children":263},{"style":262},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[264],{"type":50,"value":265}," # actual root commands\n",{"type":44,"tag":244,"props":267,"children":269},{"class":246,"line":268},2,[270,274,279,284,289,294],{"type":44,"tag":244,"props":271,"children":272},{"style":251},[273],{"type":50,"value":64},{"type":44,"tag":244,"props":275,"children":276},{"style":256},[277],{"type":50,"value":278}," run",{"type":44,"tag":244,"props":280,"children":281},{"style":256},[282],{"type":50,"value":283}," schema",{"type":44,"tag":244,"props":285,"children":286},{"style":256},[287],{"type":50,"value":288}," -o",{"type":44,"tag":244,"props":290,"children":291},{"style":256},[292],{"type":50,"value":293}," json",{"type":44,"tag":244,"props":295,"children":296},{"style":262},[297],{"type":50,"value":298}," # authoritative config schema\n",{"type":44,"tag":244,"props":300,"children":302},{"class":246,"line":301},3,[303,307,311,316],{"type":44,"tag":244,"props":304,"children":305},{"style":251},[306],{"type":50,"value":64},{"type":44,"tag":244,"props":308,"children":309},{"style":256},[310],{"type":50,"value":278},{"type":44,"tag":244,"props":312,"children":313},{"style":256},[314],{"type":50,"value":315}," explain-config",{"type":44,"tag":244,"props":317,"children":318},{"style":262},[319],{"type":50,"value":320}," # field reference with statuses\n",{"type":44,"tag":53,"props":322,"children":323},{},[324,326,332,334,340],{"type":50,"value":325},"If a command you expect is missing, the binary is old — say so rather than\nwriting instructions against a surface the user does not have. They upgrade on\nLinux or macOS with\n",{"type":44,"tag":59,"props":327,"children":329},{"className":328},[],[330],{"type":50,"value":331},"curl -fsSL https:\u002F\u002Fgithub.com\u002FAzure\u002Ftaugrid\u002Freleases\u002Flatest\u002Fdownload\u002Finstall.sh | sh",{"type":50,"value":333},";\nthe installer verifies the matching binary against ",{"type":44,"tag":59,"props":335,"children":337},{"className":336},[],[338],{"type":50,"value":339},"SHA256SUMS",{"type":50,"value":169},{"type":44,"tag":53,"props":342,"children":343},{},[344,354,356,362,363,369,370,376,377,383,385,391,392,398,399,405,407,412,413,419,421,427,429,435,437,443,444,450,451,457,458,464,465,471,472,478,479,485,486,492,493,499],{"type":44,"tag":68,"props":345,"children":346},{},[347,352],{"type":44,"tag":59,"props":348,"children":350},{"className":349},[],[351],{"type":50,"value":64},{"type":50,"value":353}," has exactly seven roots:",{"type":50,"value":355}," ",{"type":44,"tag":59,"props":357,"children":359},{"className":358},[],[360],{"type":50,"value":361},"cluster",{"type":50,"value":207},{"type":44,"tag":59,"props":364,"children":366},{"className":365},[],[367],{"type":50,"value":368},"workspace",{"type":50,"value":207},{"type":44,"tag":59,"props":371,"children":373},{"className":372},[],[374],{"type":50,"value":375},"run",{"type":50,"value":207},{"type":44,"tag":59,"props":378,"children":380},{"className":379},[],[381],{"type":50,"value":382},"serve",{"type":50,"value":384},",\n",{"type":44,"tag":59,"props":386,"children":388},{"className":387},[],[389],{"type":50,"value":390},"data",{"type":50,"value":207},{"type":44,"tag":59,"props":393,"children":395},{"className":394},[],[396],{"type":50,"value":397},"python",{"type":50,"value":207},{"type":44,"tag":59,"props":400,"children":402},{"className":401},[],[403],{"type":50,"value":404},"version",{"type":50,"value":406},". Experiment tracking and the observability portal\nlive in a ",{"type":44,"tag":68,"props":408,"children":409},{},[410],{"type":50,"value":411},"separate binary",{"type":50,"value":207},{"type":44,"tag":59,"props":414,"children":416},{"className":415},[],[417],{"type":50,"value":418},"taugrid-portal",{"type":50,"value":420}," — if a user still has\n",{"type":44,"tag":59,"props":422,"children":424},{"className":423},[],[425],{"type":50,"value":426},"tau experiment",{"type":50,"value":428}," or ",{"type":44,"tag":59,"props":430,"children":432},{"className":431},[],[433],{"type":50,"value":434},"tau portal",{"type":50,"value":436},", they are on a pre-split build. Pre-v0.5 flat\nroots (",{"type":44,"tag":59,"props":438,"children":440},{"className":439},[],[441],{"type":50,"value":442},"submit",{"type":50,"value":207},{"type":44,"tag":59,"props":445,"children":447},{"className":446},[],[448],{"type":50,"value":449},"finetune",{"type":50,"value":207},{"type":44,"tag":59,"props":452,"children":454},{"className":453},[],[455],{"type":50,"value":456},"status",{"type":50,"value":207},{"type":44,"tag":59,"props":459,"children":461},{"className":460},[],[462],{"type":50,"value":463},"logs",{"type":50,"value":207},{"type":44,"tag":59,"props":466,"children":468},{"className":467},[],[469],{"type":50,"value":470},"ray",{"type":50,"value":207},{"type":44,"tag":59,"props":473,"children":475},{"className":474},[],[476],{"type":50,"value":477},"exp",{"type":50,"value":207},{"type":44,"tag":59,"props":480,"children":482},{"className":481},[],[483],{"type":50,"value":484},"queue",{"type":50,"value":207},{"type":44,"tag":59,"props":487,"children":489},{"className":488},[],[490],{"type":50,"value":491},"model",{"type":50,"value":384},{"type":44,"tag":59,"props":494,"children":496},{"className":495},[],[497],{"type":50,"value":498},"dataset",{"type":50,"value":500},", …) are deleted, not deprecated.",{"type":44,"tag":53,"props":502,"children":503},{},[504,515],{"type":44,"tag":68,"props":505,"children":506},{},[507,513],{"type":44,"tag":59,"props":508,"children":510},{"className":509},[],[511],{"type":50,"value":512},"tau run TARGET",{"type":50,"value":514}," is a positional argument, not a subcommand.",{"type":50,"value":516}," The single\neasiest thing to get wrong, since it reads identically to a real subcommand:",{"type":44,"tag":130,"props":518,"children":519},{},[520,552],{"type":44,"tag":134,"props":521,"children":522},{},[523,529,531,536,538,544,546],{"type":44,"tag":59,"props":524,"children":526},{"className":525},[],[527],{"type":50,"value":528},"tau run train",{"type":50,"value":530}," → ",{"type":44,"tag":59,"props":532,"children":534},{"className":533},[],[535],{"type":50,"value":375},{"type":50,"value":537}," root with ",{"type":44,"tag":59,"props":539,"children":541},{"className":540},[],[542],{"type":50,"value":543},"TARGET=train",{"type":50,"value":545},", resolving ",{"type":44,"tag":59,"props":547,"children":549},{"className":548},[],[550],{"type":50,"value":551},"tau\u002Ftrain.yaml",{"type":44,"tag":134,"props":553,"children":554},{},[555,561,563,568],{"type":44,"tag":59,"props":556,"children":558},{"className":557},[],[559],{"type":50,"value":560},"tau run status",{"type":50,"value":562}," → the real ",{"type":44,"tag":59,"props":564,"children":566},{"className":565},[],[567],{"type":50,"value":456},{"type":50,"value":569}," subcommand",{"type":44,"tag":53,"props":571,"children":572},{},[573,575,581,582,588,589,595,596,602,603,608,609,614,615,621,622,628,629,635,636,642,644,650],{"type":50,"value":574},"Subcommands are exactly: ",{"type":44,"tag":59,"props":576,"children":578},{"className":577},[],[579],{"type":50,"value":580},"validate",{"type":50,"value":207},{"type":44,"tag":59,"props":583,"children":585},{"className":584},[],[586],{"type":50,"value":587},"schema",{"type":50,"value":207},{"type":44,"tag":59,"props":590,"children":592},{"className":591},[],[593],{"type":50,"value":594},"explain-config",{"type":50,"value":207},{"type":44,"tag":59,"props":597,"children":599},{"className":598},[],[600],{"type":50,"value":601},"list",{"type":50,"value":384},{"type":44,"tag":59,"props":604,"children":606},{"className":605},[],[607],{"type":50,"value":456},{"type":50,"value":207},{"type":44,"tag":59,"props":610,"children":612},{"className":611},[],[613],{"type":50,"value":463},{"type":50,"value":207},{"type":44,"tag":59,"props":616,"children":618},{"className":617},[],[619],{"type":50,"value":620},"get",{"type":50,"value":207},{"type":44,"tag":59,"props":623,"children":625},{"className":624},[],[626],{"type":50,"value":627},"cancel",{"type":50,"value":207},{"type":44,"tag":59,"props":630,"children":632},{"className":631},[],[633],{"type":50,"value":634},"resume",{"type":50,"value":207},{"type":44,"tag":59,"props":637,"children":639},{"className":638},[],[640],{"type":50,"value":641},"history",{"type":50,"value":643},". Anything else is a\ntarget name — which is why ",{"type":44,"tag":59,"props":645,"children":647},{"className":646},[],[648],{"type":50,"value":649},"tau run retry",{"type":50,"value":651}," appears to \"work\" while doing\nnothing useful. There is no retry subcommand; retry is config behavior.",{"type":44,"tag":53,"props":653,"children":654},{},[655,675,680,682,687,689,694],{"type":44,"tag":68,"props":656,"children":657},{},[658,660,665,667,673],{"type":50,"value":659},"Prefer ",{"type":44,"tag":59,"props":661,"children":663},{"className":662},[],[664],{"type":50,"value":64},{"type":50,"value":666}," over raw ",{"type":44,"tag":59,"props":668,"children":670},{"className":669},[],[671],{"type":50,"value":672},"kubectl",{"type":50,"value":674}," for Tau-managed workloads.",{"type":44,"tag":59,"props":676,"children":678},{"className":677},[],[679],{"type":50,"value":560},{"type":50,"value":681}," already merges Job\u002FRayJob state, Kueue admission, the startup\nphase tree, and pods into one ordered view. Reach for ",{"type":44,"tag":59,"props":683,"children":685},{"className":684},[],[686],{"type":50,"value":672},{"type":50,"value":688}," only to confirm\nsomething ",{"type":44,"tag":59,"props":690,"children":692},{"className":691},[],[693],{"type":50,"value":560},{"type":50,"value":695}," has already pointed at, and say why you're doing it.",{"type":44,"tag":118,"props":697,"children":699},{"id":698},"authoring-a-run-config",[700],{"type":50,"value":701},"Authoring a run config",{"type":44,"tag":53,"props":703,"children":704},{},[705,707,712,714,720],{"type":50,"value":706},"The normal contract is a checked-in ",{"type":44,"tag":68,"props":708,"children":709},{},[710],{"type":50,"value":711},"direct run config",{"type":50,"value":713}," — hand-written YAML\nthat ",{"type":44,"tag":59,"props":715,"children":717},{"className":716},[],[718],{"type":50,"value":719},"tau run --config",{"type":50,"value":721}," reads. Minimal shape:",{"type":44,"tag":233,"props":723,"children":727},{"className":724,"code":725,"language":726,"meta":238,"style":238},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","name: train # run name; also derives the default checkpoint path\nengine: rayjob # job | rayjob\nentrypoint: train.py # resolved relative to the config file\n\nruntime:\n image: \u003Cpinned-image> # pin a tag or digest; never :latest\n\ncompute:\n workers: 2 # Ray worker pods\n gpus_per_worker: 8\n\nstorage:\n data_pvc: training-data # mounted at \u002Fdata\n output: \u002Fdata\u002Fcheckpoints\u002Fworkflows\u002Ftrain\n","yaml",[728],{"type":44,"tag":59,"props":729,"children":730},{"__ignoreMap":238},[731,756,778,800,810,823,846,854,867,891,909,917,930,953],{"type":44,"tag":244,"props":732,"children":733},{"class":246,"line":247},[734,740,746,751],{"type":44,"tag":244,"props":735,"children":737},{"style":736},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[738],{"type":50,"value":739},"name",{"type":44,"tag":244,"props":741,"children":743},{"style":742},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[744],{"type":50,"value":745},":",{"type":44,"tag":244,"props":747,"children":748},{"style":256},[749],{"type":50,"value":750}," train",{"type":44,"tag":244,"props":752,"children":753},{"style":262},[754],{"type":50,"value":755}," # run name; also derives the default checkpoint path\n",{"type":44,"tag":244,"props":757,"children":758},{"class":246,"line":268},[759,764,768,773],{"type":44,"tag":244,"props":760,"children":761},{"style":736},[762],{"type":50,"value":763},"engine",{"type":44,"tag":244,"props":765,"children":766},{"style":742},[767],{"type":50,"value":745},{"type":44,"tag":244,"props":769,"children":770},{"style":256},[771],{"type":50,"value":772}," rayjob",{"type":44,"tag":244,"props":774,"children":775},{"style":262},[776],{"type":50,"value":777}," # job | rayjob\n",{"type":44,"tag":244,"props":779,"children":780},{"class":246,"line":301},[781,786,790,795],{"type":44,"tag":244,"props":782,"children":783},{"style":736},[784],{"type":50,"value":785},"entrypoint",{"type":44,"tag":244,"props":787,"children":788},{"style":742},[789],{"type":50,"value":745},{"type":44,"tag":244,"props":791,"children":792},{"style":256},[793],{"type":50,"value":794}," train.py",{"type":44,"tag":244,"props":796,"children":797},{"style":262},[798],{"type":50,"value":799}," # resolved relative to the config file\n",{"type":44,"tag":244,"props":801,"children":803},{"class":246,"line":802},4,[804],{"type":44,"tag":244,"props":805,"children":807},{"emptyLinePlaceholder":806},true,[808],{"type":50,"value":809},"\n",{"type":44,"tag":244,"props":811,"children":812},{"class":246,"line":27},[813,818],{"type":44,"tag":244,"props":814,"children":815},{"style":736},[816],{"type":50,"value":817},"runtime",{"type":44,"tag":244,"props":819,"children":820},{"style":742},[821],{"type":50,"value":822},":\n",{"type":44,"tag":244,"props":824,"children":826},{"class":246,"line":825},6,[827,832,836,841],{"type":44,"tag":244,"props":828,"children":829},{"style":736},[830],{"type":50,"value":831}," image",{"type":44,"tag":244,"props":833,"children":834},{"style":742},[835],{"type":50,"value":745},{"type":44,"tag":244,"props":837,"children":838},{"style":256},[839],{"type":50,"value":840}," \u003Cpinned-image>",{"type":44,"tag":244,"props":842,"children":843},{"style":262},[844],{"type":50,"value":845}," # pin a tag or digest; never :latest\n",{"type":44,"tag":244,"props":847,"children":849},{"class":246,"line":848},7,[850],{"type":44,"tag":244,"props":851,"children":852},{"emptyLinePlaceholder":806},[853],{"type":50,"value":809},{"type":44,"tag":244,"props":855,"children":857},{"class":246,"line":856},8,[858,863],{"type":44,"tag":244,"props":859,"children":860},{"style":736},[861],{"type":50,"value":862},"compute",{"type":44,"tag":244,"props":864,"children":865},{"style":742},[866],{"type":50,"value":822},{"type":44,"tag":244,"props":868,"children":870},{"class":246,"line":869},9,[871,876,880,886],{"type":44,"tag":244,"props":872,"children":873},{"style":736},[874],{"type":50,"value":875}," workers",{"type":44,"tag":244,"props":877,"children":878},{"style":742},[879],{"type":50,"value":745},{"type":44,"tag":244,"props":881,"children":883},{"style":882},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[884],{"type":50,"value":885}," 2",{"type":44,"tag":244,"props":887,"children":888},{"style":262},[889],{"type":50,"value":890}," # Ray worker pods\n",{"type":44,"tag":244,"props":892,"children":894},{"class":246,"line":893},10,[895,900,904],{"type":44,"tag":244,"props":896,"children":897},{"style":736},[898],{"type":50,"value":899}," gpus_per_worker",{"type":44,"tag":244,"props":901,"children":902},{"style":742},[903],{"type":50,"value":745},{"type":44,"tag":244,"props":905,"children":906},{"style":882},[907],{"type":50,"value":908}," 8\n",{"type":44,"tag":244,"props":910,"children":912},{"class":246,"line":911},11,[913],{"type":44,"tag":244,"props":914,"children":915},{"emptyLinePlaceholder":806},[916],{"type":50,"value":809},{"type":44,"tag":244,"props":918,"children":920},{"class":246,"line":919},12,[921,926],{"type":44,"tag":244,"props":922,"children":923},{"style":736},[924],{"type":50,"value":925},"storage",{"type":44,"tag":244,"props":927,"children":928},{"style":742},[929],{"type":50,"value":822},{"type":44,"tag":244,"props":931,"children":933},{"class":246,"line":932},13,[934,939,943,948],{"type":44,"tag":244,"props":935,"children":936},{"style":736},[937],{"type":50,"value":938}," data_pvc",{"type":44,"tag":244,"props":940,"children":941},{"style":742},[942],{"type":50,"value":745},{"type":44,"tag":244,"props":944,"children":945},{"style":256},[946],{"type":50,"value":947}," training-data",{"type":44,"tag":244,"props":949,"children":950},{"style":262},[951],{"type":50,"value":952}," # mounted at \u002Fdata\n",{"type":44,"tag":244,"props":954,"children":956},{"class":246,"line":955},14,[957,962,966],{"type":44,"tag":244,"props":958,"children":959},{"style":736},[960],{"type":50,"value":961}," output",{"type":44,"tag":244,"props":963,"children":964},{"style":742},[965],{"type":50,"value":745},{"type":44,"tag":244,"props":967,"children":968},{"style":256},[969],{"type":50,"value":970}," \u002Fdata\u002Fcheckpoints\u002Fworkflows\u002Ftrain\n",{"type":44,"tag":53,"props":972,"children":973},{},[974,976,981,982,987,988,993,994,1000,1001,1007,1008,1014,1015,1020,1021,1026,1027,1033,1034,1040,1041,1046,1047,1053,1054,1060,1061,1067,1068,1074,1075,1080,1081,1087,1089,1095,1097,1102,1104,1110],{"type":50,"value":975},"Top-level groups: ",{"type":44,"tag":59,"props":977,"children":979},{"className":978},[],[980],{"type":50,"value":739},{"type":50,"value":207},{"type":44,"tag":59,"props":983,"children":985},{"className":984},[],[986],{"type":50,"value":763},{"type":50,"value":207},{"type":44,"tag":59,"props":989,"children":991},{"className":990},[],[992],{"type":50,"value":785},{"type":50,"value":207},{"type":44,"tag":59,"props":995,"children":997},{"className":996},[],[998],{"type":50,"value":999},"script",{"type":50,"value":207},{"type":44,"tag":59,"props":1002,"children":1004},{"className":1003},[],[1005],{"type":50,"value":1006},"image",{"type":50,"value":384},{"type":44,"tag":59,"props":1009,"children":1011},{"className":1010},[],[1012],{"type":50,"value":1013},"schema_version",{"type":50,"value":207},{"type":44,"tag":59,"props":1016,"children":1018},{"className":1017},[],[1019],{"type":50,"value":817},{"type":50,"value":207},{"type":44,"tag":59,"props":1022,"children":1024},{"className":1023},[],[1025],{"type":50,"value":862},{"type":50,"value":207},{"type":44,"tag":59,"props":1028,"children":1030},{"className":1029},[],[1031],{"type":50,"value":1032},"execution",{"type":50,"value":207},{"type":44,"tag":59,"props":1035,"children":1037},{"className":1036},[],[1038],{"type":50,"value":1039},"policy",{"type":50,"value":207},{"type":44,"tag":59,"props":1042,"children":1044},{"className":1043},[],[1045],{"type":50,"value":925},{"type":50,"value":384},{"type":44,"tag":59,"props":1048,"children":1050},{"className":1049},[],[1051],{"type":50,"value":1052},"resilience",{"type":50,"value":207},{"type":44,"tag":59,"props":1055,"children":1057},{"className":1056},[],[1058],{"type":50,"value":1059},"profiler",{"type":50,"value":207},{"type":44,"tag":59,"props":1062,"children":1064},{"className":1063},[],[1065],{"type":50,"value":1066},"experiment",{"type":50,"value":207},{"type":44,"tag":59,"props":1069,"children":1071},{"className":1070},[],[1072],{"type":50,"value":1073},"metrics",{"type":50,"value":207},{"type":44,"tag":59,"props":1076,"children":1078},{"className":1077},[],[1079],{"type":50,"value":375},{"type":50,"value":207},{"type":44,"tag":59,"props":1082,"children":1084},{"className":1083},[],[1085],{"type":50,"value":1086},"workflow",{"type":50,"value":1088},". There is\nno ",{"type":44,"tag":59,"props":1090,"children":1092},{"className":1091},[],[1093],{"type":50,"value":1094},"eval",{"type":50,"value":1096}," group — evaluation is an ordinary run. Full field list in\n",{"type":44,"tag":163,"props":1098,"children":1100},{"href":1099},"references\u002Frun-config.md",[1101],{"type":50,"value":1099},{"type":50,"value":1103},"; ",{"type":44,"tag":59,"props":1105,"children":1107},{"className":1106},[],[1108],{"type":50,"value":1109},"tau run schema -o json",{"type":50,"value":1111},"\nis the final authority since it is generated from the implementation.",{"type":44,"tag":53,"props":1113,"children":1114},{},[1115,1120,1122,1128,1130,1136],{"type":44,"tag":68,"props":1116,"children":1117},{},[1118],{"type":50,"value":1119},"The schema is strict — unknown fields are hard errors.",{"type":50,"value":1121}," A typo like\n",{"type":44,"tag":59,"props":1123,"children":1125},{"className":1124},[],[1126],{"type":50,"value":1127},"compute.gpu",{"type":50,"value":1129}," fails with ",{"type":44,"tag":59,"props":1131,"children":1133},{"className":1132},[],[1134],{"type":50,"value":1135},"field gpu not found in type runconfig.Compute",{"type":50,"value":169},{"type":44,"tag":53,"props":1138,"children":1139},{},[1140,1150,1155,1157,1163],{"type":44,"tag":68,"props":1141,"children":1142},{},[1143,1148],{"type":44,"tag":59,"props":1144,"children":1146},{"className":1145},[],[1147],{"type":50,"value":580},{"type":50,"value":1149}," alone is not enough — always follow it with a client dry-run.",{"type":44,"tag":59,"props":1151,"children":1153},{"className":1152},[],[1154],{"type":50,"value":580},{"type":50,"value":1156}," is offline schema checking; preset resolution and GPU-count\narithmetic happen at ",{"type":44,"tag":1158,"props":1159,"children":1160},"em",{},[1161],{"type":50,"value":1162},"render",{"type":50,"value":1164}," time, so a config can validate clean and still be\nunrunnable:",{"type":44,"tag":233,"props":1166,"children":1168},{"className":235,"code":1167,"language":237,"meta":238,"style":238},"tau run validate --config tau\u002Ftrain.yaml # schema only\ntau run train --dry-run=client # renders; catches the rest\n",[1169],{"type":44,"tag":59,"props":1170,"children":1171},{"__ignoreMap":238},[1172,1203],{"type":44,"tag":244,"props":1173,"children":1174},{"class":246,"line":247},[1175,1179,1183,1188,1193,1198],{"type":44,"tag":244,"props":1176,"children":1177},{"style":251},[1178],{"type":50,"value":64},{"type":44,"tag":244,"props":1180,"children":1181},{"style":256},[1182],{"type":50,"value":278},{"type":44,"tag":244,"props":1184,"children":1185},{"style":256},[1186],{"type":50,"value":1187}," validate",{"type":44,"tag":244,"props":1189,"children":1190},{"style":256},[1191],{"type":50,"value":1192}," --config",{"type":44,"tag":244,"props":1194,"children":1195},{"style":256},[1196],{"type":50,"value":1197}," tau\u002Ftrain.yaml",{"type":44,"tag":244,"props":1199,"children":1200},{"style":262},[1201],{"type":50,"value":1202}," # schema only\n",{"type":44,"tag":244,"props":1204,"children":1205},{"class":246,"line":268},[1206,1210,1214,1218,1223],{"type":44,"tag":244,"props":1207,"children":1208},{"style":251},[1209],{"type":50,"value":64},{"type":44,"tag":244,"props":1211,"children":1212},{"style":256},[1213],{"type":50,"value":278},{"type":44,"tag":244,"props":1215,"children":1216},{"style":256},[1217],{"type":50,"value":750},{"type":44,"tag":244,"props":1219,"children":1220},{"style":256},[1221],{"type":50,"value":1222}," --dry-run=client",{"type":44,"tag":244,"props":1224,"children":1225},{"style":262},[1226],{"type":50,"value":1227}," # renders; catches the rest\n",{"type":44,"tag":53,"props":1229,"children":1230},{},[1231,1233,1238],{"type":50,"value":1232},"Two failures ",{"type":44,"tag":59,"props":1234,"children":1236},{"className":1235},[],[1237],{"type":50,"value":580},{"type":50,"value":1239}," will pass:",{"type":44,"tag":1241,"props":1242,"children":1243},"table",{},[1244,1275],{"type":44,"tag":1245,"props":1246,"children":1247},"thead",{},[1248],{"type":44,"tag":1249,"props":1250,"children":1251},"tr",{},[1252,1258,1266],{"type":44,"tag":1253,"props":1254,"children":1255},"th",{},[1256],{"type":50,"value":1257},"Config",{"type":44,"tag":1253,"props":1259,"children":1260},{},[1261],{"type":44,"tag":59,"props":1262,"children":1264},{"className":1263},[],[1265],{"type":50,"value":580},{"type":44,"tag":1253,"props":1267,"children":1268},{},[1269],{"type":44,"tag":59,"props":1270,"children":1272},{"className":1271},[],[1273],{"type":50,"value":1274},"--dry-run=client",{"type":44,"tag":1276,"props":1277,"children":1278},"tbody",{},[1279,1315],{"type":44,"tag":1249,"props":1280,"children":1281},{},[1282,1301,1306],{"type":44,"tag":1283,"props":1284,"children":1285},"td",{},[1286,1288,1294,1295],{"type":50,"value":1287},"No ",{"type":44,"tag":59,"props":1289,"children":1291},{"className":1290},[],[1292],{"type":50,"value":1293},"policy.preset",{"type":50,"value":428},{"type":44,"tag":59,"props":1296,"children":1298},{"className":1297},[],[1299],{"type":50,"value":1300},"policy.profile",{"type":44,"tag":1283,"props":1302,"children":1303},{},[1304],{"type":50,"value":1305},"is valid",{"type":44,"tag":1283,"props":1307,"children":1308},{},[1309],{"type":44,"tag":59,"props":1310,"children":1312},{"className":1311},[],[1313],{"type":50,"value":1314},"policy.profile or policy.preset is required",{"type":44,"tag":1249,"props":1316,"children":1317},{},[1318,1329,1333],{"type":44,"tag":1283,"props":1319,"children":1320},{},[1321,1327],{"type":44,"tag":59,"props":1322,"children":1324},{"className":1323},[],[1325],{"type":50,"value":1326},"processes_per_node: 99",{"type":50,"value":1328}," on an 8-GPU preset",{"type":44,"tag":1283,"props":1330,"children":1331},{},[1332],{"type":50,"value":1305},{"type":44,"tag":1283,"props":1334,"children":1335},{},[1336],{"type":44,"tag":59,"props":1337,"children":1339},{"className":1338},[],[1340],{"type":50,"value":1341},"processes_per_node (99) exceeds profile GPU count (8)",{"type":44,"tag":53,"props":1343,"children":1344},{},[1345,1347,1352],{"type":50,"value":1346},"Both are free and offline, so run both before telling a user a config is good.\nNote that dry-run resolves ",{"type":44,"tag":59,"props":1348,"children":1350},{"className":1349},[],[1351],{"type":50,"value":785},{"type":50,"value":1353}," on disk, so run it from the config's\ndirectory or the script will appear missing.",{"type":44,"tag":53,"props":1355,"children":1356},{},[1357,1363,1365,1371,1373,1379],{"type":44,"tag":59,"props":1358,"children":1360},{"className":1359},[],[1361],{"type":50,"value":1362},"--config",{"type":50,"value":1364}," takes an explicit path; a bare ",{"type":44,"tag":59,"props":1366,"children":1368},{"className":1367},[],[1369],{"type":50,"value":1370},"TARGET",{"type":50,"value":1372}," resolves ",{"type":44,"tag":59,"props":1374,"children":1376},{"className":1375},[],[1377],{"type":50,"value":1378},"tau\u002FTARGET.yaml",{"type":50,"value":1380},".\nThey are not interchangeable.",{"type":44,"tag":1382,"props":1383,"children":1385},"h3",{"id":1384},"engine-choice-constrains-everything-else",[1386],{"type":50,"value":1387},"Engine choice constrains everything else",{"type":44,"tag":53,"props":1389,"children":1390},{},[1391,1397,1399,1405,1407,1413,1415,1421],{"type":44,"tag":59,"props":1392,"children":1394},{"className":1393},[],[1395],{"type":50,"value":1396},"engine: job",{"type":50,"value":1398}," renders a ",{"type":44,"tag":59,"props":1400,"children":1402},{"className":1401},[],[1403],{"type":50,"value":1404},"batch\u002Fv1",{"type":50,"value":1406}," Job — one pod, or an Indexed Job with\ntorchrun. ",{"type":44,"tag":59,"props":1408,"children":1410},{"className":1409},[],[1411],{"type":50,"value":1412},"engine: rayjob",{"type":50,"value":1414}," renders a KubeRay RayJob (head + ",{"type":44,"tag":59,"props":1416,"children":1418},{"className":1417},[],[1419],{"type":50,"value":1420},"compute.workers",{"type":50,"value":1422},"\nworkers). Mixing the two vocabularies is the most common authoring failure:",{"type":44,"tag":1241,"props":1424,"children":1425},{},[1426,1447],{"type":44,"tag":1245,"props":1427,"children":1428},{},[1429],{"type":44,"tag":1249,"props":1430,"children":1431},{},[1432,1437,1442],{"type":44,"tag":1253,"props":1433,"children":1434},{},[1435],{"type":50,"value":1436},"Intent",{"type":44,"tag":1253,"props":1438,"children":1439},{},[1440],{"type":50,"value":1441},"Correct",{"type":44,"tag":1253,"props":1443,"children":1444},{},[1445],{"type":50,"value":1446},"Common mistake",{"type":44,"tag":1276,"props":1448,"children":1449},{},[1450,1497,1534,1567],{"type":44,"tag":1249,"props":1451,"children":1452},{},[1453,1458,1481],{"type":44,"tag":1283,"props":1454,"children":1455},{},[1456],{"type":50,"value":1457},"Multi-node PyTorch DDP",{"type":44,"tag":1283,"props":1459,"children":1460},{},[1461,1466,1468,1474,1475],{"type":44,"tag":59,"props":1462,"children":1464},{"className":1463},[],[1465],{"type":50,"value":1396},{"type":50,"value":1467}," + ",{"type":44,"tag":59,"props":1469,"children":1471},{"className":1470},[],[1472],{"type":50,"value":1473},"launcher: torchrun",{"type":50,"value":1467},{"type":44,"tag":59,"props":1476,"children":1478},{"className":1477},[],[1479],{"type":50,"value":1480},"execution.nodes: N",{"type":44,"tag":1283,"props":1482,"children":1483},{},[1484,1490,1492],{"type":44,"tag":59,"props":1485,"children":1487},{"className":1486},[],[1488],{"type":50,"value":1489},"execution.nodes",{"type":50,"value":1491}," with ",{"type":44,"tag":59,"props":1493,"children":1495},{"className":1494},[],[1496],{"type":50,"value":1412},{"type":44,"tag":1249,"props":1498,"children":1499},{},[1500,1505,1520],{"type":44,"tag":1283,"props":1501,"children":1502},{},[1503],{"type":50,"value":1504},"Ray Train distributed",{"type":44,"tag":1283,"props":1506,"children":1507},{},[1508,1513,1514],{"type":44,"tag":59,"props":1509,"children":1511},{"className":1510},[],[1512],{"type":50,"value":1412},{"type":50,"value":1467},{"type":44,"tag":59,"props":1515,"children":1517},{"className":1516},[],[1518],{"type":50,"value":1519},"compute.workers: N",{"type":44,"tag":1283,"props":1521,"children":1522},{},[1523,1528,1529],{"type":44,"tag":59,"props":1524,"children":1526},{"className":1525},[],[1527],{"type":50,"value":1473},{"type":50,"value":1491},{"type":44,"tag":59,"props":1530,"children":1532},{"className":1531},[],[1533],{"type":50,"value":1412},{"type":44,"tag":1249,"props":1535,"children":1536},{},[1537,1542,1552],{"type":44,"tag":1283,"props":1538,"children":1539},{},[1540],{"type":50,"value":1541},"GPUs per pod on a Job",{"type":44,"tag":1283,"props":1543,"children":1544},{},[1545,1550],{"type":44,"tag":59,"props":1546,"children":1548},{"className":1547},[],[1549],{"type":50,"value":1293},{"type":50,"value":1551}," (the node shape)",{"type":44,"tag":1283,"props":1553,"children":1554},{},[1555,1561,1562],{"type":44,"tag":59,"props":1556,"children":1558},{"className":1557},[],[1559],{"type":50,"value":1560},"compute.gpus_per_worker",{"type":50,"value":1491},{"type":44,"tag":59,"props":1563,"children":1565},{"className":1564},[],[1566],{"type":50,"value":1396},{"type":44,"tag":1249,"props":1568,"children":1569},{},[1570,1575,1590],{"type":44,"tag":1283,"props":1571,"children":1572},{},[1573],{"type":50,"value":1574},"Extra PVC mounts",{"type":44,"tag":1283,"props":1576,"children":1577},{},[1578,1583,1584],{"type":44,"tag":59,"props":1579,"children":1581},{"className":1580},[],[1582],{"type":50,"value":1396},{"type":50,"value":1467},{"type":44,"tag":59,"props":1585,"children":1587},{"className":1586},[],[1588],{"type":50,"value":1589},"storage.mounts",{"type":44,"tag":1283,"props":1591,"children":1592},{},[1593,1598,1599],{"type":44,"tag":59,"props":1594,"children":1596},{"className":1595},[],[1597],{"type":50,"value":1589},{"type":50,"value":1491},{"type":44,"tag":59,"props":1600,"children":1602},{"className":1601},[],[1603],{"type":50,"value":1412},{"type":44,"tag":53,"props":1605,"children":1606},{},[1607,1612,1614,1620,1621,1627,1628,1634,1635,1641,1643,1649,1651,1661,1663,1669,1671,1676,1678,1684,1686,1692,1694,1699,1700,1706,1707,1712,1713,1719,1720,1726],{"type":44,"tag":59,"props":1608,"children":1610},{"className":1609},[],[1611],{"type":50,"value":862},{"type":50,"value":1613},"'s Ray-shaped fields — ",{"type":44,"tag":59,"props":1615,"children":1617},{"className":1616},[],[1618],{"type":50,"value":1619},"workers",{"type":50,"value":207},{"type":44,"tag":59,"props":1622,"children":1624},{"className":1623},[],[1625],{"type":50,"value":1626},"gpus_per_worker",{"type":50,"value":207},{"type":44,"tag":59,"props":1629,"children":1631},{"className":1630},[],[1632],{"type":50,"value":1633},"runtime.pip",{"type":50,"value":384},{"type":44,"tag":59,"props":1636,"children":1638},{"className":1637},[],[1639],{"type":50,"value":1640},"head_*",{"type":50,"value":1642},"\u002F",{"type":44,"tag":59,"props":1644,"children":1646},{"className":1645},[],[1647],{"type":50,"value":1648},"worker_*",{"type":50,"value":1650}," — are ",{"type":44,"tag":68,"props":1652,"children":1653},{},[1654,1656],{"type":50,"value":1655},"rejected on ",{"type":44,"tag":59,"props":1657,"children":1659},{"className":1658},[],[1660],{"type":50,"value":1396},{"type":50,"value":1662},". For a Job, GPU count\ncomes from the resolved preset, and ",{"type":44,"tag":59,"props":1664,"children":1666},{"className":1665},[],[1667],{"type":50,"value":1668},"execution.processes_per_node",{"type":50,"value":1670}," is validated\nagainst it. \"8 GPUs per node\" feels like a ",{"type":44,"tag":59,"props":1672,"children":1674},{"className":1673},[],[1675],{"type":50,"value":862},{"type":50,"value":1677}," concern but is a placement\none. ",{"type":44,"tag":59,"props":1679,"children":1681},{"className":1680},[],[1682],{"type":50,"value":1683},"execution.launcher",{"type":50,"value":1685}," is engine-scoped: ",{"type":44,"tag":59,"props":1687,"children":1689},{"className":1688},[],[1690],{"type":50,"value":1691},"job",{"type":50,"value":1693}," takes ",{"type":44,"tag":59,"props":1695,"children":1697},{"className":1696},[],[1698],{"type":50,"value":397},{"type":50,"value":1642},{"type":44,"tag":59,"props":1701,"children":1703},{"className":1702},[],[1704],{"type":50,"value":1705},"torchrun",{"type":50,"value":384},{"type":44,"tag":59,"props":1708,"children":1710},{"className":1709},[],[1711],{"type":50,"value":470},{"type":50,"value":1693},{"type":44,"tag":59,"props":1714,"children":1716},{"className":1715},[],[1717],{"type":50,"value":1718},"ray-train",{"type":50,"value":1642},{"type":44,"tag":59,"props":1721,"children":1723},{"className":1722},[],[1724],{"type":50,"value":1725},"ray-tune",{"type":50,"value":169},{"type":44,"tag":53,"props":1728,"children":1729},{},[1730,1732,1738,1739,1745,1746,1752,1753,1759,1760,1766,1768,1774,1776,1782,1784,1790,1792,1797],{"type":50,"value":1731},"Tau owns the distributed-training env vars. ",{"type":44,"tag":59,"props":1733,"children":1735},{"className":1734},[],[1736],{"type":50,"value":1737},"MASTER_ADDR",{"type":50,"value":207},{"type":44,"tag":59,"props":1740,"children":1742},{"className":1741},[],[1743],{"type":50,"value":1744},"MASTER_PORT",{"type":50,"value":384},{"type":44,"tag":59,"props":1747,"children":1749},{"className":1748},[],[1750],{"type":50,"value":1751},"TAU_WORLD_SIZE",{"type":50,"value":207},{"type":44,"tag":59,"props":1754,"children":1756},{"className":1755},[],[1757],{"type":50,"value":1758},"TAU_DIST_BACKEND",{"type":50,"value":207},{"type":44,"tag":59,"props":1761,"children":1763},{"className":1762},[],[1764],{"type":50,"value":1765},"TAU_NUM_WORKERS",{"type":50,"value":1767},", and ",{"type":44,"tag":59,"props":1769,"children":1771},{"className":1770},[],[1772],{"type":50,"value":1773},"NCCL_*",{"type":50,"value":1775}," are\nrejected in ",{"type":44,"tag":59,"props":1777,"children":1779},{"className":1778},[],[1780],{"type":50,"value":1781},"runtime.env",{"type":50,"value":1783}," so rendezvous stays consistent with the rendered\ntopology. ",{"type":44,"tag":59,"props":1785,"children":1787},{"className":1786},[],[1788],{"type":50,"value":1789},"execution.allow_nccl_override: true",{"type":50,"value":1791}," unblocks only the ",{"type":44,"tag":59,"props":1793,"children":1795},{"className":1794},[],[1796],{"type":50,"value":1773},{"type":50,"value":1798},"\nones, for deliberate tuning.",{"type":44,"tag":1382,"props":1800,"children":1802},{"id":1801},"secrets",[1803],{"type":50,"value":1804},"Secrets",{"type":44,"tag":53,"props":1806,"children":1807},{},[1808],{"type":50,"value":1809},"Never put a secret value in a run config:",{"type":44,"tag":130,"props":1811,"children":1812},{},[1813,1838],{"type":44,"tag":134,"props":1814,"children":1815},{},[1816,1822,1824,1830,1831,1837],{"type":44,"tag":59,"props":1817,"children":1819},{"className":1818},[],[1820],{"type":50,"value":1821},"runtime.env_secret",{"type":50,"value":1823}," — ",{"type":44,"tag":59,"props":1825,"children":1827},{"className":1826},[],[1828],{"type":50,"value":1829},"KEY: \"secret-name:key\"",{"type":50,"value":530},{"type":44,"tag":59,"props":1832,"children":1834},{"className":1833},[],[1835],{"type":50,"value":1836},"valueFrom.secretKeyRef",{"type":50,"value":169},{"type":44,"tag":134,"props":1839,"children":1840},{},[1841,1847],{"type":44,"tag":59,"props":1842,"children":1844},{"className":1843},[],[1845],{"type":50,"value":1846},"runtime.env_kv",{"type":50,"value":1848}," — Azure Key Vault via Secrets Store CSI; all entries must\nresolve to one vault.",{"type":44,"tag":53,"props":1850,"children":1851},{},[1852],{"type":50,"value":1853},"Client dry-run redacts both while keeping the dependency shape visible.",{"type":44,"tag":118,"props":1855,"children":1857},{"id":1856},"running-and-observing",[1858],{"type":50,"value":1859},"Running and observing",{"type":44,"tag":233,"props":1861,"children":1863},{"className":235,"code":1862,"language":237,"meta":238,"style":238},"tau run train # submit the checked-in target\ntau run status \u003Crun-name> --watch # startup phase tree, live\ntau run logs \u003Crun-name> # Ray driver output, or Job pod logs\ntau run get \u003Crun-name> # durable results recorded by storage.output\ntau run cancel \u003Crun-name> # delete workload; Kueue reclaims quota\ntau run list -n \u003Cnamespace> # Tau-managed Jobs and RayJobs\n",[1864],{"type":44,"tag":59,"props":1865,"children":1866},{"__ignoreMap":238},[1867,1887,1934,1971,2008,2045],{"type":44,"tag":244,"props":1868,"children":1869},{"class":246,"line":247},[1870,1874,1878,1882],{"type":44,"tag":244,"props":1871,"children":1872},{"style":251},[1873],{"type":50,"value":64},{"type":44,"tag":244,"props":1875,"children":1876},{"style":256},[1877],{"type":50,"value":278},{"type":44,"tag":244,"props":1879,"children":1880},{"style":256},[1881],{"type":50,"value":750},{"type":44,"tag":244,"props":1883,"children":1884},{"style":262},[1885],{"type":50,"value":1886}," # submit the checked-in target\n",{"type":44,"tag":244,"props":1888,"children":1889},{"class":246,"line":268},[1890,1894,1898,1903,1908,1913,1919,1924,1929],{"type":44,"tag":244,"props":1891,"children":1892},{"style":251},[1893],{"type":50,"value":64},{"type":44,"tag":244,"props":1895,"children":1896},{"style":256},[1897],{"type":50,"value":278},{"type":44,"tag":244,"props":1899,"children":1900},{"style":256},[1901],{"type":50,"value":1902}," status",{"type":44,"tag":244,"props":1904,"children":1905},{"style":742},[1906],{"type":50,"value":1907}," \u003C",{"type":44,"tag":244,"props":1909,"children":1910},{"style":256},[1911],{"type":50,"value":1912},"run-nam",{"type":44,"tag":244,"props":1914,"children":1916},{"style":1915},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1917],{"type":50,"value":1918},"e",{"type":44,"tag":244,"props":1920,"children":1921},{"style":742},[1922],{"type":50,"value":1923},">",{"type":44,"tag":244,"props":1925,"children":1926},{"style":256},[1927],{"type":50,"value":1928}," --watch",{"type":44,"tag":244,"props":1930,"children":1931},{"style":262},[1932],{"type":50,"value":1933}," # startup phase tree, live\n",{"type":44,"tag":244,"props":1935,"children":1936},{"class":246,"line":301},[1937,1941,1945,1950,1954,1958,1962,1966],{"type":44,"tag":244,"props":1938,"children":1939},{"style":251},[1940],{"type":50,"value":64},{"type":44,"tag":244,"props":1942,"children":1943},{"style":256},[1944],{"type":50,"value":278},{"type":44,"tag":244,"props":1946,"children":1947},{"style":256},[1948],{"type":50,"value":1949}," logs",{"type":44,"tag":244,"props":1951,"children":1952},{"style":742},[1953],{"type":50,"value":1907},{"type":44,"tag":244,"props":1955,"children":1956},{"style":256},[1957],{"type":50,"value":1912},{"type":44,"tag":244,"props":1959,"children":1960},{"style":1915},[1961],{"type":50,"value":1918},{"type":44,"tag":244,"props":1963,"children":1964},{"style":742},[1965],{"type":50,"value":1923},{"type":44,"tag":244,"props":1967,"children":1968},{"style":262},[1969],{"type":50,"value":1970}," # Ray driver output, or Job pod logs\n",{"type":44,"tag":244,"props":1972,"children":1973},{"class":246,"line":802},[1974,1978,1982,1987,1991,1995,1999,2003],{"type":44,"tag":244,"props":1975,"children":1976},{"style":251},[1977],{"type":50,"value":64},{"type":44,"tag":244,"props":1979,"children":1980},{"style":256},[1981],{"type":50,"value":278},{"type":44,"tag":244,"props":1983,"children":1984},{"style":256},[1985],{"type":50,"value":1986}," get",{"type":44,"tag":244,"props":1988,"children":1989},{"style":742},[1990],{"type":50,"value":1907},{"type":44,"tag":244,"props":1992,"children":1993},{"style":256},[1994],{"type":50,"value":1912},{"type":44,"tag":244,"props":1996,"children":1997},{"style":1915},[1998],{"type":50,"value":1918},{"type":44,"tag":244,"props":2000,"children":2001},{"style":742},[2002],{"type":50,"value":1923},{"type":44,"tag":244,"props":2004,"children":2005},{"style":262},[2006],{"type":50,"value":2007}," # durable results recorded by storage.output\n",{"type":44,"tag":244,"props":2009,"children":2010},{"class":246,"line":27},[2011,2015,2019,2024,2028,2032,2036,2040],{"type":44,"tag":244,"props":2012,"children":2013},{"style":251},[2014],{"type":50,"value":64},{"type":44,"tag":244,"props":2016,"children":2017},{"style":256},[2018],{"type":50,"value":278},{"type":44,"tag":244,"props":2020,"children":2021},{"style":256},[2022],{"type":50,"value":2023}," cancel",{"type":44,"tag":244,"props":2025,"children":2026},{"style":742},[2027],{"type":50,"value":1907},{"type":44,"tag":244,"props":2029,"children":2030},{"style":256},[2031],{"type":50,"value":1912},{"type":44,"tag":244,"props":2033,"children":2034},{"style":1915},[2035],{"type":50,"value":1918},{"type":44,"tag":244,"props":2037,"children":2038},{"style":742},[2039],{"type":50,"value":1923},{"type":44,"tag":244,"props":2041,"children":2042},{"style":262},[2043],{"type":50,"value":2044}," # delete workload; Kueue reclaims quota\n",{"type":44,"tag":244,"props":2046,"children":2047},{"class":246,"line":825},[2048,2052,2056,2061,2066,2070,2075,2079,2083],{"type":44,"tag":244,"props":2049,"children":2050},{"style":251},[2051],{"type":50,"value":64},{"type":44,"tag":244,"props":2053,"children":2054},{"style":256},[2055],{"type":50,"value":278},{"type":44,"tag":244,"props":2057,"children":2058},{"style":256},[2059],{"type":50,"value":2060}," list",{"type":44,"tag":244,"props":2062,"children":2063},{"style":256},[2064],{"type":50,"value":2065}," -n",{"type":44,"tag":244,"props":2067,"children":2068},{"style":742},[2069],{"type":50,"value":1907},{"type":44,"tag":244,"props":2071,"children":2072},{"style":256},[2073],{"type":50,"value":2074},"namespac",{"type":44,"tag":244,"props":2076,"children":2077},{"style":1915},[2078],{"type":50,"value":1918},{"type":44,"tag":244,"props":2080,"children":2081},{"style":742},[2082],{"type":50,"value":1923},{"type":44,"tag":244,"props":2084,"children":2085},{"style":262},[2086],{"type":50,"value":2087}," # Tau-managed Jobs and RayJobs\n",{"type":44,"tag":53,"props":2089,"children":2090},{},[2091,2093,2099,2101,2105],{"type":50,"value":2092},"On a repository's first cluster-backed ",{"type":44,"tag":59,"props":2094,"children":2096},{"className":2095},[],[2097],{"type":50,"value":2098},"tau run",{"type":50,"value":2100},", Tau resolves credentials\nthrough the descriptor's access method: it either isolates an existing\nkubeconfig context or obtains AKS credentials with the user's Azure identity.\nThe dedicated kubeconfig avoids mutating their main one; it is ",{"type":44,"tag":68,"props":2102,"children":2103},{},[2104],{"type":50,"value":109},{"type":50,"value":2106}," a\nresearcher-isolation boundary and should not be described as one.",{"type":44,"tag":53,"props":2108,"children":2109},{},[2110,2115],{"type":44,"tag":59,"props":2111,"children":2113},{"className":2112},[],[2114],{"type":50,"value":560},{"type":50,"value":2116}," is the canonical lifecycle view. It walks an ordered phase\ntree — Submitted, Kueue admission, (RayCluster), pod scheduling, DRA\nallocation, image pull, init containers, container start, Ready, (RayJob\nstatus) — where each phase reports pending, active, done, warning, or skipped.",{"type":44,"tag":53,"props":2118,"children":2119},{},[2120,2132],{"type":44,"tag":68,"props":2121,"children":2122},{},[2123,2125,2131],{"type":50,"value":2124},"Read it top to bottom and stop at the first phase that is not ",{"type":44,"tag":59,"props":2126,"children":2128},{"className":2127},[],[2129],{"type":50,"value":2130},"done",{"type":50,"value":169},{"type":50,"value":2133}," That\nphase is the layer to investigate; everything after it is downstream noise.",{"type":44,"tag":53,"props":2135,"children":2136},{},[2137],{"type":50,"value":2138},"Three distinctions that matter when interpreting it:",{"type":44,"tag":130,"props":2140,"children":2141},{},[2142,2152,2162],{"type":44,"tag":134,"props":2143,"children":2144},{},[2145,2150],{"type":44,"tag":68,"props":2146,"children":2147},{},[2148],{"type":50,"value":2149},"Admitted ≠ scheduled.",{"type":50,"value":2151}," Kueue reserved quota; no pod exists yet.",{"type":44,"tag":134,"props":2153,"children":2154},{},[2155,2160],{"type":44,"tag":68,"props":2156,"children":2157},{},[2158],{"type":50,"value":2159},"Running ≠ progressing.",{"type":50,"value":2161}," Containers started; the training loop may be hung.",{"type":44,"tag":134,"props":2163,"children":2164},{},[2165,2170],{"type":44,"tag":68,"props":2166,"children":2167},{},[2168],{"type":50,"value":2169},"Completed ≠ evidence preserved.",{"type":50,"value":2171}," Check that artifacts actually landed.",{"type":44,"tag":53,"props":2173,"children":2174},{},[2175,2177,2183],{"type":50,"value":2176},"A ",{"type":44,"tag":59,"props":2178,"children":2180},{"className":2179},[],[2181],{"type":50,"value":2182},"skipped",{"type":50,"value":2184}," DRA phase is normal when the workload requests GPUs through the\ndevice plugin rather than DRA. It is not a failure.",{"type":44,"tag":118,"props":2186,"children":2188},{"id":2187},"diagnosing-a-stuck-or-failed-run",[2189],{"type":50,"value":2190},"Diagnosing a stuck or failed run",{"type":44,"tag":53,"props":2192,"children":2193},{},[2194,2196,2202],{"type":50,"value":2195},"Work the layers in order. Jumping to ",{"type":44,"tag":59,"props":2197,"children":2199},{"className":2198},[],[2200],{"type":50,"value":2201},"kubectl describe pod",{"type":50,"value":2203}," when the real\nproblem is quota (layer 4) or a Degraded workspace (layer 2) wastes time and\nproduces a misdiagnosis.",{"type":44,"tag":1241,"props":2205,"children":2206},{},[2207,2233],{"type":44,"tag":1245,"props":2208,"children":2209},{},[2210],{"type":44,"tag":1249,"props":2211,"children":2212},{},[2213,2218,2223,2228],{"type":44,"tag":1253,"props":2214,"children":2215},{},[2216],{"type":50,"value":2217},"#",{"type":44,"tag":1253,"props":2219,"children":2220},{},[2221],{"type":50,"value":2222},"Layer",{"type":44,"tag":1253,"props":2224,"children":2225},{},[2226],{"type":50,"value":2227},"Command",{"type":44,"tag":1253,"props":2229,"children":2230},{},[2231],{"type":50,"value":2232},"Owner if it fails",{"type":44,"tag":1276,"props":2234,"children":2235},{},[2236,2273,2299,2325,2354,2381,2416,2450],{"type":44,"tag":1249,"props":2237,"children":2238},{},[2239,2244,2249,2268],{"type":44,"tag":1283,"props":2240,"children":2241},{},[2242],{"type":50,"value":2243},"1",{"type":44,"tag":1283,"props":2245,"children":2246},{},[2247],{"type":50,"value":2248},"Repo\u002Fconnection resolution and access",{"type":44,"tag":1283,"props":2250,"children":2251},{},[2252,2258,2260,2266],{"type":44,"tag":59,"props":2253,"children":2255},{"className":2254},[],[2256],{"type":50,"value":2257},"tau workspace connection",{"type":50,"value":2259}," (",{"type":44,"tag":59,"props":2261,"children":2263},{"className":2262},[],[2264],{"type":50,"value":2265},"--offline",{"type":50,"value":2267}," for local configuration only)",{"type":44,"tag":1283,"props":2269,"children":2270},{},[2271],{"type":50,"value":2272},"Researcher (descriptor) \u002F platform (access)",{"type":44,"tag":1249,"props":2274,"children":2275},{},[2276,2281,2286,2295],{"type":44,"tag":1283,"props":2277,"children":2278},{},[2279],{"type":50,"value":2280},"2",{"type":44,"tag":1283,"props":2282,"children":2283},{},[2284],{"type":50,"value":2285},"TauWorkspace readiness",{"type":44,"tag":1283,"props":2287,"children":2288},{},[2289],{"type":44,"tag":59,"props":2290,"children":2292},{"className":2291},[],[2293],{"type":50,"value":2294},"tau workspace status \u003Cname>",{"type":44,"tag":1283,"props":2296,"children":2297},{},[2298],{"type":50,"value":159},{"type":44,"tag":1249,"props":2300,"children":2301},{},[2302,2307,2312,2321],{"type":44,"tag":1283,"props":2303,"children":2304},{},[2305],{"type":50,"value":2306},"3",{"type":44,"tag":1283,"props":2308,"children":2309},{},[2310],{"type":50,"value":2311},"Config validation and render",{"type":44,"tag":1283,"props":2313,"children":2314},{},[2315],{"type":44,"tag":59,"props":2316,"children":2318},{"className":2317},[],[2319],{"type":50,"value":2320},"tau run validate --config \u003Cpath>",{"type":44,"tag":1283,"props":2322,"children":2323},{},[2324],{"type":50,"value":141},{"type":44,"tag":1249,"props":2326,"children":2327},{},[2328,2333,2338,2349],{"type":44,"tag":1283,"props":2329,"children":2330},{},[2331],{"type":50,"value":2332},"4",{"type":44,"tag":1283,"props":2334,"children":2335},{},[2336],{"type":50,"value":2337},"Kueue admission and quota",{"type":44,"tag":1283,"props":2339,"children":2340},{},[2341,2347],{"type":44,"tag":59,"props":2342,"children":2344},{"className":2343},[],[2345],{"type":50,"value":2346},"tau run status \u003Crun>",{"type":50,"value":2348}," (admission phase)",{"type":44,"tag":1283,"props":2350,"children":2351},{},[2352],{"type":50,"value":2353},"Queue owner",{"type":44,"tag":1249,"props":2355,"children":2356},{},[2357,2362,2367,2376],{"type":44,"tag":1283,"props":2358,"children":2359},{},[2360],{"type":50,"value":2361},"5",{"type":44,"tag":1283,"props":2363,"children":2364},{},[2365],{"type":50,"value":2366},"Scheduling, DRA, image pull, init",{"type":44,"tag":1283,"props":2368,"children":2369},{},[2370],{"type":44,"tag":59,"props":2371,"children":2373},{"className":2372},[],[2374],{"type":50,"value":2375},"tau run status \u003Crun> --watch",{"type":44,"tag":1283,"props":2377,"children":2378},{},[2379],{"type":50,"value":2380},"Platform or researcher",{"type":44,"tag":1249,"props":2382,"children":2383},{},[2384,2389,2394,2411],{"type":44,"tag":1283,"props":2385,"children":2386},{},[2387],{"type":50,"value":2388},"6",{"type":44,"tag":1283,"props":2390,"children":2391},{},[2392],{"type":50,"value":2393},"GPU\u002Fnode\u002Ftopology health",{"type":44,"tag":1283,"props":2395,"children":2396},{},[2397,2403,2405],{"type":44,"tag":59,"props":2398,"children":2400},{"className":2399},[],[2401],{"type":50,"value":2402},"tau cluster validate nodes",{"type":50,"value":2404}," \u002F ",{"type":44,"tag":59,"props":2406,"children":2408},{"className":2407},[],[2409],{"type":50,"value":2410},"... topology",{"type":44,"tag":1283,"props":2412,"children":2413},{},[2414],{"type":50,"value":2415},"Node-pool operator",{"type":44,"tag":1249,"props":2417,"children":2418},{},[2419,2424,2429,2445],{"type":44,"tag":1283,"props":2420,"children":2421},{},[2422],{"type":50,"value":2423},"7",{"type":44,"tag":1283,"props":2425,"children":2426},{},[2427],{"type":50,"value":2428},"Runtime progress and evidence",{"type":44,"tag":1283,"props":2430,"children":2431},{},[2432,2438,2439],{"type":44,"tag":59,"props":2433,"children":2435},{"className":2434},[],[2436],{"type":50,"value":2437},"tau run logs \u003Crun>",{"type":50,"value":1467},{"type":44,"tag":59,"props":2440,"children":2442},{"className":2441},[],[2443],{"type":50,"value":2444},"taugrid-portal experiment status \u003Cname>",{"type":44,"tag":1283,"props":2446,"children":2447},{},[2448],{"type":50,"value":2449},"Researcher \u002F platform",{"type":44,"tag":1249,"props":2451,"children":2452},{},[2453,2458,2463,2468],{"type":44,"tag":1283,"props":2454,"children":2455},{},[2456],{"type":50,"value":2457},"8",{"type":44,"tag":1283,"props":2459,"children":2460},{},[2461],{"type":50,"value":2462},"Recovery",{"type":44,"tag":1283,"props":2464,"children":2465},{},[2466],{"type":50,"value":2467},"see retry\u002Fresume below",{"type":44,"tag":1283,"props":2469,"children":2470},{},[2471],{"type":50,"value":2472},"—",{"type":44,"tag":53,"props":2474,"children":2475},{},[2476,2478,2483,2485,2490],{"type":50,"value":2477},"Layers 1–3 are offline and cost nothing, so run them first when a symptom is\nambiguous. Full per-layer detail, including the operator-only ",{"type":44,"tag":59,"props":2479,"children":2481},{"className":2480},[],[2482],{"type":50,"value":672},{"type":50,"value":2484},"\ncommands for each, is in\n",{"type":44,"tag":163,"props":2486,"children":2488},{"href":2487},"references\u002Ftroubleshooting.md",[2489],{"type":50,"value":2487},{"type":50,"value":169},{"type":44,"tag":53,"props":2492,"children":2493},{},[2494],{"type":50,"value":2495},"Two guardrails worth stating to users directly:",{"type":44,"tag":130,"props":2497,"children":2498},{},[2499,2504],{"type":44,"tag":134,"props":2500,"children":2501},{},[2502],{"type":50,"value":2503},"Do not add namespace, queue, kubeconfig, or cloud credentials to a project\nconfig to work around a platform readiness failure. Those are workspace\nconcerns, the submission gate is enforced server-side, and there is no\nclient-side bypass.",{"type":44,"tag":134,"props":2505,"children":2506},{},[2507],{"type":50,"value":2508},"Do not resubmit blindly after a failure. Locate the first failed layer, then\nchoose retry or resume deliberately.",{"type":44,"tag":118,"props":2510,"children":2512},{"id":2511},"retry-and-resume",[2513],{"type":50,"value":2514},"Retry and resume",{"type":44,"tag":53,"props":2516,"children":2517},{},[2518],{"type":50,"value":2519},"Automatic retry is configuration, not a command:",{"type":44,"tag":233,"props":2521,"children":2523},{"className":724,"code":2522,"language":726,"meta":238,"style":238},"resilience:\n max_retries: 2\n retry_on: [\"Preempted\", \"Evicted\"] # default; OOMKilled is opt-in\n backoff_initial: 30s\n backoff_max: 5m\n checkpoint_path: \u002Fdata\u002Fcheckpoints\u002Ffinetunes\u002F\u003Cname>\n",[2524],{"type":44,"tag":59,"props":2525,"children":2526},{"__ignoreMap":238},[2527,2538,2555,2615,2632,2649],{"type":44,"tag":244,"props":2528,"children":2529},{"class":246,"line":247},[2530,2534],{"type":44,"tag":244,"props":2531,"children":2532},{"style":736},[2533],{"type":50,"value":1052},{"type":44,"tag":244,"props":2535,"children":2536},{"style":742},[2537],{"type":50,"value":822},{"type":44,"tag":244,"props":2539,"children":2540},{"class":246,"line":268},[2541,2546,2550],{"type":44,"tag":244,"props":2542,"children":2543},{"style":736},[2544],{"type":50,"value":2545}," max_retries",{"type":44,"tag":244,"props":2547,"children":2548},{"style":742},[2549],{"type":50,"value":745},{"type":44,"tag":244,"props":2551,"children":2552},{"style":882},[2553],{"type":50,"value":2554}," 2\n",{"type":44,"tag":244,"props":2556,"children":2557},{"class":246,"line":301},[2558,2563,2567,2572,2577,2582,2586,2591,2596,2601,2605,2610],{"type":44,"tag":244,"props":2559,"children":2560},{"style":736},[2561],{"type":50,"value":2562}," retry_on",{"type":44,"tag":244,"props":2564,"children":2565},{"style":742},[2566],{"type":50,"value":745},{"type":44,"tag":244,"props":2568,"children":2569},{"style":742},[2570],{"type":50,"value":2571}," [",{"type":44,"tag":244,"props":2573,"children":2574},{"style":742},[2575],{"type":50,"value":2576},"\"",{"type":44,"tag":244,"props":2578,"children":2579},{"style":256},[2580],{"type":50,"value":2581},"Preempted",{"type":44,"tag":244,"props":2583,"children":2584},{"style":742},[2585],{"type":50,"value":2576},{"type":44,"tag":244,"props":2587,"children":2588},{"style":742},[2589],{"type":50,"value":2590},",",{"type":44,"tag":244,"props":2592,"children":2593},{"style":742},[2594],{"type":50,"value":2595}," \"",{"type":44,"tag":244,"props":2597,"children":2598},{"style":256},[2599],{"type":50,"value":2600},"Evicted",{"type":44,"tag":244,"props":2602,"children":2603},{"style":742},[2604],{"type":50,"value":2576},{"type":44,"tag":244,"props":2606,"children":2607},{"style":742},[2608],{"type":50,"value":2609},"]",{"type":44,"tag":244,"props":2611,"children":2612},{"style":262},[2613],{"type":50,"value":2614}," # default; OOMKilled is opt-in\n",{"type":44,"tag":244,"props":2616,"children":2617},{"class":246,"line":802},[2618,2623,2627],{"type":44,"tag":244,"props":2619,"children":2620},{"style":736},[2621],{"type":50,"value":2622}," backoff_initial",{"type":44,"tag":244,"props":2624,"children":2625},{"style":742},[2626],{"type":50,"value":745},{"type":44,"tag":244,"props":2628,"children":2629},{"style":256},[2630],{"type":50,"value":2631}," 30s\n",{"type":44,"tag":244,"props":2633,"children":2634},{"class":246,"line":27},[2635,2640,2644],{"type":44,"tag":244,"props":2636,"children":2637},{"style":736},[2638],{"type":50,"value":2639}," backoff_max",{"type":44,"tag":244,"props":2641,"children":2642},{"style":742},[2643],{"type":50,"value":745},{"type":44,"tag":244,"props":2645,"children":2646},{"style":256},[2647],{"type":50,"value":2648}," 5m\n",{"type":44,"tag":244,"props":2650,"children":2651},{"class":246,"line":825},[2652,2657,2661],{"type":44,"tag":244,"props":2653,"children":2654},{"style":736},[2655],{"type":50,"value":2656}," checkpoint_path",{"type":44,"tag":244,"props":2658,"children":2659},{"style":742},[2660],{"type":50,"value":745},{"type":44,"tag":244,"props":2662,"children":2663},{"style":256},[2664],{"type":50,"value":2665}," \u002Fdata\u002Fcheckpoints\u002Ffinetunes\u002F\u003Cname>\n",{"type":44,"tag":53,"props":2667,"children":2668},{},[2669,2671,2677,2679,2684,2686,2692,2694,2699],{"type":50,"value":2670},"When ",{"type":44,"tag":59,"props":2672,"children":2674},{"className":2673},[],[2675],{"type":50,"value":2676},"max_retries > 0",{"type":50,"value":2678}," and no dry-run is set, ",{"type":44,"tag":59,"props":2680,"children":2682},{"className":2681},[],[2683],{"type":50,"value":2098},{"type":50,"value":2685}," waits for terminal\nstate, classifies the failure, checks it against ",{"type":44,"tag":59,"props":2687,"children":2689},{"className":2688},[],[2690],{"type":50,"value":2691},"retry_on",{"type":50,"value":2693},", applies bounded\nexponential backoff, injects the checkpoint path and attempt number, then\ndeletes and resubmits. If the reason is not in ",{"type":44,"tag":59,"props":2695,"children":2697},{"className":2696},[],[2698],{"type":50,"value":2691},{"type":50,"value":2700},", Tau exits with an\nerror naming it — unexpected failures surface instead of looping.",{"type":44,"tag":53,"props":2702,"children":2703},{},[2704,2710,2712,2717,2719,2725],{"type":44,"tag":59,"props":2705,"children":2707},{"className":2706},[],[2708],{"type":50,"value":2709},"OOMKilled",{"type":50,"value":2711}," is deliberately excluded by default: the same config with the same\nmemory usually reproduces the same OOM, so retrying without changing anything\njust spends queue time to fail identically. Suggest adding it only after\n",{"type":44,"tag":59,"props":2713,"children":2715},{"className":2714},[],[2716],{"type":50,"value":862},{"type":50,"value":2718}," memory has been raised. ",{"type":44,"tag":59,"props":2720,"children":2722},{"className":2721},[],[2723],{"type":50,"value":2724},"Unknown",{"type":50,"value":2726}," is never retryable in either path.",{"type":44,"tag":53,"props":2728,"children":2729},{},[2730],{"type":50,"value":2731},"Manual resume:",{"type":44,"tag":233,"props":2733,"children":2735},{"className":235,"code":2734,"language":237,"meta":238,"style":238},"tau run resume \u003Crun-name> --config tau\u002Ftrain.yaml # --config is required\n",[2736],{"type":44,"tag":59,"props":2737,"children":2738},{"__ignoreMap":238},[2739],{"type":44,"tag":244,"props":2740,"children":2741},{"class":246,"line":247},[2742,2746,2750,2755,2759,2763,2767,2771,2775,2779],{"type":44,"tag":244,"props":2743,"children":2744},{"style":251},[2745],{"type":50,"value":64},{"type":44,"tag":244,"props":2747,"children":2748},{"style":256},[2749],{"type":50,"value":278},{"type":44,"tag":244,"props":2751,"children":2752},{"style":256},[2753],{"type":50,"value":2754}," resume",{"type":44,"tag":244,"props":2756,"children":2757},{"style":742},[2758],{"type":50,"value":1907},{"type":44,"tag":244,"props":2760,"children":2761},{"style":256},[2762],{"type":50,"value":1912},{"type":44,"tag":244,"props":2764,"children":2765},{"style":1915},[2766],{"type":50,"value":1918},{"type":44,"tag":244,"props":2768,"children":2769},{"style":742},[2770],{"type":50,"value":1923},{"type":44,"tag":244,"props":2772,"children":2773},{"style":256},[2774],{"type":50,"value":1192},{"type":44,"tag":244,"props":2776,"children":2777},{"style":256},[2778],{"type":50,"value":1197},{"type":44,"tag":244,"props":2780,"children":2781},{"style":262},[2782],{"type":50,"value":2783}," # --config is required\n",{"type":44,"tag":53,"props":2785,"children":2786},{},[2787,2789,2795,2797,2802,2804,2810],{"type":50,"value":2788},"Resume needs a durable checkpoint under ",{"type":44,"tag":59,"props":2790,"children":2792},{"className":2791},[],[2793],{"type":50,"value":2794},"\u002Fdata",{"type":50,"value":2796}," — a workload that only wrote to\nnode-local scratch has nothing to resume from, since that state does not\nsurvive workload deletion. If the original failure was ",{"type":44,"tag":59,"props":2798,"children":2800},{"className":2799},[],[2801],{"type":50,"value":2709},{"type":50,"value":2803},", resume\nrequires ",{"type":44,"tag":59,"props":2805,"children":2807},{"className":2806},[],[2808],{"type":50,"value":2809},"--force",{"type":50,"value":169},{"type":44,"tag":118,"props":2812,"children":2814},{"id":2813},"serving-a-model",[2815],{"type":50,"value":2816},"Serving a model",{"type":44,"tag":53,"props":2818,"children":2819},{},[2820,2826],{"type":44,"tag":59,"props":2821,"children":2823},{"className":2822},[],[2824],{"type":50,"value":2825},"tau serve",{"type":50,"value":2827}," renders a KubeRay RayService (default) or a plain Deployment. A\nservice is not a run — it does not inherit run lifecycle commands.",{"type":44,"tag":233,"props":2829,"children":2831},{"className":235,"code":2830,"language":237,"meta":238,"style":238},"tau serve deploy \u003Cname> --kind=rayservice --profile \u003Cp> --image \u003Cpinned> \\\n --import-path serve:app --checkpoint \u003Cpath> --checkpoint-pvc \u003Cpvc> \\\n --namespace \u003Cns> --context \u003Cctx> --dry-run=client\ntau serve status \u003Cname> --kind=rayservice -n \u003Cns>\ntau serve scale \u003Cname> --kind=deployment --replicas 3 -n \u003Cns>\ntau serve delete \u003Cname> --kind=rayservice -n \u003Cns>\n",[2832],{"type":44,"tag":59,"props":2833,"children":2834},{"__ignoreMap":238},[2835,2919,2982,3036,3092,3160],{"type":44,"tag":244,"props":2836,"children":2837},{"class":246,"line":247},[2838,2842,2847,2852,2856,2861,2865,2869,2874,2879,2883,2887,2891,2896,2900,2905,2910,2914],{"type":44,"tag":244,"props":2839,"children":2840},{"style":251},[2841],{"type":50,"value":64},{"type":44,"tag":244,"props":2843,"children":2844},{"style":256},[2845],{"type":50,"value":2846}," serve",{"type":44,"tag":244,"props":2848,"children":2849},{"style":256},[2850],{"type":50,"value":2851}," deploy",{"type":44,"tag":244,"props":2853,"children":2854},{"style":742},[2855],{"type":50,"value":1907},{"type":44,"tag":244,"props":2857,"children":2858},{"style":256},[2859],{"type":50,"value":2860},"nam",{"type":44,"tag":244,"props":2862,"children":2863},{"style":1915},[2864],{"type":50,"value":1918},{"type":44,"tag":244,"props":2866,"children":2867},{"style":742},[2868],{"type":50,"value":1923},{"type":44,"tag":244,"props":2870,"children":2871},{"style":256},[2872],{"type":50,"value":2873}," --kind=rayservice",{"type":44,"tag":244,"props":2875,"children":2876},{"style":256},[2877],{"type":50,"value":2878}," --profile",{"type":44,"tag":244,"props":2880,"children":2881},{"style":742},[2882],{"type":50,"value":1907},{"type":44,"tag":244,"props":2884,"children":2885},{"style":1915},[2886],{"type":50,"value":53},{"type":44,"tag":244,"props":2888,"children":2889},{"style":742},[2890],{"type":50,"value":1923},{"type":44,"tag":244,"props":2892,"children":2893},{"style":256},[2894],{"type":50,"value":2895}," --image",{"type":44,"tag":244,"props":2897,"children":2898},{"style":742},[2899],{"type":50,"value":1907},{"type":44,"tag":244,"props":2901,"children":2902},{"style":256},[2903],{"type":50,"value":2904},"pinne",{"type":44,"tag":244,"props":2906,"children":2907},{"style":1915},[2908],{"type":50,"value":2909},"d",{"type":44,"tag":244,"props":2911,"children":2912},{"style":742},[2913],{"type":50,"value":1923},{"type":44,"tag":244,"props":2915,"children":2916},{"style":1915},[2917],{"type":50,"value":2918}," \\\n",{"type":44,"tag":244,"props":2920,"children":2921},{"class":246,"line":268},[2922,2927,2932,2937,2941,2946,2951,2955,2960,2964,2969,2974,2978],{"type":44,"tag":244,"props":2923,"children":2924},{"style":256},[2925],{"type":50,"value":2926}," --import-path",{"type":44,"tag":244,"props":2928,"children":2929},{"style":256},[2930],{"type":50,"value":2931}," serve:app",{"type":44,"tag":244,"props":2933,"children":2934},{"style":256},[2935],{"type":50,"value":2936}," --checkpoint",{"type":44,"tag":244,"props":2938,"children":2939},{"style":742},[2940],{"type":50,"value":1907},{"type":44,"tag":244,"props":2942,"children":2943},{"style":256},[2944],{"type":50,"value":2945},"pat",{"type":44,"tag":244,"props":2947,"children":2948},{"style":1915},[2949],{"type":50,"value":2950},"h",{"type":44,"tag":244,"props":2952,"children":2953},{"style":742},[2954],{"type":50,"value":1923},{"type":44,"tag":244,"props":2956,"children":2957},{"style":256},[2958],{"type":50,"value":2959}," --checkpoint-pvc",{"type":44,"tag":244,"props":2961,"children":2962},{"style":742},[2963],{"type":50,"value":1907},{"type":44,"tag":244,"props":2965,"children":2966},{"style":256},[2967],{"type":50,"value":2968},"pv",{"type":44,"tag":244,"props":2970,"children":2971},{"style":1915},[2972],{"type":50,"value":2973},"c",{"type":44,"tag":244,"props":2975,"children":2976},{"style":742},[2977],{"type":50,"value":1923},{"type":44,"tag":244,"props":2979,"children":2980},{"style":1915},[2981],{"type":50,"value":2918},{"type":44,"tag":244,"props":2983,"children":2984},{"class":246,"line":301},[2985,2990,2994,2999,3004,3008,3013,3017,3022,3027,3031],{"type":44,"tag":244,"props":2986,"children":2987},{"style":256},[2988],{"type":50,"value":2989}," --namespace",{"type":44,"tag":244,"props":2991,"children":2992},{"style":742},[2993],{"type":50,"value":1907},{"type":44,"tag":244,"props":2995,"children":2996},{"style":256},[2997],{"type":50,"value":2998},"n",{"type":44,"tag":244,"props":3000,"children":3001},{"style":1915},[3002],{"type":50,"value":3003},"s",{"type":44,"tag":244,"props":3005,"children":3006},{"style":742},[3007],{"type":50,"value":1923},{"type":44,"tag":244,"props":3009,"children":3010},{"style":256},[3011],{"type":50,"value":3012}," --context",{"type":44,"tag":244,"props":3014,"children":3015},{"style":742},[3016],{"type":50,"value":1907},{"type":44,"tag":244,"props":3018,"children":3019},{"style":256},[3020],{"type":50,"value":3021},"ct",{"type":44,"tag":244,"props":3023,"children":3024},{"style":1915},[3025],{"type":50,"value":3026},"x",{"type":44,"tag":244,"props":3028,"children":3029},{"style":742},[3030],{"type":50,"value":1923},{"type":44,"tag":244,"props":3032,"children":3033},{"style":256},[3034],{"type":50,"value":3035}," --dry-run=client\n",{"type":44,"tag":244,"props":3037,"children":3038},{"class":246,"line":802},[3039,3043,3047,3051,3055,3059,3063,3067,3071,3075,3079,3083,3087],{"type":44,"tag":244,"props":3040,"children":3041},{"style":251},[3042],{"type":50,"value":64},{"type":44,"tag":244,"props":3044,"children":3045},{"style":256},[3046],{"type":50,"value":2846},{"type":44,"tag":244,"props":3048,"children":3049},{"style":256},[3050],{"type":50,"value":1902},{"type":44,"tag":244,"props":3052,"children":3053},{"style":742},[3054],{"type":50,"value":1907},{"type":44,"tag":244,"props":3056,"children":3057},{"style":256},[3058],{"type":50,"value":2860},{"type":44,"tag":244,"props":3060,"children":3061},{"style":1915},[3062],{"type":50,"value":1918},{"type":44,"tag":244,"props":3064,"children":3065},{"style":742},[3066],{"type":50,"value":1923},{"type":44,"tag":244,"props":3068,"children":3069},{"style":256},[3070],{"type":50,"value":2873},{"type":44,"tag":244,"props":3072,"children":3073},{"style":256},[3074],{"type":50,"value":2065},{"type":44,"tag":244,"props":3076,"children":3077},{"style":742},[3078],{"type":50,"value":1907},{"type":44,"tag":244,"props":3080,"children":3081},{"style":256},[3082],{"type":50,"value":2998},{"type":44,"tag":244,"props":3084,"children":3085},{"style":1915},[3086],{"type":50,"value":3003},{"type":44,"tag":244,"props":3088,"children":3089},{"style":742},[3090],{"type":50,"value":3091},">\n",{"type":44,"tag":244,"props":3093,"children":3094},{"class":246,"line":27},[3095,3099,3103,3108,3113,3117,3121,3125,3130,3135,3140,3144,3148,3152,3156],{"type":44,"tag":244,"props":3096,"children":3097},{"style":251},[3098],{"type":50,"value":64},{"type":44,"tag":244,"props":3100,"children":3101},{"style":256},[3102],{"type":50,"value":2846},{"type":44,"tag":244,"props":3104,"children":3105},{"style":256},[3106],{"type":50,"value":3107}," scale",{"type":44,"tag":244,"props":3109,"children":3110},{"style":742},[3111],{"type":50,"value":3112}," \u003C",{"type":44,"tag":244,"props":3114,"children":3115},{"style":256},[3116],{"type":50,"value":2860},{"type":44,"tag":244,"props":3118,"children":3119},{"style":1915},[3120],{"type":50,"value":1918},{"type":44,"tag":244,"props":3122,"children":3123},{"style":742},[3124],{"type":50,"value":1923},{"type":44,"tag":244,"props":3126,"children":3127},{"style":256},[3128],{"type":50,"value":3129}," --kind=deployment",{"type":44,"tag":244,"props":3131,"children":3132},{"style":256},[3133],{"type":50,"value":3134}," --replicas",{"type":44,"tag":244,"props":3136,"children":3137},{"style":882},[3138],{"type":50,"value":3139}," 3",{"type":44,"tag":244,"props":3141,"children":3142},{"style":256},[3143],{"type":50,"value":2065},{"type":44,"tag":244,"props":3145,"children":3146},{"style":742},[3147],{"type":50,"value":1907},{"type":44,"tag":244,"props":3149,"children":3150},{"style":256},[3151],{"type":50,"value":2998},{"type":44,"tag":244,"props":3153,"children":3154},{"style":1915},[3155],{"type":50,"value":3003},{"type":44,"tag":244,"props":3157,"children":3158},{"style":742},[3159],{"type":50,"value":3091},{"type":44,"tag":244,"props":3161,"children":3162},{"class":246,"line":825},[3163,3167,3171,3176,3180,3184,3188,3192,3196,3200,3204,3208,3212],{"type":44,"tag":244,"props":3164,"children":3165},{"style":251},[3166],{"type":50,"value":64},{"type":44,"tag":244,"props":3168,"children":3169},{"style":256},[3170],{"type":50,"value":2846},{"type":44,"tag":244,"props":3172,"children":3173},{"style":256},[3174],{"type":50,"value":3175}," delete",{"type":44,"tag":244,"props":3177,"children":3178},{"style":742},[3179],{"type":50,"value":1907},{"type":44,"tag":244,"props":3181,"children":3182},{"style":256},[3183],{"type":50,"value":2860},{"type":44,"tag":244,"props":3185,"children":3186},{"style":1915},[3187],{"type":50,"value":1918},{"type":44,"tag":244,"props":3189,"children":3190},{"style":742},[3191],{"type":50,"value":1923},{"type":44,"tag":244,"props":3193,"children":3194},{"style":256},[3195],{"type":50,"value":2873},{"type":44,"tag":244,"props":3197,"children":3198},{"style":256},[3199],{"type":50,"value":2065},{"type":44,"tag":244,"props":3201,"children":3202},{"style":742},[3203],{"type":50,"value":1907},{"type":44,"tag":244,"props":3205,"children":3206},{"style":256},[3207],{"type":50,"value":2998},{"type":44,"tag":244,"props":3209,"children":3210},{"style":1915},[3211],{"type":50,"value":3003},{"type":44,"tag":244,"props":3213,"children":3214},{"style":742},[3215],{"type":50,"value":3091},{"type":44,"tag":53,"props":3217,"children":3218},{},[3219,3225,3227,3232,3234,3240,3242,3248,3250,3256,3258,3264,3266,3272,3273,3279,3281,3287,3288,3294,3296,3302],{"type":44,"tag":59,"props":3220,"children":3222},{"className":3221},[],[3223],{"type":50,"value":3224},"--checkpoint",{"type":50,"value":3226}," mounts the PVC at ",{"type":44,"tag":59,"props":3228,"children":3230},{"className":3229},[],[3231],{"type":50,"value":2794},{"type":50,"value":3233},", resolves relative paths under\n",{"type":44,"tag":59,"props":3235,"children":3237},{"className":3236},[],[3238],{"type":50,"value":3239},"\u002Fdata\u002Fcheckpoints",{"type":50,"value":3241},", and sets ",{"type":44,"tag":59,"props":3243,"children":3245},{"className":3244},[],[3246],{"type":50,"value":3247},"TAU_MODEL_PATH",{"type":50,"value":3249},"; the app still owns loading.\nDirect ",{"type":44,"tag":59,"props":3251,"children":3253},{"className":3252},[],[3254],{"type":50,"value":3255},"scale",{"type":50,"value":3257}," works only for ",{"type":44,"tag":59,"props":3259,"children":3261},{"className":3260},[],[3262],{"type":50,"value":3263},"--kind=deployment",{"type":50,"value":3265}," — a RayService's Serve config\nis a serialized field, so redeploy or set ",{"type":44,"tag":59,"props":3267,"children":3269},{"className":3268},[],[3270],{"type":50,"value":3271},"--min-replicas",{"type":50,"value":1642},{"type":44,"tag":59,"props":3274,"children":3276},{"className":3275},[],[3277],{"type":50,"value":3278},"--max-replicas",{"type":50,"value":3280}," at\ncreation. ",{"type":44,"tag":59,"props":3282,"children":3284},{"className":3283},[],[3285],{"type":50,"value":3286},"--from-finetune",{"type":50,"value":1642},{"type":44,"tag":59,"props":3289,"children":3291},{"className":3290},[],[3292],{"type":50,"value":3293},"--from-model",{"type":50,"value":3295}," read cluster metadata, so they\ncannot use client dry-run. Serving does not activate\n",{"type":44,"tag":59,"props":3297,"children":3299},{"className":3298},[],[3300],{"type":50,"value":3301},"tau\u002Fworkspace.connection.yaml",{"type":50,"value":3303}," — pass namespace and context explicitly.",{"type":44,"tag":118,"props":3305,"children":3307},{"id":3306},"datasets-and-models",[3308],{"type":50,"value":3309},"Datasets and models",{"type":44,"tag":233,"props":3311,"children":3313},{"className":235,"code":3312,"language":237,"meta":238,"style":238},"tau data dataset list|show|ref|verify # curated dataset registry\ntau data model list|show|best|alias # durable checkpoints and aliases\n",[3314],{"type":44,"tag":59,"props":3315,"children":3316},{"__ignoreMap":238},[3317,3371],{"type":44,"tag":244,"props":3318,"children":3319},{"class":246,"line":247},[3320,3324,3329,3334,3338,3343,3348,3352,3357,3361,3366],{"type":44,"tag":244,"props":3321,"children":3322},{"style":251},[3323],{"type":50,"value":64},{"type":44,"tag":244,"props":3325,"children":3326},{"style":256},[3327],{"type":50,"value":3328}," data",{"type":44,"tag":244,"props":3330,"children":3331},{"style":256},[3332],{"type":50,"value":3333}," dataset",{"type":44,"tag":244,"props":3335,"children":3336},{"style":256},[3337],{"type":50,"value":2060},{"type":44,"tag":244,"props":3339,"children":3340},{"style":742},[3341],{"type":50,"value":3342},"|",{"type":44,"tag":244,"props":3344,"children":3345},{"style":251},[3346],{"type":50,"value":3347},"show",{"type":44,"tag":244,"props":3349,"children":3350},{"style":742},[3351],{"type":50,"value":3342},{"type":44,"tag":244,"props":3353,"children":3354},{"style":251},[3355],{"type":50,"value":3356},"ref",{"type":44,"tag":244,"props":3358,"children":3359},{"style":742},[3360],{"type":50,"value":3342},{"type":44,"tag":244,"props":3362,"children":3363},{"style":251},[3364],{"type":50,"value":3365},"verify",{"type":44,"tag":244,"props":3367,"children":3368},{"style":262},[3369],{"type":50,"value":3370}," # curated dataset registry\n",{"type":44,"tag":244,"props":3372,"children":3373},{"class":246,"line":268},[3374,3378,3382,3387,3391,3395,3399,3403,3408,3412,3418],{"type":44,"tag":244,"props":3375,"children":3376},{"style":251},[3377],{"type":50,"value":64},{"type":44,"tag":244,"props":3379,"children":3380},{"style":256},[3381],{"type":50,"value":3328},{"type":44,"tag":244,"props":3383,"children":3384},{"style":256},[3385],{"type":50,"value":3386}," model",{"type":44,"tag":244,"props":3388,"children":3389},{"style":256},[3390],{"type":50,"value":2060},{"type":44,"tag":244,"props":3392,"children":3393},{"style":742},[3394],{"type":50,"value":3342},{"type":44,"tag":244,"props":3396,"children":3397},{"style":251},[3398],{"type":50,"value":3347},{"type":44,"tag":244,"props":3400,"children":3401},{"style":742},[3402],{"type":50,"value":3342},{"type":44,"tag":244,"props":3404,"children":3405},{"style":251},[3406],{"type":50,"value":3407},"best",{"type":44,"tag":244,"props":3409,"children":3410},{"style":742},[3411],{"type":50,"value":3342},{"type":44,"tag":244,"props":3413,"children":3415},{"style":3414},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[3416],{"type":50,"value":3417},"alias",{"type":44,"tag":244,"props":3419,"children":3420},{"style":262},[3421],{"type":50,"value":3422}," # durable checkpoints and aliases\n",{"type":44,"tag":53,"props":3424,"children":3425},{},[3426],{"type":50,"value":3427},"The registry is a catalog, not a data plane: records point at storage accessed\nby workload identity, and are immutable once registered — only aliases move.\nReference a dataset from a run through its resolved path on the mounted PVC.",{"type":44,"tag":118,"props":3429,"children":3431},{"id":3430},"evidence-experiments-and-artifacts",[3432],{"type":50,"value":3433},"Evidence: experiments and artifacts",{"type":44,"tag":53,"props":3435,"children":3436},{},[3437,3439,3447,3449,3454],{"type":50,"value":3438},"Run evidence lives in the ",{"type":44,"tag":68,"props":3440,"children":3441},{},[3442],{"type":44,"tag":59,"props":3443,"children":3445},{"className":3444},[],[3446],{"type":50,"value":418},{"type":50,"value":3448}," binary, not ",{"type":44,"tag":59,"props":3450,"children":3452},{"className":3451},[],[3453],{"type":50,"value":64},{"type":50,"value":745},{"type":44,"tag":233,"props":3456,"children":3458},{"className":235,"code":3457,"language":237,"meta":238,"style":238},"taugrid-portal experiment search # find indexed runs\ntaugrid-portal experiment stellar \u003Crun-name> # local dashboard (-o html|json|tui)\ntaugrid-portal experiment open \u003Crun-name> # serve and open in a browser\ntaugrid-portal experiment status \u003Cname> # durable lifecycle record\ntaugrid-portal portal serve # unified observability portal\n",[3459],{"type":44,"tag":59,"props":3460,"children":3461},{"__ignoreMap":238},[3462,3484,3521,3558,3594],{"type":44,"tag":244,"props":3463,"children":3464},{"class":246,"line":247},[3465,3469,3474,3479],{"type":44,"tag":244,"props":3466,"children":3467},{"style":251},[3468],{"type":50,"value":418},{"type":44,"tag":244,"props":3470,"children":3471},{"style":256},[3472],{"type":50,"value":3473}," experiment",{"type":44,"tag":244,"props":3475,"children":3476},{"style":256},[3477],{"type":50,"value":3478}," search",{"type":44,"tag":244,"props":3480,"children":3481},{"style":262},[3482],{"type":50,"value":3483}," # find indexed runs\n",{"type":44,"tag":244,"props":3485,"children":3486},{"class":246,"line":268},[3487,3491,3495,3500,3504,3508,3512,3516],{"type":44,"tag":244,"props":3488,"children":3489},{"style":251},[3490],{"type":50,"value":418},{"type":44,"tag":244,"props":3492,"children":3493},{"style":256},[3494],{"type":50,"value":3473},{"type":44,"tag":244,"props":3496,"children":3497},{"style":256},[3498],{"type":50,"value":3499}," stellar",{"type":44,"tag":244,"props":3501,"children":3502},{"style":742},[3503],{"type":50,"value":1907},{"type":44,"tag":244,"props":3505,"children":3506},{"style":256},[3507],{"type":50,"value":1912},{"type":44,"tag":244,"props":3509,"children":3510},{"style":1915},[3511],{"type":50,"value":1918},{"type":44,"tag":244,"props":3513,"children":3514},{"style":742},[3515],{"type":50,"value":1923},{"type":44,"tag":244,"props":3517,"children":3518},{"style":262},[3519],{"type":50,"value":3520}," # local dashboard (-o html|json|tui)\n",{"type":44,"tag":244,"props":3522,"children":3523},{"class":246,"line":301},[3524,3528,3532,3537,3541,3545,3549,3553],{"type":44,"tag":244,"props":3525,"children":3526},{"style":251},[3527],{"type":50,"value":418},{"type":44,"tag":244,"props":3529,"children":3530},{"style":256},[3531],{"type":50,"value":3473},{"type":44,"tag":244,"props":3533,"children":3534},{"style":256},[3535],{"type":50,"value":3536}," open",{"type":44,"tag":244,"props":3538,"children":3539},{"style":742},[3540],{"type":50,"value":1907},{"type":44,"tag":244,"props":3542,"children":3543},{"style":256},[3544],{"type":50,"value":1912},{"type":44,"tag":244,"props":3546,"children":3547},{"style":1915},[3548],{"type":50,"value":1918},{"type":44,"tag":244,"props":3550,"children":3551},{"style":742},[3552],{"type":50,"value":1923},{"type":44,"tag":244,"props":3554,"children":3555},{"style":262},[3556],{"type":50,"value":3557}," # serve and open in a browser\n",{"type":44,"tag":244,"props":3559,"children":3560},{"class":246,"line":802},[3561,3565,3569,3573,3577,3581,3585,3589],{"type":44,"tag":244,"props":3562,"children":3563},{"style":251},[3564],{"type":50,"value":418},{"type":44,"tag":244,"props":3566,"children":3567},{"style":256},[3568],{"type":50,"value":3473},{"type":44,"tag":244,"props":3570,"children":3571},{"style":256},[3572],{"type":50,"value":1902},{"type":44,"tag":244,"props":3574,"children":3575},{"style":742},[3576],{"type":50,"value":1907},{"type":44,"tag":244,"props":3578,"children":3579},{"style":256},[3580],{"type":50,"value":2860},{"type":44,"tag":244,"props":3582,"children":3583},{"style":1915},[3584],{"type":50,"value":1918},{"type":44,"tag":244,"props":3586,"children":3587},{"style":742},[3588],{"type":50,"value":1923},{"type":44,"tag":244,"props":3590,"children":3591},{"style":262},[3592],{"type":50,"value":3593}," # durable lifecycle record\n",{"type":44,"tag":244,"props":3595,"children":3596},{"class":246,"line":27},[3597,3601,3606,3610],{"type":44,"tag":244,"props":3598,"children":3599},{"style":251},[3600],{"type":50,"value":418},{"type":44,"tag":244,"props":3602,"children":3603},{"style":256},[3604],{"type":50,"value":3605}," portal",{"type":44,"tag":244,"props":3607,"children":3608},{"style":256},[3609],{"type":50,"value":2846},{"type":44,"tag":244,"props":3611,"children":3612},{"style":262},[3613],{"type":50,"value":3614}," # unified observability portal\n",{"type":44,"tag":53,"props":3616,"children":3617},{},[3618],{"type":50,"value":3619},"The local expstore is the authoritative packet; ADX\u002FKusto and the hosted\nStellar UI are optional scalar projections. So \"the dashboard is empty\" is a\nprojection problem, not necessarily lost data.",{"type":44,"tag":53,"props":3621,"children":3622},{},[3623],{"type":50,"value":3624},"Add discovery metadata so runs group correctly:",{"type":44,"tag":233,"props":3626,"children":3628},{"className":724,"code":3627,"language":726,"meta":238,"style":238},"name: \u003Crun-name>\nscript: train.py\ncompute:\n gpus: 0\nexperiment:\n project: \u003Cproject>\n name: \u003Cthe set of runs being compared>\n group: \u003Cnamed subset>\n",[3629],{"type":44,"tag":59,"props":3630,"children":3631},{"__ignoreMap":238},[3632,3648,3664,3675,3692,3703,3720,3737],{"type":44,"tag":244,"props":3633,"children":3634},{"class":246,"line":247},[3635,3639,3643],{"type":44,"tag":244,"props":3636,"children":3637},{"style":736},[3638],{"type":50,"value":739},{"type":44,"tag":244,"props":3640,"children":3641},{"style":742},[3642],{"type":50,"value":745},{"type":44,"tag":244,"props":3644,"children":3645},{"style":256},[3646],{"type":50,"value":3647}," \u003Crun-name>\n",{"type":44,"tag":244,"props":3649,"children":3650},{"class":246,"line":268},[3651,3655,3659],{"type":44,"tag":244,"props":3652,"children":3653},{"style":736},[3654],{"type":50,"value":999},{"type":44,"tag":244,"props":3656,"children":3657},{"style":742},[3658],{"type":50,"value":745},{"type":44,"tag":244,"props":3660,"children":3661},{"style":256},[3662],{"type":50,"value":3663}," train.py\n",{"type":44,"tag":244,"props":3665,"children":3666},{"class":246,"line":301},[3667,3671],{"type":44,"tag":244,"props":3668,"children":3669},{"style":736},[3670],{"type":50,"value":862},{"type":44,"tag":244,"props":3672,"children":3673},{"style":742},[3674],{"type":50,"value":822},{"type":44,"tag":244,"props":3676,"children":3677},{"class":246,"line":802},[3678,3683,3687],{"type":44,"tag":244,"props":3679,"children":3680},{"style":736},[3681],{"type":50,"value":3682}," gpus",{"type":44,"tag":244,"props":3684,"children":3685},{"style":742},[3686],{"type":50,"value":745},{"type":44,"tag":244,"props":3688,"children":3689},{"style":882},[3690],{"type":50,"value":3691}," 0\n",{"type":44,"tag":244,"props":3693,"children":3694},{"class":246,"line":27},[3695,3699],{"type":44,"tag":244,"props":3696,"children":3697},{"style":736},[3698],{"type":50,"value":1066},{"type":44,"tag":244,"props":3700,"children":3701},{"style":742},[3702],{"type":50,"value":822},{"type":44,"tag":244,"props":3704,"children":3705},{"class":246,"line":825},[3706,3711,3715],{"type":44,"tag":244,"props":3707,"children":3708},{"style":736},[3709],{"type":50,"value":3710}," project",{"type":44,"tag":244,"props":3712,"children":3713},{"style":742},[3714],{"type":50,"value":745},{"type":44,"tag":244,"props":3716,"children":3717},{"style":256},[3718],{"type":50,"value":3719}," \u003Cproject>\n",{"type":44,"tag":244,"props":3721,"children":3722},{"class":246,"line":848},[3723,3728,3732],{"type":44,"tag":244,"props":3724,"children":3725},{"style":736},[3726],{"type":50,"value":3727}," name",{"type":44,"tag":244,"props":3729,"children":3730},{"style":742},[3731],{"type":50,"value":745},{"type":44,"tag":244,"props":3733,"children":3734},{"style":256},[3735],{"type":50,"value":3736}," \u003Cthe set of runs being compared>\n",{"type":44,"tag":244,"props":3738,"children":3739},{"class":246,"line":856},[3740,3745,3749],{"type":44,"tag":244,"props":3741,"children":3742},{"style":736},[3743],{"type":50,"value":3744}," group",{"type":44,"tag":244,"props":3746,"children":3747},{"style":742},[3748],{"type":50,"value":745},{"type":44,"tag":244,"props":3750,"children":3751},{"style":256},[3752],{"type":50,"value":3753}," \u003Cnamed subset>\n",{"type":44,"tag":53,"props":3755,"children":3756},{},[3757,3762,3764,3770,3772,3778],{"type":44,"tag":59,"props":3758,"children":3760},{"className":3759},[],[3761],{"type":50,"value":1066},{"type":50,"value":3763}," has no ",{"type":44,"tag":59,"props":3765,"children":3767},{"className":3766},[],[3768],{"type":50,"value":3769},"question",{"type":50,"value":3771}," field — that is an expstore\u002FStellar concept.\nThe research question a set of runs answers is carried by ",{"type":44,"tag":59,"props":3773,"children":3775},{"className":3774},[],[3776],{"type":50,"value":3777},"experiment.name",{"type":50,"value":169},{"type":44,"tag":118,"props":3780,"children":3782},{"id":3781},"platform-operator-work",[3783],{"type":50,"value":3784},"Platform operator work",{"type":44,"tag":53,"props":3786,"children":3787},{},[3788,3790,3795],{"type":50,"value":3789},"Operator commands need cluster-admin-level access and are deliberately not\nresearcher-facing. ",{"type":44,"tag":59,"props":3791,"children":3793},{"className":3792},[],[3794],{"type":50,"value":2402},{"type":50,"value":3796}," creates privileged pods.",{"type":44,"tag":233,"props":3798,"children":3800},{"className":235,"code":3799,"language":237,"meta":238,"style":238},"tau cluster install --version \u003Cversion> --values \u003Cfile>\ntau cluster validate nodes --gpu-class \u003Cc> --min-healthy \u003Cn>\ntau cluster validate topology --preset \u003Cpreset> # or --cluster-queue (default taugrid-cq)\ntau cluster uninstall --yes # Helm-owned resources only\n\ntau workspace check \u003Cname> # exits non-zero unless Ready\ntau workspace status \u003Cname> -o json\n",[3801],{"type":44,"tag":59,"props":3802,"children":3803},{"__ignoreMap":238},[3804,3865,3919,3967,3993,4000,4038],{"type":44,"tag":244,"props":3805,"children":3806},{"class":246,"line":247},[3807,3811,3816,3821,3826,3830,3835,3839,3843,3848,3852,3857,3861],{"type":44,"tag":244,"props":3808,"children":3809},{"style":251},[3810],{"type":50,"value":64},{"type":44,"tag":244,"props":3812,"children":3813},{"style":256},[3814],{"type":50,"value":3815}," cluster",{"type":44,"tag":244,"props":3817,"children":3818},{"style":256},[3819],{"type":50,"value":3820}," install",{"type":44,"tag":244,"props":3822,"children":3823},{"style":256},[3824],{"type":50,"value":3825}," --version",{"type":44,"tag":244,"props":3827,"children":3828},{"style":742},[3829],{"type":50,"value":1907},{"type":44,"tag":244,"props":3831,"children":3832},{"style":256},[3833],{"type":50,"value":3834},"versio",{"type":44,"tag":244,"props":3836,"children":3837},{"style":1915},[3838],{"type":50,"value":2998},{"type":44,"tag":244,"props":3840,"children":3841},{"style":742},[3842],{"type":50,"value":1923},{"type":44,"tag":244,"props":3844,"children":3845},{"style":256},[3846],{"type":50,"value":3847}," --values",{"type":44,"tag":244,"props":3849,"children":3850},{"style":742},[3851],{"type":50,"value":1907},{"type":44,"tag":244,"props":3853,"children":3854},{"style":256},[3855],{"type":50,"value":3856},"fil",{"type":44,"tag":244,"props":3858,"children":3859},{"style":1915},[3860],{"type":50,"value":1918},{"type":44,"tag":244,"props":3862,"children":3863},{"style":742},[3864],{"type":50,"value":3091},{"type":44,"tag":244,"props":3866,"children":3867},{"class":246,"line":268},[3868,3872,3876,3880,3885,3890,3894,3898,3902,3907,3911,3915],{"type":44,"tag":244,"props":3869,"children":3870},{"style":251},[3871],{"type":50,"value":64},{"type":44,"tag":244,"props":3873,"children":3874},{"style":256},[3875],{"type":50,"value":3815},{"type":44,"tag":244,"props":3877,"children":3878},{"style":256},[3879],{"type":50,"value":1187},{"type":44,"tag":244,"props":3881,"children":3882},{"style":256},[3883],{"type":50,"value":3884}," nodes",{"type":44,"tag":244,"props":3886,"children":3887},{"style":256},[3888],{"type":50,"value":3889}," --gpu-class",{"type":44,"tag":244,"props":3891,"children":3892},{"style":742},[3893],{"type":50,"value":1907},{"type":44,"tag":244,"props":3895,"children":3896},{"style":1915},[3897],{"type":50,"value":2973},{"type":44,"tag":244,"props":3899,"children":3900},{"style":742},[3901],{"type":50,"value":1923},{"type":44,"tag":244,"props":3903,"children":3904},{"style":256},[3905],{"type":50,"value":3906}," --min-healthy",{"type":44,"tag":244,"props":3908,"children":3909},{"style":742},[3910],{"type":50,"value":1907},{"type":44,"tag":244,"props":3912,"children":3913},{"style":1915},[3914],{"type":50,"value":2998},{"type":44,"tag":244,"props":3916,"children":3917},{"style":742},[3918],{"type":50,"value":3091},{"type":44,"tag":244,"props":3920,"children":3921},{"class":246,"line":301},[3922,3926,3930,3934,3939,3944,3948,3953,3958,3962],{"type":44,"tag":244,"props":3923,"children":3924},{"style":251},[3925],{"type":50,"value":64},{"type":44,"tag":244,"props":3927,"children":3928},{"style":256},[3929],{"type":50,"value":3815},{"type":44,"tag":244,"props":3931,"children":3932},{"style":256},[3933],{"type":50,"value":1187},{"type":44,"tag":244,"props":3935,"children":3936},{"style":256},[3937],{"type":50,"value":3938}," topology",{"type":44,"tag":244,"props":3940,"children":3941},{"style":256},[3942],{"type":50,"value":3943}," --preset",{"type":44,"tag":244,"props":3945,"children":3946},{"style":742},[3947],{"type":50,"value":1907},{"type":44,"tag":244,"props":3949,"children":3950},{"style":256},[3951],{"type":50,"value":3952},"prese",{"type":44,"tag":244,"props":3954,"children":3955},{"style":1915},[3956],{"type":50,"value":3957},"t",{"type":44,"tag":244,"props":3959,"children":3960},{"style":742},[3961],{"type":50,"value":1923},{"type":44,"tag":244,"props":3963,"children":3964},{"style":262},[3965],{"type":50,"value":3966}," # or --cluster-queue (default taugrid-cq)\n",{"type":44,"tag":244,"props":3968,"children":3969},{"class":246,"line":802},[3970,3974,3978,3983,3988],{"type":44,"tag":244,"props":3971,"children":3972},{"style":251},[3973],{"type":50,"value":64},{"type":44,"tag":244,"props":3975,"children":3976},{"style":256},[3977],{"type":50,"value":3815},{"type":44,"tag":244,"props":3979,"children":3980},{"style":256},[3981],{"type":50,"value":3982}," uninstall",{"type":44,"tag":244,"props":3984,"children":3985},{"style":256},[3986],{"type":50,"value":3987}," --yes",{"type":44,"tag":244,"props":3989,"children":3990},{"style":262},[3991],{"type":50,"value":3992}," # Helm-owned resources only\n",{"type":44,"tag":244,"props":3994,"children":3995},{"class":246,"line":27},[3996],{"type":44,"tag":244,"props":3997,"children":3998},{"emptyLinePlaceholder":806},[3999],{"type":50,"value":809},{"type":44,"tag":244,"props":4001,"children":4002},{"class":246,"line":825},[4003,4007,4012,4017,4021,4025,4029,4033],{"type":44,"tag":244,"props":4004,"children":4005},{"style":251},[4006],{"type":50,"value":64},{"type":44,"tag":244,"props":4008,"children":4009},{"style":256},[4010],{"type":50,"value":4011}," workspace",{"type":44,"tag":244,"props":4013,"children":4014},{"style":256},[4015],{"type":50,"value":4016}," check",{"type":44,"tag":244,"props":4018,"children":4019},{"style":742},[4020],{"type":50,"value":1907},{"type":44,"tag":244,"props":4022,"children":4023},{"style":256},[4024],{"type":50,"value":2860},{"type":44,"tag":244,"props":4026,"children":4027},{"style":1915},[4028],{"type":50,"value":1918},{"type":44,"tag":244,"props":4030,"children":4031},{"style":742},[4032],{"type":50,"value":1923},{"type":44,"tag":244,"props":4034,"children":4035},{"style":262},[4036],{"type":50,"value":4037}," # exits non-zero unless Ready\n",{"type":44,"tag":244,"props":4039,"children":4040},{"class":246,"line":848},[4041,4045,4049,4053,4057,4061,4065,4069,4073],{"type":44,"tag":244,"props":4042,"children":4043},{"style":251},[4044],{"type":50,"value":64},{"type":44,"tag":244,"props":4046,"children":4047},{"style":256},[4048],{"type":50,"value":4011},{"type":44,"tag":244,"props":4050,"children":4051},{"style":256},[4052],{"type":50,"value":1902},{"type":44,"tag":244,"props":4054,"children":4055},{"style":742},[4056],{"type":50,"value":1907},{"type":44,"tag":244,"props":4058,"children":4059},{"style":256},[4060],{"type":50,"value":2860},{"type":44,"tag":244,"props":4062,"children":4063},{"style":1915},[4064],{"type":50,"value":1918},{"type":44,"tag":244,"props":4066,"children":4067},{"style":742},[4068],{"type":50,"value":1923},{"type":44,"tag":244,"props":4070,"children":4071},{"style":256},[4072],{"type":50,"value":288},{"type":44,"tag":244,"props":4074,"children":4075},{"style":256},[4076],{"type":50,"value":4077}," json\n",{"type":44,"tag":53,"props":4079,"children":4080},{},[4081,4087,4089,4095,4097,4103],{"type":44,"tag":59,"props":4082,"children":4084},{"className":4083},[],[4085],{"type":50,"value":4086},"cluster install",{"type":50,"value":4088}," and ",{"type":44,"tag":59,"props":4090,"children":4092},{"className":4091},[],[4093],{"type":50,"value":4094},"cluster uninstall",{"type":50,"value":4096}," are Helm-only. Platform owners create\n",{"type":44,"tag":59,"props":4098,"children":4100},{"className":4099},[],[4101],{"type":50,"value":4102},"TauWorkspace",{"type":50,"value":4104}," resources through reviewed Helm\u002FGitOps\u002FIaC. TauGrid 0.1 does not\ncreate or mutate StorageClasses, PVCs, PVs, CSI configuration, cloud storage,\ncredentials, or Secrets.",{"type":44,"tag":53,"props":4106,"children":4107},{},[4108,4110,4116,4118,4124,4125,4131,4133,4139,4141,4146,4147,4153,4155,4167,4169,4175,4177,4181],{"type":50,"value":4109},"Workspace ",{"type":44,"tag":59,"props":4111,"children":4113},{"className":4112},[],[4114],{"type":50,"value":4115},"Ready",{"type":50,"value":4117}," requires ",{"type":44,"tag":59,"props":4119,"children":4121},{"className":4120},[],[4122],{"type":50,"value":4123},"RBACReady",{"type":50,"value":4088},{"type":44,"tag":59,"props":4126,"children":4128},{"className":4127},[],[4129],{"type":50,"value":4130},"QueueReady",{"type":50,"value":4132}," true with no drift.\nTauGrid 0.1 has no ",{"type":44,"tag":59,"props":4134,"children":4136},{"className":4135},[],[4137],{"type":50,"value":4138},"StorageReady",{"type":50,"value":4140}," condition on ",{"type":44,"tag":59,"props":4142,"children":4144},{"className":4143},[],[4145],{"type":50,"value":4102},{"type":50,"value":428},{"type":44,"tag":59,"props":4148,"children":4150},{"className":4149},[],[4151],{"type":50,"value":4152},"TauCluster",{"type":50,"value":4154},",\nso ",{"type":44,"tag":68,"props":4156,"children":4157},{},[4158,4160,4165],{"type":50,"value":4159},"a ",{"type":44,"tag":59,"props":4161,"children":4163},{"className":4162},[],[4164],{"type":50,"value":4115},{"type":50,"value":4166}," workspace can still have a missing PVC",{"type":50,"value":4168},". Verify the\nplatform-managed claim before handing off. ",{"type":44,"tag":59,"props":4170,"children":4172},{"className":4171},[],[4173],{"type":50,"value":4174},"WorkloadIdentityReady",{"type":50,"value":4176}," is also\ndiagnostic only. Degraded-condition recovery and the install\u002Funinstall contract\nare in ",{"type":44,"tag":163,"props":4178,"children":4179},{"href":165},[4180],{"type":50,"value":165},{"type":50,"value":169},{"type":44,"tag":118,"props":4183,"children":4185},{"id":4184},"boundaries-what-to-tell-users-tau-will-not-do",[4186],{"type":50,"value":4187},"Boundaries — what to tell users Tau will not do",{"type":44,"tag":53,"props":4189,"children":4190},{},[4191,4193,4197],{"type":50,"value":4192},"Tau owns resolution, validation, rendering, submission, and lifecycle. It does\n",{"type":44,"tag":68,"props":4194,"children":4195},{},[4196],{"type":50,"value":109},{"type":50,"value":4198}," own:",{"type":44,"tag":130,"props":4200,"children":4201},{},[4202,4207,4212,4217],{"type":44,"tag":134,"props":4203,"children":4204},{},[4205],{"type":50,"value":4206},"Azure\u002FAKS provisioning, node pools, or cloud RBAC",{"type":44,"tag":134,"props":4208,"children":4209},{},[4210],{"type":50,"value":4211},"Kubernetes scheduling, or Kueue quota and admission decisions",{"type":44,"tag":134,"props":4213,"children":4214},{},[4215],{"type":50,"value":4216},"Ray Train, PyTorch, or model-framework behavior",{"type":44,"tag":134,"props":4218,"children":4219},{},[4220],{"type":50,"value":4221},"Project data preparation, model code, or serving application semantics",{"type":44,"tag":53,"props":4223,"children":4224},{},[4225],{"type":50,"value":4226},"When a request falls outside these — \"make Tau create my GPU node pool\", \"have\nTau raise my quota\" — name the system that actually owns it and who to ask.\nGuessing a plausible-sounding flag is worse than saying so.",{"type":44,"tag":118,"props":4228,"children":4230},{"id":4229},"contributing-to-this-repository",[4231],{"type":50,"value":4232},"Contributing to this repository",{"type":44,"tag":53,"props":4234,"children":4235},{},[4236,4238,4244,4246,4251,4253,4258],{"type":50,"value":4237},"Module layout, CI guards, the ",{"type":44,"tag":59,"props":4239,"children":4241},{"className":4240},[],[4242],{"type":50,"value":4243},"tau.azure.com\u002F*",{"type":50,"value":4245}," label contract, and how to\nverify a change: ",{"type":44,"tag":163,"props":4247,"children":4249},{"href":4248},"references\u002Fcontributing.md",[4250],{"type":50,"value":4248},{"type":50,"value":4252},".\nUsers of the ",{"type":44,"tag":59,"props":4254,"children":4256},{"className":4255},[],[4257],{"type":50,"value":64},{"type":50,"value":4259}," binary do not need it.",{"type":44,"tag":4261,"props":4262,"children":4263},"style",{},[4264],{"type":50,"value":4265},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":4267,"total":4444},[4268,4286,4305,4321,4334,4352,4363,4376,4386,4398,4411,4431],{"slug":4269,"name":4269,"fn":4270,"description":4271,"org":4272,"tags":4273,"stars":4283,"repoUrl":4284,"updatedAt":4285},"azure-arg-external-evaluation-policy-author","author and test Azure Resource Graph policies","Use when the user wants to author, design, or test an Azure Policy that queries Azure Resource Graph (ARG) at request-time — i.e. a policy whose deny\u002Faudit decision depends on data from elsewhere in the subscription (sibling\u002Fparent resource state, RG-wide invariants, multi-hop relationships, etc.). Formally called Azure Policy External Evaluation; sometimes referred to colloquially as \"Invoke\". Drives an iterative KQL co-design loop against the user's real subscription via `az graph query`, then emits a policy definition, assignment, `.http` test flow, and an `EXPLANATION.md` companion. Read-only; never provisions anything.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4274,4275,4278,4281],{"name":11,"slug":8,"type":16},{"name":4276,"slug":4277,"type":16},"Compliance","compliance",{"name":4279,"slug":4280,"type":16},"Governance","governance",{"name":4282,"slug":1039,"type":16},"Policy",1689,"https:\u002F\u002Fgithub.com\u002FAzure\u002Fazure-policy","2026-07-12T08:17:48.378432",{"slug":4287,"name":4287,"fn":4288,"description":4289,"org":4290,"tags":4291,"stars":4302,"repoUrl":4303,"updatedAt":4304},"azure-blueprints-migration","migrate Azure Blueprints to Template Specs","Use when a user needs to migrate off Azure Blueprints (definitions and\u002For assignments) to Template Specs and Deployment Stacks before the January 31, 2027 retirement. Covers inventory, export, conversion to Bicep, policy decoupling, Template Spec publishing, Deployment Stack deployment with deny-settings, validation, and cutover.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4292,4293,4296,4299],{"name":11,"slug":8,"type":16},{"name":4294,"slug":4295,"type":16},"Deployment","deployment",{"name":4297,"slug":4298,"type":16},"Infrastructure as Code","infrastructure-as-code",{"name":4300,"slug":4301,"type":16},"Migration","migration",261,"https:\u002F\u002Fgithub.com\u002FAzure\u002Fazure-blueprints","2026-08-20T03:28:18.136156",{"slug":4306,"name":4306,"fn":4307,"description":4308,"org":4309,"tags":4310,"stars":4318,"repoUrl":4319,"updatedAt":4320},"sdaf-bom-selection","select SAP Bill of Materials","Pick the right SDAF BOM for a target SAP product \u002F release \u002F DB platform \u002F version \u002F kernel \u002F topology. Explains where BOMs live in the samples repo (`SAP\u002F` for whole products, `BOM\u002F` for components), decodes the `ms`, `v####`, and `latest` suffix conventions, points at the compatibility guardrails (`supportedPlatforms`, `supportedDBVersions`, `supportedKernels`), and reminds the operator that the download flow needs four explicit BOM keys in `sap-parameters.yaml`. Use when a user says \"which BOM do I use\", \"product BOM vs component BOM\", \"SAP samples BOM catalog\", \"BOM_CATALOG\", \"S4\u002F2023 BOM\", \"HANA BOM\", or \"how do I pick a BOM for HANA\u002FOracle\u002FDB2\u002FASE\". Do NOT use to author a new BOM, to acquire media (`sdaf-media-acquisition`), or to troubleshoot a checksum \u002F 404 (`sdaf-media-diagnostics`).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4311,4312,4315],{"name":11,"slug":8,"type":16},{"name":4313,"slug":4314,"type":16},"Configuration","configuration",{"name":4316,"slug":4317,"type":16},"SAP","sap",143,"https:\u002F\u002Fgithub.com\u002FAzure\u002Fsap-automation","2026-09-02T07:47:45.597063",{"slug":4322,"name":4322,"fn":4323,"description":4324,"org":4325,"tags":4326,"stars":4318,"repoUrl":4319,"updatedAt":4333},"sdaf-control-plane-bootstrap","deploy SDAF control plane infrastructure","Deploy the SDAF control plane locally: prepare DEPLOYER and LIBRARY tfvars, review the plan, run `deploy_controlplane.sh` per `docs\u002Flocal\u002F03-00-control-plane.md § Run`, and validate the deployer + library + state hand-off. Grounded in `docs\u002Flocal\u002F03-00-control-plane.md`. Use when a user says \"deploy the SDAF control plane\", \"run deploy_controlplane.sh\", \"bootstrap SDAF from an empty subscription\", \"install the deployer\" or \"create the SAP library\". Do NOT use for workload zone (see sdaf-workload-zone), SAP system (see sdaf-sap-system), removal (`sdaf-safe-removal`), or Azure Government \u002F sovereign-cloud deltas (`sdaf-sovereign-cloud`).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4327,4328,4329,4332],{"name":11,"slug":8,"type":16},{"name":4294,"slug":4295,"type":16},{"name":4330,"slug":4331,"type":16},"Infrastructure","infrastructure",{"name":4316,"slug":4317,"type":16},"2026-09-02T07:46:51.573641",{"slug":4335,"name":4335,"fn":4336,"description":4337,"org":4338,"tags":4339,"stars":4318,"repoUrl":4319,"updatedAt":4351},"sdaf-failure-triage","triage failed SAP automation runs","Triage a failed or suspicious SDAF run: distinguish a real failure from a green no-op or a clean plan reported as failure, map the observed symptom to a documented cause in `docs\u002Flocal\u002Ftroubleshooting.md`, and hand off to the stage-owning skill for retry. Use when a user says \"SDAF run failed\", \"my deploy exited non-zero\", \"the plan was clean but exit 1\", \"the run said success but nothing was deployed\", \"state lock error\", \"unexpected replacement\", \"control plane stopped partway\", \"generated hosts.yaml missing\", \"workload-zone private endpoint failure\", \"workload-zone subnet policy failure\", or \"SDAF exit 2\". Do NOT use to actually deploy or to redesign the workspace layout.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4340,4341,4344,4347,4348],{"name":11,"slug":8,"type":16},{"name":4342,"slug":4343,"type":16},"Diagnostics","diagnostics",{"name":4345,"slug":4346,"type":16},"Operations","operations",{"name":4316,"slug":4317,"type":16},{"name":4349,"slug":4350,"type":16},"Triage","triage","2026-09-02T07:47:40.607437",{"slug":4353,"name":4353,"fn":4354,"description":4355,"org":4356,"tags":4357,"stars":4318,"repoUrl":4319,"updatedAt":4362},"sdaf-ha-diagnostics","diagnose SDAF high-availability cluster failures","Diagnose a live or recently failed SDAF high-availability cluster without changing cluster state. Start read-first: capture `crm status full` or `pcs status --full`, inspect SBD or fencing evidence, confirm current resource placement, and reuse existing quality-assurance or HCMT artifacts. Grounded in `docs\u002Flocal\u002F07-10-quality-assurance.md`, the shipped Pacemaker role vars\u002Ftasks, and the SAP Automation QA setup role. Use when a user says \"diagnose my HANA failover\", \"crm status\", \"pcs status\", \"check fencing\", \"why is the SCS\u002FERS cluster unhealthy\", \"offline_validation\u002Fcib\", or \"review an HCMT result zip\". Do NOT use for pre-deploy topology or design choices (see `sdaf-ha-topology`), fresh installation\u002Fdeploy, or disruptive failover\u002Ffencing exercises (see `sdaf-quality-assurance`).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4358,4359,4360,4361],{"name":11,"slug":8,"type":16},{"name":4342,"slug":4343,"type":16},{"name":4345,"slug":4346,"type":16},{"name":4316,"slug":4317,"type":16},"2026-09-02T07:48:04.154501",{"slug":4364,"name":4364,"fn":4365,"description":4366,"org":4367,"tags":4368,"stars":4318,"repoUrl":4319,"updatedAt":4375},"sdaf-ha-topology","design high-availability SAP topologies","Choose the documented SDAF high-availability design before SAP-system deployment. Use when a user asks which HA topology to deploy, whether to use AFA or SBD, AFS or ANF, HANA scale-up or scale-out, whether ANGI is allowed, or which HA inputs must be set before deployment. Ground answers in the current support matrix, Ansible roles, sample tfvars, and validation logic. Do NOT use for live cluster diagnosis (sdaf-ha-diagnostics) or for running the installers (sdaf-sap-installation).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4369,4372,4373,4374],{"name":4370,"slug":4371,"type":16},"Architecture","architecture",{"name":11,"slug":8,"type":16},{"name":4330,"slug":4331,"type":16},{"name":4316,"slug":4317,"type":16},"2026-09-02T07:47:43.405138",{"slug":4377,"name":4377,"fn":4378,"description":4379,"org":4380,"tags":4381,"stars":4318,"repoUrl":4319,"updatedAt":4385},"sdaf-media-acquisition","acquire SAP installation media","Acquire approved SAP media into the SDAF library after SAP-system infrastructure exists. Drives local `deploy\u002Fansible\u002Fdownload_menu.sh` per `docs\u002Flocal\u002F06-00-software-and-installation.md § Download software`, enforces the documented `BOM_CATALOG` root and the four required `sap-parameters.yaml` BOM keys (`application_bom_name`, `database_bom_name`, `sap_kernel_bom_name`, `save_bom_as`), and explains the documented download and validation flow. Use for \"download SAP media\", \"run download_menu.sh\", \"acquire software into the library\", \"BOM Downloader\", \"BOM_CATALOG\", or \"assemble application\u002Fdatabase\u002Fkernel BOMs for download\". Do NOT use to choose a BOM, troubleshoot a failed download, or run installation playbooks.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4382,4383,4384],{"name":11,"slug":8,"type":16},{"name":4345,"slug":4346,"type":16},{"name":4316,"slug":4317,"type":16},"2026-09-02T07:46:44.504107",{"slug":4387,"name":4387,"fn":4388,"description":4389,"org":4390,"tags":4391,"stars":4318,"repoUrl":4319,"updatedAt":4397},"sdaf-media-diagnostics","diagnose SDAF media and BOM processing","Diagnose SDAF software-download and BOM-processing failures after a BOM is already selected. Covers the documented local media path, SAP download, storage-account upload\u002Fdownload, checksum validation, and `SAPCAR` \u002F `.EXE` extraction. Use when a user says \"download_menu failed\", \"BOM downloader 404\", \"missing archive\", \"checksum mismatch\", \"SAPCAR failed\", \"bom-processing-done missing\", or \"my SPS07 media run can't find a file\". Do NOT use to choose a BOM (see `sdaf-bom-selection`), to run a clean first-time media download with no failure, or to diagnose a later non-media install failure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4392,4393,4394,4395],{"name":11,"slug":8,"type":16},{"name":4342,"slug":4343,"type":16},{"name":4316,"slug":4317,"type":16},{"name":4396,"slug":925,"type":16},"Storage","2026-09-02T07:48:06.369058",{"slug":4399,"name":4399,"fn":4400,"description":4401,"org":4402,"tags":4403,"stars":4318,"repoUrl":4319,"updatedAt":4410},"sdaf-orientation-and-surface","orient users to SDAF architecture","Orient a newcomer to the SAP Deployment Automation Framework (SDAF): explain the spine (control plane → workload zone → SAP system → software → install → operate\u002Fremove), summarise the three execution surfaces (Local, Azure DevOps, GitHub Actions), and print the install command for the matching surface bootstrap plugin — as documented in this repository's `docs\u002FPLUGINS.md` — so the operator can run it manually. Use when a user says \"what is SDAF\", \"how is SDAF structured\", \"where do I start with SDAF\", \"which surface should I use\", \"SDAF Local vs ADO vs GitHub\", \"workstation vs Azure Pipelines vs GitHub workflows\", \"how do I install the SDAF ADO\u002FGitHub plugin\", or \"I'm new to SDAF\". Do NOT use for readiness pre-flight (see sdaf-readiness-check), workspace\u002Ftfvars layout (see sdaf-workspace-and-tfvars), or troubleshooting a failed run (see sdaf-failure-triage).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4404,4405,4406,4409],{"name":4370,"slug":4371,"type":16},{"name":11,"slug":8,"type":16},{"name":4407,"slug":4408,"type":16},"Documentation","documentation",{"name":4316,"slug":4317,"type":16},"2026-09-02T07:46:52.116557",{"slug":4412,"name":4412,"fn":4413,"description":4414,"org":4415,"tags":4416,"stars":4318,"repoUrl":4319,"updatedAt":4430},"sdaf-plan-and-test-semantics","manage SDAF plan and test semantics","Explain SDAF plan-only \u002F test \u002F apply semantics without pretending they are universal. Compares the documented Local commands, Azure DevOps wrapper pipelines, and GitHub Actions workflows for control plane, workload zone, and SAP system: local stage commands show Terraform plans and then apply after approval; ADO\u002FGitHub workload-zone and SAP-system test runs stop after the plan; the current hosted control-plane test options do not provide a plan-only run. Use when a user says \"what does test do in SDAF\", \"is this a real dry run\", \"plan-only vs apply\", \"workflow 01 dry-run\", \"pipeline 02 test\", or \"TEST_ONLY\". Do NOT use to actually deploy a stage or to triage a failed run (see sdaf-control-plane-bootstrap \u002F sdaf-workload-zone \u002F sdaf-sap-system \u002F sdaf-failure-triage).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4417,4420,4421,4424,4427],{"name":4418,"slug":4419,"type":16},"Automation","automation",{"name":11,"slug":8,"type":16},{"name":4422,"slug":4423,"type":16},"Azure DevOps","azure-devops",{"name":4425,"slug":4426,"type":16},"CI\u002FCD","ci-cd",{"name":4428,"slug":4429,"type":16},"GitHub Actions","github-actions","2026-09-02T07:48:08.951811",{"slug":4432,"name":4432,"fn":4433,"description":4434,"org":4435,"tags":4436,"stars":4318,"repoUrl":4319,"updatedAt":4443},"sdaf-quality-assurance","validate deployed SDAF SAP systems","Validate a deployed SDAF SAP system through the SDAF-owned QA entry points: the local quality-assurance menu and the documented Azure DevOps pipeline 13 path. Choose configuration checks vs functional tests, apply `TEST_GROUPS` \u002F `TEST_CASES` safely, interpret `quality_assurance\u002F` and `logs\u002F`, and enforce the documented boundary that the GitHub Actions wrapper is still pending. Use when a user says \"run quality assurance\", \"run configuration checks\", \"pipeline 13 QA\", \"offline HA validation\", or \"why does the report say No test results found\". Do NOT use to deploy the SAP system or generate `sap-parameters.yaml` \u002F `\u003CSID>_hosts.yaml` (see `sdaf-sap-system`), or for a generic failed-run report with no QA-stage context (see `sdaf-failure-triage`).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4437,4438,4439,4442],{"name":4418,"slug":4419,"type":16},{"name":11,"slug":8,"type":16},{"name":4440,"slug":4441,"type":16},"Quality Assurance","quality-assurance",{"name":4316,"slug":4317,"type":16},"2026-09-02T07:46:49.932123",145,{"items":4446,"total":247},[4447],{"slug":4,"name":4,"fn":5,"description":6,"org":4448,"tags":4449,"stars":27,"repoUrl":28,"updatedAt":29},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4450,4451,4452,4453,4454],{"name":22,"slug":23,"type":16},{"name":11,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":25,"slug":26,"type":16},{"name":14,"slug":15,"type":16}]