[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-launch-nemo-rl":3,"mdc--sd5xl-key":37,"related-repo-nvidia-launch-nemo-rl":4085,"related-org-nvidia-launch-nemo-rl":4191},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":32,"sourceUrl":35,"mdContent":36},"launch-nemo-rl","launch and monitor NeMo-RL recipes","Playbook for launching, monitoring, stopping, and debugging NeMo-RL recipes on a Kubernetes cluster via the nrl-k8s CLI. Covers ephemeral vs long-lived RayCluster modes, iterating on runs, and debugging hung or failed training jobs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20,23],{"name":13,"slug":14,"type":15},"Machine Learning","machine-learning","tag",{"name":17,"slug":18,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Kubernetes","kubernetes",{"name":24,"slug":25,"type":15},"Debugging","debugging",2473,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills","2026-07-14T05:27:22.975529","Apache-2.0",281,[],{"repoUrl":27,"stars":26,"forks":30,"topics":33,"description":34},[],"AI agent skills published by NVIDIA","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fskills\u002Ftree\u002FHEAD\u002Fskills\u002Flaunch-nemo-rl","---\nname: launch-nemo-rl\nlicense: Apache-2.0\ndescription: Playbook for launching, monitoring, stopping, and debugging NeMo-RL recipes on a Kubernetes cluster via the nrl-k8s CLI. Covers ephemeral vs long-lived RayCluster modes, iterating on runs, and debugging hung or failed training jobs.\nwhen_to_use:\n  - \"run this recipe on k8s\"\n  - \"launch on the cluster\"\n  - \"submit a training job\"\n  - \"tear down the cluster\"\n  - \"resubmit as rayjob\"\n  - \"why is the run stuck\"\n  - \"how do I get logs for job X\"\n  - \"bring the cluster back up\"\nallowed-tools: Bash Read Grep Glob Edit Write\n---\n\n# launch-nemo-rl — running NeMo-RL recipes on Kubernetes via nrl-k8s\n\nThis is the playbook for the `nrl-k8s` CLI at `infra\u002Fnrl_k8s\u002F`. Follow it when the user asks to launch \u002F iterate \u002F debug a NeMo-RL recipe on a Kubernetes cluster. Verify current state (`kubectl`, `git log`, the recipe + infra files) before acting — the cluster is shared and the cost of a wrong action is high.\n\n## 1. One command, two modes\n\nThere is a single top-level submission command: **`nrl-k8s run`**. It has two lifecycle modes.\n\n| Mode               | Invocation        | When to use                                                                                                                                                                   | Cluster after? |\n| :----------------- | :---------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :------------- |\n| Ephemeral (default)  | `nrl-k8s run`              | One-shot. KubeRay applies a RayJob, runs, tears the cluster down. Best for most runs.                                                                                          | No (auto)      |\n| Long-lived           | `nrl-k8s run --raycluster` | Dev loop. Reuses a matching live cluster, applies if absent, warns + reuses on drift (pass `--recreate` to replace). Then submits daemons and training. First-choice for iteration. | Yes            |\n\nAsk: *Do I need this cluster after the run?* If yes, use `--raycluster`. Otherwise use the default (ephemeral).\n\nThe rest of the CLI is observability \u002F stage-by-stage control:\n\n| Command                 | Purpose                                                                                         |\n| :---------------------- | :---------------------------------------------------------------------------------------------- |\n| `nrl-k8s check`         | Validate a recipe + infra pair; optionally write the fully-resolved manifests (`-o`).           |\n| `nrl-k8s status`        | Per-role RayCluster state, head pod phase, worker pod phases, daemon job status.                |\n| `nrl-k8s cluster up\u002Fdown\u002Flist\u002Fdashboard` | Manage RayClusters independently of a run (e.g. render a manifest with `--dry-run`). |\n| `nrl-k8s job list\u002Flogs\u002Fstop` | Observability over Ray Jobs already submitted to a role's cluster.                         |\n| `nrl-k8s logs`          | Tail a role's pod \u002F daemon logs without needing a submission id.                                |\n\n## 2. Recipe + infra pair\n\nEvery launch takes two files. Pass the infra with `--infra`, not merged inline:\n\n```\nnrl-k8s run infra\u002Fnrl_k8s\u002Fexamples\u002F\u003Crecipe>.yaml \\\n  --infra infra\u002Fnrl_k8s\u002Fexamples\u002F\u003Crecipe>.\u003Cprofile>.infra.yaml\n```\n\n- **Recipe** (e.g. `qwen3_30b_math_8n_4gpu.yaml`) — NeMo-RL config: model, GRPO\u002FSFT knobs, `cluster.{gpus_per_node,num_nodes}`. Uses `defaults:` to inherit from `examples\u002Fconfigs\u002Frecipes\u002Fllm\u002F...`.\n- **Infra** (e.g. `*.\u003Cprofile>.infra.yaml`) — K8s\u002FRay shape: namespace, image, service account, RayCluster spec under `kuberay:`, optional Deployments under `deployments:`, `submit.submitter`, `launch.{mode,codeSource,codePath,entrypoint}`. Pair names follow `\u003Crecipe>.\u003Cprofile>[.prod].infra.yaml` where `\u003Cprofile>` names the hardware target (e.g. `gb300`).\n\nExample pairs in `infra\u002Fnrl_k8s\u002Fexamples\u002F` — read the neighbouring files to see the current conventions for the target profile.\n\n## 3. Long-lived mode flags\n\nThree independent dimensions. `--mode` is a macro that picks defaults; individual flags override it.\n\n```\n--mode interactive   → --submitter portForward  --code-source upload  (tails logs)\n--mode batch         → --submitter exec         --code-source image   (returns after nohup)\n```\n\n- **Submitter**: `portForward` uses `kubectl port-forward` + Ray Job SDK (gets a `submission_id` the dashboard tracks). `exec` uses `kubectl exec` + `nohup` on the head pod (no submission_id; driver appears as `type=DRIVER` in the dashboard).\n- **Code source**: `upload` stages a working_dir from the laptop (Ray 100 MiB cap). `image` \u002F `lustre` expect code on the pod's filesystem — paired with `--code-path` (typically `\u002Fopt\u002Fnemo-rl`), which is a subPath of the shared-filesystem PVC mount in the standard infra examples.\n- **Wait**: `--wait` tails logs until terminal; `--no-wait` returns as soon as the driver is running.\n\nOther long-lived-only flags:\n\n- `--replace` — stop any running training \u002F daemon job before submitting new ones (suffixes daemon submissionIds with a timestamp so Ray accepts the resubmit).\n- `--recreate` — delete + re-apply a RayCluster whose live spec has drifted from the rendered manifest (default is warn + reuse).\n- `--skip-daemons` — bring up all declared clusters but only submit training. Use on disagg recipes where gym\u002Fgeneration are already healthy.\n\nGotcha: on infra where the entrypoint does `cd \u002Fopt\u002Fnemo-rl` (or another in-image \u002F Lustre path) and loads the recipe from there, **`--code-source upload` does NOT override the recipe on the pod** — the uploaded working_dir sits in `\u002Ftmp\u002Fray\u002F...` but the entrypoint `cd`s away from it. To actually test a local recipe change, either sync your edits to the shared filesystem mounted into the pods or flip the Hydra overrides in the entrypoint.\n\n## 4. Ephemeral mode flags (`--rayjob`)\n\nWhen `--rayjob` is set, `run` branches into the RayJob code path. Relevant flags:\n\n- `--rayjob-name NAME` — RayJob metadata name (defaults to the training cluster name).\n- `--shutdown \u002F --no-shutdown` — default `true`: KubeRay deletes the RayCluster once the Ray Job reaches a terminal state.\n- `--ttl SECONDS` — default 3600s: keep the RayJob object around after the run finishes for post-mortem log access.\n- `--wait \u002F --no-wait` — default `wait`: poll `jobDeploymentStatus` until Complete\u002FFailed. `--no-wait` returns as soon as the RayJob is applied.\n- `--timeout SECONDS` — default 86400s (24h): bound the `--wait` poll.\n- `--dry-run` — render the RayJob manifest and print it; do not apply.\n\n`--replace` \u002F `--recreate` \u002F `--skip-daemons` are silently ignored in `--rayjob` mode (KubeRay owns lifecycle).\n\n## 5. Iterating on a config without touching the shared filesystem\n\nWhen the recipe on the pod filesystem has the wrong value for your experiment, use Hydra overrides on the entrypoint instead of forking the recipe. Pattern:\n\n```yaml\nentrypoint: |\n  set -eu\n  cd \u002Fopt\u002Fnemo-rl\n  RUN_ID=\"\\${RAY_JOB_SUBMISSION_ID:-\\${NRL_K8S_RUN_ID:-$(date -u +%Y%m%d-%H%M%S)}}\"\n  python -u examples\u002Frun_grpo.py \\\n    --config infra\u002Fnrl_k8s\u002Fexamples\u002F\u003Crecipe>.yaml \\\n    logger.wandb_enabled=true \\\n    logger.wandb.project=\u003Cproject> \\\n    \"logger.wandb.name=\u003Crun-name>-\\${RUN_ID}\"\n```\n\n**Escape `${…}`** with a backslash. OmegaConf otherwise interprets it as interpolation and errors on shell-style `${VAR:-default}`. `RUN_ID` resolves to `RAY_JOB_SUBMISSION_ID` (injected by KubeRay in rayjob mode) → `NRL_K8S_RUN_ID` (injected by the CLI in long-lived mode) → local timestamp — so the name is unique across either path.\n\n## 6. Per-profile concerns (hardware + scheduler + DRA)\n\nEvery infra YAML encodes a hardware\u002Fscheduler profile. The concrete examples in `infra\u002Fnrl_k8s\u002Fexamples\u002F` are authoritative for the profiles they target — read the neighbouring infra file before writing a new one. Things that commonly vary:\n\n- **Per-node GPUs** (e.g. 4 vs 8) — must match `cluster.gpus_per_node` in the recipe, otherwise workers stay `Pending`.\n- **Node selectors** — head pods usually land on a CPU-only node pool; GPU workers match on `nvidia.com\u002Fgpu.product` or a node-group label.\n- **Scheduler** — KAI (`schedulerName: kai-scheduler` + `kai.scheduler\u002Fqueue` label) with topology annotations (`kai.scheduler\u002Ftopology`, `kai.scheduler\u002Ftopology-required-placement`) gang-schedules workers into one clique. Without it, pods may land on different racks and NVLink\u002FRoCE won't span them.\n- **DRA claims** — ComputeDomain + RoCE are attached via `resourceClaims` referencing `ResourceClaimTemplate`s. The CLI auto-creates\u002Fdeletes these when the worker pod spec contains DRA claim references — no manual setup needed.\n- **Secrets** — always via `secretKeyRef` (`wandb-api-key`, image pull secret). Never embed.\n- **Shared filesystem mounts** — typically a Lustre PVC mounted twice: once at the code path (e.g. `\u002Fopt\u002Fnemo-rl` with a user-scoped `subPath`) and once at a workspace root (e.g. `\u002Fmnt\u002Frl-workspace`) for datasets, HF cache, and checkpoints.\n\nBefore applying an infra, verify prereqs exist in the target namespace:\n\n```bash\nkubectl get pvc \u003Cworkspace-pvc>\nkubectl get secret \u003Cwandb-secret> \u003Cimage-pull-secret>\nkubectl get sa \u003Cservice-account>\n```\n\n## 7. End-to-end workflows\n\n### 7a. Fresh one-shot run (rayjob)\n```bash\n# From the NeMo-RL repo root:\nnrl-k8s check \u003Crecipe> --infra \u003Cinfra>                               # validate first\nnrl-k8s run \u003Crecipe> --infra \u003Cinfra> --rayjob --dry-run              # render RayJob manifest\nnrl-k8s run \u003Crecipe> --infra \u003Cinfra> --rayjob --no-wait              # apply, returns fast\n```\n\nWatch status + teardown (works even after your laptop disconnects because KubeRay owns the lifecycle):\n```bash\nkubectl get rayjob -n default \u003Cname> -w\nkubectl get raycluster -n default                                    # empty = teardown succeeded\n```\n\n### 7b. Dev loop (long-lived)\n```bash\nnrl-k8s run \u003Crecipe> --infra \u003Cinfra> --run-id $(date +%Y%m%d-%H%M%S)\n# Edits in the recipe? Just re-run — reuses the live cluster.\n# Pod spec changed? Add --recreate to delete + re-apply.\n# Disagg recipe with gym\u002Fgen already healthy? --skip-daemons.\n```\n\n### 7c. First-time disaggregated bring-up\n```bash\nnrl-k8s run \u003Crecipe> --infra \u003Cdisagg-infra> --mode batch --code-source image\n```\n\n### 7d. Cluster-only lifecycle\n```bash\nnrl-k8s cluster up   \u003Crecipe> --infra \u003Cinfra> --target kuberay.training --wait\nnrl-k8s cluster up   \u003Crecipe> --infra \u003Cinfra> --target kuberay.training --dry-run   # render manifest\nnrl-k8s cluster down \u003Crecipe> --infra \u003Cinfra> --target kuberay.training --wait\nnrl-k8s cluster down \u003Crecipe> --infra \u003Cinfra>                                       # tear down all\nnrl-k8s cluster list -n default\nnrl-k8s cluster dashboard \u003Ccluster-name>                                  # port-forward + browser\n```\n\n### 7e. Deployments (e.g. nemo-skills sandbox)\n```bash\n# Bring up just the deployment\nnrl-k8s cluster up \u003Crecipe> --infra \u003Cinfra> --target deployments.nemo_skills\n# Tear down just the deployment\nnrl-k8s cluster down \u003Crecipe> --infra \u003Cinfra> --target deployments.nemo_skills\n# Tear down everything (RayClusters + Deployments)\nnrl-k8s cluster down \u003Crecipe> --infra \u003Cinfra>\n```\n\nThe `deployments:` section in infra YAML declares Kubernetes Deployments managed alongside RayClusters. The CLI patches image, imagePullSecrets, and serviceAccountName from the top-level infra keys (same as RayClusters). Deployments start in parallel with cluster bring-up — no ordering dependency.\n\n## 8. Monitoring a run\n\n```bash\n# Status\nnrl-k8s status \u003Crecipe> --infra \u003Cinfra>\nkubectl get rayjob,raycluster -n default\n\n# Follow the driver\nnrl-k8s job list \u003Crecipe> --infra \u003Cinfra> --role training\nnrl-k8s job logs \u003Crun-id> \u003Crecipe> --infra \u003Cinfra> --role training -f\n```\n\nWhen the `nrl-k8s job logs -f` subprocess dies (`kubectl port-forward` i\u002Fo timeout after ~15 min idle), just re-run it. The training job keeps going.\n\nTo fetch driver logs for a terminal job (SUCCEEDED\u002FFAILED) or a RayJob via the dashboard API:\n```bash\nRC=$(kubectl get rayjob -n default \u003Crayjob-name> -o jsonpath='{.status.rayClusterName}')\nkubectl port-forward -n default svc\u002F${RC}-head-svc 18266:8265 &\ncurl -s http:\u002F\u002Flocalhost:18266\u002Fapi\u002Fjobs\u002F                              # lists jobs, find submission_id\ncurl -s \"http:\u002F\u002Flocalhost:18266\u002Fapi\u002Fjobs\u002F\u003Csubmission_id>\u002Flogs\"        # full driver log\n```\n\n`type=DRIVER` with `submission_id=null` means an exec-submitter run (no dashboard log endpoint — use `nrl-k8s job logs` instead). `type=SUBMISSION` has `submission_id` set and `\u002Fapi\u002Fjobs\u002F\u003Cid>\u002Flogs` works.\n\nWandb URL appears in the driver log on the first `wandb.init` call; grep `grep -oE 'https:\u002F\u002Fwandb\\.ai\u002F[A-Za-z0-9_.\u002F-]+'`.\n\n## 9. Stopping things\n\n| What to stop                     | Command                                                                              |\n| :------------------------------- | :----------------------------------------------------------------------------------- |\n| One training run                 | `nrl-k8s job stop \u003Crun-id> \u003Crecipe> --infra \u003Cinfra> --role training`                 |\n| All running Ray jobs on a cluster (+ submit new) | `nrl-k8s run \u003Crecipe> --infra \u003Cinfra> --replace`                         |\n| A long-lived RayCluster          | `nrl-k8s cluster down \u003Crecipe> --infra \u003Cinfra> --target kuberay.training --wait`     |\n| A RayJob (ephemeral)             | `kubectl delete rayjob \u003Cname> -n default` — only if `shutdownAfterJobFinishes` didn't fire |\n\nConfirm before deleting shared infra. The cost of `cluster down` on someone else's cluster is high.\n\n## 10. Verifying RayJob teardown\n\nAfter a `run --rayjob` completes with `--shutdown` (default), KubeRay should delete the RayCluster:\n\n```bash\nkubectl get rayjob   -n default \u003Crayjob-name>                        # jobDeploymentStatus = Complete\nkubectl get raycluster -n default | grep \u003Crayjob-name>               # no output = torn down\n```\n\nThe RayJob object itself sticks around for `--ttl` seconds (default 3600s) so you can still fetch logs.\n\n## 11. Common gotchas\n\n- **OmegaConf interpolation** eats `${VAR}` in recipe\u002Finfra YAML. Escape shell variables with `\\${VAR}` so OmegaConf passes them through to the pod shell verbatim.\n- **Megatron optimizer configs** don't carry `foreach` \u002F `fused`. Overrides like `~policy.optimizer.kwargs.foreach ~policy.optimizer.kwargs.fused` (valid for DTensor configs) break on Megatron recipes. Omit them for Megatron.\n- **DTensor vs Megatron** — MoE recipes typically use `megatron_cfg.enabled=true`; ensure `dtensor_cfg.enabled=false` in inherited defaults.\n- **Shared filesystem vs git divergence** — `codeSource: image|lustre` reads from the pod filesystem. If your local edits aren't on the shared filesystem the pods mount, the run is testing the on-disk version, not yours. Either sync via a helper pod (head pod exec is often blocked) or override via Hydra flags.\n- **Ephemeral-storage + readinessProbe** are injected by kuberay\u002FCDI webhooks at pod-apply time. Do NOT add them to the inline RayCluster spec.\n- **Node taints** vary per cluster. `tolerations: [{operator: Exists}]` on workers is defensive and worth keeping.\n- **Dashboard blank page** — Ray 2.52 installs dashboard assets as symlinks by default; `nrl-k8s cluster dashboard \u003Cname>` auto-reinstalls `ray[default] --link-mode=copy` to fix it. Bake `ENV UV_LINK_MODE=copy` in the image to avoid this entirely.\n- **`kubectl exec` is usually blocked** in automation — route around with `kubectl get ... -o yaml`, `kubectl logs`, and `kubectl port-forward` + Ray dashboard APIs.\n\n## 12. Checklist before calling a run \"done\"\n\nBefore reporting a launch as successful, verify:\n\n1. `kubectl get rayjob\u002Fraycluster -n default` shows the expected objects.\n2. `nrl-k8s job list` (or `curl \u002Fapi\u002Fjobs\u002F`) shows the job in `RUNNING` \u002F `SUCCEEDED`.\n3. Driver log contains `wandb.ai\u002F\u003Cproject>\u002Fruns\u002F\u003Cid>` (if wandb is enabled) — share the URL with the user.\n4. At least one `Processed prompts: 100%` line appears (confirms generation is wired).\n5. For `--rayjob` mode only: after `jobDeploymentStatus=Complete`, confirm `kubectl get raycluster | grep \u003Cname>` is empty (teardown worked).\n\n## 13. Dev pod\n\n`nrl-k8s dev` manages a lightweight CPU pod on the cluster for code syncing, debugging, and running `kubectl`\u002F`nrl-k8s` from within the cluster.\n\n```bash\n# One-time: set up secrets (HF token, wandb, SSH key, rclone)\nnrl-k8s dev setup-secrets --ssh-key ~\u002F.ssh\u002Fid_rsa --add-rclone\n\n# Create pod and exec in (idempotent — reuses existing pod)\nnrl-k8s dev connect\n\n# Switch image (must stop first — image change is warned but not auto-applied)\nnrl-k8s dev stop\nnrl-k8s dev connect --image nvcr.io\u002Fnvidian\u002Fnemo-rl:v0.7.0\n\n# Tear down\nnrl-k8s dev stop\n```\n\nThe dev pod:\n- Runs on a CPU-only node (anti-affinity to GPU nodes)\n- Mounts the shared `rl-workspace` PVC at `\u002Fmnt\u002Frl-workspace`\n- Sets `USER` env var to the `nrl-k8s` username (so `$USER` and `getpass.getuser()` work correctly despite running as root)\n- Installs `kubectl`, `rclone` (if configured) on first boot\n- Injects SSH keys and tokens via `envFrom` on a per-user K8s Secret\n\nThe pod's `default` service account needs an `edit` RoleBinding in the namespace for `kubectl` to work inside. `dev connect` checks this and prints the required YAML if missing.\n\n## 14. Where things live in the repo\n\n- CLI code: `infra\u002Fnrl_k8s\u002Fsrc\u002Fnrl_k8s\u002F` (`cli.py`, `orchestrate.py`, `manifest.py`, `rayjob.py`, `k8s.py`, `submitters\u002F`, `schema.py`).\n- Tests: `infra\u002Fnrl_k8s\u002Ftests\u002Funit\u002F` — run with `uv run --extra test pytest -x -q` from `infra\u002Fnrl_k8s\u002F`.\n- Recipe + infra examples: `infra\u002Fnrl_k8s\u002Fexamples\u002F`.\n- Base recipes this tool wraps: `examples\u002Fconfigs\u002Frecipes\u002Fllm\u002F…` and `examples\u002Fnemo_gym\u002F…`.\n",{"data":38,"body":49},{"name":4,"license":29,"description":6,"when_to_use":39,"allowed-tools":48},[40,41,42,43,44,45,46,47],"run this recipe on k8s","launch on the cluster","submit a training job","tear down the cluster","resubmit as rayjob","why is the run stuck","how do I get logs for job X","bring the cluster back up","Bash Read Grep Glob Edit Write",{"type":50,"children":51},"root",[52,61,100,107,124,224,245,250,372,378,391,403,520,533,539,552,561,703,708,743,785,799,819,924,953,959,964,1070,1118,1124,1136,1307,1312,1445,1451,1458,1656,1661,1745,1751,1854,1860,1935,1941,2268,2274,2475,2487,2493,2743,2763,2768,2961,3010,3030,3036,3135,3148,3154,3175,3281,3294,3300,3505,3511,3516,3618,3624,3649,3817,3822,3916,3952,3958,4079],{"type":53,"tag":54,"props":55,"children":57},"element","h1",{"id":56},"launch-nemo-rl-running-nemo-rl-recipes-on-kubernetes-via-nrl-k8s",[58],{"type":59,"value":60},"text","launch-nemo-rl — running NeMo-RL recipes on Kubernetes via nrl-k8s",{"type":53,"tag":62,"props":63,"children":64},"p",{},[65,67,74,76,82,84,90,92,98],{"type":59,"value":66},"This is the playbook for the ",{"type":53,"tag":68,"props":69,"children":71},"code",{"className":70},[],[72],{"type":59,"value":73},"nrl-k8s",{"type":59,"value":75}," CLI at ",{"type":53,"tag":68,"props":77,"children":79},{"className":78},[],[80],{"type":59,"value":81},"infra\u002Fnrl_k8s\u002F",{"type":59,"value":83},". Follow it when the user asks to launch \u002F iterate \u002F debug a NeMo-RL recipe on a Kubernetes cluster. Verify current state (",{"type":53,"tag":68,"props":85,"children":87},{"className":86},[],[88],{"type":59,"value":89},"kubectl",{"type":59,"value":91},", ",{"type":53,"tag":68,"props":93,"children":95},{"className":94},[],[96],{"type":59,"value":97},"git log",{"type":59,"value":99},", the recipe + infra files) before acting — the cluster is shared and the cost of a wrong action is high.",{"type":53,"tag":101,"props":102,"children":104},"h2",{"id":103},"_1-one-command-two-modes",[105],{"type":59,"value":106},"1. One command, two modes",{"type":53,"tag":62,"props":108,"children":109},{},[110,112,122],{"type":59,"value":111},"There is a single top-level submission command: ",{"type":53,"tag":113,"props":114,"children":115},"strong",{},[116],{"type":53,"tag":68,"props":117,"children":119},{"className":118},[],[120],{"type":59,"value":121},"nrl-k8s run",{"type":59,"value":123},". It has two lifecycle modes.",{"type":53,"tag":125,"props":126,"children":127},"table",{},[128,158],{"type":53,"tag":129,"props":130,"children":131},"thead",{},[132],{"type":53,"tag":133,"props":134,"children":135},"tr",{},[136,143,148,153],{"type":53,"tag":137,"props":138,"children":140},"th",{"align":139},"left",[141],{"type":59,"value":142},"Mode",{"type":53,"tag":137,"props":144,"children":145},{"align":139},[146],{"type":59,"value":147},"Invocation",{"type":53,"tag":137,"props":149,"children":150},{"align":139},[151],{"type":59,"value":152},"When to use",{"type":53,"tag":137,"props":154,"children":155},{"align":139},[156],{"type":59,"value":157},"Cluster after?",{"type":53,"tag":159,"props":160,"children":161},"tbody",{},[162,189],{"type":53,"tag":133,"props":163,"children":164},{},[165,171,179,184],{"type":53,"tag":166,"props":167,"children":168},"td",{"align":139},[169],{"type":59,"value":170},"Ephemeral (default)",{"type":53,"tag":166,"props":172,"children":173},{"align":139},[174],{"type":53,"tag":68,"props":175,"children":177},{"className":176},[],[178],{"type":59,"value":121},{"type":53,"tag":166,"props":180,"children":181},{"align":139},[182],{"type":59,"value":183},"One-shot. KubeRay applies a RayJob, runs, tears the cluster down. Best for most runs.",{"type":53,"tag":166,"props":185,"children":186},{"align":139},[187],{"type":59,"value":188},"No (auto)",{"type":53,"tag":133,"props":190,"children":191},{},[192,197,206,219],{"type":53,"tag":166,"props":193,"children":194},{"align":139},[195],{"type":59,"value":196},"Long-lived",{"type":53,"tag":166,"props":198,"children":199},{"align":139},[200],{"type":53,"tag":68,"props":201,"children":203},{"className":202},[],[204],{"type":59,"value":205},"nrl-k8s run --raycluster",{"type":53,"tag":166,"props":207,"children":208},{"align":139},[209,211,217],{"type":59,"value":210},"Dev loop. Reuses a matching live cluster, applies if absent, warns + reuses on drift (pass ",{"type":53,"tag":68,"props":212,"children":214},{"className":213},[],[215],{"type":59,"value":216},"--recreate",{"type":59,"value":218}," to replace). Then submits daemons and training. First-choice for iteration.",{"type":53,"tag":166,"props":220,"children":221},{"align":139},[222],{"type":59,"value":223},"Yes",{"type":53,"tag":62,"props":225,"children":226},{},[227,229,235,237,243],{"type":59,"value":228},"Ask: ",{"type":53,"tag":230,"props":231,"children":232},"em",{},[233],{"type":59,"value":234},"Do I need this cluster after the run?",{"type":59,"value":236}," If yes, use ",{"type":53,"tag":68,"props":238,"children":240},{"className":239},[],[241],{"type":59,"value":242},"--raycluster",{"type":59,"value":244},". Otherwise use the default (ephemeral).",{"type":53,"tag":62,"props":246,"children":247},{},[248],{"type":59,"value":249},"The rest of the CLI is observability \u002F stage-by-stage control:",{"type":53,"tag":125,"props":251,"children":252},{},[253,269],{"type":53,"tag":129,"props":254,"children":255},{},[256],{"type":53,"tag":133,"props":257,"children":258},{},[259,264],{"type":53,"tag":137,"props":260,"children":261},{"align":139},[262],{"type":59,"value":263},"Command",{"type":53,"tag":137,"props":265,"children":266},{"align":139},[267],{"type":59,"value":268},"Purpose",{"type":53,"tag":159,"props":270,"children":271},{},[272,297,314,338,355],{"type":53,"tag":133,"props":273,"children":274},{},[275,284],{"type":53,"tag":166,"props":276,"children":277},{"align":139},[278],{"type":53,"tag":68,"props":279,"children":281},{"className":280},[],[282],{"type":59,"value":283},"nrl-k8s check",{"type":53,"tag":166,"props":285,"children":286},{"align":139},[287,289,295],{"type":59,"value":288},"Validate a recipe + infra pair; optionally write the fully-resolved manifests (",{"type":53,"tag":68,"props":290,"children":292},{"className":291},[],[293],{"type":59,"value":294},"-o",{"type":59,"value":296},").",{"type":53,"tag":133,"props":298,"children":299},{},[300,309],{"type":53,"tag":166,"props":301,"children":302},{"align":139},[303],{"type":53,"tag":68,"props":304,"children":306},{"className":305},[],[307],{"type":59,"value":308},"nrl-k8s status",{"type":53,"tag":166,"props":310,"children":311},{"align":139},[312],{"type":59,"value":313},"Per-role RayCluster state, head pod phase, worker pod phases, daemon job status.",{"type":53,"tag":133,"props":315,"children":316},{},[317,326],{"type":53,"tag":166,"props":318,"children":319},{"align":139},[320],{"type":53,"tag":68,"props":321,"children":323},{"className":322},[],[324],{"type":59,"value":325},"nrl-k8s cluster up\u002Fdown\u002Flist\u002Fdashboard",{"type":53,"tag":166,"props":327,"children":328},{"align":139},[329,331,337],{"type":59,"value":330},"Manage RayClusters independently of a run (e.g. render a manifest with ",{"type":53,"tag":68,"props":332,"children":334},{"className":333},[],[335],{"type":59,"value":336},"--dry-run",{"type":59,"value":296},{"type":53,"tag":133,"props":339,"children":340},{},[341,350],{"type":53,"tag":166,"props":342,"children":343},{"align":139},[344],{"type":53,"tag":68,"props":345,"children":347},{"className":346},[],[348],{"type":59,"value":349},"nrl-k8s job list\u002Flogs\u002Fstop",{"type":53,"tag":166,"props":351,"children":352},{"align":139},[353],{"type":59,"value":354},"Observability over Ray Jobs already submitted to a role's cluster.",{"type":53,"tag":133,"props":356,"children":357},{},[358,367],{"type":53,"tag":166,"props":359,"children":360},{"align":139},[361],{"type":53,"tag":68,"props":362,"children":364},{"className":363},[],[365],{"type":59,"value":366},"nrl-k8s logs",{"type":53,"tag":166,"props":368,"children":369},{"align":139},[370],{"type":59,"value":371},"Tail a role's pod \u002F daemon logs without needing a submission id.",{"type":53,"tag":101,"props":373,"children":375},{"id":374},"_2-recipe-infra-pair",[376],{"type":59,"value":377},"2. Recipe + infra pair",{"type":53,"tag":62,"props":379,"children":380},{},[381,383,389],{"type":59,"value":382},"Every launch takes two files. Pass the infra with ",{"type":53,"tag":68,"props":384,"children":386},{"className":385},[],[387],{"type":59,"value":388},"--infra",{"type":59,"value":390},", not merged inline:",{"type":53,"tag":392,"props":393,"children":397},"pre",{"className":394,"code":396,"language":59},[395],"language-text","nrl-k8s run infra\u002Fnrl_k8s\u002Fexamples\u002F\u003Crecipe>.yaml \\\n  --infra infra\u002Fnrl_k8s\u002Fexamples\u002F\u003Crecipe>.\u003Cprofile>.infra.yaml\n",[398],{"type":53,"tag":68,"props":399,"children":401},{"__ignoreMap":400},"",[402],{"type":59,"value":396},{"type":53,"tag":404,"props":405,"children":406},"ul",{},[407,450],{"type":53,"tag":408,"props":409,"children":410},"li",{},[411,416,418,424,426,432,434,440,442,448],{"type":53,"tag":113,"props":412,"children":413},{},[414],{"type":59,"value":415},"Recipe",{"type":59,"value":417}," (e.g. ",{"type":53,"tag":68,"props":419,"children":421},{"className":420},[],[422],{"type":59,"value":423},"qwen3_30b_math_8n_4gpu.yaml",{"type":59,"value":425},") — NeMo-RL config: model, GRPO\u002FSFT knobs, ",{"type":53,"tag":68,"props":427,"children":429},{"className":428},[],[430],{"type":59,"value":431},"cluster.{gpus_per_node,num_nodes}",{"type":59,"value":433},". Uses ",{"type":53,"tag":68,"props":435,"children":437},{"className":436},[],[438],{"type":59,"value":439},"defaults:",{"type":59,"value":441}," to inherit from ",{"type":53,"tag":68,"props":443,"children":445},{"className":444},[],[446],{"type":59,"value":447},"examples\u002Fconfigs\u002Frecipes\u002Fllm\u002F...",{"type":59,"value":449},".",{"type":53,"tag":408,"props":451,"children":452},{},[453,458,459,465,467,473,475,481,482,488,489,495,497,503,505,511,513,519],{"type":53,"tag":113,"props":454,"children":455},{},[456],{"type":59,"value":457},"Infra",{"type":59,"value":417},{"type":53,"tag":68,"props":460,"children":462},{"className":461},[],[463],{"type":59,"value":464},"*.\u003Cprofile>.infra.yaml",{"type":59,"value":466},") — K8s\u002FRay shape: namespace, image, service account, RayCluster spec under ",{"type":53,"tag":68,"props":468,"children":470},{"className":469},[],[471],{"type":59,"value":472},"kuberay:",{"type":59,"value":474},", optional Deployments under ",{"type":53,"tag":68,"props":476,"children":478},{"className":477},[],[479],{"type":59,"value":480},"deployments:",{"type":59,"value":91},{"type":53,"tag":68,"props":483,"children":485},{"className":484},[],[486],{"type":59,"value":487},"submit.submitter",{"type":59,"value":91},{"type":53,"tag":68,"props":490,"children":492},{"className":491},[],[493],{"type":59,"value":494},"launch.{mode,codeSource,codePath,entrypoint}",{"type":59,"value":496},". Pair names follow ",{"type":53,"tag":68,"props":498,"children":500},{"className":499},[],[501],{"type":59,"value":502},"\u003Crecipe>.\u003Cprofile>[.prod].infra.yaml",{"type":59,"value":504}," where ",{"type":53,"tag":68,"props":506,"children":508},{"className":507},[],[509],{"type":59,"value":510},"\u003Cprofile>",{"type":59,"value":512}," names the hardware target (e.g. ",{"type":53,"tag":68,"props":514,"children":516},{"className":515},[],[517],{"type":59,"value":518},"gb300",{"type":59,"value":296},{"type":53,"tag":62,"props":521,"children":522},{},[523,525,531],{"type":59,"value":524},"Example pairs in ",{"type":53,"tag":68,"props":526,"children":528},{"className":527},[],[529],{"type":59,"value":530},"infra\u002Fnrl_k8s\u002Fexamples\u002F",{"type":59,"value":532}," — read the neighbouring files to see the current conventions for the target profile.",{"type":53,"tag":101,"props":534,"children":536},{"id":535},"_3-long-lived-mode-flags",[537],{"type":59,"value":538},"3. Long-lived mode flags",{"type":53,"tag":62,"props":540,"children":541},{},[542,544,550],{"type":59,"value":543},"Three independent dimensions. ",{"type":53,"tag":68,"props":545,"children":547},{"className":546},[],[548],{"type":59,"value":549},"--mode",{"type":59,"value":551}," is a macro that picks defaults; individual flags override it.",{"type":53,"tag":392,"props":553,"children":556},{"className":554,"code":555,"language":59},[395],"--mode interactive   → --submitter portForward  --code-source upload  (tails logs)\n--mode batch         → --submitter exec         --code-source image   (returns after nohup)\n",[557],{"type":53,"tag":68,"props":558,"children":559},{"__ignoreMap":400},[560],{"type":59,"value":555},{"type":53,"tag":404,"props":562,"children":563},{},[564,629,678],{"type":53,"tag":408,"props":565,"children":566},{},[567,572,574,580,582,588,590,596,598,604,605,611,613,619,621,627],{"type":53,"tag":113,"props":568,"children":569},{},[570],{"type":59,"value":571},"Submitter",{"type":59,"value":573},": ",{"type":53,"tag":68,"props":575,"children":577},{"className":576},[],[578],{"type":59,"value":579},"portForward",{"type":59,"value":581}," uses ",{"type":53,"tag":68,"props":583,"children":585},{"className":584},[],[586],{"type":59,"value":587},"kubectl port-forward",{"type":59,"value":589}," + Ray Job SDK (gets a ",{"type":53,"tag":68,"props":591,"children":593},{"className":592},[],[594],{"type":59,"value":595},"submission_id",{"type":59,"value":597}," the dashboard tracks). ",{"type":53,"tag":68,"props":599,"children":601},{"className":600},[],[602],{"type":59,"value":603},"exec",{"type":59,"value":581},{"type":53,"tag":68,"props":606,"children":608},{"className":607},[],[609],{"type":59,"value":610},"kubectl exec",{"type":59,"value":612}," + ",{"type":53,"tag":68,"props":614,"children":616},{"className":615},[],[617],{"type":59,"value":618},"nohup",{"type":59,"value":620}," on the head pod (no submission_id; driver appears as ",{"type":53,"tag":68,"props":622,"children":624},{"className":623},[],[625],{"type":59,"value":626},"type=DRIVER",{"type":59,"value":628}," in the dashboard).",{"type":53,"tag":408,"props":630,"children":631},{},[632,637,638,644,646,652,654,660,662,668,670,676],{"type":53,"tag":113,"props":633,"children":634},{},[635],{"type":59,"value":636},"Code source",{"type":59,"value":573},{"type":53,"tag":68,"props":639,"children":641},{"className":640},[],[642],{"type":59,"value":643},"upload",{"type":59,"value":645}," stages a working_dir from the laptop (Ray 100 MiB cap). ",{"type":53,"tag":68,"props":647,"children":649},{"className":648},[],[650],{"type":59,"value":651},"image",{"type":59,"value":653}," \u002F ",{"type":53,"tag":68,"props":655,"children":657},{"className":656},[],[658],{"type":59,"value":659},"lustre",{"type":59,"value":661}," expect code on the pod's filesystem — paired with ",{"type":53,"tag":68,"props":663,"children":665},{"className":664},[],[666],{"type":59,"value":667},"--code-path",{"type":59,"value":669}," (typically ",{"type":53,"tag":68,"props":671,"children":673},{"className":672},[],[674],{"type":59,"value":675},"\u002Fopt\u002Fnemo-rl",{"type":59,"value":677},"), which is a subPath of the shared-filesystem PVC mount in the standard infra examples.",{"type":53,"tag":408,"props":679,"children":680},{},[681,686,687,693,695,701],{"type":53,"tag":113,"props":682,"children":683},{},[684],{"type":59,"value":685},"Wait",{"type":59,"value":573},{"type":53,"tag":68,"props":688,"children":690},{"className":689},[],[691],{"type":59,"value":692},"--wait",{"type":59,"value":694}," tails logs until terminal; ",{"type":53,"tag":68,"props":696,"children":698},{"className":697},[],[699],{"type":59,"value":700},"--no-wait",{"type":59,"value":702}," returns as soon as the driver is running.",{"type":53,"tag":62,"props":704,"children":705},{},[706],{"type":59,"value":707},"Other long-lived-only flags:",{"type":53,"tag":404,"props":709,"children":710},{},[711,722,732],{"type":53,"tag":408,"props":712,"children":713},{},[714,720],{"type":53,"tag":68,"props":715,"children":717},{"className":716},[],[718],{"type":59,"value":719},"--replace",{"type":59,"value":721}," — stop any running training \u002F daemon job before submitting new ones (suffixes daemon submissionIds with a timestamp so Ray accepts the resubmit).",{"type":53,"tag":408,"props":723,"children":724},{},[725,730],{"type":53,"tag":68,"props":726,"children":728},{"className":727},[],[729],{"type":59,"value":216},{"type":59,"value":731}," — delete + re-apply a RayCluster whose live spec has drifted from the rendered manifest (default is warn + reuse).",{"type":53,"tag":408,"props":733,"children":734},{},[735,741],{"type":53,"tag":68,"props":736,"children":738},{"className":737},[],[739],{"type":59,"value":740},"--skip-daemons",{"type":59,"value":742}," — bring up all declared clusters but only submit training. Use on disagg recipes where gym\u002Fgeneration are already healthy.",{"type":53,"tag":62,"props":744,"children":745},{},[746,748,754,756,767,769,775,777,783],{"type":59,"value":747},"Gotcha: on infra where the entrypoint does ",{"type":53,"tag":68,"props":749,"children":751},{"className":750},[],[752],{"type":59,"value":753},"cd \u002Fopt\u002Fnemo-rl",{"type":59,"value":755}," (or another in-image \u002F Lustre path) and loads the recipe from there, ",{"type":53,"tag":113,"props":757,"children":758},{},[759,765],{"type":53,"tag":68,"props":760,"children":762},{"className":761},[],[763],{"type":59,"value":764},"--code-source upload",{"type":59,"value":766}," does NOT override the recipe on the pod",{"type":59,"value":768}," — the uploaded working_dir sits in ",{"type":53,"tag":68,"props":770,"children":772},{"className":771},[],[773],{"type":59,"value":774},"\u002Ftmp\u002Fray\u002F...",{"type":59,"value":776}," but the entrypoint ",{"type":53,"tag":68,"props":778,"children":780},{"className":779},[],[781],{"type":59,"value":782},"cd",{"type":59,"value":784},"s away from it. To actually test a local recipe change, either sync your edits to the shared filesystem mounted into the pods or flip the Hydra overrides in the entrypoint.",{"type":53,"tag":101,"props":786,"children":788},{"id":787},"_4-ephemeral-mode-flags-rayjob",[789,791,797],{"type":59,"value":790},"4. Ephemeral mode flags (",{"type":53,"tag":68,"props":792,"children":794},{"className":793},[],[795],{"type":59,"value":796},"--rayjob",{"type":59,"value":798},")",{"type":53,"tag":62,"props":800,"children":801},{},[802,804,809,811,817],{"type":59,"value":803},"When ",{"type":53,"tag":68,"props":805,"children":807},{"className":806},[],[808],{"type":59,"value":796},{"type":59,"value":810}," is set, ",{"type":53,"tag":68,"props":812,"children":814},{"className":813},[],[815],{"type":59,"value":816},"run",{"type":59,"value":818}," branches into the RayJob code path. Relevant flags:",{"type":53,"tag":404,"props":820,"children":821},{},[822,833,852,863,896,914],{"type":53,"tag":408,"props":823,"children":824},{},[825,831],{"type":53,"tag":68,"props":826,"children":828},{"className":827},[],[829],{"type":59,"value":830},"--rayjob-name NAME",{"type":59,"value":832}," — RayJob metadata name (defaults to the training cluster name).",{"type":53,"tag":408,"props":834,"children":835},{},[836,842,844,850],{"type":53,"tag":68,"props":837,"children":839},{"className":838},[],[840],{"type":59,"value":841},"--shutdown \u002F --no-shutdown",{"type":59,"value":843}," — default ",{"type":53,"tag":68,"props":845,"children":847},{"className":846},[],[848],{"type":59,"value":849},"true",{"type":59,"value":851},": KubeRay deletes the RayCluster once the Ray Job reaches a terminal state.",{"type":53,"tag":408,"props":853,"children":854},{},[855,861],{"type":53,"tag":68,"props":856,"children":858},{"className":857},[],[859],{"type":59,"value":860},"--ttl SECONDS",{"type":59,"value":862}," — default 3600s: keep the RayJob object around after the run finishes for post-mortem log access.",{"type":53,"tag":408,"props":864,"children":865},{},[866,872,873,879,881,887,889,894],{"type":53,"tag":68,"props":867,"children":869},{"className":868},[],[870],{"type":59,"value":871},"--wait \u002F --no-wait",{"type":59,"value":843},{"type":53,"tag":68,"props":874,"children":876},{"className":875},[],[877],{"type":59,"value":878},"wait",{"type":59,"value":880},": poll ",{"type":53,"tag":68,"props":882,"children":884},{"className":883},[],[885],{"type":59,"value":886},"jobDeploymentStatus",{"type":59,"value":888}," until Complete\u002FFailed. ",{"type":53,"tag":68,"props":890,"children":892},{"className":891},[],[893],{"type":59,"value":700},{"type":59,"value":895}," returns as soon as the RayJob is applied.",{"type":53,"tag":408,"props":897,"children":898},{},[899,905,907,912],{"type":53,"tag":68,"props":900,"children":902},{"className":901},[],[903],{"type":59,"value":904},"--timeout SECONDS",{"type":59,"value":906}," — default 86400s (24h): bound the ",{"type":53,"tag":68,"props":908,"children":910},{"className":909},[],[911],{"type":59,"value":692},{"type":59,"value":913}," poll.",{"type":53,"tag":408,"props":915,"children":916},{},[917,922],{"type":53,"tag":68,"props":918,"children":920},{"className":919},[],[921],{"type":59,"value":336},{"type":59,"value":923}," — render the RayJob manifest and print it; do not apply.",{"type":53,"tag":62,"props":925,"children":926},{},[927,932,933,938,939,944,946,951],{"type":53,"tag":68,"props":928,"children":930},{"className":929},[],[931],{"type":59,"value":719},{"type":59,"value":653},{"type":53,"tag":68,"props":934,"children":936},{"className":935},[],[937],{"type":59,"value":216},{"type":59,"value":653},{"type":53,"tag":68,"props":940,"children":942},{"className":941},[],[943],{"type":59,"value":740},{"type":59,"value":945}," are silently ignored in ",{"type":53,"tag":68,"props":947,"children":949},{"className":948},[],[950],{"type":59,"value":796},{"type":59,"value":952}," mode (KubeRay owns lifecycle).",{"type":53,"tag":101,"props":954,"children":956},{"id":955},"_5-iterating-on-a-config-without-touching-the-shared-filesystem",[957],{"type":59,"value":958},"5. Iterating on a config without touching the shared filesystem",{"type":53,"tag":62,"props":960,"children":961},{},[962],{"type":59,"value":963},"When the recipe on the pod filesystem has the wrong value for your experiment, use Hydra overrides on the entrypoint instead of forking the recipe. Pattern:",{"type":53,"tag":392,"props":965,"children":969},{"className":966,"code":967,"language":968,"meta":400,"style":400},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","entrypoint: |\n  set -eu\n  cd \u002Fopt\u002Fnemo-rl\n  RUN_ID=\"\\${RAY_JOB_SUBMISSION_ID:-\\${NRL_K8S_RUN_ID:-$(date -u +%Y%m%d-%H%M%S)}}\"\n  python -u examples\u002Frun_grpo.py \\\n    --config infra\u002Fnrl_k8s\u002Fexamples\u002F\u003Crecipe>.yaml \\\n    logger.wandb_enabled=true \\\n    logger.wandb.project=\u003Cproject> \\\n    \"logger.wandb.name=\u003Crun-name>-\\${RUN_ID}\"\n","yaml",[970],{"type":53,"tag":68,"props":971,"children":972},{"__ignoreMap":400},[973,997,1007,1016,1025,1034,1043,1052,1061],{"type":53,"tag":974,"props":975,"children":978},"span",{"class":976,"line":977},"line",1,[979,985,991],{"type":53,"tag":974,"props":980,"children":982},{"style":981},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[983],{"type":59,"value":984},"entrypoint",{"type":53,"tag":974,"props":986,"children":988},{"style":987},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[989],{"type":59,"value":990},":",{"type":53,"tag":974,"props":992,"children":994},{"style":993},"--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic",[995],{"type":59,"value":996}," |\n",{"type":53,"tag":974,"props":998,"children":1000},{"class":976,"line":999},2,[1001],{"type":53,"tag":974,"props":1002,"children":1004},{"style":1003},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1005],{"type":59,"value":1006},"  set -eu\n",{"type":53,"tag":974,"props":1008,"children":1010},{"class":976,"line":1009},3,[1011],{"type":53,"tag":974,"props":1012,"children":1013},{"style":1003},[1014],{"type":59,"value":1015},"  cd \u002Fopt\u002Fnemo-rl\n",{"type":53,"tag":974,"props":1017,"children":1019},{"class":976,"line":1018},4,[1020],{"type":53,"tag":974,"props":1021,"children":1022},{"style":1003},[1023],{"type":59,"value":1024},"  RUN_ID=\"\\${RAY_JOB_SUBMISSION_ID:-\\${NRL_K8S_RUN_ID:-$(date -u +%Y%m%d-%H%M%S)}}\"\n",{"type":53,"tag":974,"props":1026,"children":1028},{"class":976,"line":1027},5,[1029],{"type":53,"tag":974,"props":1030,"children":1031},{"style":1003},[1032],{"type":59,"value":1033},"  python -u examples\u002Frun_grpo.py \\\n",{"type":53,"tag":974,"props":1035,"children":1037},{"class":976,"line":1036},6,[1038],{"type":53,"tag":974,"props":1039,"children":1040},{"style":1003},[1041],{"type":59,"value":1042},"    --config infra\u002Fnrl_k8s\u002Fexamples\u002F\u003Crecipe>.yaml \\\n",{"type":53,"tag":974,"props":1044,"children":1046},{"class":976,"line":1045},7,[1047],{"type":53,"tag":974,"props":1048,"children":1049},{"style":1003},[1050],{"type":59,"value":1051},"    logger.wandb_enabled=true \\\n",{"type":53,"tag":974,"props":1053,"children":1055},{"class":976,"line":1054},8,[1056],{"type":53,"tag":974,"props":1057,"children":1058},{"style":1003},[1059],{"type":59,"value":1060},"    logger.wandb.project=\u003Cproject> \\\n",{"type":53,"tag":974,"props":1062,"children":1064},{"class":976,"line":1063},9,[1065],{"type":53,"tag":974,"props":1066,"children":1067},{"style":1003},[1068],{"type":59,"value":1069},"    \"logger.wandb.name=\u003Crun-name>-\\${RUN_ID}\"\n",{"type":53,"tag":62,"props":1071,"children":1072},{},[1073,1084,1086,1092,1094,1100,1102,1108,1110,1116],{"type":53,"tag":113,"props":1074,"children":1075},{},[1076,1078],{"type":59,"value":1077},"Escape ",{"type":53,"tag":68,"props":1079,"children":1081},{"className":1080},[],[1082],{"type":59,"value":1083},"${…}",{"type":59,"value":1085}," with a backslash. OmegaConf otherwise interprets it as interpolation and errors on shell-style ",{"type":53,"tag":68,"props":1087,"children":1089},{"className":1088},[],[1090],{"type":59,"value":1091},"${VAR:-default}",{"type":59,"value":1093},". ",{"type":53,"tag":68,"props":1095,"children":1097},{"className":1096},[],[1098],{"type":59,"value":1099},"RUN_ID",{"type":59,"value":1101}," resolves to ",{"type":53,"tag":68,"props":1103,"children":1105},{"className":1104},[],[1106],{"type":59,"value":1107},"RAY_JOB_SUBMISSION_ID",{"type":59,"value":1109}," (injected by KubeRay in rayjob mode) → ",{"type":53,"tag":68,"props":1111,"children":1113},{"className":1112},[],[1114],{"type":59,"value":1115},"NRL_K8S_RUN_ID",{"type":59,"value":1117}," (injected by the CLI in long-lived mode) → local timestamp — so the name is unique across either path.",{"type":53,"tag":101,"props":1119,"children":1121},{"id":1120},"_6-per-profile-concerns-hardware-scheduler-dra",[1122],{"type":59,"value":1123},"6. Per-profile concerns (hardware + scheduler + DRA)",{"type":53,"tag":62,"props":1125,"children":1126},{},[1127,1129,1134],{"type":59,"value":1128},"Every infra YAML encodes a hardware\u002Fscheduler profile. The concrete examples in ",{"type":53,"tag":68,"props":1130,"children":1132},{"className":1131},[],[1133],{"type":59,"value":530},{"type":59,"value":1135}," are authoritative for the profiles they target — read the neighbouring infra file before writing a new one. Things that commonly vary:",{"type":53,"tag":404,"props":1137,"children":1138},{},[1139,1164,1182,1222,1248,1274],{"type":53,"tag":408,"props":1140,"children":1141},{},[1142,1147,1149,1155,1157,1163],{"type":53,"tag":113,"props":1143,"children":1144},{},[1145],{"type":59,"value":1146},"Per-node GPUs",{"type":59,"value":1148}," (e.g. 4 vs 8) — must match ",{"type":53,"tag":68,"props":1150,"children":1152},{"className":1151},[],[1153],{"type":59,"value":1154},"cluster.gpus_per_node",{"type":59,"value":1156}," in the recipe, otherwise workers stay ",{"type":53,"tag":68,"props":1158,"children":1160},{"className":1159},[],[1161],{"type":59,"value":1162},"Pending",{"type":59,"value":449},{"type":53,"tag":408,"props":1165,"children":1166},{},[1167,1172,1174,1180],{"type":53,"tag":113,"props":1168,"children":1169},{},[1170],{"type":59,"value":1171},"Node selectors",{"type":59,"value":1173}," — head pods usually land on a CPU-only node pool; GPU workers match on ",{"type":53,"tag":68,"props":1175,"children":1177},{"className":1176},[],[1178],{"type":59,"value":1179},"nvidia.com\u002Fgpu.product",{"type":59,"value":1181}," or a node-group label.",{"type":53,"tag":408,"props":1183,"children":1184},{},[1185,1190,1192,1198,1199,1205,1207,1213,1214,1220],{"type":53,"tag":113,"props":1186,"children":1187},{},[1188],{"type":59,"value":1189},"Scheduler",{"type":59,"value":1191}," — KAI (",{"type":53,"tag":68,"props":1193,"children":1195},{"className":1194},[],[1196],{"type":59,"value":1197},"schedulerName: kai-scheduler",{"type":59,"value":612},{"type":53,"tag":68,"props":1200,"children":1202},{"className":1201},[],[1203],{"type":59,"value":1204},"kai.scheduler\u002Fqueue",{"type":59,"value":1206}," label) with topology annotations (",{"type":53,"tag":68,"props":1208,"children":1210},{"className":1209},[],[1211],{"type":59,"value":1212},"kai.scheduler\u002Ftopology",{"type":59,"value":91},{"type":53,"tag":68,"props":1215,"children":1217},{"className":1216},[],[1218],{"type":59,"value":1219},"kai.scheduler\u002Ftopology-required-placement",{"type":59,"value":1221},") gang-schedules workers into one clique. Without it, pods may land on different racks and NVLink\u002FRoCE won't span them.",{"type":53,"tag":408,"props":1223,"children":1224},{},[1225,1230,1232,1238,1240,1246],{"type":53,"tag":113,"props":1226,"children":1227},{},[1228],{"type":59,"value":1229},"DRA claims",{"type":59,"value":1231}," — ComputeDomain + RoCE are attached via ",{"type":53,"tag":68,"props":1233,"children":1235},{"className":1234},[],[1236],{"type":59,"value":1237},"resourceClaims",{"type":59,"value":1239}," referencing ",{"type":53,"tag":68,"props":1241,"children":1243},{"className":1242},[],[1244],{"type":59,"value":1245},"ResourceClaimTemplate",{"type":59,"value":1247},"s. The CLI auto-creates\u002Fdeletes these when the worker pod spec contains DRA claim references — no manual setup needed.",{"type":53,"tag":408,"props":1249,"children":1250},{},[1251,1256,1258,1264,1266,1272],{"type":53,"tag":113,"props":1252,"children":1253},{},[1254],{"type":59,"value":1255},"Secrets",{"type":59,"value":1257}," — always via ",{"type":53,"tag":68,"props":1259,"children":1261},{"className":1260},[],[1262],{"type":59,"value":1263},"secretKeyRef",{"type":59,"value":1265}," (",{"type":53,"tag":68,"props":1267,"children":1269},{"className":1268},[],[1270],{"type":59,"value":1271},"wandb-api-key",{"type":59,"value":1273},", image pull secret). Never embed.",{"type":53,"tag":408,"props":1275,"children":1276},{},[1277,1282,1284,1289,1291,1297,1299,1305],{"type":53,"tag":113,"props":1278,"children":1279},{},[1280],{"type":59,"value":1281},"Shared filesystem mounts",{"type":59,"value":1283}," — typically a Lustre PVC mounted twice: once at the code path (e.g. ",{"type":53,"tag":68,"props":1285,"children":1287},{"className":1286},[],[1288],{"type":59,"value":675},{"type":59,"value":1290}," with a user-scoped ",{"type":53,"tag":68,"props":1292,"children":1294},{"className":1293},[],[1295],{"type":59,"value":1296},"subPath",{"type":59,"value":1298},") and once at a workspace root (e.g. ",{"type":53,"tag":68,"props":1300,"children":1302},{"className":1301},[],[1303],{"type":59,"value":1304},"\u002Fmnt\u002Frl-workspace",{"type":59,"value":1306},") for datasets, HF cache, and checkpoints.",{"type":53,"tag":62,"props":1308,"children":1309},{},[1310],{"type":59,"value":1311},"Before applying an infra, verify prereqs exist in the target namespace:",{"type":53,"tag":392,"props":1313,"children":1317},{"className":1314,"code":1315,"language":1316,"meta":400,"style":400},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","kubectl get pvc \u003Cworkspace-pvc>\nkubectl get secret \u003Cwandb-secret> \u003Cimage-pull-secret>\nkubectl get sa \u003Cservice-account>\n","bash",[1318],{"type":53,"tag":68,"props":1319,"children":1320},{"__ignoreMap":400},[1321,1360,1412],{"type":53,"tag":974,"props":1322,"children":1323},{"class":976,"line":977},[1324,1329,1334,1339,1344,1349,1355],{"type":53,"tag":974,"props":1325,"children":1327},{"style":1326},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1328],{"type":59,"value":89},{"type":53,"tag":974,"props":1330,"children":1331},{"style":1003},[1332],{"type":59,"value":1333}," get",{"type":53,"tag":974,"props":1335,"children":1336},{"style":1003},[1337],{"type":59,"value":1338}," pvc",{"type":53,"tag":974,"props":1340,"children":1341},{"style":987},[1342],{"type":59,"value":1343}," \u003C",{"type":53,"tag":974,"props":1345,"children":1346},{"style":1003},[1347],{"type":59,"value":1348},"workspace-pv",{"type":53,"tag":974,"props":1350,"children":1352},{"style":1351},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1353],{"type":59,"value":1354},"c",{"type":53,"tag":974,"props":1356,"children":1357},{"style":987},[1358],{"type":59,"value":1359},">\n",{"type":53,"tag":974,"props":1361,"children":1362},{"class":976,"line":999},[1363,1367,1371,1376,1380,1385,1390,1395,1399,1404,1408],{"type":53,"tag":974,"props":1364,"children":1365},{"style":1326},[1366],{"type":59,"value":89},{"type":53,"tag":974,"props":1368,"children":1369},{"style":1003},[1370],{"type":59,"value":1333},{"type":53,"tag":974,"props":1372,"children":1373},{"style":1003},[1374],{"type":59,"value":1375}," secret",{"type":53,"tag":974,"props":1377,"children":1378},{"style":987},[1379],{"type":59,"value":1343},{"type":53,"tag":974,"props":1381,"children":1382},{"style":1003},[1383],{"type":59,"value":1384},"wandb-secre",{"type":53,"tag":974,"props":1386,"children":1387},{"style":1351},[1388],{"type":59,"value":1389},"t",{"type":53,"tag":974,"props":1391,"children":1392},{"style":987},[1393],{"type":59,"value":1394},">",{"type":53,"tag":974,"props":1396,"children":1397},{"style":987},[1398],{"type":59,"value":1343},{"type":53,"tag":974,"props":1400,"children":1401},{"style":1003},[1402],{"type":59,"value":1403},"image-pull-secre",{"type":53,"tag":974,"props":1405,"children":1406},{"style":1351},[1407],{"type":59,"value":1389},{"type":53,"tag":974,"props":1409,"children":1410},{"style":987},[1411],{"type":59,"value":1359},{"type":53,"tag":974,"props":1413,"children":1414},{"class":976,"line":1009},[1415,1419,1423,1428,1432,1437,1441],{"type":53,"tag":974,"props":1416,"children":1417},{"style":1326},[1418],{"type":59,"value":89},{"type":53,"tag":974,"props":1420,"children":1421},{"style":1003},[1422],{"type":59,"value":1333},{"type":53,"tag":974,"props":1424,"children":1425},{"style":1003},[1426],{"type":59,"value":1427}," sa",{"type":53,"tag":974,"props":1429,"children":1430},{"style":987},[1431],{"type":59,"value":1343},{"type":53,"tag":974,"props":1433,"children":1434},{"style":1003},[1435],{"type":59,"value":1436},"service-accoun",{"type":53,"tag":974,"props":1438,"children":1439},{"style":1351},[1440],{"type":59,"value":1389},{"type":53,"tag":974,"props":1442,"children":1443},{"style":987},[1444],{"type":59,"value":1359},{"type":53,"tag":101,"props":1446,"children":1448},{"id":1447},"_7-end-to-end-workflows",[1449],{"type":59,"value":1450},"7. End-to-end workflows",{"type":53,"tag":1452,"props":1453,"children":1455},"h3",{"id":1454},"_7a-fresh-one-shot-run-rayjob",[1456],{"type":59,"value":1457},"7a. Fresh one-shot run (rayjob)",{"type":53,"tag":392,"props":1459,"children":1461},{"className":1314,"code":1460,"language":1316,"meta":400,"style":400},"# From the NeMo-RL repo root:\nnrl-k8s check \u003Crecipe> --infra \u003Cinfra>                               # validate first\nnrl-k8s run \u003Crecipe> --infra \u003Cinfra> --rayjob --dry-run              # render RayJob manifest\nnrl-k8s run \u003Crecipe> --infra \u003Cinfra> --rayjob --no-wait              # apply, returns fast\n",[1462],{"type":53,"tag":68,"props":1463,"children":1464},{"__ignoreMap":400},[1465,1474,1532,1595],{"type":53,"tag":974,"props":1466,"children":1467},{"class":976,"line":977},[1468],{"type":53,"tag":974,"props":1469,"children":1471},{"style":1470},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1472],{"type":59,"value":1473},"# From the NeMo-RL repo root:\n",{"type":53,"tag":974,"props":1475,"children":1476},{"class":976,"line":999},[1477,1481,1486,1490,1495,1500,1504,1509,1513,1518,1523,1527],{"type":53,"tag":974,"props":1478,"children":1479},{"style":1326},[1480],{"type":59,"value":73},{"type":53,"tag":974,"props":1482,"children":1483},{"style":1003},[1484],{"type":59,"value":1485}," check",{"type":53,"tag":974,"props":1487,"children":1488},{"style":987},[1489],{"type":59,"value":1343},{"type":53,"tag":974,"props":1491,"children":1492},{"style":1003},[1493],{"type":59,"value":1494},"recip",{"type":53,"tag":974,"props":1496,"children":1497},{"style":1351},[1498],{"type":59,"value":1499},"e",{"type":53,"tag":974,"props":1501,"children":1502},{"style":987},[1503],{"type":59,"value":1394},{"type":53,"tag":974,"props":1505,"children":1506},{"style":1003},[1507],{"type":59,"value":1508}," --infra",{"type":53,"tag":974,"props":1510,"children":1511},{"style":987},[1512],{"type":59,"value":1343},{"type":53,"tag":974,"props":1514,"children":1515},{"style":1003},[1516],{"type":59,"value":1517},"infr",{"type":53,"tag":974,"props":1519,"children":1520},{"style":1351},[1521],{"type":59,"value":1522},"a",{"type":53,"tag":974,"props":1524,"children":1525},{"style":987},[1526],{"type":59,"value":1394},{"type":53,"tag":974,"props":1528,"children":1529},{"style":1470},[1530],{"type":59,"value":1531},"                               # validate first\n",{"type":53,"tag":974,"props":1533,"children":1534},{"class":976,"line":1009},[1535,1539,1544,1548,1552,1556,1560,1564,1568,1572,1576,1580,1585,1590],{"type":53,"tag":974,"props":1536,"children":1537},{"style":1326},[1538],{"type":59,"value":73},{"type":53,"tag":974,"props":1540,"children":1541},{"style":1003},[1542],{"type":59,"value":1543}," run",{"type":53,"tag":974,"props":1545,"children":1546},{"style":987},[1547],{"type":59,"value":1343},{"type":53,"tag":974,"props":1549,"children":1550},{"style":1003},[1551],{"type":59,"value":1494},{"type":53,"tag":974,"props":1553,"children":1554},{"style":1351},[1555],{"type":59,"value":1499},{"type":53,"tag":974,"props":1557,"children":1558},{"style":987},[1559],{"type":59,"value":1394},{"type":53,"tag":974,"props":1561,"children":1562},{"style":1003},[1563],{"type":59,"value":1508},{"type":53,"tag":974,"props":1565,"children":1566},{"style":987},[1567],{"type":59,"value":1343},{"type":53,"tag":974,"props":1569,"children":1570},{"style":1003},[1571],{"type":59,"value":1517},{"type":53,"tag":974,"props":1573,"children":1574},{"style":1351},[1575],{"type":59,"value":1522},{"type":53,"tag":974,"props":1577,"children":1578},{"style":987},[1579],{"type":59,"value":1394},{"type":53,"tag":974,"props":1581,"children":1582},{"style":1003},[1583],{"type":59,"value":1584}," --rayjob",{"type":53,"tag":974,"props":1586,"children":1587},{"style":1003},[1588],{"type":59,"value":1589}," --dry-run",{"type":53,"tag":974,"props":1591,"children":1592},{"style":1470},[1593],{"type":59,"value":1594},"              # render RayJob manifest\n",{"type":53,"tag":974,"props":1596,"children":1597},{"class":976,"line":1018},[1598,1602,1606,1610,1614,1618,1622,1626,1630,1634,1638,1642,1646,1651],{"type":53,"tag":974,"props":1599,"children":1600},{"style":1326},[1601],{"type":59,"value":73},{"type":53,"tag":974,"props":1603,"children":1604},{"style":1003},[1605],{"type":59,"value":1543},{"type":53,"tag":974,"props":1607,"children":1608},{"style":987},[1609],{"type":59,"value":1343},{"type":53,"tag":974,"props":1611,"children":1612},{"style":1003},[1613],{"type":59,"value":1494},{"type":53,"tag":974,"props":1615,"children":1616},{"style":1351},[1617],{"type":59,"value":1499},{"type":53,"tag":974,"props":1619,"children":1620},{"style":987},[1621],{"type":59,"value":1394},{"type":53,"tag":974,"props":1623,"children":1624},{"style":1003},[1625],{"type":59,"value":1508},{"type":53,"tag":974,"props":1627,"children":1628},{"style":987},[1629],{"type":59,"value":1343},{"type":53,"tag":974,"props":1631,"children":1632},{"style":1003},[1633],{"type":59,"value":1517},{"type":53,"tag":974,"props":1635,"children":1636},{"style":1351},[1637],{"type":59,"value":1522},{"type":53,"tag":974,"props":1639,"children":1640},{"style":987},[1641],{"type":59,"value":1394},{"type":53,"tag":974,"props":1643,"children":1644},{"style":1003},[1645],{"type":59,"value":1584},{"type":53,"tag":974,"props":1647,"children":1648},{"style":1003},[1649],{"type":59,"value":1650}," --no-wait",{"type":53,"tag":974,"props":1652,"children":1653},{"style":1470},[1654],{"type":59,"value":1655},"              # apply, returns fast\n",{"type":53,"tag":62,"props":1657,"children":1658},{},[1659],{"type":59,"value":1660},"Watch status + teardown (works even after your laptop disconnects because KubeRay owns the lifecycle):",{"type":53,"tag":392,"props":1662,"children":1664},{"className":1314,"code":1663,"language":1316,"meta":400,"style":400},"kubectl get rayjob -n default \u003Cname> -w\nkubectl get raycluster -n default                                    # empty = teardown succeeded\n",[1665],{"type":53,"tag":68,"props":1666,"children":1667},{"__ignoreMap":400},[1668,1716],{"type":53,"tag":974,"props":1669,"children":1670},{"class":976,"line":977},[1671,1675,1679,1684,1689,1694,1698,1703,1707,1711],{"type":53,"tag":974,"props":1672,"children":1673},{"style":1326},[1674],{"type":59,"value":89},{"type":53,"tag":974,"props":1676,"children":1677},{"style":1003},[1678],{"type":59,"value":1333},{"type":53,"tag":974,"props":1680,"children":1681},{"style":1003},[1682],{"type":59,"value":1683}," rayjob",{"type":53,"tag":974,"props":1685,"children":1686},{"style":1003},[1687],{"type":59,"value":1688}," -n",{"type":53,"tag":974,"props":1690,"children":1691},{"style":1003},[1692],{"type":59,"value":1693}," default",{"type":53,"tag":974,"props":1695,"children":1696},{"style":987},[1697],{"type":59,"value":1343},{"type":53,"tag":974,"props":1699,"children":1700},{"style":1003},[1701],{"type":59,"value":1702},"nam",{"type":53,"tag":974,"props":1704,"children":1705},{"style":1351},[1706],{"type":59,"value":1499},{"type":53,"tag":974,"props":1708,"children":1709},{"style":987},[1710],{"type":59,"value":1394},{"type":53,"tag":974,"props":1712,"children":1713},{"style":1003},[1714],{"type":59,"value":1715}," -w\n",{"type":53,"tag":974,"props":1717,"children":1718},{"class":976,"line":999},[1719,1723,1727,1732,1736,1740],{"type":53,"tag":974,"props":1720,"children":1721},{"style":1326},[1722],{"type":59,"value":89},{"type":53,"tag":974,"props":1724,"children":1725},{"style":1003},[1726],{"type":59,"value":1333},{"type":53,"tag":974,"props":1728,"children":1729},{"style":1003},[1730],{"type":59,"value":1731}," raycluster",{"type":53,"tag":974,"props":1733,"children":1734},{"style":1003},[1735],{"type":59,"value":1688},{"type":53,"tag":974,"props":1737,"children":1738},{"style":1003},[1739],{"type":59,"value":1693},{"type":53,"tag":974,"props":1741,"children":1742},{"style":1470},[1743],{"type":59,"value":1744},"                                    # empty = teardown succeeded\n",{"type":53,"tag":1452,"props":1746,"children":1748},{"id":1747},"_7b-dev-loop-long-lived",[1749],{"type":59,"value":1750},"7b. Dev loop (long-lived)",{"type":53,"tag":392,"props":1752,"children":1754},{"className":1314,"code":1753,"language":1316,"meta":400,"style":400},"nrl-k8s run \u003Crecipe> --infra \u003Cinfra> --run-id $(date +%Y%m%d-%H%M%S)\n# Edits in the recipe? Just re-run — reuses the live cluster.\n# Pod spec changed? Add --recreate to delete + re-apply.\n# Disagg recipe with gym\u002Fgen already healthy? --skip-daemons.\n",[1755],{"type":53,"tag":68,"props":1756,"children":1757},{"__ignoreMap":400},[1758,1830,1838,1846],{"type":53,"tag":974,"props":1759,"children":1760},{"class":976,"line":977},[1761,1765,1769,1773,1777,1781,1785,1789,1793,1797,1801,1805,1810,1815,1820,1825],{"type":53,"tag":974,"props":1762,"children":1763},{"style":1326},[1764],{"type":59,"value":73},{"type":53,"tag":974,"props":1766,"children":1767},{"style":1003},[1768],{"type":59,"value":1543},{"type":53,"tag":974,"props":1770,"children":1771},{"style":987},[1772],{"type":59,"value":1343},{"type":53,"tag":974,"props":1774,"children":1775},{"style":1003},[1776],{"type":59,"value":1494},{"type":53,"tag":974,"props":1778,"children":1779},{"style":1351},[1780],{"type":59,"value":1499},{"type":53,"tag":974,"props":1782,"children":1783},{"style":987},[1784],{"type":59,"value":1394},{"type":53,"tag":974,"props":1786,"children":1787},{"style":1003},[1788],{"type":59,"value":1508},{"type":53,"tag":974,"props":1790,"children":1791},{"style":987},[1792],{"type":59,"value":1343},{"type":53,"tag":974,"props":1794,"children":1795},{"style":1003},[1796],{"type":59,"value":1517},{"type":53,"tag":974,"props":1798,"children":1799},{"style":1351},[1800],{"type":59,"value":1522},{"type":53,"tag":974,"props":1802,"children":1803},{"style":987},[1804],{"type":59,"value":1394},{"type":53,"tag":974,"props":1806,"children":1807},{"style":1003},[1808],{"type":59,"value":1809}," --run-id",{"type":53,"tag":974,"props":1811,"children":1812},{"style":987},[1813],{"type":59,"value":1814}," $(",{"type":53,"tag":974,"props":1816,"children":1817},{"style":1326},[1818],{"type":59,"value":1819},"date",{"type":53,"tag":974,"props":1821,"children":1822},{"style":1003},[1823],{"type":59,"value":1824}," +%Y%m%d-%H%M%S",{"type":53,"tag":974,"props":1826,"children":1827},{"style":987},[1828],{"type":59,"value":1829},")\n",{"type":53,"tag":974,"props":1831,"children":1832},{"class":976,"line":999},[1833],{"type":53,"tag":974,"props":1834,"children":1835},{"style":1470},[1836],{"type":59,"value":1837},"# Edits in the recipe? Just re-run — reuses the live cluster.\n",{"type":53,"tag":974,"props":1839,"children":1840},{"class":976,"line":1009},[1841],{"type":53,"tag":974,"props":1842,"children":1843},{"style":1470},[1844],{"type":59,"value":1845},"# Pod spec changed? Add --recreate to delete + re-apply.\n",{"type":53,"tag":974,"props":1847,"children":1848},{"class":976,"line":1018},[1849],{"type":53,"tag":974,"props":1850,"children":1851},{"style":1470},[1852],{"type":59,"value":1853},"# Disagg recipe with gym\u002Fgen already healthy? --skip-daemons.\n",{"type":53,"tag":1452,"props":1855,"children":1857},{"id":1856},"_7c-first-time-disaggregated-bring-up",[1858],{"type":59,"value":1859},"7c. First-time disaggregated bring-up",{"type":53,"tag":392,"props":1861,"children":1863},{"className":1314,"code":1862,"language":1316,"meta":400,"style":400},"nrl-k8s run \u003Crecipe> --infra \u003Cdisagg-infra> --mode batch --code-source image\n",[1864],{"type":53,"tag":68,"props":1865,"children":1866},{"__ignoreMap":400},[1867],{"type":53,"tag":974,"props":1868,"children":1869},{"class":976,"line":977},[1870,1874,1878,1882,1886,1890,1894,1898,1902,1907,1911,1915,1920,1925,1930],{"type":53,"tag":974,"props":1871,"children":1872},{"style":1326},[1873],{"type":59,"value":73},{"type":53,"tag":974,"props":1875,"children":1876},{"style":1003},[1877],{"type":59,"value":1543},{"type":53,"tag":974,"props":1879,"children":1880},{"style":987},[1881],{"type":59,"value":1343},{"type":53,"tag":974,"props":1883,"children":1884},{"style":1003},[1885],{"type":59,"value":1494},{"type":53,"tag":974,"props":1887,"children":1888},{"style":1351},[1889],{"type":59,"value":1499},{"type":53,"tag":974,"props":1891,"children":1892},{"style":987},[1893],{"type":59,"value":1394},{"type":53,"tag":974,"props":1895,"children":1896},{"style":1003},[1897],{"type":59,"value":1508},{"type":53,"tag":974,"props":1899,"children":1900},{"style":987},[1901],{"type":59,"value":1343},{"type":53,"tag":974,"props":1903,"children":1904},{"style":1003},[1905],{"type":59,"value":1906},"disagg-infr",{"type":53,"tag":974,"props":1908,"children":1909},{"style":1351},[1910],{"type":59,"value":1522},{"type":53,"tag":974,"props":1912,"children":1913},{"style":987},[1914],{"type":59,"value":1394},{"type":53,"tag":974,"props":1916,"children":1917},{"style":1003},[1918],{"type":59,"value":1919}," --mode",{"type":53,"tag":974,"props":1921,"children":1922},{"style":1003},[1923],{"type":59,"value":1924}," batch",{"type":53,"tag":974,"props":1926,"children":1927},{"style":1003},[1928],{"type":59,"value":1929}," --code-source",{"type":53,"tag":974,"props":1931,"children":1932},{"style":1003},[1933],{"type":59,"value":1934}," image\n",{"type":53,"tag":1452,"props":1936,"children":1938},{"id":1937},"_7d-cluster-only-lifecycle",[1939],{"type":59,"value":1940},"7d. Cluster-only lifecycle",{"type":53,"tag":392,"props":1942,"children":1944},{"className":1314,"code":1943,"language":1316,"meta":400,"style":400},"nrl-k8s cluster up   \u003Crecipe> --infra \u003Cinfra> --target kuberay.training --wait\nnrl-k8s cluster up   \u003Crecipe> --infra \u003Cinfra> --target kuberay.training --dry-run   # render manifest\nnrl-k8s cluster down \u003Crecipe> --infra \u003Cinfra> --target kuberay.training --wait\nnrl-k8s cluster down \u003Crecipe> --infra \u003Cinfra>                                       # tear down all\nnrl-k8s cluster list -n default\nnrl-k8s cluster dashboard \u003Ccluster-name>                                  # port-forward + browser\n",[1945],{"type":53,"tag":68,"props":1946,"children":1947},{"__ignoreMap":400},[1948,2017,2085,2149,2205,2230],{"type":53,"tag":974,"props":1949,"children":1950},{"class":976,"line":977},[1951,1955,1960,1965,1970,1974,1978,1982,1986,1990,1994,1998,2002,2007,2012],{"type":53,"tag":974,"props":1952,"children":1953},{"style":1326},[1954],{"type":59,"value":73},{"type":53,"tag":974,"props":1956,"children":1957},{"style":1003},[1958],{"type":59,"value":1959}," cluster",{"type":53,"tag":974,"props":1961,"children":1962},{"style":1003},[1963],{"type":59,"value":1964}," up",{"type":53,"tag":974,"props":1966,"children":1967},{"style":987},[1968],{"type":59,"value":1969},"   \u003C",{"type":53,"tag":974,"props":1971,"children":1972},{"style":1003},[1973],{"type":59,"value":1494},{"type":53,"tag":974,"props":1975,"children":1976},{"style":1351},[1977],{"type":59,"value":1499},{"type":53,"tag":974,"props":1979,"children":1980},{"style":987},[1981],{"type":59,"value":1394},{"type":53,"tag":974,"props":1983,"children":1984},{"style":1003},[1985],{"type":59,"value":1508},{"type":53,"tag":974,"props":1987,"children":1988},{"style":987},[1989],{"type":59,"value":1343},{"type":53,"tag":974,"props":1991,"children":1992},{"style":1003},[1993],{"type":59,"value":1517},{"type":53,"tag":974,"props":1995,"children":1996},{"style":1351},[1997],{"type":59,"value":1522},{"type":53,"tag":974,"props":1999,"children":2000},{"style":987},[2001],{"type":59,"value":1394},{"type":53,"tag":974,"props":2003,"children":2004},{"style":1003},[2005],{"type":59,"value":2006}," --target",{"type":53,"tag":974,"props":2008,"children":2009},{"style":1003},[2010],{"type":59,"value":2011}," kuberay.training",{"type":53,"tag":974,"props":2013,"children":2014},{"style":1003},[2015],{"type":59,"value":2016}," --wait\n",{"type":53,"tag":974,"props":2018,"children":2019},{"class":976,"line":999},[2020,2024,2028,2032,2036,2040,2044,2048,2052,2056,2060,2064,2068,2072,2076,2080],{"type":53,"tag":974,"props":2021,"children":2022},{"style":1326},[2023],{"type":59,"value":73},{"type":53,"tag":974,"props":2025,"children":2026},{"style":1003},[2027],{"type":59,"value":1959},{"type":53,"tag":974,"props":2029,"children":2030},{"style":1003},[2031],{"type":59,"value":1964},{"type":53,"tag":974,"props":2033,"children":2034},{"style":987},[2035],{"type":59,"value":1969},{"type":53,"tag":974,"props":2037,"children":2038},{"style":1003},[2039],{"type":59,"value":1494},{"type":53,"tag":974,"props":2041,"children":2042},{"style":1351},[2043],{"type":59,"value":1499},{"type":53,"tag":974,"props":2045,"children":2046},{"style":987},[2047],{"type":59,"value":1394},{"type":53,"tag":974,"props":2049,"children":2050},{"style":1003},[2051],{"type":59,"value":1508},{"type":53,"tag":974,"props":2053,"children":2054},{"style":987},[2055],{"type":59,"value":1343},{"type":53,"tag":974,"props":2057,"children":2058},{"style":1003},[2059],{"type":59,"value":1517},{"type":53,"tag":974,"props":2061,"children":2062},{"style":1351},[2063],{"type":59,"value":1522},{"type":53,"tag":974,"props":2065,"children":2066},{"style":987},[2067],{"type":59,"value":1394},{"type":53,"tag":974,"props":2069,"children":2070},{"style":1003},[2071],{"type":59,"value":2006},{"type":53,"tag":974,"props":2073,"children":2074},{"style":1003},[2075],{"type":59,"value":2011},{"type":53,"tag":974,"props":2077,"children":2078},{"style":1003},[2079],{"type":59,"value":1589},{"type":53,"tag":974,"props":2081,"children":2082},{"style":1470},[2083],{"type":59,"value":2084},"   # render manifest\n",{"type":53,"tag":974,"props":2086,"children":2087},{"class":976,"line":1009},[2088,2092,2096,2101,2105,2109,2113,2117,2121,2125,2129,2133,2137,2141,2145],{"type":53,"tag":974,"props":2089,"children":2090},{"style":1326},[2091],{"type":59,"value":73},{"type":53,"tag":974,"props":2093,"children":2094},{"style":1003},[2095],{"type":59,"value":1959},{"type":53,"tag":974,"props":2097,"children":2098},{"style":1003},[2099],{"type":59,"value":2100}," down",{"type":53,"tag":974,"props":2102,"children":2103},{"style":987},[2104],{"type":59,"value":1343},{"type":53,"tag":974,"props":2106,"children":2107},{"style":1003},[2108],{"type":59,"value":1494},{"type":53,"tag":974,"props":2110,"children":2111},{"style":1351},[2112],{"type":59,"value":1499},{"type":53,"tag":974,"props":2114,"children":2115},{"style":987},[2116],{"type":59,"value":1394},{"type":53,"tag":974,"props":2118,"children":2119},{"style":1003},[2120],{"type":59,"value":1508},{"type":53,"tag":974,"props":2122,"children":2123},{"style":987},[2124],{"type":59,"value":1343},{"type":53,"tag":974,"props":2126,"children":2127},{"style":1003},[2128],{"type":59,"value":1517},{"type":53,"tag":974,"props":2130,"children":2131},{"style":1351},[2132],{"type":59,"value":1522},{"type":53,"tag":974,"props":2134,"children":2135},{"style":987},[2136],{"type":59,"value":1394},{"type":53,"tag":974,"props":2138,"children":2139},{"style":1003},[2140],{"type":59,"value":2006},{"type":53,"tag":974,"props":2142,"children":2143},{"style":1003},[2144],{"type":59,"value":2011},{"type":53,"tag":974,"props":2146,"children":2147},{"style":1003},[2148],{"type":59,"value":2016},{"type":53,"tag":974,"props":2150,"children":2151},{"class":976,"line":1018},[2152,2156,2160,2164,2168,2172,2176,2180,2184,2188,2192,2196,2200],{"type":53,"tag":974,"props":2153,"children":2154},{"style":1326},[2155],{"type":59,"value":73},{"type":53,"tag":974,"props":2157,"children":2158},{"style":1003},[2159],{"type":59,"value":1959},{"type":53,"tag":974,"props":2161,"children":2162},{"style":1003},[2163],{"type":59,"value":2100},{"type":53,"tag":974,"props":2165,"children":2166},{"style":987},[2167],{"type":59,"value":1343},{"type":53,"tag":974,"props":2169,"children":2170},{"style":1003},[2171],{"type":59,"value":1494},{"type":53,"tag":974,"props":2173,"children":2174},{"style":1351},[2175],{"type":59,"value":1499},{"type":53,"tag":974,"props":2177,"children":2178},{"style":987},[2179],{"type":59,"value":1394},{"type":53,"tag":974,"props":2181,"children":2182},{"style":1003},[2183],{"type":59,"value":1508},{"type":53,"tag":974,"props":2185,"children":2186},{"style":987},[2187],{"type":59,"value":1343},{"type":53,"tag":974,"props":2189,"children":2190},{"style":1003},[2191],{"type":59,"value":1517},{"type":53,"tag":974,"props":2193,"children":2194},{"style":1351},[2195],{"type":59,"value":1522},{"type":53,"tag":974,"props":2197,"children":2198},{"style":987},[2199],{"type":59,"value":1394},{"type":53,"tag":974,"props":2201,"children":2202},{"style":1470},[2203],{"type":59,"value":2204},"                                       # tear down all\n",{"type":53,"tag":974,"props":2206,"children":2207},{"class":976,"line":1027},[2208,2212,2216,2221,2225],{"type":53,"tag":974,"props":2209,"children":2210},{"style":1326},[2211],{"type":59,"value":73},{"type":53,"tag":974,"props":2213,"children":2214},{"style":1003},[2215],{"type":59,"value":1959},{"type":53,"tag":974,"props":2217,"children":2218},{"style":1003},[2219],{"type":59,"value":2220}," list",{"type":53,"tag":974,"props":2222,"children":2223},{"style":1003},[2224],{"type":59,"value":1688},{"type":53,"tag":974,"props":2226,"children":2227},{"style":1003},[2228],{"type":59,"value":2229}," default\n",{"type":53,"tag":974,"props":2231,"children":2232},{"class":976,"line":1036},[2233,2237,2241,2246,2250,2255,2259,2263],{"type":53,"tag":974,"props":2234,"children":2235},{"style":1326},[2236],{"type":59,"value":73},{"type":53,"tag":974,"props":2238,"children":2239},{"style":1003},[2240],{"type":59,"value":1959},{"type":53,"tag":974,"props":2242,"children":2243},{"style":1003},[2244],{"type":59,"value":2245}," dashboard",{"type":53,"tag":974,"props":2247,"children":2248},{"style":987},[2249],{"type":59,"value":1343},{"type":53,"tag":974,"props":2251,"children":2252},{"style":1003},[2253],{"type":59,"value":2254},"cluster-nam",{"type":53,"tag":974,"props":2256,"children":2257},{"style":1351},[2258],{"type":59,"value":1499},{"type":53,"tag":974,"props":2260,"children":2261},{"style":987},[2262],{"type":59,"value":1394},{"type":53,"tag":974,"props":2264,"children":2265},{"style":1470},[2266],{"type":59,"value":2267},"                                  # port-forward + browser\n",{"type":53,"tag":1452,"props":2269,"children":2271},{"id":2270},"_7e-deployments-eg-nemo-skills-sandbox",[2272],{"type":59,"value":2273},"7e. Deployments (e.g. nemo-skills sandbox)",{"type":53,"tag":392,"props":2275,"children":2277},{"className":1314,"code":2276,"language":1316,"meta":400,"style":400},"# Bring up just the deployment\nnrl-k8s cluster up \u003Crecipe> --infra \u003Cinfra> --target deployments.nemo_skills\n# Tear down just the deployment\nnrl-k8s cluster down \u003Crecipe> --infra \u003Cinfra> --target deployments.nemo_skills\n# Tear down everything (RayClusters + Deployments)\nnrl-k8s cluster down \u003Crecipe> --infra \u003Cinfra>\n",[2278],{"type":53,"tag":68,"props":2279,"children":2280},{"__ignoreMap":400},[2281,2289,2349,2357,2416,2424],{"type":53,"tag":974,"props":2282,"children":2283},{"class":976,"line":977},[2284],{"type":53,"tag":974,"props":2285,"children":2286},{"style":1470},[2287],{"type":59,"value":2288},"# Bring up just the deployment\n",{"type":53,"tag":974,"props":2290,"children":2291},{"class":976,"line":999},[2292,2296,2300,2304,2308,2312,2316,2320,2324,2328,2332,2336,2340,2344],{"type":53,"tag":974,"props":2293,"children":2294},{"style":1326},[2295],{"type":59,"value":73},{"type":53,"tag":974,"props":2297,"children":2298},{"style":1003},[2299],{"type":59,"value":1959},{"type":53,"tag":974,"props":2301,"children":2302},{"style":1003},[2303],{"type":59,"value":1964},{"type":53,"tag":974,"props":2305,"children":2306},{"style":987},[2307],{"type":59,"value":1343},{"type":53,"tag":974,"props":2309,"children":2310},{"style":1003},[2311],{"type":59,"value":1494},{"type":53,"tag":974,"props":2313,"children":2314},{"style":1351},[2315],{"type":59,"value":1499},{"type":53,"tag":974,"props":2317,"children":2318},{"style":987},[2319],{"type":59,"value":1394},{"type":53,"tag":974,"props":2321,"children":2322},{"style":1003},[2323],{"type":59,"value":1508},{"type":53,"tag":974,"props":2325,"children":2326},{"style":987},[2327],{"type":59,"value":1343},{"type":53,"tag":974,"props":2329,"children":2330},{"style":1003},[2331],{"type":59,"value":1517},{"type":53,"tag":974,"props":2333,"children":2334},{"style":1351},[2335],{"type":59,"value":1522},{"type":53,"tag":974,"props":2337,"children":2338},{"style":987},[2339],{"type":59,"value":1394},{"type":53,"tag":974,"props":2341,"children":2342},{"style":1003},[2343],{"type":59,"value":2006},{"type":53,"tag":974,"props":2345,"children":2346},{"style":1003},[2347],{"type":59,"value":2348}," deployments.nemo_skills\n",{"type":53,"tag":974,"props":2350,"children":2351},{"class":976,"line":1009},[2352],{"type":53,"tag":974,"props":2353,"children":2354},{"style":1470},[2355],{"type":59,"value":2356},"# Tear down just the deployment\n",{"type":53,"tag":974,"props":2358,"children":2359},{"class":976,"line":1018},[2360,2364,2368,2372,2376,2380,2384,2388,2392,2396,2400,2404,2408,2412],{"type":53,"tag":974,"props":2361,"children":2362},{"style":1326},[2363],{"type":59,"value":73},{"type":53,"tag":974,"props":2365,"children":2366},{"style":1003},[2367],{"type":59,"value":1959},{"type":53,"tag":974,"props":2369,"children":2370},{"style":1003},[2371],{"type":59,"value":2100},{"type":53,"tag":974,"props":2373,"children":2374},{"style":987},[2375],{"type":59,"value":1343},{"type":53,"tag":974,"props":2377,"children":2378},{"style":1003},[2379],{"type":59,"value":1494},{"type":53,"tag":974,"props":2381,"children":2382},{"style":1351},[2383],{"type":59,"value":1499},{"type":53,"tag":974,"props":2385,"children":2386},{"style":987},[2387],{"type":59,"value":1394},{"type":53,"tag":974,"props":2389,"children":2390},{"style":1003},[2391],{"type":59,"value":1508},{"type":53,"tag":974,"props":2393,"children":2394},{"style":987},[2395],{"type":59,"value":1343},{"type":53,"tag":974,"props":2397,"children":2398},{"style":1003},[2399],{"type":59,"value":1517},{"type":53,"tag":974,"props":2401,"children":2402},{"style":1351},[2403],{"type":59,"value":1522},{"type":53,"tag":974,"props":2405,"children":2406},{"style":987},[2407],{"type":59,"value":1394},{"type":53,"tag":974,"props":2409,"children":2410},{"style":1003},[2411],{"type":59,"value":2006},{"type":53,"tag":974,"props":2413,"children":2414},{"style":1003},[2415],{"type":59,"value":2348},{"type":53,"tag":974,"props":2417,"children":2418},{"class":976,"line":1027},[2419],{"type":53,"tag":974,"props":2420,"children":2421},{"style":1470},[2422],{"type":59,"value":2423},"# Tear down everything (RayClusters + Deployments)\n",{"type":53,"tag":974,"props":2425,"children":2426},{"class":976,"line":1036},[2427,2431,2435,2439,2443,2447,2451,2455,2459,2463,2467,2471],{"type":53,"tag":974,"props":2428,"children":2429},{"style":1326},[2430],{"type":59,"value":73},{"type":53,"tag":974,"props":2432,"children":2433},{"style":1003},[2434],{"type":59,"value":1959},{"type":53,"tag":974,"props":2436,"children":2437},{"style":1003},[2438],{"type":59,"value":2100},{"type":53,"tag":974,"props":2440,"children":2441},{"style":987},[2442],{"type":59,"value":1343},{"type":53,"tag":974,"props":2444,"children":2445},{"style":1003},[2446],{"type":59,"value":1494},{"type":53,"tag":974,"props":2448,"children":2449},{"style":1351},[2450],{"type":59,"value":1499},{"type":53,"tag":974,"props":2452,"children":2453},{"style":987},[2454],{"type":59,"value":1394},{"type":53,"tag":974,"props":2456,"children":2457},{"style":1003},[2458],{"type":59,"value":1508},{"type":53,"tag":974,"props":2460,"children":2461},{"style":987},[2462],{"type":59,"value":1343},{"type":53,"tag":974,"props":2464,"children":2465},{"style":1003},[2466],{"type":59,"value":1517},{"type":53,"tag":974,"props":2468,"children":2469},{"style":1351},[2470],{"type":59,"value":1522},{"type":53,"tag":974,"props":2472,"children":2473},{"style":987},[2474],{"type":59,"value":1359},{"type":53,"tag":62,"props":2476,"children":2477},{},[2478,2480,2485],{"type":59,"value":2479},"The ",{"type":53,"tag":68,"props":2481,"children":2483},{"className":2482},[],[2484],{"type":59,"value":480},{"type":59,"value":2486}," section in infra YAML declares Kubernetes Deployments managed alongside RayClusters. The CLI patches image, imagePullSecrets, and serviceAccountName from the top-level infra keys (same as RayClusters). Deployments start in parallel with cluster bring-up — no ordering dependency.",{"type":53,"tag":101,"props":2488,"children":2490},{"id":2489},"_8-monitoring-a-run",[2491],{"type":59,"value":2492},"8. Monitoring a run",{"type":53,"tag":392,"props":2494,"children":2496},{"className":1314,"code":2495,"language":1316,"meta":400,"style":400},"# Status\nnrl-k8s status \u003Crecipe> --infra \u003Cinfra>\nkubectl get rayjob,raycluster -n default\n\n# Follow the driver\nnrl-k8s job list \u003Crecipe> --infra \u003Cinfra> --role training\nnrl-k8s job logs \u003Crun-id> \u003Crecipe> --infra \u003Cinfra> --role training -f\n",[2497],{"type":53,"tag":68,"props":2498,"children":2499},{"__ignoreMap":400},[2500,2508,2556,2580,2589,2597,2659],{"type":53,"tag":974,"props":2501,"children":2502},{"class":976,"line":977},[2503],{"type":53,"tag":974,"props":2504,"children":2505},{"style":1470},[2506],{"type":59,"value":2507},"# Status\n",{"type":53,"tag":974,"props":2509,"children":2510},{"class":976,"line":999},[2511,2515,2520,2524,2528,2532,2536,2540,2544,2548,2552],{"type":53,"tag":974,"props":2512,"children":2513},{"style":1326},[2514],{"type":59,"value":73},{"type":53,"tag":974,"props":2516,"children":2517},{"style":1003},[2518],{"type":59,"value":2519}," status",{"type":53,"tag":974,"props":2521,"children":2522},{"style":987},[2523],{"type":59,"value":1343},{"type":53,"tag":974,"props":2525,"children":2526},{"style":1003},[2527],{"type":59,"value":1494},{"type":53,"tag":974,"props":2529,"children":2530},{"style":1351},[2531],{"type":59,"value":1499},{"type":53,"tag":974,"props":2533,"children":2534},{"style":987},[2535],{"type":59,"value":1394},{"type":53,"tag":974,"props":2537,"children":2538},{"style":1003},[2539],{"type":59,"value":1508},{"type":53,"tag":974,"props":2541,"children":2542},{"style":987},[2543],{"type":59,"value":1343},{"type":53,"tag":974,"props":2545,"children":2546},{"style":1003},[2547],{"type":59,"value":1517},{"type":53,"tag":974,"props":2549,"children":2550},{"style":1351},[2551],{"type":59,"value":1522},{"type":53,"tag":974,"props":2553,"children":2554},{"style":987},[2555],{"type":59,"value":1359},{"type":53,"tag":974,"props":2557,"children":2558},{"class":976,"line":1009},[2559,2563,2567,2572,2576],{"type":53,"tag":974,"props":2560,"children":2561},{"style":1326},[2562],{"type":59,"value":89},{"type":53,"tag":974,"props":2564,"children":2565},{"style":1003},[2566],{"type":59,"value":1333},{"type":53,"tag":974,"props":2568,"children":2569},{"style":1003},[2570],{"type":59,"value":2571}," rayjob,raycluster",{"type":53,"tag":974,"props":2573,"children":2574},{"style":1003},[2575],{"type":59,"value":1688},{"type":53,"tag":974,"props":2577,"children":2578},{"style":1003},[2579],{"type":59,"value":2229},{"type":53,"tag":974,"props":2581,"children":2582},{"class":976,"line":1018},[2583],{"type":53,"tag":974,"props":2584,"children":2586},{"emptyLinePlaceholder":2585},true,[2587],{"type":59,"value":2588},"\n",{"type":53,"tag":974,"props":2590,"children":2591},{"class":976,"line":1027},[2592],{"type":53,"tag":974,"props":2593,"children":2594},{"style":1470},[2595],{"type":59,"value":2596},"# Follow the driver\n",{"type":53,"tag":974,"props":2598,"children":2599},{"class":976,"line":1036},[2600,2604,2609,2613,2617,2621,2625,2629,2633,2637,2641,2645,2649,2654],{"type":53,"tag":974,"props":2601,"children":2602},{"style":1326},[2603],{"type":59,"value":73},{"type":53,"tag":974,"props":2605,"children":2606},{"style":1003},[2607],{"type":59,"value":2608}," job",{"type":53,"tag":974,"props":2610,"children":2611},{"style":1003},[2612],{"type":59,"value":2220},{"type":53,"tag":974,"props":2614,"children":2615},{"style":987},[2616],{"type":59,"value":1343},{"type":53,"tag":974,"props":2618,"children":2619},{"style":1003},[2620],{"type":59,"value":1494},{"type":53,"tag":974,"props":2622,"children":2623},{"style":1351},[2624],{"type":59,"value":1499},{"type":53,"tag":974,"props":2626,"children":2627},{"style":987},[2628],{"type":59,"value":1394},{"type":53,"tag":974,"props":2630,"children":2631},{"style":1003},[2632],{"type":59,"value":1508},{"type":53,"tag":974,"props":2634,"children":2635},{"style":987},[2636],{"type":59,"value":1343},{"type":53,"tag":974,"props":2638,"children":2639},{"style":1003},[2640],{"type":59,"value":1517},{"type":53,"tag":974,"props":2642,"children":2643},{"style":1351},[2644],{"type":59,"value":1522},{"type":53,"tag":974,"props":2646,"children":2647},{"style":987},[2648],{"type":59,"value":1394},{"type":53,"tag":974,"props":2650,"children":2651},{"style":1003},[2652],{"type":59,"value":2653}," --role",{"type":53,"tag":974,"props":2655,"children":2656},{"style":1003},[2657],{"type":59,"value":2658}," training\n",{"type":53,"tag":974,"props":2660,"children":2661},{"class":976,"line":1045},[2662,2666,2670,2675,2679,2684,2689,2693,2697,2701,2705,2709,2713,2717,2721,2725,2729,2733,2738],{"type":53,"tag":974,"props":2663,"children":2664},{"style":1326},[2665],{"type":59,"value":73},{"type":53,"tag":974,"props":2667,"children":2668},{"style":1003},[2669],{"type":59,"value":2608},{"type":53,"tag":974,"props":2671,"children":2672},{"style":1003},[2673],{"type":59,"value":2674}," logs",{"type":53,"tag":974,"props":2676,"children":2677},{"style":987},[2678],{"type":59,"value":1343},{"type":53,"tag":974,"props":2680,"children":2681},{"style":1003},[2682],{"type":59,"value":2683},"run-i",{"type":53,"tag":974,"props":2685,"children":2686},{"style":1351},[2687],{"type":59,"value":2688},"d",{"type":53,"tag":974,"props":2690,"children":2691},{"style":987},[2692],{"type":59,"value":1394},{"type":53,"tag":974,"props":2694,"children":2695},{"style":987},[2696],{"type":59,"value":1343},{"type":53,"tag":974,"props":2698,"children":2699},{"style":1003},[2700],{"type":59,"value":1494},{"type":53,"tag":974,"props":2702,"children":2703},{"style":1351},[2704],{"type":59,"value":1499},{"type":53,"tag":974,"props":2706,"children":2707},{"style":987},[2708],{"type":59,"value":1394},{"type":53,"tag":974,"props":2710,"children":2711},{"style":1003},[2712],{"type":59,"value":1508},{"type":53,"tag":974,"props":2714,"children":2715},{"style":987},[2716],{"type":59,"value":1343},{"type":53,"tag":974,"props":2718,"children":2719},{"style":1003},[2720],{"type":59,"value":1517},{"type":53,"tag":974,"props":2722,"children":2723},{"style":1351},[2724],{"type":59,"value":1522},{"type":53,"tag":974,"props":2726,"children":2727},{"style":987},[2728],{"type":59,"value":1394},{"type":53,"tag":974,"props":2730,"children":2731},{"style":1003},[2732],{"type":59,"value":2653},{"type":53,"tag":974,"props":2734,"children":2735},{"style":1003},[2736],{"type":59,"value":2737}," training",{"type":53,"tag":974,"props":2739,"children":2740},{"style":1003},[2741],{"type":59,"value":2742}," -f\n",{"type":53,"tag":62,"props":2744,"children":2745},{},[2746,2748,2754,2756,2761],{"type":59,"value":2747},"When the ",{"type":53,"tag":68,"props":2749,"children":2751},{"className":2750},[],[2752],{"type":59,"value":2753},"nrl-k8s job logs -f",{"type":59,"value":2755}," subprocess dies (",{"type":53,"tag":68,"props":2757,"children":2759},{"className":2758},[],[2760],{"type":59,"value":587},{"type":59,"value":2762}," i\u002Fo timeout after ~15 min idle), just re-run it. The training job keeps going.",{"type":53,"tag":62,"props":2764,"children":2765},{},[2766],{"type":59,"value":2767},"To fetch driver logs for a terminal job (SUCCEEDED\u002FFAILED) or a RayJob via the dashboard API:",{"type":53,"tag":392,"props":2769,"children":2771},{"className":1314,"code":2770,"language":1316,"meta":400,"style":400},"RC=$(kubectl get rayjob -n default \u003Crayjob-name> -o jsonpath='{.status.rayClusterName}')\nkubectl port-forward -n default svc\u002F${RC}-head-svc 18266:8265 &\ncurl -s http:\u002F\u002Flocalhost:18266\u002Fapi\u002Fjobs\u002F                              # lists jobs, find submission_id\ncurl -s \"http:\u002F\u002Flocalhost:18266\u002Fapi\u002Fjobs\u002F\u003Csubmission_id>\u002Flogs\"        # full driver log\n",[2772],{"type":53,"tag":68,"props":2773,"children":2774},{"__ignoreMap":400},[2775,2853,2907,2930],{"type":53,"tag":974,"props":2776,"children":2777},{"class":976,"line":977},[2778,2783,2788,2792,2796,2800,2804,2808,2812,2817,2821,2825,2830,2835,2840,2845,2849],{"type":53,"tag":974,"props":2779,"children":2780},{"style":1351},[2781],{"type":59,"value":2782},"RC",{"type":53,"tag":974,"props":2784,"children":2785},{"style":987},[2786],{"type":59,"value":2787},"=$(",{"type":53,"tag":974,"props":2789,"children":2790},{"style":1326},[2791],{"type":59,"value":89},{"type":53,"tag":974,"props":2793,"children":2794},{"style":1003},[2795],{"type":59,"value":1333},{"type":53,"tag":974,"props":2797,"children":2798},{"style":1003},[2799],{"type":59,"value":1683},{"type":53,"tag":974,"props":2801,"children":2802},{"style":1003},[2803],{"type":59,"value":1688},{"type":53,"tag":974,"props":2805,"children":2806},{"style":1003},[2807],{"type":59,"value":1693},{"type":53,"tag":974,"props":2809,"children":2810},{"style":987},[2811],{"type":59,"value":1343},{"type":53,"tag":974,"props":2813,"children":2814},{"style":1003},[2815],{"type":59,"value":2816},"rayjob-nam",{"type":53,"tag":974,"props":2818,"children":2819},{"style":1351},[2820],{"type":59,"value":1499},{"type":53,"tag":974,"props":2822,"children":2823},{"style":987},[2824],{"type":59,"value":1394},{"type":53,"tag":974,"props":2826,"children":2827},{"style":1003},[2828],{"type":59,"value":2829}," -o",{"type":53,"tag":974,"props":2831,"children":2832},{"style":1003},[2833],{"type":59,"value":2834}," jsonpath=",{"type":53,"tag":974,"props":2836,"children":2837},{"style":987},[2838],{"type":59,"value":2839},"'",{"type":53,"tag":974,"props":2841,"children":2842},{"style":1003},[2843],{"type":59,"value":2844},"{.status.rayClusterName}",{"type":53,"tag":974,"props":2846,"children":2847},{"style":987},[2848],{"type":59,"value":2839},{"type":53,"tag":974,"props":2850,"children":2851},{"style":987},[2852],{"type":59,"value":1829},{"type":53,"tag":974,"props":2854,"children":2855},{"class":976,"line":999},[2856,2860,2865,2869,2873,2878,2883,2887,2892,2897,2902],{"type":53,"tag":974,"props":2857,"children":2858},{"style":1326},[2859],{"type":59,"value":89},{"type":53,"tag":974,"props":2861,"children":2862},{"style":1003},[2863],{"type":59,"value":2864}," port-forward",{"type":53,"tag":974,"props":2866,"children":2867},{"style":1003},[2868],{"type":59,"value":1688},{"type":53,"tag":974,"props":2870,"children":2871},{"style":1003},[2872],{"type":59,"value":1693},{"type":53,"tag":974,"props":2874,"children":2875},{"style":1003},[2876],{"type":59,"value":2877}," svc\u002F",{"type":53,"tag":974,"props":2879,"children":2880},{"style":987},[2881],{"type":59,"value":2882},"${",{"type":53,"tag":974,"props":2884,"children":2885},{"style":1351},[2886],{"type":59,"value":2782},{"type":53,"tag":974,"props":2888,"children":2889},{"style":987},[2890],{"type":59,"value":2891},"}",{"type":53,"tag":974,"props":2893,"children":2894},{"style":1003},[2895],{"type":59,"value":2896},"-head-svc",{"type":53,"tag":974,"props":2898,"children":2899},{"style":1003},[2900],{"type":59,"value":2901}," 18266:8265",{"type":53,"tag":974,"props":2903,"children":2904},{"style":987},[2905],{"type":59,"value":2906}," &\n",{"type":53,"tag":974,"props":2908,"children":2909},{"class":976,"line":1009},[2910,2915,2920,2925],{"type":53,"tag":974,"props":2911,"children":2912},{"style":1326},[2913],{"type":59,"value":2914},"curl",{"type":53,"tag":974,"props":2916,"children":2917},{"style":1003},[2918],{"type":59,"value":2919}," -s",{"type":53,"tag":974,"props":2921,"children":2922},{"style":1003},[2923],{"type":59,"value":2924}," http:\u002F\u002Flocalhost:18266\u002Fapi\u002Fjobs\u002F",{"type":53,"tag":974,"props":2926,"children":2927},{"style":1470},[2928],{"type":59,"value":2929},"                              # lists jobs, find submission_id\n",{"type":53,"tag":974,"props":2931,"children":2932},{"class":976,"line":1018},[2933,2937,2941,2946,2951,2956],{"type":53,"tag":974,"props":2934,"children":2935},{"style":1326},[2936],{"type":59,"value":2914},{"type":53,"tag":974,"props":2938,"children":2939},{"style":1003},[2940],{"type":59,"value":2919},{"type":53,"tag":974,"props":2942,"children":2943},{"style":987},[2944],{"type":59,"value":2945}," \"",{"type":53,"tag":974,"props":2947,"children":2948},{"style":1003},[2949],{"type":59,"value":2950},"http:\u002F\u002Flocalhost:18266\u002Fapi\u002Fjobs\u002F\u003Csubmission_id>\u002Flogs",{"type":53,"tag":974,"props":2952,"children":2953},{"style":987},[2954],{"type":59,"value":2955},"\"",{"type":53,"tag":974,"props":2957,"children":2958},{"style":1470},[2959],{"type":59,"value":2960},"        # full driver log\n",{"type":53,"tag":62,"props":2962,"children":2963},{},[2964,2969,2971,2977,2979,2985,2987,2993,2995,3000,3002,3008],{"type":53,"tag":68,"props":2965,"children":2967},{"className":2966},[],[2968],{"type":59,"value":626},{"type":59,"value":2970}," with ",{"type":53,"tag":68,"props":2972,"children":2974},{"className":2973},[],[2975],{"type":59,"value":2976},"submission_id=null",{"type":59,"value":2978}," means an exec-submitter run (no dashboard log endpoint — use ",{"type":53,"tag":68,"props":2980,"children":2982},{"className":2981},[],[2983],{"type":59,"value":2984},"nrl-k8s job logs",{"type":59,"value":2986}," instead). ",{"type":53,"tag":68,"props":2988,"children":2990},{"className":2989},[],[2991],{"type":59,"value":2992},"type=SUBMISSION",{"type":59,"value":2994}," has ",{"type":53,"tag":68,"props":2996,"children":2998},{"className":2997},[],[2999],{"type":59,"value":595},{"type":59,"value":3001}," set and ",{"type":53,"tag":68,"props":3003,"children":3005},{"className":3004},[],[3006],{"type":59,"value":3007},"\u002Fapi\u002Fjobs\u002F\u003Cid>\u002Flogs",{"type":59,"value":3009}," works.",{"type":53,"tag":62,"props":3011,"children":3012},{},[3013,3015,3021,3023,3029],{"type":59,"value":3014},"Wandb URL appears in the driver log on the first ",{"type":53,"tag":68,"props":3016,"children":3018},{"className":3017},[],[3019],{"type":59,"value":3020},"wandb.init",{"type":59,"value":3022}," call; grep ",{"type":53,"tag":68,"props":3024,"children":3026},{"className":3025},[],[3027],{"type":59,"value":3028},"grep -oE 'https:\u002F\u002Fwandb\\.ai\u002F[A-Za-z0-9_.\u002F-]+'",{"type":59,"value":449},{"type":53,"tag":101,"props":3031,"children":3033},{"id":3032},"_9-stopping-things",[3034],{"type":59,"value":3035},"9. Stopping things",{"type":53,"tag":125,"props":3037,"children":3038},{},[3039,3054],{"type":53,"tag":129,"props":3040,"children":3041},{},[3042],{"type":53,"tag":133,"props":3043,"children":3044},{},[3045,3050],{"type":53,"tag":137,"props":3046,"children":3047},{"align":139},[3048],{"type":59,"value":3049},"What to stop",{"type":53,"tag":137,"props":3051,"children":3052},{"align":139},[3053],{"type":59,"value":263},{"type":53,"tag":159,"props":3055,"children":3056},{},[3057,3074,3091,3108],{"type":53,"tag":133,"props":3058,"children":3059},{},[3060,3065],{"type":53,"tag":166,"props":3061,"children":3062},{"align":139},[3063],{"type":59,"value":3064},"One training run",{"type":53,"tag":166,"props":3066,"children":3067},{"align":139},[3068],{"type":53,"tag":68,"props":3069,"children":3071},{"className":3070},[],[3072],{"type":59,"value":3073},"nrl-k8s job stop \u003Crun-id> \u003Crecipe> --infra \u003Cinfra> --role training",{"type":53,"tag":133,"props":3075,"children":3076},{},[3077,3082],{"type":53,"tag":166,"props":3078,"children":3079},{"align":139},[3080],{"type":59,"value":3081},"All running Ray jobs on a cluster (+ submit new)",{"type":53,"tag":166,"props":3083,"children":3084},{"align":139},[3085],{"type":53,"tag":68,"props":3086,"children":3088},{"className":3087},[],[3089],{"type":59,"value":3090},"nrl-k8s run \u003Crecipe> --infra \u003Cinfra> --replace",{"type":53,"tag":133,"props":3092,"children":3093},{},[3094,3099],{"type":53,"tag":166,"props":3095,"children":3096},{"align":139},[3097],{"type":59,"value":3098},"A long-lived RayCluster",{"type":53,"tag":166,"props":3100,"children":3101},{"align":139},[3102],{"type":53,"tag":68,"props":3103,"children":3105},{"className":3104},[],[3106],{"type":59,"value":3107},"nrl-k8s cluster down \u003Crecipe> --infra \u003Cinfra> --target kuberay.training --wait",{"type":53,"tag":133,"props":3109,"children":3110},{},[3111,3116],{"type":53,"tag":166,"props":3112,"children":3113},{"align":139},[3114],{"type":59,"value":3115},"A RayJob (ephemeral)",{"type":53,"tag":166,"props":3117,"children":3118},{"align":139},[3119,3125,3127,3133],{"type":53,"tag":68,"props":3120,"children":3122},{"className":3121},[],[3123],{"type":59,"value":3124},"kubectl delete rayjob \u003Cname> -n default",{"type":59,"value":3126}," — only if ",{"type":53,"tag":68,"props":3128,"children":3130},{"className":3129},[],[3131],{"type":59,"value":3132},"shutdownAfterJobFinishes",{"type":59,"value":3134}," didn't fire",{"type":53,"tag":62,"props":3136,"children":3137},{},[3138,3140,3146],{"type":59,"value":3139},"Confirm before deleting shared infra. The cost of ",{"type":53,"tag":68,"props":3141,"children":3143},{"className":3142},[],[3144],{"type":59,"value":3145},"cluster down",{"type":59,"value":3147}," on someone else's cluster is high.",{"type":53,"tag":101,"props":3149,"children":3151},{"id":3150},"_10-verifying-rayjob-teardown",[3152],{"type":59,"value":3153},"10. Verifying RayJob teardown",{"type":53,"tag":62,"props":3155,"children":3156},{},[3157,3159,3165,3167,3173],{"type":59,"value":3158},"After a ",{"type":53,"tag":68,"props":3160,"children":3162},{"className":3161},[],[3163],{"type":59,"value":3164},"run --rayjob",{"type":59,"value":3166}," completes with ",{"type":53,"tag":68,"props":3168,"children":3170},{"className":3169},[],[3171],{"type":59,"value":3172},"--shutdown",{"type":59,"value":3174}," (default), KubeRay should delete the RayCluster:",{"type":53,"tag":392,"props":3176,"children":3178},{"className":1314,"code":3177,"language":1316,"meta":400,"style":400},"kubectl get rayjob   -n default \u003Crayjob-name>                        # jobDeploymentStatus = Complete\nkubectl get raycluster -n default | grep \u003Crayjob-name>               # no output = torn down\n",[3179],{"type":53,"tag":68,"props":3180,"children":3181},{"__ignoreMap":400},[3182,3227],{"type":53,"tag":974,"props":3183,"children":3184},{"class":976,"line":977},[3185,3189,3193,3197,3202,3206,3210,3214,3218,3222],{"type":53,"tag":974,"props":3186,"children":3187},{"style":1326},[3188],{"type":59,"value":89},{"type":53,"tag":974,"props":3190,"children":3191},{"style":1003},[3192],{"type":59,"value":1333},{"type":53,"tag":974,"props":3194,"children":3195},{"style":1003},[3196],{"type":59,"value":1683},{"type":53,"tag":974,"props":3198,"children":3199},{"style":1003},[3200],{"type":59,"value":3201},"   -n",{"type":53,"tag":974,"props":3203,"children":3204},{"style":1003},[3205],{"type":59,"value":1693},{"type":53,"tag":974,"props":3207,"children":3208},{"style":987},[3209],{"type":59,"value":1343},{"type":53,"tag":974,"props":3211,"children":3212},{"style":1003},[3213],{"type":59,"value":2816},{"type":53,"tag":974,"props":3215,"children":3216},{"style":1351},[3217],{"type":59,"value":1499},{"type":53,"tag":974,"props":3219,"children":3220},{"style":987},[3221],{"type":59,"value":1394},{"type":53,"tag":974,"props":3223,"children":3224},{"style":1470},[3225],{"type":59,"value":3226},"                        # jobDeploymentStatus = Complete\n",{"type":53,"tag":974,"props":3228,"children":3229},{"class":976,"line":999},[3230,3234,3238,3242,3246,3250,3255,3260,3264,3268,3272,3276],{"type":53,"tag":974,"props":3231,"children":3232},{"style":1326},[3233],{"type":59,"value":89},{"type":53,"tag":974,"props":3235,"children":3236},{"style":1003},[3237],{"type":59,"value":1333},{"type":53,"tag":974,"props":3239,"children":3240},{"style":1003},[3241],{"type":59,"value":1731},{"type":53,"tag":974,"props":3243,"children":3244},{"style":1003},[3245],{"type":59,"value":1688},{"type":53,"tag":974,"props":3247,"children":3248},{"style":1003},[3249],{"type":59,"value":1693},{"type":53,"tag":974,"props":3251,"children":3252},{"style":987},[3253],{"type":59,"value":3254}," |",{"type":53,"tag":974,"props":3256,"children":3257},{"style":1326},[3258],{"type":59,"value":3259}," grep",{"type":53,"tag":974,"props":3261,"children":3262},{"style":987},[3263],{"type":59,"value":1343},{"type":53,"tag":974,"props":3265,"children":3266},{"style":1003},[3267],{"type":59,"value":2816},{"type":53,"tag":974,"props":3269,"children":3270},{"style":1351},[3271],{"type":59,"value":1499},{"type":53,"tag":974,"props":3273,"children":3274},{"style":987},[3275],{"type":59,"value":1394},{"type":53,"tag":974,"props":3277,"children":3278},{"style":1470},[3279],{"type":59,"value":3280},"               # no output = torn down\n",{"type":53,"tag":62,"props":3282,"children":3283},{},[3284,3286,3292],{"type":59,"value":3285},"The RayJob object itself sticks around for ",{"type":53,"tag":68,"props":3287,"children":3289},{"className":3288},[],[3290],{"type":59,"value":3291},"--ttl",{"type":59,"value":3293}," seconds (default 3600s) so you can still fetch logs.",{"type":53,"tag":101,"props":3295,"children":3297},{"id":3296},"_11-common-gotchas",[3298],{"type":59,"value":3299},"11. Common gotchas",{"type":53,"tag":404,"props":3301,"children":3302},{},[3303,3329,3362,3388,3406,3416,3434,3468],{"type":53,"tag":408,"props":3304,"children":3305},{},[3306,3311,3313,3319,3321,3327],{"type":53,"tag":113,"props":3307,"children":3308},{},[3309],{"type":59,"value":3310},"OmegaConf interpolation",{"type":59,"value":3312}," eats ",{"type":53,"tag":68,"props":3314,"children":3316},{"className":3315},[],[3317],{"type":59,"value":3318},"${VAR}",{"type":59,"value":3320}," in recipe\u002Finfra YAML. Escape shell variables with ",{"type":53,"tag":68,"props":3322,"children":3324},{"className":3323},[],[3325],{"type":59,"value":3326},"\\${VAR}",{"type":59,"value":3328}," so OmegaConf passes them through to the pod shell verbatim.",{"type":53,"tag":408,"props":3330,"children":3331},{},[3332,3337,3339,3345,3346,3352,3354,3360],{"type":53,"tag":113,"props":3333,"children":3334},{},[3335],{"type":59,"value":3336},"Megatron optimizer configs",{"type":59,"value":3338}," don't carry ",{"type":53,"tag":68,"props":3340,"children":3342},{"className":3341},[],[3343],{"type":59,"value":3344},"foreach",{"type":59,"value":653},{"type":53,"tag":68,"props":3347,"children":3349},{"className":3348},[],[3350],{"type":59,"value":3351},"fused",{"type":59,"value":3353},". Overrides like ",{"type":53,"tag":68,"props":3355,"children":3357},{"className":3356},[],[3358],{"type":59,"value":3359},"~policy.optimizer.kwargs.foreach ~policy.optimizer.kwargs.fused",{"type":59,"value":3361}," (valid for DTensor configs) break on Megatron recipes. Omit them for Megatron.",{"type":53,"tag":408,"props":3363,"children":3364},{},[3365,3370,3372,3378,3380,3386],{"type":53,"tag":113,"props":3366,"children":3367},{},[3368],{"type":59,"value":3369},"DTensor vs Megatron",{"type":59,"value":3371}," — MoE recipes typically use ",{"type":53,"tag":68,"props":3373,"children":3375},{"className":3374},[],[3376],{"type":59,"value":3377},"megatron_cfg.enabled=true",{"type":59,"value":3379},"; ensure ",{"type":53,"tag":68,"props":3381,"children":3383},{"className":3382},[],[3384],{"type":59,"value":3385},"dtensor_cfg.enabled=false",{"type":59,"value":3387}," in inherited defaults.",{"type":53,"tag":408,"props":3389,"children":3390},{},[3391,3396,3398,3404],{"type":53,"tag":113,"props":3392,"children":3393},{},[3394],{"type":59,"value":3395},"Shared filesystem vs git divergence",{"type":59,"value":3397}," — ",{"type":53,"tag":68,"props":3399,"children":3401},{"className":3400},[],[3402],{"type":59,"value":3403},"codeSource: image|lustre",{"type":59,"value":3405}," reads from the pod filesystem. If your local edits aren't on the shared filesystem the pods mount, the run is testing the on-disk version, not yours. Either sync via a helper pod (head pod exec is often blocked) or override via Hydra flags.",{"type":53,"tag":408,"props":3407,"children":3408},{},[3409,3414],{"type":53,"tag":113,"props":3410,"children":3411},{},[3412],{"type":59,"value":3413},"Ephemeral-storage + readinessProbe",{"type":59,"value":3415}," are injected by kuberay\u002FCDI webhooks at pod-apply time. Do NOT add them to the inline RayCluster spec.",{"type":53,"tag":408,"props":3417,"children":3418},{},[3419,3424,3426,3432],{"type":53,"tag":113,"props":3420,"children":3421},{},[3422],{"type":59,"value":3423},"Node taints",{"type":59,"value":3425}," vary per cluster. ",{"type":53,"tag":68,"props":3427,"children":3429},{"className":3428},[],[3430],{"type":59,"value":3431},"tolerations: [{operator: Exists}]",{"type":59,"value":3433}," on workers is defensive and worth keeping.",{"type":53,"tag":408,"props":3435,"children":3436},{},[3437,3442,3444,3450,3452,3458,3460,3466],{"type":53,"tag":113,"props":3438,"children":3439},{},[3440],{"type":59,"value":3441},"Dashboard blank page",{"type":59,"value":3443}," — Ray 2.52 installs dashboard assets as symlinks by default; ",{"type":53,"tag":68,"props":3445,"children":3447},{"className":3446},[],[3448],{"type":59,"value":3449},"nrl-k8s cluster dashboard \u003Cname>",{"type":59,"value":3451}," auto-reinstalls ",{"type":53,"tag":68,"props":3453,"children":3455},{"className":3454},[],[3456],{"type":59,"value":3457},"ray[default] --link-mode=copy",{"type":59,"value":3459}," to fix it. Bake ",{"type":53,"tag":68,"props":3461,"children":3463},{"className":3462},[],[3464],{"type":59,"value":3465},"ENV UV_LINK_MODE=copy",{"type":59,"value":3467}," in the image to avoid this entirely.",{"type":53,"tag":408,"props":3469,"children":3470},{},[3471,3481,3483,3489,3490,3496,3498,3503],{"type":53,"tag":113,"props":3472,"children":3473},{},[3474,3479],{"type":53,"tag":68,"props":3475,"children":3477},{"className":3476},[],[3478],{"type":59,"value":610},{"type":59,"value":3480}," is usually blocked",{"type":59,"value":3482}," in automation — route around with ",{"type":53,"tag":68,"props":3484,"children":3486},{"className":3485},[],[3487],{"type":59,"value":3488},"kubectl get ... -o yaml",{"type":59,"value":91},{"type":53,"tag":68,"props":3491,"children":3493},{"className":3492},[],[3494],{"type":59,"value":3495},"kubectl logs",{"type":59,"value":3497},", and ",{"type":53,"tag":68,"props":3499,"children":3501},{"className":3500},[],[3502],{"type":59,"value":587},{"type":59,"value":3504}," + Ray dashboard APIs.",{"type":53,"tag":101,"props":3506,"children":3508},{"id":3507},"_12-checklist-before-calling-a-run-done",[3509],{"type":59,"value":3510},"12. Checklist before calling a run \"done\"",{"type":53,"tag":62,"props":3512,"children":3513},{},[3514],{"type":59,"value":3515},"Before reporting a launch as successful, verify:",{"type":53,"tag":3517,"props":3518,"children":3519},"ol",{},[3520,3531,3564,3577,3590],{"type":53,"tag":408,"props":3521,"children":3522},{},[3523,3529],{"type":53,"tag":68,"props":3524,"children":3526},{"className":3525},[],[3527],{"type":59,"value":3528},"kubectl get rayjob\u002Fraycluster -n default",{"type":59,"value":3530}," shows the expected objects.",{"type":53,"tag":408,"props":3532,"children":3533},{},[3534,3540,3542,3548,3550,3556,3557,3563],{"type":53,"tag":68,"props":3535,"children":3537},{"className":3536},[],[3538],{"type":59,"value":3539},"nrl-k8s job list",{"type":59,"value":3541}," (or ",{"type":53,"tag":68,"props":3543,"children":3545},{"className":3544},[],[3546],{"type":59,"value":3547},"curl \u002Fapi\u002Fjobs\u002F",{"type":59,"value":3549},") shows the job in ",{"type":53,"tag":68,"props":3551,"children":3553},{"className":3552},[],[3554],{"type":59,"value":3555},"RUNNING",{"type":59,"value":653},{"type":53,"tag":68,"props":3558,"children":3560},{"className":3559},[],[3561],{"type":59,"value":3562},"SUCCEEDED",{"type":59,"value":449},{"type":53,"tag":408,"props":3565,"children":3566},{},[3567,3569,3575],{"type":59,"value":3568},"Driver log contains ",{"type":53,"tag":68,"props":3570,"children":3572},{"className":3571},[],[3573],{"type":59,"value":3574},"wandb.ai\u002F\u003Cproject>\u002Fruns\u002F\u003Cid>",{"type":59,"value":3576}," (if wandb is enabled) — share the URL with the user.",{"type":53,"tag":408,"props":3578,"children":3579},{},[3580,3582,3588],{"type":59,"value":3581},"At least one ",{"type":53,"tag":68,"props":3583,"children":3585},{"className":3584},[],[3586],{"type":59,"value":3587},"Processed prompts: 100%",{"type":59,"value":3589}," line appears (confirms generation is wired).",{"type":53,"tag":408,"props":3591,"children":3592},{},[3593,3595,3600,3602,3608,3610,3616],{"type":59,"value":3594},"For ",{"type":53,"tag":68,"props":3596,"children":3598},{"className":3597},[],[3599],{"type":59,"value":796},{"type":59,"value":3601}," mode only: after ",{"type":53,"tag":68,"props":3603,"children":3605},{"className":3604},[],[3606],{"type":59,"value":3607},"jobDeploymentStatus=Complete",{"type":59,"value":3609},", confirm ",{"type":53,"tag":68,"props":3611,"children":3613},{"className":3612},[],[3614],{"type":59,"value":3615},"kubectl get raycluster | grep \u003Cname>",{"type":59,"value":3617}," is empty (teardown worked).",{"type":53,"tag":101,"props":3619,"children":3621},{"id":3620},"_13-dev-pod",[3622],{"type":59,"value":3623},"13. Dev pod",{"type":53,"tag":62,"props":3625,"children":3626},{},[3627,3633,3635,3640,3642,3647],{"type":53,"tag":68,"props":3628,"children":3630},{"className":3629},[],[3631],{"type":59,"value":3632},"nrl-k8s dev",{"type":59,"value":3634}," manages a lightweight CPU pod on the cluster for code syncing, debugging, and running ",{"type":53,"tag":68,"props":3636,"children":3638},{"className":3637},[],[3639],{"type":59,"value":89},{"type":59,"value":3641},"\u002F",{"type":53,"tag":68,"props":3643,"children":3645},{"className":3644},[],[3646],{"type":59,"value":73},{"type":59,"value":3648}," from within the cluster.",{"type":53,"tag":392,"props":3650,"children":3652},{"className":1314,"code":3651,"language":1316,"meta":400,"style":400},"# One-time: set up secrets (HF token, wandb, SSH key, rclone)\nnrl-k8s dev setup-secrets --ssh-key ~\u002F.ssh\u002Fid_rsa --add-rclone\n\n# Create pod and exec in (idempotent — reuses existing pod)\nnrl-k8s dev connect\n\n# Switch image (must stop first — image change is warned but not auto-applied)\nnrl-k8s dev stop\nnrl-k8s dev connect --image nvcr.io\u002Fnvidian\u002Fnemo-rl:v0.7.0\n\n# Tear down\nnrl-k8s dev stop\n",[3653],{"type":53,"tag":68,"props":3654,"children":3655},{"__ignoreMap":400},[3656,3664,3696,3703,3711,3727,3734,3742,3758,3784,3792,3801],{"type":53,"tag":974,"props":3657,"children":3658},{"class":976,"line":977},[3659],{"type":53,"tag":974,"props":3660,"children":3661},{"style":1470},[3662],{"type":59,"value":3663},"# One-time: set up secrets (HF token, wandb, SSH key, rclone)\n",{"type":53,"tag":974,"props":3665,"children":3666},{"class":976,"line":999},[3667,3671,3676,3681,3686,3691],{"type":53,"tag":974,"props":3668,"children":3669},{"style":1326},[3670],{"type":59,"value":73},{"type":53,"tag":974,"props":3672,"children":3673},{"style":1003},[3674],{"type":59,"value":3675}," dev",{"type":53,"tag":974,"props":3677,"children":3678},{"style":1003},[3679],{"type":59,"value":3680}," setup-secrets",{"type":53,"tag":974,"props":3682,"children":3683},{"style":1003},[3684],{"type":59,"value":3685}," --ssh-key",{"type":53,"tag":974,"props":3687,"children":3688},{"style":1003},[3689],{"type":59,"value":3690}," ~\u002F.ssh\u002Fid_rsa",{"type":53,"tag":974,"props":3692,"children":3693},{"style":1003},[3694],{"type":59,"value":3695}," --add-rclone\n",{"type":53,"tag":974,"props":3697,"children":3698},{"class":976,"line":1009},[3699],{"type":53,"tag":974,"props":3700,"children":3701},{"emptyLinePlaceholder":2585},[3702],{"type":59,"value":2588},{"type":53,"tag":974,"props":3704,"children":3705},{"class":976,"line":1018},[3706],{"type":53,"tag":974,"props":3707,"children":3708},{"style":1470},[3709],{"type":59,"value":3710},"# Create pod and exec in (idempotent — reuses existing pod)\n",{"type":53,"tag":974,"props":3712,"children":3713},{"class":976,"line":1027},[3714,3718,3722],{"type":53,"tag":974,"props":3715,"children":3716},{"style":1326},[3717],{"type":59,"value":73},{"type":53,"tag":974,"props":3719,"children":3720},{"style":1003},[3721],{"type":59,"value":3675},{"type":53,"tag":974,"props":3723,"children":3724},{"style":1003},[3725],{"type":59,"value":3726}," connect\n",{"type":53,"tag":974,"props":3728,"children":3729},{"class":976,"line":1036},[3730],{"type":53,"tag":974,"props":3731,"children":3732},{"emptyLinePlaceholder":2585},[3733],{"type":59,"value":2588},{"type":53,"tag":974,"props":3735,"children":3736},{"class":976,"line":1045},[3737],{"type":53,"tag":974,"props":3738,"children":3739},{"style":1470},[3740],{"type":59,"value":3741},"# Switch image (must stop first — image change is warned but not auto-applied)\n",{"type":53,"tag":974,"props":3743,"children":3744},{"class":976,"line":1054},[3745,3749,3753],{"type":53,"tag":974,"props":3746,"children":3747},{"style":1326},[3748],{"type":59,"value":73},{"type":53,"tag":974,"props":3750,"children":3751},{"style":1003},[3752],{"type":59,"value":3675},{"type":53,"tag":974,"props":3754,"children":3755},{"style":1003},[3756],{"type":59,"value":3757}," stop\n",{"type":53,"tag":974,"props":3759,"children":3760},{"class":976,"line":1063},[3761,3765,3769,3774,3779],{"type":53,"tag":974,"props":3762,"children":3763},{"style":1326},[3764],{"type":59,"value":73},{"type":53,"tag":974,"props":3766,"children":3767},{"style":1003},[3768],{"type":59,"value":3675},{"type":53,"tag":974,"props":3770,"children":3771},{"style":1003},[3772],{"type":59,"value":3773}," connect",{"type":53,"tag":974,"props":3775,"children":3776},{"style":1003},[3777],{"type":59,"value":3778}," --image",{"type":53,"tag":974,"props":3780,"children":3781},{"style":1003},[3782],{"type":59,"value":3783}," nvcr.io\u002Fnvidian\u002Fnemo-rl:v0.7.0\n",{"type":53,"tag":974,"props":3785,"children":3787},{"class":976,"line":3786},10,[3788],{"type":53,"tag":974,"props":3789,"children":3790},{"emptyLinePlaceholder":2585},[3791],{"type":59,"value":2588},{"type":53,"tag":974,"props":3793,"children":3795},{"class":976,"line":3794},11,[3796],{"type":53,"tag":974,"props":3797,"children":3798},{"style":1470},[3799],{"type":59,"value":3800},"# Tear down\n",{"type":53,"tag":974,"props":3802,"children":3804},{"class":976,"line":3803},12,[3805,3809,3813],{"type":53,"tag":974,"props":3806,"children":3807},{"style":1326},[3808],{"type":59,"value":73},{"type":53,"tag":974,"props":3810,"children":3811},{"style":1003},[3812],{"type":59,"value":3675},{"type":53,"tag":974,"props":3814,"children":3815},{"style":1003},[3816],{"type":59,"value":3757},{"type":53,"tag":62,"props":3818,"children":3819},{},[3820],{"type":59,"value":3821},"The dev pod:",{"type":53,"tag":404,"props":3823,"children":3824},{},[3825,3830,3848,3884,3903],{"type":53,"tag":408,"props":3826,"children":3827},{},[3828],{"type":59,"value":3829},"Runs on a CPU-only node (anti-affinity to GPU nodes)",{"type":53,"tag":408,"props":3831,"children":3832},{},[3833,3835,3841,3843],{"type":59,"value":3834},"Mounts the shared ",{"type":53,"tag":68,"props":3836,"children":3838},{"className":3837},[],[3839],{"type":59,"value":3840},"rl-workspace",{"type":59,"value":3842}," PVC at ",{"type":53,"tag":68,"props":3844,"children":3846},{"className":3845},[],[3847],{"type":59,"value":1304},{"type":53,"tag":408,"props":3849,"children":3850},{},[3851,3853,3859,3861,3866,3868,3874,3876,3882],{"type":59,"value":3852},"Sets ",{"type":53,"tag":68,"props":3854,"children":3856},{"className":3855},[],[3857],{"type":59,"value":3858},"USER",{"type":59,"value":3860}," env var to the ",{"type":53,"tag":68,"props":3862,"children":3864},{"className":3863},[],[3865],{"type":59,"value":73},{"type":59,"value":3867}," username (so ",{"type":53,"tag":68,"props":3869,"children":3871},{"className":3870},[],[3872],{"type":59,"value":3873},"$USER",{"type":59,"value":3875}," and ",{"type":53,"tag":68,"props":3877,"children":3879},{"className":3878},[],[3880],{"type":59,"value":3881},"getpass.getuser()",{"type":59,"value":3883}," work correctly despite running as root)",{"type":53,"tag":408,"props":3885,"children":3886},{},[3887,3889,3894,3895,3901],{"type":59,"value":3888},"Installs ",{"type":53,"tag":68,"props":3890,"children":3892},{"className":3891},[],[3893],{"type":59,"value":89},{"type":59,"value":91},{"type":53,"tag":68,"props":3896,"children":3898},{"className":3897},[],[3899],{"type":59,"value":3900},"rclone",{"type":59,"value":3902}," (if configured) on first boot",{"type":53,"tag":408,"props":3904,"children":3905},{},[3906,3908,3914],{"type":59,"value":3907},"Injects SSH keys and tokens via ",{"type":53,"tag":68,"props":3909,"children":3911},{"className":3910},[],[3912],{"type":59,"value":3913},"envFrom",{"type":59,"value":3915}," on a per-user K8s Secret",{"type":53,"tag":62,"props":3917,"children":3918},{},[3919,3921,3927,3929,3935,3937,3942,3944,3950],{"type":59,"value":3920},"The pod's ",{"type":53,"tag":68,"props":3922,"children":3924},{"className":3923},[],[3925],{"type":59,"value":3926},"default",{"type":59,"value":3928}," service account needs an ",{"type":53,"tag":68,"props":3930,"children":3932},{"className":3931},[],[3933],{"type":59,"value":3934},"edit",{"type":59,"value":3936}," RoleBinding in the namespace for ",{"type":53,"tag":68,"props":3938,"children":3940},{"className":3939},[],[3941],{"type":59,"value":89},{"type":59,"value":3943}," to work inside. ",{"type":53,"tag":68,"props":3945,"children":3947},{"className":3946},[],[3948],{"type":59,"value":3949},"dev connect",{"type":59,"value":3951}," checks this and prints the required YAML if missing.",{"type":53,"tag":101,"props":3953,"children":3955},{"id":3954},"_14-where-things-live-in-the-repo",[3956],{"type":59,"value":3957},"14. Where things live in the repo",{"type":53,"tag":404,"props":3959,"children":3960},{},[3961,4022,4049,4060],{"type":53,"tag":408,"props":3962,"children":3963},{},[3964,3966,3972,3973,3979,3980,3986,3987,3993,3994,4000,4001,4007,4008,4014,4015,4021],{"type":59,"value":3965},"CLI code: ",{"type":53,"tag":68,"props":3967,"children":3969},{"className":3968},[],[3970],{"type":59,"value":3971},"infra\u002Fnrl_k8s\u002Fsrc\u002Fnrl_k8s\u002F",{"type":59,"value":1265},{"type":53,"tag":68,"props":3974,"children":3976},{"className":3975},[],[3977],{"type":59,"value":3978},"cli.py",{"type":59,"value":91},{"type":53,"tag":68,"props":3981,"children":3983},{"className":3982},[],[3984],{"type":59,"value":3985},"orchestrate.py",{"type":59,"value":91},{"type":53,"tag":68,"props":3988,"children":3990},{"className":3989},[],[3991],{"type":59,"value":3992},"manifest.py",{"type":59,"value":91},{"type":53,"tag":68,"props":3995,"children":3997},{"className":3996},[],[3998],{"type":59,"value":3999},"rayjob.py",{"type":59,"value":91},{"type":53,"tag":68,"props":4002,"children":4004},{"className":4003},[],[4005],{"type":59,"value":4006},"k8s.py",{"type":59,"value":91},{"type":53,"tag":68,"props":4009,"children":4011},{"className":4010},[],[4012],{"type":59,"value":4013},"submitters\u002F",{"type":59,"value":91},{"type":53,"tag":68,"props":4016,"children":4018},{"className":4017},[],[4019],{"type":59,"value":4020},"schema.py",{"type":59,"value":296},{"type":53,"tag":408,"props":4023,"children":4024},{},[4025,4027,4033,4035,4041,4043,4048],{"type":59,"value":4026},"Tests: ",{"type":53,"tag":68,"props":4028,"children":4030},{"className":4029},[],[4031],{"type":59,"value":4032},"infra\u002Fnrl_k8s\u002Ftests\u002Funit\u002F",{"type":59,"value":4034}," — run with ",{"type":53,"tag":68,"props":4036,"children":4038},{"className":4037},[],[4039],{"type":59,"value":4040},"uv run --extra test pytest -x -q",{"type":59,"value":4042}," from ",{"type":53,"tag":68,"props":4044,"children":4046},{"className":4045},[],[4047],{"type":59,"value":81},{"type":59,"value":449},{"type":53,"tag":408,"props":4050,"children":4051},{},[4052,4054,4059],{"type":59,"value":4053},"Recipe + infra examples: ",{"type":53,"tag":68,"props":4055,"children":4057},{"className":4056},[],[4058],{"type":59,"value":530},{"type":59,"value":449},{"type":53,"tag":408,"props":4061,"children":4062},{},[4063,4065,4071,4072,4078],{"type":59,"value":4064},"Base recipes this tool wraps: ",{"type":53,"tag":68,"props":4066,"children":4068},{"className":4067},[],[4069],{"type":59,"value":4070},"examples\u002Fconfigs\u002Frecipes\u002Fllm\u002F…",{"type":59,"value":3875},{"type":53,"tag":68,"props":4073,"children":4075},{"className":4074},[],[4076],{"type":59,"value":4077},"examples\u002Fnemo_gym\u002F…",{"type":59,"value":449},{"type":53,"tag":4080,"props":4081,"children":4082},"style",{},[4083],{"type":59,"value":4084},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":4086,"total":4190},[4087,4104,4118,4132,4144,4161,4176],{"slug":4088,"name":4088,"fn":4089,"description":4090,"org":4091,"tags":4092,"stars":26,"repoUrl":27,"updatedAt":4103},"accelerated-computing-cudf","accelerate data processing with cuDF","Official NVIDIA-authored guidance for NVIDIA cuDF GPU DataFrames, pandas acceleration, dask-cuDF, ETL, joins, groupby, CSV\u002FParquet I\u002FO, nullable semantics, and multi-GPU DataFrame workloads.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4093,4096,4099,4100],{"name":4094,"slug":4095,"type":15},"Data Analysis","data-analysis",{"name":4097,"slug":4098,"type":15},"Data Engineering","data-engineering",{"name":9,"slug":8,"type":15},{"name":4101,"slug":4102,"type":15},"Performance","performance","2026-07-14T05:28:43.176466",{"slug":4105,"name":4105,"fn":4106,"description":4107,"org":4108,"tags":4109,"stars":26,"repoUrl":27,"updatedAt":4117},"aiq-deploy","deploy and manage NVIDIA AI-Q infrastructure","Use when asked to install, deploy, run, validate, troubleshoot, or stop NVIDIA AI-Q Blueprint infrastructure.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4110,4113,4116],{"name":4111,"slug":4112,"type":15},"Deployment","deployment",{"name":4114,"slug":4115,"type":15},"Infrastructure","infrastructure",{"name":9,"slug":8,"type":15},"2026-07-14T05:29:06.667109",{"slug":4119,"name":4119,"fn":4120,"description":4121,"org":4122,"tags":4123,"stars":26,"repoUrl":27,"updatedAt":4131},"aiq-research","conduct deep research with AI-Q","Use when asked to run deep research or AI-Q research through a reachable NVIDIA AI-Q Blueprint backend.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4124,4127,4128],{"name":4125,"slug":4126,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":4129,"slug":4130,"type":15},"Research","research","2026-07-14T05:28:06.816956",{"slug":4133,"name":4133,"fn":4134,"description":4135,"org":4136,"tags":4137,"stars":26,"repoUrl":27,"updatedAt":4143},"amc-run-sample-calibration","run AMC sample dataset calibration","Run end-to-end calibration on the shipped sample dataset (sdg_08_2_sample_data_010926.zip) against a running AMC microservice. Use when user says 'test sample dataset', 'run sample calibration', 'verify AMC install', or 'launch and test'.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4138,4139,4140],{"name":4094,"slug":4095,"type":15},{"name":9,"slug":8,"type":15},{"name":4141,"slug":4142,"type":15},"Testing","testing","2026-07-17T05:29:03.913266",{"slug":4145,"name":4145,"fn":4146,"description":4147,"org":4148,"tags":4149,"stars":26,"repoUrl":27,"updatedAt":4160},"amc-run-video-calibration","calibrate video datasets with AutoMagicCalib","Calibrate a new dataset from pre-recorded video files via the AutoMagicCalib REST API. Use when user has local MP4s and says 'calibrate my videos', 'run AMC on these videos', or similar. For RTSP\u002Flive streams, use amc-run-rtsp-calibration instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4150,4153,4156,4157],{"name":4151,"slug":4152,"type":15},"Automation","automation",{"name":4154,"slug":4155,"type":15},"Imaging","imaging",{"name":9,"slug":8,"type":15},{"name":4158,"slug":4159,"type":15},"Video","video","2026-07-17T05:28:53.905004",{"slug":4162,"name":4162,"fn":4163,"description":4164,"org":4165,"tags":4166,"stars":26,"repoUrl":27,"updatedAt":4175},"amc-setup-calibration-stack","deploy AutoMagicCalib microservice with Docker","Launch AutoMagicCalib microservice and web UI from NGC release images via Docker Compose. Use when user says 'deploy auto calibration', 'launch auto calibration', 'launch AMC', 'start MS+UI', or 'set up auto-magic-calib'. Requires NGC API key.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4167,4168,4171,4172],{"name":4111,"slug":4112,"type":15},{"name":4169,"slug":4170,"type":15},"Docker","docker",{"name":9,"slug":8,"type":15},{"name":4173,"slug":4174,"type":15},"Operations","operations","2026-07-17T05:28:56.913999",{"slug":4177,"name":4177,"fn":4178,"description":4179,"org":4180,"tags":4181,"stars":26,"repoUrl":27,"updatedAt":4189},"cudaq-guide","develop quantum applications with CUDA-Q","CUDA-Q onboarding guide for installation, test programs, GPU simulation, QPU hardware, and quantum applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4182,4183,4186],{"name":9,"slug":8,"type":15},{"name":4184,"slug":4185,"type":15},"Quantum Computing","quantum-computing",{"name":4187,"slug":4188,"type":15},"Simulation","simulation","2026-07-14T05:26:58.898253",305,{"items":4192,"total":4339},[4193,4211,4227,4238,4250,4262,4275,4287,4298,4307,4321,4330],{"slug":4194,"name":4194,"fn":4195,"description":4196,"org":4197,"tags":4198,"stars":4208,"repoUrl":4209,"updatedAt":4210},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4199,4202,4205],{"name":4200,"slug":4201,"type":15},"Documentation","documentation",{"name":4203,"slug":4204,"type":15},"MCP","mcp",{"name":4206,"slug":4207,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":4212,"name":4212,"fn":4213,"description":4214,"org":4215,"tags":4216,"stars":4224,"repoUrl":4225,"updatedAt":4226},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4217,4220,4221],{"name":4218,"slug":4219,"type":15},"Containers","containers",{"name":4111,"slug":4112,"type":15},{"name":4222,"slug":4223,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":4228,"name":4228,"fn":4229,"description":4230,"org":4231,"tags":4232,"stars":4224,"repoUrl":4225,"updatedAt":4237},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4233,4236],{"name":4234,"slug":4235,"type":15},"CI\u002FCD","ci-cd",{"name":4111,"slug":4112,"type":15},"2026-07-14T05:25:59.97109",{"slug":4239,"name":4239,"fn":4240,"description":4241,"org":4242,"tags":4243,"stars":4224,"repoUrl":4225,"updatedAt":4249},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4244,4245,4246],{"name":4234,"slug":4235,"type":15},{"name":4111,"slug":4112,"type":15},{"name":4247,"slug":4248,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":4251,"name":4251,"fn":4252,"description":4253,"org":4254,"tags":4255,"stars":4224,"repoUrl":4225,"updatedAt":4261},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4256,4257,4258],{"name":24,"slug":25,"type":15},{"name":4247,"slug":4248,"type":15},{"name":4259,"slug":4260,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":4263,"name":4263,"fn":4264,"description":4265,"org":4266,"tags":4267,"stars":4224,"repoUrl":4225,"updatedAt":4274},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4268,4271],{"name":4269,"slug":4270,"type":15},"Best Practices","best-practices",{"name":4272,"slug":4273,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":4276,"name":4276,"fn":4277,"description":4278,"org":4279,"tags":4280,"stars":4224,"repoUrl":4225,"updatedAt":4286},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4281,4282,4285],{"name":13,"slug":14,"type":15},{"name":4283,"slug":4284,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":4288,"name":4288,"fn":4289,"description":4290,"org":4291,"tags":4292,"stars":4224,"repoUrl":4225,"updatedAt":4297},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4293,4296],{"name":4294,"slug":4295,"type":15},"QA","qa",{"name":4141,"slug":4142,"type":15},"2026-07-14T05:25:53.673039",{"slug":4299,"name":4299,"fn":4300,"description":4301,"org":4302,"tags":4303,"stars":4224,"repoUrl":4225,"updatedAt":4306},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4304,4305],{"name":4111,"slug":4112,"type":15},{"name":4114,"slug":4115,"type":15},"2026-07-14T05:25:49.362534",{"slug":4308,"name":4308,"fn":4309,"description":4310,"org":4311,"tags":4312,"stars":4224,"repoUrl":4225,"updatedAt":4320},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4313,4316,4317],{"name":4314,"slug":4315,"type":15},"Code Review","code-review",{"name":4247,"slug":4248,"type":15},{"name":4318,"slug":4319,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":4322,"name":4322,"fn":4323,"description":4324,"org":4325,"tags":4326,"stars":4224,"repoUrl":4225,"updatedAt":4329},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4327,4328],{"name":4294,"slug":4295,"type":15},{"name":4141,"slug":4142,"type":15},"2026-07-14T05:25:54.928983",{"slug":4331,"name":4331,"fn":4332,"description":4333,"org":4334,"tags":4335,"stars":4224,"repoUrl":4225,"updatedAt":4338},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4336,4337],{"name":4151,"slug":4152,"type":15},{"name":4234,"slug":4235,"type":15},"2026-07-30T05:29:03.275638",496]