[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-writing-sflow-yaml":3,"mdc-181az5-key":34,"related-repo-nvidia-writing-sflow-yaml":4067,"related-org-nvidia-writing-sflow-yaml":4108},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"writing-sflow-yaml","configure sflow YAML workflows","Write, create, and modify sflow YAML workflow configuration files. Covers schema structure, variables, task DAGs, backends (local, slurm, docker, kubernetes), operators, probes, replicas, artifacts, result parsing, storage\u002Fuploads (S3), hardware monitoring, and modular composition. Use when the user asks to create an sflow YAML, configure a workflow, set up inference serving on Slurm or Kubernetes, or asks about sflow YAML syntax.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Configuration","configuration","tag",{"name":17,"slug":18,"type":15},"YAML","yaml",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Workflow Automation","workflow-automation",36,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnv-sflow","2026-07-23T05:43:46.500651",null,6,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"A Python CLI workflow orchestrator with pluggable backends (e.g. local, Slurm) for running declarative YAML DAGs, collecting logs, and organizing outputs consistently.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnv-sflow\u002Ftree\u002FHEAD\u002Fsrc\u002Fsflow\u002Fskills\u002Fwriting-sflow-yaml","---\nname: writing-sflow-yaml\ndescription: >-\n  Write, create, and modify sflow YAML workflow configuration files. Covers schema structure,\n  variables, task DAGs, backends (local, slurm, docker, kubernetes), operators, probes,\n  replicas, artifacts, result parsing, storage\u002Fuploads (S3), hardware monitoring, and modular\n  composition.\n  Use when the user asks to create an sflow YAML, configure a workflow, set up inference\n  serving on Slurm or Kubernetes, or asks about sflow YAML syntax.\n---\n\n# Writing sflow YAML Configurations\n\nsflow turns **one portable recipe** — tasks, wiring, resources — into backend-native\nexecution on `local` \u002F `slurm` \u002F `docker` \u002F `kubernetes`. Write recipes **shape-first**:\nget the DAG right, *then* fill in scripts, scaling, resources, and backend specifics.\n\n**Follow the steps below in order.** The most common mistake is jumping straight to GPU\ncounts and backend flags before the workflow shape is correct. Don't. Design the DAG first;\neverything else hangs off it.\n\n> Deep field reference → [schema-reference.md](schema-reference.md). Runnable, annotated\n> recipes → [examples.md](examples.md) (cited per step). Full docs →\n> [nvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Fintro](https:\u002F\u002Fnvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Fintro).\n\n## The shared skeleton\n\nEvery recipe has the same top-level shape. You design `workflow.tasks` first; the rest is\noptional scaffolding you add only when a step calls for it.\n\n```yaml\nversion: \"0.1\"        # schema version (NOT the sflow release) — required in EVERY file\nvariables: { ... }    # optional — ${{ variables.X }} in YAML, ${X} in scripts\nartifacts: [ ... ]    # optional — named paths\u002FURIs, auto-mounted at the same path\nbackends:  [ ... ]    # optional — omit ⇒ a default `local` backend\noperators: [ ... ]    # optional — reusable launch presets\nstorage:   [ ... ]    # optional — S3 upload targets\nworkflow:\n  name: ...\n  tasks: [ ... ]      # THE DAG — design this first (Step 1)\n```\n\nOmit `backends`\u002F`operators` and you get `local` + `bash` — enough to prototype a DAG on a\nlaptop before touching a cluster.\n\n---\n\n## Step 1 — Shape the DAG (do this first)\n\nThe DAG *is* the recipe. Decide **what the tasks are and how they connect** before writing\nany script body.\n\n1. **List the tasks** — one node per logical step or service (`etcd`, `frontend`,\n   `prefill`, `decode`, `benchmark`, …).\n2. **Wire them** with `depends_on` (a task lists the upstreams it waits for). A dependent\n   starts once every upstream is **READY** (if that upstream has a readiness probe, Step 4)\n   or **COMPLETED** (if it has none).\n3. **Identify the terminal task** — the one whose completion ends the run (usually the\n   client\u002F`benchmark`). Long-running servers never \"complete\"; sflow stops them when the\n   terminal task finishes.\n4. **Reusable fragments wire in reverse** with `required_by:` — a module attaches itself to\n   a shared hub (e.g. `benchmark`) without editing the hub or adding cross-file `depends_on`.\n\nCanonical serving shape (Dynamo-style — see examples.md Example 3):\n\n```text\ninfra (nats, etcd) ──▶ frontend ──▶ workers (prefill + decode | agg) ──▶ benchmark ──▶ end\n```\n\nWrite the wiring skeleton first — **script bodies come in Step 2**:\n\n```yaml\nworkflow:\n  name: serve_and_bench\n  tasks:\n    - { name: etcd }\n    - { name: frontend,  depends_on: [etcd] }\n    - { name: worker,    depends_on: [frontend] }\n    - { name: benchmark, depends_on: [worker] }        # terminal task ⇒ run ends here\n```\n\nSanity-check the shape now: *does anything start before its inputs exist? is there exactly\none clear end? do \"parallel\" branches actually need to be concurrent?* Fixing shape is cheap\nhere and expensive later.\n\n## Step 2 — Write the task scripts\n\n- `script:` is a **list of lines joined into one bash script** (the `python` operator instead\n  runs the lines as Python *source* via `python -c`).\n- **Don't inline files or hard-code paths.** Any file a script needs — launch\u002Fhelper script,\n  config, model dir, dataset — belongs in an **artifact** (Step 3), referenced as\n  `${{ artifacts.X.path }}`. Never embed Python via heredocs or `python3 -c` — quoting breaks\n  when YAML → shell → Python nests.\n- **Reference values:** `${{ variables.X }}` in YAML (plan-time), `${X}` in scripts (runtime).\n  For arithmetic\u002Fcomparison, set the variable's `type` (`integer`, …) or it stays a string.\n\n```yaml\nworkflow:\n  tasks:\n    - name: worker\n      depends_on: [frontend]\n      script:\n        - python3 ${{ artifacts.LAUNCH.path }} ${{ variables.MODEL }}   # LAUNCH: see Step 3\n```\n\nSee examples.md Examples 1–2.\n\n## Step 3 — Handle paths with artifacts (don't hand-manage paths)\n\n**An artifact is a named path\u002FURI that sflow resolves, validates, and — inside containers —\nauto-mounts at the *same absolute path* on every backend.** This is the biggest lever for\nportability: declare a path once and you stop caring *where* it lives or *how* to mount it.\n`${{ artifacts.X.path }}` (YAML, plan-time) and `${X}` (script, runtime) always agree, sflow\n**auto-checks** the reference at preflight, and **auto-mounts** it — so you never hand-write\n`container_mounts`, `docker -v`, or k8s `volumes` for it (doing so is redundant — pitfall 13).\n\nPick the scheme by what the artifact holds:\n\n- **Small code snippets & config files → a `file:\u002F\u002F` artifact with inline `content:`.** sflow\n  writes the content and mounts it into the container at the same path on *any* backend —\n  local, docker, slurm, kubernetes — so **you don't manage where it lives or how it gets into\n  the container**. Use it for launch\u002Fhelper scripts, config JSON\u002FYAML, engine specs, etc.\n  **Never** inline them via heredocs \u002F `python3 -c` (YAML → shell → Python quoting breaks) or\n  bake them into the image.\n\n  ```yaml\n  artifacts:\n    - name: LAUNCH\n      uri: file:\u002F\u002Flaunch.py             # relative → resolves under SFLOW_WORKSPACE_DIR\n      content: |\n        import sys; print(\"serving\", sys.argv[1])\n    - name: GEN_CONFIG\n      uri: file:\u002F\u002Fgen_config.yaml\n      content: |\n        max_batch_size: 256\n        kv_cache_free_gpu_mem_fraction: 0.9\n  workflow:\n    tasks:\n      - name: worker\n        script:\n          - python3 ${{ artifacts.LAUNCH.path }} --config ${{ artifacts.GEN_CONFIG.path }}\n  ```\n\n- **Large \u002F pre-existing data (models, datasets, checkpoints) → an `fs:\u002F\u002F` artifact** pointing\n  at storage the compute nodes already see. It still resolves + mounts the same path\n  everywhere, but *you* supply the bytes:\n  - **slurm** → keep it on **shared storage** (Lustre\u002FGPFS\u002FNFS) so the path exists on every node.\n  - **kubernetes** → back it with a **PVC**: declare the PVC under backend `volumes:` at a\n    `mount_path`, then point the `fs:\u002F\u002F` artifact at a path *under* that mount. An `fs:\u002F\u002F` path\n    no PVC covers and the node lacks makes the pod fail (pitfall 9).\n\n**Rule of thumb: if a task touches a path, declare it as an artifact** — small text → `file:\u002F\u002F`\nwith `content:`, big data → `fs:\u002F\u002F` on shared storage \u002F a PVC. Artifact schemes & `content:`\nrules → schema-reference.md.\n\n## Step 4 — Gate the wiring with probes\n\n`depends_on` alone only waits for a process to *start*. For services, wait for **ready**.\n\n- **Every long-running server gets a `readiness` probe** so dependents wait for real up-ness:\n  `log_watch` (server logs a ready line), `tcp_port` (port open), or `http_get`\u002F`http_post`\n  (health endpoint).\n- **Add a `failure` probe** on servers — `failure.log_watch` for\n  `\"Traceback (most recent call last)\"` fails the run fast instead of hanging until timeout.\n- `log_watch` matches **literally** by default (parentheses\u002Fdots aren't special); prefix\n  `re:` \u002F `regex:` for real regex.\n- **Don't probe short-lived one-shot tasks** (benchmarks, setup steps).\n- Defaults: `timeout` 1200 s (readiness deadline only), `interval` 5, `each_check_timeout`\n  30, `success_threshold` 1, `failure_threshold` 3.\n\n```yaml\n- name: frontend\n  depends_on: [etcd]\n  script: [ \"python -m my.frontend --port 8000\" ]\n  probes:\n    readiness: { tcp_port: { port: 8000 } }\n    failure:   { log_watch: { regex_pattern: \"Traceback (most recent call last)\" } }\n```\n\n(On **kubernetes**, tcp\u002Fhttp probes run from an in-cluster probe pod automatically — the\ndriver host usually can't route to the pod network.) See the probes doc.\n\n## Step 5 — Scale with replicas & sweeps\n\n- **Replicate a task:** `replicas: { count: N, policy: parallel|sequential }`.\n  `parallel` = all at once (downstream waits for all); `sequential` = one-by-one (sweeps).\n- **Parameter sweep:** `replicas.variables: [X]` where `X` declares a `domain:` → one replica\n  per value (Cartesian product across multiple swept vars). Each replica gets\n  `SFLOW_REPLICA_INDEX` and the swept values in its env; omit `count` and it equals the sweep\n  size.\n\nSee examples.md Example 5; the replicas doc.\n\n## Step 6 — Place on nodes & GPUs\n\n- **Pin placement** with `resources.nodes` (`indices` \u002F `count` \u002F `exclude`; negative indices\n  count from the end). **Request GPUs** with `resources.gpus.count`.\n- On `local`\u002F`slurm`\u002F`docker`, sflow slices `CUDA_VISIBLE_DEVICES` per task\u002Freplica — set the\n  backend's `gpus_per_node` so it can pack.\n- **GPU planning:** `GPUs\u002Fworker = TP × DP × PP` (framework-dependent — some use attention-DP\n  \u002F expert parallelism), `total = GPUs\u002Fworker × replicas`, `nodes = ceil(total \u002F gpus_per_node)`.\n  When unsure of a framework's parallelism, ask the user.\n- Use `resources.*.release_after` (`task_ready` \u002F `task_completion` \u002F `workflow_completion`)\n  to let later tasks reuse a reservation.\n\nSee the resources doc.\n\n## Step 7 — Backend-specific handling\n\nPick a backend and apply *its* rules. Start `local`, move to a cluster once the DAG is proven.\n\n**`local`** — no config needed (default). `bash` operator. Set `nodes: N` to simulate a\nmulti-node placement on one machine.\n\n**`slurm`** — default operator `srun`. Backend needs `partition`, `account`, `time`\n(HH:MM:SS or int minutes), `nodes`, `gpus_per_node`. Containers: put `container_image` \u002F\n`container_name` on an `srun` operator (reuse the image across tasks by `container_name`).\nKeep models \u002F workspace \u002F **output dir** on **shared storage** (Lustre\u002FGPFS\u002FNFS) so an\n`fs:\u002F\u002F` artifact auto-mounts at the same path on every node.\n\n```yaml\nbackends:\n  - name: slurm\n    type: slurm\n    default: true\n    nodes: 1\n    partition: ${{ variables.PART }}      # block style: ${{ }} values can't sit in a { } flow map\n    account: ${{ variables.ACCT }}\n    time: \"01:00:00\"                       # HH:MM:SS or int minutes\n    gpus_per_node: 8\noperators:\n  - { name: ctr, type: srun, container_image: nvcr.io\u002Fnvidia\u002Fpytorch:24.07-py3,\n      container_writable: true, mpi: pmix }\n```\n\n**`docker`** — backend is `type: docker`, its launch operator is `type: docker_run` (mind\nthe naming). The `image` can live on the backend or the operator. Remote\u002Fmulti-host via a\n`hosts:` list.\n\n**`kubernetes`** — the workload **`image` lives on the `k8s` operator**, never the backend\n(the backend reserves + pins nodes with placeholder pods). Rules that bite:\n- Cluster access is **CLI-only** (`--kubeconfig` \u002F `--kube-context` \u002F `--kube-namespace`) so\n  recipes stay cluster-agnostic; `namespace` is a backend field (one namespace per backend).\n- `resources.gpus.count` is a per-task **total**, split evenly across the task's nodes → must\n  be a multiple of the node count. For an ordinary pod sflow leaves `CUDA_VISIBLE_DEVICES`\n  unset (the device-plugin\u002FDRA assigns the GPUs); when co-located GPU tasks are **merged into\n  one pod** (`merge_colocated_gpu_pods`, default `auto`) it sets a per-member\n  `CUDA_VISIBLE_DEVICES` to pin each task's slice.\n- A single-node task → one pod; a `resources.nodes` task → one pod per node (leader = index 0).\n- **Cluster data → a PVC**: declare it under backend `volumes:` at a `mount_path`, then point\n  an `fs:\u002F\u002F` artifact at a path *under* that mount. Output dirs are ephemeral `emptyDir` — for\n  persistence use a PVC or `uploads:` (Step 8).\n- **Any MPI workload** (`mpirun`-launched, e.g. TRT-LLM `trtllm-llmapi-launch`) → `type: k8s_mpi`,\n  and write the `mpirun -np N …` **explicitly** in the script: sflow reads it to set up the MPI\n  world for **both single- and multi-node** and injects the keypair\u002Fhostfile\u002Fsshd\u002F`-x` env\n  forwarding. Reserve plain `k8s` for non-MPI, single-pod workloads.\n\n```yaml\nbackends:\n  - { name: k8s, type: kubernetes, default: true, namespace: default, nodes: 1,\n      gpus_per_node: 8, scheduling: device_plugin }   # device_plugin default; dra is WIP\noperators:\n  - { name: gpu_op, type: k8s, image: nvcr.io\u002Fnvidia\u002Fpytorch:24.12-py3 }\n```\n\nSee the backends doc; examples.md Examples 8 (k8s), 10 (k8s + PVC), 11 (slurm + Lustre).\n\n## Step 8 — Capture outputs\n\n- **Metrics:** add `result:` (a regex map, `patterns:`, or a JSON `file:`) to parse a task's\n  log\u002Foutput into `${SFLOW_TASK_OUTPUT_DIR}\u002Fresult.json` plus a workflow `results.json`.\n  Prefer it over the legacy `outputs:`. A task isn't `COMPLETED` until parsing finishes.\n- **Ship files:** name `storage:` S3 targets, then per-task `uploads:` (fire on `COMPLETED`)\n  or `workflow.upload_all` (zip the whole run). Credentials come from the **boto3 default\n  chain** — never put secrets in YAML; needs the `sflow[s3]` extra.\n- **Telemetry:** `monitor:` samples GPU\u002FCPU\u002Fmem\u002Fnet on `local`\u002F`slurm`\u002F`docker` (not k8s yet),\n  or enable it without editing the recipe via `--enable-workflow-monitor` \u002F\n  `--enable-task-monitor`.\n\nSee examples.md Examples 12 (result parsing), 7 (monitor), 9 (uploads); the results \u002F uploads \u002F monitor docs.\n\n## Step 9 — Validate before running\n\nRun the one-command preflight before every real run (static schema + reference checks + GPU\nplan + `sflow --dry-run`, with a root-cause explanation on failure). **Exit 0 = ready.**\n\n```bash\npython scripts\u002Fpreflight.py my_workflow.yaml\npython scripts\u002Fpreflight.py slurm.yaml common.yaml sglang\u002Fagg.yaml --set TP_SIZE=8   # composed\n```\n\nStandalone steps when you want just one:\n\n```bash\npython scripts\u002Fvalidate_sflow_yaml.py my_workflow.yaml   # schema, refs, k8s volumes\npython scripts\u002Fcheck_gpu_plan.py     my_workflow.yaml    # GPU plan + oversubscription\nsflow run -f my_workflow.yaml --dry-run                  # full plan + expression resolution\n```\n\n---\n\n## Modular composition (when splitting across files)\n\nPass fragments directly: `sflow run -f slurm.yaml -f common.yaml -f sglang\u002Fagg.yaml`.\nFiles are combined with a **recursive deep merge keyed on `name`** — `version` must match;\n`variables` \u002F `artifacts` \u002F `backends` \u002F `operators` \u002F `storage` and same-name\n`workflow.tasks` are deep-merged (one definition can be scattered across files); leaf\nscalars\u002Flists are last-file-wins with a warning. Prefer `required_by:` on optional fragments\n(absent targets skip silently) over `--missable-tasks`. See examples.md Example 4.\n\n## Pitfalls checklist\n\n1. Missing `version: \"0.1\"` in **every** file (including fragments).\n2. `${{ task.X.nodes }}` in a YAML field — task context works only in scripts.\n3. Forgetting `depends_on` — tasks run immediately without it.\n4. GPU oversubscription: Σ GPU counts > `nodes × gpus_per_node`.\n5. Probing a short-lived one-shot task (don't probe benchmarks).\n6. Expecting `log_watch` to be regex — it's **literal** unless prefixed `re:` \u002F `regex:`.\n7. Docker: the operator is `type: docker_run` (the backend is `type: docker`).\n8. Putting the workload `image` on the **kubernetes backend** — it belongs on the `k8s`\u002F`k8s_mpi` **operator**.\n9. K8s: an `fs:\u002F\u002F` path no `volumes:` PVC covers and the node lacks → the pod fails (hostPath). Use a PVC.\n10. Expecting K8s task outputs to persist — the output dir is an ephemeral `emptyDir`; use a PVC or `uploads:`.\n11. Multi-node Slurm with models\u002Fworkspace\u002Foutput on **node-local** storage — same-path mounts break; use shared storage.\n12. S3 secrets in YAML (use the boto3 chain) or forgetting the `sflow[s3]` extra.\n13. Hand-writing mounts for a path already declared as an artifact — redundant; artifacts auto-mount same-path.\n\n## Additional resources\n\n- Field reference & merge rules → [schema-reference.md](schema-reference.md)\n- 12 annotated, runnable recipes → [examples.md](examples.md)\n- Full user docs → [nvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Fintro](https:\u002F\u002Fnvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Fintro)\n",{"data":35,"body":36},{"name":4,"description":6},{"type":37,"children":38},"root",[39,48,108,118,152,159,172,441,475,479,485,504,636,641,651,662,905,917,923,1043,1147,1152,1158,1240,1245,1622,1660,1666,1689,1861,2069,2080,2086,2171,2176,2182,2341,2346,2352,2371,2399,2506,2795,2840,2873,3103,3335,3340,3346,3519,3524,3530,3548,3620,3625,3708,3711,3717,3803,3809,4024,4030,4061],{"type":40,"tag":41,"props":42,"children":44},"element","h1",{"id":43},"writing-sflow-yaml-configurations",[45],{"type":46,"value":47},"text","Writing sflow YAML Configurations",{"type":40,"tag":49,"props":50,"children":51},"p",{},[52,54,60,62,69,71,77,78,84,85,91,93,98,100,106],{"type":46,"value":53},"sflow turns ",{"type":40,"tag":55,"props":56,"children":57},"strong",{},[58],{"type":46,"value":59},"one portable recipe",{"type":46,"value":61}," — tasks, wiring, resources — into backend-native\nexecution on ",{"type":40,"tag":63,"props":64,"children":66},"code",{"className":65},[],[67],{"type":46,"value":68},"local",{"type":46,"value":70}," \u002F ",{"type":40,"tag":63,"props":72,"children":74},{"className":73},[],[75],{"type":46,"value":76},"slurm",{"type":46,"value":70},{"type":40,"tag":63,"props":79,"children":81},{"className":80},[],[82],{"type":46,"value":83},"docker",{"type":46,"value":70},{"type":40,"tag":63,"props":86,"children":88},{"className":87},[],[89],{"type":46,"value":90},"kubernetes",{"type":46,"value":92},". Write recipes ",{"type":40,"tag":55,"props":94,"children":95},{},[96],{"type":46,"value":97},"shape-first",{"type":46,"value":99},":\nget the DAG right, ",{"type":40,"tag":101,"props":102,"children":103},"em",{},[104],{"type":46,"value":105},"then",{"type":46,"value":107}," fill in scripts, scaling, resources, and backend specifics.",{"type":40,"tag":49,"props":109,"children":110},{},[111,116],{"type":40,"tag":55,"props":112,"children":113},{},[114],{"type":46,"value":115},"Follow the steps below in order.",{"type":46,"value":117}," The most common mistake is jumping straight to GPU\ncounts and backend flags before the workflow shape is correct. Don't. Design the DAG first;\neverything else hangs off it.",{"type":40,"tag":119,"props":120,"children":121},"blockquote",{},[122],{"type":40,"tag":49,"props":123,"children":124},{},[125,127,133,135,140,142,150],{"type":46,"value":126},"Deep field reference → ",{"type":40,"tag":128,"props":129,"children":131},"a",{"href":130},"schema-reference.md",[132],{"type":46,"value":130},{"type":46,"value":134},". Runnable, annotated\nrecipes → ",{"type":40,"tag":128,"props":136,"children":138},{"href":137},"examples.md",[139],{"type":46,"value":137},{"type":46,"value":141}," (cited per step). Full docs →\n",{"type":40,"tag":128,"props":143,"children":147},{"href":144,"rel":145},"https:\u002F\u002Fnvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Fintro",[146],"nofollow",[148],{"type":46,"value":149},"nvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Fintro",{"type":46,"value":151},".",{"type":40,"tag":153,"props":154,"children":156},"h2",{"id":155},"the-shared-skeleton",[157],{"type":46,"value":158},"The shared skeleton",{"type":40,"tag":49,"props":160,"children":161},{},[162,164,170],{"type":46,"value":163},"Every recipe has the same top-level shape. You design ",{"type":40,"tag":63,"props":165,"children":167},{"className":166},[],[168],{"type":46,"value":169},"workflow.tasks",{"type":46,"value":171}," first; the rest is\noptional scaffolding you add only when a step calls for it.",{"type":40,"tag":173,"props":174,"children":178},"pre",{"className":175,"code":176,"language":18,"meta":177,"style":177},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","version: \"0.1\"        # schema version (NOT the sflow release) — required in EVERY file\nvariables: { ... }    # optional — ${{ variables.X }} in YAML, ${X} in scripts\nartifacts: [ ... ]    # optional — named paths\u002FURIs, auto-mounted at the same path\nbackends:  [ ... ]    # optional — omit ⇒ a default `local` backend\noperators: [ ... ]    # optional — reusable launch presets\nstorage:   [ ... ]    # optional — S3 upload targets\nworkflow:\n  name: ...\n  tasks: [ ... ]      # THE DAG — design this first (Step 1)\n","",[179],{"type":40,"tag":63,"props":180,"children":181},{"__ignoreMap":177},[182,222,256,288,319,349,379,393,411],{"type":40,"tag":183,"props":184,"children":187},"span",{"class":185,"line":186},"line",1,[188,194,200,205,211,216],{"type":40,"tag":183,"props":189,"children":191},{"style":190},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[192],{"type":46,"value":193},"version",{"type":40,"tag":183,"props":195,"children":197},{"style":196},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[198],{"type":46,"value":199},":",{"type":40,"tag":183,"props":201,"children":202},{"style":196},[203],{"type":46,"value":204}," \"",{"type":40,"tag":183,"props":206,"children":208},{"style":207},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[209],{"type":46,"value":210},"0.1",{"type":40,"tag":183,"props":212,"children":213},{"style":196},[214],{"type":46,"value":215},"\"",{"type":40,"tag":183,"props":217,"children":219},{"style":218},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[220],{"type":46,"value":221},"        # schema version (NOT the sflow release) — required in EVERY file\n",{"type":40,"tag":183,"props":223,"children":225},{"class":185,"line":224},2,[226,231,235,240,246,251],{"type":40,"tag":183,"props":227,"children":228},{"style":190},[229],{"type":46,"value":230},"variables",{"type":40,"tag":183,"props":232,"children":233},{"style":196},[234],{"type":46,"value":199},{"type":40,"tag":183,"props":236,"children":237},{"style":196},[238],{"type":46,"value":239}," {",{"type":40,"tag":183,"props":241,"children":243},{"style":242},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[244],{"type":46,"value":245}," ...",{"type":40,"tag":183,"props":247,"children":248},{"style":196},[249],{"type":46,"value":250}," }",{"type":40,"tag":183,"props":252,"children":253},{"style":218},[254],{"type":46,"value":255},"    # optional — ${{ variables.X }} in YAML, ${X} in scripts\n",{"type":40,"tag":183,"props":257,"children":259},{"class":185,"line":258},3,[260,265,269,274,278,283],{"type":40,"tag":183,"props":261,"children":262},{"style":190},[263],{"type":46,"value":264},"artifacts",{"type":40,"tag":183,"props":266,"children":267},{"style":196},[268],{"type":46,"value":199},{"type":40,"tag":183,"props":270,"children":271},{"style":196},[272],{"type":46,"value":273}," [",{"type":40,"tag":183,"props":275,"children":276},{"style":242},[277],{"type":46,"value":245},{"type":40,"tag":183,"props":279,"children":280},{"style":196},[281],{"type":46,"value":282}," ]",{"type":40,"tag":183,"props":284,"children":285},{"style":218},[286],{"type":46,"value":287},"    # optional — named paths\u002FURIs, auto-mounted at the same path\n",{"type":40,"tag":183,"props":289,"children":291},{"class":185,"line":290},4,[292,297,301,306,310,314],{"type":40,"tag":183,"props":293,"children":294},{"style":190},[295],{"type":46,"value":296},"backends",{"type":40,"tag":183,"props":298,"children":299},{"style":196},[300],{"type":46,"value":199},{"type":40,"tag":183,"props":302,"children":303},{"style":196},[304],{"type":46,"value":305},"  [",{"type":40,"tag":183,"props":307,"children":308},{"style":242},[309],{"type":46,"value":245},{"type":40,"tag":183,"props":311,"children":312},{"style":196},[313],{"type":46,"value":282},{"type":40,"tag":183,"props":315,"children":316},{"style":218},[317],{"type":46,"value":318},"    # optional — omit ⇒ a default `local` backend\n",{"type":40,"tag":183,"props":320,"children":322},{"class":185,"line":321},5,[323,328,332,336,340,344],{"type":40,"tag":183,"props":324,"children":325},{"style":190},[326],{"type":46,"value":327},"operators",{"type":40,"tag":183,"props":329,"children":330},{"style":196},[331],{"type":46,"value":199},{"type":40,"tag":183,"props":333,"children":334},{"style":196},[335],{"type":46,"value":273},{"type":40,"tag":183,"props":337,"children":338},{"style":242},[339],{"type":46,"value":245},{"type":40,"tag":183,"props":341,"children":342},{"style":196},[343],{"type":46,"value":282},{"type":40,"tag":183,"props":345,"children":346},{"style":218},[347],{"type":46,"value":348},"    # optional — reusable launch presets\n",{"type":40,"tag":183,"props":350,"children":351},{"class":185,"line":27},[352,357,361,366,370,374],{"type":40,"tag":183,"props":353,"children":354},{"style":190},[355],{"type":46,"value":356},"storage",{"type":40,"tag":183,"props":358,"children":359},{"style":196},[360],{"type":46,"value":199},{"type":40,"tag":183,"props":362,"children":363},{"style":196},[364],{"type":46,"value":365},"   [",{"type":40,"tag":183,"props":367,"children":368},{"style":242},[369],{"type":46,"value":245},{"type":40,"tag":183,"props":371,"children":372},{"style":196},[373],{"type":46,"value":282},{"type":40,"tag":183,"props":375,"children":376},{"style":218},[377],{"type":46,"value":378},"    # optional — S3 upload targets\n",{"type":40,"tag":183,"props":380,"children":382},{"class":185,"line":381},7,[383,388],{"type":40,"tag":183,"props":384,"children":385},{"style":190},[386],{"type":46,"value":387},"workflow",{"type":40,"tag":183,"props":389,"children":390},{"style":196},[391],{"type":46,"value":392},":\n",{"type":40,"tag":183,"props":394,"children":396},{"class":185,"line":395},8,[397,402,406],{"type":40,"tag":183,"props":398,"children":399},{"style":190},[400],{"type":46,"value":401},"  name",{"type":40,"tag":183,"props":403,"children":404},{"style":196},[405],{"type":46,"value":199},{"type":40,"tag":183,"props":407,"children":408},{"style":242},[409],{"type":46,"value":410}," ...\n",{"type":40,"tag":183,"props":412,"children":414},{"class":185,"line":413},9,[415,420,424,428,432,436],{"type":40,"tag":183,"props":416,"children":417},{"style":190},[418],{"type":46,"value":419},"  tasks",{"type":40,"tag":183,"props":421,"children":422},{"style":196},[423],{"type":46,"value":199},{"type":40,"tag":183,"props":425,"children":426},{"style":196},[427],{"type":46,"value":273},{"type":40,"tag":183,"props":429,"children":430},{"style":242},[431],{"type":46,"value":245},{"type":40,"tag":183,"props":433,"children":434},{"style":196},[435],{"type":46,"value":282},{"type":40,"tag":183,"props":437,"children":438},{"style":218},[439],{"type":46,"value":440},"      # THE DAG — design this first (Step 1)\n",{"type":40,"tag":49,"props":442,"children":443},{},[444,446,451,453,458,460,465,467,473],{"type":46,"value":445},"Omit ",{"type":40,"tag":63,"props":447,"children":449},{"className":448},[],[450],{"type":46,"value":296},{"type":46,"value":452},"\u002F",{"type":40,"tag":63,"props":454,"children":456},{"className":455},[],[457],{"type":46,"value":327},{"type":46,"value":459}," and you get ",{"type":40,"tag":63,"props":461,"children":463},{"className":462},[],[464],{"type":46,"value":68},{"type":46,"value":466}," + ",{"type":40,"tag":63,"props":468,"children":470},{"className":469},[],[471],{"type":46,"value":472},"bash",{"type":46,"value":474}," — enough to prototype a DAG on a\nlaptop before touching a cluster.",{"type":40,"tag":476,"props":477,"children":478},"hr",{},[],{"type":40,"tag":153,"props":480,"children":482},{"id":481},"step-1-shape-the-dag-do-this-first",[483],{"type":46,"value":484},"Step 1 — Shape the DAG (do this first)",{"type":40,"tag":49,"props":486,"children":487},{},[488,490,495,497,502],{"type":46,"value":489},"The DAG ",{"type":40,"tag":101,"props":491,"children":492},{},[493],{"type":46,"value":494},"is",{"type":46,"value":496}," the recipe. Decide ",{"type":40,"tag":55,"props":498,"children":499},{},[500],{"type":46,"value":501},"what the tasks are and how they connect",{"type":46,"value":503}," before writing\nany script body.",{"type":40,"tag":505,"props":506,"children":507},"ol",{},[508,557,589,606],{"type":40,"tag":509,"props":510,"children":511},"li",{},[512,517,519,525,527,533,535,541,542,548,549,555],{"type":40,"tag":55,"props":513,"children":514},{},[515],{"type":46,"value":516},"List the tasks",{"type":46,"value":518}," — one node per logical step or service (",{"type":40,"tag":63,"props":520,"children":522},{"className":521},[],[523],{"type":46,"value":524},"etcd",{"type":46,"value":526},", ",{"type":40,"tag":63,"props":528,"children":530},{"className":529},[],[531],{"type":46,"value":532},"frontend",{"type":46,"value":534},",\n",{"type":40,"tag":63,"props":536,"children":538},{"className":537},[],[539],{"type":46,"value":540},"prefill",{"type":46,"value":526},{"type":40,"tag":63,"props":543,"children":545},{"className":544},[],[546],{"type":46,"value":547},"decode",{"type":46,"value":526},{"type":40,"tag":63,"props":550,"children":552},{"className":551},[],[553],{"type":46,"value":554},"benchmark",{"type":46,"value":556},", …).",{"type":40,"tag":509,"props":558,"children":559},{},[560,565,567,573,575,580,582,587],{"type":40,"tag":55,"props":561,"children":562},{},[563],{"type":46,"value":564},"Wire them",{"type":46,"value":566}," with ",{"type":40,"tag":63,"props":568,"children":570},{"className":569},[],[571],{"type":46,"value":572},"depends_on",{"type":46,"value":574}," (a task lists the upstreams it waits for). A dependent\nstarts once every upstream is ",{"type":40,"tag":55,"props":576,"children":577},{},[578],{"type":46,"value":579},"READY",{"type":46,"value":581}," (if that upstream has a readiness probe, Step 4)\nor ",{"type":40,"tag":55,"props":583,"children":584},{},[585],{"type":46,"value":586},"COMPLETED",{"type":46,"value":588}," (if it has none).",{"type":40,"tag":509,"props":590,"children":591},{},[592,597,599,604],{"type":40,"tag":55,"props":593,"children":594},{},[595],{"type":46,"value":596},"Identify the terminal task",{"type":46,"value":598}," — the one whose completion ends the run (usually the\nclient\u002F",{"type":40,"tag":63,"props":600,"children":602},{"className":601},[],[603],{"type":46,"value":554},{"type":46,"value":605},"). Long-running servers never \"complete\"; sflow stops them when the\nterminal task finishes.",{"type":40,"tag":509,"props":607,"children":608},{},[609,614,615,621,623,628,630,635],{"type":40,"tag":55,"props":610,"children":611},{},[612],{"type":46,"value":613},"Reusable fragments wire in reverse",{"type":46,"value":566},{"type":40,"tag":63,"props":616,"children":618},{"className":617},[],[619],{"type":46,"value":620},"required_by:",{"type":46,"value":622}," — a module attaches itself to\na shared hub (e.g. ",{"type":40,"tag":63,"props":624,"children":626},{"className":625},[],[627],{"type":46,"value":554},{"type":46,"value":629},") without editing the hub or adding cross-file ",{"type":40,"tag":63,"props":631,"children":633},{"className":632},[],[634],{"type":46,"value":572},{"type":46,"value":151},{"type":40,"tag":49,"props":637,"children":638},{},[639],{"type":46,"value":640},"Canonical serving shape (Dynamo-style — see examples.md Example 3):",{"type":40,"tag":173,"props":642,"children":646},{"className":643,"code":645,"language":46,"meta":177},[644],"language-text","infra (nats, etcd) ──▶ frontend ──▶ workers (prefill + decode | agg) ──▶ benchmark ──▶ end\n",[647],{"type":40,"tag":63,"props":648,"children":649},{"__ignoreMap":177},[650],{"type":46,"value":645},{"type":40,"tag":49,"props":652,"children":653},{},[654,656,661],{"type":46,"value":655},"Write the wiring skeleton first — ",{"type":40,"tag":55,"props":657,"children":658},{},[659],{"type":46,"value":660},"script bodies come in Step 2",{"type":46,"value":199},{"type":40,"tag":173,"props":663,"children":665},{"className":175,"code":664,"language":18,"meta":177,"style":177},"workflow:\n  name: serve_and_bench\n  tasks:\n    - { name: etcd }\n    - { name: frontend,  depends_on: [etcd] }\n    - { name: worker,    depends_on: [frontend] }\n    - { name: benchmark, depends_on: [worker] }        # terminal task ⇒ run ends here\n",[666],{"type":40,"tag":63,"props":667,"children":668},{"__ignoreMap":177},[669,680,696,707,738,793,846],{"type":40,"tag":183,"props":670,"children":671},{"class":185,"line":186},[672,676],{"type":40,"tag":183,"props":673,"children":674},{"style":190},[675],{"type":46,"value":387},{"type":40,"tag":183,"props":677,"children":678},{"style":196},[679],{"type":46,"value":392},{"type":40,"tag":183,"props":681,"children":682},{"class":185,"line":224},[683,687,691],{"type":40,"tag":183,"props":684,"children":685},{"style":190},[686],{"type":46,"value":401},{"type":40,"tag":183,"props":688,"children":689},{"style":196},[690],{"type":46,"value":199},{"type":40,"tag":183,"props":692,"children":693},{"style":207},[694],{"type":46,"value":695}," serve_and_bench\n",{"type":40,"tag":183,"props":697,"children":698},{"class":185,"line":258},[699,703],{"type":40,"tag":183,"props":700,"children":701},{"style":190},[702],{"type":46,"value":419},{"type":40,"tag":183,"props":704,"children":705},{"style":196},[706],{"type":46,"value":392},{"type":40,"tag":183,"props":708,"children":709},{"class":185,"line":290},[710,715,719,724,728,733],{"type":40,"tag":183,"props":711,"children":712},{"style":196},[713],{"type":46,"value":714},"    -",{"type":40,"tag":183,"props":716,"children":717},{"style":196},[718],{"type":46,"value":239},{"type":40,"tag":183,"props":720,"children":721},{"style":190},[722],{"type":46,"value":723}," name",{"type":40,"tag":183,"props":725,"children":726},{"style":196},[727],{"type":46,"value":199},{"type":40,"tag":183,"props":729,"children":730},{"style":207},[731],{"type":46,"value":732}," etcd",{"type":40,"tag":183,"props":734,"children":735},{"style":196},[736],{"type":46,"value":737}," }\n",{"type":40,"tag":183,"props":739,"children":740},{"class":185,"line":321},[741,745,749,753,757,762,767,772,776,780,784,789],{"type":40,"tag":183,"props":742,"children":743},{"style":196},[744],{"type":46,"value":714},{"type":40,"tag":183,"props":746,"children":747},{"style":196},[748],{"type":46,"value":239},{"type":40,"tag":183,"props":750,"children":751},{"style":190},[752],{"type":46,"value":723},{"type":40,"tag":183,"props":754,"children":755},{"style":196},[756],{"type":46,"value":199},{"type":40,"tag":183,"props":758,"children":759},{"style":207},[760],{"type":46,"value":761}," frontend",{"type":40,"tag":183,"props":763,"children":764},{"style":196},[765],{"type":46,"value":766},",",{"type":40,"tag":183,"props":768,"children":769},{"style":190},[770],{"type":46,"value":771},"  depends_on",{"type":40,"tag":183,"props":773,"children":774},{"style":196},[775],{"type":46,"value":199},{"type":40,"tag":183,"props":777,"children":778},{"style":196},[779],{"type":46,"value":273},{"type":40,"tag":183,"props":781,"children":782},{"style":207},[783],{"type":46,"value":524},{"type":40,"tag":183,"props":785,"children":786},{"style":196},[787],{"type":46,"value":788},"]",{"type":40,"tag":183,"props":790,"children":791},{"style":196},[792],{"type":46,"value":737},{"type":40,"tag":183,"props":794,"children":795},{"class":185,"line":27},[796,800,804,808,812,817,821,826,830,834,838,842],{"type":40,"tag":183,"props":797,"children":798},{"style":196},[799],{"type":46,"value":714},{"type":40,"tag":183,"props":801,"children":802},{"style":196},[803],{"type":46,"value":239},{"type":40,"tag":183,"props":805,"children":806},{"style":190},[807],{"type":46,"value":723},{"type":40,"tag":183,"props":809,"children":810},{"style":196},[811],{"type":46,"value":199},{"type":40,"tag":183,"props":813,"children":814},{"style":207},[815],{"type":46,"value":816}," worker",{"type":40,"tag":183,"props":818,"children":819},{"style":196},[820],{"type":46,"value":766},{"type":40,"tag":183,"props":822,"children":823},{"style":190},[824],{"type":46,"value":825},"    depends_on",{"type":40,"tag":183,"props":827,"children":828},{"style":196},[829],{"type":46,"value":199},{"type":40,"tag":183,"props":831,"children":832},{"style":196},[833],{"type":46,"value":273},{"type":40,"tag":183,"props":835,"children":836},{"style":207},[837],{"type":46,"value":532},{"type":40,"tag":183,"props":839,"children":840},{"style":196},[841],{"type":46,"value":788},{"type":40,"tag":183,"props":843,"children":844},{"style":196},[845],{"type":46,"value":737},{"type":40,"tag":183,"props":847,"children":848},{"class":185,"line":381},[849,853,857,861,865,870,874,879,883,887,892,896,900],{"type":40,"tag":183,"props":850,"children":851},{"style":196},[852],{"type":46,"value":714},{"type":40,"tag":183,"props":854,"children":855},{"style":196},[856],{"type":46,"value":239},{"type":40,"tag":183,"props":858,"children":859},{"style":190},[860],{"type":46,"value":723},{"type":40,"tag":183,"props":862,"children":863},{"style":196},[864],{"type":46,"value":199},{"type":40,"tag":183,"props":866,"children":867},{"style":207},[868],{"type":46,"value":869}," benchmark",{"type":40,"tag":183,"props":871,"children":872},{"style":196},[873],{"type":46,"value":766},{"type":40,"tag":183,"props":875,"children":876},{"style":190},[877],{"type":46,"value":878}," depends_on",{"type":40,"tag":183,"props":880,"children":881},{"style":196},[882],{"type":46,"value":199},{"type":40,"tag":183,"props":884,"children":885},{"style":196},[886],{"type":46,"value":273},{"type":40,"tag":183,"props":888,"children":889},{"style":207},[890],{"type":46,"value":891},"worker",{"type":40,"tag":183,"props":893,"children":894},{"style":196},[895],{"type":46,"value":788},{"type":40,"tag":183,"props":897,"children":898},{"style":196},[899],{"type":46,"value":250},{"type":40,"tag":183,"props":901,"children":902},{"style":218},[903],{"type":46,"value":904},"        # terminal task ⇒ run ends here\n",{"type":40,"tag":49,"props":906,"children":907},{},[908,910,915],{"type":46,"value":909},"Sanity-check the shape now: ",{"type":40,"tag":101,"props":911,"children":912},{},[913],{"type":46,"value":914},"does anything start before its inputs exist? is there exactly\none clear end? do \"parallel\" branches actually need to be concurrent?",{"type":46,"value":916}," Fixing shape is cheap\nhere and expensive later.",{"type":40,"tag":153,"props":918,"children":920},{"id":919},"step-2-write-the-task-scripts",[921],{"type":46,"value":922},"Step 2 — Write the task scripts",{"type":40,"tag":924,"props":925,"children":926},"ul",{},[927,968,1001],{"type":40,"tag":509,"props":928,"children":929},{},[930,936,938,943,945,951,953,958,960,966],{"type":40,"tag":63,"props":931,"children":933},{"className":932},[],[934],{"type":46,"value":935},"script:",{"type":46,"value":937}," is a ",{"type":40,"tag":55,"props":939,"children":940},{},[941],{"type":46,"value":942},"list of lines joined into one bash script",{"type":46,"value":944}," (the ",{"type":40,"tag":63,"props":946,"children":948},{"className":947},[],[949],{"type":46,"value":950},"python",{"type":46,"value":952}," operator instead\nruns the lines as Python ",{"type":40,"tag":101,"props":954,"children":955},{},[956],{"type":46,"value":957},"source",{"type":46,"value":959}," via ",{"type":40,"tag":63,"props":961,"children":963},{"className":962},[],[964],{"type":46,"value":965},"python -c",{"type":46,"value":967},").",{"type":40,"tag":509,"props":969,"children":970},{},[971,976,978,983,985,991,993,999],{"type":40,"tag":55,"props":972,"children":973},{},[974],{"type":46,"value":975},"Don't inline files or hard-code paths.",{"type":46,"value":977}," Any file a script needs — launch\u002Fhelper script,\nconfig, model dir, dataset — belongs in an ",{"type":40,"tag":55,"props":979,"children":980},{},[981],{"type":46,"value":982},"artifact",{"type":46,"value":984}," (Step 3), referenced as\n",{"type":40,"tag":63,"props":986,"children":988},{"className":987},[],[989],{"type":46,"value":990},"${{ artifacts.X.path }}",{"type":46,"value":992},". Never embed Python via heredocs or ",{"type":40,"tag":63,"props":994,"children":996},{"className":995},[],[997],{"type":46,"value":998},"python3 -c",{"type":46,"value":1000}," — quoting breaks\nwhen YAML → shell → Python nests.",{"type":40,"tag":509,"props":1002,"children":1003},{},[1004,1009,1011,1017,1019,1025,1027,1033,1035,1041],{"type":40,"tag":55,"props":1005,"children":1006},{},[1007],{"type":46,"value":1008},"Reference values:",{"type":46,"value":1010}," ",{"type":40,"tag":63,"props":1012,"children":1014},{"className":1013},[],[1015],{"type":46,"value":1016},"${{ variables.X }}",{"type":46,"value":1018}," in YAML (plan-time), ",{"type":40,"tag":63,"props":1020,"children":1022},{"className":1021},[],[1023],{"type":46,"value":1024},"${X}",{"type":46,"value":1026}," in scripts (runtime).\nFor arithmetic\u002Fcomparison, set the variable's ",{"type":40,"tag":63,"props":1028,"children":1030},{"className":1029},[],[1031],{"type":46,"value":1032},"type",{"type":46,"value":1034}," (",{"type":40,"tag":63,"props":1036,"children":1038},{"className":1037},[],[1039],{"type":46,"value":1040},"integer",{"type":46,"value":1042},", …) or it stays a string.",{"type":40,"tag":173,"props":1044,"children":1046},{"className":175,"code":1045,"language":18,"meta":177,"style":177},"workflow:\n  tasks:\n    - name: worker\n      depends_on: [frontend]\n      script:\n        - python3 ${{ artifacts.LAUNCH.path }} ${{ variables.MODEL }}   # LAUNCH: see Step 3\n",[1047],{"type":40,"tag":63,"props":1048,"children":1049},{"__ignoreMap":177},[1050,1061,1072,1092,1117,1129],{"type":40,"tag":183,"props":1051,"children":1052},{"class":185,"line":186},[1053,1057],{"type":40,"tag":183,"props":1054,"children":1055},{"style":190},[1056],{"type":46,"value":387},{"type":40,"tag":183,"props":1058,"children":1059},{"style":196},[1060],{"type":46,"value":392},{"type":40,"tag":183,"props":1062,"children":1063},{"class":185,"line":224},[1064,1068],{"type":40,"tag":183,"props":1065,"children":1066},{"style":190},[1067],{"type":46,"value":419},{"type":40,"tag":183,"props":1069,"children":1070},{"style":196},[1071],{"type":46,"value":392},{"type":40,"tag":183,"props":1073,"children":1074},{"class":185,"line":258},[1075,1079,1083,1087],{"type":40,"tag":183,"props":1076,"children":1077},{"style":196},[1078],{"type":46,"value":714},{"type":40,"tag":183,"props":1080,"children":1081},{"style":190},[1082],{"type":46,"value":723},{"type":40,"tag":183,"props":1084,"children":1085},{"style":196},[1086],{"type":46,"value":199},{"type":40,"tag":183,"props":1088,"children":1089},{"style":207},[1090],{"type":46,"value":1091}," worker\n",{"type":40,"tag":183,"props":1093,"children":1094},{"class":185,"line":290},[1095,1100,1104,1108,1112],{"type":40,"tag":183,"props":1096,"children":1097},{"style":190},[1098],{"type":46,"value":1099},"      depends_on",{"type":40,"tag":183,"props":1101,"children":1102},{"style":196},[1103],{"type":46,"value":199},{"type":40,"tag":183,"props":1105,"children":1106},{"style":196},[1107],{"type":46,"value":273},{"type":40,"tag":183,"props":1109,"children":1110},{"style":207},[1111],{"type":46,"value":532},{"type":40,"tag":183,"props":1113,"children":1114},{"style":196},[1115],{"type":46,"value":1116},"]\n",{"type":40,"tag":183,"props":1118,"children":1119},{"class":185,"line":321},[1120,1125],{"type":40,"tag":183,"props":1121,"children":1122},{"style":190},[1123],{"type":46,"value":1124},"      script",{"type":40,"tag":183,"props":1126,"children":1127},{"style":196},[1128],{"type":46,"value":392},{"type":40,"tag":183,"props":1130,"children":1131},{"class":185,"line":27},[1132,1137,1142],{"type":40,"tag":183,"props":1133,"children":1134},{"style":196},[1135],{"type":46,"value":1136},"        -",{"type":40,"tag":183,"props":1138,"children":1139},{"style":207},[1140],{"type":46,"value":1141}," python3 ${{ artifacts.LAUNCH.path }} ${{ variables.MODEL }}",{"type":40,"tag":183,"props":1143,"children":1144},{"style":218},[1145],{"type":46,"value":1146},"   # LAUNCH: see Step 3\n",{"type":40,"tag":49,"props":1148,"children":1149},{},[1150],{"type":46,"value":1151},"See examples.md Examples 1–2.",{"type":40,"tag":153,"props":1153,"children":1155},{"id":1154},"step-3-handle-paths-with-artifacts-dont-hand-manage-paths",[1156],{"type":46,"value":1157},"Step 3 — Handle paths with artifacts (don't hand-manage paths)",{"type":40,"tag":49,"props":1159,"children":1160},{},[1161,1173,1175,1180,1182,1187,1189,1194,1196,1201,1203,1208,1210,1215,1217,1223,1224,1230,1232,1238],{"type":40,"tag":55,"props":1162,"children":1163},{},[1164,1166,1171],{"type":46,"value":1165},"An artifact is a named path\u002FURI that sflow resolves, validates, and — inside containers —\nauto-mounts at the ",{"type":40,"tag":101,"props":1167,"children":1168},{},[1169],{"type":46,"value":1170},"same absolute path",{"type":46,"value":1172}," on every backend.",{"type":46,"value":1174}," This is the biggest lever for\nportability: declare a path once and you stop caring ",{"type":40,"tag":101,"props":1176,"children":1177},{},[1178],{"type":46,"value":1179},"where",{"type":46,"value":1181}," it lives or ",{"type":40,"tag":101,"props":1183,"children":1184},{},[1185],{"type":46,"value":1186},"how",{"type":46,"value":1188}," to mount it.\n",{"type":40,"tag":63,"props":1190,"children":1192},{"className":1191},[],[1193],{"type":46,"value":990},{"type":46,"value":1195}," (YAML, plan-time) and ",{"type":40,"tag":63,"props":1197,"children":1199},{"className":1198},[],[1200],{"type":46,"value":1024},{"type":46,"value":1202}," (script, runtime) always agree, sflow\n",{"type":40,"tag":55,"props":1204,"children":1205},{},[1206],{"type":46,"value":1207},"auto-checks",{"type":46,"value":1209}," the reference at preflight, and ",{"type":40,"tag":55,"props":1211,"children":1212},{},[1213],{"type":46,"value":1214},"auto-mounts",{"type":46,"value":1216}," it — so you never hand-write\n",{"type":40,"tag":63,"props":1218,"children":1220},{"className":1219},[],[1221],{"type":46,"value":1222},"container_mounts",{"type":46,"value":526},{"type":40,"tag":63,"props":1225,"children":1227},{"className":1226},[],[1228],{"type":46,"value":1229},"docker -v",{"type":46,"value":1231},", or k8s ",{"type":40,"tag":63,"props":1233,"children":1235},{"className":1234},[],[1236],{"type":46,"value":1237},"volumes",{"type":46,"value":1239}," for it (doing so is redundant — pitfall 13).",{"type":40,"tag":49,"props":1241,"children":1242},{},[1243],{"type":46,"value":1244},"Pick the scheme by what the artifact holds:",{"type":40,"tag":924,"props":1246,"children":1247},{},[1248,1525],{"type":40,"tag":509,"props":1249,"children":1250},{},[1251,1271,1273,1278,1280,1285,1287,1292,1294,1299,1301],{"type":40,"tag":55,"props":1252,"children":1253},{},[1254,1256,1262,1264,1270],{"type":46,"value":1255},"Small code snippets & config files → a ",{"type":40,"tag":63,"props":1257,"children":1259},{"className":1258},[],[1260],{"type":46,"value":1261},"file:\u002F\u002F",{"type":46,"value":1263}," artifact with inline ",{"type":40,"tag":63,"props":1265,"children":1267},{"className":1266},[],[1268],{"type":46,"value":1269},"content:",{"type":46,"value":151},{"type":46,"value":1272}," sflow\nwrites the content and mounts it into the container at the same path on ",{"type":40,"tag":101,"props":1274,"children":1275},{},[1276],{"type":46,"value":1277},"any",{"type":46,"value":1279}," backend —\nlocal, docker, slurm, kubernetes — so ",{"type":40,"tag":55,"props":1281,"children":1282},{},[1283],{"type":46,"value":1284},"you don't manage where it lives or how it gets into\nthe container",{"type":46,"value":1286},". Use it for launch\u002Fhelper scripts, config JSON\u002FYAML, engine specs, etc.\n",{"type":40,"tag":55,"props":1288,"children":1289},{},[1290],{"type":46,"value":1291},"Never",{"type":46,"value":1293}," inline them via heredocs \u002F ",{"type":40,"tag":63,"props":1295,"children":1297},{"className":1296},[],[1298],{"type":46,"value":998},{"type":46,"value":1300}," (YAML → shell → Python quoting breaks) or\nbake them into the image.",{"type":40,"tag":173,"props":1302,"children":1304},{"className":175,"code":1303,"language":18,"meta":177,"style":177},"artifacts:\n  - name: LAUNCH\n    uri: file:\u002F\u002Flaunch.py             # relative → resolves under SFLOW_WORKSPACE_DIR\n    content: |\n      import sys; print(\"serving\", sys.argv[1])\n  - name: GEN_CONFIG\n    uri: file:\u002F\u002Fgen_config.yaml\n    content: |\n      max_batch_size: 256\n      kv_cache_free_gpu_mem_fraction: 0.9\nworkflow:\n  tasks:\n    - name: worker\n      script:\n        - python3 ${{ artifacts.LAUNCH.path }} --config ${{ artifacts.GEN_CONFIG.path }}\n",[1305],{"type":40,"tag":63,"props":1306,"children":1307},{"__ignoreMap":177},[1308,1319,1340,1362,1380,1388,1408,1424,1439,1447,1456,1468,1480,1500,1512],{"type":40,"tag":183,"props":1309,"children":1310},{"class":185,"line":186},[1311,1315],{"type":40,"tag":183,"props":1312,"children":1313},{"style":190},[1314],{"type":46,"value":264},{"type":40,"tag":183,"props":1316,"children":1317},{"style":196},[1318],{"type":46,"value":392},{"type":40,"tag":183,"props":1320,"children":1321},{"class":185,"line":224},[1322,1327,1331,1335],{"type":40,"tag":183,"props":1323,"children":1324},{"style":196},[1325],{"type":46,"value":1326},"  -",{"type":40,"tag":183,"props":1328,"children":1329},{"style":190},[1330],{"type":46,"value":723},{"type":40,"tag":183,"props":1332,"children":1333},{"style":196},[1334],{"type":46,"value":199},{"type":40,"tag":183,"props":1336,"children":1337},{"style":207},[1338],{"type":46,"value":1339}," LAUNCH\n",{"type":40,"tag":183,"props":1341,"children":1342},{"class":185,"line":258},[1343,1348,1352,1357],{"type":40,"tag":183,"props":1344,"children":1345},{"style":190},[1346],{"type":46,"value":1347},"    uri",{"type":40,"tag":183,"props":1349,"children":1350},{"style":196},[1351],{"type":46,"value":199},{"type":40,"tag":183,"props":1353,"children":1354},{"style":207},[1355],{"type":46,"value":1356}," file:\u002F\u002Flaunch.py",{"type":40,"tag":183,"props":1358,"children":1359},{"style":218},[1360],{"type":46,"value":1361},"             # relative → resolves under SFLOW_WORKSPACE_DIR\n",{"type":40,"tag":183,"props":1363,"children":1364},{"class":185,"line":290},[1365,1370,1374],{"type":40,"tag":183,"props":1366,"children":1367},{"style":190},[1368],{"type":46,"value":1369},"    content",{"type":40,"tag":183,"props":1371,"children":1372},{"style":196},[1373],{"type":46,"value":199},{"type":40,"tag":183,"props":1375,"children":1377},{"style":1376},"--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic",[1378],{"type":46,"value":1379}," |\n",{"type":40,"tag":183,"props":1381,"children":1382},{"class":185,"line":321},[1383],{"type":40,"tag":183,"props":1384,"children":1385},{"style":207},[1386],{"type":46,"value":1387},"      import sys; print(\"serving\", sys.argv[1])\n",{"type":40,"tag":183,"props":1389,"children":1390},{"class":185,"line":27},[1391,1395,1399,1403],{"type":40,"tag":183,"props":1392,"children":1393},{"style":196},[1394],{"type":46,"value":1326},{"type":40,"tag":183,"props":1396,"children":1397},{"style":190},[1398],{"type":46,"value":723},{"type":40,"tag":183,"props":1400,"children":1401},{"style":196},[1402],{"type":46,"value":199},{"type":40,"tag":183,"props":1404,"children":1405},{"style":207},[1406],{"type":46,"value":1407}," GEN_CONFIG\n",{"type":40,"tag":183,"props":1409,"children":1410},{"class":185,"line":381},[1411,1415,1419],{"type":40,"tag":183,"props":1412,"children":1413},{"style":190},[1414],{"type":46,"value":1347},{"type":40,"tag":183,"props":1416,"children":1417},{"style":196},[1418],{"type":46,"value":199},{"type":40,"tag":183,"props":1420,"children":1421},{"style":207},[1422],{"type":46,"value":1423}," file:\u002F\u002Fgen_config.yaml\n",{"type":40,"tag":183,"props":1425,"children":1426},{"class":185,"line":395},[1427,1431,1435],{"type":40,"tag":183,"props":1428,"children":1429},{"style":190},[1430],{"type":46,"value":1369},{"type":40,"tag":183,"props":1432,"children":1433},{"style":196},[1434],{"type":46,"value":199},{"type":40,"tag":183,"props":1436,"children":1437},{"style":1376},[1438],{"type":46,"value":1379},{"type":40,"tag":183,"props":1440,"children":1441},{"class":185,"line":413},[1442],{"type":40,"tag":183,"props":1443,"children":1444},{"style":207},[1445],{"type":46,"value":1446},"      max_batch_size: 256\n",{"type":40,"tag":183,"props":1448,"children":1450},{"class":185,"line":1449},10,[1451],{"type":40,"tag":183,"props":1452,"children":1453},{"style":207},[1454],{"type":46,"value":1455},"      kv_cache_free_gpu_mem_fraction: 0.9\n",{"type":40,"tag":183,"props":1457,"children":1459},{"class":185,"line":1458},11,[1460,1464],{"type":40,"tag":183,"props":1461,"children":1462},{"style":190},[1463],{"type":46,"value":387},{"type":40,"tag":183,"props":1465,"children":1466},{"style":196},[1467],{"type":46,"value":392},{"type":40,"tag":183,"props":1469,"children":1471},{"class":185,"line":1470},12,[1472,1476],{"type":40,"tag":183,"props":1473,"children":1474},{"style":190},[1475],{"type":46,"value":419},{"type":40,"tag":183,"props":1477,"children":1478},{"style":196},[1479],{"type":46,"value":392},{"type":40,"tag":183,"props":1481,"children":1483},{"class":185,"line":1482},13,[1484,1488,1492,1496],{"type":40,"tag":183,"props":1485,"children":1486},{"style":196},[1487],{"type":46,"value":714},{"type":40,"tag":183,"props":1489,"children":1490},{"style":190},[1491],{"type":46,"value":723},{"type":40,"tag":183,"props":1493,"children":1494},{"style":196},[1495],{"type":46,"value":199},{"type":40,"tag":183,"props":1497,"children":1498},{"style":207},[1499],{"type":46,"value":1091},{"type":40,"tag":183,"props":1501,"children":1503},{"class":185,"line":1502},14,[1504,1508],{"type":40,"tag":183,"props":1505,"children":1506},{"style":190},[1507],{"type":46,"value":1124},{"type":40,"tag":183,"props":1509,"children":1510},{"style":196},[1511],{"type":46,"value":392},{"type":40,"tag":183,"props":1513,"children":1515},{"class":185,"line":1514},15,[1516,1520],{"type":40,"tag":183,"props":1517,"children":1518},{"style":196},[1519],{"type":46,"value":1136},{"type":40,"tag":183,"props":1521,"children":1522},{"style":207},[1523],{"type":46,"value":1524}," python3 ${{ artifacts.LAUNCH.path }} --config ${{ artifacts.GEN_CONFIG.path }}\n",{"type":40,"tag":509,"props":1526,"children":1527},{},[1528,1541,1543,1548,1550],{"type":40,"tag":55,"props":1529,"children":1530},{},[1531,1533,1539],{"type":46,"value":1532},"Large \u002F pre-existing data (models, datasets, checkpoints) → an ",{"type":40,"tag":63,"props":1534,"children":1536},{"className":1535},[],[1537],{"type":46,"value":1538},"fs:\u002F\u002F",{"type":46,"value":1540}," artifact",{"type":46,"value":1542}," pointing\nat storage the compute nodes already see. It still resolves + mounts the same path\neverywhere, but ",{"type":40,"tag":101,"props":1544,"children":1545},{},[1546],{"type":46,"value":1547},"you",{"type":46,"value":1549}," supply the bytes:",{"type":40,"tag":924,"props":1551,"children":1552},{},[1553,1569],{"type":40,"tag":509,"props":1554,"children":1555},{},[1556,1560,1562,1567],{"type":40,"tag":55,"props":1557,"children":1558},{},[1559],{"type":46,"value":76},{"type":46,"value":1561}," → keep it on ",{"type":40,"tag":55,"props":1563,"children":1564},{},[1565],{"type":46,"value":1566},"shared storage",{"type":46,"value":1568}," (Lustre\u002FGPFS\u002FNFS) so the path exists on every node.",{"type":40,"tag":509,"props":1570,"children":1571},{},[1572,1576,1578,1583,1585,1591,1593,1599,1601,1606,1608,1613,1615,1620],{"type":40,"tag":55,"props":1573,"children":1574},{},[1575],{"type":46,"value":90},{"type":46,"value":1577}," → back it with a ",{"type":40,"tag":55,"props":1579,"children":1580},{},[1581],{"type":46,"value":1582},"PVC",{"type":46,"value":1584},": declare the PVC under backend ",{"type":40,"tag":63,"props":1586,"children":1588},{"className":1587},[],[1589],{"type":46,"value":1590},"volumes:",{"type":46,"value":1592}," at a\n",{"type":40,"tag":63,"props":1594,"children":1596},{"className":1595},[],[1597],{"type":46,"value":1598},"mount_path",{"type":46,"value":1600},", then point the ",{"type":40,"tag":63,"props":1602,"children":1604},{"className":1603},[],[1605],{"type":46,"value":1538},{"type":46,"value":1607}," artifact at a path ",{"type":40,"tag":101,"props":1609,"children":1610},{},[1611],{"type":46,"value":1612},"under",{"type":46,"value":1614}," that mount. An ",{"type":40,"tag":63,"props":1616,"children":1618},{"className":1617},[],[1619],{"type":46,"value":1538},{"type":46,"value":1621}," path\nno PVC covers and the node lacks makes the pod fail (pitfall 9).",{"type":40,"tag":49,"props":1623,"children":1624},{},[1625,1630,1632,1637,1639,1644,1646,1651,1653,1658],{"type":40,"tag":55,"props":1626,"children":1627},{},[1628],{"type":46,"value":1629},"Rule of thumb: if a task touches a path, declare it as an artifact",{"type":46,"value":1631}," — small text → ",{"type":40,"tag":63,"props":1633,"children":1635},{"className":1634},[],[1636],{"type":46,"value":1261},{"type":46,"value":1638},"\nwith ",{"type":40,"tag":63,"props":1640,"children":1642},{"className":1641},[],[1643],{"type":46,"value":1269},{"type":46,"value":1645},", big data → ",{"type":40,"tag":63,"props":1647,"children":1649},{"className":1648},[],[1650],{"type":46,"value":1538},{"type":46,"value":1652}," on shared storage \u002F a PVC. Artifact schemes & ",{"type":40,"tag":63,"props":1654,"children":1656},{"className":1655},[],[1657],{"type":46,"value":1269},{"type":46,"value":1659},"\nrules → schema-reference.md.",{"type":40,"tag":153,"props":1661,"children":1663},{"id":1662},"step-4-gate-the-wiring-with-probes",[1664],{"type":46,"value":1665},"Step 4 — Gate the wiring with probes",{"type":40,"tag":49,"props":1667,"children":1668},{},[1669,1674,1676,1681,1683,1688],{"type":40,"tag":63,"props":1670,"children":1672},{"className":1671},[],[1673],{"type":46,"value":572},{"type":46,"value":1675}," alone only waits for a process to ",{"type":40,"tag":101,"props":1677,"children":1678},{},[1679],{"type":46,"value":1680},"start",{"type":46,"value":1682},". For services, wait for ",{"type":40,"tag":55,"props":1684,"children":1685},{},[1686],{"type":46,"value":1687},"ready",{"type":46,"value":151},{"type":40,"tag":924,"props":1690,"children":1691},{},[1692,1741,1774,1806,1816],{"type":40,"tag":509,"props":1693,"children":1694},{},[1695,1708,1710,1716,1718,1724,1726,1732,1733,1739],{"type":40,"tag":55,"props":1696,"children":1697},{},[1698,1700,1706],{"type":46,"value":1699},"Every long-running server gets a ",{"type":40,"tag":63,"props":1701,"children":1703},{"className":1702},[],[1704],{"type":46,"value":1705},"readiness",{"type":46,"value":1707}," probe",{"type":46,"value":1709}," so dependents wait for real up-ness:\n",{"type":40,"tag":63,"props":1711,"children":1713},{"className":1712},[],[1714],{"type":46,"value":1715},"log_watch",{"type":46,"value":1717}," (server logs a ready line), ",{"type":40,"tag":63,"props":1719,"children":1721},{"className":1720},[],[1722],{"type":46,"value":1723},"tcp_port",{"type":46,"value":1725}," (port open), or ",{"type":40,"tag":63,"props":1727,"children":1729},{"className":1728},[],[1730],{"type":46,"value":1731},"http_get",{"type":46,"value":452},{"type":40,"tag":63,"props":1734,"children":1736},{"className":1735},[],[1737],{"type":46,"value":1738},"http_post",{"type":46,"value":1740},"\n(health endpoint).",{"type":40,"tag":509,"props":1742,"children":1743},{},[1744,1756,1758,1764,1766,1772],{"type":40,"tag":55,"props":1745,"children":1746},{},[1747,1749,1755],{"type":46,"value":1748},"Add a ",{"type":40,"tag":63,"props":1750,"children":1752},{"className":1751},[],[1753],{"type":46,"value":1754},"failure",{"type":46,"value":1707},{"type":46,"value":1757}," on servers — ",{"type":40,"tag":63,"props":1759,"children":1761},{"className":1760},[],[1762],{"type":46,"value":1763},"failure.log_watch",{"type":46,"value":1765}," for\n",{"type":40,"tag":63,"props":1767,"children":1769},{"className":1768},[],[1770],{"type":46,"value":1771},"\"Traceback (most recent call last)\"",{"type":46,"value":1773}," fails the run fast instead of hanging until timeout.",{"type":40,"tag":509,"props":1775,"children":1776},{},[1777,1782,1784,1789,1791,1797,1798,1804],{"type":40,"tag":63,"props":1778,"children":1780},{"className":1779},[],[1781],{"type":46,"value":1715},{"type":46,"value":1783}," matches ",{"type":40,"tag":55,"props":1785,"children":1786},{},[1787],{"type":46,"value":1788},"literally",{"type":46,"value":1790}," by default (parentheses\u002Fdots aren't special); prefix\n",{"type":40,"tag":63,"props":1792,"children":1794},{"className":1793},[],[1795],{"type":46,"value":1796},"re:",{"type":46,"value":70},{"type":40,"tag":63,"props":1799,"children":1801},{"className":1800},[],[1802],{"type":46,"value":1803},"regex:",{"type":46,"value":1805}," for real regex.",{"type":40,"tag":509,"props":1807,"children":1808},{},[1809,1814],{"type":40,"tag":55,"props":1810,"children":1811},{},[1812],{"type":46,"value":1813},"Don't probe short-lived one-shot tasks",{"type":46,"value":1815}," (benchmarks, setup steps).",{"type":40,"tag":509,"props":1817,"children":1818},{},[1819,1821,1827,1829,1835,1837,1843,1845,1851,1853,1859],{"type":46,"value":1820},"Defaults: ",{"type":40,"tag":63,"props":1822,"children":1824},{"className":1823},[],[1825],{"type":46,"value":1826},"timeout",{"type":46,"value":1828}," 1200 s (readiness deadline only), ",{"type":40,"tag":63,"props":1830,"children":1832},{"className":1831},[],[1833],{"type":46,"value":1834},"interval",{"type":46,"value":1836}," 5, ",{"type":40,"tag":63,"props":1838,"children":1840},{"className":1839},[],[1841],{"type":46,"value":1842},"each_check_timeout",{"type":46,"value":1844},"\n30, ",{"type":40,"tag":63,"props":1846,"children":1848},{"className":1847},[],[1849],{"type":46,"value":1850},"success_threshold",{"type":46,"value":1852}," 1, ",{"type":40,"tag":63,"props":1854,"children":1856},{"className":1855},[],[1857],{"type":46,"value":1858},"failure_threshold",{"type":46,"value":1860}," 3.",{"type":40,"tag":173,"props":1862,"children":1864},{"className":175,"code":1863,"language":18,"meta":177,"style":177},"- name: frontend\n  depends_on: [etcd]\n  script: [ \"python -m my.frontend --port 8000\" ]\n  probes:\n    readiness: { tcp_port: { port: 8000 } }\n    failure:   { log_watch: { regex_pattern: \"Traceback (most recent call last)\" } }\n",[1865],{"type":40,"tag":63,"props":1866,"children":1867},{"__ignoreMap":177},[1868,1889,1912,1946,1958,2009],{"type":40,"tag":183,"props":1869,"children":1870},{"class":185,"line":186},[1871,1876,1880,1884],{"type":40,"tag":183,"props":1872,"children":1873},{"style":196},[1874],{"type":46,"value":1875},"-",{"type":40,"tag":183,"props":1877,"children":1878},{"style":190},[1879],{"type":46,"value":723},{"type":40,"tag":183,"props":1881,"children":1882},{"style":196},[1883],{"type":46,"value":199},{"type":40,"tag":183,"props":1885,"children":1886},{"style":207},[1887],{"type":46,"value":1888}," frontend\n",{"type":40,"tag":183,"props":1890,"children":1891},{"class":185,"line":224},[1892,1896,1900,1904,1908],{"type":40,"tag":183,"props":1893,"children":1894},{"style":190},[1895],{"type":46,"value":771},{"type":40,"tag":183,"props":1897,"children":1898},{"style":196},[1899],{"type":46,"value":199},{"type":40,"tag":183,"props":1901,"children":1902},{"style":196},[1903],{"type":46,"value":273},{"type":40,"tag":183,"props":1905,"children":1906},{"style":207},[1907],{"type":46,"value":524},{"type":40,"tag":183,"props":1909,"children":1910},{"style":196},[1911],{"type":46,"value":1116},{"type":40,"tag":183,"props":1913,"children":1914},{"class":185,"line":258},[1915,1920,1924,1928,1932,1937,1941],{"type":40,"tag":183,"props":1916,"children":1917},{"style":190},[1918],{"type":46,"value":1919},"  script",{"type":40,"tag":183,"props":1921,"children":1922},{"style":196},[1923],{"type":46,"value":199},{"type":40,"tag":183,"props":1925,"children":1926},{"style":196},[1927],{"type":46,"value":273},{"type":40,"tag":183,"props":1929,"children":1930},{"style":196},[1931],{"type":46,"value":204},{"type":40,"tag":183,"props":1933,"children":1934},{"style":207},[1935],{"type":46,"value":1936},"python -m my.frontend --port 8000",{"type":40,"tag":183,"props":1938,"children":1939},{"style":196},[1940],{"type":46,"value":215},{"type":40,"tag":183,"props":1942,"children":1943},{"style":196},[1944],{"type":46,"value":1945}," ]\n",{"type":40,"tag":183,"props":1947,"children":1948},{"class":185,"line":290},[1949,1954],{"type":40,"tag":183,"props":1950,"children":1951},{"style":190},[1952],{"type":46,"value":1953},"  probes",{"type":40,"tag":183,"props":1955,"children":1956},{"style":196},[1957],{"type":46,"value":392},{"type":40,"tag":183,"props":1959,"children":1960},{"class":185,"line":321},[1961,1966,1970,1974,1979,1983,1987,1992,1996,2001,2005],{"type":40,"tag":183,"props":1962,"children":1963},{"style":190},[1964],{"type":46,"value":1965},"    readiness",{"type":40,"tag":183,"props":1967,"children":1968},{"style":196},[1969],{"type":46,"value":199},{"type":40,"tag":183,"props":1971,"children":1972},{"style":196},[1973],{"type":46,"value":239},{"type":40,"tag":183,"props":1975,"children":1976},{"style":190},[1977],{"type":46,"value":1978}," tcp_port",{"type":40,"tag":183,"props":1980,"children":1981},{"style":196},[1982],{"type":46,"value":199},{"type":40,"tag":183,"props":1984,"children":1985},{"style":196},[1986],{"type":46,"value":239},{"type":40,"tag":183,"props":1988,"children":1989},{"style":190},[1990],{"type":46,"value":1991}," port",{"type":40,"tag":183,"props":1993,"children":1994},{"style":196},[1995],{"type":46,"value":199},{"type":40,"tag":183,"props":1997,"children":1998},{"style":242},[1999],{"type":46,"value":2000}," 8000",{"type":40,"tag":183,"props":2002,"children":2003},{"style":196},[2004],{"type":46,"value":250},{"type":40,"tag":183,"props":2006,"children":2007},{"style":196},[2008],{"type":46,"value":737},{"type":40,"tag":183,"props":2010,"children":2011},{"class":185,"line":27},[2012,2017,2021,2026,2031,2035,2039,2044,2048,2052,2057,2061,2065],{"type":40,"tag":183,"props":2013,"children":2014},{"style":190},[2015],{"type":46,"value":2016},"    failure",{"type":40,"tag":183,"props":2018,"children":2019},{"style":196},[2020],{"type":46,"value":199},{"type":40,"tag":183,"props":2022,"children":2023},{"style":196},[2024],{"type":46,"value":2025},"   {",{"type":40,"tag":183,"props":2027,"children":2028},{"style":190},[2029],{"type":46,"value":2030}," log_watch",{"type":40,"tag":183,"props":2032,"children":2033},{"style":196},[2034],{"type":46,"value":199},{"type":40,"tag":183,"props":2036,"children":2037},{"style":196},[2038],{"type":46,"value":239},{"type":40,"tag":183,"props":2040,"children":2041},{"style":190},[2042],{"type":46,"value":2043}," regex_pattern",{"type":40,"tag":183,"props":2045,"children":2046},{"style":196},[2047],{"type":46,"value":199},{"type":40,"tag":183,"props":2049,"children":2050},{"style":196},[2051],{"type":46,"value":204},{"type":40,"tag":183,"props":2053,"children":2054},{"style":207},[2055],{"type":46,"value":2056},"Traceback (most recent call last)",{"type":40,"tag":183,"props":2058,"children":2059},{"style":196},[2060],{"type":46,"value":215},{"type":40,"tag":183,"props":2062,"children":2063},{"style":196},[2064],{"type":46,"value":250},{"type":40,"tag":183,"props":2066,"children":2067},{"style":196},[2068],{"type":46,"value":737},{"type":40,"tag":49,"props":2070,"children":2071},{},[2072,2074,2078],{"type":46,"value":2073},"(On ",{"type":40,"tag":55,"props":2075,"children":2076},{},[2077],{"type":46,"value":90},{"type":46,"value":2079},", tcp\u002Fhttp probes run from an in-cluster probe pod automatically — the\ndriver host usually can't route to the pod network.) See the probes doc.",{"type":40,"tag":153,"props":2081,"children":2083},{"id":2082},"step-5-scale-with-replicas-sweeps",[2084],{"type":46,"value":2085},"Step 5 — Scale with replicas & sweeps",{"type":40,"tag":924,"props":2087,"children":2088},{},[2089,2122],{"type":40,"tag":509,"props":2090,"children":2091},{},[2092,2097,2098,2104,2106,2112,2114,2120],{"type":40,"tag":55,"props":2093,"children":2094},{},[2095],{"type":46,"value":2096},"Replicate a task:",{"type":46,"value":1010},{"type":40,"tag":63,"props":2099,"children":2101},{"className":2100},[],[2102],{"type":46,"value":2103},"replicas: { count: N, policy: parallel|sequential }",{"type":46,"value":2105},".\n",{"type":40,"tag":63,"props":2107,"children":2109},{"className":2108},[],[2110],{"type":46,"value":2111},"parallel",{"type":46,"value":2113}," = all at once (downstream waits for all); ",{"type":40,"tag":63,"props":2115,"children":2117},{"className":2116},[],[2118],{"type":46,"value":2119},"sequential",{"type":46,"value":2121}," = one-by-one (sweeps).",{"type":40,"tag":509,"props":2123,"children":2124},{},[2125,2130,2131,2137,2139,2145,2147,2153,2155,2161,2163,2169],{"type":40,"tag":55,"props":2126,"children":2127},{},[2128],{"type":46,"value":2129},"Parameter sweep:",{"type":46,"value":1010},{"type":40,"tag":63,"props":2132,"children":2134},{"className":2133},[],[2135],{"type":46,"value":2136},"replicas.variables: [X]",{"type":46,"value":2138}," where ",{"type":40,"tag":63,"props":2140,"children":2142},{"className":2141},[],[2143],{"type":46,"value":2144},"X",{"type":46,"value":2146}," declares a ",{"type":40,"tag":63,"props":2148,"children":2150},{"className":2149},[],[2151],{"type":46,"value":2152},"domain:",{"type":46,"value":2154}," → one replica\nper value (Cartesian product across multiple swept vars). Each replica gets\n",{"type":40,"tag":63,"props":2156,"children":2158},{"className":2157},[],[2159],{"type":46,"value":2160},"SFLOW_REPLICA_INDEX",{"type":46,"value":2162}," and the swept values in its env; omit ",{"type":40,"tag":63,"props":2164,"children":2166},{"className":2165},[],[2167],{"type":46,"value":2168},"count",{"type":46,"value":2170}," and it equals the sweep\nsize.",{"type":40,"tag":49,"props":2172,"children":2173},{},[2174],{"type":46,"value":2175},"See examples.md Example 5; the replicas doc.",{"type":40,"tag":153,"props":2177,"children":2179},{"id":2178},"step-6-place-on-nodes-gpus",[2180],{"type":46,"value":2181},"Step 6 — Place on nodes & GPUs",{"type":40,"tag":924,"props":2183,"children":2184},{},[2185,2235,2275,2307],{"type":40,"tag":509,"props":2186,"children":2187},{},[2188,2193,2194,2200,2201,2207,2208,2213,2214,2220,2222,2227,2228,2234],{"type":40,"tag":55,"props":2189,"children":2190},{},[2191],{"type":46,"value":2192},"Pin placement",{"type":46,"value":566},{"type":40,"tag":63,"props":2195,"children":2197},{"className":2196},[],[2198],{"type":46,"value":2199},"resources.nodes",{"type":46,"value":1034},{"type":40,"tag":63,"props":2202,"children":2204},{"className":2203},[],[2205],{"type":46,"value":2206},"indices",{"type":46,"value":70},{"type":40,"tag":63,"props":2209,"children":2211},{"className":2210},[],[2212],{"type":46,"value":2168},{"type":46,"value":70},{"type":40,"tag":63,"props":2215,"children":2217},{"className":2216},[],[2218],{"type":46,"value":2219},"exclude",{"type":46,"value":2221},"; negative indices\ncount from the end). ",{"type":40,"tag":55,"props":2223,"children":2224},{},[2225],{"type":46,"value":2226},"Request GPUs",{"type":46,"value":566},{"type":40,"tag":63,"props":2229,"children":2231},{"className":2230},[],[2232],{"type":46,"value":2233},"resources.gpus.count",{"type":46,"value":151},{"type":40,"tag":509,"props":2236,"children":2237},{},[2238,2240,2245,2246,2251,2252,2257,2259,2265,2267,2273],{"type":46,"value":2239},"On ",{"type":40,"tag":63,"props":2241,"children":2243},{"className":2242},[],[2244],{"type":46,"value":68},{"type":46,"value":452},{"type":40,"tag":63,"props":2247,"children":2249},{"className":2248},[],[2250],{"type":46,"value":76},{"type":46,"value":452},{"type":40,"tag":63,"props":2253,"children":2255},{"className":2254},[],[2256],{"type":46,"value":83},{"type":46,"value":2258},", sflow slices ",{"type":40,"tag":63,"props":2260,"children":2262},{"className":2261},[],[2263],{"type":46,"value":2264},"CUDA_VISIBLE_DEVICES",{"type":46,"value":2266}," per task\u002Freplica — set the\nbackend's ",{"type":40,"tag":63,"props":2268,"children":2270},{"className":2269},[],[2271],{"type":46,"value":2272},"gpus_per_node",{"type":46,"value":2274}," so it can pack.",{"type":40,"tag":509,"props":2276,"children":2277},{},[2278,2283,2284,2290,2292,2298,2299,2305],{"type":40,"tag":55,"props":2279,"children":2280},{},[2281],{"type":46,"value":2282},"GPU planning:",{"type":46,"value":1010},{"type":40,"tag":63,"props":2285,"children":2287},{"className":2286},[],[2288],{"type":46,"value":2289},"GPUs\u002Fworker = TP × DP × PP",{"type":46,"value":2291}," (framework-dependent — some use attention-DP\n\u002F expert parallelism), ",{"type":40,"tag":63,"props":2293,"children":2295},{"className":2294},[],[2296],{"type":46,"value":2297},"total = GPUs\u002Fworker × replicas",{"type":46,"value":526},{"type":40,"tag":63,"props":2300,"children":2302},{"className":2301},[],[2303],{"type":46,"value":2304},"nodes = ceil(total \u002F gpus_per_node)",{"type":46,"value":2306},".\nWhen unsure of a framework's parallelism, ask the user.",{"type":40,"tag":509,"props":2308,"children":2309},{},[2310,2312,2318,2319,2325,2326,2332,2333,2339],{"type":46,"value":2311},"Use ",{"type":40,"tag":63,"props":2313,"children":2315},{"className":2314},[],[2316],{"type":46,"value":2317},"resources.*.release_after",{"type":46,"value":1034},{"type":40,"tag":63,"props":2320,"children":2322},{"className":2321},[],[2323],{"type":46,"value":2324},"task_ready",{"type":46,"value":70},{"type":40,"tag":63,"props":2327,"children":2329},{"className":2328},[],[2330],{"type":46,"value":2331},"task_completion",{"type":46,"value":70},{"type":40,"tag":63,"props":2334,"children":2336},{"className":2335},[],[2337],{"type":46,"value":2338},"workflow_completion",{"type":46,"value":2340},")\nto let later tasks reuse a reservation.",{"type":40,"tag":49,"props":2342,"children":2343},{},[2344],{"type":46,"value":2345},"See the resources doc.",{"type":40,"tag":153,"props":2347,"children":2349},{"id":2348},"step-7-backend-specific-handling",[2350],{"type":46,"value":2351},"Step 7 — Backend-specific handling",{"type":40,"tag":49,"props":2353,"children":2354},{},[2355,2357,2362,2364,2369],{"type":46,"value":2356},"Pick a backend and apply ",{"type":40,"tag":101,"props":2358,"children":2359},{},[2360],{"type":46,"value":2361},"its",{"type":46,"value":2363}," rules. Start ",{"type":40,"tag":63,"props":2365,"children":2367},{"className":2366},[],[2368],{"type":46,"value":68},{"type":46,"value":2370},", move to a cluster once the DAG is proven.",{"type":40,"tag":49,"props":2372,"children":2373},{},[2374,2382,2384,2389,2391,2397],{"type":40,"tag":55,"props":2375,"children":2376},{},[2377],{"type":40,"tag":63,"props":2378,"children":2380},{"className":2379},[],[2381],{"type":46,"value":68},{"type":46,"value":2383}," — no config needed (default). ",{"type":40,"tag":63,"props":2385,"children":2387},{"className":2386},[],[2388],{"type":46,"value":472},{"type":46,"value":2390}," operator. Set ",{"type":40,"tag":63,"props":2392,"children":2394},{"className":2393},[],[2395],{"type":46,"value":2396},"nodes: N",{"type":46,"value":2398}," to simulate a\nmulti-node placement on one machine.",{"type":40,"tag":49,"props":2400,"children":2401},{},[2402,2410,2412,2418,2420,2426,2427,2433,2434,2440,2442,2448,2449,2454,2456,2462,2464,2470,2472,2477,2479,2484,2486,2491,2493,2497,2499,2504],{"type":40,"tag":55,"props":2403,"children":2404},{},[2405],{"type":40,"tag":63,"props":2406,"children":2408},{"className":2407},[],[2409],{"type":46,"value":76},{"type":46,"value":2411}," — default operator ",{"type":40,"tag":63,"props":2413,"children":2415},{"className":2414},[],[2416],{"type":46,"value":2417},"srun",{"type":46,"value":2419},". Backend needs ",{"type":40,"tag":63,"props":2421,"children":2423},{"className":2422},[],[2424],{"type":46,"value":2425},"partition",{"type":46,"value":526},{"type":40,"tag":63,"props":2428,"children":2430},{"className":2429},[],[2431],{"type":46,"value":2432},"account",{"type":46,"value":526},{"type":40,"tag":63,"props":2435,"children":2437},{"className":2436},[],[2438],{"type":46,"value":2439},"time",{"type":46,"value":2441},"\n(HH:MM:SS or int minutes), ",{"type":40,"tag":63,"props":2443,"children":2445},{"className":2444},[],[2446],{"type":46,"value":2447},"nodes",{"type":46,"value":526},{"type":40,"tag":63,"props":2450,"children":2452},{"className":2451},[],[2453],{"type":46,"value":2272},{"type":46,"value":2455},". Containers: put ",{"type":40,"tag":63,"props":2457,"children":2459},{"className":2458},[],[2460],{"type":46,"value":2461},"container_image",{"type":46,"value":2463}," \u002F\n",{"type":40,"tag":63,"props":2465,"children":2467},{"className":2466},[],[2468],{"type":46,"value":2469},"container_name",{"type":46,"value":2471}," on an ",{"type":40,"tag":63,"props":2473,"children":2475},{"className":2474},[],[2476],{"type":46,"value":2417},{"type":46,"value":2478}," operator (reuse the image across tasks by ",{"type":40,"tag":63,"props":2480,"children":2482},{"className":2481},[],[2483],{"type":46,"value":2469},{"type":46,"value":2485},").\nKeep models \u002F workspace \u002F ",{"type":40,"tag":55,"props":2487,"children":2488},{},[2489],{"type":46,"value":2490},"output dir",{"type":46,"value":2492}," on ",{"type":40,"tag":55,"props":2494,"children":2495},{},[2496],{"type":46,"value":1566},{"type":46,"value":2498}," (Lustre\u002FGPFS\u002FNFS) so an\n",{"type":40,"tag":63,"props":2500,"children":2502},{"className":2501},[],[2503],{"type":46,"value":1538},{"type":46,"value":2505}," artifact auto-mounts at the same path on every node.",{"type":40,"tag":173,"props":2507,"children":2509},{"className":175,"code":2508,"language":18,"meta":177,"style":177},"backends:\n  - name: slurm\n    type: slurm\n    default: true\n    nodes: 1\n    partition: ${{ variables.PART }}      # block style: ${{ }} values can't sit in a { } flow map\n    account: ${{ variables.ACCT }}\n    time: \"01:00:00\"                       # HH:MM:SS or int minutes\n    gpus_per_node: 8\noperators:\n  - { name: ctr, type: srun, container_image: nvcr.io\u002Fnvidia\u002Fpytorch:24.07-py3,\n      container_writable: true, mpi: pmix }\n",[2510],{"type":40,"tag":63,"props":2511,"children":2512},{"__ignoreMap":177},[2513,2524,2544,2560,2578,2595,2617,2634,2664,2681,2692,2756],{"type":40,"tag":183,"props":2514,"children":2515},{"class":185,"line":186},[2516,2520],{"type":40,"tag":183,"props":2517,"children":2518},{"style":190},[2519],{"type":46,"value":296},{"type":40,"tag":183,"props":2521,"children":2522},{"style":196},[2523],{"type":46,"value":392},{"type":40,"tag":183,"props":2525,"children":2526},{"class":185,"line":224},[2527,2531,2535,2539],{"type":40,"tag":183,"props":2528,"children":2529},{"style":196},[2530],{"type":46,"value":1326},{"type":40,"tag":183,"props":2532,"children":2533},{"style":190},[2534],{"type":46,"value":723},{"type":40,"tag":183,"props":2536,"children":2537},{"style":196},[2538],{"type":46,"value":199},{"type":40,"tag":183,"props":2540,"children":2541},{"style":207},[2542],{"type":46,"value":2543}," slurm\n",{"type":40,"tag":183,"props":2545,"children":2546},{"class":185,"line":258},[2547,2552,2556],{"type":40,"tag":183,"props":2548,"children":2549},{"style":190},[2550],{"type":46,"value":2551},"    type",{"type":40,"tag":183,"props":2553,"children":2554},{"style":196},[2555],{"type":46,"value":199},{"type":40,"tag":183,"props":2557,"children":2558},{"style":207},[2559],{"type":46,"value":2543},{"type":40,"tag":183,"props":2561,"children":2562},{"class":185,"line":290},[2563,2568,2572],{"type":40,"tag":183,"props":2564,"children":2565},{"style":190},[2566],{"type":46,"value":2567},"    default",{"type":40,"tag":183,"props":2569,"children":2570},{"style":196},[2571],{"type":46,"value":199},{"type":40,"tag":183,"props":2573,"children":2575},{"style":2574},"--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC",[2576],{"type":46,"value":2577}," true\n",{"type":40,"tag":183,"props":2579,"children":2580},{"class":185,"line":321},[2581,2586,2590],{"type":40,"tag":183,"props":2582,"children":2583},{"style":190},[2584],{"type":46,"value":2585},"    nodes",{"type":40,"tag":183,"props":2587,"children":2588},{"style":196},[2589],{"type":46,"value":199},{"type":40,"tag":183,"props":2591,"children":2592},{"style":242},[2593],{"type":46,"value":2594}," 1\n",{"type":40,"tag":183,"props":2596,"children":2597},{"class":185,"line":27},[2598,2603,2607,2612],{"type":40,"tag":183,"props":2599,"children":2600},{"style":190},[2601],{"type":46,"value":2602},"    partition",{"type":40,"tag":183,"props":2604,"children":2605},{"style":196},[2606],{"type":46,"value":199},{"type":40,"tag":183,"props":2608,"children":2609},{"style":207},[2610],{"type":46,"value":2611}," ${{ variables.PART }}",{"type":40,"tag":183,"props":2613,"children":2614},{"style":218},[2615],{"type":46,"value":2616},"      # block style: ${{ }} values can't sit in a { } flow map\n",{"type":40,"tag":183,"props":2618,"children":2619},{"class":185,"line":381},[2620,2625,2629],{"type":40,"tag":183,"props":2621,"children":2622},{"style":190},[2623],{"type":46,"value":2624},"    account",{"type":40,"tag":183,"props":2626,"children":2627},{"style":196},[2628],{"type":46,"value":199},{"type":40,"tag":183,"props":2630,"children":2631},{"style":207},[2632],{"type":46,"value":2633}," ${{ variables.ACCT }}\n",{"type":40,"tag":183,"props":2635,"children":2636},{"class":185,"line":395},[2637,2642,2646,2650,2655,2659],{"type":40,"tag":183,"props":2638,"children":2639},{"style":190},[2640],{"type":46,"value":2641},"    time",{"type":40,"tag":183,"props":2643,"children":2644},{"style":196},[2645],{"type":46,"value":199},{"type":40,"tag":183,"props":2647,"children":2648},{"style":196},[2649],{"type":46,"value":204},{"type":40,"tag":183,"props":2651,"children":2652},{"style":207},[2653],{"type":46,"value":2654},"01:00:00",{"type":40,"tag":183,"props":2656,"children":2657},{"style":196},[2658],{"type":46,"value":215},{"type":40,"tag":183,"props":2660,"children":2661},{"style":218},[2662],{"type":46,"value":2663},"                       # HH:MM:SS or int minutes\n",{"type":40,"tag":183,"props":2665,"children":2666},{"class":185,"line":413},[2667,2672,2676],{"type":40,"tag":183,"props":2668,"children":2669},{"style":190},[2670],{"type":46,"value":2671},"    gpus_per_node",{"type":40,"tag":183,"props":2673,"children":2674},{"style":196},[2675],{"type":46,"value":199},{"type":40,"tag":183,"props":2677,"children":2678},{"style":242},[2679],{"type":46,"value":2680}," 8\n",{"type":40,"tag":183,"props":2682,"children":2683},{"class":185,"line":1449},[2684,2688],{"type":40,"tag":183,"props":2685,"children":2686},{"style":190},[2687],{"type":46,"value":327},{"type":40,"tag":183,"props":2689,"children":2690},{"style":196},[2691],{"type":46,"value":392},{"type":40,"tag":183,"props":2693,"children":2694},{"class":185,"line":1458},[2695,2699,2703,2707,2711,2716,2720,2725,2729,2734,2738,2743,2747,2752],{"type":40,"tag":183,"props":2696,"children":2697},{"style":196},[2698],{"type":46,"value":1326},{"type":40,"tag":183,"props":2700,"children":2701},{"style":196},[2702],{"type":46,"value":239},{"type":40,"tag":183,"props":2704,"children":2705},{"style":190},[2706],{"type":46,"value":723},{"type":40,"tag":183,"props":2708,"children":2709},{"style":196},[2710],{"type":46,"value":199},{"type":40,"tag":183,"props":2712,"children":2713},{"style":207},[2714],{"type":46,"value":2715}," ctr",{"type":40,"tag":183,"props":2717,"children":2718},{"style":196},[2719],{"type":46,"value":766},{"type":40,"tag":183,"props":2721,"children":2722},{"style":190},[2723],{"type":46,"value":2724}," type",{"type":40,"tag":183,"props":2726,"children":2727},{"style":196},[2728],{"type":46,"value":199},{"type":40,"tag":183,"props":2730,"children":2731},{"style":207},[2732],{"type":46,"value":2733}," srun",{"type":40,"tag":183,"props":2735,"children":2736},{"style":196},[2737],{"type":46,"value":766},{"type":40,"tag":183,"props":2739,"children":2740},{"style":190},[2741],{"type":46,"value":2742}," container_image",{"type":40,"tag":183,"props":2744,"children":2745},{"style":196},[2746],{"type":46,"value":199},{"type":40,"tag":183,"props":2748,"children":2749},{"style":207},[2750],{"type":46,"value":2751}," nvcr.io\u002Fnvidia\u002Fpytorch:24.07-py3",{"type":40,"tag":183,"props":2753,"children":2754},{"style":196},[2755],{"type":46,"value":534},{"type":40,"tag":183,"props":2757,"children":2758},{"class":185,"line":1470},[2759,2764,2768,2773,2777,2782,2786,2791],{"type":40,"tag":183,"props":2760,"children":2761},{"style":190},[2762],{"type":46,"value":2763},"      container_writable",{"type":40,"tag":183,"props":2765,"children":2766},{"style":196},[2767],{"type":46,"value":199},{"type":40,"tag":183,"props":2769,"children":2770},{"style":2574},[2771],{"type":46,"value":2772}," true",{"type":40,"tag":183,"props":2774,"children":2775},{"style":196},[2776],{"type":46,"value":766},{"type":40,"tag":183,"props":2778,"children":2779},{"style":190},[2780],{"type":46,"value":2781}," mpi",{"type":40,"tag":183,"props":2783,"children":2784},{"style":196},[2785],{"type":46,"value":199},{"type":40,"tag":183,"props":2787,"children":2788},{"style":207},[2789],{"type":46,"value":2790}," pmix",{"type":40,"tag":183,"props":2792,"children":2793},{"style":196},[2794],{"type":46,"value":737},{"type":40,"tag":49,"props":2796,"children":2797},{},[2798,2806,2808,2814,2816,2822,2824,2830,2832,2838],{"type":40,"tag":55,"props":2799,"children":2800},{},[2801],{"type":40,"tag":63,"props":2802,"children":2804},{"className":2803},[],[2805],{"type":46,"value":83},{"type":46,"value":2807}," — backend is ",{"type":40,"tag":63,"props":2809,"children":2811},{"className":2810},[],[2812],{"type":46,"value":2813},"type: docker",{"type":46,"value":2815},", its launch operator is ",{"type":40,"tag":63,"props":2817,"children":2819},{"className":2818},[],[2820],{"type":46,"value":2821},"type: docker_run",{"type":46,"value":2823}," (mind\nthe naming). The ",{"type":40,"tag":63,"props":2825,"children":2827},{"className":2826},[],[2828],{"type":46,"value":2829},"image",{"type":46,"value":2831}," can live on the backend or the operator. Remote\u002Fmulti-host via a\n",{"type":40,"tag":63,"props":2833,"children":2835},{"className":2834},[],[2836],{"type":46,"value":2837},"hosts:",{"type":46,"value":2839}," list.",{"type":40,"tag":49,"props":2841,"children":2842},{},[2843,2851,2853,2871],{"type":40,"tag":55,"props":2844,"children":2845},{},[2846],{"type":40,"tag":63,"props":2847,"children":2849},{"className":2848},[],[2850],{"type":46,"value":90},{"type":46,"value":2852}," — the workload ",{"type":40,"tag":55,"props":2854,"children":2855},{},[2856,2861,2863,2869],{"type":40,"tag":63,"props":2857,"children":2859},{"className":2858},[],[2860],{"type":46,"value":2829},{"type":46,"value":2862}," lives on the ",{"type":40,"tag":63,"props":2864,"children":2866},{"className":2865},[],[2867],{"type":46,"value":2868},"k8s",{"type":46,"value":2870}," operator",{"type":46,"value":2872},", never the backend\n(the backend reserves + pins nodes with placeholder pods). Rules that bite:",{"type":40,"tag":924,"props":2874,"children":2875},{},[2876,2917,2970,2982,3034],{"type":40,"tag":509,"props":2877,"children":2878},{},[2879,2881,2886,2887,2893,2894,2900,2901,2907,2909,2915],{"type":46,"value":2880},"Cluster access is ",{"type":40,"tag":55,"props":2882,"children":2883},{},[2884],{"type":46,"value":2885},"CLI-only",{"type":46,"value":1034},{"type":40,"tag":63,"props":2888,"children":2890},{"className":2889},[],[2891],{"type":46,"value":2892},"--kubeconfig",{"type":46,"value":70},{"type":40,"tag":63,"props":2895,"children":2897},{"className":2896},[],[2898],{"type":46,"value":2899},"--kube-context",{"type":46,"value":70},{"type":40,"tag":63,"props":2902,"children":2904},{"className":2903},[],[2905],{"type":46,"value":2906},"--kube-namespace",{"type":46,"value":2908},") so\nrecipes stay cluster-agnostic; ",{"type":40,"tag":63,"props":2910,"children":2912},{"className":2911},[],[2913],{"type":46,"value":2914},"namespace",{"type":46,"value":2916}," is a backend field (one namespace per backend).",{"type":40,"tag":509,"props":2918,"children":2919},{},[2920,2925,2927,2932,2934,2939,2941,2946,2947,2953,2955,2961,2963,2968],{"type":40,"tag":63,"props":2921,"children":2923},{"className":2922},[],[2924],{"type":46,"value":2233},{"type":46,"value":2926}," is a per-task ",{"type":40,"tag":55,"props":2928,"children":2929},{},[2930],{"type":46,"value":2931},"total",{"type":46,"value":2933},", split evenly across the task's nodes → must\nbe a multiple of the node count. For an ordinary pod sflow leaves ",{"type":40,"tag":63,"props":2935,"children":2937},{"className":2936},[],[2938],{"type":46,"value":2264},{"type":46,"value":2940},"\nunset (the device-plugin\u002FDRA assigns the GPUs); when co-located GPU tasks are ",{"type":40,"tag":55,"props":2942,"children":2943},{},[2944],{"type":46,"value":2945},"merged into\none pod",{"type":46,"value":1034},{"type":40,"tag":63,"props":2948,"children":2950},{"className":2949},[],[2951],{"type":46,"value":2952},"merge_colocated_gpu_pods",{"type":46,"value":2954},", default ",{"type":40,"tag":63,"props":2956,"children":2958},{"className":2957},[],[2959],{"type":46,"value":2960},"auto",{"type":46,"value":2962},") it sets a per-member\n",{"type":40,"tag":63,"props":2964,"children":2966},{"className":2965},[],[2967],{"type":46,"value":2264},{"type":46,"value":2969}," to pin each task's slice.",{"type":40,"tag":509,"props":2971,"children":2972},{},[2973,2975,2980],{"type":46,"value":2974},"A single-node task → one pod; a ",{"type":40,"tag":63,"props":2976,"children":2978},{"className":2977},[],[2979],{"type":46,"value":2199},{"type":46,"value":2981}," task → one pod per node (leader = index 0).",{"type":40,"tag":509,"props":2983,"children":2984},{},[2985,2990,2992,2997,2999,3004,3006,3011,3012,3016,3018,3024,3026,3032],{"type":40,"tag":55,"props":2986,"children":2987},{},[2988],{"type":46,"value":2989},"Cluster data → a PVC",{"type":46,"value":2991},": declare it under backend ",{"type":40,"tag":63,"props":2993,"children":2995},{"className":2994},[],[2996],{"type":46,"value":1590},{"type":46,"value":2998}," at a ",{"type":40,"tag":63,"props":3000,"children":3002},{"className":3001},[],[3003],{"type":46,"value":1598},{"type":46,"value":3005},", then point\nan ",{"type":40,"tag":63,"props":3007,"children":3009},{"className":3008},[],[3010],{"type":46,"value":1538},{"type":46,"value":1607},{"type":40,"tag":101,"props":3013,"children":3014},{},[3015],{"type":46,"value":1612},{"type":46,"value":3017}," that mount. Output dirs are ephemeral ",{"type":40,"tag":63,"props":3019,"children":3021},{"className":3020},[],[3022],{"type":46,"value":3023},"emptyDir",{"type":46,"value":3025}," — for\npersistence use a PVC or ",{"type":40,"tag":63,"props":3027,"children":3029},{"className":3028},[],[3030],{"type":46,"value":3031},"uploads:",{"type":46,"value":3033}," (Step 8).",{"type":40,"tag":509,"props":3035,"children":3036},{},[3037,3042,3043,3049,3051,3057,3059,3065,3067,3073,3074,3079,3081,3086,3088,3094,3096,3101],{"type":40,"tag":55,"props":3038,"children":3039},{},[3040],{"type":46,"value":3041},"Any MPI workload",{"type":46,"value":1034},{"type":40,"tag":63,"props":3044,"children":3046},{"className":3045},[],[3047],{"type":46,"value":3048},"mpirun",{"type":46,"value":3050},"-launched, e.g. TRT-LLM ",{"type":40,"tag":63,"props":3052,"children":3054},{"className":3053},[],[3055],{"type":46,"value":3056},"trtllm-llmapi-launch",{"type":46,"value":3058},") → ",{"type":40,"tag":63,"props":3060,"children":3062},{"className":3061},[],[3063],{"type":46,"value":3064},"type: k8s_mpi",{"type":46,"value":3066},",\nand write the ",{"type":40,"tag":63,"props":3068,"children":3070},{"className":3069},[],[3071],{"type":46,"value":3072},"mpirun -np N …",{"type":46,"value":1010},{"type":40,"tag":55,"props":3075,"children":3076},{},[3077],{"type":46,"value":3078},"explicitly",{"type":46,"value":3080}," in the script: sflow reads it to set up the MPI\nworld for ",{"type":40,"tag":55,"props":3082,"children":3083},{},[3084],{"type":46,"value":3085},"both single- and multi-node",{"type":46,"value":3087}," and injects the keypair\u002Fhostfile\u002Fsshd\u002F",{"type":40,"tag":63,"props":3089,"children":3091},{"className":3090},[],[3092],{"type":46,"value":3093},"-x",{"type":46,"value":3095}," env\nforwarding. Reserve plain ",{"type":40,"tag":63,"props":3097,"children":3099},{"className":3098},[],[3100],{"type":46,"value":2868},{"type":46,"value":3102}," for non-MPI, single-pod workloads.",{"type":40,"tag":173,"props":3104,"children":3106},{"className":175,"code":3105,"language":18,"meta":177,"style":177},"backends:\n  - { name: k8s, type: kubernetes, default: true, namespace: default, nodes: 1,\n      gpus_per_node: 8, scheduling: device_plugin }   # device_plugin default; dra is WIP\noperators:\n  - { name: gpu_op, type: k8s, image: nvcr.io\u002Fnvidia\u002Fpytorch:24.12-py3 }\n",[3107],{"type":40,"tag":63,"props":3108,"children":3109},{"__ignoreMap":177},[3110,3121,3218,3262,3273],{"type":40,"tag":183,"props":3111,"children":3112},{"class":185,"line":186},[3113,3117],{"type":40,"tag":183,"props":3114,"children":3115},{"style":190},[3116],{"type":46,"value":296},{"type":40,"tag":183,"props":3118,"children":3119},{"style":196},[3120],{"type":46,"value":392},{"type":40,"tag":183,"props":3122,"children":3123},{"class":185,"line":224},[3124,3128,3132,3136,3140,3145,3149,3153,3157,3162,3166,3171,3175,3179,3183,3188,3192,3196,3200,3205,3209,3214],{"type":40,"tag":183,"props":3125,"children":3126},{"style":196},[3127],{"type":46,"value":1326},{"type":40,"tag":183,"props":3129,"children":3130},{"style":196},[3131],{"type":46,"value":239},{"type":40,"tag":183,"props":3133,"children":3134},{"style":190},[3135],{"type":46,"value":723},{"type":40,"tag":183,"props":3137,"children":3138},{"style":196},[3139],{"type":46,"value":199},{"type":40,"tag":183,"props":3141,"children":3142},{"style":207},[3143],{"type":46,"value":3144}," k8s",{"type":40,"tag":183,"props":3146,"children":3147},{"style":196},[3148],{"type":46,"value":766},{"type":40,"tag":183,"props":3150,"children":3151},{"style":190},[3152],{"type":46,"value":2724},{"type":40,"tag":183,"props":3154,"children":3155},{"style":196},[3156],{"type":46,"value":199},{"type":40,"tag":183,"props":3158,"children":3159},{"style":207},[3160],{"type":46,"value":3161}," kubernetes",{"type":40,"tag":183,"props":3163,"children":3164},{"style":196},[3165],{"type":46,"value":766},{"type":40,"tag":183,"props":3167,"children":3168},{"style":190},[3169],{"type":46,"value":3170}," default",{"type":40,"tag":183,"props":3172,"children":3173},{"style":196},[3174],{"type":46,"value":199},{"type":40,"tag":183,"props":3176,"children":3177},{"style":2574},[3178],{"type":46,"value":2772},{"type":40,"tag":183,"props":3180,"children":3181},{"style":196},[3182],{"type":46,"value":766},{"type":40,"tag":183,"props":3184,"children":3185},{"style":190},[3186],{"type":46,"value":3187}," namespace",{"type":40,"tag":183,"props":3189,"children":3190},{"style":196},[3191],{"type":46,"value":199},{"type":40,"tag":183,"props":3193,"children":3194},{"style":207},[3195],{"type":46,"value":3170},{"type":40,"tag":183,"props":3197,"children":3198},{"style":196},[3199],{"type":46,"value":766},{"type":40,"tag":183,"props":3201,"children":3202},{"style":190},[3203],{"type":46,"value":3204}," nodes",{"type":40,"tag":183,"props":3206,"children":3207},{"style":196},[3208],{"type":46,"value":199},{"type":40,"tag":183,"props":3210,"children":3211},{"style":242},[3212],{"type":46,"value":3213}," 1",{"type":40,"tag":183,"props":3215,"children":3216},{"style":196},[3217],{"type":46,"value":534},{"type":40,"tag":183,"props":3219,"children":3220},{"class":185,"line":258},[3221,3226,3230,3235,3239,3244,3248,3253,3257],{"type":40,"tag":183,"props":3222,"children":3223},{"style":190},[3224],{"type":46,"value":3225},"      gpus_per_node",{"type":40,"tag":183,"props":3227,"children":3228},{"style":196},[3229],{"type":46,"value":199},{"type":40,"tag":183,"props":3231,"children":3232},{"style":242},[3233],{"type":46,"value":3234}," 8",{"type":40,"tag":183,"props":3236,"children":3237},{"style":196},[3238],{"type":46,"value":766},{"type":40,"tag":183,"props":3240,"children":3241},{"style":190},[3242],{"type":46,"value":3243}," scheduling",{"type":40,"tag":183,"props":3245,"children":3246},{"style":196},[3247],{"type":46,"value":199},{"type":40,"tag":183,"props":3249,"children":3250},{"style":207},[3251],{"type":46,"value":3252}," device_plugin",{"type":40,"tag":183,"props":3254,"children":3255},{"style":196},[3256],{"type":46,"value":250},{"type":40,"tag":183,"props":3258,"children":3259},{"style":218},[3260],{"type":46,"value":3261},"   # device_plugin default; dra is WIP\n",{"type":40,"tag":183,"props":3263,"children":3264},{"class":185,"line":290},[3265,3269],{"type":40,"tag":183,"props":3266,"children":3267},{"style":190},[3268],{"type":46,"value":327},{"type":40,"tag":183,"props":3270,"children":3271},{"style":196},[3272],{"type":46,"value":392},{"type":40,"tag":183,"props":3274,"children":3275},{"class":185,"line":321},[3276,3280,3284,3288,3292,3297,3301,3305,3309,3313,3317,3322,3326,3331],{"type":40,"tag":183,"props":3277,"children":3278},{"style":196},[3279],{"type":46,"value":1326},{"type":40,"tag":183,"props":3281,"children":3282},{"style":196},[3283],{"type":46,"value":239},{"type":40,"tag":183,"props":3285,"children":3286},{"style":190},[3287],{"type":46,"value":723},{"type":40,"tag":183,"props":3289,"children":3290},{"style":196},[3291],{"type":46,"value":199},{"type":40,"tag":183,"props":3293,"children":3294},{"style":207},[3295],{"type":46,"value":3296}," gpu_op",{"type":40,"tag":183,"props":3298,"children":3299},{"style":196},[3300],{"type":46,"value":766},{"type":40,"tag":183,"props":3302,"children":3303},{"style":190},[3304],{"type":46,"value":2724},{"type":40,"tag":183,"props":3306,"children":3307},{"style":196},[3308],{"type":46,"value":199},{"type":40,"tag":183,"props":3310,"children":3311},{"style":207},[3312],{"type":46,"value":3144},{"type":40,"tag":183,"props":3314,"children":3315},{"style":196},[3316],{"type":46,"value":766},{"type":40,"tag":183,"props":3318,"children":3319},{"style":190},[3320],{"type":46,"value":3321}," image",{"type":40,"tag":183,"props":3323,"children":3324},{"style":196},[3325],{"type":46,"value":199},{"type":40,"tag":183,"props":3327,"children":3328},{"style":207},[3329],{"type":46,"value":3330}," nvcr.io\u002Fnvidia\u002Fpytorch:24.12-py3",{"type":40,"tag":183,"props":3332,"children":3333},{"style":196},[3334],{"type":46,"value":737},{"type":40,"tag":49,"props":3336,"children":3337},{},[3338],{"type":46,"value":3339},"See the backends doc; examples.md Examples 8 (k8s), 10 (k8s + PVC), 11 (slurm + Lustre).",{"type":40,"tag":153,"props":3341,"children":3343},{"id":3342},"step-8-capture-outputs",[3344],{"type":46,"value":3345},"Step 8 — Capture outputs",{"type":40,"tag":924,"props":3347,"children":3348},{},[3349,3414,3469],{"type":40,"tag":509,"props":3350,"children":3351},{},[3352,3357,3359,3365,3367,3373,3375,3381,3383,3389,3391,3397,3399,3405,3407,3412],{"type":40,"tag":55,"props":3353,"children":3354},{},[3355],{"type":46,"value":3356},"Metrics:",{"type":46,"value":3358}," add ",{"type":40,"tag":63,"props":3360,"children":3362},{"className":3361},[],[3363],{"type":46,"value":3364},"result:",{"type":46,"value":3366}," (a regex map, ",{"type":40,"tag":63,"props":3368,"children":3370},{"className":3369},[],[3371],{"type":46,"value":3372},"patterns:",{"type":46,"value":3374},", or a JSON ",{"type":40,"tag":63,"props":3376,"children":3378},{"className":3377},[],[3379],{"type":46,"value":3380},"file:",{"type":46,"value":3382},") to parse a task's\nlog\u002Foutput into ",{"type":40,"tag":63,"props":3384,"children":3386},{"className":3385},[],[3387],{"type":46,"value":3388},"${SFLOW_TASK_OUTPUT_DIR}\u002Fresult.json",{"type":46,"value":3390}," plus a workflow ",{"type":40,"tag":63,"props":3392,"children":3394},{"className":3393},[],[3395],{"type":46,"value":3396},"results.json",{"type":46,"value":3398},".\nPrefer it over the legacy ",{"type":40,"tag":63,"props":3400,"children":3402},{"className":3401},[],[3403],{"type":46,"value":3404},"outputs:",{"type":46,"value":3406},". A task isn't ",{"type":40,"tag":63,"props":3408,"children":3410},{"className":3409},[],[3411],{"type":46,"value":586},{"type":46,"value":3413}," until parsing finishes.",{"type":40,"tag":509,"props":3415,"children":3416},{},[3417,3422,3424,3430,3432,3437,3439,3444,3446,3452,3454,3459,3461,3467],{"type":40,"tag":55,"props":3418,"children":3419},{},[3420],{"type":46,"value":3421},"Ship files:",{"type":46,"value":3423}," name ",{"type":40,"tag":63,"props":3425,"children":3427},{"className":3426},[],[3428],{"type":46,"value":3429},"storage:",{"type":46,"value":3431}," S3 targets, then per-task ",{"type":40,"tag":63,"props":3433,"children":3435},{"className":3434},[],[3436],{"type":46,"value":3031},{"type":46,"value":3438}," (fire on ",{"type":40,"tag":63,"props":3440,"children":3442},{"className":3441},[],[3443],{"type":46,"value":586},{"type":46,"value":3445},")\nor ",{"type":40,"tag":63,"props":3447,"children":3449},{"className":3448},[],[3450],{"type":46,"value":3451},"workflow.upload_all",{"type":46,"value":3453}," (zip the whole run). Credentials come from the ",{"type":40,"tag":55,"props":3455,"children":3456},{},[3457],{"type":46,"value":3458},"boto3 default\nchain",{"type":46,"value":3460}," — never put secrets in YAML; needs the ",{"type":40,"tag":63,"props":3462,"children":3464},{"className":3463},[],[3465],{"type":46,"value":3466},"sflow[s3]",{"type":46,"value":3468}," extra.",{"type":40,"tag":509,"props":3470,"children":3471},{},[3472,3477,3478,3484,3486,3491,3492,3497,3498,3503,3505,3511,3512,3518],{"type":40,"tag":55,"props":3473,"children":3474},{},[3475],{"type":46,"value":3476},"Telemetry:",{"type":46,"value":1010},{"type":40,"tag":63,"props":3479,"children":3481},{"className":3480},[],[3482],{"type":46,"value":3483},"monitor:",{"type":46,"value":3485}," samples GPU\u002FCPU\u002Fmem\u002Fnet on ",{"type":40,"tag":63,"props":3487,"children":3489},{"className":3488},[],[3490],{"type":46,"value":68},{"type":46,"value":452},{"type":40,"tag":63,"props":3493,"children":3495},{"className":3494},[],[3496],{"type":46,"value":76},{"type":46,"value":452},{"type":40,"tag":63,"props":3499,"children":3501},{"className":3500},[],[3502],{"type":46,"value":83},{"type":46,"value":3504}," (not k8s yet),\nor enable it without editing the recipe via ",{"type":40,"tag":63,"props":3506,"children":3508},{"className":3507},[],[3509],{"type":46,"value":3510},"--enable-workflow-monitor",{"type":46,"value":2463},{"type":40,"tag":63,"props":3513,"children":3515},{"className":3514},[],[3516],{"type":46,"value":3517},"--enable-task-monitor",{"type":46,"value":151},{"type":40,"tag":49,"props":3520,"children":3521},{},[3522],{"type":46,"value":3523},"See examples.md Examples 12 (result parsing), 7 (monitor), 9 (uploads); the results \u002F uploads \u002F monitor docs.",{"type":40,"tag":153,"props":3525,"children":3527},{"id":3526},"step-9-validate-before-running",[3528],{"type":46,"value":3529},"Step 9 — Validate before running",{"type":40,"tag":49,"props":3531,"children":3532},{},[3533,3535,3541,3543],{"type":46,"value":3534},"Run the one-command preflight before every real run (static schema + reference checks + GPU\nplan + ",{"type":40,"tag":63,"props":3536,"children":3538},{"className":3537},[],[3539],{"type":46,"value":3540},"sflow --dry-run",{"type":46,"value":3542},", with a root-cause explanation on failure). ",{"type":40,"tag":55,"props":3544,"children":3545},{},[3546],{"type":46,"value":3547},"Exit 0 = ready.",{"type":40,"tag":173,"props":3549,"children":3552},{"className":3550,"code":3551,"language":472,"meta":177,"style":177},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","python scripts\u002Fpreflight.py my_workflow.yaml\npython scripts\u002Fpreflight.py slurm.yaml common.yaml sglang\u002Fagg.yaml --set TP_SIZE=8   # composed\n",[3553],{"type":40,"tag":63,"props":3554,"children":3555},{"__ignoreMap":177},[3556,3574],{"type":40,"tag":183,"props":3557,"children":3558},{"class":185,"line":186},[3559,3564,3569],{"type":40,"tag":183,"props":3560,"children":3562},{"style":3561},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[3563],{"type":46,"value":950},{"type":40,"tag":183,"props":3565,"children":3566},{"style":207},[3567],{"type":46,"value":3568}," scripts\u002Fpreflight.py",{"type":40,"tag":183,"props":3570,"children":3571},{"style":207},[3572],{"type":46,"value":3573}," my_workflow.yaml\n",{"type":40,"tag":183,"props":3575,"children":3576},{"class":185,"line":224},[3577,3581,3585,3590,3595,3600,3605,3610,3615],{"type":40,"tag":183,"props":3578,"children":3579},{"style":3561},[3580],{"type":46,"value":950},{"type":40,"tag":183,"props":3582,"children":3583},{"style":207},[3584],{"type":46,"value":3568},{"type":40,"tag":183,"props":3586,"children":3587},{"style":207},[3588],{"type":46,"value":3589}," slurm.yaml",{"type":40,"tag":183,"props":3591,"children":3592},{"style":207},[3593],{"type":46,"value":3594}," common.yaml",{"type":40,"tag":183,"props":3596,"children":3597},{"style":207},[3598],{"type":46,"value":3599}," sglang\u002Fagg.yaml",{"type":40,"tag":183,"props":3601,"children":3602},{"style":207},[3603],{"type":46,"value":3604}," --set",{"type":40,"tag":183,"props":3606,"children":3607},{"style":207},[3608],{"type":46,"value":3609}," TP_SIZE=",{"type":40,"tag":183,"props":3611,"children":3612},{"style":242},[3613],{"type":46,"value":3614},"8",{"type":40,"tag":183,"props":3616,"children":3617},{"style":218},[3618],{"type":46,"value":3619},"   # composed\n",{"type":40,"tag":49,"props":3621,"children":3622},{},[3623],{"type":46,"value":3624},"Standalone steps when you want just one:",{"type":40,"tag":173,"props":3626,"children":3628},{"className":3550,"code":3627,"language":472,"meta":177,"style":177},"python scripts\u002Fvalidate_sflow_yaml.py my_workflow.yaml   # schema, refs, k8s volumes\npython scripts\u002Fcheck_gpu_plan.py     my_workflow.yaml    # GPU plan + oversubscription\nsflow run -f my_workflow.yaml --dry-run                  # full plan + expression resolution\n",[3629],{"type":40,"tag":63,"props":3630,"children":3631},{"__ignoreMap":177},[3632,3654,3676],{"type":40,"tag":183,"props":3633,"children":3634},{"class":185,"line":186},[3635,3639,3644,3649],{"type":40,"tag":183,"props":3636,"children":3637},{"style":3561},[3638],{"type":46,"value":950},{"type":40,"tag":183,"props":3640,"children":3641},{"style":207},[3642],{"type":46,"value":3643}," scripts\u002Fvalidate_sflow_yaml.py",{"type":40,"tag":183,"props":3645,"children":3646},{"style":207},[3647],{"type":46,"value":3648}," my_workflow.yaml",{"type":40,"tag":183,"props":3650,"children":3651},{"style":218},[3652],{"type":46,"value":3653},"   # schema, refs, k8s volumes\n",{"type":40,"tag":183,"props":3655,"children":3656},{"class":185,"line":224},[3657,3661,3666,3671],{"type":40,"tag":183,"props":3658,"children":3659},{"style":3561},[3660],{"type":46,"value":950},{"type":40,"tag":183,"props":3662,"children":3663},{"style":207},[3664],{"type":46,"value":3665}," scripts\u002Fcheck_gpu_plan.py",{"type":40,"tag":183,"props":3667,"children":3668},{"style":207},[3669],{"type":46,"value":3670},"     my_workflow.yaml",{"type":40,"tag":183,"props":3672,"children":3673},{"style":218},[3674],{"type":46,"value":3675},"    # GPU plan + oversubscription\n",{"type":40,"tag":183,"props":3677,"children":3678},{"class":185,"line":258},[3679,3684,3689,3694,3698,3703],{"type":40,"tag":183,"props":3680,"children":3681},{"style":3561},[3682],{"type":46,"value":3683},"sflow",{"type":40,"tag":183,"props":3685,"children":3686},{"style":207},[3687],{"type":46,"value":3688}," run",{"type":40,"tag":183,"props":3690,"children":3691},{"style":207},[3692],{"type":46,"value":3693}," -f",{"type":40,"tag":183,"props":3695,"children":3696},{"style":207},[3697],{"type":46,"value":3648},{"type":40,"tag":183,"props":3699,"children":3700},{"style":207},[3701],{"type":46,"value":3702}," --dry-run",{"type":40,"tag":183,"props":3704,"children":3705},{"style":218},[3706],{"type":46,"value":3707},"                  # full plan + expression resolution\n",{"type":40,"tag":476,"props":3709,"children":3710},{},[],{"type":40,"tag":153,"props":3712,"children":3714},{"id":3713},"modular-composition-when-splitting-across-files",[3715],{"type":46,"value":3716},"Modular composition (when splitting across files)",{"type":40,"tag":49,"props":3718,"children":3719},{},[3720,3722,3728,3730,3741,3743,3748,3750,3755,3756,3761,3762,3767,3768,3773,3774,3779,3781,3786,3788,3793,3795,3801],{"type":46,"value":3721},"Pass fragments directly: ",{"type":40,"tag":63,"props":3723,"children":3725},{"className":3724},[],[3726],{"type":46,"value":3727},"sflow run -f slurm.yaml -f common.yaml -f sglang\u002Fagg.yaml",{"type":46,"value":3729},".\nFiles are combined with a ",{"type":40,"tag":55,"props":3731,"children":3732},{},[3733,3735],{"type":46,"value":3734},"recursive deep merge keyed on ",{"type":40,"tag":63,"props":3736,"children":3738},{"className":3737},[],[3739],{"type":46,"value":3740},"name",{"type":46,"value":3742}," — ",{"type":40,"tag":63,"props":3744,"children":3746},{"className":3745},[],[3747],{"type":46,"value":193},{"type":46,"value":3749}," must match;\n",{"type":40,"tag":63,"props":3751,"children":3753},{"className":3752},[],[3754],{"type":46,"value":230},{"type":46,"value":70},{"type":40,"tag":63,"props":3757,"children":3759},{"className":3758},[],[3760],{"type":46,"value":264},{"type":46,"value":70},{"type":40,"tag":63,"props":3763,"children":3765},{"className":3764},[],[3766],{"type":46,"value":296},{"type":46,"value":70},{"type":40,"tag":63,"props":3769,"children":3771},{"className":3770},[],[3772],{"type":46,"value":327},{"type":46,"value":70},{"type":40,"tag":63,"props":3775,"children":3777},{"className":3776},[],[3778],{"type":46,"value":356},{"type":46,"value":3780}," and same-name\n",{"type":40,"tag":63,"props":3782,"children":3784},{"className":3783},[],[3785],{"type":46,"value":169},{"type":46,"value":3787}," are deep-merged (one definition can be scattered across files); leaf\nscalars\u002Flists are last-file-wins with a warning. Prefer ",{"type":40,"tag":63,"props":3789,"children":3791},{"className":3790},[],[3792],{"type":46,"value":620},{"type":46,"value":3794}," on optional fragments\n(absent targets skip silently) over ",{"type":40,"tag":63,"props":3796,"children":3798},{"className":3797},[],[3799],{"type":46,"value":3800},"--missable-tasks",{"type":46,"value":3802},". See examples.md Example 4.",{"type":40,"tag":153,"props":3804,"children":3806},{"id":3805},"pitfalls-checklist",[3807],{"type":46,"value":3808},"Pitfalls checklist",{"type":40,"tag":505,"props":3810,"children":3811},{},[3812,3832,3843,3855,3867,3872,3903,3921,3959,3978,3996,4008,4019],{"type":40,"tag":509,"props":3813,"children":3814},{},[3815,3817,3823,3825,3830],{"type":46,"value":3816},"Missing ",{"type":40,"tag":63,"props":3818,"children":3820},{"className":3819},[],[3821],{"type":46,"value":3822},"version: \"0.1\"",{"type":46,"value":3824}," in ",{"type":40,"tag":55,"props":3826,"children":3827},{},[3828],{"type":46,"value":3829},"every",{"type":46,"value":3831}," file (including fragments).",{"type":40,"tag":509,"props":3833,"children":3834},{},[3835,3841],{"type":40,"tag":63,"props":3836,"children":3838},{"className":3837},[],[3839],{"type":46,"value":3840},"${{ task.X.nodes }}",{"type":46,"value":3842}," in a YAML field — task context works only in scripts.",{"type":40,"tag":509,"props":3844,"children":3845},{},[3846,3848,3853],{"type":46,"value":3847},"Forgetting ",{"type":40,"tag":63,"props":3849,"children":3851},{"className":3850},[],[3852],{"type":46,"value":572},{"type":46,"value":3854}," — tasks run immediately without it.",{"type":40,"tag":509,"props":3856,"children":3857},{},[3858,3860,3866],{"type":46,"value":3859},"GPU oversubscription: Σ GPU counts > ",{"type":40,"tag":63,"props":3861,"children":3863},{"className":3862},[],[3864],{"type":46,"value":3865},"nodes × gpus_per_node",{"type":46,"value":151},{"type":40,"tag":509,"props":3868,"children":3869},{},[3870],{"type":46,"value":3871},"Probing a short-lived one-shot task (don't probe benchmarks).",{"type":40,"tag":509,"props":3873,"children":3874},{},[3875,3877,3882,3884,3889,3891,3896,3897,3902],{"type":46,"value":3876},"Expecting ",{"type":40,"tag":63,"props":3878,"children":3880},{"className":3879},[],[3881],{"type":46,"value":1715},{"type":46,"value":3883}," to be regex — it's ",{"type":40,"tag":55,"props":3885,"children":3886},{},[3887],{"type":46,"value":3888},"literal",{"type":46,"value":3890}," unless prefixed ",{"type":40,"tag":63,"props":3892,"children":3894},{"className":3893},[],[3895],{"type":46,"value":1796},{"type":46,"value":70},{"type":40,"tag":63,"props":3898,"children":3900},{"className":3899},[],[3901],{"type":46,"value":1803},{"type":46,"value":151},{"type":40,"tag":509,"props":3904,"children":3905},{},[3906,3908,3913,3915,3920],{"type":46,"value":3907},"Docker: the operator is ",{"type":40,"tag":63,"props":3909,"children":3911},{"className":3910},[],[3912],{"type":46,"value":2821},{"type":46,"value":3914}," (the backend is ",{"type":40,"tag":63,"props":3916,"children":3918},{"className":3917},[],[3919],{"type":46,"value":2813},{"type":46,"value":967},{"type":40,"tag":509,"props":3922,"children":3923},{},[3924,3926,3931,3933,3938,3940,3945,3946,3952,3953,3958],{"type":46,"value":3925},"Putting the workload ",{"type":40,"tag":63,"props":3927,"children":3929},{"className":3928},[],[3930],{"type":46,"value":2829},{"type":46,"value":3932}," on the ",{"type":40,"tag":55,"props":3934,"children":3935},{},[3936],{"type":46,"value":3937},"kubernetes backend",{"type":46,"value":3939}," — it belongs on the ",{"type":40,"tag":63,"props":3941,"children":3943},{"className":3942},[],[3944],{"type":46,"value":2868},{"type":46,"value":452},{"type":40,"tag":63,"props":3947,"children":3949},{"className":3948},[],[3950],{"type":46,"value":3951},"k8s_mpi",{"type":46,"value":1010},{"type":40,"tag":55,"props":3954,"children":3955},{},[3956],{"type":46,"value":3957},"operator",{"type":46,"value":151},{"type":40,"tag":509,"props":3960,"children":3961},{},[3962,3964,3969,3971,3976],{"type":46,"value":3963},"K8s: an ",{"type":40,"tag":63,"props":3965,"children":3967},{"className":3966},[],[3968],{"type":46,"value":1538},{"type":46,"value":3970}," path no ",{"type":40,"tag":63,"props":3972,"children":3974},{"className":3973},[],[3975],{"type":46,"value":1590},{"type":46,"value":3977}," PVC covers and the node lacks → the pod fails (hostPath). Use a PVC.",{"type":40,"tag":509,"props":3979,"children":3980},{},[3981,3983,3988,3990,3995],{"type":46,"value":3982},"Expecting K8s task outputs to persist — the output dir is an ephemeral ",{"type":40,"tag":63,"props":3984,"children":3986},{"className":3985},[],[3987],{"type":46,"value":3023},{"type":46,"value":3989},"; use a PVC or ",{"type":40,"tag":63,"props":3991,"children":3993},{"className":3992},[],[3994],{"type":46,"value":3031},{"type":46,"value":151},{"type":40,"tag":509,"props":3997,"children":3998},{},[3999,4001,4006],{"type":46,"value":4000},"Multi-node Slurm with models\u002Fworkspace\u002Foutput on ",{"type":40,"tag":55,"props":4002,"children":4003},{},[4004],{"type":46,"value":4005},"node-local",{"type":46,"value":4007}," storage — same-path mounts break; use shared storage.",{"type":40,"tag":509,"props":4009,"children":4010},{},[4011,4013,4018],{"type":46,"value":4012},"S3 secrets in YAML (use the boto3 chain) or forgetting the ",{"type":40,"tag":63,"props":4014,"children":4016},{"className":4015},[],[4017],{"type":46,"value":3466},{"type":46,"value":3468},{"type":40,"tag":509,"props":4020,"children":4021},{},[4022],{"type":46,"value":4023},"Hand-writing mounts for a path already declared as an artifact — redundant; artifacts auto-mount same-path.",{"type":40,"tag":153,"props":4025,"children":4027},{"id":4026},"additional-resources",[4028],{"type":46,"value":4029},"Additional resources",{"type":40,"tag":924,"props":4031,"children":4032},{},[4033,4042,4051],{"type":40,"tag":509,"props":4034,"children":4035},{},[4036,4038],{"type":46,"value":4037},"Field reference & merge rules → ",{"type":40,"tag":128,"props":4039,"children":4040},{"href":130},[4041],{"type":46,"value":130},{"type":40,"tag":509,"props":4043,"children":4044},{},[4045,4047],{"type":46,"value":4046},"12 annotated, runnable recipes → ",{"type":40,"tag":128,"props":4048,"children":4049},{"href":137},[4050],{"type":46,"value":137},{"type":40,"tag":509,"props":4052,"children":4053},{},[4054,4056],{"type":46,"value":4055},"Full user docs → ",{"type":40,"tag":128,"props":4057,"children":4059},{"href":144,"rel":4058},[146],[4060],{"type":46,"value":149},{"type":40,"tag":4062,"props":4063,"children":4064},"style",{},[4065],{"type":46,"value":4066},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":4068,"total":258},[4069,4086,4101],{"slug":4070,"name":4070,"fn":4071,"description":4072,"org":4073,"tags":4074,"stars":23,"repoUrl":24,"updatedAt":4085},"sflow-code-review","review sflow code changes","Review sflow code changes for functional defects, modular by-purpose structure, duplicated logic that should be consolidated, and adequate unit + e2e CLI test coverage. Use when reviewing an sflow diff, branch, PR, staged changes, or when the user asks for a code review of sflow.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4075,4078,4081,4082],{"name":4076,"slug":4077,"type":15},"Code Review","code-review",{"name":4079,"slug":4080,"type":15},"Engineering","engineering",{"name":9,"slug":8,"type":15},{"name":4083,"slug":4084,"type":15},"Testing","testing","2026-07-23T06:06:17.700089",{"slug":4087,"name":4087,"fn":4088,"description":4089,"org":4090,"tags":4091,"stars":23,"repoUrl":24,"updatedAt":4100},"sflow-error-analysis","troubleshoot sflow workflow errors","Diagnose and troubleshoot sflow workflow errors from log output, error messages, and task failures. Covers config validation errors, expression resolution failures, backend issues (Slurm, Docker, Kubernetes\u002Fkubectl\u002FRBAC), probe timeouts, task crashes, S3 storage\u002Fupload failures, and batch submission problems. Use when the user encounters an sflow error, pastes error output, asks to debug a failed workflow, or asks about sflow troubleshooting.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4092,4095,4098,4099],{"name":4093,"slug":4094,"type":15},"Debugging","debugging",{"name":4096,"slug":4097,"type":15},"Logs","logs",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"2026-07-23T05:43:45.493836",{"slug":4,"name":4,"fn":5,"description":6,"org":4102,"tags":4103,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4104,4105,4106,4107],{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},{"name":17,"slug":18,"type":15},{"items":4109,"total":4261},[4110,4128,4145,4156,4168,4180,4193,4207,4218,4229,4241,4250],{"slug":4111,"name":4111,"fn":4112,"description":4113,"org":4114,"tags":4115,"stars":4125,"repoUrl":4126,"updatedAt":4127},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4116,4119,4122],{"name":4117,"slug":4118,"type":15},"Documentation","documentation",{"name":4120,"slug":4121,"type":15},"MCP","mcp",{"name":4123,"slug":4124,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":4129,"name":4129,"fn":4130,"description":4131,"org":4132,"tags":4133,"stars":4142,"repoUrl":4143,"updatedAt":4144},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4134,4137,4140],{"name":4135,"slug":4136,"type":15},"Containers","containers",{"name":4138,"slug":4139,"type":15},"Deployment","deployment",{"name":4141,"slug":950,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":4146,"name":4146,"fn":4147,"description":4148,"org":4149,"tags":4150,"stars":4142,"repoUrl":4143,"updatedAt":4155},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4151,4154],{"name":4152,"slug":4153,"type":15},"CI\u002FCD","ci-cd",{"name":4138,"slug":4139,"type":15},"2026-07-14T05:25:59.97109",{"slug":4157,"name":4157,"fn":4158,"description":4159,"org":4160,"tags":4161,"stars":4142,"repoUrl":4143,"updatedAt":4167},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4162,4163,4164],{"name":4152,"slug":4153,"type":15},{"name":4138,"slug":4139,"type":15},{"name":4165,"slug":4166,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":4169,"name":4169,"fn":4170,"description":4171,"org":4172,"tags":4173,"stars":4142,"repoUrl":4143,"updatedAt":4179},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4174,4175,4176],{"name":4093,"slug":4094,"type":15},{"name":4165,"slug":4166,"type":15},{"name":4177,"slug":4178,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":4181,"name":4181,"fn":4182,"description":4183,"org":4184,"tags":4185,"stars":4142,"repoUrl":4143,"updatedAt":4192},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4186,4189],{"name":4187,"slug":4188,"type":15},"Best Practices","best-practices",{"name":4190,"slug":4191,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":4194,"name":4194,"fn":4195,"description":4196,"org":4197,"tags":4198,"stars":4142,"repoUrl":4143,"updatedAt":4206},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4199,4202,4205],{"name":4200,"slug":4201,"type":15},"Machine Learning","machine-learning",{"name":4203,"slug":4204,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":4208,"name":4208,"fn":4209,"description":4210,"org":4211,"tags":4212,"stars":4142,"repoUrl":4143,"updatedAt":4217},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4213,4216],{"name":4214,"slug":4215,"type":15},"QA","qa",{"name":4083,"slug":4084,"type":15},"2026-07-14T05:25:53.673039",{"slug":4219,"name":4219,"fn":4220,"description":4221,"org":4222,"tags":4223,"stars":4142,"repoUrl":4143,"updatedAt":4228},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4224,4225],{"name":4138,"slug":4139,"type":15},{"name":4226,"slug":4227,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":4230,"name":4230,"fn":4231,"description":4232,"org":4233,"tags":4234,"stars":4142,"repoUrl":4143,"updatedAt":4240},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4235,4236,4237],{"name":4076,"slug":4077,"type":15},{"name":4165,"slug":4166,"type":15},{"name":4238,"slug":4239,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":4242,"name":4242,"fn":4243,"description":4244,"org":4245,"tags":4246,"stars":4142,"repoUrl":4143,"updatedAt":4249},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4247,4248],{"name":4214,"slug":4215,"type":15},{"name":4083,"slug":4084,"type":15},"2026-07-14T05:25:54.928983",{"slug":4251,"name":4251,"fn":4252,"description":4253,"org":4254,"tags":4255,"stars":4142,"repoUrl":4143,"updatedAt":4260},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[4256,4259],{"name":4257,"slug":4258,"type":15},"Automation","automation",{"name":4152,"slug":4153,"type":15},"2026-07-30T05:29:03.275638",496]