[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-sflow-error-analysis":3,"mdc--8ojukk-key":34,"related-org-nvidia-sflow-error-analysis":2288,"related-repo-nvidia-sflow-error-analysis":2446},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"sflow-error-analysis","troubleshoot sflow workflow errors","Diagnose and troubleshoot sflow workflow errors from log output, error messages, and task failures. Covers config validation errors, expression resolution failures, backend issues (Slurm, Docker, Kubernetes\u002Fkubectl\u002FRBAC), probe timeouts, task crashes, S3 storage\u002Fupload failures, and batch submission problems. Use when the user encounters an sflow error, pastes error output, asks to debug a failed workflow, or asks about sflow troubleshooting.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,17,20],{"name":13,"slug":14,"type":15},"Logs","logs","tag",{"name":9,"slug":8,"type":15},{"name":18,"slug":19,"type":15},"Debugging","debugging",{"name":21,"slug":22,"type":15},"Workflow Automation","workflow-automation",36,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnv-sflow","2026-07-23T05:43:45.493836",null,6,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"A Python CLI workflow orchestrator with pluggable backends (e.g. local, Slurm) for running declarative YAML DAGs, collecting logs, and organizing outputs consistently.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fnv-sflow\u002Ftree\u002FHEAD\u002Fsrc\u002Fsflow\u002Fskills\u002Fsflow-error-analysis","---\nname: sflow-error-analysis\ndescription: >-\n  Diagnose and troubleshoot sflow workflow errors from log output, error messages, and\n  task failures. Covers config validation errors, expression resolution failures, backend\n  issues (Slurm, Docker, Kubernetes\u002Fkubectl\u002FRBAC), probe timeouts, task crashes, S3\n  storage\u002Fupload failures, and batch submission problems. Use when the user encounters an\n  sflow error, pastes error output, asks to debug a failed workflow, or asks about sflow\n  troubleshooting.\n---\n\n# sflow Error Analysis\n\nDiagnose a failure in a fixed order: **capture → classify → locate → fix → re-verify.**\nThe first move depends on *when* it broke:\n\n- **Launch \u002F validation failure** — nothing ran (bad config, YAML, expression, artifact,\n  merge, CSV). Caught by `--dry-run`; the fix is in the YAML.\n- **Runtime failure** — a task started and died (backend launch, probe timeout, crash,\n  upload). You have a run dir; the fix is in the logs.\n\nFigure out which you have, then follow that path below.\n\n## Step 1 — Capture the failure\n\n- **A task ran and failed (you have a run dir)** → start with the one-shot triage. It walks\n  per-task status → orchestrator `sflow.log` → each failed `\u003Ctask>\u002F\u003Ctask>.log` and prints a\n  single root cause + fix. Drill in from its output:\n\n  ```bash\n  python scripts\u002Ftriage.py \u003Coutput_dir>\u002F\u003Crun_id>\u002F\n  ```\n\n- **It failed at launch (no run dir)** → reproduce with a dry-run and parse it:\n\n  ```bash\n  sflow run -f config.yaml --dry-run\n  python scripts\u002Fparse_sflow_errors.py - \u003C \u003C(sflow run -f config.yaml --dry-run 2>&1)\n  ```\n\n- **Only have a pasted error?** Match its marker in Step 2 to pick the path.\n\n## Step 2 — Classify (marker → where to look)\n\n| Category            | Marker in the output                                     | Path    | Where to look             |\n|---------------------|----------------------------------------------------------|---------|---------------------------|\n| Config loading      | `Configuration error:`, `File not found:`                | launch  | CLI output \u002F sflow.log    |\n| YAML syntax         | `Error parsing YAML`                                     | launch  | CLI output                |\n| Expression          | `Undefined variable`, `Invalid expression syntax`        | launch  | CLI output                |\n| Artifact            | `Artifact path validation failed`                        | launch  | CLI output \u002F dry-run      |\n| Schema              | `type 'docker' is not valid` (use `docker_run`), `mutually exclusive` | launch | CLI output (validation) |\n| Merge conflict      | `Version conflict` (name\u002Fdup-task now warn+merge)        | launch  | CLI output (multi-file)   |\n| Batch \u002F CSV         | `CSV file`, `sflow_config_file column`                   | launch  | CLI output                |\n| SLURM               | `salloc failed`, `sbatch failed`                         | runtime | CLI \u002F sbatch stderr       |\n| Kubernetes          | `kubectl`, RBAC\u002Fpreflight, pod `Pending`\u002F`ImagePullBackOff` | runtime | CLI \u002F sflow.log \u002F `kubectl describe` |\n| Docker              | `docker run` errors, remote host, NVIDIA toolkit         | runtime | `\u003Ctask>\u002F\u003Ctask>.log`       |\n| Storage \u002F uploads   | `requires boto3`, `No AWS credentials`, upload failures  | runtime | CLI \u002F dry-run \u002F sflow.log |\n| Probe timeout       | Task stuck waiting                                       | runtime | `\u003Ctask>\u002F\u003Ctask>.log`       |\n| Task failure        | `Traceback`, non-zero exit                              | runtime | `\u003Ctask>\u002F\u003Ctask>.log`       |\n\n## Step 3 — Locate the evidence\n\n```text\n\u003Coutput_dir>\u002F\u003Crun_id>\u002F\n  sflow.log                    # orchestrator log (launch + status lines)\n  sflow_summary.log            # live per-task status + failure hints\n  *_cmds.log                   # command-only launch logs, by command family\n  results.json                 # workflow-level parsed metrics (aggregated from task result: blocks)\n  sflow_monitor.log            # hardware monitor overview (if monitor: enabled — local\u002Fslurm\u002Fdocker only, NOT k8s)\n  sflow_monitor\u002F               # detailed monitor reports (csv\u002Fsvg\u002Fpng) + raw\u002F samples\n  \u003Ctask_name>\u002F\u003Ctask_name>.log  # task stdout\u002Fstderr — the crash lives here\n  \u003Ctask_name>\u002Fresult.json      # that task's parsed metrics (if it declares a result: block)\n  \u003Ctask_name>_0\u002F               # replica 0\n```\n\n- **Parsed results:** a task with a `result:` block writes `\u003Ctask>\u002Fresult.json`\n  (`ok`\u002F`values`\u002F`errors`) and updates the run-level `results.json`. Result parsing is\n  **best-effort** — a missed regex or failed `required:` spec logs a warning and sets\n  `ok=false`, but does not by itself fail the task; check `errors` there if a metric is empty.\n  Parsing runs at task finalize, so a task shows `COMPLETED` only *after* it (and any\n  `uploads:`) finish.\n- **No monitor output?** Hardware monitoring is unsupported on **kubernetes**\n  (`supports_host_monitoring=False`) — sflow logs \"node-level hardware monitoring is not\n  implemented for this backend yet\" and produces no `sflow_monitor*`. On local\u002Fslurm\u002Fdocker,\n  an empty monitor usually means the `used_by_tasks` target resolved to no nodes.\n\n- **Batch jobs:** also read the generated `.sh` and sbatch stdout\u002Fstderr\n  (`sflow_output\u002F%j-sflow-submit.out` \u002F `.err`).\n- **Kubernetes:** `kubectl describe pod \u003Cpod>` and `kubectl get events` for\n  scheduling\u002Fimage issues the pod log can't show.\n\n## Step 4 — Root-cause & fix\n\n### 4a. Launch \u002F validation — fix in the YAML, re-run `--dry-run`\n\n| Error text                                               | Fix                                                        |\n|----------------------------------------------------------|------------------------------------------------------------|\n| `Configuration file not found: \u003Cpath>`                   | Check the `-f` path                                        |\n| `Error parsing YAML configuration: \u003Cdetail>`             | Fix YAML syntax at the indicated line                      |\n| `Configuration validation failed: \u003Cdetail>`              | Check field types\u002Fvalues (schema-reference.md)             |\n| `Operator type 'docker' is not valid`                    | Use `type: docker_run` for the operator (backend is `type: docker`) |\n| `Variable '\u003Ckey>' ... is not defined`                    | Declare the variable in YAML before `--set`-ing it         |\n| `Undefined variable in expression`                       | Fix spelling; ensure the variable is declared              |\n| `Invalid expression syntax`                              | Fix the `${{ }}` Jinja2 syntax                             |\n| `Artifact path validation failed`                        | Fix the `fs:\u002F\u002F` path or create it                          |\n| `Version conflict`                                       | Every file must use `version: \"0.1\"`                       |\n| `CSV file must contain a 'sflow_config_file' column`     | Add the required column to the CSV                         |\n\n### 4b. Runtime — read the logs (backend launch, then task crash)\n\n| Symptom                                                  | Fix                                                        |\n|----------------------------------------------------------|------------------------------------------------------------|\n| `salloc failed` \u002F `sbatch failed`                        | Check partition\u002Faccount\u002Favailability (`sinfo`, `sacctmgr`) |\n| kube preflight: missing permission \u002F namespace           | Fix RBAC or `--kube-namespace`; `SFLOW_SKIP_K8S_PREFLIGHT=1` to bypass |\n| pod stuck `Pending`                                      | `kubectl describe pod`: no node has the GPUs — check `gpus_per_node`, `scheduling` mode, node-selector\u002Ftolerations, and that `gpus.count` is a multiple of the node count |\n| pod `ImagePullBackOff` \u002F container not found             | Wrong `image` or missing `image_pull_secrets`; verify registry access |\n| Docker GPU task fails                                    | Install the NVIDIA container toolkit on the host; set `gpus_per_node` |\n| Probe timeout (task hangs waiting)                       | Read the **Probe Traces (last attempt)** section of `sflow_summary.log` — it shows what each probe last saw (the last log line for `log_watch`, the endpoint result for `tcp_port`\u002F`http`). Then compare server output to `log_watch` — it matches **literally** unless prefixed `re:`\u002F`regex:`; adjust the pattern or `timeout` |\n| `Traceback` in `\u003Ctask>.log`                              | Read the traceback for the real cause (below are the usual ones) |\n| CUDA OOM                                                 | Reduce batch size, TP\u002FDP, or model size                    |\n| `Address already in use` \u002F port conflict                | Kill stale processes or offset the port                    |\n| NCCL \u002F communication errors                              | Set `NCCL_SOCKET_IFNAME`; check the inter-node network     |\n| `S3 storage requires boto3` \u002F `no AWS credentials` \u002F upload fails | `pip install 'sflow[s3]'`; provide creds via the boto3 chain (`AWS_*` \u002F `~\u002F.aws\u002Fcredentials` \u002F IAM — never in YAML); verify bucket\u002Fregion |\n\nFor a pattern-by-pattern catalog (exact message, cause, and fix per error), see\n[error-catalog.md](error-catalog.md).\n\n## Step 5 — Re-verify\n\nRe-run the relevant capture command — clean exit means fixed:\n\n```bash\nsflow run -f config.yaml --dry-run          # launch fixes: exit 0 = ready\npython scripts\u002Ftriage.py \u003Crun_dir>\u002F          # runtime fixes: no failed tasks\n```\n\n## Diagnostic commands (reference)\n\n```bash\nsflow run -f config.yaml --dry-run                       # validate + resolve without executing\nsflow compose f1.yaml f2.yaml --resolve -o merged.yaml   # see the merged + resolved config\npython scripts\u002Ftriage.py \u003Coutput_dir>\u002F\u003Crun_id>\u002F          # one-shot: summary + logs -> root cause\npython scripts\u002Fsummarize_run.py \u003Coutput_dir>\u002F\u003Crun_id>\u002F   #   per-task status + tracebacks only\npython scripts\u002Fparse_sflow_errors.py \u003Crun_id>\u002Fsflow.log  #   categorize an orchestrator log\nsinfo -p \u003Cpartition>; sacctmgr show account \u003Cname>       # Slurm environment\nkubectl -n \u003Cns> get pods && kubectl -n \u003Cns> describe pod \u003Cpod>   # Kubernetes scheduling\u002Fimage\nkubectl -n \u003Cns> get events --sort-by=.lastTimestamp\n```\n\n## Additional resources\n\n- Exhaustive error patterns → [error-catalog.md](error-catalog.md)\n- Docs → [faq](https:\u002F\u002Fnvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Ffaq),\n  [cli](https:\u002F\u002Fnvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Fcli),\n  [configuration](https:\u002F\u002Fnvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Fconfiguration),\n  [backends](https:\u002F\u002Fnvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Fbackends)\n",{"data":35,"body":36},{"name":4,"description":6},{"type":37,"children":38},"root",[39,47,69,103,108,115,320,326,788,794,804,1016,1022,1034,1276,1282,1679,1692,1698,1703,1777,1783,2226,2232,2282],{"type":40,"tag":41,"props":42,"children":43},"element","h1",{"id":4},[44],{"type":45,"value":46},"text","sflow Error Analysis",{"type":40,"tag":48,"props":49,"children":50},"p",{},[51,53,59,61,67],{"type":45,"value":52},"Diagnose a failure in a fixed order: ",{"type":40,"tag":54,"props":55,"children":56},"strong",{},[57],{"type":45,"value":58},"capture → classify → locate → fix → re-verify.",{"type":45,"value":60},"\nThe first move depends on ",{"type":40,"tag":62,"props":63,"children":64},"em",{},[65],{"type":45,"value":66},"when",{"type":45,"value":68}," it broke:",{"type":40,"tag":70,"props":71,"children":72},"ul",{},[73,93],{"type":40,"tag":74,"props":75,"children":76},"li",{},[77,82,84,91],{"type":40,"tag":54,"props":78,"children":79},{},[80],{"type":45,"value":81},"Launch \u002F validation failure",{"type":45,"value":83}," — nothing ran (bad config, YAML, expression, artifact,\nmerge, CSV). Caught by ",{"type":40,"tag":85,"props":86,"children":88},"code",{"className":87},[],[89],{"type":45,"value":90},"--dry-run",{"type":45,"value":92},"; the fix is in the YAML.",{"type":40,"tag":74,"props":94,"children":95},{},[96,101],{"type":40,"tag":54,"props":97,"children":98},{},[99],{"type":45,"value":100},"Runtime failure",{"type":45,"value":102}," — a task started and died (backend launch, probe timeout, crash,\nupload). You have a run dir; the fix is in the logs.",{"type":40,"tag":48,"props":104,"children":105},{},[106],{"type":45,"value":107},"Figure out which you have, then follow that path below.",{"type":40,"tag":109,"props":110,"children":112},"h2",{"id":111},"step-1-capture-the-failure",[113],{"type":45,"value":114},"Step 1 — Capture the failure",{"type":40,"tag":70,"props":116,"children":117},{},[118,224,310],{"type":40,"tag":74,"props":119,"children":120},{},[121,126,128,134,136,142,144],{"type":40,"tag":54,"props":122,"children":123},{},[124],{"type":45,"value":125},"A task ran and failed (you have a run dir)",{"type":45,"value":127}," → start with the one-shot triage. It walks\nper-task status → orchestrator ",{"type":40,"tag":85,"props":129,"children":131},{"className":130},[],[132],{"type":45,"value":133},"sflow.log",{"type":45,"value":135}," → each failed ",{"type":40,"tag":85,"props":137,"children":139},{"className":138},[],[140],{"type":45,"value":141},"\u003Ctask>\u002F\u003Ctask>.log",{"type":45,"value":143}," and prints a\nsingle root cause + fix. Drill in from its output:",{"type":40,"tag":145,"props":146,"children":151},"pre",{"className":147,"code":148,"language":149,"meta":150,"style":150},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","python scripts\u002Ftriage.py \u003Coutput_dir>\u002F\u003Crun_id>\u002F\n","bash","",[152],{"type":40,"tag":85,"props":153,"children":154},{"__ignoreMap":150},[155],{"type":40,"tag":156,"props":157,"children":160},"span",{"class":158,"line":159},"line",1,[161,167,173,179,184,190,195,200,205,210,215,219],{"type":40,"tag":156,"props":162,"children":164},{"style":163},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[165],{"type":45,"value":166},"python",{"type":40,"tag":156,"props":168,"children":170},{"style":169},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[171],{"type":45,"value":172}," scripts\u002Ftriage.py",{"type":40,"tag":156,"props":174,"children":176},{"style":175},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[177],{"type":45,"value":178}," \u003C",{"type":40,"tag":156,"props":180,"children":181},{"style":169},[182],{"type":45,"value":183},"output_di",{"type":40,"tag":156,"props":185,"children":187},{"style":186},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[188],{"type":45,"value":189},"r",{"type":40,"tag":156,"props":191,"children":192},{"style":175},[193],{"type":45,"value":194},">",{"type":40,"tag":156,"props":196,"children":197},{"style":169},[198],{"type":45,"value":199},"\u002F",{"type":40,"tag":156,"props":201,"children":202},{"style":175},[203],{"type":45,"value":204},"\u003C",{"type":40,"tag":156,"props":206,"children":207},{"style":169},[208],{"type":45,"value":209},"run_i",{"type":40,"tag":156,"props":211,"children":212},{"style":186},[213],{"type":45,"value":214},"d",{"type":40,"tag":156,"props":216,"children":217},{"style":175},[218],{"type":45,"value":194},{"type":40,"tag":156,"props":220,"children":221},{"style":169},[222],{"type":45,"value":223},"\u002F\n",{"type":40,"tag":74,"props":225,"children":226},{},[227,232,234],{"type":40,"tag":54,"props":228,"children":229},{},[230],{"type":45,"value":231},"It failed at launch (no run dir)",{"type":45,"value":233}," → reproduce with a dry-run and parse it:",{"type":40,"tag":145,"props":235,"children":237},{"className":147,"code":236,"language":149,"meta":150,"style":150},"sflow run -f config.yaml --dry-run\npython scripts\u002Fparse_sflow_errors.py - \u003C \u003C(sflow run -f config.yaml --dry-run 2>&1)\n",[238],{"type":40,"tag":85,"props":239,"children":240},{"__ignoreMap":150},[241,269],{"type":40,"tag":156,"props":242,"children":243},{"class":158,"line":159},[244,249,254,259,264],{"type":40,"tag":156,"props":245,"children":246},{"style":163},[247],{"type":45,"value":248},"sflow",{"type":40,"tag":156,"props":250,"children":251},{"style":169},[252],{"type":45,"value":253}," run",{"type":40,"tag":156,"props":255,"children":256},{"style":169},[257],{"type":45,"value":258}," -f",{"type":40,"tag":156,"props":260,"children":261},{"style":169},[262],{"type":45,"value":263}," config.yaml",{"type":40,"tag":156,"props":265,"children":266},{"style":169},[267],{"type":45,"value":268}," --dry-run\n",{"type":40,"tag":156,"props":270,"children":272},{"class":158,"line":271},2,[273,277,282,287,291,296,300,305],{"type":40,"tag":156,"props":274,"children":275},{"style":163},[276],{"type":45,"value":166},{"type":40,"tag":156,"props":278,"children":279},{"style":169},[280],{"type":45,"value":281}," scripts\u002Fparse_sflow_errors.py",{"type":40,"tag":156,"props":283,"children":284},{"style":169},[285],{"type":45,"value":286}," -",{"type":40,"tag":156,"props":288,"children":289},{"style":175},[290],{"type":45,"value":178},{"type":40,"tag":156,"props":292,"children":293},{"style":175},[294],{"type":45,"value":295}," \u003C(",{"type":40,"tag":156,"props":297,"children":298},{"style":163},[299],{"type":45,"value":248},{"type":40,"tag":156,"props":301,"children":302},{"style":169},[303],{"type":45,"value":304}," run -f config.yaml --dry-run ",{"type":40,"tag":156,"props":306,"children":307},{"style":175},[308],{"type":45,"value":309},"2>&1)\n",{"type":40,"tag":74,"props":311,"children":312},{},[313,318],{"type":40,"tag":54,"props":314,"children":315},{},[316],{"type":45,"value":317},"Only have a pasted error?",{"type":45,"value":319}," Match its marker in Step 2 to pick the path.",{"type":40,"tag":109,"props":321,"children":323},{"id":322},"step-2-classify-marker-where-to-look",[324],{"type":45,"value":325},"Step 2 — Classify (marker → where to look)",{"type":40,"tag":327,"props":328,"children":329},"table",{},[330,359],{"type":40,"tag":331,"props":332,"children":333},"thead",{},[334],{"type":40,"tag":335,"props":336,"children":337},"tr",{},[338,344,349,354],{"type":40,"tag":339,"props":340,"children":341},"th",{},[342],{"type":45,"value":343},"Category",{"type":40,"tag":339,"props":345,"children":346},{},[347],{"type":45,"value":348},"Marker in the output",{"type":40,"tag":339,"props":350,"children":351},{},[352],{"type":45,"value":353},"Path",{"type":40,"tag":339,"props":355,"children":356},{},[357],{"type":45,"value":358},"Where to look",{"type":40,"tag":360,"props":361,"children":362},"tbody",{},[363,399,425,457,483,525,553,585,619,666,697,732,757],{"type":40,"tag":335,"props":364,"children":365},{},[366,372,389,394],{"type":40,"tag":367,"props":368,"children":369},"td",{},[370],{"type":45,"value":371},"Config loading",{"type":40,"tag":367,"props":373,"children":374},{},[375,381,383],{"type":40,"tag":85,"props":376,"children":378},{"className":377},[],[379],{"type":45,"value":380},"Configuration error:",{"type":45,"value":382},", ",{"type":40,"tag":85,"props":384,"children":386},{"className":385},[],[387],{"type":45,"value":388},"File not found:",{"type":40,"tag":367,"props":390,"children":391},{},[392],{"type":45,"value":393},"launch",{"type":40,"tag":367,"props":395,"children":396},{},[397],{"type":45,"value":398},"CLI output \u002F sflow.log",{"type":40,"tag":335,"props":400,"children":401},{},[402,407,416,420],{"type":40,"tag":367,"props":403,"children":404},{},[405],{"type":45,"value":406},"YAML syntax",{"type":40,"tag":367,"props":408,"children":409},{},[410],{"type":40,"tag":85,"props":411,"children":413},{"className":412},[],[414],{"type":45,"value":415},"Error parsing YAML",{"type":40,"tag":367,"props":417,"children":418},{},[419],{"type":45,"value":393},{"type":40,"tag":367,"props":421,"children":422},{},[423],{"type":45,"value":424},"CLI output",{"type":40,"tag":335,"props":426,"children":427},{},[428,433,449,453],{"type":40,"tag":367,"props":429,"children":430},{},[431],{"type":45,"value":432},"Expression",{"type":40,"tag":367,"props":434,"children":435},{},[436,442,443],{"type":40,"tag":85,"props":437,"children":439},{"className":438},[],[440],{"type":45,"value":441},"Undefined variable",{"type":45,"value":382},{"type":40,"tag":85,"props":444,"children":446},{"className":445},[],[447],{"type":45,"value":448},"Invalid expression syntax",{"type":40,"tag":367,"props":450,"children":451},{},[452],{"type":45,"value":393},{"type":40,"tag":367,"props":454,"children":455},{},[456],{"type":45,"value":424},{"type":40,"tag":335,"props":458,"children":459},{},[460,465,474,478],{"type":40,"tag":367,"props":461,"children":462},{},[463],{"type":45,"value":464},"Artifact",{"type":40,"tag":367,"props":466,"children":467},{},[468],{"type":40,"tag":85,"props":469,"children":471},{"className":470},[],[472],{"type":45,"value":473},"Artifact path validation failed",{"type":40,"tag":367,"props":475,"children":476},{},[477],{"type":45,"value":393},{"type":40,"tag":367,"props":479,"children":480},{},[481],{"type":45,"value":482},"CLI output \u002F dry-run",{"type":40,"tag":335,"props":484,"children":485},{},[486,491,516,520],{"type":40,"tag":367,"props":487,"children":488},{},[489],{"type":45,"value":490},"Schema",{"type":40,"tag":367,"props":492,"children":493},{},[494,500,502,508,510],{"type":40,"tag":85,"props":495,"children":497},{"className":496},[],[498],{"type":45,"value":499},"type 'docker' is not valid",{"type":45,"value":501}," (use ",{"type":40,"tag":85,"props":503,"children":505},{"className":504},[],[506],{"type":45,"value":507},"docker_run",{"type":45,"value":509},"), ",{"type":40,"tag":85,"props":511,"children":513},{"className":512},[],[514],{"type":45,"value":515},"mutually exclusive",{"type":40,"tag":367,"props":517,"children":518},{},[519],{"type":45,"value":393},{"type":40,"tag":367,"props":521,"children":522},{},[523],{"type":45,"value":524},"CLI output (validation)",{"type":40,"tag":335,"props":526,"children":527},{},[528,533,544,548],{"type":40,"tag":367,"props":529,"children":530},{},[531],{"type":45,"value":532},"Merge conflict",{"type":40,"tag":367,"props":534,"children":535},{},[536,542],{"type":40,"tag":85,"props":537,"children":539},{"className":538},[],[540],{"type":45,"value":541},"Version conflict",{"type":45,"value":543}," (name\u002Fdup-task now warn+merge)",{"type":40,"tag":367,"props":545,"children":546},{},[547],{"type":45,"value":393},{"type":40,"tag":367,"props":549,"children":550},{},[551],{"type":45,"value":552},"CLI output (multi-file)",{"type":40,"tag":335,"props":554,"children":555},{},[556,561,577,581],{"type":40,"tag":367,"props":557,"children":558},{},[559],{"type":45,"value":560},"Batch \u002F CSV",{"type":40,"tag":367,"props":562,"children":563},{},[564,570,571],{"type":40,"tag":85,"props":565,"children":567},{"className":566},[],[568],{"type":45,"value":569},"CSV file",{"type":45,"value":382},{"type":40,"tag":85,"props":572,"children":574},{"className":573},[],[575],{"type":45,"value":576},"sflow_config_file column",{"type":40,"tag":367,"props":578,"children":579},{},[580],{"type":45,"value":393},{"type":40,"tag":367,"props":582,"children":583},{},[584],{"type":45,"value":424},{"type":40,"tag":335,"props":586,"children":587},{},[588,593,609,614],{"type":40,"tag":367,"props":589,"children":590},{},[591],{"type":45,"value":592},"SLURM",{"type":40,"tag":367,"props":594,"children":595},{},[596,602,603],{"type":40,"tag":85,"props":597,"children":599},{"className":598},[],[600],{"type":45,"value":601},"salloc failed",{"type":45,"value":382},{"type":40,"tag":85,"props":604,"children":606},{"className":605},[],[607],{"type":45,"value":608},"sbatch failed",{"type":40,"tag":367,"props":610,"children":611},{},[612],{"type":45,"value":613},"runtime",{"type":40,"tag":367,"props":615,"children":616},{},[617],{"type":45,"value":618},"CLI \u002F sbatch stderr",{"type":40,"tag":335,"props":620,"children":621},{},[622,627,651,655],{"type":40,"tag":367,"props":623,"children":624},{},[625],{"type":45,"value":626},"Kubernetes",{"type":40,"tag":367,"props":628,"children":629},{},[630,636,638,644,645],{"type":40,"tag":85,"props":631,"children":633},{"className":632},[],[634],{"type":45,"value":635},"kubectl",{"type":45,"value":637},", RBAC\u002Fpreflight, pod ",{"type":40,"tag":85,"props":639,"children":641},{"className":640},[],[642],{"type":45,"value":643},"Pending",{"type":45,"value":199},{"type":40,"tag":85,"props":646,"children":648},{"className":647},[],[649],{"type":45,"value":650},"ImagePullBackOff",{"type":40,"tag":367,"props":652,"children":653},{},[654],{"type":45,"value":613},{"type":40,"tag":367,"props":656,"children":657},{},[658,660],{"type":45,"value":659},"CLI \u002F sflow.log \u002F ",{"type":40,"tag":85,"props":661,"children":663},{"className":662},[],[664],{"type":45,"value":665},"kubectl describe",{"type":40,"tag":335,"props":667,"children":668},{},[669,674,685,689],{"type":40,"tag":367,"props":670,"children":671},{},[672],{"type":45,"value":673},"Docker",{"type":40,"tag":367,"props":675,"children":676},{},[677,683],{"type":40,"tag":85,"props":678,"children":680},{"className":679},[],[681],{"type":45,"value":682},"docker run",{"type":45,"value":684}," errors, remote host, NVIDIA toolkit",{"type":40,"tag":367,"props":686,"children":687},{},[688],{"type":45,"value":613},{"type":40,"tag":367,"props":690,"children":691},{},[692],{"type":40,"tag":85,"props":693,"children":695},{"className":694},[],[696],{"type":45,"value":141},{"type":40,"tag":335,"props":698,"children":699},{},[700,705,723,727],{"type":40,"tag":367,"props":701,"children":702},{},[703],{"type":45,"value":704},"Storage \u002F uploads",{"type":40,"tag":367,"props":706,"children":707},{},[708,714,715,721],{"type":40,"tag":85,"props":709,"children":711},{"className":710},[],[712],{"type":45,"value":713},"requires boto3",{"type":45,"value":382},{"type":40,"tag":85,"props":716,"children":718},{"className":717},[],[719],{"type":45,"value":720},"No AWS credentials",{"type":45,"value":722},", upload failures",{"type":40,"tag":367,"props":724,"children":725},{},[726],{"type":45,"value":613},{"type":40,"tag":367,"props":728,"children":729},{},[730],{"type":45,"value":731},"CLI \u002F dry-run \u002F sflow.log",{"type":40,"tag":335,"props":733,"children":734},{},[735,740,745,749],{"type":40,"tag":367,"props":736,"children":737},{},[738],{"type":45,"value":739},"Probe timeout",{"type":40,"tag":367,"props":741,"children":742},{},[743],{"type":45,"value":744},"Task stuck waiting",{"type":40,"tag":367,"props":746,"children":747},{},[748],{"type":45,"value":613},{"type":40,"tag":367,"props":750,"children":751},{},[752],{"type":40,"tag":85,"props":753,"children":755},{"className":754},[],[756],{"type":45,"value":141},{"type":40,"tag":335,"props":758,"children":759},{},[760,765,776,780],{"type":40,"tag":367,"props":761,"children":762},{},[763],{"type":45,"value":764},"Task failure",{"type":40,"tag":367,"props":766,"children":767},{},[768,774],{"type":40,"tag":85,"props":769,"children":771},{"className":770},[],[772],{"type":45,"value":773},"Traceback",{"type":45,"value":775},", non-zero exit",{"type":40,"tag":367,"props":777,"children":778},{},[779],{"type":45,"value":613},{"type":40,"tag":367,"props":781,"children":782},{},[783],{"type":40,"tag":85,"props":784,"children":786},{"className":785},[],[787],{"type":45,"value":141},{"type":40,"tag":109,"props":789,"children":791},{"id":790},"step-3-locate-the-evidence",[792],{"type":45,"value":793},"Step 3 — Locate the evidence",{"type":40,"tag":145,"props":795,"children":799},{"className":796,"code":798,"language":45,"meta":150},[797],"language-text","\u003Coutput_dir>\u002F\u003Crun_id>\u002F\n  sflow.log                    # orchestrator log (launch + status lines)\n  sflow_summary.log            # live per-task status + failure hints\n  *_cmds.log                   # command-only launch logs, by command family\n  results.json                 # workflow-level parsed metrics (aggregated from task result: blocks)\n  sflow_monitor.log            # hardware monitor overview (if monitor: enabled — local\u002Fslurm\u002Fdocker only, NOT k8s)\n  sflow_monitor\u002F               # detailed monitor reports (csv\u002Fsvg\u002Fpng) + raw\u002F samples\n  \u003Ctask_name>\u002F\u003Ctask_name>.log  # task stdout\u002Fstderr — the crash lives here\n  \u003Ctask_name>\u002Fresult.json      # that task's parsed metrics (if it declares a result: block)\n  \u003Ctask_name>_0\u002F               # replica 0\n",[800],{"type":40,"tag":85,"props":801,"children":802},{"__ignoreMap":150},[803],{"type":45,"value":798},{"type":40,"tag":70,"props":805,"children":806},{},[807,916,956,990],{"type":40,"tag":74,"props":808,"children":809},{},[810,815,817,823,825,831,833,839,840,846,847,853,855,861,863,868,870,876,878,884,886,891,893,899,901,906,908,914],{"type":40,"tag":54,"props":811,"children":812},{},[813],{"type":45,"value":814},"Parsed results:",{"type":45,"value":816}," a task with a ",{"type":40,"tag":85,"props":818,"children":820},{"className":819},[],[821],{"type":45,"value":822},"result:",{"type":45,"value":824}," block writes ",{"type":40,"tag":85,"props":826,"children":828},{"className":827},[],[829],{"type":45,"value":830},"\u003Ctask>\u002Fresult.json",{"type":45,"value":832},"\n(",{"type":40,"tag":85,"props":834,"children":836},{"className":835},[],[837],{"type":45,"value":838},"ok",{"type":45,"value":199},{"type":40,"tag":85,"props":841,"children":843},{"className":842},[],[844],{"type":45,"value":845},"values",{"type":45,"value":199},{"type":40,"tag":85,"props":848,"children":850},{"className":849},[],[851],{"type":45,"value":852},"errors",{"type":45,"value":854},") and updates the run-level ",{"type":40,"tag":85,"props":856,"children":858},{"className":857},[],[859],{"type":45,"value":860},"results.json",{"type":45,"value":862},". Result parsing is\n",{"type":40,"tag":54,"props":864,"children":865},{},[866],{"type":45,"value":867},"best-effort",{"type":45,"value":869}," — a missed regex or failed ",{"type":40,"tag":85,"props":871,"children":873},{"className":872},[],[874],{"type":45,"value":875},"required:",{"type":45,"value":877}," spec logs a warning and sets\n",{"type":40,"tag":85,"props":879,"children":881},{"className":880},[],[882],{"type":45,"value":883},"ok=false",{"type":45,"value":885},", but does not by itself fail the task; check ",{"type":40,"tag":85,"props":887,"children":889},{"className":888},[],[890],{"type":45,"value":852},{"type":45,"value":892}," there if a metric is empty.\nParsing runs at task finalize, so a task shows ",{"type":40,"tag":85,"props":894,"children":896},{"className":895},[],[897],{"type":45,"value":898},"COMPLETED",{"type":45,"value":900}," only ",{"type":40,"tag":62,"props":902,"children":903},{},[904],{"type":45,"value":905},"after",{"type":45,"value":907}," it (and any\n",{"type":40,"tag":85,"props":909,"children":911},{"className":910},[],[912],{"type":45,"value":913},"uploads:",{"type":45,"value":915},") finish.",{"type":40,"tag":74,"props":917,"children":918},{},[919,924,926,931,932,938,940,946,948,954],{"type":40,"tag":54,"props":920,"children":921},{},[922],{"type":45,"value":923},"No monitor output?",{"type":45,"value":925}," Hardware monitoring is unsupported on ",{"type":40,"tag":54,"props":927,"children":928},{},[929],{"type":45,"value":930},"kubernetes",{"type":45,"value":832},{"type":40,"tag":85,"props":933,"children":935},{"className":934},[],[936],{"type":45,"value":937},"supports_host_monitoring=False",{"type":45,"value":939},") — sflow logs \"node-level hardware monitoring is not\nimplemented for this backend yet\" and produces no ",{"type":40,"tag":85,"props":941,"children":943},{"className":942},[],[944],{"type":45,"value":945},"sflow_monitor*",{"type":45,"value":947},". On local\u002Fslurm\u002Fdocker,\nan empty monitor usually means the ",{"type":40,"tag":85,"props":949,"children":951},{"className":950},[],[952],{"type":45,"value":953},"used_by_tasks",{"type":45,"value":955}," target resolved to no nodes.",{"type":40,"tag":74,"props":957,"children":958},{},[959,964,966,972,974,980,982,988],{"type":40,"tag":54,"props":960,"children":961},{},[962],{"type":45,"value":963},"Batch jobs:",{"type":45,"value":965}," also read the generated ",{"type":40,"tag":85,"props":967,"children":969},{"className":968},[],[970],{"type":45,"value":971},".sh",{"type":45,"value":973}," and sbatch stdout\u002Fstderr\n(",{"type":40,"tag":85,"props":975,"children":977},{"className":976},[],[978],{"type":45,"value":979},"sflow_output\u002F%j-sflow-submit.out",{"type":45,"value":981}," \u002F ",{"type":40,"tag":85,"props":983,"children":985},{"className":984},[],[986],{"type":45,"value":987},".err",{"type":45,"value":989},").",{"type":40,"tag":74,"props":991,"children":992},{},[993,998,1000,1006,1008,1014],{"type":40,"tag":54,"props":994,"children":995},{},[996],{"type":45,"value":997},"Kubernetes:",{"type":45,"value":999}," ",{"type":40,"tag":85,"props":1001,"children":1003},{"className":1002},[],[1004],{"type":45,"value":1005},"kubectl describe pod \u003Cpod>",{"type":45,"value":1007}," and ",{"type":40,"tag":85,"props":1009,"children":1011},{"className":1010},[],[1012],{"type":45,"value":1013},"kubectl get events",{"type":45,"value":1015}," for\nscheduling\u002Fimage issues the pod log can't show.",{"type":40,"tag":109,"props":1017,"children":1019},{"id":1018},"step-4-root-cause-fix",[1020],{"type":45,"value":1021},"Step 4 — Root-cause & fix",{"type":40,"tag":1023,"props":1024,"children":1026},"h3",{"id":1025},"_4a-launch-validation-fix-in-the-yaml-re-run-dry-run",[1027,1029],{"type":45,"value":1028},"4a. Launch \u002F validation — fix in the YAML, re-run ",{"type":40,"tag":85,"props":1030,"children":1032},{"className":1031},[],[1033],{"type":45,"value":90},{"type":40,"tag":327,"props":1035,"children":1036},{},[1037,1053],{"type":40,"tag":331,"props":1038,"children":1039},{},[1040],{"type":40,"tag":335,"props":1041,"children":1042},{},[1043,1048],{"type":40,"tag":339,"props":1044,"children":1045},{},[1046],{"type":45,"value":1047},"Error text",{"type":40,"tag":339,"props":1049,"children":1050},{},[1051],{"type":45,"value":1052},"Fix",{"type":40,"tag":360,"props":1054,"children":1055},{},[1056,1081,1098,1115,1148,1173,1190,1214,1237,1259],{"type":40,"tag":335,"props":1057,"children":1058},{},[1059,1068],{"type":40,"tag":367,"props":1060,"children":1061},{},[1062],{"type":40,"tag":85,"props":1063,"children":1065},{"className":1064},[],[1066],{"type":45,"value":1067},"Configuration file not found: \u003Cpath>",{"type":40,"tag":367,"props":1069,"children":1070},{},[1071,1073,1079],{"type":45,"value":1072},"Check the ",{"type":40,"tag":85,"props":1074,"children":1076},{"className":1075},[],[1077],{"type":45,"value":1078},"-f",{"type":45,"value":1080}," path",{"type":40,"tag":335,"props":1082,"children":1083},{},[1084,1093],{"type":40,"tag":367,"props":1085,"children":1086},{},[1087],{"type":40,"tag":85,"props":1088,"children":1090},{"className":1089},[],[1091],{"type":45,"value":1092},"Error parsing YAML configuration: \u003Cdetail>",{"type":40,"tag":367,"props":1094,"children":1095},{},[1096],{"type":45,"value":1097},"Fix YAML syntax at the indicated line",{"type":40,"tag":335,"props":1099,"children":1100},{},[1101,1110],{"type":40,"tag":367,"props":1102,"children":1103},{},[1104],{"type":40,"tag":85,"props":1105,"children":1107},{"className":1106},[],[1108],{"type":45,"value":1109},"Configuration validation failed: \u003Cdetail>",{"type":40,"tag":367,"props":1111,"children":1112},{},[1113],{"type":45,"value":1114},"Check field types\u002Fvalues (schema-reference.md)",{"type":40,"tag":335,"props":1116,"children":1117},{},[1118,1127],{"type":40,"tag":367,"props":1119,"children":1120},{},[1121],{"type":40,"tag":85,"props":1122,"children":1124},{"className":1123},[],[1125],{"type":45,"value":1126},"Operator type 'docker' is not valid",{"type":40,"tag":367,"props":1128,"children":1129},{},[1130,1132,1138,1140,1146],{"type":45,"value":1131},"Use ",{"type":40,"tag":85,"props":1133,"children":1135},{"className":1134},[],[1136],{"type":45,"value":1137},"type: docker_run",{"type":45,"value":1139}," for the operator (backend is ",{"type":40,"tag":85,"props":1141,"children":1143},{"className":1142},[],[1144],{"type":45,"value":1145},"type: docker",{"type":45,"value":1147},")",{"type":40,"tag":335,"props":1149,"children":1150},{},[1151,1160],{"type":40,"tag":367,"props":1152,"children":1153},{},[1154],{"type":40,"tag":85,"props":1155,"children":1157},{"className":1156},[],[1158],{"type":45,"value":1159},"Variable '\u003Ckey>' ... is not defined",{"type":40,"tag":367,"props":1161,"children":1162},{},[1163,1165,1171],{"type":45,"value":1164},"Declare the variable in YAML before ",{"type":40,"tag":85,"props":1166,"children":1168},{"className":1167},[],[1169],{"type":45,"value":1170},"--set",{"type":45,"value":1172},"-ing it",{"type":40,"tag":335,"props":1174,"children":1175},{},[1176,1185],{"type":40,"tag":367,"props":1177,"children":1178},{},[1179],{"type":40,"tag":85,"props":1180,"children":1182},{"className":1181},[],[1183],{"type":45,"value":1184},"Undefined variable in expression",{"type":40,"tag":367,"props":1186,"children":1187},{},[1188],{"type":45,"value":1189},"Fix spelling; ensure the variable is declared",{"type":40,"tag":335,"props":1191,"children":1192},{},[1193,1201],{"type":40,"tag":367,"props":1194,"children":1195},{},[1196],{"type":40,"tag":85,"props":1197,"children":1199},{"className":1198},[],[1200],{"type":45,"value":448},{"type":40,"tag":367,"props":1202,"children":1203},{},[1204,1206,1212],{"type":45,"value":1205},"Fix the ",{"type":40,"tag":85,"props":1207,"children":1209},{"className":1208},[],[1210],{"type":45,"value":1211},"${{ }}",{"type":45,"value":1213}," Jinja2 syntax",{"type":40,"tag":335,"props":1215,"children":1216},{},[1217,1225],{"type":40,"tag":367,"props":1218,"children":1219},{},[1220],{"type":40,"tag":85,"props":1221,"children":1223},{"className":1222},[],[1224],{"type":45,"value":473},{"type":40,"tag":367,"props":1226,"children":1227},{},[1228,1229,1235],{"type":45,"value":1205},{"type":40,"tag":85,"props":1230,"children":1232},{"className":1231},[],[1233],{"type":45,"value":1234},"fs:\u002F\u002F",{"type":45,"value":1236}," path or create it",{"type":40,"tag":335,"props":1238,"children":1239},{},[1240,1248],{"type":40,"tag":367,"props":1241,"children":1242},{},[1243],{"type":40,"tag":85,"props":1244,"children":1246},{"className":1245},[],[1247],{"type":45,"value":541},{"type":40,"tag":367,"props":1249,"children":1250},{},[1251,1253],{"type":45,"value":1252},"Every file must use ",{"type":40,"tag":85,"props":1254,"children":1256},{"className":1255},[],[1257],{"type":45,"value":1258},"version: \"0.1\"",{"type":40,"tag":335,"props":1260,"children":1261},{},[1262,1271],{"type":40,"tag":367,"props":1263,"children":1264},{},[1265],{"type":40,"tag":85,"props":1266,"children":1268},{"className":1267},[],[1269],{"type":45,"value":1270},"CSV file must contain a 'sflow_config_file' column",{"type":40,"tag":367,"props":1272,"children":1273},{},[1274],{"type":45,"value":1275},"Add the required column to the CSV",{"type":40,"tag":1023,"props":1277,"children":1279},{"id":1278},"_4b-runtime-read-the-logs-backend-launch-then-task-crash",[1280],{"type":45,"value":1281},"4b. Runtime — read the logs (backend launch, then task crash)",{"type":40,"tag":327,"props":1283,"children":1284},{},[1285,1300],{"type":40,"tag":331,"props":1286,"children":1287},{},[1288],{"type":40,"tag":335,"props":1289,"children":1290},{},[1291,1296],{"type":40,"tag":339,"props":1292,"children":1293},{},[1294],{"type":45,"value":1295},"Symptom",{"type":40,"tag":339,"props":1297,"children":1298},{},[1299],{"type":45,"value":1052},{"type":40,"tag":360,"props":1301,"children":1302},{},[1303,1339,1368,1415,1451,1469,1555,1579,1592,1611,1632],{"type":40,"tag":335,"props":1304,"children":1305},{},[1306,1320],{"type":40,"tag":367,"props":1307,"children":1308},{},[1309,1314,1315],{"type":40,"tag":85,"props":1310,"children":1312},{"className":1311},[],[1313],{"type":45,"value":601},{"type":45,"value":981},{"type":40,"tag":85,"props":1316,"children":1318},{"className":1317},[],[1319],{"type":45,"value":608},{"type":40,"tag":367,"props":1321,"children":1322},{},[1323,1325,1331,1332,1338],{"type":45,"value":1324},"Check partition\u002Faccount\u002Favailability (",{"type":40,"tag":85,"props":1326,"children":1328},{"className":1327},[],[1329],{"type":45,"value":1330},"sinfo",{"type":45,"value":382},{"type":40,"tag":85,"props":1333,"children":1335},{"className":1334},[],[1336],{"type":45,"value":1337},"sacctmgr",{"type":45,"value":1147},{"type":40,"tag":335,"props":1340,"children":1341},{},[1342,1347],{"type":40,"tag":367,"props":1343,"children":1344},{},[1345],{"type":45,"value":1346},"kube preflight: missing permission \u002F namespace",{"type":40,"tag":367,"props":1348,"children":1349},{},[1350,1352,1358,1360,1366],{"type":45,"value":1351},"Fix RBAC or ",{"type":40,"tag":85,"props":1353,"children":1355},{"className":1354},[],[1356],{"type":45,"value":1357},"--kube-namespace",{"type":45,"value":1359},"; ",{"type":40,"tag":85,"props":1361,"children":1363},{"className":1362},[],[1364],{"type":45,"value":1365},"SFLOW_SKIP_K8S_PREFLIGHT=1",{"type":45,"value":1367}," to bypass",{"type":40,"tag":335,"props":1369,"children":1370},{},[1371,1381],{"type":40,"tag":367,"props":1372,"children":1373},{},[1374,1376],{"type":45,"value":1375},"pod stuck ",{"type":40,"tag":85,"props":1377,"children":1379},{"className":1378},[],[1380],{"type":45,"value":643},{"type":40,"tag":367,"props":1382,"children":1383},{},[1384,1390,1392,1398,1399,1405,1407,1413],{"type":40,"tag":85,"props":1385,"children":1387},{"className":1386},[],[1388],{"type":45,"value":1389},"kubectl describe pod",{"type":45,"value":1391},": no node has the GPUs — check ",{"type":40,"tag":85,"props":1393,"children":1395},{"className":1394},[],[1396],{"type":45,"value":1397},"gpus_per_node",{"type":45,"value":382},{"type":40,"tag":85,"props":1400,"children":1402},{"className":1401},[],[1403],{"type":45,"value":1404},"scheduling",{"type":45,"value":1406}," mode, node-selector\u002Ftolerations, and that ",{"type":40,"tag":85,"props":1408,"children":1410},{"className":1409},[],[1411],{"type":45,"value":1412},"gpus.count",{"type":45,"value":1414}," is a multiple of the node count",{"type":40,"tag":335,"props":1416,"children":1417},{},[1418,1430],{"type":40,"tag":367,"props":1419,"children":1420},{},[1421,1423,1428],{"type":45,"value":1422},"pod ",{"type":40,"tag":85,"props":1424,"children":1426},{"className":1425},[],[1427],{"type":45,"value":650},{"type":45,"value":1429}," \u002F container not found",{"type":40,"tag":367,"props":1431,"children":1432},{},[1433,1435,1441,1443,1449],{"type":45,"value":1434},"Wrong ",{"type":40,"tag":85,"props":1436,"children":1438},{"className":1437},[],[1439],{"type":45,"value":1440},"image",{"type":45,"value":1442}," or missing ",{"type":40,"tag":85,"props":1444,"children":1446},{"className":1445},[],[1447],{"type":45,"value":1448},"image_pull_secrets",{"type":45,"value":1450},"; verify registry access",{"type":40,"tag":335,"props":1452,"children":1453},{},[1454,1459],{"type":40,"tag":367,"props":1455,"children":1456},{},[1457],{"type":45,"value":1458},"Docker GPU task fails",{"type":40,"tag":367,"props":1460,"children":1461},{},[1462,1464],{"type":45,"value":1463},"Install the NVIDIA container toolkit on the host; set ",{"type":40,"tag":85,"props":1465,"children":1467},{"className":1466},[],[1468],{"type":45,"value":1397},{"type":40,"tag":335,"props":1470,"children":1471},{},[1472,1477],{"type":40,"tag":367,"props":1473,"children":1474},{},[1475],{"type":45,"value":1476},"Probe timeout (task hangs waiting)",{"type":40,"tag":367,"props":1478,"children":1479},{},[1480,1482,1487,1489,1495,1497,1503,1505,1511,1512,1518,1520,1525,1527,1532,1534,1540,1541,1547,1549],{"type":45,"value":1481},"Read the ",{"type":40,"tag":54,"props":1483,"children":1484},{},[1485],{"type":45,"value":1486},"Probe Traces (last attempt)",{"type":45,"value":1488}," section of ",{"type":40,"tag":85,"props":1490,"children":1492},{"className":1491},[],[1493],{"type":45,"value":1494},"sflow_summary.log",{"type":45,"value":1496}," — it shows what each probe last saw (the last log line for ",{"type":40,"tag":85,"props":1498,"children":1500},{"className":1499},[],[1501],{"type":45,"value":1502},"log_watch",{"type":45,"value":1504},", the endpoint result for ",{"type":40,"tag":85,"props":1506,"children":1508},{"className":1507},[],[1509],{"type":45,"value":1510},"tcp_port",{"type":45,"value":199},{"type":40,"tag":85,"props":1513,"children":1515},{"className":1514},[],[1516],{"type":45,"value":1517},"http",{"type":45,"value":1519},"). Then compare server output to ",{"type":40,"tag":85,"props":1521,"children":1523},{"className":1522},[],[1524],{"type":45,"value":1502},{"type":45,"value":1526}," — it matches ",{"type":40,"tag":54,"props":1528,"children":1529},{},[1530],{"type":45,"value":1531},"literally",{"type":45,"value":1533}," unless prefixed ",{"type":40,"tag":85,"props":1535,"children":1537},{"className":1536},[],[1538],{"type":45,"value":1539},"re:",{"type":45,"value":199},{"type":40,"tag":85,"props":1542,"children":1544},{"className":1543},[],[1545],{"type":45,"value":1546},"regex:",{"type":45,"value":1548},"; adjust the pattern or ",{"type":40,"tag":85,"props":1550,"children":1552},{"className":1551},[],[1553],{"type":45,"value":1554},"timeout",{"type":40,"tag":335,"props":1556,"children":1557},{},[1558,1574],{"type":40,"tag":367,"props":1559,"children":1560},{},[1561,1566,1568],{"type":40,"tag":85,"props":1562,"children":1564},{"className":1563},[],[1565],{"type":45,"value":773},{"type":45,"value":1567}," in ",{"type":40,"tag":85,"props":1569,"children":1571},{"className":1570},[],[1572],{"type":45,"value":1573},"\u003Ctask>.log",{"type":40,"tag":367,"props":1575,"children":1576},{},[1577],{"type":45,"value":1578},"Read the traceback for the real cause (below are the usual ones)",{"type":40,"tag":335,"props":1580,"children":1581},{},[1582,1587],{"type":40,"tag":367,"props":1583,"children":1584},{},[1585],{"type":45,"value":1586},"CUDA OOM",{"type":40,"tag":367,"props":1588,"children":1589},{},[1590],{"type":45,"value":1591},"Reduce batch size, TP\u002FDP, or model size",{"type":40,"tag":335,"props":1593,"children":1594},{},[1595,1606],{"type":40,"tag":367,"props":1596,"children":1597},{},[1598,1604],{"type":40,"tag":85,"props":1599,"children":1601},{"className":1600},[],[1602],{"type":45,"value":1603},"Address already in use",{"type":45,"value":1605}," \u002F port conflict",{"type":40,"tag":367,"props":1607,"children":1608},{},[1609],{"type":45,"value":1610},"Kill stale processes or offset the port",{"type":40,"tag":335,"props":1612,"children":1613},{},[1614,1619],{"type":40,"tag":367,"props":1615,"children":1616},{},[1617],{"type":45,"value":1618},"NCCL \u002F communication errors",{"type":40,"tag":367,"props":1620,"children":1621},{},[1622,1624,1630],{"type":45,"value":1623},"Set ",{"type":40,"tag":85,"props":1625,"children":1627},{"className":1626},[],[1628],{"type":45,"value":1629},"NCCL_SOCKET_IFNAME",{"type":45,"value":1631},"; check the inter-node network",{"type":40,"tag":335,"props":1633,"children":1634},{},[1635,1653],{"type":40,"tag":367,"props":1636,"children":1637},{},[1638,1644,1645,1651],{"type":40,"tag":85,"props":1639,"children":1641},{"className":1640},[],[1642],{"type":45,"value":1643},"S3 storage requires boto3",{"type":45,"value":981},{"type":40,"tag":85,"props":1646,"children":1648},{"className":1647},[],[1649],{"type":45,"value":1650},"no AWS credentials",{"type":45,"value":1652}," \u002F upload fails",{"type":40,"tag":367,"props":1654,"children":1655},{},[1656,1662,1664,1670,1671,1677],{"type":40,"tag":85,"props":1657,"children":1659},{"className":1658},[],[1660],{"type":45,"value":1661},"pip install 'sflow[s3]'",{"type":45,"value":1663},"; provide creds via the boto3 chain (",{"type":40,"tag":85,"props":1665,"children":1667},{"className":1666},[],[1668],{"type":45,"value":1669},"AWS_*",{"type":45,"value":981},{"type":40,"tag":85,"props":1672,"children":1674},{"className":1673},[],[1675],{"type":45,"value":1676},"~\u002F.aws\u002Fcredentials",{"type":45,"value":1678}," \u002F IAM — never in YAML); verify bucket\u002Fregion",{"type":40,"tag":48,"props":1680,"children":1681},{},[1682,1684,1690],{"type":45,"value":1683},"For a pattern-by-pattern catalog (exact message, cause, and fix per error), see\n",{"type":40,"tag":1685,"props":1686,"children":1688},"a",{"href":1687},"error-catalog.md",[1689],{"type":45,"value":1687},{"type":45,"value":1691},".",{"type":40,"tag":109,"props":1693,"children":1695},{"id":1694},"step-5-re-verify",[1696],{"type":45,"value":1697},"Step 5 — Re-verify",{"type":40,"tag":48,"props":1699,"children":1700},{},[1701],{"type":45,"value":1702},"Re-run the relevant capture command — clean exit means fixed:",{"type":40,"tag":145,"props":1704,"children":1706},{"className":147,"code":1705,"language":149,"meta":150,"style":150},"sflow run -f config.yaml --dry-run          # launch fixes: exit 0 = ready\npython scripts\u002Ftriage.py \u003Crun_dir>\u002F          # runtime fixes: no failed tasks\n",[1707],{"type":40,"tag":85,"props":1708,"children":1709},{"__ignoreMap":150},[1710,1740],{"type":40,"tag":156,"props":1711,"children":1712},{"class":158,"line":159},[1713,1717,1721,1725,1729,1734],{"type":40,"tag":156,"props":1714,"children":1715},{"style":163},[1716],{"type":45,"value":248},{"type":40,"tag":156,"props":1718,"children":1719},{"style":169},[1720],{"type":45,"value":253},{"type":40,"tag":156,"props":1722,"children":1723},{"style":169},[1724],{"type":45,"value":258},{"type":40,"tag":156,"props":1726,"children":1727},{"style":169},[1728],{"type":45,"value":263},{"type":40,"tag":156,"props":1730,"children":1731},{"style":169},[1732],{"type":45,"value":1733}," --dry-run",{"type":40,"tag":156,"props":1735,"children":1737},{"style":1736},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1738],{"type":45,"value":1739},"          # launch fixes: exit 0 = ready\n",{"type":40,"tag":156,"props":1741,"children":1742},{"class":158,"line":271},[1743,1747,1751,1755,1760,1764,1768,1772],{"type":40,"tag":156,"props":1744,"children":1745},{"style":163},[1746],{"type":45,"value":166},{"type":40,"tag":156,"props":1748,"children":1749},{"style":169},[1750],{"type":45,"value":172},{"type":40,"tag":156,"props":1752,"children":1753},{"style":175},[1754],{"type":45,"value":178},{"type":40,"tag":156,"props":1756,"children":1757},{"style":169},[1758],{"type":45,"value":1759},"run_di",{"type":40,"tag":156,"props":1761,"children":1762},{"style":186},[1763],{"type":45,"value":189},{"type":40,"tag":156,"props":1765,"children":1766},{"style":175},[1767],{"type":45,"value":194},{"type":40,"tag":156,"props":1769,"children":1770},{"style":169},[1771],{"type":45,"value":199},{"type":40,"tag":156,"props":1773,"children":1774},{"style":1736},[1775],{"type":45,"value":1776},"          # runtime fixes: no failed tasks\n",{"type":40,"tag":109,"props":1778,"children":1780},{"id":1779},"diagnostic-commands-reference",[1781],{"type":45,"value":1782},"Diagnostic commands (reference)",{"type":40,"tag":145,"props":1784,"children":1786},{"className":147,"code":1785,"language":149,"meta":150,"style":150},"sflow run -f config.yaml --dry-run                       # validate + resolve without executing\nsflow compose f1.yaml f2.yaml --resolve -o merged.yaml   # see the merged + resolved config\npython scripts\u002Ftriage.py \u003Coutput_dir>\u002F\u003Crun_id>\u002F          # one-shot: summary + logs -> root cause\npython scripts\u002Fsummarize_run.py \u003Coutput_dir>\u002F\u003Crun_id>\u002F   #   per-task status + tracebacks only\npython scripts\u002Fparse_sflow_errors.py \u003Crun_id>\u002Fsflow.log  #   categorize an orchestrator log\nsinfo -p \u003Cpartition>; sacctmgr show account \u003Cname>       # Slurm environment\nkubectl -n \u003Cns> get pods && kubectl -n \u003Cns> describe pod \u003Cpod>   # Kubernetes scheduling\u002Fimage\nkubectl -n \u003Cns> get events --sort-by=.lastTimestamp\n",[1787],{"type":40,"tag":85,"props":1788,"children":1789},{"__ignoreMap":150},[1790,1818,1860,1917,1975,2013,2082,2184],{"type":40,"tag":156,"props":1791,"children":1792},{"class":158,"line":159},[1793,1797,1801,1805,1809,1813],{"type":40,"tag":156,"props":1794,"children":1795},{"style":163},[1796],{"type":45,"value":248},{"type":40,"tag":156,"props":1798,"children":1799},{"style":169},[1800],{"type":45,"value":253},{"type":40,"tag":156,"props":1802,"children":1803},{"style":169},[1804],{"type":45,"value":258},{"type":40,"tag":156,"props":1806,"children":1807},{"style":169},[1808],{"type":45,"value":263},{"type":40,"tag":156,"props":1810,"children":1811},{"style":169},[1812],{"type":45,"value":1733},{"type":40,"tag":156,"props":1814,"children":1815},{"style":1736},[1816],{"type":45,"value":1817},"                       # validate + resolve without executing\n",{"type":40,"tag":156,"props":1819,"children":1820},{"class":158,"line":271},[1821,1825,1830,1835,1840,1845,1850,1855],{"type":40,"tag":156,"props":1822,"children":1823},{"style":163},[1824],{"type":45,"value":248},{"type":40,"tag":156,"props":1826,"children":1827},{"style":169},[1828],{"type":45,"value":1829}," compose",{"type":40,"tag":156,"props":1831,"children":1832},{"style":169},[1833],{"type":45,"value":1834}," f1.yaml",{"type":40,"tag":156,"props":1836,"children":1837},{"style":169},[1838],{"type":45,"value":1839}," f2.yaml",{"type":40,"tag":156,"props":1841,"children":1842},{"style":169},[1843],{"type":45,"value":1844}," --resolve",{"type":40,"tag":156,"props":1846,"children":1847},{"style":169},[1848],{"type":45,"value":1849}," -o",{"type":40,"tag":156,"props":1851,"children":1852},{"style":169},[1853],{"type":45,"value":1854}," merged.yaml",{"type":40,"tag":156,"props":1856,"children":1857},{"style":1736},[1858],{"type":45,"value":1859},"   # see the merged + resolved config\n",{"type":40,"tag":156,"props":1861,"children":1863},{"class":158,"line":1862},3,[1864,1868,1872,1876,1880,1884,1888,1892,1896,1900,1904,1908,1912],{"type":40,"tag":156,"props":1865,"children":1866},{"style":163},[1867],{"type":45,"value":166},{"type":40,"tag":156,"props":1869,"children":1870},{"style":169},[1871],{"type":45,"value":172},{"type":40,"tag":156,"props":1873,"children":1874},{"style":175},[1875],{"type":45,"value":178},{"type":40,"tag":156,"props":1877,"children":1878},{"style":169},[1879],{"type":45,"value":183},{"type":40,"tag":156,"props":1881,"children":1882},{"style":186},[1883],{"type":45,"value":189},{"type":40,"tag":156,"props":1885,"children":1886},{"style":175},[1887],{"type":45,"value":194},{"type":40,"tag":156,"props":1889,"children":1890},{"style":169},[1891],{"type":45,"value":199},{"type":40,"tag":156,"props":1893,"children":1894},{"style":175},[1895],{"type":45,"value":204},{"type":40,"tag":156,"props":1897,"children":1898},{"style":169},[1899],{"type":45,"value":209},{"type":40,"tag":156,"props":1901,"children":1902},{"style":186},[1903],{"type":45,"value":214},{"type":40,"tag":156,"props":1905,"children":1906},{"style":175},[1907],{"type":45,"value":194},{"type":40,"tag":156,"props":1909,"children":1910},{"style":169},[1911],{"type":45,"value":199},{"type":40,"tag":156,"props":1913,"children":1914},{"style":1736},[1915],{"type":45,"value":1916},"          # one-shot: summary + logs -> root cause\n",{"type":40,"tag":156,"props":1918,"children":1920},{"class":158,"line":1919},4,[1921,1925,1930,1934,1938,1942,1946,1950,1954,1958,1962,1966,1970],{"type":40,"tag":156,"props":1922,"children":1923},{"style":163},[1924],{"type":45,"value":166},{"type":40,"tag":156,"props":1926,"children":1927},{"style":169},[1928],{"type":45,"value":1929}," scripts\u002Fsummarize_run.py",{"type":40,"tag":156,"props":1931,"children":1932},{"style":175},[1933],{"type":45,"value":178},{"type":40,"tag":156,"props":1935,"children":1936},{"style":169},[1937],{"type":45,"value":183},{"type":40,"tag":156,"props":1939,"children":1940},{"style":186},[1941],{"type":45,"value":189},{"type":40,"tag":156,"props":1943,"children":1944},{"style":175},[1945],{"type":45,"value":194},{"type":40,"tag":156,"props":1947,"children":1948},{"style":169},[1949],{"type":45,"value":199},{"type":40,"tag":156,"props":1951,"children":1952},{"style":175},[1953],{"type":45,"value":204},{"type":40,"tag":156,"props":1955,"children":1956},{"style":169},[1957],{"type":45,"value":209},{"type":40,"tag":156,"props":1959,"children":1960},{"style":186},[1961],{"type":45,"value":214},{"type":40,"tag":156,"props":1963,"children":1964},{"style":175},[1965],{"type":45,"value":194},{"type":40,"tag":156,"props":1967,"children":1968},{"style":169},[1969],{"type":45,"value":199},{"type":40,"tag":156,"props":1971,"children":1972},{"style":1736},[1973],{"type":45,"value":1974},"   #   per-task status + tracebacks only\n",{"type":40,"tag":156,"props":1976,"children":1978},{"class":158,"line":1977},5,[1979,1983,1987,1991,1995,1999,2003,2008],{"type":40,"tag":156,"props":1980,"children":1981},{"style":163},[1982],{"type":45,"value":166},{"type":40,"tag":156,"props":1984,"children":1985},{"style":169},[1986],{"type":45,"value":281},{"type":40,"tag":156,"props":1988,"children":1989},{"style":175},[1990],{"type":45,"value":178},{"type":40,"tag":156,"props":1992,"children":1993},{"style":169},[1994],{"type":45,"value":209},{"type":40,"tag":156,"props":1996,"children":1997},{"style":186},[1998],{"type":45,"value":214},{"type":40,"tag":156,"props":2000,"children":2001},{"style":175},[2002],{"type":45,"value":194},{"type":40,"tag":156,"props":2004,"children":2005},{"style":169},[2006],{"type":45,"value":2007},"\u002Fsflow.log",{"type":40,"tag":156,"props":2009,"children":2010},{"style":1736},[2011],{"type":45,"value":2012},"  #   categorize an orchestrator log\n",{"type":40,"tag":156,"props":2014,"children":2015},{"class":158,"line":27},[2016,2020,2025,2029,2034,2039,2044,2049,2054,2059,2063,2068,2073,2077],{"type":40,"tag":156,"props":2017,"children":2018},{"style":163},[2019],{"type":45,"value":1330},{"type":40,"tag":156,"props":2021,"children":2022},{"style":169},[2023],{"type":45,"value":2024}," -p",{"type":40,"tag":156,"props":2026,"children":2027},{"style":175},[2028],{"type":45,"value":178},{"type":40,"tag":156,"props":2030,"children":2031},{"style":169},[2032],{"type":45,"value":2033},"partitio",{"type":40,"tag":156,"props":2035,"children":2036},{"style":186},[2037],{"type":45,"value":2038},"n",{"type":40,"tag":156,"props":2040,"children":2041},{"style":175},[2042],{"type":45,"value":2043},">;",{"type":40,"tag":156,"props":2045,"children":2046},{"style":163},[2047],{"type":45,"value":2048}," sacctmgr",{"type":40,"tag":156,"props":2050,"children":2051},{"style":169},[2052],{"type":45,"value":2053}," show",{"type":40,"tag":156,"props":2055,"children":2056},{"style":169},[2057],{"type":45,"value":2058}," account",{"type":40,"tag":156,"props":2060,"children":2061},{"style":175},[2062],{"type":45,"value":178},{"type":40,"tag":156,"props":2064,"children":2065},{"style":169},[2066],{"type":45,"value":2067},"nam",{"type":40,"tag":156,"props":2069,"children":2070},{"style":186},[2071],{"type":45,"value":2072},"e",{"type":40,"tag":156,"props":2074,"children":2075},{"style":175},[2076],{"type":45,"value":194},{"type":40,"tag":156,"props":2078,"children":2079},{"style":1736},[2080],{"type":45,"value":2081},"       # Slurm environment\n",{"type":40,"tag":156,"props":2083,"children":2085},{"class":158,"line":2084},7,[2086,2090,2095,2099,2103,2108,2112,2117,2122,2127,2132,2136,2140,2144,2148,2152,2157,2162,2166,2171,2175,2179],{"type":40,"tag":156,"props":2087,"children":2088},{"style":163},[2089],{"type":45,"value":635},{"type":40,"tag":156,"props":2091,"children":2092},{"style":169},[2093],{"type":45,"value":2094}," -n",{"type":40,"tag":156,"props":2096,"children":2097},{"style":175},[2098],{"type":45,"value":178},{"type":40,"tag":156,"props":2100,"children":2101},{"style":169},[2102],{"type":45,"value":2038},{"type":40,"tag":156,"props":2104,"children":2105},{"style":186},[2106],{"type":45,"value":2107},"s",{"type":40,"tag":156,"props":2109,"children":2110},{"style":175},[2111],{"type":45,"value":194},{"type":40,"tag":156,"props":2113,"children":2114},{"style":169},[2115],{"type":45,"value":2116}," get",{"type":40,"tag":156,"props":2118,"children":2119},{"style":169},[2120],{"type":45,"value":2121}," pods",{"type":40,"tag":156,"props":2123,"children":2124},{"style":175},[2125],{"type":45,"value":2126}," &&",{"type":40,"tag":156,"props":2128,"children":2129},{"style":163},[2130],{"type":45,"value":2131}," kubectl",{"type":40,"tag":156,"props":2133,"children":2134},{"style":169},[2135],{"type":45,"value":2094},{"type":40,"tag":156,"props":2137,"children":2138},{"style":175},[2139],{"type":45,"value":178},{"type":40,"tag":156,"props":2141,"children":2142},{"style":169},[2143],{"type":45,"value":2038},{"type":40,"tag":156,"props":2145,"children":2146},{"style":186},[2147],{"type":45,"value":2107},{"type":40,"tag":156,"props":2149,"children":2150},{"style":175},[2151],{"type":45,"value":194},{"type":40,"tag":156,"props":2153,"children":2154},{"style":169},[2155],{"type":45,"value":2156}," describe",{"type":40,"tag":156,"props":2158,"children":2159},{"style":169},[2160],{"type":45,"value":2161}," pod",{"type":40,"tag":156,"props":2163,"children":2164},{"style":175},[2165],{"type":45,"value":178},{"type":40,"tag":156,"props":2167,"children":2168},{"style":169},[2169],{"type":45,"value":2170},"po",{"type":40,"tag":156,"props":2172,"children":2173},{"style":186},[2174],{"type":45,"value":214},{"type":40,"tag":156,"props":2176,"children":2177},{"style":175},[2178],{"type":45,"value":194},{"type":40,"tag":156,"props":2180,"children":2181},{"style":1736},[2182],{"type":45,"value":2183},"   # Kubernetes scheduling\u002Fimage\n",{"type":40,"tag":156,"props":2185,"children":2187},{"class":158,"line":2186},8,[2188,2192,2196,2200,2204,2208,2212,2216,2221],{"type":40,"tag":156,"props":2189,"children":2190},{"style":163},[2191],{"type":45,"value":635},{"type":40,"tag":156,"props":2193,"children":2194},{"style":169},[2195],{"type":45,"value":2094},{"type":40,"tag":156,"props":2197,"children":2198},{"style":175},[2199],{"type":45,"value":178},{"type":40,"tag":156,"props":2201,"children":2202},{"style":169},[2203],{"type":45,"value":2038},{"type":40,"tag":156,"props":2205,"children":2206},{"style":186},[2207],{"type":45,"value":2107},{"type":40,"tag":156,"props":2209,"children":2210},{"style":175},[2211],{"type":45,"value":194},{"type":40,"tag":156,"props":2213,"children":2214},{"style":169},[2215],{"type":45,"value":2116},{"type":40,"tag":156,"props":2217,"children":2218},{"style":169},[2219],{"type":45,"value":2220}," events",{"type":40,"tag":156,"props":2222,"children":2223},{"style":169},[2224],{"type":45,"value":2225}," --sort-by=.lastTimestamp\n",{"type":40,"tag":109,"props":2227,"children":2229},{"id":2228},"additional-resources",[2230],{"type":45,"value":2231},"Additional resources",{"type":40,"tag":70,"props":2233,"children":2234},{},[2235,2244],{"type":40,"tag":74,"props":2236,"children":2237},{},[2238,2240],{"type":45,"value":2239},"Exhaustive error patterns → ",{"type":40,"tag":1685,"props":2241,"children":2242},{"href":1687},[2243],{"type":45,"value":1687},{"type":40,"tag":74,"props":2245,"children":2246},{},[2247,2249,2257,2259,2266,2267,2274,2275],{"type":45,"value":2248},"Docs → ",{"type":40,"tag":1685,"props":2250,"children":2254},{"href":2251,"rel":2252},"https:\u002F\u002Fnvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Ffaq",[2253],"nofollow",[2255],{"type":45,"value":2256},"faq",{"type":45,"value":2258},",\n",{"type":40,"tag":1685,"props":2260,"children":2263},{"href":2261,"rel":2262},"https:\u002F\u002Fnvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Fcli",[2253],[2264],{"type":45,"value":2265},"cli",{"type":45,"value":2258},{"type":40,"tag":1685,"props":2268,"children":2271},{"href":2269,"rel":2270},"https:\u002F\u002Fnvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Fconfiguration",[2253],[2272],{"type":45,"value":2273},"configuration",{"type":45,"value":2258},{"type":40,"tag":1685,"props":2276,"children":2279},{"href":2277,"rel":2278},"https:\u002F\u002Fnvidia.github.io\u002Fnv-sflow\u002Fdocs\u002Fuser\u002Fbackends",[2253],[2280],{"type":45,"value":2281},"backends",{"type":40,"tag":2283,"props":2284,"children":2285},"style",{},[2286],{"type":45,"value":2287},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2289,"total":2445},[2290,2308,2325,2336,2348,2360,2373,2387,2400,2411,2425,2434],{"slug":2291,"name":2291,"fn":2292,"description":2293,"org":2294,"tags":2295,"stars":2305,"repoUrl":2306,"updatedAt":2307},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2296,2299,2302],{"name":2297,"slug":2298,"type":15},"Documentation","documentation",{"name":2300,"slug":2301,"type":15},"MCP","mcp",{"name":2303,"slug":2304,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2309,"name":2309,"fn":2310,"description":2311,"org":2312,"tags":2313,"stars":2322,"repoUrl":2323,"updatedAt":2324},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2314,2317,2320],{"name":2315,"slug":2316,"type":15},"Containers","containers",{"name":2318,"slug":2319,"type":15},"Deployment","deployment",{"name":2321,"slug":166,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2326,"name":2326,"fn":2327,"description":2328,"org":2329,"tags":2330,"stars":2322,"repoUrl":2323,"updatedAt":2335},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2331,2334],{"name":2332,"slug":2333,"type":15},"CI\u002FCD","ci-cd",{"name":2318,"slug":2319,"type":15},"2026-07-14T05:25:59.97109",{"slug":2337,"name":2337,"fn":2338,"description":2339,"org":2340,"tags":2341,"stars":2322,"repoUrl":2323,"updatedAt":2347},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2342,2343,2344],{"name":2332,"slug":2333,"type":15},{"name":2318,"slug":2319,"type":15},{"name":2345,"slug":2346,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2349,"name":2349,"fn":2350,"description":2351,"org":2352,"tags":2353,"stars":2322,"repoUrl":2323,"updatedAt":2359},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2354,2355,2356],{"name":18,"slug":19,"type":15},{"name":2345,"slug":2346,"type":15},{"name":2357,"slug":2358,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2361,"name":2361,"fn":2362,"description":2363,"org":2364,"tags":2365,"stars":2322,"repoUrl":2323,"updatedAt":2372},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2366,2369],{"name":2367,"slug":2368,"type":15},"Best Practices","best-practices",{"name":2370,"slug":2371,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2374,"name":2374,"fn":2375,"description":2376,"org":2377,"tags":2378,"stars":2322,"repoUrl":2323,"updatedAt":2386},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2379,2382,2385],{"name":2380,"slug":2381,"type":15},"Machine Learning","machine-learning",{"name":2383,"slug":2384,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2388,"name":2388,"fn":2389,"description":2390,"org":2391,"tags":2392,"stars":2322,"repoUrl":2323,"updatedAt":2399},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2393,2396],{"name":2394,"slug":2395,"type":15},"QA","qa",{"name":2397,"slug":2398,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":2401,"name":2401,"fn":2402,"description":2403,"org":2404,"tags":2405,"stars":2322,"repoUrl":2323,"updatedAt":2410},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2406,2407],{"name":2318,"slug":2319,"type":15},{"name":2408,"slug":2409,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":2412,"name":2412,"fn":2413,"description":2414,"org":2415,"tags":2416,"stars":2322,"repoUrl":2323,"updatedAt":2424},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2417,2420,2421],{"name":2418,"slug":2419,"type":15},"Code Review","code-review",{"name":2345,"slug":2346,"type":15},{"name":2422,"slug":2423,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2426,"name":2426,"fn":2427,"description":2428,"org":2429,"tags":2430,"stars":2322,"repoUrl":2323,"updatedAt":2433},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2431,2432],{"name":2394,"slug":2395,"type":15},{"name":2397,"slug":2398,"type":15},"2026-07-14T05:25:54.928983",{"slug":2435,"name":2435,"fn":2436,"description":2437,"org":2438,"tags":2439,"stars":2322,"repoUrl":2323,"updatedAt":2444},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2440,2443],{"name":2441,"slug":2442,"type":15},"Automation","automation",{"name":2332,"slug":2333,"type":15},"2026-07-30T05:29:03.275638",496,{"items":2447,"total":1862},[2448,2461,2468],{"slug":2449,"name":2449,"fn":2450,"description":2451,"org":2452,"tags":2453,"stars":23,"repoUrl":24,"updatedAt":2460},"sflow-code-review","review sflow code changes","Review sflow code changes for functional defects, modular by-purpose structure, duplicated logic that should be consolidated, and adequate unit + e2e CLI test coverage. Use when reviewing an sflow diff, branch, PR, staged changes, or when the user asks for a code review of sflow.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2454,2455,2458,2459],{"name":2418,"slug":2419,"type":15},{"name":2456,"slug":2457,"type":15},"Engineering","engineering",{"name":9,"slug":8,"type":15},{"name":2397,"slug":2398,"type":15},"2026-07-23T06:06:17.700089",{"slug":4,"name":4,"fn":5,"description":6,"org":2462,"tags":2463,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2464,2465,2466,2467],{"name":18,"slug":19,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},{"slug":2469,"name":2469,"fn":2470,"description":2471,"org":2472,"tags":2473,"stars":23,"repoUrl":24,"updatedAt":2481},"writing-sflow-yaml","configure sflow YAML workflows","Write, create, and modify sflow YAML workflow configuration files. Covers schema structure, variables, task DAGs, backends (local, slurm, docker, kubernetes), operators, probes, replicas, artifacts, result parsing, storage\u002Fuploads (S3), hardware monitoring, and modular composition. Use when the user asks to create an sflow YAML, configure a workflow, set up inference serving on Slurm or Kubernetes, or asks about sflow YAML syntax.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2474,2476,2477,2478],{"name":2475,"slug":2273,"type":15},"Configuration",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},{"name":2479,"slug":2480,"type":15},"YAML","yaml","2026-07-23T05:43:46.500651"]