[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-sop-by-action-eval":3,"mdc-tsaee1-key":34,"related-repo-nvidia-sop-by-action-eval":1982,"related-org-nvidia-sop-by-action-eval":2076},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"sop-by-action-eval","run by-action VLM evaluations","Use when running by-action VLM evaluation (per-action-clip inference + accuracy metrics) against the BP evaluation-ms HTTP API. Invoked as \u002Fsop-by-action-eval \u003Cinputs.yaml> [natural language parameter overrides]",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"Monitoring","monitoring","tag",{"name":17,"slug":18,"type":15},"Machine Learning","machine-learning",{"name":20,"slug":21,"type":15},"Evals","evals",{"name":9,"slug":8,"type":15},39,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fsop-monitoring-blueprints","2026-07-14T05:36:13.260585","CC-BY-4.0 AND Apache-2.0",12,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"Industrial SOP Monitoring Blueprints for Training & Inference","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002Fsop-monitoring-blueprints\u002Ftree\u002FHEAD\u002Fagentic\u002Fsop-agentic-ft\u002Fplugins\u002Fsop-evaluation-plugin\u002Fskills\u002Fsop-by-action-eval","---\nname: sop-by-action-eval\ndescription: Use when running by-action VLM evaluation (per-action-clip inference + accuracy metrics) against the BP evaluation-ms HTTP API. Invoked as \u002Fsop-by-action-eval \u003Cinputs.yaml> [natural language parameter overrides]\nlicense: \"CC-BY-4.0 AND Apache-2.0\"\n---\n\n# By-Action Evaluation (API-driven)\n\nRun per-action VLM evaluation by POSTing to the BP `evaluation-ms` HTTP service. The service decides on its own where to write output (a uuid-named directory under `\u003CRESULTS_ROOT>\u002F\u003Ceval_job_id>\u002F`), runs the inference subprocess, and updates Postgres + a job cache. This skill submits the request, polls until terminal, resolves the host-side output directory, and emits a structured JSON envelope on stdout.\n\nWhen this skill is invoked, follow the steps below in order. Stop immediately if any step fails — show the full error output to the user, but do NOT attempt to fix it automatically.\n\n**Bundled resources** (paths relative to this skill's directory):\n- `scripts\u002Feval_api_client.py` — single Python helper that POSTs, polls, resolves paths, prints the envelope.\n- `references\u002Finputs-template.yaml` — configuration template with all parameters documented.\n\n**Path conventions**:\n- `SKILL_DIR` = this skill's `scripts` directory (absolute path resolved at invocation time).\n\n## Step 0: Parse Overrides (if any)\n\nIf the user provided natural language parameter overrides alongside the inputs.yaml:\n\n1. Parse the overrides. If any part of the input cannot be clearly mapped to a yaml field, list the ambiguous parts and ask the user to clarify before proceeding. Map to the yaml structure (see `references\u002Finputs-template.yaml` for the full list):\n   - Required: `training_job_id`, `val_dataset_id`\n   - Optional: `eval_host`, `eval_port`, `host_results_root`, `backend`, `fps`, `temperature`, `top_p`, `checkpoint_step`, `resolution_config`, `gpu_id`, `poll_interval_sec`, `timeout_sec`\n\n2. Write the overrides to `\u002Ftmp\u002Fby_action_overrides.yaml` using the Write tool. The api client merges them onto the base inputs.yaml at invocation time — no separate merge step.\n\n3. Show the user what was overridden.\n\nIf no overrides, skip the overrides file.\n\n## Step 1: Preflight — eval-ms reachability\n\nBefore submitting, make sure the eval-ms service is up. Read `eval_host` and `eval_port` from inputs.yaml (defaults: `localhost:32090`) and probe the health endpoint:\n\n```bash\ncurl -fsS http:\u002F\u002F${EVAL_HOST:-localhost}:${EVAL_PORT:-32090}\u002Fhealth\n```\n\n- **Connection refused \u002F non-2xx** → stop and tell the user to bring eval-ms up (`docker compose up evaluation-ms` from the BP deployment root) before retrying.\n- **2xx** → continue.\n\n## Step 2: Submit + Poll\n\n`eval_api_client.py` is a **blocking** call. It POSTs the request, polls `\u002Fapi\u002Fv1\u002Fevaluation\u002Fstatus\u002F{eval_job_id}` every `poll_interval_sec` (default 20s) until terminal status (`completed` \u002F `failed` \u002F `cancelled`) or `timeout_sec` (default 3600), prints progress to stderr, and emits a single JSON envelope on the last stdout line. The script returns only when eval-ms reports a terminal state.\n\n```bash\npython3 SKILL_DIR\u002Feval_api_client.py by-action \u003Cinputs.yaml> [--overrides \u002Ftmp\u002Fby_action_overrides.yaml]\n```\n\n### Choosing run_in_background vs synchronous\n\nThe client is blocking either way — `run_in_background` only governs how *you* (the caller) wait for it.\n\n- **Interactive Claude Code (main agent)** — use `run_in_background: true` so the main loop is free for other work; you receive a completion notification carrying the script's stdout (the envelope).\n- **Subagent or any non-interactive context** — call **synchronously** (do NOT pass `run_in_background: true`). A subagent has no completion-notification channel; a backgrounded process is killed when the subagent's bash session ends and the envelope is lost. Block the agent's bash call until the script returns. The Bash tool's default timeout may need to be raised via the `timeout` parameter (e.g. `600000` ms = 10 min) for typical eval runs.\n\nIn both cases the envelope appears on stdout once the script exits; that is the only thing the next step needs.\n\nTell the user the evaluation is running and you will report when it finishes.\n\n**IMPORTANT: Do NOT poll, monitor, or periodically read the log file while waiting.** The script's own poll loop is the only one needed. Do not use `Read` or `Bash` to check progress — wait for the script to return (or, in the interactive case, for the background-completion notification).\n\n## Step 3: Report Results\n\nWhen the background command completes:\n\n- **Failure** (non-zero exit, or stdout envelope has `\"status\": \"failed\"`):\n  - Parse the last stdout line as JSON. If `host_output_dir` is present, read `\u003Chost_output_dir>\u002Flog.txt` (last 50 lines) and show the error.\n  - Also surface the `error` field from the envelope.\n\n- **Success** (`\"status\": \"completed\"`):\n  - Parse the JSON envelope from stdout. The envelope carries `headline_metrics.overall_accuracy` (0.0–1.0) directly — that is the by-action accuracy without any extra parsing.\n  - Cite `host_output_dir`, `artifacts.inference_results_json`, and `artifacts.log` so the user can navigate to the raw outputs.\n  - Tell the user that any downstream RCA invocation should pass `\u003Chost_output_dir>\u002Flog.txt` (or `inference_results.json`) as the by-action input.\n\nDo NOT load `inference_results.json` directly into the context — it can be large.\n\n## Reference\n\n### inputs.yaml Format\n\nSee `references\u002Finputs-template.yaml` for the canonical template with comments. Minimum required fields:\n\n```yaml\ntraining_job_id: \u003Cuuid>     # registered training job (eval-ms validates it's completed)\nval_dataset_id: \u003Cuuid>      # validation dataset id (actions.json is resolved by eval-ms)\nhost_results_root: \u002Fabs\u002Fpath\u002Fto\u002Fresults   # docker-compose maps this -> \u002Fworkspace\u002Fsop-eval-ms\u002Fassets\u002Fresults\n```\n\nAll other fields have sensible defaults; see template for details.\n\n### Eval-ms request body\n\n| inputs.yaml field | Mapped to request body | Default |\n|---|---|---|\n| `training_job_id` | `training_job_id` | required |\n| `val_dataset_id` | `val_dataset_id` | required |\n| `fps` | `fps` | 8 |\n| `temperature` | `temperature` | 0.0 |\n| `top_p` | `top_p` | 1.0 |\n| `backend` | `backend` | `vllm` |\n| `checkpoint_step` | `checkpoint_step` | latest |\n| `resolution_config` | `resolution_config` | training-mirror defaults |\n| `gpu_id` | `gpu_id` | all visible GPUs |\n\n`backend: transformers` is required when evaluating LoRA-only checkpoints — vLLM's tokenizer\u002Fconfig compatibility breaks on some Qwen3-VL releases when `flash_attn` is not installed.\n\n### Outputs\n\nThe eval-ms service writes everything to `\u003Chost_results_root>\u002F\u003Ceval_job_id>\u002F` (host-side path, resolved by this skill from the docker-compose volume mapping):\n\n| File | Description |\n|------|-------------|\n| `inference_results.json` | `{video: [[gt_action, vlm_response, chunk_path], ...]}` — 3-tuple format consumed by `analyze_by_action_confusion.py` |\n| `log.txt` | Full inference log with `Args: {...}` line, per-chunk `Action Chunk: \u003Cpath>` markers, and the final accuracy summary |\n\n### Structured JSON envelope (stdout)\n\n```json\n{\n  \"mode\": \"by-action\",\n  \"eval_job_id\": \"...\",\n  \"status\": \"completed\",\n  \"host_output_dir\": \"\u002Fabs\u002Fhost\u002Fpath\u002Fto\u002F\u003Ceval_job_id>\",\n  \"container_output_dir\": \"\u002Fworkspace\u002Fsop-eval-ms\u002Fassets\u002Fresults\u002F\u003Ceval_job_id>\",\n  \"artifacts\": {\n    \"inference_results_json\": \"\u003Chost_output_dir>\u002Finference_results.json\",\n    \"log\": \"\u003Chost_output_dir>\u002Flog.txt\"\n  },\n  \"headline_metrics\": {\n    \"overall_accuracy\": 0.95\n  },\n  \"error\": null\n}\n```\n\n`headline_metrics.overall_accuracy` is the fraction of chunks the VLM classified correctly across the dataset, mirroring `evaluation_job.overall_accuracy` in the DB. The eval-ms grader is strict (requires the full `(N)` token list to match); when the VLM emits multi-action responses the helper `sop-rca-plugin\u002F...\u002Fanalyze_by_action_confusion.py` (first-`(N)` match) may report higher accuracy — prefer it for orchestrator success-criteria.\n\nThe orchestrator (or any caller) reads this envelope to record `host_output_dir` and downstream artifact paths.\n\n### Troubleshooting\n\n- **HTTP 400 \"Training job not found\"**: `training_job_id` does not exist in the eval-ms Postgres. Confirm the ID via the BP UI or via `GET \u002Fapi\u002Fv1\u002Ftraining\u002Fall_jobs` on the training service.\n- **HTTP 400 \"Training job ... is not completed\"**: wait for training to finish.\n- **HTTP 400 \"actions.json not found\"**: the registered dataset directory is missing `actions.json`. Fix the dataset; eval-ms cannot proceed without it.\n- **HTTP 400 \"An evaluation is already running\"**: eval-ms allows one by-action eval at a time. Cancel the running one (`POST \u002Fapi\u002Fv1\u002Fevaluation\u002Fcancel\u002F{eval_job_id}`) or wait.\n- **Timeout** (envelope `\"status\": \"timeout\"`): the job has not reached a terminal state within `timeout_sec`. The job may still be running on eval-ms — check `GET \u002Fapi\u002Fv1\u002Fevaluation\u002Fstatus\u002F{eval_job_id}` manually before retrying.\n- **CUDA OOM** (eval-ms subprocess log): lower `fps` or pass a `resolution_config` with lower `total_pixels` \u002F `max_frames`.\n",{"data":35,"body":36},{"name":4,"description":6,"license":26},{"type":37,"children":38},"root",[39,48,71,76,87,114,124,146,153,158,302,307,313,340,432,463,469,532,587,594,615,676,681,686,712,718,723,861,873,879,885,897,972,977,983,1235,1254,1260,1273,1356,1362,1771,1812,1824,1830,1976],{"type":40,"tag":41,"props":42,"children":44},"element","h1",{"id":43},"by-action-evaluation-api-driven",[45],{"type":46,"value":47},"text","By-Action Evaluation (API-driven)",{"type":40,"tag":49,"props":50,"children":51},"p",{},[52,54,61,63,69],{"type":46,"value":53},"Run per-action VLM evaluation by POSTing to the BP ",{"type":40,"tag":55,"props":56,"children":58},"code",{"className":57},[],[59],{"type":46,"value":60},"evaluation-ms",{"type":46,"value":62}," HTTP service. The service decides on its own where to write output (a uuid-named directory under ",{"type":40,"tag":55,"props":64,"children":66},{"className":65},[],[67],{"type":46,"value":68},"\u003CRESULTS_ROOT>\u002F\u003Ceval_job_id>\u002F",{"type":46,"value":70},"), runs the inference subprocess, and updates Postgres + a job cache. This skill submits the request, polls until terminal, resolves the host-side output directory, and emits a structured JSON envelope on stdout.",{"type":40,"tag":49,"props":72,"children":73},{},[74],{"type":46,"value":75},"When this skill is invoked, follow the steps below in order. Stop immediately if any step fails — show the full error output to the user, but do NOT attempt to fix it automatically.",{"type":40,"tag":49,"props":77,"children":78},{},[79,85],{"type":40,"tag":80,"props":81,"children":82},"strong",{},[83],{"type":46,"value":84},"Bundled resources",{"type":46,"value":86}," (paths relative to this skill's directory):",{"type":40,"tag":88,"props":89,"children":90},"ul",{},[91,103],{"type":40,"tag":92,"props":93,"children":94},"li",{},[95,101],{"type":40,"tag":55,"props":96,"children":98},{"className":97},[],[99],{"type":46,"value":100},"scripts\u002Feval_api_client.py",{"type":46,"value":102}," — single Python helper that POSTs, polls, resolves paths, prints the envelope.",{"type":40,"tag":92,"props":104,"children":105},{},[106,112],{"type":40,"tag":55,"props":107,"children":109},{"className":108},[],[110],{"type":46,"value":111},"references\u002Finputs-template.yaml",{"type":46,"value":113}," — configuration template with all parameters documented.",{"type":40,"tag":49,"props":115,"children":116},{},[117,122],{"type":40,"tag":80,"props":118,"children":119},{},[120],{"type":46,"value":121},"Path conventions",{"type":46,"value":123},":",{"type":40,"tag":88,"props":125,"children":126},{},[127],{"type":40,"tag":92,"props":128,"children":129},{},[130,136,138,144],{"type":40,"tag":55,"props":131,"children":133},{"className":132},[],[134],{"type":46,"value":135},"SKILL_DIR",{"type":46,"value":137}," = this skill's ",{"type":40,"tag":55,"props":139,"children":141},{"className":140},[],[142],{"type":46,"value":143},"scripts",{"type":46,"value":145}," directory (absolute path resolved at invocation time).",{"type":40,"tag":147,"props":148,"children":150},"h2",{"id":149},"step-0-parse-overrides-if-any",[151],{"type":46,"value":152},"Step 0: Parse Overrides (if any)",{"type":40,"tag":49,"props":154,"children":155},{},[156],{"type":46,"value":157},"If the user provided natural language parameter overrides alongside the inputs.yaml:",{"type":40,"tag":159,"props":160,"children":161},"ol",{},[162,284,297],{"type":40,"tag":92,"props":163,"children":164},{},[165,167,172,174],{"type":46,"value":166},"Parse the overrides. If any part of the input cannot be clearly mapped to a yaml field, list the ambiguous parts and ask the user to clarify before proceeding. Map to the yaml structure (see ",{"type":40,"tag":55,"props":168,"children":170},{"className":169},[],[171],{"type":46,"value":111},{"type":46,"value":173}," for the full list):",{"type":40,"tag":88,"props":175,"children":176},{},[177,196],{"type":40,"tag":92,"props":178,"children":179},{},[180,182,188,190],{"type":46,"value":181},"Required: ",{"type":40,"tag":55,"props":183,"children":185},{"className":184},[],[186],{"type":46,"value":187},"training_job_id",{"type":46,"value":189},", ",{"type":40,"tag":55,"props":191,"children":193},{"className":192},[],[194],{"type":46,"value":195},"val_dataset_id",{"type":40,"tag":92,"props":197,"children":198},{},[199,201,207,208,214,215,221,222,228,229,235,236,242,243,249,250,256,257,263,264,270,271,277,278],{"type":46,"value":200},"Optional: ",{"type":40,"tag":55,"props":202,"children":204},{"className":203},[],[205],{"type":46,"value":206},"eval_host",{"type":46,"value":189},{"type":40,"tag":55,"props":209,"children":211},{"className":210},[],[212],{"type":46,"value":213},"eval_port",{"type":46,"value":189},{"type":40,"tag":55,"props":216,"children":218},{"className":217},[],[219],{"type":46,"value":220},"host_results_root",{"type":46,"value":189},{"type":40,"tag":55,"props":223,"children":225},{"className":224},[],[226],{"type":46,"value":227},"backend",{"type":46,"value":189},{"type":40,"tag":55,"props":230,"children":232},{"className":231},[],[233],{"type":46,"value":234},"fps",{"type":46,"value":189},{"type":40,"tag":55,"props":237,"children":239},{"className":238},[],[240],{"type":46,"value":241},"temperature",{"type":46,"value":189},{"type":40,"tag":55,"props":244,"children":246},{"className":245},[],[247],{"type":46,"value":248},"top_p",{"type":46,"value":189},{"type":40,"tag":55,"props":251,"children":253},{"className":252},[],[254],{"type":46,"value":255},"checkpoint_step",{"type":46,"value":189},{"type":40,"tag":55,"props":258,"children":260},{"className":259},[],[261],{"type":46,"value":262},"resolution_config",{"type":46,"value":189},{"type":40,"tag":55,"props":265,"children":267},{"className":266},[],[268],{"type":46,"value":269},"gpu_id",{"type":46,"value":189},{"type":40,"tag":55,"props":272,"children":274},{"className":273},[],[275],{"type":46,"value":276},"poll_interval_sec",{"type":46,"value":189},{"type":40,"tag":55,"props":279,"children":281},{"className":280},[],[282],{"type":46,"value":283},"timeout_sec",{"type":40,"tag":92,"props":285,"children":286},{},[287,289,295],{"type":46,"value":288},"Write the overrides to ",{"type":40,"tag":55,"props":290,"children":292},{"className":291},[],[293],{"type":46,"value":294},"\u002Ftmp\u002Fby_action_overrides.yaml",{"type":46,"value":296}," using the Write tool. The api client merges them onto the base inputs.yaml at invocation time — no separate merge step.",{"type":40,"tag":92,"props":298,"children":299},{},[300],{"type":46,"value":301},"Show the user what was overridden.",{"type":40,"tag":49,"props":303,"children":304},{},[305],{"type":46,"value":306},"If no overrides, skip the overrides file.",{"type":40,"tag":147,"props":308,"children":310},{"id":309},"step-1-preflight-eval-ms-reachability",[311],{"type":46,"value":312},"Step 1: Preflight — eval-ms reachability",{"type":40,"tag":49,"props":314,"children":315},{},[316,318,323,325,330,332,338],{"type":46,"value":317},"Before submitting, make sure the eval-ms service is up. Read ",{"type":40,"tag":55,"props":319,"children":321},{"className":320},[],[322],{"type":46,"value":206},{"type":46,"value":324}," and ",{"type":40,"tag":55,"props":326,"children":328},{"className":327},[],[329],{"type":46,"value":213},{"type":46,"value":331}," from inputs.yaml (defaults: ",{"type":40,"tag":55,"props":333,"children":335},{"className":334},[],[336],{"type":46,"value":337},"localhost:32090",{"type":46,"value":339},") and probe the health endpoint:",{"type":40,"tag":341,"props":342,"children":347},"pre",{"className":343,"code":344,"language":345,"meta":346,"style":346},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","curl -fsS http:\u002F\u002F${EVAL_HOST:-localhost}:${EVAL_PORT:-32090}\u002Fhealth\n","bash","",[348],{"type":40,"tag":55,"props":349,"children":350},{"__ignoreMap":346},[351],{"type":40,"tag":352,"props":353,"children":356},"span",{"class":354,"line":355},"line",1,[357,363,369,374,380,386,391,396,401,405,409,414,418,423,427],{"type":40,"tag":352,"props":358,"children":360},{"style":359},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[361],{"type":46,"value":362},"curl",{"type":40,"tag":352,"props":364,"children":366},{"style":365},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[367],{"type":46,"value":368}," -fsS",{"type":40,"tag":352,"props":370,"children":371},{"style":365},[372],{"type":46,"value":373}," http:\u002F\u002F",{"type":40,"tag":352,"props":375,"children":377},{"style":376},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[378],{"type":46,"value":379},"${",{"type":40,"tag":352,"props":381,"children":383},{"style":382},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[384],{"type":46,"value":385},"EVAL_HOST",{"type":40,"tag":352,"props":387,"children":388},{"style":376},[389],{"type":46,"value":390},":-",{"type":40,"tag":352,"props":392,"children":393},{"style":382},[394],{"type":46,"value":395},"localhost",{"type":40,"tag":352,"props":397,"children":398},{"style":376},[399],{"type":46,"value":400},"}",{"type":40,"tag":352,"props":402,"children":403},{"style":365},[404],{"type":46,"value":123},{"type":40,"tag":352,"props":406,"children":407},{"style":376},[408],{"type":46,"value":379},{"type":40,"tag":352,"props":410,"children":411},{"style":382},[412],{"type":46,"value":413},"EVAL_PORT",{"type":40,"tag":352,"props":415,"children":416},{"style":376},[417],{"type":46,"value":390},{"type":40,"tag":352,"props":419,"children":420},{"style":382},[421],{"type":46,"value":422},"32090",{"type":40,"tag":352,"props":424,"children":425},{"style":376},[426],{"type":46,"value":400},{"type":40,"tag":352,"props":428,"children":429},{"style":365},[430],{"type":46,"value":431},"\u002Fhealth\n",{"type":40,"tag":88,"props":433,"children":434},{},[435,453],{"type":40,"tag":92,"props":436,"children":437},{},[438,443,445,451],{"type":40,"tag":80,"props":439,"children":440},{},[441],{"type":46,"value":442},"Connection refused \u002F non-2xx",{"type":46,"value":444}," → stop and tell the user to bring eval-ms up (",{"type":40,"tag":55,"props":446,"children":448},{"className":447},[],[449],{"type":46,"value":450},"docker compose up evaluation-ms",{"type":46,"value":452}," from the BP deployment root) before retrying.",{"type":40,"tag":92,"props":454,"children":455},{},[456,461],{"type":40,"tag":80,"props":457,"children":458},{},[459],{"type":46,"value":460},"2xx",{"type":46,"value":462}," → continue.",{"type":40,"tag":147,"props":464,"children":466},{"id":465},"step-2-submit-poll",[467],{"type":46,"value":468},"Step 2: Submit + Poll",{"type":40,"tag":49,"props":470,"children":471},{},[472,478,480,485,487,493,495,500,502,508,510,516,517,523,525,530],{"type":40,"tag":55,"props":473,"children":475},{"className":474},[],[476],{"type":46,"value":477},"eval_api_client.py",{"type":46,"value":479}," is a ",{"type":40,"tag":80,"props":481,"children":482},{},[483],{"type":46,"value":484},"blocking",{"type":46,"value":486}," call. It POSTs the request, polls ",{"type":40,"tag":55,"props":488,"children":490},{"className":489},[],[491],{"type":46,"value":492},"\u002Fapi\u002Fv1\u002Fevaluation\u002Fstatus\u002F{eval_job_id}",{"type":46,"value":494}," every ",{"type":40,"tag":55,"props":496,"children":498},{"className":497},[],[499],{"type":46,"value":276},{"type":46,"value":501}," (default 20s) until terminal status (",{"type":40,"tag":55,"props":503,"children":505},{"className":504},[],[506],{"type":46,"value":507},"completed",{"type":46,"value":509}," \u002F ",{"type":40,"tag":55,"props":511,"children":513},{"className":512},[],[514],{"type":46,"value":515},"failed",{"type":46,"value":509},{"type":40,"tag":55,"props":518,"children":520},{"className":519},[],[521],{"type":46,"value":522},"cancelled",{"type":46,"value":524},") or ",{"type":40,"tag":55,"props":526,"children":528},{"className":527},[],[529],{"type":46,"value":283},{"type":46,"value":531}," (default 3600), prints progress to stderr, and emits a single JSON envelope on the last stdout line. The script returns only when eval-ms reports a terminal state.",{"type":40,"tag":341,"props":533,"children":535},{"className":343,"code":534,"language":345,"meta":346,"style":346},"python3 SKILL_DIR\u002Feval_api_client.py by-action \u003Cinputs.yaml> [--overrides \u002Ftmp\u002Fby_action_overrides.yaml]\n",[536],{"type":40,"tag":55,"props":537,"children":538},{"__ignoreMap":346},[539],{"type":40,"tag":352,"props":540,"children":541},{"class":354,"line":355},[542,547,552,557,562,567,572,577,582],{"type":40,"tag":352,"props":543,"children":544},{"style":359},[545],{"type":46,"value":546},"python3",{"type":40,"tag":352,"props":548,"children":549},{"style":365},[550],{"type":46,"value":551}," SKILL_DIR\u002Feval_api_client.py",{"type":40,"tag":352,"props":553,"children":554},{"style":365},[555],{"type":46,"value":556}," by-action",{"type":40,"tag":352,"props":558,"children":559},{"style":376},[560],{"type":46,"value":561}," \u003C",{"type":40,"tag":352,"props":563,"children":564},{"style":365},[565],{"type":46,"value":566},"inputs.yam",{"type":40,"tag":352,"props":568,"children":569},{"style":382},[570],{"type":46,"value":571},"l",{"type":40,"tag":352,"props":573,"children":574},{"style":376},[575],{"type":46,"value":576},">",{"type":40,"tag":352,"props":578,"children":579},{"style":382},[580],{"type":46,"value":581}," [--overrides ",{"type":40,"tag":352,"props":583,"children":584},{"style":365},[585],{"type":46,"value":586},"\u002Ftmp\u002Fby_action_overrides.yaml]\n",{"type":40,"tag":588,"props":589,"children":591},"h3",{"id":590},"choosing-run_in_background-vs-synchronous",[592],{"type":46,"value":593},"Choosing run_in_background vs synchronous",{"type":40,"tag":49,"props":595,"children":596},{},[597,599,605,607,613],{"type":46,"value":598},"The client is blocking either way — ",{"type":40,"tag":55,"props":600,"children":602},{"className":601},[],[603],{"type":46,"value":604},"run_in_background",{"type":46,"value":606}," only governs how ",{"type":40,"tag":608,"props":609,"children":610},"em",{},[611],{"type":46,"value":612},"you",{"type":46,"value":614}," (the caller) wait for it.",{"type":40,"tag":88,"props":616,"children":617},{},[618,636],{"type":40,"tag":92,"props":619,"children":620},{},[621,626,628,634],{"type":40,"tag":80,"props":622,"children":623},{},[624],{"type":46,"value":625},"Interactive Claude Code (main agent)",{"type":46,"value":627}," — use ",{"type":40,"tag":55,"props":629,"children":631},{"className":630},[],[632],{"type":46,"value":633},"run_in_background: true",{"type":46,"value":635}," so the main loop is free for other work; you receive a completion notification carrying the script's stdout (the envelope).",{"type":40,"tag":92,"props":637,"children":638},{},[639,644,646,651,653,658,660,666,668,674],{"type":40,"tag":80,"props":640,"children":641},{},[642],{"type":46,"value":643},"Subagent or any non-interactive context",{"type":46,"value":645}," — call ",{"type":40,"tag":80,"props":647,"children":648},{},[649],{"type":46,"value":650},"synchronously",{"type":46,"value":652}," (do NOT pass ",{"type":40,"tag":55,"props":654,"children":656},{"className":655},[],[657],{"type":46,"value":633},{"type":46,"value":659},"). A subagent has no completion-notification channel; a backgrounded process is killed when the subagent's bash session ends and the envelope is lost. Block the agent's bash call until the script returns. The Bash tool's default timeout may need to be raised via the ",{"type":40,"tag":55,"props":661,"children":663},{"className":662},[],[664],{"type":46,"value":665},"timeout",{"type":46,"value":667}," parameter (e.g. ",{"type":40,"tag":55,"props":669,"children":671},{"className":670},[],[672],{"type":46,"value":673},"600000",{"type":46,"value":675}," ms = 10 min) for typical eval runs.",{"type":40,"tag":49,"props":677,"children":678},{},[679],{"type":46,"value":680},"In both cases the envelope appears on stdout once the script exits; that is the only thing the next step needs.",{"type":40,"tag":49,"props":682,"children":683},{},[684],{"type":46,"value":685},"Tell the user the evaluation is running and you will report when it finishes.",{"type":40,"tag":49,"props":687,"children":688},{},[689,694,696,702,704,710],{"type":40,"tag":80,"props":690,"children":691},{},[692],{"type":46,"value":693},"IMPORTANT: Do NOT poll, monitor, or periodically read the log file while waiting.",{"type":46,"value":695}," The script's own poll loop is the only one needed. Do not use ",{"type":40,"tag":55,"props":697,"children":699},{"className":698},[],[700],{"type":46,"value":701},"Read",{"type":46,"value":703}," or ",{"type":40,"tag":55,"props":705,"children":707},{"className":706},[],[708],{"type":46,"value":709},"Bash",{"type":46,"value":711}," to check progress — wait for the script to return (or, in the interactive case, for the background-completion notification).",{"type":40,"tag":147,"props":713,"children":715},{"id":714},"step-3-report-results",[716],{"type":46,"value":717},"Step 3: Report Results",{"type":40,"tag":49,"props":719,"children":720},{},[721],{"type":46,"value":722},"When the background command completes:",{"type":40,"tag":88,"props":724,"children":725},{},[726,781],{"type":40,"tag":92,"props":727,"children":728},{},[729,734,736,742,744],{"type":40,"tag":80,"props":730,"children":731},{},[732],{"type":46,"value":733},"Failure",{"type":46,"value":735}," (non-zero exit, or stdout envelope has ",{"type":40,"tag":55,"props":737,"children":739},{"className":738},[],[740],{"type":46,"value":741},"\"status\": \"failed\"",{"type":46,"value":743},"):",{"type":40,"tag":88,"props":745,"children":746},{},[747,768],{"type":40,"tag":92,"props":748,"children":749},{},[750,752,758,760,766],{"type":46,"value":751},"Parse the last stdout line as JSON. If ",{"type":40,"tag":55,"props":753,"children":755},{"className":754},[],[756],{"type":46,"value":757},"host_output_dir",{"type":46,"value":759}," is present, read ",{"type":40,"tag":55,"props":761,"children":763},{"className":762},[],[764],{"type":46,"value":765},"\u003Chost_output_dir>\u002Flog.txt",{"type":46,"value":767}," (last 50 lines) and show the error.",{"type":40,"tag":92,"props":769,"children":770},{},[771,773,779],{"type":46,"value":772},"Also surface the ",{"type":40,"tag":55,"props":774,"children":776},{"className":775},[],[777],{"type":46,"value":778},"error",{"type":46,"value":780}," field from the envelope.",{"type":40,"tag":92,"props":782,"children":783},{},[784,789,791,797,798],{"type":40,"tag":80,"props":785,"children":786},{},[787],{"type":46,"value":788},"Success",{"type":46,"value":790}," (",{"type":40,"tag":55,"props":792,"children":794},{"className":793},[],[795],{"type":46,"value":796},"\"status\": \"completed\"",{"type":46,"value":743},{"type":40,"tag":88,"props":799,"children":800},{},[801,814,841],{"type":40,"tag":92,"props":802,"children":803},{},[804,806,812],{"type":46,"value":805},"Parse the JSON envelope from stdout. The envelope carries ",{"type":40,"tag":55,"props":807,"children":809},{"className":808},[],[810],{"type":46,"value":811},"headline_metrics.overall_accuracy",{"type":46,"value":813}," (0.0–1.0) directly — that is the by-action accuracy without any extra parsing.",{"type":40,"tag":92,"props":815,"children":816},{},[817,819,824,825,831,833,839],{"type":46,"value":818},"Cite ",{"type":40,"tag":55,"props":820,"children":822},{"className":821},[],[823],{"type":46,"value":757},{"type":46,"value":189},{"type":40,"tag":55,"props":826,"children":828},{"className":827},[],[829],{"type":46,"value":830},"artifacts.inference_results_json",{"type":46,"value":832},", and ",{"type":40,"tag":55,"props":834,"children":836},{"className":835},[],[837],{"type":46,"value":838},"artifacts.log",{"type":46,"value":840}," so the user can navigate to the raw outputs.",{"type":40,"tag":92,"props":842,"children":843},{},[844,846,851,853,859],{"type":46,"value":845},"Tell the user that any downstream RCA invocation should pass ",{"type":40,"tag":55,"props":847,"children":849},{"className":848},[],[850],{"type":46,"value":765},{"type":46,"value":852}," (or ",{"type":40,"tag":55,"props":854,"children":856},{"className":855},[],[857],{"type":46,"value":858},"inference_results.json",{"type":46,"value":860},") as the by-action input.",{"type":40,"tag":49,"props":862,"children":863},{},[864,866,871],{"type":46,"value":865},"Do NOT load ",{"type":40,"tag":55,"props":867,"children":869},{"className":868},[],[870],{"type":46,"value":858},{"type":46,"value":872}," directly into the context — it can be large.",{"type":40,"tag":147,"props":874,"children":876},{"id":875},"reference",[877],{"type":46,"value":878},"Reference",{"type":40,"tag":588,"props":880,"children":882},{"id":881},"inputsyaml-format",[883],{"type":46,"value":884},"inputs.yaml Format",{"type":40,"tag":49,"props":886,"children":887},{},[888,890,895],{"type":46,"value":889},"See ",{"type":40,"tag":55,"props":891,"children":893},{"className":892},[],[894],{"type":46,"value":111},{"type":46,"value":896}," for the canonical template with comments. Minimum required fields:",{"type":40,"tag":341,"props":898,"children":902},{"className":899,"code":900,"language":901,"meta":346,"style":346},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","training_job_id: \u003Cuuid>     # registered training job (eval-ms validates it's completed)\nval_dataset_id: \u003Cuuid>      # validation dataset id (actions.json is resolved by eval-ms)\nhost_results_root: \u002Fabs\u002Fpath\u002Fto\u002Fresults   # docker-compose maps this -> \u002Fworkspace\u002Fsop-eval-ms\u002Fassets\u002Fresults\n","yaml",[903],{"type":40,"tag":55,"props":904,"children":905},{"__ignoreMap":346},[906,929,950],{"type":40,"tag":352,"props":907,"children":908},{"class":354,"line":355},[909,914,918,923],{"type":40,"tag":352,"props":910,"children":912},{"style":911},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[913],{"type":46,"value":187},{"type":40,"tag":352,"props":915,"children":916},{"style":376},[917],{"type":46,"value":123},{"type":40,"tag":352,"props":919,"children":920},{"style":365},[921],{"type":46,"value":922}," \u003Cuuid>",{"type":40,"tag":352,"props":924,"children":926},{"style":925},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[927],{"type":46,"value":928},"     # registered training job (eval-ms validates it's completed)\n",{"type":40,"tag":352,"props":930,"children":932},{"class":354,"line":931},2,[933,937,941,945],{"type":40,"tag":352,"props":934,"children":935},{"style":911},[936],{"type":46,"value":195},{"type":40,"tag":352,"props":938,"children":939},{"style":376},[940],{"type":46,"value":123},{"type":40,"tag":352,"props":942,"children":943},{"style":365},[944],{"type":46,"value":922},{"type":40,"tag":352,"props":946,"children":947},{"style":925},[948],{"type":46,"value":949},"      # validation dataset id (actions.json is resolved by eval-ms)\n",{"type":40,"tag":352,"props":951,"children":953},{"class":354,"line":952},3,[954,958,962,967],{"type":40,"tag":352,"props":955,"children":956},{"style":911},[957],{"type":46,"value":220},{"type":40,"tag":352,"props":959,"children":960},{"style":376},[961],{"type":46,"value":123},{"type":40,"tag":352,"props":963,"children":964},{"style":365},[965],{"type":46,"value":966}," \u002Fabs\u002Fpath\u002Fto\u002Fresults",{"type":40,"tag":352,"props":968,"children":969},{"style":925},[970],{"type":46,"value":971},"   # docker-compose maps this -> \u002Fworkspace\u002Fsop-eval-ms\u002Fassets\u002Fresults\n",{"type":40,"tag":49,"props":973,"children":974},{},[975],{"type":46,"value":976},"All other fields have sensible defaults; see template for details.",{"type":40,"tag":588,"props":978,"children":980},{"id":979},"eval-ms-request-body",[981],{"type":46,"value":982},"Eval-ms request body",{"type":40,"tag":984,"props":985,"children":986},"table",{},[987,1011],{"type":40,"tag":988,"props":989,"children":990},"thead",{},[991],{"type":40,"tag":992,"props":993,"children":994},"tr",{},[995,1001,1006],{"type":40,"tag":996,"props":997,"children":998},"th",{},[999],{"type":46,"value":1000},"inputs.yaml field",{"type":40,"tag":996,"props":1002,"children":1003},{},[1004],{"type":46,"value":1005},"Mapped to request body",{"type":40,"tag":996,"props":1007,"children":1008},{},[1009],{"type":46,"value":1010},"Default",{"type":40,"tag":1012,"props":1013,"children":1014},"tbody",{},[1015,1040,1063,1087,1111,1135,1163,1187,1211],{"type":40,"tag":992,"props":1016,"children":1017},{},[1018,1027,1035],{"type":40,"tag":1019,"props":1020,"children":1021},"td",{},[1022],{"type":40,"tag":55,"props":1023,"children":1025},{"className":1024},[],[1026],{"type":46,"value":187},{"type":40,"tag":1019,"props":1028,"children":1029},{},[1030],{"type":40,"tag":55,"props":1031,"children":1033},{"className":1032},[],[1034],{"type":46,"value":187},{"type":40,"tag":1019,"props":1036,"children":1037},{},[1038],{"type":46,"value":1039},"required",{"type":40,"tag":992,"props":1041,"children":1042},{},[1043,1051,1059],{"type":40,"tag":1019,"props":1044,"children":1045},{},[1046],{"type":40,"tag":55,"props":1047,"children":1049},{"className":1048},[],[1050],{"type":46,"value":195},{"type":40,"tag":1019,"props":1052,"children":1053},{},[1054],{"type":40,"tag":55,"props":1055,"children":1057},{"className":1056},[],[1058],{"type":46,"value":195},{"type":40,"tag":1019,"props":1060,"children":1061},{},[1062],{"type":46,"value":1039},{"type":40,"tag":992,"props":1064,"children":1065},{},[1066,1074,1082],{"type":40,"tag":1019,"props":1067,"children":1068},{},[1069],{"type":40,"tag":55,"props":1070,"children":1072},{"className":1071},[],[1073],{"type":46,"value":234},{"type":40,"tag":1019,"props":1075,"children":1076},{},[1077],{"type":40,"tag":55,"props":1078,"children":1080},{"className":1079},[],[1081],{"type":46,"value":234},{"type":40,"tag":1019,"props":1083,"children":1084},{},[1085],{"type":46,"value":1086},"8",{"type":40,"tag":992,"props":1088,"children":1089},{},[1090,1098,1106],{"type":40,"tag":1019,"props":1091,"children":1092},{},[1093],{"type":40,"tag":55,"props":1094,"children":1096},{"className":1095},[],[1097],{"type":46,"value":241},{"type":40,"tag":1019,"props":1099,"children":1100},{},[1101],{"type":40,"tag":55,"props":1102,"children":1104},{"className":1103},[],[1105],{"type":46,"value":241},{"type":40,"tag":1019,"props":1107,"children":1108},{},[1109],{"type":46,"value":1110},"0.0",{"type":40,"tag":992,"props":1112,"children":1113},{},[1114,1122,1130],{"type":40,"tag":1019,"props":1115,"children":1116},{},[1117],{"type":40,"tag":55,"props":1118,"children":1120},{"className":1119},[],[1121],{"type":46,"value":248},{"type":40,"tag":1019,"props":1123,"children":1124},{},[1125],{"type":40,"tag":55,"props":1126,"children":1128},{"className":1127},[],[1129],{"type":46,"value":248},{"type":40,"tag":1019,"props":1131,"children":1132},{},[1133],{"type":46,"value":1134},"1.0",{"type":40,"tag":992,"props":1136,"children":1137},{},[1138,1146,1154],{"type":40,"tag":1019,"props":1139,"children":1140},{},[1141],{"type":40,"tag":55,"props":1142,"children":1144},{"className":1143},[],[1145],{"type":46,"value":227},{"type":40,"tag":1019,"props":1147,"children":1148},{},[1149],{"type":40,"tag":55,"props":1150,"children":1152},{"className":1151},[],[1153],{"type":46,"value":227},{"type":40,"tag":1019,"props":1155,"children":1156},{},[1157],{"type":40,"tag":55,"props":1158,"children":1160},{"className":1159},[],[1161],{"type":46,"value":1162},"vllm",{"type":40,"tag":992,"props":1164,"children":1165},{},[1166,1174,1182],{"type":40,"tag":1019,"props":1167,"children":1168},{},[1169],{"type":40,"tag":55,"props":1170,"children":1172},{"className":1171},[],[1173],{"type":46,"value":255},{"type":40,"tag":1019,"props":1175,"children":1176},{},[1177],{"type":40,"tag":55,"props":1178,"children":1180},{"className":1179},[],[1181],{"type":46,"value":255},{"type":40,"tag":1019,"props":1183,"children":1184},{},[1185],{"type":46,"value":1186},"latest",{"type":40,"tag":992,"props":1188,"children":1189},{},[1190,1198,1206],{"type":40,"tag":1019,"props":1191,"children":1192},{},[1193],{"type":40,"tag":55,"props":1194,"children":1196},{"className":1195},[],[1197],{"type":46,"value":262},{"type":40,"tag":1019,"props":1199,"children":1200},{},[1201],{"type":40,"tag":55,"props":1202,"children":1204},{"className":1203},[],[1205],{"type":46,"value":262},{"type":40,"tag":1019,"props":1207,"children":1208},{},[1209],{"type":46,"value":1210},"training-mirror defaults",{"type":40,"tag":992,"props":1212,"children":1213},{},[1214,1222,1230],{"type":40,"tag":1019,"props":1215,"children":1216},{},[1217],{"type":40,"tag":55,"props":1218,"children":1220},{"className":1219},[],[1221],{"type":46,"value":269},{"type":40,"tag":1019,"props":1223,"children":1224},{},[1225],{"type":40,"tag":55,"props":1226,"children":1228},{"className":1227},[],[1229],{"type":46,"value":269},{"type":40,"tag":1019,"props":1231,"children":1232},{},[1233],{"type":46,"value":1234},"all visible GPUs",{"type":40,"tag":49,"props":1236,"children":1237},{},[1238,1244,1246,1252],{"type":40,"tag":55,"props":1239,"children":1241},{"className":1240},[],[1242],{"type":46,"value":1243},"backend: transformers",{"type":46,"value":1245}," is required when evaluating LoRA-only checkpoints — vLLM's tokenizer\u002Fconfig compatibility breaks on some Qwen3-VL releases when ",{"type":40,"tag":55,"props":1247,"children":1249},{"className":1248},[],[1250],{"type":46,"value":1251},"flash_attn",{"type":46,"value":1253}," is not installed.",{"type":40,"tag":588,"props":1255,"children":1257},{"id":1256},"outputs",[1258],{"type":46,"value":1259},"Outputs",{"type":40,"tag":49,"props":1261,"children":1262},{},[1263,1265,1271],{"type":46,"value":1264},"The eval-ms service writes everything to ",{"type":40,"tag":55,"props":1266,"children":1268},{"className":1267},[],[1269],{"type":46,"value":1270},"\u003Chost_results_root>\u002F\u003Ceval_job_id>\u002F",{"type":46,"value":1272}," (host-side path, resolved by this skill from the docker-compose volume mapping):",{"type":40,"tag":984,"props":1274,"children":1275},{},[1276,1292],{"type":40,"tag":988,"props":1277,"children":1278},{},[1279],{"type":40,"tag":992,"props":1280,"children":1281},{},[1282,1287],{"type":40,"tag":996,"props":1283,"children":1284},{},[1285],{"type":46,"value":1286},"File",{"type":40,"tag":996,"props":1288,"children":1289},{},[1290],{"type":46,"value":1291},"Description",{"type":40,"tag":1012,"props":1293,"children":1294},{},[1295,1323],{"type":40,"tag":992,"props":1296,"children":1297},{},[1298,1306],{"type":40,"tag":1019,"props":1299,"children":1300},{},[1301],{"type":40,"tag":55,"props":1302,"children":1304},{"className":1303},[],[1305],{"type":46,"value":858},{"type":40,"tag":1019,"props":1307,"children":1308},{},[1309,1315,1317],{"type":40,"tag":55,"props":1310,"children":1312},{"className":1311},[],[1313],{"type":46,"value":1314},"{video: [[gt_action, vlm_response, chunk_path], ...]}",{"type":46,"value":1316}," — 3-tuple format consumed by ",{"type":40,"tag":55,"props":1318,"children":1320},{"className":1319},[],[1321],{"type":46,"value":1322},"analyze_by_action_confusion.py",{"type":40,"tag":992,"props":1324,"children":1325},{},[1326,1335],{"type":40,"tag":1019,"props":1327,"children":1328},{},[1329],{"type":40,"tag":55,"props":1330,"children":1332},{"className":1331},[],[1333],{"type":46,"value":1334},"log.txt",{"type":40,"tag":1019,"props":1336,"children":1337},{},[1338,1340,1346,1348,1354],{"type":46,"value":1339},"Full inference log with ",{"type":40,"tag":55,"props":1341,"children":1343},{"className":1342},[],[1344],{"type":46,"value":1345},"Args: {...}",{"type":46,"value":1347}," line, per-chunk ",{"type":40,"tag":55,"props":1349,"children":1351},{"className":1350},[],[1352],{"type":46,"value":1353},"Action Chunk: \u003Cpath>",{"type":46,"value":1355}," markers, and the final accuracy summary",{"type":40,"tag":588,"props":1357,"children":1359},{"id":1358},"structured-json-envelope-stdout",[1360],{"type":46,"value":1361},"Structured JSON envelope (stdout)",{"type":40,"tag":341,"props":1363,"children":1367},{"className":1364,"code":1365,"language":1366,"meta":346,"style":346},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","{\n  \"mode\": \"by-action\",\n  \"eval_job_id\": \"...\",\n  \"status\": \"completed\",\n  \"host_output_dir\": \"\u002Fabs\u002Fhost\u002Fpath\u002Fto\u002F\u003Ceval_job_id>\",\n  \"container_output_dir\": \"\u002Fworkspace\u002Fsop-eval-ms\u002Fassets\u002Fresults\u002F\u003Ceval_job_id>\",\n  \"artifacts\": {\n    \"inference_results_json\": \"\u003Chost_output_dir>\u002Finference_results.json\",\n    \"log\": \"\u003Chost_output_dir>\u002Flog.txt\"\n  },\n  \"headline_metrics\": {\n    \"overall_accuracy\": 0.95\n  },\n  \"error\": null\n}\n","json",[1368],{"type":40,"tag":55,"props":1369,"children":1370},{"__ignoreMap":346},[1371,1379,1421,1458,1495,1532,1570,1596,1635,1669,1678,1703,1729,1737,1762],{"type":40,"tag":352,"props":1372,"children":1373},{"class":354,"line":355},[1374],{"type":40,"tag":352,"props":1375,"children":1376},{"style":376},[1377],{"type":46,"value":1378},"{\n",{"type":40,"tag":352,"props":1380,"children":1381},{"class":354,"line":931},[1382,1387,1393,1398,1402,1407,1412,1416],{"type":40,"tag":352,"props":1383,"children":1384},{"style":376},[1385],{"type":46,"value":1386},"  \"",{"type":40,"tag":352,"props":1388,"children":1390},{"style":1389},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1391],{"type":46,"value":1392},"mode",{"type":40,"tag":352,"props":1394,"children":1395},{"style":376},[1396],{"type":46,"value":1397},"\"",{"type":40,"tag":352,"props":1399,"children":1400},{"style":376},[1401],{"type":46,"value":123},{"type":40,"tag":352,"props":1403,"children":1404},{"style":376},[1405],{"type":46,"value":1406}," \"",{"type":40,"tag":352,"props":1408,"children":1409},{"style":365},[1410],{"type":46,"value":1411},"by-action",{"type":40,"tag":352,"props":1413,"children":1414},{"style":376},[1415],{"type":46,"value":1397},{"type":40,"tag":352,"props":1417,"children":1418},{"style":376},[1419],{"type":46,"value":1420},",\n",{"type":40,"tag":352,"props":1422,"children":1423},{"class":354,"line":952},[1424,1428,1433,1437,1441,1445,1450,1454],{"type":40,"tag":352,"props":1425,"children":1426},{"style":376},[1427],{"type":46,"value":1386},{"type":40,"tag":352,"props":1429,"children":1430},{"style":1389},[1431],{"type":46,"value":1432},"eval_job_id",{"type":40,"tag":352,"props":1434,"children":1435},{"style":376},[1436],{"type":46,"value":1397},{"type":40,"tag":352,"props":1438,"children":1439},{"style":376},[1440],{"type":46,"value":123},{"type":40,"tag":352,"props":1442,"children":1443},{"style":376},[1444],{"type":46,"value":1406},{"type":40,"tag":352,"props":1446,"children":1447},{"style":365},[1448],{"type":46,"value":1449},"...",{"type":40,"tag":352,"props":1451,"children":1452},{"style":376},[1453],{"type":46,"value":1397},{"type":40,"tag":352,"props":1455,"children":1456},{"style":376},[1457],{"type":46,"value":1420},{"type":40,"tag":352,"props":1459,"children":1461},{"class":354,"line":1460},4,[1462,1466,1471,1475,1479,1483,1487,1491],{"type":40,"tag":352,"props":1463,"children":1464},{"style":376},[1465],{"type":46,"value":1386},{"type":40,"tag":352,"props":1467,"children":1468},{"style":1389},[1469],{"type":46,"value":1470},"status",{"type":40,"tag":352,"props":1472,"children":1473},{"style":376},[1474],{"type":46,"value":1397},{"type":40,"tag":352,"props":1476,"children":1477},{"style":376},[1478],{"type":46,"value":123},{"type":40,"tag":352,"props":1480,"children":1481},{"style":376},[1482],{"type":46,"value":1406},{"type":40,"tag":352,"props":1484,"children":1485},{"style":365},[1486],{"type":46,"value":507},{"type":40,"tag":352,"props":1488,"children":1489},{"style":376},[1490],{"type":46,"value":1397},{"type":40,"tag":352,"props":1492,"children":1493},{"style":376},[1494],{"type":46,"value":1420},{"type":40,"tag":352,"props":1496,"children":1498},{"class":354,"line":1497},5,[1499,1503,1507,1511,1515,1519,1524,1528],{"type":40,"tag":352,"props":1500,"children":1501},{"style":376},[1502],{"type":46,"value":1386},{"type":40,"tag":352,"props":1504,"children":1505},{"style":1389},[1506],{"type":46,"value":757},{"type":40,"tag":352,"props":1508,"children":1509},{"style":376},[1510],{"type":46,"value":1397},{"type":40,"tag":352,"props":1512,"children":1513},{"style":376},[1514],{"type":46,"value":123},{"type":40,"tag":352,"props":1516,"children":1517},{"style":376},[1518],{"type":46,"value":1406},{"type":40,"tag":352,"props":1520,"children":1521},{"style":365},[1522],{"type":46,"value":1523},"\u002Fabs\u002Fhost\u002Fpath\u002Fto\u002F\u003Ceval_job_id>",{"type":40,"tag":352,"props":1525,"children":1526},{"style":376},[1527],{"type":46,"value":1397},{"type":40,"tag":352,"props":1529,"children":1530},{"style":376},[1531],{"type":46,"value":1420},{"type":40,"tag":352,"props":1533,"children":1535},{"class":354,"line":1534},6,[1536,1540,1545,1549,1553,1557,1562,1566],{"type":40,"tag":352,"props":1537,"children":1538},{"style":376},[1539],{"type":46,"value":1386},{"type":40,"tag":352,"props":1541,"children":1542},{"style":1389},[1543],{"type":46,"value":1544},"container_output_dir",{"type":40,"tag":352,"props":1546,"children":1547},{"style":376},[1548],{"type":46,"value":1397},{"type":40,"tag":352,"props":1550,"children":1551},{"style":376},[1552],{"type":46,"value":123},{"type":40,"tag":352,"props":1554,"children":1555},{"style":376},[1556],{"type":46,"value":1406},{"type":40,"tag":352,"props":1558,"children":1559},{"style":365},[1560],{"type":46,"value":1561},"\u002Fworkspace\u002Fsop-eval-ms\u002Fassets\u002Fresults\u002F\u003Ceval_job_id>",{"type":40,"tag":352,"props":1563,"children":1564},{"style":376},[1565],{"type":46,"value":1397},{"type":40,"tag":352,"props":1567,"children":1568},{"style":376},[1569],{"type":46,"value":1420},{"type":40,"tag":352,"props":1571,"children":1573},{"class":354,"line":1572},7,[1574,1578,1583,1587,1591],{"type":40,"tag":352,"props":1575,"children":1576},{"style":376},[1577],{"type":46,"value":1386},{"type":40,"tag":352,"props":1579,"children":1580},{"style":1389},[1581],{"type":46,"value":1582},"artifacts",{"type":40,"tag":352,"props":1584,"children":1585},{"style":376},[1586],{"type":46,"value":1397},{"type":40,"tag":352,"props":1588,"children":1589},{"style":376},[1590],{"type":46,"value":123},{"type":40,"tag":352,"props":1592,"children":1593},{"style":376},[1594],{"type":46,"value":1595}," {\n",{"type":40,"tag":352,"props":1597,"children":1599},{"class":354,"line":1598},8,[1600,1605,1610,1614,1618,1622,1627,1631],{"type":40,"tag":352,"props":1601,"children":1602},{"style":376},[1603],{"type":46,"value":1604},"    \"",{"type":40,"tag":352,"props":1606,"children":1607},{"style":359},[1608],{"type":46,"value":1609},"inference_results_json",{"type":40,"tag":352,"props":1611,"children":1612},{"style":376},[1613],{"type":46,"value":1397},{"type":40,"tag":352,"props":1615,"children":1616},{"style":376},[1617],{"type":46,"value":123},{"type":40,"tag":352,"props":1619,"children":1620},{"style":376},[1621],{"type":46,"value":1406},{"type":40,"tag":352,"props":1623,"children":1624},{"style":365},[1625],{"type":46,"value":1626},"\u003Chost_output_dir>\u002Finference_results.json",{"type":40,"tag":352,"props":1628,"children":1629},{"style":376},[1630],{"type":46,"value":1397},{"type":40,"tag":352,"props":1632,"children":1633},{"style":376},[1634],{"type":46,"value":1420},{"type":40,"tag":352,"props":1636,"children":1638},{"class":354,"line":1637},9,[1639,1643,1648,1652,1656,1660,1664],{"type":40,"tag":352,"props":1640,"children":1641},{"style":376},[1642],{"type":46,"value":1604},{"type":40,"tag":352,"props":1644,"children":1645},{"style":359},[1646],{"type":46,"value":1647},"log",{"type":40,"tag":352,"props":1649,"children":1650},{"style":376},[1651],{"type":46,"value":1397},{"type":40,"tag":352,"props":1653,"children":1654},{"style":376},[1655],{"type":46,"value":123},{"type":40,"tag":352,"props":1657,"children":1658},{"style":376},[1659],{"type":46,"value":1406},{"type":40,"tag":352,"props":1661,"children":1662},{"style":365},[1663],{"type":46,"value":765},{"type":40,"tag":352,"props":1665,"children":1666},{"style":376},[1667],{"type":46,"value":1668},"\"\n",{"type":40,"tag":352,"props":1670,"children":1672},{"class":354,"line":1671},10,[1673],{"type":40,"tag":352,"props":1674,"children":1675},{"style":376},[1676],{"type":46,"value":1677},"  },\n",{"type":40,"tag":352,"props":1679,"children":1681},{"class":354,"line":1680},11,[1682,1686,1691,1695,1699],{"type":40,"tag":352,"props":1683,"children":1684},{"style":376},[1685],{"type":46,"value":1386},{"type":40,"tag":352,"props":1687,"children":1688},{"style":1389},[1689],{"type":46,"value":1690},"headline_metrics",{"type":40,"tag":352,"props":1692,"children":1693},{"style":376},[1694],{"type":46,"value":1397},{"type":40,"tag":352,"props":1696,"children":1697},{"style":376},[1698],{"type":46,"value":123},{"type":40,"tag":352,"props":1700,"children":1701},{"style":376},[1702],{"type":46,"value":1595},{"type":40,"tag":352,"props":1704,"children":1705},{"class":354,"line":27},[1706,1710,1715,1719,1723],{"type":40,"tag":352,"props":1707,"children":1708},{"style":376},[1709],{"type":46,"value":1604},{"type":40,"tag":352,"props":1711,"children":1712},{"style":359},[1713],{"type":46,"value":1714},"overall_accuracy",{"type":40,"tag":352,"props":1716,"children":1717},{"style":376},[1718],{"type":46,"value":1397},{"type":40,"tag":352,"props":1720,"children":1721},{"style":376},[1722],{"type":46,"value":123},{"type":40,"tag":352,"props":1724,"children":1726},{"style":1725},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1727],{"type":46,"value":1728}," 0.95\n",{"type":40,"tag":352,"props":1730,"children":1732},{"class":354,"line":1731},13,[1733],{"type":40,"tag":352,"props":1734,"children":1735},{"style":376},[1736],{"type":46,"value":1677},{"type":40,"tag":352,"props":1738,"children":1740},{"class":354,"line":1739},14,[1741,1745,1749,1753,1757],{"type":40,"tag":352,"props":1742,"children":1743},{"style":376},[1744],{"type":46,"value":1386},{"type":40,"tag":352,"props":1746,"children":1747},{"style":1389},[1748],{"type":46,"value":778},{"type":40,"tag":352,"props":1750,"children":1751},{"style":376},[1752],{"type":46,"value":1397},{"type":40,"tag":352,"props":1754,"children":1755},{"style":376},[1756],{"type":46,"value":123},{"type":40,"tag":352,"props":1758,"children":1759},{"style":376},[1760],{"type":46,"value":1761}," null\n",{"type":40,"tag":352,"props":1763,"children":1765},{"class":354,"line":1764},15,[1766],{"type":40,"tag":352,"props":1767,"children":1768},{"style":376},[1769],{"type":46,"value":1770},"}\n",{"type":40,"tag":49,"props":1772,"children":1773},{},[1774,1779,1781,1787,1789,1795,1797,1803,1805,1810],{"type":40,"tag":55,"props":1775,"children":1777},{"className":1776},[],[1778],{"type":46,"value":811},{"type":46,"value":1780}," is the fraction of chunks the VLM classified correctly across the dataset, mirroring ",{"type":40,"tag":55,"props":1782,"children":1784},{"className":1783},[],[1785],{"type":46,"value":1786},"evaluation_job.overall_accuracy",{"type":46,"value":1788}," in the DB. The eval-ms grader is strict (requires the full ",{"type":40,"tag":55,"props":1790,"children":1792},{"className":1791},[],[1793],{"type":46,"value":1794},"(N)",{"type":46,"value":1796}," token list to match); when the VLM emits multi-action responses the helper ",{"type":40,"tag":55,"props":1798,"children":1800},{"className":1799},[],[1801],{"type":46,"value":1802},"sop-rca-plugin\u002F...\u002Fanalyze_by_action_confusion.py",{"type":46,"value":1804}," (first-",{"type":40,"tag":55,"props":1806,"children":1808},{"className":1807},[],[1809],{"type":46,"value":1794},{"type":46,"value":1811}," match) may report higher accuracy — prefer it for orchestrator success-criteria.",{"type":40,"tag":49,"props":1813,"children":1814},{},[1815,1817,1822],{"type":46,"value":1816},"The orchestrator (or any caller) reads this envelope to record ",{"type":40,"tag":55,"props":1818,"children":1820},{"className":1819},[],[1821],{"type":46,"value":757},{"type":46,"value":1823}," and downstream artifact paths.",{"type":40,"tag":588,"props":1825,"children":1827},{"id":1826},"troubleshooting",[1828],{"type":46,"value":1829},"Troubleshooting",{"type":40,"tag":88,"props":1831,"children":1832},{},[1833,1858,1868,1886,1904,1937],{"type":40,"tag":92,"props":1834,"children":1835},{},[1836,1841,1843,1848,1850,1856],{"type":40,"tag":80,"props":1837,"children":1838},{},[1839],{"type":46,"value":1840},"HTTP 400 \"Training job not found\"",{"type":46,"value":1842},": ",{"type":40,"tag":55,"props":1844,"children":1846},{"className":1845},[],[1847],{"type":46,"value":187},{"type":46,"value":1849}," does not exist in the eval-ms Postgres. Confirm the ID via the BP UI or via ",{"type":40,"tag":55,"props":1851,"children":1853},{"className":1852},[],[1854],{"type":46,"value":1855},"GET \u002Fapi\u002Fv1\u002Ftraining\u002Fall_jobs",{"type":46,"value":1857}," on the training service.",{"type":40,"tag":92,"props":1859,"children":1860},{},[1861,1866],{"type":40,"tag":80,"props":1862,"children":1863},{},[1864],{"type":46,"value":1865},"HTTP 400 \"Training job ... is not completed\"",{"type":46,"value":1867},": wait for training to finish.",{"type":40,"tag":92,"props":1869,"children":1870},{},[1871,1876,1878,1884],{"type":40,"tag":80,"props":1872,"children":1873},{},[1874],{"type":46,"value":1875},"HTTP 400 \"actions.json not found\"",{"type":46,"value":1877},": the registered dataset directory is missing ",{"type":40,"tag":55,"props":1879,"children":1881},{"className":1880},[],[1882],{"type":46,"value":1883},"actions.json",{"type":46,"value":1885},". Fix the dataset; eval-ms cannot proceed without it.",{"type":40,"tag":92,"props":1887,"children":1888},{},[1889,1894,1896,1902],{"type":40,"tag":80,"props":1890,"children":1891},{},[1892],{"type":46,"value":1893},"HTTP 400 \"An evaluation is already running\"",{"type":46,"value":1895},": eval-ms allows one by-action eval at a time. Cancel the running one (",{"type":40,"tag":55,"props":1897,"children":1899},{"className":1898},[],[1900],{"type":46,"value":1901},"POST \u002Fapi\u002Fv1\u002Fevaluation\u002Fcancel\u002F{eval_job_id}",{"type":46,"value":1903},") or wait.",{"type":40,"tag":92,"props":1905,"children":1906},{},[1907,1912,1914,1920,1922,1927,1929,1935],{"type":40,"tag":80,"props":1908,"children":1909},{},[1910],{"type":46,"value":1911},"Timeout",{"type":46,"value":1913}," (envelope ",{"type":40,"tag":55,"props":1915,"children":1917},{"className":1916},[],[1918],{"type":46,"value":1919},"\"status\": \"timeout\"",{"type":46,"value":1921},"): the job has not reached a terminal state within ",{"type":40,"tag":55,"props":1923,"children":1925},{"className":1924},[],[1926],{"type":46,"value":283},{"type":46,"value":1928},". The job may still be running on eval-ms — check ",{"type":40,"tag":55,"props":1930,"children":1932},{"className":1931},[],[1933],{"type":46,"value":1934},"GET \u002Fapi\u002Fv1\u002Fevaluation\u002Fstatus\u002F{eval_job_id}",{"type":46,"value":1936}," manually before retrying.",{"type":40,"tag":92,"props":1938,"children":1939},{},[1940,1945,1947,1952,1954,1959,1961,1967,1968,1974],{"type":40,"tag":80,"props":1941,"children":1942},{},[1943],{"type":46,"value":1944},"CUDA OOM",{"type":46,"value":1946}," (eval-ms subprocess log): lower ",{"type":40,"tag":55,"props":1948,"children":1950},{"className":1949},[],[1951],{"type":46,"value":234},{"type":46,"value":1953}," or pass a ",{"type":40,"tag":55,"props":1955,"children":1957},{"className":1956},[],[1958],{"type":46,"value":262},{"type":46,"value":1960}," with lower ",{"type":40,"tag":55,"props":1962,"children":1964},{"className":1963},[],[1965],{"type":46,"value":1966},"total_pixels",{"type":46,"value":509},{"type":40,"tag":55,"props":1969,"children":1971},{"className":1970},[],[1972],{"type":46,"value":1973},"max_frames",{"type":46,"value":1975},".",{"type":40,"tag":1977,"props":1978,"children":1979},"style",{},[1980],{"type":46,"value":1981},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1983,"total":1680},[1984,2002,2009,2022,2040,2052,2063],{"slug":1985,"name":1985,"fn":1986,"description":1987,"org":1988,"tags":1989,"stars":23,"repoUrl":24,"updatedAt":2001},"sop-build","orchestrate end-to-end SOP monitoring pipelines","Orchestrate the end-to-end SOP pipeline, including preflight prerequisite checks, verifying models and downloading assets, generating the DeepStream SOP microservice with RTSP output, evaluating the microservice, and building, deploying, and testing the VSS SOP blueprint. Use when asked to run the full SOP pipeline, set up the SOP pipeline from scratch, execute preflight checks, verify models, download assets, generate the SOP microservice, evaluate the microservice, build the VSS blueprint, deploy the VSS blueprint, test the VSS blueprint, or manage the complete build-evaluate-deploy-test cycle.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[1990,1993,1996,1997,1998],{"name":1991,"slug":1992,"type":15},"Automation","automation",{"name":1994,"slug":1995,"type":15},"Engineering","engineering",{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"name":1999,"slug":2000,"type":15},"Video","video","2026-07-14T05:32:48.503678",{"slug":4,"name":4,"fn":5,"description":6,"org":2003,"tags":2004,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2005,2006,2007,2008],{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},{"slug":2010,"name":2010,"fn":2011,"description":2012,"org":2013,"tags":2014,"stars":23,"repoUrl":24,"updatedAt":2021},"sop-cr-finetuning","fine-tune VLM models for SOP monitoring","Fine-tune Cosmos-Reason2 (CR2) VLM for SOP monitoring. Use when you need to launch and monitor a VLM training run with a given dataset ID.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2015,2018,2019,2020],{"name":2016,"slug":2017,"type":15},"AI Infrastructure","ai-infrastructure",{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:36:22.874218",{"slug":2023,"name":2023,"fn":2024,"description":2025,"org":2026,"tags":2027,"stars":23,"repoUrl":24,"updatedAt":2039},"sop-data-augmentation","augment annotated datasets","Use when the user wants to run data augmentation on an annotated dataset, configure augmentation parameters, check augmentation status, or understand what each QA augmentation type does (BCQ, MCQ, GQA, DMCQ, DSQA, ENQA)",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2028,2031,2034,2035,2036],{"name":2029,"slug":2030,"type":15},"Data Engineering","data-engineering",{"name":2032,"slug":2033,"type":15},"Datasets","datasets",{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":2037,"slug":2038,"type":15},"QA","qa","2026-07-14T05:36:20.305657",{"slug":2041,"name":2041,"fn":2042,"description":2043,"org":2044,"tags":2045,"stars":23,"repoUrl":24,"updatedAt":2051},"sop-ddm-finetuning","fine-tune DDM-Net models for SOP monitoring","Fine-tune DDM-Net temporal boundary detector for SOP monitoring. Use when you need to launch and monitor a DDM-Net training run with a given dataset ID.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2046,2049,2050],{"name":2047,"slug":2048,"type":15},"Deep Learning","deep-learning",{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:35:16.203684",{"slug":2053,"name":2053,"fn":2054,"description":2055,"org":2056,"tags":2057,"stars":23,"repoUrl":24,"updatedAt":2062},"sop-e2e-inference","run end-to-end inference evaluations","Use when running the e2e evaluation pipeline (temporal segmentation + action recognition + accuracy) against the BP evaluation-ms HTTP API. Invoked as \u002Fsop-e2e-inference \u003Cinputs.yaml> [natural language parameter overrides]",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2058,2059,2060,2061],{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":9,"slug":8,"type":15},"2026-07-14T05:36:15.23958",{"slug":2064,"name":2064,"fn":2065,"description":2066,"org":2067,"tags":2068,"stars":23,"repoUrl":24,"updatedAt":2075},"sop-ft-orchestrate","orchestrate SOP fine-tuning pipelines","Autonomous end-to-end orchestrator for SOP fine-tuning. Runs the full Import → Augment → DDM Train → VLM Train → Evaluate → RCA loop. Interprets RCA findings across DDM, VLM and augment axes, applies config fixes autonomously, and iterates until success criteria are met or max_pipeline_iterations reached. Call with a path to an inputs.yaml or with natural language.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2069,2070,2071,2072],{"name":1991,"slug":1992,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":2073,"slug":2074,"type":15},"Orchestration","orchestration","2026-07-14T05:36:28.162686",{"items":2077,"total":2230},[2078,2096,2114,2125,2137,2151,2164,2176,2187,2198,2212,2221],{"slug":2079,"name":2079,"fn":2080,"description":2081,"org":2082,"tags":2083,"stars":2093,"repoUrl":2094,"updatedAt":2095},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2084,2087,2090],{"name":2085,"slug":2086,"type":15},"Documentation","documentation",{"name":2088,"slug":2089,"type":15},"MCP","mcp",{"name":2091,"slug":2092,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-07-20T06:00:01.461044",{"slug":2097,"name":2097,"fn":2098,"description":2099,"org":2100,"tags":2101,"stars":2111,"repoUrl":2112,"updatedAt":2113},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2102,2105,2108],{"name":2103,"slug":2104,"type":15},"Containers","containers",{"name":2106,"slug":2107,"type":15},"Deployment","deployment",{"name":2109,"slug":2110,"type":15},"Python","python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2115,"name":2115,"fn":2116,"description":2117,"org":2118,"tags":2119,"stars":2111,"repoUrl":2112,"updatedAt":2124},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2120,2123],{"name":2121,"slug":2122,"type":15},"CI\u002FCD","ci-cd",{"name":2106,"slug":2107,"type":15},"2026-07-14T05:25:59.97109",{"slug":2126,"name":2126,"fn":2127,"description":2128,"org":2129,"tags":2130,"stars":2111,"repoUrl":2112,"updatedAt":2136},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2131,2132,2133],{"name":2121,"slug":2122,"type":15},{"name":2106,"slug":2107,"type":15},{"name":2134,"slug":2135,"type":15},"GitHub","github","2026-07-27T06:06:12.278222",{"slug":2138,"name":2138,"fn":2139,"description":2140,"org":2141,"tags":2142,"stars":2111,"repoUrl":2112,"updatedAt":2150},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2143,2146,2147],{"name":2144,"slug":2145,"type":15},"Debugging","debugging",{"name":2134,"slug":2135,"type":15},{"name":2148,"slug":2149,"type":15},"Triage","triage","2026-07-14T05:25:57.442089",{"slug":2152,"name":2152,"fn":2153,"description":2154,"org":2155,"tags":2156,"stars":2111,"repoUrl":2112,"updatedAt":2163},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2157,2160],{"name":2158,"slug":2159,"type":15},"Best Practices","best-practices",{"name":2161,"slug":2162,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2165,"name":2165,"fn":2166,"description":2167,"org":2168,"tags":2169,"stars":2111,"repoUrl":2112,"updatedAt":2175},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2170,2171,2174],{"name":17,"slug":18,"type":15},{"name":2172,"slug":2173,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-07-17T06:07:11.777011",{"slug":2177,"name":2177,"fn":2178,"description":2179,"org":2180,"tags":2181,"stars":2111,"repoUrl":2112,"updatedAt":2186},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2182,2183],{"name":2037,"slug":2038,"type":15},{"name":2184,"slug":2185,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":2188,"name":2188,"fn":2189,"description":2190,"org":2191,"tags":2192,"stars":2111,"repoUrl":2112,"updatedAt":2197},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2193,2194],{"name":2106,"slug":2107,"type":15},{"name":2195,"slug":2196,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":2199,"name":2199,"fn":2200,"description":2201,"org":2202,"tags":2203,"stars":2111,"repoUrl":2112,"updatedAt":2211},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2204,2207,2208],{"name":2205,"slug":2206,"type":15},"Code Review","code-review",{"name":2134,"slug":2135,"type":15},{"name":2209,"slug":2210,"type":15},"Pull Requests","pull-requests","2026-07-14T05:26:01.226578",{"slug":2213,"name":2213,"fn":2214,"description":2215,"org":2216,"tags":2217,"stars":2111,"repoUrl":2112,"updatedAt":2220},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2218,2219],{"name":2037,"slug":2038,"type":15},{"name":2184,"slug":2185,"type":15},"2026-07-14T05:25:54.928983",{"slug":2222,"name":2222,"fn":2223,"description":2224,"org":2225,"tags":2226,"stars":2111,"repoUrl":2112,"updatedAt":2229},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2227,2228],{"name":1991,"slug":1992,"type":15},{"name":2121,"slug":2122,"type":15},"2026-07-30T05:29:03.275638",496]