[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-nvidia-day0-release":3,"mdc--2ip88u-key":34,"related-org-nvidia-day0-release":2305,"related-repo-nvidia-day0-release":2458},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"day0-release","automate quantized checkpoint releases","Deterministic end-to-end driver for day-0 quantized-checkpoint releases — chains PTQ → evaluation → comparison with enforced gates between stages (the evaluation stage deploys the checkpoint itself), and returns a publish decision (ACCEPT \u002F REGRESSION \u002F ANOMALOUS \u002F INFEASIBLE). Use when the user asks to \"release a model at day-0\", \"quantize and validate model X is within N% of baseline and tell me if it's publishable\", or \"run the full day-0 workflow\". Do NOT use for single-stage requests — quantizing only (use ptq), serving only (use deployment), evaluating only (use evaluation), or comparing two existing runs (use compare-results).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},"nvidia","NVIDIA","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fnvidia.png",[12,16,19,20],{"name":13,"slug":14,"type":15},"Automation","automation","tag",{"name":17,"slug":18,"type":15},"Deployment","deployment",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"CI\u002FCD","ci-cd",3227,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer","2026-08-31T09:18:55.832925","Apache-2.0",498,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"A unified library of SOTA model optimization techniques like quantization, distillation, pruning, neural architecture search, speculative decoding, etc. It compresses deep learning models for downstream deployment frameworks like TensorRT-LLM, TensorRT, vLLM, etc. to optimize inference speed.","https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FModel-Optimizer\u002Ftree\u002FHEAD\u002Fplugins\u002Fmodelopt\u002Fskills\u002Fday0-release","---\nname: day0-release\ndescription: Deterministic end-to-end driver for day-0 quantized-checkpoint releases — chains PTQ → evaluation → comparison with enforced gates between stages (the evaluation stage deploys the checkpoint itself), and returns a publish decision (ACCEPT \u002F REGRESSION \u002F ANOMALOUS \u002F INFEASIBLE). Use when the user asks to \"release a model at day-0\", \"quantize and validate model X is within N% of baseline and tell me if it's publishable\", or \"run the full day-0 workflow\". Do NOT use for single-stage requests — quantizing only (use ptq), serving only (use deployment), evaluating only (use evaluation), or comparing two existing runs (use compare-results).\nlicense: Apache-2.0\n---\n\n# Day-0 Release\n\nDrive a model from a pretrained checkpoint to a publish decision for a quantized\ncheckpoint, in a fixed sequence with a gate after every stage. This skill is a\n**conductor**: it sequences the existing domain skills and enforces the gates —\nit does not re-implement quantization, serving, evaluation, or comparison.\n\n**Goal (the default day-0 criterion):** a quantized checkpoint smaller than the\nsource, with accuracy drop within the threshold (default \u003C1%) on the standard\nbenchmark set versus the matching baseline, plus a publish recommendation.\n\n## When to use\n\nUse only for the full goal-driven release. For a single stage, route to the\ndomain skill directly: quantize → **ptq**, serve → **deployment**, evaluate →\n**evaluation**, compare two existing runs → **compare-results**.\n\n## Inputs\n\nResolve these before starting (ask the user for anything missing):\n\n- **Model** — HF handle or checkpoint path.\n- **Recipe \u002F qformat** — e.g. `nvfp4`, `fp8`, or a recipe path. One candidate for v1.\n- **Cluster \u002F launcher** — from `clusters.yaml` (see the common skill's\n  `environment-setup.md`).\n- **Eval set** — defaults to the evaluation skill's AA suite\n  (`recipes\u002Ftasks\u002Faa\u002F`).\n- **Threshold** — max accuracy drop; default `0.01` (1%).\n\n## The chain\n\n```text\nsetup ─▶ PTQ ─▶ canary ─▶ baseline-eval ─▶ quantized-eval ─▶ compare ─▶ verbosity ─▶ closeout\n          │        │           │                │              │           │\n       gate_ptq  \u002Fhealth    gate_run         gate_run    gate_compare  gate_verbosity\n                 + 1 gen\n```\n\nThe **evaluation** skill deploys the model it evaluates (it stands up its own\nendpoint per run), so there is no separate deploy *stage* — a serving failure\nduring evaluation surfaces through the eval gate (`DEPLOYMENT_HEALTH_FAILED`) and\ntriages to the **deployment** skill (see Step 4). The Step 2b **canary** is not\nthat: it runs *before* any evaluation precisely so an unservable checkpoint is\ncaught in ~15 min rather than after a multi-hour eval.\n\nAccuracy (Step 5) and verbosity (Step 5b) are **independent gates**; closeout\nrequires both.\n\nRun each stage by invoking the domain skill, then run its gate before\nproceeding. **Do not advance past a failed gate.** Copy this checklist and track\nprogress:\n\n```text\n- [ ] Step 0: Resolve inputs; confirm threshold and eval set\n- [ ] Step 1: Setup gate — creds present, cluster reachable\n- [ ] Step 2: PTQ (ptq skill) → gate_ptq.py\n- [ ] Step 2b: Serving canary — \u002Fhealth + one generation (deployment skill)\n- [ ] Step 3: Baseline eval (evaluation skill, deploys source) → gate_run.py\n- [ ] Step 4: Quantized eval (evaluation skill, deploys candidate) → gate_run.py\n- [ ] Step 5: Compare (compare-results skill) → external sanity → gate_compare.py → decision\n- [ ] Step 5b: Verbosity gate → gate_verbosity.py\n- [ ] Step 6: Closeout — report + publish recommendation\n```\n\n### Step 1 — Setup gate\n\nUse the common skill's `credentials.md` and `remote-execution.md` to confirm\ncredentials and cluster reachability. If either fails, stop with\n`SYSTEMIC` — do not start PTQ.\n\n### Step 2 — PTQ\n\nInvoke the **ptq** skill to produce the quantized checkpoint. Then gate:\n\n```bash\n# The ptq skill's post-PTQ validation produces a validation-summary JSON (size\n# ratio + layer-precision counts + metadata diffs; see the ptq skill's\n# references\u002Fcheckpoint-validation.md). v1 gates on that summary:\npython \"$SKILL_DIR\u002Fscripts\u002Fgate_ptq.py\" --summary \u003Cvalidation-summary.json>\n#   add `--recipe \u003Cqformat>` to override the recipe recorded in the summary\n```\n\n`gate_ptq.py` returns JSON `{pass, failure_class, detail}`. On `pass: false`,\nbranch on `failure_class` (see **Triage** below). Do not evaluate an\nunvalidated checkpoint.\n\n#### Step 2b — serving canary (MANDATORY before Step 3)\n\nThe canary itself is **already specified** by the ptq skill: see\n`ptq\u002Freferences\u002Fcheckpoint-validation.md` (required gate, canary query and the\n`Serving canary` row of its report table) and `ptq\u002FSKILL.md`. Run it there rather\nthan re-deriving it here — `gate_ptq.py` checks size, coverage and metadata, not\nwhether the checkpoint *loads*, and skipping the canary has cost a full baseline\neval against a checkpoint the serving stack could never load.\n\nOn failure use `failure_class: CHECKPOINT_NOT_SERVABLE` and drop to the\n**deployment** skill; do not proceed to Step 3.\n\n**What that spec does not cover: writing a canary that cannot lie.** Both of these\nproduced a wrong verdict on a large MoE, in opposite directions:\n\n- **Poll ceiling > load time**, with headroom. A 50 min poll against a 51 min load\n  reported failure for a checkpoint that serves fine. Large MoE loads are\n  CPU-bound fp8 dequant (~50 min); 0% GPU during *load* is normal.\n- **Print an explicit `RESULT:` on every path** and exit non-zero on failure. A\n  poll loop that falls through to the generation test exits 0 and reads as PASS.\n- Server log on shared storage, not node-local `\u002Ftmp`.\n- Canary the **as-exported** artifact, not a copy you modified to make it work.\n\n### Step 3 — Baseline eval\n\nThe baseline is the **source** (pre-quantization) model on the same task set and\nsampling params. Always run a fresh baseline via the **evaluation** skill,\nwhich deploys the source model itself. Gate with `gate_run.py`.\n\n### Step 4 — Quantized eval\n\nInvoke the **evaluation** skill on the quantized checkpoint, matching the\nbaseline's task set and sampling params. The evaluation skill stands up the\nserving endpoint itself (it builds the `deployment.command`, e.g. a\n`vllm serve …`), so a serving failure surfaces here as a failed `gate_run.py`\nwith `DEPLOYMENT_HEALTH_FAILED`. When that happens, **drop to the deployment\nskill** to reproduce and debug serving in isolation (serve the checkpoint\nstandalone, confirm `\u002Fhealth` + one generation, iterate on flags \u002F TP \u002F image \u002F\nenv vars) rather than burning full eval cycles on a broken endpoint — then carry\nthe working command back into NEL's `deployment.command` and resume the eval. If\nthe checkpoint genuinely can't serve, `POINT_INFEASIBLE`.\n\n**Before submitting: assert baseline\u002Fcandidate config parity.** The candidate\nconfig must differ from the baseline's in nothing but checkpoint path and served\nmodel name. Diff mechanically — an eyeball pass misses this:\n\n```bash\ndiff \u003C(grep -vE 'checkpoint_path|served_model_name' baseline.yaml) \\\n     \u003C(grep -vE 'checkpoint_path|served_model_name' candidate.yaml)\n```\n\nAny other difference biases the comparison and invalidates the gate: a mismatched\n`parallelism` between the two sides was worth ~2 pp, enough to invert the sign of\nthe delta. If the model card splits sampling params per scenario, apply the same\nsplit to both sides. Never set an unbounded `request_timeout` (`1e9`) — it turns a\ntransient stall into a job that holds its GPUs until the wall clock kills it.\n\nGate:\n\n```bash\npython \"$SKILL_DIR\u002Fscripts\u002Fgate_run.py\" --run \u003Crun-summary.json>\n```\n\nA `pass: false` here means the run is incomplete or invalid (judge\u002Fparse error,\ndropped samples) — do **not** compare scores from it.\n\n### Step 5 — Compare\n\nInvoke the **compare-results** skill. It must perform the shared external\nbaseline sanity check before the candidate-delta gate. A failed check is\n`ANOMALOUS` with failure class `EXTERNAL_BASELINE_MISMATCH`: investigate and\nrerun the baseline. If no credible comparable external score exists, record the\nbaseline as externally unverified and continue using the validated measured\nbaseline.\n\n#### Statistical power — check BEFORE trusting any per-task verdict\n\nA task whose measurement noise rivals the threshold cannot decide a gate. Confirm\neach task's repeat count gives a standard error below the threshold; otherwise\nmark it `INDETERMINATE` rather than reporting a pass\u002Ffail.\n\nFor example, on a 1 % gate on DeepSeek-V4-Pro, both tasks that originally failed passed once measured properly:\n\n| task | runs pooled | drop | verdict |\n| --- | --- | --- | --- |\n| SciCode | 1 | 2.96 pp | REGRESSION |\n| SciCode | 8 | **-0.96 pp** | PASS |\n| IFBench | 5 | 2.73 pp | REGRESSION |\n| IFBench | 16 | **0.63 pp** | PASS |\n\nAdd precision by submitting the benchmark **more times**, not by raising\n`num_repeats` within a run — see `recipes\u002Ftasks\u002Faa\u002Fscicode.md` for why.\n\n**Re-running does not guarantee fresh samples.** With a warm NEL response cache a \"re-run\" can\nreplay cached responses — two runs came back bit-identical to 16 digits. Confirm the score\nactually moved before counting a run as an independent repeat.\n\nAfter recording the external status, produce per-task deltas and run:\n\n```bash\npython \"$SKILL_DIR\u002Fscripts\u002Fgate_compare.py\" \\\n    --baseline \u003Cbaseline_scores.json> --candidate \u003Ccandidate_scores.json> \\\n    --threshold 0.01\n```\n\nThe threshold is a fraction of each task's score scale. Most AA tasks report\n0-100, but some (e.g. `tau2_bench_telecom` `Result`) report 0-1; the gate infers\neach task's scale (0-1 if both scores are within [0, 1], else 0-100) and\nnormalizes the drop accordingly, so `--threshold 0.01` means \"≤1 pt on a 0-100\ntask \u002F ≤0.01 on a 0-1 task\" uniformly. Pass `--scales '{\"task\": max}'` to\noverride inference if a task's scores happen to fall in an ambiguous range.\n\n`gate_compare.py` checks only the candidate delta; it cannot override a failed\nexternal baseline check. Combined decision:\n\n- **ACCEPT (accuracy only)** — no external check failed and every task is within\n  the candidate threshold → continue to **Step 5b**. This verdict covers accuracy\n  alone; advance to Step 6 only once the verbosity gate also passes. A missing\n  comparable external score is not a failure; report it as externally unverified.\n- **REGRESSION** — one or more tasks exceed threshold. **v1 stops here and\n  reports** which tasks regressed by how much. (Picking the next recipe and\n  re-running is deferred — see Scope.)\n- **ANOMALOUS** — external baseline sanity failed, or scores are otherwise\n  implausible (e.g. baseline lower than candidate by a large margin, or a task\n  score is outside its valid range) → correct the baseline or surface it.\n\n#### Step 5b — Verbosity gate (MANDATORY; independent of accuracy)\n\n`gate_compare.py` does not measure verbosity, so stopping after Step 5 leaves a\nhard gate unmeasured. Step 5's ACCEPT is accuracy-only — Step 6 requires both.\n\n```bash\npython \"$SKILL_DIR\u002Fscripts\u002Fgate_verbosity.py\" \\\n    --baseline \u003Cbaseline_eval_root> --candidate \u003Ccandidate_eval_root> \\\n    --glob 'eval_*' --exclude _high --threshold 0.05\n```\n\nExit codes follow the other gates: `0` pass, `1` the gate failed, `2` it could not read its input.\nA `2` means fix the invocation, not the checkpoint — check `--baseline`\u002F`--candidate`, `--glob`\n(`no_files_matched` names the pattern that missed), and `--exclude`. If the detail names\n`collapsed_keys`, the artifact tree gave several tasks one name: point the gate at a tree whose\ntask dirs are `\u003Charness>.\u003Ctask>` rather than `\u003Cinvocation_id>.\u003Cjob_index>`, or re-sync so each\n`artifacts\u002F` carries its own `config.yml`.\n\nRead `response_stats.avg_completion_tokens` from each task's\n`artifacts\u002Feval_factory_metrics.json`. Do **not** use the `reasoning.*` fields:\n`reasoning.*_tokens` are always `0` (only the reasoning\u002Fcontent split is missing)\nand the `*_words` siblings are a proxy that can disagree with the gate — one task\nread +6.10% FAIL in words and +1.32% PASS in tokens.\n\nThe gate is two-sided; a large drop in output length is also a change.\n\nTwo filters are mandatory: same reasoning effort (pass `--exclude` explicitly — it is empty by\ndefault, since which tier is canonical is run-specific) and complete runs only —\nruns within 1% of the largest `successful_count` that **both** sides can match (exact equality\nwould make one dropped sample unmeasurable). A task with no such count on both sides is\n`not_comparable`. When the matched count is below a run\none side has, the task carries `truncated_comparison`.\n\nTasks sharing no sample count are `not_comparable`, not a delta. Means under\n~1000 tokens carry a `short_output_warning` — read absolute counts there.\n\n### Step 6 — Closeout\n\nReport the decision with: source vs output size + ratio, per-task baseline \u002F\ncandidate \u002F delta \u002F within-threshold, **the verbosity verdict from Step 5b**,\nexternal source and sanity status, MLflow run IDs, and a publish recommendation\n(publish \u002F do-not-publish). Archive artifacts to the workspace.\n\n**Publish exactly what was evaluated.** Verify mechanically, not by path\nconvention: inode-compare a shard in the evaluated directory against the one being\npublished. A day-0 run leaves near-identical sibling exports\nthat differ by one calibration suffix — prefix rejected ones `REJECTED-` so the\nartifact cannot be picked by autocomplete, and re-run the Step 2b canary against\nthe final path after any move.\n\n## Triage (gate failure → decision)\n\nMap a gate's `failure_class` to the next action:\n\n| `failure_class` | Action |\n| --- | --- |\n| `INFRA_TRANSIENT` | Retry the stage once; if it recurs, `SYSTEMIC`. |\n| `MODEL_UNSUPPORTED` | PATCH: fix the recipe pattern \u002F add model support (ptq skill owns the patch loop), then retry. If unpatchable, `POINT_INFEASIBLE`. |\n| `QUANT_COVERAGE_FAILURE` | PATCH: fix the recipe wildcard so intended layers are covered; re-run PTQ. |\n| `SIZE_NOT_REDUCED` | The output is not smaller than the source. If the recipe cannot shrink the source (e.g. mxfp4 under nvfp4, or fp8 under fp8), record `source_precision` in the validation summary and re-run the gate — that states *why* the growth is expected. Otherwise treat it as a real compression failure: check that the recipe matched the intended parameter mass and that the exporter did not retain the original tensors. `accept_size_growth: true` waives it unconditionally (no growth bound) and is a last resort, not the fix — it records no reason, so prefer `source_precision` where the claim is checkable. |\n| `CHECKPOINT_NOT_SERVABLE` | The Step 2b canary could not load\u002Fgenerate. Usually a tensor-naming or config-schema mismatch between the exporter and the serving stack, or missing\u002Fdangling auxiliary files (tokenizer). Fix the export; do not evaluate. |\n| `VERBOSITY_EXCEEDED` | Re-check run hygiene first (mixed reasoning effort, partial runs, unequal sample counts) — that has explained every false positive so far. If the delta survives matched, complete runs, it is a real behavioural change; do not publish on accuracy alone. |\n| `DEPLOYMENT_HEALTH_FAILED` | Drop to the **deployment** skill: reproduce serving standalone (`\u002Fhealth` + one generation), debug flags \u002F image \u002F TP \u002F env, then carry the working command into NEL's `deployment.command` and retry the eval. If it can't serve, `POINT_INFEASIBLE`. |\n| `EVAL_JUDGE_FAILED` | Usually transient (auth \u002F rate limit) — wait and retry. |\n| `SAMPLE_ACCOUNTING_FAILED` | Investigate dropped\u002Ffailed samples before trusting scores. |\n| `EXTERNAL_BASELINE_MISMATCH` | Investigate baseline configuration, correct it, rerun the baseline, and repeat external sanity before comparison. |\n| `USER_CONFIG_ERROR` | Correct it from the request, workspace, or model\u002Fconfig metadata and retry; if irrecoverable, return `ANOMALOUS` with evidence. |\n| `UNKNOWN` | Investigate with the owning domain skill; if unresolved, return `ANOMALOUS` with the evidence and next automated retry or patch action. |\n\n`gate_ptq.py` also emits non-blocking `notes` (present on every result). Size growth is\n**blocking by default**, waived to a note only when the validation summary's `source_precision`\nis already at or below the recipe's target bits and the growth is within what that explains. Recording\n`source_precision` is part of the ptq skill's validation table, so the waiver is reachable from the\nnormal pipeline. A BF16 source that failed to compress still fails, which is the case this check\nexists for.\n\n`SYSTEMIC` (cluster down, dataset unavailable) aborts the whole run.\n`POINT_INFEASIBLE` means this (model, recipe) can't work as configured.\n\n## Output\n\nReturn a decision, not a raw artifact:\n\n- `ACCEPT` + report + publish recommendation\n- `REGRESSION` + which tasks failed the threshold and by how much\n- `ANOMALOUS` \u002F `INFEASIBLE` + reason and next automated action\n- Always: workspace path + MLflow run IDs for traceability\n\n## Scope (v1)\n\nIn v1: the linear chain + gates + report. On `REGRESSION`, v1 reports and stops.\nDeferred to a follow-up: the evaluator-optimizer recipe loop (compare → pick the\nnext recipe → re-run PTQ), which needs the bigpareto integration and a shared\nconfig\u002Fresult schema.\n",{"data":35,"body":36},{"name":4,"description":6,"license":26},{"type":37,"children":38},"root",[39,48,62,72,79,111,117,122,225,231,243,290,302,314,323,330,359,365,376,482,524,531,581,600,610,672,678,703,709,778,788,885,914,919,972,991,997,1023,1029,1042,1047,1179,1207,1217,1222,1323,1367,1378,1423,1429,1439,1566,1679,1737,1742,1784,1804,1810,1822,1840,1846,1858,2167,2206,2223,2229,2234,2281,2287,2299],{"type":40,"tag":41,"props":42,"children":44},"element","h1",{"id":43},"day-0-release",[45],{"type":46,"value":47},"text","Day-0 Release",{"type":40,"tag":49,"props":50,"children":51},"p",{},[52,54,60],{"type":46,"value":53},"Drive a model from a pretrained checkpoint to a publish decision for a quantized\ncheckpoint, in a fixed sequence with a gate after every stage. This skill is a\n",{"type":40,"tag":55,"props":56,"children":57},"strong",{},[58],{"type":46,"value":59},"conductor",{"type":46,"value":61},": it sequences the existing domain skills and enforces the gates —\nit does not re-implement quantization, serving, evaluation, or comparison.",{"type":40,"tag":49,"props":63,"children":64},{},[65,70],{"type":40,"tag":55,"props":66,"children":67},{},[68],{"type":46,"value":69},"Goal (the default day-0 criterion):",{"type":46,"value":71}," a quantized checkpoint smaller than the\nsource, with accuracy drop within the threshold (default \u003C1%) on the standard\nbenchmark set versus the matching baseline, plus a publish recommendation.",{"type":40,"tag":73,"props":74,"children":76},"h2",{"id":75},"when-to-use",[77],{"type":46,"value":78},"When to use",{"type":40,"tag":49,"props":80,"children":81},{},[82,84,89,91,95,97,102,104,109],{"type":46,"value":83},"Use only for the full goal-driven release. For a single stage, route to the\ndomain skill directly: quantize → ",{"type":40,"tag":55,"props":85,"children":86},{},[87],{"type":46,"value":88},"ptq",{"type":46,"value":90},", serve → ",{"type":40,"tag":55,"props":92,"children":93},{},[94],{"type":46,"value":18},{"type":46,"value":96},", evaluate →\n",{"type":40,"tag":55,"props":98,"children":99},{},[100],{"type":46,"value":101},"evaluation",{"type":46,"value":103},", compare two existing runs → ",{"type":40,"tag":55,"props":105,"children":106},{},[107],{"type":46,"value":108},"compare-results",{"type":46,"value":110},".",{"type":40,"tag":73,"props":112,"children":114},{"id":113},"inputs",[115],{"type":46,"value":116},"Inputs",{"type":40,"tag":49,"props":118,"children":119},{},[120],{"type":46,"value":121},"Resolve these before starting (ask the user for anything missing):",{"type":40,"tag":123,"props":124,"children":125},"ul",{},[126,137,164,190,207],{"type":40,"tag":127,"props":128,"children":129},"li",{},[130,135],{"type":40,"tag":55,"props":131,"children":132},{},[133],{"type":46,"value":134},"Model",{"type":46,"value":136}," — HF handle or checkpoint path.",{"type":40,"tag":127,"props":138,"children":139},{},[140,145,147,154,156,162],{"type":40,"tag":55,"props":141,"children":142},{},[143],{"type":46,"value":144},"Recipe \u002F qformat",{"type":46,"value":146}," — e.g. ",{"type":40,"tag":148,"props":149,"children":151},"code",{"className":150},[],[152],{"type":46,"value":153},"nvfp4",{"type":46,"value":155},", ",{"type":40,"tag":148,"props":157,"children":159},{"className":158},[],[160],{"type":46,"value":161},"fp8",{"type":46,"value":163},", or a recipe path. One candidate for v1.",{"type":40,"tag":127,"props":165,"children":166},{},[167,172,174,180,182,188],{"type":40,"tag":55,"props":168,"children":169},{},[170],{"type":46,"value":171},"Cluster \u002F launcher",{"type":46,"value":173}," — from ",{"type":40,"tag":148,"props":175,"children":177},{"className":176},[],[178],{"type":46,"value":179},"clusters.yaml",{"type":46,"value":181}," (see the common skill's\n",{"type":40,"tag":148,"props":183,"children":185},{"className":184},[],[186],{"type":46,"value":187},"environment-setup.md",{"type":46,"value":189},").",{"type":40,"tag":127,"props":191,"children":192},{},[193,198,200,206],{"type":40,"tag":55,"props":194,"children":195},{},[196],{"type":46,"value":197},"Eval set",{"type":46,"value":199}," — defaults to the evaluation skill's AA suite\n(",{"type":40,"tag":148,"props":201,"children":203},{"className":202},[],[204],{"type":46,"value":205},"recipes\u002Ftasks\u002Faa\u002F",{"type":46,"value":189},{"type":40,"tag":127,"props":208,"children":209},{},[210,215,217,223],{"type":40,"tag":55,"props":211,"children":212},{},[213],{"type":46,"value":214},"Threshold",{"type":46,"value":216}," — max accuracy drop; default ",{"type":40,"tag":148,"props":218,"children":220},{"className":219},[],[221],{"type":46,"value":222},"0.01",{"type":46,"value":224}," (1%).",{"type":40,"tag":73,"props":226,"children":228},{"id":227},"the-chain",[229],{"type":46,"value":230},"The chain",{"type":40,"tag":232,"props":233,"children":238},"pre",{"className":234,"code":236,"language":46,"meta":237},[235],"language-text","setup ─▶ PTQ ─▶ canary ─▶ baseline-eval ─▶ quantized-eval ─▶ compare ─▶ verbosity ─▶ closeout\n          │        │           │                │              │           │\n       gate_ptq  \u002Fhealth    gate_run         gate_run    gate_compare  gate_verbosity\n                 + 1 gen\n","",[239],{"type":40,"tag":148,"props":240,"children":241},{"__ignoreMap":237},[242],{"type":46,"value":236},{"type":40,"tag":49,"props":244,"children":245},{},[246,248,252,254,260,262,268,270,274,276,281,283,288],{"type":46,"value":247},"The ",{"type":40,"tag":55,"props":249,"children":250},{},[251],{"type":46,"value":101},{"type":46,"value":253}," skill deploys the model it evaluates (it stands up its own\nendpoint per run), so there is no separate deploy ",{"type":40,"tag":255,"props":256,"children":257},"em",{},[258],{"type":46,"value":259},"stage",{"type":46,"value":261}," — a serving failure\nduring evaluation surfaces through the eval gate (",{"type":40,"tag":148,"props":263,"children":265},{"className":264},[],[266],{"type":46,"value":267},"DEPLOYMENT_HEALTH_FAILED",{"type":46,"value":269},") and\ntriages to the ",{"type":40,"tag":55,"props":271,"children":272},{},[273],{"type":46,"value":18},{"type":46,"value":275}," skill (see Step 4). The Step 2b ",{"type":40,"tag":55,"props":277,"children":278},{},[279],{"type":46,"value":280},"canary",{"type":46,"value":282}," is not\nthat: it runs ",{"type":40,"tag":255,"props":284,"children":285},{},[286],{"type":46,"value":287},"before",{"type":46,"value":289}," any evaluation precisely so an unservable checkpoint is\ncaught in ~15 min rather than after a multi-hour eval.",{"type":40,"tag":49,"props":291,"children":292},{},[293,295,300],{"type":46,"value":294},"Accuracy (Step 5) and verbosity (Step 5b) are ",{"type":40,"tag":55,"props":296,"children":297},{},[298],{"type":46,"value":299},"independent gates",{"type":46,"value":301},"; closeout\nrequires both.",{"type":40,"tag":49,"props":303,"children":304},{},[305,307,312],{"type":46,"value":306},"Run each stage by invoking the domain skill, then run its gate before\nproceeding. ",{"type":40,"tag":55,"props":308,"children":309},{},[310],{"type":46,"value":311},"Do not advance past a failed gate.",{"type":46,"value":313}," Copy this checklist and track\nprogress:",{"type":40,"tag":232,"props":315,"children":318},{"className":316,"code":317,"language":46,"meta":237},[235],"- [ ] Step 0: Resolve inputs; confirm threshold and eval set\n- [ ] Step 1: Setup gate — creds present, cluster reachable\n- [ ] Step 2: PTQ (ptq skill) → gate_ptq.py\n- [ ] Step 2b: Serving canary — \u002Fhealth + one generation (deployment skill)\n- [ ] Step 3: Baseline eval (evaluation skill, deploys source) → gate_run.py\n- [ ] Step 4: Quantized eval (evaluation skill, deploys candidate) → gate_run.py\n- [ ] Step 5: Compare (compare-results skill) → external sanity → gate_compare.py → decision\n- [ ] Step 5b: Verbosity gate → gate_verbosity.py\n- [ ] Step 6: Closeout — report + publish recommendation\n",[319],{"type":40,"tag":148,"props":320,"children":321},{"__ignoreMap":237},[322],{"type":46,"value":317},{"type":40,"tag":324,"props":325,"children":327},"h3",{"id":326},"step-1-setup-gate",[328],{"type":46,"value":329},"Step 1 — Setup gate",{"type":40,"tag":49,"props":331,"children":332},{},[333,335,341,343,349,351,357],{"type":46,"value":334},"Use the common skill's ",{"type":40,"tag":148,"props":336,"children":338},{"className":337},[],[339],{"type":46,"value":340},"credentials.md",{"type":46,"value":342}," and ",{"type":40,"tag":148,"props":344,"children":346},{"className":345},[],[347],{"type":46,"value":348},"remote-execution.md",{"type":46,"value":350}," to confirm\ncredentials and cluster reachability. If either fails, stop with\n",{"type":40,"tag":148,"props":352,"children":354},{"className":353},[],[355],{"type":46,"value":356},"SYSTEMIC",{"type":46,"value":358}," — do not start PTQ.",{"type":40,"tag":324,"props":360,"children":362},{"id":361},"step-2-ptq",[363],{"type":46,"value":364},"Step 2 — PTQ",{"type":40,"tag":49,"props":366,"children":367},{},[368,370,374],{"type":46,"value":369},"Invoke the ",{"type":40,"tag":55,"props":371,"children":372},{},[373],{"type":46,"value":88},{"type":46,"value":375}," skill to produce the quantized checkpoint. Then gate:",{"type":40,"tag":232,"props":377,"children":381},{"className":378,"code":379,"language":380,"meta":237,"style":237},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# The ptq skill's post-PTQ validation produces a validation-summary JSON (size\n# ratio + layer-precision counts + metadata diffs; see the ptq skill's\n# references\u002Fcheckpoint-validation.md). v1 gates on that summary:\npython \"$SKILL_DIR\u002Fscripts\u002Fgate_ptq.py\" --summary \u003Cvalidation-summary.json>\n#   add `--recipe \u003Cqformat>` to override the recipe recorded in the summary\n","bash",[382],{"type":40,"tag":148,"props":383,"children":384},{"__ignoreMap":237},[385,397,406,415,473],{"type":40,"tag":386,"props":387,"children":390},"span",{"class":388,"line":389},"line",1,[391],{"type":40,"tag":386,"props":392,"children":394},{"style":393},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[395],{"type":46,"value":396},"# The ptq skill's post-PTQ validation produces a validation-summary JSON (size\n",{"type":40,"tag":386,"props":398,"children":400},{"class":388,"line":399},2,[401],{"type":40,"tag":386,"props":402,"children":403},{"style":393},[404],{"type":46,"value":405},"# ratio + layer-precision counts + metadata diffs; see the ptq skill's\n",{"type":40,"tag":386,"props":407,"children":409},{"class":388,"line":408},3,[410],{"type":40,"tag":386,"props":411,"children":412},{"style":393},[413],{"type":46,"value":414},"# references\u002Fcheckpoint-validation.md). v1 gates on that summary:\n",{"type":40,"tag":386,"props":416,"children":418},{"class":388,"line":417},4,[419,425,431,437,443,448,453,458,463,468],{"type":40,"tag":386,"props":420,"children":422},{"style":421},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[423],{"type":46,"value":424},"python",{"type":40,"tag":386,"props":426,"children":428},{"style":427},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[429],{"type":46,"value":430}," \"",{"type":40,"tag":386,"props":432,"children":434},{"style":433},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[435],{"type":46,"value":436},"$SKILL_DIR",{"type":40,"tag":386,"props":438,"children":440},{"style":439},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[441],{"type":46,"value":442},"\u002Fscripts\u002Fgate_ptq.py",{"type":40,"tag":386,"props":444,"children":445},{"style":427},[446],{"type":46,"value":447},"\"",{"type":40,"tag":386,"props":449,"children":450},{"style":439},[451],{"type":46,"value":452}," --summary",{"type":40,"tag":386,"props":454,"children":455},{"style":427},[456],{"type":46,"value":457}," \u003C",{"type":40,"tag":386,"props":459,"children":460},{"style":439},[461],{"type":46,"value":462},"validation-summary.jso",{"type":40,"tag":386,"props":464,"children":465},{"style":433},[466],{"type":46,"value":467},"n",{"type":40,"tag":386,"props":469,"children":470},{"style":427},[471],{"type":46,"value":472},">\n",{"type":40,"tag":386,"props":474,"children":476},{"class":388,"line":475},5,[477],{"type":40,"tag":386,"props":478,"children":479},{"style":393},[480],{"type":46,"value":481},"#   add `--recipe \u003Cqformat>` to override the recipe recorded in the summary\n",{"type":40,"tag":49,"props":483,"children":484},{},[485,491,493,499,501,507,509,515,517,522],{"type":40,"tag":148,"props":486,"children":488},{"className":487},[],[489],{"type":46,"value":490},"gate_ptq.py",{"type":46,"value":492}," returns JSON ",{"type":40,"tag":148,"props":494,"children":496},{"className":495},[],[497],{"type":46,"value":498},"{pass, failure_class, detail}",{"type":46,"value":500},". On ",{"type":40,"tag":148,"props":502,"children":504},{"className":503},[],[505],{"type":46,"value":506},"pass: false",{"type":46,"value":508},",\nbranch on ",{"type":40,"tag":148,"props":510,"children":512},{"className":511},[],[513],{"type":46,"value":514},"failure_class",{"type":46,"value":516}," (see ",{"type":40,"tag":55,"props":518,"children":519},{},[520],{"type":46,"value":521},"Triage",{"type":46,"value":523}," below). Do not evaluate an\nunvalidated checkpoint.",{"type":40,"tag":525,"props":526,"children":528},"h4",{"id":527},"step-2b-serving-canary-mandatory-before-step-3",[529],{"type":46,"value":530},"Step 2b — serving canary (MANDATORY before Step 3)",{"type":40,"tag":49,"props":532,"children":533},{},[534,536,541,543,549,551,557,559,565,567,572,574,579],{"type":46,"value":535},"The canary itself is ",{"type":40,"tag":55,"props":537,"children":538},{},[539],{"type":46,"value":540},"already specified",{"type":46,"value":542}," by the ptq skill: see\n",{"type":40,"tag":148,"props":544,"children":546},{"className":545},[],[547],{"type":46,"value":548},"ptq\u002Freferences\u002Fcheckpoint-validation.md",{"type":46,"value":550}," (required gate, canary query and the\n",{"type":40,"tag":148,"props":552,"children":554},{"className":553},[],[555],{"type":46,"value":556},"Serving canary",{"type":46,"value":558}," row of its report table) and ",{"type":40,"tag":148,"props":560,"children":562},{"className":561},[],[563],{"type":46,"value":564},"ptq\u002FSKILL.md",{"type":46,"value":566},". Run it there rather\nthan re-deriving it here — ",{"type":40,"tag":148,"props":568,"children":570},{"className":569},[],[571],{"type":46,"value":490},{"type":46,"value":573}," checks size, coverage and metadata, not\nwhether the checkpoint ",{"type":40,"tag":255,"props":575,"children":576},{},[577],{"type":46,"value":578},"loads",{"type":46,"value":580},", and skipping the canary has cost a full baseline\neval against a checkpoint the serving stack could never load.",{"type":40,"tag":49,"props":582,"children":583},{},[584,586,592,594,598],{"type":46,"value":585},"On failure use ",{"type":40,"tag":148,"props":587,"children":589},{"className":588},[],[590],{"type":46,"value":591},"failure_class: CHECKPOINT_NOT_SERVABLE",{"type":46,"value":593}," and drop to the\n",{"type":40,"tag":55,"props":595,"children":596},{},[597],{"type":46,"value":18},{"type":46,"value":599}," skill; do not proceed to Step 3.",{"type":40,"tag":49,"props":601,"children":602},{},[603,608],{"type":40,"tag":55,"props":604,"children":605},{},[606],{"type":46,"value":607},"What that spec does not cover: writing a canary that cannot lie.",{"type":46,"value":609}," Both of these\nproduced a wrong verdict on a large MoE, in opposite directions:",{"type":40,"tag":123,"props":611,"children":612},{},[613,630,648,660],{"type":40,"tag":127,"props":614,"children":615},{},[616,621,623,628],{"type":40,"tag":55,"props":617,"children":618},{},[619],{"type":46,"value":620},"Poll ceiling > load time",{"type":46,"value":622},", with headroom. A 50 min poll against a 51 min load\nreported failure for a checkpoint that serves fine. Large MoE loads are\nCPU-bound fp8 dequant (~50 min); 0% GPU during ",{"type":40,"tag":255,"props":624,"children":625},{},[626],{"type":46,"value":627},"load",{"type":46,"value":629}," is normal.",{"type":40,"tag":127,"props":631,"children":632},{},[633,646],{"type":40,"tag":55,"props":634,"children":635},{},[636,638,644],{"type":46,"value":637},"Print an explicit ",{"type":40,"tag":148,"props":639,"children":641},{"className":640},[],[642],{"type":46,"value":643},"RESULT:",{"type":46,"value":645}," on every path",{"type":46,"value":647}," and exit non-zero on failure. A\npoll loop that falls through to the generation test exits 0 and reads as PASS.",{"type":40,"tag":127,"props":649,"children":650},{},[651,653,659],{"type":46,"value":652},"Server log on shared storage, not node-local ",{"type":40,"tag":148,"props":654,"children":656},{"className":655},[],[657],{"type":46,"value":658},"\u002Ftmp",{"type":46,"value":110},{"type":40,"tag":127,"props":661,"children":662},{},[663,665,670],{"type":46,"value":664},"Canary the ",{"type":40,"tag":55,"props":666,"children":667},{},[668],{"type":46,"value":669},"as-exported",{"type":46,"value":671}," artifact, not a copy you modified to make it work.",{"type":40,"tag":324,"props":673,"children":675},{"id":674},"step-3-baseline-eval",[676],{"type":46,"value":677},"Step 3 — Baseline eval",{"type":40,"tag":49,"props":679,"children":680},{},[681,683,688,690,694,696,702],{"type":46,"value":682},"The baseline is the ",{"type":40,"tag":55,"props":684,"children":685},{},[686],{"type":46,"value":687},"source",{"type":46,"value":689}," (pre-quantization) model on the same task set and\nsampling params. Always run a fresh baseline via the ",{"type":40,"tag":55,"props":691,"children":692},{},[693],{"type":46,"value":101},{"type":46,"value":695}," skill,\nwhich deploys the source model itself. Gate with ",{"type":40,"tag":148,"props":697,"children":699},{"className":698},[],[700],{"type":46,"value":701},"gate_run.py",{"type":46,"value":110},{"type":40,"tag":324,"props":704,"children":706},{"id":705},"step-4-quantized-eval",[707],{"type":46,"value":708},"Step 4 — Quantized eval",{"type":40,"tag":49,"props":710,"children":711},{},[712,713,717,719,725,727,733,735,740,742,747,749,754,756,762,764,769,771,777],{"type":46,"value":369},{"type":40,"tag":55,"props":714,"children":715},{},[716],{"type":46,"value":101},{"type":46,"value":718}," skill on the quantized checkpoint, matching the\nbaseline's task set and sampling params. The evaluation skill stands up the\nserving endpoint itself (it builds the ",{"type":40,"tag":148,"props":720,"children":722},{"className":721},[],[723],{"type":46,"value":724},"deployment.command",{"type":46,"value":726},", e.g. a\n",{"type":40,"tag":148,"props":728,"children":730},{"className":729},[],[731],{"type":46,"value":732},"vllm serve …",{"type":46,"value":734},"), so a serving failure surfaces here as a failed ",{"type":40,"tag":148,"props":736,"children":738},{"className":737},[],[739],{"type":46,"value":701},{"type":46,"value":741},"\nwith ",{"type":40,"tag":148,"props":743,"children":745},{"className":744},[],[746],{"type":46,"value":267},{"type":46,"value":748},". When that happens, ",{"type":40,"tag":55,"props":750,"children":751},{},[752],{"type":46,"value":753},"drop to the deployment\nskill",{"type":46,"value":755}," to reproduce and debug serving in isolation (serve the checkpoint\nstandalone, confirm ",{"type":40,"tag":148,"props":757,"children":759},{"className":758},[],[760],{"type":46,"value":761},"\u002Fhealth",{"type":46,"value":763}," + one generation, iterate on flags \u002F TP \u002F image \u002F\nenv vars) rather than burning full eval cycles on a broken endpoint — then carry\nthe working command back into NEL's ",{"type":40,"tag":148,"props":765,"children":767},{"className":766},[],[768],{"type":46,"value":724},{"type":46,"value":770}," and resume the eval. If\nthe checkpoint genuinely can't serve, ",{"type":40,"tag":148,"props":772,"children":774},{"className":773},[],[775],{"type":46,"value":776},"POINT_INFEASIBLE",{"type":46,"value":110},{"type":40,"tag":49,"props":779,"children":780},{},[781,786],{"type":40,"tag":55,"props":782,"children":783},{},[784],{"type":46,"value":785},"Before submitting: assert baseline\u002Fcandidate config parity.",{"type":46,"value":787}," The candidate\nconfig must differ from the baseline's in nothing but checkpoint path and served\nmodel name. Diff mechanically — an eyeball pass misses this:",{"type":40,"tag":232,"props":789,"children":791},{"className":378,"code":790,"language":380,"meta":237,"style":237},"diff \u003C(grep -vE 'checkpoint_path|served_model_name' baseline.yaml) \\\n     \u003C(grep -vE 'checkpoint_path|served_model_name' candidate.yaml)\n",[792],{"type":40,"tag":148,"props":793,"children":794},{"__ignoreMap":237},[795,847],{"type":40,"tag":386,"props":796,"children":797},{"class":388,"line":389},[798,803,808,813,818,823,828,832,837,842],{"type":40,"tag":386,"props":799,"children":800},{"style":421},[801],{"type":46,"value":802},"diff",{"type":40,"tag":386,"props":804,"children":805},{"style":427},[806],{"type":46,"value":807}," \u003C(",{"type":40,"tag":386,"props":809,"children":810},{"style":421},[811],{"type":46,"value":812},"grep",{"type":40,"tag":386,"props":814,"children":815},{"style":439},[816],{"type":46,"value":817}," -vE ",{"type":40,"tag":386,"props":819,"children":820},{"style":427},[821],{"type":46,"value":822},"'",{"type":40,"tag":386,"props":824,"children":825},{"style":439},[826],{"type":46,"value":827},"checkpoint_path|served_model_name",{"type":40,"tag":386,"props":829,"children":830},{"style":427},[831],{"type":46,"value":822},{"type":40,"tag":386,"props":833,"children":834},{"style":439},[835],{"type":46,"value":836}," baseline.yaml",{"type":40,"tag":386,"props":838,"children":839},{"style":427},[840],{"type":46,"value":841},")",{"type":40,"tag":386,"props":843,"children":844},{"style":433},[845],{"type":46,"value":846}," \\\n",{"type":40,"tag":386,"props":848,"children":849},{"class":388,"line":399},[850,855,859,863,867,871,875,880],{"type":40,"tag":386,"props":851,"children":852},{"style":427},[853],{"type":46,"value":854},"     \u003C(",{"type":40,"tag":386,"props":856,"children":857},{"style":421},[858],{"type":46,"value":812},{"type":40,"tag":386,"props":860,"children":861},{"style":439},[862],{"type":46,"value":817},{"type":40,"tag":386,"props":864,"children":865},{"style":427},[866],{"type":46,"value":822},{"type":40,"tag":386,"props":868,"children":869},{"style":439},[870],{"type":46,"value":827},{"type":40,"tag":386,"props":872,"children":873},{"style":427},[874],{"type":46,"value":822},{"type":40,"tag":386,"props":876,"children":877},{"style":439},[878],{"type":46,"value":879}," candidate.yaml",{"type":40,"tag":386,"props":881,"children":882},{"style":427},[883],{"type":46,"value":884},")\n",{"type":40,"tag":49,"props":886,"children":887},{},[888,890,896,898,904,906,912],{"type":46,"value":889},"Any other difference biases the comparison and invalidates the gate: a mismatched\n",{"type":40,"tag":148,"props":891,"children":893},{"className":892},[],[894],{"type":46,"value":895},"parallelism",{"type":46,"value":897}," between the two sides was worth ~2 pp, enough to invert the sign of\nthe delta. If the model card splits sampling params per scenario, apply the same\nsplit to both sides. Never set an unbounded ",{"type":40,"tag":148,"props":899,"children":901},{"className":900},[],[902],{"type":46,"value":903},"request_timeout",{"type":46,"value":905}," (",{"type":40,"tag":148,"props":907,"children":909},{"className":908},[],[910],{"type":46,"value":911},"1e9",{"type":46,"value":913},") — it turns a\ntransient stall into a job that holds its GPUs until the wall clock kills it.",{"type":40,"tag":49,"props":915,"children":916},{},[917],{"type":46,"value":918},"Gate:",{"type":40,"tag":232,"props":920,"children":922},{"className":378,"code":921,"language":380,"meta":237,"style":237},"python \"$SKILL_DIR\u002Fscripts\u002Fgate_run.py\" --run \u003Crun-summary.json>\n",[923],{"type":40,"tag":148,"props":924,"children":925},{"__ignoreMap":237},[926],{"type":40,"tag":386,"props":927,"children":928},{"class":388,"line":389},[929,933,937,941,946,950,955,959,964,968],{"type":40,"tag":386,"props":930,"children":931},{"style":421},[932],{"type":46,"value":424},{"type":40,"tag":386,"props":934,"children":935},{"style":427},[936],{"type":46,"value":430},{"type":40,"tag":386,"props":938,"children":939},{"style":433},[940],{"type":46,"value":436},{"type":40,"tag":386,"props":942,"children":943},{"style":439},[944],{"type":46,"value":945},"\u002Fscripts\u002Fgate_run.py",{"type":40,"tag":386,"props":947,"children":948},{"style":427},[949],{"type":46,"value":447},{"type":40,"tag":386,"props":951,"children":952},{"style":439},[953],{"type":46,"value":954}," --run",{"type":40,"tag":386,"props":956,"children":957},{"style":427},[958],{"type":46,"value":457},{"type":40,"tag":386,"props":960,"children":961},{"style":439},[962],{"type":46,"value":963},"run-summary.jso",{"type":40,"tag":386,"props":965,"children":966},{"style":433},[967],{"type":46,"value":467},{"type":40,"tag":386,"props":969,"children":970},{"style":427},[971],{"type":46,"value":472},{"type":40,"tag":49,"props":973,"children":974},{},[975,977,982,984,989],{"type":46,"value":976},"A ",{"type":40,"tag":148,"props":978,"children":980},{"className":979},[],[981],{"type":46,"value":506},{"type":46,"value":983}," here means the run is incomplete or invalid (judge\u002Fparse error,\ndropped samples) — do ",{"type":40,"tag":55,"props":985,"children":986},{},[987],{"type":46,"value":988},"not",{"type":46,"value":990}," compare scores from it.",{"type":40,"tag":324,"props":992,"children":994},{"id":993},"step-5-compare",[995],{"type":46,"value":996},"Step 5 — Compare",{"type":40,"tag":49,"props":998,"children":999},{},[1000,1001,1005,1007,1013,1015,1021],{"type":46,"value":369},{"type":40,"tag":55,"props":1002,"children":1003},{},[1004],{"type":46,"value":108},{"type":46,"value":1006}," skill. It must perform the shared external\nbaseline sanity check before the candidate-delta gate. A failed check is\n",{"type":40,"tag":148,"props":1008,"children":1010},{"className":1009},[],[1011],{"type":46,"value":1012},"ANOMALOUS",{"type":46,"value":1014}," with failure class ",{"type":40,"tag":148,"props":1016,"children":1018},{"className":1017},[],[1019],{"type":46,"value":1020},"EXTERNAL_BASELINE_MISMATCH",{"type":46,"value":1022},": investigate and\nrerun the baseline. If no credible comparable external score exists, record the\nbaseline as externally unverified and continue using the validated measured\nbaseline.",{"type":40,"tag":525,"props":1024,"children":1026},{"id":1025},"statistical-power-check-before-trusting-any-per-task-verdict",[1027],{"type":46,"value":1028},"Statistical power — check BEFORE trusting any per-task verdict",{"type":40,"tag":49,"props":1030,"children":1031},{},[1032,1034,1040],{"type":46,"value":1033},"A task whose measurement noise rivals the threshold cannot decide a gate. Confirm\neach task's repeat count gives a standard error below the threshold; otherwise\nmark it ",{"type":40,"tag":148,"props":1035,"children":1037},{"className":1036},[],[1038],{"type":46,"value":1039},"INDETERMINATE",{"type":46,"value":1041}," rather than reporting a pass\u002Ffail.",{"type":40,"tag":49,"props":1043,"children":1044},{},[1045],{"type":46,"value":1046},"For example, on a 1 % gate on DeepSeek-V4-Pro, both tasks that originally failed passed once measured properly:",{"type":40,"tag":1048,"props":1049,"children":1050},"table",{},[1051,1080],{"type":40,"tag":1052,"props":1053,"children":1054},"thead",{},[1055],{"type":40,"tag":1056,"props":1057,"children":1058},"tr",{},[1059,1065,1070,1075],{"type":40,"tag":1060,"props":1061,"children":1062},"th",{},[1063],{"type":46,"value":1064},"task",{"type":40,"tag":1060,"props":1066,"children":1067},{},[1068],{"type":46,"value":1069},"runs pooled",{"type":40,"tag":1060,"props":1071,"children":1072},{},[1073],{"type":46,"value":1074},"drop",{"type":40,"tag":1060,"props":1076,"children":1077},{},[1078],{"type":46,"value":1079},"verdict",{"type":40,"tag":1081,"props":1082,"children":1083},"tbody",{},[1084,1108,1133,1155],{"type":40,"tag":1056,"props":1085,"children":1086},{},[1087,1093,1098,1103],{"type":40,"tag":1088,"props":1089,"children":1090},"td",{},[1091],{"type":46,"value":1092},"SciCode",{"type":40,"tag":1088,"props":1094,"children":1095},{},[1096],{"type":46,"value":1097},"1",{"type":40,"tag":1088,"props":1099,"children":1100},{},[1101],{"type":46,"value":1102},"2.96 pp",{"type":40,"tag":1088,"props":1104,"children":1105},{},[1106],{"type":46,"value":1107},"REGRESSION",{"type":40,"tag":1056,"props":1109,"children":1110},{},[1111,1115,1120,1128],{"type":40,"tag":1088,"props":1112,"children":1113},{},[1114],{"type":46,"value":1092},{"type":40,"tag":1088,"props":1116,"children":1117},{},[1118],{"type":46,"value":1119},"8",{"type":40,"tag":1088,"props":1121,"children":1122},{},[1123],{"type":40,"tag":55,"props":1124,"children":1125},{},[1126],{"type":46,"value":1127},"-0.96 pp",{"type":40,"tag":1088,"props":1129,"children":1130},{},[1131],{"type":46,"value":1132},"PASS",{"type":40,"tag":1056,"props":1134,"children":1135},{},[1136,1141,1146,1151],{"type":40,"tag":1088,"props":1137,"children":1138},{},[1139],{"type":46,"value":1140},"IFBench",{"type":40,"tag":1088,"props":1142,"children":1143},{},[1144],{"type":46,"value":1145},"5",{"type":40,"tag":1088,"props":1147,"children":1148},{},[1149],{"type":46,"value":1150},"2.73 pp",{"type":40,"tag":1088,"props":1152,"children":1153},{},[1154],{"type":46,"value":1107},{"type":40,"tag":1056,"props":1156,"children":1157},{},[1158,1162,1167,1175],{"type":40,"tag":1088,"props":1159,"children":1160},{},[1161],{"type":46,"value":1140},{"type":40,"tag":1088,"props":1163,"children":1164},{},[1165],{"type":46,"value":1166},"16",{"type":40,"tag":1088,"props":1168,"children":1169},{},[1170],{"type":40,"tag":55,"props":1171,"children":1172},{},[1173],{"type":46,"value":1174},"0.63 pp",{"type":40,"tag":1088,"props":1176,"children":1177},{},[1178],{"type":46,"value":1132},{"type":40,"tag":49,"props":1180,"children":1181},{},[1182,1184,1189,1191,1197,1199,1205],{"type":46,"value":1183},"Add precision by submitting the benchmark ",{"type":40,"tag":55,"props":1185,"children":1186},{},[1187],{"type":46,"value":1188},"more times",{"type":46,"value":1190},", not by raising\n",{"type":40,"tag":148,"props":1192,"children":1194},{"className":1193},[],[1195],{"type":46,"value":1196},"num_repeats",{"type":46,"value":1198}," within a run — see ",{"type":40,"tag":148,"props":1200,"children":1202},{"className":1201},[],[1203],{"type":46,"value":1204},"recipes\u002Ftasks\u002Faa\u002Fscicode.md",{"type":46,"value":1206}," for why.",{"type":40,"tag":49,"props":1208,"children":1209},{},[1210,1215],{"type":40,"tag":55,"props":1211,"children":1212},{},[1213],{"type":46,"value":1214},"Re-running does not guarantee fresh samples.",{"type":46,"value":1216}," With a warm NEL response cache a \"re-run\" can\nreplay cached responses — two runs came back bit-identical to 16 digits. Confirm the score\nactually moved before counting a run as an independent repeat.",{"type":40,"tag":49,"props":1218,"children":1219},{},[1220],{"type":46,"value":1221},"After recording the external status, produce per-task deltas and run:",{"type":40,"tag":232,"props":1223,"children":1225},{"className":378,"code":1224,"language":380,"meta":237,"style":237},"python \"$SKILL_DIR\u002Fscripts\u002Fgate_compare.py\" \\\n    --baseline \u003Cbaseline_scores.json> --candidate \u003Ccandidate_scores.json> \\\n    --threshold 0.01\n",[1226],{"type":40,"tag":148,"props":1227,"children":1228},{"__ignoreMap":237},[1229,1257,1309],{"type":40,"tag":386,"props":1230,"children":1231},{"class":388,"line":389},[1232,1236,1240,1244,1249,1253],{"type":40,"tag":386,"props":1233,"children":1234},{"style":421},[1235],{"type":46,"value":424},{"type":40,"tag":386,"props":1237,"children":1238},{"style":427},[1239],{"type":46,"value":430},{"type":40,"tag":386,"props":1241,"children":1242},{"style":433},[1243],{"type":46,"value":436},{"type":40,"tag":386,"props":1245,"children":1246},{"style":439},[1247],{"type":46,"value":1248},"\u002Fscripts\u002Fgate_compare.py",{"type":40,"tag":386,"props":1250,"children":1251},{"style":427},[1252],{"type":46,"value":447},{"type":40,"tag":386,"props":1254,"children":1255},{"style":433},[1256],{"type":46,"value":846},{"type":40,"tag":386,"props":1258,"children":1259},{"class":388,"line":399},[1260,1265,1269,1274,1278,1283,1288,1292,1297,1301,1305],{"type":40,"tag":386,"props":1261,"children":1262},{"style":439},[1263],{"type":46,"value":1264},"    --baseline",{"type":40,"tag":386,"props":1266,"children":1267},{"style":427},[1268],{"type":46,"value":457},{"type":40,"tag":386,"props":1270,"children":1271},{"style":439},[1272],{"type":46,"value":1273},"baseline_scores.jso",{"type":40,"tag":386,"props":1275,"children":1276},{"style":433},[1277],{"type":46,"value":467},{"type":40,"tag":386,"props":1279,"children":1280},{"style":427},[1281],{"type":46,"value":1282},">",{"type":40,"tag":386,"props":1284,"children":1285},{"style":439},[1286],{"type":46,"value":1287}," --candidate",{"type":40,"tag":386,"props":1289,"children":1290},{"style":427},[1291],{"type":46,"value":457},{"type":40,"tag":386,"props":1293,"children":1294},{"style":439},[1295],{"type":46,"value":1296},"candidate_scores.jso",{"type":40,"tag":386,"props":1298,"children":1299},{"style":433},[1300],{"type":46,"value":467},{"type":40,"tag":386,"props":1302,"children":1303},{"style":427},[1304],{"type":46,"value":1282},{"type":40,"tag":386,"props":1306,"children":1307},{"style":433},[1308],{"type":46,"value":846},{"type":40,"tag":386,"props":1310,"children":1311},{"class":388,"line":408},[1312,1317],{"type":40,"tag":386,"props":1313,"children":1314},{"style":439},[1315],{"type":46,"value":1316},"    --threshold",{"type":40,"tag":386,"props":1318,"children":1320},{"style":1319},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1321],{"type":46,"value":1322}," 0.01\n",{"type":40,"tag":49,"props":1324,"children":1325},{},[1326,1328,1334,1336,1342,1344,1349,1351,1357,1359,1365],{"type":46,"value":1327},"The threshold is a fraction of each task's score scale. Most AA tasks report\n0-100, but some (e.g. ",{"type":40,"tag":148,"props":1329,"children":1331},{"className":1330},[],[1332],{"type":46,"value":1333},"tau2_bench_telecom",{"type":46,"value":1335}," ",{"type":40,"tag":148,"props":1337,"children":1339},{"className":1338},[],[1340],{"type":46,"value":1341},"Result",{"type":46,"value":1343},") report 0-1; the gate infers\neach task's scale (0-1 if both scores are within ",{"type":40,"tag":386,"props":1345,"children":1346},{},[1347],{"type":46,"value":1348},"0, 1",{"type":46,"value":1350},", else 0-100) and\nnormalizes the drop accordingly, so ",{"type":40,"tag":148,"props":1352,"children":1354},{"className":1353},[],[1355],{"type":46,"value":1356},"--threshold 0.01",{"type":46,"value":1358}," means \"≤1 pt on a 0-100\ntask \u002F ≤0.01 on a 0-1 task\" uniformly. Pass ",{"type":40,"tag":148,"props":1360,"children":1362},{"className":1361},[],[1363],{"type":46,"value":1364},"--scales '{\"task\": max}'",{"type":46,"value":1366}," to\noverride inference if a task's scores happen to fall in an ambiguous range.",{"type":40,"tag":49,"props":1368,"children":1369},{},[1370,1376],{"type":40,"tag":148,"props":1371,"children":1373},{"className":1372},[],[1374],{"type":46,"value":1375},"gate_compare.py",{"type":46,"value":1377}," checks only the candidate delta; it cannot override a failed\nexternal baseline check. Combined decision:",{"type":40,"tag":123,"props":1379,"children":1380},{},[1381,1398,1414],{"type":40,"tag":127,"props":1382,"children":1383},{},[1384,1389,1391,1396],{"type":40,"tag":55,"props":1385,"children":1386},{},[1387],{"type":46,"value":1388},"ACCEPT (accuracy only)",{"type":46,"value":1390}," — no external check failed and every task is within\nthe candidate threshold → continue to ",{"type":40,"tag":55,"props":1392,"children":1393},{},[1394],{"type":46,"value":1395},"Step 5b",{"type":46,"value":1397},". This verdict covers accuracy\nalone; advance to Step 6 only once the verbosity gate also passes. A missing\ncomparable external score is not a failure; report it as externally unverified.",{"type":40,"tag":127,"props":1399,"children":1400},{},[1401,1405,1407,1412],{"type":40,"tag":55,"props":1402,"children":1403},{},[1404],{"type":46,"value":1107},{"type":46,"value":1406}," — one or more tasks exceed threshold. ",{"type":40,"tag":55,"props":1408,"children":1409},{},[1410],{"type":46,"value":1411},"v1 stops here and\nreports",{"type":46,"value":1413}," which tasks regressed by how much. (Picking the next recipe and\nre-running is deferred — see Scope.)",{"type":40,"tag":127,"props":1415,"children":1416},{},[1417,1421],{"type":40,"tag":55,"props":1418,"children":1419},{},[1420],{"type":46,"value":1012},{"type":46,"value":1422}," — external baseline sanity failed, or scores are otherwise\nimplausible (e.g. baseline lower than candidate by a large margin, or a task\nscore is outside its valid range) → correct the baseline or surface it.",{"type":40,"tag":525,"props":1424,"children":1426},{"id":1425},"step-5b-verbosity-gate-mandatory-independent-of-accuracy",[1427],{"type":46,"value":1428},"Step 5b — Verbosity gate (MANDATORY; independent of accuracy)",{"type":40,"tag":49,"props":1430,"children":1431},{},[1432,1437],{"type":40,"tag":148,"props":1433,"children":1435},{"className":1434},[],[1436],{"type":46,"value":1375},{"type":46,"value":1438}," does not measure verbosity, so stopping after Step 5 leaves a\nhard gate unmeasured. Step 5's ACCEPT is accuracy-only — Step 6 requires both.",{"type":40,"tag":232,"props":1440,"children":1442},{"className":378,"code":1441,"language":380,"meta":237,"style":237},"python \"$SKILL_DIR\u002Fscripts\u002Fgate_verbosity.py\" \\\n    --baseline \u003Cbaseline_eval_root> --candidate \u003Ccandidate_eval_root> \\\n    --glob 'eval_*' --exclude _high --threshold 0.05\n",[1443],{"type":40,"tag":148,"props":1444,"children":1445},{"__ignoreMap":237},[1446,1474,1524],{"type":40,"tag":386,"props":1447,"children":1448},{"class":388,"line":389},[1449,1453,1457,1461,1466,1470],{"type":40,"tag":386,"props":1450,"children":1451},{"style":421},[1452],{"type":46,"value":424},{"type":40,"tag":386,"props":1454,"children":1455},{"style":427},[1456],{"type":46,"value":430},{"type":40,"tag":386,"props":1458,"children":1459},{"style":433},[1460],{"type":46,"value":436},{"type":40,"tag":386,"props":1462,"children":1463},{"style":439},[1464],{"type":46,"value":1465},"\u002Fscripts\u002Fgate_verbosity.py",{"type":40,"tag":386,"props":1467,"children":1468},{"style":427},[1469],{"type":46,"value":447},{"type":40,"tag":386,"props":1471,"children":1472},{"style":433},[1473],{"type":46,"value":846},{"type":40,"tag":386,"props":1475,"children":1476},{"class":388,"line":399},[1477,1481,1485,1490,1495,1499,1503,1507,1512,1516,1520],{"type":40,"tag":386,"props":1478,"children":1479},{"style":439},[1480],{"type":46,"value":1264},{"type":40,"tag":386,"props":1482,"children":1483},{"style":427},[1484],{"type":46,"value":457},{"type":40,"tag":386,"props":1486,"children":1487},{"style":439},[1488],{"type":46,"value":1489},"baseline_eval_roo",{"type":40,"tag":386,"props":1491,"children":1492},{"style":433},[1493],{"type":46,"value":1494},"t",{"type":40,"tag":386,"props":1496,"children":1497},{"style":427},[1498],{"type":46,"value":1282},{"type":40,"tag":386,"props":1500,"children":1501},{"style":439},[1502],{"type":46,"value":1287},{"type":40,"tag":386,"props":1504,"children":1505},{"style":427},[1506],{"type":46,"value":457},{"type":40,"tag":386,"props":1508,"children":1509},{"style":439},[1510],{"type":46,"value":1511},"candidate_eval_roo",{"type":40,"tag":386,"props":1513,"children":1514},{"style":433},[1515],{"type":46,"value":1494},{"type":40,"tag":386,"props":1517,"children":1518},{"style":427},[1519],{"type":46,"value":1282},{"type":40,"tag":386,"props":1521,"children":1522},{"style":433},[1523],{"type":46,"value":846},{"type":40,"tag":386,"props":1525,"children":1526},{"class":388,"line":408},[1527,1532,1537,1542,1546,1551,1556,1561],{"type":40,"tag":386,"props":1528,"children":1529},{"style":439},[1530],{"type":46,"value":1531},"    --glob",{"type":40,"tag":386,"props":1533,"children":1534},{"style":427},[1535],{"type":46,"value":1536}," '",{"type":40,"tag":386,"props":1538,"children":1539},{"style":439},[1540],{"type":46,"value":1541},"eval_*",{"type":40,"tag":386,"props":1543,"children":1544},{"style":427},[1545],{"type":46,"value":822},{"type":40,"tag":386,"props":1547,"children":1548},{"style":439},[1549],{"type":46,"value":1550}," --exclude",{"type":40,"tag":386,"props":1552,"children":1553},{"style":439},[1554],{"type":46,"value":1555}," _high",{"type":40,"tag":386,"props":1557,"children":1558},{"style":439},[1559],{"type":46,"value":1560}," --threshold",{"type":40,"tag":386,"props":1562,"children":1563},{"style":1319},[1564],{"type":46,"value":1565}," 0.05\n",{"type":40,"tag":49,"props":1567,"children":1568},{},[1569,1571,1577,1579,1584,1586,1592,1594,1599,1601,1607,1609,1615,1616,1622,1624,1630,1632,1638,1640,1646,1648,1654,1656,1662,1664,1670,1672,1678],{"type":46,"value":1570},"Exit codes follow the other gates: ",{"type":40,"tag":148,"props":1572,"children":1574},{"className":1573},[],[1575],{"type":46,"value":1576},"0",{"type":46,"value":1578}," pass, ",{"type":40,"tag":148,"props":1580,"children":1582},{"className":1581},[],[1583],{"type":46,"value":1097},{"type":46,"value":1585}," the gate failed, ",{"type":40,"tag":148,"props":1587,"children":1589},{"className":1588},[],[1590],{"type":46,"value":1591},"2",{"type":46,"value":1593}," it could not read its input.\nA ",{"type":40,"tag":148,"props":1595,"children":1597},{"className":1596},[],[1598],{"type":46,"value":1591},{"type":46,"value":1600}," means fix the invocation, not the checkpoint — check ",{"type":40,"tag":148,"props":1602,"children":1604},{"className":1603},[],[1605],{"type":46,"value":1606},"--baseline",{"type":46,"value":1608},"\u002F",{"type":40,"tag":148,"props":1610,"children":1612},{"className":1611},[],[1613],{"type":46,"value":1614},"--candidate",{"type":46,"value":155},{"type":40,"tag":148,"props":1617,"children":1619},{"className":1618},[],[1620],{"type":46,"value":1621},"--glob",{"type":46,"value":1623},"\n(",{"type":40,"tag":148,"props":1625,"children":1627},{"className":1626},[],[1628],{"type":46,"value":1629},"no_files_matched",{"type":46,"value":1631}," names the pattern that missed), and ",{"type":40,"tag":148,"props":1633,"children":1635},{"className":1634},[],[1636],{"type":46,"value":1637},"--exclude",{"type":46,"value":1639},". If the detail names\n",{"type":40,"tag":148,"props":1641,"children":1643},{"className":1642},[],[1644],{"type":46,"value":1645},"collapsed_keys",{"type":46,"value":1647},", the artifact tree gave several tasks one name: point the gate at a tree whose\ntask dirs are ",{"type":40,"tag":148,"props":1649,"children":1651},{"className":1650},[],[1652],{"type":46,"value":1653},"\u003Charness>.\u003Ctask>",{"type":46,"value":1655}," rather than ",{"type":40,"tag":148,"props":1657,"children":1659},{"className":1658},[],[1660],{"type":46,"value":1661},"\u003Cinvocation_id>.\u003Cjob_index>",{"type":46,"value":1663},", or re-sync so each\n",{"type":40,"tag":148,"props":1665,"children":1667},{"className":1666},[],[1668],{"type":46,"value":1669},"artifacts\u002F",{"type":46,"value":1671}," carries its own ",{"type":40,"tag":148,"props":1673,"children":1675},{"className":1674},[],[1676],{"type":46,"value":1677},"config.yml",{"type":46,"value":110},{"type":40,"tag":49,"props":1680,"children":1681},{},[1682,1684,1690,1692,1698,1700,1704,1706,1712,1714,1720,1722,1727,1729,1735],{"type":46,"value":1683},"Read ",{"type":40,"tag":148,"props":1685,"children":1687},{"className":1686},[],[1688],{"type":46,"value":1689},"response_stats.avg_completion_tokens",{"type":46,"value":1691}," from each task's\n",{"type":40,"tag":148,"props":1693,"children":1695},{"className":1694},[],[1696],{"type":46,"value":1697},"artifacts\u002Feval_factory_metrics.json",{"type":46,"value":1699},". Do ",{"type":40,"tag":55,"props":1701,"children":1702},{},[1703],{"type":46,"value":988},{"type":46,"value":1705}," use the ",{"type":40,"tag":148,"props":1707,"children":1709},{"className":1708},[],[1710],{"type":46,"value":1711},"reasoning.*",{"type":46,"value":1713}," fields:\n",{"type":40,"tag":148,"props":1715,"children":1717},{"className":1716},[],[1718],{"type":46,"value":1719},"reasoning.*_tokens",{"type":46,"value":1721}," are always ",{"type":40,"tag":148,"props":1723,"children":1725},{"className":1724},[],[1726],{"type":46,"value":1576},{"type":46,"value":1728}," (only the reasoning\u002Fcontent split is missing)\nand the ",{"type":40,"tag":148,"props":1730,"children":1732},{"className":1731},[],[1733],{"type":46,"value":1734},"*_words",{"type":46,"value":1736}," siblings are a proxy that can disagree with the gate — one task\nread +6.10% FAIL in words and +1.32% PASS in tokens.",{"type":40,"tag":49,"props":1738,"children":1739},{},[1740],{"type":46,"value":1741},"The gate is two-sided; a large drop in output length is also a change.",{"type":40,"tag":49,"props":1743,"children":1744},{},[1745,1747,1752,1754,1760,1762,1767,1769,1775,1777,1783],{"type":46,"value":1746},"Two filters are mandatory: same reasoning effort (pass ",{"type":40,"tag":148,"props":1748,"children":1750},{"className":1749},[],[1751],{"type":46,"value":1637},{"type":46,"value":1753}," explicitly — it is empty by\ndefault, since which tier is canonical is run-specific) and complete runs only —\nruns within 1% of the largest ",{"type":40,"tag":148,"props":1755,"children":1757},{"className":1756},[],[1758],{"type":46,"value":1759},"successful_count",{"type":46,"value":1761}," that ",{"type":40,"tag":55,"props":1763,"children":1764},{},[1765],{"type":46,"value":1766},"both",{"type":46,"value":1768}," sides can match (exact equality\nwould make one dropped sample unmeasurable). A task with no such count on both sides is\n",{"type":40,"tag":148,"props":1770,"children":1772},{"className":1771},[],[1773],{"type":46,"value":1774},"not_comparable",{"type":46,"value":1776},". When the matched count is below a run\none side has, the task carries ",{"type":40,"tag":148,"props":1778,"children":1780},{"className":1779},[],[1781],{"type":46,"value":1782},"truncated_comparison",{"type":46,"value":110},{"type":40,"tag":49,"props":1785,"children":1786},{},[1787,1789,1794,1796,1802],{"type":46,"value":1788},"Tasks sharing no sample count are ",{"type":40,"tag":148,"props":1790,"children":1792},{"className":1791},[],[1793],{"type":46,"value":1774},{"type":46,"value":1795},", not a delta. Means under\n~1000 tokens carry a ",{"type":40,"tag":148,"props":1797,"children":1799},{"className":1798},[],[1800],{"type":46,"value":1801},"short_output_warning",{"type":46,"value":1803}," — read absolute counts there.",{"type":40,"tag":324,"props":1805,"children":1807},{"id":1806},"step-6-closeout",[1808],{"type":46,"value":1809},"Step 6 — Closeout",{"type":40,"tag":49,"props":1811,"children":1812},{},[1813,1815,1820],{"type":46,"value":1814},"Report the decision with: source vs output size + ratio, per-task baseline \u002F\ncandidate \u002F delta \u002F within-threshold, ",{"type":40,"tag":55,"props":1816,"children":1817},{},[1818],{"type":46,"value":1819},"the verbosity verdict from Step 5b",{"type":46,"value":1821},",\nexternal source and sanity status, MLflow run IDs, and a publish recommendation\n(publish \u002F do-not-publish). Archive artifacts to the workspace.",{"type":40,"tag":49,"props":1823,"children":1824},{},[1825,1830,1832,1838],{"type":40,"tag":55,"props":1826,"children":1827},{},[1828],{"type":46,"value":1829},"Publish exactly what was evaluated.",{"type":46,"value":1831}," Verify mechanically, not by path\nconvention: inode-compare a shard in the evaluated directory against the one being\npublished. A day-0 run leaves near-identical sibling exports\nthat differ by one calibration suffix — prefix rejected ones ",{"type":40,"tag":148,"props":1833,"children":1835},{"className":1834},[],[1836],{"type":46,"value":1837},"REJECTED-",{"type":46,"value":1839}," so the\nartifact cannot be picked by autocomplete, and re-run the Step 2b canary against\nthe final path after any move.",{"type":40,"tag":73,"props":1841,"children":1843},{"id":1842},"triage-gate-failure-decision",[1844],{"type":46,"value":1845},"Triage (gate failure → decision)",{"type":40,"tag":49,"props":1847,"children":1848},{},[1849,1851,1856],{"type":46,"value":1850},"Map a gate's ",{"type":40,"tag":148,"props":1852,"children":1854},{"className":1853},[],[1855],{"type":46,"value":514},{"type":46,"value":1857}," to the next action:",{"type":40,"tag":1048,"props":1859,"children":1860},{},[1861,1880],{"type":40,"tag":1052,"props":1862,"children":1863},{},[1864],{"type":40,"tag":1056,"props":1865,"children":1866},{},[1867,1875],{"type":40,"tag":1060,"props":1868,"children":1869},{},[1870],{"type":40,"tag":148,"props":1871,"children":1873},{"className":1872},[],[1874],{"type":46,"value":514},{"type":40,"tag":1060,"props":1876,"children":1877},{},[1878],{"type":46,"value":1879},"Action",{"type":40,"tag":1081,"props":1881,"children":1882},{},[1883,1906,1929,1946,1993,2010,2027,2069,2086,2103,2119,2143],{"type":40,"tag":1056,"props":1884,"children":1885},{},[1886,1895],{"type":40,"tag":1088,"props":1887,"children":1888},{},[1889],{"type":40,"tag":148,"props":1890,"children":1892},{"className":1891},[],[1893],{"type":46,"value":1894},"INFRA_TRANSIENT",{"type":40,"tag":1088,"props":1896,"children":1897},{},[1898,1900,1905],{"type":46,"value":1899},"Retry the stage once; if it recurs, ",{"type":40,"tag":148,"props":1901,"children":1903},{"className":1902},[],[1904],{"type":46,"value":356},{"type":46,"value":110},{"type":40,"tag":1056,"props":1907,"children":1908},{},[1909,1918],{"type":40,"tag":1088,"props":1910,"children":1911},{},[1912],{"type":40,"tag":148,"props":1913,"children":1915},{"className":1914},[],[1916],{"type":46,"value":1917},"MODEL_UNSUPPORTED",{"type":40,"tag":1088,"props":1919,"children":1920},{},[1921,1923,1928],{"type":46,"value":1922},"PATCH: fix the recipe pattern \u002F add model support (ptq skill owns the patch loop), then retry. If unpatchable, ",{"type":40,"tag":148,"props":1924,"children":1926},{"className":1925},[],[1927],{"type":46,"value":776},{"type":46,"value":110},{"type":40,"tag":1056,"props":1930,"children":1931},{},[1932,1941],{"type":40,"tag":1088,"props":1933,"children":1934},{},[1935],{"type":40,"tag":148,"props":1936,"children":1938},{"className":1937},[],[1939],{"type":46,"value":1940},"QUANT_COVERAGE_FAILURE",{"type":40,"tag":1088,"props":1942,"children":1943},{},[1944],{"type":46,"value":1945},"PATCH: fix the recipe wildcard so intended layers are covered; re-run PTQ.",{"type":40,"tag":1056,"props":1947,"children":1948},{},[1949,1958],{"type":40,"tag":1088,"props":1950,"children":1951},{},[1952],{"type":40,"tag":148,"props":1953,"children":1955},{"className":1954},[],[1956],{"type":46,"value":1957},"SIZE_NOT_REDUCED",{"type":40,"tag":1088,"props":1959,"children":1960},{},[1961,1963,1969,1971,1976,1978,1984,1986,1991],{"type":46,"value":1962},"The output is not smaller than the source. If the recipe cannot shrink the source (e.g. mxfp4 under nvfp4, or fp8 under fp8), record ",{"type":40,"tag":148,"props":1964,"children":1966},{"className":1965},[],[1967],{"type":46,"value":1968},"source_precision",{"type":46,"value":1970}," in the validation summary and re-run the gate — that states ",{"type":40,"tag":255,"props":1972,"children":1973},{},[1974],{"type":46,"value":1975},"why",{"type":46,"value":1977}," the growth is expected. Otherwise treat it as a real compression failure: check that the recipe matched the intended parameter mass and that the exporter did not retain the original tensors. ",{"type":40,"tag":148,"props":1979,"children":1981},{"className":1980},[],[1982],{"type":46,"value":1983},"accept_size_growth: true",{"type":46,"value":1985}," waives it unconditionally (no growth bound) and is a last resort, not the fix — it records no reason, so prefer ",{"type":40,"tag":148,"props":1987,"children":1989},{"className":1988},[],[1990],{"type":46,"value":1968},{"type":46,"value":1992}," where the claim is checkable.",{"type":40,"tag":1056,"props":1994,"children":1995},{},[1996,2005],{"type":40,"tag":1088,"props":1997,"children":1998},{},[1999],{"type":40,"tag":148,"props":2000,"children":2002},{"className":2001},[],[2003],{"type":46,"value":2004},"CHECKPOINT_NOT_SERVABLE",{"type":40,"tag":1088,"props":2006,"children":2007},{},[2008],{"type":46,"value":2009},"The Step 2b canary could not load\u002Fgenerate. Usually a tensor-naming or config-schema mismatch between the exporter and the serving stack, or missing\u002Fdangling auxiliary files (tokenizer). Fix the export; do not evaluate.",{"type":40,"tag":1056,"props":2011,"children":2012},{},[2013,2022],{"type":40,"tag":1088,"props":2014,"children":2015},{},[2016],{"type":40,"tag":148,"props":2017,"children":2019},{"className":2018},[],[2020],{"type":46,"value":2021},"VERBOSITY_EXCEEDED",{"type":40,"tag":1088,"props":2023,"children":2024},{},[2025],{"type":46,"value":2026},"Re-check run hygiene first (mixed reasoning effort, partial runs, unequal sample counts) — that has explained every false positive so far. If the delta survives matched, complete runs, it is a real behavioural change; do not publish on accuracy alone.",{"type":40,"tag":1056,"props":2028,"children":2029},{},[2030,2038],{"type":40,"tag":1088,"props":2031,"children":2032},{},[2033],{"type":40,"tag":148,"props":2034,"children":2036},{"className":2035},[],[2037],{"type":46,"value":267},{"type":40,"tag":1088,"props":2039,"children":2040},{},[2041,2043,2047,2049,2054,2056,2061,2063,2068],{"type":46,"value":2042},"Drop to the ",{"type":40,"tag":55,"props":2044,"children":2045},{},[2046],{"type":46,"value":18},{"type":46,"value":2048}," skill: reproduce serving standalone (",{"type":40,"tag":148,"props":2050,"children":2052},{"className":2051},[],[2053],{"type":46,"value":761},{"type":46,"value":2055}," + one generation), debug flags \u002F image \u002F TP \u002F env, then carry the working command into NEL's ",{"type":40,"tag":148,"props":2057,"children":2059},{"className":2058},[],[2060],{"type":46,"value":724},{"type":46,"value":2062}," and retry the eval. If it can't serve, ",{"type":40,"tag":148,"props":2064,"children":2066},{"className":2065},[],[2067],{"type":46,"value":776},{"type":46,"value":110},{"type":40,"tag":1056,"props":2070,"children":2071},{},[2072,2081],{"type":40,"tag":1088,"props":2073,"children":2074},{},[2075],{"type":40,"tag":148,"props":2076,"children":2078},{"className":2077},[],[2079],{"type":46,"value":2080},"EVAL_JUDGE_FAILED",{"type":40,"tag":1088,"props":2082,"children":2083},{},[2084],{"type":46,"value":2085},"Usually transient (auth \u002F rate limit) — wait and retry.",{"type":40,"tag":1056,"props":2087,"children":2088},{},[2089,2098],{"type":40,"tag":1088,"props":2090,"children":2091},{},[2092],{"type":40,"tag":148,"props":2093,"children":2095},{"className":2094},[],[2096],{"type":46,"value":2097},"SAMPLE_ACCOUNTING_FAILED",{"type":40,"tag":1088,"props":2099,"children":2100},{},[2101],{"type":46,"value":2102},"Investigate dropped\u002Ffailed samples before trusting scores.",{"type":40,"tag":1056,"props":2104,"children":2105},{},[2106,2114],{"type":40,"tag":1088,"props":2107,"children":2108},{},[2109],{"type":40,"tag":148,"props":2110,"children":2112},{"className":2111},[],[2113],{"type":46,"value":1020},{"type":40,"tag":1088,"props":2115,"children":2116},{},[2117],{"type":46,"value":2118},"Investigate baseline configuration, correct it, rerun the baseline, and repeat external sanity before comparison.",{"type":40,"tag":1056,"props":2120,"children":2121},{},[2122,2131],{"type":40,"tag":1088,"props":2123,"children":2124},{},[2125],{"type":40,"tag":148,"props":2126,"children":2128},{"className":2127},[],[2129],{"type":46,"value":2130},"USER_CONFIG_ERROR",{"type":40,"tag":1088,"props":2132,"children":2133},{},[2134,2136,2141],{"type":46,"value":2135},"Correct it from the request, workspace, or model\u002Fconfig metadata and retry; if irrecoverable, return ",{"type":40,"tag":148,"props":2137,"children":2139},{"className":2138},[],[2140],{"type":46,"value":1012},{"type":46,"value":2142}," with evidence.",{"type":40,"tag":1056,"props":2144,"children":2145},{},[2146,2155],{"type":40,"tag":1088,"props":2147,"children":2148},{},[2149],{"type":40,"tag":148,"props":2150,"children":2152},{"className":2151},[],[2153],{"type":46,"value":2154},"UNKNOWN",{"type":40,"tag":1088,"props":2156,"children":2157},{},[2158,2160,2165],{"type":46,"value":2159},"Investigate with the owning domain skill; if unresolved, return ",{"type":40,"tag":148,"props":2161,"children":2163},{"className":2162},[],[2164],{"type":46,"value":1012},{"type":46,"value":2166}," with the evidence and next automated retry or patch action.",{"type":40,"tag":49,"props":2168,"children":2169},{},[2170,2175,2177,2183,2185,2190,2192,2197,2199,2204],{"type":40,"tag":148,"props":2171,"children":2173},{"className":2172},[],[2174],{"type":46,"value":490},{"type":46,"value":2176}," also emits non-blocking ",{"type":40,"tag":148,"props":2178,"children":2180},{"className":2179},[],[2181],{"type":46,"value":2182},"notes",{"type":46,"value":2184}," (present on every result). Size growth is\n",{"type":40,"tag":55,"props":2186,"children":2187},{},[2188],{"type":46,"value":2189},"blocking by default",{"type":46,"value":2191},", waived to a note only when the validation summary's ",{"type":40,"tag":148,"props":2193,"children":2195},{"className":2194},[],[2196],{"type":46,"value":1968},{"type":46,"value":2198},"\nis already at or below the recipe's target bits and the growth is within what that explains. Recording\n",{"type":40,"tag":148,"props":2200,"children":2202},{"className":2201},[],[2203],{"type":46,"value":1968},{"type":46,"value":2205}," is part of the ptq skill's validation table, so the waiver is reachable from the\nnormal pipeline. A BF16 source that failed to compress still fails, which is the case this check\nexists for.",{"type":40,"tag":49,"props":2207,"children":2208},{},[2209,2214,2216,2221],{"type":40,"tag":148,"props":2210,"children":2212},{"className":2211},[],[2213],{"type":46,"value":356},{"type":46,"value":2215}," (cluster down, dataset unavailable) aborts the whole run.\n",{"type":40,"tag":148,"props":2217,"children":2219},{"className":2218},[],[2220],{"type":46,"value":776},{"type":46,"value":2222}," means this (model, recipe) can't work as configured.",{"type":40,"tag":73,"props":2224,"children":2226},{"id":2225},"output",[2227],{"type":46,"value":2228},"Output",{"type":40,"tag":49,"props":2230,"children":2231},{},[2232],{"type":46,"value":2233},"Return a decision, not a raw artifact:",{"type":40,"tag":123,"props":2235,"children":2236},{},[2237,2248,2258,2276],{"type":40,"tag":127,"props":2238,"children":2239},{},[2240,2246],{"type":40,"tag":148,"props":2241,"children":2243},{"className":2242},[],[2244],{"type":46,"value":2245},"ACCEPT",{"type":46,"value":2247}," + report + publish recommendation",{"type":40,"tag":127,"props":2249,"children":2250},{},[2251,2256],{"type":40,"tag":148,"props":2252,"children":2254},{"className":2253},[],[2255],{"type":46,"value":1107},{"type":46,"value":2257}," + which tasks failed the threshold and by how much",{"type":40,"tag":127,"props":2259,"children":2260},{},[2261,2266,2268,2274],{"type":40,"tag":148,"props":2262,"children":2264},{"className":2263},[],[2265],{"type":46,"value":1012},{"type":46,"value":2267}," \u002F ",{"type":40,"tag":148,"props":2269,"children":2271},{"className":2270},[],[2272],{"type":46,"value":2273},"INFEASIBLE",{"type":46,"value":2275}," + reason and next automated action",{"type":40,"tag":127,"props":2277,"children":2278},{},[2279],{"type":46,"value":2280},"Always: workspace path + MLflow run IDs for traceability",{"type":40,"tag":73,"props":2282,"children":2284},{"id":2283},"scope-v1",[2285],{"type":46,"value":2286},"Scope (v1)",{"type":40,"tag":49,"props":2288,"children":2289},{},[2290,2292,2297],{"type":46,"value":2291},"In v1: the linear chain + gates + report. On ",{"type":40,"tag":148,"props":2293,"children":2295},{"className":2294},[],[2296],{"type":46,"value":1107},{"type":46,"value":2298},", v1 reports and stops.\nDeferred to a follow-up: the evaluator-optimizer recipe loop (compare → pick the\nnext recipe → re-run PTQ), which needs the bigpareto integration and a shared\nconfig\u002Fresult schema.",{"type":40,"tag":2300,"props":2301,"children":2302},"style",{},[2303],{"type":46,"value":2304},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2306,"total":2457},[2307,2325,2340,2349,2361,2374,2387,2401,2414,2425,2439,2448],{"slug":2308,"name":2308,"fn":2309,"description":2310,"org":2311,"tags":2312,"stars":2322,"repoUrl":2323,"updatedAt":2324},"nemoclaw-user-guide","retrieve NemoClaw documentation and configuration","Guides human users' AI agents to the NemoClaw docs MCP server and canonical Fern documentation in Markdown form. Use when users ask how to install, configure, operate, troubleshoot, secure, or learn NemoClaw with an AI coding assistant. Trigger keywords - nemoclaw docs, use nemoclaw with ai agent, nemoclaw mcp docs, nemoclaw install help, nemoclaw quickstart, nemoclaw markdown docs, llms.txt, agent skills.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2313,2316,2319],{"name":2314,"slug":2315,"type":15},"Documentation","documentation",{"name":2317,"slug":2318,"type":15},"MCP","mcp",{"name":2320,"slug":2321,"type":15},"Search","search",21777,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FNemoClaw","2026-08-25T03:29:57.273192",{"slug":2326,"name":2326,"fn":2327,"description":2328,"org":2329,"tags":2330,"stars":2337,"repoUrl":2338,"updatedAt":2339},"mcore-build-and-dependency","manage Megatron-LM development environments","Container-based dev environment setup and dependency management for Megatron-LM. Covers acquiring and launching the CI container, uv package management, and updating uv.lock.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2331,2334,2335],{"name":2332,"slug":2333,"type":15},"Containers","containers",{"name":17,"slug":18,"type":15},{"name":2336,"slug":424,"type":15},"Python",17049,"https:\u002F\u002Fgithub.com\u002FNVIDIA\u002FMegatron-LM","2026-07-27T06:06:11.249662",{"slug":2341,"name":2341,"fn":2342,"description":2343,"org":2344,"tags":2345,"stars":2337,"repoUrl":2338,"updatedAt":2348},"mcore-bump-base-image","update NVIDIA PyTorch base images","Bump the NVIDIA PyTorch base image (`nvcr.io\u002Fnvidia\u002Fpytorch:YY.MM-py3`) used by Megatron-LM CI. Covers the two pin sites (GitHub CI in `docker\u002F.ngc_version.dev` and GitLab CI in `.gitlab\u002Fstages\u002F01.build.yml`), the post-bump CI loop (re-run functional tests, refresh golden values, mark broken tests), and the gotchas that bit PRs",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2346,2347],{"name":21,"slug":22,"type":15},{"name":17,"slug":18,"type":15},"2026-07-14T05:25:59.97109",{"slug":2350,"name":2350,"fn":2351,"description":2352,"org":2353,"tags":2354,"stars":2337,"repoUrl":2338,"updatedAt":2360},"mcore-cicd","manage CI\u002FCD pipelines for Megatron-LM","CI\u002FCD reference for Megatron-LM. Covers CI pipeline structure, PR scope labels, triggering internal GitLab CI (which force-pushes the current branch to a pull-request\u002FBRANCH ref — always dry-run and verify the destination first; never run against shared or protected branches), and CI failure investigation.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2355,2356,2357],{"name":21,"slug":22,"type":15},{"name":17,"slug":18,"type":15},{"name":2358,"slug":2359,"type":15},"GitHub","github","2026-08-28T14:38:16.959248",{"slug":2362,"name":2362,"fn":2363,"description":2364,"org":2365,"tags":2366,"stars":2337,"repoUrl":2338,"updatedAt":2373},"mcore-create-issue","investigate CI failures and create issues","Investigate a failing GitHub Actions run or job and create a GitHub issue for the failure.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2367,2370,2371],{"name":2368,"slug":2369,"type":15},"Debugging","debugging",{"name":2358,"slug":2359,"type":15},{"name":521,"slug":2372,"type":15},"triage","2026-07-14T05:25:57.442089",{"slug":2375,"name":2375,"fn":2376,"description":2377,"org":2378,"tags":2379,"stars":2337,"repoUrl":2338,"updatedAt":2386},"mcore-linting-and-formatting","lint and format Megatron-LM code","Linting and formatting for Megatron-LM. Covers running autoformat.sh, tools (ruff, black, isort, pylint, mypy), and code style rules.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2380,2383],{"name":2381,"slug":2382,"type":15},"Best Practices","best-practices",{"name":2384,"slug":2385,"type":15},"Code Analysis","code-analysis","2026-07-14T05:25:56.18433",{"slug":2388,"name":2388,"fn":2389,"description":2390,"org":2391,"tags":2392,"stars":2337,"repoUrl":2338,"updatedAt":2400},"mcore-migrate-gpt-to-hybrid","migrate Megatron-LM models to HybridModel","Migration guide for moving Megatron Core GPTModel checkpoints, model providers, training commands, and layer mappings to HybridModel, including the mechanical steps for transferring an existing pretrain_gpt.py launch script.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2393,2396,2399],{"name":2394,"slug":2395,"type":15},"Machine Learning","machine-learning",{"name":2397,"slug":2398,"type":15},"Migration","migration",{"name":9,"slug":8,"type":15},"2026-08-21T03:36:57.470256",{"slug":2402,"name":2402,"fn":2403,"description":2404,"org":2405,"tags":2406,"stars":2337,"repoUrl":2338,"updatedAt":2413},"mcore-onboard-gb200-1node-tests","onboard functional tests for GB200","Onboard 1-node GitHub MR functional tests for GB200 from existing mr-scoped 2-node tests.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2407,2410],{"name":2408,"slug":2409,"type":15},"QA","qa",{"name":2411,"slug":2412,"type":15},"Testing","testing","2026-07-14T05:25:53.673039",{"slug":2415,"name":2415,"fn":2416,"description":2417,"org":2418,"tags":2419,"stars":2337,"repoUrl":2338,"updatedAt":2424},"mcore-run-on-slurm","launch distributed training jobs on SLURM","How to launch distributed Megatron-LM training jobs on a SLURM cluster. Covers a minimal sbatch skeleton, environment-variable setup for torch.distributed.run, CUDA_DEVICE_MAX_CONNECTIONS rules across hardware and parallelism modes, container conventions, monitoring, and per-rank failure diagnosis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2420,2421],{"name":17,"slug":18,"type":15},{"name":2422,"slug":2423,"type":15},"Infrastructure","infrastructure","2026-07-14T05:25:49.362534",{"slug":2426,"name":2426,"fn":2427,"description":2428,"org":2429,"tags":2430,"stars":2337,"repoUrl":2338,"updatedAt":2438},"mcore-split-pr","split pull requests to reduce review load","Split a PR into multiple PRs to reduce the number of required CODEOWNERS reviewer groups.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2431,2434,2435],{"name":2432,"slug":2433,"type":15},"Code Review","code-review",{"name":2358,"slug":2359,"type":15},{"name":2436,"slug":2437,"type":15},"Pull Requests","pull-requests","2026-08-25T03:29:16.211287",{"slug":2440,"name":2440,"fn":2441,"description":2442,"org":2443,"tags":2444,"stars":2337,"repoUrl":2338,"updatedAt":2447},"mcore-testing","run and manage Megatron-LM tests","Test system for Megatron-LM. Covers test layout, recipe YAML structure, adding and running unit and functional tests, golden values, marker filters, and CI parity.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2445,2446],{"name":2408,"slug":2409,"type":15},{"name":2411,"slug":2412,"type":15},"2026-07-14T05:25:54.928983",{"slug":2449,"name":2449,"fn":2450,"description":2451,"org":2452,"tags":2453,"stars":2337,"repoUrl":2338,"updatedAt":2456},"nightly-sync","manage nightly main-to-dev sync workflows","Domain knowledge for the nightly main-to-dev sync workflow. Covers merge strategy, CI architecture, failure investigation, and known issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2454,2455],{"name":13,"slug":14,"type":15},{"name":21,"slug":22,"type":15},"2026-07-30T05:29:03.275638",563,{"items":2459,"total":408},[2460,2467,2481],{"slug":4,"name":4,"fn":5,"description":6,"org":2461,"tags":2462,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2463,2464,2465,2466],{"name":13,"slug":14,"type":15},{"name":21,"slug":22,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"slug":18,"name":18,"fn":2468,"description":2469,"org":2470,"tags":2471,"stars":23,"repoUrl":24,"updatedAt":2480},"deploy LLM checkpoints as API endpoints","Serve a quantized or unquantized LLM checkpoint as an OpenAI-compatible API endpoint using vLLM, SGLang, or TRT-LLM. Use when user says \"deploy model\", \"serve model\", \"start vLLM server\", \"launch SGLang\", \"TRT-LLM deploy\", \"AutoDeploy\", \"benchmark throughput\", \"serve checkpoint\", or needs an inference endpoint from a HuggingFace or ModelOpt-quantized checkpoint. Do NOT use for quantizing models (use ptq) or evaluating accuracy (use evaluation).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2472,2473,2476,2477],{"name":17,"slug":18,"type":15},{"name":2474,"slug":2475,"type":15},"LLM","llm",{"name":9,"slug":8,"type":15},{"name":2478,"slug":2479,"type":15},"vLLM","vllm","2026-08-31T09:18:55.144088",{"slug":101,"name":101,"fn":2482,"description":2483,"org":2484,"tags":2485,"stars":23,"repoUrl":24,"updatedAt":2494},"evaluate LLM accuracy with NeMo","Evaluates accuracy of quantized or unquantized LLMs using NeMo Evaluator Launcher (NEL). Triggers on \"evaluate model\", \"benchmark accuracy\", \"run MMLU\", \"evaluate quantized model\", \"run nel\". Handles deployment, config generation, and evaluation execution. Not for quantizing models (use ptq), deploying\u002Fserving models (use deployment), or comparing completed baseline-vs-quantized results (use compare-results).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":9},[2486,2489,2492,2493],{"name":2487,"slug":2488,"type":15},"Benchmarking","benchmarking",{"name":2490,"slug":2491,"type":15},"Evals","evals",{"name":2474,"slug":2475,"type":15},{"name":9,"slug":8,"type":15},"2026-08-31T09:19:35.777526"]