[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-braintrust-braintrust-eval":3,"mdc--g5phzh-key":32,"related-org-braintrust-braintrust-eval":2085,"related-repo-braintrust-braintrust-eval":2259},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":21,"repoUrl":22,"updatedAt":23,"license":24,"forks":25,"topics":26,"repo":27,"sourceUrl":30,"mdContent":31},"braintrust-eval","execute Braintrust evaluation experiments","Execute an eval end to end against a live Braintrust project — org and credential selection, finding and importing a dataset, writing the task and scorer code, running the experiment behind smoke gates and quota preflight, and tracing agentic `claude -p` runs. Use when an eval has to actually run: \"run this eval,\" \"set up an eval for X,\" \"compare these models in Braintrust.\" Do not use to design a dataset, scorer, experiment, or release gate in the abstract — the lifecycle cards own what to build and why; this owns making it happen.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"braintrust","Braintrust","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fbraintrust.png","braintrustdata",[13,17,18],{"name":14,"slug":15,"type":16},"Evals","evals","tag",{"name":9,"slug":8,"type":16},{"name":19,"slug":20,"type":16},"Testing","testing",7,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library","2026-08-20T03:53:29.405249",null,0,[],{"repoUrl":22,"stars":21,"forks":25,"topics":28,"description":29},[],"Braintrust eval skills library","https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library\u002Ftree\u002FHEAD\u002Fskills\u002Fbraintrust-eval","---\nname: braintrust-eval\ndescription: >-\n  Execute an eval end to end against a live Braintrust project — org and credential\n  selection, finding and importing a dataset, writing the task and scorer code,\n  running the experiment behind smoke gates and quota preflight, and tracing agentic\n  `claude -p` runs. Use when an eval has to actually run: \"run this eval,\" \"set up an\n  eval for X,\" \"compare these models in Braintrust.\" Do not use to design a dataset,\n  scorer, experiment, or release gate in the abstract — the lifecycle cards own what\n  to build and why; this owns making it happen.\n---\n\n# Braintrust Eval\n\nThe **runbook**. Take the user from \"I want to eval X\" to a clean, comparable\nBraintrust experiment that actually ran. Deliberately interactive: ask, propose,\nget approval, then spend money.\n\nThis skill owns **execution** — credentials, imports, code, run hygiene, tracing,\nand the approval gates that keep a paid run from starting on an unreviewed plan.\nIt does not own methodology: each phase below names the lifecycle card that does,\nand those are worth loading when a judgment call is live rather than routine.\n\nShared platform mechanics — pinning, naming, safe reads, run hygiene — are in\n`references\u002Fplatform-mechanics.md`, not restated per phase.\n\n## When to use \u002F when not\n- **Use this** to run a new eval or model\u002Fprompt comparison against a real project.\n- **Getting data into Braintrust** (a HuggingFace \u002F CSV \u002F trace source into a BT\n  dataset)? Phase 3, via `references\u002Fdataset-import.md`.\n- **Pulling or plotting results** from experiments that already ran? Out of scope —\n  that is `braintrust-analyze-eval-experiment`.\n\nWork through the phases in order. Do not skip the approval gates — the whole point\nis that the user signs off on the dataset and the scorers before a paid run.\n\n---\n\n## Phase 1 — Preflight (before anything else)\n\n**1a. Ask which org.** Braintrust API keys are org-scoped, so this decides which\ncredential the run uses — and it's a data-governance decision.\n\n> \"Which org should this run in?\" — list the orgs the environment has keys for.\n> Skip the question entirely if there is only one.\n\nGovernance default, state it if relevant: *if the eval involves sensitive,\ncustomer, or proprietary data, default to the most restricted org available unless\nthe user says otherwise.*\n\n**1b. Select + verify the key.** Keys live in **one canonical file**, resolved as\n`$BRAINTRUST_ENV_FILE` → `~\u002F.braintrust\u002F.env` → the current project's `.env` (that\nlast one only if neither of the others exists). Each org's key is\n`BRAINTRUST_API_KEY_\u003CORG>`; a single-org setup can use a plain\n`BRAINTRUST_API_KEY`.\n\n**Do not hunt for keys** across sub-project `.env` files — that trips credential-\nscanning guards. If the chosen key is missing or empty: create\u002Fappend the stub\nline to the canonical `.env`, tell the user exactly which variable to fill in\n(key from Braintrust app → Settings → API Keys, per org), and wait. **Never ask\nthe user to paste a key into the chat, and never print one.** Verify the filled\nkey by listing the org's **projects** via the API (not `\u002Fv1\u002Forganization` — it can\nreturn empty for member keys). (See `references\u002Fbt-auth.md`.)\n\n**1c. Ask which project**, then confirm it exists in that org via the API. If it\ndoesn't exist, offer to create it.\n\n## Phase 2 — Intake\n\nThree open prompts (answer in prose), plus one quick config pick:\n\n1. **What do you want to eval?** (They'll usually name the model and\u002For prompt\n   variant here — that's the thing under test.)\n2. **Describe what the dataset should be.** (~3 sentences: the inputs, what a good\n   answer looks like, any domains\u002Fedge cases to cover.)\n3. **Describe what scores you'd like to see.** (What does \"good\" mean here? What\n   failures matter most?)\n4. **Which language should the eval be written in?** Braintrust ships full SDKs —\n   tracing, the `Eval()` framework, and scorers — in **Python, TypeScript, Go,\n   Java, Ruby, and C#**. Surface all six and **default to Python** unless the user\n   picks another. This drives the scorer and trace code that gets generated.\n\nRead the answers back as a short restatement so the user can correct you before\nyou plan anything.\n\n## Phase 3 — Dataset planning → **APPROVAL GATE**\n\nFrom the intake, decide the sourcing route and propose it:\n\n- **Existing dataset likely exists** → search Hugging Face and surface 2–3\n  candidates, each with a one-line \"why this fits\" and its size\u002Fsplit. (See\n  `references\u002Fhf-dataset-search.md`.)\n- **Niche \u002F sensitive \u002F synthetic-friendly** → propose generating it\n  synthetically, and include a **clean control bucket** (inputs whose correct\n  answer is true by construction) so at least one slice has unimpeachable labels.\n\nRecommend one route, explain the tradeoff in a sentence, and **wait for the user\nto pick.** Then map + push the chosen source per `references\u002Fdataset-import.md`.\n\n`braintrust-build-eval-dataset` owns the rest of the sourcing argument — population, strata,\nlabel provenance, splits, headroom — and `braintrust-size-eval-dataset` owns how many items.\nLoad them when the composition is a real decision rather than \"use this benchmark.\"\n\n**Provenance check (after the pick, before import).** Unique to this phase, and\nworth the few minutes: establish *why the dataset is trustworthy* and log it to\n`ANALYSIS-SUMMARY.md` — who built it (org\u002Flab), HF downloads + likes, the\nassociated paper and where it was published\u002Fcited, GitHub stars, and whether\nother benchmarks\u002Fleaderboards use it. If the trail is thin, say so — a\nlow-provenance dataset is usable but the writeup should carry that caveat.\n(Recipe: `references\u002Fhf-dataset-search.md` § Provenance.)\n\n## Phase 4 — Score planning → **TWO APPROVAL GATES**\n\n**Gate 1 — the lineup.** Propose scorers as a table and let the user\nadd\u002Fdrop\u002Fchange before you write any code:\n\n> | Scorer | Type | What it checks |\n> |---|---|---|\n> | `entity_recall` | Deterministic | % of expected entities present in the output |\n> | `exact_match` | Deterministic | Output equals `expected` |\n> | `answer_quality` | LLM judge | Does the answer resolve the request? (binary) |\n> | `policy_adherence` | LLM judge | Did it follow the stated policy? (binary) |\n\n**Gate 2 — the implementation.** For each approved scorer, show the *actual*\nthing before it runs — the **rubric\u002Fprompt** for a judge, or the **logic** for a\ndeterministic scorer — and get sign-off.\n\n**New scorers always re-open the gates.** Any scorer introduced after Gate 1 —\nmid-eval, for a new track, or as a \"small addition\" — gets the same treatment\nbefore it runs: **explicitly alert the user that a new scorer is being added,\nshow what it measures and how (prompt\u002Flogic verbatim), and wait for sign-off.**\nNever let a new metric slip into an experiment inside a bigger change.\n\n`braintrust-write-eval-scorer` owns how a scorer is written — method choice, anchored\nrubrics, emitting classes rather than numbers, failure handling — and\n`braintrust-validate-eval-scorer` owns whether it can be trusted for anything beyond\nexploration. Two run-shaped defaults belong here, though:\n\n- **Measure the thing the change actually moves.** A broad overall score can hide\n  a targeted effect; scope the metric to the slice under test.\n- **\"Deterministic\" means code-computed, not pass\u002Ffail.** Say this when presenting\n  the lineup, or the table reads as \"exact match\" and the user approves something\n  narrower than you meant.\n\n**Where the code lives.** Data always lands as a BT Dataset, never inline in code.\nPush scorers so they're visible in the UI: LLM-judge prompts as pushed scorer\nfunctions; heavy code scorers (browser, torch) run locally but record their file\nplus git\u002Fcontent hash in experiment metadata, so every run points at an exact\nscorer version even though the code never left your machine.\n\n## Phase 5 — Run loop\n\nRun the experiment(s) over the dataset. One `Eval()` per variant in the matrix.\n\n**Model as a variable (offer it).** Default the agent\u002Ftask model to\n**claude-sonnet-5** (cheap enough to iterate, strong enough to be interesting)\n— but once a track has baseline results, **proactively suggest a\nmodel-comparison rerun** (e.g. same rows\u002Ftrials on a stronger model like\nFable 5) as a follow-up experiment. Model changes are a new variable: new\nexperiment names carrying a model slug, smoke + gate as usual, and cost\nestimated up front (stronger models can be several × the per-run cost).\n\n**Small dataset → trials.** If **n \u003C 30**, run **3 trials per row**\n(`Eval(trial_count=3)`) so per-row noise doesn't masquerade as a variant\neffect; report scores as means across trials. State the trial count in the\nexperiment metadata and the results table. Trials multiply cost and quota —\ninclude them in the preflight math below. (`braintrust-design-eval-experiment` owns K for\nanything gating.)\n\n**Naming.** `v1_\u003Cmodel>` \u002F `v2_\u003Cmodel>`, per `references\u002Fplatform-mechanics.md`\n§5. The rename recipe, for when `Eval()` generates its own suffix, is in\n`references\u002Fexperiment-hygiene.md`.\n\n**Headless subprocess agents: brainstorm the tool allowlist deliberately.**\nA headless `claude -p` agent **cannot answer permission prompts** — any call to\nan un-allowlisted tool makes it stall mid-run asking for approval that never\ncomes, producing an empty-output failure that *looks like a variant defect*.\n`--permission-mode acceptEdits` covers file edits and trivially-safe reads\nonly; **Bash is NOT allowed by default.** Before the first smoke, run a\ndedicated brainstorm: \"what would a resourceful human reach for mid-task?\" —\ne.g. image-heavy tasks → Bash (Python\u002FPIL color sampling, cropping, base64\nencoding), asset tasks → download\u002Fencode tools, plus Read\u002FWrite\u002FEdit and the\n`mcp__\u003Cserver>` entries under test. Allowlist each need or *consciously\nexclude it and record the exclusion as a harness condition*. Keep the\nallowlist **identical across variants** and **pin it in experiment metadata**\nso failures can be attributed later. Failure signature to watch for: run ends\nwith the agent asking a question (\"Could you approve…\") and no output file.\n\n**Quota preflight (before any full agentic run).** If the task depends on an\nexternal service — an MCP server, a third-party API, a rate-limited provider —\n**check its plan\u002Fquota limits before launching the batch**, not after rows\nstart failing. Do the math out loud: `rows × variants × est. calls per run`\nvs. the plan's limit (remember smoke runs and deleted reruns spend the same\nquota). If the budget doesn't clearly fit: ask the user to upgrade, shrink the\nsubset, or split the run across quota windows. Then **preflight one real\nmutation call** through the service (read-only calls succeeding proves\nnothing — quotas often gate writes only). Quota exhaustion mid-batch\ninvalidates the variant: stop, delete the partial, rerun when quota allows.\n\n**Smoke first, then delete it.** Run a **5-example smoke experiment** first\n(named e.g. `smoke_\u003Cvariant>`) purely to confirm the pipeline runs end-to-end\nwith **0 errors**. A smoke run is never a result — **delete the smoke experiment\nbefore the full run** so it can't pollute comparisons.\n\n**→ HARD GATE between smoke and full run.** After presenting smoke results,\n**stop and wait**. Give the user room to ask questions, inspect scorer outputs,\nand request tweaks — then ask explicitly (\"ready to start the full pass?\") via a\nclickable confirmation (AskUserQuestion) and **do not launch the full run until\nthey say yes.** Never roll from smoke into the full run in one motion, even when\nthe smoke is clean.\n\n**Throttle + retry.** Bounded concurrency (`maxConcurrency` \u002F `EVAL_CONCURRENCY`,\nper `references\u002Fplatform-mechanics.md` §7) and 429 backoff honoring `Retry-After`;\ndrop concurrency further for free-tier providers.\n\n**Structure the trace.** Log spans the way these projects settled on:\nroot span `type:\"task\"` carrying the overall input\u002Foutput\u002Fmetadata; one named\nchild span per stage (or per LLM call) with its own input\u002Foutput + `metrics`\n(tokens, timing); surface errors\u002Ftool-failures onto the root so Topics\u002FIssues can\ncluster on them. **Media lives in the trace:** any image\u002Faudio the task consumes\nor produces (input images, rendered screenshots, artifacts) goes on the span as\nan `Attachment` so the trace view shows it inline — never just a local file path.\n(See `references\u002Ftrace-structure.md`.)\nFor **agentic evals running `claude -p` subprocesses**: enable the\n`trace-claude-code` plugin per run so each agent's full internal trace (every\ntool\u002FMCP call) lands in the project's **Logs**, and put an `agent_trace_url`\npermalink (`...\u002Flogs?r=\u003Csession_id>`) in the experiment span's metadata.\nDeliver the env config via `--settings` file (user settings.json env overrides\nsubprocess env!) and **verify spans actually landed** — hooks fail silently.\nFull recipe + failure catalog: `references\u002Fsubprocess-tracing.md`.\n\n**Verify completion — don't trust \"completed.\"** `ok = roots − errored`, before\nbelieving any aggregate (`references\u002Fplatform-mechanics.md` §2; the row-level\ndefinitions are in `references\u002Fexperiment-hygiene.md`).\n\n**One variable at a time.** Keep the dataset and inputs fixed across variants.\nDon't slip in an extra fix mid-run — rerun all variants together or not at all.\n\n## Phase 6 — Batch summary & sanity check\n\nA first implementation is usually not fully correct, so pressure-test the batch\nbefore reporting anything from it. This phase asks one question — *did the run\nwork?* — and stops there. What the numbers mean is `braintrust-analyze-eval-experiment`.\n\n1. **Results table**, one row per experiment, always with `n (ok)` beside the\n   scores so a half-errored run can't masquerade as a clean one. Shape and the\n   integrity check: `references\u002Fexperiment-hygiene.md`.\n\n2. **Scan the traces.** Open the lowest-scoring rows and every errored one and\n   look for signs the *run* is broken rather than the model being bad — the\n   checklist is in `references\u002Fexperiment-hygiene.md`. Two red flags matter most\n   and are easy to miss because neither produces an error:\n   - A score **identical across variants on every row** — the scorer is measuring\n     something the variants cannot affect, or both saturate a ceiling.\n   - In agentic evals, **the agent never using the capability under test.** Log\n     tool usage per run and check it; a comparison where the variable is never\n     exercised is Sonnet-vs-Sonnet with extra steps.\n\n3. **Say what you found**, in a few sentences: which variant leads, where it\n   fails, and specifically whether anything looks like a harness bug rather than\n   a model difference. Flag suspicion explicitly — \"these numbers may not be\n   trustworthy because X\" beats a confident wrong summary, and this is the last\n   point where a broken run is cheap to catch.\n\n## Phase 7 — Hand off to deeper analysis\nFor intervals, pairing, subgroups, and fragility, hand off to **`braintrust-analyze-eval-experiment`** —\nit continues the running log below.\n\n**Turning the log into a blog?** That's a separate step, and not everyone wants\nit — hand off to a dedicated writing skill that owns the research-blog structure, the\njargon rule, and the voice pass. Not included in this library.\n\n**Agentic-eval analysis menu** (stats worth computing beyond score means —\npick what the experiment makes interesting):\n- **Reliability**: failed\u002Fzero-score trials per variant with a failure\n  taxonomy (API drop \u002F agent stall \u002F tool quota \u002F render fail — read the\n  agents' final messages); scores recomputed excluding failures, so\n  \"worse quality\" vs \"worse survival\" separate cleanly.\n- **Efficiency**: duration, cost, and tool calls per run; cost per quality\n  point across variants.\n- **Behavior**: tool-mix histograms; self-correction loops (e.g.\n  screenshots\u002Frun) and their **correlation with quality scores** (scatter —\n  a weak correlation is a finding); iteration granularity (few-big-writes vs\n  many-small-edits).\n- **Markup\u002Foutput forensics**: semantic-tag usage, div ratios, output size,\n  technique adoption greps (e.g. backdrop-filter) per variant\u002Fprompt.\n- **Judge forensics**: grade distributions; recurring loss-themes in judge\n  reasons; CLIP-vs-judge agreement.\n- **Slices**: per-row win matrix, score-by-design-category, score vs. launch\n  order (drift\u002Fquota degradation over a batch).\n\n## Running log — `ANALYSIS-SUMMARY.md`\nKeep an `ANALYSIS-SUMMARY.md` at the project root and append to it *as you go*, so\nthe eval ends up blog-ready without reconstructing it later. Log the **methodology**\n(what's tested, dataset, scorers, run conditions), the **dataset provenance blurb**\n(who built it, downloads\u002Fstars, paper + citations — see Phase 3), and **only the big learnings** —\n**Every behavioral or structural claim needs at least one concrete example\npulled from a specific trace\u002Flog** — an actual output snippet, tool-call\nsequence, or judge explanation, cited by experiment + row. \"Variant A writes\nmore conventional markup\" is an assertion; a side-by-side of the two outputs'\n`\u003Cbody>` openings from the same row is evidence. If no example can be found,\nthe claim doesn't go in.\nsurprising results, confounds you found, decisions and why. Skip routine steps and\nraw dumps. Deeper analysis continues this same file.\n\n---\n\n## Overridable defaults (sticky for the session, say so to override)\n\nRecaps of the phases above, in checklist form: quota preflight before any run that\nleans on an external service; 5-example smoke at 0 errors, then delete it; pause\nafter every smoke for an explicit clickable yes; `n \u003C 30 → trial_count=3`; delete\nthe prior partial or smoke automatically before a same-named rerun, and confirm\nbefore deleting a completed one; one variable at a time with the dataset fixed;\nappend methodology and big learnings to `ANALYSIS-SUMMARY.md` as you go. Smoke\ntests verify that it *runs* — never conclude which variant is better from n=5.\n\nThe three below are stated only here, and each one changes what the numbers mean:\n\n- **Failed runs: attribute before you retry — failures can be signal, not\n  noise.** Classify each failure first: (a) *transient infrastructure* (API\n  drops) → retry and purge the dead original, disclosing the retry count;\n  (b) *harness-caused* (missing tool permission, config bug) → fix the\n  harness, disclose, excludable; (c) *variant-caused* (agent stalls, gives\n  up, its tool quota\u002FAPI fails) → **keep it — that IS the product\n  experience**. Retry policy must be **symmetric across variants**. Always\n  report both all-runs and successful-only scores plus the survival rate as\n  its own metric — the decomposition is often the finding.\n- If you do retry, **dedupe — don't delete signal**: `Eval(update=True)`\n  **appends — it does not replace the dead row**. When a retried row now\n  *succeeds*, purge only its **superseded dead original** (delete that one\n  event) so the case isn't double-counted. **Never purge a failure that is\n  still a failure** — it counts toward the survival rate. Re-pull the summary\n  after; aggregates that still count the duplicate can flip a comparison's\n  apparent winner.\n- **`ANALYSIS-SUMMARY.md` mirrors the live experiments — it is a summary of\n  current learnings, not an append-only log.** When the user asks to delete a\n  completed experiment or rerun a new version of it, **ask them first: archive\n  the analysis or remove it?** (AskUserQuestion — archive moves results +\n  learnings to a clearly-marked ARCHIVED block, quotable as \"from the deleted\n  run\"; remove leaves only a one-line pointer to what replaced it.) Episodes\n  with narrative value — a confound caught, a diagnosis story — usually\n  deserve the archive. Either way, remove the numbers from the *live*\n  sections in the same breath. Remember Braintrust experiments are\n  unrecoverable once deleted — settle the archive question and capture the\n  doc content BEFORE deleting, and keep any plots\u002Fcaches wanted for the story.\n\n## Troubleshooting\n- **`BRAINTRUST_API_KEY_* not set`** → the chosen org's key is missing from `.env`;\n  see `references\u002Fbt-auth.md`.\n- **`ModuleNotFoundError: braintrust` (but an import check passed earlier)** → the\n  workspace root's `braintrust\u002F` repo clone shadows the SDK. Use a per-eval venv\n  and `.venv\u002Fbin\u002Fpython`; never test SDK presence from the workspace root.\n- **Nested `claude -p` returns 401 (agentic evals)** — check in this order:\n  1. An **invalid `ANTHROPIC_API_KEY`** shadowing the claude.ai login. It can hide\n     in `~\u002F.zshrc`\u002F`~\u002F.bashrc` AND `~\u002F.claude\u002Fsettings.local.json` (`env` block —\n     applies even with a clean process env). Remove it everywhere.\n  2. **Parent-session env leakage**: a harness running inside Claude Code exports\n     `ANTHROPIC_BASE_URL` + `CLAUDE_CODE_*` OAuth vars that break a child `claude`.\n     Always strip those two patterns from the subprocess env before spawning.\n  3. **Stale CLI OAuth token**: the desktop app rotates refresh tokens, orphaning\n     the CLI's keychain copy. Only the user can fix this — have them run\n     `claude \u002Flogin` in a fresh terminal. A smoke run costing $0.00 with 0 tokens\n     is the signature of all three.\n- **ECONNRESET \u002F connection storm at scale** → concurrency too high; lower\n  `EVAL_CONCURRENCY`.\n- **Run \"completed\" but averages look off** → check `ok = roots − errored`; likely\n  most rows errored (rate limits).\n- **Aggregate score barely moves after a targeted change** → likely the wrong\n  metric; scope the score to the slice the change actually affects.\n- **Agentic eval: rows suddenly produce empty output, no agent_error, normal\n  turn counts** → read the agents' final messages before blaming the harness —\n  a third-party tool quota (e.g. an MCP server's weekly limit) may have run\n  out mid-batch. Budget tool-call quota like tokens before launching (rows ×\n  calls\u002Frow vs. the plan's limit), preflight one *mutation* call per MCP, and\n  treat mid-run exhaustion as invalidating the variant: stop, delete the\n  partial, rerun when quota allows.\n\n## References\nAuthored inside this skill folder, loaded on demand:\n- `references\u002Fexperiment-hygiene.md` — naming (good\u002Fbad examples), rename, delete,\n  verify-completion, and the batch results-summary table\n- `references\u002Ftrace-structure.md` — the span-tree convention\n- `references\u002Fhf-dataset-search.md` — finding an existing HF dataset\n- `references\u002Fdataset-import.md` — landing a chosen source as a BT Dataset\u002FLogs\n- `references\u002Fdataset-mapping.md` — the `to_record` mapping recipes\n- `references\u002Fsubprocess-tracing.md` — full agent-internals tracing for\n  `claude -p` subprocess evals (trace-claude-code plugin recipe + failure catalog)\n- `references\u002Fbt-setup.md`, `references\u002Fbt-auth.md` — install + org-scoped keys\n\n**Downstream, not included here:** a writing skill that turns the finished\n`ANALYSIS-SUMMARY.md` into a publishable post (research-blog structure, jargon\nrule, voice pass).\n",{"data":33,"body":34},{"name":4,"description":6},{"type":35,"children":36},"root",[37,45,59,71,85,92,142,147,151,157,167,176,187,243,297,307,313,318,384,389,400,405,442,460,479,511,522,532,660,691,708,727,750,760,766,778,802,842,889,965,997,1036,1060,1100,1216,1247,1257,1263,1281,1369,1375,1390,1400,1410,1480,1491,1546,1549,1555,1582,1587,1710,1716,1960,1966,1971,2068],{"type":38,"tag":39,"props":40,"children":41},"element","h1",{"id":4},[42],{"type":43,"value":44},"text","Braintrust Eval",{"type":38,"tag":46,"props":47,"children":48},"p",{},[49,51,57],{"type":43,"value":50},"The ",{"type":38,"tag":52,"props":53,"children":54},"strong",{},[55],{"type":43,"value":56},"runbook",{"type":43,"value":58},". Take the user from \"I want to eval X\" to a clean, comparable\nBraintrust experiment that actually ran. Deliberately interactive: ask, propose,\nget approval, then spend money.",{"type":38,"tag":46,"props":60,"children":61},{},[62,64,69],{"type":43,"value":63},"This skill owns ",{"type":38,"tag":52,"props":65,"children":66},{},[67],{"type":43,"value":68},"execution",{"type":43,"value":70}," — credentials, imports, code, run hygiene, tracing,\nand the approval gates that keep a paid run from starting on an unreviewed plan.\nIt does not own methodology: each phase below names the lifecycle card that does,\nand those are worth loading when a judgment call is live rather than routine.",{"type":38,"tag":46,"props":72,"children":73},{},[74,76,83],{"type":43,"value":75},"Shared platform mechanics — pinning, naming, safe reads, run hygiene — are in\n",{"type":38,"tag":77,"props":78,"children":80},"code",{"className":79},[],[81],{"type":43,"value":82},"references\u002Fplatform-mechanics.md",{"type":43,"value":84},", not restated per phase.",{"type":38,"tag":86,"props":87,"children":89},"h2",{"id":88},"when-to-use-when-not",[90],{"type":43,"value":91},"When to use \u002F when not",{"type":38,"tag":93,"props":94,"children":95},"ul",{},[96,107,125],{"type":38,"tag":97,"props":98,"children":99},"li",{},[100,105],{"type":38,"tag":52,"props":101,"children":102},{},[103],{"type":43,"value":104},"Use this",{"type":43,"value":106}," to run a new eval or model\u002Fprompt comparison against a real project.",{"type":38,"tag":97,"props":108,"children":109},{},[110,115,117,123],{"type":38,"tag":52,"props":111,"children":112},{},[113],{"type":43,"value":114},"Getting data into Braintrust",{"type":43,"value":116}," (a HuggingFace \u002F CSV \u002F trace source into a BT\ndataset)? Phase 3, via ",{"type":38,"tag":77,"props":118,"children":120},{"className":119},[],[121],{"type":43,"value":122},"references\u002Fdataset-import.md",{"type":43,"value":124},".",{"type":38,"tag":97,"props":126,"children":127},{},[128,133,135,141],{"type":38,"tag":52,"props":129,"children":130},{},[131],{"type":43,"value":132},"Pulling or plotting results",{"type":43,"value":134}," from experiments that already ran? Out of scope —\nthat is ",{"type":38,"tag":77,"props":136,"children":138},{"className":137},[],[139],{"type":43,"value":140},"braintrust-analyze-eval-experiment",{"type":43,"value":124},{"type":38,"tag":46,"props":143,"children":144},{},[145],{"type":43,"value":146},"Work through the phases in order. Do not skip the approval gates — the whole point\nis that the user signs off on the dataset and the scorers before a paid run.",{"type":38,"tag":148,"props":149,"children":150},"hr",{},[],{"type":38,"tag":86,"props":152,"children":154},{"id":153},"phase-1-preflight-before-anything-else",[155],{"type":43,"value":156},"Phase 1 — Preflight (before anything else)",{"type":38,"tag":46,"props":158,"children":159},{},[160,165],{"type":38,"tag":52,"props":161,"children":162},{},[163],{"type":43,"value":164},"1a. Ask which org.",{"type":43,"value":166}," Braintrust API keys are org-scoped, so this decides which\ncredential the run uses — and it's a data-governance decision.",{"type":38,"tag":168,"props":169,"children":170},"blockquote",{},[171],{"type":38,"tag":46,"props":172,"children":173},{},[174],{"type":43,"value":175},"\"Which org should this run in?\" — list the orgs the environment has keys for.\nSkip the question entirely if there is only one.",{"type":38,"tag":46,"props":177,"children":178},{},[179,181],{"type":43,"value":180},"Governance default, state it if relevant: ",{"type":38,"tag":182,"props":183,"children":184},"em",{},[185],{"type":43,"value":186},"if the eval involves sensitive,\ncustomer, or proprietary data, default to the most restricted org available unless\nthe user says otherwise.",{"type":38,"tag":46,"props":188,"children":189},{},[190,195,197,202,204,210,212,218,220,226,228,234,236,242],{"type":38,"tag":52,"props":191,"children":192},{},[193],{"type":43,"value":194},"1b. Select + verify the key.",{"type":43,"value":196}," Keys live in ",{"type":38,"tag":52,"props":198,"children":199},{},[200],{"type":43,"value":201},"one canonical file",{"type":43,"value":203},", resolved as\n",{"type":38,"tag":77,"props":205,"children":207},{"className":206},[],[208],{"type":43,"value":209},"$BRAINTRUST_ENV_FILE",{"type":43,"value":211}," → ",{"type":38,"tag":77,"props":213,"children":215},{"className":214},[],[216],{"type":43,"value":217},"~\u002F.braintrust\u002F.env",{"type":43,"value":219}," → the current project's ",{"type":38,"tag":77,"props":221,"children":223},{"className":222},[],[224],{"type":43,"value":225},".env",{"type":43,"value":227}," (that\nlast one only if neither of the others exists). Each org's key is\n",{"type":38,"tag":77,"props":229,"children":231},{"className":230},[],[232],{"type":43,"value":233},"BRAINTRUST_API_KEY_\u003CORG>",{"type":43,"value":235},"; a single-org setup can use a plain\n",{"type":38,"tag":77,"props":237,"children":239},{"className":238},[],[240],{"type":43,"value":241},"BRAINTRUST_API_KEY",{"type":43,"value":124},{"type":38,"tag":46,"props":244,"children":245},{},[246,251,253,258,260,265,267,272,274,279,281,287,289,295],{"type":38,"tag":52,"props":247,"children":248},{},[249],{"type":43,"value":250},"Do not hunt for keys",{"type":43,"value":252}," across sub-project ",{"type":38,"tag":77,"props":254,"children":256},{"className":255},[],[257],{"type":43,"value":225},{"type":43,"value":259}," files — that trips credential-\nscanning guards. If the chosen key is missing or empty: create\u002Fappend the stub\nline to the canonical ",{"type":38,"tag":77,"props":261,"children":263},{"className":262},[],[264],{"type":43,"value":225},{"type":43,"value":266},", tell the user exactly which variable to fill in\n(key from Braintrust app → Settings → API Keys, per org), and wait. ",{"type":38,"tag":52,"props":268,"children":269},{},[270],{"type":43,"value":271},"Never ask\nthe user to paste a key into the chat, and never print one.",{"type":43,"value":273}," Verify the filled\nkey by listing the org's ",{"type":38,"tag":52,"props":275,"children":276},{},[277],{"type":43,"value":278},"projects",{"type":43,"value":280}," via the API (not ",{"type":38,"tag":77,"props":282,"children":284},{"className":283},[],[285],{"type":43,"value":286},"\u002Fv1\u002Forganization",{"type":43,"value":288}," — it can\nreturn empty for member keys). (See ",{"type":38,"tag":77,"props":290,"children":292},{"className":291},[],[293],{"type":43,"value":294},"references\u002Fbt-auth.md",{"type":43,"value":296},".)",{"type":38,"tag":46,"props":298,"children":299},{},[300,305],{"type":38,"tag":52,"props":301,"children":302},{},[303],{"type":43,"value":304},"1c. Ask which project",{"type":43,"value":306},", then confirm it exists in that org via the API. If it\ndoesn't exist, offer to create it.",{"type":38,"tag":86,"props":308,"children":310},{"id":309},"phase-2-intake",[311],{"type":43,"value":312},"Phase 2 — Intake",{"type":38,"tag":46,"props":314,"children":315},{},[316],{"type":43,"value":317},"Three open prompts (answer in prose), plus one quick config pick:",{"type":38,"tag":319,"props":320,"children":321},"ol",{},[322,332,342,352],{"type":38,"tag":97,"props":323,"children":324},{},[325,330],{"type":38,"tag":52,"props":326,"children":327},{},[328],{"type":43,"value":329},"What do you want to eval?",{"type":43,"value":331}," (They'll usually name the model and\u002For prompt\nvariant here — that's the thing under test.)",{"type":38,"tag":97,"props":333,"children":334},{},[335,340],{"type":38,"tag":52,"props":336,"children":337},{},[338],{"type":43,"value":339},"Describe what the dataset should be.",{"type":43,"value":341}," (~3 sentences: the inputs, what a good\nanswer looks like, any domains\u002Fedge cases to cover.)",{"type":38,"tag":97,"props":343,"children":344},{},[345,350],{"type":38,"tag":52,"props":346,"children":347},{},[348],{"type":43,"value":349},"Describe what scores you'd like to see.",{"type":43,"value":351}," (What does \"good\" mean here? What\nfailures matter most?)",{"type":38,"tag":97,"props":353,"children":354},{},[355,360,362,368,370,375,377,382],{"type":38,"tag":52,"props":356,"children":357},{},[358],{"type":43,"value":359},"Which language should the eval be written in?",{"type":43,"value":361}," Braintrust ships full SDKs —\ntracing, the ",{"type":38,"tag":77,"props":363,"children":365},{"className":364},[],[366],{"type":43,"value":367},"Eval()",{"type":43,"value":369}," framework, and scorers — in ",{"type":38,"tag":52,"props":371,"children":372},{},[373],{"type":43,"value":374},"Python, TypeScript, Go,\nJava, Ruby, and C#",{"type":43,"value":376},". Surface all six and ",{"type":38,"tag":52,"props":378,"children":379},{},[380],{"type":43,"value":381},"default to Python",{"type":43,"value":383}," unless the user\npicks another. This drives the scorer and trace code that gets generated.",{"type":38,"tag":46,"props":385,"children":386},{},[387],{"type":43,"value":388},"Read the answers back as a short restatement so the user can correct you before\nyou plan anything.",{"type":38,"tag":86,"props":390,"children":392},{"id":391},"phase-3-dataset-planning-approval-gate",[393,395],{"type":43,"value":394},"Phase 3 — Dataset planning → ",{"type":38,"tag":52,"props":396,"children":397},{},[398],{"type":43,"value":399},"APPROVAL GATE",{"type":38,"tag":46,"props":401,"children":402},{},[403],{"type":43,"value":404},"From the intake, decide the sourcing route and propose it:",{"type":38,"tag":93,"props":406,"children":407},{},[408,425],{"type":38,"tag":97,"props":409,"children":410},{},[411,416,418,424],{"type":38,"tag":52,"props":412,"children":413},{},[414],{"type":43,"value":415},"Existing dataset likely exists",{"type":43,"value":417}," → search Hugging Face and surface 2–3\ncandidates, each with a one-line \"why this fits\" and its size\u002Fsplit. (See\n",{"type":38,"tag":77,"props":419,"children":421},{"className":420},[],[422],{"type":43,"value":423},"references\u002Fhf-dataset-search.md",{"type":43,"value":296},{"type":38,"tag":97,"props":426,"children":427},{},[428,433,435,440],{"type":38,"tag":52,"props":429,"children":430},{},[431],{"type":43,"value":432},"Niche \u002F sensitive \u002F synthetic-friendly",{"type":43,"value":434}," → propose generating it\nsynthetically, and include a ",{"type":38,"tag":52,"props":436,"children":437},{},[438],{"type":43,"value":439},"clean control bucket",{"type":43,"value":441}," (inputs whose correct\nanswer is true by construction) so at least one slice has unimpeachable labels.",{"type":38,"tag":46,"props":443,"children":444},{},[445,447,452,454,459],{"type":43,"value":446},"Recommend one route, explain the tradeoff in a sentence, and ",{"type":38,"tag":52,"props":448,"children":449},{},[450],{"type":43,"value":451},"wait for the user\nto pick.",{"type":43,"value":453}," Then map + push the chosen source per ",{"type":38,"tag":77,"props":455,"children":457},{"className":456},[],[458],{"type":43,"value":122},{"type":43,"value":124},{"type":38,"tag":46,"props":461,"children":462},{},[463,469,471,477],{"type":38,"tag":77,"props":464,"children":466},{"className":465},[],[467],{"type":43,"value":468},"braintrust-build-eval-dataset",{"type":43,"value":470}," owns the rest of the sourcing argument — population, strata,\nlabel provenance, splits, headroom — and ",{"type":38,"tag":77,"props":472,"children":474},{"className":473},[],[475],{"type":43,"value":476},"braintrust-size-eval-dataset",{"type":43,"value":478}," owns how many items.\nLoad them when the composition is a real decision rather than \"use this benchmark.\"",{"type":38,"tag":46,"props":480,"children":481},{},[482,487,489,494,496,502,504,509],{"type":38,"tag":52,"props":483,"children":484},{},[485],{"type":43,"value":486},"Provenance check (after the pick, before import).",{"type":43,"value":488}," Unique to this phase, and\nworth the few minutes: establish ",{"type":38,"tag":182,"props":490,"children":491},{},[492],{"type":43,"value":493},"why the dataset is trustworthy",{"type":43,"value":495}," and log it to\n",{"type":38,"tag":77,"props":497,"children":499},{"className":498},[],[500],{"type":43,"value":501},"ANALYSIS-SUMMARY.md",{"type":43,"value":503}," — who built it (org\u002Flab), HF downloads + likes, the\nassociated paper and where it was published\u002Fcited, GitHub stars, and whether\nother benchmarks\u002Fleaderboards use it. If the trail is thin, say so — a\nlow-provenance dataset is usable but the writeup should carry that caveat.\n(Recipe: ",{"type":38,"tag":77,"props":505,"children":507},{"className":506},[],[508],{"type":43,"value":423},{"type":43,"value":510}," § Provenance.)",{"type":38,"tag":86,"props":512,"children":514},{"id":513},"phase-4-score-planning-two-approval-gates",[515,517],{"type":43,"value":516},"Phase 4 — Score planning → ",{"type":38,"tag":52,"props":518,"children":519},{},[520],{"type":43,"value":521},"TWO APPROVAL GATES",{"type":38,"tag":46,"props":523,"children":524},{},[525,530],{"type":38,"tag":52,"props":526,"children":527},{},[528],{"type":43,"value":529},"Gate 1 — the lineup.",{"type":43,"value":531}," Propose scorers as a table and let the user\nadd\u002Fdrop\u002Fchange before you write any code:",{"type":38,"tag":168,"props":533,"children":534},{},[535],{"type":38,"tag":536,"props":537,"children":538},"table",{},[539,563],{"type":38,"tag":540,"props":541,"children":542},"thead",{},[543],{"type":38,"tag":544,"props":545,"children":546},"tr",{},[547,553,558],{"type":38,"tag":548,"props":549,"children":550},"th",{},[551],{"type":43,"value":552},"Scorer",{"type":38,"tag":548,"props":554,"children":555},{},[556],{"type":43,"value":557},"Type",{"type":38,"tag":548,"props":559,"children":560},{},[561],{"type":43,"value":562},"What it checks",{"type":38,"tag":564,"props":565,"children":566},"tbody",{},[567,590,617,639],{"type":38,"tag":544,"props":568,"children":569},{},[570,580,585],{"type":38,"tag":571,"props":572,"children":573},"td",{},[574],{"type":38,"tag":77,"props":575,"children":577},{"className":576},[],[578],{"type":43,"value":579},"entity_recall",{"type":38,"tag":571,"props":581,"children":582},{},[583],{"type":43,"value":584},"Deterministic",{"type":38,"tag":571,"props":586,"children":587},{},[588],{"type":43,"value":589},"% of expected entities present in the output",{"type":38,"tag":544,"props":591,"children":592},{},[593,602,606],{"type":38,"tag":571,"props":594,"children":595},{},[596],{"type":38,"tag":77,"props":597,"children":599},{"className":598},[],[600],{"type":43,"value":601},"exact_match",{"type":38,"tag":571,"props":603,"children":604},{},[605],{"type":43,"value":584},{"type":38,"tag":571,"props":607,"children":608},{},[609,611],{"type":43,"value":610},"Output equals ",{"type":38,"tag":77,"props":612,"children":614},{"className":613},[],[615],{"type":43,"value":616},"expected",{"type":38,"tag":544,"props":618,"children":619},{},[620,629,634],{"type":38,"tag":571,"props":621,"children":622},{},[623],{"type":38,"tag":77,"props":624,"children":626},{"className":625},[],[627],{"type":43,"value":628},"answer_quality",{"type":38,"tag":571,"props":630,"children":631},{},[632],{"type":43,"value":633},"LLM judge",{"type":38,"tag":571,"props":635,"children":636},{},[637],{"type":43,"value":638},"Does the answer resolve the request? (binary)",{"type":38,"tag":544,"props":640,"children":641},{},[642,651,655],{"type":38,"tag":571,"props":643,"children":644},{},[645],{"type":38,"tag":77,"props":646,"children":648},{"className":647},[],[649],{"type":43,"value":650},"policy_adherence",{"type":38,"tag":571,"props":652,"children":653},{},[654],{"type":43,"value":633},{"type":38,"tag":571,"props":656,"children":657},{},[658],{"type":43,"value":659},"Did it follow the stated policy? (binary)",{"type":38,"tag":46,"props":661,"children":662},{},[663,668,670,675,677,682,684,689],{"type":38,"tag":52,"props":664,"children":665},{},[666],{"type":43,"value":667},"Gate 2 — the implementation.",{"type":43,"value":669}," For each approved scorer, show the ",{"type":38,"tag":182,"props":671,"children":672},{},[673],{"type":43,"value":674},"actual",{"type":43,"value":676},"\nthing before it runs — the ",{"type":38,"tag":52,"props":678,"children":679},{},[680],{"type":43,"value":681},"rubric\u002Fprompt",{"type":43,"value":683}," for a judge, or the ",{"type":38,"tag":52,"props":685,"children":686},{},[687],{"type":43,"value":688},"logic",{"type":43,"value":690}," for a\ndeterministic scorer — and get sign-off.",{"type":38,"tag":46,"props":692,"children":693},{},[694,699,701,706],{"type":38,"tag":52,"props":695,"children":696},{},[697],{"type":43,"value":698},"New scorers always re-open the gates.",{"type":43,"value":700}," Any scorer introduced after Gate 1 —\nmid-eval, for a new track, or as a \"small addition\" — gets the same treatment\nbefore it runs: ",{"type":38,"tag":52,"props":702,"children":703},{},[704],{"type":43,"value":705},"explicitly alert the user that a new scorer is being added,\nshow what it measures and how (prompt\u002Flogic verbatim), and wait for sign-off.",{"type":43,"value":707},"\nNever let a new metric slip into an experiment inside a bigger change.",{"type":38,"tag":46,"props":709,"children":710},{},[711,717,719,725],{"type":38,"tag":77,"props":712,"children":714},{"className":713},[],[715],{"type":43,"value":716},"braintrust-write-eval-scorer",{"type":43,"value":718}," owns how a scorer is written — method choice, anchored\nrubrics, emitting classes rather than numbers, failure handling — and\n",{"type":38,"tag":77,"props":720,"children":722},{"className":721},[],[723],{"type":43,"value":724},"braintrust-validate-eval-scorer",{"type":43,"value":726}," owns whether it can be trusted for anything beyond\nexploration. Two run-shaped defaults belong here, though:",{"type":38,"tag":93,"props":728,"children":729},{},[730,740],{"type":38,"tag":97,"props":731,"children":732},{},[733,738],{"type":38,"tag":52,"props":734,"children":735},{},[736],{"type":43,"value":737},"Measure the thing the change actually moves.",{"type":43,"value":739}," A broad overall score can hide\na targeted effect; scope the metric to the slice under test.",{"type":38,"tag":97,"props":741,"children":742},{},[743,748],{"type":38,"tag":52,"props":744,"children":745},{},[746],{"type":43,"value":747},"\"Deterministic\" means code-computed, not pass\u002Ffail.",{"type":43,"value":749}," Say this when presenting\nthe lineup, or the table reads as \"exact match\" and the user approves something\nnarrower than you meant.",{"type":38,"tag":46,"props":751,"children":752},{},[753,758],{"type":38,"tag":52,"props":754,"children":755},{},[756],{"type":43,"value":757},"Where the code lives.",{"type":43,"value":759}," Data always lands as a BT Dataset, never inline in code.\nPush scorers so they're visible in the UI: LLM-judge prompts as pushed scorer\nfunctions; heavy code scorers (browser, torch) run locally but record their file\nplus git\u002Fcontent hash in experiment metadata, so every run points at an exact\nscorer version even though the code never left your machine.",{"type":38,"tag":86,"props":761,"children":763},{"id":762},"phase-5-run-loop",[764],{"type":43,"value":765},"Phase 5 — Run loop",{"type":38,"tag":46,"props":767,"children":768},{},[769,771,776],{"type":43,"value":770},"Run the experiment(s) over the dataset. One ",{"type":38,"tag":77,"props":772,"children":774},{"className":773},[],[775],{"type":43,"value":367},{"type":43,"value":777}," per variant in the matrix.",{"type":38,"tag":46,"props":779,"children":780},{},[781,786,788,793,795,800],{"type":38,"tag":52,"props":782,"children":783},{},[784],{"type":43,"value":785},"Model as a variable (offer it).",{"type":43,"value":787}," Default the agent\u002Ftask model to\n",{"type":38,"tag":52,"props":789,"children":790},{},[791],{"type":43,"value":792},"claude-sonnet-5",{"type":43,"value":794}," (cheap enough to iterate, strong enough to be interesting)\n— but once a track has baseline results, ",{"type":38,"tag":52,"props":796,"children":797},{},[798],{"type":43,"value":799},"proactively suggest a\nmodel-comparison rerun",{"type":43,"value":801}," (e.g. same rows\u002Ftrials on a stronger model like\nFable 5) as a follow-up experiment. Model changes are a new variable: new\nexperiment names carrying a model slug, smoke + gate as usual, and cost\nestimated up front (stronger models can be several × the per-run cost).",{"type":38,"tag":46,"props":803,"children":804},{},[805,810,812,817,819,824,826,832,834,840],{"type":38,"tag":52,"props":806,"children":807},{},[808],{"type":43,"value":809},"Small dataset → trials.",{"type":43,"value":811}," If ",{"type":38,"tag":52,"props":813,"children":814},{},[815],{"type":43,"value":816},"n \u003C 30",{"type":43,"value":818},", run ",{"type":38,"tag":52,"props":820,"children":821},{},[822],{"type":43,"value":823},"3 trials per row",{"type":43,"value":825},"\n(",{"type":38,"tag":77,"props":827,"children":829},{"className":828},[],[830],{"type":43,"value":831},"Eval(trial_count=3)",{"type":43,"value":833},") so per-row noise doesn't masquerade as a variant\neffect; report scores as means across trials. State the trial count in the\nexperiment metadata and the results table. Trials multiply cost and quota —\ninclude them in the preflight math below. (",{"type":38,"tag":77,"props":835,"children":837},{"className":836},[],[838],{"type":43,"value":839},"braintrust-design-eval-experiment",{"type":43,"value":841}," owns K for\nanything gating.)",{"type":38,"tag":46,"props":843,"children":844},{},[845,850,852,858,860,866,868,873,875,880,882,888],{"type":38,"tag":52,"props":846,"children":847},{},[848],{"type":43,"value":849},"Naming.",{"type":43,"value":851}," ",{"type":38,"tag":77,"props":853,"children":855},{"className":854},[],[856],{"type":43,"value":857},"v1_\u003Cmodel>",{"type":43,"value":859}," \u002F ",{"type":38,"tag":77,"props":861,"children":863},{"className":862},[],[864],{"type":43,"value":865},"v2_\u003Cmodel>",{"type":43,"value":867},", per ",{"type":38,"tag":77,"props":869,"children":871},{"className":870},[],[872],{"type":43,"value":82},{"type":43,"value":874},"\n§5. The rename recipe, for when ",{"type":38,"tag":77,"props":876,"children":878},{"className":877},[],[879],{"type":43,"value":367},{"type":43,"value":881}," generates its own suffix, is in\n",{"type":38,"tag":77,"props":883,"children":885},{"className":884},[],[886],{"type":43,"value":887},"references\u002Fexperiment-hygiene.md",{"type":43,"value":124},{"type":38,"tag":46,"props":890,"children":891},{},[892,897,899,905,907,912,914,919,921,927,929,934,936,942,944,949,951,956,958,963],{"type":38,"tag":52,"props":893,"children":894},{},[895],{"type":43,"value":896},"Headless subprocess agents: brainstorm the tool allowlist deliberately.",{"type":43,"value":898},"\nA headless ",{"type":38,"tag":77,"props":900,"children":902},{"className":901},[],[903],{"type":43,"value":904},"claude -p",{"type":43,"value":906}," agent ",{"type":38,"tag":52,"props":908,"children":909},{},[910],{"type":43,"value":911},"cannot answer permission prompts",{"type":43,"value":913}," — any call to\nan un-allowlisted tool makes it stall mid-run asking for approval that never\ncomes, producing an empty-output failure that ",{"type":38,"tag":182,"props":915,"children":916},{},[917],{"type":43,"value":918},"looks like a variant defect",{"type":43,"value":920},".\n",{"type":38,"tag":77,"props":922,"children":924},{"className":923},[],[925],{"type":43,"value":926},"--permission-mode acceptEdits",{"type":43,"value":928}," covers file edits and trivially-safe reads\nonly; ",{"type":38,"tag":52,"props":930,"children":931},{},[932],{"type":43,"value":933},"Bash is NOT allowed by default.",{"type":43,"value":935}," Before the first smoke, run a\ndedicated brainstorm: \"what would a resourceful human reach for mid-task?\" —\ne.g. image-heavy tasks → Bash (Python\u002FPIL color sampling, cropping, base64\nencoding), asset tasks → download\u002Fencode tools, plus Read\u002FWrite\u002FEdit and the\n",{"type":38,"tag":77,"props":937,"children":939},{"className":938},[],[940],{"type":43,"value":941},"mcp__\u003Cserver>",{"type":43,"value":943}," entries under test. Allowlist each need or ",{"type":38,"tag":182,"props":945,"children":946},{},[947],{"type":43,"value":948},"consciously\nexclude it and record the exclusion as a harness condition",{"type":43,"value":950},". Keep the\nallowlist ",{"type":38,"tag":52,"props":952,"children":953},{},[954],{"type":43,"value":955},"identical across variants",{"type":43,"value":957}," and ",{"type":38,"tag":52,"props":959,"children":960},{},[961],{"type":43,"value":962},"pin it in experiment metadata",{"type":43,"value":964},"\nso failures can be attributed later. Failure signature to watch for: run ends\nwith the agent asking a question (\"Could you approve…\") and no output file.",{"type":38,"tag":46,"props":966,"children":967},{},[968,973,975,980,982,988,990,995],{"type":38,"tag":52,"props":969,"children":970},{},[971],{"type":43,"value":972},"Quota preflight (before any full agentic run).",{"type":43,"value":974}," If the task depends on an\nexternal service — an MCP server, a third-party API, a rate-limited provider —\n",{"type":38,"tag":52,"props":976,"children":977},{},[978],{"type":43,"value":979},"check its plan\u002Fquota limits before launching the batch",{"type":43,"value":981},", not after rows\nstart failing. Do the math out loud: ",{"type":38,"tag":77,"props":983,"children":985},{"className":984},[],[986],{"type":43,"value":987},"rows × variants × est. calls per run",{"type":43,"value":989},"\nvs. the plan's limit (remember smoke runs and deleted reruns spend the same\nquota). If the budget doesn't clearly fit: ask the user to upgrade, shrink the\nsubset, or split the run across quota windows. Then ",{"type":38,"tag":52,"props":991,"children":992},{},[993],{"type":43,"value":994},"preflight one real\nmutation call",{"type":43,"value":996}," through the service (read-only calls succeeding proves\nnothing — quotas often gate writes only). Quota exhaustion mid-batch\ninvalidates the variant: stop, delete the partial, rerun when quota allows.",{"type":38,"tag":46,"props":998,"children":999},{},[1000,1005,1007,1012,1014,1020,1022,1027,1029,1034],{"type":38,"tag":52,"props":1001,"children":1002},{},[1003],{"type":43,"value":1004},"Smoke first, then delete it.",{"type":43,"value":1006}," Run a ",{"type":38,"tag":52,"props":1008,"children":1009},{},[1010],{"type":43,"value":1011},"5-example smoke experiment",{"type":43,"value":1013}," first\n(named e.g. ",{"type":38,"tag":77,"props":1015,"children":1017},{"className":1016},[],[1018],{"type":43,"value":1019},"smoke_\u003Cvariant>",{"type":43,"value":1021},") purely to confirm the pipeline runs end-to-end\nwith ",{"type":38,"tag":52,"props":1023,"children":1024},{},[1025],{"type":43,"value":1026},"0 errors",{"type":43,"value":1028},". A smoke run is never a result — ",{"type":38,"tag":52,"props":1030,"children":1031},{},[1032],{"type":43,"value":1033},"delete the smoke experiment\nbefore the full run",{"type":43,"value":1035}," so it can't pollute comparisons.",{"type":38,"tag":46,"props":1037,"children":1038},{},[1039,1044,1046,1051,1053,1058],{"type":38,"tag":52,"props":1040,"children":1041},{},[1042],{"type":43,"value":1043},"→ HARD GATE between smoke and full run.",{"type":43,"value":1045}," After presenting smoke results,\n",{"type":38,"tag":52,"props":1047,"children":1048},{},[1049],{"type":43,"value":1050},"stop and wait",{"type":43,"value":1052},". Give the user room to ask questions, inspect scorer outputs,\nand request tweaks — then ask explicitly (\"ready to start the full pass?\") via a\nclickable confirmation (AskUserQuestion) and ",{"type":38,"tag":52,"props":1054,"children":1055},{},[1056],{"type":43,"value":1057},"do not launch the full run until\nthey say yes.",{"type":43,"value":1059}," Never roll from smoke into the full run in one motion, even when\nthe smoke is clean.",{"type":38,"tag":46,"props":1061,"children":1062},{},[1063,1068,1070,1076,1077,1083,1085,1090,1092,1098],{"type":38,"tag":52,"props":1064,"children":1065},{},[1066],{"type":43,"value":1067},"Throttle + retry.",{"type":43,"value":1069}," Bounded concurrency (",{"type":38,"tag":77,"props":1071,"children":1073},{"className":1072},[],[1074],{"type":43,"value":1075},"maxConcurrency",{"type":43,"value":859},{"type":38,"tag":77,"props":1078,"children":1080},{"className":1079},[],[1081],{"type":43,"value":1082},"EVAL_CONCURRENCY",{"type":43,"value":1084},",\nper ",{"type":38,"tag":77,"props":1086,"children":1088},{"className":1087},[],[1089],{"type":43,"value":82},{"type":43,"value":1091}," §7) and 429 backoff honoring ",{"type":38,"tag":77,"props":1093,"children":1095},{"className":1094},[],[1096],{"type":43,"value":1097},"Retry-After",{"type":43,"value":1099},";\ndrop concurrency further for free-tier providers.",{"type":38,"tag":46,"props":1101,"children":1102},{},[1103,1108,1110,1116,1118,1124,1126,1131,1133,1139,1141,1147,1149,1161,1163,1169,1171,1176,1178,1184,1186,1192,1194,1200,1202,1207,1209,1215],{"type":38,"tag":52,"props":1104,"children":1105},{},[1106],{"type":43,"value":1107},"Structure the trace.",{"type":43,"value":1109}," Log spans the way these projects settled on:\nroot span ",{"type":38,"tag":77,"props":1111,"children":1113},{"className":1112},[],[1114],{"type":43,"value":1115},"type:\"task\"",{"type":43,"value":1117}," carrying the overall input\u002Foutput\u002Fmetadata; one named\nchild span per stage (or per LLM call) with its own input\u002Foutput + ",{"type":38,"tag":77,"props":1119,"children":1121},{"className":1120},[],[1122],{"type":43,"value":1123},"metrics",{"type":43,"value":1125},"\n(tokens, timing); surface errors\u002Ftool-failures onto the root so Topics\u002FIssues can\ncluster on them. ",{"type":38,"tag":52,"props":1127,"children":1128},{},[1129],{"type":43,"value":1130},"Media lives in the trace:",{"type":43,"value":1132}," any image\u002Faudio the task consumes\nor produces (input images, rendered screenshots, artifacts) goes on the span as\nan ",{"type":38,"tag":77,"props":1134,"children":1136},{"className":1135},[],[1137],{"type":43,"value":1138},"Attachment",{"type":43,"value":1140}," so the trace view shows it inline — never just a local file path.\n(See ",{"type":38,"tag":77,"props":1142,"children":1144},{"className":1143},[],[1145],{"type":43,"value":1146},"references\u002Ftrace-structure.md",{"type":43,"value":1148},".)\nFor ",{"type":38,"tag":52,"props":1150,"children":1151},{},[1152,1154,1159],{"type":43,"value":1153},"agentic evals running ",{"type":38,"tag":77,"props":1155,"children":1157},{"className":1156},[],[1158],{"type":43,"value":904},{"type":43,"value":1160}," subprocesses",{"type":43,"value":1162},": enable the\n",{"type":38,"tag":77,"props":1164,"children":1166},{"className":1165},[],[1167],{"type":43,"value":1168},"trace-claude-code",{"type":43,"value":1170}," plugin per run so each agent's full internal trace (every\ntool\u002FMCP call) lands in the project's ",{"type":38,"tag":52,"props":1172,"children":1173},{},[1174],{"type":43,"value":1175},"Logs",{"type":43,"value":1177},", and put an ",{"type":38,"tag":77,"props":1179,"children":1181},{"className":1180},[],[1182],{"type":43,"value":1183},"agent_trace_url",{"type":43,"value":1185},"\npermalink (",{"type":38,"tag":77,"props":1187,"children":1189},{"className":1188},[],[1190],{"type":43,"value":1191},"...\u002Flogs?r=\u003Csession_id>",{"type":43,"value":1193},") in the experiment span's metadata.\nDeliver the env config via ",{"type":38,"tag":77,"props":1195,"children":1197},{"className":1196},[],[1198],{"type":43,"value":1199},"--settings",{"type":43,"value":1201}," file (user settings.json env overrides\nsubprocess env!) and ",{"type":38,"tag":52,"props":1203,"children":1204},{},[1205],{"type":43,"value":1206},"verify spans actually landed",{"type":43,"value":1208}," — hooks fail silently.\nFull recipe + failure catalog: ",{"type":38,"tag":77,"props":1210,"children":1212},{"className":1211},[],[1213],{"type":43,"value":1214},"references\u002Fsubprocess-tracing.md",{"type":43,"value":124},{"type":38,"tag":46,"props":1217,"children":1218},{},[1219,1224,1225,1231,1233,1238,1240,1245],{"type":38,"tag":52,"props":1220,"children":1221},{},[1222],{"type":43,"value":1223},"Verify completion — don't trust \"completed.\"",{"type":43,"value":851},{"type":38,"tag":77,"props":1226,"children":1228},{"className":1227},[],[1229],{"type":43,"value":1230},"ok = roots − errored",{"type":43,"value":1232},", before\nbelieving any aggregate (",{"type":38,"tag":77,"props":1234,"children":1236},{"className":1235},[],[1237],{"type":43,"value":82},{"type":43,"value":1239}," §2; the row-level\ndefinitions are in ",{"type":38,"tag":77,"props":1241,"children":1243},{"className":1242},[],[1244],{"type":43,"value":887},{"type":43,"value":1246},").",{"type":38,"tag":46,"props":1248,"children":1249},{},[1250,1255],{"type":38,"tag":52,"props":1251,"children":1252},{},[1253],{"type":43,"value":1254},"One variable at a time.",{"type":43,"value":1256}," Keep the dataset and inputs fixed across variants.\nDon't slip in an extra fix mid-run — rerun all variants together or not at all.",{"type":38,"tag":86,"props":1258,"children":1260},{"id":1259},"phase-6-batch-summary-sanity-check",[1261],{"type":43,"value":1262},"Phase 6 — Batch summary & sanity check",{"type":38,"tag":46,"props":1264,"children":1265},{},[1266,1268,1273,1275,1280],{"type":43,"value":1267},"A first implementation is usually not fully correct, so pressure-test the batch\nbefore reporting anything from it. This phase asks one question — ",{"type":38,"tag":182,"props":1269,"children":1270},{},[1271],{"type":43,"value":1272},"did the run\nwork?",{"type":43,"value":1274}," — and stops there. What the numbers mean is ",{"type":38,"tag":77,"props":1276,"children":1278},{"className":1277},[],[1279],{"type":43,"value":140},{"type":43,"value":124},{"type":38,"tag":319,"props":1282,"children":1283},{},[1284,1308,1359],{"type":38,"tag":97,"props":1285,"children":1286},{},[1287,1292,1294,1300,1302,1307],{"type":38,"tag":52,"props":1288,"children":1289},{},[1290],{"type":43,"value":1291},"Results table",{"type":43,"value":1293},", one row per experiment, always with ",{"type":38,"tag":77,"props":1295,"children":1297},{"className":1296},[],[1298],{"type":43,"value":1299},"n (ok)",{"type":43,"value":1301}," beside the\nscores so a half-errored run can't masquerade as a clean one. Shape and the\nintegrity check: ",{"type":38,"tag":77,"props":1303,"children":1305},{"className":1304},[],[1306],{"type":43,"value":887},{"type":43,"value":124},{"type":38,"tag":97,"props":1309,"children":1310},{},[1311,1316,1318,1323,1325,1330,1332],{"type":38,"tag":52,"props":1312,"children":1313},{},[1314],{"type":43,"value":1315},"Scan the traces.",{"type":43,"value":1317}," Open the lowest-scoring rows and every errored one and\nlook for signs the ",{"type":38,"tag":182,"props":1319,"children":1320},{},[1321],{"type":43,"value":1322},"run",{"type":43,"value":1324}," is broken rather than the model being bad — the\nchecklist is in ",{"type":38,"tag":77,"props":1326,"children":1328},{"className":1327},[],[1329],{"type":43,"value":887},{"type":43,"value":1331},". Two red flags matter most\nand are easy to miss because neither produces an error:",{"type":38,"tag":93,"props":1333,"children":1334},{},[1335,1347],{"type":38,"tag":97,"props":1336,"children":1337},{},[1338,1340,1345],{"type":43,"value":1339},"A score ",{"type":38,"tag":52,"props":1341,"children":1342},{},[1343],{"type":43,"value":1344},"identical across variants on every row",{"type":43,"value":1346}," — the scorer is measuring\nsomething the variants cannot affect, or both saturate a ceiling.",{"type":38,"tag":97,"props":1348,"children":1349},{},[1350,1352,1357],{"type":43,"value":1351},"In agentic evals, ",{"type":38,"tag":52,"props":1353,"children":1354},{},[1355],{"type":43,"value":1356},"the agent never using the capability under test.",{"type":43,"value":1358}," Log\ntool usage per run and check it; a comparison where the variable is never\nexercised is Sonnet-vs-Sonnet with extra steps.",{"type":38,"tag":97,"props":1360,"children":1361},{},[1362,1367],{"type":38,"tag":52,"props":1363,"children":1364},{},[1365],{"type":43,"value":1366},"Say what you found",{"type":43,"value":1368},", in a few sentences: which variant leads, where it\nfails, and specifically whether anything looks like a harness bug rather than\na model difference. Flag suspicion explicitly — \"these numbers may not be\ntrustworthy because X\" beats a confident wrong summary, and this is the last\npoint where a broken run is cheap to catch.",{"type":38,"tag":86,"props":1370,"children":1372},{"id":1371},"phase-7-hand-off-to-deeper-analysis",[1373],{"type":43,"value":1374},"Phase 7 — Hand off to deeper analysis",{"type":38,"tag":46,"props":1376,"children":1377},{},[1378,1380,1388],{"type":43,"value":1379},"For intervals, pairing, subgroups, and fragility, hand off to ",{"type":38,"tag":52,"props":1381,"children":1382},{},[1383],{"type":38,"tag":77,"props":1384,"children":1386},{"className":1385},[],[1387],{"type":43,"value":140},{"type":43,"value":1389}," —\nit continues the running log below.",{"type":38,"tag":46,"props":1391,"children":1392},{},[1393,1398],{"type":38,"tag":52,"props":1394,"children":1395},{},[1396],{"type":43,"value":1397},"Turning the log into a blog?",{"type":43,"value":1399}," That's a separate step, and not everyone wants\nit — hand off to a dedicated writing skill that owns the research-blog structure, the\njargon rule, and the voice pass. Not included in this library.",{"type":38,"tag":46,"props":1401,"children":1402},{},[1403,1408],{"type":38,"tag":52,"props":1404,"children":1405},{},[1406],{"type":43,"value":1407},"Agentic-eval analysis menu",{"type":43,"value":1409}," (stats worth computing beyond score means —\npick what the experiment makes interesting):",{"type":38,"tag":93,"props":1411,"children":1412},{},[1413,1423,1433,1450,1460,1470],{"type":38,"tag":97,"props":1414,"children":1415},{},[1416,1421],{"type":38,"tag":52,"props":1417,"children":1418},{},[1419],{"type":43,"value":1420},"Reliability",{"type":43,"value":1422},": failed\u002Fzero-score trials per variant with a failure\ntaxonomy (API drop \u002F agent stall \u002F tool quota \u002F render fail — read the\nagents' final messages); scores recomputed excluding failures, so\n\"worse quality\" vs \"worse survival\" separate cleanly.",{"type":38,"tag":97,"props":1424,"children":1425},{},[1426,1431],{"type":38,"tag":52,"props":1427,"children":1428},{},[1429],{"type":43,"value":1430},"Efficiency",{"type":43,"value":1432},": duration, cost, and tool calls per run; cost per quality\npoint across variants.",{"type":38,"tag":97,"props":1434,"children":1435},{},[1436,1441,1443,1448],{"type":38,"tag":52,"props":1437,"children":1438},{},[1439],{"type":43,"value":1440},"Behavior",{"type":43,"value":1442},": tool-mix histograms; self-correction loops (e.g.\nscreenshots\u002Frun) and their ",{"type":38,"tag":52,"props":1444,"children":1445},{},[1446],{"type":43,"value":1447},"correlation with quality scores",{"type":43,"value":1449}," (scatter —\na weak correlation is a finding); iteration granularity (few-big-writes vs\nmany-small-edits).",{"type":38,"tag":97,"props":1451,"children":1452},{},[1453,1458],{"type":38,"tag":52,"props":1454,"children":1455},{},[1456],{"type":43,"value":1457},"Markup\u002Foutput forensics",{"type":43,"value":1459},": semantic-tag usage, div ratios, output size,\ntechnique adoption greps (e.g. backdrop-filter) per variant\u002Fprompt.",{"type":38,"tag":97,"props":1461,"children":1462},{},[1463,1468],{"type":38,"tag":52,"props":1464,"children":1465},{},[1466],{"type":43,"value":1467},"Judge forensics",{"type":43,"value":1469},": grade distributions; recurring loss-themes in judge\nreasons; CLIP-vs-judge agreement.",{"type":38,"tag":97,"props":1471,"children":1472},{},[1473,1478],{"type":38,"tag":52,"props":1474,"children":1475},{},[1476],{"type":43,"value":1477},"Slices",{"type":43,"value":1479},": per-row win matrix, score-by-design-category, score vs. launch\norder (drift\u002Fquota degradation over a batch).",{"type":38,"tag":86,"props":1481,"children":1483},{"id":1482},"running-log-analysis-summarymd",[1484,1486],{"type":43,"value":1485},"Running log — ",{"type":38,"tag":77,"props":1487,"children":1489},{"className":1488},[],[1490],{"type":43,"value":501},{"type":38,"tag":46,"props":1492,"children":1493},{},[1494,1496,1501,1503,1508,1510,1515,1517,1522,1524,1529,1531,1536,1538,1544],{"type":43,"value":1495},"Keep an ",{"type":38,"tag":77,"props":1497,"children":1499},{"className":1498},[],[1500],{"type":43,"value":501},{"type":43,"value":1502}," at the project root and append to it ",{"type":38,"tag":182,"props":1504,"children":1505},{},[1506],{"type":43,"value":1507},"as you go",{"type":43,"value":1509},", so\nthe eval ends up blog-ready without reconstructing it later. Log the ",{"type":38,"tag":52,"props":1511,"children":1512},{},[1513],{"type":43,"value":1514},"methodology",{"type":43,"value":1516},"\n(what's tested, dataset, scorers, run conditions), the ",{"type":38,"tag":52,"props":1518,"children":1519},{},[1520],{"type":43,"value":1521},"dataset provenance blurb",{"type":43,"value":1523},"\n(who built it, downloads\u002Fstars, paper + citations — see Phase 3), and ",{"type":38,"tag":52,"props":1525,"children":1526},{},[1527],{"type":43,"value":1528},"only the big learnings",{"type":43,"value":1530}," —\n",{"type":38,"tag":52,"props":1532,"children":1533},{},[1534],{"type":43,"value":1535},"Every behavioral or structural claim needs at least one concrete example\npulled from a specific trace\u002Flog",{"type":43,"value":1537}," — an actual output snippet, tool-call\nsequence, or judge explanation, cited by experiment + row. \"Variant A writes\nmore conventional markup\" is an assertion; a side-by-side of the two outputs'\n",{"type":38,"tag":77,"props":1539,"children":1541},{"className":1540},[],[1542],{"type":43,"value":1543},"\u003Cbody>",{"type":43,"value":1545}," openings from the same row is evidence. If no example can be found,\nthe claim doesn't go in.\nsurprising results, confounds you found, decisions and why. Skip routine steps and\nraw dumps. Deeper analysis continues this same file.",{"type":38,"tag":148,"props":1547,"children":1548},{},[],{"type":38,"tag":86,"props":1550,"children":1552},{"id":1551},"overridable-defaults-sticky-for-the-session-say-so-to-override",[1553],{"type":43,"value":1554},"Overridable defaults (sticky for the session, say so to override)",{"type":38,"tag":46,"props":1556,"children":1557},{},[1558,1560,1566,1568,1573,1575,1580],{"type":43,"value":1559},"Recaps of the phases above, in checklist form: quota preflight before any run that\nleans on an external service; 5-example smoke at 0 errors, then delete it; pause\nafter every smoke for an explicit clickable yes; ",{"type":38,"tag":77,"props":1561,"children":1563},{"className":1562},[],[1564],{"type":43,"value":1565},"n \u003C 30 → trial_count=3",{"type":43,"value":1567},"; delete\nthe prior partial or smoke automatically before a same-named rerun, and confirm\nbefore deleting a completed one; one variable at a time with the dataset fixed;\nappend methodology and big learnings to ",{"type":38,"tag":77,"props":1569,"children":1571},{"className":1570},[],[1572],{"type":43,"value":501},{"type":43,"value":1574}," as you go. Smoke\ntests verify that it ",{"type":38,"tag":182,"props":1576,"children":1577},{},[1578],{"type":43,"value":1579},"runs",{"type":43,"value":1581}," — never conclude which variant is better from n=5.",{"type":38,"tag":46,"props":1583,"children":1584},{},[1585],{"type":43,"value":1586},"The three below are stated only here, and each one changes what the numbers mean:",{"type":38,"tag":93,"props":1588,"children":1589},{},[1590,1635,1681],{"type":38,"tag":97,"props":1591,"children":1592},{},[1593,1598,1600,1605,1607,1612,1614,1619,1621,1626,1628,1633],{"type":38,"tag":52,"props":1594,"children":1595},{},[1596],{"type":43,"value":1597},"Failed runs: attribute before you retry — failures can be signal, not\nnoise.",{"type":43,"value":1599}," Classify each failure first: (a) ",{"type":38,"tag":182,"props":1601,"children":1602},{},[1603],{"type":43,"value":1604},"transient infrastructure",{"type":43,"value":1606}," (API\ndrops) → retry and purge the dead original, disclosing the retry count;\n(b) ",{"type":38,"tag":182,"props":1608,"children":1609},{},[1610],{"type":43,"value":1611},"harness-caused",{"type":43,"value":1613}," (missing tool permission, config bug) → fix the\nharness, disclose, excludable; (c) ",{"type":38,"tag":182,"props":1615,"children":1616},{},[1617],{"type":43,"value":1618},"variant-caused",{"type":43,"value":1620}," (agent stalls, gives\nup, its tool quota\u002FAPI fails) → ",{"type":38,"tag":52,"props":1622,"children":1623},{},[1624],{"type":43,"value":1625},"keep it — that IS the product\nexperience",{"type":43,"value":1627},". Retry policy must be ",{"type":38,"tag":52,"props":1629,"children":1630},{},[1631],{"type":43,"value":1632},"symmetric across variants",{"type":43,"value":1634},". Always\nreport both all-runs and successful-only scores plus the survival rate as\nits own metric — the decomposition is often the finding.",{"type":38,"tag":97,"props":1636,"children":1637},{},[1638,1640,1645,1647,1653,1658,1660,1665,1667,1672,1674,1679],{"type":43,"value":1639},"If you do retry, ",{"type":38,"tag":52,"props":1641,"children":1642},{},[1643],{"type":43,"value":1644},"dedupe — don't delete signal",{"type":43,"value":1646},": ",{"type":38,"tag":77,"props":1648,"children":1650},{"className":1649},[],[1651],{"type":43,"value":1652},"Eval(update=True)",{"type":38,"tag":52,"props":1654,"children":1655},{},[1656],{"type":43,"value":1657},"appends — it does not replace the dead row",{"type":43,"value":1659},". When a retried row now\n",{"type":38,"tag":182,"props":1661,"children":1662},{},[1663],{"type":43,"value":1664},"succeeds",{"type":43,"value":1666},", purge only its ",{"type":38,"tag":52,"props":1668,"children":1669},{},[1670],{"type":43,"value":1671},"superseded dead original",{"type":43,"value":1673}," (delete that one\nevent) so the case isn't double-counted. ",{"type":38,"tag":52,"props":1675,"children":1676},{},[1677],{"type":43,"value":1678},"Never purge a failure that is\nstill a failure",{"type":43,"value":1680}," — it counts toward the survival rate. Re-pull the summary\nafter; aggregates that still count the duplicate can flip a comparison's\napparent winner.",{"type":38,"tag":97,"props":1682,"children":1683},{},[1684,1694,1696,1701,1703,1708],{"type":38,"tag":52,"props":1685,"children":1686},{},[1687,1692],{"type":38,"tag":77,"props":1688,"children":1690},{"className":1689},[],[1691],{"type":43,"value":501},{"type":43,"value":1693}," mirrors the live experiments — it is a summary of\ncurrent learnings, not an append-only log.",{"type":43,"value":1695}," When the user asks to delete a\ncompleted experiment or rerun a new version of it, ",{"type":38,"tag":52,"props":1697,"children":1698},{},[1699],{"type":43,"value":1700},"ask them first: archive\nthe analysis or remove it?",{"type":43,"value":1702}," (AskUserQuestion — archive moves results +\nlearnings to a clearly-marked ARCHIVED block, quotable as \"from the deleted\nrun\"; remove leaves only a one-line pointer to what replaced it.) Episodes\nwith narrative value — a confound caught, a diagnosis story — usually\ndeserve the archive. Either way, remove the numbers from the ",{"type":38,"tag":182,"props":1704,"children":1705},{},[1706],{"type":43,"value":1707},"live",{"type":43,"value":1709},"\nsections in the same breath. Remember Braintrust experiments are\nunrecoverable once deleted — settle the archive question and capture the\ndoc content BEFORE deleting, and keep any plots\u002Fcaches wanted for the story.",{"type":38,"tag":86,"props":1711,"children":1713},{"id":1712},"troubleshooting",[1714],{"type":43,"value":1715},"Troubleshooting",{"type":38,"tag":93,"props":1717,"children":1718},{},[1719,1746,1778,1900,1916,1933,1943],{"type":38,"tag":97,"props":1720,"children":1721},{},[1722,1731,1733,1738,1740,1745],{"type":38,"tag":52,"props":1723,"children":1724},{},[1725],{"type":38,"tag":77,"props":1726,"children":1728},{"className":1727},[],[1729],{"type":43,"value":1730},"BRAINTRUST_API_KEY_* not set",{"type":43,"value":1732}," → the chosen org's key is missing from ",{"type":38,"tag":77,"props":1734,"children":1736},{"className":1735},[],[1737],{"type":43,"value":225},{"type":43,"value":1739},";\nsee ",{"type":38,"tag":77,"props":1741,"children":1743},{"className":1742},[],[1744],{"type":43,"value":294},{"type":43,"value":124},{"type":38,"tag":97,"props":1747,"children":1748},{},[1749,1760,1762,1768,1770,1776],{"type":38,"tag":52,"props":1750,"children":1751},{},[1752,1758],{"type":38,"tag":77,"props":1753,"children":1755},{"className":1754},[],[1756],{"type":43,"value":1757},"ModuleNotFoundError: braintrust",{"type":43,"value":1759}," (but an import check passed earlier)",{"type":43,"value":1761}," → the\nworkspace root's ",{"type":38,"tag":77,"props":1763,"children":1765},{"className":1764},[],[1766],{"type":43,"value":1767},"braintrust\u002F",{"type":43,"value":1769}," repo clone shadows the SDK. Use a per-eval venv\nand ",{"type":38,"tag":77,"props":1771,"children":1773},{"className":1772},[],[1774],{"type":43,"value":1775},".venv\u002Fbin\u002Fpython",{"type":43,"value":1777},"; never test SDK presence from the workspace root.",{"type":38,"tag":97,"props":1779,"children":1780},{},[1781,1793,1795],{"type":38,"tag":52,"props":1782,"children":1783},{},[1784,1786,1791],{"type":43,"value":1785},"Nested ",{"type":38,"tag":77,"props":1787,"children":1789},{"className":1788},[],[1790],{"type":43,"value":904},{"type":43,"value":1792}," returns 401 (agentic evals)",{"type":43,"value":1794}," — check in this order:\n",{"type":38,"tag":319,"props":1796,"children":1797},{},[1798,1848,1882],{"type":38,"tag":97,"props":1799,"children":1800},{},[1801,1803,1814,1816,1822,1824,1830,1832,1838,1840,1846],{"type":43,"value":1802},"An ",{"type":38,"tag":52,"props":1804,"children":1805},{},[1806,1808],{"type":43,"value":1807},"invalid ",{"type":38,"tag":77,"props":1809,"children":1811},{"className":1810},[],[1812],{"type":43,"value":1813},"ANTHROPIC_API_KEY",{"type":43,"value":1815}," shadowing the claude.ai login. It can hide\nin ",{"type":38,"tag":77,"props":1817,"children":1819},{"className":1818},[],[1820],{"type":43,"value":1821},"~\u002F.zshrc",{"type":43,"value":1823},"\u002F",{"type":38,"tag":77,"props":1825,"children":1827},{"className":1826},[],[1828],{"type":43,"value":1829},"~\u002F.bashrc",{"type":43,"value":1831}," AND ",{"type":38,"tag":77,"props":1833,"children":1835},{"className":1834},[],[1836],{"type":43,"value":1837},"~\u002F.claude\u002Fsettings.local.json",{"type":43,"value":1839}," (",{"type":38,"tag":77,"props":1841,"children":1843},{"className":1842},[],[1844],{"type":43,"value":1845},"env",{"type":43,"value":1847}," block —\napplies even with a clean process env). Remove it everywhere.",{"type":38,"tag":97,"props":1849,"children":1850},{},[1851,1856,1858,1864,1866,1872,1874,1880],{"type":38,"tag":52,"props":1852,"children":1853},{},[1854],{"type":43,"value":1855},"Parent-session env leakage",{"type":43,"value":1857},": a harness running inside Claude Code exports\n",{"type":38,"tag":77,"props":1859,"children":1861},{"className":1860},[],[1862],{"type":43,"value":1863},"ANTHROPIC_BASE_URL",{"type":43,"value":1865}," + ",{"type":38,"tag":77,"props":1867,"children":1869},{"className":1868},[],[1870],{"type":43,"value":1871},"CLAUDE_CODE_*",{"type":43,"value":1873}," OAuth vars that break a child ",{"type":38,"tag":77,"props":1875,"children":1877},{"className":1876},[],[1878],{"type":43,"value":1879},"claude",{"type":43,"value":1881},".\nAlways strip those two patterns from the subprocess env before spawning.",{"type":38,"tag":97,"props":1883,"children":1884},{},[1885,1890,1892,1898],{"type":38,"tag":52,"props":1886,"children":1887},{},[1888],{"type":43,"value":1889},"Stale CLI OAuth token",{"type":43,"value":1891},": the desktop app rotates refresh tokens, orphaning\nthe CLI's keychain copy. Only the user can fix this — have them run\n",{"type":38,"tag":77,"props":1893,"children":1895},{"className":1894},[],[1896],{"type":43,"value":1897},"claude \u002Flogin",{"type":43,"value":1899}," in a fresh terminal. A smoke run costing $0.00 with 0 tokens\nis the signature of all three.",{"type":38,"tag":97,"props":1901,"children":1902},{},[1903,1908,1910,1915],{"type":38,"tag":52,"props":1904,"children":1905},{},[1906],{"type":43,"value":1907},"ECONNRESET \u002F connection storm at scale",{"type":43,"value":1909}," → concurrency too high; lower\n",{"type":38,"tag":77,"props":1911,"children":1913},{"className":1912},[],[1914],{"type":43,"value":1082},{"type":43,"value":124},{"type":38,"tag":97,"props":1917,"children":1918},{},[1919,1924,1926,1931],{"type":38,"tag":52,"props":1920,"children":1921},{},[1922],{"type":43,"value":1923},"Run \"completed\" but averages look off",{"type":43,"value":1925}," → check ",{"type":38,"tag":77,"props":1927,"children":1929},{"className":1928},[],[1930],{"type":43,"value":1230},{"type":43,"value":1932},"; likely\nmost rows errored (rate limits).",{"type":38,"tag":97,"props":1934,"children":1935},{},[1936,1941],{"type":38,"tag":52,"props":1937,"children":1938},{},[1939],{"type":43,"value":1940},"Aggregate score barely moves after a targeted change",{"type":43,"value":1942}," → likely the wrong\nmetric; scope the score to the slice the change actually affects.",{"type":38,"tag":97,"props":1944,"children":1945},{},[1946,1951,1953,1958],{"type":38,"tag":52,"props":1947,"children":1948},{},[1949],{"type":43,"value":1950},"Agentic eval: rows suddenly produce empty output, no agent_error, normal\nturn counts",{"type":43,"value":1952}," → read the agents' final messages before blaming the harness —\na third-party tool quota (e.g. an MCP server's weekly limit) may have run\nout mid-batch. Budget tool-call quota like tokens before launching (rows ×\ncalls\u002Frow vs. the plan's limit), preflight one ",{"type":38,"tag":182,"props":1954,"children":1955},{},[1956],{"type":43,"value":1957},"mutation",{"type":43,"value":1959}," call per MCP, and\ntreat mid-run exhaustion as invalidating the variant: stop, delete the\npartial, rerun when quota allows.",{"type":38,"tag":86,"props":1961,"children":1963},{"id":1962},"references",[1964],{"type":43,"value":1965},"References",{"type":38,"tag":46,"props":1967,"children":1968},{},[1969],{"type":43,"value":1970},"Authored inside this skill folder, loaded on demand:",{"type":38,"tag":93,"props":1972,"children":1973},{},[1974,1984,1994,2004,2014,2033,2050],{"type":38,"tag":97,"props":1975,"children":1976},{},[1977,1982],{"type":38,"tag":77,"props":1978,"children":1980},{"className":1979},[],[1981],{"type":43,"value":887},{"type":43,"value":1983}," — naming (good\u002Fbad examples), rename, delete,\nverify-completion, and the batch results-summary table",{"type":38,"tag":97,"props":1985,"children":1986},{},[1987,1992],{"type":38,"tag":77,"props":1988,"children":1990},{"className":1989},[],[1991],{"type":43,"value":1146},{"type":43,"value":1993}," — the span-tree convention",{"type":38,"tag":97,"props":1995,"children":1996},{},[1997,2002],{"type":38,"tag":77,"props":1998,"children":2000},{"className":1999},[],[2001],{"type":43,"value":423},{"type":43,"value":2003}," — finding an existing HF dataset",{"type":38,"tag":97,"props":2005,"children":2006},{},[2007,2012],{"type":38,"tag":77,"props":2008,"children":2010},{"className":2009},[],[2011],{"type":43,"value":122},{"type":43,"value":2013}," — landing a chosen source as a BT Dataset\u002FLogs",{"type":38,"tag":97,"props":2015,"children":2016},{},[2017,2023,2025,2031],{"type":38,"tag":77,"props":2018,"children":2020},{"className":2019},[],[2021],{"type":43,"value":2022},"references\u002Fdataset-mapping.md",{"type":43,"value":2024}," — the ",{"type":38,"tag":77,"props":2026,"children":2028},{"className":2027},[],[2029],{"type":43,"value":2030},"to_record",{"type":43,"value":2032}," mapping recipes",{"type":38,"tag":97,"props":2034,"children":2035},{},[2036,2041,2043,2048],{"type":38,"tag":77,"props":2037,"children":2039},{"className":2038},[],[2040],{"type":43,"value":1214},{"type":43,"value":2042}," — full agent-internals tracing for\n",{"type":38,"tag":77,"props":2044,"children":2046},{"className":2045},[],[2047],{"type":43,"value":904},{"type":43,"value":2049}," subprocess evals (trace-claude-code plugin recipe + failure catalog)",{"type":38,"tag":97,"props":2051,"children":2052},{},[2053,2059,2061,2066],{"type":38,"tag":77,"props":2054,"children":2056},{"className":2055},[],[2057],{"type":43,"value":2058},"references\u002Fbt-setup.md",{"type":43,"value":2060},", ",{"type":38,"tag":77,"props":2062,"children":2064},{"className":2063},[],[2065],{"type":43,"value":294},{"type":43,"value":2067}," — install + org-scoped keys",{"type":38,"tag":46,"props":2069,"children":2070},{},[2071,2076,2078,2083],{"type":38,"tag":52,"props":2072,"children":2073},{},[2074],{"type":43,"value":2075},"Downstream, not included here:",{"type":43,"value":2077}," a writing skill that turns the finished\n",{"type":38,"tag":77,"props":2079,"children":2081},{"className":2080},[],[2082],{"type":43,"value":501},{"type":43,"value":2084}," into a publishable post (research-blog structure, jargon\nrule, voice pass).",{"items":2086,"total":2258},[2087,2103,2120,2134,2149,2165,2179,2191,2204,2220,2232,2244],{"slug":2088,"name":2088,"fn":2089,"description":2090,"org":2091,"tags":2092,"stars":2100,"repoUrl":2101,"updatedAt":2102},"troubleshoot-braintrust-mcp","configure and troubleshoot Braintrust MCP servers","This plugin auto-configures a \"braintrust\" MCP server. If you can't see it or reach it, activate this skill\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2093,2094,2097],{"name":9,"slug":8,"type":16},{"name":2095,"slug":2096,"type":16},"Debugging","debugging",{"name":2098,"slug":2099,"type":16},"MCP","mcp",18,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Fbraintrust-claude-plugin","2026-07-12T08:36:13.889274",{"slug":140,"name":140,"fn":2104,"description":2105,"org":2106,"tags":2107,"stars":21,"repoUrl":22,"updatedAt":2119},"analyze LLM and agent eval experiments","Analyze completed LLM or agent eval experiments using uncertainty-aware and decision-relevant methods. Use to audit run completeness and pairing, calculate confidence intervals, run paired comparisons, report wins, losses, and ties, incorporate run-to-run variance, handle multiple comparisons, inspect subgroup performance, and test fragility to favorable slices. Use when results already exist and someone asks what they mean, whether a difference is real, or which model won. Do not use to design an experiment that has not yet collected results.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2108,2111,2112,2113,2116],{"name":2109,"slug":2110,"type":16},"Analysis","analysis",{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":2114,"slug":2115,"type":16},"LLM","llm",{"name":2117,"slug":2118,"type":16},"Statistics","statistics","2026-08-20T03:53:01.13806",{"slug":2121,"name":2121,"fn":2122,"description":2123,"org":2124,"tags":2125,"stars":21,"repoUrl":22,"updatedAt":2133},"braintrust-attribute-multi-variable-change","attribute performance changes to multiple variables","Attribute an observed change when several things moved at once — model plus prompt plus tools, a provider migration, a framework upgrade, or a vendor swap that bundles serving stack with model. Use when asked which part of a change caused the result, when a comparison's arms differ in more than one way, when a treatment has no uniform implementation across vendors, or when a serving-stack difference is confounded with a model difference. Do not use for a clean single-variable comparison, or to design an experiment that has not yet run.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2126,2127,2128,2129,2130],{"name":2109,"slug":2110,"type":16},{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":2114,"slug":2115,"type":16},{"name":2131,"slug":2132,"type":16},"Performance","performance","2026-08-20T03:53:40.036077",{"slug":468,"name":468,"fn":2135,"description":2136,"org":2137,"tags":2138,"stars":21,"repoUrl":22,"updatedAt":2148},"create and manage LLM eval datasets","Create, edit, audit, or compare eval datasets for LLM applications and agents, including target-population definition, case sourcing from production traces, stratified sampling, label provenance and label audits, expected values as constraints for open-ended tasks, dev\u002Ftest splits, contamination and leakage controls, headroom checks, refresh policy, and datasheets. Use when working on the content or lifecycle of an eval dataset. Do not use for sample-size or power calculations, scorer implementation, or open-ended adversarial failure discovery.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2139,2142,2143,2146,2147],{"name":2140,"slug":2141,"type":16},"Agents","agents",{"name":9,"slug":8,"type":16},{"name":2144,"slug":2145,"type":16},"Datasets","datasets",{"name":14,"slug":15,"type":16},{"name":2114,"slug":2115,"type":16},"2026-08-20T03:53:33.304815",{"slug":2150,"name":2150,"fn":2151,"description":2152,"org":2153,"tags":2154,"stars":21,"repoUrl":22,"updatedAt":2164},"braintrust-define-eval-objective","define LLM evaluation objectives","Create, edit, or audit an eval objective by working backward from a product decision to the target outcome, construct, population, intended claim, and verification-versus-validation questions. Use when a team is unsure what an eval should establish, asks \"what are we actually trying to measure,\" \"is this eval measuring the right thing,\" \"does this benchmark support our claim,\" or needs to turn a product goal into an eval objective and state which claims are out of scope. Do not use to select detailed metrics, design datasets, or implement scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2155,2156,2157,2158,2161],{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":2114,"slug":2115,"type":16},{"name":2159,"slug":2160,"type":16},"Product Management","product-management",{"name":2162,"slug":2163,"type":16},"Strategy","strategy","2026-08-20T03:53:00.07097",{"slug":2166,"name":2166,"fn":2167,"description":2168,"org":2169,"tags":2170,"stars":21,"repoUrl":22,"updatedAt":2178},"braintrust-define-eval-release-gate","configure release gates for LLM applications","Create, edit, audit, or apply release gates for LLM applications and agents. Use to combine minimum meaningful improvement, statistical significance, regression rate, subgroup consistency, worst-run stability, all-attempts reliability, safety upper bounds, latency, and cost into an explicit ship-or-hold policy, to turn metrics into a CI gate, or to explain why a candidate failed a gate and what evidence would justify reconsideration. Do not use for general result analysis without a deployment decision.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2171,2172,2173,2176,2177],{"name":2140,"slug":2141,"type":16},{"name":9,"slug":8,"type":16},{"name":2174,"slug":2175,"type":16},"CI\u002FCD","ci-cd",{"name":14,"slug":15,"type":16},{"name":2114,"slug":2115,"type":16},"2026-08-20T03:53:15.686158",{"slug":2180,"name":2180,"fn":2181,"description":2182,"org":2183,"tags":2184,"stars":21,"repoUrl":22,"updatedAt":2190},"braintrust-deploy-evaluator","deploy evaluators to Braintrust","Take a validated scorer or classifier from definition to running instrument in Braintrust — scope selection, inline testing before saving, saving as an evaluator, attaching an online-scoring rule, activating it for new traffic, and backfilling history with a rewind. Use when a scorer needs to actually run against production logs, when an online-scoring rule needs to be created or changed, or when historical traces need scoring. Do not use to decide what the scorer should measure, to write its rubric, or to establish that it agrees with human judgment.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2185,2186,2189],{"name":9,"slug":8,"type":16},{"name":2187,"slug":2188,"type":16},"Deployment","deployment",{"name":14,"slug":15,"type":16},"2026-08-20T03:53:32.558937",{"slug":839,"name":839,"fn":2192,"description":2193,"org":2194,"tags":2195,"stars":21,"repoUrl":22,"updatedAt":2203},"design controlled LLM eval experiments","Design or audit controlled eval experiments for model, prompt, retrieval, tool, guardrail, or agent-architecture changes. Use before data collection to state directional and minimum-effect hypotheses, name independent, dependent, and control variables including the serving environment and tool surface, choose paired designs, set repetitions and allocation, distinguish exploratory from confirmatory comparisons, and pre-specify stopping, exclusion, multiplicity, and analysis rules. Do not use primarily to analyze results already collected.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2196,2197,2198,2201,2202],{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":2199,"slug":2200,"type":16},"Experiments","experiments",{"name":2114,"slug":2115,"type":16},{"name":2162,"slug":2163,"type":16},"2026-08-20T03:53:36.534554",{"slug":2205,"name":2205,"fn":2206,"description":2207,"org":2208,"tags":2209,"stars":21,"repoUrl":22,"updatedAt":2219},"braintrust-design-eval-instrumentation","design trace and evaluation dataset schemas","Design the trace and eval-dataset schema for an LLM app or agent, and wire the system to emit it. Use when deciding what to log, designing a trace schema, setting up tracing or observability before evals, or when failures cannot be debugged or sliced from existing traces — covering inputs, outputs, spans for tool and LLM calls, state changes, metadata, resolved configuration, serving path, tool manifest, per-item status, attachments, and subgroup variables. Do not use to decide what the evidence should mean, or to write scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2210,2211,2212,2213,2216],{"name":9,"slug":8,"type":16},{"name":2144,"slug":2145,"type":16},{"name":14,"slug":15,"type":16},{"name":2214,"slug":2215,"type":16},"Observability","observability",{"name":2217,"slug":2218,"type":16},"Tracing","tracing","2026-08-20T03:53:37.274703",{"slug":2221,"name":2221,"fn":2222,"description":2223,"org":2224,"tags":2225,"stars":21,"repoUrl":22,"updatedAt":2231},"braintrust-design-eval-metric-bundle","create multi-objective evaluation metric bundles","Create, edit, audit, or compare a multi-objective eval metric bundle covering product quality, safety, reliability, latency, and cost. Use to choose metrics for an eval, define a goodness bundle, distinguish optimization metrics from non-regression guardrails, expose tradeoffs, audit a KPI or single composite score for Goodhart and metric-gaming risk, or answer \"what should improve and what must not regress.\" Do not use to design trace schemas, build datasets, or implement scoring methods.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2226,2227,2228,2229,2230],{"name":2140,"slug":2141,"type":16},{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":2114,"slug":2115,"type":16},{"name":2131,"slug":2132,"type":16},"2026-08-20T03:53:16.078609",{"slug":2233,"name":2233,"fn":2234,"description":2235,"org":2236,"tags":2237,"stars":21,"repoUrl":22,"updatedAt":2243},"braintrust-design-human-eval-review","design human evaluation and review workflows","Design or audit human evaluation workflows and golden datasets for LLM applications and agents. Use to set up expert review, select review cases, write reviewer instructions, assign raters, capture rationales and confidence, measure inter-rater agreement with kappa or alpha, adjudicate disagreements, and preserve reviewed examples with provenance as a versioned reference set. Do not use to elicit the criteria or rubric in the first place, to validate a scorer once reference labels exist, or to implement the scorer.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2238,2239,2240,2241,2242],{"name":2140,"slug":2141,"type":16},{"name":9,"slug":8,"type":16},{"name":2144,"slug":2145,"type":16},{"name":14,"slug":15,"type":16},{"name":2114,"slug":2115,"type":16},"2026-08-20T03:53:36.900129",{"slug":2245,"name":2245,"fn":2246,"description":2247,"org":2248,"tags":2249,"stars":21,"repoUrl":22,"updatedAt":2257},"braintrust-discover-agent-failures","identify and classify agent failure modes","Search open-endedly for unanticipated agent failure modes and convert them into a named taxonomy and durable regression items. Use for requests to find out what goes wrong, surface unknown or silent failures, do error analysis over traces, cluster and triage production failures, or build a failure taxonomy — where the goal is discovering modes nobody thought to test rather than measuring a predefined criterion. Produces datasets and taxonomies, not headline scores. Do not use for adversarial attacks against a threat model, or to measure a known criterion.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2250,2251,2252,2253,2254],{"name":2140,"slug":2141,"type":16},{"name":9,"slug":8,"type":16},{"name":2095,"slug":2096,"type":16},{"name":14,"slug":15,"type":16},{"name":2255,"slug":2256,"type":16},"Triage","triage","2026-08-20T03:53:00.423941",27,{"items":2260,"total":2315},[2261,2269,2277,2285,2293,2301,2307],{"slug":140,"name":140,"fn":2104,"description":2105,"org":2262,"tags":2263,"stars":21,"repoUrl":22,"updatedAt":2119},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2264,2265,2266,2267,2268],{"name":2109,"slug":2110,"type":16},{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":2114,"slug":2115,"type":16},{"name":2117,"slug":2118,"type":16},{"slug":2121,"name":2121,"fn":2122,"description":2123,"org":2270,"tags":2271,"stars":21,"repoUrl":22,"updatedAt":2133},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2272,2273,2274,2275,2276],{"name":2109,"slug":2110,"type":16},{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":2114,"slug":2115,"type":16},{"name":2131,"slug":2132,"type":16},{"slug":468,"name":468,"fn":2135,"description":2136,"org":2278,"tags":2279,"stars":21,"repoUrl":22,"updatedAt":2148},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2280,2281,2282,2283,2284],{"name":2140,"slug":2141,"type":16},{"name":9,"slug":8,"type":16},{"name":2144,"slug":2145,"type":16},{"name":14,"slug":15,"type":16},{"name":2114,"slug":2115,"type":16},{"slug":2150,"name":2150,"fn":2151,"description":2152,"org":2286,"tags":2287,"stars":21,"repoUrl":22,"updatedAt":2164},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2288,2289,2290,2291,2292],{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":2114,"slug":2115,"type":16},{"name":2159,"slug":2160,"type":16},{"name":2162,"slug":2163,"type":16},{"slug":2166,"name":2166,"fn":2167,"description":2168,"org":2294,"tags":2295,"stars":21,"repoUrl":22,"updatedAt":2178},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2296,2297,2298,2299,2300],{"name":2140,"slug":2141,"type":16},{"name":9,"slug":8,"type":16},{"name":2174,"slug":2175,"type":16},{"name":14,"slug":15,"type":16},{"name":2114,"slug":2115,"type":16},{"slug":2180,"name":2180,"fn":2181,"description":2182,"org":2302,"tags":2303,"stars":21,"repoUrl":22,"updatedAt":2190},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2304,2305,2306],{"name":9,"slug":8,"type":16},{"name":2187,"slug":2188,"type":16},{"name":14,"slug":15,"type":16},{"slug":839,"name":839,"fn":2192,"description":2193,"org":2308,"tags":2309,"stars":21,"repoUrl":22,"updatedAt":2203},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2310,2311,2312,2313,2314],{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":2199,"slug":2200,"type":16},{"name":2114,"slug":2115,"type":16},{"name":2162,"slug":2163,"type":16},24]