[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-braintrust-agent-auto-improvement":3,"mdc--ctc81v-key":34,"related-repo-braintrust-agent-auto-improvement":2906,"related-org-braintrust-agent-auto-improvement":2915},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":24,"repoUrl":25,"updatedAt":26,"license":27,"forks":28,"topics":29,"repo":30,"sourceUrl":32,"mdContent":33},"agent-auto-improvement","improve Braintrust agents with production traces","Improve any Braintrust-instrumented agent by turning production traces into an offline eval loop: mine bad traces, build a failure taxonomy, capture cases in a remote Braintrust dataset, write scorers and an eval file, iterate on prompts\u002Ftools, and only then push online scorers. Use when the user wants to write evals for their agent, fix recurring production failures, or set up an auto-improvement loop for an LLM app.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"braintrust","Braintrust","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fbraintrust.png","braintrustdata",[13,17,20,23],{"name":14,"slug":15,"type":16},"LLM","llm","tag",{"name":18,"slug":19,"type":16},"Evals","evals",{"name":21,"slug":22,"type":16},"Agents","agents",{"name":9,"slug":8,"type":16},1,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Fbraintrust-skills","2026-07-12T08:36:16.200552",null,0,[],{"repoUrl":25,"stars":24,"forks":28,"topics":31,"description":27},[],"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Fbraintrust-skills\u002Ftree\u002FHEAD\u002Fskills\u002Fagent-auto-improvement","---\nname: agent-auto-improvement\ndescription: \"Improve any Braintrust-instrumented agent by turning production traces into an offline eval loop: mine bad traces, build a failure taxonomy, capture cases in a remote Braintrust dataset, write scorers and an eval file, iterate on prompts\u002Ftools, and only then push online scorers. Use when the user wants to write evals for their agent, fix recurring production failures, or set up an auto-improvement loop for an LLM app.\"\n---\n\n# Agent auto-improvement with Braintrust evals\n\n## Goal\n\nUse real production traces to improve a specific behavior of an agent or LLM app. The source of truth is traced production data and a remote Braintrust dataset, not hand-written examples or hardcoded local JSON.\n\nThe loop is:\n\n1. Pick one narrow, scoreable behavior to improve.\n2. Inspect bad production traces.\n3. Categorize what is wrong into a failure taxonomy.\n4. Capture those cases in a remote Braintrust dataset.\n5. Write scorers that codify the taxonomy.\n6. Run an offline eval against the dataset.\n7. Improve the system prompt, tool definitions, or runtime logic.\n8. Rerun the eval and compare experiments.\n9. Only after the offline loop is healthy, add online scorers and automations.\n\n## Prerequisites\n\n- The app is instrumented with the Braintrust SDK and logging traces to a project. If not, run `bt setup instrument` first.\n- The `bt` CLI is installed and authenticated (`bt setup`, or `BRAINTRUST_API_KEY` in CI).\n- SQL reference: `https:\u002F\u002Fwww.braintrust.dev\u002Fdocs\u002Freference\u002Fsql` (also prefetched at `.bt\u002Fskills\u002Fdocs\u002Freference\u002Fsql.md` if `bt setup skills` ran).\n\n## Workflow\n\n### 1. Pick the target behavior and verify context\n\nDo not try to eval \"the whole agent\" at once. Pick one decision point that is observable in traces and cheap to replay offline, for example:\n\n- the first tool call the agent makes for a request\n- the arguments passed to one specific tool\n- a routing or classification decision\n- the final response for a single-turn task\n\nFind where that behavior lives in the repo: the system prompt, the tool schema\u002Fdescription, and the runtime code that executes it. These are the files the loop will improve.\n\nThen confirm `bt` points at the right org and project, and capture the project ID — SQL table functions like `project_logs(...)` require IDs, not names:\n\n```bash\nbt status --json\nbt projects list --json\n```\n\n### 2. Inspect bad production traces\n\nStart with trace-derived evidence, not prompt intuition. Query for failures with `bt sql`, scoped to the spans that capture the target behavior. Every query needs a time-range filter on `created` (the query linter blocks unscoped queries), and queries that project large fields like `input`\u002F`output` need `LIMIT 100` or less:\n\n```bash\n# Errors on a specific tool's spans\nbt sql \"SELECT id, root_span_id, input, output, error \\\n  FROM project_logs('\u003Cproject-id>') \\\n  WHERE created > now() - interval 30 day \\\n    AND span_attributes.name = '\u003Ctool-or-span-name>' AND error IS NOT NULL \\\n  LIMIT 100\" --json\n\n# Low-scoring or negative-feedback traces\nbt sql \"SELECT id, root_span_id, input, output, scores \\\n  FROM project_logs('\u003Cproject-id>') \\\n  WHERE created > now() - interval 30 day AND scores['\u003Cscore-name>'] \u003C 0.5 \\\n  LIMIT 100\" --json\n\n# Slow spans (duration in seconds)\nbt sql \"SELECT id, root_span_id, metrics.duration AS duration \\\n  FROM project_logs('\u003Cproject-id>') \\\n  WHERE created > now() - interval 30 day AND metrics.duration > 20 \\\n  ORDER BY duration DESC LIMIT 100\" --json\n```\n\nIf the project has no scores and the `error` field is never set, failures are usually latent in span contents instead: search tool span `output` payloads for failure signatures (non-zero exit codes, \"not found\", rejection messages), and look for behavioral patterns like retry loops, repeated identical calls, or a cheaper tool that should have been used. Aggregate by `span_attributes.name` first to see where the volume is.\n\nUse `bt view trace --object-ref project_logs:\u003Cproject-id> --trace-id \u003Croot-span-id>` to read full traces, and `bt view span --id \u003Crow-id>` for full payloads. Read enough individual failures to understand them concretely before generalizing.\n\n### 3. Build a failure taxonomy\n\nGroup what you saw into named categories, and separate them into three buckets:\n\n- **Hard failures** the system should prevent: wrong tool chosen, malformed or over-broad arguments, ignoring available context, violating an explicit instruction, wrong output format.\n- **Soft inefficiencies** that are undesirable but not wrong: a working-but-wasteful query, an unnecessarily verbose answer, an avoidable retry.\n- **Non-quality failures** to exclude: network errors, backend outages, upstream bugs. These are real but not fixable by prompt or tool changes.\n\nFocus the loop on hard failures first. Write the taxonomy down (category id, description, how to detect it) — it becomes the spec for your scorers.\n\n### 4. Capture cases in a remote dataset\n\nTurn real failing traces into a remote dataset using a dataset pipeline. Each row should contain everything needed to replay the decision offline:\n\n- `input`: the captured context at the decision point (prior messages, available state)\n- `expected` (optional): a known-good output, if one exists\n- `metadata`: the baseline production output and its failure categories, plus the source trace id\n\n```typescript pipeline.ts\nimport { DatasetPipeline } from \"braintrust\";\n\nDatasetPipeline({\n  source: {\n    projectName: \"\u003Cproject-name>\",\n    filter:\n      \"span_attributes.name = '\u003Ctool-or-span-name>' AND error IS NOT NULL\",\n    scope: \"span\",\n  },\n  transform: ({ id, input, output, metadata, trace }) => ({\n    input,\n    metadata: {\n      source_span_id: id,\n      baseline_output: output,\n      baseline_categories: categorize(output),\n    },\n  }),\n  target: {\n    projectName: \"\u003Cproject-name>\",\n    datasetName: \"\u003Cagent>-\u003Cbehavior>-offline\",\n  },\n});\n```\n\n```bash\nbt datasets pipeline run .\u002Fpipeline.ts --limit 200 --window 30d\n```\n\nThe pipeline runner needs `tsx` (or `vite-node`) installed as a local dev dependency for TypeScript pipelines; pass `--runner tsx` if auto-detection fails.\n\nImplement `categorize` from the taxonomy in step 3 — it is shared between the pipeline and the scorers. Refresh the dataset (rerun the pipeline) whenever it feels stale, too small, or skewed toward one category. Then verify the dataset actually contains the failures you want to attack:\n\n```bash\nbt datasets view \u003Cagent>-\u003Cbehavior>-offline --limit 50\nbt sql \"SELECT metadata.primary_category AS cat, count(1) AS n \\\n  FROM dataset('\u003Cdataset-id>') \\\n  WHERE created > now() - interval 1 day \\\n  GROUP BY cat ORDER BY n DESC\" --json\n```\n\n### 5. Write scorers that codify the taxonomy\n\nPrefer deterministic code scorers for hard categories — they are free, fast, and stable across runs. Use LLM-as-a-judge (e.g. from `autoevals`) only for qualities a deterministic check cannot capture, and keep judges few and focused.\n\nThe scorer set that makes this loop work:\n\n- **Hard quality** (north star): binary, `1` only when the output has zero hard failure categories.\n- **Issue count**: how many hard categories the output still triggers; gives gradient between 0 and 1.\n- **Beats baseline**: whether the generated output has fewer issues than the captured production baseline in `metadata.baseline_output`.\n- **Avoids primary issue**: whether the output avoids the case's main captured failure category.\n- **Soft efficiency**: weighted `[0,1]` score over the soft-inefficiency categories.\n\n```typescript scorers.ts\nimport { categorize, HARD_CATEGORIES } from \".\u002Fcategorize\";\n\nexport function hardQuality({ output }) {\n  const issues = categorize(output).filter((c) => HARD_CATEGORIES.has(c));\n  return {\n    name: \"hard_quality\",\n    score: issues.length === 0 ? 1 : 0,\n    metadata: { issues },\n  };\n}\n```\n\nUnit-test the categorizer against known-bad and known-good outputs before trusting eval scores built on it.\n\n### 6. Write the offline eval\n\nThe task function replays the captured context against the current system and returns the target behavior's output (for tool-call behaviors, capture the first relevant tool call rather than running the tool):\n\n```typescript first-step.eval.ts\nimport { Eval, initDataset } from \"braintrust\";\nimport {\n  hardQuality,\n  issueCount,\n  beatsBaseline,\n  softEfficiency,\n} from \".\u002Fscorers\";\n\nEval(\"\u003Cproject-name>\", {\n  experimentName: process.env.EVAL_EXPERIMENT,\n  data: initDataset({\n    project: \"\u003Cproject-name>\",\n    dataset: \"\u003Cagent>-\u003Cbehavior>-offline\",\n    useOutputAsExpected: false,\n  }),\n  task: async (input) => runAgentDecision(input),\n  scores: [hardQuality, issueCount, beatsBaseline, softEfficiency],\n});\n```\n\n`runAgentDecision` must call the same prompt, tools, and model configuration as production — import them from the app code rather than copying them, so prompt improvements are picked up automatically.\n\nTwo practical replay issues:\n\n- Captured message histories often start mid-conversation. Normalize them before sending to the model: drop `tool_result` blocks whose matching `tool_use` was trimmed away, and start the history on a user message. Otherwise the provider API rejects the request.\n- If no model provider key is available, route the task's LLM calls through the Braintrust AI proxy with `BRAINTRUST_API_KEY`.\n\n### 7. Run the eval\n\nBounded smoke run first, full run once it is directionally better:\n\n```bash\nbt eval --first 20 first-step.eval.ts   # bounded, non-final\nbt eval first-step.eval.ts              # full dataset, final\n```\n\nNote the experiment name printed in the summary, then resolve its ID — `experiment(...)` requires the ID, not the name:\n\n```bash\nbt experiments list --json\nbt sql \"SELECT root_span_id, scores \\\n  FROM experiment('\u003Cexperiment-id>') \\\n  WHERE created > now() - interval 1 day \\\n    AND span_attributes.type = 'score' AND scores['hard_quality'] \u003C 1\" --json\n```\n\n### 8. Read the scores correctly\n\nPriority order:\n\n1. Keep the task producing the target behavior at all (it must not stop calling the tool, answering, etc.).\n2. Reduce the hard issue count.\n3. Raise the binary hard-quality rate.\n4. Then tighten soft efficiency.\n\nDo not trade away the hard score to optimize the soft score.\n\n### 9. Improve the system, not the case\n\nPrefer fixes that generalize across many traces:\n\n- **System prompt**: tighten guidance for the target behavior; add concrete bad-vs-better examples drawn from real failures; resolve the ambiguities the taxonomy exposed.\n- **Tool definitions**: improve tool descriptions, schemas, and parameter docs so the model cannot misread them.\n- **Runtime**: validate, normalize, or reject bad outputs in code when the contract should be strict rather than suggested.\n- **Categorizer\u002Fscorers**: keep them aligned with the guidance — when you add a prompt rule, add or adjust the category that detects violations of it.\n\nIf a fix only helps one or two dataset rows, it is overfitting; push on repeated categories instead.\n\n### 10. Iterate\n\nAfter every meaningful change:\n\n1. Rerun the bounded eval.\n2. Compare against the previous best experiment (the run summary shows the diff; the experiment comparison UI shows per-case regressions).\n3. If directionally better, rerun the full dataset. The full dataset is the real check.\n\nWhen new production failures appear later: refresh the dataset from the pipeline, rerun, keep iterating.\n\n### 11. Graduate to online scoring\n\nOnly after the offline loop is healthy:\n\n1. Push the same scorer logic as project scorers with `bt functions push`, keeping it aligned with the offline categorizer.\n2. Configure online scoring or automations on production logs so regressions surface as scored traces.\n3. Feed newly flagged traces back into the dataset via the pipeline.\n\nUntil then, the offline eval is the primary optimization surface.\n\n## Guardrails\n\n- Do not invent placeholder cases or fabricate expected outputs; every dataset row must come from a real trace or an explicitly user-provided example.\n- Do not assume the current dataset is representative; inspect it and refresh it.\n- Do not optimize around one or two traces. Push on repeated categories.\n- Do not let scorer logic drift from prompt guidance — they encode the same contract.\n- Do not eval against live mutable data as the primary loop; the remote dataset is the frozen surface.\n- Do not start with LLM judges when a deterministic check exists for the same category.\n\n## Expected outcome\n\nA good iteration improves the target behavior across many real traced cases by:\n\n- reducing hard failure categories\n- increasing the clean-output rate\n- beating the captured production baselines\n- trimming soft inefficiencies without sacrificing hard quality\n\nIf a change improves the bounded run but regresses the full dataset, keep iterating — the full dataset is the real check.\n",{"data":35,"body":36},{"name":4,"description":6},{"type":37,"children":38},"root",[39,48,55,61,66,116,122,198,204,211,216,239,244,264,318,324,369,640,668,689,695,700,734,739,745,750,785,1284,1339,1368,1381,1526,1532,1545,1550,1627,1959,1964,1970,1975,2424,2435,2440,2475,2481,2486,2545,2558,2648,2654,2659,2682,2687,2693,2698,2741,2746,2752,2757,2775,2780,2786,2791,2817,2822,2828,2861,2867,2872,2895,2900],{"type":40,"tag":41,"props":42,"children":44},"element","h1",{"id":43},"agent-auto-improvement-with-braintrust-evals",[45],{"type":46,"value":47},"text","Agent auto-improvement with Braintrust evals",{"type":40,"tag":49,"props":50,"children":52},"h2",{"id":51},"goal",[53],{"type":46,"value":54},"Goal",{"type":40,"tag":56,"props":57,"children":58},"p",{},[59],{"type":46,"value":60},"Use real production traces to improve a specific behavior of an agent or LLM app. The source of truth is traced production data and a remote Braintrust dataset, not hand-written examples or hardcoded local JSON.",{"type":40,"tag":56,"props":62,"children":63},{},[64],{"type":46,"value":65},"The loop is:",{"type":40,"tag":67,"props":68,"children":69},"ol",{},[70,76,81,86,91,96,101,106,111],{"type":40,"tag":71,"props":72,"children":73},"li",{},[74],{"type":46,"value":75},"Pick one narrow, scoreable behavior to improve.",{"type":40,"tag":71,"props":77,"children":78},{},[79],{"type":46,"value":80},"Inspect bad production traces.",{"type":40,"tag":71,"props":82,"children":83},{},[84],{"type":46,"value":85},"Categorize what is wrong into a failure taxonomy.",{"type":40,"tag":71,"props":87,"children":88},{},[89],{"type":46,"value":90},"Capture those cases in a remote Braintrust dataset.",{"type":40,"tag":71,"props":92,"children":93},{},[94],{"type":46,"value":95},"Write scorers that codify the taxonomy.",{"type":40,"tag":71,"props":97,"children":98},{},[99],{"type":46,"value":100},"Run an offline eval against the dataset.",{"type":40,"tag":71,"props":102,"children":103},{},[104],{"type":46,"value":105},"Improve the system prompt, tool definitions, or runtime logic.",{"type":40,"tag":71,"props":107,"children":108},{},[109],{"type":46,"value":110},"Rerun the eval and compare experiments.",{"type":40,"tag":71,"props":112,"children":113},{},[114],{"type":46,"value":115},"Only after the offline loop is healthy, add online scorers and automations.",{"type":40,"tag":49,"props":117,"children":119},{"id":118},"prerequisites",[120],{"type":46,"value":121},"Prerequisites",{"type":40,"tag":123,"props":124,"children":125},"ul",{},[126,140,169],{"type":40,"tag":71,"props":127,"children":128},{},[129,131,138],{"type":46,"value":130},"The app is instrumented with the Braintrust SDK and logging traces to a project. If not, run ",{"type":40,"tag":132,"props":133,"children":135},"code",{"className":134},[],[136],{"type":46,"value":137},"bt setup instrument",{"type":46,"value":139}," first.",{"type":40,"tag":71,"props":141,"children":142},{},[143,145,151,153,159,161,167],{"type":46,"value":144},"The ",{"type":40,"tag":132,"props":146,"children":148},{"className":147},[],[149],{"type":46,"value":150},"bt",{"type":46,"value":152}," CLI is installed and authenticated (",{"type":40,"tag":132,"props":154,"children":156},{"className":155},[],[157],{"type":46,"value":158},"bt setup",{"type":46,"value":160},", or ",{"type":40,"tag":132,"props":162,"children":164},{"className":163},[],[165],{"type":46,"value":166},"BRAINTRUST_API_KEY",{"type":46,"value":168}," in CI).",{"type":40,"tag":71,"props":170,"children":171},{},[172,174,180,182,188,190,196],{"type":46,"value":173},"SQL reference: ",{"type":40,"tag":132,"props":175,"children":177},{"className":176},[],[178],{"type":46,"value":179},"https:\u002F\u002Fwww.braintrust.dev\u002Fdocs\u002Freference\u002Fsql",{"type":46,"value":181}," (also prefetched at ",{"type":40,"tag":132,"props":183,"children":185},{"className":184},[],[186],{"type":46,"value":187},".bt\u002Fskills\u002Fdocs\u002Freference\u002Fsql.md",{"type":46,"value":189}," if ",{"type":40,"tag":132,"props":191,"children":193},{"className":192},[],[194],{"type":46,"value":195},"bt setup skills",{"type":46,"value":197}," ran).",{"type":40,"tag":49,"props":199,"children":201},{"id":200},"workflow",[202],{"type":46,"value":203},"Workflow",{"type":40,"tag":205,"props":206,"children":208},"h3",{"id":207},"_1-pick-the-target-behavior-and-verify-context",[209],{"type":46,"value":210},"1. Pick the target behavior and verify context",{"type":40,"tag":56,"props":212,"children":213},{},[214],{"type":46,"value":215},"Do not try to eval \"the whole agent\" at once. Pick one decision point that is observable in traces and cheap to replay offline, for example:",{"type":40,"tag":123,"props":217,"children":218},{},[219,224,229,234],{"type":40,"tag":71,"props":220,"children":221},{},[222],{"type":46,"value":223},"the first tool call the agent makes for a request",{"type":40,"tag":71,"props":225,"children":226},{},[227],{"type":46,"value":228},"the arguments passed to one specific tool",{"type":40,"tag":71,"props":230,"children":231},{},[232],{"type":46,"value":233},"a routing or classification decision",{"type":40,"tag":71,"props":235,"children":236},{},[237],{"type":46,"value":238},"the final response for a single-turn task",{"type":40,"tag":56,"props":240,"children":241},{},[242],{"type":46,"value":243},"Find where that behavior lives in the repo: the system prompt, the tool schema\u002Fdescription, and the runtime code that executes it. These are the files the loop will improve.",{"type":40,"tag":56,"props":245,"children":246},{},[247,249,254,256,262],{"type":46,"value":248},"Then confirm ",{"type":40,"tag":132,"props":250,"children":252},{"className":251},[],[253],{"type":46,"value":150},{"type":46,"value":255}," points at the right org and project, and capture the project ID — SQL table functions like ",{"type":40,"tag":132,"props":257,"children":259},{"className":258},[],[260],{"type":46,"value":261},"project_logs(...)",{"type":46,"value":263}," require IDs, not names:",{"type":40,"tag":265,"props":266,"children":271},"pre",{"className":267,"code":268,"language":269,"meta":270,"style":270},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","bt status --json\nbt projects list --json\n","bash","",[272],{"type":40,"tag":132,"props":273,"children":274},{"__ignoreMap":270},[275,296],{"type":40,"tag":276,"props":277,"children":279},"span",{"class":278,"line":24},"line",[280,285,291],{"type":40,"tag":276,"props":281,"children":283},{"style":282},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[284],{"type":46,"value":150},{"type":40,"tag":276,"props":286,"children":288},{"style":287},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[289],{"type":46,"value":290}," status",{"type":40,"tag":276,"props":292,"children":293},{"style":287},[294],{"type":46,"value":295}," --json\n",{"type":40,"tag":276,"props":297,"children":299},{"class":278,"line":298},2,[300,304,309,314],{"type":40,"tag":276,"props":301,"children":302},{"style":282},[303],{"type":46,"value":150},{"type":40,"tag":276,"props":305,"children":306},{"style":287},[307],{"type":46,"value":308}," projects",{"type":40,"tag":276,"props":310,"children":311},{"style":287},[312],{"type":46,"value":313}," list",{"type":40,"tag":276,"props":315,"children":316},{"style":287},[317],{"type":46,"value":295},{"type":40,"tag":205,"props":319,"children":321},{"id":320},"_2-inspect-bad-production-traces",[322],{"type":46,"value":323},"2. Inspect bad production traces",{"type":40,"tag":56,"props":325,"children":326},{},[327,329,335,337,343,345,351,353,359,361,367],{"type":46,"value":328},"Start with trace-derived evidence, not prompt intuition. Query for failures with ",{"type":40,"tag":132,"props":330,"children":332},{"className":331},[],[333],{"type":46,"value":334},"bt sql",{"type":46,"value":336},", scoped to the spans that capture the target behavior. Every query needs a time-range filter on ",{"type":40,"tag":132,"props":338,"children":340},{"className":339},[],[341],{"type":46,"value":342},"created",{"type":46,"value":344}," (the query linter blocks unscoped queries), and queries that project large fields like ",{"type":40,"tag":132,"props":346,"children":348},{"className":347},[],[349],{"type":46,"value":350},"input",{"type":46,"value":352},"\u002F",{"type":40,"tag":132,"props":354,"children":356},{"className":355},[],[357],{"type":46,"value":358},"output",{"type":46,"value":360}," need ",{"type":40,"tag":132,"props":362,"children":364},{"className":363},[],[365],{"type":46,"value":366},"LIMIT 100",{"type":46,"value":368}," or less:",{"type":40,"tag":265,"props":370,"children":372},{"className":267,"code":371,"language":269,"meta":270,"style":270},"# Errors on a specific tool's spans\nbt sql \"SELECT id, root_span_id, input, output, error \\\n  FROM project_logs('\u003Cproject-id>') \\\n  WHERE created > now() - interval 30 day \\\n    AND span_attributes.name = '\u003Ctool-or-span-name>' AND error IS NOT NULL \\\n  LIMIT 100\" --json\n\n# Low-scoring or negative-feedback traces\nbt sql \"SELECT id, root_span_id, input, output, scores \\\n  FROM project_logs('\u003Cproject-id>') \\\n  WHERE created > now() - interval 30 day AND scores['\u003Cscore-name>'] \u003C 0.5 \\\n  LIMIT 100\" --json\n\n# Slow spans (duration in seconds)\nbt sql \"SELECT id, root_span_id, metrics.duration AS duration \\\n  FROM project_logs('\u003Cproject-id>') \\\n  WHERE created > now() - interval 30 day AND metrics.duration > 20 \\\n  ORDER BY duration DESC LIMIT 100\" --json\n",[373],{"type":40,"tag":132,"props":374,"children":375},{"__ignoreMap":270},[376,385,414,427,440,453,471,481,490,515,527,540,556,564,573,598,610,623],{"type":40,"tag":276,"props":377,"children":378},{"class":278,"line":24},[379],{"type":40,"tag":276,"props":380,"children":382},{"style":381},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[383],{"type":46,"value":384},"# Errors on a specific tool's spans\n",{"type":40,"tag":276,"props":386,"children":387},{"class":278,"line":298},[388,392,397,403,408],{"type":40,"tag":276,"props":389,"children":390},{"style":282},[391],{"type":46,"value":150},{"type":40,"tag":276,"props":393,"children":394},{"style":287},[395],{"type":46,"value":396}," sql",{"type":40,"tag":276,"props":398,"children":400},{"style":399},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[401],{"type":46,"value":402}," \"",{"type":40,"tag":276,"props":404,"children":405},{"style":287},[406],{"type":46,"value":407},"SELECT id, root_span_id, input, output, error ",{"type":40,"tag":276,"props":409,"children":411},{"style":410},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[412],{"type":46,"value":413},"\\\n",{"type":40,"tag":276,"props":415,"children":417},{"class":278,"line":416},3,[418,423],{"type":40,"tag":276,"props":419,"children":420},{"style":287},[421],{"type":46,"value":422},"  FROM project_logs('\u003Cproject-id>') ",{"type":40,"tag":276,"props":424,"children":425},{"style":410},[426],{"type":46,"value":413},{"type":40,"tag":276,"props":428,"children":430},{"class":278,"line":429},4,[431,436],{"type":40,"tag":276,"props":432,"children":433},{"style":287},[434],{"type":46,"value":435},"  WHERE created > now() - interval 30 day ",{"type":40,"tag":276,"props":437,"children":438},{"style":410},[439],{"type":46,"value":413},{"type":40,"tag":276,"props":441,"children":443},{"class":278,"line":442},5,[444,449],{"type":40,"tag":276,"props":445,"children":446},{"style":287},[447],{"type":46,"value":448},"    AND span_attributes.name = '\u003Ctool-or-span-name>' AND error IS NOT NULL ",{"type":40,"tag":276,"props":450,"children":451},{"style":410},[452],{"type":46,"value":413},{"type":40,"tag":276,"props":454,"children":456},{"class":278,"line":455},6,[457,462,467],{"type":40,"tag":276,"props":458,"children":459},{"style":287},[460],{"type":46,"value":461},"  LIMIT 100",{"type":40,"tag":276,"props":463,"children":464},{"style":399},[465],{"type":46,"value":466},"\"",{"type":40,"tag":276,"props":468,"children":469},{"style":287},[470],{"type":46,"value":295},{"type":40,"tag":276,"props":472,"children":474},{"class":278,"line":473},7,[475],{"type":40,"tag":276,"props":476,"children":478},{"emptyLinePlaceholder":477},true,[479],{"type":46,"value":480},"\n",{"type":40,"tag":276,"props":482,"children":484},{"class":278,"line":483},8,[485],{"type":40,"tag":276,"props":486,"children":487},{"style":381},[488],{"type":46,"value":489},"# Low-scoring or negative-feedback traces\n",{"type":40,"tag":276,"props":491,"children":493},{"class":278,"line":492},9,[494,498,502,506,511],{"type":40,"tag":276,"props":495,"children":496},{"style":282},[497],{"type":46,"value":150},{"type":40,"tag":276,"props":499,"children":500},{"style":287},[501],{"type":46,"value":396},{"type":40,"tag":276,"props":503,"children":504},{"style":399},[505],{"type":46,"value":402},{"type":40,"tag":276,"props":507,"children":508},{"style":287},[509],{"type":46,"value":510},"SELECT id, root_span_id, input, output, scores ",{"type":40,"tag":276,"props":512,"children":513},{"style":410},[514],{"type":46,"value":413},{"type":40,"tag":276,"props":516,"children":518},{"class":278,"line":517},10,[519,523],{"type":40,"tag":276,"props":520,"children":521},{"style":287},[522],{"type":46,"value":422},{"type":40,"tag":276,"props":524,"children":525},{"style":410},[526],{"type":46,"value":413},{"type":40,"tag":276,"props":528,"children":530},{"class":278,"line":529},11,[531,536],{"type":40,"tag":276,"props":532,"children":533},{"style":287},[534],{"type":46,"value":535},"  WHERE created > now() - interval 30 day AND scores['\u003Cscore-name>'] \u003C 0.5 ",{"type":40,"tag":276,"props":537,"children":538},{"style":410},[539],{"type":46,"value":413},{"type":40,"tag":276,"props":541,"children":543},{"class":278,"line":542},12,[544,548,552],{"type":40,"tag":276,"props":545,"children":546},{"style":287},[547],{"type":46,"value":461},{"type":40,"tag":276,"props":549,"children":550},{"style":399},[551],{"type":46,"value":466},{"type":40,"tag":276,"props":553,"children":554},{"style":287},[555],{"type":46,"value":295},{"type":40,"tag":276,"props":557,"children":559},{"class":278,"line":558},13,[560],{"type":40,"tag":276,"props":561,"children":562},{"emptyLinePlaceholder":477},[563],{"type":46,"value":480},{"type":40,"tag":276,"props":565,"children":567},{"class":278,"line":566},14,[568],{"type":40,"tag":276,"props":569,"children":570},{"style":381},[571],{"type":46,"value":572},"# Slow spans (duration in seconds)\n",{"type":40,"tag":276,"props":574,"children":576},{"class":278,"line":575},15,[577,581,585,589,594],{"type":40,"tag":276,"props":578,"children":579},{"style":282},[580],{"type":46,"value":150},{"type":40,"tag":276,"props":582,"children":583},{"style":287},[584],{"type":46,"value":396},{"type":40,"tag":276,"props":586,"children":587},{"style":399},[588],{"type":46,"value":402},{"type":40,"tag":276,"props":590,"children":591},{"style":287},[592],{"type":46,"value":593},"SELECT id, root_span_id, metrics.duration AS duration ",{"type":40,"tag":276,"props":595,"children":596},{"style":410},[597],{"type":46,"value":413},{"type":40,"tag":276,"props":599,"children":601},{"class":278,"line":600},16,[602,606],{"type":40,"tag":276,"props":603,"children":604},{"style":287},[605],{"type":46,"value":422},{"type":40,"tag":276,"props":607,"children":608},{"style":410},[609],{"type":46,"value":413},{"type":40,"tag":276,"props":611,"children":613},{"class":278,"line":612},17,[614,619],{"type":40,"tag":276,"props":615,"children":616},{"style":287},[617],{"type":46,"value":618},"  WHERE created > now() - interval 30 day AND metrics.duration > 20 ",{"type":40,"tag":276,"props":620,"children":621},{"style":410},[622],{"type":46,"value":413},{"type":40,"tag":276,"props":624,"children":626},{"class":278,"line":625},18,[627,632,636],{"type":40,"tag":276,"props":628,"children":629},{"style":287},[630],{"type":46,"value":631},"  ORDER BY duration DESC LIMIT 100",{"type":40,"tag":276,"props":633,"children":634},{"style":399},[635],{"type":46,"value":466},{"type":40,"tag":276,"props":637,"children":638},{"style":287},[639],{"type":46,"value":295},{"type":40,"tag":56,"props":641,"children":642},{},[643,645,651,653,658,660,666],{"type":46,"value":644},"If the project has no scores and the ",{"type":40,"tag":132,"props":646,"children":648},{"className":647},[],[649],{"type":46,"value":650},"error",{"type":46,"value":652}," field is never set, failures are usually latent in span contents instead: search tool span ",{"type":40,"tag":132,"props":654,"children":656},{"className":655},[],[657],{"type":46,"value":358},{"type":46,"value":659}," payloads for failure signatures (non-zero exit codes, \"not found\", rejection messages), and look for behavioral patterns like retry loops, repeated identical calls, or a cheaper tool that should have been used. Aggregate by ",{"type":40,"tag":132,"props":661,"children":663},{"className":662},[],[664],{"type":46,"value":665},"span_attributes.name",{"type":46,"value":667}," first to see where the volume is.",{"type":40,"tag":56,"props":669,"children":670},{},[671,673,679,681,687],{"type":46,"value":672},"Use ",{"type":40,"tag":132,"props":674,"children":676},{"className":675},[],[677],{"type":46,"value":678},"bt view trace --object-ref project_logs:\u003Cproject-id> --trace-id \u003Croot-span-id>",{"type":46,"value":680}," to read full traces, and ",{"type":40,"tag":132,"props":682,"children":684},{"className":683},[],[685],{"type":46,"value":686},"bt view span --id \u003Crow-id>",{"type":46,"value":688}," for full payloads. Read enough individual failures to understand them concretely before generalizing.",{"type":40,"tag":205,"props":690,"children":692},{"id":691},"_3-build-a-failure-taxonomy",[693],{"type":46,"value":694},"3. Build a failure taxonomy",{"type":40,"tag":56,"props":696,"children":697},{},[698],{"type":46,"value":699},"Group what you saw into named categories, and separate them into three buckets:",{"type":40,"tag":123,"props":701,"children":702},{},[703,714,724],{"type":40,"tag":71,"props":704,"children":705},{},[706,712],{"type":40,"tag":707,"props":708,"children":709},"strong",{},[710],{"type":46,"value":711},"Hard failures",{"type":46,"value":713}," the system should prevent: wrong tool chosen, malformed or over-broad arguments, ignoring available context, violating an explicit instruction, wrong output format.",{"type":40,"tag":71,"props":715,"children":716},{},[717,722],{"type":40,"tag":707,"props":718,"children":719},{},[720],{"type":46,"value":721},"Soft inefficiencies",{"type":46,"value":723}," that are undesirable but not wrong: a working-but-wasteful query, an unnecessarily verbose answer, an avoidable retry.",{"type":40,"tag":71,"props":725,"children":726},{},[727,732],{"type":40,"tag":707,"props":728,"children":729},{},[730],{"type":46,"value":731},"Non-quality failures",{"type":46,"value":733}," to exclude: network errors, backend outages, upstream bugs. These are real but not fixable by prompt or tool changes.",{"type":40,"tag":56,"props":735,"children":736},{},[737],{"type":46,"value":738},"Focus the loop on hard failures first. Write the taxonomy down (category id, description, how to detect it) — it becomes the spec for your scorers.",{"type":40,"tag":205,"props":740,"children":742},{"id":741},"_4-capture-cases-in-a-remote-dataset",[743],{"type":46,"value":744},"4. Capture cases in a remote dataset",{"type":40,"tag":56,"props":746,"children":747},{},[748],{"type":46,"value":749},"Turn real failing traces into a remote dataset using a dataset pipeline. Each row should contain everything needed to replay the decision offline:",{"type":40,"tag":123,"props":751,"children":752},{},[753,763,774],{"type":40,"tag":71,"props":754,"children":755},{},[756,761],{"type":40,"tag":132,"props":757,"children":759},{"className":758},[],[760],{"type":46,"value":350},{"type":46,"value":762},": the captured context at the decision point (prior messages, available state)",{"type":40,"tag":71,"props":764,"children":765},{},[766,772],{"type":40,"tag":132,"props":767,"children":769},{"className":768},[],[770],{"type":46,"value":771},"expected",{"type":46,"value":773}," (optional): a known-good output, if one exists",{"type":40,"tag":71,"props":775,"children":776},{},[777,783],{"type":40,"tag":132,"props":778,"children":780},{"className":779},[],[781],{"type":46,"value":782},"metadata",{"type":46,"value":784},": the baseline production output and its failure categories, plus the source trace id",{"type":40,"tag":265,"props":786,"children":791},{"className":787,"code":788,"language":789,"meta":790,"style":270},"language-typescript shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","import { DatasetPipeline } from \"braintrust\";\n\nDatasetPipeline({\n  source: {\n    projectName: \"\u003Cproject-name>\",\n    filter:\n      \"span_attributes.name = '\u003Ctool-or-span-name>' AND error IS NOT NULL\",\n    scope: \"span\",\n  },\n  transform: ({ id, input, output, metadata, trace }) => ({\n    input,\n    metadata: {\n      source_span_id: id,\n      baseline_output: output,\n      baseline_categories: categorize(output),\n    },\n  }),\n  target: {\n    projectName: \"\u003Cproject-name>\",\n    datasetName: \"\u003Cagent>-\u003Cbehavior>-offline\",\n  },\n});\n","typescript","pipeline.ts",[792],{"type":40,"tag":132,"props":793,"children":794},{"__ignoreMap":270},[795,841,848,867,886,916,929,950,978,986,1066,1078,1094,1114,1134,1160,1168,1185,1201,1229,1259,1267],{"type":40,"tag":276,"props":796,"children":797},{"class":278,"line":24},[798,804,809,814,819,824,828,832,836],{"type":40,"tag":276,"props":799,"children":801},{"style":800},"--shiki-light:#39ADB5;--shiki-light-font-style:italic;--shiki-default:#89DDFF;--shiki-default-font-style:italic;--shiki-dark:#89DDFF;--shiki-dark-font-style:italic",[802],{"type":46,"value":803},"import",{"type":40,"tag":276,"props":805,"children":806},{"style":399},[807],{"type":46,"value":808}," {",{"type":40,"tag":276,"props":810,"children":811},{"style":410},[812],{"type":46,"value":813}," DatasetPipeline",{"type":40,"tag":276,"props":815,"children":816},{"style":399},[817],{"type":46,"value":818}," }",{"type":40,"tag":276,"props":820,"children":821},{"style":800},[822],{"type":46,"value":823}," from",{"type":40,"tag":276,"props":825,"children":826},{"style":399},[827],{"type":46,"value":402},{"type":40,"tag":276,"props":829,"children":830},{"style":287},[831],{"type":46,"value":8},{"type":40,"tag":276,"props":833,"children":834},{"style":399},[835],{"type":46,"value":466},{"type":40,"tag":276,"props":837,"children":838},{"style":399},[839],{"type":46,"value":840},";\n",{"type":40,"tag":276,"props":842,"children":843},{"class":278,"line":298},[844],{"type":40,"tag":276,"props":845,"children":846},{"emptyLinePlaceholder":477},[847],{"type":46,"value":480},{"type":40,"tag":276,"props":849,"children":850},{"class":278,"line":416},[851,857,862],{"type":40,"tag":276,"props":852,"children":854},{"style":853},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[855],{"type":46,"value":856},"DatasetPipeline",{"type":40,"tag":276,"props":858,"children":859},{"style":410},[860],{"type":46,"value":861},"(",{"type":40,"tag":276,"props":863,"children":864},{"style":399},[865],{"type":46,"value":866},"{\n",{"type":40,"tag":276,"props":868,"children":869},{"class":278,"line":429},[870,876,881],{"type":40,"tag":276,"props":871,"children":873},{"style":872},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[874],{"type":46,"value":875},"  source",{"type":40,"tag":276,"props":877,"children":878},{"style":399},[879],{"type":46,"value":880},":",{"type":40,"tag":276,"props":882,"children":883},{"style":399},[884],{"type":46,"value":885}," {\n",{"type":40,"tag":276,"props":887,"children":888},{"class":278,"line":442},[889,894,898,902,907,911],{"type":40,"tag":276,"props":890,"children":891},{"style":872},[892],{"type":46,"value":893},"    projectName",{"type":40,"tag":276,"props":895,"children":896},{"style":399},[897],{"type":46,"value":880},{"type":40,"tag":276,"props":899,"children":900},{"style":399},[901],{"type":46,"value":402},{"type":40,"tag":276,"props":903,"children":904},{"style":287},[905],{"type":46,"value":906},"\u003Cproject-name>",{"type":40,"tag":276,"props":908,"children":909},{"style":399},[910],{"type":46,"value":466},{"type":40,"tag":276,"props":912,"children":913},{"style":399},[914],{"type":46,"value":915},",\n",{"type":40,"tag":276,"props":917,"children":918},{"class":278,"line":455},[919,924],{"type":40,"tag":276,"props":920,"children":921},{"style":872},[922],{"type":46,"value":923},"    filter",{"type":40,"tag":276,"props":925,"children":926},{"style":399},[927],{"type":46,"value":928},":\n",{"type":40,"tag":276,"props":930,"children":931},{"class":278,"line":473},[932,937,942,946],{"type":40,"tag":276,"props":933,"children":934},{"style":399},[935],{"type":46,"value":936},"      \"",{"type":40,"tag":276,"props":938,"children":939},{"style":287},[940],{"type":46,"value":941},"span_attributes.name = '\u003Ctool-or-span-name>' AND error IS NOT NULL",{"type":40,"tag":276,"props":943,"children":944},{"style":399},[945],{"type":46,"value":466},{"type":40,"tag":276,"props":947,"children":948},{"style":399},[949],{"type":46,"value":915},{"type":40,"tag":276,"props":951,"children":952},{"class":278,"line":483},[953,958,962,966,970,974],{"type":40,"tag":276,"props":954,"children":955},{"style":872},[956],{"type":46,"value":957},"    scope",{"type":40,"tag":276,"props":959,"children":960},{"style":399},[961],{"type":46,"value":880},{"type":40,"tag":276,"props":963,"children":964},{"style":399},[965],{"type":46,"value":402},{"type":40,"tag":276,"props":967,"children":968},{"style":287},[969],{"type":46,"value":276},{"type":40,"tag":276,"props":971,"children":972},{"style":399},[973],{"type":46,"value":466},{"type":40,"tag":276,"props":975,"children":976},{"style":399},[977],{"type":46,"value":915},{"type":40,"tag":276,"props":979,"children":980},{"class":278,"line":492},[981],{"type":40,"tag":276,"props":982,"children":983},{"style":399},[984],{"type":46,"value":985},"  },\n",{"type":40,"tag":276,"props":987,"children":988},{"class":278,"line":517},[989,994,998,1003,1009,1014,1019,1023,1028,1032,1037,1041,1046,1051,1057,1062],{"type":40,"tag":276,"props":990,"children":991},{"style":853},[992],{"type":46,"value":993},"  transform",{"type":40,"tag":276,"props":995,"children":996},{"style":399},[997],{"type":46,"value":880},{"type":40,"tag":276,"props":999,"children":1000},{"style":399},[1001],{"type":46,"value":1002}," ({",{"type":40,"tag":276,"props":1004,"children":1006},{"style":1005},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#EEFFFF;--shiki-default-font-style:italic;--shiki-dark:#BABED8;--shiki-dark-font-style:italic",[1007],{"type":46,"value":1008}," id",{"type":40,"tag":276,"props":1010,"children":1011},{"style":399},[1012],{"type":46,"value":1013},",",{"type":40,"tag":276,"props":1015,"children":1016},{"style":1005},[1017],{"type":46,"value":1018}," input",{"type":40,"tag":276,"props":1020,"children":1021},{"style":399},[1022],{"type":46,"value":1013},{"type":40,"tag":276,"props":1024,"children":1025},{"style":1005},[1026],{"type":46,"value":1027}," output",{"type":40,"tag":276,"props":1029,"children":1030},{"style":399},[1031],{"type":46,"value":1013},{"type":40,"tag":276,"props":1033,"children":1034},{"style":1005},[1035],{"type":46,"value":1036}," metadata",{"type":40,"tag":276,"props":1038,"children":1039},{"style":399},[1040],{"type":46,"value":1013},{"type":40,"tag":276,"props":1042,"children":1043},{"style":1005},[1044],{"type":46,"value":1045}," trace",{"type":40,"tag":276,"props":1047,"children":1048},{"style":399},[1049],{"type":46,"value":1050}," })",{"type":40,"tag":276,"props":1052,"children":1054},{"style":1053},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1055],{"type":46,"value":1056}," =>",{"type":40,"tag":276,"props":1058,"children":1059},{"style":410},[1060],{"type":46,"value":1061}," (",{"type":40,"tag":276,"props":1063,"children":1064},{"style":399},[1065],{"type":46,"value":866},{"type":40,"tag":276,"props":1067,"children":1068},{"class":278,"line":529},[1069,1074],{"type":40,"tag":276,"props":1070,"children":1071},{"style":410},[1072],{"type":46,"value":1073},"    input",{"type":40,"tag":276,"props":1075,"children":1076},{"style":399},[1077],{"type":46,"value":915},{"type":40,"tag":276,"props":1079,"children":1080},{"class":278,"line":542},[1081,1086,1090],{"type":40,"tag":276,"props":1082,"children":1083},{"style":872},[1084],{"type":46,"value":1085},"    metadata",{"type":40,"tag":276,"props":1087,"children":1088},{"style":399},[1089],{"type":46,"value":880},{"type":40,"tag":276,"props":1091,"children":1092},{"style":399},[1093],{"type":46,"value":885},{"type":40,"tag":276,"props":1095,"children":1096},{"class":278,"line":558},[1097,1102,1106,1110],{"type":40,"tag":276,"props":1098,"children":1099},{"style":872},[1100],{"type":46,"value":1101},"      source_span_id",{"type":40,"tag":276,"props":1103,"children":1104},{"style":399},[1105],{"type":46,"value":880},{"type":40,"tag":276,"props":1107,"children":1108},{"style":410},[1109],{"type":46,"value":1008},{"type":40,"tag":276,"props":1111,"children":1112},{"style":399},[1113],{"type":46,"value":915},{"type":40,"tag":276,"props":1115,"children":1116},{"class":278,"line":566},[1117,1122,1126,1130],{"type":40,"tag":276,"props":1118,"children":1119},{"style":872},[1120],{"type":46,"value":1121},"      baseline_output",{"type":40,"tag":276,"props":1123,"children":1124},{"style":399},[1125],{"type":46,"value":880},{"type":40,"tag":276,"props":1127,"children":1128},{"style":410},[1129],{"type":46,"value":1027},{"type":40,"tag":276,"props":1131,"children":1132},{"style":399},[1133],{"type":46,"value":915},{"type":40,"tag":276,"props":1135,"children":1136},{"class":278,"line":575},[1137,1142,1146,1151,1156],{"type":40,"tag":276,"props":1138,"children":1139},{"style":872},[1140],{"type":46,"value":1141},"      baseline_categories",{"type":40,"tag":276,"props":1143,"children":1144},{"style":399},[1145],{"type":46,"value":880},{"type":40,"tag":276,"props":1147,"children":1148},{"style":853},[1149],{"type":46,"value":1150}," categorize",{"type":40,"tag":276,"props":1152,"children":1153},{"style":410},[1154],{"type":46,"value":1155},"(output)",{"type":40,"tag":276,"props":1157,"children":1158},{"style":399},[1159],{"type":46,"value":915},{"type":40,"tag":276,"props":1161,"children":1162},{"class":278,"line":600},[1163],{"type":40,"tag":276,"props":1164,"children":1165},{"style":399},[1166],{"type":46,"value":1167},"    },\n",{"type":40,"tag":276,"props":1169,"children":1170},{"class":278,"line":612},[1171,1176,1181],{"type":40,"tag":276,"props":1172,"children":1173},{"style":399},[1174],{"type":46,"value":1175},"  }",{"type":40,"tag":276,"props":1177,"children":1178},{"style":410},[1179],{"type":46,"value":1180},")",{"type":40,"tag":276,"props":1182,"children":1183},{"style":399},[1184],{"type":46,"value":915},{"type":40,"tag":276,"props":1186,"children":1187},{"class":278,"line":625},[1188,1193,1197],{"type":40,"tag":276,"props":1189,"children":1190},{"style":872},[1191],{"type":46,"value":1192},"  target",{"type":40,"tag":276,"props":1194,"children":1195},{"style":399},[1196],{"type":46,"value":880},{"type":40,"tag":276,"props":1198,"children":1199},{"style":399},[1200],{"type":46,"value":885},{"type":40,"tag":276,"props":1202,"children":1204},{"class":278,"line":1203},19,[1205,1209,1213,1217,1221,1225],{"type":40,"tag":276,"props":1206,"children":1207},{"style":872},[1208],{"type":46,"value":893},{"type":40,"tag":276,"props":1210,"children":1211},{"style":399},[1212],{"type":46,"value":880},{"type":40,"tag":276,"props":1214,"children":1215},{"style":399},[1216],{"type":46,"value":402},{"type":40,"tag":276,"props":1218,"children":1219},{"style":287},[1220],{"type":46,"value":906},{"type":40,"tag":276,"props":1222,"children":1223},{"style":399},[1224],{"type":46,"value":466},{"type":40,"tag":276,"props":1226,"children":1227},{"style":399},[1228],{"type":46,"value":915},{"type":40,"tag":276,"props":1230,"children":1232},{"class":278,"line":1231},20,[1233,1238,1242,1246,1251,1255],{"type":40,"tag":276,"props":1234,"children":1235},{"style":872},[1236],{"type":46,"value":1237},"    datasetName",{"type":40,"tag":276,"props":1239,"children":1240},{"style":399},[1241],{"type":46,"value":880},{"type":40,"tag":276,"props":1243,"children":1244},{"style":399},[1245],{"type":46,"value":402},{"type":40,"tag":276,"props":1247,"children":1248},{"style":287},[1249],{"type":46,"value":1250},"\u003Cagent>-\u003Cbehavior>-offline",{"type":40,"tag":276,"props":1252,"children":1253},{"style":399},[1254],{"type":46,"value":466},{"type":40,"tag":276,"props":1256,"children":1257},{"style":399},[1258],{"type":46,"value":915},{"type":40,"tag":276,"props":1260,"children":1262},{"class":278,"line":1261},21,[1263],{"type":40,"tag":276,"props":1264,"children":1265},{"style":399},[1266],{"type":46,"value":985},{"type":40,"tag":276,"props":1268,"children":1270},{"class":278,"line":1269},22,[1271,1276,1280],{"type":40,"tag":276,"props":1272,"children":1273},{"style":399},[1274],{"type":46,"value":1275},"}",{"type":40,"tag":276,"props":1277,"children":1278},{"style":410},[1279],{"type":46,"value":1180},{"type":40,"tag":276,"props":1281,"children":1282},{"style":399},[1283],{"type":46,"value":840},{"type":40,"tag":265,"props":1285,"children":1287},{"className":267,"code":1286,"language":269,"meta":270,"style":270},"bt datasets pipeline run .\u002Fpipeline.ts --limit 200 --window 30d\n",[1288],{"type":40,"tag":132,"props":1289,"children":1290},{"__ignoreMap":270},[1291],{"type":40,"tag":276,"props":1292,"children":1293},{"class":278,"line":24},[1294,1298,1303,1308,1313,1318,1323,1329,1334],{"type":40,"tag":276,"props":1295,"children":1296},{"style":282},[1297],{"type":46,"value":150},{"type":40,"tag":276,"props":1299,"children":1300},{"style":287},[1301],{"type":46,"value":1302}," datasets",{"type":40,"tag":276,"props":1304,"children":1305},{"style":287},[1306],{"type":46,"value":1307}," pipeline",{"type":40,"tag":276,"props":1309,"children":1310},{"style":287},[1311],{"type":46,"value":1312}," run",{"type":40,"tag":276,"props":1314,"children":1315},{"style":287},[1316],{"type":46,"value":1317}," .\u002Fpipeline.ts",{"type":40,"tag":276,"props":1319,"children":1320},{"style":287},[1321],{"type":46,"value":1322}," --limit",{"type":40,"tag":276,"props":1324,"children":1326},{"style":1325},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1327],{"type":46,"value":1328}," 200",{"type":40,"tag":276,"props":1330,"children":1331},{"style":287},[1332],{"type":46,"value":1333}," --window",{"type":40,"tag":276,"props":1335,"children":1336},{"style":287},[1337],{"type":46,"value":1338}," 30d\n",{"type":40,"tag":56,"props":1340,"children":1341},{},[1342,1344,1350,1352,1358,1360,1366],{"type":46,"value":1343},"The pipeline runner needs ",{"type":40,"tag":132,"props":1345,"children":1347},{"className":1346},[],[1348],{"type":46,"value":1349},"tsx",{"type":46,"value":1351}," (or ",{"type":40,"tag":132,"props":1353,"children":1355},{"className":1354},[],[1356],{"type":46,"value":1357},"vite-node",{"type":46,"value":1359},") installed as a local dev dependency for TypeScript pipelines; pass ",{"type":40,"tag":132,"props":1361,"children":1363},{"className":1362},[],[1364],{"type":46,"value":1365},"--runner tsx",{"type":46,"value":1367}," if auto-detection fails.",{"type":40,"tag":56,"props":1369,"children":1370},{},[1371,1373,1379],{"type":46,"value":1372},"Implement ",{"type":40,"tag":132,"props":1374,"children":1376},{"className":1375},[],[1377],{"type":46,"value":1378},"categorize",{"type":46,"value":1380}," from the taxonomy in step 3 — it is shared between the pipeline and the scorers. Refresh the dataset (rerun the pipeline) whenever it feels stale, too small, or skewed toward one category. Then verify the dataset actually contains the failures you want to attack:",{"type":40,"tag":265,"props":1382,"children":1384},{"className":267,"code":1383,"language":269,"meta":270,"style":270},"bt datasets view \u003Cagent>-\u003Cbehavior>-offline --limit 50\nbt sql \"SELECT metadata.primary_category AS cat, count(1) AS n \\\n  FROM dataset('\u003Cdataset-id>') \\\n  WHERE created > now() - interval 1 day \\\n  GROUP BY cat ORDER BY n DESC\" --json\n",[1385],{"type":40,"tag":132,"props":1386,"children":1387},{"__ignoreMap":270},[1388,1462,1486,1498,1510],{"type":40,"tag":276,"props":1389,"children":1390},{"class":278,"line":24},[1391,1395,1399,1404,1409,1414,1419,1424,1429,1434,1439,1444,1448,1453,1457],{"type":40,"tag":276,"props":1392,"children":1393},{"style":282},[1394],{"type":46,"value":150},{"type":40,"tag":276,"props":1396,"children":1397},{"style":287},[1398],{"type":46,"value":1302},{"type":40,"tag":276,"props":1400,"children":1401},{"style":287},[1402],{"type":46,"value":1403}," view",{"type":40,"tag":276,"props":1405,"children":1406},{"style":399},[1407],{"type":46,"value":1408}," \u003C",{"type":40,"tag":276,"props":1410,"children":1411},{"style":287},[1412],{"type":46,"value":1413},"agen",{"type":40,"tag":276,"props":1415,"children":1416},{"style":410},[1417],{"type":46,"value":1418},"t",{"type":40,"tag":276,"props":1420,"children":1421},{"style":399},[1422],{"type":46,"value":1423},">",{"type":40,"tag":276,"props":1425,"children":1426},{"style":287},[1427],{"type":46,"value":1428},"-",{"type":40,"tag":276,"props":1430,"children":1431},{"style":399},[1432],{"type":46,"value":1433},"\u003C",{"type":40,"tag":276,"props":1435,"children":1436},{"style":287},[1437],{"type":46,"value":1438},"behavio",{"type":40,"tag":276,"props":1440,"children":1441},{"style":410},[1442],{"type":46,"value":1443},"r",{"type":40,"tag":276,"props":1445,"children":1446},{"style":399},[1447],{"type":46,"value":1423},{"type":40,"tag":276,"props":1449,"children":1450},{"style":287},[1451],{"type":46,"value":1452},"-offline",{"type":40,"tag":276,"props":1454,"children":1455},{"style":287},[1456],{"type":46,"value":1322},{"type":40,"tag":276,"props":1458,"children":1459},{"style":1325},[1460],{"type":46,"value":1461}," 50\n",{"type":40,"tag":276,"props":1463,"children":1464},{"class":278,"line":298},[1465,1469,1473,1477,1482],{"type":40,"tag":276,"props":1466,"children":1467},{"style":282},[1468],{"type":46,"value":150},{"type":40,"tag":276,"props":1470,"children":1471},{"style":287},[1472],{"type":46,"value":396},{"type":40,"tag":276,"props":1474,"children":1475},{"style":399},[1476],{"type":46,"value":402},{"type":40,"tag":276,"props":1478,"children":1479},{"style":287},[1480],{"type":46,"value":1481},"SELECT metadata.primary_category AS cat, count(1) AS n ",{"type":40,"tag":276,"props":1483,"children":1484},{"style":410},[1485],{"type":46,"value":413},{"type":40,"tag":276,"props":1487,"children":1488},{"class":278,"line":416},[1489,1494],{"type":40,"tag":276,"props":1490,"children":1491},{"style":287},[1492],{"type":46,"value":1493},"  FROM dataset('\u003Cdataset-id>') ",{"type":40,"tag":276,"props":1495,"children":1496},{"style":410},[1497],{"type":46,"value":413},{"type":40,"tag":276,"props":1499,"children":1500},{"class":278,"line":429},[1501,1506],{"type":40,"tag":276,"props":1502,"children":1503},{"style":287},[1504],{"type":46,"value":1505},"  WHERE created > now() - interval 1 day ",{"type":40,"tag":276,"props":1507,"children":1508},{"style":410},[1509],{"type":46,"value":413},{"type":40,"tag":276,"props":1511,"children":1512},{"class":278,"line":442},[1513,1518,1522],{"type":40,"tag":276,"props":1514,"children":1515},{"style":287},[1516],{"type":46,"value":1517},"  GROUP BY cat ORDER BY n DESC",{"type":40,"tag":276,"props":1519,"children":1520},{"style":399},[1521],{"type":46,"value":466},{"type":40,"tag":276,"props":1523,"children":1524},{"style":287},[1525],{"type":46,"value":295},{"type":40,"tag":205,"props":1527,"children":1529},{"id":1528},"_5-write-scorers-that-codify-the-taxonomy",[1530],{"type":46,"value":1531},"5. Write scorers that codify the taxonomy",{"type":40,"tag":56,"props":1533,"children":1534},{},[1535,1537,1543],{"type":46,"value":1536},"Prefer deterministic code scorers for hard categories — they are free, fast, and stable across runs. Use LLM-as-a-judge (e.g. from ",{"type":40,"tag":132,"props":1538,"children":1540},{"className":1539},[],[1541],{"type":46,"value":1542},"autoevals",{"type":46,"value":1544},") only for qualities a deterministic check cannot capture, and keep judges few and focused.",{"type":40,"tag":56,"props":1546,"children":1547},{},[1548],{"type":46,"value":1549},"The scorer set that makes this loop work:",{"type":40,"tag":123,"props":1551,"children":1552},{},[1553,1571,1581,1599,1609],{"type":40,"tag":71,"props":1554,"children":1555},{},[1556,1561,1563,1569],{"type":40,"tag":707,"props":1557,"children":1558},{},[1559],{"type":46,"value":1560},"Hard quality",{"type":46,"value":1562}," (north star): binary, ",{"type":40,"tag":132,"props":1564,"children":1566},{"className":1565},[],[1567],{"type":46,"value":1568},"1",{"type":46,"value":1570}," only when the output has zero hard failure categories.",{"type":40,"tag":71,"props":1572,"children":1573},{},[1574,1579],{"type":40,"tag":707,"props":1575,"children":1576},{},[1577],{"type":46,"value":1578},"Issue count",{"type":46,"value":1580},": how many hard categories the output still triggers; gives gradient between 0 and 1.",{"type":40,"tag":71,"props":1582,"children":1583},{},[1584,1589,1591,1597],{"type":40,"tag":707,"props":1585,"children":1586},{},[1587],{"type":46,"value":1588},"Beats baseline",{"type":46,"value":1590},": whether the generated output has fewer issues than the captured production baseline in ",{"type":40,"tag":132,"props":1592,"children":1594},{"className":1593},[],[1595],{"type":46,"value":1596},"metadata.baseline_output",{"type":46,"value":1598},".",{"type":40,"tag":71,"props":1600,"children":1601},{},[1602,1607],{"type":40,"tag":707,"props":1603,"children":1604},{},[1605],{"type":46,"value":1606},"Avoids primary issue",{"type":46,"value":1608},": whether the output avoids the case's main captured failure category.",{"type":40,"tag":71,"props":1610,"children":1611},{},[1612,1617,1619,1625],{"type":40,"tag":707,"props":1613,"children":1614},{},[1615],{"type":46,"value":1616},"Soft efficiency",{"type":46,"value":1618},": weighted ",{"type":40,"tag":132,"props":1620,"children":1622},{"className":1621},[],[1623],{"type":46,"value":1624},"[0,1]",{"type":46,"value":1626}," score over the soft-inefficiency categories.",{"type":40,"tag":265,"props":1628,"children":1631},{"className":787,"code":1629,"language":789,"meta":1630,"style":270},"import { categorize, HARD_CATEGORIES } from \".\u002Fcategorize\";\n\nexport function hardQuality({ output }) {\n  const issues = categorize(output).filter((c) => HARD_CATEGORIES.has(c));\n  return {\n    name: \"hard_quality\",\n    score: issues.length === 0 ? 1 : 0,\n    metadata: { issues },\n  };\n}\n","scorers.ts",[1632],{"type":40,"tag":132,"props":1633,"children":1634},{"__ignoreMap":270},[1635,1684,1691,1726,1820,1832,1861,1919,1943,1951],{"type":40,"tag":276,"props":1636,"children":1637},{"class":278,"line":24},[1638,1642,1646,1650,1654,1659,1663,1667,1671,1676,1680],{"type":40,"tag":276,"props":1639,"children":1640},{"style":800},[1641],{"type":46,"value":803},{"type":40,"tag":276,"props":1643,"children":1644},{"style":399},[1645],{"type":46,"value":808},{"type":40,"tag":276,"props":1647,"children":1648},{"style":410},[1649],{"type":46,"value":1150},{"type":40,"tag":276,"props":1651,"children":1652},{"style":399},[1653],{"type":46,"value":1013},{"type":40,"tag":276,"props":1655,"children":1656},{"style":410},[1657],{"type":46,"value":1658}," HARD_CATEGORIES",{"type":40,"tag":276,"props":1660,"children":1661},{"style":399},[1662],{"type":46,"value":818},{"type":40,"tag":276,"props":1664,"children":1665},{"style":800},[1666],{"type":46,"value":823},{"type":40,"tag":276,"props":1668,"children":1669},{"style":399},[1670],{"type":46,"value":402},{"type":40,"tag":276,"props":1672,"children":1673},{"style":287},[1674],{"type":46,"value":1675},".\u002Fcategorize",{"type":40,"tag":276,"props":1677,"children":1678},{"style":399},[1679],{"type":46,"value":466},{"type":40,"tag":276,"props":1681,"children":1682},{"style":399},[1683],{"type":46,"value":840},{"type":40,"tag":276,"props":1685,"children":1686},{"class":278,"line":298},[1687],{"type":40,"tag":276,"props":1688,"children":1689},{"emptyLinePlaceholder":477},[1690],{"type":46,"value":480},{"type":40,"tag":276,"props":1692,"children":1693},{"class":278,"line":416},[1694,1699,1704,1709,1714,1718,1722],{"type":40,"tag":276,"props":1695,"children":1696},{"style":800},[1697],{"type":46,"value":1698},"export",{"type":40,"tag":276,"props":1700,"children":1701},{"style":1053},[1702],{"type":46,"value":1703}," function",{"type":40,"tag":276,"props":1705,"children":1706},{"style":853},[1707],{"type":46,"value":1708}," hardQuality",{"type":40,"tag":276,"props":1710,"children":1711},{"style":399},[1712],{"type":46,"value":1713},"({",{"type":40,"tag":276,"props":1715,"children":1716},{"style":1005},[1717],{"type":46,"value":1027},{"type":40,"tag":276,"props":1719,"children":1720},{"style":399},[1721],{"type":46,"value":1050},{"type":40,"tag":276,"props":1723,"children":1724},{"style":399},[1725],{"type":46,"value":885},{"type":40,"tag":276,"props":1727,"children":1728},{"class":278,"line":429},[1729,1734,1739,1744,1748,1752,1756,1760,1764,1769,1773,1777,1782,1786,1790,1794,1798,1803,1807,1811,1816],{"type":40,"tag":276,"props":1730,"children":1731},{"style":1053},[1732],{"type":46,"value":1733},"  const",{"type":40,"tag":276,"props":1735,"children":1736},{"style":410},[1737],{"type":46,"value":1738}," issues",{"type":40,"tag":276,"props":1740,"children":1741},{"style":399},[1742],{"type":46,"value":1743}," =",{"type":40,"tag":276,"props":1745,"children":1746},{"style":853},[1747],{"type":46,"value":1150},{"type":40,"tag":276,"props":1749,"children":1750},{"style":872},[1751],{"type":46,"value":861},{"type":40,"tag":276,"props":1753,"children":1754},{"style":410},[1755],{"type":46,"value":358},{"type":40,"tag":276,"props":1757,"children":1758},{"style":872},[1759],{"type":46,"value":1180},{"type":40,"tag":276,"props":1761,"children":1762},{"style":399},[1763],{"type":46,"value":1598},{"type":40,"tag":276,"props":1765,"children":1766},{"style":853},[1767],{"type":46,"value":1768},"filter",{"type":40,"tag":276,"props":1770,"children":1771},{"style":872},[1772],{"type":46,"value":861},{"type":40,"tag":276,"props":1774,"children":1775},{"style":399},[1776],{"type":46,"value":861},{"type":40,"tag":276,"props":1778,"children":1779},{"style":1005},[1780],{"type":46,"value":1781},"c",{"type":40,"tag":276,"props":1783,"children":1784},{"style":399},[1785],{"type":46,"value":1180},{"type":40,"tag":276,"props":1787,"children":1788},{"style":1053},[1789],{"type":46,"value":1056},{"type":40,"tag":276,"props":1791,"children":1792},{"style":410},[1793],{"type":46,"value":1658},{"type":40,"tag":276,"props":1795,"children":1796},{"style":399},[1797],{"type":46,"value":1598},{"type":40,"tag":276,"props":1799,"children":1800},{"style":853},[1801],{"type":46,"value":1802},"has",{"type":40,"tag":276,"props":1804,"children":1805},{"style":872},[1806],{"type":46,"value":861},{"type":40,"tag":276,"props":1808,"children":1809},{"style":410},[1810],{"type":46,"value":1781},{"type":40,"tag":276,"props":1812,"children":1813},{"style":872},[1814],{"type":46,"value":1815},"))",{"type":40,"tag":276,"props":1817,"children":1818},{"style":399},[1819],{"type":46,"value":840},{"type":40,"tag":276,"props":1821,"children":1822},{"class":278,"line":442},[1823,1828],{"type":40,"tag":276,"props":1824,"children":1825},{"style":800},[1826],{"type":46,"value":1827},"  return",{"type":40,"tag":276,"props":1829,"children":1830},{"style":399},[1831],{"type":46,"value":885},{"type":40,"tag":276,"props":1833,"children":1834},{"class":278,"line":455},[1835,1840,1844,1848,1853,1857],{"type":40,"tag":276,"props":1836,"children":1837},{"style":872},[1838],{"type":46,"value":1839},"    name",{"type":40,"tag":276,"props":1841,"children":1842},{"style":399},[1843],{"type":46,"value":880},{"type":40,"tag":276,"props":1845,"children":1846},{"style":399},[1847],{"type":46,"value":402},{"type":40,"tag":276,"props":1849,"children":1850},{"style":287},[1851],{"type":46,"value":1852},"hard_quality",{"type":40,"tag":276,"props":1854,"children":1855},{"style":399},[1856],{"type":46,"value":466},{"type":40,"tag":276,"props":1858,"children":1859},{"style":399},[1860],{"type":46,"value":915},{"type":40,"tag":276,"props":1862,"children":1863},{"class":278,"line":473},[1864,1869,1873,1877,1881,1886,1891,1896,1901,1906,1911,1915],{"type":40,"tag":276,"props":1865,"children":1866},{"style":872},[1867],{"type":46,"value":1868},"    score",{"type":40,"tag":276,"props":1870,"children":1871},{"style":399},[1872],{"type":46,"value":880},{"type":40,"tag":276,"props":1874,"children":1875},{"style":410},[1876],{"type":46,"value":1738},{"type":40,"tag":276,"props":1878,"children":1879},{"style":399},[1880],{"type":46,"value":1598},{"type":40,"tag":276,"props":1882,"children":1883},{"style":410},[1884],{"type":46,"value":1885},"length",{"type":40,"tag":276,"props":1887,"children":1888},{"style":399},[1889],{"type":46,"value":1890}," ===",{"type":40,"tag":276,"props":1892,"children":1893},{"style":1325},[1894],{"type":46,"value":1895}," 0",{"type":40,"tag":276,"props":1897,"children":1898},{"style":399},[1899],{"type":46,"value":1900}," ?",{"type":40,"tag":276,"props":1902,"children":1903},{"style":1325},[1904],{"type":46,"value":1905}," 1",{"type":40,"tag":276,"props":1907,"children":1908},{"style":399},[1909],{"type":46,"value":1910}," :",{"type":40,"tag":276,"props":1912,"children":1913},{"style":1325},[1914],{"type":46,"value":1895},{"type":40,"tag":276,"props":1916,"children":1917},{"style":399},[1918],{"type":46,"value":915},{"type":40,"tag":276,"props":1920,"children":1921},{"class":278,"line":483},[1922,1926,1930,1934,1938],{"type":40,"tag":276,"props":1923,"children":1924},{"style":872},[1925],{"type":46,"value":1085},{"type":40,"tag":276,"props":1927,"children":1928},{"style":399},[1929],{"type":46,"value":880},{"type":40,"tag":276,"props":1931,"children":1932},{"style":399},[1933],{"type":46,"value":808},{"type":40,"tag":276,"props":1935,"children":1936},{"style":410},[1937],{"type":46,"value":1738},{"type":40,"tag":276,"props":1939,"children":1940},{"style":399},[1941],{"type":46,"value":1942}," },\n",{"type":40,"tag":276,"props":1944,"children":1945},{"class":278,"line":492},[1946],{"type":40,"tag":276,"props":1947,"children":1948},{"style":399},[1949],{"type":46,"value":1950},"  };\n",{"type":40,"tag":276,"props":1952,"children":1953},{"class":278,"line":517},[1954],{"type":40,"tag":276,"props":1955,"children":1956},{"style":399},[1957],{"type":46,"value":1958},"}\n",{"type":40,"tag":56,"props":1960,"children":1961},{},[1962],{"type":46,"value":1963},"Unit-test the categorizer against known-bad and known-good outputs before trusting eval scores built on it.",{"type":40,"tag":205,"props":1965,"children":1967},{"id":1966},"_6-write-the-offline-eval",[1968],{"type":46,"value":1969},"6. Write the offline eval",{"type":40,"tag":56,"props":1971,"children":1972},{},[1973],{"type":46,"value":1974},"The task function replays the captured context against the current system and returns the target behavior's output (for tool-call behaviors, capture the first relevant tool call rather than running the tool):",{"type":40,"tag":265,"props":1976,"children":1979},{"className":787,"code":1977,"language":789,"meta":1978,"style":270},"import { Eval, initDataset } from \"braintrust\";\nimport {\n  hardQuality,\n  issueCount,\n  beatsBaseline,\n  softEfficiency,\n} from \".\u002Fscorers\";\n\nEval(\"\u003Cproject-name>\", {\n  experimentName: process.env.EVAL_EXPERIMENT,\n  data: initDataset({\n    project: \"\u003Cproject-name>\",\n    dataset: \"\u003Cagent>-\u003Cbehavior>-offline\",\n    useOutputAsExpected: false,\n  }),\n  task: async (input) => runAgentDecision(input),\n  scores: [hardQuality, issueCount, beatsBaseline, softEfficiency],\n});\n","first-step.eval.ts",[1980],{"type":40,"tag":132,"props":1981,"children":1982},{"__ignoreMap":270},[1983,2032,2043,2055,2067,2079,2091,2119,2126,2158,2197,2221,2249,2277,2299,2314,2361,2409],{"type":40,"tag":276,"props":1984,"children":1985},{"class":278,"line":24},[1986,1990,1994,1999,2003,2008,2012,2016,2020,2024,2028],{"type":40,"tag":276,"props":1987,"children":1988},{"style":800},[1989],{"type":46,"value":803},{"type":40,"tag":276,"props":1991,"children":1992},{"style":399},[1993],{"type":46,"value":808},{"type":40,"tag":276,"props":1995,"children":1996},{"style":410},[1997],{"type":46,"value":1998}," Eval",{"type":40,"tag":276,"props":2000,"children":2001},{"style":399},[2002],{"type":46,"value":1013},{"type":40,"tag":276,"props":2004,"children":2005},{"style":410},[2006],{"type":46,"value":2007}," initDataset",{"type":40,"tag":276,"props":2009,"children":2010},{"style":399},[2011],{"type":46,"value":818},{"type":40,"tag":276,"props":2013,"children":2014},{"style":800},[2015],{"type":46,"value":823},{"type":40,"tag":276,"props":2017,"children":2018},{"style":399},[2019],{"type":46,"value":402},{"type":40,"tag":276,"props":2021,"children":2022},{"style":287},[2023],{"type":46,"value":8},{"type":40,"tag":276,"props":2025,"children":2026},{"style":399},[2027],{"type":46,"value":466},{"type":40,"tag":276,"props":2029,"children":2030},{"style":399},[2031],{"type":46,"value":840},{"type":40,"tag":276,"props":2033,"children":2034},{"class":278,"line":298},[2035,2039],{"type":40,"tag":276,"props":2036,"children":2037},{"style":800},[2038],{"type":46,"value":803},{"type":40,"tag":276,"props":2040,"children":2041},{"style":399},[2042],{"type":46,"value":885},{"type":40,"tag":276,"props":2044,"children":2045},{"class":278,"line":416},[2046,2051],{"type":40,"tag":276,"props":2047,"children":2048},{"style":410},[2049],{"type":46,"value":2050},"  hardQuality",{"type":40,"tag":276,"props":2052,"children":2053},{"style":399},[2054],{"type":46,"value":915},{"type":40,"tag":276,"props":2056,"children":2057},{"class":278,"line":429},[2058,2063],{"type":40,"tag":276,"props":2059,"children":2060},{"style":410},[2061],{"type":46,"value":2062},"  issueCount",{"type":40,"tag":276,"props":2064,"children":2065},{"style":399},[2066],{"type":46,"value":915},{"type":40,"tag":276,"props":2068,"children":2069},{"class":278,"line":442},[2070,2075],{"type":40,"tag":276,"props":2071,"children":2072},{"style":410},[2073],{"type":46,"value":2074},"  beatsBaseline",{"type":40,"tag":276,"props":2076,"children":2077},{"style":399},[2078],{"type":46,"value":915},{"type":40,"tag":276,"props":2080,"children":2081},{"class":278,"line":455},[2082,2087],{"type":40,"tag":276,"props":2083,"children":2084},{"style":410},[2085],{"type":46,"value":2086},"  softEfficiency",{"type":40,"tag":276,"props":2088,"children":2089},{"style":399},[2090],{"type":46,"value":915},{"type":40,"tag":276,"props":2092,"children":2093},{"class":278,"line":473},[2094,2098,2102,2106,2111,2115],{"type":40,"tag":276,"props":2095,"children":2096},{"style":399},[2097],{"type":46,"value":1275},{"type":40,"tag":276,"props":2099,"children":2100},{"style":800},[2101],{"type":46,"value":823},{"type":40,"tag":276,"props":2103,"children":2104},{"style":399},[2105],{"type":46,"value":402},{"type":40,"tag":276,"props":2107,"children":2108},{"style":287},[2109],{"type":46,"value":2110},".\u002Fscorers",{"type":40,"tag":276,"props":2112,"children":2113},{"style":399},[2114],{"type":46,"value":466},{"type":40,"tag":276,"props":2116,"children":2117},{"style":399},[2118],{"type":46,"value":840},{"type":40,"tag":276,"props":2120,"children":2121},{"class":278,"line":483},[2122],{"type":40,"tag":276,"props":2123,"children":2124},{"emptyLinePlaceholder":477},[2125],{"type":46,"value":480},{"type":40,"tag":276,"props":2127,"children":2128},{"class":278,"line":492},[2129,2134,2138,2142,2146,2150,2154],{"type":40,"tag":276,"props":2130,"children":2131},{"style":853},[2132],{"type":46,"value":2133},"Eval",{"type":40,"tag":276,"props":2135,"children":2136},{"style":410},[2137],{"type":46,"value":861},{"type":40,"tag":276,"props":2139,"children":2140},{"style":399},[2141],{"type":46,"value":466},{"type":40,"tag":276,"props":2143,"children":2144},{"style":287},[2145],{"type":46,"value":906},{"type":40,"tag":276,"props":2147,"children":2148},{"style":399},[2149],{"type":46,"value":466},{"type":40,"tag":276,"props":2151,"children":2152},{"style":399},[2153],{"type":46,"value":1013},{"type":40,"tag":276,"props":2155,"children":2156},{"style":399},[2157],{"type":46,"value":885},{"type":40,"tag":276,"props":2159,"children":2160},{"class":278,"line":517},[2161,2166,2170,2175,2179,2184,2188,2193],{"type":40,"tag":276,"props":2162,"children":2163},{"style":872},[2164],{"type":46,"value":2165},"  experimentName",{"type":40,"tag":276,"props":2167,"children":2168},{"style":399},[2169],{"type":46,"value":880},{"type":40,"tag":276,"props":2171,"children":2172},{"style":410},[2173],{"type":46,"value":2174}," process",{"type":40,"tag":276,"props":2176,"children":2177},{"style":399},[2178],{"type":46,"value":1598},{"type":40,"tag":276,"props":2180,"children":2181},{"style":410},[2182],{"type":46,"value":2183},"env",{"type":40,"tag":276,"props":2185,"children":2186},{"style":399},[2187],{"type":46,"value":1598},{"type":40,"tag":276,"props":2189,"children":2190},{"style":410},[2191],{"type":46,"value":2192},"EVAL_EXPERIMENT",{"type":40,"tag":276,"props":2194,"children":2195},{"style":399},[2196],{"type":46,"value":915},{"type":40,"tag":276,"props":2198,"children":2199},{"class":278,"line":529},[2200,2205,2209,2213,2217],{"type":40,"tag":276,"props":2201,"children":2202},{"style":872},[2203],{"type":46,"value":2204},"  data",{"type":40,"tag":276,"props":2206,"children":2207},{"style":399},[2208],{"type":46,"value":880},{"type":40,"tag":276,"props":2210,"children":2211},{"style":853},[2212],{"type":46,"value":2007},{"type":40,"tag":276,"props":2214,"children":2215},{"style":410},[2216],{"type":46,"value":861},{"type":40,"tag":276,"props":2218,"children":2219},{"style":399},[2220],{"type":46,"value":866},{"type":40,"tag":276,"props":2222,"children":2223},{"class":278,"line":542},[2224,2229,2233,2237,2241,2245],{"type":40,"tag":276,"props":2225,"children":2226},{"style":872},[2227],{"type":46,"value":2228},"    project",{"type":40,"tag":276,"props":2230,"children":2231},{"style":399},[2232],{"type":46,"value":880},{"type":40,"tag":276,"props":2234,"children":2235},{"style":399},[2236],{"type":46,"value":402},{"type":40,"tag":276,"props":2238,"children":2239},{"style":287},[2240],{"type":46,"value":906},{"type":40,"tag":276,"props":2242,"children":2243},{"style":399},[2244],{"type":46,"value":466},{"type":40,"tag":276,"props":2246,"children":2247},{"style":399},[2248],{"type":46,"value":915},{"type":40,"tag":276,"props":2250,"children":2251},{"class":278,"line":558},[2252,2257,2261,2265,2269,2273],{"type":40,"tag":276,"props":2253,"children":2254},{"style":872},[2255],{"type":46,"value":2256},"    dataset",{"type":40,"tag":276,"props":2258,"children":2259},{"style":399},[2260],{"type":46,"value":880},{"type":40,"tag":276,"props":2262,"children":2263},{"style":399},[2264],{"type":46,"value":402},{"type":40,"tag":276,"props":2266,"children":2267},{"style":287},[2268],{"type":46,"value":1250},{"type":40,"tag":276,"props":2270,"children":2271},{"style":399},[2272],{"type":46,"value":466},{"type":40,"tag":276,"props":2274,"children":2275},{"style":399},[2276],{"type":46,"value":915},{"type":40,"tag":276,"props":2278,"children":2279},{"class":278,"line":566},[2280,2285,2289,2295],{"type":40,"tag":276,"props":2281,"children":2282},{"style":872},[2283],{"type":46,"value":2284},"    useOutputAsExpected",{"type":40,"tag":276,"props":2286,"children":2287},{"style":399},[2288],{"type":46,"value":880},{"type":40,"tag":276,"props":2290,"children":2292},{"style":2291},"--shiki-light:#FF5370;--shiki-default:#FF9CAC;--shiki-dark:#FF9CAC",[2293],{"type":46,"value":2294}," false",{"type":40,"tag":276,"props":2296,"children":2297},{"style":399},[2298],{"type":46,"value":915},{"type":40,"tag":276,"props":2300,"children":2301},{"class":278,"line":575},[2302,2306,2310],{"type":40,"tag":276,"props":2303,"children":2304},{"style":399},[2305],{"type":46,"value":1175},{"type":40,"tag":276,"props":2307,"children":2308},{"style":410},[2309],{"type":46,"value":1180},{"type":40,"tag":276,"props":2311,"children":2312},{"style":399},[2313],{"type":46,"value":915},{"type":40,"tag":276,"props":2315,"children":2316},{"class":278,"line":600},[2317,2322,2326,2331,2335,2339,2343,2347,2352,2357],{"type":40,"tag":276,"props":2318,"children":2319},{"style":853},[2320],{"type":46,"value":2321},"  task",{"type":40,"tag":276,"props":2323,"children":2324},{"style":399},[2325],{"type":46,"value":880},{"type":40,"tag":276,"props":2327,"children":2328},{"style":1053},[2329],{"type":46,"value":2330}," async",{"type":40,"tag":276,"props":2332,"children":2333},{"style":399},[2334],{"type":46,"value":1061},{"type":40,"tag":276,"props":2336,"children":2337},{"style":1005},[2338],{"type":46,"value":350},{"type":40,"tag":276,"props":2340,"children":2341},{"style":399},[2342],{"type":46,"value":1180},{"type":40,"tag":276,"props":2344,"children":2345},{"style":1053},[2346],{"type":46,"value":1056},{"type":40,"tag":276,"props":2348,"children":2349},{"style":853},[2350],{"type":46,"value":2351}," runAgentDecision",{"type":40,"tag":276,"props":2353,"children":2354},{"style":410},[2355],{"type":46,"value":2356},"(input)",{"type":40,"tag":276,"props":2358,"children":2359},{"style":399},[2360],{"type":46,"value":915},{"type":40,"tag":276,"props":2362,"children":2363},{"class":278,"line":612},[2364,2369,2373,2378,2382,2387,2391,2396,2400,2405],{"type":40,"tag":276,"props":2365,"children":2366},{"style":872},[2367],{"type":46,"value":2368},"  scores",{"type":40,"tag":276,"props":2370,"children":2371},{"style":399},[2372],{"type":46,"value":880},{"type":40,"tag":276,"props":2374,"children":2375},{"style":410},[2376],{"type":46,"value":2377}," [hardQuality",{"type":40,"tag":276,"props":2379,"children":2380},{"style":399},[2381],{"type":46,"value":1013},{"type":40,"tag":276,"props":2383,"children":2384},{"style":410},[2385],{"type":46,"value":2386}," issueCount",{"type":40,"tag":276,"props":2388,"children":2389},{"style":399},[2390],{"type":46,"value":1013},{"type":40,"tag":276,"props":2392,"children":2393},{"style":410},[2394],{"type":46,"value":2395}," beatsBaseline",{"type":40,"tag":276,"props":2397,"children":2398},{"style":399},[2399],{"type":46,"value":1013},{"type":40,"tag":276,"props":2401,"children":2402},{"style":410},[2403],{"type":46,"value":2404}," softEfficiency]",{"type":40,"tag":276,"props":2406,"children":2407},{"style":399},[2408],{"type":46,"value":915},{"type":40,"tag":276,"props":2410,"children":2411},{"class":278,"line":625},[2412,2416,2420],{"type":40,"tag":276,"props":2413,"children":2414},{"style":399},[2415],{"type":46,"value":1275},{"type":40,"tag":276,"props":2417,"children":2418},{"style":410},[2419],{"type":46,"value":1180},{"type":40,"tag":276,"props":2421,"children":2422},{"style":399},[2423],{"type":46,"value":840},{"type":40,"tag":56,"props":2425,"children":2426},{},[2427,2433],{"type":40,"tag":132,"props":2428,"children":2430},{"className":2429},[],[2431],{"type":46,"value":2432},"runAgentDecision",{"type":46,"value":2434}," must call the same prompt, tools, and model configuration as production — import them from the app code rather than copying them, so prompt improvements are picked up automatically.",{"type":40,"tag":56,"props":2436,"children":2437},{},[2438],{"type":46,"value":2439},"Two practical replay issues:",{"type":40,"tag":123,"props":2441,"children":2442},{},[2443,2464],{"type":40,"tag":71,"props":2444,"children":2445},{},[2446,2448,2454,2456,2462],{"type":46,"value":2447},"Captured message histories often start mid-conversation. Normalize them before sending to the model: drop ",{"type":40,"tag":132,"props":2449,"children":2451},{"className":2450},[],[2452],{"type":46,"value":2453},"tool_result",{"type":46,"value":2455}," blocks whose matching ",{"type":40,"tag":132,"props":2457,"children":2459},{"className":2458},[],[2460],{"type":46,"value":2461},"tool_use",{"type":46,"value":2463}," was trimmed away, and start the history on a user message. Otherwise the provider API rejects the request.",{"type":40,"tag":71,"props":2465,"children":2466},{},[2467,2469,2474],{"type":46,"value":2468},"If no model provider key is available, route the task's LLM calls through the Braintrust AI proxy with ",{"type":40,"tag":132,"props":2470,"children":2472},{"className":2471},[],[2473],{"type":46,"value":166},{"type":46,"value":1598},{"type":40,"tag":205,"props":2476,"children":2478},{"id":2477},"_7-run-the-eval",[2479],{"type":46,"value":2480},"7. Run the eval",{"type":40,"tag":56,"props":2482,"children":2483},{},[2484],{"type":46,"value":2485},"Bounded smoke run first, full run once it is directionally better:",{"type":40,"tag":265,"props":2487,"children":2489},{"className":267,"code":2488,"language":269,"meta":270,"style":270},"bt eval --first 20 first-step.eval.ts   # bounded, non-final\nbt eval first-step.eval.ts              # full dataset, final\n",[2490],{"type":40,"tag":132,"props":2491,"children":2492},{"__ignoreMap":270},[2493,2525],{"type":40,"tag":276,"props":2494,"children":2495},{"class":278,"line":24},[2496,2500,2505,2510,2515,2520],{"type":40,"tag":276,"props":2497,"children":2498},{"style":282},[2499],{"type":46,"value":150},{"type":40,"tag":276,"props":2501,"children":2502},{"style":287},[2503],{"type":46,"value":2504}," eval",{"type":40,"tag":276,"props":2506,"children":2507},{"style":287},[2508],{"type":46,"value":2509}," --first",{"type":40,"tag":276,"props":2511,"children":2512},{"style":1325},[2513],{"type":46,"value":2514}," 20",{"type":40,"tag":276,"props":2516,"children":2517},{"style":287},[2518],{"type":46,"value":2519}," first-step.eval.ts",{"type":40,"tag":276,"props":2521,"children":2522},{"style":381},[2523],{"type":46,"value":2524},"   # bounded, non-final\n",{"type":40,"tag":276,"props":2526,"children":2527},{"class":278,"line":298},[2528,2532,2536,2540],{"type":40,"tag":276,"props":2529,"children":2530},{"style":282},[2531],{"type":46,"value":150},{"type":40,"tag":276,"props":2533,"children":2534},{"style":287},[2535],{"type":46,"value":2504},{"type":40,"tag":276,"props":2537,"children":2538},{"style":287},[2539],{"type":46,"value":2519},{"type":40,"tag":276,"props":2541,"children":2542},{"style":381},[2543],{"type":46,"value":2544},"              # full dataset, final\n",{"type":40,"tag":56,"props":2546,"children":2547},{},[2548,2550,2556],{"type":46,"value":2549},"Note the experiment name printed in the summary, then resolve its ID — ",{"type":40,"tag":132,"props":2551,"children":2553},{"className":2552},[],[2554],{"type":46,"value":2555},"experiment(...)",{"type":46,"value":2557}," requires the ID, not the name:",{"type":40,"tag":265,"props":2559,"children":2561},{"className":267,"code":2560,"language":269,"meta":270,"style":270},"bt experiments list --json\nbt sql \"SELECT root_span_id, scores \\\n  FROM experiment('\u003Cexperiment-id>') \\\n  WHERE created > now() - interval 1 day \\\n    AND span_attributes.type = 'score' AND scores['hard_quality'] \u003C 1\" --json\n",[2562],{"type":40,"tag":132,"props":2563,"children":2564},{"__ignoreMap":270},[2565,2585,2609,2621,2632],{"type":40,"tag":276,"props":2566,"children":2567},{"class":278,"line":24},[2568,2572,2577,2581],{"type":40,"tag":276,"props":2569,"children":2570},{"style":282},[2571],{"type":46,"value":150},{"type":40,"tag":276,"props":2573,"children":2574},{"style":287},[2575],{"type":46,"value":2576}," experiments",{"type":40,"tag":276,"props":2578,"children":2579},{"style":287},[2580],{"type":46,"value":313},{"type":40,"tag":276,"props":2582,"children":2583},{"style":287},[2584],{"type":46,"value":295},{"type":40,"tag":276,"props":2586,"children":2587},{"class":278,"line":298},[2588,2592,2596,2600,2605],{"type":40,"tag":276,"props":2589,"children":2590},{"style":282},[2591],{"type":46,"value":150},{"type":40,"tag":276,"props":2593,"children":2594},{"style":287},[2595],{"type":46,"value":396},{"type":40,"tag":276,"props":2597,"children":2598},{"style":399},[2599],{"type":46,"value":402},{"type":40,"tag":276,"props":2601,"children":2602},{"style":287},[2603],{"type":46,"value":2604},"SELECT root_span_id, scores ",{"type":40,"tag":276,"props":2606,"children":2607},{"style":410},[2608],{"type":46,"value":413},{"type":40,"tag":276,"props":2610,"children":2611},{"class":278,"line":416},[2612,2617],{"type":40,"tag":276,"props":2613,"children":2614},{"style":287},[2615],{"type":46,"value":2616},"  FROM experiment('\u003Cexperiment-id>') ",{"type":40,"tag":276,"props":2618,"children":2619},{"style":410},[2620],{"type":46,"value":413},{"type":40,"tag":276,"props":2622,"children":2623},{"class":278,"line":429},[2624,2628],{"type":40,"tag":276,"props":2625,"children":2626},{"style":287},[2627],{"type":46,"value":1505},{"type":40,"tag":276,"props":2629,"children":2630},{"style":410},[2631],{"type":46,"value":413},{"type":40,"tag":276,"props":2633,"children":2634},{"class":278,"line":442},[2635,2640,2644],{"type":40,"tag":276,"props":2636,"children":2637},{"style":287},[2638],{"type":46,"value":2639},"    AND span_attributes.type = 'score' AND scores['hard_quality'] \u003C 1",{"type":40,"tag":276,"props":2641,"children":2642},{"style":399},[2643],{"type":46,"value":466},{"type":40,"tag":276,"props":2645,"children":2646},{"style":287},[2647],{"type":46,"value":295},{"type":40,"tag":205,"props":2649,"children":2651},{"id":2650},"_8-read-the-scores-correctly",[2652],{"type":46,"value":2653},"8. Read the scores correctly",{"type":40,"tag":56,"props":2655,"children":2656},{},[2657],{"type":46,"value":2658},"Priority order:",{"type":40,"tag":67,"props":2660,"children":2661},{},[2662,2667,2672,2677],{"type":40,"tag":71,"props":2663,"children":2664},{},[2665],{"type":46,"value":2666},"Keep the task producing the target behavior at all (it must not stop calling the tool, answering, etc.).",{"type":40,"tag":71,"props":2668,"children":2669},{},[2670],{"type":46,"value":2671},"Reduce the hard issue count.",{"type":40,"tag":71,"props":2673,"children":2674},{},[2675],{"type":46,"value":2676},"Raise the binary hard-quality rate.",{"type":40,"tag":71,"props":2678,"children":2679},{},[2680],{"type":46,"value":2681},"Then tighten soft efficiency.",{"type":40,"tag":56,"props":2683,"children":2684},{},[2685],{"type":46,"value":2686},"Do not trade away the hard score to optimize the soft score.",{"type":40,"tag":205,"props":2688,"children":2690},{"id":2689},"_9-improve-the-system-not-the-case",[2691],{"type":46,"value":2692},"9. Improve the system, not the case",{"type":40,"tag":56,"props":2694,"children":2695},{},[2696],{"type":46,"value":2697},"Prefer fixes that generalize across many traces:",{"type":40,"tag":123,"props":2699,"children":2700},{},[2701,2711,2721,2731],{"type":40,"tag":71,"props":2702,"children":2703},{},[2704,2709],{"type":40,"tag":707,"props":2705,"children":2706},{},[2707],{"type":46,"value":2708},"System prompt",{"type":46,"value":2710},": tighten guidance for the target behavior; add concrete bad-vs-better examples drawn from real failures; resolve the ambiguities the taxonomy exposed.",{"type":40,"tag":71,"props":2712,"children":2713},{},[2714,2719],{"type":40,"tag":707,"props":2715,"children":2716},{},[2717],{"type":46,"value":2718},"Tool definitions",{"type":46,"value":2720},": improve tool descriptions, schemas, and parameter docs so the model cannot misread them.",{"type":40,"tag":71,"props":2722,"children":2723},{},[2724,2729],{"type":40,"tag":707,"props":2725,"children":2726},{},[2727],{"type":46,"value":2728},"Runtime",{"type":46,"value":2730},": validate, normalize, or reject bad outputs in code when the contract should be strict rather than suggested.",{"type":40,"tag":71,"props":2732,"children":2733},{},[2734,2739],{"type":40,"tag":707,"props":2735,"children":2736},{},[2737],{"type":46,"value":2738},"Categorizer\u002Fscorers",{"type":46,"value":2740},": keep them aligned with the guidance — when you add a prompt rule, add or adjust the category that detects violations of it.",{"type":40,"tag":56,"props":2742,"children":2743},{},[2744],{"type":46,"value":2745},"If a fix only helps one or two dataset rows, it is overfitting; push on repeated categories instead.",{"type":40,"tag":205,"props":2747,"children":2749},{"id":2748},"_10-iterate",[2750],{"type":46,"value":2751},"10. Iterate",{"type":40,"tag":56,"props":2753,"children":2754},{},[2755],{"type":46,"value":2756},"After every meaningful change:",{"type":40,"tag":67,"props":2758,"children":2759},{},[2760,2765,2770],{"type":40,"tag":71,"props":2761,"children":2762},{},[2763],{"type":46,"value":2764},"Rerun the bounded eval.",{"type":40,"tag":71,"props":2766,"children":2767},{},[2768],{"type":46,"value":2769},"Compare against the previous best experiment (the run summary shows the diff; the experiment comparison UI shows per-case regressions).",{"type":40,"tag":71,"props":2771,"children":2772},{},[2773],{"type":46,"value":2774},"If directionally better, rerun the full dataset. The full dataset is the real check.",{"type":40,"tag":56,"props":2776,"children":2777},{},[2778],{"type":46,"value":2779},"When new production failures appear later: refresh the dataset from the pipeline, rerun, keep iterating.",{"type":40,"tag":205,"props":2781,"children":2783},{"id":2782},"_11-graduate-to-online-scoring",[2784],{"type":46,"value":2785},"11. Graduate to online scoring",{"type":40,"tag":56,"props":2787,"children":2788},{},[2789],{"type":46,"value":2790},"Only after the offline loop is healthy:",{"type":40,"tag":67,"props":2792,"children":2793},{},[2794,2807,2812],{"type":40,"tag":71,"props":2795,"children":2796},{},[2797,2799,2805],{"type":46,"value":2798},"Push the same scorer logic as project scorers with ",{"type":40,"tag":132,"props":2800,"children":2802},{"className":2801},[],[2803],{"type":46,"value":2804},"bt functions push",{"type":46,"value":2806},", keeping it aligned with the offline categorizer.",{"type":40,"tag":71,"props":2808,"children":2809},{},[2810],{"type":46,"value":2811},"Configure online scoring or automations on production logs so regressions surface as scored traces.",{"type":40,"tag":71,"props":2813,"children":2814},{},[2815],{"type":46,"value":2816},"Feed newly flagged traces back into the dataset via the pipeline.",{"type":40,"tag":56,"props":2818,"children":2819},{},[2820],{"type":46,"value":2821},"Until then, the offline eval is the primary optimization surface.",{"type":40,"tag":49,"props":2823,"children":2825},{"id":2824},"guardrails",[2826],{"type":46,"value":2827},"Guardrails",{"type":40,"tag":123,"props":2829,"children":2830},{},[2831,2836,2841,2846,2851,2856],{"type":40,"tag":71,"props":2832,"children":2833},{},[2834],{"type":46,"value":2835},"Do not invent placeholder cases or fabricate expected outputs; every dataset row must come from a real trace or an explicitly user-provided example.",{"type":40,"tag":71,"props":2837,"children":2838},{},[2839],{"type":46,"value":2840},"Do not assume the current dataset is representative; inspect it and refresh it.",{"type":40,"tag":71,"props":2842,"children":2843},{},[2844],{"type":46,"value":2845},"Do not optimize around one or two traces. Push on repeated categories.",{"type":40,"tag":71,"props":2847,"children":2848},{},[2849],{"type":46,"value":2850},"Do not let scorer logic drift from prompt guidance — they encode the same contract.",{"type":40,"tag":71,"props":2852,"children":2853},{},[2854],{"type":46,"value":2855},"Do not eval against live mutable data as the primary loop; the remote dataset is the frozen surface.",{"type":40,"tag":71,"props":2857,"children":2858},{},[2859],{"type":46,"value":2860},"Do not start with LLM judges when a deterministic check exists for the same category.",{"type":40,"tag":49,"props":2862,"children":2864},{"id":2863},"expected-outcome",[2865],{"type":46,"value":2866},"Expected outcome",{"type":40,"tag":56,"props":2868,"children":2869},{},[2870],{"type":46,"value":2871},"A good iteration improves the target behavior across many real traced cases by:",{"type":40,"tag":123,"props":2873,"children":2874},{},[2875,2880,2885,2890],{"type":40,"tag":71,"props":2876,"children":2877},{},[2878],{"type":46,"value":2879},"reducing hard failure categories",{"type":40,"tag":71,"props":2881,"children":2882},{},[2883],{"type":46,"value":2884},"increasing the clean-output rate",{"type":40,"tag":71,"props":2886,"children":2887},{},[2888],{"type":46,"value":2889},"beating the captured production baselines",{"type":40,"tag":71,"props":2891,"children":2892},{},[2893],{"type":46,"value":2894},"trimming soft inefficiencies without sacrificing hard quality",{"type":40,"tag":56,"props":2896,"children":2897},{},[2898],{"type":46,"value":2899},"If a change improves the bounded run but regresses the full dataset, keep iterating — the full dataset is the real check.",{"type":40,"tag":2901,"props":2902,"children":2903},"style",{},[2904],{"type":46,"value":2905},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2907,"total":24},[2908],{"slug":4,"name":4,"fn":5,"description":6,"org":2909,"tags":2910,"stars":24,"repoUrl":25,"updatedAt":26},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2911,2912,2913,2914],{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"items":2916,"total":416},[2917,2932,2948],{"slug":2918,"name":2918,"fn":2919,"description":2920,"org":2921,"tags":2922,"stars":625,"repoUrl":2930,"updatedAt":2931},"troubleshoot-braintrust-mcp","configure and troubleshoot Braintrust MCP servers","This plugin auto-configures a \"braintrust\" MCP server. If you can't see it or reach it, activate this skill\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2923,2924,2927],{"name":9,"slug":8,"type":16},{"name":2925,"slug":2926,"type":16},"Debugging","debugging",{"name":2928,"slug":2929,"type":16},"MCP","mcp","https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Fbraintrust-claude-plugin","2026-07-12T08:36:13.889274",{"slug":8,"name":8,"fn":2933,"description":2934,"org":2935,"tags":2936,"stars":298,"repoUrl":2946,"updatedAt":2947},"inspect Braintrust projects and logs","Use the official Braintrust MCP server to search Braintrust docs, inspect projects, query logs and experiments, summarize eval results, and generate permalinks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2937,2940,2941,2944,2945],{"name":2938,"slug":2939,"type":16},"Analytics","analytics",{"name":9,"slug":8,"type":16},{"name":2942,"slug":2943,"type":16},"Documentation","documentation",{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Fbraintrust-codex-plugin","2026-07-12T08:36:17.655627",{"slug":4,"name":4,"fn":5,"description":6,"org":2949,"tags":2950,"stars":24,"repoUrl":25,"updatedAt":26},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2951,2952,2953,2954],{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16}]