[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-braintrust-braintrust-write-eval-scorer":3,"mdc--eolrul-key":38,"related-repo-braintrust-braintrust-write-eval-scorer":455,"related-org-braintrust-braintrust-write-eval-scorer":557},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":27,"repoUrl":28,"updatedAt":29,"license":30,"forks":31,"topics":32,"repo":33,"sourceUrl":36,"mdContent":37},"braintrust-write-eval-scorer","design and implement LLM eval scorers","Design, implement, edit, or audit narrow eval scorers for LLM applications and agents, including deterministic checks, reference or final-state comparisons, trace and tool-call checks, and anchored LLM-as-judge rubrics. Use when translating one observable criterion into scoring logic, choosing between deterministic and judge-based scoring, repairing a vague rubric, or defining handling for refusals, errors, timeouts, and parse failures. Do not use to validate scorer agreement against human labels or to design the human review workflow.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"braintrust","Braintrust","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fbraintrust.png","braintrustdata",[13,17,20,23,24],{"name":14,"slug":15,"type":16},"LLM","llm","tag",{"name":18,"slug":19,"type":16},"Evals","evals",{"name":21,"slug":22,"type":16},"Agents","agents",{"name":9,"slug":8,"type":16},{"name":25,"slug":26,"type":16},"Code Analysis","code-analysis",7,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library","2026-08-20T03:52:56.65537",null,0,[],{"repoUrl":28,"stars":27,"forks":31,"topics":34,"description":35},[],"Braintrust eval skills library","https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library\u002Ftree\u002FHEAD\u002Fskills\u002Fbraintrust-write-eval-scorer","---\nname: braintrust-write-eval-scorer\ndescription: >-\n  Design, implement, edit, or audit narrow eval scorers for LLM applications and agents,\n  including deterministic checks, reference or final-state comparisons, trace and tool-call\n  checks, and anchored LLM-as-judge rubrics. Use when translating one observable criterion into\n  scoring logic, choosing between deterministic and judge-based scoring, repairing a vague\n  rubric, or defining handling for refusals, errors, timeouts, and parse failures. Do not use to\n  validate scorer agreement against human labels or to design the human review workflow.\n---\n\n# Write one eval scorer\n\nContract: `references\u002Finteraction-contract.md`. Calibration, templates, provenance: `references\u002Fscorer-patterns.md`.\n\n## Trigger\n\n- \"Write a scorer for this criterion.\" \u002F \"Should this be deterministic or a judge?\"\n- Turning a behavior-spec clause or evidence-map signal into a check.\n- A rubric producing inconsistent scores, or a scorer bundling several qualities.\n\n## Do\n\n1. Name **one** criterion, and its output contract: a **score** (0–1, needs a numeric mapping) or\n   a **classification** (one label from a fixed set, needs no-match behavior). If the request\n   combines several criteria, split them before writing any code or rubric.\n2. Match the method to the evidence — and let **stakes override convenience**. Objective →\n   deterministic. Subjective → anchored rubric or human. A checkable outcome on a\n   safety-critical path still needs sampled human review, because what is most likely wrong is\n   the check's *scope*.\n3. Set two **independent** axes and keep them apart. **Input scope** — span, trace, or group; how\n   much one evaluator call sees, default trace. **Reporting level** — per-item scoring localizes\n   failures, aggregate detects regressions. Most criteria need both reporting levels.\n4. For rubrics, write criteria as **anchored examples, not descriptions**, each scored\n   separately, requiring structured output that carries the evidence used. Have the model emit\n   **semantic classes, not numbers**, and map classes to scores outside the model.\n5. Define failure handling, keeping the kinds apart: **system** failures (refusal, invalid\n   output, wrong state) are scored, never dropped; **harness** failures (your own parallelism,\n   exhausted credits) are missing data in the status field.\n6. Version the scorer and pin the version in experiment metadata.\n\n## Avoid\n\n- Do not use a judge where objective state can be checked deterministically, or a judge from the\n  same model family as the system under test.\n- Do not combine unrelated qualities into one score.\n- Do not silently drop errors, refusals, or unparseable outputs.\n- Do not tune the scorer until the numbers improve — adopting a normalizer *after* seeing which\n  arm it helps is scorer-side p-hacking.\n- Do not validate the scorer here.\n\n## Check\n\n- Exactly one criterion; method justified by evidence type **and** stakes.\n- Tested against clear successes, clear failures, edges, refusals, timeouts, parse failures.\n- Returns interpretable evidence, not just a number.\n- Order counterbalanced in pairwise setups; length sensitivity checked.\n- Versioned; known artifacts documented with the arms they disadvantage.\n\n## Risk\n\n- Judge bias — self-preference, position, verbosity — produces confident, invalid scores.\n- A judge reads text the system under test wrote, and that text can address the judge directly.\n  The bias controls assume a miscalibrated judge, not one being spoken to — contract §9.\n- A scorer without trace access can only judge the final answer, however the criterion was\n  written.\n- Silent exclusions are the most dangerous failure: the aggregate answers \"how did the system do\n  on the items that survived\" while appearing to answer the question asked.\n\n## Braintrust\n\nDeterministic criteria → **code scorers**; subjective → **rubric scorers**. Shared mechanics:\n`references\u002Fplatform-mechanics.md`. **§5** is the one this stage creates rather than consumes —\none criterion, one scorer, one name, chosen here and depended on by every downstream diff.\n\nScores in native `scores` (0–1); the judge's **evidence in span output**, which is what makes a\ndisagreement adjudicable during validation; **scorer name and version in span metadata**, so a\nrubric revision traces to the results it changed.\n\nThe harness-vs-system split needs two destinations: system failure → a real score in `scores`;\nharness failure → the per-item **status** field, excluded from the aggregate. If both land in\n`scores`, the aggregate silently becomes \"performance on surviving items\" with no way to recover\nthe distinction.\n\nIterate the definition **inline before saving it**, then save, then re-test the *saved* version on\nthe same examples — saving is a step that can change behavior, and only the second test catches it.\nDefault the judge to a small model and escalate on measured failure, not on the suspicion that a\nbigger one would do better; judge cost is paid per item on every run, forever. Getting a saved\nevaluator onto live traffic — rule, sampling, activation, backfill — is\n`braintrust-deploy-evaluator`.\n",{"data":39,"body":40},{"name":4,"description":6},{"type":41,"children":42},"root",[43,52,75,82,102,108,227,233,268,274,309,315,338,342,376,403,429],{"type":44,"tag":45,"props":46,"children":48},"element","h1",{"id":47},"write-one-eval-scorer",[49],{"type":50,"value":51},"text","Write one eval scorer",{"type":44,"tag":53,"props":54,"children":55},"p",{},[56,58,65,67,73],{"type":50,"value":57},"Contract: ",{"type":44,"tag":59,"props":60,"children":62},"code",{"className":61},[],[63],{"type":50,"value":64},"references\u002Finteraction-contract.md",{"type":50,"value":66},". Calibration, templates, provenance: ",{"type":44,"tag":59,"props":68,"children":70},{"className":69},[],[71],{"type":50,"value":72},"references\u002Fscorer-patterns.md",{"type":50,"value":74},".",{"type":44,"tag":76,"props":77,"children":79},"h2",{"id":78},"trigger",[80],{"type":50,"value":81},"Trigger",{"type":44,"tag":83,"props":84,"children":85},"ul",{},[86,92,97],{"type":44,"tag":87,"props":88,"children":89},"li",{},[90],{"type":50,"value":91},"\"Write a scorer for this criterion.\" \u002F \"Should this be deterministic or a judge?\"",{"type":44,"tag":87,"props":93,"children":94},{},[95],{"type":50,"value":96},"Turning a behavior-spec clause or evidence-map signal into a check.",{"type":44,"tag":87,"props":98,"children":99},{},[100],{"type":50,"value":101},"A rubric producing inconsistent scores, or a scorer bundling several qualities.",{"type":44,"tag":76,"props":103,"children":105},{"id":104},"do",[106],{"type":50,"value":107},"Do",{"type":44,"tag":109,"props":110,"children":111},"ol",{},[112,139,158,184,203,222],{"type":44,"tag":87,"props":113,"children":114},{},[115,117,123,125,130,132,137],{"type":50,"value":116},"Name ",{"type":44,"tag":118,"props":119,"children":120},"strong",{},[121],{"type":50,"value":122},"one",{"type":50,"value":124}," criterion, and its output contract: a ",{"type":44,"tag":118,"props":126,"children":127},{},[128],{"type":50,"value":129},"score",{"type":50,"value":131}," (0–1, needs a numeric mapping) or\na ",{"type":44,"tag":118,"props":133,"children":134},{},[135],{"type":50,"value":136},"classification",{"type":50,"value":138}," (one label from a fixed set, needs no-match behavior). If the request\ncombines several criteria, split them before writing any code or rubric.",{"type":44,"tag":87,"props":140,"children":141},{},[142,144,149,151,157],{"type":50,"value":143},"Match the method to the evidence — and let ",{"type":44,"tag":118,"props":145,"children":146},{},[147],{"type":50,"value":148},"stakes override convenience",{"type":50,"value":150},". Objective →\ndeterministic. Subjective → anchored rubric or human. A checkable outcome on a\nsafety-critical path still needs sampled human review, because what is most likely wrong is\nthe check's ",{"type":44,"tag":152,"props":153,"children":154},"em",{},[155],{"type":50,"value":156},"scope",{"type":50,"value":74},{"type":44,"tag":87,"props":159,"children":160},{},[161,163,168,170,175,177,182],{"type":50,"value":162},"Set two ",{"type":44,"tag":118,"props":164,"children":165},{},[166],{"type":50,"value":167},"independent",{"type":50,"value":169}," axes and keep them apart. ",{"type":44,"tag":118,"props":171,"children":172},{},[173],{"type":50,"value":174},"Input scope",{"type":50,"value":176}," — span, trace, or group; how\nmuch one evaluator call sees, default trace. ",{"type":44,"tag":118,"props":178,"children":179},{},[180],{"type":50,"value":181},"Reporting level",{"type":50,"value":183}," — per-item scoring localizes\nfailures, aggregate detects regressions. Most criteria need both reporting levels.",{"type":44,"tag":87,"props":185,"children":186},{},[187,189,194,196,201],{"type":50,"value":188},"For rubrics, write criteria as ",{"type":44,"tag":118,"props":190,"children":191},{},[192],{"type":50,"value":193},"anchored examples, not descriptions",{"type":50,"value":195},", each scored\nseparately, requiring structured output that carries the evidence used. Have the model emit\n",{"type":44,"tag":118,"props":197,"children":198},{},[199],{"type":50,"value":200},"semantic classes, not numbers",{"type":50,"value":202},", and map classes to scores outside the model.",{"type":44,"tag":87,"props":204,"children":205},{},[206,208,213,215,220],{"type":50,"value":207},"Define failure handling, keeping the kinds apart: ",{"type":44,"tag":118,"props":209,"children":210},{},[211],{"type":50,"value":212},"system",{"type":50,"value":214}," failures (refusal, invalid\noutput, wrong state) are scored, never dropped; ",{"type":44,"tag":118,"props":216,"children":217},{},[218],{"type":50,"value":219},"harness",{"type":50,"value":221}," failures (your own parallelism,\nexhausted credits) are missing data in the status field.",{"type":44,"tag":87,"props":223,"children":224},{},[225],{"type":50,"value":226},"Version the scorer and pin the version in experiment metadata.",{"type":44,"tag":76,"props":228,"children":230},{"id":229},"avoid",[231],{"type":50,"value":232},"Avoid",{"type":44,"tag":83,"props":234,"children":235},{},[236,241,246,251,263],{"type":44,"tag":87,"props":237,"children":238},{},[239],{"type":50,"value":240},"Do not use a judge where objective state can be checked deterministically, or a judge from the\nsame model family as the system under test.",{"type":44,"tag":87,"props":242,"children":243},{},[244],{"type":50,"value":245},"Do not combine unrelated qualities into one score.",{"type":44,"tag":87,"props":247,"children":248},{},[249],{"type":50,"value":250},"Do not silently drop errors, refusals, or unparseable outputs.",{"type":44,"tag":87,"props":252,"children":253},{},[254,256,261],{"type":50,"value":255},"Do not tune the scorer until the numbers improve — adopting a normalizer ",{"type":44,"tag":152,"props":257,"children":258},{},[259],{"type":50,"value":260},"after",{"type":50,"value":262}," seeing which\narm it helps is scorer-side p-hacking.",{"type":44,"tag":87,"props":264,"children":265},{},[266],{"type":50,"value":267},"Do not validate the scorer here.",{"type":44,"tag":76,"props":269,"children":271},{"id":270},"check",[272],{"type":50,"value":273},"Check",{"type":44,"tag":83,"props":275,"children":276},{},[277,289,294,299,304],{"type":44,"tag":87,"props":278,"children":279},{},[280,282,287],{"type":50,"value":281},"Exactly one criterion; method justified by evidence type ",{"type":44,"tag":118,"props":283,"children":284},{},[285],{"type":50,"value":286},"and",{"type":50,"value":288}," stakes.",{"type":44,"tag":87,"props":290,"children":291},{},[292],{"type":50,"value":293},"Tested against clear successes, clear failures, edges, refusals, timeouts, parse failures.",{"type":44,"tag":87,"props":295,"children":296},{},[297],{"type":50,"value":298},"Returns interpretable evidence, not just a number.",{"type":44,"tag":87,"props":300,"children":301},{},[302],{"type":50,"value":303},"Order counterbalanced in pairwise setups; length sensitivity checked.",{"type":44,"tag":87,"props":305,"children":306},{},[307],{"type":50,"value":308},"Versioned; known artifacts documented with the arms they disadvantage.",{"type":44,"tag":76,"props":310,"children":312},{"id":311},"risk",[313],{"type":50,"value":314},"Risk",{"type":44,"tag":83,"props":316,"children":317},{},[318,323,328,333],{"type":44,"tag":87,"props":319,"children":320},{},[321],{"type":50,"value":322},"Judge bias — self-preference, position, verbosity — produces confident, invalid scores.",{"type":44,"tag":87,"props":324,"children":325},{},[326],{"type":50,"value":327},"A judge reads text the system under test wrote, and that text can address the judge directly.\nThe bias controls assume a miscalibrated judge, not one being spoken to — contract §9.",{"type":44,"tag":87,"props":329,"children":330},{},[331],{"type":50,"value":332},"A scorer without trace access can only judge the final answer, however the criterion was\nwritten.",{"type":44,"tag":87,"props":334,"children":335},{},[336],{"type":50,"value":337},"Silent exclusions are the most dangerous failure: the aggregate answers \"how did the system do\non the items that survived\" while appearing to answer the question asked.",{"type":44,"tag":76,"props":339,"children":340},{"id":8},[341],{"type":50,"value":9},{"type":44,"tag":53,"props":343,"children":344},{},[345,347,352,354,359,361,367,369,374],{"type":50,"value":346},"Deterministic criteria → ",{"type":44,"tag":118,"props":348,"children":349},{},[350],{"type":50,"value":351},"code scorers",{"type":50,"value":353},"; subjective → ",{"type":44,"tag":118,"props":355,"children":356},{},[357],{"type":50,"value":358},"rubric scorers",{"type":50,"value":360},". Shared mechanics:\n",{"type":44,"tag":59,"props":362,"children":364},{"className":363},[],[365],{"type":50,"value":366},"references\u002Fplatform-mechanics.md",{"type":50,"value":368},". ",{"type":44,"tag":118,"props":370,"children":371},{},[372],{"type":50,"value":373},"§5",{"type":50,"value":375}," is the one this stage creates rather than consumes —\none criterion, one scorer, one name, chosen here and depended on by every downstream diff.",{"type":44,"tag":53,"props":377,"children":378},{},[379,381,387,389,394,396,401],{"type":50,"value":380},"Scores in native ",{"type":44,"tag":59,"props":382,"children":384},{"className":383},[],[385],{"type":50,"value":386},"scores",{"type":50,"value":388}," (0–1); the judge's ",{"type":44,"tag":118,"props":390,"children":391},{},[392],{"type":50,"value":393},"evidence in span output",{"type":50,"value":395},", which is what makes a\ndisagreement adjudicable during validation; ",{"type":44,"tag":118,"props":397,"children":398},{},[399],{"type":50,"value":400},"scorer name and version in span metadata",{"type":50,"value":402},", so a\nrubric revision traces to the results it changed.",{"type":44,"tag":53,"props":404,"children":405},{},[406,408,413,415,420,422,427],{"type":50,"value":407},"The harness-vs-system split needs two destinations: system failure → a real score in ",{"type":44,"tag":59,"props":409,"children":411},{"className":410},[],[412],{"type":50,"value":386},{"type":50,"value":414},";\nharness failure → the per-item ",{"type":44,"tag":118,"props":416,"children":417},{},[418],{"type":50,"value":419},"status",{"type":50,"value":421}," field, excluded from the aggregate. If both land in\n",{"type":44,"tag":59,"props":423,"children":425},{"className":424},[],[426],{"type":50,"value":386},{"type":50,"value":428},", the aggregate silently becomes \"performance on surviving items\" with no way to recover\nthe distinction.",{"type":44,"tag":53,"props":430,"children":431},{},[432,434,439,441,446,448,454],{"type":50,"value":433},"Iterate the definition ",{"type":44,"tag":118,"props":435,"children":436},{},[437],{"type":50,"value":438},"inline before saving it",{"type":50,"value":440},", then save, then re-test the ",{"type":44,"tag":152,"props":442,"children":443},{},[444],{"type":50,"value":445},"saved",{"type":50,"value":447}," version on\nthe same examples — saving is a step that can change behavior, and only the second test catches it.\nDefault the judge to a small model and escalate on measured failure, not on the suspicion that a\nbigger one would do better; judge cost is paid per item on every run, forever. Getting a saved\nevaluator onto live traffic — rule, sampling, activation, backfill — is\n",{"type":44,"tag":59,"props":449,"children":451},{"className":450},[],[452],{"type":50,"value":453},"braintrust-deploy-evaluator",{"type":50,"value":74},{"items":456,"total":556},[457,473,487,501,517,531,542],{"slug":458,"name":458,"fn":459,"description":460,"org":461,"tags":462,"stars":27,"repoUrl":28,"updatedAt":472},"braintrust-analyze-eval-experiment","analyze LLM and agent eval experiments","Analyze completed LLM or agent eval experiments using uncertainty-aware and decision-relevant methods. Use to audit run completeness and pairing, calculate confidence intervals, run paired comparisons, report wins, losses, and ties, incorporate run-to-run variance, handle multiple comparisons, inspect subgroup performance, and test fragility to favorable slices. Use when results already exist and someone asks what they mean, whether a difference is real, or which model won. Do not use to design an experiment that has not yet collected results.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[463,466,467,468,469],{"name":464,"slug":465,"type":16},"Analysis","analysis",{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":470,"slug":471,"type":16},"Statistics","statistics","2026-08-20T03:53:01.13806",{"slug":474,"name":474,"fn":475,"description":476,"org":477,"tags":478,"stars":27,"repoUrl":28,"updatedAt":486},"braintrust-attribute-multi-variable-change","attribute performance changes to multiple variables","Attribute an observed change when several things moved at once — model plus prompt plus tools, a provider migration, a framework upgrade, or a vendor swap that bundles serving stack with model. Use when asked which part of a change caused the result, when a comparison's arms differ in more than one way, when a treatment has no uniform implementation across vendors, or when a serving-stack difference is confounded with a model difference. Do not use for a clean single-variable comparison, or to design an experiment that has not yet run.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[479,480,481,482,483],{"name":464,"slug":465,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":484,"slug":485,"type":16},"Performance","performance","2026-08-20T03:53:40.036077",{"slug":488,"name":488,"fn":489,"description":490,"org":491,"tags":492,"stars":27,"repoUrl":28,"updatedAt":500},"braintrust-build-eval-dataset","create and manage LLM eval datasets","Create, edit, audit, or compare eval datasets for LLM applications and agents, including target-population definition, case sourcing from production traces, stratified sampling, label provenance and label audits, expected values as constraints for open-ended tasks, dev\u002Ftest splits, contamination and leakage controls, headroom checks, refresh policy, and datasheets. Use when working on the content or lifecycle of an eval dataset. Do not use for sample-size or power calculations, scorer implementation, or open-ended adversarial failure discovery.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[493,494,495,498,499],{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},{"name":496,"slug":497,"type":16},"Datasets","datasets",{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"2026-08-20T03:53:33.304815",{"slug":502,"name":502,"fn":503,"description":504,"org":505,"tags":506,"stars":27,"repoUrl":28,"updatedAt":516},"braintrust-define-eval-objective","define LLM evaluation objectives","Create, edit, or audit an eval objective by working backward from a product decision to the target outcome, construct, population, intended claim, and verification-versus-validation questions. Use when a team is unsure what an eval should establish, asks \"what are we actually trying to measure,\" \"is this eval measuring the right thing,\" \"does this benchmark support our claim,\" or needs to turn a product goal into an eval objective and state which claims are out of scope. Do not use to select detailed metrics, design datasets, or implement scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[507,508,509,510,513],{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":511,"slug":512,"type":16},"Product Management","product-management",{"name":514,"slug":515,"type":16},"Strategy","strategy","2026-08-20T03:53:00.07097",{"slug":518,"name":518,"fn":519,"description":520,"org":521,"tags":522,"stars":27,"repoUrl":28,"updatedAt":530},"braintrust-define-eval-release-gate","configure release gates for LLM applications","Create, edit, audit, or apply release gates for LLM applications and agents. Use to combine minimum meaningful improvement, statistical significance, regression rate, subgroup consistency, worst-run stability, all-attempts reliability, safety upper bounds, latency, and cost into an explicit ship-or-hold policy, to turn metrics into a CI gate, or to explain why a candidate failed a gate and what evidence would justify reconsideration. Do not use for general result analysis without a deployment decision.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[523,524,525,528,529],{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},{"name":526,"slug":527,"type":16},"CI\u002FCD","ci-cd",{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"2026-08-20T03:53:15.686158",{"slug":453,"name":453,"fn":532,"description":533,"org":534,"tags":535,"stars":27,"repoUrl":28,"updatedAt":541},"deploy evaluators to Braintrust","Take a validated scorer or classifier from definition to running instrument in Braintrust — scope selection, inline testing before saving, saving as an evaluator, attaching an online-scoring rule, activating it for new traffic, and backfilling history with a rewind. Use when a scorer needs to actually run against production logs, when an online-scoring rule needs to be created or changed, or when historical traces need scoring. Do not use to decide what the scorer should measure, to write its rubric, or to establish that it agrees with human judgment.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[536,537,540],{"name":9,"slug":8,"type":16},{"name":538,"slug":539,"type":16},"Deployment","deployment",{"name":18,"slug":19,"type":16},"2026-08-20T03:53:32.558937",{"slug":543,"name":543,"fn":544,"description":545,"org":546,"tags":547,"stars":27,"repoUrl":28,"updatedAt":555},"braintrust-design-eval-experiment","design controlled LLM eval experiments","Design or audit controlled eval experiments for model, prompt, retrieval, tool, guardrail, or agent-architecture changes. Use before data collection to state directional and minimum-effect hypotheses, name independent, dependent, and control variables including the serving environment and tool surface, choose paired designs, set repetitions and allocation, distinguish exploratory from confirmatory comparisons, and pre-specify stopping, exclusion, multiplicity, and analysis rules. Do not use primarily to analyze results already collected.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[548,549,550,553,554],{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":551,"slug":552,"type":16},"Experiments","experiments",{"name":14,"slug":15,"type":16},{"name":514,"slug":515,"type":16},"2026-08-20T03:53:36.534554",24,{"items":558,"total":683},[559,575,583,591,599,607,615,621,629,645,657,669],{"slug":560,"name":560,"fn":561,"description":562,"org":563,"tags":564,"stars":572,"repoUrl":573,"updatedAt":574},"troubleshoot-braintrust-mcp","configure and troubleshoot Braintrust MCP servers","This plugin auto-configures a \"braintrust\" MCP server. If you can't see it or reach it, activate this skill\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[565,566,569],{"name":9,"slug":8,"type":16},{"name":567,"slug":568,"type":16},"Debugging","debugging",{"name":570,"slug":571,"type":16},"MCP","mcp",18,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Fbraintrust-claude-plugin","2026-07-12T08:36:13.889274",{"slug":458,"name":458,"fn":459,"description":460,"org":576,"tags":577,"stars":27,"repoUrl":28,"updatedAt":472},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[578,579,580,581,582],{"name":464,"slug":465,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":470,"slug":471,"type":16},{"slug":474,"name":474,"fn":475,"description":476,"org":584,"tags":585,"stars":27,"repoUrl":28,"updatedAt":486},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[586,587,588,589,590],{"name":464,"slug":465,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":484,"slug":485,"type":16},{"slug":488,"name":488,"fn":489,"description":490,"org":592,"tags":593,"stars":27,"repoUrl":28,"updatedAt":500},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[594,595,596,597,598],{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},{"name":496,"slug":497,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":502,"name":502,"fn":503,"description":504,"org":600,"tags":601,"stars":27,"repoUrl":28,"updatedAt":516},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[602,603,604,605,606],{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":511,"slug":512,"type":16},{"name":514,"slug":515,"type":16},{"slug":518,"name":518,"fn":519,"description":520,"org":608,"tags":609,"stars":27,"repoUrl":28,"updatedAt":530},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[610,611,612,613,614],{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},{"name":526,"slug":527,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":453,"name":453,"fn":532,"description":533,"org":616,"tags":617,"stars":27,"repoUrl":28,"updatedAt":541},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[618,619,620],{"name":9,"slug":8,"type":16},{"name":538,"slug":539,"type":16},{"name":18,"slug":19,"type":16},{"slug":543,"name":543,"fn":544,"description":545,"org":622,"tags":623,"stars":27,"repoUrl":28,"updatedAt":555},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[624,625,626,627,628],{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":551,"slug":552,"type":16},{"name":14,"slug":15,"type":16},{"name":514,"slug":515,"type":16},{"slug":630,"name":630,"fn":631,"description":632,"org":633,"tags":634,"stars":27,"repoUrl":28,"updatedAt":644},"braintrust-design-eval-instrumentation","design trace and evaluation dataset schemas","Design the trace and eval-dataset schema for an LLM app or agent, and wire the system to emit it. Use when deciding what to log, designing a trace schema, setting up tracing or observability before evals, or when failures cannot be debugged or sliced from existing traces — covering inputs, outputs, spans for tool and LLM calls, state changes, metadata, resolved configuration, serving path, tool manifest, per-item status, attachments, and subgroup variables. Do not use to decide what the evidence should mean, or to write scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[635,636,637,638,641],{"name":9,"slug":8,"type":16},{"name":496,"slug":497,"type":16},{"name":18,"slug":19,"type":16},{"name":639,"slug":640,"type":16},"Observability","observability",{"name":642,"slug":643,"type":16},"Tracing","tracing","2026-08-20T03:53:37.274703",{"slug":646,"name":646,"fn":647,"description":648,"org":649,"tags":650,"stars":27,"repoUrl":28,"updatedAt":656},"braintrust-design-eval-metric-bundle","create multi-objective evaluation metric bundles","Create, edit, audit, or compare a multi-objective eval metric bundle covering product quality, safety, reliability, latency, and cost. Use to choose metrics for an eval, define a goodness bundle, distinguish optimization metrics from non-regression guardrails, expose tradeoffs, audit a KPI or single composite score for Goodhart and metric-gaming risk, or answer \"what should improve and what must not regress.\" Do not use to design trace schemas, build datasets, or implement scoring methods.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[651,652,653,654,655],{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":484,"slug":485,"type":16},"2026-08-20T03:53:16.078609",{"slug":658,"name":658,"fn":659,"description":660,"org":661,"tags":662,"stars":27,"repoUrl":28,"updatedAt":668},"braintrust-design-human-eval-review","design human evaluation and review workflows","Design or audit human evaluation workflows and golden datasets for LLM applications and agents. Use to set up expert review, select review cases, write reviewer instructions, assign raters, capture rationales and confidence, measure inter-rater agreement with kappa or alpha, adjudicate disagreements, and preserve reviewed examples with provenance as a versioned reference set. Do not use to elicit the criteria or rubric in the first place, to validate a scorer once reference labels exist, or to implement the scorer.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[663,664,665,666,667],{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},{"name":496,"slug":497,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"2026-08-20T03:53:36.900129",{"slug":670,"name":670,"fn":671,"description":672,"org":673,"tags":674,"stars":27,"repoUrl":28,"updatedAt":682},"braintrust-discover-agent-failures","identify and classify agent failure modes","Search open-endedly for unanticipated agent failure modes and convert them into a named taxonomy and durable regression items. Use for requests to find out what goes wrong, surface unknown or silent failures, do error analysis over traces, cluster and triage production failures, or build a failure taxonomy — where the goal is discovering modes nobody thought to test rather than measuring a predefined criterion. Produces datasets and taxonomies, not headline scores. Do not use for adversarial attacks against a threat model, or to measure a known criterion.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[675,676,677,678,679],{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},{"name":567,"slug":568,"type":16},{"name":18,"slug":19,"type":16},{"name":680,"slug":681,"type":16},"Triage","triage","2026-08-20T03:53:00.423941",27]