[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-braintrust-braintrust-validate-eval-scorer":3,"mdc-1mauks-key":38,"related-org-braintrust-braintrust-validate-eval-scorer":411,"related-repo-braintrust-braintrust-validate-eval-scorer":584},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":27,"repoUrl":28,"updatedAt":29,"license":30,"forks":31,"topics":32,"repo":33,"sourceUrl":36,"mdContent":37},"braintrust-validate-eval-scorer","validate automated eval scorers","Validate automated eval scorers and LLM judges against expert-reviewed reference data. Use to compare scorer output with human labels, calculate agreement (kappa, alpha) with uncertainty, inspect confusion by class and severity, analyze subgroup failures, test shortcut and gaming cases, propagate scorer error into headline numbers, document blind spots, and decide whether a scorer is fit for exploration, trend monitoring, or release gating. Do not use to create the initial scorer or to design the human review workflow.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"braintrust","Braintrust","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fbraintrust.png","braintrustdata",[13,17,20,23,24],{"name":14,"slug":15,"type":16},"LLM","llm","tag",{"name":18,"slug":19,"type":16},"Evals","evals",{"name":21,"slug":22,"type":16},"Quality Assurance","quality-assurance",{"name":9,"slug":8,"type":16},{"name":25,"slug":26,"type":16},"Debugging","debugging",7,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library","2026-08-20T03:52:59.694515",null,0,[],{"repoUrl":28,"stars":27,"forks":31,"topics":34,"description":35},[],"Braintrust eval skills library","https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library\u002Ftree\u002FHEAD\u002Fskills\u002Fbraintrust-validate-eval-scorer","---\nname: braintrust-validate-eval-scorer\ndescription: >-\n  Validate automated eval scorers and LLM judges against expert-reviewed reference data. Use to\n  compare scorer output with human labels, calculate agreement (kappa, alpha) with uncertainty,\n  inspect confusion by class and severity, analyze subgroup failures, test shortcut and gaming\n  cases, propagate scorer error into headline numbers, document blind spots, and decide whether a\n  scorer is fit for exploration, trend monitoring, or release gating. Do not use to create the\n  initial scorer or to design the human review workflow.\n---\n\n# Validate the scorer\n\nContract: `references\u002Finteraction-contract.md`. Calibration, templates, provenance: `references\u002Fvalidation-recipe.md`.\n\n## Trigger\n\n- \"Validate this LLM judge.\" \u002F \"Can this scorer gate releases?\"\n- A scorer already gating releases that has never been checked against humans.\n- A scorer-version bump — validation is a regression test for the instrument.\n\n## Do\n\n1. Name the **reference tier** before computing anything. Adjudicated human labels are the only\n   tier the fitness bands in `reference.md` are calibrated against. A strong-model reference is a\n   cheaper tier that supports iteration and nothing else — say which one you have, in the verdict.\n2. Verify alignment: scorer outputs and reference labels must line up at the item **and\n   criterion** level. Misalignment invalidates every number after it.\n3. Report **agreement (κ or α) with uncertainty**, not raw accuracy. Fitness bands and their\n   hedges are in `reference.md`.\n4. Lead with the most decision-relevant **false acceptance** before any aggregate, and enumerate\n   the dangerous cells case by case.\n5. Break errors down by class and **severity**. Confusing \"excellent\" with \"good\" may be fine;\n   missing harmful outputs is disqualifying for gating.\n6. Test sensitivity and shortcuts: inject known regressions and improvements and confirm the\n   scorer moves; probe whether length, confidence, or polish raise the score independent of\n   quality; probe whether text addressed to the judge moves it; slice agreement by subgroup.\n7. **Propagate scorer error into headline numbers**, then state the fitness verdict: **allowed\n   uses**, **prohibited uses**, **revalidation triggers**.\n\n## Avoid\n\n- Do not rely on raw accuracy — with skewed classes it is high and meaningless.\n- Do not read the fitness bands against a model-generated reference; they are calibrated for\n  adjudicated human labels and mean something weaker here.\n- Do not validate on the same examples used to tune the scorer.\n- Do not report one aggregate agreement figure as the verdict.\n- Do not repair the scorer here unless asked; audit and repair are different modes.\n- Do not build the scorer or produce the reference labels here; both must already exist.\n\n## Check\n\n- False acceptances and rejections separate, severity-weighted; dangerous cells enumerated\n  individually.\n- Subgroup agreement reported; shortcut probes run.\n- Blind spots documented as understood failure modes.\n- Verdict names allowed uses, prohibited uses, revalidation triggers.\n- Scorer error reflected in the uncertainty of any headline number it produces.\n\n## Risk\n\n- High average agreement conceals rare severe misses — exactly the errors that make a scorer\n  unsafe for gating.\n- A scorer validated once and trusted indefinitely drifts; validity is dated, not permanent.\n- Validating against a golden set with systematic label error certifies agreement with the error.\n  A model-generated reference is the acute case: agreement then measures how well the cheap\n  instrument imitates the expensive one, which says nothing about either tracking the construct.\n- The goal is not a perfect scorer but one that tracks the distinctions the team cares about and\n  **fails in understood ways**.\n\n## Braintrust\n\nThe validation run is an experiment, and following the shape exactly is what makes it repeatable\nrather than a one-off notebook: golden set as a **versioned dataset** → run the candidate scorer\nover it **as an experiment** → **diff against human labels** → compute κ and confusion heatmaps\nin **custom columns or an exported notebook** (`references\u002Fplatform-mechanics.md` §8) → **re-run\non every scorer-version bump**.\n\nThis only works if the scorer wrote its **evidence into span output**; without it a disagreement\nshows two numbers and no way to adjudicate. **Scorer name and version in span metadata** is what\nlets a report name the version it validated.\n\nPut the verdict where the gate reads: allowed and prohibited uses in the **scorer's own\ndescription**, validation date and golden-set version in the experiment description. Where a\nscorer is prohibited from gating, do not attach it to a gate check at all — absence is more\nreliable than a warning. Route disagreement items back into the review queue.\n",{"data":39,"body":40},{"name":4,"description":6},{"type":41,"children":42},"root",[43,52,75,82,102,108,221,227,260,266,294,300,329,333,380,399],{"type":44,"tag":45,"props":46,"children":48},"element","h1",{"id":47},"validate-the-scorer",[49],{"type":50,"value":51},"text","Validate the scorer",{"type":44,"tag":53,"props":54,"children":55},"p",{},[56,58,65,67,73],{"type":50,"value":57},"Contract: ",{"type":44,"tag":59,"props":60,"children":62},"code",{"className":61},[],[63],{"type":50,"value":64},"references\u002Finteraction-contract.md",{"type":50,"value":66},". Calibration, templates, provenance: ",{"type":44,"tag":59,"props":68,"children":70},{"className":69},[],[71],{"type":50,"value":72},"references\u002Fvalidation-recipe.md",{"type":50,"value":74},".",{"type":44,"tag":76,"props":77,"children":79},"h2",{"id":78},"trigger",[80],{"type":50,"value":81},"Trigger",{"type":44,"tag":83,"props":84,"children":85},"ul",{},[86,92,97],{"type":44,"tag":87,"props":88,"children":89},"li",{},[90],{"type":50,"value":91},"\"Validate this LLM judge.\" \u002F \"Can this scorer gate releases?\"",{"type":44,"tag":87,"props":93,"children":94},{},[95],{"type":50,"value":96},"A scorer already gating releases that has never been checked against humans.",{"type":44,"tag":87,"props":98,"children":99},{},[100],{"type":50,"value":101},"A scorer-version bump — validation is a regression test for the instrument.",{"type":44,"tag":76,"props":103,"children":105},{"id":104},"do",[106],{"type":50,"value":107},"Do",{"type":44,"tag":109,"props":110,"children":111},"ol",{},[112,133,145,163,175,187,192],{"type":44,"tag":87,"props":113,"children":114},{},[115,117,123,125,131],{"type":50,"value":116},"Name the ",{"type":44,"tag":118,"props":119,"children":120},"strong",{},[121],{"type":50,"value":122},"reference tier",{"type":50,"value":124}," before computing anything. Adjudicated human labels are the only\ntier the fitness bands in ",{"type":44,"tag":59,"props":126,"children":128},{"className":127},[],[129],{"type":50,"value":130},"reference.md",{"type":50,"value":132}," are calibrated against. A strong-model reference is a\ncheaper tier that supports iteration and nothing else — say which one you have, in the verdict.",{"type":44,"tag":87,"props":134,"children":135},{},[136,138,143],{"type":50,"value":137},"Verify alignment: scorer outputs and reference labels must line up at the item ",{"type":44,"tag":118,"props":139,"children":140},{},[141],{"type":50,"value":142},"and\ncriterion",{"type":50,"value":144}," level. Misalignment invalidates every number after it.",{"type":44,"tag":87,"props":146,"children":147},{},[148,150,155,157,162],{"type":50,"value":149},"Report ",{"type":44,"tag":118,"props":151,"children":152},{},[153],{"type":50,"value":154},"agreement (κ or α) with uncertainty",{"type":50,"value":156},", not raw accuracy. Fitness bands and their\nhedges are in ",{"type":44,"tag":59,"props":158,"children":160},{"className":159},[],[161],{"type":50,"value":130},{"type":50,"value":74},{"type":44,"tag":87,"props":164,"children":165},{},[166,168,173],{"type":50,"value":167},"Lead with the most decision-relevant ",{"type":44,"tag":118,"props":169,"children":170},{},[171],{"type":50,"value":172},"false acceptance",{"type":50,"value":174}," before any aggregate, and enumerate\nthe dangerous cells case by case.",{"type":44,"tag":87,"props":176,"children":177},{},[178,180,185],{"type":50,"value":179},"Break errors down by class and ",{"type":44,"tag":118,"props":181,"children":182},{},[183],{"type":50,"value":184},"severity",{"type":50,"value":186},". Confusing \"excellent\" with \"good\" may be fine;\nmissing harmful outputs is disqualifying for gating.",{"type":44,"tag":87,"props":188,"children":189},{},[190],{"type":50,"value":191},"Test sensitivity and shortcuts: inject known regressions and improvements and confirm the\nscorer moves; probe whether length, confidence, or polish raise the score independent of\nquality; probe whether text addressed to the judge moves it; slice agreement by subgroup.",{"type":44,"tag":87,"props":193,"children":194},{},[195,200,202,207,209,214,215,220],{"type":44,"tag":118,"props":196,"children":197},{},[198],{"type":50,"value":199},"Propagate scorer error into headline numbers",{"type":50,"value":201},", then state the fitness verdict: ",{"type":44,"tag":118,"props":203,"children":204},{},[205],{"type":50,"value":206},"allowed\nuses",{"type":50,"value":208},", ",{"type":44,"tag":118,"props":210,"children":211},{},[212],{"type":50,"value":213},"prohibited uses",{"type":50,"value":208},{"type":44,"tag":118,"props":216,"children":217},{},[218],{"type":50,"value":219},"revalidation triggers",{"type":50,"value":74},{"type":44,"tag":76,"props":222,"children":224},{"id":223},"avoid",[225],{"type":50,"value":226},"Avoid",{"type":44,"tag":83,"props":228,"children":229},{},[230,235,240,245,250,255],{"type":44,"tag":87,"props":231,"children":232},{},[233],{"type":50,"value":234},"Do not rely on raw accuracy — with skewed classes it is high and meaningless.",{"type":44,"tag":87,"props":236,"children":237},{},[238],{"type":50,"value":239},"Do not read the fitness bands against a model-generated reference; they are calibrated for\nadjudicated human labels and mean something weaker here.",{"type":44,"tag":87,"props":241,"children":242},{},[243],{"type":50,"value":244},"Do not validate on the same examples used to tune the scorer.",{"type":44,"tag":87,"props":246,"children":247},{},[248],{"type":50,"value":249},"Do not report one aggregate agreement figure as the verdict.",{"type":44,"tag":87,"props":251,"children":252},{},[253],{"type":50,"value":254},"Do not repair the scorer here unless asked; audit and repair are different modes.",{"type":44,"tag":87,"props":256,"children":257},{},[258],{"type":50,"value":259},"Do not build the scorer or produce the reference labels here; both must already exist.",{"type":44,"tag":76,"props":261,"children":263},{"id":262},"check",[264],{"type":50,"value":265},"Check",{"type":44,"tag":83,"props":267,"children":268},{},[269,274,279,284,289],{"type":44,"tag":87,"props":270,"children":271},{},[272],{"type":50,"value":273},"False acceptances and rejections separate, severity-weighted; dangerous cells enumerated\nindividually.",{"type":44,"tag":87,"props":275,"children":276},{},[277],{"type":50,"value":278},"Subgroup agreement reported; shortcut probes run.",{"type":44,"tag":87,"props":280,"children":281},{},[282],{"type":50,"value":283},"Blind spots documented as understood failure modes.",{"type":44,"tag":87,"props":285,"children":286},{},[287],{"type":50,"value":288},"Verdict names allowed uses, prohibited uses, revalidation triggers.",{"type":44,"tag":87,"props":290,"children":291},{},[292],{"type":50,"value":293},"Scorer error reflected in the uncertainty of any headline number it produces.",{"type":44,"tag":76,"props":295,"children":297},{"id":296},"risk",[298],{"type":50,"value":299},"Risk",{"type":44,"tag":83,"props":301,"children":302},{},[303,308,313,318],{"type":44,"tag":87,"props":304,"children":305},{},[306],{"type":50,"value":307},"High average agreement conceals rare severe misses — exactly the errors that make a scorer\nunsafe for gating.",{"type":44,"tag":87,"props":309,"children":310},{},[311],{"type":50,"value":312},"A scorer validated once and trusted indefinitely drifts; validity is dated, not permanent.",{"type":44,"tag":87,"props":314,"children":315},{},[316],{"type":50,"value":317},"Validating against a golden set with systematic label error certifies agreement with the error.\nA model-generated reference is the acute case: agreement then measures how well the cheap\ninstrument imitates the expensive one, which says nothing about either tracking the construct.",{"type":44,"tag":87,"props":319,"children":320},{},[321,323,328],{"type":50,"value":322},"The goal is not a perfect scorer but one that tracks the distinctions the team cares about and\n",{"type":44,"tag":118,"props":324,"children":325},{},[326],{"type":50,"value":327},"fails in understood ways",{"type":50,"value":74},{"type":44,"tag":76,"props":330,"children":331},{"id":8},[332],{"type":50,"value":9},{"type":44,"tag":53,"props":334,"children":335},{},[336,338,343,345,350,352,357,359,364,366,372,374,379],{"type":50,"value":337},"The validation run is an experiment, and following the shape exactly is what makes it repeatable\nrather than a one-off notebook: golden set as a ",{"type":44,"tag":118,"props":339,"children":340},{},[341],{"type":50,"value":342},"versioned dataset",{"type":50,"value":344}," → run the candidate scorer\nover it ",{"type":44,"tag":118,"props":346,"children":347},{},[348],{"type":50,"value":349},"as an experiment",{"type":50,"value":351}," → ",{"type":44,"tag":118,"props":353,"children":354},{},[355],{"type":50,"value":356},"diff against human labels",{"type":50,"value":358}," → compute κ and confusion heatmaps\nin ",{"type":44,"tag":118,"props":360,"children":361},{},[362],{"type":50,"value":363},"custom columns or an exported notebook",{"type":50,"value":365}," (",{"type":44,"tag":59,"props":367,"children":369},{"className":368},[],[370],{"type":50,"value":371},"references\u002Fplatform-mechanics.md",{"type":50,"value":373}," §8) → ",{"type":44,"tag":118,"props":375,"children":376},{},[377],{"type":50,"value":378},"re-run\non every scorer-version bump",{"type":50,"value":74},{"type":44,"tag":53,"props":381,"children":382},{},[383,385,390,392,397],{"type":50,"value":384},"This only works if the scorer wrote its ",{"type":44,"tag":118,"props":386,"children":387},{},[388],{"type":50,"value":389},"evidence into span output",{"type":50,"value":391},"; without it a disagreement\nshows two numbers and no way to adjudicate. ",{"type":44,"tag":118,"props":393,"children":394},{},[395],{"type":50,"value":396},"Scorer name and version in span metadata",{"type":50,"value":398}," is what\nlets a report name the version it validated.",{"type":44,"tag":53,"props":400,"children":401},{},[402,404,409],{"type":50,"value":403},"Put the verdict where the gate reads: allowed and prohibited uses in the ",{"type":44,"tag":118,"props":405,"children":406},{},[407],{"type":50,"value":408},"scorer's own\ndescription",{"type":50,"value":410},", validation date and golden-set version in the experiment description. Where a\nscorer is prohibited from gating, do not attach it to a gate check at all — absence is more\nreliable than a warning. Route disagreement items back into the review queue.",{"items":412,"total":583},[413,427,443,457,473,489,503,515,529,545,557,569],{"slug":414,"name":414,"fn":415,"description":416,"org":417,"tags":418,"stars":424,"repoUrl":425,"updatedAt":426},"troubleshoot-braintrust-mcp","configure and troubleshoot Braintrust MCP servers","This plugin auto-configures a \"braintrust\" MCP server. If you can't see it or reach it, activate this skill\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[419,420,421],{"name":9,"slug":8,"type":16},{"name":25,"slug":26,"type":16},{"name":422,"slug":423,"type":16},"MCP","mcp",18,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Fbraintrust-claude-plugin","2026-07-12T08:36:13.889274",{"slug":428,"name":428,"fn":429,"description":430,"org":431,"tags":432,"stars":27,"repoUrl":28,"updatedAt":442},"braintrust-analyze-eval-experiment","analyze LLM and agent eval experiments","Analyze completed LLM or agent eval experiments using uncertainty-aware and decision-relevant methods. Use to audit run completeness and pairing, calculate confidence intervals, run paired comparisons, report wins, losses, and ties, incorporate run-to-run variance, handle multiple comparisons, inspect subgroup performance, and test fragility to favorable slices. Use when results already exist and someone asks what they mean, whether a difference is real, or which model won. Do not use to design an experiment that has not yet collected results.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[433,436,437,438,439],{"name":434,"slug":435,"type":16},"Analysis","analysis",{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":440,"slug":441,"type":16},"Statistics","statistics","2026-08-20T03:53:01.13806",{"slug":444,"name":444,"fn":445,"description":446,"org":447,"tags":448,"stars":27,"repoUrl":28,"updatedAt":456},"braintrust-attribute-multi-variable-change","attribute performance changes to multiple variables","Attribute an observed change when several things moved at once — model plus prompt plus tools, a provider migration, a framework upgrade, or a vendor swap that bundles serving stack with model. Use when asked which part of a change caused the result, when a comparison's arms differ in more than one way, when a treatment has no uniform implementation across vendors, or when a serving-stack difference is confounded with a model difference. Do not use for a clean single-variable comparison, or to design an experiment that has not yet run.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[449,450,451,452,453],{"name":434,"slug":435,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":454,"slug":455,"type":16},"Performance","performance","2026-08-20T03:53:40.036077",{"slug":458,"name":458,"fn":459,"description":460,"org":461,"tags":462,"stars":27,"repoUrl":28,"updatedAt":472},"braintrust-build-eval-dataset","create and manage LLM eval datasets","Create, edit, audit, or compare eval datasets for LLM applications and agents, including target-population definition, case sourcing from production traces, stratified sampling, label provenance and label audits, expected values as constraints for open-ended tasks, dev\u002Ftest splits, contamination and leakage controls, headroom checks, refresh policy, and datasheets. Use when working on the content or lifecycle of an eval dataset. Do not use for sample-size or power calculations, scorer implementation, or open-ended adversarial failure discovery.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[463,466,467,470,471],{"name":464,"slug":465,"type":16},"Agents","agents",{"name":9,"slug":8,"type":16},{"name":468,"slug":469,"type":16},"Datasets","datasets",{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"2026-08-20T03:53:33.304815",{"slug":474,"name":474,"fn":475,"description":476,"org":477,"tags":478,"stars":27,"repoUrl":28,"updatedAt":488},"braintrust-define-eval-objective","define LLM evaluation objectives","Create, edit, or audit an eval objective by working backward from a product decision to the target outcome, construct, population, intended claim, and verification-versus-validation questions. Use when a team is unsure what an eval should establish, asks \"what are we actually trying to measure,\" \"is this eval measuring the right thing,\" \"does this benchmark support our claim,\" or needs to turn a product goal into an eval objective and state which claims are out of scope. Do not use to select detailed metrics, design datasets, or implement scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[479,480,481,482,485],{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":483,"slug":484,"type":16},"Product Management","product-management",{"name":486,"slug":487,"type":16},"Strategy","strategy","2026-08-20T03:53:00.07097",{"slug":490,"name":490,"fn":491,"description":492,"org":493,"tags":494,"stars":27,"repoUrl":28,"updatedAt":502},"braintrust-define-eval-release-gate","configure release gates for LLM applications","Create, edit, audit, or apply release gates for LLM applications and agents. Use to combine minimum meaningful improvement, statistical significance, regression rate, subgroup consistency, worst-run stability, all-attempts reliability, safety upper bounds, latency, and cost into an explicit ship-or-hold policy, to turn metrics into a CI gate, or to explain why a candidate failed a gate and what evidence would justify reconsideration. Do not use for general result analysis without a deployment decision.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[495,496,497,500,501],{"name":464,"slug":465,"type":16},{"name":9,"slug":8,"type":16},{"name":498,"slug":499,"type":16},"CI\u002FCD","ci-cd",{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"2026-08-20T03:53:15.686158",{"slug":504,"name":504,"fn":505,"description":506,"org":507,"tags":508,"stars":27,"repoUrl":28,"updatedAt":514},"braintrust-deploy-evaluator","deploy evaluators to Braintrust","Take a validated scorer or classifier from definition to running instrument in Braintrust — scope selection, inline testing before saving, saving as an evaluator, attaching an online-scoring rule, activating it for new traffic, and backfilling history with a rewind. Use when a scorer needs to actually run against production logs, when an online-scoring rule needs to be created or changed, or when historical traces need scoring. Do not use to decide what the scorer should measure, to write its rubric, or to establish that it agrees with human judgment.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[509,510,513],{"name":9,"slug":8,"type":16},{"name":511,"slug":512,"type":16},"Deployment","deployment",{"name":18,"slug":19,"type":16},"2026-08-20T03:53:32.558937",{"slug":516,"name":516,"fn":517,"description":518,"org":519,"tags":520,"stars":27,"repoUrl":28,"updatedAt":528},"braintrust-design-eval-experiment","design controlled LLM eval experiments","Design or audit controlled eval experiments for model, prompt, retrieval, tool, guardrail, or agent-architecture changes. Use before data collection to state directional and minimum-effect hypotheses, name independent, dependent, and control variables including the serving environment and tool surface, choose paired designs, set repetitions and allocation, distinguish exploratory from confirmatory comparisons, and pre-specify stopping, exclusion, multiplicity, and analysis rules. Do not use primarily to analyze results already collected.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[521,522,523,526,527],{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":524,"slug":525,"type":16},"Experiments","experiments",{"name":14,"slug":15,"type":16},{"name":486,"slug":487,"type":16},"2026-08-20T03:53:36.534554",{"slug":530,"name":530,"fn":531,"description":532,"org":533,"tags":534,"stars":27,"repoUrl":28,"updatedAt":544},"braintrust-design-eval-instrumentation","design trace and evaluation dataset schemas","Design the trace and eval-dataset schema for an LLM app or agent, and wire the system to emit it. Use when deciding what to log, designing a trace schema, setting up tracing or observability before evals, or when failures cannot be debugged or sliced from existing traces — covering inputs, outputs, spans for tool and LLM calls, state changes, metadata, resolved configuration, serving path, tool manifest, per-item status, attachments, and subgroup variables. Do not use to decide what the evidence should mean, or to write scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[535,536,537,538,541],{"name":9,"slug":8,"type":16},{"name":468,"slug":469,"type":16},{"name":18,"slug":19,"type":16},{"name":539,"slug":540,"type":16},"Observability","observability",{"name":542,"slug":543,"type":16},"Tracing","tracing","2026-08-20T03:53:37.274703",{"slug":546,"name":546,"fn":547,"description":548,"org":549,"tags":550,"stars":27,"repoUrl":28,"updatedAt":556},"braintrust-design-eval-metric-bundle","create multi-objective evaluation metric bundles","Create, edit, audit, or compare a multi-objective eval metric bundle covering product quality, safety, reliability, latency, and cost. Use to choose metrics for an eval, define a goodness bundle, distinguish optimization metrics from non-regression guardrails, expose tradeoffs, audit a KPI or single composite score for Goodhart and metric-gaming risk, or answer \"what should improve and what must not regress.\" Do not use to design trace schemas, build datasets, or implement scoring methods.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[551,552,553,554,555],{"name":464,"slug":465,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":454,"slug":455,"type":16},"2026-08-20T03:53:16.078609",{"slug":558,"name":558,"fn":559,"description":560,"org":561,"tags":562,"stars":27,"repoUrl":28,"updatedAt":568},"braintrust-design-human-eval-review","design human evaluation and review workflows","Design or audit human evaluation workflows and golden datasets for LLM applications and agents. Use to set up expert review, select review cases, write reviewer instructions, assign raters, capture rationales and confidence, measure inter-rater agreement with kappa or alpha, adjudicate disagreements, and preserve reviewed examples with provenance as a versioned reference set. Do not use to elicit the criteria or rubric in the first place, to validate a scorer once reference labels exist, or to implement the scorer.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[563,564,565,566,567],{"name":464,"slug":465,"type":16},{"name":9,"slug":8,"type":16},{"name":468,"slug":469,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"2026-08-20T03:53:36.900129",{"slug":570,"name":570,"fn":571,"description":572,"org":573,"tags":574,"stars":27,"repoUrl":28,"updatedAt":582},"braintrust-discover-agent-failures","identify and classify agent failure modes","Search open-endedly for unanticipated agent failure modes and convert them into a named taxonomy and durable regression items. Use for requests to find out what goes wrong, surface unknown or silent failures, do error analysis over traces, cluster and triage production failures, or build a failure taxonomy — where the goal is discovering modes nobody thought to test rather than measuring a predefined criterion. Produces datasets and taxonomies, not headline scores. Do not use for adversarial attacks against a threat model, or to measure a known criterion.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[575,576,577,578,579],{"name":464,"slug":465,"type":16},{"name":9,"slug":8,"type":16},{"name":25,"slug":26,"type":16},{"name":18,"slug":19,"type":16},{"name":580,"slug":581,"type":16},"Triage","triage","2026-08-20T03:53:00.423941",27,{"items":585,"total":640},[586,594,602,610,618,626,632],{"slug":428,"name":428,"fn":429,"description":430,"org":587,"tags":588,"stars":27,"repoUrl":28,"updatedAt":442},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[589,590,591,592,593],{"name":434,"slug":435,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":440,"slug":441,"type":16},{"slug":444,"name":444,"fn":445,"description":446,"org":595,"tags":596,"stars":27,"repoUrl":28,"updatedAt":456},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[597,598,599,600,601],{"name":434,"slug":435,"type":16},{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":454,"slug":455,"type":16},{"slug":458,"name":458,"fn":459,"description":460,"org":603,"tags":604,"stars":27,"repoUrl":28,"updatedAt":472},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[605,606,607,608,609],{"name":464,"slug":465,"type":16},{"name":9,"slug":8,"type":16},{"name":468,"slug":469,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":474,"name":474,"fn":475,"description":476,"org":611,"tags":612,"stars":27,"repoUrl":28,"updatedAt":488},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[613,614,615,616,617],{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":483,"slug":484,"type":16},{"name":486,"slug":487,"type":16},{"slug":490,"name":490,"fn":491,"description":492,"org":619,"tags":620,"stars":27,"repoUrl":28,"updatedAt":502},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[621,622,623,624,625],{"name":464,"slug":465,"type":16},{"name":9,"slug":8,"type":16},{"name":498,"slug":499,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":504,"name":504,"fn":505,"description":506,"org":627,"tags":628,"stars":27,"repoUrl":28,"updatedAt":514},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[629,630,631],{"name":9,"slug":8,"type":16},{"name":511,"slug":512,"type":16},{"name":18,"slug":19,"type":16},{"slug":516,"name":516,"fn":517,"description":518,"org":633,"tags":634,"stars":27,"repoUrl":28,"updatedAt":528},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[635,636,637,638,639],{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":524,"slug":525,"type":16},{"name":14,"slug":15,"type":16},{"name":486,"slug":487,"type":16},24]