[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-braintrust-braintrust-build-eval-dataset":3,"mdc--vypmqj-key":38,"related-org-braintrust-braintrust-build-eval-dataset":470,"related-repo-braintrust-braintrust-build-eval-dataset":637},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":27,"repoUrl":28,"updatedAt":29,"license":30,"forks":31,"topics":32,"repo":33,"sourceUrl":36,"mdContent":37},"braintrust-build-eval-dataset","create and manage LLM eval datasets","Create, edit, audit, or compare eval datasets for LLM applications and agents, including target-population definition, case sourcing from production traces, stratified sampling, label provenance and label audits, expected values as constraints for open-ended tasks, dev\u002Ftest splits, contamination and leakage controls, headroom checks, refresh policy, and datasheets. Use when working on the content or lifecycle of an eval dataset. Do not use for sample-size or power calculations, scorer implementation, or open-ended adversarial failure discovery.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"braintrust","Braintrust","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fbraintrust.png","braintrustdata",[13,17,20,23,26],{"name":14,"slug":15,"type":16},"Datasets","datasets","tag",{"name":18,"slug":19,"type":16},"LLM","llm",{"name":21,"slug":22,"type":16},"Evals","evals",{"name":24,"slug":25,"type":16},"Agents","agents",{"name":9,"slug":8,"type":16},7,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library","2026-08-20T03:53:33.304815",null,0,[],{"repoUrl":28,"stars":27,"forks":31,"topics":34,"description":35},[],"Braintrust eval skills library","https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library\u002Ftree\u002FHEAD\u002Fskills\u002Fbraintrust-build-eval-dataset","---\nname: braintrust-build-eval-dataset\ndescription: >-\n  Create, edit, audit, or compare eval datasets for LLM applications and agents, including\n  target-population definition, case sourcing from production traces, stratified sampling,\n  label provenance and label audits, expected values as constraints for open-ended tasks,\n  dev\u002Ftest splits, contamination and leakage controls, headroom checks, refresh policy, and\n  datasheets. Use when working on the content or lifecycle of an eval dataset. Do not use for\n  sample-size or power calculations, scorer implementation, or open-ended adversarial failure\n  discovery.\n---\n\n# Build or audit an eval dataset\n\nContract: `references\u002Finteraction-contract.md`. Calibration, templates, provenance: `references\u002Fdataset-construction.md`.\n\n## Trigger\n\n- Building eval cases, sampling production traces, or auditing an inherited set.\n- Questions about representativeness, labels, splits, contamination, or refresh.\n- Open-ended tasks where one gold string is being forced onto many valid answers.\n- A set that no longer discriminates: everything passes, or everything fails.\n\n## Do\n\n1. Inspect existing cases first and write the population sentence they appear to represent. Ask\n   for the target population only if it cannot be inferred.\n2. Source in the priority order in `reference.md`: production traces → expert-authored →\n   synthetic to fill documented gaps → public benchmarks for cold start, harness calibration,\n   anchoring, or gap mapping only.\n3. Stratify rather than vibe: list strata from the instrumentation's metadata fields, measure\n   production frequency per stratum, sample proportionally or oversample rare-but-critical\n   strata and reweight when aggregating, then deduplicate.\n4. Audit labels: provenance per item, double-label a calibration sample, report agreement,\n   adjudicate. Require claimed effects to exceed the estimated label error rate. Assume error is\n   **systematic**, not just random.\n5. For open-ended tasks encode `expected` as **constraints or a rubric, not a string**, and ship\n   the constraint set with the item.\n6. Set the lifecycle: headroom at both ends, dev\u002Ftest separation with **counted test touches**,\n   the leakage controls in `reference.md`, refresh against current production mix, and the\n   datasheet.\n7. State the **iteration budget before iterating** (`reference.md`) — how many experiments before\n   stopping to reflect, one coherent change each, the same dev slice throughout, and the winner\n   running once against test. A budget set afterwards is a description of what you did, not a\n   control on it.\n\n## Avoid\n\n- Do not present a public-benchmark score as product-validity proof.\n- Do not force one gold string onto an open-ended task.\n- Do not do the sample-size arithmetic here.\n- Do not treat the transform pipeline as neutral plumbing — assert an invariant after every\n  transform and version the pipeline like a scorer.\n- Do not store text derived by a component you are still changing. Store the reference to the\n  source and re-derive, or the dataset freezes a snapshot of the thing under iteration.\n- Do not down-weight a corpus whose labels prove unusable; drop it, and report the drop.\n\n## Check\n\n- Construct, population, non-use documented; coverage matrix including negative cases (refuse,\n  ask, do nothing).\n- Label provenance per item, agreement on the calibration sample, error rate estimated.\n- Difficulty spread checked; headroom confirmed at both ends.\n- Splits defined, touches counted, leakage controls in place, refresh cadence set.\n- Open-ended items carry constraints or rubrics; a production-failure ingestion path is owned.\n\n## Risk\n\n- Convenient or noisy cases support narrow benchmark claims while misrepresenting deployment.\n- Label noise floors detectable effects and can reorder rankings; systematic error penalizes\n  exactly the systems faithful to the real input.\n- Leakage converts a capability measure into a memory measure, and no corpus can be proven clean.\n- Structurally scraped items probe retrieval and precision, not judgment.\n\n## Braintrust\n\nPush the dataset and **version it**. This stage produces the artifact that\n`references\u002Fplatform-mechanics.md` §3 depends on — every paired comparison downstream is only\npossible because a version exists to pin to, and a dataset change invalidates cross-experiment\ncomparisons exactly like a scorer change.\n\nPopulate `metadata` with every stratification field — `split`, `source`, `category`,\n`difficulty`, `adversarial`, `trap`, `ground_truth_by_construction`, `label_provenance` — since\n§4 is unforgiving here: a stratum missing at build time cannot be recovered at report time.\n\nBuild the production-to-dataset pipeline with **human review queues**: sample live traces,\nreview, append. Keep benchmark scaffolding in **separate, clearly named datasets** — once mixed,\nthe distinction between measuring your construct and the benchmark's is unrecoverable. Counting\ntest touches is a discipline the platform will not enforce; track it in the dataset description.\n\nFor items sourced from logs, store the **reference** — `trace_ref`, `group_ref`, or\n`_full_row_id_ref` — not materialized text derived from the trace. Deduplicate on the stable\nidentity of the referenced unit. Row shapes are in\n`braintrust-discover-trace-topics\u002Freferences\u002Ffacet-pipeline.md`.\n",{"data":39,"body":40},{"name":4,"description":6},{"type":41,"children":42},"root",[43,52,75,82,107,113,211,217,250,256,284,290,313,317,337,409,428],{"type":44,"tag":45,"props":46,"children":48},"element","h1",{"id":47},"build-or-audit-an-eval-dataset",[49],{"type":50,"value":51},"text","Build or audit an eval dataset",{"type":44,"tag":53,"props":54,"children":55},"p",{},[56,58,65,67,73],{"type":50,"value":57},"Contract: ",{"type":44,"tag":59,"props":60,"children":62},"code",{"className":61},[],[63],{"type":50,"value":64},"references\u002Finteraction-contract.md",{"type":50,"value":66},". Calibration, templates, provenance: ",{"type":44,"tag":59,"props":68,"children":70},{"className":69},[],[71],{"type":50,"value":72},"references\u002Fdataset-construction.md",{"type":50,"value":74},".",{"type":44,"tag":76,"props":77,"children":79},"h2",{"id":78},"trigger",[80],{"type":50,"value":81},"Trigger",{"type":44,"tag":83,"props":84,"children":85},"ul",{},[86,92,97,102],{"type":44,"tag":87,"props":88,"children":89},"li",{},[90],{"type":50,"value":91},"Building eval cases, sampling production traces, or auditing an inherited set.",{"type":44,"tag":87,"props":93,"children":94},{},[95],{"type":50,"value":96},"Questions about representativeness, labels, splits, contamination, or refresh.",{"type":44,"tag":87,"props":98,"children":99},{},[100],{"type":50,"value":101},"Open-ended tasks where one gold string is being forced onto many valid answers.",{"type":44,"tag":87,"props":103,"children":104},{},[105],{"type":50,"value":106},"A set that no longer discriminates: everything passes, or everything fails.",{"type":44,"tag":76,"props":108,"children":110},{"id":109},"do",[111],{"type":50,"value":112},"Do",{"type":44,"tag":114,"props":115,"children":116},"ol",{},[117,122,135,140,153,173,192],{"type":44,"tag":87,"props":118,"children":119},{},[120],{"type":50,"value":121},"Inspect existing cases first and write the population sentence they appear to represent. Ask\nfor the target population only if it cannot be inferred.",{"type":44,"tag":87,"props":123,"children":124},{},[125,127,133],{"type":50,"value":126},"Source in the priority order in ",{"type":44,"tag":59,"props":128,"children":130},{"className":129},[],[131],{"type":50,"value":132},"reference.md",{"type":50,"value":134},": production traces → expert-authored →\nsynthetic to fill documented gaps → public benchmarks for cold start, harness calibration,\nanchoring, or gap mapping only.",{"type":44,"tag":87,"props":136,"children":137},{},[138],{"type":50,"value":139},"Stratify rather than vibe: list strata from the instrumentation's metadata fields, measure\nproduction frequency per stratum, sample proportionally or oversample rare-but-critical\nstrata and reweight when aggregating, then deduplicate.",{"type":44,"tag":87,"props":141,"children":142},{},[143,145,151],{"type":50,"value":144},"Audit labels: provenance per item, double-label a calibration sample, report agreement,\nadjudicate. Require claimed effects to exceed the estimated label error rate. Assume error is\n",{"type":44,"tag":146,"props":147,"children":148},"strong",{},[149],{"type":50,"value":150},"systematic",{"type":50,"value":152},", not just random.",{"type":44,"tag":87,"props":154,"children":155},{},[156,158,164,166,171],{"type":50,"value":157},"For open-ended tasks encode ",{"type":44,"tag":59,"props":159,"children":161},{"className":160},[],[162],{"type":50,"value":163},"expected",{"type":50,"value":165}," as ",{"type":44,"tag":146,"props":167,"children":168},{},[169],{"type":50,"value":170},"constraints or a rubric, not a string",{"type":50,"value":172},", and ship\nthe constraint set with the item.",{"type":44,"tag":87,"props":174,"children":175},{},[176,178,183,185,190],{"type":50,"value":177},"Set the lifecycle: headroom at both ends, dev\u002Ftest separation with ",{"type":44,"tag":146,"props":179,"children":180},{},[181],{"type":50,"value":182},"counted test touches",{"type":50,"value":184},",\nthe leakage controls in ",{"type":44,"tag":59,"props":186,"children":188},{"className":187},[],[189],{"type":50,"value":132},{"type":50,"value":191},", refresh against current production mix, and the\ndatasheet.",{"type":44,"tag":87,"props":193,"children":194},{},[195,197,202,204,209],{"type":50,"value":196},"State the ",{"type":44,"tag":146,"props":198,"children":199},{},[200],{"type":50,"value":201},"iteration budget before iterating",{"type":50,"value":203}," (",{"type":44,"tag":59,"props":205,"children":207},{"className":206},[],[208],{"type":50,"value":132},{"type":50,"value":210},") — how many experiments before\nstopping to reflect, one coherent change each, the same dev slice throughout, and the winner\nrunning once against test. A budget set afterwards is a description of what you did, not a\ncontrol on it.",{"type":44,"tag":76,"props":212,"children":214},{"id":213},"avoid",[215],{"type":50,"value":216},"Avoid",{"type":44,"tag":83,"props":218,"children":219},{},[220,225,230,235,240,245],{"type":44,"tag":87,"props":221,"children":222},{},[223],{"type":50,"value":224},"Do not present a public-benchmark score as product-validity proof.",{"type":44,"tag":87,"props":226,"children":227},{},[228],{"type":50,"value":229},"Do not force one gold string onto an open-ended task.",{"type":44,"tag":87,"props":231,"children":232},{},[233],{"type":50,"value":234},"Do not do the sample-size arithmetic here.",{"type":44,"tag":87,"props":236,"children":237},{},[238],{"type":50,"value":239},"Do not treat the transform pipeline as neutral plumbing — assert an invariant after every\ntransform and version the pipeline like a scorer.",{"type":44,"tag":87,"props":241,"children":242},{},[243],{"type":50,"value":244},"Do not store text derived by a component you are still changing. Store the reference to the\nsource and re-derive, or the dataset freezes a snapshot of the thing under iteration.",{"type":44,"tag":87,"props":246,"children":247},{},[248],{"type":50,"value":249},"Do not down-weight a corpus whose labels prove unusable; drop it, and report the drop.",{"type":44,"tag":76,"props":251,"children":253},{"id":252},"check",[254],{"type":50,"value":255},"Check",{"type":44,"tag":83,"props":257,"children":258},{},[259,264,269,274,279],{"type":44,"tag":87,"props":260,"children":261},{},[262],{"type":50,"value":263},"Construct, population, non-use documented; coverage matrix including negative cases (refuse,\nask, do nothing).",{"type":44,"tag":87,"props":265,"children":266},{},[267],{"type":50,"value":268},"Label provenance per item, agreement on the calibration sample, error rate estimated.",{"type":44,"tag":87,"props":270,"children":271},{},[272],{"type":50,"value":273},"Difficulty spread checked; headroom confirmed at both ends.",{"type":44,"tag":87,"props":275,"children":276},{},[277],{"type":50,"value":278},"Splits defined, touches counted, leakage controls in place, refresh cadence set.",{"type":44,"tag":87,"props":280,"children":281},{},[282],{"type":50,"value":283},"Open-ended items carry constraints or rubrics; a production-failure ingestion path is owned.",{"type":44,"tag":76,"props":285,"children":287},{"id":286},"risk",[288],{"type":50,"value":289},"Risk",{"type":44,"tag":83,"props":291,"children":292},{},[293,298,303,308],{"type":44,"tag":87,"props":294,"children":295},{},[296],{"type":50,"value":297},"Convenient or noisy cases support narrow benchmark claims while misrepresenting deployment.",{"type":44,"tag":87,"props":299,"children":300},{},[301],{"type":50,"value":302},"Label noise floors detectable effects and can reorder rankings; systematic error penalizes\nexactly the systems faithful to the real input.",{"type":44,"tag":87,"props":304,"children":305},{},[306],{"type":50,"value":307},"Leakage converts a capability measure into a memory measure, and no corpus can be proven clean.",{"type":44,"tag":87,"props":309,"children":310},{},[311],{"type":50,"value":312},"Structurally scraped items probe retrieval and precision, not judgment.",{"type":44,"tag":76,"props":314,"children":315},{"id":8},[316],{"type":50,"value":9},{"type":44,"tag":53,"props":318,"children":319},{},[320,322,327,329,335],{"type":50,"value":321},"Push the dataset and ",{"type":44,"tag":146,"props":323,"children":324},{},[325],{"type":50,"value":326},"version it",{"type":50,"value":328},". This stage produces the artifact that\n",{"type":44,"tag":59,"props":330,"children":332},{"className":331},[],[333],{"type":50,"value":334},"references\u002Fplatform-mechanics.md",{"type":50,"value":336}," §3 depends on — every paired comparison downstream is only\npossible because a version exists to pin to, and a dataset change invalidates cross-experiment\ncomparisons exactly like a scorer change.",{"type":44,"tag":53,"props":338,"children":339},{},[340,342,348,350,356,358,364,365,371,373,379,380,386,387,393,394,400,401,407],{"type":50,"value":341},"Populate ",{"type":44,"tag":59,"props":343,"children":345},{"className":344},[],[346],{"type":50,"value":347},"metadata",{"type":50,"value":349}," with every stratification field — ",{"type":44,"tag":59,"props":351,"children":353},{"className":352},[],[354],{"type":50,"value":355},"split",{"type":50,"value":357},", ",{"type":44,"tag":59,"props":359,"children":361},{"className":360},[],[362],{"type":50,"value":363},"source",{"type":50,"value":357},{"type":44,"tag":59,"props":366,"children":368},{"className":367},[],[369],{"type":50,"value":370},"category",{"type":50,"value":372},",\n",{"type":44,"tag":59,"props":374,"children":376},{"className":375},[],[377],{"type":50,"value":378},"difficulty",{"type":50,"value":357},{"type":44,"tag":59,"props":381,"children":383},{"className":382},[],[384],{"type":50,"value":385},"adversarial",{"type":50,"value":357},{"type":44,"tag":59,"props":388,"children":390},{"className":389},[],[391],{"type":50,"value":392},"trap",{"type":50,"value":357},{"type":44,"tag":59,"props":395,"children":397},{"className":396},[],[398],{"type":50,"value":399},"ground_truth_by_construction",{"type":50,"value":357},{"type":44,"tag":59,"props":402,"children":404},{"className":403},[],[405],{"type":50,"value":406},"label_provenance",{"type":50,"value":408}," — since\n§4 is unforgiving here: a stratum missing at build time cannot be recovered at report time.",{"type":44,"tag":53,"props":410,"children":411},{},[412,414,419,421,426],{"type":50,"value":413},"Build the production-to-dataset pipeline with ",{"type":44,"tag":146,"props":415,"children":416},{},[417],{"type":50,"value":418},"human review queues",{"type":50,"value":420},": sample live traces,\nreview, append. Keep benchmark scaffolding in ",{"type":44,"tag":146,"props":422,"children":423},{},[424],{"type":50,"value":425},"separate, clearly named datasets",{"type":50,"value":427}," — once mixed,\nthe distinction between measuring your construct and the benchmark's is unrecoverable. Counting\ntest touches is a discipline the platform will not enforce; track it in the dataset description.",{"type":44,"tag":53,"props":429,"children":430},{},[431,433,438,440,446,447,453,455,461,463,469],{"type":50,"value":432},"For items sourced from logs, store the ",{"type":44,"tag":146,"props":434,"children":435},{},[436],{"type":50,"value":437},"reference",{"type":50,"value":439}," — ",{"type":44,"tag":59,"props":441,"children":443},{"className":442},[],[444],{"type":50,"value":445},"trace_ref",{"type":50,"value":357},{"type":44,"tag":59,"props":448,"children":450},{"className":449},[],[451],{"type":50,"value":452},"group_ref",{"type":50,"value":454},", or\n",{"type":44,"tag":59,"props":456,"children":458},{"className":457},[],[459],{"type":50,"value":460},"_full_row_id_ref",{"type":50,"value":462}," — not materialized text derived from the trace. Deduplicate on the stable\nidentity of the referenced unit. Row shapes are in\n",{"type":44,"tag":59,"props":464,"children":466},{"className":465},[],[467],{"type":50,"value":468},"braintrust-discover-trace-topics\u002Freferences\u002Ffacet-pipeline.md",{"type":50,"value":74},{"items":471,"total":636},[472,488,504,518,526,542,556,568,582,598,610,622],{"slug":473,"name":473,"fn":474,"description":475,"org":476,"tags":477,"stars":485,"repoUrl":486,"updatedAt":487},"troubleshoot-braintrust-mcp","configure and troubleshoot Braintrust MCP servers","This plugin auto-configures a \"braintrust\" MCP server. If you can't see it or reach it, activate this skill\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[478,479,482],{"name":9,"slug":8,"type":16},{"name":480,"slug":481,"type":16},"Debugging","debugging",{"name":483,"slug":484,"type":16},"MCP","mcp",18,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Fbraintrust-claude-plugin","2026-07-12T08:36:13.889274",{"slug":489,"name":489,"fn":490,"description":491,"org":492,"tags":493,"stars":27,"repoUrl":28,"updatedAt":503},"braintrust-analyze-eval-experiment","analyze LLM and agent eval experiments","Analyze completed LLM or agent eval experiments using uncertainty-aware and decision-relevant methods. Use to audit run completeness and pairing, calculate confidence intervals, run paired comparisons, report wins, losses, and ties, incorporate run-to-run variance, handle multiple comparisons, inspect subgroup performance, and test fragility to favorable slices. Use when results already exist and someone asks what they mean, whether a difference is real, or which model won. Do not use to design an experiment that has not yet collected results.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[494,497,498,499,500],{"name":495,"slug":496,"type":16},"Analysis","analysis",{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":501,"slug":502,"type":16},"Statistics","statistics","2026-08-20T03:53:01.13806",{"slug":505,"name":505,"fn":506,"description":507,"org":508,"tags":509,"stars":27,"repoUrl":28,"updatedAt":517},"braintrust-attribute-multi-variable-change","attribute performance changes to multiple variables","Attribute an observed change when several things moved at once — model plus prompt plus tools, a provider migration, a framework upgrade, or a vendor swap that bundles serving stack with model. Use when asked which part of a change caused the result, when a comparison's arms differ in more than one way, when a treatment has no uniform implementation across vendors, or when a serving-stack difference is confounded with a model difference. Do not use for a clean single-variable comparison, or to design an experiment that has not yet run.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[510,511,512,513,514],{"name":495,"slug":496,"type":16},{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":515,"slug":516,"type":16},"Performance","performance","2026-08-20T03:53:40.036077",{"slug":4,"name":4,"fn":5,"description":6,"org":519,"tags":520,"stars":27,"repoUrl":28,"updatedAt":29},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[521,522,523,524,525],{"name":24,"slug":25,"type":16},{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"slug":527,"name":527,"fn":528,"description":529,"org":530,"tags":531,"stars":27,"repoUrl":28,"updatedAt":541},"braintrust-define-eval-objective","define LLM evaluation objectives","Create, edit, or audit an eval objective by working backward from a product decision to the target outcome, construct, population, intended claim, and verification-versus-validation questions. Use when a team is unsure what an eval should establish, asks \"what are we actually trying to measure,\" \"is this eval measuring the right thing,\" \"does this benchmark support our claim,\" or needs to turn a product goal into an eval objective and state which claims are out of scope. Do not use to select detailed metrics, design datasets, or implement scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[532,533,534,535,538],{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":536,"slug":537,"type":16},"Product Management","product-management",{"name":539,"slug":540,"type":16},"Strategy","strategy","2026-08-20T03:53:00.07097",{"slug":543,"name":543,"fn":544,"description":545,"org":546,"tags":547,"stars":27,"repoUrl":28,"updatedAt":555},"braintrust-define-eval-release-gate","configure release gates for LLM applications","Create, edit, audit, or apply release gates for LLM applications and agents. Use to combine minimum meaningful improvement, statistical significance, regression rate, subgroup consistency, worst-run stability, all-attempts reliability, safety upper bounds, latency, and cost into an explicit ship-or-hold policy, to turn metrics into a CI gate, or to explain why a candidate failed a gate and what evidence would justify reconsideration. Do not use for general result analysis without a deployment decision.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[548,549,550,553,554],{"name":24,"slug":25,"type":16},{"name":9,"slug":8,"type":16},{"name":551,"slug":552,"type":16},"CI\u002FCD","ci-cd",{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},"2026-08-20T03:53:15.686158",{"slug":557,"name":557,"fn":558,"description":559,"org":560,"tags":561,"stars":27,"repoUrl":28,"updatedAt":567},"braintrust-deploy-evaluator","deploy evaluators to Braintrust","Take a validated scorer or classifier from definition to running instrument in Braintrust — scope selection, inline testing before saving, saving as an evaluator, attaching an online-scoring rule, activating it for new traffic, and backfilling history with a rewind. Use when a scorer needs to actually run against production logs, when an online-scoring rule needs to be created or changed, or when historical traces need scoring. Do not use to decide what the scorer should measure, to write its rubric, or to establish that it agrees with human judgment.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[562,563,566],{"name":9,"slug":8,"type":16},{"name":564,"slug":565,"type":16},"Deployment","deployment",{"name":21,"slug":22,"type":16},"2026-08-20T03:53:32.558937",{"slug":569,"name":569,"fn":570,"description":571,"org":572,"tags":573,"stars":27,"repoUrl":28,"updatedAt":581},"braintrust-design-eval-experiment","design controlled LLM eval experiments","Design or audit controlled eval experiments for model, prompt, retrieval, tool, guardrail, or agent-architecture changes. Use before data collection to state directional and minimum-effect hypotheses, name independent, dependent, and control variables including the serving environment and tool surface, choose paired designs, set repetitions and allocation, distinguish exploratory from confirmatory comparisons, and pre-specify stopping, exclusion, multiplicity, and analysis rules. Do not use primarily to analyze results already collected.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[574,575,576,579,580],{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":577,"slug":578,"type":16},"Experiments","experiments",{"name":18,"slug":19,"type":16},{"name":539,"slug":540,"type":16},"2026-08-20T03:53:36.534554",{"slug":583,"name":583,"fn":584,"description":585,"org":586,"tags":587,"stars":27,"repoUrl":28,"updatedAt":597},"braintrust-design-eval-instrumentation","design trace and evaluation dataset schemas","Design the trace and eval-dataset schema for an LLM app or agent, and wire the system to emit it. Use when deciding what to log, designing a trace schema, setting up tracing or observability before evals, or when failures cannot be debugged or sliced from existing traces — covering inputs, outputs, spans for tool and LLM calls, state changes, metadata, resolved configuration, serving path, tool manifest, per-item status, attachments, and subgroup variables. Do not use to decide what the evidence should mean, or to write scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[588,589,590,591,594],{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":21,"slug":22,"type":16},{"name":592,"slug":593,"type":16},"Observability","observability",{"name":595,"slug":596,"type":16},"Tracing","tracing","2026-08-20T03:53:37.274703",{"slug":599,"name":599,"fn":600,"description":601,"org":602,"tags":603,"stars":27,"repoUrl":28,"updatedAt":609},"braintrust-design-eval-metric-bundle","create multi-objective evaluation metric bundles","Create, edit, audit, or compare a multi-objective eval metric bundle covering product quality, safety, reliability, latency, and cost. Use to choose metrics for an eval, define a goodness bundle, distinguish optimization metrics from non-regression guardrails, expose tradeoffs, audit a KPI or single composite score for Goodhart and metric-gaming risk, or answer \"what should improve and what must not regress.\" Do not use to design trace schemas, build datasets, or implement scoring methods.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[604,605,606,607,608],{"name":24,"slug":25,"type":16},{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":515,"slug":516,"type":16},"2026-08-20T03:53:16.078609",{"slug":611,"name":611,"fn":612,"description":613,"org":614,"tags":615,"stars":27,"repoUrl":28,"updatedAt":621},"braintrust-design-human-eval-review","design human evaluation and review workflows","Design or audit human evaluation workflows and golden datasets for LLM applications and agents. Use to set up expert review, select review cases, write reviewer instructions, assign raters, capture rationales and confidence, measure inter-rater agreement with kappa or alpha, adjudicate disagreements, and preserve reviewed examples with provenance as a versioned reference set. Do not use to elicit the criteria or rubric in the first place, to validate a scorer once reference labels exist, or to implement the scorer.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[616,617,618,619,620],{"name":24,"slug":25,"type":16},{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},"2026-08-20T03:53:36.900129",{"slug":623,"name":623,"fn":624,"description":625,"org":626,"tags":627,"stars":27,"repoUrl":28,"updatedAt":635},"braintrust-discover-agent-failures","identify and classify agent failure modes","Search open-endedly for unanticipated agent failure modes and convert them into a named taxonomy and durable regression items. Use for requests to find out what goes wrong, surface unknown or silent failures, do error analysis over traces, cluster and triage production failures, or build a failure taxonomy — where the goal is discovering modes nobody thought to test rather than measuring a predefined criterion. Produces datasets and taxonomies, not headline scores. Do not use for adversarial attacks against a threat model, or to measure a known criterion.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[628,629,630,631,632],{"name":24,"slug":25,"type":16},{"name":9,"slug":8,"type":16},{"name":480,"slug":481,"type":16},{"name":21,"slug":22,"type":16},{"name":633,"slug":634,"type":16},"Triage","triage","2026-08-20T03:53:00.423941",27,{"items":638,"total":693},[639,647,655,663,671,679,685],{"slug":489,"name":489,"fn":490,"description":491,"org":640,"tags":641,"stars":27,"repoUrl":28,"updatedAt":503},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[642,643,644,645,646],{"name":495,"slug":496,"type":16},{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":501,"slug":502,"type":16},{"slug":505,"name":505,"fn":506,"description":507,"org":648,"tags":649,"stars":27,"repoUrl":28,"updatedAt":517},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[650,651,652,653,654],{"name":495,"slug":496,"type":16},{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":515,"slug":516,"type":16},{"slug":4,"name":4,"fn":5,"description":6,"org":656,"tags":657,"stars":27,"repoUrl":28,"updatedAt":29},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[658,659,660,661,662],{"name":24,"slug":25,"type":16},{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"slug":527,"name":527,"fn":528,"description":529,"org":664,"tags":665,"stars":27,"repoUrl":28,"updatedAt":541},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[666,667,668,669,670],{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":536,"slug":537,"type":16},{"name":539,"slug":540,"type":16},{"slug":543,"name":543,"fn":544,"description":545,"org":672,"tags":673,"stars":27,"repoUrl":28,"updatedAt":555},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[674,675,676,677,678],{"name":24,"slug":25,"type":16},{"name":9,"slug":8,"type":16},{"name":551,"slug":552,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"slug":557,"name":557,"fn":558,"description":559,"org":680,"tags":681,"stars":27,"repoUrl":28,"updatedAt":567},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[682,683,684],{"name":9,"slug":8,"type":16},{"name":564,"slug":565,"type":16},{"name":21,"slug":22,"type":16},{"slug":569,"name":569,"fn":570,"description":571,"org":686,"tags":687,"stars":27,"repoUrl":28,"updatedAt":581},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[688,689,690,691,692],{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":577,"slug":578,"type":16},{"name":18,"slug":19,"type":16},{"name":539,"slug":540,"type":16},24]