[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-braintrust-braintrust-deploy-evaluator":3,"mdc-mie16-key":32,"related-org-braintrust-braintrust-deploy-evaluator":618,"related-repo-braintrust-braintrust-deploy-evaluator":789},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":21,"repoUrl":22,"updatedAt":23,"license":24,"forks":25,"topics":26,"repo":27,"sourceUrl":30,"mdContent":31},"braintrust-deploy-evaluator","deploy evaluators to Braintrust","Take a validated scorer or classifier from definition to running instrument in Braintrust — scope selection, inline testing before saving, saving as an evaluator, attaching an online-scoring rule, activating it for new traffic, and backfilling history with a rewind. Use when a scorer needs to actually run against production logs, when an online-scoring rule needs to be created or changed, or when historical traces need scoring. Do not use to decide what the scorer should measure, to write its rubric, or to establish that it agrees with human judgment.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"braintrust","Braintrust","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fbraintrust.png","braintrustdata",[13,17,20],{"name":14,"slug":15,"type":16},"Evals","evals","tag",{"name":18,"slug":19,"type":16},"Deployment","deployment",{"name":9,"slug":8,"type":16},7,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library","2026-08-20T03:53:32.558937",null,0,[],{"repoUrl":22,"stars":21,"forks":25,"topics":28,"description":29},[],"Braintrust eval skills library","https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library\u002Ftree\u002FHEAD\u002Fskills\u002Fbraintrust-deploy-evaluator","---\nname: braintrust-deploy-evaluator\ndescription: >-\n  Take a validated scorer or classifier from definition to running instrument in Braintrust —\n  scope selection, inline testing before saving, saving as an evaluator, attaching an\n  online-scoring rule, activating it for new traffic, and backfilling history with a rewind. Use\n  when a scorer needs to actually run against production logs, when an online-scoring rule needs\n  to be created or changed, or when historical traces need scoring. Do not use to decide what the\n  scorer should measure, to write its rubric, or to establish that it agrees with human judgment.\n---\n\n# Deploy an evaluator to production traffic\n\nContract: `references\u002Finteraction-contract.md`. Calibration, templates, provenance: `references\u002Fevaluator-deployment.md`.\n\n## Trigger\n\n- \"Run this scorer against our logs.\" \u002F \"Score the last two weeks.\"\n- A validated scorer that still only exists inside an offline experiment.\n- An online-scoring rule to create, retarget, resample, or pause.\n- Backfilling a newly deployed evaluator over history.\n\n## Do\n\n1. Confirm the evaluator has a **fitness verdict** before deploying it. Trend monitoring and\n   release gating are different approvals; a scorer cleared for one is not cleared for the other.\n   No verdict yet → `braintrust-validate-eval-scorer` first.\n2. Fix **scope** — span for a single LLM call or tool call, trace for behavior spanning a request,\n   group for behavior spanning several traces joined by a key. Default to trace. Confirm the\n   grouping key exists in real metadata rather than assuming it.\n3. Fix the **output contract**: a scorer returns 0–1 and needs its numeric mapping; a classifier\n   returns one of a fixed label set and needs its no-match behavior. Declare skip behavior for\n   both.\n4. **Test inline before saving.** Iterate on the definition against representative examples with\n   no saved artifact, then save, then re-test the *saved* version on the same examples — saving is\n   a step that can change behavior, and the second test is what catches it.\n5. Treat **activation** and **rewind** as two separate authorizations. Neither implies the other.\n   A paused rule will not process a rewound range until it is activated; say so rather than\n   letting the user believe the backfill is running.\n6. Before rewinding, **estimate the eligible target count** over the exact rule configuration and\n   window, and account for the sampling rate in the number you report.\n7. After results land, read the distribution and inspect representative passes, failures, and\n   borderline cases before drawing any conclusion from the aggregate.\n\n## Avoid\n\n- Do not deploy an evaluator whose approved uses are undocumented — absence of a verdict is not\n  approval for trend monitoring.\n- Do not set 100% sampling without explicit approval; sampling rate is the primary cost control.\n- Do not resubmit a rewind to check on it. Resubmission is not a progress query.\n- Do not rename an evaluator across deployments (`references\u002Fplatform-mechanics.md` §5) — the\n  name is the join key for every cross-experiment and offline-to-online comparison.\n- Do not report a preliminary backfill aggregate without saying it is preliminary.\n\n## Check\n\n- Fitness verdict exists and names this deployment's use as allowed.\n- Scope justified; grouping key verified present in real data.\n- Output contract declared, including skip and no-match behavior.\n- Inline test passed, then the saved version re-tested on the same examples.\n- Sampling rate and filters approved; rule status intentional, not defaulted into.\n- Rewind window and estimated target count stated before the rewind, not after.\n\n## Risk\n\n- An evaluator deployed without a verdict produces an authoritative-looking series that no one can\n  say is valid, and it accumulates history that later looks like a baseline.\n- Activation and rewind get conflated constantly. The usual outcome is a user who believes history\n  is being scored while a paused rule does nothing.\n- Backfill cost scales with eligible units × sampling and is easy to underestimate by an order of\n  magnitude on a high-volume project.\n- Judge drift under a provider model update is indistinguishable from a product regression in the\n  resulting series unless the evaluator version is recorded alongside it.\n\n## Braintrust\n\n**Build and test.** `sql_query` first to find representative data and locate where the behavior\nactually appears — `input`, `output`, `metadata`, or elsewhere. Code evaluators (Python or\nTypeScript) for anything checkable without a model; LLM evaluators where judgment is required.\nFollow the project's existing language. Test with `test_evaluator` and\n`function: {inline_evaluator: \u003Cdefinition>}` — the same definition `create_evaluator` accepts, no\nsave required. Inline code must define a top-level function named `handler`.\n\n**Save.** `create_evaluator` with `output_type: \"score\"` and `choice_scores` for numeric output, or\n`output_type: \"classification\"` and `choices` for labels. Code evaluators declare `output_type` but\nuse neither. Then re-run `test_evaluator` with `function: {function_id: \"\u003Csaved id>\"}` on the same\nexamples to confirm the save round-tripped.\n\n**Rule.** `update_online_scoring_rule` with `operation: \"save\"`, the evaluator `function_ids`,\nscope, filters, sampling rate, and logging behavior. New rules default to `status: \"paused\"`; pass\n`status: \"active\"` only on explicit authorization. When updating an existing rule, **omit `status`**\nto preserve its lifecycle state unless a change was requested. Activate separately via\n`set_automation_status`.\n\n**Rewind.** Estimate first with `sql_query` — count spans for span scope, distinct traces for trace\nscope, distinct grouping values for group scope — then apply the sampling rate. Call\n`update_online_scoring_rule` with `operation: \"rewind\"`, the `automation_id`, and an inclusive\n`start_time`, **once**. For a natural-language window, resolve the timestamp yourself and state the\nplan (\"the last three full days, roughly 1,400 spans\") rather than making the user confirm arithmetic.\n\n**Read the results.** Distribution first, then representative cases at both ends and the boundary.\n`generate_monitor_chart` for a histogram of score distribution or a time series of trend; group by a\ncategory only where it improves interpretation. Keep previews unsaved unless asked to persist.\nPreprocessors and the `{{preprocessed}}` variable for trace- and group-scoped LLM evaluators are\ncovered in `braintrust-discover-trace-topics`.\n",{"data":33,"body":34},{"name":4,"description":6},{"type":35,"children":36},"root",[37,46,69,76,101,107,210,216,252,258,291,297,320,324,396,464,532,585],{"type":38,"tag":39,"props":40,"children":42},"element","h1",{"id":41},"deploy-an-evaluator-to-production-traffic",[43],{"type":44,"value":45},"text","Deploy an evaluator to production traffic",{"type":38,"tag":47,"props":48,"children":49},"p",{},[50,52,59,61,67],{"type":44,"value":51},"Contract: ",{"type":38,"tag":53,"props":54,"children":56},"code",{"className":55},[],[57],{"type":44,"value":58},"references\u002Finteraction-contract.md",{"type":44,"value":60},". Calibration, templates, provenance: ",{"type":38,"tag":53,"props":62,"children":64},{"className":63},[],[65],{"type":44,"value":66},"references\u002Fevaluator-deployment.md",{"type":44,"value":68},".",{"type":38,"tag":70,"props":71,"children":73},"h2",{"id":72},"trigger",[74],{"type":44,"value":75},"Trigger",{"type":38,"tag":77,"props":78,"children":79},"ul",{},[80,86,91,96],{"type":38,"tag":81,"props":82,"children":83},"li",{},[84],{"type":44,"value":85},"\"Run this scorer against our logs.\" \u002F \"Score the last two weeks.\"",{"type":38,"tag":81,"props":87,"children":88},{},[89],{"type":44,"value":90},"A validated scorer that still only exists inside an offline experiment.",{"type":38,"tag":81,"props":92,"children":93},{},[94],{"type":44,"value":95},"An online-scoring rule to create, retarget, resample, or pause.",{"type":38,"tag":81,"props":97,"children":98},{},[99],{"type":44,"value":100},"Backfilling a newly deployed evaluator over history.",{"type":38,"tag":70,"props":102,"children":104},{"id":103},"do",[105],{"type":44,"value":106},"Do",{"type":38,"tag":108,"props":109,"children":110},"ol",{},[111,132,144,156,174,193,205],{"type":38,"tag":81,"props":112,"children":113},{},[114,116,122,124,130],{"type":44,"value":115},"Confirm the evaluator has a ",{"type":38,"tag":117,"props":118,"children":119},"strong",{},[120],{"type":44,"value":121},"fitness verdict",{"type":44,"value":123}," before deploying it. Trend monitoring and\nrelease gating are different approvals; a scorer cleared for one is not cleared for the other.\nNo verdict yet → ",{"type":38,"tag":53,"props":125,"children":127},{"className":126},[],[128],{"type":44,"value":129},"braintrust-validate-eval-scorer",{"type":44,"value":131}," first.",{"type":38,"tag":81,"props":133,"children":134},{},[135,137,142],{"type":44,"value":136},"Fix ",{"type":38,"tag":117,"props":138,"children":139},{},[140],{"type":44,"value":141},"scope",{"type":44,"value":143}," — span for a single LLM call or tool call, trace for behavior spanning a request,\ngroup for behavior spanning several traces joined by a key. Default to trace. Confirm the\ngrouping key exists in real metadata rather than assuming it.",{"type":38,"tag":81,"props":145,"children":146},{},[147,149,154],{"type":44,"value":148},"Fix the ",{"type":38,"tag":117,"props":150,"children":151},{},[152],{"type":44,"value":153},"output contract",{"type":44,"value":155},": a scorer returns 0–1 and needs its numeric mapping; a classifier\nreturns one of a fixed label set and needs its no-match behavior. Declare skip behavior for\nboth.",{"type":38,"tag":81,"props":157,"children":158},{},[159,164,166,172],{"type":38,"tag":117,"props":160,"children":161},{},[162],{"type":44,"value":163},"Test inline before saving.",{"type":44,"value":165}," Iterate on the definition against representative examples with\nno saved artifact, then save, then re-test the ",{"type":38,"tag":167,"props":168,"children":169},"em",{},[170],{"type":44,"value":171},"saved",{"type":44,"value":173}," version on the same examples — saving is\na step that can change behavior, and the second test is what catches it.",{"type":38,"tag":81,"props":175,"children":176},{},[177,179,184,186,191],{"type":44,"value":178},"Treat ",{"type":38,"tag":117,"props":180,"children":181},{},[182],{"type":44,"value":183},"activation",{"type":44,"value":185}," and ",{"type":38,"tag":117,"props":187,"children":188},{},[189],{"type":44,"value":190},"rewind",{"type":44,"value":192}," as two separate authorizations. Neither implies the other.\nA paused rule will not process a rewound range until it is activated; say so rather than\nletting the user believe the backfill is running.",{"type":38,"tag":81,"props":194,"children":195},{},[196,198,203],{"type":44,"value":197},"Before rewinding, ",{"type":38,"tag":117,"props":199,"children":200},{},[201],{"type":44,"value":202},"estimate the eligible target count",{"type":44,"value":204}," over the exact rule configuration and\nwindow, and account for the sampling rate in the number you report.",{"type":38,"tag":81,"props":206,"children":207},{},[208],{"type":44,"value":209},"After results land, read the distribution and inspect representative passes, failures, and\nborderline cases before drawing any conclusion from the aggregate.",{"type":38,"tag":70,"props":211,"children":213},{"id":212},"avoid",[214],{"type":44,"value":215},"Avoid",{"type":38,"tag":77,"props":217,"children":218},{},[219,224,229,234,247],{"type":38,"tag":81,"props":220,"children":221},{},[222],{"type":44,"value":223},"Do not deploy an evaluator whose approved uses are undocumented — absence of a verdict is not\napproval for trend monitoring.",{"type":38,"tag":81,"props":225,"children":226},{},[227],{"type":44,"value":228},"Do not set 100% sampling without explicit approval; sampling rate is the primary cost control.",{"type":38,"tag":81,"props":230,"children":231},{},[232],{"type":44,"value":233},"Do not resubmit a rewind to check on it. Resubmission is not a progress query.",{"type":38,"tag":81,"props":235,"children":236},{},[237,239,245],{"type":44,"value":238},"Do not rename an evaluator across deployments (",{"type":38,"tag":53,"props":240,"children":242},{"className":241},[],[243],{"type":44,"value":244},"references\u002Fplatform-mechanics.md",{"type":44,"value":246}," §5) — the\nname is the join key for every cross-experiment and offline-to-online comparison.",{"type":38,"tag":81,"props":248,"children":249},{},[250],{"type":44,"value":251},"Do not report a preliminary backfill aggregate without saying it is preliminary.",{"type":38,"tag":70,"props":253,"children":255},{"id":254},"check",[256],{"type":44,"value":257},"Check",{"type":38,"tag":77,"props":259,"children":260},{},[261,266,271,276,281,286],{"type":38,"tag":81,"props":262,"children":263},{},[264],{"type":44,"value":265},"Fitness verdict exists and names this deployment's use as allowed.",{"type":38,"tag":81,"props":267,"children":268},{},[269],{"type":44,"value":270},"Scope justified; grouping key verified present in real data.",{"type":38,"tag":81,"props":272,"children":273},{},[274],{"type":44,"value":275},"Output contract declared, including skip and no-match behavior.",{"type":38,"tag":81,"props":277,"children":278},{},[279],{"type":44,"value":280},"Inline test passed, then the saved version re-tested on the same examples.",{"type":38,"tag":81,"props":282,"children":283},{},[284],{"type":44,"value":285},"Sampling rate and filters approved; rule status intentional, not defaulted into.",{"type":38,"tag":81,"props":287,"children":288},{},[289],{"type":44,"value":290},"Rewind window and estimated target count stated before the rewind, not after.",{"type":38,"tag":70,"props":292,"children":294},{"id":293},"risk",[295],{"type":44,"value":296},"Risk",{"type":38,"tag":77,"props":298,"children":299},{},[300,305,310,315],{"type":38,"tag":81,"props":301,"children":302},{},[303],{"type":44,"value":304},"An evaluator deployed without a verdict produces an authoritative-looking series that no one can\nsay is valid, and it accumulates history that later looks like a baseline.",{"type":38,"tag":81,"props":306,"children":307},{},[308],{"type":44,"value":309},"Activation and rewind get conflated constantly. The usual outcome is a user who believes history\nis being scored while a paused rule does nothing.",{"type":38,"tag":81,"props":311,"children":312},{},[313],{"type":44,"value":314},"Backfill cost scales with eligible units × sampling and is easy to underestimate by an order of\nmagnitude on a high-volume project.",{"type":38,"tag":81,"props":316,"children":317},{},[318],{"type":44,"value":319},"Judge drift under a provider model update is indistinguishable from a product regression in the\nresulting series unless the evaluator version is recorded alongside it.",{"type":38,"tag":70,"props":321,"children":322},{"id":8},[323],{"type":44,"value":9},{"type":38,"tag":47,"props":325,"children":326},{},[327,332,334,340,342,348,350,356,357,363,365,371,373,379,381,387,389,395],{"type":38,"tag":117,"props":328,"children":329},{},[330],{"type":44,"value":331},"Build and test.",{"type":44,"value":333}," ",{"type":38,"tag":53,"props":335,"children":337},{"className":336},[],[338],{"type":44,"value":339},"sql_query",{"type":44,"value":341}," first to find representative data and locate where the behavior\nactually appears — ",{"type":38,"tag":53,"props":343,"children":345},{"className":344},[],[346],{"type":44,"value":347},"input",{"type":44,"value":349},", ",{"type":38,"tag":53,"props":351,"children":353},{"className":352},[],[354],{"type":44,"value":355},"output",{"type":44,"value":349},{"type":38,"tag":53,"props":358,"children":360},{"className":359},[],[361],{"type":44,"value":362},"metadata",{"type":44,"value":364},", or elsewhere. Code evaluators (Python or\nTypeScript) for anything checkable without a model; LLM evaluators where judgment is required.\nFollow the project's existing language. Test with ",{"type":38,"tag":53,"props":366,"children":368},{"className":367},[],[369],{"type":44,"value":370},"test_evaluator",{"type":44,"value":372}," and\n",{"type":38,"tag":53,"props":374,"children":376},{"className":375},[],[377],{"type":44,"value":378},"function: {inline_evaluator: \u003Cdefinition>}",{"type":44,"value":380}," — the same definition ",{"type":38,"tag":53,"props":382,"children":384},{"className":383},[],[385],{"type":44,"value":386},"create_evaluator",{"type":44,"value":388}," accepts, no\nsave required. Inline code must define a top-level function named ",{"type":38,"tag":53,"props":390,"children":392},{"className":391},[],[393],{"type":44,"value":394},"handler",{"type":44,"value":68},{"type":38,"tag":47,"props":397,"children":398},{},[399,404,405,410,412,418,419,425,427,433,434,440,442,448,450,455,456,462],{"type":38,"tag":117,"props":400,"children":401},{},[402],{"type":44,"value":403},"Save.",{"type":44,"value":333},{"type":38,"tag":53,"props":406,"children":408},{"className":407},[],[409],{"type":44,"value":386},{"type":44,"value":411}," with ",{"type":38,"tag":53,"props":413,"children":415},{"className":414},[],[416],{"type":44,"value":417},"output_type: \"score\"",{"type":44,"value":185},{"type":38,"tag":53,"props":420,"children":422},{"className":421},[],[423],{"type":44,"value":424},"choice_scores",{"type":44,"value":426}," for numeric output, or\n",{"type":38,"tag":53,"props":428,"children":430},{"className":429},[],[431],{"type":44,"value":432},"output_type: \"classification\"",{"type":44,"value":185},{"type":38,"tag":53,"props":435,"children":437},{"className":436},[],[438],{"type":44,"value":439},"choices",{"type":44,"value":441}," for labels. Code evaluators declare ",{"type":38,"tag":53,"props":443,"children":445},{"className":444},[],[446],{"type":44,"value":447},"output_type",{"type":44,"value":449}," but\nuse neither. Then re-run ",{"type":38,"tag":53,"props":451,"children":453},{"className":452},[],[454],{"type":44,"value":370},{"type":44,"value":411},{"type":38,"tag":53,"props":457,"children":459},{"className":458},[],[460],{"type":44,"value":461},"function: {function_id: \"\u003Csaved id>\"}",{"type":44,"value":463}," on the same\nexamples to confirm the save round-tripped.",{"type":38,"tag":47,"props":465,"children":466},{},[467,472,473,479,480,486,488,494,496,502,504,510,512,523,525,531],{"type":38,"tag":117,"props":468,"children":469},{},[470],{"type":44,"value":471},"Rule.",{"type":44,"value":333},{"type":38,"tag":53,"props":474,"children":476},{"className":475},[],[477],{"type":44,"value":478},"update_online_scoring_rule",{"type":44,"value":411},{"type":38,"tag":53,"props":481,"children":483},{"className":482},[],[484],{"type":44,"value":485},"operation: \"save\"",{"type":44,"value":487},", the evaluator ",{"type":38,"tag":53,"props":489,"children":491},{"className":490},[],[492],{"type":44,"value":493},"function_ids",{"type":44,"value":495},",\nscope, filters, sampling rate, and logging behavior. New rules default to ",{"type":38,"tag":53,"props":497,"children":499},{"className":498},[],[500],{"type":44,"value":501},"status: \"paused\"",{"type":44,"value":503},"; pass\n",{"type":38,"tag":53,"props":505,"children":507},{"className":506},[],[508],{"type":44,"value":509},"status: \"active\"",{"type":44,"value":511}," only on explicit authorization. When updating an existing rule, ",{"type":38,"tag":117,"props":513,"children":514},{},[515,517],{"type":44,"value":516},"omit ",{"type":38,"tag":53,"props":518,"children":520},{"className":519},[],[521],{"type":44,"value":522},"status",{"type":44,"value":524},"\nto preserve its lifecycle state unless a change was requested. Activate separately via\n",{"type":38,"tag":53,"props":526,"children":528},{"className":527},[],[529],{"type":44,"value":530},"set_automation_status",{"type":44,"value":68},{"type":38,"tag":47,"props":533,"children":534},{},[535,540,542,547,549,554,555,561,563,569,571,577,578,583],{"type":38,"tag":117,"props":536,"children":537},{},[538],{"type":44,"value":539},"Rewind.",{"type":44,"value":541}," Estimate first with ",{"type":38,"tag":53,"props":543,"children":545},{"className":544},[],[546],{"type":44,"value":339},{"type":44,"value":548}," — count spans for span scope, distinct traces for trace\nscope, distinct grouping values for group scope — then apply the sampling rate. Call\n",{"type":38,"tag":53,"props":550,"children":552},{"className":551},[],[553],{"type":44,"value":478},{"type":44,"value":411},{"type":38,"tag":53,"props":556,"children":558},{"className":557},[],[559],{"type":44,"value":560},"operation: \"rewind\"",{"type":44,"value":562},", the ",{"type":38,"tag":53,"props":564,"children":566},{"className":565},[],[567],{"type":44,"value":568},"automation_id",{"type":44,"value":570},", and an inclusive\n",{"type":38,"tag":53,"props":572,"children":574},{"className":573},[],[575],{"type":44,"value":576},"start_time",{"type":44,"value":349},{"type":38,"tag":117,"props":579,"children":580},{},[581],{"type":44,"value":582},"once",{"type":44,"value":584},". For a natural-language window, resolve the timestamp yourself and state the\nplan (\"the last three full days, roughly 1,400 spans\") rather than making the user confirm arithmetic.",{"type":38,"tag":47,"props":586,"children":587},{},[588,593,595,601,603,609,611,617],{"type":38,"tag":117,"props":589,"children":590},{},[591],{"type":44,"value":592},"Read the results.",{"type":44,"value":594}," Distribution first, then representative cases at both ends and the boundary.\n",{"type":38,"tag":53,"props":596,"children":598},{"className":597},[],[599],{"type":44,"value":600},"generate_monitor_chart",{"type":44,"value":602}," for a histogram of score distribution or a time series of trend; group by a\ncategory only where it improves interpretation. Keep previews unsaved unless asked to persist.\nPreprocessors and the ",{"type":38,"tag":53,"props":604,"children":606},{"className":605},[],[607],{"type":44,"value":608},"{{preprocessed}}",{"type":44,"value":610}," variable for trace- and group-scoped LLM evaluators are\ncovered in ",{"type":38,"tag":53,"props":612,"children":614},{"className":613},[],[615],{"type":44,"value":616},"braintrust-discover-trace-topics",{"type":44,"value":68},{"items":619,"total":788},[620,636,654,668,684,700,714,720,734,750,762,774],{"slug":621,"name":621,"fn":622,"description":623,"org":624,"tags":625,"stars":633,"repoUrl":634,"updatedAt":635},"troubleshoot-braintrust-mcp","configure and troubleshoot Braintrust MCP servers","This plugin auto-configures a \"braintrust\" MCP server. If you can't see it or reach it, activate this skill\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[626,627,630],{"name":9,"slug":8,"type":16},{"name":628,"slug":629,"type":16},"Debugging","debugging",{"name":631,"slug":632,"type":16},"MCP","mcp",18,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Fbraintrust-claude-plugin","2026-07-12T08:36:13.889274",{"slug":637,"name":637,"fn":638,"description":639,"org":640,"tags":641,"stars":21,"repoUrl":22,"updatedAt":653},"braintrust-analyze-eval-experiment","analyze LLM and agent eval experiments","Analyze completed LLM or agent eval experiments using uncertainty-aware and decision-relevant methods. Use to audit run completeness and pairing, calculate confidence intervals, run paired comparisons, report wins, losses, and ties, incorporate run-to-run variance, handle multiple comparisons, inspect subgroup performance, and test fragility to favorable slices. Use when results already exist and someone asks what they mean, whether a difference is real, or which model won. Do not use to design an experiment that has not yet collected results.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[642,645,646,647,650],{"name":643,"slug":644,"type":16},"Analysis","analysis",{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":648,"slug":649,"type":16},"LLM","llm",{"name":651,"slug":652,"type":16},"Statistics","statistics","2026-08-20T03:53:01.13806",{"slug":655,"name":655,"fn":656,"description":657,"org":658,"tags":659,"stars":21,"repoUrl":22,"updatedAt":667},"braintrust-attribute-multi-variable-change","attribute performance changes to multiple variables","Attribute an observed change when several things moved at once — model plus prompt plus tools, a provider migration, a framework upgrade, or a vendor swap that bundles serving stack with model. Use when asked which part of a change caused the result, when a comparison's arms differ in more than one way, when a treatment has no uniform implementation across vendors, or when a serving-stack difference is confounded with a model difference. Do not use for a clean single-variable comparison, or to design an experiment that has not yet run.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[660,661,662,663,664],{"name":643,"slug":644,"type":16},{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":648,"slug":649,"type":16},{"name":665,"slug":666,"type":16},"Performance","performance","2026-08-20T03:53:40.036077",{"slug":669,"name":669,"fn":670,"description":671,"org":672,"tags":673,"stars":21,"repoUrl":22,"updatedAt":683},"braintrust-build-eval-dataset","create and manage LLM eval datasets","Create, edit, audit, or compare eval datasets for LLM applications and agents, including target-population definition, case sourcing from production traces, stratified sampling, label provenance and label audits, expected values as constraints for open-ended tasks, dev\u002Ftest splits, contamination and leakage controls, headroom checks, refresh policy, and datasheets. Use when working on the content or lifecycle of an eval dataset. Do not use for sample-size or power calculations, scorer implementation, or open-ended adversarial failure discovery.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[674,677,678,681,682],{"name":675,"slug":676,"type":16},"Agents","agents",{"name":9,"slug":8,"type":16},{"name":679,"slug":680,"type":16},"Datasets","datasets",{"name":14,"slug":15,"type":16},{"name":648,"slug":649,"type":16},"2026-08-20T03:53:33.304815",{"slug":685,"name":685,"fn":686,"description":687,"org":688,"tags":689,"stars":21,"repoUrl":22,"updatedAt":699},"braintrust-define-eval-objective","define LLM evaluation objectives","Create, edit, or audit an eval objective by working backward from a product decision to the target outcome, construct, population, intended claim, and verification-versus-validation questions. Use when a team is unsure what an eval should establish, asks \"what are we actually trying to measure,\" \"is this eval measuring the right thing,\" \"does this benchmark support our claim,\" or needs to turn a product goal into an eval objective and state which claims are out of scope. Do not use to select detailed metrics, design datasets, or implement scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[690,691,692,693,696],{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":648,"slug":649,"type":16},{"name":694,"slug":695,"type":16},"Product Management","product-management",{"name":697,"slug":698,"type":16},"Strategy","strategy","2026-08-20T03:53:00.07097",{"slug":701,"name":701,"fn":702,"description":703,"org":704,"tags":705,"stars":21,"repoUrl":22,"updatedAt":713},"braintrust-define-eval-release-gate","configure release gates for LLM applications","Create, edit, audit, or apply release gates for LLM applications and agents. Use to combine minimum meaningful improvement, statistical significance, regression rate, subgroup consistency, worst-run stability, all-attempts reliability, safety upper bounds, latency, and cost into an explicit ship-or-hold policy, to turn metrics into a CI gate, or to explain why a candidate failed a gate and what evidence would justify reconsideration. Do not use for general result analysis without a deployment decision.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[706,707,708,711,712],{"name":675,"slug":676,"type":16},{"name":9,"slug":8,"type":16},{"name":709,"slug":710,"type":16},"CI\u002FCD","ci-cd",{"name":14,"slug":15,"type":16},{"name":648,"slug":649,"type":16},"2026-08-20T03:53:15.686158",{"slug":4,"name":4,"fn":5,"description":6,"org":715,"tags":716,"stars":21,"repoUrl":22,"updatedAt":23},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[717,718,719],{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":721,"name":721,"fn":722,"description":723,"org":724,"tags":725,"stars":21,"repoUrl":22,"updatedAt":733},"braintrust-design-eval-experiment","design controlled LLM eval experiments","Design or audit controlled eval experiments for model, prompt, retrieval, tool, guardrail, or agent-architecture changes. Use before data collection to state directional and minimum-effect hypotheses, name independent, dependent, and control variables including the serving environment and tool surface, choose paired designs, set repetitions and allocation, distinguish exploratory from confirmatory comparisons, and pre-specify stopping, exclusion, multiplicity, and analysis rules. Do not use primarily to analyze results already collected.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[726,727,728,731,732],{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":729,"slug":730,"type":16},"Experiments","experiments",{"name":648,"slug":649,"type":16},{"name":697,"slug":698,"type":16},"2026-08-20T03:53:36.534554",{"slug":735,"name":735,"fn":736,"description":737,"org":738,"tags":739,"stars":21,"repoUrl":22,"updatedAt":749},"braintrust-design-eval-instrumentation","design trace and evaluation dataset schemas","Design the trace and eval-dataset schema for an LLM app or agent, and wire the system to emit it. Use when deciding what to log, designing a trace schema, setting up tracing or observability before evals, or when failures cannot be debugged or sliced from existing traces — covering inputs, outputs, spans for tool and LLM calls, state changes, metadata, resolved configuration, serving path, tool manifest, per-item status, attachments, and subgroup variables. Do not use to decide what the evidence should mean, or to write scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[740,741,742,743,746],{"name":9,"slug":8,"type":16},{"name":679,"slug":680,"type":16},{"name":14,"slug":15,"type":16},{"name":744,"slug":745,"type":16},"Observability","observability",{"name":747,"slug":748,"type":16},"Tracing","tracing","2026-08-20T03:53:37.274703",{"slug":751,"name":751,"fn":752,"description":753,"org":754,"tags":755,"stars":21,"repoUrl":22,"updatedAt":761},"braintrust-design-eval-metric-bundle","create multi-objective evaluation metric bundles","Create, edit, audit, or compare a multi-objective eval metric bundle covering product quality, safety, reliability, latency, and cost. Use to choose metrics for an eval, define a goodness bundle, distinguish optimization metrics from non-regression guardrails, expose tradeoffs, audit a KPI or single composite score for Goodhart and metric-gaming risk, or answer \"what should improve and what must not regress.\" Do not use to design trace schemas, build datasets, or implement scoring methods.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[756,757,758,759,760],{"name":675,"slug":676,"type":16},{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":648,"slug":649,"type":16},{"name":665,"slug":666,"type":16},"2026-08-20T03:53:16.078609",{"slug":763,"name":763,"fn":764,"description":765,"org":766,"tags":767,"stars":21,"repoUrl":22,"updatedAt":773},"braintrust-design-human-eval-review","design human evaluation and review workflows","Design or audit human evaluation workflows and golden datasets for LLM applications and agents. Use to set up expert review, select review cases, write reviewer instructions, assign raters, capture rationales and confidence, measure inter-rater agreement with kappa or alpha, adjudicate disagreements, and preserve reviewed examples with provenance as a versioned reference set. Do not use to elicit the criteria or rubric in the first place, to validate a scorer once reference labels exist, or to implement the scorer.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[768,769,770,771,772],{"name":675,"slug":676,"type":16},{"name":9,"slug":8,"type":16},{"name":679,"slug":680,"type":16},{"name":14,"slug":15,"type":16},{"name":648,"slug":649,"type":16},"2026-08-20T03:53:36.900129",{"slug":775,"name":775,"fn":776,"description":777,"org":778,"tags":779,"stars":21,"repoUrl":22,"updatedAt":787},"braintrust-discover-agent-failures","identify and classify agent failure modes","Search open-endedly for unanticipated agent failure modes and convert them into a named taxonomy and durable regression items. Use for requests to find out what goes wrong, surface unknown or silent failures, do error analysis over traces, cluster and triage production failures, or build a failure taxonomy — where the goal is discovering modes nobody thought to test rather than measuring a predefined criterion. Produces datasets and taxonomies, not headline scores. Do not use for adversarial attacks against a threat model, or to measure a known criterion.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[780,781,782,783,784],{"name":675,"slug":676,"type":16},{"name":9,"slug":8,"type":16},{"name":628,"slug":629,"type":16},{"name":14,"slug":15,"type":16},{"name":785,"slug":786,"type":16},"Triage","triage","2026-08-20T03:53:00.423941",27,{"items":790,"total":845},[791,799,807,815,823,831,837],{"slug":637,"name":637,"fn":638,"description":639,"org":792,"tags":793,"stars":21,"repoUrl":22,"updatedAt":653},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[794,795,796,797,798],{"name":643,"slug":644,"type":16},{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":648,"slug":649,"type":16},{"name":651,"slug":652,"type":16},{"slug":655,"name":655,"fn":656,"description":657,"org":800,"tags":801,"stars":21,"repoUrl":22,"updatedAt":667},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[802,803,804,805,806],{"name":643,"slug":644,"type":16},{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":648,"slug":649,"type":16},{"name":665,"slug":666,"type":16},{"slug":669,"name":669,"fn":670,"description":671,"org":808,"tags":809,"stars":21,"repoUrl":22,"updatedAt":683},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[810,811,812,813,814],{"name":675,"slug":676,"type":16},{"name":9,"slug":8,"type":16},{"name":679,"slug":680,"type":16},{"name":14,"slug":15,"type":16},{"name":648,"slug":649,"type":16},{"slug":685,"name":685,"fn":686,"description":687,"org":816,"tags":817,"stars":21,"repoUrl":22,"updatedAt":699},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[818,819,820,821,822],{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":648,"slug":649,"type":16},{"name":694,"slug":695,"type":16},{"name":697,"slug":698,"type":16},{"slug":701,"name":701,"fn":702,"description":703,"org":824,"tags":825,"stars":21,"repoUrl":22,"updatedAt":713},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[826,827,828,829,830],{"name":675,"slug":676,"type":16},{"name":9,"slug":8,"type":16},{"name":709,"slug":710,"type":16},{"name":14,"slug":15,"type":16},{"name":648,"slug":649,"type":16},{"slug":4,"name":4,"fn":5,"description":6,"org":832,"tags":833,"stars":21,"repoUrl":22,"updatedAt":23},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[834,835,836],{"name":9,"slug":8,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":721,"name":721,"fn":722,"description":723,"org":838,"tags":839,"stars":21,"repoUrl":22,"updatedAt":733},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[840,841,842,843,844],{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":729,"slug":730,"type":16},{"name":648,"slug":649,"type":16},{"name":697,"slug":698,"type":16},24]