[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-braintrust-braintrust-discover-trace-topics":3,"mdc--dj5vz0-key":38,"related-org-braintrust-braintrust-discover-trace-topics":608,"related-repo-braintrust-braintrust-discover-trace-topics":781},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":27,"repoUrl":28,"updatedAt":29,"license":30,"forks":31,"topics":32,"repo":33,"sourceUrl":36,"mdContent":37},"braintrust-discover-trace-topics","deploy trace topic discovery pipelines","Design, evaluate, and deploy a Topics pipeline that discovers clusters in trace traffic — preprocessor, facet prompt, no-match policy, clustering, and the automation that runs it. Use when the label set is unknown and has to come out of the data: finding what a product is actually used for, surfacing recurring issues, or building the categorical vocabulary a classifier will later enforce. Also use to evaluate or repair an existing facet or preprocessor. Do not use when the label set is already known and stable — that is a classifier — or to run a hypothesis-driven or adversarial failure hunt.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"braintrust","Braintrust","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fbraintrust.png","braintrustdata",[13,17,20,23,26],{"name":14,"slug":15,"type":16},"Tracing","tracing","tag",{"name":18,"slug":19,"type":16},"LLM","llm",{"name":21,"slug":22,"type":16},"Evals","evals",{"name":24,"slug":25,"type":16},"Data Pipeline","data-pipeline",{"name":9,"slug":8,"type":16},7,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library","2026-08-20T03:53:00.778959",null,0,[],{"repoUrl":28,"stars":27,"forks":31,"topics":34,"description":35},[],"Braintrust eval skills library","https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library\u002Ftree\u002FHEAD\u002Fskills\u002Fbraintrust-discover-trace-topics","---\nname: braintrust-discover-trace-topics\ndescription: >-\n  Design, evaluate, and deploy a Topics pipeline that discovers clusters in trace traffic —\n  preprocessor, facet prompt, no-match policy, clustering, and the automation that runs it. Use\n  when the label set is unknown and has to come out of the data: finding what a product is\n  actually used for, surfacing recurring issues, or building the categorical vocabulary a\n  classifier will later enforce. Also use to evaluate or repair an existing facet or\n  preprocessor. Do not use when the label set is already known and stable — that is a classifier\n  — or to run a hypothesis-driven or adversarial failure hunt.\n---\n\n# Discover topics in trace traffic\n\nContract: `references\u002Finteraction-contract.md`. Calibration, templates, provenance: `references\u002Ffacet-pipeline.md`.\n\n## Trigger\n\n- \"What are people actually using this for?\" \u002F \"What kinds of issues are in our traffic?\"\n- No label set exists yet and one has to come out of the data.\n- An existing facet producing vague, duplicated, or mostly-skipped output.\n- A classifier proposal where nobody can enumerate the classes.\n\n## Do\n\n1. Pick the instrument before building one. Known, stable label set → **classifier**. Label set\n   unknown and to be discovered → **facet + clustering**. A facet returns 1–2 sentences of\n   unstructured, high-signal text; that looseness is exactly what makes clusters findable, and\n   exactly what makes it the wrong tool for enforcement.\n2. Fix **scope** first — span, trace, or group — and name the grouping key for group scope.\n   Default to trace.\n3. Get the **preprocessor** right before touching the facet prompt. It must carry user input,\n   model output, tool calls, and errors, and drop system and debug noise. **Run it on real\n   traces**; do not reason about what it would return.\n4. Write the **output policy** before the prompt: what a good facet output looks like, and what\n   counts as no-match. Outputs like \"no issues\" or \"neutral\" cluster into nothing — they are\n   excluded, not scored.\n5. Build ground truth with the **strong model**, then evaluate the **cheap facet model** against\n   it. Deterministic scorers for output validity and no-match correctness; a semantic-agreement\n   scorer for the rest.\n6. Iterate under a budget stated in advance (`reference.md`): one hypothesis, one coherent change\n   to preprocessor *or* prompt, rerun the same validation slice. Run the winner **once** on the\n   untouched split.\n7. Estimate the backfill cost and get explicit approval before rewinding.\n\n## Avoid\n\n- Do not enable an automation before validating the facet on the user's own data.\n- Do not use a facet where the classes are already known — you are paying discovery cost for\n  enforcement work, and getting an unvalidated instrument.\n- Do not return one joined transcript string per span from a preprocessor; deduplication operates\n  on atomic array items and a joined string defeats it.\n- Do not store preprocessed text in the eval dataset. Store the reference and let the pipeline\n  re-derive the text, or you are evaluating a frozen snapshot of a preprocessor you are still\n  changing.\n- Do not keep tuning against the same slice once the budget is spent.\n\n## Check\n\n- Scope named, grouping key named for group scope.\n- Preprocessor **executed** on real traces, not assumed; output carries input, output, tool calls,\n  and errors, with no system or debug noise.\n- No-match policy defined and its pattern tested against real skip cases.\n- Facet validated against reviewed examples before any automation is enabled.\n- Iteration budget stated up front; test split touched once.\n- Backfill cost estimated, presented with its uncertainty, and approved.\n\n## Risk\n\n- Good clusters follow from good facets. A bad preprocessor is invisible until the topic map is\n  useless, and by then the cost is already spent.\n- Mostly-skipped or duplicated facet output reads as \"no signal in this traffic\" when it is almost\n  always an instrumentation or preprocessing failure.\n- A facet promoted into an enforcement role is an unvalidated classifier with no fitness verdict\n  attached.\n- Clustering finds structure in anything. A clean-looking topic map is not evidence that the facet\n  measures what you think it measures.\n\n## Braintrust\n\n**Preprocessor.** `get_project_settings` for the configured and effective defaults; the built-in\n`thread` returns the LLM messages per span and the machinery dedups across them. Run\n`project_default` on current traces and confirm non-null output before assuming it works — custom\ntracing formats are where it silently returns `[]`. A custom preprocessor is TypeScript in QuickJS,\nruns independently per source span with no cross-span state, and must return atomic top-level array\nitems; the system merges them in trace order and exact-dedups. `test_preprocessor_on_trace` before\nattaching. If you write one and the project default is still `thread`, set it as the project default\nso existing and new facets inherit it.\n\n**Evaluating the facet.** Dataset rows carry **references, never materialized text** (`trace_ref`,\n`group_ref`, `_full_row_id_ref` — shapes in `reference.md`). Ground-truth run: `run_eval` with the\nfacet as task, `expected` unset, **no scorers**; spot-check the outputs. Subsequent runs pass\n`{\"experiment_name\": \"\u003Creference-experiment>\"}` as the dataset so those outputs become `expected`.\nInline facet tasks wrap the definition in `inline_function` — not `inline_prompt`, not fields\ndirectly under `task`. Facet execution normalizes any `no_match_pattern` hit to `no_match`, so score\nthe **normalized value**, never source strings like `NONE` or `Neutral`.\n\n**Saving and deploying.** `create_preprocessor`, then `create_facet` with the evaluated prompt and\nno-match behavior, then re-run a few reviewed examples through the saved facet to confirm\npersistence did not change its behavior. Reuse an existing automation only when data source, scope\nand grouping, filter, preprocessor, sampling, topic window, **and** generation cadence are all\ncompatible — matching scope and filter alone is not sufficient. Sharing an automation batches facet\ncalls, which is the reason to bother. Pass `automation_id` explicitly whenever more than one exists.\n\nRewind sizing and the cost formula are in `reference.md`. Present eligible count, window, token\nassumptions, base estimate, and uncertainty buffer, and get approval on the window *and* the\nestimate — permissions checks on the mutating tool are not a substitute for approving a spend.\n",{"data":39,"body":40},{"name":4,"description":6},{"type":41,"children":42},"root",[43,52,75,82,107,113,232,238,266,272,312,318,341,345,402,550,590],{"type":44,"tag":45,"props":46,"children":48},"element","h1",{"id":47},"discover-topics-in-trace-traffic",[49],{"type":50,"value":51},"text","Discover topics in trace traffic",{"type":44,"tag":53,"props":54,"children":55},"p",{},[56,58,65,67,73],{"type":50,"value":57},"Contract: ",{"type":44,"tag":59,"props":60,"children":62},"code",{"className":61},[],[63],{"type":50,"value":64},"references\u002Finteraction-contract.md",{"type":50,"value":66},". Calibration, templates, provenance: ",{"type":44,"tag":59,"props":68,"children":70},{"className":69},[],[71],{"type":50,"value":72},"references\u002Ffacet-pipeline.md",{"type":50,"value":74},".",{"type":44,"tag":76,"props":77,"children":79},"h2",{"id":78},"trigger",[80],{"type":50,"value":81},"Trigger",{"type":44,"tag":83,"props":84,"children":85},"ul",{},[86,92,97,102],{"type":44,"tag":87,"props":88,"children":89},"li",{},[90],{"type":50,"value":91},"\"What are people actually using this for?\" \u002F \"What kinds of issues are in our traffic?\"",{"type":44,"tag":87,"props":93,"children":94},{},[95],{"type":50,"value":96},"No label set exists yet and one has to come out of the data.",{"type":44,"tag":87,"props":98,"children":99},{},[100],{"type":50,"value":101},"An existing facet producing vague, duplicated, or mostly-skipped output.",{"type":44,"tag":87,"props":103,"children":104},{},[105],{"type":50,"value":106},"A classifier proposal where nobody can enumerate the classes.",{"type":44,"tag":76,"props":108,"children":110},{"id":109},"do",[111],{"type":50,"value":112},"Do",{"type":44,"tag":114,"props":115,"children":116},"ol",{},[117,137,149,168,180,199,227],{"type":44,"tag":87,"props":118,"children":119},{},[120,122,128,130,135],{"type":50,"value":121},"Pick the instrument before building one. Known, stable label set → ",{"type":44,"tag":123,"props":124,"children":125},"strong",{},[126],{"type":50,"value":127},"classifier",{"type":50,"value":129},". Label set\nunknown and to be discovered → ",{"type":44,"tag":123,"props":131,"children":132},{},[133],{"type":50,"value":134},"facet + clustering",{"type":50,"value":136},". A facet returns 1–2 sentences of\nunstructured, high-signal text; that looseness is exactly what makes clusters findable, and\nexactly what makes it the wrong tool for enforcement.",{"type":44,"tag":87,"props":138,"children":139},{},[140,142,147],{"type":50,"value":141},"Fix ",{"type":44,"tag":123,"props":143,"children":144},{},[145],{"type":50,"value":146},"scope",{"type":50,"value":148}," first — span, trace, or group — and name the grouping key for group scope.\nDefault to trace.",{"type":44,"tag":87,"props":150,"children":151},{},[152,154,159,161,166],{"type":50,"value":153},"Get the ",{"type":44,"tag":123,"props":155,"children":156},{},[157],{"type":50,"value":158},"preprocessor",{"type":50,"value":160}," right before touching the facet prompt. It must carry user input,\nmodel output, tool calls, and errors, and drop system and debug noise. ",{"type":44,"tag":123,"props":162,"children":163},{},[164],{"type":50,"value":165},"Run it on real\ntraces",{"type":50,"value":167},"; do not reason about what it would return.",{"type":44,"tag":87,"props":169,"children":170},{},[171,173,178],{"type":50,"value":172},"Write the ",{"type":44,"tag":123,"props":174,"children":175},{},[176],{"type":50,"value":177},"output policy",{"type":50,"value":179}," before the prompt: what a good facet output looks like, and what\ncounts as no-match. Outputs like \"no issues\" or \"neutral\" cluster into nothing — they are\nexcluded, not scored.",{"type":44,"tag":87,"props":181,"children":182},{},[183,185,190,192,197],{"type":50,"value":184},"Build ground truth with the ",{"type":44,"tag":123,"props":186,"children":187},{},[188],{"type":50,"value":189},"strong model",{"type":50,"value":191},", then evaluate the ",{"type":44,"tag":123,"props":193,"children":194},{},[195],{"type":50,"value":196},"cheap facet model",{"type":50,"value":198}," against\nit. Deterministic scorers for output validity and no-match correctness; a semantic-agreement\nscorer for the rest.",{"type":44,"tag":87,"props":200,"children":201},{},[202,204,210,212,218,220,225],{"type":50,"value":203},"Iterate under a budget stated in advance (",{"type":44,"tag":59,"props":205,"children":207},{"className":206},[],[208],{"type":50,"value":209},"reference.md",{"type":50,"value":211},"): one hypothesis, one coherent change\nto preprocessor ",{"type":44,"tag":213,"props":214,"children":215},"em",{},[216],{"type":50,"value":217},"or",{"type":50,"value":219}," prompt, rerun the same validation slice. Run the winner ",{"type":44,"tag":123,"props":221,"children":222},{},[223],{"type":50,"value":224},"once",{"type":50,"value":226}," on the\nuntouched split.",{"type":44,"tag":87,"props":228,"children":229},{},[230],{"type":50,"value":231},"Estimate the backfill cost and get explicit approval before rewinding.",{"type":44,"tag":76,"props":233,"children":235},{"id":234},"avoid",[236],{"type":50,"value":237},"Avoid",{"type":44,"tag":83,"props":239,"children":240},{},[241,246,251,256,261],{"type":44,"tag":87,"props":242,"children":243},{},[244],{"type":50,"value":245},"Do not enable an automation before validating the facet on the user's own data.",{"type":44,"tag":87,"props":247,"children":248},{},[249],{"type":50,"value":250},"Do not use a facet where the classes are already known — you are paying discovery cost for\nenforcement work, and getting an unvalidated instrument.",{"type":44,"tag":87,"props":252,"children":253},{},[254],{"type":50,"value":255},"Do not return one joined transcript string per span from a preprocessor; deduplication operates\non atomic array items and a joined string defeats it.",{"type":44,"tag":87,"props":257,"children":258},{},[259],{"type":50,"value":260},"Do not store preprocessed text in the eval dataset. Store the reference and let the pipeline\nre-derive the text, or you are evaluating a frozen snapshot of a preprocessor you are still\nchanging.",{"type":44,"tag":87,"props":262,"children":263},{},[264],{"type":50,"value":265},"Do not keep tuning against the same slice once the budget is spent.",{"type":44,"tag":76,"props":267,"children":269},{"id":268},"check",[270],{"type":50,"value":271},"Check",{"type":44,"tag":83,"props":273,"children":274},{},[275,280,292,297,302,307],{"type":44,"tag":87,"props":276,"children":277},{},[278],{"type":50,"value":279},"Scope named, grouping key named for group scope.",{"type":44,"tag":87,"props":281,"children":282},{},[283,285,290],{"type":50,"value":284},"Preprocessor ",{"type":44,"tag":123,"props":286,"children":287},{},[288],{"type":50,"value":289},"executed",{"type":50,"value":291}," on real traces, not assumed; output carries input, output, tool calls,\nand errors, with no system or debug noise.",{"type":44,"tag":87,"props":293,"children":294},{},[295],{"type":50,"value":296},"No-match policy defined and its pattern tested against real skip cases.",{"type":44,"tag":87,"props":298,"children":299},{},[300],{"type":50,"value":301},"Facet validated against reviewed examples before any automation is enabled.",{"type":44,"tag":87,"props":303,"children":304},{},[305],{"type":50,"value":306},"Iteration budget stated up front; test split touched once.",{"type":44,"tag":87,"props":308,"children":309},{},[310],{"type":50,"value":311},"Backfill cost estimated, presented with its uncertainty, and approved.",{"type":44,"tag":76,"props":313,"children":315},{"id":314},"risk",[316],{"type":50,"value":317},"Risk",{"type":44,"tag":83,"props":319,"children":320},{},[321,326,331,336],{"type":44,"tag":87,"props":322,"children":323},{},[324],{"type":50,"value":325},"Good clusters follow from good facets. A bad preprocessor is invisible until the topic map is\nuseless, and by then the cost is already spent.",{"type":44,"tag":87,"props":327,"children":328},{},[329],{"type":50,"value":330},"Mostly-skipped or duplicated facet output reads as \"no signal in this traffic\" when it is almost\nalways an instrumentation or preprocessing failure.",{"type":44,"tag":87,"props":332,"children":333},{},[334],{"type":50,"value":335},"A facet promoted into an enforcement role is an unvalidated classifier with no fitness verdict\nattached.",{"type":44,"tag":87,"props":337,"children":338},{},[339],{"type":50,"value":340},"Clustering finds structure in anything. A clean-looking topic map is not evidence that the facet\nmeasures what you think it measures.",{"type":44,"tag":76,"props":342,"children":343},{"id":8},[344],{"type":50,"value":9},{"type":44,"tag":53,"props":346,"children":347},{},[348,353,355,361,363,369,371,377,379,385,387,393,395,400],{"type":44,"tag":123,"props":349,"children":350},{},[351],{"type":50,"value":352},"Preprocessor.",{"type":50,"value":354}," ",{"type":44,"tag":59,"props":356,"children":358},{"className":357},[],[359],{"type":50,"value":360},"get_project_settings",{"type":50,"value":362}," for the configured and effective defaults; the built-in\n",{"type":44,"tag":59,"props":364,"children":366},{"className":365},[],[367],{"type":50,"value":368},"thread",{"type":50,"value":370}," returns the LLM messages per span and the machinery dedups across them. Run\n",{"type":44,"tag":59,"props":372,"children":374},{"className":373},[],[375],{"type":50,"value":376},"project_default",{"type":50,"value":378}," on current traces and confirm non-null output before assuming it works — custom\ntracing formats are where it silently returns ",{"type":44,"tag":59,"props":380,"children":382},{"className":381},[],[383],{"type":50,"value":384},"[]",{"type":50,"value":386},". A custom preprocessor is TypeScript in QuickJS,\nruns independently per source span with no cross-span state, and must return atomic top-level array\nitems; the system merges them in trace order and exact-dedups. ",{"type":44,"tag":59,"props":388,"children":390},{"className":389},[],[391],{"type":50,"value":392},"test_preprocessor_on_trace",{"type":50,"value":394}," before\nattaching. If you write one and the project default is still ",{"type":44,"tag":59,"props":396,"children":398},{"className":397},[],[399],{"type":50,"value":368},{"type":50,"value":401},", set it as the project default\nso existing and new facets inherit it.",{"type":44,"tag":53,"props":403,"children":404},{},[405,410,412,417,419,425,427,433,435,441,443,448,450,456,458,464,466,471,473,479,481,486,488,494,496,502,504,510,512,518,520,526,528,533,535,541,543,549],{"type":44,"tag":123,"props":406,"children":407},{},[408],{"type":50,"value":409},"Evaluating the facet.",{"type":50,"value":411}," Dataset rows carry ",{"type":44,"tag":123,"props":413,"children":414},{},[415],{"type":50,"value":416},"references, never materialized text",{"type":50,"value":418}," (",{"type":44,"tag":59,"props":420,"children":422},{"className":421},[],[423],{"type":50,"value":424},"trace_ref",{"type":50,"value":426},",\n",{"type":44,"tag":59,"props":428,"children":430},{"className":429},[],[431],{"type":50,"value":432},"group_ref",{"type":50,"value":434},", ",{"type":44,"tag":59,"props":436,"children":438},{"className":437},[],[439],{"type":50,"value":440},"_full_row_id_ref",{"type":50,"value":442}," — shapes in ",{"type":44,"tag":59,"props":444,"children":446},{"className":445},[],[447],{"type":50,"value":209},{"type":50,"value":449},"). Ground-truth run: ",{"type":44,"tag":59,"props":451,"children":453},{"className":452},[],[454],{"type":50,"value":455},"run_eval",{"type":50,"value":457}," with the\nfacet as task, ",{"type":44,"tag":59,"props":459,"children":461},{"className":460},[],[462],{"type":50,"value":463},"expected",{"type":50,"value":465}," unset, ",{"type":44,"tag":123,"props":467,"children":468},{},[469],{"type":50,"value":470},"no scorers",{"type":50,"value":472},"; spot-check the outputs. Subsequent runs pass\n",{"type":44,"tag":59,"props":474,"children":476},{"className":475},[],[477],{"type":50,"value":478},"{\"experiment_name\": \"\u003Creference-experiment>\"}",{"type":50,"value":480}," as the dataset so those outputs become ",{"type":44,"tag":59,"props":482,"children":484},{"className":483},[],[485],{"type":50,"value":463},{"type":50,"value":487},".\nInline facet tasks wrap the definition in ",{"type":44,"tag":59,"props":489,"children":491},{"className":490},[],[492],{"type":50,"value":493},"inline_function",{"type":50,"value":495}," — not ",{"type":44,"tag":59,"props":497,"children":499},{"className":498},[],[500],{"type":50,"value":501},"inline_prompt",{"type":50,"value":503},", not fields\ndirectly under ",{"type":44,"tag":59,"props":505,"children":507},{"className":506},[],[508],{"type":50,"value":509},"task",{"type":50,"value":511},". Facet execution normalizes any ",{"type":44,"tag":59,"props":513,"children":515},{"className":514},[],[516],{"type":50,"value":517},"no_match_pattern",{"type":50,"value":519}," hit to ",{"type":44,"tag":59,"props":521,"children":523},{"className":522},[],[524],{"type":50,"value":525},"no_match",{"type":50,"value":527},", so score\nthe ",{"type":44,"tag":123,"props":529,"children":530},{},[531],{"type":50,"value":532},"normalized value",{"type":50,"value":534},", never source strings like ",{"type":44,"tag":59,"props":536,"children":538},{"className":537},[],[539],{"type":50,"value":540},"NONE",{"type":50,"value":542}," or ",{"type":44,"tag":59,"props":544,"children":546},{"className":545},[],[547],{"type":50,"value":548},"Neutral",{"type":50,"value":74},{"type":44,"tag":53,"props":551,"children":552},{},[553,558,559,565,567,573,575,580,582,588],{"type":44,"tag":123,"props":554,"children":555},{},[556],{"type":50,"value":557},"Saving and deploying.",{"type":50,"value":354},{"type":44,"tag":59,"props":560,"children":562},{"className":561},[],[563],{"type":50,"value":564},"create_preprocessor",{"type":50,"value":566},", then ",{"type":44,"tag":59,"props":568,"children":570},{"className":569},[],[571],{"type":50,"value":572},"create_facet",{"type":50,"value":574}," with the evaluated prompt and\nno-match behavior, then re-run a few reviewed examples through the saved facet to confirm\npersistence did not change its behavior. Reuse an existing automation only when data source, scope\nand grouping, filter, preprocessor, sampling, topic window, ",{"type":44,"tag":123,"props":576,"children":577},{},[578],{"type":50,"value":579},"and",{"type":50,"value":581}," generation cadence are all\ncompatible — matching scope and filter alone is not sufficient. Sharing an automation batches facet\ncalls, which is the reason to bother. Pass ",{"type":44,"tag":59,"props":583,"children":585},{"className":584},[],[586],{"type":50,"value":587},"automation_id",{"type":50,"value":589}," explicitly whenever more than one exists.",{"type":44,"tag":53,"props":591,"children":592},{},[593,595,600,602,606],{"type":50,"value":594},"Rewind sizing and the cost formula are in ",{"type":44,"tag":59,"props":596,"children":598},{"className":597},[],[599],{"type":50,"value":209},{"type":50,"value":601},". Present eligible count, window, token\nassumptions, base estimate, and uncertainty buffer, and get approval on the window ",{"type":44,"tag":213,"props":603,"children":604},{},[605],{"type":50,"value":579},{"type":50,"value":607}," the\nestimate — permissions checks on the mutating tool are not a substitute for approving a spend.",{"items":609,"total":780},[610,626,642,656,672,688,702,714,728,742,754,766],{"slug":611,"name":611,"fn":612,"description":613,"org":614,"tags":615,"stars":623,"repoUrl":624,"updatedAt":625},"troubleshoot-braintrust-mcp","configure and troubleshoot Braintrust MCP servers","This plugin auto-configures a \"braintrust\" MCP server. If you can't see it or reach it, activate this skill\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[616,617,620],{"name":9,"slug":8,"type":16},{"name":618,"slug":619,"type":16},"Debugging","debugging",{"name":621,"slug":622,"type":16},"MCP","mcp",18,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Fbraintrust-claude-plugin","2026-07-12T08:36:13.889274",{"slug":627,"name":627,"fn":628,"description":629,"org":630,"tags":631,"stars":27,"repoUrl":28,"updatedAt":641},"braintrust-analyze-eval-experiment","analyze LLM and agent eval experiments","Analyze completed LLM or agent eval experiments using uncertainty-aware and decision-relevant methods. Use to audit run completeness and pairing, calculate confidence intervals, run paired comparisons, report wins, losses, and ties, incorporate run-to-run variance, handle multiple comparisons, inspect subgroup performance, and test fragility to favorable slices. Use when results already exist and someone asks what they mean, whether a difference is real, or which model won. Do not use to design an experiment that has not yet collected results.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[632,635,636,637,638],{"name":633,"slug":634,"type":16},"Analysis","analysis",{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":639,"slug":640,"type":16},"Statistics","statistics","2026-08-20T03:53:01.13806",{"slug":643,"name":643,"fn":644,"description":645,"org":646,"tags":647,"stars":27,"repoUrl":28,"updatedAt":655},"braintrust-attribute-multi-variable-change","attribute performance changes to multiple variables","Attribute an observed change when several things moved at once — model plus prompt plus tools, a provider migration, a framework upgrade, or a vendor swap that bundles serving stack with model. Use when asked which part of a change caused the result, when a comparison's arms differ in more than one way, when a treatment has no uniform implementation across vendors, or when a serving-stack difference is confounded with a model difference. Do not use for a clean single-variable comparison, or to design an experiment that has not yet run.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[648,649,650,651,652],{"name":633,"slug":634,"type":16},{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":653,"slug":654,"type":16},"Performance","performance","2026-08-20T03:53:40.036077",{"slug":657,"name":657,"fn":658,"description":659,"org":660,"tags":661,"stars":27,"repoUrl":28,"updatedAt":671},"braintrust-build-eval-dataset","create and manage LLM eval datasets","Create, edit, audit, or compare eval datasets for LLM applications and agents, including target-population definition, case sourcing from production traces, stratified sampling, label provenance and label audits, expected values as constraints for open-ended tasks, dev\u002Ftest splits, contamination and leakage controls, headroom checks, refresh policy, and datasheets. Use when working on the content or lifecycle of an eval dataset. Do not use for sample-size or power calculations, scorer implementation, or open-ended adversarial failure discovery.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[662,665,666,669,670],{"name":663,"slug":664,"type":16},"Agents","agents",{"name":9,"slug":8,"type":16},{"name":667,"slug":668,"type":16},"Datasets","datasets",{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},"2026-08-20T03:53:33.304815",{"slug":673,"name":673,"fn":674,"description":675,"org":676,"tags":677,"stars":27,"repoUrl":28,"updatedAt":687},"braintrust-define-eval-objective","define LLM evaluation objectives","Create, edit, or audit an eval objective by working backward from a product decision to the target outcome, construct, population, intended claim, and verification-versus-validation questions. Use when a team is unsure what an eval should establish, asks \"what are we actually trying to measure,\" \"is this eval measuring the right thing,\" \"does this benchmark support our claim,\" or needs to turn a product goal into an eval objective and state which claims are out of scope. Do not use to select detailed metrics, design datasets, or implement scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[678,679,680,681,684],{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":682,"slug":683,"type":16},"Product Management","product-management",{"name":685,"slug":686,"type":16},"Strategy","strategy","2026-08-20T03:53:00.07097",{"slug":689,"name":689,"fn":690,"description":691,"org":692,"tags":693,"stars":27,"repoUrl":28,"updatedAt":701},"braintrust-define-eval-release-gate","configure release gates for LLM applications","Create, edit, audit, or apply release gates for LLM applications and agents. Use to combine minimum meaningful improvement, statistical significance, regression rate, subgroup consistency, worst-run stability, all-attempts reliability, safety upper bounds, latency, and cost into an explicit ship-or-hold policy, to turn metrics into a CI gate, or to explain why a candidate failed a gate and what evidence would justify reconsideration. Do not use for general result analysis without a deployment decision.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[694,695,696,699,700],{"name":663,"slug":664,"type":16},{"name":9,"slug":8,"type":16},{"name":697,"slug":698,"type":16},"CI\u002FCD","ci-cd",{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},"2026-08-20T03:53:15.686158",{"slug":703,"name":703,"fn":704,"description":705,"org":706,"tags":707,"stars":27,"repoUrl":28,"updatedAt":713},"braintrust-deploy-evaluator","deploy evaluators to Braintrust","Take a validated scorer or classifier from definition to running instrument in Braintrust — scope selection, inline testing before saving, saving as an evaluator, attaching an online-scoring rule, activating it for new traffic, and backfilling history with a rewind. Use when a scorer needs to actually run against production logs, when an online-scoring rule needs to be created or changed, or when historical traces need scoring. Do not use to decide what the scorer should measure, to write its rubric, or to establish that it agrees with human judgment.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[708,709,712],{"name":9,"slug":8,"type":16},{"name":710,"slug":711,"type":16},"Deployment","deployment",{"name":21,"slug":22,"type":16},"2026-08-20T03:53:32.558937",{"slug":715,"name":715,"fn":716,"description":717,"org":718,"tags":719,"stars":27,"repoUrl":28,"updatedAt":727},"braintrust-design-eval-experiment","design controlled LLM eval experiments","Design or audit controlled eval experiments for model, prompt, retrieval, tool, guardrail, or agent-architecture changes. Use before data collection to state directional and minimum-effect hypotheses, name independent, dependent, and control variables including the serving environment and tool surface, choose paired designs, set repetitions and allocation, distinguish exploratory from confirmatory comparisons, and pre-specify stopping, exclusion, multiplicity, and analysis rules. Do not use primarily to analyze results already collected.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[720,721,722,725,726],{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":723,"slug":724,"type":16},"Experiments","experiments",{"name":18,"slug":19,"type":16},{"name":685,"slug":686,"type":16},"2026-08-20T03:53:36.534554",{"slug":729,"name":729,"fn":730,"description":731,"org":732,"tags":733,"stars":27,"repoUrl":28,"updatedAt":741},"braintrust-design-eval-instrumentation","design trace and evaluation dataset schemas","Design the trace and eval-dataset schema for an LLM app or agent, and wire the system to emit it. Use when deciding what to log, designing a trace schema, setting up tracing or observability before evals, or when failures cannot be debugged or sliced from existing traces — covering inputs, outputs, spans for tool and LLM calls, state changes, metadata, resolved configuration, serving path, tool manifest, per-item status, attachments, and subgroup variables. Do not use to decide what the evidence should mean, or to write scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[734,735,736,737,740],{"name":9,"slug":8,"type":16},{"name":667,"slug":668,"type":16},{"name":21,"slug":22,"type":16},{"name":738,"slug":739,"type":16},"Observability","observability",{"name":14,"slug":15,"type":16},"2026-08-20T03:53:37.274703",{"slug":743,"name":743,"fn":744,"description":745,"org":746,"tags":747,"stars":27,"repoUrl":28,"updatedAt":753},"braintrust-design-eval-metric-bundle","create multi-objective evaluation metric bundles","Create, edit, audit, or compare a multi-objective eval metric bundle covering product quality, safety, reliability, latency, and cost. Use to choose metrics for an eval, define a goodness bundle, distinguish optimization metrics from non-regression guardrails, expose tradeoffs, audit a KPI or single composite score for Goodhart and metric-gaming risk, or answer \"what should improve and what must not regress.\" Do not use to design trace schemas, build datasets, or implement scoring methods.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[748,749,750,751,752],{"name":663,"slug":664,"type":16},{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":653,"slug":654,"type":16},"2026-08-20T03:53:16.078609",{"slug":755,"name":755,"fn":756,"description":757,"org":758,"tags":759,"stars":27,"repoUrl":28,"updatedAt":765},"braintrust-design-human-eval-review","design human evaluation and review workflows","Design or audit human evaluation workflows and golden datasets for LLM applications and agents. Use to set up expert review, select review cases, write reviewer instructions, assign raters, capture rationales and confidence, measure inter-rater agreement with kappa or alpha, adjudicate disagreements, and preserve reviewed examples with provenance as a versioned reference set. Do not use to elicit the criteria or rubric in the first place, to validate a scorer once reference labels exist, or to implement the scorer.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[760,761,762,763,764],{"name":663,"slug":664,"type":16},{"name":9,"slug":8,"type":16},{"name":667,"slug":668,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},"2026-08-20T03:53:36.900129",{"slug":767,"name":767,"fn":768,"description":769,"org":770,"tags":771,"stars":27,"repoUrl":28,"updatedAt":779},"braintrust-discover-agent-failures","identify and classify agent failure modes","Search open-endedly for unanticipated agent failure modes and convert them into a named taxonomy and durable regression items. Use for requests to find out what goes wrong, surface unknown or silent failures, do error analysis over traces, cluster and triage production failures, or build a failure taxonomy — where the goal is discovering modes nobody thought to test rather than measuring a predefined criterion. Produces datasets and taxonomies, not headline scores. Do not use for adversarial attacks against a threat model, or to measure a known criterion.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[772,773,774,775,776],{"name":663,"slug":664,"type":16},{"name":9,"slug":8,"type":16},{"name":618,"slug":619,"type":16},{"name":21,"slug":22,"type":16},{"name":777,"slug":778,"type":16},"Triage","triage","2026-08-20T03:53:00.423941",27,{"items":782,"total":837},[783,791,799,807,815,823,829],{"slug":627,"name":627,"fn":628,"description":629,"org":784,"tags":785,"stars":27,"repoUrl":28,"updatedAt":641},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[786,787,788,789,790],{"name":633,"slug":634,"type":16},{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":639,"slug":640,"type":16},{"slug":643,"name":643,"fn":644,"description":645,"org":792,"tags":793,"stars":27,"repoUrl":28,"updatedAt":655},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[794,795,796,797,798],{"name":633,"slug":634,"type":16},{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":653,"slug":654,"type":16},{"slug":657,"name":657,"fn":658,"description":659,"org":800,"tags":801,"stars":27,"repoUrl":28,"updatedAt":671},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[802,803,804,805,806],{"name":663,"slug":664,"type":16},{"name":9,"slug":8,"type":16},{"name":667,"slug":668,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"slug":673,"name":673,"fn":674,"description":675,"org":808,"tags":809,"stars":27,"repoUrl":28,"updatedAt":687},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[810,811,812,813,814],{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":682,"slug":683,"type":16},{"name":685,"slug":686,"type":16},{"slug":689,"name":689,"fn":690,"description":691,"org":816,"tags":817,"stars":27,"repoUrl":28,"updatedAt":701},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[818,819,820,821,822],{"name":663,"slug":664,"type":16},{"name":9,"slug":8,"type":16},{"name":697,"slug":698,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"slug":703,"name":703,"fn":704,"description":705,"org":824,"tags":825,"stars":27,"repoUrl":28,"updatedAt":713},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[826,827,828],{"name":9,"slug":8,"type":16},{"name":710,"slug":711,"type":16},{"name":21,"slug":22,"type":16},{"slug":715,"name":715,"fn":716,"description":717,"org":830,"tags":831,"stars":27,"repoUrl":28,"updatedAt":727},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[832,833,834,835,836],{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":723,"slug":724,"type":16},{"name":18,"slug":19,"type":16},{"name":685,"slug":686,"type":16},24]