[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-braintrust-braintrust-probe-capability-and-variability":3,"mdc--sv48vu-key":38,"related-repo-braintrust-braintrust-probe-capability-and-variability":418,"related-org-braintrust-braintrust-probe-capability-and-variability":521},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":27,"repoUrl":28,"updatedAt":29,"license":30,"forks":31,"topics":32,"repo":33,"sourceUrl":36,"mdContent":37},"braintrust-probe-capability-and-variability","probe system capability and output variability","Run the same dataset under controlled variants to measure either the ceiling of what a system can do or the spread of how reliably it does it. Use for questions about sensitivity to prompt paraphrases, formatting, ordering, seeds, or repeated runs; output consistency and agreement; whether a ranking survives a different prompt; flaky results; and equally for hidden, suppressed, sandbagged, or under-elicited capability, whether a low score means \"cannot\" or \"did not,\" or how prompting, demonstrations, scaffolding, tools, and fine-tuning compare. Do not use to find unknown failure modes or run adversarial attacks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"braintrust","Braintrust","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fbraintrust.png","braintrustdata",[13,17,20,23,26],{"name":14,"slug":15,"type":16},"Benchmarking","benchmarking","tag",{"name":18,"slug":19,"type":16},"LLM","llm",{"name":21,"slug":22,"type":16},"Evals","evals",{"name":24,"slug":25,"type":16},"Experiments","experiments",{"name":9,"slug":8,"type":16},7,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library","2026-08-20T03:53:37.645766",null,0,[],{"repoUrl":28,"stars":27,"forks":31,"topics":34,"description":35},[],"Braintrust eval skills library","https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library\u002Ftree\u002FHEAD\u002Fskills\u002Fbraintrust-probe-capability-and-variability","---\nname: braintrust-probe-capability-and-variability\ndescription: >-\n  Run the same dataset under controlled variants to measure either the ceiling of what a system can\n  do or the spread of how reliably it does it. Use for questions about sensitivity to prompt\n  paraphrases, formatting, ordering, seeds, or repeated runs; output consistency and agreement;\n  whether a ranking survives a different prompt; flaky results; and equally for hidden, suppressed,\n  sandbagged, or under-elicited capability, whether a low score means \"cannot\" or \"did not,\" or how\n  prompting, demonstrations, scaffolding, tools, and fine-tuning compare. Do not use to find unknown\n  failure modes or run adversarial attacks.\n---\n\n# Probe capability and variability\n\nContract: `references\u002Finteraction-contract.md`. Calibration, templates, provenance: `references\u002Felicitation-regimes.md`.\n\n## Trigger\n\n- \"Is this result stable?\" \u002F \"Does this hold with a different prompt?\" \u002F flaky numbers.\n- \"Is the model bad at this, or is our harness bad?\" \u002F a low score read as a ceiling.\n- A ranking derived from one prompt, one seed, or one run.\n\n## Do\n\n1. **Name the regime first — it selects the statistic, and mislabeling it is the most common\n   reporting error here.** Capability = max over variants (`pass@k`), licensing \"can at least do X.\"\n   Variability = spread and worst case (`pass^k`), licensing \"reliably does X.\"\n2. Build the variant set for that regime: an **elicitation ladder** for capability (prompt →\n   demonstrations → scaffolding → tools → search → fine-tuning), or **equivalence classes** for\n   variability (one scenario, several surface forms).\n3. Change one thing per variant, hold the construct and serving path constant, and record the exact\n   setup that produced each result.\n4. Repeat each condition K = 3–5 times; report mean, SD, and the **worst run** — never only best or\n   average.\n5. Attribute the result: separate what the **system** contributed from what the **harness**\n   contributed, and state which conclusions survive which variants.\n\n## Avoid\n\n- Do not average the spread away when the spread **is** the measurement.\n- Do not read a peak-elicited score as typical deployment performance, or fold peaks into an average\n  with ordinary runs.\n- Do not change the construct while improving the harness — a scaffold solving an easier task has\n  elicited nothing, and a \"paraphrase\" that changes the correct answer measures difficulty.\n- Do not treat scenario variants as independent items; cluster at the scenario level.\n- Do not confuse harness instability (rate limits, cache warming) with behavioral sensitivity.\n\n## Check\n\n- Regime named, and the reported statistic matches it.\n- Variant matrix published: factors, levels, how the construct was held constant.\n- K stated; run distribution and worst case reported.\n- Harness contribution separated from system; each result reproducible from its recorded setup.\n- Capability claims stated as a **floor**, never a ceiling.\n\n## Risk\n\n- A narrow variant set falsely suggests robustness; prompt formatting alone can swing accuracy by\n  double digits, so single-prompt rankings are a lottery.\n- Elicitation proves presence, never absence — \"we could not elicit it\" is a bounded statement about\n  effort spent.\n- Stronger elicitation can create a deployment condition no user will be in, making the number true\n  and irrelevant; if it surfaces dangerous capability, stop and escalate.\n\n## Braintrust\n\nBoth regimes share one shape: **run the same dataset under several variants as separate\nexperiments**, then compare the max (capability) or the spread (variability). Shared mechanics:\n`references\u002Fplatform-mechanics.md`. Two apply with unusual force here, because a variant matrix\nturns any inconsistency into apparent signal: pin every variant to one dataset version (§3) or\nyou are measuring dataset drift, and keep the **scorer name and version identical** across\nvariants (§5), since a scorer change between variants is indistinguishable from a real effect.\nKeep per-run results (§7) — a stored mean cannot be un-averaged, and the spread is the\nmeasurement.\n\nPut the variant dimension in the **prefix** so the matrix reads: `r0-base`, `r1-prompt`,\n`r2-fewshot`, `r3-scaffold`; or `fmt-json_set`, `fmt-md_set`. Prefix elicitation runs distinctly\n(`elicit-*`) and say in the description that a peak is a floor, not a deployment estimate —\notherwise the project's highest number gets quoted as the product's performance.\n\nThis is where an unpinned serving path actively misleads: provider change, cache warming, or batch\nvariation all present as behavioral sensitivity. Record provider, tier, and cache state per variant\nand run the matrix close together in time. Put `scenario_id` in `metadata` or every interval over\nthe matrix overstates precision, with no retroactive fix.\n",{"data":39,"body":40},{"name":4,"description":6},{"type":41,"children":42},"root",[43,52,75,82,102,108,194,200,235,241,276,282,300,304,331,397],{"type":44,"tag":45,"props":46,"children":48},"element","h1",{"id":47},"probe-capability-and-variability",[49],{"type":50,"value":51},"text","Probe capability and variability",{"type":44,"tag":53,"props":54,"children":55},"p",{},[56,58,65,67,73],{"type":50,"value":57},"Contract: ",{"type":44,"tag":59,"props":60,"children":62},"code",{"className":61},[],[63],{"type":50,"value":64},"references\u002Finteraction-contract.md",{"type":50,"value":66},". Calibration, templates, provenance: ",{"type":44,"tag":59,"props":68,"children":70},{"className":69},[],[71],{"type":50,"value":72},"references\u002Felicitation-regimes.md",{"type":50,"value":74},".",{"type":44,"tag":76,"props":77,"children":79},"h2",{"id":78},"trigger",[80],{"type":50,"value":81},"Trigger",{"type":44,"tag":83,"props":84,"children":85},"ul",{},[86,92,97],{"type":44,"tag":87,"props":88,"children":89},"li",{},[90],{"type":50,"value":91},"\"Is this result stable?\" \u002F \"Does this hold with a different prompt?\" \u002F flaky numbers.",{"type":44,"tag":87,"props":93,"children":94},{},[95],{"type":50,"value":96},"\"Is the model bad at this, or is our harness bad?\" \u002F a low score read as a ceiling.",{"type":44,"tag":87,"props":98,"children":99},{},[100],{"type":50,"value":101},"A ranking derived from one prompt, one seed, or one run.",{"type":44,"tag":76,"props":103,"children":105},{"id":104},"do",[106],{"type":50,"value":107},"Do",{"type":44,"tag":109,"props":110,"children":111},"ol",{},[112,139,158,163,175],{"type":44,"tag":87,"props":113,"children":114},{},[115,121,123,129,131,137],{"type":44,"tag":116,"props":117,"children":118},"strong",{},[119],{"type":50,"value":120},"Name the regime first — it selects the statistic, and mislabeling it is the most common\nreporting error here.",{"type":50,"value":122}," Capability = max over variants (",{"type":44,"tag":59,"props":124,"children":126},{"className":125},[],[127],{"type":50,"value":128},"pass@k",{"type":50,"value":130},"), licensing \"can at least do X.\"\nVariability = spread and worst case (",{"type":44,"tag":59,"props":132,"children":134},{"className":133},[],[135],{"type":50,"value":136},"pass^k",{"type":50,"value":138},"), licensing \"reliably does X.\"",{"type":44,"tag":87,"props":140,"children":141},{},[142,144,149,151,156],{"type":50,"value":143},"Build the variant set for that regime: an ",{"type":44,"tag":116,"props":145,"children":146},{},[147],{"type":50,"value":148},"elicitation ladder",{"type":50,"value":150}," for capability (prompt →\ndemonstrations → scaffolding → tools → search → fine-tuning), or ",{"type":44,"tag":116,"props":152,"children":153},{},[154],{"type":50,"value":155},"equivalence classes",{"type":50,"value":157}," for\nvariability (one scenario, several surface forms).",{"type":44,"tag":87,"props":159,"children":160},{},[161],{"type":50,"value":162},"Change one thing per variant, hold the construct and serving path constant, and record the exact\nsetup that produced each result.",{"type":44,"tag":87,"props":164,"children":165},{},[166,168,173],{"type":50,"value":167},"Repeat each condition K = 3–5 times; report mean, SD, and the ",{"type":44,"tag":116,"props":169,"children":170},{},[171],{"type":50,"value":172},"worst run",{"type":50,"value":174}," — never only best or\naverage.",{"type":44,"tag":87,"props":176,"children":177},{},[178,180,185,187,192],{"type":50,"value":179},"Attribute the result: separate what the ",{"type":44,"tag":116,"props":181,"children":182},{},[183],{"type":50,"value":184},"system",{"type":50,"value":186}," contributed from what the ",{"type":44,"tag":116,"props":188,"children":189},{},[190],{"type":50,"value":191},"harness",{"type":50,"value":193},"\ncontributed, and state which conclusions survive which variants.",{"type":44,"tag":76,"props":195,"children":197},{"id":196},"avoid",[198],{"type":50,"value":199},"Avoid",{"type":44,"tag":83,"props":201,"children":202},{},[203,215,220,225,230],{"type":44,"tag":87,"props":204,"children":205},{},[206,208,213],{"type":50,"value":207},"Do not average the spread away when the spread ",{"type":44,"tag":116,"props":209,"children":210},{},[211],{"type":50,"value":212},"is",{"type":50,"value":214}," the measurement.",{"type":44,"tag":87,"props":216,"children":217},{},[218],{"type":50,"value":219},"Do not read a peak-elicited score as typical deployment performance, or fold peaks into an average\nwith ordinary runs.",{"type":44,"tag":87,"props":221,"children":222},{},[223],{"type":50,"value":224},"Do not change the construct while improving the harness — a scaffold solving an easier task has\nelicited nothing, and a \"paraphrase\" that changes the correct answer measures difficulty.",{"type":44,"tag":87,"props":226,"children":227},{},[228],{"type":50,"value":229},"Do not treat scenario variants as independent items; cluster at the scenario level.",{"type":44,"tag":87,"props":231,"children":232},{},[233],{"type":50,"value":234},"Do not confuse harness instability (rate limits, cache warming) with behavioral sensitivity.",{"type":44,"tag":76,"props":236,"children":238},{"id":237},"check",[239],{"type":50,"value":240},"Check",{"type":44,"tag":83,"props":242,"children":243},{},[244,249,254,259,264],{"type":44,"tag":87,"props":245,"children":246},{},[247],{"type":50,"value":248},"Regime named, and the reported statistic matches it.",{"type":44,"tag":87,"props":250,"children":251},{},[252],{"type":50,"value":253},"Variant matrix published: factors, levels, how the construct was held constant.",{"type":44,"tag":87,"props":255,"children":256},{},[257],{"type":50,"value":258},"K stated; run distribution and worst case reported.",{"type":44,"tag":87,"props":260,"children":261},{},[262],{"type":50,"value":263},"Harness contribution separated from system; each result reproducible from its recorded setup.",{"type":44,"tag":87,"props":265,"children":266},{},[267,269,274],{"type":50,"value":268},"Capability claims stated as a ",{"type":44,"tag":116,"props":270,"children":271},{},[272],{"type":50,"value":273},"floor",{"type":50,"value":275},", never a ceiling.",{"type":44,"tag":76,"props":277,"children":279},{"id":278},"risk",[280],{"type":50,"value":281},"Risk",{"type":44,"tag":83,"props":283,"children":284},{},[285,290,295],{"type":44,"tag":87,"props":286,"children":287},{},[288],{"type":50,"value":289},"A narrow variant set falsely suggests robustness; prompt formatting alone can swing accuracy by\ndouble digits, so single-prompt rankings are a lottery.",{"type":44,"tag":87,"props":291,"children":292},{},[293],{"type":50,"value":294},"Elicitation proves presence, never absence — \"we could not elicit it\" is a bounded statement about\neffort spent.",{"type":44,"tag":87,"props":296,"children":297},{},[298],{"type":50,"value":299},"Stronger elicitation can create a deployment condition no user will be in, making the number true\nand irrelevant; if it surfaces dangerous capability, stop and escalate.",{"type":44,"tag":76,"props":301,"children":302},{"id":8},[303],{"type":50,"value":9},{"type":44,"tag":53,"props":305,"children":306},{},[307,309,314,316,322,324,329],{"type":50,"value":308},"Both regimes share one shape: ",{"type":44,"tag":116,"props":310,"children":311},{},[312],{"type":50,"value":313},"run the same dataset under several variants as separate\nexperiments",{"type":50,"value":315},", then compare the max (capability) or the spread (variability). Shared mechanics:\n",{"type":44,"tag":59,"props":317,"children":319},{"className":318},[],[320],{"type":50,"value":321},"references\u002Fplatform-mechanics.md",{"type":50,"value":323},". Two apply with unusual force here, because a variant matrix\nturns any inconsistency into apparent signal: pin every variant to one dataset version (§3) or\nyou are measuring dataset drift, and keep the ",{"type":44,"tag":116,"props":325,"children":326},{},[327],{"type":50,"value":328},"scorer name and version identical",{"type":50,"value":330}," across\nvariants (§5), since a scorer change between variants is indistinguishable from a real effect.\nKeep per-run results (§7) — a stored mean cannot be un-averaged, and the spread is the\nmeasurement.",{"type":44,"tag":53,"props":332,"children":333},{},[334,336,341,343,349,351,357,359,365,366,372,374,380,381,387,389,395],{"type":50,"value":335},"Put the variant dimension in the ",{"type":44,"tag":116,"props":337,"children":338},{},[339],{"type":50,"value":340},"prefix",{"type":50,"value":342}," so the matrix reads: ",{"type":44,"tag":59,"props":344,"children":346},{"className":345},[],[347],{"type":50,"value":348},"r0-base",{"type":50,"value":350},", ",{"type":44,"tag":59,"props":352,"children":354},{"className":353},[],[355],{"type":50,"value":356},"r1-prompt",{"type":50,"value":358},",\n",{"type":44,"tag":59,"props":360,"children":362},{"className":361},[],[363],{"type":50,"value":364},"r2-fewshot",{"type":50,"value":350},{"type":44,"tag":59,"props":367,"children":369},{"className":368},[],[370],{"type":50,"value":371},"r3-scaffold",{"type":50,"value":373},"; or ",{"type":44,"tag":59,"props":375,"children":377},{"className":376},[],[378],{"type":50,"value":379},"fmt-json_set",{"type":50,"value":350},{"type":44,"tag":59,"props":382,"children":384},{"className":383},[],[385],{"type":50,"value":386},"fmt-md_set",{"type":50,"value":388},". Prefix elicitation runs distinctly\n(",{"type":44,"tag":59,"props":390,"children":392},{"className":391},[],[393],{"type":50,"value":394},"elicit-*",{"type":50,"value":396},") and say in the description that a peak is a floor, not a deployment estimate —\notherwise the project's highest number gets quoted as the product's performance.",{"type":44,"tag":53,"props":398,"children":399},{},[400,402,408,410,416],{"type":50,"value":401},"This is where an unpinned serving path actively misleads: provider change, cache warming, or batch\nvariation all present as behavioral sensitivity. Record provider, tier, and cache state per variant\nand run the matrix close together in time. Put ",{"type":44,"tag":59,"props":403,"children":405},{"className":404},[],[406],{"type":50,"value":407},"scenario_id",{"type":50,"value":409}," in ",{"type":44,"tag":59,"props":411,"children":413},{"className":412},[],[414],{"type":50,"value":415},"metadata",{"type":50,"value":417}," or every interval over\nthe matrix overstates precision, with no retroactive fix.",{"items":419,"total":520},[420,436,450,466,482,496,508],{"slug":421,"name":421,"fn":422,"description":423,"org":424,"tags":425,"stars":27,"repoUrl":28,"updatedAt":435},"braintrust-analyze-eval-experiment","analyze LLM and agent eval experiments","Analyze completed LLM or agent eval experiments using uncertainty-aware and decision-relevant methods. Use to audit run completeness and pairing, calculate confidence intervals, run paired comparisons, report wins, losses, and ties, incorporate run-to-run variance, handle multiple comparisons, inspect subgroup performance, and test fragility to favorable slices. Use when results already exist and someone asks what they mean, whether a difference is real, or which model won. Do not use to design an experiment that has not yet collected results.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[426,429,430,431,432],{"name":427,"slug":428,"type":16},"Analysis","analysis",{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":433,"slug":434,"type":16},"Statistics","statistics","2026-08-20T03:53:01.13806",{"slug":437,"name":437,"fn":438,"description":439,"org":440,"tags":441,"stars":27,"repoUrl":28,"updatedAt":449},"braintrust-attribute-multi-variable-change","attribute performance changes to multiple variables","Attribute an observed change when several things moved at once — model plus prompt plus tools, a provider migration, a framework upgrade, or a vendor swap that bundles serving stack with model. Use when asked which part of a change caused the result, when a comparison's arms differ in more than one way, when a treatment has no uniform implementation across vendors, or when a serving-stack difference is confounded with a model difference. Do not use for a clean single-variable comparison, or to design an experiment that has not yet run.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[442,443,444,445,446],{"name":427,"slug":428,"type":16},{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":447,"slug":448,"type":16},"Performance","performance","2026-08-20T03:53:40.036077",{"slug":451,"name":451,"fn":452,"description":453,"org":454,"tags":455,"stars":27,"repoUrl":28,"updatedAt":465},"braintrust-build-eval-dataset","create and manage LLM eval datasets","Create, edit, audit, or compare eval datasets for LLM applications and agents, including target-population definition, case sourcing from production traces, stratified sampling, label provenance and label audits, expected values as constraints for open-ended tasks, dev\u002Ftest splits, contamination and leakage controls, headroom checks, refresh policy, and datasheets. Use when working on the content or lifecycle of an eval dataset. Do not use for sample-size or power calculations, scorer implementation, or open-ended adversarial failure discovery.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[456,459,460,463,464],{"name":457,"slug":458,"type":16},"Agents","agents",{"name":9,"slug":8,"type":16},{"name":461,"slug":462,"type":16},"Datasets","datasets",{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},"2026-08-20T03:53:33.304815",{"slug":467,"name":467,"fn":468,"description":469,"org":470,"tags":471,"stars":27,"repoUrl":28,"updatedAt":481},"braintrust-define-eval-objective","define LLM evaluation objectives","Create, edit, or audit an eval objective by working backward from a product decision to the target outcome, construct, population, intended claim, and verification-versus-validation questions. Use when a team is unsure what an eval should establish, asks \"what are we actually trying to measure,\" \"is this eval measuring the right thing,\" \"does this benchmark support our claim,\" or needs to turn a product goal into an eval objective and state which claims are out of scope. Do not use to select detailed metrics, design datasets, or implement scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[472,473,474,475,478],{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":476,"slug":477,"type":16},"Product Management","product-management",{"name":479,"slug":480,"type":16},"Strategy","strategy","2026-08-20T03:53:00.07097",{"slug":483,"name":483,"fn":484,"description":485,"org":486,"tags":487,"stars":27,"repoUrl":28,"updatedAt":495},"braintrust-define-eval-release-gate","configure release gates for LLM applications","Create, edit, audit, or apply release gates for LLM applications and agents. Use to combine minimum meaningful improvement, statistical significance, regression rate, subgroup consistency, worst-run stability, all-attempts reliability, safety upper bounds, latency, and cost into an explicit ship-or-hold policy, to turn metrics into a CI gate, or to explain why a candidate failed a gate and what evidence would justify reconsideration. Do not use for general result analysis without a deployment decision.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[488,489,490,493,494],{"name":457,"slug":458,"type":16},{"name":9,"slug":8,"type":16},{"name":491,"slug":492,"type":16},"CI\u002FCD","ci-cd",{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},"2026-08-20T03:53:15.686158",{"slug":497,"name":497,"fn":498,"description":499,"org":500,"tags":501,"stars":27,"repoUrl":28,"updatedAt":507},"braintrust-deploy-evaluator","deploy evaluators to Braintrust","Take a validated scorer or classifier from definition to running instrument in Braintrust — scope selection, inline testing before saving, saving as an evaluator, attaching an online-scoring rule, activating it for new traffic, and backfilling history with a rewind. Use when a scorer needs to actually run against production logs, when an online-scoring rule needs to be created or changed, or when historical traces need scoring. Do not use to decide what the scorer should measure, to write its rubric, or to establish that it agrees with human judgment.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[502,503,506],{"name":9,"slug":8,"type":16},{"name":504,"slug":505,"type":16},"Deployment","deployment",{"name":21,"slug":22,"type":16},"2026-08-20T03:53:32.558937",{"slug":509,"name":509,"fn":510,"description":511,"org":512,"tags":513,"stars":27,"repoUrl":28,"updatedAt":519},"braintrust-design-eval-experiment","design controlled LLM eval experiments","Design or audit controlled eval experiments for model, prompt, retrieval, tool, guardrail, or agent-architecture changes. Use before data collection to state directional and minimum-effect hypotheses, name independent, dependent, and control variables including the serving environment and tool surface, choose paired designs, set repetitions and allocation, distinguish exploratory from confirmatory comparisons, and pre-specify stopping, exclusion, multiplicity, and analysis rules. Do not use primarily to analyze results already collected.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[514,515,516,517,518],{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},{"name":479,"slug":480,"type":16},"2026-08-20T03:53:36.534554",24,{"items":522,"total":647},[523,539,547,555,563,571,579,585,593,609,621,633],{"slug":524,"name":524,"fn":525,"description":526,"org":527,"tags":528,"stars":536,"repoUrl":537,"updatedAt":538},"troubleshoot-braintrust-mcp","configure and troubleshoot Braintrust MCP servers","This plugin auto-configures a \"braintrust\" MCP server. If you can't see it or reach it, activate this skill\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[529,530,533],{"name":9,"slug":8,"type":16},{"name":531,"slug":532,"type":16},"Debugging","debugging",{"name":534,"slug":535,"type":16},"MCP","mcp",18,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Fbraintrust-claude-plugin","2026-07-12T08:36:13.889274",{"slug":421,"name":421,"fn":422,"description":423,"org":540,"tags":541,"stars":27,"repoUrl":28,"updatedAt":435},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[542,543,544,545,546],{"name":427,"slug":428,"type":16},{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":433,"slug":434,"type":16},{"slug":437,"name":437,"fn":438,"description":439,"org":548,"tags":549,"stars":27,"repoUrl":28,"updatedAt":449},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[550,551,552,553,554],{"name":427,"slug":428,"type":16},{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":447,"slug":448,"type":16},{"slug":451,"name":451,"fn":452,"description":453,"org":556,"tags":557,"stars":27,"repoUrl":28,"updatedAt":465},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[558,559,560,561,562],{"name":457,"slug":458,"type":16},{"name":9,"slug":8,"type":16},{"name":461,"slug":462,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"slug":467,"name":467,"fn":468,"description":469,"org":564,"tags":565,"stars":27,"repoUrl":28,"updatedAt":481},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[566,567,568,569,570],{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":476,"slug":477,"type":16},{"name":479,"slug":480,"type":16},{"slug":483,"name":483,"fn":484,"description":485,"org":572,"tags":573,"stars":27,"repoUrl":28,"updatedAt":495},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[574,575,576,577,578],{"name":457,"slug":458,"type":16},{"name":9,"slug":8,"type":16},{"name":491,"slug":492,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"slug":497,"name":497,"fn":498,"description":499,"org":580,"tags":581,"stars":27,"repoUrl":28,"updatedAt":507},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[582,583,584],{"name":9,"slug":8,"type":16},{"name":504,"slug":505,"type":16},{"name":21,"slug":22,"type":16},{"slug":509,"name":509,"fn":510,"description":511,"org":586,"tags":587,"stars":27,"repoUrl":28,"updatedAt":519},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[588,589,590,591,592],{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},{"name":479,"slug":480,"type":16},{"slug":594,"name":594,"fn":595,"description":596,"org":597,"tags":598,"stars":27,"repoUrl":28,"updatedAt":608},"braintrust-design-eval-instrumentation","design trace and evaluation dataset schemas","Design the trace and eval-dataset schema for an LLM app or agent, and wire the system to emit it. Use when deciding what to log, designing a trace schema, setting up tracing or observability before evals, or when failures cannot be debugged or sliced from existing traces — covering inputs, outputs, spans for tool and LLM calls, state changes, metadata, resolved configuration, serving path, tool manifest, per-item status, attachments, and subgroup variables. Do not use to decide what the evidence should mean, or to write scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[599,600,601,602,605],{"name":9,"slug":8,"type":16},{"name":461,"slug":462,"type":16},{"name":21,"slug":22,"type":16},{"name":603,"slug":604,"type":16},"Observability","observability",{"name":606,"slug":607,"type":16},"Tracing","tracing","2026-08-20T03:53:37.274703",{"slug":610,"name":610,"fn":611,"description":612,"org":613,"tags":614,"stars":27,"repoUrl":28,"updatedAt":620},"braintrust-design-eval-metric-bundle","create multi-objective evaluation metric bundles","Create, edit, audit, or compare a multi-objective eval metric bundle covering product quality, safety, reliability, latency, and cost. Use to choose metrics for an eval, define a goodness bundle, distinguish optimization metrics from non-regression guardrails, expose tradeoffs, audit a KPI or single composite score for Goodhart and metric-gaming risk, or answer \"what should improve and what must not regress.\" Do not use to design trace schemas, build datasets, or implement scoring methods.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[615,616,617,618,619],{"name":457,"slug":458,"type":16},{"name":9,"slug":8,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":447,"slug":448,"type":16},"2026-08-20T03:53:16.078609",{"slug":622,"name":622,"fn":623,"description":624,"org":625,"tags":626,"stars":27,"repoUrl":28,"updatedAt":632},"braintrust-design-human-eval-review","design human evaluation and review workflows","Design or audit human evaluation workflows and golden datasets for LLM applications and agents. Use to set up expert review, select review cases, write reviewer instructions, assign raters, capture rationales and confidence, measure inter-rater agreement with kappa or alpha, adjudicate disagreements, and preserve reviewed examples with provenance as a versioned reference set. Do not use to elicit the criteria or rubric in the first place, to validate a scorer once reference labels exist, or to implement the scorer.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[627,628,629,630,631],{"name":457,"slug":458,"type":16},{"name":9,"slug":8,"type":16},{"name":461,"slug":462,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},"2026-08-20T03:53:36.900129",{"slug":634,"name":634,"fn":635,"description":636,"org":637,"tags":638,"stars":27,"repoUrl":28,"updatedAt":646},"braintrust-discover-agent-failures","identify and classify agent failure modes","Search open-endedly for unanticipated agent failure modes and convert them into a named taxonomy and durable regression items. Use for requests to find out what goes wrong, surface unknown or silent failures, do error analysis over traces, cluster and triage production failures, or build a failure taxonomy — where the goal is discovering modes nobody thought to test rather than measuring a predefined criterion. Produces datasets and taxonomies, not headline scores. Do not use for adversarial attacks against a threat model, or to measure a known criterion.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[639,640,641,642,643],{"name":457,"slug":458,"type":16},{"name":9,"slug":8,"type":16},{"name":531,"slug":532,"type":16},{"name":21,"slug":22,"type":16},{"name":644,"slug":645,"type":16},"Triage","triage","2026-08-20T03:53:00.423941",27]