[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-braintrust-braintrust-monitor-production-evals":3,"mdc--bdyl4r-key":41,"related-org-braintrust-braintrust-monitor-production-evals":431,"related-repo-braintrust-braintrust-monitor-production-evals":601},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":30,"repoUrl":31,"updatedAt":32,"license":33,"forks":34,"topics":35,"repo":36,"sourceUrl":39,"mdContent":40},"braintrust-monitor-production-evals","monitor production LLM and agent evaluations","Design or audit online evaluation of live LLM and agent traffic: trace sampling, online scoring coverage, alert thresholds and ownership, drift and failure-slice monitoring, incident review, and the pipeline that routes production failures back into the offline dataset. Use for questions about scoring production traces, monitoring quality after launch, catching regressions in the wild, alert fatigue, scorer drift, or closing the loop from incident to eval item. Do not use to design offline experiments or to interpret a controlled comparison.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"braintrust","Braintrust","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fbraintrust.png","braintrustdata",[13,17,20,23,26,29],{"name":14,"slug":15,"type":16},"Observability","observability","tag",{"name":18,"slug":19,"type":16},"LLM","llm",{"name":21,"slug":22,"type":16},"Monitoring","monitoring",{"name":24,"slug":25,"type":16},"Evals","evals",{"name":27,"slug":28,"type":16},"Agents","agents",{"name":9,"slug":8,"type":16},7,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library","2026-08-20T03:53:13.51467",null,0,[],{"repoUrl":31,"stars":30,"forks":34,"topics":37,"description":38},[],"Braintrust eval skills library","https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Feval-library\u002Ftree\u002FHEAD\u002Fskills\u002Fbraintrust-monitor-production-evals","---\nname: braintrust-monitor-production-evals\ndescription: >-\n  Design or audit online evaluation of live LLM and agent traffic: trace sampling, online scoring\n  coverage, alert thresholds and ownership, drift and failure-slice monitoring, incident review, and\n  the pipeline that routes production failures back into the offline dataset. Use for questions\n  about scoring production traces, monitoring quality after launch, catching regressions in the\n  wild, alert fatigue, scorer drift, or closing the loop from incident to eval item. Do not use to\n  design offline experiments or to interpret a controlled comparison.\n---\n\n# Monitor evals in production\n\nContract: `references\u002Finteraction-contract.md`. Calibration, templates, provenance: `references\u002Fmonitoring-design.md`.\n\n## Trigger\n\n- \"How do we know it's still working?\" \u002F \"Score our production traces.\"\n- Alerts firing that nobody acts on, or incidents that never became eval items.\n- A release gate that passed while users report problems.\n\n## Do\n\n1. Frame the loop: offline experiments gate the release, online scoring monitors the shipped system,\n   observed failures become new dataset items, and the next offline eval is more representative.\n   Online has perfect distributional realism and **no controlled comparison**.\n2. Define coverage deliberately — which traffic, at what rate, which scorers. **Stratify** so\n   rare-but-severe slices are covered; proportional sampling misses the failures that matter most.\n   The sampling rate is simultaneously the coverage decision and the **cost** decision, and the\n   two pull opposite ways; decide it as one tradeoff rather than setting coverage and discovering\n   the bill.\n3. Reuse offline scorers where they were validated for monitoring, and state which use each is\n   approved for. A scorer below the gating bar may still serve trend monitoring.\n4. Give every alert a threshold with **documented provenance**, an **owner**, and an **action**. An\n   alert with no action is a notification.\n5. Monitor drift on **both sides**: input distribution shift and scorer drift. Re-validate on a\n   schedule, not on suspicion.\n6. Watch the **silent** failures — most agent failures raise no error, so error-rate dashboards look\n   healthy while quality degrades.\n7. Close the loop mechanically: flagged traces through review into the versioned dataset, with\n   provenance and incident reference. Apply privacy and retention controls **at collection time**.\n\n## Avoid\n\n- Do not treat online scores as a controlled experiment; a production metric moving is\n  observational and licenses no causal claim.\n- Do not collect production data without privacy and retention controls in place first.\n- Do not alert on every dip — alert fatigue destroys the channel, and the fix is fewer,\n  better-owned alerts.\n- Do not let the golden set ossify while traffic drifts; a loop that never updates the dataset is\n  not closed.\n- Do not design offline experiments here.\n\n## Check\n\n- Sampling plan states rate, stratification, coverage of rare severe slices, and cost at that rate.\n- Every online scorer's approved use documented, with its validation date.\n- Any historical backfill has its window, eligible-unit count, and estimated cost stated before it\n  is submitted.\n- Every alert has threshold provenance, owner, action, and review latency.\n- Drift monitored on both input distribution and scorer behavior.\n- Ingestion path from flagged trace to versioned dataset item exists, with an owner.\n- Privacy, PII handling, and retention documented per collected field.\n\n## Risk\n\n- Distribution shift silently invalidates thresholds from an older baseline; the alert stays quiet\n  while the population underneath changes.\n- Scorer drift under a provider model update looks exactly like a product regression.\n- Privacy limits may make the most informative traces uncollectable — design around what is\n  retainable rather than discovering the constraint later.\n- Monitoring sees only what is instrumented, so trace-schema gaps become permanent production blind\n  spots.\n\n## Braintrust\n\nApply **the same scorers** that ran offline — same names, same versions\n(`references\u002Fplatform-mechanics.md` §5) — via **online scoring**. This is the case §5 exists for:\na differently-named production scorer produces a second series that looks comparable and is not.\nUse **trace classifications** for the categorical side —\nspec violations, suspected failure modes, attack families — since classifications are what you\nslice and alert on, and they double as the clustering step for open-ended failure discovery. Record\neach scorer's approved use in its description so nobody promotes a trend scorer into a gate.\n\nClose the loop with **human review queues**: sample live traces → review → append to a **versioned**\ngolden dataset, with provenance in metadata (`trace_id`, incident ref, date, reviewer,\n`source: production-\u003CYYYY-MM>`). Route the scorer-disagreement queue in too.\n\nStanding a scorer up on live traffic — rule configuration, sampling, activation, and historical\nbackfill — is `braintrust-deploy-evaluator`. Two things from it bear directly on monitoring\ndesign. **Activating a rule for new traffic and rewinding it over history are separate actions\nrequiring separate authorization**, and a paused rule accepts a rewind without processing it, which\nis how teams end up believing history is being scored when nothing is running. And a backfill's\ncost is eligible units × sampling rate × per-call judge cost, which on a high-volume project is\neasy to misjudge by an order of magnitude — estimate before submitting, and submit once.\n\nRecord each alert's baseline window in its own description, so a quiet alert is distinguishable\nfrom a moved population. Detect judge drift by re-running the **calibration experiment** on a\nschedule — the same calibration run used to validate it. For non-text inputs, the raw artifact must be a\nspan **Attachment** or incident review cannot separate \"the system failed\" from \"the input was\nunintelligible.\"\n",{"data":42,"body":43},{"name":4,"description":6},{"type":44,"children":45},"root",[46,55,78,85,105,111,212,218,246,252,290,296,319,323,357,392,412],{"type":47,"tag":48,"props":49,"children":51},"element","h1",{"id":50},"monitor-evals-in-production",[52],{"type":53,"value":54},"text","Monitor evals in production",{"type":47,"tag":56,"props":57,"children":58},"p",{},[59,61,68,70,76],{"type":53,"value":60},"Contract: ",{"type":47,"tag":62,"props":63,"children":65},"code",{"className":64},[],[66],{"type":53,"value":67},"references\u002Finteraction-contract.md",{"type":53,"value":69},". Calibration, templates, provenance: ",{"type":47,"tag":62,"props":71,"children":73},{"className":72},[],[74],{"type":53,"value":75},"references\u002Fmonitoring-design.md",{"type":53,"value":77},".",{"type":47,"tag":79,"props":80,"children":82},"h2",{"id":81},"trigger",[83],{"type":53,"value":84},"Trigger",{"type":47,"tag":86,"props":87,"children":88},"ul",{},[89,95,100],{"type":47,"tag":90,"props":91,"children":92},"li",{},[93],{"type":53,"value":94},"\"How do we know it's still working?\" \u002F \"Score our production traces.\"",{"type":47,"tag":90,"props":96,"children":97},{},[98],{"type":53,"value":99},"Alerts firing that nobody acts on, or incidents that never became eval items.",{"type":47,"tag":90,"props":101,"children":102},{},[103],{"type":53,"value":104},"A release gate that passed while users report problems.",{"type":47,"tag":79,"props":106,"children":108},{"id":107},"do",[109],{"type":53,"value":110},"Do",{"type":47,"tag":112,"props":113,"children":114},"ol",{},[115,127,146,151,177,189,201],{"type":47,"tag":90,"props":116,"children":117},{},[118,120,126],{"type":53,"value":119},"Frame the loop: offline experiments gate the release, online scoring monitors the shipped system,\nobserved failures become new dataset items, and the next offline eval is more representative.\nOnline has perfect distributional realism and ",{"type":47,"tag":121,"props":122,"children":123},"strong",{},[124],{"type":53,"value":125},"no controlled comparison",{"type":53,"value":77},{"type":47,"tag":90,"props":128,"children":129},{},[130,132,137,139,144],{"type":53,"value":131},"Define coverage deliberately — which traffic, at what rate, which scorers. ",{"type":47,"tag":121,"props":133,"children":134},{},[135],{"type":53,"value":136},"Stratify",{"type":53,"value":138}," so\nrare-but-severe slices are covered; proportional sampling misses the failures that matter most.\nThe sampling rate is simultaneously the coverage decision and the ",{"type":47,"tag":121,"props":140,"children":141},{},[142],{"type":53,"value":143},"cost",{"type":53,"value":145}," decision, and the\ntwo pull opposite ways; decide it as one tradeoff rather than setting coverage and discovering\nthe bill.",{"type":47,"tag":90,"props":147,"children":148},{},[149],{"type":53,"value":150},"Reuse offline scorers where they were validated for monitoring, and state which use each is\napproved for. A scorer below the gating bar may still serve trend monitoring.",{"type":47,"tag":90,"props":152,"children":153},{},[154,156,161,163,168,170,175],{"type":53,"value":155},"Give every alert a threshold with ",{"type":47,"tag":121,"props":157,"children":158},{},[159],{"type":53,"value":160},"documented provenance",{"type":53,"value":162},", an ",{"type":47,"tag":121,"props":164,"children":165},{},[166],{"type":53,"value":167},"owner",{"type":53,"value":169},", and an ",{"type":47,"tag":121,"props":171,"children":172},{},[173],{"type":53,"value":174},"action",{"type":53,"value":176},". An\nalert with no action is a notification.",{"type":47,"tag":90,"props":178,"children":179},{},[180,182,187],{"type":53,"value":181},"Monitor drift on ",{"type":47,"tag":121,"props":183,"children":184},{},[185],{"type":53,"value":186},"both sides",{"type":53,"value":188},": input distribution shift and scorer drift. Re-validate on a\nschedule, not on suspicion.",{"type":47,"tag":90,"props":190,"children":191},{},[192,194,199],{"type":53,"value":193},"Watch the ",{"type":47,"tag":121,"props":195,"children":196},{},[197],{"type":53,"value":198},"silent",{"type":53,"value":200}," failures — most agent failures raise no error, so error-rate dashboards look\nhealthy while quality degrades.",{"type":47,"tag":90,"props":202,"children":203},{},[204,206,211],{"type":53,"value":205},"Close the loop mechanically: flagged traces through review into the versioned dataset, with\nprovenance and incident reference. Apply privacy and retention controls ",{"type":47,"tag":121,"props":207,"children":208},{},[209],{"type":53,"value":210},"at collection time",{"type":53,"value":77},{"type":47,"tag":79,"props":213,"children":215},{"id":214},"avoid",[216],{"type":53,"value":217},"Avoid",{"type":47,"tag":86,"props":219,"children":220},{},[221,226,231,236,241],{"type":47,"tag":90,"props":222,"children":223},{},[224],{"type":53,"value":225},"Do not treat online scores as a controlled experiment; a production metric moving is\nobservational and licenses no causal claim.",{"type":47,"tag":90,"props":227,"children":228},{},[229],{"type":53,"value":230},"Do not collect production data without privacy and retention controls in place first.",{"type":47,"tag":90,"props":232,"children":233},{},[234],{"type":53,"value":235},"Do not alert on every dip — alert fatigue destroys the channel, and the fix is fewer,\nbetter-owned alerts.",{"type":47,"tag":90,"props":237,"children":238},{},[239],{"type":53,"value":240},"Do not let the golden set ossify while traffic drifts; a loop that never updates the dataset is\nnot closed.",{"type":47,"tag":90,"props":242,"children":243},{},[244],{"type":53,"value":245},"Do not design offline experiments here.",{"type":47,"tag":79,"props":247,"children":249},{"id":248},"check",[250],{"type":53,"value":251},"Check",{"type":47,"tag":86,"props":253,"children":254},{},[255,260,265,270,275,280,285],{"type":47,"tag":90,"props":256,"children":257},{},[258],{"type":53,"value":259},"Sampling plan states rate, stratification, coverage of rare severe slices, and cost at that rate.",{"type":47,"tag":90,"props":261,"children":262},{},[263],{"type":53,"value":264},"Every online scorer's approved use documented, with its validation date.",{"type":47,"tag":90,"props":266,"children":267},{},[268],{"type":53,"value":269},"Any historical backfill has its window, eligible-unit count, and estimated cost stated before it\nis submitted.",{"type":47,"tag":90,"props":271,"children":272},{},[273],{"type":53,"value":274},"Every alert has threshold provenance, owner, action, and review latency.",{"type":47,"tag":90,"props":276,"children":277},{},[278],{"type":53,"value":279},"Drift monitored on both input distribution and scorer behavior.",{"type":47,"tag":90,"props":281,"children":282},{},[283],{"type":53,"value":284},"Ingestion path from flagged trace to versioned dataset item exists, with an owner.",{"type":47,"tag":90,"props":286,"children":287},{},[288],{"type":53,"value":289},"Privacy, PII handling, and retention documented per collected field.",{"type":47,"tag":79,"props":291,"children":293},{"id":292},"risk",[294],{"type":53,"value":295},"Risk",{"type":47,"tag":86,"props":297,"children":298},{},[299,304,309,314],{"type":47,"tag":90,"props":300,"children":301},{},[302],{"type":53,"value":303},"Distribution shift silently invalidates thresholds from an older baseline; the alert stays quiet\nwhile the population underneath changes.",{"type":47,"tag":90,"props":305,"children":306},{},[307],{"type":53,"value":308},"Scorer drift under a provider model update looks exactly like a product regression.",{"type":47,"tag":90,"props":310,"children":311},{},[312],{"type":53,"value":313},"Privacy limits may make the most informative traces uncollectable — design around what is\nretainable rather than discovering the constraint later.",{"type":47,"tag":90,"props":315,"children":316},{},[317],{"type":53,"value":318},"Monitoring sees only what is instrumented, so trace-schema gaps become permanent production blind\nspots.",{"type":47,"tag":79,"props":320,"children":321},{"id":8},[322],{"type":53,"value":9},{"type":47,"tag":56,"props":324,"children":325},{},[326,328,333,335,341,343,348,350,355],{"type":53,"value":327},"Apply ",{"type":47,"tag":121,"props":329,"children":330},{},[331],{"type":53,"value":332},"the same scorers",{"type":53,"value":334}," that ran offline — same names, same versions\n(",{"type":47,"tag":62,"props":336,"children":338},{"className":337},[],[339],{"type":53,"value":340},"references\u002Fplatform-mechanics.md",{"type":53,"value":342}," §5) — via ",{"type":47,"tag":121,"props":344,"children":345},{},[346],{"type":53,"value":347},"online scoring",{"type":53,"value":349},". This is the case §5 exists for:\na differently-named production scorer produces a second series that looks comparable and is not.\nUse ",{"type":47,"tag":121,"props":351,"children":352},{},[353],{"type":53,"value":354},"trace classifications",{"type":53,"value":356}," for the categorical side —\nspec violations, suspected failure modes, attack families — since classifications are what you\nslice and alert on, and they double as the clustering step for open-ended failure discovery. Record\neach scorer's approved use in its description so nobody promotes a trend scorer into a gate.",{"type":47,"tag":56,"props":358,"children":359},{},[360,362,367,369,374,376,382,384,390],{"type":53,"value":361},"Close the loop with ",{"type":47,"tag":121,"props":363,"children":364},{},[365],{"type":53,"value":366},"human review queues",{"type":53,"value":368},": sample live traces → review → append to a ",{"type":47,"tag":121,"props":370,"children":371},{},[372],{"type":53,"value":373},"versioned",{"type":53,"value":375},"\ngolden dataset, with provenance in metadata (",{"type":47,"tag":62,"props":377,"children":379},{"className":378},[],[380],{"type":53,"value":381},"trace_id",{"type":53,"value":383},", incident ref, date, reviewer,\n",{"type":47,"tag":62,"props":385,"children":387},{"className":386},[],[388],{"type":53,"value":389},"source: production-\u003CYYYY-MM>",{"type":53,"value":391},"). Route the scorer-disagreement queue in too.",{"type":47,"tag":56,"props":393,"children":394},{},[395,397,403,405,410],{"type":53,"value":396},"Standing a scorer up on live traffic — rule configuration, sampling, activation, and historical\nbackfill — is ",{"type":47,"tag":62,"props":398,"children":400},{"className":399},[],[401],{"type":53,"value":402},"braintrust-deploy-evaluator",{"type":53,"value":404},". Two things from it bear directly on monitoring\ndesign. ",{"type":47,"tag":121,"props":406,"children":407},{},[408],{"type":53,"value":409},"Activating a rule for new traffic and rewinding it over history are separate actions\nrequiring separate authorization",{"type":53,"value":411},", and a paused rule accepts a rewind without processing it, which\nis how teams end up believing history is being scored when nothing is running. And a backfill's\ncost is eligible units × sampling rate × per-call judge cost, which on a high-volume project is\neasy to misjudge by an order of magnitude — estimate before submitting, and submit once.",{"type":47,"tag":56,"props":413,"children":414},{},[415,417,422,424,429],{"type":53,"value":416},"Record each alert's baseline window in its own description, so a quiet alert is distinguishable\nfrom a moved population. Detect judge drift by re-running the ",{"type":47,"tag":121,"props":418,"children":419},{},[420],{"type":53,"value":421},"calibration experiment",{"type":53,"value":423}," on a\nschedule — the same calibration run used to validate it. For non-text inputs, the raw artifact must be a\nspan ",{"type":47,"tag":121,"props":425,"children":426},{},[427],{"type":53,"value":428},"Attachment",{"type":53,"value":430}," or incident review cannot separate \"the system failed\" from \"the input was\nunintelligible.\"",{"items":432,"total":600},[433,449,465,479,493,509,523,534,548,562,574,586],{"slug":434,"name":434,"fn":435,"description":436,"org":437,"tags":438,"stars":446,"repoUrl":447,"updatedAt":448},"troubleshoot-braintrust-mcp","configure and troubleshoot Braintrust MCP servers","This plugin auto-configures a \"braintrust\" MCP server. If you can't see it or reach it, activate this skill\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[439,440,443],{"name":9,"slug":8,"type":16},{"name":441,"slug":442,"type":16},"Debugging","debugging",{"name":444,"slug":445,"type":16},"MCP","mcp",18,"https:\u002F\u002Fgithub.com\u002Fbraintrustdata\u002Fbraintrust-claude-plugin","2026-07-12T08:36:13.889274",{"slug":450,"name":450,"fn":451,"description":452,"org":453,"tags":454,"stars":30,"repoUrl":31,"updatedAt":464},"braintrust-analyze-eval-experiment","analyze LLM and agent eval experiments","Analyze completed LLM or agent eval experiments using uncertainty-aware and decision-relevant methods. Use to audit run completeness and pairing, calculate confidence intervals, run paired comparisons, report wins, losses, and ties, incorporate run-to-run variance, handle multiple comparisons, inspect subgroup performance, and test fragility to favorable slices. Use when results already exist and someone asks what they mean, whether a difference is real, or which model won. Do not use to design an experiment that has not yet collected results.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[455,458,459,460,461],{"name":456,"slug":457,"type":16},"Analysis","analysis",{"name":9,"slug":8,"type":16},{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},{"name":462,"slug":463,"type":16},"Statistics","statistics","2026-08-20T03:53:01.13806",{"slug":466,"name":466,"fn":467,"description":468,"org":469,"tags":470,"stars":30,"repoUrl":31,"updatedAt":478},"braintrust-attribute-multi-variable-change","attribute performance changes to multiple variables","Attribute an observed change when several things moved at once — model plus prompt plus tools, a provider migration, a framework upgrade, or a vendor swap that bundles serving stack with model. Use when asked which part of a change caused the result, when a comparison's arms differ in more than one way, when a treatment has no uniform implementation across vendors, or when a serving-stack difference is confounded with a model difference. Do not use for a clean single-variable comparison, or to design an experiment that has not yet run.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[471,472,473,474,475],{"name":456,"slug":457,"type":16},{"name":9,"slug":8,"type":16},{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},{"name":476,"slug":477,"type":16},"Performance","performance","2026-08-20T03:53:40.036077",{"slug":480,"name":480,"fn":481,"description":482,"org":483,"tags":484,"stars":30,"repoUrl":31,"updatedAt":492},"braintrust-build-eval-dataset","create and manage LLM eval datasets","Create, edit, audit, or compare eval datasets for LLM applications and agents, including target-population definition, case sourcing from production traces, stratified sampling, label provenance and label audits, expected values as constraints for open-ended tasks, dev\u002Ftest splits, contamination and leakage controls, headroom checks, refresh policy, and datasheets. Use when working on the content or lifecycle of an eval dataset. Do not use for sample-size or power calculations, scorer implementation, or open-ended adversarial failure discovery.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[485,486,487,490,491],{"name":27,"slug":28,"type":16},{"name":9,"slug":8,"type":16},{"name":488,"slug":489,"type":16},"Datasets","datasets",{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},"2026-08-20T03:53:33.304815",{"slug":494,"name":494,"fn":495,"description":496,"org":497,"tags":498,"stars":30,"repoUrl":31,"updatedAt":508},"braintrust-define-eval-objective","define LLM evaluation objectives","Create, edit, or audit an eval objective by working backward from a product decision to the target outcome, construct, population, intended claim, and verification-versus-validation questions. Use when a team is unsure what an eval should establish, asks \"what are we actually trying to measure,\" \"is this eval measuring the right thing,\" \"does this benchmark support our claim,\" or needs to turn a product goal into an eval objective and state which claims are out of scope. Do not use to select detailed metrics, design datasets, or implement scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[499,500,501,502,505],{"name":9,"slug":8,"type":16},{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},{"name":503,"slug":504,"type":16},"Product Management","product-management",{"name":506,"slug":507,"type":16},"Strategy","strategy","2026-08-20T03:53:00.07097",{"slug":510,"name":510,"fn":511,"description":512,"org":513,"tags":514,"stars":30,"repoUrl":31,"updatedAt":522},"braintrust-define-eval-release-gate","configure release gates for LLM applications","Create, edit, audit, or apply release gates for LLM applications and agents. Use to combine minimum meaningful improvement, statistical significance, regression rate, subgroup consistency, worst-run stability, all-attempts reliability, safety upper bounds, latency, and cost into an explicit ship-or-hold policy, to turn metrics into a CI gate, or to explain why a candidate failed a gate and what evidence would justify reconsideration. Do not use for general result analysis without a deployment decision.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[515,516,517,520,521],{"name":27,"slug":28,"type":16},{"name":9,"slug":8,"type":16},{"name":518,"slug":519,"type":16},"CI\u002FCD","ci-cd",{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},"2026-08-20T03:53:15.686158",{"slug":402,"name":402,"fn":524,"description":525,"org":526,"tags":527,"stars":30,"repoUrl":31,"updatedAt":533},"deploy evaluators to Braintrust","Take a validated scorer or classifier from definition to running instrument in Braintrust — scope selection, inline testing before saving, saving as an evaluator, attaching an online-scoring rule, activating it for new traffic, and backfilling history with a rewind. Use when a scorer needs to actually run against production logs, when an online-scoring rule needs to be created or changed, or when historical traces need scoring. Do not use to decide what the scorer should measure, to write its rubric, or to establish that it agrees with human judgment.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[528,529,532],{"name":9,"slug":8,"type":16},{"name":530,"slug":531,"type":16},"Deployment","deployment",{"name":24,"slug":25,"type":16},"2026-08-20T03:53:32.558937",{"slug":535,"name":535,"fn":536,"description":537,"org":538,"tags":539,"stars":30,"repoUrl":31,"updatedAt":547},"braintrust-design-eval-experiment","design controlled LLM eval experiments","Design or audit controlled eval experiments for model, prompt, retrieval, tool, guardrail, or agent-architecture changes. Use before data collection to state directional and minimum-effect hypotheses, name independent, dependent, and control variables including the serving environment and tool surface, choose paired designs, set repetitions and allocation, distinguish exploratory from confirmatory comparisons, and pre-specify stopping, exclusion, multiplicity, and analysis rules. Do not use primarily to analyze results already collected.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[540,541,542,545,546],{"name":9,"slug":8,"type":16},{"name":24,"slug":25,"type":16},{"name":543,"slug":544,"type":16},"Experiments","experiments",{"name":18,"slug":19,"type":16},{"name":506,"slug":507,"type":16},"2026-08-20T03:53:36.534554",{"slug":549,"name":549,"fn":550,"description":551,"org":552,"tags":553,"stars":30,"repoUrl":31,"updatedAt":561},"braintrust-design-eval-instrumentation","design trace and evaluation dataset schemas","Design the trace and eval-dataset schema for an LLM app or agent, and wire the system to emit it. Use when deciding what to log, designing a trace schema, setting up tracing or observability before evals, or when failures cannot be debugged or sliced from existing traces — covering inputs, outputs, spans for tool and LLM calls, state changes, metadata, resolved configuration, serving path, tool manifest, per-item status, attachments, and subgroup variables. Do not use to decide what the evidence should mean, or to write scorers.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[554,555,556,557,558],{"name":9,"slug":8,"type":16},{"name":488,"slug":489,"type":16},{"name":24,"slug":25,"type":16},{"name":14,"slug":15,"type":16},{"name":559,"slug":560,"type":16},"Tracing","tracing","2026-08-20T03:53:37.274703",{"slug":563,"name":563,"fn":564,"description":565,"org":566,"tags":567,"stars":30,"repoUrl":31,"updatedAt":573},"braintrust-design-eval-metric-bundle","create multi-objective evaluation metric bundles","Create, edit, audit, or compare a multi-objective eval metric bundle covering product quality, safety, reliability, latency, and cost. Use to choose metrics for an eval, define a goodness bundle, distinguish optimization metrics from non-regression guardrails, expose tradeoffs, audit a KPI or single composite score for Goodhart and metric-gaming risk, or answer \"what should improve and what must not regress.\" Do not use to design trace schemas, build datasets, or implement scoring methods.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[568,569,570,571,572],{"name":27,"slug":28,"type":16},{"name":9,"slug":8,"type":16},{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},{"name":476,"slug":477,"type":16},"2026-08-20T03:53:16.078609",{"slug":575,"name":575,"fn":576,"description":577,"org":578,"tags":579,"stars":30,"repoUrl":31,"updatedAt":585},"braintrust-design-human-eval-review","design human evaluation and review workflows","Design or audit human evaluation workflows and golden datasets for LLM applications and agents. Use to set up expert review, select review cases, write reviewer instructions, assign raters, capture rationales and confidence, measure inter-rater agreement with kappa or alpha, adjudicate disagreements, and preserve reviewed examples with provenance as a versioned reference set. Do not use to elicit the criteria or rubric in the first place, to validate a scorer once reference labels exist, or to implement the scorer.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[580,581,582,583,584],{"name":27,"slug":28,"type":16},{"name":9,"slug":8,"type":16},{"name":488,"slug":489,"type":16},{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},"2026-08-20T03:53:36.900129",{"slug":587,"name":587,"fn":588,"description":589,"org":590,"tags":591,"stars":30,"repoUrl":31,"updatedAt":599},"braintrust-discover-agent-failures","identify and classify agent failure modes","Search open-endedly for unanticipated agent failure modes and convert them into a named taxonomy and durable regression items. Use for requests to find out what goes wrong, surface unknown or silent failures, do error analysis over traces, cluster and triage production failures, or build a failure taxonomy — where the goal is discovering modes nobody thought to test rather than measuring a predefined criterion. Produces datasets and taxonomies, not headline scores. Do not use for adversarial attacks against a threat model, or to measure a known criterion.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[592,593,594,595,596],{"name":27,"slug":28,"type":16},{"name":9,"slug":8,"type":16},{"name":441,"slug":442,"type":16},{"name":24,"slug":25,"type":16},{"name":597,"slug":598,"type":16},"Triage","triage","2026-08-20T03:53:00.423941",27,{"items":602,"total":657},[603,611,619,627,635,643,649],{"slug":450,"name":450,"fn":451,"description":452,"org":604,"tags":605,"stars":30,"repoUrl":31,"updatedAt":464},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[606,607,608,609,610],{"name":456,"slug":457,"type":16},{"name":9,"slug":8,"type":16},{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},{"name":462,"slug":463,"type":16},{"slug":466,"name":466,"fn":467,"description":468,"org":612,"tags":613,"stars":30,"repoUrl":31,"updatedAt":478},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[614,615,616,617,618],{"name":456,"slug":457,"type":16},{"name":9,"slug":8,"type":16},{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},{"name":476,"slug":477,"type":16},{"slug":480,"name":480,"fn":481,"description":482,"org":620,"tags":621,"stars":30,"repoUrl":31,"updatedAt":492},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[622,623,624,625,626],{"name":27,"slug":28,"type":16},{"name":9,"slug":8,"type":16},{"name":488,"slug":489,"type":16},{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},{"slug":494,"name":494,"fn":495,"description":496,"org":628,"tags":629,"stars":30,"repoUrl":31,"updatedAt":508},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[630,631,632,633,634],{"name":9,"slug":8,"type":16},{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},{"name":503,"slug":504,"type":16},{"name":506,"slug":507,"type":16},{"slug":510,"name":510,"fn":511,"description":512,"org":636,"tags":637,"stars":30,"repoUrl":31,"updatedAt":522},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[638,639,640,641,642],{"name":27,"slug":28,"type":16},{"name":9,"slug":8,"type":16},{"name":518,"slug":519,"type":16},{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},{"slug":402,"name":402,"fn":524,"description":525,"org":644,"tags":645,"stars":30,"repoUrl":31,"updatedAt":533},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[646,647,648],{"name":9,"slug":8,"type":16},{"name":530,"slug":531,"type":16},{"name":24,"slug":25,"type":16},{"slug":535,"name":535,"fn":536,"description":537,"org":650,"tags":651,"stars":30,"repoUrl":31,"updatedAt":547},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[652,653,654,655,656],{"name":9,"slug":8,"type":16},{"name":24,"slug":25,"type":16},{"name":543,"slug":544,"type":16},{"name":18,"slug":19,"type":16},{"name":506,"slug":507,"type":16},24]