[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-mlflow-build-a-scorer":3,"mdc--vrod5l-key":33,"related-org-mlflow-build-a-scorer":2318,"related-repo-mlflow-build-a-scorer":2476},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":31,"mdContent":32},"build-a-scorer","build MLflow evaluation prototypes","Help the user go from zero to a shipped MLflow evaluation prototype by understanding their app, generalizing a small set of atomic quality criteria, and implementing each criterion with the cheapest reliable scorer. Use when the user wants help choosing, creating, or iterating MLflow judges\u002Fscorers for an agent, RAG app, LLM app, or GenAI workflow.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"mlflow","MLflow","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fmlflow.png",[12,14,17,20],{"name":9,"slug":8,"type":13},"tag",{"name":15,"slug":16,"type":13},"Machine Learning","machine-learning",{"name":18,"slug":19,"type":13},"Evals","evals",{"name":21,"slug":22,"type":13},"MLOps","mlops",60,"https:\u002F\u002Fgithub.com\u002Fmlflow\u002Fskills","2026-08-12T04:50:08.725497",null,19,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":26},[],"https:\u002F\u002Fgithub.com\u002Fmlflow\u002Fskills\u002Ftree\u002FHEAD\u002Fbuild-a-scorer","---\nname: build-a-scorer\ndescription: Help the user go from zero to a shipped MLflow evaluation prototype by understanding their app, generalizing a small set of atomic quality criteria, and implementing each criterion with the cheapest reliable scorer. Use when the user wants help choosing, creating, or iterating MLflow judges\u002Fscorers for an agent, RAG app, LLM app, or GenAI workflow.\nallowed-tools: Read, Write, Bash, Grep, Glob\n---\n\n# build-a-scorer\n\nHelp the user get from **zero to shipped evaluation prototype**. Not 100% correctness on the first\npass — a stable, understandable scorer suite they can run, inspect, and iterate on.\n\nYour primary job is **understanding**, and the user is the expert on what counts as wrong. Do not\nstart from \"which MLflow scorer should I use?\"\n\nNorth star:\n\n> Understand the app well enough to define a small set of atomic, durable quality criteria, then\n> implement each criterion with the cheapest reliable scorer.\n\n## Doctrine\n\n1. **Prototype first.** Ship a useful v1 that runs now. Mark v2 upgrades explicitly.\n2. **Small suite.** 3-5 high-signal scorers, hard cap. Adding one means dropping a weaker one.\n3. **One scorer, one criterion.** If a criterion contains \"and\", split it. No \"overall quality\" judge.\n4. **Cheapest reliable implementation.** Code\u002Frules beat built-ins beat hand-written LLM judges.\n   See Phase 4 — the rule most often skipped, and skipping it costs real money at scale.\n   Never hand-roll a judge for something MLflow already ships.\n5. **Cover outcomes, not just rules.** A suite of only policy checks tells you the agent behaved\n   while saying nothing about whether users succeeded. Every suite needs at least one check on\n   whether the user got what they came for. See Phase 3.\n6. **Binary outputs.** Prefer `bool`, or `\"yes\"`\u002F`\"no\"` for LLM judges — other strings are silently\n   dropped from metrics (Phase 4). Avoid `0.0-1.0` without a calibration story; pass rates debug\n   better than averaged floats.\n7. **Traces are optional.** Inspect them if they exist; otherwise use dataset + `predict_fn`. Never\n   block the user on tracing unless the criterion truly needs execution internals.\n8. **Align later.** Treat every first LLM judge as a draft. Remind the user to align it once traces\n   and human labels arrive.\n9. **Talk before code.** The conversation is where right and wrong get settled. A large code block\n   ends it — users stop reading and start implementing. Small snippets mid-conversation are fine;\n   the full implementation goes in your final message, after the user confirms the criteria.\n\n## Source of truth: introspect, don't memorize\n\nMLflow's scorer surface changes. Before recommending scorers, read the installed surface. Drop the\n`uv run` prefix if the project is not uv-managed:\n\n```bash\nuv run python -c \"\nfrom mlflow.genai.scorers import get_all_scorers\nimport mlflow.genai.scorers as s\nfor sc in get_all_scorers():\n    req = getattr(sc, 'required_columns', set())\n    print(f'{type(sc).__name__:32} requires={sorted(req)}  session={getattr(sc, \\\"is_session_level_scorer\\\", False)}')\nprint(sorted(n for n in dir(s) if n[0].isupper()))\n\"\n```\n\n- `get_all_scorers()` returns built-ins instantiable with defaults. Constructor-arg scorers\n  (`Guidelines`, `RegexMatch`, `ResponseLength`) may only appear in exported names.\n- `.required_columns` is the exact data contract. Use it; never guess.\n- LLM judges: `from mlflow.genai.judges import make_judge`. Code scorers:\n  `from mlflow.genai.scorers import scorer`.\n\nIf MLflow is not importable, say so, continue from first principles, and flag that names and\nrequired columns need verification.\n\n## User-facing language\n\nProduct language first (**quality check**, **judge**, **code check**, **built-in check**), API\nlanguage second (**scorer**, `make_judge`, `@scorer`, `required_columns`). State each recommendation as:\n\n```text\nQuality check: \u003Cplain-English name>     Layer: \u003Crule | outcome>\nCriterion: \u003Cone sentence, one thing only>\nImplementation: \u003Ccode check | built-in | LLM judge>\nWhy not cheaper: \u003Cfor LLM judges — the code check AND built-in you rejected, and why>\nOutput: \u003Cbool, or \"yes\"\u002F\"no\">          Data needed: \u003Cinputs\u002Foutputs\u002Fexpectations\u002Ftrace\u002Fsession>\n```\n\n## The workflow\n\n### Phase 0: Orient\n\nState the working agreement plainly:\n\n> We'll build a small v1 scorer suite that runs now, catches repeated failures, and gives you a\n> stable base to iterate. It will not be perfect yet.\n\n### Phase 1: Elicit the user's notion of right and wrong\n\nYour job is **criteria, not architecture** — what the user thinks good and bad look like, not their\nproduct spec.\n\n**Ask at most 3 questions before showing a draft.** Users correct a wrong draft far more easily than\nthey answer an interview.\n\nOpen with these, one at a time:\n\n1. What does the app do?\n2. What is the worst failure you could ship?\n3. What should it do when it is unsure or lacks information?\n\nThen stop asking and draft. Translate vague answers like \"good responses\" into named observable\ncriteria **yourself** and show them for correction — do not bounce the work back as another question.\n\nOut of scope; these are the user's job, not the judge's:\n\n- Which fields are required at which step of their business logic.\n- How the app decides between branches or states.\n- Internal schemas, gates, thresholds, or slot taxonomies.\n\nIf you catch yourself asking a second consecutive question about app internals, stop and draft.\n\n**Real failure to avoid:** eliciting \"it shouldn't book if information is incomplete\", then asking\nfour follow-ups about which fields count as complete. The first sentence was already enough.\n\n### Phase 2: Check what is observable\n\nOnce the user has reacted to a draft, establish what evidence a judge can actually see. A criterion\nis only real if it is checkable. One focused pass, not an interview.\n\nIf traces exist, inspect one or two: confirm `inputs`, `outputs`, `expectations`, span types, tool\ncalls, and whether the task is single-turn or session-level.\n\n**The RETRIEVER-vs-TOOL trap:** `RetrievalGroundedness`, `RetrievalRelevance`, and\n`RetrievalSufficiency` require `RETRIEVER` spans and **hard-raise** otherwise. A `web_search` or DB\nlookup instrumented as `TOOL` does not qualify.\n\nFor \"grounded in tool output\", **wrap a judge in a code scorer instead of passing `{{ trace }}`.** A\n`@scorer` gets the `trace`, so use `trace.search_spans()` to pull the evidence span and pass only it\nplus the claim to the judge. `{{ trace }}` serializes every span and makes the model find the right\none — costlier, and span selection becomes nondeterministic when there are several. Use `{{ trace }}`\nonly when the criterion is about the whole trace (tool sequencing, retries), with no single span to\nextract.\n\nIf traces do not exist, use `inputs` + optional `outputs`\u002F`expectations` with\n`mlflow.genai.evaluate(data=..., predict_fn=..., scorers=[...])`, and mark trace-only upgrades as\nfuture work.\n\n### Phase 3: Generalize into atomic criteria — across both layers\n\nCluster examples into repeated failure modes. Keep criteria likely to recur; park one-off oddities.\nName them like `answers_user_question`, `does_not_invent_facts`, `refuses_out_of_scope_requests` —\none behavior each.\n\n**Cover two layers. Suites that only do the first layer are the most common failure of this skill.**\n\n1. **Rule layer — did the agent misbehave on this turn?** Policy violations, wrong facts, missed\n   escalations, forbidden claims. Easy to name, easy to check, and where users' stated fears live.\n2. **Outcome layer — did the user get what they came for?** Task completion, abandonment,\n   frustration, having to repeat themselves. Harder to name, and the reason the product exists.\n\nAn agent can pass every rule check and still be useless. Ask directly: *\"What does success look\nlike for the user — what did they come here to do, and how would we know they did it?\"*\n\nOutcome checks are often **cheap**, not expensive — do not assume they need an LLM judge:\n\n- `task_completed` — did the trace contain the goal tool call (booking, application, order)? Code.\n- Session-level built-ins with no data requirements: `UserFrustration`,\n  `ConversationCompleteness`, `KnowledgeRetention`. Verify with `.required_columns`.\n- `had_to_repeat` — did the user restate the same request across turns? Code over the conversation.\n\n**Do not substitute a checkable proxy for the outcome that matters.** A check like\n`promotion_disclosed` (\"was a promo flag mentioned?\") looks rigorous but measures text presence, not\nwhether the renter got the best deal or felt well-served. Proxies are fine as *rule* checks; they do\nnot discharge the outcome layer. If the user's real goal is fuzzy, say so and pair a cheap proxy with\none honest outcome check rather than pretending the proxy covers it.\n\nKeep outcome checks atomic too. \"Meets customer expectation\" bundles found-what-they-wanted,\nwasn't-misled, and wasn't-frustrated; when it fails you learn nothing. Split it.\n\n### Phase 4: Route each criterion to the cheapest reliable implementation\n\nDo this **before** asking the user to confirm the suite, so they are approving real cost.\n\n```text\nCan code check it deterministically (or filter most cases)?\n  yes -> code check\n  no  -> Is there a built-in for this standard concept, and do we have its required data?\n          yes -> built-in\n          no  -> LLM judge\n```\n\n**Checkpoint — every hand-written LLM judge needs a `why not cheaper` cell naming both the code\ncheck and the built-in you rejected.** Not prose elsewhere in the message: the cell in the Phase 6\ntable. If you cannot fill it, the check is miscategorized. \"No built-in covers this\" counts only if\nyou actually looked.\n\n**Check the catalog before hand-rolling.** Run the introspection command above when MLflow is\nimportable; when it is not, these ship today — confirm names and `required_columns` before relying\non them, but do not act as though the catalog is empty:\n\n| Built-in | Requires | Use for |\n|---|---|---|\n| `Guidelines(guidelines=...)` | inputs, outputs | \"does the response follow this stated policy\" — the most under-used built-in |\n| `Correctness` | inputs, outputs | answer matches expected |\n| `RelevanceToQuery` | inputs, outputs | response addresses the question |\n| `Safety` \u002F `PIIDetection` | inputs, outputs \u002F outputs | harmful content; PII leakage |\n| `RegexMatch(pattern=...)` \u002F `ResponseLength(...)` | outputs | pattern and length rules |\n| `Completeness` \u002F `Fluency` \u002F `Summarization` | inputs, outputs | coverage; readability; summary quality |\n| `ToolCallCorrectness` \u002F `ToolCallEfficiency` | trace | right tools, no redundant calls |\n| `Retrieval{Groundedness,Relevance,Sufficiency}` | inputs, trace | **RETRIEVER spans only** — see Phase 2 |\n| `UserFrustration`, `ConversationCompleteness`, `KnowledgeRetention`, `ConversationalSafety`, `ConversationalRoleAdherence` | none (session-level) | multi-turn outcome and conversation quality |\n\n**A built-in is a better starting point than a blank prompt — not a substitute for your standard.**\nBuilt-ins are LLM judges carrying MLflow's generic instructions, so \"built-in\" does not imply correct\nfor your product; prefer them because the scaffolding and data contract are already tested. Two\nconsequences: `Guidelines` always applies, since it takes *your* policy text. And alignment cuts\nacross the built-in\u002Fbespoke line — `align()` is on `Judge`, so single-turn built-ins can be aligned to\nyour labels like `make_judge`, but session-level ones (`UserFrustration`, `ConversationCompleteness`,\n`KnowledgeRetention`) **raise `NotImplementedError`** and are stuck on MLflow's definition. Hand-write\na criterion that is important and contested, so it can be aligned.\n\nSkipping built-ins is legitimate when your standard genuinely differs. Skipping them without\nlooking is not.\n\nDefault routes:\n\n- Format, JSON shape, required field, regex\u002Flink, length, exact value, latency, PII pattern,\n  **specific phrases or keywords**: code check.\n- Relevance, correctness, safety, groundedness, retrieval quality, tool-call quality, fluency,\n  session-level conversation quality: built-in, if the data contract matches.\n- Domain tone, policy adherence, task-specific correctness, paraphrase-matching against a source,\n  nuanced refusal quality: LLM judge.\n\n**Prefer a hybrid over a pure judge.** Most \"semantic\" checks have a cheap deterministic front end:\na keyword or phrase scan catches the bright-line cases at ~zero cost and passes only ambiguous ones\nto a model. Reach for this whenever a criterion mentions specific commitments, categories, or\ntrigger topics — \"did it promise a refund\", \"did the user raise a billing dispute\". Detecting the\n*trigger* is usually code; judging the *response* may need a model.\n\nPush back when the user reaches for an LLM judge to count words or validate JSON, a regex to judge\ntone, one \"overall good\" scorer, or a float without a calibration reason.\n\n**Output types — only `bool`, numerics, and `\"yes\"`\u002F`\"no\"` aggregate.** Anything else is silently\ncast to `None` by `_cast_assessment_value_to_float` and **dropped from `results.metrics`** with no\nerror: `\"pass\"`, `\"fail\"`, `\"correct\"`, `\"not_applicable\"` all disappear. Verified against\n`mlflow\u002Fgenai\u002Fscorers\u002Faggregation.py`; `CategoricalRating` accepts only `yes`\u002F`no`\u002F`unknown`.\n\n- Prefer `bool` for pass\u002Ffail.\n- Use `\"yes\"`\u002F`\"no\"` when an LLM judge must return a string.\n- Represent \"criterion does not apply\" as `Feedback(value=True, rationale=\"not applicable: ...\")`,\n  or skip the row — not as a `\"not_applicable\"` string, which vanishes from the metrics.\n- Put extra detail in `rationale`, not in the value.\n\n### Phase 5: Sharpen each criterion with the domain expert\n\n**This is the highest-value phase — spend most of the conversation here.** The user is the expert on\nwhat counts as wrong. Your job is to find each criterion's **edge**: where a reasonable person could\ncall the same behavior acceptable or unacceptable.\n\n**Propose, don't interrogate.** Put candidate verdicts in front of the user and let them rule:\n\n> For \"does not steer by protected class\" — I'd fail these three: [...]. I'd pass this one because\n> the renter asked first: [...]. I'm unsure about this one: [...]. Where do you draw it?\n\nThis yields far more signal per turn than open questions, and costs the user a judgment call rather\nthan a specification.\n\nProbes, each carrying your own proposed answer:\n\n- **Near-miss.** \"Would you fail this one, almost identical but [X]?\"\n- **Legitimate exception.** \"Is there a case where this behavior is actually correct?\"\n- **Severity split.** \"Is one of these a red line and the other a papercut?\"\n- **Not-applicable.** \"What should the check say when the criterion doesn't apply?\"\n\nA criterion is done when the user has corrected or confirmed at least one proposed verdict, two\npeople would label the same example the same way, and you know what \"not applicable\" looks like.\n\nSmall snippets help here — an output shape, a three-item keyword list. Full implementations do not.\n\n### Phase 6: Confirm before implementing\n\nRestate the suite as a **table**, one row per check, with these columns filled in for every row:\nname | layer (rule\u002Foutcome) | implementation | why not cheaper | output. A blank cell is a design\nhole, not a formatting choice — an LLM judge with no \"why not cheaper\" is unroutable, and a suite\nwith no outcome row is incomplete. Then **ask the user to confirm or change it** and wait.\n\n> Here's the suite I'd build. Anything to add, drop, or reword before I write it?\n\n**Enforce the size cap here.** If you have more than 5 checks, do not ask the user to prune — say\nwhich you would drop and why, then let them overrule you. Adding a criterion means replacing a\nweaker one, not growing the list. A 9-check v1 is a checklist, and checklists do not get validated.\n\nDo not write the full implementation on the same turn as the confirmation request. If anything\nchanges, re-confirm the changed items.\n\n### Phase 7: Deliver the implementation\n\nOnly after confirmation. Deliver **one consolidated final message**, not code spread across turns.\n\n- Code checks: runnable, with imports, malformed-input guards, and a runnable\n  `mlflow.genai.evaluate(...)` call.\n- Built-ins: state the data contract from `.required_columns`. If the data is absent, re-route or\n  mark as a future upgrade.\n- LLM judges: give the actual `instructions` string tailored to their domain, with template\n  variables (`{{ inputs }}`, `{{ outputs }}`, `{{ expectations }}`, `{{ trace }}`,\n  `{{ conversation }}`), a `bool` or `\"yes\"`\u002F`\"no\"` output, and cost at their stated volume.\n  Always add: \"Once traces and human labels come in, align this judge and validate agreement before\n  relying on it for monitoring.\"\n\n**Judge model.** Set `model=` explicitly and start mid-tier — `anthropic:\u002Fclaude-sonnet-4-6`,\n`openai:\u002Fgpt-4.1`, or the equivalent tier on your provider. Scale up if the judge disagrees with\nhuman labels on clear-cut cases; scale down only after measuring agreement on a labelled sample.\nNever downgrade a red-line judge (safety, legal) to save cents — a code prefilter already keeps its\nvolume low. `agent-evaluation\u002Freferences\u002Fscorers.md` is authoritative for URI formats and keys.\n\n### Phase 8: Name the validation plan, then hand off\n\nDecide *how the suite will be judged* — then stop. Running the evaluation is `agent-evaluation`'s\njob, and this skill must not duplicate it.\n\n**Lead with labels the user already has.** Before proposing anyone hand-label a fresh dataset, ask\nwhat past failures are already recorded — support tickets, complaints, incident reports,\nthumbs-down feedback, refunds, escalations. Backtesting against real known-bad cases beats synthetic\nlabels and costs nothing to produce. Propose hand-labeling only if no such history exists.\n\nFor each LLM judge, state the alignment plan: which labels it will be aligned against, and that its\nfirst version is a draft. Be cautious aligning subjective axes like tone or style — alignment can\noverfit. Note where alignment is impossible: session-level built-ins raise `NotImplementedError`.\n\nClose with the confirmed suite, the implementation, parked v2 ideas, trace-only upgrades, and the\nvalidation plan.\n\n**Then hand off — do not run the evaluation yourself.**\n\n- `agent-evaluation` — registering scorers, dataset discovery\u002Fcreation, dry run, full evaluation,\n  results analysis, and iteration. Registration matters: unregistered inline scorers do not appear in\n  `mlflow scorers list` and are not reusable.\n- `instrumenting-with-mlflow-tracing` — tracing gaps surfaced in Phase 2, such as a lookup\n  instrumented as `TOOL` when a criterion needs `RETRIEVER`.\n\n> **Scope boundary.** This skill decides *what to measure and how to implement it*.\n> `agent-evaluation` decides *how to run it and what the results mean*. If you find yourself\n> preparing a dataset, registering scorers, or interpreting eval output, you have left this skill's\n> scope — hand off instead.\n\n## Anti-patterns\n\n- Starting from the scorer catalog instead of the user's app and failures.\n- Writing code before the user agreed to named criteria. If they ask to talk, talk.\n- Dumping a large code block mid-conversation; it ends the sharpening discussion.\n- Shipping a suite of all-LLM judges without stating why each one can't be a code check or built-in\n  — the most common and most expensive failure of this skill.\n- Hand-rolling a `make_judge` for something MLflow ships. Policy adherence is `Guidelines`;\n  pattern rules are `RegexMatch`; multi-turn quality has session-level built-ins.\n- Shipping only rule\u002Fpolicy checks with nothing measuring whether users succeeded. An agent can\n  pass every compliance check and still fail everyone who used it.\n- Substituting a checkable proxy for the outcome that matters (`promotion_disclosed` in place of\n  \"the renter got the best available deal\") and treating the outcome layer as covered.\n- Answering a cost objection with sampling alone when a cheaper implementation was available.\n- Proposing fresh hand-labeling before asking what labels the user already has.\n- Interviewing past 3 questions without showing a draft.\n- Asking about internal gates or state machines in order to write a judge.\n- Checking only the end state when the failure happened upstream in the conversation.\n- Blocking on traces when a dataset\u002F`predict_fn` prototype would work.\n- Combining multiple criteria into one scorer, or scoring one-off examples that don't generalize.\n- `0.0-1.0` scores by default; code with undefined placeholders.\n- `RetrievalGroundedness` on TOOL-only traces.\n",{"data":34,"body":36},{"name":4,"description":6,"allowed-tools":35},"Read, Write, Bash, Grep, Glob",{"type":37,"children":38},"root",[39,46,60,72,77,86,93,229,235,248,376,446,451,457,515,525,531,538,543,551,557,569,579,584,602,614,619,637,642,652,658,663,690,754,810,843,849,876,884,907,918,930,989,1014,1019,1025,1037,1046,1064,1081,1373,1452,1457,1462,1487,1511,1516,1641,1707,1713,1730,1740,1767,1772,1777,1827,1832,1837,1843,1861,1869,1879,1884,1890,1902,1998,2039,2045,2065,2075,2086,2091,2099,2144,2178,2184,2312],{"type":40,"tag":41,"props":42,"children":43},"element","h1",{"id":4},[44],{"type":45,"value":4},"text",{"type":40,"tag":47,"props":48,"children":49},"p",{},[50,52,58],{"type":45,"value":51},"Help the user get from ",{"type":40,"tag":53,"props":54,"children":55},"strong",{},[56],{"type":45,"value":57},"zero to shipped evaluation prototype",{"type":45,"value":59},". Not 100% correctness on the first\npass — a stable, understandable scorer suite they can run, inspect, and iterate on.",{"type":40,"tag":47,"props":61,"children":62},{},[63,65,70],{"type":45,"value":64},"Your primary job is ",{"type":40,"tag":53,"props":66,"children":67},{},[68],{"type":45,"value":69},"understanding",{"type":45,"value":71},", and the user is the expert on what counts as wrong. Do not\nstart from \"which MLflow scorer should I use?\"",{"type":40,"tag":47,"props":73,"children":74},{},[75],{"type":45,"value":76},"North star:",{"type":40,"tag":78,"props":79,"children":80},"blockquote",{},[81],{"type":40,"tag":47,"props":82,"children":83},{},[84],{"type":45,"value":85},"Understand the app well enough to define a small set of atomic, durable quality criteria, then\nimplement each criterion with the cheapest reliable scorer.",{"type":40,"tag":87,"props":88,"children":90},"h2",{"id":89},"doctrine",[91],{"type":45,"value":92},"Doctrine",{"type":40,"tag":94,"props":95,"children":96},"ol",{},[97,108,118,128,138,148,191,209,219],{"type":40,"tag":98,"props":99,"children":100},"li",{},[101,106],{"type":40,"tag":53,"props":102,"children":103},{},[104],{"type":45,"value":105},"Prototype first.",{"type":45,"value":107}," Ship a useful v1 that runs now. Mark v2 upgrades explicitly.",{"type":40,"tag":98,"props":109,"children":110},{},[111,116],{"type":40,"tag":53,"props":112,"children":113},{},[114],{"type":45,"value":115},"Small suite.",{"type":45,"value":117}," 3-5 high-signal scorers, hard cap. Adding one means dropping a weaker one.",{"type":40,"tag":98,"props":119,"children":120},{},[121,126],{"type":40,"tag":53,"props":122,"children":123},{},[124],{"type":45,"value":125},"One scorer, one criterion.",{"type":45,"value":127}," If a criterion contains \"and\", split it. No \"overall quality\" judge.",{"type":40,"tag":98,"props":129,"children":130},{},[131,136],{"type":40,"tag":53,"props":132,"children":133},{},[134],{"type":45,"value":135},"Cheapest reliable implementation.",{"type":45,"value":137}," Code\u002Frules beat built-ins beat hand-written LLM judges.\nSee Phase 4 — the rule most often skipped, and skipping it costs real money at scale.\nNever hand-roll a judge for something MLflow already ships.",{"type":40,"tag":98,"props":139,"children":140},{},[141,146],{"type":40,"tag":53,"props":142,"children":143},{},[144],{"type":45,"value":145},"Cover outcomes, not just rules.",{"type":45,"value":147}," A suite of only policy checks tells you the agent behaved\nwhile saying nothing about whether users succeeded. Every suite needs at least one check on\nwhether the user got what they came for. See Phase 3.",{"type":40,"tag":98,"props":149,"children":150},{},[151,156,158,165,167,173,175,181,183,189],{"type":40,"tag":53,"props":152,"children":153},{},[154],{"type":45,"value":155},"Binary outputs.",{"type":45,"value":157}," Prefer ",{"type":40,"tag":159,"props":160,"children":162},"code",{"className":161},[],[163],{"type":45,"value":164},"bool",{"type":45,"value":166},", or ",{"type":40,"tag":159,"props":168,"children":170},{"className":169},[],[171],{"type":45,"value":172},"\"yes\"",{"type":45,"value":174},"\u002F",{"type":40,"tag":159,"props":176,"children":178},{"className":177},[],[179],{"type":45,"value":180},"\"no\"",{"type":45,"value":182}," for LLM judges — other strings are silently\ndropped from metrics (Phase 4). Avoid ",{"type":40,"tag":159,"props":184,"children":186},{"className":185},[],[187],{"type":45,"value":188},"0.0-1.0",{"type":45,"value":190}," without a calibration story; pass rates debug\nbetter than averaged floats.",{"type":40,"tag":98,"props":192,"children":193},{},[194,199,201,207],{"type":40,"tag":53,"props":195,"children":196},{},[197],{"type":45,"value":198},"Traces are optional.",{"type":45,"value":200}," Inspect them if they exist; otherwise use dataset + ",{"type":40,"tag":159,"props":202,"children":204},{"className":203},[],[205],{"type":45,"value":206},"predict_fn",{"type":45,"value":208},". Never\nblock the user on tracing unless the criterion truly needs execution internals.",{"type":40,"tag":98,"props":210,"children":211},{},[212,217],{"type":40,"tag":53,"props":213,"children":214},{},[215],{"type":45,"value":216},"Align later.",{"type":45,"value":218}," Treat every first LLM judge as a draft. Remind the user to align it once traces\nand human labels arrive.",{"type":40,"tag":98,"props":220,"children":221},{},[222,227],{"type":40,"tag":53,"props":223,"children":224},{},[225],{"type":45,"value":226},"Talk before code.",{"type":45,"value":228}," The conversation is where right and wrong get settled. A large code block\nends it — users stop reading and start implementing. Small snippets mid-conversation are fine;\nthe full implementation goes in your final message, after the user confirms the criteria.",{"type":40,"tag":87,"props":230,"children":232},{"id":231},"source-of-truth-introspect-dont-memorize",[233],{"type":45,"value":234},"Source of truth: introspect, don't memorize",{"type":40,"tag":47,"props":236,"children":237},{},[238,240,246],{"type":45,"value":239},"MLflow's scorer surface changes. Before recommending scorers, read the installed surface. Drop the\n",{"type":40,"tag":159,"props":241,"children":243},{"className":242},[],[244],{"type":45,"value":245},"uv run",{"type":45,"value":247}," prefix if the project is not uv-managed:",{"type":40,"tag":249,"props":250,"children":255},"pre",{"className":251,"code":252,"language":253,"meta":254,"style":254},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","uv run python -c \"\nfrom mlflow.genai.scorers import get_all_scorers\nimport mlflow.genai.scorers as s\nfor sc in get_all_scorers():\n    req = getattr(sc, 'required_columns', set())\n    print(f'{type(sc).__name__:32} requires={sorted(req)}  session={getattr(sc, \\\"is_session_level_scorer\\\", False)}')\nprint(sorted(n for n in dir(s) if n[0].isupper()))\n\"\n","bash","",[256],{"type":40,"tag":159,"props":257,"children":258},{"__ignoreMap":254},[259,293,302,311,320,329,358,367],{"type":40,"tag":260,"props":261,"children":264},"span",{"class":262,"line":263},"line",1,[265,271,277,282,287],{"type":40,"tag":260,"props":266,"children":268},{"style":267},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[269],{"type":45,"value":270},"uv",{"type":40,"tag":260,"props":272,"children":274},{"style":273},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[275],{"type":45,"value":276}," run",{"type":40,"tag":260,"props":278,"children":279},{"style":273},[280],{"type":45,"value":281}," python",{"type":40,"tag":260,"props":283,"children":284},{"style":273},[285],{"type":45,"value":286}," -c",{"type":40,"tag":260,"props":288,"children":290},{"style":289},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[291],{"type":45,"value":292}," \"\n",{"type":40,"tag":260,"props":294,"children":296},{"class":262,"line":295},2,[297],{"type":40,"tag":260,"props":298,"children":299},{"style":273},[300],{"type":45,"value":301},"from mlflow.genai.scorers import get_all_scorers\n",{"type":40,"tag":260,"props":303,"children":305},{"class":262,"line":304},3,[306],{"type":40,"tag":260,"props":307,"children":308},{"style":273},[309],{"type":45,"value":310},"import mlflow.genai.scorers as s\n",{"type":40,"tag":260,"props":312,"children":314},{"class":262,"line":313},4,[315],{"type":40,"tag":260,"props":316,"children":317},{"style":273},[318],{"type":45,"value":319},"for sc in get_all_scorers():\n",{"type":40,"tag":260,"props":321,"children":323},{"class":262,"line":322},5,[324],{"type":40,"tag":260,"props":325,"children":326},{"style":273},[327],{"type":45,"value":328},"    req = getattr(sc, 'required_columns', set())\n",{"type":40,"tag":260,"props":330,"children":332},{"class":262,"line":331},6,[333,338,344,349,353],{"type":40,"tag":260,"props":334,"children":335},{"style":273},[336],{"type":45,"value":337},"    print(f'{type(sc).__name__:32} requires={sorted(req)}  session={getattr(sc, ",{"type":40,"tag":260,"props":339,"children":341},{"style":340},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[342],{"type":45,"value":343},"\\\"",{"type":40,"tag":260,"props":345,"children":346},{"style":273},[347],{"type":45,"value":348},"is_session_level_scorer",{"type":40,"tag":260,"props":350,"children":351},{"style":340},[352],{"type":45,"value":343},{"type":40,"tag":260,"props":354,"children":355},{"style":273},[356],{"type":45,"value":357},", False)}')\n",{"type":40,"tag":260,"props":359,"children":361},{"class":262,"line":360},7,[362],{"type":40,"tag":260,"props":363,"children":364},{"style":273},[365],{"type":45,"value":366},"print(sorted(n for n in dir(s) if n[0].isupper()))\n",{"type":40,"tag":260,"props":368,"children":370},{"class":262,"line":369},8,[371],{"type":40,"tag":260,"props":372,"children":373},{"style":289},[374],{"type":45,"value":375},"\"\n",{"type":40,"tag":377,"props":378,"children":379},"ul",{},[380,414,425],{"type":40,"tag":98,"props":381,"children":382},{},[383,389,391,397,399,405,406,412],{"type":40,"tag":159,"props":384,"children":386},{"className":385},[],[387],{"type":45,"value":388},"get_all_scorers()",{"type":45,"value":390}," returns built-ins instantiable with defaults. Constructor-arg scorers\n(",{"type":40,"tag":159,"props":392,"children":394},{"className":393},[],[395],{"type":45,"value":396},"Guidelines",{"type":45,"value":398},", ",{"type":40,"tag":159,"props":400,"children":402},{"className":401},[],[403],{"type":45,"value":404},"RegexMatch",{"type":45,"value":398},{"type":40,"tag":159,"props":407,"children":409},{"className":408},[],[410],{"type":45,"value":411},"ResponseLength",{"type":45,"value":413},") may only appear in exported names.",{"type":40,"tag":98,"props":415,"children":416},{},[417,423],{"type":40,"tag":159,"props":418,"children":420},{"className":419},[],[421],{"type":45,"value":422},".required_columns",{"type":45,"value":424}," is the exact data contract. Use it; never guess.",{"type":40,"tag":98,"props":426,"children":427},{},[428,430,436,438,444],{"type":45,"value":429},"LLM judges: ",{"type":40,"tag":159,"props":431,"children":433},{"className":432},[],[434],{"type":45,"value":435},"from mlflow.genai.judges import make_judge",{"type":45,"value":437},". Code scorers:\n",{"type":40,"tag":159,"props":439,"children":441},{"className":440},[],[442],{"type":45,"value":443},"from mlflow.genai.scorers import scorer",{"type":45,"value":445},".",{"type":40,"tag":47,"props":447,"children":448},{},[449],{"type":45,"value":450},"If MLflow is not importable, say so, continue from first principles, and flag that names and\nrequired columns need verification.",{"type":40,"tag":87,"props":452,"children":454},{"id":453},"user-facing-language",[455],{"type":45,"value":456},"User-facing language",{"type":40,"tag":47,"props":458,"children":459},{},[460,462,467,468,473,474,479,480,485,487,492,493,499,500,506,507,513],{"type":45,"value":461},"Product language first (",{"type":40,"tag":53,"props":463,"children":464},{},[465],{"type":45,"value":466},"quality check",{"type":45,"value":398},{"type":40,"tag":53,"props":469,"children":470},{},[471],{"type":45,"value":472},"judge",{"type":45,"value":398},{"type":40,"tag":53,"props":475,"children":476},{},[477],{"type":45,"value":478},"code check",{"type":45,"value":398},{"type":40,"tag":53,"props":481,"children":482},{},[483],{"type":45,"value":484},"built-in check",{"type":45,"value":486},"), API\nlanguage second (",{"type":40,"tag":53,"props":488,"children":489},{},[490],{"type":45,"value":491},"scorer",{"type":45,"value":398},{"type":40,"tag":159,"props":494,"children":496},{"className":495},[],[497],{"type":45,"value":498},"make_judge",{"type":45,"value":398},{"type":40,"tag":159,"props":501,"children":503},{"className":502},[],[504],{"type":45,"value":505},"@scorer",{"type":45,"value":398},{"type":40,"tag":159,"props":508,"children":510},{"className":509},[],[511],{"type":45,"value":512},"required_columns",{"type":45,"value":514},"). State each recommendation as:",{"type":40,"tag":249,"props":516,"children":520},{"className":517,"code":519,"language":45,"meta":254},[518],"language-text","Quality check: \u003Cplain-English name>     Layer: \u003Crule | outcome>\nCriterion: \u003Cone sentence, one thing only>\nImplementation: \u003Ccode check | built-in | LLM judge>\nWhy not cheaper: \u003Cfor LLM judges — the code check AND built-in you rejected, and why>\nOutput: \u003Cbool, or \"yes\"\u002F\"no\">          Data needed: \u003Cinputs\u002Foutputs\u002Fexpectations\u002Ftrace\u002Fsession>\n",[521],{"type":40,"tag":159,"props":522,"children":523},{"__ignoreMap":254},[524],{"type":45,"value":519},{"type":40,"tag":87,"props":526,"children":528},{"id":527},"the-workflow",[529],{"type":45,"value":530},"The workflow",{"type":40,"tag":532,"props":533,"children":535},"h3",{"id":534},"phase-0-orient",[536],{"type":45,"value":537},"Phase 0: Orient",{"type":40,"tag":47,"props":539,"children":540},{},[541],{"type":45,"value":542},"State the working agreement plainly:",{"type":40,"tag":78,"props":544,"children":545},{},[546],{"type":40,"tag":47,"props":547,"children":548},{},[549],{"type":45,"value":550},"We'll build a small v1 scorer suite that runs now, catches repeated failures, and gives you a\nstable base to iterate. It will not be perfect yet.",{"type":40,"tag":532,"props":552,"children":554},{"id":553},"phase-1-elicit-the-users-notion-of-right-and-wrong",[555],{"type":45,"value":556},"Phase 1: Elicit the user's notion of right and wrong",{"type":40,"tag":47,"props":558,"children":559},{},[560,562,567],{"type":45,"value":561},"Your job is ",{"type":40,"tag":53,"props":563,"children":564},{},[565],{"type":45,"value":566},"criteria, not architecture",{"type":45,"value":568}," — what the user thinks good and bad look like, not their\nproduct spec.",{"type":40,"tag":47,"props":570,"children":571},{},[572,577],{"type":40,"tag":53,"props":573,"children":574},{},[575],{"type":45,"value":576},"Ask at most 3 questions before showing a draft.",{"type":45,"value":578}," Users correct a wrong draft far more easily than\nthey answer an interview.",{"type":40,"tag":47,"props":580,"children":581},{},[582],{"type":45,"value":583},"Open with these, one at a time:",{"type":40,"tag":94,"props":585,"children":586},{},[587,592,597],{"type":40,"tag":98,"props":588,"children":589},{},[590],{"type":45,"value":591},"What does the app do?",{"type":40,"tag":98,"props":593,"children":594},{},[595],{"type":45,"value":596},"What is the worst failure you could ship?",{"type":40,"tag":98,"props":598,"children":599},{},[600],{"type":45,"value":601},"What should it do when it is unsure or lacks information?",{"type":40,"tag":47,"props":603,"children":604},{},[605,607,612],{"type":45,"value":606},"Then stop asking and draft. Translate vague answers like \"good responses\" into named observable\ncriteria ",{"type":40,"tag":53,"props":608,"children":609},{},[610],{"type":45,"value":611},"yourself",{"type":45,"value":613}," and show them for correction — do not bounce the work back as another question.",{"type":40,"tag":47,"props":615,"children":616},{},[617],{"type":45,"value":618},"Out of scope; these are the user's job, not the judge's:",{"type":40,"tag":377,"props":620,"children":621},{},[622,627,632],{"type":40,"tag":98,"props":623,"children":624},{},[625],{"type":45,"value":626},"Which fields are required at which step of their business logic.",{"type":40,"tag":98,"props":628,"children":629},{},[630],{"type":45,"value":631},"How the app decides between branches or states.",{"type":40,"tag":98,"props":633,"children":634},{},[635],{"type":45,"value":636},"Internal schemas, gates, thresholds, or slot taxonomies.",{"type":40,"tag":47,"props":638,"children":639},{},[640],{"type":45,"value":641},"If you catch yourself asking a second consecutive question about app internals, stop and draft.",{"type":40,"tag":47,"props":643,"children":644},{},[645,650],{"type":40,"tag":53,"props":646,"children":647},{},[648],{"type":45,"value":649},"Real failure to avoid:",{"type":45,"value":651}," eliciting \"it shouldn't book if information is incomplete\", then asking\nfour follow-ups about which fields count as complete. The first sentence was already enough.",{"type":40,"tag":532,"props":653,"children":655},{"id":654},"phase-2-check-what-is-observable",[656],{"type":45,"value":657},"Phase 2: Check what is observable",{"type":40,"tag":47,"props":659,"children":660},{},[661],{"type":45,"value":662},"Once the user has reacted to a draft, establish what evidence a judge can actually see. A criterion\nis only real if it is checkable. One focused pass, not an interview.",{"type":40,"tag":47,"props":664,"children":665},{},[666,668,674,675,681,682,688],{"type":45,"value":667},"If traces exist, inspect one or two: confirm ",{"type":40,"tag":159,"props":669,"children":671},{"className":670},[],[672],{"type":45,"value":673},"inputs",{"type":45,"value":398},{"type":40,"tag":159,"props":676,"children":678},{"className":677},[],[679],{"type":45,"value":680},"outputs",{"type":45,"value":398},{"type":40,"tag":159,"props":683,"children":685},{"className":684},[],[686],{"type":45,"value":687},"expectations",{"type":45,"value":689},", span types, tool\ncalls, and whether the task is single-turn or session-level.",{"type":40,"tag":47,"props":691,"children":692},{},[693,698,700,706,707,713,715,721,723,729,731,736,738,744,746,752],{"type":40,"tag":53,"props":694,"children":695},{},[696],{"type":45,"value":697},"The RETRIEVER-vs-TOOL trap:",{"type":45,"value":699}," ",{"type":40,"tag":159,"props":701,"children":703},{"className":702},[],[704],{"type":45,"value":705},"RetrievalGroundedness",{"type":45,"value":398},{"type":40,"tag":159,"props":708,"children":710},{"className":709},[],[711],{"type":45,"value":712},"RetrievalRelevance",{"type":45,"value":714},", and\n",{"type":40,"tag":159,"props":716,"children":718},{"className":717},[],[719],{"type":45,"value":720},"RetrievalSufficiency",{"type":45,"value":722}," require ",{"type":40,"tag":159,"props":724,"children":726},{"className":725},[],[727],{"type":45,"value":728},"RETRIEVER",{"type":45,"value":730}," spans and ",{"type":40,"tag":53,"props":732,"children":733},{},[734],{"type":45,"value":735},"hard-raise",{"type":45,"value":737}," otherwise. A ",{"type":40,"tag":159,"props":739,"children":741},{"className":740},[],[742],{"type":45,"value":743},"web_search",{"type":45,"value":745}," or DB\nlookup instrumented as ",{"type":40,"tag":159,"props":747,"children":749},{"className":748},[],[750],{"type":45,"value":751},"TOOL",{"type":45,"value":753}," does not qualify.",{"type":40,"tag":47,"props":755,"children":756},{},[757,759,771,773,778,780,786,788,794,796,801,803,808],{"type":45,"value":758},"For \"grounded in tool output\", ",{"type":40,"tag":53,"props":760,"children":761},{},[762,764,770],{"type":45,"value":763},"wrap a judge in a code scorer instead of passing ",{"type":40,"tag":159,"props":765,"children":767},{"className":766},[],[768],{"type":45,"value":769},"{{ trace }}",{"type":45,"value":445},{"type":45,"value":772}," A\n",{"type":40,"tag":159,"props":774,"children":776},{"className":775},[],[777],{"type":45,"value":505},{"type":45,"value":779}," gets the ",{"type":40,"tag":159,"props":781,"children":783},{"className":782},[],[784],{"type":45,"value":785},"trace",{"type":45,"value":787},", so use ",{"type":40,"tag":159,"props":789,"children":791},{"className":790},[],[792],{"type":45,"value":793},"trace.search_spans()",{"type":45,"value":795}," to pull the evidence span and pass only it\nplus the claim to the judge. ",{"type":40,"tag":159,"props":797,"children":799},{"className":798},[],[800],{"type":45,"value":769},{"type":45,"value":802}," serializes every span and makes the model find the right\none — costlier, and span selection becomes nondeterministic when there are several. Use ",{"type":40,"tag":159,"props":804,"children":806},{"className":805},[],[807],{"type":45,"value":769},{"type":45,"value":809},"\nonly when the criterion is about the whole trace (tool sequencing, retries), with no single span to\nextract.",{"type":40,"tag":47,"props":811,"children":812},{},[813,815,820,822,827,828,833,835,841],{"type":45,"value":814},"If traces do not exist, use ",{"type":40,"tag":159,"props":816,"children":818},{"className":817},[],[819],{"type":45,"value":673},{"type":45,"value":821}," + optional ",{"type":40,"tag":159,"props":823,"children":825},{"className":824},[],[826],{"type":45,"value":680},{"type":45,"value":174},{"type":40,"tag":159,"props":829,"children":831},{"className":830},[],[832],{"type":45,"value":687},{"type":45,"value":834}," with\n",{"type":40,"tag":159,"props":836,"children":838},{"className":837},[],[839],{"type":45,"value":840},"mlflow.genai.evaluate(data=..., predict_fn=..., scorers=[...])",{"type":45,"value":842},", and mark trace-only upgrades as\nfuture work.",{"type":40,"tag":532,"props":844,"children":846},{"id":845},"phase-3-generalize-into-atomic-criteria-across-both-layers",[847],{"type":45,"value":848},"Phase 3: Generalize into atomic criteria — across both layers",{"type":40,"tag":47,"props":850,"children":851},{},[852,854,860,861,867,868,874],{"type":45,"value":853},"Cluster examples into repeated failure modes. Keep criteria likely to recur; park one-off oddities.\nName them like ",{"type":40,"tag":159,"props":855,"children":857},{"className":856},[],[858],{"type":45,"value":859},"answers_user_question",{"type":45,"value":398},{"type":40,"tag":159,"props":862,"children":864},{"className":863},[],[865],{"type":45,"value":866},"does_not_invent_facts",{"type":45,"value":398},{"type":40,"tag":159,"props":869,"children":871},{"className":870},[],[872],{"type":45,"value":873},"refuses_out_of_scope_requests",{"type":45,"value":875}," —\none behavior each.",{"type":40,"tag":47,"props":877,"children":878},{},[879],{"type":40,"tag":53,"props":880,"children":881},{},[882],{"type":45,"value":883},"Cover two layers. Suites that only do the first layer are the most common failure of this skill.",{"type":40,"tag":94,"props":885,"children":886},{},[887,897],{"type":40,"tag":98,"props":888,"children":889},{},[890,895],{"type":40,"tag":53,"props":891,"children":892},{},[893],{"type":45,"value":894},"Rule layer — did the agent misbehave on this turn?",{"type":45,"value":896}," Policy violations, wrong facts, missed\nescalations, forbidden claims. Easy to name, easy to check, and where users' stated fears live.",{"type":40,"tag":98,"props":898,"children":899},{},[900,905],{"type":40,"tag":53,"props":901,"children":902},{},[903],{"type":45,"value":904},"Outcome layer — did the user get what they came for?",{"type":45,"value":906}," Task completion, abandonment,\nfrustration, having to repeat themselves. Harder to name, and the reason the product exists.",{"type":40,"tag":47,"props":908,"children":909},{},[910,912],{"type":45,"value":911},"An agent can pass every rule check and still be useless. Ask directly: ",{"type":40,"tag":913,"props":914,"children":915},"em",{},[916],{"type":45,"value":917},"\"What does success look\nlike for the user — what did they come here to do, and how would we know they did it?\"",{"type":40,"tag":47,"props":919,"children":920},{},[921,923,928],{"type":45,"value":922},"Outcome checks are often ",{"type":40,"tag":53,"props":924,"children":925},{},[926],{"type":45,"value":927},"cheap",{"type":45,"value":929},", not expensive — do not assume they need an LLM judge:",{"type":40,"tag":377,"props":931,"children":932},{},[933,944,978],{"type":40,"tag":98,"props":934,"children":935},{},[936,942],{"type":40,"tag":159,"props":937,"children":939},{"className":938},[],[940],{"type":45,"value":941},"task_completed",{"type":45,"value":943}," — did the trace contain the goal tool call (booking, application, order)? Code.",{"type":40,"tag":98,"props":945,"children":946},{},[947,949,955,957,963,964,970,972,977],{"type":45,"value":948},"Session-level built-ins with no data requirements: ",{"type":40,"tag":159,"props":950,"children":952},{"className":951},[],[953],{"type":45,"value":954},"UserFrustration",{"type":45,"value":956},",\n",{"type":40,"tag":159,"props":958,"children":960},{"className":959},[],[961],{"type":45,"value":962},"ConversationCompleteness",{"type":45,"value":398},{"type":40,"tag":159,"props":965,"children":967},{"className":966},[],[968],{"type":45,"value":969},"KnowledgeRetention",{"type":45,"value":971},". Verify with ",{"type":40,"tag":159,"props":973,"children":975},{"className":974},[],[976],{"type":45,"value":422},{"type":45,"value":445},{"type":40,"tag":98,"props":979,"children":980},{},[981,987],{"type":40,"tag":159,"props":982,"children":984},{"className":983},[],[985],{"type":45,"value":986},"had_to_repeat",{"type":45,"value":988}," — did the user restate the same request across turns? Code over the conversation.",{"type":40,"tag":47,"props":990,"children":991},{},[992,997,999,1005,1007,1012],{"type":40,"tag":53,"props":993,"children":994},{},[995],{"type":45,"value":996},"Do not substitute a checkable proxy for the outcome that matters.",{"type":45,"value":998}," A check like\n",{"type":40,"tag":159,"props":1000,"children":1002},{"className":1001},[],[1003],{"type":45,"value":1004},"promotion_disclosed",{"type":45,"value":1006}," (\"was a promo flag mentioned?\") looks rigorous but measures text presence, not\nwhether the renter got the best deal or felt well-served. Proxies are fine as ",{"type":40,"tag":913,"props":1008,"children":1009},{},[1010],{"type":45,"value":1011},"rule",{"type":45,"value":1013}," checks; they do\nnot discharge the outcome layer. If the user's real goal is fuzzy, say so and pair a cheap proxy with\none honest outcome check rather than pretending the proxy covers it.",{"type":40,"tag":47,"props":1015,"children":1016},{},[1017],{"type":45,"value":1018},"Keep outcome checks atomic too. \"Meets customer expectation\" bundles found-what-they-wanted,\nwasn't-misled, and wasn't-frustrated; when it fails you learn nothing. Split it.",{"type":40,"tag":532,"props":1020,"children":1022},{"id":1021},"phase-4-route-each-criterion-to-the-cheapest-reliable-implementation",[1023],{"type":45,"value":1024},"Phase 4: Route each criterion to the cheapest reliable implementation",{"type":40,"tag":47,"props":1026,"children":1027},{},[1028,1030,1035],{"type":45,"value":1029},"Do this ",{"type":40,"tag":53,"props":1031,"children":1032},{},[1033],{"type":45,"value":1034},"before",{"type":45,"value":1036}," asking the user to confirm the suite, so they are approving real cost.",{"type":40,"tag":249,"props":1038,"children":1041},{"className":1039,"code":1040,"language":45,"meta":254},[518],"Can code check it deterministically (or filter most cases)?\n  yes -> code check\n  no  -> Is there a built-in for this standard concept, and do we have its required data?\n          yes -> built-in\n          no  -> LLM judge\n",[1042],{"type":40,"tag":159,"props":1043,"children":1044},{"__ignoreMap":254},[1045],{"type":45,"value":1040},{"type":40,"tag":47,"props":1047,"children":1048},{},[1049,1062],{"type":40,"tag":53,"props":1050,"children":1051},{},[1052,1054,1060],{"type":45,"value":1053},"Checkpoint — every hand-written LLM judge needs a ",{"type":40,"tag":159,"props":1055,"children":1057},{"className":1056},[],[1058],{"type":45,"value":1059},"why not cheaper",{"type":45,"value":1061}," cell naming both the code\ncheck and the built-in you rejected.",{"type":45,"value":1063}," Not prose elsewhere in the message: the cell in the Phase 6\ntable. If you cannot fill it, the check is miscategorized. \"No built-in covers this\" counts only if\nyou actually looked.",{"type":40,"tag":47,"props":1065,"children":1066},{},[1067,1072,1074,1079],{"type":40,"tag":53,"props":1068,"children":1069},{},[1070],{"type":45,"value":1071},"Check the catalog before hand-rolling.",{"type":45,"value":1073}," Run the introspection command above when MLflow is\nimportable; when it is not, these ship today — confirm names and ",{"type":40,"tag":159,"props":1075,"children":1077},{"className":1076},[],[1078],{"type":45,"value":512},{"type":45,"value":1080}," before relying\non them, but do not act as though the catalog is empty:",{"type":40,"tag":1082,"props":1083,"children":1084},"table",{},[1085,1109],{"type":40,"tag":1086,"props":1087,"children":1088},"thead",{},[1089],{"type":40,"tag":1090,"props":1091,"children":1092},"tr",{},[1093,1099,1104],{"type":40,"tag":1094,"props":1095,"children":1096},"th",{},[1097],{"type":45,"value":1098},"Built-in",{"type":40,"tag":1094,"props":1100,"children":1101},{},[1102],{"type":45,"value":1103},"Requires",{"type":40,"tag":1094,"props":1105,"children":1106},{},[1107],{"type":45,"value":1108},"Use for",{"type":40,"tag":1110,"props":1111,"children":1112},"tbody",{},[1113,1136,1157,1178,1208,1236,1271,1299,1326],{"type":40,"tag":1090,"props":1114,"children":1115},{},[1116,1126,1131],{"type":40,"tag":1117,"props":1118,"children":1119},"td",{},[1120],{"type":40,"tag":159,"props":1121,"children":1123},{"className":1122},[],[1124],{"type":45,"value":1125},"Guidelines(guidelines=...)",{"type":40,"tag":1117,"props":1127,"children":1128},{},[1129],{"type":45,"value":1130},"inputs, outputs",{"type":40,"tag":1117,"props":1132,"children":1133},{},[1134],{"type":45,"value":1135},"\"does the response follow this stated policy\" — the most under-used built-in",{"type":40,"tag":1090,"props":1137,"children":1138},{},[1139,1148,1152],{"type":40,"tag":1117,"props":1140,"children":1141},{},[1142],{"type":40,"tag":159,"props":1143,"children":1145},{"className":1144},[],[1146],{"type":45,"value":1147},"Correctness",{"type":40,"tag":1117,"props":1149,"children":1150},{},[1151],{"type":45,"value":1130},{"type":40,"tag":1117,"props":1153,"children":1154},{},[1155],{"type":45,"value":1156},"answer matches expected",{"type":40,"tag":1090,"props":1158,"children":1159},{},[1160,1169,1173],{"type":40,"tag":1117,"props":1161,"children":1162},{},[1163],{"type":40,"tag":159,"props":1164,"children":1166},{"className":1165},[],[1167],{"type":45,"value":1168},"RelevanceToQuery",{"type":40,"tag":1117,"props":1170,"children":1171},{},[1172],{"type":45,"value":1130},{"type":40,"tag":1117,"props":1174,"children":1175},{},[1176],{"type":45,"value":1177},"response addresses the question",{"type":40,"tag":1090,"props":1179,"children":1180},{},[1181,1198,1203],{"type":40,"tag":1117,"props":1182,"children":1183},{},[1184,1190,1192],{"type":40,"tag":159,"props":1185,"children":1187},{"className":1186},[],[1188],{"type":45,"value":1189},"Safety",{"type":45,"value":1191}," \u002F ",{"type":40,"tag":159,"props":1193,"children":1195},{"className":1194},[],[1196],{"type":45,"value":1197},"PIIDetection",{"type":40,"tag":1117,"props":1199,"children":1200},{},[1201],{"type":45,"value":1202},"inputs, outputs \u002F outputs",{"type":40,"tag":1117,"props":1204,"children":1205},{},[1206],{"type":45,"value":1207},"harmful content; PII leakage",{"type":40,"tag":1090,"props":1209,"children":1210},{},[1211,1227,1231],{"type":40,"tag":1117,"props":1212,"children":1213},{},[1214,1220,1221],{"type":40,"tag":159,"props":1215,"children":1217},{"className":1216},[],[1218],{"type":45,"value":1219},"RegexMatch(pattern=...)",{"type":45,"value":1191},{"type":40,"tag":159,"props":1222,"children":1224},{"className":1223},[],[1225],{"type":45,"value":1226},"ResponseLength(...)",{"type":40,"tag":1117,"props":1228,"children":1229},{},[1230],{"type":45,"value":680},{"type":40,"tag":1117,"props":1232,"children":1233},{},[1234],{"type":45,"value":1235},"pattern and length rules",{"type":40,"tag":1090,"props":1237,"children":1238},{},[1239,1262,1266],{"type":40,"tag":1117,"props":1240,"children":1241},{},[1242,1248,1249,1255,1256],{"type":40,"tag":159,"props":1243,"children":1245},{"className":1244},[],[1246],{"type":45,"value":1247},"Completeness",{"type":45,"value":1191},{"type":40,"tag":159,"props":1250,"children":1252},{"className":1251},[],[1253],{"type":45,"value":1254},"Fluency",{"type":45,"value":1191},{"type":40,"tag":159,"props":1257,"children":1259},{"className":1258},[],[1260],{"type":45,"value":1261},"Summarization",{"type":40,"tag":1117,"props":1263,"children":1264},{},[1265],{"type":45,"value":1130},{"type":40,"tag":1117,"props":1267,"children":1268},{},[1269],{"type":45,"value":1270},"coverage; readability; summary quality",{"type":40,"tag":1090,"props":1272,"children":1273},{},[1274,1290,1294],{"type":40,"tag":1117,"props":1275,"children":1276},{},[1277,1283,1284],{"type":40,"tag":159,"props":1278,"children":1280},{"className":1279},[],[1281],{"type":45,"value":1282},"ToolCallCorrectness",{"type":45,"value":1191},{"type":40,"tag":159,"props":1285,"children":1287},{"className":1286},[],[1288],{"type":45,"value":1289},"ToolCallEfficiency",{"type":40,"tag":1117,"props":1291,"children":1292},{},[1293],{"type":45,"value":785},{"type":40,"tag":1117,"props":1295,"children":1296},{},[1297],{"type":45,"value":1298},"right tools, no redundant calls",{"type":40,"tag":1090,"props":1300,"children":1301},{},[1302,1311,1316],{"type":40,"tag":1117,"props":1303,"children":1304},{},[1305],{"type":40,"tag":159,"props":1306,"children":1308},{"className":1307},[],[1309],{"type":45,"value":1310},"Retrieval{Groundedness,Relevance,Sufficiency}",{"type":40,"tag":1117,"props":1312,"children":1313},{},[1314],{"type":45,"value":1315},"inputs, trace",{"type":40,"tag":1117,"props":1317,"children":1318},{},[1319,1324],{"type":40,"tag":53,"props":1320,"children":1321},{},[1322],{"type":45,"value":1323},"RETRIEVER spans only",{"type":45,"value":1325}," — see Phase 2",{"type":40,"tag":1090,"props":1327,"children":1328},{},[1329,1363,1368],{"type":40,"tag":1117,"props":1330,"children":1331},{},[1332,1337,1338,1343,1344,1349,1350,1356,1357],{"type":40,"tag":159,"props":1333,"children":1335},{"className":1334},[],[1336],{"type":45,"value":954},{"type":45,"value":398},{"type":40,"tag":159,"props":1339,"children":1341},{"className":1340},[],[1342],{"type":45,"value":962},{"type":45,"value":398},{"type":40,"tag":159,"props":1345,"children":1347},{"className":1346},[],[1348],{"type":45,"value":969},{"type":45,"value":398},{"type":40,"tag":159,"props":1351,"children":1353},{"className":1352},[],[1354],{"type":45,"value":1355},"ConversationalSafety",{"type":45,"value":398},{"type":40,"tag":159,"props":1358,"children":1360},{"className":1359},[],[1361],{"type":45,"value":1362},"ConversationalRoleAdherence",{"type":40,"tag":1117,"props":1364,"children":1365},{},[1366],{"type":45,"value":1367},"none (session-level)",{"type":40,"tag":1117,"props":1369,"children":1370},{},[1371],{"type":45,"value":1372},"multi-turn outcome and conversation quality",{"type":40,"tag":47,"props":1374,"children":1375},{},[1376,1381,1383,1388,1390,1395,1397,1403,1405,1411,1413,1418,1420,1425,1426,1431,1432,1437,1439,1450],{"type":40,"tag":53,"props":1377,"children":1378},{},[1379],{"type":45,"value":1380},"A built-in is a better starting point than a blank prompt — not a substitute for your standard.",{"type":45,"value":1382},"\nBuilt-ins are LLM judges carrying MLflow's generic instructions, so \"built-in\" does not imply correct\nfor your product; prefer them because the scaffolding and data contract are already tested. Two\nconsequences: ",{"type":40,"tag":159,"props":1384,"children":1386},{"className":1385},[],[1387],{"type":45,"value":396},{"type":45,"value":1389}," always applies, since it takes ",{"type":40,"tag":913,"props":1391,"children":1392},{},[1393],{"type":45,"value":1394},"your",{"type":45,"value":1396}," policy text. And alignment cuts\nacross the built-in\u002Fbespoke line — ",{"type":40,"tag":159,"props":1398,"children":1400},{"className":1399},[],[1401],{"type":45,"value":1402},"align()",{"type":45,"value":1404}," is on ",{"type":40,"tag":159,"props":1406,"children":1408},{"className":1407},[],[1409],{"type":45,"value":1410},"Judge",{"type":45,"value":1412},", so single-turn built-ins can be aligned to\nyour labels like ",{"type":40,"tag":159,"props":1414,"children":1416},{"className":1415},[],[1417],{"type":45,"value":498},{"type":45,"value":1419},", but session-level ones (",{"type":40,"tag":159,"props":1421,"children":1423},{"className":1422},[],[1424],{"type":45,"value":954},{"type":45,"value":398},{"type":40,"tag":159,"props":1427,"children":1429},{"className":1428},[],[1430],{"type":45,"value":962},{"type":45,"value":956},{"type":40,"tag":159,"props":1433,"children":1435},{"className":1434},[],[1436],{"type":45,"value":969},{"type":45,"value":1438},") ",{"type":40,"tag":53,"props":1440,"children":1441},{},[1442,1444],{"type":45,"value":1443},"raise ",{"type":40,"tag":159,"props":1445,"children":1447},{"className":1446},[],[1448],{"type":45,"value":1449},"NotImplementedError",{"type":45,"value":1451}," and are stuck on MLflow's definition. Hand-write\na criterion that is important and contested, so it can be aligned.",{"type":40,"tag":47,"props":1453,"children":1454},{},[1455],{"type":45,"value":1456},"Skipping built-ins is legitimate when your standard genuinely differs. Skipping them without\nlooking is not.",{"type":40,"tag":47,"props":1458,"children":1459},{},[1460],{"type":45,"value":1461},"Default routes:",{"type":40,"tag":377,"props":1463,"children":1464},{},[1465,1477,1482],{"type":40,"tag":98,"props":1466,"children":1467},{},[1468,1470,1475],{"type":45,"value":1469},"Format, JSON shape, required field, regex\u002Flink, length, exact value, latency, PII pattern,\n",{"type":40,"tag":53,"props":1471,"children":1472},{},[1473],{"type":45,"value":1474},"specific phrases or keywords",{"type":45,"value":1476},": code check.",{"type":40,"tag":98,"props":1478,"children":1479},{},[1480],{"type":45,"value":1481},"Relevance, correctness, safety, groundedness, retrieval quality, tool-call quality, fluency,\nsession-level conversation quality: built-in, if the data contract matches.",{"type":40,"tag":98,"props":1483,"children":1484},{},[1485],{"type":45,"value":1486},"Domain tone, policy adherence, task-specific correctness, paraphrase-matching against a source,\nnuanced refusal quality: LLM judge.",{"type":40,"tag":47,"props":1488,"children":1489},{},[1490,1495,1497,1502,1504,1509],{"type":40,"tag":53,"props":1491,"children":1492},{},[1493],{"type":45,"value":1494},"Prefer a hybrid over a pure judge.",{"type":45,"value":1496}," Most \"semantic\" checks have a cheap deterministic front end:\na keyword or phrase scan catches the bright-line cases at ~zero cost and passes only ambiguous ones\nto a model. Reach for this whenever a criterion mentions specific commitments, categories, or\ntrigger topics — \"did it promise a refund\", \"did the user raise a billing dispute\". Detecting the\n",{"type":40,"tag":913,"props":1498,"children":1499},{},[1500],{"type":45,"value":1501},"trigger",{"type":45,"value":1503}," is usually code; judging the ",{"type":40,"tag":913,"props":1505,"children":1506},{},[1507],{"type":45,"value":1508},"response",{"type":45,"value":1510}," may need a model.",{"type":40,"tag":47,"props":1512,"children":1513},{},[1514],{"type":45,"value":1515},"Push back when the user reaches for an LLM judge to count words or validate JSON, a regex to judge\ntone, one \"overall good\" scorer, or a float without a calibration reason.",{"type":40,"tag":47,"props":1517,"children":1518},{},[1519,1544,1546,1552,1554,1560,1562,1573,1575,1581,1582,1588,1589,1595,1596,1602,1604,1610,1612,1618,1620,1626,1627,1633,1634,1640],{"type":40,"tag":53,"props":1520,"children":1521},{},[1522,1524,1529,1531,1536,1537,1542],{"type":45,"value":1523},"Output types — only ",{"type":40,"tag":159,"props":1525,"children":1527},{"className":1526},[],[1528],{"type":45,"value":164},{"type":45,"value":1530},", numerics, and ",{"type":40,"tag":159,"props":1532,"children":1534},{"className":1533},[],[1535],{"type":45,"value":172},{"type":45,"value":174},{"type":40,"tag":159,"props":1538,"children":1540},{"className":1539},[],[1541],{"type":45,"value":180},{"type":45,"value":1543}," aggregate.",{"type":45,"value":1545}," Anything else is silently\ncast to ",{"type":40,"tag":159,"props":1547,"children":1549},{"className":1548},[],[1550],{"type":45,"value":1551},"None",{"type":45,"value":1553}," by ",{"type":40,"tag":159,"props":1555,"children":1557},{"className":1556},[],[1558],{"type":45,"value":1559},"_cast_assessment_value_to_float",{"type":45,"value":1561}," and ",{"type":40,"tag":53,"props":1563,"children":1564},{},[1565,1567],{"type":45,"value":1566},"dropped from ",{"type":40,"tag":159,"props":1568,"children":1570},{"className":1569},[],[1571],{"type":45,"value":1572},"results.metrics",{"type":45,"value":1574}," with no\nerror: ",{"type":40,"tag":159,"props":1576,"children":1578},{"className":1577},[],[1579],{"type":45,"value":1580},"\"pass\"",{"type":45,"value":398},{"type":40,"tag":159,"props":1583,"children":1585},{"className":1584},[],[1586],{"type":45,"value":1587},"\"fail\"",{"type":45,"value":398},{"type":40,"tag":159,"props":1590,"children":1592},{"className":1591},[],[1593],{"type":45,"value":1594},"\"correct\"",{"type":45,"value":398},{"type":40,"tag":159,"props":1597,"children":1599},{"className":1598},[],[1600],{"type":45,"value":1601},"\"not_applicable\"",{"type":45,"value":1603}," all disappear. Verified against\n",{"type":40,"tag":159,"props":1605,"children":1607},{"className":1606},[],[1608],{"type":45,"value":1609},"mlflow\u002Fgenai\u002Fscorers\u002Faggregation.py",{"type":45,"value":1611},"; ",{"type":40,"tag":159,"props":1613,"children":1615},{"className":1614},[],[1616],{"type":45,"value":1617},"CategoricalRating",{"type":45,"value":1619}," accepts only ",{"type":40,"tag":159,"props":1621,"children":1623},{"className":1622},[],[1624],{"type":45,"value":1625},"yes",{"type":45,"value":174},{"type":40,"tag":159,"props":1628,"children":1630},{"className":1629},[],[1631],{"type":45,"value":1632},"no",{"type":45,"value":174},{"type":40,"tag":159,"props":1635,"children":1637},{"className":1636},[],[1638],{"type":45,"value":1639},"unknown",{"type":45,"value":445},{"type":40,"tag":377,"props":1642,"children":1643},{},[1644,1656,1674,1694],{"type":40,"tag":98,"props":1645,"children":1646},{},[1647,1649,1654],{"type":45,"value":1648},"Prefer ",{"type":40,"tag":159,"props":1650,"children":1652},{"className":1651},[],[1653],{"type":45,"value":164},{"type":45,"value":1655}," for pass\u002Ffail.",{"type":40,"tag":98,"props":1657,"children":1658},{},[1659,1661,1666,1667,1672],{"type":45,"value":1660},"Use ",{"type":40,"tag":159,"props":1662,"children":1664},{"className":1663},[],[1665],{"type":45,"value":172},{"type":45,"value":174},{"type":40,"tag":159,"props":1668,"children":1670},{"className":1669},[],[1671],{"type":45,"value":180},{"type":45,"value":1673}," when an LLM judge must return a string.",{"type":40,"tag":98,"props":1675,"children":1676},{},[1677,1679,1685,1687,1692],{"type":45,"value":1678},"Represent \"criterion does not apply\" as ",{"type":40,"tag":159,"props":1680,"children":1682},{"className":1681},[],[1683],{"type":45,"value":1684},"Feedback(value=True, rationale=\"not applicable: ...\")",{"type":45,"value":1686},",\nor skip the row — not as a ",{"type":40,"tag":159,"props":1688,"children":1690},{"className":1689},[],[1691],{"type":45,"value":1601},{"type":45,"value":1693}," string, which vanishes from the metrics.",{"type":40,"tag":98,"props":1695,"children":1696},{},[1697,1699,1705],{"type":45,"value":1698},"Put extra detail in ",{"type":40,"tag":159,"props":1700,"children":1702},{"className":1701},[],[1703],{"type":45,"value":1704},"rationale",{"type":45,"value":1706},", not in the value.",{"type":40,"tag":532,"props":1708,"children":1710},{"id":1709},"phase-5-sharpen-each-criterion-with-the-domain-expert",[1711],{"type":45,"value":1712},"Phase 5: Sharpen each criterion with the domain expert",{"type":40,"tag":47,"props":1714,"children":1715},{},[1716,1721,1723,1728],{"type":40,"tag":53,"props":1717,"children":1718},{},[1719],{"type":45,"value":1720},"This is the highest-value phase — spend most of the conversation here.",{"type":45,"value":1722}," The user is the expert on\nwhat counts as wrong. Your job is to find each criterion's ",{"type":40,"tag":53,"props":1724,"children":1725},{},[1726],{"type":45,"value":1727},"edge",{"type":45,"value":1729},": where a reasonable person could\ncall the same behavior acceptable or unacceptable.",{"type":40,"tag":47,"props":1731,"children":1732},{},[1733,1738],{"type":40,"tag":53,"props":1734,"children":1735},{},[1736],{"type":45,"value":1737},"Propose, don't interrogate.",{"type":45,"value":1739}," Put candidate verdicts in front of the user and let them rule:",{"type":40,"tag":78,"props":1741,"children":1742},{},[1743],{"type":40,"tag":47,"props":1744,"children":1745},{},[1746,1748,1753,1755,1759,1761,1765],{"type":45,"value":1747},"For \"does not steer by protected class\" — I'd fail these three: ",{"type":40,"tag":260,"props":1749,"children":1750},{},[1751],{"type":45,"value":1752},"...",{"type":45,"value":1754},". I'd pass this one because\nthe renter asked first: ",{"type":40,"tag":260,"props":1756,"children":1757},{},[1758],{"type":45,"value":1752},{"type":45,"value":1760},". I'm unsure about this one: ",{"type":40,"tag":260,"props":1762,"children":1763},{},[1764],{"type":45,"value":1752},{"type":45,"value":1766},". Where do you draw it?",{"type":40,"tag":47,"props":1768,"children":1769},{},[1770],{"type":45,"value":1771},"This yields far more signal per turn than open questions, and costs the user a judgment call rather\nthan a specification.",{"type":40,"tag":47,"props":1773,"children":1774},{},[1775],{"type":45,"value":1776},"Probes, each carrying your own proposed answer:",{"type":40,"tag":377,"props":1778,"children":1779},{},[1780,1797,1807,1817],{"type":40,"tag":98,"props":1781,"children":1782},{},[1783,1788,1790,1795],{"type":40,"tag":53,"props":1784,"children":1785},{},[1786],{"type":45,"value":1787},"Near-miss.",{"type":45,"value":1789}," \"Would you fail this one, almost identical but ",{"type":40,"tag":260,"props":1791,"children":1792},{},[1793],{"type":45,"value":1794},"X",{"type":45,"value":1796},"?\"",{"type":40,"tag":98,"props":1798,"children":1799},{},[1800,1805],{"type":40,"tag":53,"props":1801,"children":1802},{},[1803],{"type":45,"value":1804},"Legitimate exception.",{"type":45,"value":1806}," \"Is there a case where this behavior is actually correct?\"",{"type":40,"tag":98,"props":1808,"children":1809},{},[1810,1815],{"type":40,"tag":53,"props":1811,"children":1812},{},[1813],{"type":45,"value":1814},"Severity split.",{"type":45,"value":1816}," \"Is one of these a red line and the other a papercut?\"",{"type":40,"tag":98,"props":1818,"children":1819},{},[1820,1825],{"type":40,"tag":53,"props":1821,"children":1822},{},[1823],{"type":45,"value":1824},"Not-applicable.",{"type":45,"value":1826}," \"What should the check say when the criterion doesn't apply?\"",{"type":40,"tag":47,"props":1828,"children":1829},{},[1830],{"type":45,"value":1831},"A criterion is done when the user has corrected or confirmed at least one proposed verdict, two\npeople would label the same example the same way, and you know what \"not applicable\" looks like.",{"type":40,"tag":47,"props":1833,"children":1834},{},[1835],{"type":45,"value":1836},"Small snippets help here — an output shape, a three-item keyword list. Full implementations do not.",{"type":40,"tag":532,"props":1838,"children":1840},{"id":1839},"phase-6-confirm-before-implementing",[1841],{"type":45,"value":1842},"Phase 6: Confirm before implementing",{"type":40,"tag":47,"props":1844,"children":1845},{},[1846,1848,1852,1854,1859],{"type":45,"value":1847},"Restate the suite as a ",{"type":40,"tag":53,"props":1849,"children":1850},{},[1851],{"type":45,"value":1082},{"type":45,"value":1853},", one row per check, with these columns filled in for every row:\nname | layer (rule\u002Foutcome) | implementation | why not cheaper | output. A blank cell is a design\nhole, not a formatting choice — an LLM judge with no \"why not cheaper\" is unroutable, and a suite\nwith no outcome row is incomplete. Then ",{"type":40,"tag":53,"props":1855,"children":1856},{},[1857],{"type":45,"value":1858},"ask the user to confirm or change it",{"type":45,"value":1860}," and wait.",{"type":40,"tag":78,"props":1862,"children":1863},{},[1864],{"type":40,"tag":47,"props":1865,"children":1866},{},[1867],{"type":45,"value":1868},"Here's the suite I'd build. Anything to add, drop, or reword before I write it?",{"type":40,"tag":47,"props":1870,"children":1871},{},[1872,1877],{"type":40,"tag":53,"props":1873,"children":1874},{},[1875],{"type":45,"value":1876},"Enforce the size cap here.",{"type":45,"value":1878}," If you have more than 5 checks, do not ask the user to prune — say\nwhich you would drop and why, then let them overrule you. Adding a criterion means replacing a\nweaker one, not growing the list. A 9-check v1 is a checklist, and checklists do not get validated.",{"type":40,"tag":47,"props":1880,"children":1881},{},[1882],{"type":45,"value":1883},"Do not write the full implementation on the same turn as the confirmation request. If anything\nchanges, re-confirm the changed items.",{"type":40,"tag":532,"props":1885,"children":1887},{"id":1886},"phase-7-deliver-the-implementation",[1888],{"type":45,"value":1889},"Phase 7: Deliver the implementation",{"type":40,"tag":47,"props":1891,"children":1892},{},[1893,1895,1900],{"type":45,"value":1894},"Only after confirmation. Deliver ",{"type":40,"tag":53,"props":1896,"children":1897},{},[1898],{"type":45,"value":1899},"one consolidated final message",{"type":45,"value":1901},", not code spread across turns.",{"type":40,"tag":377,"props":1903,"children":1904},{},[1905,1918,1930],{"type":40,"tag":98,"props":1906,"children":1907},{},[1908,1910,1916],{"type":45,"value":1909},"Code checks: runnable, with imports, malformed-input guards, and a runnable\n",{"type":40,"tag":159,"props":1911,"children":1913},{"className":1912},[],[1914],{"type":45,"value":1915},"mlflow.genai.evaluate(...)",{"type":45,"value":1917}," call.",{"type":40,"tag":98,"props":1919,"children":1920},{},[1921,1923,1928],{"type":45,"value":1922},"Built-ins: state the data contract from ",{"type":40,"tag":159,"props":1924,"children":1926},{"className":1925},[],[1927],{"type":45,"value":422},{"type":45,"value":1929},". If the data is absent, re-route or\nmark as a future upgrade.",{"type":40,"tag":98,"props":1931,"children":1932},{},[1933,1935,1941,1943,1949,1950,1956,1957,1963,1964,1969,1970,1976,1978,1983,1985,1990,1991,1996],{"type":45,"value":1934},"LLM judges: give the actual ",{"type":40,"tag":159,"props":1936,"children":1938},{"className":1937},[],[1939],{"type":45,"value":1940},"instructions",{"type":45,"value":1942}," string tailored to their domain, with template\nvariables (",{"type":40,"tag":159,"props":1944,"children":1946},{"className":1945},[],[1947],{"type":45,"value":1948},"{{ inputs }}",{"type":45,"value":398},{"type":40,"tag":159,"props":1951,"children":1953},{"className":1952},[],[1954],{"type":45,"value":1955},"{{ outputs }}",{"type":45,"value":398},{"type":40,"tag":159,"props":1958,"children":1960},{"className":1959},[],[1961],{"type":45,"value":1962},"{{ expectations }}",{"type":45,"value":398},{"type":40,"tag":159,"props":1965,"children":1967},{"className":1966},[],[1968],{"type":45,"value":769},{"type":45,"value":956},{"type":40,"tag":159,"props":1971,"children":1973},{"className":1972},[],[1974],{"type":45,"value":1975},"{{ conversation }}",{"type":45,"value":1977},"), a ",{"type":40,"tag":159,"props":1979,"children":1981},{"className":1980},[],[1982],{"type":45,"value":164},{"type":45,"value":1984}," or ",{"type":40,"tag":159,"props":1986,"children":1988},{"className":1987},[],[1989],{"type":45,"value":172},{"type":45,"value":174},{"type":40,"tag":159,"props":1992,"children":1994},{"className":1993},[],[1995],{"type":45,"value":180},{"type":45,"value":1997}," output, and cost at their stated volume.\nAlways add: \"Once traces and human labels come in, align this judge and validate agreement before\nrelying on it for monitoring.\"",{"type":40,"tag":47,"props":1999,"children":2000},{},[2001,2006,2008,2014,2016,2022,2023,2029,2031,2037],{"type":40,"tag":53,"props":2002,"children":2003},{},[2004],{"type":45,"value":2005},"Judge model.",{"type":45,"value":2007}," Set ",{"type":40,"tag":159,"props":2009,"children":2011},{"className":2010},[],[2012],{"type":45,"value":2013},"model=",{"type":45,"value":2015}," explicitly and start mid-tier — ",{"type":40,"tag":159,"props":2017,"children":2019},{"className":2018},[],[2020],{"type":45,"value":2021},"anthropic:\u002Fclaude-sonnet-4-6",{"type":45,"value":956},{"type":40,"tag":159,"props":2024,"children":2026},{"className":2025},[],[2027],{"type":45,"value":2028},"openai:\u002Fgpt-4.1",{"type":45,"value":2030},", or the equivalent tier on your provider. Scale up if the judge disagrees with\nhuman labels on clear-cut cases; scale down only after measuring agreement on a labelled sample.\nNever downgrade a red-line judge (safety, legal) to save cents — a code prefilter already keeps its\nvolume low. ",{"type":40,"tag":159,"props":2032,"children":2034},{"className":2033},[],[2035],{"type":45,"value":2036},"agent-evaluation\u002Freferences\u002Fscorers.md",{"type":45,"value":2038}," is authoritative for URI formats and keys.",{"type":40,"tag":532,"props":2040,"children":2042},{"id":2041},"phase-8-name-the-validation-plan-then-hand-off",[2043],{"type":45,"value":2044},"Phase 8: Name the validation plan, then hand off",{"type":40,"tag":47,"props":2046,"children":2047},{},[2048,2050,2055,2057,2063],{"type":45,"value":2049},"Decide ",{"type":40,"tag":913,"props":2051,"children":2052},{},[2053],{"type":45,"value":2054},"how the suite will be judged",{"type":45,"value":2056}," — then stop. Running the evaluation is ",{"type":40,"tag":159,"props":2058,"children":2060},{"className":2059},[],[2061],{"type":45,"value":2062},"agent-evaluation",{"type":45,"value":2064},"'s\njob, and this skill must not duplicate it.",{"type":40,"tag":47,"props":2066,"children":2067},{},[2068,2073],{"type":40,"tag":53,"props":2069,"children":2070},{},[2071],{"type":45,"value":2072},"Lead with labels the user already has.",{"type":45,"value":2074}," Before proposing anyone hand-label a fresh dataset, ask\nwhat past failures are already recorded — support tickets, complaints, incident reports,\nthumbs-down feedback, refunds, escalations. Backtesting against real known-bad cases beats synthetic\nlabels and costs nothing to produce. Propose hand-labeling only if no such history exists.",{"type":40,"tag":47,"props":2076,"children":2077},{},[2078,2080,2085],{"type":45,"value":2079},"For each LLM judge, state the alignment plan: which labels it will be aligned against, and that its\nfirst version is a draft. Be cautious aligning subjective axes like tone or style — alignment can\noverfit. Note where alignment is impossible: session-level built-ins raise ",{"type":40,"tag":159,"props":2081,"children":2083},{"className":2082},[],[2084],{"type":45,"value":1449},{"type":45,"value":445},{"type":40,"tag":47,"props":2087,"children":2088},{},[2089],{"type":45,"value":2090},"Close with the confirmed suite, the implementation, parked v2 ideas, trace-only upgrades, and the\nvalidation plan.",{"type":40,"tag":47,"props":2092,"children":2093},{},[2094],{"type":40,"tag":53,"props":2095,"children":2096},{},[2097],{"type":45,"value":2098},"Then hand off — do not run the evaluation yourself.",{"type":40,"tag":377,"props":2100,"children":2101},{},[2102,2120],{"type":40,"tag":98,"props":2103,"children":2104},{},[2105,2110,2112,2118],{"type":40,"tag":159,"props":2106,"children":2108},{"className":2107},[],[2109],{"type":45,"value":2062},{"type":45,"value":2111}," — registering scorers, dataset discovery\u002Fcreation, dry run, full evaluation,\nresults analysis, and iteration. Registration matters: unregistered inline scorers do not appear in\n",{"type":40,"tag":159,"props":2113,"children":2115},{"className":2114},[],[2116],{"type":45,"value":2117},"mlflow scorers list",{"type":45,"value":2119}," and are not reusable.",{"type":40,"tag":98,"props":2121,"children":2122},{},[2123,2129,2131,2136,2138,2143],{"type":40,"tag":159,"props":2124,"children":2126},{"className":2125},[],[2127],{"type":45,"value":2128},"instrumenting-with-mlflow-tracing",{"type":45,"value":2130}," — tracing gaps surfaced in Phase 2, such as a lookup\ninstrumented as ",{"type":40,"tag":159,"props":2132,"children":2134},{"className":2133},[],[2135],{"type":45,"value":751},{"type":45,"value":2137}," when a criterion needs ",{"type":40,"tag":159,"props":2139,"children":2141},{"className":2140},[],[2142],{"type":45,"value":728},{"type":45,"value":445},{"type":40,"tag":78,"props":2145,"children":2146},{},[2147],{"type":40,"tag":47,"props":2148,"children":2149},{},[2150,2155,2157,2162,2164,2169,2171,2176],{"type":40,"tag":53,"props":2151,"children":2152},{},[2153],{"type":45,"value":2154},"Scope boundary.",{"type":45,"value":2156}," This skill decides ",{"type":40,"tag":913,"props":2158,"children":2159},{},[2160],{"type":45,"value":2161},"what to measure and how to implement it",{"type":45,"value":2163},".\n",{"type":40,"tag":159,"props":2165,"children":2167},{"className":2166},[],[2168],{"type":45,"value":2062},{"type":45,"value":2170}," decides ",{"type":40,"tag":913,"props":2172,"children":2173},{},[2174],{"type":45,"value":2175},"how to run it and what the results mean",{"type":45,"value":2177},". If you find yourself\npreparing a dataset, registering scorers, or interpreting eval output, you have left this skill's\nscope — hand off instead.",{"type":40,"tag":87,"props":2179,"children":2181},{"id":2180},"anti-patterns",[2182],{"type":45,"value":2183},"Anti-patterns",{"type":40,"tag":377,"props":2185,"children":2186},{},[2187,2192,2197,2202,2207,2233,2238,2250,2255,2260,2265,2270,2275,2287,2292,2302],{"type":40,"tag":98,"props":2188,"children":2189},{},[2190],{"type":45,"value":2191},"Starting from the scorer catalog instead of the user's app and failures.",{"type":40,"tag":98,"props":2193,"children":2194},{},[2195],{"type":45,"value":2196},"Writing code before the user agreed to named criteria. If they ask to talk, talk.",{"type":40,"tag":98,"props":2198,"children":2199},{},[2200],{"type":45,"value":2201},"Dumping a large code block mid-conversation; it ends the sharpening discussion.",{"type":40,"tag":98,"props":2203,"children":2204},{},[2205],{"type":45,"value":2206},"Shipping a suite of all-LLM judges without stating why each one can't be a code check or built-in\n— the most common and most expensive failure of this skill.",{"type":40,"tag":98,"props":2208,"children":2209},{},[2210,2212,2217,2219,2224,2226,2231],{"type":45,"value":2211},"Hand-rolling a ",{"type":40,"tag":159,"props":2213,"children":2215},{"className":2214},[],[2216],{"type":45,"value":498},{"type":45,"value":2218}," for something MLflow ships. Policy adherence is ",{"type":40,"tag":159,"props":2220,"children":2222},{"className":2221},[],[2223],{"type":45,"value":396},{"type":45,"value":2225},";\npattern rules are ",{"type":40,"tag":159,"props":2227,"children":2229},{"className":2228},[],[2230],{"type":45,"value":404},{"type":45,"value":2232},"; multi-turn quality has session-level built-ins.",{"type":40,"tag":98,"props":2234,"children":2235},{},[2236],{"type":45,"value":2237},"Shipping only rule\u002Fpolicy checks with nothing measuring whether users succeeded. An agent can\npass every compliance check and still fail everyone who used it.",{"type":40,"tag":98,"props":2239,"children":2240},{},[2241,2243,2248],{"type":45,"value":2242},"Substituting a checkable proxy for the outcome that matters (",{"type":40,"tag":159,"props":2244,"children":2246},{"className":2245},[],[2247],{"type":45,"value":1004},{"type":45,"value":2249}," in place of\n\"the renter got the best available deal\") and treating the outcome layer as covered.",{"type":40,"tag":98,"props":2251,"children":2252},{},[2253],{"type":45,"value":2254},"Answering a cost objection with sampling alone when a cheaper implementation was available.",{"type":40,"tag":98,"props":2256,"children":2257},{},[2258],{"type":45,"value":2259},"Proposing fresh hand-labeling before asking what labels the user already has.",{"type":40,"tag":98,"props":2261,"children":2262},{},[2263],{"type":45,"value":2264},"Interviewing past 3 questions without showing a draft.",{"type":40,"tag":98,"props":2266,"children":2267},{},[2268],{"type":45,"value":2269},"Asking about internal gates or state machines in order to write a judge.",{"type":40,"tag":98,"props":2271,"children":2272},{},[2273],{"type":45,"value":2274},"Checking only the end state when the failure happened upstream in the conversation.",{"type":40,"tag":98,"props":2276,"children":2277},{},[2278,2280,2285],{"type":45,"value":2279},"Blocking on traces when a dataset\u002F",{"type":40,"tag":159,"props":2281,"children":2283},{"className":2282},[],[2284],{"type":45,"value":206},{"type":45,"value":2286}," prototype would work.",{"type":40,"tag":98,"props":2288,"children":2289},{},[2290],{"type":45,"value":2291},"Combining multiple criteria into one scorer, or scoring one-off examples that don't generalize.",{"type":40,"tag":98,"props":2293,"children":2294},{},[2295,2300],{"type":40,"tag":159,"props":2296,"children":2298},{"className":2297},[],[2299],{"type":45,"value":188},{"type":45,"value":2301}," scores by default; code with undefined placeholders.",{"type":40,"tag":98,"props":2303,"children":2304},{},[2305,2310],{"type":40,"tag":159,"props":2306,"children":2308},{"className":2307},[],[2309],{"type":45,"value":705},{"type":45,"value":2311}," on TOOL-only traces.",{"type":40,"tag":2313,"props":2314,"children":2315},"style",{},[2316],{"type":45,"value":2317},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2319,"total":2475},[2320,2336,2346,2360,2375,2386,2393,2409,2426,2437,2449,2464],{"slug":2321,"name":2321,"fn":2322,"description":2323,"org":2324,"tags":2325,"stars":2333,"repoUrl":2334,"updatedAt":2335},"setup","configure MLflow tracing","Configure MLflow tracing for Claude Code.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2326,2329,2330],{"name":2327,"slug":2328,"type":13},"Claude Code","claude-code",{"name":9,"slug":8,"type":13},{"name":2331,"slug":2332,"type":13},"Tracing","tracing",27014,"https:\u002F\u002Fgithub.com\u002Fmlflow\u002Fmlflow","2026-07-14T05:39:00.297769",{"slug":2337,"name":2337,"fn":2338,"description":2339,"org":2340,"tags":2341,"stars":2333,"repoUrl":2334,"updatedAt":2345},"status","display MLflow tracing configuration","Show the current MLflow tracing configuration for Claude Code.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2342,2343,2344],{"name":2327,"slug":2328,"type":13},{"name":9,"slug":8,"type":13},{"name":2331,"slug":2332,"type":13},"2026-07-14T05:39:01.540537",{"slug":2062,"name":2062,"fn":2347,"description":2348,"org":2349,"tags":2350,"stars":23,"repoUrl":24,"updatedAt":2359},"evaluate and optimize LLM agent output","Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output quality - including improving tool selection accuracy, answer quality, reducing costs, or fixing issues where the agent gives wrong\u002Fincomplete responses. Evaluates agents systematically using MLflow evaluation with datasets, scorers, and tracing. IMPORTANT - Always also load the instrumenting-with-mlflow-tracing skill before starting any work. Covers end-to-end evaluation workflow or individual components (tracing setup, dataset creation, scorer definition, evaluation execution).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2351,2354,2355,2356],{"name":2352,"slug":2353,"type":13},"Agents","agents",{"name":18,"slug":19,"type":13},{"name":9,"slug":8,"type":13},{"name":2357,"slug":2358,"type":13},"Performance","performance","2026-08-12T04:26:23.404643",{"slug":2361,"name":2361,"fn":2362,"description":2363,"org":2364,"tags":2365,"stars":23,"repoUrl":24,"updatedAt":2374},"analyzing-mlflow-session","analyze MLflow chat conversation sessions","Analyzes an MLflow session — a sequence of traces from a multi-turn chat conversation or interaction. Use when the user asks to debug a chat conversation, review session or chat history, find where a multi-turn chat went wrong, or analyze patterns across turns. Triggers on \"analyze this session\", \"what happened in this conversation\", \"debug session\", \"review chat history\", \"where did this chat go wrong\", \"session traces\", \"analyze chat\", \"debug this chat\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2366,2369,2370,2373],{"name":2367,"slug":2368,"type":13},"Debugging","debugging",{"name":9,"slug":8,"type":13},{"name":2371,"slug":2372,"type":13},"Observability","observability",{"name":2331,"slug":2332,"type":13},"2026-07-14T05:39:10.542342",{"slug":2376,"name":2376,"fn":2377,"description":2378,"org":2379,"tags":2380,"stars":23,"repoUrl":24,"updatedAt":2385},"analyzing-mlflow-trace","analyze MLflow traces","Analyzes a single MLflow trace to answer a user query about it. Use when the user provides a trace ID and asks to debug, investigate, find issues, root-cause errors, understand behavior, or analyze quality. Triggers on \"analyze this trace\", \"what went wrong with this trace\", \"debug trace\", \"investigate trace\", \"why did this trace fail\", \"root cause this trace\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2381,2382,2383,2384],{"name":2367,"slug":2368,"type":13},{"name":9,"slug":8,"type":13},{"name":2371,"slug":2372,"type":13},{"name":2331,"slug":2332,"type":13},"2026-07-14T05:39:02.874441",{"slug":4,"name":4,"fn":5,"description":6,"org":2387,"tags":2388,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2389,2390,2391,2392],{"name":18,"slug":19,"type":13},{"name":15,"slug":16,"type":13},{"name":9,"slug":8,"type":13},{"name":21,"slug":22,"type":13},{"slug":2394,"name":2394,"fn":2395,"description":2396,"org":2397,"tags":2398,"stars":23,"repoUrl":24,"updatedAt":2408},"fix-agent-issue","fix and update AI agent behavior","Drives a disciplined explore → plan → implement → verify loop for changing an AI agent's behavior with confidence — whether fixing a reported failure or introducing a new requirement, business rule, or policy. Grounds the diagnosis in MLflow traces, codifies the desired behavior as a regression test suite (`mlflow.genai.evaluate` assertions in `@mlflow.test` pytest tests), and iterates the agent — not the test — until green, resisting quick system-prompt patches when the real fix is upstream (missing tool, retrieval source, or capability). Use whenever the user wants to fix or change how an agent behaves — e.g. \"fix this issue in my agent\", \"this answer is wrong\", \"the agent is hallucinating\", \"improve my agent based on this trace\", \"make the agent do X instead of Y\", \"I want the agent to lead with\u002Fprioritize\u002Frecommend X\", \"new business rule: the agent should X\", \"always\u002Fnever do X\", \"change the agent's default behavior\" — or shares a trace they want addressed.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2399,2400,2403,2404,2407],{"name":2352,"slug":2353,"type":13},{"name":2401,"slug":2402,"type":13},"Best Practices","best-practices",{"name":2367,"slug":2368,"type":13},{"name":2405,"slug":2406,"type":13},"Engineering","engineering",{"name":9,"slug":8,"type":13},"2026-07-30T05:53:39.749464",{"slug":2128,"name":2128,"fn":2410,"description":2411,"org":2412,"tags":2413,"stars":23,"repoUrl":24,"updatedAt":2425},"instrument Python and TypeScript with MLflow Tracing","Instruments Python and TypeScript code with MLflow Tracing for observability. Must be loaded when setting up tracing as part of any workflow including agent evaluation. Triggers on adding tracing, instrumenting agents\u002FLLM apps, getting started with MLflow tracing, tracing specific frameworks (LangGraph, LangChain, OpenAI, Gemini, DSPy, CrewAI, AutoGen), or when another skill references tracing setup. Examples - \"How do I add tracing?\", \"Instrument my agent\", \"Trace my LangChain app\", \"Set up tracing for evaluation\"",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2414,2415,2416,2417,2418,2421,2422],{"name":2352,"slug":2353,"type":13},{"name":18,"slug":19,"type":13},{"name":9,"slug":8,"type":13},{"name":2371,"slug":2372,"type":13},{"name":2419,"slug":2420,"type":13},"Python","python",{"name":2331,"slug":2332,"type":13},{"name":2423,"slug":2424,"type":13},"TypeScript","typescript","2026-08-02T05:42:54.839246",{"slug":2427,"name":2427,"fn":2428,"description":2429,"org":2430,"tags":2431,"stars":23,"repoUrl":24,"updatedAt":2436},"mlflow-agent","dispatch MLflow workflows and agent tasks","Master dispatcher for all MLflow workflows. Use this skill when the user wants to do anything with MLflow — tracing, evaluating, debugging, or improving an agent. Routes to the right MLflow sub-skill automatically. Triggers on: \"use mlflow\", \"help with mlflow\", \"mlflow agent\", \"add mlflow to my project\", \"trace my agent\", \"evaluate my agent\", or any MLflow task without a specific skill in mind.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2432,2433,2434,2435],{"name":2352,"slug":2353,"type":13},{"name":9,"slug":8,"type":13},{"name":21,"slug":22,"type":13},{"name":2331,"slug":2332,"type":13},"2026-07-14T05:39:04.133424",{"slug":2438,"name":2438,"fn":2439,"description":2440,"org":2441,"tags":2442,"stars":23,"repoUrl":24,"updatedAt":2448},"mlflow-onboarding","onboard users to MLflow workflows","Onboards users to MLflow by determining their use case (GenAI agents\u002Fapps or traditional ML\u002Fdeep learning) and guiding them through relevant quickstart tutorials and initial integration. If an experiment ID is available, it should be supplied as input to help determine the use case. Use when the user asks to get started with MLflow, set up tracking, add observability, or integrate MLflow into their project. Triggers on \"get started with MLflow\", \"set up MLflow\", \"onboard to MLflow\", \"add MLflow to my project\", \"how do I use MLflow\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2443,2444,2445],{"name":9,"slug":8,"type":13},{"name":21,"slug":22,"type":13},{"name":2446,"slug":2447,"type":13},"Onboarding","onboarding","2026-07-14T05:39:06.681003",{"slug":2450,"name":2450,"fn":2451,"description":2452,"org":2453,"tags":2454,"stars":23,"repoUrl":24,"updatedAt":2463},"querying-mlflow-metrics","fetch trace metrics from MLflow servers","Fetches aggregated trace metrics (token usage, latency, trace counts, quality evaluations) from MLflow tracking servers. Triggers on requests to show metrics, analyze token usage, view LLM costs, check usage trends, or query trace statistics.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2455,2458,2461,2462],{"name":2456,"slug":2457,"type":13},"Analytics","analytics",{"name":2459,"slug":2460,"type":13},"Metrics","metrics",{"name":9,"slug":8,"type":13},{"name":2371,"slug":2372,"type":13},"2026-07-14T05:39:13.07056",{"slug":2465,"name":2465,"fn":2466,"description":2467,"org":2468,"tags":2469,"stars":23,"repoUrl":24,"updatedAt":2474},"retrieving-mlflow-traces","retrieve and query MLflow traces","Retrieves MLflow traces using CLI or Python API. Use when the user asks to get a trace by ID, find traces, filter traces by status\u002Ftags\u002Fmetadata\u002Fexecution time, query traces, or debug failed traces. Triggers on \"get trace\", \"search traces\", \"find failed traces\", \"filter traces by\", \"traces slower than\", \"query MLflow traces\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2470,2471,2472,2473],{"name":2367,"slug":2368,"type":13},{"name":9,"slug":8,"type":13},{"name":2371,"slug":2372,"type":13},{"name":2331,"slug":2332,"type":13},"2026-07-14T05:39:09.22888",14,{"items":2477,"total":2531},[2478,2485,2492,2499,2506,2514,2524],{"slug":2062,"name":2062,"fn":2347,"description":2348,"org":2479,"tags":2480,"stars":23,"repoUrl":24,"updatedAt":2359},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2481,2482,2483,2484],{"name":2352,"slug":2353,"type":13},{"name":18,"slug":19,"type":13},{"name":9,"slug":8,"type":13},{"name":2357,"slug":2358,"type":13},{"slug":2361,"name":2361,"fn":2362,"description":2363,"org":2486,"tags":2487,"stars":23,"repoUrl":24,"updatedAt":2374},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2488,2489,2490,2491],{"name":2367,"slug":2368,"type":13},{"name":9,"slug":8,"type":13},{"name":2371,"slug":2372,"type":13},{"name":2331,"slug":2332,"type":13},{"slug":2376,"name":2376,"fn":2377,"description":2378,"org":2493,"tags":2494,"stars":23,"repoUrl":24,"updatedAt":2385},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2495,2496,2497,2498],{"name":2367,"slug":2368,"type":13},{"name":9,"slug":8,"type":13},{"name":2371,"slug":2372,"type":13},{"name":2331,"slug":2332,"type":13},{"slug":4,"name":4,"fn":5,"description":6,"org":2500,"tags":2501,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2502,2503,2504,2505],{"name":18,"slug":19,"type":13},{"name":15,"slug":16,"type":13},{"name":9,"slug":8,"type":13},{"name":21,"slug":22,"type":13},{"slug":2394,"name":2394,"fn":2395,"description":2396,"org":2507,"tags":2508,"stars":23,"repoUrl":24,"updatedAt":2408},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2509,2510,2511,2512,2513],{"name":2352,"slug":2353,"type":13},{"name":2401,"slug":2402,"type":13},{"name":2367,"slug":2368,"type":13},{"name":2405,"slug":2406,"type":13},{"name":9,"slug":8,"type":13},{"slug":2128,"name":2128,"fn":2410,"description":2411,"org":2515,"tags":2516,"stars":23,"repoUrl":24,"updatedAt":2425},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2517,2518,2519,2520,2521,2522,2523],{"name":2352,"slug":2353,"type":13},{"name":18,"slug":19,"type":13},{"name":9,"slug":8,"type":13},{"name":2371,"slug":2372,"type":13},{"name":2419,"slug":2420,"type":13},{"name":2331,"slug":2332,"type":13},{"name":2423,"slug":2424,"type":13},{"slug":2427,"name":2427,"fn":2428,"description":2429,"org":2525,"tags":2526,"stars":23,"repoUrl":24,"updatedAt":2436},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2527,2528,2529,2530],{"name":2352,"slug":2353,"type":13},{"name":9,"slug":8,"type":13},{"name":21,"slug":22,"type":13},{"name":2331,"slug":2332,"type":13},12]