[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-mlflow-agent-evaluation":3,"mdc--5sdoho-key":33,"related-org-mlflow-agent-evaluation":3847,"related-repo-mlflow-agent-evaluation":4005},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":31,"mdContent":32},"agent-evaluation","evaluate and optimize LLM agent output","Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output quality - including improving tool selection accuracy, answer quality, reducing costs, or fixing issues where the agent gives wrong\u002Fincomplete responses. Evaluates agents systematically using MLflow evaluation with datasets, scorers, and tracing. IMPORTANT - Always also load the instrumenting-with-mlflow-tracing skill before starting any work. Covers end-to-end evaluation workflow or individual components (tracing setup, dataset creation, scorer definition, evaluation execution).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"mlflow","MLflow","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fmlflow.png",[12,16,17,20],{"name":13,"slug":14,"type":15},"Performance","performance","tag",{"name":9,"slug":8,"type":15},{"name":18,"slug":19,"type":15},"Evals","evals",{"name":21,"slug":22,"type":15},"Agents","agents",60,"https:\u002F\u002Fgithub.com\u002Fmlflow\u002Fskills","2026-07-14T05:39:15.600492",null,19,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":26},[],"https:\u002F\u002Fgithub.com\u002Fmlflow\u002Fskills\u002Ftree\u002FHEAD\u002Fagent-evaluation","---\nname: agent-evaluation\ndescription: Use this when you need to EVALUATE OR IMPROVE or OPTIMIZE an existing LLM agent's output quality - including improving tool selection accuracy, answer quality, reducing costs, or fixing issues where the agent gives wrong\u002Fincomplete responses. Evaluates agents systematically using MLflow evaluation with datasets, scorers, and tracing. IMPORTANT - Always also load the instrumenting-with-mlflow-tracing skill before starting any work. Covers end-to-end evaluation workflow or individual components (tracing setup, dataset creation, scorer definition, evaluation execution).\nallowed-tools: Read, Write, Bash, Grep, Glob, WebFetch\n---\n\n# Agent Evaluation with MLflow\n\nComprehensive guide for evaluating GenAI agents with MLflow. Use this skill for the complete evaluation workflow or individual components - tracing setup, environment configuration, dataset creation, scorer definition, or evaluation execution. Each section can be used independently based on your needs.\n\n## ⛔ CRITICAL: Must Use MLflow APIs\n\n**DO NOT create custom evaluation frameworks.** You MUST use MLflow's native APIs:\n\n- **Datasets**: Use `mlflow.genai.datasets.create_dataset()` - NOT custom test case files\n- **Scorers**: Use `mlflow.genai.scorers` and `mlflow.genai.judges.make_judge()` - NOT custom scorer functions\n- **Evaluation**: Use `mlflow.genai.evaluate()` - NOT custom evaluation loops\n- **Scripts**: Use the provided `scripts\u002F` directory templates - NOT custom `evaluation\u002F` directories\n\n**Why?** MLflow tracks everything (datasets, scorers, traces, results) in the experiment. Custom frameworks bypass this and lose all observability.\n\nIf you're tempted to create `evaluation\u002Feval_dataset.py` or similar custom files, STOP. Use `scripts\u002Fcreate_dataset_template.py` instead.\n\n## Table of Contents\n\n1. [Quick Start](#quick-start)\n2. [Documentation Access Protocol](#documentation-access-protocol)\n3. [Setup Overview](#setup-overview)\n4. [Evaluation Workflow](#evaluation-workflow)\n5. [References](#references)\n\n## Quick Start\n\n**⚠️ REMINDER: Use MLflow APIs from this skill. Do not create custom evaluation frameworks.**\n\n**Setup (prerequisite)**: Install MLflow 3.8+, configure environment, integrate tracing\n\n**Evaluation workflow in 5 steps** (each uses MLflow APIs):\n\n1. **Understand**: Run agent, inspect traces, understand purpose\n2. **Scorers**: Select and register scorers for quality criteria\n3. **Dataset**: ALWAYS discover existing datasets first, only create new if needed\n3.5. **Dry Run**: Run 3 questions first — catch broken tools and misconfigured scorers before full eval\n4. **Evaluate**: Run agent on dataset, apply scorers, analyze results\n\n## Command Conventions\n\n**Always use `uv run` for MLflow and Python commands:**\n\n```bash\nuv run mlflow --version          # MLflow CLI commands\nuv run python scripts\u002Fxxx.py     # Python script execution\nuv run python -c \"...\"           # Python one-liners\n```\n\nThis ensures commands run in the correct environment with proper dependencies.\n\n**CRITICAL: Separate stderr from stdout when capturing CLI output:**\n\nWhen saving CLI command output to files for parsing (JSON, CSV, etc.), always redirect stderr separately to avoid mixing logs with structured data:\n\n```bash\n# Save both separately for debugging\nuv run mlflow traces evaluate ... --output json > results.json 2> evaluation.log\n```\n\n## Documentation Access Protocol\n\n**All MLflow documentation must be accessed through llms.txt:**\n\n1. Start at: `https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fllms.txt`\n2. Query llms.txt for your topic with specific prompt\n3. If llms.txt references another doc, use WebFetch with that URL\n4. Do not use WebSearch - use WebFetch with llms.txt first\n\n**This applies to all steps**, especially:\n\n- Dataset creation (read GenAI dataset docs from llms.txt)\n- Scorer registration (check MLflow docs for scorer APIs)\n- Evaluation execution (understand mlflow.genai.evaluate API)\n\n## Discovering Agent Structure\n\n**Each project has unique structure.** Use dynamic exploration instead of assumptions:\n\n### Find Agent Entry Points\n```bash\n# Search for main agent functions\ngrep -r \"def.*agent\" . --include=\"*.py\"\ngrep -r \"def (run|stream|handle|process)\" . --include=\"*.py\"\n\n# Check common locations\nls main.py app.py src\u002F*\u002Fagent.py 2>\u002Fdev\u002Fnull\n\n# Look for API routes\ngrep -r \"@app\\.(get|post)\" . --include=\"*.py\"  # FastAPI\u002FFlask\ngrep -r \"def.*route\" . --include=\"*.py\"\n```\n\n### Understand Project Structure\n```bash\n# Check entry points in package config\ncat pyproject.toml setup.py 2>\u002Fdev\u002Fnull | grep -A 5 \"scripts\\|entry_points\"\n\n# Read project documentation\ncat README.md docs\u002F*.md 2>\u002Fdev\u002Fnull | head -100\n\n# Explore main directories\nls -la src\u002F app\u002F agent\u002F 2>\u002Fdev\u002Fnull\n```\n\n## Setup Overview\n\n### Pre-check: Use Existing Environment\n\n**Before doing ANY setup, check if `MLFLOW_TRACKING_URI` and `MLFLOW_EXPERIMENT_ID` are already set:**\n\n```bash\necho \"MLFLOW_TRACKING_URI=$MLFLOW_TRACKING_URI\"\necho \"MLFLOW_EXPERIMENT_ID=$MLFLOW_EXPERIMENT_ID\"\n```\n\n**If BOTH are already set, skip Steps 1-2 entirely.** The environment is pre-configured. Do NOT run `setup_mlflow.py`, do NOT create a `.env` file, do NOT override these values. Go directly to Step 3 (tracing integration) and the evaluation workflow.\n\n### Setup Steps (only if environment is NOT pre-configured)\n\n1. **Install MLflow** (version >=3.8.0)\n2. **Configure environment** (tracking URI and experiment)\n   - **Guide**: Follow `references\u002Fsetup-guide.md` Steps 1-2\n3. **Integrate tracing** (autolog and @mlflow.trace decorators)\n   - ⚠️ **MANDATORY**: Use the `instrumenting-with-mlflow-tracing` skill for tracing setup\n   - ✓ **VERIFY**: Run `scripts\u002Fvalidate_tracing_runtime.py` after implementing\n\n⚠️ **Tracing must work before evaluation.** If tracing fails, stop and troubleshoot.\n\n**Checkpoint - verify before proceeding:**\n\n- [ ] MLflow >=3.8.0 installed\n- [ ] MLFLOW_TRACKING_URI and MLFLOW_EXPERIMENT_ID set\n- [ ] Autolog enabled and @mlflow.trace decorators added\n- [ ] Test run creates a trace (verify trace ID is not None)\n\n**Validation scripts:**\n```bash\nuv run python scripts\u002Fvalidate_environment.py  # Check MLflow install, env vars, connectivity\nuv run python scripts\u002Fvalidate_auth.py         # Test authentication before expensive operations\n```\n\n## Evaluation Workflow\n\n### Step 1: Agent Interview (REQUIRED — do not skip)\n\nBefore doing anything else, ask the user these questions. Do NOT proceed until you have answers.\n\n**Required:**\n1. \"What does your agent do? Describe its purpose in 1-2 sentences.\"\n2. \"What are the 2-3 most important things it needs to get right?\"\n3. \"Are there common failure modes you've already noticed?\"\n\n**Use answers to:**\n- Derive scorer names and criteria (do not invent them)\n- Write the `agent_description` parameter for `generate_evals_df`\n- Set evaluation priorities\n\n**If running in automated mode:** Read agent purpose from the codebase (SKILL.md, README, or main entry point docstring). Still surface what you found and confirm before proceeding.\n\n### Step 2: Define Quality Scorers\n\n1. **Check registered scorers in your experiment:**\n   ```bash\n   uv run mlflow scorers list -x $MLFLOW_EXPERIMENT_ID\n   ```\n\n**IMPORTANT: if there are registered scorers in the experiment then they must be used for evaluation.**\n\n2. **Select additional built-in scorers that apply to the agent** \n\nSee `references\u002Fscorers.md` for the built-in scorers. Select any that are useful for assessing the agent's quality and that are not already registered. \n\n3. **Create additional custom scorers as needed**\n\nIf needed, create additional scorers using the `make_judge()` API. See `references\u002Fscorers.md` on how to create custom scorers and `references\u002Fscorers-constraints.md` for best practices.\n\n> ⚠️ **CRITICAL — Scorer Return Values:** Scorers MUST instruct the LLM judge to return `\"yes\"` or `\"no\"` (or booleans\u002Fnumerics). Return values of `\"pass\"` or `\"fail\"` are **silently cast to `None`** by `_cast_assessment_value_to_float` and **excluded from `results.metrics`** with no error or warning — results simply disappear. See `references\u002Fscorers-constraints.md` Constraint 2 for the full list of safe vs. broken return values.\n\n4. **REQUIRED: Register new scorers before evaluation** using Python API:\n   \n   ```python\n   from mlflow.genai.judges import make_judge\n   from mlflow.genai.scorers import BuiltinScorerName\n   import os\n\n   scorer = make_judge(...)  # Or, scorer = BuiltinScorerName()\n   scorer.register()\n   ```\n\n** IMPORTANT:  See `references\u002Fscorers.md` → \"Model Selection for Scorers\" to configure the `model` parameter of scorers before registration.\n\n⚠️ **Scorers MUST be registered before evaluation.** Inline scorers that aren't registered won't appear in `mlflow scorers list` and won't be reusable.\n\n5. **Verify registration:**\n   ```bash\n   uv run mlflow scorers list -x $MLFLOW_EXPERIMENT_ID  # Should show your scorers\n   ```\n\n### Step 3: Prepare Evaluation Dataset\n\n**ALWAYS discover existing datasets first** to prevent duplicate work:\n\n1. **Run dataset discovery** (mandatory):\n\n   ```bash\n   uv run python scripts\u002Flist_datasets.py  # Lists, compares, recommends datasets\n   uv run python scripts\u002Flist_datasets.py --format json  # Machine-readable output\n   uv run python scripts\u002Flist_datasets.py --help  # All options\n   ```\n\n2. **Present findings to user**:\n\n   - Show all discovered datasets with their characteristics (size, topics covered)\n   - If datasets found, highlight most relevant options based on agent type\n\n3. **Ask user about existing datasets**:\n\n   - \"I found [N] existing evaluation dataset(s). Do you want to use one of these? (y\u002Fn)\"\n   - If yes: Ask which dataset to use and record the dataset name — skip to Step 3.5\n   - If no: Proceed to Phase A below\n\n**If creating a new dataset, use the two-phase approach below.**\n\n---\n\n#### Phase A: Sanity Check (5 questions — always run first)\n\nCreate a minimal 5-question dataset manually from the Step 1 interview answers. The goal is to confirm the pipeline works end-to-end before investing in large-scale generation.\n\n```python\nimport mlflow\nfrom mlflow.genai.datasets import create_dataset\n\n# Derive 5 representative questions directly from the agent's stated purpose\n# and known failure modes identified in Step 1\nsanity_records = [\n    {\"inputs\": {\"query\": \"\u003Cquestion 1 from interview>\"}, \"expected_response\": \"\u003Cexpected answer>\"},\n    {\"inputs\": {\"query\": \"\u003Cquestion 2 from interview>\"}, \"expected_response\": \"\u003Cexpected answer>\"},\n    # ... 5 total\n]\n\nsanity_dataset = create_dataset(\n    records=sanity_records,\n    name=\"sanity-check-5q\",\n)\n```\n\nRun evaluation on this dataset (see Step 4), then **present results to the user** with this framing:\n\n> \"This is a sanity check — 5 questions confirm the pipeline works but aren't statistically meaningful. Proceeding to Phase B to generate a proper evaluation set.\"\n\nOnly proceed to Phase B once Phase A completes without errors.\n\n---\n\n#### Phase B: Proper Evaluation Dataset (100+ questions — run after Phase A passes)\n\nGenerate questions from the agent's actual corpus rather than inventing them from scratch. The approach depends on whether the project uses Databricks or OSS MLflow.\n\n**On Databricks** — use `generate_evals_df` to synthesize questions from the agent's document corpus:\n\n```python\nfrom databricks.agents.evals import generate_evals_df, estimate_synthetic_num_evals\nimport mlflow\n\n# agent_description comes from Step 1 interview answers\nagent_description = \"\u003Cagent purpose from interview>\"\n\n# docs_df: a Spark or pandas DataFrame with a \"content\" column containing\n# the documents\u002Fchunks the agent retrieves from (e.g., your Vector Search index)\nevals = generate_evals_df(\n    docs=docs_df,\n    num_evals=100,\n    agent_description=agent_description,\n)\n\n# Merge into MLflow dataset — don't create a separate dataset\ndataset = mlflow.genai.datasets.create_dataset(name=\"generated-evals-100q\")\ndataset.merge_records(evals)\n```\n\nTo estimate the right `num_evals` before generating:\n\n```python\nrecommended = estimate_synthetic_num_evals(docs_df)\nprint(f\"Recommended num_evals: {recommended}\")\n```\n\n**Dataset size guidance:**\n- **\u003C30 questions**: not statistically meaningful — avoid drawing conclusions\n- **50–100 questions**: adequate for catching regressions, suitable for most agents\n- **200+ questions**: recommended when comparing model variants or scoring multiple dimensions\n\n**On OSS MLflow** — use RAGAS `TestsetGenerator` to generate from your document corpus:\n\n```python\nfrom ragas.testset import TestsetGenerator\nfrom ragas.llms import LangchainLLMWrapper\nfrom ragas.embeddings import LangchainEmbeddingsWrapper\n\ngenerator = TestsetGenerator(\n    llm=LangchainLLMWrapper(your_llm),\n    embedding_model=LangchainEmbeddingsWrapper(your_embeddings),\n)\ntestset = generator.generate_with_langchain_docs(docs, testset_size=100)\nevals_df = testset.to_pandas()\n\n# Convert to MLflow dataset schema and merge\nimport mlflow\nrecords = [\n    {\"inputs\": {\"query\": row[\"user_input\"]}, \"expected_response\": row[\"reference\"]}\n    for _, row in evals_df.iterrows()\n]\ndataset = mlflow.genai.datasets.create_dataset(name=\"generated-evals-100q\")\ndataset.merge_records(records)\n```\n\n**If no document corpus is available** — ask the user to provide 50–100 representative queries from production logs or usage history. These are more realistic than synthetic questions and are preferable when available.\n\n---\n\n**IMPORTANT**: Do not skip dataset discovery. Always run `list_datasets.py` first, even if you plan to create a new dataset. This prevents duplicate work and ensures users are aware of existing evaluation datasets.\n\n**For complete dataset guide:** See `references\u002Fdataset-preparation.md`\n\n**Checkpoint - verify before proceeding:**\n\n- [ ] Scorers have been registered\n- [ ] Phase A sanity check passed (pipeline runs end-to-end)\n- [ ] Phase B dataset created with 50+ questions (or existing dataset selected)\n\n### Step 3.5: Dry Run (REQUIRED before full eval)\n\nRun evaluation on **3 questions** from the dataset before committing to the full run. This catches broken tools, misconfigured scorers, and auth failures early — before they silently corrupt 100-question results.\n\nIf you completed Phase A above, the pipeline is already validated — focus the dry run on scorer output only.\n\n```python\nimport mlflow\n\ndataset = mlflow.genai.datasets.get_dataset(name=\"\u003Cyour-dataset-name>\")\ndry_run_records = dataset.df.head(3)\n```\n\nRun `mlflow.genai.evaluate()` on these 3 records using the same wrapper and scorers as the full eval.\n\n**For each response, check:**\n\n1. **Tool calls** — Did the agent call any tools? If it called zero tools on questions that require retrieval, tools are likely broken (403s, rate limits, missing credentials).\n2. **Response quality** — Are responses empty or generic (\"I don't know\", \"I can't help with that\")? Empty responses score as irrelevant and will skew the full eval.\n3. **Scorer output** — Did all 3 scores come back as `0` or `None`? If so, the scorer is misconfigured (check return values — `\"pass\"`\u002F`\"fail\"` are silently cast to `None`; use `\"yes\"`\u002F`\"no\"` instead).\n\n**Decision gate:**\n\n- **If dry run shows tool failures or empty responses:** Stop. Fix the underlying issue (auth, tool config, retrieval) before proceeding. Do not run the full eval on broken infrastructure.\n- **If all 3 scorer outputs are 0 or None:** Stop. Debug scorer return values and re-register before proceeding.\n- **If dry run passes:** Report to the user: *\"Dry run passed (3\u002F3 responses non-empty, tools called, scores non-zero). Proceeding to full eval.\"* Then continue to Step 4.\n\n> **Why this matters:** Tool failures (403s from docs scraping, GitHub API rate limits) produce empty agent responses that score as 0. Running a 100-question eval only to discover all tools were failing wastes time and produces misleading results. The dry run catches this in under a minute.\n\n---\n\n### Step 4: Run Evaluation\n\n> **Large datasets (50+ questions)?** See `references\u002Fthroughput-guide.md` for throughput optimization — covers parallelism env vars, async predict functions, and dataset splitting for 200+ question evals.\n\n#### 4a. Estimate Runtime Before Starting\n\nBefore launching evaluation, tell the user how long it will take:\n\n1. **Count the dataset questions:**\n   ```python\n   import mlflow\n   dataset = mlflow.genai.datasets.get_dataset(name=\"\u003Cyour-dataset-name>\")\n   print(f\"Dataset size: {len(dataset.df)} questions\")\n   ```\n\n2. **Calculate the estimate** — each question runs the agent once and the judge scorer once:\n   - Opus-class judge models (e.g. `claude-opus-4`): ~45–90s per question\n   - Sonnet-class judge models (e.g. `claude-sonnet-4`): ~20–45s per question\n   - Multiple scorers per question add time proportionally\n\n   ```\n   Estimated time = N questions × 30–60s per question ÷ parallelism factor (typically 4–8x)\n   ```\n\n3. **Tell the user before starting:**\n   > \"This dataset has N questions. At ~30–60s per question with typical parallelism, evaluation will take approximately **X–Y minutes**. I'll run it as a background task so you can continue working — I'll summarize the results when it's done.\"\n\n#### 4b. Generate the Evaluation Script\n\n```bash\n# Generate evaluation script (specify module and entry point)\nuv run python scripts\u002Frun_evaluation_template.py \\\n  --module mlflow_agent.agent \\\n  --entry-point run_agent\n```\n\nThe generated script creates a wrapper function that:\n- Accepts keyword arguments matching the dataset's input keys\n- Provides any additional arguments the agent needs (like `llm_provider`)\n- Runs `mlflow.genai.evaluate(data=df, predict_fn=wrapper, scorers=registered_scorers)`\n- Saves results to `evaluation_results.csv`\n\n⚠️ **CRITICAL: wrapper Signature Must Match Dataset Input Keys**\n\nMLflow calls `predict_fn(**inputs)` - it unpacks the inputs dict as keyword arguments.\n\n| Dataset Record | MLflow Calls | predict_fn Must Be |\n|----------------|--------------|-------------------|\n| `{\"inputs\": {\"query\": \"...\"}}` | `predict_fn(query=\"...\")` | `def wrapper(query):` |\n| `{\"inputs\": {\"question\": \"...\", \"context\": \"...\"}}` | `predict_fn(question=\"...\", context=\"...\")` | `def wrapper(question, context):` |\n\n**Common Mistake (WRONG):**\n```python\ndef wrapper(inputs):  # ❌ WRONG - inputs is NOT a dict\n    return agent(inputs[\"query\"])\n```\n\n#### 4c. Launch as a Background Sub-Agent\n\nRun the evaluation as a background sub-agent so the main session stays available. Use the Agent tool with `run_in_background: true`:\n\n**Sub-agent instructions (pass these verbatim):**\n```\nRun the agent evaluation and write results to scratchpad.\n\nSteps:\n1. cd \u003Cproject-directory>\n2. Run: uv run python run_agent_evaluation.py\n3. When complete, write a summary to scratchpad\u002Feval-results.md with:\n   - Exit status (success or error message)\n   - Path to results file (evaluation_results.csv)\n   - Wall-clock time taken\n4. Return only: \"Evaluation complete. Results written to scratchpad\u002Feval-results.md\"\n```\n\n**In the main session, poll for completion** by checking for the scratchpad file rather than blocking:\n\n```python\n# Poll every 30s using Glob\n# Glob(\"scratchpad\u002Feval-results.md\")\n# When the file appears, read it and proceed to analysis\n```\n\nDo NOT use TaskOutput to wait for the background agent — that dumps the full transcript (~10–20k tokens) into the main context.\n\n#### 4d. Analyze Results (after evaluation completes)\n\nOnce `scratchpad\u002Feval-results.md` appears, run analysis:\n\n```bash\n# Pattern detection, failure analysis, recommendations\n# Reads the CSV produced by mlflow.genai.evaluate() above\nuv run python scripts\u002Fanalyze_results.py evaluation_results.csv\n```\n\nGenerates `evaluation_report.md` with per-scorer pass rates and improvement suggestions.\n\nThe script reads `{scorer_name}\u002Fvalue` and `{scorer_name}\u002Frationale` columns from the CSV.\nIt also accepts the legacy JSON format from `mlflow traces evaluate` for backward compatibility:\n```bash\nuv run python scripts\u002Fanalyze_results.py evaluation_results.json  # legacy format\nuv run python scripts\u002Fanalyze_results.py evaluation_results.csv --output my_report.md  # custom output\n```\n\n## References\n\nDetailed guides in `references\u002F` (load as needed):\n\n- **setup-guide.md** - Environment setup (MLflow install, tracking URI configuration)\n- **Tracing**: Use the `instrumenting-with-mlflow-tracing` skill (authoritative guide for autolog, decorators, session tracking, verification)\n- **dataset-preparation.md** - Dataset schema, APIs, creation, Unity Catalog\n- **scorers.md** - Built-in vs custom scorers, registration, testing\n- **scorers-constraints.md** - CLI requirements for custom scorers (yes\u002Fno format, templates)\n- **troubleshooting.md** - Common errors by phase with solutions\n- **throughput-guide.md** - Parallelism env vars, async predict_fn, dataset splitting for 200+ question evals\n\nScripts are self-documenting - run with `--help` for usage details.\n",{"data":34,"body":36},{"name":4,"description":6,"allowed-tools":35},"Read, Write, Bash, Grep, Glob, WebFetch",{"type":37,"children":38},"root",[39,48,54,61,72,164,174,195,201,251,256,264,274,284,333,339,355,469,474,482,487,562,567,575,604,614,632,638,648,655,939,945,1133,1138,1144,1167,1226,1252,1258,1355,1366,1374,1418,1426,1483,1488,1494,1499,1507,1525,1533,1565,1575,1581,1634,1642,1653,1666,1677,1705,1791,1860,1880,1899,1954,1960,1970,2134,2142,2146,2153,2158,2288,2300,2308,2313,2316,2322,2327,2344,2484,2497,2520,2528,2561,2579,2733,2743,2746,2764,2780,2787,2818,2824,2836,2841,2878,2890,2898,2979,2987,3028,3041,3044,3050,3070,3076,3081,3198,3204,3274,3279,3322,3331,3344,3437,3445,3468,3474,3486,3494,3503,3513,3544,3549,3555,3568,3616,3629,3657,3731,3736,3749,3828,3841],{"type":40,"tag":41,"props":42,"children":44},"element","h1",{"id":43},"agent-evaluation-with-mlflow",[45],{"type":46,"value":47},"text","Agent Evaluation with MLflow",{"type":40,"tag":49,"props":50,"children":51},"p",{},[52],{"type":46,"value":53},"Comprehensive guide for evaluating GenAI agents with MLflow. Use this skill for the complete evaluation workflow or individual components - tracing setup, environment configuration, dataset creation, scorer definition, or evaluation execution. Each section can be used independently based on your needs.",{"type":40,"tag":55,"props":56,"children":58},"h2",{"id":57},"critical-must-use-mlflow-apis",[59],{"type":46,"value":60},"⛔ CRITICAL: Must Use MLflow APIs",{"type":40,"tag":49,"props":62,"children":63},{},[64,70],{"type":40,"tag":65,"props":66,"children":67},"strong",{},[68],{"type":46,"value":69},"DO NOT create custom evaluation frameworks.",{"type":46,"value":71}," You MUST use MLflow's native APIs:",{"type":40,"tag":73,"props":74,"children":75},"ul",{},[76,96,121,138],{"type":40,"tag":77,"props":78,"children":79},"li",{},[80,85,87,94],{"type":40,"tag":65,"props":81,"children":82},{},[83],{"type":46,"value":84},"Datasets",{"type":46,"value":86},": Use ",{"type":40,"tag":88,"props":89,"children":91},"code",{"className":90},[],[92],{"type":46,"value":93},"mlflow.genai.datasets.create_dataset()",{"type":46,"value":95}," - NOT custom test case files",{"type":40,"tag":77,"props":97,"children":98},{},[99,104,105,111,113,119],{"type":40,"tag":65,"props":100,"children":101},{},[102],{"type":46,"value":103},"Scorers",{"type":46,"value":86},{"type":40,"tag":88,"props":106,"children":108},{"className":107},[],[109],{"type":46,"value":110},"mlflow.genai.scorers",{"type":46,"value":112}," and ",{"type":40,"tag":88,"props":114,"children":116},{"className":115},[],[117],{"type":46,"value":118},"mlflow.genai.judges.make_judge()",{"type":46,"value":120}," - NOT custom scorer functions",{"type":40,"tag":77,"props":122,"children":123},{},[124,129,130,136],{"type":40,"tag":65,"props":125,"children":126},{},[127],{"type":46,"value":128},"Evaluation",{"type":46,"value":86},{"type":40,"tag":88,"props":131,"children":133},{"className":132},[],[134],{"type":46,"value":135},"mlflow.genai.evaluate()",{"type":46,"value":137}," - NOT custom evaluation loops",{"type":40,"tag":77,"props":139,"children":140},{},[141,146,148,154,156,162],{"type":40,"tag":65,"props":142,"children":143},{},[144],{"type":46,"value":145},"Scripts",{"type":46,"value":147},": Use the provided ",{"type":40,"tag":88,"props":149,"children":151},{"className":150},[],[152],{"type":46,"value":153},"scripts\u002F",{"type":46,"value":155}," directory templates - NOT custom ",{"type":40,"tag":88,"props":157,"children":159},{"className":158},[],[160],{"type":46,"value":161},"evaluation\u002F",{"type":46,"value":163}," directories",{"type":40,"tag":49,"props":165,"children":166},{},[167,172],{"type":40,"tag":65,"props":168,"children":169},{},[170],{"type":46,"value":171},"Why?",{"type":46,"value":173}," MLflow tracks everything (datasets, scorers, traces, results) in the experiment. Custom frameworks bypass this and lose all observability.",{"type":40,"tag":49,"props":175,"children":176},{},[177,179,185,187,193],{"type":46,"value":178},"If you're tempted to create ",{"type":40,"tag":88,"props":180,"children":182},{"className":181},[],[183],{"type":46,"value":184},"evaluation\u002Feval_dataset.py",{"type":46,"value":186}," or similar custom files, STOP. Use ",{"type":40,"tag":88,"props":188,"children":190},{"className":189},[],[191],{"type":46,"value":192},"scripts\u002Fcreate_dataset_template.py",{"type":46,"value":194}," instead.",{"type":40,"tag":55,"props":196,"children":198},{"id":197},"table-of-contents",[199],{"type":46,"value":200},"Table of Contents",{"type":40,"tag":202,"props":203,"children":204},"ol",{},[205,215,224,233,242],{"type":40,"tag":77,"props":206,"children":207},{},[208],{"type":40,"tag":209,"props":210,"children":212},"a",{"href":211},"#quick-start",[213],{"type":46,"value":214},"Quick Start",{"type":40,"tag":77,"props":216,"children":217},{},[218],{"type":40,"tag":209,"props":219,"children":221},{"href":220},"#documentation-access-protocol",[222],{"type":46,"value":223},"Documentation Access Protocol",{"type":40,"tag":77,"props":225,"children":226},{},[227],{"type":40,"tag":209,"props":228,"children":230},{"href":229},"#setup-overview",[231],{"type":46,"value":232},"Setup Overview",{"type":40,"tag":77,"props":234,"children":235},{},[236],{"type":40,"tag":209,"props":237,"children":239},{"href":238},"#evaluation-workflow",[240],{"type":46,"value":241},"Evaluation Workflow",{"type":40,"tag":77,"props":243,"children":244},{},[245],{"type":40,"tag":209,"props":246,"children":248},{"href":247},"#references",[249],{"type":46,"value":250},"References",{"type":40,"tag":55,"props":252,"children":254},{"id":253},"quick-start",[255],{"type":46,"value":214},{"type":40,"tag":49,"props":257,"children":258},{},[259],{"type":40,"tag":65,"props":260,"children":261},{},[262],{"type":46,"value":263},"⚠️ REMINDER: Use MLflow APIs from this skill. Do not create custom evaluation frameworks.",{"type":40,"tag":49,"props":265,"children":266},{},[267,272],{"type":40,"tag":65,"props":268,"children":269},{},[270],{"type":46,"value":271},"Setup (prerequisite)",{"type":46,"value":273},": Install MLflow 3.8+, configure environment, integrate tracing",{"type":40,"tag":49,"props":275,"children":276},{},[277,282],{"type":40,"tag":65,"props":278,"children":279},{},[280],{"type":46,"value":281},"Evaluation workflow in 5 steps",{"type":46,"value":283}," (each uses MLflow APIs):",{"type":40,"tag":202,"props":285,"children":286},{},[287,297,306,323],{"type":40,"tag":77,"props":288,"children":289},{},[290,295],{"type":40,"tag":65,"props":291,"children":292},{},[293],{"type":46,"value":294},"Understand",{"type":46,"value":296},": Run agent, inspect traces, understand purpose",{"type":40,"tag":77,"props":298,"children":299},{},[300,304],{"type":40,"tag":65,"props":301,"children":302},{},[303],{"type":46,"value":103},{"type":46,"value":305},": Select and register scorers for quality criteria",{"type":40,"tag":77,"props":307,"children":308},{},[309,314,316,321],{"type":40,"tag":65,"props":310,"children":311},{},[312],{"type":46,"value":313},"Dataset",{"type":46,"value":315},": ALWAYS discover existing datasets first, only create new if needed\n3.5. ",{"type":40,"tag":65,"props":317,"children":318},{},[319],{"type":46,"value":320},"Dry Run",{"type":46,"value":322},": Run 3 questions first — catch broken tools and misconfigured scorers before full eval",{"type":40,"tag":77,"props":324,"children":325},{},[326,331],{"type":40,"tag":65,"props":327,"children":328},{},[329],{"type":46,"value":330},"Evaluate",{"type":46,"value":332},": Run agent on dataset, apply scorers, analyze results",{"type":40,"tag":55,"props":334,"children":336},{"id":335},"command-conventions",[337],{"type":46,"value":338},"Command Conventions",{"type":40,"tag":49,"props":340,"children":341},{},[342],{"type":40,"tag":65,"props":343,"children":344},{},[345,347,353],{"type":46,"value":346},"Always use ",{"type":40,"tag":88,"props":348,"children":350},{"className":349},[],[351],{"type":46,"value":352},"uv run",{"type":46,"value":354}," for MLflow and Python commands:",{"type":40,"tag":356,"props":357,"children":362},"pre",{"className":358,"code":359,"language":360,"meta":361,"style":361},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","uv run mlflow --version          # MLflow CLI commands\nuv run python scripts\u002Fxxx.py     # Python script execution\nuv run python -c \"...\"           # Python one-liners\n","bash","",[363],{"type":40,"tag":88,"props":364,"children":365},{"__ignoreMap":361},[366,400,427],{"type":40,"tag":367,"props":368,"children":371},"span",{"class":369,"line":370},"line",1,[372,378,384,389,394],{"type":40,"tag":367,"props":373,"children":375},{"style":374},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[376],{"type":46,"value":377},"uv",{"type":40,"tag":367,"props":379,"children":381},{"style":380},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[382],{"type":46,"value":383}," run",{"type":40,"tag":367,"props":385,"children":386},{"style":380},[387],{"type":46,"value":388}," mlflow",{"type":40,"tag":367,"props":390,"children":391},{"style":380},[392],{"type":46,"value":393}," --version",{"type":40,"tag":367,"props":395,"children":397},{"style":396},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[398],{"type":46,"value":399},"          # MLflow CLI commands\n",{"type":40,"tag":367,"props":401,"children":403},{"class":369,"line":402},2,[404,408,412,417,422],{"type":40,"tag":367,"props":405,"children":406},{"style":374},[407],{"type":46,"value":377},{"type":40,"tag":367,"props":409,"children":410},{"style":380},[411],{"type":46,"value":383},{"type":40,"tag":367,"props":413,"children":414},{"style":380},[415],{"type":46,"value":416}," python",{"type":40,"tag":367,"props":418,"children":419},{"style":380},[420],{"type":46,"value":421}," scripts\u002Fxxx.py",{"type":40,"tag":367,"props":423,"children":424},{"style":396},[425],{"type":46,"value":426},"     # Python script execution\n",{"type":40,"tag":367,"props":428,"children":430},{"class":369,"line":429},3,[431,435,439,443,448,454,459,464],{"type":40,"tag":367,"props":432,"children":433},{"style":374},[434],{"type":46,"value":377},{"type":40,"tag":367,"props":436,"children":437},{"style":380},[438],{"type":46,"value":383},{"type":40,"tag":367,"props":440,"children":441},{"style":380},[442],{"type":46,"value":416},{"type":40,"tag":367,"props":444,"children":445},{"style":380},[446],{"type":46,"value":447}," -c",{"type":40,"tag":367,"props":449,"children":451},{"style":450},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[452],{"type":46,"value":453}," \"",{"type":40,"tag":367,"props":455,"children":456},{"style":380},[457],{"type":46,"value":458},"...",{"type":40,"tag":367,"props":460,"children":461},{"style":450},[462],{"type":46,"value":463},"\"",{"type":40,"tag":367,"props":465,"children":466},{"style":396},[467],{"type":46,"value":468},"           # Python one-liners\n",{"type":40,"tag":49,"props":470,"children":471},{},[472],{"type":46,"value":473},"This ensures commands run in the correct environment with proper dependencies.",{"type":40,"tag":49,"props":475,"children":476},{},[477],{"type":40,"tag":65,"props":478,"children":479},{},[480],{"type":46,"value":481},"CRITICAL: Separate stderr from stdout when capturing CLI output:",{"type":40,"tag":49,"props":483,"children":484},{},[485],{"type":46,"value":486},"When saving CLI command output to files for parsing (JSON, CSV, etc.), always redirect stderr separately to avoid mixing logs with structured data:",{"type":40,"tag":356,"props":488,"children":490},{"className":358,"code":489,"language":360,"meta":361,"style":361},"# Save both separately for debugging\nuv run mlflow traces evaluate ... --output json > results.json 2> evaluation.log\n",[491],{"type":40,"tag":88,"props":492,"children":493},{"__ignoreMap":361},[494,502],{"type":40,"tag":367,"props":495,"children":496},{"class":369,"line":370},[497],{"type":40,"tag":367,"props":498,"children":499},{"style":396},[500],{"type":46,"value":501},"# Save both separately for debugging\n",{"type":40,"tag":367,"props":503,"children":504},{"class":369,"line":402},[505,509,513,517,522,527,532,537,542,547,552,557],{"type":40,"tag":367,"props":506,"children":507},{"style":374},[508],{"type":46,"value":377},{"type":40,"tag":367,"props":510,"children":511},{"style":380},[512],{"type":46,"value":383},{"type":40,"tag":367,"props":514,"children":515},{"style":380},[516],{"type":46,"value":388},{"type":40,"tag":367,"props":518,"children":519},{"style":380},[520],{"type":46,"value":521}," traces",{"type":40,"tag":367,"props":523,"children":524},{"style":380},[525],{"type":46,"value":526}," evaluate",{"type":40,"tag":367,"props":528,"children":529},{"style":380},[530],{"type":46,"value":531}," ...",{"type":40,"tag":367,"props":533,"children":534},{"style":380},[535],{"type":46,"value":536}," --output",{"type":40,"tag":367,"props":538,"children":539},{"style":380},[540],{"type":46,"value":541}," json",{"type":40,"tag":367,"props":543,"children":544},{"style":450},[545],{"type":46,"value":546}," >",{"type":40,"tag":367,"props":548,"children":549},{"style":380},[550],{"type":46,"value":551}," results.json",{"type":40,"tag":367,"props":553,"children":554},{"style":450},[555],{"type":46,"value":556}," 2>",{"type":40,"tag":367,"props":558,"children":559},{"style":380},[560],{"type":46,"value":561}," evaluation.log\n",{"type":40,"tag":55,"props":563,"children":565},{"id":564},"documentation-access-protocol",[566],{"type":46,"value":223},{"type":40,"tag":49,"props":568,"children":569},{},[570],{"type":40,"tag":65,"props":571,"children":572},{},[573],{"type":46,"value":574},"All MLflow documentation must be accessed through llms.txt:",{"type":40,"tag":202,"props":576,"children":577},{},[578,589,594,599],{"type":40,"tag":77,"props":579,"children":580},{},[581,583],{"type":46,"value":582},"Start at: ",{"type":40,"tag":88,"props":584,"children":586},{"className":585},[],[587],{"type":46,"value":588},"https:\u002F\u002Fmlflow.org\u002Fdocs\u002Flatest\u002Fllms.txt",{"type":40,"tag":77,"props":590,"children":591},{},[592],{"type":46,"value":593},"Query llms.txt for your topic with specific prompt",{"type":40,"tag":77,"props":595,"children":596},{},[597],{"type":46,"value":598},"If llms.txt references another doc, use WebFetch with that URL",{"type":40,"tag":77,"props":600,"children":601},{},[602],{"type":46,"value":603},"Do not use WebSearch - use WebFetch with llms.txt first",{"type":40,"tag":49,"props":605,"children":606},{},[607,612],{"type":40,"tag":65,"props":608,"children":609},{},[610],{"type":46,"value":611},"This applies to all steps",{"type":46,"value":613},", especially:",{"type":40,"tag":73,"props":615,"children":616},{},[617,622,627],{"type":40,"tag":77,"props":618,"children":619},{},[620],{"type":46,"value":621},"Dataset creation (read GenAI dataset docs from llms.txt)",{"type":40,"tag":77,"props":623,"children":624},{},[625],{"type":46,"value":626},"Scorer registration (check MLflow docs for scorer APIs)",{"type":40,"tag":77,"props":628,"children":629},{},[630],{"type":46,"value":631},"Evaluation execution (understand mlflow.genai.evaluate API)",{"type":40,"tag":55,"props":633,"children":635},{"id":634},"discovering-agent-structure",[636],{"type":46,"value":637},"Discovering Agent Structure",{"type":40,"tag":49,"props":639,"children":640},{},[641,646],{"type":40,"tag":65,"props":642,"children":643},{},[644],{"type":46,"value":645},"Each project has unique structure.",{"type":46,"value":647}," Use dynamic exploration instead of assumptions:",{"type":40,"tag":649,"props":650,"children":652},"h3",{"id":651},"find-agent-entry-points",[653],{"type":46,"value":654},"Find Agent Entry Points",{"type":40,"tag":356,"props":656,"children":658},{"className":358,"code":657,"language":360,"meta":361,"style":361},"# Search for main agent functions\ngrep -r \"def.*agent\" . --include=\"*.py\"\ngrep -r \"def (run|stream|handle|process)\" . --include=\"*.py\"\n\n# Check common locations\nls main.py app.py src\u002F*\u002Fagent.py 2>\u002Fdev\u002Fnull\n\n# Look for API routes\ngrep -r \"@app\\.(get|post)\" . --include=\"*.py\"  # FastAPI\u002FFlask\ngrep -r \"def.*route\" . --include=\"*.py\"\n",[659],{"type":40,"tag":88,"props":660,"children":661},{"__ignoreMap":361},[662,670,720,764,774,783,827,835,844,894],{"type":40,"tag":367,"props":663,"children":664},{"class":369,"line":370},[665],{"type":40,"tag":367,"props":666,"children":667},{"style":396},[668],{"type":46,"value":669},"# Search for main agent functions\n",{"type":40,"tag":367,"props":671,"children":672},{"class":369,"line":402},[673,678,683,687,692,696,701,706,710,715],{"type":40,"tag":367,"props":674,"children":675},{"style":374},[676],{"type":46,"value":677},"grep",{"type":40,"tag":367,"props":679,"children":680},{"style":380},[681],{"type":46,"value":682}," -r",{"type":40,"tag":367,"props":684,"children":685},{"style":450},[686],{"type":46,"value":453},{"type":40,"tag":367,"props":688,"children":689},{"style":380},[690],{"type":46,"value":691},"def.*agent",{"type":40,"tag":367,"props":693,"children":694},{"style":450},[695],{"type":46,"value":463},{"type":40,"tag":367,"props":697,"children":698},{"style":380},[699],{"type":46,"value":700}," .",{"type":40,"tag":367,"props":702,"children":703},{"style":380},[704],{"type":46,"value":705}," --include=",{"type":40,"tag":367,"props":707,"children":708},{"style":450},[709],{"type":46,"value":463},{"type":40,"tag":367,"props":711,"children":712},{"style":380},[713],{"type":46,"value":714},"*.py",{"type":40,"tag":367,"props":716,"children":717},{"style":450},[718],{"type":46,"value":719},"\"\n",{"type":40,"tag":367,"props":721,"children":722},{"class":369,"line":429},[723,727,731,735,740,744,748,752,756,760],{"type":40,"tag":367,"props":724,"children":725},{"style":374},[726],{"type":46,"value":677},{"type":40,"tag":367,"props":728,"children":729},{"style":380},[730],{"type":46,"value":682},{"type":40,"tag":367,"props":732,"children":733},{"style":450},[734],{"type":46,"value":453},{"type":40,"tag":367,"props":736,"children":737},{"style":380},[738],{"type":46,"value":739},"def (run|stream|handle|process)",{"type":40,"tag":367,"props":741,"children":742},{"style":450},[743],{"type":46,"value":463},{"type":40,"tag":367,"props":745,"children":746},{"style":380},[747],{"type":46,"value":700},{"type":40,"tag":367,"props":749,"children":750},{"style":380},[751],{"type":46,"value":705},{"type":40,"tag":367,"props":753,"children":754},{"style":450},[755],{"type":46,"value":463},{"type":40,"tag":367,"props":757,"children":758},{"style":380},[759],{"type":46,"value":714},{"type":40,"tag":367,"props":761,"children":762},{"style":450},[763],{"type":46,"value":719},{"type":40,"tag":367,"props":765,"children":767},{"class":369,"line":766},4,[768],{"type":40,"tag":367,"props":769,"children":771},{"emptyLinePlaceholder":770},true,[772],{"type":46,"value":773},"\n",{"type":40,"tag":367,"props":775,"children":777},{"class":369,"line":776},5,[778],{"type":40,"tag":367,"props":779,"children":780},{"style":396},[781],{"type":46,"value":782},"# Check common locations\n",{"type":40,"tag":367,"props":784,"children":786},{"class":369,"line":785},6,[787,792,797,802,807,813,818,822],{"type":40,"tag":367,"props":788,"children":789},{"style":374},[790],{"type":46,"value":791},"ls",{"type":40,"tag":367,"props":793,"children":794},{"style":380},[795],{"type":46,"value":796}," main.py",{"type":40,"tag":367,"props":798,"children":799},{"style":380},[800],{"type":46,"value":801}," app.py",{"type":40,"tag":367,"props":803,"children":804},{"style":380},[805],{"type":46,"value":806}," src\u002F",{"type":40,"tag":367,"props":808,"children":810},{"style":809},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[811],{"type":46,"value":812},"*",{"type":40,"tag":367,"props":814,"children":815},{"style":380},[816],{"type":46,"value":817},"\u002Fagent.py",{"type":40,"tag":367,"props":819,"children":820},{"style":450},[821],{"type":46,"value":556},{"type":40,"tag":367,"props":823,"children":824},{"style":380},[825],{"type":46,"value":826},"\u002Fdev\u002Fnull\n",{"type":40,"tag":367,"props":828,"children":830},{"class":369,"line":829},7,[831],{"type":40,"tag":367,"props":832,"children":833},{"emptyLinePlaceholder":770},[834],{"type":46,"value":773},{"type":40,"tag":367,"props":836,"children":838},{"class":369,"line":837},8,[839],{"type":40,"tag":367,"props":840,"children":841},{"style":396},[842],{"type":46,"value":843},"# Look for API routes\n",{"type":40,"tag":367,"props":845,"children":847},{"class":369,"line":846},9,[848,852,856,860,865,869,873,877,881,885,889],{"type":40,"tag":367,"props":849,"children":850},{"style":374},[851],{"type":46,"value":677},{"type":40,"tag":367,"props":853,"children":854},{"style":380},[855],{"type":46,"value":682},{"type":40,"tag":367,"props":857,"children":858},{"style":450},[859],{"type":46,"value":453},{"type":40,"tag":367,"props":861,"children":862},{"style":380},[863],{"type":46,"value":864},"@app\\.(get|post)",{"type":40,"tag":367,"props":866,"children":867},{"style":450},[868],{"type":46,"value":463},{"type":40,"tag":367,"props":870,"children":871},{"style":380},[872],{"type":46,"value":700},{"type":40,"tag":367,"props":874,"children":875},{"style":380},[876],{"type":46,"value":705},{"type":40,"tag":367,"props":878,"children":879},{"style":450},[880],{"type":46,"value":463},{"type":40,"tag":367,"props":882,"children":883},{"style":380},[884],{"type":46,"value":714},{"type":40,"tag":367,"props":886,"children":887},{"style":450},[888],{"type":46,"value":463},{"type":40,"tag":367,"props":890,"children":891},{"style":396},[892],{"type":46,"value":893},"  # FastAPI\u002FFlask\n",{"type":40,"tag":367,"props":895,"children":897},{"class":369,"line":896},10,[898,902,906,910,915,919,923,927,931,935],{"type":40,"tag":367,"props":899,"children":900},{"style":374},[901],{"type":46,"value":677},{"type":40,"tag":367,"props":903,"children":904},{"style":380},[905],{"type":46,"value":682},{"type":40,"tag":367,"props":907,"children":908},{"style":450},[909],{"type":46,"value":453},{"type":40,"tag":367,"props":911,"children":912},{"style":380},[913],{"type":46,"value":914},"def.*route",{"type":40,"tag":367,"props":916,"children":917},{"style":450},[918],{"type":46,"value":463},{"type":40,"tag":367,"props":920,"children":921},{"style":380},[922],{"type":46,"value":700},{"type":40,"tag":367,"props":924,"children":925},{"style":380},[926],{"type":46,"value":705},{"type":40,"tag":367,"props":928,"children":929},{"style":450},[930],{"type":46,"value":463},{"type":40,"tag":367,"props":932,"children":933},{"style":380},[934],{"type":46,"value":714},{"type":40,"tag":367,"props":936,"children":937},{"style":450},[938],{"type":46,"value":719},{"type":40,"tag":649,"props":940,"children":942},{"id":941},"understand-project-structure",[943],{"type":46,"value":944},"Understand Project Structure",{"type":40,"tag":356,"props":946,"children":948},{"className":358,"code":947,"language":360,"meta":361,"style":361},"# Check entry points in package config\ncat pyproject.toml setup.py 2>\u002Fdev\u002Fnull | grep -A 5 \"scripts\\|entry_points\"\n\n# Read project documentation\ncat README.md docs\u002F*.md 2>\u002Fdev\u002Fnull | head -100\n\n# Explore main directories\nls -la src\u002F app\u002F agent\u002F 2>\u002Fdev\u002Fnull\n",[949],{"type":40,"tag":88,"props":950,"children":951},{"__ignoreMap":361},[952,960,1021,1028,1036,1084,1091,1099],{"type":40,"tag":367,"props":953,"children":954},{"class":369,"line":370},[955],{"type":40,"tag":367,"props":956,"children":957},{"style":396},[958],{"type":46,"value":959},"# Check entry points in package config\n",{"type":40,"tag":367,"props":961,"children":962},{"class":369,"line":402},[963,968,973,978,982,987,992,997,1002,1008,1012,1017],{"type":40,"tag":367,"props":964,"children":965},{"style":374},[966],{"type":46,"value":967},"cat",{"type":40,"tag":367,"props":969,"children":970},{"style":380},[971],{"type":46,"value":972}," pyproject.toml",{"type":40,"tag":367,"props":974,"children":975},{"style":380},[976],{"type":46,"value":977}," setup.py",{"type":40,"tag":367,"props":979,"children":980},{"style":450},[981],{"type":46,"value":556},{"type":40,"tag":367,"props":983,"children":984},{"style":380},[985],{"type":46,"value":986},"\u002Fdev\u002Fnull",{"type":40,"tag":367,"props":988,"children":989},{"style":450},[990],{"type":46,"value":991}," |",{"type":40,"tag":367,"props":993,"children":994},{"style":374},[995],{"type":46,"value":996}," grep",{"type":40,"tag":367,"props":998,"children":999},{"style":380},[1000],{"type":46,"value":1001}," -A",{"type":40,"tag":367,"props":1003,"children":1005},{"style":1004},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1006],{"type":46,"value":1007}," 5",{"type":40,"tag":367,"props":1009,"children":1010},{"style":450},[1011],{"type":46,"value":453},{"type":40,"tag":367,"props":1013,"children":1014},{"style":380},[1015],{"type":46,"value":1016},"scripts\\|entry_points",{"type":40,"tag":367,"props":1018,"children":1019},{"style":450},[1020],{"type":46,"value":719},{"type":40,"tag":367,"props":1022,"children":1023},{"class":369,"line":429},[1024],{"type":40,"tag":367,"props":1025,"children":1026},{"emptyLinePlaceholder":770},[1027],{"type":46,"value":773},{"type":40,"tag":367,"props":1029,"children":1030},{"class":369,"line":766},[1031],{"type":40,"tag":367,"props":1032,"children":1033},{"style":396},[1034],{"type":46,"value":1035},"# Read project documentation\n",{"type":40,"tag":367,"props":1037,"children":1038},{"class":369,"line":776},[1039,1043,1048,1053,1057,1062,1066,1070,1074,1079],{"type":40,"tag":367,"props":1040,"children":1041},{"style":374},[1042],{"type":46,"value":967},{"type":40,"tag":367,"props":1044,"children":1045},{"style":380},[1046],{"type":46,"value":1047}," README.md",{"type":40,"tag":367,"props":1049,"children":1050},{"style":380},[1051],{"type":46,"value":1052}," docs\u002F",{"type":40,"tag":367,"props":1054,"children":1055},{"style":809},[1056],{"type":46,"value":812},{"type":40,"tag":367,"props":1058,"children":1059},{"style":380},[1060],{"type":46,"value":1061},".md",{"type":40,"tag":367,"props":1063,"children":1064},{"style":450},[1065],{"type":46,"value":556},{"type":40,"tag":367,"props":1067,"children":1068},{"style":380},[1069],{"type":46,"value":986},{"type":40,"tag":367,"props":1071,"children":1072},{"style":450},[1073],{"type":46,"value":991},{"type":40,"tag":367,"props":1075,"children":1076},{"style":374},[1077],{"type":46,"value":1078}," head",{"type":40,"tag":367,"props":1080,"children":1081},{"style":380},[1082],{"type":46,"value":1083}," -100\n",{"type":40,"tag":367,"props":1085,"children":1086},{"class":369,"line":785},[1087],{"type":40,"tag":367,"props":1088,"children":1089},{"emptyLinePlaceholder":770},[1090],{"type":46,"value":773},{"type":40,"tag":367,"props":1092,"children":1093},{"class":369,"line":829},[1094],{"type":40,"tag":367,"props":1095,"children":1096},{"style":396},[1097],{"type":46,"value":1098},"# Explore main directories\n",{"type":40,"tag":367,"props":1100,"children":1101},{"class":369,"line":837},[1102,1106,1111,1115,1120,1125,1129],{"type":40,"tag":367,"props":1103,"children":1104},{"style":374},[1105],{"type":46,"value":791},{"type":40,"tag":367,"props":1107,"children":1108},{"style":380},[1109],{"type":46,"value":1110}," -la",{"type":40,"tag":367,"props":1112,"children":1113},{"style":380},[1114],{"type":46,"value":806},{"type":40,"tag":367,"props":1116,"children":1117},{"style":380},[1118],{"type":46,"value":1119}," app\u002F",{"type":40,"tag":367,"props":1121,"children":1122},{"style":380},[1123],{"type":46,"value":1124}," agent\u002F",{"type":40,"tag":367,"props":1126,"children":1127},{"style":450},[1128],{"type":46,"value":556},{"type":40,"tag":367,"props":1130,"children":1131},{"style":380},[1132],{"type":46,"value":826},{"type":40,"tag":55,"props":1134,"children":1136},{"id":1135},"setup-overview",[1137],{"type":46,"value":232},{"type":40,"tag":649,"props":1139,"children":1141},{"id":1140},"pre-check-use-existing-environment",[1142],{"type":46,"value":1143},"Pre-check: Use Existing Environment",{"type":40,"tag":49,"props":1145,"children":1146},{},[1147],{"type":40,"tag":65,"props":1148,"children":1149},{},[1150,1152,1158,1159,1165],{"type":46,"value":1151},"Before doing ANY setup, check if ",{"type":40,"tag":88,"props":1153,"children":1155},{"className":1154},[],[1156],{"type":46,"value":1157},"MLFLOW_TRACKING_URI",{"type":46,"value":112},{"type":40,"tag":88,"props":1160,"children":1162},{"className":1161},[],[1163],{"type":46,"value":1164},"MLFLOW_EXPERIMENT_ID",{"type":46,"value":1166}," are already set:",{"type":40,"tag":356,"props":1168,"children":1170},{"className":358,"code":1169,"language":360,"meta":361,"style":361},"echo \"MLFLOW_TRACKING_URI=$MLFLOW_TRACKING_URI\"\necho \"MLFLOW_EXPERIMENT_ID=$MLFLOW_EXPERIMENT_ID\"\n",[1171],{"type":40,"tag":88,"props":1172,"children":1173},{"__ignoreMap":361},[1174,1201],{"type":40,"tag":367,"props":1175,"children":1176},{"class":369,"line":370},[1177,1183,1187,1192,1197],{"type":40,"tag":367,"props":1178,"children":1180},{"style":1179},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[1181],{"type":46,"value":1182},"echo",{"type":40,"tag":367,"props":1184,"children":1185},{"style":450},[1186],{"type":46,"value":453},{"type":40,"tag":367,"props":1188,"children":1189},{"style":380},[1190],{"type":46,"value":1191},"MLFLOW_TRACKING_URI=",{"type":40,"tag":367,"props":1193,"children":1194},{"style":809},[1195],{"type":46,"value":1196},"$MLFLOW_TRACKING_URI",{"type":40,"tag":367,"props":1198,"children":1199},{"style":450},[1200],{"type":46,"value":719},{"type":40,"tag":367,"props":1202,"children":1203},{"class":369,"line":402},[1204,1208,1212,1217,1222],{"type":40,"tag":367,"props":1205,"children":1206},{"style":1179},[1207],{"type":46,"value":1182},{"type":40,"tag":367,"props":1209,"children":1210},{"style":450},[1211],{"type":46,"value":453},{"type":40,"tag":367,"props":1213,"children":1214},{"style":380},[1215],{"type":46,"value":1216},"MLFLOW_EXPERIMENT_ID=",{"type":40,"tag":367,"props":1218,"children":1219},{"style":809},[1220],{"type":46,"value":1221},"$MLFLOW_EXPERIMENT_ID",{"type":40,"tag":367,"props":1223,"children":1224},{"style":450},[1225],{"type":46,"value":719},{"type":40,"tag":49,"props":1227,"children":1228},{},[1229,1234,1236,1242,1244,1250],{"type":40,"tag":65,"props":1230,"children":1231},{},[1232],{"type":46,"value":1233},"If BOTH are already set, skip Steps 1-2 entirely.",{"type":46,"value":1235}," The environment is pre-configured. Do NOT run ",{"type":40,"tag":88,"props":1237,"children":1239},{"className":1238},[],[1240],{"type":46,"value":1241},"setup_mlflow.py",{"type":46,"value":1243},", do NOT create a ",{"type":40,"tag":88,"props":1245,"children":1247},{"className":1246},[],[1248],{"type":46,"value":1249},".env",{"type":46,"value":1251}," file, do NOT override these values. Go directly to Step 3 (tracing integration) and the evaluation workflow.",{"type":40,"tag":649,"props":1253,"children":1255},{"id":1254},"setup-steps-only-if-environment-is-not-pre-configured",[1256],{"type":46,"value":1257},"Setup Steps (only if environment is NOT pre-configured)",{"type":40,"tag":202,"props":1259,"children":1260},{},[1261,1271,1302],{"type":40,"tag":77,"props":1262,"children":1263},{},[1264,1269],{"type":40,"tag":65,"props":1265,"children":1266},{},[1267],{"type":46,"value":1268},"Install MLflow",{"type":46,"value":1270}," (version >=3.8.0)",{"type":40,"tag":77,"props":1272,"children":1273},{},[1274,1279,1281],{"type":40,"tag":65,"props":1275,"children":1276},{},[1277],{"type":46,"value":1278},"Configure environment",{"type":46,"value":1280}," (tracking URI and experiment)\n",{"type":40,"tag":73,"props":1282,"children":1283},{},[1284],{"type":40,"tag":77,"props":1285,"children":1286},{},[1287,1292,1294,1300],{"type":40,"tag":65,"props":1288,"children":1289},{},[1290],{"type":46,"value":1291},"Guide",{"type":46,"value":1293},": Follow ",{"type":40,"tag":88,"props":1295,"children":1297},{"className":1296},[],[1298],{"type":46,"value":1299},"references\u002Fsetup-guide.md",{"type":46,"value":1301}," Steps 1-2",{"type":40,"tag":77,"props":1303,"children":1304},{},[1305,1310,1312],{"type":40,"tag":65,"props":1306,"children":1307},{},[1308],{"type":46,"value":1309},"Integrate tracing",{"type":46,"value":1311}," (autolog and @mlflow.trace decorators)\n",{"type":40,"tag":73,"props":1313,"children":1314},{},[1315,1335],{"type":40,"tag":77,"props":1316,"children":1317},{},[1318,1320,1325,1327,1333],{"type":46,"value":1319},"⚠️ ",{"type":40,"tag":65,"props":1321,"children":1322},{},[1323],{"type":46,"value":1324},"MANDATORY",{"type":46,"value":1326},": Use the ",{"type":40,"tag":88,"props":1328,"children":1330},{"className":1329},[],[1331],{"type":46,"value":1332},"instrumenting-with-mlflow-tracing",{"type":46,"value":1334}," skill for tracing setup",{"type":40,"tag":77,"props":1336,"children":1337},{},[1338,1340,1345,1347,1353],{"type":46,"value":1339},"✓ ",{"type":40,"tag":65,"props":1341,"children":1342},{},[1343],{"type":46,"value":1344},"VERIFY",{"type":46,"value":1346},": Run ",{"type":40,"tag":88,"props":1348,"children":1350},{"className":1349},[],[1351],{"type":46,"value":1352},"scripts\u002Fvalidate_tracing_runtime.py",{"type":46,"value":1354}," after implementing",{"type":40,"tag":49,"props":1356,"children":1357},{},[1358,1359,1364],{"type":46,"value":1319},{"type":40,"tag":65,"props":1360,"children":1361},{},[1362],{"type":46,"value":1363},"Tracing must work before evaluation.",{"type":46,"value":1365}," If tracing fails, stop and troubleshoot.",{"type":40,"tag":49,"props":1367,"children":1368},{},[1369],{"type":40,"tag":65,"props":1370,"children":1371},{},[1372],{"type":46,"value":1373},"Checkpoint - verify before proceeding:",{"type":40,"tag":73,"props":1375,"children":1378},{"className":1376},[1377],"contains-task-list",[1379,1391,1400,1409],{"type":40,"tag":77,"props":1380,"children":1383},{"className":1381},[1382],"task-list-item",[1384,1389],{"type":40,"tag":1385,"props":1386,"children":1388},"input",{"disabled":770,"type":1387},"checkbox",[],{"type":46,"value":1390}," MLflow >=3.8.0 installed",{"type":40,"tag":77,"props":1392,"children":1394},{"className":1393},[1382],[1395,1398],{"type":40,"tag":1385,"props":1396,"children":1397},{"disabled":770,"type":1387},[],{"type":46,"value":1399}," MLFLOW_TRACKING_URI and MLFLOW_EXPERIMENT_ID set",{"type":40,"tag":77,"props":1401,"children":1403},{"className":1402},[1382],[1404,1407],{"type":40,"tag":1385,"props":1405,"children":1406},{"disabled":770,"type":1387},[],{"type":46,"value":1408}," Autolog enabled and @mlflow.trace decorators added",{"type":40,"tag":77,"props":1410,"children":1412},{"className":1411},[1382],[1413,1416],{"type":40,"tag":1385,"props":1414,"children":1415},{"disabled":770,"type":1387},[],{"type":46,"value":1417}," Test run creates a trace (verify trace ID is not None)",{"type":40,"tag":49,"props":1419,"children":1420},{},[1421],{"type":40,"tag":65,"props":1422,"children":1423},{},[1424],{"type":46,"value":1425},"Validation scripts:",{"type":40,"tag":356,"props":1427,"children":1429},{"className":358,"code":1428,"language":360,"meta":361,"style":361},"uv run python scripts\u002Fvalidate_environment.py  # Check MLflow install, env vars, connectivity\nuv run python scripts\u002Fvalidate_auth.py         # Test authentication before expensive operations\n",[1430],{"type":40,"tag":88,"props":1431,"children":1432},{"__ignoreMap":361},[1433,1458],{"type":40,"tag":367,"props":1434,"children":1435},{"class":369,"line":370},[1436,1440,1444,1448,1453],{"type":40,"tag":367,"props":1437,"children":1438},{"style":374},[1439],{"type":46,"value":377},{"type":40,"tag":367,"props":1441,"children":1442},{"style":380},[1443],{"type":46,"value":383},{"type":40,"tag":367,"props":1445,"children":1446},{"style":380},[1447],{"type":46,"value":416},{"type":40,"tag":367,"props":1449,"children":1450},{"style":380},[1451],{"type":46,"value":1452}," scripts\u002Fvalidate_environment.py",{"type":40,"tag":367,"props":1454,"children":1455},{"style":396},[1456],{"type":46,"value":1457},"  # Check MLflow install, env vars, connectivity\n",{"type":40,"tag":367,"props":1459,"children":1460},{"class":369,"line":402},[1461,1465,1469,1473,1478],{"type":40,"tag":367,"props":1462,"children":1463},{"style":374},[1464],{"type":46,"value":377},{"type":40,"tag":367,"props":1466,"children":1467},{"style":380},[1468],{"type":46,"value":383},{"type":40,"tag":367,"props":1470,"children":1471},{"style":380},[1472],{"type":46,"value":416},{"type":40,"tag":367,"props":1474,"children":1475},{"style":380},[1476],{"type":46,"value":1477}," scripts\u002Fvalidate_auth.py",{"type":40,"tag":367,"props":1479,"children":1480},{"style":396},[1481],{"type":46,"value":1482},"         # Test authentication before expensive operations\n",{"type":40,"tag":55,"props":1484,"children":1486},{"id":1485},"evaluation-workflow",[1487],{"type":46,"value":241},{"type":40,"tag":649,"props":1489,"children":1491},{"id":1490},"step-1-agent-interview-required-do-not-skip",[1492],{"type":46,"value":1493},"Step 1: Agent Interview (REQUIRED — do not skip)",{"type":40,"tag":49,"props":1495,"children":1496},{},[1497],{"type":46,"value":1498},"Before doing anything else, ask the user these questions. Do NOT proceed until you have answers.",{"type":40,"tag":49,"props":1500,"children":1501},{},[1502],{"type":40,"tag":65,"props":1503,"children":1504},{},[1505],{"type":46,"value":1506},"Required:",{"type":40,"tag":202,"props":1508,"children":1509},{},[1510,1515,1520],{"type":40,"tag":77,"props":1511,"children":1512},{},[1513],{"type":46,"value":1514},"\"What does your agent do? Describe its purpose in 1-2 sentences.\"",{"type":40,"tag":77,"props":1516,"children":1517},{},[1518],{"type":46,"value":1519},"\"What are the 2-3 most important things it needs to get right?\"",{"type":40,"tag":77,"props":1521,"children":1522},{},[1523],{"type":46,"value":1524},"\"Are there common failure modes you've already noticed?\"",{"type":40,"tag":49,"props":1526,"children":1527},{},[1528],{"type":40,"tag":65,"props":1529,"children":1530},{},[1531],{"type":46,"value":1532},"Use answers to:",{"type":40,"tag":73,"props":1534,"children":1535},{},[1536,1541,1560],{"type":40,"tag":77,"props":1537,"children":1538},{},[1539],{"type":46,"value":1540},"Derive scorer names and criteria (do not invent them)",{"type":40,"tag":77,"props":1542,"children":1543},{},[1544,1546,1552,1554],{"type":46,"value":1545},"Write the ",{"type":40,"tag":88,"props":1547,"children":1549},{"className":1548},[],[1550],{"type":46,"value":1551},"agent_description",{"type":46,"value":1553}," parameter for ",{"type":40,"tag":88,"props":1555,"children":1557},{"className":1556},[],[1558],{"type":46,"value":1559},"generate_evals_df",{"type":40,"tag":77,"props":1561,"children":1562},{},[1563],{"type":46,"value":1564},"Set evaluation priorities",{"type":40,"tag":49,"props":1566,"children":1567},{},[1568,1573],{"type":40,"tag":65,"props":1569,"children":1570},{},[1571],{"type":46,"value":1572},"If running in automated mode:",{"type":46,"value":1574}," Read agent purpose from the codebase (SKILL.md, README, or main entry point docstring). Still surface what you found and confirm before proceeding.",{"type":40,"tag":649,"props":1576,"children":1578},{"id":1577},"step-2-define-quality-scorers",[1579],{"type":46,"value":1580},"Step 2: Define Quality Scorers",{"type":40,"tag":202,"props":1582,"children":1583},{},[1584],{"type":40,"tag":77,"props":1585,"children":1586},{},[1587,1592],{"type":40,"tag":65,"props":1588,"children":1589},{},[1590],{"type":46,"value":1591},"Check registered scorers in your experiment:",{"type":40,"tag":356,"props":1593,"children":1595},{"className":358,"code":1594,"language":360,"meta":361,"style":361},"uv run mlflow scorers list -x $MLFLOW_EXPERIMENT_ID\n",[1596],{"type":40,"tag":88,"props":1597,"children":1598},{"__ignoreMap":361},[1599],{"type":40,"tag":367,"props":1600,"children":1601},{"class":369,"line":370},[1602,1606,1610,1614,1619,1624,1629],{"type":40,"tag":367,"props":1603,"children":1604},{"style":374},[1605],{"type":46,"value":377},{"type":40,"tag":367,"props":1607,"children":1608},{"style":380},[1609],{"type":46,"value":383},{"type":40,"tag":367,"props":1611,"children":1612},{"style":380},[1613],{"type":46,"value":388},{"type":40,"tag":367,"props":1615,"children":1616},{"style":380},[1617],{"type":46,"value":1618}," scorers",{"type":40,"tag":367,"props":1620,"children":1621},{"style":380},[1622],{"type":46,"value":1623}," list",{"type":40,"tag":367,"props":1625,"children":1626},{"style":380},[1627],{"type":46,"value":1628}," -x",{"type":40,"tag":367,"props":1630,"children":1631},{"style":809},[1632],{"type":46,"value":1633}," $MLFLOW_EXPERIMENT_ID\n",{"type":40,"tag":49,"props":1635,"children":1636},{},[1637],{"type":40,"tag":65,"props":1638,"children":1639},{},[1640],{"type":46,"value":1641},"IMPORTANT: if there are registered scorers in the experiment then they must be used for evaluation.",{"type":40,"tag":202,"props":1643,"children":1644},{"start":402},[1645],{"type":40,"tag":77,"props":1646,"children":1647},{},[1648],{"type":40,"tag":65,"props":1649,"children":1650},{},[1651],{"type":46,"value":1652},"Select additional built-in scorers that apply to the agent",{"type":40,"tag":49,"props":1654,"children":1655},{},[1656,1658,1664],{"type":46,"value":1657},"See ",{"type":40,"tag":88,"props":1659,"children":1661},{"className":1660},[],[1662],{"type":46,"value":1663},"references\u002Fscorers.md",{"type":46,"value":1665}," for the built-in scorers. Select any that are useful for assessing the agent's quality and that are not already registered.",{"type":40,"tag":202,"props":1667,"children":1668},{"start":429},[1669],{"type":40,"tag":77,"props":1670,"children":1671},{},[1672],{"type":40,"tag":65,"props":1673,"children":1674},{},[1675],{"type":46,"value":1676},"Create additional custom scorers as needed",{"type":40,"tag":49,"props":1678,"children":1679},{},[1680,1682,1688,1690,1695,1697,1703],{"type":46,"value":1681},"If needed, create additional scorers using the ",{"type":40,"tag":88,"props":1683,"children":1685},{"className":1684},[],[1686],{"type":46,"value":1687},"make_judge()",{"type":46,"value":1689}," API. See ",{"type":40,"tag":88,"props":1691,"children":1693},{"className":1692},[],[1694],{"type":46,"value":1663},{"type":46,"value":1696}," on how to create custom scorers and ",{"type":40,"tag":88,"props":1698,"children":1700},{"className":1699},[],[1701],{"type":46,"value":1702},"references\u002Fscorers-constraints.md",{"type":46,"value":1704}," for best practices.",{"type":40,"tag":1706,"props":1707,"children":1708},"blockquote",{},[1709],{"type":40,"tag":49,"props":1710,"children":1711},{},[1712,1713,1718,1720,1726,1728,1734,1736,1742,1743,1749,1751,1762,1764,1770,1771,1782,1784,1789],{"type":46,"value":1319},{"type":40,"tag":65,"props":1714,"children":1715},{},[1716],{"type":46,"value":1717},"CRITICAL — Scorer Return Values:",{"type":46,"value":1719}," Scorers MUST instruct the LLM judge to return ",{"type":40,"tag":88,"props":1721,"children":1723},{"className":1722},[],[1724],{"type":46,"value":1725},"\"yes\"",{"type":46,"value":1727}," or ",{"type":40,"tag":88,"props":1729,"children":1731},{"className":1730},[],[1732],{"type":46,"value":1733},"\"no\"",{"type":46,"value":1735}," (or booleans\u002Fnumerics). Return values of ",{"type":40,"tag":88,"props":1737,"children":1739},{"className":1738},[],[1740],{"type":46,"value":1741},"\"pass\"",{"type":46,"value":1727},{"type":40,"tag":88,"props":1744,"children":1746},{"className":1745},[],[1747],{"type":46,"value":1748},"\"fail\"",{"type":46,"value":1750}," are ",{"type":40,"tag":65,"props":1752,"children":1753},{},[1754,1756],{"type":46,"value":1755},"silently cast to ",{"type":40,"tag":88,"props":1757,"children":1759},{"className":1758},[],[1760],{"type":46,"value":1761},"None",{"type":46,"value":1763}," by ",{"type":40,"tag":88,"props":1765,"children":1767},{"className":1766},[],[1768],{"type":46,"value":1769},"_cast_assessment_value_to_float",{"type":46,"value":112},{"type":40,"tag":65,"props":1772,"children":1773},{},[1774,1776],{"type":46,"value":1775},"excluded from ",{"type":40,"tag":88,"props":1777,"children":1779},{"className":1778},[],[1780],{"type":46,"value":1781},"results.metrics",{"type":46,"value":1783}," with no error or warning — results simply disappear. See ",{"type":40,"tag":88,"props":1785,"children":1787},{"className":1786},[],[1788],{"type":46,"value":1702},{"type":46,"value":1790}," Constraint 2 for the full list of safe vs. broken return values.",{"type":40,"tag":202,"props":1792,"children":1793},{"start":766},[1794],{"type":40,"tag":77,"props":1795,"children":1796},{},[1797,1802,1804],{"type":40,"tag":65,"props":1798,"children":1799},{},[1800],{"type":46,"value":1801},"REQUIRED: Register new scorers before evaluation",{"type":46,"value":1803}," using Python API:",{"type":40,"tag":356,"props":1805,"children":1809},{"className":1806,"code":1807,"language":1808,"meta":361,"style":361},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","from mlflow.genai.judges import make_judge\nfrom mlflow.genai.scorers import BuiltinScorerName\nimport os\n\nscorer = make_judge(...)  # Or, scorer = BuiltinScorerName()\nscorer.register()\n","python",[1810],{"type":40,"tag":88,"props":1811,"children":1812},{"__ignoreMap":361},[1813,1821,1829,1837,1844,1852],{"type":40,"tag":367,"props":1814,"children":1815},{"class":369,"line":370},[1816],{"type":40,"tag":367,"props":1817,"children":1818},{},[1819],{"type":46,"value":1820},"from mlflow.genai.judges import make_judge\n",{"type":40,"tag":367,"props":1822,"children":1823},{"class":369,"line":402},[1824],{"type":40,"tag":367,"props":1825,"children":1826},{},[1827],{"type":46,"value":1828},"from mlflow.genai.scorers import BuiltinScorerName\n",{"type":40,"tag":367,"props":1830,"children":1831},{"class":369,"line":429},[1832],{"type":40,"tag":367,"props":1833,"children":1834},{},[1835],{"type":46,"value":1836},"import os\n",{"type":40,"tag":367,"props":1838,"children":1839},{"class":369,"line":766},[1840],{"type":40,"tag":367,"props":1841,"children":1842},{"emptyLinePlaceholder":770},[1843],{"type":46,"value":773},{"type":40,"tag":367,"props":1845,"children":1846},{"class":369,"line":776},[1847],{"type":40,"tag":367,"props":1848,"children":1849},{},[1850],{"type":46,"value":1851},"scorer = make_judge(...)  # Or, scorer = BuiltinScorerName()\n",{"type":40,"tag":367,"props":1853,"children":1854},{"class":369,"line":785},[1855],{"type":40,"tag":367,"props":1856,"children":1857},{},[1858],{"type":46,"value":1859},"scorer.register()\n",{"type":40,"tag":49,"props":1861,"children":1862},{},[1863,1865,1870,1872,1878],{"type":46,"value":1864},"** IMPORTANT:  See ",{"type":40,"tag":88,"props":1866,"children":1868},{"className":1867},[],[1869],{"type":46,"value":1663},{"type":46,"value":1871}," → \"Model Selection for Scorers\" to configure the ",{"type":40,"tag":88,"props":1873,"children":1875},{"className":1874},[],[1876],{"type":46,"value":1877},"model",{"type":46,"value":1879}," parameter of scorers before registration.",{"type":40,"tag":49,"props":1881,"children":1882},{},[1883,1884,1889,1891,1897],{"type":46,"value":1319},{"type":40,"tag":65,"props":1885,"children":1886},{},[1887],{"type":46,"value":1888},"Scorers MUST be registered before evaluation.",{"type":46,"value":1890}," Inline scorers that aren't registered won't appear in ",{"type":40,"tag":88,"props":1892,"children":1894},{"className":1893},[],[1895],{"type":46,"value":1896},"mlflow scorers list",{"type":46,"value":1898}," and won't be reusable.",{"type":40,"tag":202,"props":1900,"children":1901},{"start":776},[1902],{"type":40,"tag":77,"props":1903,"children":1904},{},[1905,1910],{"type":40,"tag":65,"props":1906,"children":1907},{},[1908],{"type":46,"value":1909},"Verify registration:",{"type":40,"tag":356,"props":1911,"children":1913},{"className":358,"code":1912,"language":360,"meta":361,"style":361},"uv run mlflow scorers list -x $MLFLOW_EXPERIMENT_ID  # Should show your scorers\n",[1914],{"type":40,"tag":88,"props":1915,"children":1916},{"__ignoreMap":361},[1917],{"type":40,"tag":367,"props":1918,"children":1919},{"class":369,"line":370},[1920,1924,1928,1932,1936,1940,1944,1949],{"type":40,"tag":367,"props":1921,"children":1922},{"style":374},[1923],{"type":46,"value":377},{"type":40,"tag":367,"props":1925,"children":1926},{"style":380},[1927],{"type":46,"value":383},{"type":40,"tag":367,"props":1929,"children":1930},{"style":380},[1931],{"type":46,"value":388},{"type":40,"tag":367,"props":1933,"children":1934},{"style":380},[1935],{"type":46,"value":1618},{"type":40,"tag":367,"props":1937,"children":1938},{"style":380},[1939],{"type":46,"value":1623},{"type":40,"tag":367,"props":1941,"children":1942},{"style":380},[1943],{"type":46,"value":1628},{"type":40,"tag":367,"props":1945,"children":1946},{"style":809},[1947],{"type":46,"value":1948}," $MLFLOW_EXPERIMENT_ID  ",{"type":40,"tag":367,"props":1950,"children":1951},{"style":396},[1952],{"type":46,"value":1953},"# Should show your scorers\n",{"type":40,"tag":649,"props":1955,"children":1957},{"id":1956},"step-3-prepare-evaluation-dataset",[1958],{"type":46,"value":1959},"Step 3: Prepare Evaluation Dataset",{"type":40,"tag":49,"props":1961,"children":1962},{},[1963,1968],{"type":40,"tag":65,"props":1964,"children":1965},{},[1966],{"type":46,"value":1967},"ALWAYS discover existing datasets first",{"type":46,"value":1969}," to prevent duplicate work:",{"type":40,"tag":202,"props":1971,"children":1972},{},[1973,2077,2100],{"type":40,"tag":77,"props":1974,"children":1975},{},[1976,1981,1983],{"type":40,"tag":65,"props":1977,"children":1978},{},[1979],{"type":46,"value":1980},"Run dataset discovery",{"type":46,"value":1982}," (mandatory):",{"type":40,"tag":356,"props":1984,"children":1986},{"className":358,"code":1985,"language":360,"meta":361,"style":361},"uv run python scripts\u002Flist_datasets.py  # Lists, compares, recommends datasets\nuv run python scripts\u002Flist_datasets.py --format json  # Machine-readable output\nuv run python scripts\u002Flist_datasets.py --help  # All options\n",[1987],{"type":40,"tag":88,"props":1988,"children":1989},{"__ignoreMap":361},[1990,2015,2048],{"type":40,"tag":367,"props":1991,"children":1992},{"class":369,"line":370},[1993,1997,2001,2005,2010],{"type":40,"tag":367,"props":1994,"children":1995},{"style":374},[1996],{"type":46,"value":377},{"type":40,"tag":367,"props":1998,"children":1999},{"style":380},[2000],{"type":46,"value":383},{"type":40,"tag":367,"props":2002,"children":2003},{"style":380},[2004],{"type":46,"value":416},{"type":40,"tag":367,"props":2006,"children":2007},{"style":380},[2008],{"type":46,"value":2009}," scripts\u002Flist_datasets.py",{"type":40,"tag":367,"props":2011,"children":2012},{"style":396},[2013],{"type":46,"value":2014},"  # Lists, compares, recommends datasets\n",{"type":40,"tag":367,"props":2016,"children":2017},{"class":369,"line":402},[2018,2022,2026,2030,2034,2039,2043],{"type":40,"tag":367,"props":2019,"children":2020},{"style":374},[2021],{"type":46,"value":377},{"type":40,"tag":367,"props":2023,"children":2024},{"style":380},[2025],{"type":46,"value":383},{"type":40,"tag":367,"props":2027,"children":2028},{"style":380},[2029],{"type":46,"value":416},{"type":40,"tag":367,"props":2031,"children":2032},{"style":380},[2033],{"type":46,"value":2009},{"type":40,"tag":367,"props":2035,"children":2036},{"style":380},[2037],{"type":46,"value":2038}," --format",{"type":40,"tag":367,"props":2040,"children":2041},{"style":380},[2042],{"type":46,"value":541},{"type":40,"tag":367,"props":2044,"children":2045},{"style":396},[2046],{"type":46,"value":2047},"  # Machine-readable output\n",{"type":40,"tag":367,"props":2049,"children":2050},{"class":369,"line":429},[2051,2055,2059,2063,2067,2072],{"type":40,"tag":367,"props":2052,"children":2053},{"style":374},[2054],{"type":46,"value":377},{"type":40,"tag":367,"props":2056,"children":2057},{"style":380},[2058],{"type":46,"value":383},{"type":40,"tag":367,"props":2060,"children":2061},{"style":380},[2062],{"type":46,"value":416},{"type":40,"tag":367,"props":2064,"children":2065},{"style":380},[2066],{"type":46,"value":2009},{"type":40,"tag":367,"props":2068,"children":2069},{"style":380},[2070],{"type":46,"value":2071}," --help",{"type":40,"tag":367,"props":2073,"children":2074},{"style":396},[2075],{"type":46,"value":2076},"  # All options\n",{"type":40,"tag":77,"props":2078,"children":2079},{},[2080,2085,2087],{"type":40,"tag":65,"props":2081,"children":2082},{},[2083],{"type":46,"value":2084},"Present findings to user",{"type":46,"value":2086},":",{"type":40,"tag":73,"props":2088,"children":2089},{},[2090,2095],{"type":40,"tag":77,"props":2091,"children":2092},{},[2093],{"type":46,"value":2094},"Show all discovered datasets with their characteristics (size, topics covered)",{"type":40,"tag":77,"props":2096,"children":2097},{},[2098],{"type":46,"value":2099},"If datasets found, highlight most relevant options based on agent type",{"type":40,"tag":77,"props":2101,"children":2102},{},[2103,2108,2109],{"type":40,"tag":65,"props":2104,"children":2105},{},[2106],{"type":46,"value":2107},"Ask user about existing datasets",{"type":46,"value":2086},{"type":40,"tag":73,"props":2110,"children":2111},{},[2112,2124,2129],{"type":40,"tag":77,"props":2113,"children":2114},{},[2115,2117,2122],{"type":46,"value":2116},"\"I found ",{"type":40,"tag":367,"props":2118,"children":2119},{},[2120],{"type":46,"value":2121},"N",{"type":46,"value":2123}," existing evaluation dataset(s). Do you want to use one of these? (y\u002Fn)\"",{"type":40,"tag":77,"props":2125,"children":2126},{},[2127],{"type":46,"value":2128},"If yes: Ask which dataset to use and record the dataset name — skip to Step 3.5",{"type":40,"tag":77,"props":2130,"children":2131},{},[2132],{"type":46,"value":2133},"If no: Proceed to Phase A below",{"type":40,"tag":49,"props":2135,"children":2136},{},[2137],{"type":40,"tag":65,"props":2138,"children":2139},{},[2140],{"type":46,"value":2141},"If creating a new dataset, use the two-phase approach below.",{"type":40,"tag":2143,"props":2144,"children":2145},"hr",{},[],{"type":40,"tag":2147,"props":2148,"children":2150},"h4",{"id":2149},"phase-a-sanity-check-5-questions-always-run-first",[2151],{"type":46,"value":2152},"Phase A: Sanity Check (5 questions — always run first)",{"type":40,"tag":49,"props":2154,"children":2155},{},[2156],{"type":46,"value":2157},"Create a minimal 5-question dataset manually from the Step 1 interview answers. The goal is to confirm the pipeline works end-to-end before investing in large-scale generation.",{"type":40,"tag":356,"props":2159,"children":2161},{"className":1806,"code":2160,"language":1808,"meta":361,"style":361},"import mlflow\nfrom mlflow.genai.datasets import create_dataset\n\n# Derive 5 representative questions directly from the agent's stated purpose\n# and known failure modes identified in Step 1\nsanity_records = [\n    {\"inputs\": {\"query\": \"\u003Cquestion 1 from interview>\"}, \"expected_response\": \"\u003Cexpected answer>\"},\n    {\"inputs\": {\"query\": \"\u003Cquestion 2 from interview>\"}, \"expected_response\": \"\u003Cexpected answer>\"},\n    # ... 5 total\n]\n\nsanity_dataset = create_dataset(\n    records=sanity_records,\n    name=\"sanity-check-5q\",\n)\n",[2162],{"type":40,"tag":88,"props":2163,"children":2164},{"__ignoreMap":361},[2165,2173,2181,2188,2196,2204,2212,2220,2228,2236,2244,2252,2261,2270,2279],{"type":40,"tag":367,"props":2166,"children":2167},{"class":369,"line":370},[2168],{"type":40,"tag":367,"props":2169,"children":2170},{},[2171],{"type":46,"value":2172},"import mlflow\n",{"type":40,"tag":367,"props":2174,"children":2175},{"class":369,"line":402},[2176],{"type":40,"tag":367,"props":2177,"children":2178},{},[2179],{"type":46,"value":2180},"from mlflow.genai.datasets import create_dataset\n",{"type":40,"tag":367,"props":2182,"children":2183},{"class":369,"line":429},[2184],{"type":40,"tag":367,"props":2185,"children":2186},{"emptyLinePlaceholder":770},[2187],{"type":46,"value":773},{"type":40,"tag":367,"props":2189,"children":2190},{"class":369,"line":766},[2191],{"type":40,"tag":367,"props":2192,"children":2193},{},[2194],{"type":46,"value":2195},"# Derive 5 representative questions directly from the agent's stated purpose\n",{"type":40,"tag":367,"props":2197,"children":2198},{"class":369,"line":776},[2199],{"type":40,"tag":367,"props":2200,"children":2201},{},[2202],{"type":46,"value":2203},"# and known failure modes identified in Step 1\n",{"type":40,"tag":367,"props":2205,"children":2206},{"class":369,"line":785},[2207],{"type":40,"tag":367,"props":2208,"children":2209},{},[2210],{"type":46,"value":2211},"sanity_records = [\n",{"type":40,"tag":367,"props":2213,"children":2214},{"class":369,"line":829},[2215],{"type":40,"tag":367,"props":2216,"children":2217},{},[2218],{"type":46,"value":2219},"    {\"inputs\": {\"query\": \"\u003Cquestion 1 from interview>\"}, \"expected_response\": \"\u003Cexpected answer>\"},\n",{"type":40,"tag":367,"props":2221,"children":2222},{"class":369,"line":837},[2223],{"type":40,"tag":367,"props":2224,"children":2225},{},[2226],{"type":46,"value":2227},"    {\"inputs\": {\"query\": \"\u003Cquestion 2 from interview>\"}, \"expected_response\": \"\u003Cexpected answer>\"},\n",{"type":40,"tag":367,"props":2229,"children":2230},{"class":369,"line":846},[2231],{"type":40,"tag":367,"props":2232,"children":2233},{},[2234],{"type":46,"value":2235},"    # ... 5 total\n",{"type":40,"tag":367,"props":2237,"children":2238},{"class":369,"line":896},[2239],{"type":40,"tag":367,"props":2240,"children":2241},{},[2242],{"type":46,"value":2243},"]\n",{"type":40,"tag":367,"props":2245,"children":2247},{"class":369,"line":2246},11,[2248],{"type":40,"tag":367,"props":2249,"children":2250},{"emptyLinePlaceholder":770},[2251],{"type":46,"value":773},{"type":40,"tag":367,"props":2253,"children":2255},{"class":369,"line":2254},12,[2256],{"type":40,"tag":367,"props":2257,"children":2258},{},[2259],{"type":46,"value":2260},"sanity_dataset = create_dataset(\n",{"type":40,"tag":367,"props":2262,"children":2264},{"class":369,"line":2263},13,[2265],{"type":40,"tag":367,"props":2266,"children":2267},{},[2268],{"type":46,"value":2269},"    records=sanity_records,\n",{"type":40,"tag":367,"props":2271,"children":2273},{"class":369,"line":2272},14,[2274],{"type":40,"tag":367,"props":2275,"children":2276},{},[2277],{"type":46,"value":2278},"    name=\"sanity-check-5q\",\n",{"type":40,"tag":367,"props":2280,"children":2282},{"class":369,"line":2281},15,[2283],{"type":40,"tag":367,"props":2284,"children":2285},{},[2286],{"type":46,"value":2287},")\n",{"type":40,"tag":49,"props":2289,"children":2290},{},[2291,2293,2298],{"type":46,"value":2292},"Run evaluation on this dataset (see Step 4), then ",{"type":40,"tag":65,"props":2294,"children":2295},{},[2296],{"type":46,"value":2297},"present results to the user",{"type":46,"value":2299}," with this framing:",{"type":40,"tag":1706,"props":2301,"children":2302},{},[2303],{"type":40,"tag":49,"props":2304,"children":2305},{},[2306],{"type":46,"value":2307},"\"This is a sanity check — 5 questions confirm the pipeline works but aren't statistically meaningful. Proceeding to Phase B to generate a proper evaluation set.\"",{"type":40,"tag":49,"props":2309,"children":2310},{},[2311],{"type":46,"value":2312},"Only proceed to Phase B once Phase A completes without errors.",{"type":40,"tag":2143,"props":2314,"children":2315},{},[],{"type":40,"tag":2147,"props":2317,"children":2319},{"id":2318},"phase-b-proper-evaluation-dataset-100-questions-run-after-phase-a-passes",[2320],{"type":46,"value":2321},"Phase B: Proper Evaluation Dataset (100+ questions — run after Phase A passes)",{"type":40,"tag":49,"props":2323,"children":2324},{},[2325],{"type":46,"value":2326},"Generate questions from the agent's actual corpus rather than inventing them from scratch. The approach depends on whether the project uses Databricks or OSS MLflow.",{"type":40,"tag":49,"props":2328,"children":2329},{},[2330,2335,2337,2342],{"type":40,"tag":65,"props":2331,"children":2332},{},[2333],{"type":46,"value":2334},"On Databricks",{"type":46,"value":2336}," — use ",{"type":40,"tag":88,"props":2338,"children":2340},{"className":2339},[],[2341],{"type":46,"value":1559},{"type":46,"value":2343}," to synthesize questions from the agent's document corpus:",{"type":40,"tag":356,"props":2345,"children":2347},{"className":1806,"code":2346,"language":1808,"meta":361,"style":361},"from databricks.agents.evals import generate_evals_df, estimate_synthetic_num_evals\nimport mlflow\n\n# agent_description comes from Step 1 interview answers\nagent_description = \"\u003Cagent purpose from interview>\"\n\n# docs_df: a Spark or pandas DataFrame with a \"content\" column containing\n# the documents\u002Fchunks the agent retrieves from (e.g., your Vector Search index)\nevals = generate_evals_df(\n    docs=docs_df,\n    num_evals=100,\n    agent_description=agent_description,\n)\n\n# Merge into MLflow dataset — don't create a separate dataset\ndataset = mlflow.genai.datasets.create_dataset(name=\"generated-evals-100q\")\ndataset.merge_records(evals)\n",[2348],{"type":40,"tag":88,"props":2349,"children":2350},{"__ignoreMap":361},[2351,2359,2366,2373,2381,2389,2396,2404,2412,2420,2428,2436,2444,2451,2458,2466,2475],{"type":40,"tag":367,"props":2352,"children":2353},{"class":369,"line":370},[2354],{"type":40,"tag":367,"props":2355,"children":2356},{},[2357],{"type":46,"value":2358},"from databricks.agents.evals import generate_evals_df, estimate_synthetic_num_evals\n",{"type":40,"tag":367,"props":2360,"children":2361},{"class":369,"line":402},[2362],{"type":40,"tag":367,"props":2363,"children":2364},{},[2365],{"type":46,"value":2172},{"type":40,"tag":367,"props":2367,"children":2368},{"class":369,"line":429},[2369],{"type":40,"tag":367,"props":2370,"children":2371},{"emptyLinePlaceholder":770},[2372],{"type":46,"value":773},{"type":40,"tag":367,"props":2374,"children":2375},{"class":369,"line":766},[2376],{"type":40,"tag":367,"props":2377,"children":2378},{},[2379],{"type":46,"value":2380},"# agent_description comes from Step 1 interview answers\n",{"type":40,"tag":367,"props":2382,"children":2383},{"class":369,"line":776},[2384],{"type":40,"tag":367,"props":2385,"children":2386},{},[2387],{"type":46,"value":2388},"agent_description = \"\u003Cagent purpose from interview>\"\n",{"type":40,"tag":367,"props":2390,"children":2391},{"class":369,"line":785},[2392],{"type":40,"tag":367,"props":2393,"children":2394},{"emptyLinePlaceholder":770},[2395],{"type":46,"value":773},{"type":40,"tag":367,"props":2397,"children":2398},{"class":369,"line":829},[2399],{"type":40,"tag":367,"props":2400,"children":2401},{},[2402],{"type":46,"value":2403},"# docs_df: a Spark or pandas DataFrame with a \"content\" column containing\n",{"type":40,"tag":367,"props":2405,"children":2406},{"class":369,"line":837},[2407],{"type":40,"tag":367,"props":2408,"children":2409},{},[2410],{"type":46,"value":2411},"# the documents\u002Fchunks the agent retrieves from (e.g., your Vector Search index)\n",{"type":40,"tag":367,"props":2413,"children":2414},{"class":369,"line":846},[2415],{"type":40,"tag":367,"props":2416,"children":2417},{},[2418],{"type":46,"value":2419},"evals = generate_evals_df(\n",{"type":40,"tag":367,"props":2421,"children":2422},{"class":369,"line":896},[2423],{"type":40,"tag":367,"props":2424,"children":2425},{},[2426],{"type":46,"value":2427},"    docs=docs_df,\n",{"type":40,"tag":367,"props":2429,"children":2430},{"class":369,"line":2246},[2431],{"type":40,"tag":367,"props":2432,"children":2433},{},[2434],{"type":46,"value":2435},"    num_evals=100,\n",{"type":40,"tag":367,"props":2437,"children":2438},{"class":369,"line":2254},[2439],{"type":40,"tag":367,"props":2440,"children":2441},{},[2442],{"type":46,"value":2443},"    agent_description=agent_description,\n",{"type":40,"tag":367,"props":2445,"children":2446},{"class":369,"line":2263},[2447],{"type":40,"tag":367,"props":2448,"children":2449},{},[2450],{"type":46,"value":2287},{"type":40,"tag":367,"props":2452,"children":2453},{"class":369,"line":2272},[2454],{"type":40,"tag":367,"props":2455,"children":2456},{"emptyLinePlaceholder":770},[2457],{"type":46,"value":773},{"type":40,"tag":367,"props":2459,"children":2460},{"class":369,"line":2281},[2461],{"type":40,"tag":367,"props":2462,"children":2463},{},[2464],{"type":46,"value":2465},"# Merge into MLflow dataset — don't create a separate dataset\n",{"type":40,"tag":367,"props":2467,"children":2469},{"class":369,"line":2468},16,[2470],{"type":40,"tag":367,"props":2471,"children":2472},{},[2473],{"type":46,"value":2474},"dataset = mlflow.genai.datasets.create_dataset(name=\"generated-evals-100q\")\n",{"type":40,"tag":367,"props":2476,"children":2478},{"class":369,"line":2477},17,[2479],{"type":40,"tag":367,"props":2480,"children":2481},{},[2482],{"type":46,"value":2483},"dataset.merge_records(evals)\n",{"type":40,"tag":49,"props":2485,"children":2486},{},[2487,2489,2495],{"type":46,"value":2488},"To estimate the right ",{"type":40,"tag":88,"props":2490,"children":2492},{"className":2491},[],[2493],{"type":46,"value":2494},"num_evals",{"type":46,"value":2496}," before generating:",{"type":40,"tag":356,"props":2498,"children":2500},{"className":1806,"code":2499,"language":1808,"meta":361,"style":361},"recommended = estimate_synthetic_num_evals(docs_df)\nprint(f\"Recommended num_evals: {recommended}\")\n",[2501],{"type":40,"tag":88,"props":2502,"children":2503},{"__ignoreMap":361},[2504,2512],{"type":40,"tag":367,"props":2505,"children":2506},{"class":369,"line":370},[2507],{"type":40,"tag":367,"props":2508,"children":2509},{},[2510],{"type":46,"value":2511},"recommended = estimate_synthetic_num_evals(docs_df)\n",{"type":40,"tag":367,"props":2513,"children":2514},{"class":369,"line":402},[2515],{"type":40,"tag":367,"props":2516,"children":2517},{},[2518],{"type":46,"value":2519},"print(f\"Recommended num_evals: {recommended}\")\n",{"type":40,"tag":49,"props":2521,"children":2522},{},[2523],{"type":40,"tag":65,"props":2524,"children":2525},{},[2526],{"type":46,"value":2527},"Dataset size guidance:",{"type":40,"tag":73,"props":2529,"children":2530},{},[2531,2541,2551],{"type":40,"tag":77,"props":2532,"children":2533},{},[2534,2539],{"type":40,"tag":65,"props":2535,"children":2536},{},[2537],{"type":46,"value":2538},"\u003C30 questions",{"type":46,"value":2540},": not statistically meaningful — avoid drawing conclusions",{"type":40,"tag":77,"props":2542,"children":2543},{},[2544,2549],{"type":40,"tag":65,"props":2545,"children":2546},{},[2547],{"type":46,"value":2548},"50–100 questions",{"type":46,"value":2550},": adequate for catching regressions, suitable for most agents",{"type":40,"tag":77,"props":2552,"children":2553},{},[2554,2559],{"type":40,"tag":65,"props":2555,"children":2556},{},[2557],{"type":46,"value":2558},"200+ questions",{"type":46,"value":2560},": recommended when comparing model variants or scoring multiple dimensions",{"type":40,"tag":49,"props":2562,"children":2563},{},[2564,2569,2571,2577],{"type":40,"tag":65,"props":2565,"children":2566},{},[2567],{"type":46,"value":2568},"On OSS MLflow",{"type":46,"value":2570}," — use RAGAS ",{"type":40,"tag":88,"props":2572,"children":2574},{"className":2573},[],[2575],{"type":46,"value":2576},"TestsetGenerator",{"type":46,"value":2578}," to generate from your document corpus:",{"type":40,"tag":356,"props":2580,"children":2582},{"className":1806,"code":2581,"language":1808,"meta":361,"style":361},"from ragas.testset import TestsetGenerator\nfrom ragas.llms import LangchainLLMWrapper\nfrom ragas.embeddings import LangchainEmbeddingsWrapper\n\ngenerator = TestsetGenerator(\n    llm=LangchainLLMWrapper(your_llm),\n    embedding_model=LangchainEmbeddingsWrapper(your_embeddings),\n)\ntestset = generator.generate_with_langchain_docs(docs, testset_size=100)\nevals_df = testset.to_pandas()\n\n# Convert to MLflow dataset schema and merge\nimport mlflow\nrecords = [\n    {\"inputs\": {\"query\": row[\"user_input\"]}, \"expected_response\": row[\"reference\"]}\n    for _, row in evals_df.iterrows()\n]\ndataset = mlflow.genai.datasets.create_dataset(name=\"generated-evals-100q\")\ndataset.merge_records(records)\n",[2583],{"type":40,"tag":88,"props":2584,"children":2585},{"__ignoreMap":361},[2586,2594,2602,2610,2617,2625,2633,2641,2648,2656,2664,2671,2679,2686,2694,2702,2710,2717,2725],{"type":40,"tag":367,"props":2587,"children":2588},{"class":369,"line":370},[2589],{"type":40,"tag":367,"props":2590,"children":2591},{},[2592],{"type":46,"value":2593},"from ragas.testset import TestsetGenerator\n",{"type":40,"tag":367,"props":2595,"children":2596},{"class":369,"line":402},[2597],{"type":40,"tag":367,"props":2598,"children":2599},{},[2600],{"type":46,"value":2601},"from ragas.llms import LangchainLLMWrapper\n",{"type":40,"tag":367,"props":2603,"children":2604},{"class":369,"line":429},[2605],{"type":40,"tag":367,"props":2606,"children":2607},{},[2608],{"type":46,"value":2609},"from ragas.embeddings import LangchainEmbeddingsWrapper\n",{"type":40,"tag":367,"props":2611,"children":2612},{"class":369,"line":766},[2613],{"type":40,"tag":367,"props":2614,"children":2615},{"emptyLinePlaceholder":770},[2616],{"type":46,"value":773},{"type":40,"tag":367,"props":2618,"children":2619},{"class":369,"line":776},[2620],{"type":40,"tag":367,"props":2621,"children":2622},{},[2623],{"type":46,"value":2624},"generator = TestsetGenerator(\n",{"type":40,"tag":367,"props":2626,"children":2627},{"class":369,"line":785},[2628],{"type":40,"tag":367,"props":2629,"children":2630},{},[2631],{"type":46,"value":2632},"    llm=LangchainLLMWrapper(your_llm),\n",{"type":40,"tag":367,"props":2634,"children":2635},{"class":369,"line":829},[2636],{"type":40,"tag":367,"props":2637,"children":2638},{},[2639],{"type":46,"value":2640},"    embedding_model=LangchainEmbeddingsWrapper(your_embeddings),\n",{"type":40,"tag":367,"props":2642,"children":2643},{"class":369,"line":837},[2644],{"type":40,"tag":367,"props":2645,"children":2646},{},[2647],{"type":46,"value":2287},{"type":40,"tag":367,"props":2649,"children":2650},{"class":369,"line":846},[2651],{"type":40,"tag":367,"props":2652,"children":2653},{},[2654],{"type":46,"value":2655},"testset = generator.generate_with_langchain_docs(docs, testset_size=100)\n",{"type":40,"tag":367,"props":2657,"children":2658},{"class":369,"line":896},[2659],{"type":40,"tag":367,"props":2660,"children":2661},{},[2662],{"type":46,"value":2663},"evals_df = testset.to_pandas()\n",{"type":40,"tag":367,"props":2665,"children":2666},{"class":369,"line":2246},[2667],{"type":40,"tag":367,"props":2668,"children":2669},{"emptyLinePlaceholder":770},[2670],{"type":46,"value":773},{"type":40,"tag":367,"props":2672,"children":2673},{"class":369,"line":2254},[2674],{"type":40,"tag":367,"props":2675,"children":2676},{},[2677],{"type":46,"value":2678},"# Convert to MLflow dataset schema and merge\n",{"type":40,"tag":367,"props":2680,"children":2681},{"class":369,"line":2263},[2682],{"type":40,"tag":367,"props":2683,"children":2684},{},[2685],{"type":46,"value":2172},{"type":40,"tag":367,"props":2687,"children":2688},{"class":369,"line":2272},[2689],{"type":40,"tag":367,"props":2690,"children":2691},{},[2692],{"type":46,"value":2693},"records = [\n",{"type":40,"tag":367,"props":2695,"children":2696},{"class":369,"line":2281},[2697],{"type":40,"tag":367,"props":2698,"children":2699},{},[2700],{"type":46,"value":2701},"    {\"inputs\": {\"query\": row[\"user_input\"]}, \"expected_response\": row[\"reference\"]}\n",{"type":40,"tag":367,"props":2703,"children":2704},{"class":369,"line":2468},[2705],{"type":40,"tag":367,"props":2706,"children":2707},{},[2708],{"type":46,"value":2709},"    for _, row in evals_df.iterrows()\n",{"type":40,"tag":367,"props":2711,"children":2712},{"class":369,"line":2477},[2713],{"type":40,"tag":367,"props":2714,"children":2715},{},[2716],{"type":46,"value":2243},{"type":40,"tag":367,"props":2718,"children":2720},{"class":369,"line":2719},18,[2721],{"type":40,"tag":367,"props":2722,"children":2723},{},[2724],{"type":46,"value":2474},{"type":40,"tag":367,"props":2726,"children":2727},{"class":369,"line":27},[2728],{"type":40,"tag":367,"props":2729,"children":2730},{},[2731],{"type":46,"value":2732},"dataset.merge_records(records)\n",{"type":40,"tag":49,"props":2734,"children":2735},{},[2736,2741],{"type":40,"tag":65,"props":2737,"children":2738},{},[2739],{"type":46,"value":2740},"If no document corpus is available",{"type":46,"value":2742}," — ask the user to provide 50–100 representative queries from production logs or usage history. These are more realistic than synthetic questions and are preferable when available.",{"type":40,"tag":2143,"props":2744,"children":2745},{},[],{"type":40,"tag":49,"props":2747,"children":2748},{},[2749,2754,2756,2762],{"type":40,"tag":65,"props":2750,"children":2751},{},[2752],{"type":46,"value":2753},"IMPORTANT",{"type":46,"value":2755},": Do not skip dataset discovery. Always run ",{"type":40,"tag":88,"props":2757,"children":2759},{"className":2758},[],[2760],{"type":46,"value":2761},"list_datasets.py",{"type":46,"value":2763}," first, even if you plan to create a new dataset. This prevents duplicate work and ensures users are aware of existing evaluation datasets.",{"type":40,"tag":49,"props":2765,"children":2766},{},[2767,2772,2774],{"type":40,"tag":65,"props":2768,"children":2769},{},[2770],{"type":46,"value":2771},"For complete dataset guide:",{"type":46,"value":2773}," See ",{"type":40,"tag":88,"props":2775,"children":2777},{"className":2776},[],[2778],{"type":46,"value":2779},"references\u002Fdataset-preparation.md",{"type":40,"tag":49,"props":2781,"children":2782},{},[2783],{"type":40,"tag":65,"props":2784,"children":2785},{},[2786],{"type":46,"value":1373},{"type":40,"tag":73,"props":2788,"children":2790},{"className":2789},[1377],[2791,2800,2809],{"type":40,"tag":77,"props":2792,"children":2794},{"className":2793},[1382],[2795,2798],{"type":40,"tag":1385,"props":2796,"children":2797},{"disabled":770,"type":1387},[],{"type":46,"value":2799}," Scorers have been registered",{"type":40,"tag":77,"props":2801,"children":2803},{"className":2802},[1382],[2804,2807],{"type":40,"tag":1385,"props":2805,"children":2806},{"disabled":770,"type":1387},[],{"type":46,"value":2808}," Phase A sanity check passed (pipeline runs end-to-end)",{"type":40,"tag":77,"props":2810,"children":2812},{"className":2811},[1382],[2813,2816],{"type":40,"tag":1385,"props":2814,"children":2815},{"disabled":770,"type":1387},[],{"type":46,"value":2817}," Phase B dataset created with 50+ questions (or existing dataset selected)",{"type":40,"tag":649,"props":2819,"children":2821},{"id":2820},"step-35-dry-run-required-before-full-eval",[2822],{"type":46,"value":2823},"Step 3.5: Dry Run (REQUIRED before full eval)",{"type":40,"tag":49,"props":2825,"children":2826},{},[2827,2829,2834],{"type":46,"value":2828},"Run evaluation on ",{"type":40,"tag":65,"props":2830,"children":2831},{},[2832],{"type":46,"value":2833},"3 questions",{"type":46,"value":2835}," from the dataset before committing to the full run. This catches broken tools, misconfigured scorers, and auth failures early — before they silently corrupt 100-question results.",{"type":40,"tag":49,"props":2837,"children":2838},{},[2839],{"type":46,"value":2840},"If you completed Phase A above, the pipeline is already validated — focus the dry run on scorer output only.",{"type":40,"tag":356,"props":2842,"children":2844},{"className":1806,"code":2843,"language":1808,"meta":361,"style":361},"import mlflow\n\ndataset = mlflow.genai.datasets.get_dataset(name=\"\u003Cyour-dataset-name>\")\ndry_run_records = dataset.df.head(3)\n",[2845],{"type":40,"tag":88,"props":2846,"children":2847},{"__ignoreMap":361},[2848,2855,2862,2870],{"type":40,"tag":367,"props":2849,"children":2850},{"class":369,"line":370},[2851],{"type":40,"tag":367,"props":2852,"children":2853},{},[2854],{"type":46,"value":2172},{"type":40,"tag":367,"props":2856,"children":2857},{"class":369,"line":402},[2858],{"type":40,"tag":367,"props":2859,"children":2860},{"emptyLinePlaceholder":770},[2861],{"type":46,"value":773},{"type":40,"tag":367,"props":2863,"children":2864},{"class":369,"line":429},[2865],{"type":40,"tag":367,"props":2866,"children":2867},{},[2868],{"type":46,"value":2869},"dataset = mlflow.genai.datasets.get_dataset(name=\"\u003Cyour-dataset-name>\")\n",{"type":40,"tag":367,"props":2871,"children":2872},{"class":369,"line":766},[2873],{"type":40,"tag":367,"props":2874,"children":2875},{},[2876],{"type":46,"value":2877},"dry_run_records = dataset.df.head(3)\n",{"type":40,"tag":49,"props":2879,"children":2880},{},[2881,2883,2888],{"type":46,"value":2882},"Run ",{"type":40,"tag":88,"props":2884,"children":2886},{"className":2885},[],[2887],{"type":46,"value":135},{"type":46,"value":2889}," on these 3 records using the same wrapper and scorers as the full eval.",{"type":40,"tag":49,"props":2891,"children":2892},{},[2893],{"type":40,"tag":65,"props":2894,"children":2895},{},[2896],{"type":46,"value":2897},"For each response, check:",{"type":40,"tag":202,"props":2899,"children":2900},{},[2901,2911,2921],{"type":40,"tag":77,"props":2902,"children":2903},{},[2904,2909],{"type":40,"tag":65,"props":2905,"children":2906},{},[2907],{"type":46,"value":2908},"Tool calls",{"type":46,"value":2910}," — Did the agent call any tools? If it called zero tools on questions that require retrieval, tools are likely broken (403s, rate limits, missing credentials).",{"type":40,"tag":77,"props":2912,"children":2913},{},[2914,2919],{"type":40,"tag":65,"props":2915,"children":2916},{},[2917],{"type":46,"value":2918},"Response quality",{"type":46,"value":2920}," — Are responses empty or generic (\"I don't know\", \"I can't help with that\")? Empty responses score as irrelevant and will skew the full eval.",{"type":40,"tag":77,"props":2922,"children":2923},{},[2924,2929,2931,2937,2938,2943,2945,2950,2952,2957,2959,2964,2966,2971,2972,2977],{"type":40,"tag":65,"props":2925,"children":2926},{},[2927],{"type":46,"value":2928},"Scorer output",{"type":46,"value":2930}," — Did all 3 scores come back as ",{"type":40,"tag":88,"props":2932,"children":2934},{"className":2933},[],[2935],{"type":46,"value":2936},"0",{"type":46,"value":1727},{"type":40,"tag":88,"props":2939,"children":2941},{"className":2940},[],[2942],{"type":46,"value":1761},{"type":46,"value":2944},"? If so, the scorer is misconfigured (check return values — ",{"type":40,"tag":88,"props":2946,"children":2948},{"className":2947},[],[2949],{"type":46,"value":1741},{"type":46,"value":2951},"\u002F",{"type":40,"tag":88,"props":2953,"children":2955},{"className":2954},[],[2956],{"type":46,"value":1748},{"type":46,"value":2958}," are silently cast to ",{"type":40,"tag":88,"props":2960,"children":2962},{"className":2961},[],[2963],{"type":46,"value":1761},{"type":46,"value":2965},"; use ",{"type":40,"tag":88,"props":2967,"children":2969},{"className":2968},[],[2970],{"type":46,"value":1725},{"type":46,"value":2951},{"type":40,"tag":88,"props":2973,"children":2975},{"className":2974},[],[2976],{"type":46,"value":1733},{"type":46,"value":2978}," instead).",{"type":40,"tag":49,"props":2980,"children":2981},{},[2982],{"type":40,"tag":65,"props":2983,"children":2984},{},[2985],{"type":46,"value":2986},"Decision gate:",{"type":40,"tag":73,"props":2988,"children":2989},{},[2990,3000,3010],{"type":40,"tag":77,"props":2991,"children":2992},{},[2993,2998],{"type":40,"tag":65,"props":2994,"children":2995},{},[2996],{"type":46,"value":2997},"If dry run shows tool failures or empty responses:",{"type":46,"value":2999}," Stop. Fix the underlying issue (auth, tool config, retrieval) before proceeding. Do not run the full eval on broken infrastructure.",{"type":40,"tag":77,"props":3001,"children":3002},{},[3003,3008],{"type":40,"tag":65,"props":3004,"children":3005},{},[3006],{"type":46,"value":3007},"If all 3 scorer outputs are 0 or None:",{"type":46,"value":3009}," Stop. Debug scorer return values and re-register before proceeding.",{"type":40,"tag":77,"props":3011,"children":3012},{},[3013,3018,3020,3026],{"type":40,"tag":65,"props":3014,"children":3015},{},[3016],{"type":46,"value":3017},"If dry run passes:",{"type":46,"value":3019}," Report to the user: ",{"type":40,"tag":3021,"props":3022,"children":3023},"em",{},[3024],{"type":46,"value":3025},"\"Dry run passed (3\u002F3 responses non-empty, tools called, scores non-zero). Proceeding to full eval.\"",{"type":46,"value":3027}," Then continue to Step 4.",{"type":40,"tag":1706,"props":3029,"children":3030},{},[3031],{"type":40,"tag":49,"props":3032,"children":3033},{},[3034,3039],{"type":40,"tag":65,"props":3035,"children":3036},{},[3037],{"type":46,"value":3038},"Why this matters:",{"type":46,"value":3040}," Tool failures (403s from docs scraping, GitHub API rate limits) produce empty agent responses that score as 0. Running a 100-question eval only to discover all tools were failing wastes time and produces misleading results. The dry run catches this in under a minute.",{"type":40,"tag":2143,"props":3042,"children":3043},{},[],{"type":40,"tag":649,"props":3045,"children":3047},{"id":3046},"step-4-run-evaluation",[3048],{"type":46,"value":3049},"Step 4: Run Evaluation",{"type":40,"tag":1706,"props":3051,"children":3052},{},[3053],{"type":40,"tag":49,"props":3054,"children":3055},{},[3056,3061,3062,3068],{"type":40,"tag":65,"props":3057,"children":3058},{},[3059],{"type":46,"value":3060},"Large datasets (50+ questions)?",{"type":46,"value":2773},{"type":40,"tag":88,"props":3063,"children":3065},{"className":3064},[],[3066],{"type":46,"value":3067},"references\u002Fthroughput-guide.md",{"type":46,"value":3069}," for throughput optimization — covers parallelism env vars, async predict functions, and dataset splitting for 200+ question evals.",{"type":40,"tag":2147,"props":3071,"children":3073},{"id":3072},"_4a-estimate-runtime-before-starting",[3074],{"type":46,"value":3075},"4a. Estimate Runtime Before Starting",{"type":40,"tag":49,"props":3077,"children":3078},{},[3079],{"type":46,"value":3080},"Before launching evaluation, tell the user how long it will take:",{"type":40,"tag":202,"props":3082,"children":3083},{},[3084,3121,3175],{"type":40,"tag":77,"props":3085,"children":3086},{},[3087,3092],{"type":40,"tag":65,"props":3088,"children":3089},{},[3090],{"type":46,"value":3091},"Count the dataset questions:",{"type":40,"tag":356,"props":3093,"children":3095},{"className":1806,"code":3094,"language":1808,"meta":361,"style":361},"import mlflow\ndataset = mlflow.genai.datasets.get_dataset(name=\"\u003Cyour-dataset-name>\")\nprint(f\"Dataset size: {len(dataset.df)} questions\")\n",[3096],{"type":40,"tag":88,"props":3097,"children":3098},{"__ignoreMap":361},[3099,3106,3113],{"type":40,"tag":367,"props":3100,"children":3101},{"class":369,"line":370},[3102],{"type":40,"tag":367,"props":3103,"children":3104},{},[3105],{"type":46,"value":2172},{"type":40,"tag":367,"props":3107,"children":3108},{"class":369,"line":402},[3109],{"type":40,"tag":367,"props":3110,"children":3111},{},[3112],{"type":46,"value":2869},{"type":40,"tag":367,"props":3114,"children":3115},{"class":369,"line":429},[3116],{"type":40,"tag":367,"props":3117,"children":3118},{},[3119],{"type":46,"value":3120},"print(f\"Dataset size: {len(dataset.df)} questions\")\n",{"type":40,"tag":77,"props":3122,"children":3123},{},[3124,3129,3131,3165],{"type":40,"tag":65,"props":3125,"children":3126},{},[3127],{"type":46,"value":3128},"Calculate the estimate",{"type":46,"value":3130}," — each question runs the agent once and the judge scorer once:",{"type":40,"tag":73,"props":3132,"children":3133},{},[3134,3147,3160],{"type":40,"tag":77,"props":3135,"children":3136},{},[3137,3139,3145],{"type":46,"value":3138},"Opus-class judge models (e.g. ",{"type":40,"tag":88,"props":3140,"children":3142},{"className":3141},[],[3143],{"type":46,"value":3144},"claude-opus-4",{"type":46,"value":3146},"): ~45–90s per question",{"type":40,"tag":77,"props":3148,"children":3149},{},[3150,3152,3158],{"type":46,"value":3151},"Sonnet-class judge models (e.g. ",{"type":40,"tag":88,"props":3153,"children":3155},{"className":3154},[],[3156],{"type":46,"value":3157},"claude-sonnet-4",{"type":46,"value":3159},"): ~20–45s per question",{"type":40,"tag":77,"props":3161,"children":3162},{},[3163],{"type":46,"value":3164},"Multiple scorers per question add time proportionally",{"type":40,"tag":356,"props":3166,"children":3170},{"className":3167,"code":3169,"language":46},[3168],"language-text","Estimated time = N questions × 30–60s per question ÷ parallelism factor (typically 4–8x)\n",[3171],{"type":40,"tag":88,"props":3172,"children":3173},{"__ignoreMap":361},[3174],{"type":46,"value":3169},{"type":40,"tag":77,"props":3176,"children":3177},{},[3178,3183],{"type":40,"tag":65,"props":3179,"children":3180},{},[3181],{"type":46,"value":3182},"Tell the user before starting:",{"type":40,"tag":1706,"props":3184,"children":3185},{},[3186],{"type":40,"tag":49,"props":3187,"children":3188},{},[3189,3191,3196],{"type":46,"value":3190},"\"This dataset has N questions. At ~30–60s per question with typical parallelism, evaluation will take approximately ",{"type":40,"tag":65,"props":3192,"children":3193},{},[3194],{"type":46,"value":3195},"X–Y minutes",{"type":46,"value":3197},". I'll run it as a background task so you can continue working — I'll summarize the results when it's done.\"",{"type":40,"tag":2147,"props":3199,"children":3201},{"id":3200},"_4b-generate-the-evaluation-script",[3202],{"type":46,"value":3203},"4b. Generate the Evaluation Script",{"type":40,"tag":356,"props":3205,"children":3207},{"className":358,"code":3206,"language":360,"meta":361,"style":361},"# Generate evaluation script (specify module and entry point)\nuv run python scripts\u002Frun_evaluation_template.py \\\n  --module mlflow_agent.agent \\\n  --entry-point run_agent\n",[3208],{"type":40,"tag":88,"props":3209,"children":3210},{"__ignoreMap":361},[3211,3219,3244,3261],{"type":40,"tag":367,"props":3212,"children":3213},{"class":369,"line":370},[3214],{"type":40,"tag":367,"props":3215,"children":3216},{"style":396},[3217],{"type":46,"value":3218},"# Generate evaluation script (specify module and entry point)\n",{"type":40,"tag":367,"props":3220,"children":3221},{"class":369,"line":402},[3222,3226,3230,3234,3239],{"type":40,"tag":367,"props":3223,"children":3224},{"style":374},[3225],{"type":46,"value":377},{"type":40,"tag":367,"props":3227,"children":3228},{"style":380},[3229],{"type":46,"value":383},{"type":40,"tag":367,"props":3231,"children":3232},{"style":380},[3233],{"type":46,"value":416},{"type":40,"tag":367,"props":3235,"children":3236},{"style":380},[3237],{"type":46,"value":3238}," scripts\u002Frun_evaluation_template.py",{"type":40,"tag":367,"props":3240,"children":3241},{"style":809},[3242],{"type":46,"value":3243}," \\\n",{"type":40,"tag":367,"props":3245,"children":3246},{"class":369,"line":429},[3247,3252,3257],{"type":40,"tag":367,"props":3248,"children":3249},{"style":380},[3250],{"type":46,"value":3251},"  --module",{"type":40,"tag":367,"props":3253,"children":3254},{"style":380},[3255],{"type":46,"value":3256}," mlflow_agent.agent",{"type":40,"tag":367,"props":3258,"children":3259},{"style":809},[3260],{"type":46,"value":3243},{"type":40,"tag":367,"props":3262,"children":3263},{"class":369,"line":766},[3264,3269],{"type":40,"tag":367,"props":3265,"children":3266},{"style":380},[3267],{"type":46,"value":3268},"  --entry-point",{"type":40,"tag":367,"props":3270,"children":3271},{"style":380},[3272],{"type":46,"value":3273}," run_agent\n",{"type":40,"tag":49,"props":3275,"children":3276},{},[3277],{"type":46,"value":3278},"The generated script creates a wrapper function that:",{"type":40,"tag":73,"props":3280,"children":3281},{},[3282,3287,3300,3311],{"type":40,"tag":77,"props":3283,"children":3284},{},[3285],{"type":46,"value":3286},"Accepts keyword arguments matching the dataset's input keys",{"type":40,"tag":77,"props":3288,"children":3289},{},[3290,3292,3298],{"type":46,"value":3291},"Provides any additional arguments the agent needs (like ",{"type":40,"tag":88,"props":3293,"children":3295},{"className":3294},[],[3296],{"type":46,"value":3297},"llm_provider",{"type":46,"value":3299},")",{"type":40,"tag":77,"props":3301,"children":3302},{},[3303,3305],{"type":46,"value":3304},"Runs ",{"type":40,"tag":88,"props":3306,"children":3308},{"className":3307},[],[3309],{"type":46,"value":3310},"mlflow.genai.evaluate(data=df, predict_fn=wrapper, scorers=registered_scorers)",{"type":40,"tag":77,"props":3312,"children":3313},{},[3314,3316],{"type":46,"value":3315},"Saves results to ",{"type":40,"tag":88,"props":3317,"children":3319},{"className":3318},[],[3320],{"type":46,"value":3321},"evaluation_results.csv",{"type":40,"tag":49,"props":3323,"children":3324},{},[3325,3326],{"type":46,"value":1319},{"type":40,"tag":65,"props":3327,"children":3328},{},[3329],{"type":46,"value":3330},"CRITICAL: wrapper Signature Must Match Dataset Input Keys",{"type":40,"tag":49,"props":3332,"children":3333},{},[3334,3336,3342],{"type":46,"value":3335},"MLflow calls ",{"type":40,"tag":88,"props":3337,"children":3339},{"className":3338},[],[3340],{"type":46,"value":3341},"predict_fn(**inputs)",{"type":46,"value":3343}," - it unpacks the inputs dict as keyword arguments.",{"type":40,"tag":3345,"props":3346,"children":3347},"table",{},[3348,3372],{"type":40,"tag":3349,"props":3350,"children":3351},"thead",{},[3352],{"type":40,"tag":3353,"props":3354,"children":3355},"tr",{},[3356,3362,3367],{"type":40,"tag":3357,"props":3358,"children":3359},"th",{},[3360],{"type":46,"value":3361},"Dataset Record",{"type":40,"tag":3357,"props":3363,"children":3364},{},[3365],{"type":46,"value":3366},"MLflow Calls",{"type":40,"tag":3357,"props":3368,"children":3369},{},[3370],{"type":46,"value":3371},"predict_fn Must Be",{"type":40,"tag":3373,"props":3374,"children":3375},"tbody",{},[3376,3407],{"type":40,"tag":3353,"props":3377,"children":3378},{},[3379,3389,3398],{"type":40,"tag":3380,"props":3381,"children":3382},"td",{},[3383],{"type":40,"tag":88,"props":3384,"children":3386},{"className":3385},[],[3387],{"type":46,"value":3388},"{\"inputs\": {\"query\": \"...\"}}",{"type":40,"tag":3380,"props":3390,"children":3391},{},[3392],{"type":40,"tag":88,"props":3393,"children":3395},{"className":3394},[],[3396],{"type":46,"value":3397},"predict_fn(query=\"...\")",{"type":40,"tag":3380,"props":3399,"children":3400},{},[3401],{"type":40,"tag":88,"props":3402,"children":3404},{"className":3403},[],[3405],{"type":46,"value":3406},"def wrapper(query):",{"type":40,"tag":3353,"props":3408,"children":3409},{},[3410,3419,3428],{"type":40,"tag":3380,"props":3411,"children":3412},{},[3413],{"type":40,"tag":88,"props":3414,"children":3416},{"className":3415},[],[3417],{"type":46,"value":3418},"{\"inputs\": {\"question\": \"...\", \"context\": \"...\"}}",{"type":40,"tag":3380,"props":3420,"children":3421},{},[3422],{"type":40,"tag":88,"props":3423,"children":3425},{"className":3424},[],[3426],{"type":46,"value":3427},"predict_fn(question=\"...\", context=\"...\")",{"type":40,"tag":3380,"props":3429,"children":3430},{},[3431],{"type":40,"tag":88,"props":3432,"children":3434},{"className":3433},[],[3435],{"type":46,"value":3436},"def wrapper(question, context):",{"type":40,"tag":49,"props":3438,"children":3439},{},[3440],{"type":40,"tag":65,"props":3441,"children":3442},{},[3443],{"type":46,"value":3444},"Common Mistake (WRONG):",{"type":40,"tag":356,"props":3446,"children":3448},{"className":1806,"code":3447,"language":1808,"meta":361,"style":361},"def wrapper(inputs):  # ❌ WRONG - inputs is NOT a dict\n    return agent(inputs[\"query\"])\n",[3449],{"type":40,"tag":88,"props":3450,"children":3451},{"__ignoreMap":361},[3452,3460],{"type":40,"tag":367,"props":3453,"children":3454},{"class":369,"line":370},[3455],{"type":40,"tag":367,"props":3456,"children":3457},{},[3458],{"type":46,"value":3459},"def wrapper(inputs):  # ❌ WRONG - inputs is NOT a dict\n",{"type":40,"tag":367,"props":3461,"children":3462},{"class":369,"line":402},[3463],{"type":40,"tag":367,"props":3464,"children":3465},{},[3466],{"type":46,"value":3467},"    return agent(inputs[\"query\"])\n",{"type":40,"tag":2147,"props":3469,"children":3471},{"id":3470},"_4c-launch-as-a-background-sub-agent",[3472],{"type":46,"value":3473},"4c. Launch as a Background Sub-Agent",{"type":40,"tag":49,"props":3475,"children":3476},{},[3477,3479,3485],{"type":46,"value":3478},"Run the evaluation as a background sub-agent so the main session stays available. Use the Agent tool with ",{"type":40,"tag":88,"props":3480,"children":3482},{"className":3481},[],[3483],{"type":46,"value":3484},"run_in_background: true",{"type":46,"value":2086},{"type":40,"tag":49,"props":3487,"children":3488},{},[3489],{"type":40,"tag":65,"props":3490,"children":3491},{},[3492],{"type":46,"value":3493},"Sub-agent instructions (pass these verbatim):",{"type":40,"tag":356,"props":3495,"children":3498},{"className":3496,"code":3497,"language":46},[3168],"Run the agent evaluation and write results to scratchpad.\n\nSteps:\n1. cd \u003Cproject-directory>\n2. Run: uv run python run_agent_evaluation.py\n3. When complete, write a summary to scratchpad\u002Feval-results.md with:\n   - Exit status (success or error message)\n   - Path to results file (evaluation_results.csv)\n   - Wall-clock time taken\n4. Return only: \"Evaluation complete. Results written to scratchpad\u002Feval-results.md\"\n",[3499],{"type":40,"tag":88,"props":3500,"children":3501},{"__ignoreMap":361},[3502],{"type":46,"value":3497},{"type":40,"tag":49,"props":3504,"children":3505},{},[3506,3511],{"type":40,"tag":65,"props":3507,"children":3508},{},[3509],{"type":46,"value":3510},"In the main session, poll for completion",{"type":46,"value":3512}," by checking for the scratchpad file rather than blocking:",{"type":40,"tag":356,"props":3514,"children":3516},{"className":1806,"code":3515,"language":1808,"meta":361,"style":361},"# Poll every 30s using Glob\n# Glob(\"scratchpad\u002Feval-results.md\")\n# When the file appears, read it and proceed to analysis\n",[3517],{"type":40,"tag":88,"props":3518,"children":3519},{"__ignoreMap":361},[3520,3528,3536],{"type":40,"tag":367,"props":3521,"children":3522},{"class":369,"line":370},[3523],{"type":40,"tag":367,"props":3524,"children":3525},{},[3526],{"type":46,"value":3527},"# Poll every 30s using Glob\n",{"type":40,"tag":367,"props":3529,"children":3530},{"class":369,"line":402},[3531],{"type":40,"tag":367,"props":3532,"children":3533},{},[3534],{"type":46,"value":3535},"# Glob(\"scratchpad\u002Feval-results.md\")\n",{"type":40,"tag":367,"props":3537,"children":3538},{"class":369,"line":429},[3539],{"type":40,"tag":367,"props":3540,"children":3541},{},[3542],{"type":46,"value":3543},"# When the file appears, read it and proceed to analysis\n",{"type":40,"tag":49,"props":3545,"children":3546},{},[3547],{"type":46,"value":3548},"Do NOT use TaskOutput to wait for the background agent — that dumps the full transcript (~10–20k tokens) into the main context.",{"type":40,"tag":2147,"props":3550,"children":3552},{"id":3551},"_4d-analyze-results-after-evaluation-completes",[3553],{"type":46,"value":3554},"4d. Analyze Results (after evaluation completes)",{"type":40,"tag":49,"props":3556,"children":3557},{},[3558,3560,3566],{"type":46,"value":3559},"Once ",{"type":40,"tag":88,"props":3561,"children":3563},{"className":3562},[],[3564],{"type":46,"value":3565},"scratchpad\u002Feval-results.md",{"type":46,"value":3567}," appears, run analysis:",{"type":40,"tag":356,"props":3569,"children":3571},{"className":358,"code":3570,"language":360,"meta":361,"style":361},"# Pattern detection, failure analysis, recommendations\n# Reads the CSV produced by mlflow.genai.evaluate() above\nuv run python scripts\u002Fanalyze_results.py evaluation_results.csv\n",[3572],{"type":40,"tag":88,"props":3573,"children":3574},{"__ignoreMap":361},[3575,3583,3591],{"type":40,"tag":367,"props":3576,"children":3577},{"class":369,"line":370},[3578],{"type":40,"tag":367,"props":3579,"children":3580},{"style":396},[3581],{"type":46,"value":3582},"# Pattern detection, failure analysis, recommendations\n",{"type":40,"tag":367,"props":3584,"children":3585},{"class":369,"line":402},[3586],{"type":40,"tag":367,"props":3587,"children":3588},{"style":396},[3589],{"type":46,"value":3590},"# Reads the CSV produced by mlflow.genai.evaluate() above\n",{"type":40,"tag":367,"props":3592,"children":3593},{"class":369,"line":429},[3594,3598,3602,3606,3611],{"type":40,"tag":367,"props":3595,"children":3596},{"style":374},[3597],{"type":46,"value":377},{"type":40,"tag":367,"props":3599,"children":3600},{"style":380},[3601],{"type":46,"value":383},{"type":40,"tag":367,"props":3603,"children":3604},{"style":380},[3605],{"type":46,"value":416},{"type":40,"tag":367,"props":3607,"children":3608},{"style":380},[3609],{"type":46,"value":3610}," scripts\u002Fanalyze_results.py",{"type":40,"tag":367,"props":3612,"children":3613},{"style":380},[3614],{"type":46,"value":3615}," evaluation_results.csv\n",{"type":40,"tag":49,"props":3617,"children":3618},{},[3619,3621,3627],{"type":46,"value":3620},"Generates ",{"type":40,"tag":88,"props":3622,"children":3624},{"className":3623},[],[3625],{"type":46,"value":3626},"evaluation_report.md",{"type":46,"value":3628}," with per-scorer pass rates and improvement suggestions.",{"type":40,"tag":49,"props":3630,"children":3631},{},[3632,3634,3640,3641,3647,3649,3655],{"type":46,"value":3633},"The script reads ",{"type":40,"tag":88,"props":3635,"children":3637},{"className":3636},[],[3638],{"type":46,"value":3639},"{scorer_name}\u002Fvalue",{"type":46,"value":112},{"type":40,"tag":88,"props":3642,"children":3644},{"className":3643},[],[3645],{"type":46,"value":3646},"{scorer_name}\u002Frationale",{"type":46,"value":3648}," columns from the CSV.\nIt also accepts the legacy JSON format from ",{"type":40,"tag":88,"props":3650,"children":3652},{"className":3651},[],[3653],{"type":46,"value":3654},"mlflow traces evaluate",{"type":46,"value":3656}," for backward compatibility:",{"type":40,"tag":356,"props":3658,"children":3660},{"className":358,"code":3659,"language":360,"meta":361,"style":361},"uv run python scripts\u002Fanalyze_results.py evaluation_results.json  # legacy format\nuv run python scripts\u002Fanalyze_results.py evaluation_results.csv --output my_report.md  # custom output\n",[3661],{"type":40,"tag":88,"props":3662,"children":3663},{"__ignoreMap":361},[3664,3693],{"type":40,"tag":367,"props":3665,"children":3666},{"class":369,"line":370},[3667,3671,3675,3679,3683,3688],{"type":40,"tag":367,"props":3668,"children":3669},{"style":374},[3670],{"type":46,"value":377},{"type":40,"tag":367,"props":3672,"children":3673},{"style":380},[3674],{"type":46,"value":383},{"type":40,"tag":367,"props":3676,"children":3677},{"style":380},[3678],{"type":46,"value":416},{"type":40,"tag":367,"props":3680,"children":3681},{"style":380},[3682],{"type":46,"value":3610},{"type":40,"tag":367,"props":3684,"children":3685},{"style":380},[3686],{"type":46,"value":3687}," evaluation_results.json",{"type":40,"tag":367,"props":3689,"children":3690},{"style":396},[3691],{"type":46,"value":3692},"  # legacy format\n",{"type":40,"tag":367,"props":3694,"children":3695},{"class":369,"line":402},[3696,3700,3704,3708,3712,3717,3721,3726],{"type":40,"tag":367,"props":3697,"children":3698},{"style":374},[3699],{"type":46,"value":377},{"type":40,"tag":367,"props":3701,"children":3702},{"style":380},[3703],{"type":46,"value":383},{"type":40,"tag":367,"props":3705,"children":3706},{"style":380},[3707],{"type":46,"value":416},{"type":40,"tag":367,"props":3709,"children":3710},{"style":380},[3711],{"type":46,"value":3610},{"type":40,"tag":367,"props":3713,"children":3714},{"style":380},[3715],{"type":46,"value":3716}," evaluation_results.csv",{"type":40,"tag":367,"props":3718,"children":3719},{"style":380},[3720],{"type":46,"value":536},{"type":40,"tag":367,"props":3722,"children":3723},{"style":380},[3724],{"type":46,"value":3725}," my_report.md",{"type":40,"tag":367,"props":3727,"children":3728},{"style":396},[3729],{"type":46,"value":3730},"  # custom output\n",{"type":40,"tag":55,"props":3732,"children":3734},{"id":3733},"references",[3735],{"type":46,"value":250},{"type":40,"tag":49,"props":3737,"children":3738},{},[3739,3741,3747],{"type":46,"value":3740},"Detailed guides in ",{"type":40,"tag":88,"props":3742,"children":3744},{"className":3743},[],[3745],{"type":46,"value":3746},"references\u002F",{"type":46,"value":3748}," (load as needed):",{"type":40,"tag":73,"props":3750,"children":3751},{},[3752,3762,3778,3788,3798,3808,3818],{"type":40,"tag":77,"props":3753,"children":3754},{},[3755,3760],{"type":40,"tag":65,"props":3756,"children":3757},{},[3758],{"type":46,"value":3759},"setup-guide.md",{"type":46,"value":3761}," - Environment setup (MLflow install, tracking URI configuration)",{"type":40,"tag":77,"props":3763,"children":3764},{},[3765,3770,3771,3776],{"type":40,"tag":65,"props":3766,"children":3767},{},[3768],{"type":46,"value":3769},"Tracing",{"type":46,"value":1326},{"type":40,"tag":88,"props":3772,"children":3774},{"className":3773},[],[3775],{"type":46,"value":1332},{"type":46,"value":3777}," skill (authoritative guide for autolog, decorators, session tracking, verification)",{"type":40,"tag":77,"props":3779,"children":3780},{},[3781,3786],{"type":40,"tag":65,"props":3782,"children":3783},{},[3784],{"type":46,"value":3785},"dataset-preparation.md",{"type":46,"value":3787}," - Dataset schema, APIs, creation, Unity Catalog",{"type":40,"tag":77,"props":3789,"children":3790},{},[3791,3796],{"type":40,"tag":65,"props":3792,"children":3793},{},[3794],{"type":46,"value":3795},"scorers.md",{"type":46,"value":3797}," - Built-in vs custom scorers, registration, testing",{"type":40,"tag":77,"props":3799,"children":3800},{},[3801,3806],{"type":40,"tag":65,"props":3802,"children":3803},{},[3804],{"type":46,"value":3805},"scorers-constraints.md",{"type":46,"value":3807}," - CLI requirements for custom scorers (yes\u002Fno format, templates)",{"type":40,"tag":77,"props":3809,"children":3810},{},[3811,3816],{"type":40,"tag":65,"props":3812,"children":3813},{},[3814],{"type":46,"value":3815},"troubleshooting.md",{"type":46,"value":3817}," - Common errors by phase with solutions",{"type":40,"tag":77,"props":3819,"children":3820},{},[3821,3826],{"type":40,"tag":65,"props":3822,"children":3823},{},[3824],{"type":46,"value":3825},"throughput-guide.md",{"type":46,"value":3827}," - Parallelism env vars, async predict_fn, dataset splitting for 200+ question evals",{"type":40,"tag":49,"props":3829,"children":3830},{},[3831,3833,3839],{"type":46,"value":3832},"Scripts are self-documenting - run with ",{"type":40,"tag":88,"props":3834,"children":3836},{"className":3835},[],[3837],{"type":46,"value":3838},"--help",{"type":46,"value":3840}," for usage details.",{"type":40,"tag":3842,"props":3843,"children":3844},"style",{},[3845],{"type":46,"value":3846},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3848,"total":2263},[3849,3864,3874,3881,3896,3907,3923,3939,3952,3964,3979,3990],{"slug":3850,"name":3850,"fn":3851,"description":3852,"org":3853,"tags":3854,"stars":3861,"repoUrl":3862,"updatedAt":3863},"setup","configure MLflow tracing","Configure MLflow tracing for Claude Code.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3855,3858,3859],{"name":3856,"slug":3857,"type":15},"Claude Code","claude-code",{"name":9,"slug":8,"type":15},{"name":3769,"slug":3860,"type":15},"tracing",27014,"https:\u002F\u002Fgithub.com\u002Fmlflow\u002Fmlflow","2026-07-14T05:39:00.297769",{"slug":3865,"name":3865,"fn":3866,"description":3867,"org":3868,"tags":3869,"stars":3861,"repoUrl":3862,"updatedAt":3873},"status","display MLflow tracing configuration","Show the current MLflow tracing configuration for Claude Code.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3870,3871,3872],{"name":3856,"slug":3857,"type":15},{"name":9,"slug":8,"type":15},{"name":3769,"slug":3860,"type":15},"2026-07-14T05:39:01.540537",{"slug":4,"name":4,"fn":5,"description":6,"org":3875,"tags":3876,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3877,3878,3879,3880],{"name":21,"slug":22,"type":15},{"name":18,"slug":19,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"slug":3882,"name":3882,"fn":3883,"description":3884,"org":3885,"tags":3886,"stars":23,"repoUrl":24,"updatedAt":3895},"analyzing-mlflow-session","analyze MLflow chat conversation sessions","Analyzes an MLflow session — a sequence of traces from a multi-turn chat conversation or interaction. Use when the user asks to debug a chat conversation, review session or chat history, find where a multi-turn chat went wrong, or analyze patterns across turns. Triggers on \"analyze this session\", \"what happened in this conversation\", \"debug session\", \"review chat history\", \"where did this chat go wrong\", \"session traces\", \"analyze chat\", \"debug this chat\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3887,3890,3891,3894],{"name":3888,"slug":3889,"type":15},"Debugging","debugging",{"name":9,"slug":8,"type":15},{"name":3892,"slug":3893,"type":15},"Observability","observability",{"name":3769,"slug":3860,"type":15},"2026-07-14T05:39:10.542342",{"slug":3897,"name":3897,"fn":3898,"description":3899,"org":3900,"tags":3901,"stars":23,"repoUrl":24,"updatedAt":3906},"analyzing-mlflow-trace","analyze MLflow traces","Analyzes a single MLflow trace to answer a user query about it. Use when the user provides a trace ID and asks to debug, investigate, find issues, root-cause errors, understand behavior, or analyze quality. Triggers on \"analyze this trace\", \"what went wrong with this trace\", \"debug trace\", \"investigate trace\", \"why did this trace fail\", \"root cause this trace\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3902,3903,3904,3905],{"name":3888,"slug":3889,"type":15},{"name":9,"slug":8,"type":15},{"name":3892,"slug":3893,"type":15},{"name":3769,"slug":3860,"type":15},"2026-07-14T05:39:02.874441",{"slug":3908,"name":3908,"fn":3909,"description":3910,"org":3911,"tags":3912,"stars":23,"repoUrl":24,"updatedAt":3922},"fix-agent-issue","fix and update AI agent behavior","Drives a disciplined explore → plan → implement → verify loop for changing an AI agent's behavior with confidence — whether fixing a reported failure or introducing a new requirement, business rule, or policy. Grounds the diagnosis in MLflow traces, codifies the desired behavior as a regression test suite (`mlflow.genai.evaluate` assertions in `@mlflow.test` pytest tests), and iterates the agent — not the test — until green, resisting quick system-prompt patches when the real fix is upstream (missing tool, retrieval source, or capability). Use whenever the user wants to fix or change how an agent behaves — e.g. \"fix this issue in my agent\", \"this answer is wrong\", \"the agent is hallucinating\", \"improve my agent based on this trace\", \"make the agent do X instead of Y\", \"I want the agent to lead with\u002Fprioritize\u002Frecommend X\", \"new business rule: the agent should X\", \"always\u002Fnever do X\", \"change the agent's default behavior\" — or shares a trace they want addressed.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3913,3914,3917,3918,3921],{"name":21,"slug":22,"type":15},{"name":3915,"slug":3916,"type":15},"Best Practices","best-practices",{"name":3888,"slug":3889,"type":15},{"name":3919,"slug":3920,"type":15},"Engineering","engineering",{"name":9,"slug":8,"type":15},"2026-07-30T05:53:39.749464",{"slug":1332,"name":1332,"fn":3924,"description":3925,"org":3926,"tags":3927,"stars":23,"repoUrl":24,"updatedAt":3938},"instrument Python and TypeScript with MLflow Tracing","Instruments Python and TypeScript code with MLflow Tracing for observability. Must be loaded when setting up tracing as part of any workflow including agent evaluation. Triggers on adding tracing, instrumenting agents\u002FLLM apps, getting started with MLflow tracing, tracing specific frameworks (LangGraph, LangChain, OpenAI, Gemini, DSPy, CrewAI, AutoGen), or when another skill references tracing setup. Examples - \"How do I add tracing?\", \"Instrument my agent\", \"Trace my LangChain app\", \"Set up tracing for evaluation\"",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3928,3929,3930,3931,3932,3934,3935],{"name":21,"slug":22,"type":15},{"name":18,"slug":19,"type":15},{"name":9,"slug":8,"type":15},{"name":3892,"slug":3893,"type":15},{"name":3933,"slug":1808,"type":15},"Python",{"name":3769,"slug":3860,"type":15},{"name":3936,"slug":3937,"type":15},"TypeScript","typescript","2026-07-20T05:58:52.968218",{"slug":3940,"name":3940,"fn":3941,"description":3942,"org":3943,"tags":3944,"stars":23,"repoUrl":24,"updatedAt":3951},"mlflow-agent","dispatch MLflow workflows and agent tasks","Master dispatcher for all MLflow workflows. Use this skill when the user wants to do anything with MLflow — tracing, evaluating, debugging, or improving an agent. Routes to the right MLflow sub-skill automatically. Triggers on: \"use mlflow\", \"help with mlflow\", \"mlflow agent\", \"add mlflow to my project\", \"trace my agent\", \"evaluate my agent\", or any MLflow task without a specific skill in mind.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3945,3946,3947,3950],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":3948,"slug":3949,"type":15},"MLOps","mlops",{"name":3769,"slug":3860,"type":15},"2026-07-14T05:39:04.133424",{"slug":3953,"name":3953,"fn":3954,"description":3955,"org":3956,"tags":3957,"stars":23,"repoUrl":24,"updatedAt":3963},"mlflow-onboarding","onboard users to MLflow workflows","Onboards users to MLflow by determining their use case (GenAI agents\u002Fapps or traditional ML\u002Fdeep learning) and guiding them through relevant quickstart tutorials and initial integration. If an experiment ID is available, it should be supplied as input to help determine the use case. Use when the user asks to get started with MLflow, set up tracking, add observability, or integrate MLflow into their project. Triggers on \"get started with MLflow\", \"set up MLflow\", \"onboard to MLflow\", \"add MLflow to my project\", \"how do I use MLflow\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3958,3959,3960],{"name":9,"slug":8,"type":15},{"name":3948,"slug":3949,"type":15},{"name":3961,"slug":3962,"type":15},"Onboarding","onboarding","2026-07-14T05:39:06.681003",{"slug":3965,"name":3965,"fn":3966,"description":3967,"org":3968,"tags":3969,"stars":23,"repoUrl":24,"updatedAt":3978},"querying-mlflow-metrics","fetch trace metrics from MLflow servers","Fetches aggregated trace metrics (token usage, latency, trace counts, quality evaluations) from MLflow tracking servers. Triggers on requests to show metrics, analyze token usage, view LLM costs, check usage trends, or query trace statistics.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3970,3973,3976,3977],{"name":3971,"slug":3972,"type":15},"Analytics","analytics",{"name":3974,"slug":3975,"type":15},"Metrics","metrics",{"name":9,"slug":8,"type":15},{"name":3892,"slug":3893,"type":15},"2026-07-14T05:39:13.07056",{"slug":3980,"name":3980,"fn":3981,"description":3982,"org":3983,"tags":3984,"stars":23,"repoUrl":24,"updatedAt":3989},"retrieving-mlflow-traces","retrieve and query MLflow traces","Retrieves MLflow traces using CLI or Python API. Use when the user asks to get a trace by ID, find traces, filter traces by status\u002Ftags\u002Fmetadata\u002Fexecution time, query traces, or debug failed traces. Triggers on \"get trace\", \"search traces\", \"find failed traces\", \"filter traces by\", \"traces slower than\", \"query MLflow traces\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3985,3986,3987,3988],{"name":3888,"slug":3889,"type":15},{"name":9,"slug":8,"type":15},{"name":3892,"slug":3893,"type":15},{"name":3769,"slug":3860,"type":15},"2026-07-14T05:39:09.22888",{"slug":3991,"name":3991,"fn":3992,"description":3993,"org":3994,"tags":3995,"stars":23,"repoUrl":24,"updatedAt":4004},"sagemaker-mlflow","connect to SageMaker Managed MLflow","Connect to SageMaker Managed MLflow (mlflow-app or mlflow-tracking-server ARN) as an MLflow backend, then hand off to the other MLflow skills. Triggers on a SageMaker MLflow ARN (arn:aws:sagemaker:...:mlflow-app\u002F... or arn:aws:sagemaker:...:mlflow-tracking-server\u002F...) or \"SageMaker Managed MLflow\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3996,3999,4002,4003],{"name":3997,"slug":3998,"type":15},"AI Infrastructure","ai-infrastructure",{"name":4000,"slug":4001,"type":15},"AWS","aws",{"name":9,"slug":8,"type":15},{"name":3948,"slug":3949,"type":15},"2026-07-14T05:39:05.401801",{"items":4006,"total":2246},[4007,4014,4021,4028,4036,4046,4053],{"slug":4,"name":4,"fn":5,"description":6,"org":4008,"tags":4009,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4010,4011,4012,4013],{"name":21,"slug":22,"type":15},{"name":18,"slug":19,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"slug":3882,"name":3882,"fn":3883,"description":3884,"org":4015,"tags":4016,"stars":23,"repoUrl":24,"updatedAt":3895},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4017,4018,4019,4020],{"name":3888,"slug":3889,"type":15},{"name":9,"slug":8,"type":15},{"name":3892,"slug":3893,"type":15},{"name":3769,"slug":3860,"type":15},{"slug":3897,"name":3897,"fn":3898,"description":3899,"org":4022,"tags":4023,"stars":23,"repoUrl":24,"updatedAt":3906},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4024,4025,4026,4027],{"name":3888,"slug":3889,"type":15},{"name":9,"slug":8,"type":15},{"name":3892,"slug":3893,"type":15},{"name":3769,"slug":3860,"type":15},{"slug":3908,"name":3908,"fn":3909,"description":3910,"org":4029,"tags":4030,"stars":23,"repoUrl":24,"updatedAt":3922},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4031,4032,4033,4034,4035],{"name":21,"slug":22,"type":15},{"name":3915,"slug":3916,"type":15},{"name":3888,"slug":3889,"type":15},{"name":3919,"slug":3920,"type":15},{"name":9,"slug":8,"type":15},{"slug":1332,"name":1332,"fn":3924,"description":3925,"org":4037,"tags":4038,"stars":23,"repoUrl":24,"updatedAt":3938},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4039,4040,4041,4042,4043,4044,4045],{"name":21,"slug":22,"type":15},{"name":18,"slug":19,"type":15},{"name":9,"slug":8,"type":15},{"name":3892,"slug":3893,"type":15},{"name":3933,"slug":1808,"type":15},{"name":3769,"slug":3860,"type":15},{"name":3936,"slug":3937,"type":15},{"slug":3940,"name":3940,"fn":3941,"description":3942,"org":4047,"tags":4048,"stars":23,"repoUrl":24,"updatedAt":3951},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4049,4050,4051,4052],{"name":21,"slug":22,"type":15},{"name":9,"slug":8,"type":15},{"name":3948,"slug":3949,"type":15},{"name":3769,"slug":3860,"type":15},{"slug":3953,"name":3953,"fn":3954,"description":3955,"org":4054,"tags":4055,"stars":23,"repoUrl":24,"updatedAt":3963},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[4056,4057,4058],{"name":9,"slug":8,"type":15},{"name":3948,"slug":3949,"type":15},{"name":3961,"slug":3962,"type":15}]