[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-google-cloud-agent-eval-workflow":3,"mdc--pcu972-key":44,"related-org-google-cloud-agent-eval-workflow":1425,"related-repo-google-cloud-agent-eval-workflow":1602},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":24,"repoUrl":25,"updatedAt":26,"license":27,"forks":28,"topics":29,"repo":39,"sourceUrl":42,"mdContent":43},"agent-eval-workflow","design and run agent evaluation workflows","This skill should be used when the user wants to evaluate an AI agent end-to-end: scaffold an evaluation, design metrics that test a real hypothesis, make an agent measurable, audit generated eval config, read evaluation results, or run an improvement (\"hill climbing\") loop. Covers evaluation methodology, metric design, dataset coverage, reading deterministic vs LLM-judged metrics, and the traps that make eval runs silently measure nothing. Use alongside the tool-specific skills (agents-cli-eval, adk-eval-guide) — those cover commands and schemas, this covers the process and judgement.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"google-cloud","Google Cloud","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fgoogle-cloud.png","GoogleCloudPlatform",[13,17,20,23],{"name":14,"slug":15,"type":16},"Benchmarking","benchmarking","tag",{"name":18,"slug":19,"type":16},"Evals","evals",{"name":21,"slug":22,"type":16},"Agents","agents",{"name":9,"slug":8,"type":16},3059,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fprofessional-services","2026-08-13T05:04:18.890378",null,1465,[30,31,32,33,34,35,36,37,38],"bigquery","examples","gke","google-cloud-compute","google-cloud-dataflow","google-cloud-ml","google-cloud-platform","solutions","tools",{"repoUrl":25,"stars":24,"forks":28,"topics":40,"description":41},[30,31,32,33,34,35,36,37,38],"Common solutions and tools developed by Google Cloud's Professional Services team. This repository and its contents are not an officially supported Google product.","https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fprofessional-services\u002Ftree\u002FHEAD\u002Ftools\u002Fagent-eval\u002Fskills\u002Fagent-eval-workflow","---\nname: agent-eval-workflow\ndescription: >\n  This skill should be used when the user wants to evaluate an AI agent\n  end-to-end: scaffold an evaluation, design metrics that test a real\n  hypothesis, make an agent measurable, audit generated eval config, read\n  evaluation results, or run an improvement (\"hill climbing\") loop.\n  Covers evaluation methodology, metric design, dataset coverage, reading\n  deterministic vs LLM-judged metrics, and the traps that make eval runs\n  silently measure nothing.\n  Use alongside the tool-specific skills (agents-cli-eval, adk-eval-guide) —\n  those cover commands and schemas, this covers the process and judgement.\nmetadata:\n  author: Google Cloud Professional Services\n  license: Apache-2.0\n  version: 0.1.0\n---\n\n# Agent evaluation: the process\n\nCommands change. The process does not. This skill is about **how to evaluate an\nagent well** — the reasoning that stays true whether you drive it with\n`agent-eval`, `agents-cli eval`, or whatever replaces them.\n\nThe loop: **Hypothesize. Test. Validate.**\n\nState a falsifiable claim about the agent, write metrics that could prove you\nwrong, run them, and let the results — not your intuition — decide what to fix.\n\n---\n\n## 1. Start from a hypothesis, not from \"add some metrics\"\n\nA metric exists to settle an argument. Before generating anything, answer:\n\n- What do I believe this agent gets wrong?\n- What observable evidence would prove that?\n- What would prove me wrong?\n\nRead the agent's own instructions and tool code first. **Stated rules are\ntestable claims** — \"always confirm before deleting\", \"never approve above 10%\",\n\"always cite a source\". Those sentences convert directly into criteria.\n\nWhen a generator asks what to focus on, give it the hypothesis in plain language\nnaming real tools and thresholds. Vague guidance produces vague rubrics.\n\n> Good: \"The agent must never call `approve_discount` above 10%; it should route\n> larger requests to `sync_ask_for_approval` instead of being rejected and\n> retrying.\"\n>\n> Weak: \"Test discount handling.\"\n\n**Prefer binary rubrics (0\u002F1).** LLM judges have poor inter-rater reliability on\n1–5 scales. Several sharp binary metrics beat one fuzzy graded one, and the pass\nrate across a dataset gives you a continuous score with better statistics.\n\n---\n\n## 2. Make the agent measurable before measuring harder\n\nA judge reading prose is fuzzy. A **state variable written by the tool itself is\ndeterministic**. If a tool already makes a decision, record it:\n\n```python\ndef approve_discount(discount_type: str, value: float, reason: str,\n                     tool_context: ToolContext) -> dict:\n    if value > MAX_DISCOUNT_RATE:\n        tool_context.state[\"discount_status\"] = \"rejected\"   # hard evidence\n        return {\"status\": \"rejected\", \"message\": \"discount too large.\"}\n    tool_context.state[\"discount_status\"] = \"approved\"\n    return {\"status\": \"ok\"}\n```\n\nNow `discount_status == \"rejected\"` *proves* the agent attempted an over-limit\napproval. No judge opinion required. Improving observability is usually cheaper\nand more reliable than sharpening a rubric.\n\n---\n\n## 3. Audit generated config — always\n\nTreat AI-generated metrics and datasets as a **draft to review**. The failures\nhere are silent: nothing errors, the run just measures less than it claims.\n\n**Check coverage.** For each metric, ask *which rows does this actually score?*\nCapability flags like `requires_multi_turn` \u002F `requires_reference` silently\nexclude rows. A generated config once marked both custom metrics multi-turn-only,\nleaving every single-turn row unscored — including the sharpest test case in the\ndataset.\n\n**Check that every source column can be populated for this agent.** A metric\nbound to a column the agent can never fill will fail with an error that points at\nthe agent instead of at the metric. See the built-in tools trap below.\n\n**Prefer explicit columns over generic blobs.** Mapping a whole `state_variables`\ndict hides the signal; naming `discount_status` surfaces it to both the judge and\nany coverage report.\n\n**Beware empty-but-valid columns.** If a metric maps its response to a tool-call\nlist and the correct behaviour on some row is *call no tools*, that column is\nempty and reads as \"Response is required but missing\". Map it through a template\nso it always renders text:\n\n```jsonc\n\"response\": {\n  \"template\": \"Tool Calls:\\n{extracted_data_tool_interactions}\",\n  \"source_columns\": [\"extracted_data:tool_interactions\"]\n}\n```\n\n**Run it through the tool's own validator** rather than eyeballing it.\n\n---\n\n## 4. Design a dataset that can fail\n\nRows should include boundary and negative cases, not just happy paths:\n\n- **At the limit** and **over the limit** (10% vs 15% vs 25%)\n- The **second condition** people forget (percentage limit *and* flat-amount limit)\n- Cases where the right answer is **do nothing** \u002F **refuse**\n- **Consent given** and **consent withheld**, as separate rows\n- One clearly **out-of-scope** request\n\nIf every row passes at baseline, the dataset is too easy to teach you anything.\n\n---\n\n## 5. Read results: deterministic vs judged\n\nTwo metric families answer different questions, and **the gap between them is\nthe diagnosis**:\n\n| | Answers | Example |\n|---|---|---|\n| Deterministic (from the trace) | *Did it run?* | `tool_success_rate = 1.0` |\n| LLM-judged (rubric) | *Was it right?* | `tool_use_quality = 0.25` |\n\nA real result: `tool_success_rate` 1.0 alongside `tool_use_quality` 0.25. Both\ncorrect. Every tool call executed without raising — and the agent kept calling\nthe *wrong tool at the wrong time*. Deterministic metrics cannot see that;\njudged metrics can.\n\nWork top-down: headline scores → the specific failing rows → the judge's\nreasoning next to the actual tool calls → the source code.\n\n**A metric that scores perfectly has no headroom.** Say so and move on — a\ndisproven hypothesis is a real result, not a failed experiment.\n\n---\n\n## 6. Hill climbing\n\n1. Pick the **weakest** metric, not the most interesting one.\n2. Find the specific rows that fail it and read what actually happened.\n3. Trace the cause into source — prompt, tool docstring, or architecture.\n4. Change **one** thing, so the delta is attributable.\n5. **Predict the effect out loud before re-running.** That makes it a test.\n6. **Restart the agent server before re-running** — see below. Skipping this is\n   the single easiest way to waste an entire iteration.\n7. Re-run and compare.\n\n### Restart the agent between iterations, and verify you did\n\nA long-running agent server holds the agent module **in memory**. Editing a\nprompt, a tool docstring, or a tool signature changes nothing for a server that\nis already running — it keeps serving the code it imported at startup. The\nevaluation then dutifully measures your *old* agent and reports no improvement,\nand you conclude the fix didn't work.\n\nDon't trust that the restart happened — a new server silently exits if the port\nis still held by the old one. **Compare timestamps:**\n\n```bash\nPID=$(ss -ltnp | grep 8501 | sed -E 's\u002F.*pid=([0-9]+).*\u002F\\1\u002F')\nps -o lstart= -p \"$PID\"                       # when did the server start?\ndate -r path\u002Fto\u002Ftools.py '+%H:%M:%S'          # when did you edit the code?\n```\n\nIf the server start time is **older** than your edit, it is serving stale code —\nkill it, start a new one, and check again before spending a run.\n\nThis applies to whatever drives the agent over HTTP. In-process paths (importing\n`agent.py` directly) pick changes up automatically; anything talking to a server\ndoes not.\n\n### Treat AI root-cause analysis as a hypothesis, not a finding\n\nAutomated analysis is good at spotting *that* something is wrong and bad at\nbeing sure *why*. It will produce a specific, well-argued, source-citing\nexplanation that is confidently wrong — and it reads exactly like a correct one.\n\nA real case: analysis reported the agent was fabricating `customer_id='123'`\nfrom tool docstring examples, quoting the docstrings. Acting on it — adding an\ninstruction never to invent a customer id — made every metric worse\n(`instruction_following` 0.89 → 0.61) and introduced tool-call loops. The\nsystem prompt turned out to inject `Customer.get_customer(\"123\")` as the\nsigned-in customer. Nothing was fabricated; the agent had been told to distrust\na legitimate value.\n\n**Before acting on a diagnosis, verify its claim against the source.** Grep for\nthe value. Check whether the prompt already supplies it. Confirm the mechanism\nexists. One minute of checking beats a 25-minute run in the wrong direction.\n\nAnd when a fix makes things worse, that is a result: it usually means the\ndiagnosis was wrong, not that the fix was too small.\n\nCommon causes worth checking, in rough order of how often they turn out to be it:\n\n- **Docstring examples read as facts.** Mock values like `customer_id='123'` in\n  examples get fabricated into real calls when the agent hits a gap. Add an\n  explicit prohibition; don't rely on the model inferring that examples are fake.\n- **Undocumented rules.** A limit enforced in code but absent from the prompt\n  forces the agent to discover it by failing — wasted calls and latency.\n- **Too many tools on one agent.** Biases toward acting when the right move is\n  to stop and answer.\n- **Inconsistent return contracts.** A tool annotated `-> dict` that returns a\n  bare string on the error path makes the model spend reasoning tokens parsing it.\n\nKeep an iteration log: what changed, which metrics moved, and by how much.\n\n---\n\n## 7. Traps that make a run silently meaningless\n\n**Nothing starts your agent for you.** The eval posts to whatever is listening on\nthe port. A leftover server for a *different* agent will accept the connection\nand fail every request — and the symptoms appear as confusing *metric* errors.\nVerify the server's identity, not just that something is listening:\n\n```bash\nss -ltnp | grep 8501\ntr '\\0' ' ' \u003C \u002Fproc\u002F\u003CPID>\u002Fcmdline    # which agents dir was it launched with?\n```\n\n**Built-in tools are not tool calls.** `google_search`, code execution and\nsimilar produce grounding metadata, not `functionCall` events. Any metric reading\ntool interactions is **structurally unsatisfiable** for an agent whose only tool\nis built in. Check for real Python function tools before writing tool-trajectory\nmetrics.\n\n**A phase can report success while every row failed.** \"Captured N interactions\"\ncounts rows returned, not rows that succeeded. Check per-row status.\n\n**Comparison baselines are picked automatically.** Analysis usually compares\nagainst the most recent previous run. Stale or broken runs left in the results\nfolder will silently poison your deltas — move them aside first.\n\n**Judge failures can be environment failures.** An out-of-range SDK in the\nscoring environment can make judges return prose instead of JSON, which surfaces\nas `400 INVALID_ARGUMENT — Error parsing JSON`. Keep agent runtime and eval\nscoring dependencies separate and in their supported ranges.\n\n**Use two virtualenvs: one for the agent, one for evaluation.** They share\ndependencies (the ADK, the Vertex SDK) but need *different* versions of them.\nInstall the eval tooling into the agent's venv and the agent's lockfile wins:\nthe next `uv sync` \u002F `uv run` in that project re-resolves the environment and\ndowngrades a shared package underneath the eval tool, which then crashes on an\nimport that has nothing to do with the real problem. The supported layout is an\neval venv with the agent installed into it (`uv pip install -e .`), while the\nagent's own venv serves the agent. If a run fails with a surprising\n`ModuleNotFoundError`, check installed versions before debugging anything else.\n\n**Empty results deserve suspicion, not a rerun.** If scores are missing or rows\nare blank, find the upstream cause before spending another run.\n",{"data":45,"body":50},{"name":4,"description":6,"metadata":46},{"author":47,"license":48,"version":49},"Google Cloud Professional Services","Apache-2.0","0.1.0",{"type":51,"children":52},"root",[53,62,93,103,108,112,119,124,144,156,161,191,201,204,210,222,298,319,322,328,340,373,383,409,426,467,477,480,486,491,574,579,582,588,600,681,709,714,724,727,733,796,803,822,832,998,1010,1023,1029,1048,1077,1087,1092,1097,1155,1160,1163,1169,1193,1292,1324,1334,1344,1362,1409,1419],{"type":54,"tag":55,"props":56,"children":58},"element","h1",{"id":57},"agent-evaluation-the-process",[59],{"type":60,"value":61},"text","Agent evaluation: the process",{"type":54,"tag":63,"props":64,"children":65},"p",{},[66,68,74,76,83,85,91],{"type":60,"value":67},"Commands change. The process does not. This skill is about ",{"type":54,"tag":69,"props":70,"children":71},"strong",{},[72],{"type":60,"value":73},"how to evaluate an\nagent well",{"type":60,"value":75}," — the reasoning that stays true whether you drive it with\n",{"type":54,"tag":77,"props":78,"children":80},"code",{"className":79},[],[81],{"type":60,"value":82},"agent-eval",{"type":60,"value":84},", ",{"type":54,"tag":77,"props":86,"children":88},{"className":87},[],[89],{"type":60,"value":90},"agents-cli eval",{"type":60,"value":92},", or whatever replaces them.",{"type":54,"tag":63,"props":94,"children":95},{},[96,98],{"type":60,"value":97},"The loop: ",{"type":54,"tag":69,"props":99,"children":100},{},[101],{"type":60,"value":102},"Hypothesize. Test. Validate.",{"type":54,"tag":63,"props":104,"children":105},{},[106],{"type":60,"value":107},"State a falsifiable claim about the agent, write metrics that could prove you\nwrong, run them, and let the results — not your intuition — decide what to fix.",{"type":54,"tag":109,"props":110,"children":111},"hr",{},[],{"type":54,"tag":113,"props":114,"children":116},"h2",{"id":115},"_1-start-from-a-hypothesis-not-from-add-some-metrics",[117],{"type":60,"value":118},"1. Start from a hypothesis, not from \"add some metrics\"",{"type":54,"tag":63,"props":120,"children":121},{},[122],{"type":60,"value":123},"A metric exists to settle an argument. Before generating anything, answer:",{"type":54,"tag":125,"props":126,"children":127},"ul",{},[128,134,139],{"type":54,"tag":129,"props":130,"children":131},"li",{},[132],{"type":60,"value":133},"What do I believe this agent gets wrong?",{"type":54,"tag":129,"props":135,"children":136},{},[137],{"type":60,"value":138},"What observable evidence would prove that?",{"type":54,"tag":129,"props":140,"children":141},{},[142],{"type":60,"value":143},"What would prove me wrong?",{"type":54,"tag":63,"props":145,"children":146},{},[147,149,154],{"type":60,"value":148},"Read the agent's own instructions and tool code first. ",{"type":54,"tag":69,"props":150,"children":151},{},[152],{"type":60,"value":153},"Stated rules are\ntestable claims",{"type":60,"value":155}," — \"always confirm before deleting\", \"never approve above 10%\",\n\"always cite a source\". Those sentences convert directly into criteria.",{"type":54,"tag":63,"props":157,"children":158},{},[159],{"type":60,"value":160},"When a generator asks what to focus on, give it the hypothesis in plain language\nnaming real tools and thresholds. Vague guidance produces vague rubrics.",{"type":54,"tag":162,"props":163,"children":164},"blockquote",{},[165,186],{"type":54,"tag":63,"props":166,"children":167},{},[168,170,176,178,184],{"type":60,"value":169},"Good: \"The agent must never call ",{"type":54,"tag":77,"props":171,"children":173},{"className":172},[],[174],{"type":60,"value":175},"approve_discount",{"type":60,"value":177}," above 10%; it should route\nlarger requests to ",{"type":54,"tag":77,"props":179,"children":181},{"className":180},[],[182],{"type":60,"value":183},"sync_ask_for_approval",{"type":60,"value":185}," instead of being rejected and\nretrying.\"",{"type":54,"tag":63,"props":187,"children":188},{},[189],{"type":60,"value":190},"Weak: \"Test discount handling.\"",{"type":54,"tag":63,"props":192,"children":193},{},[194,199],{"type":54,"tag":69,"props":195,"children":196},{},[197],{"type":60,"value":198},"Prefer binary rubrics (0\u002F1).",{"type":60,"value":200}," LLM judges have poor inter-rater reliability on\n1–5 scales. Several sharp binary metrics beat one fuzzy graded one, and the pass\nrate across a dataset gives you a continuous score with better statistics.",{"type":54,"tag":109,"props":202,"children":203},{},[],{"type":54,"tag":113,"props":205,"children":207},{"id":206},"_2-make-the-agent-measurable-before-measuring-harder",[208],{"type":60,"value":209},"2. Make the agent measurable before measuring harder",{"type":54,"tag":63,"props":211,"children":212},{},[213,215,220],{"type":60,"value":214},"A judge reading prose is fuzzy. A ",{"type":54,"tag":69,"props":216,"children":217},{},[218],{"type":60,"value":219},"state variable written by the tool itself is\ndeterministic",{"type":60,"value":221},". If a tool already makes a decision, record it:",{"type":54,"tag":223,"props":224,"children":229},"pre",{"className":225,"code":226,"language":227,"meta":228,"style":228},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","def approve_discount(discount_type: str, value: float, reason: str,\n                     tool_context: ToolContext) -> dict:\n    if value > MAX_DISCOUNT_RATE:\n        tool_context.state[\"discount_status\"] = \"rejected\"   # hard evidence\n        return {\"status\": \"rejected\", \"message\": \"discount too large.\"}\n    tool_context.state[\"discount_status\"] = \"approved\"\n    return {\"status\": \"ok\"}\n","python","",[230],{"type":54,"tag":77,"props":231,"children":232},{"__ignoreMap":228},[233,244,253,262,271,280,289],{"type":54,"tag":234,"props":235,"children":238},"span",{"class":236,"line":237},"line",1,[239],{"type":54,"tag":234,"props":240,"children":241},{},[242],{"type":60,"value":243},"def approve_discount(discount_type: str, value: float, reason: str,\n",{"type":54,"tag":234,"props":245,"children":247},{"class":236,"line":246},2,[248],{"type":54,"tag":234,"props":249,"children":250},{},[251],{"type":60,"value":252},"                     tool_context: ToolContext) -> dict:\n",{"type":54,"tag":234,"props":254,"children":256},{"class":236,"line":255},3,[257],{"type":54,"tag":234,"props":258,"children":259},{},[260],{"type":60,"value":261},"    if value > MAX_DISCOUNT_RATE:\n",{"type":54,"tag":234,"props":263,"children":265},{"class":236,"line":264},4,[266],{"type":54,"tag":234,"props":267,"children":268},{},[269],{"type":60,"value":270},"        tool_context.state[\"discount_status\"] = \"rejected\"   # hard evidence\n",{"type":54,"tag":234,"props":272,"children":274},{"class":236,"line":273},5,[275],{"type":54,"tag":234,"props":276,"children":277},{},[278],{"type":60,"value":279},"        return {\"status\": \"rejected\", \"message\": \"discount too large.\"}\n",{"type":54,"tag":234,"props":281,"children":283},{"class":236,"line":282},6,[284],{"type":54,"tag":234,"props":285,"children":286},{},[287],{"type":60,"value":288},"    tool_context.state[\"discount_status\"] = \"approved\"\n",{"type":54,"tag":234,"props":290,"children":292},{"class":236,"line":291},7,[293],{"type":54,"tag":234,"props":294,"children":295},{},[296],{"type":60,"value":297},"    return {\"status\": \"ok\"}\n",{"type":54,"tag":63,"props":299,"children":300},{},[301,303,309,311,317],{"type":60,"value":302},"Now ",{"type":54,"tag":77,"props":304,"children":306},{"className":305},[],[307],{"type":60,"value":308},"discount_status == \"rejected\"",{"type":60,"value":310}," ",{"type":54,"tag":312,"props":313,"children":314},"em",{},[315],{"type":60,"value":316},"proves",{"type":60,"value":318}," the agent attempted an over-limit\napproval. No judge opinion required. Improving observability is usually cheaper\nand more reliable than sharpening a rubric.",{"type":54,"tag":109,"props":320,"children":321},{},[],{"type":54,"tag":113,"props":323,"children":325},{"id":324},"_3-audit-generated-config-always",[326],{"type":60,"value":327},"3. Audit generated config — always",{"type":54,"tag":63,"props":329,"children":330},{},[331,333,338],{"type":60,"value":332},"Treat AI-generated metrics and datasets as a ",{"type":54,"tag":69,"props":334,"children":335},{},[336],{"type":60,"value":337},"draft to review",{"type":60,"value":339},". The failures\nhere are silent: nothing errors, the run just measures less than it claims.",{"type":54,"tag":63,"props":341,"children":342},{},[343,348,350,355,357,363,365,371],{"type":54,"tag":69,"props":344,"children":345},{},[346],{"type":60,"value":347},"Check coverage.",{"type":60,"value":349}," For each metric, ask ",{"type":54,"tag":312,"props":351,"children":352},{},[353],{"type":60,"value":354},"which rows does this actually score?",{"type":60,"value":356},"\nCapability flags like ",{"type":54,"tag":77,"props":358,"children":360},{"className":359},[],[361],{"type":60,"value":362},"requires_multi_turn",{"type":60,"value":364}," \u002F ",{"type":54,"tag":77,"props":366,"children":368},{"className":367},[],[369],{"type":60,"value":370},"requires_reference",{"type":60,"value":372}," silently\nexclude rows. A generated config once marked both custom metrics multi-turn-only,\nleaving every single-turn row unscored — including the sharpest test case in the\ndataset.",{"type":54,"tag":63,"props":374,"children":375},{},[376,381],{"type":54,"tag":69,"props":377,"children":378},{},[379],{"type":60,"value":380},"Check that every source column can be populated for this agent.",{"type":60,"value":382}," A metric\nbound to a column the agent can never fill will fail with an error that points at\nthe agent instead of at the metric. See the built-in tools trap below.",{"type":54,"tag":63,"props":384,"children":385},{},[386,391,393,399,401,407],{"type":54,"tag":69,"props":387,"children":388},{},[389],{"type":60,"value":390},"Prefer explicit columns over generic blobs.",{"type":60,"value":392}," Mapping a whole ",{"type":54,"tag":77,"props":394,"children":396},{"className":395},[],[397],{"type":60,"value":398},"state_variables",{"type":60,"value":400},"\ndict hides the signal; naming ",{"type":54,"tag":77,"props":402,"children":404},{"className":403},[],[405],{"type":60,"value":406},"discount_status",{"type":60,"value":408}," surfaces it to both the judge and\nany coverage report.",{"type":54,"tag":63,"props":410,"children":411},{},[412,417,419,424],{"type":54,"tag":69,"props":413,"children":414},{},[415],{"type":60,"value":416},"Beware empty-but-valid columns.",{"type":60,"value":418}," If a metric maps its response to a tool-call\nlist and the correct behaviour on some row is ",{"type":54,"tag":312,"props":420,"children":421},{},[422],{"type":60,"value":423},"call no tools",{"type":60,"value":425},", that column is\nempty and reads as \"Response is required but missing\". Map it through a template\nso it always renders text:",{"type":54,"tag":223,"props":427,"children":431},{"className":428,"code":429,"language":430,"meta":228,"style":228},"language-jsonc shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","\"response\": {\n  \"template\": \"Tool Calls:\\n{extracted_data_tool_interactions}\",\n  \"source_columns\": [\"extracted_data:tool_interactions\"]\n}\n","jsonc",[432],{"type":54,"tag":77,"props":433,"children":434},{"__ignoreMap":228},[435,443,451,459],{"type":54,"tag":234,"props":436,"children":437},{"class":236,"line":237},[438],{"type":54,"tag":234,"props":439,"children":440},{},[441],{"type":60,"value":442},"\"response\": {\n",{"type":54,"tag":234,"props":444,"children":445},{"class":236,"line":246},[446],{"type":54,"tag":234,"props":447,"children":448},{},[449],{"type":60,"value":450},"  \"template\": \"Tool Calls:\\n{extracted_data_tool_interactions}\",\n",{"type":54,"tag":234,"props":452,"children":453},{"class":236,"line":255},[454],{"type":54,"tag":234,"props":455,"children":456},{},[457],{"type":60,"value":458},"  \"source_columns\": [\"extracted_data:tool_interactions\"]\n",{"type":54,"tag":234,"props":460,"children":461},{"class":236,"line":264},[462],{"type":54,"tag":234,"props":463,"children":464},{},[465],{"type":60,"value":466},"}\n",{"type":54,"tag":63,"props":468,"children":469},{},[470,475],{"type":54,"tag":69,"props":471,"children":472},{},[473],{"type":60,"value":474},"Run it through the tool's own validator",{"type":60,"value":476}," rather than eyeballing it.",{"type":54,"tag":109,"props":478,"children":479},{},[],{"type":54,"tag":113,"props":481,"children":483},{"id":482},"_4-design-a-dataset-that-can-fail",[484],{"type":60,"value":485},"4. Design a dataset that can fail",{"type":54,"tag":63,"props":487,"children":488},{},[489],{"type":60,"value":490},"Rows should include boundary and negative cases, not just happy paths:",{"type":54,"tag":125,"props":492,"children":493},{},[494,511,530,546,562],{"type":54,"tag":129,"props":495,"children":496},{},[497,502,504,509],{"type":54,"tag":69,"props":498,"children":499},{},[500],{"type":60,"value":501},"At the limit",{"type":60,"value":503}," and ",{"type":54,"tag":69,"props":505,"children":506},{},[507],{"type":60,"value":508},"over the limit",{"type":60,"value":510}," (10% vs 15% vs 25%)",{"type":54,"tag":129,"props":512,"children":513},{},[514,516,521,523,528],{"type":60,"value":515},"The ",{"type":54,"tag":69,"props":517,"children":518},{},[519],{"type":60,"value":520},"second condition",{"type":60,"value":522}," people forget (percentage limit ",{"type":54,"tag":312,"props":524,"children":525},{},[526],{"type":60,"value":527},"and",{"type":60,"value":529}," flat-amount limit)",{"type":54,"tag":129,"props":531,"children":532},{},[533,535,540,541],{"type":60,"value":534},"Cases where the right answer is ",{"type":54,"tag":69,"props":536,"children":537},{},[538],{"type":60,"value":539},"do nothing",{"type":60,"value":364},{"type":54,"tag":69,"props":542,"children":543},{},[544],{"type":60,"value":545},"refuse",{"type":54,"tag":129,"props":547,"children":548},{},[549,554,555,560],{"type":54,"tag":69,"props":550,"children":551},{},[552],{"type":60,"value":553},"Consent given",{"type":60,"value":503},{"type":54,"tag":69,"props":556,"children":557},{},[558],{"type":60,"value":559},"consent withheld",{"type":60,"value":561},", as separate rows",{"type":54,"tag":129,"props":563,"children":564},{},[565,567,572],{"type":60,"value":566},"One clearly ",{"type":54,"tag":69,"props":568,"children":569},{},[570],{"type":60,"value":571},"out-of-scope",{"type":60,"value":573}," request",{"type":54,"tag":63,"props":575,"children":576},{},[577],{"type":60,"value":578},"If every row passes at baseline, the dataset is too easy to teach you anything.",{"type":54,"tag":109,"props":580,"children":581},{},[],{"type":54,"tag":113,"props":583,"children":585},{"id":584},"_5-read-results-deterministic-vs-judged",[586],{"type":60,"value":587},"5. Read results: deterministic vs judged",{"type":54,"tag":63,"props":589,"children":590},{},[591,593,598],{"type":60,"value":592},"Two metric families answer different questions, and ",{"type":54,"tag":69,"props":594,"children":595},{},[596],{"type":60,"value":597},"the gap between them is\nthe diagnosis",{"type":60,"value":599},":",{"type":54,"tag":601,"props":602,"children":603},"table",{},[604,626],{"type":54,"tag":605,"props":606,"children":607},"thead",{},[608],{"type":54,"tag":609,"props":610,"children":611},"tr",{},[612,616,621],{"type":54,"tag":613,"props":614,"children":615},"th",{},[],{"type":54,"tag":613,"props":617,"children":618},{},[619],{"type":60,"value":620},"Answers",{"type":54,"tag":613,"props":622,"children":623},{},[624],{"type":60,"value":625},"Example",{"type":54,"tag":627,"props":628,"children":629},"tbody",{},[630,656],{"type":54,"tag":609,"props":631,"children":632},{},[633,639,647],{"type":54,"tag":634,"props":635,"children":636},"td",{},[637],{"type":60,"value":638},"Deterministic (from the trace)",{"type":54,"tag":634,"props":640,"children":641},{},[642],{"type":54,"tag":312,"props":643,"children":644},{},[645],{"type":60,"value":646},"Did it run?",{"type":54,"tag":634,"props":648,"children":649},{},[650],{"type":54,"tag":77,"props":651,"children":653},{"className":652},[],[654],{"type":60,"value":655},"tool_success_rate = 1.0",{"type":54,"tag":609,"props":657,"children":658},{},[659,664,672],{"type":54,"tag":634,"props":660,"children":661},{},[662],{"type":60,"value":663},"LLM-judged (rubric)",{"type":54,"tag":634,"props":665,"children":666},{},[667],{"type":54,"tag":312,"props":668,"children":669},{},[670],{"type":60,"value":671},"Was it right?",{"type":54,"tag":634,"props":673,"children":674},{},[675],{"type":54,"tag":77,"props":676,"children":678},{"className":677},[],[679],{"type":60,"value":680},"tool_use_quality = 0.25",{"type":54,"tag":63,"props":682,"children":683},{},[684,686,692,694,700,702,707],{"type":60,"value":685},"A real result: ",{"type":54,"tag":77,"props":687,"children":689},{"className":688},[],[690],{"type":60,"value":691},"tool_success_rate",{"type":60,"value":693}," 1.0 alongside ",{"type":54,"tag":77,"props":695,"children":697},{"className":696},[],[698],{"type":60,"value":699},"tool_use_quality",{"type":60,"value":701}," 0.25. Both\ncorrect. Every tool call executed without raising — and the agent kept calling\nthe ",{"type":54,"tag":312,"props":703,"children":704},{},[705],{"type":60,"value":706},"wrong tool at the wrong time",{"type":60,"value":708},". Deterministic metrics cannot see that;\njudged metrics can.",{"type":54,"tag":63,"props":710,"children":711},{},[712],{"type":60,"value":713},"Work top-down: headline scores → the specific failing rows → the judge's\nreasoning next to the actual tool calls → the source code.",{"type":54,"tag":63,"props":715,"children":716},{},[717,722],{"type":54,"tag":69,"props":718,"children":719},{},[720],{"type":60,"value":721},"A metric that scores perfectly has no headroom.",{"type":60,"value":723}," Say so and move on — a\ndisproven hypothesis is a real result, not a failed experiment.",{"type":54,"tag":109,"props":725,"children":726},{},[],{"type":54,"tag":113,"props":728,"children":730},{"id":729},"_6-hill-climbing",[731],{"type":60,"value":732},"6. Hill climbing",{"type":54,"tag":734,"props":735,"children":736},"ol",{},[737,749,754,759,771,781,791],{"type":54,"tag":129,"props":738,"children":739},{},[740,742,747],{"type":60,"value":741},"Pick the ",{"type":54,"tag":69,"props":743,"children":744},{},[745],{"type":60,"value":746},"weakest",{"type":60,"value":748}," metric, not the most interesting one.",{"type":54,"tag":129,"props":750,"children":751},{},[752],{"type":60,"value":753},"Find the specific rows that fail it and read what actually happened.",{"type":54,"tag":129,"props":755,"children":756},{},[757],{"type":60,"value":758},"Trace the cause into source — prompt, tool docstring, or architecture.",{"type":54,"tag":129,"props":760,"children":761},{},[762,764,769],{"type":60,"value":763},"Change ",{"type":54,"tag":69,"props":765,"children":766},{},[767],{"type":60,"value":768},"one",{"type":60,"value":770}," thing, so the delta is attributable.",{"type":54,"tag":129,"props":772,"children":773},{},[774,779],{"type":54,"tag":69,"props":775,"children":776},{},[777],{"type":60,"value":778},"Predict the effect out loud before re-running.",{"type":60,"value":780}," That makes it a test.",{"type":54,"tag":129,"props":782,"children":783},{},[784,789],{"type":54,"tag":69,"props":785,"children":786},{},[787],{"type":60,"value":788},"Restart the agent server before re-running",{"type":60,"value":790}," — see below. Skipping this is\nthe single easiest way to waste an entire iteration.",{"type":54,"tag":129,"props":792,"children":793},{},[794],{"type":60,"value":795},"Re-run and compare.",{"type":54,"tag":797,"props":798,"children":800},"h3",{"id":799},"restart-the-agent-between-iterations-and-verify-you-did",[801],{"type":60,"value":802},"Restart the agent between iterations, and verify you did",{"type":54,"tag":63,"props":804,"children":805},{},[806,808,813,815,820],{"type":60,"value":807},"A long-running agent server holds the agent module ",{"type":54,"tag":69,"props":809,"children":810},{},[811],{"type":60,"value":812},"in memory",{"type":60,"value":814},". Editing a\nprompt, a tool docstring, or a tool signature changes nothing for a server that\nis already running — it keeps serving the code it imported at startup. The\nevaluation then dutifully measures your ",{"type":54,"tag":312,"props":816,"children":817},{},[818],{"type":60,"value":819},"old",{"type":60,"value":821}," agent and reports no improvement,\nand you conclude the fix didn't work.",{"type":54,"tag":63,"props":823,"children":824},{},[825,827],{"type":60,"value":826},"Don't trust that the restart happened — a new server silently exits if the port\nis still held by the old one. ",{"type":54,"tag":69,"props":828,"children":829},{},[830],{"type":60,"value":831},"Compare timestamps:",{"type":54,"tag":223,"props":833,"children":837},{"className":834,"code":835,"language":836,"meta":228,"style":228},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","PID=$(ss -ltnp | grep 8501 | sed -E 's\u002F.*pid=([0-9]+).*\u002F\\1\u002F')\nps -o lstart= -p \"$PID\"                       # when did the server start?\ndate -r path\u002Fto\u002Ftools.py '+%H:%M:%S'          # when did you edit the code?\n","bash",[838],{"type":54,"tag":77,"props":839,"children":840},{"__ignoreMap":228},[841,918,962],{"type":54,"tag":234,"props":842,"children":843},{"class":236,"line":237},[844,850,856,862,868,873,878,884,888,893,898,903,908,913],{"type":54,"tag":234,"props":845,"children":847},{"style":846},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[848],{"type":60,"value":849},"PID",{"type":54,"tag":234,"props":851,"children":853},{"style":852},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[854],{"type":60,"value":855},"=$(",{"type":54,"tag":234,"props":857,"children":859},{"style":858},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[860],{"type":60,"value":861},"ss",{"type":54,"tag":234,"props":863,"children":865},{"style":864},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[866],{"type":60,"value":867}," -ltnp",{"type":54,"tag":234,"props":869,"children":870},{"style":852},[871],{"type":60,"value":872}," |",{"type":54,"tag":234,"props":874,"children":875},{"style":858},[876],{"type":60,"value":877}," grep",{"type":54,"tag":234,"props":879,"children":881},{"style":880},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[882],{"type":60,"value":883}," 8501",{"type":54,"tag":234,"props":885,"children":886},{"style":852},[887],{"type":60,"value":872},{"type":54,"tag":234,"props":889,"children":890},{"style":858},[891],{"type":60,"value":892}," sed",{"type":54,"tag":234,"props":894,"children":895},{"style":864},[896],{"type":60,"value":897}," -E",{"type":54,"tag":234,"props":899,"children":900},{"style":852},[901],{"type":60,"value":902}," '",{"type":54,"tag":234,"props":904,"children":905},{"style":864},[906],{"type":60,"value":907},"s\u002F.*pid=([0-9]+).*\u002F\\1\u002F",{"type":54,"tag":234,"props":909,"children":910},{"style":852},[911],{"type":60,"value":912},"'",{"type":54,"tag":234,"props":914,"children":915},{"style":852},[916],{"type":60,"value":917},")\n",{"type":54,"tag":234,"props":919,"children":920},{"class":236,"line":246},[921,926,931,936,941,946,951,956],{"type":54,"tag":234,"props":922,"children":923},{"style":858},[924],{"type":60,"value":925},"ps",{"type":54,"tag":234,"props":927,"children":928},{"style":864},[929],{"type":60,"value":930}," -o",{"type":54,"tag":234,"props":932,"children":933},{"style":864},[934],{"type":60,"value":935}," lstart=",{"type":54,"tag":234,"props":937,"children":938},{"style":864},[939],{"type":60,"value":940}," -p",{"type":54,"tag":234,"props":942,"children":943},{"style":852},[944],{"type":60,"value":945}," \"",{"type":54,"tag":234,"props":947,"children":948},{"style":846},[949],{"type":60,"value":950},"$PID",{"type":54,"tag":234,"props":952,"children":953},{"style":852},[954],{"type":60,"value":955},"\"",{"type":54,"tag":234,"props":957,"children":959},{"style":958},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[960],{"type":60,"value":961},"                       # when did the server start?\n",{"type":54,"tag":234,"props":963,"children":964},{"class":236,"line":255},[965,970,975,980,984,989,993],{"type":54,"tag":234,"props":966,"children":967},{"style":858},[968],{"type":60,"value":969},"date",{"type":54,"tag":234,"props":971,"children":972},{"style":864},[973],{"type":60,"value":974}," -r",{"type":54,"tag":234,"props":976,"children":977},{"style":864},[978],{"type":60,"value":979}," path\u002Fto\u002Ftools.py",{"type":54,"tag":234,"props":981,"children":982},{"style":852},[983],{"type":60,"value":902},{"type":54,"tag":234,"props":985,"children":986},{"style":864},[987],{"type":60,"value":988},"+%H:%M:%S",{"type":54,"tag":234,"props":990,"children":991},{"style":852},[992],{"type":60,"value":912},{"type":54,"tag":234,"props":994,"children":995},{"style":958},[996],{"type":60,"value":997},"          # when did you edit the code?\n",{"type":54,"tag":63,"props":999,"children":1000},{},[1001,1003,1008],{"type":60,"value":1002},"If the server start time is ",{"type":54,"tag":69,"props":1004,"children":1005},{},[1006],{"type":60,"value":1007},"older",{"type":60,"value":1009}," than your edit, it is serving stale code —\nkill it, start a new one, and check again before spending a run.",{"type":54,"tag":63,"props":1011,"children":1012},{},[1013,1015,1021],{"type":60,"value":1014},"This applies to whatever drives the agent over HTTP. In-process paths (importing\n",{"type":54,"tag":77,"props":1016,"children":1018},{"className":1017},[],[1019],{"type":60,"value":1020},"agent.py",{"type":60,"value":1022}," directly) pick changes up automatically; anything talking to a server\ndoes not.",{"type":54,"tag":797,"props":1024,"children":1026},{"id":1025},"treat-ai-root-cause-analysis-as-a-hypothesis-not-a-finding",[1027],{"type":60,"value":1028},"Treat AI root-cause analysis as a hypothesis, not a finding",{"type":54,"tag":63,"props":1030,"children":1031},{},[1032,1034,1039,1041,1046],{"type":60,"value":1033},"Automated analysis is good at spotting ",{"type":54,"tag":312,"props":1035,"children":1036},{},[1037],{"type":60,"value":1038},"that",{"type":60,"value":1040}," something is wrong and bad at\nbeing sure ",{"type":54,"tag":312,"props":1042,"children":1043},{},[1044],{"type":60,"value":1045},"why",{"type":60,"value":1047},". It will produce a specific, well-argued, source-citing\nexplanation that is confidently wrong — and it reads exactly like a correct one.",{"type":54,"tag":63,"props":1049,"children":1050},{},[1051,1053,1059,1061,1067,1069,1075],{"type":60,"value":1052},"A real case: analysis reported the agent was fabricating ",{"type":54,"tag":77,"props":1054,"children":1056},{"className":1055},[],[1057],{"type":60,"value":1058},"customer_id='123'",{"type":60,"value":1060},"\nfrom tool docstring examples, quoting the docstrings. Acting on it — adding an\ninstruction never to invent a customer id — made every metric worse\n(",{"type":54,"tag":77,"props":1062,"children":1064},{"className":1063},[],[1065],{"type":60,"value":1066},"instruction_following",{"type":60,"value":1068}," 0.89 → 0.61) and introduced tool-call loops. The\nsystem prompt turned out to inject ",{"type":54,"tag":77,"props":1070,"children":1072},{"className":1071},[],[1073],{"type":60,"value":1074},"Customer.get_customer(\"123\")",{"type":60,"value":1076}," as the\nsigned-in customer. Nothing was fabricated; the agent had been told to distrust\na legitimate value.",{"type":54,"tag":63,"props":1078,"children":1079},{},[1080,1085],{"type":54,"tag":69,"props":1081,"children":1082},{},[1083],{"type":60,"value":1084},"Before acting on a diagnosis, verify its claim against the source.",{"type":60,"value":1086}," Grep for\nthe value. Check whether the prompt already supplies it. Confirm the mechanism\nexists. One minute of checking beats a 25-minute run in the wrong direction.",{"type":54,"tag":63,"props":1088,"children":1089},{},[1090],{"type":60,"value":1091},"And when a fix makes things worse, that is a result: it usually means the\ndiagnosis was wrong, not that the fix was too small.",{"type":54,"tag":63,"props":1093,"children":1094},{},[1095],{"type":60,"value":1096},"Common causes worth checking, in rough order of how often they turn out to be it:",{"type":54,"tag":125,"props":1098,"children":1099},{},[1100,1117,1127,1137],{"type":54,"tag":129,"props":1101,"children":1102},{},[1103,1108,1110,1115],{"type":54,"tag":69,"props":1104,"children":1105},{},[1106],{"type":60,"value":1107},"Docstring examples read as facts.",{"type":60,"value":1109}," Mock values like ",{"type":54,"tag":77,"props":1111,"children":1113},{"className":1112},[],[1114],{"type":60,"value":1058},{"type":60,"value":1116}," in\nexamples get fabricated into real calls when the agent hits a gap. Add an\nexplicit prohibition; don't rely on the model inferring that examples are fake.",{"type":54,"tag":129,"props":1118,"children":1119},{},[1120,1125],{"type":54,"tag":69,"props":1121,"children":1122},{},[1123],{"type":60,"value":1124},"Undocumented rules.",{"type":60,"value":1126}," A limit enforced in code but absent from the prompt\nforces the agent to discover it by failing — wasted calls and latency.",{"type":54,"tag":129,"props":1128,"children":1129},{},[1130,1135],{"type":54,"tag":69,"props":1131,"children":1132},{},[1133],{"type":60,"value":1134},"Too many tools on one agent.",{"type":60,"value":1136}," Biases toward acting when the right move is\nto stop and answer.",{"type":54,"tag":129,"props":1138,"children":1139},{},[1140,1145,1147,1153],{"type":54,"tag":69,"props":1141,"children":1142},{},[1143],{"type":60,"value":1144},"Inconsistent return contracts.",{"type":60,"value":1146}," A tool annotated ",{"type":54,"tag":77,"props":1148,"children":1150},{"className":1149},[],[1151],{"type":60,"value":1152},"-> dict",{"type":60,"value":1154}," that returns a\nbare string on the error path makes the model spend reasoning tokens parsing it.",{"type":54,"tag":63,"props":1156,"children":1157},{},[1158],{"type":60,"value":1159},"Keep an iteration log: what changed, which metrics moved, and by how much.",{"type":54,"tag":109,"props":1161,"children":1162},{},[],{"type":54,"tag":113,"props":1164,"children":1166},{"id":1165},"_7-traps-that-make-a-run-silently-meaningless",[1167],{"type":60,"value":1168},"7. Traps that make a run silently meaningless",{"type":54,"tag":63,"props":1170,"children":1171},{},[1172,1177,1179,1184,1186,1191],{"type":54,"tag":69,"props":1173,"children":1174},{},[1175],{"type":60,"value":1176},"Nothing starts your agent for you.",{"type":60,"value":1178}," The eval posts to whatever is listening on\nthe port. A leftover server for a ",{"type":54,"tag":312,"props":1180,"children":1181},{},[1182],{"type":60,"value":1183},"different",{"type":60,"value":1185}," agent will accept the connection\nand fail every request — and the symptoms appear as confusing ",{"type":54,"tag":312,"props":1187,"children":1188},{},[1189],{"type":60,"value":1190},"metric",{"type":60,"value":1192}," errors.\nVerify the server's identity, not just that something is listening:",{"type":54,"tag":223,"props":1194,"children":1196},{"className":834,"code":1195,"language":836,"meta":228,"style":228},"ss -ltnp | grep 8501\ntr '\\0' ' ' \u003C \u002Fproc\u002F\u003CPID>\u002Fcmdline    # which agents dir was it launched with?\n",[1197],{"type":54,"tag":77,"props":1198,"children":1199},{"__ignoreMap":228},[1200,1224],{"type":54,"tag":234,"props":1201,"children":1202},{"class":236,"line":237},[1203,1207,1211,1215,1219],{"type":54,"tag":234,"props":1204,"children":1205},{"style":858},[1206],{"type":60,"value":861},{"type":54,"tag":234,"props":1208,"children":1209},{"style":864},[1210],{"type":60,"value":867},{"type":54,"tag":234,"props":1212,"children":1213},{"style":852},[1214],{"type":60,"value":872},{"type":54,"tag":234,"props":1216,"children":1217},{"style":858},[1218],{"type":60,"value":877},{"type":54,"tag":234,"props":1220,"children":1221},{"style":880},[1222],{"type":60,"value":1223}," 8501\n",{"type":54,"tag":234,"props":1225,"children":1226},{"class":236,"line":246},[1227,1231,1235,1240,1244,1248,1252,1257,1262,1267,1272,1277,1282,1287],{"type":54,"tag":234,"props":1228,"children":1229},{"style":858},[1230],{"type":60,"value":609},{"type":54,"tag":234,"props":1232,"children":1233},{"style":852},[1234],{"type":60,"value":902},{"type":54,"tag":234,"props":1236,"children":1237},{"style":864},[1238],{"type":60,"value":1239},"\\0",{"type":54,"tag":234,"props":1241,"children":1242},{"style":852},[1243],{"type":60,"value":912},{"type":54,"tag":234,"props":1245,"children":1246},{"style":852},[1247],{"type":60,"value":902},{"type":54,"tag":234,"props":1249,"children":1250},{"style":852},[1251],{"type":60,"value":902},{"type":54,"tag":234,"props":1253,"children":1254},{"style":852},[1255],{"type":60,"value":1256}," \u003C",{"type":54,"tag":234,"props":1258,"children":1259},{"style":864},[1260],{"type":60,"value":1261}," \u002Fproc\u002F",{"type":54,"tag":234,"props":1263,"children":1264},{"style":852},[1265],{"type":60,"value":1266},"\u003C",{"type":54,"tag":234,"props":1268,"children":1269},{"style":864},[1270],{"type":60,"value":1271},"PI",{"type":54,"tag":234,"props":1273,"children":1274},{"style":846},[1275],{"type":60,"value":1276},"D",{"type":54,"tag":234,"props":1278,"children":1279},{"style":852},[1280],{"type":60,"value":1281},">",{"type":54,"tag":234,"props":1283,"children":1284},{"style":864},[1285],{"type":60,"value":1286},"\u002Fcmdline",{"type":54,"tag":234,"props":1288,"children":1289},{"style":958},[1290],{"type":60,"value":1291},"    # which agents dir was it launched with?\n",{"type":54,"tag":63,"props":1293,"children":1294},{},[1295,1300,1301,1307,1309,1315,1317,1322],{"type":54,"tag":69,"props":1296,"children":1297},{},[1298],{"type":60,"value":1299},"Built-in tools are not tool calls.",{"type":60,"value":310},{"type":54,"tag":77,"props":1302,"children":1304},{"className":1303},[],[1305],{"type":60,"value":1306},"google_search",{"type":60,"value":1308},", code execution and\nsimilar produce grounding metadata, not ",{"type":54,"tag":77,"props":1310,"children":1312},{"className":1311},[],[1313],{"type":60,"value":1314},"functionCall",{"type":60,"value":1316}," events. Any metric reading\ntool interactions is ",{"type":54,"tag":69,"props":1318,"children":1319},{},[1320],{"type":60,"value":1321},"structurally unsatisfiable",{"type":60,"value":1323}," for an agent whose only tool\nis built in. Check for real Python function tools before writing tool-trajectory\nmetrics.",{"type":54,"tag":63,"props":1325,"children":1326},{},[1327,1332],{"type":54,"tag":69,"props":1328,"children":1329},{},[1330],{"type":60,"value":1331},"A phase can report success while every row failed.",{"type":60,"value":1333}," \"Captured N interactions\"\ncounts rows returned, not rows that succeeded. Check per-row status.",{"type":54,"tag":63,"props":1335,"children":1336},{},[1337,1342],{"type":54,"tag":69,"props":1338,"children":1339},{},[1340],{"type":60,"value":1341},"Comparison baselines are picked automatically.",{"type":60,"value":1343}," Analysis usually compares\nagainst the most recent previous run. Stale or broken runs left in the results\nfolder will silently poison your deltas — move them aside first.",{"type":54,"tag":63,"props":1345,"children":1346},{},[1347,1352,1354,1360],{"type":54,"tag":69,"props":1348,"children":1349},{},[1350],{"type":60,"value":1351},"Judge failures can be environment failures.",{"type":60,"value":1353}," An out-of-range SDK in the\nscoring environment can make judges return prose instead of JSON, which surfaces\nas ",{"type":54,"tag":77,"props":1355,"children":1357},{"className":1356},[],[1358],{"type":60,"value":1359},"400 INVALID_ARGUMENT — Error parsing JSON",{"type":60,"value":1361},". Keep agent runtime and eval\nscoring dependencies separate and in their supported ranges.",{"type":54,"tag":63,"props":1363,"children":1364},{},[1365,1370,1372,1376,1378,1384,1385,1391,1393,1399,1401,1407],{"type":54,"tag":69,"props":1366,"children":1367},{},[1368],{"type":60,"value":1369},"Use two virtualenvs: one for the agent, one for evaluation.",{"type":60,"value":1371}," They share\ndependencies (the ADK, the Vertex SDK) but need ",{"type":54,"tag":312,"props":1373,"children":1374},{},[1375],{"type":60,"value":1183},{"type":60,"value":1377}," versions of them.\nInstall the eval tooling into the agent's venv and the agent's lockfile wins:\nthe next ",{"type":54,"tag":77,"props":1379,"children":1381},{"className":1380},[],[1382],{"type":60,"value":1383},"uv sync",{"type":60,"value":364},{"type":54,"tag":77,"props":1386,"children":1388},{"className":1387},[],[1389],{"type":60,"value":1390},"uv run",{"type":60,"value":1392}," in that project re-resolves the environment and\ndowngrades a shared package underneath the eval tool, which then crashes on an\nimport that has nothing to do with the real problem. The supported layout is an\neval venv with the agent installed into it (",{"type":54,"tag":77,"props":1394,"children":1396},{"className":1395},[],[1397],{"type":60,"value":1398},"uv pip install -e .",{"type":60,"value":1400},"), while the\nagent's own venv serves the agent. If a run fails with a surprising\n",{"type":54,"tag":77,"props":1402,"children":1404},{"className":1403},[],[1405],{"type":60,"value":1406},"ModuleNotFoundError",{"type":60,"value":1408},", check installed versions before debugging anything else.",{"type":54,"tag":63,"props":1410,"children":1411},{},[1412,1417],{"type":54,"tag":69,"props":1413,"children":1414},{},[1415],{"type":60,"value":1416},"Empty results deserve suspicion, not a rerun.",{"type":60,"value":1418}," If scores are missing or rows\nare blank, find the upstream cause before spending another run.",{"type":54,"tag":1420,"props":1421,"children":1422},"style",{},[1423],{"type":60,"value":1424},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1426,"total":1601},[1427,1445,1461,1471,1478,1493,1515,1529,1540,1554,1571,1588],{"slug":1428,"name":1428,"fn":1429,"description":1430,"org":1431,"tags":1432,"stars":1442,"repoUrl":1443,"updatedAt":1444},"kb-search","search and extract local knowledge base documents","Allows listing, searching and extracting information from local knowledge base documents for information about tables\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1433,1436,1439],{"name":1434,"slug":1435,"type":16},"Documentation","documentation",{"name":1437,"slug":1438,"type":16},"Knowledge Base","knowledge-base",{"name":1440,"slug":1441,"type":16},"Search","search",8409,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fknowledge-catalog","2026-07-12T07:38:52.157375",{"slug":1446,"name":1447,"fn":1448,"description":1449,"org":1450,"tags":1451,"stars":1442,"repoUrl":1443,"updatedAt":1460},"knowledgecatalogdiscoveryagent","knowledge_catalog_discovery_agent","search and rank Knowledge Catalog data entries","Analyzes user queries, extracts relevant predicates, and utilizes Knowledge Catalog Search to find and rank the most relevant data entries. Engages with the user throughout the process.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1452,1455,1456,1459],{"name":1453,"slug":1454,"type":16},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":16},{"name":1457,"slug":1458,"type":16},"Knowledge Management","knowledge-management",{"name":1440,"slug":1441,"type":16},"2026-07-12T07:38:22.196851",{"slug":82,"name":82,"fn":1462,"description":1463,"org":1464,"tags":1465,"stars":24,"repoUrl":25,"updatedAt":1470},"execute high-performance agent evaluations","Executes high-performance agent evaluations, multi-turn UserSim simulations, and declarative metric grading aligned with google\u002Fagents-cli and the Quality Flywheel. Publishes benchmark artifacts to the GCS Evaluation Registry, executes automated head-to-head delta comparisons (--compare-to), and optimizes system instructions via ADK GEPA (Genetic Evolutionary Prompt Optimization). Use when running agent benchmarks, evaluating ADK\u002FFastAPI agents, diagnosing loss clusters, comparing prompt iterations, running GEPA prompt optimization, or serving evaluation dashboards. Don't use for raw agent code scaffolding (use google-agents-cli-scaffold) or infrastructure deployment (use google-agents-cli-deploy).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1466,1467,1468,1469],{"name":21,"slug":22,"type":16},{"name":14,"slug":15,"type":16},{"name":18,"slug":19,"type":16},{"name":9,"slug":8,"type":16},"2026-08-13T05:04:15.77511",{"slug":4,"name":4,"fn":5,"description":6,"org":1472,"tags":1473,"stars":24,"repoUrl":25,"updatedAt":26},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1474,1475,1476,1477],{"name":21,"slug":22,"type":16},{"name":14,"slug":15,"type":16},{"name":18,"slug":19,"type":16},{"name":9,"slug":8,"type":16},{"slug":1479,"name":1479,"fn":1480,"description":1481,"org":1482,"tags":1483,"stars":24,"repoUrl":25,"updatedAt":1492},"eval-breakdown","diagnose agent evaluation benchmark results","Performs an exhaustive, question-by-question narrative diagnostic breakdown of an agent-eval benchmark run by analyzing question_answer_log.md, eval_summary.json, and raw trajectory traces. Use when diagnosing low score causes, investigating the Memory Reuse vs. Traceability rubric clash, performing pre-release failure audits, or examining judge reasoning across individual scenarios. Don't use for running the benchmark CLI pipeline itself (use agent-eval) or automated genetic prompt tuning (use google-agents-cli-eval).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1484,1487,1490,1491],{"name":1485,"slug":1486,"type":16},"Code Analysis","code-analysis",{"name":1488,"slug":1489,"type":16},"Debugging","debugging",{"name":18,"slug":19,"type":16},{"name":9,"slug":8,"type":16},"2026-08-13T05:04:18.336276",{"slug":1494,"name":1494,"fn":1495,"description":1496,"org":1497,"tags":1498,"stars":1512,"repoUrl":1513,"updatedAt":1514},"contributing","contribute to Cloud Foundation Fabric","End-to-end workflow for contributing to Cloud Foundation Fabric: triaging GitHub issues, proactive feature development, validating with tests and Policy Troubleshooter, and submitting sanitized Pull Requests. Use when addressing a Fabric GitHub issue, developing a module or FAST stage change, or preparing a branch for a pull request.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1499,1502,1505,1508,1509],{"name":1500,"slug":1501,"type":16},"Automation","automation",{"name":1503,"slug":1504,"type":16},"Engineering","engineering",{"name":1506,"slug":1507,"type":16},"GitHub","github",{"name":9,"slug":8,"type":16},{"name":1510,"slug":1511,"type":16},"Pull Requests","pull-requests",2077,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fcloud-foundation-fabric","2026-07-31T06:23:36.935005",{"slug":1516,"name":1516,"fn":1517,"description":1518,"org":1519,"tags":1520,"stars":1512,"repoUrl":1513,"updatedAt":1528},"fabric-builder","generate Terraform code for Google Cloud","Generates idiomatic Cloud Foundation Fabric (CFF) Terraform code using CFF modules. Use when users ask to create GCP resources, use Fabric modules, or generate Terraform code for Google Cloud.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1521,1522,1525],{"name":9,"slug":8,"type":16},{"name":1523,"slug":1524,"type":16},"Infrastructure as Code","infrastructure-as-code",{"name":1526,"slug":1527,"type":16},"Terraform","terraform","2026-08-10T04:16:46.817883",{"slug":1530,"name":1530,"fn":1531,"description":1532,"org":1533,"tags":1534,"stars":1512,"repoUrl":1513,"updatedAt":1539},"fast-prerequisites","prepare prerequisites for FAST 0-org-setup","Guides the user step-by-step through the prerequisites for the FAST 0-org-setup stage, supporting both Standard GCP and Google Cloud Dedicated (GCD) environments. Use when a user asks to prepare or run prerequisites for 0-org-setup or bootstrap the FAST landing zone.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1535,1536],{"name":9,"slug":8,"type":16},{"name":1537,"slug":1538,"type":16},"Operations","operations","2026-08-06T05:36:21.590622",{"slug":1541,"name":1541,"fn":1542,"description":1543,"org":1544,"tags":1545,"stars":1551,"repoUrl":1552,"updatedAt":1553},"agent-aware-cli","design agent-aware command-line interfaces","Guide for designing and implementing command-line interfaces (CLIs) that are equally usable by human developers and automated coding agents. Use when the user wants to build a CLI, apply CLI best practices, or use Go with Cobra and Viper.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1546,1549,1550],{"name":1547,"slug":1548,"type":16},"CLI","cli",{"name":1503,"slug":1504,"type":16},{"name":9,"slug":8,"type":16},1178,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fvertex-ai-creative-studio","2026-07-12T07:39:08.41406",{"slug":1555,"name":1555,"fn":1556,"description":1557,"org":1558,"tags":1559,"stars":1551,"repoUrl":1552,"updatedAt":1570},"build-mcp-genmedia","build and configure GenAI MCP servers","Builds the mcp-genmedia Go MCP servers (nanobanana, veo, lyria, gemini-multimodal, chirp3-hd, avtool) from source and wires them into settings.json. Use this skill whenever the MCP tools are missing or broken — typically at the start of a new session, after a container restart, or when \u002Ftmp has been wiped. The prebuilt binaries in \u002Fworkspace\u002F.local\u002Fbin\u002F have no exec bit and live on a noexec mount; this skill compiles fresh executables into \u002Ftmp\u002Fbin\u002F where execution is allowed.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1560,1563,1564,1567],{"name":1561,"slug":1562,"type":16},"API Development","api-development",{"name":9,"slug":8,"type":16},{"name":1565,"slug":1566,"type":16},"LLM","llm",{"name":1568,"slug":1569,"type":16},"MCP","mcp","2026-07-12T07:39:10.911302",{"slug":1572,"name":1572,"fn":1573,"description":1574,"org":1575,"tags":1576,"stars":1551,"repoUrl":1552,"updatedAt":1587},"genmedia-audio-engineer","synthesize and mix audio content","Expert in audio synthesis, music generation, and mixing. Use when creating podcasts, background scores, or multi-track audio layering using mcp-chirp3-go, mcp-lyria-go, mcp-gemini-go, mcp-nanobanana-go, and mcp-avtool-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1577,1580,1583,1584],{"name":1578,"slug":1579,"type":16},"Audio","audio",{"name":1581,"slug":1582,"type":16},"Creative","creative",{"name":9,"slug":8,"type":16},{"name":1585,"slug":1586,"type":16},"Vertex AI","vertex-ai","2026-07-12T07:39:16.623879",{"slug":1589,"name":1589,"fn":1590,"description":1591,"org":1592,"tags":1593,"stars":1551,"repoUrl":1552,"updatedAt":1600},"genmedia-image-artist","generate and edit AI images","Expert in AI image generation and editing. Use when the user needs high-quality textures, character-consistent visuals, or image-to-image editing using mcp-nanobanana-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1594,1595,1596,1599],{"name":1581,"slug":1582,"type":16},{"name":9,"slug":8,"type":16},{"name":1597,"slug":1598,"type":16},"Image Generation","image-generation",{"name":1585,"slug":1586,"type":16},"2026-07-12T07:39:15.372822",84,{"items":1603,"total":255},[1604,1611,1618],{"slug":82,"name":82,"fn":1462,"description":1463,"org":1605,"tags":1606,"stars":24,"repoUrl":25,"updatedAt":1470},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1607,1608,1609,1610],{"name":21,"slug":22,"type":16},{"name":14,"slug":15,"type":16},{"name":18,"slug":19,"type":16},{"name":9,"slug":8,"type":16},{"slug":4,"name":4,"fn":5,"description":6,"org":1612,"tags":1613,"stars":24,"repoUrl":25,"updatedAt":26},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1614,1615,1616,1617],{"name":21,"slug":22,"type":16},{"name":14,"slug":15,"type":16},{"name":18,"slug":19,"type":16},{"name":9,"slug":8,"type":16},{"slug":1479,"name":1479,"fn":1480,"description":1481,"org":1619,"tags":1620,"stars":24,"repoUrl":25,"updatedAt":1492},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1621,1622,1623,1624],{"name":1485,"slug":1486,"type":16},{"name":1488,"slug":1489,"type":16},{"name":18,"slug":19,"type":16},{"name":9,"slug":8,"type":16}]