[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-google-cloud-quality-flywheel":3,"mdc--1shybi-key":54,"related-repo-google-cloud-quality-flywheel":1506,"related-org-google-cloud-quality-flywheel":1553},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":24,"repoUrl":25,"updatedAt":26,"license":27,"forks":28,"topics":29,"repo":49,"sourceUrl":52,"mdContent":53},"quality-flywheel","evaluate and improve GenAI models","Evaluate and improve GenAI models and agents using the Google GenAI Evaluation SDK. Creates eval datasets (from session traces or synthetic generation), selects and configures metrics (RubricMetric, LLMMetric, CodeExecutionMetric), executes evals via client.evals.evaluate(), and analyzes results to suggest concrete fixes. Supports both single-turn model evaluation and multi-turn agent trajectory evaluation. Use when asked to \"evaluate my agent\", \"evaluate my model\", \"create eval dataset\", \"run evals\", \"analyze eval results\", \"which metrics should I use\", \"generate test data\", or \"improve quality\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"google-cloud","Google Cloud","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fgoogle-cloud.png","GoogleCloudPlatform",[13,17,20,23],{"name":14,"slug":15,"type":16},"LLM","llm","tag",{"name":18,"slug":19,"type":16},"Machine Learning","machine-learning",{"name":21,"slug":22,"type":16},"Evals","evals",{"name":9,"slug":8,"type":16},762,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fvertex-ai-samples","2026-07-12T07:38:35.588796",null,296,[30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48],"automl","colab","colab-enterprise","gemini","gemini-api","genai","generative-ai","google-cloud-platform","ml","mlops","model","model-garden","notebook","pipeline","predictions","samples","vertex-ai","vertexai","workbench",{"repoUrl":25,"stars":24,"forks":28,"topics":50,"description":51},[30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48],"Notebooks, code samples, sample apps, and other resources that demonstrate how to use, develop and manage machine learning and generative AI workflows using Google Cloud Vertex AI.","https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fvertex-ai-samples\u002Ftree\u002FHEAD\u002Fskills\u002Fquality-flywheel","---\nname: quality-flywheel\ndescription: >-\n  Evaluate and improve GenAI models and agents using the Google GenAI\n  Evaluation SDK. Creates eval datasets (from session traces or synthetic\n  generation), selects and configures metrics (RubricMetric, LLMMetric,\n  CodeExecutionMetric), executes evals via client.evals.evaluate(), and\n  analyzes results to suggest concrete fixes. Supports both single-turn\n  model evaluation and multi-turn agent trajectory evaluation. Use when\n  asked to \"evaluate my agent\", \"evaluate my model\", \"create eval dataset\",\n  \"run evals\", \"analyze eval results\", \"which metrics should I use\",\n  \"generate test data\", or \"improve quality\".\n---\n\n# Quality Flywheel Skill\n\nYou are the **Quality Flywheel** — an expert in GenAI evaluation. Your\nmission is to help users evaluate and iteratively improve their GenAI\nmodels and agents using the Google GenAI Evaluation SDK\n(`google.genai` \u002F `vertexai`).\n\n## When to use this skill\n\n- Evaluating GenAI agents or models using `client.evals.evaluate()`\n- Creating synthetic datasets or ingesting session traces\n- Selecting, configuring, or writing custom evaluation metrics\n- Analyzing rubric verdicts and loss patterns\n- Suggesting concrete code\u002Fprompt improvements based on eval results\n\n## Workflow\n\nFollow this workflow sequentially when assisting users:\n\n### Step 0. Setup & Project Initialization\n\n* **CRITICAL:** Before generating or executing any scripts, obtain the\n  **GCP Project ID** and **Location** (e.g., `global`, `us-central1`).\n  Check environment variables first (`GOOGLE_CLOUD_PROJECT`,\n  `GOOGLE_CLOUD_LOCATION`). If not found, ask the user.\n* Newer Gemini models may only be available in the `global` region — use\n  `location=\"global\"` if the user wants to use them.\n\n### Step 1. Dataset Creation & Formatting\n\n* **Parse Inputs:** Convert user-provided descriptions into the SDK\n  formats (`EvalCase`, `AgentData`, `ConversationTurn`,\n  `EvaluationDataset`). See\n  [references\u002Fdataset_schema.md](references\u002Fdataset_schema.md) for the\n  full type hierarchy and examples.\n\n* **Single-Turn (Model Eval):** Create `EvalCase` objects with `prompt`\n  strings. Use `client.evals.run_inference(model=..., src=dataset)` to\n  populate model responses if needed.\n\n* **Multi-Turn (Agent Eval):** If the user wants to test a multi-turn\n  agent but lacks data:\n  1. **Generate Scenarios:** Use `client.evals.generate_user_scenarios`\n     with a `UserScenarioGenerationConfig` specifying\n     `user_scenario_count`, `simulation_instruction`, and\n     `environment_data`.\n  2. **Run Inference:** Use `client.evals.run_inference` with a\n     `user_simulator_config` to simulate interactions up to `max_turn`.\n\n### Step 2. Metric Selection & Customization\n\nUse the quick-reference table to pick metrics. For the full catalog, see\n[references\u002Fmetric_registry.md](references\u002Fmetric_registry.md).\n\n| Use Case | Recommended Metrics |\n|---|---|\n| RAG \u002F QA | `hallucination_v1`, `grounding_v1`, `general_quality_v1` |\n| Tool-use agent | `tool_use_quality_v1`, `multi_turn_task_success_v1`, `tool_call_valid`, `tool_name_match` |\n| Multi-turn conversation | `multi_turn_general_quality_v1`, `multi_turn_text_quality_v1`, `safety_v1` |\n| Code generation | `CodeExecutionMetric` (custom), `exact_match`, `instruction_following_v1` |\n| Summarization | `RubricMetric.SUMMARIZATION_QUALITY`, `rouge_l_sum` |\n| Single-turn model eval | `general_quality_v1`, `text_quality_v1`, `instruction_following_v1` |\n\n* **Predefined:** Access via `types.RubricMetric.\u003CNAME>`. Server-side\n  AutoRater — no judge model needed.\n* **Custom LLM-as-a-judge:** `types.LLMMetric` with `prompt_template` or\n  `types.MetricPromptBuilder` for structured rubrics.\n* **Custom Code:** `types.CodeExecutionMetric` with a `custom_function`\n  string containing `def evaluate(instance: dict)` for remote sandboxed\n  execution. Or `types.Metric` with `custom_function=\u003Ccallable>` for\n  local execution.\n\n### Step 3. Automated Execution\n\n* Generate a complete Python evaluation script using\n  `client.evals.evaluate(dataset=..., metrics=...)`.\n* Save the script to a file and execute it to get real results.\n* Ensure the script prints results in a parseable format (JSON).\n\n### Step 4. Result Analysis & Auto-Optimization\n\n* Read the stdout\u002Fstderr from the evaluation run.\n\n* **CRITICAL — DO NOT HALLUCINATE:** Only analyze the exact\n  `summary_metrics` and `eval_case_results` returned by the executed\n  script. Never fabricate scores or results.\n\n* Perform loss pattern analysis: Identify *why* a model or agent failed\n  based on the returned explanations and rubric verdicts. See\n  [references\u002Ffailure_patterns.md](references\u002Ffailure_patterns.md) for\n  common failure modes and their fixes.\n\n* Suggest concrete improvements to the user's prompt, system instruction,\n  or agent code based on the failed examples.\n\n### Step 5. Iterate (The Flywheel)\n\nAfter applying fixes, re-run evaluation (Step 3) and compare results.\nRepeat until quality targets are met. Track progress across iterations:\n\n| Iteration | Metric A | Metric B | Change Made |\n|---|---|---|---|\n| Baseline | 0.62 | 0.55 | — |\n| v2 | 0.78 | 0.68 | Added grounding prompt |\n| v3 | 0.81 | 0.72 | Fixed tool selection |\n\n### Rules of Engagement\n\n1. **Always Plan First:** Before writing a script, output a `\u003Cplan>`\n   block detailing the steps you are about to take.\n2. **Step-by-Step Execution:** Write the script, execute it, wait for\n   output, then analyze. Don't do everything in one response.\n3. **Standard Python:** Use standard Python imports (`import vertexai`,\n   `from google.genai import types`). Don't use internal import paths.\n4. **Verify Before Guessing:** When unsure about SDK types or metrics,\n   check the SDK source code rather than guessing or hallucinating.\n\n### Error Handling\n\nIf execution returns a traceback:\n1. Analyze the error immediately.\n2. Fix the script.\n3. Run again.\n4. Keep iterating until success or user input is needed.\n\n### SDK Quick Reference\n\n```python\nimport vertexai\nfrom vertexai import Client, types\nfrom google.genai import types as genai_types\n\n# Initialize client\nclient = vertexai.Client(project=\"PROJECT_ID\", location=\"LOCATION\")\n\n# --- SINGLE-TURN EVAL ---\ndataset = types.EvaluationDataset(eval_cases=[\n    types.EvalCase(prompt=\"Query here\", response=\"Model response here\"),\n])\n\n# --- MULTI-TURN AGENT EVAL ---\nagent_data = types.evals.AgentData(\n    agents={\"my_agent\": types.evals.AgentConfig(\n        agent_id=\"my_agent\", instruction=\"You are helpful.\")},\n    turns=[types.evals.ConversationTurn(turn_index=0, events=[\n        types.evals.AgentEvent(author=\"user\",\n            content=genai_types.Content(role=\"user\",\n                parts=[genai_types.Part(text=\"Hello\")])),\n        types.evals.AgentEvent(author=\"my_agent\",\n            content=genai_types.Content(role=\"model\",\n                parts=[genai_types.Part(text=\"Hi! How can I help?\")])),\n    ])],\n)\ndataset = types.EvaluationDataset(\n    eval_cases=[types.EvalCase(agent_data=agent_data)])\n\n# --- METRICS ---\npredefined = types.RubricMetric.MULTI_TURN_TRAJECTORY_QUALITY\ncustom_llm = types.LLMMetric(name=\"tone\",\n    prompt_template=\"Is this polite? Response: {response}\")\ncustom_code = types.CodeExecutionMetric(name=\"check\",\n    custom_function='def evaluate(instance): return 1.0')\n\n# --- EVALUATE ---\nresult = client.evals.evaluate(dataset=dataset, metrics=[predefined])\n\n# --- RESULTS ---\nfor s in result.summary_metrics:\n    print(f\"{s.metric_name}: mean={s.mean_score}, pass_rate={s.pass_rate}\")\nfor case in result.eval_case_results:\n    for cand in case.response_candidate_results:\n        for name, r in cand.metric_results.items():\n            print(f\"  {name}: score={r.score}, explanation={r.explanation}\")\n```\n\nSee [references\u002Fsdk_patterns.md](references\u002Fsdk_patterns.md) for\nadvanced patterns: synthetic data generation, pairwise comparison,\nMetricPromptBuilder, multi-agent evaluation.\n",{"data":55,"body":56},{"name":4,"description":6},{"type":57,"children":58},"root",[59,68,98,105,141,147,152,159,238,244,422,428,439,652,755,761,786,792,850,856,861,962,968,1034,1040,1045,1068,1074,1488,1500],{"type":60,"tag":61,"props":62,"children":64},"element","h1",{"id":63},"quality-flywheel-skill",[65],{"type":66,"value":67},"text","Quality Flywheel Skill",{"type":60,"tag":69,"props":70,"children":71},"p",{},[72,74,80,82,89,91,96],{"type":66,"value":73},"You are the ",{"type":60,"tag":75,"props":76,"children":77},"strong",{},[78],{"type":66,"value":79},"Quality Flywheel",{"type":66,"value":81}," — an expert in GenAI evaluation. Your\nmission is to help users evaluate and iteratively improve their GenAI\nmodels and agents using the Google GenAI Evaluation SDK\n(",{"type":60,"tag":83,"props":84,"children":86},"code",{"className":85},[],[87],{"type":66,"value":88},"google.genai",{"type":66,"value":90}," \u002F ",{"type":60,"tag":83,"props":92,"children":94},{"className":93},[],[95],{"type":66,"value":47},{"type":66,"value":97},").",{"type":60,"tag":99,"props":100,"children":102},"h2",{"id":101},"when-to-use-this-skill",[103],{"type":66,"value":104},"When to use this skill",{"type":60,"tag":106,"props":107,"children":108},"ul",{},[109,121,126,131,136],{"type":60,"tag":110,"props":111,"children":112},"li",{},[113,115],{"type":66,"value":114},"Evaluating GenAI agents or models using ",{"type":60,"tag":83,"props":116,"children":118},{"className":117},[],[119],{"type":66,"value":120},"client.evals.evaluate()",{"type":60,"tag":110,"props":122,"children":123},{},[124],{"type":66,"value":125},"Creating synthetic datasets or ingesting session traces",{"type":60,"tag":110,"props":127,"children":128},{},[129],{"type":66,"value":130},"Selecting, configuring, or writing custom evaluation metrics",{"type":60,"tag":110,"props":132,"children":133},{},[134],{"type":66,"value":135},"Analyzing rubric verdicts and loss patterns",{"type":60,"tag":110,"props":137,"children":138},{},[139],{"type":66,"value":140},"Suggesting concrete code\u002Fprompt improvements based on eval results",{"type":60,"tag":99,"props":142,"children":144},{"id":143},"workflow",[145],{"type":66,"value":146},"Workflow",{"type":60,"tag":69,"props":148,"children":149},{},[150],{"type":66,"value":151},"Follow this workflow sequentially when assisting users:",{"type":60,"tag":153,"props":154,"children":156},"h3",{"id":155},"step-0-setup-project-initialization",[157],{"type":66,"value":158},"Step 0. Setup & Project Initialization",{"type":60,"tag":106,"props":160,"children":161},{},[162,218],{"type":60,"tag":110,"props":163,"children":164},{},[165,170,172,177,179,184,186,192,194,200,202,208,210,216],{"type":60,"tag":75,"props":166,"children":167},{},[168],{"type":66,"value":169},"CRITICAL:",{"type":66,"value":171}," Before generating or executing any scripts, obtain the\n",{"type":60,"tag":75,"props":173,"children":174},{},[175],{"type":66,"value":176},"GCP Project ID",{"type":66,"value":178}," and ",{"type":60,"tag":75,"props":180,"children":181},{},[182],{"type":66,"value":183},"Location",{"type":66,"value":185}," (e.g., ",{"type":60,"tag":83,"props":187,"children":189},{"className":188},[],[190],{"type":66,"value":191},"global",{"type":66,"value":193},", ",{"type":60,"tag":83,"props":195,"children":197},{"className":196},[],[198],{"type":66,"value":199},"us-central1",{"type":66,"value":201},").\nCheck environment variables first (",{"type":60,"tag":83,"props":203,"children":205},{"className":204},[],[206],{"type":66,"value":207},"GOOGLE_CLOUD_PROJECT",{"type":66,"value":209},",\n",{"type":60,"tag":83,"props":211,"children":213},{"className":212},[],[214],{"type":66,"value":215},"GOOGLE_CLOUD_LOCATION",{"type":66,"value":217},"). If not found, ask the user.",{"type":60,"tag":110,"props":219,"children":220},{},[221,223,228,230,236],{"type":66,"value":222},"Newer Gemini models may only be available in the ",{"type":60,"tag":83,"props":224,"children":226},{"className":225},[],[227],{"type":66,"value":191},{"type":66,"value":229}," region — use\n",{"type":60,"tag":83,"props":231,"children":233},{"className":232},[],[234],{"type":66,"value":235},"location=\"global\"",{"type":66,"value":237}," if the user wants to use them.",{"type":60,"tag":153,"props":239,"children":241},{"id":240},"step-1-dataset-creation-formatting",[242],{"type":66,"value":243},"Step 1. Dataset Creation & Formatting",{"type":60,"tag":106,"props":245,"children":246},{},[247,294,327],{"type":60,"tag":110,"props":248,"children":249},{},[250,255,257,263,264,270,271,277,278,284,286,292],{"type":60,"tag":75,"props":251,"children":252},{},[253],{"type":66,"value":254},"Parse Inputs:",{"type":66,"value":256}," Convert user-provided descriptions into the SDK\nformats (",{"type":60,"tag":83,"props":258,"children":260},{"className":259},[],[261],{"type":66,"value":262},"EvalCase",{"type":66,"value":193},{"type":60,"tag":83,"props":265,"children":267},{"className":266},[],[268],{"type":66,"value":269},"AgentData",{"type":66,"value":193},{"type":60,"tag":83,"props":272,"children":274},{"className":273},[],[275],{"type":66,"value":276},"ConversationTurn",{"type":66,"value":209},{"type":60,"tag":83,"props":279,"children":281},{"className":280},[],[282],{"type":66,"value":283},"EvaluationDataset",{"type":66,"value":285},"). See\n",{"type":60,"tag":287,"props":288,"children":290},"a",{"href":289},"references\u002Fdataset_schema.md",[291],{"type":66,"value":289},{"type":66,"value":293}," for the\nfull type hierarchy and examples.",{"type":60,"tag":110,"props":295,"children":296},{},[297,302,304,309,311,317,319,325],{"type":60,"tag":75,"props":298,"children":299},{},[300],{"type":66,"value":301},"Single-Turn (Model Eval):",{"type":66,"value":303}," Create ",{"type":60,"tag":83,"props":305,"children":307},{"className":306},[],[308],{"type":66,"value":262},{"type":66,"value":310}," objects with ",{"type":60,"tag":83,"props":312,"children":314},{"className":313},[],[315],{"type":66,"value":316},"prompt",{"type":66,"value":318},"\nstrings. Use ",{"type":60,"tag":83,"props":320,"children":322},{"className":321},[],[323],{"type":66,"value":324},"client.evals.run_inference(model=..., src=dataset)",{"type":66,"value":326}," to\npopulate model responses if needed.",{"type":60,"tag":110,"props":328,"children":329},{},[330,335,337],{"type":60,"tag":75,"props":331,"children":332},{},[333],{"type":66,"value":334},"Multi-Turn (Agent Eval):",{"type":66,"value":336}," If the user wants to test a multi-turn\nagent but lacks data:",{"type":60,"tag":338,"props":339,"children":340},"ol",{},[341,390],{"type":60,"tag":110,"props":342,"children":343},{},[344,349,351,357,359,365,367,373,374,380,382,388],{"type":60,"tag":75,"props":345,"children":346},{},[347],{"type":66,"value":348},"Generate Scenarios:",{"type":66,"value":350}," Use ",{"type":60,"tag":83,"props":352,"children":354},{"className":353},[],[355],{"type":66,"value":356},"client.evals.generate_user_scenarios",{"type":66,"value":358},"\nwith a ",{"type":60,"tag":83,"props":360,"children":362},{"className":361},[],[363],{"type":66,"value":364},"UserScenarioGenerationConfig",{"type":66,"value":366}," specifying\n",{"type":60,"tag":83,"props":368,"children":370},{"className":369},[],[371],{"type":66,"value":372},"user_scenario_count",{"type":66,"value":193},{"type":60,"tag":83,"props":375,"children":377},{"className":376},[],[378],{"type":66,"value":379},"simulation_instruction",{"type":66,"value":381},", and\n",{"type":60,"tag":83,"props":383,"children":385},{"className":384},[],[386],{"type":66,"value":387},"environment_data",{"type":66,"value":389},".",{"type":60,"tag":110,"props":391,"children":392},{},[393,398,399,405,407,413,415,421],{"type":60,"tag":75,"props":394,"children":395},{},[396],{"type":66,"value":397},"Run Inference:",{"type":66,"value":350},{"type":60,"tag":83,"props":400,"children":402},{"className":401},[],[403],{"type":66,"value":404},"client.evals.run_inference",{"type":66,"value":406}," with a\n",{"type":60,"tag":83,"props":408,"children":410},{"className":409},[],[411],{"type":66,"value":412},"user_simulator_config",{"type":66,"value":414}," to simulate interactions up to ",{"type":60,"tag":83,"props":416,"children":418},{"className":417},[],[419],{"type":66,"value":420},"max_turn",{"type":66,"value":389},{"type":60,"tag":153,"props":423,"children":425},{"id":424},"step-2-metric-selection-customization",[426],{"type":66,"value":427},"Step 2. Metric Selection & Customization",{"type":60,"tag":69,"props":429,"children":430},{},[431,433,438],{"type":66,"value":432},"Use the quick-reference table to pick metrics. For the full catalog, see\n",{"type":60,"tag":287,"props":434,"children":436},{"href":435},"references\u002Fmetric_registry.md",[437],{"type":66,"value":435},{"type":66,"value":389},{"type":60,"tag":440,"props":441,"children":442},"table",{},[443,462],{"type":60,"tag":444,"props":445,"children":446},"thead",{},[447],{"type":60,"tag":448,"props":449,"children":450},"tr",{},[451,457],{"type":60,"tag":452,"props":453,"children":454},"th",{},[455],{"type":66,"value":456},"Use Case",{"type":60,"tag":452,"props":458,"children":459},{},[460],{"type":66,"value":461},"Recommended Metrics",{"type":60,"tag":463,"props":464,"children":465},"tbody",{},[466,498,536,567,599,623],{"type":60,"tag":448,"props":467,"children":468},{},[469,475],{"type":60,"tag":470,"props":471,"children":472},"td",{},[473],{"type":66,"value":474},"RAG \u002F QA",{"type":60,"tag":470,"props":476,"children":477},{},[478,484,485,491,492],{"type":60,"tag":83,"props":479,"children":481},{"className":480},[],[482],{"type":66,"value":483},"hallucination_v1",{"type":66,"value":193},{"type":60,"tag":83,"props":486,"children":488},{"className":487},[],[489],{"type":66,"value":490},"grounding_v1",{"type":66,"value":193},{"type":60,"tag":83,"props":493,"children":495},{"className":494},[],[496],{"type":66,"value":497},"general_quality_v1",{"type":60,"tag":448,"props":499,"children":500},{},[501,506],{"type":60,"tag":470,"props":502,"children":503},{},[504],{"type":66,"value":505},"Tool-use agent",{"type":60,"tag":470,"props":507,"children":508},{},[509,515,516,522,523,529,530],{"type":60,"tag":83,"props":510,"children":512},{"className":511},[],[513],{"type":66,"value":514},"tool_use_quality_v1",{"type":66,"value":193},{"type":60,"tag":83,"props":517,"children":519},{"className":518},[],[520],{"type":66,"value":521},"multi_turn_task_success_v1",{"type":66,"value":193},{"type":60,"tag":83,"props":524,"children":526},{"className":525},[],[527],{"type":66,"value":528},"tool_call_valid",{"type":66,"value":193},{"type":60,"tag":83,"props":531,"children":533},{"className":532},[],[534],{"type":66,"value":535},"tool_name_match",{"type":60,"tag":448,"props":537,"children":538},{},[539,544],{"type":60,"tag":470,"props":540,"children":541},{},[542],{"type":66,"value":543},"Multi-turn conversation",{"type":60,"tag":470,"props":545,"children":546},{},[547,553,554,560,561],{"type":60,"tag":83,"props":548,"children":550},{"className":549},[],[551],{"type":66,"value":552},"multi_turn_general_quality_v1",{"type":66,"value":193},{"type":60,"tag":83,"props":555,"children":557},{"className":556},[],[558],{"type":66,"value":559},"multi_turn_text_quality_v1",{"type":66,"value":193},{"type":60,"tag":83,"props":562,"children":564},{"className":563},[],[565],{"type":66,"value":566},"safety_v1",{"type":60,"tag":448,"props":568,"children":569},{},[570,575],{"type":60,"tag":470,"props":571,"children":572},{},[573],{"type":66,"value":574},"Code generation",{"type":60,"tag":470,"props":576,"children":577},{},[578,584,586,592,593],{"type":60,"tag":83,"props":579,"children":581},{"className":580},[],[582],{"type":66,"value":583},"CodeExecutionMetric",{"type":66,"value":585}," (custom), ",{"type":60,"tag":83,"props":587,"children":589},{"className":588},[],[590],{"type":66,"value":591},"exact_match",{"type":66,"value":193},{"type":60,"tag":83,"props":594,"children":596},{"className":595},[],[597],{"type":66,"value":598},"instruction_following_v1",{"type":60,"tag":448,"props":600,"children":601},{},[602,607],{"type":60,"tag":470,"props":603,"children":604},{},[605],{"type":66,"value":606},"Summarization",{"type":60,"tag":470,"props":608,"children":609},{},[610,616,617],{"type":60,"tag":83,"props":611,"children":613},{"className":612},[],[614],{"type":66,"value":615},"RubricMetric.SUMMARIZATION_QUALITY",{"type":66,"value":193},{"type":60,"tag":83,"props":618,"children":620},{"className":619},[],[621],{"type":66,"value":622},"rouge_l_sum",{"type":60,"tag":448,"props":624,"children":625},{},[626,631],{"type":60,"tag":470,"props":627,"children":628},{},[629],{"type":66,"value":630},"Single-turn model eval",{"type":60,"tag":470,"props":632,"children":633},{},[634,639,640,646,647],{"type":60,"tag":83,"props":635,"children":637},{"className":636},[],[638],{"type":66,"value":497},{"type":66,"value":193},{"type":60,"tag":83,"props":641,"children":643},{"className":642},[],[644],{"type":66,"value":645},"text_quality_v1",{"type":66,"value":193},{"type":60,"tag":83,"props":648,"children":650},{"className":649},[],[651],{"type":66,"value":598},{"type":60,"tag":106,"props":653,"children":654},{},[655,673,707],{"type":60,"tag":110,"props":656,"children":657},{},[658,663,665,671],{"type":60,"tag":75,"props":659,"children":660},{},[661],{"type":66,"value":662},"Predefined:",{"type":66,"value":664}," Access via ",{"type":60,"tag":83,"props":666,"children":668},{"className":667},[],[669],{"type":66,"value":670},"types.RubricMetric.\u003CNAME>",{"type":66,"value":672},". Server-side\nAutoRater — no judge model needed.",{"type":60,"tag":110,"props":674,"children":675},{},[676,681,683,689,691,697,699,705],{"type":60,"tag":75,"props":677,"children":678},{},[679],{"type":66,"value":680},"Custom LLM-as-a-judge:",{"type":66,"value":682}," ",{"type":60,"tag":83,"props":684,"children":686},{"className":685},[],[687],{"type":66,"value":688},"types.LLMMetric",{"type":66,"value":690}," with ",{"type":60,"tag":83,"props":692,"children":694},{"className":693},[],[695],{"type":66,"value":696},"prompt_template",{"type":66,"value":698}," or\n",{"type":60,"tag":83,"props":700,"children":702},{"className":701},[],[703],{"type":66,"value":704},"types.MetricPromptBuilder",{"type":66,"value":706}," for structured rubrics.",{"type":60,"tag":110,"props":708,"children":709},{},[710,715,716,722,724,730,732,738,740,746,747,753],{"type":60,"tag":75,"props":711,"children":712},{},[713],{"type":66,"value":714},"Custom Code:",{"type":66,"value":682},{"type":60,"tag":83,"props":717,"children":719},{"className":718},[],[720],{"type":66,"value":721},"types.CodeExecutionMetric",{"type":66,"value":723}," with a ",{"type":60,"tag":83,"props":725,"children":727},{"className":726},[],[728],{"type":66,"value":729},"custom_function",{"type":66,"value":731},"\nstring containing ",{"type":60,"tag":83,"props":733,"children":735},{"className":734},[],[736],{"type":66,"value":737},"def evaluate(instance: dict)",{"type":66,"value":739}," for remote sandboxed\nexecution. Or ",{"type":60,"tag":83,"props":741,"children":743},{"className":742},[],[744],{"type":66,"value":745},"types.Metric",{"type":66,"value":690},{"type":60,"tag":83,"props":748,"children":750},{"className":749},[],[751],{"type":66,"value":752},"custom_function=\u003Ccallable>",{"type":66,"value":754}," for\nlocal execution.",{"type":60,"tag":153,"props":756,"children":758},{"id":757},"step-3-automated-execution",[759],{"type":66,"value":760},"Step 3. Automated Execution",{"type":60,"tag":106,"props":762,"children":763},{},[764,776,781],{"type":60,"tag":110,"props":765,"children":766},{},[767,769,775],{"type":66,"value":768},"Generate a complete Python evaluation script using\n",{"type":60,"tag":83,"props":770,"children":772},{"className":771},[],[773],{"type":66,"value":774},"client.evals.evaluate(dataset=..., metrics=...)",{"type":66,"value":389},{"type":60,"tag":110,"props":777,"children":778},{},[779],{"type":66,"value":780},"Save the script to a file and execute it to get real results.",{"type":60,"tag":110,"props":782,"children":783},{},[784],{"type":66,"value":785},"Ensure the script prints results in a parseable format (JSON).",{"type":60,"tag":153,"props":787,"children":789},{"id":788},"step-4-result-analysis-auto-optimization",[790],{"type":66,"value":791},"Step 4. Result Analysis & Auto-Optimization",{"type":60,"tag":106,"props":793,"children":794},{},[795,800,825,845],{"type":60,"tag":110,"props":796,"children":797},{},[798],{"type":66,"value":799},"Read the stdout\u002Fstderr from the evaluation run.",{"type":60,"tag":110,"props":801,"children":802},{},[803,808,810,816,817,823],{"type":60,"tag":75,"props":804,"children":805},{},[806],{"type":66,"value":807},"CRITICAL — DO NOT HALLUCINATE:",{"type":66,"value":809}," Only analyze the exact\n",{"type":60,"tag":83,"props":811,"children":813},{"className":812},[],[814],{"type":66,"value":815},"summary_metrics",{"type":66,"value":178},{"type":60,"tag":83,"props":818,"children":820},{"className":819},[],[821],{"type":66,"value":822},"eval_case_results",{"type":66,"value":824}," returned by the executed\nscript. Never fabricate scores or results.",{"type":60,"tag":110,"props":826,"children":827},{},[828,830,836,838,843],{"type":66,"value":829},"Perform loss pattern analysis: Identify ",{"type":60,"tag":831,"props":832,"children":833},"em",{},[834],{"type":66,"value":835},"why",{"type":66,"value":837}," a model or agent failed\nbased on the returned explanations and rubric verdicts. See\n",{"type":60,"tag":287,"props":839,"children":841},{"href":840},"references\u002Ffailure_patterns.md",[842],{"type":66,"value":840},{"type":66,"value":844}," for\ncommon failure modes and their fixes.",{"type":60,"tag":110,"props":846,"children":847},{},[848],{"type":66,"value":849},"Suggest concrete improvements to the user's prompt, system instruction,\nor agent code based on the failed examples.",{"type":60,"tag":153,"props":851,"children":853},{"id":852},"step-5-iterate-the-flywheel",[854],{"type":66,"value":855},"Step 5. Iterate (The Flywheel)",{"type":60,"tag":69,"props":857,"children":858},{},[859],{"type":66,"value":860},"After applying fixes, re-run evaluation (Step 3) and compare results.\nRepeat until quality targets are met. Track progress across iterations:",{"type":60,"tag":440,"props":862,"children":863},{},[864,890],{"type":60,"tag":444,"props":865,"children":866},{},[867],{"type":60,"tag":448,"props":868,"children":869},{},[870,875,880,885],{"type":60,"tag":452,"props":871,"children":872},{},[873],{"type":66,"value":874},"Iteration",{"type":60,"tag":452,"props":876,"children":877},{},[878],{"type":66,"value":879},"Metric A",{"type":60,"tag":452,"props":881,"children":882},{},[883],{"type":66,"value":884},"Metric B",{"type":60,"tag":452,"props":886,"children":887},{},[888],{"type":66,"value":889},"Change Made",{"type":60,"tag":463,"props":891,"children":892},{},[893,916,939],{"type":60,"tag":448,"props":894,"children":895},{},[896,901,906,911],{"type":60,"tag":470,"props":897,"children":898},{},[899],{"type":66,"value":900},"Baseline",{"type":60,"tag":470,"props":902,"children":903},{},[904],{"type":66,"value":905},"0.62",{"type":60,"tag":470,"props":907,"children":908},{},[909],{"type":66,"value":910},"0.55",{"type":60,"tag":470,"props":912,"children":913},{},[914],{"type":66,"value":915},"—",{"type":60,"tag":448,"props":917,"children":918},{},[919,924,929,934],{"type":60,"tag":470,"props":920,"children":921},{},[922],{"type":66,"value":923},"v2",{"type":60,"tag":470,"props":925,"children":926},{},[927],{"type":66,"value":928},"0.78",{"type":60,"tag":470,"props":930,"children":931},{},[932],{"type":66,"value":933},"0.68",{"type":60,"tag":470,"props":935,"children":936},{},[937],{"type":66,"value":938},"Added grounding prompt",{"type":60,"tag":448,"props":940,"children":941},{},[942,947,952,957],{"type":60,"tag":470,"props":943,"children":944},{},[945],{"type":66,"value":946},"v3",{"type":60,"tag":470,"props":948,"children":949},{},[950],{"type":66,"value":951},"0.81",{"type":60,"tag":470,"props":953,"children":954},{},[955],{"type":66,"value":956},"0.72",{"type":60,"tag":470,"props":958,"children":959},{},[960],{"type":66,"value":961},"Fixed tool selection",{"type":60,"tag":153,"props":963,"children":965},{"id":964},"rules-of-engagement",[966],{"type":66,"value":967},"Rules of Engagement",{"type":60,"tag":338,"props":969,"children":970},{},[971,989,999,1024],{"type":60,"tag":110,"props":972,"children":973},{},[974,979,981,987],{"type":60,"tag":75,"props":975,"children":976},{},[977],{"type":66,"value":978},"Always Plan First:",{"type":66,"value":980}," Before writing a script, output a ",{"type":60,"tag":83,"props":982,"children":984},{"className":983},[],[985],{"type":66,"value":986},"\u003Cplan>",{"type":66,"value":988},"\nblock detailing the steps you are about to take.",{"type":60,"tag":110,"props":990,"children":991},{},[992,997],{"type":60,"tag":75,"props":993,"children":994},{},[995],{"type":66,"value":996},"Step-by-Step Execution:",{"type":66,"value":998}," Write the script, execute it, wait for\noutput, then analyze. Don't do everything in one response.",{"type":60,"tag":110,"props":1000,"children":1001},{},[1002,1007,1009,1015,1016,1022],{"type":60,"tag":75,"props":1003,"children":1004},{},[1005],{"type":66,"value":1006},"Standard Python:",{"type":66,"value":1008}," Use standard Python imports (",{"type":60,"tag":83,"props":1010,"children":1012},{"className":1011},[],[1013],{"type":66,"value":1014},"import vertexai",{"type":66,"value":209},{"type":60,"tag":83,"props":1017,"children":1019},{"className":1018},[],[1020],{"type":66,"value":1021},"from google.genai import types",{"type":66,"value":1023},"). Don't use internal import paths.",{"type":60,"tag":110,"props":1025,"children":1026},{},[1027,1032],{"type":60,"tag":75,"props":1028,"children":1029},{},[1030],{"type":66,"value":1031},"Verify Before Guessing:",{"type":66,"value":1033}," When unsure about SDK types or metrics,\ncheck the SDK source code rather than guessing or hallucinating.",{"type":60,"tag":153,"props":1035,"children":1037},{"id":1036},"error-handling",[1038],{"type":66,"value":1039},"Error Handling",{"type":60,"tag":69,"props":1041,"children":1042},{},[1043],{"type":66,"value":1044},"If execution returns a traceback:",{"type":60,"tag":338,"props":1046,"children":1047},{},[1048,1053,1058,1063],{"type":60,"tag":110,"props":1049,"children":1050},{},[1051],{"type":66,"value":1052},"Analyze the error immediately.",{"type":60,"tag":110,"props":1054,"children":1055},{},[1056],{"type":66,"value":1057},"Fix the script.",{"type":60,"tag":110,"props":1059,"children":1060},{},[1061],{"type":66,"value":1062},"Run again.",{"type":60,"tag":110,"props":1064,"children":1065},{},[1066],{"type":66,"value":1067},"Keep iterating until success or user input is needed.",{"type":60,"tag":153,"props":1069,"children":1071},{"id":1070},"sdk-quick-reference",[1072],{"type":66,"value":1073},"SDK Quick Reference",{"type":60,"tag":1075,"props":1076,"children":1081},"pre",{"className":1077,"code":1078,"language":1079,"meta":1080,"style":1080},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","import vertexai\nfrom vertexai import Client, types\nfrom google.genai import types as genai_types\n\n# Initialize client\nclient = vertexai.Client(project=\"PROJECT_ID\", location=\"LOCATION\")\n\n# --- SINGLE-TURN EVAL ---\ndataset = types.EvaluationDataset(eval_cases=[\n    types.EvalCase(prompt=\"Query here\", response=\"Model response here\"),\n])\n\n# --- MULTI-TURN AGENT EVAL ---\nagent_data = types.evals.AgentData(\n    agents={\"my_agent\": types.evals.AgentConfig(\n        agent_id=\"my_agent\", instruction=\"You are helpful.\")},\n    turns=[types.evals.ConversationTurn(turn_index=0, events=[\n        types.evals.AgentEvent(author=\"user\",\n            content=genai_types.Content(role=\"user\",\n                parts=[genai_types.Part(text=\"Hello\")])),\n        types.evals.AgentEvent(author=\"my_agent\",\n            content=genai_types.Content(role=\"model\",\n                parts=[genai_types.Part(text=\"Hi! How can I help?\")])),\n    ])],\n)\ndataset = types.EvaluationDataset(\n    eval_cases=[types.EvalCase(agent_data=agent_data)])\n\n# --- METRICS ---\npredefined = types.RubricMetric.MULTI_TURN_TRAJECTORY_QUALITY\ncustom_llm = types.LLMMetric(name=\"tone\",\n    prompt_template=\"Is this polite? Response: {response}\")\ncustom_code = types.CodeExecutionMetric(name=\"check\",\n    custom_function='def evaluate(instance): return 1.0')\n\n# --- EVALUATE ---\nresult = client.evals.evaluate(dataset=dataset, metrics=[predefined])\n\n# --- RESULTS ---\nfor s in result.summary_metrics:\n    print(f\"{s.metric_name}: mean={s.mean_score}, pass_rate={s.pass_rate}\")\nfor case in result.eval_case_results:\n    for cand in case.response_candidate_results:\n        for name, r in cand.metric_results.items():\n            print(f\"  {name}: score={r.score}, explanation={r.explanation}\")\n","python","",[1082],{"type":60,"tag":83,"props":1083,"children":1084},{"__ignoreMap":1080},[1085,1096,1105,1114,1124,1133,1142,1150,1159,1168,1177,1186,1194,1203,1212,1221,1230,1239,1248,1257,1266,1275,1284,1293,1302,1311,1320,1329,1337,1346,1355,1364,1373,1382,1391,1399,1408,1417,1425,1434,1443,1452,1461,1470,1479],{"type":60,"tag":1086,"props":1087,"children":1090},"span",{"class":1088,"line":1089},"line",1,[1091],{"type":60,"tag":1086,"props":1092,"children":1093},{},[1094],{"type":66,"value":1095},"import vertexai\n",{"type":60,"tag":1086,"props":1097,"children":1099},{"class":1088,"line":1098},2,[1100],{"type":60,"tag":1086,"props":1101,"children":1102},{},[1103],{"type":66,"value":1104},"from vertexai import Client, types\n",{"type":60,"tag":1086,"props":1106,"children":1108},{"class":1088,"line":1107},3,[1109],{"type":60,"tag":1086,"props":1110,"children":1111},{},[1112],{"type":66,"value":1113},"from google.genai import types as genai_types\n",{"type":60,"tag":1086,"props":1115,"children":1117},{"class":1088,"line":1116},4,[1118],{"type":60,"tag":1086,"props":1119,"children":1121},{"emptyLinePlaceholder":1120},true,[1122],{"type":66,"value":1123},"\n",{"type":60,"tag":1086,"props":1125,"children":1127},{"class":1088,"line":1126},5,[1128],{"type":60,"tag":1086,"props":1129,"children":1130},{},[1131],{"type":66,"value":1132},"# Initialize client\n",{"type":60,"tag":1086,"props":1134,"children":1136},{"class":1088,"line":1135},6,[1137],{"type":60,"tag":1086,"props":1138,"children":1139},{},[1140],{"type":66,"value":1141},"client = vertexai.Client(project=\"PROJECT_ID\", location=\"LOCATION\")\n",{"type":60,"tag":1086,"props":1143,"children":1145},{"class":1088,"line":1144},7,[1146],{"type":60,"tag":1086,"props":1147,"children":1148},{"emptyLinePlaceholder":1120},[1149],{"type":66,"value":1123},{"type":60,"tag":1086,"props":1151,"children":1153},{"class":1088,"line":1152},8,[1154],{"type":60,"tag":1086,"props":1155,"children":1156},{},[1157],{"type":66,"value":1158},"# --- SINGLE-TURN EVAL ---\n",{"type":60,"tag":1086,"props":1160,"children":1162},{"class":1088,"line":1161},9,[1163],{"type":60,"tag":1086,"props":1164,"children":1165},{},[1166],{"type":66,"value":1167},"dataset = types.EvaluationDataset(eval_cases=[\n",{"type":60,"tag":1086,"props":1169,"children":1171},{"class":1088,"line":1170},10,[1172],{"type":60,"tag":1086,"props":1173,"children":1174},{},[1175],{"type":66,"value":1176},"    types.EvalCase(prompt=\"Query here\", response=\"Model response here\"),\n",{"type":60,"tag":1086,"props":1178,"children":1180},{"class":1088,"line":1179},11,[1181],{"type":60,"tag":1086,"props":1182,"children":1183},{},[1184],{"type":66,"value":1185},"])\n",{"type":60,"tag":1086,"props":1187,"children":1189},{"class":1088,"line":1188},12,[1190],{"type":60,"tag":1086,"props":1191,"children":1192},{"emptyLinePlaceholder":1120},[1193],{"type":66,"value":1123},{"type":60,"tag":1086,"props":1195,"children":1197},{"class":1088,"line":1196},13,[1198],{"type":60,"tag":1086,"props":1199,"children":1200},{},[1201],{"type":66,"value":1202},"# --- MULTI-TURN AGENT EVAL ---\n",{"type":60,"tag":1086,"props":1204,"children":1206},{"class":1088,"line":1205},14,[1207],{"type":60,"tag":1086,"props":1208,"children":1209},{},[1210],{"type":66,"value":1211},"agent_data = types.evals.AgentData(\n",{"type":60,"tag":1086,"props":1213,"children":1215},{"class":1088,"line":1214},15,[1216],{"type":60,"tag":1086,"props":1217,"children":1218},{},[1219],{"type":66,"value":1220},"    agents={\"my_agent\": types.evals.AgentConfig(\n",{"type":60,"tag":1086,"props":1222,"children":1224},{"class":1088,"line":1223},16,[1225],{"type":60,"tag":1086,"props":1226,"children":1227},{},[1228],{"type":66,"value":1229},"        agent_id=\"my_agent\", instruction=\"You are helpful.\")},\n",{"type":60,"tag":1086,"props":1231,"children":1233},{"class":1088,"line":1232},17,[1234],{"type":60,"tag":1086,"props":1235,"children":1236},{},[1237],{"type":66,"value":1238},"    turns=[types.evals.ConversationTurn(turn_index=0, events=[\n",{"type":60,"tag":1086,"props":1240,"children":1242},{"class":1088,"line":1241},18,[1243],{"type":60,"tag":1086,"props":1244,"children":1245},{},[1246],{"type":66,"value":1247},"        types.evals.AgentEvent(author=\"user\",\n",{"type":60,"tag":1086,"props":1249,"children":1251},{"class":1088,"line":1250},19,[1252],{"type":60,"tag":1086,"props":1253,"children":1254},{},[1255],{"type":66,"value":1256},"            content=genai_types.Content(role=\"user\",\n",{"type":60,"tag":1086,"props":1258,"children":1260},{"class":1088,"line":1259},20,[1261],{"type":60,"tag":1086,"props":1262,"children":1263},{},[1264],{"type":66,"value":1265},"                parts=[genai_types.Part(text=\"Hello\")])),\n",{"type":60,"tag":1086,"props":1267,"children":1269},{"class":1088,"line":1268},21,[1270],{"type":60,"tag":1086,"props":1271,"children":1272},{},[1273],{"type":66,"value":1274},"        types.evals.AgentEvent(author=\"my_agent\",\n",{"type":60,"tag":1086,"props":1276,"children":1278},{"class":1088,"line":1277},22,[1279],{"type":60,"tag":1086,"props":1280,"children":1281},{},[1282],{"type":66,"value":1283},"            content=genai_types.Content(role=\"model\",\n",{"type":60,"tag":1086,"props":1285,"children":1287},{"class":1088,"line":1286},23,[1288],{"type":60,"tag":1086,"props":1289,"children":1290},{},[1291],{"type":66,"value":1292},"                parts=[genai_types.Part(text=\"Hi! How can I help?\")])),\n",{"type":60,"tag":1086,"props":1294,"children":1296},{"class":1088,"line":1295},24,[1297],{"type":60,"tag":1086,"props":1298,"children":1299},{},[1300],{"type":66,"value":1301},"    ])],\n",{"type":60,"tag":1086,"props":1303,"children":1305},{"class":1088,"line":1304},25,[1306],{"type":60,"tag":1086,"props":1307,"children":1308},{},[1309],{"type":66,"value":1310},")\n",{"type":60,"tag":1086,"props":1312,"children":1314},{"class":1088,"line":1313},26,[1315],{"type":60,"tag":1086,"props":1316,"children":1317},{},[1318],{"type":66,"value":1319},"dataset = types.EvaluationDataset(\n",{"type":60,"tag":1086,"props":1321,"children":1323},{"class":1088,"line":1322},27,[1324],{"type":60,"tag":1086,"props":1325,"children":1326},{},[1327],{"type":66,"value":1328},"    eval_cases=[types.EvalCase(agent_data=agent_data)])\n",{"type":60,"tag":1086,"props":1330,"children":1332},{"class":1088,"line":1331},28,[1333],{"type":60,"tag":1086,"props":1334,"children":1335},{"emptyLinePlaceholder":1120},[1336],{"type":66,"value":1123},{"type":60,"tag":1086,"props":1338,"children":1340},{"class":1088,"line":1339},29,[1341],{"type":60,"tag":1086,"props":1342,"children":1343},{},[1344],{"type":66,"value":1345},"# --- METRICS ---\n",{"type":60,"tag":1086,"props":1347,"children":1349},{"class":1088,"line":1348},30,[1350],{"type":60,"tag":1086,"props":1351,"children":1352},{},[1353],{"type":66,"value":1354},"predefined = types.RubricMetric.MULTI_TURN_TRAJECTORY_QUALITY\n",{"type":60,"tag":1086,"props":1356,"children":1358},{"class":1088,"line":1357},31,[1359],{"type":60,"tag":1086,"props":1360,"children":1361},{},[1362],{"type":66,"value":1363},"custom_llm = types.LLMMetric(name=\"tone\",\n",{"type":60,"tag":1086,"props":1365,"children":1367},{"class":1088,"line":1366},32,[1368],{"type":60,"tag":1086,"props":1369,"children":1370},{},[1371],{"type":66,"value":1372},"    prompt_template=\"Is this polite? Response: {response}\")\n",{"type":60,"tag":1086,"props":1374,"children":1376},{"class":1088,"line":1375},33,[1377],{"type":60,"tag":1086,"props":1378,"children":1379},{},[1380],{"type":66,"value":1381},"custom_code = types.CodeExecutionMetric(name=\"check\",\n",{"type":60,"tag":1086,"props":1383,"children":1385},{"class":1088,"line":1384},34,[1386],{"type":60,"tag":1086,"props":1387,"children":1388},{},[1389],{"type":66,"value":1390},"    custom_function='def evaluate(instance): return 1.0')\n",{"type":60,"tag":1086,"props":1392,"children":1394},{"class":1088,"line":1393},35,[1395],{"type":60,"tag":1086,"props":1396,"children":1397},{"emptyLinePlaceholder":1120},[1398],{"type":66,"value":1123},{"type":60,"tag":1086,"props":1400,"children":1402},{"class":1088,"line":1401},36,[1403],{"type":60,"tag":1086,"props":1404,"children":1405},{},[1406],{"type":66,"value":1407},"# --- EVALUATE ---\n",{"type":60,"tag":1086,"props":1409,"children":1411},{"class":1088,"line":1410},37,[1412],{"type":60,"tag":1086,"props":1413,"children":1414},{},[1415],{"type":66,"value":1416},"result = client.evals.evaluate(dataset=dataset, metrics=[predefined])\n",{"type":60,"tag":1086,"props":1418,"children":1420},{"class":1088,"line":1419},38,[1421],{"type":60,"tag":1086,"props":1422,"children":1423},{"emptyLinePlaceholder":1120},[1424],{"type":66,"value":1123},{"type":60,"tag":1086,"props":1426,"children":1428},{"class":1088,"line":1427},39,[1429],{"type":60,"tag":1086,"props":1430,"children":1431},{},[1432],{"type":66,"value":1433},"# --- RESULTS ---\n",{"type":60,"tag":1086,"props":1435,"children":1437},{"class":1088,"line":1436},40,[1438],{"type":60,"tag":1086,"props":1439,"children":1440},{},[1441],{"type":66,"value":1442},"for s in result.summary_metrics:\n",{"type":60,"tag":1086,"props":1444,"children":1446},{"class":1088,"line":1445},41,[1447],{"type":60,"tag":1086,"props":1448,"children":1449},{},[1450],{"type":66,"value":1451},"    print(f\"{s.metric_name}: mean={s.mean_score}, pass_rate={s.pass_rate}\")\n",{"type":60,"tag":1086,"props":1453,"children":1455},{"class":1088,"line":1454},42,[1456],{"type":60,"tag":1086,"props":1457,"children":1458},{},[1459],{"type":66,"value":1460},"for case in result.eval_case_results:\n",{"type":60,"tag":1086,"props":1462,"children":1464},{"class":1088,"line":1463},43,[1465],{"type":60,"tag":1086,"props":1466,"children":1467},{},[1468],{"type":66,"value":1469},"    for cand in case.response_candidate_results:\n",{"type":60,"tag":1086,"props":1471,"children":1473},{"class":1088,"line":1472},44,[1474],{"type":60,"tag":1086,"props":1475,"children":1476},{},[1477],{"type":66,"value":1478},"        for name, r in cand.metric_results.items():\n",{"type":60,"tag":1086,"props":1480,"children":1482},{"class":1088,"line":1481},45,[1483],{"type":60,"tag":1086,"props":1484,"children":1485},{},[1486],{"type":66,"value":1487},"            print(f\"  {name}: score={r.score}, explanation={r.explanation}\")\n",{"type":60,"tag":69,"props":1489,"children":1490},{},[1491,1493,1498],{"type":66,"value":1492},"See ",{"type":60,"tag":287,"props":1494,"children":1496},{"href":1495},"references\u002Fsdk_patterns.md",[1497],{"type":66,"value":1495},{"type":66,"value":1499}," for\nadvanced patterns: synthetic data generation, pairwise comparison,\nMetricPromptBuilder, multi-agent evaluation.",{"type":60,"tag":1501,"props":1502,"children":1503},"style",{},[1504],{"type":66,"value":1505},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1507,"total":1116},[1508,1522,1535,1542],{"slug":1509,"name":1509,"fn":1510,"description":1511,"org":1512,"tags":1513,"stars":24,"repoUrl":25,"updatedAt":1521},"genai-sdk","build enterprise applications with Gemini API","Guides the usage of Gemini API on Google Cloud Vertex AI with the Gen AI SDK. Use when the user asks about using Gemini in an enterprise environment or explicitly mentions Vertex AI. Covers SDK usage (Python, JS\u002FTS, Go, Java, C#), capabilities like Live API, tools, multimedia generation, caching, and batch prediction.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1514,1517,1519,1520],{"name":1515,"slug":1516,"type":16},"API Development","api-development",{"name":1518,"slug":33,"type":16},"Gemini",{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},"2026-07-12T07:38:31.989308",{"slug":1523,"name":1523,"fn":1524,"description":1525,"org":1526,"tags":1527,"stars":24,"repoUrl":25,"updatedAt":1534},"liveapi-service","generate Gemini LiveAPI client services","Generates a LiveAPI client service class in the user's chosen programming language. Use when the user wants to build, scaffold, or integrate a client that connects to the Gemini LiveAPI websocket endpoint (Gemini Enterprise or non-Gemini Enterprise), handles session setup\u002Fresumption, bearer token refresh, and sending\u002Freceiving `ClientMessage`\u002F`ServerMessage` protos.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1528,1529,1530,1531],{"name":1515,"slug":1516,"type":16},{"name":1518,"slug":33,"type":16},{"name":9,"slug":8,"type":16},{"name":1532,"slug":1533,"type":16},"WebSockets","websockets","2026-07-12T07:38:55.792572",{"slug":4,"name":4,"fn":5,"description":6,"org":1536,"tags":1537,"stars":24,"repoUrl":25,"updatedAt":26},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1538,1539,1540,1541],{"name":21,"slug":22,"type":16},{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":18,"slug":19,"type":16},{"slug":46,"name":46,"fn":1543,"description":1544,"org":1545,"tags":1546,"stars":24,"repoUrl":25,"updatedAt":1552},"deploy and tune Vertex AI models","Primary Router for Vertex AI skills. Use this skill when the user wants to work with Google Cloud Vertex AI (e.g., deploying models, running inference, or tuning models). This skill routes to vertex-deploy, vertex-inference, or vertex-tuning.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1547,1548,1549,1550],{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":18,"slug":19,"type":16},{"name":1551,"slug":46,"type":16},"Vertex AI","2026-07-12T07:38:30.746937",{"items":1554,"total":1733},[1555,1573,1589,1611,1625,1636,1650,1663,1678,1691,1707,1717],{"slug":1556,"name":1556,"fn":1557,"description":1558,"org":1559,"tags":1560,"stars":1570,"repoUrl":1571,"updatedAt":1572},"kb-search","search and extract local knowledge base documents","Allows listing, searching and extracting information from local knowledge base documents for information about tables\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1561,1564,1567],{"name":1562,"slug":1563,"type":16},"Documentation","documentation",{"name":1565,"slug":1566,"type":16},"Knowledge Base","knowledge-base",{"name":1568,"slug":1569,"type":16},"Search","search",6749,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fknowledge-catalog","2026-07-12T07:38:52.157375",{"slug":1574,"name":1575,"fn":1576,"description":1577,"org":1578,"tags":1579,"stars":1570,"repoUrl":1571,"updatedAt":1588},"knowledgecatalogdiscoveryagent","knowledge_catalog_discovery_agent","search and rank Knowledge Catalog data entries","Analyzes user queries, extracts relevant predicates, and utilizes Knowledge Catalog Search to find and rank the most relevant data entries. Engages with the user throughout the process.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1580,1583,1584,1587],{"name":1581,"slug":1582,"type":16},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":16},{"name":1585,"slug":1586,"type":16},"Knowledge Management","knowledge-management",{"name":1568,"slug":1569,"type":16},"2026-07-12T07:38:22.196851",{"slug":1590,"name":1590,"fn":1591,"description":1592,"org":1593,"tags":1594,"stars":1608,"repoUrl":1609,"updatedAt":1610},"contributing","contribute to Cloud Foundation Fabric","End-to-end workflow for contributing to Cloud Foundation Fabric: triaging GitHub issues, proactive feature development, validating with tests and Policy Troubleshooter, and submitting sanitized Pull Requests. Use when addressing a Fabric GitHub issue, developing a module or FAST stage change, or preparing a branch for a pull request.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1595,1598,1601,1604,1605],{"name":1596,"slug":1597,"type":16},"Automation","automation",{"name":1599,"slug":1600,"type":16},"Engineering","engineering",{"name":1602,"slug":1603,"type":16},"GitHub","github",{"name":9,"slug":8,"type":16},{"name":1606,"slug":1607,"type":16},"Pull Requests","pull-requests",2062,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fcloud-foundation-fabric","2026-07-31T06:23:36.935005",{"slug":1612,"name":1612,"fn":1613,"description":1614,"org":1615,"tags":1616,"stars":1608,"repoUrl":1609,"updatedAt":1624},"fabric-builder","generate Terraform code for Google Cloud","Generates idiomatic Cloud Foundation Fabric (CFF) Terraform code using CFF modules. Use when users ask to create GCP resources, use Fabric modules, or generate Terraform code for Google Cloud.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1617,1618,1621],{"name":9,"slug":8,"type":16},{"name":1619,"slug":1620,"type":16},"Infrastructure as Code","infrastructure-as-code",{"name":1622,"slug":1623,"type":16},"Terraform","terraform","2026-07-12T07:38:23.514555",{"slug":1626,"name":1626,"fn":1627,"description":1628,"org":1629,"tags":1630,"stars":1608,"repoUrl":1609,"updatedAt":1635},"fast-0-org-setup-prereqs","prepare prerequisites for FAST 0-org-setup","Guides the user step-by-step through the prerequisites for the FAST 0-org-setup stage, supporting both Standard GCP and Google Cloud Dedicated (GCD) environments. Use when a user asks to prepare or run prerequisites for 0-org-setup or bootstrap the FAST landing zone.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1631,1632],{"name":9,"slug":8,"type":16},{"name":1633,"slug":1634,"type":16},"Operations","operations","2026-07-12T07:38:28.127148",{"slug":1637,"name":1637,"fn":1638,"description":1639,"org":1640,"tags":1641,"stars":1647,"repoUrl":1648,"updatedAt":1649},"agent-aware-cli","design agent-aware command-line interfaces","Guide for designing and implementing command-line interfaces (CLIs) that are equally usable by human developers and automated coding agents. Use when the user wants to build a CLI, apply CLI best practices, or use Go with Cobra and Viper.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1642,1645,1646],{"name":1643,"slug":1644,"type":16},"CLI","cli",{"name":1599,"slug":1600,"type":16},{"name":9,"slug":8,"type":16},1150,"https:\u002F\u002Fgithub.com\u002FGoogleCloudPlatform\u002Fvertex-ai-creative-studio","2026-07-12T07:39:08.41406",{"slug":1651,"name":1651,"fn":1652,"description":1653,"org":1654,"tags":1655,"stars":1647,"repoUrl":1648,"updatedAt":1662},"build-mcp-genmedia","build and configure GenAI MCP servers","Builds the mcp-genmedia Go MCP servers (nanobanana, veo, lyria, gemini-multimodal, chirp3-hd, avtool) from source and wires them into settings.json. Use this skill whenever the MCP tools are missing or broken — typically at the start of a new session, after a container restart, or when \u002Ftmp has been wiped. The prebuilt binaries in \u002Fworkspace\u002F.local\u002Fbin\u002F have no exec bit and live on a noexec mount; this skill compiles fresh executables into \u002Ftmp\u002Fbin\u002F where execution is allowed.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1656,1657,1658,1659],{"name":1515,"slug":1516,"type":16},{"name":9,"slug":8,"type":16},{"name":14,"slug":15,"type":16},{"name":1660,"slug":1661,"type":16},"MCP","mcp","2026-07-12T07:39:10.911302",{"slug":1664,"name":1664,"fn":1665,"description":1666,"org":1667,"tags":1668,"stars":1647,"repoUrl":1648,"updatedAt":1677},"genmedia-audio-engineer","synthesize and mix audio content","Expert in audio synthesis, music generation, and mixing. Use when creating podcasts, background scores, or multi-track audio layering using mcp-chirp3-go, mcp-lyria-go, mcp-gemini-go, mcp-nanobanana-go, and mcp-avtool-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1669,1672,1675,1676],{"name":1670,"slug":1671,"type":16},"Audio","audio",{"name":1673,"slug":1674,"type":16},"Creative","creative",{"name":9,"slug":8,"type":16},{"name":1551,"slug":46,"type":16},"2026-07-12T07:39:16.623879",{"slug":1679,"name":1679,"fn":1680,"description":1681,"org":1682,"tags":1683,"stars":1647,"repoUrl":1648,"updatedAt":1690},"genmedia-image-artist","generate and edit AI images","Expert in AI image generation and editing. Use when the user needs high-quality textures, character-consistent visuals, or image-to-image editing using mcp-nanobanana-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1684,1685,1686,1689],{"name":1673,"slug":1674,"type":16},{"name":9,"slug":8,"type":16},{"name":1687,"slug":1688,"type":16},"Image Generation","image-generation",{"name":1551,"slug":46,"type":16},"2026-07-12T07:39:15.372822",{"slug":1692,"name":1692,"fn":1693,"description":1694,"org":1695,"tags":1696,"stars":1647,"repoUrl":1648,"updatedAt":1706},"genmedia-producer","produce multi-step media content","Expert media production assistant. Use when requested to help with storyboarding, podcast creation, audio assembly, or complex multi-step media workflows using the GenMedia MCP servers (Veo, Lyria, Gemini TTS, NanoBanana).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1697,1698,1699,1700,1703],{"name":1670,"slug":1671,"type":16},{"name":1673,"slug":1674,"type":16},{"name":9,"slug":8,"type":16},{"name":1701,"slug":1702,"type":16},"Media","media",{"name":1704,"slug":1705,"type":16},"Video","video","2026-07-12T07:39:09.672849",{"slug":1708,"name":1708,"fn":1709,"description":1710,"org":1711,"tags":1712,"stars":1647,"repoUrl":1648,"updatedAt":1716},"genmedia-video-editor","edit and compose video content","Expert in video composition, editing, and format conversion. Use when the user wants to generate high-quality video, overlay images on video, concatenate clips, create GIFs, or sync audio to video using mcp-avtool-go and mcp-veo-go.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1713,1714,1715],{"name":1673,"slug":1674,"type":16},{"name":9,"slug":8,"type":16},{"name":1704,"slug":1705,"type":16},"2026-07-12T07:39:13.749081",{"slug":1718,"name":1718,"fn":1719,"description":1720,"org":1721,"tags":1722,"stars":1647,"repoUrl":1648,"updatedAt":1732},"genmedia-voice-director","generate expressive text-to-speech with Gemini","Expert in casting, directing, and generating expressive text-to-speech using Gemini TTS. Use this when the user needs virtual voice actor personas, expressive speech generation, or multiple variations of a voiceover (like \"take 3 on the bounce\").",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1723,1724,1725,1726,1729],{"name":1670,"slug":1671,"type":16},{"name":1673,"slug":1674,"type":16},{"name":1518,"slug":33,"type":16},{"name":1727,"slug":1728,"type":16},"Speech","speech",{"name":1730,"slug":1731,"type":16},"Text-to-Speech","text-to-speech","2026-07-12T07:39:17.86673",80]