Agent Evaluation & Continuous Optimization Framework

This skill defines the end-to-end evaluation, benchmarking, automated optimization, and publication workflow using the agent-eval CLI Pipeline aligned with google/agents-cli and the ADK Quality Flywheel (adk.dev/optimize).

1. Reference Architecture & Deep Guides

2. The Standard agent-eval run Benchmark Command

Always execute benchmark sweeps using the standardized --feature, --tag, and --publish taxonomy against an active API server endpoint (--base-url):

export AGENT_EVAL_NO_PAUSES=1
export GOOGLE_GENAI_USE_VERTEXAI=1
export GOOGLE_CLOUD_PROJECT=<PROJECT_ID>

agent-eval run \
  --agent-dir app \
  --base-url http://localhost:8080 \
  --feature "<feature_or_branch_name>" \
  --tag "<short_iteration_tag>" \
  --description "<one-line summary of changes tested>" \
  --sim-parallelism 6 \
  --publish \
  --compare-to "<baseline_run_id_or_path>"

Key CLI Flags & Defaults

• --feature: Git feature branch or capability under test (defaults to active git branch).
• --tag: Concise iteration slug (e.g. direct-bypass-v1, calibrated-prompt-v2).
• --description: Human-readable context saved into eval_summary.json and rendered in the dashboard.
• --sim-parallelism 6: Runs 6 scenarios in parallel, cutting multi-turn sweeps down to ~2.5 minutes.
• --publish: Automatically syncs the entire output run directory to Google Cloud Storage (gs://<PROJECT_ID>-eval-artifacts/runs/<run_id>/).
• --compare-to: Resolves a baseline run (locally or directly from GCS) and generates delta percentage scorecards.

3. The 5-Stage Quality Flywheel Loop

1. Prepare Data (tests/eval/dataset.jsonl)
       └──► 2. Simulate & Generate Traces (agent-eval simulate --sim-parallelism 6)
              └──► 3. Grade Traces (agent-eval evaluate via eval_config.yaml)
                     └──► 4. Analyze & Compare (agent-eval analyze --compare-to <baseline>)
                            └──► 5. Optimize via GEPA (agent-eval optimize --optimizer gepa)

Stage 1: Prepare Data (dataset.jsonl)

• Ensure test scenarios are defined in tests/eval/dataset.jsonl (see references/dataset_schema.md).

Stage 2 & 3: Run Inference & Grade Traces

• agent-eval run executes inference and grades traces with Vertex AI AutoRaters and custom domain rubrics in one pass.

Stage 4: Analyze Failures & Compare Runs

• Ingests eval_summary.json and report.html to compute run-to-run deltas:

agent-eval analyze --results-dir tests/eval/results/<run_id> --compare-to <baseline_run_id>

Stage 5: Automated Prompt Optimization (ADK GEPA)

• When prompts underperform, run ADK's GEPARootAgentPromptOptimizer rather than manual trial-and-error:

agent-eval optimize --agent-dir app --optimizer gepa --target-metric business_logic_adherence --generations 5

4. Central GCS Evaluation Registry & Dashboard Access

All runs published with --publish are indexed in real time by the Evaluation Registry Viewer:

• 🌐 Local / Network Reverse Proxy: 👉 http://localhost:8550
• ☁️ Cloud Run Endpoint: https://<EVAL_VIEWER_SERVICE>.run.app
• 🔗 Direct Deep Link: http://localhost:8550/?run_id=<RUN_ID>
• 📄 Direct Standalone HTML Report: http://localhost:8550/report/<RUN_ID>

5. Critical Operational Gotchas & Guardrails

• Suppress Click Aborts (AGENT_EVAL_NO_PAUSES=1): Always export AGENT_EVAL_NO_PAUSES=1 in headless scripts, subagents, and tests to prevent Click prompts from hanging on stdin.
• Always Use --base-url Instead of --in-process: OpenTelemetry global tracers cannot be safely forked across multi-process workers in --in-process mode. Running against an active --base-url enables full 6-worker concurrency.
• PyOpenSSL Multithreading Locks: Batch LLM grading can trigger Context has already been used to create a Connection. Ensure urllib3.contrib.pyopenssl.extract_from_urllib3() is called to delegate to Python's native ssl engine.
• Clear Background Ports Before Starting Servers: Run fuser -k 8080/tcp (or 8502/tcp) before launching new uvicorn or adk api_server daemons to avoid [Errno 98] address already in use.
• HTML Report Type Safety: When sorting iterations in _build_iterations_data, ensure _natural_key returns homogeneous (float, str) tuples to prevent string-vs-int comparison errors.