[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-grafana-agento11y-test-starter":3,"mdc--o9lf3d-key":34,"related-repo-grafana-agento11y-test-starter":3633,"related-org-grafana-agento11y-test-starter":3729},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":29,"sourceUrl":32,"mdContent":33},"agento11y-test-starter","build and run agent test suites","Use early in an AI-agent project — before ship, before real traffic — to build a starter test suite for the agent and run it offline. Reads the agent's own code (system prompt, tools, task), writes a labeled draft suite of test cases (happy\u002Fedge\u002Fadversarial) grounded in real lines, and recommends how to score each case (the evaluators\u002Fjudges the offline runner uses). Assesses how runnable the agent is: for an easily-invoked agent it generates a runner stub (run_experiment.py) with two holes to fill and can optionally run it (only with permission, only against the endpoint the developer configured); for agents needing a harness or full runtime it points to the existing eval infra. It runs OFFLINE and never creates tenant-level evaluators, rules, or guards — that is `agento11y-prod-setup`, for a deployed agent with real traffic. Trigger on phrases like \"how do I test my agent before shipping\", \"write test cases for my agent\", \"set up tests for my agent\", \"check my agent before prod\", \"I have no traffic yet, how do I evaluate it\", \"test my agent offline\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"grafana","Grafana","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fgrafana.jpg",[12,16,19,20],{"name":13,"slug":14,"type":15},"QA","qa","tag",{"name":17,"slug":18,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":21,"slug":22,"type":15},"Testing","testing",430,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgcx","2026-07-31T05:53:51.62785",null,29,[],{"repoUrl":24,"stars":23,"forks":27,"topics":30,"description":31},[],"A CLI for managing Grafana Cloud resources. Optimized for agentic usage.","https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgcx\u002Ftree\u002FHEAD\u002Fclaude-plugin\u002Fskills\u002Fagento11y-test-starter","---\nname: agento11y-test-starter\ndescription: >\n  Use early in an AI-agent project — before ship, before real traffic — to build a starter\n  test suite for the agent and run it offline. Reads the agent's own code (system prompt, tools,\n  task), writes a labeled draft suite of test cases (happy\u002Fedge\u002Fadversarial) grounded in real\n  lines, and recommends how to score each case (the evaluators\u002Fjudges the offline runner uses).\n  Assesses how runnable the agent is: for an easily-invoked agent it generates a runner stub\n  (run_experiment.py) with two holes to fill and can optionally run it (only with permission, only\n  against the endpoint the developer configured); for agents needing a harness or full runtime it\n  points to the existing eval infra. It runs OFFLINE and never creates tenant-level evaluators,\n  rules, or guards — that is `agento11y-prod-setup`, for a deployed agent with real traffic.\n  Trigger on phrases like \"how do I test my agent before shipping\", \"write test cases for my\n  agent\", \"set up tests for my agent\", \"check my agent before prod\", \"I have no traffic yet, how\n  do I evaluate it\", \"test my agent offline\".\n---\n\n# agento11y test starter\n\nHelp a developer test an AI agent **before it ships** — while there is no real traffic yet. The\nhard part isn't running the test — it's having **cases to test against** and knowing **how to\nscore them**, grounded in the agent's actual code.\n\n> Scope: this is the **pre-production, offline** skill — it writes test cases and a local runner,\n> and never touches the tenant. Once the agent is deployed and has real traffic, setting up online\n> eval rules + guards on that traffic is a different skill, `agento11y-prod-setup`.\n\nAlways produce:\n\n1. A ranked, justified **evaluator recommendation** for this agent.\n2. A starter **suite YAML** the developer reviews and extends.\n\nThen, depending on how runnable the agent is (Step 1):\n\n3. For an easily-invoked agent, a **runner stub** (`run_experiment.py`) that wires the suite\n   to the SDK with two holes to fill — and optionally run it (Step 6), only with permission.\n   For an agent that needs a harness or full runtime, point to the existing eval infra instead\n   of a runner that can't actually call it.\n\nThis skill is language-agnostic — the reading, recommending, and YAML it produces do not\ndepend on the agent's language. What differs is how runnable the agent is: recommendations +\nYAML always apply, but the runner (Step 4) and the optional run (Step 6) adapt to whether the\nagent has a clean function seam or needs a harness \u002F full stack. For deeper run-side patterns\n(binding existing generations, cross-process verifiers) point to the per-language run skill\n(Python: `agento11y-experiments`).\n\n> Note: `agento11y-experiments` currently ships in the grafana\u002Fagento11y repo\n> (`python\u002Fskills\u002Fagento11y-experiments\u002F`), not in this gcx bundle yet — install it from there for\n> now. Consolidating it into the gcx bundle is pending.\n\n## Prerequisites\n\nThe generated runner imports the Agent Observability SDK. Install it in the agent's environment\nbefore running (Step 6):\n\n- **Python:** `pip install agento11y python-dotenv` (the experiments API lives in\n  `agento11y.experiments`; the runner uses `python-dotenv` to load the agent's `.env`, and it is\n  not a dependency of `agento11y`)\n- **Go:** add `github.com\u002Fgrafana\u002Fagento11y\u002Fgo`\n\nOnly needed to *run* the suite (Step 6). Reading the agent, recommending evaluators, and writing\nthe suite YAML (Steps 1–5) need nothing installed.\n\n## Rules\n\n- Do not create, enable, or modify evaluators, rules, or guards in any Agent Observability tenant. No\n  control-plane writes. (Running an offline experiment only publishes that run's scores — it\n  does not create tenant-level evaluators\u002Frules\u002Fguards — but only do it via Step 6.)\n- Do not rewrite the agent's prompt, optimize, or redeploy.\n- Never run the experiment without asking first (Step 6). Never run against a target the\n  developer did not configure — use their `AGENTO11Y_ENDPOINT` and `AGENTO11Y_AUTH_TOKEN`; if the\n  endpoint isn't set, ask for it, do not invent one.\n- Never mint, generate, or store credentials. The developer owns the Grafana Cloud ingestion\n  token; read it from the environment (a gitignored `.env` or an exported env var they supply\n  themselves) — **do not ask them to paste a secret token into the chat** (it is captured in the\n  transcript), and do not create one.\n- Never present the generated cases as validated. They are a draft to review and extend.\n- **The `llm_judge` uses the LLM provider the agent already uses — don't add a new one.** If the\n  agent calls OpenAI, the judge calls OpenAI; if Anthropic, Anthropic. Do NOT default the judge to\n  `litellm` (or any other provider SDK) when the app doesn't already depend on it — that adds a\n  dependency and a second provider just for scoring. Reuse the agent's existing client\u002FSDK.\n- **The target is Grafana Cloud.** Publishing scores needs `AGENTO11Y_ENDPOINT` **and**\n  `AGENTO11Y_AUTH_TOKEN` (the ingestion key from the Connection page) — the SDK raises before\n  making any request if the token is empty, so both are always required. Read the endpoint from an\n  existing `AGENTO11Y_ENDPOINT` \u002F `.env` \u002F sibling app; never invent one or mint a token.\n- If a required input is missing (entrypoint, prompt, tools), ask the developer — don't guess.\n\n## Step 1 — Read the agent\n\nFind and read these in the target repo, and record the file path and line range of each:\n\n1. The agent entrypoint (where the model is invoked).\n2. The system prompt \u002F instructions.\n3. The tool \u002F function definitions the agent can call.\n4. One or two real user requests and what a correct answer looks like.\n5. **The LLM provider and the model** — two distinct things; read them off how the client is\n   constructed and cite the line:\n   - **Provider** (who serves the LLM) → which SDK\u002Fclient the judge reuses: e.g.\n     `from openai import OpenAI` → OpenAI; `from anthropic import Anthropic` → Anthropic.\n   - **Model** (the specific id, e.g. `gpt-4o-mini`, `claude-sonnet-5`) → the `MODEL_NAME` the runner uses.\n   State both explicitly, and have the judge use the **same provider** as the agent (never a\n   different one). If you can't tell from the code, ask; don't default to a provider the app doesn't use.\n   - **Do NOT assume where the provider API key lives.** The app already loads it *somehow* — an env\n     var, a `.env` (`load_dotenv()`), a secret manager, an explicit `client(api_key=...)`. The\n     runner should reuse the **same mechanism** (it already calls `load_dotenv()`, and provider SDKs\n     read their standard env var themselves). Don't hardcode a key, and don't tell the developer to\n     \"set `OPENAI_API_KEY`\" as if that's the only way — if the run can't find the key, ask them how\n     their app loads it and mirror that.\n\nEvery recommendation must cite one of these locations.\n\nAlso assess **runnability** — how hard it is to invoke this agent for one input and get its\noutput — because it decides what Step 4 produces. Classify it as one of:\n\n- **easy — clean function seam.** A single call takes an input and returns text, with no live\n  services (a small Python\u002FTS agent, an LLM call behind one function). The generated runner\n  works as-is.\n- **in-process — needs a harness.** No 3-line call, but there is an injectable seam (e.g. a\n  Go engine whose LLM client is an interface you can fake, callable from a test binary).\n  Runnable in-process for a smoke\u002Fbehavioral eval, but not a 3-line Python `run_agent`.\n- **full-stack — needs the whole runtime.** The agent needs live backends\u002Fauth\u002Fqueues and is\n  exercised over HTTP against a running stack (tools that hit real datasources, a long\n  multi-step loop). Existing eval infra likely runs it via a dedicated harness and polling,\n  not a function.\n\nSignals that push toward `in-process`\u002F`full-stack`: the agent isn't Python; tools hit real\nAPIs\u002Fdatasources; there is already a dedicated eval harness, Docker stack, or\nscenario\u002Fground-truth files in the repo. If a repo already has eval scenarios with expected\noutputs, note them — they are better test cases than anything generated, and later steps\nshould point to or reuse them.\n\nSeparately, note the **agent's language**, because the Agent Observability experiments SDK exists only in\n**Python and Go** (not JS\u002FTS, Java, or .NET yet). This is independent of runnability — a TS\nagent can be trivially runnable yet have no experiments SDK in its language. If the agent is\nPython or Go, the runner is native. If it is any other language, say so plainly: the runner\nmust be Python or Go (calling the agent across a process boundary), or the developer waits for\nexperiments support in their language. Do not imply a native JS\u002FJava\u002F.NET experiments API\nexists.\n\n## Step 2 — Choose evaluators\n\nThe SDK models an evaluator as an id plus a **kind**. There are two kinds:\n\n- `llm_judge` — a model grades the output against a rubric (relevance, helpfulness,\n  groundedness, tone, task completion, format, safety, and similar judgment calls).\n- `deterministic` — code decides pass\u002Ffail (exact\u002Fsubstring match, JSON validity, schema\n  or regex shape, length bounds, \"not empty\", a required field is present).\n\nPick 3–6 (not more). Map each to what you read in Step 1, and give a one-line `why` per pick\nthat cites a file:line. Common mappings:\n\n| If the agent… | Evaluator | Kind |\n| --- | --- | --- |\n| answers open-ended user requests | `relevance`, `helpfulness` | `llm_judge` |\n| retrieves \u002F cites sources \u002F does RAG | `groundedness` | `llm_judge` |\n| must stay on-topic \u002F in-scope | `task_adherence` | `llm_judge` |\n| must emit JSON or a fixed shape | `json_valid` \u002F `schema_match` | `deterministic` |\n| must follow a specific output format | `format_adherence` | `llm_judge` |\n| calls tools | `tool_call_correct` | `llm_judge` (or `deterministic` if the correct call is checkable in code) |\n| handles user data that could be echoed | `pii_leak` | `llm_judge` |\n| produces public-facing text | `toxicity` | `llm_judge` |\n| must always return something | `response_not_empty` | `deterministic` |\n\nEvaluator ids are yours to choose — pick clear, stable ids. Be concrete about what \"defining\"\neach one means, because it is not an Agent Observability control-plane action here: in the offline SDK flow\nan evaluator is **code the developer writes in the runner** (Step 4). An `llm_judge` is a\nfunction that calls a model and returns a score; a `deterministic` one is a plain code check.\n`agento11y.Evaluator(evaluator_id=..., kind=...)` is only the label attached to the score, not the\nlogic. (Forking a predefined template is a separate online-eval path, not needed here.)\n\nThis skill targets the **offline** phase: run these evaluators as offline experiments against\nthe suite from Step 3, before there is traffic. Do not recommend live\u002Fonline evaluation to an\nagent with no traffic.\n\nOnce the agent ships and has traffic, the same evaluation criteria can also be applied online\n— as Agent Observability Rules over ingested conversations, or as SDK guard hooks on the request path — but\nthose are separate Agent Observability surfaces, configured elsewhere, and out of scope for this skill.\nMention this only as a one-line \"next, once you have traffic\" note; do not instruct on it here.\n\n## Step 3 — Write the suite YAML\n\nWrite a suite file in the target repo (suggest `evals\u002F\u003Cagent>-starter.yaml`). It must load\nwith the SDK's `TestSuite.from_yaml(...)`, so match this schema exactly:\n\n- Top level: `suite_id` (required), plus optional `name`, `version`, `description`, `tags`,\n  `changelog`, and `cases` (a list). `version` defaults to `1.0.0`.\n- Each case: `id` (required), plus optional `name`, `description`, `tags`, `category`,\n  `input`, `expected`, `weight`, `metadata`. `input` and `expected` are free-form (a string\n  or a mapping).\n\nDerive cases from the agent's real task. Produce at least 6, weighted toward `edge` and\n`adversarial` (generated cases skew easy). Use `category` values `happy`, `edge`,\n`adversarial`. Keep the header comment verbatim.\n\nEvery case must actually reach the agent — test the agent, not its harness. From Step 1,\nnote where the entrypoint parses\u002Fvalidates input before the model runs (e.g. a JSON parse, a\nCLI arg check). Do not generate cases that fail in that pre-agent layer (malformed JSON, wrong\nCLI flags) as if they tested the agent; if such a boundary is worth covering, label it clearly\nas a harness case in its notes, or leave it out.\n\n```yaml\n# STARTER DRAFT — review before use. Generated from your agent code (\u003Cfile:line refs>).\n# NOT validated. Add your own real cases; the edge\u002Fadversarial cases need your judgment on\n# expected behavior. Loads with agento11y.TestSuite.from_yaml(...).\nsuite_id: \u003Cagent>-starter\nname: \u003CAgent> starter suite\nversion: 1.0.0\ncases:\n  - id: happy-basic-request\n    category: happy\n    tags: [smoke]\n    input:\n      prompt: \"\u003Ca real request this agent is built to answer>\"\n    expected: \"\u003Cwhat a good answer looks like, or a rubric note>\"\n  - id: edge-underspecified\n    category: edge\n    input:\n      prompt: \"\u003Ca vague \u002F multi-part \u002F boundary request>\"\n    expected: \"\u003Chow the agent should handle ambiguity>\"\n  - id: adversarial-prompt-injection\n    category: adversarial\n    input:\n      prompt: \"\u003Can injection \u002F out-of-scope \u002F data-extraction attempt>\"\n    expected: \"\u003Cagent should refuse \u002F stay in scope \u002F not leak>\"\n```\n\n## Step 4 — Write the runner stub (branch on runnability)\n\nThe suite YAML alone does not run anything. The Agent Observability SDK stores and aggregates scores, but\nit does **not** run the agent or compute the evaluators — the developer writes both. Left at\njust the YAML, a developer new to offline eval is still blocked (\"now what?\"). So generate a\n**minimal bootstrap runner** — just enough to get one experiment running.\n\nThis is deliberately the simplest path, not the full run-side API. The `agento11y-experiments`\nskill is the reference for everything beyond bootstrap — binding an already-instrumented\nagent's real generations\u002Fconversations, auditable LLM-judge grading, cross-process verifiers\n(`TrialRef`), pass@k\u002Fpass^k. Don't reproduce those patterns here; generate the minimal runner\nand point to `agento11y-experiments` for depth.\n\nWhat you generate depends on the runnability you assessed in Step 1:\n\n- **easy** → generate the full `evals\u002Frun_experiment.py` below. One hole to fill\n  (`run_agent`).\n- **in-process** → do NOT emit a Python `run_agent` that can't actually call the agent (e.g.\n  a Go agent). Generate the same experiment wiring, but make `run_agent` shell out to a small\n  harness in the agent's language (or write that harness), and say plainly the seam is the\n  injectable LLM client. Point to any existing test that already invokes the agent in-process\n  as the template.\n- **full-stack** → do NOT emit a runner that pretends to call the agent as a function. The\n  agent runs via its existing eval infra (dedicated harness, Docker stack, HTTP + polling).\n  Deliver the recommendations + YAML, and point to that infra and to any existing\n  scenario\u002Fground-truth files as the real test cases. Be explicit that isolated runs aren't\n  the path here.\n\nAlso branch on **language** (the experiments SDK is Python\u002FGo only):\n\n- **Python or Go agent** → native runner (`run_experiment.py`, or the Go `agento11y` package).\n- **TS \u002F Java \u002F .NET agent** → there is no experiments SDK in that language. Deliver\n  recommendations + YAML (they are language-neutral), and be honest about the run path: the\n  runner must be Python or Go calling the agent across a process boundary (e.g. a Python\n  runner that shells out to `node your-agent.js` and reads its output), or the developer waits\n  for experiments support in their language. Offer the subprocess bridge only as a labeled\n  option with its cost (serializing input to the CLI, parsing output), not as a clean default.\n\nFor an **easy Python\u002FGo** agent, write `evals\u002Frun_experiment.py`. It must:\n\n- Load the suite with `TestSuite.from_yaml(...)`.\n- Open an experiment (`agento11y.experiment(...)`) and one `trial` per case.\n- Call the agent through a single clearly-marked function `run_agent(case)` — **the first of the\n  two holes the developer fills**; wire it to the real entrypoint you found in Step 1.\n- Include ONE recommended evaluator sketched end-to-end (prefer an `llm_judge` — a real model\n  call that returns a JSON `{score, passed, explanation}`), so they see the shape and can copy\n  it for the others. Reference the rest by name in a comment; do not stub all of them. **The\n  judge's model call must use the provider the agent already uses** (reuse its OpenAI\u002FAnthropic\n  client or SDK) — do not pull in `litellm` or another provider just for the judge. The one model\n  call inside the judge is **the second hole** — leave it as an explicit `NotImplementedError` so a\n  developer who only fills `run_agent` gets a clear \"fill the judge call\" error, not a `NameError`\n  on an undefined helper.\n- Record I\u002FO (`trial.record_io(...)`) and emit `trial.final_score(...)` with the evaluator.\n\nKeep the header verbatim, and be honest in it about what still needs doing:\n\n```python\n#!\u002Fusr\u002Fbin\u002Fenv python3\n\"\"\"STARTER RUNNER — generated by agento11y-test-starter, review before use.\n\nRuns \u003Cagent> over evals\u002F\u003Cagent>-starter.yaml as an Agent Observability experiment and publishes scores.\n\nYou still need to: (1) fill run_agent(case) to call YOUR agent (first hole); (2) fill the model\ncall inside judge_\u003Cevaluator> using the agent's own provider client, then tune it (second hole);\n(3) set real credentials — `AGENTO11Y_ENDPOINT` and `AGENTO11Y_AUTH_TOKEN` (your Grafana Cloud\ningestion key) and the agent's provider key. The SDK stores scores; it does not run the agent or\nthe judge.\n\nSet AGENTO11Y_INGEST_ACTOR to a stable value: the run and its trials must share one actor, or\ntrial creation fails with \"401: experiment is owned by another actor\".\n\n    AGENTO11Y_ENDPOINT=... AGENTO11Y_AUTH_TOKEN=... AGENTO11Y_INGEST_ACTOR=ingest:sdk\u002Fpython \\\n        python evals\u002Frun_experiment.py\n\"\"\"\nimport json, os, time\nfrom pathlib import Path\nfrom dotenv import load_dotenv\nfrom agento11y import experiments as agento11y\n\nload_dotenv()\nSUITE = Path(__file__).parent \u002F \"\u003Cagent>-starter.yaml\"\n\n\ndef run_agent(case: agento11y.TestCase) -> str:\n    \"\"\"FIRST HOLE YOU FILL — call your agent for this case, return its output text.\"\"\"\n    raise NotImplementedError(\"wire this to your agent entrypoint (see Step 1 refs)\")\n\n\ndef judge_\u003Cevaluator>(case_input, output) -> tuple[float, bool, str]:\n    \"\"\"Sketched llm_judge — a model call returning (score 0-1, passed, explanation). Tune it.\n\n    Uses the SAME provider client the agent uses (imported below from the agent module) — do NOT\n    swap in litellm or another provider. Adapt the call to your provider's API.\n\n    IMPORTANT for grounding\u002Frelevance judges: the judge must SEE what the agent saw. If the agent\n    retrieves context (RAG), re-run its retriever here and put the passages in the prompt —\n    otherwise the judge marks correct, cited answers as \"unverifiable\" and scores them low. Import\n    the agent's own retriever (e.g. `from agent import retrieve`) and include its output.\n    \"\"\"\n    prompt = (\n        'Grade \u003Cwhat this evaluator checks>. Reply with ONLY a JSON object, no prose, no markdown '\n        'fences, explanation LAST and short: {\"score\": \u003C0-1 float>, \"passed\": \u003Cbool>, \"explanation\": \"\u003Cone sentence>\"}.\\n\\n'\n        # For a grounding judge, prepend the retrieved context here:\n        # f\"CONTEXT:\\n{retrieved_passages}\\n\\n\"\n        f\"Input:\\n{case_input}\\n\\nOutput:\\n{output}\"\n    )\n    model = os.getenv(\"GRADER_MODEL\") or os.getenv(\"MODEL_NAME\")  # a LIVE model id; no default (dead ids 404)\n    # SECOND HOLE YOU FILL — call the model using the SAME client the agent uses (do NOT add a new\n    # provider). Import it from the agent module and make ONE call that returns the reply text into\n    # `text`. Give it enough max_tokens (~600) that the JSON verdict is never truncated mid-object.\n    #   Anthropic: from agent import client\n    #     text = client.messages.create(model=model, max_tokens=600,\n    #                                    messages=[{\"role\": \"user\", \"content\": prompt}]).content[0].text\n    #   OpenAI:    from agent import client\n    #     text = client.chat.completions.create(model=model, max_tokens=600,\n    #                                            messages=[{\"role\": \"user\", \"content\": prompt}]).choices[0].message.content\n    raise NotImplementedError(\"call the agent's provider client here and assign the reply to `text`\")\n    d = _parse_judge_json(text)\n    score = max(0.0, min(1.0, float(d.get(\"score\", 0.0))))\n    return score, bool(d.get(\"passed\", score >= 0.6)), str(d.get(\"explanation\", \"\"))\n\n\ndef _parse_judge_json(text: str) -> dict:\n    \"\"\"Robustly pull the JSON verdict out of a model reply (handles prose \u002F ```json fences).\"\"\"\n    if not text:\n        return {}\n    t = text.strip()\n    if \"```\" in t:  # strip a ```json ... ``` fence if present\n        t = t.split(\"```\")[1].removeprefix(\"json\").strip() if t.count(\"```\") >= 2 else t\n    try:\n        return json.loads(t)\n    except json.JSONDecodeError:\n        pass\n    # Fallback: scan for the first balanced {...} object rather than first-brace\u002Flast-brace.\n    depth = 0; start = -1\n    for i, ch in enumerate(t):\n        if ch == \"{\":\n            if depth == 0:\n                start = i\n            depth += 1\n        elif ch == \"}\":\n            depth -= 1\n            if depth == 0 and start >= 0:\n                try:\n                    return json.loads(t[start:i + 1])\n                except json.JSONDecodeError:\n                    start = -1\n    return {}  # unparseable → caller treats as score 0; investigate the raw reply\n\n\ndef main() -> None:\n    suite = agento11y.TestSuite.from_yaml(str(SUITE))\n    verifier = agento11y.Evaluator(evaluator_id=\"\u003Cevaluator>\", version=\"draft-0\", kind=\"llm_judge\")\n    candidate = {\n        \"agent_name\": \"\u003Cagent>\",\n        # Always send a declared agent_version. Without it Agent Observability auto-derives a version from\n        # the system-prompt hash, so you can't reliably attribute scores to a version or\n        # compare versions. Replace \"v1\" with your real version (git tag, prompt version,\n        # semver...) — the \"v1\" fallback is a placeholder, not a version worth comparing.\n        \"agent_version\": os.getenv(\"AGENT_VERSION\", \"v1\"),\n        \"git_sha\": os.getenv(\"GIT_SHA\", \"\"),\n        \"model_name\": os.getenv(\"MODEL_NAME\", \"\"),\n    }\n    with agento11y.experiment(name=\"\u003Cagent> starter\", experiment_id=f\"\u003Cagent>-starter-{int(time.time())}\",\n                          suite=suite, candidate=candidate, tags=[\"starter\"],\n                          actor=os.getenv(\"AGENTO11Y_INGEST_ACTOR\", \"ingest:sdk\u002Fpython\")) as exp:\n        for case in suite.test_cases:\n            with exp.trial(case) as trial:\n                out = run_agent(case)\n                trial.record_io(input=json.dumps(case.input), output=out,\n                                model_provider=\"\u003Cprovider>\", model_name=os.getenv(\"MODEL_NAME\", \"\"))\n                score, passed, why = judge_\u003Cevaluator>(case.input, out)\n                trial.final_score(score, passed=passed, explanation=why, evaluator=verifier)\n                print(f\"  {case.test_case_id}: score={score:.2f} passed={passed}\")\n    print(f\"\\nExperiment: {exp.url}\")\n\n\nif __name__ == \"__main__\":\n    main()\n```\n\nUse a **fresh `experiment_id`** per run (a timestamp works) — reusing an id created by a\ndifferent auth actor fails with `401: experiment is owned by another actor`.\n\n## Step 5 — Summarize and hand off\n\nOutput, in this order:\n\n1. The picked evaluators, each with its kind and its `why` (with file:line).\n   Add a one-line \"once you have traffic, these criteria can also run online (Agent Observability Rules or\n   guard hooks) — separate surfaces, not set up here.\"\n2. The paths to the two written files (`evals\u002F\u003Cagent>-starter.yaml` and\n   `evals\u002Frun_experiment.py`), and a one-line reminder to review the edge\u002Fadversarial cases\n   and add real ones.\n3. The three things they still do to run it: fill `run_agent(case)` (first hole); fill the judge's\n   model call using the agent's own provider client, then tune it and add the other recommended\n   evaluators the same way (second hole); and set credentials (`AGENTO11Y_ENDPOINT` +\n   `AGENTO11Y_AUTH_TOKEN` + the provider key).\n   State the boundary explicitly: this skill only\n   bootstraps the first run; for anything past that — binding an already-instrumented agent's\n   real generations, auditable LLM-judge grading, cross-process verifiers, repeated-sampling\n   metrics — the `agento11y-experiments` skill is the reference.\n4. One line confirming nothing was created in Agent Observability — recommendations and draft files only.\n\n## Step 6 — Offer to run it (optional, only with permission)\n\nOnly offer this for an **easy** agent (clean function seam) **in Python or Go** (the languages\nwith an experiments SDK). For `in-process`\u002F`full-stack` agents, or agents in a language with no\nexperiments SDK (TS\u002FJava\u002F.NET), don't offer to run — point to the existing harness\u002Finfra (or\nthe subprocess-bridge option) and stop; a real run there is out of scope for this skill.\n\nAfter the summary, offer to run the starter experiment for them — do not run automatically.\nAsk: \"Want me to try running this now?\" Only proceed if they say yes.\n\nIf they accept:\n\n1. Help fill `run_agent(case)` — wire it to the real entrypoint from Step 1, so the runner\n   actually calls their agent.\n2. Preflight the environment and stop with a clear ask if anything is missing. **When you ask, tell\n   the developer exactly where each value is** — for a Cloud stack they all live on the plugin\n   **Connection page**, `https:\u002F\u002F\u003Cyour-stack>.grafana.net\u002Fplugins\u002Fgrafana-agento11y-app`:\n   - `AGENTO11Y_ENDPOINT` = the **API URL** on that page. If unset, ask — never invent one.\n   - `AGENTO11Y_AUTH_TENANT_ID` = the **Instance ID** on that page.\n   - `AGENTO11Y_AUTH_TOKEN` — always required (the SDK raises before any request if it is empty).\n     The developer creates it via **\"Create a token in Cloud Access Policies\"** on the Connection page. Tell them\n     the exact scopes: **`sigil:write`, `metrics:write`, `traces:write`, `logs:write`**. Heads-up on\n     the UI: `sigil` is **not** in the default resource list — they must add it via **\"Add scope\"**\n     (then tick Write); `metrics`\u002F`traces`\u002F`logs` are already listed (tick Write). The scope is still\n     `sigil:*` (the Cloud resource keeps the old name). **Never mint or fabricate a token** — the\n     developer creates and supplies it; you only read it from the environment or ask.\n   - The **provider API key from Step 1** (e.g. `OPENAI_API_KEY` or `ANTHROPIC_API_KEY`) — name the\n     exact env var, since both the agent and the judge need it. Ask for it; never mint it.\n   - `MODEL_NAME` is a live model (dead model ids fail with a 404 not_found).\n   - A stable `AGENTO11Y_INGEST_ACTOR` so run and trials share one actor (else `401: owned by\n     another actor`).\n   - A declared `AGENT_VERSION` (in the candidate). Without it Agent Observability auto-derives a version\n     from the system-prompt hash, and the developer can't attribute scores to a version or\n     compare versions — which is the whole point of the agent's Quality view. Confirm a real\n     value (git tag \u002F prompt version \u002F semver), don't leave it defaulted.\n3. Run against the endpoint the developer configured — never a target they didn't specify.\n   Start with 1–2 cases as a smoke run before the full suite.\n4. Show the per-case scores and the `exp.url`, and note this published one experiment's\n   scores (no tenant evaluators\u002Frules\u002Fguards were created).\n5. **Don't oversell a clean sweep.** If every case passes on the first run, say so honestly: that\n   usually means the suite isn't hard enough to discriminate yet, not that the agent is flawless.\n   Nudge toward (a) wiring the recommended **deterministic** evaluators (they catch contract\n   violations an `llm_judge` forgives — e.g. an exact-string check vs a paraphrase), and (b) adding\n   a case that actually stresses the agent's likely real failure mode.\n\nIf they decline, stop after the summary.\n",{"data":35,"body":36},{"name":4,"description":6},{"type":37,"children":38},"root",[39,47,75,100,105,134,139,163,176,199,206,211,281,294,300,431,437,442,606,611,623,663,684,703,709,721,745,758,1039,1073,1085,1090,1096,1117,1271,1319,1324,1769,1775,1794,1821,1826,1885,1897,1942,1961,2101,2106,3174,3199,3205,3210,3281,3287,3318,3323,3328,3622,3627],{"type":40,"tag":41,"props":42,"children":43},"element","h1",{"id":4},[44],{"type":45,"value":46},"text","agento11y test starter",{"type":40,"tag":48,"props":49,"children":50},"p",{},[51,53,59,61,66,68,73],{"type":45,"value":52},"Help a developer test an AI agent ",{"type":40,"tag":54,"props":55,"children":56},"strong",{},[57],{"type":45,"value":58},"before it ships",{"type":45,"value":60}," — while there is no real traffic yet. The\nhard part isn't running the test — it's having ",{"type":40,"tag":54,"props":62,"children":63},{},[64],{"type":45,"value":65},"cases to test against",{"type":45,"value":67}," and knowing ",{"type":40,"tag":54,"props":69,"children":70},{},[71],{"type":45,"value":72},"how to\nscore them",{"type":45,"value":74},", grounded in the agent's actual code.",{"type":40,"tag":76,"props":77,"children":78},"blockquote",{},[79],{"type":40,"tag":48,"props":80,"children":81},{},[82,84,89,91,98],{"type":45,"value":83},"Scope: this is the ",{"type":40,"tag":54,"props":85,"children":86},{},[87],{"type":45,"value":88},"pre-production, offline",{"type":45,"value":90}," skill — it writes test cases and a local runner,\nand never touches the tenant. Once the agent is deployed and has real traffic, setting up online\neval rules + guards on that traffic is a different skill, ",{"type":40,"tag":92,"props":93,"children":95},"code",{"className":94},[],[96],{"type":45,"value":97},"agento11y-prod-setup",{"type":45,"value":99},".",{"type":40,"tag":48,"props":101,"children":102},{},[103],{"type":45,"value":104},"Always produce:",{"type":40,"tag":106,"props":107,"children":108},"ol",{},[109,122],{"type":40,"tag":110,"props":111,"children":112},"li",{},[113,115,120],{"type":45,"value":114},"A ranked, justified ",{"type":40,"tag":54,"props":116,"children":117},{},[118],{"type":45,"value":119},"evaluator recommendation",{"type":45,"value":121}," for this agent.",{"type":40,"tag":110,"props":123,"children":124},{},[125,127,132],{"type":45,"value":126},"A starter ",{"type":40,"tag":54,"props":128,"children":129},{},[130],{"type":45,"value":131},"suite YAML",{"type":45,"value":133}," the developer reviews and extends.",{"type":40,"tag":48,"props":135,"children":136},{},[137],{"type":45,"value":138},"Then, depending on how runnable the agent is (Step 1):",{"type":40,"tag":106,"props":140,"children":142},{"start":141},3,[143],{"type":40,"tag":110,"props":144,"children":145},{},[146,148,153,155,161],{"type":45,"value":147},"For an easily-invoked agent, a ",{"type":40,"tag":54,"props":149,"children":150},{},[151],{"type":45,"value":152},"runner stub",{"type":45,"value":154}," (",{"type":40,"tag":92,"props":156,"children":158},{"className":157},[],[159],{"type":45,"value":160},"run_experiment.py",{"type":45,"value":162},") that wires the suite\nto the SDK with two holes to fill — and optionally run it (Step 6), only with permission.\nFor an agent that needs a harness or full runtime, point to the existing eval infra instead\nof a runner that can't actually call it.",{"type":40,"tag":48,"props":164,"children":165},{},[166,168,174],{"type":45,"value":167},"This skill is language-agnostic — the reading, recommending, and YAML it produces do not\ndepend on the agent's language. What differs is how runnable the agent is: recommendations +\nYAML always apply, but the runner (Step 4) and the optional run (Step 6) adapt to whether the\nagent has a clean function seam or needs a harness \u002F full stack. For deeper run-side patterns\n(binding existing generations, cross-process verifiers) point to the per-language run skill\n(Python: ",{"type":40,"tag":92,"props":169,"children":171},{"className":170},[],[172],{"type":45,"value":173},"agento11y-experiments",{"type":45,"value":175},").",{"type":40,"tag":76,"props":177,"children":178},{},[179],{"type":40,"tag":48,"props":180,"children":181},{},[182,184,189,191,197],{"type":45,"value":183},"Note: ",{"type":40,"tag":92,"props":185,"children":187},{"className":186},[],[188],{"type":45,"value":173},{"type":45,"value":190}," currently ships in the grafana\u002Fagento11y repo\n(",{"type":40,"tag":92,"props":192,"children":194},{"className":193},[],[195],{"type":45,"value":196},"python\u002Fskills\u002Fagento11y-experiments\u002F",{"type":45,"value":198},"), not in this gcx bundle yet — install it from there for\nnow. Consolidating it into the gcx bundle is pending.",{"type":40,"tag":200,"props":201,"children":203},"h2",{"id":202},"prerequisites",[204],{"type":45,"value":205},"Prerequisites",{"type":40,"tag":48,"props":207,"children":208},{},[209],{"type":45,"value":210},"The generated runner imports the Agent Observability SDK. Install it in the agent's environment\nbefore running (Step 6):",{"type":40,"tag":212,"props":213,"children":214},"ul",{},[215,265],{"type":40,"tag":110,"props":216,"children":217},{},[218,223,225,231,233,239,241,247,249,255,257,263],{"type":40,"tag":54,"props":219,"children":220},{},[221],{"type":45,"value":222},"Python:",{"type":45,"value":224}," ",{"type":40,"tag":92,"props":226,"children":228},{"className":227},[],[229],{"type":45,"value":230},"pip install agento11y python-dotenv",{"type":45,"value":232}," (the experiments API lives in\n",{"type":40,"tag":92,"props":234,"children":236},{"className":235},[],[237],{"type":45,"value":238},"agento11y.experiments",{"type":45,"value":240},"; the runner uses ",{"type":40,"tag":92,"props":242,"children":244},{"className":243},[],[245],{"type":45,"value":246},"python-dotenv",{"type":45,"value":248}," to load the agent's ",{"type":40,"tag":92,"props":250,"children":252},{"className":251},[],[253],{"type":45,"value":254},".env",{"type":45,"value":256},", and it is\nnot a dependency of ",{"type":40,"tag":92,"props":258,"children":260},{"className":259},[],[261],{"type":45,"value":262},"agento11y",{"type":45,"value":264},")",{"type":40,"tag":110,"props":266,"children":267},{},[268,273,275],{"type":40,"tag":54,"props":269,"children":270},{},[271],{"type":45,"value":272},"Go:",{"type":45,"value":274}," add ",{"type":40,"tag":92,"props":276,"children":278},{"className":277},[],[279],{"type":45,"value":280},"github.com\u002Fgrafana\u002Fagento11y\u002Fgo",{"type":40,"tag":48,"props":282,"children":283},{},[284,286,292],{"type":45,"value":285},"Only needed to ",{"type":40,"tag":287,"props":288,"children":289},"em",{},[290],{"type":45,"value":291},"run",{"type":45,"value":293}," the suite (Step 6). Reading the agent, recommending evaluators, and writing\nthe suite YAML (Steps 1–5) need nothing installed.",{"type":40,"tag":200,"props":295,"children":297},{"id":296},"rules",[298],{"type":45,"value":299},"Rules",{"type":40,"tag":212,"props":301,"children":302},{},[303,308,313,334,353,358,384,426],{"type":40,"tag":110,"props":304,"children":305},{},[306],{"type":45,"value":307},"Do not create, enable, or modify evaluators, rules, or guards in any Agent Observability tenant. No\ncontrol-plane writes. (Running an offline experiment only publishes that run's scores — it\ndoes not create tenant-level evaluators\u002Frules\u002Fguards — but only do it via Step 6.)",{"type":40,"tag":110,"props":309,"children":310},{},[311],{"type":45,"value":312},"Do not rewrite the agent's prompt, optimize, or redeploy.",{"type":40,"tag":110,"props":314,"children":315},{},[316,318,324,326,332],{"type":45,"value":317},"Never run the experiment without asking first (Step 6). Never run against a target the\ndeveloper did not configure — use their ",{"type":40,"tag":92,"props":319,"children":321},{"className":320},[],[322],{"type":45,"value":323},"AGENTO11Y_ENDPOINT",{"type":45,"value":325}," and ",{"type":40,"tag":92,"props":327,"children":329},{"className":328},[],[330],{"type":45,"value":331},"AGENTO11Y_AUTH_TOKEN",{"type":45,"value":333},"; if the\nendpoint isn't set, ask for it, do not invent one.",{"type":40,"tag":110,"props":335,"children":336},{},[337,339,344,346,351],{"type":45,"value":338},"Never mint, generate, or store credentials. The developer owns the Grafana Cloud ingestion\ntoken; read it from the environment (a gitignored ",{"type":40,"tag":92,"props":340,"children":342},{"className":341},[],[343],{"type":45,"value":254},{"type":45,"value":345}," or an exported env var they supply\nthemselves) — ",{"type":40,"tag":54,"props":347,"children":348},{},[349],{"type":45,"value":350},"do not ask them to paste a secret token into the chat",{"type":45,"value":352}," (it is captured in the\ntranscript), and do not create one.",{"type":40,"tag":110,"props":354,"children":355},{},[356],{"type":45,"value":357},"Never present the generated cases as validated. They are a draft to review and extend.",{"type":40,"tag":110,"props":359,"children":360},{},[361,374,376,382],{"type":40,"tag":54,"props":362,"children":363},{},[364,366,372],{"type":45,"value":365},"The ",{"type":40,"tag":92,"props":367,"children":369},{"className":368},[],[370],{"type":45,"value":371},"llm_judge",{"type":45,"value":373}," uses the LLM provider the agent already uses — don't add a new one.",{"type":45,"value":375}," If the\nagent calls OpenAI, the judge calls OpenAI; if Anthropic, Anthropic. Do NOT default the judge to\n",{"type":40,"tag":92,"props":377,"children":379},{"className":378},[],[380],{"type":45,"value":381},"litellm",{"type":45,"value":383}," (or any other provider SDK) when the app doesn't already depend on it — that adds a\ndependency and a second provider just for scoring. Reuse the agent's existing client\u002FSDK.",{"type":40,"tag":110,"props":385,"children":386},{},[387,392,394,399,400,405,410,412,417,419,424],{"type":40,"tag":54,"props":388,"children":389},{},[390],{"type":45,"value":391},"The target is Grafana Cloud.",{"type":45,"value":393}," Publishing scores needs ",{"type":40,"tag":92,"props":395,"children":397},{"className":396},[],[398],{"type":45,"value":323},{"type":45,"value":224},{"type":40,"tag":54,"props":401,"children":402},{},[403],{"type":45,"value":404},"and",{"type":40,"tag":92,"props":406,"children":408},{"className":407},[],[409],{"type":45,"value":331},{"type":45,"value":411}," (the ingestion key from the Connection page) — the SDK raises before\nmaking any request if the token is empty, so both are always required. Read the endpoint from an\nexisting ",{"type":40,"tag":92,"props":413,"children":415},{"className":414},[],[416],{"type":45,"value":323},{"type":45,"value":418}," \u002F ",{"type":40,"tag":92,"props":420,"children":422},{"className":421},[],[423],{"type":45,"value":254},{"type":45,"value":425}," \u002F sibling app; never invent one or mint a token.",{"type":40,"tag":110,"props":427,"children":428},{},[429],{"type":45,"value":430},"If a required input is missing (entrypoint, prompt, tools), ask the developer — don't guess.",{"type":40,"tag":200,"props":432,"children":434},{"id":433},"step-1-read-the-agent",[435],{"type":45,"value":436},"Step 1 — Read the agent",{"type":40,"tag":48,"props":438,"children":439},{},[440],{"type":45,"value":441},"Find and read these in the target repo, and record the file path and line range of each:",{"type":40,"tag":106,"props":443,"children":444},{},[445,450,455,460,465],{"type":40,"tag":110,"props":446,"children":447},{},[448],{"type":45,"value":449},"The agent entrypoint (where the model is invoked).",{"type":40,"tag":110,"props":451,"children":452},{},[453],{"type":45,"value":454},"The system prompt \u002F instructions.",{"type":40,"tag":110,"props":456,"children":457},{},[458],{"type":45,"value":459},"The tool \u002F function definitions the agent can call.",{"type":40,"tag":110,"props":461,"children":462},{},[463],{"type":45,"value":464},"One or two real user requests and what a correct answer looks like.",{"type":40,"tag":110,"props":466,"children":467},{},[468,473,475],{"type":40,"tag":54,"props":469,"children":470},{},[471],{"type":45,"value":472},"The LLM provider and the model",{"type":45,"value":474}," — two distinct things; read them off how the client is\nconstructed and cite the line:\n",{"type":40,"tag":212,"props":476,"children":477},{},[478,504,545],{"type":40,"tag":110,"props":479,"children":480},{},[481,486,488,494,496,502],{"type":40,"tag":54,"props":482,"children":483},{},[484],{"type":45,"value":485},"Provider",{"type":45,"value":487}," (who serves the LLM) → which SDK\u002Fclient the judge reuses: e.g.\n",{"type":40,"tag":92,"props":489,"children":491},{"className":490},[],[492],{"type":45,"value":493},"from openai import OpenAI",{"type":45,"value":495}," → OpenAI; ",{"type":40,"tag":92,"props":497,"children":499},{"className":498},[],[500],{"type":45,"value":501},"from anthropic import Anthropic",{"type":45,"value":503}," → Anthropic.",{"type":40,"tag":110,"props":505,"children":506},{},[507,512,514,520,522,528,530,536,538,543],{"type":40,"tag":54,"props":508,"children":509},{},[510],{"type":45,"value":511},"Model",{"type":45,"value":513}," (the specific id, e.g. ",{"type":40,"tag":92,"props":515,"children":517},{"className":516},[],[518],{"type":45,"value":519},"gpt-4o-mini",{"type":45,"value":521},", ",{"type":40,"tag":92,"props":523,"children":525},{"className":524},[],[526],{"type":45,"value":527},"claude-sonnet-5",{"type":45,"value":529},") → the ",{"type":40,"tag":92,"props":531,"children":533},{"className":532},[],[534],{"type":45,"value":535},"MODEL_NAME",{"type":45,"value":537}," the runner uses.\nState both explicitly, and have the judge use the ",{"type":40,"tag":54,"props":539,"children":540},{},[541],{"type":45,"value":542},"same provider",{"type":45,"value":544}," as the agent (never a\ndifferent one). If you can't tell from the code, ask; don't default to a provider the app doesn't use.",{"type":40,"tag":110,"props":546,"children":547},{},[548,553,555,560,562,567,568,574,576,582,584,589,591,596,598,604],{"type":40,"tag":54,"props":549,"children":550},{},[551],{"type":45,"value":552},"Do NOT assume where the provider API key lives.",{"type":45,"value":554}," The app already loads it ",{"type":40,"tag":287,"props":556,"children":557},{},[558],{"type":45,"value":559},"somehow",{"type":45,"value":561}," — an env\nvar, a ",{"type":40,"tag":92,"props":563,"children":565},{"className":564},[],[566],{"type":45,"value":254},{"type":45,"value":154},{"type":40,"tag":92,"props":569,"children":571},{"className":570},[],[572],{"type":45,"value":573},"load_dotenv()",{"type":45,"value":575},"), a secret manager, an explicit ",{"type":40,"tag":92,"props":577,"children":579},{"className":578},[],[580],{"type":45,"value":581},"client(api_key=...)",{"type":45,"value":583},". The\nrunner should reuse the ",{"type":40,"tag":54,"props":585,"children":586},{},[587],{"type":45,"value":588},"same mechanism",{"type":45,"value":590}," (it already calls ",{"type":40,"tag":92,"props":592,"children":594},{"className":593},[],[595],{"type":45,"value":573},{"type":45,"value":597},", and provider SDKs\nread their standard env var themselves). Don't hardcode a key, and don't tell the developer to\n\"set ",{"type":40,"tag":92,"props":599,"children":601},{"className":600},[],[602],{"type":45,"value":603},"OPENAI_API_KEY",{"type":45,"value":605},"\" as if that's the only way — if the run can't find the key, ask them how\ntheir app loads it and mirror that.",{"type":40,"tag":48,"props":607,"children":608},{},[609],{"type":45,"value":610},"Every recommendation must cite one of these locations.",{"type":40,"tag":48,"props":612,"children":613},{},[614,616,621],{"type":45,"value":615},"Also assess ",{"type":40,"tag":54,"props":617,"children":618},{},[619],{"type":45,"value":620},"runnability",{"type":45,"value":622}," — how hard it is to invoke this agent for one input and get its\noutput — because it decides what Step 4 produces. Classify it as one of:",{"type":40,"tag":212,"props":624,"children":625},{},[626,636,653],{"type":40,"tag":110,"props":627,"children":628},{},[629,634],{"type":40,"tag":54,"props":630,"children":631},{},[632],{"type":45,"value":633},"easy — clean function seam.",{"type":45,"value":635}," A single call takes an input and returns text, with no live\nservices (a small Python\u002FTS agent, an LLM call behind one function). The generated runner\nworks as-is.",{"type":40,"tag":110,"props":637,"children":638},{},[639,644,646,652],{"type":40,"tag":54,"props":640,"children":641},{},[642],{"type":45,"value":643},"in-process — needs a harness.",{"type":45,"value":645}," No 3-line call, but there is an injectable seam (e.g. a\nGo engine whose LLM client is an interface you can fake, callable from a test binary).\nRunnable in-process for a smoke\u002Fbehavioral eval, but not a 3-line Python ",{"type":40,"tag":92,"props":647,"children":649},{"className":648},[],[650],{"type":45,"value":651},"run_agent",{"type":45,"value":99},{"type":40,"tag":110,"props":654,"children":655},{},[656,661],{"type":40,"tag":54,"props":657,"children":658},{},[659],{"type":45,"value":660},"full-stack — needs the whole runtime.",{"type":45,"value":662}," The agent needs live backends\u002Fauth\u002Fqueues and is\nexercised over HTTP against a running stack (tools that hit real datasources, a long\nmulti-step loop). Existing eval infra likely runs it via a dedicated harness and polling,\nnot a function.",{"type":40,"tag":48,"props":664,"children":665},{},[666,668,674,676,682],{"type":45,"value":667},"Signals that push toward ",{"type":40,"tag":92,"props":669,"children":671},{"className":670},[],[672],{"type":45,"value":673},"in-process",{"type":45,"value":675},"\u002F",{"type":40,"tag":92,"props":677,"children":679},{"className":678},[],[680],{"type":45,"value":681},"full-stack",{"type":45,"value":683},": the agent isn't Python; tools hit real\nAPIs\u002Fdatasources; there is already a dedicated eval harness, Docker stack, or\nscenario\u002Fground-truth files in the repo. If a repo already has eval scenarios with expected\noutputs, note them — they are better test cases than anything generated, and later steps\nshould point to or reuse them.",{"type":40,"tag":48,"props":685,"children":686},{},[687,689,694,696,701],{"type":45,"value":688},"Separately, note the ",{"type":40,"tag":54,"props":690,"children":691},{},[692],{"type":45,"value":693},"agent's language",{"type":45,"value":695},", because the Agent Observability experiments SDK exists only in\n",{"type":40,"tag":54,"props":697,"children":698},{},[699],{"type":45,"value":700},"Python and Go",{"type":45,"value":702}," (not JS\u002FTS, Java, or .NET yet). This is independent of runnability — a TS\nagent can be trivially runnable yet have no experiments SDK in its language. If the agent is\nPython or Go, the runner is native. If it is any other language, say so plainly: the runner\nmust be Python or Go (calling the agent across a process boundary), or the developer waits for\nexperiments support in their language. Do not imply a native JS\u002FJava\u002F.NET experiments API\nexists.",{"type":40,"tag":200,"props":704,"children":706},{"id":705},"step-2-choose-evaluators",[707],{"type":45,"value":708},"Step 2 — Choose evaluators",{"type":40,"tag":48,"props":710,"children":711},{},[712,714,719],{"type":45,"value":713},"The SDK models an evaluator as an id plus a ",{"type":40,"tag":54,"props":715,"children":716},{},[717],{"type":45,"value":718},"kind",{"type":45,"value":720},". There are two kinds:",{"type":40,"tag":212,"props":722,"children":723},{},[724,734],{"type":40,"tag":110,"props":725,"children":726},{},[727,732],{"type":40,"tag":92,"props":728,"children":730},{"className":729},[],[731],{"type":45,"value":371},{"type":45,"value":733}," — a model grades the output against a rubric (relevance, helpfulness,\ngroundedness, tone, task completion, format, safety, and similar judgment calls).",{"type":40,"tag":110,"props":735,"children":736},{},[737,743],{"type":40,"tag":92,"props":738,"children":740},{"className":739},[],[741],{"type":45,"value":742},"deterministic",{"type":45,"value":744}," — code decides pass\u002Ffail (exact\u002Fsubstring match, JSON validity, schema\nor regex shape, length bounds, \"not empty\", a required field is present).",{"type":40,"tag":48,"props":746,"children":747},{},[748,750,756],{"type":45,"value":749},"Pick 3–6 (not more). Map each to what you read in Step 1, and give a one-line ",{"type":40,"tag":92,"props":751,"children":753},{"className":752},[],[754],{"type":45,"value":755},"why",{"type":45,"value":757}," per pick\nthat cites a file:line. Common mappings:",{"type":40,"tag":759,"props":760,"children":761},"table",{},[762,786],{"type":40,"tag":763,"props":764,"children":765},"thead",{},[766],{"type":40,"tag":767,"props":768,"children":769},"tr",{},[770,776,781],{"type":40,"tag":771,"props":772,"children":773},"th",{},[774],{"type":45,"value":775},"If the agent…",{"type":40,"tag":771,"props":777,"children":778},{},[779],{"type":45,"value":780},"Evaluator",{"type":40,"tag":771,"props":782,"children":783},{},[784],{"type":45,"value":785},"Kind",{"type":40,"tag":787,"props":788,"children":789},"tbody",{},[790,823,848,873,905,930,964,989,1014],{"type":40,"tag":767,"props":791,"children":792},{},[793,799,815],{"type":40,"tag":794,"props":795,"children":796},"td",{},[797],{"type":45,"value":798},"answers open-ended user requests",{"type":40,"tag":794,"props":800,"children":801},{},[802,808,809],{"type":40,"tag":92,"props":803,"children":805},{"className":804},[],[806],{"type":45,"value":807},"relevance",{"type":45,"value":521},{"type":40,"tag":92,"props":810,"children":812},{"className":811},[],[813],{"type":45,"value":814},"helpfulness",{"type":40,"tag":794,"props":816,"children":817},{},[818],{"type":40,"tag":92,"props":819,"children":821},{"className":820},[],[822],{"type":45,"value":371},{"type":40,"tag":767,"props":824,"children":825},{},[826,831,840],{"type":40,"tag":794,"props":827,"children":828},{},[829],{"type":45,"value":830},"retrieves \u002F cites sources \u002F does RAG",{"type":40,"tag":794,"props":832,"children":833},{},[834],{"type":40,"tag":92,"props":835,"children":837},{"className":836},[],[838],{"type":45,"value":839},"groundedness",{"type":40,"tag":794,"props":841,"children":842},{},[843],{"type":40,"tag":92,"props":844,"children":846},{"className":845},[],[847],{"type":45,"value":371},{"type":40,"tag":767,"props":849,"children":850},{},[851,856,865],{"type":40,"tag":794,"props":852,"children":853},{},[854],{"type":45,"value":855},"must stay on-topic \u002F in-scope",{"type":40,"tag":794,"props":857,"children":858},{},[859],{"type":40,"tag":92,"props":860,"children":862},{"className":861},[],[863],{"type":45,"value":864},"task_adherence",{"type":40,"tag":794,"props":866,"children":867},{},[868],{"type":40,"tag":92,"props":869,"children":871},{"className":870},[],[872],{"type":45,"value":371},{"type":40,"tag":767,"props":874,"children":875},{},[876,881,897],{"type":40,"tag":794,"props":877,"children":878},{},[879],{"type":45,"value":880},"must emit JSON or a fixed shape",{"type":40,"tag":794,"props":882,"children":883},{},[884,890,891],{"type":40,"tag":92,"props":885,"children":887},{"className":886},[],[888],{"type":45,"value":889},"json_valid",{"type":45,"value":418},{"type":40,"tag":92,"props":892,"children":894},{"className":893},[],[895],{"type":45,"value":896},"schema_match",{"type":40,"tag":794,"props":898,"children":899},{},[900],{"type":40,"tag":92,"props":901,"children":903},{"className":902},[],[904],{"type":45,"value":742},{"type":40,"tag":767,"props":906,"children":907},{},[908,913,922],{"type":40,"tag":794,"props":909,"children":910},{},[911],{"type":45,"value":912},"must follow a specific output format",{"type":40,"tag":794,"props":914,"children":915},{},[916],{"type":40,"tag":92,"props":917,"children":919},{"className":918},[],[920],{"type":45,"value":921},"format_adherence",{"type":40,"tag":794,"props":923,"children":924},{},[925],{"type":40,"tag":92,"props":926,"children":928},{"className":927},[],[929],{"type":45,"value":371},{"type":40,"tag":767,"props":931,"children":932},{},[933,938,947],{"type":40,"tag":794,"props":934,"children":935},{},[936],{"type":45,"value":937},"calls tools",{"type":40,"tag":794,"props":939,"children":940},{},[941],{"type":40,"tag":92,"props":942,"children":944},{"className":943},[],[945],{"type":45,"value":946},"tool_call_correct",{"type":40,"tag":794,"props":948,"children":949},{},[950,955,957,962],{"type":40,"tag":92,"props":951,"children":953},{"className":952},[],[954],{"type":45,"value":371},{"type":45,"value":956}," (or ",{"type":40,"tag":92,"props":958,"children":960},{"className":959},[],[961],{"type":45,"value":742},{"type":45,"value":963}," if the correct call is checkable in code)",{"type":40,"tag":767,"props":965,"children":966},{},[967,972,981],{"type":40,"tag":794,"props":968,"children":969},{},[970],{"type":45,"value":971},"handles user data that could be echoed",{"type":40,"tag":794,"props":973,"children":974},{},[975],{"type":40,"tag":92,"props":976,"children":978},{"className":977},[],[979],{"type":45,"value":980},"pii_leak",{"type":40,"tag":794,"props":982,"children":983},{},[984],{"type":40,"tag":92,"props":985,"children":987},{"className":986},[],[988],{"type":45,"value":371},{"type":40,"tag":767,"props":990,"children":991},{},[992,997,1006],{"type":40,"tag":794,"props":993,"children":994},{},[995],{"type":45,"value":996},"produces public-facing text",{"type":40,"tag":794,"props":998,"children":999},{},[1000],{"type":40,"tag":92,"props":1001,"children":1003},{"className":1002},[],[1004],{"type":45,"value":1005},"toxicity",{"type":40,"tag":794,"props":1007,"children":1008},{},[1009],{"type":40,"tag":92,"props":1010,"children":1012},{"className":1011},[],[1013],{"type":45,"value":371},{"type":40,"tag":767,"props":1015,"children":1016},{},[1017,1022,1031],{"type":40,"tag":794,"props":1018,"children":1019},{},[1020],{"type":45,"value":1021},"must always return something",{"type":40,"tag":794,"props":1023,"children":1024},{},[1025],{"type":40,"tag":92,"props":1026,"children":1028},{"className":1027},[],[1029],{"type":45,"value":1030},"response_not_empty",{"type":40,"tag":794,"props":1032,"children":1033},{},[1034],{"type":40,"tag":92,"props":1035,"children":1037},{"className":1036},[],[1038],{"type":45,"value":742},{"type":40,"tag":48,"props":1040,"children":1041},{},[1042,1044,1049,1051,1056,1058,1063,1065,1071],{"type":45,"value":1043},"Evaluator ids are yours to choose — pick clear, stable ids. Be concrete about what \"defining\"\neach one means, because it is not an Agent Observability control-plane action here: in the offline SDK flow\nan evaluator is ",{"type":40,"tag":54,"props":1045,"children":1046},{},[1047],{"type":45,"value":1048},"code the developer writes in the runner",{"type":45,"value":1050}," (Step 4). An ",{"type":40,"tag":92,"props":1052,"children":1054},{"className":1053},[],[1055],{"type":45,"value":371},{"type":45,"value":1057}," is a\nfunction that calls a model and returns a score; a ",{"type":40,"tag":92,"props":1059,"children":1061},{"className":1060},[],[1062],{"type":45,"value":742},{"type":45,"value":1064}," one is a plain code check.\n",{"type":40,"tag":92,"props":1066,"children":1068},{"className":1067},[],[1069],{"type":45,"value":1070},"agento11y.Evaluator(evaluator_id=..., kind=...)",{"type":45,"value":1072}," is only the label attached to the score, not the\nlogic. (Forking a predefined template is a separate online-eval path, not needed here.)",{"type":40,"tag":48,"props":1074,"children":1075},{},[1076,1078,1083],{"type":45,"value":1077},"This skill targets the ",{"type":40,"tag":54,"props":1079,"children":1080},{},[1081],{"type":45,"value":1082},"offline",{"type":45,"value":1084}," phase: run these evaluators as offline experiments against\nthe suite from Step 3, before there is traffic. Do not recommend live\u002Fonline evaluation to an\nagent with no traffic.",{"type":40,"tag":48,"props":1086,"children":1087},{},[1088],{"type":45,"value":1089},"Once the agent ships and has traffic, the same evaluation criteria can also be applied online\n— as Agent Observability Rules over ingested conversations, or as SDK guard hooks on the request path — but\nthose are separate Agent Observability surfaces, configured elsewhere, and out of scope for this skill.\nMention this only as a one-line \"next, once you have traffic\" note; do not instruct on it here.",{"type":40,"tag":200,"props":1091,"children":1093},{"id":1092},"step-3-write-the-suite-yaml",[1094],{"type":45,"value":1095},"Step 3 — Write the suite YAML",{"type":40,"tag":48,"props":1097,"children":1098},{},[1099,1101,1107,1109,1115],{"type":45,"value":1100},"Write a suite file in the target repo (suggest ",{"type":40,"tag":92,"props":1102,"children":1104},{"className":1103},[],[1105],{"type":45,"value":1106},"evals\u002F\u003Cagent>-starter.yaml",{"type":45,"value":1108},"). It must load\nwith the SDK's ",{"type":40,"tag":92,"props":1110,"children":1112},{"className":1111},[],[1113],{"type":45,"value":1114},"TestSuite.from_yaml(...)",{"type":45,"value":1116},", so match this schema exactly:",{"type":40,"tag":212,"props":1118,"children":1119},{},[1120,1192],{"type":40,"tag":110,"props":1121,"children":1122},{},[1123,1125,1131,1133,1139,1140,1146,1147,1153,1154,1160,1162,1168,1170,1176,1178,1183,1185,1191],{"type":45,"value":1124},"Top level: ",{"type":40,"tag":92,"props":1126,"children":1128},{"className":1127},[],[1129],{"type":45,"value":1130},"suite_id",{"type":45,"value":1132}," (required), plus optional ",{"type":40,"tag":92,"props":1134,"children":1136},{"className":1135},[],[1137],{"type":45,"value":1138},"name",{"type":45,"value":521},{"type":40,"tag":92,"props":1141,"children":1143},{"className":1142},[],[1144],{"type":45,"value":1145},"version",{"type":45,"value":521},{"type":40,"tag":92,"props":1148,"children":1150},{"className":1149},[],[1151],{"type":45,"value":1152},"description",{"type":45,"value":521},{"type":40,"tag":92,"props":1155,"children":1157},{"className":1156},[],[1158],{"type":45,"value":1159},"tags",{"type":45,"value":1161},",\n",{"type":40,"tag":92,"props":1163,"children":1165},{"className":1164},[],[1166],{"type":45,"value":1167},"changelog",{"type":45,"value":1169},", and ",{"type":40,"tag":92,"props":1171,"children":1173},{"className":1172},[],[1174],{"type":45,"value":1175},"cases",{"type":45,"value":1177}," (a list). ",{"type":40,"tag":92,"props":1179,"children":1181},{"className":1180},[],[1182],{"type":45,"value":1145},{"type":45,"value":1184}," defaults to ",{"type":40,"tag":92,"props":1186,"children":1188},{"className":1187},[],[1189],{"type":45,"value":1190},"1.0.0",{"type":45,"value":99},{"type":40,"tag":110,"props":1193,"children":1194},{},[1195,1197,1203,1204,1209,1210,1215,1216,1221,1222,1228,1229,1235,1236,1242,1243,1249,1250,1256,1258,1263,1264,1269],{"type":45,"value":1196},"Each case: ",{"type":40,"tag":92,"props":1198,"children":1200},{"className":1199},[],[1201],{"type":45,"value":1202},"id",{"type":45,"value":1132},{"type":40,"tag":92,"props":1205,"children":1207},{"className":1206},[],[1208],{"type":45,"value":1138},{"type":45,"value":521},{"type":40,"tag":92,"props":1211,"children":1213},{"className":1212},[],[1214],{"type":45,"value":1152},{"type":45,"value":521},{"type":40,"tag":92,"props":1217,"children":1219},{"className":1218},[],[1220],{"type":45,"value":1159},{"type":45,"value":521},{"type":40,"tag":92,"props":1223,"children":1225},{"className":1224},[],[1226],{"type":45,"value":1227},"category",{"type":45,"value":1161},{"type":40,"tag":92,"props":1230,"children":1232},{"className":1231},[],[1233],{"type":45,"value":1234},"input",{"type":45,"value":521},{"type":40,"tag":92,"props":1237,"children":1239},{"className":1238},[],[1240],{"type":45,"value":1241},"expected",{"type":45,"value":521},{"type":40,"tag":92,"props":1244,"children":1246},{"className":1245},[],[1247],{"type":45,"value":1248},"weight",{"type":45,"value":521},{"type":40,"tag":92,"props":1251,"children":1253},{"className":1252},[],[1254],{"type":45,"value":1255},"metadata",{"type":45,"value":1257},". ",{"type":40,"tag":92,"props":1259,"children":1261},{"className":1260},[],[1262],{"type":45,"value":1234},{"type":45,"value":325},{"type":40,"tag":92,"props":1265,"children":1267},{"className":1266},[],[1268],{"type":45,"value":1241},{"type":45,"value":1270}," are free-form (a string\nor a mapping).",{"type":40,"tag":48,"props":1272,"children":1273},{},[1274,1276,1282,1284,1290,1292,1297,1299,1305,1306,1311,1312,1317],{"type":45,"value":1275},"Derive cases from the agent's real task. Produce at least 6, weighted toward ",{"type":40,"tag":92,"props":1277,"children":1279},{"className":1278},[],[1280],{"type":45,"value":1281},"edge",{"type":45,"value":1283}," and\n",{"type":40,"tag":92,"props":1285,"children":1287},{"className":1286},[],[1288],{"type":45,"value":1289},"adversarial",{"type":45,"value":1291}," (generated cases skew easy). Use ",{"type":40,"tag":92,"props":1293,"children":1295},{"className":1294},[],[1296],{"type":45,"value":1227},{"type":45,"value":1298}," values ",{"type":40,"tag":92,"props":1300,"children":1302},{"className":1301},[],[1303],{"type":45,"value":1304},"happy",{"type":45,"value":521},{"type":40,"tag":92,"props":1307,"children":1309},{"className":1308},[],[1310],{"type":45,"value":1281},{"type":45,"value":1161},{"type":40,"tag":92,"props":1313,"children":1315},{"className":1314},[],[1316],{"type":45,"value":1289},{"type":45,"value":1318},". Keep the header comment verbatim.",{"type":40,"tag":48,"props":1320,"children":1321},{},[1322],{"type":45,"value":1323},"Every case must actually reach the agent — test the agent, not its harness. From Step 1,\nnote where the entrypoint parses\u002Fvalidates input before the model runs (e.g. a JSON parse, a\nCLI arg check). Do not generate cases that fail in that pre-agent layer (malformed JSON, wrong\nCLI flags) as if they tested the agent; if such a boundary is worth covering, label it clearly\nas a harness case in its notes, or leave it out.",{"type":40,"tag":1325,"props":1326,"children":1331},"pre",{"className":1327,"code":1328,"language":1329,"meta":1330,"style":1330},"language-yaml shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# STARTER DRAFT — review before use. Generated from your agent code (\u003Cfile:line refs>).\n# NOT validated. Add your own real cases; the edge\u002Fadversarial cases need your judgment on\n# expected behavior. Loads with agento11y.TestSuite.from_yaml(...).\nsuite_id: \u003Cagent>-starter\nname: \u003CAgent> starter suite\nversion: 1.0.0\ncases:\n  - id: happy-basic-request\n    category: happy\n    tags: [smoke]\n    input:\n      prompt: \"\u003Ca real request this agent is built to answer>\"\n    expected: \"\u003Cwhat a good answer looks like, or a rubric note>\"\n  - id: edge-underspecified\n    category: edge\n    input:\n      prompt: \"\u003Ca vague \u002F multi-part \u002F boundary request>\"\n    expected: \"\u003Chow the agent should handle ambiguity>\"\n  - id: adversarial-prompt-injection\n    category: adversarial\n    input:\n      prompt: \"\u003Can injection \u002F out-of-scope \u002F data-extraction attempt>\"\n    expected: \"\u003Cagent should refuse \u002F stay in scope \u002F not leak>\"\n","yaml","",[1332],{"type":40,"tag":92,"props":1333,"children":1334},{"__ignoreMap":1330},[1335,1347,1356,1364,1385,1402,1420,1433,1456,1474,1502,1515,1543,1569,1590,1607,1619,1644,1669,1690,1707,1719,1744],{"type":40,"tag":1336,"props":1337,"children":1340},"span",{"class":1338,"line":1339},"line",1,[1341],{"type":40,"tag":1336,"props":1342,"children":1344},{"style":1343},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1345],{"type":45,"value":1346},"# STARTER DRAFT — review before use. Generated from your agent code (\u003Cfile:line refs>).\n",{"type":40,"tag":1336,"props":1348,"children":1350},{"class":1338,"line":1349},2,[1351],{"type":40,"tag":1336,"props":1352,"children":1353},{"style":1343},[1354],{"type":45,"value":1355},"# NOT validated. Add your own real cases; the edge\u002Fadversarial cases need your judgment on\n",{"type":40,"tag":1336,"props":1357,"children":1358},{"class":1338,"line":141},[1359],{"type":40,"tag":1336,"props":1360,"children":1361},{"style":1343},[1362],{"type":45,"value":1363},"# expected behavior. Loads with agento11y.TestSuite.from_yaml(...).\n",{"type":40,"tag":1336,"props":1365,"children":1367},{"class":1338,"line":1366},4,[1368,1373,1379],{"type":40,"tag":1336,"props":1369,"children":1371},{"style":1370},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[1372],{"type":45,"value":1130},{"type":40,"tag":1336,"props":1374,"children":1376},{"style":1375},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1377],{"type":45,"value":1378},":",{"type":40,"tag":1336,"props":1380,"children":1382},{"style":1381},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1383],{"type":45,"value":1384}," \u003Cagent>-starter\n",{"type":40,"tag":1336,"props":1386,"children":1388},{"class":1338,"line":1387},5,[1389,1393,1397],{"type":40,"tag":1336,"props":1390,"children":1391},{"style":1370},[1392],{"type":45,"value":1138},{"type":40,"tag":1336,"props":1394,"children":1395},{"style":1375},[1396],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1398,"children":1399},{"style":1381},[1400],{"type":45,"value":1401}," \u003CAgent> starter suite\n",{"type":40,"tag":1336,"props":1403,"children":1405},{"class":1338,"line":1404},6,[1406,1410,1414],{"type":40,"tag":1336,"props":1407,"children":1408},{"style":1370},[1409],{"type":45,"value":1145},{"type":40,"tag":1336,"props":1411,"children":1412},{"style":1375},[1413],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1415,"children":1417},{"style":1416},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1418],{"type":45,"value":1419}," 1.0.0\n",{"type":40,"tag":1336,"props":1421,"children":1423},{"class":1338,"line":1422},7,[1424,1428],{"type":40,"tag":1336,"props":1425,"children":1426},{"style":1370},[1427],{"type":45,"value":1175},{"type":40,"tag":1336,"props":1429,"children":1430},{"style":1375},[1431],{"type":45,"value":1432},":\n",{"type":40,"tag":1336,"props":1434,"children":1436},{"class":1338,"line":1435},8,[1437,1442,1447,1451],{"type":40,"tag":1336,"props":1438,"children":1439},{"style":1375},[1440],{"type":45,"value":1441},"  -",{"type":40,"tag":1336,"props":1443,"children":1444},{"style":1370},[1445],{"type":45,"value":1446}," id",{"type":40,"tag":1336,"props":1448,"children":1449},{"style":1375},[1450],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1452,"children":1453},{"style":1381},[1454],{"type":45,"value":1455}," happy-basic-request\n",{"type":40,"tag":1336,"props":1457,"children":1459},{"class":1338,"line":1458},9,[1460,1465,1469],{"type":40,"tag":1336,"props":1461,"children":1462},{"style":1370},[1463],{"type":45,"value":1464},"    category",{"type":40,"tag":1336,"props":1466,"children":1467},{"style":1375},[1468],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1470,"children":1471},{"style":1381},[1472],{"type":45,"value":1473}," happy\n",{"type":40,"tag":1336,"props":1475,"children":1477},{"class":1338,"line":1476},10,[1478,1483,1487,1492,1497],{"type":40,"tag":1336,"props":1479,"children":1480},{"style":1370},[1481],{"type":45,"value":1482},"    tags",{"type":40,"tag":1336,"props":1484,"children":1485},{"style":1375},[1486],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1488,"children":1489},{"style":1375},[1490],{"type":45,"value":1491}," [",{"type":40,"tag":1336,"props":1493,"children":1494},{"style":1381},[1495],{"type":45,"value":1496},"smoke",{"type":40,"tag":1336,"props":1498,"children":1499},{"style":1375},[1500],{"type":45,"value":1501},"]\n",{"type":40,"tag":1336,"props":1503,"children":1505},{"class":1338,"line":1504},11,[1506,1511],{"type":40,"tag":1336,"props":1507,"children":1508},{"style":1370},[1509],{"type":45,"value":1510},"    input",{"type":40,"tag":1336,"props":1512,"children":1513},{"style":1375},[1514],{"type":45,"value":1432},{"type":40,"tag":1336,"props":1516,"children":1518},{"class":1338,"line":1517},12,[1519,1524,1528,1533,1538],{"type":40,"tag":1336,"props":1520,"children":1521},{"style":1370},[1522],{"type":45,"value":1523},"      prompt",{"type":40,"tag":1336,"props":1525,"children":1526},{"style":1375},[1527],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1529,"children":1530},{"style":1375},[1531],{"type":45,"value":1532}," \"",{"type":40,"tag":1336,"props":1534,"children":1535},{"style":1381},[1536],{"type":45,"value":1537},"\u003Ca real request this agent is built to answer>",{"type":40,"tag":1336,"props":1539,"children":1540},{"style":1375},[1541],{"type":45,"value":1542},"\"\n",{"type":40,"tag":1336,"props":1544,"children":1546},{"class":1338,"line":1545},13,[1547,1552,1556,1560,1565],{"type":40,"tag":1336,"props":1548,"children":1549},{"style":1370},[1550],{"type":45,"value":1551},"    expected",{"type":40,"tag":1336,"props":1553,"children":1554},{"style":1375},[1555],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1557,"children":1558},{"style":1375},[1559],{"type":45,"value":1532},{"type":40,"tag":1336,"props":1561,"children":1562},{"style":1381},[1563],{"type":45,"value":1564},"\u003Cwhat a good answer looks like, or a rubric note>",{"type":40,"tag":1336,"props":1566,"children":1567},{"style":1375},[1568],{"type":45,"value":1542},{"type":40,"tag":1336,"props":1570,"children":1572},{"class":1338,"line":1571},14,[1573,1577,1581,1585],{"type":40,"tag":1336,"props":1574,"children":1575},{"style":1375},[1576],{"type":45,"value":1441},{"type":40,"tag":1336,"props":1578,"children":1579},{"style":1370},[1580],{"type":45,"value":1446},{"type":40,"tag":1336,"props":1582,"children":1583},{"style":1375},[1584],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1586,"children":1587},{"style":1381},[1588],{"type":45,"value":1589}," edge-underspecified\n",{"type":40,"tag":1336,"props":1591,"children":1593},{"class":1338,"line":1592},15,[1594,1598,1602],{"type":40,"tag":1336,"props":1595,"children":1596},{"style":1370},[1597],{"type":45,"value":1464},{"type":40,"tag":1336,"props":1599,"children":1600},{"style":1375},[1601],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1603,"children":1604},{"style":1381},[1605],{"type":45,"value":1606}," edge\n",{"type":40,"tag":1336,"props":1608,"children":1610},{"class":1338,"line":1609},16,[1611,1615],{"type":40,"tag":1336,"props":1612,"children":1613},{"style":1370},[1614],{"type":45,"value":1510},{"type":40,"tag":1336,"props":1616,"children":1617},{"style":1375},[1618],{"type":45,"value":1432},{"type":40,"tag":1336,"props":1620,"children":1622},{"class":1338,"line":1621},17,[1623,1627,1631,1635,1640],{"type":40,"tag":1336,"props":1624,"children":1625},{"style":1370},[1626],{"type":45,"value":1523},{"type":40,"tag":1336,"props":1628,"children":1629},{"style":1375},[1630],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1632,"children":1633},{"style":1375},[1634],{"type":45,"value":1532},{"type":40,"tag":1336,"props":1636,"children":1637},{"style":1381},[1638],{"type":45,"value":1639},"\u003Ca vague \u002F multi-part \u002F boundary request>",{"type":40,"tag":1336,"props":1641,"children":1642},{"style":1375},[1643],{"type":45,"value":1542},{"type":40,"tag":1336,"props":1645,"children":1647},{"class":1338,"line":1646},18,[1648,1652,1656,1660,1665],{"type":40,"tag":1336,"props":1649,"children":1650},{"style":1370},[1651],{"type":45,"value":1551},{"type":40,"tag":1336,"props":1653,"children":1654},{"style":1375},[1655],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1657,"children":1658},{"style":1375},[1659],{"type":45,"value":1532},{"type":40,"tag":1336,"props":1661,"children":1662},{"style":1381},[1663],{"type":45,"value":1664},"\u003Chow the agent should handle ambiguity>",{"type":40,"tag":1336,"props":1666,"children":1667},{"style":1375},[1668],{"type":45,"value":1542},{"type":40,"tag":1336,"props":1670,"children":1672},{"class":1338,"line":1671},19,[1673,1677,1681,1685],{"type":40,"tag":1336,"props":1674,"children":1675},{"style":1375},[1676],{"type":45,"value":1441},{"type":40,"tag":1336,"props":1678,"children":1679},{"style":1370},[1680],{"type":45,"value":1446},{"type":40,"tag":1336,"props":1682,"children":1683},{"style":1375},[1684],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1686,"children":1687},{"style":1381},[1688],{"type":45,"value":1689}," adversarial-prompt-injection\n",{"type":40,"tag":1336,"props":1691,"children":1693},{"class":1338,"line":1692},20,[1694,1698,1702],{"type":40,"tag":1336,"props":1695,"children":1696},{"style":1370},[1697],{"type":45,"value":1464},{"type":40,"tag":1336,"props":1699,"children":1700},{"style":1375},[1701],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1703,"children":1704},{"style":1381},[1705],{"type":45,"value":1706}," adversarial\n",{"type":40,"tag":1336,"props":1708,"children":1710},{"class":1338,"line":1709},21,[1711,1715],{"type":40,"tag":1336,"props":1712,"children":1713},{"style":1370},[1714],{"type":45,"value":1510},{"type":40,"tag":1336,"props":1716,"children":1717},{"style":1375},[1718],{"type":45,"value":1432},{"type":40,"tag":1336,"props":1720,"children":1722},{"class":1338,"line":1721},22,[1723,1727,1731,1735,1740],{"type":40,"tag":1336,"props":1724,"children":1725},{"style":1370},[1726],{"type":45,"value":1523},{"type":40,"tag":1336,"props":1728,"children":1729},{"style":1375},[1730],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1732,"children":1733},{"style":1375},[1734],{"type":45,"value":1532},{"type":40,"tag":1336,"props":1736,"children":1737},{"style":1381},[1738],{"type":45,"value":1739},"\u003Can injection \u002F out-of-scope \u002F data-extraction attempt>",{"type":40,"tag":1336,"props":1741,"children":1742},{"style":1375},[1743],{"type":45,"value":1542},{"type":40,"tag":1336,"props":1745,"children":1747},{"class":1338,"line":1746},23,[1748,1752,1756,1760,1765],{"type":40,"tag":1336,"props":1749,"children":1750},{"style":1370},[1751],{"type":45,"value":1551},{"type":40,"tag":1336,"props":1753,"children":1754},{"style":1375},[1755],{"type":45,"value":1378},{"type":40,"tag":1336,"props":1757,"children":1758},{"style":1375},[1759],{"type":45,"value":1532},{"type":40,"tag":1336,"props":1761,"children":1762},{"style":1381},[1763],{"type":45,"value":1764},"\u003Cagent should refuse \u002F stay in scope \u002F not leak>",{"type":40,"tag":1336,"props":1766,"children":1767},{"style":1375},[1768],{"type":45,"value":1542},{"type":40,"tag":200,"props":1770,"children":1772},{"id":1771},"step-4-write-the-runner-stub-branch-on-runnability",[1773],{"type":45,"value":1774},"Step 4 — Write the runner stub (branch on runnability)",{"type":40,"tag":48,"props":1776,"children":1777},{},[1778,1780,1785,1787,1792],{"type":45,"value":1779},"The suite YAML alone does not run anything. The Agent Observability SDK stores and aggregates scores, but\nit does ",{"type":40,"tag":54,"props":1781,"children":1782},{},[1783],{"type":45,"value":1784},"not",{"type":45,"value":1786}," run the agent or compute the evaluators — the developer writes both. Left at\njust the YAML, a developer new to offline eval is still blocked (\"now what?\"). So generate a\n",{"type":40,"tag":54,"props":1788,"children":1789},{},[1790],{"type":45,"value":1791},"minimal bootstrap runner",{"type":45,"value":1793}," — just enough to get one experiment running.",{"type":40,"tag":48,"props":1795,"children":1796},{},[1797,1799,1804,1806,1812,1814,1819],{"type":45,"value":1798},"This is deliberately the simplest path, not the full run-side API. The ",{"type":40,"tag":92,"props":1800,"children":1802},{"className":1801},[],[1803],{"type":45,"value":173},{"type":45,"value":1805},"\nskill is the reference for everything beyond bootstrap — binding an already-instrumented\nagent's real generations\u002Fconversations, auditable LLM-judge grading, cross-process verifiers\n(",{"type":40,"tag":92,"props":1807,"children":1809},{"className":1808},[],[1810],{"type":45,"value":1811},"TrialRef",{"type":45,"value":1813},"), pass@k\u002Fpass^k. Don't reproduce those patterns here; generate the minimal runner\nand point to ",{"type":40,"tag":92,"props":1815,"children":1817},{"className":1816},[],[1818],{"type":45,"value":173},{"type":45,"value":1820}," for depth.",{"type":40,"tag":48,"props":1822,"children":1823},{},[1824],{"type":45,"value":1825},"What you generate depends on the runnability you assessed in Step 1:",{"type":40,"tag":212,"props":1827,"children":1828},{},[1829,1853,1876],{"type":40,"tag":110,"props":1830,"children":1831},{},[1832,1837,1839,1845,1847,1852],{"type":40,"tag":54,"props":1833,"children":1834},{},[1835],{"type":45,"value":1836},"easy",{"type":45,"value":1838}," → generate the full ",{"type":40,"tag":92,"props":1840,"children":1842},{"className":1841},[],[1843],{"type":45,"value":1844},"evals\u002Frun_experiment.py",{"type":45,"value":1846}," below. One hole to fill\n(",{"type":40,"tag":92,"props":1848,"children":1850},{"className":1849},[],[1851],{"type":45,"value":651},{"type":45,"value":175},{"type":40,"tag":110,"props":1854,"children":1855},{},[1856,1860,1862,1867,1869,1874],{"type":40,"tag":54,"props":1857,"children":1858},{},[1859],{"type":45,"value":673},{"type":45,"value":1861}," → do NOT emit a Python ",{"type":40,"tag":92,"props":1863,"children":1865},{"className":1864},[],[1866],{"type":45,"value":651},{"type":45,"value":1868}," that can't actually call the agent (e.g.\na Go agent). Generate the same experiment wiring, but make ",{"type":40,"tag":92,"props":1870,"children":1872},{"className":1871},[],[1873],{"type":45,"value":651},{"type":45,"value":1875}," shell out to a small\nharness in the agent's language (or write that harness), and say plainly the seam is the\ninjectable LLM client. Point to any existing test that already invokes the agent in-process\nas the template.",{"type":40,"tag":110,"props":1877,"children":1878},{},[1879,1883],{"type":40,"tag":54,"props":1880,"children":1881},{},[1882],{"type":45,"value":681},{"type":45,"value":1884}," → do NOT emit a runner that pretends to call the agent as a function. The\nagent runs via its existing eval infra (dedicated harness, Docker stack, HTTP + polling).\nDeliver the recommendations + YAML, and point to that infra and to any existing\nscenario\u002Fground-truth files as the real test cases. Be explicit that isolated runs aren't\nthe path here.",{"type":40,"tag":48,"props":1886,"children":1887},{},[1888,1890,1895],{"type":45,"value":1889},"Also branch on ",{"type":40,"tag":54,"props":1891,"children":1892},{},[1893],{"type":45,"value":1894},"language",{"type":45,"value":1896}," (the experiments SDK is Python\u002FGo only):",{"type":40,"tag":212,"props":1898,"children":1899},{},[1900,1924],{"type":40,"tag":110,"props":1901,"children":1902},{},[1903,1908,1910,1915,1917,1922],{"type":40,"tag":54,"props":1904,"children":1905},{},[1906],{"type":45,"value":1907},"Python or Go agent",{"type":45,"value":1909}," → native runner (",{"type":40,"tag":92,"props":1911,"children":1913},{"className":1912},[],[1914],{"type":45,"value":160},{"type":45,"value":1916},", or the Go ",{"type":40,"tag":92,"props":1918,"children":1920},{"className":1919},[],[1921],{"type":45,"value":262},{"type":45,"value":1923}," package).",{"type":40,"tag":110,"props":1925,"children":1926},{},[1927,1932,1934,1940],{"type":40,"tag":54,"props":1928,"children":1929},{},[1930],{"type":45,"value":1931},"TS \u002F Java \u002F .NET agent",{"type":45,"value":1933}," → there is no experiments SDK in that language. Deliver\nrecommendations + YAML (they are language-neutral), and be honest about the run path: the\nrunner must be Python or Go calling the agent across a process boundary (e.g. a Python\nrunner that shells out to ",{"type":40,"tag":92,"props":1935,"children":1937},{"className":1936},[],[1938],{"type":45,"value":1939},"node your-agent.js",{"type":45,"value":1941}," and reads its output), or the developer waits\nfor experiments support in their language. Offer the subprocess bridge only as a labeled\noption with its cost (serializing input to the CLI, parsing output), not as a clean default.",{"type":40,"tag":48,"props":1943,"children":1944},{},[1945,1947,1952,1954,1959],{"type":45,"value":1946},"For an ",{"type":40,"tag":54,"props":1948,"children":1949},{},[1950],{"type":45,"value":1951},"easy Python\u002FGo",{"type":45,"value":1953}," agent, write ",{"type":40,"tag":92,"props":1955,"children":1957},{"className":1956},[],[1958],{"type":45,"value":1844},{"type":45,"value":1960},". It must:",{"type":40,"tag":212,"props":1962,"children":1963},{},[1964,1975,1996,2016,2080],{"type":40,"tag":110,"props":1965,"children":1966},{},[1967,1969,1974],{"type":45,"value":1968},"Load the suite with ",{"type":40,"tag":92,"props":1970,"children":1972},{"className":1971},[],[1973],{"type":45,"value":1114},{"type":45,"value":99},{"type":40,"tag":110,"props":1976,"children":1977},{},[1978,1980,1986,1988,1994],{"type":45,"value":1979},"Open an experiment (",{"type":40,"tag":92,"props":1981,"children":1983},{"className":1982},[],[1984],{"type":45,"value":1985},"agento11y.experiment(...)",{"type":45,"value":1987},") and one ",{"type":40,"tag":92,"props":1989,"children":1991},{"className":1990},[],[1992],{"type":45,"value":1993},"trial",{"type":45,"value":1995}," per case.",{"type":40,"tag":110,"props":1997,"children":1998},{},[1999,2001,2007,2009,2014],{"type":45,"value":2000},"Call the agent through a single clearly-marked function ",{"type":40,"tag":92,"props":2002,"children":2004},{"className":2003},[],[2005],{"type":45,"value":2006},"run_agent(case)",{"type":45,"value":2008}," — ",{"type":40,"tag":54,"props":2010,"children":2011},{},[2012],{"type":45,"value":2013},"the first of the\ntwo holes the developer fills",{"type":45,"value":2015},"; wire it to the real entrypoint you found in Step 1.",{"type":40,"tag":110,"props":2017,"children":2018},{},[2019,2021,2026,2028,2034,2036,2041,2043,2048,2050,2055,2057,2063,2065,2070,2072,2078],{"type":45,"value":2020},"Include ONE recommended evaluator sketched end-to-end (prefer an ",{"type":40,"tag":92,"props":2022,"children":2024},{"className":2023},[],[2025],{"type":45,"value":371},{"type":45,"value":2027}," — a real model\ncall that returns a JSON ",{"type":40,"tag":92,"props":2029,"children":2031},{"className":2030},[],[2032],{"type":45,"value":2033},"{score, passed, explanation}",{"type":45,"value":2035},"), so they see the shape and can copy\nit for the others. Reference the rest by name in a comment; do not stub all of them. ",{"type":40,"tag":54,"props":2037,"children":2038},{},[2039],{"type":45,"value":2040},"The\njudge's model call must use the provider the agent already uses",{"type":45,"value":2042}," (reuse its OpenAI\u002FAnthropic\nclient or SDK) — do not pull in ",{"type":40,"tag":92,"props":2044,"children":2046},{"className":2045},[],[2047],{"type":45,"value":381},{"type":45,"value":2049}," or another provider just for the judge. The one model\ncall inside the judge is ",{"type":40,"tag":54,"props":2051,"children":2052},{},[2053],{"type":45,"value":2054},"the second hole",{"type":45,"value":2056}," — leave it as an explicit ",{"type":40,"tag":92,"props":2058,"children":2060},{"className":2059},[],[2061],{"type":45,"value":2062},"NotImplementedError",{"type":45,"value":2064}," so a\ndeveloper who only fills ",{"type":40,"tag":92,"props":2066,"children":2068},{"className":2067},[],[2069],{"type":45,"value":651},{"type":45,"value":2071}," gets a clear \"fill the judge call\" error, not a ",{"type":40,"tag":92,"props":2073,"children":2075},{"className":2074},[],[2076],{"type":45,"value":2077},"NameError",{"type":45,"value":2079},"\non an undefined helper.",{"type":40,"tag":110,"props":2081,"children":2082},{},[2083,2085,2091,2093,2099],{"type":45,"value":2084},"Record I\u002FO (",{"type":40,"tag":92,"props":2086,"children":2088},{"className":2087},[],[2089],{"type":45,"value":2090},"trial.record_io(...)",{"type":45,"value":2092},") and emit ",{"type":40,"tag":92,"props":2094,"children":2096},{"className":2095},[],[2097],{"type":45,"value":2098},"trial.final_score(...)",{"type":45,"value":2100}," with the evaluator.",{"type":40,"tag":48,"props":2102,"children":2103},{},[2104],{"type":45,"value":2105},"Keep the header verbatim, and be honest in it about what still needs doing:",{"type":40,"tag":1325,"props":2107,"children":2111},{"className":2108,"code":2109,"language":2110,"meta":1330,"style":1330},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","#!\u002Fusr\u002Fbin\u002Fenv python3\n\"\"\"STARTER RUNNER — generated by agento11y-test-starter, review before use.\n\nRuns \u003Cagent> over evals\u002F\u003Cagent>-starter.yaml as an Agent Observability experiment and publishes scores.\n\nYou still need to: (1) fill run_agent(case) to call YOUR agent (first hole); (2) fill the model\ncall inside judge_\u003Cevaluator> using the agent's own provider client, then tune it (second hole);\n(3) set real credentials — `AGENTO11Y_ENDPOINT` and `AGENTO11Y_AUTH_TOKEN` (your Grafana Cloud\ningestion key) and the agent's provider key. The SDK stores scores; it does not run the agent or\nthe judge.\n\nSet AGENTO11Y_INGEST_ACTOR to a stable value: the run and its trials must share one actor, or\ntrial creation fails with \"401: experiment is owned by another actor\".\n\n    AGENTO11Y_ENDPOINT=... AGENTO11Y_AUTH_TOKEN=... AGENTO11Y_INGEST_ACTOR=ingest:sdk\u002Fpython \\\n        python evals\u002Frun_experiment.py\n\"\"\"\nimport json, os, time\nfrom pathlib import Path\nfrom dotenv import load_dotenv\nfrom agento11y import experiments as agento11y\n\nload_dotenv()\nSUITE = Path(__file__).parent \u002F \"\u003Cagent>-starter.yaml\"\n\n\ndef run_agent(case: agento11y.TestCase) -> str:\n    \"\"\"FIRST HOLE YOU FILL — call your agent for this case, return its output text.\"\"\"\n    raise NotImplementedError(\"wire this to your agent entrypoint (see Step 1 refs)\")\n\n\ndef judge_\u003Cevaluator>(case_input, output) -> tuple[float, bool, str]:\n    \"\"\"Sketched llm_judge — a model call returning (score 0-1, passed, explanation). Tune it.\n\n    Uses the SAME provider client the agent uses (imported below from the agent module) — do NOT\n    swap in litellm or another provider. Adapt the call to your provider's API.\n\n    IMPORTANT for grounding\u002Frelevance judges: the judge must SEE what the agent saw. If the agent\n    retrieves context (RAG), re-run its retriever here and put the passages in the prompt —\n    otherwise the judge marks correct, cited answers as \"unverifiable\" and scores them low. Import\n    the agent's own retriever (e.g. `from agent import retrieve`) and include its output.\n    \"\"\"\n    prompt = (\n        'Grade \u003Cwhat this evaluator checks>. Reply with ONLY a JSON object, no prose, no markdown '\n        'fences, explanation LAST and short: {\"score\": \u003C0-1 float>, \"passed\": \u003Cbool>, \"explanation\": \"\u003Cone sentence>\"}.\\n\\n'\n        # For a grounding judge, prepend the retrieved context here:\n        # f\"CONTEXT:\\n{retrieved_passages}\\n\\n\"\n        f\"Input:\\n{case_input}\\n\\nOutput:\\n{output}\"\n    )\n    model = os.getenv(\"GRADER_MODEL\") or os.getenv(\"MODEL_NAME\")  # a LIVE model id; no default (dead ids 404)\n    # SECOND HOLE YOU FILL — call the model using the SAME client the agent uses (do NOT add a new\n    # provider). Import it from the agent module and make ONE call that returns the reply text into\n    # `text`. Give it enough max_tokens (~600) that the JSON verdict is never truncated mid-object.\n    #   Anthropic: from agent import client\n    #     text = client.messages.create(model=model, max_tokens=600,\n    #                                    messages=[{\"role\": \"user\", \"content\": prompt}]).content[0].text\n    #   OpenAI:    from agent import client\n    #     text = client.chat.completions.create(model=model, max_tokens=600,\n    #                                            messages=[{\"role\": \"user\", \"content\": prompt}]).choices[0].message.content\n    raise NotImplementedError(\"call the agent's provider client here and assign the reply to `text`\")\n    d = _parse_judge_json(text)\n    score = max(0.0, min(1.0, float(d.get(\"score\", 0.0))))\n    return score, bool(d.get(\"passed\", score >= 0.6)), str(d.get(\"explanation\", \"\"))\n\n\ndef _parse_judge_json(text: str) -> dict:\n    \"\"\"Robustly pull the JSON verdict out of a model reply (handles prose \u002F ```json fences).\"\"\"\n    if not text:\n        return {}\n    t = text.strip()\n    if \"```\" in t:  # strip a ```json ... ``` fence if present\n        t = t.split(\"```\")[1].removeprefix(\"json\").strip() if t.count(\"```\") >= 2 else t\n    try:\n        return json.loads(t)\n    except json.JSONDecodeError:\n        pass\n    # Fallback: scan for the first balanced {...} object rather than first-brace\u002Flast-brace.\n    depth = 0; start = -1\n    for i, ch in enumerate(t):\n        if ch == \"{\":\n            if depth == 0:\n                start = i\n            depth += 1\n        elif ch == \"}\":\n            depth -= 1\n            if depth == 0 and start >= 0:\n                try:\n                    return json.loads(t[start:i + 1])\n                except json.JSONDecodeError:\n                    start = -1\n    return {}  # unparseable → caller treats as score 0; investigate the raw reply\n\n\ndef main() -> None:\n    suite = agento11y.TestSuite.from_yaml(str(SUITE))\n    verifier = agento11y.Evaluator(evaluator_id=\"\u003Cevaluator>\", version=\"draft-0\", kind=\"llm_judge\")\n    candidate = {\n        \"agent_name\": \"\u003Cagent>\",\n        # Always send a declared agent_version. Without it Agent Observability auto-derives a version from\n        # the system-prompt hash, so you can't reliably attribute scores to a version or\n        # compare versions. Replace \"v1\" with your real version (git tag, prompt version,\n        # semver...) — the \"v1\" fallback is a placeholder, not a version worth comparing.\n        \"agent_version\": os.getenv(\"AGENT_VERSION\", \"v1\"),\n        \"git_sha\": os.getenv(\"GIT_SHA\", \"\"),\n        \"model_name\": os.getenv(\"MODEL_NAME\", \"\"),\n    }\n    with agento11y.experiment(name=\"\u003Cagent> starter\", experiment_id=f\"\u003Cagent>-starter-{int(time.time())}\",\n                          suite=suite, candidate=candidate, tags=[\"starter\"],\n                          actor=os.getenv(\"AGENTO11Y_INGEST_ACTOR\", \"ingest:sdk\u002Fpython\")) as exp:\n        for case in suite.test_cases:\n            with exp.trial(case) as trial:\n                out = run_agent(case)\n                trial.record_io(input=json.dumps(case.input), output=out,\n                                model_provider=\"\u003Cprovider>\", model_name=os.getenv(\"MODEL_NAME\", \"\"))\n                score, passed, why = judge_\u003Cevaluator>(case.input, out)\n                trial.final_score(score, passed=passed, explanation=why, evaluator=verifier)\n                print(f\"  {case.test_case_id}: score={score:.2f} passed={passed}\")\n    print(f\"\\nExperiment: {exp.url}\")\n\n\nif __name__ == \"__main__\":\n    main()\n","python",[2112],{"type":40,"tag":92,"props":2113,"children":2114},{"__ignoreMap":1330},[2115,2123,2131,2140,2148,2155,2163,2171,2179,2187,2195,2202,2210,2218,2225,2233,2241,2249,2257,2265,2273,2281,2288,2296,2305,2313,2321,2330,2339,2347,2355,2363,2372,2381,2389,2398,2407,2415,2424,2433,2442,2451,2460,2469,2478,2487,2496,2505,2514,2523,2532,2541,2550,2559,2568,2577,2586,2595,2604,2613,2622,2631,2640,2649,2657,2665,2674,2683,2692,2701,2710,2719,2728,2737,2746,2755,2764,2773,2782,2791,2800,2809,2818,2827,2836,2845,2854,2863,2872,2881,2890,2899,2907,2915,2924,2933,2942,2951,2960,2969,2978,2987,2996,3005,3014,3023,3032,3041,3050,3059,3068,3077,3086,3095,3104,3113,3122,3131,3140,3148,3156,3165],{"type":40,"tag":1336,"props":2116,"children":2117},{"class":1338,"line":1339},[2118],{"type":40,"tag":1336,"props":2119,"children":2120},{},[2121],{"type":45,"value":2122},"#!\u002Fusr\u002Fbin\u002Fenv python3\n",{"type":40,"tag":1336,"props":2124,"children":2125},{"class":1338,"line":1349},[2126],{"type":40,"tag":1336,"props":2127,"children":2128},{},[2129],{"type":45,"value":2130},"\"\"\"STARTER RUNNER — generated by agento11y-test-starter, review before use.\n",{"type":40,"tag":1336,"props":2132,"children":2133},{"class":1338,"line":141},[2134],{"type":40,"tag":1336,"props":2135,"children":2137},{"emptyLinePlaceholder":2136},true,[2138],{"type":45,"value":2139},"\n",{"type":40,"tag":1336,"props":2141,"children":2142},{"class":1338,"line":1366},[2143],{"type":40,"tag":1336,"props":2144,"children":2145},{},[2146],{"type":45,"value":2147},"Runs \u003Cagent> over evals\u002F\u003Cagent>-starter.yaml as an Agent Observability experiment and publishes scores.\n",{"type":40,"tag":1336,"props":2149,"children":2150},{"class":1338,"line":1387},[2151],{"type":40,"tag":1336,"props":2152,"children":2153},{"emptyLinePlaceholder":2136},[2154],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2156,"children":2157},{"class":1338,"line":1404},[2158],{"type":40,"tag":1336,"props":2159,"children":2160},{},[2161],{"type":45,"value":2162},"You still need to: (1) fill run_agent(case) to call YOUR agent (first hole); (2) fill the model\n",{"type":40,"tag":1336,"props":2164,"children":2165},{"class":1338,"line":1422},[2166],{"type":40,"tag":1336,"props":2167,"children":2168},{},[2169],{"type":45,"value":2170},"call inside judge_\u003Cevaluator> using the agent's own provider client, then tune it (second hole);\n",{"type":40,"tag":1336,"props":2172,"children":2173},{"class":1338,"line":1435},[2174],{"type":40,"tag":1336,"props":2175,"children":2176},{},[2177],{"type":45,"value":2178},"(3) set real credentials — `AGENTO11Y_ENDPOINT` and `AGENTO11Y_AUTH_TOKEN` (your Grafana Cloud\n",{"type":40,"tag":1336,"props":2180,"children":2181},{"class":1338,"line":1458},[2182],{"type":40,"tag":1336,"props":2183,"children":2184},{},[2185],{"type":45,"value":2186},"ingestion key) and the agent's provider key. The SDK stores scores; it does not run the agent or\n",{"type":40,"tag":1336,"props":2188,"children":2189},{"class":1338,"line":1476},[2190],{"type":40,"tag":1336,"props":2191,"children":2192},{},[2193],{"type":45,"value":2194},"the judge.\n",{"type":40,"tag":1336,"props":2196,"children":2197},{"class":1338,"line":1504},[2198],{"type":40,"tag":1336,"props":2199,"children":2200},{"emptyLinePlaceholder":2136},[2201],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2203,"children":2204},{"class":1338,"line":1517},[2205],{"type":40,"tag":1336,"props":2206,"children":2207},{},[2208],{"type":45,"value":2209},"Set AGENTO11Y_INGEST_ACTOR to a stable value: the run and its trials must share one actor, or\n",{"type":40,"tag":1336,"props":2211,"children":2212},{"class":1338,"line":1545},[2213],{"type":40,"tag":1336,"props":2214,"children":2215},{},[2216],{"type":45,"value":2217},"trial creation fails with \"401: experiment is owned by another actor\".\n",{"type":40,"tag":1336,"props":2219,"children":2220},{"class":1338,"line":1571},[2221],{"type":40,"tag":1336,"props":2222,"children":2223},{"emptyLinePlaceholder":2136},[2224],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2226,"children":2227},{"class":1338,"line":1592},[2228],{"type":40,"tag":1336,"props":2229,"children":2230},{},[2231],{"type":45,"value":2232},"    AGENTO11Y_ENDPOINT=... AGENTO11Y_AUTH_TOKEN=... AGENTO11Y_INGEST_ACTOR=ingest:sdk\u002Fpython \\\n",{"type":40,"tag":1336,"props":2234,"children":2235},{"class":1338,"line":1609},[2236],{"type":40,"tag":1336,"props":2237,"children":2238},{},[2239],{"type":45,"value":2240},"        python evals\u002Frun_experiment.py\n",{"type":40,"tag":1336,"props":2242,"children":2243},{"class":1338,"line":1621},[2244],{"type":40,"tag":1336,"props":2245,"children":2246},{},[2247],{"type":45,"value":2248},"\"\"\"\n",{"type":40,"tag":1336,"props":2250,"children":2251},{"class":1338,"line":1646},[2252],{"type":40,"tag":1336,"props":2253,"children":2254},{},[2255],{"type":45,"value":2256},"import json, os, time\n",{"type":40,"tag":1336,"props":2258,"children":2259},{"class":1338,"line":1671},[2260],{"type":40,"tag":1336,"props":2261,"children":2262},{},[2263],{"type":45,"value":2264},"from pathlib import Path\n",{"type":40,"tag":1336,"props":2266,"children":2267},{"class":1338,"line":1692},[2268],{"type":40,"tag":1336,"props":2269,"children":2270},{},[2271],{"type":45,"value":2272},"from dotenv import load_dotenv\n",{"type":40,"tag":1336,"props":2274,"children":2275},{"class":1338,"line":1709},[2276],{"type":40,"tag":1336,"props":2277,"children":2278},{},[2279],{"type":45,"value":2280},"from agento11y import experiments as agento11y\n",{"type":40,"tag":1336,"props":2282,"children":2283},{"class":1338,"line":1721},[2284],{"type":40,"tag":1336,"props":2285,"children":2286},{"emptyLinePlaceholder":2136},[2287],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2289,"children":2290},{"class":1338,"line":1746},[2291],{"type":40,"tag":1336,"props":2292,"children":2293},{},[2294],{"type":45,"value":2295},"load_dotenv()\n",{"type":40,"tag":1336,"props":2297,"children":2299},{"class":1338,"line":2298},24,[2300],{"type":40,"tag":1336,"props":2301,"children":2302},{},[2303],{"type":45,"value":2304},"SUITE = Path(__file__).parent \u002F \"\u003Cagent>-starter.yaml\"\n",{"type":40,"tag":1336,"props":2306,"children":2308},{"class":1338,"line":2307},25,[2309],{"type":40,"tag":1336,"props":2310,"children":2311},{"emptyLinePlaceholder":2136},[2312],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2314,"children":2316},{"class":1338,"line":2315},26,[2317],{"type":40,"tag":1336,"props":2318,"children":2319},{"emptyLinePlaceholder":2136},[2320],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2322,"children":2324},{"class":1338,"line":2323},27,[2325],{"type":40,"tag":1336,"props":2326,"children":2327},{},[2328],{"type":45,"value":2329},"def run_agent(case: agento11y.TestCase) -> str:\n",{"type":40,"tag":1336,"props":2331,"children":2333},{"class":1338,"line":2332},28,[2334],{"type":40,"tag":1336,"props":2335,"children":2336},{},[2337],{"type":45,"value":2338},"    \"\"\"FIRST HOLE YOU FILL — call your agent for this case, return its output text.\"\"\"\n",{"type":40,"tag":1336,"props":2340,"children":2341},{"class":1338,"line":27},[2342],{"type":40,"tag":1336,"props":2343,"children":2344},{},[2345],{"type":45,"value":2346},"    raise NotImplementedError(\"wire this to your agent entrypoint (see Step 1 refs)\")\n",{"type":40,"tag":1336,"props":2348,"children":2350},{"class":1338,"line":2349},30,[2351],{"type":40,"tag":1336,"props":2352,"children":2353},{"emptyLinePlaceholder":2136},[2354],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2356,"children":2358},{"class":1338,"line":2357},31,[2359],{"type":40,"tag":1336,"props":2360,"children":2361},{"emptyLinePlaceholder":2136},[2362],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2364,"children":2366},{"class":1338,"line":2365},32,[2367],{"type":40,"tag":1336,"props":2368,"children":2369},{},[2370],{"type":45,"value":2371},"def judge_\u003Cevaluator>(case_input, output) -> tuple[float, bool, str]:\n",{"type":40,"tag":1336,"props":2373,"children":2375},{"class":1338,"line":2374},33,[2376],{"type":40,"tag":1336,"props":2377,"children":2378},{},[2379],{"type":45,"value":2380},"    \"\"\"Sketched llm_judge — a model call returning (score 0-1, passed, explanation). Tune it.\n",{"type":40,"tag":1336,"props":2382,"children":2384},{"class":1338,"line":2383},34,[2385],{"type":40,"tag":1336,"props":2386,"children":2387},{"emptyLinePlaceholder":2136},[2388],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2390,"children":2392},{"class":1338,"line":2391},35,[2393],{"type":40,"tag":1336,"props":2394,"children":2395},{},[2396],{"type":45,"value":2397},"    Uses the SAME provider client the agent uses (imported below from the agent module) — do NOT\n",{"type":40,"tag":1336,"props":2399,"children":2401},{"class":1338,"line":2400},36,[2402],{"type":40,"tag":1336,"props":2403,"children":2404},{},[2405],{"type":45,"value":2406},"    swap in litellm or another provider. Adapt the call to your provider's API.\n",{"type":40,"tag":1336,"props":2408,"children":2410},{"class":1338,"line":2409},37,[2411],{"type":40,"tag":1336,"props":2412,"children":2413},{"emptyLinePlaceholder":2136},[2414],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2416,"children":2418},{"class":1338,"line":2417},38,[2419],{"type":40,"tag":1336,"props":2420,"children":2421},{},[2422],{"type":45,"value":2423},"    IMPORTANT for grounding\u002Frelevance judges: the judge must SEE what the agent saw. If the agent\n",{"type":40,"tag":1336,"props":2425,"children":2427},{"class":1338,"line":2426},39,[2428],{"type":40,"tag":1336,"props":2429,"children":2430},{},[2431],{"type":45,"value":2432},"    retrieves context (RAG), re-run its retriever here and put the passages in the prompt —\n",{"type":40,"tag":1336,"props":2434,"children":2436},{"class":1338,"line":2435},40,[2437],{"type":40,"tag":1336,"props":2438,"children":2439},{},[2440],{"type":45,"value":2441},"    otherwise the judge marks correct, cited answers as \"unverifiable\" and scores them low. Import\n",{"type":40,"tag":1336,"props":2443,"children":2445},{"class":1338,"line":2444},41,[2446],{"type":40,"tag":1336,"props":2447,"children":2448},{},[2449],{"type":45,"value":2450},"    the agent's own retriever (e.g. `from agent import retrieve`) and include its output.\n",{"type":40,"tag":1336,"props":2452,"children":2454},{"class":1338,"line":2453},42,[2455],{"type":40,"tag":1336,"props":2456,"children":2457},{},[2458],{"type":45,"value":2459},"    \"\"\"\n",{"type":40,"tag":1336,"props":2461,"children":2463},{"class":1338,"line":2462},43,[2464],{"type":40,"tag":1336,"props":2465,"children":2466},{},[2467],{"type":45,"value":2468},"    prompt = (\n",{"type":40,"tag":1336,"props":2470,"children":2472},{"class":1338,"line":2471},44,[2473],{"type":40,"tag":1336,"props":2474,"children":2475},{},[2476],{"type":45,"value":2477},"        'Grade \u003Cwhat this evaluator checks>. Reply with ONLY a JSON object, no prose, no markdown '\n",{"type":40,"tag":1336,"props":2479,"children":2481},{"class":1338,"line":2480},45,[2482],{"type":40,"tag":1336,"props":2483,"children":2484},{},[2485],{"type":45,"value":2486},"        'fences, explanation LAST and short: {\"score\": \u003C0-1 float>, \"passed\": \u003Cbool>, \"explanation\": \"\u003Cone sentence>\"}.\\n\\n'\n",{"type":40,"tag":1336,"props":2488,"children":2490},{"class":1338,"line":2489},46,[2491],{"type":40,"tag":1336,"props":2492,"children":2493},{},[2494],{"type":45,"value":2495},"        # For a grounding judge, prepend the retrieved context here:\n",{"type":40,"tag":1336,"props":2497,"children":2499},{"class":1338,"line":2498},47,[2500],{"type":40,"tag":1336,"props":2501,"children":2502},{},[2503],{"type":45,"value":2504},"        # f\"CONTEXT:\\n{retrieved_passages}\\n\\n\"\n",{"type":40,"tag":1336,"props":2506,"children":2508},{"class":1338,"line":2507},48,[2509],{"type":40,"tag":1336,"props":2510,"children":2511},{},[2512],{"type":45,"value":2513},"        f\"Input:\\n{case_input}\\n\\nOutput:\\n{output}\"\n",{"type":40,"tag":1336,"props":2515,"children":2517},{"class":1338,"line":2516},49,[2518],{"type":40,"tag":1336,"props":2519,"children":2520},{},[2521],{"type":45,"value":2522},"    )\n",{"type":40,"tag":1336,"props":2524,"children":2526},{"class":1338,"line":2525},50,[2527],{"type":40,"tag":1336,"props":2528,"children":2529},{},[2530],{"type":45,"value":2531},"    model = os.getenv(\"GRADER_MODEL\") or os.getenv(\"MODEL_NAME\")  # a LIVE model id; no default (dead ids 404)\n",{"type":40,"tag":1336,"props":2533,"children":2535},{"class":1338,"line":2534},51,[2536],{"type":40,"tag":1336,"props":2537,"children":2538},{},[2539],{"type":45,"value":2540},"    # SECOND HOLE YOU FILL — call the model using the SAME client the agent uses (do NOT add a new\n",{"type":40,"tag":1336,"props":2542,"children":2544},{"class":1338,"line":2543},52,[2545],{"type":40,"tag":1336,"props":2546,"children":2547},{},[2548],{"type":45,"value":2549},"    # provider). Import it from the agent module and make ONE call that returns the reply text into\n",{"type":40,"tag":1336,"props":2551,"children":2553},{"class":1338,"line":2552},53,[2554],{"type":40,"tag":1336,"props":2555,"children":2556},{},[2557],{"type":45,"value":2558},"    # `text`. Give it enough max_tokens (~600) that the JSON verdict is never truncated mid-object.\n",{"type":40,"tag":1336,"props":2560,"children":2562},{"class":1338,"line":2561},54,[2563],{"type":40,"tag":1336,"props":2564,"children":2565},{},[2566],{"type":45,"value":2567},"    #   Anthropic: from agent import client\n",{"type":40,"tag":1336,"props":2569,"children":2571},{"class":1338,"line":2570},55,[2572],{"type":40,"tag":1336,"props":2573,"children":2574},{},[2575],{"type":45,"value":2576},"    #     text = client.messages.create(model=model, max_tokens=600,\n",{"type":40,"tag":1336,"props":2578,"children":2580},{"class":1338,"line":2579},56,[2581],{"type":40,"tag":1336,"props":2582,"children":2583},{},[2584],{"type":45,"value":2585},"    #                                    messages=[{\"role\": \"user\", \"content\": prompt}]).content[0].text\n",{"type":40,"tag":1336,"props":2587,"children":2589},{"class":1338,"line":2588},57,[2590],{"type":40,"tag":1336,"props":2591,"children":2592},{},[2593],{"type":45,"value":2594},"    #   OpenAI:    from agent import client\n",{"type":40,"tag":1336,"props":2596,"children":2598},{"class":1338,"line":2597},58,[2599],{"type":40,"tag":1336,"props":2600,"children":2601},{},[2602],{"type":45,"value":2603},"    #     text = client.chat.completions.create(model=model, max_tokens=600,\n",{"type":40,"tag":1336,"props":2605,"children":2607},{"class":1338,"line":2606},59,[2608],{"type":40,"tag":1336,"props":2609,"children":2610},{},[2611],{"type":45,"value":2612},"    #                                            messages=[{\"role\": \"user\", \"content\": prompt}]).choices[0].message.content\n",{"type":40,"tag":1336,"props":2614,"children":2616},{"class":1338,"line":2615},60,[2617],{"type":40,"tag":1336,"props":2618,"children":2619},{},[2620],{"type":45,"value":2621},"    raise NotImplementedError(\"call the agent's provider client here and assign the reply to `text`\")\n",{"type":40,"tag":1336,"props":2623,"children":2625},{"class":1338,"line":2624},61,[2626],{"type":40,"tag":1336,"props":2627,"children":2628},{},[2629],{"type":45,"value":2630},"    d = _parse_judge_json(text)\n",{"type":40,"tag":1336,"props":2632,"children":2634},{"class":1338,"line":2633},62,[2635],{"type":40,"tag":1336,"props":2636,"children":2637},{},[2638],{"type":45,"value":2639},"    score = max(0.0, min(1.0, float(d.get(\"score\", 0.0))))\n",{"type":40,"tag":1336,"props":2641,"children":2643},{"class":1338,"line":2642},63,[2644],{"type":40,"tag":1336,"props":2645,"children":2646},{},[2647],{"type":45,"value":2648},"    return score, bool(d.get(\"passed\", score >= 0.6)), str(d.get(\"explanation\", \"\"))\n",{"type":40,"tag":1336,"props":2650,"children":2652},{"class":1338,"line":2651},64,[2653],{"type":40,"tag":1336,"props":2654,"children":2655},{"emptyLinePlaceholder":2136},[2656],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2658,"children":2660},{"class":1338,"line":2659},65,[2661],{"type":40,"tag":1336,"props":2662,"children":2663},{"emptyLinePlaceholder":2136},[2664],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2666,"children":2668},{"class":1338,"line":2667},66,[2669],{"type":40,"tag":1336,"props":2670,"children":2671},{},[2672],{"type":45,"value":2673},"def _parse_judge_json(text: str) -> dict:\n",{"type":40,"tag":1336,"props":2675,"children":2677},{"class":1338,"line":2676},67,[2678],{"type":40,"tag":1336,"props":2679,"children":2680},{},[2681],{"type":45,"value":2682},"    \"\"\"Robustly pull the JSON verdict out of a model reply (handles prose \u002F ```json fences).\"\"\"\n",{"type":40,"tag":1336,"props":2684,"children":2686},{"class":1338,"line":2685},68,[2687],{"type":40,"tag":1336,"props":2688,"children":2689},{},[2690],{"type":45,"value":2691},"    if not text:\n",{"type":40,"tag":1336,"props":2693,"children":2695},{"class":1338,"line":2694},69,[2696],{"type":40,"tag":1336,"props":2697,"children":2698},{},[2699],{"type":45,"value":2700},"        return {}\n",{"type":40,"tag":1336,"props":2702,"children":2704},{"class":1338,"line":2703},70,[2705],{"type":40,"tag":1336,"props":2706,"children":2707},{},[2708],{"type":45,"value":2709},"    t = text.strip()\n",{"type":40,"tag":1336,"props":2711,"children":2713},{"class":1338,"line":2712},71,[2714],{"type":40,"tag":1336,"props":2715,"children":2716},{},[2717],{"type":45,"value":2718},"    if \"```\" in t:  # strip a ```json ... ``` fence if present\n",{"type":40,"tag":1336,"props":2720,"children":2722},{"class":1338,"line":2721},72,[2723],{"type":40,"tag":1336,"props":2724,"children":2725},{},[2726],{"type":45,"value":2727},"        t = t.split(\"```\")[1].removeprefix(\"json\").strip() if t.count(\"```\") >= 2 else t\n",{"type":40,"tag":1336,"props":2729,"children":2731},{"class":1338,"line":2730},73,[2732],{"type":40,"tag":1336,"props":2733,"children":2734},{},[2735],{"type":45,"value":2736},"    try:\n",{"type":40,"tag":1336,"props":2738,"children":2740},{"class":1338,"line":2739},74,[2741],{"type":40,"tag":1336,"props":2742,"children":2743},{},[2744],{"type":45,"value":2745},"        return json.loads(t)\n",{"type":40,"tag":1336,"props":2747,"children":2749},{"class":1338,"line":2748},75,[2750],{"type":40,"tag":1336,"props":2751,"children":2752},{},[2753],{"type":45,"value":2754},"    except json.JSONDecodeError:\n",{"type":40,"tag":1336,"props":2756,"children":2758},{"class":1338,"line":2757},76,[2759],{"type":40,"tag":1336,"props":2760,"children":2761},{},[2762],{"type":45,"value":2763},"        pass\n",{"type":40,"tag":1336,"props":2765,"children":2767},{"class":1338,"line":2766},77,[2768],{"type":40,"tag":1336,"props":2769,"children":2770},{},[2771],{"type":45,"value":2772},"    # Fallback: scan for the first balanced {...} object rather than first-brace\u002Flast-brace.\n",{"type":40,"tag":1336,"props":2774,"children":2776},{"class":1338,"line":2775},78,[2777],{"type":40,"tag":1336,"props":2778,"children":2779},{},[2780],{"type":45,"value":2781},"    depth = 0; start = -1\n",{"type":40,"tag":1336,"props":2783,"children":2785},{"class":1338,"line":2784},79,[2786],{"type":40,"tag":1336,"props":2787,"children":2788},{},[2789],{"type":45,"value":2790},"    for i, ch in enumerate(t):\n",{"type":40,"tag":1336,"props":2792,"children":2794},{"class":1338,"line":2793},80,[2795],{"type":40,"tag":1336,"props":2796,"children":2797},{},[2798],{"type":45,"value":2799},"        if ch == \"{\":\n",{"type":40,"tag":1336,"props":2801,"children":2803},{"class":1338,"line":2802},81,[2804],{"type":40,"tag":1336,"props":2805,"children":2806},{},[2807],{"type":45,"value":2808},"            if depth == 0:\n",{"type":40,"tag":1336,"props":2810,"children":2812},{"class":1338,"line":2811},82,[2813],{"type":40,"tag":1336,"props":2814,"children":2815},{},[2816],{"type":45,"value":2817},"                start = i\n",{"type":40,"tag":1336,"props":2819,"children":2821},{"class":1338,"line":2820},83,[2822],{"type":40,"tag":1336,"props":2823,"children":2824},{},[2825],{"type":45,"value":2826},"            depth += 1\n",{"type":40,"tag":1336,"props":2828,"children":2830},{"class":1338,"line":2829},84,[2831],{"type":40,"tag":1336,"props":2832,"children":2833},{},[2834],{"type":45,"value":2835},"        elif ch == \"}\":\n",{"type":40,"tag":1336,"props":2837,"children":2839},{"class":1338,"line":2838},85,[2840],{"type":40,"tag":1336,"props":2841,"children":2842},{},[2843],{"type":45,"value":2844},"            depth -= 1\n",{"type":40,"tag":1336,"props":2846,"children":2848},{"class":1338,"line":2847},86,[2849],{"type":40,"tag":1336,"props":2850,"children":2851},{},[2852],{"type":45,"value":2853},"            if depth == 0 and start >= 0:\n",{"type":40,"tag":1336,"props":2855,"children":2857},{"class":1338,"line":2856},87,[2858],{"type":40,"tag":1336,"props":2859,"children":2860},{},[2861],{"type":45,"value":2862},"                try:\n",{"type":40,"tag":1336,"props":2864,"children":2866},{"class":1338,"line":2865},88,[2867],{"type":40,"tag":1336,"props":2868,"children":2869},{},[2870],{"type":45,"value":2871},"                    return json.loads(t[start:i + 1])\n",{"type":40,"tag":1336,"props":2873,"children":2875},{"class":1338,"line":2874},89,[2876],{"type":40,"tag":1336,"props":2877,"children":2878},{},[2879],{"type":45,"value":2880},"                except json.JSONDecodeError:\n",{"type":40,"tag":1336,"props":2882,"children":2884},{"class":1338,"line":2883},90,[2885],{"type":40,"tag":1336,"props":2886,"children":2887},{},[2888],{"type":45,"value":2889},"                    start = -1\n",{"type":40,"tag":1336,"props":2891,"children":2893},{"class":1338,"line":2892},91,[2894],{"type":40,"tag":1336,"props":2895,"children":2896},{},[2897],{"type":45,"value":2898},"    return {}  # unparseable → caller treats as score 0; investigate the raw reply\n",{"type":40,"tag":1336,"props":2900,"children":2902},{"class":1338,"line":2901},92,[2903],{"type":40,"tag":1336,"props":2904,"children":2905},{"emptyLinePlaceholder":2136},[2906],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2908,"children":2910},{"class":1338,"line":2909},93,[2911],{"type":40,"tag":1336,"props":2912,"children":2913},{"emptyLinePlaceholder":2136},[2914],{"type":45,"value":2139},{"type":40,"tag":1336,"props":2916,"children":2918},{"class":1338,"line":2917},94,[2919],{"type":40,"tag":1336,"props":2920,"children":2921},{},[2922],{"type":45,"value":2923},"def main() -> None:\n",{"type":40,"tag":1336,"props":2925,"children":2927},{"class":1338,"line":2926},95,[2928],{"type":40,"tag":1336,"props":2929,"children":2930},{},[2931],{"type":45,"value":2932},"    suite = agento11y.TestSuite.from_yaml(str(SUITE))\n",{"type":40,"tag":1336,"props":2934,"children":2936},{"class":1338,"line":2935},96,[2937],{"type":40,"tag":1336,"props":2938,"children":2939},{},[2940],{"type":45,"value":2941},"    verifier = agento11y.Evaluator(evaluator_id=\"\u003Cevaluator>\", version=\"draft-0\", kind=\"llm_judge\")\n",{"type":40,"tag":1336,"props":2943,"children":2945},{"class":1338,"line":2944},97,[2946],{"type":40,"tag":1336,"props":2947,"children":2948},{},[2949],{"type":45,"value":2950},"    candidate = {\n",{"type":40,"tag":1336,"props":2952,"children":2954},{"class":1338,"line":2953},98,[2955],{"type":40,"tag":1336,"props":2956,"children":2957},{},[2958],{"type":45,"value":2959},"        \"agent_name\": \"\u003Cagent>\",\n",{"type":40,"tag":1336,"props":2961,"children":2963},{"class":1338,"line":2962},99,[2964],{"type":40,"tag":1336,"props":2965,"children":2966},{},[2967],{"type":45,"value":2968},"        # Always send a declared agent_version. Without it Agent Observability auto-derives a version from\n",{"type":40,"tag":1336,"props":2970,"children":2972},{"class":1338,"line":2971},100,[2973],{"type":40,"tag":1336,"props":2974,"children":2975},{},[2976],{"type":45,"value":2977},"        # the system-prompt hash, so you can't reliably attribute scores to a version or\n",{"type":40,"tag":1336,"props":2979,"children":2981},{"class":1338,"line":2980},101,[2982],{"type":40,"tag":1336,"props":2983,"children":2984},{},[2985],{"type":45,"value":2986},"        # compare versions. Replace \"v1\" with your real version (git tag, prompt version,\n",{"type":40,"tag":1336,"props":2988,"children":2990},{"class":1338,"line":2989},102,[2991],{"type":40,"tag":1336,"props":2992,"children":2993},{},[2994],{"type":45,"value":2995},"        # semver...) — the \"v1\" fallback is a placeholder, not a version worth comparing.\n",{"type":40,"tag":1336,"props":2997,"children":2999},{"class":1338,"line":2998},103,[3000],{"type":40,"tag":1336,"props":3001,"children":3002},{},[3003],{"type":45,"value":3004},"        \"agent_version\": os.getenv(\"AGENT_VERSION\", \"v1\"),\n",{"type":40,"tag":1336,"props":3006,"children":3008},{"class":1338,"line":3007},104,[3009],{"type":40,"tag":1336,"props":3010,"children":3011},{},[3012],{"type":45,"value":3013},"        \"git_sha\": os.getenv(\"GIT_SHA\", \"\"),\n",{"type":40,"tag":1336,"props":3015,"children":3017},{"class":1338,"line":3016},105,[3018],{"type":40,"tag":1336,"props":3019,"children":3020},{},[3021],{"type":45,"value":3022},"        \"model_name\": os.getenv(\"MODEL_NAME\", \"\"),\n",{"type":40,"tag":1336,"props":3024,"children":3026},{"class":1338,"line":3025},106,[3027],{"type":40,"tag":1336,"props":3028,"children":3029},{},[3030],{"type":45,"value":3031},"    }\n",{"type":40,"tag":1336,"props":3033,"children":3035},{"class":1338,"line":3034},107,[3036],{"type":40,"tag":1336,"props":3037,"children":3038},{},[3039],{"type":45,"value":3040},"    with agento11y.experiment(name=\"\u003Cagent> starter\", experiment_id=f\"\u003Cagent>-starter-{int(time.time())}\",\n",{"type":40,"tag":1336,"props":3042,"children":3044},{"class":1338,"line":3043},108,[3045],{"type":40,"tag":1336,"props":3046,"children":3047},{},[3048],{"type":45,"value":3049},"                          suite=suite, candidate=candidate, tags=[\"starter\"],\n",{"type":40,"tag":1336,"props":3051,"children":3053},{"class":1338,"line":3052},109,[3054],{"type":40,"tag":1336,"props":3055,"children":3056},{},[3057],{"type":45,"value":3058},"                          actor=os.getenv(\"AGENTO11Y_INGEST_ACTOR\", \"ingest:sdk\u002Fpython\")) as exp:\n",{"type":40,"tag":1336,"props":3060,"children":3062},{"class":1338,"line":3061},110,[3063],{"type":40,"tag":1336,"props":3064,"children":3065},{},[3066],{"type":45,"value":3067},"        for case in suite.test_cases:\n",{"type":40,"tag":1336,"props":3069,"children":3071},{"class":1338,"line":3070},111,[3072],{"type":40,"tag":1336,"props":3073,"children":3074},{},[3075],{"type":45,"value":3076},"            with exp.trial(case) as trial:\n",{"type":40,"tag":1336,"props":3078,"children":3080},{"class":1338,"line":3079},112,[3081],{"type":40,"tag":1336,"props":3082,"children":3083},{},[3084],{"type":45,"value":3085},"                out = run_agent(case)\n",{"type":40,"tag":1336,"props":3087,"children":3089},{"class":1338,"line":3088},113,[3090],{"type":40,"tag":1336,"props":3091,"children":3092},{},[3093],{"type":45,"value":3094},"                trial.record_io(input=json.dumps(case.input), output=out,\n",{"type":40,"tag":1336,"props":3096,"children":3098},{"class":1338,"line":3097},114,[3099],{"type":40,"tag":1336,"props":3100,"children":3101},{},[3102],{"type":45,"value":3103},"                                model_provider=\"\u003Cprovider>\", model_name=os.getenv(\"MODEL_NAME\", \"\"))\n",{"type":40,"tag":1336,"props":3105,"children":3107},{"class":1338,"line":3106},115,[3108],{"type":40,"tag":1336,"props":3109,"children":3110},{},[3111],{"type":45,"value":3112},"                score, passed, why = judge_\u003Cevaluator>(case.input, out)\n",{"type":40,"tag":1336,"props":3114,"children":3116},{"class":1338,"line":3115},116,[3117],{"type":40,"tag":1336,"props":3118,"children":3119},{},[3120],{"type":45,"value":3121},"                trial.final_score(score, passed=passed, explanation=why, evaluator=verifier)\n",{"type":40,"tag":1336,"props":3123,"children":3125},{"class":1338,"line":3124},117,[3126],{"type":40,"tag":1336,"props":3127,"children":3128},{},[3129],{"type":45,"value":3130},"                print(f\"  {case.test_case_id}: score={score:.2f} passed={passed}\")\n",{"type":40,"tag":1336,"props":3132,"children":3134},{"class":1338,"line":3133},118,[3135],{"type":40,"tag":1336,"props":3136,"children":3137},{},[3138],{"type":45,"value":3139},"    print(f\"\\nExperiment: {exp.url}\")\n",{"type":40,"tag":1336,"props":3141,"children":3143},{"class":1338,"line":3142},119,[3144],{"type":40,"tag":1336,"props":3145,"children":3146},{"emptyLinePlaceholder":2136},[3147],{"type":45,"value":2139},{"type":40,"tag":1336,"props":3149,"children":3151},{"class":1338,"line":3150},120,[3152],{"type":40,"tag":1336,"props":3153,"children":3154},{"emptyLinePlaceholder":2136},[3155],{"type":45,"value":2139},{"type":40,"tag":1336,"props":3157,"children":3159},{"class":1338,"line":3158},121,[3160],{"type":40,"tag":1336,"props":3161,"children":3162},{},[3163],{"type":45,"value":3164},"if __name__ == \"__main__\":\n",{"type":40,"tag":1336,"props":3166,"children":3168},{"class":1338,"line":3167},122,[3169],{"type":40,"tag":1336,"props":3170,"children":3171},{},[3172],{"type":45,"value":3173},"    main()\n",{"type":40,"tag":48,"props":3175,"children":3176},{},[3177,3179,3190,3192,3198],{"type":45,"value":3178},"Use a ",{"type":40,"tag":54,"props":3180,"children":3181},{},[3182,3184],{"type":45,"value":3183},"fresh ",{"type":40,"tag":92,"props":3185,"children":3187},{"className":3186},[],[3188],{"type":45,"value":3189},"experiment_id",{"type":45,"value":3191}," per run (a timestamp works) — reusing an id created by a\ndifferent auth actor fails with ",{"type":40,"tag":92,"props":3193,"children":3195},{"className":3194},[],[3196],{"type":45,"value":3197},"401: experiment is owned by another actor",{"type":45,"value":99},{"type":40,"tag":200,"props":3200,"children":3202},{"id":3201},"step-5-summarize-and-hand-off",[3203],{"type":45,"value":3204},"Step 5 — Summarize and hand off",{"type":40,"tag":48,"props":3206,"children":3207},{},[3208],{"type":45,"value":3209},"Output, in this order:",{"type":40,"tag":106,"props":3211,"children":3212},{},[3213,3225,3243,3276],{"type":40,"tag":110,"props":3214,"children":3215},{},[3216,3218,3223],{"type":45,"value":3217},"The picked evaluators, each with its kind and its ",{"type":40,"tag":92,"props":3219,"children":3221},{"className":3220},[],[3222],{"type":45,"value":755},{"type":45,"value":3224}," (with file:line).\nAdd a one-line \"once you have traffic, these criteria can also run online (Agent Observability Rules or\nguard hooks) — separate surfaces, not set up here.\"",{"type":40,"tag":110,"props":3226,"children":3227},{},[3228,3230,3235,3236,3241],{"type":45,"value":3229},"The paths to the two written files (",{"type":40,"tag":92,"props":3231,"children":3233},{"className":3232},[],[3234],{"type":45,"value":1106},{"type":45,"value":1283},{"type":40,"tag":92,"props":3237,"children":3239},{"className":3238},[],[3240],{"type":45,"value":1844},{"type":45,"value":3242},"), and a one-line reminder to review the edge\u002Fadversarial cases\nand add real ones.",{"type":40,"tag":110,"props":3244,"children":3245},{},[3246,3248,3253,3255,3260,3262,3267,3269,3274],{"type":45,"value":3247},"The three things they still do to run it: fill ",{"type":40,"tag":92,"props":3249,"children":3251},{"className":3250},[],[3252],{"type":45,"value":2006},{"type":45,"value":3254}," (first hole); fill the judge's\nmodel call using the agent's own provider client, then tune it and add the other recommended\nevaluators the same way (second hole); and set credentials (",{"type":40,"tag":92,"props":3256,"children":3258},{"className":3257},[],[3259],{"type":45,"value":323},{"type":45,"value":3261}," +\n",{"type":40,"tag":92,"props":3263,"children":3265},{"className":3264},[],[3266],{"type":45,"value":331},{"type":45,"value":3268}," + the provider key).\nState the boundary explicitly: this skill only\nbootstraps the first run; for anything past that — binding an already-instrumented agent's\nreal generations, auditable LLM-judge grading, cross-process verifiers, repeated-sampling\nmetrics — the ",{"type":40,"tag":92,"props":3270,"children":3272},{"className":3271},[],[3273],{"type":45,"value":173},{"type":45,"value":3275}," skill is the reference.",{"type":40,"tag":110,"props":3277,"children":3278},{},[3279],{"type":45,"value":3280},"One line confirming nothing was created in Agent Observability — recommendations and draft files only.",{"type":40,"tag":200,"props":3282,"children":3284},{"id":3283},"step-6-offer-to-run-it-optional-only-with-permission",[3285],{"type":45,"value":3286},"Step 6 — Offer to run it (optional, only with permission)",{"type":40,"tag":48,"props":3288,"children":3289},{},[3290,3292,3296,3298,3303,3305,3310,3311,3316],{"type":45,"value":3291},"Only offer this for an ",{"type":40,"tag":54,"props":3293,"children":3294},{},[3295],{"type":45,"value":1836},{"type":45,"value":3297}," agent (clean function seam) ",{"type":40,"tag":54,"props":3299,"children":3300},{},[3301],{"type":45,"value":3302},"in Python or Go",{"type":45,"value":3304}," (the languages\nwith an experiments SDK). For ",{"type":40,"tag":92,"props":3306,"children":3308},{"className":3307},[],[3309],{"type":45,"value":673},{"type":45,"value":675},{"type":40,"tag":92,"props":3312,"children":3314},{"className":3313},[],[3315],{"type":45,"value":681},{"type":45,"value":3317}," agents, or agents in a language with no\nexperiments SDK (TS\u002FJava\u002F.NET), don't offer to run — point to the existing harness\u002Finfra (or\nthe subprocess-bridge option) and stop; a real run there is out of scope for this skill.",{"type":40,"tag":48,"props":3319,"children":3320},{},[3321],{"type":45,"value":3322},"After the summary, offer to run the starter experiment for them — do not run automatically.\nAsk: \"Want me to try running this now?\" Only proceed if they say yes.",{"type":40,"tag":48,"props":3324,"children":3325},{},[3326],{"type":45,"value":3327},"If they accept:",{"type":40,"tag":106,"props":3329,"children":3330},{},[3331,3343,3581,3586,3599],{"type":40,"tag":110,"props":3332,"children":3333},{},[3334,3336,3341],{"type":45,"value":3335},"Help fill ",{"type":40,"tag":92,"props":3337,"children":3339},{"className":3338},[],[3340],{"type":45,"value":2006},{"type":45,"value":3342}," — wire it to the real entrypoint from Step 1, so the runner\nactually calls their agent.",{"type":40,"tag":110,"props":3344,"children":3345},{},[3346,3348,3353,3355,3360,3361,3367,3368],{"type":45,"value":3347},"Preflight the environment and stop with a clear ask if anything is missing. ",{"type":40,"tag":54,"props":3349,"children":3350},{},[3351],{"type":45,"value":3352},"When you ask, tell\nthe developer exactly where each value is",{"type":45,"value":3354}," — for a Cloud stack they all live on the plugin\n",{"type":40,"tag":54,"props":3356,"children":3357},{},[3358],{"type":45,"value":3359},"Connection page",{"type":45,"value":521},{"type":40,"tag":92,"props":3362,"children":3364},{"className":3363},[],[3365],{"type":45,"value":3366},"https:\u002F\u002F\u003Cyour-stack>.grafana.net\u002Fplugins\u002Fgrafana-agento11y-app",{"type":45,"value":1432},{"type":40,"tag":212,"props":3369,"children":3370},{},[3371,3388,3405,3512,3538,3548,3568],{"type":40,"tag":110,"props":3372,"children":3373},{},[3374,3379,3381,3386],{"type":40,"tag":92,"props":3375,"children":3377},{"className":3376},[],[3378],{"type":45,"value":323},{"type":45,"value":3380}," = the ",{"type":40,"tag":54,"props":3382,"children":3383},{},[3384],{"type":45,"value":3385},"API URL",{"type":45,"value":3387}," on that page. If unset, ask — never invent one.",{"type":40,"tag":110,"props":3389,"children":3390},{},[3391,3397,3398,3403],{"type":40,"tag":92,"props":3392,"children":3394},{"className":3393},[],[3395],{"type":45,"value":3396},"AGENTO11Y_AUTH_TENANT_ID",{"type":45,"value":3380},{"type":40,"tag":54,"props":3399,"children":3400},{},[3401],{"type":45,"value":3402},"Instance ID",{"type":45,"value":3404}," on that page.",{"type":40,"tag":110,"props":3406,"children":3407},{},[3408,3413,3415,3420,3422,3452,3454,3460,3462,3466,3468,3473,3475,3481,3482,3488,3489,3495,3497,3503,3505,3510],{"type":40,"tag":92,"props":3409,"children":3411},{"className":3410},[],[3412],{"type":45,"value":331},{"type":45,"value":3414}," — always required (the SDK raises before any request if it is empty).\nThe developer creates it via ",{"type":40,"tag":54,"props":3416,"children":3417},{},[3418],{"type":45,"value":3419},"\"Create a token in Cloud Access Policies\"",{"type":45,"value":3421}," on the Connection page. Tell them\nthe exact scopes: ",{"type":40,"tag":54,"props":3423,"children":3424},{},[3425,3431,3432,3438,3439,3445,3446],{"type":40,"tag":92,"props":3426,"children":3428},{"className":3427},[],[3429],{"type":45,"value":3430},"sigil:write",{"type":45,"value":521},{"type":40,"tag":92,"props":3433,"children":3435},{"className":3434},[],[3436],{"type":45,"value":3437},"metrics:write",{"type":45,"value":521},{"type":40,"tag":92,"props":3440,"children":3442},{"className":3441},[],[3443],{"type":45,"value":3444},"traces:write",{"type":45,"value":521},{"type":40,"tag":92,"props":3447,"children":3449},{"className":3448},[],[3450],{"type":45,"value":3451},"logs:write",{"type":45,"value":3453},". Heads-up on\nthe UI: ",{"type":40,"tag":92,"props":3455,"children":3457},{"className":3456},[],[3458],{"type":45,"value":3459},"sigil",{"type":45,"value":3461}," is ",{"type":40,"tag":54,"props":3463,"children":3464},{},[3465],{"type":45,"value":1784},{"type":45,"value":3467}," in the default resource list — they must add it via ",{"type":40,"tag":54,"props":3469,"children":3470},{},[3471],{"type":45,"value":3472},"\"Add scope\"",{"type":45,"value":3474},"\n(then tick Write); ",{"type":40,"tag":92,"props":3476,"children":3478},{"className":3477},[],[3479],{"type":45,"value":3480},"metrics",{"type":45,"value":675},{"type":40,"tag":92,"props":3483,"children":3485},{"className":3484},[],[3486],{"type":45,"value":3487},"traces",{"type":45,"value":675},{"type":40,"tag":92,"props":3490,"children":3492},{"className":3491},[],[3493],{"type":45,"value":3494},"logs",{"type":45,"value":3496}," are already listed (tick Write). The scope is still\n",{"type":40,"tag":92,"props":3498,"children":3500},{"className":3499},[],[3501],{"type":45,"value":3502},"sigil:*",{"type":45,"value":3504}," (the Cloud resource keeps the old name). ",{"type":40,"tag":54,"props":3506,"children":3507},{},[3508],{"type":45,"value":3509},"Never mint or fabricate a token",{"type":45,"value":3511}," — the\ndeveloper creates and supplies it; you only read it from the environment or ask.",{"type":40,"tag":110,"props":3513,"children":3514},{},[3515,3516,3521,3523,3528,3530,3536],{"type":45,"value":365},{"type":40,"tag":54,"props":3517,"children":3518},{},[3519],{"type":45,"value":3520},"provider API key from Step 1",{"type":45,"value":3522}," (e.g. ",{"type":40,"tag":92,"props":3524,"children":3526},{"className":3525},[],[3527],{"type":45,"value":603},{"type":45,"value":3529}," or ",{"type":40,"tag":92,"props":3531,"children":3533},{"className":3532},[],[3534],{"type":45,"value":3535},"ANTHROPIC_API_KEY",{"type":45,"value":3537},") — name the\nexact env var, since both the agent and the judge need it. Ask for it; never mint it.",{"type":40,"tag":110,"props":3539,"children":3540},{},[3541,3546],{"type":40,"tag":92,"props":3542,"children":3544},{"className":3543},[],[3545],{"type":45,"value":535},{"type":45,"value":3547}," is a live model (dead model ids fail with a 404 not_found).",{"type":40,"tag":110,"props":3549,"children":3550},{},[3551,3553,3559,3561,3567],{"type":45,"value":3552},"A stable ",{"type":40,"tag":92,"props":3554,"children":3556},{"className":3555},[],[3557],{"type":45,"value":3558},"AGENTO11Y_INGEST_ACTOR",{"type":45,"value":3560}," so run and trials share one actor (else ",{"type":40,"tag":92,"props":3562,"children":3564},{"className":3563},[],[3565],{"type":45,"value":3566},"401: owned by another actor",{"type":45,"value":175},{"type":40,"tag":110,"props":3569,"children":3570},{},[3571,3573,3579],{"type":45,"value":3572},"A declared ",{"type":40,"tag":92,"props":3574,"children":3576},{"className":3575},[],[3577],{"type":45,"value":3578},"AGENT_VERSION",{"type":45,"value":3580}," (in the candidate). Without it Agent Observability auto-derives a version\nfrom the system-prompt hash, and the developer can't attribute scores to a version or\ncompare versions — which is the whole point of the agent's Quality view. Confirm a real\nvalue (git tag \u002F prompt version \u002F semver), don't leave it defaulted.",{"type":40,"tag":110,"props":3582,"children":3583},{},[3584],{"type":45,"value":3585},"Run against the endpoint the developer configured — never a target they didn't specify.\nStart with 1–2 cases as a smoke run before the full suite.",{"type":40,"tag":110,"props":3587,"children":3588},{},[3589,3591,3597],{"type":45,"value":3590},"Show the per-case scores and the ",{"type":40,"tag":92,"props":3592,"children":3594},{"className":3593},[],[3595],{"type":45,"value":3596},"exp.url",{"type":45,"value":3598},", and note this published one experiment's\nscores (no tenant evaluators\u002Frules\u002Fguards were created).",{"type":40,"tag":110,"props":3600,"children":3601},{},[3602,3607,3609,3613,3615,3620],{"type":40,"tag":54,"props":3603,"children":3604},{},[3605],{"type":45,"value":3606},"Don't oversell a clean sweep.",{"type":45,"value":3608}," If every case passes on the first run, say so honestly: that\nusually means the suite isn't hard enough to discriminate yet, not that the agent is flawless.\nNudge toward (a) wiring the recommended ",{"type":40,"tag":54,"props":3610,"children":3611},{},[3612],{"type":45,"value":742},{"type":45,"value":3614}," evaluators (they catch contract\nviolations an ",{"type":40,"tag":92,"props":3616,"children":3618},{"className":3617},[],[3619],{"type":45,"value":371},{"type":45,"value":3621}," forgives — e.g. an exact-string check vs a paraphrase), and (b) adding\na case that actually stresses the agent's likely real failure mode.",{"type":40,"tag":48,"props":3623,"children":3624},{},[3625],{"type":45,"value":3626},"If they decline, stop after the summary.",{"type":40,"tag":3628,"props":3629,"children":3630},"style",{},[3631],{"type":45,"value":3632},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3634,"total":2298},[3635,3649,3665,3677,3684,3701,3716],{"slug":262,"name":262,"fn":3636,"description":3637,"org":3638,"tags":3639,"stars":23,"repoUrl":24,"updatedAt":3648},"manage Grafana Agent Observability resources","Inspects and manages Grafana Agent Observability resources via gcx: conversations, generations, evaluators, rules, scores, and templates. Use when the user wants to list or search conversations, inspect generations, manage evaluators (upsert, test, delete), set up evaluation rules, check scores, or browse evaluator templates. Trigger on phrases like \"list conversations\", \"search generations\", \"what did the agent do\", \"debug LLM conversation\", \"create evaluator\", \"set up evaluation rule\", \"test evaluator\", \"check scores\", \"evaluate generation quality\", or \"set up online evaluation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3640,3641,3642,3645],{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":3643,"slug":3644,"type":15},"Monitoring","monitoring",{"name":3646,"slug":3647,"type":15},"Observability","observability","2026-07-25T05:30:40.29622",{"slug":3650,"name":3650,"fn":3651,"description":3652,"org":3653,"tags":3654,"stars":23,"repoUrl":24,"updatedAt":3664},"agento11y-instrument","instrument LLM apps for agent observability","Sets up and instruments a developer's own LLM app or agent to send generations and agentic workflow to Grafana Agent Observability (the Agent Observability SDKs) — greenfield setup, fixing broken instrumentation, or filling gaps in existing instrumentation. Uses gcx for the parts a static prompt can't do: `gcx login` \u002F `gcx cloud stacks` to find the stack, and `gcx agento11y agents|conversations|generations` to VERIFY that data actually lands — so it iterates (instrument → run → verify → fix) until generations arrive, not blindly. Reads the app's code, detects language\u002Fframework, classifies instrumentation state (none \u002F partial \u002F broken), then runs a fixed gap checklist whose #1 item is the silent failure no other prompt catches: the SDK emits OTel spans\u002Fmetrics but never creates a TracerProvider\u002FMeterProvider, so without them all metrics go to a no-op and are lost. Also checks agent_version (required for per-version Performance charts), set_result completeness, SYNC vs STREAM, parent_generation_ids DAG links, and workflow-step coverage. Recommends changes citing file:line and, only with explicit confirmation, applies minimal diffs that don't change app behavior. Pulls SDK reference from agento11y's llms.txt rather than restating it, and hands off to `agento11y-test-starter` once data flows. It does NOT write test suites or set up tenant evaluations, rules, or guards — offline test suites are `agento11y-test-starter`, tenant eval rules + guards are `agento11y-prod-setup`; does NOT install coding-agent telemetry plugins (that is llms.txt \"Path A\"); does NOT mint or store credentials or invent endpoints. Trigger on phrases like \"instrument my app\", \"send my agent's traces to Grafana\", \"set up AI observability for my app\", \"my generations aren't showing up\", \"why is Performance empty\", \"add Agent Observability to my code\", \"fix my instrumentation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3655,3656,3657,3660,3663],{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":3658,"slug":3659,"type":15},"Instrumentation","instrumentation",{"name":3661,"slug":3662,"type":15},"LLM","llm",{"name":3646,"slug":3647,"type":15},"2026-07-31T05:53:52.580237",{"slug":97,"name":97,"fn":3666,"description":3667,"org":3668,"tags":3669,"stars":23,"repoUrl":24,"updatedAt":3676},"setup production evaluation for AI agents","Sets up production evaluation and guardrails for a DEPLOYED AI agent in Grafana Agent Observability, grounded in the agent's own code and its real ingested traffic. The judgment layer on top of the `agento11y` skill: it reads the agent's source (system prompt, tools, entrypoint) AND samples its live traffic via gcx, checks what evaluators\u002Frules\u002Fguards already exist, then recommends only what's missing — online eval rules (score live conversations for regressions) and guards (warn-first request-path policies that redact \u002F tool-filter and may later be promoted to deny). It drafts reviewable YAML and, only with explicit confirmation, applies via `gcx agento11y`. New guards are drafted in warn mode (safe on live traffic — warn records but never blocks). It DOES create stack-level objects — that is the point — but every write is confirmed. It never rewrites or redeploys the agent. Trigger on phrases like \"set up production evaluation\", \"my agent is in prod what should I evaluate\", \"catch quality regressions\", \"add guardrails to my agent\", \"redact PII from my agent\", \"block dangerous tools\", \"set up online evals and guards\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3670,3671,3674,3675],{"name":17,"slug":18,"type":15},{"name":3672,"slug":3673,"type":15},"Evals","evals",{"name":9,"slug":8,"type":15},{"name":3646,"slug":3647,"type":15},"2026-07-31T05:53:53.576347",{"slug":4,"name":4,"fn":5,"description":6,"org":3678,"tags":3679,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3680,3681,3682,3683],{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"name":21,"slug":22,"type":15},{"slug":3685,"name":3685,"fn":3686,"description":3687,"org":3688,"tags":3689,"stars":23,"repoUrl":24,"updatedAt":3700},"create-dashboard","create Grafana dashboards with gcx","Designs and creates Grafana dashboards with gcx, using `gcx dashboards snapshot` as a visual feedback loop. Use when the user wants to create a new Grafana dashboard, add panels, variables, or annotations to an existing dashboard, design dashboard panels, variables, queries, or layout, or make a material visual redesign. Triggers on \"create dashboard\", \"new dashboard\", \"build dashboard\", \"dashboard for \u003Cservice>\", \"add panels\", \"add variable\", \"add annotation\", \"improve this dashboard\", or \"iterate on a dashboard\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3690,3693,3696,3699],{"name":3691,"slug":3692,"type":15},"Dashboards","dashboards",{"name":3694,"slug":3695,"type":15},"Data Visualization","data-visualization",{"name":3697,"slug":3698,"type":15},"Design","design",{"name":9,"slug":8,"type":15},"2026-07-25T05:30:46.289717",{"slug":3702,"name":3702,"fn":3703,"description":3704,"org":3705,"tags":3706,"stars":23,"repoUrl":24,"updatedAt":3715},"debug-with-grafana","investigate application issues with Grafana","Structured workflow for investigating application problems with Grafana observability data (metrics, logs, traces) via gcx. Covers live firefighting AND retrospective incident analysis: incident triage, root-cause analysis, blast-radius checks (did an incident spill into other services), verifying whether a deployment or rollout triggered an incident, finding which service, endpoint, or path owns the most errors or slow requests, checking whether retries or queue backlogs piled up, and quantifying error or latency shares over a time window. Trigger on: \"my API is returning 500 errors\", \"latency is spiking\", \"investigate why requests are failing\", \"triage the incident\", \"blast radius\", \"root cause\", \"did the rollout cause it\", \"which endpoint owns the most 5xx\", \"did retries pile up\", or any request to analyse an earlier incident window using telemetry. For authoring dashboards use create-dashboard; for dashboard inventory use manage-dashboards.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3707,3710,3711,3714],{"name":3708,"slug":3709,"type":15},"Debugging","debugging",{"name":9,"slug":8,"type":15},{"name":3712,"slug":3713,"type":15},"Incident Response","incident-response",{"name":3646,"slug":3647,"type":15},"2026-07-18T05:11:10.445428",{"slug":3717,"name":3717,"fn":3718,"description":3719,"org":3720,"tags":3721,"stars":23,"repoUrl":24,"updatedAt":3728},"diagnose-entity-graph","diagnose Grafana Entity Graph issues","Diagnose Entity Graph problems: missing entities, missing edges, disconnected clusters, or filtering issues. Use when the user reports that Entity Graph doesn't look right, services are missing, edges aren't appearing, or environments can't be filtered. Triggers for: \"entity graph is empty\", \"services missing from entity graph\", \"no edges in entity graph\", \"disconnected services\", \"can't filter entity graph\", \"entity graph not working\", \"diagnose entity graph\", \"debug knowledge graph\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3722,3723,3724,3727],{"name":3708,"slug":3709,"type":15},{"name":9,"slug":8,"type":15},{"name":3725,"slug":3726,"type":15},"Graph Analysis","graph-analysis",{"name":3646,"slug":3647,"type":15},"2026-07-25T05:30:39.380934",{"items":3730,"total":2793},[3731,3748,3767,3787,3794,3802,3809,3816,3823,3830,3837,3852],{"slug":3732,"name":3732,"fn":3733,"description":3734,"org":3735,"tags":3736,"stars":3745,"repoUrl":3746,"updatedAt":3747},"faro-setup-web","instrument web apps with Grafana Faro","Instruments a web app with Grafana Faro Web SDK for frontend observability. Use when setting up error tracking, Web Vitals, session monitoring, or distributed tracing in a browser app.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3737,3740,3743,3744],{"name":3738,"slug":3739,"type":15},"Distributed Tracing","distributed-tracing",{"name":3741,"slug":3742,"type":15},"Frontend","frontend",{"name":3643,"slug":3644,"type":15},{"name":3646,"slug":3647,"type":15},1103,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Ffaro-web-sdk","2026-07-12T07:43:24.63314",{"slug":3749,"name":3749,"fn":3750,"description":3751,"org":3752,"tags":3753,"stars":3764,"repoUrl":3765,"updatedAt":3766},"configuring-yesoreyeram-infinity-datasource","configure Grafana Infinity data source","Configure the Infinity data source — base URL and allowed hosts, the authentication methods (basic, bearer token, API key, digest, OAuth passthrough, OAuth 2.0 client credentials\u002FJWT, Azure, Azure Blob, AWS), TLS, custom HTTP headers, network and security settings, the custom health check, and provisioning with a config file. Use when a user asks how to set up, configure, or change settings for the Infinity data source; how to authenticate to an API; how to allow hosts; how to provision it as YAML; or how to troubleshoot connection, authentication, or health-check issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3754,3757,3760,3763],{"name":3755,"slug":3756,"type":15},"API Development","api-development",{"name":3758,"slug":3759,"type":15},"Authentication","authentication",{"name":3761,"slug":3762,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},1056,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgrafana-infinity-datasource","2026-07-12T07:43:25.939136",{"slug":3768,"name":3768,"fn":3769,"description":3770,"org":3771,"tags":3772,"stars":3764,"repoUrl":3765,"updatedAt":3786},"querying-yesoreyeram-infinity-datasource","query data with Infinity datasource","Build queries with the Infinity data source — the query types (JSON, CSV, TSV, XML, GraphQL, HTML, UQL, GROQ, Google Sheets, Series, Transformations), the parsers (Frontend\u002Fsimple, Backend JSONata, JQ, UQL, GROQ) and which support alerting, the sources (URL, Inline, Reference, Azure Blob, Random walk), output formats (table, timeseries, logs, trace, node graph, dataframe), root selector and columns, computed columns\u002Ffilters\u002Fsummarize, pagination, and template variables. Use when a user asks how to query Infinity; how to fetch JSON\u002FCSV\u002FXML\u002FGraphQL\u002FHTML from a URL or inline; how to select rows and columns; how to transform data with UQL\u002FGROQ\u002FJSONata\u002FJQ; how to make a query work with alerting; how to paginate; or how to use variables in a query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3773,3776,3779,3780,3783],{"name":3774,"slug":3775,"type":15},"CSV","csv",{"name":3777,"slug":3778,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":3781,"slug":3782,"type":15},"GraphQL","graphql",{"name":3784,"slug":3785,"type":15},"JSON","json","2026-07-15T05:34:05.773947",{"slug":262,"name":262,"fn":3636,"description":3637,"org":3788,"tags":3789,"stars":23,"repoUrl":24,"updatedAt":3648},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3790,3791,3792,3793],{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":3643,"slug":3644,"type":15},{"name":3646,"slug":3647,"type":15},{"slug":3650,"name":3650,"fn":3651,"description":3652,"org":3795,"tags":3796,"stars":23,"repoUrl":24,"updatedAt":3664},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3797,3798,3799,3800,3801],{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":3658,"slug":3659,"type":15},{"name":3661,"slug":3662,"type":15},{"name":3646,"slug":3647,"type":15},{"slug":97,"name":97,"fn":3666,"description":3667,"org":3803,"tags":3804,"stars":23,"repoUrl":24,"updatedAt":3676},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3805,3806,3807,3808],{"name":17,"slug":18,"type":15},{"name":3672,"slug":3673,"type":15},{"name":9,"slug":8,"type":15},{"name":3646,"slug":3647,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":3810,"tags":3811,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3812,3813,3814,3815],{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"name":21,"slug":22,"type":15},{"slug":3685,"name":3685,"fn":3686,"description":3687,"org":3817,"tags":3818,"stars":23,"repoUrl":24,"updatedAt":3700},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3819,3820,3821,3822],{"name":3691,"slug":3692,"type":15},{"name":3694,"slug":3695,"type":15},{"name":3697,"slug":3698,"type":15},{"name":9,"slug":8,"type":15},{"slug":3702,"name":3702,"fn":3703,"description":3704,"org":3824,"tags":3825,"stars":23,"repoUrl":24,"updatedAt":3715},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3826,3827,3828,3829],{"name":3708,"slug":3709,"type":15},{"name":9,"slug":8,"type":15},{"name":3712,"slug":3713,"type":15},{"name":3646,"slug":3647,"type":15},{"slug":3717,"name":3717,"fn":3718,"description":3719,"org":3831,"tags":3832,"stars":23,"repoUrl":24,"updatedAt":3728},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3833,3834,3835,3836],{"name":3708,"slug":3709,"type":15},{"name":9,"slug":8,"type":15},{"name":3725,"slug":3726,"type":15},{"name":3646,"slug":3647,"type":15},{"slug":3838,"name":3838,"fn":3839,"description":3840,"org":3841,"tags":3842,"stars":23,"repoUrl":24,"updatedAt":3851},"gcx","manage Grafana Cloud resources via gcx","Manages Grafana Cloud resources via the gcx CLI. Trigger when the user wants to inspect, create, update, delete, query, or automate any Grafana resource - dashboards, datasources, alerts, SLOs, synthetic checks, oncall, incidents, fleet, k6, knowledge graph, or adaptive telemetry.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3843,3846,3847,3848],{"name":3844,"slug":3845,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":3643,"slug":3644,"type":15},{"name":3849,"slug":3850,"type":15},"Operations","operations","2026-07-31T05:53:50.587304",{"slug":3853,"name":3853,"fn":3854,"description":3855,"org":3856,"tags":3857,"stars":23,"repoUrl":24,"updatedAt":3863},"gcx-demo","present gcx demo tours","Run a narrated, read-only demo tour of gcx for customer or colleague presentations. Showcases the breadth of gcx across every Grafana Cloud product area — resources, datasources, metrics, logs, traces, SLOs, alerts, synthetic monitoring, IRM, k6, fleet, and more. All commands are strictly read-only. Trigger when the user says \"demo gcx\", \"show off gcx\", \"customer demo\", \"gcx tour\", or \"\u002Fgcx-demo\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3858,3859,3860],{"name":3844,"slug":3845,"type":15},{"name":9,"slug":8,"type":15},{"name":3861,"slug":3862,"type":15},"Presentations","presentations","2026-07-25T05:30:45.282458"]