[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-grafana-agento11y-instrument":3,"mdc--d1fmav-key":37,"related-repo-grafana-agento11y-instrument":3138,"related-org-grafana-agento11y-instrument":3232},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":32,"sourceUrl":35,"mdContent":36},"agento11y-instrument","instrument LLM apps for agent observability","Sets up and instruments a developer's own LLM app or agent to send generations and agentic workflow to Grafana Agent Observability (the Agent Observability SDKs) — greenfield setup, fixing broken instrumentation, or filling gaps in existing instrumentation. Uses gcx for the parts a static prompt can't do: `gcx login` \u002F `gcx cloud stacks` to find the stack, and `gcx agento11y agents|conversations|generations` to VERIFY that data actually lands — so it iterates (instrument → run → verify → fix) until generations arrive, not blindly. Reads the app's code, detects language\u002Fframework, classifies instrumentation state (none \u002F partial \u002F broken), then runs a fixed gap checklist whose #1 item is the silent failure no other prompt catches: the SDK emits OTel spans\u002Fmetrics but never creates a TracerProvider\u002FMeterProvider, so without them all metrics go to a no-op and are lost. Also checks agent_version (required for per-version Performance charts), set_result completeness, SYNC vs STREAM, parent_generation_ids DAG links, and workflow-step coverage. Recommends changes citing file:line and, only with explicit confirmation, applies minimal diffs that don't change app behavior. Pulls SDK reference from agento11y's llms.txt rather than restating it, and hands off to `agento11y-test-starter` once data flows. It does NOT write test suites or set up tenant evaluations, rules, or guards — offline test suites are `agento11y-test-starter`, tenant eval rules + guards are `agento11y-prod-setup`; does NOT install coding-agent telemetry plugins (that is llms.txt \"Path A\"); does NOT mint or store credentials or invent endpoints. Trigger on phrases like \"instrument my app\", \"send my agent's traces to Grafana\", \"set up AI observability for my app\", \"my generations aren't showing up\", \"why is Performance empty\", \"add Agent Observability to my code\", \"fix my instrumentation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"grafana","Grafana","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fgrafana.jpg",[12,16,19,22,23],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"LLM","llm",{"name":20,"slug":21,"type":15},"Agents","agents",{"name":9,"slug":8,"type":15},{"name":24,"slug":25,"type":15},"Instrumentation","instrumentation",430,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgcx","2026-07-31T05:53:52.580237",null,29,[],{"repoUrl":27,"stars":26,"forks":30,"topics":33,"description":34},[],"A CLI for managing Grafana Cloud resources. Optimized for agentic usage.","https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgcx\u002Ftree\u002FHEAD\u002Fclaude-plugin\u002Fskills\u002Fagento11y-instrument","---\nname: agento11y-instrument\ndescription: >\n Sets up and instruments a developer's own LLM app or agent to send generations and\n agentic workflow to Grafana Agent Observability (the Agent Observability SDKs) — greenfield setup,\n fixing broken instrumentation, or filling gaps in existing instrumentation. Uses gcx\n for the parts a static prompt can't do: `gcx login` \u002F `gcx cloud stacks` to find the\n stack, and `gcx agento11y agents|conversations|generations` to VERIFY that data actually\n lands — so it iterates (instrument → run → verify → fix) until generations arrive, not\n blindly. Reads the app's code, detects language\u002Fframework, classifies instrumentation\n state (none \u002F partial \u002F broken), then runs a fixed gap checklist whose #1 item is the\n silent failure no other prompt catches: the SDK emits OTel spans\u002Fmetrics but never\n creates a TracerProvider\u002FMeterProvider, so without them all metrics go to a no-op and are\n lost. Also checks agent_version (required for per-version Performance charts), set_result\n completeness, SYNC vs STREAM, parent_generation_ids DAG links, and workflow-step coverage.\n Recommends changes citing file:line and, only with explicit confirmation, applies minimal\n diffs that don't change app behavior. Pulls SDK reference from agento11y's llms.txt rather\n than restating it, and hands off to `agento11y-test-starter` once data flows. It does NOT\n write test suites or set up tenant evaluations, rules, or guards — offline test suites are\n `agento11y-test-starter`, tenant eval rules + guards are `agento11y-prod-setup`;\n does NOT install coding-agent telemetry plugins (that is llms.txt \"Path A\"); does NOT mint\n or store credentials or invent endpoints. Trigger on phrases like \"instrument my app\",\n \"send my agent's traces to Grafana\", \"set up AI observability for my app\", \"my generations\n aren't showing up\", \"why is Performance empty\", \"add Agent Observability to my code\", \"fix my instrumentation\".\nallowed-tools: Bash, Read, Grep, Glob, Edit, Write, WebFetch\n---\n\n# Agent Observability — instrument an LLM app\n\nHelp a developer wire **their own** LLM app or agent to Grafana Agent Observability, from zero or\nfrom a broken\u002Fpartial state, and **keep going until data actually lands in the stack**. The value\nthis skill adds over the static instrumentation prompt is two things a prompt can't do:\n\n1. A mechanical **gap checklist** run against the real code — headed by the one failure that is\n completely silent (missing OTel providers → every metric lost, no error).\n2. A **verification loop** through `gcx`: after each change, run the app and check the\n `gcx agento11y` agents \u002F conversations \u002F generations commands to confirm generations arrived.\n Diagnose the next gap from what's missing, not from guesswork.\n\nThe SDK reference (env vars, provider snippets, field lists, framework adapters, workflow steps)\nlives in agento11y's `llms.txt` \"Path B\". Fetch it rather than restating it here; this file holds\nthe flow and the decision logic. A minimal fallback lives in\n[references\u002Finstrumentation.md](references\u002Finstrumentation.md) for when the fetch is unavailable.\n\n## Rules\n\n- **Reference, don't restate.** Fetch SDK detail from\n `https:\u002F\u002Fraw.githubusercontent.com\u002Fgrafana\u002Fagento11y\u002Fmain\u002Fllms.txt` (Path B). Only inline decision\n logic here. If the fetch fails, fall back to [references\u002Finstrumentation.md](references\u002Finstrumentation.md).\n- **Never invent an endpoint or a token.** Read them from the environment (`AGENTO11Y_ENDPOINT`,\n `AGENTO11Y_PROTOCOL`, `AGENTO11Y_AUTH_MODE`, `AGENTO11Y_AUTH_TENANT_ID`, `AGENTO11Y_AUTH_TOKEN`,\n `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`) or ask the developer. Never fabricate a\n URL or mint a token.\n- **Target is Grafana Cloud.** The developer supplies the endpoint + token (Step 0), and the gcx\n verification loop (Step 5) confirms data landing against the Cloud tenant. Never fabricate the\n endpoint or token — read them from the environment or ask.\n- **Write `AGENTO11Y_*` env vars, never `SIGIL_*`.** `SIGIL_*` is a deprecated legacy fallback. Do\n this **even if sibling apps or existing `.env` files in the repo use `SIGIL_*`** — matching a stale\n local convention perpetuates it. If the app already reads `SIGIL_*`, add the `AGENTO11Y_*` names\n (the SDK still honors both) and note the old ones are deprecated. Do not \"match the siblings.\"\n- **The gcx command group is `gcx agento11y`.** The old name `aio11y` (still the internal Go package\n name) does **not** exist as a command — an invocation using aio11y instead of agento11y fails.\n Every verification command uses the `agento11y` group. Do not emit the old aio11y command name even\n if prior knowledge suggests it.\n- **Gate every code WRITE on explicit confirmation.** Report first (Step 4), apply only after the\n developer says yes (Step 5). Read-only gcx verification and re-running the app happen freely inside\n the loop; editing files does not.\n- **Keep diffs small; do not change app behavior.** Instrumentation is additive. No refactors, no\n prompt rewrites, no dependency upgrades beyond the SDK\u002Fadapter packages actually needed.\n- **Never change the model, provider, or the app's LLM config — not even with permission, not even\n \"just to run the verify loop.\"** Instrument whatever model the app already uses. This is absolute:\n changing the model is out of scope for instrumentation, full stop. If a run fails because a\n provider API key is missing, the only allowed responses are: (a) ask the developer to provide the\n key for the model the app *already* uses, or (b) skip the live run and report the wiring as\n verified-by-construction, telling the developer to run it themselves. Do **not** offer to switch\n the provider, do **not** ask \"which provider should I use?\", and do **not** add a new provider\n dependency (e.g. `langchain-anthropic`) to make the run succeed. If the developer separately says\n they *want* a different model, that is an app change they own — tell them to make it and re-invoke\n this skill; do not fold it into the instrumentation diff. Swapping the model silently changes what\n the app does and what gets observed, which defeats the point. **The provider API key\n (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, …) is the app's own concern, not the instrumentation's** —\n it authenticates the LLM call, not the telemetry export, and the app already has it if it runs at\n all. So don't ask for it, configure it, or rewire it; if the live verify-run fails on a missing\n provider key, skip the run and report verified-by-construction (see Step 5). Just don't conflate the\n two 401s: a 401 on **generation ingest** is observability auth and *is* yours to fix (usually a\n missing `AGENTO11Y_PROTOCOL`\u002F`AGENTO11Y_AUTH_MODE`); an auth error from the **model provider** is\n not — surface it and let the developer handle their own key.\n- **Do not assume language symmetry.** Verify the provider wrapper \u002F framework adapter actually\n exists for the app's language before recommending it (Python has the most adapters, JS fewer, Go\n only google-adk, Java\u002F.NET core + providers + google-adk). If it doesn't exist, hand-instrument\n with the core SDK. Prefer, in order: **provider wrapper → framework adapter → hand-instrumentation**.\n- **The loop is bounded.** At most ~3–4 instrument→verify iterations. If data still isn't landing,\n stop and report what's checked and what remains — don't loop forever.\n- **Field-name traps:** `cache_write_input_tokens`, NOT `cache_creation_input_tokens`. `agent_version`\n maps to the `gen_ai.agent.version` label and is required for per-version Performance charts.\n **`MessageRole` (Python SDK) has only `USER`, `ASSISTANT`, `TOOL` — there is no `SYSTEM` (or\n `DEVELOPER`) member**; `MessageRole.SYSTEM` raises `AttributeError`. Fold the system prompt into the\n `USER` message (or a `text_part`), and prefer the `user_text_message()` \u002F `assistant_text_message()`\n \u002F `tool_result_message()` helpers over hand-building `Message(role=...)`. Always confirm enum members\n and helper names against the installed package before running — do not assume from llms.txt.\n- **Out of scope:** offline test suites → `agento11y-test-starter`; tenant eval rules + guards on\n real traffic → `agento11y-prod-setup`. Coding-agent telemetry plugins (Claude Code, Cursor, …) →\n llms.txt \"Path A\". Any control-plane write.\n- **If a required input is missing** (entrypoint, framework, endpoint, gcx auth), ask — don't guess.\n\n## Step 0 — Credentials and endpoint\n\nThe app needs, in its environment before the SDK starts, **seven** vars — not five; the two mode\nvars are the ones most often forgotten, and their absence is a silent 401:\n\n- generation ingest: `AGENTO11Y_ENDPOINT`, `AGENTO11Y_PROTOCOL=http`, `AGENTO11Y_AUTH_MODE=basic`,\n `AGENTO11Y_AUTH_TENANT_ID`, `AGENTO11Y_AUTH_TOKEN`.\n- traces\u002Fmetrics: `OTEL_EXPORTER_OTLP_ENDPOINT`, `OTEL_EXPORTER_OTLP_HEADERS`.\n\n`AGENTO11Y_PROTOCOL=http` and `AGENTO11Y_AUTH_MODE=basic` are **required for Cloud, not optional** —\nthe SDK defaults (grpc \u002F no-auth) return a silent 401 against the Cloud HTTP ingest endpoint. (They\nscope the ingest channel only; the OTel channel's transport\u002Fauth is set entirely by the `OTEL_*`\nvars — see references\u002Finstrumentation.md.) The only var that is sometimes omittable is\n`OTEL_EXPORTER_OTLP_HEADERS`: **required when sending directly to the Cloud OTLP gateway** (the common\ncase, gateway enforces Basic auth), omittable **only** when `OTEL_EXPORTER_OTLP_ENDPOINT` points at a\nlocal Alloy \u002F OTel Collector that already holds the Cloud credentials. First check what's already set\n(including any existing `.env`) — if all are present, skip to Step 1. Watch for the near-miss where\nthe endpoint is set under the wrong name (e.g. `AGENTO11Y_API_ENDPOINT` — the SDK reads\n`AGENTO11Y_ENDPOINT`, so the wrong name is silently ignored and ingest falls back to a default host).\n\n> **When any value is missing, do NOT just list the variable names and ask — hand the developer the\n> exact place to get each one (link + clicks), every time.** The concrete sources are in point 3\n> below; surface them proactively. The most common failure of this skill is naming\n> `OTEL_EXPORTER_OTLP_ENDPOINT`\u002F`OTEL_EXPORTER_OTLP_HEADERS` and leaving the developer to guess — the\n> answer is the stack OTLP tile + \"Generate now\", which precomputes both. Give that first.\n\n**What gcx does for you (run these):**\n\n1. `gcx config current-context` — is there a working context? If not, **just ask the developer to log\n in to the stack they want the instrumentation to connect to** — e.g. \"run `gcx login` against your\n stack.\" Do not fabricate the login command yourself (don't guess the host or flags); let them run\n their own login (the Agent Observability setup screen gives them the exact command, or they use\n `gcx login`). Instrumentation itself needs no gcx login — only Step 5 verification does, so this\n never blocks writing the code.\n2. `gcx cloud stacks list`, then `gcx cloud stacks get \u003Cstack-slug>` — identify the target stack and\n its URLs. This gives you the stack to point the developer at, and confirms which tenant the Step 5\n verification will read from.\n\n**What still needs the Connection page (gcx cannot do these today):**\n\n3. gcx does **not** generate the Agent Observability OTLP gateway URL, and does **not** mint the\n ingest \u002F access-policy token. **When you ask the developer for a value, always tell them exactly\n where to get it — a link and the clicks — never just name the variable and wait.** The two channels\n come from two different places:\n\n **`OTEL_*` (traces\u002Fmetrics) — easiest, let Cloud build them.** Send the developer to the stack's\n OTLP tile: `https:\u002F\u002Fgrafana.com\u002Forgs\u002F\u003Corg-slug>\u002Fstacks\u002F\u003Cstack-id>\u002Fotlp-info`. It already shows\n `OTEL_EXPORTER_OTLP_ENDPOINT` and the Instance ID; under **Password \u002F API Token → \"Generate now\"**\n it mints a token and then fills an **Environment Variables** block with all `OTEL_*` vars — **the\n base64 `OTEL_EXPORTER_OTLP_HEADERS` is precomputed**, ready to copy. No manual base64. (In Python,\n the value uses `Basic%20…` — keep it as given.)\n\n **`AGENTO11Y_*` (generation ingest) — the plugin Connection page.** `AGENTO11Y_ENDPOINT` and the\n token come from `https:\u002F\u002F\u003Cstack>.grafana.net\u002Fplugins\u002Fgrafana-agento11y-app` → Connection tab. When the\n developer creates the token via **\"Create a token in Cloud Access Policies\"**, tell them the scopes:\n **`sigil:write`, `metrics:write`, `traces:write`, `logs:write`**. UI heads-up: `sigil` is not in\n the default resource list — add it via **\"Add scope\"** (then tick Write); the scope is still\n `sigil:*` (the Cloud resource keeps the old name). The same `glc_…` token works for both channels\n if it has all four scopes. Also set `AGENTO11Y_PROTOCOL=http` and `AGENTO11Y_AUTH_MODE=basic` (see\n references\u002Finstrumentation.md — the SDK defaults grpc\u002Fnone give a 401).\n\n Ask the developer to put the values in a gitignored `.env` (or export them) **themselves** — **do\n not ask them to paste a secret token into the chat** (it is captured in the transcript). Instrument\n the code to read from the environment and have them supply the values out-of-band. **Never invent a\n URL or mint a token.**\n\n > Two different tokens — don't confuse them. gcx logs in with its own OAuth token (`gat_`) and\n > refreshes it automatically; that is what authenticates the `gcx` commands here. It is **not** the\n > app's ingest token. The app needs a separate access-policy token (`glc_…`) in\n > `AGENTO11Y_AUTH_TOKEN` \u002F `OTEL_EXPORTER_OTLP_HEADERS`, and gcx does not create that one.\n\nOnce gcx has a working context, the Step 5 verification commands (under the `gcx agento11y` group)\nwork against the developer's tenant even before the app's own credentials are fully wired.\n\n> The Connection page is the only manual step. If a future gcx release can create the access-policy\n> token and surface the OTLP endpoint, this step collapses to gcx-only — but do not assume it can\n> today; check `gcx cloud --help` rather than promising it.\n\n## Step 1 — Read the app and detect language \u002F framework \u002F shape\n\nFind and read, recording `file:line` for each:\n\n1. The generation entrypoint(s) — where the model is invoked.\n2. How the LLM client is constructed (which provider: OpenAI \u002F Anthropic \u002F Gemini \u002F other).\n3. The app bootstrap \u002F init — where an OTel `TracerProvider` \u002F `MeterProvider` would be created.\n4. Any existing Agent Observability SDK imports (`agento11y` \u002F `@grafana\u002Fagento11y` \u002F the Go\n `agento11y` package — or the legacy `sigil_sdk` \u002F `@grafana\u002Fsigil-sdk-js` \u002F Go `sigil` names in\n older code) or `AGENTO11Y_*` usage.\n\nDetect:\n- **Language** — from the manifest \u002F extensions (`pyproject.toml`\u002F`.py`, `package.json`\u002F`.ts`,\n `go.mod`\u002F`.go`, gradle\u002F`.java`, `.csproj`\u002F`.cs`).\n- **Framework** — grep for `langgraph`, `langchain`, `openai-agents`, `llamaindex`, `google-adk`,\n `strands`, `pydantic-ai`, `litellm`, `claude-agent-sdk`, `vercel-ai-sdk`, `crewai`, or a custom\n orchestrator.\n- **Shape** — single generation vs agentic pipeline (multiple nodes \u002F a graph \u002F sub-agents). This\n decides whether workflow steps (checklist #8) and parent links (#7) are in play.\n\nBefore recommending a provider wrapper or framework adapter, confirm it exists for this language\n(fetch llms.txt \"SDK API surface\" — the matrix is not symmetric across languages).\n\n## Step 2 — Classify instrumentation state\n\nState the classification explicitly, with the `file:line` evidence that led to it:\n\n- **`none` (greenfield)** — no SDK import anywhere. Full setup from scratch.\n- **`partial`** — an SDK client is constructed and some generations are wrapped, but coverage is\n incomplete (no OTel providers, no `agent_version`, no workflow steps for an agentic pipeline, no\n parent links). Run the full checklist; recommend + apply only the gaps.\n- **`broken`** — the SDK is present but wrong: metrics silently lost (no MeterProvider), export\n misconfigured, legacy `SIGIL_*` vars, stream\u002Fnon-stream mode mismatch, `set_result`\u002F`SetResult`\n never called, or `rec.err()`\u002F`Err()` never checked. Fix first, then gap-check.\n\nAll three paths converge on the same checklist (Step 3); they differ only in how much is already done.\n\n## Step 3 — Run the instrumentation gap checklist\n\nWalk each item against the code. Record PRESENT \u002F MISSING \u002F WRONG with `file:line`. This mechanical\naudit is the skill's core value. Items 0, 1, 2, 5, 6 fail **silently** (no error) — always check them.\nItems 3, 7, 8 mean data lands but analysis is degraded. For the fix, read the named **section** of\nthe fetched llms.txt (locate it by its heading — do not trust line numbers, they drift).\n\n| # | Check | Silent-failure symptom | llms.txt section |\n|---|-------|------------------------|------------------|\n| 0 | **The `.env` actually takes effect.** Confirm the app loads its own `.env` by an explicit path (not a bare `load_dotenv()` resolved by CWD) **and** that it wins over vars already in the environment. Verify by printing `os.environ[\"AGENTO11Y_ENDPOINT\"]` \u002F `OTEL_EXPORTER_OTLP_ENDPOINT` **after** all imports, not before | **`import litellm` (and some other libs) inject localhost OTLP\u002Fingest defaults into `os.environ` at import time** (`OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4318`, `AGENTO11Y_ENDPOINT=localhost:8080`). A plain `load_dotenv()` does **not** override already-set vars → the Cloud endpoints in `.env` never apply and everything ships to localhost, returning **200 OK** if a local stack is up. Zero error signal, and gcx against the Cloud tenant shows nothing. Fix: `load_dotenv(\u003Cpath-relative-to-__file__>, override=True)` before constructing providers\u002Fclient. A bare `load_dotenv()` also resolves the wrong `.env` by CWD | \"Environment\" |\n| 1 | OTel TracerProvider **and** MeterProvider created before the SDK client (verify by construction + Performance view \u002F OTLP POSTs — **not** via gcx, which can't see OTel; see Step 5) | spans\u002Fmetrics go to no-op → all latency\u002Ftoken\u002Fcost metrics lost. The #1 failure. | \"OTel setup (required)\" |\n| 2 | Providers shut down after `shutdown()` | last batch of spans\u002Fmetrics dropped on exit | \"OTel setup (required)\" |\n| 3 | `agent_name` + `agent_version` set on generations \u002F handlers | per-version Performance charts break (join on `gen_ai.agent.version`) | \"Agent Observability architecture and ingest model\", \"Telemetry fields to prioritize\" |\n| 4 | `set_result`\u002F`SetResult` includes response_id, response_model, finish\u002Fstop reason, full token usage (incl. `cache_read_input_tokens`, `cache_write_input_tokens`, `reasoning_tokens`), **and `input`\u002F`output` populated with `Message` objects** (system+user prompt in `input`, model reply in `output`) | charts\u002Fcost blank; wrong `cache_creation_input_tokens` name silently ignored; **empty `input`\u002F`output` → the conversation thread shows \"No messages in this turn\" — tokens land but there is no visible prompt\u002Fresponse** | \"Implementation rules\", \"Telemetry fields to prioritize\" |\n| 5 | `rec.err()`\u002F`Err()` checked after the recorder closes | SDK validation\u002Fenqueue errors are silent → generations never arrive, no signal | \"Implementation rules\" |\n| 6 | SYNC (non-stream) vs STREAM (stream) set correctly | streaming metrics (TTFT) corrupted | \"Agent Observability architecture and ingest model\", \"Implementation rules\" |\n| 6b | `operation_name` is a **recognized** value — `generateText` (SYNC default), `streamText` (STREAM default), `embeddings`, `execute_tool`, `framework_chain`, `framework_retriever`. Best: omit it and take the SDK default. Do **not** invent one like `\"chat\"` | the span reaches Tempo but the UI classifies `gen_ai.operation.name` as `unknown` → the conversation renders a synthetic generation node **with no attached span** → the trace does not show in the conversation and the \"T\" (trace) icon is absent, even though `trace_id`\u002F`span_id` are set. Silent, like #1 | \"Agent Observability architecture and ingest model\", \"Implementation rules\" |\n| 7 | `parent_generation_ids` set on multi-agent \u002F fan-in generations | no dependency DAG; upstream eval failures don't propagate | \"Multi-agent dependency tracking\" |\n| 8 | Workflow steps emitted for agentic pipelines with non-LLM nodes | execution graph invisible; node input\u002Foutput state lost. Use the adapter if one exists, else `enqueue_workflow_step`; never both for one node (duplicates) | \"Workflow step instrumentation (agentic pipelines)\" |\n| 9 | Env vars are `AGENTO11Y_*` (not legacy `SIGIL_*`); client built config-free when env present | drift; duplicated config | \"Environment\" |\n| 10 | Content-capture mode intentional (SDK default `no_tool_content`); no secrets in `tags`\u002F`metadata`\u002F`user_id` | PII\u002Fsecrets leak into exports | \"Content capture\", \"Tags, metadata, and user id\" |\n| 11 | Client tags low-cardinality; end-user identity via `user_id`, not a tag | high-cardinality tags blow up metric labels | \"Tags, metadata, and user id\", \"Implementation rules\" |\n\n## Step 4 — Recommend (the report)\n\nEmit the report using llms.txt's output contract (its \"Output contract\" section): top opportunities\nfirst, and per opportunity — exact `file:line`, why it matters, a concrete diff proposal, a test\nplan, and any risk. Rank by severity: `.env` not taking effect (#0 — nothing lands at all) first,\nthen missing OTel provider (metrics data loss), then broken export, then missing `agent_version`,\nthen coverage gaps. Every recommendation cites a `file:line`. Then stop and ask\nbefore applying anything.\n\n## Step 5 — Apply, then verify (the loop)\n\nOnly after the developer confirms a diff. Bounded to ~3–4 iterations.\n\n1. **Apply** the confirmed diff (Edit\u002FWrite). Order of preference: provider wrapper → framework\n adapter → hand-instrument the core SDK — only what exists for the language. Add\u002Fupdate a focused\n test for the changed instrumentation. Preserve flush\u002Fshutdown lifecycle. Never touch app logic.\n Pull exact usage from llms.txt \u002F the per-language README \u002F `examples\u002Fgetting-started\u002F*`.\n2. **Run the app** for one turn to generate traffic (ask the developer to run it, or run it if\n there's a clean entrypoint and they approve). **If the run can't happen** — missing provider API\n key, no clean entrypoint, needs a full runtime — do **not** work around it by changing the model\n or adding a provider. Stop the loop, report the wiring as verified-by-construction (imports\n resolve, providers build, client + handler construct), and tell the developer the one thing left\n is to run one turn themselves with their key. A verified-by-construction result is a fine outcome.\n3. **Verify — two independent channels, don't conflate them.** Instrumentation sends data on two\n separate paths, and confirming one says **nothing** about the other.\n\n **First, confirm gcx reads the same tenant the app writes to.** A verification against the wrong\n tenant is worse than no verification — an empty `agents list` gets misread as \"data isn't landing\"\n when it is, just elsewhere. Before drawing any conclusion from a gcx query: read the app's\n `AGENTO11Y_ENDPOINT` + `AGENTO11Y_AUTH_TENANT_ID` from its `.env`, then check `gcx config\n current-context` and that the active context points at that same stack\u002Ftenant. If it doesn't (e.g.\n context is `local` but the app writes to a Cloud stack), switch context or ask the developer to\n `gcx login` to the right stack — do not guess the login command. (If the gcx token is merely\n expired, that blocks Step 5 verification only, not writing the code — say so and continue.)\n\n **Channel A — generations** (the SDK ingest client → `\u002Fapi\u002Fv1\u002Fgenerations:export`). Carries the\n prompt, response, tokens, cost, model, finish_reason. This is what gcx can read.\n - **Via gcx:** `gcx agento11y agents list` (does the agent appear?);\n `gcx agento11y agents get \u003Cagent-name>` (is `generation_count` climbing?). To find the run's\n conversation, either `gcx agento11y conversations list --limit \u003Cn>` (most-recent first, no\n filters — the quickest post-run check) or `gcx agento11y conversations search --filters\n 'agent = \"\u003Cagent-name>\"'` (`--filters` alone is enough; `--from`\u002F`--to`, RFC3339, are **optional**\n and only needed to narrow a busy tenant). Then `gcx agento11y conversations get \u003Cconversation-id>`\n or `gcx agento11y generations get \u003Cgeneration-id>` — check tokens, finish reason, cost, and (for\n a multi-agent pipeline) that `parent_generation_ids` reproduce the DAG. This proves generation\n ingest + `set_result` are wired. **It does NOT prove OTel is wired.**\n\n **Channel B — OTel spans\u002Fmetrics** (the TracerProvider\u002FMeterProvider → OTLP exporter →\n `\u002Fv1\u002Ftraces`, `\u002Fv1\u002Fmetrics`). Carries latency\u002Ftoken\u002Fcost **metrics**. This is checklist #1, the\n #1 silent failure — and **gcx cannot see it** (traces\u002Fmetrics land in the stack's Tempo\u002FPrometheus,\n not the Agent Observability ingest API). So verify Channel B separately, by the strongest signal\n available, in this order:\n - **By construction (always do this):** confirm in the applied code that both a TracerProvider\n **and** a MeterProvider are created *before* the SDK client and shut down after it. This is\n static but reliable — a missing\u002Flate\u002Fno-op MeterProvider is exactly checklist #1.\n - **At runtime, if you can observe it:** run the **real app** (not an isolated probe script) with\n the OTel\u002Furllib3 debug log on, and confirm you see **both** `POST …\u002Fv1\u002Ftraces → 2xx` **and**\n `POST …\u002Fv1\u002Fmetrics → 2xx`. The SDK emits spans automatically from `start_generation`\u002F`end` (one\n traces POST per generation) and metrics on an interval — a clean shutdown flush surfaces both.\n **Traces and metrics are separate exports: seeing only `\u002Fv1\u002Fmetrics` does NOT mean traces work,\n and vice-versa. Never claim \"traces\u002Fmetrics verified\" from a probe that only exercised one of\n them** — that is the exact trap that reports Channel B as done when half of it was never sent. If\n you write a throwaway verification script, it must build the TracerProvider **and** MeterProvider\n and record a real generation, or just instrument the app and read its debug output.\n - **In the UI:** the stack's **Performance \u002F metrics** view populates from metrics; **traces** land\n in the stack's **Tempo** (Explore → Tempo, filter by `service.name`). If conversations appear\n (Channel A) but Performance is empty, the MeterProvider is missing or no-op → back to checklist\n #1. Do not report OTel as wired on the strength of `generations get` alone, nor on metrics alone.\n - **Trace shows in Tempo but NOT inside the conversation (no \"T\" icon):** the span is landing but\n `gen_ai.operation.name` is an unrecognized value (e.g. `\"chat\"`) → the UI classifies it as\n `unknown` and can't attach it to the conversation node. This is checklist #6b — fix\n `operation_name` to a recognized value (or omit it for the default) and re-run.\n4. If a signal is missing, diagnose the next gap from what the checks showed, propose the fix, and\n loop back to step 1. After ~3–4 iterations without full signal, stop and report exactly what\n lands, what doesn't, and what to check next (app stderr for `agento11y:` warnings, credentials).\n\n## Step 6 — Hand off\n\nOnce generations land and metrics populate, instrumentation is done — that's the prerequisite for\neverything else. Point the developer at the next step: `agento11y-test-starter` to build an\n**offline test suite** for the agent (useful before shipping *and* for regression-testing new\nversions once it's live), and `agento11y-prod-setup` to set up **online eval rules + guards** on\nreal traffic once it's deployed. The split is offline test suite vs online rules\u002Fguards — not\nbefore-traffic vs after-traffic.\n\n## Note — keeping this skill in sync\n\nThe SDK reference (env vars, provider snippets, field lists, workflow-step schema, adapter matrix) is\nintentionally **not** duplicated here — it lives in agento11y's `llms.txt` \"Path B\" and the\nper-language READMEs, which are the shipped source of truth. This skill holds only decision logic\n(state classification + gap checklist + the gcx verification loop). When a user-facing semantic\nchanges (new SDK field, renamed env var, new framework adapter), update `llms.txt` (and its onboarding\nwizard copy); this skill points at llms.txt **by section heading, not line number** (line numbers\ndrift as llms.txt is edited), so no re-pointing is needed unless a heading itself is renamed. If you\nfind yourself pasting a provider snippet into this file, stop — fetch llms.txt instead. The\n`references\u002Finstrumentation.md` fallback is deliberately minimal for the same reason.\n",{"data":38,"body":40},{"name":4,"description":6,"allowed-tools":39},"Bash, Read, Grep, Glob, Edit, Write, WebFetch",{"type":41,"children":42},"root",[43,52,73,119,140,147,697,703,715,772,854,881,889,944,952,1246,1258,1274,1280,1293,1381,1386,1562,1567,1573,1585,1676,1681,1687,1713,2504,2510,2543,2549,2554,3048,3054,3093,3099],{"type":44,"tag":45,"props":46,"children":48},"element","h1",{"id":47},"agent-observability-instrument-an-llm-app",[49],{"type":50,"value":51},"text","Agent Observability — instrument an LLM app",{"type":44,"tag":53,"props":54,"children":55},"p",{},[56,58,64,66,71],{"type":50,"value":57},"Help a developer wire ",{"type":44,"tag":59,"props":60,"children":61},"strong",{},[62],{"type":50,"value":63},"their own",{"type":50,"value":65}," LLM app or agent to Grafana Agent Observability, from zero or\nfrom a broken\u002Fpartial state, and ",{"type":44,"tag":59,"props":67,"children":68},{},[69],{"type":50,"value":70},"keep going until data actually lands in the stack",{"type":50,"value":72},". The value\nthis skill adds over the static instrumentation prompt is two things a prompt can't do:",{"type":44,"tag":74,"props":75,"children":76},"ol",{},[77,90],{"type":44,"tag":78,"props":79,"children":80},"li",{},[81,83,88],{"type":50,"value":82},"A mechanical ",{"type":44,"tag":59,"props":84,"children":85},{},[86],{"type":50,"value":87},"gap checklist",{"type":50,"value":89}," run against the real code — headed by the one failure that is\ncompletely silent (missing OTel providers → every metric lost, no error).",{"type":44,"tag":78,"props":91,"children":92},{},[93,95,100,102,109,111,117],{"type":50,"value":94},"A ",{"type":44,"tag":59,"props":96,"children":97},{},[98],{"type":50,"value":99},"verification loop",{"type":50,"value":101}," through ",{"type":44,"tag":103,"props":104,"children":106},"code",{"className":105},[],[107],{"type":50,"value":108},"gcx",{"type":50,"value":110},": after each change, run the app and check the\n",{"type":44,"tag":103,"props":112,"children":114},{"className":113},[],[115],{"type":50,"value":116},"gcx agento11y",{"type":50,"value":118}," agents \u002F conversations \u002F generations commands to confirm generations arrived.\nDiagnose the next gap from what's missing, not from guesswork.",{"type":44,"tag":53,"props":120,"children":121},{},[122,124,130,132,138],{"type":50,"value":123},"The SDK reference (env vars, provider snippets, field lists, framework adapters, workflow steps)\nlives in agento11y's ",{"type":44,"tag":103,"props":125,"children":127},{"className":126},[],[128],{"type":50,"value":129},"llms.txt",{"type":50,"value":131}," \"Path B\". Fetch it rather than restating it here; this file holds\nthe flow and the decision logic. A minimal fallback lives in\n",{"type":44,"tag":133,"props":134,"children":136},"a",{"href":135},"references\u002Finstrumentation.md",[137],{"type":50,"value":135},{"type":50,"value":139}," for when the fetch is unavailable.",{"type":44,"tag":141,"props":142,"children":144},"h2",{"id":143},"rules",[145],{"type":50,"value":146},"Rules",{"type":44,"tag":148,"props":149,"children":150},"ul",{},[151,175,237,247,313,352,362,372,480,496,506,661,687],{"type":44,"tag":78,"props":152,"children":153},{},[154,159,161,167,169,173],{"type":44,"tag":59,"props":155,"children":156},{},[157],{"type":50,"value":158},"Reference, don't restate.",{"type":50,"value":160}," Fetch SDK detail from\n",{"type":44,"tag":103,"props":162,"children":164},{"className":163},[],[165],{"type":50,"value":166},"https:\u002F\u002Fraw.githubusercontent.com\u002Fgrafana\u002Fagento11y\u002Fmain\u002Fllms.txt",{"type":50,"value":168}," (Path B). Only inline decision\nlogic here. If the fetch fails, fall back to ",{"type":44,"tag":133,"props":170,"children":171},{"href":135},[172],{"type":50,"value":135},{"type":50,"value":174},".",{"type":44,"tag":78,"props":176,"children":177},{},[178,183,185,191,193,199,201,207,208,214,215,221,222,228,229,235],{"type":44,"tag":59,"props":179,"children":180},{},[181],{"type":50,"value":182},"Never invent an endpoint or a token.",{"type":50,"value":184}," Read them from the environment (",{"type":44,"tag":103,"props":186,"children":188},{"className":187},[],[189],{"type":50,"value":190},"AGENTO11Y_ENDPOINT",{"type":50,"value":192},",\n",{"type":44,"tag":103,"props":194,"children":196},{"className":195},[],[197],{"type":50,"value":198},"AGENTO11Y_PROTOCOL",{"type":50,"value":200},", ",{"type":44,"tag":103,"props":202,"children":204},{"className":203},[],[205],{"type":50,"value":206},"AGENTO11Y_AUTH_MODE",{"type":50,"value":200},{"type":44,"tag":103,"props":209,"children":211},{"className":210},[],[212],{"type":50,"value":213},"AGENTO11Y_AUTH_TENANT_ID",{"type":50,"value":200},{"type":44,"tag":103,"props":216,"children":218},{"className":217},[],[219],{"type":50,"value":220},"AGENTO11Y_AUTH_TOKEN",{"type":50,"value":192},{"type":44,"tag":103,"props":223,"children":225},{"className":224},[],[226],{"type":50,"value":227},"OTEL_EXPORTER_OTLP_ENDPOINT",{"type":50,"value":200},{"type":44,"tag":103,"props":230,"children":232},{"className":231},[],[233],{"type":50,"value":234},"OTEL_EXPORTER_OTLP_HEADERS",{"type":50,"value":236},") or ask the developer. Never fabricate a\nURL or mint a token.",{"type":44,"tag":78,"props":238,"children":239},{},[240,245],{"type":44,"tag":59,"props":241,"children":242},{},[243],{"type":50,"value":244},"Target is Grafana Cloud.",{"type":50,"value":246}," The developer supplies the endpoint + token (Step 0), and the gcx\nverification loop (Step 5) confirms data landing against the Cloud tenant. Never fabricate the\nendpoint or token — read them from the environment or ask.",{"type":44,"tag":78,"props":248,"children":249},{},[250,270,272,277,279,297,299,304,306,311],{"type":44,"tag":59,"props":251,"children":252},{},[253,255,261,263,269],{"type":50,"value":254},"Write ",{"type":44,"tag":103,"props":256,"children":258},{"className":257},[],[259],{"type":50,"value":260},"AGENTO11Y_*",{"type":50,"value":262}," env vars, never ",{"type":44,"tag":103,"props":264,"children":266},{"className":265},[],[267],{"type":50,"value":268},"SIGIL_*",{"type":50,"value":174},{"type":50,"value":271}," ",{"type":44,"tag":103,"props":273,"children":275},{"className":274},[],[276],{"type":50,"value":268},{"type":50,"value":278}," is a deprecated legacy fallback. Do\nthis ",{"type":44,"tag":59,"props":280,"children":281},{},[282,284,290,292],{"type":50,"value":283},"even if sibling apps or existing ",{"type":44,"tag":103,"props":285,"children":287},{"className":286},[],[288],{"type":50,"value":289},".env",{"type":50,"value":291}," files in the repo use ",{"type":44,"tag":103,"props":293,"children":295},{"className":294},[],[296],{"type":50,"value":268},{"type":50,"value":298}," — matching a stale\nlocal convention perpetuates it. If the app already reads ",{"type":44,"tag":103,"props":300,"children":302},{"className":301},[],[303],{"type":50,"value":268},{"type":50,"value":305},", add the ",{"type":44,"tag":103,"props":307,"children":309},{"className":308},[],[310],{"type":50,"value":260},{"type":50,"value":312}," names\n(the SDK still honors both) and note the old ones are deprecated. Do not \"match the siblings.\"",{"type":44,"tag":78,"props":314,"children":315},{},[316,327,329,335,337,342,344,350],{"type":44,"tag":59,"props":317,"children":318},{},[319,321,326],{"type":50,"value":320},"The gcx command group is ",{"type":44,"tag":103,"props":322,"children":324},{"className":323},[],[325],{"type":50,"value":116},{"type":50,"value":174},{"type":50,"value":328}," The old name ",{"type":44,"tag":103,"props":330,"children":332},{"className":331},[],[333],{"type":50,"value":334},"aio11y",{"type":50,"value":336}," (still the internal Go package\nname) does ",{"type":44,"tag":59,"props":338,"children":339},{},[340],{"type":50,"value":341},"not",{"type":50,"value":343}," exist as a command — an invocation using aio11y instead of agento11y fails.\nEvery verification command uses the ",{"type":44,"tag":103,"props":345,"children":347},{"className":346},[],[348],{"type":50,"value":349},"agento11y",{"type":50,"value":351}," group. Do not emit the old aio11y command name even\nif prior knowledge suggests it.",{"type":44,"tag":78,"props":353,"children":354},{},[355,360],{"type":44,"tag":59,"props":356,"children":357},{},[358],{"type":50,"value":359},"Gate every code WRITE on explicit confirmation.",{"type":50,"value":361}," Report first (Step 4), apply only after the\ndeveloper says yes (Step 5). Read-only gcx verification and re-running the app happen freely inside\nthe loop; editing files does not.",{"type":44,"tag":78,"props":363,"children":364},{},[365,370],{"type":44,"tag":59,"props":366,"children":367},{},[368],{"type":50,"value":369},"Keep diffs small; do not change app behavior.",{"type":50,"value":371}," Instrumentation is additive. No refactors, no\nprompt rewrites, no dependency upgrades beyond the SDK\u002Fadapter packages actually needed.",{"type":44,"tag":78,"props":373,"children":374},{},[375,380,382,388,390,394,396,400,402,406,408,414,416,421,423,443,445,450,452,457,459,464,466,471,473,478],{"type":44,"tag":59,"props":376,"children":377},{},[378],{"type":50,"value":379},"Never change the model, provider, or the app's LLM config — not even with permission, not even\n\"just to run the verify loop.\"",{"type":50,"value":381}," Instrument whatever model the app already uses. This is absolute:\nchanging the model is out of scope for instrumentation, full stop. If a run fails because a\nprovider API key is missing, the only allowed responses are: (a) ask the developer to provide the\nkey for the model the app ",{"type":44,"tag":383,"props":384,"children":385},"em",{},[386],{"type":50,"value":387},"already",{"type":50,"value":389}," uses, or (b) skip the live run and report the wiring as\nverified-by-construction, telling the developer to run it themselves. Do ",{"type":44,"tag":59,"props":391,"children":392},{},[393],{"type":50,"value":341},{"type":50,"value":395}," offer to switch\nthe provider, do ",{"type":44,"tag":59,"props":397,"children":398},{},[399],{"type":50,"value":341},{"type":50,"value":401}," ask \"which provider should I use?\", and do ",{"type":44,"tag":59,"props":403,"children":404},{},[405],{"type":50,"value":341},{"type":50,"value":407}," add a new provider\ndependency (e.g. ",{"type":44,"tag":103,"props":409,"children":411},{"className":410},[],[412],{"type":50,"value":413},"langchain-anthropic",{"type":50,"value":415},") to make the run succeed. If the developer separately says\nthey ",{"type":44,"tag":383,"props":417,"children":418},{},[419],{"type":50,"value":420},"want",{"type":50,"value":422}," a different model, that is an app change they own — tell them to make it and re-invoke\nthis skill; do not fold it into the instrumentation diff. Swapping the model silently changes what\nthe app does and what gets observed, which defeats the point. ",{"type":44,"tag":59,"props":424,"children":425},{},[426,428,434,435,441],{"type":50,"value":427},"The provider API key\n(",{"type":44,"tag":103,"props":429,"children":431},{"className":430},[],[432],{"type":50,"value":433},"ANTHROPIC_API_KEY",{"type":50,"value":200},{"type":44,"tag":103,"props":436,"children":438},{"className":437},[],[439],{"type":50,"value":440},"OPENAI_API_KEY",{"type":50,"value":442},", …) is the app's own concern, not the instrumentation's",{"type":50,"value":444}," —\nit authenticates the LLM call, not the telemetry export, and the app already has it if it runs at\nall. So don't ask for it, configure it, or rewire it; if the live verify-run fails on a missing\nprovider key, skip the run and report verified-by-construction (see Step 5). Just don't conflate the\ntwo 401s: a 401 on ",{"type":44,"tag":59,"props":446,"children":447},{},[448],{"type":50,"value":449},"generation ingest",{"type":50,"value":451}," is observability auth and ",{"type":44,"tag":383,"props":453,"children":454},{},[455],{"type":50,"value":456},"is",{"type":50,"value":458}," yours to fix (usually a\nmissing ",{"type":44,"tag":103,"props":460,"children":462},{"className":461},[],[463],{"type":50,"value":198},{"type":50,"value":465},"\u002F",{"type":44,"tag":103,"props":467,"children":469},{"className":468},[],[470],{"type":50,"value":206},{"type":50,"value":472},"); an auth error from the ",{"type":44,"tag":59,"props":474,"children":475},{},[476],{"type":50,"value":477},"model provider",{"type":50,"value":479}," is\nnot — surface it and let the developer handle their own key.",{"type":44,"tag":78,"props":481,"children":482},{},[483,488,490,495],{"type":44,"tag":59,"props":484,"children":485},{},[486],{"type":50,"value":487},"Do not assume language symmetry.",{"type":50,"value":489}," Verify the provider wrapper \u002F framework adapter actually\nexists for the app's language before recommending it (Python has the most adapters, JS fewer, Go\nonly google-adk, Java\u002F.NET core + providers + google-adk). If it doesn't exist, hand-instrument\nwith the core SDK. Prefer, in order: ",{"type":44,"tag":59,"props":491,"children":492},{},[493],{"type":50,"value":494},"provider wrapper → framework adapter → hand-instrumentation",{"type":50,"value":174},{"type":44,"tag":78,"props":497,"children":498},{},[499,504],{"type":44,"tag":59,"props":500,"children":501},{},[502],{"type":50,"value":503},"The loop is bounded.",{"type":50,"value":505}," At most ~3–4 instrument→verify iterations. If data still isn't landing,\nstop and report what's checked and what remains — don't loop forever.",{"type":44,"tag":78,"props":507,"children":508},{},[509,514,515,521,523,529,531,537,539,545,547,596,598,604,606,612,614,619,621,627,629,635,637,643,645,651,653,659],{"type":44,"tag":59,"props":510,"children":511},{},[512],{"type":50,"value":513},"Field-name traps:",{"type":50,"value":271},{"type":44,"tag":103,"props":516,"children":518},{"className":517},[],[519],{"type":50,"value":520},"cache_write_input_tokens",{"type":50,"value":522},", NOT ",{"type":44,"tag":103,"props":524,"children":526},{"className":525},[],[527],{"type":50,"value":528},"cache_creation_input_tokens",{"type":50,"value":530},". ",{"type":44,"tag":103,"props":532,"children":534},{"className":533},[],[535],{"type":50,"value":536},"agent_version",{"type":50,"value":538},"\nmaps to the ",{"type":44,"tag":103,"props":540,"children":542},{"className":541},[],[543],{"type":50,"value":544},"gen_ai.agent.version",{"type":50,"value":546}," label and is required for per-version Performance charts.\n",{"type":44,"tag":59,"props":548,"children":549},{},[550,556,558,564,565,571,572,578,580,586,588,594],{"type":44,"tag":103,"props":551,"children":553},{"className":552},[],[554],{"type":50,"value":555},"MessageRole",{"type":50,"value":557}," (Python SDK) has only ",{"type":44,"tag":103,"props":559,"children":561},{"className":560},[],[562],{"type":50,"value":563},"USER",{"type":50,"value":200},{"type":44,"tag":103,"props":566,"children":568},{"className":567},[],[569],{"type":50,"value":570},"ASSISTANT",{"type":50,"value":200},{"type":44,"tag":103,"props":573,"children":575},{"className":574},[],[576],{"type":50,"value":577},"TOOL",{"type":50,"value":579}," — there is no ",{"type":44,"tag":103,"props":581,"children":583},{"className":582},[],[584],{"type":50,"value":585},"SYSTEM",{"type":50,"value":587}," (or\n",{"type":44,"tag":103,"props":589,"children":591},{"className":590},[],[592],{"type":50,"value":593},"DEVELOPER",{"type":50,"value":595},") member",{"type":50,"value":597},"; ",{"type":44,"tag":103,"props":599,"children":601},{"className":600},[],[602],{"type":50,"value":603},"MessageRole.SYSTEM",{"type":50,"value":605}," raises ",{"type":44,"tag":103,"props":607,"children":609},{"className":608},[],[610],{"type":50,"value":611},"AttributeError",{"type":50,"value":613},". Fold the system prompt into the\n",{"type":44,"tag":103,"props":615,"children":617},{"className":616},[],[618],{"type":50,"value":563},{"type":50,"value":620}," message (or a ",{"type":44,"tag":103,"props":622,"children":624},{"className":623},[],[625],{"type":50,"value":626},"text_part",{"type":50,"value":628},"), and prefer the ",{"type":44,"tag":103,"props":630,"children":632},{"className":631},[],[633],{"type":50,"value":634},"user_text_message()",{"type":50,"value":636}," \u002F ",{"type":44,"tag":103,"props":638,"children":640},{"className":639},[],[641],{"type":50,"value":642},"assistant_text_message()",{"type":50,"value":644},"\n\u002F ",{"type":44,"tag":103,"props":646,"children":648},{"className":647},[],[649],{"type":50,"value":650},"tool_result_message()",{"type":50,"value":652}," helpers over hand-building ",{"type":44,"tag":103,"props":654,"children":656},{"className":655},[],[657],{"type":50,"value":658},"Message(role=...)",{"type":50,"value":660},". Always confirm enum members\nand helper names against the installed package before running — do not assume from llms.txt.",{"type":44,"tag":78,"props":662,"children":663},{},[664,669,671,677,679,685],{"type":44,"tag":59,"props":665,"children":666},{},[667],{"type":50,"value":668},"Out of scope:",{"type":50,"value":670}," offline test suites → ",{"type":44,"tag":103,"props":672,"children":674},{"className":673},[],[675],{"type":50,"value":676},"agento11y-test-starter",{"type":50,"value":678},"; tenant eval rules + guards on\nreal traffic → ",{"type":44,"tag":103,"props":680,"children":682},{"className":681},[],[683],{"type":50,"value":684},"agento11y-prod-setup",{"type":50,"value":686},". Coding-agent telemetry plugins (Claude Code, Cursor, …) →\nllms.txt \"Path A\". Any control-plane write.",{"type":44,"tag":78,"props":688,"children":689},{},[690,695],{"type":44,"tag":59,"props":691,"children":692},{},[693],{"type":50,"value":694},"If a required input is missing",{"type":50,"value":696}," (entrypoint, framework, endpoint, gcx auth), ask — don't guess.",{"type":44,"tag":141,"props":698,"children":700},{"id":699},"step-0-credentials-and-endpoint",[701],{"type":50,"value":702},"Step 0 — Credentials and endpoint",{"type":44,"tag":53,"props":704,"children":705},{},[706,708,713],{"type":50,"value":707},"The app needs, in its environment before the SDK starts, ",{"type":44,"tag":59,"props":709,"children":710},{},[711],{"type":50,"value":712},"seven",{"type":50,"value":714}," vars — not five; the two mode\nvars are the ones most often forgotten, and their absence is a silent 401:",{"type":44,"tag":148,"props":716,"children":717},{},[718,755],{"type":44,"tag":78,"props":719,"children":720},{},[721,723,728,729,735,736,742,743,748,749,754],{"type":50,"value":722},"generation ingest: ",{"type":44,"tag":103,"props":724,"children":726},{"className":725},[],[727],{"type":50,"value":190},{"type":50,"value":200},{"type":44,"tag":103,"props":730,"children":732},{"className":731},[],[733],{"type":50,"value":734},"AGENTO11Y_PROTOCOL=http",{"type":50,"value":200},{"type":44,"tag":103,"props":737,"children":739},{"className":738},[],[740],{"type":50,"value":741},"AGENTO11Y_AUTH_MODE=basic",{"type":50,"value":192},{"type":44,"tag":103,"props":744,"children":746},{"className":745},[],[747],{"type":50,"value":213},{"type":50,"value":200},{"type":44,"tag":103,"props":750,"children":752},{"className":751},[],[753],{"type":50,"value":220},{"type":50,"value":174},{"type":44,"tag":78,"props":756,"children":757},{},[758,760,765,766,771],{"type":50,"value":759},"traces\u002Fmetrics: ",{"type":44,"tag":103,"props":761,"children":763},{"className":762},[],[764],{"type":50,"value":227},{"type":50,"value":200},{"type":44,"tag":103,"props":767,"children":769},{"className":768},[],[770],{"type":50,"value":234},{"type":50,"value":174},{"type":44,"tag":53,"props":773,"children":774},{},[775,780,782,787,789,794,796,802,804,809,811,816,818,823,825,830,832,837,839,845,847,852],{"type":44,"tag":103,"props":776,"children":778},{"className":777},[],[779],{"type":50,"value":734},{"type":50,"value":781}," and ",{"type":44,"tag":103,"props":783,"children":785},{"className":784},[],[786],{"type":50,"value":741},{"type":50,"value":788}," are ",{"type":44,"tag":59,"props":790,"children":791},{},[792],{"type":50,"value":793},"required for Cloud, not optional",{"type":50,"value":795}," —\nthe SDK defaults (grpc \u002F no-auth) return a silent 401 against the Cloud HTTP ingest endpoint. (They\nscope the ingest channel only; the OTel channel's transport\u002Fauth is set entirely by the ",{"type":44,"tag":103,"props":797,"children":799},{"className":798},[],[800],{"type":50,"value":801},"OTEL_*",{"type":50,"value":803},"\nvars — see references\u002Finstrumentation.md.) The only var that is sometimes omittable is\n",{"type":44,"tag":103,"props":805,"children":807},{"className":806},[],[808],{"type":50,"value":234},{"type":50,"value":810},": ",{"type":44,"tag":59,"props":812,"children":813},{},[814],{"type":50,"value":815},"required when sending directly to the Cloud OTLP gateway",{"type":50,"value":817}," (the common\ncase, gateway enforces Basic auth), omittable ",{"type":44,"tag":59,"props":819,"children":820},{},[821],{"type":50,"value":822},"only",{"type":50,"value":824}," when ",{"type":44,"tag":103,"props":826,"children":828},{"className":827},[],[829],{"type":50,"value":227},{"type":50,"value":831}," points at a\nlocal Alloy \u002F OTel Collector that already holds the Cloud credentials. First check what's already set\n(including any existing ",{"type":44,"tag":103,"props":833,"children":835},{"className":834},[],[836],{"type":50,"value":289},{"type":50,"value":838},") — if all are present, skip to Step 1. Watch for the near-miss where\nthe endpoint is set under the wrong name (e.g. ",{"type":44,"tag":103,"props":840,"children":842},{"className":841},[],[843],{"type":50,"value":844},"AGENTO11Y_API_ENDPOINT",{"type":50,"value":846}," — the SDK reads\n",{"type":44,"tag":103,"props":848,"children":850},{"className":849},[],[851],{"type":50,"value":190},{"type":50,"value":853},", so the wrong name is silently ignored and ingest falls back to a default host).",{"type":44,"tag":855,"props":856,"children":857},"blockquote",{},[858],{"type":44,"tag":53,"props":859,"children":860},{},[861,866,868,873,874,879],{"type":44,"tag":59,"props":862,"children":863},{},[864],{"type":50,"value":865},"When any value is missing, do NOT just list the variable names and ask — hand the developer the\nexact place to get each one (link + clicks), every time.",{"type":50,"value":867}," The concrete sources are in point 3\nbelow; surface them proactively. The most common failure of this skill is naming\n",{"type":44,"tag":103,"props":869,"children":871},{"className":870},[],[872],{"type":50,"value":227},{"type":50,"value":465},{"type":44,"tag":103,"props":875,"children":877},{"className":876},[],[878],{"type":50,"value":234},{"type":50,"value":880}," and leaving the developer to guess — the\nanswer is the stack OTLP tile + \"Generate now\", which precomputes both. Give that first.",{"type":44,"tag":53,"props":882,"children":883},{},[884],{"type":44,"tag":59,"props":885,"children":886},{},[887],{"type":50,"value":888},"What gcx does for you (run these):",{"type":44,"tag":74,"props":890,"children":891},{},[892,925],{"type":44,"tag":78,"props":893,"children":894},{},[895,901,903,908,910,916,918,923],{"type":44,"tag":103,"props":896,"children":898},{"className":897},[],[899],{"type":50,"value":900},"gcx config current-context",{"type":50,"value":902}," — is there a working context? If not, ",{"type":44,"tag":59,"props":904,"children":905},{},[906],{"type":50,"value":907},"just ask the developer to log\nin to the stack they want the instrumentation to connect to",{"type":50,"value":909}," — e.g. \"run ",{"type":44,"tag":103,"props":911,"children":913},{"className":912},[],[914],{"type":50,"value":915},"gcx login",{"type":50,"value":917}," against your\nstack.\" Do not fabricate the login command yourself (don't guess the host or flags); let them run\ntheir own login (the Agent Observability setup screen gives them the exact command, or they use\n",{"type":44,"tag":103,"props":919,"children":921},{"className":920},[],[922],{"type":50,"value":915},{"type":50,"value":924},"). Instrumentation itself needs no gcx login — only Step 5 verification does, so this\nnever blocks writing the code.",{"type":44,"tag":78,"props":926,"children":927},{},[928,934,936,942],{"type":44,"tag":103,"props":929,"children":931},{"className":930},[],[932],{"type":50,"value":933},"gcx cloud stacks list",{"type":50,"value":935},", then ",{"type":44,"tag":103,"props":937,"children":939},{"className":938},[],[940],{"type":50,"value":941},"gcx cloud stacks get \u003Cstack-slug>",{"type":50,"value":943}," — identify the target stack and\nits URLs. This gives you the stack to point the developer at, and confirms which tenant the Step 5\nverification will read from.",{"type":44,"tag":53,"props":945,"children":946},{},[947],{"type":44,"tag":59,"props":948,"children":949},{},[950],{"type":50,"value":951},"What still needs the Connection page (gcx cannot do these today):",{"type":44,"tag":74,"props":953,"children":955},{"start":954},3,[956],{"type":44,"tag":78,"props":957,"children":958},{},[959,961,965,967,971,973,978,980,984,994,996,1002,1004,1009,1011,1016,1018,1023,1025,1030,1032,1044,1046,1052,1054,1057,1067,1068,1073,1075,1081,1083,1088,1090,1120,1122,1128,1130,1135,1137,1143,1145,1151,1153,1158,1159,1164,1166,1169,1171,1176,1178,1183,1185,1190,1192,1197],{"type":50,"value":960},"gcx does ",{"type":44,"tag":59,"props":962,"children":963},{},[964],{"type":50,"value":341},{"type":50,"value":966}," generate the Agent Observability OTLP gateway URL, and does ",{"type":44,"tag":59,"props":968,"children":969},{},[970],{"type":50,"value":341},{"type":50,"value":972}," mint the\ningest \u002F access-policy token. ",{"type":44,"tag":59,"props":974,"children":975},{},[976],{"type":50,"value":977},"When you ask the developer for a value, always tell them exactly\nwhere to get it — a link and the clicks — never just name the variable and wait.",{"type":50,"value":979}," The two channels\ncome from two different places:",{"type":44,"tag":981,"props":982,"children":983},"br",{},[],{"type":44,"tag":59,"props":985,"children":986},{},[987,992],{"type":44,"tag":103,"props":988,"children":990},{"className":989},[],[991],{"type":50,"value":801},{"type":50,"value":993}," (traces\u002Fmetrics) — easiest, let Cloud build them.",{"type":50,"value":995}," Send the developer to the stack's\nOTLP tile: ",{"type":44,"tag":103,"props":997,"children":999},{"className":998},[],[1000],{"type":50,"value":1001},"https:\u002F\u002Fgrafana.com\u002Forgs\u002F\u003Corg-slug>\u002Fstacks\u002F\u003Cstack-id>\u002Fotlp-info",{"type":50,"value":1003},". It already shows\n",{"type":44,"tag":103,"props":1005,"children":1007},{"className":1006},[],[1008],{"type":50,"value":227},{"type":50,"value":1010}," and the Instance ID; under ",{"type":44,"tag":59,"props":1012,"children":1013},{},[1014],{"type":50,"value":1015},"Password \u002F API Token → \"Generate now\"",{"type":50,"value":1017},"\nit mints a token and then fills an ",{"type":44,"tag":59,"props":1019,"children":1020},{},[1021],{"type":50,"value":1022},"Environment Variables",{"type":50,"value":1024}," block with all ",{"type":44,"tag":103,"props":1026,"children":1028},{"className":1027},[],[1029],{"type":50,"value":801},{"type":50,"value":1031}," vars — ",{"type":44,"tag":59,"props":1033,"children":1034},{},[1035,1037,1042],{"type":50,"value":1036},"the\nbase64 ",{"type":44,"tag":103,"props":1038,"children":1040},{"className":1039},[],[1041],{"type":50,"value":234},{"type":50,"value":1043}," is precomputed",{"type":50,"value":1045},", ready to copy. No manual base64. (In Python,\nthe value uses ",{"type":44,"tag":103,"props":1047,"children":1049},{"className":1048},[],[1050],{"type":50,"value":1051},"Basic%20…",{"type":50,"value":1053}," — keep it as given.)",{"type":44,"tag":981,"props":1055,"children":1056},{},[],{"type":44,"tag":59,"props":1058,"children":1059},{},[1060,1065],{"type":44,"tag":103,"props":1061,"children":1063},{"className":1062},[],[1064],{"type":50,"value":260},{"type":50,"value":1066}," (generation ingest) — the plugin Connection page.",{"type":50,"value":271},{"type":44,"tag":103,"props":1069,"children":1071},{"className":1070},[],[1072],{"type":50,"value":190},{"type":50,"value":1074}," and the\ntoken come from ",{"type":44,"tag":103,"props":1076,"children":1078},{"className":1077},[],[1079],{"type":50,"value":1080},"https:\u002F\u002F\u003Cstack>.grafana.net\u002Fplugins\u002Fgrafana-agento11y-app",{"type":50,"value":1082}," → Connection tab. When the\ndeveloper creates the token via ",{"type":44,"tag":59,"props":1084,"children":1085},{},[1086],{"type":50,"value":1087},"\"Create a token in Cloud Access Policies\"",{"type":50,"value":1089},", tell them the scopes:\n",{"type":44,"tag":59,"props":1091,"children":1092},{},[1093,1099,1100,1106,1107,1113,1114],{"type":44,"tag":103,"props":1094,"children":1096},{"className":1095},[],[1097],{"type":50,"value":1098},"sigil:write",{"type":50,"value":200},{"type":44,"tag":103,"props":1101,"children":1103},{"className":1102},[],[1104],{"type":50,"value":1105},"metrics:write",{"type":50,"value":200},{"type":44,"tag":103,"props":1108,"children":1110},{"className":1109},[],[1111],{"type":50,"value":1112},"traces:write",{"type":50,"value":200},{"type":44,"tag":103,"props":1115,"children":1117},{"className":1116},[],[1118],{"type":50,"value":1119},"logs:write",{"type":50,"value":1121},". UI heads-up: ",{"type":44,"tag":103,"props":1123,"children":1125},{"className":1124},[],[1126],{"type":50,"value":1127},"sigil",{"type":50,"value":1129}," is not in\nthe default resource list — add it via ",{"type":44,"tag":59,"props":1131,"children":1132},{},[1133],{"type":50,"value":1134},"\"Add scope\"",{"type":50,"value":1136}," (then tick Write); the scope is still\n",{"type":44,"tag":103,"props":1138,"children":1140},{"className":1139},[],[1141],{"type":50,"value":1142},"sigil:*",{"type":50,"value":1144}," (the Cloud resource keeps the old name). The same ",{"type":44,"tag":103,"props":1146,"children":1148},{"className":1147},[],[1149],{"type":50,"value":1150},"glc_…",{"type":50,"value":1152}," token works for both channels\nif it has all four scopes. Also set ",{"type":44,"tag":103,"props":1154,"children":1156},{"className":1155},[],[1157],{"type":50,"value":734},{"type":50,"value":781},{"type":44,"tag":103,"props":1160,"children":1162},{"className":1161},[],[1163],{"type":50,"value":741},{"type":50,"value":1165}," (see\nreferences\u002Finstrumentation.md — the SDK defaults grpc\u002Fnone give a 401).",{"type":44,"tag":981,"props":1167,"children":1168},{},[],{"type":50,"value":1170},"Ask the developer to put the values in a gitignored ",{"type":44,"tag":103,"props":1172,"children":1174},{"className":1173},[],[1175],{"type":50,"value":289},{"type":50,"value":1177}," (or export them) ",{"type":44,"tag":59,"props":1179,"children":1180},{},[1181],{"type":50,"value":1182},"themselves",{"type":50,"value":1184}," — ",{"type":44,"tag":59,"props":1186,"children":1187},{},[1188],{"type":50,"value":1189},"do\nnot ask them to paste a secret token into the chat",{"type":50,"value":1191}," (it is captured in the transcript). Instrument\nthe code to read from the environment and have them supply the values out-of-band. ",{"type":44,"tag":59,"props":1193,"children":1194},{},[1195],{"type":50,"value":1196},"Never invent a\nURL or mint a token.",{"type":44,"tag":855,"props":1198,"children":1199},{},[1200],{"type":44,"tag":53,"props":1201,"children":1202},{},[1203,1205,1211,1213,1218,1220,1224,1226,1231,1233,1238,1239,1244],{"type":50,"value":1204},"Two different tokens — don't confuse them. gcx logs in with its own OAuth token (",{"type":44,"tag":103,"props":1206,"children":1208},{"className":1207},[],[1209],{"type":50,"value":1210},"gat_",{"type":50,"value":1212},") and\nrefreshes it automatically; that is what authenticates the ",{"type":44,"tag":103,"props":1214,"children":1216},{"className":1215},[],[1217],{"type":50,"value":108},{"type":50,"value":1219}," commands here. It is ",{"type":44,"tag":59,"props":1221,"children":1222},{},[1223],{"type":50,"value":341},{"type":50,"value":1225}," the\napp's ingest token. The app needs a separate access-policy token (",{"type":44,"tag":103,"props":1227,"children":1229},{"className":1228},[],[1230],{"type":50,"value":1150},{"type":50,"value":1232},") in\n",{"type":44,"tag":103,"props":1234,"children":1236},{"className":1235},[],[1237],{"type":50,"value":220},{"type":50,"value":636},{"type":44,"tag":103,"props":1240,"children":1242},{"className":1241},[],[1243],{"type":50,"value":234},{"type":50,"value":1245},", and gcx does not create that one.",{"type":44,"tag":53,"props":1247,"children":1248},{},[1249,1251,1256],{"type":50,"value":1250},"Once gcx has a working context, the Step 5 verification commands (under the ",{"type":44,"tag":103,"props":1252,"children":1254},{"className":1253},[],[1255],{"type":50,"value":116},{"type":50,"value":1257}," group)\nwork against the developer's tenant even before the app's own credentials are fully wired.",{"type":44,"tag":855,"props":1259,"children":1260},{},[1261],{"type":44,"tag":53,"props":1262,"children":1263},{},[1264,1266,1272],{"type":50,"value":1265},"The Connection page is the only manual step. If a future gcx release can create the access-policy\ntoken and surface the OTLP endpoint, this step collapses to gcx-only — but do not assume it can\ntoday; check ",{"type":44,"tag":103,"props":1267,"children":1269},{"className":1268},[],[1270],{"type":50,"value":1271},"gcx cloud --help",{"type":50,"value":1273}," rather than promising it.",{"type":44,"tag":141,"props":1275,"children":1277},{"id":1276},"step-1-read-the-app-and-detect-language-framework-shape",[1278],{"type":50,"value":1279},"Step 1 — Read the app and detect language \u002F framework \u002F shape",{"type":44,"tag":53,"props":1281,"children":1282},{},[1283,1285,1291],{"type":50,"value":1284},"Find and read, recording ",{"type":44,"tag":103,"props":1286,"children":1288},{"className":1287},[],[1289],{"type":50,"value":1290},"file:line",{"type":50,"value":1292}," for each:",{"type":44,"tag":74,"props":1294,"children":1295},{},[1296,1301,1306,1326],{"type":44,"tag":78,"props":1297,"children":1298},{},[1299],{"type":50,"value":1300},"The generation entrypoint(s) — where the model is invoked.",{"type":44,"tag":78,"props":1302,"children":1303},{},[1304],{"type":50,"value":1305},"How the LLM client is constructed (which provider: OpenAI \u002F Anthropic \u002F Gemini \u002F other).",{"type":44,"tag":78,"props":1307,"children":1308},{},[1309,1311,1317,1318,1324],{"type":50,"value":1310},"The app bootstrap \u002F init — where an OTel ",{"type":44,"tag":103,"props":1312,"children":1314},{"className":1313},[],[1315],{"type":50,"value":1316},"TracerProvider",{"type":50,"value":636},{"type":44,"tag":103,"props":1319,"children":1321},{"className":1320},[],[1322],{"type":50,"value":1323},"MeterProvider",{"type":50,"value":1325}," would be created.",{"type":44,"tag":78,"props":1327,"children":1328},{},[1329,1331,1336,1337,1343,1345,1350,1352,1358,1359,1365,1367,1372,1374,1379],{"type":50,"value":1330},"Any existing Agent Observability SDK imports (",{"type":44,"tag":103,"props":1332,"children":1334},{"className":1333},[],[1335],{"type":50,"value":349},{"type":50,"value":636},{"type":44,"tag":103,"props":1338,"children":1340},{"className":1339},[],[1341],{"type":50,"value":1342},"@grafana\u002Fagento11y",{"type":50,"value":1344}," \u002F the Go\n",{"type":44,"tag":103,"props":1346,"children":1348},{"className":1347},[],[1349],{"type":50,"value":349},{"type":50,"value":1351}," package — or the legacy ",{"type":44,"tag":103,"props":1353,"children":1355},{"className":1354},[],[1356],{"type":50,"value":1357},"sigil_sdk",{"type":50,"value":636},{"type":44,"tag":103,"props":1360,"children":1362},{"className":1361},[],[1363],{"type":50,"value":1364},"@grafana\u002Fsigil-sdk-js",{"type":50,"value":1366}," \u002F Go ",{"type":44,"tag":103,"props":1368,"children":1370},{"className":1369},[],[1371],{"type":50,"value":1127},{"type":50,"value":1373}," names in\nolder code) or ",{"type":44,"tag":103,"props":1375,"children":1377},{"className":1376},[],[1378],{"type":50,"value":260},{"type":50,"value":1380}," usage.",{"type":44,"tag":53,"props":1382,"children":1383},{},[1384],{"type":50,"value":1385},"Detect:",{"type":44,"tag":148,"props":1387,"children":1388},{},[1389,1464,1552],{"type":44,"tag":78,"props":1390,"children":1391},{},[1392,1397,1399,1405,1406,1412,1413,1419,1420,1426,1427,1433,1434,1440,1442,1448,1449,1455,1456,1462],{"type":44,"tag":59,"props":1393,"children":1394},{},[1395],{"type":50,"value":1396},"Language",{"type":50,"value":1398}," — from the manifest \u002F extensions (",{"type":44,"tag":103,"props":1400,"children":1402},{"className":1401},[],[1403],{"type":50,"value":1404},"pyproject.toml",{"type":50,"value":465},{"type":44,"tag":103,"props":1407,"children":1409},{"className":1408},[],[1410],{"type":50,"value":1411},".py",{"type":50,"value":200},{"type":44,"tag":103,"props":1414,"children":1416},{"className":1415},[],[1417],{"type":50,"value":1418},"package.json",{"type":50,"value":465},{"type":44,"tag":103,"props":1421,"children":1423},{"className":1422},[],[1424],{"type":50,"value":1425},".ts",{"type":50,"value":192},{"type":44,"tag":103,"props":1428,"children":1430},{"className":1429},[],[1431],{"type":50,"value":1432},"go.mod",{"type":50,"value":465},{"type":44,"tag":103,"props":1435,"children":1437},{"className":1436},[],[1438],{"type":50,"value":1439},".go",{"type":50,"value":1441},", gradle\u002F",{"type":44,"tag":103,"props":1443,"children":1445},{"className":1444},[],[1446],{"type":50,"value":1447},".java",{"type":50,"value":200},{"type":44,"tag":103,"props":1450,"children":1452},{"className":1451},[],[1453],{"type":50,"value":1454},".csproj",{"type":50,"value":465},{"type":44,"tag":103,"props":1457,"children":1459},{"className":1458},[],[1460],{"type":50,"value":1461},".cs",{"type":50,"value":1463},").",{"type":44,"tag":78,"props":1465,"children":1466},{},[1467,1472,1474,1480,1481,1487,1488,1494,1495,1501,1502,1508,1509,1515,1516,1522,1523,1529,1530,1536,1537,1543,1544,1550],{"type":44,"tag":59,"props":1468,"children":1469},{},[1470],{"type":50,"value":1471},"Framework",{"type":50,"value":1473}," — grep for ",{"type":44,"tag":103,"props":1475,"children":1477},{"className":1476},[],[1478],{"type":50,"value":1479},"langgraph",{"type":50,"value":200},{"type":44,"tag":103,"props":1482,"children":1484},{"className":1483},[],[1485],{"type":50,"value":1486},"langchain",{"type":50,"value":200},{"type":44,"tag":103,"props":1489,"children":1491},{"className":1490},[],[1492],{"type":50,"value":1493},"openai-agents",{"type":50,"value":200},{"type":44,"tag":103,"props":1496,"children":1498},{"className":1497},[],[1499],{"type":50,"value":1500},"llamaindex",{"type":50,"value":200},{"type":44,"tag":103,"props":1503,"children":1505},{"className":1504},[],[1506],{"type":50,"value":1507},"google-adk",{"type":50,"value":192},{"type":44,"tag":103,"props":1510,"children":1512},{"className":1511},[],[1513],{"type":50,"value":1514},"strands",{"type":50,"value":200},{"type":44,"tag":103,"props":1517,"children":1519},{"className":1518},[],[1520],{"type":50,"value":1521},"pydantic-ai",{"type":50,"value":200},{"type":44,"tag":103,"props":1524,"children":1526},{"className":1525},[],[1527],{"type":50,"value":1528},"litellm",{"type":50,"value":200},{"type":44,"tag":103,"props":1531,"children":1533},{"className":1532},[],[1534],{"type":50,"value":1535},"claude-agent-sdk",{"type":50,"value":200},{"type":44,"tag":103,"props":1538,"children":1540},{"className":1539},[],[1541],{"type":50,"value":1542},"vercel-ai-sdk",{"type":50,"value":200},{"type":44,"tag":103,"props":1545,"children":1547},{"className":1546},[],[1548],{"type":50,"value":1549},"crewai",{"type":50,"value":1551},", or a custom\norchestrator.",{"type":44,"tag":78,"props":1553,"children":1554},{},[1555,1560],{"type":44,"tag":59,"props":1556,"children":1557},{},[1558],{"type":50,"value":1559},"Shape",{"type":50,"value":1561}," — single generation vs agentic pipeline (multiple nodes \u002F a graph \u002F sub-agents). This\ndecides whether workflow steps (checklist #8) and parent links (#7) are in play.",{"type":44,"tag":53,"props":1563,"children":1564},{},[1565],{"type":50,"value":1566},"Before recommending a provider wrapper or framework adapter, confirm it exists for this language\n(fetch llms.txt \"SDK API surface\" — the matrix is not symmetric across languages).",{"type":44,"tag":141,"props":1568,"children":1570},{"id":1569},"step-2-classify-instrumentation-state",[1571],{"type":50,"value":1572},"Step 2 — Classify instrumentation state",{"type":44,"tag":53,"props":1574,"children":1575},{},[1576,1578,1583],{"type":50,"value":1577},"State the classification explicitly, with the ",{"type":44,"tag":103,"props":1579,"children":1581},{"className":1580},[],[1582],{"type":50,"value":1290},{"type":50,"value":1584}," evidence that led to it:",{"type":44,"tag":148,"props":1586,"children":1587},{},[1588,1604,1625],{"type":44,"tag":78,"props":1589,"children":1590},{},[1591,1602],{"type":44,"tag":59,"props":1592,"children":1593},{},[1594,1600],{"type":44,"tag":103,"props":1595,"children":1597},{"className":1596},[],[1598],{"type":50,"value":1599},"none",{"type":50,"value":1601}," (greenfield)",{"type":50,"value":1603}," — no SDK import anywhere. Full setup from scratch.",{"type":44,"tag":78,"props":1605,"children":1606},{},[1607,1616,1618,1623],{"type":44,"tag":59,"props":1608,"children":1609},{},[1610],{"type":44,"tag":103,"props":1611,"children":1613},{"className":1612},[],[1614],{"type":50,"value":1615},"partial",{"type":50,"value":1617}," — an SDK client is constructed and some generations are wrapped, but coverage is\nincomplete (no OTel providers, no ",{"type":44,"tag":103,"props":1619,"children":1621},{"className":1620},[],[1622],{"type":50,"value":536},{"type":50,"value":1624},", no workflow steps for an agentic pipeline, no\nparent links). Run the full checklist; recommend + apply only the gaps.",{"type":44,"tag":78,"props":1626,"children":1627},{},[1628,1637,1639,1644,1646,1652,1653,1659,1661,1667,1668,1674],{"type":44,"tag":59,"props":1629,"children":1630},{},[1631],{"type":44,"tag":103,"props":1632,"children":1634},{"className":1633},[],[1635],{"type":50,"value":1636},"broken",{"type":50,"value":1638}," — the SDK is present but wrong: metrics silently lost (no MeterProvider), export\nmisconfigured, legacy ",{"type":44,"tag":103,"props":1640,"children":1642},{"className":1641},[],[1643],{"type":50,"value":268},{"type":50,"value":1645}," vars, stream\u002Fnon-stream mode mismatch, ",{"type":44,"tag":103,"props":1647,"children":1649},{"className":1648},[],[1650],{"type":50,"value":1651},"set_result",{"type":50,"value":465},{"type":44,"tag":103,"props":1654,"children":1656},{"className":1655},[],[1657],{"type":50,"value":1658},"SetResult",{"type":50,"value":1660},"\nnever called, or ",{"type":44,"tag":103,"props":1662,"children":1664},{"className":1663},[],[1665],{"type":50,"value":1666},"rec.err()",{"type":50,"value":465},{"type":44,"tag":103,"props":1669,"children":1671},{"className":1670},[],[1672],{"type":50,"value":1673},"Err()",{"type":50,"value":1675}," never checked. Fix first, then gap-check.",{"type":44,"tag":53,"props":1677,"children":1678},{},[1679],{"type":50,"value":1680},"All three paths converge on the same checklist (Step 3); they differ only in how much is already done.",{"type":44,"tag":141,"props":1682,"children":1684},{"id":1683},"step-3-run-the-instrumentation-gap-checklist",[1685],{"type":50,"value":1686},"Step 3 — Run the instrumentation gap checklist",{"type":44,"tag":53,"props":1688,"children":1689},{},[1690,1692,1697,1699,1704,1706,1711],{"type":50,"value":1691},"Walk each item against the code. Record PRESENT \u002F MISSING \u002F WRONG with ",{"type":44,"tag":103,"props":1693,"children":1695},{"className":1694},[],[1696],{"type":50,"value":1290},{"type":50,"value":1698},". This mechanical\naudit is the skill's core value. Items 0, 1, 2, 5, 6 fail ",{"type":44,"tag":59,"props":1700,"children":1701},{},[1702],{"type":50,"value":1703},"silently",{"type":50,"value":1705}," (no error) — always check them.\nItems 3, 7, 8 mean data lands but analysis is degraded. For the fix, read the named ",{"type":44,"tag":59,"props":1707,"children":1708},{},[1709],{"type":50,"value":1710},"section",{"type":50,"value":1712}," of\nthe fetched llms.txt (locate it by its heading — do not trust line numbers, they drift).",{"type":44,"tag":1714,"props":1715,"children":1716},"table",{},[1717,1746],{"type":44,"tag":1718,"props":1719,"children":1720},"thead",{},[1721],{"type":44,"tag":1722,"props":1723,"children":1724},"tr",{},[1725,1731,1736,1741],{"type":44,"tag":1726,"props":1727,"children":1728},"th",{},[1729],{"type":50,"value":1730},"#",{"type":44,"tag":1726,"props":1732,"children":1733},{},[1734],{"type":50,"value":1735},"Check",{"type":44,"tag":1726,"props":1737,"children":1738},{},[1739],{"type":50,"value":1740},"Silent-failure symptom",{"type":44,"tag":1726,"props":1742,"children":1743},{},[1744],{"type":50,"value":1745},"llms.txt section",{"type":44,"tag":1747,"props":1748,"children":1749},"tbody",{},[1750,1911,1946,1974,2017,2140,2174,2197,2327,2356,2387,2423,2474],{"type":44,"tag":1722,"props":1751,"children":1752},{},[1753,1759,1818,1906],{"type":44,"tag":1754,"props":1755,"children":1756},"td",{},[1757],{"type":50,"value":1758},"0",{"type":44,"tag":1754,"props":1760,"children":1761},{},[1762,1774,1776,1781,1783,1789,1791,1796,1798,1804,1805,1810,1811,1816],{"type":44,"tag":59,"props":1763,"children":1764},{},[1765,1767,1772],{"type":50,"value":1766},"The ",{"type":44,"tag":103,"props":1768,"children":1770},{"className":1769},[],[1771],{"type":50,"value":289},{"type":50,"value":1773}," actually takes effect.",{"type":50,"value":1775}," Confirm the app loads its own ",{"type":44,"tag":103,"props":1777,"children":1779},{"className":1778},[],[1780],{"type":50,"value":289},{"type":50,"value":1782}," by an explicit path (not a bare ",{"type":44,"tag":103,"props":1784,"children":1786},{"className":1785},[],[1787],{"type":50,"value":1788},"load_dotenv()",{"type":50,"value":1790}," resolved by CWD) ",{"type":44,"tag":59,"props":1792,"children":1793},{},[1794],{"type":50,"value":1795},"and",{"type":50,"value":1797}," that it wins over vars already in the environment. Verify by printing ",{"type":44,"tag":103,"props":1799,"children":1801},{"className":1800},[],[1802],{"type":50,"value":1803},"os.environ[\"AGENTO11Y_ENDPOINT\"]",{"type":50,"value":636},{"type":44,"tag":103,"props":1806,"children":1808},{"className":1807},[],[1809],{"type":50,"value":227},{"type":50,"value":271},{"type":44,"tag":59,"props":1812,"children":1813},{},[1814],{"type":50,"value":1815},"after",{"type":50,"value":1817}," all imports, not before",{"type":44,"tag":1754,"props":1819,"children":1820},{},[1821,1840,1842,1848,1849,1855,1857,1862,1864,1868,1870,1875,1877,1882,1884,1890,1892,1897,1899,1904],{"type":44,"tag":59,"props":1822,"children":1823},{},[1824,1830,1832,1838],{"type":44,"tag":103,"props":1825,"children":1827},{"className":1826},[],[1828],{"type":50,"value":1829},"import litellm",{"type":50,"value":1831}," (and some other libs) inject localhost OTLP\u002Fingest defaults into ",{"type":44,"tag":103,"props":1833,"children":1835},{"className":1834},[],[1836],{"type":50,"value":1837},"os.environ",{"type":50,"value":1839}," at import time",{"type":50,"value":1841}," (",{"type":44,"tag":103,"props":1843,"children":1845},{"className":1844},[],[1846],{"type":50,"value":1847},"OTEL_EXPORTER_OTLP_ENDPOINT=localhost:4318",{"type":50,"value":200},{"type":44,"tag":103,"props":1850,"children":1852},{"className":1851},[],[1853],{"type":50,"value":1854},"AGENTO11Y_ENDPOINT=localhost:8080",{"type":50,"value":1856},"). A plain ",{"type":44,"tag":103,"props":1858,"children":1860},{"className":1859},[],[1861],{"type":50,"value":1788},{"type":50,"value":1863}," does ",{"type":44,"tag":59,"props":1865,"children":1866},{},[1867],{"type":50,"value":341},{"type":50,"value":1869}," override already-set vars → the Cloud endpoints in ",{"type":44,"tag":103,"props":1871,"children":1873},{"className":1872},[],[1874],{"type":50,"value":289},{"type":50,"value":1876}," never apply and everything ships to localhost, returning ",{"type":44,"tag":59,"props":1878,"children":1879},{},[1880],{"type":50,"value":1881},"200 OK",{"type":50,"value":1883}," if a local stack is up. Zero error signal, and gcx against the Cloud tenant shows nothing. Fix: ",{"type":44,"tag":103,"props":1885,"children":1887},{"className":1886},[],[1888],{"type":50,"value":1889},"load_dotenv(\u003Cpath-relative-to-__file__>, override=True)",{"type":50,"value":1891}," before constructing providers\u002Fclient. A bare ",{"type":44,"tag":103,"props":1893,"children":1895},{"className":1894},[],[1896],{"type":50,"value":1788},{"type":50,"value":1898}," also resolves the wrong ",{"type":44,"tag":103,"props":1900,"children":1902},{"className":1901},[],[1903],{"type":50,"value":289},{"type":50,"value":1905}," by CWD",{"type":44,"tag":1754,"props":1907,"children":1908},{},[1909],{"type":50,"value":1910},"\"Environment\"",{"type":44,"tag":1722,"props":1912,"children":1913},{},[1914,1919,1936,1941],{"type":44,"tag":1754,"props":1915,"children":1916},{},[1917],{"type":50,"value":1918},"1",{"type":44,"tag":1754,"props":1920,"children":1921},{},[1922,1924,1928,1930,1934],{"type":50,"value":1923},"OTel TracerProvider ",{"type":44,"tag":59,"props":1925,"children":1926},{},[1927],{"type":50,"value":1795},{"type":50,"value":1929}," MeterProvider created before the SDK client (verify by construction + Performance view \u002F OTLP POSTs — ",{"type":44,"tag":59,"props":1931,"children":1932},{},[1933],{"type":50,"value":341},{"type":50,"value":1935}," via gcx, which can't see OTel; see Step 5)",{"type":44,"tag":1754,"props":1937,"children":1938},{},[1939],{"type":50,"value":1940},"spans\u002Fmetrics go to no-op → all latency\u002Ftoken\u002Fcost metrics lost. The #1 failure.",{"type":44,"tag":1754,"props":1942,"children":1943},{},[1944],{"type":50,"value":1945},"\"OTel setup (required)\"",{"type":44,"tag":1722,"props":1947,"children":1948},{},[1949,1954,1965,1970],{"type":44,"tag":1754,"props":1950,"children":1951},{},[1952],{"type":50,"value":1953},"2",{"type":44,"tag":1754,"props":1955,"children":1956},{},[1957,1959],{"type":50,"value":1958},"Providers shut down after ",{"type":44,"tag":103,"props":1960,"children":1962},{"className":1961},[],[1963],{"type":50,"value":1964},"shutdown()",{"type":44,"tag":1754,"props":1966,"children":1967},{},[1968],{"type":50,"value":1969},"last batch of spans\u002Fmetrics dropped on exit",{"type":44,"tag":1754,"props":1971,"children":1972},{},[1973],{"type":50,"value":1945},{"type":44,"tag":1722,"props":1975,"children":1976},{},[1977,1982,2000,2012],{"type":44,"tag":1754,"props":1978,"children":1979},{},[1980],{"type":50,"value":1981},"3",{"type":44,"tag":1754,"props":1983,"children":1984},{},[1985,1991,1993,1998],{"type":44,"tag":103,"props":1986,"children":1988},{"className":1987},[],[1989],{"type":50,"value":1990},"agent_name",{"type":50,"value":1992}," + ",{"type":44,"tag":103,"props":1994,"children":1996},{"className":1995},[],[1997],{"type":50,"value":536},{"type":50,"value":1999}," set on generations \u002F handlers",{"type":44,"tag":1754,"props":2001,"children":2002},{},[2003,2005,2010],{"type":50,"value":2004},"per-version Performance charts break (join on ",{"type":44,"tag":103,"props":2006,"children":2008},{"className":2007},[],[2009],{"type":50,"value":544},{"type":50,"value":2011},")",{"type":44,"tag":1754,"props":2013,"children":2014},{},[2015],{"type":50,"value":2016},"\"Agent Observability architecture and ingest model\", \"Telemetry fields to prioritize\"",{"type":44,"tag":1722,"props":2018,"children":2019},{},[2020,2025,2105,2135],{"type":44,"tag":1754,"props":2021,"children":2022},{},[2023],{"type":50,"value":2024},"4",{"type":44,"tag":1754,"props":2026,"children":2027},{},[2028,2033,2034,2039,2041,2047,2048,2053,2054,2060,2062,2090,2092,2097,2099,2104],{"type":44,"tag":103,"props":2029,"children":2031},{"className":2030},[],[2032],{"type":50,"value":1651},{"type":50,"value":465},{"type":44,"tag":103,"props":2035,"children":2037},{"className":2036},[],[2038],{"type":50,"value":1658},{"type":50,"value":2040}," includes response_id, response_model, finish\u002Fstop reason, full token usage (incl. ",{"type":44,"tag":103,"props":2042,"children":2044},{"className":2043},[],[2045],{"type":50,"value":2046},"cache_read_input_tokens",{"type":50,"value":200},{"type":44,"tag":103,"props":2049,"children":2051},{"className":2050},[],[2052],{"type":50,"value":520},{"type":50,"value":200},{"type":44,"tag":103,"props":2055,"children":2057},{"className":2056},[],[2058],{"type":50,"value":2059},"reasoning_tokens",{"type":50,"value":2061},"), ",{"type":44,"tag":59,"props":2063,"children":2064},{},[2065,2067,2073,2074,2080,2082,2088],{"type":50,"value":2066},"and ",{"type":44,"tag":103,"props":2068,"children":2070},{"className":2069},[],[2071],{"type":50,"value":2072},"input",{"type":50,"value":465},{"type":44,"tag":103,"props":2075,"children":2077},{"className":2076},[],[2078],{"type":50,"value":2079},"output",{"type":50,"value":2081}," populated with ",{"type":44,"tag":103,"props":2083,"children":2085},{"className":2084},[],[2086],{"type":50,"value":2087},"Message",{"type":50,"value":2089}," objects",{"type":50,"value":2091}," (system+user prompt in ",{"type":44,"tag":103,"props":2093,"children":2095},{"className":2094},[],[2096],{"type":50,"value":2072},{"type":50,"value":2098},", model reply in ",{"type":44,"tag":103,"props":2100,"children":2102},{"className":2101},[],[2103],{"type":50,"value":2079},{"type":50,"value":2011},{"type":44,"tag":1754,"props":2106,"children":2107},{},[2108,2110,2115,2117],{"type":50,"value":2109},"charts\u002Fcost blank; wrong ",{"type":44,"tag":103,"props":2111,"children":2113},{"className":2112},[],[2114],{"type":50,"value":528},{"type":50,"value":2116}," name silently ignored; ",{"type":44,"tag":59,"props":2118,"children":2119},{},[2120,2122,2127,2128,2133],{"type":50,"value":2121},"empty ",{"type":44,"tag":103,"props":2123,"children":2125},{"className":2124},[],[2126],{"type":50,"value":2072},{"type":50,"value":465},{"type":44,"tag":103,"props":2129,"children":2131},{"className":2130},[],[2132],{"type":50,"value":2079},{"type":50,"value":2134}," → the conversation thread shows \"No messages in this turn\" — tokens land but there is no visible prompt\u002Fresponse",{"type":44,"tag":1754,"props":2136,"children":2137},{},[2138],{"type":50,"value":2139},"\"Implementation rules\", \"Telemetry fields to prioritize\"",{"type":44,"tag":1722,"props":2141,"children":2142},{},[2143,2148,2164,2169],{"type":44,"tag":1754,"props":2144,"children":2145},{},[2146],{"type":50,"value":2147},"5",{"type":44,"tag":1754,"props":2149,"children":2150},{},[2151,2156,2157,2162],{"type":44,"tag":103,"props":2152,"children":2154},{"className":2153},[],[2155],{"type":50,"value":1666},{"type":50,"value":465},{"type":44,"tag":103,"props":2158,"children":2160},{"className":2159},[],[2161],{"type":50,"value":1673},{"type":50,"value":2163}," checked after the recorder closes",{"type":44,"tag":1754,"props":2165,"children":2166},{},[2167],{"type":50,"value":2168},"SDK validation\u002Fenqueue errors are silent → generations never arrive, no signal",{"type":44,"tag":1754,"props":2170,"children":2171},{},[2172],{"type":50,"value":2173},"\"Implementation rules\"",{"type":44,"tag":1722,"props":2175,"children":2176},{},[2177,2182,2187,2192],{"type":44,"tag":1754,"props":2178,"children":2179},{},[2180],{"type":50,"value":2181},"6",{"type":44,"tag":1754,"props":2183,"children":2184},{},[2185],{"type":50,"value":2186},"SYNC (non-stream) vs STREAM (stream) set correctly",{"type":44,"tag":1754,"props":2188,"children":2189},{},[2190],{"type":50,"value":2191},"streaming metrics (TTFT) corrupted",{"type":44,"tag":1754,"props":2193,"children":2194},{},[2195],{"type":50,"value":2196},"\"Agent Observability architecture and ingest model\", \"Implementation rules\"",{"type":44,"tag":1722,"props":2198,"children":2199},{},[2200,2205,2280,2323],{"type":44,"tag":1754,"props":2201,"children":2202},{},[2203],{"type":50,"value":2204},"6b",{"type":44,"tag":1754,"props":2206,"children":2207},{},[2208,2214,2216,2221,2223,2229,2231,2237,2239,2245,2246,2252,2253,2259,2260,2266,2268,2272,2274],{"type":44,"tag":103,"props":2209,"children":2211},{"className":2210},[],[2212],{"type":50,"value":2213},"operation_name",{"type":50,"value":2215}," is a ",{"type":44,"tag":59,"props":2217,"children":2218},{},[2219],{"type":50,"value":2220},"recognized",{"type":50,"value":2222}," value — ",{"type":44,"tag":103,"props":2224,"children":2226},{"className":2225},[],[2227],{"type":50,"value":2228},"generateText",{"type":50,"value":2230}," (SYNC default), ",{"type":44,"tag":103,"props":2232,"children":2234},{"className":2233},[],[2235],{"type":50,"value":2236},"streamText",{"type":50,"value":2238}," (STREAM default), ",{"type":44,"tag":103,"props":2240,"children":2242},{"className":2241},[],[2243],{"type":50,"value":2244},"embeddings",{"type":50,"value":200},{"type":44,"tag":103,"props":2247,"children":2249},{"className":2248},[],[2250],{"type":50,"value":2251},"execute_tool",{"type":50,"value":200},{"type":44,"tag":103,"props":2254,"children":2256},{"className":2255},[],[2257],{"type":50,"value":2258},"framework_chain",{"type":50,"value":200},{"type":44,"tag":103,"props":2261,"children":2263},{"className":2262},[],[2264],{"type":50,"value":2265},"framework_retriever",{"type":50,"value":2267},". Best: omit it and take the SDK default. Do ",{"type":44,"tag":59,"props":2269,"children":2270},{},[2271],{"type":50,"value":341},{"type":50,"value":2273}," invent one like ",{"type":44,"tag":103,"props":2275,"children":2277},{"className":2276},[],[2278],{"type":50,"value":2279},"\"chat\"",{"type":44,"tag":1754,"props":2281,"children":2282},{},[2283,2285,2291,2293,2299,2301,2306,2308,2314,2315,2321],{"type":50,"value":2284},"the span reaches Tempo but the UI classifies ",{"type":44,"tag":103,"props":2286,"children":2288},{"className":2287},[],[2289],{"type":50,"value":2290},"gen_ai.operation.name",{"type":50,"value":2292}," as ",{"type":44,"tag":103,"props":2294,"children":2296},{"className":2295},[],[2297],{"type":50,"value":2298},"unknown",{"type":50,"value":2300}," → the conversation renders a synthetic generation node ",{"type":44,"tag":59,"props":2302,"children":2303},{},[2304],{"type":50,"value":2305},"with no attached span",{"type":50,"value":2307}," → the trace does not show in the conversation and the \"T\" (trace) icon is absent, even though ",{"type":44,"tag":103,"props":2309,"children":2311},{"className":2310},[],[2312],{"type":50,"value":2313},"trace_id",{"type":50,"value":465},{"type":44,"tag":103,"props":2316,"children":2318},{"className":2317},[],[2319],{"type":50,"value":2320},"span_id",{"type":50,"value":2322}," are set. Silent, like #1",{"type":44,"tag":1754,"props":2324,"children":2325},{},[2326],{"type":50,"value":2196},{"type":44,"tag":1722,"props":2328,"children":2329},{},[2330,2335,2346,2351],{"type":44,"tag":1754,"props":2331,"children":2332},{},[2333],{"type":50,"value":2334},"7",{"type":44,"tag":1754,"props":2336,"children":2337},{},[2338,2344],{"type":44,"tag":103,"props":2339,"children":2341},{"className":2340},[],[2342],{"type":50,"value":2343},"parent_generation_ids",{"type":50,"value":2345}," set on multi-agent \u002F fan-in generations",{"type":44,"tag":1754,"props":2347,"children":2348},{},[2349],{"type":50,"value":2350},"no dependency DAG; upstream eval failures don't propagate",{"type":44,"tag":1754,"props":2352,"children":2353},{},[2354],{"type":50,"value":2355},"\"Multi-agent dependency tracking\"",{"type":44,"tag":1722,"props":2357,"children":2358},{},[2359,2364,2369,2382],{"type":44,"tag":1754,"props":2360,"children":2361},{},[2362],{"type":50,"value":2363},"8",{"type":44,"tag":1754,"props":2365,"children":2366},{},[2367],{"type":50,"value":2368},"Workflow steps emitted for agentic pipelines with non-LLM nodes",{"type":44,"tag":1754,"props":2370,"children":2371},{},[2372,2374,2380],{"type":50,"value":2373},"execution graph invisible; node input\u002Foutput state lost. Use the adapter if one exists, else ",{"type":44,"tag":103,"props":2375,"children":2377},{"className":2376},[],[2378],{"type":50,"value":2379},"enqueue_workflow_step",{"type":50,"value":2381},"; never both for one node (duplicates)",{"type":44,"tag":1754,"props":2383,"children":2384},{},[2385],{"type":50,"value":2386},"\"Workflow step instrumentation (agentic pipelines)\"",{"type":44,"tag":1722,"props":2388,"children":2389},{},[2390,2395,2414,2419],{"type":44,"tag":1754,"props":2391,"children":2392},{},[2393],{"type":50,"value":2394},"9",{"type":44,"tag":1754,"props":2396,"children":2397},{},[2398,2400,2405,2407,2412],{"type":50,"value":2399},"Env vars are ",{"type":44,"tag":103,"props":2401,"children":2403},{"className":2402},[],[2404],{"type":50,"value":260},{"type":50,"value":2406}," (not legacy ",{"type":44,"tag":103,"props":2408,"children":2410},{"className":2409},[],[2411],{"type":50,"value":268},{"type":50,"value":2413},"); client built config-free when env present",{"type":44,"tag":1754,"props":2415,"children":2416},{},[2417],{"type":50,"value":2418},"drift; duplicated config",{"type":44,"tag":1754,"props":2420,"children":2421},{},[2422],{"type":50,"value":1910},{"type":44,"tag":1722,"props":2424,"children":2425},{},[2426,2431,2464,2469],{"type":44,"tag":1754,"props":2427,"children":2428},{},[2429],{"type":50,"value":2430},"10",{"type":44,"tag":1754,"props":2432,"children":2433},{},[2434,2436,2442,2444,2450,2451,2457,2458],{"type":50,"value":2435},"Content-capture mode intentional (SDK default ",{"type":44,"tag":103,"props":2437,"children":2439},{"className":2438},[],[2440],{"type":50,"value":2441},"no_tool_content",{"type":50,"value":2443},"); no secrets in ",{"type":44,"tag":103,"props":2445,"children":2447},{"className":2446},[],[2448],{"type":50,"value":2449},"tags",{"type":50,"value":465},{"type":44,"tag":103,"props":2452,"children":2454},{"className":2453},[],[2455],{"type":50,"value":2456},"metadata",{"type":50,"value":465},{"type":44,"tag":103,"props":2459,"children":2461},{"className":2460},[],[2462],{"type":50,"value":2463},"user_id",{"type":44,"tag":1754,"props":2465,"children":2466},{},[2467],{"type":50,"value":2468},"PII\u002Fsecrets leak into exports",{"type":44,"tag":1754,"props":2470,"children":2471},{},[2472],{"type":50,"value":2473},"\"Content capture\", \"Tags, metadata, and user id\"",{"type":44,"tag":1722,"props":2475,"children":2476},{},[2477,2482,2494,2499],{"type":44,"tag":1754,"props":2478,"children":2479},{},[2480],{"type":50,"value":2481},"11",{"type":44,"tag":1754,"props":2483,"children":2484},{},[2485,2487,2492],{"type":50,"value":2486},"Client tags low-cardinality; end-user identity via ",{"type":44,"tag":103,"props":2488,"children":2490},{"className":2489},[],[2491],{"type":50,"value":2463},{"type":50,"value":2493},", not a tag",{"type":44,"tag":1754,"props":2495,"children":2496},{},[2497],{"type":50,"value":2498},"high-cardinality tags blow up metric labels",{"type":44,"tag":1754,"props":2500,"children":2501},{},[2502],{"type":50,"value":2503},"\"Tags, metadata, and user id\", \"Implementation rules\"",{"type":44,"tag":141,"props":2505,"children":2507},{"id":2506},"step-4-recommend-the-report",[2508],{"type":50,"value":2509},"Step 4 — Recommend (the report)",{"type":44,"tag":53,"props":2511,"children":2512},{},[2513,2515,2520,2522,2527,2529,2534,2536,2541],{"type":50,"value":2514},"Emit the report using llms.txt's output contract (its \"Output contract\" section): top opportunities\nfirst, and per opportunity — exact ",{"type":44,"tag":103,"props":2516,"children":2518},{"className":2517},[],[2519],{"type":50,"value":1290},{"type":50,"value":2521},", why it matters, a concrete diff proposal, a test\nplan, and any risk. Rank by severity: ",{"type":44,"tag":103,"props":2523,"children":2525},{"className":2524},[],[2526],{"type":50,"value":289},{"type":50,"value":2528}," not taking effect (#0 — nothing lands at all) first,\nthen missing OTel provider (metrics data loss), then broken export, then missing ",{"type":44,"tag":103,"props":2530,"children":2532},{"className":2531},[],[2533],{"type":50,"value":536},{"type":50,"value":2535},",\nthen coverage gaps. Every recommendation cites a ",{"type":44,"tag":103,"props":2537,"children":2539},{"className":2538},[],[2540],{"type":50,"value":1290},{"type":50,"value":2542},". Then stop and ask\nbefore applying anything.",{"type":44,"tag":141,"props":2544,"children":2546},{"id":2545},"step-5-apply-then-verify-the-loop",[2547],{"type":50,"value":2548},"Step 5 — Apply, then verify (the loop)",{"type":44,"tag":53,"props":2550,"children":2551},{},[2552],{"type":50,"value":2553},"Only after the developer confirms a diff. Bounded to ~3–4 iterations.",{"type":44,"tag":74,"props":2555,"children":2556},{},[2557,2574,2597,3035],{"type":44,"tag":78,"props":2558,"children":2559},{},[2560,2565,2567,2573],{"type":44,"tag":59,"props":2561,"children":2562},{},[2563],{"type":50,"value":2564},"Apply",{"type":50,"value":2566}," the confirmed diff (Edit\u002FWrite). Order of preference: provider wrapper → framework\nadapter → hand-instrument the core SDK — only what exists for the language. Add\u002Fupdate a focused\ntest for the changed instrumentation. Preserve flush\u002Fshutdown lifecycle. Never touch app logic.\nPull exact usage from llms.txt \u002F the per-language README \u002F ",{"type":44,"tag":103,"props":2568,"children":2570},{"className":2569},[],[2571],{"type":50,"value":2572},"examples\u002Fgetting-started\u002F*",{"type":50,"value":174},{"type":44,"tag":78,"props":2575,"children":2576},{},[2577,2582,2584,2589,2591,2595],{"type":44,"tag":59,"props":2578,"children":2579},{},[2580],{"type":50,"value":2581},"Run the app",{"type":50,"value":2583}," for one turn to generate traffic (ask the developer to run it, or run it if\nthere's a clean entrypoint and they approve). ",{"type":44,"tag":59,"props":2585,"children":2586},{},[2587],{"type":50,"value":2588},"If the run can't happen",{"type":50,"value":2590}," — missing provider API\nkey, no clean entrypoint, needs a full runtime — do ",{"type":44,"tag":59,"props":2592,"children":2593},{},[2594],{"type":50,"value":341},{"type":50,"value":2596}," work around it by changing the model\nor adding a provider. Stop the loop, report the wiring as verified-by-construction (imports\nresolve, providers build, client + handler construct), and tell the developer the one thing left\nis to run one turn themselves with their key. A verified-by-construction result is a fine outcome.",{"type":44,"tag":78,"props":2598,"children":2599},{},[2600,2605,2607,2612,2614,2617,2622,2624,2630,2632,2637,2638,2643,2645,2650,2652,2657,2659,2665,2667,2672,2674,2677,2682,2684,2690,2692,2808,2811,2816,2818,2824,2825,2831,2833,2838,2840,2845,2847],{"type":44,"tag":59,"props":2601,"children":2602},{},[2603],{"type":50,"value":2604},"Verify — two independent channels, don't conflate them.",{"type":50,"value":2606}," Instrumentation sends data on two\nseparate paths, and confirming one says ",{"type":44,"tag":59,"props":2608,"children":2609},{},[2610],{"type":50,"value":2611},"nothing",{"type":50,"value":2613}," about the other.",{"type":44,"tag":981,"props":2615,"children":2616},{},[],{"type":44,"tag":59,"props":2618,"children":2619},{},[2620],{"type":50,"value":2621},"First, confirm gcx reads the same tenant the app writes to.",{"type":50,"value":2623}," A verification against the wrong\ntenant is worse than no verification — an empty ",{"type":44,"tag":103,"props":2625,"children":2627},{"className":2626},[],[2628],{"type":50,"value":2629},"agents list",{"type":50,"value":2631}," gets misread as \"data isn't landing\"\nwhen it is, just elsewhere. Before drawing any conclusion from a gcx query: read the app's\n",{"type":44,"tag":103,"props":2633,"children":2635},{"className":2634},[],[2636],{"type":50,"value":190},{"type":50,"value":1992},{"type":44,"tag":103,"props":2639,"children":2641},{"className":2640},[],[2642],{"type":50,"value":213},{"type":50,"value":2644}," from its ",{"type":44,"tag":103,"props":2646,"children":2648},{"className":2647},[],[2649],{"type":50,"value":289},{"type":50,"value":2651},", then check ",{"type":44,"tag":103,"props":2653,"children":2655},{"className":2654},[],[2656],{"type":50,"value":900},{"type":50,"value":2658}," and that the active context points at that same stack\u002Ftenant. If it doesn't (e.g.\ncontext is ",{"type":44,"tag":103,"props":2660,"children":2662},{"className":2661},[],[2663],{"type":50,"value":2664},"local",{"type":50,"value":2666}," but the app writes to a Cloud stack), switch context or ask the developer to\n",{"type":44,"tag":103,"props":2668,"children":2670},{"className":2669},[],[2671],{"type":50,"value":915},{"type":50,"value":2673}," to the right stack — do not guess the login command. (If the gcx token is merely\nexpired, that blocks Step 5 verification only, not writing the code — say so and continue.)",{"type":44,"tag":981,"props":2675,"children":2676},{},[],{"type":44,"tag":59,"props":2678,"children":2679},{},[2680],{"type":50,"value":2681},"Channel A — generations",{"type":50,"value":2683}," (the SDK ingest client → ",{"type":44,"tag":103,"props":2685,"children":2687},{"className":2686},[],[2688],{"type":50,"value":2689},"\u002Fapi\u002Fv1\u002Fgenerations:export",{"type":50,"value":2691},"). Carries the\nprompt, response, tokens, cost, model, finish_reason. This is what gcx can read.",{"type":44,"tag":148,"props":2693,"children":2694},{},[2695],{"type":44,"tag":78,"props":2696,"children":2697},{},[2698,2703,2704,2710,2712,2718,2720,2726,2728,2734,2736,2742,2743,2749,2751,2757,2758,2764,2766,2771,2773,2779,2781,2787,2789,2794,2796,2801,2803],{"type":44,"tag":59,"props":2699,"children":2700},{},[2701],{"type":50,"value":2702},"Via gcx:",{"type":50,"value":271},{"type":44,"tag":103,"props":2705,"children":2707},{"className":2706},[],[2708],{"type":50,"value":2709},"gcx agento11y agents list",{"type":50,"value":2711}," (does the agent appear?);\n",{"type":44,"tag":103,"props":2713,"children":2715},{"className":2714},[],[2716],{"type":50,"value":2717},"gcx agento11y agents get \u003Cagent-name>",{"type":50,"value":2719}," (is ",{"type":44,"tag":103,"props":2721,"children":2723},{"className":2722},[],[2724],{"type":50,"value":2725},"generation_count",{"type":50,"value":2727}," climbing?). To find the run's\nconversation, either ",{"type":44,"tag":103,"props":2729,"children":2731},{"className":2730},[],[2732],{"type":50,"value":2733},"gcx agento11y conversations list --limit \u003Cn>",{"type":50,"value":2735}," (most-recent first, no\nfilters — the quickest post-run check) or ",{"type":44,"tag":103,"props":2737,"children":2739},{"className":2738},[],[2740],{"type":50,"value":2741},"gcx agento11y conversations search --filters 'agent = \"\u003Cagent-name>\"'",{"type":50,"value":1841},{"type":44,"tag":103,"props":2744,"children":2746},{"className":2745},[],[2747],{"type":50,"value":2748},"--filters",{"type":50,"value":2750}," alone is enough; ",{"type":44,"tag":103,"props":2752,"children":2754},{"className":2753},[],[2755],{"type":50,"value":2756},"--from",{"type":50,"value":465},{"type":44,"tag":103,"props":2759,"children":2761},{"className":2760},[],[2762],{"type":50,"value":2763},"--to",{"type":50,"value":2765},", RFC3339, are ",{"type":44,"tag":59,"props":2767,"children":2768},{},[2769],{"type":50,"value":2770},"optional",{"type":50,"value":2772},"\nand only needed to narrow a busy tenant). Then ",{"type":44,"tag":103,"props":2774,"children":2776},{"className":2775},[],[2777],{"type":50,"value":2778},"gcx agento11y conversations get \u003Cconversation-id>",{"type":50,"value":2780},"\nor ",{"type":44,"tag":103,"props":2782,"children":2784},{"className":2783},[],[2785],{"type":50,"value":2786},"gcx agento11y generations get \u003Cgeneration-id>",{"type":50,"value":2788}," — check tokens, finish reason, cost, and (for\na multi-agent pipeline) that ",{"type":44,"tag":103,"props":2790,"children":2792},{"className":2791},[],[2793],{"type":50,"value":2343},{"type":50,"value":2795}," reproduce the DAG. This proves generation\ningest + ",{"type":44,"tag":103,"props":2797,"children":2799},{"className":2798},[],[2800],{"type":50,"value":1651},{"type":50,"value":2802}," are wired. ",{"type":44,"tag":59,"props":2804,"children":2805},{},[2806],{"type":50,"value":2807},"It does NOT prove OTel is wired.",{"type":44,"tag":981,"props":2809,"children":2810},{},[],{"type":44,"tag":59,"props":2812,"children":2813},{},[2814],{"type":50,"value":2815},"Channel B — OTel spans\u002Fmetrics",{"type":50,"value":2817}," (the TracerProvider\u002FMeterProvider → OTLP exporter →\n",{"type":44,"tag":103,"props":2819,"children":2821},{"className":2820},[],[2822],{"type":50,"value":2823},"\u002Fv1\u002Ftraces",{"type":50,"value":200},{"type":44,"tag":103,"props":2826,"children":2828},{"className":2827},[],[2829],{"type":50,"value":2830},"\u002Fv1\u002Fmetrics",{"type":50,"value":2832},"). Carries latency\u002Ftoken\u002Fcost ",{"type":44,"tag":59,"props":2834,"children":2835},{},[2836],{"type":50,"value":2837},"metrics",{"type":50,"value":2839},". This is checklist #1, the\n#1 silent failure — and ",{"type":44,"tag":59,"props":2841,"children":2842},{},[2843],{"type":50,"value":2844},"gcx cannot see it",{"type":50,"value":2846}," (traces\u002Fmetrics land in the stack's Tempo\u002FPrometheus,\nnot the Agent Observability ingest API). So verify Channel B separately, by the strongest signal\navailable, in this order:",{"type":44,"tag":148,"props":2848,"children":2849},{},[2850,2873,2950,2997],{"type":44,"tag":78,"props":2851,"children":2852},{},[2853,2858,2860,2864,2866,2871],{"type":44,"tag":59,"props":2854,"children":2855},{},[2856],{"type":50,"value":2857},"By construction (always do this):",{"type":50,"value":2859}," confirm in the applied code that both a TracerProvider\n",{"type":44,"tag":59,"props":2861,"children":2862},{},[2863],{"type":50,"value":1795},{"type":50,"value":2865}," a MeterProvider are created ",{"type":44,"tag":383,"props":2867,"children":2868},{},[2869],{"type":50,"value":2870},"before",{"type":50,"value":2872}," the SDK client and shut down after it. This is\nstatic but reliable — a missing\u002Flate\u002Fno-op MeterProvider is exactly checklist #1.",{"type":44,"tag":78,"props":2874,"children":2875},{},[2876,2881,2883,2888,2890,2895,2896,2902,2903,2907,2913,2915,2921,2922,2928,2930,2942,2944,2948],{"type":44,"tag":59,"props":2877,"children":2878},{},[2879],{"type":50,"value":2880},"At runtime, if you can observe it:",{"type":50,"value":2882}," run the ",{"type":44,"tag":59,"props":2884,"children":2885},{},[2886],{"type":50,"value":2887},"real app",{"type":50,"value":2889}," (not an isolated probe script) with\nthe OTel\u002Furllib3 debug log on, and confirm you see ",{"type":44,"tag":59,"props":2891,"children":2892},{},[2893],{"type":50,"value":2894},"both",{"type":50,"value":271},{"type":44,"tag":103,"props":2897,"children":2899},{"className":2898},[],[2900],{"type":50,"value":2901},"POST …\u002Fv1\u002Ftraces → 2xx",{"type":50,"value":271},{"type":44,"tag":59,"props":2904,"children":2905},{},[2906],{"type":50,"value":1795},{"type":44,"tag":103,"props":2908,"children":2910},{"className":2909},[],[2911],{"type":50,"value":2912},"POST …\u002Fv1\u002Fmetrics → 2xx",{"type":50,"value":2914},". The SDK emits spans automatically from ",{"type":44,"tag":103,"props":2916,"children":2918},{"className":2917},[],[2919],{"type":50,"value":2920},"start_generation",{"type":50,"value":465},{"type":44,"tag":103,"props":2923,"children":2925},{"className":2924},[],[2926],{"type":50,"value":2927},"end",{"type":50,"value":2929}," (one\ntraces POST per generation) and metrics on an interval — a clean shutdown flush surfaces both.\n",{"type":44,"tag":59,"props":2931,"children":2932},{},[2933,2935,2940],{"type":50,"value":2934},"Traces and metrics are separate exports: seeing only ",{"type":44,"tag":103,"props":2936,"children":2938},{"className":2937},[],[2939],{"type":50,"value":2830},{"type":50,"value":2941}," does NOT mean traces work,\nand vice-versa. Never claim \"traces\u002Fmetrics verified\" from a probe that only exercised one of\nthem",{"type":50,"value":2943}," — that is the exact trap that reports Channel B as done when half of it was never sent. If\nyou write a throwaway verification script, it must build the TracerProvider ",{"type":44,"tag":59,"props":2945,"children":2946},{},[2947],{"type":50,"value":1795},{"type":50,"value":2949}," MeterProvider\nand record a real generation, or just instrument the app and read its debug output.",{"type":44,"tag":78,"props":2951,"children":2952},{},[2953,2958,2960,2965,2967,2972,2974,2979,2981,2987,2989,2995],{"type":44,"tag":59,"props":2954,"children":2955},{},[2956],{"type":50,"value":2957},"In the UI:",{"type":50,"value":2959}," the stack's ",{"type":44,"tag":59,"props":2961,"children":2962},{},[2963],{"type":50,"value":2964},"Performance \u002F metrics",{"type":50,"value":2966}," view populates from metrics; ",{"type":44,"tag":59,"props":2968,"children":2969},{},[2970],{"type":50,"value":2971},"traces",{"type":50,"value":2973}," land\nin the stack's ",{"type":44,"tag":59,"props":2975,"children":2976},{},[2977],{"type":50,"value":2978},"Tempo",{"type":50,"value":2980}," (Explore → Tempo, filter by ",{"type":44,"tag":103,"props":2982,"children":2984},{"className":2983},[],[2985],{"type":50,"value":2986},"service.name",{"type":50,"value":2988},"). If conversations appear\n(Channel A) but Performance is empty, the MeterProvider is missing or no-op → back to checklist\n#1. Do not report OTel as wired on the strength of ",{"type":44,"tag":103,"props":2990,"children":2992},{"className":2991},[],[2993],{"type":50,"value":2994},"generations get",{"type":50,"value":2996}," alone, nor on metrics alone.",{"type":44,"tag":78,"props":2998,"children":2999},{},[3000,3005,3007,3012,3014,3019,3021,3026,3028,3033],{"type":44,"tag":59,"props":3001,"children":3002},{},[3003],{"type":50,"value":3004},"Trace shows in Tempo but NOT inside the conversation (no \"T\" icon):",{"type":50,"value":3006}," the span is landing but\n",{"type":44,"tag":103,"props":3008,"children":3010},{"className":3009},[],[3011],{"type":50,"value":2290},{"type":50,"value":3013}," is an unrecognized value (e.g. ",{"type":44,"tag":103,"props":3015,"children":3017},{"className":3016},[],[3018],{"type":50,"value":2279},{"type":50,"value":3020},") → the UI classifies it as\n",{"type":44,"tag":103,"props":3022,"children":3024},{"className":3023},[],[3025],{"type":50,"value":2298},{"type":50,"value":3027}," and can't attach it to the conversation node. This is checklist #6b — fix\n",{"type":44,"tag":103,"props":3029,"children":3031},{"className":3030},[],[3032],{"type":50,"value":2213},{"type":50,"value":3034}," to a recognized value (or omit it for the default) and re-run.",{"type":44,"tag":78,"props":3036,"children":3037},{},[3038,3040,3046],{"type":50,"value":3039},"If a signal is missing, diagnose the next gap from what the checks showed, propose the fix, and\nloop back to step 1. After ~3–4 iterations without full signal, stop and report exactly what\nlands, what doesn't, and what to check next (app stderr for ",{"type":44,"tag":103,"props":3041,"children":3043},{"className":3042},[],[3044],{"type":50,"value":3045},"agento11y:",{"type":50,"value":3047}," warnings, credentials).",{"type":44,"tag":141,"props":3049,"children":3051},{"id":3050},"step-6-hand-off",[3052],{"type":50,"value":3053},"Step 6 — Hand off",{"type":44,"tag":53,"props":3055,"children":3056},{},[3057,3059,3064,3066,3071,3073,3077,3079,3084,3086,3091],{"type":50,"value":3058},"Once generations land and metrics populate, instrumentation is done — that's the prerequisite for\neverything else. Point the developer at the next step: ",{"type":44,"tag":103,"props":3060,"children":3062},{"className":3061},[],[3063],{"type":50,"value":676},{"type":50,"value":3065}," to build an\n",{"type":44,"tag":59,"props":3067,"children":3068},{},[3069],{"type":50,"value":3070},"offline test suite",{"type":50,"value":3072}," for the agent (useful before shipping ",{"type":44,"tag":383,"props":3074,"children":3075},{},[3076],{"type":50,"value":1795},{"type":50,"value":3078}," for regression-testing new\nversions once it's live), and ",{"type":44,"tag":103,"props":3080,"children":3082},{"className":3081},[],[3083],{"type":50,"value":684},{"type":50,"value":3085}," to set up ",{"type":44,"tag":59,"props":3087,"children":3088},{},[3089],{"type":50,"value":3090},"online eval rules + guards",{"type":50,"value":3092}," on\nreal traffic once it's deployed. The split is offline test suite vs online rules\u002Fguards — not\nbefore-traffic vs after-traffic.",{"type":44,"tag":141,"props":3094,"children":3096},{"id":3095},"note-keeping-this-skill-in-sync",[3097],{"type":50,"value":3098},"Note — keeping this skill in sync",{"type":44,"tag":53,"props":3100,"children":3101},{},[3102,3104,3108,3110,3115,3117,3122,3124,3129,3131,3136],{"type":50,"value":3103},"The SDK reference (env vars, provider snippets, field lists, workflow-step schema, adapter matrix) is\nintentionally ",{"type":44,"tag":59,"props":3105,"children":3106},{},[3107],{"type":50,"value":341},{"type":50,"value":3109}," duplicated here — it lives in agento11y's ",{"type":44,"tag":103,"props":3111,"children":3113},{"className":3112},[],[3114],{"type":50,"value":129},{"type":50,"value":3116}," \"Path B\" and the\nper-language READMEs, which are the shipped source of truth. This skill holds only decision logic\n(state classification + gap checklist + the gcx verification loop). When a user-facing semantic\nchanges (new SDK field, renamed env var, new framework adapter), update ",{"type":44,"tag":103,"props":3118,"children":3120},{"className":3119},[],[3121],{"type":50,"value":129},{"type":50,"value":3123}," (and its onboarding\nwizard copy); this skill points at llms.txt ",{"type":44,"tag":59,"props":3125,"children":3126},{},[3127],{"type":50,"value":3128},"by section heading, not line number",{"type":50,"value":3130}," (line numbers\ndrift as llms.txt is edited), so no re-pointing is needed unless a heading itself is renamed. If you\nfind yourself pasting a provider snippet into this file, stop — fetch llms.txt instead. The\n",{"type":44,"tag":103,"props":3132,"children":3134},{"className":3133},[],[3135],{"type":50,"value":135},{"type":50,"value":3137}," fallback is deliberately minimal for the same reason.",{"items":3139,"total":3231},[3140,3152,3160,3172,3186,3203,3218],{"slug":349,"name":349,"fn":3141,"description":3142,"org":3143,"tags":3144,"stars":26,"repoUrl":27,"updatedAt":3151},"manage Grafana Agent Observability resources","Inspects and manages Grafana Agent Observability resources via gcx: conversations, generations, evaluators, rules, scores, and templates. Use when the user wants to list or search conversations, inspect generations, manage evaluators (upsert, test, delete), set up evaluation rules, check scores, or browse evaluator templates. Trigger on phrases like \"list conversations\", \"search generations\", \"what did the agent do\", \"debug LLM conversation\", \"create evaluator\", \"set up evaluation rule\", \"test evaluator\", \"check scores\", \"evaluate generation quality\", or \"set up online evaluation\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3145,3146,3147,3150],{"name":20,"slug":21,"type":15},{"name":9,"slug":8,"type":15},{"name":3148,"slug":3149,"type":15},"Monitoring","monitoring",{"name":13,"slug":14,"type":15},"2026-07-25T05:30:40.29622",{"slug":4,"name":4,"fn":5,"description":6,"org":3153,"tags":3154,"stars":26,"repoUrl":27,"updatedAt":28},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3155,3156,3157,3158,3159],{"name":20,"slug":21,"type":15},{"name":9,"slug":8,"type":15},{"name":24,"slug":25,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":684,"name":684,"fn":3161,"description":3162,"org":3163,"tags":3164,"stars":26,"repoUrl":27,"updatedAt":3171},"setup production evaluation for AI agents","Sets up production evaluation and guardrails for a DEPLOYED AI agent in Grafana Agent Observability, grounded in the agent's own code and its real ingested traffic. The judgment layer on top of the `agento11y` skill: it reads the agent's source (system prompt, tools, entrypoint) AND samples its live traffic via gcx, checks what evaluators\u002Frules\u002Fguards already exist, then recommends only what's missing — online eval rules (score live conversations for regressions) and guards (warn-first request-path policies that redact \u002F tool-filter and may later be promoted to deny). It drafts reviewable YAML and, only with explicit confirmation, applies via `gcx agento11y`. New guards are drafted in warn mode (safe on live traffic — warn records but never blocks). It DOES create stack-level objects — that is the point — but every write is confirmed. It never rewrites or redeploys the agent. Trigger on phrases like \"set up production evaluation\", \"my agent is in prod what should I evaluate\", \"catch quality regressions\", \"add guardrails to my agent\", \"redact PII from my agent\", \"block dangerous tools\", \"set up online evals and guards\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3165,3166,3169,3170],{"name":20,"slug":21,"type":15},{"name":3167,"slug":3168,"type":15},"Evals","evals",{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},"2026-07-31T05:53:53.576347",{"slug":676,"name":676,"fn":3173,"description":3174,"org":3175,"tags":3176,"stars":26,"repoUrl":27,"updatedAt":3185},"build and run agent test suites","Use early in an AI-agent project — before ship, before real traffic — to build a starter test suite for the agent and run it offline. Reads the agent's own code (system prompt, tools, task), writes a labeled draft suite of test cases (happy\u002Fedge\u002Fadversarial) grounded in real lines, and recommends how to score each case (the evaluators\u002Fjudges the offline runner uses). Assesses how runnable the agent is: for an easily-invoked agent it generates a runner stub (run_experiment.py) with two holes to fill and can optionally run it (only with permission, only against the endpoint the developer configured); for agents needing a harness or full runtime it points to the existing eval infra. It runs OFFLINE and never creates tenant-level evaluators, rules, or guards — that is `agento11y-prod-setup`, for a deployed agent with real traffic. Trigger on phrases like \"how do I test my agent before shipping\", \"write test cases for my agent\", \"set up tests for my agent\", \"check my agent before prod\", \"I have no traffic yet, how do I evaluate it\", \"test my agent offline\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3177,3178,3179,3182],{"name":20,"slug":21,"type":15},{"name":9,"slug":8,"type":15},{"name":3180,"slug":3181,"type":15},"QA","qa",{"name":3183,"slug":3184,"type":15},"Testing","testing","2026-07-31T05:53:51.62785",{"slug":3187,"name":3187,"fn":3188,"description":3189,"org":3190,"tags":3191,"stars":26,"repoUrl":27,"updatedAt":3202},"create-dashboard","create Grafana dashboards with gcx","Designs and creates Grafana dashboards with gcx, using `gcx dashboards snapshot` as a visual feedback loop. Use when the user wants to create a new Grafana dashboard, add panels, variables, or annotations to an existing dashboard, design dashboard panels, variables, queries, or layout, or make a material visual redesign. Triggers on \"create dashboard\", \"new dashboard\", \"build dashboard\", \"dashboard for \u003Cservice>\", \"add panels\", \"add variable\", \"add annotation\", \"improve this dashboard\", or \"iterate on a dashboard\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3192,3195,3198,3201],{"name":3193,"slug":3194,"type":15},"Dashboards","dashboards",{"name":3196,"slug":3197,"type":15},"Data Visualization","data-visualization",{"name":3199,"slug":3200,"type":15},"Design","design",{"name":9,"slug":8,"type":15},"2026-07-25T05:30:46.289717",{"slug":3204,"name":3204,"fn":3205,"description":3206,"org":3207,"tags":3208,"stars":26,"repoUrl":27,"updatedAt":3217},"debug-with-grafana","investigate application issues with Grafana","Structured workflow for investigating application problems with Grafana observability data (metrics, logs, traces) via gcx. Covers live firefighting AND retrospective incident analysis: incident triage, root-cause analysis, blast-radius checks (did an incident spill into other services), verifying whether a deployment or rollout triggered an incident, finding which service, endpoint, or path owns the most errors or slow requests, checking whether retries or queue backlogs piled up, and quantifying error or latency shares over a time window. Trigger on: \"my API is returning 500 errors\", \"latency is spiking\", \"investigate why requests are failing\", \"triage the incident\", \"blast radius\", \"root cause\", \"did the rollout cause it\", \"which endpoint owns the most 5xx\", \"did retries pile up\", or any request to analyse an earlier incident window using telemetry. For authoring dashboards use create-dashboard; for dashboard inventory use manage-dashboards.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3209,3212,3213,3216],{"name":3210,"slug":3211,"type":15},"Debugging","debugging",{"name":9,"slug":8,"type":15},{"name":3214,"slug":3215,"type":15},"Incident Response","incident-response",{"name":13,"slug":14,"type":15},"2026-07-18T05:11:10.445428",{"slug":3219,"name":3219,"fn":3220,"description":3221,"org":3222,"tags":3223,"stars":26,"repoUrl":27,"updatedAt":3230},"diagnose-entity-graph","diagnose Grafana Entity Graph issues","Diagnose Entity Graph problems: missing entities, missing edges, disconnected clusters, or filtering issues. Use when the user reports that Entity Graph doesn't look right, services are missing, edges aren't appearing, or environments can't be filtered. Triggers for: \"entity graph is empty\", \"services missing from entity graph\", \"no edges in entity graph\", \"disconnected services\", \"can't filter entity graph\", \"entity graph not working\", \"diagnose entity graph\", \"debug knowledge graph\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3224,3225,3226,3229],{"name":3210,"slug":3211,"type":15},{"name":9,"slug":8,"type":15},{"name":3227,"slug":3228,"type":15},"Graph Analysis","graph-analysis",{"name":13,"slug":14,"type":15},"2026-07-25T05:30:39.380934",24,{"items":3233,"total":3366},[3234,3251,3270,3290,3297,3305,3312,3319,3326,3333,3340,3354],{"slug":3235,"name":3235,"fn":3236,"description":3237,"org":3238,"tags":3239,"stars":3248,"repoUrl":3249,"updatedAt":3250},"faro-setup-web","instrument web apps with Grafana Faro","Instruments a web app with Grafana Faro Web SDK for frontend observability. Use when setting up error tracking, Web Vitals, session monitoring, or distributed tracing in a browser app.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3240,3243,3246,3247],{"name":3241,"slug":3242,"type":15},"Distributed Tracing","distributed-tracing",{"name":3244,"slug":3245,"type":15},"Frontend","frontend",{"name":3148,"slug":3149,"type":15},{"name":13,"slug":14,"type":15},1103,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Ffaro-web-sdk","2026-07-12T07:43:24.63314",{"slug":3252,"name":3252,"fn":3253,"description":3254,"org":3255,"tags":3256,"stars":3267,"repoUrl":3268,"updatedAt":3269},"configuring-yesoreyeram-infinity-datasource","configure Grafana Infinity data source","Configure the Infinity data source — base URL and allowed hosts, the authentication methods (basic, bearer token, API key, digest, OAuth passthrough, OAuth 2.0 client credentials\u002FJWT, Azure, Azure Blob, AWS), TLS, custom HTTP headers, network and security settings, the custom health check, and provisioning with a config file. Use when a user asks how to set up, configure, or change settings for the Infinity data source; how to authenticate to an API; how to allow hosts; how to provision it as YAML; or how to troubleshoot connection, authentication, or health-check issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3257,3260,3263,3266],{"name":3258,"slug":3259,"type":15},"API Development","api-development",{"name":3261,"slug":3262,"type":15},"Authentication","authentication",{"name":3264,"slug":3265,"type":15},"Configuration","configuration",{"name":9,"slug":8,"type":15},1056,"https:\u002F\u002Fgithub.com\u002Fgrafana\u002Fgrafana-infinity-datasource","2026-07-12T07:43:25.939136",{"slug":3271,"name":3271,"fn":3272,"description":3273,"org":3274,"tags":3275,"stars":3267,"repoUrl":3268,"updatedAt":3289},"querying-yesoreyeram-infinity-datasource","query data with Infinity datasource","Build queries with the Infinity data source — the query types (JSON, CSV, TSV, XML, GraphQL, HTML, UQL, GROQ, Google Sheets, Series, Transformations), the parsers (Frontend\u002Fsimple, Backend JSONata, JQ, UQL, GROQ) and which support alerting, the sources (URL, Inline, Reference, Azure Blob, Random walk), output formats (table, timeseries, logs, trace, node graph, dataframe), root selector and columns, computed columns\u002Ffilters\u002Fsummarize, pagination, and template variables. Use when a user asks how to query Infinity; how to fetch JSON\u002FCSV\u002FXML\u002FGraphQL\u002FHTML from a URL or inline; how to select rows and columns; how to transform data with UQL\u002FGROQ\u002FJSONata\u002FJQ; how to make a query work with alerting; how to paginate; or how to use variables in a query.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3276,3279,3282,3283,3286],{"name":3277,"slug":3278,"type":15},"CSV","csv",{"name":3280,"slug":3281,"type":15},"Data Analysis","data-analysis",{"name":9,"slug":8,"type":15},{"name":3284,"slug":3285,"type":15},"GraphQL","graphql",{"name":3287,"slug":3288,"type":15},"JSON","json","2026-07-15T05:34:05.773947",{"slug":349,"name":349,"fn":3141,"description":3142,"org":3291,"tags":3292,"stars":26,"repoUrl":27,"updatedAt":3151},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3293,3294,3295,3296],{"name":20,"slug":21,"type":15},{"name":9,"slug":8,"type":15},{"name":3148,"slug":3149,"type":15},{"name":13,"slug":14,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":3298,"tags":3299,"stars":26,"repoUrl":27,"updatedAt":28},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3300,3301,3302,3303,3304],{"name":20,"slug":21,"type":15},{"name":9,"slug":8,"type":15},{"name":24,"slug":25,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":684,"name":684,"fn":3161,"description":3162,"org":3306,"tags":3307,"stars":26,"repoUrl":27,"updatedAt":3171},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3308,3309,3310,3311],{"name":20,"slug":21,"type":15},{"name":3167,"slug":3168,"type":15},{"name":9,"slug":8,"type":15},{"name":13,"slug":14,"type":15},{"slug":676,"name":676,"fn":3173,"description":3174,"org":3313,"tags":3314,"stars":26,"repoUrl":27,"updatedAt":3185},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3315,3316,3317,3318],{"name":20,"slug":21,"type":15},{"name":9,"slug":8,"type":15},{"name":3180,"slug":3181,"type":15},{"name":3183,"slug":3184,"type":15},{"slug":3187,"name":3187,"fn":3188,"description":3189,"org":3320,"tags":3321,"stars":26,"repoUrl":27,"updatedAt":3202},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3322,3323,3324,3325],{"name":3193,"slug":3194,"type":15},{"name":3196,"slug":3197,"type":15},{"name":3199,"slug":3200,"type":15},{"name":9,"slug":8,"type":15},{"slug":3204,"name":3204,"fn":3205,"description":3206,"org":3327,"tags":3328,"stars":26,"repoUrl":27,"updatedAt":3217},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3329,3330,3331,3332],{"name":3210,"slug":3211,"type":15},{"name":9,"slug":8,"type":15},{"name":3214,"slug":3215,"type":15},{"name":13,"slug":14,"type":15},{"slug":3219,"name":3219,"fn":3220,"description":3221,"org":3334,"tags":3335,"stars":26,"repoUrl":27,"updatedAt":3230},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3336,3337,3338,3339],{"name":3210,"slug":3211,"type":15},{"name":9,"slug":8,"type":15},{"name":3227,"slug":3228,"type":15},{"name":13,"slug":14,"type":15},{"slug":108,"name":108,"fn":3341,"description":3342,"org":3343,"tags":3344,"stars":26,"repoUrl":27,"updatedAt":3353},"manage Grafana Cloud resources via gcx","Manages Grafana Cloud resources via the gcx CLI. Trigger when the user wants to inspect, create, update, delete, query, or automate any Grafana resource - dashboards, datasources, alerts, SLOs, synthetic checks, oncall, incidents, fleet, k6, knowledge graph, or adaptive telemetry.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3345,3348,3349,3350],{"name":3346,"slug":3347,"type":15},"CLI","cli",{"name":9,"slug":8,"type":15},{"name":3148,"slug":3149,"type":15},{"name":3351,"slug":3352,"type":15},"Operations","operations","2026-07-31T05:53:50.587304",{"slug":3355,"name":3355,"fn":3356,"description":3357,"org":3358,"tags":3359,"stars":26,"repoUrl":27,"updatedAt":3365},"gcx-demo","present gcx demo tours","Run a narrated, read-only demo tour of gcx for customer or colleague presentations. Showcases the breadth of gcx across every Grafana Cloud product area — resources, datasources, metrics, logs, traces, SLOs, alerts, synthetic monitoring, IRM, k6, fleet, and more. All commands are strictly read-only. Trigger when the user says \"demo gcx\", \"show off gcx\", \"customer demo\", \"gcx tour\", or \"\u002Fgcx-demo\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3360,3361,3362],{"name":3346,"slug":3347,"type":15},{"name":9,"slug":8,"type":15},{"name":3363,"slug":3364,"type":15},"Presentations","presentations","2026-07-25T05:30:45.282458",80]