[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-datadog-labs-agent-observability-trace-rca":3,"mdc-8xfzpz-key":39,"related-repo-datadog-labs-agent-observability-trace-rca":6196,"related-org-datadog-labs-agent-observability-trace-rca":6297},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":28,"repoUrl":29,"updatedAt":30,"license":31,"forks":32,"topics":33,"repo":34,"sourceUrl":37,"mdContent":38},"agent-observability-trace-rca","diagnose LLM application failures from traces","Root cause analysis on production LLM traces. Diagnoses why an LLM application is failing — works from eval judge verdicts, runtime errors, or structural anomalies depending on what signals are present. Walks the span tree from symptom to root cause. Use when user says \"what's wrong with my app\", \"why is my eval failing\", \"analyze errors\", \"root cause analysis\", \"diagnose failures\", or wants to understand production failure patterns.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"datadog-labs","Datadog Labs","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdatadog-labs.png",[12,16,19,22,25],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"LLM","llm",{"name":20,"slug":21,"type":15},"Datadog","datadog",{"name":23,"slug":24,"type":15},"Evals","evals",{"name":26,"slug":27,"type":15},"Debugging","debugging",145,"https:\u002F\u002Fgithub.com\u002Fdatadog-labs\u002Fagent-skills","2026-06-19T09:04:24.316244",null,19,[],{"repoUrl":29,"stars":28,"forks":32,"topics":35,"description":36},[],"Public repository for Datadog Agent Skills","https:\u002F\u002Fgithub.com\u002Fdatadog-labs\u002Fagent-skills\u002Ftree\u002FHEAD\u002Fagent-observability\u002Fagent-observability-trace-rca","---\nname: agent-observability-trace-rca\ndescription: Root cause analysis on production LLM traces. Diagnoses why an LLM application is failing — works from eval judge verdicts, runtime errors, or structural anomalies depending on what signals are present. Walks the span tree from symptom to root cause. Use when user says \"what's wrong with my app\", \"why is my eval failing\", \"analyze errors\", \"root cause analysis\", \"diagnose failures\", or wants to understand production failure patterns.\n---\n\n## Backend\n\n**Detection** — At the start of every invocation, before taking any action, determine which backend to use:\n\n1. If the user passed `--backend pup` anywhere in their invocation → use **pup mode** immediately, regardless of whether MCP tools are present. Skip steps 2–4.\n2. Check whether MCP tools are present in your active tool list. The canonical signal is whether `mcp__datadog-llmo-mcp__list_llmobs_evals` appears in your available tools.\n3. If MCP tools are present → use **MCP mode** throughout. Call MCP tools exactly as named in this skill's workflow sections.\n4. If MCP tools are absent → check whether `pup` is executable: run `pup --version` via Bash. A JSON response containing `\"version\"` confirms pup is available.\n5. If pup responds → use **pup mode** throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.\n6. If neither is available → stop and tell the user:\n   > \"Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (`claude mcp add --scope user --transport http datadog-llmo-mcp 'https:\u002F\u002Fmcp.datadoghq.com\u002Fapi\u002Funstable\u002Fmcp-server\u002Fmcp?toolsets=llmobs'`) or install pup.\"\n\n`--backend pup` is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.\n\n**pup invocation rules:**\n- Invoke via Bash: `pup llm-obs \u003Csubcommand> [flags]`\n- pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in `[{\"type\": \"text\", \"text\": \"\u003Cjson>\"}]`).\n- If pup returns an auth error, tell the user to run `pup auth login` and stop.\n- Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).\n- Time flags: pup accepts bare duration strings (`1h`, `7d`, `30m`) and RFC3339 timestamps. Do **not** use `now-`-prefixed strings — strip the prefix when converting from a skill `--timeframe` argument: `now-7d` → `7d`, `now-24h` → `24h`, `now-30d` → `30d`.\n- `--summary` on `pup llm-obs spans search` strips payload fields to essential metadata only. Use it in bulk\u002Fsearch phases where content is not needed.\n\n**Invocation ID:** At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., `3a9f1c2b`). Keep it constant for the entire invocation.\n\n**Intent tagging:** On every MCP tool call, prefix `telemetry.intent` with `skill:agent-observability-trace-rca[\u003Cinv_id>] — ` followed by a description of why the tool is being called. On the **first MCP tool call only**, use `skill:agent-observability-trace-rca:start[\u003Cinv_id>] — ` instead (note the `:start` suffix). Example first call: `skill:agent-observability-trace-rca:start[3a9f1c2b] — Phase 0: discover configured evals for task-cruncher to infer analysis mode`\n\n# Agent Observability Trace RCA — Root Cause Analysis from Production LLM Traces\n\nDiagnose **why an LLM application is failing** by searching production traces and walking the span tree from symptom to root cause. The skill automatically selects the best analysis mode based on available signals:\n\n| Mode | Signal | When auto-selected |\n|------|--------|--------------------|\n| **Eval Signal** | LLM judge verdicts and reasoning (pass\u002Ffail rates, scoring) | Evaluators are configured for the app |\n| **Error Signal** | Runtime errors (`@status:error`, error types, stack traces) | No evals configured, or user explicitly asks about errors\u002Fcrashes |\n| **Generic** | Structural anomalies (latency, agent loops, retrieval misses) | Explicit `mode=generic` override, or no strong signal found in Phase 1 |\n\nThe mode is **announced (never asked)** in the first user-facing output with a one-line override hint.\n\n## Methodology\n\n**Resolve → Search → Observe → Open Coding → Axial Coding → Root Cause Navigation → Recommendations**\n\n## Usage\n\n```\nWhat's wrong with \u003Cml_app> over the last \u003Ctimeframe>\nWhy is \u003Cml_app> failing today\nAnalyze eval failures for \u003Ceval_name> on \u003Cml_app>\nLook at the errors on \u003Cml_app> over the last \u003Ctimeframe>\nRoot-cause low scores on \u003Ceval_name>\n```\n\n## Inputs\n\n| Input | Required | Default | Description |\n|-------|----------|---------|-------------|\n| `ml_app` | Yes (or `eval_name`) | — | The application to analyze. |\n| `eval_name` | No | — | One or more evaluators to focus on. Implies Eval Signal mode. Pass a list for multi-eval analysis. |\n| `timeframe` | No | `now-24h` | How far back to look. |\n| `mode` | No | inferred | Explicit mode override: `eval`, `errors`, `generic`. Skips inference entirely. |\n| `failure_filter` | No | — | Narrowing scope: `\"errors\"` (routes to Error Signal path), `\"high latency\"` (post-fetch duration sort), `\"low scores on \u003Ceval>\"` (promotes to `eval_name`), a tool name or span name (`@name:\u003Cx>` query). |\n\nIf neither `ml_app` nor `eval_name` is provided, ask the user.\n\n## Available Tools\n\n### Eval discovery & overview\n\n| Tool | Purpose |\n|------|---------|\n| `list_llmobs_evals` | Discover all configured evals for an `ml_app`. Used in Phase 0 mode inference. |\n| `get_llmobs_eval_aggregate_stats` | Pass\u002Ffail rate or score distribution for an eval over a time window. |\n| `get_llmobs_evaluator` | Full evaluator config: prompt template, assessment criteria, span filter, sampling, provider. Use instead of the deprecated `get_llmobs_eval_config`. |\n\n### Trace & span exploration\n\n| Tool | Purpose |\n|------|---------|\n| `search_llmobs_spans` | Find spans by tags, span kind, status, query syntax. Paginate with cursor. Entry point for Phase 1. |\n| `get_llmobs_span_details` | Metadata, evaluations (scores, labels, reasoning), `status`, error fields, duration, and **`content_info`** map showing available fields + sizes. |\n| `get_llmobs_span_content` | Actual content for a span field. Supports JSONPath via `path` param for targeted extraction. |\n| `get_llmobs_trace` | Full trace hierarchy as span tree with span counts by kind. |\n| `find_llmobs_error_spans` | All error spans in a trace with error type, message, stack, and propagation context. |\n| `expand_llmobs_spans` | Load children of collapsed trace nodes. |\n| `get_llmobs_agent_loop` | Chronological agent execution timeline (LLM calls, tool invocations, decisions). May return empty — see Phase 4b fallback. |\n\n### Key `get_llmobs_span_content` patterns\n\n| Field | Path | What you get |\n|-------|------|-------------|\n| `messages` | `$.messages[0]` | System prompt (first message, usually `system` role) |\n| `messages` | `$.messages[-1]` | Last assistant response |\n| `messages` | *(no path)* | Full conversation including tool calls |\n| `input` \u002F `output` | — | Span I\u002FO |\n| `documents` | — | Retrieved documents (RAG apps) |\n| `metadata` | — | Custom metadata (prompt versions, feature flags, user segments) |\n\n### How to use `search_llmobs_spans`\n\n**Always include `@ml_app:\"\u003Cml_app>\"` in the `query` string — the structured `ml_app` parameter is unreliable and can return spans from other apps.** Do not rely on the structured parameter alone.\n\nUseful query fragments — combine with space (AND):\n\n| Goal | Query |\n|------|-------|\n| Errors only | `@status:error` |\n| Eval is present on the span | `@evaluations.custom.\u003Ceval_name>:*` (presence only — pass\u002Ffail is read from `get_llmobs_span_details`, not the query) |\n| A specific tool by name | `@name:\u003Ctool_name>` |\n\nDedicated params (`span_kind`, `root_spans_only`, `ml_app`) work alongside `query`, but `query` takes precedence over `tags`.\n\n### Parallelization rules\n\n1. **`get_llmobs_span_details`**: Group span_ids by trace_id, chunk each trace's span_ids into batches of at most 20. Issue ALL chunks for a page in a **single message**.\n2. **`get_llmobs_span_content`**: Each call is independent — always issue ALL in a single message.\n3. **`get_llmobs_trace` \u002F `find_llmobs_error_spans` \u002F `get_llmobs_agent_loop`**: Parallelize across different traces in a single message.\n4. **Pipeline parallelism**: Start `get_llmobs_span_details` for page 1 results immediately — don't wait to collect all pages.\n\n---\n\n## Analysis Workflow\n\n**Output discipline**: Phases 0–5 are internal analysis. The only user-facing outputs during these phases are the Phase 1 Signal Summary and the mandatory checkpoints at Phases 2 and 3. Do NOT narrate reasoning, summarize intermediate findings, or output Phase 4 deep-dive results as prose. All detailed findings go exclusively into the Phase 6 report.\n\n---\n\n### Phase 0: Resolve Inputs & Infer Mode\n\n**First: check for classification context.** Scan the conversation for a `# Session Classification Summary` header. If found → enter **Step 0S** below and skip all remaining Phase 0 steps and Phase 1 entirely.\n\n#### Step 0S — Extract Failure Bucket from Classification Output\n\nThe canonical handoff format is the **Per-Unit Details table** inside the `# Session Classification Summary` section. Extract one row per unit:\n\n| Field | Source |\n|-------|--------|\n| `trace_id` | Link URL in the ID column: parse the `trace_id=` or `session_id=` query parameter from the link href |\n| `verdict` | Verdict column |\n| `failure_mode` | Failure Mode column (`none` for passing rows) |\n| `detail` | Reason column — use as the Phase 2 reasoning input (same role as eval judge reasoning or error messages) |\n| `app_type` | From the `# Session Classification Summary` header line (e.g. `Root span kind: agent`) — default `LLM` if absent |\n\n**Failure bucket** = all rows where verdict is `no` or `partial`.\n\n- \u003C 5 entries → note low confidence, proceed anyway.\n- Empty → report \"No failures found in the classification output\" and stop.\n\nPresent this overview before proceeding:\n\n```\n## Classification Overview (from agent-observability-session-classify)\n\n**ml_app**: \u003Cfrom summary header>  |  **Classified**: N  |  **Failures (no+partial)**: F  |  **Pass rate**: X%\n\n| Failure Mode | Count |\n|---|---|\n...\n\nProceeding to Phase 2 using F failure traces. Mode inference bypassed — classification verdict is the signal.\n```\n\nThen **skip Phase 1 and jump directly to Phase 2**. Carry forward:\n- Phase 2 reasoning input: `(trace_id, span_id, detail)` tuples — same structure as eval reasoning or error messages\n- Phase 4 navigation: use `app_type` from each trace block to choose the span navigation strategy\n- Phases 2–7: run completely unchanged — the failure bucket structure is identical regardless of source\n\n---\n\n**Standard resolution (no classification context):**\n\n1. If neither `ml_app` nor `eval_name` provided → ask the user. If `eval_name` is provided but `ml_app` is not → also ask for `ml_app` (eval names are not globally unique; without it, span searches return results from all apps sharing the eval name).\n2. If `timeframe` not provided → default to `now-24h`.\n3. **Resolve `failure_filter`** (before mode inference):\n   - `\"errors\"` → force **Error Signal** mode\n   - `\"low scores on \u003Ceval>\"` → treat as `eval_name=\u003Ceval>`, then continue inference\n   - `\"high latency\"` → note for Phase 1 (sort by duration post-fetch); continue inference\n   - Tool\u002Fspan name → note as `@name:\u003Cx>` query fragment for Phase 1; continue inference\n4. **Resolve mode** (skip if `mode` was explicitly provided):\n   - `eval_name` given → **Eval Signal**\n   - User explicitly mentioned errors\u002Fexceptions\u002Fcrashes → **Error Signal**\n   - Otherwise → call `list_llmobs_evals_by_ml_app(ml_app)`:\n     - Evals returned → **Eval Signal**\n     - No evals → **Error Signal** (announce auto-selection in Phase 1)\n5. When `eval_name` is multi-valued, note for Phase 1: run parallel per-eval searches and merge+dedup by `(trace_id, span_id)`.\n\n---\n\n### Phase 1: Find Problematic Spans\n\nThree mode-specific paths. All end with a **Signal Summary** that labels the mode and includes a one-line override hint.\n\n**Mode switch handling**: At any checkpoint, if the user says \"switch to [error|eval|generic] mode\", re-enter Phase 1 with the new mode. Phase 0 inputs do not re-resolve.\n\n**Auto-pivot**: If the selected mode finds no data (0 evals configured, 0 error spans in timeframe), announce the pivot to Generic and proceed — do not stop and ask.\n\n---\n\n#### Eval Signal path\n\n##### Step 1a: Eval overview (parallel)\n\nFor each eval, call both in a single parallel batch:\n- `get_llmobs_eval_aggregate_stats(eval_name, from, to)`\n- `get_llmobs_evaluator(eval_name)`\n\n**Interpret aggregate stats:**\n- **`total_count == 0`** → Note \"no data.\" Skip this eval (or pivot to Generic if it's the only one).\n- **Boolean `pass_rate == 1.0`** → Note \"100% pass.\" Skip unless it's the only eval.\n- **Boolean with failures** → Note counts and pass_rate. Continue.\n- **Score with assessment criteria** → Note distribution and pass\u002Ffail counts. Continue.\n- **Score WITHOUT assessment criteria** → Infer failures: bottom quartile, or below median if bimodal. Label as \"inferred failures\" in report.\n- **Categorical with assessment criteria** → Note top_values and pass\u002Ffail. Continue.\n- **Categorical WITHOUT assessment criteria** → Infer from context (e.g., \"error\", \"incomplete\", \"off_topic\" are likely failures). Ask user if genuinely ambiguous.\n\n**Interpret eval config:**\n- **Config returned** (custom) → Store `prompt_template`, `assessment_criteria`, `parsing_type`, `output_schema`.\n- **Config nil** (OOTB) → Note prompt is not inspectable.\n\n**Calibration cross-check**: When two evals share a name prefix but differ in type (e.g. `foo-boolean` and `foo-score`), compare their pass rates on overlapping spans. A discrepancy >20% is an **Evaluator Calibration Discrepancy** — flag it in the report.\n\n##### Step 1b: Collect failure spans\n\nFor each eval:\n1. `search_llmobs_spans(query=\"@evaluations.custom.\u003Ceval_name>:*\", from, limit=50)`. When multi-valued, issue one search per eval in parallel — merge result sets, dedup by `(trace_id, span_id)`.\n2. Paginate until ≥15–20 failures OR no more pages. Cap at 200 spans total.\n3. `get_llmobs_span_details` per trace_id batch (follow Parallelization Rules).\n4. Extract per row: **assessment**, **value**, **reasoning**, **span_id**, **trace_id**, **span_kind**, **content_info**.\n5. Separate into pass\u002Ffail buckets using thresholds from Step 1a.\n\n**JSON-type eval fallback**: If `@evaluations.custom.\u003Ceval_name>:*` returns 0 spans but `get_llmobs_eval_aggregate_stats` confirmed `total_count > 0`, the eval is JSON-type and scores are not indexed on this field. Fall back to: search by the span name or span kind that the eval targets (check `get_llmobs_evaluator` for the span filter), then inspect output payloads for JSON verdict fields via `get_llmobs_span_content(field=\"output\")`.\n\n##### Step 1c: Signal Summary (Eval Signal)\n\n```\n## Signal Summary: `{ml_app}` · Eval Signal\n(Inferred from {N} configured eval(s). Say `switch to error mode` or `switch to generic mode` to change.)\n\n**Timeframe**: {from} → {to}\n\n| Eval | Type | Total | Pass Rate | Status |\n|------|------|------:|:---------:|--------|\n| eval_1 | boolean | 4,891 | 37.3% | ⚠ Investigating |\n| eval_2 | score | 1,200 | — (inferred threshold) | ⚠ Investigating |\n| eval_3 | boolean | 500 | 99.2% | ✓ Healthy |\n\n**Collected**: {pass_count} passing, {fail_count} failing.\n```\n\nFor a single eval, collapse to a single-line header instead of a table.\n\n---\n\n#### Error Signal path\n\n##### Step 1a: Sample error spans\n\n`search_llmobs_spans(query=\"@ml_app:\\\"\u003Cml_app>\\\" @status:error\", from=timeframe, limit=50)`. Paginate until ≥30 error spans or no more pages.\n\n##### Step 1a.5: Soft error scan\n\nMCP tool spans sometimes report `@status:ok` but carry `\"isError\": true` in their output payload — these are invisible to `@status:error` queries and can outnumber hard errors.\n\nCall `search_llmobs_spans(query=\"@ml_app:\\\"\u003Cml_app>\\\" @status:ok\", span_kind=\"tool\", from=timeframe, limit=20)`. For a sample of 5–10 results, call `get_llmobs_span_content(field=\"output\")` in parallel. If any payloads contain `\"isError\": true`, add **MCP soft errors** as a separate row in the error frequency table with the note: *(status:ok but isError:true in payload — not queryable via @status:error)*.\n\n##### Step 1b: Group by error type\n\nGroup spans by `error_type` tag → frequency table. If `error_type` tag is absent on some spans, supplement with the `error.type` field from `get_llmobs_span_details` (fetched in Step 1d).\n\n##### Step 1c: Fetch stack traces (parallel)\n\nFor the top 3–4 error types by count, pick 2–3 representative trace IDs each. Call `find_llmobs_error_spans(trace_id)` in parallel across all selected traces. Extract:\n- Error message and stack trace\n- Origin span kind and name\n- Whether errors propagate from children to parents (cascade) or are isolated\n\n##### Step 1d: Fetch span details\n\n`get_llmobs_span_details` on representative spans for each error type (follow Parallelization Rules). Extract `content_info`, `span_kind`, duration.\n\n##### Step 1e: Signal Summary (Error Signal)\n\n```\n## Signal Summary: `{ml_app}` · Error Signal\n(No evals configured — analyzing runtime errors. Say `switch to eval mode` or `switch to generic mode` to change.)\n\n**Timeframe**: {from} → {to}  |  **Total error spans sampled**: {N}\n\n| Error Type | Spans | Cascade? | Origin Span Kind |\n|------------|------:|:--------:|-----------------|\n| TimeoutError | 42 | Yes | tool |\n| APIError 429 | 18 | No | tool |\n| ValueError | 7 | No | llm |\n| MCP soft errors (isError:true) | 23 | No | tool |\n```\n\n---\n\n#### Generic path\n\n##### Step 1a: Eval health check (when evals are configured)\n\nIf `list_llmobs_evals` returned evals in Phase 0, call `get_llmobs_eval_aggregate_stats` for each enabled eval in parallel. Flag any enabled eval with `total_count: 0` as **Broken Eval Configuration** — include in the Signal Summary anomaly table as a High severity row.\n\n##### Step 1b: Broad span search\n\n`search_llmobs_spans(query=\"@ml_app:\\\"\u003Cml_app>\\\"\", root_spans_only=true, from=timeframe, limit=50)`. Apply `failure_filter` narrowing if present (tool\u002Fspan name → `@name:\u003Cx>` query; `\"high latency\"` → sort result set by `duration` after Step 1c). Paginate until ≥30 spans.\n\n##### Step 1c: Fetch span details\n\n`get_llmobs_span_details` per trace_id batch.\n\n##### Step 1d: Rank by structural anomalies\n\nPartition spans using heuristics:\n- Top decile by `duration` (latency outliers)\n- Agent spans with >N tool\u002FLLM iterations (long-running loops)\n- Retrieval spans returning 0 documents (RAG miss)\n- Workflow spans whose child set is missing an expected step (compare against median child layout)\n- **Token efficiency**: Check if `non_cached_input_tokens ≈ input_tokens` across LLM spans. If the app has stable system prompts (>1k tokens) and cache hit rate is 0%, flag as High severity — enabling `cache_control: ephemeral` on the system prompt would cut input token costs by 60–90%\n\n##### Step 1e: Signal Summary (Generic)\n\n```\n## Signal Summary: `{ml_app}` · Generic\n(Analyzing structural anomalies. Say `switch to eval mode` or `switch to error mode` to change.)\n\n**Timeframe**: {from} → {to}  |  **Sampled**: {N} root spans\n\n| Anomaly Type | Count |\n|---|:---:|\n| Latency outliers (>p90) | 12 |\n| Long agent loops (>8 iterations) | 5 |\n| RAG retrieval misses | 3 |\n| Zero prompt cache utilization | All LLM spans |\n| Broken eval configurations | 2 |\n```\n\n---\n\n### Phase 1.5: Determine App Profile & Where the Root Cause Lives\n\nInspect `content_info` and `span_kind` across collected spans. Drives Phase 4 strategy.\n\n**App profile** (from content_info):\n\n| Signal | App profile | Phase 4 strategy |\n|--------|------------|-----------------|\n| `content_info` has `messages` | LLM\u002Fchat app | Extract system prompt via `messages[0]`, check conversation flow |\n| `content_info` has `documents` | RAG app | Check retrieval quality alongside LLM output |\n| Trace contains `agent` span kind | Agent app | Try `get_llmobs_agent_loop` first; if it returns empty use child-span reconstruction (see Phase 4b) |\n| `messages.count > 10` | Long conversation | Check for context overflow |\n| `content_info` has `metadata` | Has custom metadata | Check for clustering by metadata values (prompt version, feature flags) |\n\n> **LLM Experiments traces**: If root spans have `span_kind: experiment` and carry `input`, `output`, and `expected_output` structured fields, you are looking at a [Datadog LLM Experiments](https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fexperiments\u002F) trace. Each span represents one dataset record run. Read quality signal from the root span's `input`\u002F`output`\u002F`expected_output` fields via `get_llmobs_span_content` — not from LLM sub-span messages, which may contain stub or placeholder content. Evaluations attached to experiment spans are computed by the Experiments framework at run time and may not be registered as online Datadog evaluators (`get_llmobs_evaluator` will return 404 for them).\n\n**Where the root cause likely lives** — by symptom span kind:\n\n| Symptom span kind | Symptom looks like | But root cause is often in... |\n|-------------------|-------------------|------------------------------|\n| `llm` | Bad LLM response (eval flagged, wrong output) | **Parent** agent (bad instructions), **sibling** retrieval (bad context), **sibling** tool (bad data) |\n| `agent` | Bad orchestration | **Child** spans (wrong tool calls, bad routing), full agent loop |\n| `tool` | Bad tool result | **Parent** LLM (passed wrong parameters), tool implementation |\n| `workflow` | Bad overall output | **Child** sub-spans (which step first deviated?) |\n| `retrieval` | Bad retrieval | Query construction (parent), index\u002Fembedding config (outside trace) |\n\n**Key insight**: The signal — eval verdict, error message, latency outlier — flags one span in isolation. It's a *symptom report*, not a diagnosis. The root cause often lives in a different span: a parent that gave bad instructions, a sibling that provided bad context, or a child that made a wrong decision. Phase 4 navigates the tree to find it.\n\n---\n\n### Phase 2: Open Coding — Initial Failure Categorization\n\n**Goal**: Read per-row evidence and propose initial, concrete failure categories. Pool all problematic rows together — categories should describe app behaviors, not which signal flagged them.\n\n**Per-row \"reasoning input\"** by mode:\n- **Eval Signal**: judge assessment + reasoning from `get_llmobs_span_details`\n- **Error Signal**: error message + stack trace excerpt from `find_llmobs_error_spans`\n- **Generic**: one-line description of the structural anomaly that flagged the row\n\n**Shortcuts**:\n- **\u003C 15 problematic rows**: Combine Phases 2 and 3 into one pass. Still produce the checkpoint.\n- **> 80% share the same reasoning\u002Ferror\u002Fsymptom**: Skip to Phase 4 with the dominant pattern. Still output checkpoint.\n- **> 50 problematic rows**: Sample ~50, build taxonomy, then spot-check 10–15 more.\n\n1. **Use per-row signal from Phase 1** — do NOT re-fetch. Only call `get_llmobs_span_content(field=\"input\"\u002F\"output\")` for spans where the reasoning is insufficient (generic, empty, or just a stack trace with no app context).\n\n2. **If eval config is loaded** (Eval Signal), distinguish early:\n   - **App failures**: Output genuinely violates the eval's criteria\n   - **Eval failures**: Output seems reasonable but eval criteria are too strict\u002Fambiguous\n\n3. **Each pattern must be specific**: \"Agent called search instead of calculator for price computation\" — NOT \"tool issue.\"\n\n#### MANDATORY CHECKPOINT\n\n```\n**Open coding**: {N} problematic rows → {K} initial categories: {Category1} ({count}), {Category2} ({count}), ...\n```\n\n---\n\n### Phase 3: Axial Coding — Refine Failure Taxonomy\n\n**Goal**: 3–8 final categories, ranked by impact.\n\n1. **Merge**: Categories with \u003C 3 occurrences → parent category or drop as noise.\n2. **Split**: Categories with > 30% of failures → more specific sub-categories. Pull additional span content if needed.\n3. **Validate**: 2–3 representative examples per category confirm the label fits.\n4. **Rank**: `priority = count × severity` (severity: high \u002F medium \u002F low).\n\n#### MANDATORY CHECKPOINT\n\n```\n**Axial coding**: {merges\u002Fsplits\u002Fdrops}. Final categories:\n1. {Category} ({count}, {pct}%) — {severity}\n2. ...\n```\n\n---\n\n### Phase 4: Root Cause Analysis — Navigate from Symptom to Root Cause\n\n**Goal**: The signal flagged a span. That's the symptom. Navigate the trace tree to find the actual root cause — it's often in a different span.\n\nFor each of the top 3 categories, pick 2–3 representative traces:\n\n#### Step 4a: Trace structure + errors (parallel)\n\nFor each representative trace, call in a single message:\n- `get_llmobs_trace(trace_id)` — span hierarchy; locate the symptom span and its parent\u002Fsiblings\u002Fchildren\n- `find_llmobs_error_spans(trace_id)` — check for runtime errors anywhere in the trace\n\n**Runtime vs behavioral**: If errors exist on or near the symptom span, the root cause may be a runtime failure rather than a behavioral one. Check this first.\n\n**Distributed trace fallback**: If `get_llmobs_trace` returns \"cannot find parent\" or an empty span list (common in Ray-based or multi-process execution), reconstruct the trace manually using `get_llmobs_span_details` on the span_ids collected in Phase 1, sorted by `start_ms`.\n\n#### Step 4b: Navigate to the root cause (parallel)\n\nUse the symptom span kind (from Phase 1.5). Issue ALL calls in a single message.\n\n**If symptom is on an `llm` span** (most common):\n- `get_llmobs_span_content(field=\"messages\", path=\"$.messages[0]\")` on **symptom span** — system prompt\n- `get_llmobs_span_content(field=\"messages\")` on **symptom span** — full context received\n- `get_llmobs_span_content(field=\"documents\")` on **sibling retrieval spans** (if any)\n- `get_llmobs_span_content(field=\"input\")` on **sibling tool spans** (if any)\n- `get_llmobs_span_content(field=\"messages\", path=\"$.messages[0]\")` on **parent agent\u002Fworkflow span**\n\n**If symptom is on an `agent` span**:\n- `get_llmobs_agent_loop(trace_id, span_id)` — full decision timeline *(try first; if it returns 0 iterations, use the fallback below)*\n- `get_llmobs_span_details` on **child spans** — sort by `start_ms` to reconstruct the execution timeline\n- `get_llmobs_span_content(field=\"input\"\u002F\"output\")` on **child spans that look wrong**\n\n**Agent loop fallback** (when `get_llmobs_agent_loop` returns 0 iterations): Reconstruct the timeline from `get_llmobs_span_details` results sorted by `start_ms`. Group by `span_kind` to identify LLM → tool → LLM sequences. This fallback is frequently needed — `get_llmobs_agent_loop` returns empty for many apps.\n\n**If symptom is on a `tool` span**:\n- `get_llmobs_span_content(field=\"input\")` on **symptom span** — what parameters was it called with?\n- `get_llmobs_span_content(field=\"messages\")` on **parent LLM span** — did the LLM construct the call correctly?\n\n**If symptom is on a `workflow` span**:\n- `get_llmobs_span_details` on **all child spans** — find which step first deviated\n- `get_llmobs_span_content(field=\"input\"\u002F\"output\")` on the **deviating child**\n\n**Always also fetch**:\n- `get_llmobs_span_content(field=\"metadata\")` on the symptom span — clustering signals (prompt version, feature flags)\n\n#### Step 4c: Diagnose — from symptom to root cause\n\nFor each category, trace the causal chain:\n\n1. **Symptom** — what the signal flagged (eval reasoning, error message, anomaly note). The signal only saw one span in isolation — its reasoning may be shallow.\n2. **Trace context** — what surrounding spans reveal (parent instructions, sibling data, child decisions).\n3. **Root cause** — the specific span and decision point where the failure originated. Often NOT the symptom span itself.\n\n**For suspected eval issues** (Eval Signal, if config loaded): Compare eval criteria against evidence. Is the prompt ambiguous? Criteria too strict?\n\n**Root cause categories:**\n\n| Category | Description |\n|----------|-------------|\n| **System Prompt Deficiency** | Instructions unclear, missing, or contradictory — in symptom span OR its parent |\n| **Tool Gap** | Needed tool doesn't exist or parameters too coarse |\n| **Tool Misuse** | Wrong tool called or wrong parameters — often visible in agent loop or parent LLM |\n| **Routing\u002FHandoff Error** | Wrong sub-agent selected (multi-agent systems) |\n| **Retrieval Failure** | RAG returned irrelevant or missing context — check sibling retrieval spans |\n| **Context Overflow** | Critical info lost due to context length |\n| **Upstream Data Issue** | A sibling or parent span provided bad data that cascaded to the symptom span |\n| **Runtime Error** | Tool\u002FAPI failure, timeout, exception — from `find_llmobs_error_spans` |\n| **Evaluator Miscalibration** | Eval criteria produce false positives\u002Fnegatives (Eval Signal mode only) |\n\n---\n\n### Phase 5: Generate Recommendations\n\n**Goal**: Concrete, actionable recommendations grounded in trace evidence. Actual text\u002Fcode changes with before\u002Fafter quotes from the trace — not generic advice.\n\nRecommendation types: **System Prompt Edit** (quote actual prompt, provide before\u002Fafter), **Tool Gap\u002FMisuse** (reference agent loop steps), **Routing\u002FHandoff Fix**, **Retrieval Fix** (show retrieved vs needed), **Evaluator Prompt Edit** (flag that eval changes need re-validation; Eval Signal only), **Other**.\n\n**When run in Claude Code with codebase access**: Search the codebase for system prompt, tool definitions, or routing logic. Propose specific diffs. Always ask before modifying files.\n\n---\n\n### Phase 6: Compile RCA Report\n\nWrite the full report following the Output Format below. **This is the primary deliverable — output it directly in the chat.**\n\n---\n\n### Phase 7: Post-Analysis Actions\n\n**Do NOT take any action automatically.** After presenting the report, ask the user what they'd like to do next:\n\n1. Save the report to `agent-observability-rca-{ml_app}-{date}.md`\n2. Apply fixes (if codebase is available)\n3. Deeper investigation of remaining categories\n4. Export to a Datadog notebook — in pup mode, use `pup notebooks create` to create the notebook and `pup notebooks edit NOTEBOOK_ID --file \u002Ftmp\u002Fnb_cells.json` to append sections (see Tool Reference)\n5. Re-run on an expanded time range (e.g. `now-7d` if current window was `now-24h`)\n\n**If the user chooses option 4**, follow the notebook creation fallback pattern:\n\n1. Call `mcp__datadog-mcp-core__create_datadog_notebook` with:\n   - **`name`**: `Agent Observability RCA: {ml_app} ({mode}) — YYYY-MM-DD`\n   - **`type`**: `report`\n   - **`time_span`**: `1w`\n   - **`cells`**: one cell per section (see Notebook Cell Structure below)\n\n2. **If the MCP call fails**, inspect the error before giving up:\n   - **Auth \u002F permission error (401, 403)** → stop and tell the user.\n   - **Field validation error** (error message names a specific field) → fix that field and retry the MCP call once.\n   - **Any other error** (binding, serialization, unexpected response) → fall back to pup:\n     - Write the notebook payload to `\u002Ftmp\u002Fnb_rca_{ml_app}.json` as a full API envelope: `{\"data\": {\"attributes\": {\"name\": \"...\", \"time\": {...}, \"cells\": [...]}, \"type\": \"notebooks\"}}`\n     - Run `pup notebooks create --file \u002Ftmp\u002Fnb_rca_{ml_app}.json`\n     - If pup is not available either, render the full notebook content as markdown in chat so the user has it.\n\n3. After successful creation by either method, output the URL on its own line:\n   `RCA report exported to notebook: \u003Curl>`\n\nPrint the URL prominently — if `\u002Feval-bootstrap` runs next in the same session, it will detect this URL and offer to append the evaluator suite to the same notebook.\n\n#### Notebook Cell Structure\n\n| Cell | Content |\n|------|---------|\n| 1 — Overview | Structured header (see Overview cell format below — follow it exactly) |\n| 2 — Signal Summary | Mode-specific health table |\n| 3 — Failure Taxonomy | Taxonomy table |\n| 4…N — Failure Modes | One cell per failure mode |\n| N+1 — Action Plan + Limitations | Action plan table + bullet list |\n\n**Notebook formatting rules** (apply to every cell):\n- **No triple-backtick code blocks** — use blockquotes (`>`) for prompts\u002Frubrics, inline code (`` ` ``) for short values\n- **Evidence as tables** — not bullet lists\n- **Tool inputs as tables** — Argument | Wrong value passed | Correct approach\n- **Action plan as a table** — Priority | Action | Confidence | Impact\n\n---\n\n## Output Format\n\n---\n\n### Overview cell (notebook Cell 1 \u002F report header)\n\nThe Overview cell must follow this exact structure. No prose paragraphs. No inline-numbered findings. App description is one sentence maximum.\n\n```\n## `{ml_app}`  ·  {Eval Signal | Error Signal | Generic}  ·  {timeframe}\n**Date**: {YYYY-MM-DD}  |  **Profile**: {short app profile}  |  **Model**: `{model(s)}`\n\n{One sentence: what does this app do?}\n\n| Metric | Value |\n|--------|-------|\n{mode-appropriate rows — see below}\n\n### Findings\n\n- **{Finding 1}** (~{pct}%): one-line root cause description\n- **{Finding 2}** (~{pct}%): one-line root cause description\n- **{Finding 3}** (if present): one-line root cause description\n\n### Recommendations\n\n- **{Recommendation 1}**: specific next step tied to Finding 1\n- **{Recommendation 2}**: specific next step tied to Finding 2\n\n*Sample: {N} spans analyzed. Confidence: High | Medium | Low — {one-line reason if Medium or Low}.*\n```\n\n**Mode-appropriate metric rows:**\n\nEval Signal:\n```\n| Eval | `{eval_name}` ({type}) |\n| Spans evaluated | {total_count} |\n| Pass rate | {pass_rate}% ({pass_count} pass \u002F {fail_count} fail) |\n| Top failure mode | {name} (~{pct}%) |\n| Evals configured | {N} |\n```\n\nError Signal:\n```\n| Error spans | {N} confirmed |\n| Top error type | `{type}` ({pct}%) |\n| Affected operation | `{span_name}` |\n| Cascade pattern | Isolated \u002F Cascading |\n| Evals configured | {N} (none = no quality signal) |\n```\n\nGeneric:\n```\n| Spans sampled | {N} root spans |\n| Top anomaly | {type}: {count} spans |\n| Error spans | {N} (0 = structurally healthy) |\n| Evals configured | {N} (none = no quality signal) |\n```\n\n---\n\n### Signal Summary Table\n\n**When entering from Step 0S (classification context)**, replace the Signal Summary table with:\n\n```\n## Classification Signal Summary\n\n**Source**: agent-observability-session-classify  |  **ml_app**: {app}  |  **Signal**: content-only | content+evals\n\n| Metric | Value |\n|--------|-------|\n| Traces classified | N |\n| Failures in corpus (no+partial) | F |\n| Pass rate | X% |\n| Failure modes | list |\n\n*Root cause analysis is based on per-trace classification verdicts, not automated eval judge reasoning.*\n```\n\n**Otherwise**, mode-specific — pick the appropriate variant:\n\n**Eval Signal** — one row per eval:\n\n| Eval | Type | Total | Pass Rate | Status |\n|------|------|------:|:---------:|--------|\n| eval_1 | boolean | 4,891 | 37.3% | ⚠ Investigating |\n\n**Error Signal** — one row per error type:\n\n| Error Type | Spans | Cascade? | Origin Span Kind |\n|------------|------:|:--------:|-----------------|\n| TimeoutError | 42 | Yes | tool |\n\n**Generic** — one row per anomaly type:\n\n| Anomaly Type | Count |\n|---|:---:|\n| Latency outliers (>p90) | 12 |\n\n---\n\n### Failure Taxonomy\n\n| # | Failure Mode | Traces | % | Severity | Root Cause |\n|---|-------------|-------:|:-:|:--------:|-----------|\n| 1 | ... | ... | ...% | **High** | Tool Misuse |\n\n---\n\n### Failure Mode Sections (one per top 3–5 modes)\n\n```\n## Failure Mode N: [Name]\n\n**Count**: {n} spans, {t} traces  |  **Severity**: High\u002FMedium\u002FLow  |  **Root Cause**: [Category]\n\n[3–5 sentences: what goes wrong, when, what triggers it, causal chain.]\n\n**Evidence**\n\n{Use the mode-appropriate column set:}\n\nEval Signal — Trace | Judge verdict | What the trace revealed:\n| Trace | Judge verdict | What the trace revealed |\n|---|---|---|\n| [69de86a7...](https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_id}) | fail | Parent agent has no date format instruction |\n\nError Signal — Trace | Behavior | Version:\n| Trace | Behavior | Version |\n|---|---|---|\n| [69de86a7...](https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_id}) | 7 parallel calls, all 400 | v107624932 |\n\nGeneric — Trace | Anomaly | Signal:\n| Trace | Anomaly | Signal |\n|---|---|---|\n| [69de86a7...](https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_id}) | 94s, 12 tool calls | Latency outlier |\n\n{For tool misuse — add a tool inputs table:}\n**Tool inputs (100% of sampled calls)**\n\n| Argument | Value passed (wrong) | Correct approach |\n|---|---|---|\n| `query` | `\"monitor_id:123 group_status:alert\"` | `\"monitor_id:123\"` (name\u002Ftag only) |\n\n{For Eval Signal — add judge reasoning as a blockquote:}\n> \"{quoted judge reasoning}\"\n\n**Root cause**: [WHY this happens — specific span, parameter, or prompt.]\n\n**Fix**:\n  BEFORE: [actual text from trace]\n  AFTER:  [proposed replacement]\n\n**Impact**: Eliminates ~{n} spans \u002F {timeframe}.\n```\n\n---\n\n### Prioritized Action Plan\n\n| Priority | Action | Confidence | Impact |\n|:---:|---|:---:|---|\n| 1 | Fix `monitor_groups_search` schema — add `group_states` param | High | Eliminates ~21 spans\u002F7d |\n\n**When mode is Generic and no evals are configured**, always append as the final action plan row:\n\n| N | Configure at least one evaluator | High | Enables Eval Signal mode for future RCAs — app currently has no ongoing quality signal |\n\n---\n\n### Limitations & Follow-ups\n\nBullet list of what needs more data or follow-up action.\n\n---\n\n## Operating Rules\n\n- **Ground in evidence**: Every claim references span IDs with clickable trace links: `[Trace {first_8}...](https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_32_char_id})`.\n- **Root cause over symptom**: \"System prompt doesn't specify date format\" not \"model gave wrong answer.\"\n- **Show your math**: \"47 failures (34%)\" not \"many failures.\"\n- **Honest about uncertainty**: \u003C 5 examples = tentative. Flag it.\n- **Anonymize PII**: No emails or names. User\u002Forg IDs are fine.\n- **MCP result parsing safety**: Before writing any script that iterates over MCP tool results, inspect the raw structure first — check top-level keys and whether the payload is nested inside a content block (e.g. `[{'type': 'text', 'text': '\u003Cjson>'}]`). Extract and `json.loads()` the inner payload if needed. Never assume MCP results are bare dicts or lists.\n\n---\n\n## Tool Reference\n\nThis appendix applies only in **pup mode**. In MCP mode, use the tool names in the workflow sections directly.\n\n### Spans and traces\n\n| MCP Tool | pup Command |\n|---|---|\n| `search_llmobs_spans(query, ml_app, from, to, limit, cursor, root_spans_only, span_kind, summary)` | `pup llm-obs spans search --query \"@ml_app:A [other_filters]\" [--from F] [--to T] [--limit N] [--cursor C] [--root-spans-only] [--span-kind K] [--summary]` — **always use `--query \"@ml_app:A\"` to filter by ml_app**; the `--ml-app A` flag is unreliable and silently returns spans from other apps. |\n| `get_llmobs_span_details(trace_id, span_ids, from, to)` | `pup llm-obs spans get-details --trace-id T --span-ids S1,S2,...` |\n| `get_llmobs_span_content(trace_id, span_id, field, path)` | `pup llm-obs spans get-content --trace-id T --span-id S --field F [--path P]` |\n| `get_llmobs_trace(trace_id, include_tree)` | `pup llm-obs spans get-trace --trace-id T [--include-tree]` |\n| `get_llmobs_agent_loop(trace_id, span_id)` | `pup llm-obs spans get-agent-loop --trace-id T [--span-id S]` |\n| `find_llmobs_error_spans(trace_id)` | `pup llm-obs spans find-errors --trace-id T` |\n| `expand_llmobs_spans(trace_id, span_ids, max_depth, filter_kind)` | `pup llm-obs spans expand --trace-id T --span-ids S1,S2,... [--max-depth N] [--filter-kind K]` |\n\n### Evaluators\n\n| MCP Tool | pup Command |\n|---|---|\n| `list_llmobs_evals()` | `pup llm-obs evals list` (filter by `ml_app` client-side) |\n| `list_llmobs_evals_by_ml_app(ml_app)` | `pup llm-obs evals list-by-ml-app --ml-app A` |\n| `get_llmobs_evaluator(eval_name)` | `pup llm-obs evals get-evaluator EVAL_NAME` |\n| `get_llmobs_eval_aggregate_stats(eval_name, ml_app, from, to)` | `pup llm-obs evals get-aggregate-stats EVAL_NAME [--ml-app A] [--from F] [--to T]` |\n\n### Notebooks\n\n| MCP Tool | pup Command |\n|---|---|\n| `create_datadog_notebook(name, cells, ...)` | `pup notebooks create --title \"TITLE\" --file \u002Ftmp\u002Fnb_cells.json` — confirm exact flags with `pup notebooks create --help` |\n| `edit_datadog_notebook(id, cells, append_only=true)` | `pup notebooks edit NOTEBOOK_ID --file \u002Ftmp\u002Fnb_cells.json` (fetches current notebook, appends provided cells, writes back) |\n\nThe cells file is a JSON array of cell objects:\n```json\n[{\"attributes\": {\"definition\": {\"type\": \"markdown\", \"text\": \"## Section\\n\\nContent.\"}}, \"type\": \"notebook_cells\"}]\n```\n",{"data":40,"body":41},{"name":4,"description":6},{"type":42,"children":43},"root",[44,53,65,178,188,196,355,373,428,435,447,559,571,577,585,591,603,609,842,861,867,874,961,967,1133,1146,1328,1339,1372,1377,1458,1505,1511,1588,1592,1598,1608,1611,1617,1642,1649,1668,1820,1844,1857,1862,1871,1883,1916,1919,1927,2158,2161,2167,2179,2197,2207,2210,2216,2223,2228,2249,2257,2340,2348,2399,2432,2438,2443,2527,2573,2579,2588,2593,2596,2602,2608,2619,2625,2653,2693,2699,2734,2740,2753,2771,2777,2800,2806,2815,2818,2824,2830,2863,2869,2909,2915,2925,2931,2936,2992,2998,3007,3010,3016,3034,3044,3214,3301,3311,3477,3494,3497,3503,3512,3522,3562,3572,3605,3669,3675,3684,3687,3693,3702,3753,3758,3767,3770,3776,3785,3790,3796,3801,3825,3835,3865,3871,3876,3893,3976,3990,4046,4091,4106,4140,4154,4188,4197,4211,4217,4222,4255,4265,4273,4443,4446,4452,4461,4506,4516,4519,4525,4535,4538,4544,4554,4617,4627,4808,4821,4827,4914,4924,4983,4986,4992,4995,5001,5006,5015,5023,5028,5037,5042,5051,5056,5065,5068,5074,5084,5093,5103,5112,5179,5188,5242,5251,5286,5289,5295,5371,5374,5380,5389,5392,5398,5467,5477,5482,5485,5491,5496,5499,5505,5591,5594,5600,5611,5617,5809,5815,5926,5932,6003,6008,6190],{"type":45,"tag":46,"props":47,"children":49},"element","h2",{"id":48},"backend",[50],{"type":51,"value":52},"text","Backend",{"type":45,"tag":54,"props":55,"children":56},"p",{},[57,63],{"type":45,"tag":58,"props":59,"children":60},"strong",{},[61],{"type":51,"value":62},"Detection",{"type":51,"value":64}," — At the start of every invocation, before taking any action, determine which backend to use:",{"type":45,"tag":66,"props":67,"children":68},"ol",{},[69,91,104,116,145,156],{"type":45,"tag":70,"props":71,"children":72},"li",{},[73,75,82,84,89],{"type":51,"value":74},"If the user passed ",{"type":45,"tag":76,"props":77,"children":79},"code",{"className":78},[],[80],{"type":51,"value":81},"--backend pup",{"type":51,"value":83}," anywhere in their invocation → use ",{"type":45,"tag":58,"props":85,"children":86},{},[87],{"type":51,"value":88},"pup mode",{"type":51,"value":90}," immediately, regardless of whether MCP tools are present. Skip steps 2–4.",{"type":45,"tag":70,"props":92,"children":93},{},[94,96,102],{"type":51,"value":95},"Check whether MCP tools are present in your active tool list. The canonical signal is whether ",{"type":45,"tag":76,"props":97,"children":99},{"className":98},[],[100],{"type":51,"value":101},"mcp__datadog-llmo-mcp__list_llmobs_evals",{"type":51,"value":103}," appears in your available tools.",{"type":45,"tag":70,"props":105,"children":106},{},[107,109,114],{"type":51,"value":108},"If MCP tools are present → use ",{"type":45,"tag":58,"props":110,"children":111},{},[112],{"type":51,"value":113},"MCP mode",{"type":51,"value":115}," throughout. Call MCP tools exactly as named in this skill's workflow sections.",{"type":45,"tag":70,"props":117,"children":118},{},[119,121,127,129,135,137,143],{"type":51,"value":120},"If MCP tools are absent → check whether ",{"type":45,"tag":76,"props":122,"children":124},{"className":123},[],[125],{"type":51,"value":126},"pup",{"type":51,"value":128}," is executable: run ",{"type":45,"tag":76,"props":130,"children":132},{"className":131},[],[133],{"type":51,"value":134},"pup --version",{"type":51,"value":136}," via Bash. A JSON response containing ",{"type":45,"tag":76,"props":138,"children":140},{"className":139},[],[141],{"type":51,"value":142},"\"version\"",{"type":51,"value":144}," confirms pup is available.",{"type":45,"tag":70,"props":146,"children":147},{},[148,150,154],{"type":51,"value":149},"If pup responds → use ",{"type":45,"tag":58,"props":151,"children":152},{},[153],{"type":51,"value":88},{"type":51,"value":155}," throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.",{"type":45,"tag":70,"props":157,"children":158},{},[159,161],{"type":51,"value":160},"If neither is available → stop and tell the user:\n",{"type":45,"tag":162,"props":163,"children":164},"blockquote",{},[165],{"type":45,"tag":54,"props":166,"children":167},{},[168,170,176],{"type":51,"value":169},"\"Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (",{"type":45,"tag":76,"props":171,"children":173},{"className":172},[],[174],{"type":51,"value":175},"claude mcp add --scope user --transport http datadog-llmo-mcp 'https:\u002F\u002Fmcp.datadoghq.com\u002Fapi\u002Funstable\u002Fmcp-server\u002Fmcp?toolsets=llmobs'",{"type":51,"value":177},") or install pup.\"",{"type":45,"tag":54,"props":179,"children":180},{},[181,186],{"type":45,"tag":76,"props":182,"children":184},{"className":183},[],[185],{"type":51,"value":81},{"type":51,"value":187}," is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.",{"type":45,"tag":54,"props":189,"children":190},{},[191],{"type":45,"tag":58,"props":192,"children":193},{},[194],{"type":51,"value":195},"pup invocation rules:",{"type":45,"tag":197,"props":198,"children":199},"ul",{},[200,211,224,237,242,336],{"type":45,"tag":70,"props":201,"children":202},{},[203,205],{"type":51,"value":204},"Invoke via Bash: ",{"type":45,"tag":76,"props":206,"children":208},{"className":207},[],[209],{"type":51,"value":210},"pup llm-obs \u003Csubcommand> [flags]",{"type":45,"tag":70,"props":212,"children":213},{},[214,216,222],{"type":51,"value":215},"pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in ",{"type":45,"tag":76,"props":217,"children":219},{"className":218},[],[220],{"type":51,"value":221},"[{\"type\": \"text\", \"text\": \"\u003Cjson>\"}]",{"type":51,"value":223},").",{"type":45,"tag":70,"props":225,"children":226},{},[227,229,235],{"type":51,"value":228},"If pup returns an auth error, tell the user to run ",{"type":45,"tag":76,"props":230,"children":232},{"className":231},[],[233],{"type":51,"value":234},"pup auth login",{"type":51,"value":236}," and stop.",{"type":45,"tag":70,"props":238,"children":239},{},[240],{"type":51,"value":241},"Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).",{"type":45,"tag":70,"props":243,"children":244},{},[245,247,253,255,261,262,268,270,275,277,283,285,291,293,299,301,306,307,313,314,320,321,327,328,334],{"type":51,"value":246},"Time flags: pup accepts bare duration strings (",{"type":45,"tag":76,"props":248,"children":250},{"className":249},[],[251],{"type":51,"value":252},"1h",{"type":51,"value":254},", ",{"type":45,"tag":76,"props":256,"children":258},{"className":257},[],[259],{"type":51,"value":260},"7d",{"type":51,"value":254},{"type":45,"tag":76,"props":263,"children":265},{"className":264},[],[266],{"type":51,"value":267},"30m",{"type":51,"value":269},") and RFC3339 timestamps. Do ",{"type":45,"tag":58,"props":271,"children":272},{},[273],{"type":51,"value":274},"not",{"type":51,"value":276}," use ",{"type":45,"tag":76,"props":278,"children":280},{"className":279},[],[281],{"type":51,"value":282},"now-",{"type":51,"value":284},"-prefixed strings — strip the prefix when converting from a skill ",{"type":45,"tag":76,"props":286,"children":288},{"className":287},[],[289],{"type":51,"value":290},"--timeframe",{"type":51,"value":292}," argument: ",{"type":45,"tag":76,"props":294,"children":296},{"className":295},[],[297],{"type":51,"value":298},"now-7d",{"type":51,"value":300}," → ",{"type":45,"tag":76,"props":302,"children":304},{"className":303},[],[305],{"type":51,"value":260},{"type":51,"value":254},{"type":45,"tag":76,"props":308,"children":310},{"className":309},[],[311],{"type":51,"value":312},"now-24h",{"type":51,"value":300},{"type":45,"tag":76,"props":315,"children":317},{"className":316},[],[318],{"type":51,"value":319},"24h",{"type":51,"value":254},{"type":45,"tag":76,"props":322,"children":324},{"className":323},[],[325],{"type":51,"value":326},"now-30d",{"type":51,"value":300},{"type":45,"tag":76,"props":329,"children":331},{"className":330},[],[332],{"type":51,"value":333},"30d",{"type":51,"value":335},".",{"type":45,"tag":70,"props":337,"children":338},{},[339,345,347,353],{"type":45,"tag":76,"props":340,"children":342},{"className":341},[],[343],{"type":51,"value":344},"--summary",{"type":51,"value":346}," on ",{"type":45,"tag":76,"props":348,"children":350},{"className":349},[],[351],{"type":51,"value":352},"pup llm-obs spans search",{"type":51,"value":354}," strips payload fields to essential metadata only. Use it in bulk\u002Fsearch phases where content is not needed.",{"type":45,"tag":54,"props":356,"children":357},{},[358,363,365,371],{"type":45,"tag":58,"props":359,"children":360},{},[361],{"type":51,"value":362},"Invocation ID:",{"type":51,"value":364}," At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., ",{"type":45,"tag":76,"props":366,"children":368},{"className":367},[],[369],{"type":51,"value":370},"3a9f1c2b",{"type":51,"value":372},"). Keep it constant for the entire invocation.",{"type":45,"tag":54,"props":374,"children":375},{},[376,381,383,389,391,397,399,404,406,412,414,420,422],{"type":45,"tag":58,"props":377,"children":378},{},[379],{"type":51,"value":380},"Intent tagging:",{"type":51,"value":382}," On every MCP tool call, prefix ",{"type":45,"tag":76,"props":384,"children":386},{"className":385},[],[387],{"type":51,"value":388},"telemetry.intent",{"type":51,"value":390}," with ",{"type":45,"tag":76,"props":392,"children":394},{"className":393},[],[395],{"type":51,"value":396},"skill:agent-observability-trace-rca[\u003Cinv_id>] — ",{"type":51,"value":398}," followed by a description of why the tool is being called. On the ",{"type":45,"tag":58,"props":400,"children":401},{},[402],{"type":51,"value":403},"first MCP tool call only",{"type":51,"value":405},", use ",{"type":45,"tag":76,"props":407,"children":409},{"className":408},[],[410],{"type":51,"value":411},"skill:agent-observability-trace-rca:start[\u003Cinv_id>] — ",{"type":51,"value":413}," instead (note the ",{"type":45,"tag":76,"props":415,"children":417},{"className":416},[],[418],{"type":51,"value":419},":start",{"type":51,"value":421}," suffix). Example first call: ",{"type":45,"tag":76,"props":423,"children":425},{"className":424},[],[426],{"type":51,"value":427},"skill:agent-observability-trace-rca:start[3a9f1c2b] — Phase 0: discover configured evals for task-cruncher to infer analysis mode",{"type":45,"tag":429,"props":430,"children":432},"h1",{"id":431},"agent-observability-trace-rca-root-cause-analysis-from-production-llm-traces",[433],{"type":51,"value":434},"Agent Observability Trace RCA — Root Cause Analysis from Production LLM Traces",{"type":45,"tag":54,"props":436,"children":437},{},[438,440,445],{"type":51,"value":439},"Diagnose ",{"type":45,"tag":58,"props":441,"children":442},{},[443],{"type":51,"value":444},"why an LLM application is failing",{"type":51,"value":446}," by searching production traces and walking the span tree from symptom to root cause. The skill automatically selects the best analysis mode based on available signals:",{"type":45,"tag":448,"props":449,"children":450},"table",{},[451,475],{"type":45,"tag":452,"props":453,"children":454},"thead",{},[455],{"type":45,"tag":456,"props":457,"children":458},"tr",{},[459,465,470],{"type":45,"tag":460,"props":461,"children":462},"th",{},[463],{"type":51,"value":464},"Mode",{"type":45,"tag":460,"props":466,"children":467},{},[468],{"type":51,"value":469},"Signal",{"type":45,"tag":460,"props":471,"children":472},{},[473],{"type":51,"value":474},"When auto-selected",{"type":45,"tag":476,"props":477,"children":478},"tbody",{},[479,501,530],{"type":45,"tag":456,"props":480,"children":481},{},[482,491,496],{"type":45,"tag":483,"props":484,"children":485},"td",{},[486],{"type":45,"tag":58,"props":487,"children":488},{},[489],{"type":51,"value":490},"Eval Signal",{"type":45,"tag":483,"props":492,"children":493},{},[494],{"type":51,"value":495},"LLM judge verdicts and reasoning (pass\u002Ffail rates, scoring)",{"type":45,"tag":483,"props":497,"children":498},{},[499],{"type":51,"value":500},"Evaluators are configured for the app",{"type":45,"tag":456,"props":502,"children":503},{},[504,512,525],{"type":45,"tag":483,"props":505,"children":506},{},[507],{"type":45,"tag":58,"props":508,"children":509},{},[510],{"type":51,"value":511},"Error Signal",{"type":45,"tag":483,"props":513,"children":514},{},[515,517,523],{"type":51,"value":516},"Runtime errors (",{"type":45,"tag":76,"props":518,"children":520},{"className":519},[],[521],{"type":51,"value":522},"@status:error",{"type":51,"value":524},", error types, stack traces)",{"type":45,"tag":483,"props":526,"children":527},{},[528],{"type":51,"value":529},"No evals configured, or user explicitly asks about errors\u002Fcrashes",{"type":45,"tag":456,"props":531,"children":532},{},[533,541,546],{"type":45,"tag":483,"props":534,"children":535},{},[536],{"type":45,"tag":58,"props":537,"children":538},{},[539],{"type":51,"value":540},"Generic",{"type":45,"tag":483,"props":542,"children":543},{},[544],{"type":51,"value":545},"Structural anomalies (latency, agent loops, retrieval misses)",{"type":45,"tag":483,"props":547,"children":548},{},[549,551,557],{"type":51,"value":550},"Explicit ",{"type":45,"tag":76,"props":552,"children":554},{"className":553},[],[555],{"type":51,"value":556},"mode=generic",{"type":51,"value":558}," override, or no strong signal found in Phase 1",{"type":45,"tag":54,"props":560,"children":561},{},[562,564,569],{"type":51,"value":563},"The mode is ",{"type":45,"tag":58,"props":565,"children":566},{},[567],{"type":51,"value":568},"announced (never asked)",{"type":51,"value":570}," in the first user-facing output with a one-line override hint.",{"type":45,"tag":46,"props":572,"children":574},{"id":573},"methodology",[575],{"type":51,"value":576},"Methodology",{"type":45,"tag":54,"props":578,"children":579},{},[580],{"type":45,"tag":58,"props":581,"children":582},{},[583],{"type":51,"value":584},"Resolve → Search → Observe → Open Coding → Axial Coding → Root Cause Navigation → Recommendations",{"type":45,"tag":46,"props":586,"children":588},{"id":587},"usage",[589],{"type":51,"value":590},"Usage",{"type":45,"tag":592,"props":593,"children":597},"pre",{"className":594,"code":596,"language":51},[595],"language-text","What's wrong with \u003Cml_app> over the last \u003Ctimeframe>\nWhy is \u003Cml_app> failing today\nAnalyze eval failures for \u003Ceval_name> on \u003Cml_app>\nLook at the errors on \u003Cml_app> over the last \u003Ctimeframe>\nRoot-cause low scores on \u003Ceval_name>\n",[598],{"type":45,"tag":76,"props":599,"children":601},{"__ignoreMap":600},"",[602],{"type":51,"value":596},{"type":45,"tag":46,"props":604,"children":606},{"id":605},"inputs",[607],{"type":51,"value":608},"Inputs",{"type":45,"tag":448,"props":610,"children":611},{},[612,638],{"type":45,"tag":452,"props":613,"children":614},{},[615],{"type":45,"tag":456,"props":616,"children":617},{},[618,623,628,633],{"type":45,"tag":460,"props":619,"children":620},{},[621],{"type":51,"value":622},"Input",{"type":45,"tag":460,"props":624,"children":625},{},[626],{"type":51,"value":627},"Required",{"type":45,"tag":460,"props":629,"children":630},{},[631],{"type":51,"value":632},"Default",{"type":45,"tag":460,"props":634,"children":635},{},[636],{"type":51,"value":637},"Description",{"type":45,"tag":476,"props":639,"children":640},{},[641,676,701,730,778],{"type":45,"tag":456,"props":642,"children":643},{},[644,653,666,671],{"type":45,"tag":483,"props":645,"children":646},{},[647],{"type":45,"tag":76,"props":648,"children":650},{"className":649},[],[651],{"type":51,"value":652},"ml_app",{"type":45,"tag":483,"props":654,"children":655},{},[656,658,664],{"type":51,"value":657},"Yes (or ",{"type":45,"tag":76,"props":659,"children":661},{"className":660},[],[662],{"type":51,"value":663},"eval_name",{"type":51,"value":665},")",{"type":45,"tag":483,"props":667,"children":668},{},[669],{"type":51,"value":670},"—",{"type":45,"tag":483,"props":672,"children":673},{},[674],{"type":51,"value":675},"The application to analyze.",{"type":45,"tag":456,"props":677,"children":678},{},[679,687,692,696],{"type":45,"tag":483,"props":680,"children":681},{},[682],{"type":45,"tag":76,"props":683,"children":685},{"className":684},[],[686],{"type":51,"value":663},{"type":45,"tag":483,"props":688,"children":689},{},[690],{"type":51,"value":691},"No",{"type":45,"tag":483,"props":693,"children":694},{},[695],{"type":51,"value":670},{"type":45,"tag":483,"props":697,"children":698},{},[699],{"type":51,"value":700},"One or more evaluators to focus on. Implies Eval Signal mode. Pass a list for multi-eval analysis.",{"type":45,"tag":456,"props":702,"children":703},{},[704,713,717,725],{"type":45,"tag":483,"props":705,"children":706},{},[707],{"type":45,"tag":76,"props":708,"children":710},{"className":709},[],[711],{"type":51,"value":712},"timeframe",{"type":45,"tag":483,"props":714,"children":715},{},[716],{"type":51,"value":691},{"type":45,"tag":483,"props":718,"children":719},{},[720],{"type":45,"tag":76,"props":721,"children":723},{"className":722},[],[724],{"type":51,"value":312},{"type":45,"tag":483,"props":726,"children":727},{},[728],{"type":51,"value":729},"How far back to look.",{"type":45,"tag":456,"props":731,"children":732},{},[733,742,746,751],{"type":45,"tag":483,"props":734,"children":735},{},[736],{"type":45,"tag":76,"props":737,"children":739},{"className":738},[],[740],{"type":51,"value":741},"mode",{"type":45,"tag":483,"props":743,"children":744},{},[745],{"type":51,"value":691},{"type":45,"tag":483,"props":747,"children":748},{},[749],{"type":51,"value":750},"inferred",{"type":45,"tag":483,"props":752,"children":753},{},[754,756,762,763,769,770,776],{"type":51,"value":755},"Explicit mode override: ",{"type":45,"tag":76,"props":757,"children":759},{"className":758},[],[760],{"type":51,"value":761},"eval",{"type":51,"value":254},{"type":45,"tag":76,"props":764,"children":766},{"className":765},[],[767],{"type":51,"value":768},"errors",{"type":51,"value":254},{"type":45,"tag":76,"props":771,"children":773},{"className":772},[],[774],{"type":51,"value":775},"generic",{"type":51,"value":777},". Skips inference entirely.",{"type":45,"tag":456,"props":779,"children":780},{},[781,790,794,798],{"type":45,"tag":483,"props":782,"children":783},{},[784],{"type":45,"tag":76,"props":785,"children":787},{"className":786},[],[788],{"type":51,"value":789},"failure_filter",{"type":45,"tag":483,"props":791,"children":792},{},[793],{"type":51,"value":691},{"type":45,"tag":483,"props":795,"children":796},{},[797],{"type":51,"value":670},{"type":45,"tag":483,"props":799,"children":800},{},[801,803,809,811,817,819,825,827,832,834,840],{"type":51,"value":802},"Narrowing scope: ",{"type":45,"tag":76,"props":804,"children":806},{"className":805},[],[807],{"type":51,"value":808},"\"errors\"",{"type":51,"value":810}," (routes to Error Signal path), ",{"type":45,"tag":76,"props":812,"children":814},{"className":813},[],[815],{"type":51,"value":816},"\"high latency\"",{"type":51,"value":818}," (post-fetch duration sort), ",{"type":45,"tag":76,"props":820,"children":822},{"className":821},[],[823],{"type":51,"value":824},"\"low scores on \u003Ceval>\"",{"type":51,"value":826}," (promotes to ",{"type":45,"tag":76,"props":828,"children":830},{"className":829},[],[831],{"type":51,"value":663},{"type":51,"value":833},"), a tool name or span name (",{"type":45,"tag":76,"props":835,"children":837},{"className":836},[],[838],{"type":51,"value":839},"@name:\u003Cx>",{"type":51,"value":841}," query).",{"type":45,"tag":54,"props":843,"children":844},{},[845,847,852,854,859],{"type":51,"value":846},"If neither ",{"type":45,"tag":76,"props":848,"children":850},{"className":849},[],[851],{"type":51,"value":652},{"type":51,"value":853}," nor ",{"type":45,"tag":76,"props":855,"children":857},{"className":856},[],[858],{"type":51,"value":663},{"type":51,"value":860}," is provided, ask the user.",{"type":45,"tag":46,"props":862,"children":864},{"id":863},"available-tools",[865],{"type":51,"value":866},"Available Tools",{"type":45,"tag":868,"props":869,"children":871},"h3",{"id":870},"eval-discovery-overview",[872],{"type":51,"value":873},"Eval discovery & overview",{"type":45,"tag":448,"props":875,"children":876},{},[877,893],{"type":45,"tag":452,"props":878,"children":879},{},[880],{"type":45,"tag":456,"props":881,"children":882},{},[883,888],{"type":45,"tag":460,"props":884,"children":885},{},[886],{"type":51,"value":887},"Tool",{"type":45,"tag":460,"props":889,"children":890},{},[891],{"type":51,"value":892},"Purpose",{"type":45,"tag":476,"props":894,"children":895},{},[896,920,937],{"type":45,"tag":456,"props":897,"children":898},{},[899,908],{"type":45,"tag":483,"props":900,"children":901},{},[902],{"type":45,"tag":76,"props":903,"children":905},{"className":904},[],[906],{"type":51,"value":907},"list_llmobs_evals",{"type":45,"tag":483,"props":909,"children":910},{},[911,913,918],{"type":51,"value":912},"Discover all configured evals for an ",{"type":45,"tag":76,"props":914,"children":916},{"className":915},[],[917],{"type":51,"value":652},{"type":51,"value":919},". Used in Phase 0 mode inference.",{"type":45,"tag":456,"props":921,"children":922},{},[923,932],{"type":45,"tag":483,"props":924,"children":925},{},[926],{"type":45,"tag":76,"props":927,"children":929},{"className":928},[],[930],{"type":51,"value":931},"get_llmobs_eval_aggregate_stats",{"type":45,"tag":483,"props":933,"children":934},{},[935],{"type":51,"value":936},"Pass\u002Ffail rate or score distribution for an eval over a time window.",{"type":45,"tag":456,"props":938,"children":939},{},[940,949],{"type":45,"tag":483,"props":941,"children":942},{},[943],{"type":45,"tag":76,"props":944,"children":946},{"className":945},[],[947],{"type":51,"value":948},"get_llmobs_evaluator",{"type":45,"tag":483,"props":950,"children":951},{},[952,954,960],{"type":51,"value":953},"Full evaluator config: prompt template, assessment criteria, span filter, sampling, provider. Use instead of the deprecated ",{"type":45,"tag":76,"props":955,"children":957},{"className":956},[],[958],{"type":51,"value":959},"get_llmobs_eval_config",{"type":51,"value":335},{"type":45,"tag":868,"props":962,"children":964},{"id":963},"trace-span-exploration",[965],{"type":51,"value":966},"Trace & span exploration",{"type":45,"tag":448,"props":968,"children":969},{},[970,984],{"type":45,"tag":452,"props":971,"children":972},{},[973],{"type":45,"tag":456,"props":974,"children":975},{},[976,980],{"type":45,"tag":460,"props":977,"children":978},{},[979],{"type":51,"value":887},{"type":45,"tag":460,"props":981,"children":982},{},[983],{"type":51,"value":892},{"type":45,"tag":476,"props":985,"children":986},{},[987,1004,1040,1065,1082,1099,1116],{"type":45,"tag":456,"props":988,"children":989},{},[990,999],{"type":45,"tag":483,"props":991,"children":992},{},[993],{"type":45,"tag":76,"props":994,"children":996},{"className":995},[],[997],{"type":51,"value":998},"search_llmobs_spans",{"type":45,"tag":483,"props":1000,"children":1001},{},[1002],{"type":51,"value":1003},"Find spans by tags, span kind, status, query syntax. Paginate with cursor. Entry point for Phase 1.",{"type":45,"tag":456,"props":1005,"children":1006},{},[1007,1016],{"type":45,"tag":483,"props":1008,"children":1009},{},[1010],{"type":45,"tag":76,"props":1011,"children":1013},{"className":1012},[],[1014],{"type":51,"value":1015},"get_llmobs_span_details",{"type":45,"tag":483,"props":1017,"children":1018},{},[1019,1021,1027,1029,1038],{"type":51,"value":1020},"Metadata, evaluations (scores, labels, reasoning), ",{"type":45,"tag":76,"props":1022,"children":1024},{"className":1023},[],[1025],{"type":51,"value":1026},"status",{"type":51,"value":1028},", error fields, duration, and ",{"type":45,"tag":58,"props":1030,"children":1031},{},[1032],{"type":45,"tag":76,"props":1033,"children":1035},{"className":1034},[],[1036],{"type":51,"value":1037},"content_info",{"type":51,"value":1039}," map showing available fields + sizes.",{"type":45,"tag":456,"props":1041,"children":1042},{},[1043,1052],{"type":45,"tag":483,"props":1044,"children":1045},{},[1046],{"type":45,"tag":76,"props":1047,"children":1049},{"className":1048},[],[1050],{"type":51,"value":1051},"get_llmobs_span_content",{"type":45,"tag":483,"props":1053,"children":1054},{},[1055,1057,1063],{"type":51,"value":1056},"Actual content for a span field. Supports JSONPath via ",{"type":45,"tag":76,"props":1058,"children":1060},{"className":1059},[],[1061],{"type":51,"value":1062},"path",{"type":51,"value":1064}," param for targeted extraction.",{"type":45,"tag":456,"props":1066,"children":1067},{},[1068,1077],{"type":45,"tag":483,"props":1069,"children":1070},{},[1071],{"type":45,"tag":76,"props":1072,"children":1074},{"className":1073},[],[1075],{"type":51,"value":1076},"get_llmobs_trace",{"type":45,"tag":483,"props":1078,"children":1079},{},[1080],{"type":51,"value":1081},"Full trace hierarchy as span tree with span counts by kind.",{"type":45,"tag":456,"props":1083,"children":1084},{},[1085,1094],{"type":45,"tag":483,"props":1086,"children":1087},{},[1088],{"type":45,"tag":76,"props":1089,"children":1091},{"className":1090},[],[1092],{"type":51,"value":1093},"find_llmobs_error_spans",{"type":45,"tag":483,"props":1095,"children":1096},{},[1097],{"type":51,"value":1098},"All error spans in a trace with error type, message, stack, and propagation context.",{"type":45,"tag":456,"props":1100,"children":1101},{},[1102,1111],{"type":45,"tag":483,"props":1103,"children":1104},{},[1105],{"type":45,"tag":76,"props":1106,"children":1108},{"className":1107},[],[1109],{"type":51,"value":1110},"expand_llmobs_spans",{"type":45,"tag":483,"props":1112,"children":1113},{},[1114],{"type":51,"value":1115},"Load children of collapsed trace nodes.",{"type":45,"tag":456,"props":1117,"children":1118},{},[1119,1128],{"type":45,"tag":483,"props":1120,"children":1121},{},[1122],{"type":45,"tag":76,"props":1123,"children":1125},{"className":1124},[],[1126],{"type":51,"value":1127},"get_llmobs_agent_loop",{"type":45,"tag":483,"props":1129,"children":1130},{},[1131],{"type":51,"value":1132},"Chronological agent execution timeline (LLM calls, tool invocations, decisions). May return empty — see Phase 4b fallback.",{"type":45,"tag":868,"props":1134,"children":1136},{"id":1135},"key-get_llmobs_span_content-patterns",[1137,1139,1144],{"type":51,"value":1138},"Key ",{"type":45,"tag":76,"props":1140,"children":1142},{"className":1141},[],[1143],{"type":51,"value":1051},{"type":51,"value":1145}," patterns",{"type":45,"tag":448,"props":1147,"children":1148},{},[1149,1170],{"type":45,"tag":452,"props":1150,"children":1151},{},[1152],{"type":45,"tag":456,"props":1153,"children":1154},{},[1155,1160,1165],{"type":45,"tag":460,"props":1156,"children":1157},{},[1158],{"type":51,"value":1159},"Field",{"type":45,"tag":460,"props":1161,"children":1162},{},[1163],{"type":51,"value":1164},"Path",{"type":45,"tag":460,"props":1166,"children":1167},{},[1168],{"type":51,"value":1169},"What you get",{"type":45,"tag":476,"props":1171,"children":1172},{},[1173,1207,1232,1257,1286,1307],{"type":45,"tag":456,"props":1174,"children":1175},{},[1176,1185,1194],{"type":45,"tag":483,"props":1177,"children":1178},{},[1179],{"type":45,"tag":76,"props":1180,"children":1182},{"className":1181},[],[1183],{"type":51,"value":1184},"messages",{"type":45,"tag":483,"props":1186,"children":1187},{},[1188],{"type":45,"tag":76,"props":1189,"children":1191},{"className":1190},[],[1192],{"type":51,"value":1193},"$.messages[0]",{"type":45,"tag":483,"props":1195,"children":1196},{},[1197,1199,1205],{"type":51,"value":1198},"System prompt (first message, usually ",{"type":45,"tag":76,"props":1200,"children":1202},{"className":1201},[],[1203],{"type":51,"value":1204},"system",{"type":51,"value":1206}," role)",{"type":45,"tag":456,"props":1208,"children":1209},{},[1210,1218,1227],{"type":45,"tag":483,"props":1211,"children":1212},{},[1213],{"type":45,"tag":76,"props":1214,"children":1216},{"className":1215},[],[1217],{"type":51,"value":1184},{"type":45,"tag":483,"props":1219,"children":1220},{},[1221],{"type":45,"tag":76,"props":1222,"children":1224},{"className":1223},[],[1225],{"type":51,"value":1226},"$.messages[-1]",{"type":45,"tag":483,"props":1228,"children":1229},{},[1230],{"type":51,"value":1231},"Last assistant response",{"type":45,"tag":456,"props":1233,"children":1234},{},[1235,1243,1252],{"type":45,"tag":483,"props":1236,"children":1237},{},[1238],{"type":45,"tag":76,"props":1239,"children":1241},{"className":1240},[],[1242],{"type":51,"value":1184},{"type":45,"tag":483,"props":1244,"children":1245},{},[1246],{"type":45,"tag":1247,"props":1248,"children":1249},"em",{},[1250],{"type":51,"value":1251},"(no path)",{"type":45,"tag":483,"props":1253,"children":1254},{},[1255],{"type":51,"value":1256},"Full conversation including tool calls",{"type":45,"tag":456,"props":1258,"children":1259},{},[1260,1277,1281],{"type":45,"tag":483,"props":1261,"children":1262},{},[1263,1269,1271],{"type":45,"tag":76,"props":1264,"children":1266},{"className":1265},[],[1267],{"type":51,"value":1268},"input",{"type":51,"value":1270}," \u002F ",{"type":45,"tag":76,"props":1272,"children":1274},{"className":1273},[],[1275],{"type":51,"value":1276},"output",{"type":45,"tag":483,"props":1278,"children":1279},{},[1280],{"type":51,"value":670},{"type":45,"tag":483,"props":1282,"children":1283},{},[1284],{"type":51,"value":1285},"Span I\u002FO",{"type":45,"tag":456,"props":1287,"children":1288},{},[1289,1298,1302],{"type":45,"tag":483,"props":1290,"children":1291},{},[1292],{"type":45,"tag":76,"props":1293,"children":1295},{"className":1294},[],[1296],{"type":51,"value":1297},"documents",{"type":45,"tag":483,"props":1299,"children":1300},{},[1301],{"type":51,"value":670},{"type":45,"tag":483,"props":1303,"children":1304},{},[1305],{"type":51,"value":1306},"Retrieved documents (RAG apps)",{"type":45,"tag":456,"props":1308,"children":1309},{},[1310,1319,1323],{"type":45,"tag":483,"props":1311,"children":1312},{},[1313],{"type":45,"tag":76,"props":1314,"children":1316},{"className":1315},[],[1317],{"type":51,"value":1318},"metadata",{"type":45,"tag":483,"props":1320,"children":1321},{},[1322],{"type":51,"value":670},{"type":45,"tag":483,"props":1324,"children":1325},{},[1326],{"type":51,"value":1327},"Custom metadata (prompt versions, feature flags, user segments)",{"type":45,"tag":868,"props":1329,"children":1331},{"id":1330},"how-to-use-search_llmobs_spans",[1332,1334],{"type":51,"value":1333},"How to use ",{"type":45,"tag":76,"props":1335,"children":1337},{"className":1336},[],[1338],{"type":51,"value":998},{"type":45,"tag":54,"props":1340,"children":1341},{},[1342,1370],{"type":45,"tag":58,"props":1343,"children":1344},{},[1345,1347,1353,1355,1361,1363,1368],{"type":51,"value":1346},"Always include ",{"type":45,"tag":76,"props":1348,"children":1350},{"className":1349},[],[1351],{"type":51,"value":1352},"@ml_app:\"\u003Cml_app>\"",{"type":51,"value":1354}," in the ",{"type":45,"tag":76,"props":1356,"children":1358},{"className":1357},[],[1359],{"type":51,"value":1360},"query",{"type":51,"value":1362}," string — the structured ",{"type":45,"tag":76,"props":1364,"children":1366},{"className":1365},[],[1367],{"type":51,"value":652},{"type":51,"value":1369}," parameter is unreliable and can return spans from other apps.",{"type":51,"value":1371}," Do not rely on the structured parameter alone.",{"type":45,"tag":54,"props":1373,"children":1374},{},[1375],{"type":51,"value":1376},"Useful query fragments — combine with space (AND):",{"type":45,"tag":448,"props":1378,"children":1379},{},[1380,1396],{"type":45,"tag":452,"props":1381,"children":1382},{},[1383],{"type":45,"tag":456,"props":1384,"children":1385},{},[1386,1391],{"type":45,"tag":460,"props":1387,"children":1388},{},[1389],{"type":51,"value":1390},"Goal",{"type":45,"tag":460,"props":1392,"children":1393},{},[1394],{"type":51,"value":1395},"Query",{"type":45,"tag":476,"props":1397,"children":1398},{},[1399,1415,1441],{"type":45,"tag":456,"props":1400,"children":1401},{},[1402,1407],{"type":45,"tag":483,"props":1403,"children":1404},{},[1405],{"type":51,"value":1406},"Errors only",{"type":45,"tag":483,"props":1408,"children":1409},{},[1410],{"type":45,"tag":76,"props":1411,"children":1413},{"className":1412},[],[1414],{"type":51,"value":522},{"type":45,"tag":456,"props":1416,"children":1417},{},[1418,1423],{"type":45,"tag":483,"props":1419,"children":1420},{},[1421],{"type":51,"value":1422},"Eval is present on the span",{"type":45,"tag":483,"props":1424,"children":1425},{},[1426,1432,1434,1439],{"type":45,"tag":76,"props":1427,"children":1429},{"className":1428},[],[1430],{"type":51,"value":1431},"@evaluations.custom.\u003Ceval_name>:*",{"type":51,"value":1433}," (presence only — pass\u002Ffail is read from ",{"type":45,"tag":76,"props":1435,"children":1437},{"className":1436},[],[1438],{"type":51,"value":1015},{"type":51,"value":1440},", not the query)",{"type":45,"tag":456,"props":1442,"children":1443},{},[1444,1449],{"type":45,"tag":483,"props":1445,"children":1446},{},[1447],{"type":51,"value":1448},"A specific tool by name",{"type":45,"tag":483,"props":1450,"children":1451},{},[1452],{"type":45,"tag":76,"props":1453,"children":1455},{"className":1454},[],[1456],{"type":51,"value":1457},"@name:\u003Ctool_name>",{"type":45,"tag":54,"props":1459,"children":1460},{},[1461,1463,1469,1470,1476,1477,1482,1484,1489,1491,1496,1498,1504],{"type":51,"value":1462},"Dedicated params (",{"type":45,"tag":76,"props":1464,"children":1466},{"className":1465},[],[1467],{"type":51,"value":1468},"span_kind",{"type":51,"value":254},{"type":45,"tag":76,"props":1471,"children":1473},{"className":1472},[],[1474],{"type":51,"value":1475},"root_spans_only",{"type":51,"value":254},{"type":45,"tag":76,"props":1478,"children":1480},{"className":1479},[],[1481],{"type":51,"value":652},{"type":51,"value":1483},") work alongside ",{"type":45,"tag":76,"props":1485,"children":1487},{"className":1486},[],[1488],{"type":51,"value":1360},{"type":51,"value":1490},", but ",{"type":45,"tag":76,"props":1492,"children":1494},{"className":1493},[],[1495],{"type":51,"value":1360},{"type":51,"value":1497}," takes precedence over ",{"type":45,"tag":76,"props":1499,"children":1501},{"className":1500},[],[1502],{"type":51,"value":1503},"tags",{"type":51,"value":335},{"type":45,"tag":868,"props":1506,"children":1508},{"id":1507},"parallelization-rules",[1509],{"type":51,"value":1510},"Parallelization rules",{"type":45,"tag":66,"props":1512,"children":1513},{},[1514,1533,1546,1571],{"type":45,"tag":70,"props":1515,"children":1516},{},[1517,1525,1527,1532],{"type":45,"tag":58,"props":1518,"children":1519},{},[1520],{"type":45,"tag":76,"props":1521,"children":1523},{"className":1522},[],[1524],{"type":51,"value":1015},{"type":51,"value":1526},": Group span_ids by trace_id, chunk each trace's span_ids into batches of at most 20. Issue ALL chunks for a page in a ",{"type":45,"tag":58,"props":1528,"children":1529},{},[1530],{"type":51,"value":1531},"single message",{"type":51,"value":335},{"type":45,"tag":70,"props":1534,"children":1535},{},[1536,1544],{"type":45,"tag":58,"props":1537,"children":1538},{},[1539],{"type":45,"tag":76,"props":1540,"children":1542},{"className":1541},[],[1543],{"type":51,"value":1051},{"type":51,"value":1545},": Each call is independent — always issue ALL in a single message.",{"type":45,"tag":70,"props":1547,"children":1548},{},[1549,1569],{"type":45,"tag":58,"props":1550,"children":1551},{},[1552,1557,1558,1563,1564],{"type":45,"tag":76,"props":1553,"children":1555},{"className":1554},[],[1556],{"type":51,"value":1076},{"type":51,"value":1270},{"type":45,"tag":76,"props":1559,"children":1561},{"className":1560},[],[1562],{"type":51,"value":1093},{"type":51,"value":1270},{"type":45,"tag":76,"props":1565,"children":1567},{"className":1566},[],[1568],{"type":51,"value":1127},{"type":51,"value":1570},": Parallelize across different traces in a single message.",{"type":45,"tag":70,"props":1572,"children":1573},{},[1574,1579,1581,1586],{"type":45,"tag":58,"props":1575,"children":1576},{},[1577],{"type":51,"value":1578},"Pipeline parallelism",{"type":51,"value":1580},": Start ",{"type":45,"tag":76,"props":1582,"children":1584},{"className":1583},[],[1585],{"type":51,"value":1015},{"type":51,"value":1587}," for page 1 results immediately — don't wait to collect all pages.",{"type":45,"tag":1589,"props":1590,"children":1591},"hr",{},[],{"type":45,"tag":46,"props":1593,"children":1595},{"id":1594},"analysis-workflow",[1596],{"type":51,"value":1597},"Analysis Workflow",{"type":45,"tag":54,"props":1599,"children":1600},{},[1601,1606],{"type":45,"tag":58,"props":1602,"children":1603},{},[1604],{"type":51,"value":1605},"Output discipline",{"type":51,"value":1607},": Phases 0–5 are internal analysis. The only user-facing outputs during these phases are the Phase 1 Signal Summary and the mandatory checkpoints at Phases 2 and 3. Do NOT narrate reasoning, summarize intermediate findings, or output Phase 4 deep-dive results as prose. All detailed findings go exclusively into the Phase 6 report.",{"type":45,"tag":1589,"props":1609,"children":1610},{},[],{"type":45,"tag":868,"props":1612,"children":1614},{"id":1613},"phase-0-resolve-inputs-infer-mode",[1615],{"type":51,"value":1616},"Phase 0: Resolve Inputs & Infer Mode",{"type":45,"tag":54,"props":1618,"children":1619},{},[1620,1625,1627,1633,1635,1640],{"type":45,"tag":58,"props":1621,"children":1622},{},[1623],{"type":51,"value":1624},"First: check for classification context.",{"type":51,"value":1626}," Scan the conversation for a ",{"type":45,"tag":76,"props":1628,"children":1630},{"className":1629},[],[1631],{"type":51,"value":1632},"# Session Classification Summary",{"type":51,"value":1634}," header. If found → enter ",{"type":45,"tag":58,"props":1636,"children":1637},{},[1638],{"type":51,"value":1639},"Step 0S",{"type":51,"value":1641}," below and skip all remaining Phase 0 steps and Phase 1 entirely.",{"type":45,"tag":1643,"props":1644,"children":1646},"h4",{"id":1645},"step-0s-extract-failure-bucket-from-classification-output",[1647],{"type":51,"value":1648},"Step 0S — Extract Failure Bucket from Classification Output",{"type":45,"tag":54,"props":1650,"children":1651},{},[1652,1654,1659,1661,1666],{"type":51,"value":1653},"The canonical handoff format is the ",{"type":45,"tag":58,"props":1655,"children":1656},{},[1657],{"type":51,"value":1658},"Per-Unit Details table",{"type":51,"value":1660}," inside the ",{"type":45,"tag":76,"props":1662,"children":1664},{"className":1663},[],[1665],{"type":51,"value":1632},{"type":51,"value":1667}," section. Extract one row per unit:",{"type":45,"tag":448,"props":1669,"children":1670},{},[1671,1686],{"type":45,"tag":452,"props":1672,"children":1673},{},[1674],{"type":45,"tag":456,"props":1675,"children":1676},{},[1677,1681],{"type":45,"tag":460,"props":1678,"children":1679},{},[1680],{"type":51,"value":1159},{"type":45,"tag":460,"props":1682,"children":1683},{},[1684],{"type":51,"value":1685},"Source",{"type":45,"tag":476,"props":1687,"children":1688},{},[1689,1722,1739,1764,1781],{"type":45,"tag":456,"props":1690,"children":1691},{},[1692,1701],{"type":45,"tag":483,"props":1693,"children":1694},{},[1695],{"type":45,"tag":76,"props":1696,"children":1698},{"className":1697},[],[1699],{"type":51,"value":1700},"trace_id",{"type":45,"tag":483,"props":1702,"children":1703},{},[1704,1706,1712,1714,1720],{"type":51,"value":1705},"Link URL in the ID column: parse the ",{"type":45,"tag":76,"props":1707,"children":1709},{"className":1708},[],[1710],{"type":51,"value":1711},"trace_id=",{"type":51,"value":1713}," or ",{"type":45,"tag":76,"props":1715,"children":1717},{"className":1716},[],[1718],{"type":51,"value":1719},"session_id=",{"type":51,"value":1721}," query parameter from the link href",{"type":45,"tag":456,"props":1723,"children":1724},{},[1725,1734],{"type":45,"tag":483,"props":1726,"children":1727},{},[1728],{"type":45,"tag":76,"props":1729,"children":1731},{"className":1730},[],[1732],{"type":51,"value":1733},"verdict",{"type":45,"tag":483,"props":1735,"children":1736},{},[1737],{"type":51,"value":1738},"Verdict column",{"type":45,"tag":456,"props":1740,"children":1741},{},[1742,1751],{"type":45,"tag":483,"props":1743,"children":1744},{},[1745],{"type":45,"tag":76,"props":1746,"children":1748},{"className":1747},[],[1749],{"type":51,"value":1750},"failure_mode",{"type":45,"tag":483,"props":1752,"children":1753},{},[1754,1756,1762],{"type":51,"value":1755},"Failure Mode column (",{"type":45,"tag":76,"props":1757,"children":1759},{"className":1758},[],[1760],{"type":51,"value":1761},"none",{"type":51,"value":1763}," for passing rows)",{"type":45,"tag":456,"props":1765,"children":1766},{},[1767,1776],{"type":45,"tag":483,"props":1768,"children":1769},{},[1770],{"type":45,"tag":76,"props":1771,"children":1773},{"className":1772},[],[1774],{"type":51,"value":1775},"detail",{"type":45,"tag":483,"props":1777,"children":1778},{},[1779],{"type":51,"value":1780},"Reason column — use as the Phase 2 reasoning input (same role as eval judge reasoning or error messages)",{"type":45,"tag":456,"props":1782,"children":1783},{},[1784,1793],{"type":45,"tag":483,"props":1785,"children":1786},{},[1787],{"type":45,"tag":76,"props":1788,"children":1790},{"className":1789},[],[1791],{"type":51,"value":1792},"app_type",{"type":45,"tag":483,"props":1794,"children":1795},{},[1796,1798,1803,1805,1811,1813,1818],{"type":51,"value":1797},"From the ",{"type":45,"tag":76,"props":1799,"children":1801},{"className":1800},[],[1802],{"type":51,"value":1632},{"type":51,"value":1804}," header line (e.g. ",{"type":45,"tag":76,"props":1806,"children":1808},{"className":1807},[],[1809],{"type":51,"value":1810},"Root span kind: agent",{"type":51,"value":1812},") — default ",{"type":45,"tag":76,"props":1814,"children":1816},{"className":1815},[],[1817],{"type":51,"value":17},{"type":51,"value":1819}," if absent",{"type":45,"tag":54,"props":1821,"children":1822},{},[1823,1828,1830,1836,1837,1843],{"type":45,"tag":58,"props":1824,"children":1825},{},[1826],{"type":51,"value":1827},"Failure bucket",{"type":51,"value":1829}," = all rows where verdict is ",{"type":45,"tag":76,"props":1831,"children":1833},{"className":1832},[],[1834],{"type":51,"value":1835},"no",{"type":51,"value":1713},{"type":45,"tag":76,"props":1838,"children":1840},{"className":1839},[],[1841],{"type":51,"value":1842},"partial",{"type":51,"value":335},{"type":45,"tag":197,"props":1845,"children":1846},{},[1847,1852],{"type":45,"tag":70,"props":1848,"children":1849},{},[1850],{"type":51,"value":1851},"\u003C 5 entries → note low confidence, proceed anyway.",{"type":45,"tag":70,"props":1853,"children":1854},{},[1855],{"type":51,"value":1856},"Empty → report \"No failures found in the classification output\" and stop.",{"type":45,"tag":54,"props":1858,"children":1859},{},[1860],{"type":51,"value":1861},"Present this overview before proceeding:",{"type":45,"tag":592,"props":1863,"children":1866},{"className":1864,"code":1865,"language":51},[595],"## Classification Overview (from agent-observability-session-classify)\n\n**ml_app**: \u003Cfrom summary header>  |  **Classified**: N  |  **Failures (no+partial)**: F  |  **Pass rate**: X%\n\n| Failure Mode | Count |\n|---|---|\n...\n\nProceeding to Phase 2 using F failure traces. Mode inference bypassed — classification verdict is the signal.\n",[1867],{"type":45,"tag":76,"props":1868,"children":1869},{"__ignoreMap":600},[1870],{"type":51,"value":1865},{"type":45,"tag":54,"props":1872,"children":1873},{},[1874,1876,1881],{"type":51,"value":1875},"Then ",{"type":45,"tag":58,"props":1877,"children":1878},{},[1879],{"type":51,"value":1880},"skip Phase 1 and jump directly to Phase 2",{"type":51,"value":1882},". Carry forward:",{"type":45,"tag":197,"props":1884,"children":1885},{},[1886,1899,1911],{"type":45,"tag":70,"props":1887,"children":1888},{},[1889,1891,1897],{"type":51,"value":1890},"Phase 2 reasoning input: ",{"type":45,"tag":76,"props":1892,"children":1894},{"className":1893},[],[1895],{"type":51,"value":1896},"(trace_id, span_id, detail)",{"type":51,"value":1898}," tuples — same structure as eval reasoning or error messages",{"type":45,"tag":70,"props":1900,"children":1901},{},[1902,1904,1909],{"type":51,"value":1903},"Phase 4 navigation: use ",{"type":45,"tag":76,"props":1905,"children":1907},{"className":1906},[],[1908],{"type":51,"value":1792},{"type":51,"value":1910}," from each trace block to choose the span navigation strategy",{"type":45,"tag":70,"props":1912,"children":1913},{},[1914],{"type":51,"value":1915},"Phases 2–7: run completely unchanged — the failure bucket structure is identical regardless of source",{"type":45,"tag":1589,"props":1917,"children":1918},{},[],{"type":45,"tag":54,"props":1920,"children":1921},{},[1922],{"type":45,"tag":58,"props":1923,"children":1924},{},[1925],{"type":51,"value":1926},"Standard resolution (no classification context):",{"type":45,"tag":66,"props":1928,"children":1929},{},[1930,1968,1986,2060,2139],{"type":45,"tag":70,"props":1931,"children":1932},{},[1933,1934,1939,1940,1945,1947,1952,1954,1959,1961,1966],{"type":51,"value":846},{"type":45,"tag":76,"props":1935,"children":1937},{"className":1936},[],[1938],{"type":51,"value":652},{"type":51,"value":853},{"type":45,"tag":76,"props":1941,"children":1943},{"className":1942},[],[1944],{"type":51,"value":663},{"type":51,"value":1946}," provided → ask the user. If ",{"type":45,"tag":76,"props":1948,"children":1950},{"className":1949},[],[1951],{"type":51,"value":663},{"type":51,"value":1953}," is provided but ",{"type":45,"tag":76,"props":1955,"children":1957},{"className":1956},[],[1958],{"type":51,"value":652},{"type":51,"value":1960}," is not → also ask for ",{"type":45,"tag":76,"props":1962,"children":1964},{"className":1963},[],[1965],{"type":51,"value":652},{"type":51,"value":1967}," (eval names are not globally unique; without it, span searches return results from all apps sharing the eval name).",{"type":45,"tag":70,"props":1969,"children":1970},{},[1971,1973,1978,1980,1985],{"type":51,"value":1972},"If ",{"type":45,"tag":76,"props":1974,"children":1976},{"className":1975},[],[1977],{"type":51,"value":712},{"type":51,"value":1979}," not provided → default to ",{"type":45,"tag":76,"props":1981,"children":1983},{"className":1982},[],[1984],{"type":51,"value":312},{"type":51,"value":335},{"type":45,"tag":70,"props":1987,"children":1988},{},[1989,1999,2001],{"type":45,"tag":58,"props":1990,"children":1991},{},[1992,1994],{"type":51,"value":1993},"Resolve ",{"type":45,"tag":76,"props":1995,"children":1997},{"className":1996},[],[1998],{"type":51,"value":789},{"type":51,"value":2000}," (before mode inference):\n",{"type":45,"tag":197,"props":2002,"children":2003},{},[2004,2020,2038,2048],{"type":45,"tag":70,"props":2005,"children":2006},{},[2007,2012,2014,2018],{"type":45,"tag":76,"props":2008,"children":2010},{"className":2009},[],[2011],{"type":51,"value":808},{"type":51,"value":2013}," → force ",{"type":45,"tag":58,"props":2015,"children":2016},{},[2017],{"type":51,"value":511},{"type":51,"value":2019}," mode",{"type":45,"tag":70,"props":2021,"children":2022},{},[2023,2028,2030,2036],{"type":45,"tag":76,"props":2024,"children":2026},{"className":2025},[],[2027],{"type":51,"value":824},{"type":51,"value":2029}," → treat as ",{"type":45,"tag":76,"props":2031,"children":2033},{"className":2032},[],[2034],{"type":51,"value":2035},"eval_name=\u003Ceval>",{"type":51,"value":2037},", then continue inference",{"type":45,"tag":70,"props":2039,"children":2040},{},[2041,2046],{"type":45,"tag":76,"props":2042,"children":2044},{"className":2043},[],[2045],{"type":51,"value":816},{"type":51,"value":2047}," → note for Phase 1 (sort by duration post-fetch); continue inference",{"type":45,"tag":70,"props":2049,"children":2050},{},[2051,2053,2058],{"type":51,"value":2052},"Tool\u002Fspan name → note as ",{"type":45,"tag":76,"props":2054,"children":2056},{"className":2055},[],[2057],{"type":51,"value":839},{"type":51,"value":2059}," query fragment for Phase 1; continue inference",{"type":45,"tag":70,"props":2061,"children":2062},{},[2063,2068,2070,2075,2077],{"type":45,"tag":58,"props":2064,"children":2065},{},[2066],{"type":51,"value":2067},"Resolve mode",{"type":51,"value":2069}," (skip if ",{"type":45,"tag":76,"props":2071,"children":2073},{"className":2072},[],[2074],{"type":51,"value":741},{"type":51,"value":2076}," was explicitly provided):\n",{"type":45,"tag":197,"props":2078,"children":2079},{},[2080,2094,2103],{"type":45,"tag":70,"props":2081,"children":2082},{},[2083,2088,2090],{"type":45,"tag":76,"props":2084,"children":2086},{"className":2085},[],[2087],{"type":51,"value":663},{"type":51,"value":2089}," given → ",{"type":45,"tag":58,"props":2091,"children":2092},{},[2093],{"type":51,"value":490},{"type":45,"tag":70,"props":2095,"children":2096},{},[2097,2099],{"type":51,"value":2098},"User explicitly mentioned errors\u002Fexceptions\u002Fcrashes → ",{"type":45,"tag":58,"props":2100,"children":2101},{},[2102],{"type":51,"value":511},{"type":45,"tag":70,"props":2104,"children":2105},{},[2106,2108,2114,2116],{"type":51,"value":2107},"Otherwise → call ",{"type":45,"tag":76,"props":2109,"children":2111},{"className":2110},[],[2112],{"type":51,"value":2113},"list_llmobs_evals_by_ml_app(ml_app)",{"type":51,"value":2115},":\n",{"type":45,"tag":197,"props":2117,"children":2118},{},[2119,2128],{"type":45,"tag":70,"props":2120,"children":2121},{},[2122,2124],{"type":51,"value":2123},"Evals returned → ",{"type":45,"tag":58,"props":2125,"children":2126},{},[2127],{"type":51,"value":490},{"type":45,"tag":70,"props":2129,"children":2130},{},[2131,2133,2137],{"type":51,"value":2132},"No evals → ",{"type":45,"tag":58,"props":2134,"children":2135},{},[2136],{"type":51,"value":511},{"type":51,"value":2138}," (announce auto-selection in Phase 1)",{"type":45,"tag":70,"props":2140,"children":2141},{},[2142,2144,2149,2151,2157],{"type":51,"value":2143},"When ",{"type":45,"tag":76,"props":2145,"children":2147},{"className":2146},[],[2148],{"type":51,"value":663},{"type":51,"value":2150}," is multi-valued, note for Phase 1: run parallel per-eval searches and merge+dedup by ",{"type":45,"tag":76,"props":2152,"children":2154},{"className":2153},[],[2155],{"type":51,"value":2156},"(trace_id, span_id)",{"type":51,"value":335},{"type":45,"tag":1589,"props":2159,"children":2160},{},[],{"type":45,"tag":868,"props":2162,"children":2164},{"id":2163},"phase-1-find-problematic-spans",[2165],{"type":51,"value":2166},"Phase 1: Find Problematic Spans",{"type":45,"tag":54,"props":2168,"children":2169},{},[2170,2172,2177],{"type":51,"value":2171},"Three mode-specific paths. All end with a ",{"type":45,"tag":58,"props":2173,"children":2174},{},[2175],{"type":51,"value":2176},"Signal Summary",{"type":51,"value":2178}," that labels the mode and includes a one-line override hint.",{"type":45,"tag":54,"props":2180,"children":2181},{},[2182,2187,2189,2195],{"type":45,"tag":58,"props":2183,"children":2184},{},[2185],{"type":51,"value":2186},"Mode switch handling",{"type":51,"value":2188},": At any checkpoint, if the user says \"switch to ",{"type":45,"tag":2190,"props":2191,"children":2192},"span",{},[2193],{"type":51,"value":2194},"error|eval|generic",{"type":51,"value":2196}," mode\", re-enter Phase 1 with the new mode. Phase 0 inputs do not re-resolve.",{"type":45,"tag":54,"props":2198,"children":2199},{},[2200,2205],{"type":45,"tag":58,"props":2201,"children":2202},{},[2203],{"type":51,"value":2204},"Auto-pivot",{"type":51,"value":2206},": If the selected mode finds no data (0 evals configured, 0 error spans in timeframe), announce the pivot to Generic and proceed — do not stop and ask.",{"type":45,"tag":1589,"props":2208,"children":2209},{},[],{"type":45,"tag":1643,"props":2211,"children":2213},{"id":2212},"eval-signal-path",[2214],{"type":51,"value":2215},"Eval Signal path",{"type":45,"tag":2217,"props":2218,"children":2220},"h5",{"id":2219},"step-1a-eval-overview-parallel",[2221],{"type":51,"value":2222},"Step 1a: Eval overview (parallel)",{"type":45,"tag":54,"props":2224,"children":2225},{},[2226],{"type":51,"value":2227},"For each eval, call both in a single parallel batch:",{"type":45,"tag":197,"props":2229,"children":2230},{},[2231,2240],{"type":45,"tag":70,"props":2232,"children":2233},{},[2234],{"type":45,"tag":76,"props":2235,"children":2237},{"className":2236},[],[2238],{"type":51,"value":2239},"get_llmobs_eval_aggregate_stats(eval_name, from, to)",{"type":45,"tag":70,"props":2241,"children":2242},{},[2243],{"type":45,"tag":76,"props":2244,"children":2246},{"className":2245},[],[2247],{"type":51,"value":2248},"get_llmobs_evaluator(eval_name)",{"type":45,"tag":54,"props":2250,"children":2251},{},[2252],{"type":45,"tag":58,"props":2253,"children":2254},{},[2255],{"type":51,"value":2256},"Interpret aggregate stats:",{"type":45,"tag":197,"props":2258,"children":2259},{},[2260,2274,2290,2300,2310,2320,2330],{"type":45,"tag":70,"props":2261,"children":2262},{},[2263,2272],{"type":45,"tag":58,"props":2264,"children":2265},{},[2266],{"type":45,"tag":76,"props":2267,"children":2269},{"className":2268},[],[2270],{"type":51,"value":2271},"total_count == 0",{"type":51,"value":2273}," → Note \"no data.\" Skip this eval (or pivot to Generic if it's the only one).",{"type":45,"tag":70,"props":2275,"children":2276},{},[2277,2288],{"type":45,"tag":58,"props":2278,"children":2279},{},[2280,2282],{"type":51,"value":2281},"Boolean ",{"type":45,"tag":76,"props":2283,"children":2285},{"className":2284},[],[2286],{"type":51,"value":2287},"pass_rate == 1.0",{"type":51,"value":2289}," → Note \"100% pass.\" Skip unless it's the only eval.",{"type":45,"tag":70,"props":2291,"children":2292},{},[2293,2298],{"type":45,"tag":58,"props":2294,"children":2295},{},[2296],{"type":51,"value":2297},"Boolean with failures",{"type":51,"value":2299}," → Note counts and pass_rate. Continue.",{"type":45,"tag":70,"props":2301,"children":2302},{},[2303,2308],{"type":45,"tag":58,"props":2304,"children":2305},{},[2306],{"type":51,"value":2307},"Score with assessment criteria",{"type":51,"value":2309}," → Note distribution and pass\u002Ffail counts. Continue.",{"type":45,"tag":70,"props":2311,"children":2312},{},[2313,2318],{"type":45,"tag":58,"props":2314,"children":2315},{},[2316],{"type":51,"value":2317},"Score WITHOUT assessment criteria",{"type":51,"value":2319}," → Infer failures: bottom quartile, or below median if bimodal. Label as \"inferred failures\" in report.",{"type":45,"tag":70,"props":2321,"children":2322},{},[2323,2328],{"type":45,"tag":58,"props":2324,"children":2325},{},[2326],{"type":51,"value":2327},"Categorical with assessment criteria",{"type":51,"value":2329}," → Note top_values and pass\u002Ffail. Continue.",{"type":45,"tag":70,"props":2331,"children":2332},{},[2333,2338],{"type":45,"tag":58,"props":2334,"children":2335},{},[2336],{"type":51,"value":2337},"Categorical WITHOUT assessment criteria",{"type":51,"value":2339}," → Infer from context (e.g., \"error\", \"incomplete\", \"off_topic\" are likely failures). Ask user if genuinely ambiguous.",{"type":45,"tag":54,"props":2341,"children":2342},{},[2343],{"type":45,"tag":58,"props":2344,"children":2345},{},[2346],{"type":51,"value":2347},"Interpret eval config:",{"type":45,"tag":197,"props":2349,"children":2350},{},[2351,2389],{"type":45,"tag":70,"props":2352,"children":2353},{},[2354,2359,2361,2367,2368,2374,2375,2381,2382,2388],{"type":45,"tag":58,"props":2355,"children":2356},{},[2357],{"type":51,"value":2358},"Config returned",{"type":51,"value":2360}," (custom) → Store ",{"type":45,"tag":76,"props":2362,"children":2364},{"className":2363},[],[2365],{"type":51,"value":2366},"prompt_template",{"type":51,"value":254},{"type":45,"tag":76,"props":2369,"children":2371},{"className":2370},[],[2372],{"type":51,"value":2373},"assessment_criteria",{"type":51,"value":254},{"type":45,"tag":76,"props":2376,"children":2378},{"className":2377},[],[2379],{"type":51,"value":2380},"parsing_type",{"type":51,"value":254},{"type":45,"tag":76,"props":2383,"children":2385},{"className":2384},[],[2386],{"type":51,"value":2387},"output_schema",{"type":51,"value":335},{"type":45,"tag":70,"props":2390,"children":2391},{},[2392,2397],{"type":45,"tag":58,"props":2393,"children":2394},{},[2395],{"type":51,"value":2396},"Config nil",{"type":51,"value":2398}," (OOTB) → Note prompt is not inspectable.",{"type":45,"tag":54,"props":2400,"children":2401},{},[2402,2407,2409,2415,2417,2423,2425,2430],{"type":45,"tag":58,"props":2403,"children":2404},{},[2405],{"type":51,"value":2406},"Calibration cross-check",{"type":51,"value":2408},": When two evals share a name prefix but differ in type (e.g. ",{"type":45,"tag":76,"props":2410,"children":2412},{"className":2411},[],[2413],{"type":51,"value":2414},"foo-boolean",{"type":51,"value":2416}," and ",{"type":45,"tag":76,"props":2418,"children":2420},{"className":2419},[],[2421],{"type":51,"value":2422},"foo-score",{"type":51,"value":2424},"), compare their pass rates on overlapping spans. A discrepancy >20% is an ",{"type":45,"tag":58,"props":2426,"children":2427},{},[2428],{"type":51,"value":2429},"Evaluator Calibration Discrepancy",{"type":51,"value":2431}," — flag it in the report.",{"type":45,"tag":2217,"props":2433,"children":2435},{"id":2434},"step-1b-collect-failure-spans",[2436],{"type":51,"value":2437},"Step 1b: Collect failure spans",{"type":45,"tag":54,"props":2439,"children":2440},{},[2441],{"type":51,"value":2442},"For each eval:",{"type":45,"tag":66,"props":2444,"children":2445},{},[2446,2463,2468,2478,2522],{"type":45,"tag":70,"props":2447,"children":2448},{},[2449,2455,2457,2462],{"type":45,"tag":76,"props":2450,"children":2452},{"className":2451},[],[2453],{"type":51,"value":2454},"search_llmobs_spans(query=\"@evaluations.custom.\u003Ceval_name>:*\", from, limit=50)",{"type":51,"value":2456},". When multi-valued, issue one search per eval in parallel — merge result sets, dedup by ",{"type":45,"tag":76,"props":2458,"children":2460},{"className":2459},[],[2461],{"type":51,"value":2156},{"type":51,"value":335},{"type":45,"tag":70,"props":2464,"children":2465},{},[2466],{"type":51,"value":2467},"Paginate until ≥15–20 failures OR no more pages. Cap at 200 spans total.",{"type":45,"tag":70,"props":2469,"children":2470},{},[2471,2476],{"type":45,"tag":76,"props":2472,"children":2474},{"className":2473},[],[2475],{"type":51,"value":1015},{"type":51,"value":2477}," per trace_id batch (follow Parallelization Rules).",{"type":45,"tag":70,"props":2479,"children":2480},{},[2481,2483,2488,2489,2494,2495,2500,2501,2506,2507,2511,2512,2516,2517,2521],{"type":51,"value":2482},"Extract per row: ",{"type":45,"tag":58,"props":2484,"children":2485},{},[2486],{"type":51,"value":2487},"assessment",{"type":51,"value":254},{"type":45,"tag":58,"props":2490,"children":2491},{},[2492],{"type":51,"value":2493},"value",{"type":51,"value":254},{"type":45,"tag":58,"props":2496,"children":2497},{},[2498],{"type":51,"value":2499},"reasoning",{"type":51,"value":254},{"type":45,"tag":58,"props":2502,"children":2503},{},[2504],{"type":51,"value":2505},"span_id",{"type":51,"value":254},{"type":45,"tag":58,"props":2508,"children":2509},{},[2510],{"type":51,"value":1700},{"type":51,"value":254},{"type":45,"tag":58,"props":2513,"children":2514},{},[2515],{"type":51,"value":1468},{"type":51,"value":254},{"type":45,"tag":58,"props":2518,"children":2519},{},[2520],{"type":51,"value":1037},{"type":51,"value":335},{"type":45,"tag":70,"props":2523,"children":2524},{},[2525],{"type":51,"value":2526},"Separate into pass\u002Ffail buckets using thresholds from Step 1a.",{"type":45,"tag":54,"props":2528,"children":2529},{},[2530,2535,2537,2542,2544,2549,2551,2557,2559,2564,2566,2572],{"type":45,"tag":58,"props":2531,"children":2532},{},[2533],{"type":51,"value":2534},"JSON-type eval fallback",{"type":51,"value":2536},": If ",{"type":45,"tag":76,"props":2538,"children":2540},{"className":2539},[],[2541],{"type":51,"value":1431},{"type":51,"value":2543}," returns 0 spans but ",{"type":45,"tag":76,"props":2545,"children":2547},{"className":2546},[],[2548],{"type":51,"value":931},{"type":51,"value":2550}," confirmed ",{"type":45,"tag":76,"props":2552,"children":2554},{"className":2553},[],[2555],{"type":51,"value":2556},"total_count > 0",{"type":51,"value":2558},", the eval is JSON-type and scores are not indexed on this field. Fall back to: search by the span name or span kind that the eval targets (check ",{"type":45,"tag":76,"props":2560,"children":2562},{"className":2561},[],[2563],{"type":51,"value":948},{"type":51,"value":2565}," for the span filter), then inspect output payloads for JSON verdict fields via ",{"type":45,"tag":76,"props":2567,"children":2569},{"className":2568},[],[2570],{"type":51,"value":2571},"get_llmobs_span_content(field=\"output\")",{"type":51,"value":335},{"type":45,"tag":2217,"props":2574,"children":2576},{"id":2575},"step-1c-signal-summary-eval-signal",[2577],{"type":51,"value":2578},"Step 1c: Signal Summary (Eval Signal)",{"type":45,"tag":592,"props":2580,"children":2583},{"className":2581,"code":2582,"language":51},[595],"## Signal Summary: `{ml_app}` · Eval Signal\n(Inferred from {N} configured eval(s). Say `switch to error mode` or `switch to generic mode` to change.)\n\n**Timeframe**: {from} → {to}\n\n| Eval | Type | Total | Pass Rate | Status |\n|------|------|------:|:---------:|--------|\n| eval_1 | boolean | 4,891 | 37.3% | ⚠ Investigating |\n| eval_2 | score | 1,200 | — (inferred threshold) | ⚠ Investigating |\n| eval_3 | boolean | 500 | 99.2% | ✓ Healthy |\n\n**Collected**: {pass_count} passing, {fail_count} failing.\n",[2584],{"type":45,"tag":76,"props":2585,"children":2586},{"__ignoreMap":600},[2587],{"type":51,"value":2582},{"type":45,"tag":54,"props":2589,"children":2590},{},[2591],{"type":51,"value":2592},"For a single eval, collapse to a single-line header instead of a table.",{"type":45,"tag":1589,"props":2594,"children":2595},{},[],{"type":45,"tag":1643,"props":2597,"children":2599},{"id":2598},"error-signal-path",[2600],{"type":51,"value":2601},"Error Signal path",{"type":45,"tag":2217,"props":2603,"children":2605},{"id":2604},"step-1a-sample-error-spans",[2606],{"type":51,"value":2607},"Step 1a: Sample error spans",{"type":45,"tag":54,"props":2609,"children":2610},{},[2611,2617],{"type":45,"tag":76,"props":2612,"children":2614},{"className":2613},[],[2615],{"type":51,"value":2616},"search_llmobs_spans(query=\"@ml_app:\\\"\u003Cml_app>\\\" @status:error\", from=timeframe, limit=50)",{"type":51,"value":2618},". Paginate until ≥30 error spans or no more pages.",{"type":45,"tag":2217,"props":2620,"children":2622},{"id":2621},"step-1a5-soft-error-scan",[2623],{"type":51,"value":2624},"Step 1a.5: Soft error scan",{"type":45,"tag":54,"props":2626,"children":2627},{},[2628,2630,2636,2638,2644,2646,2651],{"type":51,"value":2629},"MCP tool spans sometimes report ",{"type":45,"tag":76,"props":2631,"children":2633},{"className":2632},[],[2634],{"type":51,"value":2635},"@status:ok",{"type":51,"value":2637}," but carry ",{"type":45,"tag":76,"props":2639,"children":2641},{"className":2640},[],[2642],{"type":51,"value":2643},"\"isError\": true",{"type":51,"value":2645}," in their output payload — these are invisible to ",{"type":45,"tag":76,"props":2647,"children":2649},{"className":2648},[],[2650],{"type":51,"value":522},{"type":51,"value":2652}," queries and can outnumber hard errors.",{"type":45,"tag":54,"props":2654,"children":2655},{},[2656,2658,2664,2666,2671,2673,2678,2680,2685,2687,2692],{"type":51,"value":2657},"Call ",{"type":45,"tag":76,"props":2659,"children":2661},{"className":2660},[],[2662],{"type":51,"value":2663},"search_llmobs_spans(query=\"@ml_app:\\\"\u003Cml_app>\\\" @status:ok\", span_kind=\"tool\", from=timeframe, limit=20)",{"type":51,"value":2665},". For a sample of 5–10 results, call ",{"type":45,"tag":76,"props":2667,"children":2669},{"className":2668},[],[2670],{"type":51,"value":2571},{"type":51,"value":2672}," in parallel. If any payloads contain ",{"type":45,"tag":76,"props":2674,"children":2676},{"className":2675},[],[2677],{"type":51,"value":2643},{"type":51,"value":2679},", add ",{"type":45,"tag":58,"props":2681,"children":2682},{},[2683],{"type":51,"value":2684},"MCP soft errors",{"type":51,"value":2686}," as a separate row in the error frequency table with the note: ",{"type":45,"tag":1247,"props":2688,"children":2689},{},[2690],{"type":51,"value":2691},"(status:ok but isError:true in payload — not queryable via @status:error)",{"type":51,"value":335},{"type":45,"tag":2217,"props":2694,"children":2696},{"id":2695},"step-1b-group-by-error-type",[2697],{"type":51,"value":2698},"Step 1b: Group by error type",{"type":45,"tag":54,"props":2700,"children":2701},{},[2702,2704,2710,2712,2717,2719,2725,2727,2732],{"type":51,"value":2703},"Group spans by ",{"type":45,"tag":76,"props":2705,"children":2707},{"className":2706},[],[2708],{"type":51,"value":2709},"error_type",{"type":51,"value":2711}," tag → frequency table. If ",{"type":45,"tag":76,"props":2713,"children":2715},{"className":2714},[],[2716],{"type":51,"value":2709},{"type":51,"value":2718}," tag is absent on some spans, supplement with the ",{"type":45,"tag":76,"props":2720,"children":2722},{"className":2721},[],[2723],{"type":51,"value":2724},"error.type",{"type":51,"value":2726}," field from ",{"type":45,"tag":76,"props":2728,"children":2730},{"className":2729},[],[2731],{"type":51,"value":1015},{"type":51,"value":2733}," (fetched in Step 1d).",{"type":45,"tag":2217,"props":2735,"children":2737},{"id":2736},"step-1c-fetch-stack-traces-parallel",[2738],{"type":51,"value":2739},"Step 1c: Fetch stack traces (parallel)",{"type":45,"tag":54,"props":2741,"children":2742},{},[2743,2745,2751],{"type":51,"value":2744},"For the top 3–4 error types by count, pick 2–3 representative trace IDs each. Call ",{"type":45,"tag":76,"props":2746,"children":2748},{"className":2747},[],[2749],{"type":51,"value":2750},"find_llmobs_error_spans(trace_id)",{"type":51,"value":2752}," in parallel across all selected traces. Extract:",{"type":45,"tag":197,"props":2754,"children":2755},{},[2756,2761,2766],{"type":45,"tag":70,"props":2757,"children":2758},{},[2759],{"type":51,"value":2760},"Error message and stack trace",{"type":45,"tag":70,"props":2762,"children":2763},{},[2764],{"type":51,"value":2765},"Origin span kind and name",{"type":45,"tag":70,"props":2767,"children":2768},{},[2769],{"type":51,"value":2770},"Whether errors propagate from children to parents (cascade) or are isolated",{"type":45,"tag":2217,"props":2772,"children":2774},{"id":2773},"step-1d-fetch-span-details",[2775],{"type":51,"value":2776},"Step 1d: Fetch span details",{"type":45,"tag":54,"props":2778,"children":2779},{},[2780,2785,2787,2792,2793,2798],{"type":45,"tag":76,"props":2781,"children":2783},{"className":2782},[],[2784],{"type":51,"value":1015},{"type":51,"value":2786}," on representative spans for each error type (follow Parallelization Rules). Extract ",{"type":45,"tag":76,"props":2788,"children":2790},{"className":2789},[],[2791],{"type":51,"value":1037},{"type":51,"value":254},{"type":45,"tag":76,"props":2794,"children":2796},{"className":2795},[],[2797],{"type":51,"value":1468},{"type":51,"value":2799},", duration.",{"type":45,"tag":2217,"props":2801,"children":2803},{"id":2802},"step-1e-signal-summary-error-signal",[2804],{"type":51,"value":2805},"Step 1e: Signal Summary (Error Signal)",{"type":45,"tag":592,"props":2807,"children":2810},{"className":2808,"code":2809,"language":51},[595],"## Signal Summary: `{ml_app}` · Error Signal\n(No evals configured — analyzing runtime errors. Say `switch to eval mode` or `switch to generic mode` to change.)\n\n**Timeframe**: {from} → {to}  |  **Total error spans sampled**: {N}\n\n| Error Type | Spans | Cascade? | Origin Span Kind |\n|------------|------:|:--------:|-----------------|\n| TimeoutError | 42 | Yes | tool |\n| APIError 429 | 18 | No | tool |\n| ValueError | 7 | No | llm |\n| MCP soft errors (isError:true) | 23 | No | tool |\n",[2811],{"type":45,"tag":76,"props":2812,"children":2813},{"__ignoreMap":600},[2814],{"type":51,"value":2809},{"type":45,"tag":1589,"props":2816,"children":2817},{},[],{"type":45,"tag":1643,"props":2819,"children":2821},{"id":2820},"generic-path",[2822],{"type":51,"value":2823},"Generic path",{"type":45,"tag":2217,"props":2825,"children":2827},{"id":2826},"step-1a-eval-health-check-when-evals-are-configured",[2828],{"type":51,"value":2829},"Step 1a: Eval health check (when evals are configured)",{"type":45,"tag":54,"props":2831,"children":2832},{},[2833,2834,2839,2841,2846,2848,2854,2856,2861],{"type":51,"value":1972},{"type":45,"tag":76,"props":2835,"children":2837},{"className":2836},[],[2838],{"type":51,"value":907},{"type":51,"value":2840}," returned evals in Phase 0, call ",{"type":45,"tag":76,"props":2842,"children":2844},{"className":2843},[],[2845],{"type":51,"value":931},{"type":51,"value":2847}," for each enabled eval in parallel. Flag any enabled eval with ",{"type":45,"tag":76,"props":2849,"children":2851},{"className":2850},[],[2852],{"type":51,"value":2853},"total_count: 0",{"type":51,"value":2855}," as ",{"type":45,"tag":58,"props":2857,"children":2858},{},[2859],{"type":51,"value":2860},"Broken Eval Configuration",{"type":51,"value":2862}," — include in the Signal Summary anomaly table as a High severity row.",{"type":45,"tag":2217,"props":2864,"children":2866},{"id":2865},"step-1b-broad-span-search",[2867],{"type":51,"value":2868},"Step 1b: Broad span search",{"type":45,"tag":54,"props":2870,"children":2871},{},[2872,2878,2880,2885,2887,2892,2894,2899,2901,2907],{"type":45,"tag":76,"props":2873,"children":2875},{"className":2874},[],[2876],{"type":51,"value":2877},"search_llmobs_spans(query=\"@ml_app:\\\"\u003Cml_app>\\\"\", root_spans_only=true, from=timeframe, limit=50)",{"type":51,"value":2879},". Apply ",{"type":45,"tag":76,"props":2881,"children":2883},{"className":2882},[],[2884],{"type":51,"value":789},{"type":51,"value":2886}," narrowing if present (tool\u002Fspan name → ",{"type":45,"tag":76,"props":2888,"children":2890},{"className":2889},[],[2891],{"type":51,"value":839},{"type":51,"value":2893}," query; ",{"type":45,"tag":76,"props":2895,"children":2897},{"className":2896},[],[2898],{"type":51,"value":816},{"type":51,"value":2900}," → sort result set by ",{"type":45,"tag":76,"props":2902,"children":2904},{"className":2903},[],[2905],{"type":51,"value":2906},"duration",{"type":51,"value":2908}," after Step 1c). Paginate until ≥30 spans.",{"type":45,"tag":2217,"props":2910,"children":2912},{"id":2911},"step-1c-fetch-span-details",[2913],{"type":51,"value":2914},"Step 1c: Fetch span details",{"type":45,"tag":54,"props":2916,"children":2917},{},[2918,2923],{"type":45,"tag":76,"props":2919,"children":2921},{"className":2920},[],[2922],{"type":51,"value":1015},{"type":51,"value":2924}," per trace_id batch.",{"type":45,"tag":2217,"props":2926,"children":2928},{"id":2927},"step-1d-rank-by-structural-anomalies",[2929],{"type":51,"value":2930},"Step 1d: Rank by structural anomalies",{"type":45,"tag":54,"props":2932,"children":2933},{},[2934],{"type":51,"value":2935},"Partition spans using heuristics:",{"type":45,"tag":197,"props":2937,"children":2938},{},[2939,2951,2956,2961,2966],{"type":45,"tag":70,"props":2940,"children":2941},{},[2942,2944,2949],{"type":51,"value":2943},"Top decile by ",{"type":45,"tag":76,"props":2945,"children":2947},{"className":2946},[],[2948],{"type":51,"value":2906},{"type":51,"value":2950}," (latency outliers)",{"type":45,"tag":70,"props":2952,"children":2953},{},[2954],{"type":51,"value":2955},"Agent spans with >N tool\u002FLLM iterations (long-running loops)",{"type":45,"tag":70,"props":2957,"children":2958},{},[2959],{"type":51,"value":2960},"Retrieval spans returning 0 documents (RAG miss)",{"type":45,"tag":70,"props":2962,"children":2963},{},[2964],{"type":51,"value":2965},"Workflow spans whose child set is missing an expected step (compare against median child layout)",{"type":45,"tag":70,"props":2967,"children":2968},{},[2969,2974,2976,2982,2984,2990],{"type":45,"tag":58,"props":2970,"children":2971},{},[2972],{"type":51,"value":2973},"Token efficiency",{"type":51,"value":2975},": Check if ",{"type":45,"tag":76,"props":2977,"children":2979},{"className":2978},[],[2980],{"type":51,"value":2981},"non_cached_input_tokens ≈ input_tokens",{"type":51,"value":2983}," across LLM spans. If the app has stable system prompts (>1k tokens) and cache hit rate is 0%, flag as High severity — enabling ",{"type":45,"tag":76,"props":2985,"children":2987},{"className":2986},[],[2988],{"type":51,"value":2989},"cache_control: ephemeral",{"type":51,"value":2991}," on the system prompt would cut input token costs by 60–90%",{"type":45,"tag":2217,"props":2993,"children":2995},{"id":2994},"step-1e-signal-summary-generic",[2996],{"type":51,"value":2997},"Step 1e: Signal Summary (Generic)",{"type":45,"tag":592,"props":2999,"children":3002},{"className":3000,"code":3001,"language":51},[595],"## Signal Summary: `{ml_app}` · Generic\n(Analyzing structural anomalies. Say `switch to eval mode` or `switch to error mode` to change.)\n\n**Timeframe**: {from} → {to}  |  **Sampled**: {N} root spans\n\n| Anomaly Type | Count |\n|---|:---:|\n| Latency outliers (>p90) | 12 |\n| Long agent loops (>8 iterations) | 5 |\n| RAG retrieval misses | 3 |\n| Zero prompt cache utilization | All LLM spans |\n| Broken eval configurations | 2 |\n",[3003],{"type":45,"tag":76,"props":3004,"children":3005},{"__ignoreMap":600},[3006],{"type":51,"value":3001},{"type":45,"tag":1589,"props":3008,"children":3009},{},[],{"type":45,"tag":868,"props":3011,"children":3013},{"id":3012},"phase-15-determine-app-profile-where-the-root-cause-lives",[3014],{"type":51,"value":3015},"Phase 1.5: Determine App Profile & Where the Root Cause Lives",{"type":45,"tag":54,"props":3017,"children":3018},{},[3019,3021,3026,3027,3032],{"type":51,"value":3020},"Inspect ",{"type":45,"tag":76,"props":3022,"children":3024},{"className":3023},[],[3025],{"type":51,"value":1037},{"type":51,"value":2416},{"type":45,"tag":76,"props":3028,"children":3030},{"className":3029},[],[3031],{"type":51,"value":1468},{"type":51,"value":3033}," across collected spans. Drives Phase 4 strategy.",{"type":45,"tag":54,"props":3035,"children":3036},{},[3037,3042],{"type":45,"tag":58,"props":3038,"children":3039},{},[3040],{"type":51,"value":3041},"App profile",{"type":51,"value":3043}," (from content_info):",{"type":45,"tag":448,"props":3045,"children":3046},{},[3047,3066],{"type":45,"tag":452,"props":3048,"children":3049},{},[3050],{"type":45,"tag":456,"props":3051,"children":3052},{},[3053,3057,3061],{"type":45,"tag":460,"props":3054,"children":3055},{},[3056],{"type":51,"value":469},{"type":45,"tag":460,"props":3058,"children":3059},{},[3060],{"type":51,"value":3041},{"type":45,"tag":460,"props":3062,"children":3063},{},[3064],{"type":51,"value":3065},"Phase 4 strategy",{"type":45,"tag":476,"props":3067,"children":3068},{},[3069,3105,3132,3165,3187],{"type":45,"tag":456,"props":3070,"children":3071},{},[3072,3087,3092],{"type":45,"tag":483,"props":3073,"children":3074},{},[3075,3080,3082],{"type":45,"tag":76,"props":3076,"children":3078},{"className":3077},[],[3079],{"type":51,"value":1037},{"type":51,"value":3081}," has ",{"type":45,"tag":76,"props":3083,"children":3085},{"className":3084},[],[3086],{"type":51,"value":1184},{"type":45,"tag":483,"props":3088,"children":3089},{},[3090],{"type":51,"value":3091},"LLM\u002Fchat app",{"type":45,"tag":483,"props":3093,"children":3094},{},[3095,3097,3103],{"type":51,"value":3096},"Extract system prompt via ",{"type":45,"tag":76,"props":3098,"children":3100},{"className":3099},[],[3101],{"type":51,"value":3102},"messages[0]",{"type":51,"value":3104},", check conversation flow",{"type":45,"tag":456,"props":3106,"children":3107},{},[3108,3122,3127],{"type":45,"tag":483,"props":3109,"children":3110},{},[3111,3116,3117],{"type":45,"tag":76,"props":3112,"children":3114},{"className":3113},[],[3115],{"type":51,"value":1037},{"type":51,"value":3081},{"type":45,"tag":76,"props":3118,"children":3120},{"className":3119},[],[3121],{"type":51,"value":1297},{"type":45,"tag":483,"props":3123,"children":3124},{},[3125],{"type":51,"value":3126},"RAG app",{"type":45,"tag":483,"props":3128,"children":3129},{},[3130],{"type":51,"value":3131},"Check retrieval quality alongside LLM output",{"type":45,"tag":456,"props":3133,"children":3134},{},[3135,3148,3153],{"type":45,"tag":483,"props":3136,"children":3137},{},[3138,3140,3146],{"type":51,"value":3139},"Trace contains ",{"type":45,"tag":76,"props":3141,"children":3143},{"className":3142},[],[3144],{"type":51,"value":3145},"agent",{"type":51,"value":3147}," span kind",{"type":45,"tag":483,"props":3149,"children":3150},{},[3151],{"type":51,"value":3152},"Agent app",{"type":45,"tag":483,"props":3154,"children":3155},{},[3156,3158,3163],{"type":51,"value":3157},"Try ",{"type":45,"tag":76,"props":3159,"children":3161},{"className":3160},[],[3162],{"type":51,"value":1127},{"type":51,"value":3164}," first; if it returns empty use child-span reconstruction (see Phase 4b)",{"type":45,"tag":456,"props":3166,"children":3167},{},[3168,3177,3182],{"type":45,"tag":483,"props":3169,"children":3170},{},[3171],{"type":45,"tag":76,"props":3172,"children":3174},{"className":3173},[],[3175],{"type":51,"value":3176},"messages.count > 10",{"type":45,"tag":483,"props":3178,"children":3179},{},[3180],{"type":51,"value":3181},"Long conversation",{"type":45,"tag":483,"props":3183,"children":3184},{},[3185],{"type":51,"value":3186},"Check for context overflow",{"type":45,"tag":456,"props":3188,"children":3189},{},[3190,3204,3209],{"type":45,"tag":483,"props":3191,"children":3192},{},[3193,3198,3199],{"type":45,"tag":76,"props":3194,"children":3196},{"className":3195},[],[3197],{"type":51,"value":1037},{"type":51,"value":3081},{"type":45,"tag":76,"props":3200,"children":3202},{"className":3201},[],[3203],{"type":51,"value":1318},{"type":45,"tag":483,"props":3205,"children":3206},{},[3207],{"type":51,"value":3208},"Has custom metadata",{"type":45,"tag":483,"props":3210,"children":3211},{},[3212],{"type":51,"value":3213},"Check for clustering by metadata values (prompt version, feature flags)",{"type":45,"tag":162,"props":3215,"children":3216},{},[3217],{"type":45,"tag":54,"props":3218,"children":3219},{},[3220,3225,3227,3233,3235,3240,3241,3246,3248,3254,3256,3265,3267,3272,3274,3279,3280,3285,3287,3292,3294,3299],{"type":45,"tag":58,"props":3221,"children":3222},{},[3223],{"type":51,"value":3224},"LLM Experiments traces",{"type":51,"value":3226},": If root spans have ",{"type":45,"tag":76,"props":3228,"children":3230},{"className":3229},[],[3231],{"type":51,"value":3232},"span_kind: experiment",{"type":51,"value":3234}," and carry ",{"type":45,"tag":76,"props":3236,"children":3238},{"className":3237},[],[3239],{"type":51,"value":1268},{"type":51,"value":254},{"type":45,"tag":76,"props":3242,"children":3244},{"className":3243},[],[3245],{"type":51,"value":1276},{"type":51,"value":3247},", and ",{"type":45,"tag":76,"props":3249,"children":3251},{"className":3250},[],[3252],{"type":51,"value":3253},"expected_output",{"type":51,"value":3255}," structured fields, you are looking at a ",{"type":45,"tag":3257,"props":3258,"children":3262},"a",{"href":3259,"rel":3260},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fexperiments\u002F",[3261],"nofollow",[3263],{"type":51,"value":3264},"Datadog LLM Experiments",{"type":51,"value":3266}," trace. Each span represents one dataset record run. Read quality signal from the root span's ",{"type":45,"tag":76,"props":3268,"children":3270},{"className":3269},[],[3271],{"type":51,"value":1268},{"type":51,"value":3273},"\u002F",{"type":45,"tag":76,"props":3275,"children":3277},{"className":3276},[],[3278],{"type":51,"value":1276},{"type":51,"value":3273},{"type":45,"tag":76,"props":3281,"children":3283},{"className":3282},[],[3284],{"type":51,"value":3253},{"type":51,"value":3286}," fields via ",{"type":45,"tag":76,"props":3288,"children":3290},{"className":3289},[],[3291],{"type":51,"value":1051},{"type":51,"value":3293}," — not from LLM sub-span messages, which may contain stub or placeholder content. Evaluations attached to experiment spans are computed by the Experiments framework at run time and may not be registered as online Datadog evaluators (",{"type":45,"tag":76,"props":3295,"children":3297},{"className":3296},[],[3298],{"type":51,"value":948},{"type":51,"value":3300}," will return 404 for them).",{"type":45,"tag":54,"props":3302,"children":3303},{},[3304,3309],{"type":45,"tag":58,"props":3305,"children":3306},{},[3307],{"type":51,"value":3308},"Where the root cause likely lives",{"type":51,"value":3310}," — by symptom span kind:",{"type":45,"tag":448,"props":3312,"children":3313},{},[3314,3335],{"type":45,"tag":452,"props":3315,"children":3316},{},[3317],{"type":45,"tag":456,"props":3318,"children":3319},{},[3320,3325,3330],{"type":45,"tag":460,"props":3321,"children":3322},{},[3323],{"type":51,"value":3324},"Symptom span kind",{"type":45,"tag":460,"props":3326,"children":3327},{},[3328],{"type":51,"value":3329},"Symptom looks like",{"type":45,"tag":460,"props":3331,"children":3332},{},[3333],{"type":51,"value":3334},"But root cause is often in...",{"type":45,"tag":476,"props":3336,"children":3337},{},[3338,3377,3403,3429,3455],{"type":45,"tag":456,"props":3339,"children":3340},{},[3341,3349,3354],{"type":45,"tag":483,"props":3342,"children":3343},{},[3344],{"type":45,"tag":76,"props":3345,"children":3347},{"className":3346},[],[3348],{"type":51,"value":18},{"type":45,"tag":483,"props":3350,"children":3351},{},[3352],{"type":51,"value":3353},"Bad LLM response (eval flagged, wrong output)",{"type":45,"tag":483,"props":3355,"children":3356},{},[3357,3362,3364,3369,3371,3375],{"type":45,"tag":58,"props":3358,"children":3359},{},[3360],{"type":51,"value":3361},"Parent",{"type":51,"value":3363}," agent (bad instructions), ",{"type":45,"tag":58,"props":3365,"children":3366},{},[3367],{"type":51,"value":3368},"sibling",{"type":51,"value":3370}," retrieval (bad context), ",{"type":45,"tag":58,"props":3372,"children":3373},{},[3374],{"type":51,"value":3368},{"type":51,"value":3376}," tool (bad data)",{"type":45,"tag":456,"props":3378,"children":3379},{},[3380,3388,3393],{"type":45,"tag":483,"props":3381,"children":3382},{},[3383],{"type":45,"tag":76,"props":3384,"children":3386},{"className":3385},[],[3387],{"type":51,"value":3145},{"type":45,"tag":483,"props":3389,"children":3390},{},[3391],{"type":51,"value":3392},"Bad orchestration",{"type":45,"tag":483,"props":3394,"children":3395},{},[3396,3401],{"type":45,"tag":58,"props":3397,"children":3398},{},[3399],{"type":51,"value":3400},"Child",{"type":51,"value":3402}," spans (wrong tool calls, bad routing), full agent loop",{"type":45,"tag":456,"props":3404,"children":3405},{},[3406,3415,3420],{"type":45,"tag":483,"props":3407,"children":3408},{},[3409],{"type":45,"tag":76,"props":3410,"children":3412},{"className":3411},[],[3413],{"type":51,"value":3414},"tool",{"type":45,"tag":483,"props":3416,"children":3417},{},[3418],{"type":51,"value":3419},"Bad tool result",{"type":45,"tag":483,"props":3421,"children":3422},{},[3423,3427],{"type":45,"tag":58,"props":3424,"children":3425},{},[3426],{"type":51,"value":3361},{"type":51,"value":3428}," LLM (passed wrong parameters), tool implementation",{"type":45,"tag":456,"props":3430,"children":3431},{},[3432,3441,3446],{"type":45,"tag":483,"props":3433,"children":3434},{},[3435],{"type":45,"tag":76,"props":3436,"children":3438},{"className":3437},[],[3439],{"type":51,"value":3440},"workflow",{"type":45,"tag":483,"props":3442,"children":3443},{},[3444],{"type":51,"value":3445},"Bad overall output",{"type":45,"tag":483,"props":3447,"children":3448},{},[3449,3453],{"type":45,"tag":58,"props":3450,"children":3451},{},[3452],{"type":51,"value":3400},{"type":51,"value":3454}," sub-spans (which step first deviated?)",{"type":45,"tag":456,"props":3456,"children":3457},{},[3458,3467,3472],{"type":45,"tag":483,"props":3459,"children":3460},{},[3461],{"type":45,"tag":76,"props":3462,"children":3464},{"className":3463},[],[3465],{"type":51,"value":3466},"retrieval",{"type":45,"tag":483,"props":3468,"children":3469},{},[3470],{"type":51,"value":3471},"Bad retrieval",{"type":45,"tag":483,"props":3473,"children":3474},{},[3475],{"type":51,"value":3476},"Query construction (parent), index\u002Fembedding config (outside trace)",{"type":45,"tag":54,"props":3478,"children":3479},{},[3480,3485,3487,3492],{"type":45,"tag":58,"props":3481,"children":3482},{},[3483],{"type":51,"value":3484},"Key insight",{"type":51,"value":3486},": The signal — eval verdict, error message, latency outlier — flags one span in isolation. It's a ",{"type":45,"tag":1247,"props":3488,"children":3489},{},[3490],{"type":51,"value":3491},"symptom report",{"type":51,"value":3493},", not a diagnosis. The root cause often lives in a different span: a parent that gave bad instructions, a sibling that provided bad context, or a child that made a wrong decision. Phase 4 navigates the tree to find it.",{"type":45,"tag":1589,"props":3495,"children":3496},{},[],{"type":45,"tag":868,"props":3498,"children":3500},{"id":3499},"phase-2-open-coding-initial-failure-categorization",[3501],{"type":51,"value":3502},"Phase 2: Open Coding — Initial Failure Categorization",{"type":45,"tag":54,"props":3504,"children":3505},{},[3506,3510],{"type":45,"tag":58,"props":3507,"children":3508},{},[3509],{"type":51,"value":1390},{"type":51,"value":3511},": Read per-row evidence and propose initial, concrete failure categories. Pool all problematic rows together — categories should describe app behaviors, not which signal flagged them.",{"type":45,"tag":54,"props":3513,"children":3514},{},[3515,3520],{"type":45,"tag":58,"props":3516,"children":3517},{},[3518],{"type":51,"value":3519},"Per-row \"reasoning input\"",{"type":51,"value":3521}," by mode:",{"type":45,"tag":197,"props":3523,"children":3524},{},[3525,3539,3553],{"type":45,"tag":70,"props":3526,"children":3527},{},[3528,3532,3534],{"type":45,"tag":58,"props":3529,"children":3530},{},[3531],{"type":51,"value":490},{"type":51,"value":3533},": judge assessment + reasoning from ",{"type":45,"tag":76,"props":3535,"children":3537},{"className":3536},[],[3538],{"type":51,"value":1015},{"type":45,"tag":70,"props":3540,"children":3541},{},[3542,3546,3548],{"type":45,"tag":58,"props":3543,"children":3544},{},[3545],{"type":51,"value":511},{"type":51,"value":3547},": error message + stack trace excerpt from ",{"type":45,"tag":76,"props":3549,"children":3551},{"className":3550},[],[3552],{"type":51,"value":1093},{"type":45,"tag":70,"props":3554,"children":3555},{},[3556,3560],{"type":45,"tag":58,"props":3557,"children":3558},{},[3559],{"type":51,"value":540},{"type":51,"value":3561},": one-line description of the structural anomaly that flagged the row",{"type":45,"tag":54,"props":3563,"children":3564},{},[3565,3570],{"type":45,"tag":58,"props":3566,"children":3567},{},[3568],{"type":51,"value":3569},"Shortcuts",{"type":51,"value":3571},":",{"type":45,"tag":197,"props":3573,"children":3574},{},[3575,3585,3595],{"type":45,"tag":70,"props":3576,"children":3577},{},[3578,3583],{"type":45,"tag":58,"props":3579,"children":3580},{},[3581],{"type":51,"value":3582},"\u003C 15 problematic rows",{"type":51,"value":3584},": Combine Phases 2 and 3 into one pass. Still produce the checkpoint.",{"type":45,"tag":70,"props":3586,"children":3587},{},[3588,3593],{"type":45,"tag":58,"props":3589,"children":3590},{},[3591],{"type":51,"value":3592},"> 80% share the same reasoning\u002Ferror\u002Fsymptom",{"type":51,"value":3594},": Skip to Phase 4 with the dominant pattern. Still output checkpoint.",{"type":45,"tag":70,"props":3596,"children":3597},{},[3598,3603],{"type":45,"tag":58,"props":3599,"children":3600},{},[3601],{"type":51,"value":3602},"> 50 problematic rows",{"type":51,"value":3604},": Sample ~50, build taxonomy, then spot-check 10–15 more.",{"type":45,"tag":66,"props":3606,"children":3607},{},[3608,3626,3659],{"type":45,"tag":70,"props":3609,"children":3610},{},[3611,3616,3618,3624],{"type":45,"tag":58,"props":3612,"children":3613},{},[3614],{"type":51,"value":3615},"Use per-row signal from Phase 1",{"type":51,"value":3617}," — do NOT re-fetch. Only call ",{"type":45,"tag":76,"props":3619,"children":3621},{"className":3620},[],[3622],{"type":51,"value":3623},"get_llmobs_span_content(field=\"input\"\u002F\"output\")",{"type":51,"value":3625}," for spans where the reasoning is insufficient (generic, empty, or just a stack trace with no app context).",{"type":45,"tag":70,"props":3627,"children":3628},{},[3629,3634,3636],{"type":45,"tag":58,"props":3630,"children":3631},{},[3632],{"type":51,"value":3633},"If eval config is loaded",{"type":51,"value":3635}," (Eval Signal), distinguish early:",{"type":45,"tag":197,"props":3637,"children":3638},{},[3639,3649],{"type":45,"tag":70,"props":3640,"children":3641},{},[3642,3647],{"type":45,"tag":58,"props":3643,"children":3644},{},[3645],{"type":51,"value":3646},"App failures",{"type":51,"value":3648},": Output genuinely violates the eval's criteria",{"type":45,"tag":70,"props":3650,"children":3651},{},[3652,3657],{"type":45,"tag":58,"props":3653,"children":3654},{},[3655],{"type":51,"value":3656},"Eval failures",{"type":51,"value":3658},": Output seems reasonable but eval criteria are too strict\u002Fambiguous",{"type":45,"tag":70,"props":3660,"children":3661},{},[3662,3667],{"type":45,"tag":58,"props":3663,"children":3664},{},[3665],{"type":51,"value":3666},"Each pattern must be specific",{"type":51,"value":3668},": \"Agent called search instead of calculator for price computation\" — NOT \"tool issue.\"",{"type":45,"tag":1643,"props":3670,"children":3672},{"id":3671},"mandatory-checkpoint",[3673],{"type":51,"value":3674},"MANDATORY CHECKPOINT",{"type":45,"tag":592,"props":3676,"children":3679},{"className":3677,"code":3678,"language":51},[595],"**Open coding**: {N} problematic rows → {K} initial categories: {Category1} ({count}), {Category2} ({count}), ...\n",[3680],{"type":45,"tag":76,"props":3681,"children":3682},{"__ignoreMap":600},[3683],{"type":51,"value":3678},{"type":45,"tag":1589,"props":3685,"children":3686},{},[],{"type":45,"tag":868,"props":3688,"children":3690},{"id":3689},"phase-3-axial-coding-refine-failure-taxonomy",[3691],{"type":51,"value":3692},"Phase 3: Axial Coding — Refine Failure Taxonomy",{"type":45,"tag":54,"props":3694,"children":3695},{},[3696,3700],{"type":45,"tag":58,"props":3697,"children":3698},{},[3699],{"type":51,"value":1390},{"type":51,"value":3701},": 3–8 final categories, ranked by impact.",{"type":45,"tag":66,"props":3703,"children":3704},{},[3705,3715,3725,3735],{"type":45,"tag":70,"props":3706,"children":3707},{},[3708,3713],{"type":45,"tag":58,"props":3709,"children":3710},{},[3711],{"type":51,"value":3712},"Merge",{"type":51,"value":3714},": Categories with \u003C 3 occurrences → parent category or drop as noise.",{"type":45,"tag":70,"props":3716,"children":3717},{},[3718,3723],{"type":45,"tag":58,"props":3719,"children":3720},{},[3721],{"type":51,"value":3722},"Split",{"type":51,"value":3724},": Categories with > 30% of failures → more specific sub-categories. Pull additional span content if needed.",{"type":45,"tag":70,"props":3726,"children":3727},{},[3728,3733],{"type":45,"tag":58,"props":3729,"children":3730},{},[3731],{"type":51,"value":3732},"Validate",{"type":51,"value":3734},": 2–3 representative examples per category confirm the label fits.",{"type":45,"tag":70,"props":3736,"children":3737},{},[3738,3743,3745,3751],{"type":45,"tag":58,"props":3739,"children":3740},{},[3741],{"type":51,"value":3742},"Rank",{"type":51,"value":3744},": ",{"type":45,"tag":76,"props":3746,"children":3748},{"className":3747},[],[3749],{"type":51,"value":3750},"priority = count × severity",{"type":51,"value":3752}," (severity: high \u002F medium \u002F low).",{"type":45,"tag":1643,"props":3754,"children":3756},{"id":3755},"mandatory-checkpoint-1",[3757],{"type":51,"value":3674},{"type":45,"tag":592,"props":3759,"children":3762},{"className":3760,"code":3761,"language":51},[595],"**Axial coding**: {merges\u002Fsplits\u002Fdrops}. Final categories:\n1. {Category} ({count}, {pct}%) — {severity}\n2. ...\n",[3763],{"type":45,"tag":76,"props":3764,"children":3765},{"__ignoreMap":600},[3766],{"type":51,"value":3761},{"type":45,"tag":1589,"props":3768,"children":3769},{},[],{"type":45,"tag":868,"props":3771,"children":3773},{"id":3772},"phase-4-root-cause-analysis-navigate-from-symptom-to-root-cause",[3774],{"type":51,"value":3775},"Phase 4: Root Cause Analysis — Navigate from Symptom to Root Cause",{"type":45,"tag":54,"props":3777,"children":3778},{},[3779,3783],{"type":45,"tag":58,"props":3780,"children":3781},{},[3782],{"type":51,"value":1390},{"type":51,"value":3784},": The signal flagged a span. That's the symptom. Navigate the trace tree to find the actual root cause — it's often in a different span.",{"type":45,"tag":54,"props":3786,"children":3787},{},[3788],{"type":51,"value":3789},"For each of the top 3 categories, pick 2–3 representative traces:",{"type":45,"tag":1643,"props":3791,"children":3793},{"id":3792},"step-4a-trace-structure-errors-parallel",[3794],{"type":51,"value":3795},"Step 4a: Trace structure + errors (parallel)",{"type":45,"tag":54,"props":3797,"children":3798},{},[3799],{"type":51,"value":3800},"For each representative trace, call in a single message:",{"type":45,"tag":197,"props":3802,"children":3803},{},[3804,3815],{"type":45,"tag":70,"props":3805,"children":3806},{},[3807,3813],{"type":45,"tag":76,"props":3808,"children":3810},{"className":3809},[],[3811],{"type":51,"value":3812},"get_llmobs_trace(trace_id)",{"type":51,"value":3814}," — span hierarchy; locate the symptom span and its parent\u002Fsiblings\u002Fchildren",{"type":45,"tag":70,"props":3816,"children":3817},{},[3818,3823],{"type":45,"tag":76,"props":3819,"children":3821},{"className":3820},[],[3822],{"type":51,"value":2750},{"type":51,"value":3824}," — check for runtime errors anywhere in the trace",{"type":45,"tag":54,"props":3826,"children":3827},{},[3828,3833],{"type":45,"tag":58,"props":3829,"children":3830},{},[3831],{"type":51,"value":3832},"Runtime vs behavioral",{"type":51,"value":3834},": If errors exist on or near the symptom span, the root cause may be a runtime failure rather than a behavioral one. Check this first.",{"type":45,"tag":54,"props":3836,"children":3837},{},[3838,3843,3844,3849,3851,3856,3858,3864],{"type":45,"tag":58,"props":3839,"children":3840},{},[3841],{"type":51,"value":3842},"Distributed trace fallback",{"type":51,"value":2536},{"type":45,"tag":76,"props":3845,"children":3847},{"className":3846},[],[3848],{"type":51,"value":1076},{"type":51,"value":3850}," returns \"cannot find parent\" or an empty span list (common in Ray-based or multi-process execution), reconstruct the trace manually using ",{"type":45,"tag":76,"props":3852,"children":3854},{"className":3853},[],[3855],{"type":51,"value":1015},{"type":51,"value":3857}," on the span_ids collected in Phase 1, sorted by ",{"type":45,"tag":76,"props":3859,"children":3861},{"className":3860},[],[3862],{"type":51,"value":3863},"start_ms",{"type":51,"value":335},{"type":45,"tag":1643,"props":3866,"children":3868},{"id":3867},"step-4b-navigate-to-the-root-cause-parallel",[3869],{"type":51,"value":3870},"Step 4b: Navigate to the root cause (parallel)",{"type":45,"tag":54,"props":3872,"children":3873},{},[3874],{"type":51,"value":3875},"Use the symptom span kind (from Phase 1.5). Issue ALL calls in a single message.",{"type":45,"tag":54,"props":3877,"children":3878},{},[3879,3891],{"type":45,"tag":58,"props":3880,"children":3881},{},[3882,3884,3889],{"type":51,"value":3883},"If symptom is on an ",{"type":45,"tag":76,"props":3885,"children":3887},{"className":3886},[],[3888],{"type":51,"value":18},{"type":51,"value":3890}," span",{"type":51,"value":3892}," (most common):",{"type":45,"tag":197,"props":3894,"children":3895},{},[3896,3913,3929,3946,3962],{"type":45,"tag":70,"props":3897,"children":3898},{},[3899,3905,3906,3911],{"type":45,"tag":76,"props":3900,"children":3902},{"className":3901},[],[3903],{"type":51,"value":3904},"get_llmobs_span_content(field=\"messages\", path=\"$.messages[0]\")",{"type":51,"value":346},{"type":45,"tag":58,"props":3907,"children":3908},{},[3909],{"type":51,"value":3910},"symptom span",{"type":51,"value":3912}," — system prompt",{"type":45,"tag":70,"props":3914,"children":3915},{},[3916,3922,3923,3927],{"type":45,"tag":76,"props":3917,"children":3919},{"className":3918},[],[3920],{"type":51,"value":3921},"get_llmobs_span_content(field=\"messages\")",{"type":51,"value":346},{"type":45,"tag":58,"props":3924,"children":3925},{},[3926],{"type":51,"value":3910},{"type":51,"value":3928}," — full context received",{"type":45,"tag":70,"props":3930,"children":3931},{},[3932,3938,3939,3944],{"type":45,"tag":76,"props":3933,"children":3935},{"className":3934},[],[3936],{"type":51,"value":3937},"get_llmobs_span_content(field=\"documents\")",{"type":51,"value":346},{"type":45,"tag":58,"props":3940,"children":3941},{},[3942],{"type":51,"value":3943},"sibling retrieval spans",{"type":51,"value":3945}," (if any)",{"type":45,"tag":70,"props":3947,"children":3948},{},[3949,3955,3956,3961],{"type":45,"tag":76,"props":3950,"children":3952},{"className":3951},[],[3953],{"type":51,"value":3954},"get_llmobs_span_content(field=\"input\")",{"type":51,"value":346},{"type":45,"tag":58,"props":3957,"children":3958},{},[3959],{"type":51,"value":3960},"sibling tool spans",{"type":51,"value":3945},{"type":45,"tag":70,"props":3963,"children":3964},{},[3965,3970,3971],{"type":45,"tag":76,"props":3966,"children":3968},{"className":3967},[],[3969],{"type":51,"value":3904},{"type":51,"value":346},{"type":45,"tag":58,"props":3972,"children":3973},{},[3974],{"type":51,"value":3975},"parent agent\u002Fworkflow span",{"type":45,"tag":54,"props":3977,"children":3978},{},[3979,3989],{"type":45,"tag":58,"props":3980,"children":3981},{},[3982,3983,3988],{"type":51,"value":3883},{"type":45,"tag":76,"props":3984,"children":3986},{"className":3985},[],[3987],{"type":51,"value":3145},{"type":51,"value":3890},{"type":51,"value":3571},{"type":45,"tag":197,"props":3991,"children":3992},{},[3993,4009,4032],{"type":45,"tag":70,"props":3994,"children":3995},{},[3996,4002,4004],{"type":45,"tag":76,"props":3997,"children":3999},{"className":3998},[],[4000],{"type":51,"value":4001},"get_llmobs_agent_loop(trace_id, span_id)",{"type":51,"value":4003}," — full decision timeline ",{"type":45,"tag":1247,"props":4005,"children":4006},{},[4007],{"type":51,"value":4008},"(try first; if it returns 0 iterations, use the fallback below)",{"type":45,"tag":70,"props":4010,"children":4011},{},[4012,4017,4018,4023,4025,4030],{"type":45,"tag":76,"props":4013,"children":4015},{"className":4014},[],[4016],{"type":51,"value":1015},{"type":51,"value":346},{"type":45,"tag":58,"props":4019,"children":4020},{},[4021],{"type":51,"value":4022},"child spans",{"type":51,"value":4024}," — sort by ",{"type":45,"tag":76,"props":4026,"children":4028},{"className":4027},[],[4029],{"type":51,"value":3863},{"type":51,"value":4031}," to reconstruct the execution timeline",{"type":45,"tag":70,"props":4033,"children":4034},{},[4035,4040,4041],{"type":45,"tag":76,"props":4036,"children":4038},{"className":4037},[],[4039],{"type":51,"value":3623},{"type":51,"value":346},{"type":45,"tag":58,"props":4042,"children":4043},{},[4044],{"type":51,"value":4045},"child spans that look wrong",{"type":45,"tag":54,"props":4047,"children":4048},{},[4049,4054,4056,4061,4063,4068,4070,4075,4077,4082,4084,4089],{"type":45,"tag":58,"props":4050,"children":4051},{},[4052],{"type":51,"value":4053},"Agent loop fallback",{"type":51,"value":4055}," (when ",{"type":45,"tag":76,"props":4057,"children":4059},{"className":4058},[],[4060],{"type":51,"value":1127},{"type":51,"value":4062}," returns 0 iterations): Reconstruct the timeline from ",{"type":45,"tag":76,"props":4064,"children":4066},{"className":4065},[],[4067],{"type":51,"value":1015},{"type":51,"value":4069}," results sorted by ",{"type":45,"tag":76,"props":4071,"children":4073},{"className":4072},[],[4074],{"type":51,"value":3863},{"type":51,"value":4076},". Group by ",{"type":45,"tag":76,"props":4078,"children":4080},{"className":4079},[],[4081],{"type":51,"value":1468},{"type":51,"value":4083}," to identify LLM → tool → LLM sequences. This fallback is frequently needed — ",{"type":45,"tag":76,"props":4085,"children":4087},{"className":4086},[],[4088],{"type":51,"value":1127},{"type":51,"value":4090}," returns empty for many apps.",{"type":45,"tag":54,"props":4092,"children":4093},{},[4094,4105],{"type":45,"tag":58,"props":4095,"children":4096},{},[4097,4099,4104],{"type":51,"value":4098},"If symptom is on a ",{"type":45,"tag":76,"props":4100,"children":4102},{"className":4101},[],[4103],{"type":51,"value":3414},{"type":51,"value":3890},{"type":51,"value":3571},{"type":45,"tag":197,"props":4107,"children":4108},{},[4109,4124],{"type":45,"tag":70,"props":4110,"children":4111},{},[4112,4117,4118,4122],{"type":45,"tag":76,"props":4113,"children":4115},{"className":4114},[],[4116],{"type":51,"value":3954},{"type":51,"value":346},{"type":45,"tag":58,"props":4119,"children":4120},{},[4121],{"type":51,"value":3910},{"type":51,"value":4123}," — what parameters was it called with?",{"type":45,"tag":70,"props":4125,"children":4126},{},[4127,4132,4133,4138],{"type":45,"tag":76,"props":4128,"children":4130},{"className":4129},[],[4131],{"type":51,"value":3921},{"type":51,"value":346},{"type":45,"tag":58,"props":4134,"children":4135},{},[4136],{"type":51,"value":4137},"parent LLM span",{"type":51,"value":4139}," — did the LLM construct the call correctly?",{"type":45,"tag":54,"props":4141,"children":4142},{},[4143,4153],{"type":45,"tag":58,"props":4144,"children":4145},{},[4146,4147,4152],{"type":51,"value":4098},{"type":45,"tag":76,"props":4148,"children":4150},{"className":4149},[],[4151],{"type":51,"value":3440},{"type":51,"value":3890},{"type":51,"value":3571},{"type":45,"tag":197,"props":4155,"children":4156},{},[4157,4173],{"type":45,"tag":70,"props":4158,"children":4159},{},[4160,4165,4166,4171],{"type":45,"tag":76,"props":4161,"children":4163},{"className":4162},[],[4164],{"type":51,"value":1015},{"type":51,"value":346},{"type":45,"tag":58,"props":4167,"children":4168},{},[4169],{"type":51,"value":4170},"all child spans",{"type":51,"value":4172}," — find which step first deviated",{"type":45,"tag":70,"props":4174,"children":4175},{},[4176,4181,4183],{"type":45,"tag":76,"props":4177,"children":4179},{"className":4178},[],[4180],{"type":51,"value":3623},{"type":51,"value":4182}," on the ",{"type":45,"tag":58,"props":4184,"children":4185},{},[4186],{"type":51,"value":4187},"deviating child",{"type":45,"tag":54,"props":4189,"children":4190},{},[4191,4196],{"type":45,"tag":58,"props":4192,"children":4193},{},[4194],{"type":51,"value":4195},"Always also fetch",{"type":51,"value":3571},{"type":45,"tag":197,"props":4198,"children":4199},{},[4200],{"type":45,"tag":70,"props":4201,"children":4202},{},[4203,4209],{"type":45,"tag":76,"props":4204,"children":4206},{"className":4205},[],[4207],{"type":51,"value":4208},"get_llmobs_span_content(field=\"metadata\")",{"type":51,"value":4210}," on the symptom span — clustering signals (prompt version, feature flags)",{"type":45,"tag":1643,"props":4212,"children":4214},{"id":4213},"step-4c-diagnose-from-symptom-to-root-cause",[4215],{"type":51,"value":4216},"Step 4c: Diagnose — from symptom to root cause",{"type":45,"tag":54,"props":4218,"children":4219},{},[4220],{"type":51,"value":4221},"For each category, trace the causal chain:",{"type":45,"tag":66,"props":4223,"children":4224},{},[4225,4235,4245],{"type":45,"tag":70,"props":4226,"children":4227},{},[4228,4233],{"type":45,"tag":58,"props":4229,"children":4230},{},[4231],{"type":51,"value":4232},"Symptom",{"type":51,"value":4234}," — what the signal flagged (eval reasoning, error message, anomaly note). The signal only saw one span in isolation — its reasoning may be shallow.",{"type":45,"tag":70,"props":4236,"children":4237},{},[4238,4243],{"type":45,"tag":58,"props":4239,"children":4240},{},[4241],{"type":51,"value":4242},"Trace context",{"type":51,"value":4244}," — what surrounding spans reveal (parent instructions, sibling data, child decisions).",{"type":45,"tag":70,"props":4246,"children":4247},{},[4248,4253],{"type":45,"tag":58,"props":4249,"children":4250},{},[4251],{"type":51,"value":4252},"Root cause",{"type":51,"value":4254}," — the specific span and decision point where the failure originated. Often NOT the symptom span itself.",{"type":45,"tag":54,"props":4256,"children":4257},{},[4258,4263],{"type":45,"tag":58,"props":4259,"children":4260},{},[4261],{"type":51,"value":4262},"For suspected eval issues",{"type":51,"value":4264}," (Eval Signal, if config loaded): Compare eval criteria against evidence. Is the prompt ambiguous? Criteria too strict?",{"type":45,"tag":54,"props":4266,"children":4267},{},[4268],{"type":45,"tag":58,"props":4269,"children":4270},{},[4271],{"type":51,"value":4272},"Root cause categories:",{"type":45,"tag":448,"props":4274,"children":4275},{},[4276,4291],{"type":45,"tag":452,"props":4277,"children":4278},{},[4279],{"type":45,"tag":456,"props":4280,"children":4281},{},[4282,4287],{"type":45,"tag":460,"props":4283,"children":4284},{},[4285],{"type":51,"value":4286},"Category",{"type":45,"tag":460,"props":4288,"children":4289},{},[4290],{"type":51,"value":637},{"type":45,"tag":476,"props":4292,"children":4293},{},[4294,4310,4326,4342,4358,4374,4390,4406,4427],{"type":45,"tag":456,"props":4295,"children":4296},{},[4297,4305],{"type":45,"tag":483,"props":4298,"children":4299},{},[4300],{"type":45,"tag":58,"props":4301,"children":4302},{},[4303],{"type":51,"value":4304},"System Prompt Deficiency",{"type":45,"tag":483,"props":4306,"children":4307},{},[4308],{"type":51,"value":4309},"Instructions unclear, missing, or contradictory — in symptom span OR its parent",{"type":45,"tag":456,"props":4311,"children":4312},{},[4313,4321],{"type":45,"tag":483,"props":4314,"children":4315},{},[4316],{"type":45,"tag":58,"props":4317,"children":4318},{},[4319],{"type":51,"value":4320},"Tool Gap",{"type":45,"tag":483,"props":4322,"children":4323},{},[4324],{"type":51,"value":4325},"Needed tool doesn't exist or parameters too coarse",{"type":45,"tag":456,"props":4327,"children":4328},{},[4329,4337],{"type":45,"tag":483,"props":4330,"children":4331},{},[4332],{"type":45,"tag":58,"props":4333,"children":4334},{},[4335],{"type":51,"value":4336},"Tool Misuse",{"type":45,"tag":483,"props":4338,"children":4339},{},[4340],{"type":51,"value":4341},"Wrong tool called or wrong parameters — often visible in agent loop or parent LLM",{"type":45,"tag":456,"props":4343,"children":4344},{},[4345,4353],{"type":45,"tag":483,"props":4346,"children":4347},{},[4348],{"type":45,"tag":58,"props":4349,"children":4350},{},[4351],{"type":51,"value":4352},"Routing\u002FHandoff Error",{"type":45,"tag":483,"props":4354,"children":4355},{},[4356],{"type":51,"value":4357},"Wrong sub-agent selected (multi-agent systems)",{"type":45,"tag":456,"props":4359,"children":4360},{},[4361,4369],{"type":45,"tag":483,"props":4362,"children":4363},{},[4364],{"type":45,"tag":58,"props":4365,"children":4366},{},[4367],{"type":51,"value":4368},"Retrieval Failure",{"type":45,"tag":483,"props":4370,"children":4371},{},[4372],{"type":51,"value":4373},"RAG returned irrelevant or missing context — check sibling retrieval spans",{"type":45,"tag":456,"props":4375,"children":4376},{},[4377,4385],{"type":45,"tag":483,"props":4378,"children":4379},{},[4380],{"type":45,"tag":58,"props":4381,"children":4382},{},[4383],{"type":51,"value":4384},"Context Overflow",{"type":45,"tag":483,"props":4386,"children":4387},{},[4388],{"type":51,"value":4389},"Critical info lost due to context length",{"type":45,"tag":456,"props":4391,"children":4392},{},[4393,4401],{"type":45,"tag":483,"props":4394,"children":4395},{},[4396],{"type":45,"tag":58,"props":4397,"children":4398},{},[4399],{"type":51,"value":4400},"Upstream Data Issue",{"type":45,"tag":483,"props":4402,"children":4403},{},[4404],{"type":51,"value":4405},"A sibling or parent span provided bad data that cascaded to the symptom span",{"type":45,"tag":456,"props":4407,"children":4408},{},[4409,4417],{"type":45,"tag":483,"props":4410,"children":4411},{},[4412],{"type":45,"tag":58,"props":4413,"children":4414},{},[4415],{"type":51,"value":4416},"Runtime Error",{"type":45,"tag":483,"props":4418,"children":4419},{},[4420,4422],{"type":51,"value":4421},"Tool\u002FAPI failure, timeout, exception — from ",{"type":45,"tag":76,"props":4423,"children":4425},{"className":4424},[],[4426],{"type":51,"value":1093},{"type":45,"tag":456,"props":4428,"children":4429},{},[4430,4438],{"type":45,"tag":483,"props":4431,"children":4432},{},[4433],{"type":45,"tag":58,"props":4434,"children":4435},{},[4436],{"type":51,"value":4437},"Evaluator Miscalibration",{"type":45,"tag":483,"props":4439,"children":4440},{},[4441],{"type":51,"value":4442},"Eval criteria produce false positives\u002Fnegatives (Eval Signal mode only)",{"type":45,"tag":1589,"props":4444,"children":4445},{},[],{"type":45,"tag":868,"props":4447,"children":4449},{"id":4448},"phase-5-generate-recommendations",[4450],{"type":51,"value":4451},"Phase 5: Generate Recommendations",{"type":45,"tag":54,"props":4453,"children":4454},{},[4455,4459],{"type":45,"tag":58,"props":4456,"children":4457},{},[4458],{"type":51,"value":1390},{"type":51,"value":4460},": Concrete, actionable recommendations grounded in trace evidence. Actual text\u002Fcode changes with before\u002Fafter quotes from the trace — not generic advice.",{"type":45,"tag":54,"props":4462,"children":4463},{},[4464,4466,4471,4473,4478,4480,4485,4486,4491,4493,4498,4500,4505],{"type":51,"value":4465},"Recommendation types: ",{"type":45,"tag":58,"props":4467,"children":4468},{},[4469],{"type":51,"value":4470},"System Prompt Edit",{"type":51,"value":4472}," (quote actual prompt, provide before\u002Fafter), ",{"type":45,"tag":58,"props":4474,"children":4475},{},[4476],{"type":51,"value":4477},"Tool Gap\u002FMisuse",{"type":51,"value":4479}," (reference agent loop steps), ",{"type":45,"tag":58,"props":4481,"children":4482},{},[4483],{"type":51,"value":4484},"Routing\u002FHandoff Fix",{"type":51,"value":254},{"type":45,"tag":58,"props":4487,"children":4488},{},[4489],{"type":51,"value":4490},"Retrieval Fix",{"type":51,"value":4492}," (show retrieved vs needed), ",{"type":45,"tag":58,"props":4494,"children":4495},{},[4496],{"type":51,"value":4497},"Evaluator Prompt Edit",{"type":51,"value":4499}," (flag that eval changes need re-validation; Eval Signal only), ",{"type":45,"tag":58,"props":4501,"children":4502},{},[4503],{"type":51,"value":4504},"Other",{"type":51,"value":335},{"type":45,"tag":54,"props":4507,"children":4508},{},[4509,4514],{"type":45,"tag":58,"props":4510,"children":4511},{},[4512],{"type":51,"value":4513},"When run in Claude Code with codebase access",{"type":51,"value":4515},": Search the codebase for system prompt, tool definitions, or routing logic. Propose specific diffs. Always ask before modifying files.",{"type":45,"tag":1589,"props":4517,"children":4518},{},[],{"type":45,"tag":868,"props":4520,"children":4522},{"id":4521},"phase-6-compile-rca-report",[4523],{"type":51,"value":4524},"Phase 6: Compile RCA Report",{"type":45,"tag":54,"props":4526,"children":4527},{},[4528,4530],{"type":51,"value":4529},"Write the full report following the Output Format below. ",{"type":45,"tag":58,"props":4531,"children":4532},{},[4533],{"type":51,"value":4534},"This is the primary deliverable — output it directly in the chat.",{"type":45,"tag":1589,"props":4536,"children":4537},{},[],{"type":45,"tag":868,"props":4539,"children":4541},{"id":4540},"phase-7-post-analysis-actions",[4542],{"type":51,"value":4543},"Phase 7: Post-Analysis Actions",{"type":45,"tag":54,"props":4545,"children":4546},{},[4547,4552],{"type":45,"tag":58,"props":4548,"children":4549},{},[4550],{"type":51,"value":4551},"Do NOT take any action automatically.",{"type":51,"value":4553}," After presenting the report, ask the user what they'd like to do next:",{"type":45,"tag":66,"props":4555,"children":4556},{},[4557,4568,4573,4578,4599],{"type":45,"tag":70,"props":4558,"children":4559},{},[4560,4562],{"type":51,"value":4561},"Save the report to ",{"type":45,"tag":76,"props":4563,"children":4565},{"className":4564},[],[4566],{"type":51,"value":4567},"agent-observability-rca-{ml_app}-{date}.md",{"type":45,"tag":70,"props":4569,"children":4570},{},[4571],{"type":51,"value":4572},"Apply fixes (if codebase is available)",{"type":45,"tag":70,"props":4574,"children":4575},{},[4576],{"type":51,"value":4577},"Deeper investigation of remaining categories",{"type":45,"tag":70,"props":4579,"children":4580},{},[4581,4583,4589,4591,4597],{"type":51,"value":4582},"Export to a Datadog notebook — in pup mode, use ",{"type":45,"tag":76,"props":4584,"children":4586},{"className":4585},[],[4587],{"type":51,"value":4588},"pup notebooks create",{"type":51,"value":4590}," to create the notebook and ",{"type":45,"tag":76,"props":4592,"children":4594},{"className":4593},[],[4595],{"type":51,"value":4596},"pup notebooks edit NOTEBOOK_ID --file \u002Ftmp\u002Fnb_cells.json",{"type":51,"value":4598}," to append sections (see Tool Reference)",{"type":45,"tag":70,"props":4600,"children":4601},{},[4602,4604,4609,4611,4616],{"type":51,"value":4603},"Re-run on an expanded time range (e.g. ",{"type":45,"tag":76,"props":4605,"children":4607},{"className":4606},[],[4608],{"type":51,"value":298},{"type":51,"value":4610}," if current window was ",{"type":45,"tag":76,"props":4612,"children":4614},{"className":4613},[],[4615],{"type":51,"value":312},{"type":51,"value":665},{"type":45,"tag":54,"props":4618,"children":4619},{},[4620,4625],{"type":45,"tag":58,"props":4621,"children":4622},{},[4623],{"type":51,"value":4624},"If the user chooses option 4",{"type":51,"value":4626},", follow the notebook creation fallback pattern:",{"type":45,"tag":66,"props":4628,"children":4629},{},[4630,4716,4797],{"type":45,"tag":70,"props":4631,"children":4632},{},[4633,4634,4640,4642],{"type":51,"value":2657},{"type":45,"tag":76,"props":4635,"children":4637},{"className":4636},[],[4638],{"type":51,"value":4639},"mcp__datadog-mcp-core__create_datadog_notebook",{"type":51,"value":4641}," with:",{"type":45,"tag":197,"props":4643,"children":4644},{},[4645,4664,4683,4702],{"type":45,"tag":70,"props":4646,"children":4647},{},[4648,4657,4658],{"type":45,"tag":58,"props":4649,"children":4650},{},[4651],{"type":45,"tag":76,"props":4652,"children":4654},{"className":4653},[],[4655],{"type":51,"value":4656},"name",{"type":51,"value":3744},{"type":45,"tag":76,"props":4659,"children":4661},{"className":4660},[],[4662],{"type":51,"value":4663},"Agent Observability RCA: {ml_app} ({mode}) — YYYY-MM-DD",{"type":45,"tag":70,"props":4665,"children":4666},{},[4667,4676,4677],{"type":45,"tag":58,"props":4668,"children":4669},{},[4670],{"type":45,"tag":76,"props":4671,"children":4673},{"className":4672},[],[4674],{"type":51,"value":4675},"type",{"type":51,"value":3744},{"type":45,"tag":76,"props":4678,"children":4680},{"className":4679},[],[4681],{"type":51,"value":4682},"report",{"type":45,"tag":70,"props":4684,"children":4685},{},[4686,4695,4696],{"type":45,"tag":58,"props":4687,"children":4688},{},[4689],{"type":45,"tag":76,"props":4690,"children":4692},{"className":4691},[],[4693],{"type":51,"value":4694},"time_span",{"type":51,"value":3744},{"type":45,"tag":76,"props":4697,"children":4699},{"className":4698},[],[4700],{"type":51,"value":4701},"1w",{"type":45,"tag":70,"props":4703,"children":4704},{},[4705,4714],{"type":45,"tag":58,"props":4706,"children":4707},{},[4708],{"type":45,"tag":76,"props":4709,"children":4711},{"className":4710},[],[4712],{"type":51,"value":4713},"cells",{"type":51,"value":4715},": one cell per section (see Notebook Cell Structure below)",{"type":45,"tag":70,"props":4717,"children":4718},{},[4719,4724,4726],{"type":45,"tag":58,"props":4720,"children":4721},{},[4722],{"type":51,"value":4723},"If the MCP call fails",{"type":51,"value":4725},", inspect the error before giving up:",{"type":45,"tag":197,"props":4727,"children":4728},{},[4729,4739,4749],{"type":45,"tag":70,"props":4730,"children":4731},{},[4732,4737],{"type":45,"tag":58,"props":4733,"children":4734},{},[4735],{"type":51,"value":4736},"Auth \u002F permission error (401, 403)",{"type":51,"value":4738}," → stop and tell the user.",{"type":45,"tag":70,"props":4740,"children":4741},{},[4742,4747],{"type":45,"tag":58,"props":4743,"children":4744},{},[4745],{"type":51,"value":4746},"Field validation error",{"type":51,"value":4748}," (error message names a specific field) → fix that field and retry the MCP call once.",{"type":45,"tag":70,"props":4750,"children":4751},{},[4752,4757,4759],{"type":45,"tag":58,"props":4753,"children":4754},{},[4755],{"type":51,"value":4756},"Any other error",{"type":51,"value":4758}," (binding, serialization, unexpected response) → fall back to pup:\n",{"type":45,"tag":197,"props":4760,"children":4761},{},[4762,4781,4792],{"type":45,"tag":70,"props":4763,"children":4764},{},[4765,4767,4773,4775],{"type":51,"value":4766},"Write the notebook payload to ",{"type":45,"tag":76,"props":4768,"children":4770},{"className":4769},[],[4771],{"type":51,"value":4772},"\u002Ftmp\u002Fnb_rca_{ml_app}.json",{"type":51,"value":4774}," as a full API envelope: ",{"type":45,"tag":76,"props":4776,"children":4778},{"className":4777},[],[4779],{"type":51,"value":4780},"{\"data\": {\"attributes\": {\"name\": \"...\", \"time\": {...}, \"cells\": [...]}, \"type\": \"notebooks\"}}",{"type":45,"tag":70,"props":4782,"children":4783},{},[4784,4786],{"type":51,"value":4785},"Run ",{"type":45,"tag":76,"props":4787,"children":4789},{"className":4788},[],[4790],{"type":51,"value":4791},"pup notebooks create --file \u002Ftmp\u002Fnb_rca_{ml_app}.json",{"type":45,"tag":70,"props":4793,"children":4794},{},[4795],{"type":51,"value":4796},"If pup is not available either, render the full notebook content as markdown in chat so the user has it.",{"type":45,"tag":70,"props":4798,"children":4799},{},[4800,4802],{"type":51,"value":4801},"After successful creation by either method, output the URL on its own line:\n",{"type":45,"tag":76,"props":4803,"children":4805},{"className":4804},[],[4806],{"type":51,"value":4807},"RCA report exported to notebook: \u003Curl>",{"type":45,"tag":54,"props":4809,"children":4810},{},[4811,4813,4819],{"type":51,"value":4812},"Print the URL prominently — if ",{"type":45,"tag":76,"props":4814,"children":4816},{"className":4815},[],[4817],{"type":51,"value":4818},"\u002Feval-bootstrap",{"type":51,"value":4820}," runs next in the same session, it will detect this URL and offer to append the evaluator suite to the same notebook.",{"type":45,"tag":1643,"props":4822,"children":4824},{"id":4823},"notebook-cell-structure",[4825],{"type":51,"value":4826},"Notebook Cell Structure",{"type":45,"tag":448,"props":4828,"children":4829},{},[4830,4846],{"type":45,"tag":452,"props":4831,"children":4832},{},[4833],{"type":45,"tag":456,"props":4834,"children":4835},{},[4836,4841],{"type":45,"tag":460,"props":4837,"children":4838},{},[4839],{"type":51,"value":4840},"Cell",{"type":45,"tag":460,"props":4842,"children":4843},{},[4844],{"type":51,"value":4845},"Content",{"type":45,"tag":476,"props":4847,"children":4848},{},[4849,4862,4875,4888,4901],{"type":45,"tag":456,"props":4850,"children":4851},{},[4852,4857],{"type":45,"tag":483,"props":4853,"children":4854},{},[4855],{"type":51,"value":4856},"1 — Overview",{"type":45,"tag":483,"props":4858,"children":4859},{},[4860],{"type":51,"value":4861},"Structured header (see Overview cell format below — follow it exactly)",{"type":45,"tag":456,"props":4863,"children":4864},{},[4865,4870],{"type":45,"tag":483,"props":4866,"children":4867},{},[4868],{"type":51,"value":4869},"2 — Signal Summary",{"type":45,"tag":483,"props":4871,"children":4872},{},[4873],{"type":51,"value":4874},"Mode-specific health table",{"type":45,"tag":456,"props":4876,"children":4877},{},[4878,4883],{"type":45,"tag":483,"props":4879,"children":4880},{},[4881],{"type":51,"value":4882},"3 — Failure Taxonomy",{"type":45,"tag":483,"props":4884,"children":4885},{},[4886],{"type":51,"value":4887},"Taxonomy table",{"type":45,"tag":456,"props":4889,"children":4890},{},[4891,4896],{"type":45,"tag":483,"props":4892,"children":4893},{},[4894],{"type":51,"value":4895},"4…N — Failure Modes",{"type":45,"tag":483,"props":4897,"children":4898},{},[4899],{"type":51,"value":4900},"One cell per failure mode",{"type":45,"tag":456,"props":4902,"children":4903},{},[4904,4909],{"type":45,"tag":483,"props":4905,"children":4906},{},[4907],{"type":51,"value":4908},"N+1 — Action Plan + Limitations",{"type":45,"tag":483,"props":4910,"children":4911},{},[4912],{"type":51,"value":4913},"Action plan table + bullet list",{"type":45,"tag":54,"props":4915,"children":4916},{},[4917,4922],{"type":45,"tag":58,"props":4918,"children":4919},{},[4920],{"type":51,"value":4921},"Notebook formatting rules",{"type":51,"value":4923}," (apply to every cell):",{"type":45,"tag":197,"props":4925,"children":4926},{},[4927,4953,4963,4973],{"type":45,"tag":70,"props":4928,"children":4929},{},[4930,4935,4937,4943,4945,4951],{"type":45,"tag":58,"props":4931,"children":4932},{},[4933],{"type":51,"value":4934},"No triple-backtick code blocks",{"type":51,"value":4936}," — use blockquotes (",{"type":45,"tag":76,"props":4938,"children":4940},{"className":4939},[],[4941],{"type":51,"value":4942},">",{"type":51,"value":4944},") for prompts\u002Frubrics, inline code (",{"type":45,"tag":76,"props":4946,"children":4948},{"className":4947},[],[4949],{"type":51,"value":4950},"`",{"type":51,"value":4952},") for short values",{"type":45,"tag":70,"props":4954,"children":4955},{},[4956,4961],{"type":45,"tag":58,"props":4957,"children":4958},{},[4959],{"type":51,"value":4960},"Evidence as tables",{"type":51,"value":4962}," — not bullet lists",{"type":45,"tag":70,"props":4964,"children":4965},{},[4966,4971],{"type":45,"tag":58,"props":4967,"children":4968},{},[4969],{"type":51,"value":4970},"Tool inputs as tables",{"type":51,"value":4972}," — Argument | Wrong value passed | Correct approach",{"type":45,"tag":70,"props":4974,"children":4975},{},[4976,4981],{"type":45,"tag":58,"props":4977,"children":4978},{},[4979],{"type":51,"value":4980},"Action plan as a table",{"type":51,"value":4982}," — Priority | Action | Confidence | Impact",{"type":45,"tag":1589,"props":4984,"children":4985},{},[],{"type":45,"tag":46,"props":4987,"children":4989},{"id":4988},"output-format",[4990],{"type":51,"value":4991},"Output Format",{"type":45,"tag":1589,"props":4993,"children":4994},{},[],{"type":45,"tag":868,"props":4996,"children":4998},{"id":4997},"overview-cell-notebook-cell-1-report-header",[4999],{"type":51,"value":5000},"Overview cell (notebook Cell 1 \u002F report header)",{"type":45,"tag":54,"props":5002,"children":5003},{},[5004],{"type":51,"value":5005},"The Overview cell must follow this exact structure. No prose paragraphs. No inline-numbered findings. App description is one sentence maximum.",{"type":45,"tag":592,"props":5007,"children":5010},{"className":5008,"code":5009,"language":51},[595],"## `{ml_app}`  ·  {Eval Signal | Error Signal | Generic}  ·  {timeframe}\n**Date**: {YYYY-MM-DD}  |  **Profile**: {short app profile}  |  **Model**: `{model(s)}`\n\n{One sentence: what does this app do?}\n\n| Metric | Value |\n|--------|-------|\n{mode-appropriate rows — see below}\n\n### Findings\n\n- **{Finding 1}** (~{pct}%): one-line root cause description\n- **{Finding 2}** (~{pct}%): one-line root cause description\n- **{Finding 3}** (if present): one-line root cause description\n\n### Recommendations\n\n- **{Recommendation 1}**: specific next step tied to Finding 1\n- **{Recommendation 2}**: specific next step tied to Finding 2\n\n*Sample: {N} spans analyzed. Confidence: High | Medium | Low — {one-line reason if Medium or Low}.*\n",[5011],{"type":45,"tag":76,"props":5012,"children":5013},{"__ignoreMap":600},[5014],{"type":51,"value":5009},{"type":45,"tag":54,"props":5016,"children":5017},{},[5018],{"type":45,"tag":58,"props":5019,"children":5020},{},[5021],{"type":51,"value":5022},"Mode-appropriate metric rows:",{"type":45,"tag":54,"props":5024,"children":5025},{},[5026],{"type":51,"value":5027},"Eval Signal:",{"type":45,"tag":592,"props":5029,"children":5032},{"className":5030,"code":5031,"language":51},[595],"| Eval | `{eval_name}` ({type}) |\n| Spans evaluated | {total_count} |\n| Pass rate | {pass_rate}% ({pass_count} pass \u002F {fail_count} fail) |\n| Top failure mode | {name} (~{pct}%) |\n| Evals configured | {N} |\n",[5033],{"type":45,"tag":76,"props":5034,"children":5035},{"__ignoreMap":600},[5036],{"type":51,"value":5031},{"type":45,"tag":54,"props":5038,"children":5039},{},[5040],{"type":51,"value":5041},"Error Signal:",{"type":45,"tag":592,"props":5043,"children":5046},{"className":5044,"code":5045,"language":51},[595],"| Error spans | {N} confirmed |\n| Top error type | `{type}` ({pct}%) |\n| Affected operation | `{span_name}` |\n| Cascade pattern | Isolated \u002F Cascading |\n| Evals configured | {N} (none = no quality signal) |\n",[5047],{"type":45,"tag":76,"props":5048,"children":5049},{"__ignoreMap":600},[5050],{"type":51,"value":5045},{"type":45,"tag":54,"props":5052,"children":5053},{},[5054],{"type":51,"value":5055},"Generic:",{"type":45,"tag":592,"props":5057,"children":5060},{"className":5058,"code":5059,"language":51},[595],"| Spans sampled | {N} root spans |\n| Top anomaly | {type}: {count} spans |\n| Error spans | {N} (0 = structurally healthy) |\n| Evals configured | {N} (none = no quality signal) |\n",[5061],{"type":45,"tag":76,"props":5062,"children":5063},{"__ignoreMap":600},[5064],{"type":51,"value":5059},{"type":45,"tag":1589,"props":5066,"children":5067},{},[],{"type":45,"tag":868,"props":5069,"children":5071},{"id":5070},"signal-summary-table",[5072],{"type":51,"value":5073},"Signal Summary Table",{"type":45,"tag":54,"props":5075,"children":5076},{},[5077,5082],{"type":45,"tag":58,"props":5078,"children":5079},{},[5080],{"type":51,"value":5081},"When entering from Step 0S (classification context)",{"type":51,"value":5083},", replace the Signal Summary table with:",{"type":45,"tag":592,"props":5085,"children":5088},{"className":5086,"code":5087,"language":51},[595],"## Classification Signal Summary\n\n**Source**: agent-observability-session-classify  |  **ml_app**: {app}  |  **Signal**: content-only | content+evals\n\n| Metric | Value |\n|--------|-------|\n| Traces classified | N |\n| Failures in corpus (no+partial) | F |\n| Pass rate | X% |\n| Failure modes | list |\n\n*Root cause analysis is based on per-trace classification verdicts, not automated eval judge reasoning.*\n",[5089],{"type":45,"tag":76,"props":5090,"children":5091},{"__ignoreMap":600},[5092],{"type":51,"value":5087},{"type":45,"tag":54,"props":5094,"children":5095},{},[5096,5101],{"type":45,"tag":58,"props":5097,"children":5098},{},[5099],{"type":51,"value":5100},"Otherwise",{"type":51,"value":5102},", mode-specific — pick the appropriate variant:",{"type":45,"tag":54,"props":5104,"children":5105},{},[5106,5110],{"type":45,"tag":58,"props":5107,"children":5108},{},[5109],{"type":51,"value":490},{"type":51,"value":5111}," — one row per eval:",{"type":45,"tag":448,"props":5113,"children":5114},{},[5115,5148],{"type":45,"tag":452,"props":5116,"children":5117},{},[5118],{"type":45,"tag":456,"props":5119,"children":5120},{},[5121,5126,5131,5137,5143],{"type":45,"tag":460,"props":5122,"children":5123},{},[5124],{"type":51,"value":5125},"Eval",{"type":45,"tag":460,"props":5127,"children":5128},{},[5129],{"type":51,"value":5130},"Type",{"type":45,"tag":460,"props":5132,"children":5134},{"align":5133},"right",[5135],{"type":51,"value":5136},"Total",{"type":45,"tag":460,"props":5138,"children":5140},{"align":5139},"center",[5141],{"type":51,"value":5142},"Pass Rate",{"type":45,"tag":460,"props":5144,"children":5145},{},[5146],{"type":51,"value":5147},"Status",{"type":45,"tag":476,"props":5149,"children":5150},{},[5151],{"type":45,"tag":456,"props":5152,"children":5153},{},[5154,5159,5164,5169,5174],{"type":45,"tag":483,"props":5155,"children":5156},{},[5157],{"type":51,"value":5158},"eval_1",{"type":45,"tag":483,"props":5160,"children":5161},{},[5162],{"type":51,"value":5163},"boolean",{"type":45,"tag":483,"props":5165,"children":5166},{"align":5133},[5167],{"type":51,"value":5168},"4,891",{"type":45,"tag":483,"props":5170,"children":5171},{"align":5139},[5172],{"type":51,"value":5173},"37.3%",{"type":45,"tag":483,"props":5175,"children":5176},{},[5177],{"type":51,"value":5178},"⚠ Investigating",{"type":45,"tag":54,"props":5180,"children":5181},{},[5182,5186],{"type":45,"tag":58,"props":5183,"children":5184},{},[5185],{"type":51,"value":511},{"type":51,"value":5187}," — one row per error type:",{"type":45,"tag":448,"props":5189,"children":5190},{},[5191,5217],{"type":45,"tag":452,"props":5192,"children":5193},{},[5194],{"type":45,"tag":456,"props":5195,"children":5196},{},[5197,5202,5207,5212],{"type":45,"tag":460,"props":5198,"children":5199},{},[5200],{"type":51,"value":5201},"Error Type",{"type":45,"tag":460,"props":5203,"children":5204},{"align":5133},[5205],{"type":51,"value":5206},"Spans",{"type":45,"tag":460,"props":5208,"children":5209},{"align":5139},[5210],{"type":51,"value":5211},"Cascade?",{"type":45,"tag":460,"props":5213,"children":5214},{},[5215],{"type":51,"value":5216},"Origin Span Kind",{"type":45,"tag":476,"props":5218,"children":5219},{},[5220],{"type":45,"tag":456,"props":5221,"children":5222},{},[5223,5228,5233,5238],{"type":45,"tag":483,"props":5224,"children":5225},{},[5226],{"type":51,"value":5227},"TimeoutError",{"type":45,"tag":483,"props":5229,"children":5230},{"align":5133},[5231],{"type":51,"value":5232},"42",{"type":45,"tag":483,"props":5234,"children":5235},{"align":5139},[5236],{"type":51,"value":5237},"Yes",{"type":45,"tag":483,"props":5239,"children":5240},{},[5241],{"type":51,"value":3414},{"type":45,"tag":54,"props":5243,"children":5244},{},[5245,5249],{"type":45,"tag":58,"props":5246,"children":5247},{},[5248],{"type":51,"value":540},{"type":51,"value":5250}," — one row per anomaly type:",{"type":45,"tag":448,"props":5252,"children":5253},{},[5254,5270],{"type":45,"tag":452,"props":5255,"children":5256},{},[5257],{"type":45,"tag":456,"props":5258,"children":5259},{},[5260,5265],{"type":45,"tag":460,"props":5261,"children":5262},{},[5263],{"type":51,"value":5264},"Anomaly Type",{"type":45,"tag":460,"props":5266,"children":5267},{"align":5139},[5268],{"type":51,"value":5269},"Count",{"type":45,"tag":476,"props":5271,"children":5272},{},[5273],{"type":45,"tag":456,"props":5274,"children":5275},{},[5276,5281],{"type":45,"tag":483,"props":5277,"children":5278},{},[5279],{"type":51,"value":5280},"Latency outliers (>p90)",{"type":45,"tag":483,"props":5282,"children":5283},{"align":5139},[5284],{"type":51,"value":5285},"12",{"type":45,"tag":1589,"props":5287,"children":5288},{},[],{"type":45,"tag":868,"props":5290,"children":5292},{"id":5291},"failure-taxonomy",[5293],{"type":51,"value":5294},"Failure Taxonomy",{"type":45,"tag":448,"props":5296,"children":5297},{},[5298,5334],{"type":45,"tag":452,"props":5299,"children":5300},{},[5301],{"type":45,"tag":456,"props":5302,"children":5303},{},[5304,5309,5314,5319,5324,5329],{"type":45,"tag":460,"props":5305,"children":5306},{},[5307],{"type":51,"value":5308},"#",{"type":45,"tag":460,"props":5310,"children":5311},{},[5312],{"type":51,"value":5313},"Failure Mode",{"type":45,"tag":460,"props":5315,"children":5316},{"align":5133},[5317],{"type":51,"value":5318},"Traces",{"type":45,"tag":460,"props":5320,"children":5321},{"align":5139},[5322],{"type":51,"value":5323},"%",{"type":45,"tag":460,"props":5325,"children":5326},{"align":5139},[5327],{"type":51,"value":5328},"Severity",{"type":45,"tag":460,"props":5330,"children":5331},{},[5332],{"type":51,"value":5333},"Root Cause",{"type":45,"tag":476,"props":5335,"children":5336},{},[5337],{"type":45,"tag":456,"props":5338,"children":5339},{},[5340,5345,5350,5354,5359,5367],{"type":45,"tag":483,"props":5341,"children":5342},{},[5343],{"type":51,"value":5344},"1",{"type":45,"tag":483,"props":5346,"children":5347},{},[5348],{"type":51,"value":5349},"...",{"type":45,"tag":483,"props":5351,"children":5352},{"align":5133},[5353],{"type":51,"value":5349},{"type":45,"tag":483,"props":5355,"children":5356},{"align":5139},[5357],{"type":51,"value":5358},"...%",{"type":45,"tag":483,"props":5360,"children":5361},{"align":5139},[5362],{"type":45,"tag":58,"props":5363,"children":5364},{},[5365],{"type":51,"value":5366},"High",{"type":45,"tag":483,"props":5368,"children":5369},{},[5370],{"type":51,"value":4336},{"type":45,"tag":1589,"props":5372,"children":5373},{},[],{"type":45,"tag":868,"props":5375,"children":5377},{"id":5376},"failure-mode-sections-one-per-top-35-modes",[5378],{"type":51,"value":5379},"Failure Mode Sections (one per top 3–5 modes)",{"type":45,"tag":592,"props":5381,"children":5384},{"className":5382,"code":5383,"language":51},[595],"## Failure Mode N: [Name]\n\n**Count**: {n} spans, {t} traces  |  **Severity**: High\u002FMedium\u002FLow  |  **Root Cause**: [Category]\n\n[3–5 sentences: what goes wrong, when, what triggers it, causal chain.]\n\n**Evidence**\n\n{Use the mode-appropriate column set:}\n\nEval Signal — Trace | Judge verdict | What the trace revealed:\n| Trace | Judge verdict | What the trace revealed |\n|---|---|---|\n| [69de86a7...](https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_id}) | fail | Parent agent has no date format instruction |\n\nError Signal — Trace | Behavior | Version:\n| Trace | Behavior | Version |\n|---|---|---|\n| [69de86a7...](https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_id}) | 7 parallel calls, all 400 | v107624932 |\n\nGeneric — Trace | Anomaly | Signal:\n| Trace | Anomaly | Signal |\n|---|---|---|\n| [69de86a7...](https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_id}) | 94s, 12 tool calls | Latency outlier |\n\n{For tool misuse — add a tool inputs table:}\n**Tool inputs (100% of sampled calls)**\n\n| Argument | Value passed (wrong) | Correct approach |\n|---|---|---|\n| `query` | `\"monitor_id:123 group_status:alert\"` | `\"monitor_id:123\"` (name\u002Ftag only) |\n\n{For Eval Signal — add judge reasoning as a blockquote:}\n> \"{quoted judge reasoning}\"\n\n**Root cause**: [WHY this happens — specific span, parameter, or prompt.]\n\n**Fix**:\n  BEFORE: [actual text from trace]\n  AFTER:  [proposed replacement]\n\n**Impact**: Eliminates ~{n} spans \u002F {timeframe}.\n",[5385],{"type":45,"tag":76,"props":5386,"children":5387},{"__ignoreMap":600},[5388],{"type":51,"value":5383},{"type":45,"tag":1589,"props":5390,"children":5391},{},[],{"type":45,"tag":868,"props":5393,"children":5395},{"id":5394},"prioritized-action-plan",[5396],{"type":51,"value":5397},"Prioritized Action Plan",{"type":45,"tag":448,"props":5399,"children":5400},{},[5401,5427],{"type":45,"tag":452,"props":5402,"children":5403},{},[5404],{"type":45,"tag":456,"props":5405,"children":5406},{},[5407,5412,5417,5422],{"type":45,"tag":460,"props":5408,"children":5409},{"align":5139},[5410],{"type":51,"value":5411},"Priority",{"type":45,"tag":460,"props":5413,"children":5414},{},[5415],{"type":51,"value":5416},"Action",{"type":45,"tag":460,"props":5418,"children":5419},{"align":5139},[5420],{"type":51,"value":5421},"Confidence",{"type":45,"tag":460,"props":5423,"children":5424},{},[5425],{"type":51,"value":5426},"Impact",{"type":45,"tag":476,"props":5428,"children":5429},{},[5430],{"type":45,"tag":456,"props":5431,"children":5432},{},[5433,5437,5458,5462],{"type":45,"tag":483,"props":5434,"children":5435},{"align":5139},[5436],{"type":51,"value":5344},{"type":45,"tag":483,"props":5438,"children":5439},{},[5440,5442,5448,5450,5456],{"type":51,"value":5441},"Fix ",{"type":45,"tag":76,"props":5443,"children":5445},{"className":5444},[],[5446],{"type":51,"value":5447},"monitor_groups_search",{"type":51,"value":5449}," schema — add ",{"type":45,"tag":76,"props":5451,"children":5453},{"className":5452},[],[5454],{"type":51,"value":5455},"group_states",{"type":51,"value":5457}," param",{"type":45,"tag":483,"props":5459,"children":5460},{"align":5139},[5461],{"type":51,"value":5366},{"type":45,"tag":483,"props":5463,"children":5464},{},[5465],{"type":51,"value":5466},"Eliminates ~21 spans\u002F7d",{"type":45,"tag":54,"props":5468,"children":5469},{},[5470,5475],{"type":45,"tag":58,"props":5471,"children":5472},{},[5473],{"type":51,"value":5474},"When mode is Generic and no evals are configured",{"type":51,"value":5476},", always append as the final action plan row:",{"type":45,"tag":54,"props":5478,"children":5479},{},[5480],{"type":51,"value":5481},"| N | Configure at least one evaluator | High | Enables Eval Signal mode for future RCAs — app currently has no ongoing quality signal |",{"type":45,"tag":1589,"props":5483,"children":5484},{},[],{"type":45,"tag":868,"props":5486,"children":5488},{"id":5487},"limitations-follow-ups",[5489],{"type":51,"value":5490},"Limitations & Follow-ups",{"type":45,"tag":54,"props":5492,"children":5493},{},[5494],{"type":51,"value":5495},"Bullet list of what needs more data or follow-up action.",{"type":45,"tag":1589,"props":5497,"children":5498},{},[],{"type":45,"tag":46,"props":5500,"children":5502},{"id":5501},"operating-rules",[5503],{"type":51,"value":5504},"Operating Rules",{"type":45,"tag":197,"props":5506,"children":5507},{},[5508,5525,5535,5545,5555,5565],{"type":45,"tag":70,"props":5509,"children":5510},{},[5511,5516,5518,5524],{"type":45,"tag":58,"props":5512,"children":5513},{},[5514],{"type":51,"value":5515},"Ground in evidence",{"type":51,"value":5517},": Every claim references span IDs with clickable trace links: ",{"type":45,"tag":76,"props":5519,"children":5521},{"className":5520},[],[5522],{"type":51,"value":5523},"[Trace {first_8}...](https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_32_char_id})",{"type":51,"value":335},{"type":45,"tag":70,"props":5526,"children":5527},{},[5528,5533],{"type":45,"tag":58,"props":5529,"children":5530},{},[5531],{"type":51,"value":5532},"Root cause over symptom",{"type":51,"value":5534},": \"System prompt doesn't specify date format\" not \"model gave wrong answer.\"",{"type":45,"tag":70,"props":5536,"children":5537},{},[5538,5543],{"type":45,"tag":58,"props":5539,"children":5540},{},[5541],{"type":51,"value":5542},"Show your math",{"type":51,"value":5544},": \"47 failures (34%)\" not \"many failures.\"",{"type":45,"tag":70,"props":5546,"children":5547},{},[5548,5553],{"type":45,"tag":58,"props":5549,"children":5550},{},[5551],{"type":51,"value":5552},"Honest about uncertainty",{"type":51,"value":5554},": \u003C 5 examples = tentative. Flag it.",{"type":45,"tag":70,"props":5556,"children":5557},{},[5558,5563],{"type":45,"tag":58,"props":5559,"children":5560},{},[5561],{"type":51,"value":5562},"Anonymize PII",{"type":51,"value":5564},": No emails or names. User\u002Forg IDs are fine.",{"type":45,"tag":70,"props":5566,"children":5567},{},[5568,5573,5575,5581,5583,5589],{"type":45,"tag":58,"props":5569,"children":5570},{},[5571],{"type":51,"value":5572},"MCP result parsing safety",{"type":51,"value":5574},": Before writing any script that iterates over MCP tool results, inspect the raw structure first — check top-level keys and whether the payload is nested inside a content block (e.g. ",{"type":45,"tag":76,"props":5576,"children":5578},{"className":5577},[],[5579],{"type":51,"value":5580},"[{'type': 'text', 'text': '\u003Cjson>'}]",{"type":51,"value":5582},"). Extract and ",{"type":45,"tag":76,"props":5584,"children":5586},{"className":5585},[],[5587],{"type":51,"value":5588},"json.loads()",{"type":51,"value":5590}," the inner payload if needed. Never assume MCP results are bare dicts or lists.",{"type":45,"tag":1589,"props":5592,"children":5593},{},[],{"type":45,"tag":46,"props":5595,"children":5597},{"id":5596},"tool-reference",[5598],{"type":51,"value":5599},"Tool Reference",{"type":45,"tag":54,"props":5601,"children":5602},{},[5603,5605,5609],{"type":51,"value":5604},"This appendix applies only in ",{"type":45,"tag":58,"props":5606,"children":5607},{},[5608],{"type":51,"value":88},{"type":51,"value":5610},". In MCP mode, use the tool names in the workflow sections directly.",{"type":45,"tag":868,"props":5612,"children":5614},{"id":5613},"spans-and-traces",[5615],{"type":51,"value":5616},"Spans and traces",{"type":45,"tag":448,"props":5618,"children":5619},{},[5620,5636],{"type":45,"tag":452,"props":5621,"children":5622},{},[5623],{"type":45,"tag":456,"props":5624,"children":5625},{},[5626,5631],{"type":45,"tag":460,"props":5627,"children":5628},{},[5629],{"type":51,"value":5630},"MCP Tool",{"type":45,"tag":460,"props":5632,"children":5633},{},[5634],{"type":51,"value":5635},"pup Command",{"type":45,"tag":476,"props":5637,"children":5638},{},[5639,5685,5706,5727,5748,5768,5788],{"type":45,"tag":456,"props":5640,"children":5641},{},[5642,5651],{"type":45,"tag":483,"props":5643,"children":5644},{},[5645],{"type":45,"tag":76,"props":5646,"children":5648},{"className":5647},[],[5649],{"type":51,"value":5650},"search_llmobs_spans(query, ml_app, from, to, limit, cursor, root_spans_only, span_kind, summary)",{"type":45,"tag":483,"props":5652,"children":5653},{},[5654,5660,5662,5675,5677,5683],{"type":45,"tag":76,"props":5655,"children":5657},{"className":5656},[],[5658],{"type":51,"value":5659},"pup llm-obs spans search --query \"@ml_app:A [other_filters]\" [--from F] [--to T] [--limit N] [--cursor C] [--root-spans-only] [--span-kind K] [--summary]",{"type":51,"value":5661}," — ",{"type":45,"tag":58,"props":5663,"children":5664},{},[5665,5667,5673],{"type":51,"value":5666},"always use ",{"type":45,"tag":76,"props":5668,"children":5670},{"className":5669},[],[5671],{"type":51,"value":5672},"--query \"@ml_app:A\"",{"type":51,"value":5674}," to filter by ml_app",{"type":51,"value":5676},"; the ",{"type":45,"tag":76,"props":5678,"children":5680},{"className":5679},[],[5681],{"type":51,"value":5682},"--ml-app A",{"type":51,"value":5684}," flag is unreliable and silently returns spans from other apps.",{"type":45,"tag":456,"props":5686,"children":5687},{},[5688,5697],{"type":45,"tag":483,"props":5689,"children":5690},{},[5691],{"type":45,"tag":76,"props":5692,"children":5694},{"className":5693},[],[5695],{"type":51,"value":5696},"get_llmobs_span_details(trace_id, span_ids, from, to)",{"type":45,"tag":483,"props":5698,"children":5699},{},[5700],{"type":45,"tag":76,"props":5701,"children":5703},{"className":5702},[],[5704],{"type":51,"value":5705},"pup llm-obs spans get-details --trace-id T --span-ids S1,S2,...",{"type":45,"tag":456,"props":5707,"children":5708},{},[5709,5718],{"type":45,"tag":483,"props":5710,"children":5711},{},[5712],{"type":45,"tag":76,"props":5713,"children":5715},{"className":5714},[],[5716],{"type":51,"value":5717},"get_llmobs_span_content(trace_id, span_id, field, path)",{"type":45,"tag":483,"props":5719,"children":5720},{},[5721],{"type":45,"tag":76,"props":5722,"children":5724},{"className":5723},[],[5725],{"type":51,"value":5726},"pup llm-obs spans get-content --trace-id T --span-id S --field F [--path P]",{"type":45,"tag":456,"props":5728,"children":5729},{},[5730,5739],{"type":45,"tag":483,"props":5731,"children":5732},{},[5733],{"type":45,"tag":76,"props":5734,"children":5736},{"className":5735},[],[5737],{"type":51,"value":5738},"get_llmobs_trace(trace_id, include_tree)",{"type":45,"tag":483,"props":5740,"children":5741},{},[5742],{"type":45,"tag":76,"props":5743,"children":5745},{"className":5744},[],[5746],{"type":51,"value":5747},"pup llm-obs spans get-trace --trace-id T [--include-tree]",{"type":45,"tag":456,"props":5749,"children":5750},{},[5751,5759],{"type":45,"tag":483,"props":5752,"children":5753},{},[5754],{"type":45,"tag":76,"props":5755,"children":5757},{"className":5756},[],[5758],{"type":51,"value":4001},{"type":45,"tag":483,"props":5760,"children":5761},{},[5762],{"type":45,"tag":76,"props":5763,"children":5765},{"className":5764},[],[5766],{"type":51,"value":5767},"pup llm-obs spans get-agent-loop --trace-id T [--span-id S]",{"type":45,"tag":456,"props":5769,"children":5770},{},[5771,5779],{"type":45,"tag":483,"props":5772,"children":5773},{},[5774],{"type":45,"tag":76,"props":5775,"children":5777},{"className":5776},[],[5778],{"type":51,"value":2750},{"type":45,"tag":483,"props":5780,"children":5781},{},[5782],{"type":45,"tag":76,"props":5783,"children":5785},{"className":5784},[],[5786],{"type":51,"value":5787},"pup llm-obs spans find-errors --trace-id T",{"type":45,"tag":456,"props":5789,"children":5790},{},[5791,5800],{"type":45,"tag":483,"props":5792,"children":5793},{},[5794],{"type":45,"tag":76,"props":5795,"children":5797},{"className":5796},[],[5798],{"type":51,"value":5799},"expand_llmobs_spans(trace_id, span_ids, max_depth, filter_kind)",{"type":45,"tag":483,"props":5801,"children":5802},{},[5803],{"type":45,"tag":76,"props":5804,"children":5806},{"className":5805},[],[5807],{"type":51,"value":5808},"pup llm-obs spans expand --trace-id T --span-ids S1,S2,... [--max-depth N] [--filter-kind K]",{"type":45,"tag":868,"props":5810,"children":5812},{"id":5811},"evaluators",[5813],{"type":51,"value":5814},"Evaluators",{"type":45,"tag":448,"props":5816,"children":5817},{},[5818,5832],{"type":45,"tag":452,"props":5819,"children":5820},{},[5821],{"type":45,"tag":456,"props":5822,"children":5823},{},[5824,5828],{"type":45,"tag":460,"props":5825,"children":5826},{},[5827],{"type":51,"value":5630},{"type":45,"tag":460,"props":5829,"children":5830},{},[5831],{"type":51,"value":5635},{"type":45,"tag":476,"props":5833,"children":5834},{},[5835,5865,5885,5905],{"type":45,"tag":456,"props":5836,"children":5837},{},[5838,5847],{"type":45,"tag":483,"props":5839,"children":5840},{},[5841],{"type":45,"tag":76,"props":5842,"children":5844},{"className":5843},[],[5845],{"type":51,"value":5846},"list_llmobs_evals()",{"type":45,"tag":483,"props":5848,"children":5849},{},[5850,5856,5858,5863],{"type":45,"tag":76,"props":5851,"children":5853},{"className":5852},[],[5854],{"type":51,"value":5855},"pup llm-obs evals list",{"type":51,"value":5857}," (filter by ",{"type":45,"tag":76,"props":5859,"children":5861},{"className":5860},[],[5862],{"type":51,"value":652},{"type":51,"value":5864}," client-side)",{"type":45,"tag":456,"props":5866,"children":5867},{},[5868,5876],{"type":45,"tag":483,"props":5869,"children":5870},{},[5871],{"type":45,"tag":76,"props":5872,"children":5874},{"className":5873},[],[5875],{"type":51,"value":2113},{"type":45,"tag":483,"props":5877,"children":5878},{},[5879],{"type":45,"tag":76,"props":5880,"children":5882},{"className":5881},[],[5883],{"type":51,"value":5884},"pup llm-obs evals list-by-ml-app --ml-app A",{"type":45,"tag":456,"props":5886,"children":5887},{},[5888,5896],{"type":45,"tag":483,"props":5889,"children":5890},{},[5891],{"type":45,"tag":76,"props":5892,"children":5894},{"className":5893},[],[5895],{"type":51,"value":2248},{"type":45,"tag":483,"props":5897,"children":5898},{},[5899],{"type":45,"tag":76,"props":5900,"children":5902},{"className":5901},[],[5903],{"type":51,"value":5904},"pup llm-obs evals get-evaluator EVAL_NAME",{"type":45,"tag":456,"props":5906,"children":5907},{},[5908,5917],{"type":45,"tag":483,"props":5909,"children":5910},{},[5911],{"type":45,"tag":76,"props":5912,"children":5914},{"className":5913},[],[5915],{"type":51,"value":5916},"get_llmobs_eval_aggregate_stats(eval_name, ml_app, from, to)",{"type":45,"tag":483,"props":5918,"children":5919},{},[5920],{"type":45,"tag":76,"props":5921,"children":5923},{"className":5922},[],[5924],{"type":51,"value":5925},"pup llm-obs evals get-aggregate-stats EVAL_NAME [--ml-app A] [--from F] [--to T]",{"type":45,"tag":868,"props":5927,"children":5929},{"id":5928},"notebooks",[5930],{"type":51,"value":5931},"Notebooks",{"type":45,"tag":448,"props":5933,"children":5934},{},[5935,5949],{"type":45,"tag":452,"props":5936,"children":5937},{},[5938],{"type":45,"tag":456,"props":5939,"children":5940},{},[5941,5945],{"type":45,"tag":460,"props":5942,"children":5943},{},[5944],{"type":51,"value":5630},{"type":45,"tag":460,"props":5946,"children":5947},{},[5948],{"type":51,"value":5635},{"type":45,"tag":476,"props":5950,"children":5951},{},[5952,5981],{"type":45,"tag":456,"props":5953,"children":5954},{},[5955,5964],{"type":45,"tag":483,"props":5956,"children":5957},{},[5958],{"type":45,"tag":76,"props":5959,"children":5961},{"className":5960},[],[5962],{"type":51,"value":5963},"create_datadog_notebook(name, cells, ...)",{"type":45,"tag":483,"props":5965,"children":5966},{},[5967,5973,5975],{"type":45,"tag":76,"props":5968,"children":5970},{"className":5969},[],[5971],{"type":51,"value":5972},"pup notebooks create --title \"TITLE\" --file \u002Ftmp\u002Fnb_cells.json",{"type":51,"value":5974}," — confirm exact flags with ",{"type":45,"tag":76,"props":5976,"children":5978},{"className":5977},[],[5979],{"type":51,"value":5980},"pup notebooks create --help",{"type":45,"tag":456,"props":5982,"children":5983},{},[5984,5993],{"type":45,"tag":483,"props":5985,"children":5986},{},[5987],{"type":45,"tag":76,"props":5988,"children":5990},{"className":5989},[],[5991],{"type":51,"value":5992},"edit_datadog_notebook(id, cells, append_only=true)",{"type":45,"tag":483,"props":5994,"children":5995},{},[5996,6001],{"type":45,"tag":76,"props":5997,"children":5999},{"className":5998},[],[6000],{"type":51,"value":4596},{"type":51,"value":6002}," (fetches current notebook, appends provided cells, writes back)",{"type":45,"tag":54,"props":6004,"children":6005},{},[6006],{"type":51,"value":6007},"The cells file is a JSON array of cell objects:",{"type":45,"tag":592,"props":6009,"children":6013},{"className":6010,"code":6011,"language":6012,"meta":600,"style":600},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","[{\"attributes\": {\"definition\": {\"type\": \"markdown\", \"text\": \"## Section\\n\\nContent.\"}}, \"type\": \"notebook_cells\"}]\n","json",[6014],{"type":45,"tag":76,"props":6015,"children":6016},{"__ignoreMap":600},[6017],{"type":45,"tag":2190,"props":6018,"children":6021},{"class":6019,"line":6020},"line",1,[6022,6028,6033,6039,6043,6047,6052,6056,6062,6066,6070,6074,6078,6083,6087,6091,6096,6102,6106,6111,6115,6119,6123,6127,6131,6136,6142,6147,6151,6156,6160,6164,6168,6172,6176,6181,6185],{"type":45,"tag":2190,"props":6023,"children":6025},{"style":6024},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[6026],{"type":51,"value":6027},"[{",{"type":45,"tag":2190,"props":6029,"children":6030},{"style":6024},[6031],{"type":51,"value":6032},"\"",{"type":45,"tag":2190,"props":6034,"children":6036},{"style":6035},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[6037],{"type":51,"value":6038},"attributes",{"type":45,"tag":2190,"props":6040,"children":6041},{"style":6024},[6042],{"type":51,"value":6032},{"type":45,"tag":2190,"props":6044,"children":6045},{"style":6024},[6046],{"type":51,"value":3571},{"type":45,"tag":2190,"props":6048,"children":6049},{"style":6024},[6050],{"type":51,"value":6051}," {",{"type":45,"tag":2190,"props":6053,"children":6054},{"style":6024},[6055],{"type":51,"value":6032},{"type":45,"tag":2190,"props":6057,"children":6059},{"style":6058},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[6060],{"type":51,"value":6061},"definition",{"type":45,"tag":2190,"props":6063,"children":6064},{"style":6024},[6065],{"type":51,"value":6032},{"type":45,"tag":2190,"props":6067,"children":6068},{"style":6024},[6069],{"type":51,"value":3571},{"type":45,"tag":2190,"props":6071,"children":6072},{"style":6024},[6073],{"type":51,"value":6051},{"type":45,"tag":2190,"props":6075,"children":6076},{"style":6024},[6077],{"type":51,"value":6032},{"type":45,"tag":2190,"props":6079,"children":6081},{"style":6080},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[6082],{"type":51,"value":4675},{"type":45,"tag":2190,"props":6084,"children":6085},{"style":6024},[6086],{"type":51,"value":6032},{"type":45,"tag":2190,"props":6088,"children":6089},{"style":6024},[6090],{"type":51,"value":3571},{"type":45,"tag":2190,"props":6092,"children":6093},{"style":6024},[6094],{"type":51,"value":6095}," \"",{"type":45,"tag":2190,"props":6097,"children":6099},{"style":6098},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[6100],{"type":51,"value":6101},"markdown",{"type":45,"tag":2190,"props":6103,"children":6104},{"style":6024},[6105],{"type":51,"value":6032},{"type":45,"tag":2190,"props":6107,"children":6108},{"style":6024},[6109],{"type":51,"value":6110},",",{"type":45,"tag":2190,"props":6112,"children":6113},{"style":6024},[6114],{"type":51,"value":6095},{"type":45,"tag":2190,"props":6116,"children":6117},{"style":6080},[6118],{"type":51,"value":51},{"type":45,"tag":2190,"props":6120,"children":6121},{"style":6024},[6122],{"type":51,"value":6032},{"type":45,"tag":2190,"props":6124,"children":6125},{"style":6024},[6126],{"type":51,"value":3571},{"type":45,"tag":2190,"props":6128,"children":6129},{"style":6024},[6130],{"type":51,"value":6095},{"type":45,"tag":2190,"props":6132,"children":6133},{"style":6098},[6134],{"type":51,"value":6135},"## Section",{"type":45,"tag":2190,"props":6137,"children":6139},{"style":6138},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[6140],{"type":51,"value":6141},"\\n\\n",{"type":45,"tag":2190,"props":6143,"children":6144},{"style":6098},[6145],{"type":51,"value":6146},"Content.",{"type":45,"tag":2190,"props":6148,"children":6149},{"style":6024},[6150],{"type":51,"value":6032},{"type":45,"tag":2190,"props":6152,"children":6153},{"style":6024},[6154],{"type":51,"value":6155},"}},",{"type":45,"tag":2190,"props":6157,"children":6158},{"style":6024},[6159],{"type":51,"value":6095},{"type":45,"tag":2190,"props":6161,"children":6162},{"style":6035},[6163],{"type":51,"value":4675},{"type":45,"tag":2190,"props":6165,"children":6166},{"style":6024},[6167],{"type":51,"value":6032},{"type":45,"tag":2190,"props":6169,"children":6170},{"style":6024},[6171],{"type":51,"value":3571},{"type":45,"tag":2190,"props":6173,"children":6174},{"style":6024},[6175],{"type":51,"value":6095},{"type":45,"tag":2190,"props":6177,"children":6178},{"style":6098},[6179],{"type":51,"value":6180},"notebook_cells",{"type":45,"tag":2190,"props":6182,"children":6183},{"style":6024},[6184],{"type":51,"value":6032},{"type":45,"tag":2190,"props":6186,"children":6187},{"style":6024},[6188],{"type":51,"value":6189},"}]\n",{"type":45,"tag":6191,"props":6192,"children":6193},"style",{},[6194],{"type":51,"value":6195},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":6197,"total":6296},[6198,6213,6225,6236,6252,6266,6282],{"slug":6199,"name":6199,"fn":6200,"description":6201,"org":6202,"tags":6203,"stars":28,"repoUrl":29,"updatedAt":6212},"agent-install","install Datadog Agent on Kubernetes","Install the Datadog Agent on Kubernetes using the Datadog Operator — required before enabling Single Step Instrumentation (SSI), which automatically instruments applications for APM without code changes. Only use if no Datadog Agent is deployed on the cluster yet.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6204,6205,6208,6211],{"name":20,"slug":21,"type":15},{"name":6206,"slug":6207,"type":15},"Deployment","deployment",{"name":6209,"slug":6210,"type":15},"Kubernetes","kubernetes",{"name":13,"slug":14,"type":15},"2026-04-15T04:57:27.489805",{"slug":6214,"name":6214,"fn":6215,"description":6216,"org":6217,"tags":6218,"stars":28,"repoUrl":29,"updatedAt":6224},"agent-observability-auto-experiment","run iterative code improvements using Datadog data","Run an iterative code-improvement hill-climb against real Datadog LLM-Obs data, locally, with Claude Code as the agent. Establishes a baseline eval, makes one focused change, re-scores with the same harness, keeps the change if it improves the score in the goal's direction (labeling within-noise gains tentative), and repeats. Use when the user says \"run an auto experiment\", \"hill-climb this code\", \"iteratively improve X and measure the delta\", \"optimize this prompt\u002Ffile against my traces\", \"auto-optimize against LLM-Obs\", or wants the local equivalent of the auto_experiments worker. Works from a local dataset file, an ml_app, a dataset_id, or a list of trace_ids.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6219,6220,6221,6222,6223],{"name":20,"slug":21,"type":15},{"name":26,"slug":27,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-07-31T05:52:13.711906",{"slug":6226,"name":6226,"fn":6227,"description":6228,"org":6229,"tags":6230,"stars":28,"repoUrl":29,"updatedAt":6235},"agent-observability-eval-bootstrap","bootstrap evaluators from production traces","Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you confirm, create them in Datadog as disabled drafts (never auto-enabled); on request emit Python SDK code or a framework-agnostic JSON spec instead. Use when user says \"bootstrap evaluators\", \"generate evaluators\", \"create evals from traces\", \"eval bootstrap\", \"write evaluators\", \"build eval suite\", \"publish evaluators\", or wants to generate BaseEvaluator\u002FLLMJudge code or online judge configs from production LLM trace data. Works with ml_app and optional RCA report or failure hypothesis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6231,6232,6233,6234],{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:19.259734",{"slug":6237,"name":6237,"fn":6238,"description":6239,"org":6240,"tags":6241,"stars":28,"repoUrl":29,"updatedAt":6251},"agent-observability-eval-pipeline","run agent observability and evaluation pipelines","End-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause failures, bootstrap evaluators, then (optionally) sample + publish a dataset, generate + run an experiment, and analyze results. Six narrated phases with a standardized banner and a \"continue\" checkpoint between each. Pure orchestration over the agent-observability sub-skills (`agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `agent-observability-experiment-py-bootstrap`, `agent-observability-experiment-analyzer`). Use when user says \"run the eval pipeline\", \"go from traces to evals\", \"bootstrap evals end to end\", \"classify then RCA then bootstrap\", \"build an eval set from scratch\", \"onboard me to datasets and experiments\", \"walk me through experiments\", \"I have an ml_app, now what\", \"Agent Observability onboarding\", \"guided experiment setup\", \"from traces to experiments\", or wants a deterministic, narrated tour from production data through evaluators, datasets, and experiments. Stop early with `--stop-after \u003Cphase>` to short-circuit at evaluators or dataset, or resume mid-flow with `--start-at \u003Cphase>`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6242,6245,6248,6249,6250],{"name":6243,"slug":6244,"type":15},"Agents","agents",{"name":6246,"slug":6247,"type":15},"Data Pipeline","data-pipeline",{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:17.231423",{"slug":6253,"name":6253,"fn":6254,"description":6255,"org":6256,"tags":6257,"stars":28,"repoUrl":29,"updatedAt":6265},"agent-observability-experiment-analyzer","analyze LLM experiment results","Analyze LLM experiment results. Handles single or comparative experiments, exploratory or Q&A modes. Use when user says \"analyze experiment\", \"compare experiments\", \"analyze against baseline\", or provides one or two experiment IDs for analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6258,6261,6262,6263,6264],{"name":6259,"slug":6260,"type":15},"Analytics","analytics",{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:21.212498",{"slug":6267,"name":6267,"fn":6268,"description":6269,"org":6270,"tags":6271,"stars":28,"repoUrl":29,"updatedAt":6281},"agent-observability-experiment-py-bootstrap","generate Python experiment clients for LLM observability","Generates a self-contained Python experiment client that uses the ddtrace.llmobs SDK. Emits either a runnable .py script or a Jupyter .ipynb notebook matching the canonical DataDog reference notebook style. Use when the user says \"generate Python experiment\", \"write an SDK experiment\", \"create a ddtrace experiment\", \"Python notebook experiment\", \"use the Agent Observability SDK\", or has `ddtrace` installed and wants idiomatic SDK code.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6272,6273,6276,6277,6278],{"name":20,"slug":21,"type":15},{"name":6274,"slug":6275,"type":15},"Jupyter","jupyter",{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":6279,"slug":6280,"type":15},"Python","python","2026-06-19T09:04:22.640384",{"slug":6283,"name":6283,"fn":6284,"description":6285,"org":6286,"tags":6287,"stars":28,"repoUrl":29,"updatedAt":6295},"agent-observability-replay-trace","iterate on LLM traces with local code","Use when a developer wants to iterate on ONE specific Agent Observability \u002F LLM Obs trace whose output they didn't like — re-running that trace against their LOCAL code, seeing a concise diff of the old vs new output, and looping (change code → replay → diff) until satisfied. Invoked as \u002Fagent-observability-replay-trace \u003Ctrace-id> [changes to test]. Signals: \"replay this trace\"; \"iterate on a trace\"; \"this trace's output is wrong, fix it and re-run\"; \"re-run trace \u003Cid> with \u003Cchange>\"; pasting a trace id from the Agent Observability UI with a description of what to fix. It fetches the trace via the datadog-llmo MCP or the pup CLI, edits code, re-runs the app to emit a NEW trace, and diffs the two — no local server, no browser. For agents traced with ddtrace \u002F LLM Obs (Python first-class), with JSON-serializable entry input. Do NOT use for: scored Experiments or the browser \"Replay\" button (that's agent-observability-replay-experiment), building an experiment from a dataset\u002FCSV, writing evaluators, root-causing failed traces, or RUM\u002FHTTP session replay.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6288,6289,6290,6291,6292],{"name":20,"slug":21,"type":15},{"name":26,"slug":27,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":6293,"slug":6294,"type":15},"Tracing","tracing","2026-07-31T05:52:08.594182",34,{"items":6298,"total":6420},[6299,6306,6314,6321,6329,6337,6345,6353,6365,6373,6385,6405],{"slug":6199,"name":6199,"fn":6200,"description":6201,"org":6300,"tags":6301,"stars":28,"repoUrl":29,"updatedAt":6212},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6302,6303,6304,6305],{"name":20,"slug":21,"type":15},{"name":6206,"slug":6207,"type":15},{"name":6209,"slug":6210,"type":15},{"name":13,"slug":14,"type":15},{"slug":6214,"name":6214,"fn":6215,"description":6216,"org":6307,"tags":6308,"stars":28,"repoUrl":29,"updatedAt":6224},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6309,6310,6311,6312,6313],{"name":20,"slug":21,"type":15},{"name":26,"slug":27,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":6226,"name":6226,"fn":6227,"description":6228,"org":6315,"tags":6316,"stars":28,"repoUrl":29,"updatedAt":6235},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6317,6318,6319,6320],{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":6237,"name":6237,"fn":6238,"description":6239,"org":6322,"tags":6323,"stars":28,"repoUrl":29,"updatedAt":6251},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6324,6325,6326,6327,6328],{"name":6243,"slug":6244,"type":15},{"name":6246,"slug":6247,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":13,"slug":14,"type":15},{"slug":6253,"name":6253,"fn":6254,"description":6255,"org":6330,"tags":6331,"stars":28,"repoUrl":29,"updatedAt":6265},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6332,6333,6334,6335,6336],{"name":6259,"slug":6260,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":6267,"name":6267,"fn":6268,"description":6269,"org":6338,"tags":6339,"stars":28,"repoUrl":29,"updatedAt":6281},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6340,6341,6342,6343,6344],{"name":20,"slug":21,"type":15},{"name":6274,"slug":6275,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":6279,"slug":6280,"type":15},{"slug":6283,"name":6283,"fn":6284,"description":6285,"org":6346,"tags":6347,"stars":28,"repoUrl":29,"updatedAt":6295},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6348,6349,6350,6351,6352],{"name":20,"slug":21,"type":15},{"name":26,"slug":27,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":6293,"slug":6294,"type":15},{"slug":6354,"name":6354,"fn":6355,"description":6356,"org":6357,"tags":6358,"stars":28,"repoUrl":29,"updatedAt":6364},"agent-observability-session-classify","evaluate user intent satisfaction in sessions","Classify whether user intent was satisfied in a Datadog Agent Observability trace or session. Three modes: (1) session_id — classify a single CMD+I assistant session with RUM; (2) trace_id — classify a single Agent Observability trace without RUM; (3) ml_app — sample and classify multiple sessions or traces from a given LLM app. Output is compact by default (verdict + one-sentence reason). Use when evaluating satisfaction, classifying sessions\u002Ftraces, labeling data, or generating signal for agent-observability-eval-pipeline or agent-observability-trace-rca.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6359,6360,6361,6362,6363],{"name":6259,"slug":6260,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:26.341497",{"slug":4,"name":4,"fn":5,"description":6,"org":6366,"tags":6367,"stars":28,"repoUrl":29,"updatedAt":30},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6368,6369,6370,6371,6372],{"name":20,"slug":21,"type":15},{"name":26,"slug":27,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":6374,"name":6374,"fn":6375,"description":6376,"org":6377,"tags":6378,"stars":28,"repoUrl":29,"updatedAt":6384},"agent-skills","use Datadog observability skills","Datadog skills for AI agents. Essential monitoring, logging, tracing and observability.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6379,6380,6383],{"name":20,"slug":21,"type":15},{"name":6381,"slug":6382,"type":15},"Monitoring","monitoring",{"name":13,"slug":14,"type":15},"2026-04-06T18:08:33.337476",{"slug":6386,"name":6386,"fn":6387,"description":6388,"org":6389,"tags":6390,"stars":28,"repoUrl":29,"updatedAt":6404},"datadog-app","build and manage Datadog applications","Guides developers building Datadog Apps with TypeScript, React, the @datadog\u002Fapps scaffolder, and @datadog\u002Fvite-plugin. Use when a user wants to scaffold, run, debug, upgrade, build, upload, publish, upload without publishing (draft upload), add an upload-no-publish script, set up CI\u002FCD, use OAuth or API\u002Fapplication key auth, trigger\u002Fpoll Workflow Automation, choose DDSQL or Action Catalog for backend data access, or query app datastores with DDSQL, including backend function troubleshooting.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6391,6392,6395,6398,6401],{"name":20,"slug":21,"type":15},{"name":6393,"slug":6394,"type":15},"Frontend","frontend",{"name":6396,"slug":6397,"type":15},"React","react",{"name":6399,"slug":6400,"type":15},"TypeScript","typescript",{"name":6402,"slug":6403,"type":15},"Vite","vite","2026-06-18T08:01:32.562331",{"slug":6406,"name":6406,"fn":6407,"description":6408,"org":6409,"tags":6410,"stars":28,"repoUrl":29,"updatedAt":6419},"dd-apm","query Datadog APM traces","APM - install, onboard, instrument, enable, set up, configure, traces, services, dependencies, performance analysis. Use for any request involving Datadog APM setup, instrumentation (SSI, ddtrace, agent install), or analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[6411,6412,6415,6416],{"name":20,"slug":21,"type":15},{"name":6413,"slug":6414,"type":15},"Distributed Tracing","distributed-tracing",{"name":13,"slug":14,"type":15},{"name":6417,"slug":6418,"type":15},"Performance","performance","2026-04-06T18:08:34.575282",35]