[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-datadog-labs-agent-observability-eval-bootstrap":3,"mdc--nj9u0q-key":36,"related-org-datadog-labs-agent-observability-eval-bootstrap":16283,"related-repo-datadog-labs-agent-observability-eval-bootstrap":16450},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":25,"repoUrl":26,"updatedAt":27,"license":28,"forks":29,"topics":30,"repo":31,"sourceUrl":34,"mdContent":35},"agent-observability-eval-bootstrap","bootstrap evaluators from production traces","Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you confirm, create them in Datadog as disabled drafts (never auto-enabled); on request emit Python SDK code or a framework-agnostic JSON spec instead. Use when user says \"bootstrap evaluators\", \"generate evaluators\", \"create evals from traces\", \"eval bootstrap\", \"write evaluators\", \"build eval suite\", \"publish evaluators\", or wants to generate BaseEvaluator\u002FLLMJudge code or online judge configs from production LLM trace data. Works with ml_app and optional RCA report or failure hypothesis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"datadog-labs","Datadog Labs","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdatadog-labs.png",[12,16,19,22],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"LLM","llm",{"name":20,"slug":21,"type":15},"Datadog","datadog",{"name":23,"slug":24,"type":15},"Evals","evals",145,"https:\u002F\u002Fgithub.com\u002Fdatadog-labs\u002Fagent-skills","2026-06-19T09:04:19.259734",null,19,[],{"repoUrl":26,"stars":25,"forks":29,"topics":32,"description":33},[],"Public repository for Datadog Agent Skills","https:\u002F\u002Fgithub.com\u002Fdatadog-labs\u002Fagent-skills\u002Ftree\u002FHEAD\u002Fagent-observability\u002Fagent-observability-eval-bootstrap","---\nname: agent-observability-eval-bootstrap\ndescription: Bootstrap evaluators from production traces — by default propose online LLM-judge evaluators and, after you confirm, create them in Datadog as disabled drafts (never auto-enabled); on request emit Python SDK code or a framework-agnostic JSON spec instead. Use when user says \"bootstrap evaluators\", \"generate evaluators\", \"create evals from traces\", \"eval bootstrap\", \"write evaluators\", \"build eval suite\", \"publish evaluators\", or wants to generate BaseEvaluator\u002FLLMJudge code or online judge configs from production LLM trace data. Works with ml_app and optional RCA report or failure hypothesis.\n---\n\n## Backend\n\n**Detection** — At the start of every invocation, before taking any action, determine which backend to use:\n\n1. If the user passed `--backend pup` anywhere in their invocation → use **pup mode** immediately, regardless of whether MCP tools are present. Skip steps 2–4.\n2. Check whether MCP tools are present in your active tool list. The canonical signal is whether `mcp__datadog-llmo-mcp__list_llmobs_evals` appears in your available tools.\n3. If MCP tools are present → use **MCP mode** throughout. Call MCP tools exactly as named in this skill's workflow sections.\n4. If MCP tools are absent → check whether `pup` is executable: run `pup --version` via Bash. A JSON response containing `\"version\"` confirms pup is available.\n5. If pup responds → use **pup mode** throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.\n6. If neither is available → stop and tell the user:\n   > \"Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (`claude mcp add --scope user --transport http datadog-llmo-mcp 'https:\u002F\u002Fmcp.datadoghq.com\u002Fapi\u002Funstable\u002Fmcp-server\u002Fmcp?toolsets=llmobs'`) or install pup.\"\n\n`--backend pup` is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.\n\n**pup invocation rules:**\n- Invoke via Bash: `pup llm-obs \u003Csubcommand> [flags]`\n- pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in `[{\"type\": \"text\", \"text\": \"\u003Cjson>\"}]`).\n- If pup returns an auth error, tell the user to run `pup auth login` and stop.\n- Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).\n- Time flags: pup accepts bare duration strings (`1h`, `7d`, `30m`) and RFC3339 timestamps. Do **not** use `now-`-prefixed strings — strip the prefix when converting from a skill `--timeframe` argument: `now-7d` → `7d`, `now-24h` → `24h`, `now-30d` → `30d`.\n- `--summary` on `pup llm-obs spans search` strips payload fields to essential metadata only. Use it in bulk\u002Fsearch phases where content is not needed.\n\n**Invocation ID:** At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., `3a9f1c2b`). Keep it constant for the entire invocation.\n\n**Intent tagging:** On every MCP tool call, prefix `telemetry.intent` with `skill:agent-observability-eval-bootstrap[\u003Cinv_id>] — ` followed by a description of why the tool is being called. On the **first MCP tool call only**, use `skill:agent-observability-eval-bootstrap:start[\u003Cinv_id>] — ` instead (note the `:start` suffix). Example first call: `skill:agent-observability-eval-bootstrap:start[3a9f1c2b] — Phase 0: map existing eval coverage for task-cruncher`\n\n# Eval Bootstrap — Generate Evaluators from Production Traces\n\nGiven a sample of production LLM traces, analyze input\u002Foutput patterns and quality dimensions, then propose a ready-to-use evaluator suite. Four output modes — **online evaluators are the default**; SDK code, the JSON spec, and the dataset-emit mode are produced on request:\n\n- **`publish`** *(default)* — propose **online** LLM-judge evaluators, then — **only after you confirm the suite** — write them to Datadog via `create_or_update_llmobs_evaluator` as **disabled drafts** (`enabled: false`). Nothing is created until you confirm at the Phase 2 checkpoint, and nothing scores any spans until **you** enable it in the UI — the skill never auto-publishes a live evaluator. Once you enable a draft, it runs automatically on matching production spans, traces, or sessions (no dataset, no task function). The skill **auto-classifies** each proposed evaluator as **span-scoped**, **trace-scoped**, or **session-scoped** based on what the judgment requires (a per-LLM-call tone check vs. an agent goal completion that needs the whole trace vs. user satisfaction across a whole multi-trace conversation) — you accept or override the classification at that checkpoint. Session-scoped evaluators are only proposed when the app's spans carry a `session_id` (verified by a probe in Phase 1).\n- **`sdk_code`** *(on request — `--sdk-code`, or ask after a publish run)* — Python `.py` file using the Datadog Evals SDK (`BaseEvaluator` \u002F `LLMJudge`) for **offline** experiments.\n- **`data_only`** *(on request — `--data-only`)* — self-contained JSON spec, framework-agnostic.\n- **`emit_dataset`** *(on request — `--emit-dataset \u003Cpath>`)* — sample production traces and write a `DatasetRecordRaw[]` JSON file shaped for `LLMObs.create_dataset(records=...)`. **Skips evaluator proposal and generation entirely** — this mode produces a dataset, not evaluators. Used by `agent-observability-eval-pipeline` (Phase 4) to seed an experiment dataset from production behavior.\n\nAfter a publish run, if the user wants the same suite as offline code or a portable spec, they just ask — the skill regenerates the **already-confirmed** suite in `sdk_code` \u002F `data_only` mode without re-exploring (see \"On-request code generation\" in Phase 3). The `emit_dataset` mode is independent of the evaluator workflow and never re-uses a prior proposal — it always re-samples traces.\n\n## Usage\n\n```\n\u002Feval-bootstrap \u003Cml_app> [--timeframe \u003Cwindow>] [--sdk-code | --data-only | --emit-dataset \u003Cpath>] [--trace-limit \u003CN>]\n```\n\nArguments: $ARGUMENTS\n\n### Inputs\n\n| Input | Required | Default | Description |\n|-------|----------|---------|-------------|\n| `ml_app` | Yes | — | ML application to scope traces |\n| `timeframe` | No | `now-7d` | How far back to look |\n| `rca_report` | No | — | Failure taxonomy from `eval-trace-rca` skill, or a free-text failure hypothesis |\n| `--sdk-code` | No | off | Emit a Python SDK `.py` file for offline experiments instead of publishing online. Mutually exclusive with `--data-only` and `--emit-dataset`. |\n| `--data-only` | No | off | Emit a self-contained JSON spec file instead of publishing online. Mutually exclusive with `--sdk-code` and `--emit-dataset`. |\n| `--emit-dataset \u003Cpath>` | No | off | **Dataset-only mode.** Sample production traces and write a `DatasetRecordRaw[]` JSON to `\u003Cpath>`. Skips the evaluator workflow entirely. Mutually exclusive with `--sdk-code` and `--data-only`. |\n| `--trace-limit` | No | `20` (cap `50`) | Max traces to sample in `emit_dataset` mode |\n\nIf `ml_app` is missing, ask the user before proceeding. With no mode flag, the skill defaults to **`publish`** — it proposes online evaluators and, only after you confirm, creates them as disabled drafts (it never auto-enables them). If more than one of `--sdk-code`, `--data-only`, `--emit-dataset` is supplied, error out and ask which mode the user wants.\n\n## Available Tools\n\n| Tool | Purpose |\n|------|---------|\n| `search_llmobs_spans` | Find spans by eval presence, tags, span kind, query syntax. Paginate with cursor. |\n| `get_llmobs_span_details` | Metadata, evaluations (scores, labels, reasoning), and `content_info` map showing available fields + sizes. |\n| `get_llmobs_span_content` | Actual content for a span field. Supports JSONPath via `path` param for targeted extraction. |\n| `get_llmobs_trace` | Full trace hierarchy as span tree with span counts by kind. |\n| `get_llmobs_agent_loop` | Chronological agent execution timeline (LLM calls, tool invocations, decisions). |\n| `list_llmobs_evals` | List every evaluator configured for the caller's org across all ml_apps, with `enabled` status and `ml_app` per result. Call once in Phase 0 to map existing coverage before proposing new evaluators — filter the result by `ml_app` client-side. |\n| `get_llmobs_evaluator` | Fetch the **full** persisted evaluator config by name (target ml_app + sampling + filter, provider, prompt template, parsing type, output schema, assessment criteria). Use in Phase 0 to understand what each existing custom eval measures, and (in publish mode) **before any update** — `create_or_update_llmobs_evaluator` is full-replace, so you must round-trip the full config to avoid clobbering fields. Not all evaluators have a stored config (notably `source=ootb`); a not-found error there is expected — skip those. |\n| `create_or_update_llmobs_evaluator` | *(publish mode)* Write an LLM-judge evaluator config to Datadog. Full-replace semantics: any omitted optional field resets to its default. See \"Publishing Conventions\" for required fields and structured output → JSON schema mapping. |\n| `delete_llmobs_evaluator` | *(publish mode)* Only used if the user explicitly asks to remove an evaluator. Never invoke speculatively. |\n\n### Key `get_llmobs_span_content` Patterns\n\nUse the `path` parameter to extract targeted data without fetching full payloads:\n\n| Field | Path | What you get |\n|-------|------|-------------|\n| `messages` | `$.messages[0]` | System prompt (first message, usually `system` role) |\n| `messages` | `$.messages[-1]` | Last assistant response |\n| `messages` | *(no path)* | Full conversation including tool calls |\n| `input` \u002F `output` | — | Span I\u002FO |\n| `documents` | — | Retrieved documents (RAG apps) |\n| `metadata` | — | Custom metadata (prompt versions, feature flags, user segments) |\n\n### How to Use `search_llmobs_spans`\n\nAdditional filters combine with space (AND): `@status:error @ml_app:my-app`. Dedicated params (`span_kind`, `root_spans_only`, `ml_app`) work alongside `query`, but `query` takes precedence over `tags`.\n\nTo find spans with a specific eval: `@evaluations.custom.\u003Ceval_name>:*` — you can only query for eval *presence*, not specific results.\n\nTo detect whether the app uses **sessions**: `session_id:*` matches any span carrying a `session_id` (`session_id` is a first-class field — no `@` prefix). The Phase 1 session probe uses this to gate session-scope evaluators.\n\n### Parallelization Rules\n\n1. **`get_llmobs_span_details`**: Group span_ids by trace_id. One call per trace_id with ALL its span_ids. Issue ALL calls for a page in a **single message**.\n2. **`get_llmobs_span_content`**: Each call is independent — always issue ALL in a single message.\n3. **`get_llmobs_trace` \u002F `get_llmobs_agent_loop`**: Parallelize across different traces in a single message.\n4. **Pipeline parallelism**: Start `get_llmobs_span_details` for page 1 results immediately — don't wait to collect all pages.\n\n---\n\n## Evaluator SDK Reference\n\n> **Applies to `sdk_code` mode only.** In `data_only` mode, use this section as domain context when writing rubric prompts — no SDK classes are emitted.\n\n### Imports\n\n```python\n# Core classes\nfrom ddtrace.llmobs._experiment import BaseEvaluator, EvaluatorContext, EvaluatorResult\n\n# LLM-as-judge\nfrom ddtrace.llmobs._evaluators.llm_judge import (\n    LLMJudge,\n    BooleanStructuredOutput,\n    ScoreStructuredOutput,\n    CategoricalStructuredOutput,\n)\n\n# Built-in evaluators (use only if needed)\nfrom ddtrace.llmobs._evaluators.format import JSONEvaluator, LengthEvaluator\nfrom ddtrace.llmobs._evaluators.string_matching import StringCheckEvaluator, RegexMatchEvaluator\n```\n\nOnly import what the generated file actually uses.\n\n### EvaluatorContext (what `evaluate()` receives)\n\n```python\n@dataclass(frozen=True)\nclass EvaluatorContext:\n    input_data: dict[str, Any]          # Task inputs (from dataset record, NOT from span)\n    output_data: Any                     # Task output (from task function return, NOT from span)\n    expected_output: Optional[JSONType] = None  # Ground truth (if available)\n    metadata: dict[str, Any] = {}        # Additional metadata\n    span_id: Optional[str] = None        # LLMObs span ID\n    trace_id: Optional[str] = None       # LLMObs trace ID\n```\n\n**Important — span data vs evaluator data**: When exploring production traces, you see span I\u002FO (e.g., `input.value`, `output.messages`). But evaluators run in offline experiments where `input_data` and `output_data` come from the user's **dataset records and task function**, not from spans. The dataset schema is user-defined and may not match span structure. Write evaluator prompts with generic `{{input_data}}` \u002F `{{output_data}}` placeholders and add comments describing what data the evaluator was designed for, so the user can adapt to their dataset shape.\n\n### EvaluatorResult (what `evaluate()` returns)\n\n```python\nEvaluatorResult(\n    value=...,                    # Required. JSONType (str, int, float, bool, None, list, dict)\n    reasoning=\"...\",              # Optional. Explanation string\n    assessment=\"pass\" or \"fail\",  # Optional. Pass\u002Ffail assessment\n    metadata={...},              # Optional. Evaluation metadata dict\n    tags={...},                  # Optional. Tags dict\n)\n```\n\n### LLMJudge — LLM-as-Judge Evaluator\n\n```python\njudge = LLMJudge(\n    user_prompt=\"...\",              # Required. Supports {{template_vars}}\n    system_prompt=\"...\",            # Optional. Does NOT support template vars\n    structured_output=...,          # Optional. Boolean\u002FScore\u002FCategorical output, or a dict for custom JSON schema\n    provider=\"openai\",              # \"openai\" | \"anthropic\" | \"azure_openai\" | \"vertexai\" | \"bedrock\"\n    model=\"gpt-4o\",                # Model identifier\n    model_params={\"temperature\": 0.0},  # Optional. Passed to LLM API\n    name=\"eval_name\",              # Optional. Must match ^[a-zA-Z0-9_-]+$\n)\n```\n\n**Template variables** in `user_prompt`: `{{input_data}}`, `{{output_data}}`, `{{expected_output}}`, `{{metadata.key}}` — resolved from `EvaluatorContext` fields via dot-path into nested dicts.\n\n### Structured Output Types\n\n**Boolean** — true\u002Ffalse with optional pass\u002Ffail:\n\n```python\nBooleanStructuredOutput(\n    description=\"Whether the response is factually accurate\",\n    reasoning=True,                    # Include reasoning field in LLM response\n    reasoning_description=None,        # Optional custom description for reasoning field\n    pass_when=True,                    # True → pass when true, False → pass when false, None → no assessment\n)\n```\n\n**Score** — numeric within a range with optional thresholds:\n\n```python\nScoreStructuredOutput(\n    description=\"Helpfulness score\",\n    min_score=1,                       # Minimum possible score\n    max_score=10,                      # Maximum possible score\n    reasoning=True,\n    reasoning_description=None,\n    min_threshold=7,                   # Scores >= 7 pass (optional)\n    max_threshold=None,                # Scores \u003C= N pass (optional)\n)\n```\n\n**Categorical** — select from predefined categories:\n\n```python\nCategoricalStructuredOutput(\n    categories={\n        \"correct\": \"The response correctly answers the question\",\n        \"partially_correct\": \"The response is partially correct but missing key information\",\n        \"incorrect\": \"The response is factually wrong or irrelevant\",\n    },\n    reasoning=True,\n    reasoning_description=None,\n    pass_values=[\"correct\"],           # Which categories count as passing (optional)\n)\n```\n\n**Custom JSON schema** — arbitrary structured responses for multi-dimensional evals:\n\n```python\n# Pass a raw dict as structured_output — used as the JSON schema directly\nstructured_output={\n    \"type\": \"object\",\n    \"properties\": {\n        \"relevance\": {\"type\": \"boolean\", \"description\": \"Whether the response addresses the question\"},\n        \"confidence\": {\"type\": \"number\", \"description\": \"Confidence score (0.0 to 1.0)\"},\n        \"reasoning\": {\"type\": \"string\", \"description\": \"Explanation for the evaluation\"},\n    },\n    \"required\": [\"relevance\", \"confidence\", \"reasoning\"],\n    \"additionalProperties\": False,\n}\n```\n\nAlways write standard JSON schema — the SDK adapts it per provider automatically (e.g., Anthropic doesn't support `minimum`\u002F`maximum` on number fields, so the SDK moves range constraints into the `description`; Vertex AI converts `const`\u002F`anyOf` to `enum`). The full parsed JSON dict becomes the eval `value`; a `\"reasoning\"` key (if present) is automatically extracted. No automatic pass\u002Ffail assessment.\n\n### LLMJudge Prompt Guidelines\n\nThe `structured_output` parameter enforces the response format via JSON schema. **Do not** prescribe the format in the prompt (no \"Answer YES\u002FNO\", \"Rate 1-10\", etc.). Instead, describe the **evaluation criteria** and let the structured output handle the format.\n\n- **system_prompt**: Set the judge's role and the app's domain context. Does NOT support template vars.\n- **user_prompt**: Present the data via `{{input_data}}` \u002F `{{output_data}}`, then describe what good vs. bad looks like for this dimension.\n\n### BaseEvaluator — Custom Code-Based Evaluator\n\nFor deterministic checks that do not need LLM judgment:\n\n```python\nclass MyEvaluator(BaseEvaluator):\n    def __init__(self, name=None, ...custom_params...):\n        super().__init__(name=name)\n        self._param = ...  # Store config as private attrs\n\n    def evaluate(self, context: EvaluatorContext) -> EvaluatorResult:\n        # Access: context.input_data, context.output_data, context.expected_output, context.metadata\n        # Must NOT modify self attributes (thread safety)\n        passed = ...  # Your logic here\n        return EvaluatorResult(\n            value=passed,\n            reasoning=\"...\",\n            assessment=\"pass\" if passed else \"fail\",\n        )\n```\n\n### Built-in Evaluators\n\n```python\n# Validate JSON syntax + optional required keys\nJSONEvaluator(required_keys=[\"name\", \"age\"], output_extractor=None, name=None)\n\n# Validate length (characters, words, or lines)\nLengthEvaluator(count_by=\"words\", min_length=10, max_length=500, output_extractor=None, name=None)\n# count_by: \"characters\" | \"words\" | \"lines\"\n\n# String matching\nStringCheckEvaluator(operation=\"contains\", expected=\"success\", case_sensitive=False, name=None)\n# operation: \"eq\" | \"ne\" | \"contains\" | \"icontains\"\n\n# Regex matching\nRegexMatchEvaluator(pattern=r\"\\d{4}-\\d{2}-\\d{2}\", match_mode=\"search\", name=None)\n# match_mode: \"search\" | \"match\" | \"fullmatch\"\n```\n\n### Evaluator Type Decision Matrix\n\n| Signal | Evaluator Type |\n|--------|---------------|\n| Output must be valid JSON | `JSONEvaluator` |\n| Output must match a regex pattern | `RegexMatchEvaluator` |\n| Output has length constraints | `LengthEvaluator` |\n| Output must contain\u002Fnot contain specific strings | `StringCheckEvaluator` |\n| Semantic quality judgment (tone, accuracy, completeness) | `LLMJudge` + `BooleanStructuredOutput` |\n| Graded quality on a scale | `LLMJudge` + `ScoreStructuredOutput` |\n| Classification into categories | `LLMJudge` + `CategoricalStructuredOutput` |\n| Multi-dimensional judgment (evaluate several aspects at once) | `LLMJudge` + custom JSON schema `dict` |\n| Complex domain logic combining multiple checks | `BaseEvaluator` subclass |\n\n### Source Verification\n\nIf you have access to dd-trace-py locally, verify the API surface by reading the corresponding modules:\n\n- `ddtrace.llmobs._evaluators.llm_judge` — `LLMJudge`, `BooleanStructuredOutput`, `ScoreStructuredOutput`, `CategoricalStructuredOutput`\n- `ddtrace.llmobs._experiment` — `BaseEvaluator`, `EvaluatorContext`, `EvaluatorResult`\n- `ddtrace.llmobs._evaluators.format` — `JSONEvaluator`, `LengthEvaluator`\n- `ddtrace.llmobs._evaluators.string_matching` — `StringCheckEvaluator`, `RegexMatchEvaluator`\n\n---\n\n## Workflow\n\n### Phase 0: Resolve Inputs & Entry Mode\n\n**Entry mode detection:**\n\n| Mode | Signal | Behavior |\n|------|--------|----------|\n| **Cold Start** | Only `ml_app` provided (no RCA, no hypothesis) | Full open discovery — understand what the app does, identify quality dimensions worth measuring, propose evals for coverage |\n| **From RCA** | Conversation contains an RCA report or user provides a failure hypothesis | Skip open discovery — use existing failure taxonomy as eval targets |\n\n**Parse arguments**: Extract `ml_app` (first non-flag argument), `--timeframe` (default `now-7d`), `--trace-limit` (default `20`), `--sdk-code`, `--data-only`, and `--emit-dataset \u003Cpath>` flags. Set `output_mode` as follows (at most one of the three mode flags may be set; error if more than one is present):\n\n- `--emit-dataset \u003Cpath>` set → `output_mode = emit_dataset`. Skip the rest of the workflow entry-mode logic and jump directly to **Phase 3D** below.\n- `--sdk-code` set → `output_mode = sdk_code`.\n- `--data-only` set → `output_mode = data_only`.\n- otherwise → `output_mode = publish` (the default — propose online evaluators, gated on user confirmation, created as disabled drafts).\n\n**Resolution steps:**\n\n1. If `ml_app` not provided → ask the user.\n2. Auto-detect entry mode:\n   - If the conversation contains an RCA report (look for \"Failure Taxonomy\" heading, structured failure modes, or severity ratings) → `from_rca`. Extract the taxonomy.\n   - If the user provides a free-text failure hypothesis (e.g., \"the system prompt lacks grounding\") → `from_rca`. Use the hypothesis as the starting eval target.\n   - Otherwise → `cold_start`.\n3. If `timeframe` not provided → default to `now-7d`.\n4. **Map existing eval coverage** — **skip if `output_mode = data_only`** (there is no Datadog eval project to check coverage against): Call `list_llmobs_evals` (org-wide; filter the result client-side to entries where `ml_app == \u003Cml_app>`). Then, for each eval with `source=custom`, call `get_llmobs_evaluator(eval_name=...)` to inspect its prompt template, target, sampling, and filter, and infer which quality dimension it covers. Issue all evaluator calls in a **single message** (parallelize). Skip `source=ootb` evals — their names are self-describing and they may not have a fetchable config.\n\n   By the end of this step you have a complete coverage map: `{eval_name → source, enabled, dimension}`. Carry this into Phase 2 for deduplication.\n\n   **In `publish` mode, also note any template-variable convention** the existing custom evaluators already use (so a new suite reads consistently). Online evaluator templates resolve against the **full span JSON**, not against `EvaluatorContext`. See the \"Online Template Variables\" section under \"Publishing Conventions\" for the supported syntax (`{{span_input}}`, `{{span_output}}`, dot-paths, array selectors, filter accessors).\n\n5. **Notebook context detection**: Scan the current conversation for a Datadog notebook URL that was produced by `\u002Feval-trace-rca` (pattern: `https:\u002F\u002Fapp.datadoghq.com\u002Fnotebook\u002F{numeric-id}`). If found, store it as `rca_notebook_url` and extract the numeric ID as `rca_notebook_id`. This is used after Phase 3 to offer appending the evaluator suite to that notebook instead of creating a new one.\n\n---\n\n### Phase 1: Explore Traces & Identify Eval Targets\n\n**Goal**: Sample production traces, understand what the app does, and identify quality dimensions worth measuring.\n\n#### Cold Start Path\n\n1. **Sample the app**: `search_llmobs_spans(query=\"@ml_app:\\\"\u003Cml_app>\\\" @status:ok\", root_spans_only=true, limit=50, from=\u003Ctimeframe>)`. Filter by `@status:ok` — error spans have no output to evaluate.\n\n   **Session probe** *(gates session-scope proposals; `publish` mode)*: in the same message, also call `search_llmobs_spans(query=\"@ml_app:\\\"\u003Cml_app>\\\" session_id:*\", limit=20, from=\u003Ctimeframe>)`.\n   - **≥ 1 result** → set `sessions_present = true`. Note the distinct `session_id` values and, critically, whether the same `session_id` appears across **multiple `trace_id`s** — that cross-trace span is the real signal that a session carries context worth a session-scope evaluator. (A `session_id` that only ever maps to one trace adds nothing over trace scope.)\n   - **0 results** → `sessions_present = false`. Do **not** propose any session-scope evaluator; record a one-line \"session scope skipped — no `session_id` on sampled spans\" note for the proposal.\n\n2. **Profile the app and identify evaluation target spans**: Call `get_llmobs_span_details` for span_ids grouped by trace_id. Inspect `content_info` to classify:\n\n   | Signal | App Profile |\n   |--------|------------|\n   | `content_info` has `messages` | LLM\u002Fchat app |\n   | `content_info` has `documents` | RAG app |\n   | Spans include `agent` kind | Agent app |\n   | `content_info` has `metadata` | Has custom metadata |\n   | Multiple span kinds in one trace (`agent` + `tool` \u002F `retrieval` + `llm` from `get_llmobs_trace`) | Multi-step app — at least one trace-scope evaluator likely belongs in the suite (`publish` mode) |\n   | Same `session_id` across **multiple `trace_id`s** (from the session probe) | Multi-trace sessions — at least one session-scope evaluator likely belongs in the suite (`publish` mode, gated on `sessions_present`) |\n\n   For agent\u002Fmulti-step apps, also call `get_llmobs_trace` on 2-3 traces to see the full span hierarchy. Compare `content_info` between the root span and its sub-spans. Then ask **two** questions for each candidate quality dimension, in this order:\n\n   1. **Does the verdict depend on more than one span?** (e.g., faithfulness depends on a `retrieval` span's documents AND an `llm` span's answer; goal completion depends on the chain of `tool` calls AND the final response.) If yes → **trace scope** in `publish` mode. Don't try to compress this into a single span.\n   2. **Only if the answer to (1) is no**: pick the single span with the richest signal for that dimension (root has the summary; LLM sub-spans have the full system prompt + tool call results + reasoning chain).\n\n   Record the span-kind histogram (agent + tool + llm + retrieval) — multiple kinds under one root is a strong signal you'll have at least one trace-scope evaluator in the suite. See Phase 2's \"Span vs. Trace vs. Session Scope Classification\" for the mandatory walk-through of canonical trace-scope use cases (and, when `sessions_present`, the canonical session-scope use cases).\n\n3. **Extract content and identify targets**: Call `get_llmobs_span_content` for representative spans. Fetch fields based on app profile:\n\n   | App Profile | Fields to Fetch |\n   |------------|----------------|\n   | LLM\u002Fchat | `messages` (`path=$.messages[0]` for system prompt), `output` |\n   | RAG | `documents`, `input`, `output` |\n   | Agent | `get_llmobs_agent_loop` for the agent span, then `messages` for detail |\n   | Any with metadata | `metadata` |\n\n   Issue all calls in a single message. As you read, capture two streams of signal:\n\n   **Generic quality signals** — what does \"success\" look like? What variance exists across outputs? Each observed quality dimension becomes a candidate evaluator, with the traces you've just read as evidence. Also look for safety signals (scope violations, sensitive data in outputs, out-of-character responses) and add a safety evaluator if you find them.\n\n   **Domain signals** — these become the *domain-specific evaluator* category in Phase 2 (the highest-leverage category). For every 5–10 traces, write down:\n   - **Recurring intents \u002F question categories** — what classes of request does this app handle? (`applying for benefit X`, `comparing flight options`, `summarizing a policy`, `creating a widget`)\n   - **Entities the app emits in outputs** — URLs, agency \u002F company names, code identifiers, monetary amounts, dates, IDs, file paths, phone numbers. Note which ones the user *acts* on downstream (those are worth a correctness evaluator) versus which are passing references.\n   - **Tool argument shapes** (for agent apps) — name each tool the agent calls and the rough schema of its inputs. Tools with non-trivial schemas (≥ 3 fields, structured types) are candidates for argument-correctness evaluators.\n   - **Persona \u002F voice rules** — does the app always cite a source, always refuse certain topics (medical, legal, financial advice), always speak in a particular tone? Extract the rules implicitly followed across observed outputs.\n   - **Failure modes specific to the domain** — fabricated identifiers, outdated policy references, currency \u002F locale mismatches, off-by-one errors in IDs, wrong units. One observed instance is enough to seed a candidate evaluator.\n\n   Don't try to enumerate domain signals exhaustively before reading traces — let the patterns surface as you read. The goal is breadth in the eventual proposal, not completeness in this exploration step.\n\n#### From RCA Path\n\n1. Extract the failure taxonomy from the RCA report. Each failure mode with High or Medium severity becomes an eval target. Also run the Phase 1 **session probe** (`query=\"session_id:*\"`) to set `sessions_present` — a failure that only manifests across a multi-trace conversation (lost context, repeated mistakes, mounting frustration) is a session-scope target.\n\n2. **Check root cause categories for infrastructure failures.** Before proposing evaluators, scan the Root Cause column of the taxonomy for any of: `Instrumentation Deficiency`, `Harness Deficiency`, `Runtime Error`, `Upstream Data Issue`, or any other root cause that points to infrastructure\u002Fenvironment rather than model behavior. If any are present, pause and ask:\n\n   > \"Some failure modes were diagnosed as infrastructure or instrumentation issues rather than model behavior (e.g., `{list the infra root causes}`). Evaluators can be designed two ways:\n   > - **Behavior-targeted** (recommended for ongoing quality): measure whether the model produces correct, specific output — useful once the infrastructure is fixed and you want to track real quality\n   > - **Artifact-targeted** (useful as regression guard): detect the specific broken output observed (e.g., generic placeholder responses) — catches regressions if the infrastructure breaks again\n   >\n   > Which approach do you want, or both?\"\n\n   - If **behavior-targeted**: design evaluators for what correct output looks like, not what the broken output looked like. Use the RCA's `expected_output` \u002F gold-standard examples as the quality bar.\n   - If **artifact-targeted**: design evaluators that detect the specific failure symptom (e.g., `StringCheckEvaluator` for a known bad string, `LLMJudge` that checks for generic placeholders).\n   - If **both**: propose each category separately, clearly labelled.\n\n   If all root causes are behavioral (System Prompt Deficiency, Tool Gap, Tool Misuse, Retrieval Failure, etc.) → skip this step and proceed directly.\n\n3. For each target: if the RCA includes trace IDs, use them directly; otherwise search for matching traces. Fetch 2-3 traces per target with `get_llmobs_span_content` to understand the concrete pattern.\n\n---\n\n### Phase 2: Propose Evaluator Suite\n\n**Goal**: Present a concrete evaluator proposal for user confirmation.\n\nIn `sdk_code` \u002F `data_only` mode — and for `eval_scope: span` in `publish` mode — each evaluator judges **one data point**: input and output for a single record\u002Fspan, not a full trace or batch. In `publish` mode, `eval_scope: trace` judges a whole trace and `eval_scope: session` a whole multi-trace session — design those against the trace \u002F session payload instead (see \"Span vs. Trace vs. Session Scope Classification\" below). Design evaluators accordingly for their scope.\n\n**Targeting depends on `output_mode`:**\n\n- `sdk_code` \u002F `data_only` → **offline experiments**. Template variables use `EvaluatorContext` fields (`{{input_data}}`, `{{output_data}}`). The actual data shape depends on the user's dataset and task function (see EvaluatorContext note in SDK Reference).\n- `publish` → **online evaluation on production spans**. Template variables resolve against the **full span JSON** via dot-paths (`{{meta.input.value}}`, `{{meta.output.messages[*].content}}`, …) or the built-in span-kind-aware aliases (`{{span_input}}`, `{{span_output}}`). For `eval_scope: trace` and `eval_scope: session`, templates resolve against the trace payload (`{{spans[...]}}`) or the session payload (`{{traces[*].spans[...]}}`) instead. See \"Online Template Variables\" under Publishing Conventions for the full syntax. Each evaluator also needs `eval_scope`, `sampling_percentage`, and (optionally) `filter` — surface these in the proposal table so the user can confirm before publishing. Session scope is only used when the Phase 1 probe set `sessions_present`.\n\nOrder proposals from broadest signal to most granular. **Propose broadly, let the user curate** — see \"How many evaluators to propose\" below.\n\n1. **Domain-specific evaluators** — What does \"good\" mean *for this specific app*? These are the highest-leverage proposals because they capture quality bars generic evaluators miss. Derive them from the **domain signals** Phase 1 captured:\n   - **Recurring intents \u002F question categories** the app handles (e.g., \"applying for a federal benefit\", \"comparing flight options\", \"explaining a policy\"). Propose an `intent_classification` or `intent_handling_correctness` evaluator scoped to the dominant intents.\n   - **Specific entities the app produces** (URLs, agency names, code identifiers, monetary amounts, dates, IDs). Propose a per-entity correctness evaluator for the ones with real downstream cost when wrong (e.g., `cited_url_is_real`, `agency_name_matches_request`, `monetary_amount_is_consistent_with_input`).\n   - **Tool argument shapes** observed across `tool` spans. Propose a per-tool argument-correctness evaluator for the tools with non-trivial schemas (e.g., `search_flights_args_match_user_request`, `update_dashboard_widget_targets_correct_widget`).\n   - **Persona \u002F voice expectations** — does the app always cite sources, always refuse out-of-scope requests, always speak in a specific tone? Propose evaluators for the voice rules you can extract from observed outputs (`cites_a_source`, `refuses_medical_advice`, `tone_matches_brand`).\n   - **Domain-specific failure modes** seen across traces (fabricated identifiers, outdated policy references, unit mismatches, currency \u002F locale mismatches). One evaluator per recurring failure mode.\n\n   Name each evaluator after the *user-facing concern*, not the technical check (`agency_url_is_real` over `regex_url_match`). Use the trace IDs you read in Phase 1 as evidence — at least one passing case and one failing case per evaluator if you saw both.\n\n2. **Outcome evaluators** — Did this span \u002F trace produce a good result for the request?\n   - Examples: `task_completion`, `answer_correctness`, `response_groundedness`\n3. **Format evaluators** — Does the output meet structural requirements?\n   - Examples: `valid_json_output`, `response_length`, `citation_format`\n4. **Safety evaluators** — Does the output stay within appropriate boundaries?\n   - Examples: `no_pii_leakage`, `scope_adherence`, `no_hallucination`\n\n##### How many evaluators to propose\n\nThe default `4-6` cap from the older skill version was too tight — it pushed the skill toward generic evaluators only and left domain signals on the table. Updated guidance:\n\n- **Aim for 8–15 evaluators** in the proposal, distributed across all four categories (with domain-specific usually the largest bucket, outcome second, format and safety smaller). For very simple single-LLM-call apps, fewer is fine; for agent \u002F RAG apps with rich domain signals, lean toward the upper end.\n- **Quality > generic**: every domain-specific proposal should be backed by at least one observed pattern in the sampled traces. Don't invent generic domain evaluators (\"`response_quality`\") if you don't have evidence for them.\n- **Let the user curate**: the MANDATORY CHECKPOINT below explicitly asks the user to **remove** what doesn't apply, not just to approve. Treat the proposal as a candidate set the user trims.\n\n#### Deduplication Against Existing Coverage\n\n**In `data_only` mode**: skip this section entirely (coverage map was not built in Phase 0). Proceed directly to the proposal table.\n\nBefore building the proposal, apply the coverage map from Phase 0. **Coverage is keyed on `(dimension, scope)` — not on dimension alone**: every OOTB evaluator runs at span scope, and an enabled OOTB eval does NOT preclude proposing a trace-scope **or session-scope** evaluator for the same dimension. The three scopes answer different questions.\n\n1. **Enabled span-scope eval (OOTB or custom)** for dimension D:\n   - Do NOT propose a new **span-scope** evaluator for D — that dimension is already covered at span scope.\n   - DO propose a **trace-scope** or **session-scope** evaluator for D when the trace or session shape calls for it (multi-step app, or multi-trace session — judgment depends on cross-span or cross-trace context). Note the relationship in the rationale: e.g., \"OOTB `Goal Completeness` evaluates each LLM span in isolation; this trace-scope `goal_completion` checks whether the agent's full sequence of steps achieved the user's request, and a session-scope `session_goal_completion` checks it across the whole conversation — three different questions.\"\n\n2. **Enabled trace-scope custom eval** for dimension D: do NOT propose another trace-scope evaluator for the same dimension; that's a real duplicate. Span-scope on the same dimension is still fair game if the data also fits a single span, and session-scope is fair game if the dimension also needs cross-trace context. Likewise, an enabled **session-scope** custom eval for D blocks only another session-scope eval for D — span and trace scope remain fair game.\n\n3. **Disabled OOTB eval**: Do NOT propose a new custom span-scope evaluator for that dimension. Instead, surface it in a short note within the proposal and suggest enabling it in the Datadog UI rather than creating a duplicate. Example:\n\n   > `hallucination` (ootb, disabled) — consider enabling in Datadog UI (Evaluations → Configure) instead of creating a custom span-scope eval. (A trace-scope `rag_faithfulness` is still in scope and covers a different question.)\n\n4. **Gap identification**: Open the proposal with a coverage summary line: \"Existing coverage: N evaluator(s) already configured ({names}, all span-scope unless noted). Proposing evaluators for uncovered dimensions and uncovered scopes.\"\n\n5. **All dimensions covered**: A dimension is \"fully covered\" only when the relevant scopes are present (span, plus trace and\u002For session where the app shape calls for them). If the coverage map accounts for every identified quality dimension at the appropriate scope(s), surface this explicitly and ask the user what they want: (a) review\u002Fimprove existing eval prompts, (b) add coverage for additional dimensions, or (c) proceed anyway.\n\nFor each proposed evaluator:\n\n- **Name**: Must match `^[a-zA-Z0-9_-]+$` (alphanumeric, underscore, hyphen only)\n- **Type**: `LLMJudge` (Boolean\u002FScore\u002FCategorical\u002Fcustom JSON schema), built-in (`JSONEvaluator`, `RegexMatchEvaluator`, etc.), or `BaseEvaluator` subclass. *In `publish` mode, only LLM-judge evaluators are supported by the MCP tool — code-based checks must NOT be silently dropped. List them in the same proposal table with `Type` set to the code-based class, mark them under a \"Not publishable in this mode\" subsection of the proposal, and tell the user they can get them as offline code on request (`--sdk-code`, or ask after the publish run) or as a `--data-only` spec. Treat the code-based proposals as part of the suite for counting and coverage purposes.*\n- **What it measures**: 1-2 sentence plain-language description\n- **Target span**: Which span's data the evaluator was designed for (e.g., \"root agent span\", \"LLM sub-span `anthropic.request`\", \"all `llm` spans\"). If the root span's I\u002FO is too lossy for the quality dimension (e.g., tool call results aren't visible), note this and specify which sub-span has the signal. *In `publish` mode this maps to a combination of `eval_scope` (`span`\u002F`trace`\u002F`session`), `root_spans_only`, and the EVP `filter` query (e.g. `@meta.span.kind:llm` or `service:web`).*\n- **Pass\u002Ffail criteria**: `pass_when=True`, `min_threshold=7`, `pass_values=[\"correct\"]`, or \"no automatic assessment\" for custom JSON schema\n- **Template variables**: Which of `input_data`, `output_data`, `expected_output`, `metadata.*` it uses (offline) — or which span paths \u002F aliases it pulls from (publish mode: `{{span_input}}`, `{{span_output}}`, `{{meta.input.messages[*].content}}`, `{{meta.metadata.\u003Ckey>}}`, etc.)\n- **Evidence**: At least one trace where it would have caught a failure (or confirmed correct behavior)\n- **Publish-only fields** *(only in `publish` mode)*: `integration_provider` (default `openai`), `model_name` (default `gpt-5.4-mini`), `sampling_percentage` (default `10`), `eval_scope` (default `span`), and any `filter` query needed to scope to the right spans. Surface defaults in the proposal so the user can override before publishing.\n- **`integration_account_id`** *(only in `publish` mode)*: the integration account the judge LLM is called through. Auto-detected from existing evaluators in the same ml_app (Phase 0 coverage map). Never asked from the user as a raw UUID. If no existing evaluator has one, the field is omitted and the user picks an account in the UI before activating. **All evaluators are published with `enabled: false` regardless** — see \"Always publish as draft\" in Phase 3C for the full activation workflow.\n\n#### Span vs. Trace vs. Session Scope Classification (`publish` mode)\n\n**Don't ask the user; classify per evaluator and let them override at the checkpoint.**\n\n##### Mandatory: walk the four canonical trace-scope use cases first\n\nIf Phase 1 found multi-step traces (≥ 2 span kinds, or any `tool` \u002F `retrieval` \u002F `workflow` span under an `agent` root), you **MUST** walk through the four canonical trace-scope use cases below before finalizing the suite. For each, decide explicitly: **applies** (include with `eval_scope: trace`) or **does not apply** (record a one-line reason in a \"Skipped trace-scope candidates\" subsection of the proposal). Skipping all four without per-item justification is a sign you've over-anchored on span scope — re-check.\n\n| Canonical use case | Triggers when |\n|---|---|\n| `goal_completion` — did the agent finish the user's request? | Any agent \u002F multi-step app. Almost always applies. |\n| `tool_use_correctness` — right tool with right arguments? | Trace contains `tool` kind spans. |\n| `rag_faithfulness` — answer grounded in retrieved documents? | Trace contains `retrieval` kind spans. |\n| `conversation_quality` — coherence across multi-turn LLM calls? | Trace contains ≥ 2 `llm` spans, or app instruments multi-turn sessions. |\n\n##### Mandatory: walk the canonical session-scope use cases (only when `sessions_present`)\n\n**Gate:** perform this walk-through **only if** the Phase 1 session probe set `sessions_present = true`. If sessions are absent, skip session scope entirely and note \"session scope skipped — no `session_id` on sampled spans\" in the proposal.\n\nWhen `sessions_present`, you **MUST** walk through the four canonical session-scope use cases below. For each, decide explicitly: **applies** (include with `eval_scope: session`) or **does not apply** (one-line reason in a \"Skipped session-scope candidates\" subsection). Session scope answers questions that span **more than one trace** under the same `session_id` — a single trace cannot see prior or later turns.\n\n| Canonical session use case | Triggers when |\n|---|---|\n| `session_goal_completion` — were the user's goals met across the whole session? | A `session_id` spans ≥ 2 traces. Almost always applies for multi-trace sessions. |\n| `multi_turn_conversation_quality` — coherence, memory, and consistent tone across turns | Multi-trace chat \u002F assistant sessions. |\n| `user_frustration_signals` — frustration, confusion, repetition, or abandonment over the session | Any multi-turn session (repeated or rephrased asks across traces). |\n| `agent_consistency_across_session` — did the agent stay consistent and recover from errors across traces? | Agent app whose sessions span ≥ 2 traces. |\n\nFor other proposed evaluators (e.g. tone, format, safety), apply this scope test in order:\n\n1. Can the judgment be answered correctly from **one** span's `meta.input` + `meta.output`, where \"correctly\" means the verdict cannot change if you considered other spans in the trace? → **`eval_scope: span`**.\n2. Otherwise, if it is answerable from the spans of a **single trace** → **`eval_scope: trace`**. Default to trace when the evaluator name contains *grounding*, *faithfulness*, *hallucination*, *completeness*, *correctness across steps*, *consistency*, or *workflow* — these almost always need cross-span context.\n3. Otherwise, if the verdict needs context from **more than one trace** in the same `session_id` (overall satisfaction, behavior over time, multi-turn coherence) **and** `sessions_present` → **`eval_scope: session`**. Default to session when the name contains *session*, *conversation*, *across turns*, *over time*, *satisfaction*, *frustration*, or *abandonment*. If `sessions_present` is false, fall back to trace scope and note the limitation.\n\n##### Trade-offs (don't let these dominate the choice)\n\nTrace scope costs more than span scope: one judgment per **completed** trace (vs. per matching span), larger prompt payloads, and a 3-minute trigger latency (Datadog waits 3 minutes of inactivity before considering a trace complete; later spans are excluded). Session scope costs the most: one judgment per **completed session** (a `session_id` is complete after **30 minutes of inactivity** — vs. 3 minutes for a trace — and spans arriving > 30 min after the prior span are excluded), with the largest payloads (every span of every trace in the session, capped at 10,000 spans). These are **cost-control** levers — handle with `sampling_percentage` and `filter`, not by demoting scope. The *correctness* of the eval is what picks the scope.\n\n##### Surface the classification\n\nAdd a **Scope** column to the proposal table and a one-sentence rationale per evaluator. If you skipped a canonical trace-scope **or session-scope** use case, list it under the matching \"Skipped …-scope candidates\" subsection with the reason — the user will see and can override.\n\n> Example rationales:\n> - `tone_check` — **span**. Judging \"is this single response polite\" needs only one LLM span's `meta.output.messages[*].content`; no other span in the trace can change that verdict.\n> - `goal_completion` — **trace**. Whether the agent finished the user's request depends on the sequence of tool calls and the final LLM response together — `meta.output` of any single span only shows that step's output.\n> - `tool_use_correctness` — **trace**. Comparing tool inputs against the request and the final response requires correlating ≥ 3 spans (root, tool, final LLM).\n> - `rag_faithfulness` — **trace**. Grounding pairs the `retrieval` span's documents with the LLM span's answer.\n> - `session_goal_completion` — **session**. Whether the user's overall goals were met depends on every trace in the `session_id`, not just the last one — only session scope sees the full conversation.\n> - `user_frustration_signals` — **session**. Frustration surfaces as repeated or rephrased asks across traces; a single trace can't reveal the pattern.\n>\n> Example \"Skipped trace-scope candidates\" entry:\n> - `conversation_quality` — skipped: traces contain a single LLM call (no multi-turn signal in this app's instrumentation).\n>\n> Example \"Skipped session-scope candidates\" entry:\n> - `session_goal_completion` — skipped: every `session_id` maps to a single trace (no cross-trace context — trace scope already covers it).\n\n#### MANDATORY CHECKPOINT\n\n**You MUST output the proposal and wait for user confirmation before proceeding.**\n\n```\n## Proposed Evaluator Suite\n\n**App profile**: {LLM | RAG | Agent | Multi-agent}\n**Entry mode**: {cold_start | from_rca}\n\n| # | Name | Type | Scope | Measures | Pass Criteria |\n|---|------|------|-------|----------|---------------|\n| 1 | task_completion | LLMJudge (Boolean) | span | Whether the task was completed on this span | pass_when=True |\n| 2 | tool_use_correctness | LLMJudge (Categorical) | trace | Right tool with right arguments across the agent run | pass_values=[\"correct\"] |\n| 3 | session_goal_completion | LLMJudge (Categorical) | session | Whether the user's goals were met across the whole multi-trace session | pass_values=[\"completed\"] |\n| 4 | ... | ... | ... | ... | ... |\n\n(Drop the **Scope** column when not in `publish` mode.)\n\nFor each evaluator:\n- **{name}**: {what it measures}\n  - Target span: {which span's data it was designed for}\n  - Rationale: {which quality dimension it covers and why}\n  - {Only in publish mode:} Scope: {span | trace | session} — {one-sentence rationale}\n  - Evidence: [Trace {id_short}](https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_id})\n\n{Only in publish mode, for multi-step apps. Required if any of the four canonical trace-scope use cases was not included above:}\n\n**Skipped trace-scope candidates:**\n- `{canonical_use_case}` — {one-line reason it does not apply to this app}\n\n{Only in publish mode, when `sessions_present`. Required if any of the four canonical session-scope use cases was not included above:}\n\n**Skipped session-scope candidates:**\n- `{canonical_use_case}` — {one-line reason it does not apply, e.g. \"every `session_id` maps to a single trace\"}\n\n{Only in publish mode, when the session probe found no sessions:}\n\n**Session scope skipped** — no `session_id` on sampled spans; session-scope evaluators not proposed.\n\n{Only in publish mode, when the suite contains code-based evaluators (JSONEvaluator, RegexMatchEvaluator, LengthEvaluator, StringCheckEvaluator, BaseEvaluator). Required when any code-based proposal exists.}\n\n**Not publishable in this mode** (code-based evaluators — the publish API is LLM-judge only):\n- `{name}` ({type}) — {what it would check}. Ask me to emit these as offline SDK code (or run `\u002Feval-bootstrap {ml_app} --sdk-code`), or `\u002Feval-bootstrap {ml_app} --data-only` for a framework-agnostic JSON spec.\n```\n\n**Which evaluators should I generate?** Treat the proposal as a candidate set — the suite below is intentionally broad so you can pick what matters for your team's quality bar. Reply with **which to keep, which to drop, and which to rename**; not every domain-specific proposal will fit your priorities. In `sdk_code` mode you may also add custom evaluators or change provider\u002Fmodel. In `publish` mode you may override `integration_provider`, `model_name`, `sampling_percentage`, `eval_scope`, `root_spans_only`, or `filter` per evaluator. (In the default `publish` mode these are created as **online drafts** in Datadog on confirmation — you review and enable them in the UI. Prefer offline SDK code or a JSON spec instead? Say so and I'll generate the confirmed suite that way.)\n\nDo NOT proceed to code generation until the user confirms.\n\n---\n\n### Phase 3: Generate Output\n\nBranch on `output_mode`:\n\n- `publish` *(default)* → skip to **Phase 3C**\n- `sdk_code` → **Phase 3A** below\n- `data_only` → skip to **Phase 3B**\n- `emit_dataset` → skip to **Phase 3D** (Phases 0 step 4, 1, and 2 are bypassed — see Phase 3D for the dataset-mode workflow)\n\n#### On-request code generation (after a publish run)\n\nThe default path publishes **online** evaluators. If the user then asks for the suite as offline code or a portable spec (e.g. \"now generate the SDK code for these\", \"give me a JSON spec\"), **do not re-run Phase 1–2**. Reuse the **already-confirmed** evaluator suite and jump straight to **Phase 3A** (`sdk_code`) or **Phase 3B** (`data_only`), translating each published online evaluator into the offline form:\n- The online prompt template (span\u002Ftrace\u002Fsession placeholders) becomes an offline LLMJudge with generic `{{input_data}}` \u002F `{{output_data}}` placeholders (offline data comes from the user's dataset\u002Ftask function, not spans — see the EvaluatorContext note), preserving the rubric and pass criteria.\n- Code-based checks that couldn't be published online (the \"Not publishable in this mode\" set) are emitted as real `BaseEvaluator` \u002F built-in evaluators here.\n\nThis works the other way too: a user who started with `--sdk-code` can ask to publish the confirmed suite online (Phase 3C). The `emit_dataset` path is separate from the evaluator workflow — it never has a \"confirmed suite\" to translate.\n\n---\n\n### Phase 3A: Generate & Write Evaluator Code\n\n**Goal**: Generate the final `.py` file and write it to disk.\n\nFor each confirmed evaluator, generate production-quality Python code following the SDK Reference patterns above.\n\n#### Code Generation Rules\n\n1. **Ground prompts in traces**: LLMJudge system prompts and user prompts must reference patterns actually observed in production traces. Never write generic prompts like \"evaluate whether the response is good\" — ground them in the app's domain, observed failure patterns, and success criteria.\n\n2. **Keep template variables generic, add comments for context**: Use `{{input_data}}` and `{{output_data}}` as top-level placeholders in prompts — do NOT reference nested span paths like `{{input_data.messages[-1].content}}`. The evaluator's data comes from the user's dataset and task function, not directly from spans. Instead, add a comment above each evaluator describing what data it was designed for and what the user should adapt:\n\n   ```python\n   # Designed for: input_data = user query, output_data = assistant response text\n   # Observed from: root agent span (input.value → output.value)\n   # If your dataset uses a different structure, adapt the prompt references below.\n   ```\n\n3. **Use the narrowest evaluator type**: If a check can be done with `JSONEvaluator`, `RegexMatchEvaluator`, `StringCheckEvaluator`, or `LengthEvaluator`, do NOT use an LLMJudge. Code-based evaluators are faster, cheaper, and deterministic.\n\n4. **BaseEvaluator subclasses**:\n   - Call `super().__init__(name=name)` in `__init__`\n   - Return `EvaluatorResult` from `evaluate()`\n   - Do NOT modify instance attributes in `evaluate()` (thread safety)\n\n5. **Names**: Must match `^[a-zA-Z0-9_-]+$`. Use snake_case descriptive names.\n\n6. **Imports**: Consolidate at the top of the file. Only import classes that are actually used.\n\n7. **Evaluator list**: Collect all evaluators into an `evaluators` list at the bottom of the file.\n\n8. **Anonymize PII**: Strip emails, names, and sensitive data from any trace content included in LLMJudge prompts or the header comment.\n\n#### Output Format\n\nThe generated `.py` file should follow this structure:\n\n```python\n\"\"\"\nAuto-generated evaluators for {ml_app}\nGenerated: {YYYY-MM-DD} by eval-bootstrap\n\nApp profile: {LLM | RAG | Agent | Multi-agent}\n\nQuality dimensions covered:\n  - {target_name}: {description}\n    Evidence: https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_id}\n  ...\n\nUsage:\n    from ddtrace.llmobs import LLMObs\n\n    experiment = LLMObs.experiment(\n        name=\"my-experiment\",\n        task=my_task_fn,\n        dataset=dataset,\n        evaluators=evaluators,\n    )\n    experiment.run()\n\"\"\"\n\n{imports — only what is used}\n\n\n# --- Outcome Evaluators ---\n\n{evaluator code}\n\n\n# --- Format Evaluators ---\n\n{evaluator code}\n\n\n# --- Safety Evaluators ---\n\n{evaluator code}\n\n\n# --- Evaluator Suite ---\n\nevaluators = [\n    {eval_1_variable_name},\n    {eval_2_variable_name},\n    ...\n]\n```\n\nOnly include section comments (Outcome\u002FFormat\u002FSafety) for categories that have evaluators.\n\n#### Write the file\n\nWrite the generated code to the output path (suggest `.\u002Fevals\u002F{ml_app}_evaluators.py` if not specified), then display a summary:\n\n```\n## Generated Evaluators\n\nWrote {N} evaluators to `{output_path}`:\n\n| # | Name | Type | Covers |\n|---|------|------|--------|\n| 1 | ... | ... | ... |\n\n### Next Steps\n\n1. **Review**: Check the generated prompts and criteria match your expectations\n2. **Test offline**: Use `LLMObs.experiment(evaluators=evaluators)` to batch-evaluate against a labeled dataset and verify scores\n```\n\n#### Notebook export (after summary)\n\nAfter displaying the summary, offer notebook export.\n\n- **If `rca_notebook_url` was detected in Phase 0**:\n  > An RCA notebook was created earlier in this session: `{rca_notebook_url}`\n  > Would you like to (a) append the evaluator suite summary to that notebook, or (b) create a new standalone notebook?\n\n  If **append**: use the notebook creation fallback pattern (see below) with `mcp__datadog-mcp__edit_datadog_notebook` (`id={rca_notebook_id}`, `append_only=true`, evaluator suite summary cell).\n\n  If **new**: use the notebook creation fallback pattern (see below) with `mcp__datadog-mcp__create_datadog_notebook`.\n\n- **If no `rca_notebook_url`**:\n  > Would you like to export this evaluator suite summary to a Datadog notebook?\n\n  If yes: use the notebook creation fallback pattern (see below) with `mcp__datadog-mcp__create_datadog_notebook`:\n  - **`name`**: `Eval Bootstrap: {ml_app} — YYYY-MM-DD`\n  - **`type`**: `report`\n  - **`cells`**: single markdown cell with the evaluator suite summary\n  - **`time`**: `{ \"live_span\": \"1h\" }`\n\n**Notebook creation fallback pattern** (apply to every `create_datadog_notebook` \u002F `edit_datadog_notebook` call):\n\n1. Try the MCP tool first.\n2. **If the MCP call fails**, inspect the error:\n   - **Auth \u002F permission error (401, 403)** → stop and tell the user.\n   - **Field validation error** (error names a specific field) → fix that field and retry the MCP call once.\n   - **Any other error** (binding, serialization, unexpected response) → fall back to pup:\n     - Write the payload to `\u002Ftmp\u002Fnb_bootstrap_{ml_app}.json` as a full API envelope: `{\"data\": {\"attributes\": {\"name\": \"...\", \"time\": {...}, \"cells\": [...]}, \"type\": \"notebooks\"}}`\n     - Run `pup notebooks create --file \u002Ftmp\u002Fnb_bootstrap_{ml_app}.json`\n     - If pup is not available either, render the notebook content as markdown in chat.\n3. After successful creation by either method, output the URL:\n   `Evaluator suite exported to notebook: \u003Curl>`\n\n**Notebook cell content** — the markdown cell should contain:\n\n```markdown\n## Eval Bootstrap: {ml_app}\n\n**Generated**: YYYY-MM-DD | **App profile**: {LLM | RAG | Agent | Multi-agent} | **Entry mode**: {cold_start | from_rca}\n**Generated code**: `{output_path}`\n\n{One sentence: what does this app do?}\n\n**Coverage**: {N} new evaluators ({comma-separated dimension names}) | {N} existing (unchanged: {names}) | {gaps if any: dimensions identified but not covered, and why}\n\n### Evaluator Suite\n\n| # | Name | Type | Measures | Pass Criteria |\n|---|------|------|----------|---------------|\n| 1 | ... | ... | ... | ... |\n\n### Evidence\n\n{For each evaluator: name — 1-line description — [Trace link]}\n\n### Next Steps\n\n1. Review generated prompts in `{output_path}`\n2. Run against a labeled dataset to validate scores\n3. Deploy to Datadog LLM Experiments\n```\n\n---\n\n### Phase 3B: Generate & Write Eval Spec JSON\n\n**Goal**: Serialize the confirmed evaluator suite and representative trace samples to a single self-contained JSON file — zero SDK dependencies.\n\n**Output path**: `.\u002Fevals\u002F{ml_app}_eval_spec.json`\n\n#### JSON Schema\n\n```json\n{\n  \"schema_version\": \"1\",\n  \"generated_at\": \"\u003CISO 8601 UTC>\",\n  \"generated_by\": \"eval-bootstrap\",\n  \"app\": {\n    \"ml_app\": \"\u003Cstring>\",\n    \"app_type\": \"LLM | RAG | Agent | Multi-agent\",\n    \"trace_window\": \"\u003Ctimeframe param, e.g. now-7d>\",\n    \"trace_count\": \"\u003Cinteger>\"\n  },\n  \"evaluators\": [\n    {\n      \"name\": \"snake_case_name\",\n      \"category\": \"outcome | format | safety\",\n      \"type\": \"llm_judge | code_check\",\n      \"description\": \"\u003C1-2 sentence plain-language description>\",\n      \"target_span\": \"\u003Cwhich span: root, llm sub-span, etc.>\",\n      \"scoring\": {\n        \"scale\": \"boolean | score_1_10 | categorical\",\n        \"categories\": [\"\u003Conly present when scale=categorical>\"],\n        \"pass_criteria\": \"\u003Chuman-readable: true, >= 7, in [correct], etc.>\"\n      },\n      \"rubric\": \"\u003Cfull prompt text for llm_judge; null for code_check>\",\n      \"implementation_hints\": {\n        \"type_if_code_check\": \"json_valid | regex | contains | length_words | null\",\n        \"pattern_if_code_check\": \"\u003Cpattern string or null>\",\n        \"notes\": \"\u003Coptional framework-agnostic implementation guidance>\"\n      },\n      \"evidence\": [\n        {\n          \"trace_id\": \"\u003C32-char hex>\",\n          \"span_id\": \"\u003C16-char hex>\",\n          \"url\": \"https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:\u003Ctrace_id>\",\n          \"observation\": \"\u003Cwhy this trace illustrates the evaluator>\"\n        }\n      ]\n    }\n  ],\n  \"sample_records\": [\n    {\n      \"trace_id\": \"\u003Cstring>\",\n      \"span_id\": \"\u003Cstring>\",\n      \"input\": {},\n      \"output\": \"\u003Cstring>\",\n      \"suggested_labels\": {\n        \"\u003Cevaluator_name>\": \"pass | fail | \u003Cscore>\"\n      }\n    }\n  ]\n}\n```\n\n#### Field Notes\n\n- **`evaluators[].type`**: `\"llm_judge\"` for semantic evaluators; `\"code_check\"` for deterministic checks (regex, length, JSON validity, etc.).\n- **`evaluators[].rubric`**: For `llm_judge` — full prompt text grounded in observed trace patterns. Use `{{input}}` and `{{output}}` as generic placeholders (not `{{input_data}}` — that's ddeval-specific). For `code_check` — null.\n- **`evaluators[].implementation_hints.notes`**: Optional framework-agnostic guidance, e.g. \"For OpenAI Evals, use `rubric` as a model-graded criterion. For Braintrust, use as an LLM scorer. For Promptfoo, use as an `llm-rubric` assertion.\"\n- **`sample_records`**: 10–20 representative traces from Phase 1. `suggested_labels` are Claude's best-read from trace inspection — not ground truth. The field name communicates this explicitly.\n- **PII rule**: Strip emails, names, and sensitive data from all `input`, `output`, and `evidence[].observation` fields before writing (same as Phase 3A).\n\n#### Writing Instructions\n\n1. Assemble the JSON object in memory following the schema above.\n2. Populate `sample_records` from traces already fetched in Phase 1. Fetch additional traces (up to 20 total) if fewer than 10 were read.\n3. Anonymize PII in all `input`, `output`, and `evidence[].observation` fields.\n4. Write the file with 2-space indentation using the Write tool.\n5. Display a completion summary:\n\n```\n## Generated Eval Spec\n\nWrote `.\u002Fevals\u002F{ml_app}_eval_spec.json`:\n\n- **{N} evaluators** ({outcome_count} outcome, {format_count} format, {safety_count} safety)\n- **{M} sample records** with suggested labels\n\n| # | Name | Category | Type | Pass Criteria |\n|---|------|----------|------|---------------|\n| 1 | ... | ... | ... | ... |\n\n### Next Steps\n\n1. **Review**: Open `.\u002Fevals\u002F{ml_app}_eval_spec.json` and verify the rubrics match your expectations\n2. **Implement**: Use the `rubric` field to configure evaluators in your framework of choice:\n   - OpenAI Evals: use `rubric` as a model-graded criterion\n   - Braintrust: create an LLM scorer with the rubric text\n   - Promptfoo: use as an `llm-rubric` assertion\n   - Custom code: call your LLM API with the rubric and parse the structured output\n3. **Label**: `suggested_labels` are Claude's best guesses from trace inspection — verify against ground truth before using as training data\n```\n\n#### Notebook export (after summary)\n\nSame logic as Phase 3A — offer to append to the RCA notebook if `rca_notebook_url` was detected, or create a new standalone notebook. Use the same notebook cell format as Phase 3A, substituting `output_path` with the JSON spec file path. In pup mode, use `pup notebooks create` \u002F `pup notebooks edit` as described in Phase 3A.\n\n---\n\n### Phase 3C: Publish Online Evaluators to Datadog (as disabled drafts)\n\n**Reached only after the user confirms the suite at the Phase 2 checkpoint — nothing below is written to Datadog before that.**\n\n**Goal**: For each confirmed evaluator, write an LLM-judge configuration to Datadog via `create_or_update_llmobs_evaluator` as a **disabled draft** (`enabled: false`). It scores **no** spans until the user reviews and enables it in the UI; once enabled, it runs automatically on matching production spans.\n\n#### Pre-publish checks (single message — parallelize)\n\nFor every proposed `eval_name`, call `get_llmobs_evaluator(eval_name=...)`:\n\n- **Not found** → safe to create.\n- **Found** → existing evaluator with the same name. Surface a diff to the user (existing dimension\u002Fprompt vs. proposed) and ask:\n  > Evaluator `{name}` already exists. Overwrite, rename, or skip?\n\n  If **overwrite**: keep the fetched config as the base and **merge** your generated fields on top, then send the **complete** object back. The MCP tool is full-replace — any field you omit (e.g. `temperature`, `max_tokens`, `filter`, `sampling_percentage`) reverts to its default. Never re-publish without round-tripping the existing config.\n\n  If **rename**: append a suffix (e.g. `_v2`) and treat as new.\n\n  If **skip**: drop from the publish set.\n\n#### Publishing Conventions\n\n**Required parameters** for each `create_or_update_llmobs_evaluator` call: `eval_name`, `application_name` (= `ml_app`), `enabled`, `integration_provider`, `model_name`, `prompt_template`, `parsing_type`, `output_schema`, plus a `telemetry.intent` string.\n\n**Defaults** to use unless the user overrides:\n\n| Field | Default |\n|-------|---------|\n| `enabled` | `false` (always — see \"Always publish as draft\") |\n| `integration_provider` | `openai` |\n| `model_name` | `gpt-5.4-mini` |\n| `temperature` | `0` |\n| `parsing_type` | `structured_output` |\n| `sampling_percentage` | `10` for span scope, `5` for trace scope, `5` for session scope (the heaviest — consider lowering) |\n| `eval_scope` | `span` (auto-promoted to `trace` or `session` per the classification rule in Phase 2) |\n\n**Prompt template**: convert the LLMJudge prompt into the MCP shape — an ordered array of `{role, content}` messages. The system prompt becomes `{role: \"system\"}`, the user prompt becomes `{role: \"user\"}`. Use **span-data placeholders** (see below) — **not** the offline `{{input_data}}` \u002F `{{output_data}}` form, which only exists in `EvaluatorContext`.\n\n##### Online Template Variables\n\nOnline evaluator prompts run through the dd-source `template` library (`domains\u002Fml-observability\u002Fshared\u002Flibs\u002Ftemplate`). Missing paths → empty string. **The data shape templates resolve against depends on `eval_scope`:**\n\n- **`eval_scope: span`** *(default)* — placeholders resolve against a **single span's JSON** (the `llmobs.Span` JSON-marshaled to a map). Use the span aliases \u002F dot-paths below directly.\n- **`eval_scope: trace`** — placeholders resolve against the **trace payload** `{ spans: [...] }`. Use `{{spans[N]...}}`, `{{spans[*]...}}`, or `{{spans[field.path:value]...}}` to select span(s) before applying field paths. The `{{span_input}}` \u002F `{{span_output}}` aliases are **not available** in trace scope — reference span data through the `spans` array instead.\n- **`eval_scope: session`** — placeholders resolve against the **session payload** `{ session_id, traces: [ { trace_id, root_span_id, spans: [...] }, … ] }`. Reach spans through `{{traces[*].spans[*]...}}` (nested one level deeper than trace scope) plus the array selectors below. The `{{span_input}}` \u002F `{{span_output}}` aliases **and** trace scope's top-level `{{spans[...]}}` are **not available** — spans live under `traces[*].spans`. Only used when `sessions_present` (Phase 1 probe).\n\n###### Span-scope (`eval_scope: span`)\n\n**Built-in span-kind-aware aliases** (preferred when the evaluator is generic across span kinds):\n\n| Alias | LLM span (`meta.span.kind = \"llm\"`) | Other spans (agent, workflow, task, …) |\n|-------|------------------------------------|----------------------------------------|\n| `{{span_input}}`  | `meta.input.messages[*].content`  | `meta.input.value`  |\n| `{{span_output}}` | `meta.output.messages[*].content` | `meta.output.value` |\n\n**Common explicit dot-paths** (use when the evaluator is purpose-built for one span kind):\n\n| Path | What you get |\n|------|--------------|\n| `{{meta.input.value}}` \u002F `{{meta.output.value}}` | Plain string I\u002FO on agent \u002F workflow \u002F task \u002F tool spans |\n| `{{meta.input.messages[*].content}}` | All input message contents on an LLM span (newline-joined) |\n| `{{meta.input.messages[0].content}}` | First message (typically system prompt) |\n| `{{meta.output.messages[*].content}}` | Assistant response(s) |\n| `{{meta.input.documents}}` | Retrieved docs (RAG) — JSON-serialized |\n| `{{meta.metadata.\u003Ckey>}}` | Custom metadata fields |\n| `{{meta.tool_definitions}}` | Available tools — JSON array |\n| `{{*}}` | Entire span as compact JSON (debug \u002F fall-back catch-all) |\n\n###### Trace-scope (`eval_scope: trace`)\n\n| Pattern | What you get |\n|---------|--------------|\n| `{{spans}}` | JSON of every span in the trace |\n| `{{spans[N].meta.input.value}}` | Single span by index — `spans[0]` is the trace root |\n| `{{spans[*].name}}` | All span names in order, newline-joined |\n| `{{spans[*].meta.output.value}}` | All spans' outputs, newline-joined (handy for \"final answer = last output\") |\n| `{{spans[name:my-span].meta.input.value}}` | Filter by span name |\n| `{{spans[meta.span.kind:llm].meta.output.value}}` | All LLM-kind span outputs |\n| `{{spans[meta.span.kind:tool]}}` | Whole tool spans as JSON, paired in\u002Fout — useful for tool-use correctness |\n| `{{spans[meta.span.kind:retrieval].meta.output.documents[*].text}}` | Text of every retrieved document — useful for RAG faithfulness |\n| `{{*}}` | Entire trace payload as JSON (debug fallback) |\n\n###### Session-scope (`eval_scope: session`)\n\n| Pattern | What you get |\n|---------|--------------|\n| `{{traces}}` | JSON of every trace in the session, each `{ trace_id, root_span_id, spans: [...] }` |\n| `{{session_id}}` | The session's ID |\n| `{{traces[*].trace_id}}` | Every trace ID in the session, newline-joined |\n| `{{traces[0].spans[0].meta.input.value}}` | First span of the first trace |\n| `{{traces[*].spans[*].name}}` | Every span name across the whole session |\n| `{{traces[*].spans[*].meta.output.value}}` | Every span's output across the session |\n| `{{traces[*].spans[meta.span.kind:llm].meta.input.messages[*].content}}` | All LLM-span input messages across the session — the user turns |\n| `{{traces[*].spans[meta.span.kind:llm].meta.output.messages[*].content}}` | All assistant responses across the session |\n| `{{traces[*].spans[meta.span.kind:tool]}}` | Every tool span (paired in\u002Fout) across the session |\n| `{{traces[*].spans[meta.span.kind:retrieval].meta.output.documents[*].text}}` | Text of every retrieved document across the session (RAG) |\n| `{{*}}` | Entire session payload as JSON (debug fallback) |\n\nNote: the filter selector applies to the **`spans`** array (`traces[*].spans[meta.span.kind:llm]`), not to `traces` — a trace object has only `trace_id`, `root_span_id`, and `spans`, so there is nothing to filter on at the trace level.\n\n###### Array selector syntax (applies to all scopes)\n\n- `[N]` — index (0-based)\n- `[START,END]` — inclusive range, `END` is clamped to slice length\n- `[*]` — wildcard (fan-out over all elements)\n- `[field.path:value]` — filter array elements by a nested field equality, e.g. `messages[role:user]` or `spans[meta.span.kind:tool]`\n\n**Resolution rules to keep in mind when writing prompts:**\n- Arrays of strings → newline-joined\n- Arrays of objects \u002F mixed values → compact JSON\n- Single empty slice → empty string\n- Implicit fan-out: `messages.content` behaves the same as `messages[*].content`\n- Negative indices are **not** supported (parse error) — use `[N]` with a known index, or `[*]` for \"last assistant turn\" semantics\n\n**When to pick which form:**\n- **Generic span evaluator** (e.g. `tone_check`, `output_format`) → use `{{span_input}}` \u002F `{{span_output}}` so it works across span kinds.\n- **LLM-span-specific evaluator** (e.g. `system_prompt_adherence`) → reach for explicit `meta.input.messages[*].content` \u002F `meta.output.messages[*].content` so you can split system vs. user vs. assistant turns.\n- **Span-scope RAG evaluator** (single retrieval+generation span) → combine `{{meta.input.documents}}` with `{{span_output}}`.\n- **Trace-scope evaluator** → see \"Trace-scope evaluator examples\" below for the four canonical patterns (goal completion, tool-use correctness, RAG faithfulness, conversation quality).\n- **Session-scope evaluator** → see \"Session-scope evaluator examples\" below for the four canonical patterns (session goal completion, multi-turn conversation quality, user frustration signals, agent consistency across the session).\n- **Metadata-aware evaluator** → reference `{{meta.metadata.\u003Ckey>}}` directly.\n\nIf the user has existing custom evaluators in the same ml_app (Phase 0 coverage map), match their convention when there is no strong reason to deviate.\n\n###### Trace-scope evaluator examples\n\nConcrete user-prompt bodies for the four canonical trace-scope use cases, drawn from the public docs ([Trace-Level Evaluations](https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fevaluations\u002Fcustom_llm_as_a_judge_evaluations\u002Ftrace_level_evaluations\u002F)). Each goes alongside a static System prompt that describes the rubric (no placeholders).\n\n| Use case | `filter` | User prompt body |\n|---|---|---|\n| **Goal completion** — agent finished the user's request | `@parent_id:undefined @meta.span.kind:agent` | `User goal:\\n{{spans[0].meta.input.value}}\\n\\nAgent steps:\\n{{spans}}` |\n| **Tool-use correctness** — right tool with right arguments | `@parent_id:undefined @meta.span.kind:agent` | `User question:\\n{{spans[0].meta.input.value}}\\n\\nTool calls:\\n{{spans[meta.span.kind:tool].meta.input.parameters}}\\n\\nFinal response:\\n{{spans[*].meta.output.value}}` |\n| **RAG faithfulness** — answer grounded in retrieved docs | `@parent_id:undefined` | `Retrieved context:\\n{{spans[meta.span.kind:retrieval].meta.output.documents[*].text}}\\n\\nFinal answer:\\n{{spans[meta.span.kind:llm].meta.output.value}}` |\n| **Conversation quality** — coherence and consistency across turns | `@parent_id:undefined` | `Conversation:\\n{{spans[meta.span.kind:llm].meta.input.messages[*].content}}\\n\\nAssistant responses:\\n{{spans[meta.span.kind:llm].meta.output.messages[*].content}}` |\n\nUse these as starting points. Adapt the `filter` and span paths to the actual span names \u002F kinds the app emits (observed during Phase 1).\n\n###### Session-scope evaluator examples\n\nConcrete user-prompt bodies for the four canonical session-scope use cases (adapted from the public [Session-Level Evaluations](https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fevaluations\u002Fcustom_llm_as_a_judge_evaluations\u002Fsession_level_evaluations\u002F) docs). Each goes alongside a static System prompt that describes the rubric (no placeholders). The `filter` is matched against the **session's root span** (not every span), so scope it with root-span attributes; there is no `@parent_id:undefined` requirement beyond that.\n\n| Use case | `filter` | User prompt body |\n|---|---|---|\n| **Session goal completion** — user's goals met across the session | `@meta.span.kind:agent` (or the app's root span kind) | `Session traces (chronological):\\n{{traces}}` |\n| **Multi-turn conversation quality** — coherence and memory across turns | `@meta.span.kind:agent` | `User messages:\\n{{traces[*].spans[meta.span.kind:llm].meta.input.messages[*].content}}\\n\\nAssistant responses:\\n{{traces[*].spans[meta.span.kind:llm].meta.output.messages[*].content}}` |\n| **User frustration signals** — frustration \u002F abandonment over the session | `@meta.span.kind:agent` | `Full session:\\n{{traces}}` |\n| **Agent consistency across the session** — consistent and recovers across traces | `@meta.span.kind:agent` | `Session traces (chronological):\\n{{traces}}` |\n\nAdapt the `filter` and span paths to the actual span kinds the app emits (observed in Phase 1).\n\n**`output_schema` wrapper format (required for all providers)**\n\nThe `output_schema` field is NOT a bare JSON Schema. It must use the OpenAI `json_schema` object shape. **`name` is a fixed type discriminator**, not the evaluator name — the UI validates it against a strict allowlist and rejects any other value:\n\n| LLMJudge type | `name` value | property key inside `schema` |\n|---------------|-------------|------------------------------|\n| Boolean | `\"boolean_eval\"` | `boolean_eval` |\n| Score | `\"score_eval\"` | `score_eval` |\n| Categorical | `\"categorical_eval\"` | `categorical_eval` |\n\nThe property key inside `schema.properties` must match `name` exactly. The `required` array may only be `[\"\u003Ctype_key>\"]` or `[\"\u003Ctype_key>\", \"reasoning\"]` — any other value is rejected. Always include `\"reasoning\": {\"type\": \"string\"}` for UI display.\n\n**Boolean** (`BooleanStructuredOutput(pass_when=True)`):\n\n```json\n{\n  \"output_schema\": {\n    \"name\": \"boolean_eval\",\n    \"strict\": true,\n    \"schema\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"boolean_eval\": {\"type\": \"boolean\", \"description\": \"Whether the criterion is met\"},\n        \"reasoning\": {\"type\": \"string\", \"description\": \"Explanation for the evaluation\"}\n      },\n      \"required\": [\"boolean_eval\", \"reasoning\"],\n      \"additionalProperties\": false\n    }\n  },\n  \"assessment_criteria\": {\"pass_when\": true}\n}\n```\n\n**Score** (`ScoreStructuredOutput(min_score=1, max_score=10, min_threshold=7)`):\n\n```json\n{\n  \"output_schema\": {\n    \"name\": \"score_eval\",\n    \"strict\": true,\n    \"schema\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"score_eval\": {\"type\": \"number\", \"description\": \"Score from 1 to 10\", \"minimum\": 1, \"maximum\": 10},\n        \"reasoning\": {\"type\": \"string\", \"description\": \"Explanation for the score\"}\n      },\n      \"required\": [\"score_eval\", \"reasoning\"],\n      \"additionalProperties\": false\n    }\n  },\n  \"assessment_criteria\": {\"min_threshold\": 7}\n}\n```\n\nAdd `max_threshold` to `assessment_criteria` if set.\n\n**Categorical** (`CategoricalStructuredOutput(categories={...}, pass_values=[...])`):\n\n```json\n{\n  \"output_schema\": {\n    \"name\": \"categorical_eval\",\n    \"strict\": true,\n    \"schema\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"categorical_eval\": {\n          \"type\": \"string\",\n          \"anyOf\": [\n            {\"const\": \"correct\", \"description\": \"The response correctly answers the question\"},\n            {\"const\": \"partially_correct\", \"description\": \"Partially correct but missing information\"},\n            {\"const\": \"incorrect\", \"description\": \"The response is wrong or irrelevant\"}\n          ]\n        },\n        \"reasoning\": {\"type\": \"string\", \"description\": \"Explanation for the category chosen\"}\n      },\n      \"required\": [\"categorical_eval\", \"reasoning\"],\n      \"additionalProperties\": false\n    }\n  },\n  \"assessment_criteria\": {\"pass_values\": [\"correct\"]}\n}\n```\n\nNote: categorical uses `\"type\": \"string\"` alongside `anyOf` (each `const` is a string value), unlike the offline SDK which uses bare `anyOf` at the property root.\n\n**Custom \u002F multi-dimensional**: not directly supported via the fixed-name schema. Implement as a score or categorical evaluator where possible, or split into multiple evaluators. The `name` must be one of the three fixed values above.\n\n**Filter scoping**: when the proposal targets a specific span kind (e.g. an LLM sub-span), translate it into an EVP `filter` query — e.g. `@meta.span.kind:llm`, `service:checkout-agent`, or a more specific tag. Combine with `root_spans_only:true` only when the target is the trace root.\n\n**For `eval_scope: trace`:**\n\n- The evaluator triggers once per **completed** trace, after a **3-minute inactivity window**. Late-arriving spans (>3 min after the prior span on the same trace) are **excluded** from the evaluation. Surface this in the proposal so the user knows about both the latency and the potential miss for sparse-activity agents (long-running agents whose steps are sparser than 3 minutes apart).\n- The `filter` query must match the trace's **root span only** — always include `@parent_id:undefined` (or `root_spans_only: true`) to avoid double-firing across descendants. Combine with `@meta.span.kind:agent` (or whatever kind the app uses for root spans, observed in Phase 1) for narrowing.\n- Sampling at trace scope is heavier than at span scope (one trace = many spans on the judge's side). Default `sampling_percentage` to **`5`** for trace-scope evaluators (instead of the span default `10`); the user can raise it after a manual review pass.\n\n**For `eval_scope: session`:**\n\n- Only publish session-scope evaluators when `sessions_present` (Phase 1 probe). Otherwise the evaluator never fires — no span carries a `session_id`.\n- The evaluator triggers once per **completed session**. A `session_id` is complete after **30 minutes of inactivity** (vs. 3 minutes for a trace); spans arriving > 30 min after the prior span are **excluded**. Surface both the latency and the exclusion — long-running or sparse sessions may be split or truncated, and very large sessions are capped at 10,000 spans.\n- The `filter` is matched against the session's **root span** (the same prefilter trace scope uses), so scope it with root-span attributes — e.g. `@meta.span.kind:agent` (or the app's root span kind, observed in Phase 1). You do **not** need `@parent_id:undefined`: the session root is already a root span.\n- Sampling at session scope is the heaviest of all (one session = many traces = many spans on the judge's side). Default `sampling_percentage` to **`5`** (matching trace) and flag it as the heaviest scope; the user can lower it further after a manual review pass.\n\n#### Always publish as draft (`enabled: false`)\n\n**Always create \u002F update evaluators with `enabled: false`** — regardless of whether `integration_account_id` was auto-detected from existing evaluators. The UI is the source of truth for activation; the skill should never auto-enable evaluators on the user's behalf. The user reviews each draft in the UI, confirms the integration account is correct (the auto-detected ID may belong to a different judge LLM than the one they want for this app), and flips the toggle when they're satisfied.\n\nThis makes the workflow safe by default: a wrong `integration_account_id`, a mistuned prompt, or an over-broad filter never goes live without a human pass. Auto-detection of the account ID still helps because the draft renders with the right account pre-selected — review is faster.\n\n#### integration_account_id resolution\n\nThe `integration_account_id` is an opaque UUID that the UI matches against the org's integration accounts list to populate the account section dropdown. Users typically don't know this value, so **never ask the user to supply a raw UUID**.\n\n**Resolution order:**\n\n1. **Inherit from existing evaluators** — in Phase 0 you called `get_llmobs_evaluator` for each existing custom evaluator. Check the `llm_provider.integration_account_id` field on those responses. If any of them have a value, use that same ID on the published drafts. If multiple different IDs appear across existing evaluators, pick the most common one and note which you chose so the user can correct it during the UI review pass.\n\n2. **Omit if no existing evaluator has one** — if no custom evaluator in the ml_app has an `integration_account_id`, omit the field from the publish payload. The draft will render without an account pre-selected; the user picks one during the UI review pass before activating.\n\nEither way, the evaluator is published with `enabled: false`. The user is the gate — see \"Always publish as draft\" above.\n\n#### Publish (single message — parallelize)\n\nIssue all `create_or_update_llmobs_evaluator` calls in a **single message** (one per evaluator). Set `telemetry.intent` to a short English description like `\"skill:agent-observability-eval-bootstrap — Bootstrap evaluator suite for ml_app=\u003Cml_app> from production trace analysis.\"`.\n\nIf any call fails, capture the error and continue with the remaining evaluators — never silently abort the batch. Report failures explicitly in the summary.\n\n#### Summary\n\n```\n## Published Evaluators (drafts — pending UI review)\n\nWrote {N} online evaluators to ml_app `{ml_app}`. **All published as drafts (`enabled: false`)** — review and activate them in the UI before they start scoring spans.\n\n| # | Name | Action | Provider\u002FModel | Sampling | Scope | Account auto-detected | Status |\n|---|------|--------|----------------|----------|-------|-----------------------|--------|\n| 1 | task_completion | created (draft) | openai\u002Fgpt-5.4-mini | 10% | span | yes | ok |\n| 2 | response_groundedness | overwrote (draft) | openai\u002Fgpt-5.4-mini | 10% | span | yes | ok |\n| 3 | scope_adherence | renamed (`_v2`) (draft) | openai\u002Fgpt-5.4-mini | 10% | span | no — pick in UI | ok |\n| 4 | citation_format | failed | openai\u002Fgpt-5.4-mini | 10% | span | — | error |\n| 5 | session_goal_completion | created (draft) | openai\u002Fgpt-5.4-mini | 5% | session | yes | ok |\n\n{If any failed:}\n**Errors**:\n- `{name}`: {error message}\n\n{If any code-based proposals were dropped:}\n**Not published** (code-based, not supported by online evaluator API):\n- `{name}` ({type}) — ask me to emit it as offline SDK code, or run `\u002Feval-bootstrap {ml_app} --sdk-code`.\n\n### Next Steps — review and activate in the UI\n\nThe drafts are intentionally not running yet. Walk through each one in the Datadog UI before flipping the enable toggle:\n\n1. **Open the drafts**: Datadog → Agent Observability → Evaluations → filter by ml_app `{ml_app}` (the new drafts appear with status `Disabled`).\n2. **For each draft**:\n   - **Verify the integration account** in the Provider section. If the column above shows `auto-detected: yes`, confirm it's the correct account for the judge LLM you want this evaluator to call through. If `no`, pick an account from the dropdown.\n   - **Skim the prompt template** and the structured-output schema — make sure the span \u002F trace \u002F session scope, filter, and sampling match what you actually want to measure.\n   - **Click into a sample span\u002Ftrace** and use the test pane to dry-run the prompt against real data. Confirm the result matches your expectation.\n3. **Enable**: once each draft passes review, toggle it to enabled. Datadog starts scoring incoming spans immediately.\n4. **Wait for first scores**: with `sampling_percentage=10` (span scope) or `5` (trace \u002F session scope), expect first results within minutes for high-traffic apps — except **session scope**, where the first result appears only after the session's 30-minute inactivity window closes.\n5. **Tune sampling\u002Ffilter**: if results are noisy or volume is too high, reduce `sampling_percentage` or tighten the `filter` from the UI. Re-running `\u002Feval-bootstrap {ml_app}` will round-trip the existing config before overwriting — your manual tweaks survive across reruns.\n6. **Prefer offline code?** These evaluators run online in Datadog. If you'd rather iterate on the same suite as offline SDK code (`sdk_code`) or a portable JSON spec (`data_only`), just ask — I'll regenerate the **confirmed** suite without re-exploring.\n```\n\n#### Notebook export (after summary)\n\nSame logic as Phase 3A — offer to append to the RCA notebook if `rca_notebook_url` was detected, or create a new standalone notebook. The notebook cell should list the published evaluators with their UI links and the `ml_app` they target. In pup mode, use `pup notebooks create` \u002F `pup notebooks edit` as described in Phase 3A.\n\n---\n\n### Phase 3D: Emit Dataset from Production Traces\n\n**Goal**: Sample production traces for `ml_app`, extract input \u002F output pairs, and write a `DatasetRecordRaw[]` JSON file shaped for `LLMObs.create_dataset(records=...)`. **This mode does not propose or generate evaluators** — Phases 1 and 2 are bypassed. The dataset is the sole artifact.\n\n**When this mode runs**: triggered by `--emit-dataset \u003Cpath>`. Skip Phase 0 step 4 (eval coverage map) — it's unrelated to dataset extraction.\n\n#### Sampling\n\n1. Call `search_llmobs_spans(query=\"@ml_app:\\\"\u003Cml_app>\\\" @status:ok\", root_spans_only=true, limit=\u003Ctrace-limit>, from=\u003Ctimeframe>)`. Filter to `@status:ok` so the resulting dataset records reflect intended app behavior (errored traces have no usable output baseline).\n2. For each sampled root span, call `get_llmobs_span_details` (group by `trace_id`) and `get_llmobs_span_content` to read the root span's `input` and `output` fields. Issue all calls for a page in a single message (parallelize).\n\n#### Record extraction\n\nFor each sampled root span, build one `DatasetRecordRaw` entry:\n\n| Field | Source | Notes |\n|---|---|---|\n| `input_data` | Root span's `meta.input` | If `meta.input.value` is a plain string, wrap as `{\"input\": \"\u003Cvalue>\"}`. If `meta.input.messages` exists, use `{\"messages\": [...]}` (preserving the role\u002Fcontent shape). For RAG roots that also have `documents`, include `{\"documents\": [...]}`. |\n| `expected_output` | Root span's `meta.output.value` (or `meta.output.messages[-1].content` for LLM-kind roots) | This is the **production behavior baseline**, not ground truth. Document this clearly in the summary so the user does not over-trust it. |\n| `metadata` | `{\"source\": \"production\", \"trace_id\": \"\u003Cid>\", \"span_id\": \"\u003Cid>\", \"ml_app\": \"\u003Cml_app>\", \"extracted_at\": \"\u003CISO 8601 UTC>\"}` | Trace\u002Fspan IDs let the user click back to the source trace in Datadog. |\n| `tags` | `[\"env:prod\", \"source:traces\", \"ml_app:\u003Cml_app>\"]` | Always `\"key:value\"` strings — bare strings will fail `Dataset.append()`. See the per-record tag rules in `agent-observability-experiment-py-bootstrap\u002FSKILL.md` (search \"Per-record tags\"). |\n\n**PII scrub.** Before writing, run the same scrub used elsewhere in the agent-observability skill set: replace email \u002F phone \u002F SSN \u002F API-key regex matches in all string values with `\u003CREDACTED:pii-type>`. Surface a one-line warning listing affected record indices in the summary.\n\n**Tag normalization (MANDATORY pre-write step).** The SDK's `Dataset.append()` calls `validate_tags_list` which rejects any tag without a `:` separator (`Tag '\u003Ctag>' is malformed. Tags must be in 'key:value' format.`). Before writing the JSON, run this normalization on every record's `tags` list:\n\n1. **Drop** any non-string, `None`, or empty-string entries.\n2. **Already `key:value`** (`\":\" in tag` and both halves non-empty) → keep as-is.\n3. **Bare string** (no `:`) → wrap as `\"tag:\u003Cvalue>\"` (so e.g. `\"interactive\"` becomes `\"tag:interactive\"`). Never silently drop — the original observation is preserved under the generic `tag:` key.\n4. **Trailing or leading `:`** (e.g. `\"foo:\"`, `\":bar\"`) → wrap as `\"tag:\u003Cfull-original-string>\"` so the malformed value is preserved without crashing the SDK.\n\nTrack the count of normalizations and surface in the summary as a one-line note (`Tag normalizations applied: N (bare strings auto-prefixed with 'tag:')`). Do not write the dataset if any record's `tags` list still contains a non-`key:value` string after normalization — that's a bug in this step.\n\n**Skip records** where:\n- `meta.output` is empty or missing (nothing usable as a baseline).\n- The root span's I\u002FO is purely structural (e.g. only tool calls, no natural-language output). Record the count of skipped records and surface in the summary.\n\n#### Write the file\n\n1. Write the JSON array (2-space indented) to `\u003Cpath>` using the Write tool. The path was supplied by the user via `--emit-dataset \u003Cpath>` — use it verbatim.\n2. **Do not** push to Datadog from this mode. Publishing to Datadog is the next step in the onboarding flow (`LLMObs.create_dataset(records=...)`) and is handled by the caller (the onboarding orchestrator or the user directly).\n\n#### Summary\n\nAfter writing, print:\n\n```\n## Generated Dataset\n\nWrote `\u003Cpath>` — {N} records sampled from `\u003Cml_app>` ({timeframe}).\n\n| Stat | Value |\n|---|---|\n| Records emitted | {N} |\n| Records skipped (no output) | {M} |\n| PII redactions applied | {P} (indices: {list or \"none\"}) |\n| Tag normalizations applied | {T} (bare strings auto-prefixed with `tag:`) |\n\n**Important**: `expected_output` is the **current production behavior baseline**, not ground truth. Use this dataset for regression-style experiments (does my refactor change observed outputs?) before promoting it to a labelled gold set.\n\n### Next Steps\n\n1. **Review**: open `\u003Cpath>` and spot-check a few records. Adjust `expected_output` where the production output was wrong.\n2. **Publish to Datadog**:\n   ```python\n   from ddtrace.llmobs import LLMObs\n   import json, os\n   LLMObs.enable(api_key=os.getenv(\"DD_API_KEY\"), app_key=os.getenv(\"DD_APPLICATION_KEY\"),\n                 site=os.getenv(\"DD_SITE\", \"datadoghq.com\"),\n                 project_name=\"\u003Cyour-project>\", agentless_enabled=True)\n   records = json.load(open(\"\u003Cpath>\"))\n   ds = LLMObs.create_dataset(dataset_name=\"\u003Cyour-name>\", records=records)\n   print(ds)\n   ```\n3. **Run an experiment** against the published dataset via `agent-observability-experiment-py-bootstrap --dataset-name \u003Cyour-name>`.\n```\n\nSkip the notebook export prompt in this mode — datasets aren't well-served by a markdown summary in a notebook, and the next step (experiment generation) will produce its own artifacts.\n\n---\n\n## Operating Rules\n\n- **Breadth over precision; let the user curate**: Propose **8–15 evaluators** distributed across **domain-specific** (largest bucket — derived from Phase 1 domain signals), outcome, format, and safety. Users can always remove what doesn't fit their quality bar; they cannot easily add what was not proposed. Anchor every domain-specific proposal in at least one observed trace pattern — don't invent generic domain evaluators without evidence.\n- **Don't overfit**: Write criteria that generalize beyond the specific sampled traces. Use examples as grounding, not as the sole criteria.\n- **Show your work**: Every proposed evaluator cites at least one trace as evidence with a clickable link: `[Trace {first_8}...](https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_32_char_id})`.\n- **New file only**: Never modify existing evaluator code or experiment configurations.\n- **Honest about uncertainty**: If fewer than 5 traces support a proposed evaluator, flag it as tentative.\n- **Session scope is gated**: only propose or publish `eval_scope: session` evaluators when the Phase 1 probe confirmed `sessions_present` (spans carry a `session_id`, ideally one that spans multiple traces). If sessions are absent, skip session scope and say so — never publish a session evaluator that can't fire.\n\n---\n\n## Tool Reference\n\nThis appendix applies only in **pup mode**. In MCP mode, use the tool names in the workflow sections directly.\n\n### Spans and traces\n\n| MCP Tool | pup Command |\n|---|---|\n| `search_llmobs_spans(query, ml_app, from, to, limit, cursor, root_spans_only, span_kind, summary)` | `pup llm-obs spans search --query \"@ml_app:A [other_filters]\" [--from F] [--to T] [--limit N] [--cursor C] [--root-spans-only] [--span-kind K] [--summary]` — **always use `--query \"@ml_app:A\"` to filter by ml_app**; the `--ml-app A` flag is unreliable and silently returns spans from other apps. |\n| `get_llmobs_span_details(trace_id, span_ids, from, to)` | `pup llm-obs spans get-details --trace-id T --span-ids S1,S2,...` |\n| `get_llmobs_span_content(trace_id, span_id, field, path)` | `pup llm-obs spans get-content --trace-id T --span-id S --field F [--path P]` |\n| `get_llmobs_trace(trace_id, include_tree)` | `pup llm-obs spans get-trace --trace-id T [--include-tree]` |\n| `get_llmobs_agent_loop(trace_id, span_id)` | `pup llm-obs spans get-agent-loop --trace-id T [--span-id S]` |\n| `find_llmobs_error_spans(trace_id)` | `pup llm-obs spans find-errors --trace-id T` |\n| `expand_llmobs_spans(trace_id, span_ids, max_depth, filter_kind)` | `pup llm-obs spans expand --trace-id T --span-ids S1,S2,... [--max-depth N] [--filter-kind K]` |\n\n### Evaluators\n\n| MCP Tool | pup Command |\n|---|---|\n| `list_llmobs_evals()` | `pup llm-obs evals list` (filter by `ml_app` client-side) |\n| `list_llmobs_evals_by_ml_app(ml_app)` | `pup llm-obs evals list-by-ml-app --ml-app A` |\n| `get_llmobs_evaluator(eval_name)` | `pup llm-obs evals get-evaluator EVAL_NAME` |\n| `get_llmobs_eval_aggregate_stats(eval_name, ml_app, from, to)` | `pup llm-obs evals get-aggregate-stats EVAL_NAME [--ml-app A] [--from F] [--to T]` |\n| `delete_llmobs_evaluator(eval_name)` | `pup llm-obs evals delete EVAL_NAME` |\n| `create_or_update_llmobs_evaluator(...)` | `pup llm-obs evals create-or-update EVAL_NAME --file \u002Ftmp\u002Feval_EVAL_NAME.json` — see flat schema note below |\n\n#### `create_or_update_llmobs_evaluator` in pup mode\n\npup uses a **flat** JSON file (all fields top-level). `get-evaluator` returns a **nested** object. Transform as follows:\n\n1. **Round-trip check**: Call `pup llm-obs evals get-evaluator EVAL_NAME` first. If it exists, start from its config.\n2. **Flatten `llm_provider`**: hoist `integration_provider`, `model_name`, `integration_account_id`, `temperature` to top level, dropping the `llm_provider` key.\n3. **Merge and set `enabled: false`**.\n4. **Write to temp file and call**:\n   ```bash\n   pup llm-obs evals create-or-update EVAL_NAME --file \u002Ftmp\u002Feval_EVAL_NAME.json\n   ```\n   Use unique temp file names when publishing multiple evaluators in parallel (e.g. `\u002Ftmp\u002Feval_toxicity.json`).\n\n| `get-evaluator` field | Flat JSON key |\n|---|---|\n| `llm_provider.integration_provider` | `integration_provider` |\n| `llm_provider.model_name` | `model_name` |\n| `llm_provider.integration_account_id` | `integration_account_id` |\n| `llm_provider.temperature` | `temperature` |\n| All other fields | Unchanged (already top-level) |\n\n### Notebooks\n\n| MCP Tool | pup Command |\n|---|---|\n| `create_datadog_notebook(name, cells, ...)` | `pup notebooks create --title \"TITLE\" --file \u002Ftmp\u002Fnb_cells.json` — confirm exact flags with `pup notebooks create --help` |\n| `edit_datadog_notebook(id, cells, append_only=true)` | `pup notebooks edit NOTEBOOK_ID --file \u002Ftmp\u002Fnb_cells.json` (fetches current notebook, appends provided cells, writes back) |\n\nThe cells file is a JSON array of cell objects:\n```json\n[{\"attributes\": {\"definition\": {\"type\": \"markdown\", \"text\": \"## Section\\n\\nContent.\"}}, \"type\": \"notebook_cells\"}]\n```\n- **MCP result parsing safety**: Before writing any script (Python, jq, etc.) that iterates over or accesses fields in an MCP tool result, inspect the raw structure first — check `type(result)`, top-level keys, and whether the payload is nested inside a content block (e.g. `[{'type': 'text', 'text': '\u003Cjson>'}]`). Extract and `json.loads()` the inner payload if needed before parsing. Never assume MCP results are bare dicts or lists.\n",{"data":37,"body":38},{"name":4,"description":6},{"type":39,"children":40},"root",[41,50,62,175,185,193,352,370,425,432,444,691,723,729,741,746,753,1065,1106,1112,1362,1375,1387,1567,1578,1634,1654,1695,1701,1772,1776,1782,1809,1815,1952,1957,1971,2042,2104,2117,2179,2185,2263,2315,2321,2331,2385,2395,2473,2483,2567,2577,2671,2739,2745,2772,2807,2813,2818,2936,2942,3058,3064,3266,3272,3277,3383,3386,3392,3398,3406,3481,3552,3625,3633,3876,3879,3885,3895,3902,4634,4640,4827,4830,4836,4845,4906,4921,5074,5086,5383,5390,5403,5451,5457,5472,5499,5637,5642,6055,6067,6075,6081,6140,6255,6267,6298,6347,6450,6455,6645,6651,6711,6717,6735,6909,6915,6923,6932,7014,7019,7022,7028,7039,7107,7113,7160,7193,7212,7215,7221,7237,7242,7248,7469,7475,7487,7891,7896,7902,7915,7924,7930,7935,8126,8151,8251,8261,8681,8684,8690,8699,8714,8720,10025,10031,10194,10200,10254,10263,10268,10303,10306,10312,10320,10356,10362,10380,10501,10507,10591,10601,10790,10856,10862,10894,11087,11100,11110,11201,11211,11370,11382,11563,11575,11787,11838,11844,11912,11920,11982,11990,12119,12124,12130,12146,12298,12310,12316,12351,12501,12513,12526,12557,12670,12721,12737,13255,13270,13828,13847,13862,14611,14645,14662,14701,14715,14811,14824,14932,14944,14966,14978,14984,15001,15009,15054,15066,15072,15104,15109,15115,15124,15129,15159,15162,15168,15203,15220,15226,15287,15293,15306,15516,15534,15581,15710,15737,15747,15765,15770,15808,15813,15818,15827,15847,15856,15868,15988,15994,16068,16073,16240,16277],{"type":42,"tag":43,"props":44,"children":46},"element","h2",{"id":45},"backend",[47],{"type":48,"value":49},"text","Backend",{"type":42,"tag":51,"props":52,"children":53},"p",{},[54,60],{"type":42,"tag":55,"props":56,"children":57},"strong",{},[58],{"type":48,"value":59},"Detection",{"type":48,"value":61}," — At the start of every invocation, before taking any action, determine which backend to use:",{"type":42,"tag":63,"props":64,"children":65},"ol",{},[66,88,101,113,142,153],{"type":42,"tag":67,"props":68,"children":69},"li",{},[70,72,79,81,86],{"type":48,"value":71},"If the user passed ",{"type":42,"tag":73,"props":74,"children":76},"code",{"className":75},[],[77],{"type":48,"value":78},"--backend pup",{"type":48,"value":80}," anywhere in their invocation → use ",{"type":42,"tag":55,"props":82,"children":83},{},[84],{"type":48,"value":85},"pup mode",{"type":48,"value":87}," immediately, regardless of whether MCP tools are present. Skip steps 2–4.",{"type":42,"tag":67,"props":89,"children":90},{},[91,93,99],{"type":48,"value":92},"Check whether MCP tools are present in your active tool list. The canonical signal is whether ",{"type":42,"tag":73,"props":94,"children":96},{"className":95},[],[97],{"type":48,"value":98},"mcp__datadog-llmo-mcp__list_llmobs_evals",{"type":48,"value":100}," appears in your available tools.",{"type":42,"tag":67,"props":102,"children":103},{},[104,106,111],{"type":48,"value":105},"If MCP tools are present → use ",{"type":42,"tag":55,"props":107,"children":108},{},[109],{"type":48,"value":110},"MCP mode",{"type":48,"value":112}," throughout. Call MCP tools exactly as named in this skill's workflow sections.",{"type":42,"tag":67,"props":114,"children":115},{},[116,118,124,126,132,134,140],{"type":48,"value":117},"If MCP tools are absent → check whether ",{"type":42,"tag":73,"props":119,"children":121},{"className":120},[],[122],{"type":48,"value":123},"pup",{"type":48,"value":125}," is executable: run ",{"type":42,"tag":73,"props":127,"children":129},{"className":128},[],[130],{"type":48,"value":131},"pup --version",{"type":48,"value":133}," via Bash. A JSON response containing ",{"type":42,"tag":73,"props":135,"children":137},{"className":136},[],[138],{"type":48,"value":139},"\"version\"",{"type":48,"value":141}," confirms pup is available.",{"type":42,"tag":67,"props":143,"children":144},{},[145,147,151],{"type":48,"value":146},"If pup responds → use ",{"type":42,"tag":55,"props":148,"children":149},{},[150],{"type":48,"value":85},{"type":48,"value":152}," throughout. Translate every MCP tool call to its pup equivalent using the Tool Reference appendix at the bottom of this file.",{"type":42,"tag":67,"props":154,"children":155},{},[156,158],{"type":48,"value":157},"If neither is available → stop and tell the user:\n",{"type":42,"tag":159,"props":160,"children":161},"blockquote",{},[162],{"type":42,"tag":51,"props":163,"children":164},{},[165,167,173],{"type":48,"value":166},"\"Neither the Datadog MCP server nor the pup CLI is available. Connect the MCP server (",{"type":42,"tag":73,"props":168,"children":170},{"className":169},[],[171],{"type":48,"value":172},"claude mcp add --scope user --transport http datadog-llmo-mcp 'https:\u002F\u002Fmcp.datadoghq.com\u002Fapi\u002Funstable\u002Fmcp-server\u002Fmcp?toolsets=llmobs'",{"type":48,"value":174},") or install pup.\"",{"type":42,"tag":51,"props":176,"children":177},{},[178,183],{"type":42,"tag":73,"props":179,"children":181},{"className":180},[],[182],{"type":48,"value":78},{"type":48,"value":184}," is accepted anywhere in the invocation arguments and is stripped before passing remaining args to the skill logic.",{"type":42,"tag":51,"props":186,"children":187},{},[188],{"type":42,"tag":55,"props":189,"children":190},{},[191],{"type":48,"value":192},"pup invocation rules:",{"type":42,"tag":194,"props":195,"children":196},"ul",{},[197,208,221,234,239,333],{"type":42,"tag":67,"props":198,"children":199},{},[200,202],{"type":48,"value":201},"Invoke via Bash: ",{"type":42,"tag":73,"props":203,"children":205},{"className":204},[],[206],{"type":48,"value":207},"pup llm-obs \u003Csubcommand> [flags]",{"type":42,"tag":67,"props":209,"children":210},{},[211,213,219],{"type":48,"value":212},"pup always outputs JSON. Parse directly — no content-block unwrapping (unlike MCP results, which may wrap JSON in ",{"type":42,"tag":73,"props":214,"children":216},{"className":215},[],[217],{"type":48,"value":218},"[{\"type\": \"text\", \"text\": \"\u003Cjson>\"}]",{"type":48,"value":220},").",{"type":42,"tag":67,"props":222,"children":223},{},[224,226,232],{"type":48,"value":225},"If pup returns an auth error, tell the user to run ",{"type":42,"tag":73,"props":227,"children":229},{"className":228},[],[230],{"type":48,"value":231},"pup auth login",{"type":48,"value":233}," and stop.",{"type":42,"tag":67,"props":235,"children":236},{},[237],{"type":48,"value":238},"Parallelization: issue multiple Bash tool calls in a single message (one pup command per call).",{"type":42,"tag":67,"props":240,"children":241},{},[242,244,250,252,258,259,265,267,272,274,280,282,288,290,296,298,303,304,310,311,317,318,324,325,331],{"type":48,"value":243},"Time flags: pup accepts bare duration strings (",{"type":42,"tag":73,"props":245,"children":247},{"className":246},[],[248],{"type":48,"value":249},"1h",{"type":48,"value":251},", ",{"type":42,"tag":73,"props":253,"children":255},{"className":254},[],[256],{"type":48,"value":257},"7d",{"type":48,"value":251},{"type":42,"tag":73,"props":260,"children":262},{"className":261},[],[263],{"type":48,"value":264},"30m",{"type":48,"value":266},") and RFC3339 timestamps. Do ",{"type":42,"tag":55,"props":268,"children":269},{},[270],{"type":48,"value":271},"not",{"type":48,"value":273}," use ",{"type":42,"tag":73,"props":275,"children":277},{"className":276},[],[278],{"type":48,"value":279},"now-",{"type":48,"value":281},"-prefixed strings — strip the prefix when converting from a skill ",{"type":42,"tag":73,"props":283,"children":285},{"className":284},[],[286],{"type":48,"value":287},"--timeframe",{"type":48,"value":289}," argument: ",{"type":42,"tag":73,"props":291,"children":293},{"className":292},[],[294],{"type":48,"value":295},"now-7d",{"type":48,"value":297}," → ",{"type":42,"tag":73,"props":299,"children":301},{"className":300},[],[302],{"type":48,"value":257},{"type":48,"value":251},{"type":42,"tag":73,"props":305,"children":307},{"className":306},[],[308],{"type":48,"value":309},"now-24h",{"type":48,"value":297},{"type":42,"tag":73,"props":312,"children":314},{"className":313},[],[315],{"type":48,"value":316},"24h",{"type":48,"value":251},{"type":42,"tag":73,"props":319,"children":321},{"className":320},[],[322],{"type":48,"value":323},"now-30d",{"type":48,"value":297},{"type":42,"tag":73,"props":326,"children":328},{"className":327},[],[329],{"type":48,"value":330},"30d",{"type":48,"value":332},".",{"type":42,"tag":67,"props":334,"children":335},{},[336,342,344,350],{"type":42,"tag":73,"props":337,"children":339},{"className":338},[],[340],{"type":48,"value":341},"--summary",{"type":48,"value":343}," on ",{"type":42,"tag":73,"props":345,"children":347},{"className":346},[],[348],{"type":48,"value":349},"pup llm-obs spans search",{"type":48,"value":351}," strips payload fields to essential metadata only. Use it in bulk\u002Fsearch phases where content is not needed.",{"type":42,"tag":51,"props":353,"children":354},{},[355,360,362,368],{"type":42,"tag":55,"props":356,"children":357},{},[358],{"type":48,"value":359},"Invocation ID:",{"type":48,"value":361}," At the very start of each invocation, before any MCP tool call, generate an 8-character hex invocation ID (e.g., ",{"type":42,"tag":73,"props":363,"children":365},{"className":364},[],[366],{"type":48,"value":367},"3a9f1c2b",{"type":48,"value":369},"). Keep it constant for the entire invocation.",{"type":42,"tag":51,"props":371,"children":372},{},[373,378,380,386,388,394,396,401,403,409,411,417,419],{"type":42,"tag":55,"props":374,"children":375},{},[376],{"type":48,"value":377},"Intent tagging:",{"type":48,"value":379}," On every MCP tool call, prefix ",{"type":42,"tag":73,"props":381,"children":383},{"className":382},[],[384],{"type":48,"value":385},"telemetry.intent",{"type":48,"value":387}," with ",{"type":42,"tag":73,"props":389,"children":391},{"className":390},[],[392],{"type":48,"value":393},"skill:agent-observability-eval-bootstrap[\u003Cinv_id>] — ",{"type":48,"value":395}," followed by a description of why the tool is being called. On the ",{"type":42,"tag":55,"props":397,"children":398},{},[399],{"type":48,"value":400},"first MCP tool call only",{"type":48,"value":402},", use ",{"type":42,"tag":73,"props":404,"children":406},{"className":405},[],[407],{"type":48,"value":408},"skill:agent-observability-eval-bootstrap:start[\u003Cinv_id>] — ",{"type":48,"value":410}," instead (note the ",{"type":42,"tag":73,"props":412,"children":414},{"className":413},[],[415],{"type":48,"value":416},":start",{"type":48,"value":418}," suffix). Example first call: ",{"type":42,"tag":73,"props":420,"children":422},{"className":421},[],[423],{"type":48,"value":424},"skill:agent-observability-eval-bootstrap:start[3a9f1c2b] — Phase 0: map existing eval coverage for task-cruncher",{"type":42,"tag":426,"props":427,"children":429},"h1",{"id":428},"eval-bootstrap-generate-evaluators-from-production-traces",[430],{"type":48,"value":431},"Eval Bootstrap — Generate Evaluators from Production Traces",{"type":42,"tag":51,"props":433,"children":434},{},[435,437,442],{"type":48,"value":436},"Given a sample of production LLM traces, analyze input\u002Foutput patterns and quality dimensions, then propose a ready-to-use evaluator suite. Four output modes — ",{"type":42,"tag":55,"props":438,"children":439},{},[440],{"type":48,"value":441},"online evaluators are the default",{"type":48,"value":443},"; SDK code, the JSON spec, and the dataset-emit mode are produced on request:",{"type":42,"tag":194,"props":445,"children":446},{},[447,548,607,634],{"type":42,"tag":67,"props":448,"children":449},{},[450,459,461,467,469,474,476,481,483,489,491,496,498,504,506,511,513,518,520,525,526,531,533,538,540,546],{"type":42,"tag":55,"props":451,"children":452},{},[453],{"type":42,"tag":73,"props":454,"children":456},{"className":455},[],[457],{"type":48,"value":458},"publish",{"type":48,"value":460}," ",{"type":42,"tag":462,"props":463,"children":464},"em",{},[465],{"type":48,"value":466},"(default)",{"type":48,"value":468}," — propose ",{"type":42,"tag":55,"props":470,"children":471},{},[472],{"type":48,"value":473},"online",{"type":48,"value":475}," LLM-judge evaluators, then — ",{"type":42,"tag":55,"props":477,"children":478},{},[479],{"type":48,"value":480},"only after you confirm the suite",{"type":48,"value":482}," — write them to Datadog via ",{"type":42,"tag":73,"props":484,"children":486},{"className":485},[],[487],{"type":48,"value":488},"create_or_update_llmobs_evaluator",{"type":48,"value":490}," as ",{"type":42,"tag":55,"props":492,"children":493},{},[494],{"type":48,"value":495},"disabled drafts",{"type":48,"value":497}," (",{"type":42,"tag":73,"props":499,"children":501},{"className":500},[],[502],{"type":48,"value":503},"enabled: false",{"type":48,"value":505},"). Nothing is created until you confirm at the Phase 2 checkpoint, and nothing scores any spans until ",{"type":42,"tag":55,"props":507,"children":508},{},[509],{"type":48,"value":510},"you",{"type":48,"value":512}," enable it in the UI — the skill never auto-publishes a live evaluator. Once you enable a draft, it runs automatically on matching production spans, traces, or sessions (no dataset, no task function). The skill ",{"type":42,"tag":55,"props":514,"children":515},{},[516],{"type":48,"value":517},"auto-classifies",{"type":48,"value":519}," each proposed evaluator as ",{"type":42,"tag":55,"props":521,"children":522},{},[523],{"type":48,"value":524},"span-scoped",{"type":48,"value":251},{"type":42,"tag":55,"props":527,"children":528},{},[529],{"type":48,"value":530},"trace-scoped",{"type":48,"value":532},", or ",{"type":42,"tag":55,"props":534,"children":535},{},[536],{"type":48,"value":537},"session-scoped",{"type":48,"value":539}," based on what the judgment requires (a per-LLM-call tone check vs. an agent goal completion that needs the whole trace vs. user satisfaction across a whole multi-trace conversation) — you accept or override the classification at that checkpoint. Session-scoped evaluators are only proposed when the app's spans carry a ",{"type":42,"tag":73,"props":541,"children":543},{"className":542},[],[544],{"type":48,"value":545},"session_id",{"type":48,"value":547}," (verified by a probe in Phase 1).",{"type":42,"tag":67,"props":549,"children":550},{},[551,560,561,574,576,582,584,590,592,598,600,605],{"type":42,"tag":55,"props":552,"children":553},{},[554],{"type":42,"tag":73,"props":555,"children":557},{"className":556},[],[558],{"type":48,"value":559},"sdk_code",{"type":48,"value":460},{"type":42,"tag":462,"props":562,"children":563},{},[564,566,572],{"type":48,"value":565},"(on request — ",{"type":42,"tag":73,"props":567,"children":569},{"className":568},[],[570],{"type":48,"value":571},"--sdk-code",{"type":48,"value":573},", or ask after a publish run)",{"type":48,"value":575}," — Python ",{"type":42,"tag":73,"props":577,"children":579},{"className":578},[],[580],{"type":48,"value":581},".py",{"type":48,"value":583}," file using the Datadog Evals SDK (",{"type":42,"tag":73,"props":585,"children":587},{"className":586},[],[588],{"type":48,"value":589},"BaseEvaluator",{"type":48,"value":591}," \u002F ",{"type":42,"tag":73,"props":593,"children":595},{"className":594},[],[596],{"type":48,"value":597},"LLMJudge",{"type":48,"value":599},") for ",{"type":42,"tag":55,"props":601,"children":602},{},[603],{"type":48,"value":604},"offline",{"type":48,"value":606}," experiments.",{"type":42,"tag":67,"props":608,"children":609},{},[610,619,620,632],{"type":42,"tag":55,"props":611,"children":612},{},[613],{"type":42,"tag":73,"props":614,"children":616},{"className":615},[],[617],{"type":48,"value":618},"data_only",{"type":48,"value":460},{"type":42,"tag":462,"props":621,"children":622},{},[623,624,630],{"type":48,"value":565},{"type":42,"tag":73,"props":625,"children":627},{"className":626},[],[628],{"type":48,"value":629},"--data-only",{"type":48,"value":631},")",{"type":48,"value":633}," — self-contained JSON spec, framework-agnostic.",{"type":42,"tag":67,"props":635,"children":636},{},[637,646,647,658,660,666,668,674,676,681,683,689],{"type":42,"tag":55,"props":638,"children":639},{},[640],{"type":42,"tag":73,"props":641,"children":643},{"className":642},[],[644],{"type":48,"value":645},"emit_dataset",{"type":48,"value":460},{"type":42,"tag":462,"props":648,"children":649},{},[650,651,657],{"type":48,"value":565},{"type":42,"tag":73,"props":652,"children":654},{"className":653},[],[655],{"type":48,"value":656},"--emit-dataset \u003Cpath>",{"type":48,"value":631},{"type":48,"value":659}," — sample production traces and write a ",{"type":42,"tag":73,"props":661,"children":663},{"className":662},[],[664],{"type":48,"value":665},"DatasetRecordRaw[]",{"type":48,"value":667}," JSON file shaped for ",{"type":42,"tag":73,"props":669,"children":671},{"className":670},[],[672],{"type":48,"value":673},"LLMObs.create_dataset(records=...)",{"type":48,"value":675},". ",{"type":42,"tag":55,"props":677,"children":678},{},[679],{"type":48,"value":680},"Skips evaluator proposal and generation entirely",{"type":48,"value":682}," — this mode produces a dataset, not evaluators. Used by ",{"type":42,"tag":73,"props":684,"children":686},{"className":685},[],[687],{"type":48,"value":688},"agent-observability-eval-pipeline",{"type":48,"value":690}," (Phase 4) to seed an experiment dataset from production behavior.",{"type":42,"tag":51,"props":692,"children":693},{},[694,696,701,703,708,709,714,716,721],{"type":48,"value":695},"After a publish run, if the user wants the same suite as offline code or a portable spec, they just ask — the skill regenerates the ",{"type":42,"tag":55,"props":697,"children":698},{},[699],{"type":48,"value":700},"already-confirmed",{"type":48,"value":702}," suite in ",{"type":42,"tag":73,"props":704,"children":706},{"className":705},[],[707],{"type":48,"value":559},{"type":48,"value":591},{"type":42,"tag":73,"props":710,"children":712},{"className":711},[],[713],{"type":48,"value":618},{"type":48,"value":715}," mode without re-exploring (see \"On-request code generation\" in Phase 3). The ",{"type":42,"tag":73,"props":717,"children":719},{"className":718},[],[720],{"type":48,"value":645},{"type":48,"value":722}," mode is independent of the evaluator workflow and never re-uses a prior proposal — it always re-samples traces.",{"type":42,"tag":43,"props":724,"children":726},{"id":725},"usage",[727],{"type":48,"value":728},"Usage",{"type":42,"tag":730,"props":731,"children":735},"pre",{"className":732,"code":734,"language":48},[733],"language-text","\u002Feval-bootstrap \u003Cml_app> [--timeframe \u003Cwindow>] [--sdk-code | --data-only | --emit-dataset \u003Cpath>] [--trace-limit \u003CN>]\n",[736],{"type":42,"tag":73,"props":737,"children":739},{"__ignoreMap":738},"",[740],{"type":48,"value":734},{"type":42,"tag":51,"props":742,"children":743},{},[744],{"type":48,"value":745},"Arguments: $ARGUMENTS",{"type":42,"tag":747,"props":748,"children":750},"h3",{"id":749},"inputs",[751],{"type":48,"value":752},"Inputs",{"type":42,"tag":754,"props":755,"children":756},"table",{},[757,786],{"type":42,"tag":758,"props":759,"children":760},"thead",{},[761],{"type":42,"tag":762,"props":763,"children":764},"tr",{},[765,771,776,781],{"type":42,"tag":766,"props":767,"children":768},"th",{},[769],{"type":48,"value":770},"Input",{"type":42,"tag":766,"props":772,"children":773},{},[774],{"type":48,"value":775},"Required",{"type":42,"tag":766,"props":777,"children":778},{},[779],{"type":48,"value":780},"Default",{"type":42,"tag":766,"props":782,"children":783},{},[784],{"type":48,"value":785},"Description",{"type":42,"tag":787,"props":788,"children":789},"tbody",{},[790,818,848,881,927,963,1019],{"type":42,"tag":762,"props":791,"children":792},{},[793,803,808,813],{"type":42,"tag":794,"props":795,"children":796},"td",{},[797],{"type":42,"tag":73,"props":798,"children":800},{"className":799},[],[801],{"type":48,"value":802},"ml_app",{"type":42,"tag":794,"props":804,"children":805},{},[806],{"type":48,"value":807},"Yes",{"type":42,"tag":794,"props":809,"children":810},{},[811],{"type":48,"value":812},"—",{"type":42,"tag":794,"props":814,"children":815},{},[816],{"type":48,"value":817},"ML application to scope traces",{"type":42,"tag":762,"props":819,"children":820},{},[821,830,835,843],{"type":42,"tag":794,"props":822,"children":823},{},[824],{"type":42,"tag":73,"props":825,"children":827},{"className":826},[],[828],{"type":48,"value":829},"timeframe",{"type":42,"tag":794,"props":831,"children":832},{},[833],{"type":48,"value":834},"No",{"type":42,"tag":794,"props":836,"children":837},{},[838],{"type":42,"tag":73,"props":839,"children":841},{"className":840},[],[842],{"type":48,"value":295},{"type":42,"tag":794,"props":844,"children":845},{},[846],{"type":48,"value":847},"How far back to look",{"type":42,"tag":762,"props":849,"children":850},{},[851,860,864,868],{"type":42,"tag":794,"props":852,"children":853},{},[854],{"type":42,"tag":73,"props":855,"children":857},{"className":856},[],[858],{"type":48,"value":859},"rca_report",{"type":42,"tag":794,"props":861,"children":862},{},[863],{"type":48,"value":834},{"type":42,"tag":794,"props":865,"children":866},{},[867],{"type":48,"value":812},{"type":42,"tag":794,"props":869,"children":870},{},[871,873,879],{"type":48,"value":872},"Failure taxonomy from ",{"type":42,"tag":73,"props":874,"children":876},{"className":875},[],[877],{"type":48,"value":878},"eval-trace-rca",{"type":48,"value":880}," skill, or a free-text failure hypothesis",{"type":42,"tag":762,"props":882,"children":883},{},[884,892,896,901],{"type":42,"tag":794,"props":885,"children":886},{},[887],{"type":42,"tag":73,"props":888,"children":890},{"className":889},[],[891],{"type":48,"value":571},{"type":42,"tag":794,"props":893,"children":894},{},[895],{"type":48,"value":834},{"type":42,"tag":794,"props":897,"children":898},{},[899],{"type":48,"value":900},"off",{"type":42,"tag":794,"props":902,"children":903},{},[904,906,911,913,918,920,926],{"type":48,"value":905},"Emit a Python SDK ",{"type":42,"tag":73,"props":907,"children":909},{"className":908},[],[910],{"type":48,"value":581},{"type":48,"value":912}," file for offline experiments instead of publishing online. Mutually exclusive with ",{"type":42,"tag":73,"props":914,"children":916},{"className":915},[],[917],{"type":48,"value":629},{"type":48,"value":919}," and ",{"type":42,"tag":73,"props":921,"children":923},{"className":922},[],[924],{"type":48,"value":925},"--emit-dataset",{"type":48,"value":332},{"type":42,"tag":762,"props":928,"children":929},{},[930,938,942,946],{"type":42,"tag":794,"props":931,"children":932},{},[933],{"type":42,"tag":73,"props":934,"children":936},{"className":935},[],[937],{"type":48,"value":629},{"type":42,"tag":794,"props":939,"children":940},{},[941],{"type":48,"value":834},{"type":42,"tag":794,"props":943,"children":944},{},[945],{"type":48,"value":900},{"type":42,"tag":794,"props":947,"children":948},{},[949,951,956,957,962],{"type":48,"value":950},"Emit a self-contained JSON spec file instead of publishing online. Mutually exclusive with ",{"type":42,"tag":73,"props":952,"children":954},{"className":953},[],[955],{"type":48,"value":571},{"type":48,"value":919},{"type":42,"tag":73,"props":958,"children":960},{"className":959},[],[961],{"type":48,"value":925},{"type":48,"value":332},{"type":42,"tag":762,"props":964,"children":965},{},[966,974,978,982],{"type":42,"tag":794,"props":967,"children":968},{},[969],{"type":42,"tag":73,"props":970,"children":972},{"className":971},[],[973],{"type":48,"value":656},{"type":42,"tag":794,"props":975,"children":976},{},[977],{"type":48,"value":834},{"type":42,"tag":794,"props":979,"children":980},{},[981],{"type":48,"value":900},{"type":42,"tag":794,"props":983,"children":984},{},[985,990,992,997,999,1005,1007,1012,1013,1018],{"type":42,"tag":55,"props":986,"children":987},{},[988],{"type":48,"value":989},"Dataset-only mode.",{"type":48,"value":991}," Sample production traces and write a ",{"type":42,"tag":73,"props":993,"children":995},{"className":994},[],[996],{"type":48,"value":665},{"type":48,"value":998}," JSON to ",{"type":42,"tag":73,"props":1000,"children":1002},{"className":1001},[],[1003],{"type":48,"value":1004},"\u003Cpath>",{"type":48,"value":1006},". Skips the evaluator workflow entirely. Mutually exclusive with ",{"type":42,"tag":73,"props":1008,"children":1010},{"className":1009},[],[1011],{"type":48,"value":571},{"type":48,"value":919},{"type":42,"tag":73,"props":1014,"children":1016},{"className":1015},[],[1017],{"type":48,"value":629},{"type":48,"value":332},{"type":42,"tag":762,"props":1020,"children":1021},{},[1022,1031,1035,1053],{"type":42,"tag":794,"props":1023,"children":1024},{},[1025],{"type":42,"tag":73,"props":1026,"children":1028},{"className":1027},[],[1029],{"type":48,"value":1030},"--trace-limit",{"type":42,"tag":794,"props":1032,"children":1033},{},[1034],{"type":48,"value":834},{"type":42,"tag":794,"props":1036,"children":1037},{},[1038,1044,1046,1052],{"type":42,"tag":73,"props":1039,"children":1041},{"className":1040},[],[1042],{"type":48,"value":1043},"20",{"type":48,"value":1045}," (cap ",{"type":42,"tag":73,"props":1047,"children":1049},{"className":1048},[],[1050],{"type":48,"value":1051},"50",{"type":48,"value":631},{"type":42,"tag":794,"props":1054,"children":1055},{},[1056,1058,1063],{"type":48,"value":1057},"Max traces to sample in ",{"type":42,"tag":73,"props":1059,"children":1061},{"className":1060},[],[1062],{"type":48,"value":645},{"type":48,"value":1064}," mode",{"type":42,"tag":51,"props":1066,"children":1067},{},[1068,1070,1075,1077,1085,1087,1092,1093,1098,1099,1104],{"type":48,"value":1069},"If ",{"type":42,"tag":73,"props":1071,"children":1073},{"className":1072},[],[1074],{"type":48,"value":802},{"type":48,"value":1076}," is missing, ask the user before proceeding. With no mode flag, the skill defaults to ",{"type":42,"tag":55,"props":1078,"children":1079},{},[1080],{"type":42,"tag":73,"props":1081,"children":1083},{"className":1082},[],[1084],{"type":48,"value":458},{"type":48,"value":1086}," — it proposes online evaluators and, only after you confirm, creates them as disabled drafts (it never auto-enables them). If more than one of ",{"type":42,"tag":73,"props":1088,"children":1090},{"className":1089},[],[1091],{"type":48,"value":571},{"type":48,"value":251},{"type":42,"tag":73,"props":1094,"children":1096},{"className":1095},[],[1097],{"type":48,"value":629},{"type":48,"value":251},{"type":42,"tag":73,"props":1100,"children":1102},{"className":1101},[],[1103],{"type":48,"value":925},{"type":48,"value":1105}," is supplied, error out and ask which mode the user wants.",{"type":42,"tag":43,"props":1107,"children":1109},{"id":1108},"available-tools",[1110],{"type":48,"value":1111},"Available Tools",{"type":42,"tag":754,"props":1113,"children":1114},{},[1115,1131],{"type":42,"tag":758,"props":1116,"children":1117},{},[1118],{"type":42,"tag":762,"props":1119,"children":1120},{},[1121,1126],{"type":42,"tag":766,"props":1122,"children":1123},{},[1124],{"type":48,"value":1125},"Tool",{"type":42,"tag":766,"props":1127,"children":1128},{},[1129],{"type":48,"value":1130},"Purpose",{"type":42,"tag":787,"props":1132,"children":1133},{},[1134,1151,1176,1201,1218,1235,1274,1320,1341],{"type":42,"tag":762,"props":1135,"children":1136},{},[1137,1146],{"type":42,"tag":794,"props":1138,"children":1139},{},[1140],{"type":42,"tag":73,"props":1141,"children":1143},{"className":1142},[],[1144],{"type":48,"value":1145},"search_llmobs_spans",{"type":42,"tag":794,"props":1147,"children":1148},{},[1149],{"type":48,"value":1150},"Find spans by eval presence, tags, span kind, query syntax. Paginate with cursor.",{"type":42,"tag":762,"props":1152,"children":1153},{},[1154,1163],{"type":42,"tag":794,"props":1155,"children":1156},{},[1157],{"type":42,"tag":73,"props":1158,"children":1160},{"className":1159},[],[1161],{"type":48,"value":1162},"get_llmobs_span_details",{"type":42,"tag":794,"props":1164,"children":1165},{},[1166,1168,1174],{"type":48,"value":1167},"Metadata, evaluations (scores, labels, reasoning), and ",{"type":42,"tag":73,"props":1169,"children":1171},{"className":1170},[],[1172],{"type":48,"value":1173},"content_info",{"type":48,"value":1175}," map showing available fields + sizes.",{"type":42,"tag":762,"props":1177,"children":1178},{},[1179,1188],{"type":42,"tag":794,"props":1180,"children":1181},{},[1182],{"type":42,"tag":73,"props":1183,"children":1185},{"className":1184},[],[1186],{"type":48,"value":1187},"get_llmobs_span_content",{"type":42,"tag":794,"props":1189,"children":1190},{},[1191,1193,1199],{"type":48,"value":1192},"Actual content for a span field. Supports JSONPath via ",{"type":42,"tag":73,"props":1194,"children":1196},{"className":1195},[],[1197],{"type":48,"value":1198},"path",{"type":48,"value":1200}," param for targeted extraction.",{"type":42,"tag":762,"props":1202,"children":1203},{},[1204,1213],{"type":42,"tag":794,"props":1205,"children":1206},{},[1207],{"type":42,"tag":73,"props":1208,"children":1210},{"className":1209},[],[1211],{"type":48,"value":1212},"get_llmobs_trace",{"type":42,"tag":794,"props":1214,"children":1215},{},[1216],{"type":48,"value":1217},"Full trace hierarchy as span tree with span counts by kind.",{"type":42,"tag":762,"props":1219,"children":1220},{},[1221,1230],{"type":42,"tag":794,"props":1222,"children":1223},{},[1224],{"type":42,"tag":73,"props":1225,"children":1227},{"className":1226},[],[1228],{"type":48,"value":1229},"get_llmobs_agent_loop",{"type":42,"tag":794,"props":1231,"children":1232},{},[1233],{"type":48,"value":1234},"Chronological agent execution timeline (LLM calls, tool invocations, decisions).",{"type":42,"tag":762,"props":1236,"children":1237},{},[1238,1247],{"type":42,"tag":794,"props":1239,"children":1240},{},[1241],{"type":42,"tag":73,"props":1242,"children":1244},{"className":1243},[],[1245],{"type":48,"value":1246},"list_llmobs_evals",{"type":42,"tag":794,"props":1248,"children":1249},{},[1250,1252,1258,1260,1265,1267,1272],{"type":48,"value":1251},"List every evaluator configured for the caller's org across all ml_apps, with ",{"type":42,"tag":73,"props":1253,"children":1255},{"className":1254},[],[1256],{"type":48,"value":1257},"enabled",{"type":48,"value":1259}," status and ",{"type":42,"tag":73,"props":1261,"children":1263},{"className":1262},[],[1264],{"type":48,"value":802},{"type":48,"value":1266}," per result. Call once in Phase 0 to map existing coverage before proposing new evaluators — filter the result by ",{"type":42,"tag":73,"props":1268,"children":1270},{"className":1269},[],[1271],{"type":48,"value":802},{"type":48,"value":1273}," client-side.",{"type":42,"tag":762,"props":1275,"children":1276},{},[1277,1286],{"type":42,"tag":794,"props":1278,"children":1279},{},[1280],{"type":42,"tag":73,"props":1281,"children":1283},{"className":1282},[],[1284],{"type":48,"value":1285},"get_llmobs_evaluator",{"type":42,"tag":794,"props":1287,"children":1288},{},[1289,1291,1296,1298,1303,1305,1310,1312,1318],{"type":48,"value":1290},"Fetch the ",{"type":42,"tag":55,"props":1292,"children":1293},{},[1294],{"type":48,"value":1295},"full",{"type":48,"value":1297}," persisted evaluator config by name (target ml_app + sampling + filter, provider, prompt template, parsing type, output schema, assessment criteria). Use in Phase 0 to understand what each existing custom eval measures, and (in publish mode) ",{"type":42,"tag":55,"props":1299,"children":1300},{},[1301],{"type":48,"value":1302},"before any update",{"type":48,"value":1304}," — ",{"type":42,"tag":73,"props":1306,"children":1308},{"className":1307},[],[1309],{"type":48,"value":488},{"type":48,"value":1311}," is full-replace, so you must round-trip the full config to avoid clobbering fields. Not all evaluators have a stored config (notably ",{"type":42,"tag":73,"props":1313,"children":1315},{"className":1314},[],[1316],{"type":48,"value":1317},"source=ootb",{"type":48,"value":1319},"); a not-found error there is expected — skip those.",{"type":42,"tag":762,"props":1321,"children":1322},{},[1323,1331],{"type":42,"tag":794,"props":1324,"children":1325},{},[1326],{"type":42,"tag":73,"props":1327,"children":1329},{"className":1328},[],[1330],{"type":48,"value":488},{"type":42,"tag":794,"props":1332,"children":1333},{},[1334,1339],{"type":42,"tag":462,"props":1335,"children":1336},{},[1337],{"type":48,"value":1338},"(publish mode)",{"type":48,"value":1340}," Write an LLM-judge evaluator config to Datadog. Full-replace semantics: any omitted optional field resets to its default. See \"Publishing Conventions\" for required fields and structured output → JSON schema mapping.",{"type":42,"tag":762,"props":1342,"children":1343},{},[1344,1353],{"type":42,"tag":794,"props":1345,"children":1346},{},[1347],{"type":42,"tag":73,"props":1348,"children":1350},{"className":1349},[],[1351],{"type":48,"value":1352},"delete_llmobs_evaluator",{"type":42,"tag":794,"props":1354,"children":1355},{},[1356,1360],{"type":42,"tag":462,"props":1357,"children":1358},{},[1359],{"type":48,"value":1338},{"type":48,"value":1361}," Only used if the user explicitly asks to remove an evaluator. Never invoke speculatively.",{"type":42,"tag":747,"props":1363,"children":1365},{"id":1364},"key-get_llmobs_span_content-patterns",[1366,1368,1373],{"type":48,"value":1367},"Key ",{"type":42,"tag":73,"props":1369,"children":1371},{"className":1370},[],[1372],{"type":48,"value":1187},{"type":48,"value":1374}," Patterns",{"type":42,"tag":51,"props":1376,"children":1377},{},[1378,1380,1385],{"type":48,"value":1379},"Use the ",{"type":42,"tag":73,"props":1381,"children":1383},{"className":1382},[],[1384],{"type":48,"value":1198},{"type":48,"value":1386}," parameter to extract targeted data without fetching full payloads:",{"type":42,"tag":754,"props":1388,"children":1389},{},[1390,1411],{"type":42,"tag":758,"props":1391,"children":1392},{},[1393],{"type":42,"tag":762,"props":1394,"children":1395},{},[1396,1401,1406],{"type":42,"tag":766,"props":1397,"children":1398},{},[1399],{"type":48,"value":1400},"Field",{"type":42,"tag":766,"props":1402,"children":1403},{},[1404],{"type":48,"value":1405},"Path",{"type":42,"tag":766,"props":1407,"children":1408},{},[1409],{"type":48,"value":1410},"What you get",{"type":42,"tag":787,"props":1412,"children":1413},{},[1414,1448,1473,1497,1525,1546],{"type":42,"tag":762,"props":1415,"children":1416},{},[1417,1426,1435],{"type":42,"tag":794,"props":1418,"children":1419},{},[1420],{"type":42,"tag":73,"props":1421,"children":1423},{"className":1422},[],[1424],{"type":48,"value":1425},"messages",{"type":42,"tag":794,"props":1427,"children":1428},{},[1429],{"type":42,"tag":73,"props":1430,"children":1432},{"className":1431},[],[1433],{"type":48,"value":1434},"$.messages[0]",{"type":42,"tag":794,"props":1436,"children":1437},{},[1438,1440,1446],{"type":48,"value":1439},"System prompt (first message, usually ",{"type":42,"tag":73,"props":1441,"children":1443},{"className":1442},[],[1444],{"type":48,"value":1445},"system",{"type":48,"value":1447}," role)",{"type":42,"tag":762,"props":1449,"children":1450},{},[1451,1459,1468],{"type":42,"tag":794,"props":1452,"children":1453},{},[1454],{"type":42,"tag":73,"props":1455,"children":1457},{"className":1456},[],[1458],{"type":48,"value":1425},{"type":42,"tag":794,"props":1460,"children":1461},{},[1462],{"type":42,"tag":73,"props":1463,"children":1465},{"className":1464},[],[1466],{"type":48,"value":1467},"$.messages[-1]",{"type":42,"tag":794,"props":1469,"children":1470},{},[1471],{"type":48,"value":1472},"Last assistant response",{"type":42,"tag":762,"props":1474,"children":1475},{},[1476,1484,1492],{"type":42,"tag":794,"props":1477,"children":1478},{},[1479],{"type":42,"tag":73,"props":1480,"children":1482},{"className":1481},[],[1483],{"type":48,"value":1425},{"type":42,"tag":794,"props":1485,"children":1486},{},[1487],{"type":42,"tag":462,"props":1488,"children":1489},{},[1490],{"type":48,"value":1491},"(no path)",{"type":42,"tag":794,"props":1493,"children":1494},{},[1495],{"type":48,"value":1496},"Full conversation including tool calls",{"type":42,"tag":762,"props":1498,"children":1499},{},[1500,1516,1520],{"type":42,"tag":794,"props":1501,"children":1502},{},[1503,1509,1510],{"type":42,"tag":73,"props":1504,"children":1506},{"className":1505},[],[1507],{"type":48,"value":1508},"input",{"type":48,"value":591},{"type":42,"tag":73,"props":1511,"children":1513},{"className":1512},[],[1514],{"type":48,"value":1515},"output",{"type":42,"tag":794,"props":1517,"children":1518},{},[1519],{"type":48,"value":812},{"type":42,"tag":794,"props":1521,"children":1522},{},[1523],{"type":48,"value":1524},"Span I\u002FO",{"type":42,"tag":762,"props":1526,"children":1527},{},[1528,1537,1541],{"type":42,"tag":794,"props":1529,"children":1530},{},[1531],{"type":42,"tag":73,"props":1532,"children":1534},{"className":1533},[],[1535],{"type":48,"value":1536},"documents",{"type":42,"tag":794,"props":1538,"children":1539},{},[1540],{"type":48,"value":812},{"type":42,"tag":794,"props":1542,"children":1543},{},[1544],{"type":48,"value":1545},"Retrieved documents (RAG apps)",{"type":42,"tag":762,"props":1547,"children":1548},{},[1549,1558,1562],{"type":42,"tag":794,"props":1550,"children":1551},{},[1552],{"type":42,"tag":73,"props":1553,"children":1555},{"className":1554},[],[1556],{"type":48,"value":1557},"metadata",{"type":42,"tag":794,"props":1559,"children":1560},{},[1561],{"type":48,"value":812},{"type":42,"tag":794,"props":1563,"children":1564},{},[1565],{"type":48,"value":1566},"Custom metadata (prompt versions, feature flags, user segments)",{"type":42,"tag":747,"props":1568,"children":1570},{"id":1569},"how-to-use-search_llmobs_spans",[1571,1573],{"type":48,"value":1572},"How to Use ",{"type":42,"tag":73,"props":1574,"children":1576},{"className":1575},[],[1577],{"type":48,"value":1145},{"type":42,"tag":51,"props":1579,"children":1580},{},[1581,1583,1589,1591,1597,1598,1604,1605,1610,1612,1618,1620,1625,1627,1633],{"type":48,"value":1582},"Additional filters combine with space (AND): ",{"type":42,"tag":73,"props":1584,"children":1586},{"className":1585},[],[1587],{"type":48,"value":1588},"@status:error @ml_app:my-app",{"type":48,"value":1590},". Dedicated params (",{"type":42,"tag":73,"props":1592,"children":1594},{"className":1593},[],[1595],{"type":48,"value":1596},"span_kind",{"type":48,"value":251},{"type":42,"tag":73,"props":1599,"children":1601},{"className":1600},[],[1602],{"type":48,"value":1603},"root_spans_only",{"type":48,"value":251},{"type":42,"tag":73,"props":1606,"children":1608},{"className":1607},[],[1609],{"type":48,"value":802},{"type":48,"value":1611},") work alongside ",{"type":42,"tag":73,"props":1613,"children":1615},{"className":1614},[],[1616],{"type":48,"value":1617},"query",{"type":48,"value":1619},", but ",{"type":42,"tag":73,"props":1621,"children":1623},{"className":1622},[],[1624],{"type":48,"value":1617},{"type":48,"value":1626}," takes precedence over ",{"type":42,"tag":73,"props":1628,"children":1630},{"className":1629},[],[1631],{"type":48,"value":1632},"tags",{"type":48,"value":332},{"type":42,"tag":51,"props":1635,"children":1636},{},[1637,1639,1645,1647,1652],{"type":48,"value":1638},"To find spans with a specific eval: ",{"type":42,"tag":73,"props":1640,"children":1642},{"className":1641},[],[1643],{"type":48,"value":1644},"@evaluations.custom.\u003Ceval_name>:*",{"type":48,"value":1646}," — you can only query for eval ",{"type":42,"tag":462,"props":1648,"children":1649},{},[1650],{"type":48,"value":1651},"presence",{"type":48,"value":1653},", not specific results.",{"type":42,"tag":51,"props":1655,"children":1656},{},[1657,1659,1664,1666,1672,1674,1679,1680,1685,1687,1693],{"type":48,"value":1658},"To detect whether the app uses ",{"type":42,"tag":55,"props":1660,"children":1661},{},[1662],{"type":48,"value":1663},"sessions",{"type":48,"value":1665},": ",{"type":42,"tag":73,"props":1667,"children":1669},{"className":1668},[],[1670],{"type":48,"value":1671},"session_id:*",{"type":48,"value":1673}," matches any span carrying a ",{"type":42,"tag":73,"props":1675,"children":1677},{"className":1676},[],[1678],{"type":48,"value":545},{"type":48,"value":497},{"type":42,"tag":73,"props":1681,"children":1683},{"className":1682},[],[1684],{"type":48,"value":545},{"type":48,"value":1686}," is a first-class field — no ",{"type":42,"tag":73,"props":1688,"children":1690},{"className":1689},[],[1691],{"type":48,"value":1692},"@",{"type":48,"value":1694}," prefix). The Phase 1 session probe uses this to gate session-scope evaluators.",{"type":42,"tag":747,"props":1696,"children":1698},{"id":1697},"parallelization-rules",[1699],{"type":48,"value":1700},"Parallelization Rules",{"type":42,"tag":63,"props":1702,"children":1703},{},[1704,1723,1736,1755],{"type":42,"tag":67,"props":1705,"children":1706},{},[1707,1715,1717,1722],{"type":42,"tag":55,"props":1708,"children":1709},{},[1710],{"type":42,"tag":73,"props":1711,"children":1713},{"className":1712},[],[1714],{"type":48,"value":1162},{"type":48,"value":1716},": Group span_ids by trace_id. One call per trace_id with ALL its span_ids. Issue ALL calls for a page in a ",{"type":42,"tag":55,"props":1718,"children":1719},{},[1720],{"type":48,"value":1721},"single message",{"type":48,"value":332},{"type":42,"tag":67,"props":1724,"children":1725},{},[1726,1734],{"type":42,"tag":55,"props":1727,"children":1728},{},[1729],{"type":42,"tag":73,"props":1730,"children":1732},{"className":1731},[],[1733],{"type":48,"value":1187},{"type":48,"value":1735},": Each call is independent — always issue ALL in a single message.",{"type":42,"tag":67,"props":1737,"children":1738},{},[1739,1753],{"type":42,"tag":55,"props":1740,"children":1741},{},[1742,1747,1748],{"type":42,"tag":73,"props":1743,"children":1745},{"className":1744},[],[1746],{"type":48,"value":1212},{"type":48,"value":591},{"type":42,"tag":73,"props":1749,"children":1751},{"className":1750},[],[1752],{"type":48,"value":1229},{"type":48,"value":1754},": Parallelize across different traces in a single message.",{"type":42,"tag":67,"props":1756,"children":1757},{},[1758,1763,1765,1770],{"type":42,"tag":55,"props":1759,"children":1760},{},[1761],{"type":48,"value":1762},"Pipeline parallelism",{"type":48,"value":1764},": Start ",{"type":42,"tag":73,"props":1766,"children":1768},{"className":1767},[],[1769],{"type":48,"value":1162},{"type":48,"value":1771}," for page 1 results immediately — don't wait to collect all pages.",{"type":42,"tag":1773,"props":1774,"children":1775},"hr",{},[],{"type":42,"tag":43,"props":1777,"children":1779},{"id":1778},"evaluator-sdk-reference",[1780],{"type":48,"value":1781},"Evaluator SDK Reference",{"type":42,"tag":159,"props":1783,"children":1784},{},[1785],{"type":42,"tag":51,"props":1786,"children":1787},{},[1788,1800,1802,1807],{"type":42,"tag":55,"props":1789,"children":1790},{},[1791,1793,1798],{"type":48,"value":1792},"Applies to ",{"type":42,"tag":73,"props":1794,"children":1796},{"className":1795},[],[1797],{"type":48,"value":559},{"type":48,"value":1799}," mode only.",{"type":48,"value":1801}," In ",{"type":42,"tag":73,"props":1803,"children":1805},{"className":1804},[],[1806],{"type":48,"value":618},{"type":48,"value":1808}," mode, use this section as domain context when writing rubric prompts — no SDK classes are emitted.",{"type":42,"tag":747,"props":1810,"children":1812},{"id":1811},"imports",[1813],{"type":48,"value":1814},"Imports",{"type":42,"tag":730,"props":1816,"children":1820},{"className":1817,"code":1818,"language":1819,"meta":738,"style":738},"language-python shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Core classes\nfrom ddtrace.llmobs._experiment import BaseEvaluator, EvaluatorContext, EvaluatorResult\n\n# LLM-as-judge\nfrom ddtrace.llmobs._evaluators.llm_judge import (\n    LLMJudge,\n    BooleanStructuredOutput,\n    ScoreStructuredOutput,\n    CategoricalStructuredOutput,\n)\n\n# Built-in evaluators (use only if needed)\nfrom ddtrace.llmobs._evaluators.format import JSONEvaluator, LengthEvaluator\nfrom ddtrace.llmobs._evaluators.string_matching import StringCheckEvaluator, RegexMatchEvaluator\n","python",[1821],{"type":42,"tag":73,"props":1822,"children":1823},{"__ignoreMap":738},[1824,1835,1844,1854,1863,1872,1881,1890,1899,1908,1917,1925,1934,1943],{"type":42,"tag":1825,"props":1826,"children":1829},"span",{"class":1827,"line":1828},"line",1,[1830],{"type":42,"tag":1825,"props":1831,"children":1832},{},[1833],{"type":48,"value":1834},"# Core classes\n",{"type":42,"tag":1825,"props":1836,"children":1838},{"class":1827,"line":1837},2,[1839],{"type":42,"tag":1825,"props":1840,"children":1841},{},[1842],{"type":48,"value":1843},"from ddtrace.llmobs._experiment import BaseEvaluator, EvaluatorContext, EvaluatorResult\n",{"type":42,"tag":1825,"props":1845,"children":1847},{"class":1827,"line":1846},3,[1848],{"type":42,"tag":1825,"props":1849,"children":1851},{"emptyLinePlaceholder":1850},true,[1852],{"type":48,"value":1853},"\n",{"type":42,"tag":1825,"props":1855,"children":1857},{"class":1827,"line":1856},4,[1858],{"type":42,"tag":1825,"props":1859,"children":1860},{},[1861],{"type":48,"value":1862},"# LLM-as-judge\n",{"type":42,"tag":1825,"props":1864,"children":1866},{"class":1827,"line":1865},5,[1867],{"type":42,"tag":1825,"props":1868,"children":1869},{},[1870],{"type":48,"value":1871},"from ddtrace.llmobs._evaluators.llm_judge import (\n",{"type":42,"tag":1825,"props":1873,"children":1875},{"class":1827,"line":1874},6,[1876],{"type":42,"tag":1825,"props":1877,"children":1878},{},[1879],{"type":48,"value":1880},"    LLMJudge,\n",{"type":42,"tag":1825,"props":1882,"children":1884},{"class":1827,"line":1883},7,[1885],{"type":42,"tag":1825,"props":1886,"children":1887},{},[1888],{"type":48,"value":1889},"    BooleanStructuredOutput,\n",{"type":42,"tag":1825,"props":1891,"children":1893},{"class":1827,"line":1892},8,[1894],{"type":42,"tag":1825,"props":1895,"children":1896},{},[1897],{"type":48,"value":1898},"    ScoreStructuredOutput,\n",{"type":42,"tag":1825,"props":1900,"children":1902},{"class":1827,"line":1901},9,[1903],{"type":42,"tag":1825,"props":1904,"children":1905},{},[1906],{"type":48,"value":1907},"    CategoricalStructuredOutput,\n",{"type":42,"tag":1825,"props":1909,"children":1911},{"class":1827,"line":1910},10,[1912],{"type":42,"tag":1825,"props":1913,"children":1914},{},[1915],{"type":48,"value":1916},")\n",{"type":42,"tag":1825,"props":1918,"children":1920},{"class":1827,"line":1919},11,[1921],{"type":42,"tag":1825,"props":1922,"children":1923},{"emptyLinePlaceholder":1850},[1924],{"type":48,"value":1853},{"type":42,"tag":1825,"props":1926,"children":1928},{"class":1827,"line":1927},12,[1929],{"type":42,"tag":1825,"props":1930,"children":1931},{},[1932],{"type":48,"value":1933},"# Built-in evaluators (use only if needed)\n",{"type":42,"tag":1825,"props":1935,"children":1937},{"class":1827,"line":1936},13,[1938],{"type":42,"tag":1825,"props":1939,"children":1940},{},[1941],{"type":48,"value":1942},"from ddtrace.llmobs._evaluators.format import JSONEvaluator, LengthEvaluator\n",{"type":42,"tag":1825,"props":1944,"children":1946},{"class":1827,"line":1945},14,[1947],{"type":42,"tag":1825,"props":1948,"children":1949},{},[1950],{"type":48,"value":1951},"from ddtrace.llmobs._evaluators.string_matching import StringCheckEvaluator, RegexMatchEvaluator\n",{"type":42,"tag":51,"props":1953,"children":1954},{},[1955],{"type":48,"value":1956},"Only import what the generated file actually uses.",{"type":42,"tag":747,"props":1958,"children":1960},{"id":1959},"evaluatorcontext-what-evaluate-receives",[1961,1963,1969],{"type":48,"value":1962},"EvaluatorContext (what ",{"type":42,"tag":73,"props":1964,"children":1966},{"className":1965},[],[1967],{"type":48,"value":1968},"evaluate()",{"type":48,"value":1970}," receives)",{"type":42,"tag":730,"props":1972,"children":1974},{"className":1817,"code":1973,"language":1819,"meta":738,"style":738},"@dataclass(frozen=True)\nclass EvaluatorContext:\n    input_data: dict[str, Any]          # Task inputs (from dataset record, NOT from span)\n    output_data: Any                     # Task output (from task function return, NOT from span)\n    expected_output: Optional[JSONType] = None  # Ground truth (if available)\n    metadata: dict[str, Any] = {}        # Additional metadata\n    span_id: Optional[str] = None        # LLMObs span ID\n    trace_id: Optional[str] = None       # LLMObs trace ID\n",[1975],{"type":42,"tag":73,"props":1976,"children":1977},{"__ignoreMap":738},[1978,1986,1994,2002,2010,2018,2026,2034],{"type":42,"tag":1825,"props":1979,"children":1980},{"class":1827,"line":1828},[1981],{"type":42,"tag":1825,"props":1982,"children":1983},{},[1984],{"type":48,"value":1985},"@dataclass(frozen=True)\n",{"type":42,"tag":1825,"props":1987,"children":1988},{"class":1827,"line":1837},[1989],{"type":42,"tag":1825,"props":1990,"children":1991},{},[1992],{"type":48,"value":1993},"class EvaluatorContext:\n",{"type":42,"tag":1825,"props":1995,"children":1996},{"class":1827,"line":1846},[1997],{"type":42,"tag":1825,"props":1998,"children":1999},{},[2000],{"type":48,"value":2001},"    input_data: dict[str, Any]          # Task inputs (from dataset record, NOT from span)\n",{"type":42,"tag":1825,"props":2003,"children":2004},{"class":1827,"line":1856},[2005],{"type":42,"tag":1825,"props":2006,"children":2007},{},[2008],{"type":48,"value":2009},"    output_data: Any                     # Task output (from task function return, NOT from span)\n",{"type":42,"tag":1825,"props":2011,"children":2012},{"class":1827,"line":1865},[2013],{"type":42,"tag":1825,"props":2014,"children":2015},{},[2016],{"type":48,"value":2017},"    expected_output: Optional[JSONType] = None  # Ground truth (if available)\n",{"type":42,"tag":1825,"props":2019,"children":2020},{"class":1827,"line":1874},[2021],{"type":42,"tag":1825,"props":2022,"children":2023},{},[2024],{"type":48,"value":2025},"    metadata: dict[str, Any] = {}        # Additional metadata\n",{"type":42,"tag":1825,"props":2027,"children":2028},{"class":1827,"line":1883},[2029],{"type":42,"tag":1825,"props":2030,"children":2031},{},[2032],{"type":48,"value":2033},"    span_id: Optional[str] = None        # LLMObs span ID\n",{"type":42,"tag":1825,"props":2035,"children":2036},{"class":1827,"line":1892},[2037],{"type":42,"tag":1825,"props":2038,"children":2039},{},[2040],{"type":48,"value":2041},"    trace_id: Optional[str] = None       # LLMObs trace ID\n",{"type":42,"tag":51,"props":2043,"children":2044},{},[2045,2050,2052,2058,2059,2065,2067,2073,2074,2080,2082,2087,2089,2095,2096,2102],{"type":42,"tag":55,"props":2046,"children":2047},{},[2048],{"type":48,"value":2049},"Important — span data vs evaluator data",{"type":48,"value":2051},": When exploring production traces, you see span I\u002FO (e.g., ",{"type":42,"tag":73,"props":2053,"children":2055},{"className":2054},[],[2056],{"type":48,"value":2057},"input.value",{"type":48,"value":251},{"type":42,"tag":73,"props":2060,"children":2062},{"className":2061},[],[2063],{"type":48,"value":2064},"output.messages",{"type":48,"value":2066},"). But evaluators run in offline experiments where ",{"type":42,"tag":73,"props":2068,"children":2070},{"className":2069},[],[2071],{"type":48,"value":2072},"input_data",{"type":48,"value":919},{"type":42,"tag":73,"props":2075,"children":2077},{"className":2076},[],[2078],{"type":48,"value":2079},"output_data",{"type":48,"value":2081}," come from the user's ",{"type":42,"tag":55,"props":2083,"children":2084},{},[2085],{"type":48,"value":2086},"dataset records and task function",{"type":48,"value":2088},", not from spans. The dataset schema is user-defined and may not match span structure. Write evaluator prompts with generic ",{"type":42,"tag":73,"props":2090,"children":2092},{"className":2091},[],[2093],{"type":48,"value":2094},"{{input_data}}",{"type":48,"value":591},{"type":42,"tag":73,"props":2097,"children":2099},{"className":2098},[],[2100],{"type":48,"value":2101},"{{output_data}}",{"type":48,"value":2103}," placeholders and add comments describing what data the evaluator was designed for, so the user can adapt to their dataset shape.",{"type":42,"tag":747,"props":2105,"children":2107},{"id":2106},"evaluatorresult-what-evaluate-returns",[2108,2110,2115],{"type":48,"value":2109},"EvaluatorResult (what ",{"type":42,"tag":73,"props":2111,"children":2113},{"className":2112},[],[2114],{"type":48,"value":1968},{"type":48,"value":2116}," returns)",{"type":42,"tag":730,"props":2118,"children":2120},{"className":1817,"code":2119,"language":1819,"meta":738,"style":738},"EvaluatorResult(\n    value=...,                    # Required. JSONType (str, int, float, bool, None, list, dict)\n    reasoning=\"...\",              # Optional. Explanation string\n    assessment=\"pass\" or \"fail\",  # Optional. Pass\u002Ffail assessment\n    metadata={...},              # Optional. Evaluation metadata dict\n    tags={...},                  # Optional. Tags dict\n)\n",[2121],{"type":42,"tag":73,"props":2122,"children":2123},{"__ignoreMap":738},[2124,2132,2140,2148,2156,2164,2172],{"type":42,"tag":1825,"props":2125,"children":2126},{"class":1827,"line":1828},[2127],{"type":42,"tag":1825,"props":2128,"children":2129},{},[2130],{"type":48,"value":2131},"EvaluatorResult(\n",{"type":42,"tag":1825,"props":2133,"children":2134},{"class":1827,"line":1837},[2135],{"type":42,"tag":1825,"props":2136,"children":2137},{},[2138],{"type":48,"value":2139},"    value=...,                    # Required. JSONType (str, int, float, bool, None, list, dict)\n",{"type":42,"tag":1825,"props":2141,"children":2142},{"class":1827,"line":1846},[2143],{"type":42,"tag":1825,"props":2144,"children":2145},{},[2146],{"type":48,"value":2147},"    reasoning=\"...\",              # Optional. Explanation string\n",{"type":42,"tag":1825,"props":2149,"children":2150},{"class":1827,"line":1856},[2151],{"type":42,"tag":1825,"props":2152,"children":2153},{},[2154],{"type":48,"value":2155},"    assessment=\"pass\" or \"fail\",  # Optional. Pass\u002Ffail assessment\n",{"type":42,"tag":1825,"props":2157,"children":2158},{"class":1827,"line":1865},[2159],{"type":42,"tag":1825,"props":2160,"children":2161},{},[2162],{"type":48,"value":2163},"    metadata={...},              # Optional. Evaluation metadata dict\n",{"type":42,"tag":1825,"props":2165,"children":2166},{"class":1827,"line":1874},[2167],{"type":42,"tag":1825,"props":2168,"children":2169},{},[2170],{"type":48,"value":2171},"    tags={...},                  # Optional. Tags dict\n",{"type":42,"tag":1825,"props":2173,"children":2174},{"class":1827,"line":1883},[2175],{"type":42,"tag":1825,"props":2176,"children":2177},{},[2178],{"type":48,"value":1916},{"type":42,"tag":747,"props":2180,"children":2182},{"id":2181},"llmjudge-llm-as-judge-evaluator",[2183],{"type":48,"value":2184},"LLMJudge — LLM-as-Judge Evaluator",{"type":42,"tag":730,"props":2186,"children":2188},{"className":1817,"code":2187,"language":1819,"meta":738,"style":738},"judge = LLMJudge(\n    user_prompt=\"...\",              # Required. Supports {{template_vars}}\n    system_prompt=\"...\",            # Optional. Does NOT support template vars\n    structured_output=...,          # Optional. Boolean\u002FScore\u002FCategorical output, or a dict for custom JSON schema\n    provider=\"openai\",              # \"openai\" | \"anthropic\" | \"azure_openai\" | \"vertexai\" | \"bedrock\"\n    model=\"gpt-4o\",                # Model identifier\n    model_params={\"temperature\": 0.0},  # Optional. Passed to LLM API\n    name=\"eval_name\",              # Optional. Must match ^[a-zA-Z0-9_-]+$\n)\n",[2189],{"type":42,"tag":73,"props":2190,"children":2191},{"__ignoreMap":738},[2192,2200,2208,2216,2224,2232,2240,2248,2256],{"type":42,"tag":1825,"props":2193,"children":2194},{"class":1827,"line":1828},[2195],{"type":42,"tag":1825,"props":2196,"children":2197},{},[2198],{"type":48,"value":2199},"judge = LLMJudge(\n",{"type":42,"tag":1825,"props":2201,"children":2202},{"class":1827,"line":1837},[2203],{"type":42,"tag":1825,"props":2204,"children":2205},{},[2206],{"type":48,"value":2207},"    user_prompt=\"...\",              # Required. Supports {{template_vars}}\n",{"type":42,"tag":1825,"props":2209,"children":2210},{"class":1827,"line":1846},[2211],{"type":42,"tag":1825,"props":2212,"children":2213},{},[2214],{"type":48,"value":2215},"    system_prompt=\"...\",            # Optional. Does NOT support template vars\n",{"type":42,"tag":1825,"props":2217,"children":2218},{"class":1827,"line":1856},[2219],{"type":42,"tag":1825,"props":2220,"children":2221},{},[2222],{"type":48,"value":2223},"    structured_output=...,          # Optional. Boolean\u002FScore\u002FCategorical output, or a dict for custom JSON schema\n",{"type":42,"tag":1825,"props":2225,"children":2226},{"class":1827,"line":1865},[2227],{"type":42,"tag":1825,"props":2228,"children":2229},{},[2230],{"type":48,"value":2231},"    provider=\"openai\",              # \"openai\" | \"anthropic\" | \"azure_openai\" | \"vertexai\" | \"bedrock\"\n",{"type":42,"tag":1825,"props":2233,"children":2234},{"class":1827,"line":1874},[2235],{"type":42,"tag":1825,"props":2236,"children":2237},{},[2238],{"type":48,"value":2239},"    model=\"gpt-4o\",                # Model identifier\n",{"type":42,"tag":1825,"props":2241,"children":2242},{"class":1827,"line":1883},[2243],{"type":42,"tag":1825,"props":2244,"children":2245},{},[2246],{"type":48,"value":2247},"    model_params={\"temperature\": 0.0},  # Optional. Passed to LLM API\n",{"type":42,"tag":1825,"props":2249,"children":2250},{"class":1827,"line":1892},[2251],{"type":42,"tag":1825,"props":2252,"children":2253},{},[2254],{"type":48,"value":2255},"    name=\"eval_name\",              # Optional. Must match ^[a-zA-Z0-9_-]+$\n",{"type":42,"tag":1825,"props":2257,"children":2258},{"class":1827,"line":1901},[2259],{"type":42,"tag":1825,"props":2260,"children":2261},{},[2262],{"type":48,"value":1916},{"type":42,"tag":51,"props":2264,"children":2265},{},[2266,2271,2273,2279,2280,2285,2286,2291,2292,2298,2299,2305,2307,2313],{"type":42,"tag":55,"props":2267,"children":2268},{},[2269],{"type":48,"value":2270},"Template variables",{"type":48,"value":2272}," in ",{"type":42,"tag":73,"props":2274,"children":2276},{"className":2275},[],[2277],{"type":48,"value":2278},"user_prompt",{"type":48,"value":1665},{"type":42,"tag":73,"props":2281,"children":2283},{"className":2282},[],[2284],{"type":48,"value":2094},{"type":48,"value":251},{"type":42,"tag":73,"props":2287,"children":2289},{"className":2288},[],[2290],{"type":48,"value":2101},{"type":48,"value":251},{"type":42,"tag":73,"props":2293,"children":2295},{"className":2294},[],[2296],{"type":48,"value":2297},"{{expected_output}}",{"type":48,"value":251},{"type":42,"tag":73,"props":2300,"children":2302},{"className":2301},[],[2303],{"type":48,"value":2304},"{{metadata.key}}",{"type":48,"value":2306}," — resolved from ",{"type":42,"tag":73,"props":2308,"children":2310},{"className":2309},[],[2311],{"type":48,"value":2312},"EvaluatorContext",{"type":48,"value":2314}," fields via dot-path into nested dicts.",{"type":42,"tag":747,"props":2316,"children":2318},{"id":2317},"structured-output-types",[2319],{"type":48,"value":2320},"Structured Output Types",{"type":42,"tag":51,"props":2322,"children":2323},{},[2324,2329],{"type":42,"tag":55,"props":2325,"children":2326},{},[2327],{"type":48,"value":2328},"Boolean",{"type":48,"value":2330}," — true\u002Ffalse with optional pass\u002Ffail:",{"type":42,"tag":730,"props":2332,"children":2334},{"className":1817,"code":2333,"language":1819,"meta":738,"style":738},"BooleanStructuredOutput(\n    description=\"Whether the response is factually accurate\",\n    reasoning=True,                    # Include reasoning field in LLM response\n    reasoning_description=None,        # Optional custom description for reasoning field\n    pass_when=True,                    # True → pass when true, False → pass when false, None → no assessment\n)\n",[2335],{"type":42,"tag":73,"props":2336,"children":2337},{"__ignoreMap":738},[2338,2346,2354,2362,2370,2378],{"type":42,"tag":1825,"props":2339,"children":2340},{"class":1827,"line":1828},[2341],{"type":42,"tag":1825,"props":2342,"children":2343},{},[2344],{"type":48,"value":2345},"BooleanStructuredOutput(\n",{"type":42,"tag":1825,"props":2347,"children":2348},{"class":1827,"line":1837},[2349],{"type":42,"tag":1825,"props":2350,"children":2351},{},[2352],{"type":48,"value":2353},"    description=\"Whether the response is factually accurate\",\n",{"type":42,"tag":1825,"props":2355,"children":2356},{"class":1827,"line":1846},[2357],{"type":42,"tag":1825,"props":2358,"children":2359},{},[2360],{"type":48,"value":2361},"    reasoning=True,                    # Include reasoning field in LLM response\n",{"type":42,"tag":1825,"props":2363,"children":2364},{"class":1827,"line":1856},[2365],{"type":42,"tag":1825,"props":2366,"children":2367},{},[2368],{"type":48,"value":2369},"    reasoning_description=None,        # Optional custom description for reasoning field\n",{"type":42,"tag":1825,"props":2371,"children":2372},{"class":1827,"line":1865},[2373],{"type":42,"tag":1825,"props":2374,"children":2375},{},[2376],{"type":48,"value":2377},"    pass_when=True,                    # True → pass when true, False → pass when false, None → no assessment\n",{"type":42,"tag":1825,"props":2379,"children":2380},{"class":1827,"line":1874},[2381],{"type":42,"tag":1825,"props":2382,"children":2383},{},[2384],{"type":48,"value":1916},{"type":42,"tag":51,"props":2386,"children":2387},{},[2388,2393],{"type":42,"tag":55,"props":2389,"children":2390},{},[2391],{"type":48,"value":2392},"Score",{"type":48,"value":2394}," — numeric within a range with optional thresholds:",{"type":42,"tag":730,"props":2396,"children":2398},{"className":1817,"code":2397,"language":1819,"meta":738,"style":738},"ScoreStructuredOutput(\n    description=\"Helpfulness score\",\n    min_score=1,                       # Minimum possible score\n    max_score=10,                      # Maximum possible score\n    reasoning=True,\n    reasoning_description=None,\n    min_threshold=7,                   # Scores >= 7 pass (optional)\n    max_threshold=None,                # Scores \u003C= N pass (optional)\n)\n",[2399],{"type":42,"tag":73,"props":2400,"children":2401},{"__ignoreMap":738},[2402,2410,2418,2426,2434,2442,2450,2458,2466],{"type":42,"tag":1825,"props":2403,"children":2404},{"class":1827,"line":1828},[2405],{"type":42,"tag":1825,"props":2406,"children":2407},{},[2408],{"type":48,"value":2409},"ScoreStructuredOutput(\n",{"type":42,"tag":1825,"props":2411,"children":2412},{"class":1827,"line":1837},[2413],{"type":42,"tag":1825,"props":2414,"children":2415},{},[2416],{"type":48,"value":2417},"    description=\"Helpfulness score\",\n",{"type":42,"tag":1825,"props":2419,"children":2420},{"class":1827,"line":1846},[2421],{"type":42,"tag":1825,"props":2422,"children":2423},{},[2424],{"type":48,"value":2425},"    min_score=1,                       # Minimum possible score\n",{"type":42,"tag":1825,"props":2427,"children":2428},{"class":1827,"line":1856},[2429],{"type":42,"tag":1825,"props":2430,"children":2431},{},[2432],{"type":48,"value":2433},"    max_score=10,                      # Maximum possible score\n",{"type":42,"tag":1825,"props":2435,"children":2436},{"class":1827,"line":1865},[2437],{"type":42,"tag":1825,"props":2438,"children":2439},{},[2440],{"type":48,"value":2441},"    reasoning=True,\n",{"type":42,"tag":1825,"props":2443,"children":2444},{"class":1827,"line":1874},[2445],{"type":42,"tag":1825,"props":2446,"children":2447},{},[2448],{"type":48,"value":2449},"    reasoning_description=None,\n",{"type":42,"tag":1825,"props":2451,"children":2452},{"class":1827,"line":1883},[2453],{"type":42,"tag":1825,"props":2454,"children":2455},{},[2456],{"type":48,"value":2457},"    min_threshold=7,                   # Scores >= 7 pass (optional)\n",{"type":42,"tag":1825,"props":2459,"children":2460},{"class":1827,"line":1892},[2461],{"type":42,"tag":1825,"props":2462,"children":2463},{},[2464],{"type":48,"value":2465},"    max_threshold=None,                # Scores \u003C= N pass (optional)\n",{"type":42,"tag":1825,"props":2467,"children":2468},{"class":1827,"line":1901},[2469],{"type":42,"tag":1825,"props":2470,"children":2471},{},[2472],{"type":48,"value":1916},{"type":42,"tag":51,"props":2474,"children":2475},{},[2476,2481],{"type":42,"tag":55,"props":2477,"children":2478},{},[2479],{"type":48,"value":2480},"Categorical",{"type":48,"value":2482}," — select from predefined categories:",{"type":42,"tag":730,"props":2484,"children":2486},{"className":1817,"code":2485,"language":1819,"meta":738,"style":738},"CategoricalStructuredOutput(\n    categories={\n        \"correct\": \"The response correctly answers the question\",\n        \"partially_correct\": \"The response is partially correct but missing key information\",\n        \"incorrect\": \"The response is factually wrong or irrelevant\",\n    },\n    reasoning=True,\n    reasoning_description=None,\n    pass_values=[\"correct\"],           # Which categories count as passing (optional)\n)\n",[2487],{"type":42,"tag":73,"props":2488,"children":2489},{"__ignoreMap":738},[2490,2498,2506,2514,2522,2530,2538,2545,2552,2560],{"type":42,"tag":1825,"props":2491,"children":2492},{"class":1827,"line":1828},[2493],{"type":42,"tag":1825,"props":2494,"children":2495},{},[2496],{"type":48,"value":2497},"CategoricalStructuredOutput(\n",{"type":42,"tag":1825,"props":2499,"children":2500},{"class":1827,"line":1837},[2501],{"type":42,"tag":1825,"props":2502,"children":2503},{},[2504],{"type":48,"value":2505},"    categories={\n",{"type":42,"tag":1825,"props":2507,"children":2508},{"class":1827,"line":1846},[2509],{"type":42,"tag":1825,"props":2510,"children":2511},{},[2512],{"type":48,"value":2513},"        \"correct\": \"The response correctly answers the question\",\n",{"type":42,"tag":1825,"props":2515,"children":2516},{"class":1827,"line":1856},[2517],{"type":42,"tag":1825,"props":2518,"children":2519},{},[2520],{"type":48,"value":2521},"        \"partially_correct\": \"The response is partially correct but missing key information\",\n",{"type":42,"tag":1825,"props":2523,"children":2524},{"class":1827,"line":1865},[2525],{"type":42,"tag":1825,"props":2526,"children":2527},{},[2528],{"type":48,"value":2529},"        \"incorrect\": \"The response is factually wrong or irrelevant\",\n",{"type":42,"tag":1825,"props":2531,"children":2532},{"class":1827,"line":1874},[2533],{"type":42,"tag":1825,"props":2534,"children":2535},{},[2536],{"type":48,"value":2537},"    },\n",{"type":42,"tag":1825,"props":2539,"children":2540},{"class":1827,"line":1883},[2541],{"type":42,"tag":1825,"props":2542,"children":2543},{},[2544],{"type":48,"value":2441},{"type":42,"tag":1825,"props":2546,"children":2547},{"class":1827,"line":1892},[2548],{"type":42,"tag":1825,"props":2549,"children":2550},{},[2551],{"type":48,"value":2449},{"type":42,"tag":1825,"props":2553,"children":2554},{"class":1827,"line":1901},[2555],{"type":42,"tag":1825,"props":2556,"children":2557},{},[2558],{"type":48,"value":2559},"    pass_values=[\"correct\"],           # Which categories count as passing (optional)\n",{"type":42,"tag":1825,"props":2561,"children":2562},{"class":1827,"line":1910},[2563],{"type":42,"tag":1825,"props":2564,"children":2565},{},[2566],{"type":48,"value":1916},{"type":42,"tag":51,"props":2568,"children":2569},{},[2570,2575],{"type":42,"tag":55,"props":2571,"children":2572},{},[2573],{"type":48,"value":2574},"Custom JSON schema",{"type":48,"value":2576}," — arbitrary structured responses for multi-dimensional evals:",{"type":42,"tag":730,"props":2578,"children":2580},{"className":1817,"code":2579,"language":1819,"meta":738,"style":738},"# Pass a raw dict as structured_output — used as the JSON schema directly\nstructured_output={\n    \"type\": \"object\",\n    \"properties\": {\n        \"relevance\": {\"type\": \"boolean\", \"description\": \"Whether the response addresses the question\"},\n        \"confidence\": {\"type\": \"number\", \"description\": \"Confidence score (0.0 to 1.0)\"},\n        \"reasoning\": {\"type\": \"string\", \"description\": \"Explanation for the evaluation\"},\n    },\n    \"required\": [\"relevance\", \"confidence\", \"reasoning\"],\n    \"additionalProperties\": False,\n}\n",[2581],{"type":42,"tag":73,"props":2582,"children":2583},{"__ignoreMap":738},[2584,2592,2600,2608,2616,2624,2632,2640,2647,2655,2663],{"type":42,"tag":1825,"props":2585,"children":2586},{"class":1827,"line":1828},[2587],{"type":42,"tag":1825,"props":2588,"children":2589},{},[2590],{"type":48,"value":2591},"# Pass a raw dict as structured_output — used as the JSON schema directly\n",{"type":42,"tag":1825,"props":2593,"children":2594},{"class":1827,"line":1837},[2595],{"type":42,"tag":1825,"props":2596,"children":2597},{},[2598],{"type":48,"value":2599},"structured_output={\n",{"type":42,"tag":1825,"props":2601,"children":2602},{"class":1827,"line":1846},[2603],{"type":42,"tag":1825,"props":2604,"children":2605},{},[2606],{"type":48,"value":2607},"    \"type\": \"object\",\n",{"type":42,"tag":1825,"props":2609,"children":2610},{"class":1827,"line":1856},[2611],{"type":42,"tag":1825,"props":2612,"children":2613},{},[2614],{"type":48,"value":2615},"    \"properties\": {\n",{"type":42,"tag":1825,"props":2617,"children":2618},{"class":1827,"line":1865},[2619],{"type":42,"tag":1825,"props":2620,"children":2621},{},[2622],{"type":48,"value":2623},"        \"relevance\": {\"type\": \"boolean\", \"description\": \"Whether the response addresses the question\"},\n",{"type":42,"tag":1825,"props":2625,"children":2626},{"class":1827,"line":1874},[2627],{"type":42,"tag":1825,"props":2628,"children":2629},{},[2630],{"type":48,"value":2631},"        \"confidence\": {\"type\": \"number\", \"description\": \"Confidence score (0.0 to 1.0)\"},\n",{"type":42,"tag":1825,"props":2633,"children":2634},{"class":1827,"line":1883},[2635],{"type":42,"tag":1825,"props":2636,"children":2637},{},[2638],{"type":48,"value":2639},"        \"reasoning\": {\"type\": \"string\", \"description\": \"Explanation for the evaluation\"},\n",{"type":42,"tag":1825,"props":2641,"children":2642},{"class":1827,"line":1892},[2643],{"type":42,"tag":1825,"props":2644,"children":2645},{},[2646],{"type":48,"value":2537},{"type":42,"tag":1825,"props":2648,"children":2649},{"class":1827,"line":1901},[2650],{"type":42,"tag":1825,"props":2651,"children":2652},{},[2653],{"type":48,"value":2654},"    \"required\": [\"relevance\", \"confidence\", \"reasoning\"],\n",{"type":42,"tag":1825,"props":2656,"children":2657},{"class":1827,"line":1910},[2658],{"type":42,"tag":1825,"props":2659,"children":2660},{},[2661],{"type":48,"value":2662},"    \"additionalProperties\": False,\n",{"type":42,"tag":1825,"props":2664,"children":2665},{"class":1827,"line":1919},[2666],{"type":42,"tag":1825,"props":2667,"children":2668},{},[2669],{"type":48,"value":2670},"}\n",{"type":42,"tag":51,"props":2672,"children":2673},{},[2674,2676,2682,2684,2690,2692,2698,2700,2706,2707,2713,2715,2721,2723,2729,2731,2737],{"type":48,"value":2675},"Always write standard JSON schema — the SDK adapts it per provider automatically (e.g., Anthropic doesn't support ",{"type":42,"tag":73,"props":2677,"children":2679},{"className":2678},[],[2680],{"type":48,"value":2681},"minimum",{"type":48,"value":2683},"\u002F",{"type":42,"tag":73,"props":2685,"children":2687},{"className":2686},[],[2688],{"type":48,"value":2689},"maximum",{"type":48,"value":2691}," on number fields, so the SDK moves range constraints into the ",{"type":42,"tag":73,"props":2693,"children":2695},{"className":2694},[],[2696],{"type":48,"value":2697},"description",{"type":48,"value":2699},"; Vertex AI converts ",{"type":42,"tag":73,"props":2701,"children":2703},{"className":2702},[],[2704],{"type":48,"value":2705},"const",{"type":48,"value":2683},{"type":42,"tag":73,"props":2708,"children":2710},{"className":2709},[],[2711],{"type":48,"value":2712},"anyOf",{"type":48,"value":2714}," to ",{"type":42,"tag":73,"props":2716,"children":2718},{"className":2717},[],[2719],{"type":48,"value":2720},"enum",{"type":48,"value":2722},"). The full parsed JSON dict becomes the eval ",{"type":42,"tag":73,"props":2724,"children":2726},{"className":2725},[],[2727],{"type":48,"value":2728},"value",{"type":48,"value":2730},"; a ",{"type":42,"tag":73,"props":2732,"children":2734},{"className":2733},[],[2735],{"type":48,"value":2736},"\"reasoning\"",{"type":48,"value":2738}," key (if present) is automatically extracted. No automatic pass\u002Ffail assessment.",{"type":42,"tag":747,"props":2740,"children":2742},{"id":2741},"llmjudge-prompt-guidelines",[2743],{"type":48,"value":2744},"LLMJudge Prompt Guidelines",{"type":42,"tag":51,"props":2746,"children":2747},{},[2748,2750,2756,2758,2763,2765,2770],{"type":48,"value":2749},"The ",{"type":42,"tag":73,"props":2751,"children":2753},{"className":2752},[],[2754],{"type":48,"value":2755},"structured_output",{"type":48,"value":2757}," parameter enforces the response format via JSON schema. ",{"type":42,"tag":55,"props":2759,"children":2760},{},[2761],{"type":48,"value":2762},"Do not",{"type":48,"value":2764}," prescribe the format in the prompt (no \"Answer YES\u002FNO\", \"Rate 1-10\", etc.). Instead, describe the ",{"type":42,"tag":55,"props":2766,"children":2767},{},[2768],{"type":48,"value":2769},"evaluation criteria",{"type":48,"value":2771}," and let the structured output handle the format.",{"type":42,"tag":194,"props":2773,"children":2774},{},[2775,2785],{"type":42,"tag":67,"props":2776,"children":2777},{},[2778,2783],{"type":42,"tag":55,"props":2779,"children":2780},{},[2781],{"type":48,"value":2782},"system_prompt",{"type":48,"value":2784},": Set the judge's role and the app's domain context. Does NOT support template vars.",{"type":42,"tag":67,"props":2786,"children":2787},{},[2788,2792,2794,2799,2800,2805],{"type":42,"tag":55,"props":2789,"children":2790},{},[2791],{"type":48,"value":2278},{"type":48,"value":2793},": Present the data via ",{"type":42,"tag":73,"props":2795,"children":2797},{"className":2796},[],[2798],{"type":48,"value":2094},{"type":48,"value":591},{"type":42,"tag":73,"props":2801,"children":2803},{"className":2802},[],[2804],{"type":48,"value":2101},{"type":48,"value":2806},", then describe what good vs. bad looks like for this dimension.",{"type":42,"tag":747,"props":2808,"children":2810},{"id":2809},"baseevaluator-custom-code-based-evaluator",[2811],{"type":48,"value":2812},"BaseEvaluator — Custom Code-Based Evaluator",{"type":42,"tag":51,"props":2814,"children":2815},{},[2816],{"type":48,"value":2817},"For deterministic checks that do not need LLM judgment:",{"type":42,"tag":730,"props":2819,"children":2821},{"className":1817,"code":2820,"language":1819,"meta":738,"style":738},"class MyEvaluator(BaseEvaluator):\n    def __init__(self, name=None, ...custom_params...):\n        super().__init__(name=name)\n        self._param = ...  # Store config as private attrs\n\n    def evaluate(self, context: EvaluatorContext) -> EvaluatorResult:\n        # Access: context.input_data, context.output_data, context.expected_output, context.metadata\n        # Must NOT modify self attributes (thread safety)\n        passed = ...  # Your logic here\n        return EvaluatorResult(\n            value=passed,\n            reasoning=\"...\",\n            assessment=\"pass\" if passed else \"fail\",\n        )\n",[2822],{"type":42,"tag":73,"props":2823,"children":2824},{"__ignoreMap":738},[2825,2833,2841,2849,2857,2864,2872,2880,2888,2896,2904,2912,2920,2928],{"type":42,"tag":1825,"props":2826,"children":2827},{"class":1827,"line":1828},[2828],{"type":42,"tag":1825,"props":2829,"children":2830},{},[2831],{"type":48,"value":2832},"class MyEvaluator(BaseEvaluator):\n",{"type":42,"tag":1825,"props":2834,"children":2835},{"class":1827,"line":1837},[2836],{"type":42,"tag":1825,"props":2837,"children":2838},{},[2839],{"type":48,"value":2840},"    def __init__(self, name=None, ...custom_params...):\n",{"type":42,"tag":1825,"props":2842,"children":2843},{"class":1827,"line":1846},[2844],{"type":42,"tag":1825,"props":2845,"children":2846},{},[2847],{"type":48,"value":2848},"        super().__init__(name=name)\n",{"type":42,"tag":1825,"props":2850,"children":2851},{"class":1827,"line":1856},[2852],{"type":42,"tag":1825,"props":2853,"children":2854},{},[2855],{"type":48,"value":2856},"        self._param = ...  # Store config as private attrs\n",{"type":42,"tag":1825,"props":2858,"children":2859},{"class":1827,"line":1865},[2860],{"type":42,"tag":1825,"props":2861,"children":2862},{"emptyLinePlaceholder":1850},[2863],{"type":48,"value":1853},{"type":42,"tag":1825,"props":2865,"children":2866},{"class":1827,"line":1874},[2867],{"type":42,"tag":1825,"props":2868,"children":2869},{},[2870],{"type":48,"value":2871},"    def evaluate(self, context: EvaluatorContext) -> EvaluatorResult:\n",{"type":42,"tag":1825,"props":2873,"children":2874},{"class":1827,"line":1883},[2875],{"type":42,"tag":1825,"props":2876,"children":2877},{},[2878],{"type":48,"value":2879},"        # Access: context.input_data, context.output_data, context.expected_output, context.metadata\n",{"type":42,"tag":1825,"props":2881,"children":2882},{"class":1827,"line":1892},[2883],{"type":42,"tag":1825,"props":2884,"children":2885},{},[2886],{"type":48,"value":2887},"        # Must NOT modify self attributes (thread safety)\n",{"type":42,"tag":1825,"props":2889,"children":2890},{"class":1827,"line":1901},[2891],{"type":42,"tag":1825,"props":2892,"children":2893},{},[2894],{"type":48,"value":2895},"        passed = ...  # Your logic here\n",{"type":42,"tag":1825,"props":2897,"children":2898},{"class":1827,"line":1910},[2899],{"type":42,"tag":1825,"props":2900,"children":2901},{},[2902],{"type":48,"value":2903},"        return EvaluatorResult(\n",{"type":42,"tag":1825,"props":2905,"children":2906},{"class":1827,"line":1919},[2907],{"type":42,"tag":1825,"props":2908,"children":2909},{},[2910],{"type":48,"value":2911},"            value=passed,\n",{"type":42,"tag":1825,"props":2913,"children":2914},{"class":1827,"line":1927},[2915],{"type":42,"tag":1825,"props":2916,"children":2917},{},[2918],{"type":48,"value":2919},"            reasoning=\"...\",\n",{"type":42,"tag":1825,"props":2921,"children":2922},{"class":1827,"line":1936},[2923],{"type":42,"tag":1825,"props":2924,"children":2925},{},[2926],{"type":48,"value":2927},"            assessment=\"pass\" if passed else \"fail\",\n",{"type":42,"tag":1825,"props":2929,"children":2930},{"class":1827,"line":1945},[2931],{"type":42,"tag":1825,"props":2932,"children":2933},{},[2934],{"type":48,"value":2935},"        )\n",{"type":42,"tag":747,"props":2937,"children":2939},{"id":2938},"built-in-evaluators",[2940],{"type":48,"value":2941},"Built-in Evaluators",{"type":42,"tag":730,"props":2943,"children":2945},{"className":1817,"code":2944,"language":1819,"meta":738,"style":738},"# Validate JSON syntax + optional required keys\nJSONEvaluator(required_keys=[\"name\", \"age\"], output_extractor=None, name=None)\n\n# Validate length (characters, words, or lines)\nLengthEvaluator(count_by=\"words\", min_length=10, max_length=500, output_extractor=None, name=None)\n# count_by: \"characters\" | \"words\" | \"lines\"\n\n# String matching\nStringCheckEvaluator(operation=\"contains\", expected=\"success\", case_sensitive=False, name=None)\n# operation: \"eq\" | \"ne\" | \"contains\" | \"icontains\"\n\n# Regex matching\nRegexMatchEvaluator(pattern=r\"\\d{4}-\\d{2}-\\d{2}\", match_mode=\"search\", name=None)\n# match_mode: \"search\" | \"match\" | \"fullmatch\"\n",[2946],{"type":42,"tag":73,"props":2947,"children":2948},{"__ignoreMap":738},[2949,2957,2965,2972,2980,2988,2996,3003,3011,3019,3027,3034,3042,3050],{"type":42,"tag":1825,"props":2950,"children":2951},{"class":1827,"line":1828},[2952],{"type":42,"tag":1825,"props":2953,"children":2954},{},[2955],{"type":48,"value":2956},"# Validate JSON syntax + optional required keys\n",{"type":42,"tag":1825,"props":2958,"children":2959},{"class":1827,"line":1837},[2960],{"type":42,"tag":1825,"props":2961,"children":2962},{},[2963],{"type":48,"value":2964},"JSONEvaluator(required_keys=[\"name\", \"age\"], output_extractor=None, name=None)\n",{"type":42,"tag":1825,"props":2966,"children":2967},{"class":1827,"line":1846},[2968],{"type":42,"tag":1825,"props":2969,"children":2970},{"emptyLinePlaceholder":1850},[2971],{"type":48,"value":1853},{"type":42,"tag":1825,"props":2973,"children":2974},{"class":1827,"line":1856},[2975],{"type":42,"tag":1825,"props":2976,"children":2977},{},[2978],{"type":48,"value":2979},"# Validate length (characters, words, or lines)\n",{"type":42,"tag":1825,"props":2981,"children":2982},{"class":1827,"line":1865},[2983],{"type":42,"tag":1825,"props":2984,"children":2985},{},[2986],{"type":48,"value":2987},"LengthEvaluator(count_by=\"words\", min_length=10, max_length=500, output_extractor=None, name=None)\n",{"type":42,"tag":1825,"props":2989,"children":2990},{"class":1827,"line":1874},[2991],{"type":42,"tag":1825,"props":2992,"children":2993},{},[2994],{"type":48,"value":2995},"# count_by: \"characters\" | \"words\" | \"lines\"\n",{"type":42,"tag":1825,"props":2997,"children":2998},{"class":1827,"line":1883},[2999],{"type":42,"tag":1825,"props":3000,"children":3001},{"emptyLinePlaceholder":1850},[3002],{"type":48,"value":1853},{"type":42,"tag":1825,"props":3004,"children":3005},{"class":1827,"line":1892},[3006],{"type":42,"tag":1825,"props":3007,"children":3008},{},[3009],{"type":48,"value":3010},"# String matching\n",{"type":42,"tag":1825,"props":3012,"children":3013},{"class":1827,"line":1901},[3014],{"type":42,"tag":1825,"props":3015,"children":3016},{},[3017],{"type":48,"value":3018},"StringCheckEvaluator(operation=\"contains\", expected=\"success\", case_sensitive=False, name=None)\n",{"type":42,"tag":1825,"props":3020,"children":3021},{"class":1827,"line":1910},[3022],{"type":42,"tag":1825,"props":3023,"children":3024},{},[3025],{"type":48,"value":3026},"# operation: \"eq\" | \"ne\" | \"contains\" | \"icontains\"\n",{"type":42,"tag":1825,"props":3028,"children":3029},{"class":1827,"line":1919},[3030],{"type":42,"tag":1825,"props":3031,"children":3032},{"emptyLinePlaceholder":1850},[3033],{"type":48,"value":1853},{"type":42,"tag":1825,"props":3035,"children":3036},{"class":1827,"line":1927},[3037],{"type":42,"tag":1825,"props":3038,"children":3039},{},[3040],{"type":48,"value":3041},"# Regex matching\n",{"type":42,"tag":1825,"props":3043,"children":3044},{"class":1827,"line":1936},[3045],{"type":42,"tag":1825,"props":3046,"children":3047},{},[3048],{"type":48,"value":3049},"RegexMatchEvaluator(pattern=r\"\\d{4}-\\d{2}-\\d{2}\", match_mode=\"search\", name=None)\n",{"type":42,"tag":1825,"props":3051,"children":3052},{"class":1827,"line":1945},[3053],{"type":42,"tag":1825,"props":3054,"children":3055},{},[3056],{"type":48,"value":3057},"# match_mode: \"search\" | \"match\" | \"fullmatch\"\n",{"type":42,"tag":747,"props":3059,"children":3061},{"id":3060},"evaluator-type-decision-matrix",[3062],{"type":48,"value":3063},"Evaluator Type Decision Matrix",{"type":42,"tag":754,"props":3065,"children":3066},{},[3067,3083],{"type":42,"tag":758,"props":3068,"children":3069},{},[3070],{"type":42,"tag":762,"props":3071,"children":3072},{},[3073,3078],{"type":42,"tag":766,"props":3074,"children":3075},{},[3076],{"type":48,"value":3077},"Signal",{"type":42,"tag":766,"props":3079,"children":3080},{},[3081],{"type":48,"value":3082},"Evaluator Type",{"type":42,"tag":787,"props":3084,"children":3085},{},[3086,3103,3120,3137,3154,3178,3201,3224,3248],{"type":42,"tag":762,"props":3087,"children":3088},{},[3089,3094],{"type":42,"tag":794,"props":3090,"children":3091},{},[3092],{"type":48,"value":3093},"Output must be valid JSON",{"type":42,"tag":794,"props":3095,"children":3096},{},[3097],{"type":42,"tag":73,"props":3098,"children":3100},{"className":3099},[],[3101],{"type":48,"value":3102},"JSONEvaluator",{"type":42,"tag":762,"props":3104,"children":3105},{},[3106,3111],{"type":42,"tag":794,"props":3107,"children":3108},{},[3109],{"type":48,"value":3110},"Output must match a regex pattern",{"type":42,"tag":794,"props":3112,"children":3113},{},[3114],{"type":42,"tag":73,"props":3115,"children":3117},{"className":3116},[],[3118],{"type":48,"value":3119},"RegexMatchEvaluator",{"type":42,"tag":762,"props":3121,"children":3122},{},[3123,3128],{"type":42,"tag":794,"props":3124,"children":3125},{},[3126],{"type":48,"value":3127},"Output has length constraints",{"type":42,"tag":794,"props":3129,"children":3130},{},[3131],{"type":42,"tag":73,"props":3132,"children":3134},{"className":3133},[],[3135],{"type":48,"value":3136},"LengthEvaluator",{"type":42,"tag":762,"props":3138,"children":3139},{},[3140,3145],{"type":42,"tag":794,"props":3141,"children":3142},{},[3143],{"type":48,"value":3144},"Output must contain\u002Fnot contain specific strings",{"type":42,"tag":794,"props":3146,"children":3147},{},[3148],{"type":42,"tag":73,"props":3149,"children":3151},{"className":3150},[],[3152],{"type":48,"value":3153},"StringCheckEvaluator",{"type":42,"tag":762,"props":3155,"children":3156},{},[3157,3162],{"type":42,"tag":794,"props":3158,"children":3159},{},[3160],{"type":48,"value":3161},"Semantic quality judgment (tone, accuracy, completeness)",{"type":42,"tag":794,"props":3163,"children":3164},{},[3165,3170,3172],{"type":42,"tag":73,"props":3166,"children":3168},{"className":3167},[],[3169],{"type":48,"value":597},{"type":48,"value":3171}," + ",{"type":42,"tag":73,"props":3173,"children":3175},{"className":3174},[],[3176],{"type":48,"value":3177},"BooleanStructuredOutput",{"type":42,"tag":762,"props":3179,"children":3180},{},[3181,3186],{"type":42,"tag":794,"props":3182,"children":3183},{},[3184],{"type":48,"value":3185},"Graded quality on a scale",{"type":42,"tag":794,"props":3187,"children":3188},{},[3189,3194,3195],{"type":42,"tag":73,"props":3190,"children":3192},{"className":3191},[],[3193],{"type":48,"value":597},{"type":48,"value":3171},{"type":42,"tag":73,"props":3196,"children":3198},{"className":3197},[],[3199],{"type":48,"value":3200},"ScoreStructuredOutput",{"type":42,"tag":762,"props":3202,"children":3203},{},[3204,3209],{"type":42,"tag":794,"props":3205,"children":3206},{},[3207],{"type":48,"value":3208},"Classification into categories",{"type":42,"tag":794,"props":3210,"children":3211},{},[3212,3217,3218],{"type":42,"tag":73,"props":3213,"children":3215},{"className":3214},[],[3216],{"type":48,"value":597},{"type":48,"value":3171},{"type":42,"tag":73,"props":3219,"children":3221},{"className":3220},[],[3222],{"type":48,"value":3223},"CategoricalStructuredOutput",{"type":42,"tag":762,"props":3225,"children":3226},{},[3227,3232],{"type":42,"tag":794,"props":3228,"children":3229},{},[3230],{"type":48,"value":3231},"Multi-dimensional judgment (evaluate several aspects at once)",{"type":42,"tag":794,"props":3233,"children":3234},{},[3235,3240,3242],{"type":42,"tag":73,"props":3236,"children":3238},{"className":3237},[],[3239],{"type":48,"value":597},{"type":48,"value":3241}," + custom JSON schema ",{"type":42,"tag":73,"props":3243,"children":3245},{"className":3244},[],[3246],{"type":48,"value":3247},"dict",{"type":42,"tag":762,"props":3249,"children":3250},{},[3251,3256],{"type":42,"tag":794,"props":3252,"children":3253},{},[3254],{"type":48,"value":3255},"Complex domain logic combining multiple checks",{"type":42,"tag":794,"props":3257,"children":3258},{},[3259,3264],{"type":42,"tag":73,"props":3260,"children":3262},{"className":3261},[],[3263],{"type":48,"value":589},{"type":48,"value":3265}," subclass",{"type":42,"tag":747,"props":3267,"children":3269},{"id":3268},"source-verification",[3270],{"type":48,"value":3271},"Source Verification",{"type":42,"tag":51,"props":3273,"children":3274},{},[3275],{"type":48,"value":3276},"If you have access to dd-trace-py locally, verify the API surface by reading the corresponding modules:",{"type":42,"tag":194,"props":3278,"children":3279},{},[3280,3313,3341,3362],{"type":42,"tag":67,"props":3281,"children":3282},{},[3283,3289,3290,3295,3296,3301,3302,3307,3308],{"type":42,"tag":73,"props":3284,"children":3286},{"className":3285},[],[3287],{"type":48,"value":3288},"ddtrace.llmobs._evaluators.llm_judge",{"type":48,"value":1304},{"type":42,"tag":73,"props":3291,"children":3293},{"className":3292},[],[3294],{"type":48,"value":597},{"type":48,"value":251},{"type":42,"tag":73,"props":3297,"children":3299},{"className":3298},[],[3300],{"type":48,"value":3177},{"type":48,"value":251},{"type":42,"tag":73,"props":3303,"children":3305},{"className":3304},[],[3306],{"type":48,"value":3200},{"type":48,"value":251},{"type":42,"tag":73,"props":3309,"children":3311},{"className":3310},[],[3312],{"type":48,"value":3223},{"type":42,"tag":67,"props":3314,"children":3315},{},[3316,3322,3323,3328,3329,3334,3335],{"type":42,"tag":73,"props":3317,"children":3319},{"className":3318},[],[3320],{"type":48,"value":3321},"ddtrace.llmobs._experiment",{"type":48,"value":1304},{"type":42,"tag":73,"props":3324,"children":3326},{"className":3325},[],[3327],{"type":48,"value":589},{"type":48,"value":251},{"type":42,"tag":73,"props":3330,"children":3332},{"className":3331},[],[3333],{"type":48,"value":2312},{"type":48,"value":251},{"type":42,"tag":73,"props":3336,"children":3338},{"className":3337},[],[3339],{"type":48,"value":3340},"EvaluatorResult",{"type":42,"tag":67,"props":3342,"children":3343},{},[3344,3350,3351,3356,3357],{"type":42,"tag":73,"props":3345,"children":3347},{"className":3346},[],[3348],{"type":48,"value":3349},"ddtrace.llmobs._evaluators.format",{"type":48,"value":1304},{"type":42,"tag":73,"props":3352,"children":3354},{"className":3353},[],[3355],{"type":48,"value":3102},{"type":48,"value":251},{"type":42,"tag":73,"props":3358,"children":3360},{"className":3359},[],[3361],{"type":48,"value":3136},{"type":42,"tag":67,"props":3363,"children":3364},{},[3365,3371,3372,3377,3378],{"type":42,"tag":73,"props":3366,"children":3368},{"className":3367},[],[3369],{"type":48,"value":3370},"ddtrace.llmobs._evaluators.string_matching",{"type":48,"value":1304},{"type":42,"tag":73,"props":3373,"children":3375},{"className":3374},[],[3376],{"type":48,"value":3153},{"type":48,"value":251},{"type":42,"tag":73,"props":3379,"children":3381},{"className":3380},[],[3382],{"type":48,"value":3119},{"type":42,"tag":1773,"props":3384,"children":3385},{},[],{"type":42,"tag":43,"props":3387,"children":3389},{"id":3388},"workflow",[3390],{"type":48,"value":3391},"Workflow",{"type":42,"tag":747,"props":3393,"children":3395},{"id":3394},"phase-0-resolve-inputs-entry-mode",[3396],{"type":48,"value":3397},"Phase 0: Resolve Inputs & Entry Mode",{"type":42,"tag":51,"props":3399,"children":3400},{},[3401],{"type":42,"tag":55,"props":3402,"children":3403},{},[3404],{"type":48,"value":3405},"Entry mode detection:",{"type":42,"tag":754,"props":3407,"children":3408},{},[3409,3429],{"type":42,"tag":758,"props":3410,"children":3411},{},[3412],{"type":42,"tag":762,"props":3413,"children":3414},{},[3415,3420,3424],{"type":42,"tag":766,"props":3416,"children":3417},{},[3418],{"type":48,"value":3419},"Mode",{"type":42,"tag":766,"props":3421,"children":3422},{},[3423],{"type":48,"value":3077},{"type":42,"tag":766,"props":3425,"children":3426},{},[3427],{"type":48,"value":3428},"Behavior",{"type":42,"tag":787,"props":3430,"children":3431},{},[3432,3460],{"type":42,"tag":762,"props":3433,"children":3434},{},[3435,3443,3455],{"type":42,"tag":794,"props":3436,"children":3437},{},[3438],{"type":42,"tag":55,"props":3439,"children":3440},{},[3441],{"type":48,"value":3442},"Cold Start",{"type":42,"tag":794,"props":3444,"children":3445},{},[3446,3448,3453],{"type":48,"value":3447},"Only ",{"type":42,"tag":73,"props":3449,"children":3451},{"className":3450},[],[3452],{"type":48,"value":802},{"type":48,"value":3454}," provided (no RCA, no hypothesis)",{"type":42,"tag":794,"props":3456,"children":3457},{},[3458],{"type":48,"value":3459},"Full open discovery — understand what the app does, identify quality dimensions worth measuring, propose evals for coverage",{"type":42,"tag":762,"props":3461,"children":3462},{},[3463,3471,3476],{"type":42,"tag":794,"props":3464,"children":3465},{},[3466],{"type":42,"tag":55,"props":3467,"children":3468},{},[3469],{"type":48,"value":3470},"From RCA",{"type":42,"tag":794,"props":3472,"children":3473},{},[3474],{"type":48,"value":3475},"Conversation contains an RCA report or user provides a failure hypothesis",{"type":42,"tag":794,"props":3477,"children":3478},{},[3479],{"type":48,"value":3480},"Skip open discovery — use existing failure taxonomy as eval targets",{"type":42,"tag":51,"props":3482,"children":3483},{},[3484,3489,3491,3496,3498,3503,3505,3510,3512,3517,3518,3523,3524,3529,3530,3535,3537,3542,3544,3550],{"type":42,"tag":55,"props":3485,"children":3486},{},[3487],{"type":48,"value":3488},"Parse arguments",{"type":48,"value":3490},": Extract ",{"type":42,"tag":73,"props":3492,"children":3494},{"className":3493},[],[3495],{"type":48,"value":802},{"type":48,"value":3497}," (first non-flag argument), ",{"type":42,"tag":73,"props":3499,"children":3501},{"className":3500},[],[3502],{"type":48,"value":287},{"type":48,"value":3504}," (default ",{"type":42,"tag":73,"props":3506,"children":3508},{"className":3507},[],[3509],{"type":48,"value":295},{"type":48,"value":3511},"), ",{"type":42,"tag":73,"props":3513,"children":3515},{"className":3514},[],[3516],{"type":48,"value":1030},{"type":48,"value":3504},{"type":42,"tag":73,"props":3519,"children":3521},{"className":3520},[],[3522],{"type":48,"value":1043},{"type":48,"value":3511},{"type":42,"tag":73,"props":3525,"children":3527},{"className":3526},[],[3528],{"type":48,"value":571},{"type":48,"value":251},{"type":42,"tag":73,"props":3531,"children":3533},{"className":3532},[],[3534],{"type":48,"value":629},{"type":48,"value":3536},", and ",{"type":42,"tag":73,"props":3538,"children":3540},{"className":3539},[],[3541],{"type":48,"value":656},{"type":48,"value":3543}," flags. Set ",{"type":42,"tag":73,"props":3545,"children":3547},{"className":3546},[],[3548],{"type":48,"value":3549},"output_mode",{"type":48,"value":3551}," as follows (at most one of the three mode flags may be set; error if more than one is present):",{"type":42,"tag":194,"props":3553,"children":3554},{},[3555,3580,3596,3612],{"type":42,"tag":67,"props":3556,"children":3557},{},[3558,3563,3565,3571,3573,3578],{"type":42,"tag":73,"props":3559,"children":3561},{"className":3560},[],[3562],{"type":48,"value":656},{"type":48,"value":3564}," set → ",{"type":42,"tag":73,"props":3566,"children":3568},{"className":3567},[],[3569],{"type":48,"value":3570},"output_mode = emit_dataset",{"type":48,"value":3572},". Skip the rest of the workflow entry-mode logic and jump directly to ",{"type":42,"tag":55,"props":3574,"children":3575},{},[3576],{"type":48,"value":3577},"Phase 3D",{"type":48,"value":3579}," below.",{"type":42,"tag":67,"props":3581,"children":3582},{},[3583,3588,3589,3595],{"type":42,"tag":73,"props":3584,"children":3586},{"className":3585},[],[3587],{"type":48,"value":571},{"type":48,"value":3564},{"type":42,"tag":73,"props":3590,"children":3592},{"className":3591},[],[3593],{"type":48,"value":3594},"output_mode = sdk_code",{"type":48,"value":332},{"type":42,"tag":67,"props":3597,"children":3598},{},[3599,3604,3605,3611],{"type":42,"tag":73,"props":3600,"children":3602},{"className":3601},[],[3603],{"type":48,"value":629},{"type":48,"value":3564},{"type":42,"tag":73,"props":3606,"children":3608},{"className":3607},[],[3609],{"type":48,"value":3610},"output_mode = data_only",{"type":48,"value":332},{"type":42,"tag":67,"props":3613,"children":3614},{},[3615,3617,3623],{"type":48,"value":3616},"otherwise → ",{"type":42,"tag":73,"props":3618,"children":3620},{"className":3619},[],[3621],{"type":48,"value":3622},"output_mode = publish",{"type":48,"value":3624}," (the default — propose online evaluators, gated on user confirmation, created as disabled drafts).",{"type":42,"tag":51,"props":3626,"children":3627},{},[3628],{"type":42,"tag":55,"props":3629,"children":3630},{},[3631],{"type":48,"value":3632},"Resolution steps:",{"type":42,"tag":63,"props":3634,"children":3635},{},[3636,3647,3692,3709,3834],{"type":42,"tag":67,"props":3637,"children":3638},{},[3639,3640,3645],{"type":48,"value":1069},{"type":42,"tag":73,"props":3641,"children":3643},{"className":3642},[],[3644],{"type":48,"value":802},{"type":48,"value":3646}," not provided → ask the user.",{"type":42,"tag":67,"props":3648,"children":3649},{},[3650,3652],{"type":48,"value":3651},"Auto-detect entry mode:",{"type":42,"tag":194,"props":3653,"children":3654},{},[3655,3668,3680],{"type":42,"tag":67,"props":3656,"children":3657},{},[3658,3660,3666],{"type":48,"value":3659},"If the conversation contains an RCA report (look for \"Failure Taxonomy\" heading, structured failure modes, or severity ratings) → ",{"type":42,"tag":73,"props":3661,"children":3663},{"className":3662},[],[3664],{"type":48,"value":3665},"from_rca",{"type":48,"value":3667},". Extract the taxonomy.",{"type":42,"tag":67,"props":3669,"children":3670},{},[3671,3673,3678],{"type":48,"value":3672},"If the user provides a free-text failure hypothesis (e.g., \"the system prompt lacks grounding\") → ",{"type":42,"tag":73,"props":3674,"children":3676},{"className":3675},[],[3677],{"type":48,"value":3665},{"type":48,"value":3679},". Use the hypothesis as the starting eval target.",{"type":42,"tag":67,"props":3681,"children":3682},{},[3683,3685,3691],{"type":48,"value":3684},"Otherwise → ",{"type":42,"tag":73,"props":3686,"children":3688},{"className":3687},[],[3689],{"type":48,"value":3690},"cold_start",{"type":48,"value":332},{"type":42,"tag":67,"props":3693,"children":3694},{},[3695,3696,3701,3703,3708],{"type":48,"value":1069},{"type":42,"tag":73,"props":3697,"children":3699},{"className":3698},[],[3700],{"type":48,"value":829},{"type":48,"value":3702}," not provided → default to ",{"type":42,"tag":73,"props":3704,"children":3706},{"className":3705},[],[3707],{"type":48,"value":295},{"type":48,"value":332},{"type":42,"tag":67,"props":3710,"children":3711},{},[3712,3717,3718,3728,3730,3735,3737,3743,3745,3751,3753,3759,3761,3765,3767,3772,3774,3778,3780,3786,3788,3791,3803,3805,3810,3812,3817,3819,3825,3826,3832],{"type":42,"tag":55,"props":3713,"children":3714},{},[3715],{"type":48,"value":3716},"Map existing eval coverage",{"type":48,"value":1304},{"type":42,"tag":55,"props":3719,"children":3720},{},[3721,3723],{"type":48,"value":3722},"skip if ",{"type":42,"tag":73,"props":3724,"children":3726},{"className":3725},[],[3727],{"type":48,"value":3610},{"type":48,"value":3729}," (there is no Datadog eval project to check coverage against): Call ",{"type":42,"tag":73,"props":3731,"children":3733},{"className":3732},[],[3734],{"type":48,"value":1246},{"type":48,"value":3736}," (org-wide; filter the result client-side to entries where ",{"type":42,"tag":73,"props":3738,"children":3740},{"className":3739},[],[3741],{"type":48,"value":3742},"ml_app == \u003Cml_app>",{"type":48,"value":3744},"). Then, for each eval with ",{"type":42,"tag":73,"props":3746,"children":3748},{"className":3747},[],[3749],{"type":48,"value":3750},"source=custom",{"type":48,"value":3752},", call ",{"type":42,"tag":73,"props":3754,"children":3756},{"className":3755},[],[3757],{"type":48,"value":3758},"get_llmobs_evaluator(eval_name=...)",{"type":48,"value":3760}," to inspect its prompt template, target, sampling, and filter, and infer which quality dimension it covers. Issue all evaluator calls in a ",{"type":42,"tag":55,"props":3762,"children":3763},{},[3764],{"type":48,"value":1721},{"type":48,"value":3766}," (parallelize). Skip ",{"type":42,"tag":73,"props":3768,"children":3770},{"className":3769},[],[3771],{"type":48,"value":1317},{"type":48,"value":3773}," evals — their names are self-describing and they may not have a fetchable config.",{"type":42,"tag":3775,"props":3776,"children":3777},"br",{},[],{"type":48,"value":3779},"By the end of this step you have a complete coverage map: ",{"type":42,"tag":73,"props":3781,"children":3783},{"className":3782},[],[3784],{"type":48,"value":3785},"{eval_name → source, enabled, dimension}",{"type":48,"value":3787},". Carry this into Phase 2 for deduplication.",{"type":42,"tag":3775,"props":3789,"children":3790},{},[],{"type":42,"tag":55,"props":3792,"children":3793},{},[3794,3796,3801],{"type":48,"value":3795},"In ",{"type":42,"tag":73,"props":3797,"children":3799},{"className":3798},[],[3800],{"type":48,"value":458},{"type":48,"value":3802}," mode, also note any template-variable convention",{"type":48,"value":3804}," the existing custom evaluators already use (so a new suite reads consistently). Online evaluator templates resolve against the ",{"type":42,"tag":55,"props":3806,"children":3807},{},[3808],{"type":48,"value":3809},"full span JSON",{"type":48,"value":3811},", not against ",{"type":42,"tag":73,"props":3813,"children":3815},{"className":3814},[],[3816],{"type":48,"value":2312},{"type":48,"value":3818},". See the \"Online Template Variables\" section under \"Publishing Conventions\" for the supported syntax (",{"type":42,"tag":73,"props":3820,"children":3822},{"className":3821},[],[3823],{"type":48,"value":3824},"{{span_input}}",{"type":48,"value":251},{"type":42,"tag":73,"props":3827,"children":3829},{"className":3828},[],[3830],{"type":48,"value":3831},"{{span_output}}",{"type":48,"value":3833},", dot-paths, array selectors, filter accessors).",{"type":42,"tag":67,"props":3835,"children":3836},{},[3837,3842,3844,3850,3852,3858,3860,3866,3868,3874],{"type":42,"tag":55,"props":3838,"children":3839},{},[3840],{"type":48,"value":3841},"Notebook context detection",{"type":48,"value":3843},": Scan the current conversation for a Datadog notebook URL that was produced by ",{"type":42,"tag":73,"props":3845,"children":3847},{"className":3846},[],[3848],{"type":48,"value":3849},"\u002Feval-trace-rca",{"type":48,"value":3851}," (pattern: ",{"type":42,"tag":73,"props":3853,"children":3855},{"className":3854},[],[3856],{"type":48,"value":3857},"https:\u002F\u002Fapp.datadoghq.com\u002Fnotebook\u002F{numeric-id}",{"type":48,"value":3859},"). If found, store it as ",{"type":42,"tag":73,"props":3861,"children":3863},{"className":3862},[],[3864],{"type":48,"value":3865},"rca_notebook_url",{"type":48,"value":3867}," and extract the numeric ID as ",{"type":42,"tag":73,"props":3869,"children":3871},{"className":3870},[],[3872],{"type":48,"value":3873},"rca_notebook_id",{"type":48,"value":3875},". This is used after Phase 3 to offer appending the evaluator suite to that notebook instead of creating a new one.",{"type":42,"tag":1773,"props":3877,"children":3878},{},[],{"type":42,"tag":747,"props":3880,"children":3882},{"id":3881},"phase-1-explore-traces-identify-eval-targets",[3883],{"type":48,"value":3884},"Phase 1: Explore Traces & Identify Eval Targets",{"type":42,"tag":51,"props":3886,"children":3887},{},[3888,3893],{"type":42,"tag":55,"props":3889,"children":3890},{},[3891],{"type":48,"value":3892},"Goal",{"type":48,"value":3894},": Sample production traces, understand what the app does, and identify quality dimensions worth measuring.",{"type":42,"tag":3896,"props":3897,"children":3899},"h4",{"id":3898},"cold-start-path",[3900],{"type":48,"value":3901},"Cold Start Path",{"type":42,"tag":63,"props":3903,"children":3904},{},[3905,4047,4373],{"type":42,"tag":67,"props":3906,"children":3907},{},[3908,3913,3914,3920,3922,3928,3930,3933,3938,3939,3951,3953,3959,3960],{"type":42,"tag":55,"props":3909,"children":3910},{},[3911],{"type":48,"value":3912},"Sample the app",{"type":48,"value":1665},{"type":42,"tag":73,"props":3915,"children":3917},{"className":3916},[],[3918],{"type":48,"value":3919},"search_llmobs_spans(query=\"@ml_app:\\\"\u003Cml_app>\\\" @status:ok\", root_spans_only=true, limit=50, from=\u003Ctimeframe>)",{"type":48,"value":3921},". Filter by ",{"type":42,"tag":73,"props":3923,"children":3925},{"className":3924},[],[3926],{"type":48,"value":3927},"@status:ok",{"type":48,"value":3929}," — error spans have no output to evaluate.",{"type":42,"tag":3775,"props":3931,"children":3932},{},[],{"type":42,"tag":55,"props":3934,"children":3935},{},[3936],{"type":48,"value":3937},"Session probe",{"type":48,"value":460},{"type":42,"tag":462,"props":3940,"children":3941},{},[3942,3944,3949],{"type":48,"value":3943},"(gates session-scope proposals; ",{"type":42,"tag":73,"props":3945,"children":3947},{"className":3946},[],[3948],{"type":48,"value":458},{"type":48,"value":3950}," mode)",{"type":48,"value":3952},": in the same message, also call ",{"type":42,"tag":73,"props":3954,"children":3956},{"className":3955},[],[3957],{"type":48,"value":3958},"search_llmobs_spans(query=\"@ml_app:\\\"\u003Cml_app>\\\" session_id:*\", limit=20, from=\u003Ctimeframe>)",{"type":48,"value":332},{"type":42,"tag":194,"props":3961,"children":3962},{},[3963,4017],{"type":42,"tag":67,"props":3964,"children":3965},{},[3966,3971,3973,3979,3981,3986,3988,3993,3995,4008,4010,4015],{"type":42,"tag":55,"props":3967,"children":3968},{},[3969],{"type":48,"value":3970},"≥ 1 result",{"type":48,"value":3972}," → set ",{"type":42,"tag":73,"props":3974,"children":3976},{"className":3975},[],[3977],{"type":48,"value":3978},"sessions_present = true",{"type":48,"value":3980},". Note the distinct ",{"type":42,"tag":73,"props":3982,"children":3984},{"className":3983},[],[3985],{"type":48,"value":545},{"type":48,"value":3987}," values and, critically, whether the same ",{"type":42,"tag":73,"props":3989,"children":3991},{"className":3990},[],[3992],{"type":48,"value":545},{"type":48,"value":3994}," appears across ",{"type":42,"tag":55,"props":3996,"children":3997},{},[3998,4000,4006],{"type":48,"value":3999},"multiple ",{"type":42,"tag":73,"props":4001,"children":4003},{"className":4002},[],[4004],{"type":48,"value":4005},"trace_id",{"type":48,"value":4007},"s",{"type":48,"value":4009}," — that cross-trace span is the real signal that a session carries context worth a session-scope evaluator. (A ",{"type":42,"tag":73,"props":4011,"children":4013},{"className":4012},[],[4014],{"type":48,"value":545},{"type":48,"value":4016}," that only ever maps to one trace adds nothing over trace scope.)",{"type":42,"tag":67,"props":4018,"children":4019},{},[4020,4025,4026,4032,4034,4038,4040,4045],{"type":42,"tag":55,"props":4021,"children":4022},{},[4023],{"type":48,"value":4024},"0 results",{"type":48,"value":297},{"type":42,"tag":73,"props":4027,"children":4029},{"className":4028},[],[4030],{"type":48,"value":4031},"sessions_present = false",{"type":48,"value":4033},". Do ",{"type":42,"tag":55,"props":4035,"children":4036},{},[4037],{"type":48,"value":271},{"type":48,"value":4039}," propose any session-scope evaluator; record a one-line \"session scope skipped — no ",{"type":42,"tag":73,"props":4041,"children":4043},{"className":4042},[],[4044],{"type":48,"value":545},{"type":48,"value":4046}," on sampled spans\" note for the proposal.",{"type":42,"tag":67,"props":4048,"children":4049},{},[4050,4055,4057,4062,4064,4069,4071,4278,4281,4283,4288,4290,4295,4297,4302,4304,4361,4364,4366,4371],{"type":42,"tag":55,"props":4051,"children":4052},{},[4053],{"type":48,"value":4054},"Profile the app and identify evaluation target spans",{"type":48,"value":4056},": Call ",{"type":42,"tag":73,"props":4058,"children":4060},{"className":4059},[],[4061],{"type":48,"value":1162},{"type":48,"value":4063}," for span_ids grouped by trace_id. Inspect ",{"type":42,"tag":73,"props":4065,"children":4067},{"className":4066},[],[4068],{"type":48,"value":1173},{"type":48,"value":4070}," to classify:",{"type":42,"tag":754,"props":4072,"children":4073},{},[4074,4089],{"type":42,"tag":758,"props":4075,"children":4076},{},[4077],{"type":42,"tag":762,"props":4078,"children":4079},{},[4080,4084],{"type":42,"tag":766,"props":4081,"children":4082},{},[4083],{"type":48,"value":3077},{"type":42,"tag":766,"props":4085,"children":4086},{},[4087],{"type":48,"value":4088},"App Profile",{"type":42,"tag":787,"props":4090,"children":4091},{},[4092,4115,4137,4158,4180,4232],{"type":42,"tag":762,"props":4093,"children":4094},{},[4095,4110],{"type":42,"tag":794,"props":4096,"children":4097},{},[4098,4103,4105],{"type":42,"tag":73,"props":4099,"children":4101},{"className":4100},[],[4102],{"type":48,"value":1173},{"type":48,"value":4104}," has ",{"type":42,"tag":73,"props":4106,"children":4108},{"className":4107},[],[4109],{"type":48,"value":1425},{"type":42,"tag":794,"props":4111,"children":4112},{},[4113],{"type":48,"value":4114},"LLM\u002Fchat app",{"type":42,"tag":762,"props":4116,"children":4117},{},[4118,4132],{"type":42,"tag":794,"props":4119,"children":4120},{},[4121,4126,4127],{"type":42,"tag":73,"props":4122,"children":4124},{"className":4123},[],[4125],{"type":48,"value":1173},{"type":48,"value":4104},{"type":42,"tag":73,"props":4128,"children":4130},{"className":4129},[],[4131],{"type":48,"value":1536},{"type":42,"tag":794,"props":4133,"children":4134},{},[4135],{"type":48,"value":4136},"RAG app",{"type":42,"tag":762,"props":4138,"children":4139},{},[4140,4153],{"type":42,"tag":794,"props":4141,"children":4142},{},[4143,4145,4151],{"type":48,"value":4144},"Spans include ",{"type":42,"tag":73,"props":4146,"children":4148},{"className":4147},[],[4149],{"type":48,"value":4150},"agent",{"type":48,"value":4152}," kind",{"type":42,"tag":794,"props":4154,"children":4155},{},[4156],{"type":48,"value":4157},"Agent app",{"type":42,"tag":762,"props":4159,"children":4160},{},[4161,4175],{"type":42,"tag":794,"props":4162,"children":4163},{},[4164,4169,4170],{"type":42,"tag":73,"props":4165,"children":4167},{"className":4166},[],[4168],{"type":48,"value":1173},{"type":48,"value":4104},{"type":42,"tag":73,"props":4171,"children":4173},{"className":4172},[],[4174],{"type":48,"value":1557},{"type":42,"tag":794,"props":4176,"children":4177},{},[4178],{"type":48,"value":4179},"Has custom metadata",{"type":42,"tag":762,"props":4181,"children":4182},{},[4183,4221],{"type":42,"tag":794,"props":4184,"children":4185},{},[4186,4188,4193,4194,4200,4201,4207,4208,4213,4215,4220],{"type":48,"value":4187},"Multiple span kinds in one trace (",{"type":42,"tag":73,"props":4189,"children":4191},{"className":4190},[],[4192],{"type":48,"value":4150},{"type":48,"value":3171},{"type":42,"tag":73,"props":4195,"children":4197},{"className":4196},[],[4198],{"type":48,"value":4199},"tool",{"type":48,"value":591},{"type":42,"tag":73,"props":4202,"children":4204},{"className":4203},[],[4205],{"type":48,"value":4206},"retrieval",{"type":48,"value":3171},{"type":42,"tag":73,"props":4209,"children":4211},{"className":4210},[],[4212],{"type":48,"value":18},{"type":48,"value":4214}," from ",{"type":42,"tag":73,"props":4216,"children":4218},{"className":4217},[],[4219],{"type":48,"value":1212},{"type":48,"value":631},{"type":42,"tag":794,"props":4222,"children":4223},{},[4224,4226,4231],{"type":48,"value":4225},"Multi-step app — at least one trace-scope evaluator likely belongs in the suite (",{"type":42,"tag":73,"props":4227,"children":4229},{"className":4228},[],[4230],{"type":48,"value":458},{"type":48,"value":3950},{"type":42,"tag":762,"props":4233,"children":4234},{},[4235,4259],{"type":42,"tag":794,"props":4236,"children":4237},{},[4238,4240,4245,4247,4257],{"type":48,"value":4239},"Same ",{"type":42,"tag":73,"props":4241,"children":4243},{"className":4242},[],[4244],{"type":48,"value":545},{"type":48,"value":4246}," across ",{"type":42,"tag":55,"props":4248,"children":4249},{},[4250,4251,4256],{"type":48,"value":3999},{"type":42,"tag":73,"props":4252,"children":4254},{"className":4253},[],[4255],{"type":48,"value":4005},{"type":48,"value":4007},{"type":48,"value":4258}," (from the session probe)",{"type":42,"tag":794,"props":4260,"children":4261},{},[4262,4264,4269,4271,4277],{"type":48,"value":4263},"Multi-trace sessions — at least one session-scope evaluator likely belongs in the suite (",{"type":42,"tag":73,"props":4265,"children":4267},{"className":4266},[],[4268],{"type":48,"value":458},{"type":48,"value":4270}," mode, gated on ",{"type":42,"tag":73,"props":4272,"children":4274},{"className":4273},[],[4275],{"type":48,"value":4276},"sessions_present",{"type":48,"value":631},{"type":42,"tag":3775,"props":4279,"children":4280},{},[],{"type":48,"value":4282},"For agent\u002Fmulti-step apps, also call ",{"type":42,"tag":73,"props":4284,"children":4286},{"className":4285},[],[4287],{"type":48,"value":1212},{"type":48,"value":4289}," on 2-3 traces to see the full span hierarchy. Compare ",{"type":42,"tag":73,"props":4291,"children":4293},{"className":4292},[],[4294],{"type":48,"value":1173},{"type":48,"value":4296}," between the root span and its sub-spans. Then ask ",{"type":42,"tag":55,"props":4298,"children":4299},{},[4300],{"type":48,"value":4301},"two",{"type":48,"value":4303}," questions for each candidate quality dimension, in this order:",{"type":42,"tag":63,"props":4305,"children":4306},{},[4307,4351],{"type":42,"tag":67,"props":4308,"children":4309},{},[4310,4315,4317,4322,4324,4329,4331,4336,4338,4343,4344,4349],{"type":42,"tag":55,"props":4311,"children":4312},{},[4313],{"type":48,"value":4314},"Does the verdict depend on more than one span?",{"type":48,"value":4316}," (e.g., faithfulness depends on a ",{"type":42,"tag":73,"props":4318,"children":4320},{"className":4319},[],[4321],{"type":48,"value":4206},{"type":48,"value":4323}," span's documents AND an ",{"type":42,"tag":73,"props":4325,"children":4327},{"className":4326},[],[4328],{"type":48,"value":18},{"type":48,"value":4330}," span's answer; goal completion depends on the chain of ",{"type":42,"tag":73,"props":4332,"children":4334},{"className":4333},[],[4335],{"type":48,"value":4199},{"type":48,"value":4337}," calls AND the final response.) If yes → ",{"type":42,"tag":55,"props":4339,"children":4340},{},[4341],{"type":48,"value":4342},"trace scope",{"type":48,"value":2272},{"type":42,"tag":73,"props":4345,"children":4347},{"className":4346},[],[4348],{"type":48,"value":458},{"type":48,"value":4350}," mode. Don't try to compress this into a single span.",{"type":42,"tag":67,"props":4352,"children":4353},{},[4354,4359],{"type":42,"tag":55,"props":4355,"children":4356},{},[4357],{"type":48,"value":4358},"Only if the answer to (1) is no",{"type":48,"value":4360},": pick the single span with the richest signal for that dimension (root has the summary; LLM sub-spans have the full system prompt + tool call results + reasoning chain).",{"type":42,"tag":3775,"props":4362,"children":4363},{},[],{"type":48,"value":4365},"Record the span-kind histogram (agent + tool + llm + retrieval) — multiple kinds under one root is a strong signal you'll have at least one trace-scope evaluator in the suite. See Phase 2's \"Span vs. Trace vs. Session Scope Classification\" for the mandatory walk-through of canonical trace-scope use cases (and, when ",{"type":42,"tag":73,"props":4367,"children":4369},{"className":4368},[],[4370],{"type":48,"value":4276},{"type":48,"value":4372},", the canonical session-scope use cases).",{"type":42,"tag":67,"props":4374,"children":4375},{},[4376,4381,4382,4387,4389,4509,4512,4514,4517,4522,4524,4527,4532,4534,4539,4541,4629,4632],{"type":42,"tag":55,"props":4377,"children":4378},{},[4379],{"type":48,"value":4380},"Extract content and identify targets",{"type":48,"value":4056},{"type":42,"tag":73,"props":4383,"children":4385},{"className":4384},[],[4386],{"type":48,"value":1187},{"type":48,"value":4388}," for representative spans. Fetch fields based on app profile:",{"type":42,"tag":754,"props":4390,"children":4391},{},[4392,4407],{"type":42,"tag":758,"props":4393,"children":4394},{},[4395],{"type":42,"tag":762,"props":4396,"children":4397},{},[4398,4402],{"type":42,"tag":766,"props":4399,"children":4400},{},[4401],{"type":48,"value":4088},{"type":42,"tag":766,"props":4403,"children":4404},{},[4405],{"type":48,"value":4406},"Fields to Fetch",{"type":42,"tag":787,"props":4408,"children":4409},{},[4410,4440,4468,4493],{"type":42,"tag":762,"props":4411,"children":4412},{},[4413,4418],{"type":42,"tag":794,"props":4414,"children":4415},{},[4416],{"type":48,"value":4417},"LLM\u002Fchat",{"type":42,"tag":794,"props":4419,"children":4420},{},[4421,4426,4427,4433,4435],{"type":42,"tag":73,"props":4422,"children":4424},{"className":4423},[],[4425],{"type":48,"value":1425},{"type":48,"value":497},{"type":42,"tag":73,"props":4428,"children":4430},{"className":4429},[],[4431],{"type":48,"value":4432},"path=$.messages[0]",{"type":48,"value":4434}," for system prompt), ",{"type":42,"tag":73,"props":4436,"children":4438},{"className":4437},[],[4439],{"type":48,"value":1515},{"type":42,"tag":762,"props":4441,"children":4442},{},[4443,4448],{"type":42,"tag":794,"props":4444,"children":4445},{},[4446],{"type":48,"value":4447},"RAG",{"type":42,"tag":794,"props":4449,"children":4450},{},[4451,4456,4457,4462,4463],{"type":42,"tag":73,"props":4452,"children":4454},{"className":4453},[],[4455],{"type":48,"value":1536},{"type":48,"value":251},{"type":42,"tag":73,"props":4458,"children":4460},{"className":4459},[],[4461],{"type":48,"value":1508},{"type":48,"value":251},{"type":42,"tag":73,"props":4464,"children":4466},{"className":4465},[],[4467],{"type":48,"value":1515},{"type":42,"tag":762,"props":4469,"children":4470},{},[4471,4476],{"type":42,"tag":794,"props":4472,"children":4473},{},[4474],{"type":48,"value":4475},"Agent",{"type":42,"tag":794,"props":4477,"children":4478},{},[4479,4484,4486,4491],{"type":42,"tag":73,"props":4480,"children":4482},{"className":4481},[],[4483],{"type":48,"value":1229},{"type":48,"value":4485}," for the agent span, then ",{"type":42,"tag":73,"props":4487,"children":4489},{"className":4488},[],[4490],{"type":48,"value":1425},{"type":48,"value":4492}," for detail",{"type":42,"tag":762,"props":4494,"children":4495},{},[4496,4501],{"type":42,"tag":794,"props":4497,"children":4498},{},[4499],{"type":48,"value":4500},"Any with metadata",{"type":42,"tag":794,"props":4502,"children":4503},{},[4504],{"type":42,"tag":73,"props":4505,"children":4507},{"className":4506},[],[4508],{"type":48,"value":1557},{"type":42,"tag":3775,"props":4510,"children":4511},{},[],{"type":48,"value":4513},"Issue all calls in a single message. As you read, capture two streams of signal:",{"type":42,"tag":3775,"props":4515,"children":4516},{},[],{"type":42,"tag":55,"props":4518,"children":4519},{},[4520],{"type":48,"value":4521},"Generic quality signals",{"type":48,"value":4523}," — what does \"success\" look like? What variance exists across outputs? Each observed quality dimension becomes a candidate evaluator, with the traces you've just read as evidence. Also look for safety signals (scope violations, sensitive data in outputs, out-of-character responses) and add a safety evaluator if you find them.",{"type":42,"tag":3775,"props":4525,"children":4526},{},[],{"type":42,"tag":55,"props":4528,"children":4529},{},[4530],{"type":48,"value":4531},"Domain signals",{"type":48,"value":4533}," — these become the ",{"type":42,"tag":462,"props":4535,"children":4536},{},[4537],{"type":48,"value":4538},"domain-specific evaluator",{"type":48,"value":4540}," category in Phase 2 (the highest-leverage category). For every 5–10 traces, write down:",{"type":42,"tag":194,"props":4542,"children":4543},{},[4544,4582,4599,4609,4619],{"type":42,"tag":67,"props":4545,"children":4546},{},[4547,4552,4554,4560,4561,4567,4568,4574,4575,4581],{"type":42,"tag":55,"props":4548,"children":4549},{},[4550],{"type":48,"value":4551},"Recurring intents \u002F question categories",{"type":48,"value":4553}," — what classes of request does this app handle? (",{"type":42,"tag":73,"props":4555,"children":4557},{"className":4556},[],[4558],{"type":48,"value":4559},"applying for benefit X",{"type":48,"value":251},{"type":42,"tag":73,"props":4562,"children":4564},{"className":4563},[],[4565],{"type":48,"value":4566},"comparing flight options",{"type":48,"value":251},{"type":42,"tag":73,"props":4569,"children":4571},{"className":4570},[],[4572],{"type":48,"value":4573},"summarizing a policy",{"type":48,"value":251},{"type":42,"tag":73,"props":4576,"children":4578},{"className":4577},[],[4579],{"type":48,"value":4580},"creating a widget",{"type":48,"value":631},{"type":42,"tag":67,"props":4583,"children":4584},{},[4585,4590,4592,4597],{"type":42,"tag":55,"props":4586,"children":4587},{},[4588],{"type":48,"value":4589},"Entities the app emits in outputs",{"type":48,"value":4591}," — URLs, agency \u002F company names, code identifiers, monetary amounts, dates, IDs, file paths, phone numbers. Note which ones the user ",{"type":42,"tag":462,"props":4593,"children":4594},{},[4595],{"type":48,"value":4596},"acts",{"type":48,"value":4598}," on downstream (those are worth a correctness evaluator) versus which are passing references.",{"type":42,"tag":67,"props":4600,"children":4601},{},[4602,4607],{"type":42,"tag":55,"props":4603,"children":4604},{},[4605],{"type":48,"value":4606},"Tool argument shapes",{"type":48,"value":4608}," (for agent apps) — name each tool the agent calls and the rough schema of its inputs. Tools with non-trivial schemas (≥ 3 fields, structured types) are candidates for argument-correctness evaluators.",{"type":42,"tag":67,"props":4610,"children":4611},{},[4612,4617],{"type":42,"tag":55,"props":4613,"children":4614},{},[4615],{"type":48,"value":4616},"Persona \u002F voice rules",{"type":48,"value":4618}," — does the app always cite a source, always refuse certain topics (medical, legal, financial advice), always speak in a particular tone? Extract the rules implicitly followed across observed outputs.",{"type":42,"tag":67,"props":4620,"children":4621},{},[4622,4627],{"type":42,"tag":55,"props":4623,"children":4624},{},[4625],{"type":48,"value":4626},"Failure modes specific to the domain",{"type":48,"value":4628}," — fabricated identifiers, outdated policy references, currency \u002F locale mismatches, off-by-one errors in IDs, wrong units. One observed instance is enough to seed a candidate evaluator.",{"type":42,"tag":3775,"props":4630,"children":4631},{},[],{"type":48,"value":4633},"Don't try to enumerate domain signals exhaustively before reading traces — let the patterns surface as you read. The goal is breadth in the eventual proposal, not completeness in this exploration step.",{"type":42,"tag":3896,"props":4635,"children":4637},{"id":4636},"from-rca-path",[4638],{"type":48,"value":4639},"From RCA Path",{"type":42,"tag":63,"props":4641,"children":4642},{},[4643,4669,4815],{"type":42,"tag":67,"props":4644,"children":4645},{},[4646,4648,4653,4654,4660,4662,4667],{"type":48,"value":4647},"Extract the failure taxonomy from the RCA report. Each failure mode with High or Medium severity becomes an eval target. Also run the Phase 1 ",{"type":42,"tag":55,"props":4649,"children":4650},{},[4651],{"type":48,"value":4652},"session probe",{"type":48,"value":497},{"type":42,"tag":73,"props":4655,"children":4657},{"className":4656},[],[4658],{"type":48,"value":4659},"query=\"session_id:*\"",{"type":48,"value":4661},") to set ",{"type":42,"tag":73,"props":4663,"children":4665},{"className":4664},[],[4666],{"type":48,"value":4276},{"type":48,"value":4668}," — a failure that only manifests across a multi-trace conversation (lost context, repeated mistakes, mounting frustration) is a session-scope target.",{"type":42,"tag":67,"props":4670,"children":4671},{},[4672,4677,4679,4685,4686,4692,4693,4699,4700,4706,4708,4752,4810,4813],{"type":42,"tag":55,"props":4673,"children":4674},{},[4675],{"type":48,"value":4676},"Check root cause categories for infrastructure failures.",{"type":48,"value":4678}," Before proposing evaluators, scan the Root Cause column of the taxonomy for any of: ",{"type":42,"tag":73,"props":4680,"children":4682},{"className":4681},[],[4683],{"type":48,"value":4684},"Instrumentation Deficiency",{"type":48,"value":251},{"type":42,"tag":73,"props":4687,"children":4689},{"className":4688},[],[4690],{"type":48,"value":4691},"Harness Deficiency",{"type":48,"value":251},{"type":42,"tag":73,"props":4694,"children":4696},{"className":4695},[],[4697],{"type":48,"value":4698},"Runtime Error",{"type":48,"value":251},{"type":42,"tag":73,"props":4701,"children":4703},{"className":4702},[],[4704],{"type":48,"value":4705},"Upstream Data Issue",{"type":48,"value":4707},", or any other root cause that points to infrastructure\u002Fenvironment rather than model behavior. If any are present, pause and ask:",{"type":42,"tag":159,"props":4709,"children":4710},{},[4711,4724,4747],{"type":42,"tag":51,"props":4712,"children":4713},{},[4714,4716,4722],{"type":48,"value":4715},"\"Some failure modes were diagnosed as infrastructure or instrumentation issues rather than model behavior (e.g., ",{"type":42,"tag":73,"props":4717,"children":4719},{"className":4718},[],[4720],{"type":48,"value":4721},"{list the infra root causes}",{"type":48,"value":4723},"). Evaluators can be designed two ways:",{"type":42,"tag":194,"props":4725,"children":4726},{},[4727,4737],{"type":42,"tag":67,"props":4728,"children":4729},{},[4730,4735],{"type":42,"tag":55,"props":4731,"children":4732},{},[4733],{"type":48,"value":4734},"Behavior-targeted",{"type":48,"value":4736}," (recommended for ongoing quality): measure whether the model produces correct, specific output — useful once the infrastructure is fixed and you want to track real quality",{"type":42,"tag":67,"props":4738,"children":4739},{},[4740,4745],{"type":42,"tag":55,"props":4741,"children":4742},{},[4743],{"type":48,"value":4744},"Artifact-targeted",{"type":48,"value":4746}," (useful as regression guard): detect the specific broken output observed (e.g., generic placeholder responses) — catches regressions if the infrastructure breaks again",{"type":42,"tag":51,"props":4748,"children":4749},{},[4750],{"type":48,"value":4751},"Which approach do you want, or both?\"",{"type":42,"tag":194,"props":4753,"children":4754},{},[4755,4774,4799],{"type":42,"tag":67,"props":4756,"children":4757},{},[4758,4759,4764,4766,4772],{"type":48,"value":1069},{"type":42,"tag":55,"props":4760,"children":4761},{},[4762],{"type":48,"value":4763},"behavior-targeted",{"type":48,"value":4765},": design evaluators for what correct output looks like, not what the broken output looked like. Use the RCA's ",{"type":42,"tag":73,"props":4767,"children":4769},{"className":4768},[],[4770],{"type":48,"value":4771},"expected_output",{"type":48,"value":4773}," \u002F gold-standard examples as the quality bar.",{"type":42,"tag":67,"props":4775,"children":4776},{},[4777,4778,4783,4785,4790,4792,4797],{"type":48,"value":1069},{"type":42,"tag":55,"props":4779,"children":4780},{},[4781],{"type":48,"value":4782},"artifact-targeted",{"type":48,"value":4784},": design evaluators that detect the specific failure symptom (e.g., ",{"type":42,"tag":73,"props":4786,"children":4788},{"className":4787},[],[4789],{"type":48,"value":3153},{"type":48,"value":4791}," for a known bad string, ",{"type":42,"tag":73,"props":4793,"children":4795},{"className":4794},[],[4796],{"type":48,"value":597},{"type":48,"value":4798}," that checks for generic placeholders).",{"type":42,"tag":67,"props":4800,"children":4801},{},[4802,4803,4808],{"type":48,"value":1069},{"type":42,"tag":55,"props":4804,"children":4805},{},[4806],{"type":48,"value":4807},"both",{"type":48,"value":4809},": propose each category separately, clearly labelled.",{"type":42,"tag":3775,"props":4811,"children":4812},{},[],{"type":48,"value":4814},"If all root causes are behavioral (System Prompt Deficiency, Tool Gap, Tool Misuse, Retrieval Failure, etc.) → skip this step and proceed directly.",{"type":42,"tag":67,"props":4816,"children":4817},{},[4818,4820,4825],{"type":48,"value":4819},"For each target: if the RCA includes trace IDs, use them directly; otherwise search for matching traces. Fetch 2-3 traces per target with ",{"type":42,"tag":73,"props":4821,"children":4823},{"className":4822},[],[4824],{"type":48,"value":1187},{"type":48,"value":4826}," to understand the concrete pattern.",{"type":42,"tag":1773,"props":4828,"children":4829},{},[],{"type":42,"tag":747,"props":4831,"children":4833},{"id":4832},"phase-2-propose-evaluator-suite",[4834],{"type":48,"value":4835},"Phase 2: Propose Evaluator Suite",{"type":42,"tag":51,"props":4837,"children":4838},{},[4839,4843],{"type":42,"tag":55,"props":4840,"children":4841},{},[4842],{"type":48,"value":3892},{"type":48,"value":4844},": Present a concrete evaluator proposal for user confirmation.",{"type":42,"tag":51,"props":4846,"children":4847},{},[4848,4849,4854,4855,4860,4862,4868,4869,4874,4876,4881,4883,4888,4890,4896,4898,4904],{"type":48,"value":3795},{"type":42,"tag":73,"props":4850,"children":4852},{"className":4851},[],[4853],{"type":48,"value":559},{"type":48,"value":591},{"type":42,"tag":73,"props":4856,"children":4858},{"className":4857},[],[4859],{"type":48,"value":618},{"type":48,"value":4861}," mode — and for ",{"type":42,"tag":73,"props":4863,"children":4865},{"className":4864},[],[4866],{"type":48,"value":4867},"eval_scope: span",{"type":48,"value":2272},{"type":42,"tag":73,"props":4870,"children":4872},{"className":4871},[],[4873],{"type":48,"value":458},{"type":48,"value":4875}," mode — each evaluator judges ",{"type":42,"tag":55,"props":4877,"children":4878},{},[4879],{"type":48,"value":4880},"one data point",{"type":48,"value":4882},": input and output for a single record\u002Fspan, not a full trace or batch. In ",{"type":42,"tag":73,"props":4884,"children":4886},{"className":4885},[],[4887],{"type":48,"value":458},{"type":48,"value":4889}," mode, ",{"type":42,"tag":73,"props":4891,"children":4893},{"className":4892},[],[4894],{"type":48,"value":4895},"eval_scope: trace",{"type":48,"value":4897}," judges a whole trace and ",{"type":42,"tag":73,"props":4899,"children":4901},{"className":4900},[],[4902],{"type":48,"value":4903},"eval_scope: session",{"type":48,"value":4905}," a whole multi-trace session — design those against the trace \u002F session payload instead (see \"Span vs. Trace vs. Session Scope Classification\" below). Design evaluators accordingly for their scope.",{"type":42,"tag":51,"props":4907,"children":4908},{},[4909],{"type":42,"tag":55,"props":4910,"children":4911},{},[4912,4914,4919],{"type":48,"value":4913},"Targeting depends on ",{"type":42,"tag":73,"props":4915,"children":4917},{"className":4916},[],[4918],{"type":48,"value":3549},{"type":48,"value":4920},":",{"type":42,"tag":194,"props":4922,"children":4923},{},[4924,4966],{"type":42,"tag":67,"props":4925,"children":4926},{},[4927,4932,4933,4938,4939,4944,4946,4951,4953,4958,4959,4964],{"type":42,"tag":73,"props":4928,"children":4930},{"className":4929},[],[4931],{"type":48,"value":559},{"type":48,"value":591},{"type":42,"tag":73,"props":4934,"children":4936},{"className":4935},[],[4937],{"type":48,"value":618},{"type":48,"value":297},{"type":42,"tag":55,"props":4940,"children":4941},{},[4942],{"type":48,"value":4943},"offline experiments",{"type":48,"value":4945},". Template variables use ",{"type":42,"tag":73,"props":4947,"children":4949},{"className":4948},[],[4950],{"type":48,"value":2312},{"type":48,"value":4952}," fields (",{"type":42,"tag":73,"props":4954,"children":4956},{"className":4955},[],[4957],{"type":48,"value":2094},{"type":48,"value":251},{"type":42,"tag":73,"props":4960,"children":4962},{"className":4961},[],[4963],{"type":48,"value":2101},{"type":48,"value":4965},"). The actual data shape depends on the user's dataset and task function (see EvaluatorContext note in SDK Reference).",{"type":42,"tag":67,"props":4967,"children":4968},{},[4969,4974,4975,4980,4982,4986,4988,4994,4995,5001,5003,5008,5009,5014,5016,5021,5022,5027,5029,5035,5037,5043,5045,5051,5052,5058,5060,5066,5068,5073],{"type":42,"tag":73,"props":4970,"children":4972},{"className":4971},[],[4973],{"type":48,"value":458},{"type":48,"value":297},{"type":42,"tag":55,"props":4976,"children":4977},{},[4978],{"type":48,"value":4979},"online evaluation on production spans",{"type":48,"value":4981},". Template variables resolve against the ",{"type":42,"tag":55,"props":4983,"children":4984},{},[4985],{"type":48,"value":3809},{"type":48,"value":4987}," via dot-paths (",{"type":42,"tag":73,"props":4989,"children":4991},{"className":4990},[],[4992],{"type":48,"value":4993},"{{meta.input.value}}",{"type":48,"value":251},{"type":42,"tag":73,"props":4996,"children":4998},{"className":4997},[],[4999],{"type":48,"value":5000},"{{meta.output.messages[*].content}}",{"type":48,"value":5002},", …) or the built-in span-kind-aware aliases (",{"type":42,"tag":73,"props":5004,"children":5006},{"className":5005},[],[5007],{"type":48,"value":3824},{"type":48,"value":251},{"type":42,"tag":73,"props":5010,"children":5012},{"className":5011},[],[5013],{"type":48,"value":3831},{"type":48,"value":5015},"). For ",{"type":42,"tag":73,"props":5017,"children":5019},{"className":5018},[],[5020],{"type":48,"value":4895},{"type":48,"value":919},{"type":42,"tag":73,"props":5023,"children":5025},{"className":5024},[],[5026],{"type":48,"value":4903},{"type":48,"value":5028},", templates resolve against the trace payload (",{"type":42,"tag":73,"props":5030,"children":5032},{"className":5031},[],[5033],{"type":48,"value":5034},"{{spans[...]}}",{"type":48,"value":5036},") or the session payload (",{"type":42,"tag":73,"props":5038,"children":5040},{"className":5039},[],[5041],{"type":48,"value":5042},"{{traces[*].spans[...]}}",{"type":48,"value":5044},") instead. See \"Online Template Variables\" under Publishing Conventions for the full syntax. Each evaluator also needs ",{"type":42,"tag":73,"props":5046,"children":5048},{"className":5047},[],[5049],{"type":48,"value":5050},"eval_scope",{"type":48,"value":251},{"type":42,"tag":73,"props":5053,"children":5055},{"className":5054},[],[5056],{"type":48,"value":5057},"sampling_percentage",{"type":48,"value":5059},", and (optionally) ",{"type":42,"tag":73,"props":5061,"children":5063},{"className":5062},[],[5064],{"type":48,"value":5065},"filter",{"type":48,"value":5067}," — surface these in the proposal table so the user can confirm before publishing. Session scope is only used when the Phase 1 probe set ",{"type":42,"tag":73,"props":5069,"children":5071},{"className":5070},[],[5072],{"type":48,"value":4276},{"type":48,"value":332},{"type":42,"tag":51,"props":5075,"children":5076},{},[5077,5079,5084],{"type":48,"value":5078},"Order proposals from broadest signal to most granular. ",{"type":42,"tag":55,"props":5080,"children":5081},{},[5082],{"type":48,"value":5083},"Propose broadly, let the user curate",{"type":48,"value":5085}," — see \"How many evaluators to propose\" below.",{"type":42,"tag":63,"props":5087,"children":5088},{},[5089,5271,5309,5346],{"type":42,"tag":67,"props":5090,"children":5091},{},[5092,5097,5099,5104,5106,5111,5113,5243,5246,5248,5253,5255,5261,5263,5269],{"type":42,"tag":55,"props":5093,"children":5094},{},[5095],{"type":48,"value":5096},"Domain-specific evaluators",{"type":48,"value":5098}," — What does \"good\" mean ",{"type":42,"tag":462,"props":5100,"children":5101},{},[5102],{"type":48,"value":5103},"for this specific app",{"type":48,"value":5105},"? These are the highest-leverage proposals because they capture quality bars generic evaluators miss. Derive them from the ",{"type":42,"tag":55,"props":5107,"children":5108},{},[5109],{"type":48,"value":5110},"domain signals",{"type":48,"value":5112}," Phase 1 captured:",{"type":42,"tag":194,"props":5114,"children":5115},{},[5116,5141,5172,5202,5233],{"type":42,"tag":67,"props":5117,"children":5118},{},[5119,5123,5125,5131,5133,5139],{"type":42,"tag":55,"props":5120,"children":5121},{},[5122],{"type":48,"value":4551},{"type":48,"value":5124}," the app handles (e.g., \"applying for a federal benefit\", \"comparing flight options\", \"explaining a policy\"). Propose an ",{"type":42,"tag":73,"props":5126,"children":5128},{"className":5127},[],[5129],{"type":48,"value":5130},"intent_classification",{"type":48,"value":5132}," or ",{"type":42,"tag":73,"props":5134,"children":5136},{"className":5135},[],[5137],{"type":48,"value":5138},"intent_handling_correctness",{"type":48,"value":5140}," evaluator scoped to the dominant intents.",{"type":42,"tag":67,"props":5142,"children":5143},{},[5144,5149,5151,5157,5158,5164,5165,5171],{"type":42,"tag":55,"props":5145,"children":5146},{},[5147],{"type":48,"value":5148},"Specific entities the app produces",{"type":48,"value":5150}," (URLs, agency names, code identifiers, monetary amounts, dates, IDs). Propose a per-entity correctness evaluator for the ones with real downstream cost when wrong (e.g., ",{"type":42,"tag":73,"props":5152,"children":5154},{"className":5153},[],[5155],{"type":48,"value":5156},"cited_url_is_real",{"type":48,"value":251},{"type":42,"tag":73,"props":5159,"children":5161},{"className":5160},[],[5162],{"type":48,"value":5163},"agency_name_matches_request",{"type":48,"value":251},{"type":42,"tag":73,"props":5166,"children":5168},{"className":5167},[],[5169],{"type":48,"value":5170},"monetary_amount_is_consistent_with_input",{"type":48,"value":220},{"type":42,"tag":67,"props":5173,"children":5174},{},[5175,5179,5181,5186,5188,5194,5195,5201],{"type":42,"tag":55,"props":5176,"children":5177},{},[5178],{"type":48,"value":4606},{"type":48,"value":5180}," observed across ",{"type":42,"tag":73,"props":5182,"children":5184},{"className":5183},[],[5185],{"type":48,"value":4199},{"type":48,"value":5187}," spans. Propose a per-tool argument-correctness evaluator for the tools with non-trivial schemas (e.g., ",{"type":42,"tag":73,"props":5189,"children":5191},{"className":5190},[],[5192],{"type":48,"value":5193},"search_flights_args_match_user_request",{"type":48,"value":251},{"type":42,"tag":73,"props":5196,"children":5198},{"className":5197},[],[5199],{"type":48,"value":5200},"update_dashboard_widget_targets_correct_widget",{"type":48,"value":220},{"type":42,"tag":67,"props":5203,"children":5204},{},[5205,5210,5212,5218,5219,5225,5226,5232],{"type":42,"tag":55,"props":5206,"children":5207},{},[5208],{"type":48,"value":5209},"Persona \u002F voice expectations",{"type":48,"value":5211}," — does the app always cite sources, always refuse out-of-scope requests, always speak in a specific tone? Propose evaluators for the voice rules you can extract from observed outputs (",{"type":42,"tag":73,"props":5213,"children":5215},{"className":5214},[],[5216],{"type":48,"value":5217},"cites_a_source",{"type":48,"value":251},{"type":42,"tag":73,"props":5220,"children":5222},{"className":5221},[],[5223],{"type":48,"value":5224},"refuses_medical_advice",{"type":48,"value":251},{"type":42,"tag":73,"props":5227,"children":5229},{"className":5228},[],[5230],{"type":48,"value":5231},"tone_matches_brand",{"type":48,"value":220},{"type":42,"tag":67,"props":5234,"children":5235},{},[5236,5241],{"type":42,"tag":55,"props":5237,"children":5238},{},[5239],{"type":48,"value":5240},"Domain-specific failure modes",{"type":48,"value":5242}," seen across traces (fabricated identifiers, outdated policy references, unit mismatches, currency \u002F locale mismatches). One evaluator per recurring failure mode.",{"type":42,"tag":3775,"props":5244,"children":5245},{},[],{"type":48,"value":5247},"Name each evaluator after the ",{"type":42,"tag":462,"props":5249,"children":5250},{},[5251],{"type":48,"value":5252},"user-facing concern",{"type":48,"value":5254},", not the technical check (",{"type":42,"tag":73,"props":5256,"children":5258},{"className":5257},[],[5259],{"type":48,"value":5260},"agency_url_is_real",{"type":48,"value":5262}," over ",{"type":42,"tag":73,"props":5264,"children":5266},{"className":5265},[],[5267],{"type":48,"value":5268},"regex_url_match",{"type":48,"value":5270},"). Use the trace IDs you read in Phase 1 as evidence — at least one passing case and one failing case per evaluator if you saw both.",{"type":42,"tag":67,"props":5272,"children":5273},{},[5274,5279,5281],{"type":42,"tag":55,"props":5275,"children":5276},{},[5277],{"type":48,"value":5278},"Outcome evaluators",{"type":48,"value":5280}," — Did this span \u002F trace produce a good result for the request?",{"type":42,"tag":194,"props":5282,"children":5283},{},[5284],{"type":42,"tag":67,"props":5285,"children":5286},{},[5287,5289,5295,5296,5302,5303],{"type":48,"value":5288},"Examples: ",{"type":42,"tag":73,"props":5290,"children":5292},{"className":5291},[],[5293],{"type":48,"value":5294},"task_completion",{"type":48,"value":251},{"type":42,"tag":73,"props":5297,"children":5299},{"className":5298},[],[5300],{"type":48,"value":5301},"answer_correctness",{"type":48,"value":251},{"type":42,"tag":73,"props":5304,"children":5306},{"className":5305},[],[5307],{"type":48,"value":5308},"response_groundedness",{"type":42,"tag":67,"props":5310,"children":5311},{},[5312,5317,5319],{"type":42,"tag":55,"props":5313,"children":5314},{},[5315],{"type":48,"value":5316},"Format evaluators",{"type":48,"value":5318}," — Does the output meet structural requirements?",{"type":42,"tag":194,"props":5320,"children":5321},{},[5322],{"type":42,"tag":67,"props":5323,"children":5324},{},[5325,5326,5332,5333,5339,5340],{"type":48,"value":5288},{"type":42,"tag":73,"props":5327,"children":5329},{"className":5328},[],[5330],{"type":48,"value":5331},"valid_json_output",{"type":48,"value":251},{"type":42,"tag":73,"props":5334,"children":5336},{"className":5335},[],[5337],{"type":48,"value":5338},"response_length",{"type":48,"value":251},{"type":42,"tag":73,"props":5341,"children":5343},{"className":5342},[],[5344],{"type":48,"value":5345},"citation_format",{"type":42,"tag":67,"props":5347,"children":5348},{},[5349,5354,5356],{"type":42,"tag":55,"props":5350,"children":5351},{},[5352],{"type":48,"value":5353},"Safety evaluators",{"type":48,"value":5355}," — Does the output stay within appropriate boundaries?",{"type":42,"tag":194,"props":5357,"children":5358},{},[5359],{"type":42,"tag":67,"props":5360,"children":5361},{},[5362,5363,5369,5370,5376,5377],{"type":48,"value":5288},{"type":42,"tag":73,"props":5364,"children":5366},{"className":5365},[],[5367],{"type":48,"value":5368},"no_pii_leakage",{"type":48,"value":251},{"type":42,"tag":73,"props":5371,"children":5373},{"className":5372},[],[5374],{"type":48,"value":5375},"scope_adherence",{"type":48,"value":251},{"type":42,"tag":73,"props":5378,"children":5380},{"className":5379},[],[5381],{"type":48,"value":5382},"no_hallucination",{"type":42,"tag":5384,"props":5385,"children":5387},"h5",{"id":5386},"how-many-evaluators-to-propose",[5388],{"type":48,"value":5389},"How many evaluators to propose",{"type":42,"tag":51,"props":5391,"children":5392},{},[5393,5395,5401],{"type":48,"value":5394},"The default ",{"type":42,"tag":73,"props":5396,"children":5398},{"className":5397},[],[5399],{"type":48,"value":5400},"4-6",{"type":48,"value":5402}," cap from the older skill version was too tight — it pushed the skill toward generic evaluators only and left domain signals on the table. Updated guidance:",{"type":42,"tag":194,"props":5404,"children":5405},{},[5406,5416,5434],{"type":42,"tag":67,"props":5407,"children":5408},{},[5409,5414],{"type":42,"tag":55,"props":5410,"children":5411},{},[5412],{"type":48,"value":5413},"Aim for 8–15 evaluators",{"type":48,"value":5415}," in the proposal, distributed across all four categories (with domain-specific usually the largest bucket, outcome second, format and safety smaller). For very simple single-LLM-call apps, fewer is fine; for agent \u002F RAG apps with rich domain signals, lean toward the upper end.",{"type":42,"tag":67,"props":5417,"children":5418},{},[5419,5424,5426,5432],{"type":42,"tag":55,"props":5420,"children":5421},{},[5422],{"type":48,"value":5423},"Quality > generic",{"type":48,"value":5425},": every domain-specific proposal should be backed by at least one observed pattern in the sampled traces. Don't invent generic domain evaluators (\"",{"type":42,"tag":73,"props":5427,"children":5429},{"className":5428},[],[5430],{"type":48,"value":5431},"response_quality",{"type":48,"value":5433},"\") if you don't have evidence for them.",{"type":42,"tag":67,"props":5435,"children":5436},{},[5437,5442,5444,5449],{"type":42,"tag":55,"props":5438,"children":5439},{},[5440],{"type":48,"value":5441},"Let the user curate",{"type":48,"value":5443},": the MANDATORY CHECKPOINT below explicitly asks the user to ",{"type":42,"tag":55,"props":5445,"children":5446},{},[5447],{"type":48,"value":5448},"remove",{"type":48,"value":5450}," what doesn't apply, not just to approve. Treat the proposal as a candidate set the user trims.",{"type":42,"tag":3896,"props":5452,"children":5454},{"id":5453},"deduplication-against-existing-coverage",[5455],{"type":48,"value":5456},"Deduplication Against Existing Coverage",{"type":42,"tag":51,"props":5458,"children":5459},{},[5460,5470],{"type":42,"tag":55,"props":5461,"children":5462},{},[5463,5464,5469],{"type":48,"value":3795},{"type":42,"tag":73,"props":5465,"children":5467},{"className":5466},[],[5468],{"type":48,"value":618},{"type":48,"value":1064},{"type":48,"value":5471},": skip this section entirely (coverage map was not built in Phase 0). Proceed directly to the proposal table.",{"type":42,"tag":51,"props":5473,"children":5474},{},[5475,5477,5490,5492,5497],{"type":48,"value":5476},"Before building the proposal, apply the coverage map from Phase 0. ",{"type":42,"tag":55,"props":5478,"children":5479},{},[5480,5482,5488],{"type":48,"value":5481},"Coverage is keyed on ",{"type":42,"tag":73,"props":5483,"children":5485},{"className":5484},[],[5486],{"type":48,"value":5487},"(dimension, scope)",{"type":48,"value":5489}," — not on dimension alone",{"type":48,"value":5491},": every OOTB evaluator runs at span scope, and an enabled OOTB eval does NOT preclude proposing a trace-scope ",{"type":42,"tag":55,"props":5493,"children":5494},{},[5495],{"type":48,"value":5496},"or session-scope",{"type":48,"value":5498}," evaluator for the same dimension. The three scopes answer different questions.",{"type":42,"tag":63,"props":5500,"children":5501},{},[5502,5569,5585,5617,5627],{"type":42,"tag":67,"props":5503,"children":5504},{},[5505,5510,5512],{"type":42,"tag":55,"props":5506,"children":5507},{},[5508],{"type":48,"value":5509},"Enabled span-scope eval (OOTB or custom)",{"type":48,"value":5511}," for dimension D:",{"type":42,"tag":194,"props":5513,"children":5514},{},[5515,5527],{"type":42,"tag":67,"props":5516,"children":5517},{},[5518,5520,5525],{"type":48,"value":5519},"Do NOT propose a new ",{"type":42,"tag":55,"props":5521,"children":5522},{},[5523],{"type":48,"value":5524},"span-scope",{"type":48,"value":5526}," evaluator for D — that dimension is already covered at span scope.",{"type":42,"tag":67,"props":5528,"children":5529},{},[5530,5532,5537,5538,5543,5545,5551,5553,5559,5561,5567],{"type":48,"value":5531},"DO propose a ",{"type":42,"tag":55,"props":5533,"children":5534},{},[5535],{"type":48,"value":5536},"trace-scope",{"type":48,"value":5132},{"type":42,"tag":55,"props":5539,"children":5540},{},[5541],{"type":48,"value":5542},"session-scope",{"type":48,"value":5544}," evaluator for D when the trace or session shape calls for it (multi-step app, or multi-trace session — judgment depends on cross-span or cross-trace context). Note the relationship in the rationale: e.g., \"OOTB ",{"type":42,"tag":73,"props":5546,"children":5548},{"className":5547},[],[5549],{"type":48,"value":5550},"Goal Completeness",{"type":48,"value":5552}," evaluates each LLM span in isolation; this trace-scope ",{"type":42,"tag":73,"props":5554,"children":5556},{"className":5555},[],[5557],{"type":48,"value":5558},"goal_completion",{"type":48,"value":5560}," checks whether the agent's full sequence of steps achieved the user's request, and a session-scope ",{"type":42,"tag":73,"props":5562,"children":5564},{"className":5563},[],[5565],{"type":48,"value":5566},"session_goal_completion",{"type":48,"value":5568}," checks it across the whole conversation — three different questions.\"",{"type":42,"tag":67,"props":5570,"children":5571},{},[5572,5577,5579,5583],{"type":42,"tag":55,"props":5573,"children":5574},{},[5575],{"type":48,"value":5576},"Enabled trace-scope custom eval",{"type":48,"value":5578}," for dimension D: do NOT propose another trace-scope evaluator for the same dimension; that's a real duplicate. Span-scope on the same dimension is still fair game if the data also fits a single span, and session-scope is fair game if the dimension also needs cross-trace context. Likewise, an enabled ",{"type":42,"tag":55,"props":5580,"children":5581},{},[5582],{"type":48,"value":5542},{"type":48,"value":5584}," custom eval for D blocks only another session-scope eval for D — span and trace scope remain fair game.",{"type":42,"tag":67,"props":5586,"children":5587},{},[5588,5593,5595],{"type":42,"tag":55,"props":5589,"children":5590},{},[5591],{"type":48,"value":5592},"Disabled OOTB eval",{"type":48,"value":5594},": Do NOT propose a new custom span-scope evaluator for that dimension. Instead, surface it in a short note within the proposal and suggest enabling it in the Datadog UI rather than creating a duplicate. Example:",{"type":42,"tag":159,"props":5596,"children":5597},{},[5598],{"type":42,"tag":51,"props":5599,"children":5600},{},[5601,5607,5609,5615],{"type":42,"tag":73,"props":5602,"children":5604},{"className":5603},[],[5605],{"type":48,"value":5606},"hallucination",{"type":48,"value":5608}," (ootb, disabled) — consider enabling in Datadog UI (Evaluations → Configure) instead of creating a custom span-scope eval. (A trace-scope ",{"type":42,"tag":73,"props":5610,"children":5612},{"className":5611},[],[5613],{"type":48,"value":5614},"rag_faithfulness",{"type":48,"value":5616}," is still in scope and covers a different question.)",{"type":42,"tag":67,"props":5618,"children":5619},{},[5620,5625],{"type":42,"tag":55,"props":5621,"children":5622},{},[5623],{"type":48,"value":5624},"Gap identification",{"type":48,"value":5626},": Open the proposal with a coverage summary line: \"Existing coverage: N evaluator(s) already configured ({names}, all span-scope unless noted). Proposing evaluators for uncovered dimensions and uncovered scopes.\"",{"type":42,"tag":67,"props":5628,"children":5629},{},[5630,5635],{"type":42,"tag":55,"props":5631,"children":5632},{},[5633],{"type":48,"value":5634},"All dimensions covered",{"type":48,"value":5636},": A dimension is \"fully covered\" only when the relevant scopes are present (span, plus trace and\u002For session where the app shape calls for them). If the coverage map accounts for every identified quality dimension at the appropriate scope(s), surface this explicitly and ask the user what they want: (a) review\u002Fimprove existing eval prompts, (b) add coverage for additional dimensions, or (c) proceed anyway.",{"type":42,"tag":51,"props":5638,"children":5639},{},[5640],{"type":48,"value":5641},"For each proposed evaluator:",{"type":42,"tag":194,"props":5643,"children":5644},{},[5645,5663,5731,5741,5831,5862,5924,5934,6016],{"type":42,"tag":67,"props":5646,"children":5647},{},[5648,5653,5655,5661],{"type":42,"tag":55,"props":5649,"children":5650},{},[5651],{"type":48,"value":5652},"Name",{"type":48,"value":5654},": Must match ",{"type":42,"tag":73,"props":5656,"children":5658},{"className":5657},[],[5659],{"type":48,"value":5660},"^[a-zA-Z0-9_-]+$",{"type":48,"value":5662}," (alphanumeric, underscore, hyphen only)",{"type":42,"tag":67,"props":5664,"children":5665},{},[5666,5671,5672,5677,5679,5684,5685,5690,5692,5697,5699],{"type":42,"tag":55,"props":5667,"children":5668},{},[5669],{"type":48,"value":5670},"Type",{"type":48,"value":1665},{"type":42,"tag":73,"props":5673,"children":5675},{"className":5674},[],[5676],{"type":48,"value":597},{"type":48,"value":5678}," (Boolean\u002FScore\u002FCategorical\u002Fcustom JSON schema), built-in (",{"type":42,"tag":73,"props":5680,"children":5682},{"className":5681},[],[5683],{"type":48,"value":3102},{"type":48,"value":251},{"type":42,"tag":73,"props":5686,"children":5688},{"className":5687},[],[5689],{"type":48,"value":3119},{"type":48,"value":5691},", etc.), or ",{"type":42,"tag":73,"props":5693,"children":5695},{"className":5694},[],[5696],{"type":48,"value":589},{"type":48,"value":5698}," subclass. ",{"type":42,"tag":462,"props":5700,"children":5701},{},[5702,5703,5708,5710,5715,5717,5722,5724,5729],{"type":48,"value":3795},{"type":42,"tag":73,"props":5704,"children":5706},{"className":5705},[],[5707],{"type":48,"value":458},{"type":48,"value":5709}," mode, only LLM-judge evaluators are supported by the MCP tool — code-based checks must NOT be silently dropped. List them in the same proposal table with ",{"type":42,"tag":73,"props":5711,"children":5713},{"className":5712},[],[5714],{"type":48,"value":5670},{"type":48,"value":5716}," set to the code-based class, mark them under a \"Not publishable in this mode\" subsection of the proposal, and tell the user they can get them as offline code on request (",{"type":42,"tag":73,"props":5718,"children":5720},{"className":5719},[],[5721],{"type":48,"value":571},{"type":48,"value":5723},", or ask after the publish run) or as a ",{"type":42,"tag":73,"props":5725,"children":5727},{"className":5726},[],[5728],{"type":48,"value":629},{"type":48,"value":5730}," spec. Treat the code-based proposals as part of the suite for counting and coverage purposes.",{"type":42,"tag":67,"props":5732,"children":5733},{},[5734,5739],{"type":42,"tag":55,"props":5735,"children":5736},{},[5737],{"type":48,"value":5738},"What it measures",{"type":48,"value":5740},": 1-2 sentence plain-language description",{"type":42,"tag":67,"props":5742,"children":5743},{},[5744,5749,5751,5757,5759,5764,5766],{"type":42,"tag":55,"props":5745,"children":5746},{},[5747],{"type":48,"value":5748},"Target span",{"type":48,"value":5750},": Which span's data the evaluator was designed for (e.g., \"root agent span\", \"LLM sub-span ",{"type":42,"tag":73,"props":5752,"children":5754},{"className":5753},[],[5755],{"type":48,"value":5756},"anthropic.request",{"type":48,"value":5758},"\", \"all ",{"type":42,"tag":73,"props":5760,"children":5762},{"className":5761},[],[5763],{"type":48,"value":18},{"type":48,"value":5765}," spans\"). If the root span's I\u002FO is too lossy for the quality dimension (e.g., tool call results aren't visible), note this and specify which sub-span has the signal. ",{"type":42,"tag":462,"props":5767,"children":5768},{},[5769,5770,5775,5777,5782,5783,5788,5789,5795,5796,5802,5803,5808,5810,5815,5817,5823,5824,5830],{"type":48,"value":3795},{"type":42,"tag":73,"props":5771,"children":5773},{"className":5772},[],[5774],{"type":48,"value":458},{"type":48,"value":5776}," mode this maps to a combination of ",{"type":42,"tag":73,"props":5778,"children":5780},{"className":5779},[],[5781],{"type":48,"value":5050},{"type":48,"value":497},{"type":42,"tag":73,"props":5784,"children":5786},{"className":5785},[],[5787],{"type":48,"value":1825},{"type":48,"value":2683},{"type":42,"tag":73,"props":5790,"children":5792},{"className":5791},[],[5793],{"type":48,"value":5794},"trace",{"type":48,"value":2683},{"type":42,"tag":73,"props":5797,"children":5799},{"className":5798},[],[5800],{"type":48,"value":5801},"session",{"type":48,"value":3511},{"type":42,"tag":73,"props":5804,"children":5806},{"className":5805},[],[5807],{"type":48,"value":1603},{"type":48,"value":5809},", and the EVP ",{"type":42,"tag":73,"props":5811,"children":5813},{"className":5812},[],[5814],{"type":48,"value":5065},{"type":48,"value":5816}," query (e.g. ",{"type":42,"tag":73,"props":5818,"children":5820},{"className":5819},[],[5821],{"type":48,"value":5822},"@meta.span.kind:llm",{"type":48,"value":5132},{"type":42,"tag":73,"props":5825,"children":5827},{"className":5826},[],[5828],{"type":48,"value":5829},"service:web",{"type":48,"value":220},{"type":42,"tag":67,"props":5832,"children":5833},{},[5834,5839,5840,5846,5847,5853,5854,5860],{"type":42,"tag":55,"props":5835,"children":5836},{},[5837],{"type":48,"value":5838},"Pass\u002Ffail criteria",{"type":48,"value":1665},{"type":42,"tag":73,"props":5841,"children":5843},{"className":5842},[],[5844],{"type":48,"value":5845},"pass_when=True",{"type":48,"value":251},{"type":42,"tag":73,"props":5848,"children":5850},{"className":5849},[],[5851],{"type":48,"value":5852},"min_threshold=7",{"type":48,"value":251},{"type":42,"tag":73,"props":5855,"children":5857},{"className":5856},[],[5858],{"type":48,"value":5859},"pass_values=[\"correct\"]",{"type":48,"value":5861},", or \"no automatic assessment\" for custom JSON schema",{"type":42,"tag":67,"props":5863,"children":5864},{},[5865,5869,5871,5876,5877,5882,5883,5888,5889,5895,5897,5902,5903,5908,5909,5915,5916,5922],{"type":42,"tag":55,"props":5866,"children":5867},{},[5868],{"type":48,"value":2270},{"type":48,"value":5870},": Which of ",{"type":42,"tag":73,"props":5872,"children":5874},{"className":5873},[],[5875],{"type":48,"value":2072},{"type":48,"value":251},{"type":42,"tag":73,"props":5878,"children":5880},{"className":5879},[],[5881],{"type":48,"value":2079},{"type":48,"value":251},{"type":42,"tag":73,"props":5884,"children":5886},{"className":5885},[],[5887],{"type":48,"value":4771},{"type":48,"value":251},{"type":42,"tag":73,"props":5890,"children":5892},{"className":5891},[],[5893],{"type":48,"value":5894},"metadata.*",{"type":48,"value":5896}," it uses (offline) — or which span paths \u002F aliases it pulls from (publish mode: ",{"type":42,"tag":73,"props":5898,"children":5900},{"className":5899},[],[5901],{"type":48,"value":3824},{"type":48,"value":251},{"type":42,"tag":73,"props":5904,"children":5906},{"className":5905},[],[5907],{"type":48,"value":3831},{"type":48,"value":251},{"type":42,"tag":73,"props":5910,"children":5912},{"className":5911},[],[5913],{"type":48,"value":5914},"{{meta.input.messages[*].content}}",{"type":48,"value":251},{"type":42,"tag":73,"props":5917,"children":5919},{"className":5918},[],[5920],{"type":48,"value":5921},"{{meta.metadata.\u003Ckey>}}",{"type":48,"value":5923},", etc.)",{"type":42,"tag":67,"props":5925,"children":5926},{},[5927,5932],{"type":42,"tag":55,"props":5928,"children":5929},{},[5930],{"type":48,"value":5931},"Evidence",{"type":48,"value":5933},": At least one trace where it would have caught a failure (or confirmed correct behavior)",{"type":42,"tag":67,"props":5935,"children":5936},{},[5937,5942,5943,5954,5955,5961,5962,5968,5969,5975,5976,5982,5983,5988,5989,5995,5996,6001,6002,6007,6009,6014],{"type":42,"tag":55,"props":5938,"children":5939},{},[5940],{"type":48,"value":5941},"Publish-only fields",{"type":48,"value":460},{"type":42,"tag":462,"props":5944,"children":5945},{},[5946,5948,5953],{"type":48,"value":5947},"(only in ",{"type":42,"tag":73,"props":5949,"children":5951},{"className":5950},[],[5952],{"type":48,"value":458},{"type":48,"value":3950},{"type":48,"value":1665},{"type":42,"tag":73,"props":5956,"children":5958},{"className":5957},[],[5959],{"type":48,"value":5960},"integration_provider",{"type":48,"value":3504},{"type":42,"tag":73,"props":5963,"children":5965},{"className":5964},[],[5966],{"type":48,"value":5967},"openai",{"type":48,"value":3511},{"type":42,"tag":73,"props":5970,"children":5972},{"className":5971},[],[5973],{"type":48,"value":5974},"model_name",{"type":48,"value":3504},{"type":42,"tag":73,"props":5977,"children":5979},{"className":5978},[],[5980],{"type":48,"value":5981},"gpt-5.4-mini",{"type":48,"value":3511},{"type":42,"tag":73,"props":5984,"children":5986},{"className":5985},[],[5987],{"type":48,"value":5057},{"type":48,"value":3504},{"type":42,"tag":73,"props":5990,"children":5992},{"className":5991},[],[5993],{"type":48,"value":5994},"10",{"type":48,"value":3511},{"type":42,"tag":73,"props":5997,"children":5999},{"className":5998},[],[6000],{"type":48,"value":5050},{"type":48,"value":3504},{"type":42,"tag":73,"props":6003,"children":6005},{"className":6004},[],[6006],{"type":48,"value":1825},{"type":48,"value":6008},"), and any ",{"type":42,"tag":73,"props":6010,"children":6012},{"className":6011},[],[6013],{"type":48,"value":5065},{"type":48,"value":6015}," query needed to scope to the right spans. Surface defaults in the proposal so the user can override before publishing.",{"type":42,"tag":67,"props":6017,"children":6018},{},[6019,6028,6029,6039,6041,6053],{"type":42,"tag":55,"props":6020,"children":6021},{},[6022],{"type":42,"tag":73,"props":6023,"children":6025},{"className":6024},[],[6026],{"type":48,"value":6027},"integration_account_id",{"type":48,"value":460},{"type":42,"tag":462,"props":6030,"children":6031},{},[6032,6033,6038],{"type":48,"value":5947},{"type":42,"tag":73,"props":6034,"children":6036},{"className":6035},[],[6037],{"type":48,"value":458},{"type":48,"value":3950},{"type":48,"value":6040},": the integration account the judge LLM is called through. Auto-detected from existing evaluators in the same ml_app (Phase 0 coverage map). Never asked from the user as a raw UUID. If no existing evaluator has one, the field is omitted and the user picks an account in the UI before activating. ",{"type":42,"tag":55,"props":6042,"children":6043},{},[6044,6046,6051],{"type":48,"value":6045},"All evaluators are published with ",{"type":42,"tag":73,"props":6047,"children":6049},{"className":6048},[],[6050],{"type":48,"value":503},{"type":48,"value":6052}," regardless",{"type":48,"value":6054}," — see \"Always publish as draft\" in Phase 3C for the full activation workflow.",{"type":42,"tag":3896,"props":6056,"children":6058},{"id":6057},"span-vs-trace-vs-session-scope-classification-publish-mode",[6059,6061,6066],{"type":48,"value":6060},"Span vs. Trace vs. Session Scope Classification (",{"type":42,"tag":73,"props":6062,"children":6064},{"className":6063},[],[6065],{"type":48,"value":458},{"type":48,"value":3950},{"type":42,"tag":51,"props":6068,"children":6069},{},[6070],{"type":42,"tag":55,"props":6071,"children":6072},{},[6073],{"type":48,"value":6074},"Don't ask the user; classify per evaluator and let them override at the checkpoint.",{"type":42,"tag":5384,"props":6076,"children":6078},{"id":6077},"mandatory-walk-the-four-canonical-trace-scope-use-cases-first",[6079],{"type":48,"value":6080},"Mandatory: walk the four canonical trace-scope use cases first",{"type":42,"tag":51,"props":6082,"children":6083},{},[6084,6086,6091,6092,6097,6098,6103,6105,6110,6112,6117,6119,6124,6126,6131,6133,6138],{"type":48,"value":6085},"If Phase 1 found multi-step traces (≥ 2 span kinds, or any ",{"type":42,"tag":73,"props":6087,"children":6089},{"className":6088},[],[6090],{"type":48,"value":4199},{"type":48,"value":591},{"type":42,"tag":73,"props":6093,"children":6095},{"className":6094},[],[6096],{"type":48,"value":4206},{"type":48,"value":591},{"type":42,"tag":73,"props":6099,"children":6101},{"className":6100},[],[6102],{"type":48,"value":3388},{"type":48,"value":6104}," span under an ",{"type":42,"tag":73,"props":6106,"children":6108},{"className":6107},[],[6109],{"type":48,"value":4150},{"type":48,"value":6111}," root), you ",{"type":42,"tag":55,"props":6113,"children":6114},{},[6115],{"type":48,"value":6116},"MUST",{"type":48,"value":6118}," walk through the four canonical trace-scope use cases below before finalizing the suite. For each, decide explicitly: ",{"type":42,"tag":55,"props":6120,"children":6121},{},[6122],{"type":48,"value":6123},"applies",{"type":48,"value":6125}," (include with ",{"type":42,"tag":73,"props":6127,"children":6129},{"className":6128},[],[6130],{"type":48,"value":4895},{"type":48,"value":6132},") or ",{"type":42,"tag":55,"props":6134,"children":6135},{},[6136],{"type":48,"value":6137},"does not apply",{"type":48,"value":6139}," (record a one-line reason in a \"Skipped trace-scope candidates\" subsection of the proposal). Skipping all four without per-item justification is a sign you've over-anchored on span scope — re-check.",{"type":42,"tag":754,"props":6141,"children":6142},{},[6143,6159],{"type":42,"tag":758,"props":6144,"children":6145},{},[6146],{"type":42,"tag":762,"props":6147,"children":6148},{},[6149,6154],{"type":42,"tag":766,"props":6150,"children":6151},{},[6152],{"type":48,"value":6153},"Canonical use case",{"type":42,"tag":766,"props":6155,"children":6156},{},[6157],{"type":48,"value":6158},"Triggers when",{"type":42,"tag":787,"props":6160,"children":6161},{},[6162,6180,6206,6229],{"type":42,"tag":762,"props":6163,"children":6164},{},[6165,6175],{"type":42,"tag":794,"props":6166,"children":6167},{},[6168,6173],{"type":42,"tag":73,"props":6169,"children":6171},{"className":6170},[],[6172],{"type":48,"value":5558},{"type":48,"value":6174}," — did the agent finish the user's request?",{"type":42,"tag":794,"props":6176,"children":6177},{},[6178],{"type":48,"value":6179},"Any agent \u002F multi-step app. Almost always applies.",{"type":42,"tag":762,"props":6181,"children":6182},{},[6183,6194],{"type":42,"tag":794,"props":6184,"children":6185},{},[6186,6192],{"type":42,"tag":73,"props":6187,"children":6189},{"className":6188},[],[6190],{"type":48,"value":6191},"tool_use_correctness",{"type":48,"value":6193}," — right tool with right arguments?",{"type":42,"tag":794,"props":6195,"children":6196},{},[6197,6199,6204],{"type":48,"value":6198},"Trace contains ",{"type":42,"tag":73,"props":6200,"children":6202},{"className":6201},[],[6203],{"type":48,"value":4199},{"type":48,"value":6205}," kind spans.",{"type":42,"tag":762,"props":6207,"children":6208},{},[6209,6219],{"type":42,"tag":794,"props":6210,"children":6211},{},[6212,6217],{"type":42,"tag":73,"props":6213,"children":6215},{"className":6214},[],[6216],{"type":48,"value":5614},{"type":48,"value":6218}," — answer grounded in retrieved documents?",{"type":42,"tag":794,"props":6220,"children":6221},{},[6222,6223,6228],{"type":48,"value":6198},{"type":42,"tag":73,"props":6224,"children":6226},{"className":6225},[],[6227],{"type":48,"value":4206},{"type":48,"value":6205},{"type":42,"tag":762,"props":6230,"children":6231},{},[6232,6243],{"type":42,"tag":794,"props":6233,"children":6234},{},[6235,6241],{"type":42,"tag":73,"props":6236,"children":6238},{"className":6237},[],[6239],{"type":48,"value":6240},"conversation_quality",{"type":48,"value":6242}," — coherence across multi-turn LLM calls?",{"type":42,"tag":794,"props":6244,"children":6245},{},[6246,6248,6253],{"type":48,"value":6247},"Trace contains ≥ 2 ",{"type":42,"tag":73,"props":6249,"children":6251},{"className":6250},[],[6252],{"type":48,"value":18},{"type":48,"value":6254}," spans, or app instruments multi-turn sessions.",{"type":42,"tag":5384,"props":6256,"children":6258},{"id":6257},"mandatory-walk-the-canonical-session-scope-use-cases-only-when-sessions_present",[6259,6261,6266],{"type":48,"value":6260},"Mandatory: walk the canonical session-scope use cases (only when ",{"type":42,"tag":73,"props":6262,"children":6264},{"className":6263},[],[6265],{"type":48,"value":4276},{"type":48,"value":631},{"type":42,"tag":51,"props":6268,"children":6269},{},[6270,6275,6277,6282,6284,6289,6291,6296],{"type":42,"tag":55,"props":6271,"children":6272},{},[6273],{"type":48,"value":6274},"Gate:",{"type":48,"value":6276}," perform this walk-through ",{"type":42,"tag":55,"props":6278,"children":6279},{},[6280],{"type":48,"value":6281},"only if",{"type":48,"value":6283}," the Phase 1 session probe set ",{"type":42,"tag":73,"props":6285,"children":6287},{"className":6286},[],[6288],{"type":48,"value":3978},{"type":48,"value":6290},". If sessions are absent, skip session scope entirely and note \"session scope skipped — no ",{"type":42,"tag":73,"props":6292,"children":6294},{"className":6293},[],[6295],{"type":48,"value":545},{"type":48,"value":6297}," on sampled spans\" in the proposal.",{"type":42,"tag":51,"props":6299,"children":6300},{},[6301,6303,6308,6310,6314,6316,6320,6321,6326,6327,6331,6333,6338,6340,6345],{"type":48,"value":6302},"When ",{"type":42,"tag":73,"props":6304,"children":6306},{"className":6305},[],[6307],{"type":48,"value":4276},{"type":48,"value":6309},", you ",{"type":42,"tag":55,"props":6311,"children":6312},{},[6313],{"type":48,"value":6116},{"type":48,"value":6315}," walk through the four canonical session-scope use cases below. For each, decide explicitly: ",{"type":42,"tag":55,"props":6317,"children":6318},{},[6319],{"type":48,"value":6123},{"type":48,"value":6125},{"type":42,"tag":73,"props":6322,"children":6324},{"className":6323},[],[6325],{"type":48,"value":4903},{"type":48,"value":6132},{"type":42,"tag":55,"props":6328,"children":6329},{},[6330],{"type":48,"value":6137},{"type":48,"value":6332}," (one-line reason in a \"Skipped session-scope candidates\" subsection). Session scope answers questions that span ",{"type":42,"tag":55,"props":6334,"children":6335},{},[6336],{"type":48,"value":6337},"more than one trace",{"type":48,"value":6339}," under the same ",{"type":42,"tag":73,"props":6341,"children":6343},{"className":6342},[],[6344],{"type":48,"value":545},{"type":48,"value":6346}," — a single trace cannot see prior or later turns.",{"type":42,"tag":754,"props":6348,"children":6349},{},[6350,6365],{"type":42,"tag":758,"props":6351,"children":6352},{},[6353],{"type":42,"tag":762,"props":6354,"children":6355},{},[6356,6361],{"type":42,"tag":766,"props":6357,"children":6358},{},[6359],{"type":48,"value":6360},"Canonical session use case",{"type":42,"tag":766,"props":6362,"children":6363},{},[6364],{"type":48,"value":6158},{"type":42,"tag":787,"props":6366,"children":6367},{},[6368,6393,6412,6431],{"type":42,"tag":762,"props":6369,"children":6370},{},[6371,6381],{"type":42,"tag":794,"props":6372,"children":6373},{},[6374,6379],{"type":42,"tag":73,"props":6375,"children":6377},{"className":6376},[],[6378],{"type":48,"value":5566},{"type":48,"value":6380}," — were the user's goals met across the whole session?",{"type":42,"tag":794,"props":6382,"children":6383},{},[6384,6386,6391],{"type":48,"value":6385},"A ",{"type":42,"tag":73,"props":6387,"children":6389},{"className":6388},[],[6390],{"type":48,"value":545},{"type":48,"value":6392}," spans ≥ 2 traces. Almost always applies for multi-trace sessions.",{"type":42,"tag":762,"props":6394,"children":6395},{},[6396,6407],{"type":42,"tag":794,"props":6397,"children":6398},{},[6399,6405],{"type":42,"tag":73,"props":6400,"children":6402},{"className":6401},[],[6403],{"type":48,"value":6404},"multi_turn_conversation_quality",{"type":48,"value":6406}," — coherence, memory, and consistent tone across turns",{"type":42,"tag":794,"props":6408,"children":6409},{},[6410],{"type":48,"value":6411},"Multi-trace chat \u002F assistant sessions.",{"type":42,"tag":762,"props":6413,"children":6414},{},[6415,6426],{"type":42,"tag":794,"props":6416,"children":6417},{},[6418,6424],{"type":42,"tag":73,"props":6419,"children":6421},{"className":6420},[],[6422],{"type":48,"value":6423},"user_frustration_signals",{"type":48,"value":6425}," — frustration, confusion, repetition, or abandonment over the session",{"type":42,"tag":794,"props":6427,"children":6428},{},[6429],{"type":48,"value":6430},"Any multi-turn session (repeated or rephrased asks across traces).",{"type":42,"tag":762,"props":6432,"children":6433},{},[6434,6445],{"type":42,"tag":794,"props":6435,"children":6436},{},[6437,6443],{"type":42,"tag":73,"props":6438,"children":6440},{"className":6439},[],[6441],{"type":48,"value":6442},"agent_consistency_across_session",{"type":48,"value":6444}," — did the agent stay consistent and recover from errors across traces?",{"type":42,"tag":794,"props":6446,"children":6447},{},[6448],{"type":48,"value":6449},"Agent app whose sessions span ≥ 2 traces.",{"type":42,"tag":51,"props":6451,"children":6452},{},[6453],{"type":48,"value":6454},"For other proposed evaluators (e.g. tone, format, safety), apply this scope test in order:",{"type":42,"tag":63,"props":6456,"children":6457},{},[6458,6494,6556],{"type":42,"tag":67,"props":6459,"children":6460},{},[6461,6463,6468,6470,6476,6477,6483,6485,6493],{"type":48,"value":6462},"Can the judgment be answered correctly from ",{"type":42,"tag":55,"props":6464,"children":6465},{},[6466],{"type":48,"value":6467},"one",{"type":48,"value":6469}," span's ",{"type":42,"tag":73,"props":6471,"children":6473},{"className":6472},[],[6474],{"type":48,"value":6475},"meta.input",{"type":48,"value":3171},{"type":42,"tag":73,"props":6478,"children":6480},{"className":6479},[],[6481],{"type":48,"value":6482},"meta.output",{"type":48,"value":6484},", where \"correctly\" means the verdict cannot change if you considered other spans in the trace? → ",{"type":42,"tag":55,"props":6486,"children":6487},{},[6488],{"type":42,"tag":73,"props":6489,"children":6491},{"className":6490},[],[6492],{"type":48,"value":4867},{"type":48,"value":332},{"type":42,"tag":67,"props":6495,"children":6496},{},[6497,6499,6504,6505,6513,6515,6520,6521,6526,6527,6531,6532,6537,6538,6543,6544,6549,6550,6554],{"type":48,"value":6498},"Otherwise, if it is answerable from the spans of a ",{"type":42,"tag":55,"props":6500,"children":6501},{},[6502],{"type":48,"value":6503},"single trace",{"type":48,"value":297},{"type":42,"tag":55,"props":6506,"children":6507},{},[6508],{"type":42,"tag":73,"props":6509,"children":6511},{"className":6510},[],[6512],{"type":48,"value":4895},{"type":48,"value":6514},". Default to trace when the evaluator name contains ",{"type":42,"tag":462,"props":6516,"children":6517},{},[6518],{"type":48,"value":6519},"grounding",{"type":48,"value":251},{"type":42,"tag":462,"props":6522,"children":6523},{},[6524],{"type":48,"value":6525},"faithfulness",{"type":48,"value":251},{"type":42,"tag":462,"props":6528,"children":6529},{},[6530],{"type":48,"value":5606},{"type":48,"value":251},{"type":42,"tag":462,"props":6533,"children":6534},{},[6535],{"type":48,"value":6536},"completeness",{"type":48,"value":251},{"type":42,"tag":462,"props":6539,"children":6540},{},[6541],{"type":48,"value":6542},"correctness across steps",{"type":48,"value":251},{"type":42,"tag":462,"props":6545,"children":6546},{},[6547],{"type":48,"value":6548},"consistency",{"type":48,"value":532},{"type":42,"tag":462,"props":6551,"children":6552},{},[6553],{"type":48,"value":3388},{"type":48,"value":6555}," — these almost always need cross-span context.",{"type":42,"tag":67,"props":6557,"children":6558},{},[6559,6561,6565,6567,6572,6574,6579,6580,6585,6586,6594,6596,6600,6601,6606,6607,6612,6613,6618,6619,6624,6625,6630,6631,6636,6638,6643],{"type":48,"value":6560},"Otherwise, if the verdict needs context from ",{"type":42,"tag":55,"props":6562,"children":6563},{},[6564],{"type":48,"value":6337},{"type":48,"value":6566}," in the same ",{"type":42,"tag":73,"props":6568,"children":6570},{"className":6569},[],[6571],{"type":48,"value":545},{"type":48,"value":6573}," (overall satisfaction, behavior over time, multi-turn coherence) ",{"type":42,"tag":55,"props":6575,"children":6576},{},[6577],{"type":48,"value":6578},"and",{"type":48,"value":460},{"type":42,"tag":73,"props":6581,"children":6583},{"className":6582},[],[6584],{"type":48,"value":4276},{"type":48,"value":297},{"type":42,"tag":55,"props":6587,"children":6588},{},[6589],{"type":42,"tag":73,"props":6590,"children":6592},{"className":6591},[],[6593],{"type":48,"value":4903},{"type":48,"value":6595},". Default to session when the name contains ",{"type":42,"tag":462,"props":6597,"children":6598},{},[6599],{"type":48,"value":5801},{"type":48,"value":251},{"type":42,"tag":462,"props":6602,"children":6603},{},[6604],{"type":48,"value":6605},"conversation",{"type":48,"value":251},{"type":42,"tag":462,"props":6608,"children":6609},{},[6610],{"type":48,"value":6611},"across turns",{"type":48,"value":251},{"type":42,"tag":462,"props":6614,"children":6615},{},[6616],{"type":48,"value":6617},"over time",{"type":48,"value":251},{"type":42,"tag":462,"props":6620,"children":6621},{},[6622],{"type":48,"value":6623},"satisfaction",{"type":48,"value":251},{"type":42,"tag":462,"props":6626,"children":6627},{},[6628],{"type":48,"value":6629},"frustration",{"type":48,"value":532},{"type":42,"tag":462,"props":6632,"children":6633},{},[6634],{"type":48,"value":6635},"abandonment",{"type":48,"value":6637},". If ",{"type":42,"tag":73,"props":6639,"children":6641},{"className":6640},[],[6642],{"type":48,"value":4276},{"type":48,"value":6644}," is false, fall back to trace scope and note the limitation.",{"type":42,"tag":5384,"props":6646,"children":6648},{"id":6647},"trade-offs-dont-let-these-dominate-the-choice",[6649],{"type":48,"value":6650},"Trade-offs (don't let these dominate the choice)",{"type":42,"tag":51,"props":6652,"children":6653},{},[6654,6656,6661,6663,6668,6670,6675,6677,6682,6684,6689,6691,6696,6697,6702,6704,6709],{"type":48,"value":6655},"Trace scope costs more than span scope: one judgment per ",{"type":42,"tag":55,"props":6657,"children":6658},{},[6659],{"type":48,"value":6660},"completed",{"type":48,"value":6662}," trace (vs. per matching span), larger prompt payloads, and a 3-minute trigger latency (Datadog waits 3 minutes of inactivity before considering a trace complete; later spans are excluded). Session scope costs the most: one judgment per ",{"type":42,"tag":55,"props":6664,"children":6665},{},[6666],{"type":48,"value":6667},"completed session",{"type":48,"value":6669}," (a ",{"type":42,"tag":73,"props":6671,"children":6673},{"className":6672},[],[6674],{"type":48,"value":545},{"type":48,"value":6676}," is complete after ",{"type":42,"tag":55,"props":6678,"children":6679},{},[6680],{"type":48,"value":6681},"30 minutes of inactivity",{"type":48,"value":6683}," — vs. 3 minutes for a trace — and spans arriving > 30 min after the prior span are excluded), with the largest payloads (every span of every trace in the session, capped at 10,000 spans). These are ",{"type":42,"tag":55,"props":6685,"children":6686},{},[6687],{"type":48,"value":6688},"cost-control",{"type":48,"value":6690}," levers — handle with ",{"type":42,"tag":73,"props":6692,"children":6694},{"className":6693},[],[6695],{"type":48,"value":5057},{"type":48,"value":919},{"type":42,"tag":73,"props":6698,"children":6700},{"className":6699},[],[6701],{"type":48,"value":5065},{"type":48,"value":6703},", not by demoting scope. The ",{"type":42,"tag":462,"props":6705,"children":6706},{},[6707],{"type":48,"value":6708},"correctness",{"type":48,"value":6710}," of the eval is what picks the scope.",{"type":42,"tag":5384,"props":6712,"children":6714},{"id":6713},"surface-the-classification",[6715],{"type":48,"value":6716},"Surface the classification",{"type":42,"tag":51,"props":6718,"children":6719},{},[6720,6722,6727,6729,6733],{"type":48,"value":6721},"Add a ",{"type":42,"tag":55,"props":6723,"children":6724},{},[6725],{"type":48,"value":6726},"Scope",{"type":48,"value":6728}," column to the proposal table and a one-sentence rationale per evaluator. If you skipped a canonical trace-scope ",{"type":42,"tag":55,"props":6730,"children":6731},{},[6732],{"type":48,"value":5496},{"type":48,"value":6734}," use case, list it under the matching \"Skipped …-scope candidates\" subsection with the reason — the user will see and can override.",{"type":42,"tag":159,"props":6736,"children":6737},{},[6738,6743,6866,6871,6884,6889],{"type":42,"tag":51,"props":6739,"children":6740},{},[6741],{"type":48,"value":6742},"Example rationales:",{"type":42,"tag":194,"props":6744,"children":6745},{},[6746,6770,6792,6807,6829,6851],{"type":42,"tag":67,"props":6747,"children":6748},{},[6749,6755,6756,6760,6762,6768],{"type":42,"tag":73,"props":6750,"children":6752},{"className":6751},[],[6753],{"type":48,"value":6754},"tone_check",{"type":48,"value":1304},{"type":42,"tag":55,"props":6757,"children":6758},{},[6759],{"type":48,"value":1825},{"type":48,"value":6761},". Judging \"is this single response polite\" needs only one LLM span's ",{"type":42,"tag":73,"props":6763,"children":6765},{"className":6764},[],[6766],{"type":48,"value":6767},"meta.output.messages[*].content",{"type":48,"value":6769},"; no other span in the trace can change that verdict.",{"type":42,"tag":67,"props":6771,"children":6772},{},[6773,6778,6779,6783,6785,6790],{"type":42,"tag":73,"props":6774,"children":6776},{"className":6775},[],[6777],{"type":48,"value":5558},{"type":48,"value":1304},{"type":42,"tag":55,"props":6780,"children":6781},{},[6782],{"type":48,"value":5794},{"type":48,"value":6784},". Whether the agent finished the user's request depends on the sequence of tool calls and the final LLM response together — ",{"type":42,"tag":73,"props":6786,"children":6788},{"className":6787},[],[6789],{"type":48,"value":6482},{"type":48,"value":6791}," of any single span only shows that step's output.",{"type":42,"tag":67,"props":6793,"children":6794},{},[6795,6800,6801,6805],{"type":42,"tag":73,"props":6796,"children":6798},{"className":6797},[],[6799],{"type":48,"value":6191},{"type":48,"value":1304},{"type":42,"tag":55,"props":6802,"children":6803},{},[6804],{"type":48,"value":5794},{"type":48,"value":6806},". Comparing tool inputs against the request and the final response requires correlating ≥ 3 spans (root, tool, final LLM).",{"type":42,"tag":67,"props":6808,"children":6809},{},[6810,6815,6816,6820,6822,6827],{"type":42,"tag":73,"props":6811,"children":6813},{"className":6812},[],[6814],{"type":48,"value":5614},{"type":48,"value":1304},{"type":42,"tag":55,"props":6817,"children":6818},{},[6819],{"type":48,"value":5794},{"type":48,"value":6821},". Grounding pairs the ",{"type":42,"tag":73,"props":6823,"children":6825},{"className":6824},[],[6826],{"type":48,"value":4206},{"type":48,"value":6828}," span's documents with the LLM span's answer.",{"type":42,"tag":67,"props":6830,"children":6831},{},[6832,6837,6838,6842,6844,6849],{"type":42,"tag":73,"props":6833,"children":6835},{"className":6834},[],[6836],{"type":48,"value":5566},{"type":48,"value":1304},{"type":42,"tag":55,"props":6839,"children":6840},{},[6841],{"type":48,"value":5801},{"type":48,"value":6843},". Whether the user's overall goals were met depends on every trace in the ",{"type":42,"tag":73,"props":6845,"children":6847},{"className":6846},[],[6848],{"type":48,"value":545},{"type":48,"value":6850},", not just the last one — only session scope sees the full conversation.",{"type":42,"tag":67,"props":6852,"children":6853},{},[6854,6859,6860,6864],{"type":42,"tag":73,"props":6855,"children":6857},{"className":6856},[],[6858],{"type":48,"value":6423},{"type":48,"value":1304},{"type":42,"tag":55,"props":6861,"children":6862},{},[6863],{"type":48,"value":5801},{"type":48,"value":6865},". Frustration surfaces as repeated or rephrased asks across traces; a single trace can't reveal the pattern.",{"type":42,"tag":51,"props":6867,"children":6868},{},[6869],{"type":48,"value":6870},"Example \"Skipped trace-scope candidates\" entry:",{"type":42,"tag":194,"props":6872,"children":6873},{},[6874],{"type":42,"tag":67,"props":6875,"children":6876},{},[6877,6882],{"type":42,"tag":73,"props":6878,"children":6880},{"className":6879},[],[6881],{"type":48,"value":6240},{"type":48,"value":6883}," — skipped: traces contain a single LLM call (no multi-turn signal in this app's instrumentation).",{"type":42,"tag":51,"props":6885,"children":6886},{},[6887],{"type":48,"value":6888},"Example \"Skipped session-scope candidates\" entry:",{"type":42,"tag":194,"props":6890,"children":6891},{},[6892],{"type":42,"tag":67,"props":6893,"children":6894},{},[6895,6900,6902,6907],{"type":42,"tag":73,"props":6896,"children":6898},{"className":6897},[],[6899],{"type":48,"value":5566},{"type":48,"value":6901}," — skipped: every ",{"type":42,"tag":73,"props":6903,"children":6905},{"className":6904},[],[6906],{"type":48,"value":545},{"type":48,"value":6908}," maps to a single trace (no cross-trace context — trace scope already covers it).",{"type":42,"tag":3896,"props":6910,"children":6912},{"id":6911},"mandatory-checkpoint",[6913],{"type":48,"value":6914},"MANDATORY CHECKPOINT",{"type":42,"tag":51,"props":6916,"children":6917},{},[6918],{"type":42,"tag":55,"props":6919,"children":6920},{},[6921],{"type":48,"value":6922},"You MUST output the proposal and wait for user confirmation before proceeding.",{"type":42,"tag":730,"props":6924,"children":6927},{"className":6925,"code":6926,"language":48},[733],"## Proposed Evaluator Suite\n\n**App profile**: {LLM | RAG | Agent | Multi-agent}\n**Entry mode**: {cold_start | from_rca}\n\n| # | Name | Type | Scope | Measures | Pass Criteria |\n|---|------|------|-------|----------|---------------|\n| 1 | task_completion | LLMJudge (Boolean) | span | Whether the task was completed on this span | pass_when=True |\n| 2 | tool_use_correctness | LLMJudge (Categorical) | trace | Right tool with right arguments across the agent run | pass_values=[\"correct\"] |\n| 3 | session_goal_completion | LLMJudge (Categorical) | session | Whether the user's goals were met across the whole multi-trace session | pass_values=[\"completed\"] |\n| 4 | ... | ... | ... | ... | ... |\n\n(Drop the **Scope** column when not in `publish` mode.)\n\nFor each evaluator:\n- **{name}**: {what it measures}\n  - Target span: {which span's data it was designed for}\n  - Rationale: {which quality dimension it covers and why}\n  - {Only in publish mode:} Scope: {span | trace | session} — {one-sentence rationale}\n  - Evidence: [Trace {id_short}](https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_id})\n\n{Only in publish mode, for multi-step apps. Required if any of the four canonical trace-scope use cases was not included above:}\n\n**Skipped trace-scope candidates:**\n- `{canonical_use_case}` — {one-line reason it does not apply to this app}\n\n{Only in publish mode, when `sessions_present`. Required if any of the four canonical session-scope use cases was not included above:}\n\n**Skipped session-scope candidates:**\n- `{canonical_use_case}` — {one-line reason it does not apply, e.g. \"every `session_id` maps to a single trace\"}\n\n{Only in publish mode, when the session probe found no sessions:}\n\n**Session scope skipped** — no `session_id` on sampled spans; session-scope evaluators not proposed.\n\n{Only in publish mode, when the suite contains code-based evaluators (JSONEvaluator, RegexMatchEvaluator, LengthEvaluator, StringCheckEvaluator, BaseEvaluator). Required when any code-based proposal exists.}\n\n**Not publishable in this mode** (code-based evaluators — the publish API is LLM-judge only):\n- `{name}` ({type}) — {what it would check}. Ask me to emit these as offline SDK code (or run `\u002Feval-bootstrap {ml_app} --sdk-code`), or `\u002Feval-bootstrap {ml_app} --data-only` for a framework-agnostic JSON spec.\n",[6928],{"type":42,"tag":73,"props":6929,"children":6930},{"__ignoreMap":738},[6931],{"type":48,"value":6926},{"type":42,"tag":51,"props":6933,"children":6934},{},[6935,6940,6942,6947,6949,6954,6956,6961,6963,6968,6969,6974,6975,6980,6981,6986,6987,6992,6993,6998,7000,7005,7007,7012],{"type":42,"tag":55,"props":6936,"children":6937},{},[6938],{"type":48,"value":6939},"Which evaluators should I generate?",{"type":48,"value":6941}," Treat the proposal as a candidate set — the suite below is intentionally broad so you can pick what matters for your team's quality bar. Reply with ",{"type":42,"tag":55,"props":6943,"children":6944},{},[6945],{"type":48,"value":6946},"which to keep, which to drop, and which to rename",{"type":48,"value":6948},"; not every domain-specific proposal will fit your priorities. In ",{"type":42,"tag":73,"props":6950,"children":6952},{"className":6951},[],[6953],{"type":48,"value":559},{"type":48,"value":6955}," mode you may also add custom evaluators or change provider\u002Fmodel. In ",{"type":42,"tag":73,"props":6957,"children":6959},{"className":6958},[],[6960],{"type":48,"value":458},{"type":48,"value":6962}," mode you may override ",{"type":42,"tag":73,"props":6964,"children":6966},{"className":6965},[],[6967],{"type":48,"value":5960},{"type":48,"value":251},{"type":42,"tag":73,"props":6970,"children":6972},{"className":6971},[],[6973],{"type":48,"value":5974},{"type":48,"value":251},{"type":42,"tag":73,"props":6976,"children":6978},{"className":6977},[],[6979],{"type":48,"value":5057},{"type":48,"value":251},{"type":42,"tag":73,"props":6982,"children":6984},{"className":6983},[],[6985],{"type":48,"value":5050},{"type":48,"value":251},{"type":42,"tag":73,"props":6988,"children":6990},{"className":6989},[],[6991],{"type":48,"value":1603},{"type":48,"value":532},{"type":42,"tag":73,"props":6994,"children":6996},{"className":6995},[],[6997],{"type":48,"value":5065},{"type":48,"value":6999}," per evaluator. (In the default ",{"type":42,"tag":73,"props":7001,"children":7003},{"className":7002},[],[7004],{"type":48,"value":458},{"type":48,"value":7006}," mode these are created as ",{"type":42,"tag":55,"props":7008,"children":7009},{},[7010],{"type":48,"value":7011},"online drafts",{"type":48,"value":7013}," in Datadog on confirmation — you review and enable them in the UI. Prefer offline SDK code or a JSON spec instead? Say so and I'll generate the confirmed suite that way.)",{"type":42,"tag":51,"props":7015,"children":7016},{},[7017],{"type":48,"value":7018},"Do NOT proceed to code generation until the user confirms.",{"type":42,"tag":1773,"props":7020,"children":7021},{},[],{"type":42,"tag":747,"props":7023,"children":7025},{"id":7024},"phase-3-generate-output",[7026],{"type":48,"value":7027},"Phase 3: Generate Output",{"type":42,"tag":51,"props":7029,"children":7030},{},[7031,7033,7038],{"type":48,"value":7032},"Branch on ",{"type":42,"tag":73,"props":7034,"children":7036},{"className":7035},[],[7037],{"type":48,"value":3549},{"type":48,"value":4920},{"type":42,"tag":194,"props":7040,"children":7041},{},[7042,7062,7078,7092],{"type":42,"tag":67,"props":7043,"children":7044},{},[7045,7050,7051,7055,7057],{"type":42,"tag":73,"props":7046,"children":7048},{"className":7047},[],[7049],{"type":48,"value":458},{"type":48,"value":460},{"type":42,"tag":462,"props":7052,"children":7053},{},[7054],{"type":48,"value":466},{"type":48,"value":7056}," → skip to ",{"type":42,"tag":55,"props":7058,"children":7059},{},[7060],{"type":48,"value":7061},"Phase 3C",{"type":42,"tag":67,"props":7063,"children":7064},{},[7065,7070,7071,7076],{"type":42,"tag":73,"props":7066,"children":7068},{"className":7067},[],[7069],{"type":48,"value":559},{"type":48,"value":297},{"type":42,"tag":55,"props":7072,"children":7073},{},[7074],{"type":48,"value":7075},"Phase 3A",{"type":48,"value":7077}," below",{"type":42,"tag":67,"props":7079,"children":7080},{},[7081,7086,7087],{"type":42,"tag":73,"props":7082,"children":7084},{"className":7083},[],[7085],{"type":48,"value":618},{"type":48,"value":7056},{"type":42,"tag":55,"props":7088,"children":7089},{},[7090],{"type":48,"value":7091},"Phase 3B",{"type":42,"tag":67,"props":7093,"children":7094},{},[7095,7100,7101,7105],{"type":42,"tag":73,"props":7096,"children":7098},{"className":7097},[],[7099],{"type":48,"value":645},{"type":48,"value":7056},{"type":42,"tag":55,"props":7102,"children":7103},{},[7104],{"type":48,"value":3577},{"type":48,"value":7106}," (Phases 0 step 4, 1, and 2 are bypassed — see Phase 3D for the dataset-mode workflow)",{"type":42,"tag":3896,"props":7108,"children":7110},{"id":7109},"on-request-code-generation-after-a-publish-run",[7111],{"type":48,"value":7112},"On-request code generation (after a publish run)",{"type":42,"tag":51,"props":7114,"children":7115},{},[7116,7118,7122,7124,7129,7131,7135,7137,7141,7142,7147,7148,7152,7153,7158],{"type":48,"value":7117},"The default path publishes ",{"type":42,"tag":55,"props":7119,"children":7120},{},[7121],{"type":48,"value":473},{"type":48,"value":7123}," evaluators. If the user then asks for the suite as offline code or a portable spec (e.g. \"now generate the SDK code for these\", \"give me a JSON spec\"), ",{"type":42,"tag":55,"props":7125,"children":7126},{},[7127],{"type":48,"value":7128},"do not re-run Phase 1–2",{"type":48,"value":7130},". Reuse the ",{"type":42,"tag":55,"props":7132,"children":7133},{},[7134],{"type":48,"value":700},{"type":48,"value":7136}," evaluator suite and jump straight to ",{"type":42,"tag":55,"props":7138,"children":7139},{},[7140],{"type":48,"value":7075},{"type":48,"value":497},{"type":42,"tag":73,"props":7143,"children":7145},{"className":7144},[],[7146],{"type":48,"value":559},{"type":48,"value":6132},{"type":42,"tag":55,"props":7149,"children":7150},{},[7151],{"type":48,"value":7091},{"type":48,"value":497},{"type":42,"tag":73,"props":7154,"children":7156},{"className":7155},[],[7157],{"type":48,"value":618},{"type":48,"value":7159},"), translating each published online evaluator into the offline form:",{"type":42,"tag":194,"props":7161,"children":7162},{},[7163,7181],{"type":42,"tag":67,"props":7164,"children":7165},{},[7166,7168,7173,7174,7179],{"type":48,"value":7167},"The online prompt template (span\u002Ftrace\u002Fsession placeholders) becomes an offline LLMJudge with generic ",{"type":42,"tag":73,"props":7169,"children":7171},{"className":7170},[],[7172],{"type":48,"value":2094},{"type":48,"value":591},{"type":42,"tag":73,"props":7175,"children":7177},{"className":7176},[],[7178],{"type":48,"value":2101},{"type":48,"value":7180}," placeholders (offline data comes from the user's dataset\u002Ftask function, not spans — see the EvaluatorContext note), preserving the rubric and pass criteria.",{"type":42,"tag":67,"props":7182,"children":7183},{},[7184,7186,7191],{"type":48,"value":7185},"Code-based checks that couldn't be published online (the \"Not publishable in this mode\" set) are emitted as real ",{"type":42,"tag":73,"props":7187,"children":7189},{"className":7188},[],[7190],{"type":48,"value":589},{"type":48,"value":7192}," \u002F built-in evaluators here.",{"type":42,"tag":51,"props":7194,"children":7195},{},[7196,7198,7203,7205,7210],{"type":48,"value":7197},"This works the other way too: a user who started with ",{"type":42,"tag":73,"props":7199,"children":7201},{"className":7200},[],[7202],{"type":48,"value":571},{"type":48,"value":7204}," can ask to publish the confirmed suite online (Phase 3C). The ",{"type":42,"tag":73,"props":7206,"children":7208},{"className":7207},[],[7209],{"type":48,"value":645},{"type":48,"value":7211}," path is separate from the evaluator workflow — it never has a \"confirmed suite\" to translate.",{"type":42,"tag":1773,"props":7213,"children":7214},{},[],{"type":42,"tag":747,"props":7216,"children":7218},{"id":7217},"phase-3a-generate-write-evaluator-code",[7219],{"type":48,"value":7220},"Phase 3A: Generate & Write Evaluator Code",{"type":42,"tag":51,"props":7222,"children":7223},{},[7224,7228,7230,7235],{"type":42,"tag":55,"props":7225,"children":7226},{},[7227],{"type":48,"value":3892},{"type":48,"value":7229},": Generate the final ",{"type":42,"tag":73,"props":7231,"children":7233},{"className":7232},[],[7234],{"type":48,"value":581},{"type":48,"value":7236}," file and write it to disk.",{"type":42,"tag":51,"props":7238,"children":7239},{},[7240],{"type":48,"value":7241},"For each confirmed evaluator, generate production-quality Python code following the SDK Reference patterns above.",{"type":42,"tag":3896,"props":7243,"children":7245},{"id":7244},"code-generation-rules",[7246],{"type":48,"value":7247},"Code Generation Rules",{"type":42,"tag":63,"props":7249,"children":7250},{},[7251,7261,7323,7358,7416,7432,7441,7459],{"type":42,"tag":67,"props":7252,"children":7253},{},[7254,7259],{"type":42,"tag":55,"props":7255,"children":7256},{},[7257],{"type":48,"value":7258},"Ground prompts in traces",{"type":48,"value":7260},": LLMJudge system prompts and user prompts must reference patterns actually observed in production traces. Never write generic prompts like \"evaluate whether the response is good\" — ground them in the app's domain, observed failure patterns, and success criteria.",{"type":42,"tag":67,"props":7262,"children":7263},{},[7264,7269,7271,7276,7277,7282,7284,7290,7292],{"type":42,"tag":55,"props":7265,"children":7266},{},[7267],{"type":48,"value":7268},"Keep template variables generic, add comments for context",{"type":48,"value":7270},": Use ",{"type":42,"tag":73,"props":7272,"children":7274},{"className":7273},[],[7275],{"type":48,"value":2094},{"type":48,"value":919},{"type":42,"tag":73,"props":7278,"children":7280},{"className":7279},[],[7281],{"type":48,"value":2101},{"type":48,"value":7283}," as top-level placeholders in prompts — do NOT reference nested span paths like ",{"type":42,"tag":73,"props":7285,"children":7287},{"className":7286},[],[7288],{"type":48,"value":7289},"{{input_data.messages[-1].content}}",{"type":48,"value":7291},". The evaluator's data comes from the user's dataset and task function, not directly from spans. Instead, add a comment above each evaluator describing what data it was designed for and what the user should adapt:",{"type":42,"tag":730,"props":7293,"children":7295},{"className":1817,"code":7294,"language":1819,"meta":738,"style":738},"# Designed for: input_data = user query, output_data = assistant response text\n# Observed from: root agent span (input.value → output.value)\n# If your dataset uses a different structure, adapt the prompt references below.\n",[7296],{"type":42,"tag":73,"props":7297,"children":7298},{"__ignoreMap":738},[7299,7307,7315],{"type":42,"tag":1825,"props":7300,"children":7301},{"class":1827,"line":1828},[7302],{"type":42,"tag":1825,"props":7303,"children":7304},{},[7305],{"type":48,"value":7306},"# Designed for: input_data = user query, output_data = assistant response text\n",{"type":42,"tag":1825,"props":7308,"children":7309},{"class":1827,"line":1837},[7310],{"type":42,"tag":1825,"props":7311,"children":7312},{},[7313],{"type":48,"value":7314},"# Observed from: root agent span (input.value → output.value)\n",{"type":42,"tag":1825,"props":7316,"children":7317},{"class":1827,"line":1846},[7318],{"type":42,"tag":1825,"props":7319,"children":7320},{},[7321],{"type":48,"value":7322},"# If your dataset uses a different structure, adapt the prompt references below.\n",{"type":42,"tag":67,"props":7324,"children":7325},{},[7326,7331,7333,7338,7339,7344,7345,7350,7351,7356],{"type":42,"tag":55,"props":7327,"children":7328},{},[7329],{"type":48,"value":7330},"Use the narrowest evaluator type",{"type":48,"value":7332},": If a check can be done with ",{"type":42,"tag":73,"props":7334,"children":7336},{"className":7335},[],[7337],{"type":48,"value":3102},{"type":48,"value":251},{"type":42,"tag":73,"props":7340,"children":7342},{"className":7341},[],[7343],{"type":48,"value":3119},{"type":48,"value":251},{"type":42,"tag":73,"props":7346,"children":7348},{"className":7347},[],[7349],{"type":48,"value":3153},{"type":48,"value":532},{"type":42,"tag":73,"props":7352,"children":7354},{"className":7353},[],[7355],{"type":48,"value":3136},{"type":48,"value":7357},", do NOT use an LLMJudge. Code-based evaluators are faster, cheaper, and deterministic.",{"type":42,"tag":67,"props":7359,"children":7360},{},[7361,7366,7367],{"type":42,"tag":55,"props":7362,"children":7363},{},[7364],{"type":48,"value":7365},"BaseEvaluator subclasses",{"type":48,"value":4920},{"type":42,"tag":194,"props":7368,"children":7369},{},[7370,7388,7404],{"type":42,"tag":67,"props":7371,"children":7372},{},[7373,7375,7381,7382],{"type":48,"value":7374},"Call ",{"type":42,"tag":73,"props":7376,"children":7378},{"className":7377},[],[7379],{"type":48,"value":7380},"super().__init__(name=name)",{"type":48,"value":2272},{"type":42,"tag":73,"props":7383,"children":7385},{"className":7384},[],[7386],{"type":48,"value":7387},"__init__",{"type":42,"tag":67,"props":7389,"children":7390},{},[7391,7393,7398,7399],{"type":48,"value":7392},"Return ",{"type":42,"tag":73,"props":7394,"children":7396},{"className":7395},[],[7397],{"type":48,"value":3340},{"type":48,"value":4214},{"type":42,"tag":73,"props":7400,"children":7402},{"className":7401},[],[7403],{"type":48,"value":1968},{"type":42,"tag":67,"props":7405,"children":7406},{},[7407,7409,7414],{"type":48,"value":7408},"Do NOT modify instance attributes in ",{"type":42,"tag":73,"props":7410,"children":7412},{"className":7411},[],[7413],{"type":48,"value":1968},{"type":48,"value":7415}," (thread safety)",{"type":42,"tag":67,"props":7417,"children":7418},{},[7419,7424,7425,7430],{"type":42,"tag":55,"props":7420,"children":7421},{},[7422],{"type":48,"value":7423},"Names",{"type":48,"value":5654},{"type":42,"tag":73,"props":7426,"children":7428},{"className":7427},[],[7429],{"type":48,"value":5660},{"type":48,"value":7431},". Use snake_case descriptive names.",{"type":42,"tag":67,"props":7433,"children":7434},{},[7435,7439],{"type":42,"tag":55,"props":7436,"children":7437},{},[7438],{"type":48,"value":1814},{"type":48,"value":7440},": Consolidate at the top of the file. Only import classes that are actually used.",{"type":42,"tag":67,"props":7442,"children":7443},{},[7444,7449,7451,7457],{"type":42,"tag":55,"props":7445,"children":7446},{},[7447],{"type":48,"value":7448},"Evaluator list",{"type":48,"value":7450},": Collect all evaluators into an ",{"type":42,"tag":73,"props":7452,"children":7454},{"className":7453},[],[7455],{"type":48,"value":7456},"evaluators",{"type":48,"value":7458}," list at the bottom of the file.",{"type":42,"tag":67,"props":7460,"children":7461},{},[7462,7467],{"type":42,"tag":55,"props":7463,"children":7464},{},[7465],{"type":48,"value":7466},"Anonymize PII",{"type":48,"value":7468},": Strip emails, names, and sensitive data from any trace content included in LLMJudge prompts or the header comment.",{"type":42,"tag":3896,"props":7470,"children":7472},{"id":7471},"output-format",[7473],{"type":48,"value":7474},"Output Format",{"type":42,"tag":51,"props":7476,"children":7477},{},[7478,7480,7485],{"type":48,"value":7479},"The generated ",{"type":42,"tag":73,"props":7481,"children":7483},{"className":7482},[],[7484],{"type":48,"value":581},{"type":48,"value":7486}," file should follow this structure:",{"type":42,"tag":730,"props":7488,"children":7490},{"className":1817,"code":7489,"language":1819,"meta":738,"style":738},"\"\"\"\nAuto-generated evaluators for {ml_app}\nGenerated: {YYYY-MM-DD} by eval-bootstrap\n\nApp profile: {LLM | RAG | Agent | Multi-agent}\n\nQuality dimensions covered:\n  - {target_name}: {description}\n    Evidence: https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_id}\n  ...\n\nUsage:\n    from ddtrace.llmobs import LLMObs\n\n    experiment = LLMObs.experiment(\n        name=\"my-experiment\",\n        task=my_task_fn,\n        dataset=dataset,\n        evaluators=evaluators,\n    )\n    experiment.run()\n\"\"\"\n\n{imports — only what is used}\n\n\n# --- Outcome Evaluators ---\n\n{evaluator code}\n\n\n# --- Format Evaluators ---\n\n{evaluator code}\n\n\n# --- Safety Evaluators ---\n\n{evaluator code}\n\n\n# --- Evaluator Suite ---\n\nevaluators = [\n    {eval_1_variable_name},\n    {eval_2_variable_name},\n    ...\n]\n",[7491],{"type":42,"tag":73,"props":7492,"children":7493},{"__ignoreMap":738},[7494,7502,7510,7518,7525,7533,7540,7548,7556,7564,7572,7579,7587,7595,7602,7611,7620,7629,7638,7646,7655,7664,7672,7680,7689,7697,7705,7714,7722,7731,7739,7747,7756,7764,7772,7780,7788,7797,7805,7813,7821,7829,7838,7846,7855,7864,7873,7882],{"type":42,"tag":1825,"props":7495,"children":7496},{"class":1827,"line":1828},[7497],{"type":42,"tag":1825,"props":7498,"children":7499},{},[7500],{"type":48,"value":7501},"\"\"\"\n",{"type":42,"tag":1825,"props":7503,"children":7504},{"class":1827,"line":1837},[7505],{"type":42,"tag":1825,"props":7506,"children":7507},{},[7508],{"type":48,"value":7509},"Auto-generated evaluators for {ml_app}\n",{"type":42,"tag":1825,"props":7511,"children":7512},{"class":1827,"line":1846},[7513],{"type":42,"tag":1825,"props":7514,"children":7515},{},[7516],{"type":48,"value":7517},"Generated: {YYYY-MM-DD} by eval-bootstrap\n",{"type":42,"tag":1825,"props":7519,"children":7520},{"class":1827,"line":1856},[7521],{"type":42,"tag":1825,"props":7522,"children":7523},{"emptyLinePlaceholder":1850},[7524],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7526,"children":7527},{"class":1827,"line":1865},[7528],{"type":42,"tag":1825,"props":7529,"children":7530},{},[7531],{"type":48,"value":7532},"App profile: {LLM | RAG | Agent | Multi-agent}\n",{"type":42,"tag":1825,"props":7534,"children":7535},{"class":1827,"line":1874},[7536],{"type":42,"tag":1825,"props":7537,"children":7538},{"emptyLinePlaceholder":1850},[7539],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7541,"children":7542},{"class":1827,"line":1883},[7543],{"type":42,"tag":1825,"props":7544,"children":7545},{},[7546],{"type":48,"value":7547},"Quality dimensions covered:\n",{"type":42,"tag":1825,"props":7549,"children":7550},{"class":1827,"line":1892},[7551],{"type":42,"tag":1825,"props":7552,"children":7553},{},[7554],{"type":48,"value":7555},"  - {target_name}: {description}\n",{"type":42,"tag":1825,"props":7557,"children":7558},{"class":1827,"line":1901},[7559],{"type":42,"tag":1825,"props":7560,"children":7561},{},[7562],{"type":48,"value":7563},"    Evidence: https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_id}\n",{"type":42,"tag":1825,"props":7565,"children":7566},{"class":1827,"line":1910},[7567],{"type":42,"tag":1825,"props":7568,"children":7569},{},[7570],{"type":48,"value":7571},"  ...\n",{"type":42,"tag":1825,"props":7573,"children":7574},{"class":1827,"line":1919},[7575],{"type":42,"tag":1825,"props":7576,"children":7577},{"emptyLinePlaceholder":1850},[7578],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7580,"children":7581},{"class":1827,"line":1927},[7582],{"type":42,"tag":1825,"props":7583,"children":7584},{},[7585],{"type":48,"value":7586},"Usage:\n",{"type":42,"tag":1825,"props":7588,"children":7589},{"class":1827,"line":1936},[7590],{"type":42,"tag":1825,"props":7591,"children":7592},{},[7593],{"type":48,"value":7594},"    from ddtrace.llmobs import LLMObs\n",{"type":42,"tag":1825,"props":7596,"children":7597},{"class":1827,"line":1945},[7598],{"type":42,"tag":1825,"props":7599,"children":7600},{"emptyLinePlaceholder":1850},[7601],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7603,"children":7605},{"class":1827,"line":7604},15,[7606],{"type":42,"tag":1825,"props":7607,"children":7608},{},[7609],{"type":48,"value":7610},"    experiment = LLMObs.experiment(\n",{"type":42,"tag":1825,"props":7612,"children":7614},{"class":1827,"line":7613},16,[7615],{"type":42,"tag":1825,"props":7616,"children":7617},{},[7618],{"type":48,"value":7619},"        name=\"my-experiment\",\n",{"type":42,"tag":1825,"props":7621,"children":7623},{"class":1827,"line":7622},17,[7624],{"type":42,"tag":1825,"props":7625,"children":7626},{},[7627],{"type":48,"value":7628},"        task=my_task_fn,\n",{"type":42,"tag":1825,"props":7630,"children":7632},{"class":1827,"line":7631},18,[7633],{"type":42,"tag":1825,"props":7634,"children":7635},{},[7636],{"type":48,"value":7637},"        dataset=dataset,\n",{"type":42,"tag":1825,"props":7639,"children":7640},{"class":1827,"line":29},[7641],{"type":42,"tag":1825,"props":7642,"children":7643},{},[7644],{"type":48,"value":7645},"        evaluators=evaluators,\n",{"type":42,"tag":1825,"props":7647,"children":7649},{"class":1827,"line":7648},20,[7650],{"type":42,"tag":1825,"props":7651,"children":7652},{},[7653],{"type":48,"value":7654},"    )\n",{"type":42,"tag":1825,"props":7656,"children":7658},{"class":1827,"line":7657},21,[7659],{"type":42,"tag":1825,"props":7660,"children":7661},{},[7662],{"type":48,"value":7663},"    experiment.run()\n",{"type":42,"tag":1825,"props":7665,"children":7667},{"class":1827,"line":7666},22,[7668],{"type":42,"tag":1825,"props":7669,"children":7670},{},[7671],{"type":48,"value":7501},{"type":42,"tag":1825,"props":7673,"children":7675},{"class":1827,"line":7674},23,[7676],{"type":42,"tag":1825,"props":7677,"children":7678},{"emptyLinePlaceholder":1850},[7679],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7681,"children":7683},{"class":1827,"line":7682},24,[7684],{"type":42,"tag":1825,"props":7685,"children":7686},{},[7687],{"type":48,"value":7688},"{imports — only what is used}\n",{"type":42,"tag":1825,"props":7690,"children":7692},{"class":1827,"line":7691},25,[7693],{"type":42,"tag":1825,"props":7694,"children":7695},{"emptyLinePlaceholder":1850},[7696],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7698,"children":7700},{"class":1827,"line":7699},26,[7701],{"type":42,"tag":1825,"props":7702,"children":7703},{"emptyLinePlaceholder":1850},[7704],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7706,"children":7708},{"class":1827,"line":7707},27,[7709],{"type":42,"tag":1825,"props":7710,"children":7711},{},[7712],{"type":48,"value":7713},"# --- Outcome Evaluators ---\n",{"type":42,"tag":1825,"props":7715,"children":7717},{"class":1827,"line":7716},28,[7718],{"type":42,"tag":1825,"props":7719,"children":7720},{"emptyLinePlaceholder":1850},[7721],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7723,"children":7725},{"class":1827,"line":7724},29,[7726],{"type":42,"tag":1825,"props":7727,"children":7728},{},[7729],{"type":48,"value":7730},"{evaluator code}\n",{"type":42,"tag":1825,"props":7732,"children":7734},{"class":1827,"line":7733},30,[7735],{"type":42,"tag":1825,"props":7736,"children":7737},{"emptyLinePlaceholder":1850},[7738],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7740,"children":7742},{"class":1827,"line":7741},31,[7743],{"type":42,"tag":1825,"props":7744,"children":7745},{"emptyLinePlaceholder":1850},[7746],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7748,"children":7750},{"class":1827,"line":7749},32,[7751],{"type":42,"tag":1825,"props":7752,"children":7753},{},[7754],{"type":48,"value":7755},"# --- Format Evaluators ---\n",{"type":42,"tag":1825,"props":7757,"children":7759},{"class":1827,"line":7758},33,[7760],{"type":42,"tag":1825,"props":7761,"children":7762},{"emptyLinePlaceholder":1850},[7763],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7765,"children":7767},{"class":1827,"line":7766},34,[7768],{"type":42,"tag":1825,"props":7769,"children":7770},{},[7771],{"type":48,"value":7730},{"type":42,"tag":1825,"props":7773,"children":7775},{"class":1827,"line":7774},35,[7776],{"type":42,"tag":1825,"props":7777,"children":7778},{"emptyLinePlaceholder":1850},[7779],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7781,"children":7783},{"class":1827,"line":7782},36,[7784],{"type":42,"tag":1825,"props":7785,"children":7786},{"emptyLinePlaceholder":1850},[7787],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7789,"children":7791},{"class":1827,"line":7790},37,[7792],{"type":42,"tag":1825,"props":7793,"children":7794},{},[7795],{"type":48,"value":7796},"# --- Safety Evaluators ---\n",{"type":42,"tag":1825,"props":7798,"children":7800},{"class":1827,"line":7799},38,[7801],{"type":42,"tag":1825,"props":7802,"children":7803},{"emptyLinePlaceholder":1850},[7804],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7806,"children":7808},{"class":1827,"line":7807},39,[7809],{"type":42,"tag":1825,"props":7810,"children":7811},{},[7812],{"type":48,"value":7730},{"type":42,"tag":1825,"props":7814,"children":7816},{"class":1827,"line":7815},40,[7817],{"type":42,"tag":1825,"props":7818,"children":7819},{"emptyLinePlaceholder":1850},[7820],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7822,"children":7824},{"class":1827,"line":7823},41,[7825],{"type":42,"tag":1825,"props":7826,"children":7827},{"emptyLinePlaceholder":1850},[7828],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7830,"children":7832},{"class":1827,"line":7831},42,[7833],{"type":42,"tag":1825,"props":7834,"children":7835},{},[7836],{"type":48,"value":7837},"# --- Evaluator Suite ---\n",{"type":42,"tag":1825,"props":7839,"children":7841},{"class":1827,"line":7840},43,[7842],{"type":42,"tag":1825,"props":7843,"children":7844},{"emptyLinePlaceholder":1850},[7845],{"type":48,"value":1853},{"type":42,"tag":1825,"props":7847,"children":7849},{"class":1827,"line":7848},44,[7850],{"type":42,"tag":1825,"props":7851,"children":7852},{},[7853],{"type":48,"value":7854},"evaluators = [\n",{"type":42,"tag":1825,"props":7856,"children":7858},{"class":1827,"line":7857},45,[7859],{"type":42,"tag":1825,"props":7860,"children":7861},{},[7862],{"type":48,"value":7863},"    {eval_1_variable_name},\n",{"type":42,"tag":1825,"props":7865,"children":7867},{"class":1827,"line":7866},46,[7868],{"type":42,"tag":1825,"props":7869,"children":7870},{},[7871],{"type":48,"value":7872},"    {eval_2_variable_name},\n",{"type":42,"tag":1825,"props":7874,"children":7876},{"class":1827,"line":7875},47,[7877],{"type":42,"tag":1825,"props":7878,"children":7879},{},[7880],{"type":48,"value":7881},"    ...\n",{"type":42,"tag":1825,"props":7883,"children":7885},{"class":1827,"line":7884},48,[7886],{"type":42,"tag":1825,"props":7887,"children":7888},{},[7889],{"type":48,"value":7890},"]\n",{"type":42,"tag":51,"props":7892,"children":7893},{},[7894],{"type":48,"value":7895},"Only include section comments (Outcome\u002FFormat\u002FSafety) for categories that have evaluators.",{"type":42,"tag":3896,"props":7897,"children":7899},{"id":7898},"write-the-file",[7900],{"type":48,"value":7901},"Write the file",{"type":42,"tag":51,"props":7903,"children":7904},{},[7905,7907,7913],{"type":48,"value":7906},"Write the generated code to the output path (suggest ",{"type":42,"tag":73,"props":7908,"children":7910},{"className":7909},[],[7911],{"type":48,"value":7912},".\u002Fevals\u002F{ml_app}_evaluators.py",{"type":48,"value":7914}," if not specified), then display a summary:",{"type":42,"tag":730,"props":7916,"children":7919},{"className":7917,"code":7918,"language":48},[733],"## Generated Evaluators\n\nWrote {N} evaluators to `{output_path}`:\n\n| # | Name | Type | Covers |\n|---|------|------|--------|\n| 1 | ... | ... | ... |\n\n### Next Steps\n\n1. **Review**: Check the generated prompts and criteria match your expectations\n2. **Test offline**: Use `LLMObs.experiment(evaluators=evaluators)` to batch-evaluate against a labeled dataset and verify scores\n",[7920],{"type":42,"tag":73,"props":7921,"children":7922},{"__ignoreMap":738},[7923],{"type":48,"value":7918},{"type":42,"tag":3896,"props":7925,"children":7927},{"id":7926},"notebook-export-after-summary",[7928],{"type":48,"value":7929},"Notebook export (after summary)",{"type":42,"tag":51,"props":7931,"children":7932},{},[7933],{"type":48,"value":7934},"After displaying the summary, offer notebook export.",{"type":42,"tag":194,"props":7936,"children":7937},{},[7938,8019],{"type":42,"tag":67,"props":7939,"children":7940},{},[7941,7952,7953,7969,7972,7973,7978,7980,7986,7987,7993,7994,8000,8002,8005,8006,8011,8012,8018],{"type":42,"tag":55,"props":7942,"children":7943},{},[7944,7945,7950],{"type":48,"value":1069},{"type":42,"tag":73,"props":7946,"children":7948},{"className":7947},[],[7949],{"type":48,"value":3865},{"type":48,"value":7951}," was detected in Phase 0",{"type":48,"value":4920},{"type":42,"tag":159,"props":7954,"children":7955},{},[7956],{"type":42,"tag":51,"props":7957,"children":7958},{},[7959,7961,7967],{"type":48,"value":7960},"An RCA notebook was created earlier in this session: ",{"type":42,"tag":73,"props":7962,"children":7964},{"className":7963},[],[7965],{"type":48,"value":7966},"{rca_notebook_url}",{"type":48,"value":7968},"\nWould you like to (a) append the evaluator suite summary to that notebook, or (b) create a new standalone notebook?",{"type":42,"tag":3775,"props":7970,"children":7971},{},[],{"type":48,"value":1069},{"type":42,"tag":55,"props":7974,"children":7975},{},[7976],{"type":48,"value":7977},"append",{"type":48,"value":7979},": use the notebook creation fallback pattern (see below) with ",{"type":42,"tag":73,"props":7981,"children":7983},{"className":7982},[],[7984],{"type":48,"value":7985},"mcp__datadog-mcp__edit_datadog_notebook",{"type":48,"value":497},{"type":42,"tag":73,"props":7988,"children":7990},{"className":7989},[],[7991],{"type":48,"value":7992},"id={rca_notebook_id}",{"type":48,"value":251},{"type":42,"tag":73,"props":7995,"children":7997},{"className":7996},[],[7998],{"type":48,"value":7999},"append_only=true",{"type":48,"value":8001},", evaluator suite summary cell).",{"type":42,"tag":3775,"props":8003,"children":8004},{},[],{"type":48,"value":1069},{"type":42,"tag":55,"props":8007,"children":8008},{},[8009],{"type":48,"value":8010},"new",{"type":48,"value":7979},{"type":42,"tag":73,"props":8013,"children":8015},{"className":8014},[],[8016],{"type":48,"value":8017},"mcp__datadog-mcp__create_datadog_notebook",{"type":48,"value":332},{"type":42,"tag":67,"props":8020,"children":8021},{},[8022,8032,8033,8041,8044,8046,8051,8052],{"type":42,"tag":55,"props":8023,"children":8024},{},[8025,8027],{"type":48,"value":8026},"If no ",{"type":42,"tag":73,"props":8028,"children":8030},{"className":8029},[],[8031],{"type":48,"value":3865},{"type":48,"value":4920},{"type":42,"tag":159,"props":8034,"children":8035},{},[8036],{"type":42,"tag":51,"props":8037,"children":8038},{},[8039],{"type":48,"value":8040},"Would you like to export this evaluator suite summary to a Datadog notebook?",{"type":42,"tag":3775,"props":8042,"children":8043},{},[],{"type":48,"value":8045},"If yes: use the notebook creation fallback pattern (see below) with ",{"type":42,"tag":73,"props":8047,"children":8049},{"className":8048},[],[8050],{"type":48,"value":8017},{"type":48,"value":4920},{"type":42,"tag":194,"props":8053,"children":8054},{},[8055,8074,8093,8107],{"type":42,"tag":67,"props":8056,"children":8057},{},[8058,8067,8068],{"type":42,"tag":55,"props":8059,"children":8060},{},[8061],{"type":42,"tag":73,"props":8062,"children":8064},{"className":8063},[],[8065],{"type":48,"value":8066},"name",{"type":48,"value":1665},{"type":42,"tag":73,"props":8069,"children":8071},{"className":8070},[],[8072],{"type":48,"value":8073},"Eval Bootstrap: {ml_app} — YYYY-MM-DD",{"type":42,"tag":67,"props":8075,"children":8076},{},[8077,8086,8087],{"type":42,"tag":55,"props":8078,"children":8079},{},[8080],{"type":42,"tag":73,"props":8081,"children":8083},{"className":8082},[],[8084],{"type":48,"value":8085},"type",{"type":48,"value":1665},{"type":42,"tag":73,"props":8088,"children":8090},{"className":8089},[],[8091],{"type":48,"value":8092},"report",{"type":42,"tag":67,"props":8094,"children":8095},{},[8096,8105],{"type":42,"tag":55,"props":8097,"children":8098},{},[8099],{"type":42,"tag":73,"props":8100,"children":8102},{"className":8101},[],[8103],{"type":48,"value":8104},"cells",{"type":48,"value":8106},": single markdown cell with the evaluator suite summary",{"type":42,"tag":67,"props":8108,"children":8109},{},[8110,8119,8120],{"type":42,"tag":55,"props":8111,"children":8112},{},[8113],{"type":42,"tag":73,"props":8114,"children":8116},{"className":8115},[],[8117],{"type":48,"value":8118},"time",{"type":48,"value":1665},{"type":42,"tag":73,"props":8121,"children":8123},{"className":8122},[],[8124],{"type":48,"value":8125},"{ \"live_span\": \"1h\" }",{"type":42,"tag":51,"props":8127,"children":8128},{},[8129,8134,8136,8142,8143,8149],{"type":42,"tag":55,"props":8130,"children":8131},{},[8132],{"type":48,"value":8133},"Notebook creation fallback pattern",{"type":48,"value":8135}," (apply to every ",{"type":42,"tag":73,"props":8137,"children":8139},{"className":8138},[],[8140],{"type":48,"value":8141},"create_datadog_notebook",{"type":48,"value":591},{"type":42,"tag":73,"props":8144,"children":8146},{"className":8145},[],[8147],{"type":48,"value":8148},"edit_datadog_notebook",{"type":48,"value":8150}," call):",{"type":42,"tag":63,"props":8152,"children":8153},{},[8154,8159,8240],{"type":42,"tag":67,"props":8155,"children":8156},{},[8157],{"type":48,"value":8158},"Try the MCP tool first.",{"type":42,"tag":67,"props":8160,"children":8161},{},[8162,8167,8169],{"type":42,"tag":55,"props":8163,"children":8164},{},[8165],{"type":48,"value":8166},"If the MCP call fails",{"type":48,"value":8168},", inspect the error:\n",{"type":42,"tag":194,"props":8170,"children":8171},{},[8172,8182,8192],{"type":42,"tag":67,"props":8173,"children":8174},{},[8175,8180],{"type":42,"tag":55,"props":8176,"children":8177},{},[8178],{"type":48,"value":8179},"Auth \u002F permission error (401, 403)",{"type":48,"value":8181}," → stop and tell the user.",{"type":42,"tag":67,"props":8183,"children":8184},{},[8185,8190],{"type":42,"tag":55,"props":8186,"children":8187},{},[8188],{"type":48,"value":8189},"Field validation error",{"type":48,"value":8191}," (error names a specific field) → fix that field and retry the MCP call once.",{"type":42,"tag":67,"props":8193,"children":8194},{},[8195,8200,8202],{"type":42,"tag":55,"props":8196,"children":8197},{},[8198],{"type":48,"value":8199},"Any other error",{"type":48,"value":8201}," (binding, serialization, unexpected response) → fall back to pup:\n",{"type":42,"tag":194,"props":8203,"children":8204},{},[8205,8224,8235],{"type":42,"tag":67,"props":8206,"children":8207},{},[8208,8210,8216,8218],{"type":48,"value":8209},"Write the payload to ",{"type":42,"tag":73,"props":8211,"children":8213},{"className":8212},[],[8214],{"type":48,"value":8215},"\u002Ftmp\u002Fnb_bootstrap_{ml_app}.json",{"type":48,"value":8217}," as a full API envelope: ",{"type":42,"tag":73,"props":8219,"children":8221},{"className":8220},[],[8222],{"type":48,"value":8223},"{\"data\": {\"attributes\": {\"name\": \"...\", \"time\": {...}, \"cells\": [...]}, \"type\": \"notebooks\"}}",{"type":42,"tag":67,"props":8225,"children":8226},{},[8227,8229],{"type":48,"value":8228},"Run ",{"type":42,"tag":73,"props":8230,"children":8232},{"className":8231},[],[8233],{"type":48,"value":8234},"pup notebooks create --file \u002Ftmp\u002Fnb_bootstrap_{ml_app}.json",{"type":42,"tag":67,"props":8236,"children":8237},{},[8238],{"type":48,"value":8239},"If pup is not available either, render the notebook content as markdown in chat.",{"type":42,"tag":67,"props":8241,"children":8242},{},[8243,8245],{"type":48,"value":8244},"After successful creation by either method, output the URL:\n",{"type":42,"tag":73,"props":8246,"children":8248},{"className":8247},[],[8249],{"type":48,"value":8250},"Evaluator suite exported to notebook: \u003Curl>",{"type":42,"tag":51,"props":8252,"children":8253},{},[8254,8259],{"type":42,"tag":55,"props":8255,"children":8256},{},[8257],{"type":48,"value":8258},"Notebook cell content",{"type":48,"value":8260}," — the markdown cell should contain:",{"type":42,"tag":730,"props":8262,"children":8266},{"className":8263,"code":8264,"language":8265,"meta":738,"style":738},"language-markdown shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","## Eval Bootstrap: {ml_app}\n\n**Generated**: YYYY-MM-DD | **App profile**: {LLM | RAG | Agent | Multi-agent} | **Entry mode**: {cold_start | from_rca}\n**Generated code**: `{output_path}`\n\n{One sentence: what does this app do?}\n\n**Coverage**: {N} new evaluators ({comma-separated dimension names}) | {N} existing (unchanged: {names}) | {gaps if any: dimensions identified but not covered, and why}\n\n### Evaluator Suite\n\n| # | Name | Type | Measures | Pass Criteria |\n|---|------|------|----------|---------------|\n| 1 | ... | ... | ... | ... |\n\n### Evidence\n\n{For each evaluator: name — 1-line description — [Trace link]}\n\n### Next Steps\n\n1. Review generated prompts in `{output_path}`\n2. Run against a labeled dataset to validate scores\n3. Deploy to Datadog LLM Experiments\n","markdown",[8267],{"type":42,"tag":73,"props":8268,"children":8269},{"__ignoreMap":738},[8270,8285,8292,8353,8389,8396,8404,8411,8432,8439,8452,8459,8513,8521,8570,8577,8589,8596,8604,8611,8623,8630,8655,8668],{"type":42,"tag":1825,"props":8271,"children":8272},{"class":1827,"line":1828},[8273,8279],{"type":42,"tag":1825,"props":8274,"children":8276},{"style":8275},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[8277],{"type":48,"value":8278},"## ",{"type":42,"tag":1825,"props":8280,"children":8282},{"style":8281},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[8283],{"type":48,"value":8284},"Eval Bootstrap: {ml_app}\n",{"type":42,"tag":1825,"props":8286,"children":8287},{"class":1827,"line":1837},[8288],{"type":42,"tag":1825,"props":8289,"children":8290},{"emptyLinePlaceholder":1850},[8291],{"type":48,"value":1853},{"type":42,"tag":1825,"props":8293,"children":8294},{"class":1827,"line":1846},[8295,8301,8307,8311,8317,8321,8326,8330,8335,8339,8344,8348],{"type":42,"tag":1825,"props":8296,"children":8298},{"style":8297},"--shiki-light:#39ADB5;--shiki-light-font-weight:bold;--shiki-default:#89DDFF;--shiki-default-font-weight:bold;--shiki-dark:#89DDFF;--shiki-dark-font-weight:bold",[8299],{"type":48,"value":8300},"**",{"type":42,"tag":1825,"props":8302,"children":8304},{"style":8303},"--shiki-light:#E53935;--shiki-light-font-weight:bold;--shiki-default:#F07178;--shiki-default-font-weight:bold;--shiki-dark:#F07178;--shiki-dark-font-weight:bold",[8305],{"type":48,"value":8306},"Generated",{"type":42,"tag":1825,"props":8308,"children":8309},{"style":8297},[8310],{"type":48,"value":8300},{"type":42,"tag":1825,"props":8312,"children":8314},{"style":8313},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[8315],{"type":48,"value":8316},": YYYY-MM-DD | ",{"type":42,"tag":1825,"props":8318,"children":8319},{"style":8297},[8320],{"type":48,"value":8300},{"type":42,"tag":1825,"props":8322,"children":8323},{"style":8303},[8324],{"type":48,"value":8325},"App profile",{"type":42,"tag":1825,"props":8327,"children":8328},{"style":8297},[8329],{"type":48,"value":8300},{"type":42,"tag":1825,"props":8331,"children":8332},{"style":8313},[8333],{"type":48,"value":8334},": {LLM | RAG | Agent | Multi-agent} | ",{"type":42,"tag":1825,"props":8336,"children":8337},{"style":8297},[8338],{"type":48,"value":8300},{"type":42,"tag":1825,"props":8340,"children":8341},{"style":8303},[8342],{"type":48,"value":8343},"Entry mode",{"type":42,"tag":1825,"props":8345,"children":8346},{"style":8297},[8347],{"type":48,"value":8300},{"type":42,"tag":1825,"props":8349,"children":8350},{"style":8313},[8351],{"type":48,"value":8352},": {cold_start | from_rca}\n",{"type":42,"tag":1825,"props":8354,"children":8355},{"class":1827,"line":1856},[8356,8360,8365,8369,8373,8378,8384],{"type":42,"tag":1825,"props":8357,"children":8358},{"style":8297},[8359],{"type":48,"value":8300},{"type":42,"tag":1825,"props":8361,"children":8362},{"style":8303},[8363],{"type":48,"value":8364},"Generated code",{"type":42,"tag":1825,"props":8366,"children":8367},{"style":8297},[8368],{"type":48,"value":8300},{"type":42,"tag":1825,"props":8370,"children":8371},{"style":8313},[8372],{"type":48,"value":1665},{"type":42,"tag":1825,"props":8374,"children":8375},{"style":8275},[8376],{"type":48,"value":8377},"`",{"type":42,"tag":1825,"props":8379,"children":8381},{"style":8380},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[8382],{"type":48,"value":8383},"{output_path}",{"type":42,"tag":1825,"props":8385,"children":8386},{"style":8275},[8387],{"type":48,"value":8388},"`\n",{"type":42,"tag":1825,"props":8390,"children":8391},{"class":1827,"line":1865},[8392],{"type":42,"tag":1825,"props":8393,"children":8394},{"emptyLinePlaceholder":1850},[8395],{"type":48,"value":1853},{"type":42,"tag":1825,"props":8397,"children":8398},{"class":1827,"line":1874},[8399],{"type":42,"tag":1825,"props":8400,"children":8401},{"style":8313},[8402],{"type":48,"value":8403},"{One sentence: what does this app do?}\n",{"type":42,"tag":1825,"props":8405,"children":8406},{"class":1827,"line":1883},[8407],{"type":42,"tag":1825,"props":8408,"children":8409},{"emptyLinePlaceholder":1850},[8410],{"type":48,"value":1853},{"type":42,"tag":1825,"props":8412,"children":8413},{"class":1827,"line":1892},[8414,8418,8423,8427],{"type":42,"tag":1825,"props":8415,"children":8416},{"style":8297},[8417],{"type":48,"value":8300},{"type":42,"tag":1825,"props":8419,"children":8420},{"style":8303},[8421],{"type":48,"value":8422},"Coverage",{"type":42,"tag":1825,"props":8424,"children":8425},{"style":8297},[8426],{"type":48,"value":8300},{"type":42,"tag":1825,"props":8428,"children":8429},{"style":8313},[8430],{"type":48,"value":8431},": {N} new evaluators ({comma-separated dimension names}) | {N} existing (unchanged: {names}) | {gaps if any: dimensions identified but not covered, and why}\n",{"type":42,"tag":1825,"props":8433,"children":8434},{"class":1827,"line":1901},[8435],{"type":42,"tag":1825,"props":8436,"children":8437},{"emptyLinePlaceholder":1850},[8438],{"type":48,"value":1853},{"type":42,"tag":1825,"props":8440,"children":8441},{"class":1827,"line":1910},[8442,8447],{"type":42,"tag":1825,"props":8443,"children":8444},{"style":8275},[8445],{"type":48,"value":8446},"### ",{"type":42,"tag":1825,"props":8448,"children":8449},{"style":8281},[8450],{"type":48,"value":8451},"Evaluator Suite\n",{"type":42,"tag":1825,"props":8453,"children":8454},{"class":1827,"line":1919},[8455],{"type":42,"tag":1825,"props":8456,"children":8457},{"emptyLinePlaceholder":1850},[8458],{"type":48,"value":1853},{"type":42,"tag":1825,"props":8460,"children":8461},{"class":1827,"line":1927},[8462,8467,8472,8476,8481,8485,8490,8494,8499,8503,8508],{"type":42,"tag":1825,"props":8463,"children":8464},{"style":8275},[8465],{"type":48,"value":8466},"|",{"type":42,"tag":1825,"props":8468,"children":8469},{"style":8313},[8470],{"type":48,"value":8471}," # ",{"type":42,"tag":1825,"props":8473,"children":8474},{"style":8275},[8475],{"type":48,"value":8466},{"type":42,"tag":1825,"props":8477,"children":8478},{"style":8313},[8479],{"type":48,"value":8480}," Name ",{"type":42,"tag":1825,"props":8482,"children":8483},{"style":8275},[8484],{"type":48,"value":8466},{"type":42,"tag":1825,"props":8486,"children":8487},{"style":8313},[8488],{"type":48,"value":8489}," Type ",{"type":42,"tag":1825,"props":8491,"children":8492},{"style":8275},[8493],{"type":48,"value":8466},{"type":42,"tag":1825,"props":8495,"children":8496},{"style":8313},[8497],{"type":48,"value":8498}," Measures ",{"type":42,"tag":1825,"props":8500,"children":8501},{"style":8275},[8502],{"type":48,"value":8466},{"type":42,"tag":1825,"props":8504,"children":8505},{"style":8313},[8506],{"type":48,"value":8507}," Pass Criteria ",{"type":42,"tag":1825,"props":8509,"children":8510},{"style":8275},[8511],{"type":48,"value":8512},"|\n",{"type":42,"tag":1825,"props":8514,"children":8515},{"class":1827,"line":1936},[8516],{"type":42,"tag":1825,"props":8517,"children":8518},{"style":8275},[8519],{"type":48,"value":8520},"|---|------|------|----------|---------------|\n",{"type":42,"tag":1825,"props":8522,"children":8523},{"class":1827,"line":1945},[8524,8528,8533,8537,8542,8546,8550,8554,8558,8562,8566],{"type":42,"tag":1825,"props":8525,"children":8526},{"style":8275},[8527],{"type":48,"value":8466},{"type":42,"tag":1825,"props":8529,"children":8530},{"style":8313},[8531],{"type":48,"value":8532}," 1 ",{"type":42,"tag":1825,"props":8534,"children":8535},{"style":8275},[8536],{"type":48,"value":8466},{"type":42,"tag":1825,"props":8538,"children":8539},{"style":8313},[8540],{"type":48,"value":8541}," ... ",{"type":42,"tag":1825,"props":8543,"children":8544},{"style":8275},[8545],{"type":48,"value":8466},{"type":42,"tag":1825,"props":8547,"children":8548},{"style":8313},[8549],{"type":48,"value":8541},{"type":42,"tag":1825,"props":8551,"children":8552},{"style":8275},[8553],{"type":48,"value":8466},{"type":42,"tag":1825,"props":8555,"children":8556},{"style":8313},[8557],{"type":48,"value":8541},{"type":42,"tag":1825,"props":8559,"children":8560},{"style":8275},[8561],{"type":48,"value":8466},{"type":42,"tag":1825,"props":8563,"children":8564},{"style":8313},[8565],{"type":48,"value":8541},{"type":42,"tag":1825,"props":8567,"children":8568},{"style":8275},[8569],{"type":48,"value":8512},{"type":42,"tag":1825,"props":8571,"children":8572},{"class":1827,"line":7604},[8573],{"type":42,"tag":1825,"props":8574,"children":8575},{"emptyLinePlaceholder":1850},[8576],{"type":48,"value":1853},{"type":42,"tag":1825,"props":8578,"children":8579},{"class":1827,"line":7613},[8580,8584],{"type":42,"tag":1825,"props":8581,"children":8582},{"style":8275},[8583],{"type":48,"value":8446},{"type":42,"tag":1825,"props":8585,"children":8586},{"style":8281},[8587],{"type":48,"value":8588},"Evidence\n",{"type":42,"tag":1825,"props":8590,"children":8591},{"class":1827,"line":7622},[8592],{"type":42,"tag":1825,"props":8593,"children":8594},{"emptyLinePlaceholder":1850},[8595],{"type":48,"value":1853},{"type":42,"tag":1825,"props":8597,"children":8598},{"class":1827,"line":7631},[8599],{"type":42,"tag":1825,"props":8600,"children":8601},{"style":8313},[8602],{"type":48,"value":8603},"{For each evaluator: name — 1-line description — [Trace link]}\n",{"type":42,"tag":1825,"props":8605,"children":8606},{"class":1827,"line":29},[8607],{"type":42,"tag":1825,"props":8608,"children":8609},{"emptyLinePlaceholder":1850},[8610],{"type":48,"value":1853},{"type":42,"tag":1825,"props":8612,"children":8613},{"class":1827,"line":7648},[8614,8618],{"type":42,"tag":1825,"props":8615,"children":8616},{"style":8275},[8617],{"type":48,"value":8446},{"type":42,"tag":1825,"props":8619,"children":8620},{"style":8281},[8621],{"type":48,"value":8622},"Next Steps\n",{"type":42,"tag":1825,"props":8624,"children":8625},{"class":1827,"line":7657},[8626],{"type":42,"tag":1825,"props":8627,"children":8628},{"emptyLinePlaceholder":1850},[8629],{"type":48,"value":1853},{"type":42,"tag":1825,"props":8631,"children":8632},{"class":1827,"line":7666},[8633,8638,8643,8647,8651],{"type":42,"tag":1825,"props":8634,"children":8635},{"style":8275},[8636],{"type":48,"value":8637},"1.",{"type":42,"tag":1825,"props":8639,"children":8640},{"style":8313},[8641],{"type":48,"value":8642}," Review generated prompts in ",{"type":42,"tag":1825,"props":8644,"children":8645},{"style":8275},[8646],{"type":48,"value":8377},{"type":42,"tag":1825,"props":8648,"children":8649},{"style":8380},[8650],{"type":48,"value":8383},{"type":42,"tag":1825,"props":8652,"children":8653},{"style":8275},[8654],{"type":48,"value":8388},{"type":42,"tag":1825,"props":8656,"children":8657},{"class":1827,"line":7674},[8658,8663],{"type":42,"tag":1825,"props":8659,"children":8660},{"style":8275},[8661],{"type":48,"value":8662},"2.",{"type":42,"tag":1825,"props":8664,"children":8665},{"style":8313},[8666],{"type":48,"value":8667}," Run against a labeled dataset to validate scores\n",{"type":42,"tag":1825,"props":8669,"children":8670},{"class":1827,"line":7682},[8671,8676],{"type":42,"tag":1825,"props":8672,"children":8673},{"style":8275},[8674],{"type":48,"value":8675},"3.",{"type":42,"tag":1825,"props":8677,"children":8678},{"style":8313},[8679],{"type":48,"value":8680}," Deploy to Datadog LLM Experiments\n",{"type":42,"tag":1773,"props":8682,"children":8683},{},[],{"type":42,"tag":747,"props":8685,"children":8687},{"id":8686},"phase-3b-generate-write-eval-spec-json",[8688],{"type":48,"value":8689},"Phase 3B: Generate & Write Eval Spec JSON",{"type":42,"tag":51,"props":8691,"children":8692},{},[8693,8697],{"type":42,"tag":55,"props":8694,"children":8695},{},[8696],{"type":48,"value":3892},{"type":48,"value":8698},": Serialize the confirmed evaluator suite and representative trace samples to a single self-contained JSON file — zero SDK dependencies.",{"type":42,"tag":51,"props":8700,"children":8701},{},[8702,8707,8708],{"type":42,"tag":55,"props":8703,"children":8704},{},[8705],{"type":48,"value":8706},"Output path",{"type":48,"value":1665},{"type":42,"tag":73,"props":8709,"children":8711},{"className":8710},[],[8712],{"type":48,"value":8713},".\u002Fevals\u002F{ml_app}_eval_spec.json",{"type":42,"tag":3896,"props":8715,"children":8717},{"id":8716},"json-schema",[8718],{"type":48,"value":8719},"JSON Schema",{"type":42,"tag":730,"props":8721,"children":8725},{"className":8722,"code":8723,"language":8724,"meta":738,"style":738},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","{\n  \"schema_version\": \"1\",\n  \"generated_at\": \"\u003CISO 8601 UTC>\",\n  \"generated_by\": \"eval-bootstrap\",\n  \"app\": {\n    \"ml_app\": \"\u003Cstring>\",\n    \"app_type\": \"LLM | RAG | Agent | Multi-agent\",\n    \"trace_window\": \"\u003Ctimeframe param, e.g. now-7d>\",\n    \"trace_count\": \"\u003Cinteger>\"\n  },\n  \"evaluators\": [\n    {\n      \"name\": \"snake_case_name\",\n      \"category\": \"outcome | format | safety\",\n      \"type\": \"llm_judge | code_check\",\n      \"description\": \"\u003C1-2 sentence plain-language description>\",\n      \"target_span\": \"\u003Cwhich span: root, llm sub-span, etc.>\",\n      \"scoring\": {\n        \"scale\": \"boolean | score_1_10 | categorical\",\n        \"categories\": [\"\u003Conly present when scale=categorical>\"],\n        \"pass_criteria\": \"\u003Chuman-readable: true, >= 7, in [correct], etc.>\"\n      },\n      \"rubric\": \"\u003Cfull prompt text for llm_judge; null for code_check>\",\n      \"implementation_hints\": {\n        \"type_if_code_check\": \"json_valid | regex | contains | length_words | null\",\n        \"pattern_if_code_check\": \"\u003Cpattern string or null>\",\n        \"notes\": \"\u003Coptional framework-agnostic implementation guidance>\"\n      },\n      \"evidence\": [\n        {\n          \"trace_id\": \"\u003C32-char hex>\",\n          \"span_id\": \"\u003C16-char hex>\",\n          \"url\": \"https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:\u003Ctrace_id>\",\n          \"observation\": \"\u003Cwhy this trace illustrates the evaluator>\"\n        }\n      ]\n    }\n  ],\n  \"sample_records\": [\n    {\n      \"trace_id\": \"\u003Cstring>\",\n      \"span_id\": \"\u003Cstring>\",\n      \"input\": {},\n      \"output\": \"\u003Cstring>\",\n      \"suggested_labels\": {\n        \"\u003Cevaluator_name>\": \"pass | fail | \u003Cscore>\"\n      }\n    }\n  ]\n}\n","json",[8726],{"type":42,"tag":73,"props":8727,"children":8728},{"__ignoreMap":738},[8729,8737,8779,8816,8853,8878,8915,8952,8989,9023,9031,9055,9063,9100,9137,9173,9209,9246,9270,9309,9352,9385,9393,9430,9454,9491,9528,9561,9568,9592,9600,9637,9674,9711,9744,9752,9760,9768,9776,9800,9807,9842,9877,9901,9936,9960,9993,10001,10008,10017],{"type":42,"tag":1825,"props":8730,"children":8731},{"class":1827,"line":1828},[8732],{"type":42,"tag":1825,"props":8733,"children":8734},{"style":8275},[8735],{"type":48,"value":8736},"{\n",{"type":42,"tag":1825,"props":8738,"children":8739},{"class":1827,"line":1837},[8740,8745,8751,8756,8760,8765,8770,8774],{"type":42,"tag":1825,"props":8741,"children":8742},{"style":8275},[8743],{"type":48,"value":8744},"  \"",{"type":42,"tag":1825,"props":8746,"children":8748},{"style":8747},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[8749],{"type":48,"value":8750},"schema_version",{"type":42,"tag":1825,"props":8752,"children":8753},{"style":8275},[8754],{"type":48,"value":8755},"\"",{"type":42,"tag":1825,"props":8757,"children":8758},{"style":8275},[8759],{"type":48,"value":4920},{"type":42,"tag":1825,"props":8761,"children":8762},{"style":8275},[8763],{"type":48,"value":8764}," \"",{"type":42,"tag":1825,"props":8766,"children":8767},{"style":8380},[8768],{"type":48,"value":8769},"1",{"type":42,"tag":1825,"props":8771,"children":8772},{"style":8275},[8773],{"type":48,"value":8755},{"type":42,"tag":1825,"props":8775,"children":8776},{"style":8275},[8777],{"type":48,"value":8778},",\n",{"type":42,"tag":1825,"props":8780,"children":8781},{"class":1827,"line":1846},[8782,8786,8791,8795,8799,8803,8808,8812],{"type":42,"tag":1825,"props":8783,"children":8784},{"style":8275},[8785],{"type":48,"value":8744},{"type":42,"tag":1825,"props":8787,"children":8788},{"style":8747},[8789],{"type":48,"value":8790},"generated_at",{"type":42,"tag":1825,"props":8792,"children":8793},{"style":8275},[8794],{"type":48,"value":8755},{"type":42,"tag":1825,"props":8796,"children":8797},{"style":8275},[8798],{"type":48,"value":4920},{"type":42,"tag":1825,"props":8800,"children":8801},{"style":8275},[8802],{"type":48,"value":8764},{"type":42,"tag":1825,"props":8804,"children":8805},{"style":8380},[8806],{"type":48,"value":8807},"\u003CISO 8601 UTC>",{"type":42,"tag":1825,"props":8809,"children":8810},{"style":8275},[8811],{"type":48,"value":8755},{"type":42,"tag":1825,"props":8813,"children":8814},{"style":8275},[8815],{"type":48,"value":8778},{"type":42,"tag":1825,"props":8817,"children":8818},{"class":1827,"line":1856},[8819,8823,8828,8832,8836,8840,8845,8849],{"type":42,"tag":1825,"props":8820,"children":8821},{"style":8275},[8822],{"type":48,"value":8744},{"type":42,"tag":1825,"props":8824,"children":8825},{"style":8747},[8826],{"type":48,"value":8827},"generated_by",{"type":42,"tag":1825,"props":8829,"children":8830},{"style":8275},[8831],{"type":48,"value":8755},{"type":42,"tag":1825,"props":8833,"children":8834},{"style":8275},[8835],{"type":48,"value":4920},{"type":42,"tag":1825,"props":8837,"children":8838},{"style":8275},[8839],{"type":48,"value":8764},{"type":42,"tag":1825,"props":8841,"children":8842},{"style":8380},[8843],{"type":48,"value":8844},"eval-bootstrap",{"type":42,"tag":1825,"props":8846,"children":8847},{"style":8275},[8848],{"type":48,"value":8755},{"type":42,"tag":1825,"props":8850,"children":8851},{"style":8275},[8852],{"type":48,"value":8778},{"type":42,"tag":1825,"props":8854,"children":8855},{"class":1827,"line":1865},[8856,8860,8865,8869,8873],{"type":42,"tag":1825,"props":8857,"children":8858},{"style":8275},[8859],{"type":48,"value":8744},{"type":42,"tag":1825,"props":8861,"children":8862},{"style":8747},[8863],{"type":48,"value":8864},"app",{"type":42,"tag":1825,"props":8866,"children":8867},{"style":8275},[8868],{"type":48,"value":8755},{"type":42,"tag":1825,"props":8870,"children":8871},{"style":8275},[8872],{"type":48,"value":4920},{"type":42,"tag":1825,"props":8874,"children":8875},{"style":8275},[8876],{"type":48,"value":8877}," {\n",{"type":42,"tag":1825,"props":8879,"children":8880},{"class":1827,"line":1874},[8881,8886,8890,8894,8898,8902,8907,8911],{"type":42,"tag":1825,"props":8882,"children":8883},{"style":8275},[8884],{"type":48,"value":8885},"    \"",{"type":42,"tag":1825,"props":8887,"children":8888},{"style":8281},[8889],{"type":48,"value":802},{"type":42,"tag":1825,"props":8891,"children":8892},{"style":8275},[8893],{"type":48,"value":8755},{"type":42,"tag":1825,"props":8895,"children":8896},{"style":8275},[8897],{"type":48,"value":4920},{"type":42,"tag":1825,"props":8899,"children":8900},{"style":8275},[8901],{"type":48,"value":8764},{"type":42,"tag":1825,"props":8903,"children":8904},{"style":8380},[8905],{"type":48,"value":8906},"\u003Cstring>",{"type":42,"tag":1825,"props":8908,"children":8909},{"style":8275},[8910],{"type":48,"value":8755},{"type":42,"tag":1825,"props":8912,"children":8913},{"style":8275},[8914],{"type":48,"value":8778},{"type":42,"tag":1825,"props":8916,"children":8917},{"class":1827,"line":1883},[8918,8922,8927,8931,8935,8939,8944,8948],{"type":42,"tag":1825,"props":8919,"children":8920},{"style":8275},[8921],{"type":48,"value":8885},{"type":42,"tag":1825,"props":8923,"children":8924},{"style":8281},[8925],{"type":48,"value":8926},"app_type",{"type":42,"tag":1825,"props":8928,"children":8929},{"style":8275},[8930],{"type":48,"value":8755},{"type":42,"tag":1825,"props":8932,"children":8933},{"style":8275},[8934],{"type":48,"value":4920},{"type":42,"tag":1825,"props":8936,"children":8937},{"style":8275},[8938],{"type":48,"value":8764},{"type":42,"tag":1825,"props":8940,"children":8941},{"style":8380},[8942],{"type":48,"value":8943},"LLM | RAG | Agent | Multi-agent",{"type":42,"tag":1825,"props":8945,"children":8946},{"style":8275},[8947],{"type":48,"value":8755},{"type":42,"tag":1825,"props":8949,"children":8950},{"style":8275},[8951],{"type":48,"value":8778},{"type":42,"tag":1825,"props":8953,"children":8954},{"class":1827,"line":1892},[8955,8959,8964,8968,8972,8976,8981,8985],{"type":42,"tag":1825,"props":8956,"children":8957},{"style":8275},[8958],{"type":48,"value":8885},{"type":42,"tag":1825,"props":8960,"children":8961},{"style":8281},[8962],{"type":48,"value":8963},"trace_window",{"type":42,"tag":1825,"props":8965,"children":8966},{"style":8275},[8967],{"type":48,"value":8755},{"type":42,"tag":1825,"props":8969,"children":8970},{"style":8275},[8971],{"type":48,"value":4920},{"type":42,"tag":1825,"props":8973,"children":8974},{"style":8275},[8975],{"type":48,"value":8764},{"type":42,"tag":1825,"props":8977,"children":8978},{"style":8380},[8979],{"type":48,"value":8980},"\u003Ctimeframe param, e.g. now-7d>",{"type":42,"tag":1825,"props":8982,"children":8983},{"style":8275},[8984],{"type":48,"value":8755},{"type":42,"tag":1825,"props":8986,"children":8987},{"style":8275},[8988],{"type":48,"value":8778},{"type":42,"tag":1825,"props":8990,"children":8991},{"class":1827,"line":1901},[8992,8996,9001,9005,9009,9013,9018],{"type":42,"tag":1825,"props":8993,"children":8994},{"style":8275},[8995],{"type":48,"value":8885},{"type":42,"tag":1825,"props":8997,"children":8998},{"style":8281},[8999],{"type":48,"value":9000},"trace_count",{"type":42,"tag":1825,"props":9002,"children":9003},{"style":8275},[9004],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9006,"children":9007},{"style":8275},[9008],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9010,"children":9011},{"style":8275},[9012],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9014,"children":9015},{"style":8380},[9016],{"type":48,"value":9017},"\u003Cinteger>",{"type":42,"tag":1825,"props":9019,"children":9020},{"style":8275},[9021],{"type":48,"value":9022},"\"\n",{"type":42,"tag":1825,"props":9024,"children":9025},{"class":1827,"line":1910},[9026],{"type":42,"tag":1825,"props":9027,"children":9028},{"style":8275},[9029],{"type":48,"value":9030},"  },\n",{"type":42,"tag":1825,"props":9032,"children":9033},{"class":1827,"line":1919},[9034,9038,9042,9046,9050],{"type":42,"tag":1825,"props":9035,"children":9036},{"style":8275},[9037],{"type":48,"value":8744},{"type":42,"tag":1825,"props":9039,"children":9040},{"style":8747},[9041],{"type":48,"value":7456},{"type":42,"tag":1825,"props":9043,"children":9044},{"style":8275},[9045],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9047,"children":9048},{"style":8275},[9049],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9051,"children":9052},{"style":8275},[9053],{"type":48,"value":9054}," [\n",{"type":42,"tag":1825,"props":9056,"children":9057},{"class":1827,"line":1927},[9058],{"type":42,"tag":1825,"props":9059,"children":9060},{"style":8275},[9061],{"type":48,"value":9062},"    {\n",{"type":42,"tag":1825,"props":9064,"children":9065},{"class":1827,"line":1936},[9066,9071,9075,9079,9083,9087,9092,9096],{"type":42,"tag":1825,"props":9067,"children":9068},{"style":8275},[9069],{"type":48,"value":9070},"      \"",{"type":42,"tag":1825,"props":9072,"children":9073},{"style":8281},[9074],{"type":48,"value":8066},{"type":42,"tag":1825,"props":9076,"children":9077},{"style":8275},[9078],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9080,"children":9081},{"style":8275},[9082],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9084,"children":9085},{"style":8275},[9086],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9088,"children":9089},{"style":8380},[9090],{"type":48,"value":9091},"snake_case_name",{"type":42,"tag":1825,"props":9093,"children":9094},{"style":8275},[9095],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9097,"children":9098},{"style":8275},[9099],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9101,"children":9102},{"class":1827,"line":1945},[9103,9107,9112,9116,9120,9124,9129,9133],{"type":42,"tag":1825,"props":9104,"children":9105},{"style":8275},[9106],{"type":48,"value":9070},{"type":42,"tag":1825,"props":9108,"children":9109},{"style":8281},[9110],{"type":48,"value":9111},"category",{"type":42,"tag":1825,"props":9113,"children":9114},{"style":8275},[9115],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9117,"children":9118},{"style":8275},[9119],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9121,"children":9122},{"style":8275},[9123],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9125,"children":9126},{"style":8380},[9127],{"type":48,"value":9128},"outcome | format | safety",{"type":42,"tag":1825,"props":9130,"children":9131},{"style":8275},[9132],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9134,"children":9135},{"style":8275},[9136],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9138,"children":9139},{"class":1827,"line":7604},[9140,9144,9148,9152,9156,9160,9165,9169],{"type":42,"tag":1825,"props":9141,"children":9142},{"style":8275},[9143],{"type":48,"value":9070},{"type":42,"tag":1825,"props":9145,"children":9146},{"style":8281},[9147],{"type":48,"value":8085},{"type":42,"tag":1825,"props":9149,"children":9150},{"style":8275},[9151],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9153,"children":9154},{"style":8275},[9155],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9157,"children":9158},{"style":8275},[9159],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9161,"children":9162},{"style":8380},[9163],{"type":48,"value":9164},"llm_judge | code_check",{"type":42,"tag":1825,"props":9166,"children":9167},{"style":8275},[9168],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9170,"children":9171},{"style":8275},[9172],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9174,"children":9175},{"class":1827,"line":7613},[9176,9180,9184,9188,9192,9196,9201,9205],{"type":42,"tag":1825,"props":9177,"children":9178},{"style":8275},[9179],{"type":48,"value":9070},{"type":42,"tag":1825,"props":9181,"children":9182},{"style":8281},[9183],{"type":48,"value":2697},{"type":42,"tag":1825,"props":9185,"children":9186},{"style":8275},[9187],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9189,"children":9190},{"style":8275},[9191],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9193,"children":9194},{"style":8275},[9195],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9197,"children":9198},{"style":8380},[9199],{"type":48,"value":9200},"\u003C1-2 sentence plain-language description>",{"type":42,"tag":1825,"props":9202,"children":9203},{"style":8275},[9204],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9206,"children":9207},{"style":8275},[9208],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9210,"children":9211},{"class":1827,"line":7622},[9212,9216,9221,9225,9229,9233,9238,9242],{"type":42,"tag":1825,"props":9213,"children":9214},{"style":8275},[9215],{"type":48,"value":9070},{"type":42,"tag":1825,"props":9217,"children":9218},{"style":8281},[9219],{"type":48,"value":9220},"target_span",{"type":42,"tag":1825,"props":9222,"children":9223},{"style":8275},[9224],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9226,"children":9227},{"style":8275},[9228],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9230,"children":9231},{"style":8275},[9232],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9234,"children":9235},{"style":8380},[9236],{"type":48,"value":9237},"\u003Cwhich span: root, llm sub-span, etc.>",{"type":42,"tag":1825,"props":9239,"children":9240},{"style":8275},[9241],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9243,"children":9244},{"style":8275},[9245],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9247,"children":9248},{"class":1827,"line":7631},[9249,9253,9258,9262,9266],{"type":42,"tag":1825,"props":9250,"children":9251},{"style":8275},[9252],{"type":48,"value":9070},{"type":42,"tag":1825,"props":9254,"children":9255},{"style":8281},[9256],{"type":48,"value":9257},"scoring",{"type":42,"tag":1825,"props":9259,"children":9260},{"style":8275},[9261],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9263,"children":9264},{"style":8275},[9265],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9267,"children":9268},{"style":8275},[9269],{"type":48,"value":8877},{"type":42,"tag":1825,"props":9271,"children":9272},{"class":1827,"line":29},[9273,9278,9284,9288,9292,9296,9301,9305],{"type":42,"tag":1825,"props":9274,"children":9275},{"style":8275},[9276],{"type":48,"value":9277},"        \"",{"type":42,"tag":1825,"props":9279,"children":9281},{"style":9280},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[9282],{"type":48,"value":9283},"scale",{"type":42,"tag":1825,"props":9285,"children":9286},{"style":8275},[9287],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9289,"children":9290},{"style":8275},[9291],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9293,"children":9294},{"style":8275},[9295],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9297,"children":9298},{"style":8380},[9299],{"type":48,"value":9300},"boolean | score_1_10 | categorical",{"type":42,"tag":1825,"props":9302,"children":9303},{"style":8275},[9304],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9306,"children":9307},{"style":8275},[9308],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9310,"children":9311},{"class":1827,"line":7648},[9312,9316,9321,9325,9329,9334,9338,9343,9347],{"type":42,"tag":1825,"props":9313,"children":9314},{"style":8275},[9315],{"type":48,"value":9277},{"type":42,"tag":1825,"props":9317,"children":9318},{"style":9280},[9319],{"type":48,"value":9320},"categories",{"type":42,"tag":1825,"props":9322,"children":9323},{"style":8275},[9324],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9326,"children":9327},{"style":8275},[9328],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9330,"children":9331},{"style":8275},[9332],{"type":48,"value":9333}," [",{"type":42,"tag":1825,"props":9335,"children":9336},{"style":8275},[9337],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9339,"children":9340},{"style":8380},[9341],{"type":48,"value":9342},"\u003Conly present when scale=categorical>",{"type":42,"tag":1825,"props":9344,"children":9345},{"style":8275},[9346],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9348,"children":9349},{"style":8275},[9350],{"type":48,"value":9351},"],\n",{"type":42,"tag":1825,"props":9353,"children":9354},{"class":1827,"line":7657},[9355,9359,9364,9368,9372,9376,9381],{"type":42,"tag":1825,"props":9356,"children":9357},{"style":8275},[9358],{"type":48,"value":9277},{"type":42,"tag":1825,"props":9360,"children":9361},{"style":9280},[9362],{"type":48,"value":9363},"pass_criteria",{"type":42,"tag":1825,"props":9365,"children":9366},{"style":8275},[9367],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9369,"children":9370},{"style":8275},[9371],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9373,"children":9374},{"style":8275},[9375],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9377,"children":9378},{"style":8380},[9379],{"type":48,"value":9380},"\u003Chuman-readable: true, >= 7, in [correct], etc.>",{"type":42,"tag":1825,"props":9382,"children":9383},{"style":8275},[9384],{"type":48,"value":9022},{"type":42,"tag":1825,"props":9386,"children":9387},{"class":1827,"line":7666},[9388],{"type":42,"tag":1825,"props":9389,"children":9390},{"style":8275},[9391],{"type":48,"value":9392},"      },\n",{"type":42,"tag":1825,"props":9394,"children":9395},{"class":1827,"line":7674},[9396,9400,9405,9409,9413,9417,9422,9426],{"type":42,"tag":1825,"props":9397,"children":9398},{"style":8275},[9399],{"type":48,"value":9070},{"type":42,"tag":1825,"props":9401,"children":9402},{"style":8281},[9403],{"type":48,"value":9404},"rubric",{"type":42,"tag":1825,"props":9406,"children":9407},{"style":8275},[9408],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9410,"children":9411},{"style":8275},[9412],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9414,"children":9415},{"style":8275},[9416],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9418,"children":9419},{"style":8380},[9420],{"type":48,"value":9421},"\u003Cfull prompt text for llm_judge; null for code_check>",{"type":42,"tag":1825,"props":9423,"children":9424},{"style":8275},[9425],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9427,"children":9428},{"style":8275},[9429],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9431,"children":9432},{"class":1827,"line":7682},[9433,9437,9442,9446,9450],{"type":42,"tag":1825,"props":9434,"children":9435},{"style":8275},[9436],{"type":48,"value":9070},{"type":42,"tag":1825,"props":9438,"children":9439},{"style":8281},[9440],{"type":48,"value":9441},"implementation_hints",{"type":42,"tag":1825,"props":9443,"children":9444},{"style":8275},[9445],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9447,"children":9448},{"style":8275},[9449],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9451,"children":9452},{"style":8275},[9453],{"type":48,"value":8877},{"type":42,"tag":1825,"props":9455,"children":9456},{"class":1827,"line":7691},[9457,9461,9466,9470,9474,9478,9483,9487],{"type":42,"tag":1825,"props":9458,"children":9459},{"style":8275},[9460],{"type":48,"value":9277},{"type":42,"tag":1825,"props":9462,"children":9463},{"style":9280},[9464],{"type":48,"value":9465},"type_if_code_check",{"type":42,"tag":1825,"props":9467,"children":9468},{"style":8275},[9469],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9471,"children":9472},{"style":8275},[9473],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9475,"children":9476},{"style":8275},[9477],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9479,"children":9480},{"style":8380},[9481],{"type":48,"value":9482},"json_valid | regex | contains | length_words | null",{"type":42,"tag":1825,"props":9484,"children":9485},{"style":8275},[9486],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9488,"children":9489},{"style":8275},[9490],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9492,"children":9493},{"class":1827,"line":7699},[9494,9498,9503,9507,9511,9515,9520,9524],{"type":42,"tag":1825,"props":9495,"children":9496},{"style":8275},[9497],{"type":48,"value":9277},{"type":42,"tag":1825,"props":9499,"children":9500},{"style":9280},[9501],{"type":48,"value":9502},"pattern_if_code_check",{"type":42,"tag":1825,"props":9504,"children":9505},{"style":8275},[9506],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9508,"children":9509},{"style":8275},[9510],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9512,"children":9513},{"style":8275},[9514],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9516,"children":9517},{"style":8380},[9518],{"type":48,"value":9519},"\u003Cpattern string or null>",{"type":42,"tag":1825,"props":9521,"children":9522},{"style":8275},[9523],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9525,"children":9526},{"style":8275},[9527],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9529,"children":9530},{"class":1827,"line":7707},[9531,9535,9540,9544,9548,9552,9557],{"type":42,"tag":1825,"props":9532,"children":9533},{"style":8275},[9534],{"type":48,"value":9277},{"type":42,"tag":1825,"props":9536,"children":9537},{"style":9280},[9538],{"type":48,"value":9539},"notes",{"type":42,"tag":1825,"props":9541,"children":9542},{"style":8275},[9543],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9545,"children":9546},{"style":8275},[9547],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9549,"children":9550},{"style":8275},[9551],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9553,"children":9554},{"style":8380},[9555],{"type":48,"value":9556},"\u003Coptional framework-agnostic implementation guidance>",{"type":42,"tag":1825,"props":9558,"children":9559},{"style":8275},[9560],{"type":48,"value":9022},{"type":42,"tag":1825,"props":9562,"children":9563},{"class":1827,"line":7716},[9564],{"type":42,"tag":1825,"props":9565,"children":9566},{"style":8275},[9567],{"type":48,"value":9392},{"type":42,"tag":1825,"props":9569,"children":9570},{"class":1827,"line":7724},[9571,9575,9580,9584,9588],{"type":42,"tag":1825,"props":9572,"children":9573},{"style":8275},[9574],{"type":48,"value":9070},{"type":42,"tag":1825,"props":9576,"children":9577},{"style":8281},[9578],{"type":48,"value":9579},"evidence",{"type":42,"tag":1825,"props":9581,"children":9582},{"style":8275},[9583],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9585,"children":9586},{"style":8275},[9587],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9589,"children":9590},{"style":8275},[9591],{"type":48,"value":9054},{"type":42,"tag":1825,"props":9593,"children":9594},{"class":1827,"line":7733},[9595],{"type":42,"tag":1825,"props":9596,"children":9597},{"style":8275},[9598],{"type":48,"value":9599},"        {\n",{"type":42,"tag":1825,"props":9601,"children":9602},{"class":1827,"line":7741},[9603,9608,9612,9616,9620,9624,9629,9633],{"type":42,"tag":1825,"props":9604,"children":9605},{"style":8275},[9606],{"type":48,"value":9607},"          \"",{"type":42,"tag":1825,"props":9609,"children":9610},{"style":9280},[9611],{"type":48,"value":4005},{"type":42,"tag":1825,"props":9613,"children":9614},{"style":8275},[9615],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9617,"children":9618},{"style":8275},[9619],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9621,"children":9622},{"style":8275},[9623],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9625,"children":9626},{"style":8380},[9627],{"type":48,"value":9628},"\u003C32-char hex>",{"type":42,"tag":1825,"props":9630,"children":9631},{"style":8275},[9632],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9634,"children":9635},{"style":8275},[9636],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9638,"children":9639},{"class":1827,"line":7749},[9640,9644,9649,9653,9657,9661,9666,9670],{"type":42,"tag":1825,"props":9641,"children":9642},{"style":8275},[9643],{"type":48,"value":9607},{"type":42,"tag":1825,"props":9645,"children":9646},{"style":9280},[9647],{"type":48,"value":9648},"span_id",{"type":42,"tag":1825,"props":9650,"children":9651},{"style":8275},[9652],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9654,"children":9655},{"style":8275},[9656],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9658,"children":9659},{"style":8275},[9660],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9662,"children":9663},{"style":8380},[9664],{"type":48,"value":9665},"\u003C16-char hex>",{"type":42,"tag":1825,"props":9667,"children":9668},{"style":8275},[9669],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9671,"children":9672},{"style":8275},[9673],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9675,"children":9676},{"class":1827,"line":7758},[9677,9681,9686,9690,9694,9698,9703,9707],{"type":42,"tag":1825,"props":9678,"children":9679},{"style":8275},[9680],{"type":48,"value":9607},{"type":42,"tag":1825,"props":9682,"children":9683},{"style":9280},[9684],{"type":48,"value":9685},"url",{"type":42,"tag":1825,"props":9687,"children":9688},{"style":8275},[9689],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9691,"children":9692},{"style":8275},[9693],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9695,"children":9696},{"style":8275},[9697],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9699,"children":9700},{"style":8380},[9701],{"type":48,"value":9702},"https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:\u003Ctrace_id>",{"type":42,"tag":1825,"props":9704,"children":9705},{"style":8275},[9706],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9708,"children":9709},{"style":8275},[9710],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9712,"children":9713},{"class":1827,"line":7766},[9714,9718,9723,9727,9731,9735,9740],{"type":42,"tag":1825,"props":9715,"children":9716},{"style":8275},[9717],{"type":48,"value":9607},{"type":42,"tag":1825,"props":9719,"children":9720},{"style":9280},[9721],{"type":48,"value":9722},"observation",{"type":42,"tag":1825,"props":9724,"children":9725},{"style":8275},[9726],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9728,"children":9729},{"style":8275},[9730],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9732,"children":9733},{"style":8275},[9734],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9736,"children":9737},{"style":8380},[9738],{"type":48,"value":9739},"\u003Cwhy this trace illustrates the evaluator>",{"type":42,"tag":1825,"props":9741,"children":9742},{"style":8275},[9743],{"type":48,"value":9022},{"type":42,"tag":1825,"props":9745,"children":9746},{"class":1827,"line":7774},[9747],{"type":42,"tag":1825,"props":9748,"children":9749},{"style":8275},[9750],{"type":48,"value":9751},"        }\n",{"type":42,"tag":1825,"props":9753,"children":9754},{"class":1827,"line":7782},[9755],{"type":42,"tag":1825,"props":9756,"children":9757},{"style":8275},[9758],{"type":48,"value":9759},"      ]\n",{"type":42,"tag":1825,"props":9761,"children":9762},{"class":1827,"line":7790},[9763],{"type":42,"tag":1825,"props":9764,"children":9765},{"style":8275},[9766],{"type":48,"value":9767},"    }\n",{"type":42,"tag":1825,"props":9769,"children":9770},{"class":1827,"line":7799},[9771],{"type":42,"tag":1825,"props":9772,"children":9773},{"style":8275},[9774],{"type":48,"value":9775},"  ],\n",{"type":42,"tag":1825,"props":9777,"children":9778},{"class":1827,"line":7807},[9779,9783,9788,9792,9796],{"type":42,"tag":1825,"props":9780,"children":9781},{"style":8275},[9782],{"type":48,"value":8744},{"type":42,"tag":1825,"props":9784,"children":9785},{"style":8747},[9786],{"type":48,"value":9787},"sample_records",{"type":42,"tag":1825,"props":9789,"children":9790},{"style":8275},[9791],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9793,"children":9794},{"style":8275},[9795],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9797,"children":9798},{"style":8275},[9799],{"type":48,"value":9054},{"type":42,"tag":1825,"props":9801,"children":9802},{"class":1827,"line":7815},[9803],{"type":42,"tag":1825,"props":9804,"children":9805},{"style":8275},[9806],{"type":48,"value":9062},{"type":42,"tag":1825,"props":9808,"children":9809},{"class":1827,"line":7823},[9810,9814,9818,9822,9826,9830,9834,9838],{"type":42,"tag":1825,"props":9811,"children":9812},{"style":8275},[9813],{"type":48,"value":9070},{"type":42,"tag":1825,"props":9815,"children":9816},{"style":8281},[9817],{"type":48,"value":4005},{"type":42,"tag":1825,"props":9819,"children":9820},{"style":8275},[9821],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9823,"children":9824},{"style":8275},[9825],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9827,"children":9828},{"style":8275},[9829],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9831,"children":9832},{"style":8380},[9833],{"type":48,"value":8906},{"type":42,"tag":1825,"props":9835,"children":9836},{"style":8275},[9837],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9839,"children":9840},{"style":8275},[9841],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9843,"children":9844},{"class":1827,"line":7831},[9845,9849,9853,9857,9861,9865,9869,9873],{"type":42,"tag":1825,"props":9846,"children":9847},{"style":8275},[9848],{"type":48,"value":9070},{"type":42,"tag":1825,"props":9850,"children":9851},{"style":8281},[9852],{"type":48,"value":9648},{"type":42,"tag":1825,"props":9854,"children":9855},{"style":8275},[9856],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9858,"children":9859},{"style":8275},[9860],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9862,"children":9863},{"style":8275},[9864],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9866,"children":9867},{"style":8380},[9868],{"type":48,"value":8906},{"type":42,"tag":1825,"props":9870,"children":9871},{"style":8275},[9872],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9874,"children":9875},{"style":8275},[9876],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9878,"children":9879},{"class":1827,"line":7840},[9880,9884,9888,9892,9896],{"type":42,"tag":1825,"props":9881,"children":9882},{"style":8275},[9883],{"type":48,"value":9070},{"type":42,"tag":1825,"props":9885,"children":9886},{"style":8281},[9887],{"type":48,"value":1508},{"type":42,"tag":1825,"props":9889,"children":9890},{"style":8275},[9891],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9893,"children":9894},{"style":8275},[9895],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9897,"children":9898},{"style":8275},[9899],{"type":48,"value":9900}," {},\n",{"type":42,"tag":1825,"props":9902,"children":9903},{"class":1827,"line":7848},[9904,9908,9912,9916,9920,9924,9928,9932],{"type":42,"tag":1825,"props":9905,"children":9906},{"style":8275},[9907],{"type":48,"value":9070},{"type":42,"tag":1825,"props":9909,"children":9910},{"style":8281},[9911],{"type":48,"value":1515},{"type":42,"tag":1825,"props":9913,"children":9914},{"style":8275},[9915],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9917,"children":9918},{"style":8275},[9919],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9921,"children":9922},{"style":8275},[9923],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9925,"children":9926},{"style":8380},[9927],{"type":48,"value":8906},{"type":42,"tag":1825,"props":9929,"children":9930},{"style":8275},[9931],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9933,"children":9934},{"style":8275},[9935],{"type":48,"value":8778},{"type":42,"tag":1825,"props":9937,"children":9938},{"class":1827,"line":7857},[9939,9943,9948,9952,9956],{"type":42,"tag":1825,"props":9940,"children":9941},{"style":8275},[9942],{"type":48,"value":9070},{"type":42,"tag":1825,"props":9944,"children":9945},{"style":8281},[9946],{"type":48,"value":9947},"suggested_labels",{"type":42,"tag":1825,"props":9949,"children":9950},{"style":8275},[9951],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9953,"children":9954},{"style":8275},[9955],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9957,"children":9958},{"style":8275},[9959],{"type":48,"value":8877},{"type":42,"tag":1825,"props":9961,"children":9962},{"class":1827,"line":7866},[9963,9967,9972,9976,9980,9984,9989],{"type":42,"tag":1825,"props":9964,"children":9965},{"style":8275},[9966],{"type":48,"value":9277},{"type":42,"tag":1825,"props":9968,"children":9969},{"style":9280},[9970],{"type":48,"value":9971},"\u003Cevaluator_name>",{"type":42,"tag":1825,"props":9973,"children":9974},{"style":8275},[9975],{"type":48,"value":8755},{"type":42,"tag":1825,"props":9977,"children":9978},{"style":8275},[9979],{"type":48,"value":4920},{"type":42,"tag":1825,"props":9981,"children":9982},{"style":8275},[9983],{"type":48,"value":8764},{"type":42,"tag":1825,"props":9985,"children":9986},{"style":8380},[9987],{"type":48,"value":9988},"pass | fail | \u003Cscore>",{"type":42,"tag":1825,"props":9990,"children":9991},{"style":8275},[9992],{"type":48,"value":9022},{"type":42,"tag":1825,"props":9994,"children":9995},{"class":1827,"line":7875},[9996],{"type":42,"tag":1825,"props":9997,"children":9998},{"style":8275},[9999],{"type":48,"value":10000},"      }\n",{"type":42,"tag":1825,"props":10002,"children":10003},{"class":1827,"line":7884},[10004],{"type":42,"tag":1825,"props":10005,"children":10006},{"style":8275},[10007],{"type":48,"value":9767},{"type":42,"tag":1825,"props":10009,"children":10011},{"class":1827,"line":10010},49,[10012],{"type":42,"tag":1825,"props":10013,"children":10014},{"style":8275},[10015],{"type":48,"value":10016},"  ]\n",{"type":42,"tag":1825,"props":10018,"children":10020},{"class":1827,"line":10019},50,[10021],{"type":42,"tag":1825,"props":10022,"children":10023},{"style":8275},[10024],{"type":48,"value":2670},{"type":42,"tag":3896,"props":10026,"children":10028},{"id":10027},"field-notes",[10029],{"type":48,"value":10030},"Field Notes",{"type":42,"tag":194,"props":10032,"children":10033},{},[10034,10063,10115,10144,10164],{"type":42,"tag":67,"props":10035,"children":10036},{},[10037,10046,10047,10053,10055,10061],{"type":42,"tag":55,"props":10038,"children":10039},{},[10040],{"type":42,"tag":73,"props":10041,"children":10043},{"className":10042},[],[10044],{"type":48,"value":10045},"evaluators[].type",{"type":48,"value":1665},{"type":42,"tag":73,"props":10048,"children":10050},{"className":10049},[],[10051],{"type":48,"value":10052},"\"llm_judge\"",{"type":48,"value":10054}," for semantic evaluators; ",{"type":42,"tag":73,"props":10056,"children":10058},{"className":10057},[],[10059],{"type":48,"value":10060},"\"code_check\"",{"type":48,"value":10062}," for deterministic checks (regex, length, JSON validity, etc.).",{"type":42,"tag":67,"props":10064,"children":10065},{},[10066,10075,10077,10083,10085,10091,10092,10098,10100,10105,10107,10113],{"type":42,"tag":55,"props":10067,"children":10068},{},[10069],{"type":42,"tag":73,"props":10070,"children":10072},{"className":10071},[],[10073],{"type":48,"value":10074},"evaluators[].rubric",{"type":48,"value":10076},": For ",{"type":42,"tag":73,"props":10078,"children":10080},{"className":10079},[],[10081],{"type":48,"value":10082},"llm_judge",{"type":48,"value":10084}," — full prompt text grounded in observed trace patterns. Use ",{"type":42,"tag":73,"props":10086,"children":10088},{"className":10087},[],[10089],{"type":48,"value":10090},"{{input}}",{"type":48,"value":919},{"type":42,"tag":73,"props":10093,"children":10095},{"className":10094},[],[10096],{"type":48,"value":10097},"{{output}}",{"type":48,"value":10099}," as generic placeholders (not ",{"type":42,"tag":73,"props":10101,"children":10103},{"className":10102},[],[10104],{"type":48,"value":2094},{"type":48,"value":10106}," — that's ddeval-specific). For ",{"type":42,"tag":73,"props":10108,"children":10110},{"className":10109},[],[10111],{"type":48,"value":10112},"code_check",{"type":48,"value":10114}," — null.",{"type":42,"tag":67,"props":10116,"children":10117},{},[10118,10127,10129,10134,10136,10142],{"type":42,"tag":55,"props":10119,"children":10120},{},[10121],{"type":42,"tag":73,"props":10122,"children":10124},{"className":10123},[],[10125],{"type":48,"value":10126},"evaluators[].implementation_hints.notes",{"type":48,"value":10128},": Optional framework-agnostic guidance, e.g. \"For OpenAI Evals, use ",{"type":42,"tag":73,"props":10130,"children":10132},{"className":10131},[],[10133],{"type":48,"value":9404},{"type":48,"value":10135}," as a model-graded criterion. For Braintrust, use as an LLM scorer. For Promptfoo, use as an ",{"type":42,"tag":73,"props":10137,"children":10139},{"className":10138},[],[10140],{"type":48,"value":10141},"llm-rubric",{"type":48,"value":10143}," assertion.\"",{"type":42,"tag":67,"props":10145,"children":10146},{},[10147,10155,10157,10162],{"type":42,"tag":55,"props":10148,"children":10149},{},[10150],{"type":42,"tag":73,"props":10151,"children":10153},{"className":10152},[],[10154],{"type":48,"value":9787},{"type":48,"value":10156},": 10–20 representative traces from Phase 1. ",{"type":42,"tag":73,"props":10158,"children":10160},{"className":10159},[],[10161],{"type":48,"value":9947},{"type":48,"value":10163}," are Claude's best-read from trace inspection — not ground truth. The field name communicates this explicitly.",{"type":42,"tag":67,"props":10165,"children":10166},{},[10167,10172,10174,10179,10180,10185,10186,10192],{"type":42,"tag":55,"props":10168,"children":10169},{},[10170],{"type":48,"value":10171},"PII rule",{"type":48,"value":10173},": Strip emails, names, and sensitive data from all ",{"type":42,"tag":73,"props":10175,"children":10177},{"className":10176},[],[10178],{"type":48,"value":1508},{"type":48,"value":251},{"type":42,"tag":73,"props":10181,"children":10183},{"className":10182},[],[10184],{"type":48,"value":1515},{"type":48,"value":3536},{"type":42,"tag":73,"props":10187,"children":10189},{"className":10188},[],[10190],{"type":48,"value":10191},"evidence[].observation",{"type":48,"value":10193}," fields before writing (same as Phase 3A).",{"type":42,"tag":3896,"props":10195,"children":10197},{"id":10196},"writing-instructions",[10198],{"type":48,"value":10199},"Writing Instructions",{"type":42,"tag":63,"props":10201,"children":10202},{},[10203,10208,10220,10244,10249],{"type":42,"tag":67,"props":10204,"children":10205},{},[10206],{"type":48,"value":10207},"Assemble the JSON object in memory following the schema above.",{"type":42,"tag":67,"props":10209,"children":10210},{},[10211,10213,10218],{"type":48,"value":10212},"Populate ",{"type":42,"tag":73,"props":10214,"children":10216},{"className":10215},[],[10217],{"type":48,"value":9787},{"type":48,"value":10219}," from traces already fetched in Phase 1. Fetch additional traces (up to 20 total) if fewer than 10 were read.",{"type":42,"tag":67,"props":10221,"children":10222},{},[10223,10225,10230,10231,10236,10237,10242],{"type":48,"value":10224},"Anonymize PII in all ",{"type":42,"tag":73,"props":10226,"children":10228},{"className":10227},[],[10229],{"type":48,"value":1508},{"type":48,"value":251},{"type":42,"tag":73,"props":10232,"children":10234},{"className":10233},[],[10235],{"type":48,"value":1515},{"type":48,"value":3536},{"type":42,"tag":73,"props":10238,"children":10240},{"className":10239},[],[10241],{"type":48,"value":10191},{"type":48,"value":10243}," fields.",{"type":42,"tag":67,"props":10245,"children":10246},{},[10247],{"type":48,"value":10248},"Write the file with 2-space indentation using the Write tool.",{"type":42,"tag":67,"props":10250,"children":10251},{},[10252],{"type":48,"value":10253},"Display a completion summary:",{"type":42,"tag":730,"props":10255,"children":10258},{"className":10256,"code":10257,"language":48},[733],"## Generated Eval Spec\n\nWrote `.\u002Fevals\u002F{ml_app}_eval_spec.json`:\n\n- **{N} evaluators** ({outcome_count} outcome, {format_count} format, {safety_count} safety)\n- **{M} sample records** with suggested labels\n\n| # | Name | Category | Type | Pass Criteria |\n|---|------|----------|------|---------------|\n| 1 | ... | ... | ... | ... |\n\n### Next Steps\n\n1. **Review**: Open `.\u002Fevals\u002F{ml_app}_eval_spec.json` and verify the rubrics match your expectations\n2. **Implement**: Use the `rubric` field to configure evaluators in your framework of choice:\n   - OpenAI Evals: use `rubric` as a model-graded criterion\n   - Braintrust: create an LLM scorer with the rubric text\n   - Promptfoo: use as an `llm-rubric` assertion\n   - Custom code: call your LLM API with the rubric and parse the structured output\n3. **Label**: `suggested_labels` are Claude's best guesses from trace inspection — verify against ground truth before using as training data\n",[10259],{"type":42,"tag":73,"props":10260,"children":10261},{"__ignoreMap":738},[10262],{"type":48,"value":10257},{"type":42,"tag":3896,"props":10264,"children":10266},{"id":10265},"notebook-export-after-summary-1",[10267],{"type":48,"value":7929},{"type":42,"tag":51,"props":10269,"children":10270},{},[10271,10273,10278,10280,10286,10288,10294,10295,10301],{"type":48,"value":10272},"Same logic as Phase 3A — offer to append to the RCA notebook if ",{"type":42,"tag":73,"props":10274,"children":10276},{"className":10275},[],[10277],{"type":48,"value":3865},{"type":48,"value":10279}," was detected, or create a new standalone notebook. Use the same notebook cell format as Phase 3A, substituting ",{"type":42,"tag":73,"props":10281,"children":10283},{"className":10282},[],[10284],{"type":48,"value":10285},"output_path",{"type":48,"value":10287}," with the JSON spec file path. In pup mode, use ",{"type":42,"tag":73,"props":10289,"children":10291},{"className":10290},[],[10292],{"type":48,"value":10293},"pup notebooks create",{"type":48,"value":591},{"type":42,"tag":73,"props":10296,"children":10298},{"className":10297},[],[10299],{"type":48,"value":10300},"pup notebooks edit",{"type":48,"value":10302}," as described in Phase 3A.",{"type":42,"tag":1773,"props":10304,"children":10305},{},[],{"type":42,"tag":747,"props":10307,"children":10309},{"id":10308},"phase-3c-publish-online-evaluators-to-datadog-as-disabled-drafts",[10310],{"type":48,"value":10311},"Phase 3C: Publish Online Evaluators to Datadog (as disabled drafts)",{"type":42,"tag":51,"props":10313,"children":10314},{},[10315],{"type":42,"tag":55,"props":10316,"children":10317},{},[10318],{"type":48,"value":10319},"Reached only after the user confirms the suite at the Phase 2 checkpoint — nothing below is written to Datadog before that.",{"type":42,"tag":51,"props":10321,"children":10322},{},[10323,10327,10329,10334,10336,10341,10342,10347,10349,10354],{"type":42,"tag":55,"props":10324,"children":10325},{},[10326],{"type":48,"value":3892},{"type":48,"value":10328},": For each confirmed evaluator, write an LLM-judge configuration to Datadog via ",{"type":42,"tag":73,"props":10330,"children":10332},{"className":10331},[],[10333],{"type":48,"value":488},{"type":48,"value":10335}," as a ",{"type":42,"tag":55,"props":10337,"children":10338},{},[10339],{"type":48,"value":10340},"disabled draft",{"type":48,"value":497},{"type":42,"tag":73,"props":10343,"children":10345},{"className":10344},[],[10346],{"type":48,"value":503},{"type":48,"value":10348},"). It scores ",{"type":42,"tag":55,"props":10350,"children":10351},{},[10352],{"type":48,"value":10353},"no",{"type":48,"value":10355}," spans until the user reviews and enables it in the UI; once enabled, it runs automatically on matching production spans.",{"type":42,"tag":3896,"props":10357,"children":10359},{"id":10358},"pre-publish-checks-single-message-parallelize",[10360],{"type":48,"value":10361},"Pre-publish checks (single message — parallelize)",{"type":42,"tag":51,"props":10363,"children":10364},{},[10365,10367,10373,10374,10379],{"type":48,"value":10366},"For every proposed ",{"type":42,"tag":73,"props":10368,"children":10370},{"className":10369},[],[10371],{"type":48,"value":10372},"eval_name",{"type":48,"value":3752},{"type":42,"tag":73,"props":10375,"children":10377},{"className":10376},[],[10378],{"type":48,"value":3758},{"type":48,"value":4920},{"type":42,"tag":194,"props":10381,"children":10382},{},[10383,10393],{"type":42,"tag":67,"props":10384,"children":10385},{},[10386,10391],{"type":42,"tag":55,"props":10387,"children":10388},{},[10389],{"type":48,"value":10390},"Not found",{"type":48,"value":10392}," → safe to create.",{"type":42,"tag":67,"props":10394,"children":10395},{},[10396,10401,10403,10419,10422,10423,10428,10430,10435,10437,10442,10444,10450,10451,10457,10458,10463,10464,10469,10471,10474,10475,10480,10482,10488,10490,10493,10494,10499],{"type":42,"tag":55,"props":10397,"children":10398},{},[10399],{"type":48,"value":10400},"Found",{"type":48,"value":10402}," → existing evaluator with the same name. Surface a diff to the user (existing dimension\u002Fprompt vs. proposed) and ask:",{"type":42,"tag":159,"props":10404,"children":10405},{},[10406],{"type":42,"tag":51,"props":10407,"children":10408},{},[10409,10411,10417],{"type":48,"value":10410},"Evaluator ",{"type":42,"tag":73,"props":10412,"children":10414},{"className":10413},[],[10415],{"type":48,"value":10416},"{name}",{"type":48,"value":10418}," already exists. Overwrite, rename, or skip?",{"type":42,"tag":3775,"props":10420,"children":10421},{},[],{"type":48,"value":1069},{"type":42,"tag":55,"props":10424,"children":10425},{},[10426],{"type":48,"value":10427},"overwrite",{"type":48,"value":10429},": keep the fetched config as the base and ",{"type":42,"tag":55,"props":10431,"children":10432},{},[10433],{"type":48,"value":10434},"merge",{"type":48,"value":10436}," your generated fields on top, then send the ",{"type":42,"tag":55,"props":10438,"children":10439},{},[10440],{"type":48,"value":10441},"complete",{"type":48,"value":10443}," object back. The MCP tool is full-replace — any field you omit (e.g. ",{"type":42,"tag":73,"props":10445,"children":10447},{"className":10446},[],[10448],{"type":48,"value":10449},"temperature",{"type":48,"value":251},{"type":42,"tag":73,"props":10452,"children":10454},{"className":10453},[],[10455],{"type":48,"value":10456},"max_tokens",{"type":48,"value":251},{"type":42,"tag":73,"props":10459,"children":10461},{"className":10460},[],[10462],{"type":48,"value":5065},{"type":48,"value":251},{"type":42,"tag":73,"props":10465,"children":10467},{"className":10466},[],[10468],{"type":48,"value":5057},{"type":48,"value":10470},") reverts to its default. Never re-publish without round-tripping the existing config.",{"type":42,"tag":3775,"props":10472,"children":10473},{},[],{"type":48,"value":1069},{"type":42,"tag":55,"props":10476,"children":10477},{},[10478],{"type":48,"value":10479},"rename",{"type":48,"value":10481},": append a suffix (e.g. ",{"type":42,"tag":73,"props":10483,"children":10485},{"className":10484},[],[10486],{"type":48,"value":10487},"_v2",{"type":48,"value":10489},") and treat as new.",{"type":42,"tag":3775,"props":10491,"children":10492},{},[],{"type":48,"value":1069},{"type":42,"tag":55,"props":10495,"children":10496},{},[10497],{"type":48,"value":10498},"skip",{"type":48,"value":10500},": drop from the publish set.",{"type":42,"tag":3896,"props":10502,"children":10504},{"id":10503},"publishing-conventions",[10505],{"type":48,"value":10506},"Publishing Conventions",{"type":42,"tag":51,"props":10508,"children":10509},{},[10510,10515,10517,10522,10524,10529,10530,10536,10538,10543,10544,10549,10550,10555,10556,10561,10562,10568,10569,10575,10576,10582,10584,10589],{"type":42,"tag":55,"props":10511,"children":10512},{},[10513],{"type":48,"value":10514},"Required parameters",{"type":48,"value":10516}," for each ",{"type":42,"tag":73,"props":10518,"children":10520},{"className":10519},[],[10521],{"type":48,"value":488},{"type":48,"value":10523}," call: ",{"type":42,"tag":73,"props":10525,"children":10527},{"className":10526},[],[10528],{"type":48,"value":10372},{"type":48,"value":251},{"type":42,"tag":73,"props":10531,"children":10533},{"className":10532},[],[10534],{"type":48,"value":10535},"application_name",{"type":48,"value":10537}," (= ",{"type":42,"tag":73,"props":10539,"children":10541},{"className":10540},[],[10542],{"type":48,"value":802},{"type":48,"value":3511},{"type":42,"tag":73,"props":10545,"children":10547},{"className":10546},[],[10548],{"type":48,"value":1257},{"type":48,"value":251},{"type":42,"tag":73,"props":10551,"children":10553},{"className":10552},[],[10554],{"type":48,"value":5960},{"type":48,"value":251},{"type":42,"tag":73,"props":10557,"children":10559},{"className":10558},[],[10560],{"type":48,"value":5974},{"type":48,"value":251},{"type":42,"tag":73,"props":10563,"children":10565},{"className":10564},[],[10566],{"type":48,"value":10567},"prompt_template",{"type":48,"value":251},{"type":42,"tag":73,"props":10570,"children":10572},{"className":10571},[],[10573],{"type":48,"value":10574},"parsing_type",{"type":48,"value":251},{"type":42,"tag":73,"props":10577,"children":10579},{"className":10578},[],[10580],{"type":48,"value":10581},"output_schema",{"type":48,"value":10583},", plus a ",{"type":42,"tag":73,"props":10585,"children":10587},{"className":10586},[],[10588],{"type":48,"value":385},{"type":48,"value":10590}," string.",{"type":42,"tag":51,"props":10592,"children":10593},{},[10594,10599],{"type":42,"tag":55,"props":10595,"children":10596},{},[10597],{"type":48,"value":10598},"Defaults",{"type":48,"value":10600}," to use unless the user overrides:",{"type":42,"tag":754,"props":10602,"children":10603},{},[10604,10618],{"type":42,"tag":758,"props":10605,"children":10606},{},[10607],{"type":42,"tag":762,"props":10608,"children":10609},{},[10610,10614],{"type":42,"tag":766,"props":10611,"children":10612},{},[10613],{"type":48,"value":1400},{"type":42,"tag":766,"props":10615,"children":10616},{},[10617],{"type":48,"value":780},{"type":42,"tag":787,"props":10619,"children":10620},{},[10621,10643,10662,10681,10701,10720,10756],{"type":42,"tag":762,"props":10622,"children":10623},{},[10624,10632],{"type":42,"tag":794,"props":10625,"children":10626},{},[10627],{"type":42,"tag":73,"props":10628,"children":10630},{"className":10629},[],[10631],{"type":48,"value":1257},{"type":42,"tag":794,"props":10633,"children":10634},{},[10635,10641],{"type":42,"tag":73,"props":10636,"children":10638},{"className":10637},[],[10639],{"type":48,"value":10640},"false",{"type":48,"value":10642}," (always — see \"Always publish as draft\")",{"type":42,"tag":762,"props":10644,"children":10645},{},[10646,10654],{"type":42,"tag":794,"props":10647,"children":10648},{},[10649],{"type":42,"tag":73,"props":10650,"children":10652},{"className":10651},[],[10653],{"type":48,"value":5960},{"type":42,"tag":794,"props":10655,"children":10656},{},[10657],{"type":42,"tag":73,"props":10658,"children":10660},{"className":10659},[],[10661],{"type":48,"value":5967},{"type":42,"tag":762,"props":10663,"children":10664},{},[10665,10673],{"type":42,"tag":794,"props":10666,"children":10667},{},[10668],{"type":42,"tag":73,"props":10669,"children":10671},{"className":10670},[],[10672],{"type":48,"value":5974},{"type":42,"tag":794,"props":10674,"children":10675},{},[10676],{"type":42,"tag":73,"props":10677,"children":10679},{"className":10678},[],[10680],{"type":48,"value":5981},{"type":42,"tag":762,"props":10682,"children":10683},{},[10684,10692],{"type":42,"tag":794,"props":10685,"children":10686},{},[10687],{"type":42,"tag":73,"props":10688,"children":10690},{"className":10689},[],[10691],{"type":48,"value":10449},{"type":42,"tag":794,"props":10693,"children":10694},{},[10695],{"type":42,"tag":73,"props":10696,"children":10698},{"className":10697},[],[10699],{"type":48,"value":10700},"0",{"type":42,"tag":762,"props":10702,"children":10703},{},[10704,10712],{"type":42,"tag":794,"props":10705,"children":10706},{},[10707],{"type":42,"tag":73,"props":10708,"children":10710},{"className":10709},[],[10711],{"type":48,"value":10574},{"type":42,"tag":794,"props":10713,"children":10714},{},[10715],{"type":42,"tag":73,"props":10716,"children":10718},{"className":10717},[],[10719],{"type":48,"value":2755},{"type":42,"tag":762,"props":10721,"children":10722},{},[10723,10731],{"type":42,"tag":794,"props":10724,"children":10725},{},[10726],{"type":42,"tag":73,"props":10727,"children":10729},{"className":10728},[],[10730],{"type":48,"value":5057},{"type":42,"tag":794,"props":10732,"children":10733},{},[10734,10739,10741,10747,10749,10754],{"type":42,"tag":73,"props":10735,"children":10737},{"className":10736},[],[10738],{"type":48,"value":5994},{"type":48,"value":10740}," for span scope, ",{"type":42,"tag":73,"props":10742,"children":10744},{"className":10743},[],[10745],{"type":48,"value":10746},"5",{"type":48,"value":10748}," for trace scope, ",{"type":42,"tag":73,"props":10750,"children":10752},{"className":10751},[],[10753],{"type":48,"value":10746},{"type":48,"value":10755}," for session scope (the heaviest — consider lowering)",{"type":42,"tag":762,"props":10757,"children":10758},{},[10759,10767],{"type":42,"tag":794,"props":10760,"children":10761},{},[10762],{"type":42,"tag":73,"props":10763,"children":10765},{"className":10764},[],[10766],{"type":48,"value":5050},{"type":42,"tag":794,"props":10768,"children":10769},{},[10770,10775,10777,10782,10783,10788],{"type":42,"tag":73,"props":10771,"children":10773},{"className":10772},[],[10774],{"type":48,"value":1825},{"type":48,"value":10776}," (auto-promoted to ",{"type":42,"tag":73,"props":10778,"children":10780},{"className":10779},[],[10781],{"type":48,"value":5794},{"type":48,"value":5132},{"type":42,"tag":73,"props":10784,"children":10786},{"className":10785},[],[10787],{"type":48,"value":5801},{"type":48,"value":10789}," per the classification rule in Phase 2)",{"type":42,"tag":51,"props":10791,"children":10792},{},[10793,10798,10800,10806,10808,10814,10816,10822,10824,10829,10831,10835,10837,10842,10843,10848,10850,10855],{"type":42,"tag":55,"props":10794,"children":10795},{},[10796],{"type":48,"value":10797},"Prompt template",{"type":48,"value":10799},": convert the LLMJudge prompt into the MCP shape — an ordered array of ",{"type":42,"tag":73,"props":10801,"children":10803},{"className":10802},[],[10804],{"type":48,"value":10805},"{role, content}",{"type":48,"value":10807}," messages. The system prompt becomes ",{"type":42,"tag":73,"props":10809,"children":10811},{"className":10810},[],[10812],{"type":48,"value":10813},"{role: \"system\"}",{"type":48,"value":10815},", the user prompt becomes ",{"type":42,"tag":73,"props":10817,"children":10819},{"className":10818},[],[10820],{"type":48,"value":10821},"{role: \"user\"}",{"type":48,"value":10823},". Use ",{"type":42,"tag":55,"props":10825,"children":10826},{},[10827],{"type":48,"value":10828},"span-data placeholders",{"type":48,"value":10830}," (see below) — ",{"type":42,"tag":55,"props":10832,"children":10833},{},[10834],{"type":48,"value":271},{"type":48,"value":10836}," the offline ",{"type":42,"tag":73,"props":10838,"children":10840},{"className":10839},[],[10841],{"type":48,"value":2094},{"type":48,"value":591},{"type":42,"tag":73,"props":10844,"children":10846},{"className":10845},[],[10847],{"type":48,"value":2101},{"type":48,"value":10849}," form, which only exists in ",{"type":42,"tag":73,"props":10851,"children":10853},{"className":10852},[],[10854],{"type":48,"value":2312},{"type":48,"value":332},{"type":42,"tag":5384,"props":10857,"children":10859},{"id":10858},"online-template-variables",[10860],{"type":48,"value":10861},"Online Template Variables",{"type":42,"tag":51,"props":10863,"children":10864},{},[10865,10867,10873,10875,10881,10883],{"type":48,"value":10866},"Online evaluator prompts run through the dd-source ",{"type":42,"tag":73,"props":10868,"children":10870},{"className":10869},[],[10871],{"type":48,"value":10872},"template",{"type":48,"value":10874}," library (",{"type":42,"tag":73,"props":10876,"children":10878},{"className":10877},[],[10879],{"type":48,"value":10880},"domains\u002Fml-observability\u002Fshared\u002Flibs\u002Ftemplate",{"type":48,"value":10882},"). Missing paths → empty string. ",{"type":42,"tag":55,"props":10884,"children":10885},{},[10886,10888,10893],{"type":48,"value":10887},"The data shape templates resolve against depends on ",{"type":42,"tag":73,"props":10889,"children":10891},{"className":10890},[],[10892],{"type":48,"value":5050},{"type":48,"value":4920},{"type":42,"tag":194,"props":10895,"children":10896},{},[10897,10930,11006],{"type":42,"tag":67,"props":10898,"children":10899},{},[10900,10908,10909,10913,10915,10920,10922,10928],{"type":42,"tag":55,"props":10901,"children":10902},{},[10903],{"type":42,"tag":73,"props":10904,"children":10906},{"className":10905},[],[10907],{"type":48,"value":4867},{"type":48,"value":460},{"type":42,"tag":462,"props":10910,"children":10911},{},[10912],{"type":48,"value":466},{"type":48,"value":10914}," — placeholders resolve against a ",{"type":42,"tag":55,"props":10916,"children":10917},{},[10918],{"type":48,"value":10919},"single span's JSON",{"type":48,"value":10921}," (the ",{"type":42,"tag":73,"props":10923,"children":10925},{"className":10924},[],[10926],{"type":48,"value":10927},"llmobs.Span",{"type":48,"value":10929}," JSON-marshaled to a map). Use the span aliases \u002F dot-paths below directly.",{"type":42,"tag":67,"props":10931,"children":10932},{},[10933,10941,10943,10948,10949,10955,10956,10962,10963,10969,10970,10976,10978,10983,10984,10989,10991,10996,10998,11004],{"type":42,"tag":55,"props":10934,"children":10935},{},[10936],{"type":42,"tag":73,"props":10937,"children":10939},{"className":10938},[],[10940],{"type":48,"value":4895},{"type":48,"value":10942}," — placeholders resolve against the ",{"type":42,"tag":55,"props":10944,"children":10945},{},[10946],{"type":48,"value":10947},"trace payload",{"type":48,"value":460},{"type":42,"tag":73,"props":10950,"children":10952},{"className":10951},[],[10953],{"type":48,"value":10954},"{ spans: [...] }",{"type":48,"value":10823},{"type":42,"tag":73,"props":10957,"children":10959},{"className":10958},[],[10960],{"type":48,"value":10961},"{{spans[N]...}}",{"type":48,"value":251},{"type":42,"tag":73,"props":10964,"children":10966},{"className":10965},[],[10967],{"type":48,"value":10968},"{{spans[*]...}}",{"type":48,"value":532},{"type":42,"tag":73,"props":10971,"children":10973},{"className":10972},[],[10974],{"type":48,"value":10975},"{{spans[field.path:value]...}}",{"type":48,"value":10977}," to select span(s) before applying field paths. The ",{"type":42,"tag":73,"props":10979,"children":10981},{"className":10980},[],[10982],{"type":48,"value":3824},{"type":48,"value":591},{"type":42,"tag":73,"props":10985,"children":10987},{"className":10986},[],[10988],{"type":48,"value":3831},{"type":48,"value":10990}," aliases are ",{"type":42,"tag":55,"props":10992,"children":10993},{},[10994],{"type":48,"value":10995},"not available",{"type":48,"value":10997}," in trace scope — reference span data through the ",{"type":42,"tag":73,"props":10999,"children":11001},{"className":11000},[],[11002],{"type":48,"value":11003},"spans",{"type":48,"value":11005}," array instead.",{"type":42,"tag":67,"props":11007,"children":11008},{},[11009,11017,11018,11023,11024,11030,11032,11038,11040,11045,11046,11051,11053,11057,11059,11064,11066,11070,11072,11078,11080,11085],{"type":42,"tag":55,"props":11010,"children":11011},{},[11012],{"type":42,"tag":73,"props":11013,"children":11015},{"className":11014},[],[11016],{"type":48,"value":4903},{"type":48,"value":10942},{"type":42,"tag":55,"props":11019,"children":11020},{},[11021],{"type":48,"value":11022},"session payload",{"type":48,"value":460},{"type":42,"tag":73,"props":11025,"children":11027},{"className":11026},[],[11028],{"type":48,"value":11029},"{ session_id, traces: [ { trace_id, root_span_id, spans: [...] }, … ] }",{"type":48,"value":11031},". Reach spans through ",{"type":42,"tag":73,"props":11033,"children":11035},{"className":11034},[],[11036],{"type":48,"value":11037},"{{traces[*].spans[*]...}}",{"type":48,"value":11039}," (nested one level deeper than trace scope) plus the array selectors below. The ",{"type":42,"tag":73,"props":11041,"children":11043},{"className":11042},[],[11044],{"type":48,"value":3824},{"type":48,"value":591},{"type":42,"tag":73,"props":11047,"children":11049},{"className":11048},[],[11050],{"type":48,"value":3831},{"type":48,"value":11052}," aliases ",{"type":42,"tag":55,"props":11054,"children":11055},{},[11056],{"type":48,"value":6578},{"type":48,"value":11058}," trace scope's top-level ",{"type":42,"tag":73,"props":11060,"children":11062},{"className":11061},[],[11063],{"type":48,"value":5034},{"type":48,"value":11065}," are ",{"type":42,"tag":55,"props":11067,"children":11068},{},[11069],{"type":48,"value":10995},{"type":48,"value":11071}," — spans live under ",{"type":42,"tag":73,"props":11073,"children":11075},{"className":11074},[],[11076],{"type":48,"value":11077},"traces[*].spans",{"type":48,"value":11079},". Only used when ",{"type":42,"tag":73,"props":11081,"children":11083},{"className":11082},[],[11084],{"type":48,"value":4276},{"type":48,"value":11086}," (Phase 1 probe).",{"type":42,"tag":11088,"props":11089,"children":11091},"h6",{"id":11090},"span-scope-eval_scope-span",[11092,11094,11099],{"type":48,"value":11093},"Span-scope (",{"type":42,"tag":73,"props":11095,"children":11097},{"className":11096},[],[11098],{"type":48,"value":4867},{"type":48,"value":631},{"type":42,"tag":51,"props":11101,"children":11102},{},[11103,11108],{"type":42,"tag":55,"props":11104,"children":11105},{},[11106],{"type":48,"value":11107},"Built-in span-kind-aware aliases",{"type":48,"value":11109}," (preferred when the evaluator is generic across span kinds):",{"type":42,"tag":754,"props":11111,"children":11112},{},[11113,11141],{"type":42,"tag":758,"props":11114,"children":11115},{},[11116],{"type":42,"tag":762,"props":11117,"children":11118},{},[11119,11124,11136],{"type":42,"tag":766,"props":11120,"children":11121},{},[11122],{"type":48,"value":11123},"Alias",{"type":42,"tag":766,"props":11125,"children":11126},{},[11127,11129,11135],{"type":48,"value":11128},"LLM span (",{"type":42,"tag":73,"props":11130,"children":11132},{"className":11131},[],[11133],{"type":48,"value":11134},"meta.span.kind = \"llm\"",{"type":48,"value":631},{"type":42,"tag":766,"props":11137,"children":11138},{},[11139],{"type":48,"value":11140},"Other spans (agent, workflow, task, …)",{"type":42,"tag":787,"props":11142,"children":11143},{},[11144,11173],{"type":42,"tag":762,"props":11145,"children":11146},{},[11147,11155,11164],{"type":42,"tag":794,"props":11148,"children":11149},{},[11150],{"type":42,"tag":73,"props":11151,"children":11153},{"className":11152},[],[11154],{"type":48,"value":3824},{"type":42,"tag":794,"props":11156,"children":11157},{},[11158],{"type":42,"tag":73,"props":11159,"children":11161},{"className":11160},[],[11162],{"type":48,"value":11163},"meta.input.messages[*].content",{"type":42,"tag":794,"props":11165,"children":11166},{},[11167],{"type":42,"tag":73,"props":11168,"children":11170},{"className":11169},[],[11171],{"type":48,"value":11172},"meta.input.value",{"type":42,"tag":762,"props":11174,"children":11175},{},[11176,11184,11192],{"type":42,"tag":794,"props":11177,"children":11178},{},[11179],{"type":42,"tag":73,"props":11180,"children":11182},{"className":11181},[],[11183],{"type":48,"value":3831},{"type":42,"tag":794,"props":11185,"children":11186},{},[11187],{"type":42,"tag":73,"props":11188,"children":11190},{"className":11189},[],[11191],{"type":48,"value":6767},{"type":42,"tag":794,"props":11193,"children":11194},{},[11195],{"type":42,"tag":73,"props":11196,"children":11198},{"className":11197},[],[11199],{"type":48,"value":11200},"meta.output.value",{"type":42,"tag":51,"props":11202,"children":11203},{},[11204,11209],{"type":42,"tag":55,"props":11205,"children":11206},{},[11207],{"type":48,"value":11208},"Common explicit dot-paths",{"type":48,"value":11210}," (use when the evaluator is purpose-built for one span kind):",{"type":42,"tag":754,"props":11212,"children":11213},{},[11214,11228],{"type":42,"tag":758,"props":11215,"children":11216},{},[11217],{"type":42,"tag":762,"props":11218,"children":11219},{},[11220,11224],{"type":42,"tag":766,"props":11221,"children":11222},{},[11223],{"type":48,"value":1405},{"type":42,"tag":766,"props":11225,"children":11226},{},[11227],{"type":48,"value":1410},{"type":42,"tag":787,"props":11229,"children":11230},{},[11231,11254,11270,11287,11303,11320,11336,11353],{"type":42,"tag":762,"props":11232,"children":11233},{},[11234,11249],{"type":42,"tag":794,"props":11235,"children":11236},{},[11237,11242,11243],{"type":42,"tag":73,"props":11238,"children":11240},{"className":11239},[],[11241],{"type":48,"value":4993},{"type":48,"value":591},{"type":42,"tag":73,"props":11244,"children":11246},{"className":11245},[],[11247],{"type":48,"value":11248},"{{meta.output.value}}",{"type":42,"tag":794,"props":11250,"children":11251},{},[11252],{"type":48,"value":11253},"Plain string I\u002FO on agent \u002F workflow \u002F task \u002F tool spans",{"type":42,"tag":762,"props":11255,"children":11256},{},[11257,11265],{"type":42,"tag":794,"props":11258,"children":11259},{},[11260],{"type":42,"tag":73,"props":11261,"children":11263},{"className":11262},[],[11264],{"type":48,"value":5914},{"type":42,"tag":794,"props":11266,"children":11267},{},[11268],{"type":48,"value":11269},"All input message contents on an LLM span (newline-joined)",{"type":42,"tag":762,"props":11271,"children":11272},{},[11273,11282],{"type":42,"tag":794,"props":11274,"children":11275},{},[11276],{"type":42,"tag":73,"props":11277,"children":11279},{"className":11278},[],[11280],{"type":48,"value":11281},"{{meta.input.messages[0].content}}",{"type":42,"tag":794,"props":11283,"children":11284},{},[11285],{"type":48,"value":11286},"First message (typically system prompt)",{"type":42,"tag":762,"props":11288,"children":11289},{},[11290,11298],{"type":42,"tag":794,"props":11291,"children":11292},{},[11293],{"type":42,"tag":73,"props":11294,"children":11296},{"className":11295},[],[11297],{"type":48,"value":5000},{"type":42,"tag":794,"props":11299,"children":11300},{},[11301],{"type":48,"value":11302},"Assistant response(s)",{"type":42,"tag":762,"props":11304,"children":11305},{},[11306,11315],{"type":42,"tag":794,"props":11307,"children":11308},{},[11309],{"type":42,"tag":73,"props":11310,"children":11312},{"className":11311},[],[11313],{"type":48,"value":11314},"{{meta.input.documents}}",{"type":42,"tag":794,"props":11316,"children":11317},{},[11318],{"type":48,"value":11319},"Retrieved docs (RAG) — JSON-serialized",{"type":42,"tag":762,"props":11321,"children":11322},{},[11323,11331],{"type":42,"tag":794,"props":11324,"children":11325},{},[11326],{"type":42,"tag":73,"props":11327,"children":11329},{"className":11328},[],[11330],{"type":48,"value":5921},{"type":42,"tag":794,"props":11332,"children":11333},{},[11334],{"type":48,"value":11335},"Custom metadata fields",{"type":42,"tag":762,"props":11337,"children":11338},{},[11339,11348],{"type":42,"tag":794,"props":11340,"children":11341},{},[11342],{"type":42,"tag":73,"props":11343,"children":11345},{"className":11344},[],[11346],{"type":48,"value":11347},"{{meta.tool_definitions}}",{"type":42,"tag":794,"props":11349,"children":11350},{},[11351],{"type":48,"value":11352},"Available tools — JSON array",{"type":42,"tag":762,"props":11354,"children":11355},{},[11356,11365],{"type":42,"tag":794,"props":11357,"children":11358},{},[11359],{"type":42,"tag":73,"props":11360,"children":11362},{"className":11361},[],[11363],{"type":48,"value":11364},"{{*}}",{"type":42,"tag":794,"props":11366,"children":11367},{},[11368],{"type":48,"value":11369},"Entire span as compact JSON (debug \u002F fall-back catch-all)",{"type":42,"tag":11088,"props":11371,"children":11373},{"id":11372},"trace-scope-eval_scope-trace",[11374,11376,11381],{"type":48,"value":11375},"Trace-scope (",{"type":42,"tag":73,"props":11377,"children":11379},{"className":11378},[],[11380],{"type":48,"value":4895},{"type":48,"value":631},{"type":42,"tag":754,"props":11383,"children":11384},{},[11385,11400],{"type":42,"tag":758,"props":11386,"children":11387},{},[11388],{"type":42,"tag":762,"props":11389,"children":11390},{},[11391,11396],{"type":42,"tag":766,"props":11392,"children":11393},{},[11394],{"type":48,"value":11395},"Pattern",{"type":42,"tag":766,"props":11397,"children":11398},{},[11399],{"type":48,"value":1410},{"type":42,"tag":787,"props":11401,"children":11402},{},[11403,11420,11445,11462,11479,11496,11513,11530,11547],{"type":42,"tag":762,"props":11404,"children":11405},{},[11406,11415],{"type":42,"tag":794,"props":11407,"children":11408},{},[11409],{"type":42,"tag":73,"props":11410,"children":11412},{"className":11411},[],[11413],{"type":48,"value":11414},"{{spans}}",{"type":42,"tag":794,"props":11416,"children":11417},{},[11418],{"type":48,"value":11419},"JSON of every span in the trace",{"type":42,"tag":762,"props":11421,"children":11422},{},[11423,11432],{"type":42,"tag":794,"props":11424,"children":11425},{},[11426],{"type":42,"tag":73,"props":11427,"children":11429},{"className":11428},[],[11430],{"type":48,"value":11431},"{{spans[N].meta.input.value}}",{"type":42,"tag":794,"props":11433,"children":11434},{},[11435,11437,11443],{"type":48,"value":11436},"Single span by index — ",{"type":42,"tag":73,"props":11438,"children":11440},{"className":11439},[],[11441],{"type":48,"value":11442},"spans[0]",{"type":48,"value":11444}," is the trace root",{"type":42,"tag":762,"props":11446,"children":11447},{},[11448,11457],{"type":42,"tag":794,"props":11449,"children":11450},{},[11451],{"type":42,"tag":73,"props":11452,"children":11454},{"className":11453},[],[11455],{"type":48,"value":11456},"{{spans[*].name}}",{"type":42,"tag":794,"props":11458,"children":11459},{},[11460],{"type":48,"value":11461},"All span names in order, newline-joined",{"type":42,"tag":762,"props":11463,"children":11464},{},[11465,11474],{"type":42,"tag":794,"props":11466,"children":11467},{},[11468],{"type":42,"tag":73,"props":11469,"children":11471},{"className":11470},[],[11472],{"type":48,"value":11473},"{{spans[*].meta.output.value}}",{"type":42,"tag":794,"props":11475,"children":11476},{},[11477],{"type":48,"value":11478},"All spans' outputs, newline-joined (handy for \"final answer = last output\")",{"type":42,"tag":762,"props":11480,"children":11481},{},[11482,11491],{"type":42,"tag":794,"props":11483,"children":11484},{},[11485],{"type":42,"tag":73,"props":11486,"children":11488},{"className":11487},[],[11489],{"type":48,"value":11490},"{{spans[name:my-span].meta.input.value}}",{"type":42,"tag":794,"props":11492,"children":11493},{},[11494],{"type":48,"value":11495},"Filter by span name",{"type":42,"tag":762,"props":11497,"children":11498},{},[11499,11508],{"type":42,"tag":794,"props":11500,"children":11501},{},[11502],{"type":42,"tag":73,"props":11503,"children":11505},{"className":11504},[],[11506],{"type":48,"value":11507},"{{spans[meta.span.kind:llm].meta.output.value}}",{"type":42,"tag":794,"props":11509,"children":11510},{},[11511],{"type":48,"value":11512},"All LLM-kind span outputs",{"type":42,"tag":762,"props":11514,"children":11515},{},[11516,11525],{"type":42,"tag":794,"props":11517,"children":11518},{},[11519],{"type":42,"tag":73,"props":11520,"children":11522},{"className":11521},[],[11523],{"type":48,"value":11524},"{{spans[meta.span.kind:tool]}}",{"type":42,"tag":794,"props":11526,"children":11527},{},[11528],{"type":48,"value":11529},"Whole tool spans as JSON, paired in\u002Fout — useful for tool-use correctness",{"type":42,"tag":762,"props":11531,"children":11532},{},[11533,11542],{"type":42,"tag":794,"props":11534,"children":11535},{},[11536],{"type":42,"tag":73,"props":11537,"children":11539},{"className":11538},[],[11540],{"type":48,"value":11541},"{{spans[meta.span.kind:retrieval].meta.output.documents[*].text}}",{"type":42,"tag":794,"props":11543,"children":11544},{},[11545],{"type":48,"value":11546},"Text of every retrieved document — useful for RAG faithfulness",{"type":42,"tag":762,"props":11548,"children":11549},{},[11550,11558],{"type":42,"tag":794,"props":11551,"children":11552},{},[11553],{"type":42,"tag":73,"props":11554,"children":11556},{"className":11555},[],[11557],{"type":48,"value":11364},{"type":42,"tag":794,"props":11559,"children":11560},{},[11561],{"type":48,"value":11562},"Entire trace payload as JSON (debug fallback)",{"type":42,"tag":11088,"props":11564,"children":11566},{"id":11565},"session-scope-eval_scope-session",[11567,11569,11574],{"type":48,"value":11568},"Session-scope (",{"type":42,"tag":73,"props":11570,"children":11572},{"className":11571},[],[11573],{"type":48,"value":4903},{"type":48,"value":631},{"type":42,"tag":754,"props":11576,"children":11577},{},[11578,11592],{"type":42,"tag":758,"props":11579,"children":11580},{},[11581],{"type":42,"tag":762,"props":11582,"children":11583},{},[11584,11588],{"type":42,"tag":766,"props":11585,"children":11586},{},[11587],{"type":48,"value":11395},{"type":42,"tag":766,"props":11589,"children":11590},{},[11591],{"type":48,"value":1410},{"type":42,"tag":787,"props":11593,"children":11594},{},[11595,11618,11635,11652,11669,11686,11703,11720,11737,11754,11771],{"type":42,"tag":762,"props":11596,"children":11597},{},[11598,11607],{"type":42,"tag":794,"props":11599,"children":11600},{},[11601],{"type":42,"tag":73,"props":11602,"children":11604},{"className":11603},[],[11605],{"type":48,"value":11606},"{{traces}}",{"type":42,"tag":794,"props":11608,"children":11609},{},[11610,11612],{"type":48,"value":11611},"JSON of every trace in the session, each ",{"type":42,"tag":73,"props":11613,"children":11615},{"className":11614},[],[11616],{"type":48,"value":11617},"{ trace_id, root_span_id, spans: [...] }",{"type":42,"tag":762,"props":11619,"children":11620},{},[11621,11630],{"type":42,"tag":794,"props":11622,"children":11623},{},[11624],{"type":42,"tag":73,"props":11625,"children":11627},{"className":11626},[],[11628],{"type":48,"value":11629},"{{session_id}}",{"type":42,"tag":794,"props":11631,"children":11632},{},[11633],{"type":48,"value":11634},"The session's ID",{"type":42,"tag":762,"props":11636,"children":11637},{},[11638,11647],{"type":42,"tag":794,"props":11639,"children":11640},{},[11641],{"type":42,"tag":73,"props":11642,"children":11644},{"className":11643},[],[11645],{"type":48,"value":11646},"{{traces[*].trace_id}}",{"type":42,"tag":794,"props":11648,"children":11649},{},[11650],{"type":48,"value":11651},"Every trace ID in the session, newline-joined",{"type":42,"tag":762,"props":11653,"children":11654},{},[11655,11664],{"type":42,"tag":794,"props":11656,"children":11657},{},[11658],{"type":42,"tag":73,"props":11659,"children":11661},{"className":11660},[],[11662],{"type":48,"value":11663},"{{traces[0].spans[0].meta.input.value}}",{"type":42,"tag":794,"props":11665,"children":11666},{},[11667],{"type":48,"value":11668},"First span of the first trace",{"type":42,"tag":762,"props":11670,"children":11671},{},[11672,11681],{"type":42,"tag":794,"props":11673,"children":11674},{},[11675],{"type":42,"tag":73,"props":11676,"children":11678},{"className":11677},[],[11679],{"type":48,"value":11680},"{{traces[*].spans[*].name}}",{"type":42,"tag":794,"props":11682,"children":11683},{},[11684],{"type":48,"value":11685},"Every span name across the whole session",{"type":42,"tag":762,"props":11687,"children":11688},{},[11689,11698],{"type":42,"tag":794,"props":11690,"children":11691},{},[11692],{"type":42,"tag":73,"props":11693,"children":11695},{"className":11694},[],[11696],{"type":48,"value":11697},"{{traces[*].spans[*].meta.output.value}}",{"type":42,"tag":794,"props":11699,"children":11700},{},[11701],{"type":48,"value":11702},"Every span's output across the session",{"type":42,"tag":762,"props":11704,"children":11705},{},[11706,11715],{"type":42,"tag":794,"props":11707,"children":11708},{},[11709],{"type":42,"tag":73,"props":11710,"children":11712},{"className":11711},[],[11713],{"type":48,"value":11714},"{{traces[*].spans[meta.span.kind:llm].meta.input.messages[*].content}}",{"type":42,"tag":794,"props":11716,"children":11717},{},[11718],{"type":48,"value":11719},"All LLM-span input messages across the session — the user turns",{"type":42,"tag":762,"props":11721,"children":11722},{},[11723,11732],{"type":42,"tag":794,"props":11724,"children":11725},{},[11726],{"type":42,"tag":73,"props":11727,"children":11729},{"className":11728},[],[11730],{"type":48,"value":11731},"{{traces[*].spans[meta.span.kind:llm].meta.output.messages[*].content}}",{"type":42,"tag":794,"props":11733,"children":11734},{},[11735],{"type":48,"value":11736},"All assistant responses across the session",{"type":42,"tag":762,"props":11738,"children":11739},{},[11740,11749],{"type":42,"tag":794,"props":11741,"children":11742},{},[11743],{"type":42,"tag":73,"props":11744,"children":11746},{"className":11745},[],[11747],{"type":48,"value":11748},"{{traces[*].spans[meta.span.kind:tool]}}",{"type":42,"tag":794,"props":11750,"children":11751},{},[11752],{"type":48,"value":11753},"Every tool span (paired in\u002Fout) across the session",{"type":42,"tag":762,"props":11755,"children":11756},{},[11757,11766],{"type":42,"tag":794,"props":11758,"children":11759},{},[11760],{"type":42,"tag":73,"props":11761,"children":11763},{"className":11762},[],[11764],{"type":48,"value":11765},"{{traces[*].spans[meta.span.kind:retrieval].meta.output.documents[*].text}}",{"type":42,"tag":794,"props":11767,"children":11768},{},[11769],{"type":48,"value":11770},"Text of every retrieved document across the session (RAG)",{"type":42,"tag":762,"props":11772,"children":11773},{},[11774,11782],{"type":42,"tag":794,"props":11775,"children":11776},{},[11777],{"type":42,"tag":73,"props":11778,"children":11780},{"className":11779},[],[11781],{"type":48,"value":11364},{"type":42,"tag":794,"props":11783,"children":11784},{},[11785],{"type":48,"value":11786},"Entire session payload as JSON (debug fallback)",{"type":42,"tag":51,"props":11788,"children":11789},{},[11790,11792,11800,11802,11808,11810,11816,11818,11823,11824,11830,11831,11836],{"type":48,"value":11791},"Note: the filter selector applies to the ",{"type":42,"tag":55,"props":11793,"children":11794},{},[11795],{"type":42,"tag":73,"props":11796,"children":11798},{"className":11797},[],[11799],{"type":48,"value":11003},{"type":48,"value":11801}," array (",{"type":42,"tag":73,"props":11803,"children":11805},{"className":11804},[],[11806],{"type":48,"value":11807},"traces[*].spans[meta.span.kind:llm]",{"type":48,"value":11809},"), not to ",{"type":42,"tag":73,"props":11811,"children":11813},{"className":11812},[],[11814],{"type":48,"value":11815},"traces",{"type":48,"value":11817}," — a trace object has only ",{"type":42,"tag":73,"props":11819,"children":11821},{"className":11820},[],[11822],{"type":48,"value":4005},{"type":48,"value":251},{"type":42,"tag":73,"props":11825,"children":11827},{"className":11826},[],[11828],{"type":48,"value":11829},"root_span_id",{"type":48,"value":3536},{"type":42,"tag":73,"props":11832,"children":11834},{"className":11833},[],[11835],{"type":48,"value":11003},{"type":48,"value":11837},", so there is nothing to filter on at the trace level.",{"type":42,"tag":11088,"props":11839,"children":11841},{"id":11840},"array-selector-syntax-applies-to-all-scopes",[11842],{"type":48,"value":11843},"Array selector syntax (applies to all scopes)",{"type":42,"tag":194,"props":11845,"children":11846},{},[11847,11858,11877,11888],{"type":42,"tag":67,"props":11848,"children":11849},{},[11850,11856],{"type":42,"tag":73,"props":11851,"children":11853},{"className":11852},[],[11854],{"type":48,"value":11855},"[N]",{"type":48,"value":11857}," — index (0-based)",{"type":42,"tag":67,"props":11859,"children":11860},{},[11861,11867,11869,11875],{"type":42,"tag":73,"props":11862,"children":11864},{"className":11863},[],[11865],{"type":48,"value":11866},"[START,END]",{"type":48,"value":11868}," — inclusive range, ",{"type":42,"tag":73,"props":11870,"children":11872},{"className":11871},[],[11873],{"type":48,"value":11874},"END",{"type":48,"value":11876}," is clamped to slice length",{"type":42,"tag":67,"props":11878,"children":11879},{},[11880,11886],{"type":42,"tag":73,"props":11881,"children":11883},{"className":11882},[],[11884],{"type":48,"value":11885},"[*]",{"type":48,"value":11887}," — wildcard (fan-out over all elements)",{"type":42,"tag":67,"props":11889,"children":11890},{},[11891,11897,11899,11905,11906],{"type":42,"tag":73,"props":11892,"children":11894},{"className":11893},[],[11895],{"type":48,"value":11896},"[field.path:value]",{"type":48,"value":11898}," — filter array elements by a nested field equality, e.g. ",{"type":42,"tag":73,"props":11900,"children":11902},{"className":11901},[],[11903],{"type":48,"value":11904},"messages[role:user]",{"type":48,"value":5132},{"type":42,"tag":73,"props":11907,"children":11909},{"className":11908},[],[11910],{"type":48,"value":11911},"spans[meta.span.kind:tool]",{"type":42,"tag":51,"props":11913,"children":11914},{},[11915],{"type":42,"tag":55,"props":11916,"children":11917},{},[11918],{"type":48,"value":11919},"Resolution rules to keep in mind when writing prompts:",{"type":42,"tag":194,"props":11921,"children":11922},{},[11923,11928,11933,11938,11957],{"type":42,"tag":67,"props":11924,"children":11925},{},[11926],{"type":48,"value":11927},"Arrays of strings → newline-joined",{"type":42,"tag":67,"props":11929,"children":11930},{},[11931],{"type":48,"value":11932},"Arrays of objects \u002F mixed values → compact JSON",{"type":42,"tag":67,"props":11934,"children":11935},{},[11936],{"type":48,"value":11937},"Single empty slice → empty string",{"type":42,"tag":67,"props":11939,"children":11940},{},[11941,11943,11949,11951],{"type":48,"value":11942},"Implicit fan-out: ",{"type":42,"tag":73,"props":11944,"children":11946},{"className":11945},[],[11947],{"type":48,"value":11948},"messages.content",{"type":48,"value":11950}," behaves the same as ",{"type":42,"tag":73,"props":11952,"children":11954},{"className":11953},[],[11955],{"type":48,"value":11956},"messages[*].content",{"type":42,"tag":67,"props":11958,"children":11959},{},[11960,11962,11966,11968,11973,11975,11980],{"type":48,"value":11961},"Negative indices are ",{"type":42,"tag":55,"props":11963,"children":11964},{},[11965],{"type":48,"value":271},{"type":48,"value":11967}," supported (parse error) — use ",{"type":42,"tag":73,"props":11969,"children":11971},{"className":11970},[],[11972],{"type":48,"value":11855},{"type":48,"value":11974}," with a known index, or ",{"type":42,"tag":73,"props":11976,"children":11978},{"className":11977},[],[11979],{"type":48,"value":11885},{"type":48,"value":11981}," for \"last assistant turn\" semantics",{"type":42,"tag":51,"props":11983,"children":11984},{},[11985],{"type":42,"tag":55,"props":11986,"children":11987},{},[11988],{"type":48,"value":11989},"When to pick which form:",{"type":42,"tag":194,"props":11991,"children":11992},{},[11993,12030,12060,12082,12092,12102],{"type":42,"tag":67,"props":11994,"children":11995},{},[11996,12001,12003,12008,12009,12015,12017,12022,12023,12028],{"type":42,"tag":55,"props":11997,"children":11998},{},[11999],{"type":48,"value":12000},"Generic span evaluator",{"type":48,"value":12002}," (e.g. ",{"type":42,"tag":73,"props":12004,"children":12006},{"className":12005},[],[12007],{"type":48,"value":6754},{"type":48,"value":251},{"type":42,"tag":73,"props":12010,"children":12012},{"className":12011},[],[12013],{"type":48,"value":12014},"output_format",{"type":48,"value":12016},") → use ",{"type":42,"tag":73,"props":12018,"children":12020},{"className":12019},[],[12021],{"type":48,"value":3824},{"type":48,"value":591},{"type":42,"tag":73,"props":12024,"children":12026},{"className":12025},[],[12027],{"type":48,"value":3831},{"type":48,"value":12029}," so it works across span kinds.",{"type":42,"tag":67,"props":12031,"children":12032},{},[12033,12038,12039,12045,12047,12052,12053,12058],{"type":42,"tag":55,"props":12034,"children":12035},{},[12036],{"type":48,"value":12037},"LLM-span-specific evaluator",{"type":48,"value":12002},{"type":42,"tag":73,"props":12040,"children":12042},{"className":12041},[],[12043],{"type":48,"value":12044},"system_prompt_adherence",{"type":48,"value":12046},") → reach for explicit ",{"type":42,"tag":73,"props":12048,"children":12050},{"className":12049},[],[12051],{"type":48,"value":11163},{"type":48,"value":591},{"type":42,"tag":73,"props":12054,"children":12056},{"className":12055},[],[12057],{"type":48,"value":6767},{"type":48,"value":12059}," so you can split system vs. user vs. assistant turns.",{"type":42,"tag":67,"props":12061,"children":12062},{},[12063,12068,12070,12075,12076,12081],{"type":42,"tag":55,"props":12064,"children":12065},{},[12066],{"type":48,"value":12067},"Span-scope RAG evaluator",{"type":48,"value":12069}," (single retrieval+generation span) → combine ",{"type":42,"tag":73,"props":12071,"children":12073},{"className":12072},[],[12074],{"type":48,"value":11314},{"type":48,"value":387},{"type":42,"tag":73,"props":12077,"children":12079},{"className":12078},[],[12080],{"type":48,"value":3831},{"type":48,"value":332},{"type":42,"tag":67,"props":12083,"children":12084},{},[12085,12090],{"type":42,"tag":55,"props":12086,"children":12087},{},[12088],{"type":48,"value":12089},"Trace-scope evaluator",{"type":48,"value":12091}," → see \"Trace-scope evaluator examples\" below for the four canonical patterns (goal completion, tool-use correctness, RAG faithfulness, conversation quality).",{"type":42,"tag":67,"props":12093,"children":12094},{},[12095,12100],{"type":42,"tag":55,"props":12096,"children":12097},{},[12098],{"type":48,"value":12099},"Session-scope evaluator",{"type":48,"value":12101}," → see \"Session-scope evaluator examples\" below for the four canonical patterns (session goal completion, multi-turn conversation quality, user frustration signals, agent consistency across the session).",{"type":42,"tag":67,"props":12103,"children":12104},{},[12105,12110,12112,12117],{"type":42,"tag":55,"props":12106,"children":12107},{},[12108],{"type":48,"value":12109},"Metadata-aware evaluator",{"type":48,"value":12111}," → reference ",{"type":42,"tag":73,"props":12113,"children":12115},{"className":12114},[],[12116],{"type":48,"value":5921},{"type":48,"value":12118}," directly.",{"type":42,"tag":51,"props":12120,"children":12121},{},[12122],{"type":48,"value":12123},"If the user has existing custom evaluators in the same ml_app (Phase 0 coverage map), match their convention when there is no strong reason to deviate.",{"type":42,"tag":11088,"props":12125,"children":12127},{"id":12126},"trace-scope-evaluator-examples",[12128],{"type":48,"value":12129},"Trace-scope evaluator examples",{"type":42,"tag":51,"props":12131,"children":12132},{},[12133,12135,12144],{"type":48,"value":12134},"Concrete user-prompt bodies for the four canonical trace-scope use cases, drawn from the public docs (",{"type":42,"tag":12136,"props":12137,"children":12141},"a",{"href":12138,"rel":12139},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fevaluations\u002Fcustom_llm_as_a_judge_evaluations\u002Ftrace_level_evaluations\u002F",[12140],"nofollow",[12142],{"type":48,"value":12143},"Trace-Level Evaluations",{"type":48,"value":12145},"). Each goes alongside a static System prompt that describes the rubric (no placeholders).",{"type":42,"tag":754,"props":12147,"children":12148},{},[12149,12173],{"type":42,"tag":758,"props":12150,"children":12151},{},[12152],{"type":42,"tag":762,"props":12153,"children":12154},{},[12155,12160,12168],{"type":42,"tag":766,"props":12156,"children":12157},{},[12158],{"type":48,"value":12159},"Use case",{"type":42,"tag":766,"props":12161,"children":12162},{},[12163],{"type":42,"tag":73,"props":12164,"children":12166},{"className":12165},[],[12167],{"type":48,"value":5065},{"type":42,"tag":766,"props":12169,"children":12170},{},[12171],{"type":48,"value":12172},"User prompt body",{"type":42,"tag":787,"props":12174,"children":12175},{},[12176,12207,12237,12268],{"type":42,"tag":762,"props":12177,"children":12178},{},[12179,12189,12198],{"type":42,"tag":794,"props":12180,"children":12181},{},[12182,12187],{"type":42,"tag":55,"props":12183,"children":12184},{},[12185],{"type":48,"value":12186},"Goal completion",{"type":48,"value":12188}," — agent finished the user's request",{"type":42,"tag":794,"props":12190,"children":12191},{},[12192],{"type":42,"tag":73,"props":12193,"children":12195},{"className":12194},[],[12196],{"type":48,"value":12197},"@parent_id:undefined @meta.span.kind:agent",{"type":42,"tag":794,"props":12199,"children":12200},{},[12201],{"type":42,"tag":73,"props":12202,"children":12204},{"className":12203},[],[12205],{"type":48,"value":12206},"User goal:\\n{{spans[0].meta.input.value}}\\n\\nAgent steps:\\n{{spans}}",{"type":42,"tag":762,"props":12208,"children":12209},{},[12210,12220,12228],{"type":42,"tag":794,"props":12211,"children":12212},{},[12213,12218],{"type":42,"tag":55,"props":12214,"children":12215},{},[12216],{"type":48,"value":12217},"Tool-use correctness",{"type":48,"value":12219}," — right tool with right arguments",{"type":42,"tag":794,"props":12221,"children":12222},{},[12223],{"type":42,"tag":73,"props":12224,"children":12226},{"className":12225},[],[12227],{"type":48,"value":12197},{"type":42,"tag":794,"props":12229,"children":12230},{},[12231],{"type":42,"tag":73,"props":12232,"children":12234},{"className":12233},[],[12235],{"type":48,"value":12236},"User question:\\n{{spans[0].meta.input.value}}\\n\\nTool calls:\\n{{spans[meta.span.kind:tool].meta.input.parameters}}\\n\\nFinal response:\\n{{spans[*].meta.output.value}}",{"type":42,"tag":762,"props":12238,"children":12239},{},[12240,12250,12259],{"type":42,"tag":794,"props":12241,"children":12242},{},[12243,12248],{"type":42,"tag":55,"props":12244,"children":12245},{},[12246],{"type":48,"value":12247},"RAG faithfulness",{"type":48,"value":12249}," — answer grounded in retrieved docs",{"type":42,"tag":794,"props":12251,"children":12252},{},[12253],{"type":42,"tag":73,"props":12254,"children":12256},{"className":12255},[],[12257],{"type":48,"value":12258},"@parent_id:undefined",{"type":42,"tag":794,"props":12260,"children":12261},{},[12262],{"type":42,"tag":73,"props":12263,"children":12265},{"className":12264},[],[12266],{"type":48,"value":12267},"Retrieved context:\\n{{spans[meta.span.kind:retrieval].meta.output.documents[*].text}}\\n\\nFinal answer:\\n{{spans[meta.span.kind:llm].meta.output.value}}",{"type":42,"tag":762,"props":12269,"children":12270},{},[12271,12281,12289],{"type":42,"tag":794,"props":12272,"children":12273},{},[12274,12279],{"type":42,"tag":55,"props":12275,"children":12276},{},[12277],{"type":48,"value":12278},"Conversation quality",{"type":48,"value":12280}," — coherence and consistency across turns",{"type":42,"tag":794,"props":12282,"children":12283},{},[12284],{"type":42,"tag":73,"props":12285,"children":12287},{"className":12286},[],[12288],{"type":48,"value":12258},{"type":42,"tag":794,"props":12290,"children":12291},{},[12292],{"type":42,"tag":73,"props":12293,"children":12295},{"className":12294},[],[12296],{"type":48,"value":12297},"Conversation:\\n{{spans[meta.span.kind:llm].meta.input.messages[*].content}}\\n\\nAssistant responses:\\n{{spans[meta.span.kind:llm].meta.output.messages[*].content}}",{"type":42,"tag":51,"props":12299,"children":12300},{},[12301,12303,12308],{"type":48,"value":12302},"Use these as starting points. Adapt the ",{"type":42,"tag":73,"props":12304,"children":12306},{"className":12305},[],[12307],{"type":48,"value":5065},{"type":48,"value":12309}," and span paths to the actual span names \u002F kinds the app emits (observed during Phase 1).",{"type":42,"tag":11088,"props":12311,"children":12313},{"id":12312},"session-scope-evaluator-examples",[12314],{"type":48,"value":12315},"Session-scope evaluator examples",{"type":42,"tag":51,"props":12317,"children":12318},{},[12319,12321,12328,12330,12335,12337,12342,12344,12349],{"type":48,"value":12320},"Concrete user-prompt bodies for the four canonical session-scope use cases (adapted from the public ",{"type":42,"tag":12136,"props":12322,"children":12325},{"href":12323,"rel":12324},"https:\u002F\u002Fdocs.datadoghq.com\u002Fllm_observability\u002Fevaluations\u002Fcustom_llm_as_a_judge_evaluations\u002Fsession_level_evaluations\u002F",[12140],[12326],{"type":48,"value":12327},"Session-Level Evaluations",{"type":48,"value":12329}," docs). Each goes alongside a static System prompt that describes the rubric (no placeholders). The ",{"type":42,"tag":73,"props":12331,"children":12333},{"className":12332},[],[12334],{"type":48,"value":5065},{"type":48,"value":12336}," is matched against the ",{"type":42,"tag":55,"props":12338,"children":12339},{},[12340],{"type":48,"value":12341},"session's root span",{"type":48,"value":12343}," (not every span), so scope it with root-span attributes; there is no ",{"type":42,"tag":73,"props":12345,"children":12347},{"className":12346},[],[12348],{"type":48,"value":12258},{"type":48,"value":12350}," requirement beyond that.",{"type":42,"tag":754,"props":12352,"children":12353},{},[12354,12376],{"type":42,"tag":758,"props":12355,"children":12356},{},[12357],{"type":42,"tag":762,"props":12358,"children":12359},{},[12360,12364,12372],{"type":42,"tag":766,"props":12361,"children":12362},{},[12363],{"type":48,"value":12159},{"type":42,"tag":766,"props":12365,"children":12366},{},[12367],{"type":42,"tag":73,"props":12368,"children":12370},{"className":12369},[],[12371],{"type":48,"value":5065},{"type":42,"tag":766,"props":12373,"children":12374},{},[12375],{"type":48,"value":12172},{"type":42,"tag":787,"props":12377,"children":12378},{},[12379,12412,12442,12472],{"type":42,"tag":762,"props":12380,"children":12381},{},[12382,12392,12403],{"type":42,"tag":794,"props":12383,"children":12384},{},[12385,12390],{"type":42,"tag":55,"props":12386,"children":12387},{},[12388],{"type":48,"value":12389},"Session goal completion",{"type":48,"value":12391}," — user's goals met across the session",{"type":42,"tag":794,"props":12393,"children":12394},{},[12395,12401],{"type":42,"tag":73,"props":12396,"children":12398},{"className":12397},[],[12399],{"type":48,"value":12400},"@meta.span.kind:agent",{"type":48,"value":12402}," (or the app's root span kind)",{"type":42,"tag":794,"props":12404,"children":12405},{},[12406],{"type":42,"tag":73,"props":12407,"children":12409},{"className":12408},[],[12410],{"type":48,"value":12411},"Session traces (chronological):\\n{{traces}}",{"type":42,"tag":762,"props":12413,"children":12414},{},[12415,12425,12433],{"type":42,"tag":794,"props":12416,"children":12417},{},[12418,12423],{"type":42,"tag":55,"props":12419,"children":12420},{},[12421],{"type":48,"value":12422},"Multi-turn conversation quality",{"type":48,"value":12424}," — coherence and memory across turns",{"type":42,"tag":794,"props":12426,"children":12427},{},[12428],{"type":42,"tag":73,"props":12429,"children":12431},{"className":12430},[],[12432],{"type":48,"value":12400},{"type":42,"tag":794,"props":12434,"children":12435},{},[12436],{"type":42,"tag":73,"props":12437,"children":12439},{"className":12438},[],[12440],{"type":48,"value":12441},"User messages:\\n{{traces[*].spans[meta.span.kind:llm].meta.input.messages[*].content}}\\n\\nAssistant responses:\\n{{traces[*].spans[meta.span.kind:llm].meta.output.messages[*].content}}",{"type":42,"tag":762,"props":12443,"children":12444},{},[12445,12455,12463],{"type":42,"tag":794,"props":12446,"children":12447},{},[12448,12453],{"type":42,"tag":55,"props":12449,"children":12450},{},[12451],{"type":48,"value":12452},"User frustration signals",{"type":48,"value":12454}," — frustration \u002F abandonment over the session",{"type":42,"tag":794,"props":12456,"children":12457},{},[12458],{"type":42,"tag":73,"props":12459,"children":12461},{"className":12460},[],[12462],{"type":48,"value":12400},{"type":42,"tag":794,"props":12464,"children":12465},{},[12466],{"type":42,"tag":73,"props":12467,"children":12469},{"className":12468},[],[12470],{"type":48,"value":12471},"Full session:\\n{{traces}}",{"type":42,"tag":762,"props":12473,"children":12474},{},[12475,12485,12493],{"type":42,"tag":794,"props":12476,"children":12477},{},[12478,12483],{"type":42,"tag":55,"props":12479,"children":12480},{},[12481],{"type":48,"value":12482},"Agent consistency across the session",{"type":48,"value":12484}," — consistent and recovers across traces",{"type":42,"tag":794,"props":12486,"children":12487},{},[12488],{"type":42,"tag":73,"props":12489,"children":12491},{"className":12490},[],[12492],{"type":48,"value":12400},{"type":42,"tag":794,"props":12494,"children":12495},{},[12496],{"type":42,"tag":73,"props":12497,"children":12499},{"className":12498},[],[12500],{"type":48,"value":12411},{"type":42,"tag":51,"props":12502,"children":12503},{},[12504,12506,12511],{"type":48,"value":12505},"Adapt the ",{"type":42,"tag":73,"props":12507,"children":12509},{"className":12508},[],[12510],{"type":48,"value":5065},{"type":48,"value":12512}," and span paths to the actual span kinds the app emits (observed in Phase 1).",{"type":42,"tag":51,"props":12514,"children":12515},{},[12516],{"type":42,"tag":55,"props":12517,"children":12518},{},[12519,12524],{"type":42,"tag":73,"props":12520,"children":12522},{"className":12521},[],[12523],{"type":48,"value":10581},{"type":48,"value":12525}," wrapper format (required for all providers)",{"type":42,"tag":51,"props":12527,"children":12528},{},[12529,12530,12535,12537,12543,12545,12555],{"type":48,"value":2749},{"type":42,"tag":73,"props":12531,"children":12533},{"className":12532},[],[12534],{"type":48,"value":10581},{"type":48,"value":12536}," field is NOT a bare JSON Schema. It must use the OpenAI ",{"type":42,"tag":73,"props":12538,"children":12540},{"className":12539},[],[12541],{"type":48,"value":12542},"json_schema",{"type":48,"value":12544}," object shape. ",{"type":42,"tag":55,"props":12546,"children":12547},{},[12548,12553],{"type":42,"tag":73,"props":12549,"children":12551},{"className":12550},[],[12552],{"type":48,"value":8066},{"type":48,"value":12554}," is a fixed type discriminator",{"type":48,"value":12556},", not the evaluator name — the UI validates it against a strict allowlist and rejects any other value:",{"type":42,"tag":754,"props":12558,"children":12559},{},[12560,12592],{"type":42,"tag":758,"props":12561,"children":12562},{},[12563],{"type":42,"tag":762,"props":12564,"children":12565},{},[12566,12571,12581],{"type":42,"tag":766,"props":12567,"children":12568},{},[12569],{"type":48,"value":12570},"LLMJudge type",{"type":42,"tag":766,"props":12572,"children":12573},{},[12574,12579],{"type":42,"tag":73,"props":12575,"children":12577},{"className":12576},[],[12578],{"type":48,"value":8066},{"type":48,"value":12580}," value",{"type":42,"tag":766,"props":12582,"children":12583},{},[12584,12586],{"type":48,"value":12585},"property key inside ",{"type":42,"tag":73,"props":12587,"children":12589},{"className":12588},[],[12590],{"type":48,"value":12591},"schema",{"type":42,"tag":787,"props":12593,"children":12594},{},[12595,12620,12645],{"type":42,"tag":762,"props":12596,"children":12597},{},[12598,12602,12611],{"type":42,"tag":794,"props":12599,"children":12600},{},[12601],{"type":48,"value":2328},{"type":42,"tag":794,"props":12603,"children":12604},{},[12605],{"type":42,"tag":73,"props":12606,"children":12608},{"className":12607},[],[12609],{"type":48,"value":12610},"\"boolean_eval\"",{"type":42,"tag":794,"props":12612,"children":12613},{},[12614],{"type":42,"tag":73,"props":12615,"children":12617},{"className":12616},[],[12618],{"type":48,"value":12619},"boolean_eval",{"type":42,"tag":762,"props":12621,"children":12622},{},[12623,12627,12636],{"type":42,"tag":794,"props":12624,"children":12625},{},[12626],{"type":48,"value":2392},{"type":42,"tag":794,"props":12628,"children":12629},{},[12630],{"type":42,"tag":73,"props":12631,"children":12633},{"className":12632},[],[12634],{"type":48,"value":12635},"\"score_eval\"",{"type":42,"tag":794,"props":12637,"children":12638},{},[12639],{"type":42,"tag":73,"props":12640,"children":12642},{"className":12641},[],[12643],{"type":48,"value":12644},"score_eval",{"type":42,"tag":762,"props":12646,"children":12647},{},[12648,12652,12661],{"type":42,"tag":794,"props":12649,"children":12650},{},[12651],{"type":48,"value":2480},{"type":42,"tag":794,"props":12653,"children":12654},{},[12655],{"type":42,"tag":73,"props":12656,"children":12658},{"className":12657},[],[12659],{"type":48,"value":12660},"\"categorical_eval\"",{"type":42,"tag":794,"props":12662,"children":12663},{},[12664],{"type":42,"tag":73,"props":12665,"children":12667},{"className":12666},[],[12668],{"type":48,"value":12669},"categorical_eval",{"type":42,"tag":51,"props":12671,"children":12672},{},[12673,12675,12681,12683,12688,12690,12696,12698,12704,12705,12711,12713,12719],{"type":48,"value":12674},"The property key inside ",{"type":42,"tag":73,"props":12676,"children":12678},{"className":12677},[],[12679],{"type":48,"value":12680},"schema.properties",{"type":48,"value":12682}," must match ",{"type":42,"tag":73,"props":12684,"children":12686},{"className":12685},[],[12687],{"type":48,"value":8066},{"type":48,"value":12689}," exactly. The ",{"type":42,"tag":73,"props":12691,"children":12693},{"className":12692},[],[12694],{"type":48,"value":12695},"required",{"type":48,"value":12697}," array may only be ",{"type":42,"tag":73,"props":12699,"children":12701},{"className":12700},[],[12702],{"type":48,"value":12703},"[\"\u003Ctype_key>\"]",{"type":48,"value":5132},{"type":42,"tag":73,"props":12706,"children":12708},{"className":12707},[],[12709],{"type":48,"value":12710},"[\"\u003Ctype_key>\", \"reasoning\"]",{"type":48,"value":12712}," — any other value is rejected. Always include ",{"type":42,"tag":73,"props":12714,"children":12716},{"className":12715},[],[12717],{"type":48,"value":12718},"\"reasoning\": {\"type\": \"string\"}",{"type":48,"value":12720}," for UI display.",{"type":42,"tag":51,"props":12722,"children":12723},{},[12724,12728,12729,12735],{"type":42,"tag":55,"props":12725,"children":12726},{},[12727],{"type":48,"value":2328},{"type":48,"value":497},{"type":42,"tag":73,"props":12730,"children":12732},{"className":12731},[],[12733],{"type":48,"value":12734},"BooleanStructuredOutput(pass_when=True)",{"type":48,"value":12736},"):",{"type":42,"tag":730,"props":12738,"children":12740},{"className":8722,"code":12739,"language":8724,"meta":738,"style":738},"{\n  \"output_schema\": {\n    \"name\": \"boolean_eval\",\n    \"strict\": true,\n    \"schema\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"boolean_eval\": {\"type\": \"boolean\", \"description\": \"Whether the criterion is met\"},\n        \"reasoning\": {\"type\": \"string\", \"description\": \"Explanation for the evaluation\"}\n      },\n      \"required\": [\"boolean_eval\", \"reasoning\"],\n      \"additionalProperties\": false\n    }\n  },\n  \"assessment_criteria\": {\"pass_when\": true}\n}\n",[12741],{"type":42,"tag":73,"props":12742,"children":12743},{"__ignoreMap":738},[12744,12751,12774,12809,12834,12857,12893,12917,13011,13101,13108,13163,13188,13195,13202,13248],{"type":42,"tag":1825,"props":12745,"children":12746},{"class":1827,"line":1828},[12747],{"type":42,"tag":1825,"props":12748,"children":12749},{"style":8275},[12750],{"type":48,"value":8736},{"type":42,"tag":1825,"props":12752,"children":12753},{"class":1827,"line":1837},[12754,12758,12762,12766,12770],{"type":42,"tag":1825,"props":12755,"children":12756},{"style":8275},[12757],{"type":48,"value":8744},{"type":42,"tag":1825,"props":12759,"children":12760},{"style":8747},[12761],{"type":48,"value":10581},{"type":42,"tag":1825,"props":12763,"children":12764},{"style":8275},[12765],{"type":48,"value":8755},{"type":42,"tag":1825,"props":12767,"children":12768},{"style":8275},[12769],{"type":48,"value":4920},{"type":42,"tag":1825,"props":12771,"children":12772},{"style":8275},[12773],{"type":48,"value":8877},{"type":42,"tag":1825,"props":12775,"children":12776},{"class":1827,"line":1846},[12777,12781,12785,12789,12793,12797,12801,12805],{"type":42,"tag":1825,"props":12778,"children":12779},{"style":8275},[12780],{"type":48,"value":8885},{"type":42,"tag":1825,"props":12782,"children":12783},{"style":8281},[12784],{"type":48,"value":8066},{"type":42,"tag":1825,"props":12786,"children":12787},{"style":8275},[12788],{"type":48,"value":8755},{"type":42,"tag":1825,"props":12790,"children":12791},{"style":8275},[12792],{"type":48,"value":4920},{"type":42,"tag":1825,"props":12794,"children":12795},{"style":8275},[12796],{"type":48,"value":8764},{"type":42,"tag":1825,"props":12798,"children":12799},{"style":8380},[12800],{"type":48,"value":12619},{"type":42,"tag":1825,"props":12802,"children":12803},{"style":8275},[12804],{"type":48,"value":8755},{"type":42,"tag":1825,"props":12806,"children":12807},{"style":8275},[12808],{"type":48,"value":8778},{"type":42,"tag":1825,"props":12810,"children":12811},{"class":1827,"line":1856},[12812,12816,12821,12825,12829],{"type":42,"tag":1825,"props":12813,"children":12814},{"style":8275},[12815],{"type":48,"value":8885},{"type":42,"tag":1825,"props":12817,"children":12818},{"style":8281},[12819],{"type":48,"value":12820},"strict",{"type":42,"tag":1825,"props":12822,"children":12823},{"style":8275},[12824],{"type":48,"value":8755},{"type":42,"tag":1825,"props":12826,"children":12827},{"style":8275},[12828],{"type":48,"value":4920},{"type":42,"tag":1825,"props":12830,"children":12831},{"style":8275},[12832],{"type":48,"value":12833}," true,\n",{"type":42,"tag":1825,"props":12835,"children":12836},{"class":1827,"line":1865},[12837,12841,12845,12849,12853],{"type":42,"tag":1825,"props":12838,"children":12839},{"style":8275},[12840],{"type":48,"value":8885},{"type":42,"tag":1825,"props":12842,"children":12843},{"style":8281},[12844],{"type":48,"value":12591},{"type":42,"tag":1825,"props":12846,"children":12847},{"style":8275},[12848],{"type":48,"value":8755},{"type":42,"tag":1825,"props":12850,"children":12851},{"style":8275},[12852],{"type":48,"value":4920},{"type":42,"tag":1825,"props":12854,"children":12855},{"style":8275},[12856],{"type":48,"value":8877},{"type":42,"tag":1825,"props":12858,"children":12859},{"class":1827,"line":1874},[12860,12864,12868,12872,12876,12880,12885,12889],{"type":42,"tag":1825,"props":12861,"children":12862},{"style":8275},[12863],{"type":48,"value":9070},{"type":42,"tag":1825,"props":12865,"children":12866},{"style":9280},[12867],{"type":48,"value":8085},{"type":42,"tag":1825,"props":12869,"children":12870},{"style":8275},[12871],{"type":48,"value":8755},{"type":42,"tag":1825,"props":12873,"children":12874},{"style":8275},[12875],{"type":48,"value":4920},{"type":42,"tag":1825,"props":12877,"children":12878},{"style":8275},[12879],{"type":48,"value":8764},{"type":42,"tag":1825,"props":12881,"children":12882},{"style":8380},[12883],{"type":48,"value":12884},"object",{"type":42,"tag":1825,"props":12886,"children":12887},{"style":8275},[12888],{"type":48,"value":8755},{"type":42,"tag":1825,"props":12890,"children":12891},{"style":8275},[12892],{"type":48,"value":8778},{"type":42,"tag":1825,"props":12894,"children":12895},{"class":1827,"line":1883},[12896,12900,12905,12909,12913],{"type":42,"tag":1825,"props":12897,"children":12898},{"style":8275},[12899],{"type":48,"value":9070},{"type":42,"tag":1825,"props":12901,"children":12902},{"style":9280},[12903],{"type":48,"value":12904},"properties",{"type":42,"tag":1825,"props":12906,"children":12907},{"style":8275},[12908],{"type":48,"value":8755},{"type":42,"tag":1825,"props":12910,"children":12911},{"style":8275},[12912],{"type":48,"value":4920},{"type":42,"tag":1825,"props":12914,"children":12915},{"style":8275},[12916],{"type":48,"value":8877},{"type":42,"tag":1825,"props":12918,"children":12919},{"class":1827,"line":1892},[12920,12924,12929,12933,12937,12942,12946,12951,12955,12959,12963,12968,12972,12977,12981,12985,12989,12993,12997,13002,13006],{"type":42,"tag":1825,"props":12921,"children":12922},{"style":8275},[12923],{"type":48,"value":9277},{"type":42,"tag":1825,"props":12925,"children":12927},{"style":12926},"--shiki-light:#E53935;--shiki-default:#F07178;--shiki-dark:#F07178",[12928],{"type":48,"value":12619},{"type":42,"tag":1825,"props":12930,"children":12931},{"style":8275},[12932],{"type":48,"value":8755},{"type":42,"tag":1825,"props":12934,"children":12935},{"style":8275},[12936],{"type":48,"value":4920},{"type":42,"tag":1825,"props":12938,"children":12939},{"style":8275},[12940],{"type":48,"value":12941}," {",{"type":42,"tag":1825,"props":12943,"children":12944},{"style":8275},[12945],{"type":48,"value":8755},{"type":42,"tag":1825,"props":12947,"children":12949},{"style":12948},"--shiki-light:#916B53;--shiki-default:#916B53;--shiki-dark:#916B53",[12950],{"type":48,"value":8085},{"type":42,"tag":1825,"props":12952,"children":12953},{"style":8275},[12954],{"type":48,"value":8755},{"type":42,"tag":1825,"props":12956,"children":12957},{"style":8275},[12958],{"type":48,"value":4920},{"type":42,"tag":1825,"props":12960,"children":12961},{"style":8275},[12962],{"type":48,"value":8764},{"type":42,"tag":1825,"props":12964,"children":12965},{"style":8380},[12966],{"type":48,"value":12967},"boolean",{"type":42,"tag":1825,"props":12969,"children":12970},{"style":8275},[12971],{"type":48,"value":8755},{"type":42,"tag":1825,"props":12973,"children":12974},{"style":8275},[12975],{"type":48,"value":12976},",",{"type":42,"tag":1825,"props":12978,"children":12979},{"style":8275},[12980],{"type":48,"value":8764},{"type":42,"tag":1825,"props":12982,"children":12983},{"style":12948},[12984],{"type":48,"value":2697},{"type":42,"tag":1825,"props":12986,"children":12987},{"style":8275},[12988],{"type":48,"value":8755},{"type":42,"tag":1825,"props":12990,"children":12991},{"style":8275},[12992],{"type":48,"value":4920},{"type":42,"tag":1825,"props":12994,"children":12995},{"style":8275},[12996],{"type":48,"value":8764},{"type":42,"tag":1825,"props":12998,"children":12999},{"style":8380},[13000],{"type":48,"value":13001},"Whether the criterion is met",{"type":42,"tag":1825,"props":13003,"children":13004},{"style":8275},[13005],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13007,"children":13008},{"style":8275},[13009],{"type":48,"value":13010},"},\n",{"type":42,"tag":1825,"props":13012,"children":13013},{"class":1827,"line":1901},[13014,13018,13023,13027,13031,13035,13039,13043,13047,13051,13055,13060,13064,13068,13072,13076,13080,13084,13088,13093,13097],{"type":42,"tag":1825,"props":13015,"children":13016},{"style":8275},[13017],{"type":48,"value":9277},{"type":42,"tag":1825,"props":13019,"children":13020},{"style":12926},[13021],{"type":48,"value":13022},"reasoning",{"type":42,"tag":1825,"props":13024,"children":13025},{"style":8275},[13026],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13028,"children":13029},{"style":8275},[13030],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13032,"children":13033},{"style":8275},[13034],{"type":48,"value":12941},{"type":42,"tag":1825,"props":13036,"children":13037},{"style":8275},[13038],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13040,"children":13041},{"style":12948},[13042],{"type":48,"value":8085},{"type":42,"tag":1825,"props":13044,"children":13045},{"style":8275},[13046],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13048,"children":13049},{"style":8275},[13050],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13052,"children":13053},{"style":8275},[13054],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13056,"children":13057},{"style":8380},[13058],{"type":48,"value":13059},"string",{"type":42,"tag":1825,"props":13061,"children":13062},{"style":8275},[13063],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13065,"children":13066},{"style":8275},[13067],{"type":48,"value":12976},{"type":42,"tag":1825,"props":13069,"children":13070},{"style":8275},[13071],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13073,"children":13074},{"style":12948},[13075],{"type":48,"value":2697},{"type":42,"tag":1825,"props":13077,"children":13078},{"style":8275},[13079],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13081,"children":13082},{"style":8275},[13083],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13085,"children":13086},{"style":8275},[13087],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13089,"children":13090},{"style":8380},[13091],{"type":48,"value":13092},"Explanation for the evaluation",{"type":42,"tag":1825,"props":13094,"children":13095},{"style":8275},[13096],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13098,"children":13099},{"style":8275},[13100],{"type":48,"value":2670},{"type":42,"tag":1825,"props":13102,"children":13103},{"class":1827,"line":1910},[13104],{"type":42,"tag":1825,"props":13105,"children":13106},{"style":8275},[13107],{"type":48,"value":9392},{"type":42,"tag":1825,"props":13109,"children":13110},{"class":1827,"line":1919},[13111,13115,13119,13123,13127,13131,13135,13139,13143,13147,13151,13155,13159],{"type":42,"tag":1825,"props":13112,"children":13113},{"style":8275},[13114],{"type":48,"value":9070},{"type":42,"tag":1825,"props":13116,"children":13117},{"style":9280},[13118],{"type":48,"value":12695},{"type":42,"tag":1825,"props":13120,"children":13121},{"style":8275},[13122],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13124,"children":13125},{"style":8275},[13126],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13128,"children":13129},{"style":8275},[13130],{"type":48,"value":9333},{"type":42,"tag":1825,"props":13132,"children":13133},{"style":8275},[13134],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13136,"children":13137},{"style":8380},[13138],{"type":48,"value":12619},{"type":42,"tag":1825,"props":13140,"children":13141},{"style":8275},[13142],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13144,"children":13145},{"style":8275},[13146],{"type":48,"value":12976},{"type":42,"tag":1825,"props":13148,"children":13149},{"style":8275},[13150],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13152,"children":13153},{"style":8380},[13154],{"type":48,"value":13022},{"type":42,"tag":1825,"props":13156,"children":13157},{"style":8275},[13158],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13160,"children":13161},{"style":8275},[13162],{"type":48,"value":9351},{"type":42,"tag":1825,"props":13164,"children":13165},{"class":1827,"line":1927},[13166,13170,13175,13179,13183],{"type":42,"tag":1825,"props":13167,"children":13168},{"style":8275},[13169],{"type":48,"value":9070},{"type":42,"tag":1825,"props":13171,"children":13172},{"style":9280},[13173],{"type":48,"value":13174},"additionalProperties",{"type":42,"tag":1825,"props":13176,"children":13177},{"style":8275},[13178],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13180,"children":13181},{"style":8275},[13182],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13184,"children":13185},{"style":8275},[13186],{"type":48,"value":13187}," false\n",{"type":42,"tag":1825,"props":13189,"children":13190},{"class":1827,"line":1936},[13191],{"type":42,"tag":1825,"props":13192,"children":13193},{"style":8275},[13194],{"type":48,"value":9767},{"type":42,"tag":1825,"props":13196,"children":13197},{"class":1827,"line":1945},[13198],{"type":42,"tag":1825,"props":13199,"children":13200},{"style":8275},[13201],{"type":48,"value":9030},{"type":42,"tag":1825,"props":13203,"children":13204},{"class":1827,"line":7604},[13205,13209,13214,13218,13222,13226,13230,13235,13239,13243],{"type":42,"tag":1825,"props":13206,"children":13207},{"style":8275},[13208],{"type":48,"value":8744},{"type":42,"tag":1825,"props":13210,"children":13211},{"style":8747},[13212],{"type":48,"value":13213},"assessment_criteria",{"type":42,"tag":1825,"props":13215,"children":13216},{"style":8275},[13217],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13219,"children":13220},{"style":8275},[13221],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13223,"children":13224},{"style":8275},[13225],{"type":48,"value":12941},{"type":42,"tag":1825,"props":13227,"children":13228},{"style":8275},[13229],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13231,"children":13232},{"style":8281},[13233],{"type":48,"value":13234},"pass_when",{"type":42,"tag":1825,"props":13236,"children":13237},{"style":8275},[13238],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13240,"children":13241},{"style":8275},[13242],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13244,"children":13245},{"style":8275},[13246],{"type":48,"value":13247}," true}\n",{"type":42,"tag":1825,"props":13249,"children":13250},{"class":1827,"line":7613},[13251],{"type":42,"tag":1825,"props":13252,"children":13253},{"style":8275},[13254],{"type":48,"value":2670},{"type":42,"tag":51,"props":13256,"children":13257},{},[13258,13262,13263,13269],{"type":42,"tag":55,"props":13259,"children":13260},{},[13261],{"type":48,"value":2392},{"type":48,"value":497},{"type":42,"tag":73,"props":13264,"children":13266},{"className":13265},[],[13267],{"type":48,"value":13268},"ScoreStructuredOutput(min_score=1, max_score=10, min_threshold=7)",{"type":48,"value":12736},{"type":42,"tag":730,"props":13271,"children":13273},{"className":8722,"code":13272,"language":8724,"meta":738,"style":738},"{\n  \"output_schema\": {\n    \"name\": \"score_eval\",\n    \"strict\": true,\n    \"schema\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"score_eval\": {\"type\": \"number\", \"description\": \"Score from 1 to 10\", \"minimum\": 1, \"maximum\": 10},\n        \"reasoning\": {\"type\": \"string\", \"description\": \"Explanation for the score\"}\n      },\n      \"required\": [\"score_eval\", \"reasoning\"],\n      \"additionalProperties\": false\n    }\n  },\n  \"assessment_criteria\": {\"min_threshold\": 7}\n}\n",[13274],{"type":42,"tag":73,"props":13275,"children":13276},{"__ignoreMap":738},[13277,13284,13307,13342,13365,13388,13423,13446,13585,13673,13680,13735,13758,13765,13772,13821],{"type":42,"tag":1825,"props":13278,"children":13279},{"class":1827,"line":1828},[13280],{"type":42,"tag":1825,"props":13281,"children":13282},{"style":8275},[13283],{"type":48,"value":8736},{"type":42,"tag":1825,"props":13285,"children":13286},{"class":1827,"line":1837},[13287,13291,13295,13299,13303],{"type":42,"tag":1825,"props":13288,"children":13289},{"style":8275},[13290],{"type":48,"value":8744},{"type":42,"tag":1825,"props":13292,"children":13293},{"style":8747},[13294],{"type":48,"value":10581},{"type":42,"tag":1825,"props":13296,"children":13297},{"style":8275},[13298],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13300,"children":13301},{"style":8275},[13302],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13304,"children":13305},{"style":8275},[13306],{"type":48,"value":8877},{"type":42,"tag":1825,"props":13308,"children":13309},{"class":1827,"line":1846},[13310,13314,13318,13322,13326,13330,13334,13338],{"type":42,"tag":1825,"props":13311,"children":13312},{"style":8275},[13313],{"type":48,"value":8885},{"type":42,"tag":1825,"props":13315,"children":13316},{"style":8281},[13317],{"type":48,"value":8066},{"type":42,"tag":1825,"props":13319,"children":13320},{"style":8275},[13321],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13323,"children":13324},{"style":8275},[13325],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13327,"children":13328},{"style":8275},[13329],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13331,"children":13332},{"style":8380},[13333],{"type":48,"value":12644},{"type":42,"tag":1825,"props":13335,"children":13336},{"style":8275},[13337],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13339,"children":13340},{"style":8275},[13341],{"type":48,"value":8778},{"type":42,"tag":1825,"props":13343,"children":13344},{"class":1827,"line":1856},[13345,13349,13353,13357,13361],{"type":42,"tag":1825,"props":13346,"children":13347},{"style":8275},[13348],{"type":48,"value":8885},{"type":42,"tag":1825,"props":13350,"children":13351},{"style":8281},[13352],{"type":48,"value":12820},{"type":42,"tag":1825,"props":13354,"children":13355},{"style":8275},[13356],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13358,"children":13359},{"style":8275},[13360],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13362,"children":13363},{"style":8275},[13364],{"type":48,"value":12833},{"type":42,"tag":1825,"props":13366,"children":13367},{"class":1827,"line":1865},[13368,13372,13376,13380,13384],{"type":42,"tag":1825,"props":13369,"children":13370},{"style":8275},[13371],{"type":48,"value":8885},{"type":42,"tag":1825,"props":13373,"children":13374},{"style":8281},[13375],{"type":48,"value":12591},{"type":42,"tag":1825,"props":13377,"children":13378},{"style":8275},[13379],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13381,"children":13382},{"style":8275},[13383],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13385,"children":13386},{"style":8275},[13387],{"type":48,"value":8877},{"type":42,"tag":1825,"props":13389,"children":13390},{"class":1827,"line":1874},[13391,13395,13399,13403,13407,13411,13415,13419],{"type":42,"tag":1825,"props":13392,"children":13393},{"style":8275},[13394],{"type":48,"value":9070},{"type":42,"tag":1825,"props":13396,"children":13397},{"style":9280},[13398],{"type":48,"value":8085},{"type":42,"tag":1825,"props":13400,"children":13401},{"style":8275},[13402],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13404,"children":13405},{"style":8275},[13406],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13408,"children":13409},{"style":8275},[13410],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13412,"children":13413},{"style":8380},[13414],{"type":48,"value":12884},{"type":42,"tag":1825,"props":13416,"children":13417},{"style":8275},[13418],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13420,"children":13421},{"style":8275},[13422],{"type":48,"value":8778},{"type":42,"tag":1825,"props":13424,"children":13425},{"class":1827,"line":1883},[13426,13430,13434,13438,13442],{"type":42,"tag":1825,"props":13427,"children":13428},{"style":8275},[13429],{"type":48,"value":9070},{"type":42,"tag":1825,"props":13431,"children":13432},{"style":9280},[13433],{"type":48,"value":12904},{"type":42,"tag":1825,"props":13435,"children":13436},{"style":8275},[13437],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13439,"children":13440},{"style":8275},[13441],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13443,"children":13444},{"style":8275},[13445],{"type":48,"value":8877},{"type":42,"tag":1825,"props":13447,"children":13448},{"class":1827,"line":1892},[13449,13453,13457,13461,13465,13469,13473,13477,13481,13485,13489,13494,13498,13502,13506,13510,13514,13518,13522,13527,13531,13535,13539,13543,13547,13551,13556,13560,13564,13568,13572,13576,13581],{"type":42,"tag":1825,"props":13450,"children":13451},{"style":8275},[13452],{"type":48,"value":9277},{"type":42,"tag":1825,"props":13454,"children":13455},{"style":12926},[13456],{"type":48,"value":12644},{"type":42,"tag":1825,"props":13458,"children":13459},{"style":8275},[13460],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13462,"children":13463},{"style":8275},[13464],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13466,"children":13467},{"style":8275},[13468],{"type":48,"value":12941},{"type":42,"tag":1825,"props":13470,"children":13471},{"style":8275},[13472],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13474,"children":13475},{"style":12948},[13476],{"type":48,"value":8085},{"type":42,"tag":1825,"props":13478,"children":13479},{"style":8275},[13480],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13482,"children":13483},{"style":8275},[13484],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13486,"children":13487},{"style":8275},[13488],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13490,"children":13491},{"style":8380},[13492],{"type":48,"value":13493},"number",{"type":42,"tag":1825,"props":13495,"children":13496},{"style":8275},[13497],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13499,"children":13500},{"style":8275},[13501],{"type":48,"value":12976},{"type":42,"tag":1825,"props":13503,"children":13504},{"style":8275},[13505],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13507,"children":13508},{"style":12948},[13509],{"type":48,"value":2697},{"type":42,"tag":1825,"props":13511,"children":13512},{"style":8275},[13513],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13515,"children":13516},{"style":8275},[13517],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13519,"children":13520},{"style":8275},[13521],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13523,"children":13524},{"style":8380},[13525],{"type":48,"value":13526},"Score from 1 to 10",{"type":42,"tag":1825,"props":13528,"children":13529},{"style":8275},[13530],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13532,"children":13533},{"style":8275},[13534],{"type":48,"value":12976},{"type":42,"tag":1825,"props":13536,"children":13537},{"style":8275},[13538],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13540,"children":13541},{"style":12948},[13542],{"type":48,"value":2681},{"type":42,"tag":1825,"props":13544,"children":13545},{"style":8275},[13546],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13548,"children":13549},{"style":8275},[13550],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13552,"children":13553},{"style":9280},[13554],{"type":48,"value":13555}," 1",{"type":42,"tag":1825,"props":13557,"children":13558},{"style":8275},[13559],{"type":48,"value":12976},{"type":42,"tag":1825,"props":13561,"children":13562},{"style":8275},[13563],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13565,"children":13566},{"style":12948},[13567],{"type":48,"value":2689},{"type":42,"tag":1825,"props":13569,"children":13570},{"style":8275},[13571],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13573,"children":13574},{"style":8275},[13575],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13577,"children":13578},{"style":9280},[13579],{"type":48,"value":13580}," 10",{"type":42,"tag":1825,"props":13582,"children":13583},{"style":8275},[13584],{"type":48,"value":13010},{"type":42,"tag":1825,"props":13586,"children":13587},{"class":1827,"line":1901},[13588,13592,13596,13600,13604,13608,13612,13616,13620,13624,13628,13632,13636,13640,13644,13648,13652,13656,13660,13665,13669],{"type":42,"tag":1825,"props":13589,"children":13590},{"style":8275},[13591],{"type":48,"value":9277},{"type":42,"tag":1825,"props":13593,"children":13594},{"style":12926},[13595],{"type":48,"value":13022},{"type":42,"tag":1825,"props":13597,"children":13598},{"style":8275},[13599],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13601,"children":13602},{"style":8275},[13603],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13605,"children":13606},{"style":8275},[13607],{"type":48,"value":12941},{"type":42,"tag":1825,"props":13609,"children":13610},{"style":8275},[13611],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13613,"children":13614},{"style":12948},[13615],{"type":48,"value":8085},{"type":42,"tag":1825,"props":13617,"children":13618},{"style":8275},[13619],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13621,"children":13622},{"style":8275},[13623],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13625,"children":13626},{"style":8275},[13627],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13629,"children":13630},{"style":8380},[13631],{"type":48,"value":13059},{"type":42,"tag":1825,"props":13633,"children":13634},{"style":8275},[13635],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13637,"children":13638},{"style":8275},[13639],{"type":48,"value":12976},{"type":42,"tag":1825,"props":13641,"children":13642},{"style":8275},[13643],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13645,"children":13646},{"style":12948},[13647],{"type":48,"value":2697},{"type":42,"tag":1825,"props":13649,"children":13650},{"style":8275},[13651],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13653,"children":13654},{"style":8275},[13655],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13657,"children":13658},{"style":8275},[13659],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13661,"children":13662},{"style":8380},[13663],{"type":48,"value":13664},"Explanation for the score",{"type":42,"tag":1825,"props":13666,"children":13667},{"style":8275},[13668],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13670,"children":13671},{"style":8275},[13672],{"type":48,"value":2670},{"type":42,"tag":1825,"props":13674,"children":13675},{"class":1827,"line":1910},[13676],{"type":42,"tag":1825,"props":13677,"children":13678},{"style":8275},[13679],{"type":48,"value":9392},{"type":42,"tag":1825,"props":13681,"children":13682},{"class":1827,"line":1919},[13683,13687,13691,13695,13699,13703,13707,13711,13715,13719,13723,13727,13731],{"type":42,"tag":1825,"props":13684,"children":13685},{"style":8275},[13686],{"type":48,"value":9070},{"type":42,"tag":1825,"props":13688,"children":13689},{"style":9280},[13690],{"type":48,"value":12695},{"type":42,"tag":1825,"props":13692,"children":13693},{"style":8275},[13694],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13696,"children":13697},{"style":8275},[13698],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13700,"children":13701},{"style":8275},[13702],{"type":48,"value":9333},{"type":42,"tag":1825,"props":13704,"children":13705},{"style":8275},[13706],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13708,"children":13709},{"style":8380},[13710],{"type":48,"value":12644},{"type":42,"tag":1825,"props":13712,"children":13713},{"style":8275},[13714],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13716,"children":13717},{"style":8275},[13718],{"type":48,"value":12976},{"type":42,"tag":1825,"props":13720,"children":13721},{"style":8275},[13722],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13724,"children":13725},{"style":8380},[13726],{"type":48,"value":13022},{"type":42,"tag":1825,"props":13728,"children":13729},{"style":8275},[13730],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13732,"children":13733},{"style":8275},[13734],{"type":48,"value":9351},{"type":42,"tag":1825,"props":13736,"children":13737},{"class":1827,"line":1927},[13738,13742,13746,13750,13754],{"type":42,"tag":1825,"props":13739,"children":13740},{"style":8275},[13741],{"type":48,"value":9070},{"type":42,"tag":1825,"props":13743,"children":13744},{"style":9280},[13745],{"type":48,"value":13174},{"type":42,"tag":1825,"props":13747,"children":13748},{"style":8275},[13749],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13751,"children":13752},{"style":8275},[13753],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13755,"children":13756},{"style":8275},[13757],{"type":48,"value":13187},{"type":42,"tag":1825,"props":13759,"children":13760},{"class":1827,"line":1936},[13761],{"type":42,"tag":1825,"props":13762,"children":13763},{"style":8275},[13764],{"type":48,"value":9767},{"type":42,"tag":1825,"props":13766,"children":13767},{"class":1827,"line":1945},[13768],{"type":42,"tag":1825,"props":13769,"children":13770},{"style":8275},[13771],{"type":48,"value":9030},{"type":42,"tag":1825,"props":13773,"children":13774},{"class":1827,"line":7604},[13775,13779,13783,13787,13791,13795,13799,13804,13808,13812,13817],{"type":42,"tag":1825,"props":13776,"children":13777},{"style":8275},[13778],{"type":48,"value":8744},{"type":42,"tag":1825,"props":13780,"children":13781},{"style":8747},[13782],{"type":48,"value":13213},{"type":42,"tag":1825,"props":13784,"children":13785},{"style":8275},[13786],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13788,"children":13789},{"style":8275},[13790],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13792,"children":13793},{"style":8275},[13794],{"type":48,"value":12941},{"type":42,"tag":1825,"props":13796,"children":13797},{"style":8275},[13798],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13800,"children":13801},{"style":8281},[13802],{"type":48,"value":13803},"min_threshold",{"type":42,"tag":1825,"props":13805,"children":13806},{"style":8275},[13807],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13809,"children":13810},{"style":8275},[13811],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13813,"children":13814},{"style":9280},[13815],{"type":48,"value":13816}," 7",{"type":42,"tag":1825,"props":13818,"children":13819},{"style":8275},[13820],{"type":48,"value":2670},{"type":42,"tag":1825,"props":13822,"children":13823},{"class":1827,"line":7613},[13824],{"type":42,"tag":1825,"props":13825,"children":13826},{"style":8275},[13827],{"type":48,"value":2670},{"type":42,"tag":51,"props":13829,"children":13830},{},[13831,13833,13839,13840,13845],{"type":48,"value":13832},"Add ",{"type":42,"tag":73,"props":13834,"children":13836},{"className":13835},[],[13837],{"type":48,"value":13838},"max_threshold",{"type":48,"value":2714},{"type":42,"tag":73,"props":13841,"children":13843},{"className":13842},[],[13844],{"type":48,"value":13213},{"type":48,"value":13846}," if set.",{"type":42,"tag":51,"props":13848,"children":13849},{},[13850,13854,13855,13861],{"type":42,"tag":55,"props":13851,"children":13852},{},[13853],{"type":48,"value":2480},{"type":48,"value":497},{"type":42,"tag":73,"props":13856,"children":13858},{"className":13857},[],[13859],{"type":48,"value":13860},"CategoricalStructuredOutput(categories={...}, pass_values=[...])",{"type":48,"value":12736},{"type":42,"tag":730,"props":13863,"children":13865},{"className":8722,"code":13864,"language":8724,"meta":738,"style":738},"{\n  \"output_schema\": {\n    \"name\": \"categorical_eval\",\n    \"strict\": true,\n    \"schema\": {\n      \"type\": \"object\",\n      \"properties\": {\n        \"categorical_eval\": {\n          \"type\": \"string\",\n          \"anyOf\": [\n            {\"const\": \"correct\", \"description\": \"The response correctly answers the question\"},\n            {\"const\": \"partially_correct\", \"description\": \"Partially correct but missing information\"},\n            {\"const\": \"incorrect\", \"description\": \"The response is wrong or irrelevant\"}\n          ]\n        },\n        \"reasoning\": {\"type\": \"string\", \"description\": \"Explanation for the category chosen\"}\n      },\n      \"required\": [\"categorical_eval\", \"reasoning\"],\n      \"additionalProperties\": false\n    }\n  },\n  \"assessment_criteria\": {\"pass_values\": [\"correct\"]}\n}\n",[13866],{"type":42,"tag":73,"props":13867,"children":13868},{"__ignoreMap":738},[13869,13876,13899,13934,13957,13980,14015,14038,14061,14096,14119,14194,14267,14340,14348,14356,14444,14451,14506,14529,14536,14543,14604],{"type":42,"tag":1825,"props":13870,"children":13871},{"class":1827,"line":1828},[13872],{"type":42,"tag":1825,"props":13873,"children":13874},{"style":8275},[13875],{"type":48,"value":8736},{"type":42,"tag":1825,"props":13877,"children":13878},{"class":1827,"line":1837},[13879,13883,13887,13891,13895],{"type":42,"tag":1825,"props":13880,"children":13881},{"style":8275},[13882],{"type":48,"value":8744},{"type":42,"tag":1825,"props":13884,"children":13885},{"style":8747},[13886],{"type":48,"value":10581},{"type":42,"tag":1825,"props":13888,"children":13889},{"style":8275},[13890],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13892,"children":13893},{"style":8275},[13894],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13896,"children":13897},{"style":8275},[13898],{"type":48,"value":8877},{"type":42,"tag":1825,"props":13900,"children":13901},{"class":1827,"line":1846},[13902,13906,13910,13914,13918,13922,13926,13930],{"type":42,"tag":1825,"props":13903,"children":13904},{"style":8275},[13905],{"type":48,"value":8885},{"type":42,"tag":1825,"props":13907,"children":13908},{"style":8281},[13909],{"type":48,"value":8066},{"type":42,"tag":1825,"props":13911,"children":13912},{"style":8275},[13913],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13915,"children":13916},{"style":8275},[13917],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13919,"children":13920},{"style":8275},[13921],{"type":48,"value":8764},{"type":42,"tag":1825,"props":13923,"children":13924},{"style":8380},[13925],{"type":48,"value":12669},{"type":42,"tag":1825,"props":13927,"children":13928},{"style":8275},[13929],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13931,"children":13932},{"style":8275},[13933],{"type":48,"value":8778},{"type":42,"tag":1825,"props":13935,"children":13936},{"class":1827,"line":1856},[13937,13941,13945,13949,13953],{"type":42,"tag":1825,"props":13938,"children":13939},{"style":8275},[13940],{"type":48,"value":8885},{"type":42,"tag":1825,"props":13942,"children":13943},{"style":8281},[13944],{"type":48,"value":12820},{"type":42,"tag":1825,"props":13946,"children":13947},{"style":8275},[13948],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13950,"children":13951},{"style":8275},[13952],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13954,"children":13955},{"style":8275},[13956],{"type":48,"value":12833},{"type":42,"tag":1825,"props":13958,"children":13959},{"class":1827,"line":1865},[13960,13964,13968,13972,13976],{"type":42,"tag":1825,"props":13961,"children":13962},{"style":8275},[13963],{"type":48,"value":8885},{"type":42,"tag":1825,"props":13965,"children":13966},{"style":8281},[13967],{"type":48,"value":12591},{"type":42,"tag":1825,"props":13969,"children":13970},{"style":8275},[13971],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13973,"children":13974},{"style":8275},[13975],{"type":48,"value":4920},{"type":42,"tag":1825,"props":13977,"children":13978},{"style":8275},[13979],{"type":48,"value":8877},{"type":42,"tag":1825,"props":13981,"children":13982},{"class":1827,"line":1874},[13983,13987,13991,13995,13999,14003,14007,14011],{"type":42,"tag":1825,"props":13984,"children":13985},{"style":8275},[13986],{"type":48,"value":9070},{"type":42,"tag":1825,"props":13988,"children":13989},{"style":9280},[13990],{"type":48,"value":8085},{"type":42,"tag":1825,"props":13992,"children":13993},{"style":8275},[13994],{"type":48,"value":8755},{"type":42,"tag":1825,"props":13996,"children":13997},{"style":8275},[13998],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14000,"children":14001},{"style":8275},[14002],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14004,"children":14005},{"style":8380},[14006],{"type":48,"value":12884},{"type":42,"tag":1825,"props":14008,"children":14009},{"style":8275},[14010],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14012,"children":14013},{"style":8275},[14014],{"type":48,"value":8778},{"type":42,"tag":1825,"props":14016,"children":14017},{"class":1827,"line":1883},[14018,14022,14026,14030,14034],{"type":42,"tag":1825,"props":14019,"children":14020},{"style":8275},[14021],{"type":48,"value":9070},{"type":42,"tag":1825,"props":14023,"children":14024},{"style":9280},[14025],{"type":48,"value":12904},{"type":42,"tag":1825,"props":14027,"children":14028},{"style":8275},[14029],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14031,"children":14032},{"style":8275},[14033],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14035,"children":14036},{"style":8275},[14037],{"type":48,"value":8877},{"type":42,"tag":1825,"props":14039,"children":14040},{"class":1827,"line":1892},[14041,14045,14049,14053,14057],{"type":42,"tag":1825,"props":14042,"children":14043},{"style":8275},[14044],{"type":48,"value":9277},{"type":42,"tag":1825,"props":14046,"children":14047},{"style":12926},[14048],{"type":48,"value":12669},{"type":42,"tag":1825,"props":14050,"children":14051},{"style":8275},[14052],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14054,"children":14055},{"style":8275},[14056],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14058,"children":14059},{"style":8275},[14060],{"type":48,"value":8877},{"type":42,"tag":1825,"props":14062,"children":14063},{"class":1827,"line":1901},[14064,14068,14072,14076,14080,14084,14088,14092],{"type":42,"tag":1825,"props":14065,"children":14066},{"style":8275},[14067],{"type":48,"value":9607},{"type":42,"tag":1825,"props":14069,"children":14070},{"style":12948},[14071],{"type":48,"value":8085},{"type":42,"tag":1825,"props":14073,"children":14074},{"style":8275},[14075],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14077,"children":14078},{"style":8275},[14079],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14081,"children":14082},{"style":8275},[14083],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14085,"children":14086},{"style":8380},[14087],{"type":48,"value":13059},{"type":42,"tag":1825,"props":14089,"children":14090},{"style":8275},[14091],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14093,"children":14094},{"style":8275},[14095],{"type":48,"value":8778},{"type":42,"tag":1825,"props":14097,"children":14098},{"class":1827,"line":1910},[14099,14103,14107,14111,14115],{"type":42,"tag":1825,"props":14100,"children":14101},{"style":8275},[14102],{"type":48,"value":9607},{"type":42,"tag":1825,"props":14104,"children":14105},{"style":12948},[14106],{"type":48,"value":2712},{"type":42,"tag":1825,"props":14108,"children":14109},{"style":8275},[14110],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14112,"children":14113},{"style":8275},[14114],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14116,"children":14117},{"style":8275},[14118],{"type":48,"value":9054},{"type":42,"tag":1825,"props":14120,"children":14121},{"class":1827,"line":1919},[14122,14127,14131,14136,14140,14144,14148,14153,14157,14161,14165,14169,14173,14177,14181,14186,14190],{"type":42,"tag":1825,"props":14123,"children":14124},{"style":8275},[14125],{"type":48,"value":14126},"            {",{"type":42,"tag":1825,"props":14128,"children":14129},{"style":8275},[14130],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14132,"children":14134},{"style":14133},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[14135],{"type":48,"value":2705},{"type":42,"tag":1825,"props":14137,"children":14138},{"style":8275},[14139],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14141,"children":14142},{"style":8275},[14143],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14145,"children":14146},{"style":8275},[14147],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14149,"children":14150},{"style":8380},[14151],{"type":48,"value":14152},"correct",{"type":42,"tag":1825,"props":14154,"children":14155},{"style":8275},[14156],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14158,"children":14159},{"style":8275},[14160],{"type":48,"value":12976},{"type":42,"tag":1825,"props":14162,"children":14163},{"style":8275},[14164],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14166,"children":14167},{"style":14133},[14168],{"type":48,"value":2697},{"type":42,"tag":1825,"props":14170,"children":14171},{"style":8275},[14172],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14174,"children":14175},{"style":8275},[14176],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14178,"children":14179},{"style":8275},[14180],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14182,"children":14183},{"style":8380},[14184],{"type":48,"value":14185},"The response correctly answers the question",{"type":42,"tag":1825,"props":14187,"children":14188},{"style":8275},[14189],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14191,"children":14192},{"style":8275},[14193],{"type":48,"value":13010},{"type":42,"tag":1825,"props":14195,"children":14196},{"class":1827,"line":1927},[14197,14201,14205,14209,14213,14217,14221,14226,14230,14234,14238,14242,14246,14250,14254,14259,14263],{"type":42,"tag":1825,"props":14198,"children":14199},{"style":8275},[14200],{"type":48,"value":14126},{"type":42,"tag":1825,"props":14202,"children":14203},{"style":8275},[14204],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14206,"children":14207},{"style":14133},[14208],{"type":48,"value":2705},{"type":42,"tag":1825,"props":14210,"children":14211},{"style":8275},[14212],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14214,"children":14215},{"style":8275},[14216],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14218,"children":14219},{"style":8275},[14220],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14222,"children":14223},{"style":8380},[14224],{"type":48,"value":14225},"partially_correct",{"type":42,"tag":1825,"props":14227,"children":14228},{"style":8275},[14229],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14231,"children":14232},{"style":8275},[14233],{"type":48,"value":12976},{"type":42,"tag":1825,"props":14235,"children":14236},{"style":8275},[14237],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14239,"children":14240},{"style":14133},[14241],{"type":48,"value":2697},{"type":42,"tag":1825,"props":14243,"children":14244},{"style":8275},[14245],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14247,"children":14248},{"style":8275},[14249],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14251,"children":14252},{"style":8275},[14253],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14255,"children":14256},{"style":8380},[14257],{"type":48,"value":14258},"Partially correct but missing information",{"type":42,"tag":1825,"props":14260,"children":14261},{"style":8275},[14262],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14264,"children":14265},{"style":8275},[14266],{"type":48,"value":13010},{"type":42,"tag":1825,"props":14268,"children":14269},{"class":1827,"line":1936},[14270,14274,14278,14282,14286,14290,14294,14299,14303,14307,14311,14315,14319,14323,14327,14332,14336],{"type":42,"tag":1825,"props":14271,"children":14272},{"style":8275},[14273],{"type":48,"value":14126},{"type":42,"tag":1825,"props":14275,"children":14276},{"style":8275},[14277],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14279,"children":14280},{"style":14133},[14281],{"type":48,"value":2705},{"type":42,"tag":1825,"props":14283,"children":14284},{"style":8275},[14285],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14287,"children":14288},{"style":8275},[14289],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14291,"children":14292},{"style":8275},[14293],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14295,"children":14296},{"style":8380},[14297],{"type":48,"value":14298},"incorrect",{"type":42,"tag":1825,"props":14300,"children":14301},{"style":8275},[14302],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14304,"children":14305},{"style":8275},[14306],{"type":48,"value":12976},{"type":42,"tag":1825,"props":14308,"children":14309},{"style":8275},[14310],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14312,"children":14313},{"style":14133},[14314],{"type":48,"value":2697},{"type":42,"tag":1825,"props":14316,"children":14317},{"style":8275},[14318],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14320,"children":14321},{"style":8275},[14322],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14324,"children":14325},{"style":8275},[14326],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14328,"children":14329},{"style":8380},[14330],{"type":48,"value":14331},"The response is wrong or irrelevant",{"type":42,"tag":1825,"props":14333,"children":14334},{"style":8275},[14335],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14337,"children":14338},{"style":8275},[14339],{"type":48,"value":2670},{"type":42,"tag":1825,"props":14341,"children":14342},{"class":1827,"line":1945},[14343],{"type":42,"tag":1825,"props":14344,"children":14345},{"style":8275},[14346],{"type":48,"value":14347},"          ]\n",{"type":42,"tag":1825,"props":14349,"children":14350},{"class":1827,"line":7604},[14351],{"type":42,"tag":1825,"props":14352,"children":14353},{"style":8275},[14354],{"type":48,"value":14355},"        },\n",{"type":42,"tag":1825,"props":14357,"children":14358},{"class":1827,"line":7613},[14359,14363,14367,14371,14375,14379,14383,14387,14391,14395,14399,14403,14407,14411,14415,14419,14423,14427,14431,14436,14440],{"type":42,"tag":1825,"props":14360,"children":14361},{"style":8275},[14362],{"type":48,"value":9277},{"type":42,"tag":1825,"props":14364,"children":14365},{"style":12926},[14366],{"type":48,"value":13022},{"type":42,"tag":1825,"props":14368,"children":14369},{"style":8275},[14370],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14372,"children":14373},{"style":8275},[14374],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14376,"children":14377},{"style":8275},[14378],{"type":48,"value":12941},{"type":42,"tag":1825,"props":14380,"children":14381},{"style":8275},[14382],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14384,"children":14385},{"style":12948},[14386],{"type":48,"value":8085},{"type":42,"tag":1825,"props":14388,"children":14389},{"style":8275},[14390],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14392,"children":14393},{"style":8275},[14394],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14396,"children":14397},{"style":8275},[14398],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14400,"children":14401},{"style":8380},[14402],{"type":48,"value":13059},{"type":42,"tag":1825,"props":14404,"children":14405},{"style":8275},[14406],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14408,"children":14409},{"style":8275},[14410],{"type":48,"value":12976},{"type":42,"tag":1825,"props":14412,"children":14413},{"style":8275},[14414],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14416,"children":14417},{"style":12948},[14418],{"type":48,"value":2697},{"type":42,"tag":1825,"props":14420,"children":14421},{"style":8275},[14422],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14424,"children":14425},{"style":8275},[14426],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14428,"children":14429},{"style":8275},[14430],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14432,"children":14433},{"style":8380},[14434],{"type":48,"value":14435},"Explanation for the category chosen",{"type":42,"tag":1825,"props":14437,"children":14438},{"style":8275},[14439],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14441,"children":14442},{"style":8275},[14443],{"type":48,"value":2670},{"type":42,"tag":1825,"props":14445,"children":14446},{"class":1827,"line":7622},[14447],{"type":42,"tag":1825,"props":14448,"children":14449},{"style":8275},[14450],{"type":48,"value":9392},{"type":42,"tag":1825,"props":14452,"children":14453},{"class":1827,"line":7631},[14454,14458,14462,14466,14470,14474,14478,14482,14486,14490,14494,14498,14502],{"type":42,"tag":1825,"props":14455,"children":14456},{"style":8275},[14457],{"type":48,"value":9070},{"type":42,"tag":1825,"props":14459,"children":14460},{"style":9280},[14461],{"type":48,"value":12695},{"type":42,"tag":1825,"props":14463,"children":14464},{"style":8275},[14465],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14467,"children":14468},{"style":8275},[14469],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14471,"children":14472},{"style":8275},[14473],{"type":48,"value":9333},{"type":42,"tag":1825,"props":14475,"children":14476},{"style":8275},[14477],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14479,"children":14480},{"style":8380},[14481],{"type":48,"value":12669},{"type":42,"tag":1825,"props":14483,"children":14484},{"style":8275},[14485],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14487,"children":14488},{"style":8275},[14489],{"type":48,"value":12976},{"type":42,"tag":1825,"props":14491,"children":14492},{"style":8275},[14493],{"type":48,"value":8764},{"type":42,"tag":1825,"props":14495,"children":14496},{"style":8380},[14497],{"type":48,"value":13022},{"type":42,"tag":1825,"props":14499,"children":14500},{"style":8275},[14501],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14503,"children":14504},{"style":8275},[14505],{"type":48,"value":9351},{"type":42,"tag":1825,"props":14507,"children":14508},{"class":1827,"line":29},[14509,14513,14517,14521,14525],{"type":42,"tag":1825,"props":14510,"children":14511},{"style":8275},[14512],{"type":48,"value":9070},{"type":42,"tag":1825,"props":14514,"children":14515},{"style":9280},[14516],{"type":48,"value":13174},{"type":42,"tag":1825,"props":14518,"children":14519},{"style":8275},[14520],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14522,"children":14523},{"style":8275},[14524],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14526,"children":14527},{"style":8275},[14528],{"type":48,"value":13187},{"type":42,"tag":1825,"props":14530,"children":14531},{"class":1827,"line":7648},[14532],{"type":42,"tag":1825,"props":14533,"children":14534},{"style":8275},[14535],{"type":48,"value":9767},{"type":42,"tag":1825,"props":14537,"children":14538},{"class":1827,"line":7657},[14539],{"type":42,"tag":1825,"props":14540,"children":14541},{"style":8275},[14542],{"type":48,"value":9030},{"type":42,"tag":1825,"props":14544,"children":14545},{"class":1827,"line":7666},[14546,14550,14554,14558,14562,14566,14570,14575,14579,14583,14587,14591,14595,14599],{"type":42,"tag":1825,"props":14547,"children":14548},{"style":8275},[14549],{"type":48,"value":8744},{"type":42,"tag":1825,"props":14551,"children":14552},{"style":8747},[14553],{"type":48,"value":13213},{"type":42,"tag":1825,"props":14555,"children":14556},{"style":8275},[14557],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14559,"children":14560},{"style":8275},[14561],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14563,"children":14564},{"style":8275},[14565],{"type":48,"value":12941},{"type":42,"tag":1825,"props":14567,"children":14568},{"style":8275},[14569],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14571,"children":14572},{"style":8281},[14573],{"type":48,"value":14574},"pass_values",{"type":42,"tag":1825,"props":14576,"children":14577},{"style":8275},[14578],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14580,"children":14581},{"style":8275},[14582],{"type":48,"value":4920},{"type":42,"tag":1825,"props":14584,"children":14585},{"style":8275},[14586],{"type":48,"value":9333},{"type":42,"tag":1825,"props":14588,"children":14589},{"style":8275},[14590],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14592,"children":14593},{"style":8380},[14594],{"type":48,"value":14152},{"type":42,"tag":1825,"props":14596,"children":14597},{"style":8275},[14598],{"type":48,"value":8755},{"type":42,"tag":1825,"props":14600,"children":14601},{"style":8275},[14602],{"type":48,"value":14603},"]}\n",{"type":42,"tag":1825,"props":14605,"children":14606},{"class":1827,"line":7674},[14607],{"type":42,"tag":1825,"props":14608,"children":14609},{"style":8275},[14610],{"type":48,"value":2670},{"type":42,"tag":51,"props":14612,"children":14613},{},[14614,14616,14622,14624,14629,14631,14636,14638,14643],{"type":48,"value":14615},"Note: categorical uses ",{"type":42,"tag":73,"props":14617,"children":14619},{"className":14618},[],[14620],{"type":48,"value":14621},"\"type\": \"string\"",{"type":48,"value":14623}," alongside ",{"type":42,"tag":73,"props":14625,"children":14627},{"className":14626},[],[14628],{"type":48,"value":2712},{"type":48,"value":14630}," (each ",{"type":42,"tag":73,"props":14632,"children":14634},{"className":14633},[],[14635],{"type":48,"value":2705},{"type":48,"value":14637}," is a string value), unlike the offline SDK which uses bare ",{"type":42,"tag":73,"props":14639,"children":14641},{"className":14640},[],[14642],{"type":48,"value":2712},{"type":48,"value":14644}," at the property root.",{"type":42,"tag":51,"props":14646,"children":14647},{},[14648,14653,14655,14660],{"type":42,"tag":55,"props":14649,"children":14650},{},[14651],{"type":48,"value":14652},"Custom \u002F multi-dimensional",{"type":48,"value":14654},": not directly supported via the fixed-name schema. Implement as a score or categorical evaluator where possible, or split into multiple evaluators. The ",{"type":42,"tag":73,"props":14656,"children":14658},{"className":14657},[],[14659],{"type":48,"value":8066},{"type":48,"value":14661}," must be one of the three fixed values above.",{"type":42,"tag":51,"props":14663,"children":14664},{},[14665,14670,14672,14677,14679,14684,14685,14691,14693,14699],{"type":42,"tag":55,"props":14666,"children":14667},{},[14668],{"type":48,"value":14669},"Filter scoping",{"type":48,"value":14671},": when the proposal targets a specific span kind (e.g. an LLM sub-span), translate it into an EVP ",{"type":42,"tag":73,"props":14673,"children":14675},{"className":14674},[],[14676],{"type":48,"value":5065},{"type":48,"value":14678}," query — e.g. ",{"type":42,"tag":73,"props":14680,"children":14682},{"className":14681},[],[14683],{"type":48,"value":5822},{"type":48,"value":251},{"type":42,"tag":73,"props":14686,"children":14688},{"className":14687},[],[14689],{"type":48,"value":14690},"service:checkout-agent",{"type":48,"value":14692},", or a more specific tag. Combine with ",{"type":42,"tag":73,"props":14694,"children":14696},{"className":14695},[],[14697],{"type":48,"value":14698},"root_spans_only:true",{"type":48,"value":14700}," only when the target is the trace root.",{"type":42,"tag":51,"props":14702,"children":14703},{},[14704],{"type":42,"tag":55,"props":14705,"children":14706},{},[14707,14709,14714],{"type":48,"value":14708},"For ",{"type":42,"tag":73,"props":14710,"children":14712},{"className":14711},[],[14713],{"type":48,"value":4895},{"type":48,"value":4920},{"type":42,"tag":194,"props":14716,"children":14717},{},[14718,14743,14783],{"type":42,"tag":67,"props":14719,"children":14720},{},[14721,14723,14727,14729,14734,14736,14741],{"type":48,"value":14722},"The evaluator triggers once per ",{"type":42,"tag":55,"props":14724,"children":14725},{},[14726],{"type":48,"value":6660},{"type":48,"value":14728}," trace, after a ",{"type":42,"tag":55,"props":14730,"children":14731},{},[14732],{"type":48,"value":14733},"3-minute inactivity window",{"type":48,"value":14735},". Late-arriving spans (>3 min after the prior span on the same trace) are ",{"type":42,"tag":55,"props":14737,"children":14738},{},[14739],{"type":48,"value":14740},"excluded",{"type":48,"value":14742}," from the evaluation. Surface this in the proposal so the user knows about both the latency and the potential miss for sparse-activity agents (long-running agents whose steps are sparser than 3 minutes apart).",{"type":42,"tag":67,"props":14744,"children":14745},{},[14746,14747,14752,14754,14759,14761,14766,14768,14774,14776,14781],{"type":48,"value":2749},{"type":42,"tag":73,"props":14748,"children":14750},{"className":14749},[],[14751],{"type":48,"value":5065},{"type":48,"value":14753}," query must match the trace's ",{"type":42,"tag":55,"props":14755,"children":14756},{},[14757],{"type":48,"value":14758},"root span only",{"type":48,"value":14760}," — always include ",{"type":42,"tag":73,"props":14762,"children":14764},{"className":14763},[],[14765],{"type":48,"value":12258},{"type":48,"value":14767}," (or ",{"type":42,"tag":73,"props":14769,"children":14771},{"className":14770},[],[14772],{"type":48,"value":14773},"root_spans_only: true",{"type":48,"value":14775},") to avoid double-firing across descendants. Combine with ",{"type":42,"tag":73,"props":14777,"children":14779},{"className":14778},[],[14780],{"type":48,"value":12400},{"type":48,"value":14782}," (or whatever kind the app uses for root spans, observed in Phase 1) for narrowing.",{"type":42,"tag":67,"props":14784,"children":14785},{},[14786,14788,14793,14794,14802,14804,14809],{"type":48,"value":14787},"Sampling at trace scope is heavier than at span scope (one trace = many spans on the judge's side). Default ",{"type":42,"tag":73,"props":14789,"children":14791},{"className":14790},[],[14792],{"type":48,"value":5057},{"type":48,"value":2714},{"type":42,"tag":55,"props":14795,"children":14796},{},[14797],{"type":42,"tag":73,"props":14798,"children":14800},{"className":14799},[],[14801],{"type":48,"value":10746},{"type":48,"value":14803}," for trace-scope evaluators (instead of the span default ",{"type":42,"tag":73,"props":14805,"children":14807},{"className":14806},[],[14808],{"type":48,"value":5994},{"type":48,"value":14810},"); the user can raise it after a manual review pass.",{"type":42,"tag":51,"props":14812,"children":14813},{},[14814],{"type":42,"tag":55,"props":14815,"children":14816},{},[14817,14818,14823],{"type":48,"value":14708},{"type":42,"tag":73,"props":14819,"children":14821},{"className":14820},[],[14822],{"type":48,"value":4903},{"type":48,"value":4920},{"type":42,"tag":194,"props":14825,"children":14826},{},[14827,14845,14873,14911],{"type":42,"tag":67,"props":14828,"children":14829},{},[14830,14832,14837,14839,14844],{"type":48,"value":14831},"Only publish session-scope evaluators when ",{"type":42,"tag":73,"props":14833,"children":14835},{"className":14834},[],[14836],{"type":48,"value":4276},{"type":48,"value":14838}," (Phase 1 probe). Otherwise the evaluator never fires — no span carries a ",{"type":42,"tag":73,"props":14840,"children":14842},{"className":14841},[],[14843],{"type":48,"value":545},{"type":48,"value":332},{"type":42,"tag":67,"props":14846,"children":14847},{},[14848,14849,14853,14855,14860,14861,14865,14867,14871],{"type":48,"value":14722},{"type":42,"tag":55,"props":14850,"children":14851},{},[14852],{"type":48,"value":6667},{"type":48,"value":14854},". A ",{"type":42,"tag":73,"props":14856,"children":14858},{"className":14857},[],[14859],{"type":48,"value":545},{"type":48,"value":6676},{"type":42,"tag":55,"props":14862,"children":14863},{},[14864],{"type":48,"value":6681},{"type":48,"value":14866}," (vs. 3 minutes for a trace); spans arriving > 30 min after the prior span are ",{"type":42,"tag":55,"props":14868,"children":14869},{},[14870],{"type":48,"value":14740},{"type":48,"value":14872},". Surface both the latency and the exclusion — long-running or sparse sessions may be split or truncated, and very large sessions are capped at 10,000 spans.",{"type":42,"tag":67,"props":14874,"children":14875},{},[14876,14877,14882,14884,14889,14891,14896,14898,14902,14904,14909],{"type":48,"value":2749},{"type":42,"tag":73,"props":14878,"children":14880},{"className":14879},[],[14881],{"type":48,"value":5065},{"type":48,"value":14883}," is matched against the session's ",{"type":42,"tag":55,"props":14885,"children":14886},{},[14887],{"type":48,"value":14888},"root span",{"type":48,"value":14890}," (the same prefilter trace scope uses), so scope it with root-span attributes — e.g. ",{"type":42,"tag":73,"props":14892,"children":14894},{"className":14893},[],[14895],{"type":48,"value":12400},{"type":48,"value":14897}," (or the app's root span kind, observed in Phase 1). You do ",{"type":42,"tag":55,"props":14899,"children":14900},{},[14901],{"type":48,"value":271},{"type":48,"value":14903}," need ",{"type":42,"tag":73,"props":14905,"children":14907},{"className":14906},[],[14908],{"type":48,"value":12258},{"type":48,"value":14910},": the session root is already a root span.",{"type":42,"tag":67,"props":14912,"children":14913},{},[14914,14916,14921,14922,14930],{"type":48,"value":14915},"Sampling at session scope is the heaviest of all (one session = many traces = many spans on the judge's side). Default ",{"type":42,"tag":73,"props":14917,"children":14919},{"className":14918},[],[14920],{"type":48,"value":5057},{"type":48,"value":2714},{"type":42,"tag":55,"props":14923,"children":14924},{},[14925],{"type":42,"tag":73,"props":14926,"children":14928},{"className":14927},[],[14929],{"type":48,"value":10746},{"type":48,"value":14931}," (matching trace) and flag it as the heaviest scope; the user can lower it further after a manual review pass.",{"type":42,"tag":3896,"props":14933,"children":14935},{"id":14934},"always-publish-as-draft-enabled-false",[14936,14938,14943],{"type":48,"value":14937},"Always publish as draft (",{"type":42,"tag":73,"props":14939,"children":14941},{"className":14940},[],[14942],{"type":48,"value":503},{"type":48,"value":631},{"type":42,"tag":51,"props":14945,"children":14946},{},[14947,14957,14959,14964],{"type":42,"tag":55,"props":14948,"children":14949},{},[14950,14952],{"type":48,"value":14951},"Always create \u002F update evaluators with ",{"type":42,"tag":73,"props":14953,"children":14955},{"className":14954},[],[14956],{"type":48,"value":503},{"type":48,"value":14958}," — regardless of whether ",{"type":42,"tag":73,"props":14960,"children":14962},{"className":14961},[],[14963],{"type":48,"value":6027},{"type":48,"value":14965}," was auto-detected from existing evaluators. The UI is the source of truth for activation; the skill should never auto-enable evaluators on the user's behalf. The user reviews each draft in the UI, confirms the integration account is correct (the auto-detected ID may belong to a different judge LLM than the one they want for this app), and flips the toggle when they're satisfied.",{"type":42,"tag":51,"props":14967,"children":14968},{},[14969,14971,14976],{"type":48,"value":14970},"This makes the workflow safe by default: a wrong ",{"type":42,"tag":73,"props":14972,"children":14974},{"className":14973},[],[14975],{"type":48,"value":6027},{"type":48,"value":14977},", a mistuned prompt, or an over-broad filter never goes live without a human pass. Auto-detection of the account ID still helps because the draft renders with the right account pre-selected — review is faster.",{"type":42,"tag":3896,"props":14979,"children":14981},{"id":14980},"integration_account_id-resolution",[14982],{"type":48,"value":14983},"integration_account_id resolution",{"type":42,"tag":51,"props":14985,"children":14986},{},[14987,14988,14993,14995,15000],{"type":48,"value":2749},{"type":42,"tag":73,"props":14989,"children":14991},{"className":14990},[],[14992],{"type":48,"value":6027},{"type":48,"value":14994}," is an opaque UUID that the UI matches against the org's integration accounts list to populate the account section dropdown. Users typically don't know this value, so ",{"type":42,"tag":55,"props":14996,"children":14997},{},[14998],{"type":48,"value":14999},"never ask the user to supply a raw UUID",{"type":48,"value":332},{"type":42,"tag":51,"props":15002,"children":15003},{},[15004],{"type":42,"tag":55,"props":15005,"children":15006},{},[15007],{"type":48,"value":15008},"Resolution order:",{"type":42,"tag":63,"props":15010,"children":15011},{},[15012,15037],{"type":42,"tag":67,"props":15013,"children":15014},{},[15015,15020,15022,15027,15029,15035],{"type":42,"tag":55,"props":15016,"children":15017},{},[15018],{"type":48,"value":15019},"Inherit from existing evaluators",{"type":48,"value":15021}," — in Phase 0 you called ",{"type":42,"tag":73,"props":15023,"children":15025},{"className":15024},[],[15026],{"type":48,"value":1285},{"type":48,"value":15028}," for each existing custom evaluator. Check the ",{"type":42,"tag":73,"props":15030,"children":15032},{"className":15031},[],[15033],{"type":48,"value":15034},"llm_provider.integration_account_id",{"type":48,"value":15036}," field on those responses. If any of them have a value, use that same ID on the published drafts. If multiple different IDs appear across existing evaluators, pick the most common one and note which you chose so the user can correct it during the UI review pass.",{"type":42,"tag":67,"props":15038,"children":15039},{},[15040,15045,15047,15052],{"type":42,"tag":55,"props":15041,"children":15042},{},[15043],{"type":48,"value":15044},"Omit if no existing evaluator has one",{"type":48,"value":15046}," — if no custom evaluator in the ml_app has an ",{"type":42,"tag":73,"props":15048,"children":15050},{"className":15049},[],[15051],{"type":48,"value":6027},{"type":48,"value":15053},", omit the field from the publish payload. The draft will render without an account pre-selected; the user picks one during the UI review pass before activating.",{"type":42,"tag":51,"props":15055,"children":15056},{},[15057,15059,15064],{"type":48,"value":15058},"Either way, the evaluator is published with ",{"type":42,"tag":73,"props":15060,"children":15062},{"className":15061},[],[15063],{"type":48,"value":503},{"type":48,"value":15065},". The user is the gate — see \"Always publish as draft\" above.",{"type":42,"tag":3896,"props":15067,"children":15069},{"id":15068},"publish-single-message-parallelize",[15070],{"type":48,"value":15071},"Publish (single message — parallelize)",{"type":42,"tag":51,"props":15073,"children":15074},{},[15075,15077,15082,15084,15088,15090,15095,15097,15103],{"type":48,"value":15076},"Issue all ",{"type":42,"tag":73,"props":15078,"children":15080},{"className":15079},[],[15081],{"type":48,"value":488},{"type":48,"value":15083}," calls in a ",{"type":42,"tag":55,"props":15085,"children":15086},{},[15087],{"type":48,"value":1721},{"type":48,"value":15089}," (one per evaluator). Set ",{"type":42,"tag":73,"props":15091,"children":15093},{"className":15092},[],[15094],{"type":48,"value":385},{"type":48,"value":15096}," to a short English description like ",{"type":42,"tag":73,"props":15098,"children":15100},{"className":15099},[],[15101],{"type":48,"value":15102},"\"skill:agent-observability-eval-bootstrap — Bootstrap evaluator suite for ml_app=\u003Cml_app> from production trace analysis.\"",{"type":48,"value":332},{"type":42,"tag":51,"props":15105,"children":15106},{},[15107],{"type":48,"value":15108},"If any call fails, capture the error and continue with the remaining evaluators — never silently abort the batch. Report failures explicitly in the summary.",{"type":42,"tag":3896,"props":15110,"children":15112},{"id":15111},"summary",[15113],{"type":48,"value":15114},"Summary",{"type":42,"tag":730,"props":15116,"children":15119},{"className":15117,"code":15118,"language":48},[733],"## Published Evaluators (drafts — pending UI review)\n\nWrote {N} online evaluators to ml_app `{ml_app}`. **All published as drafts (`enabled: false`)** — review and activate them in the UI before they start scoring spans.\n\n| # | Name | Action | Provider\u002FModel | Sampling | Scope | Account auto-detected | Status |\n|---|------|--------|----------------|----------|-------|-----------------------|--------|\n| 1 | task_completion | created (draft) | openai\u002Fgpt-5.4-mini | 10% | span | yes | ok |\n| 2 | response_groundedness | overwrote (draft) | openai\u002Fgpt-5.4-mini | 10% | span | yes | ok |\n| 3 | scope_adherence | renamed (`_v2`) (draft) | openai\u002Fgpt-5.4-mini | 10% | span | no — pick in UI | ok |\n| 4 | citation_format | failed | openai\u002Fgpt-5.4-mini | 10% | span | — | error |\n| 5 | session_goal_completion | created (draft) | openai\u002Fgpt-5.4-mini | 5% | session | yes | ok |\n\n{If any failed:}\n**Errors**:\n- `{name}`: {error message}\n\n{If any code-based proposals were dropped:}\n**Not published** (code-based, not supported by online evaluator API):\n- `{name}` ({type}) — ask me to emit it as offline SDK code, or run `\u002Feval-bootstrap {ml_app} --sdk-code`.\n\n### Next Steps — review and activate in the UI\n\nThe drafts are intentionally not running yet. Walk through each one in the Datadog UI before flipping the enable toggle:\n\n1. **Open the drafts**: Datadog → Agent Observability → Evaluations → filter by ml_app `{ml_app}` (the new drafts appear with status `Disabled`).\n2. **For each draft**:\n   - **Verify the integration account** in the Provider section. If the column above shows `auto-detected: yes`, confirm it's the correct account for the judge LLM you want this evaluator to call through. If `no`, pick an account from the dropdown.\n   - **Skim the prompt template** and the structured-output schema — make sure the span \u002F trace \u002F session scope, filter, and sampling match what you actually want to measure.\n   - **Click into a sample span\u002Ftrace** and use the test pane to dry-run the prompt against real data. Confirm the result matches your expectation.\n3. **Enable**: once each draft passes review, toggle it to enabled. Datadog starts scoring incoming spans immediately.\n4. **Wait for first scores**: with `sampling_percentage=10` (span scope) or `5` (trace \u002F session scope), expect first results within minutes for high-traffic apps — except **session scope**, where the first result appears only after the session's 30-minute inactivity window closes.\n5. **Tune sampling\u002Ffilter**: if results are noisy or volume is too high, reduce `sampling_percentage` or tighten the `filter` from the UI. Re-running `\u002Feval-bootstrap {ml_app}` will round-trip the existing config before overwriting — your manual tweaks survive across reruns.\n6. **Prefer offline code?** These evaluators run online in Datadog. If you'd rather iterate on the same suite as offline SDK code (`sdk_code`) or a portable JSON spec (`data_only`), just ask — I'll regenerate the **confirmed** suite without re-exploring.\n",[15120],{"type":42,"tag":73,"props":15121,"children":15122},{"__ignoreMap":738},[15123],{"type":48,"value":15118},{"type":42,"tag":3896,"props":15125,"children":15127},{"id":15126},"notebook-export-after-summary-2",[15128],{"type":48,"value":7929},{"type":42,"tag":51,"props":15130,"children":15131},{},[15132,15133,15138,15140,15145,15147,15152,15153,15158],{"type":48,"value":10272},{"type":42,"tag":73,"props":15134,"children":15136},{"className":15135},[],[15137],{"type":48,"value":3865},{"type":48,"value":15139}," was detected, or create a new standalone notebook. The notebook cell should list the published evaluators with their UI links and the ",{"type":42,"tag":73,"props":15141,"children":15143},{"className":15142},[],[15144],{"type":48,"value":802},{"type":48,"value":15146}," they target. In pup mode, use ",{"type":42,"tag":73,"props":15148,"children":15150},{"className":15149},[],[15151],{"type":48,"value":10293},{"type":48,"value":591},{"type":42,"tag":73,"props":15154,"children":15156},{"className":15155},[],[15157],{"type":48,"value":10300},{"type":48,"value":10302},{"type":42,"tag":1773,"props":15160,"children":15161},{},[],{"type":42,"tag":747,"props":15163,"children":15165},{"id":15164},"phase-3d-emit-dataset-from-production-traces",[15166],{"type":48,"value":15167},"Phase 3D: Emit Dataset from Production Traces",{"type":42,"tag":51,"props":15169,"children":15170},{},[15171,15175,15177,15182,15184,15189,15190,15195,15196,15201],{"type":42,"tag":55,"props":15172,"children":15173},{},[15174],{"type":48,"value":3892},{"type":48,"value":15176},": Sample production traces for ",{"type":42,"tag":73,"props":15178,"children":15180},{"className":15179},[],[15181],{"type":48,"value":802},{"type":48,"value":15183},", extract input \u002F output pairs, and write a ",{"type":42,"tag":73,"props":15185,"children":15187},{"className":15186},[],[15188],{"type":48,"value":665},{"type":48,"value":667},{"type":42,"tag":73,"props":15191,"children":15193},{"className":15192},[],[15194],{"type":48,"value":673},{"type":48,"value":675},{"type":42,"tag":55,"props":15197,"children":15198},{},[15199],{"type":48,"value":15200},"This mode does not propose or generate evaluators",{"type":48,"value":15202}," — Phases 1 and 2 are bypassed. The dataset is the sole artifact.",{"type":42,"tag":51,"props":15204,"children":15205},{},[15206,15211,15213,15218],{"type":42,"tag":55,"props":15207,"children":15208},{},[15209],{"type":48,"value":15210},"When this mode runs",{"type":48,"value":15212},": triggered by ",{"type":42,"tag":73,"props":15214,"children":15216},{"className":15215},[],[15217],{"type":48,"value":656},{"type":48,"value":15219},". Skip Phase 0 step 4 (eval coverage map) — it's unrelated to dataset extraction.",{"type":42,"tag":3896,"props":15221,"children":15223},{"id":15222},"sampling",[15224],{"type":48,"value":15225},"Sampling",{"type":42,"tag":63,"props":15227,"children":15228},{},[15229,15248],{"type":42,"tag":67,"props":15230,"children":15231},{},[15232,15233,15239,15241,15246],{"type":48,"value":7374},{"type":42,"tag":73,"props":15234,"children":15236},{"className":15235},[],[15237],{"type":48,"value":15238},"search_llmobs_spans(query=\"@ml_app:\\\"\u003Cml_app>\\\" @status:ok\", root_spans_only=true, limit=\u003Ctrace-limit>, from=\u003Ctimeframe>)",{"type":48,"value":15240},". Filter to ",{"type":42,"tag":73,"props":15242,"children":15244},{"className":15243},[],[15245],{"type":48,"value":3927},{"type":48,"value":15247}," so the resulting dataset records reflect intended app behavior (errored traces have no usable output baseline).",{"type":42,"tag":67,"props":15249,"children":15250},{},[15251,15253,15258,15260,15265,15267,15272,15274,15279,15280,15285],{"type":48,"value":15252},"For each sampled root span, call ",{"type":42,"tag":73,"props":15254,"children":15256},{"className":15255},[],[15257],{"type":48,"value":1162},{"type":48,"value":15259}," (group by ",{"type":42,"tag":73,"props":15261,"children":15263},{"className":15262},[],[15264],{"type":48,"value":4005},{"type":48,"value":15266},") and ",{"type":42,"tag":73,"props":15268,"children":15270},{"className":15269},[],[15271],{"type":48,"value":1187},{"type":48,"value":15273}," to read the root span's ",{"type":42,"tag":73,"props":15275,"children":15277},{"className":15276},[],[15278],{"type":48,"value":1508},{"type":48,"value":919},{"type":42,"tag":73,"props":15281,"children":15283},{"className":15282},[],[15284],{"type":48,"value":1515},{"type":48,"value":15286}," fields. Issue all calls for a page in a single message (parallelize).",{"type":42,"tag":3896,"props":15288,"children":15290},{"id":15289},"record-extraction",[15291],{"type":48,"value":15292},"Record extraction",{"type":42,"tag":51,"props":15294,"children":15295},{},[15296,15298,15304],{"type":48,"value":15297},"For each sampled root span, build one ",{"type":42,"tag":73,"props":15299,"children":15301},{"className":15300},[],[15302],{"type":48,"value":15303},"DatasetRecordRaw",{"type":48,"value":15305}," entry:",{"type":42,"tag":754,"props":15307,"children":15308},{},[15309,15329],{"type":42,"tag":758,"props":15310,"children":15311},{},[15312],{"type":42,"tag":762,"props":15313,"children":15314},{},[15315,15319,15324],{"type":42,"tag":766,"props":15316,"children":15317},{},[15318],{"type":48,"value":1400},{"type":42,"tag":766,"props":15320,"children":15321},{},[15322],{"type":48,"value":15323},"Source",{"type":42,"tag":766,"props":15325,"children":15326},{},[15327],{"type":48,"value":15328},"Notes",{"type":42,"tag":787,"props":15330,"children":15331},{},[15332,15401,15442,15467],{"type":42,"tag":762,"props":15333,"children":15334},{},[15335,15343,15353],{"type":42,"tag":794,"props":15336,"children":15337},{},[15338],{"type":42,"tag":73,"props":15339,"children":15341},{"className":15340},[],[15342],{"type":48,"value":2072},{"type":42,"tag":794,"props":15344,"children":15345},{},[15346,15348],{"type":48,"value":15347},"Root span's ",{"type":42,"tag":73,"props":15349,"children":15351},{"className":15350},[],[15352],{"type":48,"value":6475},{"type":42,"tag":794,"props":15354,"children":15355},{},[15356,15357,15362,15364,15370,15371,15377,15379,15385,15387,15392,15394,15400],{"type":48,"value":1069},{"type":42,"tag":73,"props":15358,"children":15360},{"className":15359},[],[15361],{"type":48,"value":11172},{"type":48,"value":15363}," is a plain string, wrap as ",{"type":42,"tag":73,"props":15365,"children":15367},{"className":15366},[],[15368],{"type":48,"value":15369},"{\"input\": \"\u003Cvalue>\"}",{"type":48,"value":6637},{"type":42,"tag":73,"props":15372,"children":15374},{"className":15373},[],[15375],{"type":48,"value":15376},"meta.input.messages",{"type":48,"value":15378}," exists, use ",{"type":42,"tag":73,"props":15380,"children":15382},{"className":15381},[],[15383],{"type":48,"value":15384},"{\"messages\": [...]}",{"type":48,"value":15386}," (preserving the role\u002Fcontent shape). For RAG roots that also have ",{"type":42,"tag":73,"props":15388,"children":15390},{"className":15389},[],[15391],{"type":48,"value":1536},{"type":48,"value":15393},", include ",{"type":42,"tag":73,"props":15395,"children":15397},{"className":15396},[],[15398],{"type":48,"value":15399},"{\"documents\": [...]}",{"type":48,"value":332},{"type":42,"tag":762,"props":15402,"children":15403},{},[15404,15412,15430],{"type":42,"tag":794,"props":15405,"children":15406},{},[15407],{"type":42,"tag":73,"props":15408,"children":15410},{"className":15409},[],[15411],{"type":48,"value":4771},{"type":42,"tag":794,"props":15413,"children":15414},{},[15415,15416,15421,15422,15428],{"type":48,"value":15347},{"type":42,"tag":73,"props":15417,"children":15419},{"className":15418},[],[15420],{"type":48,"value":11200},{"type":48,"value":14767},{"type":42,"tag":73,"props":15423,"children":15425},{"className":15424},[],[15426],{"type":48,"value":15427},"meta.output.messages[-1].content",{"type":48,"value":15429}," for LLM-kind roots)",{"type":42,"tag":794,"props":15431,"children":15432},{},[15433,15435,15440],{"type":48,"value":15434},"This is the ",{"type":42,"tag":55,"props":15436,"children":15437},{},[15438],{"type":48,"value":15439},"production behavior baseline",{"type":48,"value":15441},", not ground truth. Document this clearly in the summary so the user does not over-trust it.",{"type":42,"tag":762,"props":15443,"children":15444},{},[15445,15453,15462],{"type":42,"tag":794,"props":15446,"children":15447},{},[15448],{"type":42,"tag":73,"props":15449,"children":15451},{"className":15450},[],[15452],{"type":48,"value":1557},{"type":42,"tag":794,"props":15454,"children":15455},{},[15456],{"type":42,"tag":73,"props":15457,"children":15459},{"className":15458},[],[15460],{"type":48,"value":15461},"{\"source\": \"production\", \"trace_id\": \"\u003Cid>\", \"span_id\": \"\u003Cid>\", \"ml_app\": \"\u003Cml_app>\", \"extracted_at\": \"\u003CISO 8601 UTC>\"}",{"type":42,"tag":794,"props":15463,"children":15464},{},[15465],{"type":48,"value":15466},"Trace\u002Fspan IDs let the user click back to the source trace in Datadog.",{"type":42,"tag":762,"props":15468,"children":15469},{},[15470,15478,15487],{"type":42,"tag":794,"props":15471,"children":15472},{},[15473],{"type":42,"tag":73,"props":15474,"children":15476},{"className":15475},[],[15477],{"type":48,"value":1632},{"type":42,"tag":794,"props":15479,"children":15480},{},[15481],{"type":42,"tag":73,"props":15482,"children":15484},{"className":15483},[],[15485],{"type":48,"value":15486},"[\"env:prod\", \"source:traces\", \"ml_app:\u003Cml_app>\"]",{"type":42,"tag":794,"props":15488,"children":15489},{},[15490,15492,15498,15500,15506,15508,15514],{"type":48,"value":15491},"Always ",{"type":42,"tag":73,"props":15493,"children":15495},{"className":15494},[],[15496],{"type":48,"value":15497},"\"key:value\"",{"type":48,"value":15499}," strings — bare strings will fail ",{"type":42,"tag":73,"props":15501,"children":15503},{"className":15502},[],[15504],{"type":48,"value":15505},"Dataset.append()",{"type":48,"value":15507},". See the per-record tag rules in ",{"type":42,"tag":73,"props":15509,"children":15511},{"className":15510},[],[15512],{"type":48,"value":15513},"agent-observability-experiment-py-bootstrap\u002FSKILL.md",{"type":48,"value":15515}," (search \"Per-record tags\").",{"type":42,"tag":51,"props":15517,"children":15518},{},[15519,15524,15526,15532],{"type":42,"tag":55,"props":15520,"children":15521},{},[15522],{"type":48,"value":15523},"PII scrub.",{"type":48,"value":15525}," Before writing, run the same scrub used elsewhere in the agent-observability skill set: replace email \u002F phone \u002F SSN \u002F API-key regex matches in all string values with ",{"type":42,"tag":73,"props":15527,"children":15529},{"className":15528},[],[15530],{"type":48,"value":15531},"\u003CREDACTED:pii-type>",{"type":48,"value":15533},". Surface a one-line warning listing affected record indices in the summary.",{"type":42,"tag":51,"props":15535,"children":15536},{},[15537,15542,15544,15549,15551,15557,15559,15564,15566,15572,15574,15579],{"type":42,"tag":55,"props":15538,"children":15539},{},[15540],{"type":48,"value":15541},"Tag normalization (MANDATORY pre-write step).",{"type":48,"value":15543}," The SDK's ",{"type":42,"tag":73,"props":15545,"children":15547},{"className":15546},[],[15548],{"type":48,"value":15505},{"type":48,"value":15550}," calls ",{"type":42,"tag":73,"props":15552,"children":15554},{"className":15553},[],[15555],{"type":48,"value":15556},"validate_tags_list",{"type":48,"value":15558}," which rejects any tag without a ",{"type":42,"tag":73,"props":15560,"children":15562},{"className":15561},[],[15563],{"type":48,"value":4920},{"type":48,"value":15565}," separator (",{"type":42,"tag":73,"props":15567,"children":15569},{"className":15568},[],[15570],{"type":48,"value":15571},"Tag '\u003Ctag>' is malformed. Tags must be in 'key:value' format.",{"type":48,"value":15573},"). Before writing the JSON, run this normalization on every record's ",{"type":42,"tag":73,"props":15575,"children":15577},{"className":15576},[],[15578],{"type":48,"value":1632},{"type":48,"value":15580}," list:",{"type":42,"tag":63,"props":15582,"children":15583},{},[15584,15602,15625,15674],{"type":42,"tag":67,"props":15585,"children":15586},{},[15587,15592,15594,15600],{"type":42,"tag":55,"props":15588,"children":15589},{},[15590],{"type":48,"value":15591},"Drop",{"type":48,"value":15593}," any non-string, ",{"type":42,"tag":73,"props":15595,"children":15597},{"className":15596},[],[15598],{"type":48,"value":15599},"None",{"type":48,"value":15601},", or empty-string entries.",{"type":42,"tag":67,"props":15603,"children":15604},{},[15605,15616,15617,15623],{"type":42,"tag":55,"props":15606,"children":15607},{},[15608,15610],{"type":48,"value":15609},"Already ",{"type":42,"tag":73,"props":15611,"children":15613},{"className":15612},[],[15614],{"type":48,"value":15615},"key:value",{"type":48,"value":497},{"type":42,"tag":73,"props":15618,"children":15620},{"className":15619},[],[15621],{"type":48,"value":15622},"\":\" in tag",{"type":48,"value":15624}," and both halves non-empty) → keep as-is.",{"type":42,"tag":67,"props":15626,"children":15627},{},[15628,15633,15635,15640,15642,15648,15650,15656,15658,15664,15666,15672],{"type":42,"tag":55,"props":15629,"children":15630},{},[15631],{"type":48,"value":15632},"Bare string",{"type":48,"value":15634}," (no ",{"type":42,"tag":73,"props":15636,"children":15638},{"className":15637},[],[15639],{"type":48,"value":4920},{"type":48,"value":15641},") → wrap as ",{"type":42,"tag":73,"props":15643,"children":15645},{"className":15644},[],[15646],{"type":48,"value":15647},"\"tag:\u003Cvalue>\"",{"type":48,"value":15649}," (so e.g. ",{"type":42,"tag":73,"props":15651,"children":15653},{"className":15652},[],[15654],{"type":48,"value":15655},"\"interactive\"",{"type":48,"value":15657}," becomes ",{"type":42,"tag":73,"props":15659,"children":15661},{"className":15660},[],[15662],{"type":48,"value":15663},"\"tag:interactive\"",{"type":48,"value":15665},"). Never silently drop — the original observation is preserved under the generic ",{"type":42,"tag":73,"props":15667,"children":15669},{"className":15668},[],[15670],{"type":48,"value":15671},"tag:",{"type":48,"value":15673}," key.",{"type":42,"tag":67,"props":15675,"children":15676},{},[15677,15687,15688,15694,15695,15701,15702,15708],{"type":42,"tag":55,"props":15678,"children":15679},{},[15680,15682],{"type":48,"value":15681},"Trailing or leading ",{"type":42,"tag":73,"props":15683,"children":15685},{"className":15684},[],[15686],{"type":48,"value":4920},{"type":48,"value":12002},{"type":42,"tag":73,"props":15689,"children":15691},{"className":15690},[],[15692],{"type":48,"value":15693},"\"foo:\"",{"type":48,"value":251},{"type":42,"tag":73,"props":15696,"children":15698},{"className":15697},[],[15699],{"type":48,"value":15700},"\":bar\"",{"type":48,"value":15641},{"type":42,"tag":73,"props":15703,"children":15705},{"className":15704},[],[15706],{"type":48,"value":15707},"\"tag:\u003Cfull-original-string>\"",{"type":48,"value":15709}," so the malformed value is preserved without crashing the SDK.",{"type":42,"tag":51,"props":15711,"children":15712},{},[15713,15715,15721,15723,15728,15730,15735],{"type":48,"value":15714},"Track the count of normalizations and surface in the summary as a one-line note (",{"type":42,"tag":73,"props":15716,"children":15718},{"className":15717},[],[15719],{"type":48,"value":15720},"Tag normalizations applied: N (bare strings auto-prefixed with 'tag:')",{"type":48,"value":15722},"). Do not write the dataset if any record's ",{"type":42,"tag":73,"props":15724,"children":15726},{"className":15725},[],[15727],{"type":48,"value":1632},{"type":48,"value":15729}," list still contains a non-",{"type":42,"tag":73,"props":15731,"children":15733},{"className":15732},[],[15734],{"type":48,"value":15615},{"type":48,"value":15736}," string after normalization — that's a bug in this step.",{"type":42,"tag":51,"props":15738,"children":15739},{},[15740,15745],{"type":42,"tag":55,"props":15741,"children":15742},{},[15743],{"type":48,"value":15744},"Skip records",{"type":48,"value":15746}," where:",{"type":42,"tag":194,"props":15748,"children":15749},{},[15750,15760],{"type":42,"tag":67,"props":15751,"children":15752},{},[15753,15758],{"type":42,"tag":73,"props":15754,"children":15756},{"className":15755},[],[15757],{"type":48,"value":6482},{"type":48,"value":15759}," is empty or missing (nothing usable as a baseline).",{"type":42,"tag":67,"props":15761,"children":15762},{},[15763],{"type":48,"value":15764},"The root span's I\u002FO is purely structural (e.g. only tool calls, no natural-language output). Record the count of skipped records and surface in the summary.",{"type":42,"tag":3896,"props":15766,"children":15768},{"id":15767},"write-the-file-1",[15769],{"type":48,"value":7901},{"type":42,"tag":63,"props":15771,"children":15772},{},[15773,15792],{"type":42,"tag":67,"props":15774,"children":15775},{},[15776,15778,15783,15785,15790],{"type":48,"value":15777},"Write the JSON array (2-space indented) to ",{"type":42,"tag":73,"props":15779,"children":15781},{"className":15780},[],[15782],{"type":48,"value":1004},{"type":48,"value":15784}," using the Write tool. The path was supplied by the user via ",{"type":42,"tag":73,"props":15786,"children":15788},{"className":15787},[],[15789],{"type":48,"value":656},{"type":48,"value":15791}," — use it verbatim.",{"type":42,"tag":67,"props":15793,"children":15794},{},[15795,15799,15801,15806],{"type":42,"tag":55,"props":15796,"children":15797},{},[15798],{"type":48,"value":2762},{"type":48,"value":15800}," push to Datadog from this mode. Publishing to Datadog is the next step in the onboarding flow (",{"type":42,"tag":73,"props":15802,"children":15804},{"className":15803},[],[15805],{"type":48,"value":673},{"type":48,"value":15807},") and is handled by the caller (the onboarding orchestrator or the user directly).",{"type":42,"tag":3896,"props":15809,"children":15811},{"id":15810},"summary-1",[15812],{"type":48,"value":15114},{"type":42,"tag":51,"props":15814,"children":15815},{},[15816],{"type":48,"value":15817},"After writing, print:",{"type":42,"tag":730,"props":15819,"children":15822},{"className":15820,"code":15821,"language":48},[733],"## Generated Dataset\n\nWrote `\u003Cpath>` — {N} records sampled from `\u003Cml_app>` ({timeframe}).\n\n| Stat | Value |\n|---|---|\n| Records emitted | {N} |\n| Records skipped (no output) | {M} |\n| PII redactions applied | {P} (indices: {list or \"none\"}) |\n| Tag normalizations applied | {T} (bare strings auto-prefixed with `tag:`) |\n\n**Important**: `expected_output` is the **current production behavior baseline**, not ground truth. Use this dataset for regression-style experiments (does my refactor change observed outputs?) before promoting it to a labelled gold set.\n\n### Next Steps\n\n1. **Review**: open `\u003Cpath>` and spot-check a few records. Adjust `expected_output` where the production output was wrong.\n2. **Publish to Datadog**:\n   ```python\n   from ddtrace.llmobs import LLMObs\n   import json, os\n   LLMObs.enable(api_key=os.getenv(\"DD_API_KEY\"), app_key=os.getenv(\"DD_APPLICATION_KEY\"),\n                 site=os.getenv(\"DD_SITE\", \"datadoghq.com\"),\n                 project_name=\"\u003Cyour-project>\", agentless_enabled=True)\n   records = json.load(open(\"\u003Cpath>\"))\n   ds = LLMObs.create_dataset(dataset_name=\"\u003Cyour-name>\", records=records)\n   print(ds)\n",[15823],{"type":42,"tag":73,"props":15824,"children":15825},{"__ignoreMap":738},[15826],{"type":48,"value":15821},{"type":42,"tag":63,"props":15828,"children":15829},{"start":1846},[15830],{"type":42,"tag":67,"props":15831,"children":15832},{},[15833,15838,15840,15846],{"type":42,"tag":55,"props":15834,"children":15835},{},[15836],{"type":48,"value":15837},"Run an experiment",{"type":48,"value":15839}," against the published dataset via ",{"type":42,"tag":73,"props":15841,"children":15843},{"className":15842},[],[15844],{"type":48,"value":15845},"agent-observability-experiment-py-bootstrap --dataset-name \u003Cyour-name>",{"type":48,"value":332},{"type":42,"tag":730,"props":15848,"children":15851},{"className":15849,"code":15850,"language":48},[733],"\nSkip the notebook export prompt in this mode — datasets aren't well-served by a markdown summary in a notebook, and the next step (experiment generation) will produce its own artifacts.\n\n---\n\n## Operating Rules\n\n- **Breadth over precision; let the user curate**: Propose **8–15 evaluators** distributed across **domain-specific** (largest bucket — derived from Phase 1 domain signals), outcome, format, and safety. Users can always remove what doesn't fit their quality bar; they cannot easily add what was not proposed. Anchor every domain-specific proposal in at least one observed trace pattern — don't invent generic domain evaluators without evidence.\n- **Don't overfit**: Write criteria that generalize beyond the specific sampled traces. Use examples as grounding, not as the sole criteria.\n- **Show your work**: Every proposed evaluator cites at least one trace as evidence with a clickable link: `[Trace {first_8}...](https:\u002F\u002Fapp.datadoghq.com\u002Fllm\u002Ftraces?query=trace_id:{full_32_char_id})`.\n- **New file only**: Never modify existing evaluator code or experiment configurations.\n- **Honest about uncertainty**: If fewer than 5 traces support a proposed evaluator, flag it as tentative.\n- **Session scope is gated**: only propose or publish `eval_scope: session` evaluators when the Phase 1 probe confirmed `sessions_present` (spans carry a `session_id`, ideally one that spans multiple traces). If sessions are absent, skip session scope and say so — never publish a session evaluator that can't fire.\n\n---\n\n## Tool Reference\n\nThis appendix applies only in **pup mode**. In MCP mode, use the tool names in the workflow sections directly.\n\n### Spans and traces\n\n| MCP Tool | pup Command |\n|---|---|\n| `search_llmobs_spans(query, ml_app, from, to, limit, cursor, root_spans_only, span_kind, summary)` | `pup llm-obs spans search --query \"@ml_app:A [other_filters]\" [--from F] [--to T] [--limit N] [--cursor C] [--root-spans-only] [--span-kind K] [--summary]` — **always use `--query \"@ml_app:A\"` to filter by ml_app**; the `--ml-app A` flag is unreliable and silently returns spans from other apps. |\n| `get_llmobs_span_details(trace_id, span_ids, from, to)` | `pup llm-obs spans get-details --trace-id T --span-ids S1,S2,...` |\n| `get_llmobs_span_content(trace_id, span_id, field, path)` | `pup llm-obs spans get-content --trace-id T --span-id S --field F [--path P]` |\n| `get_llmobs_trace(trace_id, include_tree)` | `pup llm-obs spans get-trace --trace-id T [--include-tree]` |\n| `get_llmobs_agent_loop(trace_id, span_id)` | `pup llm-obs spans get-agent-loop --trace-id T [--span-id S]` |\n| `find_llmobs_error_spans(trace_id)` | `pup llm-obs spans find-errors --trace-id T` |\n| `expand_llmobs_spans(trace_id, span_ids, max_depth, filter_kind)` | `pup llm-obs spans expand --trace-id T --span-ids S1,S2,... [--max-depth N] [--filter-kind K]` |\n\n### Evaluators\n\n| MCP Tool | pup Command |\n|---|---|\n| `list_llmobs_evals()` | `pup llm-obs evals list` (filter by `ml_app` client-side) |\n| `list_llmobs_evals_by_ml_app(ml_app)` | `pup llm-obs evals list-by-ml-app --ml-app A` |\n| `get_llmobs_evaluator(eval_name)` | `pup llm-obs evals get-evaluator EVAL_NAME` |\n| `get_llmobs_eval_aggregate_stats(eval_name, ml_app, from, to)` | `pup llm-obs evals get-aggregate-stats EVAL_NAME [--ml-app A] [--from F] [--to T]` |\n| `delete_llmobs_evaluator(eval_name)` | `pup llm-obs evals delete EVAL_NAME` |\n| `create_or_update_llmobs_evaluator(...)` | `pup llm-obs evals create-or-update EVAL_NAME --file \u002Ftmp\u002Feval_EVAL_NAME.json` — see flat schema note below |\n\n#### `create_or_update_llmobs_evaluator` in pup mode\n\npup uses a **flat** JSON file (all fields top-level). `get-evaluator` returns a **nested** object. Transform as follows:\n\n1. **Round-trip check**: Call `pup llm-obs evals get-evaluator EVAL_NAME` first. If it exists, start from its config.\n2. **Flatten `llm_provider`**: hoist `integration_provider`, `model_name`, `integration_account_id`, `temperature` to top level, dropping the `llm_provider` key.\n3. **Merge and set `enabled: false`**.\n4. **Write to temp file and call**:\n   ```bash\n   pup llm-obs evals create-or-update EVAL_NAME --file \u002Ftmp\u002Feval_EVAL_NAME.json\n",[15852],{"type":42,"tag":73,"props":15853,"children":15854},{"__ignoreMap":738},[15855],{"type":48,"value":15850},{"type":42,"tag":51,"props":15857,"children":15858},{},[15859,15861,15867],{"type":48,"value":15860},"Use unique temp file names when publishing multiple evaluators in parallel (e.g. ",{"type":42,"tag":73,"props":15862,"children":15864},{"className":15863},[],[15865],{"type":48,"value":15866},"\u002Ftmp\u002Feval_toxicity.json",{"type":48,"value":220},{"type":42,"tag":754,"props":15869,"children":15870},{},[15871,15893],{"type":42,"tag":758,"props":15872,"children":15873},{},[15874],{"type":42,"tag":762,"props":15875,"children":15876},{},[15877,15888],{"type":42,"tag":766,"props":15878,"children":15879},{},[15880,15886],{"type":42,"tag":73,"props":15881,"children":15883},{"className":15882},[],[15884],{"type":48,"value":15885},"get-evaluator",{"type":48,"value":15887}," field",{"type":42,"tag":766,"props":15889,"children":15890},{},[15891],{"type":48,"value":15892},"Flat JSON key",{"type":42,"tag":787,"props":15894,"children":15895},{},[15896,15916,15936,15955,15975],{"type":42,"tag":762,"props":15897,"children":15898},{},[15899,15908],{"type":42,"tag":794,"props":15900,"children":15901},{},[15902],{"type":42,"tag":73,"props":15903,"children":15905},{"className":15904},[],[15906],{"type":48,"value":15907},"llm_provider.integration_provider",{"type":42,"tag":794,"props":15909,"children":15910},{},[15911],{"type":42,"tag":73,"props":15912,"children":15914},{"className":15913},[],[15915],{"type":48,"value":5960},{"type":42,"tag":762,"props":15917,"children":15918},{},[15919,15928],{"type":42,"tag":794,"props":15920,"children":15921},{},[15922],{"type":42,"tag":73,"props":15923,"children":15925},{"className":15924},[],[15926],{"type":48,"value":15927},"llm_provider.model_name",{"type":42,"tag":794,"props":15929,"children":15930},{},[15931],{"type":42,"tag":73,"props":15932,"children":15934},{"className":15933},[],[15935],{"type":48,"value":5974},{"type":42,"tag":762,"props":15937,"children":15938},{},[15939,15947],{"type":42,"tag":794,"props":15940,"children":15941},{},[15942],{"type":42,"tag":73,"props":15943,"children":15945},{"className":15944},[],[15946],{"type":48,"value":15034},{"type":42,"tag":794,"props":15948,"children":15949},{},[15950],{"type":42,"tag":73,"props":15951,"children":15953},{"className":15952},[],[15954],{"type":48,"value":6027},{"type":42,"tag":762,"props":15956,"children":15957},{},[15958,15967],{"type":42,"tag":794,"props":15959,"children":15960},{},[15961],{"type":42,"tag":73,"props":15962,"children":15964},{"className":15963},[],[15965],{"type":48,"value":15966},"llm_provider.temperature",{"type":42,"tag":794,"props":15968,"children":15969},{},[15970],{"type":42,"tag":73,"props":15971,"children":15973},{"className":15972},[],[15974],{"type":48,"value":10449},{"type":42,"tag":762,"props":15976,"children":15977},{},[15978,15983],{"type":42,"tag":794,"props":15979,"children":15980},{},[15981],{"type":48,"value":15982},"All other fields",{"type":42,"tag":794,"props":15984,"children":15985},{},[15986],{"type":48,"value":15987},"Unchanged (already top-level)",{"type":42,"tag":747,"props":15989,"children":15991},{"id":15990},"notebooks",[15992],{"type":48,"value":15993},"Notebooks",{"type":42,"tag":754,"props":15995,"children":15996},{},[15997,16013],{"type":42,"tag":758,"props":15998,"children":15999},{},[16000],{"type":42,"tag":762,"props":16001,"children":16002},{},[16003,16008],{"type":42,"tag":766,"props":16004,"children":16005},{},[16006],{"type":48,"value":16007},"MCP Tool",{"type":42,"tag":766,"props":16009,"children":16010},{},[16011],{"type":48,"value":16012},"pup Command",{"type":42,"tag":787,"props":16014,"children":16015},{},[16016,16045],{"type":42,"tag":762,"props":16017,"children":16018},{},[16019,16028],{"type":42,"tag":794,"props":16020,"children":16021},{},[16022],{"type":42,"tag":73,"props":16023,"children":16025},{"className":16024},[],[16026],{"type":48,"value":16027},"create_datadog_notebook(name, cells, ...)",{"type":42,"tag":794,"props":16029,"children":16030},{},[16031,16037,16039],{"type":42,"tag":73,"props":16032,"children":16034},{"className":16033},[],[16035],{"type":48,"value":16036},"pup notebooks create --title \"TITLE\" --file \u002Ftmp\u002Fnb_cells.json",{"type":48,"value":16038}," — confirm exact flags with ",{"type":42,"tag":73,"props":16040,"children":16042},{"className":16041},[],[16043],{"type":48,"value":16044},"pup notebooks create --help",{"type":42,"tag":762,"props":16046,"children":16047},{},[16048,16057],{"type":42,"tag":794,"props":16049,"children":16050},{},[16051],{"type":42,"tag":73,"props":16052,"children":16054},{"className":16053},[],[16055],{"type":48,"value":16056},"edit_datadog_notebook(id, cells, append_only=true)",{"type":42,"tag":794,"props":16058,"children":16059},{},[16060,16066],{"type":42,"tag":73,"props":16061,"children":16063},{"className":16062},[],[16064],{"type":48,"value":16065},"pup notebooks edit NOTEBOOK_ID --file \u002Ftmp\u002Fnb_cells.json",{"type":48,"value":16067}," (fetches current notebook, appends provided cells, writes back)",{"type":42,"tag":51,"props":16069,"children":16070},{},[16071],{"type":48,"value":16072},"The cells file is a JSON array of cell objects:",{"type":42,"tag":730,"props":16074,"children":16076},{"className":8722,"code":16075,"language":8724,"meta":738,"style":738},"[{\"attributes\": {\"definition\": {\"type\": \"markdown\", \"text\": \"## Section\\n\\nContent.\"}}, \"type\": \"notebook_cells\"}]\n",[16077],{"type":42,"tag":73,"props":16078,"children":16079},{"__ignoreMap":738},[16080],{"type":42,"tag":1825,"props":16081,"children":16082},{"class":1827,"line":1828},[16083,16088,16092,16097,16101,16105,16109,16113,16118,16122,16126,16130,16134,16138,16142,16146,16150,16154,16158,16162,16166,16170,16174,16178,16182,16187,16192,16197,16201,16206,16210,16214,16218,16222,16226,16231,16235],{"type":42,"tag":1825,"props":16084,"children":16085},{"style":8275},[16086],{"type":48,"value":16087},"[{",{"type":42,"tag":1825,"props":16089,"children":16090},{"style":8275},[16091],{"type":48,"value":8755},{"type":42,"tag":1825,"props":16093,"children":16094},{"style":8747},[16095],{"type":48,"value":16096},"attributes",{"type":42,"tag":1825,"props":16098,"children":16099},{"style":8275},[16100],{"type":48,"value":8755},{"type":42,"tag":1825,"props":16102,"children":16103},{"style":8275},[16104],{"type":48,"value":4920},{"type":42,"tag":1825,"props":16106,"children":16107},{"style":8275},[16108],{"type":48,"value":12941},{"type":42,"tag":1825,"props":16110,"children":16111},{"style":8275},[16112],{"type":48,"value":8755},{"type":42,"tag":1825,"props":16114,"children":16115},{"style":8281},[16116],{"type":48,"value":16117},"definition",{"type":42,"tag":1825,"props":16119,"children":16120},{"style":8275},[16121],{"type":48,"value":8755},{"type":42,"tag":1825,"props":16123,"children":16124},{"style":8275},[16125],{"type":48,"value":4920},{"type":42,"tag":1825,"props":16127,"children":16128},{"style":8275},[16129],{"type":48,"value":12941},{"type":42,"tag":1825,"props":16131,"children":16132},{"style":8275},[16133],{"type":48,"value":8755},{"type":42,"tag":1825,"props":16135,"children":16136},{"style":9280},[16137],{"type":48,"value":8085},{"type":42,"tag":1825,"props":16139,"children":16140},{"style":8275},[16141],{"type":48,"value":8755},{"type":42,"tag":1825,"props":16143,"children":16144},{"style":8275},[16145],{"type":48,"value":4920},{"type":42,"tag":1825,"props":16147,"children":16148},{"style":8275},[16149],{"type":48,"value":8764},{"type":42,"tag":1825,"props":16151,"children":16152},{"style":8380},[16153],{"type":48,"value":8265},{"type":42,"tag":1825,"props":16155,"children":16156},{"style":8275},[16157],{"type":48,"value":8755},{"type":42,"tag":1825,"props":16159,"children":16160},{"style":8275},[16161],{"type":48,"value":12976},{"type":42,"tag":1825,"props":16163,"children":16164},{"style":8275},[16165],{"type":48,"value":8764},{"type":42,"tag":1825,"props":16167,"children":16168},{"style":9280},[16169],{"type":48,"value":48},{"type":42,"tag":1825,"props":16171,"children":16172},{"style":8275},[16173],{"type":48,"value":8755},{"type":42,"tag":1825,"props":16175,"children":16176},{"style":8275},[16177],{"type":48,"value":4920},{"type":42,"tag":1825,"props":16179,"children":16180},{"style":8275},[16181],{"type":48,"value":8764},{"type":42,"tag":1825,"props":16183,"children":16184},{"style":8380},[16185],{"type":48,"value":16186},"## Section",{"type":42,"tag":1825,"props":16188,"children":16189},{"style":8313},[16190],{"type":48,"value":16191},"\\n\\n",{"type":42,"tag":1825,"props":16193,"children":16194},{"style":8380},[16195],{"type":48,"value":16196},"Content.",{"type":42,"tag":1825,"props":16198,"children":16199},{"style":8275},[16200],{"type":48,"value":8755},{"type":42,"tag":1825,"props":16202,"children":16203},{"style":8275},[16204],{"type":48,"value":16205},"}},",{"type":42,"tag":1825,"props":16207,"children":16208},{"style":8275},[16209],{"type":48,"value":8764},{"type":42,"tag":1825,"props":16211,"children":16212},{"style":8747},[16213],{"type":48,"value":8085},{"type":42,"tag":1825,"props":16215,"children":16216},{"style":8275},[16217],{"type":48,"value":8755},{"type":42,"tag":1825,"props":16219,"children":16220},{"style":8275},[16221],{"type":48,"value":4920},{"type":42,"tag":1825,"props":16223,"children":16224},{"style":8275},[16225],{"type":48,"value":8764},{"type":42,"tag":1825,"props":16227,"children":16228},{"style":8380},[16229],{"type":48,"value":16230},"notebook_cells",{"type":42,"tag":1825,"props":16232,"children":16233},{"style":8275},[16234],{"type":48,"value":8755},{"type":42,"tag":1825,"props":16236,"children":16237},{"style":8275},[16238],{"type":48,"value":16239},"}]\n",{"type":42,"tag":194,"props":16241,"children":16242},{},[16243],{"type":42,"tag":67,"props":16244,"children":16245},{},[16246,16251,16253,16259,16261,16267,16269,16275],{"type":42,"tag":55,"props":16247,"children":16248},{},[16249],{"type":48,"value":16250},"MCP result parsing safety",{"type":48,"value":16252},": Before writing any script (Python, jq, etc.) that iterates over or accesses fields in an MCP tool result, inspect the raw structure first — check ",{"type":42,"tag":73,"props":16254,"children":16256},{"className":16255},[],[16257],{"type":48,"value":16258},"type(result)",{"type":48,"value":16260},", top-level keys, and whether the payload is nested inside a content block (e.g. ",{"type":42,"tag":73,"props":16262,"children":16264},{"className":16263},[],[16265],{"type":48,"value":16266},"[{'type': 'text', 'text': '\u003Cjson>'}]",{"type":48,"value":16268},"). Extract and ",{"type":42,"tag":73,"props":16270,"children":16272},{"className":16271},[],[16273],{"type":48,"value":16274},"json.loads()",{"type":48,"value":16276}," the inner payload if needed before parsing. Never assume MCP results are bare dicts or lists.",{"type":42,"tag":16278,"props":16279,"children":16280},"style",{},[16281],{"type":48,"value":16282},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":16284,"total":7774},[16285,16300,16314,16321,16336,16350,16365,16379,16391,16403,16415,16435],{"slug":16286,"name":16286,"fn":16287,"description":16288,"org":16289,"tags":16290,"stars":25,"repoUrl":26,"updatedAt":16299},"agent-install","install Datadog Agent on Kubernetes","Install the Datadog Agent on Kubernetes using the Datadog Operator — required before enabling Single Step Instrumentation (SSI), which automatically instruments applications for APM without code changes. Only use if no Datadog Agent is deployed on the cluster yet.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16291,16292,16295,16298],{"name":20,"slug":21,"type":15},{"name":16293,"slug":16294,"type":15},"Deployment","deployment",{"name":16296,"slug":16297,"type":15},"Kubernetes","kubernetes",{"name":13,"slug":14,"type":15},"2026-04-15T04:57:27.489805",{"slug":16301,"name":16301,"fn":16302,"description":16303,"org":16304,"tags":16305,"stars":25,"repoUrl":26,"updatedAt":16313},"agent-observability-auto-experiment","run iterative code improvements using Datadog data","Run an iterative code-improvement hill-climb against real Datadog LLM-Obs data, locally, with Claude Code as the agent. Establishes a baseline eval, makes one focused change, re-scores with the same harness, keeps the change if it improves the score in the goal's direction (labeling within-noise gains tentative), and repeats. Use when the user says \"run an auto experiment\", \"hill-climb this code\", \"iteratively improve X and measure the delta\", \"optimize this prompt\u002Ffile against my traces\", \"auto-optimize against LLM-Obs\", or wants the local equivalent of the auto_experiments worker. Works from a local dataset file, an ml_app, a dataset_id, or a list of trace_ids.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16306,16307,16310,16311,16312],{"name":20,"slug":21,"type":15},{"name":16308,"slug":16309,"type":15},"Debugging","debugging",{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-07-31T05:52:13.711906",{"slug":4,"name":4,"fn":5,"description":6,"org":16315,"tags":16316,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16317,16318,16319,16320],{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":688,"name":688,"fn":16322,"description":16323,"org":16324,"tags":16325,"stars":25,"repoUrl":26,"updatedAt":16335},"run agent observability and evaluation pipelines","End-to-end Agent Observability pipeline for an instrumented ml_app — classify production traces, root-cause failures, bootstrap evaluators, then (optionally) sample + publish a dataset, generate + run an experiment, and analyze results. Six narrated phases with a standardized banner and a \"continue\" checkpoint between each. Pure orchestration over the agent-observability sub-skills (`agent-observability-session-classify`, `agent-observability-trace-rca`, `agent-observability-eval-bootstrap`, `agent-observability-experiment-py-bootstrap`, `agent-observability-experiment-analyzer`). Use when user says \"run the eval pipeline\", \"go from traces to evals\", \"bootstrap evals end to end\", \"classify then RCA then bootstrap\", \"build an eval set from scratch\", \"onboard me to datasets and experiments\", \"walk me through experiments\", \"I have an ml_app, now what\", \"Agent Observability onboarding\", \"guided experiment setup\", \"from traces to experiments\", or wants a deterministic, narrated tour from production data through evaluators, datasets, and experiments. Stop early with `--stop-after \u003Cphase>` to short-circuit at evaluators or dataset, or resume mid-flow with `--start-at \u003Cphase>`.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16326,16329,16332,16333,16334],{"name":16327,"slug":16328,"type":15},"Agents","agents",{"name":16330,"slug":16331,"type":15},"Data Pipeline","data-pipeline",{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:17.231423",{"slug":16337,"name":16337,"fn":16338,"description":16339,"org":16340,"tags":16341,"stars":25,"repoUrl":26,"updatedAt":16349},"agent-observability-experiment-analyzer","analyze LLM experiment results","Analyze LLM experiment results. Handles single or comparative experiments, exploratory or Q&A modes. Use when user says \"analyze experiment\", \"compare experiments\", \"analyze against baseline\", or provides one or two experiment IDs for analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16342,16345,16346,16347,16348],{"name":16343,"slug":16344,"type":15},"Analytics","analytics",{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:21.212498",{"slug":16351,"name":16351,"fn":16352,"description":16353,"org":16354,"tags":16355,"stars":25,"repoUrl":26,"updatedAt":16364},"agent-observability-experiment-py-bootstrap","generate Python experiment clients for LLM observability","Generates a self-contained Python experiment client that uses the ddtrace.llmobs SDK. Emits either a runnable .py script or a Jupyter .ipynb notebook matching the canonical DataDog reference notebook style. Use when the user says \"generate Python experiment\", \"write an SDK experiment\", \"create a ddtrace experiment\", \"Python notebook experiment\", \"use the Agent Observability SDK\", or has `ddtrace` installed and wants idiomatic SDK code.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16356,16357,16360,16361,16362],{"name":20,"slug":21,"type":15},{"name":16358,"slug":16359,"type":15},"Jupyter","jupyter",{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":16363,"slug":1819,"type":15},"Python","2026-06-19T09:04:22.640384",{"slug":16366,"name":16366,"fn":16367,"description":16368,"org":16369,"tags":16370,"stars":25,"repoUrl":26,"updatedAt":16378},"agent-observability-replay-trace","iterate on LLM traces with local code","Use when a developer wants to iterate on ONE specific Agent Observability \u002F LLM Obs trace whose output they didn't like — re-running that trace against their LOCAL code, seeing a concise diff of the old vs new output, and looping (change code → replay → diff) until satisfied. Invoked as \u002Fagent-observability-replay-trace \u003Ctrace-id> [changes to test]. Signals: \"replay this trace\"; \"iterate on a trace\"; \"this trace's output is wrong, fix it and re-run\"; \"re-run trace \u003Cid> with \u003Cchange>\"; pasting a trace id from the Agent Observability UI with a description of what to fix. It fetches the trace via the datadog-llmo MCP or the pup CLI, edits code, re-runs the app to emit a NEW trace, and diffs the two — no local server, no browser. For agents traced with ddtrace \u002F LLM Obs (Python first-class), with JSON-serializable entry input. Do NOT use for: scored Experiments or the browser \"Replay\" button (that's agent-observability-replay-experiment), building an experiment from a dataset\u002FCSV, writing evaluators, root-causing failed traces, or RUM\u002FHTTP session replay.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16371,16372,16373,16374,16375],{"name":20,"slug":21,"type":15},{"name":16308,"slug":16309,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":16376,"slug":16377,"type":15},"Tracing","tracing","2026-07-31T05:52:08.594182",{"slug":16380,"name":16380,"fn":16381,"description":16382,"org":16383,"tags":16384,"stars":25,"repoUrl":26,"updatedAt":16390},"agent-observability-session-classify","evaluate user intent satisfaction in sessions","Classify whether user intent was satisfied in a Datadog Agent Observability trace or session. Three modes: (1) session_id — classify a single CMD+I assistant session with RUM; (2) trace_id — classify a single Agent Observability trace without RUM; (3) ml_app — sample and classify multiple sessions or traces from a given LLM app. Output is compact by default (verdict + one-sentence reason). Use when evaluating satisfaction, classifying sessions\u002Ftraces, labeling data, or generating signal for agent-observability-eval-pipeline or agent-observability-trace-rca.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16385,16386,16387,16388,16389],{"name":16343,"slug":16344,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:26.341497",{"slug":16392,"name":16392,"fn":16393,"description":16394,"org":16395,"tags":16396,"stars":25,"repoUrl":26,"updatedAt":16402},"agent-observability-trace-rca","diagnose LLM application failures from traces","Root cause analysis on production LLM traces. Diagnoses why an LLM application is failing — works from eval judge verdicts, runtime errors, or structural anomalies depending on what signals are present. Walks the span tree from symptom to root cause. Use when user says \"what's wrong with my app\", \"why is my eval failing\", \"analyze errors\", \"root cause analysis\", \"diagnose failures\", or wants to understand production failure patterns.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16397,16398,16399,16400,16401],{"name":20,"slug":21,"type":15},{"name":16308,"slug":16309,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},"2026-06-19T09:04:24.316244",{"slug":16404,"name":16404,"fn":16405,"description":16406,"org":16407,"tags":16408,"stars":25,"repoUrl":26,"updatedAt":16414},"agent-skills","use Datadog observability skills","Datadog skills for AI agents. Essential monitoring, logging, tracing and observability.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16409,16410,16413],{"name":20,"slug":21,"type":15},{"name":16411,"slug":16412,"type":15},"Monitoring","monitoring",{"name":13,"slug":14,"type":15},"2026-04-06T18:08:33.337476",{"slug":16416,"name":16416,"fn":16417,"description":16418,"org":16419,"tags":16420,"stars":25,"repoUrl":26,"updatedAt":16434},"datadog-app","build and manage Datadog applications","Guides developers building Datadog Apps with TypeScript, React, the @datadog\u002Fapps scaffolder, and @datadog\u002Fvite-plugin. Use when a user wants to scaffold, run, debug, upgrade, build, upload, publish, upload without publishing (draft upload), add an upload-no-publish script, set up CI\u002FCD, use OAuth or API\u002Fapplication key auth, trigger\u002Fpoll Workflow Automation, choose DDSQL or Action Catalog for backend data access, or query app datastores with DDSQL, including backend function troubleshooting.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16421,16422,16425,16428,16431],{"name":20,"slug":21,"type":15},{"name":16423,"slug":16424,"type":15},"Frontend","frontend",{"name":16426,"slug":16427,"type":15},"React","react",{"name":16429,"slug":16430,"type":15},"TypeScript","typescript",{"name":16432,"slug":16433,"type":15},"Vite","vite","2026-06-18T08:01:32.562331",{"slug":16436,"name":16436,"fn":16437,"description":16438,"org":16439,"tags":16440,"stars":25,"repoUrl":26,"updatedAt":16449},"dd-apm","query Datadog APM traces","APM - install, onboard, instrument, enable, set up, configure, traces, services, dependencies, performance analysis. Use for any request involving Datadog APM setup, instrumentation (SSI, ddtrace, agent install), or analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16441,16442,16445,16446],{"name":20,"slug":21,"type":15},{"name":16443,"slug":16444,"type":15},"Distributed Tracing","distributed-tracing",{"name":13,"slug":14,"type":15},{"name":16447,"slug":16448,"type":15},"Performance","performance","2026-04-06T18:08:34.575282",{"items":16451,"total":7766},[16452,16459,16467,16474,16482,16490,16498],{"slug":16286,"name":16286,"fn":16287,"description":16288,"org":16453,"tags":16454,"stars":25,"repoUrl":26,"updatedAt":16299},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16455,16456,16457,16458],{"name":20,"slug":21,"type":15},{"name":16293,"slug":16294,"type":15},{"name":16296,"slug":16297,"type":15},{"name":13,"slug":14,"type":15},{"slug":16301,"name":16301,"fn":16302,"description":16303,"org":16460,"tags":16461,"stars":25,"repoUrl":26,"updatedAt":16313},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16462,16463,16464,16465,16466],{"name":20,"slug":21,"type":15},{"name":16308,"slug":16309,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":4,"name":4,"fn":5,"description":6,"org":16468,"tags":16469,"stars":25,"repoUrl":26,"updatedAt":27},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16470,16471,16472,16473],{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":688,"name":688,"fn":16322,"description":16323,"org":16475,"tags":16476,"stars":25,"repoUrl":26,"updatedAt":16335},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16477,16478,16479,16480,16481],{"name":16327,"slug":16328,"type":15},{"name":16330,"slug":16331,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":13,"slug":14,"type":15},{"slug":16337,"name":16337,"fn":16338,"description":16339,"org":16483,"tags":16484,"stars":25,"repoUrl":26,"updatedAt":16349},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16485,16486,16487,16488,16489],{"name":16343,"slug":16344,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"slug":16351,"name":16351,"fn":16352,"description":16353,"org":16491,"tags":16492,"stars":25,"repoUrl":26,"updatedAt":16364},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16493,16494,16495,16496,16497],{"name":20,"slug":21,"type":15},{"name":16358,"slug":16359,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":16363,"slug":1819,"type":15},{"slug":16366,"name":16366,"fn":16367,"description":16368,"org":16499,"tags":16500,"stars":25,"repoUrl":26,"updatedAt":16378},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[16501,16502,16503,16504,16505],{"name":20,"slug":21,"type":15},{"name":16308,"slug":16309,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"name":16376,"slug":16377,"type":15}]