[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-arize-arize-evaluator":3,"mdc--wv1ida-key":44,"related-org-arize-arize-evaluator":5058,"related-repo-arize-arize-evaluator":5222},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":39,"sourceUrl":42,"mdContent":43},"arize-evaluator","configure and run Arize evaluations","Handles LLM-as-judge and code evaluator workflows on Arize including creating\u002Fupdating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, code evaluator, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"arize","Arize AI","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Farize.jpg","Arize-ai",[13,17,20],{"name":14,"slug":15,"type":16},"Observability","observability","tag",{"name":18,"slug":19,"type":16},"LLM","llm",{"name":21,"slug":22,"type":16},"Evals","evals",38,"https:\u002F\u002Fgithub.com\u002FArize-ai\u002Farize-skills","2026-07-25T05:32:37.552903",null,5,[29,30,31,8,32,33,34,35,36,37,38],"agent-skills","ai-agents","ai-observability","claude-code","codex","cursor","datasets","experiments","llmops","tracing",{"repoUrl":24,"stars":23,"forks":27,"topics":40,"description":41},[29,30,31,8,32,33,34,35,36,37,38],"Agent skills for Arize — datasets, experiments, and traces via the ax CLI","https:\u002F\u002Fgithub.com\u002FArize-ai\u002Farize-skills\u002Ftree\u002FHEAD\u002Fskills\u002Farize-evaluator","---\nname: arize-evaluator\ndescription: Handles LLM-as-judge and code evaluator workflows on Arize including creating\u002Fupdating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, code evaluator, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt.\nmetadata:\n  author: arize\n  version: \"1.0\"\ncompatibility: Requires the ax CLI and a configured Arize profile with an AI integration.\n---\n\n# Arize Evaluator Skill\n\n> **`SPACE`** — All `--space` flags and the `ARIZE_SPACE` env var accept a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`.\n\nThis skill covers designing, creating, and running **evaluators** on Arize — both **LLM-as-judge (template)** evaluators and **code evaluators** (deterministic, no LLM required). An evaluator defines the judge; a **task** is how you run it against real data.\n\n---\n\n## Prerequisites\n\nProceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront.\n\nIf an `ax` command fails, troubleshoot based on the error:\n- `command not found` or version error → see [references\u002Fax-setup.md](references\u002Fax-setup.md)\n- `401 Unauthorized` \u002F missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow [references\u002Fax-profiles.md](references\u002Fax-profiles.md) to create\u002Fupdate it. If the user doesn't have their key, direct them to https:\u002F\u002Fapp.arize.com\u002Fadmin > API Keys\n- Space unknown → run `ax spaces list` to pick by name, or ask the user\n- LLM provider call fails (missing provider credentials) → run `ax ai-integrations list --space SPACE` to check for platform-managed credentials. If none exist, use the **arize-ai-provider-integration** skill — never ask the user to paste a provider key into chat.\n- **Security:** Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. Never ask the user to paste secrets into chat. For missing credentials, see [references\u002Fax-profiles.md](references\u002Fax-profiles.md).\n- **CRITICAL — Never fabricate evaluation results:** If an evaluation task fails, is cancelled, or produces no scores, report the failure clearly and explain what went wrong. Do NOT perform a \"manual evaluation,\" invent quality scores, estimate percentages, or present any agent-generated analysis as if it came from the Arize evaluation system. Instead suggest: (1) fix the identified issue and retry, (2) try running from the Arize UI, (3) verify integration credentials with `ax ai-integrations list`, (4) contact support at https:\u002F\u002Farize.com\u002Fsupport\n\n---\n\n## Concepts\n\n### What is an Evaluator?\n\nAn **evaluator** is an LLM-as-judge definition. It contains:\n\n| Field | Description |\n|-------|-------------|\n| **Template** | The judge prompt. Uses `{variable}` placeholders (e.g. `{input}`, `{output}`, `{context}`) that get filled in at run time via a task's column mappings. |\n| **Classification choices** | The set of allowed output labels (e.g. `factual` \u002F `hallucinated`). Binary is the default and most common. Each choice can optionally carry a numeric score. |\n| **AI Integration** | Stored LLM provider credentials (OpenAI, Anthropic, Bedrock, etc.) the evaluator uses to call the judge model. |\n| **Model** | The specific judge model (e.g. `gpt-4o`, `claude-sonnet-4-5`). |\n| **Invocation params** | Optional JSON of model settings like `{\"temperature\": 0}`. Low temperature is recommended for reproducibility. |\n| **Optimization direction** | Whether higher scores are better (`maximize`) or worse (`minimize`). Sets how the UI renders trends. |\n| **Data granularity** | Whether the evaluator runs at the **span**, **trace**, or **session** level. Most evaluators run at the span level. |\n\nEvaluators are **versioned** — every prompt or model change creates a new immutable version. The most recent version is active.\n\n**Code evaluators** are the deterministic alternative — no AI integration or model, just Python. They run as a class subclassing `CodeEvaluator`, not a bare function, and have their own strict import-path and `evaluate()`-signature contract. Getting either wrong makes a run cancel silently at `0\u002F0\u002F0`. See \"Custom Python code evaluators\" in [references\u002Fcli-reference.md](references\u002Fcli-reference.md) before writing one.\n\n### What is a Task?\n\nA **task** is how you run one or more evaluators against real data. Tasks are attached to a **project** (live traces\u002Fspans) or a **dataset** (experiment runs). A task contains:\n\n| Field | Description |\n|-------|-------------|\n| **Evaluators** | List of evaluators to run. You can run multiple in one task. |\n| **Column mappings** | Maps each evaluator's template variables to actual field paths on spans or experiment runs (e.g. `\"input\" → \"attributes.input.value\"`). This is what makes evaluators portable across projects and experiments. |\n| **Query filter** | SQL-style expression to select which spans\u002Fruns to evaluate (e.g. `\"span_kind = 'LLM'\"`). Optional but important for precision. |\n| **Continuous** | For project tasks: whether to automatically score new spans as they arrive. |\n| **Sampling rate** | For continuous project tasks: fraction of new spans to evaluate (0–1). |\n\n---\n\n## Data Granularity\n\nThe `--data-granularity` flag controls what unit of data the evaluator scores. It defaults to `span` and only applies to **project tasks** (not dataset\u002Fexperiment tasks — those evaluate experiment runs directly).\n\n| Level | What it evaluates | Use for | Result column prefix |\n|-------|-------------------|---------|---------------------|\n| `span` (default) | Individual spans | Q&A correctness, hallucination, relevance | `eval.{name}.label` \u002F `.score` \u002F `.explanation` |\n| `trace` | All spans in a trace, grouped by `context.trace_id` | Agent trajectory, task correctness — anything that needs the full call chain | `trace_eval.{name}.label` \u002F `.score` \u002F `.explanation` |\n| `session` | All traces in a session, grouped by `attributes.session.id` and ordered by start time | Multi-turn coherence, overall tone, conversation quality | `session_eval.{name}.label` \u002F `.score` \u002F `.explanation` |\n\n### How trace and session aggregation works\n\nFor **trace** granularity, spans sharing the same `context.trace_id` are grouped together. Column values used by the evaluator template are comma-joined into a single string (each value truncated to 100K characters) before being passed to the judge model.\n\nFor **session** granularity, the same trace-level grouping happens first, then traces are ordered by `start_time` and grouped by `attributes.session.id`. Session-level values are capped at 100K characters total.\n\n### The `{conversation}` template variable\n\nAt session granularity, `{conversation}` is a special template variable that renders as a JSON array of `{input, output}` turns across all traces in the session, built from `attributes.input.value` \u002F `attributes.llm.input_messages` (input side) and `attributes.output.value` \u002F `attributes.llm.output_messages` (output side).\n\nAt span or trace granularity, `{conversation}` is treated as a regular template variable and resolved via column mappings like any other.\n\n> **Note:** For `{conversation}` to work, spans must carry `attributes.session.id`. See the **arize-instrumentation** skill for how to emit `session.id` from application code, including the `force_flush()` pattern required for Jupyter notebooks and short-lived scripts.\n\n### Multi-evaluator tasks\n\nA task can contain evaluators at different granularities. At runtime the system uses the **highest** granularity (session > trace > span) for data fetching and automatically **splits into one child run per evaluator**. Per-evaluator `query_filter` in the task's evaluators JSON further narrows which spans are included (e.g., only tool-call spans within a session).\n\n---\n\n## Basic CRUD\n\nFull command reference for **AI integrations**, **evaluators** (template and code), and **tasks** — every flag, with examples — is in [references\u002Fcli-reference.md](references\u002Fcli-reference.md). The workflows below include the commands you need inline.\n\n---\n\n## Workflow A: Create an evaluator for a project\n\nUse this when the user says something like *\"create an evaluator for my Playground Traces project\"*.\n\n### Step 1: Confirm the project name\n\n`ax spans export` accepts a project name directly — no ID lookup needed. If you don't know the project name, list available projects:\n\n```bash\nax projects list --space SPACE -o json\n```\n\nFind the entry whose `\"name\"` matches (case-insensitive) and use that name as `PROJECT` in subsequent commands. If you later hit a validation error with a name, fall back to using the project's `\"id\"` (a base64 string) instead.\n\n### Step 2: Understand what to evaluate\n\nIf the user specified the evaluator type (hallucination, correctness, relevance, etc.) → skip to Step 3.\n\nIf not, sample recent spans to base the evaluator on actual data:\n\n```bash\nax spans export PROJECT --space SPACE -l 10 --days 30 --stdout\n```\n\nInspect `attributes.input`, `attributes.output`, span kinds, and any existing annotations. Identify failure modes (e.g. hallucinated facts, off-topic answers, missing context) and propose **1–3 concrete evaluator ideas**. Let the user pick.\n\nEach suggestion must include: the evaluator name (bold), a one-sentence description of what it judges, and the binary label pair in parentheses. Format each like:\n\n1. **Name** — Description of what is being judged. (`label_a` \u002F `label_b`)\n\nExample:\n1. **Response Correctness** — Does the agent's response correctly address the user's financial query? (`correct` \u002F `incorrect`)\n2. **Hallucination** — Does the response fabricate facts not grounded in retrieved context? (`factual` \u002F `hallucinated`)\n\n### Step 3: Confirm or create an AI integration\n\n```bash\nax ai-integrations list --space SPACE -o json\n```\n\nIf a suitable integration exists, note its ID. If not, create one using the **arize-ai-provider-integration** skill. Ask the user which provider\u002Fmodel they want for the judge.\n\n### Step 4: Create the evaluator\n\nUse the template design best practices below. Keep the evaluator name and variables **generic** — the task (Step 6) handles project-specific wiring via `column_mappings`.\n\n```bash\nax evaluators create-template-evaluator \\\n  --name \"Hallucination\" \\\n  --space SPACE \\\n  --template-name \"hallucination\" \\\n  --commit-message \"Initial version\" \\\n  --ai-integration-id INT_ID \\\n  --model-name \"gpt-4o\" \\\n  --include-explanation \\\n  --use-function-calling \\\n  --classification-choices '{\"factual\": 1, \"hallucinated\": 0}' \\\n  --template 'You are an evaluator. Given the user question and the model response, decide if the response is factual or contains unsupported claims.\n\nUser question: {input}\n\nModel response: {output}\n\nRespond with exactly one of these labels: hallucinated, factual'\n```\n\n### Step 5: Ask — backfill, continuous, or both?\n\n**Recommended approach:** Always start with a small backfill (~100 historical spans) to validate the evaluator before turning on continuous monitoring. This lets you catch column mapping errors, wrong span kinds, and template issues on known data before scoring all future production spans. Only enable continuous after a backfill confirms correct scoring.\n\nBefore creating the task, ask:\n\n> \"Would you like to:\n> (a) Run a **backfill** on historical spans (one-time)?\n> (b) Set up **continuous** evaluation on new spans going forward?\n> (c) **Both** — backfill first to validate, then keep scoring new spans automatically? (recommended)\"\n\n### Step 6: Determine column mappings from real span data\n\nDo not guess paths. Pull a sample and inspect what fields are actually present:\n\n```bash\nax spans export PROJECT --space SPACE -l 5 --days 7 --stdout\n```\n\nFor each template variable (`{input}`, `{output}`, `{context}`), find the matching JSON path. Common starting points — **always verify on your actual data before using**:\n\n| Template var | LLM span | CHAIN span |\n|---|---|---|\n| `input` | `attributes.input.value` | `attributes.input.value` |\n| `output` | `attributes.llm.output_messages.0.message.content` | `attributes.output.value` |\n| `context` | `attributes.retrieval.documents.contents` | — |\n| `tool_output` | `attributes.input.value` (fallback) | `attributes.output.value` |\n\n**Validate span kind alignment:** If the evaluator prompt assumes LLM final text but the task targets CHAIN spans (or vice versa), runs can cancel or score the wrong text. Make sure the `query_filter` on the task matches the span kind you mapped.\n\n**`query_filter` only works on indexed attributes:** The `query_filter` in the evaluators JSON is evaluated against the eval index, not the raw span store. Attributes under `attributes.metadata.*` or custom keys may not be indexed and will silently match nothing. Use well-known indexed attributes like `span_kind` or `attributes.llm.model_name` for filtering. If a filter returns 0 spans despite data existing, try removing the filter as a diagnostic step.\n\n**Full example `--evaluators` JSON:**\n\n```json\n[\n  {\n    \"evaluator_id\": \"EVAL_ID\",\n    \"query_filter\": \"span_kind = 'LLM'\",\n    \"column_mappings\": {\n      \"input\": \"attributes.input.value\",\n      \"output\": \"attributes.llm.output_messages.0.message.content\",\n      \"context\": \"attributes.retrieval.documents.contents\"\n    }\n  }\n]\n```\n\nInclude a mapping for **every** variable the template references. Omitting one causes runs to produce no valid scores.\n\n### Step 7: Create the task\n\n**Backfill only (a):**\n```bash\nax tasks create-evaluation \\\n  --name \"Hallucination Backfill\" \\\n  --task-type TEMPLATE_EVALUATION \\\n  --project PROJECT \\\n  --evaluators '[{\"evaluator_id\": \"EVAL_ID\", \"column_mappings\": {\"input\": \"attributes.input.value\", \"output\": \"attributes.output.value\"}}]' \\\n  --no-continuous\n```\n\n**Continuous only (b):**\n```bash\nax tasks create-evaluation \\\n  --name \"Hallucination Monitor\" \\\n  --task-type TEMPLATE_EVALUATION \\\n  --project PROJECT \\\n  --evaluators '[{\"evaluator_id\": \"EVAL_ID\", \"column_mappings\": {\"input\": \"attributes.input.value\", \"output\": \"attributes.output.value\"}}]' \\\n  --is-continuous \\\n  --sampling-rate 0.1\n```\n\n**Both (c):** Use `--is-continuous` on create, then also trigger a backfill run in Step 8.\n\n### Step 8: Trigger a backfill run (if requested)\n\n> **Eval index lag:** The eval index is built asynchronously from the primary trace store and can lag **1–2 hours**. For your first test run, use a time window ending at least 2 hours in the past. If you set `--data-end-time` to \"now\" on spans ingested in the last hour, the run will complete successfully but score 0 spans.\n\nFirst find what time range has data:\n```bash\nax spans export PROJECT --space SPACE -l 100 --days 1 --stdout   # try last 24h first\nax spans export PROJECT --space SPACE -l 100 --days 7 --stdout   # widen if empty\n```\n\nUse the `start_time` \u002F `end_time` fields from real spans to set the window. For the first validation run, cap `--max-spans` at ~100 to get quick feedback:\n\n```bash\nax tasks trigger-run TASK_ID \\\n  --data-start-time \"2026-03-20T00:00:00\" \\\n  --data-end-time \"2026-03-21T23:59:59\" \\\n  --max-spans 100 \\\n  --wait\n```\n\nReview scores and explanations before widening to the full backfill or enabling continuous.\n\n---\n\n## Workflow B: Create an evaluator for an experiment\n\nUse this when the user says something like *\"create an evaluator for my experiment\"* or *\"evaluate my dataset runs\"*.\n\n**If the user says \"dataset\" but doesn't have an experiment:** A task must target an experiment (not a bare dataset). Ask:\n> \"Evaluation tasks run against experiment runs, not datasets directly. Would you like help creating an experiment on that dataset first?\"\n\nIf yes, use the **arize-experiment** skill to create one, then return here.\n\n### Step 1: Find the dataset and experiment names\n\n```bash\nax datasets list --space SPACE\nax experiments list --dataset DATASET_NAME --space SPACE -o json\n```\n\nNote the dataset name and the experiment name(s) to score. These accept names or IDs in subsequent commands — names are preferred.\n\n### Step 2: Understand what to evaluate\n\nIf the user specified the evaluator type → skip to Step 3.\n\nIf not, inspect a recent experiment run to base the evaluator on actual data:\n\n```bash\nax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE --stdout | python3 -c \"import sys,json; runs=json.load(sys.stdin); print(json.dumps(runs[0], indent=2))\"\n```\n\nLook at the `output`, `input`, `evaluations`, and `metadata` fields. Identify gaps (metrics the user cares about but doesn't have yet) and propose **1–3 evaluator ideas**. Each suggestion must include: the evaluator name (bold), a one-sentence description, and the binary label pair in parentheses — same format as Workflow A, Step 2.\n\n### Step 3: Confirm or create an AI integration\n\nSame as Workflow A, Step 3.\n\n### Step 4: Create the evaluator\n\nSame as Workflow A, Step 4. Keep variables generic.\n\n### Step 5: Determine column mappings from real run data\n\nRun data shape differs from span data. Inspect:\n\n```bash\nax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE --stdout | python3 -c \"import sys,json; runs=json.load(sys.stdin); print(json.dumps(runs[0], indent=2))\"\n```\n\nCommon mapping for experiment runs:\n- `output` → `\"output\"` (top-level field on each run)\n- `input` → check if it's on the run or embedded in the linked dataset examples\n\nIf `input` is not on the run JSON, export dataset examples to find the path:\n```bash\nax datasets export DATASET_NAME --space SPACE --stdout | python3 -c \"import sys,json; ex=json.load(sys.stdin); print(json.dumps(ex[0], indent=2))\"\n```\n\n### Step 6: Create the task\n\n```bash\nax tasks create-evaluation \\\n  --name \"Experiment Correctness\" \\\n  --task-type TEMPLATE_EVALUATION \\\n  --dataset DATASET_NAME --space SPACE \\\n  --experiment-ids \"EXP_ID\" \\   # base64 ID from `ax experiments list --space SPACE -o json`\n  --evaluators '[{\"evaluator_id\": \"EVAL_ID\", \"column_mappings\": {\"output\": \"output\"}}]' \\\n  --no-continuous\n```\n\n### Step 7: Trigger and monitor\n\n```bash\nax tasks trigger-run TASK_ID \\\n  --experiment-ids \"EXP_ID\" \\   # base64 ID from `ax experiments list --space SPACE -o json`\n  --wait\n\nax tasks list-runs TASK_ID\nax tasks get-run RUN_ID\n```\n\n---\n\n## Best Practices for Template Design\n\n### 1. Use generic, portable variable names\n\nUse `{input}`, `{output}`, and `{context}` — not names tied to a specific project or span attribute (e.g. do not use `{attributes_input_value}`). The evaluator itself stays abstract; the **task's `column_mappings`** is where you wire it to the actual fields in a specific project or experiment. This lets the same evaluator run across multiple projects and experiments without modification.\n\n### 2. Default to binary labels\n\nUse exactly two clear string labels (e.g. `hallucinated` \u002F `factual`, `correct` \u002F `incorrect`, `pass` \u002F `fail`). Binary labels are:\n- Easiest for the judge model to produce consistently\n- Most common in the industry\n- Simplest to interpret in dashboards\n\nIf the user insists on more than two choices, that's fine — but recommend binary first and explain the tradeoff (more labels → more ambiguity → lower inter-rater reliability).\n\n### 3. Be explicit about what the model must return\n\nThe template must tell the judge model to respond with **only** the label string — nothing else. The label strings in the prompt must **exactly match** the labels in `--classification-choices` (same spelling, same casing).\n\nGood:\n```\nRespond with exactly one of these labels: hallucinated, factual\n```\n\nBad (too open-ended):\n```\nIs this hallucinated? Answer yes or no.\n```\n\n### 4. Keep temperature low\n\nPass `--invocation-params '{\"temperature\": 0}'` for reproducible scoring. Higher temperatures introduce noise into evaluation results.\n\n### 5. Use `--include-explanations` for debugging\n\nDuring initial setup, always include explanations so you can verify the judge is reasoning correctly before trusting the labels at scale.\n\n### 6. Pass the template in single quotes in bash\n\nSingle quotes prevent the shell from interpolating `{variable}` placeholders. Double quotes will cause issues:\n\n```bash\n# Correct\n--template 'Judge this: {input} → {output}'\n\n# Wrong — shell may interpret { } or fail\n--template \"Judge this: {input} → {output}\"\n```\n\n### 7. Always set `--classification-choices` to match your template labels\n\nThe labels in `--classification-choices` must exactly match the labels referenced in `--template` (same spelling, same casing). Omitting `--classification-choices` causes task runs to fail with \"missing rails and classification choices.\"\n\n---\n\n## Troubleshooting\n\n| Problem | Solution |\n|---------|----------|\n| `ax: command not found` | See [references\u002Fax-setup.md](references\u002Fax-setup.md) |\n| `401 Unauthorized` | API key may not have access to this space. Verify at https:\u002F\u002Fapp.arize.com\u002Fadmin > API Keys |\n| `Evaluator not found` | `ax evaluators list --space SPACE` |\n| `Integration not found` | `ax ai-integrations list --space SPACE` |\n| `Task not found` | `ax tasks list --space SPACE` |\n| `project and dataset-id are mutually exclusive` | Use only one when creating a task |\n| `experiment-ids required for dataset tasks` | Add `--experiment-ids` to `create` and `trigger-run` |\n| `sampling-rate only valid for project tasks` | Remove `--sampling-rate` from dataset tasks |\n| Validation error on `ax spans export` | Project name usually works; if you still get a validation error, look up the base64 project ID via `ax projects list --space SPACE -o json` and use the `id` field instead |\n| Template validation errors | Use single-quoted `--template '...'` in bash; single braces `{var}`, not double `{{var}}` |\n| Run stuck in `pending` | `ax tasks get-run RUN_ID`; then `ax tasks cancel-run RUN_ID` |\n| Run `cancelled` ~1s | Integration credentials invalid — check AI integration |\n| Run `cancelled` ~3min | Found spans but LLM call failed — wrong model name or bad key |\n| Run `completed`, 0 spans | Widen time window; eval index may not cover older data |\n| No scores in UI | Fix `column_mappings` to match real paths on your spans\u002Fruns |\n| Scores look wrong | Add `--include-explanations` and inspect judge reasoning on a few samples |\n| Evaluator cancels on wrong span kind | Match `query_filter` and `column_mappings` to LLM vs CHAIN spans |\n| Time format error on `trigger-run` | Use `2026-03-21T09:00:00` — no trailing `Z` |\n| Run failed: \"missing rails and classification choices\" | Add `--classification-choices '{\"label_a\": 1, \"label_b\": 0}'` to `ax evaluators create-template-evaluator` — labels must match the template |\n| Run `completed`, all spans skipped | Query filter matched spans but column mappings are wrong or template variables don't resolve — export a sample span and verify paths |\n| `query_filter` set but 0 spans scored | The filter attribute may not be indexed in the eval index. `attributes.metadata.*` and custom attributes are often not indexed. Use `span_kind` or `attributes.llm.model_name` instead, or remove the filter to confirm spans exist in the window. |\n| Custom **code** evaluator run cancels ~3s with `0\u002F0\u002F0` (successes\u002Ferrors\u002Fskipped) | Wrong import path or `evaluate()` signature — see the \"CRITICAL\" callout under **Custom Python code evaluators** in [references\u002Fcli-reference.md](references\u002Fcli-reference.md). Must import from `arize.experimental.datasets.experiments.evaluators.base` (not `arize.experiments`) and declare named `evaluate()` params, not just `**kwargs`. |\n\n### Diagnosing cancelled runs\n\nWhen a task run reports status `cancelled`, work through the ordered checklist (credentials → model name → column-mapping\u002Fpath checks → time window → span kind → variable resolution) in [references\u002Ftroubleshooting.md](references\u002Ftroubleshooting.md).\n\n---\n\n## Related Skills\n\n- **arize-ai-provider-integration**: Full CRUD for LLM provider integrations (create, update, delete credentials)\n- **arize-trace**: Export spans to discover column paths and time ranges\n- **arize-experiment**: Create experiments and export runs for experiment column mappings\n- **arize-dataset**: Export dataset examples to find input fields when runs omit them\n- **arize-prompts**: Manage prompt templates in the Prompt Hub; use with evaluators to score versioned prompts\n- **arize-link**: Deep links to evaluators and tasks in the Arize UI\n\n---\n\n## Save Credentials for Future Use\n\nSee [references\u002Fax-profiles.md](references\u002Fax-profiles.md) § Save Credentials for Future Use.\n",{"data":45,"body":49},{"name":4,"description":6,"metadata":46,"compatibility":48},{"author":8,"version":47},"1.0","Requires the ax CLI and a configured Arize profile with an AI integration.",{"type":50,"children":51},"root",[52,61,135,168,172,179,192,204,356,359,365,372,384,630,642,683,689,714,830,833,839,866,1042,1048,1066,1091,1104,1154,1166,1216,1222,1249,1252,1258,1288,1291,1297,1309,1315,1326,1378,1407,1413,1418,1423,1486,1513,1518,1547,1552,1601,1607,1646,1657,1663,1682,1981,1987,1997,2002,2031,2037,2042,2098,2129,2269,2286,2332,2348,2600,2612,2618,2626,2744,2752,2880,2898,2904,2932,2937,3052,3079,3185,3190,3193,3199,3215,3225,3233,3245,3251,3325,3330,3335,3340,3345,3420,3460,3465,3470,3475,3480,3486,3491,3560,3565,3596,3608,3671,3677,3828,3834,3947,3950,3956,3962,4006,4012,4056,4074,4079,4085,4112,4117,4127,4132,4141,4147,4160,4174,4179,4185,4197,4267,4280,4306,4309,4315,4939,4945,4963,4966,4972,5033,5036,5042,5052],{"type":53,"tag":54,"props":55,"children":57},"element","h1",{"id":56},"arize-evaluator-skill",[58],{"type":59,"value":60},"text","Arize Evaluator Skill",{"type":53,"tag":62,"props":63,"children":64},"blockquote",{},[65],{"type":53,"tag":66,"props":67,"children":68},"p",{},[69,80,82,88,90,96,98,103,105,111,113,118,119,125,127,133],{"type":53,"tag":70,"props":71,"children":72},"strong",{},[73],{"type":53,"tag":74,"props":75,"children":77},"code",{"className":76},[],[78],{"type":59,"value":79},"SPACE",{"type":59,"value":81}," — All ",{"type":53,"tag":74,"props":83,"children":85},{"className":84},[],[86],{"type":59,"value":87},"--space",{"type":59,"value":89}," flags and the ",{"type":53,"tag":74,"props":91,"children":93},{"className":92},[],[94],{"type":59,"value":95},"ARIZE_SPACE",{"type":59,"value":97}," env var accept a space ",{"type":53,"tag":70,"props":99,"children":100},{},[101],{"type":59,"value":102},"name",{"type":59,"value":104}," (e.g., ",{"type":53,"tag":74,"props":106,"children":108},{"className":107},[],[109],{"type":59,"value":110},"my-workspace",{"type":59,"value":112},") or a base64 space ",{"type":53,"tag":70,"props":114,"children":115},{},[116],{"type":59,"value":117},"ID",{"type":59,"value":104},{"type":53,"tag":74,"props":120,"children":122},{"className":121},[],[123],{"type":59,"value":124},"U3BhY2U6...",{"type":59,"value":126},"). Find yours with ",{"type":53,"tag":74,"props":128,"children":130},{"className":129},[],[131],{"type":59,"value":132},"ax spaces list",{"type":59,"value":134},".",{"type":53,"tag":66,"props":136,"children":137},{},[138,140,145,147,152,154,159,161,166],{"type":59,"value":139},"This skill covers designing, creating, and running ",{"type":53,"tag":70,"props":141,"children":142},{},[143],{"type":59,"value":144},"evaluators",{"type":59,"value":146}," on Arize — both ",{"type":53,"tag":70,"props":148,"children":149},{},[150],{"type":59,"value":151},"LLM-as-judge (template)",{"type":59,"value":153}," evaluators and ",{"type":53,"tag":70,"props":155,"children":156},{},[157],{"type":59,"value":158},"code evaluators",{"type":59,"value":160}," (deterministic, no LLM required). An evaluator defines the judge; a ",{"type":53,"tag":70,"props":162,"children":163},{},[164],{"type":59,"value":165},"task",{"type":59,"value":167}," is how you run it against real data.",{"type":53,"tag":169,"props":170,"children":171},"hr",{},[],{"type":53,"tag":173,"props":174,"children":176},"h2",{"id":175},"prerequisites",[177],{"type":59,"value":178},"Prerequisites",{"type":53,"tag":66,"props":180,"children":181},{},[182,184,190],{"type":59,"value":183},"Proceed directly with the task — run the ",{"type":53,"tag":74,"props":185,"children":187},{"className":186},[],[188],{"type":59,"value":189},"ax",{"type":59,"value":191}," command you need. Do NOT check versions, env vars, or profiles upfront.",{"type":53,"tag":66,"props":193,"children":194},{},[195,197,202],{"type":59,"value":196},"If an ",{"type":53,"tag":74,"props":198,"children":200},{"className":199},[],[201],{"type":59,"value":189},{"type":59,"value":203}," command fails, troubleshoot based on the error:",{"type":53,"tag":205,"props":206,"children":207},"ul",{},[208,226,261,273,293,332],{"type":53,"tag":209,"props":210,"children":211},"li",{},[212,218,220],{"type":53,"tag":74,"props":213,"children":215},{"className":214},[],[216],{"type":59,"value":217},"command not found",{"type":59,"value":219}," or version error → see ",{"type":53,"tag":221,"props":222,"children":224},"a",{"href":223},"references\u002Fax-setup.md",[225],{"type":59,"value":223},{"type":53,"tag":209,"props":227,"children":228},{},[229,235,237,243,245,250,252,259],{"type":53,"tag":74,"props":230,"children":232},{"className":231},[],[233],{"type":59,"value":234},"401 Unauthorized",{"type":59,"value":236}," \u002F missing API key → run ",{"type":53,"tag":74,"props":238,"children":240},{"className":239},[],[241],{"type":59,"value":242},"ax profiles show",{"type":59,"value":244}," to inspect the current profile. If the profile is missing or the API key is wrong, follow ",{"type":53,"tag":221,"props":246,"children":248},{"href":247},"references\u002Fax-profiles.md",[249],{"type":59,"value":247},{"type":59,"value":251}," to create\u002Fupdate it. If the user doesn't have their key, direct them to ",{"type":53,"tag":221,"props":253,"children":257},{"href":254,"rel":255},"https:\u002F\u002Fapp.arize.com\u002Fadmin",[256],"nofollow",[258],{"type":59,"value":254},{"type":59,"value":260}," > API Keys",{"type":53,"tag":209,"props":262,"children":263},{},[264,266,271],{"type":59,"value":265},"Space unknown → run ",{"type":53,"tag":74,"props":267,"children":269},{"className":268},[],[270],{"type":59,"value":132},{"type":59,"value":272}," to pick by name, or ask the user",{"type":53,"tag":209,"props":274,"children":275},{},[276,278,284,286,291],{"type":59,"value":277},"LLM provider call fails (missing provider credentials) → run ",{"type":53,"tag":74,"props":279,"children":281},{"className":280},[],[282],{"type":59,"value":283},"ax ai-integrations list --space SPACE",{"type":59,"value":285}," to check for platform-managed credentials. If none exist, use the ",{"type":53,"tag":70,"props":287,"children":288},{},[289],{"type":59,"value":290},"arize-ai-provider-integration",{"type":59,"value":292}," skill — never ask the user to paste a provider key into chat.",{"type":53,"tag":209,"props":294,"children":295},{},[296,301,303,309,311,317,319,325,327,331],{"type":53,"tag":70,"props":297,"children":298},{},[299],{"type":59,"value":300},"Security:",{"type":59,"value":302}," Never read ",{"type":53,"tag":74,"props":304,"children":306},{"className":305},[],[307],{"type":59,"value":308},".env",{"type":59,"value":310}," files or search the filesystem for credentials. Use ",{"type":53,"tag":74,"props":312,"children":314},{"className":313},[],[315],{"type":59,"value":316},"ax profiles",{"type":59,"value":318}," for Arize credentials and ",{"type":53,"tag":74,"props":320,"children":322},{"className":321},[],[323],{"type":59,"value":324},"ax ai-integrations",{"type":59,"value":326}," for LLM provider keys. Never ask the user to paste secrets into chat. For missing credentials, see ",{"type":53,"tag":221,"props":328,"children":329},{"href":247},[330],{"type":59,"value":247},{"type":59,"value":134},{"type":53,"tag":209,"props":333,"children":334},{},[335,340,342,348,350],{"type":53,"tag":70,"props":336,"children":337},{},[338],{"type":59,"value":339},"CRITICAL — Never fabricate evaluation results:",{"type":59,"value":341}," If an evaluation task fails, is cancelled, or produces no scores, report the failure clearly and explain what went wrong. Do NOT perform a \"manual evaluation,\" invent quality scores, estimate percentages, or present any agent-generated analysis as if it came from the Arize evaluation system. Instead suggest: (1) fix the identified issue and retry, (2) try running from the Arize UI, (3) verify integration credentials with ",{"type":53,"tag":74,"props":343,"children":345},{"className":344},[],[346],{"type":59,"value":347},"ax ai-integrations list",{"type":59,"value":349},", (4) contact support at ",{"type":53,"tag":221,"props":351,"children":354},{"href":352,"rel":353},"https:\u002F\u002Farize.com\u002Fsupport",[256],[355],{"type":59,"value":352},{"type":53,"tag":169,"props":357,"children":358},{},[],{"type":53,"tag":173,"props":360,"children":362},{"id":361},"concepts",[363],{"type":59,"value":364},"Concepts",{"type":53,"tag":366,"props":367,"children":369},"h3",{"id":368},"what-is-an-evaluator",[370],{"type":59,"value":371},"What is an Evaluator?",{"type":53,"tag":66,"props":373,"children":374},{},[375,377,382],{"type":59,"value":376},"An ",{"type":53,"tag":70,"props":378,"children":379},{},[380],{"type":59,"value":381},"evaluator",{"type":59,"value":383}," is an LLM-as-judge definition. It contains:",{"type":53,"tag":385,"props":386,"children":387},"table",{},[388,407],{"type":53,"tag":389,"props":390,"children":391},"thead",{},[392],{"type":53,"tag":393,"props":394,"children":395},"tr",{},[396,402],{"type":53,"tag":397,"props":398,"children":399},"th",{},[400],{"type":59,"value":401},"Field",{"type":53,"tag":397,"props":403,"children":404},{},[405],{"type":59,"value":406},"Description",{"type":53,"tag":408,"props":409,"children":410},"tbody",{},[411,459,491,507,538,562,594],{"type":53,"tag":393,"props":412,"children":413},{},[414,423],{"type":53,"tag":415,"props":416,"children":417},"td",{},[418],{"type":53,"tag":70,"props":419,"children":420},{},[421],{"type":59,"value":422},"Template",{"type":53,"tag":415,"props":424,"children":425},{},[426,428,434,436,442,444,450,451,457],{"type":59,"value":427},"The judge prompt. Uses ",{"type":53,"tag":74,"props":429,"children":431},{"className":430},[],[432],{"type":59,"value":433},"{variable}",{"type":59,"value":435}," placeholders (e.g. ",{"type":53,"tag":74,"props":437,"children":439},{"className":438},[],[440],{"type":59,"value":441},"{input}",{"type":59,"value":443},", ",{"type":53,"tag":74,"props":445,"children":447},{"className":446},[],[448],{"type":59,"value":449},"{output}",{"type":59,"value":443},{"type":53,"tag":74,"props":452,"children":454},{"className":453},[],[455],{"type":59,"value":456},"{context}",{"type":59,"value":458},") that get filled in at run time via a task's column mappings.",{"type":53,"tag":393,"props":460,"children":461},{},[462,470],{"type":53,"tag":415,"props":463,"children":464},{},[465],{"type":53,"tag":70,"props":466,"children":467},{},[468],{"type":59,"value":469},"Classification choices",{"type":53,"tag":415,"props":471,"children":472},{},[473,475,481,483,489],{"type":59,"value":474},"The set of allowed output labels (e.g. ",{"type":53,"tag":74,"props":476,"children":478},{"className":477},[],[479],{"type":59,"value":480},"factual",{"type":59,"value":482}," \u002F ",{"type":53,"tag":74,"props":484,"children":486},{"className":485},[],[487],{"type":59,"value":488},"hallucinated",{"type":59,"value":490},"). Binary is the default and most common. Each choice can optionally carry a numeric score.",{"type":53,"tag":393,"props":492,"children":493},{},[494,502],{"type":53,"tag":415,"props":495,"children":496},{},[497],{"type":53,"tag":70,"props":498,"children":499},{},[500],{"type":59,"value":501},"AI Integration",{"type":53,"tag":415,"props":503,"children":504},{},[505],{"type":59,"value":506},"Stored LLM provider credentials (OpenAI, Anthropic, Bedrock, etc.) the evaluator uses to call the judge model.",{"type":53,"tag":393,"props":508,"children":509},{},[510,518],{"type":53,"tag":415,"props":511,"children":512},{},[513],{"type":53,"tag":70,"props":514,"children":515},{},[516],{"type":59,"value":517},"Model",{"type":53,"tag":415,"props":519,"children":520},{},[521,523,529,530,536],{"type":59,"value":522},"The specific judge model (e.g. ",{"type":53,"tag":74,"props":524,"children":526},{"className":525},[],[527],{"type":59,"value":528},"gpt-4o",{"type":59,"value":443},{"type":53,"tag":74,"props":531,"children":533},{"className":532},[],[534],{"type":59,"value":535},"claude-sonnet-4-5",{"type":59,"value":537},").",{"type":53,"tag":393,"props":539,"children":540},{},[541,549],{"type":53,"tag":415,"props":542,"children":543},{},[544],{"type":53,"tag":70,"props":545,"children":546},{},[547],{"type":59,"value":548},"Invocation params",{"type":53,"tag":415,"props":550,"children":551},{},[552,554,560],{"type":59,"value":553},"Optional JSON of model settings like ",{"type":53,"tag":74,"props":555,"children":557},{"className":556},[],[558],{"type":59,"value":559},"{\"temperature\": 0}",{"type":59,"value":561},". Low temperature is recommended for reproducibility.",{"type":53,"tag":393,"props":563,"children":564},{},[565,573],{"type":53,"tag":415,"props":566,"children":567},{},[568],{"type":53,"tag":70,"props":569,"children":570},{},[571],{"type":59,"value":572},"Optimization direction",{"type":53,"tag":415,"props":574,"children":575},{},[576,578,584,586,592],{"type":59,"value":577},"Whether higher scores are better (",{"type":53,"tag":74,"props":579,"children":581},{"className":580},[],[582],{"type":59,"value":583},"maximize",{"type":59,"value":585},") or worse (",{"type":53,"tag":74,"props":587,"children":589},{"className":588},[],[590],{"type":59,"value":591},"minimize",{"type":59,"value":593},"). Sets how the UI renders trends.",{"type":53,"tag":393,"props":595,"children":596},{},[597,605],{"type":53,"tag":415,"props":598,"children":599},{},[600],{"type":53,"tag":70,"props":601,"children":602},{},[603],{"type":59,"value":604},"Data granularity",{"type":53,"tag":415,"props":606,"children":607},{},[608,610,615,616,621,623,628],{"type":59,"value":609},"Whether the evaluator runs at the ",{"type":53,"tag":70,"props":611,"children":612},{},[613],{"type":59,"value":614},"span",{"type":59,"value":443},{"type":53,"tag":70,"props":617,"children":618},{},[619],{"type":59,"value":620},"trace",{"type":59,"value":622},", or ",{"type":53,"tag":70,"props":624,"children":625},{},[626],{"type":59,"value":627},"session",{"type":59,"value":629}," level. Most evaluators run at the span level.",{"type":53,"tag":66,"props":631,"children":632},{},[633,635,640],{"type":59,"value":634},"Evaluators are ",{"type":53,"tag":70,"props":636,"children":637},{},[638],{"type":59,"value":639},"versioned",{"type":59,"value":641}," — every prompt or model change creates a new immutable version. The most recent version is active.",{"type":53,"tag":66,"props":643,"children":644},{},[645,650,652,658,660,666,668,674,676,681],{"type":53,"tag":70,"props":646,"children":647},{},[648],{"type":59,"value":649},"Code evaluators",{"type":59,"value":651}," are the deterministic alternative — no AI integration or model, just Python. They run as a class subclassing ",{"type":53,"tag":74,"props":653,"children":655},{"className":654},[],[656],{"type":59,"value":657},"CodeEvaluator",{"type":59,"value":659},", not a bare function, and have their own strict import-path and ",{"type":53,"tag":74,"props":661,"children":663},{"className":662},[],[664],{"type":59,"value":665},"evaluate()",{"type":59,"value":667},"-signature contract. Getting either wrong makes a run cancel silently at ",{"type":53,"tag":74,"props":669,"children":671},{"className":670},[],[672],{"type":59,"value":673},"0\u002F0\u002F0",{"type":59,"value":675},". See \"Custom Python code evaluators\" in ",{"type":53,"tag":221,"props":677,"children":679},{"href":678},"references\u002Fcli-reference.md",[680],{"type":59,"value":678},{"type":59,"value":682}," before writing one.",{"type":53,"tag":366,"props":684,"children":686},{"id":685},"what-is-a-task",[687],{"type":59,"value":688},"What is a Task?",{"type":53,"tag":66,"props":690,"children":691},{},[692,694,698,700,705,707,712],{"type":59,"value":693},"A ",{"type":53,"tag":70,"props":695,"children":696},{},[697],{"type":59,"value":165},{"type":59,"value":699}," is how you run one or more evaluators against real data. Tasks are attached to a ",{"type":53,"tag":70,"props":701,"children":702},{},[703],{"type":59,"value":704},"project",{"type":59,"value":706}," (live traces\u002Fspans) or a ",{"type":53,"tag":70,"props":708,"children":709},{},[710],{"type":59,"value":711},"dataset",{"type":59,"value":713}," (experiment runs). A task contains:",{"type":53,"tag":385,"props":715,"children":716},{},[717,731],{"type":53,"tag":389,"props":718,"children":719},{},[720],{"type":53,"tag":393,"props":721,"children":722},{},[723,727],{"type":53,"tag":397,"props":724,"children":725},{},[726],{"type":59,"value":401},{"type":53,"tag":397,"props":728,"children":729},{},[730],{"type":59,"value":406},{"type":53,"tag":408,"props":732,"children":733},{},[734,750,774,798,814],{"type":53,"tag":393,"props":735,"children":736},{},[737,745],{"type":53,"tag":415,"props":738,"children":739},{},[740],{"type":53,"tag":70,"props":741,"children":742},{},[743],{"type":59,"value":744},"Evaluators",{"type":53,"tag":415,"props":746,"children":747},{},[748],{"type":59,"value":749},"List of evaluators to run. You can run multiple in one task.",{"type":53,"tag":393,"props":751,"children":752},{},[753,761],{"type":53,"tag":415,"props":754,"children":755},{},[756],{"type":53,"tag":70,"props":757,"children":758},{},[759],{"type":59,"value":760},"Column mappings",{"type":53,"tag":415,"props":762,"children":763},{},[764,766,772],{"type":59,"value":765},"Maps each evaluator's template variables to actual field paths on spans or experiment runs (e.g. ",{"type":53,"tag":74,"props":767,"children":769},{"className":768},[],[770],{"type":59,"value":771},"\"input\" → \"attributes.input.value\"",{"type":59,"value":773},"). This is what makes evaluators portable across projects and experiments.",{"type":53,"tag":393,"props":775,"children":776},{},[777,785],{"type":53,"tag":415,"props":778,"children":779},{},[780],{"type":53,"tag":70,"props":781,"children":782},{},[783],{"type":59,"value":784},"Query filter",{"type":53,"tag":415,"props":786,"children":787},{},[788,790,796],{"type":59,"value":789},"SQL-style expression to select which spans\u002Fruns to evaluate (e.g. ",{"type":53,"tag":74,"props":791,"children":793},{"className":792},[],[794],{"type":59,"value":795},"\"span_kind = 'LLM'\"",{"type":59,"value":797},"). Optional but important for precision.",{"type":53,"tag":393,"props":799,"children":800},{},[801,809],{"type":53,"tag":415,"props":802,"children":803},{},[804],{"type":53,"tag":70,"props":805,"children":806},{},[807],{"type":59,"value":808},"Continuous",{"type":53,"tag":415,"props":810,"children":811},{},[812],{"type":59,"value":813},"For project tasks: whether to automatically score new spans as they arrive.",{"type":53,"tag":393,"props":815,"children":816},{},[817,825],{"type":53,"tag":415,"props":818,"children":819},{},[820],{"type":53,"tag":70,"props":821,"children":822},{},[823],{"type":59,"value":824},"Sampling rate",{"type":53,"tag":415,"props":826,"children":827},{},[828],{"type":59,"value":829},"For continuous project tasks: fraction of new spans to evaluate (0–1).",{"type":53,"tag":169,"props":831,"children":832},{},[],{"type":53,"tag":173,"props":834,"children":836},{"id":835},"data-granularity",[837],{"type":59,"value":838},"Data Granularity",{"type":53,"tag":66,"props":840,"children":841},{},[842,844,850,852,857,859,864],{"type":59,"value":843},"The ",{"type":53,"tag":74,"props":845,"children":847},{"className":846},[],[848],{"type":59,"value":849},"--data-granularity",{"type":59,"value":851}," flag controls what unit of data the evaluator scores. It defaults to ",{"type":53,"tag":74,"props":853,"children":855},{"className":854},[],[856],{"type":59,"value":614},{"type":59,"value":858}," and only applies to ",{"type":53,"tag":70,"props":860,"children":861},{},[862],{"type":59,"value":863},"project tasks",{"type":59,"value":865}," (not dataset\u002Fexperiment tasks — those evaluate experiment runs directly).",{"type":53,"tag":385,"props":867,"children":868},{},[869,895],{"type":53,"tag":389,"props":870,"children":871},{},[872],{"type":53,"tag":393,"props":873,"children":874},{},[875,880,885,890],{"type":53,"tag":397,"props":876,"children":877},{},[878],{"type":59,"value":879},"Level",{"type":53,"tag":397,"props":881,"children":882},{},[883],{"type":59,"value":884},"What it evaluates",{"type":53,"tag":397,"props":886,"children":887},{},[888],{"type":59,"value":889},"Use for",{"type":53,"tag":397,"props":891,"children":892},{},[893],{"type":59,"value":894},"Result column prefix",{"type":53,"tag":408,"props":896,"children":897},{},[898,944,992],{"type":53,"tag":393,"props":899,"children":900},{},[901,911,916,921],{"type":53,"tag":415,"props":902,"children":903},{},[904,909],{"type":53,"tag":74,"props":905,"children":907},{"className":906},[],[908],{"type":59,"value":614},{"type":59,"value":910}," (default)",{"type":53,"tag":415,"props":912,"children":913},{},[914],{"type":59,"value":915},"Individual spans",{"type":53,"tag":415,"props":917,"children":918},{},[919],{"type":59,"value":920},"Q&A correctness, hallucination, relevance",{"type":53,"tag":415,"props":922,"children":923},{},[924,930,931,937,938],{"type":53,"tag":74,"props":925,"children":927},{"className":926},[],[928],{"type":59,"value":929},"eval.{name}.label",{"type":59,"value":482},{"type":53,"tag":74,"props":932,"children":934},{"className":933},[],[935],{"type":59,"value":936},".score",{"type":59,"value":482},{"type":53,"tag":74,"props":939,"children":941},{"className":940},[],[942],{"type":59,"value":943},".explanation",{"type":53,"tag":393,"props":945,"children":946},{},[947,955,966,971],{"type":53,"tag":415,"props":948,"children":949},{},[950],{"type":53,"tag":74,"props":951,"children":953},{"className":952},[],[954],{"type":59,"value":620},{"type":53,"tag":415,"props":956,"children":957},{},[958,960],{"type":59,"value":959},"All spans in a trace, grouped by ",{"type":53,"tag":74,"props":961,"children":963},{"className":962},[],[964],{"type":59,"value":965},"context.trace_id",{"type":53,"tag":415,"props":967,"children":968},{},[969],{"type":59,"value":970},"Agent trajectory, task correctness — anything that needs the full call chain",{"type":53,"tag":415,"props":972,"children":973},{},[974,980,981,986,987],{"type":53,"tag":74,"props":975,"children":977},{"className":976},[],[978],{"type":59,"value":979},"trace_eval.{name}.label",{"type":59,"value":482},{"type":53,"tag":74,"props":982,"children":984},{"className":983},[],[985],{"type":59,"value":936},{"type":59,"value":482},{"type":53,"tag":74,"props":988,"children":990},{"className":989},[],[991],{"type":59,"value":943},{"type":53,"tag":393,"props":993,"children":994},{},[995,1003,1016,1021],{"type":53,"tag":415,"props":996,"children":997},{},[998],{"type":53,"tag":74,"props":999,"children":1001},{"className":1000},[],[1002],{"type":59,"value":627},{"type":53,"tag":415,"props":1004,"children":1005},{},[1006,1008,1014],{"type":59,"value":1007},"All traces in a session, grouped by ",{"type":53,"tag":74,"props":1009,"children":1011},{"className":1010},[],[1012],{"type":59,"value":1013},"attributes.session.id",{"type":59,"value":1015}," and ordered by start time",{"type":53,"tag":415,"props":1017,"children":1018},{},[1019],{"type":59,"value":1020},"Multi-turn coherence, overall tone, conversation quality",{"type":53,"tag":415,"props":1022,"children":1023},{},[1024,1030,1031,1036,1037],{"type":53,"tag":74,"props":1025,"children":1027},{"className":1026},[],[1028],{"type":59,"value":1029},"session_eval.{name}.label",{"type":59,"value":482},{"type":53,"tag":74,"props":1032,"children":1034},{"className":1033},[],[1035],{"type":59,"value":936},{"type":59,"value":482},{"type":53,"tag":74,"props":1038,"children":1040},{"className":1039},[],[1041],{"type":59,"value":943},{"type":53,"tag":366,"props":1043,"children":1045},{"id":1044},"how-trace-and-session-aggregation-works",[1046],{"type":59,"value":1047},"How trace and session aggregation works",{"type":53,"tag":66,"props":1049,"children":1050},{},[1051,1053,1057,1059,1064],{"type":59,"value":1052},"For ",{"type":53,"tag":70,"props":1054,"children":1055},{},[1056],{"type":59,"value":620},{"type":59,"value":1058}," granularity, spans sharing the same ",{"type":53,"tag":74,"props":1060,"children":1062},{"className":1061},[],[1063],{"type":59,"value":965},{"type":59,"value":1065}," are grouped together. Column values used by the evaluator template are comma-joined into a single string (each value truncated to 100K characters) before being passed to the judge model.",{"type":53,"tag":66,"props":1067,"children":1068},{},[1069,1070,1074,1076,1082,1084,1089],{"type":59,"value":1052},{"type":53,"tag":70,"props":1071,"children":1072},{},[1073],{"type":59,"value":627},{"type":59,"value":1075}," granularity, the same trace-level grouping happens first, then traces are ordered by ",{"type":53,"tag":74,"props":1077,"children":1079},{"className":1078},[],[1080],{"type":59,"value":1081},"start_time",{"type":59,"value":1083}," and grouped by ",{"type":53,"tag":74,"props":1085,"children":1087},{"className":1086},[],[1088],{"type":59,"value":1013},{"type":59,"value":1090},". Session-level values are capped at 100K characters total.",{"type":53,"tag":366,"props":1092,"children":1094},{"id":1093},"the-conversation-template-variable",[1095,1096,1102],{"type":59,"value":843},{"type":53,"tag":74,"props":1097,"children":1099},{"className":1098},[],[1100],{"type":59,"value":1101},"{conversation}",{"type":59,"value":1103}," template variable",{"type":53,"tag":66,"props":1105,"children":1106},{},[1107,1109,1114,1116,1122,1124,1130,1131,1137,1139,1145,1146,1152],{"type":59,"value":1108},"At session granularity, ",{"type":53,"tag":74,"props":1110,"children":1112},{"className":1111},[],[1113],{"type":59,"value":1101},{"type":59,"value":1115}," is a special template variable that renders as a JSON array of ",{"type":53,"tag":74,"props":1117,"children":1119},{"className":1118},[],[1120],{"type":59,"value":1121},"{input, output}",{"type":59,"value":1123}," turns across all traces in the session, built from ",{"type":53,"tag":74,"props":1125,"children":1127},{"className":1126},[],[1128],{"type":59,"value":1129},"attributes.input.value",{"type":59,"value":482},{"type":53,"tag":74,"props":1132,"children":1134},{"className":1133},[],[1135],{"type":59,"value":1136},"attributes.llm.input_messages",{"type":59,"value":1138}," (input side) and ",{"type":53,"tag":74,"props":1140,"children":1142},{"className":1141},[],[1143],{"type":59,"value":1144},"attributes.output.value",{"type":59,"value":482},{"type":53,"tag":74,"props":1147,"children":1149},{"className":1148},[],[1150],{"type":59,"value":1151},"attributes.llm.output_messages",{"type":59,"value":1153}," (output side).",{"type":53,"tag":66,"props":1155,"children":1156},{},[1157,1159,1164],{"type":59,"value":1158},"At span or trace granularity, ",{"type":53,"tag":74,"props":1160,"children":1162},{"className":1161},[],[1163],{"type":59,"value":1101},{"type":59,"value":1165}," is treated as a regular template variable and resolved via column mappings like any other.",{"type":53,"tag":62,"props":1167,"children":1168},{},[1169],{"type":53,"tag":66,"props":1170,"children":1171},{},[1172,1177,1179,1184,1186,1191,1193,1198,1200,1206,1208,1214],{"type":53,"tag":70,"props":1173,"children":1174},{},[1175],{"type":59,"value":1176},"Note:",{"type":59,"value":1178}," For ",{"type":53,"tag":74,"props":1180,"children":1182},{"className":1181},[],[1183],{"type":59,"value":1101},{"type":59,"value":1185}," to work, spans must carry ",{"type":53,"tag":74,"props":1187,"children":1189},{"className":1188},[],[1190],{"type":59,"value":1013},{"type":59,"value":1192},". See the ",{"type":53,"tag":70,"props":1194,"children":1195},{},[1196],{"type":59,"value":1197},"arize-instrumentation",{"type":59,"value":1199}," skill for how to emit ",{"type":53,"tag":74,"props":1201,"children":1203},{"className":1202},[],[1204],{"type":59,"value":1205},"session.id",{"type":59,"value":1207}," from application code, including the ",{"type":53,"tag":74,"props":1209,"children":1211},{"className":1210},[],[1212],{"type":59,"value":1213},"force_flush()",{"type":59,"value":1215}," pattern required for Jupyter notebooks and short-lived scripts.",{"type":53,"tag":366,"props":1217,"children":1219},{"id":1218},"multi-evaluator-tasks",[1220],{"type":59,"value":1221},"Multi-evaluator tasks",{"type":53,"tag":66,"props":1223,"children":1224},{},[1225,1227,1232,1234,1239,1241,1247],{"type":59,"value":1226},"A task can contain evaluators at different granularities. At runtime the system uses the ",{"type":53,"tag":70,"props":1228,"children":1229},{},[1230],{"type":59,"value":1231},"highest",{"type":59,"value":1233}," granularity (session > trace > span) for data fetching and automatically ",{"type":53,"tag":70,"props":1235,"children":1236},{},[1237],{"type":59,"value":1238},"splits into one child run per evaluator",{"type":59,"value":1240},". Per-evaluator ",{"type":53,"tag":74,"props":1242,"children":1244},{"className":1243},[],[1245],{"type":59,"value":1246},"query_filter",{"type":59,"value":1248}," in the task's evaluators JSON further narrows which spans are included (e.g., only tool-call spans within a session).",{"type":53,"tag":169,"props":1250,"children":1251},{},[],{"type":53,"tag":173,"props":1253,"children":1255},{"id":1254},"basic-crud",[1256],{"type":59,"value":1257},"Basic CRUD",{"type":53,"tag":66,"props":1259,"children":1260},{},[1261,1263,1268,1269,1273,1275,1280,1282,1286],{"type":59,"value":1262},"Full command reference for ",{"type":53,"tag":70,"props":1264,"children":1265},{},[1266],{"type":59,"value":1267},"AI integrations",{"type":59,"value":443},{"type":53,"tag":70,"props":1270,"children":1271},{},[1272],{"type":59,"value":144},{"type":59,"value":1274}," (template and code), and ",{"type":53,"tag":70,"props":1276,"children":1277},{},[1278],{"type":59,"value":1279},"tasks",{"type":59,"value":1281}," — every flag, with examples — is in ",{"type":53,"tag":221,"props":1283,"children":1284},{"href":678},[1285],{"type":59,"value":678},{"type":59,"value":1287},". The workflows below include the commands you need inline.",{"type":53,"tag":169,"props":1289,"children":1290},{},[],{"type":53,"tag":173,"props":1292,"children":1294},{"id":1293},"workflow-a-create-an-evaluator-for-a-project",[1295],{"type":59,"value":1296},"Workflow A: Create an evaluator for a project",{"type":53,"tag":66,"props":1298,"children":1299},{},[1300,1302,1308],{"type":59,"value":1301},"Use this when the user says something like ",{"type":53,"tag":1303,"props":1304,"children":1305},"em",{},[1306],{"type":59,"value":1307},"\"create an evaluator for my Playground Traces project\"",{"type":59,"value":134},{"type":53,"tag":366,"props":1310,"children":1312},{"id":1311},"step-1-confirm-the-project-name",[1313],{"type":59,"value":1314},"Step 1: Confirm the project name",{"type":53,"tag":66,"props":1316,"children":1317},{},[1318,1324],{"type":53,"tag":74,"props":1319,"children":1321},{"className":1320},[],[1322],{"type":59,"value":1323},"ax spans export",{"type":59,"value":1325}," accepts a project name directly — no ID lookup needed. If you don't know the project name, list available projects:",{"type":53,"tag":1327,"props":1328,"children":1333},"pre",{"className":1329,"code":1330,"language":1331,"meta":1332,"style":1332},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","ax projects list --space SPACE -o json\n","bash","",[1334],{"type":53,"tag":74,"props":1335,"children":1336},{"__ignoreMap":1332},[1337],{"type":53,"tag":614,"props":1338,"children":1341},{"class":1339,"line":1340},"line",1,[1342,1347,1353,1358,1363,1368,1373],{"type":53,"tag":614,"props":1343,"children":1345},{"style":1344},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1346],{"type":59,"value":189},{"type":53,"tag":614,"props":1348,"children":1350},{"style":1349},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1351],{"type":59,"value":1352}," projects",{"type":53,"tag":614,"props":1354,"children":1355},{"style":1349},[1356],{"type":59,"value":1357}," list",{"type":53,"tag":614,"props":1359,"children":1360},{"style":1349},[1361],{"type":59,"value":1362}," --space",{"type":53,"tag":614,"props":1364,"children":1365},{"style":1349},[1366],{"type":59,"value":1367}," SPACE",{"type":53,"tag":614,"props":1369,"children":1370},{"style":1349},[1371],{"type":59,"value":1372}," -o",{"type":53,"tag":614,"props":1374,"children":1375},{"style":1349},[1376],{"type":59,"value":1377}," json\n",{"type":53,"tag":66,"props":1379,"children":1380},{},[1381,1383,1389,1391,1397,1399,1405],{"type":59,"value":1382},"Find the entry whose ",{"type":53,"tag":74,"props":1384,"children":1386},{"className":1385},[],[1387],{"type":59,"value":1388},"\"name\"",{"type":59,"value":1390}," matches (case-insensitive) and use that name as ",{"type":53,"tag":74,"props":1392,"children":1394},{"className":1393},[],[1395],{"type":59,"value":1396},"PROJECT",{"type":59,"value":1398}," in subsequent commands. If you later hit a validation error with a name, fall back to using the project's ",{"type":53,"tag":74,"props":1400,"children":1402},{"className":1401},[],[1403],{"type":59,"value":1404},"\"id\"",{"type":59,"value":1406}," (a base64 string) instead.",{"type":53,"tag":366,"props":1408,"children":1410},{"id":1409},"step-2-understand-what-to-evaluate",[1411],{"type":59,"value":1412},"Step 2: Understand what to evaluate",{"type":53,"tag":66,"props":1414,"children":1415},{},[1416],{"type":59,"value":1417},"If the user specified the evaluator type (hallucination, correctness, relevance, etc.) → skip to Step 3.",{"type":53,"tag":66,"props":1419,"children":1420},{},[1421],{"type":59,"value":1422},"If not, sample recent spans to base the evaluator on actual data:",{"type":53,"tag":1327,"props":1424,"children":1426},{"className":1329,"code":1425,"language":1331,"meta":1332,"style":1332},"ax spans export PROJECT --space SPACE -l 10 --days 30 --stdout\n",[1427],{"type":53,"tag":74,"props":1428,"children":1429},{"__ignoreMap":1332},[1430],{"type":53,"tag":614,"props":1431,"children":1432},{"class":1339,"line":1340},[1433,1437,1442,1447,1452,1456,1460,1465,1471,1476,1481],{"type":53,"tag":614,"props":1434,"children":1435},{"style":1344},[1436],{"type":59,"value":189},{"type":53,"tag":614,"props":1438,"children":1439},{"style":1349},[1440],{"type":59,"value":1441}," spans",{"type":53,"tag":614,"props":1443,"children":1444},{"style":1349},[1445],{"type":59,"value":1446}," export",{"type":53,"tag":614,"props":1448,"children":1449},{"style":1349},[1450],{"type":59,"value":1451}," PROJECT",{"type":53,"tag":614,"props":1453,"children":1454},{"style":1349},[1455],{"type":59,"value":1362},{"type":53,"tag":614,"props":1457,"children":1458},{"style":1349},[1459],{"type":59,"value":1367},{"type":53,"tag":614,"props":1461,"children":1462},{"style":1349},[1463],{"type":59,"value":1464}," -l",{"type":53,"tag":614,"props":1466,"children":1468},{"style":1467},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1469],{"type":59,"value":1470}," 10",{"type":53,"tag":614,"props":1472,"children":1473},{"style":1349},[1474],{"type":59,"value":1475}," --days",{"type":53,"tag":614,"props":1477,"children":1478},{"style":1467},[1479],{"type":59,"value":1480}," 30",{"type":53,"tag":614,"props":1482,"children":1483},{"style":1349},[1484],{"type":59,"value":1485}," --stdout\n",{"type":53,"tag":66,"props":1487,"children":1488},{},[1489,1491,1497,1498,1504,1506,1511],{"type":59,"value":1490},"Inspect ",{"type":53,"tag":74,"props":1492,"children":1494},{"className":1493},[],[1495],{"type":59,"value":1496},"attributes.input",{"type":59,"value":443},{"type":53,"tag":74,"props":1499,"children":1501},{"className":1500},[],[1502],{"type":59,"value":1503},"attributes.output",{"type":59,"value":1505},", span kinds, and any existing annotations. Identify failure modes (e.g. hallucinated facts, off-topic answers, missing context) and propose ",{"type":53,"tag":70,"props":1507,"children":1508},{},[1509],{"type":59,"value":1510},"1–3 concrete evaluator ideas",{"type":59,"value":1512},". Let the user pick.",{"type":53,"tag":66,"props":1514,"children":1515},{},[1516],{"type":59,"value":1517},"Each suggestion must include: the evaluator name (bold), a one-sentence description of what it judges, and the binary label pair in parentheses. Format each like:",{"type":53,"tag":1519,"props":1520,"children":1521},"ol",{},[1522],{"type":53,"tag":209,"props":1523,"children":1524},{},[1525,1530,1532,1538,1539,1545],{"type":53,"tag":70,"props":1526,"children":1527},{},[1528],{"type":59,"value":1529},"Name",{"type":59,"value":1531}," — Description of what is being judged. (",{"type":53,"tag":74,"props":1533,"children":1535},{"className":1534},[],[1536],{"type":59,"value":1537},"label_a",{"type":59,"value":482},{"type":53,"tag":74,"props":1540,"children":1542},{"className":1541},[],[1543],{"type":59,"value":1544},"label_b",{"type":59,"value":1546},")",{"type":53,"tag":66,"props":1548,"children":1549},{},[1550],{"type":59,"value":1551},"Example:",{"type":53,"tag":1519,"props":1553,"children":1554},{},[1555,1579],{"type":53,"tag":209,"props":1556,"children":1557},{},[1558,1563,1565,1571,1572,1578],{"type":53,"tag":70,"props":1559,"children":1560},{},[1561],{"type":59,"value":1562},"Response Correctness",{"type":59,"value":1564}," — Does the agent's response correctly address the user's financial query? (",{"type":53,"tag":74,"props":1566,"children":1568},{"className":1567},[],[1569],{"type":59,"value":1570},"correct",{"type":59,"value":482},{"type":53,"tag":74,"props":1573,"children":1575},{"className":1574},[],[1576],{"type":59,"value":1577},"incorrect",{"type":59,"value":1546},{"type":53,"tag":209,"props":1580,"children":1581},{},[1582,1587,1589,1594,1595,1600],{"type":53,"tag":70,"props":1583,"children":1584},{},[1585],{"type":59,"value":1586},"Hallucination",{"type":59,"value":1588}," — Does the response fabricate facts not grounded in retrieved context? (",{"type":53,"tag":74,"props":1590,"children":1592},{"className":1591},[],[1593],{"type":59,"value":480},{"type":59,"value":482},{"type":53,"tag":74,"props":1596,"children":1598},{"className":1597},[],[1599],{"type":59,"value":488},{"type":59,"value":1546},{"type":53,"tag":366,"props":1602,"children":1604},{"id":1603},"step-3-confirm-or-create-an-ai-integration",[1605],{"type":59,"value":1606},"Step 3: Confirm or create an AI integration",{"type":53,"tag":1327,"props":1608,"children":1610},{"className":1329,"code":1609,"language":1331,"meta":1332,"style":1332},"ax ai-integrations list --space SPACE -o json\n",[1611],{"type":53,"tag":74,"props":1612,"children":1613},{"__ignoreMap":1332},[1614],{"type":53,"tag":614,"props":1615,"children":1616},{"class":1339,"line":1340},[1617,1621,1626,1630,1634,1638,1642],{"type":53,"tag":614,"props":1618,"children":1619},{"style":1344},[1620],{"type":59,"value":189},{"type":53,"tag":614,"props":1622,"children":1623},{"style":1349},[1624],{"type":59,"value":1625}," ai-integrations",{"type":53,"tag":614,"props":1627,"children":1628},{"style":1349},[1629],{"type":59,"value":1357},{"type":53,"tag":614,"props":1631,"children":1632},{"style":1349},[1633],{"type":59,"value":1362},{"type":53,"tag":614,"props":1635,"children":1636},{"style":1349},[1637],{"type":59,"value":1367},{"type":53,"tag":614,"props":1639,"children":1640},{"style":1349},[1641],{"type":59,"value":1372},{"type":53,"tag":614,"props":1643,"children":1644},{"style":1349},[1645],{"type":59,"value":1377},{"type":53,"tag":66,"props":1647,"children":1648},{},[1649,1651,1655],{"type":59,"value":1650},"If a suitable integration exists, note its ID. If not, create one using the ",{"type":53,"tag":70,"props":1652,"children":1653},{},[1654],{"type":59,"value":290},{"type":59,"value":1656}," skill. Ask the user which provider\u002Fmodel they want for the judge.",{"type":53,"tag":366,"props":1658,"children":1660},{"id":1659},"step-4-create-the-evaluator",[1661],{"type":59,"value":1662},"Step 4: Create the evaluator",{"type":53,"tag":66,"props":1664,"children":1665},{},[1666,1668,1673,1675,1681],{"type":59,"value":1667},"Use the template design best practices below. Keep the evaluator name and variables ",{"type":53,"tag":70,"props":1669,"children":1670},{},[1671],{"type":59,"value":1672},"generic",{"type":59,"value":1674}," — the task (Step 6) handles project-specific wiring via ",{"type":53,"tag":74,"props":1676,"children":1678},{"className":1677},[],[1679],{"type":59,"value":1680},"column_mappings",{"type":59,"value":134},{"type":53,"tag":1327,"props":1683,"children":1685},{"className":1329,"code":1684,"language":1331,"meta":1332,"style":1332},"ax evaluators create-template-evaluator \\\n  --name \"Hallucination\" \\\n  --space SPACE \\\n  --template-name \"hallucination\" \\\n  --commit-message \"Initial version\" \\\n  --ai-integration-id INT_ID \\\n  --model-name \"gpt-4o\" \\\n  --include-explanation \\\n  --use-function-calling \\\n  --classification-choices '{\"factual\": 1, \"hallucinated\": 0}' \\\n  --template 'You are an evaluator. Given the user question and the model response, decide if the response is factual or contains unsupported claims.\n\nUser question: {input}\n\nModel response: {output}\n\nRespond with exactly one of these labels: hallucinated, factual'\n",[1686],{"type":53,"tag":74,"props":1687,"children":1688},{"__ignoreMap":1332},[1689,1712,1740,1757,1783,1808,1826,1851,1864,1877,1905,1923,1933,1942,1950,1959,1967],{"type":53,"tag":614,"props":1690,"children":1691},{"class":1339,"line":1340},[1692,1696,1701,1706],{"type":53,"tag":614,"props":1693,"children":1694},{"style":1344},[1695],{"type":59,"value":189},{"type":53,"tag":614,"props":1697,"children":1698},{"style":1349},[1699],{"type":59,"value":1700}," evaluators",{"type":53,"tag":614,"props":1702,"children":1703},{"style":1349},[1704],{"type":59,"value":1705}," create-template-evaluator",{"type":53,"tag":614,"props":1707,"children":1709},{"style":1708},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1710],{"type":59,"value":1711}," \\\n",{"type":53,"tag":614,"props":1713,"children":1715},{"class":1339,"line":1714},2,[1716,1721,1727,1731,1736],{"type":53,"tag":614,"props":1717,"children":1718},{"style":1349},[1719],{"type":59,"value":1720},"  --name",{"type":53,"tag":614,"props":1722,"children":1724},{"style":1723},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1725],{"type":59,"value":1726}," \"",{"type":53,"tag":614,"props":1728,"children":1729},{"style":1349},[1730],{"type":59,"value":1586},{"type":53,"tag":614,"props":1732,"children":1733},{"style":1723},[1734],{"type":59,"value":1735},"\"",{"type":53,"tag":614,"props":1737,"children":1738},{"style":1708},[1739],{"type":59,"value":1711},{"type":53,"tag":614,"props":1741,"children":1743},{"class":1339,"line":1742},3,[1744,1749,1753],{"type":53,"tag":614,"props":1745,"children":1746},{"style":1349},[1747],{"type":59,"value":1748},"  --space",{"type":53,"tag":614,"props":1750,"children":1751},{"style":1349},[1752],{"type":59,"value":1367},{"type":53,"tag":614,"props":1754,"children":1755},{"style":1708},[1756],{"type":59,"value":1711},{"type":53,"tag":614,"props":1758,"children":1760},{"class":1339,"line":1759},4,[1761,1766,1770,1775,1779],{"type":53,"tag":614,"props":1762,"children":1763},{"style":1349},[1764],{"type":59,"value":1765},"  --template-name",{"type":53,"tag":614,"props":1767,"children":1768},{"style":1723},[1769],{"type":59,"value":1726},{"type":53,"tag":614,"props":1771,"children":1772},{"style":1349},[1773],{"type":59,"value":1774},"hallucination",{"type":53,"tag":614,"props":1776,"children":1777},{"style":1723},[1778],{"type":59,"value":1735},{"type":53,"tag":614,"props":1780,"children":1781},{"style":1708},[1782],{"type":59,"value":1711},{"type":53,"tag":614,"props":1784,"children":1785},{"class":1339,"line":27},[1786,1791,1795,1800,1804],{"type":53,"tag":614,"props":1787,"children":1788},{"style":1349},[1789],{"type":59,"value":1790},"  --commit-message",{"type":53,"tag":614,"props":1792,"children":1793},{"style":1723},[1794],{"type":59,"value":1726},{"type":53,"tag":614,"props":1796,"children":1797},{"style":1349},[1798],{"type":59,"value":1799},"Initial version",{"type":53,"tag":614,"props":1801,"children":1802},{"style":1723},[1803],{"type":59,"value":1735},{"type":53,"tag":614,"props":1805,"children":1806},{"style":1708},[1807],{"type":59,"value":1711},{"type":53,"tag":614,"props":1809,"children":1811},{"class":1339,"line":1810},6,[1812,1817,1822],{"type":53,"tag":614,"props":1813,"children":1814},{"style":1349},[1815],{"type":59,"value":1816},"  --ai-integration-id",{"type":53,"tag":614,"props":1818,"children":1819},{"style":1349},[1820],{"type":59,"value":1821}," INT_ID",{"type":53,"tag":614,"props":1823,"children":1824},{"style":1708},[1825],{"type":59,"value":1711},{"type":53,"tag":614,"props":1827,"children":1829},{"class":1339,"line":1828},7,[1830,1835,1839,1843,1847],{"type":53,"tag":614,"props":1831,"children":1832},{"style":1349},[1833],{"type":59,"value":1834},"  --model-name",{"type":53,"tag":614,"props":1836,"children":1837},{"style":1723},[1838],{"type":59,"value":1726},{"type":53,"tag":614,"props":1840,"children":1841},{"style":1349},[1842],{"type":59,"value":528},{"type":53,"tag":614,"props":1844,"children":1845},{"style":1723},[1846],{"type":59,"value":1735},{"type":53,"tag":614,"props":1848,"children":1849},{"style":1708},[1850],{"type":59,"value":1711},{"type":53,"tag":614,"props":1852,"children":1854},{"class":1339,"line":1853},8,[1855,1860],{"type":53,"tag":614,"props":1856,"children":1857},{"style":1349},[1858],{"type":59,"value":1859},"  --include-explanation",{"type":53,"tag":614,"props":1861,"children":1862},{"style":1708},[1863],{"type":59,"value":1711},{"type":53,"tag":614,"props":1865,"children":1867},{"class":1339,"line":1866},9,[1868,1873],{"type":53,"tag":614,"props":1869,"children":1870},{"style":1349},[1871],{"type":59,"value":1872},"  --use-function-calling",{"type":53,"tag":614,"props":1874,"children":1875},{"style":1708},[1876],{"type":59,"value":1711},{"type":53,"tag":614,"props":1878,"children":1880},{"class":1339,"line":1879},10,[1881,1886,1891,1896,1901],{"type":53,"tag":614,"props":1882,"children":1883},{"style":1349},[1884],{"type":59,"value":1885},"  --classification-choices",{"type":53,"tag":614,"props":1887,"children":1888},{"style":1723},[1889],{"type":59,"value":1890}," '",{"type":53,"tag":614,"props":1892,"children":1893},{"style":1349},[1894],{"type":59,"value":1895},"{\"factual\": 1, \"hallucinated\": 0}",{"type":53,"tag":614,"props":1897,"children":1898},{"style":1723},[1899],{"type":59,"value":1900},"'",{"type":53,"tag":614,"props":1902,"children":1903},{"style":1708},[1904],{"type":59,"value":1711},{"type":53,"tag":614,"props":1906,"children":1908},{"class":1339,"line":1907},11,[1909,1914,1918],{"type":53,"tag":614,"props":1910,"children":1911},{"style":1349},[1912],{"type":59,"value":1913},"  --template",{"type":53,"tag":614,"props":1915,"children":1916},{"style":1723},[1917],{"type":59,"value":1890},{"type":53,"tag":614,"props":1919,"children":1920},{"style":1349},[1921],{"type":59,"value":1922},"You are an evaluator. Given the user question and the model response, decide if the response is factual or contains unsupported claims.\n",{"type":53,"tag":614,"props":1924,"children":1926},{"class":1339,"line":1925},12,[1927],{"type":53,"tag":614,"props":1928,"children":1930},{"emptyLinePlaceholder":1929},true,[1931],{"type":59,"value":1932},"\n",{"type":53,"tag":614,"props":1934,"children":1936},{"class":1339,"line":1935},13,[1937],{"type":53,"tag":614,"props":1938,"children":1939},{"style":1349},[1940],{"type":59,"value":1941},"User question: {input}\n",{"type":53,"tag":614,"props":1943,"children":1945},{"class":1339,"line":1944},14,[1946],{"type":53,"tag":614,"props":1947,"children":1948},{"emptyLinePlaceholder":1929},[1949],{"type":59,"value":1932},{"type":53,"tag":614,"props":1951,"children":1953},{"class":1339,"line":1952},15,[1954],{"type":53,"tag":614,"props":1955,"children":1956},{"style":1349},[1957],{"type":59,"value":1958},"Model response: {output}\n",{"type":53,"tag":614,"props":1960,"children":1962},{"class":1339,"line":1961},16,[1963],{"type":53,"tag":614,"props":1964,"children":1965},{"emptyLinePlaceholder":1929},[1966],{"type":59,"value":1932},{"type":53,"tag":614,"props":1968,"children":1970},{"class":1339,"line":1969},17,[1971,1976],{"type":53,"tag":614,"props":1972,"children":1973},{"style":1349},[1974],{"type":59,"value":1975},"Respond with exactly one of these labels: hallucinated, factual",{"type":53,"tag":614,"props":1977,"children":1978},{"style":1723},[1979],{"type":59,"value":1980},"'\n",{"type":53,"tag":366,"props":1982,"children":1984},{"id":1983},"step-5-ask-backfill-continuous-or-both",[1985],{"type":59,"value":1986},"Step 5: Ask — backfill, continuous, or both?",{"type":53,"tag":66,"props":1988,"children":1989},{},[1990,1995],{"type":53,"tag":70,"props":1991,"children":1992},{},[1993],{"type":59,"value":1994},"Recommended approach:",{"type":59,"value":1996}," Always start with a small backfill (~100 historical spans) to validate the evaluator before turning on continuous monitoring. This lets you catch column mapping errors, wrong span kinds, and template issues on known data before scoring all future production spans. Only enable continuous after a backfill confirms correct scoring.",{"type":53,"tag":66,"props":1998,"children":1999},{},[2000],{"type":59,"value":2001},"Before creating the task, ask:",{"type":53,"tag":62,"props":2003,"children":2004},{},[2005],{"type":53,"tag":66,"props":2006,"children":2007},{},[2008,2010,2015,2017,2022,2024,2029],{"type":59,"value":2009},"\"Would you like to:\n(a) Run a ",{"type":53,"tag":70,"props":2011,"children":2012},{},[2013],{"type":59,"value":2014},"backfill",{"type":59,"value":2016}," on historical spans (one-time)?\n(b) Set up ",{"type":53,"tag":70,"props":2018,"children":2019},{},[2020],{"type":59,"value":2021},"continuous",{"type":59,"value":2023}," evaluation on new spans going forward?\n(c) ",{"type":53,"tag":70,"props":2025,"children":2026},{},[2027],{"type":59,"value":2028},"Both",{"type":59,"value":2030}," — backfill first to validate, then keep scoring new spans automatically? (recommended)\"",{"type":53,"tag":366,"props":2032,"children":2034},{"id":2033},"step-6-determine-column-mappings-from-real-span-data",[2035],{"type":59,"value":2036},"Step 6: Determine column mappings from real span data",{"type":53,"tag":66,"props":2038,"children":2039},{},[2040],{"type":59,"value":2041},"Do not guess paths. Pull a sample and inspect what fields are actually present:",{"type":53,"tag":1327,"props":2043,"children":2045},{"className":1329,"code":2044,"language":1331,"meta":1332,"style":1332},"ax spans export PROJECT --space SPACE -l 5 --days 7 --stdout\n",[2046],{"type":53,"tag":74,"props":2047,"children":2048},{"__ignoreMap":1332},[2049],{"type":53,"tag":614,"props":2050,"children":2051},{"class":1339,"line":1340},[2052,2056,2060,2064,2068,2072,2076,2080,2085,2089,2094],{"type":53,"tag":614,"props":2053,"children":2054},{"style":1344},[2055],{"type":59,"value":189},{"type":53,"tag":614,"props":2057,"children":2058},{"style":1349},[2059],{"type":59,"value":1441},{"type":53,"tag":614,"props":2061,"children":2062},{"style":1349},[2063],{"type":59,"value":1446},{"type":53,"tag":614,"props":2065,"children":2066},{"style":1349},[2067],{"type":59,"value":1451},{"type":53,"tag":614,"props":2069,"children":2070},{"style":1349},[2071],{"type":59,"value":1362},{"type":53,"tag":614,"props":2073,"children":2074},{"style":1349},[2075],{"type":59,"value":1367},{"type":53,"tag":614,"props":2077,"children":2078},{"style":1349},[2079],{"type":59,"value":1464},{"type":53,"tag":614,"props":2081,"children":2082},{"style":1467},[2083],{"type":59,"value":2084}," 5",{"type":53,"tag":614,"props":2086,"children":2087},{"style":1349},[2088],{"type":59,"value":1475},{"type":53,"tag":614,"props":2090,"children":2091},{"style":1467},[2092],{"type":59,"value":2093}," 7",{"type":53,"tag":614,"props":2095,"children":2096},{"style":1349},[2097],{"type":59,"value":1485},{"type":53,"tag":66,"props":2099,"children":2100},{},[2101,2103,2108,2109,2114,2115,2120,2122,2127],{"type":59,"value":2102},"For each template variable (",{"type":53,"tag":74,"props":2104,"children":2106},{"className":2105},[],[2107],{"type":59,"value":441},{"type":59,"value":443},{"type":53,"tag":74,"props":2110,"children":2112},{"className":2111},[],[2113],{"type":59,"value":449},{"type":59,"value":443},{"type":53,"tag":74,"props":2116,"children":2118},{"className":2117},[],[2119],{"type":59,"value":456},{"type":59,"value":2121},"), find the matching JSON path. Common starting points — ",{"type":53,"tag":70,"props":2123,"children":2124},{},[2125],{"type":59,"value":2126},"always verify on your actual data before using",{"type":59,"value":2128},":",{"type":53,"tag":385,"props":2130,"children":2131},{},[2132,2153],{"type":53,"tag":389,"props":2133,"children":2134},{},[2135],{"type":53,"tag":393,"props":2136,"children":2137},{},[2138,2143,2148],{"type":53,"tag":397,"props":2139,"children":2140},{},[2141],{"type":59,"value":2142},"Template var",{"type":53,"tag":397,"props":2144,"children":2145},{},[2146],{"type":59,"value":2147},"LLM span",{"type":53,"tag":397,"props":2149,"children":2150},{},[2151],{"type":59,"value":2152},"CHAIN span",{"type":53,"tag":408,"props":2154,"children":2155},{},[2156,2184,2213,2239],{"type":53,"tag":393,"props":2157,"children":2158},{},[2159,2168,2176],{"type":53,"tag":415,"props":2160,"children":2161},{},[2162],{"type":53,"tag":74,"props":2163,"children":2165},{"className":2164},[],[2166],{"type":59,"value":2167},"input",{"type":53,"tag":415,"props":2169,"children":2170},{},[2171],{"type":53,"tag":74,"props":2172,"children":2174},{"className":2173},[],[2175],{"type":59,"value":1129},{"type":53,"tag":415,"props":2177,"children":2178},{},[2179],{"type":53,"tag":74,"props":2180,"children":2182},{"className":2181},[],[2183],{"type":59,"value":1129},{"type":53,"tag":393,"props":2185,"children":2186},{},[2187,2196,2205],{"type":53,"tag":415,"props":2188,"children":2189},{},[2190],{"type":53,"tag":74,"props":2191,"children":2193},{"className":2192},[],[2194],{"type":59,"value":2195},"output",{"type":53,"tag":415,"props":2197,"children":2198},{},[2199],{"type":53,"tag":74,"props":2200,"children":2202},{"className":2201},[],[2203],{"type":59,"value":2204},"attributes.llm.output_messages.0.message.content",{"type":53,"tag":415,"props":2206,"children":2207},{},[2208],{"type":53,"tag":74,"props":2209,"children":2211},{"className":2210},[],[2212],{"type":59,"value":1144},{"type":53,"tag":393,"props":2214,"children":2215},{},[2216,2225,2234],{"type":53,"tag":415,"props":2217,"children":2218},{},[2219],{"type":53,"tag":74,"props":2220,"children":2222},{"className":2221},[],[2223],{"type":59,"value":2224},"context",{"type":53,"tag":415,"props":2226,"children":2227},{},[2228],{"type":53,"tag":74,"props":2229,"children":2231},{"className":2230},[],[2232],{"type":59,"value":2233},"attributes.retrieval.documents.contents",{"type":53,"tag":415,"props":2235,"children":2236},{},[2237],{"type":59,"value":2238},"—",{"type":53,"tag":393,"props":2240,"children":2241},{},[2242,2251,2261],{"type":53,"tag":415,"props":2243,"children":2244},{},[2245],{"type":53,"tag":74,"props":2246,"children":2248},{"className":2247},[],[2249],{"type":59,"value":2250},"tool_output",{"type":53,"tag":415,"props":2252,"children":2253},{},[2254,2259],{"type":53,"tag":74,"props":2255,"children":2257},{"className":2256},[],[2258],{"type":59,"value":1129},{"type":59,"value":2260}," (fallback)",{"type":53,"tag":415,"props":2262,"children":2263},{},[2264],{"type":53,"tag":74,"props":2265,"children":2267},{"className":2266},[],[2268],{"type":59,"value":1144},{"type":53,"tag":66,"props":2270,"children":2271},{},[2272,2277,2279,2284],{"type":53,"tag":70,"props":2273,"children":2274},{},[2275],{"type":59,"value":2276},"Validate span kind alignment:",{"type":59,"value":2278}," If the evaluator prompt assumes LLM final text but the task targets CHAIN spans (or vice versa), runs can cancel or score the wrong text. Make sure the ",{"type":53,"tag":74,"props":2280,"children":2282},{"className":2281},[],[2283],{"type":59,"value":1246},{"type":59,"value":2285}," on the task matches the span kind you mapped.",{"type":53,"tag":66,"props":2287,"children":2288},{},[2289,2299,2301,2306,2308,2314,2316,2322,2324,2330],{"type":53,"tag":70,"props":2290,"children":2291},{},[2292,2297],{"type":53,"tag":74,"props":2293,"children":2295},{"className":2294},[],[2296],{"type":59,"value":1246},{"type":59,"value":2298}," only works on indexed attributes:",{"type":59,"value":2300}," The ",{"type":53,"tag":74,"props":2302,"children":2304},{"className":2303},[],[2305],{"type":59,"value":1246},{"type":59,"value":2307}," in the evaluators JSON is evaluated against the eval index, not the raw span store. Attributes under ",{"type":53,"tag":74,"props":2309,"children":2311},{"className":2310},[],[2312],{"type":59,"value":2313},"attributes.metadata.*",{"type":59,"value":2315}," or custom keys may not be indexed and will silently match nothing. Use well-known indexed attributes like ",{"type":53,"tag":74,"props":2317,"children":2319},{"className":2318},[],[2320],{"type":59,"value":2321},"span_kind",{"type":59,"value":2323}," or ",{"type":53,"tag":74,"props":2325,"children":2327},{"className":2326},[],[2328],{"type":59,"value":2329},"attributes.llm.model_name",{"type":59,"value":2331}," for filtering. If a filter returns 0 spans despite data existing, try removing the filter as a diagnostic step.",{"type":53,"tag":66,"props":2333,"children":2334},{},[2335],{"type":53,"tag":70,"props":2336,"children":2337},{},[2338,2340,2346],{"type":59,"value":2339},"Full example ",{"type":53,"tag":74,"props":2341,"children":2343},{"className":2342},[],[2344],{"type":59,"value":2345},"--evaluators",{"type":59,"value":2347}," JSON:",{"type":53,"tag":1327,"props":2349,"children":2353},{"className":2350,"code":2351,"language":2352,"meta":1332,"style":1332},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","[\n  {\n    \"evaluator_id\": \"EVAL_ID\",\n    \"query_filter\": \"span_kind = 'LLM'\",\n    \"column_mappings\": {\n      \"input\": \"attributes.input.value\",\n      \"output\": \"attributes.llm.output_messages.0.message.content\",\n      \"context\": \"attributes.retrieval.documents.contents\"\n    }\n  }\n]\n","json",[2354],{"type":53,"tag":74,"props":2355,"children":2356},{"__ignoreMap":1332},[2357,2365,2373,2413,2449,2473,2509,2544,2576,2584,2592],{"type":53,"tag":614,"props":2358,"children":2359},{"class":1339,"line":1340},[2360],{"type":53,"tag":614,"props":2361,"children":2362},{"style":1723},[2363],{"type":59,"value":2364},"[\n",{"type":53,"tag":614,"props":2366,"children":2367},{"class":1339,"line":1714},[2368],{"type":53,"tag":614,"props":2369,"children":2370},{"style":1723},[2371],{"type":59,"value":2372},"  {\n",{"type":53,"tag":614,"props":2374,"children":2375},{"class":1339,"line":1742},[2376,2381,2387,2391,2395,2399,2404,2408],{"type":53,"tag":614,"props":2377,"children":2378},{"style":1723},[2379],{"type":59,"value":2380},"    \"",{"type":53,"tag":614,"props":2382,"children":2384},{"style":2383},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[2385],{"type":59,"value":2386},"evaluator_id",{"type":53,"tag":614,"props":2388,"children":2389},{"style":1723},[2390],{"type":59,"value":1735},{"type":53,"tag":614,"props":2392,"children":2393},{"style":1723},[2394],{"type":59,"value":2128},{"type":53,"tag":614,"props":2396,"children":2397},{"style":1723},[2398],{"type":59,"value":1726},{"type":53,"tag":614,"props":2400,"children":2401},{"style":1349},[2402],{"type":59,"value":2403},"EVAL_ID",{"type":53,"tag":614,"props":2405,"children":2406},{"style":1723},[2407],{"type":59,"value":1735},{"type":53,"tag":614,"props":2409,"children":2410},{"style":1723},[2411],{"type":59,"value":2412},",\n",{"type":53,"tag":614,"props":2414,"children":2415},{"class":1339,"line":1759},[2416,2420,2424,2428,2432,2436,2441,2445],{"type":53,"tag":614,"props":2417,"children":2418},{"style":1723},[2419],{"type":59,"value":2380},{"type":53,"tag":614,"props":2421,"children":2422},{"style":2383},[2423],{"type":59,"value":1246},{"type":53,"tag":614,"props":2425,"children":2426},{"style":1723},[2427],{"type":59,"value":1735},{"type":53,"tag":614,"props":2429,"children":2430},{"style":1723},[2431],{"type":59,"value":2128},{"type":53,"tag":614,"props":2433,"children":2434},{"style":1723},[2435],{"type":59,"value":1726},{"type":53,"tag":614,"props":2437,"children":2438},{"style":1349},[2439],{"type":59,"value":2440},"span_kind = 'LLM'",{"type":53,"tag":614,"props":2442,"children":2443},{"style":1723},[2444],{"type":59,"value":1735},{"type":53,"tag":614,"props":2446,"children":2447},{"style":1723},[2448],{"type":59,"value":2412},{"type":53,"tag":614,"props":2450,"children":2451},{"class":1339,"line":27},[2452,2456,2460,2464,2468],{"type":53,"tag":614,"props":2453,"children":2454},{"style":1723},[2455],{"type":59,"value":2380},{"type":53,"tag":614,"props":2457,"children":2458},{"style":2383},[2459],{"type":59,"value":1680},{"type":53,"tag":614,"props":2461,"children":2462},{"style":1723},[2463],{"type":59,"value":1735},{"type":53,"tag":614,"props":2465,"children":2466},{"style":1723},[2467],{"type":59,"value":2128},{"type":53,"tag":614,"props":2469,"children":2470},{"style":1723},[2471],{"type":59,"value":2472}," {\n",{"type":53,"tag":614,"props":2474,"children":2475},{"class":1339,"line":1810},[2476,2481,2485,2489,2493,2497,2501,2505],{"type":53,"tag":614,"props":2477,"children":2478},{"style":1723},[2479],{"type":59,"value":2480},"      \"",{"type":53,"tag":614,"props":2482,"children":2483},{"style":1344},[2484],{"type":59,"value":2167},{"type":53,"tag":614,"props":2486,"children":2487},{"style":1723},[2488],{"type":59,"value":1735},{"type":53,"tag":614,"props":2490,"children":2491},{"style":1723},[2492],{"type":59,"value":2128},{"type":53,"tag":614,"props":2494,"children":2495},{"style":1723},[2496],{"type":59,"value":1726},{"type":53,"tag":614,"props":2498,"children":2499},{"style":1349},[2500],{"type":59,"value":1129},{"type":53,"tag":614,"props":2502,"children":2503},{"style":1723},[2504],{"type":59,"value":1735},{"type":53,"tag":614,"props":2506,"children":2507},{"style":1723},[2508],{"type":59,"value":2412},{"type":53,"tag":614,"props":2510,"children":2511},{"class":1339,"line":1828},[2512,2516,2520,2524,2528,2532,2536,2540],{"type":53,"tag":614,"props":2513,"children":2514},{"style":1723},[2515],{"type":59,"value":2480},{"type":53,"tag":614,"props":2517,"children":2518},{"style":1344},[2519],{"type":59,"value":2195},{"type":53,"tag":614,"props":2521,"children":2522},{"style":1723},[2523],{"type":59,"value":1735},{"type":53,"tag":614,"props":2525,"children":2526},{"style":1723},[2527],{"type":59,"value":2128},{"type":53,"tag":614,"props":2529,"children":2530},{"style":1723},[2531],{"type":59,"value":1726},{"type":53,"tag":614,"props":2533,"children":2534},{"style":1349},[2535],{"type":59,"value":2204},{"type":53,"tag":614,"props":2537,"children":2538},{"style":1723},[2539],{"type":59,"value":1735},{"type":53,"tag":614,"props":2541,"children":2542},{"style":1723},[2543],{"type":59,"value":2412},{"type":53,"tag":614,"props":2545,"children":2546},{"class":1339,"line":1853},[2547,2551,2555,2559,2563,2567,2571],{"type":53,"tag":614,"props":2548,"children":2549},{"style":1723},[2550],{"type":59,"value":2480},{"type":53,"tag":614,"props":2552,"children":2553},{"style":1344},[2554],{"type":59,"value":2224},{"type":53,"tag":614,"props":2556,"children":2557},{"style":1723},[2558],{"type":59,"value":1735},{"type":53,"tag":614,"props":2560,"children":2561},{"style":1723},[2562],{"type":59,"value":2128},{"type":53,"tag":614,"props":2564,"children":2565},{"style":1723},[2566],{"type":59,"value":1726},{"type":53,"tag":614,"props":2568,"children":2569},{"style":1349},[2570],{"type":59,"value":2233},{"type":53,"tag":614,"props":2572,"children":2573},{"style":1723},[2574],{"type":59,"value":2575},"\"\n",{"type":53,"tag":614,"props":2577,"children":2578},{"class":1339,"line":1866},[2579],{"type":53,"tag":614,"props":2580,"children":2581},{"style":1723},[2582],{"type":59,"value":2583},"    }\n",{"type":53,"tag":614,"props":2585,"children":2586},{"class":1339,"line":1879},[2587],{"type":53,"tag":614,"props":2588,"children":2589},{"style":1723},[2590],{"type":59,"value":2591},"  }\n",{"type":53,"tag":614,"props":2593,"children":2594},{"class":1339,"line":1907},[2595],{"type":53,"tag":614,"props":2596,"children":2597},{"style":1723},[2598],{"type":59,"value":2599},"]\n",{"type":53,"tag":66,"props":2601,"children":2602},{},[2603,2605,2610],{"type":59,"value":2604},"Include a mapping for ",{"type":53,"tag":70,"props":2606,"children":2607},{},[2608],{"type":59,"value":2609},"every",{"type":59,"value":2611}," variable the template references. Omitting one causes runs to produce no valid scores.",{"type":53,"tag":366,"props":2613,"children":2615},{"id":2614},"step-7-create-the-task",[2616],{"type":59,"value":2617},"Step 7: Create the task",{"type":53,"tag":66,"props":2619,"children":2620},{},[2621],{"type":53,"tag":70,"props":2622,"children":2623},{},[2624],{"type":59,"value":2625},"Backfill only (a):",{"type":53,"tag":1327,"props":2627,"children":2629},{"className":1329,"code":2628,"language":1331,"meta":1332,"style":1332},"ax tasks create-evaluation \\\n  --name \"Hallucination Backfill\" \\\n  --task-type TEMPLATE_EVALUATION \\\n  --project PROJECT \\\n  --evaluators '[{\"evaluator_id\": \"EVAL_ID\", \"column_mappings\": {\"input\": \"attributes.input.value\", \"output\": \"attributes.output.value\"}}]' \\\n  --no-continuous\n",[2630],{"type":53,"tag":74,"props":2631,"children":2632},{"__ignoreMap":1332},[2633,2654,2678,2695,2711,2736],{"type":53,"tag":614,"props":2634,"children":2635},{"class":1339,"line":1340},[2636,2640,2645,2650],{"type":53,"tag":614,"props":2637,"children":2638},{"style":1344},[2639],{"type":59,"value":189},{"type":53,"tag":614,"props":2641,"children":2642},{"style":1349},[2643],{"type":59,"value":2644}," tasks",{"type":53,"tag":614,"props":2646,"children":2647},{"style":1349},[2648],{"type":59,"value":2649}," create-evaluation",{"type":53,"tag":614,"props":2651,"children":2652},{"style":1708},[2653],{"type":59,"value":1711},{"type":53,"tag":614,"props":2655,"children":2656},{"class":1339,"line":1714},[2657,2661,2665,2670,2674],{"type":53,"tag":614,"props":2658,"children":2659},{"style":1349},[2660],{"type":59,"value":1720},{"type":53,"tag":614,"props":2662,"children":2663},{"style":1723},[2664],{"type":59,"value":1726},{"type":53,"tag":614,"props":2666,"children":2667},{"style":1349},[2668],{"type":59,"value":2669},"Hallucination Backfill",{"type":53,"tag":614,"props":2671,"children":2672},{"style":1723},[2673],{"type":59,"value":1735},{"type":53,"tag":614,"props":2675,"children":2676},{"style":1708},[2677],{"type":59,"value":1711},{"type":53,"tag":614,"props":2679,"children":2680},{"class":1339,"line":1742},[2681,2686,2691],{"type":53,"tag":614,"props":2682,"children":2683},{"style":1349},[2684],{"type":59,"value":2685},"  --task-type",{"type":53,"tag":614,"props":2687,"children":2688},{"style":1349},[2689],{"type":59,"value":2690}," TEMPLATE_EVALUATION",{"type":53,"tag":614,"props":2692,"children":2693},{"style":1708},[2694],{"type":59,"value":1711},{"type":53,"tag":614,"props":2696,"children":2697},{"class":1339,"line":1759},[2698,2703,2707],{"type":53,"tag":614,"props":2699,"children":2700},{"style":1349},[2701],{"type":59,"value":2702},"  --project",{"type":53,"tag":614,"props":2704,"children":2705},{"style":1349},[2706],{"type":59,"value":1451},{"type":53,"tag":614,"props":2708,"children":2709},{"style":1708},[2710],{"type":59,"value":1711},{"type":53,"tag":614,"props":2712,"children":2713},{"class":1339,"line":27},[2714,2719,2723,2728,2732],{"type":53,"tag":614,"props":2715,"children":2716},{"style":1349},[2717],{"type":59,"value":2718},"  --evaluators",{"type":53,"tag":614,"props":2720,"children":2721},{"style":1723},[2722],{"type":59,"value":1890},{"type":53,"tag":614,"props":2724,"children":2725},{"style":1349},[2726],{"type":59,"value":2727},"[{\"evaluator_id\": \"EVAL_ID\", \"column_mappings\": {\"input\": \"attributes.input.value\", \"output\": \"attributes.output.value\"}}]",{"type":53,"tag":614,"props":2729,"children":2730},{"style":1723},[2731],{"type":59,"value":1900},{"type":53,"tag":614,"props":2733,"children":2734},{"style":1708},[2735],{"type":59,"value":1711},{"type":53,"tag":614,"props":2737,"children":2738},{"class":1339,"line":1810},[2739],{"type":53,"tag":614,"props":2740,"children":2741},{"style":1349},[2742],{"type":59,"value":2743},"  --no-continuous\n",{"type":53,"tag":66,"props":2745,"children":2746},{},[2747],{"type":53,"tag":70,"props":2748,"children":2749},{},[2750],{"type":59,"value":2751},"Continuous only (b):",{"type":53,"tag":1327,"props":2753,"children":2755},{"className":1329,"code":2754,"language":1331,"meta":1332,"style":1332},"ax tasks create-evaluation \\\n  --name \"Hallucination Monitor\" \\\n  --task-type TEMPLATE_EVALUATION \\\n  --project PROJECT \\\n  --evaluators '[{\"evaluator_id\": \"EVAL_ID\", \"column_mappings\": {\"input\": \"attributes.input.value\", \"output\": \"attributes.output.value\"}}]' \\\n  --is-continuous \\\n  --sampling-rate 0.1\n",[2756],{"type":53,"tag":74,"props":2757,"children":2758},{"__ignoreMap":1332},[2759,2778,2802,2817,2832,2855,2867],{"type":53,"tag":614,"props":2760,"children":2761},{"class":1339,"line":1340},[2762,2766,2770,2774],{"type":53,"tag":614,"props":2763,"children":2764},{"style":1344},[2765],{"type":59,"value":189},{"type":53,"tag":614,"props":2767,"children":2768},{"style":1349},[2769],{"type":59,"value":2644},{"type":53,"tag":614,"props":2771,"children":2772},{"style":1349},[2773],{"type":59,"value":2649},{"type":53,"tag":614,"props":2775,"children":2776},{"style":1708},[2777],{"type":59,"value":1711},{"type":53,"tag":614,"props":2779,"children":2780},{"class":1339,"line":1714},[2781,2785,2789,2794,2798],{"type":53,"tag":614,"props":2782,"children":2783},{"style":1349},[2784],{"type":59,"value":1720},{"type":53,"tag":614,"props":2786,"children":2787},{"style":1723},[2788],{"type":59,"value":1726},{"type":53,"tag":614,"props":2790,"children":2791},{"style":1349},[2792],{"type":59,"value":2793},"Hallucination Monitor",{"type":53,"tag":614,"props":2795,"children":2796},{"style":1723},[2797],{"type":59,"value":1735},{"type":53,"tag":614,"props":2799,"children":2800},{"style":1708},[2801],{"type":59,"value":1711},{"type":53,"tag":614,"props":2803,"children":2804},{"class":1339,"line":1742},[2805,2809,2813],{"type":53,"tag":614,"props":2806,"children":2807},{"style":1349},[2808],{"type":59,"value":2685},{"type":53,"tag":614,"props":2810,"children":2811},{"style":1349},[2812],{"type":59,"value":2690},{"type":53,"tag":614,"props":2814,"children":2815},{"style":1708},[2816],{"type":59,"value":1711},{"type":53,"tag":614,"props":2818,"children":2819},{"class":1339,"line":1759},[2820,2824,2828],{"type":53,"tag":614,"props":2821,"children":2822},{"style":1349},[2823],{"type":59,"value":2702},{"type":53,"tag":614,"props":2825,"children":2826},{"style":1349},[2827],{"type":59,"value":1451},{"type":53,"tag":614,"props":2829,"children":2830},{"style":1708},[2831],{"type":59,"value":1711},{"type":53,"tag":614,"props":2833,"children":2834},{"class":1339,"line":27},[2835,2839,2843,2847,2851],{"type":53,"tag":614,"props":2836,"children":2837},{"style":1349},[2838],{"type":59,"value":2718},{"type":53,"tag":614,"props":2840,"children":2841},{"style":1723},[2842],{"type":59,"value":1890},{"type":53,"tag":614,"props":2844,"children":2845},{"style":1349},[2846],{"type":59,"value":2727},{"type":53,"tag":614,"props":2848,"children":2849},{"style":1723},[2850],{"type":59,"value":1900},{"type":53,"tag":614,"props":2852,"children":2853},{"style":1708},[2854],{"type":59,"value":1711},{"type":53,"tag":614,"props":2856,"children":2857},{"class":1339,"line":1810},[2858,2863],{"type":53,"tag":614,"props":2859,"children":2860},{"style":1349},[2861],{"type":59,"value":2862},"  --is-continuous",{"type":53,"tag":614,"props":2864,"children":2865},{"style":1708},[2866],{"type":59,"value":1711},{"type":53,"tag":614,"props":2868,"children":2869},{"class":1339,"line":1828},[2870,2875],{"type":53,"tag":614,"props":2871,"children":2872},{"style":1349},[2873],{"type":59,"value":2874},"  --sampling-rate",{"type":53,"tag":614,"props":2876,"children":2877},{"style":1467},[2878],{"type":59,"value":2879}," 0.1\n",{"type":53,"tag":66,"props":2881,"children":2882},{},[2883,2888,2890,2896],{"type":53,"tag":70,"props":2884,"children":2885},{},[2886],{"type":59,"value":2887},"Both (c):",{"type":59,"value":2889}," Use ",{"type":53,"tag":74,"props":2891,"children":2893},{"className":2892},[],[2894],{"type":59,"value":2895},"--is-continuous",{"type":59,"value":2897}," on create, then also trigger a backfill run in Step 8.",{"type":53,"tag":366,"props":2899,"children":2901},{"id":2900},"step-8-trigger-a-backfill-run-if-requested",[2902],{"type":59,"value":2903},"Step 8: Trigger a backfill run (if requested)",{"type":53,"tag":62,"props":2905,"children":2906},{},[2907],{"type":53,"tag":66,"props":2908,"children":2909},{},[2910,2915,2917,2922,2924,2930],{"type":53,"tag":70,"props":2911,"children":2912},{},[2913],{"type":59,"value":2914},"Eval index lag:",{"type":59,"value":2916}," The eval index is built asynchronously from the primary trace store and can lag ",{"type":53,"tag":70,"props":2918,"children":2919},{},[2920],{"type":59,"value":2921},"1–2 hours",{"type":59,"value":2923},". For your first test run, use a time window ending at least 2 hours in the past. If you set ",{"type":53,"tag":74,"props":2925,"children":2927},{"className":2926},[],[2928],{"type":59,"value":2929},"--data-end-time",{"type":59,"value":2931}," to \"now\" on spans ingested in the last hour, the run will complete successfully but score 0 spans.",{"type":53,"tag":66,"props":2933,"children":2934},{},[2935],{"type":59,"value":2936},"First find what time range has data:",{"type":53,"tag":1327,"props":2938,"children":2940},{"className":1329,"code":2939,"language":1331,"meta":1332,"style":1332},"ax spans export PROJECT --space SPACE -l 100 --days 1 --stdout   # try last 24h first\nax spans export PROJECT --space SPACE -l 100 --days 7 --stdout   # widen if empty\n",[2941],{"type":53,"tag":74,"props":2942,"children":2943},{"__ignoreMap":1332},[2944,3000],{"type":53,"tag":614,"props":2945,"children":2946},{"class":1339,"line":1340},[2947,2951,2955,2959,2963,2967,2971,2975,2980,2984,2989,2994],{"type":53,"tag":614,"props":2948,"children":2949},{"style":1344},[2950],{"type":59,"value":189},{"type":53,"tag":614,"props":2952,"children":2953},{"style":1349},[2954],{"type":59,"value":1441},{"type":53,"tag":614,"props":2956,"children":2957},{"style":1349},[2958],{"type":59,"value":1446},{"type":53,"tag":614,"props":2960,"children":2961},{"style":1349},[2962],{"type":59,"value":1451},{"type":53,"tag":614,"props":2964,"children":2965},{"style":1349},[2966],{"type":59,"value":1362},{"type":53,"tag":614,"props":2968,"children":2969},{"style":1349},[2970],{"type":59,"value":1367},{"type":53,"tag":614,"props":2972,"children":2973},{"style":1349},[2974],{"type":59,"value":1464},{"type":53,"tag":614,"props":2976,"children":2977},{"style":1467},[2978],{"type":59,"value":2979}," 100",{"type":53,"tag":614,"props":2981,"children":2982},{"style":1349},[2983],{"type":59,"value":1475},{"type":53,"tag":614,"props":2985,"children":2986},{"style":1467},[2987],{"type":59,"value":2988}," 1",{"type":53,"tag":614,"props":2990,"children":2991},{"style":1349},[2992],{"type":59,"value":2993}," --stdout",{"type":53,"tag":614,"props":2995,"children":2997},{"style":2996},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[2998],{"type":59,"value":2999},"   # try last 24h first\n",{"type":53,"tag":614,"props":3001,"children":3002},{"class":1339,"line":1714},[3003,3007,3011,3015,3019,3023,3027,3031,3035,3039,3043,3047],{"type":53,"tag":614,"props":3004,"children":3005},{"style":1344},[3006],{"type":59,"value":189},{"type":53,"tag":614,"props":3008,"children":3009},{"style":1349},[3010],{"type":59,"value":1441},{"type":53,"tag":614,"props":3012,"children":3013},{"style":1349},[3014],{"type":59,"value":1446},{"type":53,"tag":614,"props":3016,"children":3017},{"style":1349},[3018],{"type":59,"value":1451},{"type":53,"tag":614,"props":3020,"children":3021},{"style":1349},[3022],{"type":59,"value":1362},{"type":53,"tag":614,"props":3024,"children":3025},{"style":1349},[3026],{"type":59,"value":1367},{"type":53,"tag":614,"props":3028,"children":3029},{"style":1349},[3030],{"type":59,"value":1464},{"type":53,"tag":614,"props":3032,"children":3033},{"style":1467},[3034],{"type":59,"value":2979},{"type":53,"tag":614,"props":3036,"children":3037},{"style":1349},[3038],{"type":59,"value":1475},{"type":53,"tag":614,"props":3040,"children":3041},{"style":1467},[3042],{"type":59,"value":2093},{"type":53,"tag":614,"props":3044,"children":3045},{"style":1349},[3046],{"type":59,"value":2993},{"type":53,"tag":614,"props":3048,"children":3049},{"style":2996},[3050],{"type":59,"value":3051},"   # widen if empty\n",{"type":53,"tag":66,"props":3053,"children":3054},{},[3055,3057,3062,3063,3069,3071,3077],{"type":59,"value":3056},"Use the ",{"type":53,"tag":74,"props":3058,"children":3060},{"className":3059},[],[3061],{"type":59,"value":1081},{"type":59,"value":482},{"type":53,"tag":74,"props":3064,"children":3066},{"className":3065},[],[3067],{"type":59,"value":3068},"end_time",{"type":59,"value":3070}," fields from real spans to set the window. For the first validation run, cap ",{"type":53,"tag":74,"props":3072,"children":3074},{"className":3073},[],[3075],{"type":59,"value":3076},"--max-spans",{"type":59,"value":3078}," at ~100 to get quick feedback:",{"type":53,"tag":1327,"props":3080,"children":3082},{"className":1329,"code":3081,"language":1331,"meta":1332,"style":1332},"ax tasks trigger-run TASK_ID \\\n  --data-start-time \"2026-03-20T00:00:00\" \\\n  --data-end-time \"2026-03-21T23:59:59\" \\\n  --max-spans 100 \\\n  --wait\n",[3083],{"type":53,"tag":74,"props":3084,"children":3085},{"__ignoreMap":1332},[3086,3111,3136,3161,3177],{"type":53,"tag":614,"props":3087,"children":3088},{"class":1339,"line":1340},[3089,3093,3097,3102,3107],{"type":53,"tag":614,"props":3090,"children":3091},{"style":1344},[3092],{"type":59,"value":189},{"type":53,"tag":614,"props":3094,"children":3095},{"style":1349},[3096],{"type":59,"value":2644},{"type":53,"tag":614,"props":3098,"children":3099},{"style":1349},[3100],{"type":59,"value":3101}," trigger-run",{"type":53,"tag":614,"props":3103,"children":3104},{"style":1349},[3105],{"type":59,"value":3106}," TASK_ID",{"type":53,"tag":614,"props":3108,"children":3109},{"style":1708},[3110],{"type":59,"value":1711},{"type":53,"tag":614,"props":3112,"children":3113},{"class":1339,"line":1714},[3114,3119,3123,3128,3132],{"type":53,"tag":614,"props":3115,"children":3116},{"style":1349},[3117],{"type":59,"value":3118},"  --data-start-time",{"type":53,"tag":614,"props":3120,"children":3121},{"style":1723},[3122],{"type":59,"value":1726},{"type":53,"tag":614,"props":3124,"children":3125},{"style":1349},[3126],{"type":59,"value":3127},"2026-03-20T00:00:00",{"type":53,"tag":614,"props":3129,"children":3130},{"style":1723},[3131],{"type":59,"value":1735},{"type":53,"tag":614,"props":3133,"children":3134},{"style":1708},[3135],{"type":59,"value":1711},{"type":53,"tag":614,"props":3137,"children":3138},{"class":1339,"line":1742},[3139,3144,3148,3153,3157],{"type":53,"tag":614,"props":3140,"children":3141},{"style":1349},[3142],{"type":59,"value":3143},"  --data-end-time",{"type":53,"tag":614,"props":3145,"children":3146},{"style":1723},[3147],{"type":59,"value":1726},{"type":53,"tag":614,"props":3149,"children":3150},{"style":1349},[3151],{"type":59,"value":3152},"2026-03-21T23:59:59",{"type":53,"tag":614,"props":3154,"children":3155},{"style":1723},[3156],{"type":59,"value":1735},{"type":53,"tag":614,"props":3158,"children":3159},{"style":1708},[3160],{"type":59,"value":1711},{"type":53,"tag":614,"props":3162,"children":3163},{"class":1339,"line":1759},[3164,3169,3173],{"type":53,"tag":614,"props":3165,"children":3166},{"style":1349},[3167],{"type":59,"value":3168},"  --max-spans",{"type":53,"tag":614,"props":3170,"children":3171},{"style":1467},[3172],{"type":59,"value":2979},{"type":53,"tag":614,"props":3174,"children":3175},{"style":1708},[3176],{"type":59,"value":1711},{"type":53,"tag":614,"props":3178,"children":3179},{"class":1339,"line":27},[3180],{"type":53,"tag":614,"props":3181,"children":3182},{"style":1349},[3183],{"type":59,"value":3184},"  --wait\n",{"type":53,"tag":66,"props":3186,"children":3187},{},[3188],{"type":59,"value":3189},"Review scores and explanations before widening to the full backfill or enabling continuous.",{"type":53,"tag":169,"props":3191,"children":3192},{},[],{"type":53,"tag":173,"props":3194,"children":3196},{"id":3195},"workflow-b-create-an-evaluator-for-an-experiment",[3197],{"type":59,"value":3198},"Workflow B: Create an evaluator for an experiment",{"type":53,"tag":66,"props":3200,"children":3201},{},[3202,3203,3208,3209,3214],{"type":59,"value":1301},{"type":53,"tag":1303,"props":3204,"children":3205},{},[3206],{"type":59,"value":3207},"\"create an evaluator for my experiment\"",{"type":59,"value":2323},{"type":53,"tag":1303,"props":3210,"children":3211},{},[3212],{"type":59,"value":3213},"\"evaluate my dataset runs\"",{"type":59,"value":134},{"type":53,"tag":66,"props":3216,"children":3217},{},[3218,3223],{"type":53,"tag":70,"props":3219,"children":3220},{},[3221],{"type":59,"value":3222},"If the user says \"dataset\" but doesn't have an experiment:",{"type":59,"value":3224}," A task must target an experiment (not a bare dataset). Ask:",{"type":53,"tag":62,"props":3226,"children":3227},{},[3228],{"type":53,"tag":66,"props":3229,"children":3230},{},[3231],{"type":59,"value":3232},"\"Evaluation tasks run against experiment runs, not datasets directly. Would you like help creating an experiment on that dataset first?\"",{"type":53,"tag":66,"props":3234,"children":3235},{},[3236,3238,3243],{"type":59,"value":3237},"If yes, use the ",{"type":53,"tag":70,"props":3239,"children":3240},{},[3241],{"type":59,"value":3242},"arize-experiment",{"type":59,"value":3244}," skill to create one, then return here.",{"type":53,"tag":366,"props":3246,"children":3248},{"id":3247},"step-1-find-the-dataset-and-experiment-names",[3249],{"type":59,"value":3250},"Step 1: Find the dataset and experiment names",{"type":53,"tag":1327,"props":3252,"children":3254},{"className":1329,"code":3253,"language":1331,"meta":1332,"style":1332},"ax datasets list --space SPACE\nax experiments list --dataset DATASET_NAME --space SPACE -o json\n",[3255],{"type":53,"tag":74,"props":3256,"children":3257},{"__ignoreMap":1332},[3258,3283],{"type":53,"tag":614,"props":3259,"children":3260},{"class":1339,"line":1340},[3261,3265,3270,3274,3278],{"type":53,"tag":614,"props":3262,"children":3263},{"style":1344},[3264],{"type":59,"value":189},{"type":53,"tag":614,"props":3266,"children":3267},{"style":1349},[3268],{"type":59,"value":3269}," datasets",{"type":53,"tag":614,"props":3271,"children":3272},{"style":1349},[3273],{"type":59,"value":1357},{"type":53,"tag":614,"props":3275,"children":3276},{"style":1349},[3277],{"type":59,"value":1362},{"type":53,"tag":614,"props":3279,"children":3280},{"style":1349},[3281],{"type":59,"value":3282}," SPACE\n",{"type":53,"tag":614,"props":3284,"children":3285},{"class":1339,"line":1714},[3286,3290,3295,3299,3304,3309,3313,3317,3321],{"type":53,"tag":614,"props":3287,"children":3288},{"style":1344},[3289],{"type":59,"value":189},{"type":53,"tag":614,"props":3291,"children":3292},{"style":1349},[3293],{"type":59,"value":3294}," experiments",{"type":53,"tag":614,"props":3296,"children":3297},{"style":1349},[3298],{"type":59,"value":1357},{"type":53,"tag":614,"props":3300,"children":3301},{"style":1349},[3302],{"type":59,"value":3303}," --dataset",{"type":53,"tag":614,"props":3305,"children":3306},{"style":1349},[3307],{"type":59,"value":3308}," DATASET_NAME",{"type":53,"tag":614,"props":3310,"children":3311},{"style":1349},[3312],{"type":59,"value":1362},{"type":53,"tag":614,"props":3314,"children":3315},{"style":1349},[3316],{"type":59,"value":1367},{"type":53,"tag":614,"props":3318,"children":3319},{"style":1349},[3320],{"type":59,"value":1372},{"type":53,"tag":614,"props":3322,"children":3323},{"style":1349},[3324],{"type":59,"value":1377},{"type":53,"tag":66,"props":3326,"children":3327},{},[3328],{"type":59,"value":3329},"Note the dataset name and the experiment name(s) to score. These accept names or IDs in subsequent commands — names are preferred.",{"type":53,"tag":366,"props":3331,"children":3333},{"id":3332},"step-2-understand-what-to-evaluate-1",[3334],{"type":59,"value":1412},{"type":53,"tag":66,"props":3336,"children":3337},{},[3338],{"type":59,"value":3339},"If the user specified the evaluator type → skip to Step 3.",{"type":53,"tag":66,"props":3341,"children":3342},{},[3343],{"type":59,"value":3344},"If not, inspect a recent experiment run to base the evaluator on actual data:",{"type":53,"tag":1327,"props":3346,"children":3348},{"className":1329,"code":3347,"language":1331,"meta":1332,"style":1332},"ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE --stdout | python3 -c \"import sys,json; runs=json.load(sys.stdin); print(json.dumps(runs[0], indent=2))\"\n",[3349],{"type":53,"tag":74,"props":3350,"children":3351},{"__ignoreMap":1332},[3352],{"type":53,"tag":614,"props":3353,"children":3354},{"class":1339,"line":1340},[3355,3359,3363,3367,3372,3376,3380,3384,3388,3392,3397,3402,3407,3411,3416],{"type":53,"tag":614,"props":3356,"children":3357},{"style":1344},[3358],{"type":59,"value":189},{"type":53,"tag":614,"props":3360,"children":3361},{"style":1349},[3362],{"type":59,"value":3294},{"type":53,"tag":614,"props":3364,"children":3365},{"style":1349},[3366],{"type":59,"value":1446},{"type":53,"tag":614,"props":3368,"children":3369},{"style":1349},[3370],{"type":59,"value":3371}," EXPERIMENT_NAME",{"type":53,"tag":614,"props":3373,"children":3374},{"style":1349},[3375],{"type":59,"value":3303},{"type":53,"tag":614,"props":3377,"children":3378},{"style":1349},[3379],{"type":59,"value":3308},{"type":53,"tag":614,"props":3381,"children":3382},{"style":1349},[3383],{"type":59,"value":1362},{"type":53,"tag":614,"props":3385,"children":3386},{"style":1349},[3387],{"type":59,"value":1367},{"type":53,"tag":614,"props":3389,"children":3390},{"style":1349},[3391],{"type":59,"value":2993},{"type":53,"tag":614,"props":3393,"children":3394},{"style":1723},[3395],{"type":59,"value":3396}," |",{"type":53,"tag":614,"props":3398,"children":3399},{"style":1344},[3400],{"type":59,"value":3401}," python3",{"type":53,"tag":614,"props":3403,"children":3404},{"style":1349},[3405],{"type":59,"value":3406}," -c",{"type":53,"tag":614,"props":3408,"children":3409},{"style":1723},[3410],{"type":59,"value":1726},{"type":53,"tag":614,"props":3412,"children":3413},{"style":1349},[3414],{"type":59,"value":3415},"import sys,json; runs=json.load(sys.stdin); print(json.dumps(runs[0], indent=2))",{"type":53,"tag":614,"props":3417,"children":3418},{"style":1723},[3419],{"type":59,"value":2575},{"type":53,"tag":66,"props":3421,"children":3422},{},[3423,3425,3430,3431,3436,3437,3443,3445,3451,3453,3458],{"type":59,"value":3424},"Look at the ",{"type":53,"tag":74,"props":3426,"children":3428},{"className":3427},[],[3429],{"type":59,"value":2195},{"type":59,"value":443},{"type":53,"tag":74,"props":3432,"children":3434},{"className":3433},[],[3435],{"type":59,"value":2167},{"type":59,"value":443},{"type":53,"tag":74,"props":3438,"children":3440},{"className":3439},[],[3441],{"type":59,"value":3442},"evaluations",{"type":59,"value":3444},", and ",{"type":53,"tag":74,"props":3446,"children":3448},{"className":3447},[],[3449],{"type":59,"value":3450},"metadata",{"type":59,"value":3452}," fields. Identify gaps (metrics the user cares about but doesn't have yet) and propose ",{"type":53,"tag":70,"props":3454,"children":3455},{},[3456],{"type":59,"value":3457},"1–3 evaluator ideas",{"type":59,"value":3459},". Each suggestion must include: the evaluator name (bold), a one-sentence description, and the binary label pair in parentheses — same format as Workflow A, Step 2.",{"type":53,"tag":366,"props":3461,"children":3463},{"id":3462},"step-3-confirm-or-create-an-ai-integration-1",[3464],{"type":59,"value":1606},{"type":53,"tag":66,"props":3466,"children":3467},{},[3468],{"type":59,"value":3469},"Same as Workflow A, Step 3.",{"type":53,"tag":366,"props":3471,"children":3473},{"id":3472},"step-4-create-the-evaluator-1",[3474],{"type":59,"value":1662},{"type":53,"tag":66,"props":3476,"children":3477},{},[3478],{"type":59,"value":3479},"Same as Workflow A, Step 4. Keep variables generic.",{"type":53,"tag":366,"props":3481,"children":3483},{"id":3482},"step-5-determine-column-mappings-from-real-run-data",[3484],{"type":59,"value":3485},"Step 5: Determine column mappings from real run data",{"type":53,"tag":66,"props":3487,"children":3488},{},[3489],{"type":59,"value":3490},"Run data shape differs from span data. Inspect:",{"type":53,"tag":1327,"props":3492,"children":3493},{"className":1329,"code":3347,"language":1331,"meta":1332,"style":1332},[3494],{"type":53,"tag":74,"props":3495,"children":3496},{"__ignoreMap":1332},[3497],{"type":53,"tag":614,"props":3498,"children":3499},{"class":1339,"line":1340},[3500,3504,3508,3512,3516,3520,3524,3528,3532,3536,3540,3544,3548,3552,3556],{"type":53,"tag":614,"props":3501,"children":3502},{"style":1344},[3503],{"type":59,"value":189},{"type":53,"tag":614,"props":3505,"children":3506},{"style":1349},[3507],{"type":59,"value":3294},{"type":53,"tag":614,"props":3509,"children":3510},{"style":1349},[3511],{"type":59,"value":1446},{"type":53,"tag":614,"props":3513,"children":3514},{"style":1349},[3515],{"type":59,"value":3371},{"type":53,"tag":614,"props":3517,"children":3518},{"style":1349},[3519],{"type":59,"value":3303},{"type":53,"tag":614,"props":3521,"children":3522},{"style":1349},[3523],{"type":59,"value":3308},{"type":53,"tag":614,"props":3525,"children":3526},{"style":1349},[3527],{"type":59,"value":1362},{"type":53,"tag":614,"props":3529,"children":3530},{"style":1349},[3531],{"type":59,"value":1367},{"type":53,"tag":614,"props":3533,"children":3534},{"style":1349},[3535],{"type":59,"value":2993},{"type":53,"tag":614,"props":3537,"children":3538},{"style":1723},[3539],{"type":59,"value":3396},{"type":53,"tag":614,"props":3541,"children":3542},{"style":1344},[3543],{"type":59,"value":3401},{"type":53,"tag":614,"props":3545,"children":3546},{"style":1349},[3547],{"type":59,"value":3406},{"type":53,"tag":614,"props":3549,"children":3550},{"style":1723},[3551],{"type":59,"value":1726},{"type":53,"tag":614,"props":3553,"children":3554},{"style":1349},[3555],{"type":59,"value":3415},{"type":53,"tag":614,"props":3557,"children":3558},{"style":1723},[3559],{"type":59,"value":2575},{"type":53,"tag":66,"props":3561,"children":3562},{},[3563],{"type":59,"value":3564},"Common mapping for experiment runs:",{"type":53,"tag":205,"props":3566,"children":3567},{},[3568,3586],{"type":53,"tag":209,"props":3569,"children":3570},{},[3571,3576,3578,3584],{"type":53,"tag":74,"props":3572,"children":3574},{"className":3573},[],[3575],{"type":59,"value":2195},{"type":59,"value":3577}," → ",{"type":53,"tag":74,"props":3579,"children":3581},{"className":3580},[],[3582],{"type":59,"value":3583},"\"output\"",{"type":59,"value":3585}," (top-level field on each run)",{"type":53,"tag":209,"props":3587,"children":3588},{},[3589,3594],{"type":53,"tag":74,"props":3590,"children":3592},{"className":3591},[],[3593],{"type":59,"value":2167},{"type":59,"value":3595}," → check if it's on the run or embedded in the linked dataset examples",{"type":53,"tag":66,"props":3597,"children":3598},{},[3599,3601,3606],{"type":59,"value":3600},"If ",{"type":53,"tag":74,"props":3602,"children":3604},{"className":3603},[],[3605],{"type":59,"value":2167},{"type":59,"value":3607}," is not on the run JSON, export dataset examples to find the path:",{"type":53,"tag":1327,"props":3609,"children":3611},{"className":1329,"code":3610,"language":1331,"meta":1332,"style":1332},"ax datasets export DATASET_NAME --space SPACE --stdout | python3 -c \"import sys,json; ex=json.load(sys.stdin); print(json.dumps(ex[0], indent=2))\"\n",[3612],{"type":53,"tag":74,"props":3613,"children":3614},{"__ignoreMap":1332},[3615],{"type":53,"tag":614,"props":3616,"children":3617},{"class":1339,"line":1340},[3618,3622,3626,3630,3634,3638,3642,3646,3650,3654,3658,3662,3667],{"type":53,"tag":614,"props":3619,"children":3620},{"style":1344},[3621],{"type":59,"value":189},{"type":53,"tag":614,"props":3623,"children":3624},{"style":1349},[3625],{"type":59,"value":3269},{"type":53,"tag":614,"props":3627,"children":3628},{"style":1349},[3629],{"type":59,"value":1446},{"type":53,"tag":614,"props":3631,"children":3632},{"style":1349},[3633],{"type":59,"value":3308},{"type":53,"tag":614,"props":3635,"children":3636},{"style":1349},[3637],{"type":59,"value":1362},{"type":53,"tag":614,"props":3639,"children":3640},{"style":1349},[3641],{"type":59,"value":1367},{"type":53,"tag":614,"props":3643,"children":3644},{"style":1349},[3645],{"type":59,"value":2993},{"type":53,"tag":614,"props":3647,"children":3648},{"style":1723},[3649],{"type":59,"value":3396},{"type":53,"tag":614,"props":3651,"children":3652},{"style":1344},[3653],{"type":59,"value":3401},{"type":53,"tag":614,"props":3655,"children":3656},{"style":1349},[3657],{"type":59,"value":3406},{"type":53,"tag":614,"props":3659,"children":3660},{"style":1723},[3661],{"type":59,"value":1726},{"type":53,"tag":614,"props":3663,"children":3664},{"style":1349},[3665],{"type":59,"value":3666},"import sys,json; ex=json.load(sys.stdin); print(json.dumps(ex[0], indent=2))",{"type":53,"tag":614,"props":3668,"children":3669},{"style":1723},[3670],{"type":59,"value":2575},{"type":53,"tag":366,"props":3672,"children":3674},{"id":3673},"step-6-create-the-task",[3675],{"type":59,"value":3676},"Step 6: Create the task",{"type":53,"tag":1327,"props":3678,"children":3680},{"className":1329,"code":3679,"language":1331,"meta":1332,"style":1332},"ax tasks create-evaluation \\\n  --name \"Experiment Correctness\" \\\n  --task-type TEMPLATE_EVALUATION \\\n  --dataset DATASET_NAME --space SPACE \\\n  --experiment-ids \"EXP_ID\" \\   # base64 ID from `ax experiments list --space SPACE -o json`\n  --evaluators '[{\"evaluator_id\": \"EVAL_ID\", \"column_mappings\": {\"output\": \"output\"}}]' \\\n  --no-continuous\n",[3681],{"type":53,"tag":74,"props":3682,"children":3683},{"__ignoreMap":1332},[3684,3703,3727,3742,3766,3797,3821],{"type":53,"tag":614,"props":3685,"children":3686},{"class":1339,"line":1340},[3687,3691,3695,3699],{"type":53,"tag":614,"props":3688,"children":3689},{"style":1344},[3690],{"type":59,"value":189},{"type":53,"tag":614,"props":3692,"children":3693},{"style":1349},[3694],{"type":59,"value":2644},{"type":53,"tag":614,"props":3696,"children":3697},{"style":1349},[3698],{"type":59,"value":2649},{"type":53,"tag":614,"props":3700,"children":3701},{"style":1708},[3702],{"type":59,"value":1711},{"type":53,"tag":614,"props":3704,"children":3705},{"class":1339,"line":1714},[3706,3710,3714,3719,3723],{"type":53,"tag":614,"props":3707,"children":3708},{"style":1349},[3709],{"type":59,"value":1720},{"type":53,"tag":614,"props":3711,"children":3712},{"style":1723},[3713],{"type":59,"value":1726},{"type":53,"tag":614,"props":3715,"children":3716},{"style":1349},[3717],{"type":59,"value":3718},"Experiment Correctness",{"type":53,"tag":614,"props":3720,"children":3721},{"style":1723},[3722],{"type":59,"value":1735},{"type":53,"tag":614,"props":3724,"children":3725},{"style":1708},[3726],{"type":59,"value":1711},{"type":53,"tag":614,"props":3728,"children":3729},{"class":1339,"line":1742},[3730,3734,3738],{"type":53,"tag":614,"props":3731,"children":3732},{"style":1349},[3733],{"type":59,"value":2685},{"type":53,"tag":614,"props":3735,"children":3736},{"style":1349},[3737],{"type":59,"value":2690},{"type":53,"tag":614,"props":3739,"children":3740},{"style":1708},[3741],{"type":59,"value":1711},{"type":53,"tag":614,"props":3743,"children":3744},{"class":1339,"line":1759},[3745,3750,3754,3758,3762],{"type":53,"tag":614,"props":3746,"children":3747},{"style":1349},[3748],{"type":59,"value":3749},"  --dataset",{"type":53,"tag":614,"props":3751,"children":3752},{"style":1349},[3753],{"type":59,"value":3308},{"type":53,"tag":614,"props":3755,"children":3756},{"style":1349},[3757],{"type":59,"value":1362},{"type":53,"tag":614,"props":3759,"children":3760},{"style":1349},[3761],{"type":59,"value":1367},{"type":53,"tag":614,"props":3763,"children":3764},{"style":1708},[3765],{"type":59,"value":1711},{"type":53,"tag":614,"props":3767,"children":3768},{"class":1339,"line":27},[3769,3774,3778,3783,3787,3792],{"type":53,"tag":614,"props":3770,"children":3771},{"style":1349},[3772],{"type":59,"value":3773},"  --experiment-ids",{"type":53,"tag":614,"props":3775,"children":3776},{"style":1723},[3777],{"type":59,"value":1726},{"type":53,"tag":614,"props":3779,"children":3780},{"style":1349},[3781],{"type":59,"value":3782},"EXP_ID",{"type":53,"tag":614,"props":3784,"children":3785},{"style":1723},[3786],{"type":59,"value":1735},{"type":53,"tag":614,"props":3788,"children":3789},{"style":1708},[3790],{"type":59,"value":3791}," \\   ",{"type":53,"tag":614,"props":3793,"children":3794},{"style":2996},[3795],{"type":59,"value":3796},"# base64 ID from `ax experiments list --space SPACE -o json`\n",{"type":53,"tag":614,"props":3798,"children":3799},{"class":1339,"line":1810},[3800,3804,3808,3813,3817],{"type":53,"tag":614,"props":3801,"children":3802},{"style":1344},[3803],{"type":59,"value":2718},{"type":53,"tag":614,"props":3805,"children":3806},{"style":1723},[3807],{"type":59,"value":1890},{"type":53,"tag":614,"props":3809,"children":3810},{"style":1349},[3811],{"type":59,"value":3812},"[{\"evaluator_id\": \"EVAL_ID\", \"column_mappings\": {\"output\": \"output\"}}]",{"type":53,"tag":614,"props":3814,"children":3815},{"style":1723},[3816],{"type":59,"value":1900},{"type":53,"tag":614,"props":3818,"children":3819},{"style":1708},[3820],{"type":59,"value":1711},{"type":53,"tag":614,"props":3822,"children":3823},{"class":1339,"line":1828},[3824],{"type":53,"tag":614,"props":3825,"children":3826},{"style":1349},[3827],{"type":59,"value":2743},{"type":53,"tag":366,"props":3829,"children":3831},{"id":3830},"step-7-trigger-and-monitor",[3832],{"type":59,"value":3833},"Step 7: Trigger and monitor",{"type":53,"tag":1327,"props":3835,"children":3837},{"className":1329,"code":3836,"language":1331,"meta":1332,"style":1332},"ax tasks trigger-run TASK_ID \\\n  --experiment-ids \"EXP_ID\" \\   # base64 ID from `ax experiments list --space SPACE -o json`\n  --wait\n\nax tasks list-runs TASK_ID\nax tasks get-run RUN_ID\n",[3838],{"type":53,"tag":74,"props":3839,"children":3840},{"__ignoreMap":1332},[3841,3864,3891,3898,3905,3926],{"type":53,"tag":614,"props":3842,"children":3843},{"class":1339,"line":1340},[3844,3848,3852,3856,3860],{"type":53,"tag":614,"props":3845,"children":3846},{"style":1344},[3847],{"type":59,"value":189},{"type":53,"tag":614,"props":3849,"children":3850},{"style":1349},[3851],{"type":59,"value":2644},{"type":53,"tag":614,"props":3853,"children":3854},{"style":1349},[3855],{"type":59,"value":3101},{"type":53,"tag":614,"props":3857,"children":3858},{"style":1349},[3859],{"type":59,"value":3106},{"type":53,"tag":614,"props":3861,"children":3862},{"style":1708},[3863],{"type":59,"value":1711},{"type":53,"tag":614,"props":3865,"children":3866},{"class":1339,"line":1714},[3867,3871,3875,3879,3883,3887],{"type":53,"tag":614,"props":3868,"children":3869},{"style":1349},[3870],{"type":59,"value":3773},{"type":53,"tag":614,"props":3872,"children":3873},{"style":1723},[3874],{"type":59,"value":1726},{"type":53,"tag":614,"props":3876,"children":3877},{"style":1349},[3878],{"type":59,"value":3782},{"type":53,"tag":614,"props":3880,"children":3881},{"style":1723},[3882],{"type":59,"value":1735},{"type":53,"tag":614,"props":3884,"children":3885},{"style":1708},[3886],{"type":59,"value":3791},{"type":53,"tag":614,"props":3888,"children":3889},{"style":2996},[3890],{"type":59,"value":3796},{"type":53,"tag":614,"props":3892,"children":3893},{"class":1339,"line":1742},[3894],{"type":53,"tag":614,"props":3895,"children":3896},{"style":1344},[3897],{"type":59,"value":3184},{"type":53,"tag":614,"props":3899,"children":3900},{"class":1339,"line":1759},[3901],{"type":53,"tag":614,"props":3902,"children":3903},{"emptyLinePlaceholder":1929},[3904],{"type":59,"value":1932},{"type":53,"tag":614,"props":3906,"children":3907},{"class":1339,"line":27},[3908,3912,3916,3921],{"type":53,"tag":614,"props":3909,"children":3910},{"style":1344},[3911],{"type":59,"value":189},{"type":53,"tag":614,"props":3913,"children":3914},{"style":1349},[3915],{"type":59,"value":2644},{"type":53,"tag":614,"props":3917,"children":3918},{"style":1349},[3919],{"type":59,"value":3920}," list-runs",{"type":53,"tag":614,"props":3922,"children":3923},{"style":1349},[3924],{"type":59,"value":3925}," TASK_ID\n",{"type":53,"tag":614,"props":3927,"children":3928},{"class":1339,"line":1810},[3929,3933,3937,3942],{"type":53,"tag":614,"props":3930,"children":3931},{"style":1344},[3932],{"type":59,"value":189},{"type":53,"tag":614,"props":3934,"children":3935},{"style":1349},[3936],{"type":59,"value":2644},{"type":53,"tag":614,"props":3938,"children":3939},{"style":1349},[3940],{"type":59,"value":3941}," get-run",{"type":53,"tag":614,"props":3943,"children":3944},{"style":1349},[3945],{"type":59,"value":3946}," RUN_ID\n",{"type":53,"tag":169,"props":3948,"children":3949},{},[],{"type":53,"tag":173,"props":3951,"children":3953},{"id":3952},"best-practices-for-template-design",[3954],{"type":59,"value":3955},"Best Practices for Template Design",{"type":53,"tag":366,"props":3957,"children":3959},{"id":3958},"_1-use-generic-portable-variable-names",[3960],{"type":59,"value":3961},"1. Use generic, portable variable names",{"type":53,"tag":66,"props":3963,"children":3964},{},[3965,3967,3972,3973,3978,3979,3984,3986,3992,3994,4004],{"type":59,"value":3966},"Use ",{"type":53,"tag":74,"props":3968,"children":3970},{"className":3969},[],[3971],{"type":59,"value":441},{"type":59,"value":443},{"type":53,"tag":74,"props":3974,"children":3976},{"className":3975},[],[3977],{"type":59,"value":449},{"type":59,"value":3444},{"type":53,"tag":74,"props":3980,"children":3982},{"className":3981},[],[3983],{"type":59,"value":456},{"type":59,"value":3985}," — not names tied to a specific project or span attribute (e.g. do not use ",{"type":53,"tag":74,"props":3987,"children":3989},{"className":3988},[],[3990],{"type":59,"value":3991},"{attributes_input_value}",{"type":59,"value":3993},"). The evaluator itself stays abstract; the ",{"type":53,"tag":70,"props":3995,"children":3996},{},[3997,3999],{"type":59,"value":3998},"task's ",{"type":53,"tag":74,"props":4000,"children":4002},{"className":4001},[],[4003],{"type":59,"value":1680},{"type":59,"value":4005}," is where you wire it to the actual fields in a specific project or experiment. This lets the same evaluator run across multiple projects and experiments without modification.",{"type":53,"tag":366,"props":4007,"children":4009},{"id":4008},"_2-default-to-binary-labels",[4010],{"type":59,"value":4011},"2. Default to binary labels",{"type":53,"tag":66,"props":4013,"children":4014},{},[4015,4017,4022,4023,4028,4029,4034,4035,4040,4041,4047,4048,4054],{"type":59,"value":4016},"Use exactly two clear string labels (e.g. ",{"type":53,"tag":74,"props":4018,"children":4020},{"className":4019},[],[4021],{"type":59,"value":488},{"type":59,"value":482},{"type":53,"tag":74,"props":4024,"children":4026},{"className":4025},[],[4027],{"type":59,"value":480},{"type":59,"value":443},{"type":53,"tag":74,"props":4030,"children":4032},{"className":4031},[],[4033],{"type":59,"value":1570},{"type":59,"value":482},{"type":53,"tag":74,"props":4036,"children":4038},{"className":4037},[],[4039],{"type":59,"value":1577},{"type":59,"value":443},{"type":53,"tag":74,"props":4042,"children":4044},{"className":4043},[],[4045],{"type":59,"value":4046},"pass",{"type":59,"value":482},{"type":53,"tag":74,"props":4049,"children":4051},{"className":4050},[],[4052],{"type":59,"value":4053},"fail",{"type":59,"value":4055},"). Binary labels are:",{"type":53,"tag":205,"props":4057,"children":4058},{},[4059,4064,4069],{"type":53,"tag":209,"props":4060,"children":4061},{},[4062],{"type":59,"value":4063},"Easiest for the judge model to produce consistently",{"type":53,"tag":209,"props":4065,"children":4066},{},[4067],{"type":59,"value":4068},"Most common in the industry",{"type":53,"tag":209,"props":4070,"children":4071},{},[4072],{"type":59,"value":4073},"Simplest to interpret in dashboards",{"type":53,"tag":66,"props":4075,"children":4076},{},[4077],{"type":59,"value":4078},"If the user insists on more than two choices, that's fine — but recommend binary first and explain the tradeoff (more labels → more ambiguity → lower inter-rater reliability).",{"type":53,"tag":366,"props":4080,"children":4082},{"id":4081},"_3-be-explicit-about-what-the-model-must-return",[4083],{"type":59,"value":4084},"3. Be explicit about what the model must return",{"type":53,"tag":66,"props":4086,"children":4087},{},[4088,4090,4095,4097,4102,4104,4110],{"type":59,"value":4089},"The template must tell the judge model to respond with ",{"type":53,"tag":70,"props":4091,"children":4092},{},[4093],{"type":59,"value":4094},"only",{"type":59,"value":4096}," the label string — nothing else. The label strings in the prompt must ",{"type":53,"tag":70,"props":4098,"children":4099},{},[4100],{"type":59,"value":4101},"exactly match",{"type":59,"value":4103}," the labels in ",{"type":53,"tag":74,"props":4105,"children":4107},{"className":4106},[],[4108],{"type":59,"value":4109},"--classification-choices",{"type":59,"value":4111}," (same spelling, same casing).",{"type":53,"tag":66,"props":4113,"children":4114},{},[4115],{"type":59,"value":4116},"Good:",{"type":53,"tag":1327,"props":4118,"children":4122},{"className":4119,"code":4121,"language":59},[4120],"language-text","Respond with exactly one of these labels: hallucinated, factual\n",[4123],{"type":53,"tag":74,"props":4124,"children":4125},{"__ignoreMap":1332},[4126],{"type":59,"value":4121},{"type":53,"tag":66,"props":4128,"children":4129},{},[4130],{"type":59,"value":4131},"Bad (too open-ended):",{"type":53,"tag":1327,"props":4133,"children":4136},{"className":4134,"code":4135,"language":59},[4120],"Is this hallucinated? Answer yes or no.\n",[4137],{"type":53,"tag":74,"props":4138,"children":4139},{"__ignoreMap":1332},[4140],{"type":59,"value":4135},{"type":53,"tag":366,"props":4142,"children":4144},{"id":4143},"_4-keep-temperature-low",[4145],{"type":59,"value":4146},"4. Keep temperature low",{"type":53,"tag":66,"props":4148,"children":4149},{},[4150,4152,4158],{"type":59,"value":4151},"Pass ",{"type":53,"tag":74,"props":4153,"children":4155},{"className":4154},[],[4156],{"type":59,"value":4157},"--invocation-params '{\"temperature\": 0}'",{"type":59,"value":4159}," for reproducible scoring. Higher temperatures introduce noise into evaluation results.",{"type":53,"tag":366,"props":4161,"children":4163},{"id":4162},"_5-use-include-explanations-for-debugging",[4164,4166,4172],{"type":59,"value":4165},"5. Use ",{"type":53,"tag":74,"props":4167,"children":4169},{"className":4168},[],[4170],{"type":59,"value":4171},"--include-explanations",{"type":59,"value":4173}," for debugging",{"type":53,"tag":66,"props":4175,"children":4176},{},[4177],{"type":59,"value":4178},"During initial setup, always include explanations so you can verify the judge is reasoning correctly before trusting the labels at scale.",{"type":53,"tag":366,"props":4180,"children":4182},{"id":4181},"_6-pass-the-template-in-single-quotes-in-bash",[4183],{"type":59,"value":4184},"6. Pass the template in single quotes in bash",{"type":53,"tag":66,"props":4186,"children":4187},{},[4188,4190,4195],{"type":59,"value":4189},"Single quotes prevent the shell from interpolating ",{"type":53,"tag":74,"props":4191,"children":4193},{"className":4192},[],[4194],{"type":59,"value":433},{"type":59,"value":4196}," placeholders. Double quotes will cause issues:",{"type":53,"tag":1327,"props":4198,"children":4200},{"className":1329,"code":4199,"language":1331,"meta":1332,"style":1332},"# Correct\n--template 'Judge this: {input} → {output}'\n\n# Wrong — shell may interpret { } or fail\n--template \"Judge this: {input} → {output}\"\n",[4201],{"type":53,"tag":74,"props":4202,"children":4203},{"__ignoreMap":1332},[4204,4212,4233,4240,4248],{"type":53,"tag":614,"props":4205,"children":4206},{"class":1339,"line":1340},[4207],{"type":53,"tag":614,"props":4208,"children":4209},{"style":2996},[4210],{"type":59,"value":4211},"# Correct\n",{"type":53,"tag":614,"props":4213,"children":4214},{"class":1339,"line":1714},[4215,4220,4224,4229],{"type":53,"tag":614,"props":4216,"children":4217},{"style":1344},[4218],{"type":59,"value":4219},"--template",{"type":53,"tag":614,"props":4221,"children":4222},{"style":1723},[4223],{"type":59,"value":1890},{"type":53,"tag":614,"props":4225,"children":4226},{"style":1349},[4227],{"type":59,"value":4228},"Judge this: {input} → {output}",{"type":53,"tag":614,"props":4230,"children":4231},{"style":1723},[4232],{"type":59,"value":1980},{"type":53,"tag":614,"props":4234,"children":4235},{"class":1339,"line":1742},[4236],{"type":53,"tag":614,"props":4237,"children":4238},{"emptyLinePlaceholder":1929},[4239],{"type":59,"value":1932},{"type":53,"tag":614,"props":4241,"children":4242},{"class":1339,"line":1759},[4243],{"type":53,"tag":614,"props":4244,"children":4245},{"style":2996},[4246],{"type":59,"value":4247},"# Wrong — shell may interpret { } or fail\n",{"type":53,"tag":614,"props":4249,"children":4250},{"class":1339,"line":27},[4251,4255,4259,4263],{"type":53,"tag":614,"props":4252,"children":4253},{"style":1344},[4254],{"type":59,"value":4219},{"type":53,"tag":614,"props":4256,"children":4257},{"style":1723},[4258],{"type":59,"value":1726},{"type":53,"tag":614,"props":4260,"children":4261},{"style":1349},[4262],{"type":59,"value":4228},{"type":53,"tag":614,"props":4264,"children":4265},{"style":1723},[4266],{"type":59,"value":2575},{"type":53,"tag":366,"props":4268,"children":4270},{"id":4269},"_7-always-set-classification-choices-to-match-your-template-labels",[4271,4273,4278],{"type":59,"value":4272},"7. Always set ",{"type":53,"tag":74,"props":4274,"children":4276},{"className":4275},[],[4277],{"type":59,"value":4109},{"type":59,"value":4279}," to match your template labels",{"type":53,"tag":66,"props":4281,"children":4282},{},[4283,4285,4290,4292,4297,4299,4304],{"type":59,"value":4284},"The labels in ",{"type":53,"tag":74,"props":4286,"children":4288},{"className":4287},[],[4289],{"type":59,"value":4109},{"type":59,"value":4291}," must exactly match the labels referenced in ",{"type":53,"tag":74,"props":4293,"children":4295},{"className":4294},[],[4296],{"type":59,"value":4219},{"type":59,"value":4298}," (same spelling, same casing). Omitting ",{"type":53,"tag":74,"props":4300,"children":4302},{"className":4301},[],[4303],{"type":59,"value":4109},{"type":59,"value":4305}," causes task runs to fail with \"missing rails and classification choices.\"",{"type":53,"tag":169,"props":4307,"children":4308},{},[],{"type":53,"tag":173,"props":4310,"children":4312},{"id":4311},"troubleshooting",[4313],{"type":59,"value":4314},"Troubleshooting",{"type":53,"tag":385,"props":4316,"children":4317},{},[4318,4334],{"type":53,"tag":389,"props":4319,"children":4320},{},[4321],{"type":53,"tag":393,"props":4322,"children":4323},{},[4324,4329],{"type":53,"tag":397,"props":4325,"children":4326},{},[4327],{"type":59,"value":4328},"Problem",{"type":53,"tag":397,"props":4330,"children":4331},{},[4332],{"type":59,"value":4333},"Solution",{"type":53,"tag":408,"props":4335,"children":4336},{},[4337,4358,4380,4401,4421,4442,4459,4498,4523,4557,4592,4623,4644,4663,4683,4703,4722,4748,4779,4806,4825,4863],{"type":53,"tag":393,"props":4338,"children":4339},{},[4340,4349],{"type":53,"tag":415,"props":4341,"children":4342},{},[4343],{"type":53,"tag":74,"props":4344,"children":4346},{"className":4345},[],[4347],{"type":59,"value":4348},"ax: command not found",{"type":53,"tag":415,"props":4350,"children":4351},{},[4352,4354],{"type":59,"value":4353},"See ",{"type":53,"tag":221,"props":4355,"children":4356},{"href":223},[4357],{"type":59,"value":223},{"type":53,"tag":393,"props":4359,"children":4360},{},[4361,4369],{"type":53,"tag":415,"props":4362,"children":4363},{},[4364],{"type":53,"tag":74,"props":4365,"children":4367},{"className":4366},[],[4368],{"type":59,"value":234},{"type":53,"tag":415,"props":4370,"children":4371},{},[4372,4374,4379],{"type":59,"value":4373},"API key may not have access to this space. Verify at ",{"type":53,"tag":221,"props":4375,"children":4377},{"href":254,"rel":4376},[256],[4378],{"type":59,"value":254},{"type":59,"value":260},{"type":53,"tag":393,"props":4381,"children":4382},{},[4383,4392],{"type":53,"tag":415,"props":4384,"children":4385},{},[4386],{"type":53,"tag":74,"props":4387,"children":4389},{"className":4388},[],[4390],{"type":59,"value":4391},"Evaluator not found",{"type":53,"tag":415,"props":4393,"children":4394},{},[4395],{"type":53,"tag":74,"props":4396,"children":4398},{"className":4397},[],[4399],{"type":59,"value":4400},"ax evaluators list --space SPACE",{"type":53,"tag":393,"props":4402,"children":4403},{},[4404,4413],{"type":53,"tag":415,"props":4405,"children":4406},{},[4407],{"type":53,"tag":74,"props":4408,"children":4410},{"className":4409},[],[4411],{"type":59,"value":4412},"Integration not found",{"type":53,"tag":415,"props":4414,"children":4415},{},[4416],{"type":53,"tag":74,"props":4417,"children":4419},{"className":4418},[],[4420],{"type":59,"value":283},{"type":53,"tag":393,"props":4422,"children":4423},{},[4424,4433],{"type":53,"tag":415,"props":4425,"children":4426},{},[4427],{"type":53,"tag":74,"props":4428,"children":4430},{"className":4429},[],[4431],{"type":59,"value":4432},"Task not found",{"type":53,"tag":415,"props":4434,"children":4435},{},[4436],{"type":53,"tag":74,"props":4437,"children":4439},{"className":4438},[],[4440],{"type":59,"value":4441},"ax tasks list --space SPACE",{"type":53,"tag":393,"props":4443,"children":4444},{},[4445,4454],{"type":53,"tag":415,"props":4446,"children":4447},{},[4448],{"type":53,"tag":74,"props":4449,"children":4451},{"className":4450},[],[4452],{"type":59,"value":4453},"project and dataset-id are mutually exclusive",{"type":53,"tag":415,"props":4455,"children":4456},{},[4457],{"type":59,"value":4458},"Use only one when creating a task",{"type":53,"tag":393,"props":4460,"children":4461},{},[4462,4471],{"type":53,"tag":415,"props":4463,"children":4464},{},[4465],{"type":53,"tag":74,"props":4466,"children":4468},{"className":4467},[],[4469],{"type":59,"value":4470},"experiment-ids required for dataset tasks",{"type":53,"tag":415,"props":4472,"children":4473},{},[4474,4476,4482,4484,4490,4492],{"type":59,"value":4475},"Add ",{"type":53,"tag":74,"props":4477,"children":4479},{"className":4478},[],[4480],{"type":59,"value":4481},"--experiment-ids",{"type":59,"value":4483}," to ",{"type":53,"tag":74,"props":4485,"children":4487},{"className":4486},[],[4488],{"type":59,"value":4489},"create",{"type":59,"value":4491}," and ",{"type":53,"tag":74,"props":4493,"children":4495},{"className":4494},[],[4496],{"type":59,"value":4497},"trigger-run",{"type":53,"tag":393,"props":4499,"children":4500},{},[4501,4510],{"type":53,"tag":415,"props":4502,"children":4503},{},[4504],{"type":53,"tag":74,"props":4505,"children":4507},{"className":4506},[],[4508],{"type":59,"value":4509},"sampling-rate only valid for project tasks",{"type":53,"tag":415,"props":4511,"children":4512},{},[4513,4515,4521],{"type":59,"value":4514},"Remove ",{"type":53,"tag":74,"props":4516,"children":4518},{"className":4517},[],[4519],{"type":59,"value":4520},"--sampling-rate",{"type":59,"value":4522}," from dataset tasks",{"type":53,"tag":393,"props":4524,"children":4525},{},[4526,4536],{"type":53,"tag":415,"props":4527,"children":4528},{},[4529,4531],{"type":59,"value":4530},"Validation error on ",{"type":53,"tag":74,"props":4532,"children":4534},{"className":4533},[],[4535],{"type":59,"value":1323},{"type":53,"tag":415,"props":4537,"children":4538},{},[4539,4541,4547,4549,4555],{"type":59,"value":4540},"Project name usually works; if you still get a validation error, look up the base64 project ID via ",{"type":53,"tag":74,"props":4542,"children":4544},{"className":4543},[],[4545],{"type":59,"value":4546},"ax projects list --space SPACE -o json",{"type":59,"value":4548}," and use the ",{"type":53,"tag":74,"props":4550,"children":4552},{"className":4551},[],[4553],{"type":59,"value":4554},"id",{"type":59,"value":4556}," field instead",{"type":53,"tag":393,"props":4558,"children":4559},{},[4560,4565],{"type":53,"tag":415,"props":4561,"children":4562},{},[4563],{"type":59,"value":4564},"Template validation errors",{"type":53,"tag":415,"props":4566,"children":4567},{},[4568,4570,4576,4578,4584,4586],{"type":59,"value":4569},"Use single-quoted ",{"type":53,"tag":74,"props":4571,"children":4573},{"className":4572},[],[4574],{"type":59,"value":4575},"--template '...'",{"type":59,"value":4577}," in bash; single braces ",{"type":53,"tag":74,"props":4579,"children":4581},{"className":4580},[],[4582],{"type":59,"value":4583},"{var}",{"type":59,"value":4585},", not double ",{"type":53,"tag":74,"props":4587,"children":4589},{"className":4588},[],[4590],{"type":59,"value":4591},"{{var}}",{"type":53,"tag":393,"props":4593,"children":4594},{},[4595,4606],{"type":53,"tag":415,"props":4596,"children":4597},{},[4598,4600],{"type":59,"value":4599},"Run stuck in ",{"type":53,"tag":74,"props":4601,"children":4603},{"className":4602},[],[4604],{"type":59,"value":4605},"pending",{"type":53,"tag":415,"props":4607,"children":4608},{},[4609,4615,4617],{"type":53,"tag":74,"props":4610,"children":4612},{"className":4611},[],[4613],{"type":59,"value":4614},"ax tasks get-run RUN_ID",{"type":59,"value":4616},"; then ",{"type":53,"tag":74,"props":4618,"children":4620},{"className":4619},[],[4621],{"type":59,"value":4622},"ax tasks cancel-run RUN_ID",{"type":53,"tag":393,"props":4624,"children":4625},{},[4626,4639],{"type":53,"tag":415,"props":4627,"children":4628},{},[4629,4631,4637],{"type":59,"value":4630},"Run ",{"type":53,"tag":74,"props":4632,"children":4634},{"className":4633},[],[4635],{"type":59,"value":4636},"cancelled",{"type":59,"value":4638}," ~1s",{"type":53,"tag":415,"props":4640,"children":4641},{},[4642],{"type":59,"value":4643},"Integration credentials invalid — check AI integration",{"type":53,"tag":393,"props":4645,"children":4646},{},[4647,4658],{"type":53,"tag":415,"props":4648,"children":4649},{},[4650,4651,4656],{"type":59,"value":4630},{"type":53,"tag":74,"props":4652,"children":4654},{"className":4653},[],[4655],{"type":59,"value":4636},{"type":59,"value":4657}," ~3min",{"type":53,"tag":415,"props":4659,"children":4660},{},[4661],{"type":59,"value":4662},"Found spans but LLM call failed — wrong model name or bad key",{"type":53,"tag":393,"props":4664,"children":4665},{},[4666,4678],{"type":53,"tag":415,"props":4667,"children":4668},{},[4669,4670,4676],{"type":59,"value":4630},{"type":53,"tag":74,"props":4671,"children":4673},{"className":4672},[],[4674],{"type":59,"value":4675},"completed",{"type":59,"value":4677},", 0 spans",{"type":53,"tag":415,"props":4679,"children":4680},{},[4681],{"type":59,"value":4682},"Widen time window; eval index may not cover older data",{"type":53,"tag":393,"props":4684,"children":4685},{},[4686,4691],{"type":53,"tag":415,"props":4687,"children":4688},{},[4689],{"type":59,"value":4690},"No scores in UI",{"type":53,"tag":415,"props":4692,"children":4693},{},[4694,4696,4701],{"type":59,"value":4695},"Fix ",{"type":53,"tag":74,"props":4697,"children":4699},{"className":4698},[],[4700],{"type":59,"value":1680},{"type":59,"value":4702}," to match real paths on your spans\u002Fruns",{"type":53,"tag":393,"props":4704,"children":4705},{},[4706,4711],{"type":53,"tag":415,"props":4707,"children":4708},{},[4709],{"type":59,"value":4710},"Scores look wrong",{"type":53,"tag":415,"props":4712,"children":4713},{},[4714,4715,4720],{"type":59,"value":4475},{"type":53,"tag":74,"props":4716,"children":4718},{"className":4717},[],[4719],{"type":59,"value":4171},{"type":59,"value":4721}," and inspect judge reasoning on a few samples",{"type":53,"tag":393,"props":4723,"children":4724},{},[4725,4730],{"type":53,"tag":415,"props":4726,"children":4727},{},[4728],{"type":59,"value":4729},"Evaluator cancels on wrong span kind",{"type":53,"tag":415,"props":4731,"children":4732},{},[4733,4735,4740,4741,4746],{"type":59,"value":4734},"Match ",{"type":53,"tag":74,"props":4736,"children":4738},{"className":4737},[],[4739],{"type":59,"value":1246},{"type":59,"value":4491},{"type":53,"tag":74,"props":4742,"children":4744},{"className":4743},[],[4745],{"type":59,"value":1680},{"type":59,"value":4747}," to LLM vs CHAIN spans",{"type":53,"tag":393,"props":4749,"children":4750},{},[4751,4761],{"type":53,"tag":415,"props":4752,"children":4753},{},[4754,4756],{"type":59,"value":4755},"Time format error on ",{"type":53,"tag":74,"props":4757,"children":4759},{"className":4758},[],[4760],{"type":59,"value":4497},{"type":53,"tag":415,"props":4762,"children":4763},{},[4764,4765,4771,4773],{"type":59,"value":3966},{"type":53,"tag":74,"props":4766,"children":4768},{"className":4767},[],[4769],{"type":59,"value":4770},"2026-03-21T09:00:00",{"type":59,"value":4772}," — no trailing ",{"type":53,"tag":74,"props":4774,"children":4776},{"className":4775},[],[4777],{"type":59,"value":4778},"Z",{"type":53,"tag":393,"props":4780,"children":4781},{},[4782,4787],{"type":53,"tag":415,"props":4783,"children":4784},{},[4785],{"type":59,"value":4786},"Run failed: \"missing rails and classification choices\"",{"type":53,"tag":415,"props":4788,"children":4789},{},[4790,4791,4797,4798,4804],{"type":59,"value":4475},{"type":53,"tag":74,"props":4792,"children":4794},{"className":4793},[],[4795],{"type":59,"value":4796},"--classification-choices '{\"label_a\": 1, \"label_b\": 0}'",{"type":59,"value":4483},{"type":53,"tag":74,"props":4799,"children":4801},{"className":4800},[],[4802],{"type":59,"value":4803},"ax evaluators create-template-evaluator",{"type":59,"value":4805}," — labels must match the template",{"type":53,"tag":393,"props":4807,"children":4808},{},[4809,4820],{"type":53,"tag":415,"props":4810,"children":4811},{},[4812,4813,4818],{"type":59,"value":4630},{"type":53,"tag":74,"props":4814,"children":4816},{"className":4815},[],[4817],{"type":59,"value":4675},{"type":59,"value":4819},", all spans skipped",{"type":53,"tag":415,"props":4821,"children":4822},{},[4823],{"type":59,"value":4824},"Query filter matched spans but column mappings are wrong or template variables don't resolve — export a sample span and verify paths",{"type":53,"tag":393,"props":4826,"children":4827},{},[4828,4838],{"type":53,"tag":415,"props":4829,"children":4830},{},[4831,4836],{"type":53,"tag":74,"props":4832,"children":4834},{"className":4833},[],[4835],{"type":59,"value":1246},{"type":59,"value":4837}," set but 0 spans scored",{"type":53,"tag":415,"props":4839,"children":4840},{},[4841,4843,4848,4850,4855,4856,4861],{"type":59,"value":4842},"The filter attribute may not be indexed in the eval index. ",{"type":53,"tag":74,"props":4844,"children":4846},{"className":4845},[],[4847],{"type":59,"value":2313},{"type":59,"value":4849}," and custom attributes are often not indexed. Use ",{"type":53,"tag":74,"props":4851,"children":4853},{"className":4852},[],[4854],{"type":59,"value":2321},{"type":59,"value":2323},{"type":53,"tag":74,"props":4857,"children":4859},{"className":4858},[],[4860],{"type":59,"value":2329},{"type":59,"value":4862}," instead, or remove the filter to confirm spans exist in the window.",{"type":53,"tag":393,"props":4864,"children":4865},{},[4866,4884],{"type":53,"tag":415,"props":4867,"children":4868},{},[4869,4871,4875,4877,4882],{"type":59,"value":4870},"Custom ",{"type":53,"tag":70,"props":4872,"children":4873},{},[4874],{"type":59,"value":74},{"type":59,"value":4876}," evaluator run cancels ~3s with ",{"type":53,"tag":74,"props":4878,"children":4880},{"className":4879},[],[4881],{"type":59,"value":673},{"type":59,"value":4883}," (successes\u002Ferrors\u002Fskipped)",{"type":53,"tag":415,"props":4885,"children":4886},{},[4887,4889,4894,4896,4901,4903,4907,4909,4915,4917,4923,4925,4930,4932,4938],{"type":59,"value":4888},"Wrong import path or ",{"type":53,"tag":74,"props":4890,"children":4892},{"className":4891},[],[4893],{"type":59,"value":665},{"type":59,"value":4895}," signature — see the \"CRITICAL\" callout under ",{"type":53,"tag":70,"props":4897,"children":4898},{},[4899],{"type":59,"value":4900},"Custom Python code evaluators",{"type":59,"value":4902}," in ",{"type":53,"tag":221,"props":4904,"children":4905},{"href":678},[4906],{"type":59,"value":678},{"type":59,"value":4908},". Must import from ",{"type":53,"tag":74,"props":4910,"children":4912},{"className":4911},[],[4913],{"type":59,"value":4914},"arize.experimental.datasets.experiments.evaluators.base",{"type":59,"value":4916}," (not ",{"type":53,"tag":74,"props":4918,"children":4920},{"className":4919},[],[4921],{"type":59,"value":4922},"arize.experiments",{"type":59,"value":4924},") and declare named ",{"type":53,"tag":74,"props":4926,"children":4928},{"className":4927},[],[4929],{"type":59,"value":665},{"type":59,"value":4931}," params, not just ",{"type":53,"tag":74,"props":4933,"children":4935},{"className":4934},[],[4936],{"type":59,"value":4937},"**kwargs",{"type":59,"value":134},{"type":53,"tag":366,"props":4940,"children":4942},{"id":4941},"diagnosing-cancelled-runs",[4943],{"type":59,"value":4944},"Diagnosing cancelled runs",{"type":53,"tag":66,"props":4946,"children":4947},{},[4948,4950,4955,4957,4962],{"type":59,"value":4949},"When a task run reports status ",{"type":53,"tag":74,"props":4951,"children":4953},{"className":4952},[],[4954],{"type":59,"value":4636},{"type":59,"value":4956},", work through the ordered checklist (credentials → model name → column-mapping\u002Fpath checks → time window → span kind → variable resolution) in ",{"type":53,"tag":221,"props":4958,"children":4960},{"href":4959},"references\u002Ftroubleshooting.md",[4961],{"type":59,"value":4959},{"type":59,"value":134},{"type":53,"tag":169,"props":4964,"children":4965},{},[],{"type":53,"tag":173,"props":4967,"children":4969},{"id":4968},"related-skills",[4970],{"type":59,"value":4971},"Related Skills",{"type":53,"tag":205,"props":4973,"children":4974},{},[4975,4984,4994,5003,5013,5023],{"type":53,"tag":209,"props":4976,"children":4977},{},[4978,4982],{"type":53,"tag":70,"props":4979,"children":4980},{},[4981],{"type":59,"value":290},{"type":59,"value":4983},": Full CRUD for LLM provider integrations (create, update, delete credentials)",{"type":53,"tag":209,"props":4985,"children":4986},{},[4987,4992],{"type":53,"tag":70,"props":4988,"children":4989},{},[4990],{"type":59,"value":4991},"arize-trace",{"type":59,"value":4993},": Export spans to discover column paths and time ranges",{"type":53,"tag":209,"props":4995,"children":4996},{},[4997,5001],{"type":53,"tag":70,"props":4998,"children":4999},{},[5000],{"type":59,"value":3242},{"type":59,"value":5002},": Create experiments and export runs for experiment column mappings",{"type":53,"tag":209,"props":5004,"children":5005},{},[5006,5011],{"type":53,"tag":70,"props":5007,"children":5008},{},[5009],{"type":59,"value":5010},"arize-dataset",{"type":59,"value":5012},": Export dataset examples to find input fields when runs omit them",{"type":53,"tag":209,"props":5014,"children":5015},{},[5016,5021],{"type":53,"tag":70,"props":5017,"children":5018},{},[5019],{"type":59,"value":5020},"arize-prompts",{"type":59,"value":5022},": Manage prompt templates in the Prompt Hub; use with evaluators to score versioned prompts",{"type":53,"tag":209,"props":5024,"children":5025},{},[5026,5031],{"type":53,"tag":70,"props":5027,"children":5028},{},[5029],{"type":59,"value":5030},"arize-link",{"type":59,"value":5032},": Deep links to evaluators and tasks in the Arize UI",{"type":53,"tag":169,"props":5034,"children":5035},{},[],{"type":53,"tag":173,"props":5037,"children":5039},{"id":5038},"save-credentials-for-future-use",[5040],{"type":59,"value":5041},"Save Credentials for Future Use",{"type":53,"tag":66,"props":5043,"children":5044},{},[5045,5046,5050],{"type":59,"value":4353},{"type":53,"tag":221,"props":5047,"children":5048},{"href":247},[5049],{"type":59,"value":247},{"type":59,"value":5051}," § Save Credentials for Future Use.",{"type":53,"tag":5053,"props":5054,"children":5055},"style",{},[5056],{"type":59,"value":5057},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":5059,"total":5221},[5060,5074,5086,5098,5109,5119,5133,5143,5154,5170,5189,5199],{"slug":5061,"name":5061,"fn":5062,"description":5063,"org":5064,"tags":5065,"stars":5071,"repoUrl":5072,"updatedAt":5073},"annotate-spans","annotate LLM spans and traces","Write effective, consistent annotations on LLM\u002Fagent spans and traces, and coach the user on annotation practice. Load this whenever you are about to record structured feedback with the `batch_span_annotate` tool, or when the user asks how to annotate, label, score, or review spans\u002Ftraces, build a failure taxonomy, or set up human\u002FLLM review. Do NOT load for: pure analysis with no intent to save feedback (use debug-trace), latency or cost statistics, or prompt authoring (use playground).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5066,5067,5068,5069],{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":5070,"slug":38,"type":16},"Tracing",10513,"https:\u002F\u002Fgithub.com\u002FArize-ai\u002Fphoenix","2026-07-12T08:08:14.140984",{"slug":35,"name":35,"fn":5075,"description":5076,"org":5077,"tags":5078,"stars":5071,"repoUrl":5072,"updatedAt":5085},"reason about Phoenix dataset structure","Understand what a Phoenix dataset is and reason well about its examples, outputs, splits, and how it feeds evaluators and experiments. Load this whenever a dataset is in view or the user asks what a dataset is, how splits work, what an output \"means\", or how datasets relate to experiments and evals. This skill governs the judgment; any tool descriptions govern the mechanics.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5079,5082,5084],{"name":5080,"slug":5081,"type":16},"Data Analysis","data-analysis",{"name":5083,"slug":35,"type":16},"Datasets",{"name":21,"slug":22,"type":16},"2026-07-12T08:08:21.695457",{"slug":5087,"name":5087,"fn":5088,"description":5089,"org":5090,"tags":5091,"stars":5071,"repoUrl":5072,"updatedAt":5097},"debug-trace","diagnose failures using trace investigation","Diagnose failure modes by systematically investigating traces. Trigger when the user explicitly asks for cross-trace diagnosis: \"what's going wrong?\", \"were there errors?\", \"debug this\", \"where is my agent struggling?\". Do NOT trigger on: (1) advice questions (\"what should I do?\"), (2) statistical questions (\"what's the average latency?\"), (3) summarize requests, (4) trace filtering (\"show me traces with errors\"), (5) vague questions (\"is there a problem?\"), (6) unrelated requests.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5092,5095,5096],{"name":5093,"slug":5094,"type":16},"Debugging","debugging",{"name":14,"slug":15,"type":16},{"name":5070,"slug":38,"type":16},"2026-07-12T08:08:10.44243",{"slug":144,"name":144,"fn":5099,"description":5100,"org":5101,"tags":5102,"stars":5071,"repoUrl":5072,"updatedAt":5108},"author and refine Phoenix evaluators","Author or refine a Phoenix evaluator — code or LLM-as-a-judge — that scores a run's output. Trigger when the user wants to create a new evaluator, improve an existing one's logic or rubric, choose labels, or decide what to measure on a dataset or experiment. Do NOT trigger on: (1) manual prompt drafting (use `playground`), (2) running or comparing experiments themselves (use `experiments`), (3) cross-trace failure diagnosis with no evaluator in scope (use `debug-trace`).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5103,5104,5105],{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":5106,"slug":5107,"type":16},"Testing","testing","2026-07-31T05:58:09.13624",{"slug":36,"name":36,"fn":5110,"description":5111,"org":5112,"tags":5113,"stars":5071,"repoUrl":5072,"updatedAt":5118},"run and compare dataset-backed experiments","Run, read, and compare dataset-backed experiments to find evidence that a prompt or pipeline is improving. Trigger when the user wants to iterate over a dataset with experiments, compare experiment runs, read experiment quality\u002Flatency\u002Fcost, or decide whether a change actually helped. Running a prompt over a dataset is implicitly an experiment — load this skill when dataset-backed work begins, before authoring evaluators for the experiment and before starting the recorded run, not only when reading results. Do NOT trigger on: (1) manual prompt drafting with no dataset-backed evaluation in scope (use `playground`), (2) authoring or refining an evaluator's logic or rubric (use `evaluators`), (3) cross-trace failure diagnosis with no experiment in scope (use `debug-trace`).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5114,5115,5116,5117],{"name":5083,"slug":35,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":5106,"slug":5107,"type":16},"2026-07-12T08:08:11.691477",{"slug":5120,"name":5120,"fn":5121,"description":5122,"org":5123,"tags":5124,"stars":5071,"repoUrl":5072,"updatedAt":5132},"phoenix-graphql","query Phoenix API with GraphQL","Write efficient GraphQL queries against the Phoenix API. Load this skill in two cases: (1) before composing any non-trivial GraphQL query yourself for data analysis (via the `phoenix-gql` bash command) — it contains schema entrypoints and patterns that eliminate the need for introspection; (2) when the user asks for help writing GraphQL queries for their own scripts, tools, or integrations against Phoenix — it covers the endpoint, authentication, and client examples.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5125,5128,5129],{"name":5126,"slug":5127,"type":16},"Analytics","analytics",{"name":5080,"slug":5081,"type":16},{"name":5130,"slug":5131,"type":16},"GraphQL","graphql","2026-07-12T08:08:17.163493",{"slug":5134,"name":5134,"fn":5135,"description":5136,"org":5137,"tags":5138,"stars":5071,"repoUrl":5072,"updatedAt":5142},"playground","author and iterate on prompts in Phoenix","Author, edit, or iterate on prompts in the Phoenix prompt playground, including running experiments over a dataset. Load before any playground tool call, including single-shot prompt rewrites.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5139,5140,5141],{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":5106,"slug":5107,"type":16},"2026-07-12T08:08:12.920792",{"slug":5144,"name":5144,"fn":5145,"description":5146,"org":5147,"tags":5148,"stars":5071,"repoUrl":5072,"updatedAt":5153},"span-coding","analyze and code Phoenix spans","Open-code Phoenix spans with PXI-owned notes, recover those notes for axial coding, and promote stable categories into structured annotations. Load this when analyzing spans to discover failure patterns before a taxonomy exists.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5149,5150,5151,5152],{"name":5093,"slug":5094,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":5070,"slug":38,"type":16},"2026-07-12T08:08:19.597239",{"slug":5155,"name":5155,"fn":5156,"description":5157,"org":5158,"tags":5159,"stars":23,"repoUrl":24,"updatedAt":5169},"arize-admin","manage Arize enterprise user access","Manages Arize users, organizations, spaces, projects, roles, role bindings, resource restrictions, and API keys via the ax CLI. Use for enterprise admin workflows: inviting and offboarding users, onboarding new teams, creating custom roles for SAML\u002FSSO mappings, assigning roles to users, restricting project-level access, and managing service keys for multi-tenant architectures. Covers ax users, ax organizations, ax spaces, ax projects, ax roles, ax role-bindings, and ax api-keys.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5160,5163,5166],{"name":5161,"slug":5162,"type":16},"CLI","cli",{"name":5164,"slug":5165,"type":16},"Operations","operations",{"name":5167,"slug":5168,"type":16},"Permissions","permissions","2026-07-22T05:37:21.991338",{"slug":290,"name":290,"fn":5171,"description":5172,"org":5173,"tags":5174,"stars":23,"repoUrl":24,"updatedAt":5188},"manage Arize AI provider integrations","Creates, reads, updates, and deletes Arize AI integrations that store LLM provider credentials used by evaluators and other Arize features. Supports any LLM provider (e.g. OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Gemini, NVIDIA NIM). Use when the user mentions AI integration, LLM provider credentials, create integration, list integrations, update credentials, delete integration, or connecting an LLM provider to Arize.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5175,5178,5181,5184,5185],{"name":5176,"slug":5177,"type":16},"Anthropic","anthropic",{"name":5179,"slug":5180,"type":16},"Azure","azure",{"name":5182,"slug":5183,"type":16},"Integrations","integrations",{"name":18,"slug":19,"type":16},{"name":5186,"slug":5187,"type":16},"OpenAI","openai","2026-07-22T05:37:23.90468",{"slug":5190,"name":5190,"fn":5191,"description":5192,"org":5193,"tags":5194,"stars":23,"repoUrl":24,"updatedAt":5198},"arize-annotation","manage Arize annotation workflows","Creates and manages annotation configs (categorical, continuous, freeform label schemas) and annotation queues (human review workflows) on Arize. Applies human annotations to project spans via the Python SDK. Use when the user mentions annotation config, annotation queue, label schema, human feedback, bulk annotate spans, update_annotations, labeling queue, annotate record, or human review.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5195,5196,5197],{"name":5080,"slug":5081,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"2026-07-22T05:37:19.010776",{"slug":5200,"name":5200,"fn":5201,"description":5202,"org":5203,"tags":5204,"stars":23,"repoUrl":24,"updatedAt":5220},"arize-compliance-audit","audit AI agents for regulatory compliance","INVOKE THIS SKILL when auditing an AI agent or LLM app for regulatory compliance. Covers EU AI Act, GPAI Code of Practice, GDPR, NIST AI RMF, Colorado AI Act, HIPAA, and ISO 42001. Scans the codebase for compliance gaps, cross-references Arize instrumentation for audit trail coverage, and produces an actionable remediation checklist tailored to the selected frameworks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5205,5208,5211,5214,5217],{"name":5206,"slug":5207,"type":16},"Audit","audit",{"name":5209,"slug":5210,"type":16},"Compliance","compliance",{"name":5212,"slug":5213,"type":16},"GDPR","gdpr",{"name":5215,"slug":5216,"type":16},"Legal","legal",{"name":5218,"slug":5219,"type":16},"Security","security","2026-07-19T05:39:42.632738",23,{"items":5223,"total":1935},[5224,5230,5238,5244,5252,5264,5270],{"slug":5155,"name":5155,"fn":5156,"description":5157,"org":5225,"tags":5226,"stars":23,"repoUrl":24,"updatedAt":5169},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5227,5228,5229],{"name":5161,"slug":5162,"type":16},{"name":5164,"slug":5165,"type":16},{"name":5167,"slug":5168,"type":16},{"slug":290,"name":290,"fn":5171,"description":5172,"org":5231,"tags":5232,"stars":23,"repoUrl":24,"updatedAt":5188},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5233,5234,5235,5236,5237],{"name":5176,"slug":5177,"type":16},{"name":5179,"slug":5180,"type":16},{"name":5182,"slug":5183,"type":16},{"name":18,"slug":19,"type":16},{"name":5186,"slug":5187,"type":16},{"slug":5190,"name":5190,"fn":5191,"description":5192,"org":5239,"tags":5240,"stars":23,"repoUrl":24,"updatedAt":5198},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5241,5242,5243],{"name":5080,"slug":5081,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":5200,"name":5200,"fn":5201,"description":5202,"org":5245,"tags":5246,"stars":23,"repoUrl":24,"updatedAt":5220},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5247,5248,5249,5250,5251],{"name":5206,"slug":5207,"type":16},{"name":5209,"slug":5210,"type":16},{"name":5212,"slug":5213,"type":16},{"name":5215,"slug":5216,"type":16},{"name":5218,"slug":5219,"type":16},{"slug":5010,"name":5010,"fn":5253,"description":5254,"org":5255,"tags":5256,"stars":23,"repoUrl":24,"updatedAt":5263},"manage Arize datasets and examples","Creates, manages, and queries Arize datasets and examples. Covers dataset CRUD, appending examples, exporting data, and file-based dataset creation using the ax CLI. Use when the user needs test data, evaluation examples, or mentions create dataset, list datasets, export dataset, append examples, dataset version, golden dataset, or test set.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5257,5260,5261,5262],{"name":5258,"slug":5259,"type":16},"Data Engineering","data-engineering",{"name":5083,"slug":35,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},"2026-07-22T05:37:20.943718",{"slug":4,"name":4,"fn":5,"description":6,"org":5265,"tags":5266,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5267,5268,5269],{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":3242,"name":3242,"fn":5271,"description":5272,"org":5273,"tags":5274,"stars":23,"repoUrl":24,"updatedAt":5278},"run and analyze Arize experiments","Creates, runs, and analyzes Arize experiments for evaluating and comparing model performance. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. Use when the user mentions create experiment, run experiment, compare models, model performance, evaluate AI, experiment results, benchmark, A\u002FB test models, or measure accuracy.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[5275,5276,5277],{"name":5126,"slug":5127,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},"2026-07-31T05:53:44.725539"]