[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-arize-arize-prompt-optimization":3,"mdc-69r5ca-key":44,"related-org-arize-arize-prompt-optimization":4334,"related-repo-arize-arize-prompt-optimization":4498},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":39,"sourceUrl":42,"mdContent":43},"arize-prompt-optimization","optimize LLM prompts with Arize","Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"arize","Arize AI","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Farize.jpg","Arize-ai",[13,17,20],{"name":14,"slug":15,"type":16},"Observability","observability","tag",{"name":18,"slug":19,"type":16},"LLM","llm",{"name":21,"slug":22,"type":16},"Evals","evals",38,"https:\u002F\u002Fgithub.com\u002FArize-ai\u002Farize-skills","2026-07-22T05:37:22.946231",null,5,[29,30,31,8,32,33,34,35,36,37,38],"agent-skills","ai-agents","ai-observability","claude-code","codex","cursor","datasets","experiments","llmops","tracing",{"repoUrl":24,"stars":23,"forks":27,"topics":40,"description":41},[29,30,31,8,32,33,34,35,36,37,38],"Agent skills for Arize — datasets, experiments, and traces via the ax CLI","https:\u002F\u002Fgithub.com\u002FArize-ai\u002Farize-skills\u002Ftree\u002FHEAD\u002Fskills\u002Farize-prompt-optimization","---\nname: arize-prompt-optimization\ndescription: Optimizes, improves, and debugs LLM prompts using production trace data, evaluations, and annotations. Extracts prompts from spans, gathers performance signal, and runs a data-driven optimization loop using the ax CLI. Use when the user mentions optimize prompt, improve prompt, make AI respond better, improve output quality, prompt engineering, prompt tuning, or system prompt improvement.\nmetadata:\n  author: arize\n  version: \"1.0\"\ncompatibility: Requires the ax CLI and a configured Arize profile.\n---\n\n# Arize Prompt Optimization Skill\n\n> **`SPACE`** — All `--space` flags and the `ARIZE_SPACE` env var accept a space **name** (e.g., `my-workspace`) or a base64 space **ID** (e.g., `U3BhY2U6...`). Find yours with `ax spaces list`.\n\n## Related skills\n\n- **arize-prompts**: Create, version, and label prompts in **Prompt Hub** with `ax prompts` (JSON messages, providers, labels such as `production`). Use that skill when the artifact should live in Arize; use **arize-prompt-optimization** below to improve prompt text from traces, datasets, and experiments.\n\n## Concepts\n\n### Where Prompts Live in Trace Data\n\nLLM applications emit spans following OpenInference semantic conventions. Prompts are stored in different span attributes depending on the span kind and instrumentation:\n\n| Column | What it contains | When to use |\n|--------|-----------------|-------------|\n| `attributes.llm.input_messages` | Structured chat messages (system, user, assistant, tool) in role-based format | **Primary source** for chat-based LLM prompts |\n| `attributes.llm.input_messages.roles` | Array of roles: `system`, `user`, `assistant`, `tool` | Extract individual message roles |\n| `attributes.llm.input_messages.contents` | Array of message content strings | Extract message text |\n| `attributes.input.value` | Serialized prompt or user question (generic, all span kinds) | Fallback when structured messages are not available |\n| `attributes.llm.prompt_template.template` | Template with `{variable}` placeholders (e.g., `\"Answer {question} using {context}\"`) | When the app uses prompt templates |\n| `attributes.llm.prompt_template.variables` | Template variable values (JSON object) | See what values were substituted into the template |\n| `attributes.output.value` | Model response text | See what the LLM produced |\n| `attributes.llm.output_messages` | Structured model output (including tool calls) | Inspect tool-calling responses |\n\n### Finding Prompts by Span Kind\n\n- **LLM span** (`attributes.openinference.span.kind = 'LLM'`): Check `attributes.llm.input_messages` for structured chat messages, OR `attributes.input.value` for a serialized prompt. Check `attributes.llm.prompt_template.template` for the template.\n- **Chain\u002FAgent span**: `attributes.input.value` contains the user's question. The actual LLM prompt lives on **child LLM spans** -- navigate down the trace tree.\n- **Tool span**: `attributes.input.value` has tool input, `attributes.output.value` has tool result. Not typically where prompts live.\n\n### Performance Signal Columns\n\nThese columns carry the feedback data used for optimization:\n\n| Column pattern | Source | What it tells you |\n|---------------|--------|-------------------|\n| `annotation.\u003Cname>.label` | Human reviewers | Categorical grade (e.g., `correct`, `incorrect`, `partial`) |\n| `annotation.\u003Cname>.score` | Human reviewers | Numeric quality score (e.g., 0.0 - 1.0) |\n| `annotation.\u003Cname>.text` | Human reviewers | Freeform explanation of the grade |\n| `eval.\u003Cname>.label` | LLM-as-judge evals | Automated categorical assessment |\n| `eval.\u003Cname>.score` | LLM-as-judge evals | Automated numeric score |\n| `eval.\u003Cname>.explanation` | LLM-as-judge evals | Why the eval gave that score -- **most valuable for optimization** |\n| `attributes.input.value` | Trace data | What went into the LLM |\n| `attributes.output.value` | Trace data | What the LLM produced |\n| `{experiment_name}.output` | Experiment runs | Output from a specific experiment |\n\n## Prerequisites\n\nProceed directly with the task — run the `ax` command you need. Do NOT check versions, env vars, or profiles upfront.\n\nIf an `ax` command fails, troubleshoot based on the error:\n- `command not found` or version error → see [references\u002Fax-setup.md](references\u002Fax-setup.md)\n- `401 Unauthorized` \u002F missing API key → run `ax profiles show` to inspect the current profile. If the profile is missing or the API key is wrong, follow [references\u002Fax-profiles.md](references\u002Fax-profiles.md) to create\u002Fupdate it. If the user doesn't have their key, direct them to https:\u002F\u002Fapp.arize.com\u002Fadmin > API Keys\n- Space unknown → run `ax spaces list` to pick by name, or ask the user\n- Project unclear → ask the user, or run `ax projects list -o json --limit 100` and present as selectable options\n- LLM provider call fails (missing provider credentials) → run `ax ai-integrations list --space SPACE` to check for platform-managed credentials. If none exist, use the **arize-ai-provider-integration** skill — never ask the user to paste a provider key into chat.\n- **Security:** Never read `.env` files or search the filesystem for credentials. Use `ax profiles` for Arize credentials and `ax ai-integrations` for LLM provider keys. Never ask the user to paste secrets into chat. For missing credentials, see [references\u002Fax-profiles.md](references\u002Fax-profiles.md).\n\n### When you must ask the user first\n\nStill prefer `ax spaces list`, `ax projects list`, `ax datasets list`, `ax experiments list`, and exports over open-ended questions. If you still cannot proceed (e.g. multiple projects match the name the user gave, unclear trace vs experiment path, or destructive scope), **do not** jump straight into questions — use the same explicit framing as **arize-instrumentation** when it stops for scope or confirmation:\n\n1. Acknowledge the skill, e.g.: **I found the arize-prompt-optimization skill in this repo** (you may add `skills\u002Farize-prompt-optimization\u002FSKILL.md` if helpful).\n2. Then a clear pause line, e.g.: **A few clarifying questions before I invoke it:**\n3. Ask **minimal** numbered or short bullet questions — only what blocks the next `ax` step in this skill.\n\n## Phase 1: Extract the Current Prompt\n\n### Find LLM spans containing prompts\n\n```bash\n# Sample LLM spans (where prompts live)\nax spans export PROJECT --filter \"attributes.openinference.span.kind = 'LLM'\" -l 10 --stdout\n\n# Filter by model\nax spans export PROJECT --filter \"attributes.llm.model_name = 'gpt-4o'\" -l 10 --stdout\n\n# Filter by span name (e.g., a specific LLM call)\nax spans export PROJECT --filter \"name = 'ChatCompletion'\" -l 10 --stdout\n```\n\n### Export a trace to inspect prompt structure\n\n```bash\n# Export all spans in a trace\nax spans export PROJECT --trace-id TRACE_ID\n\n# Export a single span\nax spans export PROJECT --span-id SPAN_ID\n```\n\n### Extract prompts from exported JSON\n\n```bash\n# Extract structured chat messages (system + user + assistant)\njq '.[0] | {\n  messages: .attributes.llm.input_messages,\n  model: .attributes.llm.model_name\n}' trace_*\u002Fspans.json\n\n# Extract the system prompt specifically\njq '[.[] | select(.attributes.llm.input_messages.roles[]? == \"system\")] | .[0].attributes.llm.input_messages' trace_*\u002Fspans.json\n\n# Extract prompt template and variables\njq '.[0].attributes.llm.prompt_template' trace_*\u002Fspans.json\n\n# Extract from input.value (fallback for non-structured prompts)\njq '.[0].attributes.input.value' trace_*\u002Fspans.json\n```\n\n### Reconstruct the prompt as messages\n\nOnce you have the span data, reconstruct the prompt as a messages array:\n\n```json\n[\n  {\"role\": \"system\", \"content\": \"You are a helpful assistant that...\"},\n  {\"role\": \"user\", \"content\": \"Given {input}, answer the question: {question}\"}\n]\n```\n\nIf the span has `attributes.llm.prompt_template.template`, the prompt uses variables. Preserve these placeholders (`{variable}` or `{{variable}}`) -- they are substituted at runtime.\n\n## Phase 2: Gather Performance Data\n\n### From traces (production feedback)\n\n```bash\n# Find error spans -- these indicate prompt failures\nax spans export PROJECT \\\n  --filter \"status_code = 'ERROR' AND attributes.openinference.span.kind = 'LLM'\" \\\n  -l 20 --stdout\n\n# Find spans with low eval scores\nax spans export PROJECT \\\n  --filter \"annotation.correctness.label = 'incorrect'\" \\\n  -l 20 --stdout\n\n# Find spans with high latency (may indicate overly complex prompts)\nax spans export PROJECT \\\n  --filter \"attributes.openinference.span.kind = 'LLM' AND latency_ms > 10000\" \\\n  -l 20 --stdout\n\n# Export error traces for detailed inspection\nax spans export PROJECT --trace-id TRACE_ID\n```\n\n### From datasets and experiments\n\n```bash\n# Export a dataset (ground truth examples)\nax datasets export DATASET_NAME --space SPACE\n# -> dataset_*\u002Fexamples.json\n\n# Export experiment results (what the LLM produced)\nax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE\n# -> experiment_*\u002Fruns.json\n```\n\n### Merge dataset + experiment for analysis\n\nJoin the two files by `example_id` to see inputs alongside outputs and evaluations:\n\n```bash\n# Count examples and runs\njq 'length' dataset_*\u002Fexamples.json\njq 'length' experiment_*\u002Fruns.json\n\n# View a single joined record\njq -s '\n  .[0] as $dataset |\n  .[1][0] as $run |\n  ($dataset[] | select(.id == $run.example_id)) as $example |\n  {\n    input: $example,\n    output: $run.output,\n    evaluations: $run.evaluations\n  }\n' dataset_*\u002Fexamples.json experiment_*\u002Fruns.json\n\n# Find failed examples (where eval score \u003C threshold)\njq '[.[] | select(.evaluations.correctness.score \u003C 0.5)]' experiment_*\u002Fruns.json\n```\n\n### Identify what to optimize\n\nLook for patterns across failures:\n\n1. **Compare outputs to ground truth**: Where does the LLM output differ from expected?\n2. **Read eval explanations**: `eval.*.explanation` tells you WHY something failed\n3. **Check annotation text**: Human feedback describes specific issues\n4. **Look for verbosity mismatches**: If outputs are too long\u002Fshort vs ground truth\n5. **Check format compliance**: Are outputs in the expected format?\n\n## Phase 3: Optimize the Prompt\n\n### The Optimization Meta-Prompt\n\nUse this template to generate an improved version of the prompt. Fill in the three placeholders and send it to your LLM (GPT-4o, Claude, etc.):\n\n````\nYou are an expert in prompt optimization. Given the original baseline prompt\nand the associated performance data (inputs, outputs, evaluation labels, and\nexplanations), generate a revised version that improves results.\n\nORIGINAL BASELINE PROMPT\n========================\n\n{PASTE_ORIGINAL_PROMPT_HERE}\n\n========================\n\nPERFORMANCE DATA\n================\n\nThe following records show how the current prompt performed. Each record\nincludes the input, the LLM output, and evaluation feedback:\n\n{PASTE_RECORDS_HERE}\n\n================\n\nHOW TO USE THIS DATA\n\n1. Compare outputs: Look at what the LLM generated vs what was expected\n2. Review eval scores: Check which examples scored poorly and why\n3. Examine annotations: Human feedback shows what worked and what didn't\n4. Identify patterns: Look for common issues across multiple examples\n5. Focus on failures: The rows where the output DIFFERS from the expected\n   value are the ones that need fixing\n\nALIGNMENT STRATEGY\n\n- If outputs have extra text or reasoning not present in the ground truth,\n  remove instructions that encourage explanation or verbose reasoning\n- If outputs are missing information, add instructions to include it\n- If outputs are in the wrong format, add explicit format instructions\n- Focus on the rows where the output differs from the target -- these are\n  the failures to fix\n\nRULES\n\nMaintain Structure:\n- Use the same template variables as the current prompt ({var} or {{var}})\n- Don't change sections that are already working\n- Preserve the exact return format instructions from the original prompt\n\nAvoid Overfitting:\n- DO NOT copy examples verbatim into the prompt\n- DO NOT quote specific test data outputs exactly\n- INSTEAD: Extract the ESSENCE of what makes good vs bad outputs\n- INSTEAD: Add general guidelines and principles\n- INSTEAD: If adding few-shot examples, create SYNTHETIC examples that\n  demonstrate the principle, not real data from above\n\nGoal: Create a prompt that generalizes well to new inputs, not one that\nmemorizes the test data.\n\nOUTPUT FORMAT\n\nReturn the revised prompt as a JSON array of messages:\n\n[\n  {\"role\": \"system\", \"content\": \"...\"},\n  {\"role\": \"user\", \"content\": \"...\"}\n]\n\nAlso provide a brief reasoning section (bulleted list) explaining:\n- What problems you found\n- How the revised prompt addresses each one\n````\n\n### Preparing the performance data\n\nFormat the records as a JSON array before pasting into the template:\n\n```bash\n# From dataset + experiment: join and select relevant columns\njq -s '\n  .[0] as $ds |\n  [.[1][] | . as $run |\n    ($ds[] | select(.id == $run.example_id)) as $ex |\n    {\n      input: $ex.input,\n      expected: $ex.expected_output,\n      actual_output: $run.output,\n      eval_score: $run.evaluations.correctness.score,\n      eval_label: $run.evaluations.correctness.label,\n      eval_explanation: $run.evaluations.correctness.explanation\n    }\n  ]\n' dataset_*\u002Fexamples.json experiment_*\u002Fruns.json\n\n# From exported spans: extract input\u002Foutput pairs with annotations\njq '[.[] | select(.attributes.openinference.span.kind == \"LLM\") | {\n  input: .attributes.input.value,\n  output: .attributes.output.value,\n  status: .status_code,\n  model: .attributes.llm.model_name\n}]' trace_*\u002Fspans.json\n```\n\n### Applying the revised prompt\n\nAfter the LLM returns the revised messages array:\n\n1. Compare the original and revised prompts side by side\n2. Verify all template variables are preserved\n3. Check that format instructions are intact\n4. Test on a few examples before full deployment\n\n## Phase 4: Iterate\n\n### The optimization loop\n\n```\n1. Extract prompt    -> Phase 1 (once)\n2. Run experiment    -> ax experiments create ...\n3. Export results    -> ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE\n4. Analyze failures  -> jq to find low scores\n5. Run meta-prompt   -> Phase 3 with new failure data\n6. Apply revised prompt\n7. Repeat from step 2\n```\n\n### Measure improvement\n\n```bash\n# Compare scores across experiments\n# Experiment A (baseline)\njq '[.[] | .evaluations.correctness.score] | add \u002F length' experiment_a\u002Fruns.json\n\n# Experiment B (optimized)\njq '[.[] | .evaluations.correctness.score] | add \u002F length' experiment_b\u002Fruns.json\n\n# Find examples that flipped from fail to pass\njq -s '\n  [.[0][] | select(.evaluations.correctness.label == \"incorrect\")] as $fails |\n  [.[1][] | select(.evaluations.correctness.label == \"correct\") |\n    select(.example_id as $id | $fails | any(.example_id == $id))\n  ] | length\n' experiment_a\u002Fruns.json experiment_b\u002Fruns.json\n```\n\n### A\u002FB compare two prompts\n\n1. Create two experiments against the same dataset, each using a different prompt version\n2. Export both: `ax experiments export EXP_A` and `ax experiments export EXP_B`\n3. Compare average scores, failure rates, and specific example flips\n4. Check for regressions -- examples that passed with prompt A but fail with prompt B\n\n## Prompt Engineering Best Practices\n\nApply these when writing or revising prompts:\n\n| Technique | When to apply | Example |\n|-----------|--------------|---------|\n| Clear, detailed instructions | Output is vague or off-topic | \"Classify the sentiment as exactly one of: positive, negative, neutral\" |\n| Instructions at the beginning | Model ignores later instructions | Put the task description before examples |\n| Step-by-step breakdowns | Complex multi-step processes | \"First extract entities, then classify each, then summarize\" |\n| Specific personas | Need consistent style\u002Ftone | \"You are a senior financial analyst writing for institutional investors\" |\n| Delimiter tokens | Sections blend together | Use `---`, `###`, or XML tags to separate input from instructions |\n| Few-shot examples | Output format needs clarification | Show 2-3 synthetic input\u002Foutput pairs |\n| Output length specifications | Responses are too long or short | \"Respond in exactly 2-3 sentences\" |\n| Reasoning instructions | Accuracy is critical | \"Think step by step before answering\" |\n| \"I don't know\" guidelines | Hallucination is a risk | \"If the answer is not in the provided context, say 'I don't have enough information'\" |\n\n### Variable preservation\n\nWhen optimizing prompts that use template variables:\n\n- **Single braces** (`{variable}`): Python f-string \u002F Jinja style. Most common in Arize.\n- **Double braces** (`{{variable}}`): Mustache style. Used when the framework requires it.\n- Never add or remove variable placeholders during optimization\n- Never rename variables -- the runtime substitution depends on exact names\n- If adding few-shot examples, use literal values, not variable placeholders\n\n## Workflows\n\n### Optimize a prompt from a failing trace\n\n1. Find failing traces:\n   ```bash\n   ax traces list PROJECT --filter \"status_code = 'ERROR'\" --limit 5\n   ```\n2. Export the trace:\n   ```bash\n   ax spans export PROJECT --trace-id TRACE_ID\n   ```\n3. Extract the prompt from the LLM span:\n   ```bash\n   jq '[.[] | select(.attributes.openinference.span.kind == \"LLM\")][0] | {\n     messages: .attributes.llm.input_messages,\n     template: .attributes.llm.prompt_template,\n     output: .attributes.output.value,\n     error: .attributes.exception.message\n   }' trace_*\u002Fspans.json\n   ```\n4. Identify what failed from the error message or output\n5. Fill in the optimization meta-prompt (Phase 3) with the prompt and error context\n6. Apply the revised prompt\n\n### Optimize using a dataset and experiment\n\n1. Find the dataset and experiment:\n   ```bash\n   ax datasets list --space SPACE\n   ax experiments list --dataset DATASET_NAME --space SPACE\n   ```\n2. Export both:\n   ```bash\n   ax datasets export DATASET_NAME --space SPACE\n   ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE\n   ```\n3. Prepare the joined data for the meta-prompt\n4. Run the optimization meta-prompt\n5. Create a new experiment with the revised prompt to measure improvement\n\n### Debug a prompt that produces wrong format\n\n1. Export spans where the output format is wrong:\n   ```bash\n   ax spans export PROJECT \\\n     --filter \"attributes.openinference.span.kind = 'LLM' AND annotation.format.label = 'incorrect'\" \\\n     -l 10 --stdout > bad_format.json\n   ```\n2. Look at what the LLM is producing vs what was expected\n3. Add explicit format instructions to the prompt (JSON schema, examples, delimiters)\n4. Common fix: add a few-shot example showing the exact desired output format\n\n### Reduce hallucination in a RAG prompt\n\n1. Find traces where the model hallucinated:\n   ```bash\n   ax spans export PROJECT \\\n     --filter \"annotation.faithfulness.label = 'unfaithful'\" \\\n     -l 20 --stdout\n   ```\n2. Export and inspect the retriever + LLM spans together:\n   ```bash\n   ax spans export PROJECT --trace-id TRACE_ID\n   jq '[.[] | {kind: .attributes.openinference.span.kind, name, input: .attributes.input.value, output: .attributes.output.value}]' trace_*\u002Fspans.json\n   ```\n3. Check if the retrieved context actually contained the answer\n4. Add grounding instructions to the system prompt: \"Only use information from the provided context. If the answer is not in the context, say so.\"\n\n## Troubleshooting\n\n| Problem | Solution |\n|---------|----------|\n| `ax: command not found` | See [references\u002Fax-setup.md](references\u002Fax-setup.md) |\n| `No profile found` | No profile is configured. See [references\u002Fax-profiles.md](references\u002Fax-profiles.md) to create one. |\n| No `input_messages` on span | Check span kind -- Chain\u002FAgent spans store prompts on child LLM spans, not on themselves |\n| Prompt template is `null` | Not all instrumentations emit `prompt_template`. Use `input_messages` or `input.value` instead |\n| Variables lost after optimization | Verify the revised prompt preserves all `{var}` placeholders from the original |\n| Optimization makes things worse | Check for overfitting -- the meta-prompt may have memorized test data. Ensure few-shot examples are synthetic |\n| No eval\u002Fannotation columns | Run evaluations first (via Arize UI or SDK), then re-export |\n| Experiment output column not found | The column name is `{experiment_name}.output` -- check exact experiment name via `ax experiments get` |\n| `jq` errors on span JSON | Ensure you're targeting the correct file path (e.g., `trace_*\u002Fspans.json`) |\n",{"data":45,"body":49},{"name":4,"description":6,"metadata":46,"compatibility":48},{"author":8,"version":47},"1.0","Requires the ax CLI and a configured Arize profile.",{"type":50,"children":51},"root",[52,61,135,142,186,192,199,204,462,468,557,563,568,812,818,831,843,982,988,1035,1088,1094,1100,1317,1323,1411,1417,1642,1648,1653,1830,1857,1863,1869,2149,2155,2270,2276,2289,2547,2553,2558,2618,2624,2630,2635,2645,2651,2656,2904,2910,2915,2938,2944,2950,2959,2965,3130,3136,3173,3179,3184,3388,3394,3399,3449,3455,3461,3659,3665,3823,3829,3932,3938,4096,4102,4328],{"type":53,"tag":54,"props":55,"children":57},"element","h1",{"id":56},"arize-prompt-optimization-skill",[58],{"type":59,"value":60},"text","Arize Prompt Optimization Skill",{"type":53,"tag":62,"props":63,"children":64},"blockquote",{},[65],{"type":53,"tag":66,"props":67,"children":68},"p",{},[69,80,82,88,90,96,98,103,105,111,113,118,119,125,127,133],{"type":53,"tag":70,"props":71,"children":72},"strong",{},[73],{"type":53,"tag":74,"props":75,"children":77},"code",{"className":76},[],[78],{"type":59,"value":79},"SPACE",{"type":59,"value":81}," — All ",{"type":53,"tag":74,"props":83,"children":85},{"className":84},[],[86],{"type":59,"value":87},"--space",{"type":59,"value":89}," flags and the ",{"type":53,"tag":74,"props":91,"children":93},{"className":92},[],[94],{"type":59,"value":95},"ARIZE_SPACE",{"type":59,"value":97}," env var accept a space ",{"type":53,"tag":70,"props":99,"children":100},{},[101],{"type":59,"value":102},"name",{"type":59,"value":104}," (e.g., ",{"type":53,"tag":74,"props":106,"children":108},{"className":107},[],[109],{"type":59,"value":110},"my-workspace",{"type":59,"value":112},") or a base64 space ",{"type":53,"tag":70,"props":114,"children":115},{},[116],{"type":59,"value":117},"ID",{"type":59,"value":104},{"type":53,"tag":74,"props":120,"children":122},{"className":121},[],[123],{"type":59,"value":124},"U3BhY2U6...",{"type":59,"value":126},"). Find yours with ",{"type":53,"tag":74,"props":128,"children":130},{"className":129},[],[131],{"type":59,"value":132},"ax spaces list",{"type":59,"value":134},".",{"type":53,"tag":136,"props":137,"children":139},"h2",{"id":138},"related-skills",[140],{"type":59,"value":141},"Related skills",{"type":53,"tag":143,"props":144,"children":145},"ul",{},[146],{"type":53,"tag":147,"props":148,"children":149},"li",{},[150,155,157,162,164,170,172,178,180,184],{"type":53,"tag":70,"props":151,"children":152},{},[153],{"type":59,"value":154},"arize-prompts",{"type":59,"value":156},": Create, version, and label prompts in ",{"type":53,"tag":70,"props":158,"children":159},{},[160],{"type":59,"value":161},"Prompt Hub",{"type":59,"value":163}," with ",{"type":53,"tag":74,"props":165,"children":167},{"className":166},[],[168],{"type":59,"value":169},"ax prompts",{"type":59,"value":171}," (JSON messages, providers, labels such as ",{"type":53,"tag":74,"props":173,"children":175},{"className":174},[],[176],{"type":59,"value":177},"production",{"type":59,"value":179},"). Use that skill when the artifact should live in Arize; use ",{"type":53,"tag":70,"props":181,"children":182},{},[183],{"type":59,"value":4},{"type":59,"value":185}," below to improve prompt text from traces, datasets, and experiments.",{"type":53,"tag":136,"props":187,"children":189},{"id":188},"concepts",[190],{"type":59,"value":191},"Concepts",{"type":53,"tag":193,"props":194,"children":196},"h3",{"id":195},"where-prompts-live-in-trace-data",[197],{"type":59,"value":198},"Where Prompts Live in Trace Data",{"type":53,"tag":66,"props":200,"children":201},{},[202],{"type":59,"value":203},"LLM applications emit spans following OpenInference semantic conventions. Prompts are stored in different span attributes depending on the span kind and instrumentation:",{"type":53,"tag":205,"props":206,"children":207},"table",{},[208,232],{"type":53,"tag":209,"props":210,"children":211},"thead",{},[212],{"type":53,"tag":213,"props":214,"children":215},"tr",{},[216,222,227],{"type":53,"tag":217,"props":218,"children":219},"th",{},[220],{"type":59,"value":221},"Column",{"type":53,"tag":217,"props":223,"children":224},{},[225],{"type":59,"value":226},"What it contains",{"type":53,"tag":217,"props":228,"children":229},{},[230],{"type":59,"value":231},"When to use",{"type":53,"tag":233,"props":234,"children":235},"tbody",{},[236,264,314,336,358,396,418,440],{"type":53,"tag":213,"props":237,"children":238},{},[239,249,254],{"type":53,"tag":240,"props":241,"children":242},"td",{},[243],{"type":53,"tag":74,"props":244,"children":246},{"className":245},[],[247],{"type":59,"value":248},"attributes.llm.input_messages",{"type":53,"tag":240,"props":250,"children":251},{},[252],{"type":59,"value":253},"Structured chat messages (system, user, assistant, tool) in role-based format",{"type":53,"tag":240,"props":255,"children":256},{},[257,262],{"type":53,"tag":70,"props":258,"children":259},{},[260],{"type":59,"value":261},"Primary source",{"type":59,"value":263}," for chat-based LLM prompts",{"type":53,"tag":213,"props":265,"children":266},{},[267,276,309],{"type":53,"tag":240,"props":268,"children":269},{},[270],{"type":53,"tag":74,"props":271,"children":273},{"className":272},[],[274],{"type":59,"value":275},"attributes.llm.input_messages.roles",{"type":53,"tag":240,"props":277,"children":278},{},[279,281,287,289,295,296,302,303],{"type":59,"value":280},"Array of roles: ",{"type":53,"tag":74,"props":282,"children":284},{"className":283},[],[285],{"type":59,"value":286},"system",{"type":59,"value":288},", ",{"type":53,"tag":74,"props":290,"children":292},{"className":291},[],[293],{"type":59,"value":294},"user",{"type":59,"value":288},{"type":53,"tag":74,"props":297,"children":299},{"className":298},[],[300],{"type":59,"value":301},"assistant",{"type":59,"value":288},{"type":53,"tag":74,"props":304,"children":306},{"className":305},[],[307],{"type":59,"value":308},"tool",{"type":53,"tag":240,"props":310,"children":311},{},[312],{"type":59,"value":313},"Extract individual message roles",{"type":53,"tag":213,"props":315,"children":316},{},[317,326,331],{"type":53,"tag":240,"props":318,"children":319},{},[320],{"type":53,"tag":74,"props":321,"children":323},{"className":322},[],[324],{"type":59,"value":325},"attributes.llm.input_messages.contents",{"type":53,"tag":240,"props":327,"children":328},{},[329],{"type":59,"value":330},"Array of message content strings",{"type":53,"tag":240,"props":332,"children":333},{},[334],{"type":59,"value":335},"Extract message text",{"type":53,"tag":213,"props":337,"children":338},{},[339,348,353],{"type":53,"tag":240,"props":340,"children":341},{},[342],{"type":53,"tag":74,"props":343,"children":345},{"className":344},[],[346],{"type":59,"value":347},"attributes.input.value",{"type":53,"tag":240,"props":349,"children":350},{},[351],{"type":59,"value":352},"Serialized prompt or user question (generic, all span kinds)",{"type":53,"tag":240,"props":354,"children":355},{},[356],{"type":59,"value":357},"Fallback when structured messages are not available",{"type":53,"tag":213,"props":359,"children":360},{},[361,370,391],{"type":53,"tag":240,"props":362,"children":363},{},[364],{"type":53,"tag":74,"props":365,"children":367},{"className":366},[],[368],{"type":59,"value":369},"attributes.llm.prompt_template.template",{"type":53,"tag":240,"props":371,"children":372},{},[373,375,381,383,389],{"type":59,"value":374},"Template with ",{"type":53,"tag":74,"props":376,"children":378},{"className":377},[],[379],{"type":59,"value":380},"{variable}",{"type":59,"value":382}," placeholders (e.g., ",{"type":53,"tag":74,"props":384,"children":386},{"className":385},[],[387],{"type":59,"value":388},"\"Answer {question} using {context}\"",{"type":59,"value":390},")",{"type":53,"tag":240,"props":392,"children":393},{},[394],{"type":59,"value":395},"When the app uses prompt templates",{"type":53,"tag":213,"props":397,"children":398},{},[399,408,413],{"type":53,"tag":240,"props":400,"children":401},{},[402],{"type":53,"tag":74,"props":403,"children":405},{"className":404},[],[406],{"type":59,"value":407},"attributes.llm.prompt_template.variables",{"type":53,"tag":240,"props":409,"children":410},{},[411],{"type":59,"value":412},"Template variable values (JSON object)",{"type":53,"tag":240,"props":414,"children":415},{},[416],{"type":59,"value":417},"See what values were substituted into the template",{"type":53,"tag":213,"props":419,"children":420},{},[421,430,435],{"type":53,"tag":240,"props":422,"children":423},{},[424],{"type":53,"tag":74,"props":425,"children":427},{"className":426},[],[428],{"type":59,"value":429},"attributes.output.value",{"type":53,"tag":240,"props":431,"children":432},{},[433],{"type":59,"value":434},"Model response text",{"type":53,"tag":240,"props":436,"children":437},{},[438],{"type":59,"value":439},"See what the LLM produced",{"type":53,"tag":213,"props":441,"children":442},{},[443,452,457],{"type":53,"tag":240,"props":444,"children":445},{},[446],{"type":53,"tag":74,"props":447,"children":449},{"className":448},[],[450],{"type":59,"value":451},"attributes.llm.output_messages",{"type":53,"tag":240,"props":453,"children":454},{},[455],{"type":59,"value":456},"Structured model output (including tool calls)",{"type":53,"tag":240,"props":458,"children":459},{},[460],{"type":59,"value":461},"Inspect tool-calling responses",{"type":53,"tag":193,"props":463,"children":465},{"id":464},"finding-prompts-by-span-kind",[466],{"type":59,"value":467},"Finding Prompts by Span Kind",{"type":53,"tag":143,"props":469,"children":470},{},[471,510,534],{"type":53,"tag":147,"props":472,"children":473},{},[474,479,481,487,489,494,496,501,503,508],{"type":53,"tag":70,"props":475,"children":476},{},[477],{"type":59,"value":478},"LLM span",{"type":59,"value":480}," (",{"type":53,"tag":74,"props":482,"children":484},{"className":483},[],[485],{"type":59,"value":486},"attributes.openinference.span.kind = 'LLM'",{"type":59,"value":488},"): Check ",{"type":53,"tag":74,"props":490,"children":492},{"className":491},[],[493],{"type":59,"value":248},{"type":59,"value":495}," for structured chat messages, OR ",{"type":53,"tag":74,"props":497,"children":499},{"className":498},[],[500],{"type":59,"value":347},{"type":59,"value":502}," for a serialized prompt. Check ",{"type":53,"tag":74,"props":504,"children":506},{"className":505},[],[507],{"type":59,"value":369},{"type":59,"value":509}," for the template.",{"type":53,"tag":147,"props":511,"children":512},{},[513,518,520,525,527,532],{"type":53,"tag":70,"props":514,"children":515},{},[516],{"type":59,"value":517},"Chain\u002FAgent span",{"type":59,"value":519},": ",{"type":53,"tag":74,"props":521,"children":523},{"className":522},[],[524],{"type":59,"value":347},{"type":59,"value":526}," contains the user's question. The actual LLM prompt lives on ",{"type":53,"tag":70,"props":528,"children":529},{},[530],{"type":59,"value":531},"child LLM spans",{"type":59,"value":533}," -- navigate down the trace tree.",{"type":53,"tag":147,"props":535,"children":536},{},[537,542,543,548,550,555],{"type":53,"tag":70,"props":538,"children":539},{},[540],{"type":59,"value":541},"Tool span",{"type":59,"value":519},{"type":53,"tag":74,"props":544,"children":546},{"className":545},[],[547],{"type":59,"value":347},{"type":59,"value":549}," has tool input, ",{"type":53,"tag":74,"props":551,"children":553},{"className":552},[],[554],{"type":59,"value":429},{"type":59,"value":556}," has tool result. Not typically where prompts live.",{"type":53,"tag":193,"props":558,"children":560},{"id":559},"performance-signal-columns",[561],{"type":59,"value":562},"Performance Signal Columns",{"type":53,"tag":66,"props":564,"children":565},{},[566],{"type":59,"value":567},"These columns carry the feedback data used for optimization:",{"type":53,"tag":205,"props":569,"children":570},{},[571,592],{"type":53,"tag":209,"props":572,"children":573},{},[574],{"type":53,"tag":213,"props":575,"children":576},{},[577,582,587],{"type":53,"tag":217,"props":578,"children":579},{},[580],{"type":59,"value":581},"Column pattern",{"type":53,"tag":217,"props":583,"children":584},{},[585],{"type":59,"value":586},"Source",{"type":53,"tag":217,"props":588,"children":589},{},[590],{"type":59,"value":591},"What it tells you",{"type":53,"tag":233,"props":593,"children":594},{},[595,638,659,680,702,723,749,770,790],{"type":53,"tag":213,"props":596,"children":597},{},[598,607,612],{"type":53,"tag":240,"props":599,"children":600},{},[601],{"type":53,"tag":74,"props":602,"children":604},{"className":603},[],[605],{"type":59,"value":606},"annotation.\u003Cname>.label",{"type":53,"tag":240,"props":608,"children":609},{},[610],{"type":59,"value":611},"Human reviewers",{"type":53,"tag":240,"props":613,"children":614},{},[615,617,623,624,630,631,637],{"type":59,"value":616},"Categorical grade (e.g., ",{"type":53,"tag":74,"props":618,"children":620},{"className":619},[],[621],{"type":59,"value":622},"correct",{"type":59,"value":288},{"type":53,"tag":74,"props":625,"children":627},{"className":626},[],[628],{"type":59,"value":629},"incorrect",{"type":59,"value":288},{"type":53,"tag":74,"props":632,"children":634},{"className":633},[],[635],{"type":59,"value":636},"partial",{"type":59,"value":390},{"type":53,"tag":213,"props":639,"children":640},{},[641,650,654],{"type":53,"tag":240,"props":642,"children":643},{},[644],{"type":53,"tag":74,"props":645,"children":647},{"className":646},[],[648],{"type":59,"value":649},"annotation.\u003Cname>.score",{"type":53,"tag":240,"props":651,"children":652},{},[653],{"type":59,"value":611},{"type":53,"tag":240,"props":655,"children":656},{},[657],{"type":59,"value":658},"Numeric quality score (e.g., 0.0 - 1.0)",{"type":53,"tag":213,"props":660,"children":661},{},[662,671,675],{"type":53,"tag":240,"props":663,"children":664},{},[665],{"type":53,"tag":74,"props":666,"children":668},{"className":667},[],[669],{"type":59,"value":670},"annotation.\u003Cname>.text",{"type":53,"tag":240,"props":672,"children":673},{},[674],{"type":59,"value":611},{"type":53,"tag":240,"props":676,"children":677},{},[678],{"type":59,"value":679},"Freeform explanation of the grade",{"type":53,"tag":213,"props":681,"children":682},{},[683,692,697],{"type":53,"tag":240,"props":684,"children":685},{},[686],{"type":53,"tag":74,"props":687,"children":689},{"className":688},[],[690],{"type":59,"value":691},"eval.\u003Cname>.label",{"type":53,"tag":240,"props":693,"children":694},{},[695],{"type":59,"value":696},"LLM-as-judge evals",{"type":53,"tag":240,"props":698,"children":699},{},[700],{"type":59,"value":701},"Automated categorical assessment",{"type":53,"tag":213,"props":703,"children":704},{},[705,714,718],{"type":53,"tag":240,"props":706,"children":707},{},[708],{"type":53,"tag":74,"props":709,"children":711},{"className":710},[],[712],{"type":59,"value":713},"eval.\u003Cname>.score",{"type":53,"tag":240,"props":715,"children":716},{},[717],{"type":59,"value":696},{"type":53,"tag":240,"props":719,"children":720},{},[721],{"type":59,"value":722},"Automated numeric score",{"type":53,"tag":213,"props":724,"children":725},{},[726,735,739],{"type":53,"tag":240,"props":727,"children":728},{},[729],{"type":53,"tag":74,"props":730,"children":732},{"className":731},[],[733],{"type":59,"value":734},"eval.\u003Cname>.explanation",{"type":53,"tag":240,"props":736,"children":737},{},[738],{"type":59,"value":696},{"type":53,"tag":240,"props":740,"children":741},{},[742,744],{"type":59,"value":743},"Why the eval gave that score -- ",{"type":53,"tag":70,"props":745,"children":746},{},[747],{"type":59,"value":748},"most valuable for optimization",{"type":53,"tag":213,"props":750,"children":751},{},[752,760,765],{"type":53,"tag":240,"props":753,"children":754},{},[755],{"type":53,"tag":74,"props":756,"children":758},{"className":757},[],[759],{"type":59,"value":347},{"type":53,"tag":240,"props":761,"children":762},{},[763],{"type":59,"value":764},"Trace data",{"type":53,"tag":240,"props":766,"children":767},{},[768],{"type":59,"value":769},"What went into the LLM",{"type":53,"tag":213,"props":771,"children":772},{},[773,781,785],{"type":53,"tag":240,"props":774,"children":775},{},[776],{"type":53,"tag":74,"props":777,"children":779},{"className":778},[],[780],{"type":59,"value":429},{"type":53,"tag":240,"props":782,"children":783},{},[784],{"type":59,"value":764},{"type":53,"tag":240,"props":786,"children":787},{},[788],{"type":59,"value":789},"What the LLM produced",{"type":53,"tag":213,"props":791,"children":792},{},[793,802,807],{"type":53,"tag":240,"props":794,"children":795},{},[796],{"type":53,"tag":74,"props":797,"children":799},{"className":798},[],[800],{"type":59,"value":801},"{experiment_name}.output",{"type":53,"tag":240,"props":803,"children":804},{},[805],{"type":59,"value":806},"Experiment runs",{"type":53,"tag":240,"props":808,"children":809},{},[810],{"type":59,"value":811},"Output from a specific experiment",{"type":53,"tag":136,"props":813,"children":815},{"id":814},"prerequisites",[816],{"type":59,"value":817},"Prerequisites",{"type":53,"tag":66,"props":819,"children":820},{},[821,823,829],{"type":59,"value":822},"Proceed directly with the task — run the ",{"type":53,"tag":74,"props":824,"children":826},{"className":825},[],[827],{"type":59,"value":828},"ax",{"type":59,"value":830}," command you need. Do NOT check versions, env vars, or profiles upfront.",{"type":53,"tag":66,"props":832,"children":833},{},[834,836,841],{"type":59,"value":835},"If an ",{"type":53,"tag":74,"props":837,"children":839},{"className":838},[],[840],{"type":59,"value":828},{"type":59,"value":842}," command fails, troubleshoot based on the error:",{"type":53,"tag":143,"props":844,"children":845},{},[846,863,898,910,923,943],{"type":53,"tag":147,"props":847,"children":848},{},[849,855,857],{"type":53,"tag":74,"props":850,"children":852},{"className":851},[],[853],{"type":59,"value":854},"command not found",{"type":59,"value":856}," or version error → see ",{"type":53,"tag":858,"props":859,"children":861},"a",{"href":860},"references\u002Fax-setup.md",[862],{"type":59,"value":860},{"type":53,"tag":147,"props":864,"children":865},{},[866,872,874,880,882,887,889,896],{"type":53,"tag":74,"props":867,"children":869},{"className":868},[],[870],{"type":59,"value":871},"401 Unauthorized",{"type":59,"value":873}," \u002F missing API key → run ",{"type":53,"tag":74,"props":875,"children":877},{"className":876},[],[878],{"type":59,"value":879},"ax profiles show",{"type":59,"value":881}," to inspect the current profile. If the profile is missing or the API key is wrong, follow ",{"type":53,"tag":858,"props":883,"children":885},{"href":884},"references\u002Fax-profiles.md",[886],{"type":59,"value":884},{"type":59,"value":888}," to create\u002Fupdate it. If the user doesn't have their key, direct them to ",{"type":53,"tag":858,"props":890,"children":894},{"href":891,"rel":892},"https:\u002F\u002Fapp.arize.com\u002Fadmin",[893],"nofollow",[895],{"type":59,"value":891},{"type":59,"value":897}," > API Keys",{"type":53,"tag":147,"props":899,"children":900},{},[901,903,908],{"type":59,"value":902},"Space unknown → run ",{"type":53,"tag":74,"props":904,"children":906},{"className":905},[],[907],{"type":59,"value":132},{"type":59,"value":909}," to pick by name, or ask the user",{"type":53,"tag":147,"props":911,"children":912},{},[913,915,921],{"type":59,"value":914},"Project unclear → ask the user, or run ",{"type":53,"tag":74,"props":916,"children":918},{"className":917},[],[919],{"type":59,"value":920},"ax projects list -o json --limit 100",{"type":59,"value":922}," and present as selectable options",{"type":53,"tag":147,"props":924,"children":925},{},[926,928,934,936,941],{"type":59,"value":927},"LLM provider call fails (missing provider credentials) → run ",{"type":53,"tag":74,"props":929,"children":931},{"className":930},[],[932],{"type":59,"value":933},"ax ai-integrations list --space SPACE",{"type":59,"value":935}," to check for platform-managed credentials. If none exist, use the ",{"type":53,"tag":70,"props":937,"children":938},{},[939],{"type":59,"value":940},"arize-ai-provider-integration",{"type":59,"value":942}," skill — never ask the user to paste a provider key into chat.",{"type":53,"tag":147,"props":944,"children":945},{},[946,951,953,959,961,967,969,975,977,981],{"type":53,"tag":70,"props":947,"children":948},{},[949],{"type":59,"value":950},"Security:",{"type":59,"value":952}," Never read ",{"type":53,"tag":74,"props":954,"children":956},{"className":955},[],[957],{"type":59,"value":958},".env",{"type":59,"value":960}," files or search the filesystem for credentials. Use ",{"type":53,"tag":74,"props":962,"children":964},{"className":963},[],[965],{"type":59,"value":966},"ax profiles",{"type":59,"value":968}," for Arize credentials and ",{"type":53,"tag":74,"props":970,"children":972},{"className":971},[],[973],{"type":59,"value":974},"ax ai-integrations",{"type":59,"value":976}," for LLM provider keys. Never ask the user to paste secrets into chat. For missing credentials, see ",{"type":53,"tag":858,"props":978,"children":979},{"href":884},[980],{"type":59,"value":884},{"type":59,"value":134},{"type":53,"tag":193,"props":983,"children":985},{"id":984},"when-you-must-ask-the-user-first",[986],{"type":59,"value":987},"When you must ask the user first",{"type":53,"tag":66,"props":989,"children":990},{},[991,993,998,999,1005,1006,1012,1013,1019,1021,1026,1028,1033],{"type":59,"value":992},"Still prefer ",{"type":53,"tag":74,"props":994,"children":996},{"className":995},[],[997],{"type":59,"value":132},{"type":59,"value":288},{"type":53,"tag":74,"props":1000,"children":1002},{"className":1001},[],[1003],{"type":59,"value":1004},"ax projects list",{"type":59,"value":288},{"type":53,"tag":74,"props":1007,"children":1009},{"className":1008},[],[1010],{"type":59,"value":1011},"ax datasets list",{"type":59,"value":288},{"type":53,"tag":74,"props":1014,"children":1016},{"className":1015},[],[1017],{"type":59,"value":1018},"ax experiments list",{"type":59,"value":1020},", and exports over open-ended questions. If you still cannot proceed (e.g. multiple projects match the name the user gave, unclear trace vs experiment path, or destructive scope), ",{"type":53,"tag":70,"props":1022,"children":1023},{},[1024],{"type":59,"value":1025},"do not",{"type":59,"value":1027}," jump straight into questions — use the same explicit framing as ",{"type":53,"tag":70,"props":1029,"children":1030},{},[1031],{"type":59,"value":1032},"arize-instrumentation",{"type":59,"value":1034}," when it stops for scope or confirmation:",{"type":53,"tag":1036,"props":1037,"children":1038},"ol",{},[1039,1059,1069],{"type":53,"tag":147,"props":1040,"children":1041},{},[1042,1044,1049,1051,1057],{"type":59,"value":1043},"Acknowledge the skill, e.g.: ",{"type":53,"tag":70,"props":1045,"children":1046},{},[1047],{"type":59,"value":1048},"I found the arize-prompt-optimization skill in this repo",{"type":59,"value":1050}," (you may add ",{"type":53,"tag":74,"props":1052,"children":1054},{"className":1053},[],[1055],{"type":59,"value":1056},"skills\u002Farize-prompt-optimization\u002FSKILL.md",{"type":59,"value":1058}," if helpful).",{"type":53,"tag":147,"props":1060,"children":1061},{},[1062,1064],{"type":59,"value":1063},"Then a clear pause line, e.g.: ",{"type":53,"tag":70,"props":1065,"children":1066},{},[1067],{"type":59,"value":1068},"A few clarifying questions before I invoke it:",{"type":53,"tag":147,"props":1070,"children":1071},{},[1072,1074,1079,1081,1086],{"type":59,"value":1073},"Ask ",{"type":53,"tag":70,"props":1075,"children":1076},{},[1077],{"type":59,"value":1078},"minimal",{"type":59,"value":1080}," numbered or short bullet questions — only what blocks the next ",{"type":53,"tag":74,"props":1082,"children":1084},{"className":1083},[],[1085],{"type":59,"value":828},{"type":59,"value":1087}," step in this skill.",{"type":53,"tag":136,"props":1089,"children":1091},{"id":1090},"phase-1-extract-the-current-prompt",[1092],{"type":59,"value":1093},"Phase 1: Extract the Current Prompt",{"type":53,"tag":193,"props":1095,"children":1097},{"id":1096},"find-llm-spans-containing-prompts",[1098],{"type":59,"value":1099},"Find LLM spans containing prompts",{"type":53,"tag":1101,"props":1102,"children":1107},"pre",{"className":1103,"code":1104,"language":1105,"meta":1106,"style":1106},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Sample LLM spans (where prompts live)\nax spans export PROJECT --filter \"attributes.openinference.span.kind = 'LLM'\" -l 10 --stdout\n\n# Filter by model\nax spans export PROJECT --filter \"attributes.llm.model_name = 'gpt-4o'\" -l 10 --stdout\n\n# Filter by span name (e.g., a specific LLM call)\nax spans export PROJECT --filter \"name = 'ChatCompletion'\" -l 10 --stdout\n","bash","",[1108],{"type":53,"tag":74,"props":1109,"children":1110},{"__ignoreMap":1106},[1111,1123,1184,1194,1203,1251,1259,1268],{"type":53,"tag":1112,"props":1113,"children":1116},"span",{"class":1114,"line":1115},"line",1,[1117],{"type":53,"tag":1112,"props":1118,"children":1120},{"style":1119},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[1121],{"type":59,"value":1122},"# Sample LLM spans (where prompts live)\n",{"type":53,"tag":1112,"props":1124,"children":1126},{"class":1114,"line":1125},2,[1127,1132,1138,1143,1148,1153,1159,1163,1168,1173,1179],{"type":53,"tag":1112,"props":1128,"children":1130},{"style":1129},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1131],{"type":59,"value":828},{"type":53,"tag":1112,"props":1133,"children":1135},{"style":1134},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1136],{"type":59,"value":1137}," spans",{"type":53,"tag":1112,"props":1139,"children":1140},{"style":1134},[1141],{"type":59,"value":1142}," export",{"type":53,"tag":1112,"props":1144,"children":1145},{"style":1134},[1146],{"type":59,"value":1147}," PROJECT",{"type":53,"tag":1112,"props":1149,"children":1150},{"style":1134},[1151],{"type":59,"value":1152}," --filter",{"type":53,"tag":1112,"props":1154,"children":1156},{"style":1155},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1157],{"type":59,"value":1158}," \"",{"type":53,"tag":1112,"props":1160,"children":1161},{"style":1134},[1162],{"type":59,"value":486},{"type":53,"tag":1112,"props":1164,"children":1165},{"style":1155},[1166],{"type":59,"value":1167},"\"",{"type":53,"tag":1112,"props":1169,"children":1170},{"style":1134},[1171],{"type":59,"value":1172}," -l",{"type":53,"tag":1112,"props":1174,"children":1176},{"style":1175},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1177],{"type":59,"value":1178}," 10",{"type":53,"tag":1112,"props":1180,"children":1181},{"style":1134},[1182],{"type":59,"value":1183}," --stdout\n",{"type":53,"tag":1112,"props":1185,"children":1187},{"class":1114,"line":1186},3,[1188],{"type":53,"tag":1112,"props":1189,"children":1191},{"emptyLinePlaceholder":1190},true,[1192],{"type":59,"value":1193},"\n",{"type":53,"tag":1112,"props":1195,"children":1197},{"class":1114,"line":1196},4,[1198],{"type":53,"tag":1112,"props":1199,"children":1200},{"style":1119},[1201],{"type":59,"value":1202},"# Filter by model\n",{"type":53,"tag":1112,"props":1204,"children":1205},{"class":1114,"line":27},[1206,1210,1214,1218,1222,1226,1230,1235,1239,1243,1247],{"type":53,"tag":1112,"props":1207,"children":1208},{"style":1129},[1209],{"type":59,"value":828},{"type":53,"tag":1112,"props":1211,"children":1212},{"style":1134},[1213],{"type":59,"value":1137},{"type":53,"tag":1112,"props":1215,"children":1216},{"style":1134},[1217],{"type":59,"value":1142},{"type":53,"tag":1112,"props":1219,"children":1220},{"style":1134},[1221],{"type":59,"value":1147},{"type":53,"tag":1112,"props":1223,"children":1224},{"style":1134},[1225],{"type":59,"value":1152},{"type":53,"tag":1112,"props":1227,"children":1228},{"style":1155},[1229],{"type":59,"value":1158},{"type":53,"tag":1112,"props":1231,"children":1232},{"style":1134},[1233],{"type":59,"value":1234},"attributes.llm.model_name = 'gpt-4o'",{"type":53,"tag":1112,"props":1236,"children":1237},{"style":1155},[1238],{"type":59,"value":1167},{"type":53,"tag":1112,"props":1240,"children":1241},{"style":1134},[1242],{"type":59,"value":1172},{"type":53,"tag":1112,"props":1244,"children":1245},{"style":1175},[1246],{"type":59,"value":1178},{"type":53,"tag":1112,"props":1248,"children":1249},{"style":1134},[1250],{"type":59,"value":1183},{"type":53,"tag":1112,"props":1252,"children":1254},{"class":1114,"line":1253},6,[1255],{"type":53,"tag":1112,"props":1256,"children":1257},{"emptyLinePlaceholder":1190},[1258],{"type":59,"value":1193},{"type":53,"tag":1112,"props":1260,"children":1262},{"class":1114,"line":1261},7,[1263],{"type":53,"tag":1112,"props":1264,"children":1265},{"style":1119},[1266],{"type":59,"value":1267},"# Filter by span name (e.g., a specific LLM call)\n",{"type":53,"tag":1112,"props":1269,"children":1271},{"class":1114,"line":1270},8,[1272,1276,1280,1284,1288,1292,1296,1301,1305,1309,1313],{"type":53,"tag":1112,"props":1273,"children":1274},{"style":1129},[1275],{"type":59,"value":828},{"type":53,"tag":1112,"props":1277,"children":1278},{"style":1134},[1279],{"type":59,"value":1137},{"type":53,"tag":1112,"props":1281,"children":1282},{"style":1134},[1283],{"type":59,"value":1142},{"type":53,"tag":1112,"props":1285,"children":1286},{"style":1134},[1287],{"type":59,"value":1147},{"type":53,"tag":1112,"props":1289,"children":1290},{"style":1134},[1291],{"type":59,"value":1152},{"type":53,"tag":1112,"props":1293,"children":1294},{"style":1155},[1295],{"type":59,"value":1158},{"type":53,"tag":1112,"props":1297,"children":1298},{"style":1134},[1299],{"type":59,"value":1300},"name = 'ChatCompletion'",{"type":53,"tag":1112,"props":1302,"children":1303},{"style":1155},[1304],{"type":59,"value":1167},{"type":53,"tag":1112,"props":1306,"children":1307},{"style":1134},[1308],{"type":59,"value":1172},{"type":53,"tag":1112,"props":1310,"children":1311},{"style":1175},[1312],{"type":59,"value":1178},{"type":53,"tag":1112,"props":1314,"children":1315},{"style":1134},[1316],{"type":59,"value":1183},{"type":53,"tag":193,"props":1318,"children":1320},{"id":1319},"export-a-trace-to-inspect-prompt-structure",[1321],{"type":59,"value":1322},"Export a trace to inspect prompt structure",{"type":53,"tag":1101,"props":1324,"children":1326},{"className":1103,"code":1325,"language":1105,"meta":1106,"style":1106},"# Export all spans in a trace\nax spans export PROJECT --trace-id TRACE_ID\n\n# Export a single span\nax spans export PROJECT --span-id SPAN_ID\n",[1327],{"type":53,"tag":74,"props":1328,"children":1329},{"__ignoreMap":1106},[1330,1338,1367,1374,1382],{"type":53,"tag":1112,"props":1331,"children":1332},{"class":1114,"line":1115},[1333],{"type":53,"tag":1112,"props":1334,"children":1335},{"style":1119},[1336],{"type":59,"value":1337},"# Export all spans in a trace\n",{"type":53,"tag":1112,"props":1339,"children":1340},{"class":1114,"line":1125},[1341,1345,1349,1353,1357,1362],{"type":53,"tag":1112,"props":1342,"children":1343},{"style":1129},[1344],{"type":59,"value":828},{"type":53,"tag":1112,"props":1346,"children":1347},{"style":1134},[1348],{"type":59,"value":1137},{"type":53,"tag":1112,"props":1350,"children":1351},{"style":1134},[1352],{"type":59,"value":1142},{"type":53,"tag":1112,"props":1354,"children":1355},{"style":1134},[1356],{"type":59,"value":1147},{"type":53,"tag":1112,"props":1358,"children":1359},{"style":1134},[1360],{"type":59,"value":1361}," --trace-id",{"type":53,"tag":1112,"props":1363,"children":1364},{"style":1134},[1365],{"type":59,"value":1366}," TRACE_ID\n",{"type":53,"tag":1112,"props":1368,"children":1369},{"class":1114,"line":1186},[1370],{"type":53,"tag":1112,"props":1371,"children":1372},{"emptyLinePlaceholder":1190},[1373],{"type":59,"value":1193},{"type":53,"tag":1112,"props":1375,"children":1376},{"class":1114,"line":1196},[1377],{"type":53,"tag":1112,"props":1378,"children":1379},{"style":1119},[1380],{"type":59,"value":1381},"# Export a single span\n",{"type":53,"tag":1112,"props":1383,"children":1384},{"class":1114,"line":27},[1385,1389,1393,1397,1401,1406],{"type":53,"tag":1112,"props":1386,"children":1387},{"style":1129},[1388],{"type":59,"value":828},{"type":53,"tag":1112,"props":1390,"children":1391},{"style":1134},[1392],{"type":59,"value":1137},{"type":53,"tag":1112,"props":1394,"children":1395},{"style":1134},[1396],{"type":59,"value":1142},{"type":53,"tag":1112,"props":1398,"children":1399},{"style":1134},[1400],{"type":59,"value":1147},{"type":53,"tag":1112,"props":1402,"children":1403},{"style":1134},[1404],{"type":59,"value":1405}," --span-id",{"type":53,"tag":1112,"props":1407,"children":1408},{"style":1134},[1409],{"type":59,"value":1410}," SPAN_ID\n",{"type":53,"tag":193,"props":1412,"children":1414},{"id":1413},"extract-prompts-from-exported-json",[1415],{"type":59,"value":1416},"Extract prompts from exported JSON",{"type":53,"tag":1101,"props":1418,"children":1420},{"className":1103,"code":1419,"language":1105,"meta":1106,"style":1106},"# Extract structured chat messages (system + user + assistant)\njq '.[0] | {\n  messages: .attributes.llm.input_messages,\n  model: .attributes.llm.model_name\n}' trace_*\u002Fspans.json\n\n# Extract the system prompt specifically\njq '[.[] | select(.attributes.llm.input_messages.roles[]? == \"system\")] | .[0].attributes.llm.input_messages' trace_*\u002Fspans.json\n\n# Extract prompt template and variables\njq '.[0].attributes.llm.prompt_template' trace_*\u002Fspans.json\n\n# Extract from input.value (fallback for non-structured prompts)\njq '.[0].attributes.input.value' trace_*\u002Fspans.json\n",[1421],{"type":53,"tag":74,"props":1422,"children":1423},{"__ignoreMap":1106},[1424,1432,1450,1458,1466,1495,1502,1510,1542,1550,1559,1592,1600,1609],{"type":53,"tag":1112,"props":1425,"children":1426},{"class":1114,"line":1115},[1427],{"type":53,"tag":1112,"props":1428,"children":1429},{"style":1119},[1430],{"type":59,"value":1431},"# Extract structured chat messages (system + user + assistant)\n",{"type":53,"tag":1112,"props":1433,"children":1434},{"class":1114,"line":1125},[1435,1440,1445],{"type":53,"tag":1112,"props":1436,"children":1437},{"style":1129},[1438],{"type":59,"value":1439},"jq",{"type":53,"tag":1112,"props":1441,"children":1442},{"style":1155},[1443],{"type":59,"value":1444}," '",{"type":53,"tag":1112,"props":1446,"children":1447},{"style":1134},[1448],{"type":59,"value":1449},".[0] | {\n",{"type":53,"tag":1112,"props":1451,"children":1452},{"class":1114,"line":1186},[1453],{"type":53,"tag":1112,"props":1454,"children":1455},{"style":1134},[1456],{"type":59,"value":1457},"  messages: .attributes.llm.input_messages,\n",{"type":53,"tag":1112,"props":1459,"children":1460},{"class":1114,"line":1196},[1461],{"type":53,"tag":1112,"props":1462,"children":1463},{"style":1134},[1464],{"type":59,"value":1465},"  model: .attributes.llm.model_name\n",{"type":53,"tag":1112,"props":1467,"children":1468},{"class":1114,"line":27},[1469,1474,1479,1484,1490],{"type":53,"tag":1112,"props":1470,"children":1471},{"style":1134},[1472],{"type":59,"value":1473},"}",{"type":53,"tag":1112,"props":1475,"children":1476},{"style":1155},[1477],{"type":59,"value":1478},"'",{"type":53,"tag":1112,"props":1480,"children":1481},{"style":1134},[1482],{"type":59,"value":1483}," trace_",{"type":53,"tag":1112,"props":1485,"children":1487},{"style":1486},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1488],{"type":59,"value":1489},"*",{"type":53,"tag":1112,"props":1491,"children":1492},{"style":1134},[1493],{"type":59,"value":1494},"\u002Fspans.json\n",{"type":53,"tag":1112,"props":1496,"children":1497},{"class":1114,"line":1253},[1498],{"type":53,"tag":1112,"props":1499,"children":1500},{"emptyLinePlaceholder":1190},[1501],{"type":59,"value":1193},{"type":53,"tag":1112,"props":1503,"children":1504},{"class":1114,"line":1261},[1505],{"type":53,"tag":1112,"props":1506,"children":1507},{"style":1119},[1508],{"type":59,"value":1509},"# Extract the system prompt specifically\n",{"type":53,"tag":1112,"props":1511,"children":1512},{"class":1114,"line":1270},[1513,1517,1521,1526,1530,1534,1538],{"type":53,"tag":1112,"props":1514,"children":1515},{"style":1129},[1516],{"type":59,"value":1439},{"type":53,"tag":1112,"props":1518,"children":1519},{"style":1155},[1520],{"type":59,"value":1444},{"type":53,"tag":1112,"props":1522,"children":1523},{"style":1134},[1524],{"type":59,"value":1525},"[.[] | select(.attributes.llm.input_messages.roles[]? == \"system\")] | .[0].attributes.llm.input_messages",{"type":53,"tag":1112,"props":1527,"children":1528},{"style":1155},[1529],{"type":59,"value":1478},{"type":53,"tag":1112,"props":1531,"children":1532},{"style":1134},[1533],{"type":59,"value":1483},{"type":53,"tag":1112,"props":1535,"children":1536},{"style":1486},[1537],{"type":59,"value":1489},{"type":53,"tag":1112,"props":1539,"children":1540},{"style":1134},[1541],{"type":59,"value":1494},{"type":53,"tag":1112,"props":1543,"children":1545},{"class":1114,"line":1544},9,[1546],{"type":53,"tag":1112,"props":1547,"children":1548},{"emptyLinePlaceholder":1190},[1549],{"type":59,"value":1193},{"type":53,"tag":1112,"props":1551,"children":1553},{"class":1114,"line":1552},10,[1554],{"type":53,"tag":1112,"props":1555,"children":1556},{"style":1119},[1557],{"type":59,"value":1558},"# Extract prompt template and variables\n",{"type":53,"tag":1112,"props":1560,"children":1562},{"class":1114,"line":1561},11,[1563,1567,1571,1576,1580,1584,1588],{"type":53,"tag":1112,"props":1564,"children":1565},{"style":1129},[1566],{"type":59,"value":1439},{"type":53,"tag":1112,"props":1568,"children":1569},{"style":1155},[1570],{"type":59,"value":1444},{"type":53,"tag":1112,"props":1572,"children":1573},{"style":1134},[1574],{"type":59,"value":1575},".[0].attributes.llm.prompt_template",{"type":53,"tag":1112,"props":1577,"children":1578},{"style":1155},[1579],{"type":59,"value":1478},{"type":53,"tag":1112,"props":1581,"children":1582},{"style":1134},[1583],{"type":59,"value":1483},{"type":53,"tag":1112,"props":1585,"children":1586},{"style":1486},[1587],{"type":59,"value":1489},{"type":53,"tag":1112,"props":1589,"children":1590},{"style":1134},[1591],{"type":59,"value":1494},{"type":53,"tag":1112,"props":1593,"children":1595},{"class":1114,"line":1594},12,[1596],{"type":53,"tag":1112,"props":1597,"children":1598},{"emptyLinePlaceholder":1190},[1599],{"type":59,"value":1193},{"type":53,"tag":1112,"props":1601,"children":1603},{"class":1114,"line":1602},13,[1604],{"type":53,"tag":1112,"props":1605,"children":1606},{"style":1119},[1607],{"type":59,"value":1608},"# Extract from input.value (fallback for non-structured prompts)\n",{"type":53,"tag":1112,"props":1610,"children":1612},{"class":1114,"line":1611},14,[1613,1617,1621,1626,1630,1634,1638],{"type":53,"tag":1112,"props":1614,"children":1615},{"style":1129},[1616],{"type":59,"value":1439},{"type":53,"tag":1112,"props":1618,"children":1619},{"style":1155},[1620],{"type":59,"value":1444},{"type":53,"tag":1112,"props":1622,"children":1623},{"style":1134},[1624],{"type":59,"value":1625},".[0].attributes.input.value",{"type":53,"tag":1112,"props":1627,"children":1628},{"style":1155},[1629],{"type":59,"value":1478},{"type":53,"tag":1112,"props":1631,"children":1632},{"style":1134},[1633],{"type":59,"value":1483},{"type":53,"tag":1112,"props":1635,"children":1636},{"style":1486},[1637],{"type":59,"value":1489},{"type":53,"tag":1112,"props":1639,"children":1640},{"style":1134},[1641],{"type":59,"value":1494},{"type":53,"tag":193,"props":1643,"children":1645},{"id":1644},"reconstruct-the-prompt-as-messages",[1646],{"type":59,"value":1647},"Reconstruct the prompt as messages",{"type":53,"tag":66,"props":1649,"children":1650},{},[1651],{"type":59,"value":1652},"Once you have the span data, reconstruct the prompt as a messages array:",{"type":53,"tag":1101,"props":1654,"children":1658},{"className":1655,"code":1656,"language":1657,"meta":1106,"style":1106},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","[\n  {\"role\": \"system\", \"content\": \"You are a helpful assistant that...\"},\n  {\"role\": \"user\", \"content\": \"Given {input}, answer the question: {question}\"}\n]\n","json",[1659],{"type":53,"tag":74,"props":1660,"children":1661},{"__ignoreMap":1106},[1662,1670,1749,1822],{"type":53,"tag":1112,"props":1663,"children":1664},{"class":1114,"line":1115},[1665],{"type":53,"tag":1112,"props":1666,"children":1667},{"style":1155},[1668],{"type":59,"value":1669},"[\n",{"type":53,"tag":1112,"props":1671,"children":1672},{"class":1114,"line":1125},[1673,1678,1682,1688,1692,1697,1701,1705,1709,1714,1718,1723,1727,1731,1735,1740,1744],{"type":53,"tag":1112,"props":1674,"children":1675},{"style":1155},[1676],{"type":59,"value":1677},"  {",{"type":53,"tag":1112,"props":1679,"children":1680},{"style":1155},[1681],{"type":59,"value":1167},{"type":53,"tag":1112,"props":1683,"children":1685},{"style":1684},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[1686],{"type":59,"value":1687},"role",{"type":53,"tag":1112,"props":1689,"children":1690},{"style":1155},[1691],{"type":59,"value":1167},{"type":53,"tag":1112,"props":1693,"children":1694},{"style":1155},[1695],{"type":59,"value":1696},":",{"type":53,"tag":1112,"props":1698,"children":1699},{"style":1155},[1700],{"type":59,"value":1158},{"type":53,"tag":1112,"props":1702,"children":1703},{"style":1134},[1704],{"type":59,"value":286},{"type":53,"tag":1112,"props":1706,"children":1707},{"style":1155},[1708],{"type":59,"value":1167},{"type":53,"tag":1112,"props":1710,"children":1711},{"style":1155},[1712],{"type":59,"value":1713},",",{"type":53,"tag":1112,"props":1715,"children":1716},{"style":1155},[1717],{"type":59,"value":1158},{"type":53,"tag":1112,"props":1719,"children":1720},{"style":1684},[1721],{"type":59,"value":1722},"content",{"type":53,"tag":1112,"props":1724,"children":1725},{"style":1155},[1726],{"type":59,"value":1167},{"type":53,"tag":1112,"props":1728,"children":1729},{"style":1155},[1730],{"type":59,"value":1696},{"type":53,"tag":1112,"props":1732,"children":1733},{"style":1155},[1734],{"type":59,"value":1158},{"type":53,"tag":1112,"props":1736,"children":1737},{"style":1134},[1738],{"type":59,"value":1739},"You are a helpful assistant that...",{"type":53,"tag":1112,"props":1741,"children":1742},{"style":1155},[1743],{"type":59,"value":1167},{"type":53,"tag":1112,"props":1745,"children":1746},{"style":1155},[1747],{"type":59,"value":1748},"},\n",{"type":53,"tag":1112,"props":1750,"children":1751},{"class":1114,"line":1186},[1752,1756,1760,1764,1768,1772,1776,1780,1784,1788,1792,1796,1800,1804,1808,1813,1817],{"type":53,"tag":1112,"props":1753,"children":1754},{"style":1155},[1755],{"type":59,"value":1677},{"type":53,"tag":1112,"props":1757,"children":1758},{"style":1155},[1759],{"type":59,"value":1167},{"type":53,"tag":1112,"props":1761,"children":1762},{"style":1684},[1763],{"type":59,"value":1687},{"type":53,"tag":1112,"props":1765,"children":1766},{"style":1155},[1767],{"type":59,"value":1167},{"type":53,"tag":1112,"props":1769,"children":1770},{"style":1155},[1771],{"type":59,"value":1696},{"type":53,"tag":1112,"props":1773,"children":1774},{"style":1155},[1775],{"type":59,"value":1158},{"type":53,"tag":1112,"props":1777,"children":1778},{"style":1134},[1779],{"type":59,"value":294},{"type":53,"tag":1112,"props":1781,"children":1782},{"style":1155},[1783],{"type":59,"value":1167},{"type":53,"tag":1112,"props":1785,"children":1786},{"style":1155},[1787],{"type":59,"value":1713},{"type":53,"tag":1112,"props":1789,"children":1790},{"style":1155},[1791],{"type":59,"value":1158},{"type":53,"tag":1112,"props":1793,"children":1794},{"style":1684},[1795],{"type":59,"value":1722},{"type":53,"tag":1112,"props":1797,"children":1798},{"style":1155},[1799],{"type":59,"value":1167},{"type":53,"tag":1112,"props":1801,"children":1802},{"style":1155},[1803],{"type":59,"value":1696},{"type":53,"tag":1112,"props":1805,"children":1806},{"style":1155},[1807],{"type":59,"value":1158},{"type":53,"tag":1112,"props":1809,"children":1810},{"style":1134},[1811],{"type":59,"value":1812},"Given {input}, answer the question: {question}",{"type":53,"tag":1112,"props":1814,"children":1815},{"style":1155},[1816],{"type":59,"value":1167},{"type":53,"tag":1112,"props":1818,"children":1819},{"style":1155},[1820],{"type":59,"value":1821},"}\n",{"type":53,"tag":1112,"props":1823,"children":1824},{"class":1114,"line":1196},[1825],{"type":53,"tag":1112,"props":1826,"children":1827},{"style":1155},[1828],{"type":59,"value":1829},"]\n",{"type":53,"tag":66,"props":1831,"children":1832},{},[1833,1835,1840,1842,1847,1849,1855],{"type":59,"value":1834},"If the span has ",{"type":53,"tag":74,"props":1836,"children":1838},{"className":1837},[],[1839],{"type":59,"value":369},{"type":59,"value":1841},", the prompt uses variables. Preserve these placeholders (",{"type":53,"tag":74,"props":1843,"children":1845},{"className":1844},[],[1846],{"type":59,"value":380},{"type":59,"value":1848}," or ",{"type":53,"tag":74,"props":1850,"children":1852},{"className":1851},[],[1853],{"type":59,"value":1854},"{{variable}}",{"type":59,"value":1856},") -- they are substituted at runtime.",{"type":53,"tag":136,"props":1858,"children":1860},{"id":1859},"phase-2-gather-performance-data",[1861],{"type":59,"value":1862},"Phase 2: Gather Performance Data",{"type":53,"tag":193,"props":1864,"children":1866},{"id":1865},"from-traces-production-feedback",[1867],{"type":59,"value":1868},"From traces (production feedback)",{"type":53,"tag":1101,"props":1870,"children":1872},{"className":1103,"code":1871,"language":1105,"meta":1106,"style":1106},"# Find error spans -- these indicate prompt failures\nax spans export PROJECT \\\n  --filter \"status_code = 'ERROR' AND attributes.openinference.span.kind = 'LLM'\" \\\n  -l 20 --stdout\n\n# Find spans with low eval scores\nax spans export PROJECT \\\n  --filter \"annotation.correctness.label = 'incorrect'\" \\\n  -l 20 --stdout\n\n# Find spans with high latency (may indicate overly complex prompts)\nax spans export PROJECT \\\n  --filter \"attributes.openinference.span.kind = 'LLM' AND latency_ms > 10000\" \\\n  -l 20 --stdout\n\n# Export error traces for detailed inspection\nax spans export PROJECT --trace-id TRACE_ID\n",[1873],{"type":53,"tag":74,"props":1874,"children":1875},{"__ignoreMap":1106},[1876,1884,1908,1933,1950,1957,1965,1988,2012,2027,2034,2042,2065,2089,2104,2112,2121],{"type":53,"tag":1112,"props":1877,"children":1878},{"class":1114,"line":1115},[1879],{"type":53,"tag":1112,"props":1880,"children":1881},{"style":1119},[1882],{"type":59,"value":1883},"# Find error spans -- these indicate prompt failures\n",{"type":53,"tag":1112,"props":1885,"children":1886},{"class":1114,"line":1125},[1887,1891,1895,1899,1903],{"type":53,"tag":1112,"props":1888,"children":1889},{"style":1129},[1890],{"type":59,"value":828},{"type":53,"tag":1112,"props":1892,"children":1893},{"style":1134},[1894],{"type":59,"value":1137},{"type":53,"tag":1112,"props":1896,"children":1897},{"style":1134},[1898],{"type":59,"value":1142},{"type":53,"tag":1112,"props":1900,"children":1901},{"style":1134},[1902],{"type":59,"value":1147},{"type":53,"tag":1112,"props":1904,"children":1905},{"style":1486},[1906],{"type":59,"value":1907}," \\\n",{"type":53,"tag":1112,"props":1909,"children":1910},{"class":1114,"line":1186},[1911,1916,1920,1925,1929],{"type":53,"tag":1112,"props":1912,"children":1913},{"style":1134},[1914],{"type":59,"value":1915},"  --filter",{"type":53,"tag":1112,"props":1917,"children":1918},{"style":1155},[1919],{"type":59,"value":1158},{"type":53,"tag":1112,"props":1921,"children":1922},{"style":1134},[1923],{"type":59,"value":1924},"status_code = 'ERROR' AND attributes.openinference.span.kind = 'LLM'",{"type":53,"tag":1112,"props":1926,"children":1927},{"style":1155},[1928],{"type":59,"value":1167},{"type":53,"tag":1112,"props":1930,"children":1931},{"style":1486},[1932],{"type":59,"value":1907},{"type":53,"tag":1112,"props":1934,"children":1935},{"class":1114,"line":1196},[1936,1941,1946],{"type":53,"tag":1112,"props":1937,"children":1938},{"style":1134},[1939],{"type":59,"value":1940},"  -l",{"type":53,"tag":1112,"props":1942,"children":1943},{"style":1175},[1944],{"type":59,"value":1945}," 20",{"type":53,"tag":1112,"props":1947,"children":1948},{"style":1134},[1949],{"type":59,"value":1183},{"type":53,"tag":1112,"props":1951,"children":1952},{"class":1114,"line":27},[1953],{"type":53,"tag":1112,"props":1954,"children":1955},{"emptyLinePlaceholder":1190},[1956],{"type":59,"value":1193},{"type":53,"tag":1112,"props":1958,"children":1959},{"class":1114,"line":1253},[1960],{"type":53,"tag":1112,"props":1961,"children":1962},{"style":1119},[1963],{"type":59,"value":1964},"# Find spans with low eval scores\n",{"type":53,"tag":1112,"props":1966,"children":1967},{"class":1114,"line":1261},[1968,1972,1976,1980,1984],{"type":53,"tag":1112,"props":1969,"children":1970},{"style":1129},[1971],{"type":59,"value":828},{"type":53,"tag":1112,"props":1973,"children":1974},{"style":1134},[1975],{"type":59,"value":1137},{"type":53,"tag":1112,"props":1977,"children":1978},{"style":1134},[1979],{"type":59,"value":1142},{"type":53,"tag":1112,"props":1981,"children":1982},{"style":1134},[1983],{"type":59,"value":1147},{"type":53,"tag":1112,"props":1985,"children":1986},{"style":1486},[1987],{"type":59,"value":1907},{"type":53,"tag":1112,"props":1989,"children":1990},{"class":1114,"line":1270},[1991,1995,1999,2004,2008],{"type":53,"tag":1112,"props":1992,"children":1993},{"style":1134},[1994],{"type":59,"value":1915},{"type":53,"tag":1112,"props":1996,"children":1997},{"style":1155},[1998],{"type":59,"value":1158},{"type":53,"tag":1112,"props":2000,"children":2001},{"style":1134},[2002],{"type":59,"value":2003},"annotation.correctness.label = 'incorrect'",{"type":53,"tag":1112,"props":2005,"children":2006},{"style":1155},[2007],{"type":59,"value":1167},{"type":53,"tag":1112,"props":2009,"children":2010},{"style":1486},[2011],{"type":59,"value":1907},{"type":53,"tag":1112,"props":2013,"children":2014},{"class":1114,"line":1544},[2015,2019,2023],{"type":53,"tag":1112,"props":2016,"children":2017},{"style":1134},[2018],{"type":59,"value":1940},{"type":53,"tag":1112,"props":2020,"children":2021},{"style":1175},[2022],{"type":59,"value":1945},{"type":53,"tag":1112,"props":2024,"children":2025},{"style":1134},[2026],{"type":59,"value":1183},{"type":53,"tag":1112,"props":2028,"children":2029},{"class":1114,"line":1552},[2030],{"type":53,"tag":1112,"props":2031,"children":2032},{"emptyLinePlaceholder":1190},[2033],{"type":59,"value":1193},{"type":53,"tag":1112,"props":2035,"children":2036},{"class":1114,"line":1561},[2037],{"type":53,"tag":1112,"props":2038,"children":2039},{"style":1119},[2040],{"type":59,"value":2041},"# Find spans with high latency (may indicate overly complex prompts)\n",{"type":53,"tag":1112,"props":2043,"children":2044},{"class":1114,"line":1594},[2045,2049,2053,2057,2061],{"type":53,"tag":1112,"props":2046,"children":2047},{"style":1129},[2048],{"type":59,"value":828},{"type":53,"tag":1112,"props":2050,"children":2051},{"style":1134},[2052],{"type":59,"value":1137},{"type":53,"tag":1112,"props":2054,"children":2055},{"style":1134},[2056],{"type":59,"value":1142},{"type":53,"tag":1112,"props":2058,"children":2059},{"style":1134},[2060],{"type":59,"value":1147},{"type":53,"tag":1112,"props":2062,"children":2063},{"style":1486},[2064],{"type":59,"value":1907},{"type":53,"tag":1112,"props":2066,"children":2067},{"class":1114,"line":1602},[2068,2072,2076,2081,2085],{"type":53,"tag":1112,"props":2069,"children":2070},{"style":1134},[2071],{"type":59,"value":1915},{"type":53,"tag":1112,"props":2073,"children":2074},{"style":1155},[2075],{"type":59,"value":1158},{"type":53,"tag":1112,"props":2077,"children":2078},{"style":1134},[2079],{"type":59,"value":2080},"attributes.openinference.span.kind = 'LLM' AND latency_ms > 10000",{"type":53,"tag":1112,"props":2082,"children":2083},{"style":1155},[2084],{"type":59,"value":1167},{"type":53,"tag":1112,"props":2086,"children":2087},{"style":1486},[2088],{"type":59,"value":1907},{"type":53,"tag":1112,"props":2090,"children":2091},{"class":1114,"line":1611},[2092,2096,2100],{"type":53,"tag":1112,"props":2093,"children":2094},{"style":1134},[2095],{"type":59,"value":1940},{"type":53,"tag":1112,"props":2097,"children":2098},{"style":1175},[2099],{"type":59,"value":1945},{"type":53,"tag":1112,"props":2101,"children":2102},{"style":1134},[2103],{"type":59,"value":1183},{"type":53,"tag":1112,"props":2105,"children":2107},{"class":1114,"line":2106},15,[2108],{"type":53,"tag":1112,"props":2109,"children":2110},{"emptyLinePlaceholder":1190},[2111],{"type":59,"value":1193},{"type":53,"tag":1112,"props":2113,"children":2115},{"class":1114,"line":2114},16,[2116],{"type":53,"tag":1112,"props":2117,"children":2118},{"style":1119},[2119],{"type":59,"value":2120},"# Export error traces for detailed inspection\n",{"type":53,"tag":1112,"props":2122,"children":2124},{"class":1114,"line":2123},17,[2125,2129,2133,2137,2141,2145],{"type":53,"tag":1112,"props":2126,"children":2127},{"style":1129},[2128],{"type":59,"value":828},{"type":53,"tag":1112,"props":2130,"children":2131},{"style":1134},[2132],{"type":59,"value":1137},{"type":53,"tag":1112,"props":2134,"children":2135},{"style":1134},[2136],{"type":59,"value":1142},{"type":53,"tag":1112,"props":2138,"children":2139},{"style":1134},[2140],{"type":59,"value":1147},{"type":53,"tag":1112,"props":2142,"children":2143},{"style":1134},[2144],{"type":59,"value":1361},{"type":53,"tag":1112,"props":2146,"children":2147},{"style":1134},[2148],{"type":59,"value":1366},{"type":53,"tag":193,"props":2150,"children":2152},{"id":2151},"from-datasets-and-experiments",[2153],{"type":59,"value":2154},"From datasets and experiments",{"type":53,"tag":1101,"props":2156,"children":2158},{"className":1103,"code":2157,"language":1105,"meta":1106,"style":1106},"# Export a dataset (ground truth examples)\nax datasets export DATASET_NAME --space SPACE\n# -> dataset_*\u002Fexamples.json\n\n# Export experiment results (what the LLM produced)\nax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE\n# -> experiment_*\u002Fruns.json\n",[2159],{"type":53,"tag":74,"props":2160,"children":2161},{"__ignoreMap":1106},[2162,2170,2201,2209,2216,2224,2262],{"type":53,"tag":1112,"props":2163,"children":2164},{"class":1114,"line":1115},[2165],{"type":53,"tag":1112,"props":2166,"children":2167},{"style":1119},[2168],{"type":59,"value":2169},"# Export a dataset (ground truth examples)\n",{"type":53,"tag":1112,"props":2171,"children":2172},{"class":1114,"line":1125},[2173,2177,2182,2186,2191,2196],{"type":53,"tag":1112,"props":2174,"children":2175},{"style":1129},[2176],{"type":59,"value":828},{"type":53,"tag":1112,"props":2178,"children":2179},{"style":1134},[2180],{"type":59,"value":2181}," datasets",{"type":53,"tag":1112,"props":2183,"children":2184},{"style":1134},[2185],{"type":59,"value":1142},{"type":53,"tag":1112,"props":2187,"children":2188},{"style":1134},[2189],{"type":59,"value":2190}," DATASET_NAME",{"type":53,"tag":1112,"props":2192,"children":2193},{"style":1134},[2194],{"type":59,"value":2195}," --space",{"type":53,"tag":1112,"props":2197,"children":2198},{"style":1134},[2199],{"type":59,"value":2200}," SPACE\n",{"type":53,"tag":1112,"props":2202,"children":2203},{"class":1114,"line":1186},[2204],{"type":53,"tag":1112,"props":2205,"children":2206},{"style":1119},[2207],{"type":59,"value":2208},"# -> dataset_*\u002Fexamples.json\n",{"type":53,"tag":1112,"props":2210,"children":2211},{"class":1114,"line":1196},[2212],{"type":53,"tag":1112,"props":2213,"children":2214},{"emptyLinePlaceholder":1190},[2215],{"type":59,"value":1193},{"type":53,"tag":1112,"props":2217,"children":2218},{"class":1114,"line":27},[2219],{"type":53,"tag":1112,"props":2220,"children":2221},{"style":1119},[2222],{"type":59,"value":2223},"# Export experiment results (what the LLM produced)\n",{"type":53,"tag":1112,"props":2225,"children":2226},{"class":1114,"line":1253},[2227,2231,2236,2240,2245,2250,2254,2258],{"type":53,"tag":1112,"props":2228,"children":2229},{"style":1129},[2230],{"type":59,"value":828},{"type":53,"tag":1112,"props":2232,"children":2233},{"style":1134},[2234],{"type":59,"value":2235}," experiments",{"type":53,"tag":1112,"props":2237,"children":2238},{"style":1134},[2239],{"type":59,"value":1142},{"type":53,"tag":1112,"props":2241,"children":2242},{"style":1134},[2243],{"type":59,"value":2244}," EXPERIMENT_NAME",{"type":53,"tag":1112,"props":2246,"children":2247},{"style":1134},[2248],{"type":59,"value":2249}," --dataset",{"type":53,"tag":1112,"props":2251,"children":2252},{"style":1134},[2253],{"type":59,"value":2190},{"type":53,"tag":1112,"props":2255,"children":2256},{"style":1134},[2257],{"type":59,"value":2195},{"type":53,"tag":1112,"props":2259,"children":2260},{"style":1134},[2261],{"type":59,"value":2200},{"type":53,"tag":1112,"props":2263,"children":2264},{"class":1114,"line":1261},[2265],{"type":53,"tag":1112,"props":2266,"children":2267},{"style":1119},[2268],{"type":59,"value":2269},"# -> experiment_*\u002Fruns.json\n",{"type":53,"tag":193,"props":2271,"children":2273},{"id":2272},"merge-dataset-experiment-for-analysis",[2274],{"type":59,"value":2275},"Merge dataset + experiment for analysis",{"type":53,"tag":66,"props":2277,"children":2278},{},[2279,2281,2287],{"type":59,"value":2280},"Join the two files by ",{"type":53,"tag":74,"props":2282,"children":2284},{"className":2283},[],[2285],{"type":59,"value":2286},"example_id",{"type":59,"value":2288}," to see inputs alongside outputs and evaluations:",{"type":53,"tag":1101,"props":2290,"children":2292},{"className":1103,"code":2291,"language":1105,"meta":1106,"style":1106},"# Count examples and runs\njq 'length' dataset_*\u002Fexamples.json\njq 'length' experiment_*\u002Fruns.json\n\n# View a single joined record\njq -s '\n  .[0] as $dataset |\n  .[1][0] as $run |\n  ($dataset[] | select(.id == $run.example_id)) as $example |\n  {\n    input: $example,\n    output: $run.output,\n    evaluations: $run.evaluations\n  }\n' dataset_*\u002Fexamples.json experiment_*\u002Fruns.json\n\n# Find failed examples (where eval score \u003C threshold)\njq '[.[] | select(.evaluations.correctness.score \u003C 0.5)]' experiment_*\u002Fruns.json\n",[2293],{"type":53,"tag":74,"props":2294,"children":2295},{"__ignoreMap":1106},[2296,2304,2338,2371,2378,2386,2403,2411,2419,2427,2435,2443,2451,2459,2467,2499,2506,2514],{"type":53,"tag":1112,"props":2297,"children":2298},{"class":1114,"line":1115},[2299],{"type":53,"tag":1112,"props":2300,"children":2301},{"style":1119},[2302],{"type":59,"value":2303},"# Count examples and runs\n",{"type":53,"tag":1112,"props":2305,"children":2306},{"class":1114,"line":1125},[2307,2311,2315,2320,2324,2329,2333],{"type":53,"tag":1112,"props":2308,"children":2309},{"style":1129},[2310],{"type":59,"value":1439},{"type":53,"tag":1112,"props":2312,"children":2313},{"style":1155},[2314],{"type":59,"value":1444},{"type":53,"tag":1112,"props":2316,"children":2317},{"style":1134},[2318],{"type":59,"value":2319},"length",{"type":53,"tag":1112,"props":2321,"children":2322},{"style":1155},[2323],{"type":59,"value":1478},{"type":53,"tag":1112,"props":2325,"children":2326},{"style":1134},[2327],{"type":59,"value":2328}," dataset_",{"type":53,"tag":1112,"props":2330,"children":2331},{"style":1486},[2332],{"type":59,"value":1489},{"type":53,"tag":1112,"props":2334,"children":2335},{"style":1134},[2336],{"type":59,"value":2337},"\u002Fexamples.json\n",{"type":53,"tag":1112,"props":2339,"children":2340},{"class":1114,"line":1186},[2341,2345,2349,2353,2357,2362,2366],{"type":53,"tag":1112,"props":2342,"children":2343},{"style":1129},[2344],{"type":59,"value":1439},{"type":53,"tag":1112,"props":2346,"children":2347},{"style":1155},[2348],{"type":59,"value":1444},{"type":53,"tag":1112,"props":2350,"children":2351},{"style":1134},[2352],{"type":59,"value":2319},{"type":53,"tag":1112,"props":2354,"children":2355},{"style":1155},[2356],{"type":59,"value":1478},{"type":53,"tag":1112,"props":2358,"children":2359},{"style":1134},[2360],{"type":59,"value":2361}," experiment_",{"type":53,"tag":1112,"props":2363,"children":2364},{"style":1486},[2365],{"type":59,"value":1489},{"type":53,"tag":1112,"props":2367,"children":2368},{"style":1134},[2369],{"type":59,"value":2370},"\u002Fruns.json\n",{"type":53,"tag":1112,"props":2372,"children":2373},{"class":1114,"line":1196},[2374],{"type":53,"tag":1112,"props":2375,"children":2376},{"emptyLinePlaceholder":1190},[2377],{"type":59,"value":1193},{"type":53,"tag":1112,"props":2379,"children":2380},{"class":1114,"line":27},[2381],{"type":53,"tag":1112,"props":2382,"children":2383},{"style":1119},[2384],{"type":59,"value":2385},"# View a single joined record\n",{"type":53,"tag":1112,"props":2387,"children":2388},{"class":1114,"line":1253},[2389,2393,2398],{"type":53,"tag":1112,"props":2390,"children":2391},{"style":1129},[2392],{"type":59,"value":1439},{"type":53,"tag":1112,"props":2394,"children":2395},{"style":1134},[2396],{"type":59,"value":2397}," -s",{"type":53,"tag":1112,"props":2399,"children":2400},{"style":1155},[2401],{"type":59,"value":2402}," '\n",{"type":53,"tag":1112,"props":2404,"children":2405},{"class":1114,"line":1261},[2406],{"type":53,"tag":1112,"props":2407,"children":2408},{"style":1134},[2409],{"type":59,"value":2410},"  .[0] as $dataset |\n",{"type":53,"tag":1112,"props":2412,"children":2413},{"class":1114,"line":1270},[2414],{"type":53,"tag":1112,"props":2415,"children":2416},{"style":1134},[2417],{"type":59,"value":2418},"  .[1][0] as $run |\n",{"type":53,"tag":1112,"props":2420,"children":2421},{"class":1114,"line":1544},[2422],{"type":53,"tag":1112,"props":2423,"children":2424},{"style":1134},[2425],{"type":59,"value":2426},"  ($dataset[] | select(.id == $run.example_id)) as $example |\n",{"type":53,"tag":1112,"props":2428,"children":2429},{"class":1114,"line":1552},[2430],{"type":53,"tag":1112,"props":2431,"children":2432},{"style":1134},[2433],{"type":59,"value":2434},"  {\n",{"type":53,"tag":1112,"props":2436,"children":2437},{"class":1114,"line":1561},[2438],{"type":53,"tag":1112,"props":2439,"children":2440},{"style":1134},[2441],{"type":59,"value":2442},"    input: $example,\n",{"type":53,"tag":1112,"props":2444,"children":2445},{"class":1114,"line":1594},[2446],{"type":53,"tag":1112,"props":2447,"children":2448},{"style":1134},[2449],{"type":59,"value":2450},"    output: $run.output,\n",{"type":53,"tag":1112,"props":2452,"children":2453},{"class":1114,"line":1602},[2454],{"type":53,"tag":1112,"props":2455,"children":2456},{"style":1134},[2457],{"type":59,"value":2458},"    evaluations: $run.evaluations\n",{"type":53,"tag":1112,"props":2460,"children":2461},{"class":1114,"line":1611},[2462],{"type":53,"tag":1112,"props":2463,"children":2464},{"style":1134},[2465],{"type":59,"value":2466},"  }\n",{"type":53,"tag":1112,"props":2468,"children":2469},{"class":1114,"line":2106},[2470,2474,2478,2482,2487,2491,2495],{"type":53,"tag":1112,"props":2471,"children":2472},{"style":1155},[2473],{"type":59,"value":1478},{"type":53,"tag":1112,"props":2475,"children":2476},{"style":1134},[2477],{"type":59,"value":2328},{"type":53,"tag":1112,"props":2479,"children":2480},{"style":1486},[2481],{"type":59,"value":1489},{"type":53,"tag":1112,"props":2483,"children":2484},{"style":1134},[2485],{"type":59,"value":2486},"\u002Fexamples.json",{"type":53,"tag":1112,"props":2488,"children":2489},{"style":1134},[2490],{"type":59,"value":2361},{"type":53,"tag":1112,"props":2492,"children":2493},{"style":1486},[2494],{"type":59,"value":1489},{"type":53,"tag":1112,"props":2496,"children":2497},{"style":1134},[2498],{"type":59,"value":2370},{"type":53,"tag":1112,"props":2500,"children":2501},{"class":1114,"line":2114},[2502],{"type":53,"tag":1112,"props":2503,"children":2504},{"emptyLinePlaceholder":1190},[2505],{"type":59,"value":1193},{"type":53,"tag":1112,"props":2507,"children":2508},{"class":1114,"line":2123},[2509],{"type":53,"tag":1112,"props":2510,"children":2511},{"style":1119},[2512],{"type":59,"value":2513},"# Find failed examples (where eval score \u003C threshold)\n",{"type":53,"tag":1112,"props":2515,"children":2517},{"class":1114,"line":2516},18,[2518,2522,2526,2531,2535,2539,2543],{"type":53,"tag":1112,"props":2519,"children":2520},{"style":1129},[2521],{"type":59,"value":1439},{"type":53,"tag":1112,"props":2523,"children":2524},{"style":1155},[2525],{"type":59,"value":1444},{"type":53,"tag":1112,"props":2527,"children":2528},{"style":1134},[2529],{"type":59,"value":2530},"[.[] | select(.evaluations.correctness.score \u003C 0.5)]",{"type":53,"tag":1112,"props":2532,"children":2533},{"style":1155},[2534],{"type":59,"value":1478},{"type":53,"tag":1112,"props":2536,"children":2537},{"style":1134},[2538],{"type":59,"value":2361},{"type":53,"tag":1112,"props":2540,"children":2541},{"style":1486},[2542],{"type":59,"value":1489},{"type":53,"tag":1112,"props":2544,"children":2545},{"style":1134},[2546],{"type":59,"value":2370},{"type":53,"tag":193,"props":2548,"children":2550},{"id":2549},"identify-what-to-optimize",[2551],{"type":59,"value":2552},"Identify what to optimize",{"type":53,"tag":66,"props":2554,"children":2555},{},[2556],{"type":59,"value":2557},"Look for patterns across failures:",{"type":53,"tag":1036,"props":2559,"children":2560},{},[2561,2571,2588,2598,2608],{"type":53,"tag":147,"props":2562,"children":2563},{},[2564,2569],{"type":53,"tag":70,"props":2565,"children":2566},{},[2567],{"type":59,"value":2568},"Compare outputs to ground truth",{"type":59,"value":2570},": Where does the LLM output differ from expected?",{"type":53,"tag":147,"props":2572,"children":2573},{},[2574,2579,2580,2586],{"type":53,"tag":70,"props":2575,"children":2576},{},[2577],{"type":59,"value":2578},"Read eval explanations",{"type":59,"value":519},{"type":53,"tag":74,"props":2581,"children":2583},{"className":2582},[],[2584],{"type":59,"value":2585},"eval.*.explanation",{"type":59,"value":2587}," tells you WHY something failed",{"type":53,"tag":147,"props":2589,"children":2590},{},[2591,2596],{"type":53,"tag":70,"props":2592,"children":2593},{},[2594],{"type":59,"value":2595},"Check annotation text",{"type":59,"value":2597},": Human feedback describes specific issues",{"type":53,"tag":147,"props":2599,"children":2600},{},[2601,2606],{"type":53,"tag":70,"props":2602,"children":2603},{},[2604],{"type":59,"value":2605},"Look for verbosity mismatches",{"type":59,"value":2607},": If outputs are too long\u002Fshort vs ground truth",{"type":53,"tag":147,"props":2609,"children":2610},{},[2611,2616],{"type":53,"tag":70,"props":2612,"children":2613},{},[2614],{"type":59,"value":2615},"Check format compliance",{"type":59,"value":2617},": Are outputs in the expected format?",{"type":53,"tag":136,"props":2619,"children":2621},{"id":2620},"phase-3-optimize-the-prompt",[2622],{"type":59,"value":2623},"Phase 3: Optimize the Prompt",{"type":53,"tag":193,"props":2625,"children":2627},{"id":2626},"the-optimization-meta-prompt",[2628],{"type":59,"value":2629},"The Optimization Meta-Prompt",{"type":53,"tag":66,"props":2631,"children":2632},{},[2633],{"type":59,"value":2634},"Use this template to generate an improved version of the prompt. Fill in the three placeholders and send it to your LLM (GPT-4o, Claude, etc.):",{"type":53,"tag":1101,"props":2636,"children":2640},{"className":2637,"code":2639,"language":59},[2638],"language-text","You are an expert in prompt optimization. Given the original baseline prompt\nand the associated performance data (inputs, outputs, evaluation labels, and\nexplanations), generate a revised version that improves results.\n\nORIGINAL BASELINE PROMPT\n========================\n\n{PASTE_ORIGINAL_PROMPT_HERE}\n\n========================\n\nPERFORMANCE DATA\n================\n\nThe following records show how the current prompt performed. Each record\nincludes the input, the LLM output, and evaluation feedback:\n\n{PASTE_RECORDS_HERE}\n\n================\n\nHOW TO USE THIS DATA\n\n1. Compare outputs: Look at what the LLM generated vs what was expected\n2. Review eval scores: Check which examples scored poorly and why\n3. Examine annotations: Human feedback shows what worked and what didn't\n4. Identify patterns: Look for common issues across multiple examples\n5. Focus on failures: The rows where the output DIFFERS from the expected\n   value are the ones that need fixing\n\nALIGNMENT STRATEGY\n\n- If outputs have extra text or reasoning not present in the ground truth,\n  remove instructions that encourage explanation or verbose reasoning\n- If outputs are missing information, add instructions to include it\n- If outputs are in the wrong format, add explicit format instructions\n- Focus on the rows where the output differs from the target -- these are\n  the failures to fix\n\nRULES\n\nMaintain Structure:\n- Use the same template variables as the current prompt ({var} or {{var}})\n- Don't change sections that are already working\n- Preserve the exact return format instructions from the original prompt\n\nAvoid Overfitting:\n- DO NOT copy examples verbatim into the prompt\n- DO NOT quote specific test data outputs exactly\n- INSTEAD: Extract the ESSENCE of what makes good vs bad outputs\n- INSTEAD: Add general guidelines and principles\n- INSTEAD: If adding few-shot examples, create SYNTHETIC examples that\n  demonstrate the principle, not real data from above\n\nGoal: Create a prompt that generalizes well to new inputs, not one that\nmemorizes the test data.\n\nOUTPUT FORMAT\n\nReturn the revised prompt as a JSON array of messages:\n\n[\n  {\"role\": \"system\", \"content\": \"...\"},\n  {\"role\": \"user\", \"content\": \"...\"}\n]\n\nAlso provide a brief reasoning section (bulleted list) explaining:\n- What problems you found\n- How the revised prompt addresses each one\n",[2641],{"type":53,"tag":74,"props":2642,"children":2643},{"__ignoreMap":1106},[2644],{"type":59,"value":2639},{"type":53,"tag":193,"props":2646,"children":2648},{"id":2647},"preparing-the-performance-data",[2649],{"type":59,"value":2650},"Preparing the performance data",{"type":53,"tag":66,"props":2652,"children":2653},{},[2654],{"type":59,"value":2655},"Format the records as a JSON array before pasting into the template:",{"type":53,"tag":1101,"props":2657,"children":2659},{"className":1103,"code":2658,"language":1105,"meta":1106,"style":1106},"# From dataset + experiment: join and select relevant columns\njq -s '\n  .[0] as $ds |\n  [.[1][] | . as $run |\n    ($ds[] | select(.id == $run.example_id)) as $ex |\n    {\n      input: $ex.input,\n      expected: $ex.expected_output,\n      actual_output: $run.output,\n      eval_score: $run.evaluations.correctness.score,\n      eval_label: $run.evaluations.correctness.label,\n      eval_explanation: $run.evaluations.correctness.explanation\n    }\n  ]\n' dataset_*\u002Fexamples.json experiment_*\u002Fruns.json\n\n# From exported spans: extract input\u002Foutput pairs with annotations\njq '[.[] | select(.attributes.openinference.span.kind == \"LLM\") | {\n  input: .attributes.input.value,\n  output: .attributes.output.value,\n  status: .status_code,\n  model: .attributes.llm.model_name\n}]' trace_*\u002Fspans.json\n",[2660],{"type":53,"tag":74,"props":2661,"children":2662},{"__ignoreMap":1106},[2663,2671,2686,2694,2702,2710,2718,2726,2734,2742,2750,2758,2766,2774,2782,2813,2820,2828,2844,2853,2862,2871,2879],{"type":53,"tag":1112,"props":2664,"children":2665},{"class":1114,"line":1115},[2666],{"type":53,"tag":1112,"props":2667,"children":2668},{"style":1119},[2669],{"type":59,"value":2670},"# From dataset + experiment: join and select relevant columns\n",{"type":53,"tag":1112,"props":2672,"children":2673},{"class":1114,"line":1125},[2674,2678,2682],{"type":53,"tag":1112,"props":2675,"children":2676},{"style":1129},[2677],{"type":59,"value":1439},{"type":53,"tag":1112,"props":2679,"children":2680},{"style":1134},[2681],{"type":59,"value":2397},{"type":53,"tag":1112,"props":2683,"children":2684},{"style":1155},[2685],{"type":59,"value":2402},{"type":53,"tag":1112,"props":2687,"children":2688},{"class":1114,"line":1186},[2689],{"type":53,"tag":1112,"props":2690,"children":2691},{"style":1134},[2692],{"type":59,"value":2693},"  .[0] as $ds |\n",{"type":53,"tag":1112,"props":2695,"children":2696},{"class":1114,"line":1196},[2697],{"type":53,"tag":1112,"props":2698,"children":2699},{"style":1134},[2700],{"type":59,"value":2701},"  [.[1][] | . as $run |\n",{"type":53,"tag":1112,"props":2703,"children":2704},{"class":1114,"line":27},[2705],{"type":53,"tag":1112,"props":2706,"children":2707},{"style":1134},[2708],{"type":59,"value":2709},"    ($ds[] | select(.id == $run.example_id)) as $ex |\n",{"type":53,"tag":1112,"props":2711,"children":2712},{"class":1114,"line":1253},[2713],{"type":53,"tag":1112,"props":2714,"children":2715},{"style":1134},[2716],{"type":59,"value":2717},"    {\n",{"type":53,"tag":1112,"props":2719,"children":2720},{"class":1114,"line":1261},[2721],{"type":53,"tag":1112,"props":2722,"children":2723},{"style":1134},[2724],{"type":59,"value":2725},"      input: $ex.input,\n",{"type":53,"tag":1112,"props":2727,"children":2728},{"class":1114,"line":1270},[2729],{"type":53,"tag":1112,"props":2730,"children":2731},{"style":1134},[2732],{"type":59,"value":2733},"      expected: $ex.expected_output,\n",{"type":53,"tag":1112,"props":2735,"children":2736},{"class":1114,"line":1544},[2737],{"type":53,"tag":1112,"props":2738,"children":2739},{"style":1134},[2740],{"type":59,"value":2741},"      actual_output: $run.output,\n",{"type":53,"tag":1112,"props":2743,"children":2744},{"class":1114,"line":1552},[2745],{"type":53,"tag":1112,"props":2746,"children":2747},{"style":1134},[2748],{"type":59,"value":2749},"      eval_score: $run.evaluations.correctness.score,\n",{"type":53,"tag":1112,"props":2751,"children":2752},{"class":1114,"line":1561},[2753],{"type":53,"tag":1112,"props":2754,"children":2755},{"style":1134},[2756],{"type":59,"value":2757},"      eval_label: $run.evaluations.correctness.label,\n",{"type":53,"tag":1112,"props":2759,"children":2760},{"class":1114,"line":1594},[2761],{"type":53,"tag":1112,"props":2762,"children":2763},{"style":1134},[2764],{"type":59,"value":2765},"      eval_explanation: $run.evaluations.correctness.explanation\n",{"type":53,"tag":1112,"props":2767,"children":2768},{"class":1114,"line":1602},[2769],{"type":53,"tag":1112,"props":2770,"children":2771},{"style":1134},[2772],{"type":59,"value":2773},"    }\n",{"type":53,"tag":1112,"props":2775,"children":2776},{"class":1114,"line":1611},[2777],{"type":53,"tag":1112,"props":2778,"children":2779},{"style":1134},[2780],{"type":59,"value":2781},"  ]\n",{"type":53,"tag":1112,"props":2783,"children":2784},{"class":1114,"line":2106},[2785,2789,2793,2797,2801,2805,2809],{"type":53,"tag":1112,"props":2786,"children":2787},{"style":1155},[2788],{"type":59,"value":1478},{"type":53,"tag":1112,"props":2790,"children":2791},{"style":1134},[2792],{"type":59,"value":2328},{"type":53,"tag":1112,"props":2794,"children":2795},{"style":1486},[2796],{"type":59,"value":1489},{"type":53,"tag":1112,"props":2798,"children":2799},{"style":1134},[2800],{"type":59,"value":2486},{"type":53,"tag":1112,"props":2802,"children":2803},{"style":1134},[2804],{"type":59,"value":2361},{"type":53,"tag":1112,"props":2806,"children":2807},{"style":1486},[2808],{"type":59,"value":1489},{"type":53,"tag":1112,"props":2810,"children":2811},{"style":1134},[2812],{"type":59,"value":2370},{"type":53,"tag":1112,"props":2814,"children":2815},{"class":1114,"line":2114},[2816],{"type":53,"tag":1112,"props":2817,"children":2818},{"emptyLinePlaceholder":1190},[2819],{"type":59,"value":1193},{"type":53,"tag":1112,"props":2821,"children":2822},{"class":1114,"line":2123},[2823],{"type":53,"tag":1112,"props":2824,"children":2825},{"style":1119},[2826],{"type":59,"value":2827},"# From exported spans: extract input\u002Foutput pairs with annotations\n",{"type":53,"tag":1112,"props":2829,"children":2830},{"class":1114,"line":2516},[2831,2835,2839],{"type":53,"tag":1112,"props":2832,"children":2833},{"style":1129},[2834],{"type":59,"value":1439},{"type":53,"tag":1112,"props":2836,"children":2837},{"style":1155},[2838],{"type":59,"value":1444},{"type":53,"tag":1112,"props":2840,"children":2841},{"style":1134},[2842],{"type":59,"value":2843},"[.[] | select(.attributes.openinference.span.kind == \"LLM\") | {\n",{"type":53,"tag":1112,"props":2845,"children":2847},{"class":1114,"line":2846},19,[2848],{"type":53,"tag":1112,"props":2849,"children":2850},{"style":1134},[2851],{"type":59,"value":2852},"  input: .attributes.input.value,\n",{"type":53,"tag":1112,"props":2854,"children":2856},{"class":1114,"line":2855},20,[2857],{"type":53,"tag":1112,"props":2858,"children":2859},{"style":1134},[2860],{"type":59,"value":2861},"  output: .attributes.output.value,\n",{"type":53,"tag":1112,"props":2863,"children":2865},{"class":1114,"line":2864},21,[2866],{"type":53,"tag":1112,"props":2867,"children":2868},{"style":1134},[2869],{"type":59,"value":2870},"  status: .status_code,\n",{"type":53,"tag":1112,"props":2872,"children":2874},{"class":1114,"line":2873},22,[2875],{"type":53,"tag":1112,"props":2876,"children":2877},{"style":1134},[2878],{"type":59,"value":1465},{"type":53,"tag":1112,"props":2880,"children":2882},{"class":1114,"line":2881},23,[2883,2888,2892,2896,2900],{"type":53,"tag":1112,"props":2884,"children":2885},{"style":1134},[2886],{"type":59,"value":2887},"}]",{"type":53,"tag":1112,"props":2889,"children":2890},{"style":1155},[2891],{"type":59,"value":1478},{"type":53,"tag":1112,"props":2893,"children":2894},{"style":1134},[2895],{"type":59,"value":1483},{"type":53,"tag":1112,"props":2897,"children":2898},{"style":1486},[2899],{"type":59,"value":1489},{"type":53,"tag":1112,"props":2901,"children":2902},{"style":1134},[2903],{"type":59,"value":1494},{"type":53,"tag":193,"props":2905,"children":2907},{"id":2906},"applying-the-revised-prompt",[2908],{"type":59,"value":2909},"Applying the revised prompt",{"type":53,"tag":66,"props":2911,"children":2912},{},[2913],{"type":59,"value":2914},"After the LLM returns the revised messages array:",{"type":53,"tag":1036,"props":2916,"children":2917},{},[2918,2923,2928,2933],{"type":53,"tag":147,"props":2919,"children":2920},{},[2921],{"type":59,"value":2922},"Compare the original and revised prompts side by side",{"type":53,"tag":147,"props":2924,"children":2925},{},[2926],{"type":59,"value":2927},"Verify all template variables are preserved",{"type":53,"tag":147,"props":2929,"children":2930},{},[2931],{"type":59,"value":2932},"Check that format instructions are intact",{"type":53,"tag":147,"props":2934,"children":2935},{},[2936],{"type":59,"value":2937},"Test on a few examples before full deployment",{"type":53,"tag":136,"props":2939,"children":2941},{"id":2940},"phase-4-iterate",[2942],{"type":59,"value":2943},"Phase 4: Iterate",{"type":53,"tag":193,"props":2945,"children":2947},{"id":2946},"the-optimization-loop",[2948],{"type":59,"value":2949},"The optimization loop",{"type":53,"tag":1101,"props":2951,"children":2954},{"className":2952,"code":2953,"language":59},[2638],"1. Extract prompt    -> Phase 1 (once)\n2. Run experiment    -> ax experiments create ...\n3. Export results    -> ax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE\n4. Analyze failures  -> jq to find low scores\n5. Run meta-prompt   -> Phase 3 with new failure data\n6. Apply revised prompt\n7. Repeat from step 2\n",[2955],{"type":53,"tag":74,"props":2956,"children":2957},{"__ignoreMap":1106},[2958],{"type":59,"value":2953},{"type":53,"tag":193,"props":2960,"children":2962},{"id":2961},"measure-improvement",[2963],{"type":59,"value":2964},"Measure improvement",{"type":53,"tag":1101,"props":2966,"children":2968},{"className":1103,"code":2967,"language":1105,"meta":1106,"style":1106},"# Compare scores across experiments\n# Experiment A (baseline)\njq '[.[] | .evaluations.correctness.score] | add \u002F length' experiment_a\u002Fruns.json\n\n# Experiment B (optimized)\njq '[.[] | .evaluations.correctness.score] | add \u002F length' experiment_b\u002Fruns.json\n\n# Find examples that flipped from fail to pass\njq -s '\n  [.[0][] | select(.evaluations.correctness.label == \"incorrect\")] as $fails |\n  [.[1][] | select(.evaluations.correctness.label == \"correct\") |\n    select(.example_id as $id | $fails | any(.example_id == $id))\n  ] | length\n' experiment_a\u002Fruns.json experiment_b\u002Fruns.json\n",[2969],{"type":53,"tag":74,"props":2970,"children":2971},{"__ignoreMap":1106},[2972,2980,2988,3013,3020,3028,3052,3059,3067,3082,3090,3098,3106,3114],{"type":53,"tag":1112,"props":2973,"children":2974},{"class":1114,"line":1115},[2975],{"type":53,"tag":1112,"props":2976,"children":2977},{"style":1119},[2978],{"type":59,"value":2979},"# Compare scores across experiments\n",{"type":53,"tag":1112,"props":2981,"children":2982},{"class":1114,"line":1125},[2983],{"type":53,"tag":1112,"props":2984,"children":2985},{"style":1119},[2986],{"type":59,"value":2987},"# Experiment A (baseline)\n",{"type":53,"tag":1112,"props":2989,"children":2990},{"class":1114,"line":1186},[2991,2995,2999,3004,3008],{"type":53,"tag":1112,"props":2992,"children":2993},{"style":1129},[2994],{"type":59,"value":1439},{"type":53,"tag":1112,"props":2996,"children":2997},{"style":1155},[2998],{"type":59,"value":1444},{"type":53,"tag":1112,"props":3000,"children":3001},{"style":1134},[3002],{"type":59,"value":3003},"[.[] | .evaluations.correctness.score] | add \u002F length",{"type":53,"tag":1112,"props":3005,"children":3006},{"style":1155},[3007],{"type":59,"value":1478},{"type":53,"tag":1112,"props":3009,"children":3010},{"style":1134},[3011],{"type":59,"value":3012}," experiment_a\u002Fruns.json\n",{"type":53,"tag":1112,"props":3014,"children":3015},{"class":1114,"line":1196},[3016],{"type":53,"tag":1112,"props":3017,"children":3018},{"emptyLinePlaceholder":1190},[3019],{"type":59,"value":1193},{"type":53,"tag":1112,"props":3021,"children":3022},{"class":1114,"line":27},[3023],{"type":53,"tag":1112,"props":3024,"children":3025},{"style":1119},[3026],{"type":59,"value":3027},"# Experiment B (optimized)\n",{"type":53,"tag":1112,"props":3029,"children":3030},{"class":1114,"line":1253},[3031,3035,3039,3043,3047],{"type":53,"tag":1112,"props":3032,"children":3033},{"style":1129},[3034],{"type":59,"value":1439},{"type":53,"tag":1112,"props":3036,"children":3037},{"style":1155},[3038],{"type":59,"value":1444},{"type":53,"tag":1112,"props":3040,"children":3041},{"style":1134},[3042],{"type":59,"value":3003},{"type":53,"tag":1112,"props":3044,"children":3045},{"style":1155},[3046],{"type":59,"value":1478},{"type":53,"tag":1112,"props":3048,"children":3049},{"style":1134},[3050],{"type":59,"value":3051}," experiment_b\u002Fruns.json\n",{"type":53,"tag":1112,"props":3053,"children":3054},{"class":1114,"line":1261},[3055],{"type":53,"tag":1112,"props":3056,"children":3057},{"emptyLinePlaceholder":1190},[3058],{"type":59,"value":1193},{"type":53,"tag":1112,"props":3060,"children":3061},{"class":1114,"line":1270},[3062],{"type":53,"tag":1112,"props":3063,"children":3064},{"style":1119},[3065],{"type":59,"value":3066},"# Find examples that flipped from fail to pass\n",{"type":53,"tag":1112,"props":3068,"children":3069},{"class":1114,"line":1544},[3070,3074,3078],{"type":53,"tag":1112,"props":3071,"children":3072},{"style":1129},[3073],{"type":59,"value":1439},{"type":53,"tag":1112,"props":3075,"children":3076},{"style":1134},[3077],{"type":59,"value":2397},{"type":53,"tag":1112,"props":3079,"children":3080},{"style":1155},[3081],{"type":59,"value":2402},{"type":53,"tag":1112,"props":3083,"children":3084},{"class":1114,"line":1552},[3085],{"type":53,"tag":1112,"props":3086,"children":3087},{"style":1134},[3088],{"type":59,"value":3089},"  [.[0][] | select(.evaluations.correctness.label == \"incorrect\")] as $fails |\n",{"type":53,"tag":1112,"props":3091,"children":3092},{"class":1114,"line":1561},[3093],{"type":53,"tag":1112,"props":3094,"children":3095},{"style":1134},[3096],{"type":59,"value":3097},"  [.[1][] | select(.evaluations.correctness.label == \"correct\") |\n",{"type":53,"tag":1112,"props":3099,"children":3100},{"class":1114,"line":1594},[3101],{"type":53,"tag":1112,"props":3102,"children":3103},{"style":1134},[3104],{"type":59,"value":3105},"    select(.example_id as $id | $fails | any(.example_id == $id))\n",{"type":53,"tag":1112,"props":3107,"children":3108},{"class":1114,"line":1602},[3109],{"type":53,"tag":1112,"props":3110,"children":3111},{"style":1134},[3112],{"type":59,"value":3113},"  ] | length\n",{"type":53,"tag":1112,"props":3115,"children":3116},{"class":1114,"line":1611},[3117,3121,3126],{"type":53,"tag":1112,"props":3118,"children":3119},{"style":1155},[3120],{"type":59,"value":1478},{"type":53,"tag":1112,"props":3122,"children":3123},{"style":1134},[3124],{"type":59,"value":3125}," experiment_a\u002Fruns.json",{"type":53,"tag":1112,"props":3127,"children":3128},{"style":1134},[3129],{"type":59,"value":3051},{"type":53,"tag":193,"props":3131,"children":3133},{"id":3132},"ab-compare-two-prompts",[3134],{"type":59,"value":3135},"A\u002FB compare two prompts",{"type":53,"tag":1036,"props":3137,"children":3138},{},[3139,3144,3163,3168],{"type":53,"tag":147,"props":3140,"children":3141},{},[3142],{"type":59,"value":3143},"Create two experiments against the same dataset, each using a different prompt version",{"type":53,"tag":147,"props":3145,"children":3146},{},[3147,3149,3155,3157],{"type":59,"value":3148},"Export both: ",{"type":53,"tag":74,"props":3150,"children":3152},{"className":3151},[],[3153],{"type":59,"value":3154},"ax experiments export EXP_A",{"type":59,"value":3156}," and ",{"type":53,"tag":74,"props":3158,"children":3160},{"className":3159},[],[3161],{"type":59,"value":3162},"ax experiments export EXP_B",{"type":53,"tag":147,"props":3164,"children":3165},{},[3166],{"type":59,"value":3167},"Compare average scores, failure rates, and specific example flips",{"type":53,"tag":147,"props":3169,"children":3170},{},[3171],{"type":59,"value":3172},"Check for regressions -- examples that passed with prompt A but fail with prompt B",{"type":53,"tag":136,"props":3174,"children":3176},{"id":3175},"prompt-engineering-best-practices",[3177],{"type":59,"value":3178},"Prompt Engineering Best Practices",{"type":53,"tag":66,"props":3180,"children":3181},{},[3182],{"type":59,"value":3183},"Apply these when writing or revising prompts:",{"type":53,"tag":205,"props":3185,"children":3186},{},[3187,3208],{"type":53,"tag":209,"props":3188,"children":3189},{},[3190],{"type":53,"tag":213,"props":3191,"children":3192},{},[3193,3198,3203],{"type":53,"tag":217,"props":3194,"children":3195},{},[3196],{"type":59,"value":3197},"Technique",{"type":53,"tag":217,"props":3199,"children":3200},{},[3201],{"type":59,"value":3202},"When to apply",{"type":53,"tag":217,"props":3204,"children":3205},{},[3206],{"type":59,"value":3207},"Example",{"type":53,"tag":233,"props":3209,"children":3210},{},[3211,3229,3247,3265,3283,3316,3334,3352,3370],{"type":53,"tag":213,"props":3212,"children":3213},{},[3214,3219,3224],{"type":53,"tag":240,"props":3215,"children":3216},{},[3217],{"type":59,"value":3218},"Clear, detailed instructions",{"type":53,"tag":240,"props":3220,"children":3221},{},[3222],{"type":59,"value":3223},"Output is vague or off-topic",{"type":53,"tag":240,"props":3225,"children":3226},{},[3227],{"type":59,"value":3228},"\"Classify the sentiment as exactly one of: positive, negative, neutral\"",{"type":53,"tag":213,"props":3230,"children":3231},{},[3232,3237,3242],{"type":53,"tag":240,"props":3233,"children":3234},{},[3235],{"type":59,"value":3236},"Instructions at the beginning",{"type":53,"tag":240,"props":3238,"children":3239},{},[3240],{"type":59,"value":3241},"Model ignores later instructions",{"type":53,"tag":240,"props":3243,"children":3244},{},[3245],{"type":59,"value":3246},"Put the task description before examples",{"type":53,"tag":213,"props":3248,"children":3249},{},[3250,3255,3260],{"type":53,"tag":240,"props":3251,"children":3252},{},[3253],{"type":59,"value":3254},"Step-by-step breakdowns",{"type":53,"tag":240,"props":3256,"children":3257},{},[3258],{"type":59,"value":3259},"Complex multi-step processes",{"type":53,"tag":240,"props":3261,"children":3262},{},[3263],{"type":59,"value":3264},"\"First extract entities, then classify each, then summarize\"",{"type":53,"tag":213,"props":3266,"children":3267},{},[3268,3273,3278],{"type":53,"tag":240,"props":3269,"children":3270},{},[3271],{"type":59,"value":3272},"Specific personas",{"type":53,"tag":240,"props":3274,"children":3275},{},[3276],{"type":59,"value":3277},"Need consistent style\u002Ftone",{"type":53,"tag":240,"props":3279,"children":3280},{},[3281],{"type":59,"value":3282},"\"You are a senior financial analyst writing for institutional investors\"",{"type":53,"tag":213,"props":3284,"children":3285},{},[3286,3291,3296],{"type":53,"tag":240,"props":3287,"children":3288},{},[3289],{"type":59,"value":3290},"Delimiter tokens",{"type":53,"tag":240,"props":3292,"children":3293},{},[3294],{"type":59,"value":3295},"Sections blend together",{"type":53,"tag":240,"props":3297,"children":3298},{},[3299,3301,3307,3308,3314],{"type":59,"value":3300},"Use ",{"type":53,"tag":74,"props":3302,"children":3304},{"className":3303},[],[3305],{"type":59,"value":3306},"---",{"type":59,"value":288},{"type":53,"tag":74,"props":3309,"children":3311},{"className":3310},[],[3312],{"type":59,"value":3313},"###",{"type":59,"value":3315},", or XML tags to separate input from instructions",{"type":53,"tag":213,"props":3317,"children":3318},{},[3319,3324,3329],{"type":53,"tag":240,"props":3320,"children":3321},{},[3322],{"type":59,"value":3323},"Few-shot examples",{"type":53,"tag":240,"props":3325,"children":3326},{},[3327],{"type":59,"value":3328},"Output format needs clarification",{"type":53,"tag":240,"props":3330,"children":3331},{},[3332],{"type":59,"value":3333},"Show 2-3 synthetic input\u002Foutput pairs",{"type":53,"tag":213,"props":3335,"children":3336},{},[3337,3342,3347],{"type":53,"tag":240,"props":3338,"children":3339},{},[3340],{"type":59,"value":3341},"Output length specifications",{"type":53,"tag":240,"props":3343,"children":3344},{},[3345],{"type":59,"value":3346},"Responses are too long or short",{"type":53,"tag":240,"props":3348,"children":3349},{},[3350],{"type":59,"value":3351},"\"Respond in exactly 2-3 sentences\"",{"type":53,"tag":213,"props":3353,"children":3354},{},[3355,3360,3365],{"type":53,"tag":240,"props":3356,"children":3357},{},[3358],{"type":59,"value":3359},"Reasoning instructions",{"type":53,"tag":240,"props":3361,"children":3362},{},[3363],{"type":59,"value":3364},"Accuracy is critical",{"type":53,"tag":240,"props":3366,"children":3367},{},[3368],{"type":59,"value":3369},"\"Think step by step before answering\"",{"type":53,"tag":213,"props":3371,"children":3372},{},[3373,3378,3383],{"type":53,"tag":240,"props":3374,"children":3375},{},[3376],{"type":59,"value":3377},"\"I don't know\" guidelines",{"type":53,"tag":240,"props":3379,"children":3380},{},[3381],{"type":59,"value":3382},"Hallucination is a risk",{"type":53,"tag":240,"props":3384,"children":3385},{},[3386],{"type":59,"value":3387},"\"If the answer is not in the provided context, say 'I don't have enough information'\"",{"type":53,"tag":193,"props":3389,"children":3391},{"id":3390},"variable-preservation",[3392],{"type":59,"value":3393},"Variable preservation",{"type":53,"tag":66,"props":3395,"children":3396},{},[3397],{"type":59,"value":3398},"When optimizing prompts that use template variables:",{"type":53,"tag":143,"props":3400,"children":3401},{},[3402,3418,3434,3439,3444],{"type":53,"tag":147,"props":3403,"children":3404},{},[3405,3410,3411,3416],{"type":53,"tag":70,"props":3406,"children":3407},{},[3408],{"type":59,"value":3409},"Single braces",{"type":59,"value":480},{"type":53,"tag":74,"props":3412,"children":3414},{"className":3413},[],[3415],{"type":59,"value":380},{"type":59,"value":3417},"): Python f-string \u002F Jinja style. Most common in Arize.",{"type":53,"tag":147,"props":3419,"children":3420},{},[3421,3426,3427,3432],{"type":53,"tag":70,"props":3422,"children":3423},{},[3424],{"type":59,"value":3425},"Double braces",{"type":59,"value":480},{"type":53,"tag":74,"props":3428,"children":3430},{"className":3429},[],[3431],{"type":59,"value":1854},{"type":59,"value":3433},"): Mustache style. Used when the framework requires it.",{"type":53,"tag":147,"props":3435,"children":3436},{},[3437],{"type":59,"value":3438},"Never add or remove variable placeholders during optimization",{"type":53,"tag":147,"props":3440,"children":3441},{},[3442],{"type":59,"value":3443},"Never rename variables -- the runtime substitution depends on exact names",{"type":53,"tag":147,"props":3445,"children":3446},{},[3447],{"type":59,"value":3448},"If adding few-shot examples, use literal values, not variable placeholders",{"type":53,"tag":136,"props":3450,"children":3452},{"id":3451},"workflows",[3453],{"type":59,"value":3454},"Workflows",{"type":53,"tag":193,"props":3456,"children":3458},{"id":3457},"optimize-a-prompt-from-a-failing-trace",[3459],{"type":59,"value":3460},"Optimize a prompt from a failing trace",{"type":53,"tag":1036,"props":3462,"children":3463},{},[3464,3524,3563,3644,3649,3654],{"type":53,"tag":147,"props":3465,"children":3466},{},[3467,3469],{"type":59,"value":3468},"Find failing traces:\n",{"type":53,"tag":1101,"props":3470,"children":3472},{"className":1103,"code":3471,"language":1105,"meta":1106,"style":1106},"ax traces list PROJECT --filter \"status_code = 'ERROR'\" --limit 5\n",[3473],{"type":53,"tag":74,"props":3474,"children":3475},{"__ignoreMap":1106},[3476],{"type":53,"tag":1112,"props":3477,"children":3478},{"class":1114,"line":1115},[3479,3483,3488,3493,3497,3501,3505,3510,3514,3519],{"type":53,"tag":1112,"props":3480,"children":3481},{"style":1129},[3482],{"type":59,"value":828},{"type":53,"tag":1112,"props":3484,"children":3485},{"style":1134},[3486],{"type":59,"value":3487}," traces",{"type":53,"tag":1112,"props":3489,"children":3490},{"style":1134},[3491],{"type":59,"value":3492}," list",{"type":53,"tag":1112,"props":3494,"children":3495},{"style":1134},[3496],{"type":59,"value":1147},{"type":53,"tag":1112,"props":3498,"children":3499},{"style":1134},[3500],{"type":59,"value":1152},{"type":53,"tag":1112,"props":3502,"children":3503},{"style":1155},[3504],{"type":59,"value":1158},{"type":53,"tag":1112,"props":3506,"children":3507},{"style":1134},[3508],{"type":59,"value":3509},"status_code = 'ERROR'",{"type":53,"tag":1112,"props":3511,"children":3512},{"style":1155},[3513],{"type":59,"value":1167},{"type":53,"tag":1112,"props":3515,"children":3516},{"style":1134},[3517],{"type":59,"value":3518}," --limit",{"type":53,"tag":1112,"props":3520,"children":3521},{"style":1175},[3522],{"type":59,"value":3523}," 5\n",{"type":53,"tag":147,"props":3525,"children":3526},{},[3527,3529],{"type":59,"value":3528},"Export the trace:\n",{"type":53,"tag":1101,"props":3530,"children":3532},{"className":1103,"code":3531,"language":1105,"meta":1106,"style":1106},"ax spans export PROJECT --trace-id TRACE_ID\n",[3533],{"type":53,"tag":74,"props":3534,"children":3535},{"__ignoreMap":1106},[3536],{"type":53,"tag":1112,"props":3537,"children":3538},{"class":1114,"line":1115},[3539,3543,3547,3551,3555,3559],{"type":53,"tag":1112,"props":3540,"children":3541},{"style":1129},[3542],{"type":59,"value":828},{"type":53,"tag":1112,"props":3544,"children":3545},{"style":1134},[3546],{"type":59,"value":1137},{"type":53,"tag":1112,"props":3548,"children":3549},{"style":1134},[3550],{"type":59,"value":1142},{"type":53,"tag":1112,"props":3552,"children":3553},{"style":1134},[3554],{"type":59,"value":1147},{"type":53,"tag":1112,"props":3556,"children":3557},{"style":1134},[3558],{"type":59,"value":1361},{"type":53,"tag":1112,"props":3560,"children":3561},{"style":1134},[3562],{"type":59,"value":1366},{"type":53,"tag":147,"props":3564,"children":3565},{},[3566,3568],{"type":59,"value":3567},"Extract the prompt from the LLM span:\n",{"type":53,"tag":1101,"props":3569,"children":3571},{"className":1103,"code":3570,"language":1105,"meta":1106,"style":1106},"jq '[.[] | select(.attributes.openinference.span.kind == \"LLM\")][0] | {\n  messages: .attributes.llm.input_messages,\n  template: .attributes.llm.prompt_template,\n  output: .attributes.output.value,\n  error: .attributes.exception.message\n}' trace_*\u002Fspans.json\n",[3572],{"type":53,"tag":74,"props":3573,"children":3574},{"__ignoreMap":1106},[3575,3591,3598,3606,3613,3621],{"type":53,"tag":1112,"props":3576,"children":3577},{"class":1114,"line":1115},[3578,3582,3586],{"type":53,"tag":1112,"props":3579,"children":3580},{"style":1129},[3581],{"type":59,"value":1439},{"type":53,"tag":1112,"props":3583,"children":3584},{"style":1155},[3585],{"type":59,"value":1444},{"type":53,"tag":1112,"props":3587,"children":3588},{"style":1134},[3589],{"type":59,"value":3590},"[.[] | select(.attributes.openinference.span.kind == \"LLM\")][0] | {\n",{"type":53,"tag":1112,"props":3592,"children":3593},{"class":1114,"line":1125},[3594],{"type":53,"tag":1112,"props":3595,"children":3596},{"style":1134},[3597],{"type":59,"value":1457},{"type":53,"tag":1112,"props":3599,"children":3600},{"class":1114,"line":1186},[3601],{"type":53,"tag":1112,"props":3602,"children":3603},{"style":1134},[3604],{"type":59,"value":3605},"  template: .attributes.llm.prompt_template,\n",{"type":53,"tag":1112,"props":3607,"children":3608},{"class":1114,"line":1196},[3609],{"type":53,"tag":1112,"props":3610,"children":3611},{"style":1134},[3612],{"type":59,"value":2861},{"type":53,"tag":1112,"props":3614,"children":3615},{"class":1114,"line":27},[3616],{"type":53,"tag":1112,"props":3617,"children":3618},{"style":1134},[3619],{"type":59,"value":3620},"  error: .attributes.exception.message\n",{"type":53,"tag":1112,"props":3622,"children":3623},{"class":1114,"line":1253},[3624,3628,3632,3636,3640],{"type":53,"tag":1112,"props":3625,"children":3626},{"style":1134},[3627],{"type":59,"value":1473},{"type":53,"tag":1112,"props":3629,"children":3630},{"style":1155},[3631],{"type":59,"value":1478},{"type":53,"tag":1112,"props":3633,"children":3634},{"style":1134},[3635],{"type":59,"value":1483},{"type":53,"tag":1112,"props":3637,"children":3638},{"style":1486},[3639],{"type":59,"value":1489},{"type":53,"tag":1112,"props":3641,"children":3642},{"style":1134},[3643],{"type":59,"value":1494},{"type":53,"tag":147,"props":3645,"children":3646},{},[3647],{"type":59,"value":3648},"Identify what failed from the error message or output",{"type":53,"tag":147,"props":3650,"children":3651},{},[3652],{"type":59,"value":3653},"Fill in the optimization meta-prompt (Phase 3) with the prompt and error context",{"type":53,"tag":147,"props":3655,"children":3656},{},[3657],{"type":59,"value":3658},"Apply the revised prompt",{"type":53,"tag":193,"props":3660,"children":3662},{"id":3661},"optimize-using-a-dataset-and-experiment",[3663],{"type":59,"value":3664},"Optimize using a dataset and experiment",{"type":53,"tag":1036,"props":3666,"children":3667},{},[3668,3734,3808,3813,3818],{"type":53,"tag":147,"props":3669,"children":3670},{},[3671,3673],{"type":59,"value":3672},"Find the dataset and experiment:\n",{"type":53,"tag":1101,"props":3674,"children":3676},{"className":1103,"code":3675,"language":1105,"meta":1106,"style":1106},"ax datasets list --space SPACE\nax experiments list --dataset DATASET_NAME --space SPACE\n",[3677],{"type":53,"tag":74,"props":3678,"children":3679},{"__ignoreMap":1106},[3680,3703],{"type":53,"tag":1112,"props":3681,"children":3682},{"class":1114,"line":1115},[3683,3687,3691,3695,3699],{"type":53,"tag":1112,"props":3684,"children":3685},{"style":1129},[3686],{"type":59,"value":828},{"type":53,"tag":1112,"props":3688,"children":3689},{"style":1134},[3690],{"type":59,"value":2181},{"type":53,"tag":1112,"props":3692,"children":3693},{"style":1134},[3694],{"type":59,"value":3492},{"type":53,"tag":1112,"props":3696,"children":3697},{"style":1134},[3698],{"type":59,"value":2195},{"type":53,"tag":1112,"props":3700,"children":3701},{"style":1134},[3702],{"type":59,"value":2200},{"type":53,"tag":1112,"props":3704,"children":3705},{"class":1114,"line":1125},[3706,3710,3714,3718,3722,3726,3730],{"type":53,"tag":1112,"props":3707,"children":3708},{"style":1129},[3709],{"type":59,"value":828},{"type":53,"tag":1112,"props":3711,"children":3712},{"style":1134},[3713],{"type":59,"value":2235},{"type":53,"tag":1112,"props":3715,"children":3716},{"style":1134},[3717],{"type":59,"value":3492},{"type":53,"tag":1112,"props":3719,"children":3720},{"style":1134},[3721],{"type":59,"value":2249},{"type":53,"tag":1112,"props":3723,"children":3724},{"style":1134},[3725],{"type":59,"value":2190},{"type":53,"tag":1112,"props":3727,"children":3728},{"style":1134},[3729],{"type":59,"value":2195},{"type":53,"tag":1112,"props":3731,"children":3732},{"style":1134},[3733],{"type":59,"value":2200},{"type":53,"tag":147,"props":3735,"children":3736},{},[3737,3739],{"type":59,"value":3738},"Export both:\n",{"type":53,"tag":1101,"props":3740,"children":3742},{"className":1103,"code":3741,"language":1105,"meta":1106,"style":1106},"ax datasets export DATASET_NAME --space SPACE\nax experiments export EXPERIMENT_NAME --dataset DATASET_NAME --space SPACE\n",[3743],{"type":53,"tag":74,"props":3744,"children":3745},{"__ignoreMap":1106},[3746,3773],{"type":53,"tag":1112,"props":3747,"children":3748},{"class":1114,"line":1115},[3749,3753,3757,3761,3765,3769],{"type":53,"tag":1112,"props":3750,"children":3751},{"style":1129},[3752],{"type":59,"value":828},{"type":53,"tag":1112,"props":3754,"children":3755},{"style":1134},[3756],{"type":59,"value":2181},{"type":53,"tag":1112,"props":3758,"children":3759},{"style":1134},[3760],{"type":59,"value":1142},{"type":53,"tag":1112,"props":3762,"children":3763},{"style":1134},[3764],{"type":59,"value":2190},{"type":53,"tag":1112,"props":3766,"children":3767},{"style":1134},[3768],{"type":59,"value":2195},{"type":53,"tag":1112,"props":3770,"children":3771},{"style":1134},[3772],{"type":59,"value":2200},{"type":53,"tag":1112,"props":3774,"children":3775},{"class":1114,"line":1125},[3776,3780,3784,3788,3792,3796,3800,3804],{"type":53,"tag":1112,"props":3777,"children":3778},{"style":1129},[3779],{"type":59,"value":828},{"type":53,"tag":1112,"props":3781,"children":3782},{"style":1134},[3783],{"type":59,"value":2235},{"type":53,"tag":1112,"props":3785,"children":3786},{"style":1134},[3787],{"type":59,"value":1142},{"type":53,"tag":1112,"props":3789,"children":3790},{"style":1134},[3791],{"type":59,"value":2244},{"type":53,"tag":1112,"props":3793,"children":3794},{"style":1134},[3795],{"type":59,"value":2249},{"type":53,"tag":1112,"props":3797,"children":3798},{"style":1134},[3799],{"type":59,"value":2190},{"type":53,"tag":1112,"props":3801,"children":3802},{"style":1134},[3803],{"type":59,"value":2195},{"type":53,"tag":1112,"props":3805,"children":3806},{"style":1134},[3807],{"type":59,"value":2200},{"type":53,"tag":147,"props":3809,"children":3810},{},[3811],{"type":59,"value":3812},"Prepare the joined data for the meta-prompt",{"type":53,"tag":147,"props":3814,"children":3815},{},[3816],{"type":59,"value":3817},"Run the optimization meta-prompt",{"type":53,"tag":147,"props":3819,"children":3820},{},[3821],{"type":59,"value":3822},"Create a new experiment with the revised prompt to measure improvement",{"type":53,"tag":193,"props":3824,"children":3826},{"id":3825},"debug-a-prompt-that-produces-wrong-format",[3827],{"type":59,"value":3828},"Debug a prompt that produces wrong format",{"type":53,"tag":1036,"props":3830,"children":3831},{},[3832,3917,3922,3927],{"type":53,"tag":147,"props":3833,"children":3834},{},[3835,3837],{"type":59,"value":3836},"Export spans where the output format is wrong:\n",{"type":53,"tag":1101,"props":3838,"children":3840},{"className":1103,"code":3839,"language":1105,"meta":1106,"style":1106},"ax spans export PROJECT \\\n  --filter \"attributes.openinference.span.kind = 'LLM' AND annotation.format.label = 'incorrect'\" \\\n  -l 10 --stdout > bad_format.json\n",[3841],{"type":53,"tag":74,"props":3842,"children":3843},{"__ignoreMap":1106},[3844,3867,3891],{"type":53,"tag":1112,"props":3845,"children":3846},{"class":1114,"line":1115},[3847,3851,3855,3859,3863],{"type":53,"tag":1112,"props":3848,"children":3849},{"style":1129},[3850],{"type":59,"value":828},{"type":53,"tag":1112,"props":3852,"children":3853},{"style":1134},[3854],{"type":59,"value":1137},{"type":53,"tag":1112,"props":3856,"children":3857},{"style":1134},[3858],{"type":59,"value":1142},{"type":53,"tag":1112,"props":3860,"children":3861},{"style":1134},[3862],{"type":59,"value":1147},{"type":53,"tag":1112,"props":3864,"children":3865},{"style":1486},[3866],{"type":59,"value":1907},{"type":53,"tag":1112,"props":3868,"children":3869},{"class":1114,"line":1125},[3870,3874,3878,3883,3887],{"type":53,"tag":1112,"props":3871,"children":3872},{"style":1134},[3873],{"type":59,"value":1915},{"type":53,"tag":1112,"props":3875,"children":3876},{"style":1155},[3877],{"type":59,"value":1158},{"type":53,"tag":1112,"props":3879,"children":3880},{"style":1134},[3881],{"type":59,"value":3882},"attributes.openinference.span.kind = 'LLM' AND annotation.format.label = 'incorrect'",{"type":53,"tag":1112,"props":3884,"children":3885},{"style":1155},[3886],{"type":59,"value":1167},{"type":53,"tag":1112,"props":3888,"children":3889},{"style":1486},[3890],{"type":59,"value":1907},{"type":53,"tag":1112,"props":3892,"children":3893},{"class":1114,"line":1186},[3894,3898,3902,3907,3912],{"type":53,"tag":1112,"props":3895,"children":3896},{"style":1134},[3897],{"type":59,"value":1940},{"type":53,"tag":1112,"props":3899,"children":3900},{"style":1175},[3901],{"type":59,"value":1178},{"type":53,"tag":1112,"props":3903,"children":3904},{"style":1134},[3905],{"type":59,"value":3906}," --stdout",{"type":53,"tag":1112,"props":3908,"children":3909},{"style":1155},[3910],{"type":59,"value":3911}," >",{"type":53,"tag":1112,"props":3913,"children":3914},{"style":1134},[3915],{"type":59,"value":3916}," bad_format.json\n",{"type":53,"tag":147,"props":3918,"children":3919},{},[3920],{"type":59,"value":3921},"Look at what the LLM is producing vs what was expected",{"type":53,"tag":147,"props":3923,"children":3924},{},[3925],{"type":59,"value":3926},"Add explicit format instructions to the prompt (JSON schema, examples, delimiters)",{"type":53,"tag":147,"props":3928,"children":3929},{},[3930],{"type":59,"value":3931},"Common fix: add a few-shot example showing the exact desired output format",{"type":53,"tag":193,"props":3933,"children":3935},{"id":3934},"reduce-hallucination-in-a-rag-prompt",[3936],{"type":59,"value":3937},"Reduce hallucination in a RAG prompt",{"type":53,"tag":1036,"props":3939,"children":3940},{},[3941,4015,4086,4091],{"type":53,"tag":147,"props":3942,"children":3943},{},[3944,3946],{"type":59,"value":3945},"Find traces where the model hallucinated:\n",{"type":53,"tag":1101,"props":3947,"children":3949},{"className":1103,"code":3948,"language":1105,"meta":1106,"style":1106},"ax spans export PROJECT \\\n  --filter \"annotation.faithfulness.label = 'unfaithful'\" \\\n  -l 20 --stdout\n",[3950],{"type":53,"tag":74,"props":3951,"children":3952},{"__ignoreMap":1106},[3953,3976,4000],{"type":53,"tag":1112,"props":3954,"children":3955},{"class":1114,"line":1115},[3956,3960,3964,3968,3972],{"type":53,"tag":1112,"props":3957,"children":3958},{"style":1129},[3959],{"type":59,"value":828},{"type":53,"tag":1112,"props":3961,"children":3962},{"style":1134},[3963],{"type":59,"value":1137},{"type":53,"tag":1112,"props":3965,"children":3966},{"style":1134},[3967],{"type":59,"value":1142},{"type":53,"tag":1112,"props":3969,"children":3970},{"style":1134},[3971],{"type":59,"value":1147},{"type":53,"tag":1112,"props":3973,"children":3974},{"style":1486},[3975],{"type":59,"value":1907},{"type":53,"tag":1112,"props":3977,"children":3978},{"class":1114,"line":1125},[3979,3983,3987,3992,3996],{"type":53,"tag":1112,"props":3980,"children":3981},{"style":1134},[3982],{"type":59,"value":1915},{"type":53,"tag":1112,"props":3984,"children":3985},{"style":1155},[3986],{"type":59,"value":1158},{"type":53,"tag":1112,"props":3988,"children":3989},{"style":1134},[3990],{"type":59,"value":3991},"annotation.faithfulness.label = 'unfaithful'",{"type":53,"tag":1112,"props":3993,"children":3994},{"style":1155},[3995],{"type":59,"value":1167},{"type":53,"tag":1112,"props":3997,"children":3998},{"style":1486},[3999],{"type":59,"value":1907},{"type":53,"tag":1112,"props":4001,"children":4002},{"class":1114,"line":1186},[4003,4007,4011],{"type":53,"tag":1112,"props":4004,"children":4005},{"style":1134},[4006],{"type":59,"value":1940},{"type":53,"tag":1112,"props":4008,"children":4009},{"style":1175},[4010],{"type":59,"value":1945},{"type":53,"tag":1112,"props":4012,"children":4013},{"style":1134},[4014],{"type":59,"value":1183},{"type":53,"tag":147,"props":4016,"children":4017},{},[4018,4020],{"type":59,"value":4019},"Export and inspect the retriever + LLM spans together:\n",{"type":53,"tag":1101,"props":4021,"children":4023},{"className":1103,"code":4022,"language":1105,"meta":1106,"style":1106},"ax spans export PROJECT --trace-id TRACE_ID\njq '[.[] | {kind: .attributes.openinference.span.kind, name, input: .attributes.input.value, output: .attributes.output.value}]' trace_*\u002Fspans.json\n",[4024],{"type":53,"tag":74,"props":4025,"children":4026},{"__ignoreMap":1106},[4027,4054],{"type":53,"tag":1112,"props":4028,"children":4029},{"class":1114,"line":1115},[4030,4034,4038,4042,4046,4050],{"type":53,"tag":1112,"props":4031,"children":4032},{"style":1129},[4033],{"type":59,"value":828},{"type":53,"tag":1112,"props":4035,"children":4036},{"style":1134},[4037],{"type":59,"value":1137},{"type":53,"tag":1112,"props":4039,"children":4040},{"style":1134},[4041],{"type":59,"value":1142},{"type":53,"tag":1112,"props":4043,"children":4044},{"style":1134},[4045],{"type":59,"value":1147},{"type":53,"tag":1112,"props":4047,"children":4048},{"style":1134},[4049],{"type":59,"value":1361},{"type":53,"tag":1112,"props":4051,"children":4052},{"style":1134},[4053],{"type":59,"value":1366},{"type":53,"tag":1112,"props":4055,"children":4056},{"class":1114,"line":1125},[4057,4061,4065,4070,4074,4078,4082],{"type":53,"tag":1112,"props":4058,"children":4059},{"style":1129},[4060],{"type":59,"value":1439},{"type":53,"tag":1112,"props":4062,"children":4063},{"style":1155},[4064],{"type":59,"value":1444},{"type":53,"tag":1112,"props":4066,"children":4067},{"style":1134},[4068],{"type":59,"value":4069},"[.[] | {kind: .attributes.openinference.span.kind, name, input: .attributes.input.value, output: .attributes.output.value}]",{"type":53,"tag":1112,"props":4071,"children":4072},{"style":1155},[4073],{"type":59,"value":1478},{"type":53,"tag":1112,"props":4075,"children":4076},{"style":1134},[4077],{"type":59,"value":1483},{"type":53,"tag":1112,"props":4079,"children":4080},{"style":1486},[4081],{"type":59,"value":1489},{"type":53,"tag":1112,"props":4083,"children":4084},{"style":1134},[4085],{"type":59,"value":1494},{"type":53,"tag":147,"props":4087,"children":4088},{},[4089],{"type":59,"value":4090},"Check if the retrieved context actually contained the answer",{"type":53,"tag":147,"props":4092,"children":4093},{},[4094],{"type":59,"value":4095},"Add grounding instructions to the system prompt: \"Only use information from the provided context. If the answer is not in the context, say so.\"",{"type":53,"tag":136,"props":4097,"children":4099},{"id":4098},"troubleshooting",[4100],{"type":59,"value":4101},"Troubleshooting",{"type":53,"tag":205,"props":4103,"children":4104},{},[4105,4121],{"type":53,"tag":209,"props":4106,"children":4107},{},[4108],{"type":53,"tag":213,"props":4109,"children":4110},{},[4111,4116],{"type":53,"tag":217,"props":4112,"children":4113},{},[4114],{"type":59,"value":4115},"Problem",{"type":53,"tag":217,"props":4117,"children":4118},{},[4119],{"type":59,"value":4120},"Solution",{"type":53,"tag":233,"props":4122,"children":4123},{},[4124,4145,4168,4189,4230,4251,4264,4277,4303],{"type":53,"tag":213,"props":4125,"children":4126},{},[4127,4136],{"type":53,"tag":240,"props":4128,"children":4129},{},[4130],{"type":53,"tag":74,"props":4131,"children":4133},{"className":4132},[],[4134],{"type":59,"value":4135},"ax: command not found",{"type":53,"tag":240,"props":4137,"children":4138},{},[4139,4141],{"type":59,"value":4140},"See ",{"type":53,"tag":858,"props":4142,"children":4143},{"href":860},[4144],{"type":59,"value":860},{"type":53,"tag":213,"props":4146,"children":4147},{},[4148,4157],{"type":53,"tag":240,"props":4149,"children":4150},{},[4151],{"type":53,"tag":74,"props":4152,"children":4154},{"className":4153},[],[4155],{"type":59,"value":4156},"No profile found",{"type":53,"tag":240,"props":4158,"children":4159},{},[4160,4162,4166],{"type":59,"value":4161},"No profile is configured. See ",{"type":53,"tag":858,"props":4163,"children":4164},{"href":884},[4165],{"type":59,"value":884},{"type":59,"value":4167}," to create one.",{"type":53,"tag":213,"props":4169,"children":4170},{},[4171,4184],{"type":53,"tag":240,"props":4172,"children":4173},{},[4174,4176,4182],{"type":59,"value":4175},"No ",{"type":53,"tag":74,"props":4177,"children":4179},{"className":4178},[],[4180],{"type":59,"value":4181},"input_messages",{"type":59,"value":4183}," on span",{"type":53,"tag":240,"props":4185,"children":4186},{},[4187],{"type":59,"value":4188},"Check span kind -- Chain\u002FAgent spans store prompts on child LLM spans, not on themselves",{"type":53,"tag":213,"props":4190,"children":4191},{},[4192,4203],{"type":53,"tag":240,"props":4193,"children":4194},{},[4195,4197],{"type":59,"value":4196},"Prompt template is ",{"type":53,"tag":74,"props":4198,"children":4200},{"className":4199},[],[4201],{"type":59,"value":4202},"null",{"type":53,"tag":240,"props":4204,"children":4205},{},[4206,4208,4214,4216,4221,4222,4228],{"type":59,"value":4207},"Not all instrumentations emit ",{"type":53,"tag":74,"props":4209,"children":4211},{"className":4210},[],[4212],{"type":59,"value":4213},"prompt_template",{"type":59,"value":4215},". Use ",{"type":53,"tag":74,"props":4217,"children":4219},{"className":4218},[],[4220],{"type":59,"value":4181},{"type":59,"value":1848},{"type":53,"tag":74,"props":4223,"children":4225},{"className":4224},[],[4226],{"type":59,"value":4227},"input.value",{"type":59,"value":4229}," instead",{"type":53,"tag":213,"props":4231,"children":4232},{},[4233,4238],{"type":53,"tag":240,"props":4234,"children":4235},{},[4236],{"type":59,"value":4237},"Variables lost after optimization",{"type":53,"tag":240,"props":4239,"children":4240},{},[4241,4243,4249],{"type":59,"value":4242},"Verify the revised prompt preserves all ",{"type":53,"tag":74,"props":4244,"children":4246},{"className":4245},[],[4247],{"type":59,"value":4248},"{var}",{"type":59,"value":4250}," placeholders from the original",{"type":53,"tag":213,"props":4252,"children":4253},{},[4254,4259],{"type":53,"tag":240,"props":4255,"children":4256},{},[4257],{"type":59,"value":4258},"Optimization makes things worse",{"type":53,"tag":240,"props":4260,"children":4261},{},[4262],{"type":59,"value":4263},"Check for overfitting -- the meta-prompt may have memorized test data. Ensure few-shot examples are synthetic",{"type":53,"tag":213,"props":4265,"children":4266},{},[4267,4272],{"type":53,"tag":240,"props":4268,"children":4269},{},[4270],{"type":59,"value":4271},"No eval\u002Fannotation columns",{"type":53,"tag":240,"props":4273,"children":4274},{},[4275],{"type":59,"value":4276},"Run evaluations first (via Arize UI or SDK), then re-export",{"type":53,"tag":213,"props":4278,"children":4279},{},[4280,4285],{"type":53,"tag":240,"props":4281,"children":4282},{},[4283],{"type":59,"value":4284},"Experiment output column not found",{"type":53,"tag":240,"props":4286,"children":4287},{},[4288,4290,4295,4297],{"type":59,"value":4289},"The column name is ",{"type":53,"tag":74,"props":4291,"children":4293},{"className":4292},[],[4294],{"type":59,"value":801},{"type":59,"value":4296}," -- check exact experiment name via ",{"type":53,"tag":74,"props":4298,"children":4300},{"className":4299},[],[4301],{"type":59,"value":4302},"ax experiments get",{"type":53,"tag":213,"props":4304,"children":4305},{},[4306,4316],{"type":53,"tag":240,"props":4307,"children":4308},{},[4309,4314],{"type":53,"tag":74,"props":4310,"children":4312},{"className":4311},[],[4313],{"type":59,"value":1439},{"type":59,"value":4315}," errors on span JSON",{"type":53,"tag":240,"props":4317,"children":4318},{},[4319,4321,4327],{"type":59,"value":4320},"Ensure you're targeting the correct file path (e.g., ",{"type":53,"tag":74,"props":4322,"children":4324},{"className":4323},[],[4325],{"type":59,"value":4326},"trace_*\u002Fspans.json",{"type":59,"value":390},{"type":53,"tag":4329,"props":4330,"children":4331},"style",{},[4332],{"type":59,"value":4333},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":4335,"total":2881},[4336,4350,4362,4374,4386,4396,4410,4420,4431,4447,4466,4476],{"slug":4337,"name":4337,"fn":4338,"description":4339,"org":4340,"tags":4341,"stars":4347,"repoUrl":4348,"updatedAt":4349},"annotate-spans","annotate LLM spans and traces","Write effective, consistent annotations on LLM\u002Fagent spans and traces, and coach the user on annotation practice. Load this whenever you are about to record structured feedback with the `batch_span_annotate` tool, or when the user asks how to annotate, label, score, or review spans\u002Ftraces, build a failure taxonomy, or set up human\u002FLLM review. Do NOT load for: pure analysis with no intent to save feedback (use debug-trace), latency or cost statistics, or prompt authoring (use playground).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4342,4343,4344,4345],{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":4346,"slug":38,"type":16},"Tracing",10513,"https:\u002F\u002Fgithub.com\u002FArize-ai\u002Fphoenix","2026-07-12T08:08:14.140984",{"slug":35,"name":35,"fn":4351,"description":4352,"org":4353,"tags":4354,"stars":4347,"repoUrl":4348,"updatedAt":4361},"reason about Phoenix dataset structure","Understand what a Phoenix dataset is and reason well about its examples, outputs, splits, and how it feeds evaluators and experiments. Load this whenever a dataset is in view or the user asks what a dataset is, how splits work, what an output \"means\", or how datasets relate to experiments and evals. This skill governs the judgment; any tool descriptions govern the mechanics.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4355,4358,4360],{"name":4356,"slug":4357,"type":16},"Data Analysis","data-analysis",{"name":4359,"slug":35,"type":16},"Datasets",{"name":21,"slug":22,"type":16},"2026-07-12T08:08:21.695457",{"slug":4363,"name":4363,"fn":4364,"description":4365,"org":4366,"tags":4367,"stars":4347,"repoUrl":4348,"updatedAt":4373},"debug-trace","diagnose failures using trace investigation","Diagnose failure modes by systematically investigating traces. Trigger when the user explicitly asks for cross-trace diagnosis: \"what's going wrong?\", \"were there errors?\", \"debug this\", \"where is my agent struggling?\". Do NOT trigger on: (1) advice questions (\"what should I do?\"), (2) statistical questions (\"what's the average latency?\"), (3) summarize requests, (4) trace filtering (\"show me traces with errors\"), (5) vague questions (\"is there a problem?\"), (6) unrelated requests.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4368,4371,4372],{"name":4369,"slug":4370,"type":16},"Debugging","debugging",{"name":14,"slug":15,"type":16},{"name":4346,"slug":38,"type":16},"2026-07-12T08:08:10.44243",{"slug":4375,"name":4375,"fn":4376,"description":4377,"org":4378,"tags":4379,"stars":4347,"repoUrl":4348,"updatedAt":4385},"evaluators","author and refine Phoenix evaluators","Author or refine a Phoenix evaluator — code or LLM-as-a-judge — that scores a run's output. Trigger when the user wants to create a new evaluator, improve an existing one's logic or rubric, choose labels, or decide what to measure on a dataset or experiment. Do NOT trigger on: (1) manual prompt drafting (use `playground`), (2) running or comparing experiments themselves (use `experiments`), (3) cross-trace failure diagnosis with no evaluator in scope (use `debug-trace`).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4380,4381,4382],{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":4383,"slug":4384,"type":16},"Testing","testing","2026-07-31T05:58:09.13624",{"slug":36,"name":36,"fn":4387,"description":4388,"org":4389,"tags":4390,"stars":4347,"repoUrl":4348,"updatedAt":4395},"run and compare dataset-backed experiments","Run, read, and compare dataset-backed experiments to find evidence that a prompt or pipeline is improving. Trigger when the user wants to iterate over a dataset with experiments, compare experiment runs, read experiment quality\u002Flatency\u002Fcost, or decide whether a change actually helped. Running a prompt over a dataset is implicitly an experiment — load this skill when dataset-backed work begins, before authoring evaluators for the experiment and before starting the recorded run, not only when reading results. Do NOT trigger on: (1) manual prompt drafting with no dataset-backed evaluation in scope (use `playground`), (2) authoring or refining an evaluator's logic or rubric (use `evaluators`), (3) cross-trace failure diagnosis with no experiment in scope (use `debug-trace`).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4391,4392,4393,4394],{"name":4359,"slug":35,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":4383,"slug":4384,"type":16},"2026-07-12T08:08:11.691477",{"slug":4397,"name":4397,"fn":4398,"description":4399,"org":4400,"tags":4401,"stars":4347,"repoUrl":4348,"updatedAt":4409},"phoenix-graphql","query Phoenix API with GraphQL","Write efficient GraphQL queries against the Phoenix API. Load this skill in two cases: (1) before composing any non-trivial GraphQL query yourself for data analysis (via the `phoenix-gql` bash command) — it contains schema entrypoints and patterns that eliminate the need for introspection; (2) when the user asks for help writing GraphQL queries for their own scripts, tools, or integrations against Phoenix — it covers the endpoint, authentication, and client examples.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4402,4405,4406],{"name":4403,"slug":4404,"type":16},"Analytics","analytics",{"name":4356,"slug":4357,"type":16},{"name":4407,"slug":4408,"type":16},"GraphQL","graphql","2026-07-12T08:08:17.163493",{"slug":4411,"name":4411,"fn":4412,"description":4413,"org":4414,"tags":4415,"stars":4347,"repoUrl":4348,"updatedAt":4419},"playground","author and iterate on prompts in Phoenix","Author, edit, or iterate on prompts in the Phoenix prompt playground, including running experiments over a dataset. Load before any playground tool call, including single-shot prompt rewrites.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4416,4417,4418],{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":4383,"slug":4384,"type":16},"2026-07-12T08:08:12.920792",{"slug":4421,"name":4421,"fn":4422,"description":4423,"org":4424,"tags":4425,"stars":4347,"repoUrl":4348,"updatedAt":4430},"span-coding","analyze and code Phoenix spans","Open-code Phoenix spans with PXI-owned notes, recover those notes for axial coding, and promote stable categories into structured annotations. Load this when analyzing spans to discover failure patterns before a taxonomy exists.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4426,4427,4428,4429],{"name":4369,"slug":4370,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":4346,"slug":38,"type":16},"2026-07-12T08:08:19.597239",{"slug":4432,"name":4432,"fn":4433,"description":4434,"org":4435,"tags":4436,"stars":23,"repoUrl":24,"updatedAt":4446},"arize-admin","manage Arize enterprise user access","Manages Arize users, organizations, spaces, projects, roles, role bindings, resource restrictions, and API keys via the ax CLI. Use for enterprise admin workflows: inviting and offboarding users, onboarding new teams, creating custom roles for SAML\u002FSSO mappings, assigning roles to users, restricting project-level access, and managing service keys for multi-tenant architectures. Covers ax users, ax organizations, ax spaces, ax projects, ax roles, ax role-bindings, and ax api-keys.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4437,4440,4443],{"name":4438,"slug":4439,"type":16},"CLI","cli",{"name":4441,"slug":4442,"type":16},"Operations","operations",{"name":4444,"slug":4445,"type":16},"Permissions","permissions","2026-07-22T05:37:21.991338",{"slug":940,"name":940,"fn":4448,"description":4449,"org":4450,"tags":4451,"stars":23,"repoUrl":24,"updatedAt":4465},"manage Arize AI provider integrations","Creates, reads, updates, and deletes Arize AI integrations that store LLM provider credentials used by evaluators and other Arize features. Supports any LLM provider (e.g. OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Gemini, NVIDIA NIM). Use when the user mentions AI integration, LLM provider credentials, create integration, list integrations, update credentials, delete integration, or connecting an LLM provider to Arize.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4452,4455,4458,4461,4462],{"name":4453,"slug":4454,"type":16},"Anthropic","anthropic",{"name":4456,"slug":4457,"type":16},"Azure","azure",{"name":4459,"slug":4460,"type":16},"Integrations","integrations",{"name":18,"slug":19,"type":16},{"name":4463,"slug":4464,"type":16},"OpenAI","openai","2026-07-22T05:37:23.90468",{"slug":4467,"name":4467,"fn":4468,"description":4469,"org":4470,"tags":4471,"stars":23,"repoUrl":24,"updatedAt":4475},"arize-annotation","manage Arize annotation workflows","Creates and manages annotation configs (categorical, continuous, freeform label schemas) and annotation queues (human review workflows) on Arize. Applies human annotations to project spans via the Python SDK. Use when the user mentions annotation config, annotation queue, label schema, human feedback, bulk annotate spans, update_annotations, labeling queue, annotate record, or human review.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4472,4473,4474],{"name":4356,"slug":4357,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"2026-07-22T05:37:19.010776",{"slug":4477,"name":4477,"fn":4478,"description":4479,"org":4480,"tags":4481,"stars":23,"repoUrl":24,"updatedAt":4497},"arize-compliance-audit","audit AI agents for regulatory compliance","INVOKE THIS SKILL when auditing an AI agent or LLM app for regulatory compliance. Covers EU AI Act, GPAI Code of Practice, GDPR, NIST AI RMF, Colorado AI Act, HIPAA, and ISO 42001. Scans the codebase for compliance gaps, cross-references Arize instrumentation for audit trail coverage, and produces an actionable remediation checklist tailored to the selected frameworks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4482,4485,4488,4491,4494],{"name":4483,"slug":4484,"type":16},"Audit","audit",{"name":4486,"slug":4487,"type":16},"Compliance","compliance",{"name":4489,"slug":4490,"type":16},"GDPR","gdpr",{"name":4492,"slug":4493,"type":16},"Legal","legal",{"name":4495,"slug":4496,"type":16},"Security","security","2026-07-19T05:39:42.632738",{"items":4499,"total":1602},[4500,4506,4514,4520,4528,4541,4551],{"slug":4432,"name":4432,"fn":4433,"description":4434,"org":4501,"tags":4502,"stars":23,"repoUrl":24,"updatedAt":4446},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4503,4504,4505],{"name":4438,"slug":4439,"type":16},{"name":4441,"slug":4442,"type":16},{"name":4444,"slug":4445,"type":16},{"slug":940,"name":940,"fn":4448,"description":4449,"org":4507,"tags":4508,"stars":23,"repoUrl":24,"updatedAt":4465},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4509,4510,4511,4512,4513],{"name":4453,"slug":4454,"type":16},{"name":4456,"slug":4457,"type":16},{"name":4459,"slug":4460,"type":16},{"name":18,"slug":19,"type":16},{"name":4463,"slug":4464,"type":16},{"slug":4467,"name":4467,"fn":4468,"description":4469,"org":4515,"tags":4516,"stars":23,"repoUrl":24,"updatedAt":4475},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4517,4518,4519],{"name":4356,"slug":4357,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":4477,"name":4477,"fn":4478,"description":4479,"org":4521,"tags":4522,"stars":23,"repoUrl":24,"updatedAt":4497},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4523,4524,4525,4526,4527],{"name":4483,"slug":4484,"type":16},{"name":4486,"slug":4487,"type":16},{"name":4489,"slug":4490,"type":16},{"name":4492,"slug":4493,"type":16},{"name":4495,"slug":4496,"type":16},{"slug":4529,"name":4529,"fn":4530,"description":4531,"org":4532,"tags":4533,"stars":23,"repoUrl":24,"updatedAt":4540},"arize-dataset","manage Arize datasets and examples","Creates, manages, and queries Arize datasets and examples. Covers dataset CRUD, appending examples, exporting data, and file-based dataset creation using the ax CLI. Use when the user needs test data, evaluation examples, or mentions create dataset, list datasets, export dataset, append examples, dataset version, golden dataset, or test set.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4534,4537,4538,4539],{"name":4535,"slug":4536,"type":16},"Data Engineering","data-engineering",{"name":4359,"slug":35,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},"2026-07-22T05:37:20.943718",{"slug":4542,"name":4542,"fn":4543,"description":4544,"org":4545,"tags":4546,"stars":23,"repoUrl":24,"updatedAt":4550},"arize-evaluator","configure and run Arize evaluations","Handles LLM-as-judge and code evaluator workflows on Arize including creating\u002Fupdating evaluators, running evaluations on spans or experiments, managing tasks, trigger-run operations, column mapping, and continuous monitoring. Use when the user mentions create evaluator, LLM judge, code evaluator, hallucination, faithfulness, correctness, relevance, run eval, score spans, score experiment, trigger-run, column mapping, continuous monitoring, or improve evaluator prompt.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4547,4548,4549],{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"2026-07-25T05:32:37.552903",{"slug":4552,"name":4552,"fn":4553,"description":4554,"org":4555,"tags":4556,"stars":23,"repoUrl":24,"updatedAt":4560},"arize-experiment","run and analyze Arize experiments","Creates, runs, and analyzes Arize experiments for evaluating and comparing model performance. Covers experiment CRUD, exporting runs, comparing results, and evaluation workflows using the ax CLI. Use when the user mentions create experiment, run experiment, compare models, model performance, evaluate AI, experiment results, benchmark, A\u002FB test models, or measure accuracy.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4557,4558,4559],{"name":4403,"slug":4404,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},"2026-07-31T05:53:44.725539"]