[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-arize-evaluators":3,"mdc-z6h6a6-key":49,"related-repo-arize-evaluators":553,"related-org-arize-evaluators":635},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":23,"repoUrl":24,"updatedAt":25,"license":26,"forks":27,"topics":28,"repo":44,"sourceUrl":47,"mdContent":48},"evaluators","author and refine Phoenix evaluators","Author or refine a Phoenix evaluator — code or LLM-as-a-judge — that scores a run's output. Trigger when the user wants to create a new evaluator, improve an existing one's logic or rubric, choose labels, or decide what to measure on a dataset or experiment. Do NOT trigger on: (1) manual prompt drafting (use `playground`), (2) running or comparing experiments themselves (use `experiments`), (3) cross-trace failure diagnosis with no evaluator in scope (use `debug-trace`).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"arize","Arize AI","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Farize.jpg","Arize-ai",[13,17,20],{"name":14,"slug":15,"type":16},"LLM","llm","tag",{"name":18,"slug":19,"type":16},"Evals","evals",{"name":21,"slug":22,"type":16},"Testing","testing",10513,"https:\u002F\u002Fgithub.com\u002FArize-ai\u002Fphoenix","2026-07-31T05:58:09.13624",null,977,[29,30,31,32,33,34,19,35,36,37,38,39,40,41,42,43],"agents","ai-monitoring","ai-observability","aiengineering","anthropic","datasets","langchain","llamaindex","llm-eval","llm-evaluation","llmops","llms","openai","prompt-engineering","smolagents",{"repoUrl":24,"stars":23,"forks":27,"topics":45,"description":46},[29,30,31,32,33,34,19,35,36,37,38,39,40,41,42,43],"AI Observability & Evaluation","https:\u002F\u002Fgithub.com\u002FArize-ai\u002Fphoenix\u002Ftree\u002FHEAD\u002Fsrc\u002Fphoenix\u002Fserver\u002Fagents\u002Fprompts\u002Fskills\u002Fevaluators","---\nname: evaluators\ndescription: >\n  Author or refine a Phoenix evaluator — code or LLM-as-a-judge — that scores a run's output. Trigger when the user wants to create a new evaluator, improve an existing one's logic or rubric, choose labels, or decide what to measure on a dataset or experiment. Do NOT trigger on: (1) manual prompt drafting (use `playground`), (2) running or comparing experiments themselves (use `experiments`), (3) cross-trace failure diagnosis with no evaluator in scope (use `debug-trace`).\nsummary: Design or refine a code or LLM evaluator — labels, logic or rubric, the field it reads, and representative preview cases.\n---\n\n# Evaluators\n\nA Phoenix evaluator scores a run: it reads some subset of the run's `input`, `output`, `reference`,\nand `metadata` and returns named annotations — a label, a score, or both. The two artifact kinds —\na **code evaluator** (a Python or TypeScript function) and an **LLM evaluator** (a judge prompt\nsent to a model) — share the loop and discipline below; judgment structure follows task need and\nenvironment capability, not artifact kind. The `experiments` skill reads the scores you design\nhere; the explanations you enable are its evidence channel when comparing runs.\n\n## The Authoring Loop\n\n1. Derive the grading task from the stated purpose — a hypothesis and its evaluator are one design:\n   the hypothesis names the failure mode, the evaluator scores it. Evidence comes from the stated\n   purpose, the dataset's examples, and existing run outputs, not from interrogating the user.\n2. Inventory before creating. Read the dataset's existing evaluators and check input-shape\n   compatibility — an evaluator fits when its declared inputs parse the experiment's output\n   topology. Reuse or attach on a match; create on a gap. A question is warranted only when\n   the stated purpose lacks the failure mode, target output field, or acceptable tradeoff.\n3. Decide the labels. Choose a small, mutually exclusive, collectively exhaustive set — often\n   binary (`correct`\u002F`incorrect`, `pass`\u002F`fail`). Add labels only to distinguish failure modes that\n   matter; every extra label adds boundary ambiguity.\n4. Locate the signal in the run's fields — a top-level key, a chat-style `messages` array,\n   assistant content parts, `tool_calls`, or a `function_call` — by inspecting the actual shape.\n5. Write the judgment: a function that reads the field and returns the label or score, or a judge\n   prompt whose rubric names each label and ties the decision to observable evidence in the fields.\n   Align scores with the optimization direction and enable an explanation to justify the judgment\n   and surface rubric ambiguity.\n6. Calibrate against several representative cases covering the named failure modes — one preview is\n   not calibration. Shape each payload like a real run. When a case mislabels, change one thing —\n   rubric, logic, labels, or case — and fix an unrepresentative case before blaming the logic.\n7. Iterate until the representative cases label correctly and the tradeoff is acceptable.\n   Persistence is a separate save; do not claim the evaluator is created or updated until that save\n   actually completes.\n\n## Reference Provenance\n\nThe first fork is **reference-based** versus **reference-free**: does the judgment compare the\noutput against something stored with the example, or against the input and rubric criteria alone?\nWhen reference-based, triage what the `reference` actually is before trusting it as an answer key:\n\n- **golden** — a hand-labeled ideal output. Deviation is a defect; strict comparison is warranted.\n- **baseline-snapshot** — current-state outputs culled into the dataset as a comparison point, not\n  ground truth. It anchors did-behavior-change judgments — Pairwise suits it — and losing to it is\n  a signal, not a verdict.\n- **reference-free** — no stored answer key; judge the output against the input and a rubric.\n\n## Choosing The Judgment Structure\n\nDiscover the environment first: read the capability fingerprint of the sandbox the evaluator will\nrun in — which model credentials its environment variables name, whether it has internet access,\nwhich packages are installed, and any restricted-language runtime notes. Choose by environment capability and task need, not artifact\nkind. The ladder, floor to heaviest:\n\n- **deterministic primitives** — exact-match, contains, regex-match, json-distance,\n  embedding-distance, levenshtein-distance, scikit-learn metrics, and structural `tool_calls`\n  checks. Prefer one whenever the judgment can be computed.\n- **single LLM judge** — one rubric, one model, for reading-comprehension or open-ended quality.\n- **Composite** — sub-checks (code or LLM) blended into one weighted score; per-axis breakdown in\n  the explanation.\n- **LLM Jury** — one judgment polled across several models; weighted votes in the explanation.\n- **Pairwise** — a blind head-to-head of `output` against `reference`, returning a winner or tie.\n\nA credentialed sandbox with LLM access subsumes the LLM-evaluator artifact: a code evaluator there\ncan call models, so Composite, LLM Jury, and Pairwise become layered code evaluators recording\nsteps and votes in explanation or metadata. Design a suite, not a single check: a **deterministic\nfloor** of primitives plus **judged dimensions** for what the floor cannot settle.\n\n## Matching The Field Topology\n\n- At runtime, `output` is the new run's output; a dataset-backed evaluator receives the example's\n  output as `reference` — data the run never saw, provenance triaged above. Don't conflate the two.\n- A playground-backed run's `output` always has the fixed LLM-span shape: a `messages` array\n  (`role`, `content`, and `tool_calls` with `function.name`\u002F`function.arguments`) plus a top-level\n  `available_tools`; shape calibration cases accordingly. A dataset example's output keeps the\n  user-defined dataset shape — evidence for where the signal lives, not a guarantee; inspect it\n  rather than assuming a top-level key.\n- Declare only the fields the judgment needs; the simplest evaluator often reads just `output`.\n  Add `reference` for relational checks against the example's expected, golden, or subset data.\n- Prefer reading fields directly and parsing nested or stringified JSON in the logic over a custom\n  input mapping; in relational checks, normalize `output` and `reference` the same way before\n  comparing, or matching values will spuriously diverge.\n\n## Things To Avoid\n\n- Don't edit an existing evaluator without reading its current draft; form a specific hypothesis\n  and make the smallest edit that tests it.\n- Don't change the rubric, logic, labels, and cases all in one step; you lose attribution.\n- Don't reach for a judged structure when a deterministic primitive settles the judgment.\n- Don't claim the evaluator is created or updated; persistence is a separate save.\n",{"data":50,"body":52},{"name":4,"description":6,"summary":51},"Design or refine a code or LLM evaluator — labels, logic or rubric, the field it reads, and representative preview cases.",{"type":53,"children":54},"root",[55,63,124,131,225,231,257,290,296,301,375,394,400,524,530],{"type":56,"tag":57,"props":58,"children":59},"element","h1",{"id":4},[60],{"type":61,"value":62},"text","Evaluators",{"type":56,"tag":64,"props":65,"children":66},"p",{},[67,69,76,78,84,85,91,93,99,101,107,109,114,116,122],{"type":61,"value":68},"A Phoenix evaluator scores a run: it reads some subset of the run's ",{"type":56,"tag":70,"props":71,"children":73},"code",{"className":72},[],[74],{"type":61,"value":75},"input",{"type":61,"value":77},", ",{"type":56,"tag":70,"props":79,"children":81},{"className":80},[],[82],{"type":61,"value":83},"output",{"type":61,"value":77},{"type":56,"tag":70,"props":86,"children":88},{"className":87},[],[89],{"type":61,"value":90},"reference",{"type":61,"value":92},",\nand ",{"type":56,"tag":70,"props":94,"children":96},{"className":95},[],[97],{"type":61,"value":98},"metadata",{"type":61,"value":100}," and returns named annotations — a label, a score, or both. The two artifact kinds —\na ",{"type":56,"tag":102,"props":103,"children":104},"strong",{},[105],{"type":61,"value":106},"code evaluator",{"type":61,"value":108}," (a Python or TypeScript function) and an ",{"type":56,"tag":102,"props":110,"children":111},{},[112],{"type":61,"value":113},"LLM evaluator",{"type":61,"value":115}," (a judge prompt\nsent to a model) — share the loop and discipline below; judgment structure follows task need and\nenvironment capability, not artifact kind. The ",{"type":56,"tag":70,"props":117,"children":119},{"className":118},[],[120],{"type":61,"value":121},"experiments",{"type":61,"value":123}," skill reads the scores you design\nhere; the explanations you enable are its evidence channel when comparing runs.",{"type":56,"tag":125,"props":126,"children":128},"h2",{"id":127},"the-authoring-loop",[129],{"type":61,"value":130},"The Authoring Loop",{"type":56,"tag":132,"props":133,"children":134},"ol",{},[135,141,146,181,210,215,220],{"type":56,"tag":136,"props":137,"children":138},"li",{},[139],{"type":61,"value":140},"Derive the grading task from the stated purpose — a hypothesis and its evaluator are one design:\nthe hypothesis names the failure mode, the evaluator scores it. Evidence comes from the stated\npurpose, the dataset's examples, and existing run outputs, not from interrogating the user.",{"type":56,"tag":136,"props":142,"children":143},{},[144],{"type":61,"value":145},"Inventory before creating. Read the dataset's existing evaluators and check input-shape\ncompatibility — an evaluator fits when its declared inputs parse the experiment's output\ntopology. Reuse or attach on a match; create on a gap. A question is warranted only when\nthe stated purpose lacks the failure mode, target output field, or acceptable tradeoff.",{"type":56,"tag":136,"props":147,"children":148},{},[149,151,157,159,165,166,172,173,179],{"type":61,"value":150},"Decide the labels. Choose a small, mutually exclusive, collectively exhaustive set — often\nbinary (",{"type":56,"tag":70,"props":152,"children":154},{"className":153},[],[155],{"type":61,"value":156},"correct",{"type":61,"value":158},"\u002F",{"type":56,"tag":70,"props":160,"children":162},{"className":161},[],[163],{"type":61,"value":164},"incorrect",{"type":61,"value":77},{"type":56,"tag":70,"props":167,"children":169},{"className":168},[],[170],{"type":61,"value":171},"pass",{"type":61,"value":158},{"type":56,"tag":70,"props":174,"children":176},{"className":175},[],[177],{"type":61,"value":178},"fail",{"type":61,"value":180},"). Add labels only to distinguish failure modes that\nmatter; every extra label adds boundary ambiguity.",{"type":56,"tag":136,"props":182,"children":183},{},[184,186,192,194,200,202,208],{"type":61,"value":185},"Locate the signal in the run's fields — a top-level key, a chat-style ",{"type":56,"tag":70,"props":187,"children":189},{"className":188},[],[190],{"type":61,"value":191},"messages",{"type":61,"value":193}," array,\nassistant content parts, ",{"type":56,"tag":70,"props":195,"children":197},{"className":196},[],[198],{"type":61,"value":199},"tool_calls",{"type":61,"value":201},", or a ",{"type":56,"tag":70,"props":203,"children":205},{"className":204},[],[206],{"type":61,"value":207},"function_call",{"type":61,"value":209}," — by inspecting the actual shape.",{"type":56,"tag":136,"props":211,"children":212},{},[213],{"type":61,"value":214},"Write the judgment: a function that reads the field and returns the label or score, or a judge\nprompt whose rubric names each label and ties the decision to observable evidence in the fields.\nAlign scores with the optimization direction and enable an explanation to justify the judgment\nand surface rubric ambiguity.",{"type":56,"tag":136,"props":216,"children":217},{},[218],{"type":61,"value":219},"Calibrate against several representative cases covering the named failure modes — one preview is\nnot calibration. Shape each payload like a real run. When a case mislabels, change one thing —\nrubric, logic, labels, or case — and fix an unrepresentative case before blaming the logic.",{"type":56,"tag":136,"props":221,"children":222},{},[223],{"type":61,"value":224},"Iterate until the representative cases label correctly and the tradeoff is acceptable.\nPersistence is a separate save; do not claim the evaluator is created or updated until that save\nactually completes.",{"type":56,"tag":125,"props":226,"children":228},{"id":227},"reference-provenance",[229],{"type":61,"value":230},"Reference Provenance",{"type":56,"tag":64,"props":232,"children":233},{},[234,236,241,243,248,250,255],{"type":61,"value":235},"The first fork is ",{"type":56,"tag":102,"props":237,"children":238},{},[239],{"type":61,"value":240},"reference-based",{"type":61,"value":242}," versus ",{"type":56,"tag":102,"props":244,"children":245},{},[246],{"type":61,"value":247},"reference-free",{"type":61,"value":249},": does the judgment compare the\noutput against something stored with the example, or against the input and rubric criteria alone?\nWhen reference-based, triage what the ",{"type":56,"tag":70,"props":251,"children":253},{"className":252},[],[254],{"type":61,"value":90},{"type":61,"value":256}," actually is before trusting it as an answer key:",{"type":56,"tag":258,"props":259,"children":260},"ul",{},[261,271,281],{"type":56,"tag":136,"props":262,"children":263},{},[264,269],{"type":56,"tag":102,"props":265,"children":266},{},[267],{"type":61,"value":268},"golden",{"type":61,"value":270}," — a hand-labeled ideal output. Deviation is a defect; strict comparison is warranted.",{"type":56,"tag":136,"props":272,"children":273},{},[274,279],{"type":56,"tag":102,"props":275,"children":276},{},[277],{"type":61,"value":278},"baseline-snapshot",{"type":61,"value":280}," — current-state outputs culled into the dataset as a comparison point, not\nground truth. It anchors did-behavior-change judgments — Pairwise suits it — and losing to it is\na signal, not a verdict.",{"type":56,"tag":136,"props":282,"children":283},{},[284,288],{"type":56,"tag":102,"props":285,"children":286},{},[287],{"type":61,"value":247},{"type":61,"value":289}," — no stored answer key; judge the output against the input and a rubric.",{"type":56,"tag":125,"props":291,"children":293},{"id":292},"choosing-the-judgment-structure",[294],{"type":61,"value":295},"Choosing The Judgment Structure",{"type":56,"tag":64,"props":297,"children":298},{},[299],{"type":61,"value":300},"Discover the environment first: read the capability fingerprint of the sandbox the evaluator will\nrun in — which model credentials its environment variables name, whether it has internet access,\nwhich packages are installed, and any restricted-language runtime notes. Choose by environment capability and task need, not artifact\nkind. The ladder, floor to heaviest:",{"type":56,"tag":258,"props":302,"children":303},{},[304,321,331,341,351],{"type":56,"tag":136,"props":305,"children":306},{},[307,312,314,319],{"type":56,"tag":102,"props":308,"children":309},{},[310],{"type":61,"value":311},"deterministic primitives",{"type":61,"value":313}," — exact-match, contains, regex-match, json-distance,\nembedding-distance, levenshtein-distance, scikit-learn metrics, and structural ",{"type":56,"tag":70,"props":315,"children":317},{"className":316},[],[318],{"type":61,"value":199},{"type":61,"value":320},"\nchecks. Prefer one whenever the judgment can be computed.",{"type":56,"tag":136,"props":322,"children":323},{},[324,329],{"type":56,"tag":102,"props":325,"children":326},{},[327],{"type":61,"value":328},"single LLM judge",{"type":61,"value":330}," — one rubric, one model, for reading-comprehension or open-ended quality.",{"type":56,"tag":136,"props":332,"children":333},{},[334,339],{"type":56,"tag":102,"props":335,"children":336},{},[337],{"type":61,"value":338},"Composite",{"type":61,"value":340}," — sub-checks (code or LLM) blended into one weighted score; per-axis breakdown in\nthe explanation.",{"type":56,"tag":136,"props":342,"children":343},{},[344,349],{"type":56,"tag":102,"props":345,"children":346},{},[347],{"type":61,"value":348},"LLM Jury",{"type":61,"value":350}," — one judgment polled across several models; weighted votes in the explanation.",{"type":56,"tag":136,"props":352,"children":353},{},[354,359,361,366,368,373],{"type":56,"tag":102,"props":355,"children":356},{},[357],{"type":61,"value":358},"Pairwise",{"type":61,"value":360}," — a blind head-to-head of ",{"type":56,"tag":70,"props":362,"children":364},{"className":363},[],[365],{"type":61,"value":83},{"type":61,"value":367}," against ",{"type":56,"tag":70,"props":369,"children":371},{"className":370},[],[372],{"type":61,"value":90},{"type":61,"value":374},", returning a winner or tie.",{"type":56,"tag":64,"props":376,"children":377},{},[378,380,385,387,392],{"type":61,"value":379},"A credentialed sandbox with LLM access subsumes the LLM-evaluator artifact: a code evaluator there\ncan call models, so Composite, LLM Jury, and Pairwise become layered code evaluators recording\nsteps and votes in explanation or metadata. Design a suite, not a single check: a ",{"type":56,"tag":102,"props":381,"children":382},{},[383],{"type":61,"value":384},"deterministic\nfloor",{"type":61,"value":386}," of primitives plus ",{"type":56,"tag":102,"props":388,"children":389},{},[390],{"type":61,"value":391},"judged dimensions",{"type":61,"value":393}," for what the floor cannot settle.",{"type":56,"tag":125,"props":395,"children":397},{"id":396},"matching-the-field-topology",[398],{"type":61,"value":399},"Matching The Field Topology",{"type":56,"tag":258,"props":401,"children":402},{},[403,422,486,505],{"type":56,"tag":136,"props":404,"children":405},{},[406,408,413,415,420],{"type":61,"value":407},"At runtime, ",{"type":56,"tag":70,"props":409,"children":411},{"className":410},[],[412],{"type":61,"value":83},{"type":61,"value":414}," is the new run's output; a dataset-backed evaluator receives the example's\noutput as ",{"type":56,"tag":70,"props":416,"children":418},{"className":417},[],[419],{"type":61,"value":90},{"type":61,"value":421}," — data the run never saw, provenance triaged above. Don't conflate the two.",{"type":56,"tag":136,"props":423,"children":424},{},[425,427,432,434,439,441,447,448,454,456,461,463,469,470,476,478,484],{"type":61,"value":426},"A playground-backed run's ",{"type":56,"tag":70,"props":428,"children":430},{"className":429},[],[431],{"type":61,"value":83},{"type":61,"value":433}," always has the fixed LLM-span shape: a ",{"type":56,"tag":70,"props":435,"children":437},{"className":436},[],[438],{"type":61,"value":191},{"type":61,"value":440}," array\n(",{"type":56,"tag":70,"props":442,"children":444},{"className":443},[],[445],{"type":61,"value":446},"role",{"type":61,"value":77},{"type":56,"tag":70,"props":449,"children":451},{"className":450},[],[452],{"type":61,"value":453},"content",{"type":61,"value":455},", and ",{"type":56,"tag":70,"props":457,"children":459},{"className":458},[],[460],{"type":61,"value":199},{"type":61,"value":462}," with ",{"type":56,"tag":70,"props":464,"children":466},{"className":465},[],[467],{"type":61,"value":468},"function.name",{"type":61,"value":158},{"type":56,"tag":70,"props":471,"children":473},{"className":472},[],[474],{"type":61,"value":475},"function.arguments",{"type":61,"value":477},") plus a top-level\n",{"type":56,"tag":70,"props":479,"children":481},{"className":480},[],[482],{"type":61,"value":483},"available_tools",{"type":61,"value":485},"; shape calibration cases accordingly. A dataset example's output keeps the\nuser-defined dataset shape — evidence for where the signal lives, not a guarantee; inspect it\nrather than assuming a top-level key.",{"type":56,"tag":136,"props":487,"children":488},{},[489,491,496,498,503],{"type":61,"value":490},"Declare only the fields the judgment needs; the simplest evaluator often reads just ",{"type":56,"tag":70,"props":492,"children":494},{"className":493},[],[495],{"type":61,"value":83},{"type":61,"value":497},".\nAdd ",{"type":56,"tag":70,"props":499,"children":501},{"className":500},[],[502],{"type":61,"value":90},{"type":61,"value":504}," for relational checks against the example's expected, golden, or subset data.",{"type":56,"tag":136,"props":506,"children":507},{},[508,510,515,517,522],{"type":61,"value":509},"Prefer reading fields directly and parsing nested or stringified JSON in the logic over a custom\ninput mapping; in relational checks, normalize ",{"type":56,"tag":70,"props":511,"children":513},{"className":512},[],[514],{"type":61,"value":83},{"type":61,"value":516}," and ",{"type":56,"tag":70,"props":518,"children":520},{"className":519},[],[521],{"type":61,"value":90},{"type":61,"value":523}," the same way before\ncomparing, or matching values will spuriously diverge.",{"type":56,"tag":125,"props":525,"children":527},{"id":526},"things-to-avoid",[528],{"type":61,"value":529},"Things To Avoid",{"type":56,"tag":258,"props":531,"children":532},{},[533,538,543,548],{"type":56,"tag":136,"props":534,"children":535},{},[536],{"type":61,"value":537},"Don't edit an existing evaluator without reading its current draft; form a specific hypothesis\nand make the smallest edit that tests it.",{"type":56,"tag":136,"props":539,"children":540},{},[541],{"type":61,"value":542},"Don't change the rubric, logic, labels, and cases all in one step; you lose attribution.",{"type":56,"tag":136,"props":544,"children":545},{},[546],{"type":61,"value":547},"Don't reach for a judged structure when a deterministic primitive settles the judgment.",{"type":56,"tag":136,"props":549,"children":550},{},[551],{"type":61,"value":552},"Don't claim the evaluator is created or updated; persistence is a separate save.",{"items":554,"total":634},[555,570,582,594,600,610,624],{"slug":556,"name":556,"fn":557,"description":558,"org":559,"tags":560,"stars":23,"repoUrl":24,"updatedAt":569},"annotate-spans","annotate LLM spans and traces","Write effective, consistent annotations on LLM\u002Fagent spans and traces, and coach the user on annotation practice. Load this whenever you are about to record structured feedback with the `batch_span_annotate` tool, or when the user asks how to annotate, label, score, or review spans\u002Ftraces, build a failure taxonomy, or set up human\u002FLLM review. Do NOT load for: pure analysis with no intent to save feedback (use debug-trace), latency or cost statistics, or prompt authoring (use playground).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[561,562,563,566],{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":564,"slug":565,"type":16},"Observability","observability",{"name":567,"slug":568,"type":16},"Tracing","tracing","2026-07-12T08:08:14.140984",{"slug":34,"name":34,"fn":571,"description":572,"org":573,"tags":574,"stars":23,"repoUrl":24,"updatedAt":581},"reason about Phoenix dataset structure","Understand what a Phoenix dataset is and reason well about its examples, outputs, splits, and how it feeds evaluators and experiments. Load this whenever a dataset is in view or the user asks what a dataset is, how splits work, what an output \"means\", or how datasets relate to experiments and evals. This skill governs the judgment; any tool descriptions govern the mechanics.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[575,578,580],{"name":576,"slug":577,"type":16},"Data Analysis","data-analysis",{"name":579,"slug":34,"type":16},"Datasets",{"name":18,"slug":19,"type":16},"2026-07-12T08:08:21.695457",{"slug":583,"name":583,"fn":584,"description":585,"org":586,"tags":587,"stars":23,"repoUrl":24,"updatedAt":593},"debug-trace","diagnose failures using trace investigation","Diagnose failure modes by systematically investigating traces. Trigger when the user explicitly asks for cross-trace diagnosis: \"what's going wrong?\", \"were there errors?\", \"debug this\", \"where is my agent struggling?\". Do NOT trigger on: (1) advice questions (\"what should I do?\"), (2) statistical questions (\"what's the average latency?\"), (3) summarize requests, (4) trace filtering (\"show me traces with errors\"), (5) vague questions (\"is there a problem?\"), (6) unrelated requests.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[588,591,592],{"name":589,"slug":590,"type":16},"Debugging","debugging",{"name":564,"slug":565,"type":16},{"name":567,"slug":568,"type":16},"2026-07-12T08:08:10.44243",{"slug":4,"name":4,"fn":5,"description":6,"org":595,"tags":596,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[597,598,599],{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":21,"slug":22,"type":16},{"slug":121,"name":121,"fn":601,"description":602,"org":603,"tags":604,"stars":23,"repoUrl":24,"updatedAt":609},"run and compare dataset-backed experiments","Run, read, and compare dataset-backed experiments to find evidence that a prompt or pipeline is improving. Trigger when the user wants to iterate over a dataset with experiments, compare experiment runs, read experiment quality\u002Flatency\u002Fcost, or decide whether a change actually helped. Running a prompt over a dataset is implicitly an experiment — load this skill when dataset-backed work begins, before authoring evaluators for the experiment and before starting the recorded run, not only when reading results. Do NOT trigger on: (1) manual prompt drafting with no dataset-backed evaluation in scope (use `playground`), (2) authoring or refining an evaluator's logic or rubric (use `evaluators`), (3) cross-trace failure diagnosis with no experiment in scope (use `debug-trace`).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[605,606,607,608],{"name":579,"slug":34,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":21,"slug":22,"type":16},"2026-07-12T08:08:11.691477",{"slug":611,"name":611,"fn":612,"description":613,"org":614,"tags":615,"stars":23,"repoUrl":24,"updatedAt":623},"phoenix-graphql","query Phoenix API with GraphQL","Write efficient GraphQL queries against the Phoenix API. Load this skill in two cases: (1) before composing any non-trivial GraphQL query yourself for data analysis (via the `phoenix-gql` bash command) — it contains schema entrypoints and patterns that eliminate the need for introspection; (2) when the user asks for help writing GraphQL queries for their own scripts, tools, or integrations against Phoenix — it covers the endpoint, authentication, and client examples.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[616,619,620],{"name":617,"slug":618,"type":16},"Analytics","analytics",{"name":576,"slug":577,"type":16},{"name":621,"slug":622,"type":16},"GraphQL","graphql","2026-07-12T08:08:17.163493",{"slug":625,"name":625,"fn":626,"description":627,"org":628,"tags":629,"stars":23,"repoUrl":24,"updatedAt":633},"playground","author and iterate on prompts in Phoenix","Author, edit, or iterate on prompts in the Phoenix prompt playground, including running experiments over a dataset. Load before any playground tool call, including single-shot prompt rewrites.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[630,631,632],{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":21,"slug":22,"type":16},"2026-07-12T08:08:12.920792",8,{"items":636,"total":760},[637,644,650,656,662,669,675,681,692,710,728,738],{"slug":556,"name":556,"fn":557,"description":558,"org":638,"tags":639,"stars":23,"repoUrl":24,"updatedAt":569},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[640,641,642,643],{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":564,"slug":565,"type":16},{"name":567,"slug":568,"type":16},{"slug":34,"name":34,"fn":571,"description":572,"org":645,"tags":646,"stars":23,"repoUrl":24,"updatedAt":581},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[647,648,649],{"name":576,"slug":577,"type":16},{"name":579,"slug":34,"type":16},{"name":18,"slug":19,"type":16},{"slug":583,"name":583,"fn":584,"description":585,"org":651,"tags":652,"stars":23,"repoUrl":24,"updatedAt":593},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[653,654,655],{"name":589,"slug":590,"type":16},{"name":564,"slug":565,"type":16},{"name":567,"slug":568,"type":16},{"slug":4,"name":4,"fn":5,"description":6,"org":657,"tags":658,"stars":23,"repoUrl":24,"updatedAt":25},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[659,660,661],{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":21,"slug":22,"type":16},{"slug":121,"name":121,"fn":601,"description":602,"org":663,"tags":664,"stars":23,"repoUrl":24,"updatedAt":609},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[665,666,667,668],{"name":579,"slug":34,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":21,"slug":22,"type":16},{"slug":611,"name":611,"fn":612,"description":613,"org":670,"tags":671,"stars":23,"repoUrl":24,"updatedAt":623},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[672,673,674],{"name":617,"slug":618,"type":16},{"name":576,"slug":577,"type":16},{"name":621,"slug":622,"type":16},{"slug":625,"name":625,"fn":626,"description":627,"org":676,"tags":677,"stars":23,"repoUrl":24,"updatedAt":633},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[678,679,680],{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":21,"slug":22,"type":16},{"slug":682,"name":682,"fn":683,"description":684,"org":685,"tags":686,"stars":23,"repoUrl":24,"updatedAt":691},"span-coding","analyze and code Phoenix spans","Open-code Phoenix spans with PXI-owned notes, recover those notes for axial coding, and promote stable categories into structured annotations. Load this when analyzing spans to discover failure patterns before a taxonomy exists.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[687,688,689,690],{"name":589,"slug":590,"type":16},{"name":14,"slug":15,"type":16},{"name":564,"slug":565,"type":16},{"name":567,"slug":568,"type":16},"2026-07-12T08:08:19.597239",{"slug":693,"name":693,"fn":694,"description":695,"org":696,"tags":697,"stars":707,"repoUrl":708,"updatedAt":709},"arize-admin","manage Arize enterprise user access","Manages Arize users, organizations, spaces, projects, roles, role bindings, resource restrictions, and API keys via the ax CLI. Use for enterprise admin workflows: inviting and offboarding users, onboarding new teams, creating custom roles for SAML\u002FSSO mappings, assigning roles to users, restricting project-level access, and managing service keys for multi-tenant architectures. Covers ax users, ax organizations, ax spaces, ax projects, ax roles, ax role-bindings, and ax api-keys.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[698,701,704],{"name":699,"slug":700,"type":16},"CLI","cli",{"name":702,"slug":703,"type":16},"Operations","operations",{"name":705,"slug":706,"type":16},"Permissions","permissions",38,"https:\u002F\u002Fgithub.com\u002FArize-ai\u002Farize-skills","2026-07-22T05:37:21.991338",{"slug":711,"name":711,"fn":712,"description":713,"org":714,"tags":715,"stars":707,"repoUrl":708,"updatedAt":727},"arize-ai-provider-integration","manage Arize AI provider integrations","Creates, reads, updates, and deletes Arize AI integrations that store LLM provider credentials used by evaluators and other Arize features. Supports any LLM provider (e.g. OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Gemini, NVIDIA NIM). Use when the user mentions AI integration, LLM provider credentials, create integration, list integrations, update credentials, delete integration, or connecting an LLM provider to Arize.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[716,718,721,724,725],{"name":717,"slug":33,"type":16},"Anthropic",{"name":719,"slug":720,"type":16},"Azure","azure",{"name":722,"slug":723,"type":16},"Integrations","integrations",{"name":14,"slug":15,"type":16},{"name":726,"slug":41,"type":16},"OpenAI","2026-07-22T05:37:23.90468",{"slug":729,"name":729,"fn":730,"description":731,"org":732,"tags":733,"stars":707,"repoUrl":708,"updatedAt":737},"arize-annotation","manage Arize annotation workflows","Creates and manages annotation configs (categorical, continuous, freeform label schemas) and annotation queues (human review workflows) on Arize. Applies human annotations to project spans via the Python SDK. Use when the user mentions annotation config, annotation queue, label schema, human feedback, bulk annotate spans, update_annotations, labeling queue, annotate record, or human review.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[734,735,736],{"name":576,"slug":577,"type":16},{"name":14,"slug":15,"type":16},{"name":564,"slug":565,"type":16},"2026-07-22T05:37:19.010776",{"slug":739,"name":739,"fn":740,"description":741,"org":742,"tags":743,"stars":707,"repoUrl":708,"updatedAt":759},"arize-compliance-audit","audit AI agents for regulatory compliance","INVOKE THIS SKILL when auditing an AI agent or LLM app for regulatory compliance. Covers EU AI Act, GPAI Code of Practice, GDPR, NIST AI RMF, Colorado AI Act, HIPAA, and ISO 42001. Scans the codebase for compliance gaps, cross-references Arize instrumentation for audit trail coverage, and produces an actionable remediation checklist tailored to the selected frameworks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[744,747,750,753,756],{"name":745,"slug":746,"type":16},"Audit","audit",{"name":748,"slug":749,"type":16},"Compliance","compliance",{"name":751,"slug":752,"type":16},"GDPR","gdpr",{"name":754,"slug":755,"type":16},"Legal","legal",{"name":757,"slug":758,"type":16},"Security","security","2026-07-19T05:39:42.632738",23]