[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-arize-annotate-spans":3,"mdc--v1cvd1-key":52,"related-repo-arize-annotate-spans":1474,"related-org-arize-annotate-spans":1551},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":47,"sourceUrl":50,"mdContent":51},"annotate-spans","annotate LLM spans and traces","Write effective, consistent annotations on LLM\u002Fagent spans and traces, and coach the user on annotation practice. Load this whenever you are about to record structured feedback with the `batch_span_annotate` tool, or when the user asks how to annotate, label, score, or review spans\u002Ftraces, build a failure taxonomy, or set up human\u002FLLM review. Do NOT load for: pure analysis with no intent to save feedback (use debug-trace), latency or cost statistics, or prompt authoring (use playground).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"arize","Arize AI","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Farize.jpg","Arize-ai",[13,17,20,23],{"name":14,"slug":15,"type":16},"Tracing","tracing","tag",{"name":18,"slug":19,"type":16},"Observability","observability",{"name":21,"slug":22,"type":16},"LLM","llm",{"name":24,"slug":25,"type":16},"Evals","evals",10513,"https:\u002F\u002Fgithub.com\u002FArize-ai\u002Fphoenix","2026-07-12T08:08:14.140984",null,977,[32,33,34,35,36,37,25,38,39,40,41,42,43,44,45,46],"agents","ai-monitoring","ai-observability","aiengineering","anthropic","datasets","langchain","llamaindex","llm-eval","llm-evaluation","llmops","llms","openai","prompt-engineering","smolagents",{"repoUrl":27,"stars":26,"forks":30,"topics":48,"description":49},[32,33,34,35,36,37,25,38,39,40,41,42,43,44,45,46],"AI Observability & Evaluation","https:\u002F\u002Fgithub.com\u002FArize-ai\u002Fphoenix\u002Ftree\u002FHEAD\u002Fsrc\u002Fphoenix\u002Fserver\u002Fagents\u002Fprompts\u002Fskills\u002Fannotate-spans","---\nname: annotate-spans\ndescription: >\n  Write effective, consistent annotations on LLM\u002Fagent spans and traces, and coach the user on annotation practice. Load this whenever you are about to record structured feedback with the `batch_span_annotate` tool, or when the user asks how to annotate, label, score, or review spans\u002Ftraces, build a failure taxonomy, or set up human\u002FLLM review. Do NOT load for: pure analysis with no intent to save feedback (use debug-trace), latency or cost statistics, or prompt authoring (use playground).\nsummary: Create consistent span or trace annotations and help design useful feedback taxonomies.\n---\n\n# Annotating Spans and Traces\n\nAn annotation is durable, structured feedback attached to a span or trace: a `name` (the dimension being judged), an optional `label` and\u002For `score` (the outcome), and an `explanation` (why). Annotations are not throwaway commentary — they accumulate into a dataset the user filters, aggregates, and iterates against.\n\nA good annotation earns its place by being useful *later*:\n\n- **Filterable** — `annotations['answer_relevance'].label == 'fail'` returns the spans you meant.\n- **Aggregatable** — counting labels across spans yields a failure rate that tells the user where to focus.\n- **Auditable** — months later, the explanation still justifies the judgment without rerunning anything.\n- **Curatable** — failing spans can be pulled into a dataset to drive evals or fixes.\n\nThis skill governs the *judgment* behind annotations. The `batch_span_annotate` tool description governs the *mechanics* (one array, ID requirements, update keying); follow both, and never contradict the tool's naming and identifier rules.\n\n## What Makes an Annotation Useful\n\n1. **Grounded in observed behavior, not generic quality vibes.** Annotate what actually went wrong or right in *this* span. \"Cited a refund policy that does not exist in the retrieved context\" beats a free-floating `hallucination_score: 0.3`. Generic dimensions like `helpfulness` or `coherence` are rarely grounded in the application's real failure modes — prefer names that point at a concrete behavior.\n\n2. **One dimension per annotation.** `name` is the rubric dimension; the outcome lives in `label`\u002F`score`. Use `name: \"tool_selection\"`, `label: \"incorrect\"` — not `name: \"wrong_tool\"`. If you find yourself judging two things at once (e.g., retrieval relevance *and* answer faithfulness), write two annotations.\n\n3. **Target the most specific responsible span.** Annotate the LLM span for model output, the tool span for tool behavior, the retriever span for retrieval quality. Reserve root agent\u002Fchain spans for genuinely end-to-end judgments (task success, trajectory). A faithfulness failure pinned to the right LLM span is actionable; the same label on the root span forces the user to hunt.\n\n4. **Judge the first failure, not every downstream symptom.** Errors cascade — bad retrieval produces a bad answer. Annotate the root cause where it occurred. Add a separate annotation downstream only when it reveals an independent problem, not a consequence of the first.\n\n5. **Prefer crisp labels over fuzzy scores.** A binary or small categorical label (`pass`\u002F`fail`, `relevant`\u002F`irrelevant`, `correct`\u002F`partial`\u002F`incorrect`) is easy to apply consistently and easy to aggregate. Use a numeric `score` only when the scale is genuinely meaningful and defined; put the rubric, scale, or threshold in `metadata` so the number is interpretable later.\n\n6. **Explanations are specific observations, not restatements.** Write what you saw, citing the evidence. Good: \"Returned chunks about onboarding; user asked about cancellation — no relevant chunk retrieved.\" Weak: \"The retrieval was bad.\" Always include an explanation for any score, any failure, any unclear label, or any judgment the user might want to revisit.\n\n7. **Be consistent across spans.** The same dimension must use the same `name` and the same label vocabulary everywhere, or filtering and rate computation break. The project's annotation configs *are* that shared vocabulary — annotate into a config's `name` and labels rather than deciding fresh each run (see [Work From the Project's Annotation Configs](#work-from-the-projects-annotation-configs)). Keep names stable across runs (no `_v2`\u002F`_new` suffixes).\n\n8. **Set `annotatorKind` honestly.** `LLM` for your own judgment, `HUMAN` only when recording feedback the user explicitly gave, `CODE` for deterministic checks. Don't record your own opinion as `HUMAN`.\n\n## Work From the Project's Annotation Configs\n\nAn **annotation config** is the project's codified rubric for one dimension: a `name`, a type (categorical \u002F continuous \u002F freeform), and the allowed outcomes — a categorical config's `values` (each a `label` with an optional `score`), or a continuous config's `lowerBound`\u002F`upperBound`. Configs are the source of truth for annotation vocabulary: they drive the annotation UI, keep names and labels consistent across runs, and let a later visit reuse the grading criteria instead of reinventing it. Annotate *into* configs rather than inventing a name and label set each time.\n\nBefore writing any annotation:\n\n1. **Pull the project's configs first.** Query `Project.annotationConfigs` and read the existing names and their label\u002Fscore schemes. This is the established rubric — prefer it over anything you would invent. (Resolve the project node id as described in the `phoenix-graphql` skill.)\n\n   ```graphql\n   query ProjectAnnotationConfigs($projectId: ID!) {\n     node(id: $projectId) {\n       ... on Project {\n         annotationConfigs(first: 100) {\n           edges { node {\n             __typename\n             ... on AnnotationConfigBase { name description annotationType }\n             ... on CategoricalAnnotationConfig { id optimizationDirection values { label score } }\n             ... on ContinuousAnnotationConfig { id optimizationDirection lowerBound upperBound }\n             ... on FreeformAnnotationConfig { id }\n           } }\n         }\n       }\n     }\n   }\n   ```\n2. **Reuse an existing config when one fits.** Annotate with the config's exact `name` and a `label` from its `values` (or a `score` within its bounds). This is what keeps `annotations['tool_selection'].label == 'incorrect'` filterable and aggregatable across runs. A config that already defines the scale also answers questions you would otherwise stop to ask the user (e.g. \"what numeric range?\") — don't `ask_user` for something a config already specifies.\n3. **When a new category emerges that no config covers, codify it before (or as) you annotate** — don't add an ad-hoc annotation and move on. Use the annotation-config tools rather than raw GraphQL mutations; the tools provide the approval surface and the frontend owns the write path.\n   - **No close config** → call `create_annotation_config` with the new rubric and the current project's `projectId`, then annotate against that config's `name` after the proposal is accepted.\n   - **A config is close but missing a label** → call `update_annotation_config` to replace it with the complete desired scheme, then annotate. Update is a full replace: pass the existing `values` plus the new one, keep the same `name`, and do not omit values you want to keep.\n\n   Codifying first means the next visit to this project rediscovers the criteria instead of growing a second, differently-named rubric for the same thing. Use `type: \"categorical\" | \"continuous\" | \"freeform\"` and set `optimizationDirection` to `MINIMIZE`, `MAXIMIZE`, or `NONE` when it matters.\n4. **Surface the choice explicitly.** Tell the user when you reused a config versus proposed a new one or extended an existing one, and why. Naming or changing a rubric is a decision they may want to weigh in on.\n\n**Do not cross-contaminate projects.** Config names are global, and a same-named config may be bound to other projects with different semantics. Prefer configs already associated with *this* project; only create or associate configs for the project in context. Never repurpose another project's rubric just because its name looks right.\n\n## Mode A: Coaching the User\n\nWhen the user asks *how* to annotate (rather than asking you to do it), teach the process rather than handing over a fixed rubric:\n\n1. **Start from failures they have actually seen.** Resist proposing a polished taxonomy up front — a pre-baked list causes confirmation bias. Ask what's going wrong, or use the `debug-trace` skill to surface real failure modes first.\n2. **Open-code before naming.** Encourage free-form notes on a handful of traces (\"what's the first thing wrong here?\") before committing to category names.\n3. **Axial-code into a small vocabulary.** Group similar notes into 5–10 named, mutually distinct, actionable categories. Each should be specific enough that two reviewers would label the same span the same way.\n4. **Define the label set per category.** Usually binary. Write a one-sentence definition so the boundary is unambiguous. Once the vocabulary stabilizes, codify each category as an annotation config (see [Work From the Project's Annotation Configs](#work-from-the-projects-annotation-configs)) so the rubric persists and the next session reuses it instead of re-deriving it.\n5. **Apply, then aggregate.** Label a representative sample consistently, then compute failure rates to prioritize. Highest-frequency × highest-impact wins.\n6. **Fix-first.** Remind the user that many failures (missing prompt instruction, missing tool, retrieval bug) are better *fixed* than measured. Reserve standing annotations\u002Fevals for failures they will iterate on repeatedly or that need a guardrail.\n\nExplain *why* a convention matters (\"stable names let you trend this over time\") rather than only stating it.\n\n## Mode B: Writing Annotations Yourself\n\nWhen the user asks you to save annotations:\n\n1. **Confirm scope and intent.** Only annotate when the user wants feedback persisted, not during ordinary analysis. Know which spans and which dimension(s) are in scope. If no failure modes are established yet, diagnose first with the `debug-trace` skill, then return here to persist the results — don't re-derive a taxonomy.\n2. **Inspect before judging.** Read the actual span input\u002Foutput (and relevant parent\u002Fchild spans for context). Never annotate from span status codes alone — a success status can mask an error in attributes, and an exception can be expected behavior.\n3. **Pick the dimension(s) from the project's annotation configs.** Pull the configs first and reuse a matching config's `name` and label\u002Fscore scheme; if a needed category has no config, create or extend one (see [Work From the Project's Annotation Configs](#work-from-the-projects-annotation-configs)) before writing. Then judge every span in the batch on that fixed vocabulary. If you're judging more than one dimension, settle each one's config up front.\n4. **Annotate the right span for each judgment** (principle 3) and the first failure (principle 4). Use only real IDs from context or prior tool results — never guess span IDs.\n5. **Write a grounded explanation per annotation** citing the specific evidence in that span.\n6. **Batch the related annotations into one `batch_span_annotate` call** and pick the `identifier` to match your intent (annotations are keyed by `(name, span, identifier)`):\n   - Use a **stable** identifier that names you as the author — e.g. `pxi` — when the judgment should be *updatable*: re-reviewing the same span and dimension overwrites the prior annotation instead of duplicating it.\n   - Use a **descriptive, run-scoped** identifier — e.g. `pxi:tool-misuse-2026-05-29` — for a discrete review batch you may want to query or revert as a unit later. This mirrors the Phoenix CLI's `coding-run:\u003Ctopic>-\u003Cdate>` convention: a descriptive id carries meaning for whoever opens the data later, far better than an opaque constant.\n   - Either way, the author prefix keeps your annotations distinguishable from human or other-evaluator annotations on the same span. Do not reuse one identifier for two unrelated runs you'd want to revert separately.\n7. **Report back** what you annotated, the dimension(s) and vocabulary used, and the distribution of labels — and link to a representative annotated span so the user can verify. Build links from the OpenTelemetry hex IDs in the `spanId`\u002F`traceId` GraphQL fields (never the Relay `id` field): `[short label](\u002Fredirects\u002Fspans\u002F\u003CspanId>)` for a span, or `[short label](\u002Fredirects\u002Ftraces\u002F\u003CtraceId>)` when no single span captures the judgment. A span link lands the user on the exact annotated node.\n\n## Anti-Patterns\n\n- **Generic score columns** (`hallucination_score`, `quality_score`) not tied to a concrete observed behavior.\n- **Outcome baked into the name** (`name: \"passed_relevance\"`) instead of `name: \"relevance\"`, `label: \"pass\"`.\n- **Inconsistent vocabulary** across spans (`fail` here, `bad` there, `0` elsewhere) — kills filtering and aggregation.\n- **Annotating every downstream symptom** of a single root failure.\n- **Labeling without an explanation** when the judgment is anything but trivially obvious.\n- **Annotating from status codes or metadata** without reading the actual span content.\n- **Recording your own judgment as `HUMAN`**, or guessing span IDs.\n- **Inventing a taxonomy before reading traces**, when coaching the user.\n",{"data":53,"body":55},{"name":4,"description":6,"summary":54},"Create consistent span or trace annotations and help design useful feedback taxonomies.",{"type":56,"children":57},"root",[58,67,106,119,173,200,207,521,526,589,594,957,973,979,991,1075,1087,1093,1098,1315,1321,1468],{"type":59,"tag":60,"props":61,"children":63},"element","h1",{"id":62},"annotating-spans-and-traces",[64],{"type":65,"value":66},"text","Annotating Spans and Traces",{"type":59,"tag":68,"props":69,"children":70},"p",{},[71,73,80,82,88,90,96,98,104],{"type":65,"value":72},"An annotation is durable, structured feedback attached to a span or trace: a ",{"type":59,"tag":74,"props":75,"children":77},"code",{"className":76},[],[78],{"type":65,"value":79},"name",{"type":65,"value":81}," (the dimension being judged), an optional ",{"type":59,"tag":74,"props":83,"children":85},{"className":84},[],[86],{"type":65,"value":87},"label",{"type":65,"value":89}," and\u002For ",{"type":59,"tag":74,"props":91,"children":93},{"className":92},[],[94],{"type":65,"value":95},"score",{"type":65,"value":97}," (the outcome), and an ",{"type":59,"tag":74,"props":99,"children":101},{"className":100},[],[102],{"type":65,"value":103},"explanation",{"type":65,"value":105}," (why). Annotations are not throwaway commentary — they accumulate into a dataset the user filters, aggregates, and iterates against.",{"type":59,"tag":68,"props":107,"children":108},{},[109,111,117],{"type":65,"value":110},"A good annotation earns its place by being useful ",{"type":59,"tag":112,"props":113,"children":114},"em",{},[115],{"type":65,"value":116},"later",{"type":65,"value":118},":",{"type":59,"tag":120,"props":121,"children":122},"ul",{},[123,143,153,163],{"type":59,"tag":124,"props":125,"children":126},"li",{},[127,133,135,141],{"type":59,"tag":128,"props":129,"children":130},"strong",{},[131],{"type":65,"value":132},"Filterable",{"type":65,"value":134}," — ",{"type":59,"tag":74,"props":136,"children":138},{"className":137},[],[139],{"type":65,"value":140},"annotations['answer_relevance'].label == 'fail'",{"type":65,"value":142}," returns the spans you meant.",{"type":59,"tag":124,"props":144,"children":145},{},[146,151],{"type":59,"tag":128,"props":147,"children":148},{},[149],{"type":65,"value":150},"Aggregatable",{"type":65,"value":152}," — counting labels across spans yields a failure rate that tells the user where to focus.",{"type":59,"tag":124,"props":154,"children":155},{},[156,161],{"type":59,"tag":128,"props":157,"children":158},{},[159],{"type":65,"value":160},"Auditable",{"type":65,"value":162}," — months later, the explanation still justifies the judgment without rerunning anything.",{"type":59,"tag":124,"props":164,"children":165},{},[166,171],{"type":59,"tag":128,"props":167,"children":168},{},[169],{"type":65,"value":170},"Curatable",{"type":65,"value":172}," — failing spans can be pulled into a dataset to drive evals or fixes.",{"type":59,"tag":68,"props":174,"children":175},{},[176,178,183,185,191,193,198],{"type":65,"value":177},"This skill governs the ",{"type":59,"tag":112,"props":179,"children":180},{},[181],{"type":65,"value":182},"judgment",{"type":65,"value":184}," behind annotations. The ",{"type":59,"tag":74,"props":186,"children":188},{"className":187},[],[189],{"type":65,"value":190},"batch_span_annotate",{"type":65,"value":192}," tool description governs the ",{"type":59,"tag":112,"props":194,"children":195},{},[196],{"type":65,"value":197},"mechanics",{"type":65,"value":199}," (one array, ID requirements, update keying); follow both, and never contradict the tool's naming and identifier rules.",{"type":59,"tag":201,"props":202,"children":204},"h2",{"id":203},"what-makes-an-annotation-useful",[205],{"type":65,"value":206},"What Makes an Annotation Useful",{"type":59,"tag":208,"props":209,"children":210},"ol",{},[211,252,314,324,334,409,419,474],{"type":59,"tag":124,"props":212,"children":213},{},[214,219,221,226,228,234,236,242,244,250],{"type":59,"tag":128,"props":215,"children":216},{},[217],{"type":65,"value":218},"Grounded in observed behavior, not generic quality vibes.",{"type":65,"value":220}," Annotate what actually went wrong or right in ",{"type":59,"tag":112,"props":222,"children":223},{},[224],{"type":65,"value":225},"this",{"type":65,"value":227}," span. \"Cited a refund policy that does not exist in the retrieved context\" beats a free-floating ",{"type":59,"tag":74,"props":229,"children":231},{"className":230},[],[232],{"type":65,"value":233},"hallucination_score: 0.3",{"type":65,"value":235},". Generic dimensions like ",{"type":59,"tag":74,"props":237,"children":239},{"className":238},[],[240],{"type":65,"value":241},"helpfulness",{"type":65,"value":243}," or ",{"type":59,"tag":74,"props":245,"children":247},{"className":246},[],[248],{"type":65,"value":249},"coherence",{"type":65,"value":251}," are rarely grounded in the application's real failure modes — prefer names that point at a concrete behavior.",{"type":59,"tag":124,"props":253,"children":254},{},[255,260,262,267,269,274,276,281,283,289,291,297,299,305,307,312],{"type":59,"tag":128,"props":256,"children":257},{},[258],{"type":65,"value":259},"One dimension per annotation.",{"type":65,"value":261}," ",{"type":59,"tag":74,"props":263,"children":265},{"className":264},[],[266],{"type":65,"value":79},{"type":65,"value":268}," is the rubric dimension; the outcome lives in ",{"type":59,"tag":74,"props":270,"children":272},{"className":271},[],[273],{"type":65,"value":87},{"type":65,"value":275},"\u002F",{"type":59,"tag":74,"props":277,"children":279},{"className":278},[],[280],{"type":65,"value":95},{"type":65,"value":282},". Use ",{"type":59,"tag":74,"props":284,"children":286},{"className":285},[],[287],{"type":65,"value":288},"name: \"tool_selection\"",{"type":65,"value":290},", ",{"type":59,"tag":74,"props":292,"children":294},{"className":293},[],[295],{"type":65,"value":296},"label: \"incorrect\"",{"type":65,"value":298}," — not ",{"type":59,"tag":74,"props":300,"children":302},{"className":301},[],[303],{"type":65,"value":304},"name: \"wrong_tool\"",{"type":65,"value":306},". If you find yourself judging two things at once (e.g., retrieval relevance ",{"type":59,"tag":112,"props":308,"children":309},{},[310],{"type":65,"value":311},"and",{"type":65,"value":313}," answer faithfulness), write two annotations.",{"type":59,"tag":124,"props":315,"children":316},{},[317,322],{"type":59,"tag":128,"props":318,"children":319},{},[320],{"type":65,"value":321},"Target the most specific responsible span.",{"type":65,"value":323}," Annotate the LLM span for model output, the tool span for tool behavior, the retriever span for retrieval quality. Reserve root agent\u002Fchain spans for genuinely end-to-end judgments (task success, trajectory). A faithfulness failure pinned to the right LLM span is actionable; the same label on the root span forces the user to hunt.",{"type":59,"tag":124,"props":325,"children":326},{},[327,332],{"type":59,"tag":128,"props":328,"children":329},{},[330],{"type":65,"value":331},"Judge the first failure, not every downstream symptom.",{"type":65,"value":333}," Errors cascade — bad retrieval produces a bad answer. Annotate the root cause where it occurred. Add a separate annotation downstream only when it reveals an independent problem, not a consequence of the first.",{"type":59,"tag":124,"props":335,"children":336},{},[337,342,344,350,351,357,358,364,365,371,372,378,379,385,386,392,394,399,401,407],{"type":59,"tag":128,"props":338,"children":339},{},[340],{"type":65,"value":341},"Prefer crisp labels over fuzzy scores.",{"type":65,"value":343}," A binary or small categorical label (",{"type":59,"tag":74,"props":345,"children":347},{"className":346},[],[348],{"type":65,"value":349},"pass",{"type":65,"value":275},{"type":59,"tag":74,"props":352,"children":354},{"className":353},[],[355],{"type":65,"value":356},"fail",{"type":65,"value":290},{"type":59,"tag":74,"props":359,"children":361},{"className":360},[],[362],{"type":65,"value":363},"relevant",{"type":65,"value":275},{"type":59,"tag":74,"props":366,"children":368},{"className":367},[],[369],{"type":65,"value":370},"irrelevant",{"type":65,"value":290},{"type":59,"tag":74,"props":373,"children":375},{"className":374},[],[376],{"type":65,"value":377},"correct",{"type":65,"value":275},{"type":59,"tag":74,"props":380,"children":382},{"className":381},[],[383],{"type":65,"value":384},"partial",{"type":65,"value":275},{"type":59,"tag":74,"props":387,"children":389},{"className":388},[],[390],{"type":65,"value":391},"incorrect",{"type":65,"value":393},") is easy to apply consistently and easy to aggregate. Use a numeric ",{"type":59,"tag":74,"props":395,"children":397},{"className":396},[],[398],{"type":65,"value":95},{"type":65,"value":400}," only when the scale is genuinely meaningful and defined; put the rubric, scale, or threshold in ",{"type":59,"tag":74,"props":402,"children":404},{"className":403},[],[405],{"type":65,"value":406},"metadata",{"type":65,"value":408}," so the number is interpretable later.",{"type":59,"tag":124,"props":410,"children":411},{},[412,417],{"type":59,"tag":128,"props":413,"children":414},{},[415],{"type":65,"value":416},"Explanations are specific observations, not restatements.",{"type":65,"value":418}," Write what you saw, citing the evidence. Good: \"Returned chunks about onboarding; user asked about cancellation — no relevant chunk retrieved.\" Weak: \"The retrieval was bad.\" Always include an explanation for any score, any failure, any unclear label, or any judgment the user might want to revisit.",{"type":59,"tag":124,"props":420,"children":421},{},[422,427,429,434,436,441,443,448,450,457,459,465,466,472],{"type":59,"tag":128,"props":423,"children":424},{},[425],{"type":65,"value":426},"Be consistent across spans.",{"type":65,"value":428}," The same dimension must use the same ",{"type":59,"tag":74,"props":430,"children":432},{"className":431},[],[433],{"type":65,"value":79},{"type":65,"value":435}," and the same label vocabulary everywhere, or filtering and rate computation break. The project's annotation configs ",{"type":59,"tag":112,"props":437,"children":438},{},[439],{"type":65,"value":440},"are",{"type":65,"value":442}," that shared vocabulary — annotate into a config's ",{"type":59,"tag":74,"props":444,"children":446},{"className":445},[],[447],{"type":65,"value":79},{"type":65,"value":449}," and labels rather than deciding fresh each run (see ",{"type":59,"tag":451,"props":452,"children":454},"a",{"href":453},"#work-from-the-projects-annotation-configs",[455],{"type":65,"value":456},"Work From the Project's Annotation Configs",{"type":65,"value":458},"). Keep names stable across runs (no ",{"type":59,"tag":74,"props":460,"children":462},{"className":461},[],[463],{"type":65,"value":464},"_v2",{"type":65,"value":275},{"type":59,"tag":74,"props":467,"children":469},{"className":468},[],[470],{"type":65,"value":471},"_new",{"type":65,"value":473}," suffixes).",{"type":59,"tag":124,"props":475,"children":476},{},[477,490,491,496,498,504,506,512,514,519],{"type":59,"tag":128,"props":478,"children":479},{},[480,482,488],{"type":65,"value":481},"Set ",{"type":59,"tag":74,"props":483,"children":485},{"className":484},[],[486],{"type":65,"value":487},"annotatorKind",{"type":65,"value":489}," honestly.",{"type":65,"value":261},{"type":59,"tag":74,"props":492,"children":494},{"className":493},[],[495],{"type":65,"value":21},{"type":65,"value":497}," for your own judgment, ",{"type":59,"tag":74,"props":499,"children":501},{"className":500},[],[502],{"type":65,"value":503},"HUMAN",{"type":65,"value":505}," only when recording feedback the user explicitly gave, ",{"type":59,"tag":74,"props":507,"children":509},{"className":508},[],[510],{"type":65,"value":511},"CODE",{"type":65,"value":513}," for deterministic checks. Don't record your own opinion as ",{"type":59,"tag":74,"props":515,"children":517},{"className":516},[],[518],{"type":65,"value":503},{"type":65,"value":520},".",{"type":59,"tag":201,"props":522,"children":524},{"id":523},"work-from-the-projects-annotation-configs",[525],{"type":65,"value":456},{"type":59,"tag":68,"props":527,"children":528},{},[529,531,536,538,543,545,551,553,558,560,565,567,573,574,580,582,587],{"type":65,"value":530},"An ",{"type":59,"tag":128,"props":532,"children":533},{},[534],{"type":65,"value":535},"annotation config",{"type":65,"value":537}," is the project's codified rubric for one dimension: a ",{"type":59,"tag":74,"props":539,"children":541},{"className":540},[],[542],{"type":65,"value":79},{"type":65,"value":544},", a type (categorical \u002F continuous \u002F freeform), and the allowed outcomes — a categorical config's ",{"type":59,"tag":74,"props":546,"children":548},{"className":547},[],[549],{"type":65,"value":550},"values",{"type":65,"value":552}," (each a ",{"type":59,"tag":74,"props":554,"children":556},{"className":555},[],[557],{"type":65,"value":87},{"type":65,"value":559}," with an optional ",{"type":59,"tag":74,"props":561,"children":563},{"className":562},[],[564],{"type":65,"value":95},{"type":65,"value":566},"), or a continuous config's ",{"type":59,"tag":74,"props":568,"children":570},{"className":569},[],[571],{"type":65,"value":572},"lowerBound",{"type":65,"value":275},{"type":59,"tag":74,"props":575,"children":577},{"className":576},[],[578],{"type":65,"value":579},"upperBound",{"type":65,"value":581},". Configs are the source of truth for annotation vocabulary: they drive the annotation UI, keep names and labels consistent across runs, and let a later visit reuse the grading criteria instead of reinventing it. Annotate ",{"type":59,"tag":112,"props":583,"children":584},{},[585],{"type":65,"value":586},"into",{"type":65,"value":588}," configs rather than inventing a name and label set each time.",{"type":59,"tag":68,"props":590,"children":591},{},[592],{"type":65,"value":593},"Before writing any annotation:",{"type":59,"tag":208,"props":595,"children":596},{},[597,771,825,947],{"type":59,"tag":124,"props":598,"children":599},{},[600,605,607,613,615,621,623],{"type":59,"tag":128,"props":601,"children":602},{},[603],{"type":65,"value":604},"Pull the project's configs first.",{"type":65,"value":606}," Query ",{"type":59,"tag":74,"props":608,"children":610},{"className":609},[],[611],{"type":65,"value":612},"Project.annotationConfigs",{"type":65,"value":614}," and read the existing names and their label\u002Fscore schemes. This is the established rubric — prefer it over anything you would invent. (Resolve the project node id as described in the ",{"type":59,"tag":74,"props":616,"children":618},{"className":617},[],[619],{"type":65,"value":620},"phoenix-graphql",{"type":65,"value":622}," skill.)",{"type":59,"tag":624,"props":625,"children":630},"pre",{"className":626,"code":627,"language":628,"meta":629,"style":629},"language-graphql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","query ProjectAnnotationConfigs($projectId: ID!) {\n  node(id: $projectId) {\n    ... on Project {\n      annotationConfigs(first: 100) {\n        edges { node {\n          __typename\n          ... on AnnotationConfigBase { name description annotationType }\n          ... on CategoricalAnnotationConfig { id optimizationDirection values { label score } }\n          ... on ContinuousAnnotationConfig { id optimizationDirection lowerBound upperBound }\n          ... on FreeformAnnotationConfig { id }\n        } }\n      }\n    }\n  }\n}\n","graphql","",[631],{"type":59,"tag":74,"props":632,"children":633},{"__ignoreMap":629},[634,645,654,663,672,681,690,699,708,717,726,735,744,753,762],{"type":59,"tag":635,"props":636,"children":639},"span",{"class":637,"line":638},"line",1,[640],{"type":59,"tag":635,"props":641,"children":642},{},[643],{"type":65,"value":644},"query ProjectAnnotationConfigs($projectId: ID!) {\n",{"type":59,"tag":635,"props":646,"children":648},{"class":637,"line":647},2,[649],{"type":59,"tag":635,"props":650,"children":651},{},[652],{"type":65,"value":653},"  node(id: $projectId) {\n",{"type":59,"tag":635,"props":655,"children":657},{"class":637,"line":656},3,[658],{"type":59,"tag":635,"props":659,"children":660},{},[661],{"type":65,"value":662},"    ... on Project {\n",{"type":59,"tag":635,"props":664,"children":666},{"class":637,"line":665},4,[667],{"type":59,"tag":635,"props":668,"children":669},{},[670],{"type":65,"value":671},"      annotationConfigs(first: 100) {\n",{"type":59,"tag":635,"props":673,"children":675},{"class":637,"line":674},5,[676],{"type":59,"tag":635,"props":677,"children":678},{},[679],{"type":65,"value":680},"        edges { node {\n",{"type":59,"tag":635,"props":682,"children":684},{"class":637,"line":683},6,[685],{"type":59,"tag":635,"props":686,"children":687},{},[688],{"type":65,"value":689},"          __typename\n",{"type":59,"tag":635,"props":691,"children":693},{"class":637,"line":692},7,[694],{"type":59,"tag":635,"props":695,"children":696},{},[697],{"type":65,"value":698},"          ... on AnnotationConfigBase { name description annotationType }\n",{"type":59,"tag":635,"props":700,"children":702},{"class":637,"line":701},8,[703],{"type":59,"tag":635,"props":704,"children":705},{},[706],{"type":65,"value":707},"          ... on CategoricalAnnotationConfig { id optimizationDirection values { label score } }\n",{"type":59,"tag":635,"props":709,"children":711},{"class":637,"line":710},9,[712],{"type":59,"tag":635,"props":713,"children":714},{},[715],{"type":65,"value":716},"          ... on ContinuousAnnotationConfig { id optimizationDirection lowerBound upperBound }\n",{"type":59,"tag":635,"props":718,"children":720},{"class":637,"line":719},10,[721],{"type":59,"tag":635,"props":722,"children":723},{},[724],{"type":65,"value":725},"          ... on FreeformAnnotationConfig { id }\n",{"type":59,"tag":635,"props":727,"children":729},{"class":637,"line":728},11,[730],{"type":59,"tag":635,"props":731,"children":732},{},[733],{"type":65,"value":734},"        } }\n",{"type":59,"tag":635,"props":736,"children":738},{"class":637,"line":737},12,[739],{"type":59,"tag":635,"props":740,"children":741},{},[742],{"type":65,"value":743},"      }\n",{"type":59,"tag":635,"props":745,"children":747},{"class":637,"line":746},13,[748],{"type":59,"tag":635,"props":749,"children":750},{},[751],{"type":65,"value":752},"    }\n",{"type":59,"tag":635,"props":754,"children":756},{"class":637,"line":755},14,[757],{"type":59,"tag":635,"props":758,"children":759},{},[760],{"type":65,"value":761},"  }\n",{"type":59,"tag":635,"props":763,"children":765},{"class":637,"line":764},15,[766],{"type":59,"tag":635,"props":767,"children":768},{},[769],{"type":65,"value":770},"}\n",{"type":59,"tag":124,"props":772,"children":773},{},[774,779,781,786,788,793,795,800,802,807,809,815,817,823],{"type":59,"tag":128,"props":775,"children":776},{},[777],{"type":65,"value":778},"Reuse an existing config when one fits.",{"type":65,"value":780}," Annotate with the config's exact ",{"type":59,"tag":74,"props":782,"children":784},{"className":783},[],[785],{"type":65,"value":79},{"type":65,"value":787}," and a ",{"type":59,"tag":74,"props":789,"children":791},{"className":790},[],[792],{"type":65,"value":87},{"type":65,"value":794}," from its ",{"type":59,"tag":74,"props":796,"children":798},{"className":797},[],[799],{"type":65,"value":550},{"type":65,"value":801}," (or a ",{"type":59,"tag":74,"props":803,"children":805},{"className":804},[],[806],{"type":65,"value":95},{"type":65,"value":808}," within its bounds). This is what keeps ",{"type":59,"tag":74,"props":810,"children":812},{"className":811},[],[813],{"type":65,"value":814},"annotations['tool_selection'].label == 'incorrect'",{"type":65,"value":816}," filterable and aggregatable across runs. A config that already defines the scale also answers questions you would otherwise stop to ask the user (e.g. \"what numeric range?\") — don't ",{"type":59,"tag":74,"props":818,"children":820},{"className":819},[],[821],{"type":65,"value":822},"ask_user",{"type":65,"value":824}," for something a config already specifies.",{"type":59,"tag":124,"props":826,"children":827},{},[828,833,835,902,906,908,914,916,922,924,930,931,937,939,945],{"type":59,"tag":128,"props":829,"children":830},{},[831],{"type":65,"value":832},"When a new category emerges that no config covers, codify it before (or as) you annotate",{"type":65,"value":834}," — don't add an ad-hoc annotation and move on. Use the annotation-config tools rather than raw GraphQL mutations; the tools provide the approval surface and the frontend owns the write path.",{"type":59,"tag":120,"props":836,"children":837},{},[838,871],{"type":59,"tag":124,"props":839,"children":840},{},[841,846,848,854,856,862,864,869],{"type":59,"tag":128,"props":842,"children":843},{},[844],{"type":65,"value":845},"No close config",{"type":65,"value":847}," → call ",{"type":59,"tag":74,"props":849,"children":851},{"className":850},[],[852],{"type":65,"value":853},"create_annotation_config",{"type":65,"value":855}," with the new rubric and the current project's ",{"type":59,"tag":74,"props":857,"children":859},{"className":858},[],[860],{"type":65,"value":861},"projectId",{"type":65,"value":863},", then annotate against that config's ",{"type":59,"tag":74,"props":865,"children":867},{"className":866},[],[868],{"type":65,"value":79},{"type":65,"value":870}," after the proposal is accepted.",{"type":59,"tag":124,"props":872,"children":873},{},[874,879,880,886,888,893,895,900],{"type":59,"tag":128,"props":875,"children":876},{},[877],{"type":65,"value":878},"A config is close but missing a label",{"type":65,"value":847},{"type":59,"tag":74,"props":881,"children":883},{"className":882},[],[884],{"type":65,"value":885},"update_annotation_config",{"type":65,"value":887}," to replace it with the complete desired scheme, then annotate. Update is a full replace: pass the existing ",{"type":59,"tag":74,"props":889,"children":891},{"className":890},[],[892],{"type":65,"value":550},{"type":65,"value":894}," plus the new one, keep the same ",{"type":59,"tag":74,"props":896,"children":898},{"className":897},[],[899],{"type":65,"value":79},{"type":65,"value":901},", and do not omit values you want to keep.",{"type":59,"tag":903,"props":904,"children":905},"br",{},[],{"type":65,"value":907},"Codifying first means the next visit to this project rediscovers the criteria instead of growing a second, differently-named rubric for the same thing. Use ",{"type":59,"tag":74,"props":909,"children":911},{"className":910},[],[912],{"type":65,"value":913},"type: \"categorical\" | \"continuous\" | \"freeform\"",{"type":65,"value":915}," and set ",{"type":59,"tag":74,"props":917,"children":919},{"className":918},[],[920],{"type":65,"value":921},"optimizationDirection",{"type":65,"value":923}," to ",{"type":59,"tag":74,"props":925,"children":927},{"className":926},[],[928],{"type":65,"value":929},"MINIMIZE",{"type":65,"value":290},{"type":59,"tag":74,"props":932,"children":934},{"className":933},[],[935],{"type":65,"value":936},"MAXIMIZE",{"type":65,"value":938},", or ",{"type":59,"tag":74,"props":940,"children":942},{"className":941},[],[943],{"type":65,"value":944},"NONE",{"type":65,"value":946}," when it matters.",{"type":59,"tag":124,"props":948,"children":949},{},[950,955],{"type":59,"tag":128,"props":951,"children":952},{},[953],{"type":65,"value":954},"Surface the choice explicitly.",{"type":65,"value":956}," Tell the user when you reused a config versus proposed a new one or extended an existing one, and why. Naming or changing a rubric is a decision they may want to weigh in on.",{"type":59,"tag":68,"props":958,"children":959},{},[960,965,967,971],{"type":59,"tag":128,"props":961,"children":962},{},[963],{"type":65,"value":964},"Do not cross-contaminate projects.",{"type":65,"value":966}," Config names are global, and a same-named config may be bound to other projects with different semantics. Prefer configs already associated with ",{"type":59,"tag":112,"props":968,"children":969},{},[970],{"type":65,"value":225},{"type":65,"value":972}," project; only create or associate configs for the project in context. Never repurpose another project's rubric just because its name looks right.",{"type":59,"tag":201,"props":974,"children":976},{"id":975},"mode-a-coaching-the-user",[977],{"type":65,"value":978},"Mode A: Coaching the User",{"type":59,"tag":68,"props":980,"children":981},{},[982,984,989],{"type":65,"value":983},"When the user asks ",{"type":59,"tag":112,"props":985,"children":986},{},[987],{"type":65,"value":988},"how",{"type":65,"value":990}," to annotate (rather than asking you to do it), teach the process rather than handing over a fixed rubric:",{"type":59,"tag":208,"props":992,"children":993},{},[994,1012,1022,1032,1048,1058],{"type":59,"tag":124,"props":995,"children":996},{},[997,1002,1004,1010],{"type":59,"tag":128,"props":998,"children":999},{},[1000],{"type":65,"value":1001},"Start from failures they have actually seen.",{"type":65,"value":1003}," Resist proposing a polished taxonomy up front — a pre-baked list causes confirmation bias. Ask what's going wrong, or use the ",{"type":59,"tag":74,"props":1005,"children":1007},{"className":1006},[],[1008],{"type":65,"value":1009},"debug-trace",{"type":65,"value":1011}," skill to surface real failure modes first.",{"type":59,"tag":124,"props":1013,"children":1014},{},[1015,1020],{"type":59,"tag":128,"props":1016,"children":1017},{},[1018],{"type":65,"value":1019},"Open-code before naming.",{"type":65,"value":1021}," Encourage free-form notes on a handful of traces (\"what's the first thing wrong here?\") before committing to category names.",{"type":59,"tag":124,"props":1023,"children":1024},{},[1025,1030],{"type":59,"tag":128,"props":1026,"children":1027},{},[1028],{"type":65,"value":1029},"Axial-code into a small vocabulary.",{"type":65,"value":1031}," Group similar notes into 5–10 named, mutually distinct, actionable categories. Each should be specific enough that two reviewers would label the same span the same way.",{"type":59,"tag":124,"props":1033,"children":1034},{},[1035,1040,1042,1046],{"type":59,"tag":128,"props":1036,"children":1037},{},[1038],{"type":65,"value":1039},"Define the label set per category.",{"type":65,"value":1041}," Usually binary. Write a one-sentence definition so the boundary is unambiguous. Once the vocabulary stabilizes, codify each category as an annotation config (see ",{"type":59,"tag":451,"props":1043,"children":1044},{"href":453},[1045],{"type":65,"value":456},{"type":65,"value":1047},") so the rubric persists and the next session reuses it instead of re-deriving it.",{"type":59,"tag":124,"props":1049,"children":1050},{},[1051,1056],{"type":59,"tag":128,"props":1052,"children":1053},{},[1054],{"type":65,"value":1055},"Apply, then aggregate.",{"type":65,"value":1057}," Label a representative sample consistently, then compute failure rates to prioritize. Highest-frequency × highest-impact wins.",{"type":59,"tag":124,"props":1059,"children":1060},{},[1061,1066,1068,1073],{"type":59,"tag":128,"props":1062,"children":1063},{},[1064],{"type":65,"value":1065},"Fix-first.",{"type":65,"value":1067}," Remind the user that many failures (missing prompt instruction, missing tool, retrieval bug) are better ",{"type":59,"tag":112,"props":1069,"children":1070},{},[1071],{"type":65,"value":1072},"fixed",{"type":65,"value":1074}," than measured. Reserve standing annotations\u002Fevals for failures they will iterate on repeatedly or that need a guardrail.",{"type":59,"tag":68,"props":1076,"children":1077},{},[1078,1080,1085],{"type":65,"value":1079},"Explain ",{"type":59,"tag":112,"props":1081,"children":1082},{},[1083],{"type":65,"value":1084},"why",{"type":65,"value":1086}," a convention matters (\"stable names let you trend this over time\") rather than only stating it.",{"type":59,"tag":201,"props":1088,"children":1090},{"id":1089},"mode-b-writing-annotations-yourself",[1091],{"type":65,"value":1092},"Mode B: Writing Annotations Yourself",{"type":59,"tag":68,"props":1094,"children":1095},{},[1096],{"type":65,"value":1097},"When the user asks you to save annotations:",{"type":59,"tag":208,"props":1099,"children":1100},{},[1101,1118,1128,1151,1161,1171,1266],{"type":59,"tag":124,"props":1102,"children":1103},{},[1104,1109,1111,1116],{"type":59,"tag":128,"props":1105,"children":1106},{},[1107],{"type":65,"value":1108},"Confirm scope and intent.",{"type":65,"value":1110}," Only annotate when the user wants feedback persisted, not during ordinary analysis. Know which spans and which dimension(s) are in scope. If no failure modes are established yet, diagnose first with the ",{"type":59,"tag":74,"props":1112,"children":1114},{"className":1113},[],[1115],{"type":65,"value":1009},{"type":65,"value":1117}," skill, then return here to persist the results — don't re-derive a taxonomy.",{"type":59,"tag":124,"props":1119,"children":1120},{},[1121,1126],{"type":59,"tag":128,"props":1122,"children":1123},{},[1124],{"type":65,"value":1125},"Inspect before judging.",{"type":65,"value":1127}," Read the actual span input\u002Foutput (and relevant parent\u002Fchild spans for context). Never annotate from span status codes alone — a success status can mask an error in attributes, and an exception can be expected behavior.",{"type":59,"tag":124,"props":1129,"children":1130},{},[1131,1136,1138,1143,1145,1149],{"type":59,"tag":128,"props":1132,"children":1133},{},[1134],{"type":65,"value":1135},"Pick the dimension(s) from the project's annotation configs.",{"type":65,"value":1137}," Pull the configs first and reuse a matching config's ",{"type":59,"tag":74,"props":1139,"children":1141},{"className":1140},[],[1142],{"type":65,"value":79},{"type":65,"value":1144}," and label\u002Fscore scheme; if a needed category has no config, create or extend one (see ",{"type":59,"tag":451,"props":1146,"children":1147},{"href":453},[1148],{"type":65,"value":456},{"type":65,"value":1150},") before writing. Then judge every span in the batch on that fixed vocabulary. If you're judging more than one dimension, settle each one's config up front.",{"type":59,"tag":124,"props":1152,"children":1153},{},[1154,1159],{"type":59,"tag":128,"props":1155,"children":1156},{},[1157],{"type":65,"value":1158},"Annotate the right span for each judgment",{"type":65,"value":1160}," (principle 3) and the first failure (principle 4). Use only real IDs from context or prior tool results — never guess span IDs.",{"type":59,"tag":124,"props":1162,"children":1163},{},[1164,1169],{"type":59,"tag":128,"props":1165,"children":1166},{},[1167],{"type":65,"value":1168},"Write a grounded explanation per annotation",{"type":65,"value":1170}," citing the specific evidence in that span.",{"type":59,"tag":124,"props":1172,"children":1173},{},[1174,1186,1188,1194,1196,1202,1204],{"type":59,"tag":128,"props":1175,"children":1176},{},[1177,1179,1184],{"type":65,"value":1178},"Batch the related annotations into one ",{"type":59,"tag":74,"props":1180,"children":1182},{"className":1181},[],[1183],{"type":65,"value":190},{"type":65,"value":1185}," call",{"type":65,"value":1187}," and pick the ",{"type":59,"tag":74,"props":1189,"children":1191},{"className":1190},[],[1192],{"type":65,"value":1193},"identifier",{"type":65,"value":1195}," to match your intent (annotations are keyed by ",{"type":59,"tag":74,"props":1197,"children":1199},{"className":1198},[],[1200],{"type":65,"value":1201},"(name, span, identifier)",{"type":65,"value":1203},"):\n",{"type":59,"tag":120,"props":1205,"children":1206},{},[1207,1234,1261],{"type":59,"tag":124,"props":1208,"children":1209},{},[1210,1212,1217,1219,1225,1227,1232],{"type":65,"value":1211},"Use a ",{"type":59,"tag":128,"props":1213,"children":1214},{},[1215],{"type":65,"value":1216},"stable",{"type":65,"value":1218}," identifier that names you as the author — e.g. ",{"type":59,"tag":74,"props":1220,"children":1222},{"className":1221},[],[1223],{"type":65,"value":1224},"pxi",{"type":65,"value":1226}," — when the judgment should be ",{"type":59,"tag":112,"props":1228,"children":1229},{},[1230],{"type":65,"value":1231},"updatable",{"type":65,"value":1233},": re-reviewing the same span and dimension overwrites the prior annotation instead of duplicating it.",{"type":59,"tag":124,"props":1235,"children":1236},{},[1237,1238,1243,1245,1251,1253,1259],{"type":65,"value":1211},{"type":59,"tag":128,"props":1239,"children":1240},{},[1241],{"type":65,"value":1242},"descriptive, run-scoped",{"type":65,"value":1244}," identifier — e.g. ",{"type":59,"tag":74,"props":1246,"children":1248},{"className":1247},[],[1249],{"type":65,"value":1250},"pxi:tool-misuse-2026-05-29",{"type":65,"value":1252}," — for a discrete review batch you may want to query or revert as a unit later. This mirrors the Phoenix CLI's ",{"type":59,"tag":74,"props":1254,"children":1256},{"className":1255},[],[1257],{"type":65,"value":1258},"coding-run:\u003Ctopic>-\u003Cdate>",{"type":65,"value":1260}," convention: a descriptive id carries meaning for whoever opens the data later, far better than an opaque constant.",{"type":59,"tag":124,"props":1262,"children":1263},{},[1264],{"type":65,"value":1265},"Either way, the author prefix keeps your annotations distinguishable from human or other-evaluator annotations on the same span. Do not reuse one identifier for two unrelated runs you'd want to revert separately.",{"type":59,"tag":124,"props":1267,"children":1268},{},[1269,1274,1276,1282,1283,1289,1291,1297,1299,1305,1307,1313],{"type":59,"tag":128,"props":1270,"children":1271},{},[1272],{"type":65,"value":1273},"Report back",{"type":65,"value":1275}," what you annotated, the dimension(s) and vocabulary used, and the distribution of labels — and link to a representative annotated span so the user can verify. Build links from the OpenTelemetry hex IDs in the ",{"type":59,"tag":74,"props":1277,"children":1279},{"className":1278},[],[1280],{"type":65,"value":1281},"spanId",{"type":65,"value":275},{"type":59,"tag":74,"props":1284,"children":1286},{"className":1285},[],[1287],{"type":65,"value":1288},"traceId",{"type":65,"value":1290}," GraphQL fields (never the Relay ",{"type":59,"tag":74,"props":1292,"children":1294},{"className":1293},[],[1295],{"type":65,"value":1296},"id",{"type":65,"value":1298}," field): ",{"type":59,"tag":74,"props":1300,"children":1302},{"className":1301},[],[1303],{"type":65,"value":1304},"[short label](\u002Fredirects\u002Fspans\u002F\u003CspanId>)",{"type":65,"value":1306}," for a span, or ",{"type":59,"tag":74,"props":1308,"children":1310},{"className":1309},[],[1311],{"type":65,"value":1312},"[short label](\u002Fredirects\u002Ftraces\u002F\u003CtraceId>)",{"type":65,"value":1314}," when no single span captures the judgment. A span link lands the user on the exact annotated node.",{"type":59,"tag":201,"props":1316,"children":1318},{"id":1317},"anti-patterns",[1319],{"type":65,"value":1320},"Anti-Patterns",{"type":59,"tag":120,"props":1322,"children":1323},{},[1324,1349,1380,1413,1423,1433,1443,1458],{"type":59,"tag":124,"props":1325,"children":1326},{},[1327,1332,1334,1340,1341,1347],{"type":59,"tag":128,"props":1328,"children":1329},{},[1330],{"type":65,"value":1331},"Generic score columns",{"type":65,"value":1333}," (",{"type":59,"tag":74,"props":1335,"children":1337},{"className":1336},[],[1338],{"type":65,"value":1339},"hallucination_score",{"type":65,"value":290},{"type":59,"tag":74,"props":1342,"children":1344},{"className":1343},[],[1345],{"type":65,"value":1346},"quality_score",{"type":65,"value":1348},") not tied to a concrete observed behavior.",{"type":59,"tag":124,"props":1350,"children":1351},{},[1352,1357,1358,1364,1366,1372,1373,1379],{"type":59,"tag":128,"props":1353,"children":1354},{},[1355],{"type":65,"value":1356},"Outcome baked into the name",{"type":65,"value":1333},{"type":59,"tag":74,"props":1359,"children":1361},{"className":1360},[],[1362],{"type":65,"value":1363},"name: \"passed_relevance\"",{"type":65,"value":1365},") instead of ",{"type":59,"tag":74,"props":1367,"children":1369},{"className":1368},[],[1370],{"type":65,"value":1371},"name: \"relevance\"",{"type":65,"value":290},{"type":59,"tag":74,"props":1374,"children":1376},{"className":1375},[],[1377],{"type":65,"value":1378},"label: \"pass\"",{"type":65,"value":520},{"type":59,"tag":124,"props":1381,"children":1382},{},[1383,1388,1390,1395,1397,1403,1405,1411],{"type":59,"tag":128,"props":1384,"children":1385},{},[1386],{"type":65,"value":1387},"Inconsistent vocabulary",{"type":65,"value":1389}," across spans (",{"type":59,"tag":74,"props":1391,"children":1393},{"className":1392},[],[1394],{"type":65,"value":356},{"type":65,"value":1396}," here, ",{"type":59,"tag":74,"props":1398,"children":1400},{"className":1399},[],[1401],{"type":65,"value":1402},"bad",{"type":65,"value":1404}," there, ",{"type":59,"tag":74,"props":1406,"children":1408},{"className":1407},[],[1409],{"type":65,"value":1410},"0",{"type":65,"value":1412}," elsewhere) — kills filtering and aggregation.",{"type":59,"tag":124,"props":1414,"children":1415},{},[1416,1421],{"type":59,"tag":128,"props":1417,"children":1418},{},[1419],{"type":65,"value":1420},"Annotating every downstream symptom",{"type":65,"value":1422}," of a single root failure.",{"type":59,"tag":124,"props":1424,"children":1425},{},[1426,1431],{"type":59,"tag":128,"props":1427,"children":1428},{},[1429],{"type":65,"value":1430},"Labeling without an explanation",{"type":65,"value":1432}," when the judgment is anything but trivially obvious.",{"type":59,"tag":124,"props":1434,"children":1435},{},[1436,1441],{"type":59,"tag":128,"props":1437,"children":1438},{},[1439],{"type":65,"value":1440},"Annotating from status codes or metadata",{"type":65,"value":1442}," without reading the actual span content.",{"type":59,"tag":124,"props":1444,"children":1445},{},[1446,1456],{"type":59,"tag":128,"props":1447,"children":1448},{},[1449,1451],{"type":65,"value":1450},"Recording your own judgment as ",{"type":59,"tag":74,"props":1452,"children":1454},{"className":1453},[],[1455],{"type":65,"value":503},{"type":65,"value":1457},", or guessing span IDs.",{"type":59,"tag":124,"props":1459,"children":1460},{},[1461,1466],{"type":59,"tag":128,"props":1462,"children":1463},{},[1464],{"type":65,"value":1465},"Inventing a taxonomy before reading traces",{"type":65,"value":1467},", when coaching the user.",{"type":59,"tag":1469,"props":1470,"children":1471},"style",{},[1472],{"type":65,"value":1473},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":1475,"total":701},[1476,1483,1495,1506,1518,1529,1541],{"slug":4,"name":4,"fn":5,"description":6,"org":1477,"tags":1478,"stars":26,"repoUrl":27,"updatedAt":28},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1479,1480,1481,1482],{"name":24,"slug":25,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":37,"name":37,"fn":1484,"description":1485,"org":1486,"tags":1487,"stars":26,"repoUrl":27,"updatedAt":1494},"reason about Phoenix dataset structure","Understand what a Phoenix dataset is and reason well about its examples, outputs, splits, and how it feeds evaluators and experiments. Load this whenever a dataset is in view or the user asks what a dataset is, how splits work, what an output \"means\", or how datasets relate to experiments and evals. This skill governs the judgment; any tool descriptions govern the mechanics.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1488,1491,1493],{"name":1489,"slug":1490,"type":16},"Data Analysis","data-analysis",{"name":1492,"slug":37,"type":16},"Datasets",{"name":24,"slug":25,"type":16},"2026-07-12T08:08:21.695457",{"slug":1009,"name":1009,"fn":1496,"description":1497,"org":1498,"tags":1499,"stars":26,"repoUrl":27,"updatedAt":1505},"diagnose failures using trace investigation","Diagnose failure modes by systematically investigating traces. Trigger when the user explicitly asks for cross-trace diagnosis: \"what's going wrong?\", \"were there errors?\", \"debug this\", \"where is my agent struggling?\". Do NOT trigger on: (1) advice questions (\"what should I do?\"), (2) statistical questions (\"what's the average latency?\"), (3) summarize requests, (4) trace filtering (\"show me traces with errors\"), (5) vague questions (\"is there a problem?\"), (6) unrelated requests.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1500,1503,1504],{"name":1501,"slug":1502,"type":16},"Debugging","debugging",{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"2026-07-12T08:08:10.44243",{"slug":1507,"name":1507,"fn":1508,"description":1509,"org":1510,"tags":1511,"stars":26,"repoUrl":27,"updatedAt":1517},"evaluators","author and refine Phoenix evaluators","Author or refine a Phoenix evaluator — code or LLM-as-a-judge — that scores a run's output. Trigger when the user wants to create a new evaluator, improve an existing one's logic or rubric, choose labels, or decide what to measure on a dataset or experiment. Do NOT trigger on: (1) manual prompt drafting (use `playground`), (2) running or comparing experiments themselves (use `experiments`), (3) cross-trace failure diagnosis with no evaluator in scope (use `debug-trace`).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1512,1513,1514],{"name":24,"slug":25,"type":16},{"name":21,"slug":22,"type":16},{"name":1515,"slug":1516,"type":16},"Testing","testing","2026-07-31T05:58:09.13624",{"slug":1519,"name":1519,"fn":1520,"description":1521,"org":1522,"tags":1523,"stars":26,"repoUrl":27,"updatedAt":1528},"experiments","run and compare dataset-backed experiments","Run, read, and compare dataset-backed experiments to find evidence that a prompt or pipeline is improving. Trigger when the user wants to iterate over a dataset with experiments, compare experiment runs, read experiment quality\u002Flatency\u002Fcost, or decide whether a change actually helped. Running a prompt over a dataset is implicitly an experiment — load this skill when dataset-backed work begins, before authoring evaluators for the experiment and before starting the recorded run, not only when reading results. Do NOT trigger on: (1) manual prompt drafting with no dataset-backed evaluation in scope (use `playground`), (2) authoring or refining an evaluator's logic or rubric (use `evaluators`), (3) cross-trace failure diagnosis with no experiment in scope (use `debug-trace`).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1524,1525,1526,1527],{"name":1492,"slug":37,"type":16},{"name":24,"slug":25,"type":16},{"name":21,"slug":22,"type":16},{"name":1515,"slug":1516,"type":16},"2026-07-12T08:08:11.691477",{"slug":620,"name":620,"fn":1530,"description":1531,"org":1532,"tags":1533,"stars":26,"repoUrl":27,"updatedAt":1540},"query Phoenix API with GraphQL","Write efficient GraphQL queries against the Phoenix API. Load this skill in two cases: (1) before composing any non-trivial GraphQL query yourself for data analysis (via the `phoenix-gql` bash command) — it contains schema entrypoints and patterns that eliminate the need for introspection; (2) when the user asks for help writing GraphQL queries for their own scripts, tools, or integrations against Phoenix — it covers the endpoint, authentication, and client examples.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1534,1537,1538],{"name":1535,"slug":1536,"type":16},"Analytics","analytics",{"name":1489,"slug":1490,"type":16},{"name":1539,"slug":628,"type":16},"GraphQL","2026-07-12T08:08:17.163493",{"slug":1542,"name":1542,"fn":1543,"description":1544,"org":1545,"tags":1546,"stars":26,"repoUrl":27,"updatedAt":1550},"playground","author and iterate on prompts in Phoenix","Author, edit, or iterate on prompts in the Phoenix prompt playground, including running experiments over a dataset. Load before any playground tool call, including single-shot prompt rewrites.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1547,1548,1549],{"name":24,"slug":25,"type":16},{"name":21,"slug":22,"type":16},{"name":1515,"slug":1516,"type":16},"2026-07-12T08:08:12.920792",{"items":1552,"total":1676},[1553,1560,1566,1572,1578,1585,1591,1597,1608,1626,1644,1654],{"slug":4,"name":4,"fn":5,"description":6,"org":1554,"tags":1555,"stars":26,"repoUrl":27,"updatedAt":28},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1556,1557,1558,1559],{"name":24,"slug":25,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":37,"name":37,"fn":1484,"description":1485,"org":1561,"tags":1562,"stars":26,"repoUrl":27,"updatedAt":1494},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1563,1564,1565],{"name":1489,"slug":1490,"type":16},{"name":1492,"slug":37,"type":16},{"name":24,"slug":25,"type":16},{"slug":1009,"name":1009,"fn":1496,"description":1497,"org":1567,"tags":1568,"stars":26,"repoUrl":27,"updatedAt":1505},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1569,1570,1571],{"name":1501,"slug":1502,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":1507,"name":1507,"fn":1508,"description":1509,"org":1573,"tags":1574,"stars":26,"repoUrl":27,"updatedAt":1517},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1575,1576,1577],{"name":24,"slug":25,"type":16},{"name":21,"slug":22,"type":16},{"name":1515,"slug":1516,"type":16},{"slug":1519,"name":1519,"fn":1520,"description":1521,"org":1579,"tags":1580,"stars":26,"repoUrl":27,"updatedAt":1528},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1581,1582,1583,1584],{"name":1492,"slug":37,"type":16},{"name":24,"slug":25,"type":16},{"name":21,"slug":22,"type":16},{"name":1515,"slug":1516,"type":16},{"slug":620,"name":620,"fn":1530,"description":1531,"org":1586,"tags":1587,"stars":26,"repoUrl":27,"updatedAt":1540},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1588,1589,1590],{"name":1535,"slug":1536,"type":16},{"name":1489,"slug":1490,"type":16},{"name":1539,"slug":628,"type":16},{"slug":1542,"name":1542,"fn":1543,"description":1544,"org":1592,"tags":1593,"stars":26,"repoUrl":27,"updatedAt":1550},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1594,1595,1596],{"name":24,"slug":25,"type":16},{"name":21,"slug":22,"type":16},{"name":1515,"slug":1516,"type":16},{"slug":1598,"name":1598,"fn":1599,"description":1600,"org":1601,"tags":1602,"stars":26,"repoUrl":27,"updatedAt":1607},"span-coding","analyze and code Phoenix spans","Open-code Phoenix spans with PXI-owned notes, recover those notes for axial coding, and promote stable categories into structured annotations. Load this when analyzing spans to discover failure patterns before a taxonomy exists.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1603,1604,1605,1606],{"name":1501,"slug":1502,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},"2026-07-12T08:08:19.597239",{"slug":1609,"name":1609,"fn":1610,"description":1611,"org":1612,"tags":1613,"stars":1623,"repoUrl":1624,"updatedAt":1625},"arize-admin","manage Arize enterprise user access","Manages Arize users, organizations, spaces, projects, roles, role bindings, resource restrictions, and API keys via the ax CLI. Use for enterprise admin workflows: inviting and offboarding users, onboarding new teams, creating custom roles for SAML\u002FSSO mappings, assigning roles to users, restricting project-level access, and managing service keys for multi-tenant architectures. Covers ax users, ax organizations, ax spaces, ax projects, ax roles, ax role-bindings, and ax api-keys.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1614,1617,1620],{"name":1615,"slug":1616,"type":16},"CLI","cli",{"name":1618,"slug":1619,"type":16},"Operations","operations",{"name":1621,"slug":1622,"type":16},"Permissions","permissions",38,"https:\u002F\u002Fgithub.com\u002FArize-ai\u002Farize-skills","2026-07-22T05:37:21.991338",{"slug":1627,"name":1627,"fn":1628,"description":1629,"org":1630,"tags":1631,"stars":1623,"repoUrl":1624,"updatedAt":1643},"arize-ai-provider-integration","manage Arize AI provider integrations","Creates, reads, updates, and deletes Arize AI integrations that store LLM provider credentials used by evaluators and other Arize features. Supports any LLM provider (e.g. OpenAI, Anthropic, Azure OpenAI, AWS Bedrock, Vertex AI, Gemini, NVIDIA NIM). Use when the user mentions AI integration, LLM provider credentials, create integration, list integrations, update credentials, delete integration, or connecting an LLM provider to Arize.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1632,1634,1637,1640,1641],{"name":1633,"slug":36,"type":16},"Anthropic",{"name":1635,"slug":1636,"type":16},"Azure","azure",{"name":1638,"slug":1639,"type":16},"Integrations","integrations",{"name":21,"slug":22,"type":16},{"name":1642,"slug":44,"type":16},"OpenAI","2026-07-22T05:37:23.90468",{"slug":1645,"name":1645,"fn":1646,"description":1647,"org":1648,"tags":1649,"stars":1623,"repoUrl":1624,"updatedAt":1653},"arize-annotation","manage Arize annotation workflows","Creates and manages annotation configs (categorical, continuous, freeform label schemas) and annotation queues (human review workflows) on Arize. Applies human annotations to project spans via the Python SDK. Use when the user mentions annotation config, annotation queue, label schema, human feedback, bulk annotate spans, update_annotations, labeling queue, annotate record, or human review.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1650,1651,1652],{"name":1489,"slug":1490,"type":16},{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},"2026-07-22T05:37:19.010776",{"slug":1655,"name":1655,"fn":1656,"description":1657,"org":1658,"tags":1659,"stars":1623,"repoUrl":1624,"updatedAt":1675},"arize-compliance-audit","audit AI agents for regulatory compliance","INVOKE THIS SKILL when auditing an AI agent or LLM app for regulatory compliance. Covers EU AI Act, GPAI Code of Practice, GDPR, NIST AI RMF, Colorado AI Act, HIPAA, and ISO 42001. Scans the codebase for compliance gaps, cross-references Arize instrumentation for audit trail coverage, and produces an actionable remediation checklist tailored to the selected frameworks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[1660,1663,1666,1669,1672],{"name":1661,"slug":1662,"type":16},"Audit","audit",{"name":1664,"slug":1665,"type":16},"Compliance","compliance",{"name":1667,"slug":1668,"type":16},"GDPR","gdpr",{"name":1670,"slug":1671,"type":16},"Legal","legal",{"name":1673,"slug":1674,"type":16},"Security","security","2026-07-19T05:39:42.632738",23]