[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-microsoft-eval-triage-and-improvement":3,"mdc--508p6l-key":39,"related-repo-microsoft-eval-triage-and-improvement":3190,"related-org-microsoft-eval-triage-and-improvement":3228},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":28,"repoUrl":29,"updatedAt":30,"license":31,"forks":32,"topics":33,"repo":34,"sourceUrl":37,"mdContent":38},"eval-triage-and-improvement","triage and improve Copilot agent evaluations","Use this skill when the user's Copilot Studio agent evaluations have come back and they need to interpret scores, diagnose root causes of underperforming test cases, find remediation steps, or analyze patterns to improve their agent. Always use this skill when the user mentions: \"eval failed\", \"why did this fail\", \"triage\", \"diagnose failure\", \"low pass rate\", \"fix evaluation results\", \"not passing\", \"failing test cases\", \"evaluation results\", \"improve my eval scores\", or any situation where eval scores need interpretation and action.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"microsoft","Microsoft","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fmicrosoft.png",[12,16,19,22,25],{"name":13,"slug":14,"type":15},"Observability","observability","tag",{"name":17,"slug":18,"type":15},"Evals","evals",{"name":20,"slug":21,"type":15},"Copilot Studio","copilot-studio",{"name":23,"slug":24,"type":15},"Agents","agents",{"name":26,"slug":27,"type":15},"Debugging","debugging",117,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Feval-guide","2026-04-06T18:36:55.186287",null,19,[],{"repoUrl":29,"stars":28,"forks":32,"topics":35,"description":36},[],"A plugin for AI agent evaluation. Plan evals, generate test cases, interpret results for Copilot Studio agents. Grounded in Microsoft's Eval Scenario Library & Triage Playbook.","https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Feval-guide\u002Ftree\u002FHEAD\u002Fskills\u002Feval-triage-and-improvement","---\nname: eval-triage-and-improvement\ndescription: 'Use this skill when the user''s Copilot Studio agent evaluations have come back and they need to interpret scores, diagnose root causes of underperforming test cases, find remediation steps, or analyze patterns to improve their agent. Always use this skill when the user mentions: \"eval failed\", \"why did this fail\", \"triage\", \"diagnose failure\", \"low pass rate\", \"fix evaluation results\", \"not passing\", \"failing test cases\", \"evaluation results\", \"improve my eval scores\", or any situation where eval scores need interpretation and action.'\n---\n\n# Eval Triage & Improvement\n\nYou help users interpret their agent evaluation results and find actionable next steps to improve. Follow the hybrid workflow: gather eval results first, then generate a structured triage report with Step 7 root buckets, owners, and recommended fixes.\n\nThis skill is grounded in `skills\u002Feval-guide\u002Fplaybook.md`, the canonical **Practical Guidance on Agent Evaluation: 10-step playbook**. It is the deep-dive for **Step 7 — Iterate to Diagnose Failures** and seeds **Step 9 — Optimization Loop** for production feedback. MS Learn pages and the Eval Guidance Kit remain supporting sources for Copilot Studio mechanics, lifecycle cadence, and checklist artifacts.\n\n### When to use this skill vs. eval-result-interpreter\n\nThese two skills share the same triage framework but serve different modes of work:\n\n| Use **eval-triage-and-improvement** when… | Use **eval-result-interpreter** when… |\n|---|---|\n| You want **interactive guidance** walking through diagnosis step by step | You have a CSV file or concrete results and want a **one-shot structured report** |\n| You are in an **ongoing improvement loop** — fixing, re-running, and re-triaging | This is your **first look** at results — you need a verdict and top actions fast |\n| You need **detailed remediation help** for specific eval-set failure patterns (e.g., \"wrong tool fires — now what?\") | You want a **customer-deliverable artifact** (the .docx triage report) |\n| You have **many failures** (15+) and need help prioritizing which to investigate | The eval run is relatively straightforward (\u003C20 failures) |\n| You need the playbook worked examples and deeper diagnostic walkthroughs | You need the **activity map \u002F result comparison** tool recommendations inline |\n\n**If in doubt:** Start with eval-result-interpreter to get the structured report, then switch to eval-triage-and-improvement if you need interactive help implementing the fixes.\n\n## Workflow\n\n### Step 1: Gather Eval Results\n\nAsk the user to share:\n1. **Which eval sets ran** and their pass rates (e.g., \"Faithfulness: 71%, Prompt injection: 95%\")\n2. **Methodology manifest metadata** from the companion `.docx` report or `stage-N-data.json`: `set_type`, `category`\u002Fcapability dimension, `method`, `gate` (hard\u002Fsoft), `target`, `regression_class`, human-review flag, and source\u002Fground-truth provenance\n3. **Specific failing test cases** — the test case ID, sample input, expected value, actual agent response, and eval method assigned in Copilot Studio\n4. **How many times they've run** — is this the first baseline run (Step 6) or a re-run after fixes?\n5. **What they've already tried** — any eval, agent, knowledge, or tool changes attempted so far?\n\nIf they don't have structured results, help them organize what they have. Prefer manifest metadata over inferring from filenames or question text. If they just have a general complaint (\"my agent isn't working well\"), guide them to run a baseline first using the scenario library and the 10-step playbook's Steps 1-6.\n\n### Step 2: Score Interpretation\n\nAssess readiness from the manifest's Step 4 targets and gates:\n\n```\nREADINESS ASSESSMENT\n\nAny failed hard gate            → BLOCK (trust & safety hard gates block regardless of aggregate pass rate)\nCapability set below hard floor → ITERATE or BLOCK based on risk tier and capability criticality\nSoft target missed only         → SHIP WITH KNOWN GAPS \u002F ITERATE (tracked, not blocking)\nAll hard gates + targets pass   → SHIP\n```\n\n**Setting thresholds** — don't apply fixed numbers. Use the manifest target first; if missing, derive a provisional target from the agent-level **risk tier** and flag that the manifest needs updating.\n\n| Factor | Higher Threshold When... |\n|--------|------------------------|\n| Criticality of error | Financial loss, safety risk, legal exposure |\n| Reach | External customers or large internal population |\n| Autonomy \u002F blast radius | Agent can take actions or trigger downstream systems |\n| Regulatory exposure | Regulated workflow, audit requirement, compliance obligation |\n| Data sensitivity | PII, PHI, confidential, or tenant-sensitive data |\n\n### Step 3: Pre-Triage Infrastructure Check\n\nBefore diagnosing individual failures, verify infrastructure was healthy during the eval run:\n\n- [ ] All knowledge sources accessible and fully indexed?\n- [ ] API backends and connectors returned no errors\u002Ftimeouts?\n- [ ] Authentication tokens valid throughout the run?\n- [ ] Correct agent version was published and evaluated?\n\nIf any dependency was unhealthy, recommend re-running after fixing infrastructure before triaging.\n\n### Step 4: Prioritize Failures\n\nIf the user has many failures, recommend this triage order:\n\n| Priority | Triage First | Rationale |\n|----------|-------------|-----------|\n| 1 | Failed hard gates, especially trust & safety sets | Highest consequence; blocks deploy regardless of aggregate score |\n| 2 | High-risk capability failures (accuracy, faithfulness, tool use) | Direct impact on agent value; hallucination is a faithfulness\u002Fcapability failure |\n| 3 | Lowest-scoring eval set failures | Likely systemic — fixing one pattern resolves multiple |\n| 4 | Recurring failures across baseline\u002Fre-runs | Most diagnosable and regression-prone |\n| 5 | Soft-target misses | Important but non-blocking unless pattern worsens |\n\n**15+ failures?** Don't triage every one. Review 3-5 from the lowest-scoring eval set. If they share a root bucket\u002Fsubtype pattern, fix that and re-run.\n\n### Step 5: Classify Root Cause\n\nFor each failure, work through the diagnostic questions in order. **Every failure must end in exactly one Step 7 root bucket**: `eval-setup problem` (response is acceptable; fix the eval) or `agent-quality problem` (real issue; fix the agent\u002Fplatform and log the pattern).\n\n```\nTRIAGE DECISION TREE (for each failing test case)\n\n1. Is the agent's response actually acceptable, even though it failed?\n   → YES = Eval-setup problem (grader, expected value, rubric, or method is wrong)\n\n2. Is the expected answer still current against the actual source\u002Fground truth in the manifest?\n   → NO = Eval-setup problem (expected answer outdated or source dependency drifted)\n\n3. Does the test case represent a realistic user input for this eval set's `set_type` and category?\n   → NO = Eval-setup problem (unrealistic or mis-scoped test case)\n\n4. Could a valid alternative response also be correct, but the grader rejects it?\n   → YES = Eval-setup problem (rubric\u002Fgrader too rigid)\n\n5. Is the eval method appropriate for what you're testing?\n   → NO = Eval-setup problem (wrong method; update the manifest and Copilot Studio row assignment)\n\nALL PASS → The eval is valid. Classify as **agent-quality problem** and proceed to operational subtype diagnosis:\n\n6. Does the issue come from prompt\u002Ftopic\u002Ftool\u002Fretrieval configuration or stale knowledge?\n   → YES = Agent Configuration \u002F Knowledge Issue (agent-quality problem)\n\n7. Does the behavior persist after reasonable config and knowledge fixes plus re-run?\n   → YES = Platform Limitation (agent-quality problem; log evidence and workaround)\n```\n\n### Step 5b: Conversation (Multi-Turn) Triage\n\nFor conversation eval failures, the standard decision tree still applies but you must first identify the **critical turn** — the earliest turn where the agent went wrong. Everything after a bad turn is a cascade, not independent failures.\n\n**Critical turn identification:**\n1. Walk the conversation turn by turn\n2. Find the first turn where the agent response diverges from expected behavior\n3. Classify that turn using the decision tree above\n4. Mark downstream turns as \"cascade — blocked by Turn N fix\"\n\n**Conversation-specific failure patterns and remediations:**\n\n| Pattern | How to spot it | Root cause area | Remediation |\n|---------|---------------|-----------------|-------------|\n| **Context loss** — Turn 1 fine, Turn 3+ forgets | Agent re-asks or contradicts earlier turns | Agent Config | Review topic management; ensure conversation context is preserved across topic switches |\n| **State loop** — Agent repeats the same response | Identical or near-identical agent turns in sequence | Agent Config | Check topic routing for circular references; add explicit exit conditions |\n| **Clarification failure** — Agent can't handle follow-ups | Turn 2 fails when user provides clarification or correction | Agent Config | Add follow-up handling instructions; check that topics accept partial\u002Fcorrective inputs |\n| **Last-mile failure** — Understands but can't resolve | Early turns diagnose correctly, final resolution turn fails | Agent Config or Platform | Check action\u002Fconnector configuration; verify the resolution path is wired correctly |\n| **Eval rigidity** — Conversation is acceptable but grader rejects | Reading the full conversation, the outcome is reasonable | Eval Setup | Conversation grading is limited (AI Generated or Approval Rating only); adjust rubric or expected values |\n\n**Key difference from single-response triage:** Do NOT triage each turn independently. Triage the critical turn, apply the fix, re-run, and then see which downstream turns self-resolve. Expect 40-60% of downstream failures to clear after fixing the critical turn.\n\n### Two Root Buckets + Operational Subtypes\n\n| Step 7 root bucket | Operational subtype | Who acts | What it means |\n|-----------|---------|----------|---------------|\n| **Eval-setup problem** | Eval setup issue | Eval author | The response is acceptable or the eval metadata\u002Frubric\u002Fexpected answer\u002Fmethod is wrong. Fix the eval and manifest. |\n| **Agent-quality problem** | Agent configuration issue | Agent builder | The agent genuinely produced a bad response. Fix prompt, topics, tools, retrieval, grounding, or knowledge. |\n| **Agent-quality problem** | Platform limitation | Platform team + agent owner | The eval caught a real issue caused by platform behavior. Log evidence, workaround if possible, and track the pattern. |\n\nMaintain a **failure-pattern log** for every agent-quality problem: test case, set_type\u002Fcategory, root bucket, subtype, suspected pattern, owner, fix location, verification eval set, and whether it should become a regression case (Step 8).\n\n### Step 6: Map to Remediation\n\nFor detailed remediation steps by Step 7 root bucket, operational subtype, and eval-set failure pattern, read the playbook files:\n- **Full triage decision tree**: Read `triage-and-improvement-playbook\u002Ftriage-decision-tree.md`\n- **Remediation mapping**: Read `triage-and-improvement-playbook\u002Fremediation-mapping.md`\n- **Pattern analysis**: Read `triage-and-improvement-playbook\u002Fpattern-analysis.md`\n- **Worked examples**: Read `triage-and-improvement-playbook\u002Fworked-examples.md`\n\n#### Quick Remediation Reference\n\n**Eval-setup fixes:**\n\n| Sub-Type | Fix |\n|----------|-----|\n| Outdated expected answer | Update expected value to match current source content |\n| Overly rigid grader | Switch to Compare Meaning, or broaden keyword set |\n| Unrealistic test case | Rewrite input using actual user language |\n| Wrong eval method | Change method to match the eval-set purpose and evidence type |\n| Grader error\u002Fbias | Review rubric, add examples, consider deterministic method |\n\n**Agent-quality fixes — agent configuration \u002F knowledge \u002F tools:**\n\n| Failure pattern | Common Fix |\n|-----------------|-----------|\n| Factual accuracy (wrong source) | Review knowledge source config, verify indexing, check vocabulary match |\n| Factual accuracy (wrong extraction) | Add extraction guidance to system prompt |\n| Hallucination (faithfulness capability failure) | Improve retrieval\u002Fchunking first; add instruction: \"Only answer from knowledge sources. If unavailable, say so.\" |\n| Wrong tool fires | Rewrite tool descriptions to differentiate; add negative examples |\n| Tool doesn't fire | Review trigger conditions; check if tool is enabled and accessible |\n| Wrong topic fires | Review trigger phrase overlap; adjust priority ordering |\n| Lacks empathy | Add context-specific tone instructions to system prompt |\n| Scope violation | Add explicit out-of-scope instruction |\n| PII leakage | Add PII protection instruction; review authentication scope |\n\n**Agent-quality fixes — platform limitation response:**\n- Document the limitation with evidence\n- Implement workaround where possible\n- Adjust eval thresholds to account for known platform behavior\n- File with platform team with reproduction steps\n\n### Step 7: Triage Rationale (teach the WHY)\n\nBefore generating the report, add rationale that teaches the customer the reasoning behind triage decisions — not just the conclusions. For each of these, use the actual eval data from this triage:\n\n1. **Why each failure got its root bucket and subtype** — Walk through the decision tree for at least one example per Step 7 root bucket. E.g., \"Test case KB-014 was classified as an Eval Setup Issue because the agent response is factually correct per the current knowledge source, but the expected value still references the old 14-day policy. The agent is right; the eval is stale.\"\n\n2. **Why the remediation targets config vs. content vs. eval** — Explain the logic: \"We recommended updating the knowledge source rather than changing the prompt because the agent retrieval worked correctly — it found the right document — but the document itself contains outdated information. A prompt change would mask the real problem.\"\n\n3. **Why the priority order is what it is** — Connect to blast radius and dependency chains: \"Failed hard gates come first because they block deploy and can change downstream behavior. Fix the hard-gate issue, re-run the regression suite, then triage the rest — otherwise you may diagnose failures that disappear once the blocking guardrail is corrected.\"\n\n4. **What this triage does NOT tell you** — Name the limits explicitly: \"This triage analyzed [N] failures from a single eval run. It cannot detect issues in scenarios you have not written test cases for, and it cannot distinguish between a flaky failure (non-determinism) and a real failure from a single data point. If a failure is borderline, re-run before investing in a fix.\"\n\nInclude this rationale in the triage report (see Triage Rationale section in the report template below).\n\n### Step 8: Generate Triage Report\n\nOutput a structured triage report:\n\n```markdown\n# Triage Report: [Agent Name] — [Date]\n\n## Score Summary\n| Eval Set | Set Type \u002F Category | Pass Rate | Target | Gate | Status |\n|----------|---------------------|-----------|--------|------|--------|\n| ... | capability \u002F faithfulness | ... | ... | hard\u002Fsoft | PASS\u002FBLOCK\u002FITERATE |\n\n## Readiness Assessment\n[SHIP \u002F SHIP WITH KNOWN GAPS \u002F ITERATE \u002F BLOCK]\n[Rationale]\n\n## Failure Analysis\n### Failure 1: [Test Case ID]\n- **Set Type \u002F Category:** capability or trust_safety \u002F ...\n- **Eval-set focus:** ...\n- **Sample Input:** ...\n- **Expected:** ...\n- **Actual:** ...\n- **Root Bucket:** [Eval-setup problem \u002F Agent-quality problem]\n- **Operational Subtype:** [Eval Setup \u002F Agent Config \u002F Knowledge \u002F Platform Limitation]\n- **Diagnosis:** [specific diagnosis]\n- **Owner:** [who needs to act]\n- **Remediation:** [specific action]\n- **Verification:** [how to verify the fix worked]\n\n[Repeat for each triaged failure]\n\n## Triage Rationale\n### Why these root bucket classifications\n[Walk through the decision tree for representative examples — show the reasoning, not just the label]\n\n### Why these remediations\n[Explain the logic connecting root bucket\u002Fsubtype to fix — why this fix and not an alternative]\n\n### Why this priority order\n[Connect priority to blast radius and dependency chains]\n\n### What this triage does NOT tell you\n[Name the limits: coverage gaps, single-run non-determinism, untested scenarios]\n\n## Failure-Pattern Log\n[Summarize recurring Step 7 patterns, owner, fix location, and whether each pattern should be added to the Step 8 regression suite]\n\n## Systemic Patterns\n[If 80%+ of failures share a root bucket\u002Fsubtype\u002Fcategory, call it out]\n\n## Action Items\n| # | Action | Owner | Priority | Verification |\n|---|--------|-------|----------|-------------|\n| 1 | ... | ... | ... | Re-run [eval set] |\n\n## Post-Triage Checklist\n- [ ] All failed hard gates addressed before deploy\n- [ ] Root buckets verified by reading actual responses\n- [ ] Eval-setup fixes applied to expected answers\u002Frubrics\u002Fmethod assignments\u002Fmanifest\n- [ ] Agent-quality patterns logged with owners and fix location\n- [ ] Full Step 8 regression suite re-run after fixes\n- [ ] Platform limitations filed if applicable\n\n## Human Review Required\n[Include human review checkpoints table — see Human Review Checkpoints section below]\n```\n\n### Post-Triage Verification\n\nAfter fixes are applied:\n- **Scores flat after fix?** → Wrong root bucket\u002Fsubtype, re-triage\n- **One score up, another down?** → Instruction conflict — the fix improved one behavior but degraded another\n- **80%+ of failures share a root bucket\u002Fsubtype?** → Systemic issue — fix the category, not individual test cases\n\n## Non-Determinism Handling\n\nLLM-based agents and graders produce variable outputs:\n- **Establish baselines:** Run 3+ times before treating any score as the Step 6 baseline. Use the average and record agent version + timestamp.\n- **Normal variance:** +\u002F-5% between runs is expected. Investigate if >10%.\n- **Flaky test cases** (pass sometimes, fail others): Agent may produce two valid responses but eval is too rigid. Investigate whether to broaden the expected value.\n- **Small eval sets (\u003C30 test cases):** A single test case flip changes the score by 3%+. Don't over-interpret.\n\n## Step 9 Optimization Loop: Production Signals\n\nIf the agent is deployed (even in preview), treat production feedback as the Step 9 loop: **collect signals → cluster → decide fix location → ship → re-evaluate against the Step 8 regression suite**. Prioritize signals in this order: thumbs-down (highest-signal negative feedback), escalations, manual overrides, support tickets, then qualitative comments.\n\n- **High thumbs-down on a topic where eval passes:** Coverage gap. Add or revise eval cases and tag them for regression if they represent recurring production risk.\n- **Thumbs-down clustering after a config change:** Possible regression. Re-run the Step 8 regression suite and add a case if the suite missed it.\n- **Escalations\u002Fmanual overrides:** Agent-quality problem until proven otherwise; cluster by capability or trust & safety category and choose the fix location: agent config\u002Fretrieval\u002Ftools, rubric\u002Fexpected answer, or new eval coverage.\n- **Steady thumbs-up on a topic where eval fails:** Possible eval-setup problem. Review the actual responses before weakening a gate.\n\nProduction signals are not verdicts by themselves. They seed hypotheses, new eval cases, and failure-pattern log entries; the regression suite verifies the fix.\n\n## Human Review Checkpoints\n\nBefore acting on the triage report, review these checkpoints. Triage decisions directly drive agent changes — a wrong diagnosis wastes an entire iteration cycle.\n\n| # | Checkpoint | Why it matters |\n|---|---|---|\n| 1 | **Verify Step 7 root buckets yourself** — For each failure classified as an eval-setup problem, read the agent actual response. Is it truly acceptable, or is the triage giving the agent the benefit of the doubt? | Misclassifying agent-quality problems as eval setup means real problems get ignored. The two-bucket distinction requires judgment, not score-only automation. |\n| 2 | **Confirm systemic pattern diagnoses before applying systemic fixes** — If the report says 80%+ failures share a root bucket\u002Fsubtype, verify by reading the actual responses. Similar symptoms can have different causes. | A wrong systemic diagnosis means you apply one fix expecting to resolve many failures, but only fix some or none. |\n| 3 | **Validate remediation feasibility and priority order** — Can your team actually make the suggested changes? Is the priority order right for your timeline and constraints? | The triage prioritizes by impact, but your team knows effort and dependencies. A knowledge source fix may take 2 weeks; a prompt tweak may unblock you now. |\n| 4 | **Check that proposed fixes will not regress passing scenarios** — Before making changes, consider which currently-passing test cases could be affected. Prompt changes especially have ripple effects. | Fixing 3 failures while introducing 5 new ones is a net loss. Plan to re-run the full suite after any agent configuration change. |\n| 5 | **Validate platform limitation classifications before escalating** — If a failure is classified as a platform limitation, confirm the behavior persists across multiple prompt and config variations before filing with the platform team. | Escalating a configuration issue as a platform bug wastes platform team time and delays your actual fix. |\n| 6 | **Review manifest targets and gates against your actual risk tier** — Does SHIP\u002FITERATE\u002FBLOCK honor hard gates, soft targets, and the five risk factors? | Only your team knows your real risk tolerance. A soft-target miss may be acceptable for an internal helper but a failed hard trust & safety gate blocks deploy. |\n\nInclude this table in the triage report output. Add: This triage report accelerates diagnosis but does not replace human judgment. Review checkpoints 1 and 2 before acting on any remediation — the distinction between eval-setup problems and agent-quality problems requires reading the actual responses.\n\n## Data Retention Warning\n\nCopilot Studio **deletes test run results after 89 days**. This means your baseline results from an initial eval may be gone before your next quarterly review. After every triage cycle:\n\n1. **Export the results CSV** immediately (Test set → Export results)\n2. **Store alongside your triage report** in SharePoint, a repo, or wherever your team keeps versioned artifacts\n3. **Tag with agent version and date** so future comparisons are possible\n\nIf your triage identified a fix-and-rerun cycle, export the pre-fix results *before* applying changes. You need the before\u002Fafter comparison, and Copilot Studio won't keep the \"before\" forever.\n\n## Cross-Reference\n\nThis skill uses `skills\u002Feval-guide\u002Fplaybook.md` as the methodology spine. It also works alongside the **AI Agent Evaluation Scenario Library** (`github.com\u002Fmicrosoft\u002Fai-agent-eval-scenario-library`), which defines supporting scenario patterns and quality dimensions, and the **Triage & Improvement Playbook** (`github.com\u002Fmicrosoft\u002Ftriage-and-improvement-playbook`), which provides supporting diagnostic frameworks for Step 7.\n\n### Related eval skills\n\n| After triage, if you need to... | Use this skill |\n|---|---|\n| Build or expand the eval plan with new scenarios identified during triage | `\u002Feval-suite-planner` |\n| Generate new test cases for expanded or revised scenarios | `\u002Feval-generator` |\n| Get a quick structured report from a new CSV (without interactive triage) | `\u002Feval-result-interpreter` |\n| Answer a methodology question that came up during triage | `\u002Feval-faq` |\n| Walk the customer through the full eval pipeline end-to-end | `\u002Feval-guide` |\n",{"data":40,"body":41},{"name":4,"description":6},{"type":42,"children":43},"root",[44,53,59,95,102,107,265,275,282,288,293,410,415,421,426,438,455,542,548,553,599,604,610,615,732,742,748,775,784,790,802,810,833,841,1011,1021,1027,1136,1148,1154,1159,1223,1230,1238,1325,1333,1472,1480,1503,1509,1514,1565,1570,1576,1581,2620,2626,2631,2664,2670,2675,2718,2724,2736,2779,2784,2790,2795,2955,2960,2966,2978,3011,3024,3030,3071,3077,3184],{"type":45,"tag":46,"props":47,"children":49},"element","h1",{"id":48},"eval-triage-improvement",[50],{"type":51,"value":52},"text","Eval Triage & Improvement",{"type":45,"tag":54,"props":55,"children":56},"p",{},[57],{"type":51,"value":58},"You help users interpret their agent evaluation results and find actionable next steps to improve. Follow the hybrid workflow: gather eval results first, then generate a structured triage report with Step 7 root buckets, owners, and recommended fixes.",{"type":45,"tag":54,"props":60,"children":61},{},[62,64,71,73,79,81,86,88,93],{"type":51,"value":63},"This skill is grounded in ",{"type":45,"tag":65,"props":66,"children":68},"code",{"className":67},[],[69],{"type":51,"value":70},"skills\u002Feval-guide\u002Fplaybook.md",{"type":51,"value":72},", the canonical ",{"type":45,"tag":74,"props":75,"children":76},"strong",{},[77],{"type":51,"value":78},"Practical Guidance on Agent Evaluation: 10-step playbook",{"type":51,"value":80},". It is the deep-dive for ",{"type":45,"tag":74,"props":82,"children":83},{},[84],{"type":51,"value":85},"Step 7 — Iterate to Diagnose Failures",{"type":51,"value":87}," and seeds ",{"type":45,"tag":74,"props":89,"children":90},{},[91],{"type":51,"value":92},"Step 9 — Optimization Loop",{"type":51,"value":94}," for production feedback. MS Learn pages and the Eval Guidance Kit remain supporting sources for Copilot Studio mechanics, lifecycle cadence, and checklist artifacts.",{"type":45,"tag":96,"props":97,"children":99},"h3",{"id":98},"when-to-use-this-skill-vs-eval-result-interpreter",[100],{"type":51,"value":101},"When to use this skill vs. eval-result-interpreter",{"type":45,"tag":54,"props":103,"children":104},{},[105],{"type":51,"value":106},"These two skills share the same triage framework but serve different modes of work:",{"type":45,"tag":108,"props":109,"children":110},"table",{},[111,141],{"type":45,"tag":112,"props":113,"children":114},"thead",{},[115],{"type":45,"tag":116,"props":117,"children":118},"tr",{},[119,131],{"type":45,"tag":120,"props":121,"children":122},"th",{},[123,125,129],{"type":51,"value":124},"Use ",{"type":45,"tag":74,"props":126,"children":127},{},[128],{"type":51,"value":4},{"type":51,"value":130}," when…",{"type":45,"tag":120,"props":132,"children":133},{},[134,135,140],{"type":51,"value":124},{"type":45,"tag":74,"props":136,"children":137},{},[138],{"type":51,"value":139},"eval-result-interpreter",{"type":51,"value":130},{"type":45,"tag":142,"props":143,"children":144},"tbody",{},[145,171,198,225,245],{"type":45,"tag":116,"props":146,"children":147},{},[148,161],{"type":45,"tag":149,"props":150,"children":151},"td",{},[152,154,159],{"type":51,"value":153},"You want ",{"type":45,"tag":74,"props":155,"children":156},{},[157],{"type":51,"value":158},"interactive guidance",{"type":51,"value":160}," walking through diagnosis step by step",{"type":45,"tag":149,"props":162,"children":163},{},[164,166],{"type":51,"value":165},"You have a CSV file or concrete results and want a ",{"type":45,"tag":74,"props":167,"children":168},{},[169],{"type":51,"value":170},"one-shot structured report",{"type":45,"tag":116,"props":172,"children":173},{},[174,186],{"type":45,"tag":149,"props":175,"children":176},{},[177,179,184],{"type":51,"value":178},"You are in an ",{"type":45,"tag":74,"props":180,"children":181},{},[182],{"type":51,"value":183},"ongoing improvement loop",{"type":51,"value":185}," — fixing, re-running, and re-triaging",{"type":45,"tag":149,"props":187,"children":188},{},[189,191,196],{"type":51,"value":190},"This is your ",{"type":45,"tag":74,"props":192,"children":193},{},[194],{"type":51,"value":195},"first look",{"type":51,"value":197}," at results — you need a verdict and top actions fast",{"type":45,"tag":116,"props":199,"children":200},{},[201,213],{"type":45,"tag":149,"props":202,"children":203},{},[204,206,211],{"type":51,"value":205},"You need ",{"type":45,"tag":74,"props":207,"children":208},{},[209],{"type":51,"value":210},"detailed remediation help",{"type":51,"value":212}," for specific eval-set failure patterns (e.g., \"wrong tool fires — now what?\")",{"type":45,"tag":149,"props":214,"children":215},{},[216,218,223],{"type":51,"value":217},"You want a ",{"type":45,"tag":74,"props":219,"children":220},{},[221],{"type":51,"value":222},"customer-deliverable artifact",{"type":51,"value":224}," (the .docx triage report)",{"type":45,"tag":116,"props":226,"children":227},{},[228,240],{"type":45,"tag":149,"props":229,"children":230},{},[231,233,238],{"type":51,"value":232},"You have ",{"type":45,"tag":74,"props":234,"children":235},{},[236],{"type":51,"value":237},"many failures",{"type":51,"value":239}," (15+) and need help prioritizing which to investigate",{"type":45,"tag":149,"props":241,"children":242},{},[243],{"type":51,"value":244},"The eval run is relatively straightforward (\u003C20 failures)",{"type":45,"tag":116,"props":246,"children":247},{},[248,253],{"type":45,"tag":149,"props":249,"children":250},{},[251],{"type":51,"value":252},"You need the playbook worked examples and deeper diagnostic walkthroughs",{"type":45,"tag":149,"props":254,"children":255},{},[256,258,263],{"type":51,"value":257},"You need the ",{"type":45,"tag":74,"props":259,"children":260},{},[261],{"type":51,"value":262},"activity map \u002F result comparison",{"type":51,"value":264}," tool recommendations inline",{"type":45,"tag":54,"props":266,"children":267},{},[268,273],{"type":45,"tag":74,"props":269,"children":270},{},[271],{"type":51,"value":272},"If in doubt:",{"type":51,"value":274}," Start with eval-result-interpreter to get the structured report, then switch to eval-triage-and-improvement if you need interactive help implementing the fixes.",{"type":45,"tag":276,"props":277,"children":279},"h2",{"id":278},"workflow",[280],{"type":51,"value":281},"Workflow",{"type":45,"tag":96,"props":283,"children":285},{"id":284},"step-1-gather-eval-results",[286],{"type":51,"value":287},"Step 1: Gather Eval Results",{"type":45,"tag":54,"props":289,"children":290},{},[291],{"type":51,"value":292},"Ask the user to share:",{"type":45,"tag":294,"props":295,"children":296},"ol",{},[297,308,380,390,400],{"type":45,"tag":298,"props":299,"children":300},"li",{},[301,306],{"type":45,"tag":74,"props":302,"children":303},{},[304],{"type":51,"value":305},"Which eval sets ran",{"type":51,"value":307}," and their pass rates (e.g., \"Faithfulness: 71%, Prompt injection: 95%\")",{"type":45,"tag":298,"props":309,"children":310},{},[311,316,318,324,326,332,334,340,342,348,350,356,357,363,365,371,372,378],{"type":45,"tag":74,"props":312,"children":313},{},[314],{"type":51,"value":315},"Methodology manifest metadata",{"type":51,"value":317}," from the companion ",{"type":45,"tag":65,"props":319,"children":321},{"className":320},[],[322],{"type":51,"value":323},".docx",{"type":51,"value":325}," report or ",{"type":45,"tag":65,"props":327,"children":329},{"className":328},[],[330],{"type":51,"value":331},"stage-N-data.json",{"type":51,"value":333},": ",{"type":45,"tag":65,"props":335,"children":337},{"className":336},[],[338],{"type":51,"value":339},"set_type",{"type":51,"value":341},", ",{"type":45,"tag":65,"props":343,"children":345},{"className":344},[],[346],{"type":51,"value":347},"category",{"type":51,"value":349},"\u002Fcapability dimension, ",{"type":45,"tag":65,"props":351,"children":353},{"className":352},[],[354],{"type":51,"value":355},"method",{"type":51,"value":341},{"type":45,"tag":65,"props":358,"children":360},{"className":359},[],[361],{"type":51,"value":362},"gate",{"type":51,"value":364}," (hard\u002Fsoft), ",{"type":45,"tag":65,"props":366,"children":368},{"className":367},[],[369],{"type":51,"value":370},"target",{"type":51,"value":341},{"type":45,"tag":65,"props":373,"children":375},{"className":374},[],[376],{"type":51,"value":377},"regression_class",{"type":51,"value":379},", human-review flag, and source\u002Fground-truth provenance",{"type":45,"tag":298,"props":381,"children":382},{},[383,388],{"type":45,"tag":74,"props":384,"children":385},{},[386],{"type":51,"value":387},"Specific failing test cases",{"type":51,"value":389}," — the test case ID, sample input, expected value, actual agent response, and eval method assigned in Copilot Studio",{"type":45,"tag":298,"props":391,"children":392},{},[393,398],{"type":45,"tag":74,"props":394,"children":395},{},[396],{"type":51,"value":397},"How many times they've run",{"type":51,"value":399}," — is this the first baseline run (Step 6) or a re-run after fixes?",{"type":45,"tag":298,"props":401,"children":402},{},[403,408],{"type":45,"tag":74,"props":404,"children":405},{},[406],{"type":51,"value":407},"What they've already tried",{"type":51,"value":409}," — any eval, agent, knowledge, or tool changes attempted so far?",{"type":45,"tag":54,"props":411,"children":412},{},[413],{"type":51,"value":414},"If they don't have structured results, help them organize what they have. Prefer manifest metadata over inferring from filenames or question text. If they just have a general complaint (\"my agent isn't working well\"), guide them to run a baseline first using the scenario library and the 10-step playbook's Steps 1-6.",{"type":45,"tag":96,"props":416,"children":418},{"id":417},"step-2-score-interpretation",[419],{"type":51,"value":420},"Step 2: Score Interpretation",{"type":45,"tag":54,"props":422,"children":423},{},[424],{"type":51,"value":425},"Assess readiness from the manifest's Step 4 targets and gates:",{"type":45,"tag":427,"props":428,"children":432},"pre",{"className":429,"code":431,"language":51},[430],"language-text","READINESS ASSESSMENT\n\nAny failed hard gate            → BLOCK (trust & safety hard gates block regardless of aggregate pass rate)\nCapability set below hard floor → ITERATE or BLOCK based on risk tier and capability criticality\nSoft target missed only         → SHIP WITH KNOWN GAPS \u002F ITERATE (tracked, not blocking)\nAll hard gates + targets pass   → SHIP\n",[433],{"type":45,"tag":65,"props":434,"children":436},{"__ignoreMap":435},"",[437],{"type":51,"value":431},{"type":45,"tag":54,"props":439,"children":440},{},[441,446,448,453],{"type":45,"tag":74,"props":442,"children":443},{},[444],{"type":51,"value":445},"Setting thresholds",{"type":51,"value":447}," — don't apply fixed numbers. Use the manifest target first; if missing, derive a provisional target from the agent-level ",{"type":45,"tag":74,"props":449,"children":450},{},[451],{"type":51,"value":452},"risk tier",{"type":51,"value":454}," and flag that the manifest needs updating.",{"type":45,"tag":108,"props":456,"children":457},{},[458,474],{"type":45,"tag":112,"props":459,"children":460},{},[461],{"type":45,"tag":116,"props":462,"children":463},{},[464,469],{"type":45,"tag":120,"props":465,"children":466},{},[467],{"type":51,"value":468},"Factor",{"type":45,"tag":120,"props":470,"children":471},{},[472],{"type":51,"value":473},"Higher Threshold When...",{"type":45,"tag":142,"props":475,"children":476},{},[477,490,503,516,529],{"type":45,"tag":116,"props":478,"children":479},{},[480,485],{"type":45,"tag":149,"props":481,"children":482},{},[483],{"type":51,"value":484},"Criticality of error",{"type":45,"tag":149,"props":486,"children":487},{},[488],{"type":51,"value":489},"Financial loss, safety risk, legal exposure",{"type":45,"tag":116,"props":491,"children":492},{},[493,498],{"type":45,"tag":149,"props":494,"children":495},{},[496],{"type":51,"value":497},"Reach",{"type":45,"tag":149,"props":499,"children":500},{},[501],{"type":51,"value":502},"External customers or large internal population",{"type":45,"tag":116,"props":504,"children":505},{},[506,511],{"type":45,"tag":149,"props":507,"children":508},{},[509],{"type":51,"value":510},"Autonomy \u002F blast radius",{"type":45,"tag":149,"props":512,"children":513},{},[514],{"type":51,"value":515},"Agent can take actions or trigger downstream systems",{"type":45,"tag":116,"props":517,"children":518},{},[519,524],{"type":45,"tag":149,"props":520,"children":521},{},[522],{"type":51,"value":523},"Regulatory exposure",{"type":45,"tag":149,"props":525,"children":526},{},[527],{"type":51,"value":528},"Regulated workflow, audit requirement, compliance obligation",{"type":45,"tag":116,"props":530,"children":531},{},[532,537],{"type":45,"tag":149,"props":533,"children":534},{},[535],{"type":51,"value":536},"Data sensitivity",{"type":45,"tag":149,"props":538,"children":539},{},[540],{"type":51,"value":541},"PII, PHI, confidential, or tenant-sensitive data",{"type":45,"tag":96,"props":543,"children":545},{"id":544},"step-3-pre-triage-infrastructure-check",[546],{"type":51,"value":547},"Step 3: Pre-Triage Infrastructure Check",{"type":45,"tag":54,"props":549,"children":550},{},[551],{"type":51,"value":552},"Before diagnosing individual failures, verify infrastructure was healthy during the eval run:",{"type":45,"tag":554,"props":555,"children":558},"ul",{"className":556},[557],"contains-task-list",[559,572,581,590],{"type":45,"tag":298,"props":560,"children":563},{"className":561},[562],"task-list-item",[564,570],{"type":45,"tag":565,"props":566,"children":569},"input",{"disabled":567,"type":568},true,"checkbox",[],{"type":51,"value":571}," All knowledge sources accessible and fully indexed?",{"type":45,"tag":298,"props":573,"children":575},{"className":574},[562],[576,579],{"type":45,"tag":565,"props":577,"children":578},{"disabled":567,"type":568},[],{"type":51,"value":580}," API backends and connectors returned no errors\u002Ftimeouts?",{"type":45,"tag":298,"props":582,"children":584},{"className":583},[562],[585,588],{"type":45,"tag":565,"props":586,"children":587},{"disabled":567,"type":568},[],{"type":51,"value":589}," Authentication tokens valid throughout the run?",{"type":45,"tag":298,"props":591,"children":593},{"className":592},[562],[594,597],{"type":45,"tag":565,"props":595,"children":596},{"disabled":567,"type":568},[],{"type":51,"value":598}," Correct agent version was published and evaluated?",{"type":45,"tag":54,"props":600,"children":601},{},[602],{"type":51,"value":603},"If any dependency was unhealthy, recommend re-running after fixing infrastructure before triaging.",{"type":45,"tag":96,"props":605,"children":607},{"id":606},"step-4-prioritize-failures",[608],{"type":51,"value":609},"Step 4: Prioritize Failures",{"type":45,"tag":54,"props":611,"children":612},{},[613],{"type":51,"value":614},"If the user has many failures, recommend this triage order:",{"type":45,"tag":108,"props":616,"children":617},{},[618,639],{"type":45,"tag":112,"props":619,"children":620},{},[621],{"type":45,"tag":116,"props":622,"children":623},{},[624,629,634],{"type":45,"tag":120,"props":625,"children":626},{},[627],{"type":51,"value":628},"Priority",{"type":45,"tag":120,"props":630,"children":631},{},[632],{"type":51,"value":633},"Triage First",{"type":45,"tag":120,"props":635,"children":636},{},[637],{"type":51,"value":638},"Rationale",{"type":45,"tag":142,"props":640,"children":641},{},[642,660,678,696,714],{"type":45,"tag":116,"props":643,"children":644},{},[645,650,655],{"type":45,"tag":149,"props":646,"children":647},{},[648],{"type":51,"value":649},"1",{"type":45,"tag":149,"props":651,"children":652},{},[653],{"type":51,"value":654},"Failed hard gates, especially trust & safety sets",{"type":45,"tag":149,"props":656,"children":657},{},[658],{"type":51,"value":659},"Highest consequence; blocks deploy regardless of aggregate score",{"type":45,"tag":116,"props":661,"children":662},{},[663,668,673],{"type":45,"tag":149,"props":664,"children":665},{},[666],{"type":51,"value":667},"2",{"type":45,"tag":149,"props":669,"children":670},{},[671],{"type":51,"value":672},"High-risk capability failures (accuracy, faithfulness, tool use)",{"type":45,"tag":149,"props":674,"children":675},{},[676],{"type":51,"value":677},"Direct impact on agent value; hallucination is a faithfulness\u002Fcapability failure",{"type":45,"tag":116,"props":679,"children":680},{},[681,686,691],{"type":45,"tag":149,"props":682,"children":683},{},[684],{"type":51,"value":685},"3",{"type":45,"tag":149,"props":687,"children":688},{},[689],{"type":51,"value":690},"Lowest-scoring eval set failures",{"type":45,"tag":149,"props":692,"children":693},{},[694],{"type":51,"value":695},"Likely systemic — fixing one pattern resolves multiple",{"type":45,"tag":116,"props":697,"children":698},{},[699,704,709],{"type":45,"tag":149,"props":700,"children":701},{},[702],{"type":51,"value":703},"4",{"type":45,"tag":149,"props":705,"children":706},{},[707],{"type":51,"value":708},"Recurring failures across baseline\u002Fre-runs",{"type":45,"tag":149,"props":710,"children":711},{},[712],{"type":51,"value":713},"Most diagnosable and regression-prone",{"type":45,"tag":116,"props":715,"children":716},{},[717,722,727],{"type":45,"tag":149,"props":718,"children":719},{},[720],{"type":51,"value":721},"5",{"type":45,"tag":149,"props":723,"children":724},{},[725],{"type":51,"value":726},"Soft-target misses",{"type":45,"tag":149,"props":728,"children":729},{},[730],{"type":51,"value":731},"Important but non-blocking unless pattern worsens",{"type":45,"tag":54,"props":733,"children":734},{},[735,740],{"type":45,"tag":74,"props":736,"children":737},{},[738],{"type":51,"value":739},"15+ failures?",{"type":51,"value":741}," Don't triage every one. Review 3-5 from the lowest-scoring eval set. If they share a root bucket\u002Fsubtype pattern, fix that and re-run.",{"type":45,"tag":96,"props":743,"children":745},{"id":744},"step-5-classify-root-cause",[746],{"type":51,"value":747},"Step 5: Classify Root Cause",{"type":45,"tag":54,"props":749,"children":750},{},[751,753,758,759,765,767,773],{"type":51,"value":752},"For each failure, work through the diagnostic questions in order. ",{"type":45,"tag":74,"props":754,"children":755},{},[756],{"type":51,"value":757},"Every failure must end in exactly one Step 7 root bucket",{"type":51,"value":333},{"type":45,"tag":65,"props":760,"children":762},{"className":761},[],[763],{"type":51,"value":764},"eval-setup problem",{"type":51,"value":766}," (response is acceptable; fix the eval) or ",{"type":45,"tag":65,"props":768,"children":770},{"className":769},[],[771],{"type":51,"value":772},"agent-quality problem",{"type":51,"value":774}," (real issue; fix the agent\u002Fplatform and log the pattern).",{"type":45,"tag":427,"props":776,"children":779},{"className":777,"code":778,"language":51},[430],"TRIAGE DECISION TREE (for each failing test case)\n\n1. Is the agent's response actually acceptable, even though it failed?\n   → YES = Eval-setup problem (grader, expected value, rubric, or method is wrong)\n\n2. Is the expected answer still current against the actual source\u002Fground truth in the manifest?\n   → NO = Eval-setup problem (expected answer outdated or source dependency drifted)\n\n3. Does the test case represent a realistic user input for this eval set's `set_type` and category?\n   → NO = Eval-setup problem (unrealistic or mis-scoped test case)\n\n4. Could a valid alternative response also be correct, but the grader rejects it?\n   → YES = Eval-setup problem (rubric\u002Fgrader too rigid)\n\n5. Is the eval method appropriate for what you're testing?\n   → NO = Eval-setup problem (wrong method; update the manifest and Copilot Studio row assignment)\n\nALL PASS → The eval is valid. Classify as **agent-quality problem** and proceed to operational subtype diagnosis:\n\n6. Does the issue come from prompt\u002Ftopic\u002Ftool\u002Fretrieval configuration or stale knowledge?\n   → YES = Agent Configuration \u002F Knowledge Issue (agent-quality problem)\n\n7. Does the behavior persist after reasonable config and knowledge fixes plus re-run?\n   → YES = Platform Limitation (agent-quality problem; log evidence and workaround)\n",[780],{"type":45,"tag":65,"props":781,"children":782},{"__ignoreMap":435},[783],{"type":51,"value":778},{"type":45,"tag":96,"props":785,"children":787},{"id":786},"step-5b-conversation-multi-turn-triage",[788],{"type":51,"value":789},"Step 5b: Conversation (Multi-Turn) Triage",{"type":45,"tag":54,"props":791,"children":792},{},[793,795,800],{"type":51,"value":794},"For conversation eval failures, the standard decision tree still applies but you must first identify the ",{"type":45,"tag":74,"props":796,"children":797},{},[798],{"type":51,"value":799},"critical turn",{"type":51,"value":801}," — the earliest turn where the agent went wrong. Everything after a bad turn is a cascade, not independent failures.",{"type":45,"tag":54,"props":803,"children":804},{},[805],{"type":45,"tag":74,"props":806,"children":807},{},[808],{"type":51,"value":809},"Critical turn identification:",{"type":45,"tag":294,"props":811,"children":812},{},[813,818,823,828],{"type":45,"tag":298,"props":814,"children":815},{},[816],{"type":51,"value":817},"Walk the conversation turn by turn",{"type":45,"tag":298,"props":819,"children":820},{},[821],{"type":51,"value":822},"Find the first turn where the agent response diverges from expected behavior",{"type":45,"tag":298,"props":824,"children":825},{},[826],{"type":51,"value":827},"Classify that turn using the decision tree above",{"type":45,"tag":298,"props":829,"children":830},{},[831],{"type":51,"value":832},"Mark downstream turns as \"cascade — blocked by Turn N fix\"",{"type":45,"tag":54,"props":834,"children":835},{},[836],{"type":45,"tag":74,"props":837,"children":838},{},[839],{"type":51,"value":840},"Conversation-specific failure patterns and remediations:",{"type":45,"tag":108,"props":842,"children":843},{},[844,870],{"type":45,"tag":112,"props":845,"children":846},{},[847],{"type":45,"tag":116,"props":848,"children":849},{},[850,855,860,865],{"type":45,"tag":120,"props":851,"children":852},{},[853],{"type":51,"value":854},"Pattern",{"type":45,"tag":120,"props":856,"children":857},{},[858],{"type":51,"value":859},"How to spot it",{"type":45,"tag":120,"props":861,"children":862},{},[863],{"type":51,"value":864},"Root cause area",{"type":45,"tag":120,"props":866,"children":867},{},[868],{"type":51,"value":869},"Remediation",{"type":45,"tag":142,"props":871,"children":872},{},[873,901,928,955,983],{"type":45,"tag":116,"props":874,"children":875},{},[876,886,891,896],{"type":45,"tag":149,"props":877,"children":878},{},[879,884],{"type":45,"tag":74,"props":880,"children":881},{},[882],{"type":51,"value":883},"Context loss",{"type":51,"value":885}," — Turn 1 fine, Turn 3+ forgets",{"type":45,"tag":149,"props":887,"children":888},{},[889],{"type":51,"value":890},"Agent re-asks or contradicts earlier turns",{"type":45,"tag":149,"props":892,"children":893},{},[894],{"type":51,"value":895},"Agent Config",{"type":45,"tag":149,"props":897,"children":898},{},[899],{"type":51,"value":900},"Review topic management; ensure conversation context is preserved across topic switches",{"type":45,"tag":116,"props":902,"children":903},{},[904,914,919,923],{"type":45,"tag":149,"props":905,"children":906},{},[907,912],{"type":45,"tag":74,"props":908,"children":909},{},[910],{"type":51,"value":911},"State loop",{"type":51,"value":913}," — Agent repeats the same response",{"type":45,"tag":149,"props":915,"children":916},{},[917],{"type":51,"value":918},"Identical or near-identical agent turns in sequence",{"type":45,"tag":149,"props":920,"children":921},{},[922],{"type":51,"value":895},{"type":45,"tag":149,"props":924,"children":925},{},[926],{"type":51,"value":927},"Check topic routing for circular references; add explicit exit conditions",{"type":45,"tag":116,"props":929,"children":930},{},[931,941,946,950],{"type":45,"tag":149,"props":932,"children":933},{},[934,939],{"type":45,"tag":74,"props":935,"children":936},{},[937],{"type":51,"value":938},"Clarification failure",{"type":51,"value":940}," — Agent can't handle follow-ups",{"type":45,"tag":149,"props":942,"children":943},{},[944],{"type":51,"value":945},"Turn 2 fails when user provides clarification or correction",{"type":45,"tag":149,"props":947,"children":948},{},[949],{"type":51,"value":895},{"type":45,"tag":149,"props":951,"children":952},{},[953],{"type":51,"value":954},"Add follow-up handling instructions; check that topics accept partial\u002Fcorrective inputs",{"type":45,"tag":116,"props":956,"children":957},{},[958,968,973,978],{"type":45,"tag":149,"props":959,"children":960},{},[961,966],{"type":45,"tag":74,"props":962,"children":963},{},[964],{"type":51,"value":965},"Last-mile failure",{"type":51,"value":967}," — Understands but can't resolve",{"type":45,"tag":149,"props":969,"children":970},{},[971],{"type":51,"value":972},"Early turns diagnose correctly, final resolution turn fails",{"type":45,"tag":149,"props":974,"children":975},{},[976],{"type":51,"value":977},"Agent Config or Platform",{"type":45,"tag":149,"props":979,"children":980},{},[981],{"type":51,"value":982},"Check action\u002Fconnector configuration; verify the resolution path is wired correctly",{"type":45,"tag":116,"props":984,"children":985},{},[986,996,1001,1006],{"type":45,"tag":149,"props":987,"children":988},{},[989,994],{"type":45,"tag":74,"props":990,"children":991},{},[992],{"type":51,"value":993},"Eval rigidity",{"type":51,"value":995}," — Conversation is acceptable but grader rejects",{"type":45,"tag":149,"props":997,"children":998},{},[999],{"type":51,"value":1000},"Reading the full conversation, the outcome is reasonable",{"type":45,"tag":149,"props":1002,"children":1003},{},[1004],{"type":51,"value":1005},"Eval Setup",{"type":45,"tag":149,"props":1007,"children":1008},{},[1009],{"type":51,"value":1010},"Conversation grading is limited (AI Generated or Approval Rating only); adjust rubric or expected values",{"type":45,"tag":54,"props":1012,"children":1013},{},[1014,1019],{"type":45,"tag":74,"props":1015,"children":1016},{},[1017],{"type":51,"value":1018},"Key difference from single-response triage:",{"type":51,"value":1020}," Do NOT triage each turn independently. Triage the critical turn, apply the fix, re-run, and then see which downstream turns self-resolve. Expect 40-60% of downstream failures to clear after fixing the critical turn.",{"type":45,"tag":96,"props":1022,"children":1024},{"id":1023},"two-root-buckets-operational-subtypes",[1025],{"type":51,"value":1026},"Two Root Buckets + Operational Subtypes",{"type":45,"tag":108,"props":1028,"children":1029},{},[1030,1056],{"type":45,"tag":112,"props":1031,"children":1032},{},[1033],{"type":45,"tag":116,"props":1034,"children":1035},{},[1036,1041,1046,1051],{"type":45,"tag":120,"props":1037,"children":1038},{},[1039],{"type":51,"value":1040},"Step 7 root bucket",{"type":45,"tag":120,"props":1042,"children":1043},{},[1044],{"type":51,"value":1045},"Operational subtype",{"type":45,"tag":120,"props":1047,"children":1048},{},[1049],{"type":51,"value":1050},"Who acts",{"type":45,"tag":120,"props":1052,"children":1053},{},[1054],{"type":51,"value":1055},"What it means",{"type":45,"tag":142,"props":1057,"children":1058},{},[1059,1085,1111],{"type":45,"tag":116,"props":1060,"children":1061},{},[1062,1070,1075,1080],{"type":45,"tag":149,"props":1063,"children":1064},{},[1065],{"type":45,"tag":74,"props":1066,"children":1067},{},[1068],{"type":51,"value":1069},"Eval-setup problem",{"type":45,"tag":149,"props":1071,"children":1072},{},[1073],{"type":51,"value":1074},"Eval setup issue",{"type":45,"tag":149,"props":1076,"children":1077},{},[1078],{"type":51,"value":1079},"Eval author",{"type":45,"tag":149,"props":1081,"children":1082},{},[1083],{"type":51,"value":1084},"The response is acceptable or the eval metadata\u002Frubric\u002Fexpected answer\u002Fmethod is wrong. Fix the eval and manifest.",{"type":45,"tag":116,"props":1086,"children":1087},{},[1088,1096,1101,1106],{"type":45,"tag":149,"props":1089,"children":1090},{},[1091],{"type":45,"tag":74,"props":1092,"children":1093},{},[1094],{"type":51,"value":1095},"Agent-quality problem",{"type":45,"tag":149,"props":1097,"children":1098},{},[1099],{"type":51,"value":1100},"Agent configuration issue",{"type":45,"tag":149,"props":1102,"children":1103},{},[1104],{"type":51,"value":1105},"Agent builder",{"type":45,"tag":149,"props":1107,"children":1108},{},[1109],{"type":51,"value":1110},"The agent genuinely produced a bad response. Fix prompt, topics, tools, retrieval, grounding, or knowledge.",{"type":45,"tag":116,"props":1112,"children":1113},{},[1114,1121,1126,1131],{"type":45,"tag":149,"props":1115,"children":1116},{},[1117],{"type":45,"tag":74,"props":1118,"children":1119},{},[1120],{"type":51,"value":1095},{"type":45,"tag":149,"props":1122,"children":1123},{},[1124],{"type":51,"value":1125},"Platform limitation",{"type":45,"tag":149,"props":1127,"children":1128},{},[1129],{"type":51,"value":1130},"Platform team + agent owner",{"type":45,"tag":149,"props":1132,"children":1133},{},[1134],{"type":51,"value":1135},"The eval caught a real issue caused by platform behavior. Log evidence, workaround if possible, and track the pattern.",{"type":45,"tag":54,"props":1137,"children":1138},{},[1139,1141,1146],{"type":51,"value":1140},"Maintain a ",{"type":45,"tag":74,"props":1142,"children":1143},{},[1144],{"type":51,"value":1145},"failure-pattern log",{"type":51,"value":1147}," for every agent-quality problem: test case, set_type\u002Fcategory, root bucket, subtype, suspected pattern, owner, fix location, verification eval set, and whether it should become a regression case (Step 8).",{"type":45,"tag":96,"props":1149,"children":1151},{"id":1150},"step-6-map-to-remediation",[1152],{"type":51,"value":1153},"Step 6: Map to Remediation",{"type":45,"tag":54,"props":1155,"children":1156},{},[1157],{"type":51,"value":1158},"For detailed remediation steps by Step 7 root bucket, operational subtype, and eval-set failure pattern, read the playbook files:",{"type":45,"tag":554,"props":1160,"children":1161},{},[1162,1178,1193,1208],{"type":45,"tag":298,"props":1163,"children":1164},{},[1165,1170,1172],{"type":45,"tag":74,"props":1166,"children":1167},{},[1168],{"type":51,"value":1169},"Full triage decision tree",{"type":51,"value":1171},": Read ",{"type":45,"tag":65,"props":1173,"children":1175},{"className":1174},[],[1176],{"type":51,"value":1177},"triage-and-improvement-playbook\u002Ftriage-decision-tree.md",{"type":45,"tag":298,"props":1179,"children":1180},{},[1181,1186,1187],{"type":45,"tag":74,"props":1182,"children":1183},{},[1184],{"type":51,"value":1185},"Remediation mapping",{"type":51,"value":1171},{"type":45,"tag":65,"props":1188,"children":1190},{"className":1189},[],[1191],{"type":51,"value":1192},"triage-and-improvement-playbook\u002Fremediation-mapping.md",{"type":45,"tag":298,"props":1194,"children":1195},{},[1196,1201,1202],{"type":45,"tag":74,"props":1197,"children":1198},{},[1199],{"type":51,"value":1200},"Pattern analysis",{"type":51,"value":1171},{"type":45,"tag":65,"props":1203,"children":1205},{"className":1204},[],[1206],{"type":51,"value":1207},"triage-and-improvement-playbook\u002Fpattern-analysis.md",{"type":45,"tag":298,"props":1209,"children":1210},{},[1211,1216,1217],{"type":45,"tag":74,"props":1212,"children":1213},{},[1214],{"type":51,"value":1215},"Worked examples",{"type":51,"value":1171},{"type":45,"tag":65,"props":1218,"children":1220},{"className":1219},[],[1221],{"type":51,"value":1222},"triage-and-improvement-playbook\u002Fworked-examples.md",{"type":45,"tag":1224,"props":1225,"children":1227},"h4",{"id":1226},"quick-remediation-reference",[1228],{"type":51,"value":1229},"Quick Remediation Reference",{"type":45,"tag":54,"props":1231,"children":1232},{},[1233],{"type":45,"tag":74,"props":1234,"children":1235},{},[1236],{"type":51,"value":1237},"Eval-setup fixes:",{"type":45,"tag":108,"props":1239,"children":1240},{},[1241,1257],{"type":45,"tag":112,"props":1242,"children":1243},{},[1244],{"type":45,"tag":116,"props":1245,"children":1246},{},[1247,1252],{"type":45,"tag":120,"props":1248,"children":1249},{},[1250],{"type":51,"value":1251},"Sub-Type",{"type":45,"tag":120,"props":1253,"children":1254},{},[1255],{"type":51,"value":1256},"Fix",{"type":45,"tag":142,"props":1258,"children":1259},{},[1260,1273,1286,1299,1312],{"type":45,"tag":116,"props":1261,"children":1262},{},[1263,1268],{"type":45,"tag":149,"props":1264,"children":1265},{},[1266],{"type":51,"value":1267},"Outdated expected answer",{"type":45,"tag":149,"props":1269,"children":1270},{},[1271],{"type":51,"value":1272},"Update expected value to match current source content",{"type":45,"tag":116,"props":1274,"children":1275},{},[1276,1281],{"type":45,"tag":149,"props":1277,"children":1278},{},[1279],{"type":51,"value":1280},"Overly rigid grader",{"type":45,"tag":149,"props":1282,"children":1283},{},[1284],{"type":51,"value":1285},"Switch to Compare Meaning, or broaden keyword set",{"type":45,"tag":116,"props":1287,"children":1288},{},[1289,1294],{"type":45,"tag":149,"props":1290,"children":1291},{},[1292],{"type":51,"value":1293},"Unrealistic test case",{"type":45,"tag":149,"props":1295,"children":1296},{},[1297],{"type":51,"value":1298},"Rewrite input using actual user language",{"type":45,"tag":116,"props":1300,"children":1301},{},[1302,1307],{"type":45,"tag":149,"props":1303,"children":1304},{},[1305],{"type":51,"value":1306},"Wrong eval method",{"type":45,"tag":149,"props":1308,"children":1309},{},[1310],{"type":51,"value":1311},"Change method to match the eval-set purpose and evidence type",{"type":45,"tag":116,"props":1313,"children":1314},{},[1315,1320],{"type":45,"tag":149,"props":1316,"children":1317},{},[1318],{"type":51,"value":1319},"Grader error\u002Fbias",{"type":45,"tag":149,"props":1321,"children":1322},{},[1323],{"type":51,"value":1324},"Review rubric, add examples, consider deterministic method",{"type":45,"tag":54,"props":1326,"children":1327},{},[1328],{"type":45,"tag":74,"props":1329,"children":1330},{},[1331],{"type":51,"value":1332},"Agent-quality fixes — agent configuration \u002F knowledge \u002F tools:",{"type":45,"tag":108,"props":1334,"children":1335},{},[1336,1352],{"type":45,"tag":112,"props":1337,"children":1338},{},[1339],{"type":45,"tag":116,"props":1340,"children":1341},{},[1342,1347],{"type":45,"tag":120,"props":1343,"children":1344},{},[1345],{"type":51,"value":1346},"Failure pattern",{"type":45,"tag":120,"props":1348,"children":1349},{},[1350],{"type":51,"value":1351},"Common Fix",{"type":45,"tag":142,"props":1353,"children":1354},{},[1355,1368,1381,1394,1407,1420,1433,1446,1459],{"type":45,"tag":116,"props":1356,"children":1357},{},[1358,1363],{"type":45,"tag":149,"props":1359,"children":1360},{},[1361],{"type":51,"value":1362},"Factual accuracy (wrong source)",{"type":45,"tag":149,"props":1364,"children":1365},{},[1366],{"type":51,"value":1367},"Review knowledge source config, verify indexing, check vocabulary match",{"type":45,"tag":116,"props":1369,"children":1370},{},[1371,1376],{"type":45,"tag":149,"props":1372,"children":1373},{},[1374],{"type":51,"value":1375},"Factual accuracy (wrong extraction)",{"type":45,"tag":149,"props":1377,"children":1378},{},[1379],{"type":51,"value":1380},"Add extraction guidance to system prompt",{"type":45,"tag":116,"props":1382,"children":1383},{},[1384,1389],{"type":45,"tag":149,"props":1385,"children":1386},{},[1387],{"type":51,"value":1388},"Hallucination (faithfulness capability failure)",{"type":45,"tag":149,"props":1390,"children":1391},{},[1392],{"type":51,"value":1393},"Improve retrieval\u002Fchunking first; add instruction: \"Only answer from knowledge sources. If unavailable, say so.\"",{"type":45,"tag":116,"props":1395,"children":1396},{},[1397,1402],{"type":45,"tag":149,"props":1398,"children":1399},{},[1400],{"type":51,"value":1401},"Wrong tool fires",{"type":45,"tag":149,"props":1403,"children":1404},{},[1405],{"type":51,"value":1406},"Rewrite tool descriptions to differentiate; add negative examples",{"type":45,"tag":116,"props":1408,"children":1409},{},[1410,1415],{"type":45,"tag":149,"props":1411,"children":1412},{},[1413],{"type":51,"value":1414},"Tool doesn't fire",{"type":45,"tag":149,"props":1416,"children":1417},{},[1418],{"type":51,"value":1419},"Review trigger conditions; check if tool is enabled and accessible",{"type":45,"tag":116,"props":1421,"children":1422},{},[1423,1428],{"type":45,"tag":149,"props":1424,"children":1425},{},[1426],{"type":51,"value":1427},"Wrong topic fires",{"type":45,"tag":149,"props":1429,"children":1430},{},[1431],{"type":51,"value":1432},"Review trigger phrase overlap; adjust priority ordering",{"type":45,"tag":116,"props":1434,"children":1435},{},[1436,1441],{"type":45,"tag":149,"props":1437,"children":1438},{},[1439],{"type":51,"value":1440},"Lacks empathy",{"type":45,"tag":149,"props":1442,"children":1443},{},[1444],{"type":51,"value":1445},"Add context-specific tone instructions to system prompt",{"type":45,"tag":116,"props":1447,"children":1448},{},[1449,1454],{"type":45,"tag":149,"props":1450,"children":1451},{},[1452],{"type":51,"value":1453},"Scope violation",{"type":45,"tag":149,"props":1455,"children":1456},{},[1457],{"type":51,"value":1458},"Add explicit out-of-scope instruction",{"type":45,"tag":116,"props":1460,"children":1461},{},[1462,1467],{"type":45,"tag":149,"props":1463,"children":1464},{},[1465],{"type":51,"value":1466},"PII leakage",{"type":45,"tag":149,"props":1468,"children":1469},{},[1470],{"type":51,"value":1471},"Add PII protection instruction; review authentication scope",{"type":45,"tag":54,"props":1473,"children":1474},{},[1475],{"type":45,"tag":74,"props":1476,"children":1477},{},[1478],{"type":51,"value":1479},"Agent-quality fixes — platform limitation response:",{"type":45,"tag":554,"props":1481,"children":1482},{},[1483,1488,1493,1498],{"type":45,"tag":298,"props":1484,"children":1485},{},[1486],{"type":51,"value":1487},"Document the limitation with evidence",{"type":45,"tag":298,"props":1489,"children":1490},{},[1491],{"type":51,"value":1492},"Implement workaround where possible",{"type":45,"tag":298,"props":1494,"children":1495},{},[1496],{"type":51,"value":1497},"Adjust eval thresholds to account for known platform behavior",{"type":45,"tag":298,"props":1499,"children":1500},{},[1501],{"type":51,"value":1502},"File with platform team with reproduction steps",{"type":45,"tag":96,"props":1504,"children":1506},{"id":1505},"step-7-triage-rationale-teach-the-why",[1507],{"type":51,"value":1508},"Step 7: Triage Rationale (teach the WHY)",{"type":45,"tag":54,"props":1510,"children":1511},{},[1512],{"type":51,"value":1513},"Before generating the report, add rationale that teaches the customer the reasoning behind triage decisions — not just the conclusions. For each of these, use the actual eval data from this triage:",{"type":45,"tag":294,"props":1515,"children":1516},{},[1517,1527,1537,1547],{"type":45,"tag":298,"props":1518,"children":1519},{},[1520,1525],{"type":45,"tag":74,"props":1521,"children":1522},{},[1523],{"type":51,"value":1524},"Why each failure got its root bucket and subtype",{"type":51,"value":1526}," — Walk through the decision tree for at least one example per Step 7 root bucket. E.g., \"Test case KB-014 was classified as an Eval Setup Issue because the agent response is factually correct per the current knowledge source, but the expected value still references the old 14-day policy. The agent is right; the eval is stale.\"",{"type":45,"tag":298,"props":1528,"children":1529},{},[1530,1535],{"type":45,"tag":74,"props":1531,"children":1532},{},[1533],{"type":51,"value":1534},"Why the remediation targets config vs. content vs. eval",{"type":51,"value":1536}," — Explain the logic: \"We recommended updating the knowledge source rather than changing the prompt because the agent retrieval worked correctly — it found the right document — but the document itself contains outdated information. A prompt change would mask the real problem.\"",{"type":45,"tag":298,"props":1538,"children":1539},{},[1540,1545],{"type":45,"tag":74,"props":1541,"children":1542},{},[1543],{"type":51,"value":1544},"Why the priority order is what it is",{"type":51,"value":1546}," — Connect to blast radius and dependency chains: \"Failed hard gates come first because they block deploy and can change downstream behavior. Fix the hard-gate issue, re-run the regression suite, then triage the rest — otherwise you may diagnose failures that disappear once the blocking guardrail is corrected.\"",{"type":45,"tag":298,"props":1548,"children":1549},{},[1550,1555,1557,1563],{"type":45,"tag":74,"props":1551,"children":1552},{},[1553],{"type":51,"value":1554},"What this triage does NOT tell you",{"type":51,"value":1556}," — Name the limits explicitly: \"This triage analyzed ",{"type":45,"tag":1558,"props":1559,"children":1560},"span",{},[1561],{"type":51,"value":1562},"N",{"type":51,"value":1564}," failures from a single eval run. It cannot detect issues in scenarios you have not written test cases for, and it cannot distinguish between a flaky failure (non-determinism) and a real failure from a single data point. If a failure is borderline, re-run before investing in a fix.\"",{"type":45,"tag":54,"props":1566,"children":1567},{},[1568],{"type":51,"value":1569},"Include this rationale in the triage report (see Triage Rationale section in the report template below).",{"type":45,"tag":96,"props":1571,"children":1573},{"id":1572},"step-8-generate-triage-report",[1574],{"type":51,"value":1575},"Step 8: Generate Triage Report",{"type":45,"tag":54,"props":1577,"children":1578},{},[1579],{"type":51,"value":1580},"Output a structured triage report:",{"type":45,"tag":427,"props":1582,"children":1586},{"className":1583,"code":1584,"language":1585,"meta":435,"style":435},"language-markdown shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Triage Report: [Agent Name] — [Date]\n\n## Score Summary\n| Eval Set | Set Type \u002F Category | Pass Rate | Target | Gate | Status |\n|----------|---------------------|-----------|--------|------|--------|\n| ... | capability \u002F faithfulness | ... | ... | hard\u002Fsoft | PASS\u002FBLOCK\u002FITERATE |\n\n## Readiness Assessment\n[SHIP \u002F SHIP WITH KNOWN GAPS \u002F ITERATE \u002F BLOCK]\n[Rationale]\n\n## Failure Analysis\n### Failure 1: [Test Case ID]\n- **Set Type \u002F Category:** capability or trust_safety \u002F ...\n- **Eval-set focus:** ...\n- **Sample Input:** ...\n- **Expected:** ...\n- **Actual:** ...\n- **Root Bucket:** [Eval-setup problem \u002F Agent-quality problem]\n- **Operational Subtype:** [Eval Setup \u002F Agent Config \u002F Knowledge \u002F Platform Limitation]\n- **Diagnosis:** [specific diagnosis]\n- **Owner:** [who needs to act]\n- **Remediation:** [specific action]\n- **Verification:** [how to verify the fix worked]\n\n[Repeat for each triaged failure]\n\n## Triage Rationale\n### Why these root bucket classifications\n[Walk through the decision tree for representative examples — show the reasoning, not just the label]\n\n### Why these remediations\n[Explain the logic connecting root bucket\u002Fsubtype to fix — why this fix and not an alternative]\n\n### Why this priority order\n[Connect priority to blast radius and dependency chains]\n\n### What this triage does NOT tell you\n[Name the limits: coverage gaps, single-run non-determinism, untested scenarios]\n\n## Failure-Pattern Log\n[Summarize recurring Step 7 patterns, owner, fix location, and whether each pattern should be added to the Step 8 regression suite]\n\n## Systemic Patterns\n[If 80%+ of failures share a root bucket\u002Fsubtype\u002Fcategory, call it out]\n\n## Action Items\n| # | Action | Owner | Priority | Verification |\n|---|--------|-------|----------|-------------|\n| 1 | ... | ... | ... | Re-run [eval set] |\n\n## Post-Triage Checklist\n- [ ] All failed hard gates addressed before deploy\n- [ ] Root buckets verified by reading actual responses\n- [ ] Eval-setup fixes applied to expected answers\u002Frubrics\u002Fmethod assignments\u002Fmanifest\n- [ ] Agent-quality patterns logged with owners and fix location\n- [ ] Full Step 8 regression suite re-run after fixes\n- [ ] Platform limitations filed if applicable\n\n## Human Review Required\n[Include human review checkpoints table — see Human Review Checkpoints section below]\n","markdown",[1587],{"type":45,"tag":65,"props":1588,"children":1589},{"__ignoreMap":435},[1590,1623,1632,1646,1711,1720,1780,1788,1801,1810,1826,1834,1847,1861,1892,1918,1943,1968,1993,2018,2044,2070,2096,2122,2148,2156,2165,2173,2186,2199,2208,2216,2229,2238,2246,2259,2268,2276,2289,2298,2306,2319,2328,2336,2349,2358,2366,2379,2432,2441,2491,2499,2512,2525,2538,2551,2564,2577,2590,2598,2611],{"type":45,"tag":1558,"props":1591,"children":1594},{"class":1592,"line":1593},"line",1,[1595,1601,1607,1612,1618],{"type":45,"tag":1558,"props":1596,"children":1598},{"style":1597},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[1599],{"type":51,"value":1600},"# ",{"type":45,"tag":1558,"props":1602,"children":1604},{"style":1603},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[1605],{"type":51,"value":1606},"Triage Report: [Agent Name] — ",{"type":45,"tag":1558,"props":1608,"children":1609},{"style":1597},[1610],{"type":51,"value":1611},"[",{"type":45,"tag":1558,"props":1613,"children":1615},{"style":1614},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1616],{"type":51,"value":1617},"Date",{"type":45,"tag":1558,"props":1619,"children":1620},{"style":1597},[1621],{"type":51,"value":1622},"]\n",{"type":45,"tag":1558,"props":1624,"children":1626},{"class":1592,"line":1625},2,[1627],{"type":45,"tag":1558,"props":1628,"children":1629},{"emptyLinePlaceholder":567},[1630],{"type":51,"value":1631},"\n",{"type":45,"tag":1558,"props":1633,"children":1635},{"class":1592,"line":1634},3,[1636,1641],{"type":45,"tag":1558,"props":1637,"children":1638},{"style":1597},[1639],{"type":51,"value":1640},"## ",{"type":45,"tag":1558,"props":1642,"children":1643},{"style":1603},[1644],{"type":51,"value":1645},"Score Summary\n",{"type":45,"tag":1558,"props":1647,"children":1649},{"class":1592,"line":1648},4,[1650,1655,1661,1665,1670,1674,1679,1683,1688,1692,1697,1701,1706],{"type":45,"tag":1558,"props":1651,"children":1652},{"style":1597},[1653],{"type":51,"value":1654},"|",{"type":45,"tag":1558,"props":1656,"children":1658},{"style":1657},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[1659],{"type":51,"value":1660}," Eval Set ",{"type":45,"tag":1558,"props":1662,"children":1663},{"style":1597},[1664],{"type":51,"value":1654},{"type":45,"tag":1558,"props":1666,"children":1667},{"style":1657},[1668],{"type":51,"value":1669}," Set Type \u002F Category ",{"type":45,"tag":1558,"props":1671,"children":1672},{"style":1597},[1673],{"type":51,"value":1654},{"type":45,"tag":1558,"props":1675,"children":1676},{"style":1657},[1677],{"type":51,"value":1678}," Pass Rate ",{"type":45,"tag":1558,"props":1680,"children":1681},{"style":1597},[1682],{"type":51,"value":1654},{"type":45,"tag":1558,"props":1684,"children":1685},{"style":1657},[1686],{"type":51,"value":1687}," Target ",{"type":45,"tag":1558,"props":1689,"children":1690},{"style":1597},[1691],{"type":51,"value":1654},{"type":45,"tag":1558,"props":1693,"children":1694},{"style":1657},[1695],{"type":51,"value":1696}," Gate ",{"type":45,"tag":1558,"props":1698,"children":1699},{"style":1597},[1700],{"type":51,"value":1654},{"type":45,"tag":1558,"props":1702,"children":1703},{"style":1657},[1704],{"type":51,"value":1705}," Status ",{"type":45,"tag":1558,"props":1707,"children":1708},{"style":1597},[1709],{"type":51,"value":1710},"|\n",{"type":45,"tag":1558,"props":1712,"children":1714},{"class":1592,"line":1713},5,[1715],{"type":45,"tag":1558,"props":1716,"children":1717},{"style":1597},[1718],{"type":51,"value":1719},"|----------|---------------------|-----------|--------|------|--------|\n",{"type":45,"tag":1558,"props":1721,"children":1723},{"class":1592,"line":1722},6,[1724,1728,1733,1737,1742,1746,1750,1754,1758,1762,1767,1771,1776],{"type":45,"tag":1558,"props":1725,"children":1726},{"style":1597},[1727],{"type":51,"value":1654},{"type":45,"tag":1558,"props":1729,"children":1730},{"style":1657},[1731],{"type":51,"value":1732}," ... ",{"type":45,"tag":1558,"props":1734,"children":1735},{"style":1597},[1736],{"type":51,"value":1654},{"type":45,"tag":1558,"props":1738,"children":1739},{"style":1657},[1740],{"type":51,"value":1741}," capability \u002F faithfulness ",{"type":45,"tag":1558,"props":1743,"children":1744},{"style":1597},[1745],{"type":51,"value":1654},{"type":45,"tag":1558,"props":1747,"children":1748},{"style":1657},[1749],{"type":51,"value":1732},{"type":45,"tag":1558,"props":1751,"children":1752},{"style":1597},[1753],{"type":51,"value":1654},{"type":45,"tag":1558,"props":1755,"children":1756},{"style":1657},[1757],{"type":51,"value":1732},{"type":45,"tag":1558,"props":1759,"children":1760},{"style":1597},[1761],{"type":51,"value":1654},{"type":45,"tag":1558,"props":1763,"children":1764},{"style":1657},[1765],{"type":51,"value":1766}," hard\u002Fsoft ",{"type":45,"tag":1558,"props":1768,"children":1769},{"style":1597},[1770],{"type":51,"value":1654},{"type":45,"tag":1558,"props":1772,"children":1773},{"style":1657},[1774],{"type":51,"value":1775}," PASS\u002FBLOCK\u002FITERATE ",{"type":45,"tag":1558,"props":1777,"children":1778},{"style":1597},[1779],{"type":51,"value":1710},{"type":45,"tag":1558,"props":1781,"children":1783},{"class":1592,"line":1782},7,[1784],{"type":45,"tag":1558,"props":1785,"children":1786},{"emptyLinePlaceholder":567},[1787],{"type":51,"value":1631},{"type":45,"tag":1558,"props":1789,"children":1791},{"class":1592,"line":1790},8,[1792,1796],{"type":45,"tag":1558,"props":1793,"children":1794},{"style":1597},[1795],{"type":51,"value":1640},{"type":45,"tag":1558,"props":1797,"children":1798},{"style":1603},[1799],{"type":51,"value":1800},"Readiness Assessment\n",{"type":45,"tag":1558,"props":1802,"children":1804},{"class":1592,"line":1803},9,[1805],{"type":45,"tag":1558,"props":1806,"children":1807},{"style":1657},[1808],{"type":51,"value":1809},"[SHIP \u002F SHIP WITH KNOWN GAPS \u002F ITERATE \u002F BLOCK]\n",{"type":45,"tag":1558,"props":1811,"children":1813},{"class":1592,"line":1812},10,[1814,1818,1822],{"type":45,"tag":1558,"props":1815,"children":1816},{"style":1597},[1817],{"type":51,"value":1611},{"type":45,"tag":1558,"props":1819,"children":1820},{"style":1614},[1821],{"type":51,"value":638},{"type":45,"tag":1558,"props":1823,"children":1824},{"style":1597},[1825],{"type":51,"value":1622},{"type":45,"tag":1558,"props":1827,"children":1829},{"class":1592,"line":1828},11,[1830],{"type":45,"tag":1558,"props":1831,"children":1832},{"emptyLinePlaceholder":567},[1833],{"type":51,"value":1631},{"type":45,"tag":1558,"props":1835,"children":1837},{"class":1592,"line":1836},12,[1838,1842],{"type":45,"tag":1558,"props":1839,"children":1840},{"style":1597},[1841],{"type":51,"value":1640},{"type":45,"tag":1558,"props":1843,"children":1844},{"style":1603},[1845],{"type":51,"value":1846},"Failure Analysis\n",{"type":45,"tag":1558,"props":1848,"children":1850},{"class":1592,"line":1849},13,[1851,1856],{"type":45,"tag":1558,"props":1852,"children":1853},{"style":1597},[1854],{"type":51,"value":1855},"### ",{"type":45,"tag":1558,"props":1857,"children":1858},{"style":1603},[1859],{"type":51,"value":1860},"Failure 1: [Test Case ID]\n",{"type":45,"tag":1558,"props":1862,"children":1864},{"class":1592,"line":1863},14,[1865,1870,1876,1882,1887],{"type":45,"tag":1558,"props":1866,"children":1867},{"style":1597},[1868],{"type":51,"value":1869},"-",{"type":45,"tag":1558,"props":1871,"children":1873},{"style":1872},"--shiki-light:#39ADB5;--shiki-light-font-weight:bold;--shiki-default:#89DDFF;--shiki-default-font-weight:bold;--shiki-dark:#89DDFF;--shiki-dark-font-weight:bold",[1874],{"type":51,"value":1875}," **",{"type":45,"tag":1558,"props":1877,"children":1879},{"style":1878},"--shiki-light:#E53935;--shiki-light-font-weight:bold;--shiki-default:#F07178;--shiki-default-font-weight:bold;--shiki-dark:#F07178;--shiki-dark-font-weight:bold",[1880],{"type":51,"value":1881},"Set Type \u002F Category:",{"type":45,"tag":1558,"props":1883,"children":1884},{"style":1872},[1885],{"type":51,"value":1886},"**",{"type":45,"tag":1558,"props":1888,"children":1889},{"style":1657},[1890],{"type":51,"value":1891}," capability or trust_safety \u002F ...\n",{"type":45,"tag":1558,"props":1893,"children":1895},{"class":1592,"line":1894},15,[1896,1900,1904,1909,1913],{"type":45,"tag":1558,"props":1897,"children":1898},{"style":1597},[1899],{"type":51,"value":1869},{"type":45,"tag":1558,"props":1901,"children":1902},{"style":1872},[1903],{"type":51,"value":1875},{"type":45,"tag":1558,"props":1905,"children":1906},{"style":1878},[1907],{"type":51,"value":1908},"Eval-set focus:",{"type":45,"tag":1558,"props":1910,"children":1911},{"style":1872},[1912],{"type":51,"value":1886},{"type":45,"tag":1558,"props":1914,"children":1915},{"style":1657},[1916],{"type":51,"value":1917}," ...\n",{"type":45,"tag":1558,"props":1919,"children":1921},{"class":1592,"line":1920},16,[1922,1926,1930,1935,1939],{"type":45,"tag":1558,"props":1923,"children":1924},{"style":1597},[1925],{"type":51,"value":1869},{"type":45,"tag":1558,"props":1927,"children":1928},{"style":1872},[1929],{"type":51,"value":1875},{"type":45,"tag":1558,"props":1931,"children":1932},{"style":1878},[1933],{"type":51,"value":1934},"Sample Input:",{"type":45,"tag":1558,"props":1936,"children":1937},{"style":1872},[1938],{"type":51,"value":1886},{"type":45,"tag":1558,"props":1940,"children":1941},{"style":1657},[1942],{"type":51,"value":1917},{"type":45,"tag":1558,"props":1944,"children":1946},{"class":1592,"line":1945},17,[1947,1951,1955,1960,1964],{"type":45,"tag":1558,"props":1948,"children":1949},{"style":1597},[1950],{"type":51,"value":1869},{"type":45,"tag":1558,"props":1952,"children":1953},{"style":1872},[1954],{"type":51,"value":1875},{"type":45,"tag":1558,"props":1956,"children":1957},{"style":1878},[1958],{"type":51,"value":1959},"Expected:",{"type":45,"tag":1558,"props":1961,"children":1962},{"style":1872},[1963],{"type":51,"value":1886},{"type":45,"tag":1558,"props":1965,"children":1966},{"style":1657},[1967],{"type":51,"value":1917},{"type":45,"tag":1558,"props":1969,"children":1971},{"class":1592,"line":1970},18,[1972,1976,1980,1985,1989],{"type":45,"tag":1558,"props":1973,"children":1974},{"style":1597},[1975],{"type":51,"value":1869},{"type":45,"tag":1558,"props":1977,"children":1978},{"style":1872},[1979],{"type":51,"value":1875},{"type":45,"tag":1558,"props":1981,"children":1982},{"style":1878},[1983],{"type":51,"value":1984},"Actual:",{"type":45,"tag":1558,"props":1986,"children":1987},{"style":1872},[1988],{"type":51,"value":1886},{"type":45,"tag":1558,"props":1990,"children":1991},{"style":1657},[1992],{"type":51,"value":1917},{"type":45,"tag":1558,"props":1994,"children":1995},{"class":1592,"line":32},[1996,2000,2004,2009,2013],{"type":45,"tag":1558,"props":1997,"children":1998},{"style":1597},[1999],{"type":51,"value":1869},{"type":45,"tag":1558,"props":2001,"children":2002},{"style":1872},[2003],{"type":51,"value":1875},{"type":45,"tag":1558,"props":2005,"children":2006},{"style":1878},[2007],{"type":51,"value":2008},"Root Bucket:",{"type":45,"tag":1558,"props":2010,"children":2011},{"style":1872},[2012],{"type":51,"value":1886},{"type":45,"tag":1558,"props":2014,"children":2015},{"style":1657},[2016],{"type":51,"value":2017}," [Eval-setup problem \u002F Agent-quality problem]\n",{"type":45,"tag":1558,"props":2019,"children":2021},{"class":1592,"line":2020},20,[2022,2026,2030,2035,2039],{"type":45,"tag":1558,"props":2023,"children":2024},{"style":1597},[2025],{"type":51,"value":1869},{"type":45,"tag":1558,"props":2027,"children":2028},{"style":1872},[2029],{"type":51,"value":1875},{"type":45,"tag":1558,"props":2031,"children":2032},{"style":1878},[2033],{"type":51,"value":2034},"Operational Subtype:",{"type":45,"tag":1558,"props":2036,"children":2037},{"style":1872},[2038],{"type":51,"value":1886},{"type":45,"tag":1558,"props":2040,"children":2041},{"style":1657},[2042],{"type":51,"value":2043}," [Eval Setup \u002F Agent Config \u002F Knowledge \u002F Platform Limitation]\n",{"type":45,"tag":1558,"props":2045,"children":2047},{"class":1592,"line":2046},21,[2048,2052,2056,2061,2065],{"type":45,"tag":1558,"props":2049,"children":2050},{"style":1597},[2051],{"type":51,"value":1869},{"type":45,"tag":1558,"props":2053,"children":2054},{"style":1872},[2055],{"type":51,"value":1875},{"type":45,"tag":1558,"props":2057,"children":2058},{"style":1878},[2059],{"type":51,"value":2060},"Diagnosis:",{"type":45,"tag":1558,"props":2062,"children":2063},{"style":1872},[2064],{"type":51,"value":1886},{"type":45,"tag":1558,"props":2066,"children":2067},{"style":1657},[2068],{"type":51,"value":2069}," [specific diagnosis]\n",{"type":45,"tag":1558,"props":2071,"children":2073},{"class":1592,"line":2072},22,[2074,2078,2082,2087,2091],{"type":45,"tag":1558,"props":2075,"children":2076},{"style":1597},[2077],{"type":51,"value":1869},{"type":45,"tag":1558,"props":2079,"children":2080},{"style":1872},[2081],{"type":51,"value":1875},{"type":45,"tag":1558,"props":2083,"children":2084},{"style":1878},[2085],{"type":51,"value":2086},"Owner:",{"type":45,"tag":1558,"props":2088,"children":2089},{"style":1872},[2090],{"type":51,"value":1886},{"type":45,"tag":1558,"props":2092,"children":2093},{"style":1657},[2094],{"type":51,"value":2095}," [who needs to act]\n",{"type":45,"tag":1558,"props":2097,"children":2099},{"class":1592,"line":2098},23,[2100,2104,2108,2113,2117],{"type":45,"tag":1558,"props":2101,"children":2102},{"style":1597},[2103],{"type":51,"value":1869},{"type":45,"tag":1558,"props":2105,"children":2106},{"style":1872},[2107],{"type":51,"value":1875},{"type":45,"tag":1558,"props":2109,"children":2110},{"style":1878},[2111],{"type":51,"value":2112},"Remediation:",{"type":45,"tag":1558,"props":2114,"children":2115},{"style":1872},[2116],{"type":51,"value":1886},{"type":45,"tag":1558,"props":2118,"children":2119},{"style":1657},[2120],{"type":51,"value":2121}," [specific action]\n",{"type":45,"tag":1558,"props":2123,"children":2125},{"class":1592,"line":2124},24,[2126,2130,2134,2139,2143],{"type":45,"tag":1558,"props":2127,"children":2128},{"style":1597},[2129],{"type":51,"value":1869},{"type":45,"tag":1558,"props":2131,"children":2132},{"style":1872},[2133],{"type":51,"value":1875},{"type":45,"tag":1558,"props":2135,"children":2136},{"style":1878},[2137],{"type":51,"value":2138},"Verification:",{"type":45,"tag":1558,"props":2140,"children":2141},{"style":1872},[2142],{"type":51,"value":1886},{"type":45,"tag":1558,"props":2144,"children":2145},{"style":1657},[2146],{"type":51,"value":2147}," [how to verify the fix worked]\n",{"type":45,"tag":1558,"props":2149,"children":2151},{"class":1592,"line":2150},25,[2152],{"type":45,"tag":1558,"props":2153,"children":2154},{"emptyLinePlaceholder":567},[2155],{"type":51,"value":1631},{"type":45,"tag":1558,"props":2157,"children":2159},{"class":1592,"line":2158},26,[2160],{"type":45,"tag":1558,"props":2161,"children":2162},{"style":1657},[2163],{"type":51,"value":2164},"[Repeat for each triaged failure]\n",{"type":45,"tag":1558,"props":2166,"children":2168},{"class":1592,"line":2167},27,[2169],{"type":45,"tag":1558,"props":2170,"children":2171},{"emptyLinePlaceholder":567},[2172],{"type":51,"value":1631},{"type":45,"tag":1558,"props":2174,"children":2176},{"class":1592,"line":2175},28,[2177,2181],{"type":45,"tag":1558,"props":2178,"children":2179},{"style":1597},[2180],{"type":51,"value":1640},{"type":45,"tag":1558,"props":2182,"children":2183},{"style":1603},[2184],{"type":51,"value":2185},"Triage Rationale\n",{"type":45,"tag":1558,"props":2187,"children":2189},{"class":1592,"line":2188},29,[2190,2194],{"type":45,"tag":1558,"props":2191,"children":2192},{"style":1597},[2193],{"type":51,"value":1855},{"type":45,"tag":1558,"props":2195,"children":2196},{"style":1603},[2197],{"type":51,"value":2198},"Why these root bucket classifications\n",{"type":45,"tag":1558,"props":2200,"children":2202},{"class":1592,"line":2201},30,[2203],{"type":45,"tag":1558,"props":2204,"children":2205},{"style":1657},[2206],{"type":51,"value":2207},"[Walk through the decision tree for representative examples — show the reasoning, not just the label]\n",{"type":45,"tag":1558,"props":2209,"children":2211},{"class":1592,"line":2210},31,[2212],{"type":45,"tag":1558,"props":2213,"children":2214},{"emptyLinePlaceholder":567},[2215],{"type":51,"value":1631},{"type":45,"tag":1558,"props":2217,"children":2219},{"class":1592,"line":2218},32,[2220,2224],{"type":45,"tag":1558,"props":2221,"children":2222},{"style":1597},[2223],{"type":51,"value":1855},{"type":45,"tag":1558,"props":2225,"children":2226},{"style":1603},[2227],{"type":51,"value":2228},"Why these remediations\n",{"type":45,"tag":1558,"props":2230,"children":2232},{"class":1592,"line":2231},33,[2233],{"type":45,"tag":1558,"props":2234,"children":2235},{"style":1657},[2236],{"type":51,"value":2237},"[Explain the logic connecting root bucket\u002Fsubtype to fix — why this fix and not an alternative]\n",{"type":45,"tag":1558,"props":2239,"children":2241},{"class":1592,"line":2240},34,[2242],{"type":45,"tag":1558,"props":2243,"children":2244},{"emptyLinePlaceholder":567},[2245],{"type":51,"value":1631},{"type":45,"tag":1558,"props":2247,"children":2249},{"class":1592,"line":2248},35,[2250,2254],{"type":45,"tag":1558,"props":2251,"children":2252},{"style":1597},[2253],{"type":51,"value":1855},{"type":45,"tag":1558,"props":2255,"children":2256},{"style":1603},[2257],{"type":51,"value":2258},"Why this priority order\n",{"type":45,"tag":1558,"props":2260,"children":2262},{"class":1592,"line":2261},36,[2263],{"type":45,"tag":1558,"props":2264,"children":2265},{"style":1657},[2266],{"type":51,"value":2267},"[Connect priority to blast radius and dependency chains]\n",{"type":45,"tag":1558,"props":2269,"children":2271},{"class":1592,"line":2270},37,[2272],{"type":45,"tag":1558,"props":2273,"children":2274},{"emptyLinePlaceholder":567},[2275],{"type":51,"value":1631},{"type":45,"tag":1558,"props":2277,"children":2279},{"class":1592,"line":2278},38,[2280,2284],{"type":45,"tag":1558,"props":2281,"children":2282},{"style":1597},[2283],{"type":51,"value":1855},{"type":45,"tag":1558,"props":2285,"children":2286},{"style":1603},[2287],{"type":51,"value":2288},"What this triage does NOT tell you\n",{"type":45,"tag":1558,"props":2290,"children":2292},{"class":1592,"line":2291},39,[2293],{"type":45,"tag":1558,"props":2294,"children":2295},{"style":1657},[2296],{"type":51,"value":2297},"[Name the limits: coverage gaps, single-run non-determinism, untested scenarios]\n",{"type":45,"tag":1558,"props":2299,"children":2301},{"class":1592,"line":2300},40,[2302],{"type":45,"tag":1558,"props":2303,"children":2304},{"emptyLinePlaceholder":567},[2305],{"type":51,"value":1631},{"type":45,"tag":1558,"props":2307,"children":2309},{"class":1592,"line":2308},41,[2310,2314],{"type":45,"tag":1558,"props":2311,"children":2312},{"style":1597},[2313],{"type":51,"value":1640},{"type":45,"tag":1558,"props":2315,"children":2316},{"style":1603},[2317],{"type":51,"value":2318},"Failure-Pattern Log\n",{"type":45,"tag":1558,"props":2320,"children":2322},{"class":1592,"line":2321},42,[2323],{"type":45,"tag":1558,"props":2324,"children":2325},{"style":1657},[2326],{"type":51,"value":2327},"[Summarize recurring Step 7 patterns, owner, fix location, and whether each pattern should be added to the Step 8 regression suite]\n",{"type":45,"tag":1558,"props":2329,"children":2331},{"class":1592,"line":2330},43,[2332],{"type":45,"tag":1558,"props":2333,"children":2334},{"emptyLinePlaceholder":567},[2335],{"type":51,"value":1631},{"type":45,"tag":1558,"props":2337,"children":2339},{"class":1592,"line":2338},44,[2340,2344],{"type":45,"tag":1558,"props":2341,"children":2342},{"style":1597},[2343],{"type":51,"value":1640},{"type":45,"tag":1558,"props":2345,"children":2346},{"style":1603},[2347],{"type":51,"value":2348},"Systemic Patterns\n",{"type":45,"tag":1558,"props":2350,"children":2352},{"class":1592,"line":2351},45,[2353],{"type":45,"tag":1558,"props":2354,"children":2355},{"style":1657},[2356],{"type":51,"value":2357},"[If 80%+ of failures share a root bucket\u002Fsubtype\u002Fcategory, call it out]\n",{"type":45,"tag":1558,"props":2359,"children":2361},{"class":1592,"line":2360},46,[2362],{"type":45,"tag":1558,"props":2363,"children":2364},{"emptyLinePlaceholder":567},[2365],{"type":51,"value":1631},{"type":45,"tag":1558,"props":2367,"children":2369},{"class":1592,"line":2368},47,[2370,2374],{"type":45,"tag":1558,"props":2371,"children":2372},{"style":1597},[2373],{"type":51,"value":1640},{"type":45,"tag":1558,"props":2375,"children":2376},{"style":1603},[2377],{"type":51,"value":2378},"Action Items\n",{"type":45,"tag":1558,"props":2380,"children":2382},{"class":1592,"line":2381},48,[2383,2387,2392,2396,2401,2405,2410,2414,2419,2423,2428],{"type":45,"tag":1558,"props":2384,"children":2385},{"style":1597},[2386],{"type":51,"value":1654},{"type":45,"tag":1558,"props":2388,"children":2389},{"style":1657},[2390],{"type":51,"value":2391}," # ",{"type":45,"tag":1558,"props":2393,"children":2394},{"style":1597},[2395],{"type":51,"value":1654},{"type":45,"tag":1558,"props":2397,"children":2398},{"style":1657},[2399],{"type":51,"value":2400}," Action ",{"type":45,"tag":1558,"props":2402,"children":2403},{"style":1597},[2404],{"type":51,"value":1654},{"type":45,"tag":1558,"props":2406,"children":2407},{"style":1657},[2408],{"type":51,"value":2409}," Owner ",{"type":45,"tag":1558,"props":2411,"children":2412},{"style":1597},[2413],{"type":51,"value":1654},{"type":45,"tag":1558,"props":2415,"children":2416},{"style":1657},[2417],{"type":51,"value":2418}," Priority ",{"type":45,"tag":1558,"props":2420,"children":2421},{"style":1597},[2422],{"type":51,"value":1654},{"type":45,"tag":1558,"props":2424,"children":2425},{"style":1657},[2426],{"type":51,"value":2427}," Verification ",{"type":45,"tag":1558,"props":2429,"children":2430},{"style":1597},[2431],{"type":51,"value":1710},{"type":45,"tag":1558,"props":2433,"children":2435},{"class":1592,"line":2434},49,[2436],{"type":45,"tag":1558,"props":2437,"children":2438},{"style":1597},[2439],{"type":51,"value":2440},"|---|--------|-------|----------|-------------|\n",{"type":45,"tag":1558,"props":2442,"children":2444},{"class":1592,"line":2443},50,[2445,2449,2454,2458,2462,2466,2470,2474,2478,2482,2487],{"type":45,"tag":1558,"props":2446,"children":2447},{"style":1597},[2448],{"type":51,"value":1654},{"type":45,"tag":1558,"props":2450,"children":2451},{"style":1657},[2452],{"type":51,"value":2453}," 1 ",{"type":45,"tag":1558,"props":2455,"children":2456},{"style":1597},[2457],{"type":51,"value":1654},{"type":45,"tag":1558,"props":2459,"children":2460},{"style":1657},[2461],{"type":51,"value":1732},{"type":45,"tag":1558,"props":2463,"children":2464},{"style":1597},[2465],{"type":51,"value":1654},{"type":45,"tag":1558,"props":2467,"children":2468},{"style":1657},[2469],{"type":51,"value":1732},{"type":45,"tag":1558,"props":2471,"children":2472},{"style":1597},[2473],{"type":51,"value":1654},{"type":45,"tag":1558,"props":2475,"children":2476},{"style":1657},[2477],{"type":51,"value":1732},{"type":45,"tag":1558,"props":2479,"children":2480},{"style":1597},[2481],{"type":51,"value":1654},{"type":45,"tag":1558,"props":2483,"children":2484},{"style":1657},[2485],{"type":51,"value":2486}," Re-run [eval set] ",{"type":45,"tag":1558,"props":2488,"children":2489},{"style":1597},[2490],{"type":51,"value":1710},{"type":45,"tag":1558,"props":2492,"children":2494},{"class":1592,"line":2493},51,[2495],{"type":45,"tag":1558,"props":2496,"children":2497},{"emptyLinePlaceholder":567},[2498],{"type":51,"value":1631},{"type":45,"tag":1558,"props":2500,"children":2502},{"class":1592,"line":2501},52,[2503,2507],{"type":45,"tag":1558,"props":2504,"children":2505},{"style":1597},[2506],{"type":51,"value":1640},{"type":45,"tag":1558,"props":2508,"children":2509},{"style":1603},[2510],{"type":51,"value":2511},"Post-Triage Checklist\n",{"type":45,"tag":1558,"props":2513,"children":2515},{"class":1592,"line":2514},53,[2516,2520],{"type":45,"tag":1558,"props":2517,"children":2518},{"style":1597},[2519],{"type":51,"value":1869},{"type":45,"tag":1558,"props":2521,"children":2522},{"style":1657},[2523],{"type":51,"value":2524}," [ ] All failed hard gates addressed before deploy\n",{"type":45,"tag":1558,"props":2526,"children":2528},{"class":1592,"line":2527},54,[2529,2533],{"type":45,"tag":1558,"props":2530,"children":2531},{"style":1597},[2532],{"type":51,"value":1869},{"type":45,"tag":1558,"props":2534,"children":2535},{"style":1657},[2536],{"type":51,"value":2537}," [ ] Root buckets verified by reading actual responses\n",{"type":45,"tag":1558,"props":2539,"children":2541},{"class":1592,"line":2540},55,[2542,2546],{"type":45,"tag":1558,"props":2543,"children":2544},{"style":1597},[2545],{"type":51,"value":1869},{"type":45,"tag":1558,"props":2547,"children":2548},{"style":1657},[2549],{"type":51,"value":2550}," [ ] Eval-setup fixes applied to expected answers\u002Frubrics\u002Fmethod assignments\u002Fmanifest\n",{"type":45,"tag":1558,"props":2552,"children":2554},{"class":1592,"line":2553},56,[2555,2559],{"type":45,"tag":1558,"props":2556,"children":2557},{"style":1597},[2558],{"type":51,"value":1869},{"type":45,"tag":1558,"props":2560,"children":2561},{"style":1657},[2562],{"type":51,"value":2563}," [ ] Agent-quality patterns logged with owners and fix location\n",{"type":45,"tag":1558,"props":2565,"children":2567},{"class":1592,"line":2566},57,[2568,2572],{"type":45,"tag":1558,"props":2569,"children":2570},{"style":1597},[2571],{"type":51,"value":1869},{"type":45,"tag":1558,"props":2573,"children":2574},{"style":1657},[2575],{"type":51,"value":2576}," [ ] Full Step 8 regression suite re-run after fixes\n",{"type":45,"tag":1558,"props":2578,"children":2580},{"class":1592,"line":2579},58,[2581,2585],{"type":45,"tag":1558,"props":2582,"children":2583},{"style":1597},[2584],{"type":51,"value":1869},{"type":45,"tag":1558,"props":2586,"children":2587},{"style":1657},[2588],{"type":51,"value":2589}," [ ] Platform limitations filed if applicable\n",{"type":45,"tag":1558,"props":2591,"children":2593},{"class":1592,"line":2592},59,[2594],{"type":45,"tag":1558,"props":2595,"children":2596},{"emptyLinePlaceholder":567},[2597],{"type":51,"value":1631},{"type":45,"tag":1558,"props":2599,"children":2601},{"class":1592,"line":2600},60,[2602,2606],{"type":45,"tag":1558,"props":2603,"children":2604},{"style":1597},[2605],{"type":51,"value":1640},{"type":45,"tag":1558,"props":2607,"children":2608},{"style":1603},[2609],{"type":51,"value":2610},"Human Review Required\n",{"type":45,"tag":1558,"props":2612,"children":2614},{"class":1592,"line":2613},61,[2615],{"type":45,"tag":1558,"props":2616,"children":2617},{"style":1657},[2618],{"type":51,"value":2619},"[Include human review checkpoints table — see Human Review Checkpoints section below]\n",{"type":45,"tag":96,"props":2621,"children":2623},{"id":2622},"post-triage-verification",[2624],{"type":51,"value":2625},"Post-Triage Verification",{"type":45,"tag":54,"props":2627,"children":2628},{},[2629],{"type":51,"value":2630},"After fixes are applied:",{"type":45,"tag":554,"props":2632,"children":2633},{},[2634,2644,2654],{"type":45,"tag":298,"props":2635,"children":2636},{},[2637,2642],{"type":45,"tag":74,"props":2638,"children":2639},{},[2640],{"type":51,"value":2641},"Scores flat after fix?",{"type":51,"value":2643}," → Wrong root bucket\u002Fsubtype, re-triage",{"type":45,"tag":298,"props":2645,"children":2646},{},[2647,2652],{"type":45,"tag":74,"props":2648,"children":2649},{},[2650],{"type":51,"value":2651},"One score up, another down?",{"type":51,"value":2653}," → Instruction conflict — the fix improved one behavior but degraded another",{"type":45,"tag":298,"props":2655,"children":2656},{},[2657,2662],{"type":45,"tag":74,"props":2658,"children":2659},{},[2660],{"type":51,"value":2661},"80%+ of failures share a root bucket\u002Fsubtype?",{"type":51,"value":2663}," → Systemic issue — fix the category, not individual test cases",{"type":45,"tag":276,"props":2665,"children":2667},{"id":2666},"non-determinism-handling",[2668],{"type":51,"value":2669},"Non-Determinism Handling",{"type":45,"tag":54,"props":2671,"children":2672},{},[2673],{"type":51,"value":2674},"LLM-based agents and graders produce variable outputs:",{"type":45,"tag":554,"props":2676,"children":2677},{},[2678,2688,2698,2708],{"type":45,"tag":298,"props":2679,"children":2680},{},[2681,2686],{"type":45,"tag":74,"props":2682,"children":2683},{},[2684],{"type":51,"value":2685},"Establish baselines:",{"type":51,"value":2687}," Run 3+ times before treating any score as the Step 6 baseline. Use the average and record agent version + timestamp.",{"type":45,"tag":298,"props":2689,"children":2690},{},[2691,2696],{"type":45,"tag":74,"props":2692,"children":2693},{},[2694],{"type":51,"value":2695},"Normal variance:",{"type":51,"value":2697}," +\u002F-5% between runs is expected. Investigate if >10%.",{"type":45,"tag":298,"props":2699,"children":2700},{},[2701,2706],{"type":45,"tag":74,"props":2702,"children":2703},{},[2704],{"type":51,"value":2705},"Flaky test cases",{"type":51,"value":2707}," (pass sometimes, fail others): Agent may produce two valid responses but eval is too rigid. Investigate whether to broaden the expected value.",{"type":45,"tag":298,"props":2709,"children":2710},{},[2711,2716],{"type":45,"tag":74,"props":2712,"children":2713},{},[2714],{"type":51,"value":2715},"Small eval sets (\u003C30 test cases):",{"type":51,"value":2717}," A single test case flip changes the score by 3%+. Don't over-interpret.",{"type":45,"tag":276,"props":2719,"children":2721},{"id":2720},"step-9-optimization-loop-production-signals",[2722],{"type":51,"value":2723},"Step 9 Optimization Loop: Production Signals",{"type":45,"tag":54,"props":2725,"children":2726},{},[2727,2729,2734],{"type":51,"value":2728},"If the agent is deployed (even in preview), treat production feedback as the Step 9 loop: ",{"type":45,"tag":74,"props":2730,"children":2731},{},[2732],{"type":51,"value":2733},"collect signals → cluster → decide fix location → ship → re-evaluate against the Step 8 regression suite",{"type":51,"value":2735},". Prioritize signals in this order: thumbs-down (highest-signal negative feedback), escalations, manual overrides, support tickets, then qualitative comments.",{"type":45,"tag":554,"props":2737,"children":2738},{},[2739,2749,2759,2769],{"type":45,"tag":298,"props":2740,"children":2741},{},[2742,2747],{"type":45,"tag":74,"props":2743,"children":2744},{},[2745],{"type":51,"value":2746},"High thumbs-down on a topic where eval passes:",{"type":51,"value":2748}," Coverage gap. Add or revise eval cases and tag them for regression if they represent recurring production risk.",{"type":45,"tag":298,"props":2750,"children":2751},{},[2752,2757],{"type":45,"tag":74,"props":2753,"children":2754},{},[2755],{"type":51,"value":2756},"Thumbs-down clustering after a config change:",{"type":51,"value":2758}," Possible regression. Re-run the Step 8 regression suite and add a case if the suite missed it.",{"type":45,"tag":298,"props":2760,"children":2761},{},[2762,2767],{"type":45,"tag":74,"props":2763,"children":2764},{},[2765],{"type":51,"value":2766},"Escalations\u002Fmanual overrides:",{"type":51,"value":2768}," Agent-quality problem until proven otherwise; cluster by capability or trust & safety category and choose the fix location: agent config\u002Fretrieval\u002Ftools, rubric\u002Fexpected answer, or new eval coverage.",{"type":45,"tag":298,"props":2770,"children":2771},{},[2772,2777],{"type":45,"tag":74,"props":2773,"children":2774},{},[2775],{"type":51,"value":2776},"Steady thumbs-up on a topic where eval fails:",{"type":51,"value":2778}," Possible eval-setup problem. Review the actual responses before weakening a gate.",{"type":45,"tag":54,"props":2780,"children":2781},{},[2782],{"type":51,"value":2783},"Production signals are not verdicts by themselves. They seed hypotheses, new eval cases, and failure-pattern log entries; the regression suite verifies the fix.",{"type":45,"tag":276,"props":2785,"children":2787},{"id":2786},"human-review-checkpoints",[2788],{"type":51,"value":2789},"Human Review Checkpoints",{"type":45,"tag":54,"props":2791,"children":2792},{},[2793],{"type":51,"value":2794},"Before acting on the triage report, review these checkpoints. Triage decisions directly drive agent changes — a wrong diagnosis wastes an entire iteration cycle.",{"type":45,"tag":108,"props":2796,"children":2797},{},[2798,2819],{"type":45,"tag":112,"props":2799,"children":2800},{},[2801],{"type":45,"tag":116,"props":2802,"children":2803},{},[2804,2809,2814],{"type":45,"tag":120,"props":2805,"children":2806},{},[2807],{"type":51,"value":2808},"#",{"type":45,"tag":120,"props":2810,"children":2811},{},[2812],{"type":51,"value":2813},"Checkpoint",{"type":45,"tag":120,"props":2815,"children":2816},{},[2817],{"type":51,"value":2818},"Why it matters",{"type":45,"tag":142,"props":2820,"children":2821},{},[2822,2844,2866,2888,2910,2932],{"type":45,"tag":116,"props":2823,"children":2824},{},[2825,2829,2839],{"type":45,"tag":149,"props":2826,"children":2827},{},[2828],{"type":51,"value":649},{"type":45,"tag":149,"props":2830,"children":2831},{},[2832,2837],{"type":45,"tag":74,"props":2833,"children":2834},{},[2835],{"type":51,"value":2836},"Verify Step 7 root buckets yourself",{"type":51,"value":2838}," — For each failure classified as an eval-setup problem, read the agent actual response. Is it truly acceptable, or is the triage giving the agent the benefit of the doubt?",{"type":45,"tag":149,"props":2840,"children":2841},{},[2842],{"type":51,"value":2843},"Misclassifying agent-quality problems as eval setup means real problems get ignored. The two-bucket distinction requires judgment, not score-only automation.",{"type":45,"tag":116,"props":2845,"children":2846},{},[2847,2851,2861],{"type":45,"tag":149,"props":2848,"children":2849},{},[2850],{"type":51,"value":667},{"type":45,"tag":149,"props":2852,"children":2853},{},[2854,2859],{"type":45,"tag":74,"props":2855,"children":2856},{},[2857],{"type":51,"value":2858},"Confirm systemic pattern diagnoses before applying systemic fixes",{"type":51,"value":2860}," — If the report says 80%+ failures share a root bucket\u002Fsubtype, verify by reading the actual responses. Similar symptoms can have different causes.",{"type":45,"tag":149,"props":2862,"children":2863},{},[2864],{"type":51,"value":2865},"A wrong systemic diagnosis means you apply one fix expecting to resolve many failures, but only fix some or none.",{"type":45,"tag":116,"props":2867,"children":2868},{},[2869,2873,2883],{"type":45,"tag":149,"props":2870,"children":2871},{},[2872],{"type":51,"value":685},{"type":45,"tag":149,"props":2874,"children":2875},{},[2876,2881],{"type":45,"tag":74,"props":2877,"children":2878},{},[2879],{"type":51,"value":2880},"Validate remediation feasibility and priority order",{"type":51,"value":2882}," — Can your team actually make the suggested changes? Is the priority order right for your timeline and constraints?",{"type":45,"tag":149,"props":2884,"children":2885},{},[2886],{"type":51,"value":2887},"The triage prioritizes by impact, but your team knows effort and dependencies. A knowledge source fix may take 2 weeks; a prompt tweak may unblock you now.",{"type":45,"tag":116,"props":2889,"children":2890},{},[2891,2895,2905],{"type":45,"tag":149,"props":2892,"children":2893},{},[2894],{"type":51,"value":703},{"type":45,"tag":149,"props":2896,"children":2897},{},[2898,2903],{"type":45,"tag":74,"props":2899,"children":2900},{},[2901],{"type":51,"value":2902},"Check that proposed fixes will not regress passing scenarios",{"type":51,"value":2904}," — Before making changes, consider which currently-passing test cases could be affected. Prompt changes especially have ripple effects.",{"type":45,"tag":149,"props":2906,"children":2907},{},[2908],{"type":51,"value":2909},"Fixing 3 failures while introducing 5 new ones is a net loss. Plan to re-run the full suite after any agent configuration change.",{"type":45,"tag":116,"props":2911,"children":2912},{},[2913,2917,2927],{"type":45,"tag":149,"props":2914,"children":2915},{},[2916],{"type":51,"value":721},{"type":45,"tag":149,"props":2918,"children":2919},{},[2920,2925],{"type":45,"tag":74,"props":2921,"children":2922},{},[2923],{"type":51,"value":2924},"Validate platform limitation classifications before escalating",{"type":51,"value":2926}," — If a failure is classified as a platform limitation, confirm the behavior persists across multiple prompt and config variations before filing with the platform team.",{"type":45,"tag":149,"props":2928,"children":2929},{},[2930],{"type":51,"value":2931},"Escalating a configuration issue as a platform bug wastes platform team time and delays your actual fix.",{"type":45,"tag":116,"props":2933,"children":2934},{},[2935,2940,2950],{"type":45,"tag":149,"props":2936,"children":2937},{},[2938],{"type":51,"value":2939},"6",{"type":45,"tag":149,"props":2941,"children":2942},{},[2943,2948],{"type":45,"tag":74,"props":2944,"children":2945},{},[2946],{"type":51,"value":2947},"Review manifest targets and gates against your actual risk tier",{"type":51,"value":2949}," — Does SHIP\u002FITERATE\u002FBLOCK honor hard gates, soft targets, and the five risk factors?",{"type":45,"tag":149,"props":2951,"children":2952},{},[2953],{"type":51,"value":2954},"Only your team knows your real risk tolerance. A soft-target miss may be acceptable for an internal helper but a failed hard trust & safety gate blocks deploy.",{"type":45,"tag":54,"props":2956,"children":2957},{},[2958],{"type":51,"value":2959},"Include this table in the triage report output. Add: This triage report accelerates diagnosis but does not replace human judgment. Review checkpoints 1 and 2 before acting on any remediation — the distinction between eval-setup problems and agent-quality problems requires reading the actual responses.",{"type":45,"tag":276,"props":2961,"children":2963},{"id":2962},"data-retention-warning",[2964],{"type":51,"value":2965},"Data Retention Warning",{"type":45,"tag":54,"props":2967,"children":2968},{},[2969,2971,2976],{"type":51,"value":2970},"Copilot Studio ",{"type":45,"tag":74,"props":2972,"children":2973},{},[2974],{"type":51,"value":2975},"deletes test run results after 89 days",{"type":51,"value":2977},". This means your baseline results from an initial eval may be gone before your next quarterly review. After every triage cycle:",{"type":45,"tag":294,"props":2979,"children":2980},{},[2981,2991,3001],{"type":45,"tag":298,"props":2982,"children":2983},{},[2984,2989],{"type":45,"tag":74,"props":2985,"children":2986},{},[2987],{"type":51,"value":2988},"Export the results CSV",{"type":51,"value":2990}," immediately (Test set → Export results)",{"type":45,"tag":298,"props":2992,"children":2993},{},[2994,2999],{"type":45,"tag":74,"props":2995,"children":2996},{},[2997],{"type":51,"value":2998},"Store alongside your triage report",{"type":51,"value":3000}," in SharePoint, a repo, or wherever your team keeps versioned artifacts",{"type":45,"tag":298,"props":3002,"children":3003},{},[3004,3009],{"type":45,"tag":74,"props":3005,"children":3006},{},[3007],{"type":51,"value":3008},"Tag with agent version and date",{"type":51,"value":3010}," so future comparisons are possible",{"type":45,"tag":54,"props":3012,"children":3013},{},[3014,3016,3022],{"type":51,"value":3015},"If your triage identified a fix-and-rerun cycle, export the pre-fix results ",{"type":45,"tag":3017,"props":3018,"children":3019},"em",{},[3020],{"type":51,"value":3021},"before",{"type":51,"value":3023}," applying changes. You need the before\u002Fafter comparison, and Copilot Studio won't keep the \"before\" forever.",{"type":45,"tag":276,"props":3025,"children":3027},{"id":3026},"cross-reference",[3028],{"type":51,"value":3029},"Cross-Reference",{"type":45,"tag":54,"props":3031,"children":3032},{},[3033,3035,3040,3042,3047,3049,3055,3057,3062,3063,3069],{"type":51,"value":3034},"This skill uses ",{"type":45,"tag":65,"props":3036,"children":3038},{"className":3037},[],[3039],{"type":51,"value":70},{"type":51,"value":3041}," as the methodology spine. It also works alongside the ",{"type":45,"tag":74,"props":3043,"children":3044},{},[3045],{"type":51,"value":3046},"AI Agent Evaluation Scenario Library",{"type":51,"value":3048}," (",{"type":45,"tag":65,"props":3050,"children":3052},{"className":3051},[],[3053],{"type":51,"value":3054},"github.com\u002Fmicrosoft\u002Fai-agent-eval-scenario-library",{"type":51,"value":3056},"), which defines supporting scenario patterns and quality dimensions, and the ",{"type":45,"tag":74,"props":3058,"children":3059},{},[3060],{"type":51,"value":3061},"Triage & Improvement Playbook",{"type":51,"value":3048},{"type":45,"tag":65,"props":3064,"children":3066},{"className":3065},[],[3067],{"type":51,"value":3068},"github.com\u002Fmicrosoft\u002Ftriage-and-improvement-playbook",{"type":51,"value":3070},"), which provides supporting diagnostic frameworks for Step 7.",{"type":45,"tag":96,"props":3072,"children":3074},{"id":3073},"related-eval-skills",[3075],{"type":51,"value":3076},"Related eval skills",{"type":45,"tag":108,"props":3078,"children":3079},{},[3080,3096],{"type":45,"tag":112,"props":3081,"children":3082},{},[3083],{"type":45,"tag":116,"props":3084,"children":3085},{},[3086,3091],{"type":45,"tag":120,"props":3087,"children":3088},{},[3089],{"type":51,"value":3090},"After triage, if you need to...",{"type":45,"tag":120,"props":3092,"children":3093},{},[3094],{"type":51,"value":3095},"Use this skill",{"type":45,"tag":142,"props":3097,"children":3098},{},[3099,3116,3133,3150,3167],{"type":45,"tag":116,"props":3100,"children":3101},{},[3102,3107],{"type":45,"tag":149,"props":3103,"children":3104},{},[3105],{"type":51,"value":3106},"Build or expand the eval plan with new scenarios identified during triage",{"type":45,"tag":149,"props":3108,"children":3109},{},[3110],{"type":45,"tag":65,"props":3111,"children":3113},{"className":3112},[],[3114],{"type":51,"value":3115},"\u002Feval-suite-planner",{"type":45,"tag":116,"props":3117,"children":3118},{},[3119,3124],{"type":45,"tag":149,"props":3120,"children":3121},{},[3122],{"type":51,"value":3123},"Generate new test cases for expanded or revised scenarios",{"type":45,"tag":149,"props":3125,"children":3126},{},[3127],{"type":45,"tag":65,"props":3128,"children":3130},{"className":3129},[],[3131],{"type":51,"value":3132},"\u002Feval-generator",{"type":45,"tag":116,"props":3134,"children":3135},{},[3136,3141],{"type":45,"tag":149,"props":3137,"children":3138},{},[3139],{"type":51,"value":3140},"Get a quick structured report from a new CSV (without interactive triage)",{"type":45,"tag":149,"props":3142,"children":3143},{},[3144],{"type":45,"tag":65,"props":3145,"children":3147},{"className":3146},[],[3148],{"type":51,"value":3149},"\u002Feval-result-interpreter",{"type":45,"tag":116,"props":3151,"children":3152},{},[3153,3158],{"type":45,"tag":149,"props":3154,"children":3155},{},[3156],{"type":51,"value":3157},"Answer a methodology question that came up during triage",{"type":45,"tag":149,"props":3159,"children":3160},{},[3161],{"type":45,"tag":65,"props":3162,"children":3164},{"className":3163},[],[3165],{"type":51,"value":3166},"\u002Feval-faq",{"type":45,"tag":116,"props":3168,"children":3169},{},[3170,3175],{"type":45,"tag":149,"props":3171,"children":3172},{},[3173],{"type":51,"value":3174},"Walk the customer through the full eval pipeline end-to-end",{"type":45,"tag":149,"props":3176,"children":3177},{},[3178],{"type":45,"tag":65,"props":3179,"children":3181},{"className":3180},[],[3182],{"type":51,"value":3183},"\u002Feval-guide",{"type":45,"tag":3185,"props":3186,"children":3187},"style",{},[3188],{"type":51,"value":3189},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3191,"total":1634},[3192,3205,3220],{"slug":3193,"name":3193,"fn":3194,"description":3195,"org":3196,"tags":3197,"stars":28,"repoUrl":29,"updatedAt":3204},"eval-faq","answer AI agent evaluation methodology questions","Answers AI agent evaluation methodology questions with practical, opinionated guidance grounded primarily in Microsoft's agent evaluation ecosystem (MS Learn, Eval Scenario Library, Triage & Improvement Playbook, Eval Guidance Kit) supplemented by select industry sources.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3198,3199,3200,3201],{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":3202,"slug":3203,"type":15},"Strategy","strategy","2026-04-06T18:37:00.491978",{"slug":139,"name":139,"fn":3206,"description":3207,"org":3208,"tags":3209,"stars":28,"repoUrl":29,"updatedAt":3219},"interpret Copilot Studio evaluation results","Analyzes Copilot Studio evaluation results using Practical Guidance on Agent Evaluation's 10-step playbook (Steps 6, 7, and 9) plus Microsoft's triage diagnostics. Returns a gate-based SHIP \u002F ITERATE \u002F BLOCK verdict with root cause classification, remediation, and pattern analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3210,3213,3214,3215,3216],{"name":3211,"slug":3212,"type":15},"Analytics","analytics",{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},{"name":9,"slug":8,"type":15},{"name":3217,"slug":3218,"type":15},"Triage","triage","2026-04-06T18:36:56.479055",{"slug":4,"name":4,"fn":5,"description":6,"org":3221,"tags":3222,"stars":28,"repoUrl":29,"updatedAt":30},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3223,3224,3225,3226,3227],{"name":23,"slug":24,"type":15},{"name":20,"slug":21,"type":15},{"name":26,"slug":27,"type":15},{"name":17,"slug":18,"type":15},{"name":13,"slug":14,"type":15},{"items":3229,"total":3420},[3230,3252,3271,3290,3305,3322,3333,3346,3361,3376,3395,3408],{"slug":3231,"name":3231,"fn":3232,"description":3233,"org":3234,"tags":3235,"stars":3249,"repoUrl":3250,"updatedAt":3251},"rushstack-best-practices","manage Rush monorepos with best practices","Provides best practices and guidance for working with Rush monorepos. Use when the user is working in a Rush-based repository, asks about Rush commands (install, update, build, rebuild), needs help with project selection, dependency management, build caching, subspace configuration, or troubleshooting Rush-specific issues.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3236,3239,3242,3243,3246],{"name":3237,"slug":3238,"type":15},"Engineering","engineering",{"name":3240,"slug":3241,"type":15},"Local Development","local-development",{"name":9,"slug":8,"type":15},{"name":3244,"slug":3245,"type":15},"Project Management","project-management",{"name":3247,"slug":3248,"type":15},"Rush","rush",6484,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Frushstack","2026-04-06T18:34:44.965032",{"slug":3253,"name":3253,"fn":3254,"description":3255,"org":3256,"tags":3257,"stars":3268,"repoUrl":3269,"updatedAt":3270},"azure-ai-agents-persistent-dotnet","build AI agents with Azure .NET SDK","Azure AI Agents Persistent SDK for .NET. Low-level SDK for creating and managing AI agents with threads, messages, runs, and tools. Use for agent CRUD, conversation threads, streaming responses, function calling, file search, and code interpreter. Triggers: \"PersistentAgentsClient\", \"persistent agents\", \"agent threads\", \"agent runs\", \"streaming agents\", \"function calling agents .NET\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3258,3261,3262,3265],{"name":3259,"slug":3260,"type":15},".NET","net",{"name":23,"slug":24,"type":15},{"name":3263,"slug":3264,"type":15},"Azure","azure",{"name":3266,"slug":3267,"type":15},"LLM","llm",2804,"https:\u002F\u002Fgithub.com\u002Fmicrosoft\u002Fskills","2026-07-03T16:32:10.297433",{"slug":3272,"name":3272,"fn":3273,"description":3274,"org":3275,"tags":3276,"stars":3268,"repoUrl":3269,"updatedAt":3289},"azure-ai-anomalydetector-java","build anomaly detection applications with Java","Build anomaly detection applications with Azure AI Anomaly Detector SDK for Java. Use when implementing univariate\u002Fmultivariate anomaly detection, time-series analysis, or AI-powered monitoring.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3277,3278,3279,3282,3285,3286],{"name":3211,"slug":3212,"type":15},{"name":3263,"slug":3264,"type":15},{"name":3280,"slug":3281,"type":15},"Data Analysis","data-analysis",{"name":3283,"slug":3284,"type":15},"Java","java",{"name":9,"slug":8,"type":15},{"name":3287,"slug":3288,"type":15},"Monitoring","monitoring","2026-05-13T06:14:16.261754",{"slug":3291,"name":3291,"fn":3292,"description":3293,"org":3294,"tags":3295,"stars":3268,"repoUrl":3269,"updatedAt":3304},"azure-ai-contentsafety-java","build content moderation applications with Azure AI","Build content moderation applications with Azure AI Content Safety SDK for Java. Use when implementing text\u002Fimage analysis, blocklist management, or harm detection for hate, violence, sexual content, and self-harm.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3296,3299,3300,3301],{"name":3297,"slug":3298,"type":15},"AI Infrastructure","ai-infrastructure",{"name":3263,"slug":3264,"type":15},{"name":3283,"slug":3284,"type":15},{"name":3302,"slug":3303,"type":15},"Security","security","2026-07-07T06:53:31.293235",{"slug":3306,"name":3306,"fn":3307,"description":3308,"org":3309,"tags":3310,"stars":3268,"repoUrl":3269,"updatedAt":3321},"azure-ai-contentsafety-py","detect harmful content with Azure AI Content Safety","Azure AI Content Safety SDK for Python. Use for detecting harmful content in text and images with multi-severity classification.\nTriggers: \"azure-ai-contentsafety\", \"ContentSafetyClient\", \"content moderation\", \"harmful content\", \"text analysis\", \"image analysis\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3311,3312,3315,3316,3317,3320],{"name":3263,"slug":3264,"type":15},{"name":3313,"slug":3314,"type":15},"Compliance","compliance",{"name":3266,"slug":3267,"type":15},{"name":9,"slug":8,"type":15},{"name":3318,"slug":3319,"type":15},"Python","python",{"name":3302,"slug":3303,"type":15},"2026-07-18T05:14:23.017504",{"slug":3323,"name":3323,"fn":3324,"description":3325,"org":3326,"tags":3327,"stars":3268,"repoUrl":3269,"updatedAt":3332},"azure-ai-language-conversations-py","implement conversational language understanding with Python","Implement Conversational Language Understanding (CLU) using the azure-ai-language-conversations Python SDK. Use when working with ConversationAnalysisClient to analyze conversation intent and entities, building NLP features, or integrating language understanding into applications.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3328,3329,3330,3331],{"name":3211,"slug":3212,"type":15},{"name":3263,"slug":3264,"type":15},{"name":3266,"slug":3267,"type":15},{"name":3318,"slug":3319,"type":15},"2026-07-31T05:54:29.068751",{"slug":3334,"name":3334,"fn":3335,"description":3336,"org":3337,"tags":3338,"stars":3268,"repoUrl":3269,"updatedAt":3345},"azure-ai-translation-text-py","translate text using Azure AI services","Azure AI Text Translation SDK for real-time text translation, transliteration, language detection, and dictionary lookup. Use for translating text content in applications.\nTriggers: \"text translation\", \"translator\", \"translate text\", \"transliterate\", \"TextTranslationClient\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3339,3342,3343,3344],{"name":3340,"slug":3341,"type":15},"API Development","api-development",{"name":3263,"slug":3264,"type":15},{"name":9,"slug":8,"type":15},{"name":3318,"slug":3319,"type":15},"2026-07-18T05:14:16.988376",{"slug":3347,"name":3347,"fn":3348,"description":3349,"org":3350,"tags":3351,"stars":3268,"repoUrl":3269,"updatedAt":3360},"azure-ai-vision-imageanalysis-py","analyze images with Azure AI Vision","Azure AI Vision Image Analysis SDK for captions, tags, objects, OCR, people detection, and smart cropping. Use for computer vision and image understanding tasks.\nTriggers: \"image analysis\", \"computer vision\", \"OCR\", \"object detection\", \"ImageAnalysisClient\", \"image caption\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3352,3353,3356,3359],{"name":3263,"slug":3264,"type":15},{"name":3354,"slug":3355,"type":15},"Computer Vision","computer-vision",{"name":3357,"slug":3358,"type":15},"Images","images",{"name":3318,"slug":3319,"type":15},"2026-07-18T05:14:18.007737",{"slug":3362,"name":3362,"fn":3363,"description":3364,"org":3365,"tags":3366,"stars":3268,"repoUrl":3269,"updatedAt":3375},"azure-appconfiguration-java","manage configuration with Azure App Configuration","Azure App Configuration SDK for Java. Centralized application configuration management with key-value settings, feature flags, and snapshots.\nTriggers: \"ConfigurationClient java\", \"app configuration java\", \"feature flag java\", \"configuration setting java\", \"azure config java\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3367,3368,3371,3374],{"name":3263,"slug":3264,"type":15},{"name":3369,"slug":3370,"type":15},"Configuration","configuration",{"name":3372,"slug":3373,"type":15},"Feature Flags","feature-flags",{"name":3283,"slug":3284,"type":15},"2026-07-03T16:32:01.278468",{"slug":3377,"name":3377,"fn":3378,"description":3379,"org":3380,"tags":3381,"stars":3268,"repoUrl":3269,"updatedAt":3394},"azure-cosmos-rust","build applications with Azure Cosmos DB","Azure Cosmos DB library for Rust (NoSQL API). Document CRUD, containers, and globally distributed data.\nTriggers: \"cosmos db rust\", \"CosmosClient rust\", \"document crud rust\", \"NoSQL rust\", \"partition key rust\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3382,3385,3388,3391],{"name":3383,"slug":3384,"type":15},"Cosmos DB","cosmos-db",{"name":3386,"slug":3387,"type":15},"Database","database",{"name":3389,"slug":3390,"type":15},"NoSQL","nosql",{"name":3392,"slug":3393,"type":15},"Rust","rust","2026-07-31T05:54:27.021432",{"slug":3396,"name":3396,"fn":3378,"description":3397,"org":3398,"tags":3399,"stars":3268,"repoUrl":3269,"updatedAt":3407},"azure-cosmos-ts","Azure Cosmos DB JavaScript\u002FTypeScript SDK (@azure\u002Fcosmos) for data plane operations. Use for CRUD operations on documents, queries, bulk operations, and container management. Triggers: \"Cosmos DB\", \"@azure\u002Fcosmos\", \"CosmosClient\", \"document CRUD\", \"NoSQL queries\", \"bulk operations\", \"partition key\", \"container.items\".\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3400,3401,3402,3403,3404],{"name":3383,"slug":3384,"type":15},{"name":3386,"slug":3387,"type":15},{"name":9,"slug":8,"type":15},{"name":3389,"slug":3390,"type":15},{"name":3405,"slug":3406,"type":15},"TypeScript","typescript","2026-07-03T16:31:19.368382",{"slug":3409,"name":3409,"fn":3410,"description":3411,"org":3412,"tags":3413,"stars":3268,"repoUrl":3269,"updatedAt":3419},"azure-data-tables-java","build table storage applications with Java","Build table storage applications with Azure Tables SDK for Java. Use when working with Azure Table Storage or Cosmos DB Table API for NoSQL key-value data, schemaless storage, or structured data at scale.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3414,3415,3416,3417,3418],{"name":3263,"slug":3264,"type":15},{"name":3383,"slug":3384,"type":15},{"name":3386,"slug":3387,"type":15},{"name":3283,"slug":3284,"type":15},{"name":3389,"slug":3390,"type":15},"2026-05-13T06:14:17.582229",267]