[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-dotnet-pipeline-investigation":3,"mdc--ulbc6w-key":36,"related-org-dotnet-pipeline-investigation":3178,"related-repo-dotnet-pipeline-investigation":3342},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":25,"repoUrl":26,"updatedAt":27,"license":28,"forks":29,"topics":30,"repo":31,"sourceUrl":34,"mdContent":35},"pipeline-investigation","investigate Azure DevOps pipeline failures","Investigate AzDO pipeline failures beyond Helix — build errors, infra tooling crashes, validation test flakiness, artifact cascade failures. USE FOR: \"why did the unified-build fail\", \"what's breaking the pipeline\", \"how often does this failure occur\", \"drill into build task logs\", \"1ES scan failures\", \"SourcelinkTests flaky\", \"NetAnalyzers build error\", analyzing AzDO build timelines and task logs, failure frequency\u002Ftrend analysis. DO NOT USE FOR: Helix test failures (use helix-investigation), CI status overview (use ci-analysis), codeflow PRs (use flow-analysis). INVOKES: AzDO, Helix, and binlog MCP tools, az CLI for internal auth, gh CLI.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"dotnet",".NET (Microsoft)","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdotnet.png",[12,16,19,22],{"name":13,"slug":14,"type":15},".NET","net","tag",{"name":17,"slug":18,"type":15},"Engineering","engineering",{"name":20,"slug":21,"type":15},"CI\u002FCD","ci-cd",{"name":23,"slug":24,"type":15},"Debugging","debugging",12,"https:\u002F\u002Fgithub.com\u002Fdotnet\u002Farcade-skills","2026-07-12T08:21:58.244188",null,18,[],{"repoUrl":26,"stars":25,"forks":29,"topics":32,"description":33},[],"Reuseable AI skills, plugins, agents for use in the dotnet product repos, such as dotnet\u002Fruntime","https:\u002F\u002Fgithub.com\u002Fdotnet\u002Farcade-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fdotnet-dnceng\u002Fskills\u002Fpipeline-investigation","---\nname: pipeline-investigation\ndescription: >\n  Investigate AzDO pipeline failures beyond Helix — build errors, infra tooling\n  crashes, validation test flakiness, artifact cascade failures.\n  USE FOR: \"why did the unified-build fail\", \"what's breaking the pipeline\",\n  \"how often does this failure occur\", \"drill into build task logs\",\n  \"1ES scan failures\", \"SourcelinkTests flaky\", \"NetAnalyzers build error\",\n  analyzing AzDO build timelines and task logs, failure frequency\u002Ftrend analysis.\n  DO NOT USE FOR: Helix test failures (use helix-investigation), CI status\n  overview (use ci-analysis), codeflow PRs (use flow-analysis).\n  INVOKES: AzDO, Helix, and binlog MCP tools, az CLI for internal auth, gh CLI.\n---\n\n# Pipeline Investigation\n\nInvestigate AzDO pipeline failures that aren't Helix test failures — build errors, infrastructure tooling crashes, validation test flakiness, and artifact cascade failures. Complements helix-investigation by covering everything else in the pipeline.\n\n## When to Use This Skill\n\n- User has an AzDO build URL with a non-Helix failure (build step, validation, infra task)\n- User asks \"why did the pipeline fail\" and the failure is in a build\u002Fscan\u002Fvalidation task\n- User wants to know how often a specific failure occurs (frequency\u002Ftrend analysis)\n- User sees `exit code null`, 1ES PT errors, or MSBuild failures\n- User wants to understand artifact cascade failures (\"missing artifacts from prior build\")\n- User asks about SourcelinkTests, Binary Analysis Scan, or installer validation failures\n\n## Output Formats\n\nThis skill produces two distinct report types. Match the format to the request:\n\n### Health Assessment (\"pipeline health\", \"are builds passing\", \"pipeline status\")\n\nFollow [references\u002Fhealth-assessment-format.md](references\u002Fhealth-assessment-format.md). Output MUST include these two tables:\n\n1. **Failed Builds Table** — every failed build, classified and investigated:\n   | Build | Type | Source | Failure Detail |\n   - **Type**: Rolling, Forward Flow, or Other PR (classify via `gh pr view`)\n   - **Source**: Rolling → branch name. Forward Flow → `target ← source-repo`. Other PR → short description.\n\n2. **Summary Table** — pass\u002Ffail breakdown by build type:\n   | Type | Completed | ✅ Pass | ❌ Fail | Pass Rate |\n\n3. **Failure Trends Table** (conditional — include when 3+ builds in scope and at least one pattern recurs; cap at top 5):\n   | Pattern | Hits | Window | Status |\n   - **Status**: ❌ No issue filed, ✅ Fix merged, 🔄 Known issue (link), ⏳ Fix in progress\n\nSee the reference for build classification rules, branch filtering, and codeflow analysis methodology.\n\n**Save the report:** After presenting the health assessment, save it to `reviews\u002Fpipeline-health-\u003Cslug>-YYYY-MM-DD-HHMM.md` in the repo, where `\u003Cslug>` identifies the pipeline or scope (e.g., `unified-build`, `runtime-ci`). Include the timestamp to ensure uniqueness across multiple investigations per day. Include a Methodology section at the end documenting data sources and classification approach.\n\n### Individual Failure Investigation (\"why did this build fail\", single build URL)\n\nUse Step 7's numbered format: failure category, frequency, root cause, affected branches, owner, existing issues, recommended action.\n\n## Prerequisites\n\n- AzDO MCP tools (ado-* prefix) for querying builds and timelines on public projects\n- Binlog MCP server (`binlog` namespace) for analyzing MSBuild binary logs from build artifacts via `Microsoft.AITools.BinlogMcp`\n- `az account get-access-token` or `azureauth ado token` for authenticated REST API access to `dnceng\u002Finternal`\n- `curl` for downloading task logs and build artifacts\n- `gh` CLI for searching related issues and source code\n\n## Authentication\n\nThe `dnceng\u002Finternal` project requires authentication. Two methods:\n\n```bash\n# Method 1: Azure CLI (preferred — works in most environments)\nTOKEN=$(az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query accessToken -o tsv)\n\n# Method 2: azureauth tool\nTOKEN=$(azureauth ado token --output token --prompt-hint \"copilot-cli\")\n\n# Usage\ncurl -s -H \"Authorization: Bearer $TOKEN\" \"https:\u002F\u002Fdev.azure.com\u002Fdnceng\u002Finternal\u002F_apis\u002Fbuild\u002Fbuilds\u002F{buildId}\u002Ftimeline?api-version=7.1\"\n```\n\n> ⚠️ Tokens expire. Re-acquire if you get a 401 or redirect to a sign-in page.\n> ⚠️ AzDO MCP tools do NOT work against `dnceng\u002Finternal`. Use curl with Bearer token for all internal queries.\n\n## Workflow\n\n### Step 1: Get the build timeline\n\nGiven a build ID or URL, query the timeline to find all failed records:\n\n```\nhttps:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds\u002F{buildId}\u002Ftimeline?api-version=7.1\n```\n\nFilter records by `result == \"failed\"` or `result == \"succeededWithIssues\"`, with `type == \"Task\"`. Don't skip `succeededWithIssues` — these contain real failures (signing validation errors, Component Governance warnings) that didn't block the overall job. For health assessments, also include builds with overall result `partiallySucceeded`.\n\nEach failed or warning task has:\n- **name** — the task that failed\n- **parentId** — links to the parent Job record (tells you which leg)\n- **issues** — error messages with type and message fields\n- **log.url** — URL to download the full task log\n\n### Step 2: Categorize the failure\n\n| Pattern | Category | Owner |\n|---------|----------|-------|\n| `Binary Analysis Scan` task, `exit code null` | 1ES infra tooling crash | 1ES PT team |\n| `MSB3073` \u002F `NetAnalyzers.Package.csproj` error | Build error | Source code \u002F SDK |\n| `Validate installer packages` — expected vs actual package list mismatch | Installer validation regex | dotnet\u002Finstaller |\n| `Run Tests` with `SourcelinkTests.VerifySourcelinks` | Sourcelink validation | dotnet\u002Fdotnet |\n| `Run Tests` with `The task has timed out` across many `SB_*_Validation_*` legs | Run Tests timeout epidemic | dotnet\u002Fdotnet |\n| `Run Tests` with locale test failure (e.g., `tr-TR`) | Scenario test bug | dotnet\u002Ftemplating |\n| `Build` with `IBCMerge` \u002F PGO error in `Windows_Pgo_*` | PGO optimization failure | dotnet\u002Fruntime |\n| `Download Previous Build` with \"Artifact not found\" | Artifact cascade failure | Prior build failed first |\n| `Build` with `curl` \u002F `tar` \u002F download failure (exit code 2, \"not recoverable\") | External resource fetch failure | Retry first; if persistent, check URL\u002Fversion |\n| `Build` with `npm ci` ETIMEDOUT \u002F ECONNREFUSED | npm network timeout | Transient; shell retry wrapper needed |\n| `Build` task with compilation errors | Source code build break | Varies by component |\n| `Crossgen.targets` with `exit code 57005` (0xDEAD) | crossgen2 fatal crash | dotnet\u002Fruntime (area-crossgen2-coreclr) |\n| ESRP `MacSignFailed` \u002F `FailDoNotRetry` \u002F notarization errors | Signing\u002Fnotarization failure | dotnet\u002Fsdk or ESRP team |\n| `exit code null` on cross-compilation legs in containers | Container OOM kill | Infrastructure — pool\u002Fcontainer config |\n| Build start == finish, HTTP 204 timeline, `validationResults` has errors | YAML pre-flight rejection | PR author — pipeline YAML invalid |\n\n> **Routing: AzDO tests vs Helix tests.** If the test runs directly as an AzDO task (e.g., `Run Tests` in `SB_*_Validation_*` legs — SourcelinkTests, scenario tests), it's pipeline-investigation. If the test is submitted to Helix (has a Helix job ID, work items, console logs at helix.dot.net), use helix-investigation.\n\n### Step 3: Download and analyze task logs\n\n```bash\nTOKEN=$(azureauth ado token --output token --prompt-hint \"copilot-cli\")\ncurl -s -H \"Authorization: Bearer $TOKEN\" \"{log_url}\" > \u002Ftmp\u002Ftask-{buildId}.log\n```\n\nKey patterns to search for:\n- **`exit code null`** — process terminated by signal (OOM, timeout). Check for memory-related warnings before the crash.\n- **`exit code 57005`** (0xDEAD) — crossgen2 fatal error. Check which assembly was being compiled. Intermittent crashes need crash dumps for diagnosis.\n- **`[ERROR]`** \u002F **`##[error]`** — explicit error messages from the task\n- **`[WARNING]` floods** — excessive warnings (e.g., 1000+ hardlink failures) indicate resource exhaustion\n- **MSBuild error codes** — `MSB3073` (command failed), `NETSDK*` (SDK errors), `NU1105` (invalid target framework — often transient forward flow)\n- **`MacSignFailed`** \u002F **`FailDoNotRetry`** — ESRP signing rejection. `FailDoNotRetry` means deterministic content failure, not transient.\n- **Stack traces** — .NET exceptions in test output\n- **`ETIMEDOUT`** \u002F **`ECONNREFUSED`** — TCP-level network failures in npm\u002Fcurl that aren't retried by default\n\n### Step 4: Determine if it's a one-off or recurring\n\nQuery failed builds over a time range and count how many hit the same failure:\n\n```bash\n# Get failed builds for a pipeline definition\ncurl -s -H \"Authorization: Bearer $TOKEN\" \\\n  \"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds?definitions={defId}&resultFilter=failed&minTime={iso8601}&$top=200&api-version=7.1\"\n```\n\n> 💡 **Rolling builds give cleaner signal.** Add `&reasonFilter=schedule` to filter to scheduled builds only. PR builds include broken branches that inflate failure counts and obscure systemic trends.\n\nFor each build, check its timeline for the same failure pattern. Track in a SQL table:\n\n```sql\nCREATE TABLE pipeline_failures (\n  build_id INT, branch TEXT, queued TEXT, failure_category TEXT, \n  failed_task TEXT, notes TEXT\n);\n```\n\nLook for:\n- **Burst pattern** — many occurrences in a short window → something changed (tooling update, artifact size growth)\n- **Steady trickle** — consistent low rate → chronic issue\n- **Single occurrence** — likely transient, not worth investigating further\n\n### Step 5: Check for known issues\n\nSearch the relevant repo for existing issues:\n\n```bash\ngh search issues \"{failure_pattern}\" --repo dotnet\u002Fdotnet --limit 5\n```\n\nIf found, check if the issue is being worked on. If not found and the failure is recurring, consider filing one.\n\n### Step 6: Root cause analysis\n\nFor non-obvious failures, pull up the source code of the failing test or tool:\n\n```bash\ngh api \"repos\u002F{owner}\u002F{repo}\u002Fcontents\u002F{path}\" --jq '.content' | base64 -d\n```\n\nLook for:\n- **Unbounded parallelism** — `Parallel.ForEach` without `MaxDegreeOfParallelism`\n- **Tight timeouts** — processes timing out under contention\n- **Resource assumptions** — disk space, memory, network that may not hold in CI\n- **Non-determinism** — race conditions, order-dependent assertions\n\n### Step 6b: Binlog analysis for deep investigation\n\nWhen task logs don't reveal enough, download the build's binlog artifacts for detailed MSBuild analysis:\n\n```bash\n# List build artifacts to find binlogs\ncurl -s -H \"Authorization: Bearer $TOKEN\" \\\n  \"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds\u002F{buildId}\u002Fartifacts?api-version=7.1\"\n\n# Download the artifact zip\ncurl -s -H \"Authorization: Bearer $TOKEN\" -o \u002Ftmp\u002Flogs.zip \"{downloadUrl}\"\nunzip -o \u002Ftmp\u002Flogs.zip \"*\u002FBuild.binlog\" -d \u002Ftmp\u002Fbinlogs\u002F\n```\n\nThen use the binlog MCP tools (`Microsoft.AITools.BinlogMcp`, wired under the `binlog` MCP namespace). Call `tools\u002Flist` on the `binlog` namespace first if you're unsure which capabilities your server version exposes — names below are typical for `Microsoft.AITools.BinlogMcp`:\n\n- **Build configuration + which target failed** (typically `binlog_overview`)\n- **Structured diagnostics list** with `{code, message, file, line, column, projectFile, targetName, taskName}` per row (typically `binlog_errors` \u002F `binlog_warnings`)\n- **Find specific patterns** like `MacSignFailed` or `crossgen2` (typically `binlog_search`, same StructuredLog-Viewer query syntax)\n- **Exact command lines + parameters for failed tasks** — Exec's `Command`, RAR's `Assemblies`\u002F`AppConfigFile`, etc. (typically `binlog_task_details`)\n- **All tasks in a target** — useful when one target invokes a task 126 times and only one failed (typically `binlog_tasks_in_target`)\n- **Automated \"what broke\" report** — newer servers include a root-cause diagnostic (typically `binlog_diagnose`; not present on older servers)\n- **Trace where a property got its value** across imports\u002Ftargets\u002Ftasks (typically `binlog_explain_property`; newer servers only)\n\n> 💡 Binlogs capture the full MSBuild execution including command lines, environment variables, and interleaved output that task logs lose. Essential for signing, crossgen2, and SBRP failures.\n\n### Step 7: Report findings (individual failure investigations only)\n\nFor health assessments, use the Output Formats section above instead.\n\nProvide:\n1. **Failure category** — which pattern from the table above\n2. **Frequency** — how often in the last N days, trending up\u002Fdown\u002Fstable\n3. **Root cause** — with evidence from logs and\u002For source code\n4. **Affected branches** — main only, or also release branches\n5. **Owner** — who should fix it (1ES, dotnet\u002Fdotnet, SDK team, etc.)\n6. **Existing issues** — links to any filed issues\n7. **Recommended action** — file issue, retry, wait for fix, etc.\n\n## Artifact Cascade Failures\n\nWhen a build leg fails, downstream legs that depend on its artifacts will also fail with \"Artifact not found.\" This creates a cascade:\n\n```\nSB_CentOS_Build (fails: NetAnalyzers error)\n  → SB_CentOS_Validation (fails: missing artifacts)\n  → SB_Fedora_Offline (fails: missing artifacts)\n```\n\n**Don't investigate the cascade — find the root failure.** Look for the first failed task chronologically, or filter out \"Download Previous Build\" failures to find the real cause.\n\n## Category-Specific Investigation Techniques\n\nFor detailed investigation techniques for each failure category in the table above, see [references\u002Finvestigation-techniques.md](references\u002Finvestigation-techniques.md). Covers: ESRP signing\u002Fnotarization, container OOM, crossgen2 crashes, YAML pre-flight rejections, and network transient failures.\n\n## Fix Verification\n\nWhen a fix is merged, verify it's tested by checking builds queued **after** the merge — not builds that finished after it.\n\n### Query all build statuses\n\n**Always include in-progress and not-started builds**, not just completed ones. The AzDO builds API requires explicit `statusFilter` to return active builds.\n\n> ⚠️ **The AzDO API rejects combining `completed` with `inProgress`\u002F`notStarted` in a single `statusFilter`.** You must make separate calls and merge the results:\n\n```bash\n# Completed builds\ncurl -s -H \"Authorization: Bearer $TOKEN\" \\\n  \"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds?definitions={defId}&branchName=refs\u002Fheads\u002F{branch}&statusFilter=completed&api-version=7.1\"\n\n# Active builds (in-progress and queued)\ncurl -s -H \"Authorization: Bearer $TOKEN\" \\\n  \"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds?definitions={defId}&branchName=refs\u002Fheads\u002F{branch}&statusFilter=inProgress,notStarted&api-version=7.1\"\n```\n\nIf you only query completed builds, you'll miss active builds that are currently testing the fix and incorrectly conclude \"not tested yet.\"\n\n### Multi-branch fixes\n\nFixes often need backporting to multiple release branches. For each branch:\n\n1. Find the fix PR (or backport PR) and its merge time\n2. Query builds on that branch with **all statuses**\n3. Compare each build's `queueTime` to the merge time\n4. Partition into pre-merge (doesn't have fix) and post-merge (has fix)\n\n> ⚠️ **batchedCI builds pick up source at queue time.** A build queued before a PR merges runs against old source, even if it finishes hours later.\n\n## Stop Signals\n\n- **Stop after categorizing** if it's a known issue with an existing bug. Link to it and move on.\n- **Stop frequency analysis** after 14 days of data. Longer trends are rarely actionable.\n- **Stop investigating cascades** — always trace back to the root failure.\n- **Present single-occurrence failures** — even one-offs deserve a summary. Let the user decide whether to dig deeper or move on.\n\n## Anti-Patterns\n\n> 🚨 **Don't confuse cascade failures with root causes.** \"Missing artifacts\" means a prior leg failed — find that leg.\n\n> 🚨 **Don't assume `exit code null` is a code bug.** It usually means the process was terminated externally (OOM, signal). Look for resource exhaustion in the log. In containers, check memory limits.\n\n> 🚨 **Don't investigate all 100+ failed builds.** Sample 3-5 spread across the time range to confirm the pattern, then count occurrences.\n\n> 🚨 **Don't assume ESRP `FailDoNotRetry` is transient.** It means the binary content is deterministically invalid. Investigate the source commits, don't retry the build.\n\n> 🚨 **Don't skip infrastructure failures.** Container OOM, agent timeouts, signing failures, and network issues deserve the same depth of investigation as code failures. They often have actionable fixes (memory tuning, retry wrappers, pool changes).\n\n## Discovering Pipeline Definitions\n\nNever hardcode definition IDs — they vary across projects and can change. Discover them at runtime:\n\n```bash\n# Public pipelines (AzDO MCP tools work here too)\ncurl -s \"https:\u002F\u002Fdev.azure.com\u002Fdnceng-public\u002Fpublic\u002F_apis\u002Fbuild\u002Fdefinitions?name=dotnet-unified-build&api-version=7.1\" | jq '.value[] | {id, name, project: .project.name}'\n\n# Internal pipelines (requires auth)\nTOKEN=$(az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query accessToken -o tsv)\ncurl -s -H \"Authorization: Bearer $TOKEN\" \\\n  \"https:\u002F\u002Fdev.azure.com\u002Fdnceng\u002Finternal\u002F_apis\u002Fbuild\u002Fdefinitions?name=dotnet-unified-build&api-version=7.1\" \\\n  | jq '.value[] | {id, name, project: .project.name}'\n```\n\nIf querying internal fails (401\u002Fredirect), fall back to public-only analysis and note the limitation in the report.\n\n### Health Assessment Report Format\n\n**Required format** for pipeline health assessments. See Output Formats section above for routing, and [references\u002Fhealth-assessment-format.md](references\u002Fhealth-assessment-format.md) for the complete specification including build classification rules, branch filtering, and codeflow analysis methodology.\n\n",{"data":37,"body":38},{"name":4,"description":6},{"type":39,"children":40},"root",[41,49,55,62,106,112,117,124,137,224,229,271,277,282,288,362,368,380,617,633,639,645,650,660,704,709,752,758,1236,1263,1269,1381,1386,1551,1557,1562,1640,1663,1668,1709,1714,1747,1753,1758,1816,1821,1827,1832,1899,1903,1959,1965,1970,2143,2184,2360,2368,2374,2379,2384,2456,2462,2467,2476,2486,2492,2504,2510,2522,2528,2546,2589,2721,2726,2732,2737,2773,2787,2793,2836,2842,2857,2878,2892,2913,2927,2933,2938,3145,3150,3156,3172],{"type":42,"tag":43,"props":44,"children":45},"element","h1",{"id":4},[46],{"type":47,"value":48},"text","Pipeline Investigation",{"type":42,"tag":50,"props":51,"children":52},"p",{},[53],{"type":47,"value":54},"Investigate AzDO pipeline failures that aren't Helix test failures — build errors, infrastructure tooling crashes, validation test flakiness, and artifact cascade failures. Complements helix-investigation by covering everything else in the pipeline.",{"type":42,"tag":56,"props":57,"children":59},"h2",{"id":58},"when-to-use-this-skill",[60],{"type":47,"value":61},"When to Use This Skill",{"type":42,"tag":63,"props":64,"children":65},"ul",{},[66,72,77,82,96,101],{"type":42,"tag":67,"props":68,"children":69},"li",{},[70],{"type":47,"value":71},"User has an AzDO build URL with a non-Helix failure (build step, validation, infra task)",{"type":42,"tag":67,"props":73,"children":74},{},[75],{"type":47,"value":76},"User asks \"why did the pipeline fail\" and the failure is in a build\u002Fscan\u002Fvalidation task",{"type":42,"tag":67,"props":78,"children":79},{},[80],{"type":47,"value":81},"User wants to know how often a specific failure occurs (frequency\u002Ftrend analysis)",{"type":42,"tag":67,"props":83,"children":84},{},[85,87,94],{"type":47,"value":86},"User sees ",{"type":42,"tag":88,"props":89,"children":91},"code",{"className":90},[],[92],{"type":47,"value":93},"exit code null",{"type":47,"value":95},", 1ES PT errors, or MSBuild failures",{"type":42,"tag":67,"props":97,"children":98},{},[99],{"type":47,"value":100},"User wants to understand artifact cascade failures (\"missing artifacts from prior build\")",{"type":42,"tag":67,"props":102,"children":103},{},[104],{"type":47,"value":105},"User asks about SourcelinkTests, Binary Analysis Scan, or installer validation failures",{"type":42,"tag":56,"props":107,"children":109},{"id":108},"output-formats",[110],{"type":47,"value":111},"Output Formats",{"type":42,"tag":50,"props":113,"children":114},{},[115],{"type":47,"value":116},"This skill produces two distinct report types. Match the format to the request:",{"type":42,"tag":118,"props":119,"children":121},"h3",{"id":120},"health-assessment-pipeline-health-are-builds-passing-pipeline-status",[122],{"type":47,"value":123},"Health Assessment (\"pipeline health\", \"are builds passing\", \"pipeline status\")",{"type":42,"tag":50,"props":125,"children":126},{},[127,129,135],{"type":47,"value":128},"Follow ",{"type":42,"tag":130,"props":131,"children":133},"a",{"href":132},"references\u002Fhealth-assessment-format.md",[134],{"type":47,"value":132},{"type":47,"value":136},". Output MUST include these two tables:",{"type":42,"tag":138,"props":139,"children":140},"ol",{},[141,191,201],{"type":42,"tag":67,"props":142,"children":143},{},[144,150,152],{"type":42,"tag":145,"props":146,"children":147},"strong",{},[148],{"type":47,"value":149},"Failed Builds Table",{"type":47,"value":151}," — every failed build, classified and investigated:\n| Build | Type | Source | Failure Detail |",{"type":42,"tag":63,"props":153,"children":154},{},[155,173],{"type":42,"tag":67,"props":156,"children":157},{},[158,163,165,171],{"type":42,"tag":145,"props":159,"children":160},{},[161],{"type":47,"value":162},"Type",{"type":47,"value":164},": Rolling, Forward Flow, or Other PR (classify via ",{"type":42,"tag":88,"props":166,"children":168},{"className":167},[],[169],{"type":47,"value":170},"gh pr view",{"type":47,"value":172},")",{"type":42,"tag":67,"props":174,"children":175},{},[176,181,183,189],{"type":42,"tag":145,"props":177,"children":178},{},[179],{"type":47,"value":180},"Source",{"type":47,"value":182},": Rolling → branch name. Forward Flow → ",{"type":42,"tag":88,"props":184,"children":186},{"className":185},[],[187],{"type":47,"value":188},"target ← source-repo",{"type":47,"value":190},". Other PR → short description.",{"type":42,"tag":67,"props":192,"children":193},{},[194,199],{"type":42,"tag":145,"props":195,"children":196},{},[197],{"type":47,"value":198},"Summary Table",{"type":47,"value":200}," — pass\u002Ffail breakdown by build type:\n| Type | Completed | ✅ Pass | ❌ Fail | Pass Rate |",{"type":42,"tag":67,"props":202,"children":203},{},[204,209,211],{"type":42,"tag":145,"props":205,"children":206},{},[207],{"type":47,"value":208},"Failure Trends Table",{"type":47,"value":210}," (conditional — include when 3+ builds in scope and at least one pattern recurs; cap at top 5):\n| Pattern | Hits | Window | Status |",{"type":42,"tag":63,"props":212,"children":213},{},[214],{"type":42,"tag":67,"props":215,"children":216},{},[217,222],{"type":42,"tag":145,"props":218,"children":219},{},[220],{"type":47,"value":221},"Status",{"type":47,"value":223},": ❌ No issue filed, ✅ Fix merged, 🔄 Known issue (link), ⏳ Fix in progress",{"type":42,"tag":50,"props":225,"children":226},{},[227],{"type":47,"value":228},"See the reference for build classification rules, branch filtering, and codeflow analysis methodology.",{"type":42,"tag":50,"props":230,"children":231},{},[232,237,239,245,247,253,255,261,263,269],{"type":42,"tag":145,"props":233,"children":234},{},[235],{"type":47,"value":236},"Save the report:",{"type":47,"value":238}," After presenting the health assessment, save it to ",{"type":42,"tag":88,"props":240,"children":242},{"className":241},[],[243],{"type":47,"value":244},"reviews\u002Fpipeline-health-\u003Cslug>-YYYY-MM-DD-HHMM.md",{"type":47,"value":246}," in the repo, where ",{"type":42,"tag":88,"props":248,"children":250},{"className":249},[],[251],{"type":47,"value":252},"\u003Cslug>",{"type":47,"value":254}," identifies the pipeline or scope (e.g., ",{"type":42,"tag":88,"props":256,"children":258},{"className":257},[],[259],{"type":47,"value":260},"unified-build",{"type":47,"value":262},", ",{"type":42,"tag":88,"props":264,"children":266},{"className":265},[],[267],{"type":47,"value":268},"runtime-ci",{"type":47,"value":270},"). Include the timestamp to ensure uniqueness across multiple investigations per day. Include a Methodology section at the end documenting data sources and classification approach.",{"type":42,"tag":118,"props":272,"children":274},{"id":273},"individual-failure-investigation-why-did-this-build-fail-single-build-url",[275],{"type":47,"value":276},"Individual Failure Investigation (\"why did this build fail\", single build URL)",{"type":42,"tag":50,"props":278,"children":279},{},[280],{"type":47,"value":281},"Use Step 7's numbered format: failure category, frequency, root cause, affected branches, owner, existing issues, recommended action.",{"type":42,"tag":56,"props":283,"children":285},{"id":284},"prerequisites",[286],{"type":47,"value":287},"Prerequisites",{"type":42,"tag":63,"props":289,"children":290},{},[291,296,315,340,351],{"type":42,"tag":67,"props":292,"children":293},{},[294],{"type":47,"value":295},"AzDO MCP tools (ado-* prefix) for querying builds and timelines on public projects",{"type":42,"tag":67,"props":297,"children":298},{},[299,301,307,309],{"type":47,"value":300},"Binlog MCP server (",{"type":42,"tag":88,"props":302,"children":304},{"className":303},[],[305],{"type":47,"value":306},"binlog",{"type":47,"value":308}," namespace) for analyzing MSBuild binary logs from build artifacts via ",{"type":42,"tag":88,"props":310,"children":312},{"className":311},[],[313],{"type":47,"value":314},"Microsoft.AITools.BinlogMcp",{"type":42,"tag":67,"props":316,"children":317},{},[318,324,326,332,334],{"type":42,"tag":88,"props":319,"children":321},{"className":320},[],[322],{"type":47,"value":323},"az account get-access-token",{"type":47,"value":325}," or ",{"type":42,"tag":88,"props":327,"children":329},{"className":328},[],[330],{"type":47,"value":331},"azureauth ado token",{"type":47,"value":333}," for authenticated REST API access to ",{"type":42,"tag":88,"props":335,"children":337},{"className":336},[],[338],{"type":47,"value":339},"dnceng\u002Finternal",{"type":42,"tag":67,"props":341,"children":342},{},[343,349],{"type":42,"tag":88,"props":344,"children":346},{"className":345},[],[347],{"type":47,"value":348},"curl",{"type":47,"value":350}," for downloading task logs and build artifacts",{"type":42,"tag":67,"props":352,"children":353},{},[354,360],{"type":42,"tag":88,"props":355,"children":357},{"className":356},[],[358],{"type":47,"value":359},"gh",{"type":47,"value":361}," CLI for searching related issues and source code",{"type":42,"tag":56,"props":363,"children":365},{"id":364},"authentication",[366],{"type":47,"value":367},"Authentication",{"type":42,"tag":50,"props":369,"children":370},{},[371,373,378],{"type":47,"value":372},"The ",{"type":42,"tag":88,"props":374,"children":376},{"className":375},[],[377],{"type":47,"value":339},{"type":47,"value":379}," project requires authentication. Two methods:",{"type":42,"tag":381,"props":382,"children":387},"pre",{"className":383,"code":384,"language":385,"meta":386,"style":386},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Method 1: Azure CLI (preferred — works in most environments)\nTOKEN=$(az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query accessToken -o tsv)\n\n# Method 2: azureauth tool\nTOKEN=$(azureauth ado token --output token --prompt-hint \"copilot-cli\")\n\n# Usage\ncurl -s -H \"Authorization: Bearer $TOKEN\" \"https:\u002F\u002Fdev.azure.com\u002Fdnceng\u002Finternal\u002F_apis\u002Fbuild\u002Fbuilds\u002F{buildId}\u002Ftimeline?api-version=7.1\"\n","bash","",[388],{"type":42,"tag":88,"props":389,"children":390},{"__ignoreMap":386},[391,403,471,481,490,550,558,567],{"type":42,"tag":392,"props":393,"children":396},"span",{"class":394,"line":395},"line",1,[397],{"type":42,"tag":392,"props":398,"children":400},{"style":399},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[401],{"type":47,"value":402},"# Method 1: Azure CLI (preferred — works in most environments)\n",{"type":42,"tag":392,"props":404,"children":406},{"class":394,"line":405},2,[407,413,419,425,431,436,441,446,451,456,461,466],{"type":42,"tag":392,"props":408,"children":410},{"style":409},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[411],{"type":47,"value":412},"TOKEN",{"type":42,"tag":392,"props":414,"children":416},{"style":415},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[417],{"type":47,"value":418},"=$(",{"type":42,"tag":392,"props":420,"children":422},{"style":421},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[423],{"type":47,"value":424},"az",{"type":42,"tag":392,"props":426,"children":428},{"style":427},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[429],{"type":47,"value":430}," account",{"type":42,"tag":392,"props":432,"children":433},{"style":427},[434],{"type":47,"value":435}," get-access-token",{"type":42,"tag":392,"props":437,"children":438},{"style":427},[439],{"type":47,"value":440}," --resource",{"type":42,"tag":392,"props":442,"children":443},{"style":427},[444],{"type":47,"value":445}," 499b84ac-1321-427f-aa17-267ca6975798",{"type":42,"tag":392,"props":447,"children":448},{"style":427},[449],{"type":47,"value":450}," --query",{"type":42,"tag":392,"props":452,"children":453},{"style":427},[454],{"type":47,"value":455}," accessToken",{"type":42,"tag":392,"props":457,"children":458},{"style":427},[459],{"type":47,"value":460}," -o",{"type":42,"tag":392,"props":462,"children":463},{"style":427},[464],{"type":47,"value":465}," tsv",{"type":42,"tag":392,"props":467,"children":468},{"style":415},[469],{"type":47,"value":470},")\n",{"type":42,"tag":392,"props":472,"children":474},{"class":394,"line":473},3,[475],{"type":42,"tag":392,"props":476,"children":478},{"emptyLinePlaceholder":477},true,[479],{"type":47,"value":480},"\n",{"type":42,"tag":392,"props":482,"children":484},{"class":394,"line":483},4,[485],{"type":42,"tag":392,"props":486,"children":487},{"style":399},[488],{"type":47,"value":489},"# Method 2: azureauth tool\n",{"type":42,"tag":392,"props":491,"children":493},{"class":394,"line":492},5,[494,498,502,507,512,517,522,526,531,536,541,546],{"type":42,"tag":392,"props":495,"children":496},{"style":409},[497],{"type":47,"value":412},{"type":42,"tag":392,"props":499,"children":500},{"style":415},[501],{"type":47,"value":418},{"type":42,"tag":392,"props":503,"children":504},{"style":421},[505],{"type":47,"value":506},"azureauth",{"type":42,"tag":392,"props":508,"children":509},{"style":427},[510],{"type":47,"value":511}," ado",{"type":42,"tag":392,"props":513,"children":514},{"style":427},[515],{"type":47,"value":516}," token",{"type":42,"tag":392,"props":518,"children":519},{"style":427},[520],{"type":47,"value":521}," --output",{"type":42,"tag":392,"props":523,"children":524},{"style":427},[525],{"type":47,"value":516},{"type":42,"tag":392,"props":527,"children":528},{"style":427},[529],{"type":47,"value":530}," --prompt-hint",{"type":42,"tag":392,"props":532,"children":533},{"style":415},[534],{"type":47,"value":535}," \"",{"type":42,"tag":392,"props":537,"children":538},{"style":427},[539],{"type":47,"value":540},"copilot-cli",{"type":42,"tag":392,"props":542,"children":543},{"style":415},[544],{"type":47,"value":545},"\"",{"type":42,"tag":392,"props":547,"children":548},{"style":415},[549],{"type":47,"value":470},{"type":42,"tag":392,"props":551,"children":553},{"class":394,"line":552},6,[554],{"type":42,"tag":392,"props":555,"children":556},{"emptyLinePlaceholder":477},[557],{"type":47,"value":480},{"type":42,"tag":392,"props":559,"children":561},{"class":394,"line":560},7,[562],{"type":42,"tag":392,"props":563,"children":564},{"style":399},[565],{"type":47,"value":566},"# Usage\n",{"type":42,"tag":392,"props":568,"children":570},{"class":394,"line":569},8,[571,575,580,585,589,594,599,603,607,612],{"type":42,"tag":392,"props":572,"children":573},{"style":421},[574],{"type":47,"value":348},{"type":42,"tag":392,"props":576,"children":577},{"style":427},[578],{"type":47,"value":579}," -s",{"type":42,"tag":392,"props":581,"children":582},{"style":427},[583],{"type":47,"value":584}," -H",{"type":42,"tag":392,"props":586,"children":587},{"style":415},[588],{"type":47,"value":535},{"type":42,"tag":392,"props":590,"children":591},{"style":427},[592],{"type":47,"value":593},"Authorization: Bearer ",{"type":42,"tag":392,"props":595,"children":596},{"style":409},[597],{"type":47,"value":598},"$TOKEN",{"type":42,"tag":392,"props":600,"children":601},{"style":415},[602],{"type":47,"value":545},{"type":42,"tag":392,"props":604,"children":605},{"style":415},[606],{"type":47,"value":535},{"type":42,"tag":392,"props":608,"children":609},{"style":427},[610],{"type":47,"value":611},"https:\u002F\u002Fdev.azure.com\u002Fdnceng\u002Finternal\u002F_apis\u002Fbuild\u002Fbuilds\u002F{buildId}\u002Ftimeline?api-version=7.1",{"type":42,"tag":392,"props":613,"children":614},{"style":415},[615],{"type":47,"value":616},"\"\n",{"type":42,"tag":618,"props":619,"children":620},"blockquote",{},[621],{"type":42,"tag":50,"props":622,"children":623},{},[624,626,631],{"type":47,"value":625},"⚠️ Tokens expire. Re-acquire if you get a 401 or redirect to a sign-in page.\n⚠️ AzDO MCP tools do NOT work against ",{"type":42,"tag":88,"props":627,"children":629},{"className":628},[],[630],{"type":47,"value":339},{"type":47,"value":632},". Use curl with Bearer token for all internal queries.",{"type":42,"tag":56,"props":634,"children":636},{"id":635},"workflow",[637],{"type":47,"value":638},"Workflow",{"type":42,"tag":118,"props":640,"children":642},{"id":641},"step-1-get-the-build-timeline",[643],{"type":47,"value":644},"Step 1: Get the build timeline",{"type":42,"tag":50,"props":646,"children":647},{},[648],{"type":47,"value":649},"Given a build ID or URL, query the timeline to find all failed records:",{"type":42,"tag":381,"props":651,"children":655},{"className":652,"code":654,"language":47},[653],"language-text","https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds\u002F{buildId}\u002Ftimeline?api-version=7.1\n",[656],{"type":42,"tag":88,"props":657,"children":658},{"__ignoreMap":386},[659],{"type":47,"value":654},{"type":42,"tag":50,"props":661,"children":662},{},[663,665,671,672,678,680,686,688,694,696,702],{"type":47,"value":664},"Filter records by ",{"type":42,"tag":88,"props":666,"children":668},{"className":667},[],[669],{"type":47,"value":670},"result == \"failed\"",{"type":47,"value":325},{"type":42,"tag":88,"props":673,"children":675},{"className":674},[],[676],{"type":47,"value":677},"result == \"succeededWithIssues\"",{"type":47,"value":679},", with ",{"type":42,"tag":88,"props":681,"children":683},{"className":682},[],[684],{"type":47,"value":685},"type == \"Task\"",{"type":47,"value":687},". Don't skip ",{"type":42,"tag":88,"props":689,"children":691},{"className":690},[],[692],{"type":47,"value":693},"succeededWithIssues",{"type":47,"value":695}," — these contain real failures (signing validation errors, Component Governance warnings) that didn't block the overall job. For health assessments, also include builds with overall result ",{"type":42,"tag":88,"props":697,"children":699},{"className":698},[],[700],{"type":47,"value":701},"partiallySucceeded",{"type":47,"value":703},".",{"type":42,"tag":50,"props":705,"children":706},{},[707],{"type":47,"value":708},"Each failed or warning task has:",{"type":42,"tag":63,"props":710,"children":711},{},[712,722,732,742],{"type":42,"tag":67,"props":713,"children":714},{},[715,720],{"type":42,"tag":145,"props":716,"children":717},{},[718],{"type":47,"value":719},"name",{"type":47,"value":721}," — the task that failed",{"type":42,"tag":67,"props":723,"children":724},{},[725,730],{"type":42,"tag":145,"props":726,"children":727},{},[728],{"type":47,"value":729},"parentId",{"type":47,"value":731}," — links to the parent Job record (tells you which leg)",{"type":42,"tag":67,"props":733,"children":734},{},[735,740],{"type":42,"tag":145,"props":736,"children":737},{},[738],{"type":47,"value":739},"issues",{"type":47,"value":741}," — error messages with type and message fields",{"type":42,"tag":67,"props":743,"children":744},{},[745,750],{"type":42,"tag":145,"props":746,"children":747},{},[748],{"type":47,"value":749},"log.url",{"type":47,"value":751}," — URL to download the full task log",{"type":42,"tag":118,"props":753,"children":755},{"id":754},"step-2-categorize-the-failure",[756],{"type":47,"value":757},"Step 2: Categorize the failure",{"type":42,"tag":759,"props":760,"children":761},"table",{},[762,786],{"type":42,"tag":763,"props":764,"children":765},"thead",{},[766],{"type":42,"tag":767,"props":768,"children":769},"tr",{},[770,776,781],{"type":42,"tag":771,"props":772,"children":773},"th",{},[774],{"type":47,"value":775},"Pattern",{"type":42,"tag":771,"props":777,"children":778},{},[779],{"type":47,"value":780},"Category",{"type":42,"tag":771,"props":782,"children":783},{},[784],{"type":47,"value":785},"Owner",{"type":42,"tag":787,"props":788,"children":789},"tbody",{},[790,820,852,876,906,943,973,1010,1034,1070,1100,1123,1154,1187,1210],{"type":42,"tag":767,"props":791,"children":792},{},[793,810,815],{"type":42,"tag":794,"props":795,"children":796},"td",{},[797,803,805],{"type":42,"tag":88,"props":798,"children":800},{"className":799},[],[801],{"type":47,"value":802},"Binary Analysis Scan",{"type":47,"value":804}," task, ",{"type":42,"tag":88,"props":806,"children":808},{"className":807},[],[809],{"type":47,"value":93},{"type":42,"tag":794,"props":811,"children":812},{},[813],{"type":47,"value":814},"1ES infra tooling crash",{"type":42,"tag":794,"props":816,"children":817},{},[818],{"type":47,"value":819},"1ES PT team",{"type":42,"tag":767,"props":821,"children":822},{},[823,842,847],{"type":42,"tag":794,"props":824,"children":825},{},[826,832,834,840],{"type":42,"tag":88,"props":827,"children":829},{"className":828},[],[830],{"type":47,"value":831},"MSB3073",{"type":47,"value":833}," \u002F ",{"type":42,"tag":88,"props":835,"children":837},{"className":836},[],[838],{"type":47,"value":839},"NetAnalyzers.Package.csproj",{"type":47,"value":841}," error",{"type":42,"tag":794,"props":843,"children":844},{},[845],{"type":47,"value":846},"Build error",{"type":42,"tag":794,"props":848,"children":849},{},[850],{"type":47,"value":851},"Source code \u002F SDK",{"type":42,"tag":767,"props":853,"children":854},{},[855,866,871],{"type":42,"tag":794,"props":856,"children":857},{},[858,864],{"type":42,"tag":88,"props":859,"children":861},{"className":860},[],[862],{"type":47,"value":863},"Validate installer packages",{"type":47,"value":865}," — expected vs actual package list mismatch",{"type":42,"tag":794,"props":867,"children":868},{},[869],{"type":47,"value":870},"Installer validation regex",{"type":42,"tag":794,"props":872,"children":873},{},[874],{"type":47,"value":875},"dotnet\u002Finstaller",{"type":42,"tag":767,"props":877,"children":878},{},[879,896,901],{"type":42,"tag":794,"props":880,"children":881},{},[882,888,890],{"type":42,"tag":88,"props":883,"children":885},{"className":884},[],[886],{"type":47,"value":887},"Run Tests",{"type":47,"value":889}," with ",{"type":42,"tag":88,"props":891,"children":893},{"className":892},[],[894],{"type":47,"value":895},"SourcelinkTests.VerifySourcelinks",{"type":42,"tag":794,"props":897,"children":898},{},[899],{"type":47,"value":900},"Sourcelink validation",{"type":42,"tag":794,"props":902,"children":903},{},[904],{"type":47,"value":905},"dotnet\u002Fdotnet",{"type":42,"tag":767,"props":907,"children":908},{},[909,934,939],{"type":42,"tag":794,"props":910,"children":911},{},[912,917,918,924,926,932],{"type":42,"tag":88,"props":913,"children":915},{"className":914},[],[916],{"type":47,"value":887},{"type":47,"value":889},{"type":42,"tag":88,"props":919,"children":921},{"className":920},[],[922],{"type":47,"value":923},"The task has timed out",{"type":47,"value":925}," across many ",{"type":42,"tag":88,"props":927,"children":929},{"className":928},[],[930],{"type":47,"value":931},"SB_*_Validation_*",{"type":47,"value":933}," legs",{"type":42,"tag":794,"props":935,"children":936},{},[937],{"type":47,"value":938},"Run Tests timeout epidemic",{"type":42,"tag":794,"props":940,"children":941},{},[942],{"type":47,"value":905},{"type":42,"tag":767,"props":944,"children":945},{},[946,963,968],{"type":42,"tag":794,"props":947,"children":948},{},[949,954,956,962],{"type":42,"tag":88,"props":950,"children":952},{"className":951},[],[953],{"type":47,"value":887},{"type":47,"value":955}," with locale test failure (e.g., ",{"type":42,"tag":88,"props":957,"children":959},{"className":958},[],[960],{"type":47,"value":961},"tr-TR",{"type":47,"value":172},{"type":42,"tag":794,"props":964,"children":965},{},[966],{"type":47,"value":967},"Scenario test bug",{"type":42,"tag":794,"props":969,"children":970},{},[971],{"type":47,"value":972},"dotnet\u002Ftemplating",{"type":42,"tag":767,"props":974,"children":975},{},[976,1000,1005],{"type":42,"tag":794,"props":977,"children":978},{},[979,985,986,992,994],{"type":42,"tag":88,"props":980,"children":982},{"className":981},[],[983],{"type":47,"value":984},"Build",{"type":47,"value":889},{"type":42,"tag":88,"props":987,"children":989},{"className":988},[],[990],{"type":47,"value":991},"IBCMerge",{"type":47,"value":993}," \u002F PGO error in ",{"type":42,"tag":88,"props":995,"children":997},{"className":996},[],[998],{"type":47,"value":999},"Windows_Pgo_*",{"type":42,"tag":794,"props":1001,"children":1002},{},[1003],{"type":47,"value":1004},"PGO optimization failure",{"type":42,"tag":794,"props":1006,"children":1007},{},[1008],{"type":47,"value":1009},"dotnet\u002Fruntime",{"type":42,"tag":767,"props":1011,"children":1012},{},[1013,1024,1029],{"type":42,"tag":794,"props":1014,"children":1015},{},[1016,1022],{"type":42,"tag":88,"props":1017,"children":1019},{"className":1018},[],[1020],{"type":47,"value":1021},"Download Previous Build",{"type":47,"value":1023}," with \"Artifact not found\"",{"type":42,"tag":794,"props":1025,"children":1026},{},[1027],{"type":47,"value":1028},"Artifact cascade failure",{"type":42,"tag":794,"props":1030,"children":1031},{},[1032],{"type":47,"value":1033},"Prior build failed first",{"type":42,"tag":767,"props":1035,"children":1036},{},[1037,1060,1065],{"type":42,"tag":794,"props":1038,"children":1039},{},[1040,1045,1046,1051,1052,1058],{"type":42,"tag":88,"props":1041,"children":1043},{"className":1042},[],[1044],{"type":47,"value":984},{"type":47,"value":889},{"type":42,"tag":88,"props":1047,"children":1049},{"className":1048},[],[1050],{"type":47,"value":348},{"type":47,"value":833},{"type":42,"tag":88,"props":1053,"children":1055},{"className":1054},[],[1056],{"type":47,"value":1057},"tar",{"type":47,"value":1059}," \u002F download failure (exit code 2, \"not recoverable\")",{"type":42,"tag":794,"props":1061,"children":1062},{},[1063],{"type":47,"value":1064},"External resource fetch failure",{"type":42,"tag":794,"props":1066,"children":1067},{},[1068],{"type":47,"value":1069},"Retry first; if persistent, check URL\u002Fversion",{"type":42,"tag":767,"props":1071,"children":1072},{},[1073,1090,1095],{"type":42,"tag":794,"props":1074,"children":1075},{},[1076,1081,1082,1088],{"type":42,"tag":88,"props":1077,"children":1079},{"className":1078},[],[1080],{"type":47,"value":984},{"type":47,"value":889},{"type":42,"tag":88,"props":1083,"children":1085},{"className":1084},[],[1086],{"type":47,"value":1087},"npm ci",{"type":47,"value":1089}," ETIMEDOUT \u002F ECONNREFUSED",{"type":42,"tag":794,"props":1091,"children":1092},{},[1093],{"type":47,"value":1094},"npm network timeout",{"type":42,"tag":794,"props":1096,"children":1097},{},[1098],{"type":47,"value":1099},"Transient; shell retry wrapper needed",{"type":42,"tag":767,"props":1101,"children":1102},{},[1103,1113,1118],{"type":42,"tag":794,"props":1104,"children":1105},{},[1106,1111],{"type":42,"tag":88,"props":1107,"children":1109},{"className":1108},[],[1110],{"type":47,"value":984},{"type":47,"value":1112}," task with compilation errors",{"type":42,"tag":794,"props":1114,"children":1115},{},[1116],{"type":47,"value":1117},"Source code build break",{"type":42,"tag":794,"props":1119,"children":1120},{},[1121],{"type":47,"value":1122},"Varies by component",{"type":42,"tag":767,"props":1124,"children":1125},{},[1126,1144,1149],{"type":42,"tag":794,"props":1127,"children":1128},{},[1129,1135,1136,1142],{"type":42,"tag":88,"props":1130,"children":1132},{"className":1131},[],[1133],{"type":47,"value":1134},"Crossgen.targets",{"type":47,"value":889},{"type":42,"tag":88,"props":1137,"children":1139},{"className":1138},[],[1140],{"type":47,"value":1141},"exit code 57005",{"type":47,"value":1143}," (0xDEAD)",{"type":42,"tag":794,"props":1145,"children":1146},{},[1147],{"type":47,"value":1148},"crossgen2 fatal crash",{"type":42,"tag":794,"props":1150,"children":1151},{},[1152],{"type":47,"value":1153},"dotnet\u002Fruntime (area-crossgen2-coreclr)",{"type":42,"tag":767,"props":1155,"children":1156},{},[1157,1177,1182],{"type":42,"tag":794,"props":1158,"children":1159},{},[1160,1162,1168,1169,1175],{"type":47,"value":1161},"ESRP ",{"type":42,"tag":88,"props":1163,"children":1165},{"className":1164},[],[1166],{"type":47,"value":1167},"MacSignFailed",{"type":47,"value":833},{"type":42,"tag":88,"props":1170,"children":1172},{"className":1171},[],[1173],{"type":47,"value":1174},"FailDoNotRetry",{"type":47,"value":1176}," \u002F notarization errors",{"type":42,"tag":794,"props":1178,"children":1179},{},[1180],{"type":47,"value":1181},"Signing\u002Fnotarization failure",{"type":42,"tag":794,"props":1183,"children":1184},{},[1185],{"type":47,"value":1186},"dotnet\u002Fsdk or ESRP team",{"type":42,"tag":767,"props":1188,"children":1189},{},[1190,1200,1205],{"type":42,"tag":794,"props":1191,"children":1192},{},[1193,1198],{"type":42,"tag":88,"props":1194,"children":1196},{"className":1195},[],[1197],{"type":47,"value":93},{"type":47,"value":1199}," on cross-compilation legs in containers",{"type":42,"tag":794,"props":1201,"children":1202},{},[1203],{"type":47,"value":1204},"Container OOM kill",{"type":42,"tag":794,"props":1206,"children":1207},{},[1208],{"type":47,"value":1209},"Infrastructure — pool\u002Fcontainer config",{"type":42,"tag":767,"props":1211,"children":1212},{},[1213,1226,1231],{"type":42,"tag":794,"props":1214,"children":1215},{},[1216,1218,1224],{"type":47,"value":1217},"Build start == finish, HTTP 204 timeline, ",{"type":42,"tag":88,"props":1219,"children":1221},{"className":1220},[],[1222],{"type":47,"value":1223},"validationResults",{"type":47,"value":1225}," has errors",{"type":42,"tag":794,"props":1227,"children":1228},{},[1229],{"type":47,"value":1230},"YAML pre-flight rejection",{"type":42,"tag":794,"props":1232,"children":1233},{},[1234],{"type":47,"value":1235},"PR author — pipeline YAML invalid",{"type":42,"tag":618,"props":1237,"children":1238},{},[1239],{"type":42,"tag":50,"props":1240,"children":1241},{},[1242,1247,1249,1254,1256,1261],{"type":42,"tag":145,"props":1243,"children":1244},{},[1245],{"type":47,"value":1246},"Routing: AzDO tests vs Helix tests.",{"type":47,"value":1248}," If the test runs directly as an AzDO task (e.g., ",{"type":42,"tag":88,"props":1250,"children":1252},{"className":1251},[],[1253],{"type":47,"value":887},{"type":47,"value":1255}," in ",{"type":42,"tag":88,"props":1257,"children":1259},{"className":1258},[],[1260],{"type":47,"value":931},{"type":47,"value":1262}," legs — SourcelinkTests, scenario tests), it's pipeline-investigation. If the test is submitted to Helix (has a Helix job ID, work items, console logs at helix.dot.net), use helix-investigation.",{"type":42,"tag":118,"props":1264,"children":1266},{"id":1265},"step-3-download-and-analyze-task-logs",[1267],{"type":47,"value":1268},"Step 3: Download and analyze task logs",{"type":42,"tag":381,"props":1270,"children":1272},{"className":383,"code":1271,"language":385,"meta":386,"style":386},"TOKEN=$(azureauth ado token --output token --prompt-hint \"copilot-cli\")\ncurl -s -H \"Authorization: Bearer $TOKEN\" \"{log_url}\" > \u002Ftmp\u002Ftask-{buildId}.log\n",[1273],{"type":42,"tag":88,"props":1274,"children":1275},{"__ignoreMap":386},[1276,1327],{"type":42,"tag":392,"props":1277,"children":1278},{"class":394,"line":395},[1279,1283,1287,1291,1295,1299,1303,1307,1311,1315,1319,1323],{"type":42,"tag":392,"props":1280,"children":1281},{"style":409},[1282],{"type":47,"value":412},{"type":42,"tag":392,"props":1284,"children":1285},{"style":415},[1286],{"type":47,"value":418},{"type":42,"tag":392,"props":1288,"children":1289},{"style":421},[1290],{"type":47,"value":506},{"type":42,"tag":392,"props":1292,"children":1293},{"style":427},[1294],{"type":47,"value":511},{"type":42,"tag":392,"props":1296,"children":1297},{"style":427},[1298],{"type":47,"value":516},{"type":42,"tag":392,"props":1300,"children":1301},{"style":427},[1302],{"type":47,"value":521},{"type":42,"tag":392,"props":1304,"children":1305},{"style":427},[1306],{"type":47,"value":516},{"type":42,"tag":392,"props":1308,"children":1309},{"style":427},[1310],{"type":47,"value":530},{"type":42,"tag":392,"props":1312,"children":1313},{"style":415},[1314],{"type":47,"value":535},{"type":42,"tag":392,"props":1316,"children":1317},{"style":427},[1318],{"type":47,"value":540},{"type":42,"tag":392,"props":1320,"children":1321},{"style":415},[1322],{"type":47,"value":545},{"type":42,"tag":392,"props":1324,"children":1325},{"style":415},[1326],{"type":47,"value":470},{"type":42,"tag":392,"props":1328,"children":1329},{"class":394,"line":405},[1330,1334,1338,1342,1346,1350,1354,1358,1362,1367,1371,1376],{"type":42,"tag":392,"props":1331,"children":1332},{"style":421},[1333],{"type":47,"value":348},{"type":42,"tag":392,"props":1335,"children":1336},{"style":427},[1337],{"type":47,"value":579},{"type":42,"tag":392,"props":1339,"children":1340},{"style":427},[1341],{"type":47,"value":584},{"type":42,"tag":392,"props":1343,"children":1344},{"style":415},[1345],{"type":47,"value":535},{"type":42,"tag":392,"props":1347,"children":1348},{"style":427},[1349],{"type":47,"value":593},{"type":42,"tag":392,"props":1351,"children":1352},{"style":409},[1353],{"type":47,"value":598},{"type":42,"tag":392,"props":1355,"children":1356},{"style":415},[1357],{"type":47,"value":545},{"type":42,"tag":392,"props":1359,"children":1360},{"style":415},[1361],{"type":47,"value":535},{"type":42,"tag":392,"props":1363,"children":1364},{"style":427},[1365],{"type":47,"value":1366},"{log_url}",{"type":42,"tag":392,"props":1368,"children":1369},{"style":415},[1370],{"type":47,"value":545},{"type":42,"tag":392,"props":1372,"children":1373},{"style":415},[1374],{"type":47,"value":1375}," >",{"type":42,"tag":392,"props":1377,"children":1378},{"style":427},[1379],{"type":47,"value":1380}," \u002Ftmp\u002Ftask-{buildId}.log\n",{"type":42,"tag":50,"props":1382,"children":1383},{},[1384],{"type":47,"value":1385},"Key patterns to search for:",{"type":42,"tag":63,"props":1387,"children":1388},{},[1389,1402,1415,1439,1455,1488,1517,1527],{"type":42,"tag":67,"props":1390,"children":1391},{},[1392,1400],{"type":42,"tag":145,"props":1393,"children":1394},{},[1395],{"type":42,"tag":88,"props":1396,"children":1398},{"className":1397},[],[1399],{"type":47,"value":93},{"type":47,"value":1401}," — process terminated by signal (OOM, timeout). Check for memory-related warnings before the crash.",{"type":42,"tag":67,"props":1403,"children":1404},{},[1405,1413],{"type":42,"tag":145,"props":1406,"children":1407},{},[1408],{"type":42,"tag":88,"props":1409,"children":1411},{"className":1410},[],[1412],{"type":47,"value":1141},{"type":47,"value":1414}," (0xDEAD) — crossgen2 fatal error. Check which assembly was being compiled. Intermittent crashes need crash dumps for diagnosis.",{"type":42,"tag":67,"props":1416,"children":1417},{},[1418,1427,1428,1437],{"type":42,"tag":145,"props":1419,"children":1420},{},[1421],{"type":42,"tag":88,"props":1422,"children":1424},{"className":1423},[],[1425],{"type":47,"value":1426},"[ERROR]",{"type":47,"value":833},{"type":42,"tag":145,"props":1429,"children":1430},{},[1431],{"type":42,"tag":88,"props":1432,"children":1434},{"className":1433},[],[1435],{"type":47,"value":1436},"##[error]",{"type":47,"value":1438}," — explicit error messages from the task",{"type":42,"tag":67,"props":1440,"children":1441},{},[1442,1453],{"type":42,"tag":145,"props":1443,"children":1444},{},[1445,1451],{"type":42,"tag":88,"props":1446,"children":1448},{"className":1447},[],[1449],{"type":47,"value":1450},"[WARNING]",{"type":47,"value":1452}," floods",{"type":47,"value":1454}," — excessive warnings (e.g., 1000+ hardlink failures) indicate resource exhaustion",{"type":42,"tag":67,"props":1456,"children":1457},{},[1458,1463,1465,1470,1472,1478,1480,1486],{"type":42,"tag":145,"props":1459,"children":1460},{},[1461],{"type":47,"value":1462},"MSBuild error codes",{"type":47,"value":1464}," — ",{"type":42,"tag":88,"props":1466,"children":1468},{"className":1467},[],[1469],{"type":47,"value":831},{"type":47,"value":1471}," (command failed), ",{"type":42,"tag":88,"props":1473,"children":1475},{"className":1474},[],[1476],{"type":47,"value":1477},"NETSDK*",{"type":47,"value":1479}," (SDK errors), ",{"type":42,"tag":88,"props":1481,"children":1483},{"className":1482},[],[1484],{"type":47,"value":1485},"NU1105",{"type":47,"value":1487}," (invalid target framework — often transient forward flow)",{"type":42,"tag":67,"props":1489,"children":1490},{},[1491,1499,1500,1508,1510,1515],{"type":42,"tag":145,"props":1492,"children":1493},{},[1494],{"type":42,"tag":88,"props":1495,"children":1497},{"className":1496},[],[1498],{"type":47,"value":1167},{"type":47,"value":833},{"type":42,"tag":145,"props":1501,"children":1502},{},[1503],{"type":42,"tag":88,"props":1504,"children":1506},{"className":1505},[],[1507],{"type":47,"value":1174},{"type":47,"value":1509}," — ESRP signing rejection. ",{"type":42,"tag":88,"props":1511,"children":1513},{"className":1512},[],[1514],{"type":47,"value":1174},{"type":47,"value":1516}," means deterministic content failure, not transient.",{"type":42,"tag":67,"props":1518,"children":1519},{},[1520,1525],{"type":42,"tag":145,"props":1521,"children":1522},{},[1523],{"type":47,"value":1524},"Stack traces",{"type":47,"value":1526}," — .NET exceptions in test output",{"type":42,"tag":67,"props":1528,"children":1529},{},[1530,1539,1540,1549],{"type":42,"tag":145,"props":1531,"children":1532},{},[1533],{"type":42,"tag":88,"props":1534,"children":1536},{"className":1535},[],[1537],{"type":47,"value":1538},"ETIMEDOUT",{"type":47,"value":833},{"type":42,"tag":145,"props":1541,"children":1542},{},[1543],{"type":42,"tag":88,"props":1544,"children":1546},{"className":1545},[],[1547],{"type":47,"value":1548},"ECONNREFUSED",{"type":47,"value":1550}," — TCP-level network failures in npm\u002Fcurl that aren't retried by default",{"type":42,"tag":118,"props":1552,"children":1554},{"id":1553},"step-4-determine-if-its-a-one-off-or-recurring",[1555],{"type":47,"value":1556},"Step 4: Determine if it's a one-off or recurring",{"type":42,"tag":50,"props":1558,"children":1559},{},[1560],{"type":47,"value":1561},"Query failed builds over a time range and count how many hit the same failure:",{"type":42,"tag":381,"props":1563,"children":1565},{"className":383,"code":1564,"language":385,"meta":386,"style":386},"# Get failed builds for a pipeline definition\ncurl -s -H \"Authorization: Bearer $TOKEN\" \\\n  \"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds?definitions={defId}&resultFilter=failed&minTime={iso8601}&$top=200&api-version=7.1\"\n",[1566],{"type":42,"tag":88,"props":1567,"children":1568},{"__ignoreMap":386},[1569,1577,1613],{"type":42,"tag":392,"props":1570,"children":1571},{"class":394,"line":395},[1572],{"type":42,"tag":392,"props":1573,"children":1574},{"style":399},[1575],{"type":47,"value":1576},"# Get failed builds for a pipeline definition\n",{"type":42,"tag":392,"props":1578,"children":1579},{"class":394,"line":405},[1580,1584,1588,1592,1596,1600,1604,1608],{"type":42,"tag":392,"props":1581,"children":1582},{"style":421},[1583],{"type":47,"value":348},{"type":42,"tag":392,"props":1585,"children":1586},{"style":427},[1587],{"type":47,"value":579},{"type":42,"tag":392,"props":1589,"children":1590},{"style":427},[1591],{"type":47,"value":584},{"type":42,"tag":392,"props":1593,"children":1594},{"style":415},[1595],{"type":47,"value":535},{"type":42,"tag":392,"props":1597,"children":1598},{"style":427},[1599],{"type":47,"value":593},{"type":42,"tag":392,"props":1601,"children":1602},{"style":409},[1603],{"type":47,"value":598},{"type":42,"tag":392,"props":1605,"children":1606},{"style":415},[1607],{"type":47,"value":545},{"type":42,"tag":392,"props":1609,"children":1610},{"style":409},[1611],{"type":47,"value":1612}," \\\n",{"type":42,"tag":392,"props":1614,"children":1615},{"class":394,"line":473},[1616,1621,1626,1631,1636],{"type":42,"tag":392,"props":1617,"children":1618},{"style":415},[1619],{"type":47,"value":1620},"  \"",{"type":42,"tag":392,"props":1622,"children":1623},{"style":427},[1624],{"type":47,"value":1625},"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds?definitions={defId}&resultFilter=failed&minTime={iso8601}&",{"type":42,"tag":392,"props":1627,"children":1628},{"style":409},[1629],{"type":47,"value":1630},"$top",{"type":42,"tag":392,"props":1632,"children":1633},{"style":427},[1634],{"type":47,"value":1635},"=200&api-version=7.1",{"type":42,"tag":392,"props":1637,"children":1638},{"style":415},[1639],{"type":47,"value":616},{"type":42,"tag":618,"props":1641,"children":1642},{},[1643],{"type":42,"tag":50,"props":1644,"children":1645},{},[1646,1648,1653,1655,1661],{"type":47,"value":1647},"💡 ",{"type":42,"tag":145,"props":1649,"children":1650},{},[1651],{"type":47,"value":1652},"Rolling builds give cleaner signal.",{"type":47,"value":1654}," Add ",{"type":42,"tag":88,"props":1656,"children":1658},{"className":1657},[],[1659],{"type":47,"value":1660},"&reasonFilter=schedule",{"type":47,"value":1662}," to filter to scheduled builds only. PR builds include broken branches that inflate failure counts and obscure systemic trends.",{"type":42,"tag":50,"props":1664,"children":1665},{},[1666],{"type":47,"value":1667},"For each build, check its timeline for the same failure pattern. Track in a SQL table:",{"type":42,"tag":381,"props":1669,"children":1673},{"className":1670,"code":1671,"language":1672,"meta":386,"style":386},"language-sql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","CREATE TABLE pipeline_failures (\n  build_id INT, branch TEXT, queued TEXT, failure_category TEXT, \n  failed_task TEXT, notes TEXT\n);\n","sql",[1674],{"type":42,"tag":88,"props":1675,"children":1676},{"__ignoreMap":386},[1677,1685,1693,1701],{"type":42,"tag":392,"props":1678,"children":1679},{"class":394,"line":395},[1680],{"type":42,"tag":392,"props":1681,"children":1682},{},[1683],{"type":47,"value":1684},"CREATE TABLE pipeline_failures (\n",{"type":42,"tag":392,"props":1686,"children":1687},{"class":394,"line":405},[1688],{"type":42,"tag":392,"props":1689,"children":1690},{},[1691],{"type":47,"value":1692},"  build_id INT, branch TEXT, queued TEXT, failure_category TEXT, \n",{"type":42,"tag":392,"props":1694,"children":1695},{"class":394,"line":473},[1696],{"type":42,"tag":392,"props":1697,"children":1698},{},[1699],{"type":47,"value":1700},"  failed_task TEXT, notes TEXT\n",{"type":42,"tag":392,"props":1702,"children":1703},{"class":394,"line":483},[1704],{"type":42,"tag":392,"props":1705,"children":1706},{},[1707],{"type":47,"value":1708},");\n",{"type":42,"tag":50,"props":1710,"children":1711},{},[1712],{"type":47,"value":1713},"Look for:",{"type":42,"tag":63,"props":1715,"children":1716},{},[1717,1727,1737],{"type":42,"tag":67,"props":1718,"children":1719},{},[1720,1725],{"type":42,"tag":145,"props":1721,"children":1722},{},[1723],{"type":47,"value":1724},"Burst pattern",{"type":47,"value":1726}," — many occurrences in a short window → something changed (tooling update, artifact size growth)",{"type":42,"tag":67,"props":1728,"children":1729},{},[1730,1735],{"type":42,"tag":145,"props":1731,"children":1732},{},[1733],{"type":47,"value":1734},"Steady trickle",{"type":47,"value":1736}," — consistent low rate → chronic issue",{"type":42,"tag":67,"props":1738,"children":1739},{},[1740,1745],{"type":42,"tag":145,"props":1741,"children":1742},{},[1743],{"type":47,"value":1744},"Single occurrence",{"type":47,"value":1746}," — likely transient, not worth investigating further",{"type":42,"tag":118,"props":1748,"children":1750},{"id":1749},"step-5-check-for-known-issues",[1751],{"type":47,"value":1752},"Step 5: Check for known issues",{"type":42,"tag":50,"props":1754,"children":1755},{},[1756],{"type":47,"value":1757},"Search the relevant repo for existing issues:",{"type":42,"tag":381,"props":1759,"children":1761},{"className":383,"code":1760,"language":385,"meta":386,"style":386},"gh search issues \"{failure_pattern}\" --repo dotnet\u002Fdotnet --limit 5\n",[1762],{"type":42,"tag":88,"props":1763,"children":1764},{"__ignoreMap":386},[1765],{"type":42,"tag":392,"props":1766,"children":1767},{"class":394,"line":395},[1768,1772,1777,1782,1786,1791,1795,1800,1805,1810],{"type":42,"tag":392,"props":1769,"children":1770},{"style":421},[1771],{"type":47,"value":359},{"type":42,"tag":392,"props":1773,"children":1774},{"style":427},[1775],{"type":47,"value":1776}," search",{"type":42,"tag":392,"props":1778,"children":1779},{"style":427},[1780],{"type":47,"value":1781}," issues",{"type":42,"tag":392,"props":1783,"children":1784},{"style":415},[1785],{"type":47,"value":535},{"type":42,"tag":392,"props":1787,"children":1788},{"style":427},[1789],{"type":47,"value":1790},"{failure_pattern}",{"type":42,"tag":392,"props":1792,"children":1793},{"style":415},[1794],{"type":47,"value":545},{"type":42,"tag":392,"props":1796,"children":1797},{"style":427},[1798],{"type":47,"value":1799}," --repo",{"type":42,"tag":392,"props":1801,"children":1802},{"style":427},[1803],{"type":47,"value":1804}," dotnet\u002Fdotnet",{"type":42,"tag":392,"props":1806,"children":1807},{"style":427},[1808],{"type":47,"value":1809}," --limit",{"type":42,"tag":392,"props":1811,"children":1813},{"style":1812},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1814],{"type":47,"value":1815}," 5\n",{"type":42,"tag":50,"props":1817,"children":1818},{},[1819],{"type":47,"value":1820},"If found, check if the issue is being worked on. If not found and the failure is recurring, consider filing one.",{"type":42,"tag":118,"props":1822,"children":1824},{"id":1823},"step-6-root-cause-analysis",[1825],{"type":47,"value":1826},"Step 6: Root cause analysis",{"type":42,"tag":50,"props":1828,"children":1829},{},[1830],{"type":47,"value":1831},"For non-obvious failures, pull up the source code of the failing test or tool:",{"type":42,"tag":381,"props":1833,"children":1835},{"className":383,"code":1834,"language":385,"meta":386,"style":386},"gh api \"repos\u002F{owner}\u002F{repo}\u002Fcontents\u002F{path}\" --jq '.content' | base64 -d\n",[1836],{"type":42,"tag":88,"props":1837,"children":1838},{"__ignoreMap":386},[1839],{"type":42,"tag":392,"props":1840,"children":1841},{"class":394,"line":395},[1842,1846,1851,1855,1860,1864,1869,1874,1879,1884,1889,1894],{"type":42,"tag":392,"props":1843,"children":1844},{"style":421},[1845],{"type":47,"value":359},{"type":42,"tag":392,"props":1847,"children":1848},{"style":427},[1849],{"type":47,"value":1850}," api",{"type":42,"tag":392,"props":1852,"children":1853},{"style":415},[1854],{"type":47,"value":535},{"type":42,"tag":392,"props":1856,"children":1857},{"style":427},[1858],{"type":47,"value":1859},"repos\u002F{owner}\u002F{repo}\u002Fcontents\u002F{path}",{"type":42,"tag":392,"props":1861,"children":1862},{"style":415},[1863],{"type":47,"value":545},{"type":42,"tag":392,"props":1865,"children":1866},{"style":427},[1867],{"type":47,"value":1868}," --jq",{"type":42,"tag":392,"props":1870,"children":1871},{"style":415},[1872],{"type":47,"value":1873}," '",{"type":42,"tag":392,"props":1875,"children":1876},{"style":427},[1877],{"type":47,"value":1878},".content",{"type":42,"tag":392,"props":1880,"children":1881},{"style":415},[1882],{"type":47,"value":1883},"'",{"type":42,"tag":392,"props":1885,"children":1886},{"style":415},[1887],{"type":47,"value":1888}," |",{"type":42,"tag":392,"props":1890,"children":1891},{"style":421},[1892],{"type":47,"value":1893}," base64",{"type":42,"tag":392,"props":1895,"children":1896},{"style":427},[1897],{"type":47,"value":1898}," -d\n",{"type":42,"tag":50,"props":1900,"children":1901},{},[1902],{"type":47,"value":1713},{"type":42,"tag":63,"props":1904,"children":1905},{},[1906,1929,1939,1949],{"type":42,"tag":67,"props":1907,"children":1908},{},[1909,1914,1915,1921,1923],{"type":42,"tag":145,"props":1910,"children":1911},{},[1912],{"type":47,"value":1913},"Unbounded parallelism",{"type":47,"value":1464},{"type":42,"tag":88,"props":1916,"children":1918},{"className":1917},[],[1919],{"type":47,"value":1920},"Parallel.ForEach",{"type":47,"value":1922}," without ",{"type":42,"tag":88,"props":1924,"children":1926},{"className":1925},[],[1927],{"type":47,"value":1928},"MaxDegreeOfParallelism",{"type":42,"tag":67,"props":1930,"children":1931},{},[1932,1937],{"type":42,"tag":145,"props":1933,"children":1934},{},[1935],{"type":47,"value":1936},"Tight timeouts",{"type":47,"value":1938}," — processes timing out under contention",{"type":42,"tag":67,"props":1940,"children":1941},{},[1942,1947],{"type":42,"tag":145,"props":1943,"children":1944},{},[1945],{"type":47,"value":1946},"Resource assumptions",{"type":47,"value":1948}," — disk space, memory, network that may not hold in CI",{"type":42,"tag":67,"props":1950,"children":1951},{},[1952,1957],{"type":42,"tag":145,"props":1953,"children":1954},{},[1955],{"type":47,"value":1956},"Non-determinism",{"type":47,"value":1958}," — race conditions, order-dependent assertions",{"type":42,"tag":118,"props":1960,"children":1962},{"id":1961},"step-6b-binlog-analysis-for-deep-investigation",[1963],{"type":47,"value":1964},"Step 6b: Binlog analysis for deep investigation",{"type":42,"tag":50,"props":1966,"children":1967},{},[1968],{"type":47,"value":1969},"When task logs don't reveal enough, download the build's binlog artifacts for detailed MSBuild analysis:",{"type":42,"tag":381,"props":1971,"children":1973},{"className":383,"code":1972,"language":385,"meta":386,"style":386},"# List build artifacts to find binlogs\ncurl -s -H \"Authorization: Bearer $TOKEN\" \\\n  \"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds\u002F{buildId}\u002Fartifacts?api-version=7.1\"\n\n# Download the artifact zip\ncurl -s -H \"Authorization: Bearer $TOKEN\" -o \u002Ftmp\u002Flogs.zip \"{downloadUrl}\"\nunzip -o \u002Ftmp\u002Flogs.zip \"*\u002FBuild.binlog\" -d \u002Ftmp\u002Fbinlogs\u002F\n",[1974],{"type":42,"tag":88,"props":1975,"children":1976},{"__ignoreMap":386},[1977,1985,2020,2036,2043,2051,2104],{"type":42,"tag":392,"props":1978,"children":1979},{"class":394,"line":395},[1980],{"type":42,"tag":392,"props":1981,"children":1982},{"style":399},[1983],{"type":47,"value":1984},"# List build artifacts to find binlogs\n",{"type":42,"tag":392,"props":1986,"children":1987},{"class":394,"line":405},[1988,1992,1996,2000,2004,2008,2012,2016],{"type":42,"tag":392,"props":1989,"children":1990},{"style":421},[1991],{"type":47,"value":348},{"type":42,"tag":392,"props":1993,"children":1994},{"style":427},[1995],{"type":47,"value":579},{"type":42,"tag":392,"props":1997,"children":1998},{"style":427},[1999],{"type":47,"value":584},{"type":42,"tag":392,"props":2001,"children":2002},{"style":415},[2003],{"type":47,"value":535},{"type":42,"tag":392,"props":2005,"children":2006},{"style":427},[2007],{"type":47,"value":593},{"type":42,"tag":392,"props":2009,"children":2010},{"style":409},[2011],{"type":47,"value":598},{"type":42,"tag":392,"props":2013,"children":2014},{"style":415},[2015],{"type":47,"value":545},{"type":42,"tag":392,"props":2017,"children":2018},{"style":409},[2019],{"type":47,"value":1612},{"type":42,"tag":392,"props":2021,"children":2022},{"class":394,"line":473},[2023,2027,2032],{"type":42,"tag":392,"props":2024,"children":2025},{"style":415},[2026],{"type":47,"value":1620},{"type":42,"tag":392,"props":2028,"children":2029},{"style":427},[2030],{"type":47,"value":2031},"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds\u002F{buildId}\u002Fartifacts?api-version=7.1",{"type":42,"tag":392,"props":2033,"children":2034},{"style":415},[2035],{"type":47,"value":616},{"type":42,"tag":392,"props":2037,"children":2038},{"class":394,"line":483},[2039],{"type":42,"tag":392,"props":2040,"children":2041},{"emptyLinePlaceholder":477},[2042],{"type":47,"value":480},{"type":42,"tag":392,"props":2044,"children":2045},{"class":394,"line":492},[2046],{"type":42,"tag":392,"props":2047,"children":2048},{"style":399},[2049],{"type":47,"value":2050},"# Download the artifact zip\n",{"type":42,"tag":392,"props":2052,"children":2053},{"class":394,"line":552},[2054,2058,2062,2066,2070,2074,2078,2082,2086,2091,2095,2100],{"type":42,"tag":392,"props":2055,"children":2056},{"style":421},[2057],{"type":47,"value":348},{"type":42,"tag":392,"props":2059,"children":2060},{"style":427},[2061],{"type":47,"value":579},{"type":42,"tag":392,"props":2063,"children":2064},{"style":427},[2065],{"type":47,"value":584},{"type":42,"tag":392,"props":2067,"children":2068},{"style":415},[2069],{"type":47,"value":535},{"type":42,"tag":392,"props":2071,"children":2072},{"style":427},[2073],{"type":47,"value":593},{"type":42,"tag":392,"props":2075,"children":2076},{"style":409},[2077],{"type":47,"value":598},{"type":42,"tag":392,"props":2079,"children":2080},{"style":415},[2081],{"type":47,"value":545},{"type":42,"tag":392,"props":2083,"children":2084},{"style":427},[2085],{"type":47,"value":460},{"type":42,"tag":392,"props":2087,"children":2088},{"style":427},[2089],{"type":47,"value":2090}," \u002Ftmp\u002Flogs.zip",{"type":42,"tag":392,"props":2092,"children":2093},{"style":415},[2094],{"type":47,"value":535},{"type":42,"tag":392,"props":2096,"children":2097},{"style":427},[2098],{"type":47,"value":2099},"{downloadUrl}",{"type":42,"tag":392,"props":2101,"children":2102},{"style":415},[2103],{"type":47,"value":616},{"type":42,"tag":392,"props":2105,"children":2106},{"class":394,"line":560},[2107,2112,2116,2120,2124,2129,2133,2138],{"type":42,"tag":392,"props":2108,"children":2109},{"style":421},[2110],{"type":47,"value":2111},"unzip",{"type":42,"tag":392,"props":2113,"children":2114},{"style":427},[2115],{"type":47,"value":460},{"type":42,"tag":392,"props":2117,"children":2118},{"style":427},[2119],{"type":47,"value":2090},{"type":42,"tag":392,"props":2121,"children":2122},{"style":415},[2123],{"type":47,"value":535},{"type":42,"tag":392,"props":2125,"children":2126},{"style":427},[2127],{"type":47,"value":2128},"*\u002FBuild.binlog",{"type":42,"tag":392,"props":2130,"children":2131},{"style":415},[2132],{"type":47,"value":545},{"type":42,"tag":392,"props":2134,"children":2135},{"style":427},[2136],{"type":47,"value":2137}," -d",{"type":42,"tag":392,"props":2139,"children":2140},{"style":427},[2141],{"type":47,"value":2142}," \u002Ftmp\u002Fbinlogs\u002F\n",{"type":42,"tag":50,"props":2144,"children":2145},{},[2146,2148,2153,2155,2160,2162,2168,2170,2175,2177,2182],{"type":47,"value":2147},"Then use the binlog MCP tools (",{"type":42,"tag":88,"props":2149,"children":2151},{"className":2150},[],[2152],{"type":47,"value":314},{"type":47,"value":2154},", wired under the ",{"type":42,"tag":88,"props":2156,"children":2158},{"className":2157},[],[2159],{"type":47,"value":306},{"type":47,"value":2161}," MCP namespace). Call ",{"type":42,"tag":88,"props":2163,"children":2165},{"className":2164},[],[2166],{"type":47,"value":2167},"tools\u002Flist",{"type":47,"value":2169}," on the ",{"type":42,"tag":88,"props":2171,"children":2173},{"className":2172},[],[2174],{"type":47,"value":306},{"type":47,"value":2176}," namespace first if you're unsure which capabilities your server version exposes — names below are typical for ",{"type":42,"tag":88,"props":2178,"children":2180},{"className":2179},[],[2181],{"type":47,"value":314},{"type":47,"value":2183},":",{"type":42,"tag":63,"props":2185,"children":2186},{},[2187,2204,2235,2266,2307,2324,2342],{"type":42,"tag":67,"props":2188,"children":2189},{},[2190,2195,2197,2203],{"type":42,"tag":145,"props":2191,"children":2192},{},[2193],{"type":47,"value":2194},"Build configuration + which target failed",{"type":47,"value":2196}," (typically ",{"type":42,"tag":88,"props":2198,"children":2200},{"className":2199},[],[2201],{"type":47,"value":2202},"binlog_overview",{"type":47,"value":172},{"type":42,"tag":67,"props":2205,"children":2206},{},[2207,2212,2213,2219,2221,2227,2228,2234],{"type":42,"tag":145,"props":2208,"children":2209},{},[2210],{"type":47,"value":2211},"Structured diagnostics list",{"type":47,"value":889},{"type":42,"tag":88,"props":2214,"children":2216},{"className":2215},[],[2217],{"type":47,"value":2218},"{code, message, file, line, column, projectFile, targetName, taskName}",{"type":47,"value":2220}," per row (typically ",{"type":42,"tag":88,"props":2222,"children":2224},{"className":2223},[],[2225],{"type":47,"value":2226},"binlog_errors",{"type":47,"value":833},{"type":42,"tag":88,"props":2229,"children":2231},{"className":2230},[],[2232],{"type":47,"value":2233},"binlog_warnings",{"type":47,"value":172},{"type":42,"tag":67,"props":2236,"children":2237},{},[2238,2243,2245,2250,2251,2257,2258,2264],{"type":42,"tag":145,"props":2239,"children":2240},{},[2241],{"type":47,"value":2242},"Find specific patterns",{"type":47,"value":2244}," like ",{"type":42,"tag":88,"props":2246,"children":2248},{"className":2247},[],[2249],{"type":47,"value":1167},{"type":47,"value":325},{"type":42,"tag":88,"props":2252,"children":2254},{"className":2253},[],[2255],{"type":47,"value":2256},"crossgen2",{"type":47,"value":2196},{"type":42,"tag":88,"props":2259,"children":2261},{"className":2260},[],[2262],{"type":47,"value":2263},"binlog_search",{"type":47,"value":2265},", same StructuredLog-Viewer query syntax)",{"type":42,"tag":67,"props":2267,"children":2268},{},[2269,2274,2276,2282,2284,2290,2292,2298,2300,2306],{"type":42,"tag":145,"props":2270,"children":2271},{},[2272],{"type":47,"value":2273},"Exact command lines + parameters for failed tasks",{"type":47,"value":2275}," — Exec's ",{"type":42,"tag":88,"props":2277,"children":2279},{"className":2278},[],[2280],{"type":47,"value":2281},"Command",{"type":47,"value":2283},", RAR's ",{"type":42,"tag":88,"props":2285,"children":2287},{"className":2286},[],[2288],{"type":47,"value":2289},"Assemblies",{"type":47,"value":2291},"\u002F",{"type":42,"tag":88,"props":2293,"children":2295},{"className":2294},[],[2296],{"type":47,"value":2297},"AppConfigFile",{"type":47,"value":2299},", etc. (typically ",{"type":42,"tag":88,"props":2301,"children":2303},{"className":2302},[],[2304],{"type":47,"value":2305},"binlog_task_details",{"type":47,"value":172},{"type":42,"tag":67,"props":2308,"children":2309},{},[2310,2315,2317,2323],{"type":42,"tag":145,"props":2311,"children":2312},{},[2313],{"type":47,"value":2314},"All tasks in a target",{"type":47,"value":2316}," — useful when one target invokes a task 126 times and only one failed (typically ",{"type":42,"tag":88,"props":2318,"children":2320},{"className":2319},[],[2321],{"type":47,"value":2322},"binlog_tasks_in_target",{"type":47,"value":172},{"type":42,"tag":67,"props":2325,"children":2326},{},[2327,2332,2334,2340],{"type":42,"tag":145,"props":2328,"children":2329},{},[2330],{"type":47,"value":2331},"Automated \"what broke\" report",{"type":47,"value":2333}," — newer servers include a root-cause diagnostic (typically ",{"type":42,"tag":88,"props":2335,"children":2337},{"className":2336},[],[2338],{"type":47,"value":2339},"binlog_diagnose",{"type":47,"value":2341},"; not present on older servers)",{"type":42,"tag":67,"props":2343,"children":2344},{},[2345,2350,2352,2358],{"type":42,"tag":145,"props":2346,"children":2347},{},[2348],{"type":47,"value":2349},"Trace where a property got its value",{"type":47,"value":2351}," across imports\u002Ftargets\u002Ftasks (typically ",{"type":42,"tag":88,"props":2353,"children":2355},{"className":2354},[],[2356],{"type":47,"value":2357},"binlog_explain_property",{"type":47,"value":2359},"; newer servers only)",{"type":42,"tag":618,"props":2361,"children":2362},{},[2363],{"type":42,"tag":50,"props":2364,"children":2365},{},[2366],{"type":47,"value":2367},"💡 Binlogs capture the full MSBuild execution including command lines, environment variables, and interleaved output that task logs lose. Essential for signing, crossgen2, and SBRP failures.",{"type":42,"tag":118,"props":2369,"children":2371},{"id":2370},"step-7-report-findings-individual-failure-investigations-only",[2372],{"type":47,"value":2373},"Step 7: Report findings (individual failure investigations only)",{"type":42,"tag":50,"props":2375,"children":2376},{},[2377],{"type":47,"value":2378},"For health assessments, use the Output Formats section above instead.",{"type":42,"tag":50,"props":2380,"children":2381},{},[2382],{"type":47,"value":2383},"Provide:",{"type":42,"tag":138,"props":2385,"children":2386},{},[2387,2397,2407,2417,2427,2436,2446],{"type":42,"tag":67,"props":2388,"children":2389},{},[2390,2395],{"type":42,"tag":145,"props":2391,"children":2392},{},[2393],{"type":47,"value":2394},"Failure category",{"type":47,"value":2396}," — which pattern from the table above",{"type":42,"tag":67,"props":2398,"children":2399},{},[2400,2405],{"type":42,"tag":145,"props":2401,"children":2402},{},[2403],{"type":47,"value":2404},"Frequency",{"type":47,"value":2406}," — how often in the last N days, trending up\u002Fdown\u002Fstable",{"type":42,"tag":67,"props":2408,"children":2409},{},[2410,2415],{"type":42,"tag":145,"props":2411,"children":2412},{},[2413],{"type":47,"value":2414},"Root cause",{"type":47,"value":2416}," — with evidence from logs and\u002For source code",{"type":42,"tag":67,"props":2418,"children":2419},{},[2420,2425],{"type":42,"tag":145,"props":2421,"children":2422},{},[2423],{"type":47,"value":2424},"Affected branches",{"type":47,"value":2426}," — main only, or also release branches",{"type":42,"tag":67,"props":2428,"children":2429},{},[2430,2434],{"type":42,"tag":145,"props":2431,"children":2432},{},[2433],{"type":47,"value":785},{"type":47,"value":2435}," — who should fix it (1ES, dotnet\u002Fdotnet, SDK team, etc.)",{"type":42,"tag":67,"props":2437,"children":2438},{},[2439,2444],{"type":42,"tag":145,"props":2440,"children":2441},{},[2442],{"type":47,"value":2443},"Existing issues",{"type":47,"value":2445}," — links to any filed issues",{"type":42,"tag":67,"props":2447,"children":2448},{},[2449,2454],{"type":42,"tag":145,"props":2450,"children":2451},{},[2452],{"type":47,"value":2453},"Recommended action",{"type":47,"value":2455}," — file issue, retry, wait for fix, etc.",{"type":42,"tag":56,"props":2457,"children":2459},{"id":2458},"artifact-cascade-failures",[2460],{"type":47,"value":2461},"Artifact Cascade Failures",{"type":42,"tag":50,"props":2463,"children":2464},{},[2465],{"type":47,"value":2466},"When a build leg fails, downstream legs that depend on its artifacts will also fail with \"Artifact not found.\" This creates a cascade:",{"type":42,"tag":381,"props":2468,"children":2471},{"className":2469,"code":2470,"language":47},[653],"SB_CentOS_Build (fails: NetAnalyzers error)\n  → SB_CentOS_Validation (fails: missing artifacts)\n  → SB_Fedora_Offline (fails: missing artifacts)\n",[2472],{"type":42,"tag":88,"props":2473,"children":2474},{"__ignoreMap":386},[2475],{"type":47,"value":2470},{"type":42,"tag":50,"props":2477,"children":2478},{},[2479,2484],{"type":42,"tag":145,"props":2480,"children":2481},{},[2482],{"type":47,"value":2483},"Don't investigate the cascade — find the root failure.",{"type":47,"value":2485}," Look for the first failed task chronologically, or filter out \"Download Previous Build\" failures to find the real cause.",{"type":42,"tag":56,"props":2487,"children":2489},{"id":2488},"category-specific-investigation-techniques",[2490],{"type":47,"value":2491},"Category-Specific Investigation Techniques",{"type":42,"tag":50,"props":2493,"children":2494},{},[2495,2497,2502],{"type":47,"value":2496},"For detailed investigation techniques for each failure category in the table above, see ",{"type":42,"tag":130,"props":2498,"children":2500},{"href":2499},"references\u002Finvestigation-techniques.md",[2501],{"type":47,"value":2499},{"type":47,"value":2503},". Covers: ESRP signing\u002Fnotarization, container OOM, crossgen2 crashes, YAML pre-flight rejections, and network transient failures.",{"type":42,"tag":56,"props":2505,"children":2507},{"id":2506},"fix-verification",[2508],{"type":47,"value":2509},"Fix Verification",{"type":42,"tag":50,"props":2511,"children":2512},{},[2513,2515,2520],{"type":47,"value":2514},"When a fix is merged, verify it's tested by checking builds queued ",{"type":42,"tag":145,"props":2516,"children":2517},{},[2518],{"type":47,"value":2519},"after",{"type":47,"value":2521}," the merge — not builds that finished after it.",{"type":42,"tag":118,"props":2523,"children":2525},{"id":2524},"query-all-build-statuses",[2526],{"type":47,"value":2527},"Query all build statuses",{"type":42,"tag":50,"props":2529,"children":2530},{},[2531,2536,2538,2544],{"type":42,"tag":145,"props":2532,"children":2533},{},[2534],{"type":47,"value":2535},"Always include in-progress and not-started builds",{"type":47,"value":2537},", not just completed ones. The AzDO builds API requires explicit ",{"type":42,"tag":88,"props":2539,"children":2541},{"className":2540},[],[2542],{"type":47,"value":2543},"statusFilter",{"type":47,"value":2545}," to return active builds.",{"type":42,"tag":618,"props":2547,"children":2548},{},[2549],{"type":42,"tag":50,"props":2550,"children":2551},{},[2552,2554,2587],{"type":47,"value":2553},"⚠️ ",{"type":42,"tag":145,"props":2555,"children":2556},{},[2557,2559,2565,2566,2572,2573,2579,2581,2586],{"type":47,"value":2558},"The AzDO API rejects combining ",{"type":42,"tag":88,"props":2560,"children":2562},{"className":2561},[],[2563],{"type":47,"value":2564},"completed",{"type":47,"value":889},{"type":42,"tag":88,"props":2567,"children":2569},{"className":2568},[],[2570],{"type":47,"value":2571},"inProgress",{"type":47,"value":2291},{"type":42,"tag":88,"props":2574,"children":2576},{"className":2575},[],[2577],{"type":47,"value":2578},"notStarted",{"type":47,"value":2580}," in a single ",{"type":42,"tag":88,"props":2582,"children":2584},{"className":2583},[],[2585],{"type":47,"value":2543},{"type":47,"value":703},{"type":47,"value":2588}," You must make separate calls and merge the results:",{"type":42,"tag":381,"props":2590,"children":2592},{"className":383,"code":2591,"language":385,"meta":386,"style":386},"# Completed builds\ncurl -s -H \"Authorization: Bearer $TOKEN\" \\\n  \"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds?definitions={defId}&branchName=refs\u002Fheads\u002F{branch}&statusFilter=completed&api-version=7.1\"\n\n# Active builds (in-progress and queued)\ncurl -s -H \"Authorization: Bearer $TOKEN\" \\\n  \"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds?definitions={defId}&branchName=refs\u002Fheads\u002F{branch}&statusFilter=inProgress,notStarted&api-version=7.1\"\n",[2593],{"type":42,"tag":88,"props":2594,"children":2595},{"__ignoreMap":386},[2596,2604,2639,2655,2662,2670,2705],{"type":42,"tag":392,"props":2597,"children":2598},{"class":394,"line":395},[2599],{"type":42,"tag":392,"props":2600,"children":2601},{"style":399},[2602],{"type":47,"value":2603},"# Completed builds\n",{"type":42,"tag":392,"props":2605,"children":2606},{"class":394,"line":405},[2607,2611,2615,2619,2623,2627,2631,2635],{"type":42,"tag":392,"props":2608,"children":2609},{"style":421},[2610],{"type":47,"value":348},{"type":42,"tag":392,"props":2612,"children":2613},{"style":427},[2614],{"type":47,"value":579},{"type":42,"tag":392,"props":2616,"children":2617},{"style":427},[2618],{"type":47,"value":584},{"type":42,"tag":392,"props":2620,"children":2621},{"style":415},[2622],{"type":47,"value":535},{"type":42,"tag":392,"props":2624,"children":2625},{"style":427},[2626],{"type":47,"value":593},{"type":42,"tag":392,"props":2628,"children":2629},{"style":409},[2630],{"type":47,"value":598},{"type":42,"tag":392,"props":2632,"children":2633},{"style":415},[2634],{"type":47,"value":545},{"type":42,"tag":392,"props":2636,"children":2637},{"style":409},[2638],{"type":47,"value":1612},{"type":42,"tag":392,"props":2640,"children":2641},{"class":394,"line":473},[2642,2646,2651],{"type":42,"tag":392,"props":2643,"children":2644},{"style":415},[2645],{"type":47,"value":1620},{"type":42,"tag":392,"props":2647,"children":2648},{"style":427},[2649],{"type":47,"value":2650},"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds?definitions={defId}&branchName=refs\u002Fheads\u002F{branch}&statusFilter=completed&api-version=7.1",{"type":42,"tag":392,"props":2652,"children":2653},{"style":415},[2654],{"type":47,"value":616},{"type":42,"tag":392,"props":2656,"children":2657},{"class":394,"line":483},[2658],{"type":42,"tag":392,"props":2659,"children":2660},{"emptyLinePlaceholder":477},[2661],{"type":47,"value":480},{"type":42,"tag":392,"props":2663,"children":2664},{"class":394,"line":492},[2665],{"type":42,"tag":392,"props":2666,"children":2667},{"style":399},[2668],{"type":47,"value":2669},"# Active builds (in-progress and queued)\n",{"type":42,"tag":392,"props":2671,"children":2672},{"class":394,"line":552},[2673,2677,2681,2685,2689,2693,2697,2701],{"type":42,"tag":392,"props":2674,"children":2675},{"style":421},[2676],{"type":47,"value":348},{"type":42,"tag":392,"props":2678,"children":2679},{"style":427},[2680],{"type":47,"value":579},{"type":42,"tag":392,"props":2682,"children":2683},{"style":427},[2684],{"type":47,"value":584},{"type":42,"tag":392,"props":2686,"children":2687},{"style":415},[2688],{"type":47,"value":535},{"type":42,"tag":392,"props":2690,"children":2691},{"style":427},[2692],{"type":47,"value":593},{"type":42,"tag":392,"props":2694,"children":2695},{"style":409},[2696],{"type":47,"value":598},{"type":42,"tag":392,"props":2698,"children":2699},{"style":415},[2700],{"type":47,"value":545},{"type":42,"tag":392,"props":2702,"children":2703},{"style":409},[2704],{"type":47,"value":1612},{"type":42,"tag":392,"props":2706,"children":2707},{"class":394,"line":560},[2708,2712,2717],{"type":42,"tag":392,"props":2709,"children":2710},{"style":415},[2711],{"type":47,"value":1620},{"type":42,"tag":392,"props":2713,"children":2714},{"style":427},[2715],{"type":47,"value":2716},"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds?definitions={defId}&branchName=refs\u002Fheads\u002F{branch}&statusFilter=inProgress,notStarted&api-version=7.1",{"type":42,"tag":392,"props":2718,"children":2719},{"style":415},[2720],{"type":47,"value":616},{"type":42,"tag":50,"props":2722,"children":2723},{},[2724],{"type":47,"value":2725},"If you only query completed builds, you'll miss active builds that are currently testing the fix and incorrectly conclude \"not tested yet.\"",{"type":42,"tag":118,"props":2727,"children":2729},{"id":2728},"multi-branch-fixes",[2730],{"type":47,"value":2731},"Multi-branch fixes",{"type":42,"tag":50,"props":2733,"children":2734},{},[2735],{"type":47,"value":2736},"Fixes often need backporting to multiple release branches. For each branch:",{"type":42,"tag":138,"props":2738,"children":2739},{},[2740,2745,2755,2768],{"type":42,"tag":67,"props":2741,"children":2742},{},[2743],{"type":47,"value":2744},"Find the fix PR (or backport PR) and its merge time",{"type":42,"tag":67,"props":2746,"children":2747},{},[2748,2750],{"type":47,"value":2749},"Query builds on that branch with ",{"type":42,"tag":145,"props":2751,"children":2752},{},[2753],{"type":47,"value":2754},"all statuses",{"type":42,"tag":67,"props":2756,"children":2757},{},[2758,2760,2766],{"type":47,"value":2759},"Compare each build's ",{"type":42,"tag":88,"props":2761,"children":2763},{"className":2762},[],[2764],{"type":47,"value":2765},"queueTime",{"type":47,"value":2767}," to the merge time",{"type":42,"tag":67,"props":2769,"children":2770},{},[2771],{"type":47,"value":2772},"Partition into pre-merge (doesn't have fix) and post-merge (has fix)",{"type":42,"tag":618,"props":2774,"children":2775},{},[2776],{"type":42,"tag":50,"props":2777,"children":2778},{},[2779,2780,2785],{"type":47,"value":2553},{"type":42,"tag":145,"props":2781,"children":2782},{},[2783],{"type":47,"value":2784},"batchedCI builds pick up source at queue time.",{"type":47,"value":2786}," A build queued before a PR merges runs against old source, even if it finishes hours later.",{"type":42,"tag":56,"props":2788,"children":2790},{"id":2789},"stop-signals",[2791],{"type":47,"value":2792},"Stop Signals",{"type":42,"tag":63,"props":2794,"children":2795},{},[2796,2806,2816,2826],{"type":42,"tag":67,"props":2797,"children":2798},{},[2799,2804],{"type":42,"tag":145,"props":2800,"children":2801},{},[2802],{"type":47,"value":2803},"Stop after categorizing",{"type":47,"value":2805}," if it's a known issue with an existing bug. Link to it and move on.",{"type":42,"tag":67,"props":2807,"children":2808},{},[2809,2814],{"type":42,"tag":145,"props":2810,"children":2811},{},[2812],{"type":47,"value":2813},"Stop frequency analysis",{"type":47,"value":2815}," after 14 days of data. Longer trends are rarely actionable.",{"type":42,"tag":67,"props":2817,"children":2818},{},[2819,2824],{"type":42,"tag":145,"props":2820,"children":2821},{},[2822],{"type":47,"value":2823},"Stop investigating cascades",{"type":47,"value":2825}," — always trace back to the root failure.",{"type":42,"tag":67,"props":2827,"children":2828},{},[2829,2834],{"type":42,"tag":145,"props":2830,"children":2831},{},[2832],{"type":47,"value":2833},"Present single-occurrence failures",{"type":47,"value":2835}," — even one-offs deserve a summary. Let the user decide whether to dig deeper or move on.",{"type":42,"tag":56,"props":2837,"children":2839},{"id":2838},"anti-patterns",[2840],{"type":47,"value":2841},"Anti-Patterns",{"type":42,"tag":618,"props":2843,"children":2844},{},[2845],{"type":42,"tag":50,"props":2846,"children":2847},{},[2848,2850,2855],{"type":47,"value":2849},"🚨 ",{"type":42,"tag":145,"props":2851,"children":2852},{},[2853],{"type":47,"value":2854},"Don't confuse cascade failures with root causes.",{"type":47,"value":2856}," \"Missing artifacts\" means a prior leg failed — find that leg.",{"type":42,"tag":618,"props":2858,"children":2859},{},[2860],{"type":42,"tag":50,"props":2861,"children":2862},{},[2863,2864,2876],{"type":47,"value":2849},{"type":42,"tag":145,"props":2865,"children":2866},{},[2867,2869,2874],{"type":47,"value":2868},"Don't assume ",{"type":42,"tag":88,"props":2870,"children":2872},{"className":2871},[],[2873],{"type":47,"value":93},{"type":47,"value":2875}," is a code bug.",{"type":47,"value":2877}," It usually means the process was terminated externally (OOM, signal). Look for resource exhaustion in the log. In containers, check memory limits.",{"type":42,"tag":618,"props":2879,"children":2880},{},[2881],{"type":42,"tag":50,"props":2882,"children":2883},{},[2884,2885,2890],{"type":47,"value":2849},{"type":42,"tag":145,"props":2886,"children":2887},{},[2888],{"type":47,"value":2889},"Don't investigate all 100+ failed builds.",{"type":47,"value":2891}," Sample 3-5 spread across the time range to confirm the pattern, then count occurrences.",{"type":42,"tag":618,"props":2893,"children":2894},{},[2895],{"type":42,"tag":50,"props":2896,"children":2897},{},[2898,2899,2911],{"type":47,"value":2849},{"type":42,"tag":145,"props":2900,"children":2901},{},[2902,2904,2909],{"type":47,"value":2903},"Don't assume ESRP ",{"type":42,"tag":88,"props":2905,"children":2907},{"className":2906},[],[2908],{"type":47,"value":1174},{"type":47,"value":2910}," is transient.",{"type":47,"value":2912}," It means the binary content is deterministically invalid. Investigate the source commits, don't retry the build.",{"type":42,"tag":618,"props":2914,"children":2915},{},[2916],{"type":42,"tag":50,"props":2917,"children":2918},{},[2919,2920,2925],{"type":47,"value":2849},{"type":42,"tag":145,"props":2921,"children":2922},{},[2923],{"type":47,"value":2924},"Don't skip infrastructure failures.",{"type":47,"value":2926}," Container OOM, agent timeouts, signing failures, and network issues deserve the same depth of investigation as code failures. They often have actionable fixes (memory tuning, retry wrappers, pool changes).",{"type":42,"tag":56,"props":2928,"children":2930},{"id":2929},"discovering-pipeline-definitions",[2931],{"type":47,"value":2932},"Discovering Pipeline Definitions",{"type":42,"tag":50,"props":2934,"children":2935},{},[2936],{"type":47,"value":2937},"Never hardcode definition IDs — they vary across projects and can change. Discover them at runtime:",{"type":42,"tag":381,"props":2939,"children":2941},{"className":383,"code":2940,"language":385,"meta":386,"style":386},"# Public pipelines (AzDO MCP tools work here too)\ncurl -s \"https:\u002F\u002Fdev.azure.com\u002Fdnceng-public\u002Fpublic\u002F_apis\u002Fbuild\u002Fdefinitions?name=dotnet-unified-build&api-version=7.1\" | jq '.value[] | {id, name, project: .project.name}'\n\n# Internal pipelines (requires auth)\nTOKEN=$(az account get-access-token --resource 499b84ac-1321-427f-aa17-267ca6975798 --query accessToken -o tsv)\ncurl -s -H \"Authorization: Bearer $TOKEN\" \\\n  \"https:\u002F\u002Fdev.azure.com\u002Fdnceng\u002Finternal\u002F_apis\u002Fbuild\u002Fdefinitions?name=dotnet-unified-build&api-version=7.1\" \\\n  | jq '.value[] | {id, name, project: .project.name}'\n",[2942],{"type":42,"tag":88,"props":2943,"children":2944},{"__ignoreMap":386},[2945,2953,3000,3007,3015,3066,3101,3121],{"type":42,"tag":392,"props":2946,"children":2947},{"class":394,"line":395},[2948],{"type":42,"tag":392,"props":2949,"children":2950},{"style":399},[2951],{"type":47,"value":2952},"# Public pipelines (AzDO MCP tools work here too)\n",{"type":42,"tag":392,"props":2954,"children":2955},{"class":394,"line":405},[2956,2960,2964,2968,2973,2977,2981,2986,2990,2995],{"type":42,"tag":392,"props":2957,"children":2958},{"style":421},[2959],{"type":47,"value":348},{"type":42,"tag":392,"props":2961,"children":2962},{"style":427},[2963],{"type":47,"value":579},{"type":42,"tag":392,"props":2965,"children":2966},{"style":415},[2967],{"type":47,"value":535},{"type":42,"tag":392,"props":2969,"children":2970},{"style":427},[2971],{"type":47,"value":2972},"https:\u002F\u002Fdev.azure.com\u002Fdnceng-public\u002Fpublic\u002F_apis\u002Fbuild\u002Fdefinitions?name=dotnet-unified-build&api-version=7.1",{"type":42,"tag":392,"props":2974,"children":2975},{"style":415},[2976],{"type":47,"value":545},{"type":42,"tag":392,"props":2978,"children":2979},{"style":415},[2980],{"type":47,"value":1888},{"type":42,"tag":392,"props":2982,"children":2983},{"style":421},[2984],{"type":47,"value":2985}," jq",{"type":42,"tag":392,"props":2987,"children":2988},{"style":415},[2989],{"type":47,"value":1873},{"type":42,"tag":392,"props":2991,"children":2992},{"style":427},[2993],{"type":47,"value":2994},".value[] | {id, name, project: .project.name}",{"type":42,"tag":392,"props":2996,"children":2997},{"style":415},[2998],{"type":47,"value":2999},"'\n",{"type":42,"tag":392,"props":3001,"children":3002},{"class":394,"line":473},[3003],{"type":42,"tag":392,"props":3004,"children":3005},{"emptyLinePlaceholder":477},[3006],{"type":47,"value":480},{"type":42,"tag":392,"props":3008,"children":3009},{"class":394,"line":483},[3010],{"type":42,"tag":392,"props":3011,"children":3012},{"style":399},[3013],{"type":47,"value":3014},"# Internal pipelines (requires auth)\n",{"type":42,"tag":392,"props":3016,"children":3017},{"class":394,"line":492},[3018,3022,3026,3030,3034,3038,3042,3046,3050,3054,3058,3062],{"type":42,"tag":392,"props":3019,"children":3020},{"style":409},[3021],{"type":47,"value":412},{"type":42,"tag":392,"props":3023,"children":3024},{"style":415},[3025],{"type":47,"value":418},{"type":42,"tag":392,"props":3027,"children":3028},{"style":421},[3029],{"type":47,"value":424},{"type":42,"tag":392,"props":3031,"children":3032},{"style":427},[3033],{"type":47,"value":430},{"type":42,"tag":392,"props":3035,"children":3036},{"style":427},[3037],{"type":47,"value":435},{"type":42,"tag":392,"props":3039,"children":3040},{"style":427},[3041],{"type":47,"value":440},{"type":42,"tag":392,"props":3043,"children":3044},{"style":427},[3045],{"type":47,"value":445},{"type":42,"tag":392,"props":3047,"children":3048},{"style":427},[3049],{"type":47,"value":450},{"type":42,"tag":392,"props":3051,"children":3052},{"style":427},[3053],{"type":47,"value":455},{"type":42,"tag":392,"props":3055,"children":3056},{"style":427},[3057],{"type":47,"value":460},{"type":42,"tag":392,"props":3059,"children":3060},{"style":427},[3061],{"type":47,"value":465},{"type":42,"tag":392,"props":3063,"children":3064},{"style":415},[3065],{"type":47,"value":470},{"type":42,"tag":392,"props":3067,"children":3068},{"class":394,"line":552},[3069,3073,3077,3081,3085,3089,3093,3097],{"type":42,"tag":392,"props":3070,"children":3071},{"style":421},[3072],{"type":47,"value":348},{"type":42,"tag":392,"props":3074,"children":3075},{"style":427},[3076],{"type":47,"value":579},{"type":42,"tag":392,"props":3078,"children":3079},{"style":427},[3080],{"type":47,"value":584},{"type":42,"tag":392,"props":3082,"children":3083},{"style":415},[3084],{"type":47,"value":535},{"type":42,"tag":392,"props":3086,"children":3087},{"style":427},[3088],{"type":47,"value":593},{"type":42,"tag":392,"props":3090,"children":3091},{"style":409},[3092],{"type":47,"value":598},{"type":42,"tag":392,"props":3094,"children":3095},{"style":415},[3096],{"type":47,"value":545},{"type":42,"tag":392,"props":3098,"children":3099},{"style":409},[3100],{"type":47,"value":1612},{"type":42,"tag":392,"props":3102,"children":3103},{"class":394,"line":560},[3104,3108,3113,3117],{"type":42,"tag":392,"props":3105,"children":3106},{"style":415},[3107],{"type":47,"value":1620},{"type":42,"tag":392,"props":3109,"children":3110},{"style":427},[3111],{"type":47,"value":3112},"https:\u002F\u002Fdev.azure.com\u002Fdnceng\u002Finternal\u002F_apis\u002Fbuild\u002Fdefinitions?name=dotnet-unified-build&api-version=7.1",{"type":42,"tag":392,"props":3114,"children":3115},{"style":415},[3116],{"type":47,"value":545},{"type":42,"tag":392,"props":3118,"children":3119},{"style":409},[3120],{"type":47,"value":1612},{"type":42,"tag":392,"props":3122,"children":3123},{"class":394,"line":569},[3124,3129,3133,3137,3141],{"type":42,"tag":392,"props":3125,"children":3126},{"style":415},[3127],{"type":47,"value":3128},"  |",{"type":42,"tag":392,"props":3130,"children":3131},{"style":421},[3132],{"type":47,"value":2985},{"type":42,"tag":392,"props":3134,"children":3135},{"style":415},[3136],{"type":47,"value":1873},{"type":42,"tag":392,"props":3138,"children":3139},{"style":427},[3140],{"type":47,"value":2994},{"type":42,"tag":392,"props":3142,"children":3143},{"style":415},[3144],{"type":47,"value":2999},{"type":42,"tag":50,"props":3146,"children":3147},{},[3148],{"type":47,"value":3149},"If querying internal fails (401\u002Fredirect), fall back to public-only analysis and note the limitation in the report.",{"type":42,"tag":118,"props":3151,"children":3153},{"id":3152},"health-assessment-report-format",[3154],{"type":47,"value":3155},"Health Assessment Report Format",{"type":42,"tag":50,"props":3157,"children":3158},{},[3159,3164,3166,3170],{"type":42,"tag":145,"props":3160,"children":3161},{},[3162],{"type":47,"value":3163},"Required format",{"type":47,"value":3165}," for pipeline health assessments. See Output Formats section above for routing, and ",{"type":42,"tag":130,"props":3167,"children":3168},{"href":132},[3169],{"type":47,"value":132},{"type":47,"value":3171}," for the complete specification including build classification rules, branch filtering, and codeflow analysis methodology.",{"type":42,"tag":3173,"props":3174,"children":3175},"style",{},[3176],{"type":47,"value":3177},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3179,"total":3341},[3180,3194,3209,3224,3242,3256,3276,3286,3297,3307,3320,3331],{"slug":3181,"name":3181,"fn":3182,"description":3183,"org":3184,"tags":3185,"stars":3191,"repoUrl":3192,"updatedAt":3193},"multithreaded-task-migration","migrate MSBuild tasks to multithreaded mode","Guide for migrating MSBuild tasks to multithreaded mode support, including compatibility red-team review. Use this when converting tasks to thread-safe versions, implementing IMultiThreadableTask, adding TaskEnvironment support, or auditing migrations for behavioral compatibility.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3186,3187,3188],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"name":3189,"slug":3190,"type":15},"Performance","performance",5535,"https:\u002F\u002Fgithub.com\u002Fdotnet\u002Fmsbuild","2026-07-22T05:37:33.965588",{"slug":3195,"name":3195,"fn":3196,"description":3197,"org":3198,"tags":3199,"stars":3206,"repoUrl":3207,"updatedAt":3208},"analyzing-dotnet-performance","analyze .NET code for performance anti-patterns","Scans .NET code for ~50 performance anti-patterns across async, memory, strings, collections, LINQ, regex, serialization, and I\u002FO with tiered severity classification. Use when analyzing .NET code for optimization opportunities, reviewing hot paths, or auditing allocation-heavy patterns.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3200,3201,3204,3205],{"name":13,"slug":14,"type":15},{"name":3202,"slug":3203,"type":15},"Code Analysis","code-analysis",{"name":23,"slug":24,"type":15},{"name":3189,"slug":3190,"type":15},4576,"https:\u002F\u002Fgithub.com\u002Fdotnet\u002Fskills","2026-07-12T08:23:25.400375",{"slug":3210,"name":3210,"fn":3211,"description":3212,"org":3213,"tags":3214,"stars":3206,"repoUrl":3207,"updatedAt":3223},"android-tombstone-symbolication","symbolicate .NET runtime frames in Android tombstones","Symbolicate the .NET runtime frames in an Android tombstone file. Extracts BuildIds and PC offsets from the native backtrace, downloads debug symbols from the Microsoft symbol server, and runs llvm-symbolizer to produce function names with source file and line numbers. USE FOR triaging a .NET MAUI or Mono Android app crash from a tombstone, resolving native backtrace frames in libmonosgen-2.0.so or libcoreclr.so to .NET runtime source code, or investigating SIGABRT, SIGSEGV, or other native signals originating from the .NET runtime on Android. DO NOT USE FOR pure Java\u002FKotlin crashes, managed .NET exceptions that are already captured in logcat, or iOS crash logs. INVOKES Symbolicate-Tombstone.ps1 script, llvm-symbolizer, Microsoft symbol server.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3215,3216,3219,3220],{"name":13,"slug":14,"type":15},{"name":3217,"slug":3218,"type":15},"Android","android",{"name":23,"slug":24,"type":15},{"name":3221,"slug":3222,"type":15},"Microsoft","microsoft","2026-07-12T08:23:21.595572",{"slug":3225,"name":3225,"fn":3226,"description":3227,"org":3228,"tags":3229,"stars":3206,"repoUrl":3207,"updatedAt":3241},"apple-crash-symbolication","symbolicate .NET runtime frames in crash logs","Symbolicate .NET runtime frames in Apple platform .ips crash logs (iOS, tvOS, Mac Catalyst, macOS). Extracts UUIDs and addresses from the native backtrace, locates dSYM debug symbols, and runs atos to produce function names with source file and line numbers. Automatically downloads .dwarf symbols from the Microsoft symbol server using Mach-O UUIDs. USE FOR triaging a .NET MAUI or Mono app crash from an .ips file on any Apple platform, resolving native backtrace frames in libcoreclr or libmonosgen-2.0 to .NET runtime source code, retrieving .ips crash logs from a connected iOS device or iPhone, or investigating EXC_CRASH, EXC_BAD_ACCESS, SIGABRT, or SIGSEGV originating from the .NET runtime. DO NOT USE FOR pure Swift\u002FObjective-C crashes with no .NET components, or Android tombstone files. INVOKES Symbolicate-Crash.ps1 script, atos, dwarfdump, idevicecrashreport.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3230,3231,3232,3235,3238],{"name":13,"slug":14,"type":15},{"name":23,"slug":24,"type":15},{"name":3233,"slug":3234,"type":15},"iOS","ios",{"name":3236,"slug":3237,"type":15},"macOS","macos",{"name":3239,"slug":3240,"type":15},"Observability","observability","2026-07-12T08:23:20.369986",{"slug":3243,"name":3243,"fn":3244,"description":3245,"org":3246,"tags":3247,"stars":3206,"repoUrl":3207,"updatedAt":3255},"assertion-quality","evaluate assertion quality in test suites","Analyzes the variety and depth of assertions across test suites in any language. Use when the user asks to evaluate assertion quality, find shallow tests, identify assertion-free tests (no assertions or only trivial ones like Assert.IsNotNull \u002F toBeTruthy()), flag self-referential or tautological assertions, measure assertion diversity, or audit whether tests verify different facets of behavior. Polyglot: .NET, Python, TS\u002FJS, Java, Go, Ruby, Rust, Swift, Kotlin, PowerShell, C++. DO NOT USE FOR: writing new tests (use code-testing-agent \u002F writing-mstest-tests), mutation reasoning about whether tests would catch a bug (use test-gap-analysis), or a general severity-ranked anti-pattern audit (use test-anti-patterns), fixing or rewriting assertions, or writing, fixing, or modernizing MSTest tests, assertions, or attributes (use writing-mstest-tests).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3248,3249,3252],{"name":3202,"slug":3203,"type":15},{"name":3250,"slug":3251,"type":15},"QA","qa",{"name":3253,"slug":3254,"type":15},"Testing","testing","2026-07-12T08:23:51.277743",{"slug":3257,"name":3257,"fn":3258,"description":3259,"org":3260,"tags":3261,"stars":3206,"repoUrl":3207,"updatedAt":3275},"author-component","create and review Blazor components","Create or review Blazor components (.razor files) with correct architecture. USE FOR: writing new Blazor components that do NOT involve JavaScript interop, implementing parameters and EventCallback, RenderFragment slots, component lifecycle (OnInitializedAsync, OnParametersSet), async patterns, IAsyncDisposable, CancellationToken, CSS isolation, code-behind. DO NOT USE FOR: creating new projects (use create-blazor-project), JavaScript interop or calling browser APIs from Blazor (use use-js-interop), forms and validation (use collect-user-input), prerendering issues (use support-prerendering), HTTP data fetching patterns (use fetch-and-send-data), coordinating state between unrelated components (use coordinate-components).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3262,3263,3266,3269,3272],{"name":13,"slug":14,"type":15},{"name":3264,"slug":3265,"type":15},"Blazor","blazor",{"name":3267,"slug":3268,"type":15},"C#","csharp",{"name":3270,"slug":3271,"type":15},"UI Components","ui-components",{"name":3273,"slug":3274,"type":15},"Web Development","web-development","2026-07-15T06:03:29.216359",{"slug":3277,"name":3277,"fn":3278,"description":3279,"org":3280,"tags":3281,"stars":3206,"repoUrl":3207,"updatedAt":3285},"binlog-failure-analysis","analyze MSBuild binary logs","Analyze MSBuild binary logs to diagnose build failures. USE FOR: build errors that are unclear from console output, diagnosing cascading failures across multi-project builds, tracing MSBuild target execution order, and generally any MSBuild build issues. Requires an existing .binlog file. DO NOT USE FOR: generating binlogs (use binlog-generation), non-MSBuild build systems.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3282,3283,3284],{"name":3202,"slug":3203,"type":15},{"name":23,"slug":24,"type":15},{"name":3221,"slug":3222,"type":15},"2026-07-12T08:21:34.637923",{"slug":3287,"name":3287,"fn":3288,"description":3289,"org":3290,"tags":3291,"stars":3206,"repoUrl":3207,"updatedAt":3296},"binlog-generation","generate MSBuild binary logs for diagnostics","Generate MSBuild binary logs (binlogs) for build diagnostics and analysis. USE FOR: adding \u002Fbl:{} to any dotnet build, test, pack, publish, or restore command to capture a full build execution trace, prerequisite for binlog-failure-analysis and build-perf-diagnostics skills, enabling post-build investigation of errors or performance. Requires MSBuild 17.8+ \u002F .NET 8 SDK+ for {} placeholder; PowerShell needs -bl:{{}}. DO NOT USE FOR: non-MSBuild build systems (npm, Maven, CMake), analyzing an existing binlog (use binlog-failure-analysis instead).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3292,3294,3295],{"name":984,"slug":3293,"type":15},"build",{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},"2026-07-19T05:38:19.340791",{"slug":3298,"name":3298,"fn":3299,"description":3300,"org":3301,"tags":3302,"stars":3206,"repoUrl":3207,"updatedAt":3306},"build-parallelism","optimize MSBuild build parallelism","Diagnose and fix under-parallelized MSBuild builds. USE WHEN a multi-project solution build is slower than expected, doesn't speed up when you add cores, pegs a single core while others idle, or you want to know why `-m` isn't helping. Note: `\u002Fmaxcpucount` default is 1 (sequential) — always pass `-m` for parallel builds. Covers finding the critical path (longest serial ProjectReference chain), graph build (`\u002Fgraph`), BuildInParallel, and solution filters (`.slnf`). DO NOT USE FOR: single-project builds, incremental issues (use incremental-build), compilation slowness inside one project (use build-perf-diagnostics), non-MSBuild build systems.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3303,3304,3305],{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"name":3189,"slug":3190,"type":15},"2026-07-19T05:38:18.364937",{"slug":3308,"name":3308,"fn":3309,"description":3310,"org":3311,"tags":3312,"stars":3206,"repoUrl":3207,"updatedAt":3319},"build-perf-baseline","establish and optimize build performance baselines","Establish build performance baselines and apply systematic optimization techniques. USE FOR: diagnosing slow builds, establishing before\u002Fafter measurements (cold, warm, no-op scenarios), applying optimization strategies like MSBuild Server, static graph builds, artifacts output, and dependency graph trimming. Start here before diving into build-perf-diagnostics, incremental-build, or build-parallelism. DO NOT USE FOR: non-MSBuild build systems, detailed bottleneck analysis (use build-perf-diagnostics after baselining).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3313,3314,3317,3318],{"name":17,"slug":18,"type":15},{"name":3315,"slug":3316,"type":15},"Monitoring","monitoring",{"name":3189,"slug":3190,"type":15},{"name":3253,"slug":3254,"type":15},"2026-07-12T08:21:35.865649",{"slug":3321,"name":3321,"fn":3322,"description":3323,"org":3324,"tags":3325,"stars":3206,"repoUrl":3207,"updatedAt":3330},"build-perf-diagnostics","diagnose MSBuild build performance bottlenecks","Diagnose MSBuild build performance bottlenecks using binary log analysis. USE FOR: identifying why builds are slow by analyzing binlog performance summaries, detecting ResolveAssemblyReference (RAR) taking >5s, Roslyn analyzers consuming >30% of Csc time, single targets dominating >50% of build time, node utilization below 80%, excessive Copy tasks, NuGet restore running every build. Covers timeline analysis, Target\u002FTask Performance Summary interpretation, and 7 common bottleneck categories. Use after build-perf-baseline has established measurements. DO NOT USE FOR: establishing initial baselines (use build-perf-baseline first), fixing incremental build issues (use incremental-build), parallelism tuning (use build-parallelism), non-MSBuild build systems.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3326,3327,3328,3329],{"name":13,"slug":14,"type":15},{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":3189,"slug":3190,"type":15},"2026-07-12T08:21:40.961722",{"slug":3332,"name":3332,"fn":3333,"description":3334,"org":3335,"tags":3336,"stars":3206,"repoUrl":3207,"updatedAt":3340},"check-bin-obj-clash","detect MSBuild output path conflicts","Detects MSBuild projects with conflicting OutputPath or IntermediateOutputPath. USE FOR: builds failing with 'Cannot create a file when that file already exists', 'The process cannot access the file because it is being used by another process', intermittent build failures that succeed on retry, or missing\u002Foverwritten outputs in multi-project or multi-targeting builds where bin\u002Fobj (or project.assets.json) collide. Common causes: shared OutputPath, missing AppendTargetFrameworkToOutputPath, extra global properties (e.g. PublishReadyToRun), or SetTargetFramework on a ProjectReference to a single-targeting project. DO NOT USE FOR: file access errors unrelated to MSBuild (OS-level locking), single-project single-TFM builds, non-MSBuild build systems.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3337,3338,3339],{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":3250,"slug":3251,"type":15},"2026-07-19T05:38:14.336279",144,{"items":3343,"total":25},[3344,3359,3372,3385,3396,3411,3421],{"slug":3345,"name":3345,"fn":3346,"description":3347,"org":3348,"tags":3349,"stars":25,"repoUrl":26,"updatedAt":3358},"binskim-analysis","investigate BinSkim SDL security findings","Investigate BinSkim SDL findings from official pipelines — understand Guardian filtering, compare raw vs merged SARIF, decode portal results, and determine fix ownership. Use when asked about SDL scan results, portal findings, Guardian filtering, rule meanings, or discrepancies between local and official results. Also use when asked \"why does the portal show X\", \"what's filtered\", \"explain Guardian\", \"investigate SDL findings\", \"portal BA2008\", \"binskim failures in pipeline\", or \"what rules are required\". DO NOT USE FOR: running BinSkim locally (use binskim-scan), source code analysis (use CodeQL), or credential scanning (use CredScan).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3350,3351,3354,3355],{"name":13,"slug":14,"type":15},{"name":3352,"slug":3353,"type":15},"Audit","audit",{"name":3202,"slug":3203,"type":15},{"name":3356,"slug":3357,"type":15},"Security","security","2026-07-12T08:23:11.212231",{"slug":3360,"name":3360,"fn":3361,"description":3362,"org":3363,"tags":3364,"stars":25,"repoUrl":26,"updatedAt":3371},"binskim-scan","run BinSkim binary security analysis","Run BinSkim binary security analysis locally against a dotnet repository. Use when asked to scan binaries, check BinSkim compliance, verify a fix for a rule violation, or run a local SDL scan. Also use when asked \"run binskim\", \"binary security scan\", \"scan binaries\", \"check binskim\", \"verify my fix\", \"repro BA2008 locally\", or \"verify BA2008 fix\". DO NOT USE FOR: investigating official pipeline results or portal findings (use binskim-analysis), source code analysis (use CodeQL), credential scanning (use CredScan), or general build\u002Ftest failures (use ci-analysis).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3365,3366,3369,3370],{"name":13,"slug":14,"type":15},{"name":3367,"slug":3368,"type":15},"CLI","cli",{"name":3202,"slug":3203,"type":15},{"name":3356,"slug":3357,"type":15},"2026-07-12T08:23:09.911458",{"slug":3373,"name":3373,"fn":3374,"description":3375,"org":3376,"tags":3377,"stars":25,"repoUrl":26,"updatedAt":3384},"ci-analysis","analyze .NET CI build and test status","Analyze CI build and test status from Azure DevOps and Helix for dotnet repository PRs. Use when checking CI status, investigating failures, determining if a PR is ready to merge, or given URLs containing dev.azure.com or helix.dot.net. Also use when asked \"why is CI red\", \"test failures\", \"retry CI\", \"rerun tests\", \"is CI green\", \"build failed\", \"checks failing\", or \"flaky tests\". DO NOT USE FOR: investigating stale codeflow PRs or dependency update health, tracing whether a commit has flowed from one repo to another, reviewing code changes for correctness or style.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3378,3379,3382,3383],{"name":13,"slug":14,"type":15},{"name":3380,"slug":3381,"type":15},"Azure","azure",{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},"2026-07-12T08:21:49.360882",{"slug":3386,"name":3386,"fn":3387,"description":3388,"org":3389,"tags":3390,"stars":25,"repoUrl":26,"updatedAt":3395},"ci-crash-dump","debug CI crash dumps","Download and debug crash dumps from CI test failures in dotnet repositories. Use when a CI test crashed (not just failed), when the user wants to debug a crash dump from a PR or build, or when asked \"debug dump\", \"download dump\", \"crash dump from CI\", \"test crashed\", \"analyze crash in PR\", or \"why did the test crash\". DO NOT USE FOR: test failures that are not crashes (use ci-analysis), build failures, performance analysis, or analyzing dumps you already have locally.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3391,3392,3393,3394],{"name":13,"slug":14,"type":15},{"name":20,"slug":21,"type":15},{"name":23,"slug":24,"type":15},{"name":3253,"slug":3254,"type":15},"2026-07-12T08:21:56.945844",{"slug":3397,"name":3397,"fn":3398,"description":3399,"org":3400,"tags":3401,"stars":25,"repoUrl":26,"updatedAt":3410},"flow-analysis","analyze VMR codeflow health","Analyze VMR codeflow health using maestro MCP tools and GitHub MCP tools. USE FOR: investigating stale codeflow PRs, checking if fixes have flowed through the VMR pipeline, debugging dependency update issues, checking overall flow status for a repo, diagnosing why backflow PRs are missing or blocked, subscription health, build freshness, URLs containing dotnet-maestro or \"Source code updates from dotnet\u002Fdotnet\". DO NOT USE FOR: CI build failures (use ci-analysis skill), code review (use code-review skill), general PR investigation without codeflow context, tracing whether a specific commit\u002FPR has reached another repo (use flow-tracing skill). INVOKES: maestro and GitHub MCP tools, flow-health.cs script.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3402,3403,3404,3407],{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},{"name":3405,"slug":3406,"type":15},"GitHub","github",{"name":3408,"slug":3409,"type":15},"MCP","mcp","2026-07-12T08:21:53.151866",{"slug":3412,"name":3412,"fn":3413,"description":3414,"org":3415,"tags":3416,"stars":25,"repoUrl":26,"updatedAt":3420},"flow-tracing","trace .NET dependency flow in VMR","Trace dependency flow across .NET repos through the VMR pipeline. USE FOR: checking if a PR\u002Fcommit from repo A has reached repo B, finding what runtime SHA is in an SDK build, tracing dependency versions through the VMR, checking if a commit is included in an SDK build, decoding SDK version strings, \"has my fix reached runtime\", \"did roslyn#80873 flow to runtime\", \"what SHA is in SDK version X\", cross-repo dependency tracing, mapping SDK versions to VMR commits. DO NOT USE FOR: codeflow PR health or staleness (use flow-analysis skill), CI build failures (use ci-analysis skill). INVOKES: maestro and GitHub MCP tools, Get-SdkVersionTrace.ps1 script.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3417],{"name":3418,"slug":3419,"type":15},"Data Pipeline","data-pipeline","2026-07-12T08:21:55.689056",{"slug":3422,"name":3422,"fn":3423,"description":3424,"org":3425,"tags":3426,"stars":25,"repoUrl":26,"updatedAt":3430},"helix-investigation","investigate Helix test failures","Deep-dive investigation of Helix test failures starting from AzDO build legs. USE FOR: investigating recurring Helix test failures, downloading and analyzing Helix console logs, comparing passing vs failing runs, identifying machine-specific issues, XHarness timeout analysis, Android emulator DEVICE_NOT_FOUND errors, bulk failure aggregation across legs, \"why does this test fail on some machines\", \"top 5 failing tests in the last 2 days\", \"download helix logs for build X\", \"compare passing and failing helix runs\", \"what are the most common failures\". DO NOT USE FOR: high-level CI status checks (use ci-analysis), codeflow PRs (use flow-analysis). INVOKES: Helix and AzDO MCP tools, curl, gh CLI.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3427,3428,3429],{"name":23,"slug":24,"type":15},{"name":17,"slug":18,"type":15},{"name":3253,"slug":3254,"type":15},"2026-07-12T08:21:46.445586"]