[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-dotnet-helix-investigation":3,"mdc-stwr0a-key":33,"related-repo-dotnet-helix-investigation":2827,"related-org-dotnet-helix-investigation":2918},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":11,"stars":22,"repoUrl":23,"updatedAt":24,"license":25,"forks":26,"topics":27,"repo":28,"sourceUrl":31,"mdContent":32},"helix-investigation","investigate Helix test failures","Deep-dive investigation of Helix test failures starting from AzDO build legs. USE FOR: investigating recurring Helix test failures, downloading and analyzing Helix console logs, comparing passing vs failing runs, identifying machine-specific issues, XHarness timeout analysis, Android emulator DEVICE_NOT_FOUND errors, bulk failure aggregation across legs, \"why does this test fail on some machines\", \"top 5 failing tests in the last 2 days\", \"download helix logs for build X\", \"compare passing and failing helix runs\", \"what are the most common failures\". DO NOT USE FOR: high-level CI status checks (use ci-analysis), codeflow PRs (use flow-analysis). INVOKES: Helix and AzDO MCP tools, curl, gh CLI.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},"dotnet",".NET (Microsoft)","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Fdotnet.png",[12,16,19],{"name":13,"slug":14,"type":15},"Engineering","engineering","tag",{"name":17,"slug":18,"type":15},"Testing","testing",{"name":20,"slug":21,"type":15},"Debugging","debugging",12,"https:\u002F\u002Fgithub.com\u002Fdotnet\u002Farcade-skills","2026-07-12T08:21:46.445586",null,18,[],{"repoUrl":23,"stars":22,"forks":26,"topics":29,"description":30},[],"Reuseable AI skills, plugins, agents for use in the dotnet product repos, such as dotnet\u002Fruntime","https:\u002F\u002Fgithub.com\u002Fdotnet\u002Farcade-skills\u002Ftree\u002FHEAD\u002Fplugins\u002Fdotnet-dnceng\u002Fskills\u002Fhelix-investigation","---\nname: helix-investigation\ndescription: >\n  Deep-dive investigation of Helix test failures starting from AzDO build legs.\n  USE FOR: investigating recurring Helix test failures, downloading and analyzing\n  Helix console logs, comparing passing vs failing runs, identifying machine-specific\n  issues, XHarness timeout analysis, Android emulator DEVICE_NOT_FOUND errors,\n  bulk failure aggregation across legs, \"why does this test fail on some machines\",\n  \"top 5 failing tests in the last 2 days\", \"download helix logs for build X\",\n  \"compare passing and failing helix runs\", \"what are the most common failures\".\n  DO NOT USE FOR: high-level CI status checks (use ci-analysis), codeflow PRs\n  (use flow-analysis).\n  INVOKES: Helix and AzDO MCP tools, curl, gh CLI.\n---\n\n# Helix Investigation\n\nDeep-dive into Helix test failures starting from an AzDO build leg. Goes beyond CI status checks to download raw console logs, compare passing vs failing runs, and identify root causes at the machine\u002Fdevice level.\n\n## When to Use This Skill\n\n- User has an AzDO build URL with a failing test leg and wants to understand why\n- User wants to see raw Helix console logs for a specific work item\n- User notices a test fails intermittently and wants pass\u002Ffail comparison\n- User asks \"why does this test only fail on some machines?\"\n- User wants to correlate AzDO build → Helix job → work item → console output\n- User mentions XHarness, device tests, or Helix timeouts\n- User asks for a \"top N\" list of failures across a pipeline or time range\n- User wants to know if failures are machine-specific or systemic\n- User sees Android DEVICE_NOT_FOUND or emulator boot failures\n\n## Prerequisites\n\n- Helix MCP tools (hlx-* prefix) for querying jobs, work items, and logs\n- AzDO MCP tools (ado-* prefix) for querying builds and timelines\n- `curl` for downloading raw log files when MCP search is insufficient\n- Access to `https:\u002F\u002Fhelix.dot.net\u002F` API endpoints\n\n## Workflow\n\n### Step 1: Start from the AzDO build\n\nGiven an AzDO build URL or pipeline definition + build ID:\n\n1. Query the build timeline to find the failing job\u002Fleg\n2. Identify the Helix job ID from the timeline records — look for `HelixJobId` in record properties or Helix URLs in the log output\n3. Note the job name, queue, and failure category\n\n```\nAzDO Build → Timeline API → Failing job record → Helix job ID\n```\n\n> ⚠️ AzDO MCP tools may become unavailable mid-session. Fall back to the REST API: `https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds\u002F{buildId}\u002Ftimeline?api-version=7.1`\n\n### Step 2: Enumerate Helix work items\n\nQuery the Helix job for its work items. Each work item is a test group that ran on a specific machine.\n\nKey fields to capture per work item:\n- **Work item name** (e.g., `iOS.CoreCLR.R2R.Test`, `System.Runtime.Tests`)\n- **Exit code** (0 = pass, 143 = SIGTERM\u002Ftimeout, 71 = infra error)\n- **Machine name** (e.g., `DNCENGMAC036`, `DNCENGTVOS-106`)\n- **Duration** — long durations with exit 143 indicate timeout kills\n- **Failure category** — `Crash`, `InfrastructureError`, etc.\n\n### Step 3: Download console logs\n\nThe Helix console log URL pattern:\n\n```\nhttps:\u002F\u002Fhelix.dot.net\u002Fapi\u002F2019-06-17\u002Fjobs\u002F{jobId}\u002Fworkitems\u002F{workItemName}\u002Fconsole\n```\n\nDownload with curl for local analysis — this is often more reliable than Helix MCP search tools for finding specific patterns:\n\n```bash\ncurl -s \"https:\u002F\u002Fhelix.dot.net\u002Fapi\u002F2019-06-17\u002Fjobs\u002F{jobId}\u002Fworkitems\u002F{workItemName}\u002Fconsole\" > \u002Ftmp\u002F{jobIdPrefix}-console.log\n```\n\nAlso check for additional log files (test output, device logs):\n\n```\nhttps:\u002F\u002Fhelix.dot.net\u002Fapi\u002F2019-06-17\u002Fjobs\u002F{jobId}\u002Fworkitems\u002F{workItemName}\u002Ffiles\n```\n\n**testResults.xml**: Many work items produce xUnit\u002FNUnit result XML. Download it for structured failure data — test names, failure messages, stack traces, and pass\u002Ffail counts — especially when console logs lack detail:\n\n```bash\n# List files to find the results XML\ncurl -s \"https:\u002F\u002Fhelix.dot.net\u002Fapi\u002F2019-06-17\u002Fjobs\u002F{jobId}\u002Fworkitems\u002F{workItemName}\u002Ffiles\" | python3 -c \"import sys,json; [print(f['Name']) for f in json.load(sys.stdin)]\"\n# Download (path varies: testResults.xml, xharness-output\u002FtestResults.xml, etc.)\ncurl -sL \"https:\u002F\u002Fhelix.dot.net\u002Fapi\u002F2019-06-17\u002Fjobs\u002F{jobId}\u002Fworkitems\u002F{workItemName}\u002Ffiles\u002F{fileName}\" > \u002Ftmp\u002Ftest-results.xml\n# Quick failure summary\ngrep -c 'result=\"Fail\"' \u002Ftmp\u002Ftest-results.xml\ngrep 'result=\"Fail\"' -A5 \u002Ftmp\u002Ftest-results.xml | grep 'name='\n```\n\n> 💡 Helix MCP search tools (`hlx_search_log`, `hlx_search_file`) may miss patterns — they can be case-sensitive or have limited regex support. When searching for specific strings, download the file and use local `grep` for reliability.\n\n### Step 4: Analyze the console log\n\nLook for these key patterns in order:\n\n1. **Setup phase** — machine name, queue, environment (OS version, Xcode version, device model)\n2. **Test execution** — command line used (e.g., XHarness `apple run` with flags), app launch, test output\n3. **Completion signals** — exit codes, timeout messages, crash reports\n4. **Helix wrapper** — `WORKLOAD TIMED OUT`, `exit_code=`, `Command exited with`\n\nCommon failure signatures:\n\n| Pattern | Meaning |\n|---------|---------|\n| `exit_code=143` + `WORKLOAD TIMED OUT` | Helix killed the process after timeout (SIGTERM) |\n| `exit_code=71` | Infrastructure error |\n| `exit_code=81` \u002F `DEVICE_NOT_FOUND` | Android emulator not running or not responding |\n| `exit_code=80` \u002F `APP_CRASH` | XHarness device error — **may be false** (see below) |\n| `Run timed out after N seconds` | XHarness hit its own timeout before Helix |\n| `TIMED_OUT` (XHarness) | XHarness-level timeout (distinct from Helix timeout) |\n| `mlaunch exited with {code}` | App completed — check if XHarness noticed |\n| `Detected test end tag` then `APP_CRASH` | **False failure** — tests passed but device communication failed afterward |\n| `Mercury error 1000` \u002F `RSD error 0xE8000003` | devicectl file copy failure (tvOS\u002FiOS) |\n| Device log file empty\u002Fremoved | Device communication issue (tvOS\u002FiOS) |\n| `emulator-5554` found but not `emulator-5556` | Wrong emulator architecture booted |\n| `Attempt.3` in work item name | All retries exhausted |\n| `Expected 0 exit code but got 1` (dotnet build) | SDK\u002FMSBuild build failure — 100% fail rate, not a test bug |\n| `System.TimeoutException: Timeout 30000ms exceeded` | Playwright browser automation timeout (Blazor) |\n| `exit_code=133` (128+5=SIGTRAP) + `ASSERT FAILED` | Runtime assertion failure \u002F core dump — every test suite crashes |\n\n> ⚠️ **False failure detection is critical.** XHarness can report APP_CRASH (exit 80) or TIMED_OUT (exit 143) even when **all tests pass**. Always check for `\"Detected test end tag\"` or `\"Test run completed\"` in the log before trusting the exit code. If tests completed successfully but XHarness still reports failure, the issue is post-completion device communication, not a test bug.\n\n### Step 5: Compare passing vs failing runs\n\nThis is the highest-value step for intermittent failures. Find a passing run of the same work item and download its console log too.\n\nCompare side-by-side:\n1. **Machine\u002Fdevice** — are failures concentrated on specific machines?\n2. **Environment** — OS version, SDK version, device model differences\n3. **Log divergence point** — where does the passing run succeed and the failing run hang or crash?\n4. **File artifacts** — does the passing run produce files (device logs, test results) that the failing run doesn't?\n\nUse a SQL table to track findings across multiple runs:\n\n```sql\nCREATE TABLE helix_runs (\n  job_id TEXT, work_item TEXT, machine TEXT, exit_code INT,\n  duration_sec INT, queue TEXT, os_version TEXT, passed BOOLEAN,\n  notes TEXT\n);\n```\n\n### Step 6: Identify root cause pattern\n\nCommon root cause categories:\n\n| Category | Signal | Action |\n|----------|--------|--------|\n| **False failure** | Tests pass (`\"Test run completed\"`) but exit 80\u002F143 reported | Device communication failed post-completion — file\u002Fupdate on dotnet\u002Fxharness |\n| **Machine-specific** | Same test, same code, different outcomes by machine | Report machine name, suggest pool removal |\n| **Device communication** | Empty device logs, log stream hangs, `Mercury error 1000`, `RSD error 0xE8000003` | USB\u002Fconnectivity issue on host |\n| **Emulator failure** | DEVICE_NOT_FOUND (exit 81), wrong emulator arch | Systemic if spread across machines; file on dotnet\u002Fxharness for recovery logic |\n| **Test infrastructure** | XHarness\u002FHelix wrapper bug | File issue on dotnet\u002Fxharness or dotnet\u002Farcade |\n| **Deterministic test bug** | 100% failure rate, same assertion, every run | Test expectation is wrong, not infra — file on dotnet\u002Fruntime, check for `blocking-clean-ci` label |\n| **Genuine test failure** | Test code exercises broken runtime behavior | File issue on dotnet\u002Fruntime |\n| **Timeout misconfiguration** | Test passes but harness timeout too short | Adjust timeout parameter |\n| **Retry exhaustion** | `Attempt.3` still failing, retries hit same broken machine | Helix retries on same machine before reboot takes effect |\n| **XML truncation** | All tests pass but app killed mid-result-write → truncated XML → crash reported | Large test suites on constrained devices (tvOS memory limits) |\n| **SDK\u002Fbuild break** | `dotnet build` exit 1, 100% fail rate, error in SDK targets | Not a test bug — file on dotnet\u002Faspnetcore or dotnet\u002Fsdk |\n| **Browser automation** | Playwright timeout on UI element, app builds fine | Resource contention or Blazor rendering regression |\n| **Poison PR** | Exit 133 (SIGTRAP), ALL suites fail on same build, 1x per platform per suite | Runtime assertion failure — single bad commit crashes before tests run. Filter out this build; failures are misleading |\n\n### Step 7: Report findings\n\nProvide:\n1. **Summary table** — all analyzed runs with machine, exit code, pass\u002Ffail, key observation\n2. **Root cause** — which category, with evidence from the logs\n3. **Direct links** — Helix console log URLs, AzDO build URLs\n4. **Recommended action** — machine removal, issue to file, timeout adjustment, etc.\n\n## Fix Verification\n\nWhen a PR is merged to fix a test failure, verify it's working by checking builds **queued after** the merge — not builds that **finished** after it.\n\n### Why queue time matters\n\nA build queued 10 minutes before a PR merges runs against the **old** source, even if it finishes hours later. Only builds queued after the merge include the fix.\n\n### Workflow\n\n1. Find the fix PR and its merge timestamp: `gh pr view {number} --repo {owner\u002Frepo} --json mergedAt`\n2. Query recent builds for the pipeline — **include all statuses** (completed, inProgress, notStarted). If you only check completed builds, you'll miss active builds currently testing the fix.\n3. For each build with the relevant failure, check its **queueTime** (not finishTime)\n4. Partition into pre-merge and post-merge builds\n5. If post-merge builds still show the failure → fix didn't work or there's a second issue\n6. If only pre-merge builds fail → fix is confirmed clean\n\n> ⚠️ **Common mistake**: Seeing a failure \"after the merge\" and concluding the fix didn't work. Always check queue time — the build may have started before the merge landed.\n\n### Ancestry check shortcut\n\nWhen queue time is ambiguous (merge happened mid-build), use commit ancestry:\n\n```bash\ngh api repos\u002F{owner}\u002F{repo}\u002Fcompare\u002F{merge_commit}...{build_source_version} --jq '.status'\n# \"ahead\" or \"identical\" = fix is included; \"behind\" = fix is NOT included\n```\n\n## Bulk Failure Aggregation\n\nWhen the user asks \"what are the top N failures\" across a pipeline or time range, use a different approach than single-build investigation.\n\n### Approach\n\n1. Query AzDO for all failed builds in the time range (e.g., last 2 days)\n2. For each failed build, query the timeline to find which jobs\u002Flegs failed\n3. For Helix-based legs, query work item results to get test names, exit codes, machines\n4. Aggregate into a SQL table for analysis:\n\n```sql\nCREATE TABLE failure_survey (\n  build_id INT, leg TEXT, work_item TEXT, exit_code INT,\n  machine TEXT, failure_category TEXT, helix_job TEXT,\n  is_pr_specific BOOLEAN, notes TEXT\n);\n```\n\n5. Group by work item to find the most frequent failures\n6. Separate PR-specific build errors (hit all legs at once) from recurring test failures\n\n### Classification\n\n| Type | Signal | Priority |\n|------|--------|----------|\n| **Recurring test failure** | Same work item fails across many PRs | High — investigate root cause |\n| **PR-specific build error** | Multiple legs fail on same build only | Low — PR author's problem |\n| **Poison PR** | ALL suites fail on one build, exit 133, 1x per platform | Low — filter out; single bad commit crashes runtime before tests run |\n| **Infrastructure error** | exit 71, exit 81, DEVICE_NOT_FOUND | Medium — systemic infra issue |\n| **Sporadic infra** | Single occurrence, no pattern | Ignore — noise |\n\n### Rolling Builds vs PR Builds\n\nWhen assessing **main branch health**, filter to rolling (scheduled) builds using `reasonFilter=schedule`. PR builds include broken PR branches that dominate failure counts and obscure systemic issues.\n\nReal-world example: over the same 2-day window, PR builds produced **423 work item failures** (dominated by a single poison PR with 119 failures). Rolling builds had only **46 failures** — all genuine systemic issues.\n\n```bash\n# Rolling builds only (scheduled, typically 2x daily on main)\ncurl -s \"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds?definitions={defId}&reasonFilter=schedule&resultFilter=failed&minTime={iso8601}&$top=50&api-version=7.1\"\n```\n\nUse rolling builds to:\n- Assess current main branch health\n- Identify systemic test failures vs PR-introduced regressions\n- Verify fix impact (compare rolling failure counts before\u002Fafter merge)\n\n### Cross-Leg View\n\nSome tests fail on multiple platforms. When reporting top-N, group by test name but break out by leg to show which platforms are affected:\n\n```\nSystem.Runtime.Tests: 61 total\n  → ios-arm64 Mono (13), tvos-arm64 Mono (20), android (9), ...\n```\n\nThis reveals whether a failure is platform-specific or cross-platform.\n\n## Machine Distribution Analysis\n\nAfter identifying a recurring failure, determine whether it's **machine-specific** or **systemic**:\n\n1. Collect the machine name from each failing work item\n2. Count failures per machine\n3. Compare to the pool size\n\n| Distribution | Diagnosis | Action |\n|-------------|-----------|--------|\n| 1-2 machines account for most failures | **Machine-specific** — hardware\u002Fconfig issue | Report machines, suggest pool removal |\n| Failures spread evenly across many machines | **Systemic** — software\u002Finfra issue | File infrastructure bug |\n| Slight concentration on a few + spread | **Mixed** — systemic with some worse machines | File bug + flag worst offenders |\n\nExample from real investigation: tvOS failures concentrated on DNCENGTVOS-036 and DNCENGTVOS-022 = machine-specific. Android DEVICE_NOT_FOUND spread across 17 machines = systemic emulator issue.\n\n## Filing Issues from Investigation\n\nWhen investigation reveals a clear root cause, file issues on the appropriate repo:\n\n| Root Cause | File On | Include |\n|------------|---------|---------|\n| XHarness bug (timeout handling, exit code detection, device recovery) | [dotnet\u002Fxharness](https:\u002F\u002Fgithub.com\u002Fdotnet\u002Fxharness) | Pass\u002Ffail comparison evidence, console log URLs, affected machines |\n| Helix infrastructure (retry behavior, machine management) | [dotnet\u002Farcade](https:\u002F\u002Fgithub.com\u002Fdotnet\u002Farcade) or dnceng issue tracker | Failure counts, machine distribution, retry logs |\n| Test code bug | [dotnet\u002Fruntime](https:\u002F\u002Fgithub.com\u002Fdotnet\u002Fruntime) | Failing test name, stack trace, repro steps |\n| Machine hardware | dnceng infrastructure team | Machine names, failure frequency, environment details |\n\nAlways include:\n- **Evidence**: Helix console log URLs and AzDO build URLs\n- **Failure count**: How many times this occurred in the analysis window\n- **Machine list**: Which machines are affected\n- **Pass\u002Ffail comparison**: If intermittent, what differs between passing and failing runs\n\n## Stop Signals\n\n- **Stop downloading logs** after you have 1 clear failing run + 1 clear passing run for comparison. Don't download all 50 builds.\n- **Stop analyzing** when you can explain why the test fails on machine A but passes on machine B. That's the root cause.\n- **Stop if no passing runs exist** — the failure is 100% reproducible, not intermittent. Shift to analyzing the test code itself.\n\n## Anti-Patterns\n\n> 🚨 **Don't rely solely on Helix MCP search tools for log analysis.** Download the console log with curl and use local grep — it's faster and more reliable for pattern matching.\n\n> 🚨 **Don't assume \"flaky\" without evidence.** If the same machine fails every time, it's machine-specific, not flaky. Capture the machine name.\n\n> 🚨 **Don't ignore the \"Exit code detection\" warning as the root cause.** On tvOS\u002FiOS 15+, XHarness logs this warning but it may be a red herring — the real issue can be device log streaming, not exit code detection itself.\n\n> 🚨 **Don't analyze all builds in a pipeline.** Filter to the specific job\u002Fleg first, then look at 3-5 recent runs maximum.\n\n## Reference: Known Helix Queue Patterns\n\n| Queue Pattern | Platform |\n|---------------|----------|\n| `osx.*.amd64.appletv.open` | Apple TV devices (tvOS) |\n| `osx.*.amd64.iphone.open` | iPhone devices (iOS) |\n| `ubuntu.*.amd64.android.*.open` | Android emulators |\n| `ubuntu.*` | Linux |\n| `windows.*` | Windows |\n| `browser-wasm.*` | WASM (Chrome\u002FFirefox) |\n\nQueue name is in the first line of every Helix console log: `workitem ... ({queue}) executed on machine {machine}`.\n\n## Reference: Platform-Specific Resources\n\n- [references\u002Fxharness-patterns.md](references\u002Fxharness-patterns.md) — tvOS\u002FiOS XHarness exit code detection, device log streaming, timeout chain\n- [references\u002Fandroid-patterns.md](references\u002Fandroid-patterns.md) — Android emulator DEVICE_NOT_FOUND, retry\u002Freboot flow, common test failures\n- [references\u002Fwasm-patterns.md](references\u002Fwasm-patterns.md) — WASM\u002Fbrowser SDK build errors, Playwright timeouts, work item naming\n",{"data":34,"body":35},{"name":4,"description":6},{"type":36,"children":37},"root",[38,46,52,59,109,115,153,159,166,171,198,210,225,231,236,241,339,345,350,359,364,416,421,430,440,648,678,684,689,759,764,1113,1151,1157,1162,1167,1210,1215,1264,1270,1275,1611,1617,1622,1665,1671,1690,1696,1708,1713,1766,1780,1786,1791,1842,1848,1853,1859,1882,1928,1941,1947,2076,2082,2102,2121,2171,2176,2194,2200,2205,2214,2219,2225,2243,2261,2355,2360,2366,2371,2489,2494,2537,2543,2576,2582,2597,2611,2625,2639,2645,2769,2782,2788,2821],{"type":39,"tag":40,"props":41,"children":42},"element","h1",{"id":4},[43],{"type":44,"value":45},"text","Helix Investigation",{"type":39,"tag":47,"props":48,"children":49},"p",{},[50],{"type":44,"value":51},"Deep-dive into Helix test failures starting from an AzDO build leg. Goes beyond CI status checks to download raw console logs, compare passing vs failing runs, and identify root causes at the machine\u002Fdevice level.",{"type":39,"tag":53,"props":54,"children":56},"h2",{"id":55},"when-to-use-this-skill",[57],{"type":44,"value":58},"When to Use This Skill",{"type":39,"tag":60,"props":61,"children":62},"ul",{},[63,69,74,79,84,89,94,99,104],{"type":39,"tag":64,"props":65,"children":66},"li",{},[67],{"type":44,"value":68},"User has an AzDO build URL with a failing test leg and wants to understand why",{"type":39,"tag":64,"props":70,"children":71},{},[72],{"type":44,"value":73},"User wants to see raw Helix console logs for a specific work item",{"type":39,"tag":64,"props":75,"children":76},{},[77],{"type":44,"value":78},"User notices a test fails intermittently and wants pass\u002Ffail comparison",{"type":39,"tag":64,"props":80,"children":81},{},[82],{"type":44,"value":83},"User asks \"why does this test only fail on some machines?\"",{"type":39,"tag":64,"props":85,"children":86},{},[87],{"type":44,"value":88},"User wants to correlate AzDO build → Helix job → work item → console output",{"type":39,"tag":64,"props":90,"children":91},{},[92],{"type":44,"value":93},"User mentions XHarness, device tests, or Helix timeouts",{"type":39,"tag":64,"props":95,"children":96},{},[97],{"type":44,"value":98},"User asks for a \"top N\" list of failures across a pipeline or time range",{"type":39,"tag":64,"props":100,"children":101},{},[102],{"type":44,"value":103},"User wants to know if failures are machine-specific or systemic",{"type":39,"tag":64,"props":105,"children":106},{},[107],{"type":44,"value":108},"User sees Android DEVICE_NOT_FOUND or emulator boot failures",{"type":39,"tag":53,"props":110,"children":112},{"id":111},"prerequisites",[113],{"type":44,"value":114},"Prerequisites",{"type":39,"tag":60,"props":116,"children":117},{},[118,123,128,140],{"type":39,"tag":64,"props":119,"children":120},{},[121],{"type":44,"value":122},"Helix MCP tools (hlx-* prefix) for querying jobs, work items, and logs",{"type":39,"tag":64,"props":124,"children":125},{},[126],{"type":44,"value":127},"AzDO MCP tools (ado-* prefix) for querying builds and timelines",{"type":39,"tag":64,"props":129,"children":130},{},[131,138],{"type":39,"tag":132,"props":133,"children":135},"code",{"className":134},[],[136],{"type":44,"value":137},"curl",{"type":44,"value":139}," for downloading raw log files when MCP search is insufficient",{"type":39,"tag":64,"props":141,"children":142},{},[143,145,151],{"type":44,"value":144},"Access to ",{"type":39,"tag":132,"props":146,"children":148},{"className":147},[],[149],{"type":44,"value":150},"https:\u002F\u002Fhelix.dot.net\u002F",{"type":44,"value":152}," API endpoints",{"type":39,"tag":53,"props":154,"children":156},{"id":155},"workflow",[157],{"type":44,"value":158},"Workflow",{"type":39,"tag":160,"props":161,"children":163},"h3",{"id":162},"step-1-start-from-the-azdo-build",[164],{"type":44,"value":165},"Step 1: Start from the AzDO build",{"type":39,"tag":47,"props":167,"children":168},{},[169],{"type":44,"value":170},"Given an AzDO build URL or pipeline definition + build ID:",{"type":39,"tag":172,"props":173,"children":174},"ol",{},[175,180,193],{"type":39,"tag":64,"props":176,"children":177},{},[178],{"type":44,"value":179},"Query the build timeline to find the failing job\u002Fleg",{"type":39,"tag":64,"props":181,"children":182},{},[183,185,191],{"type":44,"value":184},"Identify the Helix job ID from the timeline records — look for ",{"type":39,"tag":132,"props":186,"children":188},{"className":187},[],[189],{"type":44,"value":190},"HelixJobId",{"type":44,"value":192}," in record properties or Helix URLs in the log output",{"type":39,"tag":64,"props":194,"children":195},{},[196],{"type":44,"value":197},"Note the job name, queue, and failure category",{"type":39,"tag":199,"props":200,"children":204},"pre",{"className":201,"code":203,"language":44},[202],"language-text","AzDO Build → Timeline API → Failing job record → Helix job ID\n",[205],{"type":39,"tag":132,"props":206,"children":208},{"__ignoreMap":207},"",[209],{"type":44,"value":203},{"type":39,"tag":211,"props":212,"children":213},"blockquote",{},[214],{"type":39,"tag":47,"props":215,"children":216},{},[217,219],{"type":44,"value":218},"⚠️ AzDO MCP tools may become unavailable mid-session. Fall back to the REST API: ",{"type":39,"tag":132,"props":220,"children":222},{"className":221},[],[223],{"type":44,"value":224},"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds\u002F{buildId}\u002Ftimeline?api-version=7.1",{"type":39,"tag":160,"props":226,"children":228},{"id":227},"step-2-enumerate-helix-work-items",[229],{"type":44,"value":230},"Step 2: Enumerate Helix work items",{"type":39,"tag":47,"props":232,"children":233},{},[234],{"type":44,"value":235},"Query the Helix job for its work items. Each work item is a test group that ran on a specific machine.",{"type":39,"tag":47,"props":237,"children":238},{},[239],{"type":44,"value":240},"Key fields to capture per work item:",{"type":39,"tag":60,"props":242,"children":243},{},[244,271,281,304,314],{"type":39,"tag":64,"props":245,"children":246},{},[247,253,255,261,263,269],{"type":39,"tag":248,"props":249,"children":250},"strong",{},[251],{"type":44,"value":252},"Work item name",{"type":44,"value":254}," (e.g., ",{"type":39,"tag":132,"props":256,"children":258},{"className":257},[],[259],{"type":44,"value":260},"iOS.CoreCLR.R2R.Test",{"type":44,"value":262},", ",{"type":39,"tag":132,"props":264,"children":266},{"className":265},[],[267],{"type":44,"value":268},"System.Runtime.Tests",{"type":44,"value":270},")",{"type":39,"tag":64,"props":272,"children":273},{},[274,279],{"type":39,"tag":248,"props":275,"children":276},{},[277],{"type":44,"value":278},"Exit code",{"type":44,"value":280}," (0 = pass, 143 = SIGTERM\u002Ftimeout, 71 = infra error)",{"type":39,"tag":64,"props":282,"children":283},{},[284,289,290,296,297,303],{"type":39,"tag":248,"props":285,"children":286},{},[287],{"type":44,"value":288},"Machine name",{"type":44,"value":254},{"type":39,"tag":132,"props":291,"children":293},{"className":292},[],[294],{"type":44,"value":295},"DNCENGMAC036",{"type":44,"value":262},{"type":39,"tag":132,"props":298,"children":300},{"className":299},[],[301],{"type":44,"value":302},"DNCENGTVOS-106",{"type":44,"value":270},{"type":39,"tag":64,"props":305,"children":306},{},[307,312],{"type":39,"tag":248,"props":308,"children":309},{},[310],{"type":44,"value":311},"Duration",{"type":44,"value":313}," — long durations with exit 143 indicate timeout kills",{"type":39,"tag":64,"props":315,"children":316},{},[317,322,324,330,331,337],{"type":39,"tag":248,"props":318,"children":319},{},[320],{"type":44,"value":321},"Failure category",{"type":44,"value":323}," — ",{"type":39,"tag":132,"props":325,"children":327},{"className":326},[],[328],{"type":44,"value":329},"Crash",{"type":44,"value":262},{"type":39,"tag":132,"props":332,"children":334},{"className":333},[],[335],{"type":44,"value":336},"InfrastructureError",{"type":44,"value":338},", etc.",{"type":39,"tag":160,"props":340,"children":342},{"id":341},"step-3-download-console-logs",[343],{"type":44,"value":344},"Step 3: Download console logs",{"type":39,"tag":47,"props":346,"children":347},{},[348],{"type":44,"value":349},"The Helix console log URL pattern:",{"type":39,"tag":199,"props":351,"children":354},{"className":352,"code":353,"language":44},[202],"https:\u002F\u002Fhelix.dot.net\u002Fapi\u002F2019-06-17\u002Fjobs\u002F{jobId}\u002Fworkitems\u002F{workItemName}\u002Fconsole\n",[355],{"type":39,"tag":132,"props":356,"children":357},{"__ignoreMap":207},[358],{"type":44,"value":353},{"type":39,"tag":47,"props":360,"children":361},{},[362],{"type":44,"value":363},"Download with curl for local analysis — this is often more reliable than Helix MCP search tools for finding specific patterns:",{"type":39,"tag":199,"props":365,"children":369},{"className":366,"code":367,"language":368,"meta":207,"style":207},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","curl -s \"https:\u002F\u002Fhelix.dot.net\u002Fapi\u002F2019-06-17\u002Fjobs\u002F{jobId}\u002Fworkitems\u002F{workItemName}\u002Fconsole\" > \u002Ftmp\u002F{jobIdPrefix}-console.log\n","bash",[370],{"type":39,"tag":132,"props":371,"children":372},{"__ignoreMap":207},[373],{"type":39,"tag":374,"props":375,"children":378},"span",{"class":376,"line":377},"line",1,[379,384,390,396,401,406,411],{"type":39,"tag":374,"props":380,"children":382},{"style":381},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[383],{"type":44,"value":137},{"type":39,"tag":374,"props":385,"children":387},{"style":386},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[388],{"type":44,"value":389}," -s",{"type":39,"tag":374,"props":391,"children":393},{"style":392},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[394],{"type":44,"value":395}," \"",{"type":39,"tag":374,"props":397,"children":398},{"style":386},[399],{"type":44,"value":400},"https:\u002F\u002Fhelix.dot.net\u002Fapi\u002F2019-06-17\u002Fjobs\u002F{jobId}\u002Fworkitems\u002F{workItemName}\u002Fconsole",{"type":39,"tag":374,"props":402,"children":403},{"style":392},[404],{"type":44,"value":405},"\"",{"type":39,"tag":374,"props":407,"children":408},{"style":392},[409],{"type":44,"value":410}," >",{"type":39,"tag":374,"props":412,"children":413},{"style":386},[414],{"type":44,"value":415}," \u002Ftmp\u002F{jobIdPrefix}-console.log\n",{"type":39,"tag":47,"props":417,"children":418},{},[419],{"type":44,"value":420},"Also check for additional log files (test output, device logs):",{"type":39,"tag":199,"props":422,"children":425},{"className":423,"code":424,"language":44},[202],"https:\u002F\u002Fhelix.dot.net\u002Fapi\u002F2019-06-17\u002Fjobs\u002F{jobId}\u002Fworkitems\u002F{workItemName}\u002Ffiles\n",[426],{"type":39,"tag":132,"props":427,"children":428},{"__ignoreMap":207},[429],{"type":44,"value":424},{"type":39,"tag":47,"props":431,"children":432},{},[433,438],{"type":39,"tag":248,"props":434,"children":435},{},[436],{"type":44,"value":437},"testResults.xml",{"type":44,"value":439},": Many work items produce xUnit\u002FNUnit result XML. Download it for structured failure data — test names, failure messages, stack traces, and pass\u002Ffail counts — especially when console logs lack detail:",{"type":39,"tag":199,"props":441,"children":443},{"className":366,"code":442,"language":368,"meta":207,"style":207},"# List files to find the results XML\ncurl -s \"https:\u002F\u002Fhelix.dot.net\u002Fapi\u002F2019-06-17\u002Fjobs\u002F{jobId}\u002Fworkitems\u002F{workItemName}\u002Ffiles\" | python3 -c \"import sys,json; [print(f['Name']) for f in json.load(sys.stdin)]\"\n# Download (path varies: testResults.xml, xharness-output\u002FtestResults.xml, etc.)\ncurl -sL \"https:\u002F\u002Fhelix.dot.net\u002Fapi\u002F2019-06-17\u002Fjobs\u002F{jobId}\u002Fworkitems\u002F{workItemName}\u002Ffiles\u002F{fileName}\" > \u002Ftmp\u002Ftest-results.xml\n# Quick failure summary\ngrep -c 'result=\"Fail\"' \u002Ftmp\u002Ftest-results.xml\ngrep 'result=\"Fail\"' -A5 \u002Ftmp\u002Ftest-results.xml | grep 'name='\n",[444],{"type":39,"tag":132,"props":445,"children":446},{"__ignoreMap":207},[447,456,510,519,554,563,595],{"type":39,"tag":374,"props":448,"children":449},{"class":376,"line":377},[450],{"type":39,"tag":374,"props":451,"children":453},{"style":452},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[454],{"type":44,"value":455},"# List files to find the results XML\n",{"type":39,"tag":374,"props":457,"children":459},{"class":376,"line":458},2,[460,464,468,472,477,481,486,491,496,500,505],{"type":39,"tag":374,"props":461,"children":462},{"style":381},[463],{"type":44,"value":137},{"type":39,"tag":374,"props":465,"children":466},{"style":386},[467],{"type":44,"value":389},{"type":39,"tag":374,"props":469,"children":470},{"style":392},[471],{"type":44,"value":395},{"type":39,"tag":374,"props":473,"children":474},{"style":386},[475],{"type":44,"value":476},"https:\u002F\u002Fhelix.dot.net\u002Fapi\u002F2019-06-17\u002Fjobs\u002F{jobId}\u002Fworkitems\u002F{workItemName}\u002Ffiles",{"type":39,"tag":374,"props":478,"children":479},{"style":392},[480],{"type":44,"value":405},{"type":39,"tag":374,"props":482,"children":483},{"style":392},[484],{"type":44,"value":485}," |",{"type":39,"tag":374,"props":487,"children":488},{"style":381},[489],{"type":44,"value":490}," python3",{"type":39,"tag":374,"props":492,"children":493},{"style":386},[494],{"type":44,"value":495}," -c",{"type":39,"tag":374,"props":497,"children":498},{"style":392},[499],{"type":44,"value":395},{"type":39,"tag":374,"props":501,"children":502},{"style":386},[503],{"type":44,"value":504},"import sys,json; [print(f['Name']) for f in json.load(sys.stdin)]",{"type":39,"tag":374,"props":506,"children":507},{"style":392},[508],{"type":44,"value":509},"\"\n",{"type":39,"tag":374,"props":511,"children":513},{"class":376,"line":512},3,[514],{"type":39,"tag":374,"props":515,"children":516},{"style":452},[517],{"type":44,"value":518},"# Download (path varies: testResults.xml, xharness-output\u002FtestResults.xml, etc.)\n",{"type":39,"tag":374,"props":520,"children":522},{"class":376,"line":521},4,[523,527,532,536,541,545,549],{"type":39,"tag":374,"props":524,"children":525},{"style":381},[526],{"type":44,"value":137},{"type":39,"tag":374,"props":528,"children":529},{"style":386},[530],{"type":44,"value":531}," -sL",{"type":39,"tag":374,"props":533,"children":534},{"style":392},[535],{"type":44,"value":395},{"type":39,"tag":374,"props":537,"children":538},{"style":386},[539],{"type":44,"value":540},"https:\u002F\u002Fhelix.dot.net\u002Fapi\u002F2019-06-17\u002Fjobs\u002F{jobId}\u002Fworkitems\u002F{workItemName}\u002Ffiles\u002F{fileName}",{"type":39,"tag":374,"props":542,"children":543},{"style":392},[544],{"type":44,"value":405},{"type":39,"tag":374,"props":546,"children":547},{"style":392},[548],{"type":44,"value":410},{"type":39,"tag":374,"props":550,"children":551},{"style":386},[552],{"type":44,"value":553}," \u002Ftmp\u002Ftest-results.xml\n",{"type":39,"tag":374,"props":555,"children":557},{"class":376,"line":556},5,[558],{"type":39,"tag":374,"props":559,"children":560},{"style":452},[561],{"type":44,"value":562},"# Quick failure summary\n",{"type":39,"tag":374,"props":564,"children":566},{"class":376,"line":565},6,[567,572,576,581,586,591],{"type":39,"tag":374,"props":568,"children":569},{"style":381},[570],{"type":44,"value":571},"grep",{"type":39,"tag":374,"props":573,"children":574},{"style":386},[575],{"type":44,"value":495},{"type":39,"tag":374,"props":577,"children":578},{"style":392},[579],{"type":44,"value":580}," '",{"type":39,"tag":374,"props":582,"children":583},{"style":386},[584],{"type":44,"value":585},"result=\"Fail\"",{"type":39,"tag":374,"props":587,"children":588},{"style":392},[589],{"type":44,"value":590},"'",{"type":39,"tag":374,"props":592,"children":593},{"style":386},[594],{"type":44,"value":553},{"type":39,"tag":374,"props":596,"children":598},{"class":376,"line":597},7,[599,603,607,611,615,620,625,629,634,638,643],{"type":39,"tag":374,"props":600,"children":601},{"style":381},[602],{"type":44,"value":571},{"type":39,"tag":374,"props":604,"children":605},{"style":392},[606],{"type":44,"value":580},{"type":39,"tag":374,"props":608,"children":609},{"style":386},[610],{"type":44,"value":585},{"type":39,"tag":374,"props":612,"children":613},{"style":392},[614],{"type":44,"value":590},{"type":39,"tag":374,"props":616,"children":617},{"style":386},[618],{"type":44,"value":619}," -A5",{"type":39,"tag":374,"props":621,"children":622},{"style":386},[623],{"type":44,"value":624}," \u002Ftmp\u002Ftest-results.xml",{"type":39,"tag":374,"props":626,"children":627},{"style":392},[628],{"type":44,"value":485},{"type":39,"tag":374,"props":630,"children":631},{"style":381},[632],{"type":44,"value":633}," grep",{"type":39,"tag":374,"props":635,"children":636},{"style":392},[637],{"type":44,"value":580},{"type":39,"tag":374,"props":639,"children":640},{"style":386},[641],{"type":44,"value":642},"name=",{"type":39,"tag":374,"props":644,"children":645},{"style":392},[646],{"type":44,"value":647},"'\n",{"type":39,"tag":211,"props":649,"children":650},{},[651],{"type":39,"tag":47,"props":652,"children":653},{},[654,656,662,663,669,671,676],{"type":44,"value":655},"💡 Helix MCP search tools (",{"type":39,"tag":132,"props":657,"children":659},{"className":658},[],[660],{"type":44,"value":661},"hlx_search_log",{"type":44,"value":262},{"type":39,"tag":132,"props":664,"children":666},{"className":665},[],[667],{"type":44,"value":668},"hlx_search_file",{"type":44,"value":670},") may miss patterns — they can be case-sensitive or have limited regex support. When searching for specific strings, download the file and use local ",{"type":39,"tag":132,"props":672,"children":674},{"className":673},[],[675],{"type":44,"value":571},{"type":44,"value":677}," for reliability.",{"type":39,"tag":160,"props":679,"children":681},{"id":680},"step-4-analyze-the-console-log",[682],{"type":44,"value":683},"Step 4: Analyze the console log",{"type":39,"tag":47,"props":685,"children":686},{},[687],{"type":44,"value":688},"Look for these key patterns in order:",{"type":39,"tag":172,"props":690,"children":691},{},[692,702,720,730],{"type":39,"tag":64,"props":693,"children":694},{},[695,700],{"type":39,"tag":248,"props":696,"children":697},{},[698],{"type":44,"value":699},"Setup phase",{"type":44,"value":701}," — machine name, queue, environment (OS version, Xcode version, device model)",{"type":39,"tag":64,"props":703,"children":704},{},[705,710,712,718],{"type":39,"tag":248,"props":706,"children":707},{},[708],{"type":44,"value":709},"Test execution",{"type":44,"value":711}," — command line used (e.g., XHarness ",{"type":39,"tag":132,"props":713,"children":715},{"className":714},[],[716],{"type":44,"value":717},"apple run",{"type":44,"value":719}," with flags), app launch, test output",{"type":39,"tag":64,"props":721,"children":722},{},[723,728],{"type":39,"tag":248,"props":724,"children":725},{},[726],{"type":44,"value":727},"Completion signals",{"type":44,"value":729}," — exit codes, timeout messages, crash reports",{"type":39,"tag":64,"props":731,"children":732},{},[733,738,739,745,746,752,753],{"type":39,"tag":248,"props":734,"children":735},{},[736],{"type":44,"value":737},"Helix wrapper",{"type":44,"value":323},{"type":39,"tag":132,"props":740,"children":742},{"className":741},[],[743],{"type":44,"value":744},"WORKLOAD TIMED OUT",{"type":44,"value":262},{"type":39,"tag":132,"props":747,"children":749},{"className":748},[],[750],{"type":44,"value":751},"exit_code=",{"type":44,"value":262},{"type":39,"tag":132,"props":754,"children":756},{"className":755},[],[757],{"type":44,"value":758},"Command exited with",{"type":39,"tag":47,"props":760,"children":761},{},[762],{"type":44,"value":763},"Common failure signatures:",{"type":39,"tag":765,"props":766,"children":767},"table",{},[768,787],{"type":39,"tag":769,"props":770,"children":771},"thead",{},[772],{"type":39,"tag":773,"props":774,"children":775},"tr",{},[776,782],{"type":39,"tag":777,"props":778,"children":779},"th",{},[780],{"type":44,"value":781},"Pattern",{"type":39,"tag":777,"props":783,"children":784},{},[785],{"type":44,"value":786},"Meaning",{"type":39,"tag":788,"props":789,"children":790},"tbody",{},[791,816,833,858,889,906,925,942,971,995,1008,1033,1052,1071,1088],{"type":39,"tag":773,"props":792,"children":793},{},[794,811],{"type":39,"tag":795,"props":796,"children":797},"td",{},[798,804,806],{"type":39,"tag":132,"props":799,"children":801},{"className":800},[],[802],{"type":44,"value":803},"exit_code=143",{"type":44,"value":805}," + ",{"type":39,"tag":132,"props":807,"children":809},{"className":808},[],[810],{"type":44,"value":744},{"type":39,"tag":795,"props":812,"children":813},{},[814],{"type":44,"value":815},"Helix killed the process after timeout (SIGTERM)",{"type":39,"tag":773,"props":817,"children":818},{},[819,828],{"type":39,"tag":795,"props":820,"children":821},{},[822],{"type":39,"tag":132,"props":823,"children":825},{"className":824},[],[826],{"type":44,"value":827},"exit_code=71",{"type":39,"tag":795,"props":829,"children":830},{},[831],{"type":44,"value":832},"Infrastructure error",{"type":39,"tag":773,"props":834,"children":835},{},[836,853],{"type":39,"tag":795,"props":837,"children":838},{},[839,845,847],{"type":39,"tag":132,"props":840,"children":842},{"className":841},[],[843],{"type":44,"value":844},"exit_code=81",{"type":44,"value":846}," \u002F ",{"type":39,"tag":132,"props":848,"children":850},{"className":849},[],[851],{"type":44,"value":852},"DEVICE_NOT_FOUND",{"type":39,"tag":795,"props":854,"children":855},{},[856],{"type":44,"value":857},"Android emulator not running or not responding",{"type":39,"tag":773,"props":859,"children":860},{},[861,877],{"type":39,"tag":795,"props":862,"children":863},{},[864,870,871],{"type":39,"tag":132,"props":865,"children":867},{"className":866},[],[868],{"type":44,"value":869},"exit_code=80",{"type":44,"value":846},{"type":39,"tag":132,"props":872,"children":874},{"className":873},[],[875],{"type":44,"value":876},"APP_CRASH",{"type":39,"tag":795,"props":878,"children":879},{},[880,882,887],{"type":44,"value":881},"XHarness device error — ",{"type":39,"tag":248,"props":883,"children":884},{},[885],{"type":44,"value":886},"may be false",{"type":44,"value":888}," (see below)",{"type":39,"tag":773,"props":890,"children":891},{},[892,901],{"type":39,"tag":795,"props":893,"children":894},{},[895],{"type":39,"tag":132,"props":896,"children":898},{"className":897},[],[899],{"type":44,"value":900},"Run timed out after N seconds",{"type":39,"tag":795,"props":902,"children":903},{},[904],{"type":44,"value":905},"XHarness hit its own timeout before Helix",{"type":39,"tag":773,"props":907,"children":908},{},[909,920],{"type":39,"tag":795,"props":910,"children":911},{},[912,918],{"type":39,"tag":132,"props":913,"children":915},{"className":914},[],[916],{"type":44,"value":917},"TIMED_OUT",{"type":44,"value":919}," (XHarness)",{"type":39,"tag":795,"props":921,"children":922},{},[923],{"type":44,"value":924},"XHarness-level timeout (distinct from Helix timeout)",{"type":39,"tag":773,"props":926,"children":927},{},[928,937],{"type":39,"tag":795,"props":929,"children":930},{},[931],{"type":39,"tag":132,"props":932,"children":934},{"className":933},[],[935],{"type":44,"value":936},"mlaunch exited with {code}",{"type":39,"tag":795,"props":938,"children":939},{},[940],{"type":44,"value":941},"App completed — check if XHarness noticed",{"type":39,"tag":773,"props":943,"children":944},{},[945,961],{"type":39,"tag":795,"props":946,"children":947},{},[948,954,956],{"type":39,"tag":132,"props":949,"children":951},{"className":950},[],[952],{"type":44,"value":953},"Detected test end tag",{"type":44,"value":955}," then ",{"type":39,"tag":132,"props":957,"children":959},{"className":958},[],[960],{"type":44,"value":876},{"type":39,"tag":795,"props":962,"children":963},{},[964,969],{"type":39,"tag":248,"props":965,"children":966},{},[967],{"type":44,"value":968},"False failure",{"type":44,"value":970}," — tests passed but device communication failed afterward",{"type":39,"tag":773,"props":972,"children":973},{},[974,990],{"type":39,"tag":795,"props":975,"children":976},{},[977,983,984],{"type":39,"tag":132,"props":978,"children":980},{"className":979},[],[981],{"type":44,"value":982},"Mercury error 1000",{"type":44,"value":846},{"type":39,"tag":132,"props":985,"children":987},{"className":986},[],[988],{"type":44,"value":989},"RSD error 0xE8000003",{"type":39,"tag":795,"props":991,"children":992},{},[993],{"type":44,"value":994},"devicectl file copy failure (tvOS\u002FiOS)",{"type":39,"tag":773,"props":996,"children":997},{},[998,1003],{"type":39,"tag":795,"props":999,"children":1000},{},[1001],{"type":44,"value":1002},"Device log file empty\u002Fremoved",{"type":39,"tag":795,"props":1004,"children":1005},{},[1006],{"type":44,"value":1007},"Device communication issue (tvOS\u002FiOS)",{"type":39,"tag":773,"props":1009,"children":1010},{},[1011,1028],{"type":39,"tag":795,"props":1012,"children":1013},{},[1014,1020,1022],{"type":39,"tag":132,"props":1015,"children":1017},{"className":1016},[],[1018],{"type":44,"value":1019},"emulator-5554",{"type":44,"value":1021}," found but not ",{"type":39,"tag":132,"props":1023,"children":1025},{"className":1024},[],[1026],{"type":44,"value":1027},"emulator-5556",{"type":39,"tag":795,"props":1029,"children":1030},{},[1031],{"type":44,"value":1032},"Wrong emulator architecture booted",{"type":39,"tag":773,"props":1034,"children":1035},{},[1036,1047],{"type":39,"tag":795,"props":1037,"children":1038},{},[1039,1045],{"type":39,"tag":132,"props":1040,"children":1042},{"className":1041},[],[1043],{"type":44,"value":1044},"Attempt.3",{"type":44,"value":1046}," in work item name",{"type":39,"tag":795,"props":1048,"children":1049},{},[1050],{"type":44,"value":1051},"All retries exhausted",{"type":39,"tag":773,"props":1053,"children":1054},{},[1055,1066],{"type":39,"tag":795,"props":1056,"children":1057},{},[1058,1064],{"type":39,"tag":132,"props":1059,"children":1061},{"className":1060},[],[1062],{"type":44,"value":1063},"Expected 0 exit code but got 1",{"type":44,"value":1065}," (dotnet build)",{"type":39,"tag":795,"props":1067,"children":1068},{},[1069],{"type":44,"value":1070},"SDK\u002FMSBuild build failure — 100% fail rate, not a test bug",{"type":39,"tag":773,"props":1072,"children":1073},{},[1074,1083],{"type":39,"tag":795,"props":1075,"children":1076},{},[1077],{"type":39,"tag":132,"props":1078,"children":1080},{"className":1079},[],[1081],{"type":44,"value":1082},"System.TimeoutException: Timeout 30000ms exceeded",{"type":39,"tag":795,"props":1084,"children":1085},{},[1086],{"type":44,"value":1087},"Playwright browser automation timeout (Blazor)",{"type":39,"tag":773,"props":1089,"children":1090},{},[1091,1108],{"type":39,"tag":795,"props":1092,"children":1093},{},[1094,1100,1102],{"type":39,"tag":132,"props":1095,"children":1097},{"className":1096},[],[1098],{"type":44,"value":1099},"exit_code=133",{"type":44,"value":1101}," (128+5=SIGTRAP) + ",{"type":39,"tag":132,"props":1103,"children":1105},{"className":1104},[],[1106],{"type":44,"value":1107},"ASSERT FAILED",{"type":39,"tag":795,"props":1109,"children":1110},{},[1111],{"type":44,"value":1112},"Runtime assertion failure \u002F core dump — every test suite crashes",{"type":39,"tag":211,"props":1114,"children":1115},{},[1116],{"type":39,"tag":47,"props":1117,"children":1118},{},[1119,1121,1126,1128,1133,1135,1141,1143,1149],{"type":44,"value":1120},"⚠️ ",{"type":39,"tag":248,"props":1122,"children":1123},{},[1124],{"type":44,"value":1125},"False failure detection is critical.",{"type":44,"value":1127}," XHarness can report APP_CRASH (exit 80) or TIMED_OUT (exit 143) even when ",{"type":39,"tag":248,"props":1129,"children":1130},{},[1131],{"type":44,"value":1132},"all tests pass",{"type":44,"value":1134},". Always check for ",{"type":39,"tag":132,"props":1136,"children":1138},{"className":1137},[],[1139],{"type":44,"value":1140},"\"Detected test end tag\"",{"type":44,"value":1142}," or ",{"type":39,"tag":132,"props":1144,"children":1146},{"className":1145},[],[1147],{"type":44,"value":1148},"\"Test run completed\"",{"type":44,"value":1150}," in the log before trusting the exit code. If tests completed successfully but XHarness still reports failure, the issue is post-completion device communication, not a test bug.",{"type":39,"tag":160,"props":1152,"children":1154},{"id":1153},"step-5-compare-passing-vs-failing-runs",[1155],{"type":44,"value":1156},"Step 5: Compare passing vs failing runs",{"type":39,"tag":47,"props":1158,"children":1159},{},[1160],{"type":44,"value":1161},"This is the highest-value step for intermittent failures. Find a passing run of the same work item and download its console log too.",{"type":39,"tag":47,"props":1163,"children":1164},{},[1165],{"type":44,"value":1166},"Compare side-by-side:",{"type":39,"tag":172,"props":1168,"children":1169},{},[1170,1180,1190,1200],{"type":39,"tag":64,"props":1171,"children":1172},{},[1173,1178],{"type":39,"tag":248,"props":1174,"children":1175},{},[1176],{"type":44,"value":1177},"Machine\u002Fdevice",{"type":44,"value":1179}," — are failures concentrated on specific machines?",{"type":39,"tag":64,"props":1181,"children":1182},{},[1183,1188],{"type":39,"tag":248,"props":1184,"children":1185},{},[1186],{"type":44,"value":1187},"Environment",{"type":44,"value":1189}," — OS version, SDK version, device model differences",{"type":39,"tag":64,"props":1191,"children":1192},{},[1193,1198],{"type":39,"tag":248,"props":1194,"children":1195},{},[1196],{"type":44,"value":1197},"Log divergence point",{"type":44,"value":1199}," — where does the passing run succeed and the failing run hang or crash?",{"type":39,"tag":64,"props":1201,"children":1202},{},[1203,1208],{"type":39,"tag":248,"props":1204,"children":1205},{},[1206],{"type":44,"value":1207},"File artifacts",{"type":44,"value":1209}," — does the passing run produce files (device logs, test results) that the failing run doesn't?",{"type":39,"tag":47,"props":1211,"children":1212},{},[1213],{"type":44,"value":1214},"Use a SQL table to track findings across multiple runs:",{"type":39,"tag":199,"props":1216,"children":1220},{"className":1217,"code":1218,"language":1219,"meta":207,"style":207},"language-sql shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","CREATE TABLE helix_runs (\n  job_id TEXT, work_item TEXT, machine TEXT, exit_code INT,\n  duration_sec INT, queue TEXT, os_version TEXT, passed BOOLEAN,\n  notes TEXT\n);\n","sql",[1221],{"type":39,"tag":132,"props":1222,"children":1223},{"__ignoreMap":207},[1224,1232,1240,1248,1256],{"type":39,"tag":374,"props":1225,"children":1226},{"class":376,"line":377},[1227],{"type":39,"tag":374,"props":1228,"children":1229},{},[1230],{"type":44,"value":1231},"CREATE TABLE helix_runs (\n",{"type":39,"tag":374,"props":1233,"children":1234},{"class":376,"line":458},[1235],{"type":39,"tag":374,"props":1236,"children":1237},{},[1238],{"type":44,"value":1239},"  job_id TEXT, work_item TEXT, machine TEXT, exit_code INT,\n",{"type":39,"tag":374,"props":1241,"children":1242},{"class":376,"line":512},[1243],{"type":39,"tag":374,"props":1244,"children":1245},{},[1246],{"type":44,"value":1247},"  duration_sec INT, queue TEXT, os_version TEXT, passed BOOLEAN,\n",{"type":39,"tag":374,"props":1249,"children":1250},{"class":376,"line":521},[1251],{"type":39,"tag":374,"props":1252,"children":1253},{},[1254],{"type":44,"value":1255},"  notes TEXT\n",{"type":39,"tag":374,"props":1257,"children":1258},{"class":376,"line":556},[1259],{"type":39,"tag":374,"props":1260,"children":1261},{},[1262],{"type":44,"value":1263},");\n",{"type":39,"tag":160,"props":1265,"children":1267},{"id":1266},"step-6-identify-root-cause-pattern",[1268],{"type":44,"value":1269},"Step 6: Identify root cause pattern",{"type":39,"tag":47,"props":1271,"children":1272},{},[1273],{"type":44,"value":1274},"Common root cause categories:",{"type":39,"tag":765,"props":1276,"children":1277},{},[1278,1299],{"type":39,"tag":769,"props":1279,"children":1280},{},[1281],{"type":39,"tag":773,"props":1282,"children":1283},{},[1284,1289,1294],{"type":39,"tag":777,"props":1285,"children":1286},{},[1287],{"type":44,"value":1288},"Category",{"type":39,"tag":777,"props":1290,"children":1291},{},[1292],{"type":44,"value":1293},"Signal",{"type":39,"tag":777,"props":1295,"children":1296},{},[1297],{"type":44,"value":1298},"Action",{"type":39,"tag":788,"props":1300,"children":1301},{},[1302,1329,1350,1382,1403,1424,1453,1474,1495,1521,1542,1569,1590],{"type":39,"tag":773,"props":1303,"children":1304},{},[1305,1312,1324],{"type":39,"tag":795,"props":1306,"children":1307},{},[1308],{"type":39,"tag":248,"props":1309,"children":1310},{},[1311],{"type":44,"value":968},{"type":39,"tag":795,"props":1313,"children":1314},{},[1315,1317,1322],{"type":44,"value":1316},"Tests pass (",{"type":39,"tag":132,"props":1318,"children":1320},{"className":1319},[],[1321],{"type":44,"value":1148},{"type":44,"value":1323},") but exit 80\u002F143 reported",{"type":39,"tag":795,"props":1325,"children":1326},{},[1327],{"type":44,"value":1328},"Device communication failed post-completion — file\u002Fupdate on dotnet\u002Fxharness",{"type":39,"tag":773,"props":1330,"children":1331},{},[1332,1340,1345],{"type":39,"tag":795,"props":1333,"children":1334},{},[1335],{"type":39,"tag":248,"props":1336,"children":1337},{},[1338],{"type":44,"value":1339},"Machine-specific",{"type":39,"tag":795,"props":1341,"children":1342},{},[1343],{"type":44,"value":1344},"Same test, same code, different outcomes by machine",{"type":39,"tag":795,"props":1346,"children":1347},{},[1348],{"type":44,"value":1349},"Report machine name, suggest pool removal",{"type":39,"tag":773,"props":1351,"children":1352},{},[1353,1361,1377],{"type":39,"tag":795,"props":1354,"children":1355},{},[1356],{"type":39,"tag":248,"props":1357,"children":1358},{},[1359],{"type":44,"value":1360},"Device communication",{"type":39,"tag":795,"props":1362,"children":1363},{},[1364,1366,1371,1372],{"type":44,"value":1365},"Empty device logs, log stream hangs, ",{"type":39,"tag":132,"props":1367,"children":1369},{"className":1368},[],[1370],{"type":44,"value":982},{"type":44,"value":262},{"type":39,"tag":132,"props":1373,"children":1375},{"className":1374},[],[1376],{"type":44,"value":989},{"type":39,"tag":795,"props":1378,"children":1379},{},[1380],{"type":44,"value":1381},"USB\u002Fconnectivity issue on host",{"type":39,"tag":773,"props":1383,"children":1384},{},[1385,1393,1398],{"type":39,"tag":795,"props":1386,"children":1387},{},[1388],{"type":39,"tag":248,"props":1389,"children":1390},{},[1391],{"type":44,"value":1392},"Emulator failure",{"type":39,"tag":795,"props":1394,"children":1395},{},[1396],{"type":44,"value":1397},"DEVICE_NOT_FOUND (exit 81), wrong emulator arch",{"type":39,"tag":795,"props":1399,"children":1400},{},[1401],{"type":44,"value":1402},"Systemic if spread across machines; file on dotnet\u002Fxharness for recovery logic",{"type":39,"tag":773,"props":1404,"children":1405},{},[1406,1414,1419],{"type":39,"tag":795,"props":1407,"children":1408},{},[1409],{"type":39,"tag":248,"props":1410,"children":1411},{},[1412],{"type":44,"value":1413},"Test infrastructure",{"type":39,"tag":795,"props":1415,"children":1416},{},[1417],{"type":44,"value":1418},"XHarness\u002FHelix wrapper bug",{"type":39,"tag":795,"props":1420,"children":1421},{},[1422],{"type":44,"value":1423},"File issue on dotnet\u002Fxharness or dotnet\u002Farcade",{"type":39,"tag":773,"props":1425,"children":1426},{},[1427,1435,1440],{"type":39,"tag":795,"props":1428,"children":1429},{},[1430],{"type":39,"tag":248,"props":1431,"children":1432},{},[1433],{"type":44,"value":1434},"Deterministic test bug",{"type":39,"tag":795,"props":1436,"children":1437},{},[1438],{"type":44,"value":1439},"100% failure rate, same assertion, every run",{"type":39,"tag":795,"props":1441,"children":1442},{},[1443,1445,1451],{"type":44,"value":1444},"Test expectation is wrong, not infra — file on dotnet\u002Fruntime, check for ",{"type":39,"tag":132,"props":1446,"children":1448},{"className":1447},[],[1449],{"type":44,"value":1450},"blocking-clean-ci",{"type":44,"value":1452}," label",{"type":39,"tag":773,"props":1454,"children":1455},{},[1456,1464,1469],{"type":39,"tag":795,"props":1457,"children":1458},{},[1459],{"type":39,"tag":248,"props":1460,"children":1461},{},[1462],{"type":44,"value":1463},"Genuine test failure",{"type":39,"tag":795,"props":1465,"children":1466},{},[1467],{"type":44,"value":1468},"Test code exercises broken runtime behavior",{"type":39,"tag":795,"props":1470,"children":1471},{},[1472],{"type":44,"value":1473},"File issue on dotnet\u002Fruntime",{"type":39,"tag":773,"props":1475,"children":1476},{},[1477,1485,1490],{"type":39,"tag":795,"props":1478,"children":1479},{},[1480],{"type":39,"tag":248,"props":1481,"children":1482},{},[1483],{"type":44,"value":1484},"Timeout misconfiguration",{"type":39,"tag":795,"props":1486,"children":1487},{},[1488],{"type":44,"value":1489},"Test passes but harness timeout too short",{"type":39,"tag":795,"props":1491,"children":1492},{},[1493],{"type":44,"value":1494},"Adjust timeout parameter",{"type":39,"tag":773,"props":1496,"children":1497},{},[1498,1506,1516],{"type":39,"tag":795,"props":1499,"children":1500},{},[1501],{"type":39,"tag":248,"props":1502,"children":1503},{},[1504],{"type":44,"value":1505},"Retry exhaustion",{"type":39,"tag":795,"props":1507,"children":1508},{},[1509,1514],{"type":39,"tag":132,"props":1510,"children":1512},{"className":1511},[],[1513],{"type":44,"value":1044},{"type":44,"value":1515}," still failing, retries hit same broken machine",{"type":39,"tag":795,"props":1517,"children":1518},{},[1519],{"type":44,"value":1520},"Helix retries on same machine before reboot takes effect",{"type":39,"tag":773,"props":1522,"children":1523},{},[1524,1532,1537],{"type":39,"tag":795,"props":1525,"children":1526},{},[1527],{"type":39,"tag":248,"props":1528,"children":1529},{},[1530],{"type":44,"value":1531},"XML truncation",{"type":39,"tag":795,"props":1533,"children":1534},{},[1535],{"type":44,"value":1536},"All tests pass but app killed mid-result-write → truncated XML → crash reported",{"type":39,"tag":795,"props":1538,"children":1539},{},[1540],{"type":44,"value":1541},"Large test suites on constrained devices (tvOS memory limits)",{"type":39,"tag":773,"props":1543,"children":1544},{},[1545,1553,1564],{"type":39,"tag":795,"props":1546,"children":1547},{},[1548],{"type":39,"tag":248,"props":1549,"children":1550},{},[1551],{"type":44,"value":1552},"SDK\u002Fbuild break",{"type":39,"tag":795,"props":1554,"children":1555},{},[1556,1562],{"type":39,"tag":132,"props":1557,"children":1559},{"className":1558},[],[1560],{"type":44,"value":1561},"dotnet build",{"type":44,"value":1563}," exit 1, 100% fail rate, error in SDK targets",{"type":39,"tag":795,"props":1565,"children":1566},{},[1567],{"type":44,"value":1568},"Not a test bug — file on dotnet\u002Faspnetcore or dotnet\u002Fsdk",{"type":39,"tag":773,"props":1570,"children":1571},{},[1572,1580,1585],{"type":39,"tag":795,"props":1573,"children":1574},{},[1575],{"type":39,"tag":248,"props":1576,"children":1577},{},[1578],{"type":44,"value":1579},"Browser automation",{"type":39,"tag":795,"props":1581,"children":1582},{},[1583],{"type":44,"value":1584},"Playwright timeout on UI element, app builds fine",{"type":39,"tag":795,"props":1586,"children":1587},{},[1588],{"type":44,"value":1589},"Resource contention or Blazor rendering regression",{"type":39,"tag":773,"props":1591,"children":1592},{},[1593,1601,1606],{"type":39,"tag":795,"props":1594,"children":1595},{},[1596],{"type":39,"tag":248,"props":1597,"children":1598},{},[1599],{"type":44,"value":1600},"Poison PR",{"type":39,"tag":795,"props":1602,"children":1603},{},[1604],{"type":44,"value":1605},"Exit 133 (SIGTRAP), ALL suites fail on same build, 1x per platform per suite",{"type":39,"tag":795,"props":1607,"children":1608},{},[1609],{"type":44,"value":1610},"Runtime assertion failure — single bad commit crashes before tests run. Filter out this build; failures are misleading",{"type":39,"tag":160,"props":1612,"children":1614},{"id":1613},"step-7-report-findings",[1615],{"type":44,"value":1616},"Step 7: Report findings",{"type":39,"tag":47,"props":1618,"children":1619},{},[1620],{"type":44,"value":1621},"Provide:",{"type":39,"tag":172,"props":1623,"children":1624},{},[1625,1635,1645,1655],{"type":39,"tag":64,"props":1626,"children":1627},{},[1628,1633],{"type":39,"tag":248,"props":1629,"children":1630},{},[1631],{"type":44,"value":1632},"Summary table",{"type":44,"value":1634}," — all analyzed runs with machine, exit code, pass\u002Ffail, key observation",{"type":39,"tag":64,"props":1636,"children":1637},{},[1638,1643],{"type":39,"tag":248,"props":1639,"children":1640},{},[1641],{"type":44,"value":1642},"Root cause",{"type":44,"value":1644}," — which category, with evidence from the logs",{"type":39,"tag":64,"props":1646,"children":1647},{},[1648,1653],{"type":39,"tag":248,"props":1649,"children":1650},{},[1651],{"type":44,"value":1652},"Direct links",{"type":44,"value":1654}," — Helix console log URLs, AzDO build URLs",{"type":39,"tag":64,"props":1656,"children":1657},{},[1658,1663],{"type":39,"tag":248,"props":1659,"children":1660},{},[1661],{"type":44,"value":1662},"Recommended action",{"type":44,"value":1664}," — machine removal, issue to file, timeout adjustment, etc.",{"type":39,"tag":53,"props":1666,"children":1668},{"id":1667},"fix-verification",[1669],{"type":44,"value":1670},"Fix Verification",{"type":39,"tag":47,"props":1672,"children":1673},{},[1674,1676,1681,1683,1688],{"type":44,"value":1675},"When a PR is merged to fix a test failure, verify it's working by checking builds ",{"type":39,"tag":248,"props":1677,"children":1678},{},[1679],{"type":44,"value":1680},"queued after",{"type":44,"value":1682}," the merge — not builds that ",{"type":39,"tag":248,"props":1684,"children":1685},{},[1686],{"type":44,"value":1687},"finished",{"type":44,"value":1689}," after it.",{"type":39,"tag":160,"props":1691,"children":1693},{"id":1692},"why-queue-time-matters",[1694],{"type":44,"value":1695},"Why queue time matters",{"type":39,"tag":47,"props":1697,"children":1698},{},[1699,1701,1706],{"type":44,"value":1700},"A build queued 10 minutes before a PR merges runs against the ",{"type":39,"tag":248,"props":1702,"children":1703},{},[1704],{"type":44,"value":1705},"old",{"type":44,"value":1707}," source, even if it finishes hours later. Only builds queued after the merge include the fix.",{"type":39,"tag":160,"props":1709,"children":1711},{"id":1710},"workflow-1",[1712],{"type":44,"value":158},{"type":39,"tag":172,"props":1714,"children":1715},{},[1716,1727,1739,1751,1756,1761],{"type":39,"tag":64,"props":1717,"children":1718},{},[1719,1721],{"type":44,"value":1720},"Find the fix PR and its merge timestamp: ",{"type":39,"tag":132,"props":1722,"children":1724},{"className":1723},[],[1725],{"type":44,"value":1726},"gh pr view {number} --repo {owner\u002Frepo} --json mergedAt",{"type":39,"tag":64,"props":1728,"children":1729},{},[1730,1732,1737],{"type":44,"value":1731},"Query recent builds for the pipeline — ",{"type":39,"tag":248,"props":1733,"children":1734},{},[1735],{"type":44,"value":1736},"include all statuses",{"type":44,"value":1738}," (completed, inProgress, notStarted). If you only check completed builds, you'll miss active builds currently testing the fix.",{"type":39,"tag":64,"props":1740,"children":1741},{},[1742,1744,1749],{"type":44,"value":1743},"For each build with the relevant failure, check its ",{"type":39,"tag":248,"props":1745,"children":1746},{},[1747],{"type":44,"value":1748},"queueTime",{"type":44,"value":1750}," (not finishTime)",{"type":39,"tag":64,"props":1752,"children":1753},{},[1754],{"type":44,"value":1755},"Partition into pre-merge and post-merge builds",{"type":39,"tag":64,"props":1757,"children":1758},{},[1759],{"type":44,"value":1760},"If post-merge builds still show the failure → fix didn't work or there's a second issue",{"type":39,"tag":64,"props":1762,"children":1763},{},[1764],{"type":44,"value":1765},"If only pre-merge builds fail → fix is confirmed clean",{"type":39,"tag":211,"props":1767,"children":1768},{},[1769],{"type":39,"tag":47,"props":1770,"children":1771},{},[1772,1773,1778],{"type":44,"value":1120},{"type":39,"tag":248,"props":1774,"children":1775},{},[1776],{"type":44,"value":1777},"Common mistake",{"type":44,"value":1779},": Seeing a failure \"after the merge\" and concluding the fix didn't work. Always check queue time — the build may have started before the merge landed.",{"type":39,"tag":160,"props":1781,"children":1783},{"id":1782},"ancestry-check-shortcut",[1784],{"type":44,"value":1785},"Ancestry check shortcut",{"type":39,"tag":47,"props":1787,"children":1788},{},[1789],{"type":44,"value":1790},"When queue time is ambiguous (merge happened mid-build), use commit ancestry:",{"type":39,"tag":199,"props":1792,"children":1794},{"className":366,"code":1793,"language":368,"meta":207,"style":207},"gh api repos\u002F{owner}\u002F{repo}\u002Fcompare\u002F{merge_commit}...{build_source_version} --jq '.status'\n# \"ahead\" or \"identical\" = fix is included; \"behind\" = fix is NOT included\n",[1795],{"type":39,"tag":132,"props":1796,"children":1797},{"__ignoreMap":207},[1798,1834],{"type":39,"tag":374,"props":1799,"children":1800},{"class":376,"line":377},[1801,1806,1811,1816,1821,1825,1830],{"type":39,"tag":374,"props":1802,"children":1803},{"style":381},[1804],{"type":44,"value":1805},"gh",{"type":39,"tag":374,"props":1807,"children":1808},{"style":386},[1809],{"type":44,"value":1810}," api",{"type":39,"tag":374,"props":1812,"children":1813},{"style":386},[1814],{"type":44,"value":1815}," repos\u002F{owner}\u002F{repo}\u002Fcompare\u002F{merge_commit}...{build_source_version}",{"type":39,"tag":374,"props":1817,"children":1818},{"style":386},[1819],{"type":44,"value":1820}," --jq",{"type":39,"tag":374,"props":1822,"children":1823},{"style":392},[1824],{"type":44,"value":580},{"type":39,"tag":374,"props":1826,"children":1827},{"style":386},[1828],{"type":44,"value":1829},".status",{"type":39,"tag":374,"props":1831,"children":1832},{"style":392},[1833],{"type":44,"value":647},{"type":39,"tag":374,"props":1835,"children":1836},{"class":376,"line":458},[1837],{"type":39,"tag":374,"props":1838,"children":1839},{"style":452},[1840],{"type":44,"value":1841},"# \"ahead\" or \"identical\" = fix is included; \"behind\" = fix is NOT included\n",{"type":39,"tag":53,"props":1843,"children":1845},{"id":1844},"bulk-failure-aggregation",[1846],{"type":44,"value":1847},"Bulk Failure Aggregation",{"type":39,"tag":47,"props":1849,"children":1850},{},[1851],{"type":44,"value":1852},"When the user asks \"what are the top N failures\" across a pipeline or time range, use a different approach than single-build investigation.",{"type":39,"tag":160,"props":1854,"children":1856},{"id":1855},"approach",[1857],{"type":44,"value":1858},"Approach",{"type":39,"tag":172,"props":1860,"children":1861},{},[1862,1867,1872,1877],{"type":39,"tag":64,"props":1863,"children":1864},{},[1865],{"type":44,"value":1866},"Query AzDO for all failed builds in the time range (e.g., last 2 days)",{"type":39,"tag":64,"props":1868,"children":1869},{},[1870],{"type":44,"value":1871},"For each failed build, query the timeline to find which jobs\u002Flegs failed",{"type":39,"tag":64,"props":1873,"children":1874},{},[1875],{"type":44,"value":1876},"For Helix-based legs, query work item results to get test names, exit codes, machines",{"type":39,"tag":64,"props":1878,"children":1879},{},[1880],{"type":44,"value":1881},"Aggregate into a SQL table for analysis:",{"type":39,"tag":199,"props":1883,"children":1885},{"className":1217,"code":1884,"language":1219,"meta":207,"style":207},"CREATE TABLE failure_survey (\n  build_id INT, leg TEXT, work_item TEXT, exit_code INT,\n  machine TEXT, failure_category TEXT, helix_job TEXT,\n  is_pr_specific BOOLEAN, notes TEXT\n);\n",[1886],{"type":39,"tag":132,"props":1887,"children":1888},{"__ignoreMap":207},[1889,1897,1905,1913,1921],{"type":39,"tag":374,"props":1890,"children":1891},{"class":376,"line":377},[1892],{"type":39,"tag":374,"props":1893,"children":1894},{},[1895],{"type":44,"value":1896},"CREATE TABLE failure_survey (\n",{"type":39,"tag":374,"props":1898,"children":1899},{"class":376,"line":458},[1900],{"type":39,"tag":374,"props":1901,"children":1902},{},[1903],{"type":44,"value":1904},"  build_id INT, leg TEXT, work_item TEXT, exit_code INT,\n",{"type":39,"tag":374,"props":1906,"children":1907},{"class":376,"line":512},[1908],{"type":39,"tag":374,"props":1909,"children":1910},{},[1911],{"type":44,"value":1912},"  machine TEXT, failure_category TEXT, helix_job TEXT,\n",{"type":39,"tag":374,"props":1914,"children":1915},{"class":376,"line":521},[1916],{"type":39,"tag":374,"props":1917,"children":1918},{},[1919],{"type":44,"value":1920},"  is_pr_specific BOOLEAN, notes TEXT\n",{"type":39,"tag":374,"props":1922,"children":1923},{"class":376,"line":556},[1924],{"type":39,"tag":374,"props":1925,"children":1926},{},[1927],{"type":44,"value":1263},{"type":39,"tag":172,"props":1929,"children":1930},{"start":556},[1931,1936],{"type":39,"tag":64,"props":1932,"children":1933},{},[1934],{"type":44,"value":1935},"Group by work item to find the most frequent failures",{"type":39,"tag":64,"props":1937,"children":1938},{},[1939],{"type":44,"value":1940},"Separate PR-specific build errors (hit all legs at once) from recurring test failures",{"type":39,"tag":160,"props":1942,"children":1944},{"id":1943},"classification",[1945],{"type":44,"value":1946},"Classification",{"type":39,"tag":765,"props":1948,"children":1949},{},[1950,1970],{"type":39,"tag":769,"props":1951,"children":1952},{},[1953],{"type":39,"tag":773,"props":1954,"children":1955},{},[1956,1961,1965],{"type":39,"tag":777,"props":1957,"children":1958},{},[1959],{"type":44,"value":1960},"Type",{"type":39,"tag":777,"props":1962,"children":1963},{},[1964],{"type":44,"value":1293},{"type":39,"tag":777,"props":1966,"children":1967},{},[1968],{"type":44,"value":1969},"Priority",{"type":39,"tag":788,"props":1971,"children":1972},{},[1973,1994,2015,2035,2055],{"type":39,"tag":773,"props":1974,"children":1975},{},[1976,1984,1989],{"type":39,"tag":795,"props":1977,"children":1978},{},[1979],{"type":39,"tag":248,"props":1980,"children":1981},{},[1982],{"type":44,"value":1983},"Recurring test failure",{"type":39,"tag":795,"props":1985,"children":1986},{},[1987],{"type":44,"value":1988},"Same work item fails across many PRs",{"type":39,"tag":795,"props":1990,"children":1991},{},[1992],{"type":44,"value":1993},"High — investigate root cause",{"type":39,"tag":773,"props":1995,"children":1996},{},[1997,2005,2010],{"type":39,"tag":795,"props":1998,"children":1999},{},[2000],{"type":39,"tag":248,"props":2001,"children":2002},{},[2003],{"type":44,"value":2004},"PR-specific build error",{"type":39,"tag":795,"props":2006,"children":2007},{},[2008],{"type":44,"value":2009},"Multiple legs fail on same build only",{"type":39,"tag":795,"props":2011,"children":2012},{},[2013],{"type":44,"value":2014},"Low — PR author's problem",{"type":39,"tag":773,"props":2016,"children":2017},{},[2018,2025,2030],{"type":39,"tag":795,"props":2019,"children":2020},{},[2021],{"type":39,"tag":248,"props":2022,"children":2023},{},[2024],{"type":44,"value":1600},{"type":39,"tag":795,"props":2026,"children":2027},{},[2028],{"type":44,"value":2029},"ALL suites fail on one build, exit 133, 1x per platform",{"type":39,"tag":795,"props":2031,"children":2032},{},[2033],{"type":44,"value":2034},"Low — filter out; single bad commit crashes runtime before tests run",{"type":39,"tag":773,"props":2036,"children":2037},{},[2038,2045,2050],{"type":39,"tag":795,"props":2039,"children":2040},{},[2041],{"type":39,"tag":248,"props":2042,"children":2043},{},[2044],{"type":44,"value":832},{"type":39,"tag":795,"props":2046,"children":2047},{},[2048],{"type":44,"value":2049},"exit 71, exit 81, DEVICE_NOT_FOUND",{"type":39,"tag":795,"props":2051,"children":2052},{},[2053],{"type":44,"value":2054},"Medium — systemic infra issue",{"type":39,"tag":773,"props":2056,"children":2057},{},[2058,2066,2071],{"type":39,"tag":795,"props":2059,"children":2060},{},[2061],{"type":39,"tag":248,"props":2062,"children":2063},{},[2064],{"type":44,"value":2065},"Sporadic infra",{"type":39,"tag":795,"props":2067,"children":2068},{},[2069],{"type":44,"value":2070},"Single occurrence, no pattern",{"type":39,"tag":795,"props":2072,"children":2073},{},[2074],{"type":44,"value":2075},"Ignore — noise",{"type":39,"tag":160,"props":2077,"children":2079},{"id":2078},"rolling-builds-vs-pr-builds",[2080],{"type":44,"value":2081},"Rolling Builds vs PR Builds",{"type":39,"tag":47,"props":2083,"children":2084},{},[2085,2087,2092,2094,2100],{"type":44,"value":2086},"When assessing ",{"type":39,"tag":248,"props":2088,"children":2089},{},[2090],{"type":44,"value":2091},"main branch health",{"type":44,"value":2093},", filter to rolling (scheduled) builds using ",{"type":39,"tag":132,"props":2095,"children":2097},{"className":2096},[],[2098],{"type":44,"value":2099},"reasonFilter=schedule",{"type":44,"value":2101},". PR builds include broken PR branches that dominate failure counts and obscure systemic issues.",{"type":39,"tag":47,"props":2103,"children":2104},{},[2105,2107,2112,2114,2119],{"type":44,"value":2106},"Real-world example: over the same 2-day window, PR builds produced ",{"type":39,"tag":248,"props":2108,"children":2109},{},[2110],{"type":44,"value":2111},"423 work item failures",{"type":44,"value":2113}," (dominated by a single poison PR with 119 failures). Rolling builds had only ",{"type":39,"tag":248,"props":2115,"children":2116},{},[2117],{"type":44,"value":2118},"46 failures",{"type":44,"value":2120}," — all genuine systemic issues.",{"type":39,"tag":199,"props":2122,"children":2124},{"className":366,"code":2123,"language":368,"meta":207,"style":207},"# Rolling builds only (scheduled, typically 2x daily on main)\ncurl -s \"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds?definitions={defId}&reasonFilter=schedule&resultFilter=failed&minTime={iso8601}&$top=50&api-version=7.1\"\n",[2125],{"type":39,"tag":132,"props":2126,"children":2127},{"__ignoreMap":207},[2128,2136],{"type":39,"tag":374,"props":2129,"children":2130},{"class":376,"line":377},[2131],{"type":39,"tag":374,"props":2132,"children":2133},{"style":452},[2134],{"type":44,"value":2135},"# Rolling builds only (scheduled, typically 2x daily on main)\n",{"type":39,"tag":374,"props":2137,"children":2138},{"class":376,"line":458},[2139,2143,2147,2151,2156,2162,2167],{"type":39,"tag":374,"props":2140,"children":2141},{"style":381},[2142],{"type":44,"value":137},{"type":39,"tag":374,"props":2144,"children":2145},{"style":386},[2146],{"type":44,"value":389},{"type":39,"tag":374,"props":2148,"children":2149},{"style":392},[2150],{"type":44,"value":395},{"type":39,"tag":374,"props":2152,"children":2153},{"style":386},[2154],{"type":44,"value":2155},"https:\u002F\u002Fdev.azure.com\u002F{org}\u002F{project}\u002F_apis\u002Fbuild\u002Fbuilds?definitions={defId}&reasonFilter=schedule&resultFilter=failed&minTime={iso8601}&",{"type":39,"tag":374,"props":2157,"children":2159},{"style":2158},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[2160],{"type":44,"value":2161},"$top",{"type":39,"tag":374,"props":2163,"children":2164},{"style":386},[2165],{"type":44,"value":2166},"=50&api-version=7.1",{"type":39,"tag":374,"props":2168,"children":2169},{"style":392},[2170],{"type":44,"value":509},{"type":39,"tag":47,"props":2172,"children":2173},{},[2174],{"type":44,"value":2175},"Use rolling builds to:",{"type":39,"tag":60,"props":2177,"children":2178},{},[2179,2184,2189],{"type":39,"tag":64,"props":2180,"children":2181},{},[2182],{"type":44,"value":2183},"Assess current main branch health",{"type":39,"tag":64,"props":2185,"children":2186},{},[2187],{"type":44,"value":2188},"Identify systemic test failures vs PR-introduced regressions",{"type":39,"tag":64,"props":2190,"children":2191},{},[2192],{"type":44,"value":2193},"Verify fix impact (compare rolling failure counts before\u002Fafter merge)",{"type":39,"tag":160,"props":2195,"children":2197},{"id":2196},"cross-leg-view",[2198],{"type":44,"value":2199},"Cross-Leg View",{"type":39,"tag":47,"props":2201,"children":2202},{},[2203],{"type":44,"value":2204},"Some tests fail on multiple platforms. When reporting top-N, group by test name but break out by leg to show which platforms are affected:",{"type":39,"tag":199,"props":2206,"children":2209},{"className":2207,"code":2208,"language":44},[202],"System.Runtime.Tests: 61 total\n  → ios-arm64 Mono (13), tvos-arm64 Mono (20), android (9), ...\n",[2210],{"type":39,"tag":132,"props":2211,"children":2212},{"__ignoreMap":207},[2213],{"type":44,"value":2208},{"type":39,"tag":47,"props":2215,"children":2216},{},[2217],{"type":44,"value":2218},"This reveals whether a failure is platform-specific or cross-platform.",{"type":39,"tag":53,"props":2220,"children":2222},{"id":2221},"machine-distribution-analysis",[2223],{"type":44,"value":2224},"Machine Distribution Analysis",{"type":39,"tag":47,"props":2226,"children":2227},{},[2228,2230,2235,2236,2241],{"type":44,"value":2229},"After identifying a recurring failure, determine whether it's ",{"type":39,"tag":248,"props":2231,"children":2232},{},[2233],{"type":44,"value":2234},"machine-specific",{"type":44,"value":1142},{"type":39,"tag":248,"props":2237,"children":2238},{},[2239],{"type":44,"value":2240},"systemic",{"type":44,"value":2242},":",{"type":39,"tag":172,"props":2244,"children":2245},{},[2246,2251,2256],{"type":39,"tag":64,"props":2247,"children":2248},{},[2249],{"type":44,"value":2250},"Collect the machine name from each failing work item",{"type":39,"tag":64,"props":2252,"children":2253},{},[2254],{"type":44,"value":2255},"Count failures per machine",{"type":39,"tag":64,"props":2257,"children":2258},{},[2259],{"type":44,"value":2260},"Compare to the pool size",{"type":39,"tag":765,"props":2262,"children":2263},{},[2264,2284],{"type":39,"tag":769,"props":2265,"children":2266},{},[2267],{"type":39,"tag":773,"props":2268,"children":2269},{},[2270,2275,2280],{"type":39,"tag":777,"props":2271,"children":2272},{},[2273],{"type":44,"value":2274},"Distribution",{"type":39,"tag":777,"props":2276,"children":2277},{},[2278],{"type":44,"value":2279},"Diagnosis",{"type":39,"tag":777,"props":2281,"children":2282},{},[2283],{"type":44,"value":1298},{"type":39,"tag":788,"props":2285,"children":2286},{},[2287,2309,2332],{"type":39,"tag":773,"props":2288,"children":2289},{},[2290,2295,2304],{"type":39,"tag":795,"props":2291,"children":2292},{},[2293],{"type":44,"value":2294},"1-2 machines account for most failures",{"type":39,"tag":795,"props":2296,"children":2297},{},[2298,2302],{"type":39,"tag":248,"props":2299,"children":2300},{},[2301],{"type":44,"value":1339},{"type":44,"value":2303}," — hardware\u002Fconfig issue",{"type":39,"tag":795,"props":2305,"children":2306},{},[2307],{"type":44,"value":2308},"Report machines, suggest pool removal",{"type":39,"tag":773,"props":2310,"children":2311},{},[2312,2317,2327],{"type":39,"tag":795,"props":2313,"children":2314},{},[2315],{"type":44,"value":2316},"Failures spread evenly across many machines",{"type":39,"tag":795,"props":2318,"children":2319},{},[2320,2325],{"type":39,"tag":248,"props":2321,"children":2322},{},[2323],{"type":44,"value":2324},"Systemic",{"type":44,"value":2326}," — software\u002Finfra issue",{"type":39,"tag":795,"props":2328,"children":2329},{},[2330],{"type":44,"value":2331},"File infrastructure bug",{"type":39,"tag":773,"props":2333,"children":2334},{},[2335,2340,2350],{"type":39,"tag":795,"props":2336,"children":2337},{},[2338],{"type":44,"value":2339},"Slight concentration on a few + spread",{"type":39,"tag":795,"props":2341,"children":2342},{},[2343,2348],{"type":39,"tag":248,"props":2344,"children":2345},{},[2346],{"type":44,"value":2347},"Mixed",{"type":44,"value":2349}," — systemic with some worse machines",{"type":39,"tag":795,"props":2351,"children":2352},{},[2353],{"type":44,"value":2354},"File bug + flag worst offenders",{"type":39,"tag":47,"props":2356,"children":2357},{},[2358],{"type":44,"value":2359},"Example from real investigation: tvOS failures concentrated on DNCENGTVOS-036 and DNCENGTVOS-022 = machine-specific. Android DEVICE_NOT_FOUND spread across 17 machines = systemic emulator issue.",{"type":39,"tag":53,"props":2361,"children":2363},{"id":2362},"filing-issues-from-investigation",[2364],{"type":44,"value":2365},"Filing Issues from Investigation",{"type":39,"tag":47,"props":2367,"children":2368},{},[2369],{"type":44,"value":2370},"When investigation reveals a clear root cause, file issues on the appropriate repo:",{"type":39,"tag":765,"props":2372,"children":2373},{},[2374,2395],{"type":39,"tag":769,"props":2375,"children":2376},{},[2377],{"type":39,"tag":773,"props":2378,"children":2379},{},[2380,2385,2390],{"type":39,"tag":777,"props":2381,"children":2382},{},[2383],{"type":44,"value":2384},"Root Cause",{"type":39,"tag":777,"props":2386,"children":2387},{},[2388],{"type":44,"value":2389},"File On",{"type":39,"tag":777,"props":2391,"children":2392},{},[2393],{"type":44,"value":2394},"Include",{"type":39,"tag":788,"props":2396,"children":2397},{},[2398,2423,2448,2471],{"type":39,"tag":773,"props":2399,"children":2400},{},[2401,2406,2418],{"type":39,"tag":795,"props":2402,"children":2403},{},[2404],{"type":44,"value":2405},"XHarness bug (timeout handling, exit code detection, device recovery)",{"type":39,"tag":795,"props":2407,"children":2408},{},[2409],{"type":39,"tag":2410,"props":2411,"children":2415},"a",{"href":2412,"rel":2413},"https:\u002F\u002Fgithub.com\u002Fdotnet\u002Fxharness",[2414],"nofollow",[2416],{"type":44,"value":2417},"dotnet\u002Fxharness",{"type":39,"tag":795,"props":2419,"children":2420},{},[2421],{"type":44,"value":2422},"Pass\u002Ffail comparison evidence, console log URLs, affected machines",{"type":39,"tag":773,"props":2424,"children":2425},{},[2426,2431,2443],{"type":39,"tag":795,"props":2427,"children":2428},{},[2429],{"type":44,"value":2430},"Helix infrastructure (retry behavior, machine management)",{"type":39,"tag":795,"props":2432,"children":2433},{},[2434,2441],{"type":39,"tag":2410,"props":2435,"children":2438},{"href":2436,"rel":2437},"https:\u002F\u002Fgithub.com\u002Fdotnet\u002Farcade",[2414],[2439],{"type":44,"value":2440},"dotnet\u002Farcade",{"type":44,"value":2442}," or dnceng issue tracker",{"type":39,"tag":795,"props":2444,"children":2445},{},[2446],{"type":44,"value":2447},"Failure counts, machine distribution, retry logs",{"type":39,"tag":773,"props":2449,"children":2450},{},[2451,2456,2466],{"type":39,"tag":795,"props":2452,"children":2453},{},[2454],{"type":44,"value":2455},"Test code bug",{"type":39,"tag":795,"props":2457,"children":2458},{},[2459],{"type":39,"tag":2410,"props":2460,"children":2463},{"href":2461,"rel":2462},"https:\u002F\u002Fgithub.com\u002Fdotnet\u002Fruntime",[2414],[2464],{"type":44,"value":2465},"dotnet\u002Fruntime",{"type":39,"tag":795,"props":2467,"children":2468},{},[2469],{"type":44,"value":2470},"Failing test name, stack trace, repro steps",{"type":39,"tag":773,"props":2472,"children":2473},{},[2474,2479,2484],{"type":39,"tag":795,"props":2475,"children":2476},{},[2477],{"type":44,"value":2478},"Machine hardware",{"type":39,"tag":795,"props":2480,"children":2481},{},[2482],{"type":44,"value":2483},"dnceng infrastructure team",{"type":39,"tag":795,"props":2485,"children":2486},{},[2487],{"type":44,"value":2488},"Machine names, failure frequency, environment details",{"type":39,"tag":47,"props":2490,"children":2491},{},[2492],{"type":44,"value":2493},"Always include:",{"type":39,"tag":60,"props":2495,"children":2496},{},[2497,2507,2517,2527],{"type":39,"tag":64,"props":2498,"children":2499},{},[2500,2505],{"type":39,"tag":248,"props":2501,"children":2502},{},[2503],{"type":44,"value":2504},"Evidence",{"type":44,"value":2506},": Helix console log URLs and AzDO build URLs",{"type":39,"tag":64,"props":2508,"children":2509},{},[2510,2515],{"type":39,"tag":248,"props":2511,"children":2512},{},[2513],{"type":44,"value":2514},"Failure count",{"type":44,"value":2516},": How many times this occurred in the analysis window",{"type":39,"tag":64,"props":2518,"children":2519},{},[2520,2525],{"type":39,"tag":248,"props":2521,"children":2522},{},[2523],{"type":44,"value":2524},"Machine list",{"type":44,"value":2526},": Which machines are affected",{"type":39,"tag":64,"props":2528,"children":2529},{},[2530,2535],{"type":39,"tag":248,"props":2531,"children":2532},{},[2533],{"type":44,"value":2534},"Pass\u002Ffail comparison",{"type":44,"value":2536},": If intermittent, what differs between passing and failing runs",{"type":39,"tag":53,"props":2538,"children":2540},{"id":2539},"stop-signals",[2541],{"type":44,"value":2542},"Stop Signals",{"type":39,"tag":60,"props":2544,"children":2545},{},[2546,2556,2566],{"type":39,"tag":64,"props":2547,"children":2548},{},[2549,2554],{"type":39,"tag":248,"props":2550,"children":2551},{},[2552],{"type":44,"value":2553},"Stop downloading logs",{"type":44,"value":2555}," after you have 1 clear failing run + 1 clear passing run for comparison. Don't download all 50 builds.",{"type":39,"tag":64,"props":2557,"children":2558},{},[2559,2564],{"type":39,"tag":248,"props":2560,"children":2561},{},[2562],{"type":44,"value":2563},"Stop analyzing",{"type":44,"value":2565}," when you can explain why the test fails on machine A but passes on machine B. That's the root cause.",{"type":39,"tag":64,"props":2567,"children":2568},{},[2569,2574],{"type":39,"tag":248,"props":2570,"children":2571},{},[2572],{"type":44,"value":2573},"Stop if no passing runs exist",{"type":44,"value":2575}," — the failure is 100% reproducible, not intermittent. Shift to analyzing the test code itself.",{"type":39,"tag":53,"props":2577,"children":2579},{"id":2578},"anti-patterns",[2580],{"type":44,"value":2581},"Anti-Patterns",{"type":39,"tag":211,"props":2583,"children":2584},{},[2585],{"type":39,"tag":47,"props":2586,"children":2587},{},[2588,2590,2595],{"type":44,"value":2589},"🚨 ",{"type":39,"tag":248,"props":2591,"children":2592},{},[2593],{"type":44,"value":2594},"Don't rely solely on Helix MCP search tools for log analysis.",{"type":44,"value":2596}," Download the console log with curl and use local grep — it's faster and more reliable for pattern matching.",{"type":39,"tag":211,"props":2598,"children":2599},{},[2600],{"type":39,"tag":47,"props":2601,"children":2602},{},[2603,2604,2609],{"type":44,"value":2589},{"type":39,"tag":248,"props":2605,"children":2606},{},[2607],{"type":44,"value":2608},"Don't assume \"flaky\" without evidence.",{"type":44,"value":2610}," If the same machine fails every time, it's machine-specific, not flaky. Capture the machine name.",{"type":39,"tag":211,"props":2612,"children":2613},{},[2614],{"type":39,"tag":47,"props":2615,"children":2616},{},[2617,2618,2623],{"type":44,"value":2589},{"type":39,"tag":248,"props":2619,"children":2620},{},[2621],{"type":44,"value":2622},"Don't ignore the \"Exit code detection\" warning as the root cause.",{"type":44,"value":2624}," On tvOS\u002FiOS 15+, XHarness logs this warning but it may be a red herring — the real issue can be device log streaming, not exit code detection itself.",{"type":39,"tag":211,"props":2626,"children":2627},{},[2628],{"type":39,"tag":47,"props":2629,"children":2630},{},[2631,2632,2637],{"type":44,"value":2589},{"type":39,"tag":248,"props":2633,"children":2634},{},[2635],{"type":44,"value":2636},"Don't analyze all builds in a pipeline.",{"type":44,"value":2638}," Filter to the specific job\u002Fleg first, then look at 3-5 recent runs maximum.",{"type":39,"tag":53,"props":2640,"children":2642},{"id":2641},"reference-known-helix-queue-patterns",[2643],{"type":44,"value":2644},"Reference: Known Helix Queue Patterns",{"type":39,"tag":765,"props":2646,"children":2647},{},[2648,2664],{"type":39,"tag":769,"props":2649,"children":2650},{},[2651],{"type":39,"tag":773,"props":2652,"children":2653},{},[2654,2659],{"type":39,"tag":777,"props":2655,"children":2656},{},[2657],{"type":44,"value":2658},"Queue Pattern",{"type":39,"tag":777,"props":2660,"children":2661},{},[2662],{"type":44,"value":2663},"Platform",{"type":39,"tag":788,"props":2665,"children":2666},{},[2667,2684,2701,2718,2735,2752],{"type":39,"tag":773,"props":2668,"children":2669},{},[2670,2679],{"type":39,"tag":795,"props":2671,"children":2672},{},[2673],{"type":39,"tag":132,"props":2674,"children":2676},{"className":2675},[],[2677],{"type":44,"value":2678},"osx.*.amd64.appletv.open",{"type":39,"tag":795,"props":2680,"children":2681},{},[2682],{"type":44,"value":2683},"Apple TV devices (tvOS)",{"type":39,"tag":773,"props":2685,"children":2686},{},[2687,2696],{"type":39,"tag":795,"props":2688,"children":2689},{},[2690],{"type":39,"tag":132,"props":2691,"children":2693},{"className":2692},[],[2694],{"type":44,"value":2695},"osx.*.amd64.iphone.open",{"type":39,"tag":795,"props":2697,"children":2698},{},[2699],{"type":44,"value":2700},"iPhone devices (iOS)",{"type":39,"tag":773,"props":2702,"children":2703},{},[2704,2713],{"type":39,"tag":795,"props":2705,"children":2706},{},[2707],{"type":39,"tag":132,"props":2708,"children":2710},{"className":2709},[],[2711],{"type":44,"value":2712},"ubuntu.*.amd64.android.*.open",{"type":39,"tag":795,"props":2714,"children":2715},{},[2716],{"type":44,"value":2717},"Android emulators",{"type":39,"tag":773,"props":2719,"children":2720},{},[2721,2730],{"type":39,"tag":795,"props":2722,"children":2723},{},[2724],{"type":39,"tag":132,"props":2725,"children":2727},{"className":2726},[],[2728],{"type":44,"value":2729},"ubuntu.*",{"type":39,"tag":795,"props":2731,"children":2732},{},[2733],{"type":44,"value":2734},"Linux",{"type":39,"tag":773,"props":2736,"children":2737},{},[2738,2747],{"type":39,"tag":795,"props":2739,"children":2740},{},[2741],{"type":39,"tag":132,"props":2742,"children":2744},{"className":2743},[],[2745],{"type":44,"value":2746},"windows.*",{"type":39,"tag":795,"props":2748,"children":2749},{},[2750],{"type":44,"value":2751},"Windows",{"type":39,"tag":773,"props":2753,"children":2754},{},[2755,2764],{"type":39,"tag":795,"props":2756,"children":2757},{},[2758],{"type":39,"tag":132,"props":2759,"children":2761},{"className":2760},[],[2762],{"type":44,"value":2763},"browser-wasm.*",{"type":39,"tag":795,"props":2765,"children":2766},{},[2767],{"type":44,"value":2768},"WASM (Chrome\u002FFirefox)",{"type":39,"tag":47,"props":2770,"children":2771},{},[2772,2774,2780],{"type":44,"value":2773},"Queue name is in the first line of every Helix console log: ",{"type":39,"tag":132,"props":2775,"children":2777},{"className":2776},[],[2778],{"type":44,"value":2779},"workitem ... ({queue}) executed on machine {machine}",{"type":44,"value":2781},".",{"type":39,"tag":53,"props":2783,"children":2785},{"id":2784},"reference-platform-specific-resources",[2786],{"type":44,"value":2787},"Reference: Platform-Specific Resources",{"type":39,"tag":60,"props":2789,"children":2790},{},[2791,2801,2811],{"type":39,"tag":64,"props":2792,"children":2793},{},[2794,2799],{"type":39,"tag":2410,"props":2795,"children":2797},{"href":2796},"references\u002Fxharness-patterns.md",[2798],{"type":44,"value":2796},{"type":44,"value":2800}," — tvOS\u002FiOS XHarness exit code detection, device log streaming, timeout chain",{"type":39,"tag":64,"props":2802,"children":2803},{},[2804,2809],{"type":39,"tag":2410,"props":2805,"children":2807},{"href":2806},"references\u002Fandroid-patterns.md",[2808],{"type":44,"value":2806},{"type":44,"value":2810}," — Android emulator DEVICE_NOT_FOUND, retry\u002Freboot flow, common test failures",{"type":39,"tag":64,"props":2812,"children":2813},{},[2814,2819],{"type":39,"tag":2410,"props":2815,"children":2817},{"href":2816},"references\u002Fwasm-patterns.md",[2818],{"type":44,"value":2816},{"type":44,"value":2820}," — WASM\u002Fbrowser SDK build errors, Playwright timeouts, work item naming",{"type":39,"tag":2822,"props":2823,"children":2824},"style",{},[2825],{"type":44,"value":2826},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2828,"total":22},[2829,2848,2861,2876,2887,2902,2912],{"slug":2830,"name":2830,"fn":2831,"description":2832,"org":2833,"tags":2834,"stars":22,"repoUrl":23,"updatedAt":2847},"binskim-analysis","investigate BinSkim SDL security findings","Investigate BinSkim SDL findings from official pipelines — understand Guardian filtering, compare raw vs merged SARIF, decode portal results, and determine fix ownership. Use when asked about SDL scan results, portal findings, Guardian filtering, rule meanings, or discrepancies between local and official results. Also use when asked \"why does the portal show X\", \"what's filtered\", \"explain Guardian\", \"investigate SDL findings\", \"portal BA2008\", \"binskim failures in pipeline\", or \"what rules are required\". DO NOT USE FOR: running BinSkim locally (use binskim-scan), source code analysis (use CodeQL), or credential scanning (use CredScan).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2835,2838,2841,2844],{"name":2836,"slug":2837,"type":15},".NET","net",{"name":2839,"slug":2840,"type":15},"Audit","audit",{"name":2842,"slug":2843,"type":15},"Code Analysis","code-analysis",{"name":2845,"slug":2846,"type":15},"Security","security","2026-07-12T08:23:11.212231",{"slug":2849,"name":2849,"fn":2850,"description":2851,"org":2852,"tags":2853,"stars":22,"repoUrl":23,"updatedAt":2860},"binskim-scan","run BinSkim binary security analysis","Run BinSkim binary security analysis locally against a dotnet repository. Use when asked to scan binaries, check BinSkim compliance, verify a fix for a rule violation, or run a local SDL scan. Also use when asked \"run binskim\", \"binary security scan\", \"scan binaries\", \"check binskim\", \"verify my fix\", \"repro BA2008 locally\", or \"verify BA2008 fix\". DO NOT USE FOR: investigating official pipeline results or portal findings (use binskim-analysis), source code analysis (use CodeQL), credential scanning (use CredScan), or general build\u002Ftest failures (use ci-analysis).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2854,2855,2858,2859],{"name":2836,"slug":2837,"type":15},{"name":2856,"slug":2857,"type":15},"CLI","cli",{"name":2842,"slug":2843,"type":15},{"name":2845,"slug":2846,"type":15},"2026-07-12T08:23:09.911458",{"slug":2862,"name":2862,"fn":2863,"description":2864,"org":2865,"tags":2866,"stars":22,"repoUrl":23,"updatedAt":2875},"ci-analysis","analyze .NET CI build and test status","Analyze CI build and test status from Azure DevOps and Helix for dotnet repository PRs. Use when checking CI status, investigating failures, determining if a PR is ready to merge, or given URLs containing dev.azure.com or helix.dot.net. Also use when asked \"why is CI red\", \"test failures\", \"retry CI\", \"rerun tests\", \"is CI green\", \"build failed\", \"checks failing\", or \"flaky tests\". DO NOT USE FOR: investigating stale codeflow PRs or dependency update health, tracing whether a commit has flowed from one repo to another, reviewing code changes for correctness or style.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2867,2868,2871,2874],{"name":2836,"slug":2837,"type":15},{"name":2869,"slug":2870,"type":15},"Azure","azure",{"name":2872,"slug":2873,"type":15},"CI\u002FCD","ci-cd",{"name":20,"slug":21,"type":15},"2026-07-12T08:21:49.360882",{"slug":2877,"name":2877,"fn":2878,"description":2879,"org":2880,"tags":2881,"stars":22,"repoUrl":23,"updatedAt":2886},"ci-crash-dump","debug CI crash dumps","Download and debug crash dumps from CI test failures in dotnet repositories. Use when a CI test crashed (not just failed), when the user wants to debug a crash dump from a PR or build, or when asked \"debug dump\", \"download dump\", \"crash dump from CI\", \"test crashed\", \"analyze crash in PR\", or \"why did the test crash\". DO NOT USE FOR: test failures that are not crashes (use ci-analysis), build failures, performance analysis, or analyzing dumps you already have locally.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2882,2883,2884,2885],{"name":2836,"slug":2837,"type":15},{"name":2872,"slug":2873,"type":15},{"name":20,"slug":21,"type":15},{"name":17,"slug":18,"type":15},"2026-07-12T08:21:56.945844",{"slug":2888,"name":2888,"fn":2889,"description":2890,"org":2891,"tags":2892,"stars":22,"repoUrl":23,"updatedAt":2901},"flow-analysis","analyze VMR codeflow health","Analyze VMR codeflow health using maestro MCP tools and GitHub MCP tools. USE FOR: investigating stale codeflow PRs, checking if fixes have flowed through the VMR pipeline, debugging dependency update issues, checking overall flow status for a repo, diagnosing why backflow PRs are missing or blocked, subscription health, build freshness, URLs containing dotnet-maestro or \"Source code updates from dotnet\u002Fdotnet\". DO NOT USE FOR: CI build failures (use ci-analysis skill), code review (use code-review skill), general PR investigation without codeflow context, tracing whether a specific commit\u002FPR has reached another repo (use flow-tracing skill). INVOKES: maestro and GitHub MCP tools, flow-health.cs script.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2893,2894,2895,2898],{"name":2872,"slug":2873,"type":15},{"name":13,"slug":14,"type":15},{"name":2896,"slug":2897,"type":15},"GitHub","github",{"name":2899,"slug":2900,"type":15},"MCP","mcp","2026-07-12T08:21:53.151866",{"slug":2903,"name":2903,"fn":2904,"description":2905,"org":2906,"tags":2907,"stars":22,"repoUrl":23,"updatedAt":2911},"flow-tracing","trace .NET dependency flow in VMR","Trace dependency flow across .NET repos through the VMR pipeline. USE FOR: checking if a PR\u002Fcommit from repo A has reached repo B, finding what runtime SHA is in an SDK build, tracing dependency versions through the VMR, checking if a commit is included in an SDK build, decoding SDK version strings, \"has my fix reached runtime\", \"did roslyn#80873 flow to runtime\", \"what SHA is in SDK version X\", cross-repo dependency tracing, mapping SDK versions to VMR commits. DO NOT USE FOR: codeflow PR health or staleness (use flow-analysis skill), CI build failures (use ci-analysis skill). INVOKES: maestro and GitHub MCP tools, Get-SdkVersionTrace.ps1 script.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2908],{"name":2909,"slug":2910,"type":15},"Data Pipeline","data-pipeline","2026-07-12T08:21:55.689056",{"slug":4,"name":4,"fn":5,"description":6,"org":2913,"tags":2914,"stars":22,"repoUrl":23,"updatedAt":24},{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2915,2916,2917],{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},{"name":17,"slug":18,"type":15},{"items":2919,"total":3078},[2920,2934,2947,2962,2980,2992,3012,3022,3034,3044,3057,3068],{"slug":2921,"name":2921,"fn":2922,"description":2923,"org":2924,"tags":2925,"stars":2931,"repoUrl":2932,"updatedAt":2933},"multithreaded-task-migration","migrate MSBuild tasks to multithreaded mode","Guide for migrating MSBuild tasks to multithreaded mode support, including compatibility red-team review. Use this when converting tasks to thread-safe versions, implementing IMultiThreadableTask, adding TaskEnvironment support, or auditing migrations for behavioral compatibility.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2926,2927,2928],{"name":2836,"slug":2837,"type":15},{"name":13,"slug":14,"type":15},{"name":2929,"slug":2930,"type":15},"Performance","performance",5535,"https:\u002F\u002Fgithub.com\u002Fdotnet\u002Fmsbuild","2026-07-22T05:37:33.965588",{"slug":2935,"name":2935,"fn":2936,"description":2937,"org":2938,"tags":2939,"stars":2944,"repoUrl":2945,"updatedAt":2946},"analyzing-dotnet-performance","analyze .NET code for performance anti-patterns","Scans .NET code for ~50 performance anti-patterns across async, memory, strings, collections, LINQ, regex, serialization, and I\u002FO with tiered severity classification. Use when analyzing .NET code for optimization opportunities, reviewing hot paths, or auditing allocation-heavy patterns.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2940,2941,2942,2943],{"name":2836,"slug":2837,"type":15},{"name":2842,"slug":2843,"type":15},{"name":20,"slug":21,"type":15},{"name":2929,"slug":2930,"type":15},4576,"https:\u002F\u002Fgithub.com\u002Fdotnet\u002Fskills","2026-07-12T08:23:25.400375",{"slug":2948,"name":2948,"fn":2949,"description":2950,"org":2951,"tags":2952,"stars":2944,"repoUrl":2945,"updatedAt":2961},"android-tombstone-symbolication","symbolicate .NET runtime frames in Android tombstones","Symbolicate the .NET runtime frames in an Android tombstone file. Extracts BuildIds and PC offsets from the native backtrace, downloads debug symbols from the Microsoft symbol server, and runs llvm-symbolizer to produce function names with source file and line numbers. USE FOR triaging a .NET MAUI or Mono Android app crash from a tombstone, resolving native backtrace frames in libmonosgen-2.0.so or libcoreclr.so to .NET runtime source code, or investigating SIGABRT, SIGSEGV, or other native signals originating from the .NET runtime on Android. DO NOT USE FOR pure Java\u002FKotlin crashes, managed .NET exceptions that are already captured in logcat, or iOS crash logs. INVOKES Symbolicate-Tombstone.ps1 script, llvm-symbolizer, Microsoft symbol server.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2953,2954,2957,2958],{"name":2836,"slug":2837,"type":15},{"name":2955,"slug":2956,"type":15},"Android","android",{"name":20,"slug":21,"type":15},{"name":2959,"slug":2960,"type":15},"Microsoft","microsoft","2026-07-12T08:23:21.595572",{"slug":2963,"name":2963,"fn":2964,"description":2965,"org":2966,"tags":2967,"stars":2944,"repoUrl":2945,"updatedAt":2979},"apple-crash-symbolication","symbolicate .NET runtime frames in crash logs","Symbolicate .NET runtime frames in Apple platform .ips crash logs (iOS, tvOS, Mac Catalyst, macOS). Extracts UUIDs and addresses from the native backtrace, locates dSYM debug symbols, and runs atos to produce function names with source file and line numbers. Automatically downloads .dwarf symbols from the Microsoft symbol server using Mach-O UUIDs. USE FOR triaging a .NET MAUI or Mono app crash from an .ips file on any Apple platform, resolving native backtrace frames in libcoreclr or libmonosgen-2.0 to .NET runtime source code, retrieving .ips crash logs from a connected iOS device or iPhone, or investigating EXC_CRASH, EXC_BAD_ACCESS, SIGABRT, or SIGSEGV originating from the .NET runtime. DO NOT USE FOR pure Swift\u002FObjective-C crashes with no .NET components, or Android tombstone files. INVOKES Symbolicate-Crash.ps1 script, atos, dwarfdump, idevicecrashreport.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2968,2969,2970,2973,2976],{"name":2836,"slug":2837,"type":15},{"name":20,"slug":21,"type":15},{"name":2971,"slug":2972,"type":15},"iOS","ios",{"name":2974,"slug":2975,"type":15},"macOS","macos",{"name":2977,"slug":2978,"type":15},"Observability","observability","2026-07-12T08:23:20.369986",{"slug":2981,"name":2981,"fn":2982,"description":2983,"org":2984,"tags":2985,"stars":2944,"repoUrl":2945,"updatedAt":2991},"assertion-quality","evaluate assertion quality in test suites","Analyzes the variety and depth of assertions across test suites in any language. Use when the user asks to evaluate assertion quality, find shallow tests, identify assertion-free tests (no assertions or only trivial ones like Assert.IsNotNull \u002F toBeTruthy()), flag self-referential or tautological assertions, measure assertion diversity, or audit whether tests verify different facets of behavior. Polyglot: .NET, Python, TS\u002FJS, Java, Go, Ruby, Rust, Swift, Kotlin, PowerShell, C++. DO NOT USE FOR: writing new tests (use code-testing-agent \u002F writing-mstest-tests), mutation reasoning about whether tests would catch a bug (use test-gap-analysis), or a general severity-ranked anti-pattern audit (use test-anti-patterns), fixing or rewriting assertions, or writing, fixing, or modernizing MSTest tests, assertions, or attributes (use writing-mstest-tests).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2986,2987,2990],{"name":2842,"slug":2843,"type":15},{"name":2988,"slug":2989,"type":15},"QA","qa",{"name":17,"slug":18,"type":15},"2026-07-12T08:23:51.277743",{"slug":2993,"name":2993,"fn":2994,"description":2995,"org":2996,"tags":2997,"stars":2944,"repoUrl":2945,"updatedAt":3011},"author-component","create and review Blazor components","Create or review Blazor components (.razor files) with correct architecture. USE FOR: writing new Blazor components that do NOT involve JavaScript interop, implementing parameters and EventCallback, RenderFragment slots, component lifecycle (OnInitializedAsync, OnParametersSet), async patterns, IAsyncDisposable, CancellationToken, CSS isolation, code-behind. DO NOT USE FOR: creating new projects (use create-blazor-project), JavaScript interop or calling browser APIs from Blazor (use use-js-interop), forms and validation (use collect-user-input), prerendering issues (use support-prerendering), HTTP data fetching patterns (use fetch-and-send-data), coordinating state between unrelated components (use coordinate-components).\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[2998,2999,3002,3005,3008],{"name":2836,"slug":2837,"type":15},{"name":3000,"slug":3001,"type":15},"Blazor","blazor",{"name":3003,"slug":3004,"type":15},"C#","csharp",{"name":3006,"slug":3007,"type":15},"UI Components","ui-components",{"name":3009,"slug":3010,"type":15},"Web Development","web-development","2026-07-15T06:03:29.216359",{"slug":3013,"name":3013,"fn":3014,"description":3015,"org":3016,"tags":3017,"stars":2944,"repoUrl":2945,"updatedAt":3021},"binlog-failure-analysis","analyze MSBuild binary logs","Analyze MSBuild binary logs to diagnose build failures. USE FOR: build errors that are unclear from console output, diagnosing cascading failures across multi-project builds, tracing MSBuild target execution order, and generally any MSBuild build issues. Requires an existing .binlog file. DO NOT USE FOR: generating binlogs (use binlog-generation), non-MSBuild build systems.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3018,3019,3020],{"name":2842,"slug":2843,"type":15},{"name":20,"slug":21,"type":15},{"name":2959,"slug":2960,"type":15},"2026-07-12T08:21:34.637923",{"slug":3023,"name":3023,"fn":3024,"description":3025,"org":3026,"tags":3027,"stars":2944,"repoUrl":2945,"updatedAt":3033},"binlog-generation","generate MSBuild binary logs for diagnostics","Generate MSBuild binary logs (binlogs) for build diagnostics and analysis. USE FOR: adding \u002Fbl:{} to any dotnet build, test, pack, publish, or restore command to capture a full build execution trace, prerequisite for binlog-failure-analysis and build-perf-diagnostics skills, enabling post-build investigation of errors or performance. Requires MSBuild 17.8+ \u002F .NET 8 SDK+ for {} placeholder; PowerShell needs -bl:{{}}. DO NOT USE FOR: non-MSBuild build systems (npm, Maven, CMake), analyzing an existing binlog (use binlog-failure-analysis instead).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3028,3031,3032],{"name":3029,"slug":3030,"type":15},"Build","build",{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},"2026-07-19T05:38:19.340791",{"slug":3035,"name":3035,"fn":3036,"description":3037,"org":3038,"tags":3039,"stars":2944,"repoUrl":2945,"updatedAt":3043},"build-parallelism","optimize MSBuild build parallelism","Diagnose and fix under-parallelized MSBuild builds. USE WHEN a multi-project solution build is slower than expected, doesn't speed up when you add cores, pegs a single core while others idle, or you want to know why `-m` isn't helping. Note: `\u002Fmaxcpucount` default is 1 (sequential) — always pass `-m` for parallel builds. Covers finding the critical path (longest serial ProjectReference chain), graph build (`\u002Fgraph`), BuildInParallel, and solution filters (`.slnf`). DO NOT USE FOR: single-project builds, incremental issues (use incremental-build), compilation slowness inside one project (use build-perf-diagnostics), non-MSBuild build systems.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3040,3041,3042],{"name":2836,"slug":2837,"type":15},{"name":13,"slug":14,"type":15},{"name":2929,"slug":2930,"type":15},"2026-07-19T05:38:18.364937",{"slug":3045,"name":3045,"fn":3046,"description":3047,"org":3048,"tags":3049,"stars":2944,"repoUrl":2945,"updatedAt":3056},"build-perf-baseline","establish and optimize build performance baselines","Establish build performance baselines and apply systematic optimization techniques. USE FOR: diagnosing slow builds, establishing before\u002Fafter measurements (cold, warm, no-op scenarios), applying optimization strategies like MSBuild Server, static graph builds, artifacts output, and dependency graph trimming. Start here before diving into build-perf-diagnostics, incremental-build, or build-parallelism. DO NOT USE FOR: non-MSBuild build systems, detailed bottleneck analysis (use build-perf-diagnostics after baselining).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3050,3051,3054,3055],{"name":13,"slug":14,"type":15},{"name":3052,"slug":3053,"type":15},"Monitoring","monitoring",{"name":2929,"slug":2930,"type":15},{"name":17,"slug":18,"type":15},"2026-07-12T08:21:35.865649",{"slug":3058,"name":3058,"fn":3059,"description":3060,"org":3061,"tags":3062,"stars":2944,"repoUrl":2945,"updatedAt":3067},"build-perf-diagnostics","diagnose MSBuild build performance bottlenecks","Diagnose MSBuild build performance bottlenecks using binary log analysis. USE FOR: identifying why builds are slow by analyzing binlog performance summaries, detecting ResolveAssemblyReference (RAR) taking >5s, Roslyn analyzers consuming >30% of Csc time, single targets dominating >50% of build time, node utilization below 80%, excessive Copy tasks, NuGet restore running every build. Covers timeline analysis, Target\u002FTask Performance Summary interpretation, and 7 common bottleneck categories. Use after build-perf-baseline has established measurements. DO NOT USE FOR: establishing initial baselines (use build-perf-baseline first), fixing incremental build issues (use incremental-build), parallelism tuning (use build-parallelism), non-MSBuild build systems.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3063,3064,3065,3066],{"name":2836,"slug":2837,"type":15},{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},{"name":2929,"slug":2930,"type":15},"2026-07-12T08:21:40.961722",{"slug":3069,"name":3069,"fn":3070,"description":3071,"org":3072,"tags":3073,"stars":2944,"repoUrl":2945,"updatedAt":3077},"check-bin-obj-clash","detect MSBuild output path conflicts","Detects MSBuild projects with conflicting OutputPath or IntermediateOutputPath. USE FOR: builds failing with 'Cannot create a file when that file already exists', 'The process cannot access the file because it is being used by another process', intermittent build failures that succeed on retry, or missing\u002Foverwritten outputs in multi-project or multi-targeting builds where bin\u002Fobj (or project.assets.json) collide. Common causes: shared OutputPath, missing AppendTargetFrameworkToOutputPath, extra global properties (e.g. PublishReadyToRun), or SetTargetFramework on a ProjectReference to a single-targeting project. DO NOT USE FOR: file access errors unrelated to MSBuild (OS-level locking), single-project single-TFM builds, non-MSBuild build systems.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":8},[3074,3075,3076],{"name":20,"slug":21,"type":15},{"name":13,"slug":14,"type":15},{"name":2988,"slug":2989,"type":15},"2026-07-19T05:38:14.336279",144]