[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-aws-labs-hyperpod-incident-rca":3,"mdc-36zvea-key":48,"related-org-aws-labs-hyperpod-incident-rca":4594,"related-repo-aws-labs-hyperpod-incident-rca":4769},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":43,"sourceUrl":46,"mdContent":47},"hyperpod-incident-rca","perform root-cause analysis for HyperPod incidents","Root-cause analysis for a SageMaker HyperPod incident, after triage has decided to PROCEED. Runs at the INCIDENT_RCA stage. Reads describe-cluster, list-cluster-nodes, list-cluster-events, and HMA CloudWatch streams; reconstructs a timeline; classifies as Suppress \u002F Monitor \u002F Escalate \u002F Resolved against time budgets and recurrence statistics from the HyperPod mental model. Produces a human-readable verdict report with recommended operator actions. The complementary INCIDENT_TRIAGE skill `hyperpod-incident-triage` decides LINKED \u002F SKIPPED \u002F PROCEED before this skill runs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"aws-labs","AWS Labs","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Faws-labs.png","awslabs",[13,17,20,23],{"name":14,"slug":15,"type":16},"Operations","operations","tag",{"name":18,"slug":19,"type":16},"Incident Response","incident-response",{"name":21,"slug":22,"type":16},"AWS","aws",{"name":24,"slug":25,"type":16},"Debugging","debugging",472,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fawsome-distributed-ai","2026-09-02T07:47:38.297082",null,206,[22,32,33,34,35,36,37,38,39,40,41,42],"distributed-inference","distributed-training","efa","eks","generative-ai","gpu","hyperpod","kubernetes","parallelcluster","physical-ai","slurm",{"repoUrl":27,"stars":26,"forks":30,"topics":44,"description":45},[22,32,33,34,35,36,37,38,39,40,41,42],"Collection of best practices, reference architectures, model training examples and utilities to train large models on AWS. ","https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fawsome-distributed-ai\u002Ftree\u002FHEAD\u002Farchitectures\u002Fsagemaker-hyperpod-slurm\u002Ftools\u002Fdevops-agent\u002Fskills\u002Fhyperpod-incident-rca","---\nname: hyperpod-incident-rca\ndescription: Root-cause analysis for a SageMaker HyperPod incident, after triage has decided to PROCEED. Runs at the INCIDENT_RCA stage. Reads describe-cluster, list-cluster-nodes, list-cluster-events, and HMA CloudWatch streams; reconstructs a timeline; classifies as Suppress \u002F Monitor \u002F Escalate \u002F Resolved against time budgets and recurrence statistics from the HyperPod mental model. Produces a human-readable verdict report with recommended operator actions. The complementary INCIDENT_TRIAGE skill `hyperpod-incident-triage` decides LINKED \u002F SKIPPED \u002F PROCEED before this skill runs.\nmetadata:\n  version: \"0.6.1\"\n  agent_types: [\"INCIDENT_RCA\"]\n---\n\n# HyperPod incident RCA skill\n\nThis skill runs at the **INCIDENT_RCA stage**, after DevOps Agent's\ntriage stage decided to `PROCEED` with a full investigation. By the\ntime this skill loads, the task already has a primary investigation\nslot and the agent has full AWS API access.\n\n**The complementary triage skill** `hyperpod-incident-triage` runs at\nthe INCIDENT_TRIAGE stage BEFORE this skill, and decides whether an\nincoming event should be `LINKED` \u002F `SKIPPED` \u002F `PROCEED` using concise\ndeclarative correlation rules (keep distinct fault types on the same\ninstance group separate; skip concurrent periodic audits). Separately,\nfor periodic audits, the **audit Lambda gates volume** — it inspects\nKubernetes state itself and only invokes an investigation when a real\nissue is present (plus a daily heartbeat), so a \"periodic audit\"\ninvestigation reaching this skill already corresponds to something\nworth looking at.\n\nWhen this skill DOES run, it does triage-like classification and\nroot-cause analysis in one pass because they need the same evidence:\nsnapshot data\n(`describe-cluster`, `describe-cluster-node`) alone cannot distinguish\n\"HyperPod is auto-recovering\" from \"HyperPod has given up\" — only the\ncross-source timeline can.\n\n**Operating policy.** Read-only. Never call a state-changing API.\nPresent every action that mutates anything (operator-driven replacement,\nSSM session, label change) as a **Recommended action (operator runs\nthis)** block. The customer's operator decides whether to execute it.\n\n**Required reading.** Open\n[references\u002Fhyperpod-mental-model.md](references\u002Fhyperpod-mental-model.md)\nin parallel with this skill. The mental-model doc is the ground truth\nfor what each HyperPod signal means; this skill is the decision\nprocedure that consumes those meanings.\n\n## Why one skill, not two\n\nA single failed instance can vanish from `list-cluster-nodes` between\nretry attempts, and HyperPod may auto-retry from `Failed` status —\nneither is a terminal escalation signal on its own. Distinguishing\n\"still retrying within budget\" from \"stuck\" requires the full\n`list-cluster-events` chain, the HMA CloudWatch stream, and the cluster\nstatus all aligned on a wall-clock timeline. Building a separate\n\"triage\" skill that decides without that data would mean re-deciding\nincorrectly on every event.\n\n## Relationship to `hyperpod-incident-triage` and the audit Lambda\n\nTwo things run before this RCA skill and mean it does NOT need to\nre-implement duplicate\u002Fstale-audit suppression itself:\n\n- **`hyperpod-incident-triage`** makes the LINK \u002F SKIP \u002F PROCEED\n  decision with concise declarative rules (same fault on the same\n  component links; different fault types on the same instance group\n  stay separate; concurrent periodic audits are skipped).\n- **The periodic-audit Lambda gates volume**: for scheduled audits it\n  inspects Kubernetes state (CrashLoopBackOff \u002F NotReady) and POSTs the\n  webhook only when a real issue exists — so a periodic-audit\n  investigation reaching this skill already corresponds to a real\n  finding, not an idle poll. HyperPod control-plane faults arrive via\n  the event-driven webhook bridge, not the audit.\n\nRCA still does the full Phase 1 gather (including Pod\u002FNode inspection\nand Phase 3d threshold checks) because the reasoning stage needs the\nfull evidence. Earlier RCA \"stale-evidence\" rules (3 \u002F 3b) were removed\nand are not reintroduced — that concern is handled upstream now.\n\n## Workflow\n\n### Trigger modes\n\nThe skill is loaded from two trigger sources, and Phase 1 \u002F Phase 3\nbehavior differs slightly between them:\n\n| Mode | Trigger | Scope |\n|---|---|---|\n| **Incident mode** | Webhook fire from the bridge Lambda (an EventBridge HyperPod event passed the noise filter). The investigation context carries `clusterName` and usually an `instanceId`. | Focused on the specific incident referenced in the trigger. |\n| **Audit mode** | A synthesized webhook POST from the periodic-audit Lambda. Unlike a raw scheduled trigger, this payload **does** carry context: the description names the cluster (`clusterName`) and, when the Lambda-side checks fired, lists `detectedIssues` and a `k8sChecks` config line. | Confirm the detected issues, then scan the named cluster for any in-flight or recently-resolved fault chains and re-classify each. Catches: (1) `Monitor` incidents that have now succeeded (emit `Resolved`), (2) `Monitor` incidents stuck past their re-check budget (escalate), (3) recurring patterns that didn't trigger an EventBridge event recently but persist statistically. |\n\nIn both modes the same Phase 1 \u002F Phase 2 \u002F Phase 3 \u002F Phase 4 logic\napplies. The differences are noted inline below.\n\n### Phase 1 — Gather (run in parallel)\n\nFor the incident referenced in the trigger event (cluster name + optional\ninstance id), collect these in parallel. **Do not stop on a single\nsignal — gather all of them before classifying.**\n\n**Audit mode**: the trigger payload **does** carry the cluster name\nin its description (the periodic-audit Lambda synthesizes it), so use\nthat. Only if the name is genuinely absent, fall back to the cluster(s)\nreachable from this Agent Space's AWS account association (typically\njust one HyperPod cluster — discover it via `sagemaker list-clusters`).\nRun the full Phase 1 gather for the cluster. Then in Phase 3, classify\nper open fault chain found in the cluster-events window (plus any\n`detectedIssues` the payload already carries), not per single trigger.\n\n1. **Cluster state**: `aws sagemaker describe-cluster --cluster-name \u003Cname>`\n   — current `ClusterStatus`, `NodeRecovery`, `Orchestrator` (Eks vs.\n   Slurm), `InstanceGroups[].CurrentCount` \u002F `TargetCount` \u002F `Status`.\n2. **Node inventory**: `aws sagemaker list-cluster-nodes --cluster-name \u003Cname>`\n   plus `describe-cluster-node` for any instance id named in the trigger\n   event. **An instance id mentioned in the event that is NOT in\n   `list-cluster-nodes` is a signal, not an error** — the node may have\n   been removed mid-retry.\n3. **Cluster events chain**: `aws sagemaker list-cluster-events --cluster-name \u003Cname>`,\n   paginated to ≥500 entries or until events reach **7 days** back,\n   whichever comes first. This is the **canonical record of replacement\n   attempts including failed ones** and survives nodes disappearing\n   from the node list. The wider 7-day window also feeds the recurring-\n   pattern classification rules in Phase 3 — don't shorten the lookback\n   even when the trigger event is recent.\n   - Available on EKS clusters and on Slurm clusters with **Continuous\n     Provisioning** enabled. If the API returns\n     `ValidationException`\u002Fequivalent on a Slurm cluster without CP,\n     note \"event timeline unavailable; confidence degraded\" in the\n     final report and proceed with the remaining sources.\n4. **HMA CloudWatch stream**: log group\n   `\u002Faws\u002Fsagemaker\u002FClusters\u002F\u003CNAME>\u002F\u003CCLUSTER_ID>`, streams matching\n   `SagemakerHealthMonitoringAgent\u002F*` (filter to the affected instance\n   group \u002F instance if known). Filter for `HealthMonitoringAgentDetectionEvent`\n   entries and any Xid \u002F DCGM \u002F EFA \u002F OOM messages within the same\n   window as Phase 1 step 3.\n5. **Lifecycle script stream** (only if a replace attempt is in the\n   timeline): same log group, streams matching `LifecycleConfig\u002F*` for\n   the affected instance group \u002F instance id. Look for non-zero exit,\n   timeout, S3 \u002F IAM errors.\n6. **EKS node state** (EKS only): `kubectl get node \u003Cname> -o yaml` for\n   the affected node (if still present). Surface\n   `sagemaker.amazonaws.com\u002Fnode-health-status`, `fault-types`,\n   `fault-reasons`, `fault-details` labels\u002Fannotations and any taints.\n7. **Slurm node state** (Slurm only, if SSM not required): use what's\n   reachable from the control plane — `describe-cluster-node` gives\n   most of what we need; deep `scontrol`\u002F`sinfo` requires SSM and is\n   out of scope here.\n8. **Kubernetes state — MANDATORY in audit mode on EKS clusters.**\n   You MUST execute both of the following before entering Phase 2,\n   even if the trigger payload has no `k8sChecks` block, even if\n   earlier gather steps already suggest a plausible verdict, and\n   even if `list-cluster-events` returned a rich history. Live\n   Pod\u002FNode state is the source of truth for \"what is broken *right\n   now*\" — the wider event history is context, not a substitute.\n   - `kubectl get pods -A -o json` — full cluster Pod state.\n   - `kubectl get nodes -o json` — full Node state including\n     `status.conditions[]` and their `lastTransitionTime`.\n\n   Skip this step only if: (a) the cluster's orchestrator is Slurm\n   (not EKS), OR (b) the trigger is incident-mode (webhook-triggered\n   for a specific fault; Pod\u002FNode scan belongs in periodic audit).\n\n   The trigger's `k8sChecks` block supplies **thresholds and\n   namespace filters** used by Phase 3d — not a gate on whether\n   kubectl runs. Fields:\n\n   ```json\n   {\n     \"enabled\": true,\n     \"crashLoopMinRestarts\": 5,\n     \"crashLoopRecencyMinutes\": 15,\n     \"notReadyNodePercentThreshold\": 10,\n     \"notReadyDurationMinutes\": 15,\n     \"ignoreNamespaces\": [\"kube-public\", \"kube-node-lease\"],\n     \"systemNamespaces\": [\"kube-system\", \"aws-hyperpod\", \"amazon-cloudwatch\"]\n   }\n   ```\n\n   **Where to find this block at runtime.** The DevOps Agent\n   platform preserves the top-level `description` string of the\n   incoming task verbatim, but drops nested sub-objects from the\n   webhook payload. The audit Lambda therefore inlines the\n   `k8sChecks` block into the task description text on a line that\n   begins:\n\n   ```\n   k8sChecks configuration (parse as JSON, then apply per Phase 1 step 8 + Phase 3d):\n   { ... }\n   ```\n\n   Locate that line in the task description, extract the JSON\n   object on the next line, and use its fields for Phase 3d\n   thresholds and namespaces. Do NOT default any field unless the\n   whole block is absent — the block's values override the built-in\n   defaults regardless of whether they equal the defaults.\n\n   If the block is absent from the description entirely, use\n   defaults: `crashLoopMinRestarts=5`, `crashLoopRecencyMinutes=15`,\n   `notReadyNodePercentThreshold=10`, `notReadyDurationMinutes=15`,\n   `ignoreNamespaces=[\"kube-public\",\"kube-node-lease\"]`,\n   `systemNamespaces=[\"kube-system\",\"aws-hyperpod\",\"amazon-cloudwatch\"]`.\n\n   If the block has `enabled: false`, still run the kubectl commands\n   (Phase 1's job is discovery), but Phase 3d will not fire — see\n   note there.\n\n### Phase 1 gather sanity gate — verify BEFORE entering Phase 2\n\nBefore starting Phase 2 (timeline reconstruction), confirm that\nPhase 1 executed the required steps:\n\n- Steps 1, 2, 3 (describe-cluster, list-cluster-nodes,\n  list-cluster-events): required for both incident and audit modes.\n- Step 8 (kubectl get pods, kubectl get nodes): required in audit\n  mode on EKS. Missing here is a **hard error** — do not proceed to\n  Phase 2. Instead: run step 8 now, then re-enter this gate.\n\nDo not rationalize skipping step 8 with reasoning like \"the\n`list-cluster-events` window looks quiet so I don't need to check\npods\" or \"the LCS storm from earlier is more interesting.\" Those\nare outputs of Phase 3 reasoning, not inputs to Phase 1. Pod state\nmust be gathered as **evidence** before Phase 3 rules can weigh\n\"live k8s problem\" against \"historical event pattern.\"\n\n### Phase 2 — Reconstruct the timeline\n\nBuild a single ordered timeline keyed by UTC timestamp across all\nsources, restricted to the affected scope (cluster, instance group, or\ninstance). Mark each entry with its source. The shape should be:\n\n```\nT+0:00   [HMA]            HealthMonitoringAgentDetectionEvent — Xid 79 on GPU 3\nT+0:32   [ClusterEvent]   Action:Replace marked on i-aaa (NodeRecovery=Automatic)\nT+0:33   [Node]           list-cluster-nodes: i-aaa removed\nT+0:34   [ClusterEvent]   Replacement started for instance group worker1\nT+18:21  [ClusterEvent]   Replacement failed: EFA health checks did not run successfully\nT+18:25  [Node]           list-cluster-nodes: still missing\nT+20:10  [ClusterEvent]   Replacement started for instance group worker1   ← second attempt\n...\n```\n\nThis is the artifact the classification phase reasons over. Include it\nin the final report regardless of verdict — operators need it to\ndouble-check the agent's call.\n\n### Phase 2b — Build the fault-content signature for each event\n\nEvery fault event (Error\u002FWarn cluster event or HMA detection within\nthe 4h window for signature-set computation, and within the 7d window\nfor recurrence statistics) gets a **signature string**, formed by\nconcatenating the event's Description with any FailureMessage the\nbridge Lambda enriched it with, prefixed by the InstanceGroup name.\n\nSignature format:\n\n```\n\u003Cig>:\u003Cfull-description-and-failure-message-content>\n```\n\nSources of the content, in order of preference:\n\n1. The bridge Lambda's enriched description (top-level `description`\n   field in the incoming trigger payload). This already includes\n   `Description: ...` + `FailureMessage: ...` + `InstanceMetadata: ...`\n   fields the bridge assembled by calling `DescribeClusterEvent`.\n2. For events retrieved via `list-cluster-events` during Phase 1\n   pagination (not through the bridge), call\n   `describe-cluster-event --event-id \u003Ceid>` yourself for each\n   Error\u002FWarn EventId. Concatenate `Description` +\n   `EventDetails.EventMetadata.Instance.FailureMessage` (and any\n   other populated `Instance.*` fields) the same way the bridge does.\n3. For HMA CloudWatch stream events: use the full\n   `HealthMonitoringAgentDetectionEvent` text.\n4. Cluster State Change events: use `ClusterStatus + Description`.\n\n**Why full concatenated content, not hard-coded categories.**\nClassifying events into a fixed enum of categories (`gpu-xid`,\n`lifecycle-script-failed`, `capacity-insufficient`, etc.) via regex\nrules merges distinct fault types — for example, \"capacity for\nml.g5.8xlarge\", \"capacity for ml.p5.48xlarge\", \"EFA health check\nfailed\", and \"generic provisioning failure\" all collapse to the same\n`instance-creation-failed:generic` key, because the top-level\n`Description` field only says `\"Failed to provision EC2 Instance in\nCluster ...\"`. The actual root cause lives in `FailureMessage` (a\ndifferent string per fault type), so include it. Concatenated raw\ncontent is more robust than enum-based categories.\n\n**InstanceGroup extraction**:\n\n- Cluster Event: `detail.EventDetails.InstanceGroupName`\n- Node Health \u002F Cluster State Change: `detail.InstanceGroupName`\n- Cluster-level fault: empty (`\"\"`) — no IG prefix.\n\n### Phase 2c — Compute recurrence statistics\n\nOver the 7-day `list-cluster-events` window from Phase 1 (and HMA\nCloudWatch stream over the same window), compute and record (used by\nPhase 3 rules 6–8):\n\n- `replacements_7d_total` — count of `Replace` actions \u002F replacement-\n  started cluster events across the whole cluster in the last 7 days.\n- `replacements_7d_by_group[\u003Cig>]` — same, partitioned by InstanceGroup.\n- `replacements_24h_total` — same metric, 24h window.\n- `signature_count_7d[\u003Csignature>]` — count of distinct fault events\n  with the same signature string in the last 7 days. Uses the\n  Phase 2b signature format `\u003Cig>:\u003Ccontent>`.\n  **Exclusion**: events whose Description contains\n  `\"lost orchestration-ready status\"` MUST be excluded from\n  signature counting. These are scale-in-progress noise (see\n  mental-model § \"Scale-in-progress emits spurious Warn events\"),\n  not fault signals. Do NOT count them toward rule 6 thresholds.\n\nInclude these counts in the verdict description's \"What HyperPod is\ndoing right now\" paragraph when they're ≥2 — even if the verdict\nitself doesn't change, the operator should see the count.\n\n### Phase 2d — Compute the signature set\n\nThe signature set is used for verdict-title generation. Encoding it in\nthe title lets the DevOps Agent platform's title-based dedup absorb\nback-to-back audits of the same fault set (identical set → identical\ntitle), while a new fault set produces a new title and a fresh\ninvestigation.\n\n- `current_signature_set` — sorted, deduplicated set of signature\n  strings (per Phase 2b) for every distinct fault event in the\n  4-hour window. Examples:\n  - `{\"worker4:Description: Failed to provision EC2 Instance in Cluster my-cluster and InstanceGroup worker4. FailureMessage: We currently do not have sufficient capacity to launch new ml.g5.8xlarge instances. Please try again.\"}`\n  - `{\"worker2:Description: Instance i-XXXX is unhealthy. HyperPod Health Monitoring Agent (HMA) has detected fault type NvidiaGPUUnhealthy on this node and is unhealthy. Repair action: Replace.\"}`\n\n  Render in the verdict title as a sorted comma-separated list.\n  Full strings; do NOT truncate. If total title length exceeds\n  platform limits (rare but possible for very long FailureMessages),\n  trim signatures from the middle with `...` markers to preserve\n  the beginning (where the discriminating text lives).\n- `current_most_recent_event_at` — the latest `EventTime` of any\n  Error\u002FWarn cluster event or HMA detection within the window.\n\n### Phase 3 — Classify\n\n**Ordering — MANDATORY.**\n\n1. **First**, run **Phase 3d** (Kubernetes-state checks — CrashLoopBackOff,\n   NotReady). Emit any Escalate verdicts from Phase 3d immediately.\n2. **Then**, run the fault-chain rules below in this Phase 3 table.\n\nReason: Phase 3d looks at **live current state**, which is the most\nactionable signal for a periodic audit. The main rule table below\nlooks at the 4h\u002F24h\u002F7d event history, which is context. When both\nwould fire, the operator needs the live state surfaced regardless\nof whether historical patterns are ALSO present. Do not skip Phase\n3d because \"the event history already provides an interesting\nverdict\" — that produced the 2026-07-09 miss where a live\nCrashLoopBackOff was overshadowed by a recurring-LCS-pattern\nverdict from a *already-resolved* 07-08 storm.\n\nEmitting both a Phase 3d verdict and a Phase 3 verdict on the same\naudit is expected and correct. Each becomes its own symptom record;\nthe operator gets both signals via email.\n\n#### Phase 3d — Kubernetes-state checks (audit mode, EKS only)\n\nRun this **BEFORE** the fault-chain classification rules below\n(see the \"Ordering — MANDATORY\" note at the top of Phase 3). Phase\n3d uses the `kubectl get pods` \u002F `kubectl get nodes` output from\nPhase 1 step 8, which is mandatory in audit-mode-on-EKS regardless\nof the k8sChecks payload block. Emit each verdict as an\nindependent symptom record.\n\n**Threshold + namespace configuration.** Use the `k8sChecks` block\nparsed from the task **description line** per Phase 1 step 8 — the\nDevOps Agent platform preserves the top-level `description` verbatim\nbut flattens nested payload sub-objects, so `data.metadata.k8sChecks`\nis NOT reliably present; the description-inline copy is the source of\ntruth. If no block is in the description, use the defaults documented\nin Phase 1 step 8. If the block is present with `enabled: false`, DO\nNOT emit Phase 3d verdicts (customer has opted out of the k8s-state\nEscalate) — but Phase 1's kubectl gather still ran because it's part\nof general state discovery, and its data may still surface in Phase 4\nreport context.\n\n**Pod check — CrashLoopBackOff loop.** For each Pod in the\n`kubectl get pods -A -o json` output:\n\n1. **Namespace classification.** Look up `pod.metadata.namespace`:\n   - If it is in `k8sChecks.ignoreNamespaces` → skip this pod entirely.\n   - If it is in `k8sChecks.systemNamespaces` → workload class is\n     `system-workload`.\n   - Otherwise → workload class is `customer-workload`.\n\n   This is a plain set-membership lookup, in this exact order. No\n   pattern matching, no wildcards, no precedence rules. The Lambda\n   has already validated that `ignoreNamespaces` and\n   `systemNamespaces` do not overlap, so a pod's namespace matches\n   at most one list.\n\n2. **CrashLoopBackOff detection.** For each container in\n   `pod.status.containerStatuses[]`, decide whether it is in a\n   persistent crash loop. Kubernetes does **not** expose when the\n   loop started in a snapshot — `containerStatuses[].state.waiting`\n   carries only `reason`\u002F`message` (no timestamp), and\n   `lastState.terminated` holds only the *most recent* crashed run —\n   so use crash evidence, not a duration:\n   - **Loop present** when `state.waiting.reason == \"CrashLoopBackOff\"`\n     OR a `lastState.terminated` record exists (persistent evidence,\n     stable across the backoff cycle).\n   - **Enough restarts:** `restartCount >= k8sChecks.crashLoopMinRestarts`.\n   - **Still active:** the last crash\n     (`lastState.terminated.finishedAt`) is within\n     `k8sChecks.crashLoopRecencyMinutes` minutes — or the container is\n     currently in `CrashLoopBackOff` (treat a missing `finishedAt` as\n     active only in that case).\n3. **Verdict.** If all three hold → emit an `Escalate` verdict with:\n   - Title: `Triage verdict: Escalate — CrashLoopBackOff loop active :: (\u003Cnamespace>\u002F\u003Cpod>:\u003Ccontainer>)`\n   - Workload class tag in the description (`system-workload` or\n     `customer-workload`) so downstream email routing can decide who\n     to page.\n   - Include: pod name, namespace, container name, `restartCount`,\n     whether it is currently in backoff, and the most recent\n     `lastState.terminated.reason` \u002F `.exitCode` if available.\n\n**Node check — NotReady percentage.** From\n`kubectl get nodes -o json`:\n\n1. Total node count = length of `items[]`.\n2. For each node, examine `status.conditions[]` for\n   `type: Ready`. If `status: \"False\"` or `\"Unknown\"` AND\n   `now() - lastTransitionTime` exceeds\n   `k8sChecks.notReadyDurationMinutes` minutes → count as NotReady.\n3. If `(NotReady count \u002F total count) * 100` meets or exceeds\n   `k8sChecks.notReadyNodePercentThreshold` → emit an `Escalate`\n   verdict with:\n   - Title: `Triage verdict: Escalate — NotReady nodes exceeded threshold :: (\u003Cn>\u002F\u003Ctotal> nodes NotReady)`\n   - Include the affected node names, their NotReady durations, and\n     any `taints[]` that would explain the state.\n\n**Interaction with the fault-chain classification.** If a HyperPod\nfault chain in the main rule table already covers the same\nnode (e.g. rule 11 says the retry chain is stuck for a specific\ninstance), the Phase 3d NotReady check may re-flag the same node.\nThat's fine — the two verdicts have different scopes and different\nrecommended actions. Emit both; the operator gets richer context.\n\n**Interaction with `Suppress — periodic audit, no open incidents`.**\nRule 1 fires when no HyperPod fault chains are open. Phase 3d can\nstill fire independently — a CrashLoopBackOff pod is an incident\neven when HyperPod's own event stream is quiet. If Phase 3d\nproduces any verdict, do NOT emit the rule 1 Suppress; Phase 3d has\nfound something to report.\n\n#### Fault-chain classification rules\n\nApply the rules below **in order**. Stop at the first match.\n\n**Audit mode**: classify each open fault chain found in the\ncluster-events window independently. Emit one verdict per chain\n(plus one `Suppress — periodic audit, no open incidents` if no\nchains are open AND Phase 3d also emitted nothing). A fault chain\nis \"open\" if it had a fault event within the last 4 hours and has\nnot yet emitted a successful `Running` transition + 30 min\nclean-window.\n\n**Historical-only chains are not open.** If the only fault events\nin the 4-hour window belong to a chain that has already completed\n(e.g., all replacements succeeded and all affected nodes are back\nin `Running` for ≥30 min), the chain is closed. Do not emit an\n`Escalate` verdict on the basis of `signature_count_7d` alone\nagainst a closed chain unless the recurrence rule 6\u002F7\u002F8 fires from\nstatistics that INCLUDE at least one event within the last 4 hours.\nA 7-day-old already-recovered pattern is context, not an actionable\nincident.\n\nThe recurring-pattern rules (6–8) are checked **before** the\nsingle-incident rules (9–13) — a node that's auto-recovering for the\n3rd time this week should be flagged as a pattern, not silently\nclassified as `Monitor — first attempt`.\n\n| # | Signal pattern | Verdict |\n|---|---|---|\n| 1 | **Audit mode** AND no fault events in the 4-hour window AND no `Monitor` fault chain still open | **Suppress — periodic audit, no open incidents** (skip Phase 4 entirely or emit a minimal record; nothing actionable) |\n| 2 | A previously-`Monitor` fault chain now shows the affected instance(s) back in `Running` AND no new HMA detection for that instance in the last 30 min AND cluster status is `InService` | **Resolved — auto-recovery succeeded** (operator gets a closure email; include the original detection time and total elapsed) |\n| 3 | Trigger detail-type is `Cluster Event` with `EventLevel=Info` and the timeline shows no node-health activity | **Suppress** |\n| 4 | Cluster status is `Failed` or `RollingBack` | **Escalate** (cluster-level) |\n| 5 | `NodeRecovery=None` on the cluster AND a node has been marked `Action:*` \u002F `UnschedulablePending*` AND no replacement has started within 5 minutes | **Escalate** — auto-recovery is off; operator must trigger replacement |\n| 6 | `signature_count_7d[\u003Csignature>] ≥ 3` for ANY signature — same fault content (Description + FailureMessage) has driven ≥3 replacements on the same InstanceGroup in the last 7 days (auto-recovery may be succeeding each time). Because signatures are the full concatenated content, this fires cleanly for repeated same-cause faults (e.g. 3× capacity errors for the same instance type) but does NOT over-merge distinct causes. | **Escalate — recurring fault pattern**: HyperPod is repairing the symptom, but the underlying cause hasn't gone away. Include the signature (first ~200 chars) and the timestamps of all prior occurrences. Recommend operator actions appropriate to the fault content (vendor-exclusion for hardware Xid faults; LCS bug for lifecycle-script failures; capacity request or IG-move for capacity failures; VPC\u002FSG review for EFA health-check failures). Infer the appropriate action class from the FailureMessage content, not from a hard-coded category enum. |\n| 7 | `replacements_24h_total ≥ 5` — five or more replacements anywhere in the cluster within 24 hours | **Escalate — fleet-wide instability**: the rate of node churn is abnormal regardless of individual root causes. |\n| 8 | `replacements_7d_by_group[\u003Cig>] ≥ 5` — five or more replacements on the same InstanceGroup in 7 days | **Escalate — instance-group instability**: the affected IG (which may be a specific SKU or topology placement) is failing more often than the rest of the cluster. |\n| 9 | Exactly one replacement attempt in flight, started within the last 30 minutes, no prior failure in the chain | **Monitor — first attempt** (next re-check in 30 min via scheduled audit) |\n| 10 | Multiple replacement attempts in the chain, total elapsed since the first failure ≤ 90 minutes, the most recent attempt is *Running* or *Started* (not yet failed) | **Monitor — elevated** (retry in progress, watch closely) |\n| 11 | Multiple replacement attempts, total elapsed > 90 minutes, no successful `Running` transition, AND no new attempt started within the last 30 minutes | **Escalate** — retry chain is stuck |\n| 12 | Node was in `Failed` state AND no new replacement attempt has started within the last 30 minutes AND total time in failing chain > 60 minutes | **Escalate** — HyperPod has given up |\n| 13 | Instance id from the trigger event is missing from `list-cluster-nodes` AND `list-cluster-events` shows no new attempt for the last 30 minutes AND the most recent attempt failed | **Escalate** — instance vanished, no retry |\n| 14 | HMA detection event present but no corresponding `Action:*` \u002F replacement event in the timeline within 10 minutes | **Escalate** — HMA fired without escalating; investigate why (mismatch in node-recovery config, signal didn't classify) |\n| 15 | None of the above match | **Monitor — uncategorized** (include the full timeline; flag for review) |\n\n> **Duplicate \u002F stale-audit suppression is handled upstream**, not by\n> this skill: the `hyperpod-incident-triage` skill LINKs\u002FSKIPs\n> duplicate incident events and concurrent audits, and the\n> periodic-audit Lambda only invokes an investigation when a real issue\n> is present. The former RCA \"stale-evidence\" rules (3 \u002F 3b) were\n> removed and are not reintroduced.\n\n**Rule 2 (`Resolved`) closes the loop on prior `Monitor` verdicts.**\nA previous `Monitor` verdict promised a re-check; this rule provides\nthat re-check via the scheduled audit. The verdict description\nshould explicitly say \"Incident detected at \u003CT0> is now resolved.\nTotal auto-recovery time: \u003Cduration>. No operator action required.\"\nThis is the closure email the operator needs.\n\n**Rule 1 (`Suppress — periodic audit, no open incidents`) is the no-op\ncase.** When the scheduled audit fires on a healthy cluster, emit a\nsingle minimal record acknowledging the audit ran. The email notifier\nfilters `Suppress` verdicts so no email is sent.\n\n**Recurring-pattern verdicts (6–8) are `Escalate` even when the\nindividual incident is auto-recovering correctly.** The reasoning is\nin your operational goals: HyperPod's resiliency repairs the symptom,\nbut a 3× recurrence of a categorized fault on the same IG is an\nunderlying cause (vendor \u002F capacity pool for `gpu-xid`, code bug for\n`lifecycle-script-failed`, pool exhaustion for `capacity-insufficient`,\netc.) that auto-recovery can't fix. The verdict explanation should\nexplicitly say \"this incident is auto-recovering, but the pattern\nacross the last \u003CN> days warrants human attention\" so operators\nunderstand the verdict isn't about *this* incident's resolution.\n\n**Time budgets are not hardcoded constants — they encode the\nmental-model doc's \"How long things take\" section.** A single replace\ntakes 20–30 min; two attempts plus a slack gap = ~90 min. Don't change\nthese without updating the mental-model doc first.\n\n### Phase 4 — Report (using DevOps Agent's native schema)\n\nDevOps Agent's investigation output is structured: the terminal tool\nis `write_final_investigation_report`, and the agent emits `symptom`\nand `finding` records along the way that get serialized into the\nfinal report. **Author the verdict and timeline into that schema\ndirectly — do NOT invent a separate four-section markdown format**\n(it would be at the agent's mercy during serialization and may not\nsurvive into the final report).\n\nThe agent's schema supports these record types:\n\n| Record type | Used for |\n|---|---|\n| `symptom` | Observable state the operator would notice (a node went unhealthy, an investigation was triggered, the cluster is in `Failed` state). First-class; survives serialization with title + description verbatim. |\n| `finding` with `finding_type: \"root_cause\"` | An identified hardware\u002Fsoftware root cause that cascades to one or more symptoms. |\n| `finding` with `finding_type: \"cause\"` | An intermediate cause linking root_cause to symptoms. |\n| `finding` with `finding_type: \"hypothesis\"` | An unverified explanation. |\n| `investigation_gaps[]` | What the agent could not verify and would need the operator to confirm. |\n\n**Emit exactly these records, in this order:**\n\n1. **First `symptom`** — titled\n   `Triage verdict: \u003Cverdict-name> :: \u003Csignature-set>` where\n   `\u003Cverdict-name>` is one of `Suppress`, `Monitor — first attempt`,\n   `Monitor — elevated`, `Monitor — uncategorized`, `Resolved — auto-recovery succeeded`,\n   or `Escalate — \u003Creason>`. **The `\u003Csignature-set>` suffix is\n   required for any verdict that names specific fault evidence**\n   (all Escalate \u002F Monitor \u002F Resolved variants) — it's a stable,\n   sorted-and-deduplicated representation of the `current_signature_set`\n   computed in Phase 2d, formatted as a comma-separated list of the\n   `\u003Cig>:\u003Ccontent>` signature strings (per Phase 2b) that drove the\n   verdict. Verdict titles use the full concatenated fault-content\n   signature, NOT a hard-coded category name. Examples (line-wrapped\n   here for readability; the actual titles are single-line):\n\n   - `Triage verdict: Escalate — recurring fault pattern :: (worker4:Description: Failed to provision EC2 Instance ... FailureMessage: We currently do not have sufficient capacity to launch new ml.g5.8xlarge instances. Please try again.)`\n   - `Triage verdict: Monitor — first attempt :: (worker2:Description: Instance i-XXXX is unhealthy. HyperPod Health Monitoring Agent (HMA) has detected fault type NvidiaGPUUnhealthy on this node ...)`\n   - `Triage verdict: Resolved — auto-recovery succeeded :: (worker2:Description: Instance i-XXXX is unhealthy ...)`\n   - `Triage verdict: Escalate — fleet-wide instability :: (worker2:..., worker3:..., worker4:...)`\n   - `Triage verdict: Suppress — periodic audit, no open incidents` (no suffix when there's no signature set)\n\n   **Purpose**: the DevOps Agent platform's automatic task-dedup uses\n   title equality within a ~30 minute window. Encoding the signature\n   set in the title means a NEW signature (a new Xid type on the same\n   IG, or the same Xid spreading to a new IG) produces a different\n   title and the platform does NOT dedup it — the new investigation\n   runs. Conversely, identical signature sets produce identical titles\n   and platform dedup quietly absorbs the duplicate audit.\n\n   The description is the prose justification in the shape:\n\n   ```\n   Verdict: \u003Cname>\n\n   Summary:\n   \u003CA single plain-English paragraph (2–4 sentences) that flows naturally and\n   weaves together THREE things, in this order: (1) WHAT HAPPENED — the specific\n   observed problem: fault type, affected instance group \u002F node \u002F instance type,\n   and how many times \u002F how long (never just the generic verdict category);\n   (2) THE LIKELY CAUSE — the most probable root cause inferred from the\n   FailureMessage \u002F event content and cluster context (not a hardcoded category);\n   (3) THE RECOMMENDED ACTION — concrete operator next step(s) appropriate to the\n   cause. Write it as prose, NOT as labeled fields or bullets. Lead the paragraph\n   with \"what happened\" so the first sentence works as a standalone headline.\n   Example:\n   \"Repeated capacity errors are failing to provision ml.p5.48xlarge for instance\n   group 'faulttest' on cluster my-slurm-cluster — 5 attempts in the last 40 minutes. The\n   likely cause is that on-demand capacity for ml.p5.48xlarge is unavailable in\n   this Availability Zone and the instance group is not backed by a training plan\n   or reserved capacity, so every Continuous-Provisioning retry hits the same\n   wall. Recommended: launch in an AZ\u002FRegion where you hold reserved capacity for\n   this instance type or associate the group with a training plan; otherwise\n   reduce the target count to stop the retry loop and request a capacity increase.\">\n\n   What HyperPod is doing right now:\n   \u003Cone paragraph plain English summary referencing the timeline>\n\n   Timeline (UTC):\n   \u003Cthe timeline reconstructed in Phase 2, one event per line, source-tagged>\n\n   Most recent event at:\n   \u003CUTC ISO 8601 timestamp of the most recent fault event in the 4-hour window.\n   Recommended for all verdicts that contain a non-empty signature set so\n   operators reading the report can correlate with their own timelines. Example:\n   2026-06-30T01:36:49Z>\n\n   Next re-check:\n   \u003Conly for Monitor verdicts: UTC timestamp 30 min from now if \"first attempt\",\n   15 min from now if \"elevated\">\n\n   Recommended actions (operator runs these):\n   \u003Conly for Escalate verdicts: explicit operator-runnable remediation steps —\n   see the per-verdict guidance below and the SSM GPU-UUID step for\n   recurring-pattern verdicts. Omit this heading for Suppress \u002F Monitor \u002F\n   Resolved verdicts.>\n   ```\n\n   **The `Summary:` paragraph is REQUIRED for every non-Suppress verdict.** Write\n   it as ONE natural-language paragraph (2–4 sentences) that covers, in order,\n   what happened → the likely cause → the recommended action. Do NOT use labeled\n   fields, bullets, or JSON — just prose. Lead with \"what happened\" so the first\n   sentence stands alone as a headline (the email notifier uses the paragraph as\n   the body summary and its first sentence as the subject). Name the *specific*\n   problem (instance type, instance group, counts, AZ) — never restate the\n   generic verdict category. Do NOT put `[direct]`\u002F`[proxy]`\u002F`[unverified]`\n   confidence tags in this paragraph; those belong on the detailed evidence\n   claims later in the description. Suppress verdicts may omit it (no email).\n\n   `related_resources` on this symptom: `[\"HyperPod cluster \u003Cname>\"]`\n   plus any affected instance IDs.\n\n2. **Additional `symptom` records** — one per observable failure\n   condition (e.g. \"Instance i-xxxx marked unhealthy with\n   NvidiaGPUUnhealthy\"). These are the per-resource symptoms the\n   agent would naturally produce; keep them as separate records so\n   they cross-link with `cascades_to` from the findings.\n\n3. **`finding` records** as appropriate (root cause, intermediate\n   cause, hypothesis). Cross-link via `cascades_to: [\u003Csymptom-id>]`.\n\n4. **`investigation_gaps[]`** — anything Phase 1 couldn't reach.\n   Examples: `list-cluster-events` unavailable on a Slurm cluster\n   without Continuous Provisioning; HMA CloudWatch log group not\n   yet populated; no SSM access from the agent (always include this\n   one with a note that operator can confirm via the suggested SSM\n   command).\n\n   **Emit an `investigation_gap` when `DescribeClusterEvent` returned\n   no FailureMessage for any Error\u002FWarn event** (i.e. the API returned\n   `EventDetails.EventDetails.EventMetadata.Instance` as null or with\n   only `NodeLogicalId`). Title: `FailureMessage missing from N event(s)`.\n   Description: list the affected `EventId`s + timestamps. This surfaces\n   the known HyperPod bug where FailureMessage is populated in the API\n   response for capacity errors but omitted for other fault categories —\n   operators reviewing the report should escalate to the HyperPod team\n   with those EventIds as evidence.\n\n5. **`write_final_investigation_report`** — listing the verdict\n   symptom FIRST in `symptoms[]`, then the per-resource symptoms.\n   Findings listed under their `finding_type` arrays (`root_cause[]`,\n   `cause[]`, `hypothesis[]`). `investigation_gaps[]` populated from\n   step 4.\n\n### CRITICAL: the FIRST symptom is the verdict symptom\n\nDownstream automation (email notifier, dashboards, dedup) keys off the\nFIRST symptom's title matching `Triage verdict: \u003Cname> :: \u003Csignature>`.\nIf you emit a descriptively-titled symptom first (e.g.\n`\"worker1 lifecycle script execution failures across multiple nodes on my-cluster\"`)\nthe verdict is invisible to the pipeline and no email is sent.\n\nThis has been observed to fail in RCA runs. **The first\n`symptom` record you emit MUST have `title` beginning with\n`Triage verdict:`.** Descriptive titles are for the *second* and later\nsymptom records, which capture per-resource observations.\n\n### Few-shot examples of the first symptom record\n\nCopy the shape and titles exactly. Substitute your investigation's\nconcrete data.\n\n**Example A — Escalate for a recurring capacity fault:**\n\n```\ntitle: \"Triage verdict: Escalate — recurring fault pattern :: (worker4:Description: Failed to provision EC2 Instance in Cluster prod-01 and InstanceGroup worker4. FailureMessage: We currently do not have sufficient capacity to launch new ml.g5.8xlarge instances. Please try again.)\"\n\ndescription: |\n  Verdict: Escalate — recurring fault pattern\n\n  Summary:\n  Repeated capacity errors are failing to provision ml.g5.8xlarge for instance group \"worker4\" — 4 failed replacements in the last 24 hours. The likely cause is that on-demand capacity for ml.g5.8xlarge is unavailable in us-west-2 for this cluster and worker4 is not backed by a training plan or reserved capacity, so each Continuous-Provisioning retry hits the same capacity wall. Recommended: launch worker4 in an AZ\u002FRegion where you hold reserved capacity or a training plan for ml.g5.8xlarge, or switch to a SKU with availability (e.g. ml.g6.8xlarge); alternatively lower worker4's target count to stop the retry loop while you request a capacity increase.\n\n  What HyperPod is doing right now:\n  HyperPod is auto-retrying, but the same insufficient-capacity error has driven 4 replacements on worker4 in the last 24 hours. Continuous Provisioning is looping without progress; each new attempt hits the same on-demand capacity wall for ml.g5.8xlarge in us-west-2. Operator action is required — the pattern will not self-resolve.\n\n  Timeline (UTC):\n  2026-07-08T15:12:03Z  aws.sagemaker Cluster Event Error  Failed to provision EC2 Instance in worker4 (attempt 1)\n  2026-07-08T15:31:47Z  aws.sagemaker Cluster Event Error  Failed to provision EC2 Instance in worker4 (attempt 2)\n  2026-07-08T15:53:22Z  aws.sagemaker Cluster Event Error  Failed to provision EC2 Instance in worker4 (attempt 3)\n  2026-07-08T16:14:59Z  aws.sagemaker Cluster Event Error  Failed to provision EC2 Instance in worker4 (attempt 4)\n\n  Most recent event at:\n  2026-07-08T16:14:59Z\n```\n\n**Example B — Escalate for coordinated lifecycle-script failures across multiple instances:**\n\n```\ntitle: \"Triage verdict: Escalate — coordinated lifecycle-script failure :: (worker1:Description: Lifecycle scripts did not run successfully. Ensure the scripts exist in provided S3 path, are accessible, and run without errors.)\"\n\ndescription: |\n  Verdict: Escalate — coordinated lifecycle-script failure\n\n  Summary:\n  Every new worker1 instance on cluster my-cluster is failing bootstrap with the same lifecycle-script (LCS) execution error — 6 instances failed across two waves at 2026-07-08T19:21Z and 19:32Z. The likely cause is a bug or missing\u002Finaccessible file in the on_create script in the instance group's S3 path, so each freshly provisioned node hits the identical failure. Recommended: inspect the LCS log stream for one of the affected instances to find the failing command, then fix on_create.sh (or on_create_main.sh) in the cluster's S3 bucket; the retry loop clears on the next attempt once the script succeeds.\n\n  What HyperPod is doing right now:\n  HyperPod is retrying, but every new worker1 instance is failing bootstrap with the same LCS execution error. 3 instances (i-0aaaa1111bbbb2222, i-0cccc3333dddd4444, i-0eeee5555ffff6666) failed simultaneously at 2026-07-08T19:21Z, then 3 more at 2026-07-08T19:32Z with the same error. Continuous Provisioning + Automatic NodeRecovery will keep respawning these logical nodes at ~10-minute intervals until an operator fixes the LCS.\n\n  Timeline (UTC):\n  2026-07-08T19:20:01Z  aws.sagemaker Cluster Event Info  Instance lifecycle script execution for i-0aaaa1... has Started\n  2026-07-08T19:20:05Z  aws.sagemaker Cluster Event Info  Instance lifecycle script execution for i-0cccc3... has Started\n  2026-07-08T19:20:07Z  aws.sagemaker Cluster Event Info  Instance lifecycle script execution for i-0eeee5... has Started\n  2026-07-08T19:21:02Z  aws.sagemaker Cluster Event Error Lifecycle scripts did not run successfully (i-0aaaa1...)\n  2026-07-08T19:21:05Z  aws.sagemaker Cluster Event Error Lifecycle scripts did not run successfully (i-0cccc3...)\n  2026-07-08T19:21:08Z  aws.sagemaker Cluster Event Error Lifecycle scripts did not run successfully (i-0eeee5...)\n  2026-07-08T19:31:53Z  aws.sagemaker Cluster Event Error Lifecycle scripts did not run successfully (i-07777a...)\n  2026-07-08T19:32:04Z  aws.sagemaker Cluster Event Error Lifecycle scripts did not run successfully (i-08888b...)\n  2026-07-08T19:32:13Z  aws.sagemaker Cluster Event Error Lifecycle scripts did not run successfully (i-09999c...)\n\n  Most recent event at:\n  2026-07-08T19:32:13Z\n\n  Recommended actions (operator runs these):\n  Inspect the LCS log stream \u002Faws\u002Fsagemaker\u002FClusters\u002Fmy-cluster\u002F\u003Ccluster-id>\u002FLifecycleConfig\u002Fworker1\u002F\u003Cinstance-id> for any of the affected instances to identify the failing command. Fix on_create.sh (or on_create_main.sh) in s3:\u002F\u002F\u003Cyour-lcs-bucket>. Once fixed the retry loop will clear on its next attempt.\n\nrelated_resources: [\"HyperPod cluster my-cluster\", \"i-0aaaa1111bbbb2222\", \"i-0cccc3333dddd4444\", \"i-0eeee5555ffff6666\"]\n```\n\n**Example C — Monitor (first attempt in flight):**\n\n```\ntitle: \"Triage verdict: Monitor — first attempt :: (worker2:Description: Instance i-0abcdef1234567890 is unhealthy. HyperPod Health Monitoring Agent (HMA) has detected fault type NvidiaGPUUnhealthy on this node and is unhealthy. Repair action: Replace.)\"\n\ndescription: |\n  Verdict: Monitor — first attempt\n\n  Summary:\n  HMA flagged instance i-0abcdef1234567890 in instance group \"worker2\" as unhealthy (NvidiaGPUUnhealthy) and requested a Replace, and HyperPod has already started the first replacement attempt. The likely cause is a transient or isolated GPU fault on that single node, which HyperPod's automatic recovery is equipped to handle on its own. No operator action is needed right now — the replacement is expected to reach InService within 20-30 min; the next scheduled audit will re-check and emit a Resolved closure if it succeeds.\n\n  What HyperPod is doing right now:\n  HMA has flagged i-0abcdef1234567890 as unhealthy (NvidiaGPUUnhealthy) and requested Replace. HyperPod has started the replacement and this is the first attempt in the chain. Expected wall-clock: 20-30 min for the new instance to reach InService.\n\n  Timeline (UTC):\n  2026-07-08T18:45:12Z  aws.sagemaker Cluster Node Health Event  Instance i-0abcdef1234567890 unhealthy: NvidiaGPUUnhealthy\n  2026-07-08T18:45:15Z  aws.sagemaker Cluster Event Info         Instance deletion is starting as part of instance replacement\n\n  Most recent event at:\n  2026-07-08T18:45:15Z\n\n  Next re-check:\n  2026-07-08T19:15:15Z\n```\n\n**Example D — Suppress (audit found nothing):**\n\n```\ntitle: \"Triage verdict: Suppress — periodic audit, no open incidents\"\n\ndescription: |\n  Verdict: Suppress — periodic audit, no open incidents\n\n  What HyperPod is doing right now:\n  Scheduled audit at 2026-07-08T19:00Z; scanned the last 4 hours of cluster events. No fault events, no open Monitor chains, cluster status InService, all instance groups at target. Nothing to investigate.\n```\n\n### Anti-example — do NOT do this\n\nIf your first symptom looks like this, downstream automation is broken:\n\n```\ntitle: \"worker1 lifecycle script execution failures across multiple nodes on my-cluster\"    ← WRONG: missing \"Triage verdict:\" prefix\ndescription: \"HyperPod cluster my-cluster emitted coordinated lifecycle-script execution failures...\"\n```\n\nThe content is fine as a *second* symptom record. But the FIRST symptom\nmust be the verdict.\n\n**For `Suppress` verdict:** still emit the verdict symptom (so the\noperator can audit what was suppressed and why), but skip per-resource\nsymptoms and findings — there's nothing to root-cause.\n\n**For `Monitor` verdicts:** the verdict symptom's description tells\nthe human \"no action needed; HyperPod is recovering; expected\ncompletion by HH:MM UTC. You will be notified again only if the\nsituation changes.\" This is the key UX improvement — silence is bad;\n\"we're watching and here's why we're not alarming you\" is good.\n\n**For `Escalate` verdicts:** include explicit operator-runnable\nremediation in the verdict symptom's description, under a\n`Recommended actions (operator runs these):` heading. The agent\ncannot execute these; the operator must.\n\n**Confidence annotations:** for every material claim in the verdict\ndescription, prefix with one of `[direct]` (observed via API\u002Flog\noutput the agent itself ran), `[proxy]` (inferred from HMA\nclassification or a correlated source), or `[unverified]` (would need\non-node SSM or AWS-internal data the agent can't reach). This\nreplaces the separate \"Confidence\" section.\n\n**`[direct]` requires direct observation, not inference.** A claim\nlike \"all three replacements landed on the same physical host\" is\n**not** `[direct]` even when three identical fault signatures are\nobserved — the customer surface (NodeId, InstanceId, ENI, K8s node\nname) does not expose physical hardware identity. Such claims are\n`[unverified]` at best. Mislabeling inference as `[direct]` is a\nserious failure mode because operators trust the annotation to mean\n\"the agent saw this in the data.\"\n\n**Hypothesis discipline for recurring-pattern verdicts\n(rules 6–8).** When the verdict is one of `Escalate — recurring\nfault pattern`, `Escalate — fleet-wide instability`, or\n`Escalate — instance-group instability`, the verdict description\nMUST enumerate at least **two** competing hypotheses for the root\ncause, each labeled `[unverified]` or `[proxy]`, and each paired with\na discriminating operator action. Do not commit to a single root\ncause without `[direct]` evidence. The required hypothesis classes\nare:\n\n1. **Software \u002F workload** — the workload running on the IG triggers\n   the fault on whatever GPU it lands on (NCCL pattern, driver \u002F CUDA\n   version, application code path). Discriminator: change the\n   workload, or move the IG to a different node and see if the fault\n   follows.\n2. **Infrastructure path** — an EFA fabric path, leaf switch, or\n   shared network resource surfaces as GPU-level errors on workloads\n   that hit it. Discriminator: move the IG to a different subnet \u002F AZ.\n3. **Statistical hardware** — a bad batch of the same SKU is over-\n   represented in the capacity pool. Discriminator: open an AWS\n   Support case with the fault signature requesting hardware\n   exclusion, or wait + retry later from a different time\u002Fpool.\n\nThe verdict should NOT include \"every replacement is landing on the\nsame physical hardware\" as a stated cause — that's the explanation\n**operators are conditioned to expect from on-prem clusters**, but\non HyperPod the EC2 instance is owned by the service account, the\nunderlying physical host is not exposed on the customer surface, and\nEC2 placement is non-deterministic per replacement. Read the\n\"Recurring fault signature does NOT prove physical-host affinity\"\nsection in [references\u002Fhyperpod-mental-model.md](references\u002Fhyperpod-mental-model.md)\nbefore authoring this part of the verdict.\n\n**The GPU UUID check is the only way to confirm or refute\nphysical-host affinity, and it requires SSM (operator-only).** The\nverdict's \"Recommended actions\" section MUST include this check as\nan explicit operator step whenever a recurring-pattern verdict is\nemitted (rules 6–8). The wording should be:\n\n```\nN. Verify or refute \"same physical GPU\" by capturing GPU UUIDs.\n   The agent cannot run this check (requires SSM, which is outside\n   the DevOps Agent permission guardrail). For each affected\n   instance ID, run:\n\n       aws ssm start-session \\\n         --target sagemaker-cluster:\u003Ccluster-id>_\u003Cgroup>-\u003Cinstance-id> \\\n         --document-name AWS-StartNonInteractiveCommand \\\n         --parameters '{\"command\":[\"nvidia-smi -L\"]}'\n\n   Compare the UUID strings across the affected instances. If they\n   match, the same physical GPU is being recycled — that elevates\n   the \"statistical hardware\" hypothesis to [direct] evidence and\n   strengthens the case for an AWS Support exclusion request. If\n   they differ, \"same physical hardware\" is RULED OUT and the\n   investigation should pivot to the software\u002Fworkload and\n   infrastructure-path hypotheses instead.\n```\n\nReplace `\u003Ccluster-id>`, `\u003Cgroup>`, `\u003Cinstance-id>` with the actual\nvalues from the affected instances in the timeline. List each\ninstance separately so the operator can run the commands in parallel.\n\n## Inputs the skill expects from the trigger\n\nThe webhook payload built by the bridge Lambda carries these fields. Per the\nflattening note in Phase 1 step 8, read them from the task `description` at\nruntime — the `data.metadata.*` \u002F `data.originalEvent.detail.*` paths below\nname the logical source:\n\n- `data.metadata.clusterName` — HyperPod cluster name (required)\n- `data.metadata.detailType` — `Cluster State Change` \u002F\n  `Cluster Node Health Event` \u002F `Cluster Event`\n- `data.originalEvent.detail.InstanceId` — affected instance (for node\n  health events)\n- `data.originalEvent.detail.EventDetails.InstanceGroupName` \u002F\n  `InstanceId` (for cluster events)\n\nIf the cluster name is missing, abort with \"skill cannot run without a\nHyperPod cluster name — check the webhook bridge's payload mapping.\"\n\n## What this skill does NOT do\n\n- **It does not SSH or run SSM on nodes.** The DevOps Agent permission\n  guardrail blocks `ssm:StartSession` \u002F `ssm:SendCommand`. When on-node\n  evidence is needed (Xid in `dmesg`, DCGM counters, kubelet journal),\n  the skill emits a recommendation that includes the exact SSM command\n  for the operator to run, never executes it.\n- **It does not replace nodes.** `batch-replace-cluster-nodes` is a\n  state-changing call; the skill recommends it but never invokes it.\n- **It does not modify Slurm or EKS state.** No `scontrol update`, no\n  `kubectl cordon`, no label edits.\n\n## Tools used (in-guardrail)\n\n- `sagemaker:DescribeCluster`, `ListClusterNodes`, `DescribeClusterNode`,\n  `ListClusterEvents`\n- `logs:FilterLogEvents`, `logs:DescribeLogStreams` on\n  `\u002Faws\u002Fsagemaker\u002FClusters\u002F*`\n- `eks:DescribeCluster` (read-only) and `kubectl get\u002Fdescribe`\n  (via the `AWS::EKS::AccessEntry` created by the CloudFormation stack)\n- `cloudwatch:GetMetricData` on `ClusterAgent` and\n  `SagemakerHealthMonitoringAgent` namespaces — useful for time-series\n  confirmation when individual log events are sparse\n",{"data":49,"body":54},{"name":4,"description":6,"metadata":50},{"version":51,"agent_types":52},"0.6.1",[53],"INCIDENT_RCA",{"type":55,"children":56},"root",[57,66,89,136,157,174,192,199,228,241,246,274,279,285,292,297,441,446,452,462,492,1302,1308,1313,1333,1352,1358,1363,1372,1377,1383,1395,1400,1409,1414,1527,1590,1599,1637,1643,1655,1733,1738,1744,1749,1816,1822,1830,1860,1879,1884,1891,1918,1964,1981,2295,2311,2432,2442,2459,2465,2477,2501,2533,2552,3059,3080,3123,3147,3198,3208,3214,3250,3255,3386,3394,3815,3821,3842,3883,3889,3894,3902,3911,3919,3928,3936,3945,3953,3962,3968,3973,3982,3993,4010,4026,4049,4080,4121,4178,4211,4229,4239,4248,4275,4281,4308,4382,4387,4393,4472,4478,4588],{"type":58,"tag":59,"props":60,"children":62},"element","h1",{"id":61},"hyperpod-incident-rca-skill",[63],{"type":64,"value":65},"text","HyperPod incident RCA skill",{"type":58,"tag":67,"props":68,"children":69},"p",{},[70,72,78,80,87],{"type":64,"value":71},"This skill runs at the ",{"type":58,"tag":73,"props":74,"children":75},"strong",{},[76],{"type":64,"value":77},"INCIDENT_RCA stage",{"type":64,"value":79},", after DevOps Agent's\ntriage stage decided to ",{"type":58,"tag":81,"props":82,"children":84},"code",{"className":83},[],[85],{"type":64,"value":86},"PROCEED",{"type":64,"value":88}," with a full investigation. By the\ntime this skill loads, the task already has a primary investigation\nslot and the agent has full AWS API access.",{"type":58,"tag":67,"props":90,"children":91},{},[92,97,99,105,107,113,115,121,122,127,129,134],{"type":58,"tag":73,"props":93,"children":94},{},[95],{"type":64,"value":96},"The complementary triage skill",{"type":64,"value":98}," ",{"type":58,"tag":81,"props":100,"children":102},{"className":101},[],[103],{"type":64,"value":104},"hyperpod-incident-triage",{"type":64,"value":106}," runs at\nthe INCIDENT_TRIAGE stage BEFORE this skill, and decides whether an\nincoming event should be ",{"type":58,"tag":81,"props":108,"children":110},{"className":109},[],[111],{"type":64,"value":112},"LINKED",{"type":64,"value":114}," \u002F ",{"type":58,"tag":81,"props":116,"children":118},{"className":117},[],[119],{"type":64,"value":120},"SKIPPED",{"type":64,"value":114},{"type":58,"tag":81,"props":123,"children":125},{"className":124},[],[126],{"type":64,"value":86},{"type":64,"value":128}," using concise\ndeclarative correlation rules (keep distinct fault types on the same\ninstance group separate; skip concurrent periodic audits). Separately,\nfor periodic audits, the ",{"type":58,"tag":73,"props":130,"children":131},{},[132],{"type":64,"value":133},"audit Lambda gates volume",{"type":64,"value":135}," — it inspects\nKubernetes state itself and only invokes an investigation when a real\nissue is present (plus a daily heartbeat), so a \"periodic audit\"\ninvestigation reaching this skill already corresponds to something\nworth looking at.",{"type":58,"tag":67,"props":137,"children":138},{},[139,141,147,149,155],{"type":64,"value":140},"When this skill DOES run, it does triage-like classification and\nroot-cause analysis in one pass because they need the same evidence:\nsnapshot data\n(",{"type":58,"tag":81,"props":142,"children":144},{"className":143},[],[145],{"type":64,"value":146},"describe-cluster",{"type":64,"value":148},", ",{"type":58,"tag":81,"props":150,"children":152},{"className":151},[],[153],{"type":64,"value":154},"describe-cluster-node",{"type":64,"value":156},") alone cannot distinguish\n\"HyperPod is auto-recovering\" from \"HyperPod has given up\" — only the\ncross-source timeline can.",{"type":58,"tag":67,"props":158,"children":159},{},[160,165,167,172],{"type":58,"tag":73,"props":161,"children":162},{},[163],{"type":64,"value":164},"Operating policy.",{"type":64,"value":166}," Read-only. Never call a state-changing API.\nPresent every action that mutates anything (operator-driven replacement,\nSSM session, label change) as a ",{"type":58,"tag":73,"props":168,"children":169},{},[170],{"type":64,"value":171},"Recommended action (operator runs\nthis)",{"type":64,"value":173}," block. The customer's operator decides whether to execute it.",{"type":58,"tag":67,"props":175,"children":176},{},[177,182,184,190],{"type":58,"tag":73,"props":178,"children":179},{},[180],{"type":64,"value":181},"Required reading.",{"type":64,"value":183}," Open\n",{"type":58,"tag":185,"props":186,"children":188},"a",{"href":187},"references\u002Fhyperpod-mental-model.md",[189],{"type":64,"value":187},{"type":64,"value":191},"\nin parallel with this skill. The mental-model doc is the ground truth\nfor what each HyperPod signal means; this skill is the decision\nprocedure that consumes those meanings.",{"type":58,"tag":193,"props":194,"children":196},"h2",{"id":195},"why-one-skill-not-two",[197],{"type":64,"value":198},"Why one skill, not two",{"type":58,"tag":67,"props":200,"children":201},{},[202,204,210,212,218,220,226],{"type":64,"value":203},"A single failed instance can vanish from ",{"type":58,"tag":81,"props":205,"children":207},{"className":206},[],[208],{"type":64,"value":209},"list-cluster-nodes",{"type":64,"value":211}," between\nretry attempts, and HyperPod may auto-retry from ",{"type":58,"tag":81,"props":213,"children":215},{"className":214},[],[216],{"type":64,"value":217},"Failed",{"type":64,"value":219}," status —\nneither is a terminal escalation signal on its own. Distinguishing\n\"still retrying within budget\" from \"stuck\" requires the full\n",{"type":58,"tag":81,"props":221,"children":223},{"className":222},[],[224],{"type":64,"value":225},"list-cluster-events",{"type":64,"value":227}," chain, the HMA CloudWatch stream, and the cluster\nstatus all aligned on a wall-clock timeline. Building a separate\n\"triage\" skill that decides without that data would mean re-deciding\nincorrectly on every event.",{"type":58,"tag":193,"props":229,"children":231},{"id":230},"relationship-to-hyperpod-incident-triage-and-the-audit-lambda",[232,234,239],{"type":64,"value":233},"Relationship to ",{"type":58,"tag":81,"props":235,"children":237},{"className":236},[],[238],{"type":64,"value":104},{"type":64,"value":240}," and the audit Lambda",{"type":58,"tag":67,"props":242,"children":243},{},[244],{"type":64,"value":245},"Two things run before this RCA skill and mean it does NOT need to\nre-implement duplicate\u002Fstale-audit suppression itself:",{"type":58,"tag":247,"props":248,"children":249},"ul",{},[250,264],{"type":58,"tag":251,"props":252,"children":253},"li",{},[254,262],{"type":58,"tag":73,"props":255,"children":256},{},[257],{"type":58,"tag":81,"props":258,"children":260},{"className":259},[],[261],{"type":64,"value":104},{"type":64,"value":263}," makes the LINK \u002F SKIP \u002F PROCEED\ndecision with concise declarative rules (same fault on the same\ncomponent links; different fault types on the same instance group\nstay separate; concurrent periodic audits are skipped).",{"type":58,"tag":251,"props":265,"children":266},{},[267,272],{"type":58,"tag":73,"props":268,"children":269},{},[270],{"type":64,"value":271},"The periodic-audit Lambda gates volume",{"type":64,"value":273},": for scheduled audits it\ninspects Kubernetes state (CrashLoopBackOff \u002F NotReady) and POSTs the\nwebhook only when a real issue exists — so a periodic-audit\ninvestigation reaching this skill already corresponds to a real\nfinding, not an idle poll. HyperPod control-plane faults arrive via\nthe event-driven webhook bridge, not the audit.",{"type":58,"tag":67,"props":275,"children":276},{},[277],{"type":64,"value":278},"RCA still does the full Phase 1 gather (including Pod\u002FNode inspection\nand Phase 3d threshold checks) because the reasoning stage needs the\nfull evidence. Earlier RCA \"stale-evidence\" rules (3 \u002F 3b) were removed\nand are not reintroduced — that concern is handled upstream now.",{"type":58,"tag":193,"props":280,"children":282},{"id":281},"workflow",[283],{"type":64,"value":284},"Workflow",{"type":58,"tag":286,"props":287,"children":289},"h3",{"id":288},"trigger-modes",[290],{"type":64,"value":291},"Trigger modes",{"type":58,"tag":67,"props":293,"children":294},{},[295],{"type":64,"value":296},"The skill is loaded from two trigger sources, and Phase 1 \u002F Phase 3\nbehavior differs slightly between them:",{"type":58,"tag":298,"props":299,"children":300},"table",{},[301,325],{"type":58,"tag":302,"props":303,"children":304},"thead",{},[305],{"type":58,"tag":306,"props":307,"children":308},"tr",{},[309,315,320],{"type":58,"tag":310,"props":311,"children":312},"th",{},[313],{"type":64,"value":314},"Mode",{"type":58,"tag":310,"props":316,"children":317},{},[318],{"type":64,"value":319},"Trigger",{"type":58,"tag":310,"props":321,"children":322},{},[323],{"type":64,"value":324},"Scope",{"type":58,"tag":326,"props":327,"children":328},"tbody",{},[329,367],{"type":58,"tag":306,"props":330,"children":331},{},[332,341,362],{"type":58,"tag":333,"props":334,"children":335},"td",{},[336],{"type":58,"tag":73,"props":337,"children":338},{},[339],{"type":64,"value":340},"Incident mode",{"type":58,"tag":333,"props":342,"children":343},{},[344,346,352,354,360],{"type":64,"value":345},"Webhook fire from the bridge Lambda (an EventBridge HyperPod event passed the noise filter). The investigation context carries ",{"type":58,"tag":81,"props":347,"children":349},{"className":348},[],[350],{"type":64,"value":351},"clusterName",{"type":64,"value":353}," and usually an ",{"type":58,"tag":81,"props":355,"children":357},{"className":356},[],[358],{"type":64,"value":359},"instanceId",{"type":64,"value":361},".",{"type":58,"tag":333,"props":363,"children":364},{},[365],{"type":64,"value":366},"Focused on the specific incident referenced in the trigger.",{"type":58,"tag":306,"props":368,"children":369},{},[370,378,413],{"type":58,"tag":333,"props":371,"children":372},{},[373],{"type":58,"tag":73,"props":374,"children":375},{},[376],{"type":64,"value":377},"Audit mode",{"type":58,"tag":333,"props":379,"children":380},{},[381,383,388,390,395,397,403,405,411],{"type":64,"value":382},"A synthesized webhook POST from the periodic-audit Lambda. Unlike a raw scheduled trigger, this payload ",{"type":58,"tag":73,"props":384,"children":385},{},[386],{"type":64,"value":387},"does",{"type":64,"value":389}," carry context: the description names the cluster (",{"type":58,"tag":81,"props":391,"children":393},{"className":392},[],[394],{"type":64,"value":351},{"type":64,"value":396},") and, when the Lambda-side checks fired, lists ",{"type":58,"tag":81,"props":398,"children":400},{"className":399},[],[401],{"type":64,"value":402},"detectedIssues",{"type":64,"value":404}," and a ",{"type":58,"tag":81,"props":406,"children":408},{"className":407},[],[409],{"type":64,"value":410},"k8sChecks",{"type":64,"value":412}," config line.",{"type":58,"tag":333,"props":414,"children":415},{},[416,418,424,426,432,434,439],{"type":64,"value":417},"Confirm the detected issues, then scan the named cluster for any in-flight or recently-resolved fault chains and re-classify each. Catches: (1) ",{"type":58,"tag":81,"props":419,"children":421},{"className":420},[],[422],{"type":64,"value":423},"Monitor",{"type":64,"value":425}," incidents that have now succeeded (emit ",{"type":58,"tag":81,"props":427,"children":429},{"className":428},[],[430],{"type":64,"value":431},"Resolved",{"type":64,"value":433},"), (2) ",{"type":58,"tag":81,"props":435,"children":437},{"className":436},[],[438],{"type":64,"value":423},{"type":64,"value":440}," incidents stuck past their re-check budget (escalate), (3) recurring patterns that didn't trigger an EventBridge event recently but persist statistically.",{"type":58,"tag":67,"props":442,"children":443},{},[444],{"type":64,"value":445},"In both modes the same Phase 1 \u002F Phase 2 \u002F Phase 3 \u002F Phase 4 logic\napplies. The differences are noted inline below.",{"type":58,"tag":286,"props":447,"children":449},{"id":448},"phase-1-gather-run-in-parallel",[450],{"type":64,"value":451},"Phase 1 — Gather (run in parallel)",{"type":58,"tag":67,"props":453,"children":454},{},[455,457],{"type":64,"value":456},"For the incident referenced in the trigger event (cluster name + optional\ninstance id), collect these in parallel. ",{"type":58,"tag":73,"props":458,"children":459},{},[460],{"type":64,"value":461},"Do not stop on a single\nsignal — gather all of them before classifying.",{"type":58,"tag":67,"props":463,"children":464},{},[465,469,471,475,477,483,485,490],{"type":58,"tag":73,"props":466,"children":467},{},[468],{"type":64,"value":377},{"type":64,"value":470},": the trigger payload ",{"type":58,"tag":73,"props":472,"children":473},{},[474],{"type":64,"value":387},{"type":64,"value":476}," carry the cluster name\nin its description (the periodic-audit Lambda synthesizes it), so use\nthat. Only if the name is genuinely absent, fall back to the cluster(s)\nreachable from this Agent Space's AWS account association (typically\njust one HyperPod cluster — discover it via ",{"type":58,"tag":81,"props":478,"children":480},{"className":479},[],[481],{"type":64,"value":482},"sagemaker list-clusters",{"type":64,"value":484},").\nRun the full Phase 1 gather for the cluster. Then in Phase 3, classify\nper open fault chain found in the cluster-events window (plus any\n",{"type":58,"tag":81,"props":486,"children":488},{"className":487},[],[489],{"type":64,"value":402},{"type":64,"value":491}," the payload already carries), not per single trigger.",{"type":58,"tag":493,"props":494,"children":495},"ol",{},[496,557,595,649,683,701,749,782],{"type":58,"tag":251,"props":497,"children":498},{},[499,504,506,512,514,520,521,527,528,534,536,542,543,549,550,556],{"type":58,"tag":73,"props":500,"children":501},{},[502],{"type":64,"value":503},"Cluster state",{"type":64,"value":505},": ",{"type":58,"tag":81,"props":507,"children":509},{"className":508},[],[510],{"type":64,"value":511},"aws sagemaker describe-cluster --cluster-name \u003Cname>",{"type":64,"value":513},"\n— current ",{"type":58,"tag":81,"props":515,"children":517},{"className":516},[],[518],{"type":64,"value":519},"ClusterStatus",{"type":64,"value":148},{"type":58,"tag":81,"props":522,"children":524},{"className":523},[],[525],{"type":64,"value":526},"NodeRecovery",{"type":64,"value":148},{"type":58,"tag":81,"props":529,"children":531},{"className":530},[],[532],{"type":64,"value":533},"Orchestrator",{"type":64,"value":535}," (Eks vs.\nSlurm), ",{"type":58,"tag":81,"props":537,"children":539},{"className":538},[],[540],{"type":64,"value":541},"InstanceGroups[].CurrentCount",{"type":64,"value":114},{"type":58,"tag":81,"props":544,"children":546},{"className":545},[],[547],{"type":64,"value":548},"TargetCount",{"type":64,"value":114},{"type":58,"tag":81,"props":551,"children":553},{"className":552},[],[554],{"type":64,"value":555},"Status",{"type":64,"value":361},{"type":58,"tag":251,"props":558,"children":559},{},[560,565,566,572,574,579,581,593],{"type":58,"tag":73,"props":561,"children":562},{},[563],{"type":64,"value":564},"Node inventory",{"type":64,"value":505},{"type":58,"tag":81,"props":567,"children":569},{"className":568},[],[570],{"type":64,"value":571},"aws sagemaker list-cluster-nodes --cluster-name \u003Cname>",{"type":64,"value":573},"\nplus ",{"type":58,"tag":81,"props":575,"children":577},{"className":576},[],[578],{"type":64,"value":154},{"type":64,"value":580}," for any instance id named in the trigger\nevent. ",{"type":58,"tag":73,"props":582,"children":583},{},[584,586,591],{"type":64,"value":585},"An instance id mentioned in the event that is NOT in\n",{"type":58,"tag":81,"props":587,"children":589},{"className":588},[],[590],{"type":64,"value":209},{"type":64,"value":592}," is a signal, not an error",{"type":64,"value":594}," — the node may have\nbeen removed mid-retry.",{"type":58,"tag":251,"props":596,"children":597},{},[598,603,604,610,612,617,619,624,626],{"type":58,"tag":73,"props":599,"children":600},{},[601],{"type":64,"value":602},"Cluster events chain",{"type":64,"value":505},{"type":58,"tag":81,"props":605,"children":607},{"className":606},[],[608],{"type":64,"value":609},"aws sagemaker list-cluster-events --cluster-name \u003Cname>",{"type":64,"value":611},",\npaginated to ≥500 entries or until events reach ",{"type":58,"tag":73,"props":613,"children":614},{},[615],{"type":64,"value":616},"7 days",{"type":64,"value":618}," back,\nwhichever comes first. This is the ",{"type":58,"tag":73,"props":620,"children":621},{},[622],{"type":64,"value":623},"canonical record of replacement\nattempts including failed ones",{"type":64,"value":625}," and survives nodes disappearing\nfrom the node list. The wider 7-day window also feeds the recurring-\npattern classification rules in Phase 3 — don't shorten the lookback\neven when the trigger event is recent.",{"type":58,"tag":247,"props":627,"children":628},{},[629],{"type":58,"tag":251,"props":630,"children":631},{},[632,634,639,641,647],{"type":64,"value":633},"Available on EKS clusters and on Slurm clusters with ",{"type":58,"tag":73,"props":635,"children":636},{},[637],{"type":64,"value":638},"Continuous\nProvisioning",{"type":64,"value":640}," enabled. If the API returns\n",{"type":58,"tag":81,"props":642,"children":644},{"className":643},[],[645],{"type":64,"value":646},"ValidationException",{"type":64,"value":648},"\u002Fequivalent on a Slurm cluster without CP,\nnote \"event timeline unavailable; confidence degraded\" in the\nfinal report and proceed with the remaining sources.",{"type":58,"tag":251,"props":650,"children":651},{},[652,657,659,665,667,673,675,681],{"type":58,"tag":73,"props":653,"children":654},{},[655],{"type":64,"value":656},"HMA CloudWatch stream",{"type":64,"value":658},": log group\n",{"type":58,"tag":81,"props":660,"children":662},{"className":661},[],[663],{"type":64,"value":664},"\u002Faws\u002Fsagemaker\u002FClusters\u002F\u003CNAME>\u002F\u003CCLUSTER_ID>",{"type":64,"value":666},", streams matching\n",{"type":58,"tag":81,"props":668,"children":670},{"className":669},[],[671],{"type":64,"value":672},"SagemakerHealthMonitoringAgent\u002F*",{"type":64,"value":674}," (filter to the affected instance\ngroup \u002F instance if known). Filter for ",{"type":58,"tag":81,"props":676,"children":678},{"className":677},[],[679],{"type":64,"value":680},"HealthMonitoringAgentDetectionEvent",{"type":64,"value":682},"\nentries and any Xid \u002F DCGM \u002F EFA \u002F OOM messages within the same\nwindow as Phase 1 step 3.",{"type":58,"tag":251,"props":684,"children":685},{},[686,691,693,699],{"type":58,"tag":73,"props":687,"children":688},{},[689],{"type":64,"value":690},"Lifecycle script stream",{"type":64,"value":692}," (only if a replace attempt is in the\ntimeline): same log group, streams matching ",{"type":58,"tag":81,"props":694,"children":696},{"className":695},[],[697],{"type":64,"value":698},"LifecycleConfig\u002F*",{"type":64,"value":700}," for\nthe affected instance group \u002F instance id. Look for non-zero exit,\ntimeout, S3 \u002F IAM errors.",{"type":58,"tag":251,"props":702,"children":703},{},[704,709,711,717,719,725,726,732,734,740,741,747],{"type":58,"tag":73,"props":705,"children":706},{},[707],{"type":64,"value":708},"EKS node state",{"type":64,"value":710}," (EKS only): ",{"type":58,"tag":81,"props":712,"children":714},{"className":713},[],[715],{"type":64,"value":716},"kubectl get node \u003Cname> -o yaml",{"type":64,"value":718}," for\nthe affected node (if still present). Surface\n",{"type":58,"tag":81,"props":720,"children":722},{"className":721},[],[723],{"type":64,"value":724},"sagemaker.amazonaws.com\u002Fnode-health-status",{"type":64,"value":148},{"type":58,"tag":81,"props":727,"children":729},{"className":728},[],[730],{"type":64,"value":731},"fault-types",{"type":64,"value":733},",\n",{"type":58,"tag":81,"props":735,"children":737},{"className":736},[],[738],{"type":64,"value":739},"fault-reasons",{"type":64,"value":148},{"type":58,"tag":81,"props":742,"children":744},{"className":743},[],[745],{"type":64,"value":746},"fault-details",{"type":64,"value":748}," labels\u002Fannotations and any taints.",{"type":58,"tag":251,"props":750,"children":751},{},[752,757,759,764,766,772,774,780],{"type":58,"tag":73,"props":753,"children":754},{},[755],{"type":64,"value":756},"Slurm node state",{"type":64,"value":758}," (Slurm only, if SSM not required): use what's\nreachable from the control plane — ",{"type":58,"tag":81,"props":760,"children":762},{"className":761},[],[763],{"type":64,"value":154},{"type":64,"value":765}," gives\nmost of what we need; deep ",{"type":58,"tag":81,"props":767,"children":769},{"className":768},[],[770],{"type":64,"value":771},"scontrol",{"type":64,"value":773},"\u002F",{"type":58,"tag":81,"props":775,"children":777},{"className":776},[],[778],{"type":64,"value":779},"sinfo",{"type":64,"value":781}," requires SSM and is\nout of scope here.",{"type":58,"tag":251,"props":783,"children":784},{},[785,790,792,797,799,804,806,812,814,854,858,860,863,865,870,872,877,879,1202,1205,1210,1212,1218,1220,1225,1227,1237,1240,1242,1245,1247,1253,1254,1260,1261,1267,1268,1274,1275,1281,1282,1288,1289,1292,1294,1300],{"type":58,"tag":73,"props":786,"children":787},{},[788],{"type":64,"value":789},"Kubernetes state — MANDATORY in audit mode on EKS clusters.",{"type":64,"value":791},"\nYou MUST execute both of the following before entering Phase 2,\neven if the trigger payload has no ",{"type":58,"tag":81,"props":793,"children":795},{"className":794},[],[796],{"type":64,"value":410},{"type":64,"value":798}," block, even if\nearlier gather steps already suggest a plausible verdict, and\neven if ",{"type":58,"tag":81,"props":800,"children":802},{"className":801},[],[803],{"type":64,"value":225},{"type":64,"value":805}," returned a rich history. Live\nPod\u002FNode state is the source of truth for \"what is broken ",{"type":58,"tag":807,"props":808,"children":809},"em",{},[810],{"type":64,"value":811},"right\nnow",{"type":64,"value":813},"\" — the wider event history is context, not a substitute.",{"type":58,"tag":247,"props":815,"children":816},{},[817,828],{"type":58,"tag":251,"props":818,"children":819},{},[820,826],{"type":58,"tag":81,"props":821,"children":823},{"className":822},[],[824],{"type":64,"value":825},"kubectl get pods -A -o json",{"type":64,"value":827}," — full cluster Pod state.",{"type":58,"tag":251,"props":829,"children":830},{},[831,837,839,845,847,853],{"type":58,"tag":81,"props":832,"children":834},{"className":833},[],[835],{"type":64,"value":836},"kubectl get nodes -o json",{"type":64,"value":838}," — full Node state including\n",{"type":58,"tag":81,"props":840,"children":842},{"className":841},[],[843],{"type":64,"value":844},"status.conditions[]",{"type":64,"value":846}," and their ",{"type":58,"tag":81,"props":848,"children":850},{"className":849},[],[851],{"type":64,"value":852},"lastTransitionTime",{"type":64,"value":361},{"type":58,"tag":855,"props":856,"children":857},"br",{},[],{"type":64,"value":859},"Skip this step only if: (a) the cluster's orchestrator is Slurm\n(not EKS), OR (b) the trigger is incident-mode (webhook-triggered\nfor a specific fault; Pod\u002FNode scan belongs in periodic audit).",{"type":58,"tag":855,"props":861,"children":862},{},[],{"type":64,"value":864},"The trigger's ",{"type":58,"tag":81,"props":866,"children":868},{"className":867},[],[869],{"type":64,"value":410},{"type":64,"value":871}," block supplies ",{"type":58,"tag":73,"props":873,"children":874},{},[875],{"type":64,"value":876},"thresholds and\nnamespace filters",{"type":64,"value":878}," used by Phase 3d — not a gate on whether\nkubectl runs. Fields:",{"type":58,"tag":880,"props":881,"children":886},"pre",{"className":882,"code":883,"language":884,"meta":885,"style":885},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","{\n  \"enabled\": true,\n  \"crashLoopMinRestarts\": 5,\n  \"crashLoopRecencyMinutes\": 15,\n  \"notReadyNodePercentThreshold\": 10,\n  \"notReadyDurationMinutes\": 15,\n  \"ignoreNamespaces\": [\"kube-public\", \"kube-node-lease\"],\n  \"systemNamespaces\": [\"kube-system\", \"aws-hyperpod\", \"amazon-cloudwatch\"]\n}\n","json","",[887],{"type":58,"tag":81,"props":888,"children":889},{"__ignoreMap":885},[890,902,932,963,993,1023,1052,1116,1193],{"type":58,"tag":891,"props":892,"children":895},"span",{"class":893,"line":894},"line",1,[896],{"type":58,"tag":891,"props":897,"children":899},{"style":898},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[900],{"type":64,"value":901},"{\n",{"type":58,"tag":891,"props":903,"children":905},{"class":893,"line":904},2,[906,911,917,922,927],{"type":58,"tag":891,"props":907,"children":908},{"style":898},[909],{"type":64,"value":910},"  \"",{"type":58,"tag":891,"props":912,"children":914},{"style":913},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[915],{"type":64,"value":916},"enabled",{"type":58,"tag":891,"props":918,"children":919},{"style":898},[920],{"type":64,"value":921},"\"",{"type":58,"tag":891,"props":923,"children":924},{"style":898},[925],{"type":64,"value":926},":",{"type":58,"tag":891,"props":928,"children":929},{"style":898},[930],{"type":64,"value":931}," true,\n",{"type":58,"tag":891,"props":933,"children":935},{"class":893,"line":934},3,[936,940,945,949,953,959],{"type":58,"tag":891,"props":937,"children":938},{"style":898},[939],{"type":64,"value":910},{"type":58,"tag":891,"props":941,"children":942},{"style":913},[943],{"type":64,"value":944},"crashLoopMinRestarts",{"type":58,"tag":891,"props":946,"children":947},{"style":898},[948],{"type":64,"value":921},{"type":58,"tag":891,"props":950,"children":951},{"style":898},[952],{"type":64,"value":926},{"type":58,"tag":891,"props":954,"children":956},{"style":955},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[957],{"type":64,"value":958}," 5",{"type":58,"tag":891,"props":960,"children":961},{"style":898},[962],{"type":64,"value":733},{"type":58,"tag":891,"props":964,"children":966},{"class":893,"line":965},4,[967,971,976,980,984,989],{"type":58,"tag":891,"props":968,"children":969},{"style":898},[970],{"type":64,"value":910},{"type":58,"tag":891,"props":972,"children":973},{"style":913},[974],{"type":64,"value":975},"crashLoopRecencyMinutes",{"type":58,"tag":891,"props":977,"children":978},{"style":898},[979],{"type":64,"value":921},{"type":58,"tag":891,"props":981,"children":982},{"style":898},[983],{"type":64,"value":926},{"type":58,"tag":891,"props":985,"children":986},{"style":955},[987],{"type":64,"value":988}," 15",{"type":58,"tag":891,"props":990,"children":991},{"style":898},[992],{"type":64,"value":733},{"type":58,"tag":891,"props":994,"children":996},{"class":893,"line":995},5,[997,1001,1006,1010,1014,1019],{"type":58,"tag":891,"props":998,"children":999},{"style":898},[1000],{"type":64,"value":910},{"type":58,"tag":891,"props":1002,"children":1003},{"style":913},[1004],{"type":64,"value":1005},"notReadyNodePercentThreshold",{"type":58,"tag":891,"props":1007,"children":1008},{"style":898},[1009],{"type":64,"value":921},{"type":58,"tag":891,"props":1011,"children":1012},{"style":898},[1013],{"type":64,"value":926},{"type":58,"tag":891,"props":1015,"children":1016},{"style":955},[1017],{"type":64,"value":1018}," 10",{"type":58,"tag":891,"props":1020,"children":1021},{"style":898},[1022],{"type":64,"value":733},{"type":58,"tag":891,"props":1024,"children":1026},{"class":893,"line":1025},6,[1027,1031,1036,1040,1044,1048],{"type":58,"tag":891,"props":1028,"children":1029},{"style":898},[1030],{"type":64,"value":910},{"type":58,"tag":891,"props":1032,"children":1033},{"style":913},[1034],{"type":64,"value":1035},"notReadyDurationMinutes",{"type":58,"tag":891,"props":1037,"children":1038},{"style":898},[1039],{"type":64,"value":921},{"type":58,"tag":891,"props":1041,"children":1042},{"style":898},[1043],{"type":64,"value":926},{"type":58,"tag":891,"props":1045,"children":1046},{"style":955},[1047],{"type":64,"value":988},{"type":58,"tag":891,"props":1049,"children":1050},{"style":898},[1051],{"type":64,"value":733},{"type":58,"tag":891,"props":1053,"children":1055},{"class":893,"line":1054},7,[1056,1060,1065,1069,1073,1078,1082,1088,1092,1097,1102,1107,1111],{"type":58,"tag":891,"props":1057,"children":1058},{"style":898},[1059],{"type":64,"value":910},{"type":58,"tag":891,"props":1061,"children":1062},{"style":913},[1063],{"type":64,"value":1064},"ignoreNamespaces",{"type":58,"tag":891,"props":1066,"children":1067},{"style":898},[1068],{"type":64,"value":921},{"type":58,"tag":891,"props":1070,"children":1071},{"style":898},[1072],{"type":64,"value":926},{"type":58,"tag":891,"props":1074,"children":1075},{"style":898},[1076],{"type":64,"value":1077}," [",{"type":58,"tag":891,"props":1079,"children":1080},{"style":898},[1081],{"type":64,"value":921},{"type":58,"tag":891,"props":1083,"children":1085},{"style":1084},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[1086],{"type":64,"value":1087},"kube-public",{"type":58,"tag":891,"props":1089,"children":1090},{"style":898},[1091],{"type":64,"value":921},{"type":58,"tag":891,"props":1093,"children":1094},{"style":898},[1095],{"type":64,"value":1096},",",{"type":58,"tag":891,"props":1098,"children":1099},{"style":898},[1100],{"type":64,"value":1101}," \"",{"type":58,"tag":891,"props":1103,"children":1104},{"style":1084},[1105],{"type":64,"value":1106},"kube-node-lease",{"type":58,"tag":891,"props":1108,"children":1109},{"style":898},[1110],{"type":64,"value":921},{"type":58,"tag":891,"props":1112,"children":1113},{"style":898},[1114],{"type":64,"value":1115},"],\n",{"type":58,"tag":891,"props":1117,"children":1119},{"class":893,"line":1118},8,[1120,1124,1129,1133,1137,1141,1145,1150,1154,1158,1162,1167,1171,1175,1179,1184,1188],{"type":58,"tag":891,"props":1121,"children":1122},{"style":898},[1123],{"type":64,"value":910},{"type":58,"tag":891,"props":1125,"children":1126},{"style":913},[1127],{"type":64,"value":1128},"systemNamespaces",{"type":58,"tag":891,"props":1130,"children":1131},{"style":898},[1132],{"type":64,"value":921},{"type":58,"tag":891,"props":1134,"children":1135},{"style":898},[1136],{"type":64,"value":926},{"type":58,"tag":891,"props":1138,"children":1139},{"style":898},[1140],{"type":64,"value":1077},{"type":58,"tag":891,"props":1142,"children":1143},{"style":898},[1144],{"type":64,"value":921},{"type":58,"tag":891,"props":1146,"children":1147},{"style":1084},[1148],{"type":64,"value":1149},"kube-system",{"type":58,"tag":891,"props":1151,"children":1152},{"style":898},[1153],{"type":64,"value":921},{"type":58,"tag":891,"props":1155,"children":1156},{"style":898},[1157],{"type":64,"value":1096},{"type":58,"tag":891,"props":1159,"children":1160},{"style":898},[1161],{"type":64,"value":1101},{"type":58,"tag":891,"props":1163,"children":1164},{"style":1084},[1165],{"type":64,"value":1166},"aws-hyperpod",{"type":58,"tag":891,"props":1168,"children":1169},{"style":898},[1170],{"type":64,"value":921},{"type":58,"tag":891,"props":1172,"children":1173},{"style":898},[1174],{"type":64,"value":1096},{"type":58,"tag":891,"props":1176,"children":1177},{"style":898},[1178],{"type":64,"value":1101},{"type":58,"tag":891,"props":1180,"children":1181},{"style":1084},[1182],{"type":64,"value":1183},"amazon-cloudwatch",{"type":58,"tag":891,"props":1185,"children":1186},{"style":898},[1187],{"type":64,"value":921},{"type":58,"tag":891,"props":1189,"children":1190},{"style":898},[1191],{"type":64,"value":1192},"]\n",{"type":58,"tag":891,"props":1194,"children":1196},{"class":893,"line":1195},9,[1197],{"type":58,"tag":891,"props":1198,"children":1199},{"style":898},[1200],{"type":64,"value":1201},"}\n",{"type":58,"tag":855,"props":1203,"children":1204},{},[],{"type":58,"tag":73,"props":1206,"children":1207},{},[1208],{"type":64,"value":1209},"Where to find this block at runtime.",{"type":64,"value":1211}," The DevOps Agent\nplatform preserves the top-level ",{"type":58,"tag":81,"props":1213,"children":1215},{"className":1214},[],[1216],{"type":64,"value":1217},"description",{"type":64,"value":1219}," string of the\nincoming task verbatim, but drops nested sub-objects from the\nwebhook payload. The audit Lambda therefore inlines the\n",{"type":58,"tag":81,"props":1221,"children":1223},{"className":1222},[],[1224],{"type":64,"value":410},{"type":64,"value":1226}," block into the task description text on a line that\nbegins:",{"type":58,"tag":880,"props":1228,"children":1232},{"className":1229,"code":1231,"language":64},[1230],"language-text","k8sChecks configuration (parse as JSON, then apply per Phase 1 step 8 + Phase 3d):\n{ ... }\n",[1233],{"type":58,"tag":81,"props":1234,"children":1235},{"__ignoreMap":885},[1236],{"type":64,"value":1231},{"type":58,"tag":855,"props":1238,"children":1239},{},[],{"type":64,"value":1241},"Locate that line in the task description, extract the JSON\nobject on the next line, and use its fields for Phase 3d\nthresholds and namespaces. Do NOT default any field unless the\nwhole block is absent — the block's values override the built-in\ndefaults regardless of whether they equal the defaults.",{"type":58,"tag":855,"props":1243,"children":1244},{},[],{"type":64,"value":1246},"If the block is absent from the description entirely, use\ndefaults: ",{"type":58,"tag":81,"props":1248,"children":1250},{"className":1249},[],[1251],{"type":64,"value":1252},"crashLoopMinRestarts=5",{"type":64,"value":148},{"type":58,"tag":81,"props":1255,"children":1257},{"className":1256},[],[1258],{"type":64,"value":1259},"crashLoopRecencyMinutes=15",{"type":64,"value":733},{"type":58,"tag":81,"props":1262,"children":1264},{"className":1263},[],[1265],{"type":64,"value":1266},"notReadyNodePercentThreshold=10",{"type":64,"value":148},{"type":58,"tag":81,"props":1269,"children":1271},{"className":1270},[],[1272],{"type":64,"value":1273},"notReadyDurationMinutes=15",{"type":64,"value":733},{"type":58,"tag":81,"props":1276,"children":1278},{"className":1277},[],[1279],{"type":64,"value":1280},"ignoreNamespaces=[\"kube-public\",\"kube-node-lease\"]",{"type":64,"value":733},{"type":58,"tag":81,"props":1283,"children":1285},{"className":1284},[],[1286],{"type":64,"value":1287},"systemNamespaces=[\"kube-system\",\"aws-hyperpod\",\"amazon-cloudwatch\"]",{"type":64,"value":361},{"type":58,"tag":855,"props":1290,"children":1291},{},[],{"type":64,"value":1293},"If the block has ",{"type":58,"tag":81,"props":1295,"children":1297},{"className":1296},[],[1298],{"type":64,"value":1299},"enabled: false",{"type":64,"value":1301},", still run the kubectl commands\n(Phase 1's job is discovery), but Phase 3d will not fire — see\nnote there.",{"type":58,"tag":286,"props":1303,"children":1305},{"id":1304},"phase-1-gather-sanity-gate-verify-before-entering-phase-2",[1306],{"type":64,"value":1307},"Phase 1 gather sanity gate — verify BEFORE entering Phase 2",{"type":58,"tag":67,"props":1309,"children":1310},{},[1311],{"type":64,"value":1312},"Before starting Phase 2 (timeline reconstruction), confirm that\nPhase 1 executed the required steps:",{"type":58,"tag":247,"props":1314,"children":1315},{},[1316,1321],{"type":58,"tag":251,"props":1317,"children":1318},{},[1319],{"type":64,"value":1320},"Steps 1, 2, 3 (describe-cluster, list-cluster-nodes,\nlist-cluster-events): required for both incident and audit modes.",{"type":58,"tag":251,"props":1322,"children":1323},{},[1324,1326,1331],{"type":64,"value":1325},"Step 8 (kubectl get pods, kubectl get nodes): required in audit\nmode on EKS. Missing here is a ",{"type":58,"tag":73,"props":1327,"children":1328},{},[1329],{"type":64,"value":1330},"hard error",{"type":64,"value":1332}," — do not proceed to\nPhase 2. Instead: run step 8 now, then re-enter this gate.",{"type":58,"tag":67,"props":1334,"children":1335},{},[1336,1338,1343,1345,1350],{"type":64,"value":1337},"Do not rationalize skipping step 8 with reasoning like \"the\n",{"type":58,"tag":81,"props":1339,"children":1341},{"className":1340},[],[1342],{"type":64,"value":225},{"type":64,"value":1344}," window looks quiet so I don't need to check\npods\" or \"the LCS storm from earlier is more interesting.\" Those\nare outputs of Phase 3 reasoning, not inputs to Phase 1. Pod state\nmust be gathered as ",{"type":58,"tag":73,"props":1346,"children":1347},{},[1348],{"type":64,"value":1349},"evidence",{"type":64,"value":1351}," before Phase 3 rules can weigh\n\"live k8s problem\" against \"historical event pattern.\"",{"type":58,"tag":286,"props":1353,"children":1355},{"id":1354},"phase-2-reconstruct-the-timeline",[1356],{"type":64,"value":1357},"Phase 2 — Reconstruct the timeline",{"type":58,"tag":67,"props":1359,"children":1360},{},[1361],{"type":64,"value":1362},"Build a single ordered timeline keyed by UTC timestamp across all\nsources, restricted to the affected scope (cluster, instance group, or\ninstance). Mark each entry with its source. The shape should be:",{"type":58,"tag":880,"props":1364,"children":1367},{"className":1365,"code":1366,"language":64},[1230],"T+0:00   [HMA]            HealthMonitoringAgentDetectionEvent — Xid 79 on GPU 3\nT+0:32   [ClusterEvent]   Action:Replace marked on i-aaa (NodeRecovery=Automatic)\nT+0:33   [Node]           list-cluster-nodes: i-aaa removed\nT+0:34   [ClusterEvent]   Replacement started for instance group worker1\nT+18:21  [ClusterEvent]   Replacement failed: EFA health checks did not run successfully\nT+18:25  [Node]           list-cluster-nodes: still missing\nT+20:10  [ClusterEvent]   Replacement started for instance group worker1   ← second attempt\n...\n",[1368],{"type":58,"tag":81,"props":1369,"children":1370},{"__ignoreMap":885},[1371],{"type":64,"value":1366},{"type":58,"tag":67,"props":1373,"children":1374},{},[1375],{"type":64,"value":1376},"This is the artifact the classification phase reasons over. Include it\nin the final report regardless of verdict — operators need it to\ndouble-check the agent's call.",{"type":58,"tag":286,"props":1378,"children":1380},{"id":1379},"phase-2b-build-the-fault-content-signature-for-each-event",[1381],{"type":64,"value":1382},"Phase 2b — Build the fault-content signature for each event",{"type":58,"tag":67,"props":1384,"children":1385},{},[1386,1388,1393],{"type":64,"value":1387},"Every fault event (Error\u002FWarn cluster event or HMA detection within\nthe 4h window for signature-set computation, and within the 7d window\nfor recurrence statistics) gets a ",{"type":58,"tag":73,"props":1389,"children":1390},{},[1391],{"type":64,"value":1392},"signature string",{"type":64,"value":1394},", formed by\nconcatenating the event's Description with any FailureMessage the\nbridge Lambda enriched it with, prefixed by the InstanceGroup name.",{"type":58,"tag":67,"props":1396,"children":1397},{},[1398],{"type":64,"value":1399},"Signature format:",{"type":58,"tag":880,"props":1401,"children":1404},{"className":1402,"code":1403,"language":64},[1230],"\u003Cig>:\u003Cfull-description-and-failure-message-content>\n",[1405],{"type":58,"tag":81,"props":1406,"children":1407},{"__ignoreMap":885},[1408],{"type":64,"value":1403},{"type":58,"tag":67,"props":1410,"children":1411},{},[1412],{"type":64,"value":1413},"Sources of the content, in order of preference:",{"type":58,"tag":493,"props":1415,"children":1416},{},[1417,1459,1503,1515],{"type":58,"tag":251,"props":1418,"children":1419},{},[1420,1422,1427,1429,1435,1437,1443,1444,1450,1452,1458],{"type":64,"value":1421},"The bridge Lambda's enriched description (top-level ",{"type":58,"tag":81,"props":1423,"children":1425},{"className":1424},[],[1426],{"type":64,"value":1217},{"type":64,"value":1428},"\nfield in the incoming trigger payload). This already includes\n",{"type":58,"tag":81,"props":1430,"children":1432},{"className":1431},[],[1433],{"type":64,"value":1434},"Description: ...",{"type":64,"value":1436}," + ",{"type":58,"tag":81,"props":1438,"children":1440},{"className":1439},[],[1441],{"type":64,"value":1442},"FailureMessage: ...",{"type":64,"value":1436},{"type":58,"tag":81,"props":1445,"children":1447},{"className":1446},[],[1448],{"type":64,"value":1449},"InstanceMetadata: ...",{"type":64,"value":1451},"\nfields the bridge assembled by calling ",{"type":58,"tag":81,"props":1453,"children":1455},{"className":1454},[],[1456],{"type":64,"value":1457},"DescribeClusterEvent",{"type":64,"value":361},{"type":58,"tag":251,"props":1460,"children":1461},{},[1462,1464,1469,1471,1477,1479,1485,1487,1493,1495,1501],{"type":64,"value":1463},"For events retrieved via ",{"type":58,"tag":81,"props":1465,"children":1467},{"className":1466},[],[1468],{"type":64,"value":225},{"type":64,"value":1470}," during Phase 1\npagination (not through the bridge), call\n",{"type":58,"tag":81,"props":1472,"children":1474},{"className":1473},[],[1475],{"type":64,"value":1476},"describe-cluster-event --event-id \u003Ceid>",{"type":64,"value":1478}," yourself for each\nError\u002FWarn EventId. Concatenate ",{"type":58,"tag":81,"props":1480,"children":1482},{"className":1481},[],[1483],{"type":64,"value":1484},"Description",{"type":64,"value":1486}," +\n",{"type":58,"tag":81,"props":1488,"children":1490},{"className":1489},[],[1491],{"type":64,"value":1492},"EventDetails.EventMetadata.Instance.FailureMessage",{"type":64,"value":1494}," (and any\nother populated ",{"type":58,"tag":81,"props":1496,"children":1498},{"className":1497},[],[1499],{"type":64,"value":1500},"Instance.*",{"type":64,"value":1502}," fields) the same way the bridge does.",{"type":58,"tag":251,"props":1504,"children":1505},{},[1506,1508,1513],{"type":64,"value":1507},"For HMA CloudWatch stream events: use the full\n",{"type":58,"tag":81,"props":1509,"children":1511},{"className":1510},[],[1512],{"type":64,"value":680},{"type":64,"value":1514}," text.",{"type":58,"tag":251,"props":1516,"children":1517},{},[1518,1520,1526],{"type":64,"value":1519},"Cluster State Change events: use ",{"type":58,"tag":81,"props":1521,"children":1523},{"className":1522},[],[1524],{"type":64,"value":1525},"ClusterStatus + Description",{"type":64,"value":361},{"type":58,"tag":67,"props":1528,"children":1529},{},[1530,1535,1537,1543,1544,1550,1551,1557,1559,1565,1567,1572,1574,1580,1582,1588],{"type":58,"tag":73,"props":1531,"children":1532},{},[1533],{"type":64,"value":1534},"Why full concatenated content, not hard-coded categories.",{"type":64,"value":1536},"\nClassifying events into a fixed enum of categories (",{"type":58,"tag":81,"props":1538,"children":1540},{"className":1539},[],[1541],{"type":64,"value":1542},"gpu-xid",{"type":64,"value":733},{"type":58,"tag":81,"props":1545,"children":1547},{"className":1546},[],[1548],{"type":64,"value":1549},"lifecycle-script-failed",{"type":64,"value":148},{"type":58,"tag":81,"props":1552,"children":1554},{"className":1553},[],[1555],{"type":64,"value":1556},"capacity-insufficient",{"type":64,"value":1558},", etc.) via regex\nrules merges distinct fault types — for example, \"capacity for\nml.g5.8xlarge\", \"capacity for ml.p5.48xlarge\", \"EFA health check\nfailed\", and \"generic provisioning failure\" all collapse to the same\n",{"type":58,"tag":81,"props":1560,"children":1562},{"className":1561},[],[1563],{"type":64,"value":1564},"instance-creation-failed:generic",{"type":64,"value":1566}," key, because the top-level\n",{"type":58,"tag":81,"props":1568,"children":1570},{"className":1569},[],[1571],{"type":64,"value":1484},{"type":64,"value":1573}," field only says ",{"type":58,"tag":81,"props":1575,"children":1577},{"className":1576},[],[1578],{"type":64,"value":1579},"\"Failed to provision EC2 Instance in Cluster ...\"",{"type":64,"value":1581},". The actual root cause lives in ",{"type":58,"tag":81,"props":1583,"children":1585},{"className":1584},[],[1586],{"type":64,"value":1587},"FailureMessage",{"type":64,"value":1589}," (a\ndifferent string per fault type), so include it. Concatenated raw\ncontent is more robust than enum-based categories.",{"type":58,"tag":67,"props":1591,"children":1592},{},[1593,1598],{"type":58,"tag":73,"props":1594,"children":1595},{},[1596],{"type":64,"value":1597},"InstanceGroup extraction",{"type":64,"value":926},{"type":58,"tag":247,"props":1600,"children":1601},{},[1602,1613,1624],{"type":58,"tag":251,"props":1603,"children":1604},{},[1605,1607],{"type":64,"value":1606},"Cluster Event: ",{"type":58,"tag":81,"props":1608,"children":1610},{"className":1609},[],[1611],{"type":64,"value":1612},"detail.EventDetails.InstanceGroupName",{"type":58,"tag":251,"props":1614,"children":1615},{},[1616,1618],{"type":64,"value":1617},"Node Health \u002F Cluster State Change: ",{"type":58,"tag":81,"props":1619,"children":1621},{"className":1620},[],[1622],{"type":64,"value":1623},"detail.InstanceGroupName",{"type":58,"tag":251,"props":1625,"children":1626},{},[1627,1629,1635],{"type":64,"value":1628},"Cluster-level fault: empty (",{"type":58,"tag":81,"props":1630,"children":1632},{"className":1631},[],[1633],{"type":64,"value":1634},"\"\"",{"type":64,"value":1636},") — no IG prefix.",{"type":58,"tag":286,"props":1638,"children":1640},{"id":1639},"phase-2c-compute-recurrence-statistics",[1641],{"type":64,"value":1642},"Phase 2c — Compute recurrence statistics",{"type":58,"tag":67,"props":1644,"children":1645},{},[1646,1648,1653],{"type":64,"value":1647},"Over the 7-day ",{"type":58,"tag":81,"props":1649,"children":1651},{"className":1650},[],[1652],{"type":64,"value":225},{"type":64,"value":1654}," window from Phase 1 (and HMA\nCloudWatch stream over the same window), compute and record (used by\nPhase 3 rules 6–8):",{"type":58,"tag":247,"props":1656,"children":1657},{},[1658,1677,1688,1699],{"type":58,"tag":251,"props":1659,"children":1660},{},[1661,1667,1669,1675],{"type":58,"tag":81,"props":1662,"children":1664},{"className":1663},[],[1665],{"type":64,"value":1666},"replacements_7d_total",{"type":64,"value":1668}," — count of ",{"type":58,"tag":81,"props":1670,"children":1672},{"className":1671},[],[1673],{"type":64,"value":1674},"Replace",{"type":64,"value":1676}," actions \u002F replacement-\nstarted cluster events across the whole cluster in the last 7 days.",{"type":58,"tag":251,"props":1678,"children":1679},{},[1680,1686],{"type":58,"tag":81,"props":1681,"children":1683},{"className":1682},[],[1684],{"type":64,"value":1685},"replacements_7d_by_group[\u003Cig>]",{"type":64,"value":1687}," — same, partitioned by InstanceGroup.",{"type":58,"tag":251,"props":1689,"children":1690},{},[1691,1697],{"type":58,"tag":81,"props":1692,"children":1694},{"className":1693},[],[1695],{"type":64,"value":1696},"replacements_24h_total",{"type":64,"value":1698}," — same metric, 24h window.",{"type":58,"tag":251,"props":1700,"children":1701},{},[1702,1708,1710,1716,1718,1723,1725,1731],{"type":58,"tag":81,"props":1703,"children":1705},{"className":1704},[],[1706],{"type":64,"value":1707},"signature_count_7d[\u003Csignature>]",{"type":64,"value":1709}," — count of distinct fault events\nwith the same signature string in the last 7 days. Uses the\nPhase 2b signature format ",{"type":58,"tag":81,"props":1711,"children":1713},{"className":1712},[],[1714],{"type":64,"value":1715},"\u003Cig>:\u003Ccontent>",{"type":64,"value":1717},".\n",{"type":58,"tag":73,"props":1719,"children":1720},{},[1721],{"type":64,"value":1722},"Exclusion",{"type":64,"value":1724},": events whose Description contains\n",{"type":58,"tag":81,"props":1726,"children":1728},{"className":1727},[],[1729],{"type":64,"value":1730},"\"lost orchestration-ready status\"",{"type":64,"value":1732}," MUST be excluded from\nsignature counting. These are scale-in-progress noise (see\nmental-model § \"Scale-in-progress emits spurious Warn events\"),\nnot fault signals. Do NOT count them toward rule 6 thresholds.",{"type":58,"tag":67,"props":1734,"children":1735},{},[1736],{"type":64,"value":1737},"Include these counts in the verdict description's \"What HyperPod is\ndoing right now\" paragraph when they're ≥2 — even if the verdict\nitself doesn't change, the operator should see the count.",{"type":58,"tag":286,"props":1739,"children":1741},{"id":1740},"phase-2d-compute-the-signature-set",[1742],{"type":64,"value":1743},"Phase 2d — Compute the signature set",{"type":58,"tag":67,"props":1745,"children":1746},{},[1747],{"type":64,"value":1748},"The signature set is used for verdict-title generation. Encoding it in\nthe title lets the DevOps Agent platform's title-based dedup absorb\nback-to-back audits of the same fault set (identical set → identical\ntitle), while a new fault set produces a new title and a fresh\ninvestigation.",{"type":58,"tag":247,"props":1750,"children":1751},{},[1752,1797],{"type":58,"tag":251,"props":1753,"children":1754},{},[1755,1761,1763,1784,1787,1789,1795],{"type":58,"tag":81,"props":1756,"children":1758},{"className":1757},[],[1759],{"type":64,"value":1760},"current_signature_set",{"type":64,"value":1762}," — sorted, deduplicated set of signature\nstrings (per Phase 2b) for every distinct fault event in the\n4-hour window. Examples:",{"type":58,"tag":247,"props":1764,"children":1765},{},[1766,1775],{"type":58,"tag":251,"props":1767,"children":1768},{},[1769],{"type":58,"tag":81,"props":1770,"children":1772},{"className":1771},[],[1773],{"type":64,"value":1774},"{\"worker4:Description: Failed to provision EC2 Instance in Cluster my-cluster and InstanceGroup worker4. FailureMessage: We currently do not have sufficient capacity to launch new ml.g5.8xlarge instances. Please try again.\"}",{"type":58,"tag":251,"props":1776,"children":1777},{},[1778],{"type":58,"tag":81,"props":1779,"children":1781},{"className":1780},[],[1782],{"type":64,"value":1783},"{\"worker2:Description: Instance i-XXXX is unhealthy. HyperPod Health Monitoring Agent (HMA) has detected fault type NvidiaGPUUnhealthy on this node and is unhealthy. Repair action: Replace.\"}",{"type":58,"tag":855,"props":1785,"children":1786},{},[],{"type":64,"value":1788},"Render in the verdict title as a sorted comma-separated list.\nFull strings; do NOT truncate. If total title length exceeds\nplatform limits (rare but possible for very long FailureMessages),\ntrim signatures from the middle with ",{"type":58,"tag":81,"props":1790,"children":1792},{"className":1791},[],[1793],{"type":64,"value":1794},"...",{"type":64,"value":1796}," markers to preserve\nthe beginning (where the discriminating text lives).",{"type":58,"tag":251,"props":1798,"children":1799},{},[1800,1806,1808,1814],{"type":58,"tag":81,"props":1801,"children":1803},{"className":1802},[],[1804],{"type":64,"value":1805},"current_most_recent_event_at",{"type":64,"value":1807}," — the latest ",{"type":58,"tag":81,"props":1809,"children":1811},{"className":1810},[],[1812],{"type":64,"value":1813},"EventTime",{"type":64,"value":1815}," of any\nError\u002FWarn cluster event or HMA detection within the window.",{"type":58,"tag":286,"props":1817,"children":1819},{"id":1818},"phase-3-classify",[1820],{"type":64,"value":1821},"Phase 3 — Classify",{"type":58,"tag":67,"props":1823,"children":1824},{},[1825],{"type":58,"tag":73,"props":1826,"children":1827},{},[1828],{"type":64,"value":1829},"Ordering — MANDATORY.",{"type":58,"tag":493,"props":1831,"children":1832},{},[1833,1850],{"type":58,"tag":251,"props":1834,"children":1835},{},[1836,1841,1843,1848],{"type":58,"tag":73,"props":1837,"children":1838},{},[1839],{"type":64,"value":1840},"First",{"type":64,"value":1842},", run ",{"type":58,"tag":73,"props":1844,"children":1845},{},[1846],{"type":64,"value":1847},"Phase 3d",{"type":64,"value":1849}," (Kubernetes-state checks — CrashLoopBackOff,\nNotReady). Emit any Escalate verdicts from Phase 3d immediately.",{"type":58,"tag":251,"props":1851,"children":1852},{},[1853,1858],{"type":58,"tag":73,"props":1854,"children":1855},{},[1856],{"type":64,"value":1857},"Then",{"type":64,"value":1859},", run the fault-chain rules below in this Phase 3 table.",{"type":58,"tag":67,"props":1861,"children":1862},{},[1863,1865,1870,1872,1877],{"type":64,"value":1864},"Reason: Phase 3d looks at ",{"type":58,"tag":73,"props":1866,"children":1867},{},[1868],{"type":64,"value":1869},"live current state",{"type":64,"value":1871},", which is the most\nactionable signal for a periodic audit. The main rule table below\nlooks at the 4h\u002F24h\u002F7d event history, which is context. When both\nwould fire, the operator needs the live state surfaced regardless\nof whether historical patterns are ALSO present. Do not skip Phase\n3d because \"the event history already provides an interesting\nverdict\" — that produced the 2026-07-09 miss where a live\nCrashLoopBackOff was overshadowed by a recurring-LCS-pattern\nverdict from a ",{"type":58,"tag":807,"props":1873,"children":1874},{},[1875],{"type":64,"value":1876},"already-resolved",{"type":64,"value":1878}," 07-08 storm.",{"type":58,"tag":67,"props":1880,"children":1881},{},[1882],{"type":64,"value":1883},"Emitting both a Phase 3d verdict and a Phase 3 verdict on the same\naudit is expected and correct. Each becomes its own symptom record;\nthe operator gets both signals via email.",{"type":58,"tag":1885,"props":1886,"children":1888},"h4",{"id":1887},"phase-3d-kubernetes-state-checks-audit-mode-eks-only",[1889],{"type":64,"value":1890},"Phase 3d — Kubernetes-state checks (audit mode, EKS only)",{"type":58,"tag":67,"props":1892,"children":1893},{},[1894,1896,1901,1903,1909,1910,1916],{"type":64,"value":1895},"Run this ",{"type":58,"tag":73,"props":1897,"children":1898},{},[1899],{"type":64,"value":1900},"BEFORE",{"type":64,"value":1902}," the fault-chain classification rules below\n(see the \"Ordering — MANDATORY\" note at the top of Phase 3). Phase\n3d uses the ",{"type":58,"tag":81,"props":1904,"children":1906},{"className":1905},[],[1907],{"type":64,"value":1908},"kubectl get pods",{"type":64,"value":114},{"type":58,"tag":81,"props":1911,"children":1913},{"className":1912},[],[1914],{"type":64,"value":1915},"kubectl get nodes",{"type":64,"value":1917}," output from\nPhase 1 step 8, which is mandatory in audit-mode-on-EKS regardless\nof the k8sChecks payload block. Emit each verdict as an\nindependent symptom record.",{"type":58,"tag":67,"props":1919,"children":1920},{},[1921,1926,1928,1933,1935,1940,1942,1947,1949,1955,1957,1962],{"type":58,"tag":73,"props":1922,"children":1923},{},[1924],{"type":64,"value":1925},"Threshold + namespace configuration.",{"type":64,"value":1927}," Use the ",{"type":58,"tag":81,"props":1929,"children":1931},{"className":1930},[],[1932],{"type":64,"value":410},{"type":64,"value":1934}," block\nparsed from the task ",{"type":58,"tag":73,"props":1936,"children":1937},{},[1938],{"type":64,"value":1939},"description line",{"type":64,"value":1941}," per Phase 1 step 8 — the\nDevOps Agent platform preserves the top-level ",{"type":58,"tag":81,"props":1943,"children":1945},{"className":1944},[],[1946],{"type":64,"value":1217},{"type":64,"value":1948}," verbatim\nbut flattens nested payload sub-objects, so ",{"type":58,"tag":81,"props":1950,"children":1952},{"className":1951},[],[1953],{"type":64,"value":1954},"data.metadata.k8sChecks",{"type":64,"value":1956},"\nis NOT reliably present; the description-inline copy is the source of\ntruth. If no block is in the description, use the defaults documented\nin Phase 1 step 8. If the block is present with ",{"type":58,"tag":81,"props":1958,"children":1960},{"className":1959},[],[1961],{"type":64,"value":1299},{"type":64,"value":1963},", DO\nNOT emit Phase 3d verdicts (customer has opted out of the k8s-state\nEscalate) — but Phase 1's kubectl gather still ran because it's part\nof general state discovery, and its data may still surface in Phase 4\nreport context.",{"type":58,"tag":67,"props":1965,"children":1966},{},[1967,1972,1974,1979],{"type":58,"tag":73,"props":1968,"children":1969},{},[1970],{"type":64,"value":1971},"Pod check — CrashLoopBackOff loop.",{"type":64,"value":1973}," For each Pod in the\n",{"type":58,"tag":81,"props":1975,"children":1977},{"className":1976},[],[1978],{"type":64,"value":825},{"type":64,"value":1980}," output:",{"type":58,"tag":493,"props":1982,"children":1983},{},[1984,2067,2216],{"type":58,"tag":251,"props":1985,"children":1986},{},[1987,1992,1994,2000,2001,2048,2051,2053,2058,2060,2065],{"type":58,"tag":73,"props":1988,"children":1989},{},[1990],{"type":64,"value":1991},"Namespace classification.",{"type":64,"value":1993}," Look up ",{"type":58,"tag":81,"props":1995,"children":1997},{"className":1996},[],[1998],{"type":64,"value":1999},"pod.metadata.namespace",{"type":64,"value":926},{"type":58,"tag":247,"props":2002,"children":2003},{},[2004,2017,2036],{"type":58,"tag":251,"props":2005,"children":2006},{},[2007,2009,2015],{"type":64,"value":2008},"If it is in ",{"type":58,"tag":81,"props":2010,"children":2012},{"className":2011},[],[2013],{"type":64,"value":2014},"k8sChecks.ignoreNamespaces",{"type":64,"value":2016}," → skip this pod entirely.",{"type":58,"tag":251,"props":2018,"children":2019},{},[2020,2021,2027,2029,2035],{"type":64,"value":2008},{"type":58,"tag":81,"props":2022,"children":2024},{"className":2023},[],[2025],{"type":64,"value":2026},"k8sChecks.systemNamespaces",{"type":64,"value":2028}," → workload class is\n",{"type":58,"tag":81,"props":2030,"children":2032},{"className":2031},[],[2033],{"type":64,"value":2034},"system-workload",{"type":64,"value":361},{"type":58,"tag":251,"props":2037,"children":2038},{},[2039,2041,2047],{"type":64,"value":2040},"Otherwise → workload class is ",{"type":58,"tag":81,"props":2042,"children":2044},{"className":2043},[],[2045],{"type":64,"value":2046},"customer-workload",{"type":64,"value":361},{"type":58,"tag":855,"props":2049,"children":2050},{},[],{"type":64,"value":2052},"This is a plain set-membership lookup, in this exact order. No\npattern matching, no wildcards, no precedence rules. The Lambda\nhas already validated that ",{"type":58,"tag":81,"props":2054,"children":2056},{"className":2055},[],[2057],{"type":64,"value":1064},{"type":64,"value":2059}," and\n",{"type":58,"tag":81,"props":2061,"children":2063},{"className":2062},[],[2064],{"type":64,"value":1128},{"type":64,"value":2066}," do not overlap, so a pod's namespace matches\nat most one list.",{"type":58,"tag":251,"props":2068,"children":2069},{},[2070,2075,2077,2083,2085,2090,2092,2098,2100,2106,2107,2113,2115,2121,2123,2128,2130],{"type":58,"tag":73,"props":2071,"children":2072},{},[2073],{"type":64,"value":2074},"CrashLoopBackOff detection.",{"type":64,"value":2076}," For each container in\n",{"type":58,"tag":81,"props":2078,"children":2080},{"className":2079},[],[2081],{"type":64,"value":2082},"pod.status.containerStatuses[]",{"type":64,"value":2084},", decide whether it is in a\npersistent crash loop. Kubernetes does ",{"type":58,"tag":73,"props":2086,"children":2087},{},[2088],{"type":64,"value":2089},"not",{"type":64,"value":2091}," expose when the\nloop started in a snapshot — ",{"type":58,"tag":81,"props":2093,"children":2095},{"className":2094},[],[2096],{"type":64,"value":2097},"containerStatuses[].state.waiting",{"type":64,"value":2099},"\ncarries only ",{"type":58,"tag":81,"props":2101,"children":2103},{"className":2102},[],[2104],{"type":64,"value":2105},"reason",{"type":64,"value":773},{"type":58,"tag":81,"props":2108,"children":2110},{"className":2109},[],[2111],{"type":64,"value":2112},"message",{"type":64,"value":2114}," (no timestamp), and\n",{"type":58,"tag":81,"props":2116,"children":2118},{"className":2117},[],[2119],{"type":64,"value":2120},"lastState.terminated",{"type":64,"value":2122}," holds only the ",{"type":58,"tag":807,"props":2124,"children":2125},{},[2126],{"type":64,"value":2127},"most recent",{"type":64,"value":2129}," crashed run —\nso use crash evidence, not a duration:",{"type":58,"tag":247,"props":2131,"children":2132},{},[2133,2158,2174],{"type":58,"tag":251,"props":2134,"children":2135},{},[2136,2141,2143,2149,2151,2156],{"type":58,"tag":73,"props":2137,"children":2138},{},[2139],{"type":64,"value":2140},"Loop present",{"type":64,"value":2142}," when ",{"type":58,"tag":81,"props":2144,"children":2146},{"className":2145},[],[2147],{"type":64,"value":2148},"state.waiting.reason == \"CrashLoopBackOff\"",{"type":64,"value":2150},"\nOR a ",{"type":58,"tag":81,"props":2152,"children":2154},{"className":2153},[],[2155],{"type":64,"value":2120},{"type":64,"value":2157}," record exists (persistent evidence,\nstable across the backoff cycle).",{"type":58,"tag":251,"props":2159,"children":2160},{},[2161,2166,2167,2173],{"type":58,"tag":73,"props":2162,"children":2163},{},[2164],{"type":64,"value":2165},"Enough restarts:",{"type":64,"value":98},{"type":58,"tag":81,"props":2168,"children":2170},{"className":2169},[],[2171],{"type":64,"value":2172},"restartCount >= k8sChecks.crashLoopMinRestarts",{"type":64,"value":361},{"type":58,"tag":251,"props":2175,"children":2176},{},[2177,2182,2184,2190,2192,2198,2200,2206,2208,2214],{"type":58,"tag":73,"props":2178,"children":2179},{},[2180],{"type":64,"value":2181},"Still active:",{"type":64,"value":2183}," the last crash\n(",{"type":58,"tag":81,"props":2185,"children":2187},{"className":2186},[],[2188],{"type":64,"value":2189},"lastState.terminated.finishedAt",{"type":64,"value":2191},") is within\n",{"type":58,"tag":81,"props":2193,"children":2195},{"className":2194},[],[2196],{"type":64,"value":2197},"k8sChecks.crashLoopRecencyMinutes",{"type":64,"value":2199}," minutes — or the container is\ncurrently in ",{"type":58,"tag":81,"props":2201,"children":2203},{"className":2202},[],[2204],{"type":64,"value":2205},"CrashLoopBackOff",{"type":64,"value":2207}," (treat a missing ",{"type":58,"tag":81,"props":2209,"children":2211},{"className":2210},[],[2212],{"type":64,"value":2213},"finishedAt",{"type":64,"value":2215}," as\nactive only in that case).",{"type":58,"tag":251,"props":2217,"children":2218},{},[2219,2224,2226,2232,2234],{"type":58,"tag":73,"props":2220,"children":2221},{},[2222],{"type":64,"value":2223},"Verdict.",{"type":64,"value":2225}," If all three hold → emit an ",{"type":58,"tag":81,"props":2227,"children":2229},{"className":2228},[],[2230],{"type":64,"value":2231},"Escalate",{"type":64,"value":2233}," verdict with:",{"type":58,"tag":247,"props":2235,"children":2236},{},[2237,2248,2267],{"type":58,"tag":251,"props":2238,"children":2239},{},[2240,2242],{"type":64,"value":2241},"Title: ",{"type":58,"tag":81,"props":2243,"children":2245},{"className":2244},[],[2246],{"type":64,"value":2247},"Triage verdict: Escalate — CrashLoopBackOff loop active :: (\u003Cnamespace>\u002F\u003Cpod>:\u003Ccontainer>)",{"type":58,"tag":251,"props":2249,"children":2250},{},[2251,2253,2258,2260,2265],{"type":64,"value":2252},"Workload class tag in the description (",{"type":58,"tag":81,"props":2254,"children":2256},{"className":2255},[],[2257],{"type":64,"value":2034},{"type":64,"value":2259}," or\n",{"type":58,"tag":81,"props":2261,"children":2263},{"className":2262},[],[2264],{"type":64,"value":2046},{"type":64,"value":2266},") so downstream email routing can decide who\nto page.",{"type":58,"tag":251,"props":2268,"children":2269},{},[2270,2272,2278,2280,2286,2287,2293],{"type":64,"value":2271},"Include: pod name, namespace, container name, ",{"type":58,"tag":81,"props":2273,"children":2275},{"className":2274},[],[2276],{"type":64,"value":2277},"restartCount",{"type":64,"value":2279},",\nwhether it is currently in backoff, and the most recent\n",{"type":58,"tag":81,"props":2281,"children":2283},{"className":2282},[],[2284],{"type":64,"value":2285},"lastState.terminated.reason",{"type":64,"value":114},{"type":58,"tag":81,"props":2288,"children":2290},{"className":2289},[],[2291],{"type":64,"value":2292},".exitCode",{"type":64,"value":2294}," if available.",{"type":58,"tag":67,"props":2296,"children":2297},{},[2298,2303,2305,2310],{"type":58,"tag":73,"props":2299,"children":2300},{},[2301],{"type":64,"value":2302},"Node check — NotReady percentage.",{"type":64,"value":2304}," From\n",{"type":58,"tag":81,"props":2306,"children":2308},{"className":2307},[],[2309],{"type":64,"value":836},{"type":64,"value":926},{"type":58,"tag":493,"props":2312,"children":2313},{},[2314,2326,2378],{"type":58,"tag":251,"props":2315,"children":2316},{},[2317,2319,2325],{"type":64,"value":2318},"Total node count = length of ",{"type":58,"tag":81,"props":2320,"children":2322},{"className":2321},[],[2323],{"type":64,"value":2324},"items[]",{"type":64,"value":361},{"type":58,"tag":251,"props":2327,"children":2328},{},[2329,2331,2336,2338,2344,2346,2352,2354,2360,2362,2368,2370,2376],{"type":64,"value":2330},"For each node, examine ",{"type":58,"tag":81,"props":2332,"children":2334},{"className":2333},[],[2335],{"type":64,"value":844},{"type":64,"value":2337}," for\n",{"type":58,"tag":81,"props":2339,"children":2341},{"className":2340},[],[2342],{"type":64,"value":2343},"type: Ready",{"type":64,"value":2345},". If ",{"type":58,"tag":81,"props":2347,"children":2349},{"className":2348},[],[2350],{"type":64,"value":2351},"status: \"False\"",{"type":64,"value":2353}," or ",{"type":58,"tag":81,"props":2355,"children":2357},{"className":2356},[],[2358],{"type":64,"value":2359},"\"Unknown\"",{"type":64,"value":2361}," AND\n",{"type":58,"tag":81,"props":2363,"children":2365},{"className":2364},[],[2366],{"type":64,"value":2367},"now() - lastTransitionTime",{"type":64,"value":2369}," exceeds\n",{"type":58,"tag":81,"props":2371,"children":2373},{"className":2372},[],[2374],{"type":64,"value":2375},"k8sChecks.notReadyDurationMinutes",{"type":64,"value":2377}," minutes → count as NotReady.",{"type":58,"tag":251,"props":2379,"children":2380},{},[2381,2383,2389,2391,2397,2399,2404,2406],{"type":64,"value":2382},"If ",{"type":58,"tag":81,"props":2384,"children":2386},{"className":2385},[],[2387],{"type":64,"value":2388},"(NotReady count \u002F total count) * 100",{"type":64,"value":2390}," meets or exceeds\n",{"type":58,"tag":81,"props":2392,"children":2394},{"className":2393},[],[2395],{"type":64,"value":2396},"k8sChecks.notReadyNodePercentThreshold",{"type":64,"value":2398}," → emit an ",{"type":58,"tag":81,"props":2400,"children":2402},{"className":2401},[],[2403],{"type":64,"value":2231},{"type":64,"value":2405},"\nverdict with:\n",{"type":58,"tag":247,"props":2407,"children":2408},{},[2409,2419],{"type":58,"tag":251,"props":2410,"children":2411},{},[2412,2413],{"type":64,"value":2241},{"type":58,"tag":81,"props":2414,"children":2416},{"className":2415},[],[2417],{"type":64,"value":2418},"Triage verdict: Escalate — NotReady nodes exceeded threshold :: (\u003Cn>\u002F\u003Ctotal> nodes NotReady)",{"type":58,"tag":251,"props":2420,"children":2421},{},[2422,2424,2430],{"type":64,"value":2423},"Include the affected node names, their NotReady durations, and\nany ",{"type":58,"tag":81,"props":2425,"children":2427},{"className":2426},[],[2428],{"type":64,"value":2429},"taints[]",{"type":64,"value":2431}," that would explain the state.",{"type":58,"tag":67,"props":2433,"children":2434},{},[2435,2440],{"type":58,"tag":73,"props":2436,"children":2437},{},[2438],{"type":64,"value":2439},"Interaction with the fault-chain classification.",{"type":64,"value":2441}," If a HyperPod\nfault chain in the main rule table already covers the same\nnode (e.g. rule 11 says the retry chain is stuck for a specific\ninstance), the Phase 3d NotReady check may re-flag the same node.\nThat's fine — the two verdicts have different scopes and different\nrecommended actions. Emit both; the operator gets richer context.",{"type":58,"tag":67,"props":2443,"children":2444},{},[2445,2457],{"type":58,"tag":73,"props":2446,"children":2447},{},[2448,2450,2456],{"type":64,"value":2449},"Interaction with ",{"type":58,"tag":81,"props":2451,"children":2453},{"className":2452},[],[2454],{"type":64,"value":2455},"Suppress — periodic audit, no open incidents",{"type":64,"value":361},{"type":64,"value":2458},"\nRule 1 fires when no HyperPod fault chains are open. Phase 3d can\nstill fire independently — a CrashLoopBackOff pod is an incident\neven when HyperPod's own event stream is quiet. If Phase 3d\nproduces any verdict, do NOT emit the rule 1 Suppress; Phase 3d has\nfound something to report.",{"type":58,"tag":1885,"props":2460,"children":2462},{"id":2461},"fault-chain-classification-rules",[2463],{"type":64,"value":2464},"Fault-chain classification rules",{"type":58,"tag":67,"props":2466,"children":2467},{},[2468,2470,2475],{"type":64,"value":2469},"Apply the rules below ",{"type":58,"tag":73,"props":2471,"children":2472},{},[2473],{"type":64,"value":2474},"in order",{"type":64,"value":2476},". Stop at the first match.",{"type":58,"tag":67,"props":2478,"children":2479},{},[2480,2484,2486,2491,2493,2499],{"type":58,"tag":73,"props":2481,"children":2482},{},[2483],{"type":64,"value":377},{"type":64,"value":2485},": classify each open fault chain found in the\ncluster-events window independently. Emit one verdict per chain\n(plus one ",{"type":58,"tag":81,"props":2487,"children":2489},{"className":2488},[],[2490],{"type":64,"value":2455},{"type":64,"value":2492}," if no\nchains are open AND Phase 3d also emitted nothing). A fault chain\nis \"open\" if it had a fault event within the last 4 hours and has\nnot yet emitted a successful ",{"type":58,"tag":81,"props":2494,"children":2496},{"className":2495},[],[2497],{"type":64,"value":2498},"Running",{"type":64,"value":2500}," transition + 30 min\nclean-window.",{"type":58,"tag":67,"props":2502,"children":2503},{},[2504,2509,2511,2516,2518,2523,2525,2531],{"type":58,"tag":73,"props":2505,"children":2506},{},[2507],{"type":64,"value":2508},"Historical-only chains are not open.",{"type":64,"value":2510}," If the only fault events\nin the 4-hour window belong to a chain that has already completed\n(e.g., all replacements succeeded and all affected nodes are back\nin ",{"type":58,"tag":81,"props":2512,"children":2514},{"className":2513},[],[2515],{"type":64,"value":2498},{"type":64,"value":2517}," for ≥30 min), the chain is closed. Do not emit an\n",{"type":58,"tag":81,"props":2519,"children":2521},{"className":2520},[],[2522],{"type":64,"value":2231},{"type":64,"value":2524}," verdict on the basis of ",{"type":58,"tag":81,"props":2526,"children":2528},{"className":2527},[],[2529],{"type":64,"value":2530},"signature_count_7d",{"type":64,"value":2532}," alone\nagainst a closed chain unless the recurrence rule 6\u002F7\u002F8 fires from\nstatistics that INCLUDE at least one event within the last 4 hours.\nA 7-day-old already-recovered pattern is context, not an actionable\nincident.",{"type":58,"tag":67,"props":2534,"children":2535},{},[2536,2538,2543,2545,2551],{"type":64,"value":2537},"The recurring-pattern rules (6–8) are checked ",{"type":58,"tag":73,"props":2539,"children":2540},{},[2541],{"type":64,"value":2542},"before",{"type":64,"value":2544}," the\nsingle-incident rules (9–13) — a node that's auto-recovering for the\n3rd time this week should be flagged as a pattern, not silently\nclassified as ",{"type":58,"tag":81,"props":2546,"children":2548},{"className":2547},[],[2549],{"type":64,"value":2550},"Monitor — first attempt",{"type":64,"value":361},{"type":58,"tag":298,"props":2553,"children":2554},{},[2555,2576],{"type":58,"tag":302,"props":2556,"children":2557},{},[2558],{"type":58,"tag":306,"props":2559,"children":2560},{},[2561,2566,2571],{"type":58,"tag":310,"props":2562,"children":2563},{},[2564],{"type":64,"value":2565},"#",{"type":58,"tag":310,"props":2567,"children":2568},{},[2569],{"type":64,"value":2570},"Signal pattern",{"type":58,"tag":310,"props":2572,"children":2573},{},[2574],{"type":64,"value":2575},"Verdict",{"type":58,"tag":326,"props":2577,"children":2578},{},[2579,2612,2655,2692,2726,2769,2798,2827,2856,2878,2913,2942,2971,3007,3036],{"type":58,"tag":306,"props":2580,"children":2581},{},[2582,2587,2603],{"type":58,"tag":333,"props":2583,"children":2584},{},[2585],{"type":64,"value":2586},"1",{"type":58,"tag":333,"props":2588,"children":2589},{},[2590,2594,2596,2601],{"type":58,"tag":73,"props":2591,"children":2592},{},[2593],{"type":64,"value":377},{"type":64,"value":2595}," AND no fault events in the 4-hour window AND no ",{"type":58,"tag":81,"props":2597,"children":2599},{"className":2598},[],[2600],{"type":64,"value":423},{"type":64,"value":2602}," fault chain still open",{"type":58,"tag":333,"props":2604,"children":2605},{},[2606,2610],{"type":58,"tag":73,"props":2607,"children":2608},{},[2609],{"type":64,"value":2455},{"type":64,"value":2611}," (skip Phase 4 entirely or emit a minimal record; nothing actionable)",{"type":58,"tag":306,"props":2613,"children":2614},{},[2615,2620,2645],{"type":58,"tag":333,"props":2616,"children":2617},{},[2618],{"type":64,"value":2619},"2",{"type":58,"tag":333,"props":2621,"children":2622},{},[2623,2625,2630,2632,2637,2639],{"type":64,"value":2624},"A previously-",{"type":58,"tag":81,"props":2626,"children":2628},{"className":2627},[],[2629],{"type":64,"value":423},{"type":64,"value":2631}," fault chain now shows the affected instance(s) back in ",{"type":58,"tag":81,"props":2633,"children":2635},{"className":2634},[],[2636],{"type":64,"value":2498},{"type":64,"value":2638}," AND no new HMA detection for that instance in the last 30 min AND cluster status is ",{"type":58,"tag":81,"props":2640,"children":2642},{"className":2641},[],[2643],{"type":64,"value":2644},"InService",{"type":58,"tag":333,"props":2646,"children":2647},{},[2648,2653],{"type":58,"tag":73,"props":2649,"children":2650},{},[2651],{"type":64,"value":2652},"Resolved — auto-recovery succeeded",{"type":64,"value":2654}," (operator gets a closure email; include the original detection time and total elapsed)",{"type":58,"tag":306,"props":2656,"children":2657},{},[2658,2663,2684],{"type":58,"tag":333,"props":2659,"children":2660},{},[2661],{"type":64,"value":2662},"3",{"type":58,"tag":333,"props":2664,"children":2665},{},[2666,2668,2674,2676,2682],{"type":64,"value":2667},"Trigger detail-type is ",{"type":58,"tag":81,"props":2669,"children":2671},{"className":2670},[],[2672],{"type":64,"value":2673},"Cluster Event",{"type":64,"value":2675}," with ",{"type":58,"tag":81,"props":2677,"children":2679},{"className":2678},[],[2680],{"type":64,"value":2681},"EventLevel=Info",{"type":64,"value":2683}," and the timeline shows no node-health activity",{"type":58,"tag":333,"props":2685,"children":2686},{},[2687],{"type":58,"tag":73,"props":2688,"children":2689},{},[2690],{"type":64,"value":2691},"Suppress",{"type":58,"tag":306,"props":2693,"children":2694},{},[2695,2700,2717],{"type":58,"tag":333,"props":2696,"children":2697},{},[2698],{"type":64,"value":2699},"4",{"type":58,"tag":333,"props":2701,"children":2702},{},[2703,2705,2710,2711],{"type":64,"value":2704},"Cluster status is ",{"type":58,"tag":81,"props":2706,"children":2708},{"className":2707},[],[2709],{"type":64,"value":217},{"type":64,"value":2353},{"type":58,"tag":81,"props":2712,"children":2714},{"className":2713},[],[2715],{"type":64,"value":2716},"RollingBack",{"type":58,"tag":333,"props":2718,"children":2719},{},[2720,2724],{"type":58,"tag":73,"props":2721,"children":2722},{},[2723],{"type":64,"value":2231},{"type":64,"value":2725}," (cluster-level)",{"type":58,"tag":306,"props":2727,"children":2728},{},[2729,2734,2760],{"type":58,"tag":333,"props":2730,"children":2731},{},[2732],{"type":64,"value":2733},"5",{"type":58,"tag":333,"props":2735,"children":2736},{},[2737,2743,2745,2751,2752,2758],{"type":58,"tag":81,"props":2738,"children":2740},{"className":2739},[],[2741],{"type":64,"value":2742},"NodeRecovery=None",{"type":64,"value":2744}," on the cluster AND a node has been marked ",{"type":58,"tag":81,"props":2746,"children":2748},{"className":2747},[],[2749],{"type":64,"value":2750},"Action:*",{"type":64,"value":114},{"type":58,"tag":81,"props":2753,"children":2755},{"className":2754},[],[2756],{"type":64,"value":2757},"UnschedulablePending*",{"type":64,"value":2759}," AND no replacement has started within 5 minutes",{"type":58,"tag":333,"props":2761,"children":2762},{},[2763,2767],{"type":58,"tag":73,"props":2764,"children":2765},{},[2766],{"type":64,"value":2231},{"type":64,"value":2768}," — auto-recovery is off; operator must trigger replacement",{"type":58,"tag":306,"props":2770,"children":2771},{},[2772,2777,2788],{"type":58,"tag":333,"props":2773,"children":2774},{},[2775],{"type":64,"value":2776},"6",{"type":58,"tag":333,"props":2778,"children":2779},{},[2780,2786],{"type":58,"tag":81,"props":2781,"children":2783},{"className":2782},[],[2784],{"type":64,"value":2785},"signature_count_7d[\u003Csignature>] ≥ 3",{"type":64,"value":2787}," for ANY signature — same fault content (Description + FailureMessage) has driven ≥3 replacements on the same InstanceGroup in the last 7 days (auto-recovery may be succeeding each time). Because signatures are the full concatenated content, this fires cleanly for repeated same-cause faults (e.g. 3× capacity errors for the same instance type) but does NOT over-merge distinct causes.",{"type":58,"tag":333,"props":2789,"children":2790},{},[2791,2796],{"type":58,"tag":73,"props":2792,"children":2793},{},[2794],{"type":64,"value":2795},"Escalate — recurring fault pattern",{"type":64,"value":2797},": HyperPod is repairing the symptom, but the underlying cause hasn't gone away. Include the signature (first ~200 chars) and the timestamps of all prior occurrences. Recommend operator actions appropriate to the fault content (vendor-exclusion for hardware Xid faults; LCS bug for lifecycle-script failures; capacity request or IG-move for capacity failures; VPC\u002FSG review for EFA health-check failures). Infer the appropriate action class from the FailureMessage content, not from a hard-coded category enum.",{"type":58,"tag":306,"props":2799,"children":2800},{},[2801,2806,2817],{"type":58,"tag":333,"props":2802,"children":2803},{},[2804],{"type":64,"value":2805},"7",{"type":58,"tag":333,"props":2807,"children":2808},{},[2809,2815],{"type":58,"tag":81,"props":2810,"children":2812},{"className":2811},[],[2813],{"type":64,"value":2814},"replacements_24h_total ≥ 5",{"type":64,"value":2816}," — five or more replacements anywhere in the cluster within 24 hours",{"type":58,"tag":333,"props":2818,"children":2819},{},[2820,2825],{"type":58,"tag":73,"props":2821,"children":2822},{},[2823],{"type":64,"value":2824},"Escalate — fleet-wide instability",{"type":64,"value":2826},": the rate of node churn is abnormal regardless of individual root causes.",{"type":58,"tag":306,"props":2828,"children":2829},{},[2830,2835,2846],{"type":58,"tag":333,"props":2831,"children":2832},{},[2833],{"type":64,"value":2834},"8",{"type":58,"tag":333,"props":2836,"children":2837},{},[2838,2844],{"type":58,"tag":81,"props":2839,"children":2841},{"className":2840},[],[2842],{"type":64,"value":2843},"replacements_7d_by_group[\u003Cig>] ≥ 5",{"type":64,"value":2845}," — five or more replacements on the same InstanceGroup in 7 days",{"type":58,"tag":333,"props":2847,"children":2848},{},[2849,2854],{"type":58,"tag":73,"props":2850,"children":2851},{},[2852],{"type":64,"value":2853},"Escalate — instance-group instability",{"type":64,"value":2855},": the affected IG (which may be a specific SKU or topology placement) is failing more often than the rest of the cluster.",{"type":58,"tag":306,"props":2857,"children":2858},{},[2859,2864,2869],{"type":58,"tag":333,"props":2860,"children":2861},{},[2862],{"type":64,"value":2863},"9",{"type":58,"tag":333,"props":2865,"children":2866},{},[2867],{"type":64,"value":2868},"Exactly one replacement attempt in flight, started within the last 30 minutes, no prior failure in the chain",{"type":58,"tag":333,"props":2870,"children":2871},{},[2872,2876],{"type":58,"tag":73,"props":2873,"children":2874},{},[2875],{"type":64,"value":2550},{"type":64,"value":2877}," (next re-check in 30 min via scheduled audit)",{"type":58,"tag":306,"props":2879,"children":2880},{},[2881,2886,2903],{"type":58,"tag":333,"props":2882,"children":2883},{},[2884],{"type":64,"value":2885},"10",{"type":58,"tag":333,"props":2887,"children":2888},{},[2889,2891,2895,2896,2901],{"type":64,"value":2890},"Multiple replacement attempts in the chain, total elapsed since the first failure ≤ 90 minutes, the most recent attempt is ",{"type":58,"tag":807,"props":2892,"children":2893},{},[2894],{"type":64,"value":2498},{"type":64,"value":2353},{"type":58,"tag":807,"props":2897,"children":2898},{},[2899],{"type":64,"value":2900},"Started",{"type":64,"value":2902}," (not yet failed)",{"type":58,"tag":333,"props":2904,"children":2905},{},[2906,2911],{"type":58,"tag":73,"props":2907,"children":2908},{},[2909],{"type":64,"value":2910},"Monitor — elevated",{"type":64,"value":2912}," (retry in progress, watch closely)",{"type":58,"tag":306,"props":2914,"children":2915},{},[2916,2921,2933],{"type":58,"tag":333,"props":2917,"children":2918},{},[2919],{"type":64,"value":2920},"11",{"type":58,"tag":333,"props":2922,"children":2923},{},[2924,2926,2931],{"type":64,"value":2925},"Multiple replacement attempts, total elapsed > 90 minutes, no successful ",{"type":58,"tag":81,"props":2927,"children":2929},{"className":2928},[],[2930],{"type":64,"value":2498},{"type":64,"value":2932}," transition, AND no new attempt started within the last 30 minutes",{"type":58,"tag":333,"props":2934,"children":2935},{},[2936,2940],{"type":58,"tag":73,"props":2937,"children":2938},{},[2939],{"type":64,"value":2231},{"type":64,"value":2941}," — retry chain is stuck",{"type":58,"tag":306,"props":2943,"children":2944},{},[2945,2950,2962],{"type":58,"tag":333,"props":2946,"children":2947},{},[2948],{"type":64,"value":2949},"12",{"type":58,"tag":333,"props":2951,"children":2952},{},[2953,2955,2960],{"type":64,"value":2954},"Node was in ",{"type":58,"tag":81,"props":2956,"children":2958},{"className":2957},[],[2959],{"type":64,"value":217},{"type":64,"value":2961}," state AND no new replacement attempt has started within the last 30 minutes AND total time in failing chain > 60 minutes",{"type":58,"tag":333,"props":2963,"children":2964},{},[2965,2969],{"type":58,"tag":73,"props":2966,"children":2967},{},[2968],{"type":64,"value":2231},{"type":64,"value":2970}," — HyperPod has given up",{"type":58,"tag":306,"props":2972,"children":2973},{},[2974,2979,2998],{"type":58,"tag":333,"props":2975,"children":2976},{},[2977],{"type":64,"value":2978},"13",{"type":58,"tag":333,"props":2980,"children":2981},{},[2982,2984,2989,2991,2996],{"type":64,"value":2983},"Instance id from the trigger event is missing from ",{"type":58,"tag":81,"props":2985,"children":2987},{"className":2986},[],[2988],{"type":64,"value":209},{"type":64,"value":2990}," AND ",{"type":58,"tag":81,"props":2992,"children":2994},{"className":2993},[],[2995],{"type":64,"value":225},{"type":64,"value":2997}," shows no new attempt for the last 30 minutes AND the most recent attempt failed",{"type":58,"tag":333,"props":2999,"children":3000},{},[3001,3005],{"type":58,"tag":73,"props":3002,"children":3003},{},[3004],{"type":64,"value":2231},{"type":64,"value":3006}," — instance vanished, no retry",{"type":58,"tag":306,"props":3008,"children":3009},{},[3010,3015,3027],{"type":58,"tag":333,"props":3011,"children":3012},{},[3013],{"type":64,"value":3014},"14",{"type":58,"tag":333,"props":3016,"children":3017},{},[3018,3020,3025],{"type":64,"value":3019},"HMA detection event present but no corresponding ",{"type":58,"tag":81,"props":3021,"children":3023},{"className":3022},[],[3024],{"type":64,"value":2750},{"type":64,"value":3026}," \u002F replacement event in the timeline within 10 minutes",{"type":58,"tag":333,"props":3028,"children":3029},{},[3030,3034],{"type":58,"tag":73,"props":3031,"children":3032},{},[3033],{"type":64,"value":2231},{"type":64,"value":3035}," — HMA fired without escalating; investigate why (mismatch in node-recovery config, signal didn't classify)",{"type":58,"tag":306,"props":3037,"children":3038},{},[3039,3044,3049],{"type":58,"tag":333,"props":3040,"children":3041},{},[3042],{"type":64,"value":3043},"15",{"type":58,"tag":333,"props":3045,"children":3046},{},[3047],{"type":64,"value":3048},"None of the above match",{"type":58,"tag":333,"props":3050,"children":3051},{},[3052,3057],{"type":58,"tag":73,"props":3053,"children":3054},{},[3055],{"type":64,"value":3056},"Monitor — uncategorized",{"type":64,"value":3058}," (include the full timeline; flag for review)",{"type":58,"tag":3060,"props":3061,"children":3062},"blockquote",{},[3063],{"type":58,"tag":67,"props":3064,"children":3065},{},[3066,3071,3073,3078],{"type":58,"tag":73,"props":3067,"children":3068},{},[3069],{"type":64,"value":3070},"Duplicate \u002F stale-audit suppression is handled upstream",{"type":64,"value":3072},", not by\nthis skill: the ",{"type":58,"tag":81,"props":3074,"children":3076},{"className":3075},[],[3077],{"type":64,"value":104},{"type":64,"value":3079}," skill LINKs\u002FSKIPs\nduplicate incident events and concurrent audits, and the\nperiodic-audit Lambda only invokes an investigation when a real issue\nis present. The former RCA \"stale-evidence\" rules (3 \u002F 3b) were\nremoved and are not reintroduced.",{"type":58,"tag":67,"props":3081,"children":3082},{},[3083,3102,3104,3109,3111],{"type":58,"tag":73,"props":3084,"children":3085},{},[3086,3088,3093,3095,3100],{"type":64,"value":3087},"Rule 2 (",{"type":58,"tag":81,"props":3089,"children":3091},{"className":3090},[],[3092],{"type":64,"value":431},{"type":64,"value":3094},") closes the loop on prior ",{"type":58,"tag":81,"props":3096,"children":3098},{"className":3097},[],[3099],{"type":64,"value":423},{"type":64,"value":3101}," verdicts.",{"type":64,"value":3103},"\nA previous ",{"type":58,"tag":81,"props":3105,"children":3107},{"className":3106},[],[3108],{"type":64,"value":423},{"type":64,"value":3110}," verdict promised a re-check; this rule provides\nthat re-check via the scheduled audit. The verdict description\nshould explicitly say \"Incident detected at ",{"type":58,"tag":3112,"props":3113,"children":3114},"t0",{},[3115,3117],{"type":64,"value":3116}," is now resolved.\nTotal auto-recovery time: ",{"type":58,"tag":3118,"props":3119,"children":3120},"duration",{},[3121],{"type":64,"value":3122},". No operator action required.\"\nThis is the closure email the operator needs.",{"type":58,"tag":67,"props":3124,"children":3125},{},[3126,3138,3140,3145],{"type":58,"tag":73,"props":3127,"children":3128},{},[3129,3131,3136],{"type":64,"value":3130},"Rule 1 (",{"type":58,"tag":81,"props":3132,"children":3134},{"className":3133},[],[3135],{"type":64,"value":2455},{"type":64,"value":3137},") is the no-op\ncase.",{"type":64,"value":3139}," When the scheduled audit fires on a healthy cluster, emit a\nsingle minimal record acknowledging the audit ran. The email notifier\nfilters ",{"type":58,"tag":81,"props":3141,"children":3143},{"className":3142},[],[3144],{"type":64,"value":2691},{"type":64,"value":3146}," verdicts so no email is sent.",{"type":58,"tag":67,"props":3148,"children":3149},{},[3150,3162,3164,3169,3171,3176,3178,3183,3185],{"type":58,"tag":73,"props":3151,"children":3152},{},[3153,3155,3160],{"type":64,"value":3154},"Recurring-pattern verdicts (6–8) are ",{"type":58,"tag":81,"props":3156,"children":3158},{"className":3157},[],[3159],{"type":64,"value":2231},{"type":64,"value":3161}," even when the\nindividual incident is auto-recovering correctly.",{"type":64,"value":3163}," The reasoning is\nin your operational goals: HyperPod's resiliency repairs the symptom,\nbut a 3× recurrence of a categorized fault on the same IG is an\nunderlying cause (vendor \u002F capacity pool for ",{"type":58,"tag":81,"props":3165,"children":3167},{"className":3166},[],[3168],{"type":64,"value":1542},{"type":64,"value":3170},", code bug for\n",{"type":58,"tag":81,"props":3172,"children":3174},{"className":3173},[],[3175],{"type":64,"value":1549},{"type":64,"value":3177},", pool exhaustion for ",{"type":58,"tag":81,"props":3179,"children":3181},{"className":3180},[],[3182],{"type":64,"value":1556},{"type":64,"value":3184},",\netc.) that auto-recovery can't fix. The verdict explanation should\nexplicitly say \"this incident is auto-recovering, but the pattern\nacross the last ",{"type":58,"tag":3186,"props":3187,"children":3188},"n",{},[3189,3191,3196],{"type":64,"value":3190}," days warrants human attention\" so operators\nunderstand the verdict isn't about ",{"type":58,"tag":807,"props":3192,"children":3193},{},[3194],{"type":64,"value":3195},"this",{"type":64,"value":3197}," incident's resolution.",{"type":58,"tag":67,"props":3199,"children":3200},{},[3201,3206],{"type":58,"tag":73,"props":3202,"children":3203},{},[3204],{"type":64,"value":3205},"Time budgets are not hardcoded constants — they encode the\nmental-model doc's \"How long things take\" section.",{"type":64,"value":3207}," A single replace\ntakes 20–30 min; two attempts plus a slack gap = ~90 min. Don't change\nthese without updating the mental-model doc first.",{"type":58,"tag":286,"props":3209,"children":3211},{"id":3210},"phase-4-report-using-devops-agents-native-schema",[3212],{"type":64,"value":3213},"Phase 4 — Report (using DevOps Agent's native schema)",{"type":58,"tag":67,"props":3215,"children":3216},{},[3217,3219,3225,3227,3233,3235,3241,3243,3248],{"type":64,"value":3218},"DevOps Agent's investigation output is structured: the terminal tool\nis ",{"type":58,"tag":81,"props":3220,"children":3222},{"className":3221},[],[3223],{"type":64,"value":3224},"write_final_investigation_report",{"type":64,"value":3226},", and the agent emits ",{"type":58,"tag":81,"props":3228,"children":3230},{"className":3229},[],[3231],{"type":64,"value":3232},"symptom",{"type":64,"value":3234},"\nand ",{"type":58,"tag":81,"props":3236,"children":3238},{"className":3237},[],[3239],{"type":64,"value":3240},"finding",{"type":64,"value":3242}," records along the way that get serialized into the\nfinal report. ",{"type":58,"tag":73,"props":3244,"children":3245},{},[3246],{"type":64,"value":3247},"Author the verdict and timeline into that schema\ndirectly — do NOT invent a separate four-section markdown format",{"type":64,"value":3249},"\n(it would be at the agent's mercy during serialization and may not\nsurvive into the final report).",{"type":58,"tag":67,"props":3251,"children":3252},{},[3253],{"type":64,"value":3254},"The agent's schema supports these record types:",{"type":58,"tag":298,"props":3256,"children":3257},{},[3258,3274],{"type":58,"tag":302,"props":3259,"children":3260},{},[3261],{"type":58,"tag":306,"props":3262,"children":3263},{},[3264,3269],{"type":58,"tag":310,"props":3265,"children":3266},{},[3267],{"type":64,"value":3268},"Record type",{"type":58,"tag":310,"props":3270,"children":3271},{},[3272],{"type":64,"value":3273},"Used for",{"type":58,"tag":326,"props":3275,"children":3276},{},[3277,3300,3323,3346,3369],{"type":58,"tag":306,"props":3278,"children":3279},{},[3280,3288],{"type":58,"tag":333,"props":3281,"children":3282},{},[3283],{"type":58,"tag":81,"props":3284,"children":3286},{"className":3285},[],[3287],{"type":64,"value":3232},{"type":58,"tag":333,"props":3289,"children":3290},{},[3291,3293,3298],{"type":64,"value":3292},"Observable state the operator would notice (a node went unhealthy, an investigation was triggered, the cluster is in ",{"type":58,"tag":81,"props":3294,"children":3296},{"className":3295},[],[3297],{"type":64,"value":217},{"type":64,"value":3299}," state). First-class; survives serialization with title + description verbatim.",{"type":58,"tag":306,"props":3301,"children":3302},{},[3303,3318],{"type":58,"tag":333,"props":3304,"children":3305},{},[3306,3311,3312],{"type":58,"tag":81,"props":3307,"children":3309},{"className":3308},[],[3310],{"type":64,"value":3240},{"type":64,"value":2675},{"type":58,"tag":81,"props":3313,"children":3315},{"className":3314},[],[3316],{"type":64,"value":3317},"finding_type: \"root_cause\"",{"type":58,"tag":333,"props":3319,"children":3320},{},[3321],{"type":64,"value":3322},"An identified hardware\u002Fsoftware root cause that cascades to one or more symptoms.",{"type":58,"tag":306,"props":3324,"children":3325},{},[3326,3341],{"type":58,"tag":333,"props":3327,"children":3328},{},[3329,3334,3335],{"type":58,"tag":81,"props":3330,"children":3332},{"className":3331},[],[3333],{"type":64,"value":3240},{"type":64,"value":2675},{"type":58,"tag":81,"props":3336,"children":3338},{"className":3337},[],[3339],{"type":64,"value":3340},"finding_type: \"cause\"",{"type":58,"tag":333,"props":3342,"children":3343},{},[3344],{"type":64,"value":3345},"An intermediate cause linking root_cause to symptoms.",{"type":58,"tag":306,"props":3347,"children":3348},{},[3349,3364],{"type":58,"tag":333,"props":3350,"children":3351},{},[3352,3357,3358],{"type":58,"tag":81,"props":3353,"children":3355},{"className":3354},[],[3356],{"type":64,"value":3240},{"type":64,"value":2675},{"type":58,"tag":81,"props":3359,"children":3361},{"className":3360},[],[3362],{"type":64,"value":3363},"finding_type: \"hypothesis\"",{"type":58,"tag":333,"props":3365,"children":3366},{},[3367],{"type":64,"value":3368},"An unverified explanation.",{"type":58,"tag":306,"props":3370,"children":3371},{},[3372,3381],{"type":58,"tag":333,"props":3373,"children":3374},{},[3375],{"type":58,"tag":81,"props":3376,"children":3378},{"className":3377},[],[3379],{"type":64,"value":3380},"investigation_gaps[]",{"type":58,"tag":333,"props":3382,"children":3383},{},[3384],{"type":64,"value":3385},"What the agent could not verify and would need the operator to confirm.",{"type":58,"tag":67,"props":3387,"children":3388},{},[3389],{"type":58,"tag":73,"props":3390,"children":3391},{},[3392],{"type":64,"value":3393},"Emit exactly these records, in this order:",{"type":58,"tag":493,"props":3395,"children":3396},{},[3397,3635,3660,3681,3757],{"type":58,"tag":251,"props":3398,"children":3399},{},[3400,3410,3412,3418,3420,3426,3428,3433,3434,3439,3440,3445,3446,3451,3452,3457,3459,3465,3467,3480,3482,3487,3489,3494,3496,3546,3549,3554,3556,3559,3561,3570,3573,3585,3587,3592,3594,3600,3601,3607,3608,3614,3616,3619,3625,3627,3633],{"type":58,"tag":73,"props":3401,"children":3402},{},[3403,3405],{"type":64,"value":3404},"First ",{"type":58,"tag":81,"props":3406,"children":3408},{"className":3407},[],[3409],{"type":64,"value":3232},{"type":64,"value":3411}," — titled\n",{"type":58,"tag":81,"props":3413,"children":3415},{"className":3414},[],[3416],{"type":64,"value":3417},"Triage verdict: \u003Cverdict-name> :: \u003Csignature-set>",{"type":64,"value":3419}," where\n",{"type":58,"tag":81,"props":3421,"children":3423},{"className":3422},[],[3424],{"type":64,"value":3425},"\u003Cverdict-name>",{"type":64,"value":3427}," is one of ",{"type":58,"tag":81,"props":3429,"children":3431},{"className":3430},[],[3432],{"type":64,"value":2691},{"type":64,"value":148},{"type":58,"tag":81,"props":3435,"children":3437},{"className":3436},[],[3438],{"type":64,"value":2550},{"type":64,"value":733},{"type":58,"tag":81,"props":3441,"children":3443},{"className":3442},[],[3444],{"type":64,"value":2910},{"type":64,"value":148},{"type":58,"tag":81,"props":3447,"children":3449},{"className":3448},[],[3450],{"type":64,"value":3056},{"type":64,"value":148},{"type":58,"tag":81,"props":3453,"children":3455},{"className":3454},[],[3456],{"type":64,"value":2652},{"type":64,"value":3458},",\nor ",{"type":58,"tag":81,"props":3460,"children":3462},{"className":3461},[],[3463],{"type":64,"value":3464},"Escalate — \u003Creason>",{"type":64,"value":3466},". ",{"type":58,"tag":73,"props":3468,"children":3469},{},[3470,3472,3478],{"type":64,"value":3471},"The ",{"type":58,"tag":81,"props":3473,"children":3475},{"className":3474},[],[3476],{"type":64,"value":3477},"\u003Csignature-set>",{"type":64,"value":3479}," suffix is\nrequired for any verdict that names specific fault evidence",{"type":64,"value":3481},"\n(all Escalate \u002F Monitor \u002F Resolved variants) — it's a stable,\nsorted-and-deduplicated representation of the ",{"type":58,"tag":81,"props":3483,"children":3485},{"className":3484},[],[3486],{"type":64,"value":1760},{"type":64,"value":3488},"\ncomputed in Phase 2d, formatted as a comma-separated list of the\n",{"type":58,"tag":81,"props":3490,"children":3492},{"className":3491},[],[3493],{"type":64,"value":1715},{"type":64,"value":3495}," signature strings (per Phase 2b) that drove the\nverdict. Verdict titles use the full concatenated fault-content\nsignature, NOT a hard-coded category name. Examples (line-wrapped\nhere for readability; the actual titles are single-line):",{"type":58,"tag":247,"props":3497,"children":3498},{},[3499,3508,3517,3526,3535],{"type":58,"tag":251,"props":3500,"children":3501},{},[3502],{"type":58,"tag":81,"props":3503,"children":3505},{"className":3504},[],[3506],{"type":64,"value":3507},"Triage verdict: Escalate — recurring fault pattern :: (worker4:Description: Failed to provision EC2 Instance ... FailureMessage: We currently do not have sufficient capacity to launch new ml.g5.8xlarge instances. Please try again.)",{"type":58,"tag":251,"props":3509,"children":3510},{},[3511],{"type":58,"tag":81,"props":3512,"children":3514},{"className":3513},[],[3515],{"type":64,"value":3516},"Triage verdict: Monitor — first attempt :: (worker2:Description: Instance i-XXXX is unhealthy. HyperPod Health Monitoring Agent (HMA) has detected fault type NvidiaGPUUnhealthy on this node ...)",{"type":58,"tag":251,"props":3518,"children":3519},{},[3520],{"type":58,"tag":81,"props":3521,"children":3523},{"className":3522},[],[3524],{"type":64,"value":3525},"Triage verdict: Resolved — auto-recovery succeeded :: (worker2:Description: Instance i-XXXX is unhealthy ...)",{"type":58,"tag":251,"props":3527,"children":3528},{},[3529],{"type":58,"tag":81,"props":3530,"children":3532},{"className":3531},[],[3533],{"type":64,"value":3534},"Triage verdict: Escalate — fleet-wide instability :: (worker2:..., worker3:..., worker4:...)",{"type":58,"tag":251,"props":3536,"children":3537},{},[3538,3544],{"type":58,"tag":81,"props":3539,"children":3541},{"className":3540},[],[3542],{"type":64,"value":3543},"Triage verdict: Suppress — periodic audit, no open incidents",{"type":64,"value":3545}," (no suffix when there's no signature set)",{"type":58,"tag":855,"props":3547,"children":3548},{},[],{"type":58,"tag":73,"props":3550,"children":3551},{},[3552],{"type":64,"value":3553},"Purpose",{"type":64,"value":3555},": the DevOps Agent platform's automatic task-dedup uses\ntitle equality within a ~30 minute window. Encoding the signature\nset in the title means a NEW signature (a new Xid type on the same\nIG, or the same Xid spreading to a new IG) produces a different\ntitle and the platform does NOT dedup it — the new investigation\nruns. Conversely, identical signature sets produce identical titles\nand platform dedup quietly absorbs the duplicate audit.",{"type":58,"tag":855,"props":3557,"children":3558},{},[],{"type":64,"value":3560},"The description is the prose justification in the shape:",{"type":58,"tag":880,"props":3562,"children":3565},{"className":3563,"code":3564,"language":64},[1230],"Verdict: \u003Cname>\n\nSummary:\n\u003CA single plain-English paragraph (2–4 sentences) that flows naturally and\nweaves together THREE things, in this order: (1) WHAT HAPPENED — the specific\nobserved problem: fault type, affected instance group \u002F node \u002F instance type,\nand how many times \u002F how long (never just the generic verdict category);\n(2) THE LIKELY CAUSE — the most probable root cause inferred from the\nFailureMessage \u002F event content and cluster context (not a hardcoded category);\n(3) THE RECOMMENDED ACTION — concrete operator next step(s) appropriate to the\ncause. Write it as prose, NOT as labeled fields or bullets. Lead the paragraph\nwith \"what happened\" so the first sentence works as a standalone headline.\nExample:\n\"Repeated capacity errors are failing to provision ml.p5.48xlarge for instance\ngroup 'faulttest' on cluster my-slurm-cluster — 5 attempts in the last 40 minutes. The\nlikely cause is that on-demand capacity for ml.p5.48xlarge is unavailable in\nthis Availability Zone and the instance group is not backed by a training plan\nor reserved capacity, so every Continuous-Provisioning retry hits the same\nwall. Recommended: launch in an AZ\u002FRegion where you hold reserved capacity for\nthis instance type or associate the group with a training plan; otherwise\nreduce the target count to stop the retry loop and request a capacity increase.\">\n\nWhat HyperPod is doing right now:\n\u003Cone paragraph plain English summary referencing the timeline>\n\nTimeline (UTC):\n\u003Cthe timeline reconstructed in Phase 2, one event per line, source-tagged>\n\nMost recent event at:\n\u003CUTC ISO 8601 timestamp of the most recent fault event in the 4-hour window.\nRecommended for all verdicts that contain a non-empty signature set so\noperators reading the report can correlate with their own timelines. Example:\n2026-06-30T01:36:49Z>\n\nNext re-check:\n\u003Conly for Monitor verdicts: UTC timestamp 30 min from now if \"first attempt\",\n15 min from now if \"elevated\">\n\nRecommended actions (operator runs these):\n\u003Conly for Escalate verdicts: explicit operator-runnable remediation steps —\nsee the per-verdict guidance below and the SSM GPU-UUID step for\nrecurring-pattern verdicts. Omit this heading for Suppress \u002F Monitor \u002F\nResolved verdicts.>\n",[3566],{"type":58,"tag":81,"props":3567,"children":3568},{"__ignoreMap":885},[3569],{"type":64,"value":3564},{"type":58,"tag":855,"props":3571,"children":3572},{},[],{"type":58,"tag":73,"props":3574,"children":3575},{},[3576,3577,3583],{"type":64,"value":3471},{"type":58,"tag":81,"props":3578,"children":3580},{"className":3579},[],[3581],{"type":64,"value":3582},"Summary:",{"type":64,"value":3584}," paragraph is REQUIRED for every non-Suppress verdict.",{"type":64,"value":3586}," Write\nit as ONE natural-language paragraph (2–4 sentences) that covers, in order,\nwhat happened → the likely cause → the recommended action. Do NOT use labeled\nfields, bullets, or JSON — just prose. Lead with \"what happened\" so the first\nsentence stands alone as a headline (the email notifier uses the paragraph as\nthe body summary and its first sentence as the subject). Name the ",{"type":58,"tag":807,"props":3588,"children":3589},{},[3590],{"type":64,"value":3591},"specific",{"type":64,"value":3593},"\nproblem (instance type, instance group, counts, AZ) — never restate the\ngeneric verdict category. Do NOT put ",{"type":58,"tag":81,"props":3595,"children":3597},{"className":3596},[],[3598],{"type":64,"value":3599},"[direct]",{"type":64,"value":773},{"type":58,"tag":81,"props":3602,"children":3604},{"className":3603},[],[3605],{"type":64,"value":3606},"[proxy]",{"type":64,"value":773},{"type":58,"tag":81,"props":3609,"children":3611},{"className":3610},[],[3612],{"type":64,"value":3613},"[unverified]",{"type":64,"value":3615},"\nconfidence tags in this paragraph; those belong on the detailed evidence\nclaims later in the description. Suppress verdicts may omit it (no email).",{"type":58,"tag":855,"props":3617,"children":3618},{},[],{"type":58,"tag":81,"props":3620,"children":3622},{"className":3621},[],[3623],{"type":64,"value":3624},"related_resources",{"type":64,"value":3626}," on this symptom: ",{"type":58,"tag":81,"props":3628,"children":3630},{"className":3629},[],[3631],{"type":64,"value":3632},"[\"HyperPod cluster \u003Cname>\"]",{"type":64,"value":3634},"\nplus any affected instance IDs.",{"type":58,"tag":251,"props":3636,"children":3637},{},[3638,3650,3652,3658],{"type":58,"tag":73,"props":3639,"children":3640},{},[3641,3643,3648],{"type":64,"value":3642},"Additional ",{"type":58,"tag":81,"props":3644,"children":3646},{"className":3645},[],[3647],{"type":64,"value":3232},{"type":64,"value":3649}," records",{"type":64,"value":3651}," — one per observable failure\ncondition (e.g. \"Instance i-xxxx marked unhealthy with\nNvidiaGPUUnhealthy\"). These are the per-resource symptoms the\nagent would naturally produce; keep them as separate records so\nthey cross-link with ",{"type":58,"tag":81,"props":3653,"children":3655},{"className":3654},[],[3656],{"type":64,"value":3657},"cascades_to",{"type":64,"value":3659}," from the findings.",{"type":58,"tag":251,"props":3661,"children":3662},{},[3663,3672,3674,3680],{"type":58,"tag":73,"props":3664,"children":3665},{},[3666,3671],{"type":58,"tag":81,"props":3667,"children":3669},{"className":3668},[],[3670],{"type":64,"value":3240},{"type":64,"value":3649},{"type":64,"value":3673}," as appropriate (root cause, intermediate\ncause, hypothesis). Cross-link via ",{"type":58,"tag":81,"props":3675,"children":3677},{"className":3676},[],[3678],{"type":64,"value":3679},"cascades_to: [\u003Csymptom-id>]",{"type":64,"value":361},{"type":58,"tag":251,"props":3682,"children":3683},{},[3684,3692,3694,3699,3701,3704,3723,3725,3731,3733,3739,3741,3747,3749,3755],{"type":58,"tag":73,"props":3685,"children":3686},{},[3687],{"type":58,"tag":81,"props":3688,"children":3690},{"className":3689},[],[3691],{"type":64,"value":3380},{"type":64,"value":3693}," — anything Phase 1 couldn't reach.\nExamples: ",{"type":58,"tag":81,"props":3695,"children":3697},{"className":3696},[],[3698],{"type":64,"value":225},{"type":64,"value":3700}," unavailable on a Slurm cluster\nwithout Continuous Provisioning; HMA CloudWatch log group not\nyet populated; no SSM access from the agent (always include this\none with a note that operator can confirm via the suggested SSM\ncommand).",{"type":58,"tag":855,"props":3702,"children":3703},{},[],{"type":58,"tag":73,"props":3705,"children":3706},{},[3707,3709,3715,3716,3721],{"type":64,"value":3708},"Emit an ",{"type":58,"tag":81,"props":3710,"children":3712},{"className":3711},[],[3713],{"type":64,"value":3714},"investigation_gap",{"type":64,"value":2142},{"type":58,"tag":81,"props":3717,"children":3719},{"className":3718},[],[3720],{"type":64,"value":1457},{"type":64,"value":3722}," returned\nno FailureMessage for any Error\u002FWarn event",{"type":64,"value":3724}," (i.e. the API returned\n",{"type":58,"tag":81,"props":3726,"children":3728},{"className":3727},[],[3729],{"type":64,"value":3730},"EventDetails.EventDetails.EventMetadata.Instance",{"type":64,"value":3732}," as null or with\nonly ",{"type":58,"tag":81,"props":3734,"children":3736},{"className":3735},[],[3737],{"type":64,"value":3738},"NodeLogicalId",{"type":64,"value":3740},"). Title: ",{"type":58,"tag":81,"props":3742,"children":3744},{"className":3743},[],[3745],{"type":64,"value":3746},"FailureMessage missing from N event(s)",{"type":64,"value":3748},".\nDescription: list the affected ",{"type":58,"tag":81,"props":3750,"children":3752},{"className":3751},[],[3753],{"type":64,"value":3754},"EventId",{"type":64,"value":3756},"s + timestamps. This surfaces\nthe known HyperPod bug where FailureMessage is populated in the API\nresponse for capacity errors but omitted for other fault categories —\noperators reviewing the report should escalate to the HyperPod team\nwith those EventIds as evidence.",{"type":58,"tag":251,"props":3758,"children":3759},{},[3760,3768,3770,3776,3778,3784,3786,3792,3793,3799,3800,3806,3808,3813],{"type":58,"tag":73,"props":3761,"children":3762},{},[3763],{"type":58,"tag":81,"props":3764,"children":3766},{"className":3765},[],[3767],{"type":64,"value":3224},{"type":64,"value":3769}," — listing the verdict\nsymptom FIRST in ",{"type":58,"tag":81,"props":3771,"children":3773},{"className":3772},[],[3774],{"type":64,"value":3775},"symptoms[]",{"type":64,"value":3777},", then the per-resource symptoms.\nFindings listed under their ",{"type":58,"tag":81,"props":3779,"children":3781},{"className":3780},[],[3782],{"type":64,"value":3783},"finding_type",{"type":64,"value":3785}," arrays (",{"type":58,"tag":81,"props":3787,"children":3789},{"className":3788},[],[3790],{"type":64,"value":3791},"root_cause[]",{"type":64,"value":733},{"type":58,"tag":81,"props":3794,"children":3796},{"className":3795},[],[3797],{"type":64,"value":3798},"cause[]",{"type":64,"value":148},{"type":58,"tag":81,"props":3801,"children":3803},{"className":3802},[],[3804],{"type":64,"value":3805},"hypothesis[]",{"type":64,"value":3807},"). ",{"type":58,"tag":81,"props":3809,"children":3811},{"className":3810},[],[3812],{"type":64,"value":3380},{"type":64,"value":3814}," populated from\nstep 4.",{"type":58,"tag":286,"props":3816,"children":3818},{"id":3817},"critical-the-first-symptom-is-the-verdict-symptom",[3819],{"type":64,"value":3820},"CRITICAL: the FIRST symptom is the verdict symptom",{"type":58,"tag":67,"props":3822,"children":3823},{},[3824,3826,3832,3834,3840],{"type":64,"value":3825},"Downstream automation (email notifier, dashboards, dedup) keys off the\nFIRST symptom's title matching ",{"type":58,"tag":81,"props":3827,"children":3829},{"className":3828},[],[3830],{"type":64,"value":3831},"Triage verdict: \u003Cname> :: \u003Csignature>",{"type":64,"value":3833},".\nIf you emit a descriptively-titled symptom first (e.g.\n",{"type":58,"tag":81,"props":3835,"children":3837},{"className":3836},[],[3838],{"type":64,"value":3839},"\"worker1 lifecycle script execution failures across multiple nodes on my-cluster\"",{"type":64,"value":3841},")\nthe verdict is invisible to the pipeline and no email is sent.",{"type":58,"tag":67,"props":3843,"children":3844},{},[3845,3847,3874,3876,3881],{"type":64,"value":3846},"This has been observed to fail in RCA runs. ",{"type":58,"tag":73,"props":3848,"children":3849},{},[3850,3852,3857,3859,3865,3867,3873],{"type":64,"value":3851},"The first\n",{"type":58,"tag":81,"props":3853,"children":3855},{"className":3854},[],[3856],{"type":64,"value":3232},{"type":64,"value":3858}," record you emit MUST have ",{"type":58,"tag":81,"props":3860,"children":3862},{"className":3861},[],[3863],{"type":64,"value":3864},"title",{"type":64,"value":3866}," beginning with\n",{"type":58,"tag":81,"props":3868,"children":3870},{"className":3869},[],[3871],{"type":64,"value":3872},"Triage verdict:",{"type":64,"value":361},{"type":64,"value":3875}," Descriptive titles are for the ",{"type":58,"tag":807,"props":3877,"children":3878},{},[3879],{"type":64,"value":3880},"second",{"type":64,"value":3882}," and later\nsymptom records, which capture per-resource observations.",{"type":58,"tag":286,"props":3884,"children":3886},{"id":3885},"few-shot-examples-of-the-first-symptom-record",[3887],{"type":64,"value":3888},"Few-shot examples of the first symptom record",{"type":58,"tag":67,"props":3890,"children":3891},{},[3892],{"type":64,"value":3893},"Copy the shape and titles exactly. Substitute your investigation's\nconcrete data.",{"type":58,"tag":67,"props":3895,"children":3896},{},[3897],{"type":58,"tag":73,"props":3898,"children":3899},{},[3900],{"type":64,"value":3901},"Example A — Escalate for a recurring capacity fault:",{"type":58,"tag":880,"props":3903,"children":3906},{"className":3904,"code":3905,"language":64},[1230],"title: \"Triage verdict: Escalate — recurring fault pattern :: (worker4:Description: Failed to provision EC2 Instance in Cluster prod-01 and InstanceGroup worker4. FailureMessage: We currently do not have sufficient capacity to launch new ml.g5.8xlarge instances. Please try again.)\"\n\ndescription: |\n  Verdict: Escalate — recurring fault pattern\n\n  Summary:\n  Repeated capacity errors are failing to provision ml.g5.8xlarge for instance group \"worker4\" — 4 failed replacements in the last 24 hours. The likely cause is that on-demand capacity for ml.g5.8xlarge is unavailable in us-west-2 for this cluster and worker4 is not backed by a training plan or reserved capacity, so each Continuous-Provisioning retry hits the same capacity wall. Recommended: launch worker4 in an AZ\u002FRegion where you hold reserved capacity or a training plan for ml.g5.8xlarge, or switch to a SKU with availability (e.g. ml.g6.8xlarge); alternatively lower worker4's target count to stop the retry loop while you request a capacity increase.\n\n  What HyperPod is doing right now:\n  HyperPod is auto-retrying, but the same insufficient-capacity error has driven 4 replacements on worker4 in the last 24 hours. Continuous Provisioning is looping without progress; each new attempt hits the same on-demand capacity wall for ml.g5.8xlarge in us-west-2. Operator action is required — the pattern will not self-resolve.\n\n  Timeline (UTC):\n  2026-07-08T15:12:03Z  aws.sagemaker Cluster Event Error  Failed to provision EC2 Instance in worker4 (attempt 1)\n  2026-07-08T15:31:47Z  aws.sagemaker Cluster Event Error  Failed to provision EC2 Instance in worker4 (attempt 2)\n  2026-07-08T15:53:22Z  aws.sagemaker Cluster Event Error  Failed to provision EC2 Instance in worker4 (attempt 3)\n  2026-07-08T16:14:59Z  aws.sagemaker Cluster Event Error  Failed to provision EC2 Instance in worker4 (attempt 4)\n\n  Most recent event at:\n  2026-07-08T16:14:59Z\n",[3907],{"type":58,"tag":81,"props":3908,"children":3909},{"__ignoreMap":885},[3910],{"type":64,"value":3905},{"type":58,"tag":67,"props":3912,"children":3913},{},[3914],{"type":58,"tag":73,"props":3915,"children":3916},{},[3917],{"type":64,"value":3918},"Example B — Escalate for coordinated lifecycle-script failures across multiple instances:",{"type":58,"tag":880,"props":3920,"children":3923},{"className":3921,"code":3922,"language":64},[1230],"title: \"Triage verdict: Escalate — coordinated lifecycle-script failure :: (worker1:Description: Lifecycle scripts did not run successfully. Ensure the scripts exist in provided S3 path, are accessible, and run without errors.)\"\n\ndescription: |\n  Verdict: Escalate — coordinated lifecycle-script failure\n\n  Summary:\n  Every new worker1 instance on cluster my-cluster is failing bootstrap with the same lifecycle-script (LCS) execution error — 6 instances failed across two waves at 2026-07-08T19:21Z and 19:32Z. The likely cause is a bug or missing\u002Finaccessible file in the on_create script in the instance group's S3 path, so each freshly provisioned node hits the identical failure. Recommended: inspect the LCS log stream for one of the affected instances to find the failing command, then fix on_create.sh (or on_create_main.sh) in the cluster's S3 bucket; the retry loop clears on the next attempt once the script succeeds.\n\n  What HyperPod is doing right now:\n  HyperPod is retrying, but every new worker1 instance is failing bootstrap with the same LCS execution error. 3 instances (i-0aaaa1111bbbb2222, i-0cccc3333dddd4444, i-0eeee5555ffff6666) failed simultaneously at 2026-07-08T19:21Z, then 3 more at 2026-07-08T19:32Z with the same error. Continuous Provisioning + Automatic NodeRecovery will keep respawning these logical nodes at ~10-minute intervals until an operator fixes the LCS.\n\n  Timeline (UTC):\n  2026-07-08T19:20:01Z  aws.sagemaker Cluster Event Info  Instance lifecycle script execution for i-0aaaa1... has Started\n  2026-07-08T19:20:05Z  aws.sagemaker Cluster Event Info  Instance lifecycle script execution for i-0cccc3... has Started\n  2026-07-08T19:20:07Z  aws.sagemaker Cluster Event Info  Instance lifecycle script execution for i-0eeee5... has Started\n  2026-07-08T19:21:02Z  aws.sagemaker Cluster Event Error Lifecycle scripts did not run successfully (i-0aaaa1...)\n  2026-07-08T19:21:05Z  aws.sagemaker Cluster Event Error Lifecycle scripts did not run successfully (i-0cccc3...)\n  2026-07-08T19:21:08Z  aws.sagemaker Cluster Event Error Lifecycle scripts did not run successfully (i-0eeee5...)\n  2026-07-08T19:31:53Z  aws.sagemaker Cluster Event Error Lifecycle scripts did not run successfully (i-07777a...)\n  2026-07-08T19:32:04Z  aws.sagemaker Cluster Event Error Lifecycle scripts did not run successfully (i-08888b...)\n  2026-07-08T19:32:13Z  aws.sagemaker Cluster Event Error Lifecycle scripts did not run successfully (i-09999c...)\n\n  Most recent event at:\n  2026-07-08T19:32:13Z\n\n  Recommended actions (operator runs these):\n  Inspect the LCS log stream \u002Faws\u002Fsagemaker\u002FClusters\u002Fmy-cluster\u002F\u003Ccluster-id>\u002FLifecycleConfig\u002Fworker1\u002F\u003Cinstance-id> for any of the affected instances to identify the failing command. Fix on_create.sh (or on_create_main.sh) in s3:\u002F\u002F\u003Cyour-lcs-bucket>. Once fixed the retry loop will clear on its next attempt.\n\nrelated_resources: [\"HyperPod cluster my-cluster\", \"i-0aaaa1111bbbb2222\", \"i-0cccc3333dddd4444\", \"i-0eeee5555ffff6666\"]\n",[3924],{"type":58,"tag":81,"props":3925,"children":3926},{"__ignoreMap":885},[3927],{"type":64,"value":3922},{"type":58,"tag":67,"props":3929,"children":3930},{},[3931],{"type":58,"tag":73,"props":3932,"children":3933},{},[3934],{"type":64,"value":3935},"Example C — Monitor (first attempt in flight):",{"type":58,"tag":880,"props":3937,"children":3940},{"className":3938,"code":3939,"language":64},[1230],"title: \"Triage verdict: Monitor — first attempt :: (worker2:Description: Instance i-0abcdef1234567890 is unhealthy. HyperPod Health Monitoring Agent (HMA) has detected fault type NvidiaGPUUnhealthy on this node and is unhealthy. Repair action: Replace.)\"\n\ndescription: |\n  Verdict: Monitor — first attempt\n\n  Summary:\n  HMA flagged instance i-0abcdef1234567890 in instance group \"worker2\" as unhealthy (NvidiaGPUUnhealthy) and requested a Replace, and HyperPod has already started the first replacement attempt. The likely cause is a transient or isolated GPU fault on that single node, which HyperPod's automatic recovery is equipped to handle on its own. No operator action is needed right now — the replacement is expected to reach InService within 20-30 min; the next scheduled audit will re-check and emit a Resolved closure if it succeeds.\n\n  What HyperPod is doing right now:\n  HMA has flagged i-0abcdef1234567890 as unhealthy (NvidiaGPUUnhealthy) and requested Replace. HyperPod has started the replacement and this is the first attempt in the chain. Expected wall-clock: 20-30 min for the new instance to reach InService.\n\n  Timeline (UTC):\n  2026-07-08T18:45:12Z  aws.sagemaker Cluster Node Health Event  Instance i-0abcdef1234567890 unhealthy: NvidiaGPUUnhealthy\n  2026-07-08T18:45:15Z  aws.sagemaker Cluster Event Info         Instance deletion is starting as part of instance replacement\n\n  Most recent event at:\n  2026-07-08T18:45:15Z\n\n  Next re-check:\n  2026-07-08T19:15:15Z\n",[3941],{"type":58,"tag":81,"props":3942,"children":3943},{"__ignoreMap":885},[3944],{"type":64,"value":3939},{"type":58,"tag":67,"props":3946,"children":3947},{},[3948],{"type":58,"tag":73,"props":3949,"children":3950},{},[3951],{"type":64,"value":3952},"Example D — Suppress (audit found nothing):",{"type":58,"tag":880,"props":3954,"children":3957},{"className":3955,"code":3956,"language":64},[1230],"title: \"Triage verdict: Suppress — periodic audit, no open incidents\"\n\ndescription: |\n  Verdict: Suppress — periodic audit, no open incidents\n\n  What HyperPod is doing right now:\n  Scheduled audit at 2026-07-08T19:00Z; scanned the last 4 hours of cluster events. No fault events, no open Monitor chains, cluster status InService, all instance groups at target. Nothing to investigate.\n",[3958],{"type":58,"tag":81,"props":3959,"children":3960},{"__ignoreMap":885},[3961],{"type":64,"value":3956},{"type":58,"tag":286,"props":3963,"children":3965},{"id":3964},"anti-example-do-not-do-this",[3966],{"type":64,"value":3967},"Anti-example — do NOT do this",{"type":58,"tag":67,"props":3969,"children":3970},{},[3971],{"type":64,"value":3972},"If your first symptom looks like this, downstream automation is broken:",{"type":58,"tag":880,"props":3974,"children":3977},{"className":3975,"code":3976,"language":64},[1230],"title: \"worker1 lifecycle script execution failures across multiple nodes on my-cluster\"    ← WRONG: missing \"Triage verdict:\" prefix\ndescription: \"HyperPod cluster my-cluster emitted coordinated lifecycle-script execution failures...\"\n",[3978],{"type":58,"tag":81,"props":3979,"children":3980},{"__ignoreMap":885},[3981],{"type":64,"value":3976},{"type":58,"tag":67,"props":3983,"children":3984},{},[3985,3987,3991],{"type":64,"value":3986},"The content is fine as a ",{"type":58,"tag":807,"props":3988,"children":3989},{},[3990],{"type":64,"value":3880},{"type":64,"value":3992}," symptom record. But the FIRST symptom\nmust be the verdict.",{"type":58,"tag":67,"props":3994,"children":3995},{},[3996,4008],{"type":58,"tag":73,"props":3997,"children":3998},{},[3999,4001,4006],{"type":64,"value":4000},"For ",{"type":58,"tag":81,"props":4002,"children":4004},{"className":4003},[],[4005],{"type":64,"value":2691},{"type":64,"value":4007}," verdict:",{"type":64,"value":4009}," still emit the verdict symptom (so the\noperator can audit what was suppressed and why), but skip per-resource\nsymptoms and findings — there's nothing to root-cause.",{"type":58,"tag":67,"props":4011,"children":4012},{},[4013,4024],{"type":58,"tag":73,"props":4014,"children":4015},{},[4016,4017,4022],{"type":64,"value":4000},{"type":58,"tag":81,"props":4018,"children":4020},{"className":4019},[],[4021],{"type":64,"value":423},{"type":64,"value":4023}," verdicts:",{"type":64,"value":4025}," the verdict symptom's description tells\nthe human \"no action needed; HyperPod is recovering; expected\ncompletion by HH:MM UTC. You will be notified again only if the\nsituation changes.\" This is the key UX improvement — silence is bad;\n\"we're watching and here's why we're not alarming you\" is good.",{"type":58,"tag":67,"props":4027,"children":4028},{},[4029,4039,4041,4047],{"type":58,"tag":73,"props":4030,"children":4031},{},[4032,4033,4038],{"type":64,"value":4000},{"type":58,"tag":81,"props":4034,"children":4036},{"className":4035},[],[4037],{"type":64,"value":2231},{"type":64,"value":4023},{"type":64,"value":4040}," include explicit operator-runnable\nremediation in the verdict symptom's description, under a\n",{"type":58,"tag":81,"props":4042,"children":4044},{"className":4043},[],[4045],{"type":64,"value":4046},"Recommended actions (operator runs these):",{"type":64,"value":4048}," heading. The agent\ncannot execute these; the operator must.",{"type":58,"tag":67,"props":4050,"children":4051},{},[4052,4057,4059,4064,4066,4071,4073,4078],{"type":58,"tag":73,"props":4053,"children":4054},{},[4055],{"type":64,"value":4056},"Confidence annotations:",{"type":64,"value":4058}," for every material claim in the verdict\ndescription, prefix with one of ",{"type":58,"tag":81,"props":4060,"children":4062},{"className":4061},[],[4063],{"type":64,"value":3599},{"type":64,"value":4065}," (observed via API\u002Flog\noutput the agent itself ran), ",{"type":58,"tag":81,"props":4067,"children":4069},{"className":4068},[],[4070],{"type":64,"value":3606},{"type":64,"value":4072}," (inferred from HMA\nclassification or a correlated source), or ",{"type":58,"tag":81,"props":4074,"children":4076},{"className":4075},[],[4077],{"type":64,"value":3613},{"type":64,"value":4079}," (would need\non-node SSM or AWS-internal data the agent can't reach). This\nreplaces the separate \"Confidence\" section.",{"type":58,"tag":67,"props":4081,"children":4082},{},[4083,4093,4095,4099,4100,4105,4107,4112,4114,4119],{"type":58,"tag":73,"props":4084,"children":4085},{},[4086,4091],{"type":58,"tag":81,"props":4087,"children":4089},{"className":4088},[],[4090],{"type":64,"value":3599},{"type":64,"value":4092}," requires direct observation, not inference.",{"type":64,"value":4094}," A claim\nlike \"all three replacements landed on the same physical host\" is\n",{"type":58,"tag":73,"props":4096,"children":4097},{},[4098],{"type":64,"value":2089},{"type":64,"value":98},{"type":58,"tag":81,"props":4101,"children":4103},{"className":4102},[],[4104],{"type":64,"value":3599},{"type":64,"value":4106}," even when three identical fault signatures are\nobserved — the customer surface (NodeId, InstanceId, ENI, K8s node\nname) does not expose physical hardware identity. Such claims are\n",{"type":58,"tag":81,"props":4108,"children":4110},{"className":4109},[],[4111],{"type":64,"value":3613},{"type":64,"value":4113}," at best. Mislabeling inference as ",{"type":58,"tag":81,"props":4115,"children":4117},{"className":4116},[],[4118],{"type":64,"value":3599},{"type":64,"value":4120}," is a\nserious failure mode because operators trust the annotation to mean\n\"the agent saw this in the data.\"",{"type":58,"tag":67,"props":4122,"children":4123},{},[4124,4129,4131,4136,4137,4142,4144,4149,4151,4156,4158,4163,4164,4169,4171,4176],{"type":58,"tag":73,"props":4125,"children":4126},{},[4127],{"type":64,"value":4128},"Hypothesis discipline for recurring-pattern verdicts\n(rules 6–8).",{"type":64,"value":4130}," When the verdict is one of ",{"type":58,"tag":81,"props":4132,"children":4134},{"className":4133},[],[4135],{"type":64,"value":2795},{"type":64,"value":148},{"type":58,"tag":81,"props":4138,"children":4140},{"className":4139},[],[4141],{"type":64,"value":2824},{"type":64,"value":4143},", or\n",{"type":58,"tag":81,"props":4145,"children":4147},{"className":4146},[],[4148],{"type":64,"value":2853},{"type":64,"value":4150},", the verdict description\nMUST enumerate at least ",{"type":58,"tag":73,"props":4152,"children":4153},{},[4154],{"type":64,"value":4155},"two",{"type":64,"value":4157}," competing hypotheses for the root\ncause, each labeled ",{"type":58,"tag":81,"props":4159,"children":4161},{"className":4160},[],[4162],{"type":64,"value":3613},{"type":64,"value":2353},{"type":58,"tag":81,"props":4165,"children":4167},{"className":4166},[],[4168],{"type":64,"value":3606},{"type":64,"value":4170},", and each paired with\na discriminating operator action. Do not commit to a single root\ncause without ",{"type":58,"tag":81,"props":4172,"children":4174},{"className":4173},[],[4175],{"type":64,"value":3599},{"type":64,"value":4177}," evidence. The required hypothesis classes\nare:",{"type":58,"tag":493,"props":4179,"children":4180},{},[4181,4191,4201],{"type":58,"tag":251,"props":4182,"children":4183},{},[4184,4189],{"type":58,"tag":73,"props":4185,"children":4186},{},[4187],{"type":64,"value":4188},"Software \u002F workload",{"type":64,"value":4190}," — the workload running on the IG triggers\nthe fault on whatever GPU it lands on (NCCL pattern, driver \u002F CUDA\nversion, application code path). Discriminator: change the\nworkload, or move the IG to a different node and see if the fault\nfollows.",{"type":58,"tag":251,"props":4192,"children":4193},{},[4194,4199],{"type":58,"tag":73,"props":4195,"children":4196},{},[4197],{"type":64,"value":4198},"Infrastructure path",{"type":64,"value":4200}," — an EFA fabric path, leaf switch, or\nshared network resource surfaces as GPU-level errors on workloads\nthat hit it. Discriminator: move the IG to a different subnet \u002F AZ.",{"type":58,"tag":251,"props":4202,"children":4203},{},[4204,4209],{"type":58,"tag":73,"props":4205,"children":4206},{},[4207],{"type":64,"value":4208},"Statistical hardware",{"type":64,"value":4210}," — a bad batch of the same SKU is over-\nrepresented in the capacity pool. Discriminator: open an AWS\nSupport case with the fault signature requesting hardware\nexclusion, or wait + retry later from a different time\u002Fpool.",{"type":58,"tag":67,"props":4212,"children":4213},{},[4214,4216,4221,4223,4227],{"type":64,"value":4215},"The verdict should NOT include \"every replacement is landing on the\nsame physical hardware\" as a stated cause — that's the explanation\n",{"type":58,"tag":73,"props":4217,"children":4218},{},[4219],{"type":64,"value":4220},"operators are conditioned to expect from on-prem clusters",{"type":64,"value":4222},", but\non HyperPod the EC2 instance is owned by the service account, the\nunderlying physical host is not exposed on the customer surface, and\nEC2 placement is non-deterministic per replacement. Read the\n\"Recurring fault signature does NOT prove physical-host affinity\"\nsection in ",{"type":58,"tag":185,"props":4224,"children":4225},{"href":187},[4226],{"type":64,"value":187},{"type":64,"value":4228},"\nbefore authoring this part of the verdict.",{"type":58,"tag":67,"props":4230,"children":4231},{},[4232,4237],{"type":58,"tag":73,"props":4233,"children":4234},{},[4235],{"type":64,"value":4236},"The GPU UUID check is the only way to confirm or refute\nphysical-host affinity, and it requires SSM (operator-only).",{"type":64,"value":4238}," The\nverdict's \"Recommended actions\" section MUST include this check as\nan explicit operator step whenever a recurring-pattern verdict is\nemitted (rules 6–8). The wording should be:",{"type":58,"tag":880,"props":4240,"children":4243},{"className":4241,"code":4242,"language":64},[1230],"N. Verify or refute \"same physical GPU\" by capturing GPU UUIDs.\n   The agent cannot run this check (requires SSM, which is outside\n   the DevOps Agent permission guardrail). For each affected\n   instance ID, run:\n\n       aws ssm start-session \\\n         --target sagemaker-cluster:\u003Ccluster-id>_\u003Cgroup>-\u003Cinstance-id> \\\n         --document-name AWS-StartNonInteractiveCommand \\\n         --parameters '{\"command\":[\"nvidia-smi -L\"]}'\n\n   Compare the UUID strings across the affected instances. If they\n   match, the same physical GPU is being recycled — that elevates\n   the \"statistical hardware\" hypothesis to [direct] evidence and\n   strengthens the case for an AWS Support exclusion request. If\n   they differ, \"same physical hardware\" is RULED OUT and the\n   investigation should pivot to the software\u002Fworkload and\n   infrastructure-path hypotheses instead.\n",[4244],{"type":58,"tag":81,"props":4245,"children":4246},{"__ignoreMap":885},[4247],{"type":64,"value":4242},{"type":58,"tag":67,"props":4249,"children":4250},{},[4251,4253,4259,4260,4266,4267,4273],{"type":64,"value":4252},"Replace ",{"type":58,"tag":81,"props":4254,"children":4256},{"className":4255},[],[4257],{"type":64,"value":4258},"\u003Ccluster-id>",{"type":64,"value":148},{"type":58,"tag":81,"props":4261,"children":4263},{"className":4262},[],[4264],{"type":64,"value":4265},"\u003Cgroup>",{"type":64,"value":148},{"type":58,"tag":81,"props":4268,"children":4270},{"className":4269},[],[4271],{"type":64,"value":4272},"\u003Cinstance-id>",{"type":64,"value":4274}," with the actual\nvalues from the affected instances in the timeline. List each\ninstance separately so the operator can run the commands in parallel.",{"type":58,"tag":193,"props":4276,"children":4278},{"id":4277},"inputs-the-skill-expects-from-the-trigger",[4279],{"type":64,"value":4280},"Inputs the skill expects from the trigger",{"type":58,"tag":67,"props":4282,"children":4283},{},[4284,4286,4291,4293,4299,4300,4306],{"type":64,"value":4285},"The webhook payload built by the bridge Lambda carries these fields. Per the\nflattening note in Phase 1 step 8, read them from the task ",{"type":58,"tag":81,"props":4287,"children":4289},{"className":4288},[],[4290],{"type":64,"value":1217},{"type":64,"value":4292}," at\nruntime — the ",{"type":58,"tag":81,"props":4294,"children":4296},{"className":4295},[],[4297],{"type":64,"value":4298},"data.metadata.*",{"type":64,"value":114},{"type":58,"tag":81,"props":4301,"children":4303},{"className":4302},[],[4304],{"type":64,"value":4305},"data.originalEvent.detail.*",{"type":64,"value":4307}," paths below\nname the logical source:",{"type":58,"tag":247,"props":4309,"children":4310},{},[4311,4322,4353,4364],{"type":58,"tag":251,"props":4312,"children":4313},{},[4314,4320],{"type":58,"tag":81,"props":4315,"children":4317},{"className":4316},[],[4318],{"type":64,"value":4319},"data.metadata.clusterName",{"type":64,"value":4321}," — HyperPod cluster name (required)",{"type":58,"tag":251,"props":4323,"children":4324},{},[4325,4331,4333,4339,4341,4347,4348],{"type":58,"tag":81,"props":4326,"children":4328},{"className":4327},[],[4329],{"type":64,"value":4330},"data.metadata.detailType",{"type":64,"value":4332}," — ",{"type":58,"tag":81,"props":4334,"children":4336},{"className":4335},[],[4337],{"type":64,"value":4338},"Cluster State Change",{"type":64,"value":4340}," \u002F\n",{"type":58,"tag":81,"props":4342,"children":4344},{"className":4343},[],[4345],{"type":64,"value":4346},"Cluster Node Health Event",{"type":64,"value":114},{"type":58,"tag":81,"props":4349,"children":4351},{"className":4350},[],[4352],{"type":64,"value":2673},{"type":58,"tag":251,"props":4354,"children":4355},{},[4356,4362],{"type":58,"tag":81,"props":4357,"children":4359},{"className":4358},[],[4360],{"type":64,"value":4361},"data.originalEvent.detail.InstanceId",{"type":64,"value":4363}," — affected instance (for node\nhealth events)",{"type":58,"tag":251,"props":4365,"children":4366},{},[4367,4373,4374,4380],{"type":58,"tag":81,"props":4368,"children":4370},{"className":4369},[],[4371],{"type":64,"value":4372},"data.originalEvent.detail.EventDetails.InstanceGroupName",{"type":64,"value":4340},{"type":58,"tag":81,"props":4375,"children":4377},{"className":4376},[],[4378],{"type":64,"value":4379},"InstanceId",{"type":64,"value":4381}," (for cluster events)",{"type":58,"tag":67,"props":4383,"children":4384},{},[4385],{"type":64,"value":4386},"If the cluster name is missing, abort with \"skill cannot run without a\nHyperPod cluster name — check the webhook bridge's payload mapping.\"",{"type":58,"tag":193,"props":4388,"children":4390},{"id":4389},"what-this-skill-does-not-do",[4391],{"type":64,"value":4392},"What this skill does NOT do",{"type":58,"tag":247,"props":4394,"children":4395},{},[4396,4429,4446],{"type":58,"tag":251,"props":4397,"children":4398},{},[4399,4404,4406,4412,4413,4419,4421,4427],{"type":58,"tag":73,"props":4400,"children":4401},{},[4402],{"type":64,"value":4403},"It does not SSH or run SSM on nodes.",{"type":64,"value":4405}," The DevOps Agent permission\nguardrail blocks ",{"type":58,"tag":81,"props":4407,"children":4409},{"className":4408},[],[4410],{"type":64,"value":4411},"ssm:StartSession",{"type":64,"value":114},{"type":58,"tag":81,"props":4414,"children":4416},{"className":4415},[],[4417],{"type":64,"value":4418},"ssm:SendCommand",{"type":64,"value":4420},". When on-node\nevidence is needed (Xid in ",{"type":58,"tag":81,"props":4422,"children":4424},{"className":4423},[],[4425],{"type":64,"value":4426},"dmesg",{"type":64,"value":4428},", DCGM counters, kubelet journal),\nthe skill emits a recommendation that includes the exact SSM command\nfor the operator to run, never executes it.",{"type":58,"tag":251,"props":4430,"children":4431},{},[4432,4437,4438,4444],{"type":58,"tag":73,"props":4433,"children":4434},{},[4435],{"type":64,"value":4436},"It does not replace nodes.",{"type":64,"value":98},{"type":58,"tag":81,"props":4439,"children":4441},{"className":4440},[],[4442],{"type":64,"value":4443},"batch-replace-cluster-nodes",{"type":64,"value":4445}," is a\nstate-changing call; the skill recommends it but never invokes it.",{"type":58,"tag":251,"props":4447,"children":4448},{},[4449,4454,4456,4462,4464,4470],{"type":58,"tag":73,"props":4450,"children":4451},{},[4452],{"type":64,"value":4453},"It does not modify Slurm or EKS state.",{"type":64,"value":4455}," No ",{"type":58,"tag":81,"props":4457,"children":4459},{"className":4458},[],[4460],{"type":64,"value":4461},"scontrol update",{"type":64,"value":4463},", no\n",{"type":58,"tag":81,"props":4465,"children":4467},{"className":4466},[],[4468],{"type":64,"value":4469},"kubectl cordon",{"type":64,"value":4471},", no label edits.",{"type":58,"tag":193,"props":4473,"children":4475},{"id":4474},"tools-used-in-guardrail",[4476],{"type":64,"value":4477},"Tools used (in-guardrail)",{"type":58,"tag":247,"props":4479,"children":4480},{},[4481,4511,4535,4562],{"type":58,"tag":251,"props":4482,"children":4483},{},[4484,4490,4491,4497,4498,4504,4505],{"type":58,"tag":81,"props":4485,"children":4487},{"className":4486},[],[4488],{"type":64,"value":4489},"sagemaker:DescribeCluster",{"type":64,"value":148},{"type":58,"tag":81,"props":4492,"children":4494},{"className":4493},[],[4495],{"type":64,"value":4496},"ListClusterNodes",{"type":64,"value":148},{"type":58,"tag":81,"props":4499,"children":4501},{"className":4500},[],[4502],{"type":64,"value":4503},"DescribeClusterNode",{"type":64,"value":733},{"type":58,"tag":81,"props":4506,"children":4508},{"className":4507},[],[4509],{"type":64,"value":4510},"ListClusterEvents",{"type":58,"tag":251,"props":4512,"children":4513},{},[4514,4520,4521,4527,4529],{"type":58,"tag":81,"props":4515,"children":4517},{"className":4516},[],[4518],{"type":64,"value":4519},"logs:FilterLogEvents",{"type":64,"value":148},{"type":58,"tag":81,"props":4522,"children":4524},{"className":4523},[],[4525],{"type":64,"value":4526},"logs:DescribeLogStreams",{"type":64,"value":4528}," on\n",{"type":58,"tag":81,"props":4530,"children":4532},{"className":4531},[],[4533],{"type":64,"value":4534},"\u002Faws\u002Fsagemaker\u002FClusters\u002F*",{"type":58,"tag":251,"props":4536,"children":4537},{},[4538,4544,4546,4552,4554,4560],{"type":58,"tag":81,"props":4539,"children":4541},{"className":4540},[],[4542],{"type":64,"value":4543},"eks:DescribeCluster",{"type":64,"value":4545}," (read-only) and ",{"type":58,"tag":81,"props":4547,"children":4549},{"className":4548},[],[4550],{"type":64,"value":4551},"kubectl get\u002Fdescribe",{"type":64,"value":4553},"\n(via the ",{"type":58,"tag":81,"props":4555,"children":4557},{"className":4556},[],[4558],{"type":64,"value":4559},"AWS::EKS::AccessEntry",{"type":64,"value":4561}," created by the CloudFormation stack)",{"type":58,"tag":251,"props":4563,"children":4564},{},[4565,4571,4573,4579,4580,4586],{"type":58,"tag":81,"props":4566,"children":4568},{"className":4567},[],[4569],{"type":64,"value":4570},"cloudwatch:GetMetricData",{"type":64,"value":4572}," on ",{"type":58,"tag":81,"props":4574,"children":4576},{"className":4575},[],[4577],{"type":64,"value":4578},"ClusterAgent",{"type":64,"value":2059},{"type":58,"tag":81,"props":4581,"children":4583},{"className":4582},[],[4584],{"type":64,"value":4585},"SagemakerHealthMonitoringAgent",{"type":64,"value":4587}," namespaces — useful for time-series\nconfirmation when individual log events are sparse",{"type":58,"tag":4589,"props":4590,"children":4591},"style",{},[4592],{"type":64,"value":4593},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":4595,"total":4768},[4596,4613,4634,4644,4657,4670,4680,4690,4706,4717,4737,4753],{"slug":4597,"name":4597,"fn":4598,"description":4599,"org":4600,"tags":4601,"stars":4610,"repoUrl":4611,"updatedAt":4612},"agentcore-investigation","investigate Bedrock AgentCore runtime sessions","Investigate Bedrock AgentCore runtime sessions via CloudWatch Logs Insights — resolve session\u002Ftrace IDs, query OTEL spans, filter noise, build timelines. Use when debugging AgentCore agent sessions, tracing tool calls, or analyzing latency.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4602,4603,4604,4607],{"name":21,"slug":22,"type":16},{"name":24,"slug":25,"type":16},{"name":4605,"slug":4606,"type":16},"Logs","logs",{"name":4608,"slug":4609,"type":16},"Observability","observability",9645,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fmcp","2026-07-12T08:37:22.601527",{"slug":4614,"name":4615,"fn":4616,"description":4617,"org":4618,"tags":4619,"stars":4610,"repoUrl":4611,"updatedAt":4633},"amazon-aurora-dsql","amazon aurora dsql","build applications with Aurora DSQL","Deprecated compatibility redirect for Aurora DSQL guidance. Use when a request concerns DSQL, Aurora DSQL, distributed SQL, DSQL schemas, migrations, queries, authentication, performance, or application development.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4620,4623,4624,4627,4630],{"name":4621,"slug":4622,"type":16},"Aurora","aurora",{"name":21,"slug":22,"type":16},{"name":4625,"slug":4626,"type":16},"Database","database",{"name":4628,"slug":4629,"type":16},"Serverless","serverless",{"name":4631,"slug":4632,"type":16},"SQL","sql","2026-09-02T07:20:51.53702",{"slug":4635,"name":4636,"fn":4616,"description":4617,"org":4637,"tags":4638,"stars":4610,"repoUrl":4611,"updatedAt":4643},"aurora-dsql","aurora dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4639,4640,4641,4642],{"name":21,"slug":22,"type":16},{"name":4625,"slug":4626,"type":16},{"name":4628,"slug":4629,"type":16},{"name":4631,"slug":4632,"type":16},"2026-09-02T07:20:46.533217",{"slug":4645,"name":4646,"fn":4616,"description":4617,"org":4647,"tags":4648,"stars":4610,"repoUrl":4611,"updatedAt":4656},"aws-dsql","aws dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4649,4650,4651,4654,4655],{"name":21,"slug":22,"type":16},{"name":4625,"slug":4626,"type":16},{"name":4652,"slug":4653,"type":16},"Migration","migration",{"name":4628,"slug":4629,"type":16},{"name":4631,"slug":4632,"type":16},"2026-09-02T07:20:49.531712",{"slug":4658,"name":4659,"fn":4616,"description":4617,"org":4660,"tags":4661,"stars":4610,"repoUrl":4611,"updatedAt":4669},"distributed-postgres","distributed postgres",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4662,4663,4664,4667,4668],{"name":21,"slug":22,"type":16},{"name":4625,"slug":4626,"type":16},{"name":4665,"slug":4666,"type":16},"PostgreSQL","postgresql",{"name":4628,"slug":4629,"type":16},{"name":4631,"slug":4632,"type":16},"2026-09-02T07:20:47.592534",{"slug":4671,"name":4672,"fn":4616,"description":4617,"org":4673,"tags":4674,"stars":4610,"repoUrl":4611,"updatedAt":4679},"distributed-sql","distributed sql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4675,4676,4677,4678],{"name":21,"slug":22,"type":16},{"name":4625,"slug":4626,"type":16},{"name":4628,"slug":4629,"type":16},{"name":4631,"slug":4632,"type":16},"2026-09-02T07:20:50.520015",{"slug":4681,"name":4681,"fn":4616,"description":4617,"org":4682,"tags":4683,"stars":4610,"repoUrl":4611,"updatedAt":4689},"dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4684,4685,4686,4687,4688],{"name":21,"slug":22,"type":16},{"name":4625,"slug":4626,"type":16},{"name":4652,"slug":4653,"type":16},{"name":4628,"slug":4629,"type":16},{"name":4631,"slug":4632,"type":16},"2026-09-02T07:20:48.570617",{"slug":4691,"name":4691,"fn":4692,"description":4693,"org":4694,"tags":4695,"stars":4703,"repoUrl":4704,"updatedAt":4705},"aidlc","orchestrate AI-driven development lifecycle workflows","AI-DLC workflow orchestrator. Start, resume, or manage an AI-driven development lifecycle. Scopes are defined one file per scope under `.kiro\u002Fscopes\u002F`; run `bun .kiro\u002Ftools\u002Faidlc-utility.ts help` for the authoritative list and descriptions. Utilities: --status, --doctor, --stage, --phase, --scope, --depth, --test-strategy, --review, --version, --help, plus the intent and space verbs. Or describe what you want to build and the scope will be auto-detected.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4696,4699,4702],{"name":4697,"slug":4698,"type":16},"Agents","agents",{"name":4700,"slug":4701,"type":16},"Automation","automation",{"name":284,"slug":281,"type":16},4261,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Faidlc-workflows","2026-09-02T07:47:34.75815",{"slug":4707,"name":4707,"fn":4708,"description":4709,"org":4710,"tags":4711,"stars":4703,"repoUrl":4704,"updatedAt":4716},"aidlc-jump","navigate AI-DLC workflow stages and phases","Jump the active AI-DLC workflow to a stage or phase. A Cursor-native shortcut for `\u002Faidlc --stage \u003Ctarget>` or `\u002Faidlc --phase \u003Ctarget>`.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4712,4713],{"name":4700,"slug":4701,"type":16},{"name":4714,"slug":4715,"type":16},"Navigation","navigation","2026-09-02T07:48:09.505949",{"slug":4718,"name":4718,"fn":4719,"description":4720,"org":4721,"tags":4722,"stars":4703,"repoUrl":4704,"updatedAt":4736},"aidlc-knowledge","index documents for AI-DLC agent citation","Index the team's own documents — PDFs, Word files, Markdown, plain text — into a per-space catalog the AI-DLC agents can cite. Wraps `aidlc-knowledge.ts`: onboard, sync, list, show, associate, dissociate, rebind, summarize. Every catalog row is written by the tool under a workspace lock; this skill never edits the catalog by hand and never advances workflow state.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4723,4724,4727,4730,4733],{"name":4697,"slug":4698,"type":16},{"name":4725,"slug":4726,"type":16},"Documents","documents",{"name":4728,"slug":4729,"type":16},"Knowledge Management","knowledge-management",{"name":4731,"slug":4732,"type":16},"Markdown","markdown",{"name":4734,"slug":4735,"type":16},"PDF","pdf","2026-09-02T07:48:10.614775",{"slug":4738,"name":4738,"fn":4739,"description":4740,"org":4741,"tags":4742,"stars":4703,"repoUrl":4704,"updatedAt":4752},"aidlc-outcomes-pack","generate AI-DLC workflow handover documentation","Generate a comprehensive handover document at workflow close so the team can own, operate, and continue the system without re-running the workflow. Stage\u002Fphase\u002Flearning counts come from `aidlc-runtime.ts summary`; prose comes from the artefacts. Writes OUTCOMES.md but never mutates workflow state or emits audit events.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4743,4746,4749],{"name":4744,"slug":4745,"type":16},"Documentation","documentation",{"name":4747,"slug":4748,"type":16},"Process Documentation","process-documentation",{"name":4750,"slug":4751,"type":16},"Reporting","reporting","2026-09-02T07:47:34.212738",{"slug":4754,"name":4754,"fn":4755,"description":4756,"org":4757,"tags":4758,"stars":4703,"repoUrl":4704,"updatedAt":4767},"aidlc-replay","generate AI-DLC session narrative reports","Print a structured session narrative for stakeholders who weren't in the room. Numbers (stage counts, phase rollup, duration) come from `aidlc-runtime.ts summary`; prose comes from the audit trail and artefacts. Renders to the terminal only — writes no file, never mutates workflow state, never emits audit events.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4759,4762,4763,4766],{"name":4760,"slug":4761,"type":16},"Audit","audit",{"name":4744,"slug":4745,"type":16},{"name":4764,"slug":4765,"type":16},"Engineering","engineering",{"name":4750,"slug":4751,"type":16},"2026-09-02T07:47:45.055994",142,{"items":4770,"total":934},[4771,4787,4794],{"slug":4772,"name":4772,"fn":4773,"description":4774,"org":4775,"tags":4776,"stars":26,"repoUrl":27,"updatedAt":4786},"hyperpod-devops-agent-solution","monitor SageMaker HyperPod infrastructure","How this Agent Space monitors SageMaker HyperPod — the design and intent of the HyperPod x DevOps Agent solution (event-driven webhook bridge, Lambda-gated periodic audit, triage\u002FRCA skills, email notifications). Read this to understand WHY a HyperPod investigation was created and what the monitoring pipeline does. For the concrete resource\u002Ftopology map (ARNs, IDs, log groups), see the understanding-agent-space skill instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4777,4778,4779,4782,4785],{"name":4697,"slug":4698,"type":16},{"name":21,"slug":22,"type":16},{"name":4780,"slug":4781,"type":16},"DevOps","devops",{"name":4783,"slug":4784,"type":16},"Monitoring","monitoring",{"name":4608,"slug":4609,"type":16},"2026-09-02T07:47:35.452244",{"slug":4,"name":4,"fn":5,"description":6,"org":4788,"tags":4789,"stars":26,"repoUrl":27,"updatedAt":28},{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4790,4791,4792,4793],{"name":21,"slug":22,"type":16},{"name":24,"slug":25,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"slug":104,"name":104,"fn":4795,"description":4796,"org":4797,"tags":4798,"stars":26,"repoUrl":27,"updatedAt":4805},"triage SageMaker HyperPod incidents","Correlation and skip rules for SageMaker HyperPod incident triage. Keeps distinct fault types on the same instance group as separate investigations (the default correlator merges them), and prevents periodic-audit re-investigation of an unchanged cluster. Applies at the Incident Triage stage before any investigation runs.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[4799,4800,4801,4802],{"name":21,"slug":22,"type":16},{"name":18,"slug":19,"type":16},{"name":14,"slug":15,"type":16},{"name":4803,"slug":4804,"type":16},"Triage","triage","2026-09-02T07:47:39.537858"]