HyperPod x DevOps Agent — how this solution monitors the cluster

This Agent Space is wired to monitor one or more SageMaker HyperPod clusters. This document explains the design and intent of that monitoring so investigations reason correctly and operators (via chat) can understand how their cluster is watched.

Scope: this is the how it works doc. For the concrete resource map (what exists, ARNs, instance IDs, log groups, VPC/subnet/SG IDs), use the understanding-agent-space skill.

Two independent detection paths

HyperPod problems reach DevOps Agent through two paths, by design:

1. Event-driven webhook bridge (all HyperPod control-plane faults). A Lambda subscribes to aws.sagemaker HyperPod EventBridge events (Cluster State Change, Node Health, Cluster Event), drops routine Info-level noise, and HMAC-POSTs the rest to the DevOps Agent generic webhook — which creates an investigation. This is how node health faults, capacity errors, lifecycle-script failures, and cluster state changes are detected, on both EKS and Slurm (Slurm requires Continuous Provisioning for the correct event shape).
2. Periodic-audit Lambda (Kubernetes state only). Some conditions are NOT in the HyperPod event stream — notably Kubernetes CrashLoopBackOff pods and NotReady nodes. A scheduled Lambda inspects those via the EKS API and only invokes an investigation when a real issue is present (thresholds are configurable). On a healthy cluster it posts nothing. On Slurm there is no Kubernetes to inspect, so this path does nothing except the heartbeat below.

Implication for investigations: if you are investigating a "HyperPod periodic audit" task, the audit Lambda already detected a concrete issue (or a control-plane fault arrived via the bridge). Treat the reported issue as real and confirm + explain it — don't assume it was a routine poll. The Lambda inlines what it found into the task description string (one - [tag] type on resource: detail line per issue) — the DevOps Agent platform preserves the top-level description verbatim but flattens nested payload sub-objects, so the description text, not a data.metadata.* field, is where the skill reads the findings.

Daily heartbeat (liveness)

Once per day the audit Lambda fires a heartbeat. Its purpose is pipeline liveness — letting operators confirm the solution itself is alive and reaching the console — not incident detection (the 15-minute audit and the event-driven bridge do that). Do not treat a heartbeat as an incident.

The heartbeat run still inspects cluster state:

• Healthy cluster (the usual case): an "all clear, no open issues" heartbeat — a silent liveness signal, visible in the console but never emailed. Triage skips it (nothing to investigate); the skipped task is itself the liveness signal.
• A real issue exists at heartbeat time: the Lambda builds an issues payload instead, which is triaged and root-caused like any detection and can email.

Triage → RCA → notification

• Triage (hyperpod-incident-triage, INCIDENT_TRIAGE) decides LINK / SKIP / PROCEED. It keeps different fault types on the same instance group as separate investigations (the default correlator would merge them) and skips concurrent periodic audits.
• RCA (hyperpod-incident-rca, INCIDENT_RCA) runs on PROCEED: reconstructs the timeline from list-cluster-events, cluster/node state, and HMA CloudWatch streams; classifies as Suppress / Monitor / Escalate / Resolved; and writes a verdict with a plain-English Summary (what happened → likely cause → recommended action).
• Email notifier sends one SES email per completed investigation (scoped to this Agent Space). It suppresses Suppress verdicts, no-finding investigations, and heartbeats, and leads the subject with the Summary's first sentence.

What this solution does NOT do

• It does not poll for HyperPod control-plane faults — those are event-driven via the bridge. The audit only covers Kubernetes state.
• The DevOps Agent runtime cannot open SSM sessions to nodes (a fixed permission guardrail), so on-node signals (dmesg Xid lines, DCGM counters, EFA fabric errors, slurmctld logs) are reached only indirectly via HMA CloudWatch streams / K8s labels / control-plane events — not by shelling into the node.