[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-aws-labs-hyperpod-node-debugger":3,"mdc-eni1x9-key":42,"related-repo-aws-labs-hyperpod-node-debugger":3604,"related-org-aws-labs-hyperpod-node-debugger":3709},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":37,"sourceUrl":40,"mdContent":41},"hyperpod-node-debugger","diagnose and remediate HyperPod cluster nodes","Diagnose and remediate per-node issues on a HyperPod cluster (EKS or Slurm) — a specific node is unhealthy, unresponsive, stuck, or needs replacing. Covers on-node EFA, GPU \u002F accelerator hardware (XID, ECC, NVLink, row-remap, DCGM), Slurm node down\u002Fdrained, disk and memory pressure, per-node lifecycle-script failures, SSM agent, container runtime, kernel panics, pod networking. Read-only. Not for cluster-wide provisioning (→ hyperpod-cluster-debugger), NCCL (→ hyperpod-nccl), or MFU (→ hyperpod-mfu-debugger).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"aws-labs","AWS Labs","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Faws-labs.png","awslabs",[13,17,20,23],{"name":14,"slug":15,"type":16},"Observability","observability","tag",{"name":18,"slug":19,"type":16},"Infrastructure","infrastructure",{"name":21,"slug":22,"type":16},"AWS","aws",{"name":24,"slug":25,"type":16},"Debugging","debugging",831,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagent-plugins","2026-07-12T08:39:26.704231",null,127,[32,33,34,22,35,36],"agent-plugins","agent-skills","agents","coding-agent-skills","coding-agents",{"repoUrl":27,"stars":26,"forks":30,"topics":38,"description":39},[32,33,34,22,35,36],"Agent Plugins for AWS equip AI coding agents with the skills to help you architect, deploy, and operate on AWS.","https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagent-plugins\u002Ftree\u002FHEAD\u002Fplugins\u002Fsagemaker-ai\u002Fskills\u002Fhyperpod-node-debugger","---\nname: hyperpod-node-debugger\ndescription: Diagnose and remediate per-node issues on a HyperPod cluster (EKS or Slurm) — a specific node is unhealthy, unresponsive, stuck, or needs replacing. Covers on-node EFA, GPU \u002F accelerator hardware (XID, ECC, NVLink, row-remap, DCGM), Slurm node down\u002Fdrained, disk and memory pressure, per-node lifecycle-script failures, SSM agent, container runtime, kernel panics, pod networking. Read-only. Not for cluster-wide provisioning (→ hyperpod-cluster-debugger), NCCL (→ hyperpod-nccl), or MFU (→ hyperpod-mfu-debugger).\nmetadata:\n  version: \"0.0.1\"\n---\n\n# HyperPod Node Debugger\n\n**Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a **Suggested command (run this yourself)** block and wait for the customer. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes). Never discard training state, logs, or caches on speculation.\n\n**IaC note (always include with mutation commands).** When you suggest any command that changes cluster, VPC, SG, subnet, or EKS configuration (e.g. `authorize-security-group-*`, `modify-vpc-attribute`, `update-cluster`, `kubectl label\u002Fcordon\u002Fdrain`, `create namespace`, `set env daemonset`), ask the customer first whether the cluster \u002F VPC \u002F SG is managed by Infrastructure-as-Code (CloudFormation, CDK, Terraform, Pulumi). If yes, tell them: \"Apply this change in your IaC source first, then deploy through the pipeline — running the command directly will drift from your template and the next stack update may overwrite it.\" If they need to fix the issue immediately and the IaC change will follow, flag the drift explicitly so they remember to reconcile.\n\nRead-only triage. `scripts\u002Ftriage-cluster.sh` (and helpers `check-efa-sg.sh`, `check-node-reachability.sh`, `check-vpc-config.sh`) read state and print each issue as `[FAIL] ... → references\u002Fnode-diagnostics-detail.md § \u003Csection>`. Catalog of customer-ticket patterns: [references\u002Fnode-issue-catalog.md](references\u002Fnode-issue-catalog.md).\n\n---\n\n## Workflow\n\n1. Collect cluster name, region, suspect instance ID, exact error string from logs.\n2. Run `scripts\u002Ftriage-cluster.sh` (add `--node \u003CINSTANCE-ID>` to focus one node).\n3. For every `[FAIL]` \u002F issue entry, `Read` the referenced section.\n4. Present: what script detected (copy the line verbatim), root cause, exact command(s) with instance\u002FSG IDs filled in, blast radius (e.g. \"reboots i-xxx\", \"wipes volumes on replacement\"). For any command that mutates cluster\u002FVPC\u002FSG\u002FEKS state, ask whether the affected resource is IaC-managed and surface the drift warning from the operating-policy note above.\n5. Wait for explicit customer approval. Destructive order: investigate → reboot → replace.\n6. Re-run triage to confirm. Iterate if not cleared.\n\n## Step 1: Triage\n\n```bash\nbash scripts\u002Ftriage-cluster.sh --cluster \u003CCLUSTER_NAME_OR_ARN> --region \u003CREGION>\n\n# Focus on one node:\nbash scripts\u002Ftriage-cluster.sh --cluster \u003CCLUSTER_NAME_OR_ARN> --region \u003CREGION> --node \u003CINSTANCE_ID>\n```\n\nOne pass collects: cluster status + NodeRecovery, events, per-node health (HyperPod + EKS labels, Slurm states), VPC\u002FSG snapshot, CloudWatch availability, SSM readiness, on-node resource checks (disk, memory, \u002Fdev\u002Fshm, OOM, NVMe, time sync, SSM agent), Slurm node→instance mapping.\n\nTags: `[PASS]` passed · `[FAIL]` issue with a `→ references\u002F...` pointer · `[WARN]` advisory · `[INFO]` informational. Priorities: **P0** blocks operation · **P1** degraded · **P2** informational.\n\n## Step 2: Match signal → section\n\n**Events (`list-cluster-events`) — provisioning-time:**\n\n| Event                                                                       | Section                                                         |\n| --------------------------------------------------------------------------- | --------------------------------------------------------------- |\n| `\"EFA health checks did not run successfully\"` (public-doc verbatim signal) | **[A: EFA\u002FSG](#a-efa--security-group)**                         |\n| Instance bootstrap or network-misconfiguration event                        | **[A](#a-efa--security-group)** + **[B: VPC](#b-vpc--routing)** |\n| Lifecycle-script failure or timeout                                         | **[D: Lifecycle](#d-lifecycle-scripts)**                        |\n| Insufficient-capacity or AZ-mismatch failure at creation                    | **[C: Capacity](#c-capacity--az)**                              |\n| Hardware failure \u002F `UnschedulablePendingReplacement`                        | **[F: Hardware](#f-hardware--auto-repair)**                     |\n\n**EKS labels:**\n\n| Label                                                 | Section                                                          |\n| ----------------------------------------------------- | ---------------------------------------------------------------- |\n| `node-health-status: UnschedulablePendingReplacement` | **[F](#f-hardware--auto-repair)**                                |\n| `node-health-status: UnschedulablePendingReboot`      | **[F](#f-hardware--auto-repair)**                                |\n| `deep-health-check-status: Failed`                    | **[G](#g-gpu--accelerator)** → **[F](#f-hardware--auto-repair)** |\n\n**Symptoms:**\n\n| Symptom                                                  | Section                                                         |\n| -------------------------------------------------------- | --------------------------------------------------------------- |\n| Training hangs at NCCL init \u002F AllReduce                  | **[A](#a-efa--security-group)** → **[E](#e-software-versions)** |\n| Slurm node `down` \u002F `\"Node unexpectedly rebooted\"`       | **[H: Slurm](#h-slurm-node-management)**                        |\n| Jobs stuck PENDING \u002F COMPLETING                          | **[H](#h-slurm-node-management)**                               |\n| Auto-repair not triggering                               | **[F](#f-hardware--auto-repair)**                               |\n| GPU not visible \u002F XID \u002F ECC errors                       | **[G](#g-gpu--accelerator)**                                    |\n| GPU row-remap pending\u002Ffailed \u002F silent NaNs \u002F DCGM Fail   | **[G § G.1.a\u002Fb](#g-gpu--accelerator)**                          |\n| Disk full \u002F OOM \u002F `\"Cannot allocate memory\"`             | **[I: Resources](#i-resource-exhaustion)**                      |\n| Wrong vCPU count (e.g. 96 instead of 192 on p5.48xlarge) | **[J: Config](#j-configuration)**                               |\n| Container CrashLoopBackOff \u002F runtime crash               | **[M: Container Runtime](#m-container-runtime)**                |\n| `aws-node` CrashLoopBackOff \u002F gRPC 50051 refused         | **[O: CNI \u002F Pod Networking](#o-cni--pod-networking)**           |\n| Pods stuck Pending with no IP \u002F CNI error                | **[O](#o-cni--pod-networking)**                                 |\n| DNS resolution \u002F `enableDnsSupport`                      | **[B § B.2](#b-vpc--routing)**                                  |\n| Public subnet \u002F IGW misconfigured                        | **[B § B.3](#b-vpc--routing)**                                  |\n| Missing VPC endpoints (ECR \u002F STS \u002F FSx)                  | **[B § B.4](#b-vpc--routing)**                                  |\n| EKS VPC \u002F SG mismatch with HyperPod                      | **[B § B.5](#b-vpc--routing)**                                  |\n| Kernel panic \u002F watchdog \u002F hung task                      | **[N: Kernel](#n-kernel--system)**                              |\n| Need shell on a node                                     | **[K: SSM](#k-node-access-via-ssm)**                            |\n| Collect logs for AWS Support                             | **[L: Log Collection](#l-log-collection)**                      |\n\n---\n\n## A: EFA \u002F Security Group\n\nPer the HyperPod prerequisites doc, the SG must allow all inbound and outbound to itself. `scripts\u002Fcheck-efa-sg.sh` validates self-ref rules on every cluster SG. On-node EFA check via `scripts\u002Fcheck-node-reachability.sh` over SSM. Full: [§ A](references\u002Fnode-diagnostics-detail.md#a-efa--security-group).\n\n## B: VPC \u002F Routing\n\nSG\u002Fsubnet VPC mismatch, missing S3 Gateway endpoint, EKS auth mode, worker→controller routing, VPC DNS support, private-subnet + NAT \u002F VPC endpoints, EKS↔HyperPod VPC alignment. `scripts\u002Fcheck-vpc-config.sh`. Full: [§ B](references\u002Fnode-diagnostics-detail.md#b-vpc--routing).\n\n## C: Capacity \u002F AZ\n\nInsufficient-capacity failure at creation, or no subnets in the AZ where capacity is available. Check AZ offerings via `describe-instance-type-offerings`, then change subnet AZ or use Flexible Training Plans \u002F ODCR. Full: [§ C](references\u002Fnode-diagnostics-detail.md#c-capacity--az).\n\n## D: Lifecycle Scripts\n\nSurfaced in cluster events + CloudWatch under `LifecycleConfig\u002F\u003Cgroup>\u002F\u003Cinstance-id>`. Common: S3 connectivity, IAM gaps, CRLF line endings, infinite loops, parameter-name mismatch. Full: [§ D](references\u002Fnode-diagnostics-detail.md#d-lifecycle-scripts).\n\n## E: Software Versions\n\nDelegate to `hyperpod-version-checker` to compare NVIDIA driver, CUDA, NCCL, EFA installer, OFI NCCL, PyTorch across nodes. Ensure job env has `FI_PROVIDER=efa`, `FI_EFA_USE_DEVICE_RDMA=1`, `NCCL_SOCKET_IFNAME=^lo,docker`. Full: [§ E](references\u002Fnode-diagnostics-detail.md#e-software-versions).\n\n## F: Hardware \u002F Auto-Repair\n\nConfirm `NodeRecovery=Automatic`, inspect the EKS health labels + `sagemaker.amazonaws.com\u002Ffault-details` annotation, and read the `SagemakerHealthMonitoringAgent\u002F\u003Cgroup>\u002F\u003Cinstance>` CloudWatch stream. HMA runs passive background checks on GPU and Neuron state and **reboots** the node on count mismatch (per the HMA doc: \"if there's a mismatch between the expected number of GPUs … and the count returned by `nvidia-smi`, then HMA reboots the node\"; same for `neuron-ls`). Manual recovery order: reboot first, replace only if reboot fails; the preferred path is the batch APIs (`BatchReboot`\u002F`BatchReplaceClusterNodes`). Full: [§ F](references\u002Fnode-diagnostics-detail.md#f-hardware--auto-repair) · patterns: [node-issue-catalog.md](references\u002Fnode-issue-catalog.md).\n\n## G: GPU \u002F Accelerator\n\n**NVIDIA (p4d\u002Fp5\u002Fg5\u002Fg6):** `nvidia-smi` + `dmesg` over SSM for Xid, ECC, thermal throttling. Xid classification per NVIDIA's catalog: 13 Graphics Engine Exception (application-level), 31 GPU memory page fault (application, can be driver\u002FHW), 63 GPU memory remapping event (HW\u002FECC), 71 CE4 Error (HW copy engine), 74 NVLink Error (HW), 79 GPU has fallen off the bus (PCIe bus), 109 Context Switch Timeout Error (HW). Any uncorrectable ECC → drain and replace. Row-remap state is the authoritative silent-degradation signal (§ G.1.a).\n\n**Trainium \u002F Inferentia (trn1\u002Ftrn2\u002Finf2):** Neuron SDK — `neuron-ls`, `neuron-top`, `neuron-monitor`. `nvidia-smi` does not apply.\n\nGPU \u002F accelerator failures flow into § F for reboot \u002F replace. Full: [§ G](references\u002Fnode-diagnostics-detail.md#g-gpuaccelerator).\n\n## H: Slurm Node Management\n\nNode down\u002Funresponsive, unexpected reboots, stuck PENDING\u002FCOMPLETING jobs, Slurm-to-instance-ID translation. Primary access is SSM; diagnose `slurmd` first, fix the root cause, then start\u002Fresume the node per § H. Full: [§ H](references\u002Fnode-diagnostics-detail.md#h-slurm-node-management).\n\n## I: Resource Exhaustion\n\nDisk full (HyperPod root volume defaults to 100 GB and is not intended to grow post-creation), OOM, `os.fork()` memory error, `\u002Fdev\u002Fshm` exhaustion, inode exhaustion. Fork-memory fix: `export FI_EFA_USE_HUGE_PAGE=0`. Redirect bulk data to `\u002Fopt\u002Fsagemaker` (secondary EBS) or `\u002Fopt\u002Fdlami\u002Fnvme` (instance store). Full: [§ I](references\u002Fnode-diagnostics-detail.md#i-resource-exhaustion).\n\n## J: Configuration\n\np5.48xlarge reports 96 vCPU instead of 192 → set `ThreadsPerCore=2` via `update-cluster`. Full: [§ J](references\u002Fnode-diagnostics-detail.md#j-configuration).\n\n## K: Node Access via SSM\n\nNo direct SSH on HyperPod. Target format `sagemaker-cluster:\u003CCLUSTER_ID>_\u003CGROUP>-\u003CINSTANCE_ID>`. Failures: plugin missing, wrong prefix, IAM, VPC endpoints. Full: [§ K](references\u002Fnode-diagnostics-detail.md#k-node-access-via-ssm).\n\n## L: Log Collection\n\nDelegate to `hyperpod-issue-report` for S3-stored bundles. Key CloudWatch streams: `LifecycleConfig\u002F\u003Cgroup>\u002F\u003Cinstance-id>`, `SagemakerHealthMonitoringAgent\u002F\u003Cgroup>\u002F\u003Cinstance-id>`. Full: [§ L](references\u002Fnode-diagnostics-detail.md#l-log-collection).\n\n## M: Container Runtime\n\nCrashLoopBackOff, OOMKilled, ImagePullBackOff, RunContainerError on EKS. `kubectl describe pod` + on-node `crictl ps -a`, `journalctl -u containerd`. Full: [§ M](references\u002Fnode-diagnostics-detail.md#m-container-runtime).\n\n## N: Kernel & System\n\nKernel panic, watchdog timeout, soft lockup, unexpected reboots not explained by HyperPod health monitoring. `dmesg | grep -iE 'panic|watchdog|hung_task|NMI'` + `journalctl -b -1`. nvrm-related signatures point at NVIDIA driver crashes. Full: [§ N](references\u002Fnode-diagnostics-detail.md#n-kernel--system).\n\n## O: CNI \u002F Pod Networking\n\nVPC CNI (`aws-node`) failures, IPAMD errors, gRPC 127.0.0.1:50051 refused, pods stuck `Pending` with `FailedCreatePodSandBox`. Script auto-checks `aws-node`, `kube-proxy`, CoreDNS. Full: [§ O](references\u002Fnode-diagnostics-detail.md#o-cni--pod-networking).\n\n---\n\n## Prerequisites\n\n- `aws` CLI v2, recent enough to support the HyperPod cluster commands (`describe-cluster`, `list-cluster-nodes`, `batch-reboot-cluster-nodes`, `batch-replace-cluster-nodes`)\n- `python3`, `bash` 4+ (associative arrays are required by the scripts)\n- `kubectl` authenticated to the EKS cluster (K8s checks skipped if absent)\n- `session-manager-plugin` for on-node hardware checks\n- `unbuffer` (from the `expect` package) — optional; if missing, SSM on-node probes are skipped while the rest of the triage still runs. Install via `yum install expect` \u002F `apt install expect`.\n\n## Defaults\n\n- **Region** — required: pass `--region` or set `$AWS_DEFAULT_REGION`.\n- **Target scope** — all nodes; `--node \u003CID>` focuses one.\n- **Event window** — up to 500 most recent events (5 × 100, paginated).\n- **Node list cap** — up to 20,000 nodes (200 × 100); warns on cap.\n- **SSM probes** — 180 s per node with retry-on-throttle.\n- **Colors** — auto-disabled on non-TTY; `--no-color` to force off.\n\n## Error handling\n\n| Failure                                         | Script                                                 | Tell the customer                                                    |\n| ----------------------------------------------- | ------------------------------------------------------ | -------------------------------------------------------------------- |\n| `aws sts get-caller-identity` fails             | Exit 1                                                 | \"Fix AWS credentials and rerun.\"                                     |\n| `describe-cluster` fails                        | Exit 1 after listing region's clusters                 | \"Confirm cluster name and region.\"                                   |\n| `sagemaker:*` \u002F `ec2:*` \u002F `logs:*` AccessDenied | Warn, add `Missing IAM permission for \u003CAPI>`, continue | \"Grant the listed IAM action and rerun.\"                             |\n| `kubectl` absent or unauthenticated             | Skip K8s checks                                        | \"Install\u002Fauthenticate kubectl (see § K).\"                            |\n| `session-manager-plugin` absent                 | Skip on-node probes                                    | \"Install session-manager-plugin (see § K).\"                          |\n| SSM `start-session` fails or times out (180s)   | Mark node unreachable with `→ § K` pointer             | \"Rerun with `--node \u003CID>` to isolate; verify SSM agent on the node.\" |\n| Cluster > 20,000 nodes                          | First 20,000 paginated; warn                           | \"Use `--node` to target specific nodes.\"                             |\n\nExit codes: `0` triage complete · `1` cluster not found or fatal prerequisite missing.\n\n## IAM permissions\n\nRead-only diagnostic — covers `triage-cluster.sh`, `check-efa-sg.sh`, `check-vpc-config.sh`, and `check-node-reachability.sh`:\n\n```json\n{\n  \"Action\": [\n    \"sagemaker:DescribeCluster\",\n    \"sagemaker:DescribeClusterNode\",\n    \"sagemaker:ListClusterNodes\",\n    \"sagemaker:ListClusterEvents\",\n    \"sagemaker:ListClusters\",\n    \"eks:DescribeCluster\",\n    \"ec2:DescribeSecurityGroups\",\n    \"ec2:DescribeSubnets\",\n    \"ec2:DescribeVpcs\",\n    \"ec2:DescribeVpcAttribute\",\n    \"ec2:DescribeVpcEndpoints\",\n    \"ec2:DescribeRouteTables\",\n    \"ec2:DescribeNetworkInterfaces\",\n    \"ec2:DescribeInstances\",\n    \"ec2:DescribeInstanceTypeOfferings\",\n    \"ec2:DescribeInstanceTypes\",\n    \"logs:DescribeLogGroups\",\n    \"logs:DescribeLogStreams\",\n    \"logs:FilterLogEvents\",\n    \"ssm:StartSession\",\n    \"ssm:TerminateSession\",\n    \"service-quotas:GetServiceQuota\"\n  ]\n}\n```\n\n`sts:GetCallerIdentity` is implicit — it requires no IAM action. SSM on HyperPod uses `start-session` against `sagemaker-cluster:\u003Ccluster-id>_\u003Cgroup>-\u003Ciid>` targets — not `send-command` against bare instance IDs. For remediation commands, grant the matching write permission (e.g. `ec2:AuthorizeSecurityGroupIngress` \u002F `Egress`, `ec2:RevokeSecurityGroupIngress` \u002F `Egress`, `ec2:ModifyVpcAttribute`, `sagemaker:UpdateCluster`, `sagemaker:BatchRebootClusterNodes`, `sagemaker:BatchReplaceClusterNodes`). Not needed for the diagnostic itself.\n\n## Skill delegation\n\n| Need                                                   | Use                                                          |\n| ------------------------------------------------------ | ------------------------------------------------------------ |\n| Cluster creation \u002F deployment failures                 | `hyperpod-cluster-debugger` (§ A \u002F B \u002F C \u002F H + `--validate`) |\n| Cluster-wide SSM outage                                | `hyperpod-cluster-debugger` § F                              |\n| Single-node SSM failure                                | stay here — § K                                              |\n| Cluster-wide EFA health-check failure at creation time | `hyperpod-cluster-debugger` § A                              |\n| Single-node EFA failure post-provisioning              | stay here — § A                                              |\n| NCCL AllReduce \u002F collective-op timeouts (distributed)  | `hyperpod-nccl`                                              |\n| Silent GPU NaNs on a specific node (row-remap \u002F DCGM)  | stay here — § G.1 (even if discovered by NCCL)               |\n| Post-deployment cluster-wide management                | `hyperpod-cluster-debugger`                                  |\n| Shell \u002F commands on nodes                              | `hyperpod-ssm`                                               |\n| CUDA \u002F NCCL \u002F EFA version comparison                   | `hyperpod-version-checker`                                   |\n| Diagnostic bundle for AWS Support                      | `hyperpod-issue-report`                                      |\n| Training performance \u002F MFU degradation                 | `hyperpod-mfu-debugger`                                      |\n\n## Escalate to AWS Support\n\nEscalate when:\n\n1. SG rules correct and reachability passes but EFA still fails.\n2. VPC correct but K8s bootstrap fails — check VPC flow logs for REJECT.\n3. Hardware failure where replacement keeps failing (bad physical host).\n4. Node replacement fails with an insufficient-capacity signal despite a valid ODCR.\n\n### Before opening the case\n\n```bash\n# 1. Cluster identity + affected node status\naws sagemaker describe-cluster --cluster-name \u003CCLUSTER> --region \u003CREGION>\naws sagemaker list-cluster-nodes --cluster-name \u003CCLUSTER> --region \u003CREGION> \\\n  --query \"ClusterNodeSummaries[?InstanceId=='\u003CINSTANCE_ID>']\"\n\n# 2. Triage bundle (scoped to the affected node where possible)\nbash scripts\u002Ftriage-cluster.sh --cluster \u003CCLUSTER> --region \u003CREGION> --node \u003CINSTANCE_ID> > triage.txt\n\n# 3. Per-node log\u002Fconfig bundle to S3 (delegates to hyperpod-issue-report)\n#    See skills\u002Fhyperpod-issue-report\u002FSKILL.md for the exact invocation.\n```\n\n### Include in the case\n\n- Cluster name + ARN and AWS region\n- Orchestrator (EKS or Slurm)\n- Affected instance IDs \u002F node names \u002F instance-group names\n- Timestamp window (UTC start \u002F end) of the failure\n- Exact error strings observed (copy verbatim from pod logs, CloudWatch, dmesg, events)\n- XID numbers \u002F ECC counts \u002F DCGM output where hardware is implicated\n- `triage.txt` from step 2 above\n- S3 URI of the `hyperpod-issue-report` bundle from step 3\n\nPatterns from real customer tickets: [node-issue-catalog.md](references\u002Fnode-issue-catalog.md).\n",{"data":43,"body":46},{"name":4,"description":6,"metadata":44},{"version":45},"0.0.1",{"type":47,"children":48},"root",[49,57,83,138,189,193,200,266,272,445,450,515,521,537,687,695,794,802,1212,1215,1221,1249,1255,1275,1281,1301,1307,1327,1333,1374,1380,1462,1468,1492,1530,1542,1548,1568,1574,1626,1632,1658,1664,1684,1689,1721,1726,1760,1766,1793,1798,1847,1850,1856,1971,1977,2071,2077,2309,2330,2336,2368,2890,2980,2986,3208,3214,3219,3242,3249,3526,3532,3588,3598],{"type":50,"tag":51,"props":52,"children":53},"element","h1",{"id":4},[54],{"type":55,"value":56},"text","HyperPod Node Debugger",{"type":50,"tag":58,"props":59,"children":60},"p",{},[61,67,69,74,76,81],{"type":50,"tag":62,"props":63,"children":64},"strong",{},[65],{"type":55,"value":66},"Operating policy.",{"type":55,"value":68}," Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a ",{"type":50,"tag":62,"props":70,"children":71},{},[72],{"type":55,"value":73},"Suggested command (run this yourself)",{"type":55,"value":75}," block and wait for the customer. Destructive order: ",{"type":50,"tag":62,"props":77,"children":78},{},[79],{"type":55,"value":80},"investigate → reboot → replace",{"type":55,"value":82}," (replace destroys root + secondary volumes; not supported on Slurm controller nodes). Never discard training state, logs, or caches on speculation.",{"type":50,"tag":58,"props":84,"children":85},{},[86,91,93,100,102,108,109,115,116,122,123,129,130,136],{"type":50,"tag":62,"props":87,"children":88},{},[89],{"type":55,"value":90},"IaC note (always include with mutation commands).",{"type":55,"value":92}," When you suggest any command that changes cluster, VPC, SG, subnet, or EKS configuration (e.g. ",{"type":50,"tag":94,"props":95,"children":97},"code",{"className":96},[],[98],{"type":55,"value":99},"authorize-security-group-*",{"type":55,"value":101},", ",{"type":50,"tag":94,"props":103,"children":105},{"className":104},[],[106],{"type":55,"value":107},"modify-vpc-attribute",{"type":55,"value":101},{"type":50,"tag":94,"props":110,"children":112},{"className":111},[],[113],{"type":55,"value":114},"update-cluster",{"type":55,"value":101},{"type":50,"tag":94,"props":117,"children":119},{"className":118},[],[120],{"type":55,"value":121},"kubectl label\u002Fcordon\u002Fdrain",{"type":55,"value":101},{"type":50,"tag":94,"props":124,"children":126},{"className":125},[],[127],{"type":55,"value":128},"create namespace",{"type":55,"value":101},{"type":50,"tag":94,"props":131,"children":133},{"className":132},[],[134],{"type":55,"value":135},"set env daemonset",{"type":55,"value":137},"), ask the customer first whether the cluster \u002F VPC \u002F SG is managed by Infrastructure-as-Code (CloudFormation, CDK, Terraform, Pulumi). If yes, tell them: \"Apply this change in your IaC source first, then deploy through the pipeline — running the command directly will drift from your template and the next stack update may overwrite it.\" If they need to fix the issue immediately and the IaC change will follow, flag the drift explicitly so they remember to reconcile.",{"type":50,"tag":58,"props":139,"children":140},{},[141,143,149,151,157,158,164,165,171,173,179,181,187],{"type":55,"value":142},"Read-only triage. ",{"type":50,"tag":94,"props":144,"children":146},{"className":145},[],[147],{"type":55,"value":148},"scripts\u002Ftriage-cluster.sh",{"type":55,"value":150}," (and helpers ",{"type":50,"tag":94,"props":152,"children":154},{"className":153},[],[155],{"type":55,"value":156},"check-efa-sg.sh",{"type":55,"value":101},{"type":50,"tag":94,"props":159,"children":161},{"className":160},[],[162],{"type":55,"value":163},"check-node-reachability.sh",{"type":55,"value":101},{"type":50,"tag":94,"props":166,"children":168},{"className":167},[],[169],{"type":55,"value":170},"check-vpc-config.sh",{"type":55,"value":172},") read state and print each issue as ",{"type":50,"tag":94,"props":174,"children":176},{"className":175},[],[177],{"type":55,"value":178},"[FAIL] ... → references\u002Fnode-diagnostics-detail.md § \u003Csection>",{"type":55,"value":180},". Catalog of customer-ticket patterns: ",{"type":50,"tag":182,"props":183,"children":185},"a",{"href":184},"references\u002Fnode-issue-catalog.md",[186],{"type":55,"value":184},{"type":55,"value":188},".",{"type":50,"tag":190,"props":191,"children":192},"hr",{},[],{"type":50,"tag":194,"props":195,"children":197},"h2",{"id":196},"workflow",[198],{"type":55,"value":199},"Workflow",{"type":50,"tag":201,"props":202,"children":203},"ol",{},[204,210,230,251,256,261],{"type":50,"tag":205,"props":206,"children":207},"li",{},[208],{"type":55,"value":209},"Collect cluster name, region, suspect instance ID, exact error string from logs.",{"type":50,"tag":205,"props":211,"children":212},{},[213,215,220,222,228],{"type":55,"value":214},"Run ",{"type":50,"tag":94,"props":216,"children":218},{"className":217},[],[219],{"type":55,"value":148},{"type":55,"value":221}," (add ",{"type":50,"tag":94,"props":223,"children":225},{"className":224},[],[226],{"type":55,"value":227},"--node \u003CINSTANCE-ID>",{"type":55,"value":229}," to focus one node).",{"type":50,"tag":205,"props":231,"children":232},{},[233,235,241,243,249],{"type":55,"value":234},"For every ",{"type":50,"tag":94,"props":236,"children":238},{"className":237},[],[239],{"type":55,"value":240},"[FAIL]",{"type":55,"value":242}," \u002F issue entry, ",{"type":50,"tag":94,"props":244,"children":246},{"className":245},[],[247],{"type":55,"value":248},"Read",{"type":55,"value":250}," the referenced section.",{"type":50,"tag":205,"props":252,"children":253},{},[254],{"type":55,"value":255},"Present: what script detected (copy the line verbatim), root cause, exact command(s) with instance\u002FSG IDs filled in, blast radius (e.g. \"reboots i-xxx\", \"wipes volumes on replacement\"). For any command that mutates cluster\u002FVPC\u002FSG\u002FEKS state, ask whether the affected resource is IaC-managed and surface the drift warning from the operating-policy note above.",{"type":50,"tag":205,"props":257,"children":258},{},[259],{"type":55,"value":260},"Wait for explicit customer approval. Destructive order: investigate → reboot → replace.",{"type":50,"tag":205,"props":262,"children":263},{},[264],{"type":55,"value":265},"Re-run triage to confirm. Iterate if not cleared.",{"type":50,"tag":194,"props":267,"children":269},{"id":268},"step-1-triage",[270],{"type":55,"value":271},"Step 1: Triage",{"type":50,"tag":273,"props":274,"children":279},"pre",{"className":275,"code":276,"language":277,"meta":278,"style":278},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","bash scripts\u002Ftriage-cluster.sh --cluster \u003CCLUSTER_NAME_OR_ARN> --region \u003CREGION>\n\n# Focus on one node:\nbash scripts\u002Ftriage-cluster.sh --cluster \u003CCLUSTER_NAME_OR_ARN> --region \u003CREGION> --node \u003CINSTANCE_ID>\n","bash","",[280],{"type":50,"tag":94,"props":281,"children":282},{"__ignoreMap":278},[283,350,360,370],{"type":50,"tag":284,"props":285,"children":288},"span",{"class":286,"line":287},"line",1,[289,294,300,305,311,316,322,327,332,336,341,345],{"type":50,"tag":284,"props":290,"children":292},{"style":291},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[293],{"type":55,"value":277},{"type":50,"tag":284,"props":295,"children":297},{"style":296},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[298],{"type":55,"value":299}," scripts\u002Ftriage-cluster.sh",{"type":50,"tag":284,"props":301,"children":302},{"style":296},[303],{"type":55,"value":304}," --cluster",{"type":50,"tag":284,"props":306,"children":308},{"style":307},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[309],{"type":55,"value":310}," \u003C",{"type":50,"tag":284,"props":312,"children":313},{"style":296},[314],{"type":55,"value":315},"CLUSTER_NAME_OR_AR",{"type":50,"tag":284,"props":317,"children":319},{"style":318},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[320],{"type":55,"value":321},"N",{"type":50,"tag":284,"props":323,"children":324},{"style":307},[325],{"type":55,"value":326},">",{"type":50,"tag":284,"props":328,"children":329},{"style":296},[330],{"type":55,"value":331}," --region",{"type":50,"tag":284,"props":333,"children":334},{"style":307},[335],{"type":55,"value":310},{"type":50,"tag":284,"props":337,"children":338},{"style":296},[339],{"type":55,"value":340},"REGIO",{"type":50,"tag":284,"props":342,"children":343},{"style":318},[344],{"type":55,"value":321},{"type":50,"tag":284,"props":346,"children":347},{"style":307},[348],{"type":55,"value":349},">\n",{"type":50,"tag":284,"props":351,"children":353},{"class":286,"line":352},2,[354],{"type":50,"tag":284,"props":355,"children":357},{"emptyLinePlaceholder":356},true,[358],{"type":55,"value":359},"\n",{"type":50,"tag":284,"props":361,"children":363},{"class":286,"line":362},3,[364],{"type":50,"tag":284,"props":365,"children":367},{"style":366},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[368],{"type":55,"value":369},"# Focus on one node:\n",{"type":50,"tag":284,"props":371,"children":373},{"class":286,"line":372},4,[374,378,382,386,390,394,398,402,406,410,414,418,422,427,431,436,441],{"type":50,"tag":284,"props":375,"children":376},{"style":291},[377],{"type":55,"value":277},{"type":50,"tag":284,"props":379,"children":380},{"style":296},[381],{"type":55,"value":299},{"type":50,"tag":284,"props":383,"children":384},{"style":296},[385],{"type":55,"value":304},{"type":50,"tag":284,"props":387,"children":388},{"style":307},[389],{"type":55,"value":310},{"type":50,"tag":284,"props":391,"children":392},{"style":296},[393],{"type":55,"value":315},{"type":50,"tag":284,"props":395,"children":396},{"style":318},[397],{"type":55,"value":321},{"type":50,"tag":284,"props":399,"children":400},{"style":307},[401],{"type":55,"value":326},{"type":50,"tag":284,"props":403,"children":404},{"style":296},[405],{"type":55,"value":331},{"type":50,"tag":284,"props":407,"children":408},{"style":307},[409],{"type":55,"value":310},{"type":50,"tag":284,"props":411,"children":412},{"style":296},[413],{"type":55,"value":340},{"type":50,"tag":284,"props":415,"children":416},{"style":318},[417],{"type":55,"value":321},{"type":50,"tag":284,"props":419,"children":420},{"style":307},[421],{"type":55,"value":326},{"type":50,"tag":284,"props":423,"children":424},{"style":296},[425],{"type":55,"value":426}," --node",{"type":50,"tag":284,"props":428,"children":429},{"style":307},[430],{"type":55,"value":310},{"type":50,"tag":284,"props":432,"children":433},{"style":296},[434],{"type":55,"value":435},"INSTANCE_I",{"type":50,"tag":284,"props":437,"children":438},{"style":318},[439],{"type":55,"value":440},"D",{"type":50,"tag":284,"props":442,"children":443},{"style":307},[444],{"type":55,"value":349},{"type":50,"tag":58,"props":446,"children":447},{},[448],{"type":55,"value":449},"One pass collects: cluster status + NodeRecovery, events, per-node health (HyperPod + EKS labels, Slurm states), VPC\u002FSG snapshot, CloudWatch availability, SSM readiness, on-node resource checks (disk, memory, \u002Fdev\u002Fshm, OOM, NVMe, time sync, SSM agent), Slurm node→instance mapping.",{"type":50,"tag":58,"props":451,"children":452},{},[453,455,461,463,468,470,476,478,484,486,492,494,499,501,506,508,513],{"type":55,"value":454},"Tags: ",{"type":50,"tag":94,"props":456,"children":458},{"className":457},[],[459],{"type":55,"value":460},"[PASS]",{"type":55,"value":462}," passed · ",{"type":50,"tag":94,"props":464,"children":466},{"className":465},[],[467],{"type":55,"value":240},{"type":55,"value":469}," issue with a ",{"type":50,"tag":94,"props":471,"children":473},{"className":472},[],[474],{"type":55,"value":475},"→ references\u002F...",{"type":55,"value":477}," pointer · ",{"type":50,"tag":94,"props":479,"children":481},{"className":480},[],[482],{"type":55,"value":483},"[WARN]",{"type":55,"value":485}," advisory · ",{"type":50,"tag":94,"props":487,"children":489},{"className":488},[],[490],{"type":55,"value":491},"[INFO]",{"type":55,"value":493}," informational. Priorities: ",{"type":50,"tag":62,"props":495,"children":496},{},[497],{"type":55,"value":498},"P0",{"type":55,"value":500}," blocks operation · ",{"type":50,"tag":62,"props":502,"children":503},{},[504],{"type":55,"value":505},"P1",{"type":55,"value":507}," degraded · ",{"type":50,"tag":62,"props":509,"children":510},{},[511],{"type":55,"value":512},"P2",{"type":55,"value":514}," informational.",{"type":50,"tag":194,"props":516,"children":518},{"id":517},"step-2-match-signal-section",[519],{"type":55,"value":520},"Step 2: Match signal → section",{"type":50,"tag":58,"props":522,"children":523},{},[524],{"type":50,"tag":62,"props":525,"children":526},{},[527,529,535],{"type":55,"value":528},"Events (",{"type":50,"tag":94,"props":530,"children":532},{"className":531},[],[533],{"type":55,"value":534},"list-cluster-events",{"type":55,"value":536},") — provisioning-time:",{"type":50,"tag":538,"props":539,"children":540},"table",{},[541,560],{"type":50,"tag":542,"props":543,"children":544},"thead",{},[545],{"type":50,"tag":546,"props":547,"children":548},"tr",{},[549,555],{"type":50,"tag":550,"props":551,"children":552},"th",{},[553],{"type":55,"value":554},"Event",{"type":50,"tag":550,"props":556,"children":557},{},[558],{"type":55,"value":559},"Section",{"type":50,"tag":561,"props":562,"children":563},"tbody",{},[564,591,621,641,661],{"type":50,"tag":546,"props":565,"children":566},{},[567,579],{"type":50,"tag":568,"props":569,"children":570},"td",{},[571,577],{"type":50,"tag":94,"props":572,"children":574},{"className":573},[],[575],{"type":55,"value":576},"\"EFA health checks did not run successfully\"",{"type":55,"value":578}," (public-doc verbatim signal)",{"type":50,"tag":568,"props":580,"children":581},{},[582],{"type":50,"tag":62,"props":583,"children":584},{},[585],{"type":50,"tag":182,"props":586,"children":588},{"href":587},"#a-efa--security-group",[589],{"type":55,"value":590},"A: EFA\u002FSG",{"type":50,"tag":546,"props":592,"children":593},{},[594,599],{"type":50,"tag":568,"props":595,"children":596},{},[597],{"type":55,"value":598},"Instance bootstrap or network-misconfiguration event",{"type":50,"tag":568,"props":600,"children":601},{},[602,610,612],{"type":50,"tag":62,"props":603,"children":604},{},[605],{"type":50,"tag":182,"props":606,"children":607},{"href":587},[608],{"type":55,"value":609},"A",{"type":55,"value":611}," + ",{"type":50,"tag":62,"props":613,"children":614},{},[615],{"type":50,"tag":182,"props":616,"children":618},{"href":617},"#b-vpc--routing",[619],{"type":55,"value":620},"B: VPC",{"type":50,"tag":546,"props":622,"children":623},{},[624,629],{"type":50,"tag":568,"props":625,"children":626},{},[627],{"type":55,"value":628},"Lifecycle-script failure or timeout",{"type":50,"tag":568,"props":630,"children":631},{},[632],{"type":50,"tag":62,"props":633,"children":634},{},[635],{"type":50,"tag":182,"props":636,"children":638},{"href":637},"#d-lifecycle-scripts",[639],{"type":55,"value":640},"D: Lifecycle",{"type":50,"tag":546,"props":642,"children":643},{},[644,649],{"type":50,"tag":568,"props":645,"children":646},{},[647],{"type":55,"value":648},"Insufficient-capacity or AZ-mismatch failure at creation",{"type":50,"tag":568,"props":650,"children":651},{},[652],{"type":50,"tag":62,"props":653,"children":654},{},[655],{"type":50,"tag":182,"props":656,"children":658},{"href":657},"#c-capacity--az",[659],{"type":55,"value":660},"C: Capacity",{"type":50,"tag":546,"props":662,"children":663},{},[664,675],{"type":50,"tag":568,"props":665,"children":666},{},[667,669],{"type":55,"value":668},"Hardware failure \u002F ",{"type":50,"tag":94,"props":670,"children":672},{"className":671},[],[673],{"type":55,"value":674},"UnschedulablePendingReplacement",{"type":50,"tag":568,"props":676,"children":677},{},[678],{"type":50,"tag":62,"props":679,"children":680},{},[681],{"type":50,"tag":182,"props":682,"children":684},{"href":683},"#f-hardware--auto-repair",[685],{"type":55,"value":686},"F: Hardware",{"type":50,"tag":58,"props":688,"children":689},{},[690],{"type":50,"tag":62,"props":691,"children":692},{},[693],{"type":55,"value":694},"EKS labels:",{"type":50,"tag":538,"props":696,"children":697},{},[698,713],{"type":50,"tag":542,"props":699,"children":700},{},[701],{"type":50,"tag":546,"props":702,"children":703},{},[704,709],{"type":50,"tag":550,"props":705,"children":706},{},[707],{"type":55,"value":708},"Label",{"type":50,"tag":550,"props":710,"children":711},{},[712],{"type":55,"value":559},{"type":50,"tag":561,"props":714,"children":715},{},[716,739,761],{"type":50,"tag":546,"props":717,"children":718},{},[719,728],{"type":50,"tag":568,"props":720,"children":721},{},[722],{"type":50,"tag":94,"props":723,"children":725},{"className":724},[],[726],{"type":55,"value":727},"node-health-status: UnschedulablePendingReplacement",{"type":50,"tag":568,"props":729,"children":730},{},[731],{"type":50,"tag":62,"props":732,"children":733},{},[734],{"type":50,"tag":182,"props":735,"children":736},{"href":683},[737],{"type":55,"value":738},"F",{"type":50,"tag":546,"props":740,"children":741},{},[742,751],{"type":50,"tag":568,"props":743,"children":744},{},[745],{"type":50,"tag":94,"props":746,"children":748},{"className":747},[],[749],{"type":55,"value":750},"node-health-status: UnschedulablePendingReboot",{"type":50,"tag":568,"props":752,"children":753},{},[754],{"type":50,"tag":62,"props":755,"children":756},{},[757],{"type":50,"tag":182,"props":758,"children":759},{"href":683},[760],{"type":55,"value":738},{"type":50,"tag":546,"props":762,"children":763},{},[764,773],{"type":50,"tag":568,"props":765,"children":766},{},[767],{"type":50,"tag":94,"props":768,"children":770},{"className":769},[],[771],{"type":55,"value":772},"deep-health-check-status: Failed",{"type":50,"tag":568,"props":774,"children":775},{},[776,785,787],{"type":50,"tag":62,"props":777,"children":778},{},[779],{"type":50,"tag":182,"props":780,"children":782},{"href":781},"#g-gpu--accelerator",[783],{"type":55,"value":784},"G",{"type":55,"value":786}," → ",{"type":50,"tag":62,"props":788,"children":789},{},[790],{"type":50,"tag":182,"props":791,"children":792},{"href":683},[793],{"type":55,"value":738},{"type":50,"tag":58,"props":795,"children":796},{},[797],{"type":50,"tag":62,"props":798,"children":799},{},[800],{"type":55,"value":801},"Symptoms:",{"type":50,"tag":538,"props":803,"children":804},{},[805,820],{"type":50,"tag":542,"props":806,"children":807},{},[808],{"type":50,"tag":546,"props":809,"children":810},{},[811,816],{"type":50,"tag":550,"props":812,"children":813},{},[814],{"type":55,"value":815},"Symptom",{"type":50,"tag":550,"props":817,"children":818},{},[819],{"type":55,"value":559},{"type":50,"tag":561,"props":821,"children":822},{},[823,851,885,904,922,940,959,985,1005,1025,1051,1070,1095,1114,1133,1152,1172,1192],{"type":50,"tag":546,"props":824,"children":825},{},[826,831],{"type":50,"tag":568,"props":827,"children":828},{},[829],{"type":55,"value":830},"Training hangs at NCCL init \u002F AllReduce",{"type":50,"tag":568,"props":832,"children":833},{},[834,841,842],{"type":50,"tag":62,"props":835,"children":836},{},[837],{"type":50,"tag":182,"props":838,"children":839},{"href":587},[840],{"type":55,"value":609},{"type":55,"value":786},{"type":50,"tag":62,"props":843,"children":844},{},[845],{"type":50,"tag":182,"props":846,"children":848},{"href":847},"#e-software-versions",[849],{"type":55,"value":850},"E",{"type":50,"tag":546,"props":852,"children":853},{},[854,873],{"type":50,"tag":568,"props":855,"children":856},{},[857,859,865,867],{"type":55,"value":858},"Slurm node ",{"type":50,"tag":94,"props":860,"children":862},{"className":861},[],[863],{"type":55,"value":864},"down",{"type":55,"value":866}," \u002F ",{"type":50,"tag":94,"props":868,"children":870},{"className":869},[],[871],{"type":55,"value":872},"\"Node unexpectedly rebooted\"",{"type":50,"tag":568,"props":874,"children":875},{},[876],{"type":50,"tag":62,"props":877,"children":878},{},[879],{"type":50,"tag":182,"props":880,"children":882},{"href":881},"#h-slurm-node-management",[883],{"type":55,"value":884},"H: Slurm",{"type":50,"tag":546,"props":886,"children":887},{},[888,893],{"type":50,"tag":568,"props":889,"children":890},{},[891],{"type":55,"value":892},"Jobs stuck PENDING \u002F COMPLETING",{"type":50,"tag":568,"props":894,"children":895},{},[896],{"type":50,"tag":62,"props":897,"children":898},{},[899],{"type":50,"tag":182,"props":900,"children":901},{"href":881},[902],{"type":55,"value":903},"H",{"type":50,"tag":546,"props":905,"children":906},{},[907,912],{"type":50,"tag":568,"props":908,"children":909},{},[910],{"type":55,"value":911},"Auto-repair not triggering",{"type":50,"tag":568,"props":913,"children":914},{},[915],{"type":50,"tag":62,"props":916,"children":917},{},[918],{"type":50,"tag":182,"props":919,"children":920},{"href":683},[921],{"type":55,"value":738},{"type":50,"tag":546,"props":923,"children":924},{},[925,930],{"type":50,"tag":568,"props":926,"children":927},{},[928],{"type":55,"value":929},"GPU not visible \u002F XID \u002F ECC errors",{"type":50,"tag":568,"props":931,"children":932},{},[933],{"type":50,"tag":62,"props":934,"children":935},{},[936],{"type":50,"tag":182,"props":937,"children":938},{"href":781},[939],{"type":55,"value":784},{"type":50,"tag":546,"props":941,"children":942},{},[943,948],{"type":50,"tag":568,"props":944,"children":945},{},[946],{"type":55,"value":947},"GPU row-remap pending\u002Ffailed \u002F silent NaNs \u002F DCGM Fail",{"type":50,"tag":568,"props":949,"children":950},{},[951],{"type":50,"tag":62,"props":952,"children":953},{},[954],{"type":50,"tag":182,"props":955,"children":956},{"href":781},[957],{"type":55,"value":958},"G § G.1.a\u002Fb",{"type":50,"tag":546,"props":960,"children":961},{},[962,973],{"type":50,"tag":568,"props":963,"children":964},{},[965,967],{"type":55,"value":966},"Disk full \u002F OOM \u002F ",{"type":50,"tag":94,"props":968,"children":970},{"className":969},[],[971],{"type":55,"value":972},"\"Cannot allocate memory\"",{"type":50,"tag":568,"props":974,"children":975},{},[976],{"type":50,"tag":62,"props":977,"children":978},{},[979],{"type":50,"tag":182,"props":980,"children":982},{"href":981},"#i-resource-exhaustion",[983],{"type":55,"value":984},"I: Resources",{"type":50,"tag":546,"props":986,"children":987},{},[988,993],{"type":50,"tag":568,"props":989,"children":990},{},[991],{"type":55,"value":992},"Wrong vCPU count (e.g. 96 instead of 192 on p5.48xlarge)",{"type":50,"tag":568,"props":994,"children":995},{},[996],{"type":50,"tag":62,"props":997,"children":998},{},[999],{"type":50,"tag":182,"props":1000,"children":1002},{"href":1001},"#j-configuration",[1003],{"type":55,"value":1004},"J: Config",{"type":50,"tag":546,"props":1006,"children":1007},{},[1008,1013],{"type":50,"tag":568,"props":1009,"children":1010},{},[1011],{"type":55,"value":1012},"Container CrashLoopBackOff \u002F runtime crash",{"type":50,"tag":568,"props":1014,"children":1015},{},[1016],{"type":50,"tag":62,"props":1017,"children":1018},{},[1019],{"type":50,"tag":182,"props":1020,"children":1022},{"href":1021},"#m-container-runtime",[1023],{"type":55,"value":1024},"M: Container Runtime",{"type":50,"tag":546,"props":1026,"children":1027},{},[1028,1039],{"type":50,"tag":568,"props":1029,"children":1030},{},[1031,1037],{"type":50,"tag":94,"props":1032,"children":1034},{"className":1033},[],[1035],{"type":55,"value":1036},"aws-node",{"type":55,"value":1038}," CrashLoopBackOff \u002F gRPC 50051 refused",{"type":50,"tag":568,"props":1040,"children":1041},{},[1042],{"type":50,"tag":62,"props":1043,"children":1044},{},[1045],{"type":50,"tag":182,"props":1046,"children":1048},{"href":1047},"#o-cni--pod-networking",[1049],{"type":55,"value":1050},"O: CNI \u002F Pod Networking",{"type":50,"tag":546,"props":1052,"children":1053},{},[1054,1059],{"type":50,"tag":568,"props":1055,"children":1056},{},[1057],{"type":55,"value":1058},"Pods stuck Pending with no IP \u002F CNI error",{"type":50,"tag":568,"props":1060,"children":1061},{},[1062],{"type":50,"tag":62,"props":1063,"children":1064},{},[1065],{"type":50,"tag":182,"props":1066,"children":1067},{"href":1047},[1068],{"type":55,"value":1069},"O",{"type":50,"tag":546,"props":1071,"children":1072},{},[1073,1084],{"type":50,"tag":568,"props":1074,"children":1075},{},[1076,1078],{"type":55,"value":1077},"DNS resolution \u002F ",{"type":50,"tag":94,"props":1079,"children":1081},{"className":1080},[],[1082],{"type":55,"value":1083},"enableDnsSupport",{"type":50,"tag":568,"props":1085,"children":1086},{},[1087],{"type":50,"tag":62,"props":1088,"children":1089},{},[1090],{"type":50,"tag":182,"props":1091,"children":1092},{"href":617},[1093],{"type":55,"value":1094},"B § B.2",{"type":50,"tag":546,"props":1096,"children":1097},{},[1098,1103],{"type":50,"tag":568,"props":1099,"children":1100},{},[1101],{"type":55,"value":1102},"Public subnet \u002F IGW misconfigured",{"type":50,"tag":568,"props":1104,"children":1105},{},[1106],{"type":50,"tag":62,"props":1107,"children":1108},{},[1109],{"type":50,"tag":182,"props":1110,"children":1111},{"href":617},[1112],{"type":55,"value":1113},"B § B.3",{"type":50,"tag":546,"props":1115,"children":1116},{},[1117,1122],{"type":50,"tag":568,"props":1118,"children":1119},{},[1120],{"type":55,"value":1121},"Missing VPC endpoints (ECR \u002F STS \u002F FSx)",{"type":50,"tag":568,"props":1123,"children":1124},{},[1125],{"type":50,"tag":62,"props":1126,"children":1127},{},[1128],{"type":50,"tag":182,"props":1129,"children":1130},{"href":617},[1131],{"type":55,"value":1132},"B § B.4",{"type":50,"tag":546,"props":1134,"children":1135},{},[1136,1141],{"type":50,"tag":568,"props":1137,"children":1138},{},[1139],{"type":55,"value":1140},"EKS VPC \u002F SG mismatch with HyperPod",{"type":50,"tag":568,"props":1142,"children":1143},{},[1144],{"type":50,"tag":62,"props":1145,"children":1146},{},[1147],{"type":50,"tag":182,"props":1148,"children":1149},{"href":617},[1150],{"type":55,"value":1151},"B § B.5",{"type":50,"tag":546,"props":1153,"children":1154},{},[1155,1160],{"type":50,"tag":568,"props":1156,"children":1157},{},[1158],{"type":55,"value":1159},"Kernel panic \u002F watchdog \u002F hung task",{"type":50,"tag":568,"props":1161,"children":1162},{},[1163],{"type":50,"tag":62,"props":1164,"children":1165},{},[1166],{"type":50,"tag":182,"props":1167,"children":1169},{"href":1168},"#n-kernel--system",[1170],{"type":55,"value":1171},"N: Kernel",{"type":50,"tag":546,"props":1173,"children":1174},{},[1175,1180],{"type":50,"tag":568,"props":1176,"children":1177},{},[1178],{"type":55,"value":1179},"Need shell on a node",{"type":50,"tag":568,"props":1181,"children":1182},{},[1183],{"type":50,"tag":62,"props":1184,"children":1185},{},[1186],{"type":50,"tag":182,"props":1187,"children":1189},{"href":1188},"#k-node-access-via-ssm",[1190],{"type":55,"value":1191},"K: SSM",{"type":50,"tag":546,"props":1193,"children":1194},{},[1195,1200],{"type":50,"tag":568,"props":1196,"children":1197},{},[1198],{"type":55,"value":1199},"Collect logs for AWS Support",{"type":50,"tag":568,"props":1201,"children":1202},{},[1203],{"type":50,"tag":62,"props":1204,"children":1205},{},[1206],{"type":50,"tag":182,"props":1207,"children":1209},{"href":1208},"#l-log-collection",[1210],{"type":55,"value":1211},"L: Log Collection",{"type":50,"tag":190,"props":1213,"children":1214},{},[],{"type":50,"tag":194,"props":1216,"children":1218},{"id":1217},"a-efa-security-group",[1219],{"type":55,"value":1220},"A: EFA \u002F Security Group",{"type":50,"tag":58,"props":1222,"children":1223},{},[1224,1226,1232,1234,1240,1242,1248],{"type":55,"value":1225},"Per the HyperPod prerequisites doc, the SG must allow all inbound and outbound to itself. ",{"type":50,"tag":94,"props":1227,"children":1229},{"className":1228},[],[1230],{"type":55,"value":1231},"scripts\u002Fcheck-efa-sg.sh",{"type":55,"value":1233}," validates self-ref rules on every cluster SG. On-node EFA check via ",{"type":50,"tag":94,"props":1235,"children":1237},{"className":1236},[],[1238],{"type":55,"value":1239},"scripts\u002Fcheck-node-reachability.sh",{"type":55,"value":1241}," over SSM. Full: ",{"type":50,"tag":182,"props":1243,"children":1245},{"href":1244},"references\u002Fnode-diagnostics-detail.md#a-efa--security-group",[1246],{"type":55,"value":1247},"§ A",{"type":55,"value":188},{"type":50,"tag":194,"props":1250,"children":1252},{"id":1251},"b-vpc-routing",[1253],{"type":55,"value":1254},"B: VPC \u002F Routing",{"type":50,"tag":58,"props":1256,"children":1257},{},[1258,1260,1266,1268,1274],{"type":55,"value":1259},"SG\u002Fsubnet VPC mismatch, missing S3 Gateway endpoint, EKS auth mode, worker→controller routing, VPC DNS support, private-subnet + NAT \u002F VPC endpoints, EKS↔HyperPod VPC alignment. ",{"type":50,"tag":94,"props":1261,"children":1263},{"className":1262},[],[1264],{"type":55,"value":1265},"scripts\u002Fcheck-vpc-config.sh",{"type":55,"value":1267},". Full: ",{"type":50,"tag":182,"props":1269,"children":1271},{"href":1270},"references\u002Fnode-diagnostics-detail.md#b-vpc--routing",[1272],{"type":55,"value":1273},"§ B",{"type":55,"value":188},{"type":50,"tag":194,"props":1276,"children":1278},{"id":1277},"c-capacity-az",[1279],{"type":55,"value":1280},"C: Capacity \u002F AZ",{"type":50,"tag":58,"props":1282,"children":1283},{},[1284,1286,1292,1294,1300],{"type":55,"value":1285},"Insufficient-capacity failure at creation, or no subnets in the AZ where capacity is available. Check AZ offerings via ",{"type":50,"tag":94,"props":1287,"children":1289},{"className":1288},[],[1290],{"type":55,"value":1291},"describe-instance-type-offerings",{"type":55,"value":1293},", then change subnet AZ or use Flexible Training Plans \u002F ODCR. Full: ",{"type":50,"tag":182,"props":1295,"children":1297},{"href":1296},"references\u002Fnode-diagnostics-detail.md#c-capacity--az",[1298],{"type":55,"value":1299},"§ C",{"type":55,"value":188},{"type":50,"tag":194,"props":1302,"children":1304},{"id":1303},"d-lifecycle-scripts",[1305],{"type":55,"value":1306},"D: Lifecycle Scripts",{"type":50,"tag":58,"props":1308,"children":1309},{},[1310,1312,1318,1320,1326],{"type":55,"value":1311},"Surfaced in cluster events + CloudWatch under ",{"type":50,"tag":94,"props":1313,"children":1315},{"className":1314},[],[1316],{"type":55,"value":1317},"LifecycleConfig\u002F\u003Cgroup>\u002F\u003Cinstance-id>",{"type":55,"value":1319},". Common: S3 connectivity, IAM gaps, CRLF line endings, infinite loops, parameter-name mismatch. Full: ",{"type":50,"tag":182,"props":1321,"children":1323},{"href":1322},"references\u002Fnode-diagnostics-detail.md#d-lifecycle-scripts",[1324],{"type":55,"value":1325},"§ D",{"type":55,"value":188},{"type":50,"tag":194,"props":1328,"children":1330},{"id":1329},"e-software-versions",[1331],{"type":55,"value":1332},"E: Software Versions",{"type":50,"tag":58,"props":1334,"children":1335},{},[1336,1338,1344,1346,1352,1353,1359,1360,1366,1367,1373],{"type":55,"value":1337},"Delegate to ",{"type":50,"tag":94,"props":1339,"children":1341},{"className":1340},[],[1342],{"type":55,"value":1343},"hyperpod-version-checker",{"type":55,"value":1345}," to compare NVIDIA driver, CUDA, NCCL, EFA installer, OFI NCCL, PyTorch across nodes. Ensure job env has ",{"type":50,"tag":94,"props":1347,"children":1349},{"className":1348},[],[1350],{"type":55,"value":1351},"FI_PROVIDER=efa",{"type":55,"value":101},{"type":50,"tag":94,"props":1354,"children":1356},{"className":1355},[],[1357],{"type":55,"value":1358},"FI_EFA_USE_DEVICE_RDMA=1",{"type":55,"value":101},{"type":50,"tag":94,"props":1361,"children":1363},{"className":1362},[],[1364],{"type":55,"value":1365},"NCCL_SOCKET_IFNAME=^lo,docker",{"type":55,"value":1267},{"type":50,"tag":182,"props":1368,"children":1370},{"href":1369},"references\u002Fnode-diagnostics-detail.md#e-software-versions",[1371],{"type":55,"value":1372},"§ E",{"type":55,"value":188},{"type":50,"tag":194,"props":1375,"children":1377},{"id":1376},"f-hardware-auto-repair",[1378],{"type":55,"value":1379},"F: Hardware \u002F Auto-Repair",{"type":50,"tag":58,"props":1381,"children":1382},{},[1383,1385,1391,1393,1399,1401,1407,1409,1414,1416,1422,1424,1430,1432,1438,1440,1446,1448,1454,1456,1461],{"type":55,"value":1384},"Confirm ",{"type":50,"tag":94,"props":1386,"children":1388},{"className":1387},[],[1389],{"type":55,"value":1390},"NodeRecovery=Automatic",{"type":55,"value":1392},", inspect the EKS health labels + ",{"type":50,"tag":94,"props":1394,"children":1396},{"className":1395},[],[1397],{"type":55,"value":1398},"sagemaker.amazonaws.com\u002Ffault-details",{"type":55,"value":1400}," annotation, and read the ",{"type":50,"tag":94,"props":1402,"children":1404},{"className":1403},[],[1405],{"type":55,"value":1406},"SagemakerHealthMonitoringAgent\u002F\u003Cgroup>\u002F\u003Cinstance>",{"type":55,"value":1408}," CloudWatch stream. HMA runs passive background checks on GPU and Neuron state and ",{"type":50,"tag":62,"props":1410,"children":1411},{},[1412],{"type":55,"value":1413},"reboots",{"type":55,"value":1415}," the node on count mismatch (per the HMA doc: \"if there's a mismatch between the expected number of GPUs … and the count returned by ",{"type":50,"tag":94,"props":1417,"children":1419},{"className":1418},[],[1420],{"type":55,"value":1421},"nvidia-smi",{"type":55,"value":1423},", then HMA reboots the node\"; same for ",{"type":50,"tag":94,"props":1425,"children":1427},{"className":1426},[],[1428],{"type":55,"value":1429},"neuron-ls",{"type":55,"value":1431},"). Manual recovery order: reboot first, replace only if reboot fails; the preferred path is the batch APIs (",{"type":50,"tag":94,"props":1433,"children":1435},{"className":1434},[],[1436],{"type":55,"value":1437},"BatchReboot",{"type":55,"value":1439},"\u002F",{"type":50,"tag":94,"props":1441,"children":1443},{"className":1442},[],[1444],{"type":55,"value":1445},"BatchReplaceClusterNodes",{"type":55,"value":1447},"). Full: ",{"type":50,"tag":182,"props":1449,"children":1451},{"href":1450},"references\u002Fnode-diagnostics-detail.md#f-hardware--auto-repair",[1452],{"type":55,"value":1453},"§ F",{"type":55,"value":1455}," · patterns: ",{"type":50,"tag":182,"props":1457,"children":1458},{"href":184},[1459],{"type":55,"value":1460},"node-issue-catalog.md",{"type":55,"value":188},{"type":50,"tag":194,"props":1463,"children":1465},{"id":1464},"g-gpu-accelerator",[1466],{"type":55,"value":1467},"G: GPU \u002F Accelerator",{"type":50,"tag":58,"props":1469,"children":1470},{},[1471,1476,1478,1483,1484,1490],{"type":50,"tag":62,"props":1472,"children":1473},{},[1474],{"type":55,"value":1475},"NVIDIA (p4d\u002Fp5\u002Fg5\u002Fg6):",{"type":55,"value":1477}," ",{"type":50,"tag":94,"props":1479,"children":1481},{"className":1480},[],[1482],{"type":55,"value":1421},{"type":55,"value":611},{"type":50,"tag":94,"props":1485,"children":1487},{"className":1486},[],[1488],{"type":55,"value":1489},"dmesg",{"type":55,"value":1491}," over SSM for Xid, ECC, thermal throttling. Xid classification per NVIDIA's catalog: 13 Graphics Engine Exception (application-level), 31 GPU memory page fault (application, can be driver\u002FHW), 63 GPU memory remapping event (HW\u002FECC), 71 CE4 Error (HW copy engine), 74 NVLink Error (HW), 79 GPU has fallen off the bus (PCIe bus), 109 Context Switch Timeout Error (HW). Any uncorrectable ECC → drain and replace. Row-remap state is the authoritative silent-degradation signal (§ G.1.a).",{"type":50,"tag":58,"props":1493,"children":1494},{},[1495,1500,1502,1507,1508,1514,1515,1521,1523,1528],{"type":50,"tag":62,"props":1496,"children":1497},{},[1498],{"type":55,"value":1499},"Trainium \u002F Inferentia (trn1\u002Ftrn2\u002Finf2):",{"type":55,"value":1501}," Neuron SDK — ",{"type":50,"tag":94,"props":1503,"children":1505},{"className":1504},[],[1506],{"type":55,"value":1429},{"type":55,"value":101},{"type":50,"tag":94,"props":1509,"children":1511},{"className":1510},[],[1512],{"type":55,"value":1513},"neuron-top",{"type":55,"value":101},{"type":50,"tag":94,"props":1516,"children":1518},{"className":1517},[],[1519],{"type":55,"value":1520},"neuron-monitor",{"type":55,"value":1522},". ",{"type":50,"tag":94,"props":1524,"children":1526},{"className":1525},[],[1527],{"type":55,"value":1421},{"type":55,"value":1529}," does not apply.",{"type":50,"tag":58,"props":1531,"children":1532},{},[1533,1535,1541],{"type":55,"value":1534},"GPU \u002F accelerator failures flow into § F for reboot \u002F replace. Full: ",{"type":50,"tag":182,"props":1536,"children":1538},{"href":1537},"references\u002Fnode-diagnostics-detail.md#g-gpuaccelerator",[1539],{"type":55,"value":1540},"§ G",{"type":55,"value":188},{"type":50,"tag":194,"props":1543,"children":1545},{"id":1544},"h-slurm-node-management",[1546],{"type":55,"value":1547},"H: Slurm Node Management",{"type":50,"tag":58,"props":1549,"children":1550},{},[1551,1553,1559,1561,1567],{"type":55,"value":1552},"Node down\u002Funresponsive, unexpected reboots, stuck PENDING\u002FCOMPLETING jobs, Slurm-to-instance-ID translation. Primary access is SSM; diagnose ",{"type":50,"tag":94,"props":1554,"children":1556},{"className":1555},[],[1557],{"type":55,"value":1558},"slurmd",{"type":55,"value":1560}," first, fix the root cause, then start\u002Fresume the node per § H. Full: ",{"type":50,"tag":182,"props":1562,"children":1564},{"href":1563},"references\u002Fnode-diagnostics-detail.md#h-slurm-node-management",[1565],{"type":55,"value":1566},"§ H",{"type":55,"value":188},{"type":50,"tag":194,"props":1569,"children":1571},{"id":1570},"i-resource-exhaustion",[1572],{"type":55,"value":1573},"I: Resource Exhaustion",{"type":50,"tag":58,"props":1575,"children":1576},{},[1577,1579,1585,1587,1593,1595,1601,1603,1609,1611,1617,1619,1625],{"type":55,"value":1578},"Disk full (HyperPod root volume defaults to 100 GB and is not intended to grow post-creation), OOM, ",{"type":50,"tag":94,"props":1580,"children":1582},{"className":1581},[],[1583],{"type":55,"value":1584},"os.fork()",{"type":55,"value":1586}," memory error, ",{"type":50,"tag":94,"props":1588,"children":1590},{"className":1589},[],[1591],{"type":55,"value":1592},"\u002Fdev\u002Fshm",{"type":55,"value":1594}," exhaustion, inode exhaustion. Fork-memory fix: ",{"type":50,"tag":94,"props":1596,"children":1598},{"className":1597},[],[1599],{"type":55,"value":1600},"export FI_EFA_USE_HUGE_PAGE=0",{"type":55,"value":1602},". Redirect bulk data to ",{"type":50,"tag":94,"props":1604,"children":1606},{"className":1605},[],[1607],{"type":55,"value":1608},"\u002Fopt\u002Fsagemaker",{"type":55,"value":1610}," (secondary EBS) or ",{"type":50,"tag":94,"props":1612,"children":1614},{"className":1613},[],[1615],{"type":55,"value":1616},"\u002Fopt\u002Fdlami\u002Fnvme",{"type":55,"value":1618}," (instance store). Full: ",{"type":50,"tag":182,"props":1620,"children":1622},{"href":1621},"references\u002Fnode-diagnostics-detail.md#i-resource-exhaustion",[1623],{"type":55,"value":1624},"§ I",{"type":55,"value":188},{"type":50,"tag":194,"props":1627,"children":1629},{"id":1628},"j-configuration",[1630],{"type":55,"value":1631},"J: Configuration",{"type":50,"tag":58,"props":1633,"children":1634},{},[1635,1637,1643,1645,1650,1651,1657],{"type":55,"value":1636},"p5.48xlarge reports 96 vCPU instead of 192 → set ",{"type":50,"tag":94,"props":1638,"children":1640},{"className":1639},[],[1641],{"type":55,"value":1642},"ThreadsPerCore=2",{"type":55,"value":1644}," via ",{"type":50,"tag":94,"props":1646,"children":1648},{"className":1647},[],[1649],{"type":55,"value":114},{"type":55,"value":1267},{"type":50,"tag":182,"props":1652,"children":1654},{"href":1653},"references\u002Fnode-diagnostics-detail.md#j-configuration",[1655],{"type":55,"value":1656},"§ J",{"type":55,"value":188},{"type":50,"tag":194,"props":1659,"children":1661},{"id":1660},"k-node-access-via-ssm",[1662],{"type":55,"value":1663},"K: Node Access via SSM",{"type":50,"tag":58,"props":1665,"children":1666},{},[1667,1669,1675,1677,1683],{"type":55,"value":1668},"No direct SSH on HyperPod. Target format ",{"type":50,"tag":94,"props":1670,"children":1672},{"className":1671},[],[1673],{"type":55,"value":1674},"sagemaker-cluster:\u003CCLUSTER_ID>_\u003CGROUP>-\u003CINSTANCE_ID>",{"type":55,"value":1676},". Failures: plugin missing, wrong prefix, IAM, VPC endpoints. Full: ",{"type":50,"tag":182,"props":1678,"children":1680},{"href":1679},"references\u002Fnode-diagnostics-detail.md#k-node-access-via-ssm",[1681],{"type":55,"value":1682},"§ K",{"type":55,"value":188},{"type":50,"tag":194,"props":1685,"children":1687},{"id":1686},"l-log-collection",[1688],{"type":55,"value":1211},{"type":50,"tag":58,"props":1690,"children":1691},{},[1692,1693,1699,1701,1706,1707,1713,1714,1720],{"type":55,"value":1337},{"type":50,"tag":94,"props":1694,"children":1696},{"className":1695},[],[1697],{"type":55,"value":1698},"hyperpod-issue-report",{"type":55,"value":1700}," for S3-stored bundles. Key CloudWatch streams: ",{"type":50,"tag":94,"props":1702,"children":1704},{"className":1703},[],[1705],{"type":55,"value":1317},{"type":55,"value":101},{"type":50,"tag":94,"props":1708,"children":1710},{"className":1709},[],[1711],{"type":55,"value":1712},"SagemakerHealthMonitoringAgent\u002F\u003Cgroup>\u002F\u003Cinstance-id>",{"type":55,"value":1267},{"type":50,"tag":182,"props":1715,"children":1717},{"href":1716},"references\u002Fnode-diagnostics-detail.md#l-log-collection",[1718],{"type":55,"value":1719},"§ L",{"type":55,"value":188},{"type":50,"tag":194,"props":1722,"children":1724},{"id":1723},"m-container-runtime",[1725],{"type":55,"value":1024},{"type":50,"tag":58,"props":1727,"children":1728},{},[1729,1731,1737,1739,1745,1746,1752,1753,1759],{"type":55,"value":1730},"CrashLoopBackOff, OOMKilled, ImagePullBackOff, RunContainerError on EKS. ",{"type":50,"tag":94,"props":1732,"children":1734},{"className":1733},[],[1735],{"type":55,"value":1736},"kubectl describe pod",{"type":55,"value":1738}," + on-node ",{"type":50,"tag":94,"props":1740,"children":1742},{"className":1741},[],[1743],{"type":55,"value":1744},"crictl ps -a",{"type":55,"value":101},{"type":50,"tag":94,"props":1747,"children":1749},{"className":1748},[],[1750],{"type":55,"value":1751},"journalctl -u containerd",{"type":55,"value":1267},{"type":50,"tag":182,"props":1754,"children":1756},{"href":1755},"references\u002Fnode-diagnostics-detail.md#m-container-runtime",[1757],{"type":55,"value":1758},"§ M",{"type":55,"value":188},{"type":50,"tag":194,"props":1761,"children":1763},{"id":1762},"n-kernel-system",[1764],{"type":55,"value":1765},"N: Kernel & System",{"type":50,"tag":58,"props":1767,"children":1768},{},[1769,1771,1777,1778,1784,1786,1792],{"type":55,"value":1770},"Kernel panic, watchdog timeout, soft lockup, unexpected reboots not explained by HyperPod health monitoring. ",{"type":50,"tag":94,"props":1772,"children":1774},{"className":1773},[],[1775],{"type":55,"value":1776},"dmesg | grep -iE 'panic|watchdog|hung_task|NMI'",{"type":55,"value":611},{"type":50,"tag":94,"props":1779,"children":1781},{"className":1780},[],[1782],{"type":55,"value":1783},"journalctl -b -1",{"type":55,"value":1785},". nvrm-related signatures point at NVIDIA driver crashes. Full: ",{"type":50,"tag":182,"props":1787,"children":1789},{"href":1788},"references\u002Fnode-diagnostics-detail.md#n-kernel--system",[1790],{"type":55,"value":1791},"§ N",{"type":55,"value":188},{"type":50,"tag":194,"props":1794,"children":1796},{"id":1795},"o-cni-pod-networking",[1797],{"type":55,"value":1050},{"type":50,"tag":58,"props":1799,"children":1800},{},[1801,1803,1808,1810,1816,1818,1824,1826,1831,1832,1838,1840,1846],{"type":55,"value":1802},"VPC CNI (",{"type":50,"tag":94,"props":1804,"children":1806},{"className":1805},[],[1807],{"type":55,"value":1036},{"type":55,"value":1809},") failures, IPAMD errors, gRPC 127.0.0.1:50051 refused, pods stuck ",{"type":50,"tag":94,"props":1811,"children":1813},{"className":1812},[],[1814],{"type":55,"value":1815},"Pending",{"type":55,"value":1817}," with ",{"type":50,"tag":94,"props":1819,"children":1821},{"className":1820},[],[1822],{"type":55,"value":1823},"FailedCreatePodSandBox",{"type":55,"value":1825},". Script auto-checks ",{"type":50,"tag":94,"props":1827,"children":1829},{"className":1828},[],[1830],{"type":55,"value":1036},{"type":55,"value":101},{"type":50,"tag":94,"props":1833,"children":1835},{"className":1834},[],[1836],{"type":55,"value":1837},"kube-proxy",{"type":55,"value":1839},", CoreDNS. Full: ",{"type":50,"tag":182,"props":1841,"children":1843},{"href":1842},"references\u002Fnode-diagnostics-detail.md#o-cni--pod-networking",[1844],{"type":55,"value":1845},"§ O",{"type":55,"value":188},{"type":50,"tag":190,"props":1848,"children":1849},{},[],{"type":50,"tag":194,"props":1851,"children":1853},{"id":1852},"prerequisites",[1854],{"type":55,"value":1855},"Prerequisites",{"type":50,"tag":1857,"props":1858,"children":1859},"ul",{},[1860,1899,1916,1927,1938],{"type":50,"tag":205,"props":1861,"children":1862},{},[1863,1868,1870,1876,1877,1883,1884,1890,1891,1897],{"type":50,"tag":94,"props":1864,"children":1866},{"className":1865},[],[1867],{"type":55,"value":22},{"type":55,"value":1869}," CLI v2, recent enough to support the HyperPod cluster commands (",{"type":50,"tag":94,"props":1871,"children":1873},{"className":1872},[],[1874],{"type":55,"value":1875},"describe-cluster",{"type":55,"value":101},{"type":50,"tag":94,"props":1878,"children":1880},{"className":1879},[],[1881],{"type":55,"value":1882},"list-cluster-nodes",{"type":55,"value":101},{"type":50,"tag":94,"props":1885,"children":1887},{"className":1886},[],[1888],{"type":55,"value":1889},"batch-reboot-cluster-nodes",{"type":55,"value":101},{"type":50,"tag":94,"props":1892,"children":1894},{"className":1893},[],[1895],{"type":55,"value":1896},"batch-replace-cluster-nodes",{"type":55,"value":1898},")",{"type":50,"tag":205,"props":1900,"children":1901},{},[1902,1908,1909,1914],{"type":50,"tag":94,"props":1903,"children":1905},{"className":1904},[],[1906],{"type":55,"value":1907},"python3",{"type":55,"value":101},{"type":50,"tag":94,"props":1910,"children":1912},{"className":1911},[],[1913],{"type":55,"value":277},{"type":55,"value":1915}," 4+ (associative arrays are required by the scripts)",{"type":50,"tag":205,"props":1917,"children":1918},{},[1919,1925],{"type":50,"tag":94,"props":1920,"children":1922},{"className":1921},[],[1923],{"type":55,"value":1924},"kubectl",{"type":55,"value":1926}," authenticated to the EKS cluster (K8s checks skipped if absent)",{"type":50,"tag":205,"props":1928,"children":1929},{},[1930,1936],{"type":50,"tag":94,"props":1931,"children":1933},{"className":1932},[],[1934],{"type":55,"value":1935},"session-manager-plugin",{"type":55,"value":1937}," for on-node hardware checks",{"type":50,"tag":205,"props":1939,"children":1940},{},[1941,1947,1949,1955,1957,1963,1964,1970],{"type":50,"tag":94,"props":1942,"children":1944},{"className":1943},[],[1945],{"type":55,"value":1946},"unbuffer",{"type":55,"value":1948}," (from the ",{"type":50,"tag":94,"props":1950,"children":1952},{"className":1951},[],[1953],{"type":55,"value":1954},"expect",{"type":55,"value":1956}," package) — optional; if missing, SSM on-node probes are skipped while the rest of the triage still runs. Install via ",{"type":50,"tag":94,"props":1958,"children":1960},{"className":1959},[],[1961],{"type":55,"value":1962},"yum install expect",{"type":55,"value":866},{"type":50,"tag":94,"props":1965,"children":1967},{"className":1966},[],[1968],{"type":55,"value":1969},"apt install expect",{"type":55,"value":188},{"type":50,"tag":194,"props":1972,"children":1974},{"id":1973},"defaults",[1975],{"type":55,"value":1976},"Defaults",{"type":50,"tag":1857,"props":1978,"children":1979},{},[1980,2005,2023,2033,2043,2053],{"type":50,"tag":205,"props":1981,"children":1982},{},[1983,1988,1990,1996,1998,2004],{"type":50,"tag":62,"props":1984,"children":1985},{},[1986],{"type":55,"value":1987},"Region",{"type":55,"value":1989}," — required: pass ",{"type":50,"tag":94,"props":1991,"children":1993},{"className":1992},[],[1994],{"type":55,"value":1995},"--region",{"type":55,"value":1997}," or set ",{"type":50,"tag":94,"props":1999,"children":2001},{"className":2000},[],[2002],{"type":55,"value":2003},"$AWS_DEFAULT_REGION",{"type":55,"value":188},{"type":50,"tag":205,"props":2006,"children":2007},{},[2008,2013,2015,2021],{"type":50,"tag":62,"props":2009,"children":2010},{},[2011],{"type":55,"value":2012},"Target scope",{"type":55,"value":2014}," — all nodes; ",{"type":50,"tag":94,"props":2016,"children":2018},{"className":2017},[],[2019],{"type":55,"value":2020},"--node \u003CID>",{"type":55,"value":2022}," focuses one.",{"type":50,"tag":205,"props":2024,"children":2025},{},[2026,2031],{"type":50,"tag":62,"props":2027,"children":2028},{},[2029],{"type":55,"value":2030},"Event window",{"type":55,"value":2032}," — up to 500 most recent events (5 × 100, paginated).",{"type":50,"tag":205,"props":2034,"children":2035},{},[2036,2041],{"type":50,"tag":62,"props":2037,"children":2038},{},[2039],{"type":55,"value":2040},"Node list cap",{"type":55,"value":2042}," — up to 20,000 nodes (200 × 100); warns on cap.",{"type":50,"tag":205,"props":2044,"children":2045},{},[2046,2051],{"type":50,"tag":62,"props":2047,"children":2048},{},[2049],{"type":55,"value":2050},"SSM probes",{"type":55,"value":2052}," — 180 s per node with retry-on-throttle.",{"type":50,"tag":205,"props":2054,"children":2055},{},[2056,2061,2063,2069],{"type":50,"tag":62,"props":2057,"children":2058},{},[2059],{"type":55,"value":2060},"Colors",{"type":55,"value":2062}," — auto-disabled on non-TTY; ",{"type":50,"tag":94,"props":2064,"children":2066},{"className":2065},[],[2067],{"type":55,"value":2068},"--no-color",{"type":55,"value":2070}," to force off.",{"type":50,"tag":194,"props":2072,"children":2074},{"id":2073},"error-handling",[2075],{"type":55,"value":2076},"Error handling",{"type":50,"tag":538,"props":2078,"children":2079},{},[2080,2101],{"type":50,"tag":542,"props":2081,"children":2082},{},[2083],{"type":50,"tag":546,"props":2084,"children":2085},{},[2086,2091,2096],{"type":50,"tag":550,"props":2087,"children":2088},{},[2089],{"type":55,"value":2090},"Failure",{"type":50,"tag":550,"props":2092,"children":2093},{},[2094],{"type":55,"value":2095},"Script",{"type":50,"tag":550,"props":2097,"children":2098},{},[2099],{"type":55,"value":2100},"Tell the customer",{"type":50,"tag":561,"props":2102,"children":2103},{},[2104,2128,2150,2196,2219,2242,2283],{"type":50,"tag":546,"props":2105,"children":2106},{},[2107,2118,2123],{"type":50,"tag":568,"props":2108,"children":2109},{},[2110,2116],{"type":50,"tag":94,"props":2111,"children":2113},{"className":2112},[],[2114],{"type":55,"value":2115},"aws sts get-caller-identity",{"type":55,"value":2117}," fails",{"type":50,"tag":568,"props":2119,"children":2120},{},[2121],{"type":55,"value":2122},"Exit 1",{"type":50,"tag":568,"props":2124,"children":2125},{},[2126],{"type":55,"value":2127},"\"Fix AWS credentials and rerun.\"",{"type":50,"tag":546,"props":2129,"children":2130},{},[2131,2140,2145],{"type":50,"tag":568,"props":2132,"children":2133},{},[2134,2139],{"type":50,"tag":94,"props":2135,"children":2137},{"className":2136},[],[2138],{"type":55,"value":1875},{"type":55,"value":2117},{"type":50,"tag":568,"props":2141,"children":2142},{},[2143],{"type":55,"value":2144},"Exit 1 after listing region's clusters",{"type":50,"tag":568,"props":2146,"children":2147},{},[2148],{"type":55,"value":2149},"\"Confirm cluster name and region.\"",{"type":50,"tag":546,"props":2151,"children":2152},{},[2153,2178,2191],{"type":50,"tag":568,"props":2154,"children":2155},{},[2156,2162,2163,2169,2170,2176],{"type":50,"tag":94,"props":2157,"children":2159},{"className":2158},[],[2160],{"type":55,"value":2161},"sagemaker:*",{"type":55,"value":866},{"type":50,"tag":94,"props":2164,"children":2166},{"className":2165},[],[2167],{"type":55,"value":2168},"ec2:*",{"type":55,"value":866},{"type":50,"tag":94,"props":2171,"children":2173},{"className":2172},[],[2174],{"type":55,"value":2175},"logs:*",{"type":55,"value":2177}," AccessDenied",{"type":50,"tag":568,"props":2179,"children":2180},{},[2181,2183,2189],{"type":55,"value":2182},"Warn, add ",{"type":50,"tag":94,"props":2184,"children":2186},{"className":2185},[],[2187],{"type":55,"value":2188},"Missing IAM permission for \u003CAPI>",{"type":55,"value":2190},", continue",{"type":50,"tag":568,"props":2192,"children":2193},{},[2194],{"type":55,"value":2195},"\"Grant the listed IAM action and rerun.\"",{"type":50,"tag":546,"props":2197,"children":2198},{},[2199,2209,2214],{"type":50,"tag":568,"props":2200,"children":2201},{},[2202,2207],{"type":50,"tag":94,"props":2203,"children":2205},{"className":2204},[],[2206],{"type":55,"value":1924},{"type":55,"value":2208}," absent or unauthenticated",{"type":50,"tag":568,"props":2210,"children":2211},{},[2212],{"type":55,"value":2213},"Skip K8s checks",{"type":50,"tag":568,"props":2215,"children":2216},{},[2217],{"type":55,"value":2218},"\"Install\u002Fauthenticate kubectl (see § K).\"",{"type":50,"tag":546,"props":2220,"children":2221},{},[2222,2232,2237],{"type":50,"tag":568,"props":2223,"children":2224},{},[2225,2230],{"type":50,"tag":94,"props":2226,"children":2228},{"className":2227},[],[2229],{"type":55,"value":1935},{"type":55,"value":2231}," absent",{"type":50,"tag":568,"props":2233,"children":2234},{},[2235],{"type":55,"value":2236},"Skip on-node probes",{"type":50,"tag":568,"props":2238,"children":2239},{},[2240],{"type":55,"value":2241},"\"Install session-manager-plugin (see § K).\"",{"type":50,"tag":546,"props":2243,"children":2244},{},[2245,2258,2271],{"type":50,"tag":568,"props":2246,"children":2247},{},[2248,2250,2256],{"type":55,"value":2249},"SSM ",{"type":50,"tag":94,"props":2251,"children":2253},{"className":2252},[],[2254],{"type":55,"value":2255},"start-session",{"type":55,"value":2257}," fails or times out (180s)",{"type":50,"tag":568,"props":2259,"children":2260},{},[2261,2263,2269],{"type":55,"value":2262},"Mark node unreachable with ",{"type":50,"tag":94,"props":2264,"children":2266},{"className":2265},[],[2267],{"type":55,"value":2268},"→ § K",{"type":55,"value":2270}," pointer",{"type":50,"tag":568,"props":2272,"children":2273},{},[2274,2276,2281],{"type":55,"value":2275},"\"Rerun with ",{"type":50,"tag":94,"props":2277,"children":2279},{"className":2278},[],[2280],{"type":55,"value":2020},{"type":55,"value":2282}," to isolate; verify SSM agent on the node.\"",{"type":50,"tag":546,"props":2284,"children":2285},{},[2286,2291,2296],{"type":50,"tag":568,"props":2287,"children":2288},{},[2289],{"type":55,"value":2290},"Cluster > 20,000 nodes",{"type":50,"tag":568,"props":2292,"children":2293},{},[2294],{"type":55,"value":2295},"First 20,000 paginated; warn",{"type":50,"tag":568,"props":2297,"children":2298},{},[2299,2301,2307],{"type":55,"value":2300},"\"Use ",{"type":50,"tag":94,"props":2302,"children":2304},{"className":2303},[],[2305],{"type":55,"value":2306},"--node",{"type":55,"value":2308}," to target specific nodes.\"",{"type":50,"tag":58,"props":2310,"children":2311},{},[2312,2314,2320,2322,2328],{"type":55,"value":2313},"Exit codes: ",{"type":50,"tag":94,"props":2315,"children":2317},{"className":2316},[],[2318],{"type":55,"value":2319},"0",{"type":55,"value":2321}," triage complete · ",{"type":50,"tag":94,"props":2323,"children":2325},{"className":2324},[],[2326],{"type":55,"value":2327},"1",{"type":55,"value":2329}," cluster not found or fatal prerequisite missing.",{"type":50,"tag":194,"props":2331,"children":2333},{"id":2332},"iam-permissions",[2334],{"type":55,"value":2335},"IAM permissions",{"type":50,"tag":58,"props":2337,"children":2338},{},[2339,2341,2347,2348,2353,2354,2359,2361,2366],{"type":55,"value":2340},"Read-only diagnostic — covers ",{"type":50,"tag":94,"props":2342,"children":2344},{"className":2343},[],[2345],{"type":55,"value":2346},"triage-cluster.sh",{"type":55,"value":101},{"type":50,"tag":94,"props":2349,"children":2351},{"className":2350},[],[2352],{"type":55,"value":156},{"type":55,"value":101},{"type":50,"tag":94,"props":2355,"children":2357},{"className":2356},[],[2358],{"type":55,"value":170},{"type":55,"value":2360},", and ",{"type":50,"tag":94,"props":2362,"children":2364},{"className":2363},[],[2365],{"type":55,"value":163},{"type":55,"value":2367},":",{"type":50,"tag":273,"props":2369,"children":2373},{"className":2370,"code":2371,"language":2372,"meta":278,"style":278},"language-json shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","{\n  \"Action\": [\n    \"sagemaker:DescribeCluster\",\n    \"sagemaker:DescribeClusterNode\",\n    \"sagemaker:ListClusterNodes\",\n    \"sagemaker:ListClusterEvents\",\n    \"sagemaker:ListClusters\",\n    \"eks:DescribeCluster\",\n    \"ec2:DescribeSecurityGroups\",\n    \"ec2:DescribeSubnets\",\n    \"ec2:DescribeVpcs\",\n    \"ec2:DescribeVpcAttribute\",\n    \"ec2:DescribeVpcEndpoints\",\n    \"ec2:DescribeRouteTables\",\n    \"ec2:DescribeNetworkInterfaces\",\n    \"ec2:DescribeInstances\",\n    \"ec2:DescribeInstanceTypeOfferings\",\n    \"ec2:DescribeInstanceTypes\",\n    \"logs:DescribeLogGroups\",\n    \"logs:DescribeLogStreams\",\n    \"logs:FilterLogEvents\",\n    \"ssm:StartSession\",\n    \"ssm:TerminateSession\",\n    \"service-quotas:GetServiceQuota\"\n  ]\n}\n","json",[2374],{"type":50,"tag":94,"props":2375,"children":2376},{"__ignoreMap":278},[2377,2385,2413,2435,2455,2476,2497,2518,2539,2560,2581,2602,2623,2644,2665,2686,2707,2728,2749,2770,2791,2812,2833,2854,2872,2881],{"type":50,"tag":284,"props":2378,"children":2379},{"class":286,"line":287},[2380],{"type":50,"tag":284,"props":2381,"children":2382},{"style":307},[2383],{"type":55,"value":2384},"{\n",{"type":50,"tag":284,"props":2386,"children":2387},{"class":286,"line":352},[2388,2393,2399,2404,2408],{"type":50,"tag":284,"props":2389,"children":2390},{"style":307},[2391],{"type":55,"value":2392},"  \"",{"type":50,"tag":284,"props":2394,"children":2396},{"style":2395},"--shiki-light:#9C3EDA;--shiki-default:#C792EA;--shiki-dark:#C792EA",[2397],{"type":55,"value":2398},"Action",{"type":50,"tag":284,"props":2400,"children":2401},{"style":307},[2402],{"type":55,"value":2403},"\"",{"type":50,"tag":284,"props":2405,"children":2406},{"style":307},[2407],{"type":55,"value":2367},{"type":50,"tag":284,"props":2409,"children":2410},{"style":307},[2411],{"type":55,"value":2412}," [\n",{"type":50,"tag":284,"props":2414,"children":2415},{"class":286,"line":362},[2416,2421,2426,2430],{"type":50,"tag":284,"props":2417,"children":2418},{"style":307},[2419],{"type":55,"value":2420},"    \"",{"type":50,"tag":284,"props":2422,"children":2423},{"style":296},[2424],{"type":55,"value":2425},"sagemaker:DescribeCluster",{"type":50,"tag":284,"props":2427,"children":2428},{"style":307},[2429],{"type":55,"value":2403},{"type":50,"tag":284,"props":2431,"children":2432},{"style":307},[2433],{"type":55,"value":2434},",\n",{"type":50,"tag":284,"props":2436,"children":2437},{"class":286,"line":372},[2438,2442,2447,2451],{"type":50,"tag":284,"props":2439,"children":2440},{"style":307},[2441],{"type":55,"value":2420},{"type":50,"tag":284,"props":2443,"children":2444},{"style":296},[2445],{"type":55,"value":2446},"sagemaker:DescribeClusterNode",{"type":50,"tag":284,"props":2448,"children":2449},{"style":307},[2450],{"type":55,"value":2403},{"type":50,"tag":284,"props":2452,"children":2453},{"style":307},[2454],{"type":55,"value":2434},{"type":50,"tag":284,"props":2456,"children":2458},{"class":286,"line":2457},5,[2459,2463,2468,2472],{"type":50,"tag":284,"props":2460,"children":2461},{"style":307},[2462],{"type":55,"value":2420},{"type":50,"tag":284,"props":2464,"children":2465},{"style":296},[2466],{"type":55,"value":2467},"sagemaker:ListClusterNodes",{"type":50,"tag":284,"props":2469,"children":2470},{"style":307},[2471],{"type":55,"value":2403},{"type":50,"tag":284,"props":2473,"children":2474},{"style":307},[2475],{"type":55,"value":2434},{"type":50,"tag":284,"props":2477,"children":2479},{"class":286,"line":2478},6,[2480,2484,2489,2493],{"type":50,"tag":284,"props":2481,"children":2482},{"style":307},[2483],{"type":55,"value":2420},{"type":50,"tag":284,"props":2485,"children":2486},{"style":296},[2487],{"type":55,"value":2488},"sagemaker:ListClusterEvents",{"type":50,"tag":284,"props":2490,"children":2491},{"style":307},[2492],{"type":55,"value":2403},{"type":50,"tag":284,"props":2494,"children":2495},{"style":307},[2496],{"type":55,"value":2434},{"type":50,"tag":284,"props":2498,"children":2500},{"class":286,"line":2499},7,[2501,2505,2510,2514],{"type":50,"tag":284,"props":2502,"children":2503},{"style":307},[2504],{"type":55,"value":2420},{"type":50,"tag":284,"props":2506,"children":2507},{"style":296},[2508],{"type":55,"value":2509},"sagemaker:ListClusters",{"type":50,"tag":284,"props":2511,"children":2512},{"style":307},[2513],{"type":55,"value":2403},{"type":50,"tag":284,"props":2515,"children":2516},{"style":307},[2517],{"type":55,"value":2434},{"type":50,"tag":284,"props":2519,"children":2521},{"class":286,"line":2520},8,[2522,2526,2531,2535],{"type":50,"tag":284,"props":2523,"children":2524},{"style":307},[2525],{"type":55,"value":2420},{"type":50,"tag":284,"props":2527,"children":2528},{"style":296},[2529],{"type":55,"value":2530},"eks:DescribeCluster",{"type":50,"tag":284,"props":2532,"children":2533},{"style":307},[2534],{"type":55,"value":2403},{"type":50,"tag":284,"props":2536,"children":2537},{"style":307},[2538],{"type":55,"value":2434},{"type":50,"tag":284,"props":2540,"children":2542},{"class":286,"line":2541},9,[2543,2547,2552,2556],{"type":50,"tag":284,"props":2544,"children":2545},{"style":307},[2546],{"type":55,"value":2420},{"type":50,"tag":284,"props":2548,"children":2549},{"style":296},[2550],{"type":55,"value":2551},"ec2:DescribeSecurityGroups",{"type":50,"tag":284,"props":2553,"children":2554},{"style":307},[2555],{"type":55,"value":2403},{"type":50,"tag":284,"props":2557,"children":2558},{"style":307},[2559],{"type":55,"value":2434},{"type":50,"tag":284,"props":2561,"children":2563},{"class":286,"line":2562},10,[2564,2568,2573,2577],{"type":50,"tag":284,"props":2565,"children":2566},{"style":307},[2567],{"type":55,"value":2420},{"type":50,"tag":284,"props":2569,"children":2570},{"style":296},[2571],{"type":55,"value":2572},"ec2:DescribeSubnets",{"type":50,"tag":284,"props":2574,"children":2575},{"style":307},[2576],{"type":55,"value":2403},{"type":50,"tag":284,"props":2578,"children":2579},{"style":307},[2580],{"type":55,"value":2434},{"type":50,"tag":284,"props":2582,"children":2584},{"class":286,"line":2583},11,[2585,2589,2594,2598],{"type":50,"tag":284,"props":2586,"children":2587},{"style":307},[2588],{"type":55,"value":2420},{"type":50,"tag":284,"props":2590,"children":2591},{"style":296},[2592],{"type":55,"value":2593},"ec2:DescribeVpcs",{"type":50,"tag":284,"props":2595,"children":2596},{"style":307},[2597],{"type":55,"value":2403},{"type":50,"tag":284,"props":2599,"children":2600},{"style":307},[2601],{"type":55,"value":2434},{"type":50,"tag":284,"props":2603,"children":2605},{"class":286,"line":2604},12,[2606,2610,2615,2619],{"type":50,"tag":284,"props":2607,"children":2608},{"style":307},[2609],{"type":55,"value":2420},{"type":50,"tag":284,"props":2611,"children":2612},{"style":296},[2613],{"type":55,"value":2614},"ec2:DescribeVpcAttribute",{"type":50,"tag":284,"props":2616,"children":2617},{"style":307},[2618],{"type":55,"value":2403},{"type":50,"tag":284,"props":2620,"children":2621},{"style":307},[2622],{"type":55,"value":2434},{"type":50,"tag":284,"props":2624,"children":2626},{"class":286,"line":2625},13,[2627,2631,2636,2640],{"type":50,"tag":284,"props":2628,"children":2629},{"style":307},[2630],{"type":55,"value":2420},{"type":50,"tag":284,"props":2632,"children":2633},{"style":296},[2634],{"type":55,"value":2635},"ec2:DescribeVpcEndpoints",{"type":50,"tag":284,"props":2637,"children":2638},{"style":307},[2639],{"type":55,"value":2403},{"type":50,"tag":284,"props":2641,"children":2642},{"style":307},[2643],{"type":55,"value":2434},{"type":50,"tag":284,"props":2645,"children":2647},{"class":286,"line":2646},14,[2648,2652,2657,2661],{"type":50,"tag":284,"props":2649,"children":2650},{"style":307},[2651],{"type":55,"value":2420},{"type":50,"tag":284,"props":2653,"children":2654},{"style":296},[2655],{"type":55,"value":2656},"ec2:DescribeRouteTables",{"type":50,"tag":284,"props":2658,"children":2659},{"style":307},[2660],{"type":55,"value":2403},{"type":50,"tag":284,"props":2662,"children":2663},{"style":307},[2664],{"type":55,"value":2434},{"type":50,"tag":284,"props":2666,"children":2668},{"class":286,"line":2667},15,[2669,2673,2678,2682],{"type":50,"tag":284,"props":2670,"children":2671},{"style":307},[2672],{"type":55,"value":2420},{"type":50,"tag":284,"props":2674,"children":2675},{"style":296},[2676],{"type":55,"value":2677},"ec2:DescribeNetworkInterfaces",{"type":50,"tag":284,"props":2679,"children":2680},{"style":307},[2681],{"type":55,"value":2403},{"type":50,"tag":284,"props":2683,"children":2684},{"style":307},[2685],{"type":55,"value":2434},{"type":50,"tag":284,"props":2687,"children":2689},{"class":286,"line":2688},16,[2690,2694,2699,2703],{"type":50,"tag":284,"props":2691,"children":2692},{"style":307},[2693],{"type":55,"value":2420},{"type":50,"tag":284,"props":2695,"children":2696},{"style":296},[2697],{"type":55,"value":2698},"ec2:DescribeInstances",{"type":50,"tag":284,"props":2700,"children":2701},{"style":307},[2702],{"type":55,"value":2403},{"type":50,"tag":284,"props":2704,"children":2705},{"style":307},[2706],{"type":55,"value":2434},{"type":50,"tag":284,"props":2708,"children":2710},{"class":286,"line":2709},17,[2711,2715,2720,2724],{"type":50,"tag":284,"props":2712,"children":2713},{"style":307},[2714],{"type":55,"value":2420},{"type":50,"tag":284,"props":2716,"children":2717},{"style":296},[2718],{"type":55,"value":2719},"ec2:DescribeInstanceTypeOfferings",{"type":50,"tag":284,"props":2721,"children":2722},{"style":307},[2723],{"type":55,"value":2403},{"type":50,"tag":284,"props":2725,"children":2726},{"style":307},[2727],{"type":55,"value":2434},{"type":50,"tag":284,"props":2729,"children":2731},{"class":286,"line":2730},18,[2732,2736,2741,2745],{"type":50,"tag":284,"props":2733,"children":2734},{"style":307},[2735],{"type":55,"value":2420},{"type":50,"tag":284,"props":2737,"children":2738},{"style":296},[2739],{"type":55,"value":2740},"ec2:DescribeInstanceTypes",{"type":50,"tag":284,"props":2742,"children":2743},{"style":307},[2744],{"type":55,"value":2403},{"type":50,"tag":284,"props":2746,"children":2747},{"style":307},[2748],{"type":55,"value":2434},{"type":50,"tag":284,"props":2750,"children":2752},{"class":286,"line":2751},19,[2753,2757,2762,2766],{"type":50,"tag":284,"props":2754,"children":2755},{"style":307},[2756],{"type":55,"value":2420},{"type":50,"tag":284,"props":2758,"children":2759},{"style":296},[2760],{"type":55,"value":2761},"logs:DescribeLogGroups",{"type":50,"tag":284,"props":2763,"children":2764},{"style":307},[2765],{"type":55,"value":2403},{"type":50,"tag":284,"props":2767,"children":2768},{"style":307},[2769],{"type":55,"value":2434},{"type":50,"tag":284,"props":2771,"children":2773},{"class":286,"line":2772},20,[2774,2778,2783,2787],{"type":50,"tag":284,"props":2775,"children":2776},{"style":307},[2777],{"type":55,"value":2420},{"type":50,"tag":284,"props":2779,"children":2780},{"style":296},[2781],{"type":55,"value":2782},"logs:DescribeLogStreams",{"type":50,"tag":284,"props":2784,"children":2785},{"style":307},[2786],{"type":55,"value":2403},{"type":50,"tag":284,"props":2788,"children":2789},{"style":307},[2790],{"type":55,"value":2434},{"type":50,"tag":284,"props":2792,"children":2794},{"class":286,"line":2793},21,[2795,2799,2804,2808],{"type":50,"tag":284,"props":2796,"children":2797},{"style":307},[2798],{"type":55,"value":2420},{"type":50,"tag":284,"props":2800,"children":2801},{"style":296},[2802],{"type":55,"value":2803},"logs:FilterLogEvents",{"type":50,"tag":284,"props":2805,"children":2806},{"style":307},[2807],{"type":55,"value":2403},{"type":50,"tag":284,"props":2809,"children":2810},{"style":307},[2811],{"type":55,"value":2434},{"type":50,"tag":284,"props":2813,"children":2815},{"class":286,"line":2814},22,[2816,2820,2825,2829],{"type":50,"tag":284,"props":2817,"children":2818},{"style":307},[2819],{"type":55,"value":2420},{"type":50,"tag":284,"props":2821,"children":2822},{"style":296},[2823],{"type":55,"value":2824},"ssm:StartSession",{"type":50,"tag":284,"props":2826,"children":2827},{"style":307},[2828],{"type":55,"value":2403},{"type":50,"tag":284,"props":2830,"children":2831},{"style":307},[2832],{"type":55,"value":2434},{"type":50,"tag":284,"props":2834,"children":2836},{"class":286,"line":2835},23,[2837,2841,2846,2850],{"type":50,"tag":284,"props":2838,"children":2839},{"style":307},[2840],{"type":55,"value":2420},{"type":50,"tag":284,"props":2842,"children":2843},{"style":296},[2844],{"type":55,"value":2845},"ssm:TerminateSession",{"type":50,"tag":284,"props":2847,"children":2848},{"style":307},[2849],{"type":55,"value":2403},{"type":50,"tag":284,"props":2851,"children":2852},{"style":307},[2853],{"type":55,"value":2434},{"type":50,"tag":284,"props":2855,"children":2857},{"class":286,"line":2856},24,[2858,2862,2867],{"type":50,"tag":284,"props":2859,"children":2860},{"style":307},[2861],{"type":55,"value":2420},{"type":50,"tag":284,"props":2863,"children":2864},{"style":296},[2865],{"type":55,"value":2866},"service-quotas:GetServiceQuota",{"type":50,"tag":284,"props":2868,"children":2869},{"style":307},[2870],{"type":55,"value":2871},"\"\n",{"type":50,"tag":284,"props":2873,"children":2875},{"class":286,"line":2874},25,[2876],{"type":50,"tag":284,"props":2877,"children":2878},{"style":307},[2879],{"type":55,"value":2880},"  ]\n",{"type":50,"tag":284,"props":2882,"children":2884},{"class":286,"line":2883},26,[2885],{"type":50,"tag":284,"props":2886,"children":2887},{"style":307},[2888],{"type":55,"value":2889},"}\n",{"type":50,"tag":58,"props":2891,"children":2892},{},[2893,2899,2901,2906,2908,2914,2916,2922,2924,2930,2931,2937,2938,2944,2945,2950,2951,2957,2958,2964,2965,2971,2972,2978],{"type":50,"tag":94,"props":2894,"children":2896},{"className":2895},[],[2897],{"type":55,"value":2898},"sts:GetCallerIdentity",{"type":55,"value":2900}," is implicit — it requires no IAM action. SSM on HyperPod uses ",{"type":50,"tag":94,"props":2902,"children":2904},{"className":2903},[],[2905],{"type":55,"value":2255},{"type":55,"value":2907}," against ",{"type":50,"tag":94,"props":2909,"children":2911},{"className":2910},[],[2912],{"type":55,"value":2913},"sagemaker-cluster:\u003Ccluster-id>_\u003Cgroup>-\u003Ciid>",{"type":55,"value":2915}," targets — not ",{"type":50,"tag":94,"props":2917,"children":2919},{"className":2918},[],[2920],{"type":55,"value":2921},"send-command",{"type":55,"value":2923}," against bare instance IDs. For remediation commands, grant the matching write permission (e.g. ",{"type":50,"tag":94,"props":2925,"children":2927},{"className":2926},[],[2928],{"type":55,"value":2929},"ec2:AuthorizeSecurityGroupIngress",{"type":55,"value":866},{"type":50,"tag":94,"props":2932,"children":2934},{"className":2933},[],[2935],{"type":55,"value":2936},"Egress",{"type":55,"value":101},{"type":50,"tag":94,"props":2939,"children":2941},{"className":2940},[],[2942],{"type":55,"value":2943},"ec2:RevokeSecurityGroupIngress",{"type":55,"value":866},{"type":50,"tag":94,"props":2946,"children":2948},{"className":2947},[],[2949],{"type":55,"value":2936},{"type":55,"value":101},{"type":50,"tag":94,"props":2952,"children":2954},{"className":2953},[],[2955],{"type":55,"value":2956},"ec2:ModifyVpcAttribute",{"type":55,"value":101},{"type":50,"tag":94,"props":2959,"children":2961},{"className":2960},[],[2962],{"type":55,"value":2963},"sagemaker:UpdateCluster",{"type":55,"value":101},{"type":50,"tag":94,"props":2966,"children":2968},{"className":2967},[],[2969],{"type":55,"value":2970},"sagemaker:BatchRebootClusterNodes",{"type":55,"value":101},{"type":50,"tag":94,"props":2973,"children":2975},{"className":2974},[],[2976],{"type":55,"value":2977},"sagemaker:BatchReplaceClusterNodes",{"type":55,"value":2979},"). Not needed for the diagnostic itself.",{"type":50,"tag":194,"props":2981,"children":2983},{"id":2982},"skill-delegation",[2984],{"type":55,"value":2985},"Skill delegation",{"type":50,"tag":538,"props":2987,"children":2988},{},[2989,3005],{"type":50,"tag":542,"props":2990,"children":2991},{},[2992],{"type":50,"tag":546,"props":2993,"children":2994},{},[2995,3000],{"type":50,"tag":550,"props":2996,"children":2997},{},[2998],{"type":55,"value":2999},"Need",{"type":50,"tag":550,"props":3001,"children":3002},{},[3003],{"type":55,"value":3004},"Use",{"type":50,"tag":561,"props":3006,"children":3007},{},[3008,3034,3052,3065,3083,3096,3113,3126,3142,3159,3175,3191],{"type":50,"tag":546,"props":3009,"children":3010},{},[3011,3016],{"type":50,"tag":568,"props":3012,"children":3013},{},[3014],{"type":55,"value":3015},"Cluster creation \u002F deployment failures",{"type":50,"tag":568,"props":3017,"children":3018},{},[3019,3025,3027,3033],{"type":50,"tag":94,"props":3020,"children":3022},{"className":3021},[],[3023],{"type":55,"value":3024},"hyperpod-cluster-debugger",{"type":55,"value":3026}," (§ A \u002F B \u002F C \u002F H + ",{"type":50,"tag":94,"props":3028,"children":3030},{"className":3029},[],[3031],{"type":55,"value":3032},"--validate",{"type":55,"value":1898},{"type":50,"tag":546,"props":3035,"children":3036},{},[3037,3042],{"type":50,"tag":568,"props":3038,"children":3039},{},[3040],{"type":55,"value":3041},"Cluster-wide SSM outage",{"type":50,"tag":568,"props":3043,"children":3044},{},[3045,3050],{"type":50,"tag":94,"props":3046,"children":3048},{"className":3047},[],[3049],{"type":55,"value":3024},{"type":55,"value":3051}," § F",{"type":50,"tag":546,"props":3053,"children":3054},{},[3055,3060],{"type":50,"tag":568,"props":3056,"children":3057},{},[3058],{"type":55,"value":3059},"Single-node SSM failure",{"type":50,"tag":568,"props":3061,"children":3062},{},[3063],{"type":55,"value":3064},"stay here — § K",{"type":50,"tag":546,"props":3066,"children":3067},{},[3068,3073],{"type":50,"tag":568,"props":3069,"children":3070},{},[3071],{"type":55,"value":3072},"Cluster-wide EFA health-check failure at creation time",{"type":50,"tag":568,"props":3074,"children":3075},{},[3076,3081],{"type":50,"tag":94,"props":3077,"children":3079},{"className":3078},[],[3080],{"type":55,"value":3024},{"type":55,"value":3082}," § A",{"type":50,"tag":546,"props":3084,"children":3085},{},[3086,3091],{"type":50,"tag":568,"props":3087,"children":3088},{},[3089],{"type":55,"value":3090},"Single-node EFA failure post-provisioning",{"type":50,"tag":568,"props":3092,"children":3093},{},[3094],{"type":55,"value":3095},"stay here — § A",{"type":50,"tag":546,"props":3097,"children":3098},{},[3099,3104],{"type":50,"tag":568,"props":3100,"children":3101},{},[3102],{"type":55,"value":3103},"NCCL AllReduce \u002F collective-op timeouts (distributed)",{"type":50,"tag":568,"props":3105,"children":3106},{},[3107],{"type":50,"tag":94,"props":3108,"children":3110},{"className":3109},[],[3111],{"type":55,"value":3112},"hyperpod-nccl",{"type":50,"tag":546,"props":3114,"children":3115},{},[3116,3121],{"type":50,"tag":568,"props":3117,"children":3118},{},[3119],{"type":55,"value":3120},"Silent GPU NaNs on a specific node (row-remap \u002F DCGM)",{"type":50,"tag":568,"props":3122,"children":3123},{},[3124],{"type":55,"value":3125},"stay here — § G.1 (even if discovered by NCCL)",{"type":50,"tag":546,"props":3127,"children":3128},{},[3129,3134],{"type":50,"tag":568,"props":3130,"children":3131},{},[3132],{"type":55,"value":3133},"Post-deployment cluster-wide management",{"type":50,"tag":568,"props":3135,"children":3136},{},[3137],{"type":50,"tag":94,"props":3138,"children":3140},{"className":3139},[],[3141],{"type":55,"value":3024},{"type":50,"tag":546,"props":3143,"children":3144},{},[3145,3150],{"type":50,"tag":568,"props":3146,"children":3147},{},[3148],{"type":55,"value":3149},"Shell \u002F commands on nodes",{"type":50,"tag":568,"props":3151,"children":3152},{},[3153],{"type":50,"tag":94,"props":3154,"children":3156},{"className":3155},[],[3157],{"type":55,"value":3158},"hyperpod-ssm",{"type":50,"tag":546,"props":3160,"children":3161},{},[3162,3167],{"type":50,"tag":568,"props":3163,"children":3164},{},[3165],{"type":55,"value":3166},"CUDA \u002F NCCL \u002F EFA version comparison",{"type":50,"tag":568,"props":3168,"children":3169},{},[3170],{"type":50,"tag":94,"props":3171,"children":3173},{"className":3172},[],[3174],{"type":55,"value":1343},{"type":50,"tag":546,"props":3176,"children":3177},{},[3178,3183],{"type":50,"tag":568,"props":3179,"children":3180},{},[3181],{"type":55,"value":3182},"Diagnostic bundle for AWS Support",{"type":50,"tag":568,"props":3184,"children":3185},{},[3186],{"type":50,"tag":94,"props":3187,"children":3189},{"className":3188},[],[3190],{"type":55,"value":1698},{"type":50,"tag":546,"props":3192,"children":3193},{},[3194,3199],{"type":50,"tag":568,"props":3195,"children":3196},{},[3197],{"type":55,"value":3198},"Training performance \u002F MFU degradation",{"type":50,"tag":568,"props":3200,"children":3201},{},[3202],{"type":50,"tag":94,"props":3203,"children":3205},{"className":3204},[],[3206],{"type":55,"value":3207},"hyperpod-mfu-debugger",{"type":50,"tag":194,"props":3209,"children":3211},{"id":3210},"escalate-to-aws-support",[3212],{"type":55,"value":3213},"Escalate to AWS Support",{"type":50,"tag":58,"props":3215,"children":3216},{},[3217],{"type":55,"value":3218},"Escalate when:",{"type":50,"tag":201,"props":3220,"children":3221},{},[3222,3227,3232,3237],{"type":50,"tag":205,"props":3223,"children":3224},{},[3225],{"type":55,"value":3226},"SG rules correct and reachability passes but EFA still fails.",{"type":50,"tag":205,"props":3228,"children":3229},{},[3230],{"type":55,"value":3231},"VPC correct but K8s bootstrap fails — check VPC flow logs for REJECT.",{"type":50,"tag":205,"props":3233,"children":3234},{},[3235],{"type":55,"value":3236},"Hardware failure where replacement keeps failing (bad physical host).",{"type":50,"tag":205,"props":3238,"children":3239},{},[3240],{"type":55,"value":3241},"Node replacement fails with an insufficient-capacity signal despite a valid ODCR.",{"type":50,"tag":3243,"props":3244,"children":3246},"h3",{"id":3245},"before-opening-the-case",[3247],{"type":55,"value":3248},"Before opening the case",{"type":50,"tag":273,"props":3250,"children":3252},{"className":275,"code":3251,"language":277,"meta":278,"style":278},"# 1. Cluster identity + affected node status\naws sagemaker describe-cluster --cluster-name \u003CCLUSTER> --region \u003CREGION>\naws sagemaker list-cluster-nodes --cluster-name \u003CCLUSTER> --region \u003CREGION> \\\n  --query \"ClusterNodeSummaries[?InstanceId=='\u003CINSTANCE_ID>']\"\n\n# 2. Triage bundle (scoped to the affected node where possible)\nbash scripts\u002Ftriage-cluster.sh --cluster \u003CCLUSTER> --region \u003CREGION> --node \u003CINSTANCE_ID> > triage.txt\n\n# 3. Per-node log\u002Fconfig bundle to S3 (delegates to hyperpod-issue-report)\n#    See skills\u002Fhyperpod-issue-report\u002FSKILL.md for the exact invocation.\n",[3253],{"type":50,"tag":94,"props":3254,"children":3255},{"__ignoreMap":278},[3256,3264,3324,3385,3407,3414,3422,3503,3510,3518],{"type":50,"tag":284,"props":3257,"children":3258},{"class":286,"line":287},[3259],{"type":50,"tag":284,"props":3260,"children":3261},{"style":366},[3262],{"type":55,"value":3263},"# 1. Cluster identity + affected node status\n",{"type":50,"tag":284,"props":3265,"children":3266},{"class":286,"line":352},[3267,3271,3276,3281,3286,3290,3295,3300,3304,3308,3312,3316,3320],{"type":50,"tag":284,"props":3268,"children":3269},{"style":291},[3270],{"type":55,"value":22},{"type":50,"tag":284,"props":3272,"children":3273},{"style":296},[3274],{"type":55,"value":3275}," sagemaker",{"type":50,"tag":284,"props":3277,"children":3278},{"style":296},[3279],{"type":55,"value":3280}," describe-cluster",{"type":50,"tag":284,"props":3282,"children":3283},{"style":296},[3284],{"type":55,"value":3285}," --cluster-name",{"type":50,"tag":284,"props":3287,"children":3288},{"style":307},[3289],{"type":55,"value":310},{"type":50,"tag":284,"props":3291,"children":3292},{"style":296},[3293],{"type":55,"value":3294},"CLUSTE",{"type":50,"tag":284,"props":3296,"children":3297},{"style":318},[3298],{"type":55,"value":3299},"R",{"type":50,"tag":284,"props":3301,"children":3302},{"style":307},[3303],{"type":55,"value":326},{"type":50,"tag":284,"props":3305,"children":3306},{"style":296},[3307],{"type":55,"value":331},{"type":50,"tag":284,"props":3309,"children":3310},{"style":307},[3311],{"type":55,"value":310},{"type":50,"tag":284,"props":3313,"children":3314},{"style":296},[3315],{"type":55,"value":340},{"type":50,"tag":284,"props":3317,"children":3318},{"style":318},[3319],{"type":55,"value":321},{"type":50,"tag":284,"props":3321,"children":3322},{"style":307},[3323],{"type":55,"value":349},{"type":50,"tag":284,"props":3325,"children":3326},{"class":286,"line":362},[3327,3331,3335,3340,3344,3348,3352,3356,3360,3364,3368,3372,3376,3380],{"type":50,"tag":284,"props":3328,"children":3329},{"style":291},[3330],{"type":55,"value":22},{"type":50,"tag":284,"props":3332,"children":3333},{"style":296},[3334],{"type":55,"value":3275},{"type":50,"tag":284,"props":3336,"children":3337},{"style":296},[3338],{"type":55,"value":3339}," list-cluster-nodes",{"type":50,"tag":284,"props":3341,"children":3342},{"style":296},[3343],{"type":55,"value":3285},{"type":50,"tag":284,"props":3345,"children":3346},{"style":307},[3347],{"type":55,"value":310},{"type":50,"tag":284,"props":3349,"children":3350},{"style":296},[3351],{"type":55,"value":3294},{"type":50,"tag":284,"props":3353,"children":3354},{"style":318},[3355],{"type":55,"value":3299},{"type":50,"tag":284,"props":3357,"children":3358},{"style":307},[3359],{"type":55,"value":326},{"type":50,"tag":284,"props":3361,"children":3362},{"style":296},[3363],{"type":55,"value":331},{"type":50,"tag":284,"props":3365,"children":3366},{"style":307},[3367],{"type":55,"value":310},{"type":50,"tag":284,"props":3369,"children":3370},{"style":296},[3371],{"type":55,"value":340},{"type":50,"tag":284,"props":3373,"children":3374},{"style":318},[3375],{"type":55,"value":321},{"type":50,"tag":284,"props":3377,"children":3378},{"style":307},[3379],{"type":55,"value":326},{"type":50,"tag":284,"props":3381,"children":3382},{"style":318},[3383],{"type":55,"value":3384}," \\\n",{"type":50,"tag":284,"props":3386,"children":3387},{"class":286,"line":372},[3388,3393,3398,3403],{"type":50,"tag":284,"props":3389,"children":3390},{"style":296},[3391],{"type":55,"value":3392},"  --query",{"type":50,"tag":284,"props":3394,"children":3395},{"style":307},[3396],{"type":55,"value":3397}," \"",{"type":50,"tag":284,"props":3399,"children":3400},{"style":296},[3401],{"type":55,"value":3402},"ClusterNodeSummaries[?InstanceId=='\u003CINSTANCE_ID>']",{"type":50,"tag":284,"props":3404,"children":3405},{"style":307},[3406],{"type":55,"value":2871},{"type":50,"tag":284,"props":3408,"children":3409},{"class":286,"line":2457},[3410],{"type":50,"tag":284,"props":3411,"children":3412},{"emptyLinePlaceholder":356},[3413],{"type":55,"value":359},{"type":50,"tag":284,"props":3415,"children":3416},{"class":286,"line":2478},[3417],{"type":50,"tag":284,"props":3418,"children":3419},{"style":366},[3420],{"type":55,"value":3421},"# 2. Triage bundle (scoped to the affected node where possible)\n",{"type":50,"tag":284,"props":3423,"children":3424},{"class":286,"line":2499},[3425,3429,3433,3437,3441,3445,3449,3453,3457,3461,3465,3469,3473,3477,3481,3485,3489,3493,3498],{"type":50,"tag":284,"props":3426,"children":3427},{"style":291},[3428],{"type":55,"value":277},{"type":50,"tag":284,"props":3430,"children":3431},{"style":296},[3432],{"type":55,"value":299},{"type":50,"tag":284,"props":3434,"children":3435},{"style":296},[3436],{"type":55,"value":304},{"type":50,"tag":284,"props":3438,"children":3439},{"style":307},[3440],{"type":55,"value":310},{"type":50,"tag":284,"props":3442,"children":3443},{"style":296},[3444],{"type":55,"value":3294},{"type":50,"tag":284,"props":3446,"children":3447},{"style":318},[3448],{"type":55,"value":3299},{"type":50,"tag":284,"props":3450,"children":3451},{"style":307},[3452],{"type":55,"value":326},{"type":50,"tag":284,"props":3454,"children":3455},{"style":296},[3456],{"type":55,"value":331},{"type":50,"tag":284,"props":3458,"children":3459},{"style":307},[3460],{"type":55,"value":310},{"type":50,"tag":284,"props":3462,"children":3463},{"style":296},[3464],{"type":55,"value":340},{"type":50,"tag":284,"props":3466,"children":3467},{"style":318},[3468],{"type":55,"value":321},{"type":50,"tag":284,"props":3470,"children":3471},{"style":307},[3472],{"type":55,"value":326},{"type":50,"tag":284,"props":3474,"children":3475},{"style":296},[3476],{"type":55,"value":426},{"type":50,"tag":284,"props":3478,"children":3479},{"style":307},[3480],{"type":55,"value":310},{"type":50,"tag":284,"props":3482,"children":3483},{"style":296},[3484],{"type":55,"value":435},{"type":50,"tag":284,"props":3486,"children":3487},{"style":318},[3488],{"type":55,"value":440},{"type":50,"tag":284,"props":3490,"children":3491},{"style":307},[3492],{"type":55,"value":326},{"type":50,"tag":284,"props":3494,"children":3495},{"style":307},[3496],{"type":55,"value":3497}," >",{"type":50,"tag":284,"props":3499,"children":3500},{"style":296},[3501],{"type":55,"value":3502}," triage.txt\n",{"type":50,"tag":284,"props":3504,"children":3505},{"class":286,"line":2520},[3506],{"type":50,"tag":284,"props":3507,"children":3508},{"emptyLinePlaceholder":356},[3509],{"type":55,"value":359},{"type":50,"tag":284,"props":3511,"children":3512},{"class":286,"line":2541},[3513],{"type":50,"tag":284,"props":3514,"children":3515},{"style":366},[3516],{"type":55,"value":3517},"# 3. Per-node log\u002Fconfig bundle to S3 (delegates to hyperpod-issue-report)\n",{"type":50,"tag":284,"props":3519,"children":3520},{"class":286,"line":2562},[3521],{"type":50,"tag":284,"props":3522,"children":3523},{"style":366},[3524],{"type":55,"value":3525},"#    See skills\u002Fhyperpod-issue-report\u002FSKILL.md for the exact invocation.\n",{"type":50,"tag":3243,"props":3527,"children":3529},{"id":3528},"include-in-the-case",[3530],{"type":55,"value":3531},"Include in the case",{"type":50,"tag":1857,"props":3533,"children":3534},{},[3535,3540,3545,3550,3555,3560,3565,3576],{"type":50,"tag":205,"props":3536,"children":3537},{},[3538],{"type":55,"value":3539},"Cluster name + ARN and AWS region",{"type":50,"tag":205,"props":3541,"children":3542},{},[3543],{"type":55,"value":3544},"Orchestrator (EKS or Slurm)",{"type":50,"tag":205,"props":3546,"children":3547},{},[3548],{"type":55,"value":3549},"Affected instance IDs \u002F node names \u002F instance-group names",{"type":50,"tag":205,"props":3551,"children":3552},{},[3553],{"type":55,"value":3554},"Timestamp window (UTC start \u002F end) of the failure",{"type":50,"tag":205,"props":3556,"children":3557},{},[3558],{"type":55,"value":3559},"Exact error strings observed (copy verbatim from pod logs, CloudWatch, dmesg, events)",{"type":50,"tag":205,"props":3561,"children":3562},{},[3563],{"type":55,"value":3564},"XID numbers \u002F ECC counts \u002F DCGM output where hardware is implicated",{"type":50,"tag":205,"props":3566,"children":3567},{},[3568,3574],{"type":50,"tag":94,"props":3569,"children":3571},{"className":3570},[],[3572],{"type":55,"value":3573},"triage.txt",{"type":55,"value":3575}," from step 2 above",{"type":50,"tag":205,"props":3577,"children":3578},{},[3579,3581,3586],{"type":55,"value":3580},"S3 URI of the ",{"type":50,"tag":94,"props":3582,"children":3584},{"className":3583},[],[3585],{"type":55,"value":1698},{"type":55,"value":3587}," bundle from step 3",{"type":50,"tag":58,"props":3589,"children":3590},{},[3591,3593,3597],{"type":55,"value":3592},"Patterns from real customer tickets: ",{"type":50,"tag":182,"props":3594,"children":3595},{"href":184},[3596],{"type":55,"value":1460},{"type":55,"value":188},{"type":50,"tag":3599,"props":3600,"children":3601},"style",{},[3602],{"type":55,"value":3603},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":3605,"total":3708},[3606,3623,3643,3655,3672,3685,3695],{"slug":3607,"name":3607,"fn":3608,"description":3609,"org":3610,"tags":3611,"stars":26,"repoUrl":27,"updatedAt":3622},"amazon-location-service","integrate Amazon Location Service maps","Integrates Amazon Location Service APIs for AWS applications. Use this skill when users want to add maps (interactive MapLibre or static images); geocode addresses to coordinates or reverse geocode coordinates to addresses; calculate routes, travel times, or service areas; find places and businesses through text search, nearby search, or autocomplete suggestions; retrieve detailed place information including hours, contacts, and addresses; monitor geographical boundaries with geofences; or track device locations. Covers authentication, SDK integration, and all Amazon Location Service capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3612,3615,3616,3619],{"name":3613,"slug":3614,"type":16},"API Development","api-development",{"name":21,"slug":22,"type":16},{"name":3617,"slug":3618,"type":16},"Maps","maps",{"name":3620,"slug":3621,"type":16},"Navigation","navigation","2026-07-12T08:39:49.88311",{"slug":3624,"name":3624,"fn":3625,"description":3626,"org":3627,"tags":3628,"stars":26,"repoUrl":27,"updatedAt":3642},"amplify-workflow","build and deploy apps with AWS Amplify","Build and deploy full-stack web and mobile apps with AWS Amplify Gen2 (TypeScript code-first). Covers auth (Cognito), data (AppSync\u002FDynamoDB including schema modeling, enum types, relationships, authorization rules), storage (S3), functions, APIs, and AI (Amplify AI Kit with Bedrock). Supports React, Next.js, Vue, Angular, React Native, Flutter, Swift, and Android. Always use this skill for Amplify Gen2 topics — even for questions you think you know — it contains validated, version-specific patterns that prevent common mistakes. TRIGGER when: user mentions Amplify Gen2; project has amplify\u002F directory or amplify_outputs; code imports @aws-amplify packages; user asks about defineBackend, defineAuth, defineData, defineStorage, or npx ampx. SKIP: Amplify Gen1 (amplify CLI v6), standalone SAM\u002FCDK without Amplify (use aws-serverless), direct Bedrock without Amplify AI Kit (use bedrock).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3629,3632,3633,3636,3639],{"name":3630,"slug":3631,"type":16},"Auth","auth",{"name":21,"slug":22,"type":16},{"name":3634,"slug":3635,"type":16},"Database","database",{"name":3637,"slug":3638,"type":16},"Deployment","deployment",{"name":3640,"slug":3641,"type":16},"TypeScript","typescript","2026-07-12T08:39:43.500162",{"slug":3644,"name":3644,"fn":3645,"description":3646,"org":3647,"tags":3648,"stars":26,"repoUrl":27,"updatedAt":3654},"api-gateway","build and manage Amazon API Gateway APIs","Build, manage, and operate APIs with Amazon API Gateway (REST, HTTP, and WebSocket). Triggers on phrases like: API Gateway, REST API, HTTP API, WebSocket API, custom domain, Lambda authorizer, usage plan, throttling, CORS, VPC link, private API. Also covers troubleshooting API Gateway errors (4xx, 5xx, timeout, CORS failures) and IaC templates containing API Gateway resources. For general REST API design unrelated to AWS, do not trigger.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3649,3650,3651],{"name":3613,"slug":3614,"type":16},{"name":21,"slug":22,"type":16},{"name":3652,"slug":3653,"type":16},"REST API","rest-api","2026-07-12T08:39:00.149339",{"slug":3656,"name":3656,"fn":3657,"description":3658,"org":3659,"tags":3660,"stars":26,"repoUrl":27,"updatedAt":3671},"aws-architecture-diagram","generate AWS architecture diagrams","Generate validated AWS architecture diagrams as draw.io XML using official AWS4 icon libraries. Use this skill whenever the user wants to create, generate, or design AWS architecture diagrams, cloud infrastructure diagrams, or system design visuals. Also triggers for requests to visualize existing infrastructure from CloudFormation, CDK, or Terraform code. Supports two modes: analyze an existing codebase to auto-generate diagrams, or brainstorm interactively from scratch. Exports .drawio files with optional PNG\u002FSVG\u002FPDF export via draw.io desktop CLI.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3661,3664,3665,3668],{"name":3662,"slug":3663,"type":16},"Architecture","architecture",{"name":21,"slug":22,"type":16},{"name":3666,"slug":3667,"type":16},"Design","design",{"name":3669,"slug":3670,"type":16},"Diagrams","diagrams","2026-07-12T08:37:11.012278",{"slug":3673,"name":3673,"fn":3674,"description":3675,"org":3676,"tags":3677,"stars":26,"repoUrl":27,"updatedAt":3684},"aws-lambda","build and deploy AWS Lambda functions","Design, build, deploy, test, and debug serverless applications with AWS Lambda. Triggers on phrases like: Lambda function, event source, serverless application, API Gateway, EventBridge, Step Functions, serverless API, event-driven architecture, Lambda trigger. For deploying non-serverless apps to AWS, use deploy-on-aws plugin instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3678,3679,3680,3681],{"name":3613,"slug":3614,"type":16},{"name":21,"slug":22,"type":16},{"name":3637,"slug":3638,"type":16},{"name":3682,"slug":3683,"type":16},"Serverless","serverless","2026-07-12T08:38:58.598492",{"slug":3686,"name":3686,"fn":3687,"description":3688,"org":3689,"tags":3690,"stars":26,"repoUrl":27,"updatedAt":3694},"aws-lambda-durable-functions","build resilient AWS Lambda durable functions","Build resilient, long-running, multi-step applications with AWS Lambda durable functions with automatic state persistence, retry logic, and orchestration for long-running executions. Covers the critical replay model, step operations, wait\u002Fcallback patterns, error handling with saga pattern, testing with LocalDurableTestRunner. Triggers on phrases like: lambda durable functions, workflow orchestration, state machines, retry\u002Fcheckpoint patterns, long-running stateful Lambda functions, saga pattern, human-in-the-loop callbacks, and reliable serverless applications.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3691,3692,3693],{"name":3662,"slug":3663,"type":16},{"name":21,"slug":22,"type":16},{"name":3682,"slug":3683,"type":16},"2026-07-12T08:39:05.546173",{"slug":3696,"name":3696,"fn":3697,"description":3698,"org":3699,"tags":3700,"stars":26,"repoUrl":27,"updatedAt":3707},"aws-lambda-managed-instances","configure AWS Lambda Managed Instances","Evaluate, configure, and migrate workloads to AWS Lambda Managed Instances (LMI). Triggers on: Lambda Managed Instances, LMI, capacity provider, multi-concurrency Lambda, dedicated instance Lambda, EC2-backed Lambda, cold start elimination, Graviton Lambda, instance type for Lambda, scheduled scaling for LMI, Lambda cost optimization with Reserved Instances or Savings Plans. Also trigger when users describe high-volume predictable workloads seeking cost savings, want to scale LMI capacity on a schedule, or compare Lambda vs EC2 for steady-state traffic. For standard Lambda without LMI, use the aws-lambda skill instead.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3701,3702,3703,3704],{"name":21,"slug":22,"type":16},{"name":3637,"slug":3638,"type":16},{"name":18,"slug":19,"type":16},{"name":3705,"slug":3706,"type":16},"Performance","performance","2026-07-12T08:39:07.007071",33,{"items":3710,"total":3880},[3711,3726,3743,3753,3766,3779,3789,3799,3820,3835,3850,3865],{"slug":3712,"name":3712,"fn":3713,"description":3714,"org":3715,"tags":3716,"stars":3723,"repoUrl":3724,"updatedAt":3725},"agentcore-investigation","investigate Bedrock AgentCore runtime sessions","Investigate Bedrock AgentCore runtime sessions via CloudWatch Logs Insights — resolve session\u002Ftrace IDs, query OTEL spans, filter noise, build timelines. Use when debugging AgentCore agent sessions, tracing tool calls, or analyzing latency.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3717,3718,3719,3722],{"name":21,"slug":22,"type":16},{"name":24,"slug":25,"type":16},{"name":3720,"slug":3721,"type":16},"Logs","logs",{"name":14,"slug":15,"type":16},9427,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fmcp","2026-07-12T08:37:22.601527",{"slug":3727,"name":3728,"fn":3729,"description":3730,"org":3731,"tags":3732,"stars":3723,"repoUrl":3724,"updatedAt":3742},"amazon-aurora-dsql","amazon aurora dsql","build applications with Aurora DSQL","Build with Aurora DSQL — manage schemas, execute queries, handle migrations, diagnose query plans, load data, and develop applications with a serverless, distributed SQL database. Covers IAM auth, multi-tenant patterns, MySQL-to-DSQL and PostgreSQL-to-DSQL schema conversion, FK replacement code generation, OCC retry patterns, ORM migration (Django\u002FHibernate\u002FRails), DDL operations, query plan explainability, SQL compatibility validation, and bulk data loading. Triggers on phrases like: DSQL, Aurora DSQL, create DSQL table, DSQL schema, migrate to DSQL, distributed SQL database, serverless PostgreSQL-compatible database, DSQL query plan, DSQL EXPLAIN ANALYZE, why is my DSQL query slow, DSQL foreign key, DSQL OCC retry, DSQL multi-region, load into DSQL, load CSV into DSQL, bulk load DSQL, aurora-dsql-loader.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3733,3736,3737,3738,3739],{"name":3734,"slug":3735,"type":16},"Aurora","aurora",{"name":21,"slug":22,"type":16},{"name":3634,"slug":3635,"type":16},{"name":3682,"slug":3683,"type":16},{"name":3740,"slug":3741,"type":16},"SQL","sql","2026-07-12T08:36:45.053393",{"slug":3744,"name":3745,"fn":3729,"description":3730,"org":3746,"tags":3747,"stars":3723,"repoUrl":3724,"updatedAt":3752},"aurora-dsql","aurora dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3748,3749,3750,3751],{"name":21,"slug":22,"type":16},{"name":3634,"slug":3635,"type":16},{"name":3682,"slug":3683,"type":16},{"name":3740,"slug":3741,"type":16},"2026-07-12T08:36:42.694299",{"slug":3754,"name":3755,"fn":3729,"description":3730,"org":3756,"tags":3757,"stars":3723,"repoUrl":3724,"updatedAt":3765},"aws-dsql","aws dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3758,3759,3760,3763,3764],{"name":21,"slug":22,"type":16},{"name":3634,"slug":3635,"type":16},{"name":3761,"slug":3762,"type":16},"Migration","migration",{"name":3682,"slug":3683,"type":16},{"name":3740,"slug":3741,"type":16},"2026-07-12T08:36:38.584057",{"slug":3767,"name":3768,"fn":3729,"description":3730,"org":3769,"tags":3770,"stars":3723,"repoUrl":3724,"updatedAt":3778},"distributed-postgres","distributed postgres",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3771,3772,3773,3776,3777],{"name":21,"slug":22,"type":16},{"name":3634,"slug":3635,"type":16},{"name":3774,"slug":3775,"type":16},"PostgreSQL","postgresql",{"name":3682,"slug":3683,"type":16},{"name":3740,"slug":3741,"type":16},"2026-07-12T08:36:46.530743",{"slug":3780,"name":3781,"fn":3729,"description":3730,"org":3782,"tags":3783,"stars":3723,"repoUrl":3724,"updatedAt":3788},"distributed-sql","distributed sql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3784,3785,3786,3787],{"name":21,"slug":22,"type":16},{"name":3634,"slug":3635,"type":16},{"name":3682,"slug":3683,"type":16},{"name":3740,"slug":3741,"type":16},"2026-07-12T08:36:48.104182",{"slug":3790,"name":3790,"fn":3729,"description":3730,"org":3791,"tags":3792,"stars":3723,"repoUrl":3724,"updatedAt":3798},"dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3793,3794,3795,3796,3797],{"name":21,"slug":22,"type":16},{"name":3634,"slug":3635,"type":16},{"name":3761,"slug":3762,"type":16},{"name":3682,"slug":3683,"type":16},{"name":3740,"slug":3741,"type":16},"2026-07-12T08:36:36.374512",{"slug":3800,"name":3800,"fn":3801,"description":3802,"org":3803,"tags":3804,"stars":3817,"repoUrl":3818,"updatedAt":3819},"cost-efficiency-analyzer","analyze cost efficiency and expenses","Analyzes cost structure, cost efficiency, and expense management from P&L data. Use when the user asks about costs, expenses, COGS, operating expenses, cost ratios, cost control, spending efficiency, margin compression from cost side, or wants to understand where money is going. Also use for \"are we spending too much\", \"cost breakdown\", \"expense analysis\", or \"how efficient are our operations\". NOT for revenue or top-line analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3805,3808,3811,3814],{"name":3806,"slug":3807,"type":16},"Accounting","accounting",{"name":3809,"slug":3810,"type":16},"Analytics","analytics",{"name":3812,"slug":3813,"type":16},"Cost Optimization","cost-optimization",{"name":3815,"slug":3816,"type":16},"Finance","finance",3176,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagentcore-samples","2026-07-12T08:40:03.29555",{"slug":3821,"name":3821,"fn":3822,"description":3823,"org":3824,"tags":3825,"stars":3817,"repoUrl":3818,"updatedAt":3834},"executive-financial-briefing","generate executive financial briefings","Generates a concise executive-level financial briefing or summary suitable for a CEO, CFO, or board presentation. Use when the user asks for a summary, briefing, executive summary, board update, financial overview, financial health check, or \"how is the business doing\". Covers the full P&L picture in one page. Also use for \"give me the highlights\", \"what do I need to know\", or \"quick financial update\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3826,3827,3828,3831],{"name":21,"slug":22,"type":16},{"name":3815,"slug":3816,"type":16},{"name":3829,"slug":3830,"type":16},"Management","management",{"name":3832,"slug":3833,"type":16},"Reporting","reporting","2026-07-12T08:40:02.066471",{"slug":3836,"name":3836,"fn":3837,"description":3838,"org":3839,"tags":3840,"stars":3817,"repoUrl":3818,"updatedAt":3849},"multi-quarter-trend-analysis","analyze multi-quarter financial trends","Analyzes financial trends across multiple quarters by comparing P&L metrics over time. Use when the user wants to see trends, patterns, trajectories, or directional movement across 3 or more quarters. Also use for \"how are we trending\", \"show me the trend\", \"track performance over time\", \"quarter over quarter comparison across all quarters\", or any multi-period longitudinal analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3841,3842,3843,3846],{"name":3809,"slug":3810,"type":16},{"name":3815,"slug":3816,"type":16},{"name":3844,"slug":3845,"type":16},"Financial Statements","financial-statements",{"name":3847,"slug":3848,"type":16},"Variance Analysis","variance-analysis","2026-07-12T08:40:00.79141",{"slug":3851,"name":3851,"fn":3852,"description":3853,"org":3854,"tags":3855,"stars":3817,"repoUrl":3818,"updatedAt":3864},"pdf","process and manipulate PDF documents","Use this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text\u002Ftables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting\u002Fdecrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3856,3859,3862],{"name":3857,"slug":3858,"type":16},"Automation","automation",{"name":3860,"slug":3861,"type":16},"Documents","documents",{"name":3863,"slug":3851,"type":16},"PDF","2026-07-12T08:41:44.135656",{"slug":3866,"name":3866,"fn":3867,"description":3868,"org":3869,"tags":3870,"stars":3817,"repoUrl":3818,"updatedAt":3879},"quarterly-kpi-calculator","calculate quarterly financial KPIs","Calculates quarterly financial KPIs from P&L data. P&L figures can be provided directly by the user or fetched from the financial data MCP server. Use when the user wants KPI calculations such as Gross Margin %, EBITDA Margin %, Operating Expense Ratio, or Revenue Growth % QoQ. Also use for quarterly performance review, P&L analysis, or interpreting financial ratios against benchmarks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[3871,3872,3875,3876],{"name":3806,"slug":3807,"type":16},{"name":3873,"slug":3874,"type":16},"Data Analysis","data-analysis",{"name":3815,"slug":3816,"type":16},{"name":3877,"slug":3878,"type":16},"KPI","kpi","2026-07-12T08:39:59.54971",150]