[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-aws-labs-hyperpod-slurm-debugger":3,"mdc--yorp2v-key":42,"related-org-aws-labs-hyperpod-slurm-debugger":2308,"related-repo-aws-labs-hyperpod-slurm-debugger":2486},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":37,"sourceUrl":40,"mdContent":41},"hyperpod-slurm-debugger","debug Slurm scheduler issues on HyperPod clusters","Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker HyperPod Slurm clusters. Scope mirrors the HyperPod troubleshooting guide. Invoke when the user reports a Slurm node stuck in down\u002Fdrain, \"Node unexpectedly rebooted\" after auto-repair, slurmd not running, jobs stuck PENDING with REASON=Resources while sinfo shows idle nodes, jobs stuck COMPLETING after node replacement, GRES\u002FGPU counts wrong, scontrol ping failing, slurmctld unresponsive, an Action:Reboot\u002FReplace request that did not trigger HyperPod auto-recovery, or auto-resume not restarting a job. Also triggers on \"drain before reboot\", \"diagnose a Slurm node\", \"investigate stuck jobs.\"",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"aws-labs","AWS Labs","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Faws-labs.png","awslabs",[13,17,20,23],{"name":14,"slug":15,"type":16},"SRE","sre","tag",{"name":18,"slug":19,"type":16},"Infrastructure","infrastructure",{"name":21,"slug":22,"type":16},"AWS","aws",{"name":24,"slug":25,"type":16},"Debugging","debugging",831,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagent-plugins","2026-07-12T08:39:30.666952",null,127,[32,33,34,22,35,36],"agent-plugins","agent-skills","agents","coding-agent-skills","coding-agents",{"repoUrl":27,"stars":26,"forks":30,"topics":38,"description":39},[32,33,34,22,35,36],"Agent Plugins for AWS equip AI coding agents with the skills to help you architect, deploy, and operate on AWS.","https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagent-plugins\u002Ftree\u002FHEAD\u002Fplugins\u002Fsagemaker-ai\u002Fskills\u002Fhyperpod-slurm-debugger","---\nname: hyperpod-slurm-debugger\ndescription: Diagnostic-only skill for Slurm scheduler and node-daemon issues on Amazon SageMaker HyperPod Slurm clusters. Scope mirrors the HyperPod troubleshooting guide. Invoke when the user reports a Slurm node stuck in down\u002Fdrain, \"Node unexpectedly rebooted\" after auto-repair, slurmd not running, jobs stuck PENDING with REASON=Resources while sinfo shows idle nodes, jobs stuck COMPLETING after node replacement, GRES\u002FGPU counts wrong, scontrol ping failing, slurmctld unresponsive, an Action:Reboot\u002FReplace request that did not trigger HyperPod auto-recovery, or auto-resume not restarting a job. Also triggers on \"drain before reboot\", \"diagnose a Slurm node\", \"investigate stuck jobs.\"\nmetadata:\n  version: \"0.0.1\"\n---\n\n# HyperPod Slurm Debugger\n\nDiagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on\nHyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command.\nFor remediation, link to the official AWS or Slurm documentation.\n\n## When to invoke\n\nInvoke when the user reports any of the symptoms in the [decision table](#decision-table).\n\n## When NOT to invoke\n\n- Cluster has `Orchestrator.Eks` — invoke `hyperpod-node-debugger` or `hyperpod-nccl`.\n- Single-node hardware fault with healthy Slurm scheduler — invoke `hyperpod-node-debugger`.\n- NCCL training-hang investigation — invoke `hyperpod-nccl`.\n- Node unreachable via SSM — invoke `hyperpod-ssm`.\n\n## Constraints\n\n- Read-only. Do not run, recommend, or print state-mutating commands.\n- For any remediation, link to AWS or Slurm docs. The user authorizes and executes.\n- IaC-managed cluster (Terraform \u002F CloudFormation \u002F CDK): warn that direct mutation\n  drifts the live state from the IaC plan.\n\nCanonical recovery URLs:\n[references\u002Fslurm-details.md → Authoritative recovery documentation](references\u002Fslurm-details.md).\n\n## Prerequisites\n\n- AWS CLI v2, authenticated for the target account and region with permissions:\n  - `sagemaker:DescribeCluster`, `sagemaker:ListClusterNodes`\n  - `ssm:StartSession` on the HyperPod-created SSM document\n- [Session Manager plugin](https:\u002F\u002Fdocs.aws.amazon.com\u002Fsystems-manager\u002Flatest\u002Fuserguide\u002Fsession-manager-working-with-install-plugin.html)\n  installed locally.\n- `jq` ≥ 1.6.\n- `unbuffer` (from the `expect` package). Required — without it `aws ssm start-session`\n  returns empty stdout intermittently with `Cannot perform start session: EOF` and every\n  check silently misreports. Install: `expect` package on Amazon Linux \u002F RHEL \u002F Debian \u002F\n  Ubuntu \u002F macOS. Script exits at prerequisite check if missing.\n\n## Procedure\n\n### Step 1 — Collect inputs\n\nAsk the user for:\n\n1. HyperPod cluster name (not Slurm partition name).\n2. AWS region.\n3. Optional: a specific Slurm node name.\n\n### Step 2 — Confirm orchestrator\n\n```bash\naws sagemaker describe-cluster --cluster-name \u003CNAME\u002FARN> --region \u003CREGION> \\\n  --query 'Orchestrator' --output json\n```\n\nIf `Orchestrator.Eks` is present, stop. Route per [When NOT to invoke](#when-not-to-invoke).\n\n### Step 3 — Run the diagnostic script\n\n```bash\nbash scripts\u002Fslurm-diagnose.sh --cluster \u003CNAME> --region \u003CREGION>\n# Scope to a node:\nbash scripts\u002Fslurm-diagnose.sh --cluster \u003CNAME> --region \u003CREGION> --node \u003CSLURM_NODE>\n```\n\nRelay the script output to the user verbatim.\n\n### Step 4 — Map findings → docs\n\nFor each finding, look up the section in the [decision table](#decision-table) and link\nthe user to the corresponding AWS \u002F Slurm doc. Do not type out remediation commands.\n\n## Decision table\n\n| Symptom (`sinfo -o \"%N %T %30E\"` or script finding)         | Section                                                |\n| ----------------------------------------------------------- | ------------------------------------------------------ |\n| Node state = `down` or `down*`, reason other than below     | [A: Node Down](#a-node-down)                           |\n| Node state = `down*`, Reason = `Node unexpectedly rebooted` | [B: Unexpected Reboot](#b-unexpected-reboot)           |\n| Jobs `PENDING` with `REASON=Resources` while nodes are idle | [C: Controller State](#c-controller-state)             |\n| Jobs stuck `COMPLETING` after node replacement              | [C: Controller State](#c-controller-state)             |\n| `scontrol ping` returns `DOWN` for the controller           | [C: Controller State](#c-controller-state)             |\n| GRES (GPU) counts incorrect or not released                 | [C: Controller State](#c-controller-state)             |\n| `state=fail` issued but no recovery occurred                | [D: Action Reason Mismatch](#d-action-reason-mismatch) |\n| Accounting errors or RPC errors mentioning `dbd`            | [C: Controller State](#c-controller-state) (slurmdbd)  |\n| `slurm.conf` edited; new partitions or nodes not visible    | [C: Controller State](#c-controller-state) (config)    |\n| Job exited on a hardware failure but did not restart        | [E: Auto-resume](#e-auto-resume)                       |\n\n## Defaults\n\n| Behavior             | Default                                                                                            | Override                   |\n| -------------------- | -------------------------------------------------------------------------------------------------- | -------------------------- |\n| Mode                 | read-only — always; no remediation flag exists                                                     | n\u002Fa                        |\n| Region               | `$AWS_DEFAULT_REGION`, falling back to `us-east-1`                                                 | `--region \u003CR>`             |\n| Scope                | all nodes in `down` \u002F `drain` \u002F `fail` \u002F \"unexpectedly rebooted\"                                   | `--node \u003CSLURM_NODE_NAME>` |\n| Output               | colorized terminal                                                                                 | `--no-color`               |\n| SSM target format    | `sagemaker-cluster:\u003CclusterId>_\u003CinstanceGroupName>-\u003CinstanceId>` (derived)                         | n\u002Fa                        |\n| Controller discovery | `--controller-group` (if set) → `SlurmConfig.NodeType=Controller` → `provisioning_parameters.json` | `--controller-group \u003CN>`   |\n\n## Error handling\n\n| Failure                                            | Skill behavior                         | Required user action                            |\n| -------------------------------------------------- | -------------------------------------- | ----------------------------------------------- |\n| `describe-cluster` fails                           | Print AWS error; exit 1                | Fix credentials\u002Fregion; verify cluster name     |\n| Cluster has `Orchestrator.Eks`                     | Exit 1 with pointer to EKS-side skills | Use `hyperpod-node-debugger` or `hyperpod-nccl` |\n| `session-manager-plugin` missing \u002F SSM unreachable | `sinfo` returns empty; exit 1          | Install plugin; verify node `InService`         |\n| Disk ≥ 95 % full on a `down` node                  | Report finding `disk-full-\u003Cnode>`      | Refer to AWS troubleshooting docs               |\n| Missing `jq` or `aws`                              | Exit 1 at prerequisite check           | Install per [Prerequisites](#prerequisites)     |\n\n---\n\n## A: Node Down\n\nNode is `down` because `slurmd` stopped responding. Causes: `slurmd` crash, disk full,\nOOM, network partition, hardware fault.\n\nScript checks: `systemctl is-active slurmd`, `srun -w \u003CNODE> hostname` (RPC layer), disk,\nmemory.\n\nLink: \u003Chttps:\u002F\u002Fgithub.com\u002Faws\u002Fsagemaker-hyperpod-cluster-setup\u002Fblob\u002Ftroubleshooting-doc-20250917\u002Ftroubleshoot\u002Findex.md>\n\nIf node returns to `down` after a manual resume → escalate to `hyperpod-node-debugger`.\n\nContext: [references\u002Fslurm-details.md § A](references\u002Fslurm-details.md#-a-node-down--diagnostic-context).\n\n---\n\n## B: Unexpected Reboot\n\nNode is `down*` with Reason `\"Node unexpectedly rebooted\"` because `slurmd`\nre-registered after an out-of-band reboot. Upstream Slurm behavior, not HyperPod.\nNode is typically healthy.\n\nLinks:\n\n- \u003Chttps:\u002F\u002Fgithub.com\u002Faws\u002Fsagemaker-hyperpod-cluster-setup\u002Fblob\u002Ftroubleshooting-doc-20250917\u002Ftroubleshoot\u002Findex.md>\n- \u003Chttps:\u002F\u002Fslurm.schedmd.com\u002Fscontrol.html> (`state=resume` semantics)\n\nIf node reboots again within minutes → escalate to `hyperpod-node-debugger`.\n\nContext: [references\u002Fslurm-details.md § B](references\u002Fslurm-details.md#-b-unexpected-reboot--diagnostic-context).\n\n---\n\n## C: Controller State\n\n`slurmctld` in-memory state can desync from the on-disk state. A controller restart reloads from `StateSaveLocation` and clears bad caches. User decides and executes.\n\nRestart may help:\n\n| Symptom                                            | Why                                         |\n| -------------------------------------------------- | ------------------------------------------- |\n| `PENDING` with `REASON=Resources`, idle nodes      | Re-evaluates the queue                      |\n| Jobs stuck `COMPLETING` after node replacement     | Controller held a reference to the old node |\n| GRES (GPU, EFA) not released after a job ends      | Resource accounting de-synced               |\n| Nodes stuck `Unknown` after reboot, `slurmd` is up | Re-registration was not processed           |\n| `scontrol ping` times out                          | Controller event loop is hung               |\n| Lost connection to `slurmdbd` \u002F RPC errors         | DBD connection wedged                       |\n\nDo NOT restart when:\n\n- HyperPod replacement (`Action:Replace`) in progress on any node — concurrent changes\n  fail the replacement.\n- Only one compute node is bad — restart `slurmd` on that node.\n- `sinfo` and `squeue` are responsive — problem is elsewhere.\n- `journalctl -u slurmctld` not reviewed yet — panic \u002F OOM will reproduce.\n- `slurm.conf` was just edited — try `scontrol reconfigure` first.\n\n### Folded triggers\n\n- **slurmdbd disconnected** — `sacct` fails, accounting fields show `Unknown`,\n  controller log spams `Unable to contact slurmdbd`. Restore `slurmdbd` before\n  considering controller restart.\n  \u003Chttps:\u002F\u002Fslurm.schedmd.com\u002Faccounting.html> ·\n  [details](references\u002Fslurm-details.md#slurmdbd-connectivity).\n- **Stale config** — `slurm.conf` \u002F `topology.conf` mtime > slurmctld start.\n  `scontrol reconfigure` first; restart is fallback.\n  \u003Chttps:\u002F\u002Fslurm.schedmd.com\u002Fscontrol.html> ·\n  [details](references\u002Fslurm-details.md#scontrol-reconfigure-vs-restart).\n\nRestart procedure \u002F what's preserved:\n\n- \u003Chttps:\u002F\u002Fslurm.schedmd.com\u002Fslurmctld.html>\n- \u003Chttps:\u002F\u002Fgithub.com\u002Faws\u002Fsagemaker-hyperpod-cluster-setup\u002Fblob\u002Ftroubleshooting-doc-20250917\u002Ftroubleshoot\u002Findex.md>\n\nContext: [references\u002Fslurm-details.md § C](references\u002Fslurm-details.md#-c-controller-state--diagnostic-context).\n\n---\n\n## D: Action Reason Mismatch\n\n`scontrol update state=fail reason=...` was issued with a `reason` that does not match\n`Action:Reboot` or `Action:Replace` exactly. HyperPod silently ignores anything else.\nScript detects near-misses on nodes in `fail` state.\n\nRequired strings (case-sensitive, no whitespace, no punctuation):\n\n- `Action:Reboot`\n- `Action:Replace`\n\nLink: \u003Chttps:\u002F\u002Fdocs.aws.amazon.com\u002Fsagemaker\u002Flatest\u002Fdg\u002Fsagemaker-hyperpod-resiliency-slurm-replace-faulty-instance.html>\n\nContext: [references\u002Fslurm-details.md § Action reason-string validation](references\u002Fslurm-details.md#action-reason-string-validation).\n\n---\n\n## E: Auto-resume\n\n`--auto-resume=1` is an `srun` step option. It re-runs the step after HMA (the Health\nMonitoring Agent) flags a node and Automatic node recovery replaces it.\n\nWhy it didn't restart the job:\n\n- Flag on `sbatch` not `srun` — per-step; `sbatch` directives are silently ignored.\n- HMA did not flag the node — failure was application\u002Ftransient, not hardware. Step\n  exits as a normal Slurm failure.\n- Cluster `NodeRecovery` is `None` — faulty nodes are labeled but not replaced.\n- No checkpointing — step restarts from process zero each iteration.\n- AMI predates HMA support (released 2025-09-11) — needs AMI \u002F cluster-software update.\n\nLink: \u003Chttps:\u002F\u002Fdocs.aws.amazon.com\u002Fsagemaker\u002Flatest\u002Fdg\u002Fsagemaker-hyperpod-resiliency-slurm-auto-resume.html>\n\nContext: [references\u002Fslurm-details.md § HyperPod auto-resume](references\u002Fslurm-details.md#hyperpod-auto-resume).\n\n---\n\n## Escalation\n\n| Condition                                                       | Next skill                            |\n| --------------------------------------------------------------- | ------------------------------------- |\n| Node returns to `down` shortly after a manual resume            | `hyperpod-node-debugger` (hardware)   |\n| `slurmd` logs contain CUDA \u002F NVIDIA \u002F XID errors                | `hyperpod-node-debugger` § G          |\n| Disk full or `\u002Fdev\u002Fshm` exhausted                               | `hyperpod-node-debugger` § I          |\n| Node unreachable via SSM                                        | `hyperpod-ssm`                        |\n| Controller restart does not clear `COMPLETING` after 2 attempts | `hyperpod-issue-report` + AWS Support |\n",{"data":43,"body":46},{"name":4,"description":6,"metadata":44},{"version":45},"0.0.1",{"type":47,"children":48},"root",[49,57,63,70,84,90,158,164,182,194,200,305,311,318,323,342,348,469,487,493,639,644,650,661,667,950,956,1166,1172,1357,1361,1366,1393,1413,1424,1442,1454,1457,1462,1487,1492,1522,1533,1544,1547,1552,1571,1576,1720,1725,1800,1806,1907,1912,1932,1943,1946,1951,1991,1996,2015,2025,2036,2039,2044,2063,2068,2134,2144,2155,2158,2164,2302],{"type":50,"tag":51,"props":52,"children":53},"element","h1",{"id":4},[54],{"type":55,"value":56},"text","HyperPod Slurm Debugger",{"type":50,"tag":58,"props":59,"children":60},"p",{},[61],{"type":55,"value":62},"Diagnostic-only. Identify and classify Slurm scheduler and node-daemon issues on\nHyperPod Slurm clusters. Do not run, recommend, or print any state-mutating command.\nFor remediation, link to the official AWS or Slurm documentation.",{"type":50,"tag":64,"props":65,"children":67},"h2",{"id":66},"when-to-invoke",[68],{"type":55,"value":69},"When to invoke",{"type":50,"tag":58,"props":71,"children":72},{},[73,75,82],{"type":55,"value":74},"Invoke when the user reports any of the symptoms in the ",{"type":50,"tag":76,"props":77,"children":79},"a",{"href":78},"#decision-table",[80],{"type":55,"value":81},"decision table",{"type":55,"value":83},".",{"type":50,"tag":64,"props":85,"children":87},{"id":86},"when-not-to-invoke",[88],{"type":55,"value":89},"When NOT to invoke",{"type":50,"tag":91,"props":92,"children":93},"ul",{},[94,124,135,146],{"type":50,"tag":95,"props":96,"children":97},"li",{},[98,100,107,109,115,117,123],{"type":55,"value":99},"Cluster has ",{"type":50,"tag":101,"props":102,"children":104},"code",{"className":103},[],[105],{"type":55,"value":106},"Orchestrator.Eks",{"type":55,"value":108}," — invoke ",{"type":50,"tag":101,"props":110,"children":112},{"className":111},[],[113],{"type":55,"value":114},"hyperpod-node-debugger",{"type":55,"value":116}," or ",{"type":50,"tag":101,"props":118,"children":120},{"className":119},[],[121],{"type":55,"value":122},"hyperpod-nccl",{"type":55,"value":83},{"type":50,"tag":95,"props":125,"children":126},{},[127,129,134],{"type":55,"value":128},"Single-node hardware fault with healthy Slurm scheduler — invoke ",{"type":50,"tag":101,"props":130,"children":132},{"className":131},[],[133],{"type":55,"value":114},{"type":55,"value":83},{"type":50,"tag":95,"props":136,"children":137},{},[138,140,145],{"type":55,"value":139},"NCCL training-hang investigation — invoke ",{"type":50,"tag":101,"props":141,"children":143},{"className":142},[],[144],{"type":55,"value":122},{"type":55,"value":83},{"type":50,"tag":95,"props":147,"children":148},{},[149,151,157],{"type":55,"value":150},"Node unreachable via SSM — invoke ",{"type":50,"tag":101,"props":152,"children":154},{"className":153},[],[155],{"type":55,"value":156},"hyperpod-ssm",{"type":55,"value":83},{"type":50,"tag":64,"props":159,"children":161},{"id":160},"constraints",[162],{"type":55,"value":163},"Constraints",{"type":50,"tag":91,"props":165,"children":166},{},[167,172,177],{"type":50,"tag":95,"props":168,"children":169},{},[170],{"type":55,"value":171},"Read-only. Do not run, recommend, or print state-mutating commands.",{"type":50,"tag":95,"props":173,"children":174},{},[175],{"type":55,"value":176},"For any remediation, link to AWS or Slurm docs. The user authorizes and executes.",{"type":50,"tag":95,"props":178,"children":179},{},[180],{"type":55,"value":181},"IaC-managed cluster (Terraform \u002F CloudFormation \u002F CDK): warn that direct mutation\ndrifts the live state from the IaC plan.",{"type":50,"tag":58,"props":183,"children":184},{},[185,187,193],{"type":55,"value":186},"Canonical recovery URLs:\n",{"type":50,"tag":76,"props":188,"children":190},{"href":189},"references\u002Fslurm-details.md",[191],{"type":55,"value":192},"references\u002Fslurm-details.md → Authoritative recovery documentation",{"type":55,"value":83},{"type":50,"tag":64,"props":195,"children":197},{"id":196},"prerequisites",[198],{"type":55,"value":199},"Prerequisites",{"type":50,"tag":91,"props":201,"children":202},{},[203,239,252,263],{"type":50,"tag":95,"props":204,"children":205},{},[206,208],{"type":55,"value":207},"AWS CLI v2, authenticated for the target account and region with permissions:\n",{"type":50,"tag":91,"props":209,"children":210},{},[211,228],{"type":50,"tag":95,"props":212,"children":213},{},[214,220,222],{"type":50,"tag":101,"props":215,"children":217},{"className":216},[],[218],{"type":55,"value":219},"sagemaker:DescribeCluster",{"type":55,"value":221},", ",{"type":50,"tag":101,"props":223,"children":225},{"className":224},[],[226],{"type":55,"value":227},"sagemaker:ListClusterNodes",{"type":50,"tag":95,"props":229,"children":230},{},[231,237],{"type":50,"tag":101,"props":232,"children":234},{"className":233},[],[235],{"type":55,"value":236},"ssm:StartSession",{"type":55,"value":238}," on the HyperPod-created SSM document",{"type":50,"tag":95,"props":240,"children":241},{},[242,250],{"type":50,"tag":76,"props":243,"children":247},{"href":244,"rel":245},"https:\u002F\u002Fdocs.aws.amazon.com\u002Fsystems-manager\u002Flatest\u002Fuserguide\u002Fsession-manager-working-with-install-plugin.html",[246],"nofollow",[248],{"type":55,"value":249},"Session Manager plugin",{"type":55,"value":251},"\ninstalled locally.",{"type":50,"tag":95,"props":253,"children":254},{},[255,261],{"type":50,"tag":101,"props":256,"children":258},{"className":257},[],[259],{"type":55,"value":260},"jq",{"type":55,"value":262}," ≥ 1.6.",{"type":50,"tag":95,"props":264,"children":265},{},[266,272,274,280,282,288,290,296,298,303],{"type":50,"tag":101,"props":267,"children":269},{"className":268},[],[270],{"type":55,"value":271},"unbuffer",{"type":55,"value":273}," (from the ",{"type":50,"tag":101,"props":275,"children":277},{"className":276},[],[278],{"type":55,"value":279},"expect",{"type":55,"value":281}," package). Required — without it ",{"type":50,"tag":101,"props":283,"children":285},{"className":284},[],[286],{"type":55,"value":287},"aws ssm start-session",{"type":55,"value":289},"\nreturns empty stdout intermittently with ",{"type":50,"tag":101,"props":291,"children":293},{"className":292},[],[294],{"type":55,"value":295},"Cannot perform start session: EOF",{"type":55,"value":297}," and every\ncheck silently misreports. Install: ",{"type":50,"tag":101,"props":299,"children":301},{"className":300},[],[302],{"type":55,"value":279},{"type":55,"value":304}," package on Amazon Linux \u002F RHEL \u002F Debian \u002F\nUbuntu \u002F macOS. Script exits at prerequisite check if missing.",{"type":50,"tag":64,"props":306,"children":308},{"id":307},"procedure",[309],{"type":55,"value":310},"Procedure",{"type":50,"tag":312,"props":313,"children":315},"h3",{"id":314},"step-1-collect-inputs",[316],{"type":55,"value":317},"Step 1 — Collect inputs",{"type":50,"tag":58,"props":319,"children":320},{},[321],{"type":55,"value":322},"Ask the user for:",{"type":50,"tag":324,"props":325,"children":326},"ol",{},[327,332,337],{"type":50,"tag":95,"props":328,"children":329},{},[330],{"type":55,"value":331},"HyperPod cluster name (not Slurm partition name).",{"type":50,"tag":95,"props":333,"children":334},{},[335],{"type":55,"value":336},"AWS region.",{"type":50,"tag":95,"props":338,"children":339},{},[340],{"type":55,"value":341},"Optional: a specific Slurm node name.",{"type":50,"tag":312,"props":343,"children":345},{"id":344},"step-2-confirm-orchestrator",[346],{"type":55,"value":347},"Step 2 — Confirm orchestrator",{"type":50,"tag":349,"props":350,"children":355},"pre",{"className":351,"code":352,"language":353,"meta":354,"style":354},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","aws sagemaker describe-cluster --cluster-name \u003CNAME\u002FARN> --region \u003CREGION> \\\n  --query 'Orchestrator' --output json\n","bash","",[356],{"type":50,"tag":101,"props":357,"children":358},{"__ignoreMap":354},[359,435],{"type":50,"tag":360,"props":361,"children":364},"span",{"class":362,"line":363},"line",1,[365,370,376,381,386,392,397,403,408,413,417,422,426,430],{"type":50,"tag":360,"props":366,"children":368},{"style":367},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[369],{"type":55,"value":22},{"type":50,"tag":360,"props":371,"children":373},{"style":372},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[374],{"type":55,"value":375}," sagemaker",{"type":50,"tag":360,"props":377,"children":378},{"style":372},[379],{"type":55,"value":380}," describe-cluster",{"type":50,"tag":360,"props":382,"children":383},{"style":372},[384],{"type":55,"value":385}," --cluster-name",{"type":50,"tag":360,"props":387,"children":389},{"style":388},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[390],{"type":55,"value":391}," \u003C",{"type":50,"tag":360,"props":393,"children":394},{"style":372},[395],{"type":55,"value":396},"NAME\u002FAR",{"type":50,"tag":360,"props":398,"children":400},{"style":399},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[401],{"type":55,"value":402},"N",{"type":50,"tag":360,"props":404,"children":405},{"style":388},[406],{"type":55,"value":407},">",{"type":50,"tag":360,"props":409,"children":410},{"style":372},[411],{"type":55,"value":412}," --region",{"type":50,"tag":360,"props":414,"children":415},{"style":388},[416],{"type":55,"value":391},{"type":50,"tag":360,"props":418,"children":419},{"style":372},[420],{"type":55,"value":421},"REGIO",{"type":50,"tag":360,"props":423,"children":424},{"style":399},[425],{"type":55,"value":402},{"type":50,"tag":360,"props":427,"children":428},{"style":388},[429],{"type":55,"value":407},{"type":50,"tag":360,"props":431,"children":432},{"style":399},[433],{"type":55,"value":434}," \\\n",{"type":50,"tag":360,"props":436,"children":438},{"class":362,"line":437},2,[439,444,449,454,459,464],{"type":50,"tag":360,"props":440,"children":441},{"style":372},[442],{"type":55,"value":443},"  --query",{"type":50,"tag":360,"props":445,"children":446},{"style":388},[447],{"type":55,"value":448}," '",{"type":50,"tag":360,"props":450,"children":451},{"style":372},[452],{"type":55,"value":453},"Orchestrator",{"type":50,"tag":360,"props":455,"children":456},{"style":388},[457],{"type":55,"value":458},"'",{"type":50,"tag":360,"props":460,"children":461},{"style":372},[462],{"type":55,"value":463}," --output",{"type":50,"tag":360,"props":465,"children":466},{"style":372},[467],{"type":55,"value":468}," json\n",{"type":50,"tag":58,"props":470,"children":471},{},[472,474,479,481,486],{"type":55,"value":473},"If ",{"type":50,"tag":101,"props":475,"children":477},{"className":476},[],[478],{"type":55,"value":106},{"type":55,"value":480}," is present, stop. Route per ",{"type":50,"tag":76,"props":482,"children":484},{"href":483},"#when-not-to-invoke",[485],{"type":55,"value":89},{"type":55,"value":83},{"type":50,"tag":312,"props":488,"children":490},{"id":489},"step-3-run-the-diagnostic-script",[491],{"type":55,"value":492},"Step 3 — Run the diagnostic script",{"type":50,"tag":349,"props":494,"children":496},{"className":351,"code":495,"language":353,"meta":354,"style":354},"bash scripts\u002Fslurm-diagnose.sh --cluster \u003CNAME> --region \u003CREGION>\n# Scope to a node:\nbash scripts\u002Fslurm-diagnose.sh --cluster \u003CNAME> --region \u003CREGION> --node \u003CSLURM_NODE>\n",[497],{"type":50,"tag":101,"props":498,"children":499},{"__ignoreMap":354},[500,556,565],{"type":50,"tag":360,"props":501,"children":502},{"class":362,"line":363},[503,507,512,517,521,526,531,535,539,543,547,551],{"type":50,"tag":360,"props":504,"children":505},{"style":367},[506],{"type":55,"value":353},{"type":50,"tag":360,"props":508,"children":509},{"style":372},[510],{"type":55,"value":511}," scripts\u002Fslurm-diagnose.sh",{"type":50,"tag":360,"props":513,"children":514},{"style":372},[515],{"type":55,"value":516}," --cluster",{"type":50,"tag":360,"props":518,"children":519},{"style":388},[520],{"type":55,"value":391},{"type":50,"tag":360,"props":522,"children":523},{"style":372},[524],{"type":55,"value":525},"NAM",{"type":50,"tag":360,"props":527,"children":528},{"style":399},[529],{"type":55,"value":530},"E",{"type":50,"tag":360,"props":532,"children":533},{"style":388},[534],{"type":55,"value":407},{"type":50,"tag":360,"props":536,"children":537},{"style":372},[538],{"type":55,"value":412},{"type":50,"tag":360,"props":540,"children":541},{"style":388},[542],{"type":55,"value":391},{"type":50,"tag":360,"props":544,"children":545},{"style":372},[546],{"type":55,"value":421},{"type":50,"tag":360,"props":548,"children":549},{"style":399},[550],{"type":55,"value":402},{"type":50,"tag":360,"props":552,"children":553},{"style":388},[554],{"type":55,"value":555},">\n",{"type":50,"tag":360,"props":557,"children":558},{"class":362,"line":437},[559],{"type":50,"tag":360,"props":560,"children":562},{"style":561},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[563],{"type":55,"value":564},"# Scope to a node:\n",{"type":50,"tag":360,"props":566,"children":568},{"class":362,"line":567},3,[569,573,577,581,585,589,593,597,601,605,609,613,617,622,626,631,635],{"type":50,"tag":360,"props":570,"children":571},{"style":367},[572],{"type":55,"value":353},{"type":50,"tag":360,"props":574,"children":575},{"style":372},[576],{"type":55,"value":511},{"type":50,"tag":360,"props":578,"children":579},{"style":372},[580],{"type":55,"value":516},{"type":50,"tag":360,"props":582,"children":583},{"style":388},[584],{"type":55,"value":391},{"type":50,"tag":360,"props":586,"children":587},{"style":372},[588],{"type":55,"value":525},{"type":50,"tag":360,"props":590,"children":591},{"style":399},[592],{"type":55,"value":530},{"type":50,"tag":360,"props":594,"children":595},{"style":388},[596],{"type":55,"value":407},{"type":50,"tag":360,"props":598,"children":599},{"style":372},[600],{"type":55,"value":412},{"type":50,"tag":360,"props":602,"children":603},{"style":388},[604],{"type":55,"value":391},{"type":50,"tag":360,"props":606,"children":607},{"style":372},[608],{"type":55,"value":421},{"type":50,"tag":360,"props":610,"children":611},{"style":399},[612],{"type":55,"value":402},{"type":50,"tag":360,"props":614,"children":615},{"style":388},[616],{"type":55,"value":407},{"type":50,"tag":360,"props":618,"children":619},{"style":372},[620],{"type":55,"value":621}," --node",{"type":50,"tag":360,"props":623,"children":624},{"style":388},[625],{"type":55,"value":391},{"type":50,"tag":360,"props":627,"children":628},{"style":372},[629],{"type":55,"value":630},"SLURM_NOD",{"type":50,"tag":360,"props":632,"children":633},{"style":399},[634],{"type":55,"value":530},{"type":50,"tag":360,"props":636,"children":637},{"style":388},[638],{"type":55,"value":555},{"type":50,"tag":58,"props":640,"children":641},{},[642],{"type":55,"value":643},"Relay the script output to the user verbatim.",{"type":50,"tag":312,"props":645,"children":647},{"id":646},"step-4-map-findings-docs",[648],{"type":55,"value":649},"Step 4 — Map findings → docs",{"type":50,"tag":58,"props":651,"children":652},{},[653,655,659],{"type":55,"value":654},"For each finding, look up the section in the ",{"type":50,"tag":76,"props":656,"children":657},{"href":78},[658],{"type":55,"value":81},{"type":55,"value":660}," and link\nthe user to the corresponding AWS \u002F Slurm doc. Do not type out remediation commands.",{"type":50,"tag":64,"props":662,"children":664},{"id":663},"decision-table",[665],{"type":55,"value":666},"Decision table",{"type":50,"tag":668,"props":669,"children":670},"table",{},[671,698],{"type":50,"tag":672,"props":673,"children":674},"thead",{},[675],{"type":50,"tag":676,"props":677,"children":678},"tr",{},[679,693],{"type":50,"tag":680,"props":681,"children":682},"th",{},[683,685,691],{"type":55,"value":684},"Symptom (",{"type":50,"tag":101,"props":686,"children":688},{"className":687},[],[689],{"type":55,"value":690},"sinfo -o \"%N %T %30E\"",{"type":55,"value":692}," or script finding)",{"type":50,"tag":680,"props":694,"children":695},{},[696],{"type":55,"value":697},"Section",{"type":50,"tag":699,"props":700,"children":701},"tbody",{},[702,735,764,797,820,849,864,887,910,933],{"type":50,"tag":676,"props":703,"children":704},{},[705,726],{"type":50,"tag":706,"props":707,"children":708},"td",{},[709,711,717,718,724],{"type":55,"value":710},"Node state = ",{"type":50,"tag":101,"props":712,"children":714},{"className":713},[],[715],{"type":55,"value":716},"down",{"type":55,"value":116},{"type":50,"tag":101,"props":719,"children":721},{"className":720},[],[722],{"type":55,"value":723},"down*",{"type":55,"value":725},", reason other than below",{"type":50,"tag":706,"props":727,"children":728},{},[729],{"type":50,"tag":76,"props":730,"children":732},{"href":731},"#a-node-down",[733],{"type":55,"value":734},"A: Node Down",{"type":50,"tag":676,"props":736,"children":737},{},[738,755],{"type":50,"tag":706,"props":739,"children":740},{},[741,742,747,749],{"type":55,"value":710},{"type":50,"tag":101,"props":743,"children":745},{"className":744},[],[746],{"type":55,"value":723},{"type":55,"value":748},", Reason = ",{"type":50,"tag":101,"props":750,"children":752},{"className":751},[],[753],{"type":55,"value":754},"Node unexpectedly rebooted",{"type":50,"tag":706,"props":756,"children":757},{},[758],{"type":50,"tag":76,"props":759,"children":761},{"href":760},"#b-unexpected-reboot",[762],{"type":55,"value":763},"B: Unexpected Reboot",{"type":50,"tag":676,"props":765,"children":766},{},[767,788],{"type":50,"tag":706,"props":768,"children":769},{},[770,772,778,780,786],{"type":55,"value":771},"Jobs ",{"type":50,"tag":101,"props":773,"children":775},{"className":774},[],[776],{"type":55,"value":777},"PENDING",{"type":55,"value":779}," with ",{"type":50,"tag":101,"props":781,"children":783},{"className":782},[],[784],{"type":55,"value":785},"REASON=Resources",{"type":55,"value":787}," while nodes are idle",{"type":50,"tag":706,"props":789,"children":790},{},[791],{"type":50,"tag":76,"props":792,"children":794},{"href":793},"#c-controller-state",[795],{"type":55,"value":796},"C: Controller State",{"type":50,"tag":676,"props":798,"children":799},{},[800,813],{"type":50,"tag":706,"props":801,"children":802},{},[803,805,811],{"type":55,"value":804},"Jobs stuck ",{"type":50,"tag":101,"props":806,"children":808},{"className":807},[],[809],{"type":55,"value":810},"COMPLETING",{"type":55,"value":812}," after node replacement",{"type":50,"tag":706,"props":814,"children":815},{},[816],{"type":50,"tag":76,"props":817,"children":818},{"href":793},[819],{"type":55,"value":796},{"type":50,"tag":676,"props":821,"children":822},{},[823,842],{"type":50,"tag":706,"props":824,"children":825},{},[826,832,834,840],{"type":50,"tag":101,"props":827,"children":829},{"className":828},[],[830],{"type":55,"value":831},"scontrol ping",{"type":55,"value":833}," returns ",{"type":50,"tag":101,"props":835,"children":837},{"className":836},[],[838],{"type":55,"value":839},"DOWN",{"type":55,"value":841}," for the controller",{"type":50,"tag":706,"props":843,"children":844},{},[845],{"type":50,"tag":76,"props":846,"children":847},{"href":793},[848],{"type":55,"value":796},{"type":50,"tag":676,"props":850,"children":851},{},[852,857],{"type":50,"tag":706,"props":853,"children":854},{},[855],{"type":55,"value":856},"GRES (GPU) counts incorrect or not released",{"type":50,"tag":706,"props":858,"children":859},{},[860],{"type":50,"tag":76,"props":861,"children":862},{"href":793},[863],{"type":55,"value":796},{"type":50,"tag":676,"props":865,"children":866},{},[867,878],{"type":50,"tag":706,"props":868,"children":869},{},[870,876],{"type":50,"tag":101,"props":871,"children":873},{"className":872},[],[874],{"type":55,"value":875},"state=fail",{"type":55,"value":877}," issued but no recovery occurred",{"type":50,"tag":706,"props":879,"children":880},{},[881],{"type":50,"tag":76,"props":882,"children":884},{"href":883},"#d-action-reason-mismatch",[885],{"type":55,"value":886},"D: Action Reason Mismatch",{"type":50,"tag":676,"props":888,"children":889},{},[890,901],{"type":50,"tag":706,"props":891,"children":892},{},[893,895],{"type":55,"value":894},"Accounting errors or RPC errors mentioning ",{"type":50,"tag":101,"props":896,"children":898},{"className":897},[],[899],{"type":55,"value":900},"dbd",{"type":50,"tag":706,"props":902,"children":903},{},[904,908],{"type":50,"tag":76,"props":905,"children":906},{"href":793},[907],{"type":55,"value":796},{"type":55,"value":909}," (slurmdbd)",{"type":50,"tag":676,"props":911,"children":912},{},[913,924],{"type":50,"tag":706,"props":914,"children":915},{},[916,922],{"type":50,"tag":101,"props":917,"children":919},{"className":918},[],[920],{"type":55,"value":921},"slurm.conf",{"type":55,"value":923}," edited; new partitions or nodes not visible",{"type":50,"tag":706,"props":925,"children":926},{},[927,931],{"type":50,"tag":76,"props":928,"children":929},{"href":793},[930],{"type":55,"value":796},{"type":55,"value":932}," (config)",{"type":50,"tag":676,"props":934,"children":935},{},[936,941],{"type":50,"tag":706,"props":937,"children":938},{},[939],{"type":55,"value":940},"Job exited on a hardware failure but did not restart",{"type":50,"tag":706,"props":942,"children":943},{},[944],{"type":50,"tag":76,"props":945,"children":947},{"href":946},"#e-auto-resume",[948],{"type":55,"value":949},"E: Auto-resume",{"type":50,"tag":64,"props":951,"children":953},{"id":952},"defaults",[954],{"type":55,"value":955},"Defaults",{"type":50,"tag":668,"props":957,"children":958},{},[959,980],{"type":50,"tag":672,"props":960,"children":961},{},[962],{"type":50,"tag":676,"props":963,"children":964},{},[965,970,975],{"type":50,"tag":680,"props":966,"children":967},{},[968],{"type":55,"value":969},"Behavior",{"type":50,"tag":680,"props":971,"children":972},{},[973],{"type":55,"value":974},"Default",{"type":50,"tag":680,"props":976,"children":977},{},[978],{"type":55,"value":979},"Override",{"type":50,"tag":699,"props":981,"children":982},{},[983,1001,1035,1079,1101,1124],{"type":50,"tag":676,"props":984,"children":985},{},[986,991,996],{"type":50,"tag":706,"props":987,"children":988},{},[989],{"type":55,"value":990},"Mode",{"type":50,"tag":706,"props":992,"children":993},{},[994],{"type":55,"value":995},"read-only — always; no remediation flag exists",{"type":50,"tag":706,"props":997,"children":998},{},[999],{"type":55,"value":1000},"n\u002Fa",{"type":50,"tag":676,"props":1002,"children":1003},{},[1004,1009,1026],{"type":50,"tag":706,"props":1005,"children":1006},{},[1007],{"type":55,"value":1008},"Region",{"type":50,"tag":706,"props":1010,"children":1011},{},[1012,1018,1020],{"type":50,"tag":101,"props":1013,"children":1015},{"className":1014},[],[1016],{"type":55,"value":1017},"$AWS_DEFAULT_REGION",{"type":55,"value":1019},", falling back to ",{"type":50,"tag":101,"props":1021,"children":1023},{"className":1022},[],[1024],{"type":55,"value":1025},"us-east-1",{"type":50,"tag":706,"props":1027,"children":1028},{},[1029],{"type":50,"tag":101,"props":1030,"children":1032},{"className":1031},[],[1033],{"type":55,"value":1034},"--region \u003CR>",{"type":50,"tag":676,"props":1036,"children":1037},{},[1038,1043,1070],{"type":50,"tag":706,"props":1039,"children":1040},{},[1041],{"type":55,"value":1042},"Scope",{"type":50,"tag":706,"props":1044,"children":1045},{},[1046,1048,1053,1055,1061,1062,1068],{"type":55,"value":1047},"all nodes in ",{"type":50,"tag":101,"props":1049,"children":1051},{"className":1050},[],[1052],{"type":55,"value":716},{"type":55,"value":1054}," \u002F ",{"type":50,"tag":101,"props":1056,"children":1058},{"className":1057},[],[1059],{"type":55,"value":1060},"drain",{"type":55,"value":1054},{"type":50,"tag":101,"props":1063,"children":1065},{"className":1064},[],[1066],{"type":55,"value":1067},"fail",{"type":55,"value":1069}," \u002F \"unexpectedly rebooted\"",{"type":50,"tag":706,"props":1071,"children":1072},{},[1073],{"type":50,"tag":101,"props":1074,"children":1076},{"className":1075},[],[1077],{"type":55,"value":1078},"--node \u003CSLURM_NODE_NAME>",{"type":50,"tag":676,"props":1080,"children":1081},{},[1082,1087,1092],{"type":50,"tag":706,"props":1083,"children":1084},{},[1085],{"type":55,"value":1086},"Output",{"type":50,"tag":706,"props":1088,"children":1089},{},[1090],{"type":55,"value":1091},"colorized terminal",{"type":50,"tag":706,"props":1093,"children":1094},{},[1095],{"type":50,"tag":101,"props":1096,"children":1098},{"className":1097},[],[1099],{"type":55,"value":1100},"--no-color",{"type":50,"tag":676,"props":1102,"children":1103},{},[1104,1109,1120],{"type":50,"tag":706,"props":1105,"children":1106},{},[1107],{"type":55,"value":1108},"SSM target format",{"type":50,"tag":706,"props":1110,"children":1111},{},[1112,1118],{"type":50,"tag":101,"props":1113,"children":1115},{"className":1114},[],[1116],{"type":55,"value":1117},"sagemaker-cluster:\u003CclusterId>_\u003CinstanceGroupName>-\u003CinstanceId>",{"type":55,"value":1119}," (derived)",{"type":50,"tag":706,"props":1121,"children":1122},{},[1123],{"type":55,"value":1000},{"type":50,"tag":676,"props":1125,"children":1126},{},[1127,1132,1157],{"type":50,"tag":706,"props":1128,"children":1129},{},[1130],{"type":55,"value":1131},"Controller discovery",{"type":50,"tag":706,"props":1133,"children":1134},{},[1135,1141,1143,1149,1151],{"type":50,"tag":101,"props":1136,"children":1138},{"className":1137},[],[1139],{"type":55,"value":1140},"--controller-group",{"type":55,"value":1142}," (if set) → ",{"type":50,"tag":101,"props":1144,"children":1146},{"className":1145},[],[1147],{"type":55,"value":1148},"SlurmConfig.NodeType=Controller",{"type":55,"value":1150}," → ",{"type":50,"tag":101,"props":1152,"children":1154},{"className":1153},[],[1155],{"type":55,"value":1156},"provisioning_parameters.json",{"type":50,"tag":706,"props":1158,"children":1159},{},[1160],{"type":50,"tag":101,"props":1161,"children":1163},{"className":1162},[],[1164],{"type":55,"value":1165},"--controller-group \u003CN>",{"type":50,"tag":64,"props":1167,"children":1169},{"id":1168},"error-handling",[1170],{"type":55,"value":1171},"Error handling",{"type":50,"tag":668,"props":1173,"children":1174},{},[1175,1196],{"type":50,"tag":672,"props":1176,"children":1177},{},[1178],{"type":50,"tag":676,"props":1179,"children":1180},{},[1181,1186,1191],{"type":50,"tag":680,"props":1182,"children":1183},{},[1184],{"type":55,"value":1185},"Failure",{"type":50,"tag":680,"props":1187,"children":1188},{},[1189],{"type":55,"value":1190},"Skill behavior",{"type":50,"tag":680,"props":1192,"children":1193},{},[1194],{"type":55,"value":1195},"Required user action",{"type":50,"tag":699,"props":1197,"children":1198},{},[1199,1223,1256,1292,1323],{"type":50,"tag":676,"props":1200,"children":1201},{},[1202,1213,1218],{"type":50,"tag":706,"props":1203,"children":1204},{},[1205,1211],{"type":50,"tag":101,"props":1206,"children":1208},{"className":1207},[],[1209],{"type":55,"value":1210},"describe-cluster",{"type":55,"value":1212}," fails",{"type":50,"tag":706,"props":1214,"children":1215},{},[1216],{"type":55,"value":1217},"Print AWS error; exit 1",{"type":50,"tag":706,"props":1219,"children":1220},{},[1221],{"type":55,"value":1222},"Fix credentials\u002Fregion; verify cluster name",{"type":50,"tag":676,"props":1224,"children":1225},{},[1226,1235,1240],{"type":50,"tag":706,"props":1227,"children":1228},{},[1229,1230],{"type":55,"value":99},{"type":50,"tag":101,"props":1231,"children":1233},{"className":1232},[],[1234],{"type":55,"value":106},{"type":50,"tag":706,"props":1236,"children":1237},{},[1238],{"type":55,"value":1239},"Exit 1 with pointer to EKS-side skills",{"type":50,"tag":706,"props":1241,"children":1242},{},[1243,1245,1250,1251],{"type":55,"value":1244},"Use ",{"type":50,"tag":101,"props":1246,"children":1248},{"className":1247},[],[1249],{"type":55,"value":114},{"type":55,"value":116},{"type":50,"tag":101,"props":1252,"children":1254},{"className":1253},[],[1255],{"type":55,"value":122},{"type":50,"tag":676,"props":1257,"children":1258},{},[1259,1270,1281],{"type":50,"tag":706,"props":1260,"children":1261},{},[1262,1268],{"type":50,"tag":101,"props":1263,"children":1265},{"className":1264},[],[1266],{"type":55,"value":1267},"session-manager-plugin",{"type":55,"value":1269}," missing \u002F SSM unreachable",{"type":50,"tag":706,"props":1271,"children":1272},{},[1273,1279],{"type":50,"tag":101,"props":1274,"children":1276},{"className":1275},[],[1277],{"type":55,"value":1278},"sinfo",{"type":55,"value":1280}," returns empty; exit 1",{"type":50,"tag":706,"props":1282,"children":1283},{},[1284,1286],{"type":55,"value":1285},"Install plugin; verify node ",{"type":50,"tag":101,"props":1287,"children":1289},{"className":1288},[],[1290],{"type":55,"value":1291},"InService",{"type":50,"tag":676,"props":1293,"children":1294},{},[1295,1307,1318],{"type":50,"tag":706,"props":1296,"children":1297},{},[1298,1300,1305],{"type":55,"value":1299},"Disk ≥ 95 % full on a ",{"type":50,"tag":101,"props":1301,"children":1303},{"className":1302},[],[1304],{"type":55,"value":716},{"type":55,"value":1306}," node",{"type":50,"tag":706,"props":1308,"children":1309},{},[1310,1312],{"type":55,"value":1311},"Report finding ",{"type":50,"tag":101,"props":1313,"children":1315},{"className":1314},[],[1316],{"type":55,"value":1317},"disk-full-\u003Cnode>",{"type":50,"tag":706,"props":1319,"children":1320},{},[1321],{"type":55,"value":1322},"Refer to AWS troubleshooting docs",{"type":50,"tag":676,"props":1324,"children":1325},{},[1326,1342,1347],{"type":50,"tag":706,"props":1327,"children":1328},{},[1329,1331,1336,1337],{"type":55,"value":1330},"Missing ",{"type":50,"tag":101,"props":1332,"children":1334},{"className":1333},[],[1335],{"type":55,"value":260},{"type":55,"value":116},{"type":50,"tag":101,"props":1338,"children":1340},{"className":1339},[],[1341],{"type":55,"value":22},{"type":50,"tag":706,"props":1343,"children":1344},{},[1345],{"type":55,"value":1346},"Exit 1 at prerequisite check",{"type":50,"tag":706,"props":1348,"children":1349},{},[1350,1352],{"type":55,"value":1351},"Install per ",{"type":50,"tag":76,"props":1353,"children":1355},{"href":1354},"#prerequisites",[1356],{"type":55,"value":199},{"type":50,"tag":1358,"props":1359,"children":1360},"hr",{},[],{"type":50,"tag":64,"props":1362,"children":1364},{"id":1363},"a-node-down",[1365],{"type":55,"value":734},{"type":50,"tag":58,"props":1367,"children":1368},{},[1369,1371,1376,1378,1384,1386,1391],{"type":55,"value":1370},"Node is ",{"type":50,"tag":101,"props":1372,"children":1374},{"className":1373},[],[1375],{"type":55,"value":716},{"type":55,"value":1377}," because ",{"type":50,"tag":101,"props":1379,"children":1381},{"className":1380},[],[1382],{"type":55,"value":1383},"slurmd",{"type":55,"value":1385}," stopped responding. Causes: ",{"type":50,"tag":101,"props":1387,"children":1389},{"className":1388},[],[1390],{"type":55,"value":1383},{"type":55,"value":1392}," crash, disk full,\nOOM, network partition, hardware fault.",{"type":50,"tag":58,"props":1394,"children":1395},{},[1396,1398,1404,1405,1411],{"type":55,"value":1397},"Script checks: ",{"type":50,"tag":101,"props":1399,"children":1401},{"className":1400},[],[1402],{"type":55,"value":1403},"systemctl is-active slurmd",{"type":55,"value":221},{"type":50,"tag":101,"props":1406,"children":1408},{"className":1407},[],[1409],{"type":55,"value":1410},"srun -w \u003CNODE> hostname",{"type":55,"value":1412}," (RPC layer), disk,\nmemory.",{"type":50,"tag":58,"props":1414,"children":1415},{},[1416,1418],{"type":55,"value":1417},"Link: ",{"type":50,"tag":76,"props":1419,"children":1422},{"href":1420,"rel":1421},"https:\u002F\u002Fgithub.com\u002Faws\u002Fsagemaker-hyperpod-cluster-setup\u002Fblob\u002Ftroubleshooting-doc-20250917\u002Ftroubleshoot\u002Findex.md",[246],[1423],{"type":55,"value":1420},{"type":50,"tag":58,"props":1425,"children":1426},{},[1427,1429,1434,1436,1441],{"type":55,"value":1428},"If node returns to ",{"type":50,"tag":101,"props":1430,"children":1432},{"className":1431},[],[1433],{"type":55,"value":716},{"type":55,"value":1435}," after a manual resume → escalate to ",{"type":50,"tag":101,"props":1437,"children":1439},{"className":1438},[],[1440],{"type":55,"value":114},{"type":55,"value":83},{"type":50,"tag":58,"props":1443,"children":1444},{},[1445,1447,1453],{"type":55,"value":1446},"Context: ",{"type":50,"tag":76,"props":1448,"children":1450},{"href":1449},"references\u002Fslurm-details.md#-a-node-down--diagnostic-context",[1451],{"type":55,"value":1452},"references\u002Fslurm-details.md § A",{"type":55,"value":83},{"type":50,"tag":1358,"props":1455,"children":1456},{},[],{"type":50,"tag":64,"props":1458,"children":1460},{"id":1459},"b-unexpected-reboot",[1461],{"type":55,"value":763},{"type":50,"tag":58,"props":1463,"children":1464},{},[1465,1466,1471,1473,1479,1480,1485],{"type":55,"value":1370},{"type":50,"tag":101,"props":1467,"children":1469},{"className":1468},[],[1470],{"type":55,"value":723},{"type":55,"value":1472}," with Reason ",{"type":50,"tag":101,"props":1474,"children":1476},{"className":1475},[],[1477],{"type":55,"value":1478},"\"Node unexpectedly rebooted\"",{"type":55,"value":1377},{"type":50,"tag":101,"props":1481,"children":1483},{"className":1482},[],[1484],{"type":55,"value":1383},{"type":55,"value":1486},"\nre-registered after an out-of-band reboot. Upstream Slurm behavior, not HyperPod.\nNode is typically healthy.",{"type":50,"tag":58,"props":1488,"children":1489},{},[1490],{"type":55,"value":1491},"Links:",{"type":50,"tag":91,"props":1493,"children":1494},{},[1495,1503],{"type":50,"tag":95,"props":1496,"children":1497},{},[1498],{"type":50,"tag":76,"props":1499,"children":1501},{"href":1420,"rel":1500},[246],[1502],{"type":55,"value":1420},{"type":50,"tag":95,"props":1504,"children":1505},{},[1506,1512,1514,1520],{"type":50,"tag":76,"props":1507,"children":1510},{"href":1508,"rel":1509},"https:\u002F\u002Fslurm.schedmd.com\u002Fscontrol.html",[246],[1511],{"type":55,"value":1508},{"type":55,"value":1513}," (",{"type":50,"tag":101,"props":1515,"children":1517},{"className":1516},[],[1518],{"type":55,"value":1519},"state=resume",{"type":55,"value":1521}," semantics)",{"type":50,"tag":58,"props":1523,"children":1524},{},[1525,1527,1532],{"type":55,"value":1526},"If node reboots again within minutes → escalate to ",{"type":50,"tag":101,"props":1528,"children":1530},{"className":1529},[],[1531],{"type":55,"value":114},{"type":55,"value":83},{"type":50,"tag":58,"props":1534,"children":1535},{},[1536,1537,1543],{"type":55,"value":1446},{"type":50,"tag":76,"props":1538,"children":1540},{"href":1539},"references\u002Fslurm-details.md#-b-unexpected-reboot--diagnostic-context",[1541],{"type":55,"value":1542},"references\u002Fslurm-details.md § B",{"type":55,"value":83},{"type":50,"tag":1358,"props":1545,"children":1546},{},[],{"type":50,"tag":64,"props":1548,"children":1550},{"id":1549},"c-controller-state",[1551],{"type":55,"value":796},{"type":50,"tag":58,"props":1553,"children":1554},{},[1555,1561,1563,1569],{"type":50,"tag":101,"props":1556,"children":1558},{"className":1557},[],[1559],{"type":55,"value":1560},"slurmctld",{"type":55,"value":1562}," in-memory state can desync from the on-disk state. A controller restart reloads from ",{"type":50,"tag":101,"props":1564,"children":1566},{"className":1565},[],[1567],{"type":55,"value":1568},"StateSaveLocation",{"type":55,"value":1570}," and clears bad caches. User decides and executes.",{"type":50,"tag":58,"props":1572,"children":1573},{},[1574],{"type":55,"value":1575},"Restart may help:",{"type":50,"tag":668,"props":1577,"children":1578},{},[1579,1595],{"type":50,"tag":672,"props":1580,"children":1581},{},[1582],{"type":50,"tag":676,"props":1583,"children":1584},{},[1585,1590],{"type":50,"tag":680,"props":1586,"children":1587},{},[1588],{"type":55,"value":1589},"Symptom",{"type":50,"tag":680,"props":1591,"children":1592},{},[1593],{"type":55,"value":1594},"Why",{"type":50,"tag":699,"props":1596,"children":1597},{},[1598,1622,1640,1653,1681,1699],{"type":50,"tag":676,"props":1599,"children":1600},{},[1601,1617],{"type":50,"tag":706,"props":1602,"children":1603},{},[1604,1609,1610,1615],{"type":50,"tag":101,"props":1605,"children":1607},{"className":1606},[],[1608],{"type":55,"value":777},{"type":55,"value":779},{"type":50,"tag":101,"props":1611,"children":1613},{"className":1612},[],[1614],{"type":55,"value":785},{"type":55,"value":1616},", idle nodes",{"type":50,"tag":706,"props":1618,"children":1619},{},[1620],{"type":55,"value":1621},"Re-evaluates the queue",{"type":50,"tag":676,"props":1623,"children":1624},{},[1625,1635],{"type":50,"tag":706,"props":1626,"children":1627},{},[1628,1629,1634],{"type":55,"value":804},{"type":50,"tag":101,"props":1630,"children":1632},{"className":1631},[],[1633],{"type":55,"value":810},{"type":55,"value":812},{"type":50,"tag":706,"props":1636,"children":1637},{},[1638],{"type":55,"value":1639},"Controller held a reference to the old node",{"type":50,"tag":676,"props":1641,"children":1642},{},[1643,1648],{"type":50,"tag":706,"props":1644,"children":1645},{},[1646],{"type":55,"value":1647},"GRES (GPU, EFA) not released after a job ends",{"type":50,"tag":706,"props":1649,"children":1650},{},[1651],{"type":55,"value":1652},"Resource accounting de-synced",{"type":50,"tag":676,"props":1654,"children":1655},{},[1656,1676],{"type":50,"tag":706,"props":1657,"children":1658},{},[1659,1661,1667,1669,1674],{"type":55,"value":1660},"Nodes stuck ",{"type":50,"tag":101,"props":1662,"children":1664},{"className":1663},[],[1665],{"type":55,"value":1666},"Unknown",{"type":55,"value":1668}," after reboot, ",{"type":50,"tag":101,"props":1670,"children":1672},{"className":1671},[],[1673],{"type":55,"value":1383},{"type":55,"value":1675}," is up",{"type":50,"tag":706,"props":1677,"children":1678},{},[1679],{"type":55,"value":1680},"Re-registration was not processed",{"type":50,"tag":676,"props":1682,"children":1683},{},[1684,1694],{"type":50,"tag":706,"props":1685,"children":1686},{},[1687,1692],{"type":50,"tag":101,"props":1688,"children":1690},{"className":1689},[],[1691],{"type":55,"value":831},{"type":55,"value":1693}," times out",{"type":50,"tag":706,"props":1695,"children":1696},{},[1697],{"type":55,"value":1698},"Controller event loop is hung",{"type":50,"tag":676,"props":1700,"children":1701},{},[1702,1715],{"type":50,"tag":706,"props":1703,"children":1704},{},[1705,1707,1713],{"type":55,"value":1706},"Lost connection to ",{"type":50,"tag":101,"props":1708,"children":1710},{"className":1709},[],[1711],{"type":55,"value":1712},"slurmdbd",{"type":55,"value":1714}," \u002F RPC errors",{"type":50,"tag":706,"props":1716,"children":1717},{},[1718],{"type":55,"value":1719},"DBD connection wedged",{"type":50,"tag":58,"props":1721,"children":1722},{},[1723],{"type":55,"value":1724},"Do NOT restart when:",{"type":50,"tag":91,"props":1726,"children":1727},{},[1728,1741,1753,1771,1782],{"type":50,"tag":95,"props":1729,"children":1730},{},[1731,1733,1739],{"type":55,"value":1732},"HyperPod replacement (",{"type":50,"tag":101,"props":1734,"children":1736},{"className":1735},[],[1737],{"type":55,"value":1738},"Action:Replace",{"type":55,"value":1740},") in progress on any node — concurrent changes\nfail the replacement.",{"type":50,"tag":95,"props":1742,"children":1743},{},[1744,1746,1751],{"type":55,"value":1745},"Only one compute node is bad — restart ",{"type":50,"tag":101,"props":1747,"children":1749},{"className":1748},[],[1750],{"type":55,"value":1383},{"type":55,"value":1752}," on that node.",{"type":50,"tag":95,"props":1754,"children":1755},{},[1756,1761,1763,1769],{"type":50,"tag":101,"props":1757,"children":1759},{"className":1758},[],[1760],{"type":55,"value":1278},{"type":55,"value":1762}," and ",{"type":50,"tag":101,"props":1764,"children":1766},{"className":1765},[],[1767],{"type":55,"value":1768},"squeue",{"type":55,"value":1770}," are responsive — problem is elsewhere.",{"type":50,"tag":95,"props":1772,"children":1773},{},[1774,1780],{"type":50,"tag":101,"props":1775,"children":1777},{"className":1776},[],[1778],{"type":55,"value":1779},"journalctl -u slurmctld",{"type":55,"value":1781}," not reviewed yet — panic \u002F OOM will reproduce.",{"type":50,"tag":95,"props":1783,"children":1784},{},[1785,1790,1792,1798],{"type":50,"tag":101,"props":1786,"children":1788},{"className":1787},[],[1789],{"type":55,"value":921},{"type":55,"value":1791}," was just edited — try ",{"type":50,"tag":101,"props":1793,"children":1795},{"className":1794},[],[1796],{"type":55,"value":1797},"scontrol reconfigure",{"type":55,"value":1799}," first.",{"type":50,"tag":312,"props":1801,"children":1803},{"id":1802},"folded-triggers",[1804],{"type":55,"value":1805},"Folded triggers",{"type":50,"tag":91,"props":1807,"children":1808},{},[1809,1865],{"type":50,"tag":95,"props":1810,"children":1811},{},[1812,1818,1820,1826,1828,1833,1835,1841,1843,1848,1850,1856,1858,1864],{"type":50,"tag":1813,"props":1814,"children":1815},"strong",{},[1816],{"type":55,"value":1817},"slurmdbd disconnected",{"type":55,"value":1819}," — ",{"type":50,"tag":101,"props":1821,"children":1823},{"className":1822},[],[1824],{"type":55,"value":1825},"sacct",{"type":55,"value":1827}," fails, accounting fields show ",{"type":50,"tag":101,"props":1829,"children":1831},{"className":1830},[],[1832],{"type":55,"value":1666},{"type":55,"value":1834},",\ncontroller log spams ",{"type":50,"tag":101,"props":1836,"children":1838},{"className":1837},[],[1839],{"type":55,"value":1840},"Unable to contact slurmdbd",{"type":55,"value":1842},". Restore ",{"type":50,"tag":101,"props":1844,"children":1846},{"className":1845},[],[1847],{"type":55,"value":1712},{"type":55,"value":1849}," before\nconsidering controller restart.\n",{"type":50,"tag":76,"props":1851,"children":1854},{"href":1852,"rel":1853},"https:\u002F\u002Fslurm.schedmd.com\u002Faccounting.html",[246],[1855],{"type":55,"value":1852},{"type":55,"value":1857}," ·\n",{"type":50,"tag":76,"props":1859,"children":1861},{"href":1860},"references\u002Fslurm-details.md#slurmdbd-connectivity",[1862],{"type":55,"value":1863},"details",{"type":55,"value":83},{"type":50,"tag":95,"props":1866,"children":1867},{},[1868,1873,1874,1879,1880,1886,1888,1893,1895,1900,1901,1906],{"type":50,"tag":1813,"props":1869,"children":1870},{},[1871],{"type":55,"value":1872},"Stale config",{"type":55,"value":1819},{"type":50,"tag":101,"props":1875,"children":1877},{"className":1876},[],[1878],{"type":55,"value":921},{"type":55,"value":1054},{"type":50,"tag":101,"props":1881,"children":1883},{"className":1882},[],[1884],{"type":55,"value":1885},"topology.conf",{"type":55,"value":1887}," mtime > slurmctld start.\n",{"type":50,"tag":101,"props":1889,"children":1891},{"className":1890},[],[1892],{"type":55,"value":1797},{"type":55,"value":1894}," first; restart is fallback.\n",{"type":50,"tag":76,"props":1896,"children":1898},{"href":1508,"rel":1897},[246],[1899],{"type":55,"value":1508},{"type":55,"value":1857},{"type":50,"tag":76,"props":1902,"children":1904},{"href":1903},"references\u002Fslurm-details.md#scontrol-reconfigure-vs-restart",[1905],{"type":55,"value":1863},{"type":55,"value":83},{"type":50,"tag":58,"props":1908,"children":1909},{},[1910],{"type":55,"value":1911},"Restart procedure \u002F what's preserved:",{"type":50,"tag":91,"props":1913,"children":1914},{},[1915,1924],{"type":50,"tag":95,"props":1916,"children":1917},{},[1918],{"type":50,"tag":76,"props":1919,"children":1922},{"href":1920,"rel":1921},"https:\u002F\u002Fslurm.schedmd.com\u002Fslurmctld.html",[246],[1923],{"type":55,"value":1920},{"type":50,"tag":95,"props":1925,"children":1926},{},[1927],{"type":50,"tag":76,"props":1928,"children":1930},{"href":1420,"rel":1929},[246],[1931],{"type":55,"value":1420},{"type":50,"tag":58,"props":1933,"children":1934},{},[1935,1936,1942],{"type":55,"value":1446},{"type":50,"tag":76,"props":1937,"children":1939},{"href":1938},"references\u002Fslurm-details.md#-c-controller-state--diagnostic-context",[1940],{"type":55,"value":1941},"references\u002Fslurm-details.md § C",{"type":55,"value":83},{"type":50,"tag":1358,"props":1944,"children":1945},{},[],{"type":50,"tag":64,"props":1947,"children":1949},{"id":1948},"d-action-reason-mismatch",[1950],{"type":55,"value":886},{"type":50,"tag":58,"props":1952,"children":1953},{},[1954,1960,1962,1968,1970,1976,1977,1982,1984,1989],{"type":50,"tag":101,"props":1955,"children":1957},{"className":1956},[],[1958],{"type":55,"value":1959},"scontrol update state=fail reason=...",{"type":55,"value":1961}," was issued with a ",{"type":50,"tag":101,"props":1963,"children":1965},{"className":1964},[],[1966],{"type":55,"value":1967},"reason",{"type":55,"value":1969}," that does not match\n",{"type":50,"tag":101,"props":1971,"children":1973},{"className":1972},[],[1974],{"type":55,"value":1975},"Action:Reboot",{"type":55,"value":116},{"type":50,"tag":101,"props":1978,"children":1980},{"className":1979},[],[1981],{"type":55,"value":1738},{"type":55,"value":1983}," exactly. HyperPod silently ignores anything else.\nScript detects near-misses on nodes in ",{"type":50,"tag":101,"props":1985,"children":1987},{"className":1986},[],[1988],{"type":55,"value":1067},{"type":55,"value":1990}," state.",{"type":50,"tag":58,"props":1992,"children":1993},{},[1994],{"type":55,"value":1995},"Required strings (case-sensitive, no whitespace, no punctuation):",{"type":50,"tag":91,"props":1997,"children":1998},{},[1999,2007],{"type":50,"tag":95,"props":2000,"children":2001},{},[2002],{"type":50,"tag":101,"props":2003,"children":2005},{"className":2004},[],[2006],{"type":55,"value":1975},{"type":50,"tag":95,"props":2008,"children":2009},{},[2010],{"type":50,"tag":101,"props":2011,"children":2013},{"className":2012},[],[2014],{"type":55,"value":1738},{"type":50,"tag":58,"props":2016,"children":2017},{},[2018,2019],{"type":55,"value":1417},{"type":50,"tag":76,"props":2020,"children":2023},{"href":2021,"rel":2022},"https:\u002F\u002Fdocs.aws.amazon.com\u002Fsagemaker\u002Flatest\u002Fdg\u002Fsagemaker-hyperpod-resiliency-slurm-replace-faulty-instance.html",[246],[2024],{"type":55,"value":2021},{"type":50,"tag":58,"props":2026,"children":2027},{},[2028,2029,2035],{"type":55,"value":1446},{"type":50,"tag":76,"props":2030,"children":2032},{"href":2031},"references\u002Fslurm-details.md#action-reason-string-validation",[2033],{"type":55,"value":2034},"references\u002Fslurm-details.md § Action reason-string validation",{"type":55,"value":83},{"type":50,"tag":1358,"props":2037,"children":2038},{},[],{"type":50,"tag":64,"props":2040,"children":2042},{"id":2041},"e-auto-resume",[2043],{"type":55,"value":949},{"type":50,"tag":58,"props":2045,"children":2046},{},[2047,2053,2055,2061],{"type":50,"tag":101,"props":2048,"children":2050},{"className":2049},[],[2051],{"type":55,"value":2052},"--auto-resume=1",{"type":55,"value":2054}," is an ",{"type":50,"tag":101,"props":2056,"children":2058},{"className":2057},[],[2059],{"type":55,"value":2060},"srun",{"type":55,"value":2062}," step option. It re-runs the step after HMA (the Health\nMonitoring Agent) flags a node and Automatic node recovery replaces it.",{"type":50,"tag":58,"props":2064,"children":2065},{},[2066],{"type":55,"value":2067},"Why it didn't restart the job:",{"type":50,"tag":91,"props":2069,"children":2070},{},[2071,2098,2103,2124,2129],{"type":50,"tag":95,"props":2072,"children":2073},{},[2074,2076,2082,2084,2089,2091,2096],{"type":55,"value":2075},"Flag on ",{"type":50,"tag":101,"props":2077,"children":2079},{"className":2078},[],[2080],{"type":55,"value":2081},"sbatch",{"type":55,"value":2083}," not ",{"type":50,"tag":101,"props":2085,"children":2087},{"className":2086},[],[2088],{"type":55,"value":2060},{"type":55,"value":2090}," — per-step; ",{"type":50,"tag":101,"props":2092,"children":2094},{"className":2093},[],[2095],{"type":55,"value":2081},{"type":55,"value":2097}," directives are silently ignored.",{"type":50,"tag":95,"props":2099,"children":2100},{},[2101],{"type":55,"value":2102},"HMA did not flag the node — failure was application\u002Ftransient, not hardware. Step\nexits as a normal Slurm failure.",{"type":50,"tag":95,"props":2104,"children":2105},{},[2106,2108,2114,2116,2122],{"type":55,"value":2107},"Cluster ",{"type":50,"tag":101,"props":2109,"children":2111},{"className":2110},[],[2112],{"type":55,"value":2113},"NodeRecovery",{"type":55,"value":2115}," is ",{"type":50,"tag":101,"props":2117,"children":2119},{"className":2118},[],[2120],{"type":55,"value":2121},"None",{"type":55,"value":2123}," — faulty nodes are labeled but not replaced.",{"type":50,"tag":95,"props":2125,"children":2126},{},[2127],{"type":55,"value":2128},"No checkpointing — step restarts from process zero each iteration.",{"type":50,"tag":95,"props":2130,"children":2131},{},[2132],{"type":55,"value":2133},"AMI predates HMA support (released 2025-09-11) — needs AMI \u002F cluster-software update.",{"type":50,"tag":58,"props":2135,"children":2136},{},[2137,2138],{"type":55,"value":1417},{"type":50,"tag":76,"props":2139,"children":2142},{"href":2140,"rel":2141},"https:\u002F\u002Fdocs.aws.amazon.com\u002Fsagemaker\u002Flatest\u002Fdg\u002Fsagemaker-hyperpod-resiliency-slurm-auto-resume.html",[246],[2143],{"type":55,"value":2140},{"type":50,"tag":58,"props":2145,"children":2146},{},[2147,2148,2154],{"type":55,"value":1446},{"type":50,"tag":76,"props":2149,"children":2151},{"href":2150},"references\u002Fslurm-details.md#hyperpod-auto-resume",[2152],{"type":55,"value":2153},"references\u002Fslurm-details.md § HyperPod auto-resume",{"type":55,"value":83},{"type":50,"tag":1358,"props":2156,"children":2157},{},[],{"type":50,"tag":64,"props":2159,"children":2161},{"id":2160},"escalation",[2162],{"type":55,"value":2163},"Escalation",{"type":50,"tag":668,"props":2165,"children":2166},{},[2167,2183],{"type":50,"tag":672,"props":2168,"children":2169},{},[2170],{"type":50,"tag":676,"props":2171,"children":2172},{},[2173,2178],{"type":50,"tag":680,"props":2174,"children":2175},{},[2176],{"type":55,"value":2177},"Condition",{"type":50,"tag":680,"props":2179,"children":2180},{},[2181],{"type":55,"value":2182},"Next skill",{"type":50,"tag":699,"props":2184,"children":2185},{},[2186,2211,2234,2260,2276],{"type":50,"tag":676,"props":2187,"children":2188},{},[2189,2201],{"type":50,"tag":706,"props":2190,"children":2191},{},[2192,2194,2199],{"type":55,"value":2193},"Node returns to ",{"type":50,"tag":101,"props":2195,"children":2197},{"className":2196},[],[2198],{"type":55,"value":716},{"type":55,"value":2200}," shortly after a manual resume",{"type":50,"tag":706,"props":2202,"children":2203},{},[2204,2209],{"type":50,"tag":101,"props":2205,"children":2207},{"className":2206},[],[2208],{"type":55,"value":114},{"type":55,"value":2210}," (hardware)",{"type":50,"tag":676,"props":2212,"children":2213},{},[2214,2224],{"type":50,"tag":706,"props":2215,"children":2216},{},[2217,2222],{"type":50,"tag":101,"props":2218,"children":2220},{"className":2219},[],[2221],{"type":55,"value":1383},{"type":55,"value":2223}," logs contain CUDA \u002F NVIDIA \u002F XID errors",{"type":50,"tag":706,"props":2225,"children":2226},{},[2227,2232],{"type":50,"tag":101,"props":2228,"children":2230},{"className":2229},[],[2231],{"type":55,"value":114},{"type":55,"value":2233}," § G",{"type":50,"tag":676,"props":2235,"children":2236},{},[2237,2250],{"type":50,"tag":706,"props":2238,"children":2239},{},[2240,2242,2248],{"type":55,"value":2241},"Disk full or ",{"type":50,"tag":101,"props":2243,"children":2245},{"className":2244},[],[2246],{"type":55,"value":2247},"\u002Fdev\u002Fshm",{"type":55,"value":2249}," exhausted",{"type":50,"tag":706,"props":2251,"children":2252},{},[2253,2258],{"type":50,"tag":101,"props":2254,"children":2256},{"className":2255},[],[2257],{"type":55,"value":114},{"type":55,"value":2259}," § I",{"type":50,"tag":676,"props":2261,"children":2262},{},[2263,2268],{"type":50,"tag":706,"props":2264,"children":2265},{},[2266],{"type":55,"value":2267},"Node unreachable via SSM",{"type":50,"tag":706,"props":2269,"children":2270},{},[2271],{"type":50,"tag":101,"props":2272,"children":2274},{"className":2273},[],[2275],{"type":55,"value":156},{"type":50,"tag":676,"props":2277,"children":2278},{},[2279,2291],{"type":50,"tag":706,"props":2280,"children":2281},{},[2282,2284,2289],{"type":55,"value":2283},"Controller restart does not clear ",{"type":50,"tag":101,"props":2285,"children":2287},{"className":2286},[],[2288],{"type":55,"value":810},{"type":55,"value":2290}," after 2 attempts",{"type":50,"tag":706,"props":2292,"children":2293},{},[2294,2300],{"type":50,"tag":101,"props":2295,"children":2297},{"className":2296},[],[2298],{"type":55,"value":2299},"hyperpod-issue-report",{"type":55,"value":2301}," + AWS Support",{"type":50,"tag":2303,"props":2304,"children":2305},"style",{},[2306],{"type":55,"value":2307},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2309,"total":2485},[2310,2327,2348,2358,2371,2384,2394,2404,2425,2440,2455,2470],{"slug":2311,"name":2311,"fn":2312,"description":2313,"org":2314,"tags":2315,"stars":2324,"repoUrl":2325,"updatedAt":2326},"agentcore-investigation","investigate Bedrock AgentCore runtime sessions","Investigate Bedrock AgentCore runtime sessions via CloudWatch Logs Insights — resolve session\u002Ftrace IDs, query OTEL spans, filter noise, build timelines. Use when debugging AgentCore agent sessions, tracing tool calls, or analyzing latency.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2316,2317,2318,2321],{"name":21,"slug":22,"type":16},{"name":24,"slug":25,"type":16},{"name":2319,"slug":2320,"type":16},"Logs","logs",{"name":2322,"slug":2323,"type":16},"Observability","observability",9427,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fmcp","2026-07-12T08:37:22.601527",{"slug":2328,"name":2329,"fn":2330,"description":2331,"org":2332,"tags":2333,"stars":2324,"repoUrl":2325,"updatedAt":2347},"amazon-aurora-dsql","amazon aurora dsql","build applications with Aurora DSQL","Build with Aurora DSQL — manage schemas, execute queries, handle migrations, diagnose query plans, load data, and develop applications with a serverless, distributed SQL database. Covers IAM auth, multi-tenant patterns, MySQL-to-DSQL and PostgreSQL-to-DSQL schema conversion, FK replacement code generation, OCC retry patterns, ORM migration (Django\u002FHibernate\u002FRails), DDL operations, query plan explainability, SQL compatibility validation, and bulk data loading. Triggers on phrases like: DSQL, Aurora DSQL, create DSQL table, DSQL schema, migrate to DSQL, distributed SQL database, serverless PostgreSQL-compatible database, DSQL query plan, DSQL EXPLAIN ANALYZE, why is my DSQL query slow, DSQL foreign key, DSQL OCC retry, DSQL multi-region, load into DSQL, load CSV into DSQL, bulk load DSQL, aurora-dsql-loader.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2334,2337,2338,2341,2344],{"name":2335,"slug":2336,"type":16},"Aurora","aurora",{"name":21,"slug":22,"type":16},{"name":2339,"slug":2340,"type":16},"Database","database",{"name":2342,"slug":2343,"type":16},"Serverless","serverless",{"name":2345,"slug":2346,"type":16},"SQL","sql","2026-07-12T08:36:45.053393",{"slug":2349,"name":2350,"fn":2330,"description":2331,"org":2351,"tags":2352,"stars":2324,"repoUrl":2325,"updatedAt":2357},"aurora-dsql","aurora dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2353,2354,2355,2356],{"name":21,"slug":22,"type":16},{"name":2339,"slug":2340,"type":16},{"name":2342,"slug":2343,"type":16},{"name":2345,"slug":2346,"type":16},"2026-07-12T08:36:42.694299",{"slug":2359,"name":2360,"fn":2330,"description":2331,"org":2361,"tags":2362,"stars":2324,"repoUrl":2325,"updatedAt":2370},"aws-dsql","aws dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2363,2364,2365,2368,2369],{"name":21,"slug":22,"type":16},{"name":2339,"slug":2340,"type":16},{"name":2366,"slug":2367,"type":16},"Migration","migration",{"name":2342,"slug":2343,"type":16},{"name":2345,"slug":2346,"type":16},"2026-07-12T08:36:38.584057",{"slug":2372,"name":2373,"fn":2330,"description":2331,"org":2374,"tags":2375,"stars":2324,"repoUrl":2325,"updatedAt":2383},"distributed-postgres","distributed postgres",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2376,2377,2378,2381,2382],{"name":21,"slug":22,"type":16},{"name":2339,"slug":2340,"type":16},{"name":2379,"slug":2380,"type":16},"PostgreSQL","postgresql",{"name":2342,"slug":2343,"type":16},{"name":2345,"slug":2346,"type":16},"2026-07-12T08:36:46.530743",{"slug":2385,"name":2386,"fn":2330,"description":2331,"org":2387,"tags":2388,"stars":2324,"repoUrl":2325,"updatedAt":2393},"distributed-sql","distributed sql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2389,2390,2391,2392],{"name":21,"slug":22,"type":16},{"name":2339,"slug":2340,"type":16},{"name":2342,"slug":2343,"type":16},{"name":2345,"slug":2346,"type":16},"2026-07-12T08:36:48.104182",{"slug":2395,"name":2395,"fn":2330,"description":2331,"org":2396,"tags":2397,"stars":2324,"repoUrl":2325,"updatedAt":2403},"dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2398,2399,2400,2401,2402],{"name":21,"slug":22,"type":16},{"name":2339,"slug":2340,"type":16},{"name":2366,"slug":2367,"type":16},{"name":2342,"slug":2343,"type":16},{"name":2345,"slug":2346,"type":16},"2026-07-12T08:36:36.374512",{"slug":2405,"name":2405,"fn":2406,"description":2407,"org":2408,"tags":2409,"stars":2422,"repoUrl":2423,"updatedAt":2424},"cost-efficiency-analyzer","analyze cost efficiency and expenses","Analyzes cost structure, cost efficiency, and expense management from P&L data. Use when the user asks about costs, expenses, COGS, operating expenses, cost ratios, cost control, spending efficiency, margin compression from cost side, or wants to understand where money is going. Also use for \"are we spending too much\", \"cost breakdown\", \"expense analysis\", or \"how efficient are our operations\". NOT for revenue or top-line analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2410,2413,2416,2419],{"name":2411,"slug":2412,"type":16},"Accounting","accounting",{"name":2414,"slug":2415,"type":16},"Analytics","analytics",{"name":2417,"slug":2418,"type":16},"Cost Optimization","cost-optimization",{"name":2420,"slug":2421,"type":16},"Finance","finance",3176,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagentcore-samples","2026-07-12T08:40:03.29555",{"slug":2426,"name":2426,"fn":2427,"description":2428,"org":2429,"tags":2430,"stars":2422,"repoUrl":2423,"updatedAt":2439},"executive-financial-briefing","generate executive financial briefings","Generates a concise executive-level financial briefing or summary suitable for a CEO, CFO, or board presentation. Use when the user asks for a summary, briefing, executive summary, board update, financial overview, financial health check, or \"how is the business doing\". Covers the full P&L picture in one page. Also use for \"give me the highlights\", \"what do I need to know\", or \"quick financial update\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2431,2432,2433,2436],{"name":21,"slug":22,"type":16},{"name":2420,"slug":2421,"type":16},{"name":2434,"slug":2435,"type":16},"Management","management",{"name":2437,"slug":2438,"type":16},"Reporting","reporting","2026-07-12T08:40:02.066471",{"slug":2441,"name":2441,"fn":2442,"description":2443,"org":2444,"tags":2445,"stars":2422,"repoUrl":2423,"updatedAt":2454},"multi-quarter-trend-analysis","analyze multi-quarter financial trends","Analyzes financial trends across multiple quarters by comparing P&L metrics over time. Use when the user wants to see trends, patterns, trajectories, or directional movement across 3 or more quarters. Also use for \"how are we trending\", \"show me the trend\", \"track performance over time\", \"quarter over quarter comparison across all quarters\", or any multi-period longitudinal analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2446,2447,2448,2451],{"name":2414,"slug":2415,"type":16},{"name":2420,"slug":2421,"type":16},{"name":2449,"slug":2450,"type":16},"Financial Statements","financial-statements",{"name":2452,"slug":2453,"type":16},"Variance Analysis","variance-analysis","2026-07-12T08:40:00.79141",{"slug":2456,"name":2456,"fn":2457,"description":2458,"org":2459,"tags":2460,"stars":2422,"repoUrl":2423,"updatedAt":2469},"pdf","process and manipulate PDF documents","Use this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text\u002Ftables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting\u002Fdecrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2461,2464,2467],{"name":2462,"slug":2463,"type":16},"Automation","automation",{"name":2465,"slug":2466,"type":16},"Documents","documents",{"name":2468,"slug":2456,"type":16},"PDF","2026-07-12T08:41:44.135656",{"slug":2471,"name":2471,"fn":2472,"description":2473,"org":2474,"tags":2475,"stars":2422,"repoUrl":2423,"updatedAt":2484},"quarterly-kpi-calculator","calculate quarterly financial KPIs","Calculates quarterly financial KPIs from P&L data. P&L figures can be provided directly by the user or fetched from the financial data MCP server. Use when the user wants KPI calculations such as Gross Margin %, EBITDA Margin %, Operating Expense Ratio, or Revenue Growth % QoQ. Also use for quarterly performance review, P&L analysis, or interpreting financial ratios against benchmarks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2476,2477,2480,2481],{"name":2411,"slug":2412,"type":16},{"name":2478,"slug":2479,"type":16},"Data Analysis","data-analysis",{"name":2420,"slug":2421,"type":16},{"name":2482,"slug":2483,"type":16},"KPI","kpi","2026-07-12T08:39:59.54971",150,{"items":2487,"total":2586},[2488,2505,2523,2535,2552,2563,2573],{"slug":2489,"name":2489,"fn":2490,"description":2491,"org":2492,"tags":2493,"stars":26,"repoUrl":27,"updatedAt":2504},"amazon-location-service","integrate Amazon Location Service maps","Integrates Amazon Location Service APIs for AWS applications. Use this skill when users want to add maps (interactive MapLibre or static images); geocode addresses to coordinates or reverse geocode coordinates to addresses; calculate routes, travel times, or service areas; find places and businesses through text search, nearby search, or autocomplete suggestions; retrieve detailed place information including hours, contacts, and addresses; monitor geographical boundaries with geofences; or track device locations. Covers authentication, SDK integration, and all Amazon Location Service capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2494,2497,2498,2501],{"name":2495,"slug":2496,"type":16},"API Development","api-development",{"name":21,"slug":22,"type":16},{"name":2499,"slug":2500,"type":16},"Maps","maps",{"name":2502,"slug":2503,"type":16},"Navigation","navigation","2026-07-12T08:39:49.88311",{"slug":2506,"name":2506,"fn":2507,"description":2508,"org":2509,"tags":2510,"stars":26,"repoUrl":27,"updatedAt":2522},"amplify-workflow","build and deploy apps with AWS Amplify","Build and deploy full-stack web and mobile apps with AWS Amplify Gen2 (TypeScript code-first). Covers auth (Cognito), data (AppSync\u002FDynamoDB including schema modeling, enum types, relationships, authorization rules), storage (S3), functions, APIs, and AI (Amplify AI Kit with Bedrock). Supports React, Next.js, Vue, Angular, React Native, Flutter, Swift, and Android. Always use this skill for Amplify Gen2 topics — even for questions you think you know — it contains validated, version-specific patterns that prevent common mistakes. TRIGGER when: user mentions Amplify Gen2; project has amplify\u002F directory or amplify_outputs; code imports @aws-amplify packages; user asks about defineBackend, defineAuth, defineData, defineStorage, or npx ampx. SKIP: Amplify Gen1 (amplify CLI v6), standalone SAM\u002FCDK without Amplify (use aws-serverless), direct Bedrock without Amplify AI Kit (use bedrock).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2511,2514,2515,2516,2519],{"name":2512,"slug":2513,"type":16},"Auth","auth",{"name":21,"slug":22,"type":16},{"name":2339,"slug":2340,"type":16},{"name":2517,"slug":2518,"type":16},"Deployment","deployment",{"name":2520,"slug":2521,"type":16},"TypeScript","typescript","2026-07-12T08:39:43.500162",{"slug":2524,"name":2524,"fn":2525,"description":2526,"org":2527,"tags":2528,"stars":26,"repoUrl":27,"updatedAt":2534},"api-gateway","build and manage Amazon API Gateway APIs","Build, manage, and operate APIs with Amazon API Gateway (REST, HTTP, and WebSocket). Triggers on phrases like: API Gateway, REST API, HTTP API, WebSocket API, custom domain, Lambda authorizer, usage plan, throttling, CORS, VPC link, private API. Also covers troubleshooting API Gateway errors (4xx, 5xx, timeout, CORS failures) and IaC templates containing API Gateway resources. For general REST API design unrelated to AWS, do not trigger.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2529,2530,2531],{"name":2495,"slug":2496,"type":16},{"name":21,"slug":22,"type":16},{"name":2532,"slug":2533,"type":16},"REST API","rest-api","2026-07-12T08:39:00.149339",{"slug":2536,"name":2536,"fn":2537,"description":2538,"org":2539,"tags":2540,"stars":26,"repoUrl":27,"updatedAt":2551},"aws-architecture-diagram","generate AWS architecture diagrams","Generate validated AWS architecture diagrams as draw.io XML using official AWS4 icon libraries. Use this skill whenever the user wants to create, generate, or design AWS architecture diagrams, cloud infrastructure diagrams, or system design visuals. Also triggers for requests to visualize existing infrastructure from CloudFormation, CDK, or Terraform code. Supports two modes: analyze an existing codebase to auto-generate diagrams, or brainstorm interactively from scratch. Exports .drawio files with optional PNG\u002FSVG\u002FPDF export via draw.io desktop CLI.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2541,2544,2545,2548],{"name":2542,"slug":2543,"type":16},"Architecture","architecture",{"name":21,"slug":22,"type":16},{"name":2546,"slug":2547,"type":16},"Design","design",{"name":2549,"slug":2550,"type":16},"Diagrams","diagrams","2026-07-12T08:37:11.012278",{"slug":2553,"name":2553,"fn":2554,"description":2555,"org":2556,"tags":2557,"stars":26,"repoUrl":27,"updatedAt":2562},"aws-lambda","build and deploy AWS Lambda functions","Design, build, deploy, test, and debug serverless applications with AWS Lambda. Triggers on phrases like: Lambda function, event source, serverless application, API Gateway, EventBridge, Step Functions, serverless API, event-driven architecture, Lambda trigger. For deploying non-serverless apps to AWS, use deploy-on-aws plugin instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2558,2559,2560,2561],{"name":2495,"slug":2496,"type":16},{"name":21,"slug":22,"type":16},{"name":2517,"slug":2518,"type":16},{"name":2342,"slug":2343,"type":16},"2026-07-12T08:38:58.598492",{"slug":2564,"name":2564,"fn":2565,"description":2566,"org":2567,"tags":2568,"stars":26,"repoUrl":27,"updatedAt":2572},"aws-lambda-durable-functions","build resilient AWS Lambda durable functions","Build resilient, long-running, multi-step applications with AWS Lambda durable functions with automatic state persistence, retry logic, and orchestration for long-running executions. Covers the critical replay model, step operations, wait\u002Fcallback patterns, error handling with saga pattern, testing with LocalDurableTestRunner. Triggers on phrases like: lambda durable functions, workflow orchestration, state machines, retry\u002Fcheckpoint patterns, long-running stateful Lambda functions, saga pattern, human-in-the-loop callbacks, and reliable serverless applications.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2569,2570,2571],{"name":2542,"slug":2543,"type":16},{"name":21,"slug":22,"type":16},{"name":2342,"slug":2343,"type":16},"2026-07-12T08:39:05.546173",{"slug":2574,"name":2574,"fn":2575,"description":2576,"org":2577,"tags":2578,"stars":26,"repoUrl":27,"updatedAt":2585},"aws-lambda-managed-instances","configure AWS Lambda Managed Instances","Evaluate, configure, and migrate workloads to AWS Lambda Managed Instances (LMI). Triggers on: Lambda Managed Instances, LMI, capacity provider, multi-concurrency Lambda, dedicated instance Lambda, EC2-backed Lambda, cold start elimination, Graviton Lambda, instance type for Lambda, scheduled scaling for LMI, Lambda cost optimization with Reserved Instances or Savings Plans. Also trigger when users describe high-volume predictable workloads seeking cost savings, want to scale LMI capacity on a schedule, or compare Lambda vs EC2 for steady-state traffic. For standard Lambda without LMI, use the aws-lambda skill instead.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2579,2580,2581,2582],{"name":21,"slug":22,"type":16},{"name":2517,"slug":2518,"type":16},{"name":18,"slug":19,"type":16},{"name":2583,"slug":2584,"type":16},"Performance","performance","2026-07-12T08:39:07.007071",33]