[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-aws-labs-hyperpod-cluster-debugger":3,"mdc-2viunz-key":45,"related-org-aws-labs-hyperpod-cluster-debugger":2229,"related-repo-aws-labs-hyperpod-cluster-debugger":2407},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":29,"repoUrl":30,"updatedAt":31,"license":32,"forks":33,"topics":34,"repo":40,"sourceUrl":43,"mdContent":44},"hyperpod-cluster-debugger","diagnose and remediate HyperPod cluster failures","Diagnose and remediate cluster-wide HyperPod (EKS or Slurm) problems — creation \u002F deployment failures (CloudFormation, EFA health check, lifecycle scripts, capacity), EKS access, node replacement, CloudFormation nested-stack errors, post-maintenance rollback state, dangling nodes, autoscaler conflicts. Includes `--validate` pre-flight. Read-only.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"aws-labs","AWS Labs","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Faws-labs.png","awslabs",[13,17,20,23,26],{"name":14,"slug":15,"type":16},"Operations","operations","tag",{"name":18,"slug":19,"type":16},"Kubernetes","kubernetes",{"name":21,"slug":22,"type":16},"Infrastructure","infrastructure",{"name":24,"slug":25,"type":16},"AWS","aws",{"name":27,"slug":28,"type":16},"Debugging","debugging",831,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagent-plugins","2026-07-12T08:39:33.335106",null,127,[35,36,37,25,38,39],"agent-plugins","agent-skills","agents","coding-agent-skills","coding-agents",{"repoUrl":30,"stars":29,"forks":33,"topics":41,"description":42},[35,36,37,25,38,39],"Agent Plugins for AWS equip AI coding agents with the skills to help you architect, deploy, and operate on AWS.","https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagent-plugins\u002Ftree\u002FHEAD\u002Fplugins\u002Fsagemaker-ai\u002Fskills\u002Fhyperpod-cluster-debugger","---\nname: hyperpod-cluster-debugger\ndescription: Diagnose and remediate cluster-wide HyperPod (EKS or Slurm) problems — creation \u002F deployment failures (CloudFormation, EFA health check, lifecycle scripts, capacity), EKS access, node replacement, CloudFormation nested-stack errors, post-maintenance rollback state, dangling nodes, autoscaler conflicts. Includes `--validate` pre-flight. Read-only.\nmetadata:\n  version: \"0.0.1\"\n---\n\n# HyperPod Cluster Debugger\n\n**Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a **Suggested command (run this yourself)** block and wait for the customer to run it. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes).\n\n**Before any state-changing CLI: ask if it's IaC-managed.** HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation \u002F CDK \u002F Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review).\n\n`scripts\u002Fdiagnose-cluster.sh` is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as `[FAIL] ... → references\u002F\u003Cfile>.md § \u003Csection>`.\n\n| Reference                                                                 | Open when                                                           |\n| ------------------------------------------------------------------------- | ------------------------------------------------------------------- |\n| [cluster-diagnostics-detail.md](references\u002Fcluster-diagnostics-detail.md) | Per-finding remediation runbook (§ A–L)                             |\n| [cluster-operations.md](references\u002Fcluster-operations.md)                 | Operational deep-dives (EFA SG, EKS access, SSM, Slurm, filesystem) |\n| [cloudformation-errors.md](references\u002Fcloudformation-errors.md)           | § H needs the full per-resource CFN error catalog                   |\n| [capacity-planning.md](references\u002Fcapacity-planning.md)                   | § B or `--validate` flags capacity \u002F subnet sizing                  |\n| [lifecycle-scripts.md](references\u002Flifecycle-scripts.md)                   | § C points at a specific lifecycle failure                          |\n| [iam-permissions.md](references\u002Fiam-permissions.md)                       | Full IAM policy for the diagnostic                                  |\n\n---\n\n## Workflow\n\n1. Collect HyperPod cluster name (not EKS name), region, exact error string.\n2. Run `scripts\u002Fdiagnose-cluster.sh` (or `--validate` for pre-create).\n3. For every `[FAIL]` line, `Read` the referenced section.\n4. Present finding, root cause, and the Suggested-command block verbatim. Wait for customer approval.\n5. Re-run the diagnostic to confirm.\n\n---\n\n## Step 1: Run diagnostics\n\n```bash\n# Diagnose an existing cluster:\nbash scripts\u002Fdiagnose-cluster.sh --cluster \u003CCLUSTER_NAME_OR_ARN> --region \u003CREGION>\n\n# Pre-flight (no cluster needed) — validates SGs, subnets, IAM, VPC endpoints,\n# optionally S3 lifecycle scripts and per-AZ capacity:\nbash scripts\u002Fdiagnose-cluster.sh --validate --region \u003CREGION> \\\n  --sg-ids \u003Csg-1,sg-2> --subnet-ids \u003Csub-1,sub-2> [--iam-role \u003Crole-arn>] \\\n  [--s3-uri s3:\u002F\u002F\u003CBUCKET>\u002Fpath\u002F] [--instance-type ml.p5.48xlarge]\n```\n\nPass `--instance-type` when the target instance type is known — enables the per-AZ capacity check (warns if none of the provided subnets are in an AZ that offers that type, which causes insufficient-capacity failures at creation time).\n\nTags: `[PASS]` · `[FAIL]` (counted, has `→ references\u002F...` pointer) · `[WARN]` · `[INFO]`. Priorities: **P0** blocks operation · **P1** degraded · **P2** informational.\n\n---\n\n## Step 2: Match signal → section\n\n**Error messages \u002F events:**\n\n| Signal                                                                       | Section                                                        |\n| ---------------------------------------------------------------------------- | -------------------------------------------------------------- |\n| `\"EFA health checks did not run successfully\"` (public-doc verbatim signal)  | **[A: EFA Health Checks](#a-efa-health-checks)**               |\n| Insufficient-capacity or AZ-mismatch failure at creation                     | **[B: Capacity & AZ](#b-capacity--az)**                        |\n| Lifecycle-script failure or timeout during provisioning                      | **[C: Lifecycle Scripts](#c-lifecycle-scripts)**               |\n| kubectl auth error (server asks for credentials \u002F no API group list)         | **[D: EKS Access](#d-eks-access--kubectl)**                    |\n| `InService` but not all instances visible                                    | **[E: Cluster Provisioning](#e-cluster-provisioning)**         |\n| `\"Target is not connected\"` \u002F SSM errors                                     | **[F: SSM Connectivity](#f-ssm-connectivity)**                 |\n| Node replacement not happening \u002F `batch-replace` not working                 | **[G: Node Replacement](#g-node-replacement)**                 |\n| `\"Embedded stack failed\"` \u002F any CloudFormation error                         | **[H: CloudFormation Errors](#h-cloudformation-errors)**       |\n| `UpdateClusterSoftware` failed or cluster in post-maintenance rollback state | **[J: AMI & Cluster Updates](#j-ami--cluster-updates)**        |\n| Dangling \u002F orphaned nodes in EKS vs `list-cluster-nodes`                     | **[K: Dangling Nodes & Cleanup](#k-dangling-nodes--cleanup)**  |\n| Cluster Autoscaler breaks after HyperPod attached                            | **[L: Autoscaler Compatibility](#l-autoscaler-compatibility)** |\n| Slow I\u002FO, FSx throughput saturated                                           | [cluster-operations.md § 9](references\u002Fcluster-operations.md)  |\n| Slurm node name → instance ID lookup                                         | **[I: Utilities](#i-utilities)**                               |\n\n---\n\n## A: EFA Health Checks\n\nSG missing self-reference. Add inbound + outbound self-ref to every SG on the cluster, plus least-privilege egress for the AWS APIs the node needs (HTTPS 443 to S3 \u002F ECR \u002F SageMaker \u002F SSM \u002F STS \u002F CloudWatch Logs — via VPC-endpoint prefix-lists when possible). Full procedure: [cluster-diagnostics-detail.md § A](references\u002Fcluster-diagnostics-detail.md#a-efa-health-checks).\n\n## B: Capacity & AZ\n\nInstance type unavailable in the requested AZ. Verify with `describe-instance-type-offerings`, then change AZ, use Flexible Training Plans, or request ODCR. Full: [§ B](references\u002Fcluster-diagnostics-detail.md#b-capacity--az) · strategy: [capacity-planning.md](references\u002Fcapacity-planning.md).\n\n## C: Lifecycle Scripts\n\nScript failed or timed out during provisioning. Read CloudWatch under `\u002Faws\u002Fsagemaker\u002FClusters\u002F\u003Cname>\u002F\u003Cid>` — common causes: missing S3 VPC endpoint, IAM gap, CRLF line endings, instance-group name mismatch. Full: [§ C](references\u002Fcluster-diagnostics-detail.md#c-lifecycle-scripts) · layout: [lifecycle-scripts.md](references\u002Flifecycle-scripts.md).\n\n## D: EKS Access \u002F kubectl\n\nIAM identity not in EKS access entries. Verify with `sts get-caller-identity`, create an access entry with admin policy, update kubeconfig. Full: [§ D](references\u002Fcluster-diagnostics-detail.md#d-eks-access--kubectl).\n\n## E: Cluster Provisioning\n\n`InService` without all instances is expected under Continuous Provisioning — failures surface as events, not cluster errors. For stuck `Creating`\u002F`Updating`\u002F`Deleting`: check CFN nested stacks (§ H), IAM, capacity, events; if stuck `Deleting` check VPC ENI dependencies. Full: [§ E](references\u002Fcluster-diagnostics-detail.md#e-cluster-provisioning).\n\n## F: SSM Connectivity\n\n`Target is not connected`: use `sagemaker-cluster:\u003CCLUSTER_ID>_\u003CGROUP>-\u003CINSTANCE_ID>` format (not raw EC2 ID), install session-manager-plugin, confirm node `Running`. Check IAM + VPC endpoints on timeouts. Full: [§ F](references\u002Fcluster-diagnostics-detail.md#f-ssm-connectivity).\n\n## G: Node Replacement\n\nAuto-repair: confirm `NodeRecovery=Automatic`, check Health Monitoring Agent (HMA) logs + node labels \u002F Slurm reason, confirm capacity. Manual: reboot first, replace only if reboot fails. Replace requires the cluster to have been patched via `UpdateClusterSoftware` at least once and cannot target a Slurm controller node. Full: [§ G](references\u002Fcluster-diagnostics-detail.md#g-node-replacement).\n\n## H: CloudFormation Errors\n\n`Embedded stack failed` hides the real error. Drill into nested stacks via Events tab (filter Failed) until you reach a non-stack resource. CLI: `describe-stack-events --query 'StackEvents[?ResourceStatus==\\`CREATE_FAILED\\`]'`. Also covers SLR creation failures and permission-boundary denials. Full: [§ H](references\u002Fcluster-diagnostics-detail.md#h-cloudformation-errors) · catalog: [cloudformation-errors.md](references\u002Fcloudformation-errors.md).\n\n## I: Utilities\n\nMap Slurm node names (`ip-10-x-y-z`) to HyperPod instance IDs via `list-cluster-nodes` or on-node `\u002Fopt\u002Fml\u002Fconfig\u002Fresource_config.json`. Full: [§ I](references\u002Fcluster-diagnostics-detail.md#i-utilities).\n\n## J: AMI & Cluster Updates\n\n`UpdateClusterSoftware` fails and rolls back, or the cluster stays in a post-maintenance rollback state. Common causes: lifecycle script incompatible with new AMI, HMA version too old, insufficient rolling-update capacity. If the cluster has active nodes, collect diagnostics and escalate rather than delete-and-recreate. Full: [§ J](references\u002Fcluster-diagnostics-detail.md#j-ami--cluster-updates).\n\n## K: Dangling Nodes & Cleanup\n\nNodes in `kubectl get nodes` but not in `list-cluster-nodes` (ghost EKS nodes), or the inverse (HyperPod nodes that never registered kubelet). Script flags both. Full: [§ K](references\u002Fcluster-diagnostics-detail.md#k-dangling-nodes--cleanup).\n\n## L: Autoscaler Compatibility\n\nCluster Autoscaler errors on HyperPod provider IDs and breaks autoscaling for all node groups. No officially endorsed workaround — escalate to AWS Support. Karpenter does not conflict with HyperPod nodes by default. Full: [§ L](references\u002Fcluster-diagnostics-detail.md#l-autoscaler-compatibility).\n\n---\n\n## Prerequisites\n\n- `aws` CLI v2.13+ authenticated to the cluster's account\n- `jq`, `python3`, `bash` 4.2+\n- `kubectl` authenticated to the EKS cluster (EKS checks skipped if absent)\n- `session-manager-plugin` (Slurm controller health checks only)\n\nIAM policy: [references\u002Fiam-permissions.md](references\u002Fiam-permissions.md).\n\n## Defaults\n\n- **Region** — required: pass `--region` or set `$AWS_DEFAULT_REGION`.\n- **Mode** — `--cluster \u003CNAME>` (diagnose) or `--validate` (pre-create).\n- **Event window** — up to 500 most recent events (5 × 100, paginated).\n- **Colors** — auto-disabled on non-TTY; `--no-color` to force off.\n\n## Error handling\n\n| Failure                                             | Script                                                     | Tell the customer                                     |\n| --------------------------------------------------- | ---------------------------------------------------------- | ----------------------------------------------------- |\n| `aws sts get-caller-identity` fails                 | Exit 1                                                     | \"Fix AWS credentials and rerun.\"                      |\n| Cluster not found                                   | Exit 1 after listing region's clusters                     | \"Confirm HyperPod cluster name (not EKS) and region.\" |\n| `sagemaker:*` \u002F `ec2:*` \u002F `eks:*` \u002F `logs:*` denied | Warn, add `Missing IAM permission for \u003CAPI>`, continue     | \"Grant the listed IAM action and rerun.\"              |\n| `kubectl` absent or unauthenticated                 | Skip EKS checks (access entries, add-ons, aws-auth, nodes) | \"Install\u002Fauthenticate kubectl.\"                       |\n| `session-manager-plugin` absent (Slurm)             | Skip Slurm controller probe                                | \"Install session-manager-plugin.\"                     |\n| SSM throttled \u002F times out (180s)                    | Retry with backoff; warn and continue if still failing     | \"Rerun later — script is idempotent.\"                 |\n| CloudWatch log group not found                      | Skip CloudWatch check                                      | \"CloudWatch not configured on this cluster.\"          |\n\nExit codes: `0` no critical failures · `1` one or more critical failures (cluster not found, fatal prerequisite missing, or any `[FAIL]` in diagnose or `--validate` mode). `[WARN]` lines do not affect the exit code.\n\n## Skill delegation\n\n| Need                            | Use                        |\n| ------------------------------- | -------------------------- |\n| Shell on nodes                  | `hyperpod-ssm`             |\n| Version comparison across nodes | `hyperpod-version-checker` |\n\n## Escalate to AWS Support\n\nEscalate when:\n\n1. EFA health checks fail despite correct SG rules.\n2. Capacity errors persist despite a valid Flexible Training Plan \u002F ODCR.\n3. Node replacement fails repeatedly without clear events \u002F log signal.\n4. Cluster stuck in a non-terminal state (`Creating`, `Updating`, or a post-maintenance rollback state) for an extended period.\n5. CloudFormation root-cause is an internal service error.\n\n### Before opening the case\n\nRun these commands and attach the output. Goal: AWS Support has everything at case open.\n\n```bash\n# 1. Cluster identity + status (confirms region, ARN, orchestrator, instance groups)\naws sagemaker describe-cluster --cluster-name \u003CCLUSTER> --region \u003CREGION>\n\n# 2. Full cluster-level diagnostic bundle\nbash scripts\u002Fdiagnose-cluster.sh --cluster \u003CCLUSTER> --region \u003CREGION> > diag.txt\n\n# 3. Per-node log\u002Fconfig bundle to S3 (delegates to hyperpod-issue-report skill)\n#    See skills\u002Fhyperpod-issue-report\u002FSKILL.md for the exact invocation.\n```\n\n### Include in the case\n\n- Cluster name + ARN (or `ClusterId` suffix) and AWS region\n- `ClusterStatus` + `FailureMessage` from `describe-cluster`\n- Timestamp window (UTC start \u002F end) of the failure\n- Exact error strings observed (copy verbatim from events \u002F logs \u002F console)\n- Affected instance IDs \u002F `NodeLogicalId`s \u002F instance group names\n- `diag.txt` from step 2 above\n- S3 URI of the `hyperpod-issue-report` bundle from step 3\n",{"data":46,"body":49},{"name":4,"description":6,"metadata":47},{"version":48},"0.0.1",{"type":50,"children":51},"root",[52,60,86,96,116,255,259,266,326,329,335,614,627,691,694,700,708,1030,1033,1038,1050,1055,1081,1086,1112,1118,1138,1143,1190,1195,1229,1234,1261,1266,1298,1303,1338,1343,1360,1365,1392,1397,1409,1412,1418,1479,1489,1495,1576,1582,1787,1829,1835,1891,1897,1902,1943,1950,1955,2129,2135,2223],{"type":53,"tag":54,"props":55,"children":56},"element","h1",{"id":4},[57],{"type":58,"value":59},"text","HyperPod Cluster Debugger",{"type":53,"tag":61,"props":62,"children":63},"p",{},[64,70,72,77,79,84],{"type":53,"tag":65,"props":66,"children":67},"strong",{},[68],{"type":58,"value":69},"Operating policy.",{"type":58,"value":71}," Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a ",{"type":53,"tag":65,"props":73,"children":74},{},[75],{"type":58,"value":76},"Suggested command (run this yourself)",{"type":58,"value":78}," block and wait for the customer to run it. Destructive order: ",{"type":53,"tag":65,"props":80,"children":81},{},[82],{"type":58,"value":83},"investigate → reboot → replace",{"type":58,"value":85}," (replace destroys root + secondary volumes; not supported on Slurm controller nodes).",{"type":53,"tag":61,"props":87,"children":88},{},[89,94],{"type":53,"tag":65,"props":90,"children":91},{},[92],{"type":58,"value":93},"Before any state-changing CLI: ask if it's IaC-managed.",{"type":58,"value":95}," HyperPod clusters, SGs, EKS access entries, and IAM are usually provisioned via CloudFormation \u002F CDK \u002F Terraform. If yes, the fix belongs in IaC — running the CLI will drift and the next deploy reverts it. Use the CLI only when IaC is unavailable (locked out, predates IaC, mid-review).",{"type":53,"tag":61,"props":97,"children":98},{},[99,106,108,114],{"type":53,"tag":100,"props":101,"children":103},"code",{"className":102},[],[104],{"type":58,"value":105},"scripts\u002Fdiagnose-cluster.sh",{"type":58,"value":107}," is read-only: it collects state via AWS APIs (and SSM for Slurm controller health) and prints each issue as ",{"type":53,"tag":100,"props":109,"children":111},{"className":110},[],[112],{"type":58,"value":113},"[FAIL] ... → references\u002F\u003Cfile>.md § \u003Csection>",{"type":58,"value":115},".",{"type":53,"tag":117,"props":118,"children":119},"table",{},[120,139],{"type":53,"tag":121,"props":122,"children":123},"thead",{},[124],{"type":53,"tag":125,"props":126,"children":127},"tr",{},[128,134],{"type":53,"tag":129,"props":130,"children":131},"th",{},[132],{"type":58,"value":133},"Reference",{"type":53,"tag":129,"props":135,"children":136},{},[137],{"type":58,"value":138},"Open when",{"type":53,"tag":140,"props":141,"children":142},"tbody",{},[143,162,179,196,221,238],{"type":53,"tag":125,"props":144,"children":145},{},[146,157],{"type":53,"tag":147,"props":148,"children":149},"td",{},[150],{"type":53,"tag":151,"props":152,"children":154},"a",{"href":153},"references\u002Fcluster-diagnostics-detail.md",[155],{"type":58,"value":156},"cluster-diagnostics-detail.md",{"type":53,"tag":147,"props":158,"children":159},{},[160],{"type":58,"value":161},"Per-finding remediation runbook (§ A–L)",{"type":53,"tag":125,"props":163,"children":164},{},[165,174],{"type":53,"tag":147,"props":166,"children":167},{},[168],{"type":53,"tag":151,"props":169,"children":171},{"href":170},"references\u002Fcluster-operations.md",[172],{"type":58,"value":173},"cluster-operations.md",{"type":53,"tag":147,"props":175,"children":176},{},[177],{"type":58,"value":178},"Operational deep-dives (EFA SG, EKS access, SSM, Slurm, filesystem)",{"type":53,"tag":125,"props":180,"children":181},{},[182,191],{"type":53,"tag":147,"props":183,"children":184},{},[185],{"type":53,"tag":151,"props":186,"children":188},{"href":187},"references\u002Fcloudformation-errors.md",[189],{"type":58,"value":190},"cloudformation-errors.md",{"type":53,"tag":147,"props":192,"children":193},{},[194],{"type":58,"value":195},"§ H needs the full per-resource CFN error catalog",{"type":53,"tag":125,"props":197,"children":198},{},[199,208],{"type":53,"tag":147,"props":200,"children":201},{},[202],{"type":53,"tag":151,"props":203,"children":205},{"href":204},"references\u002Fcapacity-planning.md",[206],{"type":58,"value":207},"capacity-planning.md",{"type":53,"tag":147,"props":209,"children":210},{},[211,213,219],{"type":58,"value":212},"§ B or ",{"type":53,"tag":100,"props":214,"children":216},{"className":215},[],[217],{"type":58,"value":218},"--validate",{"type":58,"value":220}," flags capacity \u002F subnet sizing",{"type":53,"tag":125,"props":222,"children":223},{},[224,233],{"type":53,"tag":147,"props":225,"children":226},{},[227],{"type":53,"tag":151,"props":228,"children":230},{"href":229},"references\u002Flifecycle-scripts.md",[231],{"type":58,"value":232},"lifecycle-scripts.md",{"type":53,"tag":147,"props":234,"children":235},{},[236],{"type":58,"value":237},"§ C points at a specific lifecycle failure",{"type":53,"tag":125,"props":239,"children":240},{},[241,250],{"type":53,"tag":147,"props":242,"children":243},{},[244],{"type":53,"tag":151,"props":245,"children":247},{"href":246},"references\u002Fiam-permissions.md",[248],{"type":58,"value":249},"iam-permissions.md",{"type":53,"tag":147,"props":251,"children":252},{},[253],{"type":58,"value":254},"Full IAM policy for the diagnostic",{"type":53,"tag":256,"props":257,"children":258},"hr",{},[],{"type":53,"tag":260,"props":261,"children":263},"h2",{"id":262},"workflow",[264],{"type":58,"value":265},"Workflow",{"type":53,"tag":267,"props":268,"children":269},"ol",{},[270,276,295,316,321],{"type":53,"tag":271,"props":272,"children":273},"li",{},[274],{"type":58,"value":275},"Collect HyperPod cluster name (not EKS name), region, exact error string.",{"type":53,"tag":271,"props":277,"children":278},{},[279,281,286,288,293],{"type":58,"value":280},"Run ",{"type":53,"tag":100,"props":282,"children":284},{"className":283},[],[285],{"type":58,"value":105},{"type":58,"value":287}," (or ",{"type":53,"tag":100,"props":289,"children":291},{"className":290},[],[292],{"type":58,"value":218},{"type":58,"value":294}," for pre-create).",{"type":53,"tag":271,"props":296,"children":297},{},[298,300,306,308,314],{"type":58,"value":299},"For every ",{"type":53,"tag":100,"props":301,"children":303},{"className":302},[],[304],{"type":58,"value":305},"[FAIL]",{"type":58,"value":307}," line, ",{"type":53,"tag":100,"props":309,"children":311},{"className":310},[],[312],{"type":58,"value":313},"Read",{"type":58,"value":315}," the referenced section.",{"type":53,"tag":271,"props":317,"children":318},{},[319],{"type":58,"value":320},"Present finding, root cause, and the Suggested-command block verbatim. Wait for customer approval.",{"type":53,"tag":271,"props":322,"children":323},{},[324],{"type":58,"value":325},"Re-run the diagnostic to confirm.",{"type":53,"tag":256,"props":327,"children":328},{},[],{"type":53,"tag":260,"props":330,"children":332},{"id":331},"step-1-run-diagnostics",[333],{"type":58,"value":334},"Step 1: Run diagnostics",{"type":53,"tag":336,"props":337,"children":342},"pre",{"className":338,"code":339,"language":340,"meta":341,"style":341},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","# Diagnose an existing cluster:\nbash scripts\u002Fdiagnose-cluster.sh --cluster \u003CCLUSTER_NAME_OR_ARN> --region \u003CREGION>\n\n# Pre-flight (no cluster needed) — validates SGs, subnets, IAM, VPC endpoints,\n# optionally S3 lifecycle scripts and per-AZ capacity:\nbash scripts\u002Fdiagnose-cluster.sh --validate --region \u003CREGION> \\\n  --sg-ids \u003Csg-1,sg-2> --subnet-ids \u003Csub-1,sub-2> [--iam-role \u003Crole-arn>] \\\n  [--s3-uri s3:\u002F\u002F\u003CBUCKET>\u002Fpath\u002F] [--instance-type ml.p5.48xlarge]\n","bash","",[343],{"type":53,"tag":100,"props":344,"children":345},{"__ignoreMap":341},[346,358,423,433,442,451,493,567],{"type":53,"tag":347,"props":348,"children":351},"span",{"class":349,"line":350},"line",1,[352],{"type":53,"tag":347,"props":353,"children":355},{"style":354},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[356],{"type":58,"value":357},"# Diagnose an existing cluster:\n",{"type":53,"tag":347,"props":359,"children":361},{"class":349,"line":360},2,[362,367,373,378,384,389,395,400,405,409,414,418],{"type":53,"tag":347,"props":363,"children":365},{"style":364},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[366],{"type":58,"value":340},{"type":53,"tag":347,"props":368,"children":370},{"style":369},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[371],{"type":58,"value":372}," scripts\u002Fdiagnose-cluster.sh",{"type":53,"tag":347,"props":374,"children":375},{"style":369},[376],{"type":58,"value":377}," --cluster",{"type":53,"tag":347,"props":379,"children":381},{"style":380},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[382],{"type":58,"value":383}," \u003C",{"type":53,"tag":347,"props":385,"children":386},{"style":369},[387],{"type":58,"value":388},"CLUSTER_NAME_OR_AR",{"type":53,"tag":347,"props":390,"children":392},{"style":391},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[393],{"type":58,"value":394},"N",{"type":53,"tag":347,"props":396,"children":397},{"style":380},[398],{"type":58,"value":399},">",{"type":53,"tag":347,"props":401,"children":402},{"style":369},[403],{"type":58,"value":404}," --region",{"type":53,"tag":347,"props":406,"children":407},{"style":380},[408],{"type":58,"value":383},{"type":53,"tag":347,"props":410,"children":411},{"style":369},[412],{"type":58,"value":413},"REGIO",{"type":53,"tag":347,"props":415,"children":416},{"style":391},[417],{"type":58,"value":394},{"type":53,"tag":347,"props":419,"children":420},{"style":380},[421],{"type":58,"value":422},">\n",{"type":53,"tag":347,"props":424,"children":426},{"class":349,"line":425},3,[427],{"type":53,"tag":347,"props":428,"children":430},{"emptyLinePlaceholder":429},true,[431],{"type":58,"value":432},"\n",{"type":53,"tag":347,"props":434,"children":436},{"class":349,"line":435},4,[437],{"type":53,"tag":347,"props":438,"children":439},{"style":354},[440],{"type":58,"value":441},"# Pre-flight (no cluster needed) — validates SGs, subnets, IAM, VPC endpoints,\n",{"type":53,"tag":347,"props":443,"children":445},{"class":349,"line":444},5,[446],{"type":53,"tag":347,"props":447,"children":448},{"style":354},[449],{"type":58,"value":450},"# optionally S3 lifecycle scripts and per-AZ capacity:\n",{"type":53,"tag":347,"props":452,"children":454},{"class":349,"line":453},6,[455,459,463,468,472,476,480,484,488],{"type":53,"tag":347,"props":456,"children":457},{"style":364},[458],{"type":58,"value":340},{"type":53,"tag":347,"props":460,"children":461},{"style":369},[462],{"type":58,"value":372},{"type":53,"tag":347,"props":464,"children":465},{"style":369},[466],{"type":58,"value":467}," --validate",{"type":53,"tag":347,"props":469,"children":470},{"style":369},[471],{"type":58,"value":404},{"type":53,"tag":347,"props":473,"children":474},{"style":380},[475],{"type":58,"value":383},{"type":53,"tag":347,"props":477,"children":478},{"style":369},[479],{"type":58,"value":413},{"type":53,"tag":347,"props":481,"children":482},{"style":391},[483],{"type":58,"value":394},{"type":53,"tag":347,"props":485,"children":486},{"style":380},[487],{"type":58,"value":399},{"type":53,"tag":347,"props":489,"children":490},{"style":391},[491],{"type":58,"value":492}," \\\n",{"type":53,"tag":347,"props":494,"children":496},{"class":349,"line":495},7,[497,502,506,511,516,521,525,530,534,539,544,549,554,558,563],{"type":53,"tag":347,"props":498,"children":499},{"style":369},[500],{"type":58,"value":501},"  --sg-ids",{"type":53,"tag":347,"props":503,"children":504},{"style":380},[505],{"type":58,"value":383},{"type":53,"tag":347,"props":507,"children":508},{"style":369},[509],{"type":58,"value":510},"sg-1,sg-",{"type":53,"tag":347,"props":512,"children":513},{"style":380},[514],{"type":58,"value":515},"2>",{"type":53,"tag":347,"props":517,"children":518},{"style":369},[519],{"type":58,"value":520}," --subnet-ids",{"type":53,"tag":347,"props":522,"children":523},{"style":380},[524],{"type":58,"value":383},{"type":53,"tag":347,"props":526,"children":527},{"style":369},[528],{"type":58,"value":529},"sub-1,sub-",{"type":53,"tag":347,"props":531,"children":532},{"style":380},[533],{"type":58,"value":515},{"type":53,"tag":347,"props":535,"children":536},{"style":391},[537],{"type":58,"value":538}," [--iam-role ",{"type":53,"tag":347,"props":540,"children":541},{"style":380},[542],{"type":58,"value":543},"\u003C",{"type":53,"tag":347,"props":545,"children":546},{"style":369},[547],{"type":58,"value":548},"role-ar",{"type":53,"tag":347,"props":550,"children":551},{"style":391},[552],{"type":58,"value":553},"n",{"type":53,"tag":347,"props":555,"children":556},{"style":380},[557],{"type":58,"value":399},{"type":53,"tag":347,"props":559,"children":560},{"style":369},[561],{"type":58,"value":562},"]",{"type":53,"tag":347,"props":564,"children":565},{"style":391},[566],{"type":58,"value":492},{"type":53,"tag":347,"props":568,"children":570},{"class":349,"line":569},8,[571,576,581,585,590,595,599,604,609],{"type":53,"tag":347,"props":572,"children":573},{"style":391},[574],{"type":58,"value":575},"  [--s3-uri ",{"type":53,"tag":347,"props":577,"children":578},{"style":369},[579],{"type":58,"value":580},"s3:\u002F\u002F",{"type":53,"tag":347,"props":582,"children":583},{"style":380},[584],{"type":58,"value":543},{"type":53,"tag":347,"props":586,"children":587},{"style":369},[588],{"type":58,"value":589},"BUCKE",{"type":53,"tag":347,"props":591,"children":592},{"style":391},[593],{"type":58,"value":594},"T",{"type":53,"tag":347,"props":596,"children":597},{"style":380},[598],{"type":58,"value":399},{"type":53,"tag":347,"props":600,"children":601},{"style":369},[602],{"type":58,"value":603},"\u002Fpath\u002F]",{"type":53,"tag":347,"props":605,"children":606},{"style":391},[607],{"type":58,"value":608}," [--instance-type ",{"type":53,"tag":347,"props":610,"children":611},{"style":369},[612],{"type":58,"value":613},"ml.p5.48xlarge]\n",{"type":53,"tag":61,"props":615,"children":616},{},[617,619,625],{"type":58,"value":618},"Pass ",{"type":53,"tag":100,"props":620,"children":622},{"className":621},[],[623],{"type":58,"value":624},"--instance-type",{"type":58,"value":626}," when the target instance type is known — enables the per-AZ capacity check (warns if none of the provided subnets are in an AZ that offers that type, which causes insufficient-capacity failures at creation time).",{"type":53,"tag":61,"props":628,"children":629},{},[630,632,638,640,645,647,653,655,661,662,668,670,675,677,682,684,689],{"type":58,"value":631},"Tags: ",{"type":53,"tag":100,"props":633,"children":635},{"className":634},[],[636],{"type":58,"value":637},"[PASS]",{"type":58,"value":639}," · ",{"type":53,"tag":100,"props":641,"children":643},{"className":642},[],[644],{"type":58,"value":305},{"type":58,"value":646}," (counted, has ",{"type":53,"tag":100,"props":648,"children":650},{"className":649},[],[651],{"type":58,"value":652},"→ references\u002F...",{"type":58,"value":654}," pointer) · ",{"type":53,"tag":100,"props":656,"children":658},{"className":657},[],[659],{"type":58,"value":660},"[WARN]",{"type":58,"value":639},{"type":53,"tag":100,"props":663,"children":665},{"className":664},[],[666],{"type":58,"value":667},"[INFO]",{"type":58,"value":669},". Priorities: ",{"type":53,"tag":65,"props":671,"children":672},{},[673],{"type":58,"value":674},"P0",{"type":58,"value":676}," blocks operation · ",{"type":53,"tag":65,"props":678,"children":679},{},[680],{"type":58,"value":681},"P1",{"type":58,"value":683}," degraded · ",{"type":53,"tag":65,"props":685,"children":686},{},[687],{"type":58,"value":688},"P2",{"type":58,"value":690}," informational.",{"type":53,"tag":256,"props":692,"children":693},{},[],{"type":53,"tag":260,"props":695,"children":697},{"id":696},"step-2-match-signal-section",[698],{"type":58,"value":699},"Step 2: Match signal → section",{"type":53,"tag":61,"props":701,"children":702},{},[703],{"type":53,"tag":65,"props":704,"children":705},{},[706],{"type":58,"value":707},"Error messages \u002F events:",{"type":53,"tag":117,"props":709,"children":710},{},[711,727],{"type":53,"tag":121,"props":712,"children":713},{},[714],{"type":53,"tag":125,"props":715,"children":716},{},[717,722],{"type":53,"tag":129,"props":718,"children":719},{},[720],{"type":58,"value":721},"Signal",{"type":53,"tag":129,"props":723,"children":724},{},[725],{"type":58,"value":726},"Section",{"type":53,"tag":140,"props":728,"children":729},{},[730,756,776,796,816,842,868,896,922,948,974,994,1010],{"type":53,"tag":125,"props":731,"children":732},{},[733,744],{"type":53,"tag":147,"props":734,"children":735},{},[736,742],{"type":53,"tag":100,"props":737,"children":739},{"className":738},[],[740],{"type":58,"value":741},"\"EFA health checks did not run successfully\"",{"type":58,"value":743}," (public-doc verbatim signal)",{"type":53,"tag":147,"props":745,"children":746},{},[747],{"type":53,"tag":65,"props":748,"children":749},{},[750],{"type":53,"tag":151,"props":751,"children":753},{"href":752},"#a-efa-health-checks",[754],{"type":58,"value":755},"A: EFA Health Checks",{"type":53,"tag":125,"props":757,"children":758},{},[759,764],{"type":53,"tag":147,"props":760,"children":761},{},[762],{"type":58,"value":763},"Insufficient-capacity or AZ-mismatch failure at creation",{"type":53,"tag":147,"props":765,"children":766},{},[767],{"type":53,"tag":65,"props":768,"children":769},{},[770],{"type":53,"tag":151,"props":771,"children":773},{"href":772},"#b-capacity--az",[774],{"type":58,"value":775},"B: Capacity & AZ",{"type":53,"tag":125,"props":777,"children":778},{},[779,784],{"type":53,"tag":147,"props":780,"children":781},{},[782],{"type":58,"value":783},"Lifecycle-script failure or timeout during provisioning",{"type":53,"tag":147,"props":785,"children":786},{},[787],{"type":53,"tag":65,"props":788,"children":789},{},[790],{"type":53,"tag":151,"props":791,"children":793},{"href":792},"#c-lifecycle-scripts",[794],{"type":58,"value":795},"C: Lifecycle Scripts",{"type":53,"tag":125,"props":797,"children":798},{},[799,804],{"type":53,"tag":147,"props":800,"children":801},{},[802],{"type":58,"value":803},"kubectl auth error (server asks for credentials \u002F no API group list)",{"type":53,"tag":147,"props":805,"children":806},{},[807],{"type":53,"tag":65,"props":808,"children":809},{},[810],{"type":53,"tag":151,"props":811,"children":813},{"href":812},"#d-eks-access--kubectl",[814],{"type":58,"value":815},"D: EKS Access",{"type":53,"tag":125,"props":817,"children":818},{},[819,830],{"type":53,"tag":147,"props":820,"children":821},{},[822,828],{"type":53,"tag":100,"props":823,"children":825},{"className":824},[],[826],{"type":58,"value":827},"InService",{"type":58,"value":829}," but not all instances visible",{"type":53,"tag":147,"props":831,"children":832},{},[833],{"type":53,"tag":65,"props":834,"children":835},{},[836],{"type":53,"tag":151,"props":837,"children":839},{"href":838},"#e-cluster-provisioning",[840],{"type":58,"value":841},"E: Cluster Provisioning",{"type":53,"tag":125,"props":843,"children":844},{},[845,856],{"type":53,"tag":147,"props":846,"children":847},{},[848,854],{"type":53,"tag":100,"props":849,"children":851},{"className":850},[],[852],{"type":58,"value":853},"\"Target is not connected\"",{"type":58,"value":855}," \u002F SSM errors",{"type":53,"tag":147,"props":857,"children":858},{},[859],{"type":53,"tag":65,"props":860,"children":861},{},[862],{"type":53,"tag":151,"props":863,"children":865},{"href":864},"#f-ssm-connectivity",[866],{"type":58,"value":867},"F: SSM Connectivity",{"type":53,"tag":125,"props":869,"children":870},{},[871,884],{"type":53,"tag":147,"props":872,"children":873},{},[874,876,882],{"type":58,"value":875},"Node replacement not happening \u002F ",{"type":53,"tag":100,"props":877,"children":879},{"className":878},[],[880],{"type":58,"value":881},"batch-replace",{"type":58,"value":883}," not working",{"type":53,"tag":147,"props":885,"children":886},{},[887],{"type":53,"tag":65,"props":888,"children":889},{},[890],{"type":53,"tag":151,"props":891,"children":893},{"href":892},"#g-node-replacement",[894],{"type":58,"value":895},"G: Node Replacement",{"type":53,"tag":125,"props":897,"children":898},{},[899,910],{"type":53,"tag":147,"props":900,"children":901},{},[902,908],{"type":53,"tag":100,"props":903,"children":905},{"className":904},[],[906],{"type":58,"value":907},"\"Embedded stack failed\"",{"type":58,"value":909}," \u002F any CloudFormation error",{"type":53,"tag":147,"props":911,"children":912},{},[913],{"type":53,"tag":65,"props":914,"children":915},{},[916],{"type":53,"tag":151,"props":917,"children":919},{"href":918},"#h-cloudformation-errors",[920],{"type":58,"value":921},"H: CloudFormation Errors",{"type":53,"tag":125,"props":923,"children":924},{},[925,936],{"type":53,"tag":147,"props":926,"children":927},{},[928,934],{"type":53,"tag":100,"props":929,"children":931},{"className":930},[],[932],{"type":58,"value":933},"UpdateClusterSoftware",{"type":58,"value":935}," failed or cluster in post-maintenance rollback state",{"type":53,"tag":147,"props":937,"children":938},{},[939],{"type":53,"tag":65,"props":940,"children":941},{},[942],{"type":53,"tag":151,"props":943,"children":945},{"href":944},"#j-ami--cluster-updates",[946],{"type":58,"value":947},"J: AMI & Cluster Updates",{"type":53,"tag":125,"props":949,"children":950},{},[951,962],{"type":53,"tag":147,"props":952,"children":953},{},[954,956],{"type":58,"value":955},"Dangling \u002F orphaned nodes in EKS vs ",{"type":53,"tag":100,"props":957,"children":959},{"className":958},[],[960],{"type":58,"value":961},"list-cluster-nodes",{"type":53,"tag":147,"props":963,"children":964},{},[965],{"type":53,"tag":65,"props":966,"children":967},{},[968],{"type":53,"tag":151,"props":969,"children":971},{"href":970},"#k-dangling-nodes--cleanup",[972],{"type":58,"value":973},"K: Dangling Nodes & Cleanup",{"type":53,"tag":125,"props":975,"children":976},{},[977,982],{"type":53,"tag":147,"props":978,"children":979},{},[980],{"type":58,"value":981},"Cluster Autoscaler breaks after HyperPod attached",{"type":53,"tag":147,"props":983,"children":984},{},[985],{"type":53,"tag":65,"props":986,"children":987},{},[988],{"type":53,"tag":151,"props":989,"children":991},{"href":990},"#l-autoscaler-compatibility",[992],{"type":58,"value":993},"L: Autoscaler Compatibility",{"type":53,"tag":125,"props":995,"children":996},{},[997,1002],{"type":53,"tag":147,"props":998,"children":999},{},[1000],{"type":58,"value":1001},"Slow I\u002FO, FSx throughput saturated",{"type":53,"tag":147,"props":1003,"children":1004},{},[1005],{"type":53,"tag":151,"props":1006,"children":1007},{"href":170},[1008],{"type":58,"value":1009},"cluster-operations.md § 9",{"type":53,"tag":125,"props":1011,"children":1012},{},[1013,1018],{"type":53,"tag":147,"props":1014,"children":1015},{},[1016],{"type":58,"value":1017},"Slurm node name → instance ID lookup",{"type":53,"tag":147,"props":1019,"children":1020},{},[1021],{"type":53,"tag":65,"props":1022,"children":1023},{},[1024],{"type":53,"tag":151,"props":1025,"children":1027},{"href":1026},"#i-utilities",[1028],{"type":58,"value":1029},"I: Utilities",{"type":53,"tag":256,"props":1031,"children":1032},{},[],{"type":53,"tag":260,"props":1034,"children":1036},{"id":1035},"a-efa-health-checks",[1037],{"type":58,"value":755},{"type":53,"tag":61,"props":1039,"children":1040},{},[1041,1043,1049],{"type":58,"value":1042},"SG missing self-reference. Add inbound + outbound self-ref to every SG on the cluster, plus least-privilege egress for the AWS APIs the node needs (HTTPS 443 to S3 \u002F ECR \u002F SageMaker \u002F SSM \u002F STS \u002F CloudWatch Logs — via VPC-endpoint prefix-lists when possible). Full procedure: ",{"type":53,"tag":151,"props":1044,"children":1046},{"href":1045},"references\u002Fcluster-diagnostics-detail.md#a-efa-health-checks",[1047],{"type":58,"value":1048},"cluster-diagnostics-detail.md § A",{"type":58,"value":115},{"type":53,"tag":260,"props":1051,"children":1053},{"id":1052},"b-capacity-az",[1054],{"type":58,"value":775},{"type":53,"tag":61,"props":1056,"children":1057},{},[1058,1060,1066,1068,1074,1076,1080],{"type":58,"value":1059},"Instance type unavailable in the requested AZ. Verify with ",{"type":53,"tag":100,"props":1061,"children":1063},{"className":1062},[],[1064],{"type":58,"value":1065},"describe-instance-type-offerings",{"type":58,"value":1067},", then change AZ, use Flexible Training Plans, or request ODCR. Full: ",{"type":53,"tag":151,"props":1069,"children":1071},{"href":1070},"references\u002Fcluster-diagnostics-detail.md#b-capacity--az",[1072],{"type":58,"value":1073},"§ B",{"type":58,"value":1075}," · strategy: ",{"type":53,"tag":151,"props":1077,"children":1078},{"href":204},[1079],{"type":58,"value":207},{"type":58,"value":115},{"type":53,"tag":260,"props":1082,"children":1084},{"id":1083},"c-lifecycle-scripts",[1085],{"type":58,"value":795},{"type":53,"tag":61,"props":1087,"children":1088},{},[1089,1091,1097,1099,1105,1107,1111],{"type":58,"value":1090},"Script failed or timed out during provisioning. Read CloudWatch under ",{"type":53,"tag":100,"props":1092,"children":1094},{"className":1093},[],[1095],{"type":58,"value":1096},"\u002Faws\u002Fsagemaker\u002FClusters\u002F\u003Cname>\u002F\u003Cid>",{"type":58,"value":1098}," — common causes: missing S3 VPC endpoint, IAM gap, CRLF line endings, instance-group name mismatch. Full: ",{"type":53,"tag":151,"props":1100,"children":1102},{"href":1101},"references\u002Fcluster-diagnostics-detail.md#c-lifecycle-scripts",[1103],{"type":58,"value":1104},"§ C",{"type":58,"value":1106}," · layout: ",{"type":53,"tag":151,"props":1108,"children":1109},{"href":229},[1110],{"type":58,"value":232},{"type":58,"value":115},{"type":53,"tag":260,"props":1113,"children":1115},{"id":1114},"d-eks-access-kubectl",[1116],{"type":58,"value":1117},"D: EKS Access \u002F kubectl",{"type":53,"tag":61,"props":1119,"children":1120},{},[1121,1123,1129,1131,1137],{"type":58,"value":1122},"IAM identity not in EKS access entries. Verify with ",{"type":53,"tag":100,"props":1124,"children":1126},{"className":1125},[],[1127],{"type":58,"value":1128},"sts get-caller-identity",{"type":58,"value":1130},", create an access entry with admin policy, update kubeconfig. Full: ",{"type":53,"tag":151,"props":1132,"children":1134},{"href":1133},"references\u002Fcluster-diagnostics-detail.md#d-eks-access--kubectl",[1135],{"type":58,"value":1136},"§ D",{"type":58,"value":115},{"type":53,"tag":260,"props":1139,"children":1141},{"id":1140},"e-cluster-provisioning",[1142],{"type":58,"value":841},{"type":53,"tag":61,"props":1144,"children":1145},{},[1146,1151,1153,1159,1161,1167,1168,1174,1176,1181,1183,1189],{"type":53,"tag":100,"props":1147,"children":1149},{"className":1148},[],[1150],{"type":58,"value":827},{"type":58,"value":1152}," without all instances is expected under Continuous Provisioning — failures surface as events, not cluster errors. For stuck ",{"type":53,"tag":100,"props":1154,"children":1156},{"className":1155},[],[1157],{"type":58,"value":1158},"Creating",{"type":58,"value":1160},"\u002F",{"type":53,"tag":100,"props":1162,"children":1164},{"className":1163},[],[1165],{"type":58,"value":1166},"Updating",{"type":58,"value":1160},{"type":53,"tag":100,"props":1169,"children":1171},{"className":1170},[],[1172],{"type":58,"value":1173},"Deleting",{"type":58,"value":1175},": check CFN nested stacks (§ H), IAM, capacity, events; if stuck ",{"type":53,"tag":100,"props":1177,"children":1179},{"className":1178},[],[1180],{"type":58,"value":1173},{"type":58,"value":1182}," check VPC ENI dependencies. Full: ",{"type":53,"tag":151,"props":1184,"children":1186},{"href":1185},"references\u002Fcluster-diagnostics-detail.md#e-cluster-provisioning",[1187],{"type":58,"value":1188},"§ E",{"type":58,"value":115},{"type":53,"tag":260,"props":1191,"children":1193},{"id":1192},"f-ssm-connectivity",[1194],{"type":58,"value":867},{"type":53,"tag":61,"props":1196,"children":1197},{},[1198,1204,1206,1212,1214,1220,1222,1228],{"type":53,"tag":100,"props":1199,"children":1201},{"className":1200},[],[1202],{"type":58,"value":1203},"Target is not connected",{"type":58,"value":1205},": use ",{"type":53,"tag":100,"props":1207,"children":1209},{"className":1208},[],[1210],{"type":58,"value":1211},"sagemaker-cluster:\u003CCLUSTER_ID>_\u003CGROUP>-\u003CINSTANCE_ID>",{"type":58,"value":1213}," format (not raw EC2 ID), install session-manager-plugin, confirm node ",{"type":53,"tag":100,"props":1215,"children":1217},{"className":1216},[],[1218],{"type":58,"value":1219},"Running",{"type":58,"value":1221},". Check IAM + VPC endpoints on timeouts. Full: ",{"type":53,"tag":151,"props":1223,"children":1225},{"href":1224},"references\u002Fcluster-diagnostics-detail.md#f-ssm-connectivity",[1226],{"type":58,"value":1227},"§ F",{"type":58,"value":115},{"type":53,"tag":260,"props":1230,"children":1232},{"id":1231},"g-node-replacement",[1233],{"type":58,"value":895},{"type":53,"tag":61,"props":1235,"children":1236},{},[1237,1239,1245,1247,1252,1254,1260],{"type":58,"value":1238},"Auto-repair: confirm ",{"type":53,"tag":100,"props":1240,"children":1242},{"className":1241},[],[1243],{"type":58,"value":1244},"NodeRecovery=Automatic",{"type":58,"value":1246},", check Health Monitoring Agent (HMA) logs + node labels \u002F Slurm reason, confirm capacity. Manual: reboot first, replace only if reboot fails. Replace requires the cluster to have been patched via ",{"type":53,"tag":100,"props":1248,"children":1250},{"className":1249},[],[1251],{"type":58,"value":933},{"type":58,"value":1253}," at least once and cannot target a Slurm controller node. Full: ",{"type":53,"tag":151,"props":1255,"children":1257},{"href":1256},"references\u002Fcluster-diagnostics-detail.md#g-node-replacement",[1258],{"type":58,"value":1259},"§ G",{"type":58,"value":115},{"type":53,"tag":260,"props":1262,"children":1264},{"id":1263},"h-cloudformation-errors",[1265],{"type":58,"value":921},{"type":53,"tag":61,"props":1267,"children":1268},{},[1269,1275,1277,1283,1285,1291,1293,1297],{"type":53,"tag":100,"props":1270,"children":1272},{"className":1271},[],[1273],{"type":58,"value":1274},"Embedded stack failed",{"type":58,"value":1276}," hides the real error. Drill into nested stacks via Events tab (filter Failed) until you reach a non-stack resource. CLI: ",{"type":53,"tag":100,"props":1278,"children":1280},{"className":1279},[],[1281],{"type":58,"value":1282},"describe-stack-events --query 'StackEvents[?ResourceStatus==\\",{"type":58,"value":1284},"CREATE_FAILED`]'`. Also covers SLR creation failures and permission-boundary denials. Full: ",{"type":53,"tag":151,"props":1286,"children":1288},{"href":1287},"references\u002Fcluster-diagnostics-detail.md#h-cloudformation-errors",[1289],{"type":58,"value":1290},"§ H",{"type":58,"value":1292}," · catalog: ",{"type":53,"tag":151,"props":1294,"children":1295},{"href":187},[1296],{"type":58,"value":190},{"type":58,"value":115},{"type":53,"tag":260,"props":1299,"children":1301},{"id":1300},"i-utilities",[1302],{"type":58,"value":1029},{"type":53,"tag":61,"props":1304,"children":1305},{},[1306,1308,1314,1316,1321,1323,1329,1331,1337],{"type":58,"value":1307},"Map Slurm node names (",{"type":53,"tag":100,"props":1309,"children":1311},{"className":1310},[],[1312],{"type":58,"value":1313},"ip-10-x-y-z",{"type":58,"value":1315},") to HyperPod instance IDs via ",{"type":53,"tag":100,"props":1317,"children":1319},{"className":1318},[],[1320],{"type":58,"value":961},{"type":58,"value":1322}," or on-node ",{"type":53,"tag":100,"props":1324,"children":1326},{"className":1325},[],[1327],{"type":58,"value":1328},"\u002Fopt\u002Fml\u002Fconfig\u002Fresource_config.json",{"type":58,"value":1330},". Full: ",{"type":53,"tag":151,"props":1332,"children":1334},{"href":1333},"references\u002Fcluster-diagnostics-detail.md#i-utilities",[1335],{"type":58,"value":1336},"§ I",{"type":58,"value":115},{"type":53,"tag":260,"props":1339,"children":1341},{"id":1340},"j-ami-cluster-updates",[1342],{"type":58,"value":947},{"type":53,"tag":61,"props":1344,"children":1345},{},[1346,1351,1353,1359],{"type":53,"tag":100,"props":1347,"children":1349},{"className":1348},[],[1350],{"type":58,"value":933},{"type":58,"value":1352}," fails and rolls back, or the cluster stays in a post-maintenance rollback state. Common causes: lifecycle script incompatible with new AMI, HMA version too old, insufficient rolling-update capacity. If the cluster has active nodes, collect diagnostics and escalate rather than delete-and-recreate. Full: ",{"type":53,"tag":151,"props":1354,"children":1356},{"href":1355},"references\u002Fcluster-diagnostics-detail.md#j-ami--cluster-updates",[1357],{"type":58,"value":1358},"§ J",{"type":58,"value":115},{"type":53,"tag":260,"props":1361,"children":1363},{"id":1362},"k-dangling-nodes-cleanup",[1364],{"type":58,"value":973},{"type":53,"tag":61,"props":1366,"children":1367},{},[1368,1370,1376,1378,1383,1385,1391],{"type":58,"value":1369},"Nodes in ",{"type":53,"tag":100,"props":1371,"children":1373},{"className":1372},[],[1374],{"type":58,"value":1375},"kubectl get nodes",{"type":58,"value":1377}," but not in ",{"type":53,"tag":100,"props":1379,"children":1381},{"className":1380},[],[1382],{"type":58,"value":961},{"type":58,"value":1384}," (ghost EKS nodes), or the inverse (HyperPod nodes that never registered kubelet). Script flags both. Full: ",{"type":53,"tag":151,"props":1386,"children":1388},{"href":1387},"references\u002Fcluster-diagnostics-detail.md#k-dangling-nodes--cleanup",[1389],{"type":58,"value":1390},"§ K",{"type":58,"value":115},{"type":53,"tag":260,"props":1393,"children":1395},{"id":1394},"l-autoscaler-compatibility",[1396],{"type":58,"value":993},{"type":53,"tag":61,"props":1398,"children":1399},{},[1400,1402,1408],{"type":58,"value":1401},"Cluster Autoscaler errors on HyperPod provider IDs and breaks autoscaling for all node groups. No officially endorsed workaround — escalate to AWS Support. Karpenter does not conflict with HyperPod nodes by default. Full: ",{"type":53,"tag":151,"props":1403,"children":1405},{"href":1404},"references\u002Fcluster-diagnostics-detail.md#l-autoscaler-compatibility",[1406],{"type":58,"value":1407},"§ L",{"type":58,"value":115},{"type":53,"tag":256,"props":1410,"children":1411},{},[],{"type":53,"tag":260,"props":1413,"children":1415},{"id":1414},"prerequisites",[1416],{"type":58,"value":1417},"Prerequisites",{"type":53,"tag":1419,"props":1420,"children":1421},"ul",{},[1422,1432,1457,1468],{"type":53,"tag":271,"props":1423,"children":1424},{},[1425,1430],{"type":53,"tag":100,"props":1426,"children":1428},{"className":1427},[],[1429],{"type":58,"value":25},{"type":58,"value":1431}," CLI v2.13+ authenticated to the cluster's account",{"type":53,"tag":271,"props":1433,"children":1434},{},[1435,1441,1443,1449,1450,1455],{"type":53,"tag":100,"props":1436,"children":1438},{"className":1437},[],[1439],{"type":58,"value":1440},"jq",{"type":58,"value":1442},", ",{"type":53,"tag":100,"props":1444,"children":1446},{"className":1445},[],[1447],{"type":58,"value":1448},"python3",{"type":58,"value":1442},{"type":53,"tag":100,"props":1451,"children":1453},{"className":1452},[],[1454],{"type":58,"value":340},{"type":58,"value":1456}," 4.2+",{"type":53,"tag":271,"props":1458,"children":1459},{},[1460,1466],{"type":53,"tag":100,"props":1461,"children":1463},{"className":1462},[],[1464],{"type":58,"value":1465},"kubectl",{"type":58,"value":1467}," authenticated to the EKS cluster (EKS checks skipped if absent)",{"type":53,"tag":271,"props":1469,"children":1470},{},[1471,1477],{"type":53,"tag":100,"props":1472,"children":1474},{"className":1473},[],[1475],{"type":58,"value":1476},"session-manager-plugin",{"type":58,"value":1478}," (Slurm controller health checks only)",{"type":53,"tag":61,"props":1480,"children":1481},{},[1482,1484,1488],{"type":58,"value":1483},"IAM policy: ",{"type":53,"tag":151,"props":1485,"children":1486},{"href":246},[1487],{"type":58,"value":246},{"type":58,"value":115},{"type":53,"tag":260,"props":1490,"children":1492},{"id":1491},"defaults",[1493],{"type":58,"value":1494},"Defaults",{"type":53,"tag":1419,"props":1496,"children":1497},{},[1498,1523,1548,1558],{"type":53,"tag":271,"props":1499,"children":1500},{},[1501,1506,1508,1514,1516,1522],{"type":53,"tag":65,"props":1502,"children":1503},{},[1504],{"type":58,"value":1505},"Region",{"type":58,"value":1507}," — required: pass ",{"type":53,"tag":100,"props":1509,"children":1511},{"className":1510},[],[1512],{"type":58,"value":1513},"--region",{"type":58,"value":1515}," or set ",{"type":53,"tag":100,"props":1517,"children":1519},{"className":1518},[],[1520],{"type":58,"value":1521},"$AWS_DEFAULT_REGION",{"type":58,"value":115},{"type":53,"tag":271,"props":1524,"children":1525},{},[1526,1531,1533,1539,1541,1546],{"type":53,"tag":65,"props":1527,"children":1528},{},[1529],{"type":58,"value":1530},"Mode",{"type":58,"value":1532}," — ",{"type":53,"tag":100,"props":1534,"children":1536},{"className":1535},[],[1537],{"type":58,"value":1538},"--cluster \u003CNAME>",{"type":58,"value":1540}," (diagnose) or ",{"type":53,"tag":100,"props":1542,"children":1544},{"className":1543},[],[1545],{"type":58,"value":218},{"type":58,"value":1547}," (pre-create).",{"type":53,"tag":271,"props":1549,"children":1550},{},[1551,1556],{"type":53,"tag":65,"props":1552,"children":1553},{},[1554],{"type":58,"value":1555},"Event window",{"type":58,"value":1557}," — up to 500 most recent events (5 × 100, paginated).",{"type":53,"tag":271,"props":1559,"children":1560},{},[1561,1566,1568,1574],{"type":53,"tag":65,"props":1562,"children":1563},{},[1564],{"type":58,"value":1565},"Colors",{"type":58,"value":1567}," — auto-disabled on non-TTY; ",{"type":53,"tag":100,"props":1569,"children":1571},{"className":1570},[],[1572],{"type":58,"value":1573},"--no-color",{"type":58,"value":1575}," to force off.",{"type":53,"tag":260,"props":1577,"children":1579},{"id":1578},"error-handling",[1580],{"type":58,"value":1581},"Error handling",{"type":53,"tag":117,"props":1583,"children":1584},{},[1585,1606],{"type":53,"tag":121,"props":1586,"children":1587},{},[1588],{"type":53,"tag":125,"props":1589,"children":1590},{},[1591,1596,1601],{"type":53,"tag":129,"props":1592,"children":1593},{},[1594],{"type":58,"value":1595},"Failure",{"type":53,"tag":129,"props":1597,"children":1598},{},[1599],{"type":58,"value":1600},"Script",{"type":53,"tag":129,"props":1602,"children":1603},{},[1604],{"type":58,"value":1605},"Tell the customer",{"type":53,"tag":140,"props":1607,"children":1608},{},[1609,1633,1651,1705,1728,1751,1769],{"type":53,"tag":125,"props":1610,"children":1611},{},[1612,1623,1628],{"type":53,"tag":147,"props":1613,"children":1614},{},[1615,1621],{"type":53,"tag":100,"props":1616,"children":1618},{"className":1617},[],[1619],{"type":58,"value":1620},"aws sts get-caller-identity",{"type":58,"value":1622}," fails",{"type":53,"tag":147,"props":1624,"children":1625},{},[1626],{"type":58,"value":1627},"Exit 1",{"type":53,"tag":147,"props":1629,"children":1630},{},[1631],{"type":58,"value":1632},"\"Fix AWS credentials and rerun.\"",{"type":53,"tag":125,"props":1634,"children":1635},{},[1636,1641,1646],{"type":53,"tag":147,"props":1637,"children":1638},{},[1639],{"type":58,"value":1640},"Cluster not found",{"type":53,"tag":147,"props":1642,"children":1643},{},[1644],{"type":58,"value":1645},"Exit 1 after listing region's clusters",{"type":53,"tag":147,"props":1647,"children":1648},{},[1649],{"type":58,"value":1650},"\"Confirm HyperPod cluster name (not EKS) and region.\"",{"type":53,"tag":125,"props":1652,"children":1653},{},[1654,1687,1700],{"type":53,"tag":147,"props":1655,"children":1656},{},[1657,1663,1665,1671,1672,1678,1679,1685],{"type":53,"tag":100,"props":1658,"children":1660},{"className":1659},[],[1661],{"type":58,"value":1662},"sagemaker:*",{"type":58,"value":1664}," \u002F ",{"type":53,"tag":100,"props":1666,"children":1668},{"className":1667},[],[1669],{"type":58,"value":1670},"ec2:*",{"type":58,"value":1664},{"type":53,"tag":100,"props":1673,"children":1675},{"className":1674},[],[1676],{"type":58,"value":1677},"eks:*",{"type":58,"value":1664},{"type":53,"tag":100,"props":1680,"children":1682},{"className":1681},[],[1683],{"type":58,"value":1684},"logs:*",{"type":58,"value":1686}," denied",{"type":53,"tag":147,"props":1688,"children":1689},{},[1690,1692,1698],{"type":58,"value":1691},"Warn, add ",{"type":53,"tag":100,"props":1693,"children":1695},{"className":1694},[],[1696],{"type":58,"value":1697},"Missing IAM permission for \u003CAPI>",{"type":58,"value":1699},", continue",{"type":53,"tag":147,"props":1701,"children":1702},{},[1703],{"type":58,"value":1704},"\"Grant the listed IAM action and rerun.\"",{"type":53,"tag":125,"props":1706,"children":1707},{},[1708,1718,1723],{"type":53,"tag":147,"props":1709,"children":1710},{},[1711,1716],{"type":53,"tag":100,"props":1712,"children":1714},{"className":1713},[],[1715],{"type":58,"value":1465},{"type":58,"value":1717}," absent or unauthenticated",{"type":53,"tag":147,"props":1719,"children":1720},{},[1721],{"type":58,"value":1722},"Skip EKS checks (access entries, add-ons, aws-auth, nodes)",{"type":53,"tag":147,"props":1724,"children":1725},{},[1726],{"type":58,"value":1727},"\"Install\u002Fauthenticate kubectl.\"",{"type":53,"tag":125,"props":1729,"children":1730},{},[1731,1741,1746],{"type":53,"tag":147,"props":1732,"children":1733},{},[1734,1739],{"type":53,"tag":100,"props":1735,"children":1737},{"className":1736},[],[1738],{"type":58,"value":1476},{"type":58,"value":1740}," absent (Slurm)",{"type":53,"tag":147,"props":1742,"children":1743},{},[1744],{"type":58,"value":1745},"Skip Slurm controller probe",{"type":53,"tag":147,"props":1747,"children":1748},{},[1749],{"type":58,"value":1750},"\"Install session-manager-plugin.\"",{"type":53,"tag":125,"props":1752,"children":1753},{},[1754,1759,1764],{"type":53,"tag":147,"props":1755,"children":1756},{},[1757],{"type":58,"value":1758},"SSM throttled \u002F times out (180s)",{"type":53,"tag":147,"props":1760,"children":1761},{},[1762],{"type":58,"value":1763},"Retry with backoff; warn and continue if still failing",{"type":53,"tag":147,"props":1765,"children":1766},{},[1767],{"type":58,"value":1768},"\"Rerun later — script is idempotent.\"",{"type":53,"tag":125,"props":1770,"children":1771},{},[1772,1777,1782],{"type":53,"tag":147,"props":1773,"children":1774},{},[1775],{"type":58,"value":1776},"CloudWatch log group not found",{"type":53,"tag":147,"props":1778,"children":1779},{},[1780],{"type":58,"value":1781},"Skip CloudWatch check",{"type":53,"tag":147,"props":1783,"children":1784},{},[1785],{"type":58,"value":1786},"\"CloudWatch not configured on this cluster.\"",{"type":53,"tag":61,"props":1788,"children":1789},{},[1790,1792,1798,1800,1806,1808,1813,1815,1820,1822,1827],{"type":58,"value":1791},"Exit codes: ",{"type":53,"tag":100,"props":1793,"children":1795},{"className":1794},[],[1796],{"type":58,"value":1797},"0",{"type":58,"value":1799}," no critical failures · ",{"type":53,"tag":100,"props":1801,"children":1803},{"className":1802},[],[1804],{"type":58,"value":1805},"1",{"type":58,"value":1807}," one or more critical failures (cluster not found, fatal prerequisite missing, or any ",{"type":53,"tag":100,"props":1809,"children":1811},{"className":1810},[],[1812],{"type":58,"value":305},{"type":58,"value":1814}," in diagnose or ",{"type":53,"tag":100,"props":1816,"children":1818},{"className":1817},[],[1819],{"type":58,"value":218},{"type":58,"value":1821}," mode). ",{"type":53,"tag":100,"props":1823,"children":1825},{"className":1824},[],[1826],{"type":58,"value":660},{"type":58,"value":1828}," lines do not affect the exit code.",{"type":53,"tag":260,"props":1830,"children":1832},{"id":1831},"skill-delegation",[1833],{"type":58,"value":1834},"Skill delegation",{"type":53,"tag":117,"props":1836,"children":1837},{},[1838,1854],{"type":53,"tag":121,"props":1839,"children":1840},{},[1841],{"type":53,"tag":125,"props":1842,"children":1843},{},[1844,1849],{"type":53,"tag":129,"props":1845,"children":1846},{},[1847],{"type":58,"value":1848},"Need",{"type":53,"tag":129,"props":1850,"children":1851},{},[1852],{"type":58,"value":1853},"Use",{"type":53,"tag":140,"props":1855,"children":1856},{},[1857,1874],{"type":53,"tag":125,"props":1858,"children":1859},{},[1860,1865],{"type":53,"tag":147,"props":1861,"children":1862},{},[1863],{"type":58,"value":1864},"Shell on nodes",{"type":53,"tag":147,"props":1866,"children":1867},{},[1868],{"type":53,"tag":100,"props":1869,"children":1871},{"className":1870},[],[1872],{"type":58,"value":1873},"hyperpod-ssm",{"type":53,"tag":125,"props":1875,"children":1876},{},[1877,1882],{"type":53,"tag":147,"props":1878,"children":1879},{},[1880],{"type":58,"value":1881},"Version comparison across nodes",{"type":53,"tag":147,"props":1883,"children":1884},{},[1885],{"type":53,"tag":100,"props":1886,"children":1888},{"className":1887},[],[1889],{"type":58,"value":1890},"hyperpod-version-checker",{"type":53,"tag":260,"props":1892,"children":1894},{"id":1893},"escalate-to-aws-support",[1895],{"type":58,"value":1896},"Escalate to AWS Support",{"type":53,"tag":61,"props":1898,"children":1899},{},[1900],{"type":58,"value":1901},"Escalate when:",{"type":53,"tag":267,"props":1903,"children":1904},{},[1905,1910,1915,1920,1938],{"type":53,"tag":271,"props":1906,"children":1907},{},[1908],{"type":58,"value":1909},"EFA health checks fail despite correct SG rules.",{"type":53,"tag":271,"props":1911,"children":1912},{},[1913],{"type":58,"value":1914},"Capacity errors persist despite a valid Flexible Training Plan \u002F ODCR.",{"type":53,"tag":271,"props":1916,"children":1917},{},[1918],{"type":58,"value":1919},"Node replacement fails repeatedly without clear events \u002F log signal.",{"type":53,"tag":271,"props":1921,"children":1922},{},[1923,1925,1930,1931,1936],{"type":58,"value":1924},"Cluster stuck in a non-terminal state (",{"type":53,"tag":100,"props":1926,"children":1928},{"className":1927},[],[1929],{"type":58,"value":1158},{"type":58,"value":1442},{"type":53,"tag":100,"props":1932,"children":1934},{"className":1933},[],[1935],{"type":58,"value":1166},{"type":58,"value":1937},", or a post-maintenance rollback state) for an extended period.",{"type":53,"tag":271,"props":1939,"children":1940},{},[1941],{"type":58,"value":1942},"CloudFormation root-cause is an internal service error.",{"type":53,"tag":1944,"props":1945,"children":1947},"h3",{"id":1946},"before-opening-the-case",[1948],{"type":58,"value":1949},"Before opening the case",{"type":53,"tag":61,"props":1951,"children":1952},{},[1953],{"type":58,"value":1954},"Run these commands and attach the output. Goal: AWS Support has everything at case open.",{"type":53,"tag":336,"props":1956,"children":1958},{"className":338,"code":1957,"language":340,"meta":341,"style":341},"# 1. Cluster identity + status (confirms region, ARN, orchestrator, instance groups)\naws sagemaker describe-cluster --cluster-name \u003CCLUSTER> --region \u003CREGION>\n\n# 2. Full cluster-level diagnostic bundle\nbash scripts\u002Fdiagnose-cluster.sh --cluster \u003CCLUSTER> --region \u003CREGION> > diag.txt\n\n# 3. Per-node log\u002Fconfig bundle to S3 (delegates to hyperpod-issue-report skill)\n#    See skills\u002Fhyperpod-issue-report\u002FSKILL.md for the exact invocation.\n",[1959],{"type":53,"tag":100,"props":1960,"children":1961},{"__ignoreMap":341},[1962,1970,2030,2037,2045,2106,2113,2121],{"type":53,"tag":347,"props":1963,"children":1964},{"class":349,"line":350},[1965],{"type":53,"tag":347,"props":1966,"children":1967},{"style":354},[1968],{"type":58,"value":1969},"# 1. Cluster identity + status (confirms region, ARN, orchestrator, instance groups)\n",{"type":53,"tag":347,"props":1971,"children":1972},{"class":349,"line":360},[1973,1977,1982,1987,1992,1996,2001,2006,2010,2014,2018,2022,2026],{"type":53,"tag":347,"props":1974,"children":1975},{"style":364},[1976],{"type":58,"value":25},{"type":53,"tag":347,"props":1978,"children":1979},{"style":369},[1980],{"type":58,"value":1981}," sagemaker",{"type":53,"tag":347,"props":1983,"children":1984},{"style":369},[1985],{"type":58,"value":1986}," describe-cluster",{"type":53,"tag":347,"props":1988,"children":1989},{"style":369},[1990],{"type":58,"value":1991}," --cluster-name",{"type":53,"tag":347,"props":1993,"children":1994},{"style":380},[1995],{"type":58,"value":383},{"type":53,"tag":347,"props":1997,"children":1998},{"style":369},[1999],{"type":58,"value":2000},"CLUSTE",{"type":53,"tag":347,"props":2002,"children":2003},{"style":391},[2004],{"type":58,"value":2005},"R",{"type":53,"tag":347,"props":2007,"children":2008},{"style":380},[2009],{"type":58,"value":399},{"type":53,"tag":347,"props":2011,"children":2012},{"style":369},[2013],{"type":58,"value":404},{"type":53,"tag":347,"props":2015,"children":2016},{"style":380},[2017],{"type":58,"value":383},{"type":53,"tag":347,"props":2019,"children":2020},{"style":369},[2021],{"type":58,"value":413},{"type":53,"tag":347,"props":2023,"children":2024},{"style":391},[2025],{"type":58,"value":394},{"type":53,"tag":347,"props":2027,"children":2028},{"style":380},[2029],{"type":58,"value":422},{"type":53,"tag":347,"props":2031,"children":2032},{"class":349,"line":425},[2033],{"type":53,"tag":347,"props":2034,"children":2035},{"emptyLinePlaceholder":429},[2036],{"type":58,"value":432},{"type":53,"tag":347,"props":2038,"children":2039},{"class":349,"line":435},[2040],{"type":53,"tag":347,"props":2041,"children":2042},{"style":354},[2043],{"type":58,"value":2044},"# 2. Full cluster-level diagnostic bundle\n",{"type":53,"tag":347,"props":2046,"children":2047},{"class":349,"line":444},[2048,2052,2056,2060,2064,2068,2072,2076,2080,2084,2088,2092,2096,2101],{"type":53,"tag":347,"props":2049,"children":2050},{"style":364},[2051],{"type":58,"value":340},{"type":53,"tag":347,"props":2053,"children":2054},{"style":369},[2055],{"type":58,"value":372},{"type":53,"tag":347,"props":2057,"children":2058},{"style":369},[2059],{"type":58,"value":377},{"type":53,"tag":347,"props":2061,"children":2062},{"style":380},[2063],{"type":58,"value":383},{"type":53,"tag":347,"props":2065,"children":2066},{"style":369},[2067],{"type":58,"value":2000},{"type":53,"tag":347,"props":2069,"children":2070},{"style":391},[2071],{"type":58,"value":2005},{"type":53,"tag":347,"props":2073,"children":2074},{"style":380},[2075],{"type":58,"value":399},{"type":53,"tag":347,"props":2077,"children":2078},{"style":369},[2079],{"type":58,"value":404},{"type":53,"tag":347,"props":2081,"children":2082},{"style":380},[2083],{"type":58,"value":383},{"type":53,"tag":347,"props":2085,"children":2086},{"style":369},[2087],{"type":58,"value":413},{"type":53,"tag":347,"props":2089,"children":2090},{"style":391},[2091],{"type":58,"value":394},{"type":53,"tag":347,"props":2093,"children":2094},{"style":380},[2095],{"type":58,"value":399},{"type":53,"tag":347,"props":2097,"children":2098},{"style":380},[2099],{"type":58,"value":2100}," >",{"type":53,"tag":347,"props":2102,"children":2103},{"style":369},[2104],{"type":58,"value":2105}," diag.txt\n",{"type":53,"tag":347,"props":2107,"children":2108},{"class":349,"line":453},[2109],{"type":53,"tag":347,"props":2110,"children":2111},{"emptyLinePlaceholder":429},[2112],{"type":58,"value":432},{"type":53,"tag":347,"props":2114,"children":2115},{"class":349,"line":495},[2116],{"type":53,"tag":347,"props":2117,"children":2118},{"style":354},[2119],{"type":58,"value":2120},"# 3. Per-node log\u002Fconfig bundle to S3 (delegates to hyperpod-issue-report skill)\n",{"type":53,"tag":347,"props":2122,"children":2123},{"class":349,"line":569},[2124],{"type":53,"tag":347,"props":2125,"children":2126},{"style":354},[2127],{"type":58,"value":2128},"#    See skills\u002Fhyperpod-issue-report\u002FSKILL.md for the exact invocation.\n",{"type":53,"tag":1944,"props":2130,"children":2132},{"id":2131},"include-in-the-case",[2133],{"type":58,"value":2134},"Include in the case",{"type":53,"tag":1419,"props":2136,"children":2137},{},[2138,2151,2176,2181,2186,2199,2210],{"type":53,"tag":271,"props":2139,"children":2140},{},[2141,2143,2149],{"type":58,"value":2142},"Cluster name + ARN (or ",{"type":53,"tag":100,"props":2144,"children":2146},{"className":2145},[],[2147],{"type":58,"value":2148},"ClusterId",{"type":58,"value":2150}," suffix) and AWS region",{"type":53,"tag":271,"props":2152,"children":2153},{},[2154,2160,2162,2168,2170],{"type":53,"tag":100,"props":2155,"children":2157},{"className":2156},[],[2158],{"type":58,"value":2159},"ClusterStatus",{"type":58,"value":2161}," + ",{"type":53,"tag":100,"props":2163,"children":2165},{"className":2164},[],[2166],{"type":58,"value":2167},"FailureMessage",{"type":58,"value":2169}," from ",{"type":53,"tag":100,"props":2171,"children":2173},{"className":2172},[],[2174],{"type":58,"value":2175},"describe-cluster",{"type":53,"tag":271,"props":2177,"children":2178},{},[2179],{"type":58,"value":2180},"Timestamp window (UTC start \u002F end) of the failure",{"type":53,"tag":271,"props":2182,"children":2183},{},[2184],{"type":58,"value":2185},"Exact error strings observed (copy verbatim from events \u002F logs \u002F console)",{"type":53,"tag":271,"props":2187,"children":2188},{},[2189,2191,2197],{"type":58,"value":2190},"Affected instance IDs \u002F ",{"type":53,"tag":100,"props":2192,"children":2194},{"className":2193},[],[2195],{"type":58,"value":2196},"NodeLogicalId",{"type":58,"value":2198},"s \u002F instance group names",{"type":53,"tag":271,"props":2200,"children":2201},{},[2202,2208],{"type":53,"tag":100,"props":2203,"children":2205},{"className":2204},[],[2206],{"type":58,"value":2207},"diag.txt",{"type":58,"value":2209}," from step 2 above",{"type":53,"tag":271,"props":2211,"children":2212},{},[2213,2215,2221],{"type":58,"value":2214},"S3 URI of the ",{"type":53,"tag":100,"props":2216,"children":2218},{"className":2217},[],[2219],{"type":58,"value":2220},"hyperpod-issue-report",{"type":58,"value":2222}," bundle from step 3",{"type":53,"tag":2224,"props":2225,"children":2226},"style",{},[2227],{"type":58,"value":2228},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2230,"total":2406},[2231,2248,2269,2279,2292,2305,2315,2325,2346,2361,2376,2391],{"slug":2232,"name":2232,"fn":2233,"description":2234,"org":2235,"tags":2236,"stars":2245,"repoUrl":2246,"updatedAt":2247},"agentcore-investigation","investigate Bedrock AgentCore runtime sessions","Investigate Bedrock AgentCore runtime sessions via CloudWatch Logs Insights — resolve session\u002Ftrace IDs, query OTEL spans, filter noise, build timelines. Use when debugging AgentCore agent sessions, tracing tool calls, or analyzing latency.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2237,2238,2239,2242],{"name":24,"slug":25,"type":16},{"name":27,"slug":28,"type":16},{"name":2240,"slug":2241,"type":16},"Logs","logs",{"name":2243,"slug":2244,"type":16},"Observability","observability",9427,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fmcp","2026-07-12T08:37:22.601527",{"slug":2249,"name":2250,"fn":2251,"description":2252,"org":2253,"tags":2254,"stars":2245,"repoUrl":2246,"updatedAt":2268},"amazon-aurora-dsql","amazon aurora dsql","build applications with Aurora DSQL","Build with Aurora DSQL — manage schemas, execute queries, handle migrations, diagnose query plans, load data, and develop applications with a serverless, distributed SQL database. Covers IAM auth, multi-tenant patterns, MySQL-to-DSQL and PostgreSQL-to-DSQL schema conversion, FK replacement code generation, OCC retry patterns, ORM migration (Django\u002FHibernate\u002FRails), DDL operations, query plan explainability, SQL compatibility validation, and bulk data loading. Triggers on phrases like: DSQL, Aurora DSQL, create DSQL table, DSQL schema, migrate to DSQL, distributed SQL database, serverless PostgreSQL-compatible database, DSQL query plan, DSQL EXPLAIN ANALYZE, why is my DSQL query slow, DSQL foreign key, DSQL OCC retry, DSQL multi-region, load into DSQL, load CSV into DSQL, bulk load DSQL, aurora-dsql-loader.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2255,2258,2259,2262,2265],{"name":2256,"slug":2257,"type":16},"Aurora","aurora",{"name":24,"slug":25,"type":16},{"name":2260,"slug":2261,"type":16},"Database","database",{"name":2263,"slug":2264,"type":16},"Serverless","serverless",{"name":2266,"slug":2267,"type":16},"SQL","sql","2026-07-12T08:36:45.053393",{"slug":2270,"name":2271,"fn":2251,"description":2252,"org":2272,"tags":2273,"stars":2245,"repoUrl":2246,"updatedAt":2278},"aurora-dsql","aurora dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2274,2275,2276,2277],{"name":24,"slug":25,"type":16},{"name":2260,"slug":2261,"type":16},{"name":2263,"slug":2264,"type":16},{"name":2266,"slug":2267,"type":16},"2026-07-12T08:36:42.694299",{"slug":2280,"name":2281,"fn":2251,"description":2252,"org":2282,"tags":2283,"stars":2245,"repoUrl":2246,"updatedAt":2291},"aws-dsql","aws dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2284,2285,2286,2289,2290],{"name":24,"slug":25,"type":16},{"name":2260,"slug":2261,"type":16},{"name":2287,"slug":2288,"type":16},"Migration","migration",{"name":2263,"slug":2264,"type":16},{"name":2266,"slug":2267,"type":16},"2026-07-12T08:36:38.584057",{"slug":2293,"name":2294,"fn":2251,"description":2252,"org":2295,"tags":2296,"stars":2245,"repoUrl":2246,"updatedAt":2304},"distributed-postgres","distributed postgres",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2297,2298,2299,2302,2303],{"name":24,"slug":25,"type":16},{"name":2260,"slug":2261,"type":16},{"name":2300,"slug":2301,"type":16},"PostgreSQL","postgresql",{"name":2263,"slug":2264,"type":16},{"name":2266,"slug":2267,"type":16},"2026-07-12T08:36:46.530743",{"slug":2306,"name":2307,"fn":2251,"description":2252,"org":2308,"tags":2309,"stars":2245,"repoUrl":2246,"updatedAt":2314},"distributed-sql","distributed sql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2310,2311,2312,2313],{"name":24,"slug":25,"type":16},{"name":2260,"slug":2261,"type":16},{"name":2263,"slug":2264,"type":16},{"name":2266,"slug":2267,"type":16},"2026-07-12T08:36:48.104182",{"slug":2316,"name":2316,"fn":2251,"description":2252,"org":2317,"tags":2318,"stars":2245,"repoUrl":2246,"updatedAt":2324},"dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2319,2320,2321,2322,2323],{"name":24,"slug":25,"type":16},{"name":2260,"slug":2261,"type":16},{"name":2287,"slug":2288,"type":16},{"name":2263,"slug":2264,"type":16},{"name":2266,"slug":2267,"type":16},"2026-07-12T08:36:36.374512",{"slug":2326,"name":2326,"fn":2327,"description":2328,"org":2329,"tags":2330,"stars":2343,"repoUrl":2344,"updatedAt":2345},"cost-efficiency-analyzer","analyze cost efficiency and expenses","Analyzes cost structure, cost efficiency, and expense management from P&L data. Use when the user asks about costs, expenses, COGS, operating expenses, cost ratios, cost control, spending efficiency, margin compression from cost side, or wants to understand where money is going. Also use for \"are we spending too much\", \"cost breakdown\", \"expense analysis\", or \"how efficient are our operations\". NOT for revenue or top-line analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2331,2334,2337,2340],{"name":2332,"slug":2333,"type":16},"Accounting","accounting",{"name":2335,"slug":2336,"type":16},"Analytics","analytics",{"name":2338,"slug":2339,"type":16},"Cost Optimization","cost-optimization",{"name":2341,"slug":2342,"type":16},"Finance","finance",3176,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagentcore-samples","2026-07-12T08:40:03.29555",{"slug":2347,"name":2347,"fn":2348,"description":2349,"org":2350,"tags":2351,"stars":2343,"repoUrl":2344,"updatedAt":2360},"executive-financial-briefing","generate executive financial briefings","Generates a concise executive-level financial briefing or summary suitable for a CEO, CFO, or board presentation. Use when the user asks for a summary, briefing, executive summary, board update, financial overview, financial health check, or \"how is the business doing\". Covers the full P&L picture in one page. Also use for \"give me the highlights\", \"what do I need to know\", or \"quick financial update\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2352,2353,2354,2357],{"name":24,"slug":25,"type":16},{"name":2341,"slug":2342,"type":16},{"name":2355,"slug":2356,"type":16},"Management","management",{"name":2358,"slug":2359,"type":16},"Reporting","reporting","2026-07-12T08:40:02.066471",{"slug":2362,"name":2362,"fn":2363,"description":2364,"org":2365,"tags":2366,"stars":2343,"repoUrl":2344,"updatedAt":2375},"multi-quarter-trend-analysis","analyze multi-quarter financial trends","Analyzes financial trends across multiple quarters by comparing P&L metrics over time. Use when the user wants to see trends, patterns, trajectories, or directional movement across 3 or more quarters. Also use for \"how are we trending\", \"show me the trend\", \"track performance over time\", \"quarter over quarter comparison across all quarters\", or any multi-period longitudinal analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2367,2368,2369,2372],{"name":2335,"slug":2336,"type":16},{"name":2341,"slug":2342,"type":16},{"name":2370,"slug":2371,"type":16},"Financial Statements","financial-statements",{"name":2373,"slug":2374,"type":16},"Variance Analysis","variance-analysis","2026-07-12T08:40:00.79141",{"slug":2377,"name":2377,"fn":2378,"description":2379,"org":2380,"tags":2381,"stars":2343,"repoUrl":2344,"updatedAt":2390},"pdf","process and manipulate PDF documents","Use this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text\u002Ftables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting\u002Fdecrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2382,2385,2388],{"name":2383,"slug":2384,"type":16},"Automation","automation",{"name":2386,"slug":2387,"type":16},"Documents","documents",{"name":2389,"slug":2377,"type":16},"PDF","2026-07-12T08:41:44.135656",{"slug":2392,"name":2392,"fn":2393,"description":2394,"org":2395,"tags":2396,"stars":2343,"repoUrl":2344,"updatedAt":2405},"quarterly-kpi-calculator","calculate quarterly financial KPIs","Calculates quarterly financial KPIs from P&L data. P&L figures can be provided directly by the user or fetched from the financial data MCP server. Use when the user wants KPI calculations such as Gross Margin %, EBITDA Margin %, Operating Expense Ratio, or Revenue Growth % QoQ. Also use for quarterly performance review, P&L analysis, or interpreting financial ratios against benchmarks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2397,2398,2401,2402],{"name":2332,"slug":2333,"type":16},{"name":2399,"slug":2400,"type":16},"Data Analysis","data-analysis",{"name":2341,"slug":2342,"type":16},{"name":2403,"slug":2404,"type":16},"KPI","kpi","2026-07-12T08:39:59.54971",150,{"items":2408,"total":2507},[2409,2426,2444,2456,2473,2484,2494],{"slug":2410,"name":2410,"fn":2411,"description":2412,"org":2413,"tags":2414,"stars":29,"repoUrl":30,"updatedAt":2425},"amazon-location-service","integrate Amazon Location Service maps","Integrates Amazon Location Service APIs for AWS applications. Use this skill when users want to add maps (interactive MapLibre or static images); geocode addresses to coordinates or reverse geocode coordinates to addresses; calculate routes, travel times, or service areas; find places and businesses through text search, nearby search, or autocomplete suggestions; retrieve detailed place information including hours, contacts, and addresses; monitor geographical boundaries with geofences; or track device locations. Covers authentication, SDK integration, and all Amazon Location Service capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2415,2418,2419,2422],{"name":2416,"slug":2417,"type":16},"API Development","api-development",{"name":24,"slug":25,"type":16},{"name":2420,"slug":2421,"type":16},"Maps","maps",{"name":2423,"slug":2424,"type":16},"Navigation","navigation","2026-07-12T08:39:49.88311",{"slug":2427,"name":2427,"fn":2428,"description":2429,"org":2430,"tags":2431,"stars":29,"repoUrl":30,"updatedAt":2443},"amplify-workflow","build and deploy apps with AWS Amplify","Build and deploy full-stack web and mobile apps with AWS Amplify Gen2 (TypeScript code-first). Covers auth (Cognito), data (AppSync\u002FDynamoDB including schema modeling, enum types, relationships, authorization rules), storage (S3), functions, APIs, and AI (Amplify AI Kit with Bedrock). Supports React, Next.js, Vue, Angular, React Native, Flutter, Swift, and Android. Always use this skill for Amplify Gen2 topics — even for questions you think you know — it contains validated, version-specific patterns that prevent common mistakes. TRIGGER when: user mentions Amplify Gen2; project has amplify\u002F directory or amplify_outputs; code imports @aws-amplify packages; user asks about defineBackend, defineAuth, defineData, defineStorage, or npx ampx. SKIP: Amplify Gen1 (amplify CLI v6), standalone SAM\u002FCDK without Amplify (use aws-serverless), direct Bedrock without Amplify AI Kit (use bedrock).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2432,2435,2436,2437,2440],{"name":2433,"slug":2434,"type":16},"Auth","auth",{"name":24,"slug":25,"type":16},{"name":2260,"slug":2261,"type":16},{"name":2438,"slug":2439,"type":16},"Deployment","deployment",{"name":2441,"slug":2442,"type":16},"TypeScript","typescript","2026-07-12T08:39:43.500162",{"slug":2445,"name":2445,"fn":2446,"description":2447,"org":2448,"tags":2449,"stars":29,"repoUrl":30,"updatedAt":2455},"api-gateway","build and manage Amazon API Gateway APIs","Build, manage, and operate APIs with Amazon API Gateway (REST, HTTP, and WebSocket). Triggers on phrases like: API Gateway, REST API, HTTP API, WebSocket API, custom domain, Lambda authorizer, usage plan, throttling, CORS, VPC link, private API. Also covers troubleshooting API Gateway errors (4xx, 5xx, timeout, CORS failures) and IaC templates containing API Gateway resources. For general REST API design unrelated to AWS, do not trigger.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2450,2451,2452],{"name":2416,"slug":2417,"type":16},{"name":24,"slug":25,"type":16},{"name":2453,"slug":2454,"type":16},"REST API","rest-api","2026-07-12T08:39:00.149339",{"slug":2457,"name":2457,"fn":2458,"description":2459,"org":2460,"tags":2461,"stars":29,"repoUrl":30,"updatedAt":2472},"aws-architecture-diagram","generate AWS architecture diagrams","Generate validated AWS architecture diagrams as draw.io XML using official AWS4 icon libraries. Use this skill whenever the user wants to create, generate, or design AWS architecture diagrams, cloud infrastructure diagrams, or system design visuals. Also triggers for requests to visualize existing infrastructure from CloudFormation, CDK, or Terraform code. Supports two modes: analyze an existing codebase to auto-generate diagrams, or brainstorm interactively from scratch. Exports .drawio files with optional PNG\u002FSVG\u002FPDF export via draw.io desktop CLI.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2462,2465,2466,2469],{"name":2463,"slug":2464,"type":16},"Architecture","architecture",{"name":24,"slug":25,"type":16},{"name":2467,"slug":2468,"type":16},"Design","design",{"name":2470,"slug":2471,"type":16},"Diagrams","diagrams","2026-07-12T08:37:11.012278",{"slug":2474,"name":2474,"fn":2475,"description":2476,"org":2477,"tags":2478,"stars":29,"repoUrl":30,"updatedAt":2483},"aws-lambda","build and deploy AWS Lambda functions","Design, build, deploy, test, and debug serverless applications with AWS Lambda. Triggers on phrases like: Lambda function, event source, serverless application, API Gateway, EventBridge, Step Functions, serverless API, event-driven architecture, Lambda trigger. For deploying non-serverless apps to AWS, use deploy-on-aws plugin instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2479,2480,2481,2482],{"name":2416,"slug":2417,"type":16},{"name":24,"slug":25,"type":16},{"name":2438,"slug":2439,"type":16},{"name":2263,"slug":2264,"type":16},"2026-07-12T08:38:58.598492",{"slug":2485,"name":2485,"fn":2486,"description":2487,"org":2488,"tags":2489,"stars":29,"repoUrl":30,"updatedAt":2493},"aws-lambda-durable-functions","build resilient AWS Lambda durable functions","Build resilient, long-running, multi-step applications with AWS Lambda durable functions with automatic state persistence, retry logic, and orchestration for long-running executions. Covers the critical replay model, step operations, wait\u002Fcallback patterns, error handling with saga pattern, testing with LocalDurableTestRunner. Triggers on phrases like: lambda durable functions, workflow orchestration, state machines, retry\u002Fcheckpoint patterns, long-running stateful Lambda functions, saga pattern, human-in-the-loop callbacks, and reliable serverless applications.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2490,2491,2492],{"name":2463,"slug":2464,"type":16},{"name":24,"slug":25,"type":16},{"name":2263,"slug":2264,"type":16},"2026-07-12T08:39:05.546173",{"slug":2495,"name":2495,"fn":2496,"description":2497,"org":2498,"tags":2499,"stars":29,"repoUrl":30,"updatedAt":2506},"aws-lambda-managed-instances","configure AWS Lambda Managed Instances","Evaluate, configure, and migrate workloads to AWS Lambda Managed Instances (LMI). Triggers on: Lambda Managed Instances, LMI, capacity provider, multi-concurrency Lambda, dedicated instance Lambda, EC2-backed Lambda, cold start elimination, Graviton Lambda, instance type for Lambda, scheduled scaling for LMI, Lambda cost optimization with Reserved Instances or Savings Plans. Also trigger when users describe high-volume predictable workloads seeking cost savings, want to scale LMI capacity on a schedule, or compare Lambda vs EC2 for steady-state traffic. For standard Lambda without LMI, use the aws-lambda skill instead.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2500,2501,2502,2503],{"name":24,"slug":25,"type":16},{"name":2438,"slug":2439,"type":16},{"name":21,"slug":22,"type":16},{"name":2504,"slug":2505,"type":16},"Performance","performance","2026-07-12T08:39:07.007071",33]