[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-aws-labs-hyperpod-nccl":3,"mdc--5jyfud-key":42,"related-org-aws-labs-hyperpod-nccl":2576,"related-repo-aws-labs-hyperpod-nccl":2752},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":37,"sourceUrl":40,"mdContent":41},"hyperpod-nccl","diagnose NCCL and GPU training cluster failures","Diagnose NCCL failures and adjacent training-pod failures on HyperPod GPU clusters (EKS or Slurm) — training hangs, AllReduce \u002F collective-op timeouts, EFA or libfabric errors, rendezvous failures, EFA TCP fallback, \u002Fdev\u002Fshm or memlock issues, NCCL version mismatch across pods, container OOM \u002F exit-137 \u002F OOMKilled, GPU OOM (CUDA out of memory), CrashLoopBackOff \u002F Pending pods, MASTER_ADDR DNS, NetworkPolicy blocking. Not for single-node hardware faults (→ hyperpod-node-debugger § G) or cluster-creation EFA \u002F SSM failures (→ hyperpod-cluster-debugger § A \u002F § F).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"aws-labs","AWS Labs","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Faws-labs.png","awslabs",[13,17,20,23],{"name":14,"slug":15,"type":16},"Observability","observability","tag",{"name":18,"slug":19,"type":16},"Engineering","engineering",{"name":21,"slug":22,"type":16},"AWS","aws",{"name":24,"slug":25,"type":16},"Debugging","debugging",831,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagent-plugins","2026-07-12T08:39:12.365564",null,127,[32,33,34,22,35,36],"agent-plugins","agent-skills","agents","coding-agent-skills","coding-agents",{"repoUrl":27,"stars":26,"forks":30,"topics":38,"description":39},[32,33,34,22,35,36],"Agent Plugins for AWS equip AI coding agents with the skills to help you architect, deploy, and operate on AWS.","https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagent-plugins\u002Ftree\u002FHEAD\u002Fplugins\u002Fsagemaker-ai\u002Fskills\u002Fhyperpod-nccl","---\nname: hyperpod-nccl\ndescription: Diagnose NCCL failures and adjacent training-pod failures on HyperPod GPU clusters (EKS or Slurm) — training hangs, AllReduce \u002F collective-op timeouts, EFA or libfabric errors, rendezvous failures, EFA TCP fallback, \u002Fdev\u002Fshm or memlock issues, NCCL version mismatch across pods, container OOM \u002F exit-137 \u002F OOMKilled, GPU OOM (CUDA out of memory), CrashLoopBackOff \u002F Pending pods, MASTER_ADDR DNS, NetworkPolicy blocking. Not for single-node hardware faults (→ hyperpod-node-debugger § G) or cluster-creation EFA \u002F SSM failures (→ hyperpod-cluster-debugger § A \u002F § F).\nmetadata:\n  version: \"0.0.1\"\n---\n\n# HyperPod NCCL Debugger\n\n**Operating policy.** Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a **Suggested command (run this yourself)** block and wait for the customer. Destructive order: **investigate → reboot → replace** (replace destroys root + secondary volumes; not supported on Slurm controller nodes). Never discard training state on speculation.\n\nDiagnose NCCL failures on SageMaker HyperPod (EKS and Slurm). `scripts\u002Fnccl-diagnose.sh` reads state via AWS APIs, kubectl, and SSM, then prints each issue as `[FAIL] ... → references\u002F\u003Cfile>.md § \u003Csection>`. Read-only.\n\n**Signal sourcing:** `list-cluster-events` carries infrastructure-level state only (lifecycle, bootstrap, EFA health check, capacity, replacement, reboot, AMI rollback). It does **not** carry NCCL timeouts, GPU XID\u002FECC, or per-pod training signals — those come from pod logs, CloudWatch training streams, on-node SSM probes, and NCCL env audit. \"No events\" on a training-time NCCL issue is expected, not a clean bill of health.\n\n---\n\n## Workflow\n\n1. Collect cluster name, region, namespace\u002Fjob (EKS), exact NCCL error string.\n2. Run the diagnostic (always — the output drives everything else).\n3. For every `[FAIL]` line, `Read` the referenced section.\n4. Present finding, root cause, and the Suggested-command block with concrete values (instance IDs, SG IDs, namespaces) filled in from the script output. Wait for customer approval.\n5. Re-run the diagnostic to confirm.\n\nIf a finding has no matching section, report it as a bug — do not invent a fix.\n\n## Step 1: Authenticate kubectl (EKS)\n\n```bash\nEKS_ARN=$(aws sagemaker describe-cluster --cluster-name \u003CHYPERPOD-NAME> --region \u003CREGION> \\\n  --query 'Orchestrator.Eks.ClusterArn' --output text)\nEKS_NAME=$(echo \"$EKS_ARN\" | awk -F'\u002F' '{print $NF}')\naws eks update-kubeconfig --name \"$EKS_NAME\" --region \u003CREGION>\nkubectl get nodes\n```\n\n## Step 2: Run the diagnostic\n\n```bash\n# Basic:\nbash scripts\u002Fnccl-diagnose.sh --cluster \u003CHYPERPOD-NAME> --region \u003CREGION>\n\n# Scope to an EKS job\u002Fnamespace:\nbash scripts\u002Fnccl-diagnose.sh --cluster \u003CNAME> --region \u003CREGION> --namespace \u003CNS> --job \u003CJOB>\n\n# Force orchestrator:\nbash scripts\u002Fnccl-diagnose.sh --cluster \u003CNAME> --region \u003CREGION> --orchestrator slurm\n\n# Larger hardware sample (default 3):\nbash scripts\u002Fnccl-diagnose.sh --cluster \u003CNAME> --region \u003CREGION> --sample-nodes 10\n\n# Specific node only:\nbash scripts\u002Fnccl-diagnose.sh --cluster \u003CNAME> --region \u003CREGION> --node i-0abc123def456\n```\n\nTags: `[PASS]` · `[FAIL]` (counted in `Issues Found`, has reference pointer) · `[WARN]` · `[INFO]`. Priorities: **P0** blocks training · **P1** degraded · **P2** informational.\n\n---\n\n## Remediation index\n\nEach `[FAIL]` line in the script already points directly at the right section. This table is a lookup for manual triage.\n\n| Finding                                    | Section                                                                                             |\n| ------------------------------------------ | --------------------------------------------------------------------------------------------------- |\n| SG missing inbound\u002Foutbound self-reference | [operations.md § 8](references\u002Foperations.md)                                                       |\n| Blocking NetworkPolicy \u002F allow-all missing | [operations.md § 8](references\u002Foperations.md)                                                       |\n| Slurm node DOWN \u002F DRAINING \u002F RemoveIPC     | [operations.md § 7](references\u002Foperations.md)                                                       |\n| GPU XID \u002F SYSTEM_ERROR \u002F hardware fault    | [hyperpod-node-debugger § F \u002F § G](..\u002Fhyperpod-node-debugger\u002Freferences\u002Fnode-diagnostics-detail.md) |\n| GPU row-remap \u002F DCGM Fail \u002F silent NaNs    | [hyperpod-node-debugger § G.1.a\u002Fb](..\u002Fhyperpod-node-debugger\u002Freferences\u002Fnode-diagnostics-detail.md) |\n| NCCL timeout \u002F rendezvous \u002F straggler      | [debugging-guide.md § 1](references\u002Fdebugging-guide.md)                                             |\n| EFA configuration \u002F not used               | [debugging-guide.md § 6](references\u002Fdebugging-guide.md)                                             |\n| EFA TCP fallback (`NET\u002FOFI Using TCP`)     | [debugging-guide.md § 13](references\u002Fdebugging-guide.md)                                            |\n| NCCL version mismatch across pods          | [debugging-guide.md § 10](references\u002Fdebugging-guide.md)                                            |\n| Container OOM (pod killed, exit 137)       | [debugging-guide.md § 4](references\u002Fdebugging-guide.md)                                             |\n| GPU OOM (`CUDA out of memory`)             | [debugging-guide.md § 11](references\u002Fdebugging-guide.md)                                            |\n| RDMA memlock \u002F `\u002Fdev\u002Fshm` too small        | [debugging-guide.md § 17](references\u002Fdebugging-guide.md)                                            |\n| MASTER_ADDR DNS \u002F headless Service         | [debugging-guide.md § 12](references\u002Fdebugging-guide.md)                                            |\n| NVLS \u002F PXN \u002F topology tuning               | [debugging-guide.md § 19](references\u002Fdebugging-guide.md)                                            |\n| Any NCCL \u002F EFA \u002F rendezvous log pattern    | [error-patterns-quick-ref.md](references\u002Ferror-patterns-quick-ref.md)                               |\n| Performance \u002F nccl-tests \u002F bandwidth       | [performance-testing.md](references\u002Fperformance-testing.md)                                         |\n\n---\n\n## Prerequisites\n\n- `aws` CLI v2.13+ authenticated (`aws sts get-caller-identity`)\n- `jq`, `python3`, `bash` 4.2+\n- `unbuffer` (from the `expect` package: `yum install expect` \u002F `apt install expect`)\n- `kubectl` authenticated to the EKS cluster (K8s checks skipped if absent)\n- `session-manager-plugin` for on-node hardware checks\n\n## Defaults\n\n- **Region** — required: pass `--region` or set `$AWS_DEFAULT_REGION`.\n- **Orchestrator** — auto-detected; override with `--orchestrator eks|slurm`.\n- **Namespace \u002F job (EKS)** — all namespaces; scope with `--namespace \u003CNS> --job \u003CJOB>`.\n- **Hardware sampling** — 3 nodes over SSM (capped at 50). `--node \u003CID>` for a specific node. Node probes run **serially** (180 s per node): `--sample-nodes 10` can take ~30 min.\n- **CloudWatch window** — last 2 hours.\n- **Colors** — auto-disabled on non-TTY or `TERM=dumb`.\n\n## Error handling\n\n| Failure                             | Script                                                | Tell the customer                                                         |\n| ----------------------------------- | ----------------------------------------------------- | ------------------------------------------------------------------------- |\n| `aws sts get-caller-identity` fails | Exit 1 with the AWS error                             | \"Fix AWS credentials and rerun.\"                                          |\n| `describe-cluster` AccessDenied     | Warn, add `Missing IAM for sagemaker:DescribeCluster` | \"Grant `sagemaker:DescribeCluster` (operations.md § 2).\"                  |\n| Cluster not found                   | Exit 1 after listing region's clusters                | \"Confirm HyperPod cluster name and region.\"                               |\n| `kubectl` absent \u002F unauthenticated  | Warn, skip K8s checks                                 | \"`aws eks update-kubeconfig --name \u003CEKS> --region \u003CR>`.\"                  |\n| SSM plugin absent                   | Warn, skip on-node hardware checks                    | \"Install session-manager-plugin.\"                                         |\n| SSM times out (180s)                | Partial output, mark node unreachable                 | \"Rerun with `--node \u003CID> --sample-nodes 1`; check SSM agent on the node.\" |\n| CloudWatch log group not found      | Skip CloudWatch scan                                  | \"Enable CloudWatch on the cluster (operations.md § 4).\"                   |\n| Cluster events API throttled        | Warn, continue with partial data                      | \"Rerun later — script is idempotent.\"                                     |\n\nExit codes: `0` diagnostic complete · `1` fatal prerequisite missing or cluster unreachable.\n\n## IAM permissions\n\nFull policy + RBAC in [operations.md § 2](references\u002Foperations.md#2-iam). SSM on HyperPod uses `start-session` against `sagemaker-cluster:\u003Ccluster-id>_\u003Cgroup>-\u003Ciid>` targets — grant `ssm:StartSession` \u002F `ssm:TerminateSession`, not `ssm:SendCommand`.\n\n## Scale strategy\n\n| Scope           | Method                                   | Coverage                 |\n| --------------- | ---------------------------------------- | ------------------------ |\n| All nodes       | `sagemaker:ListClusterNodes` (paginated) | 100% nodes               |\n| All K8s objects | `kubectl`                                | 100% pods\u002Fnodes\u002Fpolicies |\n| Hardware        | SSM `--sample-nodes N` (default 3)       | Sampled                  |\n| Node logs       | CloudWatch                               | 100% nodes               |\n\n**Large clusters:** the PyTorch NCCL backend defaults to a 10-minute collective-op timeout (per the PyTorch distributed docs). Large clusters routinely exceed that on first rendezvous; raise it via `torch.distributed.init_process_group(timeout=timedelta(seconds=\u003CN>))`. HyperPod support has also observed NCCL topology-graph-search hangs on 256+ node clusters when `memlock` is `unlimited`; using a large fixed memlock (e.g. `8388608`) in pod `securityContext` or `\u002Fetc\u002Fsecurity\u002Flimits.conf` has cleared these in field cases. This memlock pattern is a field observation, not AWS- or NCCL-documented behavior.\n\nFor **FSDP**, **DeepSpeed**, or **Megatron-LM** tuning: [debugging-guide.md § 18](references\u002Fdebugging-guide.md).\n\n## Skill delegation\n\n| Need                                                                   | Use                                                          |\n| ---------------------------------------------------------------------- | ------------------------------------------------------------ |\n| Cluster creation \u002F deployment failures                                 | `hyperpod-cluster-debugger` (§ A \u002F B \u002F C \u002F H + `--validate`) |\n| Post-deployment cluster-wide management                                | `hyperpod-cluster-debugger`                                  |\n| Per-node issues (disk, lifecycle, hardware)                            | `hyperpod-node-debugger`                                     |\n| Trainium\u002FInferentia collective-comm (AWS Neuron Collectives, not NCCL) | `hyperpod-node-debugger` § G.2                               |\n| Shell on nodes                                                         | `hyperpod-ssm`                                               |\n| Version comparison across nodes                                        | `hyperpod-version-checker`                                   |\n| Diagnostic bundle for AWS Support                                      | `hyperpod-issue-report`                                      |\n| MFU \u002F performance degradation                                          | `hyperpod-mfu-debugger`                                      |\n\n## Escalate to AWS Support\n\nEscalate when:\n\n1. All SG rules correct, EFA verified on-node, but NCCL still times out.\n2. Hardware checks pass on all nodes but AllReduce still hangs.\n3. `Issues Found: 0` but training still fails.\n4. GPU XID errors persist after node replacement.\n5. Collective-op timeout raised and memlock workaround applied but large-cluster rendezvous still hangs.\n\n### Before opening the case\n\n```bash\n# 1. Cluster identity + status\naws sagemaker describe-cluster --cluster-name \u003CC> --region \u003CR>\n\n# 2. Full NCCL diagnostic (sample more nodes for escalation)\nbash scripts\u002Fnccl-diagnose.sh --cluster \u003CC> --region \u003CR> --sample-nodes 10 > nccl-diag.txt\n\n# 3. Per-node log\u002Fconfig bundle to S3 (delegates to hyperpod-issue-report)\n#    See skills\u002Fhyperpod-issue-report\u002FSKILL.md for the exact invocation.\n```\n\n### Include in the case\n\n- Cluster name + ARN and AWS region\n- Orchestrator (EKS or Slurm) and EKS cluster name \u002F Slurm controller node\n- Timestamp window (UTC start \u002F end) of the failure\n- Exact NCCL \u002F libfabric error strings (copy verbatim from pod logs or journalctl)\n- Affected instance IDs \u002F node names \u002F pod names \u002F namespace \u002F job name\n- `nccl-diag.txt` from step 2 above\n- S3 URI of the `hyperpod-issue-report` bundle from step 3\n- NCCL env vars in effect (`printenv | grep -E '^NCCL|^FI_|^TORCH_'` from one pod)\n\n## References\n\n- [error-patterns-quick-ref.md](references\u002Ferror-patterns-quick-ref.md) — log pattern → code → fix table\n- [debugging-guide.md](references\u002Fdebugging-guide.md) — per-scenario procedures (21 sections incl. NVLS\u002FPXN\u002Ftopology)\n- [performance-testing.md](references\u002Fperformance-testing.md) — nccl-tests, bandwidth thresholds, straggler detection\n- [operations.md](references\u002Foperations.md) — IAM, SSM format, CloudWatch, env-var reference, node labels, Slurm ops, remediations\n",{"data":43,"body":46},{"name":4,"description":6,"metadata":44},{"version":45},"0.0.1",{"type":47,"children":48},"root",[49,58,84,106,131,135,142,188,193,199,491,497,918,982,985,991,1003,1315,1318,1324,1425,1431,1554,1560,1776,1797,1803,1854,1860,1975,2033,2064,2070,2237,2243,2248,2282,2289,2453,2459,2523,2529,2570],{"type":50,"tag":51,"props":52,"children":54},"element","h1",{"id":53},"hyperpod-nccl-debugger",[55],{"type":56,"value":57},"text","HyperPod NCCL Debugger",{"type":50,"tag":59,"props":60,"children":61},"p",{},[62,68,70,75,77,82],{"type":50,"tag":63,"props":64,"children":65},"strong",{},[66],{"type":56,"value":67},"Operating policy.",{"type":56,"value":69}," Run read-only diagnostics yourself. Never run a command that changes cluster, node, or workload state — present each one as a ",{"type":50,"tag":63,"props":71,"children":72},{},[73],{"type":56,"value":74},"Suggested command (run this yourself)",{"type":56,"value":76}," block and wait for the customer. Destructive order: ",{"type":50,"tag":63,"props":78,"children":79},{},[80],{"type":56,"value":81},"investigate → reboot → replace",{"type":56,"value":83}," (replace destroys root + secondary volumes; not supported on Slurm controller nodes). Never discard training state on speculation.",{"type":50,"tag":59,"props":85,"children":86},{},[87,89,96,98,104],{"type":56,"value":88},"Diagnose NCCL failures on SageMaker HyperPod (EKS and Slurm). ",{"type":50,"tag":90,"props":91,"children":93},"code",{"className":92},[],[94],{"type":56,"value":95},"scripts\u002Fnccl-diagnose.sh",{"type":56,"value":97}," reads state via AWS APIs, kubectl, and SSM, then prints each issue as ",{"type":50,"tag":90,"props":99,"children":101},{"className":100},[],[102],{"type":56,"value":103},"[FAIL] ... → references\u002F\u003Cfile>.md § \u003Csection>",{"type":56,"value":105},". Read-only.",{"type":50,"tag":59,"props":107,"children":108},{},[109,114,116,122,124,129],{"type":50,"tag":63,"props":110,"children":111},{},[112],{"type":56,"value":113},"Signal sourcing:",{"type":56,"value":115}," ",{"type":50,"tag":90,"props":117,"children":119},{"className":118},[],[120],{"type":56,"value":121},"list-cluster-events",{"type":56,"value":123}," carries infrastructure-level state only (lifecycle, bootstrap, EFA health check, capacity, replacement, reboot, AMI rollback). It does ",{"type":50,"tag":63,"props":125,"children":126},{},[127],{"type":56,"value":128},"not",{"type":56,"value":130}," carry NCCL timeouts, GPU XID\u002FECC, or per-pod training signals — those come from pod logs, CloudWatch training streams, on-node SSM probes, and NCCL env audit. \"No events\" on a training-time NCCL issue is expected, not a clean bill of health.",{"type":50,"tag":132,"props":133,"children":134},"hr",{},[],{"type":50,"tag":136,"props":137,"children":139},"h2",{"id":138},"workflow",[140],{"type":56,"value":141},"Workflow",{"type":50,"tag":143,"props":144,"children":145},"ol",{},[146,152,157,178,183],{"type":50,"tag":147,"props":148,"children":149},"li",{},[150],{"type":56,"value":151},"Collect cluster name, region, namespace\u002Fjob (EKS), exact NCCL error string.",{"type":50,"tag":147,"props":153,"children":154},{},[155],{"type":56,"value":156},"Run the diagnostic (always — the output drives everything else).",{"type":50,"tag":147,"props":158,"children":159},{},[160,162,168,170,176],{"type":56,"value":161},"For every ",{"type":50,"tag":90,"props":163,"children":165},{"className":164},[],[166],{"type":56,"value":167},"[FAIL]",{"type":56,"value":169}," line, ",{"type":50,"tag":90,"props":171,"children":173},{"className":172},[],[174],{"type":56,"value":175},"Read",{"type":56,"value":177}," the referenced section.",{"type":50,"tag":147,"props":179,"children":180},{},[181],{"type":56,"value":182},"Present finding, root cause, and the Suggested-command block with concrete values (instance IDs, SG IDs, namespaces) filled in from the script output. Wait for customer approval.",{"type":50,"tag":147,"props":184,"children":185},{},[186],{"type":56,"value":187},"Re-run the diagnostic to confirm.",{"type":50,"tag":59,"props":189,"children":190},{},[191],{"type":56,"value":192},"If a finding has no matching section, report it as a bug — do not invent a fix.",{"type":50,"tag":136,"props":194,"children":196},{"id":195},"step-1-authenticate-kubectl-eks",[197],{"type":56,"value":198},"Step 1: Authenticate kubectl (EKS)",{"type":50,"tag":200,"props":201,"children":206},"pre",{"className":202,"code":203,"language":204,"meta":205,"style":205},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","EKS_ARN=$(aws sagemaker describe-cluster --cluster-name \u003CHYPERPOD-NAME> --region \u003CREGION> \\\n  --query 'Orchestrator.Eks.ClusterArn' --output text)\nEKS_NAME=$(echo \"$EKS_ARN\" | awk -F'\u002F' '{print $NF}')\naws eks update-kubeconfig --name \"$EKS_NAME\" --region \u003CREGION>\nkubectl get nodes\n","bash","",[207],{"type":50,"tag":90,"props":208,"children":209},{"__ignoreMap":205},[210,297,336,415,472],{"type":50,"tag":211,"props":212,"children":215},"span",{"class":213,"line":214},"line",1,[216,222,228,233,239,244,249,254,259,264,269,274,278,283,288,292],{"type":50,"tag":211,"props":217,"children":219},{"style":218},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[220],{"type":56,"value":221},"EKS_ARN",{"type":50,"tag":211,"props":223,"children":225},{"style":224},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[226],{"type":56,"value":227},"=$(",{"type":50,"tag":211,"props":229,"children":231},{"style":230},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[232],{"type":56,"value":22},{"type":50,"tag":211,"props":234,"children":236},{"style":235},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[237],{"type":56,"value":238}," sagemaker",{"type":50,"tag":211,"props":240,"children":241},{"style":235},[242],{"type":56,"value":243}," describe-cluster",{"type":50,"tag":211,"props":245,"children":246},{"style":235},[247],{"type":56,"value":248}," --cluster-name",{"type":50,"tag":211,"props":250,"children":251},{"style":224},[252],{"type":56,"value":253}," \u003C",{"type":50,"tag":211,"props":255,"children":256},{"style":235},[257],{"type":56,"value":258},"HYPERPOD-NAM",{"type":50,"tag":211,"props":260,"children":261},{"style":218},[262],{"type":56,"value":263},"E",{"type":50,"tag":211,"props":265,"children":266},{"style":224},[267],{"type":56,"value":268},">",{"type":50,"tag":211,"props":270,"children":271},{"style":235},[272],{"type":56,"value":273}," --region",{"type":50,"tag":211,"props":275,"children":276},{"style":224},[277],{"type":56,"value":253},{"type":50,"tag":211,"props":279,"children":280},{"style":235},[281],{"type":56,"value":282},"REGIO",{"type":50,"tag":211,"props":284,"children":285},{"style":218},[286],{"type":56,"value":287},"N",{"type":50,"tag":211,"props":289,"children":290},{"style":224},[291],{"type":56,"value":268},{"type":50,"tag":211,"props":293,"children":294},{"style":218},[295],{"type":56,"value":296}," \\\n",{"type":50,"tag":211,"props":298,"children":300},{"class":213,"line":299},2,[301,306,311,316,321,326,331],{"type":50,"tag":211,"props":302,"children":303},{"style":235},[304],{"type":56,"value":305},"  --query",{"type":50,"tag":211,"props":307,"children":308},{"style":224},[309],{"type":56,"value":310}," '",{"type":50,"tag":211,"props":312,"children":313},{"style":235},[314],{"type":56,"value":315},"Orchestrator.Eks.ClusterArn",{"type":50,"tag":211,"props":317,"children":318},{"style":224},[319],{"type":56,"value":320},"'",{"type":50,"tag":211,"props":322,"children":323},{"style":235},[324],{"type":56,"value":325}," --output",{"type":50,"tag":211,"props":327,"children":328},{"style":235},[329],{"type":56,"value":330}," text",{"type":50,"tag":211,"props":332,"children":333},{"style":224},[334],{"type":56,"value":335},")\n",{"type":50,"tag":211,"props":337,"children":339},{"class":213,"line":338},3,[340,345,349,355,360,365,370,375,380,385,389,394,398,402,407,411],{"type":50,"tag":211,"props":341,"children":342},{"style":218},[343],{"type":56,"value":344},"EKS_NAME",{"type":50,"tag":211,"props":346,"children":347},{"style":224},[348],{"type":56,"value":227},{"type":50,"tag":211,"props":350,"children":352},{"style":351},"--shiki-light:#6182B8;--shiki-default:#82AAFF;--shiki-dark:#82AAFF",[353],{"type":56,"value":354},"echo",{"type":50,"tag":211,"props":356,"children":357},{"style":224},[358],{"type":56,"value":359}," \"",{"type":50,"tag":211,"props":361,"children":362},{"style":218},[363],{"type":56,"value":364},"$EKS_ARN",{"type":50,"tag":211,"props":366,"children":367},{"style":224},[368],{"type":56,"value":369},"\"",{"type":50,"tag":211,"props":371,"children":372},{"style":224},[373],{"type":56,"value":374}," |",{"type":50,"tag":211,"props":376,"children":377},{"style":230},[378],{"type":56,"value":379}," awk",{"type":50,"tag":211,"props":381,"children":382},{"style":235},[383],{"type":56,"value":384}," -F",{"type":50,"tag":211,"props":386,"children":387},{"style":224},[388],{"type":56,"value":320},{"type":50,"tag":211,"props":390,"children":391},{"style":235},[392],{"type":56,"value":393},"\u002F",{"type":50,"tag":211,"props":395,"children":396},{"style":224},[397],{"type":56,"value":320},{"type":50,"tag":211,"props":399,"children":400},{"style":224},[401],{"type":56,"value":310},{"type":50,"tag":211,"props":403,"children":404},{"style":235},[405],{"type":56,"value":406},"{print $NF}",{"type":50,"tag":211,"props":408,"children":409},{"style":224},[410],{"type":56,"value":320},{"type":50,"tag":211,"props":412,"children":413},{"style":224},[414],{"type":56,"value":335},{"type":50,"tag":211,"props":416,"children":418},{"class":213,"line":417},4,[419,423,428,433,438,442,447,451,455,459,463,467],{"type":50,"tag":211,"props":420,"children":421},{"style":230},[422],{"type":56,"value":22},{"type":50,"tag":211,"props":424,"children":425},{"style":235},[426],{"type":56,"value":427}," eks",{"type":50,"tag":211,"props":429,"children":430},{"style":235},[431],{"type":56,"value":432}," update-kubeconfig",{"type":50,"tag":211,"props":434,"children":435},{"style":235},[436],{"type":56,"value":437}," --name",{"type":50,"tag":211,"props":439,"children":440},{"style":224},[441],{"type":56,"value":359},{"type":50,"tag":211,"props":443,"children":444},{"style":218},[445],{"type":56,"value":446},"$EKS_NAME",{"type":50,"tag":211,"props":448,"children":449},{"style":224},[450],{"type":56,"value":369},{"type":50,"tag":211,"props":452,"children":453},{"style":235},[454],{"type":56,"value":273},{"type":50,"tag":211,"props":456,"children":457},{"style":224},[458],{"type":56,"value":253},{"type":50,"tag":211,"props":460,"children":461},{"style":235},[462],{"type":56,"value":282},{"type":50,"tag":211,"props":464,"children":465},{"style":218},[466],{"type":56,"value":287},{"type":50,"tag":211,"props":468,"children":469},{"style":224},[470],{"type":56,"value":471},">\n",{"type":50,"tag":211,"props":473,"children":475},{"class":213,"line":474},5,[476,481,486],{"type":50,"tag":211,"props":477,"children":478},{"style":230},[479],{"type":56,"value":480},"kubectl",{"type":50,"tag":211,"props":482,"children":483},{"style":235},[484],{"type":56,"value":485}," get",{"type":50,"tag":211,"props":487,"children":488},{"style":235},[489],{"type":56,"value":490}," nodes\n",{"type":50,"tag":136,"props":492,"children":494},{"id":493},"step-2-run-the-diagnostic",[495],{"type":56,"value":496},"Step 2: Run the diagnostic",{"type":50,"tag":200,"props":498,"children":500},{"className":202,"code":499,"language":204,"meta":205,"style":205},"# Basic:\nbash scripts\u002Fnccl-diagnose.sh --cluster \u003CHYPERPOD-NAME> --region \u003CREGION>\n\n# Scope to an EKS job\u002Fnamespace:\nbash scripts\u002Fnccl-diagnose.sh --cluster \u003CNAME> --region \u003CREGION> --namespace \u003CNS> --job \u003CJOB>\n\n# Force orchestrator:\nbash scripts\u002Fnccl-diagnose.sh --cluster \u003CNAME> --region \u003CREGION> --orchestrator slurm\n\n# Larger hardware sample (default 3):\nbash scripts\u002Fnccl-diagnose.sh --cluster \u003CNAME> --region \u003CREGION> --sample-nodes 10\n\n# Specific node only:\nbash scripts\u002Fnccl-diagnose.sh --cluster \u003CNAME> --region \u003CREGION> --node i-0abc123def456\n",[501],{"type":50,"tag":90,"props":502,"children":503},{"__ignoreMap":205},[504,513,566,575,583,680,688,697,759,767,776,839,847,856],{"type":50,"tag":211,"props":505,"children":506},{"class":213,"line":214},[507],{"type":50,"tag":211,"props":508,"children":510},{"style":509},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[511],{"type":56,"value":512},"# Basic:\n",{"type":50,"tag":211,"props":514,"children":515},{"class":213,"line":299},[516,520,525,530,534,538,542,546,550,554,558,562],{"type":50,"tag":211,"props":517,"children":518},{"style":230},[519],{"type":56,"value":204},{"type":50,"tag":211,"props":521,"children":522},{"style":235},[523],{"type":56,"value":524}," scripts\u002Fnccl-diagnose.sh",{"type":50,"tag":211,"props":526,"children":527},{"style":235},[528],{"type":56,"value":529}," --cluster",{"type":50,"tag":211,"props":531,"children":532},{"style":224},[533],{"type":56,"value":253},{"type":50,"tag":211,"props":535,"children":536},{"style":235},[537],{"type":56,"value":258},{"type":50,"tag":211,"props":539,"children":540},{"style":218},[541],{"type":56,"value":263},{"type":50,"tag":211,"props":543,"children":544},{"style":224},[545],{"type":56,"value":268},{"type":50,"tag":211,"props":547,"children":548},{"style":235},[549],{"type":56,"value":273},{"type":50,"tag":211,"props":551,"children":552},{"style":224},[553],{"type":56,"value":253},{"type":50,"tag":211,"props":555,"children":556},{"style":235},[557],{"type":56,"value":282},{"type":50,"tag":211,"props":559,"children":560},{"style":218},[561],{"type":56,"value":287},{"type":50,"tag":211,"props":563,"children":564},{"style":224},[565],{"type":56,"value":471},{"type":50,"tag":211,"props":567,"children":568},{"class":213,"line":338},[569],{"type":50,"tag":211,"props":570,"children":572},{"emptyLinePlaceholder":571},true,[573],{"type":56,"value":574},"\n",{"type":50,"tag":211,"props":576,"children":577},{"class":213,"line":417},[578],{"type":50,"tag":211,"props":579,"children":580},{"style":509},[581],{"type":56,"value":582},"# Scope to an EKS job\u002Fnamespace:\n",{"type":50,"tag":211,"props":584,"children":585},{"class":213,"line":474},[586,590,594,598,602,607,611,615,619,623,627,631,635,640,644,648,653,657,662,666,671,676],{"type":50,"tag":211,"props":587,"children":588},{"style":230},[589],{"type":56,"value":204},{"type":50,"tag":211,"props":591,"children":592},{"style":235},[593],{"type":56,"value":524},{"type":50,"tag":211,"props":595,"children":596},{"style":235},[597],{"type":56,"value":529},{"type":50,"tag":211,"props":599,"children":600},{"style":224},[601],{"type":56,"value":253},{"type":50,"tag":211,"props":603,"children":604},{"style":235},[605],{"type":56,"value":606},"NAM",{"type":50,"tag":211,"props":608,"children":609},{"style":218},[610],{"type":56,"value":263},{"type":50,"tag":211,"props":612,"children":613},{"style":224},[614],{"type":56,"value":268},{"type":50,"tag":211,"props":616,"children":617},{"style":235},[618],{"type":56,"value":273},{"type":50,"tag":211,"props":620,"children":621},{"style":224},[622],{"type":56,"value":253},{"type":50,"tag":211,"props":624,"children":625},{"style":235},[626],{"type":56,"value":282},{"type":50,"tag":211,"props":628,"children":629},{"style":218},[630],{"type":56,"value":287},{"type":50,"tag":211,"props":632,"children":633},{"style":224},[634],{"type":56,"value":268},{"type":50,"tag":211,"props":636,"children":637},{"style":235},[638],{"type":56,"value":639}," --namespace",{"type":50,"tag":211,"props":641,"children":642},{"style":224},[643],{"type":56,"value":253},{"type":50,"tag":211,"props":645,"children":646},{"style":235},[647],{"type":56,"value":287},{"type":50,"tag":211,"props":649,"children":650},{"style":218},[651],{"type":56,"value":652},"S",{"type":50,"tag":211,"props":654,"children":655},{"style":224},[656],{"type":56,"value":268},{"type":50,"tag":211,"props":658,"children":659},{"style":235},[660],{"type":56,"value":661}," --job",{"type":50,"tag":211,"props":663,"children":664},{"style":224},[665],{"type":56,"value":253},{"type":50,"tag":211,"props":667,"children":668},{"style":235},[669],{"type":56,"value":670},"JO",{"type":50,"tag":211,"props":672,"children":673},{"style":218},[674],{"type":56,"value":675},"B",{"type":50,"tag":211,"props":677,"children":678},{"style":224},[679],{"type":56,"value":471},{"type":50,"tag":211,"props":681,"children":683},{"class":213,"line":682},6,[684],{"type":50,"tag":211,"props":685,"children":686},{"emptyLinePlaceholder":571},[687],{"type":56,"value":574},{"type":50,"tag":211,"props":689,"children":691},{"class":213,"line":690},7,[692],{"type":50,"tag":211,"props":693,"children":694},{"style":509},[695],{"type":56,"value":696},"# Force orchestrator:\n",{"type":50,"tag":211,"props":698,"children":700},{"class":213,"line":699},8,[701,705,709,713,717,721,725,729,733,737,741,745,749,754],{"type":50,"tag":211,"props":702,"children":703},{"style":230},[704],{"type":56,"value":204},{"type":50,"tag":211,"props":706,"children":707},{"style":235},[708],{"type":56,"value":524},{"type":50,"tag":211,"props":710,"children":711},{"style":235},[712],{"type":56,"value":529},{"type":50,"tag":211,"props":714,"children":715},{"style":224},[716],{"type":56,"value":253},{"type":50,"tag":211,"props":718,"children":719},{"style":235},[720],{"type":56,"value":606},{"type":50,"tag":211,"props":722,"children":723},{"style":218},[724],{"type":56,"value":263},{"type":50,"tag":211,"props":726,"children":727},{"style":224},[728],{"type":56,"value":268},{"type":50,"tag":211,"props":730,"children":731},{"style":235},[732],{"type":56,"value":273},{"type":50,"tag":211,"props":734,"children":735},{"style":224},[736],{"type":56,"value":253},{"type":50,"tag":211,"props":738,"children":739},{"style":235},[740],{"type":56,"value":282},{"type":50,"tag":211,"props":742,"children":743},{"style":218},[744],{"type":56,"value":287},{"type":50,"tag":211,"props":746,"children":747},{"style":224},[748],{"type":56,"value":268},{"type":50,"tag":211,"props":750,"children":751},{"style":235},[752],{"type":56,"value":753}," --orchestrator",{"type":50,"tag":211,"props":755,"children":756},{"style":235},[757],{"type":56,"value":758}," slurm\n",{"type":50,"tag":211,"props":760,"children":762},{"class":213,"line":761},9,[763],{"type":50,"tag":211,"props":764,"children":765},{"emptyLinePlaceholder":571},[766],{"type":56,"value":574},{"type":50,"tag":211,"props":768,"children":770},{"class":213,"line":769},10,[771],{"type":50,"tag":211,"props":772,"children":773},{"style":509},[774],{"type":56,"value":775},"# Larger hardware sample (default 3):\n",{"type":50,"tag":211,"props":777,"children":779},{"class":213,"line":778},11,[780,784,788,792,796,800,804,808,812,816,820,824,828,833],{"type":50,"tag":211,"props":781,"children":782},{"style":230},[783],{"type":56,"value":204},{"type":50,"tag":211,"props":785,"children":786},{"style":235},[787],{"type":56,"value":524},{"type":50,"tag":211,"props":789,"children":790},{"style":235},[791],{"type":56,"value":529},{"type":50,"tag":211,"props":793,"children":794},{"style":224},[795],{"type":56,"value":253},{"type":50,"tag":211,"props":797,"children":798},{"style":235},[799],{"type":56,"value":606},{"type":50,"tag":211,"props":801,"children":802},{"style":218},[803],{"type":56,"value":263},{"type":50,"tag":211,"props":805,"children":806},{"style":224},[807],{"type":56,"value":268},{"type":50,"tag":211,"props":809,"children":810},{"style":235},[811],{"type":56,"value":273},{"type":50,"tag":211,"props":813,"children":814},{"style":224},[815],{"type":56,"value":253},{"type":50,"tag":211,"props":817,"children":818},{"style":235},[819],{"type":56,"value":282},{"type":50,"tag":211,"props":821,"children":822},{"style":218},[823],{"type":56,"value":287},{"type":50,"tag":211,"props":825,"children":826},{"style":224},[827],{"type":56,"value":268},{"type":50,"tag":211,"props":829,"children":830},{"style":235},[831],{"type":56,"value":832}," --sample-nodes",{"type":50,"tag":211,"props":834,"children":836},{"style":835},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[837],{"type":56,"value":838}," 10\n",{"type":50,"tag":211,"props":840,"children":842},{"class":213,"line":841},12,[843],{"type":50,"tag":211,"props":844,"children":845},{"emptyLinePlaceholder":571},[846],{"type":56,"value":574},{"type":50,"tag":211,"props":848,"children":850},{"class":213,"line":849},13,[851],{"type":50,"tag":211,"props":852,"children":853},{"style":509},[854],{"type":56,"value":855},"# Specific node only:\n",{"type":50,"tag":211,"props":857,"children":859},{"class":213,"line":858},14,[860,864,868,872,876,880,884,888,892,896,900,904,908,913],{"type":50,"tag":211,"props":861,"children":862},{"style":230},[863],{"type":56,"value":204},{"type":50,"tag":211,"props":865,"children":866},{"style":235},[867],{"type":56,"value":524},{"type":50,"tag":211,"props":869,"children":870},{"style":235},[871],{"type":56,"value":529},{"type":50,"tag":211,"props":873,"children":874},{"style":224},[875],{"type":56,"value":253},{"type":50,"tag":211,"props":877,"children":878},{"style":235},[879],{"type":56,"value":606},{"type":50,"tag":211,"props":881,"children":882},{"style":218},[883],{"type":56,"value":263},{"type":50,"tag":211,"props":885,"children":886},{"style":224},[887],{"type":56,"value":268},{"type":50,"tag":211,"props":889,"children":890},{"style":235},[891],{"type":56,"value":273},{"type":50,"tag":211,"props":893,"children":894},{"style":224},[895],{"type":56,"value":253},{"type":50,"tag":211,"props":897,"children":898},{"style":235},[899],{"type":56,"value":282},{"type":50,"tag":211,"props":901,"children":902},{"style":218},[903],{"type":56,"value":287},{"type":50,"tag":211,"props":905,"children":906},{"style":224},[907],{"type":56,"value":268},{"type":50,"tag":211,"props":909,"children":910},{"style":235},[911],{"type":56,"value":912}," --node",{"type":50,"tag":211,"props":914,"children":915},{"style":235},[916],{"type":56,"value":917}," i-0abc123def456\n",{"type":50,"tag":59,"props":919,"children":920},{},[921,923,929,931,936,938,944,946,952,953,959,961,966,968,973,975,980],{"type":56,"value":922},"Tags: ",{"type":50,"tag":90,"props":924,"children":926},{"className":925},[],[927],{"type":56,"value":928},"[PASS]",{"type":56,"value":930}," · ",{"type":50,"tag":90,"props":932,"children":934},{"className":933},[],[935],{"type":56,"value":167},{"type":56,"value":937}," (counted in ",{"type":50,"tag":90,"props":939,"children":941},{"className":940},[],[942],{"type":56,"value":943},"Issues Found",{"type":56,"value":945},", has reference pointer) · ",{"type":50,"tag":90,"props":947,"children":949},{"className":948},[],[950],{"type":56,"value":951},"[WARN]",{"type":56,"value":930},{"type":50,"tag":90,"props":954,"children":956},{"className":955},[],[957],{"type":56,"value":958},"[INFO]",{"type":56,"value":960},". Priorities: ",{"type":50,"tag":63,"props":962,"children":963},{},[964],{"type":56,"value":965},"P0",{"type":56,"value":967}," blocks training · ",{"type":50,"tag":63,"props":969,"children":970},{},[971],{"type":56,"value":972},"P1",{"type":56,"value":974}," degraded · ",{"type":50,"tag":63,"props":976,"children":977},{},[978],{"type":56,"value":979},"P2",{"type":56,"value":981}," informational.",{"type":50,"tag":132,"props":983,"children":984},{},[],{"type":50,"tag":136,"props":986,"children":988},{"id":987},"remediation-index",[989],{"type":56,"value":990},"Remediation index",{"type":50,"tag":59,"props":992,"children":993},{},[994,996,1001],{"type":56,"value":995},"Each ",{"type":50,"tag":90,"props":997,"children":999},{"className":998},[],[1000],{"type":56,"value":167},{"type":56,"value":1002}," line in the script already points directly at the right section. This table is a lookup for manual triage.",{"type":50,"tag":1004,"props":1005,"children":1006},"table",{},[1007,1026],{"type":50,"tag":1008,"props":1009,"children":1010},"thead",{},[1011],{"type":50,"tag":1012,"props":1013,"children":1014},"tr",{},[1015,1021],{"type":50,"tag":1016,"props":1017,"children":1018},"th",{},[1019],{"type":56,"value":1020},"Finding",{"type":50,"tag":1016,"props":1022,"children":1023},{},[1024],{"type":56,"value":1025},"Section",{"type":50,"tag":1027,"props":1028,"children":1029},"tbody",{},[1030,1049,1064,1080,1097,1113,1130,1146,1170,1186,1202,1225,1249,1265,1281,1298],{"type":50,"tag":1012,"props":1031,"children":1032},{},[1033,1039],{"type":50,"tag":1034,"props":1035,"children":1036},"td",{},[1037],{"type":56,"value":1038},"SG missing inbound\u002Foutbound self-reference",{"type":50,"tag":1034,"props":1040,"children":1041},{},[1042],{"type":50,"tag":1043,"props":1044,"children":1046},"a",{"href":1045},"references\u002Foperations.md",[1047],{"type":56,"value":1048},"operations.md § 8",{"type":50,"tag":1012,"props":1050,"children":1051},{},[1052,1057],{"type":50,"tag":1034,"props":1053,"children":1054},{},[1055],{"type":56,"value":1056},"Blocking NetworkPolicy \u002F allow-all missing",{"type":50,"tag":1034,"props":1058,"children":1059},{},[1060],{"type":50,"tag":1043,"props":1061,"children":1062},{"href":1045},[1063],{"type":56,"value":1048},{"type":50,"tag":1012,"props":1065,"children":1066},{},[1067,1072],{"type":50,"tag":1034,"props":1068,"children":1069},{},[1070],{"type":56,"value":1071},"Slurm node DOWN \u002F DRAINING \u002F RemoveIPC",{"type":50,"tag":1034,"props":1073,"children":1074},{},[1075],{"type":50,"tag":1043,"props":1076,"children":1077},{"href":1045},[1078],{"type":56,"value":1079},"operations.md § 7",{"type":50,"tag":1012,"props":1081,"children":1082},{},[1083,1088],{"type":50,"tag":1034,"props":1084,"children":1085},{},[1086],{"type":56,"value":1087},"GPU XID \u002F SYSTEM_ERROR \u002F hardware fault",{"type":50,"tag":1034,"props":1089,"children":1090},{},[1091],{"type":50,"tag":1043,"props":1092,"children":1094},{"href":1093},"..\u002Fhyperpod-node-debugger\u002Freferences\u002Fnode-diagnostics-detail.md",[1095],{"type":56,"value":1096},"hyperpod-node-debugger § F \u002F § G",{"type":50,"tag":1012,"props":1098,"children":1099},{},[1100,1105],{"type":50,"tag":1034,"props":1101,"children":1102},{},[1103],{"type":56,"value":1104},"GPU row-remap \u002F DCGM Fail \u002F silent NaNs",{"type":50,"tag":1034,"props":1106,"children":1107},{},[1108],{"type":50,"tag":1043,"props":1109,"children":1110},{"href":1093},[1111],{"type":56,"value":1112},"hyperpod-node-debugger § G.1.a\u002Fb",{"type":50,"tag":1012,"props":1114,"children":1115},{},[1116,1121],{"type":50,"tag":1034,"props":1117,"children":1118},{},[1119],{"type":56,"value":1120},"NCCL timeout \u002F rendezvous \u002F straggler",{"type":50,"tag":1034,"props":1122,"children":1123},{},[1124],{"type":50,"tag":1043,"props":1125,"children":1127},{"href":1126},"references\u002Fdebugging-guide.md",[1128],{"type":56,"value":1129},"debugging-guide.md § 1",{"type":50,"tag":1012,"props":1131,"children":1132},{},[1133,1138],{"type":50,"tag":1034,"props":1134,"children":1135},{},[1136],{"type":56,"value":1137},"EFA configuration \u002F not used",{"type":50,"tag":1034,"props":1139,"children":1140},{},[1141],{"type":50,"tag":1043,"props":1142,"children":1143},{"href":1126},[1144],{"type":56,"value":1145},"debugging-guide.md § 6",{"type":50,"tag":1012,"props":1147,"children":1148},{},[1149,1162],{"type":50,"tag":1034,"props":1150,"children":1151},{},[1152,1154,1160],{"type":56,"value":1153},"EFA TCP fallback (",{"type":50,"tag":90,"props":1155,"children":1157},{"className":1156},[],[1158],{"type":56,"value":1159},"NET\u002FOFI Using TCP",{"type":56,"value":1161},")",{"type":50,"tag":1034,"props":1163,"children":1164},{},[1165],{"type":50,"tag":1043,"props":1166,"children":1167},{"href":1126},[1168],{"type":56,"value":1169},"debugging-guide.md § 13",{"type":50,"tag":1012,"props":1171,"children":1172},{},[1173,1178],{"type":50,"tag":1034,"props":1174,"children":1175},{},[1176],{"type":56,"value":1177},"NCCL version mismatch across pods",{"type":50,"tag":1034,"props":1179,"children":1180},{},[1181],{"type":50,"tag":1043,"props":1182,"children":1183},{"href":1126},[1184],{"type":56,"value":1185},"debugging-guide.md § 10",{"type":50,"tag":1012,"props":1187,"children":1188},{},[1189,1194],{"type":50,"tag":1034,"props":1190,"children":1191},{},[1192],{"type":56,"value":1193},"Container OOM (pod killed, exit 137)",{"type":50,"tag":1034,"props":1195,"children":1196},{},[1197],{"type":50,"tag":1043,"props":1198,"children":1199},{"href":1126},[1200],{"type":56,"value":1201},"debugging-guide.md § 4",{"type":50,"tag":1012,"props":1203,"children":1204},{},[1205,1217],{"type":50,"tag":1034,"props":1206,"children":1207},{},[1208,1210,1216],{"type":56,"value":1209},"GPU OOM (",{"type":50,"tag":90,"props":1211,"children":1213},{"className":1212},[],[1214],{"type":56,"value":1215},"CUDA out of memory",{"type":56,"value":1161},{"type":50,"tag":1034,"props":1218,"children":1219},{},[1220],{"type":50,"tag":1043,"props":1221,"children":1222},{"href":1126},[1223],{"type":56,"value":1224},"debugging-guide.md § 11",{"type":50,"tag":1012,"props":1226,"children":1227},{},[1228,1241],{"type":50,"tag":1034,"props":1229,"children":1230},{},[1231,1233,1239],{"type":56,"value":1232},"RDMA memlock \u002F ",{"type":50,"tag":90,"props":1234,"children":1236},{"className":1235},[],[1237],{"type":56,"value":1238},"\u002Fdev\u002Fshm",{"type":56,"value":1240}," too small",{"type":50,"tag":1034,"props":1242,"children":1243},{},[1244],{"type":50,"tag":1043,"props":1245,"children":1246},{"href":1126},[1247],{"type":56,"value":1248},"debugging-guide.md § 17",{"type":50,"tag":1012,"props":1250,"children":1251},{},[1252,1257],{"type":50,"tag":1034,"props":1253,"children":1254},{},[1255],{"type":56,"value":1256},"MASTER_ADDR DNS \u002F headless Service",{"type":50,"tag":1034,"props":1258,"children":1259},{},[1260],{"type":50,"tag":1043,"props":1261,"children":1262},{"href":1126},[1263],{"type":56,"value":1264},"debugging-guide.md § 12",{"type":50,"tag":1012,"props":1266,"children":1267},{},[1268,1273],{"type":50,"tag":1034,"props":1269,"children":1270},{},[1271],{"type":56,"value":1272},"NVLS \u002F PXN \u002F topology tuning",{"type":50,"tag":1034,"props":1274,"children":1275},{},[1276],{"type":50,"tag":1043,"props":1277,"children":1278},{"href":1126},[1279],{"type":56,"value":1280},"debugging-guide.md § 19",{"type":50,"tag":1012,"props":1282,"children":1283},{},[1284,1289],{"type":50,"tag":1034,"props":1285,"children":1286},{},[1287],{"type":56,"value":1288},"Any NCCL \u002F EFA \u002F rendezvous log pattern",{"type":50,"tag":1034,"props":1290,"children":1291},{},[1292],{"type":50,"tag":1043,"props":1293,"children":1295},{"href":1294},"references\u002Ferror-patterns-quick-ref.md",[1296],{"type":56,"value":1297},"error-patterns-quick-ref.md",{"type":50,"tag":1012,"props":1299,"children":1300},{},[1301,1306],{"type":50,"tag":1034,"props":1302,"children":1303},{},[1304],{"type":56,"value":1305},"Performance \u002F nccl-tests \u002F bandwidth",{"type":50,"tag":1034,"props":1307,"children":1308},{},[1309],{"type":50,"tag":1043,"props":1310,"children":1312},{"href":1311},"references\u002Fperformance-testing.md",[1313],{"type":56,"value":1314},"performance-testing.md",{"type":50,"tag":132,"props":1316,"children":1317},{},[],{"type":50,"tag":136,"props":1319,"children":1321},{"id":1320},"prerequisites",[1322],{"type":56,"value":1323},"Prerequisites",{"type":50,"tag":1325,"props":1326,"children":1327},"ul",{},[1328,1345,1370,1404,1414],{"type":50,"tag":147,"props":1329,"children":1330},{},[1331,1336,1338,1344],{"type":50,"tag":90,"props":1332,"children":1334},{"className":1333},[],[1335],{"type":56,"value":22},{"type":56,"value":1337}," CLI v2.13+ authenticated (",{"type":50,"tag":90,"props":1339,"children":1341},{"className":1340},[],[1342],{"type":56,"value":1343},"aws sts get-caller-identity",{"type":56,"value":1161},{"type":50,"tag":147,"props":1346,"children":1347},{},[1348,1354,1356,1362,1363,1368],{"type":50,"tag":90,"props":1349,"children":1351},{"className":1350},[],[1352],{"type":56,"value":1353},"jq",{"type":56,"value":1355},", ",{"type":50,"tag":90,"props":1357,"children":1359},{"className":1358},[],[1360],{"type":56,"value":1361},"python3",{"type":56,"value":1355},{"type":50,"tag":90,"props":1364,"children":1366},{"className":1365},[],[1367],{"type":56,"value":204},{"type":56,"value":1369}," 4.2+",{"type":50,"tag":147,"props":1371,"children":1372},{},[1373,1379,1381,1387,1389,1395,1397,1403],{"type":50,"tag":90,"props":1374,"children":1376},{"className":1375},[],[1377],{"type":56,"value":1378},"unbuffer",{"type":56,"value":1380}," (from the ",{"type":50,"tag":90,"props":1382,"children":1384},{"className":1383},[],[1385],{"type":56,"value":1386},"expect",{"type":56,"value":1388}," package: ",{"type":50,"tag":90,"props":1390,"children":1392},{"className":1391},[],[1393],{"type":56,"value":1394},"yum install expect",{"type":56,"value":1396}," \u002F ",{"type":50,"tag":90,"props":1398,"children":1400},{"className":1399},[],[1401],{"type":56,"value":1402},"apt install expect",{"type":56,"value":1161},{"type":50,"tag":147,"props":1405,"children":1406},{},[1407,1412],{"type":50,"tag":90,"props":1408,"children":1410},{"className":1409},[],[1411],{"type":56,"value":480},{"type":56,"value":1413}," authenticated to the EKS cluster (K8s checks skipped if absent)",{"type":50,"tag":147,"props":1415,"children":1416},{},[1417,1423],{"type":50,"tag":90,"props":1418,"children":1420},{"className":1419},[],[1421],{"type":56,"value":1422},"session-manager-plugin",{"type":56,"value":1424}," for on-node hardware checks",{"type":50,"tag":136,"props":1426,"children":1428},{"id":1427},"defaults",[1429],{"type":56,"value":1430},"Defaults",{"type":50,"tag":1325,"props":1432,"children":1433},{},[1434,1460,1477,1494,1527,1537],{"type":50,"tag":147,"props":1435,"children":1436},{},[1437,1442,1444,1450,1452,1458],{"type":50,"tag":63,"props":1438,"children":1439},{},[1440],{"type":56,"value":1441},"Region",{"type":56,"value":1443}," — required: pass ",{"type":50,"tag":90,"props":1445,"children":1447},{"className":1446},[],[1448],{"type":56,"value":1449},"--region",{"type":56,"value":1451}," or set ",{"type":50,"tag":90,"props":1453,"children":1455},{"className":1454},[],[1456],{"type":56,"value":1457},"$AWS_DEFAULT_REGION",{"type":56,"value":1459},".",{"type":50,"tag":147,"props":1461,"children":1462},{},[1463,1468,1470,1476],{"type":50,"tag":63,"props":1464,"children":1465},{},[1466],{"type":56,"value":1467},"Orchestrator",{"type":56,"value":1469}," — auto-detected; override with ",{"type":50,"tag":90,"props":1471,"children":1473},{"className":1472},[],[1474],{"type":56,"value":1475},"--orchestrator eks|slurm",{"type":56,"value":1459},{"type":50,"tag":147,"props":1478,"children":1479},{},[1480,1485,1487,1493],{"type":50,"tag":63,"props":1481,"children":1482},{},[1483],{"type":56,"value":1484},"Namespace \u002F job (EKS)",{"type":56,"value":1486}," — all namespaces; scope with ",{"type":50,"tag":90,"props":1488,"children":1490},{"className":1489},[],[1491],{"type":56,"value":1492},"--namespace \u003CNS> --job \u003CJOB>",{"type":56,"value":1459},{"type":50,"tag":147,"props":1495,"children":1496},{},[1497,1502,1504,1510,1512,1517,1519,1525],{"type":50,"tag":63,"props":1498,"children":1499},{},[1500],{"type":56,"value":1501},"Hardware sampling",{"type":56,"value":1503}," — 3 nodes over SSM (capped at 50). ",{"type":50,"tag":90,"props":1505,"children":1507},{"className":1506},[],[1508],{"type":56,"value":1509},"--node \u003CID>",{"type":56,"value":1511}," for a specific node. Node probes run ",{"type":50,"tag":63,"props":1513,"children":1514},{},[1515],{"type":56,"value":1516},"serially",{"type":56,"value":1518}," (180 s per node): ",{"type":50,"tag":90,"props":1520,"children":1522},{"className":1521},[],[1523],{"type":56,"value":1524},"--sample-nodes 10",{"type":56,"value":1526}," can take ~30 min.",{"type":50,"tag":147,"props":1528,"children":1529},{},[1530,1535],{"type":50,"tag":63,"props":1531,"children":1532},{},[1533],{"type":56,"value":1534},"CloudWatch window",{"type":56,"value":1536}," — last 2 hours.",{"type":50,"tag":147,"props":1538,"children":1539},{},[1540,1545,1547,1553],{"type":50,"tag":63,"props":1541,"children":1542},{},[1543],{"type":56,"value":1544},"Colors",{"type":56,"value":1546}," — auto-disabled on non-TTY or ",{"type":50,"tag":90,"props":1548,"children":1550},{"className":1549},[],[1551],{"type":56,"value":1552},"TERM=dumb",{"type":56,"value":1459},{"type":50,"tag":136,"props":1555,"children":1557},{"id":1556},"error-handling",[1558],{"type":56,"value":1559},"Error handling",{"type":50,"tag":1004,"props":1561,"children":1562},{},[1563,1584],{"type":50,"tag":1008,"props":1564,"children":1565},{},[1566],{"type":50,"tag":1012,"props":1567,"children":1568},{},[1569,1574,1579],{"type":50,"tag":1016,"props":1570,"children":1571},{},[1572],{"type":56,"value":1573},"Failure",{"type":50,"tag":1016,"props":1575,"children":1576},{},[1577],{"type":56,"value":1578},"Script",{"type":50,"tag":1016,"props":1580,"children":1581},{},[1582],{"type":56,"value":1583},"Tell the customer",{"type":50,"tag":1027,"props":1585,"children":1586},{},[1587,1610,1648,1666,1696,1714,1740,1758],{"type":50,"tag":1012,"props":1588,"children":1589},{},[1590,1600,1605],{"type":50,"tag":1034,"props":1591,"children":1592},{},[1593,1598],{"type":50,"tag":90,"props":1594,"children":1596},{"className":1595},[],[1597],{"type":56,"value":1343},{"type":56,"value":1599}," fails",{"type":50,"tag":1034,"props":1601,"children":1602},{},[1603],{"type":56,"value":1604},"Exit 1 with the AWS error",{"type":50,"tag":1034,"props":1606,"children":1607},{},[1608],{"type":56,"value":1609},"\"Fix AWS credentials and rerun.\"",{"type":50,"tag":1012,"props":1611,"children":1612},{},[1613,1624,1635],{"type":50,"tag":1034,"props":1614,"children":1615},{},[1616,1622],{"type":50,"tag":90,"props":1617,"children":1619},{"className":1618},[],[1620],{"type":56,"value":1621},"describe-cluster",{"type":56,"value":1623}," AccessDenied",{"type":50,"tag":1034,"props":1625,"children":1626},{},[1627,1629],{"type":56,"value":1628},"Warn, add ",{"type":50,"tag":90,"props":1630,"children":1632},{"className":1631},[],[1633],{"type":56,"value":1634},"Missing IAM for sagemaker:DescribeCluster",{"type":50,"tag":1034,"props":1636,"children":1637},{},[1638,1640,1646],{"type":56,"value":1639},"\"Grant ",{"type":50,"tag":90,"props":1641,"children":1643},{"className":1642},[],[1644],{"type":56,"value":1645},"sagemaker:DescribeCluster",{"type":56,"value":1647}," (operations.md § 2).\"",{"type":50,"tag":1012,"props":1649,"children":1650},{},[1651,1656,1661],{"type":50,"tag":1034,"props":1652,"children":1653},{},[1654],{"type":56,"value":1655},"Cluster not found",{"type":50,"tag":1034,"props":1657,"children":1658},{},[1659],{"type":56,"value":1660},"Exit 1 after listing region's clusters",{"type":50,"tag":1034,"props":1662,"children":1663},{},[1664],{"type":56,"value":1665},"\"Confirm HyperPod cluster name and region.\"",{"type":50,"tag":1012,"props":1667,"children":1668},{},[1669,1679,1684],{"type":50,"tag":1034,"props":1670,"children":1671},{},[1672,1677],{"type":50,"tag":90,"props":1673,"children":1675},{"className":1674},[],[1676],{"type":56,"value":480},{"type":56,"value":1678}," absent \u002F unauthenticated",{"type":50,"tag":1034,"props":1680,"children":1681},{},[1682],{"type":56,"value":1683},"Warn, skip K8s checks",{"type":50,"tag":1034,"props":1685,"children":1686},{},[1687,1688,1694],{"type":56,"value":369},{"type":50,"tag":90,"props":1689,"children":1691},{"className":1690},[],[1692],{"type":56,"value":1693},"aws eks update-kubeconfig --name \u003CEKS> --region \u003CR>",{"type":56,"value":1695},".\"",{"type":50,"tag":1012,"props":1697,"children":1698},{},[1699,1704,1709],{"type":50,"tag":1034,"props":1700,"children":1701},{},[1702],{"type":56,"value":1703},"SSM plugin absent",{"type":50,"tag":1034,"props":1705,"children":1706},{},[1707],{"type":56,"value":1708},"Warn, skip on-node hardware checks",{"type":50,"tag":1034,"props":1710,"children":1711},{},[1712],{"type":56,"value":1713},"\"Install session-manager-plugin.\"",{"type":50,"tag":1012,"props":1715,"children":1716},{},[1717,1722,1727],{"type":50,"tag":1034,"props":1718,"children":1719},{},[1720],{"type":56,"value":1721},"SSM times out (180s)",{"type":50,"tag":1034,"props":1723,"children":1724},{},[1725],{"type":56,"value":1726},"Partial output, mark node unreachable",{"type":50,"tag":1034,"props":1728,"children":1729},{},[1730,1732,1738],{"type":56,"value":1731},"\"Rerun with ",{"type":50,"tag":90,"props":1733,"children":1735},{"className":1734},[],[1736],{"type":56,"value":1737},"--node \u003CID> --sample-nodes 1",{"type":56,"value":1739},"; check SSM agent on the node.\"",{"type":50,"tag":1012,"props":1741,"children":1742},{},[1743,1748,1753],{"type":50,"tag":1034,"props":1744,"children":1745},{},[1746],{"type":56,"value":1747},"CloudWatch log group not found",{"type":50,"tag":1034,"props":1749,"children":1750},{},[1751],{"type":56,"value":1752},"Skip CloudWatch scan",{"type":50,"tag":1034,"props":1754,"children":1755},{},[1756],{"type":56,"value":1757},"\"Enable CloudWatch on the cluster (operations.md § 4).\"",{"type":50,"tag":1012,"props":1759,"children":1760},{},[1761,1766,1771],{"type":50,"tag":1034,"props":1762,"children":1763},{},[1764],{"type":56,"value":1765},"Cluster events API throttled",{"type":50,"tag":1034,"props":1767,"children":1768},{},[1769],{"type":56,"value":1770},"Warn, continue with partial data",{"type":50,"tag":1034,"props":1772,"children":1773},{},[1774],{"type":56,"value":1775},"\"Rerun later — script is idempotent.\"",{"type":50,"tag":59,"props":1777,"children":1778},{},[1779,1781,1787,1789,1795],{"type":56,"value":1780},"Exit codes: ",{"type":50,"tag":90,"props":1782,"children":1784},{"className":1783},[],[1785],{"type":56,"value":1786},"0",{"type":56,"value":1788}," diagnostic complete · ",{"type":50,"tag":90,"props":1790,"children":1792},{"className":1791},[],[1793],{"type":56,"value":1794},"1",{"type":56,"value":1796}," fatal prerequisite missing or cluster unreachable.",{"type":50,"tag":136,"props":1798,"children":1800},{"id":1799},"iam-permissions",[1801],{"type":56,"value":1802},"IAM permissions",{"type":50,"tag":59,"props":1804,"children":1805},{},[1806,1808,1814,1816,1822,1824,1830,1832,1838,1839,1845,1847,1853],{"type":56,"value":1807},"Full policy + RBAC in ",{"type":50,"tag":1043,"props":1809,"children":1811},{"href":1810},"references\u002Foperations.md#2-iam",[1812],{"type":56,"value":1813},"operations.md § 2",{"type":56,"value":1815},". SSM on HyperPod uses ",{"type":50,"tag":90,"props":1817,"children":1819},{"className":1818},[],[1820],{"type":56,"value":1821},"start-session",{"type":56,"value":1823}," against ",{"type":50,"tag":90,"props":1825,"children":1827},{"className":1826},[],[1828],{"type":56,"value":1829},"sagemaker-cluster:\u003Ccluster-id>_\u003Cgroup>-\u003Ciid>",{"type":56,"value":1831}," targets — grant ",{"type":50,"tag":90,"props":1833,"children":1835},{"className":1834},[],[1836],{"type":56,"value":1837},"ssm:StartSession",{"type":56,"value":1396},{"type":50,"tag":90,"props":1840,"children":1842},{"className":1841},[],[1843],{"type":56,"value":1844},"ssm:TerminateSession",{"type":56,"value":1846},", not ",{"type":50,"tag":90,"props":1848,"children":1850},{"className":1849},[],[1851],{"type":56,"value":1852},"ssm:SendCommand",{"type":56,"value":1459},{"type":50,"tag":136,"props":1855,"children":1857},{"id":1856},"scale-strategy",[1858],{"type":56,"value":1859},"Scale strategy",{"type":50,"tag":1004,"props":1861,"children":1862},{},[1863,1884],{"type":50,"tag":1008,"props":1864,"children":1865},{},[1866],{"type":50,"tag":1012,"props":1867,"children":1868},{},[1869,1874,1879],{"type":50,"tag":1016,"props":1870,"children":1871},{},[1872],{"type":56,"value":1873},"Scope",{"type":50,"tag":1016,"props":1875,"children":1876},{},[1877],{"type":56,"value":1878},"Method",{"type":50,"tag":1016,"props":1880,"children":1881},{},[1882],{"type":56,"value":1883},"Coverage",{"type":50,"tag":1027,"props":1885,"children":1886},{},[1887,1911,1932,1958],{"type":50,"tag":1012,"props":1888,"children":1889},{},[1890,1895,1906],{"type":50,"tag":1034,"props":1891,"children":1892},{},[1893],{"type":56,"value":1894},"All nodes",{"type":50,"tag":1034,"props":1896,"children":1897},{},[1898,1904],{"type":50,"tag":90,"props":1899,"children":1901},{"className":1900},[],[1902],{"type":56,"value":1903},"sagemaker:ListClusterNodes",{"type":56,"value":1905}," (paginated)",{"type":50,"tag":1034,"props":1907,"children":1908},{},[1909],{"type":56,"value":1910},"100% nodes",{"type":50,"tag":1012,"props":1912,"children":1913},{},[1914,1919,1927],{"type":50,"tag":1034,"props":1915,"children":1916},{},[1917],{"type":56,"value":1918},"All K8s objects",{"type":50,"tag":1034,"props":1920,"children":1921},{},[1922],{"type":50,"tag":90,"props":1923,"children":1925},{"className":1924},[],[1926],{"type":56,"value":480},{"type":50,"tag":1034,"props":1928,"children":1929},{},[1930],{"type":56,"value":1931},"100% pods\u002Fnodes\u002Fpolicies",{"type":50,"tag":1012,"props":1933,"children":1934},{},[1935,1940,1953],{"type":50,"tag":1034,"props":1936,"children":1937},{},[1938],{"type":56,"value":1939},"Hardware",{"type":50,"tag":1034,"props":1941,"children":1942},{},[1943,1945,1951],{"type":56,"value":1944},"SSM ",{"type":50,"tag":90,"props":1946,"children":1948},{"className":1947},[],[1949],{"type":56,"value":1950},"--sample-nodes N",{"type":56,"value":1952}," (default 3)",{"type":50,"tag":1034,"props":1954,"children":1955},{},[1956],{"type":56,"value":1957},"Sampled",{"type":50,"tag":1012,"props":1959,"children":1960},{},[1961,1966,1971],{"type":50,"tag":1034,"props":1962,"children":1963},{},[1964],{"type":56,"value":1965},"Node logs",{"type":50,"tag":1034,"props":1967,"children":1968},{},[1969],{"type":56,"value":1970},"CloudWatch",{"type":50,"tag":1034,"props":1972,"children":1973},{},[1974],{"type":56,"value":1910},{"type":50,"tag":59,"props":1976,"children":1977},{},[1978,1983,1985,1991,1993,1999,2001,2007,2009,2015,2017,2023,2025,2031],{"type":50,"tag":63,"props":1979,"children":1980},{},[1981],{"type":56,"value":1982},"Large clusters:",{"type":56,"value":1984}," the PyTorch NCCL backend defaults to a 10-minute collective-op timeout (per the PyTorch distributed docs). Large clusters routinely exceed that on first rendezvous; raise it via ",{"type":50,"tag":90,"props":1986,"children":1988},{"className":1987},[],[1989],{"type":56,"value":1990},"torch.distributed.init_process_group(timeout=timedelta(seconds=\u003CN>))",{"type":56,"value":1992},". HyperPod support has also observed NCCL topology-graph-search hangs on 256+ node clusters when ",{"type":50,"tag":90,"props":1994,"children":1996},{"className":1995},[],[1997],{"type":56,"value":1998},"memlock",{"type":56,"value":2000}," is ",{"type":50,"tag":90,"props":2002,"children":2004},{"className":2003},[],[2005],{"type":56,"value":2006},"unlimited",{"type":56,"value":2008},"; using a large fixed memlock (e.g. ",{"type":50,"tag":90,"props":2010,"children":2012},{"className":2011},[],[2013],{"type":56,"value":2014},"8388608",{"type":56,"value":2016},") in pod ",{"type":50,"tag":90,"props":2018,"children":2020},{"className":2019},[],[2021],{"type":56,"value":2022},"securityContext",{"type":56,"value":2024}," or ",{"type":50,"tag":90,"props":2026,"children":2028},{"className":2027},[],[2029],{"type":56,"value":2030},"\u002Fetc\u002Fsecurity\u002Flimits.conf",{"type":56,"value":2032}," has cleared these in field cases. This memlock pattern is a field observation, not AWS- or NCCL-documented behavior.",{"type":50,"tag":59,"props":2034,"children":2035},{},[2036,2038,2043,2044,2049,2051,2056,2058,2063],{"type":56,"value":2037},"For ",{"type":50,"tag":63,"props":2039,"children":2040},{},[2041],{"type":56,"value":2042},"FSDP",{"type":56,"value":1355},{"type":50,"tag":63,"props":2045,"children":2046},{},[2047],{"type":56,"value":2048},"DeepSpeed",{"type":56,"value":2050},", or ",{"type":50,"tag":63,"props":2052,"children":2053},{},[2054],{"type":56,"value":2055},"Megatron-LM",{"type":56,"value":2057}," tuning: ",{"type":50,"tag":1043,"props":2059,"children":2060},{"href":1126},[2061],{"type":56,"value":2062},"debugging-guide.md § 18",{"type":56,"value":1459},{"type":50,"tag":136,"props":2065,"children":2067},{"id":2066},"skill-delegation",[2068],{"type":56,"value":2069},"Skill delegation",{"type":50,"tag":1004,"props":2071,"children":2072},{},[2073,2089],{"type":50,"tag":1008,"props":2074,"children":2075},{},[2076],{"type":50,"tag":1012,"props":2077,"children":2078},{},[2079,2084],{"type":50,"tag":1016,"props":2080,"children":2081},{},[2082],{"type":56,"value":2083},"Need",{"type":50,"tag":1016,"props":2085,"children":2086},{},[2087],{"type":56,"value":2088},"Use",{"type":50,"tag":1027,"props":2090,"children":2091},{},[2092,2118,2134,2151,2169,2186,2203,2220],{"type":50,"tag":1012,"props":2093,"children":2094},{},[2095,2100],{"type":50,"tag":1034,"props":2096,"children":2097},{},[2098],{"type":56,"value":2099},"Cluster creation \u002F deployment failures",{"type":50,"tag":1034,"props":2101,"children":2102},{},[2103,2109,2111,2117],{"type":50,"tag":90,"props":2104,"children":2106},{"className":2105},[],[2107],{"type":56,"value":2108},"hyperpod-cluster-debugger",{"type":56,"value":2110}," (§ A \u002F B \u002F C \u002F H + ",{"type":50,"tag":90,"props":2112,"children":2114},{"className":2113},[],[2115],{"type":56,"value":2116},"--validate",{"type":56,"value":1161},{"type":50,"tag":1012,"props":2119,"children":2120},{},[2121,2126],{"type":50,"tag":1034,"props":2122,"children":2123},{},[2124],{"type":56,"value":2125},"Post-deployment cluster-wide management",{"type":50,"tag":1034,"props":2127,"children":2128},{},[2129],{"type":50,"tag":90,"props":2130,"children":2132},{"className":2131},[],[2133],{"type":56,"value":2108},{"type":50,"tag":1012,"props":2135,"children":2136},{},[2137,2142],{"type":50,"tag":1034,"props":2138,"children":2139},{},[2140],{"type":56,"value":2141},"Per-node issues (disk, lifecycle, hardware)",{"type":50,"tag":1034,"props":2143,"children":2144},{},[2145],{"type":50,"tag":90,"props":2146,"children":2148},{"className":2147},[],[2149],{"type":56,"value":2150},"hyperpod-node-debugger",{"type":50,"tag":1012,"props":2152,"children":2153},{},[2154,2159],{"type":50,"tag":1034,"props":2155,"children":2156},{},[2157],{"type":56,"value":2158},"Trainium\u002FInferentia collective-comm (AWS Neuron Collectives, not NCCL)",{"type":50,"tag":1034,"props":2160,"children":2161},{},[2162,2167],{"type":50,"tag":90,"props":2163,"children":2165},{"className":2164},[],[2166],{"type":56,"value":2150},{"type":56,"value":2168}," § G.2",{"type":50,"tag":1012,"props":2170,"children":2171},{},[2172,2177],{"type":50,"tag":1034,"props":2173,"children":2174},{},[2175],{"type":56,"value":2176},"Shell on nodes",{"type":50,"tag":1034,"props":2178,"children":2179},{},[2180],{"type":50,"tag":90,"props":2181,"children":2183},{"className":2182},[],[2184],{"type":56,"value":2185},"hyperpod-ssm",{"type":50,"tag":1012,"props":2187,"children":2188},{},[2189,2194],{"type":50,"tag":1034,"props":2190,"children":2191},{},[2192],{"type":56,"value":2193},"Version comparison across nodes",{"type":50,"tag":1034,"props":2195,"children":2196},{},[2197],{"type":50,"tag":90,"props":2198,"children":2200},{"className":2199},[],[2201],{"type":56,"value":2202},"hyperpod-version-checker",{"type":50,"tag":1012,"props":2204,"children":2205},{},[2206,2211],{"type":50,"tag":1034,"props":2207,"children":2208},{},[2209],{"type":56,"value":2210},"Diagnostic bundle for AWS Support",{"type":50,"tag":1034,"props":2212,"children":2213},{},[2214],{"type":50,"tag":90,"props":2215,"children":2217},{"className":2216},[],[2218],{"type":56,"value":2219},"hyperpod-issue-report",{"type":50,"tag":1012,"props":2221,"children":2222},{},[2223,2228],{"type":50,"tag":1034,"props":2224,"children":2225},{},[2226],{"type":56,"value":2227},"MFU \u002F performance degradation",{"type":50,"tag":1034,"props":2229,"children":2230},{},[2231],{"type":50,"tag":90,"props":2232,"children":2234},{"className":2233},[],[2235],{"type":56,"value":2236},"hyperpod-mfu-debugger",{"type":50,"tag":136,"props":2238,"children":2240},{"id":2239},"escalate-to-aws-support",[2241],{"type":56,"value":2242},"Escalate to AWS Support",{"type":50,"tag":59,"props":2244,"children":2245},{},[2246],{"type":56,"value":2247},"Escalate when:",{"type":50,"tag":143,"props":2249,"children":2250},{},[2251,2256,2261,2272,2277],{"type":50,"tag":147,"props":2252,"children":2253},{},[2254],{"type":56,"value":2255},"All SG rules correct, EFA verified on-node, but NCCL still times out.",{"type":50,"tag":147,"props":2257,"children":2258},{},[2259],{"type":56,"value":2260},"Hardware checks pass on all nodes but AllReduce still hangs.",{"type":50,"tag":147,"props":2262,"children":2263},{},[2264,2270],{"type":50,"tag":90,"props":2265,"children":2267},{"className":2266},[],[2268],{"type":56,"value":2269},"Issues Found: 0",{"type":56,"value":2271}," but training still fails.",{"type":50,"tag":147,"props":2273,"children":2274},{},[2275],{"type":56,"value":2276},"GPU XID errors persist after node replacement.",{"type":50,"tag":147,"props":2278,"children":2279},{},[2280],{"type":56,"value":2281},"Collective-op timeout raised and memlock workaround applied but large-cluster rendezvous still hangs.",{"type":50,"tag":2283,"props":2284,"children":2286},"h3",{"id":2285},"before-opening-the-case",[2287],{"type":56,"value":2288},"Before opening the case",{"type":50,"tag":200,"props":2290,"children":2292},{"className":202,"code":2291,"language":204,"meta":205,"style":205},"# 1. Cluster identity + status\naws sagemaker describe-cluster --cluster-name \u003CC> --region \u003CR>\n\n# 2. Full NCCL diagnostic (sample more nodes for escalation)\nbash scripts\u002Fnccl-diagnose.sh --cluster \u003CC> --region \u003CR> --sample-nodes 10 > nccl-diag.txt\n\n# 3. Per-node log\u002Fconfig bundle to S3 (delegates to hyperpod-issue-report)\n#    See skills\u002Fhyperpod-issue-report\u002FSKILL.md for the exact invocation.\n",[2293],{"type":50,"tag":90,"props":2294,"children":2295},{"__ignoreMap":205},[2296,2304,2353,2360,2368,2430,2437,2445],{"type":50,"tag":211,"props":2297,"children":2298},{"class":213,"line":214},[2299],{"type":50,"tag":211,"props":2300,"children":2301},{"style":509},[2302],{"type":56,"value":2303},"# 1. Cluster identity + status\n",{"type":50,"tag":211,"props":2305,"children":2306},{"class":213,"line":299},[2307,2311,2315,2319,2323,2327,2332,2336,2340,2344,2349],{"type":50,"tag":211,"props":2308,"children":2309},{"style":230},[2310],{"type":56,"value":22},{"type":50,"tag":211,"props":2312,"children":2313},{"style":235},[2314],{"type":56,"value":238},{"type":50,"tag":211,"props":2316,"children":2317},{"style":235},[2318],{"type":56,"value":243},{"type":50,"tag":211,"props":2320,"children":2321},{"style":235},[2322],{"type":56,"value":248},{"type":50,"tag":211,"props":2324,"children":2325},{"style":224},[2326],{"type":56,"value":253},{"type":50,"tag":211,"props":2328,"children":2329},{"style":218},[2330],{"type":56,"value":2331},"C",{"type":50,"tag":211,"props":2333,"children":2334},{"style":224},[2335],{"type":56,"value":268},{"type":50,"tag":211,"props":2337,"children":2338},{"style":235},[2339],{"type":56,"value":273},{"type":50,"tag":211,"props":2341,"children":2342},{"style":224},[2343],{"type":56,"value":253},{"type":50,"tag":211,"props":2345,"children":2346},{"style":218},[2347],{"type":56,"value":2348},"R",{"type":50,"tag":211,"props":2350,"children":2351},{"style":224},[2352],{"type":56,"value":471},{"type":50,"tag":211,"props":2354,"children":2355},{"class":213,"line":338},[2356],{"type":50,"tag":211,"props":2357,"children":2358},{"emptyLinePlaceholder":571},[2359],{"type":56,"value":574},{"type":50,"tag":211,"props":2361,"children":2362},{"class":213,"line":417},[2363],{"type":50,"tag":211,"props":2364,"children":2365},{"style":509},[2366],{"type":56,"value":2367},"# 2. Full NCCL diagnostic (sample more nodes for escalation)\n",{"type":50,"tag":211,"props":2369,"children":2370},{"class":213,"line":474},[2371,2375,2379,2383,2387,2391,2395,2399,2403,2407,2411,2415,2420,2425],{"type":50,"tag":211,"props":2372,"children":2373},{"style":230},[2374],{"type":56,"value":204},{"type":50,"tag":211,"props":2376,"children":2377},{"style":235},[2378],{"type":56,"value":524},{"type":50,"tag":211,"props":2380,"children":2381},{"style":235},[2382],{"type":56,"value":529},{"type":50,"tag":211,"props":2384,"children":2385},{"style":224},[2386],{"type":56,"value":253},{"type":50,"tag":211,"props":2388,"children":2389},{"style":218},[2390],{"type":56,"value":2331},{"type":50,"tag":211,"props":2392,"children":2393},{"style":224},[2394],{"type":56,"value":268},{"type":50,"tag":211,"props":2396,"children":2397},{"style":235},[2398],{"type":56,"value":273},{"type":50,"tag":211,"props":2400,"children":2401},{"style":224},[2402],{"type":56,"value":253},{"type":50,"tag":211,"props":2404,"children":2405},{"style":218},[2406],{"type":56,"value":2348},{"type":50,"tag":211,"props":2408,"children":2409},{"style":224},[2410],{"type":56,"value":268},{"type":50,"tag":211,"props":2412,"children":2413},{"style":235},[2414],{"type":56,"value":832},{"type":50,"tag":211,"props":2416,"children":2417},{"style":835},[2418],{"type":56,"value":2419}," 10",{"type":50,"tag":211,"props":2421,"children":2422},{"style":224},[2423],{"type":56,"value":2424}," >",{"type":50,"tag":211,"props":2426,"children":2427},{"style":235},[2428],{"type":56,"value":2429}," nccl-diag.txt\n",{"type":50,"tag":211,"props":2431,"children":2432},{"class":213,"line":682},[2433],{"type":50,"tag":211,"props":2434,"children":2435},{"emptyLinePlaceholder":571},[2436],{"type":56,"value":574},{"type":50,"tag":211,"props":2438,"children":2439},{"class":213,"line":690},[2440],{"type":50,"tag":211,"props":2441,"children":2442},{"style":509},[2443],{"type":56,"value":2444},"# 3. Per-node log\u002Fconfig bundle to S3 (delegates to hyperpod-issue-report)\n",{"type":50,"tag":211,"props":2446,"children":2447},{"class":213,"line":699},[2448],{"type":50,"tag":211,"props":2449,"children":2450},{"style":509},[2451],{"type":56,"value":2452},"#    See skills\u002Fhyperpod-issue-report\u002FSKILL.md for the exact invocation.\n",{"type":50,"tag":2283,"props":2454,"children":2456},{"id":2455},"include-in-the-case",[2457],{"type":56,"value":2458},"Include in the case",{"type":50,"tag":1325,"props":2460,"children":2461},{},[2462,2467,2472,2477,2482,2487,2498,2510],{"type":50,"tag":147,"props":2463,"children":2464},{},[2465],{"type":56,"value":2466},"Cluster name + ARN and AWS region",{"type":50,"tag":147,"props":2468,"children":2469},{},[2470],{"type":56,"value":2471},"Orchestrator (EKS or Slurm) and EKS cluster name \u002F Slurm controller node",{"type":50,"tag":147,"props":2473,"children":2474},{},[2475],{"type":56,"value":2476},"Timestamp window (UTC start \u002F end) of the failure",{"type":50,"tag":147,"props":2478,"children":2479},{},[2480],{"type":56,"value":2481},"Exact NCCL \u002F libfabric error strings (copy verbatim from pod logs or journalctl)",{"type":50,"tag":147,"props":2483,"children":2484},{},[2485],{"type":56,"value":2486},"Affected instance IDs \u002F node names \u002F pod names \u002F namespace \u002F job name",{"type":50,"tag":147,"props":2488,"children":2489},{},[2490,2496],{"type":50,"tag":90,"props":2491,"children":2493},{"className":2492},[],[2494],{"type":56,"value":2495},"nccl-diag.txt",{"type":56,"value":2497}," from step 2 above",{"type":50,"tag":147,"props":2499,"children":2500},{},[2501,2503,2508],{"type":56,"value":2502},"S3 URI of the ",{"type":50,"tag":90,"props":2504,"children":2506},{"className":2505},[],[2507],{"type":56,"value":2219},{"type":56,"value":2509}," bundle from step 3",{"type":50,"tag":147,"props":2511,"children":2512},{},[2513,2515,2521],{"type":56,"value":2514},"NCCL env vars in effect (",{"type":50,"tag":90,"props":2516,"children":2518},{"className":2517},[],[2519],{"type":56,"value":2520},"printenv | grep -E '^NCCL|^FI_|^TORCH_'",{"type":56,"value":2522}," from one pod)",{"type":50,"tag":136,"props":2524,"children":2526},{"id":2525},"references",[2527],{"type":56,"value":2528},"References",{"type":50,"tag":1325,"props":2530,"children":2531},{},[2532,2541,2551,2560],{"type":50,"tag":147,"props":2533,"children":2534},{},[2535,2539],{"type":50,"tag":1043,"props":2536,"children":2537},{"href":1294},[2538],{"type":56,"value":1297},{"type":56,"value":2540}," — log pattern → code → fix table",{"type":50,"tag":147,"props":2542,"children":2543},{},[2544,2549],{"type":50,"tag":1043,"props":2545,"children":2546},{"href":1126},[2547],{"type":56,"value":2548},"debugging-guide.md",{"type":56,"value":2550}," — per-scenario procedures (21 sections incl. NVLS\u002FPXN\u002Ftopology)",{"type":50,"tag":147,"props":2552,"children":2553},{},[2554,2558],{"type":50,"tag":1043,"props":2555,"children":2556},{"href":1311},[2557],{"type":56,"value":1314},{"type":56,"value":2559}," — nccl-tests, bandwidth thresholds, straggler detection",{"type":50,"tag":147,"props":2561,"children":2562},{},[2563,2568],{"type":50,"tag":1043,"props":2564,"children":2565},{"href":1045},[2566],{"type":56,"value":2567},"operations.md",{"type":56,"value":2569}," — IAM, SSM format, CloudWatch, env-var reference, node labels, Slurm ops, remediations",{"type":50,"tag":2571,"props":2572,"children":2573},"style",{},[2574],{"type":56,"value":2575},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2577,"total":2751},[2578,2593,2614,2624,2637,2650,2660,2670,2691,2706,2721,2736],{"slug":2579,"name":2579,"fn":2580,"description":2581,"org":2582,"tags":2583,"stars":2590,"repoUrl":2591,"updatedAt":2592},"agentcore-investigation","investigate Bedrock AgentCore runtime sessions","Investigate Bedrock AgentCore runtime sessions via CloudWatch Logs Insights — resolve session\u002Ftrace IDs, query OTEL spans, filter noise, build timelines. Use when debugging AgentCore agent sessions, tracing tool calls, or analyzing latency.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2584,2585,2586,2589],{"name":21,"slug":22,"type":16},{"name":24,"slug":25,"type":16},{"name":2587,"slug":2588,"type":16},"Logs","logs",{"name":14,"slug":15,"type":16},9427,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fmcp","2026-07-12T08:37:22.601527",{"slug":2594,"name":2595,"fn":2596,"description":2597,"org":2598,"tags":2599,"stars":2590,"repoUrl":2591,"updatedAt":2613},"amazon-aurora-dsql","amazon aurora dsql","build applications with Aurora DSQL","Build with Aurora DSQL — manage schemas, execute queries, handle migrations, diagnose query plans, load data, and develop applications with a serverless, distributed SQL database. Covers IAM auth, multi-tenant patterns, MySQL-to-DSQL and PostgreSQL-to-DSQL schema conversion, FK replacement code generation, OCC retry patterns, ORM migration (Django\u002FHibernate\u002FRails), DDL operations, query plan explainability, SQL compatibility validation, and bulk data loading. Triggers on phrases like: DSQL, Aurora DSQL, create DSQL table, DSQL schema, migrate to DSQL, distributed SQL database, serverless PostgreSQL-compatible database, DSQL query plan, DSQL EXPLAIN ANALYZE, why is my DSQL query slow, DSQL foreign key, DSQL OCC retry, DSQL multi-region, load into DSQL, load CSV into DSQL, bulk load DSQL, aurora-dsql-loader.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2600,2603,2604,2607,2610],{"name":2601,"slug":2602,"type":16},"Aurora","aurora",{"name":21,"slug":22,"type":16},{"name":2605,"slug":2606,"type":16},"Database","database",{"name":2608,"slug":2609,"type":16},"Serverless","serverless",{"name":2611,"slug":2612,"type":16},"SQL","sql","2026-07-12T08:36:45.053393",{"slug":2615,"name":2616,"fn":2596,"description":2597,"org":2617,"tags":2618,"stars":2590,"repoUrl":2591,"updatedAt":2623},"aurora-dsql","aurora dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2619,2620,2621,2622],{"name":21,"slug":22,"type":16},{"name":2605,"slug":2606,"type":16},{"name":2608,"slug":2609,"type":16},{"name":2611,"slug":2612,"type":16},"2026-07-12T08:36:42.694299",{"slug":2625,"name":2626,"fn":2596,"description":2597,"org":2627,"tags":2628,"stars":2590,"repoUrl":2591,"updatedAt":2636},"aws-dsql","aws dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2629,2630,2631,2634,2635],{"name":21,"slug":22,"type":16},{"name":2605,"slug":2606,"type":16},{"name":2632,"slug":2633,"type":16},"Migration","migration",{"name":2608,"slug":2609,"type":16},{"name":2611,"slug":2612,"type":16},"2026-07-12T08:36:38.584057",{"slug":2638,"name":2639,"fn":2596,"description":2597,"org":2640,"tags":2641,"stars":2590,"repoUrl":2591,"updatedAt":2649},"distributed-postgres","distributed postgres",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2642,2643,2644,2647,2648],{"name":21,"slug":22,"type":16},{"name":2605,"slug":2606,"type":16},{"name":2645,"slug":2646,"type":16},"PostgreSQL","postgresql",{"name":2608,"slug":2609,"type":16},{"name":2611,"slug":2612,"type":16},"2026-07-12T08:36:46.530743",{"slug":2651,"name":2652,"fn":2596,"description":2597,"org":2653,"tags":2654,"stars":2590,"repoUrl":2591,"updatedAt":2659},"distributed-sql","distributed sql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2655,2656,2657,2658],{"name":21,"slug":22,"type":16},{"name":2605,"slug":2606,"type":16},{"name":2608,"slug":2609,"type":16},{"name":2611,"slug":2612,"type":16},"2026-07-12T08:36:48.104182",{"slug":2661,"name":2661,"fn":2596,"description":2597,"org":2662,"tags":2663,"stars":2590,"repoUrl":2591,"updatedAt":2669},"dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2664,2665,2666,2667,2668],{"name":21,"slug":22,"type":16},{"name":2605,"slug":2606,"type":16},{"name":2632,"slug":2633,"type":16},{"name":2608,"slug":2609,"type":16},{"name":2611,"slug":2612,"type":16},"2026-07-12T08:36:36.374512",{"slug":2671,"name":2671,"fn":2672,"description":2673,"org":2674,"tags":2675,"stars":2688,"repoUrl":2689,"updatedAt":2690},"cost-efficiency-analyzer","analyze cost efficiency and expenses","Analyzes cost structure, cost efficiency, and expense management from P&L data. Use when the user asks about costs, expenses, COGS, operating expenses, cost ratios, cost control, spending efficiency, margin compression from cost side, or wants to understand where money is going. Also use for \"are we spending too much\", \"cost breakdown\", \"expense analysis\", or \"how efficient are our operations\". NOT for revenue or top-line analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2676,2679,2682,2685],{"name":2677,"slug":2678,"type":16},"Accounting","accounting",{"name":2680,"slug":2681,"type":16},"Analytics","analytics",{"name":2683,"slug":2684,"type":16},"Cost Optimization","cost-optimization",{"name":2686,"slug":2687,"type":16},"Finance","finance",3176,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagentcore-samples","2026-07-12T08:40:03.29555",{"slug":2692,"name":2692,"fn":2693,"description":2694,"org":2695,"tags":2696,"stars":2688,"repoUrl":2689,"updatedAt":2705},"executive-financial-briefing","generate executive financial briefings","Generates a concise executive-level financial briefing or summary suitable for a CEO, CFO, or board presentation. Use when the user asks for a summary, briefing, executive summary, board update, financial overview, financial health check, or \"how is the business doing\". Covers the full P&L picture in one page. Also use for \"give me the highlights\", \"what do I need to know\", or \"quick financial update\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2697,2698,2699,2702],{"name":21,"slug":22,"type":16},{"name":2686,"slug":2687,"type":16},{"name":2700,"slug":2701,"type":16},"Management","management",{"name":2703,"slug":2704,"type":16},"Reporting","reporting","2026-07-12T08:40:02.066471",{"slug":2707,"name":2707,"fn":2708,"description":2709,"org":2710,"tags":2711,"stars":2688,"repoUrl":2689,"updatedAt":2720},"multi-quarter-trend-analysis","analyze multi-quarter financial trends","Analyzes financial trends across multiple quarters by comparing P&L metrics over time. Use when the user wants to see trends, patterns, trajectories, or directional movement across 3 or more quarters. Also use for \"how are we trending\", \"show me the trend\", \"track performance over time\", \"quarter over quarter comparison across all quarters\", or any multi-period longitudinal analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2712,2713,2714,2717],{"name":2680,"slug":2681,"type":16},{"name":2686,"slug":2687,"type":16},{"name":2715,"slug":2716,"type":16},"Financial Statements","financial-statements",{"name":2718,"slug":2719,"type":16},"Variance Analysis","variance-analysis","2026-07-12T08:40:00.79141",{"slug":2722,"name":2722,"fn":2723,"description":2724,"org":2725,"tags":2726,"stars":2688,"repoUrl":2689,"updatedAt":2735},"pdf","process and manipulate PDF documents","Use this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text\u002Ftables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting\u002Fdecrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2727,2730,2733],{"name":2728,"slug":2729,"type":16},"Automation","automation",{"name":2731,"slug":2732,"type":16},"Documents","documents",{"name":2734,"slug":2722,"type":16},"PDF","2026-07-12T08:41:44.135656",{"slug":2737,"name":2737,"fn":2738,"description":2739,"org":2740,"tags":2741,"stars":2688,"repoUrl":2689,"updatedAt":2750},"quarterly-kpi-calculator","calculate quarterly financial KPIs","Calculates quarterly financial KPIs from P&L data. P&L figures can be provided directly by the user or fetched from the financial data MCP server. Use when the user wants KPI calculations such as Gross Margin %, EBITDA Margin %, Operating Expense Ratio, or Revenue Growth % QoQ. Also use for quarterly performance review, P&L analysis, or interpreting financial ratios against benchmarks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2742,2743,2746,2747],{"name":2677,"slug":2678,"type":16},{"name":2744,"slug":2745,"type":16},"Data Analysis","data-analysis",{"name":2686,"slug":2687,"type":16},{"name":2748,"slug":2749,"type":16},"KPI","kpi","2026-07-12T08:39:59.54971",150,{"items":2753,"total":2854},[2754,2771,2789,2801,2818,2829,2839],{"slug":2755,"name":2755,"fn":2756,"description":2757,"org":2758,"tags":2759,"stars":26,"repoUrl":27,"updatedAt":2770},"amazon-location-service","integrate Amazon Location Service maps","Integrates Amazon Location Service APIs for AWS applications. Use this skill when users want to add maps (interactive MapLibre or static images); geocode addresses to coordinates or reverse geocode coordinates to addresses; calculate routes, travel times, or service areas; find places and businesses through text search, nearby search, or autocomplete suggestions; retrieve detailed place information including hours, contacts, and addresses; monitor geographical boundaries with geofences; or track device locations. Covers authentication, SDK integration, and all Amazon Location Service capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2760,2763,2764,2767],{"name":2761,"slug":2762,"type":16},"API Development","api-development",{"name":21,"slug":22,"type":16},{"name":2765,"slug":2766,"type":16},"Maps","maps",{"name":2768,"slug":2769,"type":16},"Navigation","navigation","2026-07-12T08:39:49.88311",{"slug":2772,"name":2772,"fn":2773,"description":2774,"org":2775,"tags":2776,"stars":26,"repoUrl":27,"updatedAt":2788},"amplify-workflow","build and deploy apps with AWS Amplify","Build and deploy full-stack web and mobile apps with AWS Amplify Gen2 (TypeScript code-first). Covers auth (Cognito), data (AppSync\u002FDynamoDB including schema modeling, enum types, relationships, authorization rules), storage (S3), functions, APIs, and AI (Amplify AI Kit with Bedrock). Supports React, Next.js, Vue, Angular, React Native, Flutter, Swift, and Android. Always use this skill for Amplify Gen2 topics — even for questions you think you know — it contains validated, version-specific patterns that prevent common mistakes. TRIGGER when: user mentions Amplify Gen2; project has amplify\u002F directory or amplify_outputs; code imports @aws-amplify packages; user asks about defineBackend, defineAuth, defineData, defineStorage, or npx ampx. SKIP: Amplify Gen1 (amplify CLI v6), standalone SAM\u002FCDK without Amplify (use aws-serverless), direct Bedrock without Amplify AI Kit (use bedrock).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2777,2780,2781,2782,2785],{"name":2778,"slug":2779,"type":16},"Auth","auth",{"name":21,"slug":22,"type":16},{"name":2605,"slug":2606,"type":16},{"name":2783,"slug":2784,"type":16},"Deployment","deployment",{"name":2786,"slug":2787,"type":16},"TypeScript","typescript","2026-07-12T08:39:43.500162",{"slug":2790,"name":2790,"fn":2791,"description":2792,"org":2793,"tags":2794,"stars":26,"repoUrl":27,"updatedAt":2800},"api-gateway","build and manage Amazon API Gateway APIs","Build, manage, and operate APIs with Amazon API Gateway (REST, HTTP, and WebSocket). Triggers on phrases like: API Gateway, REST API, HTTP API, WebSocket API, custom domain, Lambda authorizer, usage plan, throttling, CORS, VPC link, private API. Also covers troubleshooting API Gateway errors (4xx, 5xx, timeout, CORS failures) and IaC templates containing API Gateway resources. For general REST API design unrelated to AWS, do not trigger.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2795,2796,2797],{"name":2761,"slug":2762,"type":16},{"name":21,"slug":22,"type":16},{"name":2798,"slug":2799,"type":16},"REST API","rest-api","2026-07-12T08:39:00.149339",{"slug":2802,"name":2802,"fn":2803,"description":2804,"org":2805,"tags":2806,"stars":26,"repoUrl":27,"updatedAt":2817},"aws-architecture-diagram","generate AWS architecture diagrams","Generate validated AWS architecture diagrams as draw.io XML using official AWS4 icon libraries. Use this skill whenever the user wants to create, generate, or design AWS architecture diagrams, cloud infrastructure diagrams, or system design visuals. Also triggers for requests to visualize existing infrastructure from CloudFormation, CDK, or Terraform code. Supports two modes: analyze an existing codebase to auto-generate diagrams, or brainstorm interactively from scratch. Exports .drawio files with optional PNG\u002FSVG\u002FPDF export via draw.io desktop CLI.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2807,2810,2811,2814],{"name":2808,"slug":2809,"type":16},"Architecture","architecture",{"name":21,"slug":22,"type":16},{"name":2812,"slug":2813,"type":16},"Design","design",{"name":2815,"slug":2816,"type":16},"Diagrams","diagrams","2026-07-12T08:37:11.012278",{"slug":2819,"name":2819,"fn":2820,"description":2821,"org":2822,"tags":2823,"stars":26,"repoUrl":27,"updatedAt":2828},"aws-lambda","build and deploy AWS Lambda functions","Design, build, deploy, test, and debug serverless applications with AWS Lambda. Triggers on phrases like: Lambda function, event source, serverless application, API Gateway, EventBridge, Step Functions, serverless API, event-driven architecture, Lambda trigger. For deploying non-serverless apps to AWS, use deploy-on-aws plugin instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2824,2825,2826,2827],{"name":2761,"slug":2762,"type":16},{"name":21,"slug":22,"type":16},{"name":2783,"slug":2784,"type":16},{"name":2608,"slug":2609,"type":16},"2026-07-12T08:38:58.598492",{"slug":2830,"name":2830,"fn":2831,"description":2832,"org":2833,"tags":2834,"stars":26,"repoUrl":27,"updatedAt":2838},"aws-lambda-durable-functions","build resilient AWS Lambda durable functions","Build resilient, long-running, multi-step applications with AWS Lambda durable functions with automatic state persistence, retry logic, and orchestration for long-running executions. Covers the critical replay model, step operations, wait\u002Fcallback patterns, error handling with saga pattern, testing with LocalDurableTestRunner. Triggers on phrases like: lambda durable functions, workflow orchestration, state machines, retry\u002Fcheckpoint patterns, long-running stateful Lambda functions, saga pattern, human-in-the-loop callbacks, and reliable serverless applications.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2835,2836,2837],{"name":2808,"slug":2809,"type":16},{"name":21,"slug":22,"type":16},{"name":2608,"slug":2609,"type":16},"2026-07-12T08:39:05.546173",{"slug":2840,"name":2840,"fn":2841,"description":2842,"org":2843,"tags":2844,"stars":26,"repoUrl":27,"updatedAt":2853},"aws-lambda-managed-instances","configure AWS Lambda Managed Instances","Evaluate, configure, and migrate workloads to AWS Lambda Managed Instances (LMI). Triggers on: Lambda Managed Instances, LMI, capacity provider, multi-concurrency Lambda, dedicated instance Lambda, EC2-backed Lambda, cold start elimination, Graviton Lambda, instance type for Lambda, scheduled scaling for LMI, Lambda cost optimization with Reserved Instances or Savings Plans. Also trigger when users describe high-volume predictable workloads seeking cost savings, want to scale LMI capacity on a schedule, or compare Lambda vs EC2 for steady-state traffic. For standard Lambda without LMI, use the aws-lambda skill instead.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2845,2846,2847,2850],{"name":21,"slug":22,"type":16},{"name":2783,"slug":2784,"type":16},{"name":2848,"slug":2849,"type":16},"Infrastructure","infrastructure",{"name":2851,"slug":2852,"type":16},"Performance","performance","2026-07-12T08:39:07.007071",33]