[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"skill-aws-labs-hyperpod-performance-debugger":3,"mdc--bblttq-key":42,"related-org-aws-labs-hyperpod-performance-debugger":2067,"related-repo-aws-labs-hyperpod-performance-debugger":2243},{"slug":4,"name":4,"fn":5,"description":6,"org":7,"tags":12,"stars":26,"repoUrl":27,"updatedAt":28,"license":29,"forks":30,"topics":31,"repo":37,"sourceUrl":40,"mdContent":41},"hyperpod-performance-debugger","debug performance on SageMaker HyperPod clusters","Diagnose performance issues on Amazon SageMaker HyperPod clusters — uneven NCCL bandwidth across nodes and poor filesystem throughput. Read-only. Surfaces host-side signals (Xid, ECC, NVLink, EFA reachability, FSx saturation) and routes to the appropriate sibling skill (hyperpod-node-debugger, hyperpod-nccl, hyperpod-version-checker, hyperpod-issue-report) for any remediation. Triggers on uneven NCCL across nodes, straggler node, FSx slow, checkpoint slow, dataloader slow, filesystem bottleneck, FSx throughput, cross-AZ latency, topology mismatch.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},"aws-labs","AWS Labs","https:\u002F\u002Fpexgzepcugksgbtrxkhf.supabase.co\u002Fstorage\u002Fv1\u002Fobject\u002Fpublic\u002Forg-logos\u002Faws-labs.png","awslabs",[13,17,20,23],{"name":14,"slug":15,"type":16},"Observability","observability","tag",{"name":18,"slug":19,"type":16},"Performance","performance",{"name":21,"slug":22,"type":16},"AWS","aws",{"name":24,"slug":25,"type":16},"Debugging","debugging",831,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagent-plugins","2026-07-12T08:39:34.626279",null,127,[32,33,34,22,35,36],"agent-plugins","agent-skills","agents","coding-agent-skills","coding-agents",{"repoUrl":27,"stars":26,"forks":30,"topics":38,"description":39},[32,33,34,22,35,36],"Agent Plugins for AWS equip AI coding agents with the skills to help you architect, deploy, and operate on AWS.","https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagent-plugins\u002Ftree\u002FHEAD\u002Fplugins\u002Fsagemaker-ai\u002Fskills\u002Fhyperpod-performance-debugger","---\nname: hyperpod-performance-debugger\ndescription: Diagnose performance issues on Amazon SageMaker HyperPod clusters — uneven NCCL bandwidth across nodes and poor filesystem throughput. Read-only. Surfaces host-side signals (Xid, ECC, NVLink, EFA reachability, FSx saturation) and routes to the appropriate sibling skill (hyperpod-node-debugger, hyperpod-nccl, hyperpod-version-checker, hyperpod-issue-report) for any remediation. Triggers on uneven NCCL across nodes, straggler node, FSx slow, checkpoint slow, dataloader slow, filesystem bottleneck, FSx throughput, cross-AZ latency, topology mismatch.\nmetadata:\n  version: \"0.0.1\"\n---\n\n# HyperPod Performance Debugger\n\n1. **Uneven NCCL performance across nodes** — workload faster on some node sets than others, pairwise bandwidth variance, suspected straggler.\n2. **Poor filesystem performance** — training stalled on data loading, checkpoint save\u002Fload dominating step time, FSx throughput saturated.\n\n## Scope and delegation\n\nRoute findings outside the two in-scope scenarios to the owner skill below.\n\n| Concern observed                                                       | Route to                                                     |\n| ---------------------------------------------------------------------- | ------------------------------------------------------------ |\n| GPU hardware fault, ECC, NVLink, Xid, DCGM diagnostics, drain\u002Freplace  | `hyperpod-node-debugger` (§ F Hardware\u002FAuto-Repair, § G GPU) |\n| `Cannot allocate memory` at `os.fork()`, root volume exhausted         | `hyperpod-node-debugger` (§ I Resource Exhaustion)           |\n| NCCL timeouts, hangs, AllReduce stalls, EFA TCP fallback, RDMA memlock | `hyperpod-nccl`                                              |\n| EFA \u002F NCCL \u002F CUDA \u002F NVIDIA driver version drift across nodes           | `hyperpod-version-checker`                                   |\n| EFA self-referencing security-group rule missing — single node         | `hyperpod-node-debugger` § A (EFA \u002F Security Group)          |\n| EFA self-referencing security-group rule missing — cluster-wide        | `hyperpod-cluster-debugger` § A (EFA Health Checks)          |\n| Slurm node state changes (drain \u002F resume \u002F reboot)                     | `hyperpod-slurm-debugger`                                    |\n| Diagnostic bundle for AWS Support                                      | `hyperpod-issue-report`                                      |\n| Shell access on a node                                                 | `hyperpod-ssm`                                               |\n\n## Operating policy\n\n- Read-only. Print commands the customer runs; do not execute commands that modify state.\n- Container vs host version comparisons go through `hyperpod-version-checker`.\n- Xid lines, ECC counts, NVLink lane state, and thermal readings get surfaced; the catalog and verdict live in `hyperpod-node-debugger` § G.\n\n## Workflow\n\n1. Confirm the symptom is **uneven NCCL** or **poor filesystem performance**. If neither, route to the matching sibling skill above.\n2. Run `scripts\u002Fperf-snapshot.sh` (read-only) to gather host-side signals for the suspect node and FSx filesystems mounted on it.\n3. For each `[CONCERN]` line in the script output, open the matching section below and read the supporting reference.\n4. After the per-incident diagnosis, recommend the HyperPod platform health features in [§ Continuous health coverage](#continuous-health-coverage) so the customer gets ongoing protection.\n\n## Step 1: Run the snapshot\n\n```bash\nbash scripts\u002Fperf-snapshot.sh --cluster \u003CCLUSTER_NAME_OR_ARN> --region \u003CREGION>\n\n# Scope to one suspect node:\nbash scripts\u002Fperf-snapshot.sh --cluster \u003CC> --region \u003CR> --node \u003CINSTANCE_ID>\n```\n\nThe script samples one node by default. It collects host-side data via `hyperpod-ssm`: `nvidia-smi` output (temperature, SM clocks, PCIe link width, ECC, NVLink, `topo -m`), recent `dmesg` Xid lines, EFA port state and `fi_info` provider visibility, EFA installer + kernel module versions, CPU governor, NVL72 Fabric Manager state, FSx CloudWatch utilization, `df -h` \u002F `lfs df -h` per mount, host iowait, `\u002Fdev\u002Fshm` size, and root-volume usage. All read-only.\n\nTags: `[OK]` healthy · `[CONCERN]` signal worth investigating (carries a `→` pointer to the owner skill) · `[INFO]` informational.\n\n**Host vs container scope.** The script runs on the host via SSM and reports host-scope values. Many setups ship the EFA \u002F libfabric \u002F OFI-NCCL \u002F CUDA stack inside the training container by design — a host value of `unknown` is not by itself a defect. What matters for performance is the stack the workload actually uses. Verify versions inside the container (and across nodes) via `hyperpod-version-checker` before drawing conclusions.\n\n## Step 2: Match signal → section\n\n| Observation                                                                   | Section                                                                                                                                |\n| ----------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------- |\n| Pairwise NCCL bandwidth varies across node pairs \u002F suspected straggler        | **[A: Uneven NCCL Performance](#a-uneven-nccl-performance)**                                                                           |\n| Nodes spread across AZs \u002F network-node-layer labels \u002F UltraServer boundaries  | **[A](#a-uneven-nccl-performance)**                                                                                                    |\n| EFA port not ACTIVE on a node, missing OFI plugin, or FI provider not visible | **[A](#a-uneven-nccl-performance)** + route to `hyperpod-node-debugger` § A; `hyperpod-version-checker` for cross-node version compare |\n| `iostat` shows high iowait, FSx CloudWatch utilization sustained near 100%    | **[B: Poor Filesystem Performance](#b-poor-filesystem-performance)**                                                                   |\n| DataLoader stalls, checkpoint dominates step time                             | **[B](#b-poor-filesystem-performance)**                                                                                                |\n| Xid line in `dmesg`, uncorrectable ECC, inactive NVLink lane, GPU ≥ 88°C      | Route to `hyperpod-node-debugger` § G                                                                                                  |\n| Container vs host version drift suspected                                     | Route to `hyperpod-version-checker`                                                                                                    |\n| `Cannot allocate memory` at `os.fork()`, root volume full, OOM events         | Route to `hyperpod-node-debugger` § I                                                                                                  |\n| NCCL timeout, hang, TCP fallback (`NET\u002FOFI Using TCP`), RDMA memlock          | Route to `hyperpod-nccl`                                                                                                               |\n\n---\n\n## A: Uneven NCCL Performance\n\nThe customer reports identical training jobs running with different step times on different node sets, pairwise bandwidth variance, or some allocations consistently slower than others despite identical code.\n\nPer the official troubleshooting guide, the common contributing factors are network topology differences between nodes (cross-AZ, cross-rack, cross-UltraServer), degraded EFA performance on some nodes, mixed instance types or generations within an instance group, and CPU frequency scaling differences.\n\n### Diagnostic pass (read-only)\n\nThe host-side data points — GPU thermal\u002FECC\u002FPCIe\u002Fclocks, Xid, NVLink lanes, EFA port state and provider visibility, CPU governor, EFA\u002FOFI\u002Fdriver versions, `nvidia-smi topo -m` — are all collected by `scripts\u002Fperf-snapshot.sh` (Step 1 above). The script tags `[CONCERN]` with thresholds and emits routing pointers; rerun it per suspect node via `--node \u003CINSTANCE_ID>`.\n\nFor driver \u002F CUDA \u002F NCCL \u002F EFA \u002F OFI version drift across nodes, run `hyperpod-version-checker` skill.\n\n### Pairwise NCCL bandwidth test\n\nRun the standard `nccl-tests` recipes from [awslabs\u002Fawsome-distributed-training](https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fawsome-distributed-training\u002Ftree\u002Fmain\u002Fmicro-benchmarks\u002Fnccl-tests). For an N-node cluster, run all-reduce across every pair and record `busbw` for each pair. Pairs more than ~5% below the run mean (the threshold the AWS validation script flags) are problematic candidates.\n\nExpected `busbw` per SKU is published in the [AI-on-HyperPod NCCL test guide](https:\u002F\u002Fawslabs.github.io\u002Fai-on-sagemaker-hyperpod\u002Fdocs\u002Fslurm-orchestration\u002Fvalidation-and-testing\u002Fperformance-testing\u002Fnccl-tests). Benchmark the specific instance type before relying on a number.\n\nPairwise scripts, HyperPod topology surfaces (HyperPod API, EKS labels, Slurm `topology.conf`), and GB200 NVL72 specifics are in [references\u002Fperf-details.md § Uneven NCCL](references\u002Fperf-details.md#uneven-nccl).\n\n### Topology verification\n\nHyperPod exposes topology through three operator-visible surfaces:\n\n- **HyperPod API**: `aws sagemaker describe-cluster-node` returns `NodeDetails.Placement.AvailabilityZone` \u002F `AvailabilityZoneId` and `NodeDetails.UltraServerInfo.Id` (UltraServer SKUs only).\n- **EKS labels**: `topology.kubernetes.io\u002Fzone`, `topology.k8s.aws\u002Fnetwork-node-layer-{1,2,3}` (highest-numbered = closest to instance), `topology.k8s.aws\u002Fultraserver-id`.\n- **Slurm**: HyperPod auto-generates `topology.conf`. Inspect via `scontrol show topology`.\n\nTightly coupled work shares the same AZ, the same highest-numbered network-node-layer label (EKS) or the same Slurm topology block, and — for NVL72 jobs — the same `UltraServerInfo.Id` \u002F `topology.k8s.aws\u002Fultraserver-id`. If the cluster is spread across AZs or layers, topology must be re-established at provisioning time. Route provisioning changes to `hyperpod-cluster-debugger` § B (Capacity & AZ).\n\n---\n\n## B: Poor Filesystem Performance\n\nThe customer reports training bottlenecked on data loading, checkpoint save\u002Fload dominating step time, executables\u002Fscripts loading slowly, or `iowait` high.\n\nPer the official troubleshooting guide, the resolution path follows this order:\n\n1. Check CloudWatch metrics on the filesystem.\n2. Check the provisioned performance configuration against workload requirements.\n3. Investigate which operations are causing the I\u002FO — workload demand vs inefficient pattern.\n4. Consider upgrading provisioned performance.\n5. Choose the filesystem type that matches the I\u002FO pattern.\n\nThis skill covers steps 1–3. Steps 4–5 are customer decisions; surface the data and let the customer pick.\n\n### Diagnostic pass (read-only)\n\n`scripts\u002Fperf-snapshot.sh` (Step 1 above) covers the on-node side of this pass: it discovers FSx mounts, calls `aws cloudwatch get-metric-statistics` on `DataReadBytes` and (for OpenZFS) `FileServerDiskIopsUtilization`, prints `df -h` for `\u002Ffsx \u002Fopt\u002Fdlami\u002Fnvme \u002Fopt\u002Fsagemaker`, runs `lfs df -h` per Lustre mount, and reports `iostat` iowait. It tags `[CONCERN]` when OpenZFS IOPS utilization sustains ≥ 80% or iowait > 20%.\n\nFor longer windows or additional metrics (`DataWriteBytes`, Lustre `DiskIopsUtilization`, OpenZFS `FileServerDiskThroughputUtilization`), drive the query directly:\n\n```bash\naws cloudwatch get-metric-statistics --region \u003CREGION> \\\n  --namespace AWS\u002FFSx --metric-name DataReadBytes \\\n  --dimensions Name=FileSystemId,Value=\u003CFSID> \\\n  --start-time \"$(date -u -d '3 hours ago' +%Y-%m-%dT%H:%M:%S)\" \\\n  --end-time   \"$(date -u +%Y-%m-%dT%H:%M:%S)\" \\\n  --period 60 --statistics Sum Maximum\n```\n\nThe full per-filesystem-type metric catalog is in [references\u002Fperf-details.md § Filesystem](references\u002Fperf-details.md#filesystem).\n\n### Branches\n\n**Provisioned capacity is saturated.** CloudWatch utilization sustained near 100% across the workload window. Customer decision: scale up the filesystem.\n\n- FSx for Lustre throughput scales with `StorageCapacity × PerUnitStorageThroughput`; capacity changes are non-disruptive.\n- FSx for OpenZFS — increase provisioned IOPS or throughput.\n\n**I\u002FO pattern is inefficient.** CloudWatch shows headroom but the workload is still I\u002FO-bound. Customer decision: change the application.\n\n- DataLoader: raise `num_workers`, set `pin_memory=True`, `persistent_workers=True`.\n- Checkpointing: use async + sharded (`torch.distributed.checkpoint.async_save` plus FSDP `SHARDED_STATE_DICT`). `FULL_STATE_DICT` serializes through rank 0 and is a frequent root cause.\n- Small-file workloads: Lustre is optimized for large sequential I\u002FO. For millions of small files, use WebDataset \u002F tar shards, FSx for OpenZFS, or NVMe scratch.\n\nFilesystem-selection guidance and the async-checkpoint pattern are in [references\u002Fperf-details.md § Filesystem](references\u002Fperf-details.md#filesystem).\n\n---\n\n## Continuous health coverage\n\nOnce the immediate incident is diagnosed, recommend HyperPod's built-in health features so problems are caught before the next training run rather than after another customer-reported regression.\n\n- **Enable `NodeRecovery=Automatic`** on the cluster. The Health Monitoring Agent (HMA) continuously monitors GPU- and Trainium-based instances and marks instances unhealthy on detected failure. With auto-recovery enabled, HyperPod reboots or replaces the node — no operator intervention.\n- **Enable `OnStartDeepHealthChecks` on every GPU instance group** with both check categories:\n  - `InstanceStress` — `stress-ng` on CPU\u002Fmemory\u002Fdisk, GPU and PCI device count verification, DCGM level-4 diagnostics (memory test included), and EFA loopback bandwidth\u002Flatency.\n  - `InstanceConnectivity` — multi-node NCCL all-reduce.\n\n  Every newly provisioned or auto-replaced node passes the same hardware bar before accepting jobs.\n\n- **Run on-demand deep health checks** when this skill or any sibling surfaces a hardware concern but the cluster is mid-workload. `aws sagemaker start-cluster-health-check` runs the same checks against a specific instance group; nodes are placed in a Slurm maintenance reservation and the check is queued until any running job completes (not preempted). Console: **HyperPod → Clusters → Instances → Run deep health checks**.\n\n  Not supported when `NodeProvisioningMode=Continuous`; one on-demand request per cluster at a time. Requires the latest AMI — run `UpdateClusterSoftware` first.\n\nLogs land in CloudWatch at `\u002Faws\u002Fsagemaker\u002FClusters\u002F\u003Ccluster_name>\u002F\u003Ccluster_id>` under `DeepHealthCheckResults\u002F\u003Clog_stream_id>`, and on each node at `\u002Fvar\u002Flog\u002Faws\u002Fclusters\u002Fsagemaker-deep-health-check.log`.\n\n## References\n\n- [references\u002Fperf-details.md](references\u002Fperf-details.md) — pairwise NCCL test recipes, HyperPod topology check, GB200 NVL72 placement; CloudWatch metric catalog per filesystem type, async-checkpoint pattern, filesystem selection guide.\n\nExternal:\n\n- Amazon SageMaker HyperPod troubleshooting guide (official): \u003Chttps:\u002F\u002Fgithub.com\u002Faws\u002Fsagemaker-hyperpod-cluster-setup\u002Fblob\u002Ftroubleshooting-doc-20250917\u002Ftroubleshoot\u002Findex.md>\n- AI-on-HyperPod NCCL performance tests (expected `busbw` per SKU): \u003Chttps:\u002F\u002Fawslabs.github.io\u002Fai-on-sagemaker-hyperpod\u002Fdocs\u002Fslurm-orchestration\u002Fvalidation-and-testing\u002Fperformance-testing\u002Fnccl-tests>\n- Amazon SageMaker HyperPod resiliency (NodeRecovery, HMA, auto-resume): \u003Chttps:\u002F\u002Fdocs.aws.amazon.com\u002Fsagemaker\u002Flatest\u002Fdg\u002Fsagemaker-hyperpod-resiliency.html>\n- Amazon SageMaker HyperPod deep health checks: \u003Chttps:\u002F\u002Fdocs.aws.amazon.com\u002Fsagemaker\u002Flatest\u002Fdg\u002Fsagemaker-hyperpod-resiliency-slurm-deep-health-checks.html>\n- StartClusterHealthCheck API: \u003Chttps:\u002F\u002Fdocs.aws.amazon.com\u002Fsagemaker\u002Flatest\u002FAPIReference\u002FAPI_StartClusterHealthCheck.html>\n- Amazon EC2 instance topology \u002F network-node-layer labels: \u003Chttps:\u002F\u002Fdocs.aws.amazon.com\u002FAWSEC2\u002Flatest\u002FUserGuide\u002Fhow-ec2-instance-topology-works.html>\n- Amazon FSx for Lustre performance: \u003Chttps:\u002F\u002Fdocs.aws.amazon.com\u002Ffsx\u002Flatest\u002FLustreGuide\u002Fperformance.html>\n- Amazon FSx for OpenZFS metrics: \u003Chttps:\u002F\u002Fdocs.aws.amazon.com\u002Ffsx\u002Flatest\u002FOpenZFSGuide\u002Ffsx-openzfs-metrics.html>\n- AWS Elastic Fabric Adapter and NCCL: \u003Chttps:\u002F\u002Fdocs.aws.amazon.com\u002FAWSEC2\u002Flatest\u002FUserGuide\u002Fefa-start-nccl.html>\n- awslabs\u002Fawsome-distributed-training NCCL tests: \u003Chttps:\u002F\u002Fgithub.com\u002Fawslabs\u002Fawsome-distributed-training\u002Ftree\u002Fmain\u002Fmicro-benchmarks\u002Fnccl-tests>\n",{"data":43,"body":46},{"name":4,"description":6,"metadata":44},{"version":45},"0.0.1",{"type":47,"children":48},"root",[49,57,83,90,96,298,304,337,343,405,411,578,646,682,707,713,952,956,961,966,971,978,1012,1024,1030,1061,1082,1102,1108,1113,1212,1238,1241,1246,1259,1264,1292,1297,1302,1372,1401,1624,1636,1642,1652,1673,1683,1747,1757,1760,1766,1771,1891,1919,1925,1938,1943,2061],{"type":50,"tag":51,"props":52,"children":53},"element","h1",{"id":4},[54],{"type":55,"value":56},"text","HyperPod Performance Debugger",{"type":50,"tag":58,"props":59,"children":60},"ol",{},[61,73],{"type":50,"tag":62,"props":63,"children":64},"li",{},[65,71],{"type":50,"tag":66,"props":67,"children":68},"strong",{},[69],{"type":55,"value":70},"Uneven NCCL performance across nodes",{"type":55,"value":72}," — workload faster on some node sets than others, pairwise bandwidth variance, suspected straggler.",{"type":50,"tag":62,"props":74,"children":75},{},[76,81],{"type":50,"tag":66,"props":77,"children":78},{},[79],{"type":55,"value":80},"Poor filesystem performance",{"type":55,"value":82}," — training stalled on data loading, checkpoint save\u002Fload dominating step time, FSx throughput saturated.",{"type":50,"tag":84,"props":85,"children":87},"h2",{"id":86},"scope-and-delegation",[88],{"type":55,"value":89},"Scope and delegation",{"type":50,"tag":91,"props":92,"children":93},"p",{},[94],{"type":55,"value":95},"Route findings outside the two in-scope scenarios to the owner skill below.",{"type":50,"tag":97,"props":98,"children":99},"table",{},[100,119],{"type":50,"tag":101,"props":102,"children":103},"thead",{},[104],{"type":50,"tag":105,"props":106,"children":107},"tr",{},[108,114],{"type":50,"tag":109,"props":110,"children":111},"th",{},[112],{"type":55,"value":113},"Concern observed",{"type":50,"tag":109,"props":115,"children":116},{},[117],{"type":55,"value":118},"Route to",{"type":50,"tag":120,"props":121,"children":122},"tbody",{},[123,144,176,193,210,228,247,264,281],{"type":50,"tag":105,"props":124,"children":125},{},[126,132],{"type":50,"tag":127,"props":128,"children":129},"td",{},[130],{"type":55,"value":131},"GPU hardware fault, ECC, NVLink, Xid, DCGM diagnostics, drain\u002Freplace",{"type":50,"tag":127,"props":133,"children":134},{},[135,142],{"type":50,"tag":136,"props":137,"children":139},"code",{"className":138},[],[140],{"type":55,"value":141},"hyperpod-node-debugger",{"type":55,"value":143}," (§ F Hardware\u002FAuto-Repair, § G GPU)",{"type":50,"tag":105,"props":145,"children":146},{},[147,166],{"type":50,"tag":127,"props":148,"children":149},{},[150,156,158,164],{"type":50,"tag":136,"props":151,"children":153},{"className":152},[],[154],{"type":55,"value":155},"Cannot allocate memory",{"type":55,"value":157}," at ",{"type":50,"tag":136,"props":159,"children":161},{"className":160},[],[162],{"type":55,"value":163},"os.fork()",{"type":55,"value":165},", root volume exhausted",{"type":50,"tag":127,"props":167,"children":168},{},[169,174],{"type":50,"tag":136,"props":170,"children":172},{"className":171},[],[173],{"type":55,"value":141},{"type":55,"value":175}," (§ I Resource Exhaustion)",{"type":50,"tag":105,"props":177,"children":178},{},[179,184],{"type":50,"tag":127,"props":180,"children":181},{},[182],{"type":55,"value":183},"NCCL timeouts, hangs, AllReduce stalls, EFA TCP fallback, RDMA memlock",{"type":50,"tag":127,"props":185,"children":186},{},[187],{"type":50,"tag":136,"props":188,"children":190},{"className":189},[],[191],{"type":55,"value":192},"hyperpod-nccl",{"type":50,"tag":105,"props":194,"children":195},{},[196,201],{"type":50,"tag":127,"props":197,"children":198},{},[199],{"type":55,"value":200},"EFA \u002F NCCL \u002F CUDA \u002F NVIDIA driver version drift across nodes",{"type":50,"tag":127,"props":202,"children":203},{},[204],{"type":50,"tag":136,"props":205,"children":207},{"className":206},[],[208],{"type":55,"value":209},"hyperpod-version-checker",{"type":50,"tag":105,"props":211,"children":212},{},[213,218],{"type":50,"tag":127,"props":214,"children":215},{},[216],{"type":55,"value":217},"EFA self-referencing security-group rule missing — single node",{"type":50,"tag":127,"props":219,"children":220},{},[221,226],{"type":50,"tag":136,"props":222,"children":224},{"className":223},[],[225],{"type":55,"value":141},{"type":55,"value":227}," § A (EFA \u002F Security Group)",{"type":50,"tag":105,"props":229,"children":230},{},[231,236],{"type":50,"tag":127,"props":232,"children":233},{},[234],{"type":55,"value":235},"EFA self-referencing security-group rule missing — cluster-wide",{"type":50,"tag":127,"props":237,"children":238},{},[239,245],{"type":50,"tag":136,"props":240,"children":242},{"className":241},[],[243],{"type":55,"value":244},"hyperpod-cluster-debugger",{"type":55,"value":246}," § A (EFA Health Checks)",{"type":50,"tag":105,"props":248,"children":249},{},[250,255],{"type":50,"tag":127,"props":251,"children":252},{},[253],{"type":55,"value":254},"Slurm node state changes (drain \u002F resume \u002F reboot)",{"type":50,"tag":127,"props":256,"children":257},{},[258],{"type":50,"tag":136,"props":259,"children":261},{"className":260},[],[262],{"type":55,"value":263},"hyperpod-slurm-debugger",{"type":50,"tag":105,"props":265,"children":266},{},[267,272],{"type":50,"tag":127,"props":268,"children":269},{},[270],{"type":55,"value":271},"Diagnostic bundle for AWS Support",{"type":50,"tag":127,"props":273,"children":274},{},[275],{"type":50,"tag":136,"props":276,"children":278},{"className":277},[],[279],{"type":55,"value":280},"hyperpod-issue-report",{"type":50,"tag":105,"props":282,"children":283},{},[284,289],{"type":50,"tag":127,"props":285,"children":286},{},[287],{"type":55,"value":288},"Shell access on a node",{"type":50,"tag":127,"props":290,"children":291},{},[292],{"type":50,"tag":136,"props":293,"children":295},{"className":294},[],[296],{"type":55,"value":297},"hyperpod-ssm",{"type":50,"tag":84,"props":299,"children":301},{"id":300},"operating-policy",[302],{"type":55,"value":303},"Operating policy",{"type":50,"tag":305,"props":306,"children":307},"ul",{},[308,313,325],{"type":50,"tag":62,"props":309,"children":310},{},[311],{"type":55,"value":312},"Read-only. Print commands the customer runs; do not execute commands that modify state.",{"type":50,"tag":62,"props":314,"children":315},{},[316,318,323],{"type":55,"value":317},"Container vs host version comparisons go through ",{"type":50,"tag":136,"props":319,"children":321},{"className":320},[],[322],{"type":55,"value":209},{"type":55,"value":324},".",{"type":50,"tag":62,"props":326,"children":327},{},[328,330,335],{"type":55,"value":329},"Xid lines, ECC counts, NVLink lane state, and thermal readings get surfaced; the catalog and verdict live in ",{"type":50,"tag":136,"props":331,"children":333},{"className":332},[],[334],{"type":55,"value":141},{"type":55,"value":336}," § G.",{"type":50,"tag":84,"props":338,"children":340},{"id":339},"workflow",[341],{"type":55,"value":342},"Workflow",{"type":50,"tag":58,"props":344,"children":345},{},[346,365,378,391],{"type":50,"tag":62,"props":347,"children":348},{},[349,351,356,358,363],{"type":55,"value":350},"Confirm the symptom is ",{"type":50,"tag":66,"props":352,"children":353},{},[354],{"type":55,"value":355},"uneven NCCL",{"type":55,"value":357}," or ",{"type":50,"tag":66,"props":359,"children":360},{},[361],{"type":55,"value":362},"poor filesystem performance",{"type":55,"value":364},". If neither, route to the matching sibling skill above.",{"type":50,"tag":62,"props":366,"children":367},{},[368,370,376],{"type":55,"value":369},"Run ",{"type":50,"tag":136,"props":371,"children":373},{"className":372},[],[374],{"type":55,"value":375},"scripts\u002Fperf-snapshot.sh",{"type":55,"value":377}," (read-only) to gather host-side signals for the suspect node and FSx filesystems mounted on it.",{"type":50,"tag":62,"props":379,"children":380},{},[381,383,389],{"type":55,"value":382},"For each ",{"type":50,"tag":136,"props":384,"children":386},{"className":385},[],[387],{"type":55,"value":388},"[CONCERN]",{"type":55,"value":390}," line in the script output, open the matching section below and read the supporting reference.",{"type":50,"tag":62,"props":392,"children":393},{},[394,396,403],{"type":55,"value":395},"After the per-incident diagnosis, recommend the HyperPod platform health features in ",{"type":50,"tag":397,"props":398,"children":400},"a",{"href":399},"#continuous-health-coverage",[401],{"type":55,"value":402},"§ Continuous health coverage",{"type":55,"value":404}," so the customer gets ongoing protection.",{"type":50,"tag":84,"props":406,"children":408},{"id":407},"step-1-run-the-snapshot",[409],{"type":55,"value":410},"Step 1: Run the snapshot",{"type":50,"tag":412,"props":413,"children":418},"pre",{"className":414,"code":415,"language":416,"meta":417,"style":417},"language-bash shiki shiki-themes material-theme-lighter material-theme material-theme-palenight","bash scripts\u002Fperf-snapshot.sh --cluster \u003CCLUSTER_NAME_OR_ARN> --region \u003CREGION>\n\n# Scope to one suspect node:\nbash scripts\u002Fperf-snapshot.sh --cluster \u003CC> --region \u003CR> --node \u003CINSTANCE_ID>\n","bash","",[419],{"type":50,"tag":136,"props":420,"children":421},{"__ignoreMap":417},[422,489,499,509],{"type":50,"tag":423,"props":424,"children":427},"span",{"class":425,"line":426},"line",1,[428,433,439,444,450,455,461,466,471,475,480,484],{"type":50,"tag":423,"props":429,"children":431},{"style":430},"--shiki-light:#E2931D;--shiki-default:#FFCB6B;--shiki-dark:#FFCB6B",[432],{"type":55,"value":416},{"type":50,"tag":423,"props":434,"children":436},{"style":435},"--shiki-light:#91B859;--shiki-default:#C3E88D;--shiki-dark:#C3E88D",[437],{"type":55,"value":438}," scripts\u002Fperf-snapshot.sh",{"type":50,"tag":423,"props":440,"children":441},{"style":435},[442],{"type":55,"value":443}," --cluster",{"type":50,"tag":423,"props":445,"children":447},{"style":446},"--shiki-light:#39ADB5;--shiki-default:#89DDFF;--shiki-dark:#89DDFF",[448],{"type":55,"value":449}," \u003C",{"type":50,"tag":423,"props":451,"children":452},{"style":435},[453],{"type":55,"value":454},"CLUSTER_NAME_OR_AR",{"type":50,"tag":423,"props":456,"children":458},{"style":457},"--shiki-light:#90A4AE;--shiki-default:#EEFFFF;--shiki-dark:#BABED8",[459],{"type":55,"value":460},"N",{"type":50,"tag":423,"props":462,"children":463},{"style":446},[464],{"type":55,"value":465},">",{"type":50,"tag":423,"props":467,"children":468},{"style":435},[469],{"type":55,"value":470}," --region",{"type":50,"tag":423,"props":472,"children":473},{"style":446},[474],{"type":55,"value":449},{"type":50,"tag":423,"props":476,"children":477},{"style":435},[478],{"type":55,"value":479},"REGIO",{"type":50,"tag":423,"props":481,"children":482},{"style":457},[483],{"type":55,"value":460},{"type":50,"tag":423,"props":485,"children":486},{"style":446},[487],{"type":55,"value":488},">\n",{"type":50,"tag":423,"props":490,"children":492},{"class":425,"line":491},2,[493],{"type":50,"tag":423,"props":494,"children":496},{"emptyLinePlaceholder":495},true,[497],{"type":55,"value":498},"\n",{"type":50,"tag":423,"props":500,"children":502},{"class":425,"line":501},3,[503],{"type":50,"tag":423,"props":504,"children":506},{"style":505},"--shiki-light:#90A4AE;--shiki-light-font-style:italic;--shiki-default:#546E7A;--shiki-default-font-style:italic;--shiki-dark:#676E95;--shiki-dark-font-style:italic",[507],{"type":55,"value":508},"# Scope to one suspect node:\n",{"type":50,"tag":423,"props":510,"children":512},{"class":425,"line":511},4,[513,517,521,525,529,534,538,542,546,551,555,560,564,569,574],{"type":50,"tag":423,"props":514,"children":515},{"style":430},[516],{"type":55,"value":416},{"type":50,"tag":423,"props":518,"children":519},{"style":435},[520],{"type":55,"value":438},{"type":50,"tag":423,"props":522,"children":523},{"style":435},[524],{"type":55,"value":443},{"type":50,"tag":423,"props":526,"children":527},{"style":446},[528],{"type":55,"value":449},{"type":50,"tag":423,"props":530,"children":531},{"style":457},[532],{"type":55,"value":533},"C",{"type":50,"tag":423,"props":535,"children":536},{"style":446},[537],{"type":55,"value":465},{"type":50,"tag":423,"props":539,"children":540},{"style":435},[541],{"type":55,"value":470},{"type":50,"tag":423,"props":543,"children":544},{"style":446},[545],{"type":55,"value":449},{"type":50,"tag":423,"props":547,"children":548},{"style":457},[549],{"type":55,"value":550},"R",{"type":50,"tag":423,"props":552,"children":553},{"style":446},[554],{"type":55,"value":465},{"type":50,"tag":423,"props":556,"children":557},{"style":435},[558],{"type":55,"value":559}," --node",{"type":50,"tag":423,"props":561,"children":562},{"style":446},[563],{"type":55,"value":449},{"type":50,"tag":423,"props":565,"children":566},{"style":435},[567],{"type":55,"value":568},"INSTANCE_I",{"type":50,"tag":423,"props":570,"children":571},{"style":457},[572],{"type":55,"value":573},"D",{"type":50,"tag":423,"props":575,"children":576},{"style":446},[577],{"type":55,"value":488},{"type":50,"tag":91,"props":579,"children":580},{},[581,583,588,590,596,598,604,606,612,614,620,622,628,630,636,638,644],{"type":55,"value":582},"The script samples one node by default. It collects host-side data via ",{"type":50,"tag":136,"props":584,"children":586},{"className":585},[],[587],{"type":55,"value":297},{"type":55,"value":589},": ",{"type":50,"tag":136,"props":591,"children":593},{"className":592},[],[594],{"type":55,"value":595},"nvidia-smi",{"type":55,"value":597}," output (temperature, SM clocks, PCIe link width, ECC, NVLink, ",{"type":50,"tag":136,"props":599,"children":601},{"className":600},[],[602],{"type":55,"value":603},"topo -m",{"type":55,"value":605},"), recent ",{"type":50,"tag":136,"props":607,"children":609},{"className":608},[],[610],{"type":55,"value":611},"dmesg",{"type":55,"value":613}," Xid lines, EFA port state and ",{"type":50,"tag":136,"props":615,"children":617},{"className":616},[],[618],{"type":55,"value":619},"fi_info",{"type":55,"value":621}," provider visibility, EFA installer + kernel module versions, CPU governor, NVL72 Fabric Manager state, FSx CloudWatch utilization, ",{"type":50,"tag":136,"props":623,"children":625},{"className":624},[],[626],{"type":55,"value":627},"df -h",{"type":55,"value":629}," \u002F ",{"type":50,"tag":136,"props":631,"children":633},{"className":632},[],[634],{"type":55,"value":635},"lfs df -h",{"type":55,"value":637}," per mount, host iowait, ",{"type":50,"tag":136,"props":639,"children":641},{"className":640},[],[642],{"type":55,"value":643},"\u002Fdev\u002Fshm",{"type":55,"value":645}," size, and root-volume usage. All read-only.",{"type":50,"tag":91,"props":647,"children":648},{},[649,651,657,659,664,666,672,674,680],{"type":55,"value":650},"Tags: ",{"type":50,"tag":136,"props":652,"children":654},{"className":653},[],[655],{"type":55,"value":656},"[OK]",{"type":55,"value":658}," healthy · ",{"type":50,"tag":136,"props":660,"children":662},{"className":661},[],[663],{"type":55,"value":388},{"type":55,"value":665}," signal worth investigating (carries a ",{"type":50,"tag":136,"props":667,"children":669},{"className":668},[],[670],{"type":55,"value":671},"→",{"type":55,"value":673}," pointer to the owner skill) · ",{"type":50,"tag":136,"props":675,"children":677},{"className":676},[],[678],{"type":55,"value":679},"[INFO]",{"type":55,"value":681}," informational.",{"type":50,"tag":91,"props":683,"children":684},{},[685,690,692,698,700,705],{"type":50,"tag":66,"props":686,"children":687},{},[688],{"type":55,"value":689},"Host vs container scope.",{"type":55,"value":691}," The script runs on the host via SSM and reports host-scope values. Many setups ship the EFA \u002F libfabric \u002F OFI-NCCL \u002F CUDA stack inside the training container by design — a host value of ",{"type":50,"tag":136,"props":693,"children":695},{"className":694},[],[696],{"type":55,"value":697},"unknown",{"type":55,"value":699}," is not by itself a defect. What matters for performance is the stack the workload actually uses. Verify versions inside the container (and across nodes) via ",{"type":50,"tag":136,"props":701,"children":703},{"className":702},[],[704],{"type":55,"value":209},{"type":55,"value":706}," before drawing conclusions.",{"type":50,"tag":84,"props":708,"children":710},{"id":709},"step-2-match-signal-section",[711],{"type":55,"value":712},"Step 2: Match signal → section",{"type":50,"tag":97,"props":714,"children":715},{},[716,732],{"type":50,"tag":101,"props":717,"children":718},{},[719],{"type":50,"tag":105,"props":720,"children":721},{},[722,727],{"type":50,"tag":109,"props":723,"children":724},{},[725],{"type":55,"value":726},"Observation",{"type":50,"tag":109,"props":728,"children":729},{},[730],{"type":55,"value":731},"Section",{"type":50,"tag":120,"props":733,"children":734},{},[735,755,774,808,834,853,880,897,927],{"type":50,"tag":105,"props":736,"children":737},{},[738,743],{"type":50,"tag":127,"props":739,"children":740},{},[741],{"type":55,"value":742},"Pairwise NCCL bandwidth varies across node pairs \u002F suspected straggler",{"type":50,"tag":127,"props":744,"children":745},{},[746],{"type":50,"tag":66,"props":747,"children":748},{},[749],{"type":50,"tag":397,"props":750,"children":752},{"href":751},"#a-uneven-nccl-performance",[753],{"type":55,"value":754},"A: Uneven NCCL Performance",{"type":50,"tag":105,"props":756,"children":757},{},[758,763],{"type":50,"tag":127,"props":759,"children":760},{},[761],{"type":55,"value":762},"Nodes spread across AZs \u002F network-node-layer labels \u002F UltraServer boundaries",{"type":50,"tag":127,"props":764,"children":765},{},[766],{"type":50,"tag":66,"props":767,"children":768},{},[769],{"type":50,"tag":397,"props":770,"children":771},{"href":751},[772],{"type":55,"value":773},"A",{"type":50,"tag":105,"props":775,"children":776},{},[777,782],{"type":50,"tag":127,"props":778,"children":779},{},[780],{"type":55,"value":781},"EFA port not ACTIVE on a node, missing OFI plugin, or FI provider not visible",{"type":50,"tag":127,"props":783,"children":784},{},[785,792,794,799,801,806],{"type":50,"tag":66,"props":786,"children":787},{},[788],{"type":50,"tag":397,"props":789,"children":790},{"href":751},[791],{"type":55,"value":773},{"type":55,"value":793}," + route to ",{"type":50,"tag":136,"props":795,"children":797},{"className":796},[],[798],{"type":55,"value":141},{"type":55,"value":800}," § A; ",{"type":50,"tag":136,"props":802,"children":804},{"className":803},[],[805],{"type":55,"value":209},{"type":55,"value":807}," for cross-node version compare",{"type":50,"tag":105,"props":809,"children":810},{},[811,822],{"type":50,"tag":127,"props":812,"children":813},{},[814,820],{"type":50,"tag":136,"props":815,"children":817},{"className":816},[],[818],{"type":55,"value":819},"iostat",{"type":55,"value":821}," shows high iowait, FSx CloudWatch utilization sustained near 100%",{"type":50,"tag":127,"props":823,"children":824},{},[825],{"type":50,"tag":66,"props":826,"children":827},{},[828],{"type":50,"tag":397,"props":829,"children":831},{"href":830},"#b-poor-filesystem-performance",[832],{"type":55,"value":833},"B: Poor Filesystem Performance",{"type":50,"tag":105,"props":835,"children":836},{},[837,842],{"type":50,"tag":127,"props":838,"children":839},{},[840],{"type":55,"value":841},"DataLoader stalls, checkpoint dominates step time",{"type":50,"tag":127,"props":843,"children":844},{},[845],{"type":50,"tag":66,"props":846,"children":847},{},[848],{"type":50,"tag":397,"props":849,"children":850},{"href":830},[851],{"type":55,"value":852},"B",{"type":50,"tag":105,"props":854,"children":855},{},[856,868],{"type":50,"tag":127,"props":857,"children":858},{},[859,861,866],{"type":55,"value":860},"Xid line in ",{"type":50,"tag":136,"props":862,"children":864},{"className":863},[],[865],{"type":55,"value":611},{"type":55,"value":867},", uncorrectable ECC, inactive NVLink lane, GPU ≥ 88°C",{"type":50,"tag":127,"props":869,"children":870},{},[871,873,878],{"type":55,"value":872},"Route to ",{"type":50,"tag":136,"props":874,"children":876},{"className":875},[],[877],{"type":55,"value":141},{"type":55,"value":879}," § G",{"type":50,"tag":105,"props":881,"children":882},{},[883,888],{"type":50,"tag":127,"props":884,"children":885},{},[886],{"type":55,"value":887},"Container vs host version drift suspected",{"type":50,"tag":127,"props":889,"children":890},{},[891,892],{"type":55,"value":872},{"type":50,"tag":136,"props":893,"children":895},{"className":894},[],[896],{"type":55,"value":209},{"type":50,"tag":105,"props":898,"children":899},{},[900,916],{"type":50,"tag":127,"props":901,"children":902},{},[903,908,909,914],{"type":50,"tag":136,"props":904,"children":906},{"className":905},[],[907],{"type":55,"value":155},{"type":55,"value":157},{"type":50,"tag":136,"props":910,"children":912},{"className":911},[],[913],{"type":55,"value":163},{"type":55,"value":915},", root volume full, OOM events",{"type":50,"tag":127,"props":917,"children":918},{},[919,920,925],{"type":55,"value":872},{"type":50,"tag":136,"props":921,"children":923},{"className":922},[],[924],{"type":55,"value":141},{"type":55,"value":926}," § I",{"type":50,"tag":105,"props":928,"children":929},{},[930,943],{"type":50,"tag":127,"props":931,"children":932},{},[933,935,941],{"type":55,"value":934},"NCCL timeout, hang, TCP fallback (",{"type":50,"tag":136,"props":936,"children":938},{"className":937},[],[939],{"type":55,"value":940},"NET\u002FOFI Using TCP",{"type":55,"value":942},"), RDMA memlock",{"type":50,"tag":127,"props":944,"children":945},{},[946,947],{"type":55,"value":872},{"type":50,"tag":136,"props":948,"children":950},{"className":949},[],[951],{"type":55,"value":192},{"type":50,"tag":953,"props":954,"children":955},"hr",{},[],{"type":50,"tag":84,"props":957,"children":959},{"id":958},"a-uneven-nccl-performance",[960],{"type":55,"value":754},{"type":50,"tag":91,"props":962,"children":963},{},[964],{"type":55,"value":965},"The customer reports identical training jobs running with different step times on different node sets, pairwise bandwidth variance, or some allocations consistently slower than others despite identical code.",{"type":50,"tag":91,"props":967,"children":968},{},[969],{"type":55,"value":970},"Per the official troubleshooting guide, the common contributing factors are network topology differences between nodes (cross-AZ, cross-rack, cross-UltraServer), degraded EFA performance on some nodes, mixed instance types or generations within an instance group, and CPU frequency scaling differences.",{"type":50,"tag":972,"props":973,"children":975},"h3",{"id":974},"diagnostic-pass-read-only",[976],{"type":55,"value":977},"Diagnostic pass (read-only)",{"type":50,"tag":91,"props":979,"children":980},{},[981,983,989,991,996,998,1003,1005,1011],{"type":55,"value":982},"The host-side data points — GPU thermal\u002FECC\u002FPCIe\u002Fclocks, Xid, NVLink lanes, EFA port state and provider visibility, CPU governor, EFA\u002FOFI\u002Fdriver versions, ",{"type":50,"tag":136,"props":984,"children":986},{"className":985},[],[987],{"type":55,"value":988},"nvidia-smi topo -m",{"type":55,"value":990}," — are all collected by ",{"type":50,"tag":136,"props":992,"children":994},{"className":993},[],[995],{"type":55,"value":375},{"type":55,"value":997}," (Step 1 above). The script tags ",{"type":50,"tag":136,"props":999,"children":1001},{"className":1000},[],[1002],{"type":55,"value":388},{"type":55,"value":1004}," with thresholds and emits routing pointers; rerun it per suspect node via ",{"type":50,"tag":136,"props":1006,"children":1008},{"className":1007},[],[1009],{"type":55,"value":1010},"--node \u003CINSTANCE_ID>",{"type":55,"value":324},{"type":50,"tag":91,"props":1013,"children":1014},{},[1015,1017,1022],{"type":55,"value":1016},"For driver \u002F CUDA \u002F NCCL \u002F EFA \u002F OFI version drift across nodes, run ",{"type":50,"tag":136,"props":1018,"children":1020},{"className":1019},[],[1021],{"type":55,"value":209},{"type":55,"value":1023}," skill.",{"type":50,"tag":972,"props":1025,"children":1027},{"id":1026},"pairwise-nccl-bandwidth-test",[1028],{"type":55,"value":1029},"Pairwise NCCL bandwidth test",{"type":50,"tag":91,"props":1031,"children":1032},{},[1033,1035,1041,1043,1051,1053,1059],{"type":55,"value":1034},"Run the standard ",{"type":50,"tag":136,"props":1036,"children":1038},{"className":1037},[],[1039],{"type":55,"value":1040},"nccl-tests",{"type":55,"value":1042}," recipes from ",{"type":50,"tag":397,"props":1044,"children":1048},{"href":1045,"rel":1046},"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fawsome-distributed-training\u002Ftree\u002Fmain\u002Fmicro-benchmarks\u002Fnccl-tests",[1047],"nofollow",[1049],{"type":55,"value":1050},"awslabs\u002Fawsome-distributed-training",{"type":55,"value":1052},". For an N-node cluster, run all-reduce across every pair and record ",{"type":50,"tag":136,"props":1054,"children":1056},{"className":1055},[],[1057],{"type":55,"value":1058},"busbw",{"type":55,"value":1060}," for each pair. Pairs more than ~5% below the run mean (the threshold the AWS validation script flags) are problematic candidates.",{"type":50,"tag":91,"props":1062,"children":1063},{},[1064,1066,1071,1073,1080],{"type":55,"value":1065},"Expected ",{"type":50,"tag":136,"props":1067,"children":1069},{"className":1068},[],[1070],{"type":55,"value":1058},{"type":55,"value":1072}," per SKU is published in the ",{"type":50,"tag":397,"props":1074,"children":1077},{"href":1075,"rel":1076},"https:\u002F\u002Fawslabs.github.io\u002Fai-on-sagemaker-hyperpod\u002Fdocs\u002Fslurm-orchestration\u002Fvalidation-and-testing\u002Fperformance-testing\u002Fnccl-tests",[1047],[1078],{"type":55,"value":1079},"AI-on-HyperPod NCCL test guide",{"type":55,"value":1081},". Benchmark the specific instance type before relying on a number.",{"type":50,"tag":91,"props":1083,"children":1084},{},[1085,1087,1093,1095,1101],{"type":55,"value":1086},"Pairwise scripts, HyperPod topology surfaces (HyperPod API, EKS labels, Slurm ",{"type":50,"tag":136,"props":1088,"children":1090},{"className":1089},[],[1091],{"type":55,"value":1092},"topology.conf",{"type":55,"value":1094},"), and GB200 NVL72 specifics are in ",{"type":50,"tag":397,"props":1096,"children":1098},{"href":1097},"references\u002Fperf-details.md#uneven-nccl",[1099],{"type":55,"value":1100},"references\u002Fperf-details.md § Uneven NCCL",{"type":55,"value":324},{"type":50,"tag":972,"props":1103,"children":1105},{"id":1104},"topology-verification",[1106],{"type":55,"value":1107},"Topology verification",{"type":50,"tag":91,"props":1109,"children":1110},{},[1111],{"type":55,"value":1112},"HyperPod exposes topology through three operator-visible surfaces:",{"type":50,"tag":305,"props":1114,"children":1115},{},[1116,1156,1188],{"type":50,"tag":62,"props":1117,"children":1118},{},[1119,1124,1125,1131,1133,1139,1140,1146,1148,1154],{"type":50,"tag":66,"props":1120,"children":1121},{},[1122],{"type":55,"value":1123},"HyperPod API",{"type":55,"value":589},{"type":50,"tag":136,"props":1126,"children":1128},{"className":1127},[],[1129],{"type":55,"value":1130},"aws sagemaker describe-cluster-node",{"type":55,"value":1132}," returns ",{"type":50,"tag":136,"props":1134,"children":1136},{"className":1135},[],[1137],{"type":55,"value":1138},"NodeDetails.Placement.AvailabilityZone",{"type":55,"value":629},{"type":50,"tag":136,"props":1141,"children":1143},{"className":1142},[],[1144],{"type":55,"value":1145},"AvailabilityZoneId",{"type":55,"value":1147}," and ",{"type":50,"tag":136,"props":1149,"children":1151},{"className":1150},[],[1152],{"type":55,"value":1153},"NodeDetails.UltraServerInfo.Id",{"type":55,"value":1155}," (UltraServer SKUs only).",{"type":50,"tag":62,"props":1157,"children":1158},{},[1159,1164,1165,1171,1173,1179,1181,1187],{"type":50,"tag":66,"props":1160,"children":1161},{},[1162],{"type":55,"value":1163},"EKS labels",{"type":55,"value":589},{"type":50,"tag":136,"props":1166,"children":1168},{"className":1167},[],[1169],{"type":55,"value":1170},"topology.kubernetes.io\u002Fzone",{"type":55,"value":1172},", ",{"type":50,"tag":136,"props":1174,"children":1176},{"className":1175},[],[1177],{"type":55,"value":1178},"topology.k8s.aws\u002Fnetwork-node-layer-{1,2,3}",{"type":55,"value":1180}," (highest-numbered = closest to instance), ",{"type":50,"tag":136,"props":1182,"children":1184},{"className":1183},[],[1185],{"type":55,"value":1186},"topology.k8s.aws\u002Fultraserver-id",{"type":55,"value":324},{"type":50,"tag":62,"props":1189,"children":1190},{},[1191,1196,1198,1203,1205,1211],{"type":50,"tag":66,"props":1192,"children":1193},{},[1194],{"type":55,"value":1195},"Slurm",{"type":55,"value":1197},": HyperPod auto-generates ",{"type":50,"tag":136,"props":1199,"children":1201},{"className":1200},[],[1202],{"type":55,"value":1092},{"type":55,"value":1204},". Inspect via ",{"type":50,"tag":136,"props":1206,"children":1208},{"className":1207},[],[1209],{"type":55,"value":1210},"scontrol show topology",{"type":55,"value":324},{"type":50,"tag":91,"props":1213,"children":1214},{},[1215,1217,1223,1224,1229,1231,1236],{"type":55,"value":1216},"Tightly coupled work shares the same AZ, the same highest-numbered network-node-layer label (EKS) or the same Slurm topology block, and — for NVL72 jobs — the same ",{"type":50,"tag":136,"props":1218,"children":1220},{"className":1219},[],[1221],{"type":55,"value":1222},"UltraServerInfo.Id",{"type":55,"value":629},{"type":50,"tag":136,"props":1225,"children":1227},{"className":1226},[],[1228],{"type":55,"value":1186},{"type":55,"value":1230},". If the cluster is spread across AZs or layers, topology must be re-established at provisioning time. Route provisioning changes to ",{"type":50,"tag":136,"props":1232,"children":1234},{"className":1233},[],[1235],{"type":55,"value":244},{"type":55,"value":1237}," § B (Capacity & AZ).",{"type":50,"tag":953,"props":1239,"children":1240},{},[],{"type":50,"tag":84,"props":1242,"children":1244},{"id":1243},"b-poor-filesystem-performance",[1245],{"type":55,"value":833},{"type":50,"tag":91,"props":1247,"children":1248},{},[1249,1251,1257],{"type":55,"value":1250},"The customer reports training bottlenecked on data loading, checkpoint save\u002Fload dominating step time, executables\u002Fscripts loading slowly, or ",{"type":50,"tag":136,"props":1252,"children":1254},{"className":1253},[],[1255],{"type":55,"value":1256},"iowait",{"type":55,"value":1258}," high.",{"type":50,"tag":91,"props":1260,"children":1261},{},[1262],{"type":55,"value":1263},"Per the official troubleshooting guide, the resolution path follows this order:",{"type":50,"tag":58,"props":1265,"children":1266},{},[1267,1272,1277,1282,1287],{"type":50,"tag":62,"props":1268,"children":1269},{},[1270],{"type":55,"value":1271},"Check CloudWatch metrics on the filesystem.",{"type":50,"tag":62,"props":1273,"children":1274},{},[1275],{"type":55,"value":1276},"Check the provisioned performance configuration against workload requirements.",{"type":50,"tag":62,"props":1278,"children":1279},{},[1280],{"type":55,"value":1281},"Investigate which operations are causing the I\u002FO — workload demand vs inefficient pattern.",{"type":50,"tag":62,"props":1283,"children":1284},{},[1285],{"type":55,"value":1286},"Consider upgrading provisioned performance.",{"type":50,"tag":62,"props":1288,"children":1289},{},[1290],{"type":55,"value":1291},"Choose the filesystem type that matches the I\u002FO pattern.",{"type":50,"tag":91,"props":1293,"children":1294},{},[1295],{"type":55,"value":1296},"This skill covers steps 1–3. Steps 4–5 are customer decisions; surface the data and let the customer pick.",{"type":50,"tag":972,"props":1298,"children":1300},{"id":1299},"diagnostic-pass-read-only-1",[1301],{"type":55,"value":977},{"type":50,"tag":91,"props":1303,"children":1304},{},[1305,1310,1312,1318,1320,1326,1328,1334,1336,1341,1343,1349,1351,1356,1358,1363,1365,1370],{"type":50,"tag":136,"props":1306,"children":1308},{"className":1307},[],[1309],{"type":55,"value":375},{"type":55,"value":1311}," (Step 1 above) covers the on-node side of this pass: it discovers FSx mounts, calls ",{"type":50,"tag":136,"props":1313,"children":1315},{"className":1314},[],[1316],{"type":55,"value":1317},"aws cloudwatch get-metric-statistics",{"type":55,"value":1319}," on ",{"type":50,"tag":136,"props":1321,"children":1323},{"className":1322},[],[1324],{"type":55,"value":1325},"DataReadBytes",{"type":55,"value":1327}," and (for OpenZFS) ",{"type":50,"tag":136,"props":1329,"children":1331},{"className":1330},[],[1332],{"type":55,"value":1333},"FileServerDiskIopsUtilization",{"type":55,"value":1335},", prints ",{"type":50,"tag":136,"props":1337,"children":1339},{"className":1338},[],[1340],{"type":55,"value":627},{"type":55,"value":1342}," for ",{"type":50,"tag":136,"props":1344,"children":1346},{"className":1345},[],[1347],{"type":55,"value":1348},"\u002Ffsx \u002Fopt\u002Fdlami\u002Fnvme \u002Fopt\u002Fsagemaker",{"type":55,"value":1350},", runs ",{"type":50,"tag":136,"props":1352,"children":1354},{"className":1353},[],[1355],{"type":55,"value":635},{"type":55,"value":1357}," per Lustre mount, and reports ",{"type":50,"tag":136,"props":1359,"children":1361},{"className":1360},[],[1362],{"type":55,"value":819},{"type":55,"value":1364}," iowait. It tags ",{"type":50,"tag":136,"props":1366,"children":1368},{"className":1367},[],[1369],{"type":55,"value":388},{"type":55,"value":1371}," when OpenZFS IOPS utilization sustains ≥ 80% or iowait > 20%.",{"type":50,"tag":91,"props":1373,"children":1374},{},[1375,1377,1383,1385,1391,1393,1399],{"type":55,"value":1376},"For longer windows or additional metrics (",{"type":50,"tag":136,"props":1378,"children":1380},{"className":1379},[],[1381],{"type":55,"value":1382},"DataWriteBytes",{"type":55,"value":1384},", Lustre ",{"type":50,"tag":136,"props":1386,"children":1388},{"className":1387},[],[1389],{"type":55,"value":1390},"DiskIopsUtilization",{"type":55,"value":1392},", OpenZFS ",{"type":50,"tag":136,"props":1394,"children":1396},{"className":1395},[],[1397],{"type":55,"value":1398},"FileServerDiskThroughputUtilization",{"type":55,"value":1400},"), drive the query directly:",{"type":50,"tag":412,"props":1402,"children":1404},{"className":414,"code":1403,"language":416,"meta":417,"style":417},"aws cloudwatch get-metric-statistics --region \u003CREGION> \\\n  --namespace AWS\u002FFSx --metric-name DataReadBytes \\\n  --dimensions Name=FileSystemId,Value=\u003CFSID> \\\n  --start-time \"$(date -u -d '3 hours ago' +%Y-%m-%dT%H:%M:%S)\" \\\n  --end-time   \"$(date -u +%Y-%m-%dT%H:%M:%S)\" \\\n  --period 60 --statistics Sum Maximum\n",[1405],{"type":50,"tag":136,"props":1406,"children":1407},{"__ignoreMap":417},[1408,1450,1477,1512,1563,1594],{"type":50,"tag":423,"props":1409,"children":1410},{"class":425,"line":426},[1411,1415,1420,1425,1429,1433,1437,1441,1445],{"type":50,"tag":423,"props":1412,"children":1413},{"style":430},[1414],{"type":55,"value":22},{"type":50,"tag":423,"props":1416,"children":1417},{"style":435},[1418],{"type":55,"value":1419}," cloudwatch",{"type":50,"tag":423,"props":1421,"children":1422},{"style":435},[1423],{"type":55,"value":1424}," get-metric-statistics",{"type":50,"tag":423,"props":1426,"children":1427},{"style":435},[1428],{"type":55,"value":470},{"type":50,"tag":423,"props":1430,"children":1431},{"style":446},[1432],{"type":55,"value":449},{"type":50,"tag":423,"props":1434,"children":1435},{"style":435},[1436],{"type":55,"value":479},{"type":50,"tag":423,"props":1438,"children":1439},{"style":457},[1440],{"type":55,"value":460},{"type":50,"tag":423,"props":1442,"children":1443},{"style":446},[1444],{"type":55,"value":465},{"type":50,"tag":423,"props":1446,"children":1447},{"style":457},[1448],{"type":55,"value":1449}," \\\n",{"type":50,"tag":423,"props":1451,"children":1452},{"class":425,"line":491},[1453,1458,1463,1468,1473],{"type":50,"tag":423,"props":1454,"children":1455},{"style":435},[1456],{"type":55,"value":1457},"  --namespace",{"type":50,"tag":423,"props":1459,"children":1460},{"style":435},[1461],{"type":55,"value":1462}," AWS\u002FFSx",{"type":50,"tag":423,"props":1464,"children":1465},{"style":435},[1466],{"type":55,"value":1467}," --metric-name",{"type":50,"tag":423,"props":1469,"children":1470},{"style":435},[1471],{"type":55,"value":1472}," DataReadBytes",{"type":50,"tag":423,"props":1474,"children":1475},{"style":457},[1476],{"type":55,"value":1449},{"type":50,"tag":423,"props":1478,"children":1479},{"class":425,"line":501},[1480,1485,1490,1495,1500,1504,1508],{"type":50,"tag":423,"props":1481,"children":1482},{"style":435},[1483],{"type":55,"value":1484},"  --dimensions",{"type":50,"tag":423,"props":1486,"children":1487},{"style":435},[1488],{"type":55,"value":1489}," Name=FileSystemId,Value=",{"type":50,"tag":423,"props":1491,"children":1492},{"style":446},[1493],{"type":55,"value":1494},"\u003C",{"type":50,"tag":423,"props":1496,"children":1497},{"style":435},[1498],{"type":55,"value":1499},"FSI",{"type":50,"tag":423,"props":1501,"children":1502},{"style":457},[1503],{"type":55,"value":573},{"type":50,"tag":423,"props":1505,"children":1506},{"style":446},[1507],{"type":55,"value":465},{"type":50,"tag":423,"props":1509,"children":1510},{"style":457},[1511],{"type":55,"value":1449},{"type":50,"tag":423,"props":1513,"children":1514},{"class":425,"line":511},[1515,1520,1525,1530,1535,1540,1545,1549,1554,1559],{"type":50,"tag":423,"props":1516,"children":1517},{"style":435},[1518],{"type":55,"value":1519},"  --start-time",{"type":50,"tag":423,"props":1521,"children":1522},{"style":446},[1523],{"type":55,"value":1524}," \"$(",{"type":50,"tag":423,"props":1526,"children":1527},{"style":430},[1528],{"type":55,"value":1529},"date",{"type":50,"tag":423,"props":1531,"children":1532},{"style":435},[1533],{"type":55,"value":1534}," -u -d ",{"type":50,"tag":423,"props":1536,"children":1537},{"style":446},[1538],{"type":55,"value":1539},"'",{"type":50,"tag":423,"props":1541,"children":1542},{"style":435},[1543],{"type":55,"value":1544},"3 hours ago",{"type":50,"tag":423,"props":1546,"children":1547},{"style":446},[1548],{"type":55,"value":1539},{"type":50,"tag":423,"props":1550,"children":1551},{"style":435},[1552],{"type":55,"value":1553}," +%Y-%m-%dT%H:%M:%S",{"type":50,"tag":423,"props":1555,"children":1556},{"style":446},[1557],{"type":55,"value":1558},")\"",{"type":50,"tag":423,"props":1560,"children":1561},{"style":457},[1562],{"type":55,"value":1449},{"type":50,"tag":423,"props":1564,"children":1566},{"class":425,"line":1565},5,[1567,1572,1577,1581,1586,1590],{"type":50,"tag":423,"props":1568,"children":1569},{"style":435},[1570],{"type":55,"value":1571},"  --end-time",{"type":50,"tag":423,"props":1573,"children":1574},{"style":446},[1575],{"type":55,"value":1576},"   \"$(",{"type":50,"tag":423,"props":1578,"children":1579},{"style":430},[1580],{"type":55,"value":1529},{"type":50,"tag":423,"props":1582,"children":1583},{"style":435},[1584],{"type":55,"value":1585}," -u +%Y-%m-%dT%H:%M:%S",{"type":50,"tag":423,"props":1587,"children":1588},{"style":446},[1589],{"type":55,"value":1558},{"type":50,"tag":423,"props":1591,"children":1592},{"style":457},[1593],{"type":55,"value":1449},{"type":50,"tag":423,"props":1595,"children":1597},{"class":425,"line":1596},6,[1598,1603,1609,1614,1619],{"type":50,"tag":423,"props":1599,"children":1600},{"style":435},[1601],{"type":55,"value":1602},"  --period",{"type":50,"tag":423,"props":1604,"children":1606},{"style":1605},"--shiki-light:#F76D47;--shiki-default:#F78C6C;--shiki-dark:#F78C6C",[1607],{"type":55,"value":1608}," 60",{"type":50,"tag":423,"props":1610,"children":1611},{"style":435},[1612],{"type":55,"value":1613}," --statistics",{"type":50,"tag":423,"props":1615,"children":1616},{"style":435},[1617],{"type":55,"value":1618}," Sum",{"type":50,"tag":423,"props":1620,"children":1621},{"style":435},[1622],{"type":55,"value":1623}," Maximum\n",{"type":50,"tag":91,"props":1625,"children":1626},{},[1627,1629,1635],{"type":55,"value":1628},"The full per-filesystem-type metric catalog is in ",{"type":50,"tag":397,"props":1630,"children":1632},{"href":1631},"references\u002Fperf-details.md#filesystem",[1633],{"type":55,"value":1634},"references\u002Fperf-details.md § Filesystem",{"type":55,"value":324},{"type":50,"tag":972,"props":1637,"children":1639},{"id":1638},"branches",[1640],{"type":55,"value":1641},"Branches",{"type":50,"tag":91,"props":1643,"children":1644},{},[1645,1650],{"type":50,"tag":66,"props":1646,"children":1647},{},[1648],{"type":55,"value":1649},"Provisioned capacity is saturated.",{"type":55,"value":1651}," CloudWatch utilization sustained near 100% across the workload window. Customer decision: scale up the filesystem.",{"type":50,"tag":305,"props":1653,"children":1654},{},[1655,1668],{"type":50,"tag":62,"props":1656,"children":1657},{},[1658,1660,1666],{"type":55,"value":1659},"FSx for Lustre throughput scales with ",{"type":50,"tag":136,"props":1661,"children":1663},{"className":1662},[],[1664],{"type":55,"value":1665},"StorageCapacity × PerUnitStorageThroughput",{"type":55,"value":1667},"; capacity changes are non-disruptive.",{"type":50,"tag":62,"props":1669,"children":1670},{},[1671],{"type":55,"value":1672},"FSx for OpenZFS — increase provisioned IOPS or throughput.",{"type":50,"tag":91,"props":1674,"children":1675},{},[1676,1681],{"type":50,"tag":66,"props":1677,"children":1678},{},[1679],{"type":55,"value":1680},"I\u002FO pattern is inefficient.",{"type":55,"value":1682}," CloudWatch shows headroom but the workload is still I\u002FO-bound. Customer decision: change the application.",{"type":50,"tag":305,"props":1684,"children":1685},{},[1686,1713,1742],{"type":50,"tag":62,"props":1687,"children":1688},{},[1689,1691,1697,1699,1705,1706,1712],{"type":55,"value":1690},"DataLoader: raise ",{"type":50,"tag":136,"props":1692,"children":1694},{"className":1693},[],[1695],{"type":55,"value":1696},"num_workers",{"type":55,"value":1698},", set ",{"type":50,"tag":136,"props":1700,"children":1702},{"className":1701},[],[1703],{"type":55,"value":1704},"pin_memory=True",{"type":55,"value":1172},{"type":50,"tag":136,"props":1707,"children":1709},{"className":1708},[],[1710],{"type":55,"value":1711},"persistent_workers=True",{"type":55,"value":324},{"type":50,"tag":62,"props":1714,"children":1715},{},[1716,1718,1724,1726,1732,1734,1740],{"type":55,"value":1717},"Checkpointing: use async + sharded (",{"type":50,"tag":136,"props":1719,"children":1721},{"className":1720},[],[1722],{"type":55,"value":1723},"torch.distributed.checkpoint.async_save",{"type":55,"value":1725}," plus FSDP ",{"type":50,"tag":136,"props":1727,"children":1729},{"className":1728},[],[1730],{"type":55,"value":1731},"SHARDED_STATE_DICT",{"type":55,"value":1733},"). ",{"type":50,"tag":136,"props":1735,"children":1737},{"className":1736},[],[1738],{"type":55,"value":1739},"FULL_STATE_DICT",{"type":55,"value":1741}," serializes through rank 0 and is a frequent root cause.",{"type":50,"tag":62,"props":1743,"children":1744},{},[1745],{"type":55,"value":1746},"Small-file workloads: Lustre is optimized for large sequential I\u002FO. For millions of small files, use WebDataset \u002F tar shards, FSx for OpenZFS, or NVMe scratch.",{"type":50,"tag":91,"props":1748,"children":1749},{},[1750,1752,1756],{"type":55,"value":1751},"Filesystem-selection guidance and the async-checkpoint pattern are in ",{"type":50,"tag":397,"props":1753,"children":1754},{"href":1631},[1755],{"type":55,"value":1634},{"type":55,"value":324},{"type":50,"tag":953,"props":1758,"children":1759},{},[],{"type":50,"tag":84,"props":1761,"children":1763},{"id":1762},"continuous-health-coverage",[1764],{"type":55,"value":1765},"Continuous health coverage",{"type":50,"tag":91,"props":1767,"children":1768},{},[1769],{"type":55,"value":1770},"Once the immediate incident is diagnosed, recommend HyperPod's built-in health features so problems are caught before the next training run rather than after another customer-reported regression.",{"type":50,"tag":305,"props":1772,"children":1773},{},[1774,1790,1846],{"type":50,"tag":62,"props":1775,"children":1776},{},[1777,1788],{"type":50,"tag":66,"props":1778,"children":1779},{},[1780,1782],{"type":55,"value":1781},"Enable ",{"type":50,"tag":136,"props":1783,"children":1785},{"className":1784},[],[1786],{"type":55,"value":1787},"NodeRecovery=Automatic",{"type":55,"value":1789}," on the cluster. The Health Monitoring Agent (HMA) continuously monitors GPU- and Trainium-based instances and marks instances unhealthy on detected failure. With auto-recovery enabled, HyperPod reboots or replaces the node — no operator intervention.",{"type":50,"tag":62,"props":1791,"children":1792},{},[1793,1805,1807,1840,1844],{"type":50,"tag":66,"props":1794,"children":1795},{},[1796,1797,1803],{"type":55,"value":1781},{"type":50,"tag":136,"props":1798,"children":1800},{"className":1799},[],[1801],{"type":55,"value":1802},"OnStartDeepHealthChecks",{"type":55,"value":1804}," on every GPU instance group",{"type":55,"value":1806}," with both check categories:",{"type":50,"tag":305,"props":1808,"children":1809},{},[1810,1829],{"type":50,"tag":62,"props":1811,"children":1812},{},[1813,1819,1821,1827],{"type":50,"tag":136,"props":1814,"children":1816},{"className":1815},[],[1817],{"type":55,"value":1818},"InstanceStress",{"type":55,"value":1820}," — ",{"type":50,"tag":136,"props":1822,"children":1824},{"className":1823},[],[1825],{"type":55,"value":1826},"stress-ng",{"type":55,"value":1828}," on CPU\u002Fmemory\u002Fdisk, GPU and PCI device count verification, DCGM level-4 diagnostics (memory test included), and EFA loopback bandwidth\u002Flatency.",{"type":50,"tag":62,"props":1830,"children":1831},{},[1832,1838],{"type":50,"tag":136,"props":1833,"children":1835},{"className":1834},[],[1836],{"type":55,"value":1837},"InstanceConnectivity",{"type":55,"value":1839}," — multi-node NCCL all-reduce.",{"type":50,"tag":1841,"props":1842,"children":1843},"br",{},[],{"type":55,"value":1845},"Every newly provisioned or auto-replaced node passes the same hardware bar before accepting jobs.",{"type":50,"tag":62,"props":1847,"children":1848},{},[1849,1854,1856,1862,1864,1869,1870,1873,1875,1881,1883,1889],{"type":50,"tag":66,"props":1850,"children":1851},{},[1852],{"type":55,"value":1853},"Run on-demand deep health checks",{"type":55,"value":1855}," when this skill or any sibling surfaces a hardware concern but the cluster is mid-workload. ",{"type":50,"tag":136,"props":1857,"children":1859},{"className":1858},[],[1860],{"type":55,"value":1861},"aws sagemaker start-cluster-health-check",{"type":55,"value":1863}," runs the same checks against a specific instance group; nodes are placed in a Slurm maintenance reservation and the check is queued until any running job completes (not preempted). Console: ",{"type":50,"tag":66,"props":1865,"children":1866},{},[1867],{"type":55,"value":1868},"HyperPod → Clusters → Instances → Run deep health checks",{"type":55,"value":324},{"type":50,"tag":1841,"props":1871,"children":1872},{},[],{"type":55,"value":1874},"Not supported when ",{"type":50,"tag":136,"props":1876,"children":1878},{"className":1877},[],[1879],{"type":55,"value":1880},"NodeProvisioningMode=Continuous",{"type":55,"value":1882},"; one on-demand request per cluster at a time. Requires the latest AMI — run ",{"type":50,"tag":136,"props":1884,"children":1886},{"className":1885},[],[1887],{"type":55,"value":1888},"UpdateClusterSoftware",{"type":55,"value":1890}," first.",{"type":50,"tag":91,"props":1892,"children":1893},{},[1894,1896,1902,1904,1910,1912,1918],{"type":55,"value":1895},"Logs land in CloudWatch at ",{"type":50,"tag":136,"props":1897,"children":1899},{"className":1898},[],[1900],{"type":55,"value":1901},"\u002Faws\u002Fsagemaker\u002FClusters\u002F\u003Ccluster_name>\u002F\u003Ccluster_id>",{"type":55,"value":1903}," under ",{"type":50,"tag":136,"props":1905,"children":1907},{"className":1906},[],[1908],{"type":55,"value":1909},"DeepHealthCheckResults\u002F\u003Clog_stream_id>",{"type":55,"value":1911},", and on each node at ",{"type":50,"tag":136,"props":1913,"children":1915},{"className":1914},[],[1916],{"type":55,"value":1917},"\u002Fvar\u002Flog\u002Faws\u002Fclusters\u002Fsagemaker-deep-health-check.log",{"type":55,"value":324},{"type":50,"tag":84,"props":1920,"children":1922},{"id":1921},"references",[1923],{"type":55,"value":1924},"References",{"type":50,"tag":305,"props":1926,"children":1927},{},[1928],{"type":50,"tag":62,"props":1929,"children":1930},{},[1931,1936],{"type":50,"tag":397,"props":1932,"children":1934},{"href":1933},"references\u002Fperf-details.md",[1935],{"type":55,"value":1933},{"type":55,"value":1937}," — pairwise NCCL test recipes, HyperPod topology check, GB200 NVL72 placement; CloudWatch metric catalog per filesystem type, async-checkpoint pattern, filesystem selection guide.",{"type":50,"tag":91,"props":1939,"children":1940},{},[1941],{"type":55,"value":1942},"External:",{"type":50,"tag":305,"props":1944,"children":1945},{},[1946,1957,1974,1985,1996,2007,2018,2029,2040,2051],{"type":50,"tag":62,"props":1947,"children":1948},{},[1949,1951],{"type":55,"value":1950},"Amazon SageMaker HyperPod troubleshooting guide (official): ",{"type":50,"tag":397,"props":1952,"children":1955},{"href":1953,"rel":1954},"https:\u002F\u002Fgithub.com\u002Faws\u002Fsagemaker-hyperpod-cluster-setup\u002Fblob\u002Ftroubleshooting-doc-20250917\u002Ftroubleshoot\u002Findex.md",[1047],[1956],{"type":55,"value":1953},{"type":50,"tag":62,"props":1958,"children":1959},{},[1960,1962,1967,1969],{"type":55,"value":1961},"AI-on-HyperPod NCCL performance tests (expected ",{"type":50,"tag":136,"props":1963,"children":1965},{"className":1964},[],[1966],{"type":55,"value":1058},{"type":55,"value":1968}," per SKU): ",{"type":50,"tag":397,"props":1970,"children":1972},{"href":1075,"rel":1971},[1047],[1973],{"type":55,"value":1075},{"type":50,"tag":62,"props":1975,"children":1976},{},[1977,1979],{"type":55,"value":1978},"Amazon SageMaker HyperPod resiliency (NodeRecovery, HMA, auto-resume): ",{"type":50,"tag":397,"props":1980,"children":1983},{"href":1981,"rel":1982},"https:\u002F\u002Fdocs.aws.amazon.com\u002Fsagemaker\u002Flatest\u002Fdg\u002Fsagemaker-hyperpod-resiliency.html",[1047],[1984],{"type":55,"value":1981},{"type":50,"tag":62,"props":1986,"children":1987},{},[1988,1990],{"type":55,"value":1989},"Amazon SageMaker HyperPod deep health checks: ",{"type":50,"tag":397,"props":1991,"children":1994},{"href":1992,"rel":1993},"https:\u002F\u002Fdocs.aws.amazon.com\u002Fsagemaker\u002Flatest\u002Fdg\u002Fsagemaker-hyperpod-resiliency-slurm-deep-health-checks.html",[1047],[1995],{"type":55,"value":1992},{"type":50,"tag":62,"props":1997,"children":1998},{},[1999,2001],{"type":55,"value":2000},"StartClusterHealthCheck API: ",{"type":50,"tag":397,"props":2002,"children":2005},{"href":2003,"rel":2004},"https:\u002F\u002Fdocs.aws.amazon.com\u002Fsagemaker\u002Flatest\u002FAPIReference\u002FAPI_StartClusterHealthCheck.html",[1047],[2006],{"type":55,"value":2003},{"type":50,"tag":62,"props":2008,"children":2009},{},[2010,2012],{"type":55,"value":2011},"Amazon EC2 instance topology \u002F network-node-layer labels: ",{"type":50,"tag":397,"props":2013,"children":2016},{"href":2014,"rel":2015},"https:\u002F\u002Fdocs.aws.amazon.com\u002FAWSEC2\u002Flatest\u002FUserGuide\u002Fhow-ec2-instance-topology-works.html",[1047],[2017],{"type":55,"value":2014},{"type":50,"tag":62,"props":2019,"children":2020},{},[2021,2023],{"type":55,"value":2022},"Amazon FSx for Lustre performance: ",{"type":50,"tag":397,"props":2024,"children":2027},{"href":2025,"rel":2026},"https:\u002F\u002Fdocs.aws.amazon.com\u002Ffsx\u002Flatest\u002FLustreGuide\u002Fperformance.html",[1047],[2028],{"type":55,"value":2025},{"type":50,"tag":62,"props":2030,"children":2031},{},[2032,2034],{"type":55,"value":2033},"Amazon FSx for OpenZFS metrics: ",{"type":50,"tag":397,"props":2035,"children":2038},{"href":2036,"rel":2037},"https:\u002F\u002Fdocs.aws.amazon.com\u002Ffsx\u002Flatest\u002FOpenZFSGuide\u002Ffsx-openzfs-metrics.html",[1047],[2039],{"type":55,"value":2036},{"type":50,"tag":62,"props":2041,"children":2042},{},[2043,2045],{"type":55,"value":2044},"AWS Elastic Fabric Adapter and NCCL: ",{"type":50,"tag":397,"props":2046,"children":2049},{"href":2047,"rel":2048},"https:\u002F\u002Fdocs.aws.amazon.com\u002FAWSEC2\u002Flatest\u002FUserGuide\u002Fefa-start-nccl.html",[1047],[2050],{"type":55,"value":2047},{"type":50,"tag":62,"props":2052,"children":2053},{},[2054,2056],{"type":55,"value":2055},"awslabs\u002Fawsome-distributed-training NCCL tests: ",{"type":50,"tag":397,"props":2057,"children":2059},{"href":1045,"rel":2058},[1047],[2060],{"type":55,"value":1045},{"type":50,"tag":2062,"props":2063,"children":2064},"style",{},[2065],{"type":55,"value":2066},"html .light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html.light .shiki span {color: var(--shiki-light);background: var(--shiki-light-bg);font-style: var(--shiki-light-font-style);font-weight: var(--shiki-light-font-weight);text-decoration: var(--shiki-light-text-decoration);}html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"items":2068,"total":2242},[2069,2084,2105,2115,2128,2141,2151,2161,2182,2197,2212,2227],{"slug":2070,"name":2070,"fn":2071,"description":2072,"org":2073,"tags":2074,"stars":2081,"repoUrl":2082,"updatedAt":2083},"agentcore-investigation","investigate Bedrock AgentCore runtime sessions","Investigate Bedrock AgentCore runtime sessions via CloudWatch Logs Insights — resolve session\u002Ftrace IDs, query OTEL spans, filter noise, build timelines. Use when debugging AgentCore agent sessions, tracing tool calls, or analyzing latency.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2075,2076,2077,2080],{"name":21,"slug":22,"type":16},{"name":24,"slug":25,"type":16},{"name":2078,"slug":2079,"type":16},"Logs","logs",{"name":14,"slug":15,"type":16},9427,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fmcp","2026-07-12T08:37:22.601527",{"slug":2085,"name":2086,"fn":2087,"description":2088,"org":2089,"tags":2090,"stars":2081,"repoUrl":2082,"updatedAt":2104},"amazon-aurora-dsql","amazon aurora dsql","build applications with Aurora DSQL","Build with Aurora DSQL — manage schemas, execute queries, handle migrations, diagnose query plans, load data, and develop applications with a serverless, distributed SQL database. Covers IAM auth, multi-tenant patterns, MySQL-to-DSQL and PostgreSQL-to-DSQL schema conversion, FK replacement code generation, OCC retry patterns, ORM migration (Django\u002FHibernate\u002FRails), DDL operations, query plan explainability, SQL compatibility validation, and bulk data loading. Triggers on phrases like: DSQL, Aurora DSQL, create DSQL table, DSQL schema, migrate to DSQL, distributed SQL database, serverless PostgreSQL-compatible database, DSQL query plan, DSQL EXPLAIN ANALYZE, why is my DSQL query slow, DSQL foreign key, DSQL OCC retry, DSQL multi-region, load into DSQL, load CSV into DSQL, bulk load DSQL, aurora-dsql-loader.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2091,2094,2095,2098,2101],{"name":2092,"slug":2093,"type":16},"Aurora","aurora",{"name":21,"slug":22,"type":16},{"name":2096,"slug":2097,"type":16},"Database","database",{"name":2099,"slug":2100,"type":16},"Serverless","serverless",{"name":2102,"slug":2103,"type":16},"SQL","sql","2026-07-12T08:36:45.053393",{"slug":2106,"name":2107,"fn":2087,"description":2088,"org":2108,"tags":2109,"stars":2081,"repoUrl":2082,"updatedAt":2114},"aurora-dsql","aurora dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2110,2111,2112,2113],{"name":21,"slug":22,"type":16},{"name":2096,"slug":2097,"type":16},{"name":2099,"slug":2100,"type":16},{"name":2102,"slug":2103,"type":16},"2026-07-12T08:36:42.694299",{"slug":2116,"name":2117,"fn":2087,"description":2088,"org":2118,"tags":2119,"stars":2081,"repoUrl":2082,"updatedAt":2127},"aws-dsql","aws dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2120,2121,2122,2125,2126],{"name":21,"slug":22,"type":16},{"name":2096,"slug":2097,"type":16},{"name":2123,"slug":2124,"type":16},"Migration","migration",{"name":2099,"slug":2100,"type":16},{"name":2102,"slug":2103,"type":16},"2026-07-12T08:36:38.584057",{"slug":2129,"name":2130,"fn":2087,"description":2088,"org":2131,"tags":2132,"stars":2081,"repoUrl":2082,"updatedAt":2140},"distributed-postgres","distributed postgres",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2133,2134,2135,2138,2139],{"name":21,"slug":22,"type":16},{"name":2096,"slug":2097,"type":16},{"name":2136,"slug":2137,"type":16},"PostgreSQL","postgresql",{"name":2099,"slug":2100,"type":16},{"name":2102,"slug":2103,"type":16},"2026-07-12T08:36:46.530743",{"slug":2142,"name":2143,"fn":2087,"description":2088,"org":2144,"tags":2145,"stars":2081,"repoUrl":2082,"updatedAt":2150},"distributed-sql","distributed sql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2146,2147,2148,2149],{"name":21,"slug":22,"type":16},{"name":2096,"slug":2097,"type":16},{"name":2099,"slug":2100,"type":16},{"name":2102,"slug":2103,"type":16},"2026-07-12T08:36:48.104182",{"slug":2152,"name":2152,"fn":2087,"description":2088,"org":2153,"tags":2154,"stars":2081,"repoUrl":2082,"updatedAt":2160},"dsql",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2155,2156,2157,2158,2159],{"name":21,"slug":22,"type":16},{"name":2096,"slug":2097,"type":16},{"name":2123,"slug":2124,"type":16},{"name":2099,"slug":2100,"type":16},{"name":2102,"slug":2103,"type":16},"2026-07-12T08:36:36.374512",{"slug":2162,"name":2162,"fn":2163,"description":2164,"org":2165,"tags":2166,"stars":2179,"repoUrl":2180,"updatedAt":2181},"cost-efficiency-analyzer","analyze cost efficiency and expenses","Analyzes cost structure, cost efficiency, and expense management from P&L data. Use when the user asks about costs, expenses, COGS, operating expenses, cost ratios, cost control, spending efficiency, margin compression from cost side, or wants to understand where money is going. Also use for \"are we spending too much\", \"cost breakdown\", \"expense analysis\", or \"how efficient are our operations\". NOT for revenue or top-line analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2167,2170,2173,2176],{"name":2168,"slug":2169,"type":16},"Accounting","accounting",{"name":2171,"slug":2172,"type":16},"Analytics","analytics",{"name":2174,"slug":2175,"type":16},"Cost Optimization","cost-optimization",{"name":2177,"slug":2178,"type":16},"Finance","finance",3176,"https:\u002F\u002Fgithub.com\u002Fawslabs\u002Fagentcore-samples","2026-07-12T08:40:03.29555",{"slug":2183,"name":2183,"fn":2184,"description":2185,"org":2186,"tags":2187,"stars":2179,"repoUrl":2180,"updatedAt":2196},"executive-financial-briefing","generate executive financial briefings","Generates a concise executive-level financial briefing or summary suitable for a CEO, CFO, or board presentation. Use when the user asks for a summary, briefing, executive summary, board update, financial overview, financial health check, or \"how is the business doing\". Covers the full P&L picture in one page. Also use for \"give me the highlights\", \"what do I need to know\", or \"quick financial update\".",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2188,2189,2190,2193],{"name":21,"slug":22,"type":16},{"name":2177,"slug":2178,"type":16},{"name":2191,"slug":2192,"type":16},"Management","management",{"name":2194,"slug":2195,"type":16},"Reporting","reporting","2026-07-12T08:40:02.066471",{"slug":2198,"name":2198,"fn":2199,"description":2200,"org":2201,"tags":2202,"stars":2179,"repoUrl":2180,"updatedAt":2211},"multi-quarter-trend-analysis","analyze multi-quarter financial trends","Analyzes financial trends across multiple quarters by comparing P&L metrics over time. Use when the user wants to see trends, patterns, trajectories, or directional movement across 3 or more quarters. Also use for \"how are we trending\", \"show me the trend\", \"track performance over time\", \"quarter over quarter comparison across all quarters\", or any multi-period longitudinal analysis.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2203,2204,2205,2208],{"name":2171,"slug":2172,"type":16},{"name":2177,"slug":2178,"type":16},{"name":2206,"slug":2207,"type":16},"Financial Statements","financial-statements",{"name":2209,"slug":2210,"type":16},"Variance Analysis","variance-analysis","2026-07-12T08:40:00.79141",{"slug":2213,"name":2213,"fn":2214,"description":2215,"org":2216,"tags":2217,"stars":2179,"repoUrl":2180,"updatedAt":2226},"pdf","process and manipulate PDF documents","Use this skill whenever the user wants to do anything with PDF files. This includes reading or extracting text\u002Ftables from PDFs, combining or merging multiple PDFs into one, splitting PDFs apart, rotating pages, adding watermarks, creating new PDFs, filling PDF forms, encrypting\u002Fdecrypting PDFs, extracting images, and OCR on scanned PDFs to make them searchable. If the user mentions a .pdf file or asks to produce one, use this skill.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2218,2221,2224],{"name":2219,"slug":2220,"type":16},"Automation","automation",{"name":2222,"slug":2223,"type":16},"Documents","documents",{"name":2225,"slug":2213,"type":16},"PDF","2026-07-12T08:41:44.135656",{"slug":2228,"name":2228,"fn":2229,"description":2230,"org":2231,"tags":2232,"stars":2179,"repoUrl":2180,"updatedAt":2241},"quarterly-kpi-calculator","calculate quarterly financial KPIs","Calculates quarterly financial KPIs from P&L data. P&L figures can be provided directly by the user or fetched from the financial data MCP server. Use when the user wants KPI calculations such as Gross Margin %, EBITDA Margin %, Operating Expense Ratio, or Revenue Growth % QoQ. Also use for quarterly performance review, P&L analysis, or interpreting financial ratios against benchmarks.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2233,2234,2237,2238],{"name":2168,"slug":2169,"type":16},{"name":2235,"slug":2236,"type":16},"Data Analysis","data-analysis",{"name":2177,"slug":2178,"type":16},{"name":2239,"slug":2240,"type":16},"KPI","kpi","2026-07-12T08:39:59.54971",150,{"items":2244,"total":2343},[2245,2262,2280,2292,2309,2320,2330],{"slug":2246,"name":2246,"fn":2247,"description":2248,"org":2249,"tags":2250,"stars":26,"repoUrl":27,"updatedAt":2261},"amazon-location-service","integrate Amazon Location Service maps","Integrates Amazon Location Service APIs for AWS applications. Use this skill when users want to add maps (interactive MapLibre or static images); geocode addresses to coordinates or reverse geocode coordinates to addresses; calculate routes, travel times, or service areas; find places and businesses through text search, nearby search, or autocomplete suggestions; retrieve detailed place information including hours, contacts, and addresses; monitor geographical boundaries with geofences; or track device locations. Covers authentication, SDK integration, and all Amazon Location Service capabilities.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2251,2254,2255,2258],{"name":2252,"slug":2253,"type":16},"API Development","api-development",{"name":21,"slug":22,"type":16},{"name":2256,"slug":2257,"type":16},"Maps","maps",{"name":2259,"slug":2260,"type":16},"Navigation","navigation","2026-07-12T08:39:49.88311",{"slug":2263,"name":2263,"fn":2264,"description":2265,"org":2266,"tags":2267,"stars":26,"repoUrl":27,"updatedAt":2279},"amplify-workflow","build and deploy apps with AWS Amplify","Build and deploy full-stack web and mobile apps with AWS Amplify Gen2 (TypeScript code-first). Covers auth (Cognito), data (AppSync\u002FDynamoDB including schema modeling, enum types, relationships, authorization rules), storage (S3), functions, APIs, and AI (Amplify AI Kit with Bedrock). Supports React, Next.js, Vue, Angular, React Native, Flutter, Swift, and Android. Always use this skill for Amplify Gen2 topics — even for questions you think you know — it contains validated, version-specific patterns that prevent common mistakes. TRIGGER when: user mentions Amplify Gen2; project has amplify\u002F directory or amplify_outputs; code imports @aws-amplify packages; user asks about defineBackend, defineAuth, defineData, defineStorage, or npx ampx. SKIP: Amplify Gen1 (amplify CLI v6), standalone SAM\u002FCDK without Amplify (use aws-serverless), direct Bedrock without Amplify AI Kit (use bedrock).",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2268,2271,2272,2273,2276],{"name":2269,"slug":2270,"type":16},"Auth","auth",{"name":21,"slug":22,"type":16},{"name":2096,"slug":2097,"type":16},{"name":2274,"slug":2275,"type":16},"Deployment","deployment",{"name":2277,"slug":2278,"type":16},"TypeScript","typescript","2026-07-12T08:39:43.500162",{"slug":2281,"name":2281,"fn":2282,"description":2283,"org":2284,"tags":2285,"stars":26,"repoUrl":27,"updatedAt":2291},"api-gateway","build and manage Amazon API Gateway APIs","Build, manage, and operate APIs with Amazon API Gateway (REST, HTTP, and WebSocket). Triggers on phrases like: API Gateway, REST API, HTTP API, WebSocket API, custom domain, Lambda authorizer, usage plan, throttling, CORS, VPC link, private API. Also covers troubleshooting API Gateway errors (4xx, 5xx, timeout, CORS failures) and IaC templates containing API Gateway resources. For general REST API design unrelated to AWS, do not trigger.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2286,2287,2288],{"name":2252,"slug":2253,"type":16},{"name":21,"slug":22,"type":16},{"name":2289,"slug":2290,"type":16},"REST API","rest-api","2026-07-12T08:39:00.149339",{"slug":2293,"name":2293,"fn":2294,"description":2295,"org":2296,"tags":2297,"stars":26,"repoUrl":27,"updatedAt":2308},"aws-architecture-diagram","generate AWS architecture diagrams","Generate validated AWS architecture diagrams as draw.io XML using official AWS4 icon libraries. Use this skill whenever the user wants to create, generate, or design AWS architecture diagrams, cloud infrastructure diagrams, or system design visuals. Also triggers for requests to visualize existing infrastructure from CloudFormation, CDK, or Terraform code. Supports two modes: analyze an existing codebase to auto-generate diagrams, or brainstorm interactively from scratch. Exports .drawio files with optional PNG\u002FSVG\u002FPDF export via draw.io desktop CLI.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2298,2301,2302,2305],{"name":2299,"slug":2300,"type":16},"Architecture","architecture",{"name":21,"slug":22,"type":16},{"name":2303,"slug":2304,"type":16},"Design","design",{"name":2306,"slug":2307,"type":16},"Diagrams","diagrams","2026-07-12T08:37:11.012278",{"slug":2310,"name":2310,"fn":2311,"description":2312,"org":2313,"tags":2314,"stars":26,"repoUrl":27,"updatedAt":2319},"aws-lambda","build and deploy AWS Lambda functions","Design, build, deploy, test, and debug serverless applications with AWS Lambda. Triggers on phrases like: Lambda function, event source, serverless application, API Gateway, EventBridge, Step Functions, serverless API, event-driven architecture, Lambda trigger. For deploying non-serverless apps to AWS, use deploy-on-aws plugin instead.",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2315,2316,2317,2318],{"name":2252,"slug":2253,"type":16},{"name":21,"slug":22,"type":16},{"name":2274,"slug":2275,"type":16},{"name":2099,"slug":2100,"type":16},"2026-07-12T08:38:58.598492",{"slug":2321,"name":2321,"fn":2322,"description":2323,"org":2324,"tags":2325,"stars":26,"repoUrl":27,"updatedAt":2329},"aws-lambda-durable-functions","build resilient AWS Lambda durable functions","Build resilient, long-running, multi-step applications with AWS Lambda durable functions with automatic state persistence, retry logic, and orchestration for long-running executions. Covers the critical replay model, step operations, wait\u002Fcallback patterns, error handling with saga pattern, testing with LocalDurableTestRunner. Triggers on phrases like: lambda durable functions, workflow orchestration, state machines, retry\u002Fcheckpoint patterns, long-running stateful Lambda functions, saga pattern, human-in-the-loop callbacks, and reliable serverless applications.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2326,2327,2328],{"name":2299,"slug":2300,"type":16},{"name":21,"slug":22,"type":16},{"name":2099,"slug":2100,"type":16},"2026-07-12T08:39:05.546173",{"slug":2331,"name":2331,"fn":2332,"description":2333,"org":2334,"tags":2335,"stars":26,"repoUrl":27,"updatedAt":2342},"aws-lambda-managed-instances","configure AWS Lambda Managed Instances","Evaluate, configure, and migrate workloads to AWS Lambda Managed Instances (LMI). Triggers on: Lambda Managed Instances, LMI, capacity provider, multi-concurrency Lambda, dedicated instance Lambda, EC2-backed Lambda, cold start elimination, Graviton Lambda, instance type for Lambda, scheduled scaling for LMI, Lambda cost optimization with Reserved Instances or Savings Plans. Also trigger when users describe high-volume predictable workloads seeking cost savings, want to scale LMI capacity on a schedule, or compare Lambda vs EC2 for steady-state traffic. For standard Lambda without LMI, use the aws-lambda skill instead.\n",{"slug":8,"name":9,"logoUrl":10,"githubOrg":11},[2336,2337,2338,2341],{"name":21,"slug":22,"type":16},{"name":2274,"slug":2275,"type":16},{"name":2339,"slug":2340,"type":16},"Infrastructure","infrastructure",{"name":18,"slug":19,"type":16},"2026-07-12T08:39:07.007071",33]