
Skill
aws-vpc-dns-investigation
investigate VPC DNS resolution issues
Description
Use this skill when a name is not resolving as expected inside a VPC, or before applying a DNS control-plane change. Activate on symptoms such as NXDOMAIN or SERVFAIL from an EC2 instance, a hostname resolving to a public address when a private endpoint was expected, an AWS service endpoint that stopped resolving after a VPC endpoint or Route 53 change, an application reaching the wrong IP, resolution that works from one instance but not another, IPv6 or dualstack resolution differences, a suspected on-premises forwarding or hybrid DNS problem, or a request to check whether enabling private DNS, adding a Resolver rule, associating a private hosted zone, attaching DNS Firewall, or associating a Route 53 Profile would break anything. It drives the aws-vpc-dns-diagnostics MCP server to observe live resolution from inside the subnet and to simulate a proposed change before it is applied.
SKILL.md
Investigate VPC DNS Resolution
Use the tools on the connected aws-vpc-dns-diagnostics MCP server.
Step 1: Classify the request as Mode A or Mode B
Before calling any tool, determine which mode applies:
- Mode A (live diagnosis): The operator reports a resolution symptom from a running instance. They provide an instance ID (or you can identify one). The goal is to observe what actually resolves and compare resolvers.
- Mode B (pre-change validation): The operator asks whether a proposed DNS change is safe. They provide account, region, VPC, and a change descriptor. No instance is required.
If the request is ambiguous, ask the operator to clarify. Do not default to Mode A when the input lacks an instance ID, and do not default to Mode B when the operator describes a live symptom.
Step 2: Load safety rules
Regardless of mode, call get_sop with slug A-critical-safety-rules and
follow every rule it contains. These are non-negotiable constraints on how you
interpret results, handle opaque constructs, and report findings.
Mode A route: live diagnosis
Required inputs
account_id, region, instance_id, and the failing DNS name.
Tool sequence (in order)
dns_probe_context— establishes VPC-attribute preconditions: enableDnsSupport, enableDnsHostnames, address family, DHCP option set. A resolution result means nothing until you know whether the VPC resolver is answering.dns_probe_compare— runs the allowlisted probe set inside the instance via SSM. Returns each resolver's answer and the resolver's own identity fromhostname.bind. The VPC DHCP resolver is auto-added for comparison.get_sop— load the pattern runbook matching the observed signature (see trap-to-SOP mapping below).
Interpretation rules
- If
enableDnsSupportis false: loadA-resolver-disabled-precondition. The VPC resolver is intentionally dark and every probe failure follows from that. - Compare the instance's
/etc/resolv.conf(from the probe output) against the DHCP option set. A mismatch means the instance is not using the VPC-intended resolver. - Judge answers by name category (load
A-name-category-classification), not by whether resolvers agree. Two resolvers returning the same wrong answer is still a failure.
Mode A trap-to-SOP mapping
| Observed signature | SOP slug |
|---|---|
| Custom resolver answers differently from VPC .2 | A-custom-resolver-divergence |
| FORWARD rule and PHZ both match the name | A-forward-vs-phz-precedence-collision |
| A record works, AAAA fails (or vice versa) | A-address-family-divergence |
| enableDnsSupport is false | A-resolver-disabled-precondition |
| General live comparison procedure | A-mode-a-live-resolver-comparison |
Reporting format for Mode A
Label every finding as Observed (ground truth from the probe). State which resolver answered and what it returned. When Mode A and Mode B produce different conclusions for the same name, Mode A wins because it is ground truth from inside the subnet.
Mode B route: pre-change validation
Required inputs
account_id, region, vpc_id, and a change descriptor (structured dict with type
and type-specific fields). No instance required.
Tool sequence (in order)
dns_simulate_effective_config— returns the VPC's effective DNS config: the union of directly attached resources and anything inherited through an associated Route 53 Profile, each construct tagged by source.dns_simulate_change— applies the proposed change symbolically and returns a per-name impact report (before/after, delta, traps, severity, volume).get_sop— load runbooks for any traps reported in the impact table (see trap-to-SOP mapping below).
Interpretation rules
- Never recommend applying a change without simulating it first. A broad FORWARD rule, enabling private DNS on an interface endpoint, or a Profile association can silently redirect names that currently resolve correctly.
- The candidate set is limited to API-derived names (PHZ records, rule domains, VPCE apexes, Firewall domain lists) or operator-supplied names. It is not exhaustive. State the coverage boundary.
- If the operator supplies
volumes(from Resolver Query Logs), names are ranked by traffic. This is enrichment; absence does not invalidate the simulation.
Mode B trap-to-SOP mapping
| Trap label in impact report | SOP slug |
|---|---|
| VPCE-shadow-NXDOMAIN | B-vpce-shadow-nxdomain |
| broad-FORWARD-sweep | B-broad-forward-sweep |
| flag-AND-mismatch | B-flag-and-mismatch |
| DNS-Firewall-block | B-dns-firewall-block |
| profile-union-shift | B-profile-propagation-timing |
| General pre-change procedure | B-mode-b-pre-change-validation |
Reporting format for Mode B
Label every finding as Predicted (symbolic, not ground truth). State the candidate-set size, its source (API-derived or operator-supplied), and that names outside this set were not evaluated. Include the propagation timing caveat for Profile changes.
Cross-account opacity
Call get_sop with slug C-cross-account-opaque-constructs when the effective
config or impact report contains opaque markers. Cross-account constructs shared
via RAM or a Route 53 Profile may be enumerable but their contents are not
readable from the consumer account. Report them as "present but unknown content"
rather than treating them as absent or inferring past them.
Limitations
Call get_sop with slug C-limitations-and-boundaries and state the relevant
boundaries to the operator. Key constraints:
- All tools are read-only. Do not modify, delete, or create DNS resources.
- Mode A requires SSM reachability (ssm, ssmmessages, ec2messages VPC endpoints and an instance role with AmazonSSMManagedInstanceCore).
- Mode B candidate sets are not exhaustive. The "no impacts" conclusion applies only within the tested set.
- Opaque constructs cannot be resolved from this account.
- Resolver Query Log ingestion is not implemented; volumes must be supplied by the operator.
Final response requirements
Every response produced by this skill must include:
- Each finding labelled Observed (Mode A) or Predicted (Mode B).
- When both modes were used, state "Mode A wins" for any conflict.
- The candidate-set coverage: how many names, what source, what was not tested.
- Any opaque constructs and their impact on the conclusion.
- Recommended next steps or the specific change to apply (never apply it).
Prerequisites
Requires the aws-vpc-dns-diagnostics MCP server registered in the Agent Space
with its tools allowlisted. The server is at mcp/aws-vpc-dns-diagnostics-mcp/.
If the server is not registered or SSM is unreachable, report that as the blocker
rather than guessing at the resolution path.
More skills from the tools-for-devops-agent repository
View all 15 skillsaws-health-events
analyze AWS Health events for incidents
Aug 20AWSDebuggingIncident ResponseMonitoringcrm-production-investigation-guidelines
investigate CRM production incidents
Aug 20AWSCRMDebuggingIncident Response +1database-migration-service-expertise
troubleshoot AWS Database Migration Service
Aug 20AWSDatabaseDebuggingMigration +1database-rds-devops
diagnose Aurora MySQL and PostgreSQL databases
Aug 20AWSDatabaseDebuggingMySQL +1eks-operation-review
audit Amazon EKS cluster operations
Aug 20AuditAWSKubernetesOperationsenrich-with-aws-security-agent
investigate security root causes in AWS
Aug 20AuthAWSDebuggingIncident Response +1
More from AWS (Amazon)
View publisheragents-build
add capabilities to existing agent projects
agent-toolkit-for-aws
Aug 20AgentsAutomationAWSEngineeringagents-connect
connect agents to external services
agent-toolkit-for-aws
Jul 16AgentsAPI DevelopmentAuthenticationAWSagents-debug
debug agent and environment issues
agent-toolkit-for-aws
Jul 16AgentsAWSDebuggingObservabilityagents-deploy
deploy AI agents to AWS
agent-toolkit-for-aws
Jul 12AgentsAWSCI/CDDeploymentagents-get-started
scaffold and deploy new agent projects
agent-toolkit-for-aws
Jul 12AgentsAWSDeploymentagents-harden
harden agents for production
agent-toolkit-for-aws
Jul 16AgentsAWSBest PracticesSecurity