The network failures that present as success
Most AWS networking incidents announce themselves with a timeout. A smaller set return a valid answer and do the wrong thing, and every diagnostic instinct that works on the first kind has nothing to grab on the second.
· Venerable Networks
Almost every networking failure worth writing about ends in a timeout. A packet leaves and nothing comes back, someone gets paged, and the investigation follows the failing request until it finds where the packet died.
A smaller class does not fail. It succeeds, at the wrong thing, and it is worse for three reasons: nothing alarms, the failing-request method has no failing request to follow, and the duration is measured in months rather than minutes.
Four examples, all documented, all encountered while building diagnostic labs.
A DNS answer that is valid and stale
Two things can claim the same domain in one VPC: a private hosted zone and a Route 53 Resolver forwarding rule. At equal specificity the Resolver rule wins and the hosted zone is never consulted.
So a migration creates a correct private hosted zone, and an eighteen-month-old forwarding rule sends every lookup to a corporate resolver holding the pre-migration record. The application resolves the name, connects successfully, and talks to the host that was supposed to be decommissioned.
The obvious check passes. "Is the DNS record right?" is the correct first question and the answer is yes — verifying a record tells you what the zone contains, not whether the zone is being read.
The cheap signal is TTL 0 on the answer, because a forwarded response is relayed rather than served authoritatively. If your record has a TTL of 60 and the answer arrives with 0, it did not come from your zone.
Segmentation that only works one way
A Transit Gateway isolation route table with nothing propagated into it genuinely prevents the isolated VPC from initiating anything. Verified, correct, holds.
It says nothing about traffic arriving from elsewhere, because route lookup happens against the route table associated with the attachment traffic arrived on. And since DefaultRouteTableAssociation and DefaultRouteTablePropagation are both enabled by default, any attachment nobody explicitly associated joins the default route table where every prefix was propagated automatically.
Two teams end up correct and in disagreement. Networking tested the direction their table controls. Security tested the direction nobody's table controlled.
Nothing fails. It is an audit finding, discovered by whoever runs Reachability Analyzer, whenever they next run it.
An address family that is dead behind a working one
Enable IPv6 on a VPC and forget the ::/0 route. Instances get global addresses and an IPv6 default route from the VPC router, so they prefer IPv6 and their packets die at the VPC route table.
Happy Eyeballs then covers it. Clients try IPv6, wait out a timeout — 200 ms in curl by default — and fall back to IPv4. Every dual-stack destination gets slower by a fixed amount and nothing breaks. Only an IPv6-only dependency fails outright, whenever one appears.
A fixed latency penalty on dual-stack destinations only is a specific enough fingerprint to diagnose from, and it is exactly the kind of thing that gets written off as noise.
An endpoint that bills and serves nothing
An interface endpoint with private_dns_enabled = false — the Terraform default — reaches available, has a healthy ENI, passes reachability checks, and claims no hostname. Nothing routes to it and it costs the same per hour as one that works.
This one does eventually produce a timeout, so it is the mildest of the four. It earns its place because the waste is silent even after the connectivity problem is fixed some other way: an account can carry a dozen of these indefinitely, each one a line item with no function.
What the four have in common
The obvious check passes. In every case the first thing a competent engineer verifies is genuinely correct. The record is right. The isolation table is right. IPv6 is enabled. The endpoint is available.
The failure is in a relationship, not an object. Which source answers. Which table governs. Which family routes. Which name resolves where. There is no resource to inspect, so resource-by-resource review is blind to all of it.
The signal exists and nobody is reading it. Resolver query logging would have caught the first. Reachability Analyzer the second. A forced-IPv6 synthetic check the third. An endpoint inventory the fourth. None are expensive; all are things you only add if you have thought about this failure mode.
Asserting values instead of detecting errors
The alarm design question that catches this class is different from the usual one. "What would tell us if this broke?" gets built. "What would tell us if this were quietly wrong?" usually does not.
The answers tend to look like assertions rather than error detectors:
# Does this name resolve to what we think?
test "$(dig +short payments.corp.internal)" = "$EXPECTED_IP" || alert
# Does IPv6 actually work, not just exist?
curl -6 -s -m 5 -o /dev/null https://ipv6.google.com/ || alert
# Does the service hostname resolve inside the VPC?
getent hosts secretsmanager.us-east-1.amazonaws.com | grep -q '^10\.' || alert
# Which route tables can reach the prefix that is supposed to be isolated?
aws ec2 search-transit-gateway-routes --transit-gateway-route-table-id "$rtb" \
--filters "Name=state,Values=active" --query 'Routes[].DestinationCidrBlock'Each one encodes an expectation. That is the difference: a health check confirms something is responding, and an assertion confirms it is responding with the right thing.
Enumerate rather than test
The other habit worth adopting, particularly for the segmentation case: a test can prove a path exists and cannot prove no path exists, because that requires testing every pair. Configuration enumeration can.
Route tables, endpoint attributes, and DNS precedence are all readable in bulk. Reading them exhaustively and comparing against the design is a different activity from testing, and it is the only one that produces a negative result you can trust.
Each of the four has a lab if you would rather find them than read about them: Resolver rule precedence, the Transit Gateway default route table, the missing IPv6 route, and the endpoint that resolved to the internet. The first two are the ones that succeed most convincingly.