AWS
38 posts tagged AWS.
The CloudFront prefix list proves a request came from CloudFront, not from your CloudFront
A security group rule on the managed prefix list admits every CloudFront origin-facing server on earth, including the ones serving someone else's distribution pointed at your origin. AWS's own guidance puts a secret header first and the prefix list as an optional extra, the list costs 55 of your 60 rules, and VPC origins replace both with a service-managed security group that is scoped to your distributions and nothing else.
Networking7 minStop and start an instance and one of its four addresses changes, which is the one in the firewall
Private IPv4, IPv6 and an Elastic IP all survive a stop and start. The auto-assigned public IPv4 does not: it is released on stop and a new one is assigned on start, and the old one can never be recovered. Associate an Elastic IP later and the public address is released then too. Every address costs the same $0.005 an hour, so the one that changes is the one with no reason to exist.
Networking6 minAn S3 endpoint policy that names only your buckets breaks dnf and docker pull
Amazon Linux package repositories and ECR image layers are S3 objects in buckets AWS owns. Lock a gateway endpoint policy down to your own buckets and both stop working, with an access-denied that names a bucket you have never heard of. The policy language has three more traps, and the default policy is the only one that cannot hit them.
Networking7 minDual-stack on a Network Load Balancer is a listener rule, not a checkbox
Flip an NLB to dualstack and IPv6 clients connect, but every one of them reaches an IPv4 target as the load balancer's own address, because translated traffic is never source-preserved. The fix is a listener rule that routes by address family to an IPv6 target group, and it brings four requirements the checkbox never mentioned, one of which makes a UDP service fail to build at all.
Networking7 minA DNS Firewall allowlist blocks the name you allowed, because the CNAME behind it was never on the list
Route 53 Resolver DNS Firewall inspects every name in a redirection chain by default. Allow app.example.com, and the query is still blocked when that record is a CNAME to a CDN hostname the list does not mention. The default fail mode is closed, the block response can be made to look like a missing record, and the fix is one setting whose two values mean opposite things.
Networking9 minA stale security group rule allows nothing, until the peering it was written for comes back
Delete a peering connection, a transit gateway, or a referenced security group and every rule that pointed at it goes stale. Stale rules are not removed; they sit in the group admitting no one. Peer the same two VPCs again for an unrelated reason and the rule is no longer stale. There are four ways a rule gets there, one command that finds them, and one command to run before you delete anything.
Networking8 minTransit Gateway charges whoever sends, so an inspection VPC pays for every gigabyte it did not originate
Data processing is billed per gigabyte sent to the Transit Gateway, to the owner of the attachment that sent it. Route spoke traffic through an inspection VPC and every gigabyte is sent twice: once by the spoke, once by the inspection VPC on the way back. The charge doubles and half of it lands on the central network account. Flexible Cost Allocation exists to move it, with conditions.
Networking8 minThe NAT Gateway bills the download too, and three other lines that never say NAT
Inbound data transfer from the internet is free. A gigabyte downloaded through a NAT Gateway still costs $0.045, because the processing charge applies regardless of the traffic's source or destination. That is the line people model. The Elastic IP, the zone boundary, and the data transfer out stack on top of it, and none of them is labelled NAT Gateway on the bill.
Networking7 minThe one network limit that does not grow when you scale up
Every instance can send 1,024 packets per second to the VPC resolver, per network interface, and the number is the same on a t3.nano and a u-24tb1.metal. Throttled queries never reach anything that logs, so the symptom is intermittent DNS failure with no evidence anywhere — and scaling up, the instinctive fix, changes nothing.
Networking5 minYour 25 Gbps instance does 5 Gbps to the internet, and that is documented
EC2 instance bandwidth is one number on the spec sheet and at least three in practice. Traffic through an internet gateway is capped at 5 Gbps on anything under 32 vCPUs, and a single flow is capped at 5 Gbps nearly everywhere. Neither is a fault, and both get filed as one.
Networking6 minTen kinds of traffic VPC Flow Logs never record
A flow log with no REJECT lines is routinely read as proof that nothing was blocked. AWS documents ten categories of traffic that flow logs exclude by design, and several of them are precisely the traffic you would be looking for when something is wrong.
Networking6 minA regional NAT Gateway makes the most common NAT mistake impossible to express
The zonal NAT Gateway has a placement rule, a per-AZ route table requirement, and a single-zone dependency, and each one is a documented outage. Regional mode does not fix those bugs. It removes the configuration they live in.
Networking7 minus-east-1a is not the same place in your account and theirs
Availability Zone names are mapped per account, deliberately and randomly, in every Region that existed before November 2012. A PrivateLink consumer who builds in us-east-1a can be refused an endpoint for a service that is, by the provider's naming, in us-east-1a. The identifier that is the same everywhere is the AZ ID, and almost nothing is written in it.
Networking6 minA custom DNS server in your DHCP options turns off private DNS for every endpoint in the VPC
Private hosted zones, interface endpoint private DNS, and Route 53 Resolver rules all live in one place: the VPC resolver at .2. Point the DHCP option set at your own DNS servers and every instance stops asking it. Nothing is deleted, nothing is disabled, and every one of those features quietly stops applying.
Networking5 minClient VPN connected, authenticated, and reaching nothing
A Client VPN endpoint has its own route table and its own authorization rules, and a destination has to be in both before a connected user can reach it. Add one without the other and the tunnel is up, the client has an address, and every packet goes nowhere — with a different silence depending on which one you forgot.
Networking6 minThe Site-to-Site VPN tunnel that waits to be called
By default, AWS never initiates a Site-to-Site VPN tunnel. Your device has to. So a tunnel that only carries traffic AWS originates — monitoring, a scheduled job, a backup pull — comes up, idles, drops, and stays down until something on your side sends a packet. The fix is one setting that most people have never seen, and it requires IKEv2.
Networking6 minDirect Connect has a button that breaks it on purpose, and almost nobody presses it
StartBgpFailoverTest takes a virtual interface's BGP session down for a duration you choose, so you can watch traffic move to the backup path before an outage does it for you. It is the only honest way to know a hybrid failover works, it has a three-hour default most people do not expect, and it can be stopped at any time.
Networking6 minThe network failures that present as success
Most AWS networking incidents announce themselves with a timeout. A smaller set return a valid answer and do the wrong thing, and every diagnostic instinct that works on the first kind has nothing to grab on the second.
Networking5 minSmall transfers work, large ones hang, and the ICMP that would explain it was never sent
Path MTU Discovery depends on the device that drops an oversized packet sending an ICMP back. On AWS, which devices do that is documented per attachment type — and a Site-to-Site VPN is not one of them. Jumbo-frame instances talking to on-premises over a VPN see the classic hang, and no security group rule in the world will produce the ICMP that was never generated.
Networking8 minOn-premises cannot reach a NAT Gateway through a virtual private gateway
The same design works with a Transit Gateway and does not work with a VGW. It is one sentence in the documentation, and it decides whether centralised egress for on-premises traffic is possible at all.
Networking6 minPrivateLink traffic always arrives from the load balancer, whatever you configure
Client IP preservation has no effect on inbound PrivateLink traffic. If you turn it on and write client-IP security group rules, direct clients work and every PrivateLink consumer breaks — which is a difficult failure to see from the provider side.
Networking7 minYou can route around the local route, and that is how intra-VPC inspection works
"The local route always wins and cannot be overridden" is the version most people learn. It is wrong twice: a route more specific than the VPC CIDR wins by longest prefix match, and the local route's target can be replaced outright. The first is what makes inspecting traffic between two subnets possible.
Networking5 minBFD negotiates to the slower peer, so your failover is as fast as their config
You can set a 300 millisecond interval on your side and still get whatever the other end asked for. Intervals negotiate to the slower value and can be asymmetric, which means detection time is not a number you control alone.
Networking5 minA stale static route silently shadows the BGP route you are advertising
VPC route priority puts static routes above propagated ones at the same prefix. So a leftover static route wins over a healthy BGP advertisement, and the routing table looks correct because the route is there.
Networking5 minThe NAT Gateway drops idle connections after 350 seconds and sends a RST
It is not a silent timeout. The connection is actively reset, which means your application sees a connection error at a moment when nothing was wrong, and connection pools are the most common casualty.
Networking4 minA 200 millisecond penalty is what a completely broken IPv6 path looks like
Happy Eyeballs means a dead address family costs you latency instead of errors. The fixed delay on dual-stack destinations is a specific enough fingerprint to diagnose from, and it can sit unnoticed indefinitely.
Networking7 minProxy protocol is sent on health checks, where it carries nothing useful
Enable it on an NLB target group and the header is prepended to health check connections too. On those connections it contains no client information at all, so the feature adds nothing there and still marks every target unhealthy.
Networking5 minA Resolver rule outranks your private hosted zone, and the zone is never consulted
Two DNS sources can claim the same domain in one VPC. The Resolver rule wins at equal specificity and the hosted zone is never read — and nothing in the answer reliably tells you which source served it, so the check has to be a configuration check.
Networking6 minYou cannot borrow a peered VPC's NAT Gateway, but the reverse works
Routing a peered VPC's egress through your NAT Gateway is unsupported edge-to-edge routing. Sending NAT Gateway traffic through a peering to a peered destination is supported, and AWS documents a Return to Sender behaviour there that surprises people.
Networking5 minSecurity group referencing across a Transit Gateway needs two flags, not one
It has to be enabled on the transit gateway and on every VPC attachment. Enable it in one place and it does nothing, with no error. The limitation list is also longer than most people check.
Networking5 minAppliance mode has a prerequisite that silently undoes it
Enable appliance mode and forget route propagation on the associated Transit Gateway route table, and every flow falls back to hash-based Availability Zone selection. The attachment still reports appliance mode enabled.
Networking8 minAn interface endpoint with private DNS off is a bill with no benefit
It costs the same per hour as one that works, reaches available, passes every reachability check, and nothing routes to it. The Terraform default is false, which is most of why this ships.
Networking5 minVPC peering has no reverse path forwarding, and AWS documents what that costs
A hub peered to two VPCs that share a CIDR will deliver one of them's replies to the other. AWS states the behaviour outright, and the configuration is legal to build because the overlap guardrail is narrower than people think.
Networking5 minThe misconfiguration that does not appear in the API output
A subnet with no explicit route table association is returned by nothing. DescribeRouteTables says so in one line of its own reference, and it means "check the route tables" structurally cannot find the problem.
Networking4 min"The ephemeral port range" means four different things
Network ACLs get written with 32768-65535 because that is the number everyone knows. It is nobody's actual range — it is AWS's example. A NAT Gateway uses 1024-65535, and the gap between the two is an intermittent failure that retries hide for weeks.
Networking6 minOne Transit Gateway rule explains both an outage and a security hole
Route lookup happens against the route table associated with the attachment traffic arrived on. That single sentence produces a connection that should work and does not, and a connection that should be impossible and is not.
Networking5 minWhen the AWS docs contradict themselves: NLB client IP preservation
Two pages in the same Network Load Balancer user guide state opposite defaults for client IP preservation. The disagreement decides which security group rules are correct, and getting it wrong produces a target that passes health checks and refuses every client.
Networking6 minThree AWS networking limits that changed shape, not just value
A remembered quota is worse than no quota, because it feels like knowledge. Three AWS networking limits where the mechanism changed underneath the number, and the design decision each one now points at.
Networking8 min