Writing
Blog
Findings from building the labs — quota mechanics, documentation that disagrees with itself, and the design decisions those details drive. Where the labs teach mechanics, this is where the sharp edges get written down.
The one network limit that does not grow when you scale up
Every instance can send 1,024 packets per second to the VPC resolver, per network interface, and the number is the same on a t3.nano and a u-24tb1.metal. Throttled queries never reach anything that logs, so the symptom is intermittent DNS failure with no evidence anywhere — and scaling up, the instinctive fix, changes nothing.
5 min readDNSQuotasYour 25 Gbps instance does 5 Gbps to the internet, and that is documented
EC2 instance bandwidth is one number on the spec sheet and at least three in practice. Traffic through an internet gateway is capped at 5 Gbps on anything under 32 vCPUs, and a single flow is capped at 5 Gbps nearly everywhere. Neither is a fault, and both get filed as one.
6 min readQuotasRoute TablesTen kinds of traffic VPC Flow Logs never record
A flow log with no REJECT lines is routinely read as proof that nothing was blocked. AWS documents ten categories of traffic that flow logs exclude by design, and several of them are precisely the traffic you would be looking for when something is wrong.
6 min readDocumentationSecurityA regional NAT Gateway makes the most common NAT mistake impossible to express
The zonal NAT Gateway has a placement rule, a per-AZ route table requirement, and a single-zone dependency, and each one is a documented outage. Regional mode does not fix those bugs. It removes the configuration they live in.
7 min readNAT GatewayRoute Tablesus-east-1a is not the same place in your account and theirs
Availability Zone names are mapped per account, deliberately and randomly, in every Region that existed before November 2012. A PrivateLink consumer who builds in us-east-1a can be refused an endpoint for a service that is, by the provider's naming, in us-east-1a. The identifier that is the same everywhere is the AZ ID, and almost nothing is written in it.
6 min readPrivateLinkDocumentationA custom DNS server in your DHCP options turns off private DNS for every endpoint in the VPC
Private hosted zones, interface endpoint private DNS, and Route 53 Resolver rules all live in one place: the VPC resolver at .2. Point the DHCP option set at your own DNS servers and every instance stops asking it. Nothing is deleted, nothing is disabled, and every one of those features quietly stops applying.
5 min readDNSPrivateLinkClient VPN connected, authenticated, and reaching nothing
A Client VPN endpoint has its own route table and its own authorization rules, and a destination has to be in both before a connected user can reach it. Add one without the other and the tunnel is up, the client has an address, and every packet goes nowhere — with a different silence depending on which one you forgot.
6 min readVPNSecurityThe Site-to-Site VPN tunnel that waits to be called
By default, AWS never initiates a Site-to-Site VPN tunnel. Your device has to. So a tunnel that only carries traffic AWS originates — monitoring, a scheduled job, a backup pull — comes up, idles, drops, and stays down until something on your side sends a packet. The fix is one setting that most people have never seen, and it requires IKEv2.
6 min readVPNBGPDirect Connect has a button that breaks it on purpose, and almost nobody presses it
StartBgpFailoverTest takes a virtual interface's BGP session down for a duration you choose, so you can watch traffic move to the backup path before an outage does it for you. It is the only honest way to know a hybrid failover works, it has a three-hour default most people do not expect, and it can be stopped at any time.
6 min readDirect ConnectBGPThe network failures that present as success
Most AWS networking incidents announce themselves with a timeout. A smaller set return a valid answer and do the wrong thing, and every diagnostic instinct that works on the first kind has nothing to grab on the second.
5 min readDNSSecurityDocumentationOn-premises cannot reach a NAT Gateway through a virtual private gateway
The same design works with a Transit Gateway and does not work with a VGW. It is one sentence in the documentation, and it decides whether centralised egress for on-premises traffic is possible at all.
6 min readNAT GatewayVPNPrivateLink traffic always arrives from the load balancer, whatever you configure
Client IP preservation has no effect on inbound PrivateLink traffic. If you turn it on and write client-IP security group rules, direct clients work and every PrivateLink consumer breaks — which is a difficult failure to see from the provider side.
7 min readPrivateLinkNLBYou can route around the local route, and that is how intra-VPC inspection works
"The local route always wins and cannot be overridden" is the version most people learn. It is wrong twice: a route more specific than the VPC CIDR wins by longest prefix match, and the local route's target can be replaced outright. The first is what makes inspecting traffic between two subnets possible.
5 min readRoute TablesSecurityBFD negotiates to the slower peer, so your failover is as fast as their config
You can set a 300 millisecond interval on your side and still get whatever the other end asked for. Intervals negotiate to the slower value and can be asymmetric, which means detection time is not a number you control alone.
5 min readBGPDirect ConnectA stale static route silently shadows the BGP route you are advertising
VPC route priority puts static routes above propagated ones at the same prefix. So a leftover static route wins over a healthy BGP advertisement, and the routing table looks correct because the route is there.
5 min readBGPRoute TablesThe NAT Gateway drops idle connections after 350 seconds and sends a RST
It is not a silent timeout. The connection is actively reset, which means your application sees a connection error at a moment when nothing was wrong, and connection pools are the most common casualty.
4 min readNAT GatewayQuotasA 200 millisecond penalty is what a completely broken IPv6 path looks like
Happy Eyeballs means a dead address family costs you latency instead of errors. The fixed delay on dual-stack destinations is a specific enough fingerprint to diagnose from, and it can sit unnoticed indefinitely.
7 min readIPv6Route TablesProxy protocol is sent on health checks, where it carries nothing useful
Enable it on an NLB target group and the header is prepended to health check connections too. On those connections it contains no client information at all, so the feature adds nothing there and still marks every target unhealthy.
5 min readNLBDocumentationA Resolver rule outranks your private hosted zone, and the zone is never consulted
Two DNS sources can claim the same domain in one VPC. The Resolver rule wins at equal specificity and the hosted zone is never read — and nothing in the answer reliably tells you which source served it, so the check has to be a configuration check.
6 min readDNSDocumentationYou cannot borrow a peered VPC's NAT Gateway, but the reverse works
Routing a peered VPC's egress through your NAT Gateway is unsupported edge-to-edge routing. Sending NAT Gateway traffic through a peering to a peered destination is supported, and AWS documents a Return to Sender behaviour there that surprises people.
5 min readVPC PeeringNAT GatewaySecurity group referencing across a Transit Gateway needs two flags, not one
It has to be enabled on the transit gateway and on every VPC attachment. Enable it in one place and it does nothing, with no error. The limitation list is also longer than most people check.
5 min readTransit GatewaySecurityAppliance mode has a prerequisite that silently undoes it
Enable appliance mode and forget route propagation on the associated Transit Gateway route table, and every flow falls back to hash-based Availability Zone selection. The attachment still reports appliance mode enabled.
8 min readTransit GatewaySecurityAn interface endpoint with private DNS off is a bill with no benefit
It costs the same per hour as one that works, reaches available, passes every reachability check, and nothing routes to it. The Terraform default is false, which is most of why this ships.
5 min readPrivateLinkDNSVPC peering has no reverse path forwarding, and AWS documents what that costs
A hub peered to two VPCs that share a CIDR will deliver one of them's replies to the other. AWS states the behaviour outright, and the configuration is legal to build because the overlap guardrail is narrower than people think.
5 min readVPC PeeringRoute TablesThe misconfiguration that does not appear in the API output
A subnet with no explicit route table association is returned by nothing. DescribeRouteTables says so in one line of its own reference, and it means "check the route tables" structurally cannot find the problem.
4 min readRoute TablesDocumentation"The ephemeral port range" means four different things
Network ACLs get written with 32768-65535 because that is the number everyone knows. It is nobody's actual range — it is AWS's example. A NAT Gateway uses 1024-65535, and the gap between the two is an intermittent failure that retries hide for weeks.
6 min readNAT GatewayNLBSecurityOne Transit Gateway rule explains both an outage and a security hole
Route lookup happens against the route table associated with the attachment traffic arrived on. That single sentence produces a connection that should work and does not, and a connection that should be impossible and is not.
5 min readTransit GatewayRoute TablesWhen the AWS docs contradict themselves: NLB client IP preservation
Two pages in the same Network Load Balancer user guide state opposite defaults for client IP preservation. The disagreement decides which security group rules are correct, and getting it wrong produces a target that passes health checks and refuses every client.
6 min readNLBDocumentationThree AWS networking limits that changed shape, not just value
A remembered quota is worse than no quota, because it feels like knowledge. Three AWS networking limits where the mechanism changed underneath the number, and the design decision each one now points at.
8 min readQuotasNAT GatewayDirect Connect