Small transfers work, large ones hang, and the ICMP that would explain it was never sent

The shape is always the same. An SSH session works. A small file copies. curl against the API answers. A larger file, a database dump, a page with a big response body — hangs. Not slowly: completely, with a TCP connection that is established and a transfer that never finishes.

That is the signature of a path MTU problem, and the usual explanation is that something is blocking ICMP. On AWS the answer is more specific: sometimes nothing is blocking it, because nothing sent it.

How PMTUD is supposed to work#

A host sends a packet larger than some link along the path can carry, with the Don't Fragment bit set — which modern TCP always does. The device that cannot forward it drops it and sends back an ICMP message naming the size it can carry. The host shrinks and retries. From the EC2 MTU documentation:

For IPv4, when a host sends a packet that's larger than the MTU of the receiving host or that's larger than the MTU of a device along the path, the receiving host or device drops the packet, and then returns the following ICMP message: Destination Unreachable: Fragmentation Needed and Don't Fragment was Set (Type 3, Code 4).

Two things have to happen: the dropping device has to send the ICMP, and the sending host has to receive it. Most PMTUD troubleshooting assumes the first and looks for a block on the second. On AWS, the first is not guaranteed.

Where the boundaries are#

Every current-generation instance supports a 9001-byte MTU, and inside a VPC that is what you get. The boundaries where it drops are enumerated:

MTU limit 1500 bytes: Traffic over an internet gateway. Traffic over VPN connections. Traffic between AWS Regions, unless a transit gateway is used. MTU limit 8500 bytes: Traffic over an inter-region VPC peering connection.

And for a Site-to-Site VPN specifically, the number is lower than 1500. From the VPN quotas:

Site-to-Site VPN supports a maximum transmission unit (MTU) of 1446 bytes and a corresponding maximum segment size (MSS) of 1406 bytes. However, certain algorithms that use larger TCP headers can effectively reduce that maximum value.

1446, because IPsec overhead eats the rest of 1500, and less than that with some cipher suites. A jumbo-frame instance sending a 9001-byte packet toward on-premises over a VPN is 7,555 bytes over.

Which devices send the ICMP#

This is the part most explanations miss. The Transit Gateway quotas page states it per attachment type:

Transit gateways support Path MTU Discovery (PMTUD) for traffic ingressing on VPC and Connect attachments. Transit gateway generates the FRAG_NEEDED for ICMPv4 packets and Packet Too Big (PTB) for ICMPv6 packets. Transit gateways does not support PMTUD on Site-to-site VPN, Direct Connect, and Peering attachments.

And the VPN quotas page, in one sentence:

A Site-to-Site VPN connection does not support Path MTU Discovery.

So: a packet from a VPC that is too large for the Transit Gateway to forward out a VPN attachment — does the Transit Gateway send ICMP back? The traffic ingressed on a VPC attachment, so by the first quote, yes. But the VPN itself has no PMTUD, so a packet that makes it into the tunnel and is too large for the far side gets no ICMP from anywhere. And a VGW-terminated VPN, with no Transit Gateway in the path, has no PMTUD at all.

In the no rows, the oversized packet is dropped and nothing comes back. The sending host's TCP stack sees loss, retransmits the same oversized segment, sees loss again, and backs off. Small segments — the ones under the real path MTU — go through fine, because they were never too large. That is exactly the "small works, large hangs" shape, and it will persist for as long as the connection does.

Why the security group is not the answer#

The standard advice — and it is correct where it applies — is from the network ACL and PMTUD page:

If the maximum transmission unit (MTU) between hosts in your subnets is different, or your instances communicate with peers over the internet, you must add the following network ACL rule, both inbound and outbound. […] Select Custom ICMP Rule for the type and Destination Unreachable, fragmentation required, and DF flag set for the port range (type 3, code 4).

And the EC2 page adds the security group condition: ICMP responses arrive "if connections are automatically tracked, are untracked, or if your security group rules allow inbound ICMP traffic."

Both of those are about receiving the ICMP. They matter — a stateful security group with no inbound ICMP rule will drop the Frag-Needed reply for an untracked connection, and that is a real and common failure. But they fix nothing on a path where the ICMP was never generated. An engineer who adds the Type 3 Code 4 rule, confirms it is in place, and still sees the hang has not misconfigured anything. They are on a path with no PMTUD, and the fix is on the other side of the equation.

The actual fixes#

Since the network will not tell the host to shrink, the host has to be told some other way. Three options, all documented.

Clamp MSS on the customer gateway. The device terminating the VPN rewrites the TCP MSS option in every SYN that crosses it, so both ends agree on a segment size that fits before any data is sent. AWS's customer gateway best practices publish the values per cipher suite, and the spread is the reason to look them up rather than assume 1446:

Forty bytes of difference between the best and worst case, and a tunnel behind a NAT device with SHA-512 is forty bytes smaller than the headline number. This is the fix that works for every host behind the tunnel without touching any of them, and it is the one to reach for first. The Transit Gateway, for its part, "enforces Maximum Segment Size (MSS) clamping for all packets" — but that governs what it forwards, not what a VGW-terminated tunnel sees.

Set the instance MTU for the route. The EC2 page describes the approach: "you can configure the MTU size by route, or use multiple elastic network interfaces with different MTU sizes and different routes." An instance keeps 9001 for in-VPC traffic and uses 1446, or lower, toward the on-premises prefix:

ip route change 10.50.0.0/16 via 10.0.1.1 dev ens5 mtu 1400

Narrow, and correct, and has to be done on every instance that talks to on-premises.

Enable TCP MTU probing on the host. The AWS VPN performance post points at RFC 4821 packetization-layer PMTUD, which finds the path MTU by probing rather than waiting for ICMP:

sysctl -w net.ipv4.tcp_mtu_probing=1

This makes the host robust to any path that eats ICMP, which is a broader fix than the problem — and the same post says to test its performance impact before using it in production.

What does not work is the thing people try first: lowering the instance MTU to 1500 globally. That still exceeds 1446, so it moves the hang from "9001-byte packets" to "1500-byte packets" and changes nothing visible.

Reproducing it deliberately#

The diagnostic the AWS post recommends, with the flags that make it meaningful:

When using the ping command, make sure to use flags for Don't Fragment (Windows -f | Linux/Mac -D) and Send Buffer Size (Windows -l <bytes> | Linux/Mac -s <bytes>).

# Assumes AES-GCM-16 with NAT-T off, MTU 1446. Adjust from the table above.
# 1418 payload + 20 IP + 8 ICMP = 1446. Should pass.
ping -M do -s 1418 -c 3 10.50.0.10
 
# 1419 + 28 = 1447. Should fail.
ping -M do -s 1419 -c 3 10.50.0.10

On a path with PMTUD, the second ping returns Frag needed and DF set (mtu = 1446) — the ICMP, in plain text, naming the number. On a VPN path it returns nothing: 100% packet loss, no message, because the message does not exist. The absence of the Frag needed line, on a ping sized just over the boundary, is the diagnosis.

What to do with it#

Know which of your paths have no PMTUD. Anything crossing a Site-to-Site VPN, a Direct Connect attachment, or a Transit Gateway peering attachment is on the list. For each one, the fix has to be on the host or the customer gateway, because the network will not negotiate.

Clamp MSS at the VPN device, as a matter of course. It is one configuration line on the customer gateway, it covers every host behind it, and the value is documented per cipher. A VPN without it is a VPN waiting for the first large transfer.

Keep the ICMP rules too. Paths that do generate Frag-Needed — in-VPC, through a Transit Gateway from a VPC attachment, across the internet — still need the Type 3 Code 4 allow on network ACLs and, for untracked connections, on security groups. The two fixes address different halves of the same mechanism.

When the hang appears, ping with -M do at the boundary. Thirty seconds, no access to the far side needed, and it distinguishes "ICMP blocked" from "ICMP never sent" by whether a Frag needed line appears at all. The two look identical from the application and need opposite fixes.

The through-line is the one that runs through the flow log exclusions and the DNS throttle that logs nowhere: a mechanism that depends on a signal, on a path that is documented not to produce the signal. The documentation is clear. It is just not on the page the symptom points at.