MTU, MSS and encapsulation overhead

It is tempting to treat MTU and MSS as two names for the same number. They are not, and the confusion causes a large share of tunnel performance problems. The MTU is a property of a link: the largest frame that link will carry. For standard Ethernet that is 1500 bytes, and it is why an oversized ping does not get through a badly configured network.

MSS is a property of a TCP connection: the largest payload either endpoint will put in a single segment. It is not a link setting and it is not negotiated from the MTU — each endpoint announces the MSS it can receive, and each subtracts the headers it knows about from its own MTU.

MSS = MTU − IP header − TCP header

IPv4: 1500 − 20 − 20 = 1460 bytes
IPv6: 1500 − 40 − 20 = 1440 bytes

The 1460-byte figure is so common it is worth memorising. If you see a value well below it on a working connection, something is adding headers — which is the whole subject of this page.

MTU is measured at the outermost layer, so every byte of header between the link frame and the application data reduces how much data fits. On a plain path that is 40 bytes of overhead out of 1500, about 2.7%. Add a tunnel and it climbs fast.

The last two rows are why IPsec and MPLS sites so often need a reduced MTU. An ESP tunnel with AES-CBC and a SHA-256 integrity check adds around 73 bytes on its own, before the outer IP header.

Take a GRE tunnel over the public internet, carrying traffic from a private site. The packet on the wire is the public Ethernet frame; the private packet is the payload inside it. Working from the outside in:

The same arithmetic with VXLAN shows why overlay networks are so often set to a 1450-byte inner MTU: Ethernet, IPv4, UDP, VXLAN, then the inner Ethernet, IPv4 and TCP headers total 14 + 20 + 8 + 8 + 14 + 20 + 20 = 104 bytes, leaving 1396 for the inner TCP payload at a 1500-byte outer MTU.

PMTUD is the mechanism that should make all of this automatic. A sender sets the Don't Fragment bit and sizes packets to what it believes the path supports. When a router finds the next hop's MTU is smaller, it is supposed to drop the packet and send back an ICMP message — “fragmentation needed”, with the smaller MTU. The sender reduces its size and retries.

The failure mode is that ICMP is frequently blocked by firewalls, either deliberately as a security measure or as collateral damage from rate limiting. When the message never arrives, the sender keeps using the oversized packet, every one is dropped, and the connection appears to hang. This is a PMTUD blackhole, and it is characterised by a very specific pattern:

The quickest diagnostic is to send an oversized ping with Don't Fragment set and binary-search the payload size until it gets through. On a tunnel that should give you the real path MTU, which you can compare against what the endpoints believe.

Where PMTUD cannot be relied on, MSS clamping is the usual fix. A router on the path rewrites the MSS field in the SYN and SYN-ACK packets, so both endpoints immediately use a value that fits without needing to discover anything. It is common on PPPoE links, where the 8-byte PPPoE header reduces the effective MTU to 1492 and MSS to 1452.

Clamping has limits worth knowing. It only affects connections whose handshake passes through the clamping device, so it does nothing for hosts on the far side of a router that is not in the path. It also does not touch a connection that is already established. And because it rewrites a field the endpoints agreed on, it can mask a genuine misconfiguration rather than fixing it.

The alternatives are to set the MSS explicitly on the endpoints, to reduce the interface MTU on the tunnel so the arithmetic works out naturally, or — for IPv6 — to respect that PMTUD is mandatory and make sure ICMPv6 type 2 messages are never filtered.

• Size MSS from the outermost MTU, not from 1500. Add up the headers you actually use. The MTU and MSS calculator will do the arithmetic per layer.
• Never block ICMP wholesale. If you must filter it, allow fragmentation-needed (type 3, code 4) inbound — blocking it breaks PMTUD for every path that terminates behind you. For IPv6, allow Packet Too Big (type 2).
• Remember that 1460 is only correct for a plain IPv4 Ethernet path. IPv6 gives 1440, PPPoE gives 1452, and anything with a tunnel in it is smaller still.
• Expect tunnels to cost more than the header arithmetic suggests. A GRE-over-IPsec stack can consume well over 100 bytes, taking a 1500-byte path down to an MSS near 1330.
• Check both directions. Asymmetric routing means the return path can have a different MTU, and MSS is announced per direction.