I recently ran into an issue whereby traffic ceased passing over a link that had been in use for weeks and months without any issues. Post investigations and packet captures the root cause ended up being a mis-match in MTU size, but I thought I’d share my experience……
The Maximum Transmission Unit (MTU) is the largest number of bytes an individual datagram can have on a particular data communications link. When encapsulation, encryption or overlay network protocols are used the end-to-end effective MTU size is reduced. Some applications may not work well with the reduced MTU size and fail to perform Path MTU Discovery. In response, it would be nice to be able to increase the MTU size of the network links.
For most Ethernet networks the MTU size is, by default set to 1500 bytes, however today’s networks with numerous overlays and encapsulation on-top of encapsulation it can be difficult to determine what you should be setting the MTU size to throughout your network and across your WAN.
If we take the below image as a reference and pick SMTP as our example protocol. We have the corporate LAN on the left where our traffic will be sourced and the remote DC on the right where our Exchange servers reside.
The communication should flow as follows:
- Mail sent from Client to local Exchange server
- Exchange processes and determines next-hop, which will then likely be a mail gateway of sorts.
- Mail gateway performs it’s own interrogation, i.e. potentially blocks attachments/key words that are not permitted to leave the network, but if processed successfully traffic will will be forward on towards the egress point of the network. Additional upstream appliances could include IDPS, additional proxies etc.
- Traffic reaches egress firewall and provided there is an ACL in place to permit, the traffic is forwarded on to CE router and onto the WAN.
Let’s say the egress interface on the corporate firewall has an MTU of 1500 bytes, but the CE router interface has an MTU of 1400 bytes – what’s going to happen?
Well, in my recent experience I found SMTP traffic was leaving the firewall destined for the DC and a successful 3-way handshake performed and a connection established. However, data was leaving the firewall at 1500 bytes, being chopped up by the router and sent on it’s way. The reply from the remote DC device (using ICMP) was being dropped by the firewall (as why would you want your egress firewall to be pingable….)
Upon performing a packet capture from the corporate network I could see SMTP traffic leaving, but receiving TCP retransmit messages, but also messages stating:
Timeout waiting for client input
On first look I assumed this related to authentication, but after some expert googling it turned out to be an idiosyncrasy of Windows, which actually related to the network and the MTU size of the packets being transmitted.
There was a change performed on the CE router, which had reduced said MTU size, and although SMTP 3-way handshakes were still successful large amounts of drops were occuring and it ground mail to a standstill.
Tip: MTU size is always a great thing to check, but so are the approved changes that took place the time of the issue 😉
The below list is great to have when looking to determine your MTU size and how many bytes will be added to the frame:
GRE (IP Protocol 47) (RFC 2784) adds 24 bytes (20 byte IPv4 header, 4 byte GRE header)
6in4 encapsulation (IP Protocol 41, RFC 4213) adds 20 bytes
4in6 encapsulation (e.g. DS-Lite RFC 6333) adds 40 bytes
Any time you add another outer IPv4 header adds 20 bytes
IPsec encryption performed by the DMVPN adds 73 bytes for ESP-AES-256 and ESP-SHA-HMAC overhead (overhead depends on transport or tunnel mode and the encryption/authentication algorithm and HMAC)
MPLS adds 4 bytes for each label in the stack
IEEE 802.1Q tag adds 4 bytes (Q-in-Q would add 8 bytes)
VXLAN adds 50 bytes
OTV adds 42 bytes
LISP adds 36 bytes for IPv4 and 56 bytes for IPv6 encapsulation
NVGRE adds 42 bytes
STT adds 54 bytes
Source: Network World