TCP throughput and the BDP

A 1 Gbps link between two offices 100 ms apart should move a file in a fraction of a second. In practice a single TCP connection often reaches a tenth of that. The link is not the problem — the protocol is waiting. TCP requires data to be acknowledged before it can be replaced in flight, and over a long path each acknowledgement takes a full round trip to arrive. If the sender runs out of permitted unacknowledged data before the first acknowledgement returns, the link sits idle for part of every RTT.

The quantity that determines whether this happens is the bandwidth-delay product: how much data has to be in flight at once to keep the pipe full. It is just bandwidth multiplied by round-trip time.

BDP (bits)  = bandwidth (bits/s) × RTT (s)
BDP (bytes) = BDP (bits) ÷ 8

1 Gbps, 100 ms RTT:
  BDP = 1,000,000,000 × 0.1      = 100,000,000 bits
      = 100,000,000 ÷ 8          = 12,500,000 bytes
      = 12.5 MB

So this link needs 12.5 MB of data outstanding at all times to stay busy. That number is the target that every other tuning decision is trying to reach.

A TCP sender cannot have more unacknowledged data outstanding than the receiver's advertised window allows. That gives an upper bound on throughput that has nothing to do with the link rate:

Throughput (B/s) = window (bytes) ÷ RTT (s)

65,535-byte window, 100 ms RTT:
  Throughput = 65,535 ÷ 0.1 = 655,350 B/s
             = 655,350 × 8  = 5,242,800 bps ≈ 5.24 Mbps

This is the single most important number in TCP performance. A stock 65,535-byte window on a 100 ms path caps a single flow at about 5.24 Mbps — on a link of any speed. The same window over a 10 ms RTT reaches 52.4 Mbps, which is why the problem is invisible on a LAN and severe across an ocean.

RFC 7323 (originally RFC 1323) adds the window scale option. During the handshake each side advertises a shift count, and every window value sent afterwards is interpreted as value × 2^scale. A scale factor of 7 multiplies the ceiling by 128:

Because the shift count is negotiated, both endpoints must support it — a modern Linux or macOS host talking to an old appliance may still get an unscaled window. Note that the scale is applied to the window field but not to the value in a window update during the handshake itself, which is a classic source of off-by-a-factor-of-128 confusion when reading captures.

Even with a large window, loss imposes its own limit. Congestion control treats a dropped packet as evidence of congestion and reduces the sending rate; recovery is gradual. The Mathis equation gives an order-of-magnitude ceiling for a single Reno-style AIMD flow:

Throughput ≈ (MSS ÷ RTT) × (C ÷ √p)

  MSS = maximum segment size in bytes
  RTT = round-trip time in seconds
  p   = packet loss probability
  C   ≈ 1.22 for a simple AIMD (Reno) sender

MSS 1460 B, RTT 100 ms, p = 1%:
  1460 ÷ 0.1        = 14,600
  1.22 ÷ √0.01      = 1.22 ÷ 0.1 = 12.2
  Throughput        = 14,600 × 12.2 = 178,120 B/s
                    ≈ 1.42 Mbps

The loss term sits under a square root, which makes it unusually punishing. Going from 0.01% loss to 1% loss — a hundredfold increase — costs a factor of ten in throughput, not a hundred. It also means that a path with a small amount of persistent loss has a hard ceiling that no amount of window tuning can raise.

When a transfer is slower than expected, the useful question is which of the three limits is binding. Each has a different fix, and applying the wrong one wastes time.

The quick sanity check: multiply the observed throughput by the RTT. If the result is suspiciously close to 64 KB, or to a round number of megabytes, the window is the limit. If it is close to the BDP but well below the link rate, look at loss.

The figures above are upper bounds for a single idealised flow. Real transfers are also limited by TCP slow start — a connection begins with a small congestion window and doubles it each RTT until it reaches the path capacity, so short transfers never get near the ceiling. Delayed acknowledgements, receive-window autotuning, competing traffic, queuing delay that increases the effective RTT under load, and CUBIC's different response to loss all move real behaviour away from the model.

Use these numbers to work out which limit applies and how much headroom tuning could buy, not to predict a specific transfer's duration. The bandwidth calculator handles the more basic question of how long a file takes at a given rate, and the MTU and MSS calculator covers the encapsulation overhead that reduces the MSS these formulas depend on.