How Window Scaling keeps TCP moving at top speed
Protocol choices made in older internet standards don’t scale to today’s network speeds. TCP (Transmission Control Protocol) has been making networks go for a very, very long time. As with many of the early internet protocols, limitations that seemed reasonable then can have a negative effect on performance now. Protocol designers allowed for future options to augment existing fields in order to keep them working effectively into the future. Let’s take a closer look at TCP Window Size and the Window Scaling option to see how they can impact your network.
Note: This CloudShark Collection contains the example here and all of the captures related to it in one easy to access page. Use CloudShark when you need to share multiple capture files alongside detailed explanations like this! Learn more…
What is the TCP window?
TCP uses a sliding window flow-control protocol to keep both sides of a connection sending and receiving data at rates that the opposite side can handle.
Chris Greer gave a great talk about this topic at SharkFest ‘18 where he compared the TCP Window to a bucket of water. We highly recommend you check out his video. He is a wonderful instructor!
Water through a drain pipe
Chris’s analogy describes water (data) flowing through a pipe (the network) and into a bucket on the receiver’s side (their receive window). The sender will only send as much water as the receiver can fit in their bucket at the other end, otherwise, it will spill out all over the floor! On the receiver’s side, the application layer is responsible for scooping water out of that bucket to keep water flowing. If the bucket fills up, the flow slows down or stops.
The Window value is sent in every TCP packet and contains a 16-bit field which represents how much additional data that side is willing to buffer for this connection. Back when TCP was first defined, it seemed reasonable that a field of this size would be enough. After all, the highest value that fits into a 2-byte field is 0xFFFF in hexadecimal, or 65,535. An endpoint with 65k of buffer space per connection was probably unheard of in 1981!
Take a look at this packet. Expand the “Transmission Control Protocol data” field and notice the Window size value is set to 29,200 bytes.
https://www.cloudshark.org/captures/ccbb6d25241f?filter=frame.number==1
This means that the receiver can accept an additional 29,200 bytes of data from the sender before its buffer fills up. 29k is not a lot of data, and today’s multi-gigabit network speeds can transfer that in microseconds. The sender then has to wait a full round-trip to get the receivers updated window size. If this can’t scale beyond sending 65k at a time, the capacity is going to be limited.
The TCP Window Scaling option provides the mechanism to increase that window scale value and push more data into the pipe at once.
How does TCP Window Scaling work?
Those working on TCP wanted to expand this ability. But since the field for window size was already 2 bytes, simply increasing the size of the field would have broken many existing implementations of TCP. To get around this they introduced a TCP “Window Scaling” option that can be used in the options field.
Take a look at this SYN packet. If you expand the Transmission Control Protocol data, you’ll see that the Window size value is still set to 29,200. However, if you expand the Options field, you’ll see the TCP Option - Window scale field included.
https://www.cloudshark.org/captures/e7b26c8d0f96?filter=frame.number==1
Here, it has a value of 7. The window scale option is defined as a “shift count” – meaning the window value is bit-shifted by an amount defined in the option. Another way to think about it is to multiply the value of the window size by 2^(window scale). The value 7 in this packet means the window scale should be multiplied by 2^7 (128). You can see these values broken out in CloudShark.
Window Scaling only works if both sides include it as an option during the initial 3-way handshake (even if their multipliers are different). The protocol recommends using the option even if the shift-count is 0 (meaning 2^0, which would multiply the window size by 1). That way, if one side doesn’t plan on using scaling, it doesn’t prevent the other side from taking advantage of the increased buffer space.
The maximum scale factor allowed by the standard is 14 - Which would allow us to have TCP windows of up to 2^30 bytes (a little over 1GB).
After the connection has been established and data flowing, the receiver can increase their window size to accept even more data. We can see this in packet #10 here. Notice that the Window Size value is only 251, but the Calculated Window Size is 32,128 because of the scaling option.
https://www.cloudshark.org/captures/e7b26c8d0f96?filter=frame.number==10
Jumping forward in time, towards the end of the transfer, we can see the receiver has drastically increased their window. The Window Size field is now 24,576 (much less than the max 65,535 that will fit in that field) but because of the window scale, the Calculated Window Size is 3,145,728 bytes - that means we can have almost 3MB in-flight!
https://www.cloudshark.org/captures/e7b26c8d0f96?filter=frame.number==57779
Note:
You won’t see the window scale option in every TCP packet. It is transmitted only during the 3-way TCP handshake. After that, it is remembered by each side of the connection. Without capturing that handshake, tools like CloudShark would not be able to determine the actual window size available.
TCP window scaling before and after
To see the effects of window scaling on a file transfer, let’s look at some examples with it disabled and enabled.
Scaling disabled
This is a capture of a TCP session with window scaling disabled. If we look at a graph of TCP data “in flight” during the first 500 packets, we can see the bytes-in-flight value approach the window size fairly often, sometimes getting close to its maximum value:
Since TCP must not send any more data than will fit in the receivers window, the sender will wait before transmitting additional data once the window size has been reached. This reduces the speed of the transfer since the sender is waiting for the receiver’s next ACK to arrive in order to know how much additional window the receiver has.
Scaling enabled
This is a capture of a TCP session with window scaling enabled. Again, when we look at a graph of the TCP data, we can see the bytes-in-flight approach the window size within the first second. With TCP scaling enabled, the receiver is able to advertise a larger window as more resources are allocated for this connection. We see the max-window size step up periodically during the transfer as these resources are allocated.
The light-blue Window Size quickly jumps up to and then above 500k within the first couple of seconds - which is then filled by the darker-blue sender. This graph actually shows a good representation of that water in the pipe and the available bucket it’s filling up. As the transfer continues, the receiver puts a bigger and bigger bucket to catch the water.
Thanks to the Window Scaling option, these higher receive buffer sizes are possible and the sender is then able to push more data into the wire to complete the transfer faster!
Conclusion
If we look at the conversation stats of both the first capture and second capture, we can see there’s a difference in overall bandwidth on our 100 Mbps link of almost 14 Mbps! As we expected, having Window Scaling enabled allowed us to transfer the data about 18% faster than without it.
| Duration | Data Rate Server -> Client | Frames Server -> Client | |
|---|---|---|---|
| Disabled | 10.628 sec | 74.5 Mbits/s | 70,734 |
| Enabled | 8.9982 sec | 87.9 Mbits/s | 68,509 |
That’s a pretty significant difference. With scaling disabled, the sender can fill the receive window almost immediately, and then has to wait for data to be ACKed so it can get more bytes in flight. However, with window scaling enabled, that window can grow beyond 65k as necessary, and there is plenty of room to send more and more data. Fewer frames can be sent as well, since the receiver can ACK multiple segments with a single packet.
The TCP window size is always adapting based on the resources available to the host and the particular TCP algorithm in use. The addition of window scaling let’s an endpoint go well beyond the 65k window size in the original TCP specification. This shows how flexible TCP really is, and that standards always need to be refreshed for uses that no one imagined in the beginning. Who knows - will the maximum window scale value of 14 be enough in the future?
When you’re diagnosing a slow network, Window Scale is one of the details to pay close attention to. Only the packets have the answer!
Additional Resources
- Kary from PacketBomb has a great video diving into more details on TCP Windows His videos are always very entertaining.
- Chris Greer gave a 2 part SharkFest talk on TCP. We linked to the first part above, but recommend taking the time to watch both:
- This is part of our ongoing series on TCP. Subscribe to our newsletter to be notified when we publish a new topic.
Want articles like this delivered right to your inbox?
No spam, just good networking
