Solution to Packet Capture Challenge #5 - TCP Fast Open and Home Routers

October 04, 2012 • 4 min read

This challenge is now concluded! Read the solution below or scroll down for the original challenge!

The Solution

So, what’s going on here?

This communication is happening over a home gateway using Network Address Translation, or NAT. This is very common in home networks as it allows a Service Provider to use only one public address to represent many hosts. It also has an interesting side effect of acting as a natural firewall.

In our packet capture, we can see both the LAN and WAN connections of the gateway using pcap-ng’s multi-interface feature. In packet 2, the host at requests a TCP Fast Open cookie from the server at The home gateway doesn’t see anything wrong here and forwards the information just fine. The host receives the Fast Open cookie for later use.

The host then tries to use the cookie for TCP Fast Open at packet 14. However, to do this, it is also including 4 bytes of data in the TCP SYN. Normally this can only happen after the three-way-handshake completes, and this potential problem is clarified in section 2 of the TCP Fast Open IETF Draft:

` 2. Data In SYN

RFC793 (section 3.4) already allows data in SYN packets but forbids the receiver to deliver the data to the application until 3WHS is completed. This is because TCP’s initial handshake serves to capture

  • Old or duplicate SYNs
  • SYNs with spoofed IP addresses

TFO allows data to be delivered to the application before 3WHS is completed, thus opening itself to a possible data integrity problem caused by the dubious SYN packets above. `

The home gateway drops this SYN packet and prevents the TCP Fast Open connection from completing.

Thanks for trying Packet Capture Challenge #5! Keep with us for our next updates and don’t forget to follow us on twitter for more challenges and announcements!

The Challenge

We have so much going on here at CloudShark, we decided it’s time to throw you all another capture challenge!

As some of you may know, CloudShark is brought to by QA Cafe. Our flagship testing product, CDRouter, is the industry standard platform for testing consumer and SOHO grade broadband gateways (also known as “home gateways”, “home routers”), and other CPE (Customer Premises Equipment). This is actually how CloudShark was born - and so we have a lot of experience dealing with the nuances and problems of broadband networking.

Coming in CDRouter 7.2 will be some tests for TCP Fast Open. If you’ve never heard of TCP fast open, its a new technique to speed up the TCP connection establishment process between clients and servers when multiple connections will be made. Client-side support was just released in Linux 3.6.

Unfortunately, this technique does not always work with many of the CPE devices out there. Testing this requires that we observe both the WAN and LAN links of the CPE. Luckily, CloudShark 1.6 has pcapng support which lets us look at multiple interfaces in the same capture file!

So here is your challenge. The following pcapng trace contains both the LAN and WAN interfaces of a CPE device where the TCP Fast Open attempt fails. There are two TCP connections in the trace. The first one sets up a TCP fast open cookie. The second session attempts to use the TCP fast open cookie. The challenge is to determine why the second TCP session fails and suggest a reason why it fails. What could possibly go wrong? Why doesn’t this just work?

We placed a few annotations inside our pcapng trace to point you in the right direction.

Send your answers to

This is a tough one, so to make things interesting, we’re upping the stakes. The first five people who give us the full correct answer will get their very own CloudShark “P-CAP”, because we love puns:

If you don’t make it in time, don’t fret! We’ll still give you a CloudShark t-shirt:

Happy packet surfing!