Introduction
A new field in CloudShark called "TCP Completeness" calculates which packets of a TCP connection are seen in a PCAP file. This field allows you to determine if the TCP 3-way handshake has been captured while viewing any packet in the connection. This article will explain why it is essential to know if these packets have been captured, how the value for this field is calculated, and how you can use it.
A TCP connection between a client and a server is considered complete after it has been established and terminated. During the 3-way handshake, the client and server establish that both endpoints are ready to communicate. Multiple options, such as the maximum segment size and a scaling factor for the window size field used in subsequent packets in the connection, are also negotiated. Knowing what options have been negotiated and their values is vital when analyzing a TCP connection.
Establishing a connection is performed by completing a 3-way handshake. The handshake begins with a packet sent from the client to the server with the SYN flag set. If the server is willing to establish a TCP connection, it replies with a packet with the SYN and ACK flag set. The client will then send a packet with the ACK flag set, completing the handshake.
The connection is terminated by the client and server sending a packet with the FIN flag set. A packet with the RST flag set sent in either direction will also terminate the connection. This ladder diagram shows a complete TCP connection.
Clicking on any packet and expanding the TCP layer to inspect each field shows us the tcp.completeness field displayed as [Conversation completeness: Complete, WITH_DATA (31)].
How it is calculated
This field is enclosed in square brackets, which means the value of this field is calculated and never actually sent on the wire. It is computed by examining all of the packets in the TCP connection.
The value of the tcp.completeness bitmap field is calculated by examining each packet in the connection and creating a binary bitmap for each packet seen with the TCP flags set below:
Note that there is no TCP flag set in the packet for DATA, this bitmap field instead indicates that there was a TCP packet containing data.. Using a bitmap for the tcp.completeness field results in a unique value for any combination of packets seen with the flags above.
Looking at our example of a complete TCP connection, we would have the following bitmap for the value of tcp.completeness:
Adding up all of the values in this bitmap gives us 31 which is displayed as Complete, WITH_DATA (31). The first part, Complete, indicates all of the packets in the 3-way handshake, along with a packet with either the FIN or RST flag set. WITH_DATA means that data was also observed being sent over this connection. Finally, the value (31) was assigned to the tcp.completeness field.
If this connection were terminated with a TCP packet with the RST flag, instead we would have the following bitmap:
Adding all of these values give us 47 which is displayed as Complete, WITH_DATA (47).
Bitwise AND Filter
Because tcp.completeness is a bitmap, this allows us to use the bitwise AND operation to test if one or more of these bits is true. For example, the value 8, creates a bitmask for any value of tcp.completeness with the DATA flag set to true indicating that data was sent over the TCP connection. This can be calculated by setting the bit to true for the flags you want to test, and calculating the decimal value of the binary number:
Once we have calculated the bitmask it can be used with a filter using the bitwise AND operator. Now, to filter for any TCP packets that have had data sent over the connection the filter tcp.completeness & 8 can be used. This example shows this filter used to find TCP connections with different completeness values that all have data sent. The included TCP Completeness profile contains a similar saved filter used to find any connections with the RST flag sent using tcp.completeness & 32.
When using the bitmask AND operator, it is important to remember that setting multiple bits in the bitmask to true will match packets with any of these same bits in the field to set true. Take the following bitmask for example:
Next, examine the packets displayed in this capture when the filter tcp.completeness & 7 is used. In addition to displaying packets with all of these bits set to true, it also displays packets with any of the SYN, SYN/ACK, and ACK bits in the tcp.completeness field set to true. You can observe this in TCP stream 8 of this capture, where the only packet captured in the TCP connection was the initial TCP SYN packet.
When to use it
The tcp.completeness field is in every packet of the TCP connection. This makes it immediately obvious when looking at any packet in the connection what the overall state is of the connection based on the packets that have been captured.
The initial 3-way handshake is often not captured. This can happen when the capture is started in the middle of a conversation or if packets were dropped by the device doing the capturing. In addition to establishing that both endpoints are ready to transmit data, the initial 3-way handshake contains negotiated options.
The only time the values for these options are sent on the wire is during the handshake. Knowing the value of the Window scale option is necessary to know what values are used for the window size in later packets. This field gives you a quick way to know if the handshake and these options are available for the TCP connection any packet belongs to.
TCP Fast Open may also send data in the initial 3-way handshake. Without the handshake, data may be missing in the reassembled TCP stream.
When it is wrong
Complete?
The tcp.completeness field does not always mean that all of the packets in the 3-way handshake or the connection termination are in the pcap. This ladder diagram shows an example of where this calculation can be incorrect.
In this example, there are only 3 packets, yet the tcp.completeness field indicates 'Complete, NO_DATA (23)'. No data was transmitted over this connection, but how can it be 'Compete' when there are only 3 packets?
The trouble comes from how the ACK and FIN bits of the tcp.completeness field are handled. In this capture, we see a packet from the client with the SYN flag set (1), followed by the server sending its response with the SYN and ACK flags set (2). A termination packet from the server follows with the FIN and ACK flags set. This enables the ACK and FIN bits of the bitmap:
Adding up all these values we get 1 + 2 + 4 + 16 = 21 which translates to Complete, NO_DATA (23).
Half-open
Here is a connection that is still "half-open".
This connection is incomplete and has not been terminated. While the client is done transmitting data and has sent a packet with the FIN flag set, the server has not. The server can still send data over this TCP connection, and the client should process this data.
Interesting Values
35 & 39
A TCP completeness bitmap value of 35 indicates the following:
Value 39 indicates something similar with the addition of a packet with the ACK flag set.
These values indicate that a connection was established, meaning that both endpoints are ready to send data, then one endpoint terminates the connection with a TCP Reset packet. That's like knocking on a door, waiting for whoever is home to open it, then grabbing the door and slamming it. Not something a well-behaved person would do! So why would you see this behavior on a computer network?
Portscanning software behaves exactly this way! A TCP port scan will attempt to start a TCP connection on various ports to see if there is software listening for clients to connect. Often these are done to scan for vulnerabilities by attackers, or defenders testing their own network. Behavior like this can be searched for using the filter tcp.completeness in {35,39}.
Conclusion
The tcp.completeness field makes it quick and easy for an analyst to determine if the options in the TCP handshake are present in the capture. Without the handshake, it is often difficult or even impossible to analyze whether either endpoint is behaving correctly. While the value may not tell the analyst the whole story, it is a great way to determine if you're ready to start diving into an analysis or if you need to reexamine how you're capturing traffic first!
