Packet capture on VMware virtual machines using vmnet-sniffer

One of the most powerful tools we use when testing CloudShark is a combination of VMware Workstation and the Vagrant API interface. With Vagrant, we can test every permutation of CloudShark via a barrage of automated testing.

Along the way, we had to learn some of the lower level interfaces of these tools. We became aware of a vmnet-sniffer command that comes with VMware Workstation and VMware Fusion, which we use on our OS X workstations for development, and realized that it’s a great tool for capturing on virtual machines or in a cloud environment when used with CloudShark for analysis.

Why use packet capture in a virtual environment?

You have this VMware environment. It has virtual machines. Maybe just one, but probably more than one, and maybe way too many. And every day you’re logging into random virtual machines, trying to keep your SSH keys all in order, or worse, deploying LDAP client configurations just so you can log in for five minutes. Maybe you’re doing support services on an in-house thin client cluster, or you’re debugging some webserver on some regional node. Hunting down where a VM is hiding is not always quick and easy without some copious planning.

Why go through all this trouble? You already have a tool to skip all this: vmnet-sniffer. Log into your VMware host system, capture the traffic to all your virtual machines at once, and then sort it out later in CloudShark.

How vmnet-sniffer Works

This vmnet-sniffer doesn’t have much documentation available. Fortunately there aren’t too many options, so it’s a quick explanation. On OS X with VMware Fusion:

sudo /Applications/VMware\ Fusion.app/Contents/Library/vmnet-sniffer -e -w my_capture.pcap vmnet8

On Linux with VMware Workstation:

sudo /usr/bin/vmnet-sniffer -e -w my_capture.pcap /dev/vmnet8

With either of these commands, you just press control-c when you want to stop capturing.

You might be on vmnet1 if you are using host-only networks. Just give both a shot if you aren’t sure.

Note how the Linux command requires you to specify the full device path, but OS X does not. In both commands, the argument order is imperative! Otherwise you will probably get a nice text report but no pcap file.

Uploading to CloudShark

Once you have generated the my_capture.pcap file, you can upload it to a CloudShark system using whatever method is the most convenient. The API Upload is the most versatile, since you can integrate it with any custom tool you already have.

Using the upload API in a bash script

You can also put your CloudShark API token into your .bash_profile and upload the file with a little bash function. Put this into your ~/.bash_profile file and then open a new terminal (enterprise users should replace cloudshark.org with their own host name):

CS_TOKEN=8d9e1eace0dbb71e8fdebc0d15810272
CLOUDSHARK_ROOT=https://www.cloudshark.org
CLOUDSHARK_UPLOAD=$CLOUDSHARK_ROOT/api/v1/$CS_TOKEN/upload

function cloudshark_upload () {
  JSONID=$(curl -s -F "file=@$1" -F "additional_tags=upload_by_$USER" $CLOUDSHARK_UPLOAD | python -mjson.tool | grep id)
  if [ "$JSONID" != "" ]; then
    ID=$(echo $JSONID | sed 's/:/ /1' | awk -F" " '{ print $2 }'| sed 's/\"//g')
    echo "$CLOUDSHARK_ROOT/captures/$ID"
  else
    echo "Could not upload capture to CloudShark"
  fi
}

Now every time you load a terminal, you’ll have the cloudshark_upload command available:

# sudo /usr/bin/vmnet-sniffer -e -w my_capture.pcap /dev/vmnet8
# cloudshark_upload my_capture.pcap
Command returns: https://your-cloudshark-hostname/captures/44562ef7bb73

Using auto-import

CloudShark Entprise users can make this very easy by exporting the file to an NFS directory configuring CloudShark’s Auto Importer for it. Let’s pretend you have /captures mounted on your system, which is configured as an Auto Import directory already. This command will put the capture directly into the auto-import directory when it is completed:

sudo /usr/bin/vmnet-sniffer -e -w /captures/my_capture.pcap /dev/vmnet8

The capture will then appear in your CloudShark capture list.

These are some fun ways to get your pcap file imported. You can always just drag it from your Desktop onto the CloudShark webpage, as well.

How to Use

It Since your traffic on the vmnet8 or vmnet1 interface is relayed via a private address of your host system, you’re going to see a lot of traffic to and from this address. For example, on our settings, we have 172.16.254.1 set as the NAT gateway for our virtual machines.

For a bridged VMware environment, you don’t need vmnet-sniffer. You can just use tcpdump or your regular sniffer of choice, on the regular eth0/eth1/etc interface.

You’ll also be able to see all of the traffic the virtual machines are sending to each other, and this is where the vmnet-sniffer tool comes in really handy. You have a total scope of this private network from the top down. It’s straight-forward to apply CloudShark filters to zoom in on a specific area of interest. Some examples:

Isolating HTTP traffic that is being sent through a host level load balancer to a random virtual machine
Determining where a specific DNS query is getting picked up by some fault-tolerant DNS VM
Developing a new low level network management packet protocol without causing danger to your corporate network

There are plenty of scenarios we can invent where this environment is particularly useful. It’s an exercise to you, the VMware user, to come up with yet another brilliant network model best solved by virtual network isolation. If you have your own, email us! We’re geeks at heart and always love a great networking anecdote to share.

Want articles like this delivered right to your inbox?