Watch our in-depth webinar on troubleshooting AWS environments with CloudShark
Packet capture in private cloud environments has been a holy grail for IT since virtualized environments became the norm. Whether troubleshooting application performance, detecting security anomalies, or bringing a cloud deployment into compliance with the rest of your infrastructure, you will need to get to the packets to fully resolve and report on issues. CloudShark is, of course, a natural fit for analyzing captures in a cloud environment - but how do you gather captures in such an environment in the first place?
AWS VPC Traffic Mirroring
Thankfully, Amazon has announced VPC traffic mirroring in its AWS service. This allows AWS users to create a virtual interface that mirrors the network traffic of another AWS network resource’s interface. This powerful feature provides a perfect “capture point”, like a network tap, to source packet capture data and send the capture files to your own CS Enterprise instance running in AWS, where your entire team can collaborate on the resolution of network, application, and security issues.
From the article:
You can think of VPC Traffic Mirroring as a “virtual fiber tap” that gives you direct access to the network packets flowing through your VPC. As you will soon see, you can choose to capture all traffic or you can use filters to capture the packets that are of particular interest to you, with an option to limit the number of bytes captured per packet. You can use VPC Traffic Mirroring in a multi-account AWS environment, capturing traffic from VPCs spread across many AWS accounts and then routing it to a central VPC for inspection.
Capture workflow with traffic mirroring
VPC traffic mirroring works by selecting a network resource’s (such as an EC2) network interface and an elastic network interface or load balancer on another EC2 instance.
As an example, we set this up using these instructions in AWS and mirrored the traffic from one EC2 instance connected to the internet to an ENI connected to another capturing EC2 instance running Amazon Linux.
Here’s a diagram of our setup:
Dealing with VXLAN and other network traffic
The traffic being sent to our mirror destination is encapsulating using a VXLAN.
To properly capture the traffic we can create a VXLAN interface on the destination of our mirror.
sudo ip link add capture0 type vxlan id 4058602 local 10.0.0.83 remote 10.0.0.84 dev eth1 dstport 4789 sudo ip link set capture0 up
Next we will disable ipv6 on this interface so that it doesn’t transmit any IPv6 packets:
sudo sysctl net.ipv6.conf.capture0.disable_ipv6=1
Capturing the data
tcpdump is a standard, fast way to perform packet captures on a system, though one of many tools that can be used. Since tcpdump is installed by default on Amazon Linux, it’s great for capturing what was being sent within the AWS cloud to the mirror destination. Once we set up the proper filters for VXLAN and IPv6 (above), we can capture on our new capture0 interface using the command:
sudo tcpdump -i capture0 -w aws_capture0.pcap
Uploading to CloudShark
Once the capture is complete, it can be uploaded to CloudShark using our upload API. This can be done in a command line using cURL, but Tom has a script (cloudshark_upload.sh) that can be used with a CloudShark API key to upload captures to CloudShark:
https://gist.github.com/thomasp11/fa7b6d1ef80bfbb0d4dcb28e9bce94af
This script uses cURL and makes it easy to work with a CS Enterprise instance.
The wild cloud west
As an experiment, we started a capture on an EC2 interface exposed to the Internet and let this run for about 2 hours. Even in that short of an amount of time we had quite a few attempts to connect from all over the world! A lot of IP addresses were also flagged due to their reputation in Threat Assessment. Here is a capture of what we found:
https://www.cloudshark.org/captures/747dc632ad41
You can see the threat assessment analysis of the events here:
https://www.cloudshark.org/analysis/747dc632ad41/threats_vectors
That’s a lot of activity that we might have only noticed by looking at the packets!
CloudShark Enterprise on AWS
To create a seamless packet workflow with captures generated through VPC traffic mirroring, CloudShark Enterprise can be installed on certain AWS instances. You can read how to do it here to learn more.
To start a trial of CS Enterprise and try this for your own cloud deployment, request an evaluation and get started!
Want articles like this delivered right to your inbox?
No spam, just good networking
