An Introduction to Monitoring Encrypted Network Traffic with "Joy" Philip Perricone (SE) Bill Hudson (TL) Blake Anderson (TL) David McGrew (Fellow)
Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot#
Agenda Obtaining Joy Running Joy Inspecting Traffic with Sleuth Learning Classifiers Conclusions
Obtaining Joy
Dependencies sudo apt-get install build-essential libssl-dev libpcap-dev libcurl4-openssl-dev sudo apt-get install python-dev python-numpy pythonsetuptools python-scipy sudo easy_install -U scikit-learn 6
Download, Build, Run, Install Download git clone https://github.com/cisco/joy Build cd joy./config make Run./bin/joy [options] (Optional) Install make install 7
Running Joy
Components joy Offline pcap joy json sleuth Online joy json model.py exporter collector 9
Processing Pcaps Run the executable: Joy:~/joy$ bin/joy bidir=1 http=1 dns=1 tls=1 dist=1 \../benign/capture.pcap > capture.gz View the compressed JSON output: Joy:~/joy$ zless capture.gz 10
Inspecting Traffic with Sleuth
Sleuth Overview Sleuth is a tool designed for users to make SQL-like queries on JSON data. It reads a stream of flow (json) objects, processes each one in sequence, and writes out the resulting objects. The python script named sleuth located within the Joy repository provides many useful commands. On the remaining slides of this section, short descriptions of the commands and simple examples are presented. More technical, in-depth guides can be found in the wiki: https://github.com/cisco/joy/wiki 12
Sleuth Basic Options No options: Joy:~/joy$./sleuth capture.gz less --stitch Joy:~/joy$./sleuth capture.gz --stitch less --select <fields> Joy:~/joy$./sleuth capture.gz --stitch --select "sa, da, dp" --where <condition> Joy:~/joy$./sleuth capture.gz --stitch --select "sa, da, dp" --where "pr=17" 13
Sleuth --dist <fields> Calculate the distribution of each unique value for the selected field elements. Show the distribution of HTTP User-Agent and Accept-Encoding headers: Joy:~/joy$./slueth capture.gz --stitch --select "ohttp{user-agent,accept-encoding}" --dist 14
Sleuth --groupby <fields> Split the output into multiple streams, grouping by the field elements specified. Create a separate stream for each destination address: Joy:~/joy$./sleuth capture.gz --stitch --select "sa, da, dp" --groupby da Create a stream for each distinct (destination address, destination port) tuple: Joy:~/joy$./sleuth capture.gz --stitch --select "sa, da, dp" --groupby "da, dp" 15
Sleuth --sum <fields> For each field element specified, compute the sum of their values. Find which destinations had the most data sent to them (outbound bytes): Joy:~/joy$./sleuth capture.gz --stitch --select "da, ob" --groupby "da" --sum "ob" 16
Learning Classifiers
Identify malware flows based upon SPLT and BD Run Joy with the classify option on: Joy:~/joy$ bin/joy bidir=1 dist=1 classify=1../benign/capture.pcap > capture.gz Use Sleuth to show flows with probability of malware greater than 1%: Joy:~/joy$./sleuth capture.gz --select "da, dp, p_malware" --where "p_malware > 0.01" Example output: { "name": [ { "da": "151.101.1.69", "dp": 80, "p_malware": 0.084 }, { "da": "172.217.1.195", "dp": 443, "p_malware": 0.021 } ] } 18
Identify malware flows based upon SPLT and BD Run Joy with the classify option on: Joy:~/joy$ bin/joy bidir=1 dist=1 classify=1 \../malware/133d773a8ca64c24bd81b594cf5240dd.pcap > malware.gz Use Sleuth to show flows with probability of malware greater than 99%: Joy:~/joy$./sleuth malware.gz --select "da, dp, p_malware" --where "p_malware > 0.99" Example output: { "name": [ { "da": 86.59.21.38, "dp": 443, "p_malware": 0.997 }, { "da": 108.61.179.216, "dp": 443, "p_malware": 0.995 }, { "da": 5.9.123.81, "dp": 9001, "p_malware": 0.999 }, { "da": 109.238.6.25, "dp": 443, "p_malware": 0.992 }, { "da": 176.31.159.231, "dp": 443, "p_malware": 0.998 },... 19
Build a malware classifier Create the positive (malware) dataset: Joy:~/joy$ bin/joy bidir=1 dist=1../malware/*.pcap >../malware_train/malware.gz Create the negative (benign) dataset: Joy:~/joy$ bin/joy bidir=1 dist=1../benign/*.pcap >../benign_train/benign.gz Finally, generate the classifier model using the datasets: Joy:~/joy$ python saltui/model.py -m -l -t -p../malware_train/ \ -n../benign_train/ -o params.txt 20
Using the new classifier Run Joy with the model option to specify the new classifier: Joy:~/joy$ bin/joy bidir=1 dist=1 classify=1 model="params.txt:params_bd.txt" \../everything > new_class.gz Use Sleuth to show flows with probability of malware greater than 99%: Joy:~/joy$./sleuth new_class.gz --select "da, dp, p_malware" --where "p_malware > 0.99" Example output: { "name": [ { "da": "239.255.255.250", "dp": 1900, "p_malware": 1.000 }, { "da": "172.16.255.255", "dp": 137, "p_malware": 0.995 }, { "da": "104.237.132.39", "dp": 443, "p_malware": 0.996 },... 21
Conclusions Machine learning and rules applied to passively obtained network traffic can: Detect malware communication Detect misused cryptography SPLT, Byte Distribution and TLS handshake data are valuable. Training classifiers is key. Better than MITM (Man in the middle) with respect to security and privacy. 22
Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click Join the Discussion 3. Install Spark or go directly to the space 4. Enter messages/questions in the space cs.co/ciscolivebot#
Please complete your Online Session Evaluations after each session Complete 4 Session Evaluations & the Overall Conference Evaluation (available from Thursday) to receive your Cisco Live T-shirt All surveys can be completed via the Cisco Live Mobile App or the Communication Stations Complete Your Online Session Evaluation Don t forget: Cisco Live sessions will be available for viewing on-demand after the event at www.ciscolive.com/global/on-demand-library/.
Continue Your Education Demos in the Cisco campus Walk-in Self-Paced Labs Tech Circle Meet the Engineer 1:1 meetings Related sessions 25
Thank you