Behavioral Anomaly Detection of Malware on Home Routers

Transcription

1 Behavioral Anomaly Detection of Malware on Home Routers Ni An, Alex Duff, Gaurav Naik, Michalis Faloutsos, Steven Weber, Spiros Mancoridis CAE Tech Talk 10/19/2017

2 Our team Ni An Alex Duff Gaurav Naik Michalis Faloutsos Steven Weber Spiros Mancoridis October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 2

3 Malware Conference 2017 October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 3

4 Agenda Motivation Malware Experiment setup System call distribution Training, testing, metrics Malware detection algorithms Experimental results Conclusions and future work October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 4

5 Internet of Things (IoT) they re here October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 5

6 IoT s obligation to security and privacy Connected devices [are] collecting, transmitting, storing, and often sharing vast amounts of consumer data, some of it highly personal companies are investing billions of dollars in this growing industry; they should also make appropriate investments in privacy and security. The stakes are too high to do otherwise. Edith Ramirez Former Chairwoman Federal Trade Commission Ramirez E (2015) Privacy and the IoT: navigating policy issues, International consumer electronics show, Las Vegas. Chamberlain A (2017) Special theme on privacy and the Internet of things, Springer Journal on Personal and Ubiquitous Computing, August, October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 6

7 The difficulty of securing the IoT The primary challenge involving the IoT is twofold. First of all, the vast increase in the number of network-enabled devices, which increase the range of possible avenues of attack. Secondly, because many devices that are part of the IoT are themselves vulnerable, they may provide hackers with an easy route to launch on attack on an otherwise secured network. Delgado R (September 22, 2017) Dealing with the endpoint security weaknesses of the Internet of Things, IoTnews. October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 7

8 Malware on the IoT October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 8

9 Basic facts about Mirai malware Malware that turns networked Linux devices into remotely controlled bots (botnet) These bots scan for IoT devices and home network routers and seek to control them via default username/passwords Mirai bots may be remotely controlled for DDoS attacks Mirai botnet first detected August 2016 Krebs On Security attack September 2016 Dyn DNS provider attack October 2016 Known to have infected at least ~500k devices October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 9

10 Behavioral anomaly detection Traditional Anti-virus (AV) approach to malware detection: Modern machine-learning based approach to malware detection: Maintain a list of hashes (signatures, fingerprints) of known malware Drawbacks: Difficult to maintain / update (zero day vulnerabilities) Hashes are imperfect capture of malware (compilation artifacts) Malware designed to adapt to avoid signature-based detection (polymorphism) Advantages: Does not require sensors, behavior monitoring, or baseline behavior Monitor behavior through sensors (features), establish ``normal behavior baseline, and seek to detect anomalies (abnormal behavior) Drawbacks: May be difficult to find the right features Need reliable normal baseline Errors: false positive and missed detections Advantages: Does not require a malware library (less vulnerable to zero day attacks, and polymorphic malware) October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 10

11 Talk overview Question: how well can IoT malware (like Mirai) be detected on IoT-like devices (like home routers), using behavioral anomaly detection (specifically system call analysis)? Sensors: system calls on the home router Features: system-call n-grams Detection: three standard approaches Principal Component Analysis (PCA) One-class Support Vector Machine (SVM) Unseen n-gram detector Intuition: because IoT devices have relatively simple functionality, their normal behavior is easier to characterize, and as such abnormal behavior should be easier to detect. October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 11

12 Agenda Motivation Malware Experiment setup System call distribution Training, testing, metrics Malware detection algorithms Experimental results Conclusions and future work October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 12

13 A Mirai mugshot from VirusShare October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 13

14 Malware used in this project Five different malware binaries: Mr. Black (one variant) Mirai (four variants) 32-bit ELF binaries compiled for ARM architecture Different AV platforms label binaries with different names Malware binaries downloaded from VirusShare October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 14

15 Agenda Motivation Malware Experiment setup System call distribution Training, testing, metrics Malware detection algorithms Experimental results Conclusions and future work October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 15

16 Experimental setup MAWI Client 1 MAWI Client 2 MAWI Client 3 TCP traffic 1 TCP traffic 2 TCP traffic 3 Emulated router (Linux ARM VM) System call sensor built upon ftrace Internet Kernel function system call logs October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 16

17 Network traffic No home router network traffic archive known to us Instead, use (filtered) network backbone traffic from MAWI (Measurement and Analysis of Wide-area Internet, Selected 9 distinct MAWI pcap (packet capture) files, each roughly 15 minutes in duration Found a source IP address / port with HTTP (80) / HTTPS (443) traffic in most of file, and retained only traffic to/from that user Translated the user s IP network address to (private IP network address often used by home routers) Combined these 9 traffic files by forming all possible subsets of size <= 4 (255 distinct combinations) October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 17

18 Number of TCP packets 255 distinct network traffic files 5 # Pcap file index October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 18

19 Home router virtual machine VM: used QEMU (Quick Emulator, qemu.org) open source hypervisor Chosen for its support of ARM architecture Router: used LEDE (Linux Embedded Development Environment, lede-project.org), an open source firmware project Fork of the OpenWrt project Used a customized ARM router firmware Customization includes turning on ftrace to enable system call monitoring October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 19

20 LEDE configuration for QEMU ARM VM October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 20

21 LEDE configuration: enable ftrace ftrace is a tracing framework in the Linux kernel that can be used to record system calls October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 21

22 LEDE config: heimdall s syscall-trace utility heimdall is a malware / anomaly detection utility written by Alex Duff. Includes syscall-trace utility for archiving and filtering the system calls recorded by ftrace October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 22

23 Home router kernel system calls home router kernel includes 379 distinct system calls System calls were recorded for each of the 255 traffic traces System call trace length statistics: Average trace length: 113,464 Maximum trace length: 127,484 Minimum trace length: 46,052 System call diversity statistics Distinct 1-grams (system call): 80 (21.1% of 379) Distinct 2-grams: 2,298 (1.60% of 379^2) Distinct 3-grams: 13,522 (0.02% of 379^3) October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 23

24 Agenda Motivation Malware Experiment setup System call distribution Training, testing, metrics Malware detection algorithms Experimental results Conclusions and future work October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 24

25 PMF of system call bi-grams Distinct bi-grams from traffic traces: 2,298 Count occurrence of each bi-gram, normalize, sort Repeat with syscalls under MrBlack malware, using same order PMF of clean bi-grams PMF of bi-grams of traces infected by MrBlack P (a) Training traces (b) Traces infected by MrBlack October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 25 (c)

26 CDF of system call bi-grams i v-1 1 Clean Mirai-v1 Mirai-v2 0.8 Mirai-v3 Mirai-v4 Mr. Black Distributional differences among Mirai variants Some Mirai variants have disbns close to the clean disbn. 0.9 (d) Bi-gram CDFs October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 26

27 Syscall bi-gram distribution distance A distribution distance measure quantifies the distance between two PMFs Jenson-Shannon divergence (JSD): JSD p, q = 1 KLD pԡz + 1 KLD qԡz 2 2 z 1 (p + q) is the average of (p,q) 2 KLD pԡr σ i p i log p(i) is the Kullback-Leibler divergence r(i) between PMFs (p,r) KLD is the increase in entropy from using the wrong distribution when making a code JSD shows Mirai-v{1,2,3} similar distance from clean, and ten-times closer than MrBlack MrBlack Mirai-v1 Mirai-v2 Mirai-v3 Mirai-v4 JSD October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 27

28 Agenda Motivation Malware Experiment setup System call distribution Training, testing, metrics Malware detection algorithms Experimental results Conclusions and future work October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 28

29 Data preprocessing System call trace truncation: Truncate a system call trace to fragments of length L (e.g., 1000) Bag-of-n-gram (e.g., 1,2,3): n-gram: a sequence of system call numbers appearing in a small window of length n Bag-of-n-gram model: the sum of the one-hot vectors of n-grams in appearing in a system-call fragment x i = (1,1,1,2,1,1) 2-gram 192, , 6 1 6, 5 1 5, 5 2 5, , 6 1 Freq. October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 29

30 TF-IDF transformation Term-frequency inverse document frequency Here: terms are n-grams, documents are syscall fragments Matrix X: rows are syscall fragment indices, columns are n-gram indices X i,j counts the number of occurrences of n-gram j in fragment i of N i : X i,j > 0 counts the number of chunks in which n-gram j occurs TF-IDF transformation of X TF IDF X i,j = TF X i,j IDF(X i,j ) TF X i,j = 1 + log X i,j, X i,j > 0 0, X i,j = 0 IDF X i,j = 1 + log 1+N 1+ i :X i,j >0 Intuition: TF compresses range of X, IDF increases weight of rare n-grams Final step: standardize each feature vector to unit norm October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 30

31 Training, testing, cross-validation Approach: 5-fold cross-validation Divide the 255 clean call-traces (one per traffic file) into 5 samples, each sample with 51 call-traces For each (of 5) clean sample: Training: omitting the clean sample, use the remaining four clean samples for training Testing: add one of the five malware traces to the omitted clean sample, use the combined trace for testing Classify each fragment in testing as either normal or abnormal Classify each call-trace as abnormal if one or more fragments is classified as abnormal Repeat each experiment five times and average Repeat for each of the five malware samples October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 31

32 Performance metrics True positive rate TPR = # of infected traces labeled abnormal # of infected traces fraction of malware samples we captured False alarm rate FAR = # of clean traces labeled abnormal # of clean traces fraction of clean samples labeled as malware October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 32

33 Agenda Motivation Malware Experiment setup System call distribution Training, testing, metrics Malware detection algorithms Experimental results Conclusions and future work October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 33

34 Malware detection algorithms Three standard semi-supervised learning approaches to detection SSL: labeled examples of normal, no labeled examples of abnormal, in training Alg. #1: Principal Component Analysis (PCA) Alg. #2: One-class Support Vector Machine (1SVM) Alg. #3: Naïve anomaly detector (NAD) using unseen n-grams October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 34

35 { { Alg. #1: Principal Component Analysis (PCA) PCA is a standard approach to embedding a collection of points (rows of X) into a lower dimensional subspace PCA-based statistical anomaly detection labels points as (ab)normal if the distance from the point to the subspace is (above) below a specified threshold x i x i (residual) Ŝ ˆx i x T i V (k ) N V (k ) T N x i : Principal subspace October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 35

36 Alg. #2: One-class Support Vector Machine (1SVM) Training: find a hyperplane that has the largest margin separating the training dataset from the origin. Testing: label point x as normal if w T x > a normal if, sweep a for ROC Training Test Anomalies Test Normal w w Support vector Separation hyperplane: w T φ(x) = 0 Inversely proportional to the margin Upper-bounds the fraction of outliers in the training set 1 min w,ξ,ρ 2 wt w + 1 σ νn i=1 N ξ i ρ Subject to w T φ x i ρ ξ i and ξ i 0. A training sample Bias term Penalty term for error October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 36

37 Alg. #3: Naïve anomaly detector (NAD) using unseen n-grams Training: build a normal dictionary D, consisting of all n-grams that appears in the training system call traces Testing: N A = of n-grams that are not in D If N A > τ A, label fragment as abnormal otherwise, label the fragment as normal. October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 37

38 Agenda Motivation Malware Experiment setup System call distribution Training, testing, metrics Malware detection algorithms Experimental results Conclusions and future work October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 38

39 Alg. #1: Principal Component Analysis (PCA) Fig.1. ROC curves of PCA-SAD with L = 1000, n = 2 October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 39

40 Alg. #2: One-class Support Vector Machine (1SVM) Fig.2. ROC curves of one-class SVM with PNS, L = 3000, n = 2 October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 40

41 Alg. #3: Naïve anomaly detector (NAD) using unseen n-grams Fig.3. ROC curves of NAD with L = 1000, n = 2 October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 41

42 System calls in infected traces Malware name Example of system-calls appearing ONLY in infected traces MrBlack 19 (lseek), 78 (gettimeofday), 94 (fchmod), 120 (clone), 191 (vfork), 207 (fchown32), 289 (send), 338 (set_robust_list) Mirai-v1 Mirai-v2 Mirai-v3 Mirai-v4 1(exit), 55 (fcntl), 78 (gettimeofday), 208 (setresuid32) 78 (gettimeofday), 120 (clone), 208(setresuid32) 1(exit), 55 (fcntl), 78 (gettimeofday) 43 (times), 78 (gettimeofday), 120 (clone), 141 (getdents) 10/20/2017 Drexel University Isaac L. Auerbach Cybersecurity Institute 42

43 Distribution of 1-grams PMF of One-grams infected by Mirai (v4) 0PMF One-grams infected by Mr. Black PMF of clean One-grams 0PMF of infected by Mirai (v1) 0PMF of of One-grams infected by M 10 0 PMF of clean One-grams PMF of One-grams infected by Mirai (v1) (v1) PMF of of One-grams infected by Mirai (v2) PMF of One-grams infected by Mirai (v3) PMF of One-grams infected by Mirai (v4) PMF of One-grams infected by Mirai (v3) 10 0PMF of One-grams infected by Mirai (v4) PMF of One-grams infected by Mr. 0PMF of One-grams infected by Mirai (v3) 0PMF of One-grams infected by Mirai (v4) (v4) 10/20/ Drexel -6 University Isaac L. Auerbach Cybersecurity Institute PMF of One-grams infected by Mr. Black

44 Impact of L, n on detection accuracy October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 44

45 Impact of L, n on detection accuracy PCA-SAD and one-class SVM with partial n-gram space are sensitive to the change of L; as L increases, the detection accuracy increases. L has negligible impact on the NAD and the oneclass SVM with complete n-gram space The superiority of the NAD and the one-class SVM with complete n-gram space shows the importance of the information of the new n- grams caused by the malware A larger n does not necessarily result in a higher detection accuracy October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 45

46 Agenda Motivation Malware Experiment setup System call distribution Training, testing, metrics Malware detection algorithms Experimental results Conclusions and future work October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 46

47 Limitations of the analysis Emulated home router s behavior is relatively simple: the router s functionality is restricted to relaying traffic Botnet malware not able to connect to a command and control (C&C) server within the VM October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 47

48 Conclusion and future work Syscall-based behavioral anomaly detection algorithms can effectively detect previously unknown malware on home routers The one-class SVM using the complete 2-gram space and the NAD achieve best detection Future work #1: joint syscall and traffic analysis Future work #2: investigate benefit of shared training across multiple home routers October 19, 2017 CAE Tech Talk - Drexel U. Isaac L. Auerbach Cybersecurity Institute 48