Towards Scalable Cluster Auditing through Grammatical Inference over Provenance Graphs

Transcription

1 Towards Scalable Cluster Auditing through Grammatical Inference over Provenance Graphs Wajih Ul Hassan, Mark Lemay, Nuraini Aguse, Adam Bates, Thomas Moyer NDSS Symposium 2018 Feb 20, 2018

2 Notable Data Breach in

3 Notable Data Breach in 2017 Equifax Data Breach Timeline 2017 Breached Detected Breached Announced apr may jun jul aug sep oct Hackers in Equifax Servers Patched 3

4 Notable Data Breach in 2017 Equifax Data Breach Timeline 2017 Breached Detected Breached Announced 3 Months of crucial attack audit logs apr may jun jul aug sep oct Hackers in Equifax Servers Patched 4

5 Notable Data Breach in 2017 Equifax Data Breach Timeline 2017 Breached Detected apr may jun jul aug sep oct Breached Announced 3 Months of crucial attack audit Are current auditing systems scalable? logs Hackers in Equifax Servers Patched 5

6 Data Provenance aka Audit log Lineage of system activities Represented as Directed Acyclic Graph (DAG) Used for forensic analysis Code Audit log Provenance Graph Bash: exec(./nginx ); NGINX: recv(, abc.com ); fread( index.html ); 1. Bash, Spawns NGINX 2. NGINX, Receives from abc.com 3. NGINX, Reads File index.html Bash NGINX abc.com index.html 6

7 Data Provenance in a Cluster Worker Nodes Centralized auditing not practical due to two limitations Master Node 7

8 Limitation#1: Graph Complexity NGINX and MySQL running for 5 mins on a single machine Finding needle in a haystack problem 8

9 Limitation#2: Storage overhead Leads to network overhead as logs are transferred to master node Audit Log Size Growth for a Single NGINX server Uncompressed Compressed [VALUE] GB 2.54 GB/Day on a single machine 10 Log Size (GB) [VALUE] GB [VALUE] GB With compression >1GB/Day 2 0 [VALUE] GB Day1 Day2 Day3 Day4 Day5 9

10 Winnower Cluster applications are replicated in accordance with microservice architecture principle Replicated apps produce highly homogeneous provenance graphs core execution behaviour is similar Key Idea: Remove redundancy from provenance graphs across cluster before sending to master node 10

11 Master Node View with Winnower Before After Bash * NGINX Other library file ver@ces mysql mysqld /up/* * /db/* 11

12 Winnower Build consensus model across cluster using graph grammars Like string grammar, graph grammars provide rule-based mechanisms For generating, manipulating and analyzing graphs Induction produce grammar from a given graph Parsing membership test of a given graph is in a grammar t t t t S A T A a B a a b b b a a b e T B t b S Graph S e Graph Grammar 12

13 Architecture Worker Nodes Worker Node Audit Module Prov. Graph Master Node Model Aggregator Model graphs/grammars from cluster Winnower Agent Fetch graph at each epoch Fine-grained Graph Graph Abstraction Abstracted Graph Graph Induction Model Graph 13

14 Architecture Worker Nodes Worker Node Audit Module Prov. Graph Master Node Model Aggregator Aggregated Model Only send Model updates Winnower Agent Fetch graph at next epoch Fine-grained Graph Graph Abstraction Abstracted Graph Graph Induction Model Graph 14

15 Architecture Worker Nodes Worker Node Audit Module Master Node Model Aggregator Query part of Provenance High-fidelity graph Provenance graph Winnower Agent 15

16 Provenance Graph Abstraction Graph Induction process builds a model/grammar that concisely describe the whole graph However, instance-specific fields frustrate any attempts to build a generic application behaviour model Node 1 Node 2 listener pid:2791 pid:2788 worker pid:2795 pid:2780 listener pid:2789 worker pid: Graph Induc@on No General model as instance specific informa@on such PID is different among graphs /up/file1 Inode:3 /up/file2 Inode:

17 Provenance Graph Abstraction Provenance graph vertices have well defined fields E.g. pid:1234, FilePath:/etc/ld.so Defined rules manually that remove or generalize these fields Node 1 Node 2 pid:2788 pid: Node /24 Node /24 listener pid:2791 worker pid:2795 listener pid:2789 worker pid:2797 Graph Abstrac@on listener worker listener worker /up/file1 Inode:3 /up/file2 Inode: /24 /up/* /up/* /24

18 Provenance Graph Induction Deterministic Finite Automata (DFA) Learning to generate grammar Encodes the causality in generated models In DFA learning the present state of a vertex includes the path taken to reach the vertex (provenance ancestry) Winnower extends it to remember descendants (provenance progeny) State of each vertex consist of three items: 1. Label 2. Provenance ancestry 3. Provenance progeny Bash gzip Ancestry of gzip vertex File1.txt File1.txt Progeny of gzip vertex 18

19 Provenance Graph Induction Finds repetitive patterns using standard implicit and explicit state merging algorithm Implicit state merging combines two subgraphs if states of each vertex are same in both subgraphs Node /24 Node / /24 listener listener worker listener worker Graph Induc@on worker /24 /up/* /24 /up/* /up/* /24 Confidence level Legend 2 19

20 Explicit State Merging At high-level explicit state merging Picks two nodes and make their states same Check if subgraph can be merged implicitly Consider a chained map reduce job java java mapper data Node 1 java java mapper data java Graph Grammar := S S := A S := T V Node 1 := A -> X A -> Y X := B -> W Y := C -> W data W := D D -> S java mapper T java java mapper B A D data java reducer Merge two nodes java reducer java reducer A B C D := data := java mapper := java reducer := java java reducer C 20

21 Provenance Graph Induction Consider a graph with a malicious activity Malicious behavior is visible in the final model listener worker bash wget Node 1 /up/* x.x.x.x Malicious file /up/* /up/* Node 2 Node 3 listener worker listener worker Graph Induc@on /up/* Master Node listener worker bash x.x.x.x wget Confidence level Legend 1 3 Malicious file 21

22 Evaluation Setup Setup 1 VM as master node, 4 VMs as worker nodes SPADE and Docker Swarm Epoch size 50 sec Metrics Storage Overhead Computational Cost Effectiveness 22

23 Storage Overhead on Master Node 98.7% decrease 0.17 MySQL 130 Winnower Raw 0.12 ProFTPD HTTPD LOG SIZE IN MB 23

24 Storage Reduction on Master Node LOG SIZE (MB) Raw(Uncompressed) Raw(Compressed) Winnower Apache Webserver with moderate workload Note the log scale on y- axis 7z compression is not suitable: No global view of cluster Oblivious to previous batch TIME (SEC) 24

25 Evaluation: Computation Cost Average time spent in induction and membership test at each epoch 35 Generate Model for 30 Average Time (sec) Apache MySQL ProFTPD Membership check in model Heterogeneous Workload -> Updates model Elapsed Time (sec) 25

26 Case Study: Ransomware Attack Attacker exploits Redis database server vulnerability version < 3.2 Vulnerability allows attacker to change SSH key and log in as Root Attacker deletes the database and left a note using vim to send bitcoins get database back 26

27 Traditional Graph of Attack 10 instances of redis running in the cluster ~80k vertices and ~83K edges with 161 MB size Part of provenance graph shown below 27

28 Winnower Generated Provenance graph 54 vertices and 68 edges with 0.7 MB size Part of graph is shown below: * bash * Nginx * Worker /uploads/* /proc/12743/stat /var/log/redis/redis.log Confidence level Legend 1 10 redis-server /var/lib/redis/dump.rdb /24 x.x.x.x /dev/tty /root/.ssh/authorized_keys x.x.x.x sshd bash Other library files Attack Provenance vim /root/ransomware.note 28

29 Winnower Generated Provenance graph What happens if we attack all the nodes in the cluster * bash * Nginx * Worker /uploads/* /proc/12743/stat /var/log/redis/redis.log Confidence level Legend 10 redis-server /var/lib/redis/dump.rdb /24 x.x.x.x /dev/tty /root/.ssh/authorized_keys x.x.x.x sshd bash Other library files Attack Provenance vim /root/ransomware.note 29

30 Conclusion Winnower is the first practical system for provenance-based auditing of clusters at scale with low overhead Winnower significantly improves attack identification and investigation in a large cluster 30

31 Questions Thank you for your time. 31

32 Backup Slides 32

33 Threat model Assumptions Winnower only tracks user-space attacks i.e. trusts the OS Log integrity is maintained Attack surface Distributed application replicated on Worker nodes Attacker motive Gain control over worker node by exploiting a software vulnerability in the distributed application 33