Data Provenance at Internet Scale: Architecture, Experiences, and the Road Ahead. Ang Chen, Yang Wu, Andreas Haeberlen, Boon Thau Loo, Wenchao Zhou

Transcription

1 Data Provenance at Internet Scale: Architecture, Experiences, and the Road Ahead Ang Chen, Yang Wu, Andreas Haeberlen, Boon Thau Loo, Wenchao Zhou

2 Motivation D E Alice A B C foo.com An example scenario: network routing System administrator observes strange behavior Example: the route to foo.com has suddenly changed Anomalies in distributed systems Need a way to explain system behavior. 2

3 Motivation D E Alice Route r 1 A foo.com An example scenario: network routing System administrator observes strange behavior Example: the route to foo.com has suddenly changed Anomalies in distributed systems Need a way to explain system behavior. B C 2

4 Motivation D Route r 2 E Alice A B C foo.com An example scenario: network routing System administrator observes strange behavior Example: the route to foo.com has suddenly changed Anomalies in distributed systems Need a way to explain system behavior. 2

5 Motivation Why did my route to foo.com change?! D Route r 2 E Alice A B C foo.com An example scenario: network routing System administrator observes strange behavior Example: the route to foo.com has suddenly changed Anomalies in distributed systems Need a way to explain system behavior. 2

6 Motivation Why did my route to foo.com change?! D Route r 2 E Innocent Reason? Alice A B C foo.com An example scenario: network routing System administrator observes strange behavior Example: the route to foo.com has suddenly changed Anomalies in distributed systems Need a way to explain system behavior. 2

7 Motivation Why did my route to foo.com change?! D Route r 2 E Innocent Reason? Software Bugs? Alice A B C foo.com An example scenario: network routing System administrator observes strange behavior Example: the route to foo.com has suddenly changed Anomalies in distributed systems Need a way to explain system behavior. 2

8 Motivation Why did my route to foo.com change?! D Route r 2 E Innocent Reason? Software Bugs? Malicious Attack? Alice A B C foo.com An example scenario: network routing System administrator observes strange behavior Example: the route to foo.com has suddenly changed Anomalies in distributed systems Need a way to explain system behavior. 2

9 Data-centric Perspective on Network Debugging D E foo.com Alice A B C We assume a general distributed system Network consists of nodes (routers, middleboxes,...) The state of a node is a set of tuples (routes, config,...) 3

10 Data-centric Perspective on Network Debugging D route(a, foo.com) E foo.com Alice A route(a, B) route(a, D) B C We assume a general distributed system Network consists of nodes (routers, middleboxes,...) The state of a node is a set of tuples (routes, config,...) 3

11 Data-centric Perspective on Network Debugging D route(a, foo.com) E foo.com Alice A link(a, B) link(a, D) route(a, B) route(a, D) B C We assume a general distributed system Network consists of nodes (routers, middleboxes,...) The state of a node is a set of tuples (routes, config,...) 3

12 Data-centric Perspective on Network Debugging D route(a, foo.com) E foo.com Alice A B C We assume a general distributed system Network consists of nodes (routers, middleboxes,...) The state of a node is a set of tuples (routes, config,...) Idea: Explanation as reasoning about distributed state dependencies 3

13 Data-centric Perspective on Network Debugging D E route(a, foo.com) foo.com Alice A link(a, B) route(b, foo.com) B C We assume a general distributed system Network consists of nodes (routers, middleboxes,...) The state of a node is a set of tuples (routes, config,...) Idea: Explanation as reasoning about distributed state dependencies 3

14 Data-centric Perspective on Network Debugging D E route(a, foo.com) foo.com Alice A link(a, B) route(b, foo.com) route(c, foo.com) B link(b, C) C link(c, foo.com) We assume a general distributed system Network consists of nodes (routers, middleboxes,...) The state of a node is a set of tuples (routes, config,...) Idea: Explanation as reasoning about distributed state dependencies 3

15 Network Provenance [SIGMOD 2010] route(d, foo.com) route(e, foo.com) D link(d, E) route(a, foo.com) E link(e, B) foo.com Alice A link(a, B) route(b, foo.com) route(c, foo.com) B link(b, C) C link(c, foo.com) 4

16 Network Provenance [SIGMOD 2010] route(d, foo.com) link(d, E) route(e, foo.com) link(e, B) route(a, foo.com) link(a, B) route(b, foo.com) route(c, foo.com) link(b, C) Provenance for encoding distributed state dependencies Explains the derivation of tuples Captures the dependencies between tuples as a graph link(c, foo.com) 4

17 Network Provenance [SIGMOD 2010] route(a, foo.com) link(a, B) route(b, foo.com) route(c, foo.com) link(b, C) link(c, foo.com) Provenance for encoding distributed state dependencies Explains the derivation of tuples Captures the dependencies between tuples as a graph Explanation of a tuple is an acyclic graph rooted at the tuple 4

18 NetTrails: First Generation Network Provenance Tool [SIGMOD 2011 demo] 5

19 Network Provenance Research Network provenance [SIGMOD 10] ( ) Secure network provenance [SOSP 11] Provenance in dynamic environments [VLDB 13] Explanations Negative provenance [SIGCOMM 14] Distributed provenance compression [SIGMOD 17] Differential provenance [SIGCOMM 16] Meta-provenance [NSDI 17] Deeper diagnostics and repair Ph.D. dissertation work of Ang Chen (2017), Chen Chen (2017), Yang Wu (2017), and Wenchao Zhou (2012). 6

20 Assumption #1: All nodes in the network can be trusted Why did my route to foo.com change?! D Route r 2 E Alice A B C foo.com 7

21 Assumption #1: All nodes in the network can be trusted Q: Explain why the route to foo.com changed to r2. D Route r 2 E Alice A B C foo.com The Network 7

22 Assumption #1: All nodes in the network can be trusted Q: Explain why the route to foo.com changed to r2. D Route r 2 E Alice A B C foo.com The Network A: Because someone accessed Router D and changed the configuration from X to Y. 7

23 Assumption #1: All nodes in the network can be trusted Q: Explain why the route to foo.com changed to r2. D Route r 2 E Alice A B C foo.com The Network A: Because someone accessed Router D and changed the configuration from X to Y. Not realistic: adversary can tell lies 7

24 Challenge: Adversaries Can Lie I should cover up the intrusion. Q: Explain why the route to foo.com changed to r2. D Route r 2 E Alice A B C foo.com The Network Problem: adversary can... fabricate plausible (yet incorrect) response point accusation towards innocent nodes 8

25 Challenge: Adversaries Can Lie Everything is fine. Router E advertised a new route. Q: Explain why the route to foo.com changed to r2. D Route r 2 E Alice A B C foo.com The Network Problem: adversary can... fabricate plausible (yet incorrect) response point accusation towards innocent nodes 8

26 Secure Network Provenance (SNP) SOSP 2011 route(d, foo.com) link(d, E) route(e, foo.com) link(e, B) route(a, foo.com) link(a, B) route(b, foo.com) link(b, C) route(c, foo.com) link(c, foo.com) Step 1: Each node keeps vertices about local actions Split cross-node communications 9

27 Secure Network Provenance (SNP) SOSP 2011 Step 1: Each node keeps vertices about local actions Split cross-node communications 9

28 Secure Network Provenance (SNP) SOSP 2011 Step 1: Each node keeps vertices about local actions Split cross-node communications 9

29 Secure Network Provenance (SNP) SOSP 2011 Step 1: Each node keeps vertices about local actions Split cross-node communications 9

30 Secure Network Provenance (SNP) SOSP 2011 RECV SEND Step 1: Each node keeps vertices about local actions Split cross-node communications 9

31 Secure Network Provenance (SNP) SOSP 2011 RECV SEND Step 1: Each node keeps vertices about local actions Split cross-node communications Step 2: Make the graph tamper-evident 9

32 SNP Guarantees Q: Why did my route to foo.com change to r2? D Route r 2 E Alice A B The Network C foo.com A: Because someone accessed Router D and changed its router configuration from X to Y. No faults: Explanation is complete and accurate Byzantine fault: Explanation identifies at least one faulty node 10

33 SNP Guarantees Q: Why did my route to foo.com change to r2? D Route r 2 E Alice A B The Network C foo.com A: Because someone accessed Router D and changed its router configuration from X to Y. No faults: Explanation is complete and accurate Byzantine fault: Explanation identifies at least one faulty node 10

34 SNP Guarantees Q: Why did my route to foo.com change to r2? D Route r 2 E Alice A B The Network C foo.com A: Because someone accessed Router D and changed its router configuration from X to Y. No faults: Explanation is complete and accurate Byzantine fault: Explanation identifies at least one faulty node 10

35 SNP Guarantees Q: Why did my route to foo.com change to r2? D Route r 2 E Alice A B The Network C foo.com Aha, at least I know which node is compromised. A: Because someone accessed Router D and changed its router configuration from X to Y. No faults: Explanation is complete and accurate Byzantine fault: Explanation identifies at least one faulty node 10

36 Assumption #2: Operators react only to presence of anomaly events 11

37 Assumption #2: Operators react only to presence of anomaly events - What if something expected is not happening? - Missing events cannot be handled by existing tools 11

38 Assumption #2: Operators react only to presence of anomaly events - What if something expected is not happening? - Missing events cannot be handled by existing tools Controller Internet Data Center Network HTTP Server 11

39 Assumption #2: Operators react only to presence of anomaly events - What if something expected is not happening? - Missing events cannot be handled by existing tools Controller Why is the HTTP server NOT getting requests???? Internet Data Center Network HTTP Server 11

40 How common are missing events? - Missing events are consistently in the majority - Lengthier threads for missing events Missing events NANOG-user floodlight-dev Positive events Outages 26% 17% 52% 48% 74% 83% NANOG-user Floodlight-dev Outages 12

41 Negative Provenance [SIGCOMM 2014] 13

42 Negative Provenance [SIGCOMM 2014] Controller Internet Data Center Network HTTP Server 13

43 Negative Provenance [SIGCOMM 2014] No HTTP Packet arrived at HTTP Server Why is the HTTP server NOT getting requests? Controller??? Internet Data Center Network HTTP Server 13

44 Negative Provenance [SIGCOMM 2014] No HTTP Packet arrived at HTTP Server No Forwarding-FlowEntry installed at Switch Controller Why is the HTTP server NOT getting requests??????? Internet Data Center Network HTTP Server 13

45 Negative Provenance [SIGCOMM 2014] No HTTP Packet arrived at HTTP Server No Forwarding-FlowEntry installed at Switch Controller Why is the HTTP server NOT getting requests? HTTP Packet received at Switch HTTP Packet?????? Internet Data Center Network HTTP Server 13

46 Negative Provenance [SIGCOMM 2014] No HTTP Packet arrived at HTTP Server No Forwarding-FlowEntry installed at Switch Controller Why is the HTTP server NOT getting requests? HTTP Packet received at Switch Dropping-FlowEntry existed at Switch HTTP Packet Dropping- FlowEntry?????? Internet Data Center Network HTTP Server 13

47 Negative Provenance [SIGCOMM 2014] No HTTP Packet arrived at HTTP Server No Forwarding-FlowEntry installed at Switch Controller Why is the HTTP server NOT getting requests? HTTP Packet received at Switch Dropping-FlowEntry existed at Switch Program HTTP Packet Dropping- FlowEntry?????? Internet Data Center Network HTTP Server 13

48 Negative Provenance [SIGCOMM 2014] No HTTP Packet arrived at HTTP Server No Forwarding-FlowEntry installed at Switch Controller Why is the HTTP server NOT getting requests? HTTP Packet received at Switch Dropping-FlowEntry existed at Switch Program HTTP??? Packet Dropping- FlowEntry??? Internet Data Center Network HTTP Server 13

49 Approach: Counter-factual reasoning Find all the ways a missing event could have occurred, and show why each of them did not happen. 14

50 Approach: Counter-factual reasoning Find all the ways a missing event could have occurred, and show why each of them did not happen. Philly 14

51 Approach: Counter-factual reasoning Find all the ways a missing event could have occurred, and show why each of them did not happen. Philly Santa Cruz 14

52 Approach: Counter-factual reasoning Find all the ways a missing event could have occurred, and show why each of them did not happen. Why did Bob NOT arrive at CIDR? Philly Santa Cruz 14

53 Approach: Counter-factual reasoning Find all the ways a missing event could have occurred, and show why each of them did not happen. Why did Bob NOT arrive at CIDR? Philly Santa Cruz 14

54 Approach: Counter-factual reasoning Find all the ways a missing event could have occurred, and show why each of them did not happen. Why did Bob NOT arrive at CIDR? Philly Santa Cruz 14

55 Approach: Counter-factual reasoning Find all the ways a missing event could have occurred, and show why each of them did not happen. Why did Bob NOT arrive at CIDR? Philly Santa Cruz 14

56 Approach: Counter-factual reasoning Find all the ways a missing event could have occurred, and show why each of them did not happen. Why did Bob NOT arrive at CIDR? Philly Santa Cruz 14

57 Assumption #3: Provenance trees alone are sufficient for diagnostics 15

58 Assumption #3: Provenance trees alone are sufficient for diagnostics root Symptom Root cause 15

59 Assumption #3: Provenance trees alone are sufficient for diagnostics C received packet B sent packet Rule match on B root Symptom Root cause 15

60 Assumption #3: Provenance trees alone are sufficient for diagnostics root Symptom Root cause 15

61 Assumption #3: Provenance trees alone are sufficient for diagnostics Packet arrives at the wrong server root Symptom Root cause 15

62 Assumption #3: Provenance trees alone are sufficient for diagnostics Packet arrives at the wrong server Rule 7: Next-hop=port2 root Symptom Root cause 15

63 What can we do? 16

64 What can we do? From: To: Admin Title: Help! My server is receiving suspicious traffic from /24--it should have been sent to the low-security server. Packets from /24 are still being routed correctly. Can you help? 16

65 What can we do? From: To: Admin Title: Help! My server is receiving suspicious traffic from /24--it should have been sent to the low-security server. Packets from /24 are still being routed correctly. Can you help? Working reference! 16

66 What can we do? From: To: Admin Title: Help! My server is receiving suspicious traffic from /24--it should have been sent to the low-security server. Packets from /24 are still being routed correctly. Can you help? Outages mailing list Sept. Dec. 2014: Working reference! 66% have references! 16

67 What can we do? S1 S2 S3 S4 S5 S6 Web server 1 DPI Web server 2 Bob 16

68 What can we do? S1 S2 S3 S4 S5 S6 Web server 1 DPI Web server 2 Bob Idea: Reason about the differences between the symptom and the reference 16

69 Differential provenance [SIGCOMM 2016] fails works Input: a bad symptom and a good reference 17

70 Differential provenance [SIGCOMM 2016] fails works Differential Provenance Input: a bad symptom and a good reference 17

71 Differential provenance [SIGCOMM 2016] fails works Differential Provenance Input: a bad symptom and a good reference Debugger reasons about the differences 17

72 Differential provenance [SIGCOMM 2016] fails works Differential Provenance Rule 7 s next hop is wrong! Input: a bad symptom and a good reference Debugger reasons about the differences Output: root cause 17

73 Strawman solution faulty rule - = root? root Provenance (Symptom) Provenance (Reference) Strawman solution: Find vertexes that are different in the two trees 18

74 Strawman solution faulty rule - = root root Provenance (Symptom) Provenance (Reference) Strawman solution: Find vertexes that are different in the two trees Problem: The diff can be larger than the individual trees! 18

75 Overly Simplified Approach in a nutshell Roll back the execution to a divergence point Change the faulty node to be like the correct node Roll forward the execution to align the trees 19

76 Assumption #4: Software is correct and static - Networks are software and can have bugs 20

77 Assumption #4: Software is correct and static - Networks are software and can have bugs S2 5 S0 3 4 S1 Backup Web Server Main Web Server DNS Server 20

78 Assumption #4: Software is correct and static - Networks are software and can have bugs SDN Controller S2 5 S0 3 4 S1 Backup Web Server Main Web Server DNS Server 20

79 Assumption #4: Software is correct and static - Networks are software and can have bugs SDN Controller S2 5 S0 HTTP traffic 3 4 S1 Backup Web Server Main Web Server DNS Server 20

80 Assumption #4: Software is correct and static - Networks are software and can have bugs SDN Controller S2 5 Off-loading HTTP S0 HTTP traffic 3 4 S1 Backup Web Server Main Web Server DNS Server 20

81 Assumption #4: Software is correct and static - Networks are software and can have bugs else if (switch == S1 && protocol == HTTP) then action = output:3. SDN Controller S2 5 Off-loading HTTP S0 HTTP traffic 3 4 S1 Backup Web Server Main Web Server DNS Server 20

82 Assumption #4: Software is correct and static - Networks are software and can have bugs else if (switch == S1 && protocol == HTTP) then action = output:3. SDN Controller S2 5 Off-loading HTTP S0 HTTP traffic 3 4 S1 Backup Web Server Main Web Server DNS Server 20

83 Assumption #4: Software is correct and static - Networks are software and can have bugs else if (switch == S1 && protocol == HTTP) then action = output:3. else if (switch == S1 && protocol == HTTP) then action = output:5. SDN Controller S2 5 Off-loading HTTP S0 HTTP traffic 3 4 S1 Backup Web Server Main Web Server DNS Server 20

84 Assumption #4: Software is correct and static - Networks are software and can have bugs else if (switch == S1 && protocol == HTTP) then action = output:3. else if (switch == S1 && protocol == HTTP) then action = output:5. SDN Controller S2 5 Off-loading HTTP S0 HTTP traffic 3 4 S1 Backup Web Server Main Web Server DNS Server 20

85 Assumption #4: Software is correct and static - Networks are software and can have bugs else if (switch == S1 && protocol == HTTP) then action = output:3. else if (switch == S1 && protocol == HTTP) then action = output:5. Copy-and-paste bug!!! SDN Controller S2 5 Off-loading HTTP S0 HTTP traffic 3 4 S1 Backup Web Server Main Web Server DNS Server 20

86 Assumption #4: Software is correct and static - Networks are software and can have bugs else if (switch == S1 && protocol == HTTP) then action = output:3. else if (switch == S1 && protocol == HTTP) then action = output:5. Copy-and-paste bug!!! SDN Controller S2 5 Off-loading HTTP S0 HTTP traffic 3 4 S1 Backup Web Server Main Web Server DNS Server 20

87 Assumption #4: Software is correct and static - Networks are software and can have bugs else if (switch == S1 && protocol == HTTP) then action = output:3. else if (switch == S1 && protocol == HTTP) then action = output:5. Copy-and-paste bug!!! Why is the backup web server not getting requests? SDN Controller S2 5 Off-loading HTTP S0 HTTP traffic 3 4 S1 Backup Web Server Main Web Server DNS Server 20

88 Assumption #4: Software is correct and static - Networks are software and can have bugs - How can we find and fix bugs quickly? else if (switch == S1 && protocol == HTTP) then action = output:3. else if (switch == S1 && protocol == HTTP) then action = output:5. Copy-and-paste bug!!! Why is the backup web server not getting requests? SDN Controller S2 5 Off-loading HTTP S0 HTTP traffic 3 4 S1 Backup Web Server Main Web Server DNS Server 20

89 Approach: Meta provenance [NSDI 2017] - Problem: Finding fixes is hard - Idea: Provenance can pinpoint the root cause - But previous provenance focus exclusively on data - Key idea: Treating program as data HTTP Packet received at Main Web Server meta provenance Matching Flow Entry installed at S1 PacketIn received at Controller Executed If Clause in Controller Program 21

90 Repairing Software-defined Networking Programs Full support of Network Datalog (declarative) Support a subset of Pyretic (Python + DSL) Support a subset of Trema (imperative Ruby) Uses Z3 SMT solver to enumerate repairs More details are in [HotNets 15, NSDI 17] 22

91 Network Provenance Research ( ) Network provenance model [SIGMOD 10] Secure network provenance [SOSP 11] Provenance in dynamic environments [VLDB 13] Explanations Negative provenance [SIGCOMM 14] Distributed provenance compression [SIGMOD 17] Differential provenance [SIGCOMM 16] Meta-provenance [NSDI 17] Deeper diagnostics and repair Ph.D. dissertation work of Ang Chen (2017), Chen Chen (2017), Yang Wu (2017), and Wenchao Zhou (2012). 23

92 The Road Ahead Network forensics meets data provenance is a rich area of exploration! Sampling of the problems we are working on: Network forensics on the data plane Privacy-preserving provenance on sensitive networks Probabilistic provenance Automated repairs of complex events Timing-based provenance and more. 24

93 Thank You! Network provenance team at Penn/Georgetown: Ang Chen, Chen Chen, Ling Ding, Qiong Fei, Andreas Haeberlen, Zachary Ives, Yang Li, Boon Thau Loo, Suyog Mapara, Arjun Narayan, Yiqing Ren, Micah Sherr, Shengzhi Sun, Tao Tao, Yang Wu, Mingchen Zhao, Wenchao Zhou 25