SDNRacer. Concurrency Analysis for SDNs. Ahmed El-Hassany Jeremie Miserez Pavol Bielik Laurent Vanbever Martin Vechev.

Transcription

1 SDNRacer Concurrency Analysis for SDNs Ahmed El-Hassany Jeremie Miserez Pavol Bielik Laurent Vanbever Martin Vechev ETH Zürich April 29 th,

2 Load Balancer Application 2

3 Load Balancer Application Controller load-balancer Internet Host#1 S2 Replica#1 Replica#2 3

4 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) Controller load-balancer Internet Host#1 S2 Replica#1 Replica#2 4

5 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) Controller load-balancer read(s1, pkt) Internet Host#1 1 S2 Replica#1 Replica#2 5

6 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) Controller load-balancer PktIn(s1, pkt) 2 Internet Host#1 1 S2 Replica#1 Replica#2 6

7 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) Select Replica #1 packet_out(pkt,in sw) Controller load-balancer PktIn(s1, pkt) 2 Internet Host#1 1 S2 Replica#1 Replica#2 7

8 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) Controller load-balancer write(s1, match, out S2) 2 3 Internet Host#1 1 src H1 out S2 S2 Replica#1 Replica#2 8

9 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) Controller load-balancer write(s2, match, out R1) Host#1 Internet S2 Replica#1 src H1 out S2 src H1 out R1 Replica#2 9

10 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) write(s1, match -1, out Internet) Controller load-balancer Internet Host#1 1 src H1 out S2 dst H1 out I S2 Replica#1 src H1 out R1 Replica#2 10

11 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) Controller load-balancer write(s2, match -1, out ) Internet Host#1 1 src H1 out S2 dst H1 out I S2 src H1 out R1 dst H1 out Replica#1 Replica#2 11

12 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) Controller load-balancer PktOut(s1, pkt) Internet Host#1 1 src H1 out S2 dst H1 out I S2 Replica#1 src H1 out R1 dst H1 out Replica#2 12

13 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) Controller load-balancer Host#1 Internet 1 2 src H1 out S2 dst H1 out I 3 read(s2, pkt) S2 src H1 out R1 dst H1 out Replica#1 Replica#2 13

14 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 What is the result of install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) read(s2, pkt)? Controller load-balancer Host#1 Internet src H1 out S2 dst H1 out I read(s2, pkt) S2 src H1 out R1 dst H1 out Replica#1 Replica#2 14

15 Load Balancer Application Did read(s2, pkt) if dst == server: rep = rep[idx] idx = (idx+1)%2 happened before or install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) after write(s2, match, out R1)? 2 Internet Controller load-balancer write(s2, match, out R1) read(s2, pkt) Host#1 1 src H1 out S2 dst H1 out I S2 Replica#1 src H1 out R1 dst H1 out Replica#2 15

16 Load Balancer Application What if read(s2, pkt) if dst == server: rep = rep[idx] idx = (idx+1)%2 happened before install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) write(s2, match, out R1)? Controller load-balancer Internet read(s2, pkt) Host#1 1 src H1 out S2 dst H1 out I S2 Replica#1 src H1 out R1 dst H1 out Replica#2 16

17 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) Controller load-balancer PktIn(s2, pkt) Internet Host#1 src H1 out S2 dst H1 out I S2 Replica#1 src H1 out R1 dst H1 out Replica#2 17

18 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) Select Replica #2 packet_out(pkt,in sw) Controller load-balancer PktIn(s2, pkt) Internet Host#1 src H1 out S2 dst H1 out I S2 Replica#1 src H1 out R1 dst H1 out Replica#2 18

19 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) Controller load-balancer write(s2, match, out ) Internet Host#1 src H1 out S2 dst H1 out I S2 src H1 out R1 dst H1 out src H1 out Replica#1 Replica#2 19

20 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) Controller load-balancer write(s3, match, out R2) Internet Host#1 src H1 out S2 dst H1 out I src H1 out S2 src H1 out R1 dst H1 out src H1 out Replica#1 Replica#2 src H1 out R2 20

21 Load Balancer Application if dst == server: rep = rep[idx] idx = (idx+1)%2 install_path(src, rep) install_path(rep, src) packet_out(pkt,in sw) Controller load-balancer write(s3, match, out R2) Internet Host#1 src H1 out S2 dst H1 out I src H1 out S2 src H1 out R1 dst H1 out src H1 out Replica#1 Indeterministic Loop! 21 Replica#2 src H1 out R2

22 What is the result of read(s2, pkt)??? 22

23 What is the result of read(s2, pkt)??? Did read(s2, pkt) happened before or after write(s2, match, out R1)? 23

24 What is the result of read(s2, pkt)??? Did read(s2, pkt) happened before or after write(s2, match, out R1)? To ensure correctness read(s2, pkt) must happen after write(s2, match, out R1). 24

25 What is the result of read(s2, pkt)??? Did read(s2, pkt) happened before or after write(s2, match, out R1)? To ensure correctness read(s2, pkt) must happen after write(s2, match, out R1). This is a real bug in Floodlight s Load balancer! 25

26 SDNRacer Detecting Concurrency Violations Filtering false-positives Checking for High level properties Implementation and Evaluation 26

27 SDNRacer Detecting Concurrency Violations Filtering false-positives Checking for High level properties Implementation and Evaluation 27

28 Detecting Concurrency Violations How to model the SDN network? How to capture asynchrony? 28

29 SDN Programs Flow tables are just memory locations. The controller writes and reads to/from memory. Packet reads memory. 29

30 Capturing Asynchrony Identifying the causality between events. 30

31 Happens-Before relation Defines partial ordering for events. a b reads event b happened before event a. HB relation is: Transitive Irreflexive Antisymmetric 31

32 Happens-Before relation A switch sends a PktIn message after reading it. read(s1, pkt) PktIn(s1, pkt) A controller may issue FlowMods after PktIn. PktIn(s1, pkt) write(s1, match, out S2) PktIn(s1, pkt) write(s2, match, out R1) 32

33 Load Balancer Application HB Relations: (1) (2) (2) (3) (2) (5) (2) (6) (2) (7) (2) (8) (8) (9) Host# Internet 1 Controller load-balancer S2 Replica# Replica#

34 Load Balancer Application HB Relations: (1) (2) (2) (3) (2) (5) (2) (6) (2) (7) (2) (8) (8) (9) Host#1 What about events (4) and (9)? Internet 1 Controller load-balancer S2 Replica#1 Replica#2 34

35 Real Example Floodlight Load Balancer Host#1 Replica#1 Host#2 Replica#2 35

36 Real Example Floodlight Load Balancer 30 seconds experiment. Host#1 Replica#1 Host#2 Replica#2 36

37 Real Example Floodlight Load Balancer 30 seconds experiment 822 Write event 476 Read event Host#1 Replica#1 Host#2 Replica#2 37

38 Real Example Floodlight Load Balancer 30 seconds experiment 822 Write event 476 Read event Host#1 Replica#1 703,864 Possible Races!!! Host#2 Replica#2 38

39 SDNRacer Detecting Concurrency Violations Filtering false-positives Checking for High level properties Implementation and Evaluation 39

40 Filtering False-Positives Identifying harmless race conditions. Identifying impossible race conditions due to physical limitations. 40

41 Harmful vs Harmless Non-Overlapping Events read(s1, dst=x) write(s1, dst=y, output port 1) write(s1, dst=z, output port 1) 41

42 Harmful vs Harmless Events with the same net effect write(s1, dst=x, output port 1) write(s1, dst=x, output port 1) 42

43 Real Example Floodlight Load Balancer 30 seconds experiment Host#1 Replica#1 822 Write event 476 Read event 703,864 Possible Races!!! 685,158 Commuting events Host#2 Replica#2 43

44 Real Example Floodlight Load Balancer 30 seconds experiment Host#1 Replica#1 822 Write event 476 Read event 703,864 Possible Races 685,158 Commuting events Host#2 Replica#2 What about the remaining 18,706 events??? 44

45 None feasible races write(s1, pkt, out replica 1) 45

46 None feasible races write(s1, pkt, out replica 1) 46

47 None feasible races write(s1, pkt, out replica 1) read(s1, pkt) 47

48 Real Example Floodlight Load Balancer 30 seconds experiment Host#1 Replica#1 822 Write event 476 Read event 703,864 Possible Races 685,158 Commuting events 16,492 Filtered by time 2214 (0.31%) Remaining 48 Host#2 Replica#2

49 SDNRacer Detecting Concurrency Violations Filtering False-Positives Checking for High level properties Implementation and Evaluation 49

50 Can we leverage low-level race detector to check for high-level properties? 50

51 Network Update A set of write events that together reflects highlevel network-wide policy change. Network updates are either reactive or proactive. 51

52 Network Update Isolation Network updates are isolated if they are serializable. 52

53 Network Update Isolation We check if there is no race between the two updates sets. 53

54 Real Bug: POX L2_Multi 54

55 Packet Coherence A packet is coherent if it is processed entirely using one consistent global network configuration. 55

56 Packet Coherence Check Take note if the first version the packet sees. Check at every switch if the read operations triggered by the packet races with write operations from different version than originally observed. 56

57 Real Bug: POX L2_Multi 57

58 SDNRacer Detecting Concurrency Violations Filtering False-Positives Checking for High level properties Implementation and Evaluation 58

59 Effect of Time-Based Filter Choosing smaller δ filters more races. 59

60 SDNRacer Filters More than 90% of all races are filtered in 89% of the cases 60

61 SDNRacer Runtime In 90% of the cases SDNRacer can analyze the traces in less than 30 seconds. 61

62 Conclusion SDNRacer can: 1. detect concurrency violation 2. detect violations of high-level network properties In short time. 62