Toward Intrusion Tolerant Cloud Infrastructure

Transcription

1 Toward Intrusion Tolerant Cloud Infrastructure Daniel Obenshain, Tom Tantillo, Yair Amir Department of Computer Science Johns Hopkins University Andrew Newell, Cristina Nita-Rotaru Department of Computer Science Purdue University 1

2 DSN03: The Overlay Paradigm Overlay paradigm: In contrast to keep it simple in the middle and smart at the edge Move intelligence and resources to the middle Software-based overlay routers working on top of the internet Overlay links translated to Internet paths Smaller overlay scale (# nodes) è smarter algorithms, better performance, and new services. 2

3 DSN03: Hop-by-Hop Reliability 50 millisecond network, five hops 10 milliseconds to tell node DAL about the loss 10 milliseconds to get the packet back from DAL Only 20 milliseconds to recover a lost packet Lost packet sent twice only on link DAL ATL In contrast to at least 100 milliseconds on the Internet 56 5 LAX PHX DAL ATL DCA BWI 5 3

4 DSN03: Average Latency and Jitter Simulation Spines on Emulab TCP End-to-end Hop-by-hop Linux TCP End-to-end Spines Hop-by-hop Latency Simulation Average delay (ms) Emulab Average delay (ms) Loss rate (%) Loss rate (%) TCP End-to-end Hop-by-hop Linux TCP End-to-end Spines Hop-by-hop Jitter Simulation Jitter (ms) Loss rate (%) Emulab Jitter (ms) Loss rate (%) 4

5 The Spines Platform [DSN03, NOSSDAV05, TOM06, Mobisys06, TOCS10] Daemons create an overlay network on the fly Clients are identified by the IP address of their daemon and a port ID Clients feel they are working with UDP and TCP using their IP and port identifiers Protocols designed to support up to 1000 daemons (locations), each daemon can handle up to about 1000 clients 5

6 Outline From overlay networks to clouds Going back in time it all started with a DSN03 paper From research to practice lessons learned Toward intrusion-tolerant cloud infrastructure Intrusion-tolerant cloud monitoring and control Monitoring: Priority-based flooding with source fairness Control: Reliable flooding with source destination fairness Intelligent use of diversity to increase resiliency The Diversity Assignment Problem (DAP) Optimal assignments on a cloud Naïve assignments can hurt Application patterns matter Summary 6

7 : The Siemens VoIP Challenge Can we maintain a good enough phone call quality over the Internet? High quality calls demand predictable performance VoIP is interactive. Humans perceive delays at 100ms The best-effort service offered by the Internet was not designed to offer any quality guarantees Communication subject to dynamic loss, delay, jitter, path failures PESQ - Average Normal 25% burst 50% burst 75% burst PSTN PESQ - 5 percentile Normal 25% burst 50% burst 75% burst Loss rate (%) Loss rate (%) 50ms network delay 7

8 VoIP Quality Improvement PESQ - Average Spines Normal Spines 25% Spines 50% Spines 75% UDP Normal UDP 75% PESQ - 5 percentile Spines Normal Spines 25% Spines 50% Spines 75% UDP Normal UDP 75% Loss rate (%) Loss rate (%) Spines overlay 5 links of 10ms each 10 VoIP streams sending in parallel Loss on middle link C-D 10ms 10ms 10ms 10ms 10ms A 10Mbps B 10Mbps C 10Mbps D 10Mbps E 10Mbps F 50ms network delay 8

9 The Spines Architecture (2006) Reliable Session Daemon-Client Interface Session Application Client API Library Routing Overlay Node Simple Routing Forwarder Link State Hello Protocol Group State State Flood Best Effort Data Link Real-time Data Link Control Link Reliable Data Link Overlay Link Datalink (UDP/IP unicast) 9

10 2008-Present: The LiveTimeNet Cloud 10

11 Almost Reliable, Real-time Protocol for Live TV Network packet loss on one link (assuming 66% burstiness) Loss experienced by flows on the LTN Network 2% < % 5% < 0.003% 10% < 0.03% 11

12 Zooming on a Single Channel 12

13 From Research to Practice 13

14 Toward Intrusion Tolerant Clouds Main premise: The large gap in constructing resilient clouds is the vulnerability to intrusions No good algorithms to construct distributed messaging systems and consistent global state at cloud scale, guaranteeing their integrity and performance even under intrusion attacks Main goal: Invent, develop and transition the overlay messaging and replication tools necessary to make public and private clouds resilient to sophisticated intrusion attacks 14

15 Outline From overlay networks to clouds Going back in time it all started with a DSN03 paper From research to practice lessons learned Toward intrusion-tolerant cloud infrastructure Intrusion-tolerant cloud monitoring and control Monitoring: Priority-based flooding with source fairness Control: Reliable flooding with source destination fairness Intelligent use of diversity to increase resiliency The Diversity Assignment Problem (DAP) Optimal assignments on a cloud Naïve assignments can hurt Application patterns matter Summary 15

16 Intrusion-Tolerant Cloud Monitoring and Control Cloud infrastructure is remote to its administrators Cloud management must be done through monitoring and control messaging The chicken and the egg at least part of the cloud software has to work to allow its administrators to make it work (e.g. react to attacks) Result: Monitoring and control messaging has to be intrusion-tolerant Instances of the messaging service may reside on compromised nodes But: no practical intrusion-tolerant messaging that can perform well in cloud environments 16

17 Controlled Authenticated Flooding Uses overlay topology and authentication to limit the power of incorrect (compromised) nodes Floods messages at most twice on each overlay link Optimal intrusion tolerance Optimal latency High cost up to times higher than secure link-state overlay routing on relevant topologies Two protocols with differing properties and guarantees: Monitoring: priority-based flooding with source fairness Control: reliable flooding with source-destination fairness 17

18 Priority-based Flooding with Source Fairness The Need: Motivated by the demands of a monitoring system in a cloud infrastructure Distributing continuous streams of messages across a network that may contain compromised nodes Any node in the network can be a source Some messages are more critical than others Delivery should be timely If resources are constrained, critical information must pass 18

19 Priority-based Flooding with Source Fairness Main ideas: Each overlay link schedules the next outgoing message by selecting a source with messages contending for this link based on a fair round-robin scheme The highest priority message from the selected source is sent first Memory buffer space and outgoing bandwidth is allocated on each overlay link based on a source-fair scheme When memory is exhausted, the lowest priority message of the source with the most messages in the overlay link buffers is dropped first 19

20 Source Fairness for Malicious Environments Example Source A is sending at 10 Mbps, Source B at 50 Mbps, Source C at 60 Mbps, and link s capacity is 100 Mbps Source A gets all 10 Mbps Source B gets 45 out of the 50 Mbps it wants Source C gets 45 out of the 60 Mbps it wants + + = + 20

21 Priority-based Flooding with Source Fairness Flag Ptr 5 source ngbr 1 NEED 12 seq_num ngbr 2 ACK 11:59 am orig_time One at each Node ngbr D SEND 12:00 pm 2:15 pm... arrival expire *neighbor status packet *msg 2 need_count Storage for Sources Messages Source ID Selection Queue Message Priority Queue head tail tail head MAX PRIORITY Source ID One for each Neighboring Link urgent queue head tail normal queue Priority Level

22 Priority-based Flooding with Source Fairness Intrusion-Tolerant Point-to-Point Reliable Link Protocol Compromised nodes can lie to cause good nodes to run fast Use up good system resources (processing, bandwidth) Intrusion-Tolerant Link Protocol ensures nodes can t ACK packets they never received We include a nonce on each packet that must be included in the corresponding ACK A ACK dishonestly Data B 22

23 Priority-based Flooding Validation on a Cloud U.S.-Wide Cloud Topology NYC DEN CHI LAX JHU DAL ATL 23

24 Reliable Flooding with Source-Destination Fairness The Need: Motivated by the demands of a control system in a cloud infrastructure Any node in the network can be a source All messages are critical because they change the state of the cloud Messages must be delivered reliably Delivery should be timely 24

25 The Problem of Source-based Fairness in Reliable Communication If we used source based fairness, a malicious destination could block a good source C A B A sends to C and D, via B D is malicious and refuses to acknowledge packets A cannot make progress with either C or D (because it s a reliable protocol) D 25

26 Source-Destination Fairness Instead, treat each source-destination flow separately. C A B D The A-D flow becomes blocked The A-C flow does not 26

27 Reliable Flooding with Source-Destination Fairness Main ideas: A correct node maintains a message until either of the following conditions is met: 1) All of its direct neighbors have it 2) An end-to-end acknowledgement is received Memory buffer space and outgoing bandwidth are allocated on each overlay link based on a sourcedestination pair fair scheme, so that bad destinations cannot block other flows by not acknowledging messages Back pressure is employed all the way to the source if a source-destination pair exhausts its memory buffer 27

28 Reliable Flooding with Source-Destination Fairness Per-Flow Buffers One at each Node for each Destination Up-to-Date End-to-End ACK 1 N Flow ID Selection Queue head tail Flow ARU Matrix Destination ID Flow Start of Window Matrix Destination ID One for each Neighboring Link urgent queue Source ID Source ID head tail normal queue 28

29 The Spines Architecture Reliable Session Daemon-Client Interface Session Application Client API Library Routing Overlay Node Simple Routing Forwarder Link State Hello Protocol Group State State Flood Best Effort Data Link Real-time Data Link Control Link Reliable Data Link Overlay Link Datalink (UDP/IP unicast) 29

30 The New Spines Architecture Reliable Session Daemon-Client Interface Session Application Client API Library Routing Overlay Node Simple Routing Forwarder Link State Hello Protocol Group State State Flood Intrusion- Tolerant Priority Flooding Intrusion- Tolerant Reliable Flooding Best Effort Data Link Real-time Data Link Control Link Reliable Data Link Intrusion Tolerant Data Link Overlay Link Datalink (UDP/IP unicast) 30

31 Outline From overlay networks to clouds Going back in time it all started with a DSN03 paper From research to practice lessons learned Toward intrusion-tolerant cloud infrastructure Intrusion-tolerant cloud monitoring and control Monitoring: Priority-based flooding with source fairness Control: Reliable flooding with source destination fairness Intelligent use of diversity to increase resiliency The Diversity Assignment Problem (DAP) Optimal assignments on a cloud Naïve assignments can hurt Application patterns matter Summary 31

32 The Diversity Assignment Problem (DAP) l l l l l Undirected graph of client and network nodes Variants are compromised independently with a certain probability over a period of time If a variant is compromised, all nodes of that variant are assumed compromised Goodness is generally measured by overall client-to-client expected connectivity Find the diversity assignment that maximizes goodness NP-Hard problem" 32

33 Diversity Assignment Example Red: 90% resilience Blue: 85% resilience l Expected connectivity: 83.8% l Expected connectivity: 95.7% How one places variants in the network matters" 33

34 Diversity Assignment on a Cloud Topology l l Topology based on how the LiveTimeNet guys build their global cloud - 20 overlay nodes (datacenters) - 10 clients randomly placed - Up to 4 variants with differing resilient properties Model - (Red 90%) (Blue 85%) (Green 80%) (Yellow 75%) - Variants are compromised independently with a certain probability over a period of time - If a variant is compromised, all nodes of that variant are assumed compromised - Goodness is generally measured by overall client-to-client expected connectivity 34

35 Baseline: No Diversity l Without diversity just pick the most resilient variant l Variant resilience: (Red 90%) l Expected connectivity: 90% 35

36 Diversity with Two Variants l Variant resilience: (Red 90%) (Blue 85%) l Expected connectivity: 98.5% Inclusion of a more vulnerable variant results in higher resilience 36

37 Diversity with Three Variants l Variant resilience: (Red 90%) (Blue 85%) (Green 80%) l Expected connectivity: 99.7% l Even higher expected connectivity with another, even weaker variant 37

38 Naïve Assignments can Hurt l 100,000 random assignments with 3 variants - Many random assignments have less than 90% expected connectivity (worse than without diversity) - Best value: 98.8% - Expected dis-connectivity = (1 expected connectivity) - Best random is 4x worse in terms of expected dis-connectivity: - 1.2% vs 0.3% 38

39 Diversity with Four Variants l Resilience: (Red 90%) (Blue 85%) (Green 80%) (Yellow 75%) l Expected client connectivity: 99.75% 39

40 Application-Specific Diversity A BFT Test Case l l Run BFT on top of this diversified network - The state machine replicas are the clients from the point of view of the network - Rather than maximize any-to-any connectivity, we maximize the chance that BFT can make progress - BFT needs a connected component of at least 2f+1 good replicas out of a total of 3f+1 replicas Maximize the probability that a connected component of correct replicas exists 40

41 Diversity with Four Variants Fault model: 1 Byzantine replica 2 partitioned replicas l Resilience: (Red 90%) (Blue 85%) (Green 80%) (Yellow 75%) l Expected client connectivity: 99.75% l Probability of BFT progress: 99.7% 41

42 BFT-Specific Diversity Fault model: 1 Byzantine replica 2 partitioned replicas l Resilience: (Red 90%) (Blue 85%) (Green 80%) (Yellow 75%) l Expected client connectivity: 98.06% l Probability of BFT progress: % 42

43 Outline From overlay networks to clouds Going back in time it all started with a DSN03 paper From research to practice lessons learned Toward intrusion-tolerant cloud infrastructure Intrusion-tolerant cloud monitoring and control Monitoring: Priority-based flooding with source fairness Control: Reliable flooding with source destination fairness Intelligent use of diversity to increase resiliency The Diversity Assignment Problem (DAP) Optimal assignments on a cloud Naïve assignments can hurt Application patterns matter Summary 43