Anti-fragile Cloud Architectures. Agim Emruli - mimacom

Transcription

1 Anti-fragile Cloud Architectures Agim Emruli - mimacom

2 Antifragility is beyond resilience or robustness. The resilient resists shocks and stays the same; the antifragile gets better. Nasim Nicholas Taleb

3 Fragile Robust ANTI-Fragile Non-linear (Konkav) Post-traumatic Syndrom Centralized linear Non-linear (Konvex) Post-traumatic Growth Decentralized

4 Fragile Robust

5 Fragile Anti-Fragile

6 Demand Time

7 Scale cube Infinite Scale Modularize Partition Start Duplicate

8 Start

9 Duplication

10 Demand Time

11 Modularization

12 Demand Time

13 PArtioning North America Europe

14 Demand Time

15 {you Name it} Service

16 a microservices architecture IS a service-oriented architecture composed of loosely coupled elements that have bounded contexts

17 Therapy Dosing Therapy Patients Partner Conformist Dose Calculation product Management Shared Kernel Patient Hospital Product

18 Component Component Component Component

19 The network is reliable Latency is zero Bandwith is Infinite The network is secure Topology doesn t Change There is one Administrator Transport cost is zero The network is homogeneous

20 The network is reliable Latency is zero Bandwith is Infinite The network The network is secure Topology doesn t Change There is one Administrator Fallacies Transport cost is zero The network is homogeneous

21 Timeout

22 public class MessageReceiver { private Session session; private Destination destination; public void doreceive() throws Exception{ MessageConsumer consumer = session.createconsumer(destination); } } consumer.receive();

23 public class MessageReceiver { private Session session; private Destination destination; public void doreceive() throws Exception{ MessageConsumer consumer = session.createconsumer(destination); } } consumer.receive(20l);

24 public class HttpReceiver { public String getresource() throws IOException { URL url = new URL(" InputStream inputstream = url.openstream(); return ; } }

25 public class HttpReceiver { public String getresource() throws IOException { URL url = new URL(" URLConnection urlconn = url.openconnection(); urlconn.setconnecttimeout(10); urlconn.setreadtimeout(10); InputStream inputstream = url.getinputstream(); return ; } }

26 public class DataSourceConfig { public void DataSource setupdatasource(){ BasicDataSource basicdatasource = new BasicDataSource(); basicdatasource.setmaxwait(30l); } }

27 Add Latency tc qdisc add dev eth0 root latency delay 1000ms 500ms Corrupt Packages tc qdisc add dev eth0 root netem corrupt 5% Drop Packages tc qdisc add dev eth0 root netem loss 7% 25% Block DNS iptables -A INPUT -p tcp -m tcp --dport 53 -j DROP

28 Patterns Stability Capacity Transparency

29 37% Internet Traffic

30 XD BOOT GRAILS CLOUD IO EXECUTION STREAMS, TAPS, JOBS FULL STACK, WEB SERVICE REGISTRY, CIRCUIT BREAKER, METRICS IO COORDINATION BOOTABLE, MINIMAL, OPS-READY INTEGRATION BATCH BIG DATA WEB IO FOUNDATION CHANNELS, FILTERS, ADAPTERS, TRANSFORMERS DATA JOBS, STEPS, READERS, WRITERS RELATIONAL DATA INGESTION, EXPORT, ORCHESTRATION, HADOOP NON-RELATIONAL DATA CONTROLLERS, REST, WEBSOCKET CORE FRAMEWORK SECURITY GROOVY REACTOR

31 Application Tomcat (Jetty, Undertow) Actuator java -jar myapplication.jar Data Source Java Runtime Environment

32 Circuit Breaker

33 Fragile Robust ANTI-Fragile No Timeout Timeout CIRCUIT-BREAKER

34 Execute Command Run Fallback

35 Execute Command Circuit Open? Close Circuit Run Fallback

36 Thread - 1 Thread - 2 Thread - 3 Thread - 4 Thread - 5 Thread - 6 Thread - 7 Thread - 8 Tomcat Thread Pool Service Service XService Service

37 COMMAND THREAD COMMAND THREAD COMMAND THREAD

38 COMMAND THREAD TRACE --- [http-nio-auto-1-exec-10] COMMAND outside command THREAD TRACE --- [hystrix-restcurrencyexchange-10] inside command Thread locals

39 @SpringCloudApplication public class SearchGateway = "fallback") public List<SearchHit> search(string query) { return ; } public List<SearchHit> fallback() { return Collections.emptyList(); } }

40

41 Service Registry

42 Fragile Robust ANTI-Fragile Point-To-Point Service-Registry

43 Client Service Service Service v1.0 Service v1.1 Service

44 Eureka Consul Zookeeper Availability Partitioning Consistency Availability Consistency Availability

45 Availability Zookeeper Eureka Time

46 PArtioning North America Europe Eureka Eureka

47 Discovery

48 Client Discovery DNS Service Registry Client Consul Load Balance Service Instance Service Instance Service Instance Kubernetes Execution Environment

49 Client Discovery Api Discovery Client Service Registry Client Load Balance Service Instance Service Instance Service Instance Execution Environment

50 Server Discovery Service Registry Client Proxy Sidecar Execution Environment Load Balance Service Instance Service Instance Service Instance

51 Server Discovery Service Registry Client Request ZUUL Edge Gateway Execution Environment Load Balance Service Instance Service Instance Service Instance

52 Load Balancing Client Client Client X Service Service Service Service Service 2.3 Service 0.7 Round Robin Availability Filtering Weighted Response Time LINEAR NON-LInear NON-LInear

53 Add Latency tc qdisc add dev eth0 root latency delay 1000ms 500ms Corrupt Packages tc qdisc add dev eth0 root netem corrupt 5% Drop Packages tc qdisc add dev eth0 root netem loss 7% 25% Block DNS iptables -A INPUT -p tcp -m tcp --dport 53 -j DROP Simulate heavy IO while true; do dd if=/dev/urandom of=/burn bs=1m count=1024 iflag=fullblock done Burn CPU while true; do openssl speed; done

54 API Gateway

55 Resource Client Client Resource Resource Client API Gateway Client Client Resource Client (C OR R1 OR R2 OR R3 OR R4) (C OR A) AND (R1 OR R2 OR R3 OR R4)

56 Running Distributed Architectures

57 Load Performance Stress

58 Robust At 12:24 PM Pacific Time on December 24 network traffic stopped on a few ELBs.. At around 3:30 PM on December 24, network traffic stopped on additional ELBs..Netflix is designed to handle failure of all or part of a single availability zone in a region as we run across three zones and operate with no loss of functionality on two. We are working on ways of extending our resiliency to handle partial or complete regional outages.

59

60 Impact Time Hormesis

61

62 Anti-fragile On Sunday, at 2:19am PDT, there was a brief network disruption that impacted.. So, when the network disruption occurred on Sunday morning, and a number of storage servers simultaneously requested their membership data,.. By 2:37am PDT, the error rate in customer requests to DynamoDB had risen far beyond any level experienced in the last 3 years After several failed attempts at adding capacity, at 5:06am PDT, we decided to pause requests to the metadata service.

63 Anti-fragile Despite being run entirely from AWS' cloud platform the online streaming giant Netflix reports a quick recovery from Sunday's disruption - demonstrating the importance of its approach of building cloud-based systems to "fail".

64 Cassandra AWS:REBOOT Nodes 218 Rebooted 22 Dead

65 Thanks Agim Emruli -