The Fallacies of Distributed Computing: What if the Network Fails?

Transcription

1 The Fallacies of Distributed Computing: What if the Network Fails? Bert Ertman and Willem Dekker Those who stand for nothing, fall for anything - Alexander Hamilton

2 About us Bert Ertman Fellow Willem Dekker Architect @BertErtman

3 Introduction In 1994 Peter Deutsch drafted 7 assumptions architects and designers of distributed systems are likely to make, which prove wrong in the long run In 1997 James Gosling added another fallacy

4 The 8 Fallacies 1. The network is reliable 2. Latency is zero 3. Bandwidth is infinite 4. The network is secure 5. Topology doesn't change 6. There is one administrator 7. Transport cost is zero 8. The network is homogeneous

5 Freely translated, the paper says.

6 The Network hates us!

7 The Cloud hates us too!

8 Latency control

9 Fault tolerance

10 Fault tolerance It s-all-your-fault tolerance

11 Resilience The ability of a system to handle unexpected situations at all times - without the user noticing it (best case) - with graceful degradation of service (worst case)

12 Case

13 Case introduction Liquid Sunshine is a modern web shop selling well liquid sunshine a.k.a. Whisky We sell great products, aim for customer intimacy, and offer great service Our architecture looks like this

14 web application database

15 web application web application web application database

16 web application web application web application database

17 web application web application web application database

18 But suddenly we realised OMG, we ve built a MONOLITH!

19 At that point we switched to Microservices because well just because! and Amazon does it too! :-P

20 Shop UI Search Special Offers Products Order Auth Accounts Stock Billing Shipping Returns Care Center

21 And we were all happy

22

23 Loading Please Wait

24

25 bad response times

26 Shop UI Search Special Offers Products Order Auth Accounts Stock Billing Shipping Returns Care Center

27 Shop UI Special Offers Purchase history Cascading failures

28 Web Application thread pool

29 Web Application thread pool Special Offers

30 Web Application thread pool Special Offers

31 Web Application thread pool Special Offers

32 Web Application thread pool Special Offers Products Order

33 Web Application thread pool Special Offers Products Order

34 Web Application thread pool Special Offers Products Order

35 Problem public class SpecialOffersClient { private final URL endpoint; public Offers getoffers() throws IOException { HttpURLConnection connection = (HttpURLConnection) endpoint.openconnection(); connection.connect(); } InputStream inputstream = connection.getinputstream(); return readtoobject(inputstream);

36 What could possibly go wrong? Well, actually a lot! - connection might take a while to establish - the target service might be slow - the network might be slow or experience hiccups - a response might comprise of multiple packets

37 Dealing with the problem HttpURLConnection connection = (HttpURLConnection) endpoint.openconnection(); connection.setconnecttimeout(300); connection.setreadtimeout(1000); connection.connect(); InputStream inputstream = connection.getinputstream(); Offers result = readtoobject(inputstream);

38 Solution Use timeouts to prevent blocked threads Set aggressive timeouts Apply to every call here here too! Shop UI Special Offers Purchase history

39 And we were all happy

40

41 Loading Please Wait

42

43 bad response times awful throughput

44 Shop UI Search Special Offers Products Order Auth Accounts Stock Billing Shipping Returns Care Center

45 Shop UI Special Offers Timeouts

46 Web Application thread pool timeout Special Offers

47 Web Application thread pool Special Offers many timeouts

48 Throughput < incoming requests Our response times degraded to 5 rps because of timeouts But we had > 10 rps incoming requests

49 Web Application thread pool Special Offers many timeouts

50 Problem Someone had moved the special deals section to the template that is included for every page on the site Throughput of the Special Offers service became lower than the number of incoming requests Result: many timeouts, blocked threads, and queues causing our site to be unreachable

51 Solution For frequently called services, timeouts alone are not enough! You have to make sure that calls to broken services fail fast Guard services with a Circuit Breaker

52 Circuit Breaker try 1 exception/timeout try 1 exception/timeout circuit is CLOSED Initiating Service try 2 exception/timeout try 3 exception/timeout Circuit Breaker try 2 exception/timeout Target Service circuit trips and is OPEN try 83 exception/timeout try 3 exception/timeout circuit is HALF-OPEN

53 Circuit Breaker example public class GetSpecialOffersCommand extends HystrixCommand<Offers> { private final SpecialOffersClient client; public GetSpecialOffersCommand(SpecialOffersClient client) { super(factory.askey("special-offers")); this.client = client; protected Offers run() throws Exception { return client.getspecialoffers(); } } GetSpecialOffersCommand command = new GetSpecialOffersCommand(client); Offers result = command.execute();

54 Solution (revisited) With a Circuit Breaker - calls to broken services fail fast - broken services will be offloaded as well

55 And we were all happy

56

57 Loading Please Wait

58

59 bad response times awful throughput again?!!

60 Shop UI Search Special Offers Products Order Auth Accounts Stock Billing Shipping Returns Care Center

61 Shop UI Special Offers Slow response

62 Web Application thread pool Special Offers slow responses

63 Web Application thread pool Special Offers throughput < incoming requests (again) slow responses

64 Problem Timeouts and Circuit Breakers are not enough if response time < timeout and a sudden increase in load occurs Somehow we need another means of isolating services to prevent cascading failures

65 Solution Introduce Bulkheads Limit the number of concurrent calls to a service Set an upper bound on the number of waiting threads

66 Bulkheads In shipping: partition the load into sections allowing you to seal them off if there is a breach In software: isolate services to prevent cascading failures to cripple the entire system

67 Web Application thread pool bulkhead (size=3) Special Offers

68 Web Application thread pool bulkhead (size=3) Special Offers

69 Web Application thread pool bulkhead (size=3) Special Offers

70 Web Application thread pool bulkhead (size=3) slow response with special offers Special Offers fast response no special offers

71 Bulkheads for each service Shop UI BH BH BH BH BH Search Special Offers Products Order Auth

72 Tuning bulkheads As a rule of thumb: - Determine peak load when healthy 25 rps w/ 200 ms response time 25 x 0.2 = 5 add a bit of breathing room (+2) = 7

73 Bulkheads for each service Shop UI B=5 B=7 B=10 B=4 B=2 Search Special Offers Products Order Auth 28

74 Bulkhead example Semaphore semaphore = new Semaphore(3); Offers result; if (semaphore.tryacquire()) { result = client.getspecialoffers() semaphore.release(); } else { result = Offers.EMPTY; } return result;

75 And we were all happy

76

77 Loading Please Wait

78

79 All service calls are being rejected!

80 But how can this be? The site had a sudden increase in load But we had all our measures in place: - Bulkheads - Circuit Breakers - Timeouts What happened?

81 Shop UI B=5 B=7 B=10 B=4 B=2 Search Special Offers Products Order Auth

82 Problem It turned out we d upgraded our client library and this messed up our settings - timeouts were gone Client libraries are mostly black box - implementation changes can cause disruptions

83 Solution Introduce thread pool handovers - hand over the service call to another thread pool so the initial caller can walk away after timeout

84 Web Application request thread pool service thread pool Service

85 request thread pool Web Application timeout service thread pool Service

86 Web Application request thread pool error service thread pool timeouts are still required! X X X Service

87 Web Application request thread pool service thread pool Bulkhead included Service X X error

88 Solution (revisited) Thread pool handovers are a powerful construct - generic timeouts - bonus bulkhead included when fixing the thread pool size - client library issues are confined to just a single service thread pool

89 ThreadHandOver example ExecutorService executor = new ThreadPoolExecutor(3, 3, ); try { Future<Offers> future = executor.submit(specialoffersclient::getoffers); return future.get(1, TimeUnit.SECONDS); } catch (RejectedExecutionException TimeoutException e) { return Offers.EMPTY; }

90 Hystrix example GetSpecialOffersCommand command = Future<Offers> future = command.queue(); return future.get(); public class GetSpecialOffersCommand extends HystrixCommand<Offers> protected Offers getfallback() { return Offers.EMPTY; }

91

92 Monitoring Monitor all service calls - remote dependencies == integration points Monitor stability measures (problem detector) - errors, timeouts, waits - validate configuration

93 Monitoring TODO - example screenshot van Hystrix monitoring?

94 Monitoring TODO - example screenshot van Hystrix monitoring?

95 Testing Focus on client demand rather than supplier s definition - BDD-style testing Lots of stuff happens on the wire - Wiremock, Saboteur, Simian Army, Chaos Monkey, etc - JMeter, Gatling, Jenkins Performance Plugin

96 Moral of the story Now this case was just to show you what could go wrong with just a very insignificant service and some distribution. - we focused on client side protection, but there are other patterns that can be applied e.g. on the service-side Image what could go wrong on a bigger scale

97 Thanks!