Resilience, Service- Discovery and Z- D Deployment. Bodo Junglas, York Xylander

Transcription

1 Resilience, Service- Discovery and Z- D Deployment Bodo Junglas, York Xylander

2 Who is leanovate? people, HQ in Kreuzberg Mission: Build learning organizaions

3 What do we do? Dev Meth Doing Coaching ConsulIng Training Java,Scala, SPA,.. Kanban, Scrum, Lean*,..

4 Promises of µservices architectures Independent (parallel) development Independent release cycles Independent scaling

5 Challenges of µservices architectures Developing & Running Configuring Debugging Deploying Tough to learn & understand! Discovering Resilience

6 Microzon: A lab for µservices hyps://github.com/leanovate/microzon

7 Case Study: Microzon- Shop 7 DEMO Browse through catalog Select product Put into cart Checkout

8 Case Study: Microzon- Shop 8 Load-Balancer / Firewall Web Facade Customer Product Cart Billing mysql mongo mysql mysql

9 Technology- Babylon 9 Web Facade Play 2.3 Scala 2.11 Customer Product Cart Billing Spring-Boot JPA/Hibernate Spray 1.3 / Akka 2.3 Scala 2.10 ReactiveMongo Scalatra Jetty finatra 1.6 async-mysql finagle 6.20 Dropwizard 0.7 JDBI mysql mongo mysql mysql

10 Challenges: 10 Running: How long does it take to get a dev system up and running for a new team member? How to run your ci system it in your favorite cloud? Configuring Debugging

11 Case Study: Microzon- Shop 11 DEMO Start system with docker Create products Logstash Zipkin

12 Why is that a challenge? 12 Web Facade Customer Product Cart Billing mysql mongo mysql mysql

13 Why is that a challenge? 13 Web Facade Customer Product Cart Billing mysql mongo mysql mysql

14 Why is that a challenge? 14 Web Facade Customer Product Cart Billing mysql mongo mysql mysql Support UI Marketing tool Reporting Payment adapter

15 Why is that a challenge? 15 Web Facade Customer Product Cart Billing mysql mongo mysql mysql Support UI Marketing tool Reporting Payment?adapter

16 Takeaway Running/Configuring/Debugging will quickly become a non- trivial mayer We have chosen docker for development system puppet for»producion«elas.csearch/logstash/kibana for distributed logging zipkin for request tracing but that is not this focus of this talk

17 Challenge: Deploying 17 Zero-downtime deployment strategies/variants: Blue/Green, Wave, Canary, Service Version n Loadbalancer/ Router Service Version n+1 Deploy Delivery Pipeline Service Version n

18 Challenge: Service discovery 18 How to remove service nodes from the cluster or take them temporary offline? How to add new ones or take them online again? How do services find each other? => ServiceDiscovery

19 Service Discovery by configurafon 19 You (mis)use your configuraion management tool (puppet/chef) to generate service configuraion with explicit endpoints Node1 Node2 Node3 Config Management update add/remove node entry

20 Service Discovery by configurafon 20 Pros: Simply works No extra technology involved (and thereby no extra point of failure) Cons: To take a service node offline one has to update all of its consumers (or wait for them to be updated) All consumers have to be able to reload their configuraion without restart

21 Service Discovery by DNS 21 DNS is actually a service discovery 10.X.X.1 Product Service1 10.X.X.2 Product Service2 10.X.X.3 Product Service3 Customer Service {10.X.X.1, 10.X.X.2, 10.X.X.3} product.loc? DNS

22 DNS 22 Pros: Old technology that is proven to work on a very very large scale Supported by almost everyone Cons: Rather crude (and very inconvenient) interface (especially for updates) Resolved service names might be cached on muliple levels Focusses purely on service nodes, not on the services itself

23 Also: DNS might lead to wrong assumpfons 23 Apache HttpClient 4.3: org.apache.http.impl.conn.httpclientconnectionoperator public void connect(... ) throws IOException {... final InetAddress[] addresses = this.dnsresolver.resolve(host.gethostname());... for (int i = 0; i < addresses.length; i++) { final InetAddress address = addresses[i]; final boolean last = i == addresses.length - 1; } } Socket sock = sf.createsocket(context);... try { sock.setsotimeout(socketconfig.getsotimeout());... conn.bind(sock); return; } catch (final SocketTimeoutException ex) {... } catch (final ConnectException ex) {... }

24 Service discovery service 24 Zookeeper Curator Customer consul etcd doozerd Product Discovery Service Cart SkyDNS Eureka Billing

25 EvaluaFon criteria 25 - High availability - Consistency (consistency over availability?) - Support for service- level checks - API (hyp, DNS?, ) - Footprint (Memory/CPU) - MulIple datacenter support - UI - Template engine (for config files)

26 Consul 26 Consul server cluster consul agent service node A health check http / dns consul server consul server Raft consensus consul server consul template (optional) restart reload write service config files service node A consul server consul server consul agent health check http / dns restart reload service consul template (optional) write config files

27 Consul 27 Consul server cluster consul agent service node A health check http / dns consul server consul server Raft consensus consul server consul template (optional) restart reload write service config files service node A consul server consul server consul agent health check http / dns restart reload service consul template (optional) write config files

28 Consul 28 DEMO Show consul UI Show web- service status page Add cart/remove cart

29 Challenge: Resilience 29 Every system has a retry!

30 Do not do failover/retry on connecfon 30 GET ok DELETE ok PUT ok POST??? Duplicates might be ok (e.g. create new shopping cart) or not (e.g. register new customer) might be solved by a request token (e.g. the xsrf token from the web) as long as the service supports this GET really ok? What about streaming?

31 Failover should be part of the business 31 Usually the failover strategy depends on the concrete use- case Handling failover on the protocol layer (hyp- client) might hide error scenarios from the programmer It can be difficult to disinguish between technical and business error on the protocol layer As a rule of thumb: You want to retry all technical error, but not the business errors though even that is discussable in some cases

32 How»not«to do service failover 32 public class ServiceFailover { private static final Random RANDOM = new Random(System.currentTimeMillis()); public static <E, R> R retry(final List<E> endpoints, final Requester<E, R> requester) throws IOException { final int size = endpoints.size(); if (size == 0) throw new RuntimeException("No active endpoints found"); final int offset = RANDOM.nextInt(size); IOException lastexception = null; for (int idx = 0; idx < size; idx++) { final E endpoint = endpoints.get((idx + offset) % size); try { return requester.performtry(endpoint); } catch (IOException e) { lastexception = e; } } throw lastexception; public interface Requester<E, R> { R performtry(e endpoint) throws IOException; } }

33 Resilience 33 DEMO Kill 2 consul nodes Kill one cart node

34 Hystrix 34 Circuit- Breaking Fail- Early Developer s are»forced«to think in commands with potenial fallback result rather than REST- calls

35 How it actually should look like 35 according to finagle Observe Service Timeout Request Draining Load Balancer Monitor Trace Observe Failure Accrual Request Timeout Pool Fail Fast Expiration Dispatcher

36 What other people do 36 Nenlix Many libraries and tools that build up on top of each other RxJava/ReacIveX: ReacIve extension/reacive streaming Based on neyy: RxNeYy Hystrix: Basic command system for circuit- breaking/fail- early Eureka: Service registry Ribbon: REST- Client with failover/service discovery based on Hystrix/Eureka

37 What other people do 37 TwiYer Services based on finagle (scala) which is itself based on neyy which contains als the basics for for failover/retry/service- discovery/monitoring Service- Discovery done via zookeeper, but can be adapted/extended to other tools Several frameworks/connectors build on top: finatra, async- mysql- connector

38 Takeaway Service discovery Helps a lot to realize any kind of zero- downime deployment strategy a self- healing micro- service jungle Does not create a fully resilient system by itself, even though it is the basis of it Might conflict with your exising configuraion system (when creaing config files via templates) Might be just another central component that fails

39 Takeaway Failover/Retry The failover strategy usually depends on the business case A full failover stack is quite a piece of work Emerging frameworks might make life easier or at least provide a reference implementaion