Freeing the Whale How to Fail at Scale

Transcription

1 from QConSF, November 9, 2016 Freeing the Whale How to Fail at Scale oliver gould cto, buoyant

2 2010 A FAILWHALE ODYSSEY

3

4

5 Twitter, users 10 7 tweets/day 10 2 engineers 10 1 ops eng 10 1 services 10 1 deploys/week 10 2 hosts 0 datacenters 10 1 user-facing outages/week

6 objective reliability flexibility

7 objective solution reliability platform flexibility SOA + devops i.e. microservices

8 Resilience is an imperative: our software runs on the truly dismal computers we call datacenters. Besides being heinously complex they are unreliable and prone to operator error. Marius RPC Redux

9 software you didn t write hardware you can t touch network you can t trace break in new and surprising ways and your customers shouldn t notice

10 freeing the whale photo: Johanan Ottensooser

11 mesos.apache.org UC Berkeley, 2010 Twitter, 2011 Apache, 2012 Abstracts compute resources Promise: don t worry about the hosts

12 aurora.apache.org Twitter, 2011 Apache, 2013 Schedules processes on Mesos Promise: no more puppet, monit, etc

13 timelines users notifications x800 x300 x1000 Aurora (or Marathon, or ) Mesos host host host host host host

14 timelines users notifications x800 x300 x1000 Aurora (or Marathon, or ) Mesos host host host host host

15 service discovery timelines users create ephemeral /svc/users/node_ { host : host-abc, port : 4321} zookeeper

16 service discovery timelines users watch /svc/users/* zookeeper

17 service discovery GetUser(olix0r) timelines users zookeeper

18 service discovery GetUser(olix0r) timelines users uh oh. zookeeper

19 service discovery GetUser(olix0r) timelines users client caches results zookeeper

20 service discovery GetUser(olix0r) timelines users zookeeper serves empty results?! zookeeper

21 service discovery GetUser(olix0r) timelines users service discovery is advisory zookeeper

22 github.com/twitter/finagle RPC library (JVM) asynchronous built on Netty scala functional strongly typed first commit: Oct 2010

23 business [7] application languages, libraries [6] presentation json, protobuf, thrift, rpc [5] session http/2, mux, [4] transport kubernetes, mesos, swarm, [3] network datacenter canal, weave, [2] link aws, azure, digitalocean, gce, [1] physical

24 It s slow is the hardest problem you ll ever debug. Jeff Notes on Distributed Systems for Young Bloods

25 observability counters (e.g. client/users/failures) histograms (e.g. client/users/latency/p99) tracing

26 tracing

27 timeouts & retries web timeout=400ms retries=3 timelines timeout=400ms retries=2 users timeout=200ms retries=3 db

28 timeouts & retries web timeout=400ms retries=3 timelines timeout=400ms retries=2 800ms! users timeout=200ms retries=3 600ms! db

29 deadlines web timeout=400ms 77ms elapsed timelines deadline=323ms 113ms elapsed users deadline=210ms db

30 retries typical: retries=3

31 retries typical: retries=3 worst-case: 300% more load!!!

32 budgets typical: retries=3 worst-case: 300% more load!!! better: retrybudget=20% worst-case: 20% more load

33 load shedding via cancellation web timelines timeout! users db

34 load shedding via cancellation web timelines timeout! users db

35 backpressure web 1000 requests timelines 1000 requests users 100 requests db

36 backpressure web 1000 failed timelines 1000 failed users db

37 backpressure web 100 ok failed/redirected/etc timelines 100 ok users 100 ok db

38 request-level load balancing lb algorithms: round-robin fewest connections queue depth exponentially-weighted moving average (ewma) aperture

39

40 So just rewrite everything in Finagle!?

41 linkerd

42 github.com/buoyantio/linkerd service mesh proxy built on finagle & netty suuuuper pluggable http, thrift, etcd, consul, kubernetes, marathon, zookeeper,

43 Linkers and Loaders, John R. Levine, Academic Press

44 linker for the datacenter

45 logical naming applications refer to logical names /s/users requests are bound to concrete names /#/io.l5d.zk/prod/users /#/io.l5d.zk/staging/users delegations express routing /s => /#/io.l5d.zk/prod

46 per-request routing: staging GET / HTTP/1.1 Host: mysite.com l5d-dtab: /s/b => /s/b2

47 per-request routing: debug proxy GET / HTTP/1.1 Host: mysite.com l5d-dtab: /s/e => /s/p/s/e

48 linkerd service mesh transport security service discovery circuit breaking backpressure deadlines retries Service A instance Service B instance Service C instance tracing metrics keep-alive linkerd linkerd linkerd multiplexing load balancing per-request routing service-level objectives

49 demo: gob s microservice

50 web l5d gen l5d word l5d

51 web l5d gen l5d word l5d gen-v2 l5d

52 namerd web l5d gen l5d word l5d gen-v2 l5d

53 github.com/buoyantio/linkerd-examples

54 linkerd roadmap Battle test HTTP/2 TLS client certs Deadlines Dark Traffic All configurable everything

55 thanks! more at linkerd.io slack: slack.linkerd.io