100% Containers Powered Carpooling

Size: px

Start display at page:

Download "100% Containers Powered Carpooling"

Loraine McKinney
5 years ago
Views:

1 100% Containers Powered Carpooling

2 Maxime Fouilleul Database Reliability Engineer

3 BlaBlaCar - Facts & Figures Today s agenda Infrastructure Ecosystem - 100% containers powered carpooling Stateful Services into containers - MariaDB as an example Next challenges - Kubernetes, the Cloud

4 BlaBlaCar Facts & Figures

Facts and Figures 60 million members 30 million mobile app downloads Founded in 2006 15 million travellers /quarter 1 million tonnes less CO2 Currently in 22 countries In the past year iphone and

5 Facts and Figures 60 million members 30 million mobile app downloads Founded in million travellers /quarter 1 million tonnes less CO2 Currently in 22 countries In the past year iphone and Android France, Spain, UK, Italy, Poland, Hungary, Croatia, Serbia, Romania, Germany, Belgium, India, Mexico, The Netherlands, Luxembourg, Portugal, Ukraine, Czech Republic, Slovakia, Russia, Brazil and Turkey.

6 Our prod data ecosystem MariaDB ElasticSearch Redis Cassandra PostgreSQL Kafka Transactional Search Volatile Distributed Spatial Stream

7 Infrastructure Ecosystem 100% containers powered carpooling

8 Why containers?

9 Homogeneous Hardware From this srv_001 svc_001 srv_006 svc_006 srv_009 svc_009 srv_013 svc_013 srv_007 srv_002 svc_007 svc_002 srv_003 svc_003 srv_004 svc_004 srv_005 svc_005 srv_010 srv_008 svc_008 svc_010 srv_011 svc_011 srv_012 svc_012 srv_014 svc_014

10 Homogeneous Hardware To that srv_001 srv_003 svc_001 svc_008 svc_002 svc_003 svc_004 svc_005 svc_011 srv_002 svc_006 srv_004 svc_009 svc_007 svc_010 svc_012 srv_005 srv_007 svc_013 srv_006 svc_014 srv_008

11 Homogeneous Hardware - Pets vs Cattle Easier to replace broken hardware Cost Effective Easier to manage

12 Homogeneous Deployment trip-meeting-point application redis trip-meeting-point cat./prod-dc1/services/trip-meeting-point/service-manifest.yml --containers: - aci.blbl.cr/aci-trip-meeting-point: v-979da34 - aci.blbl.cr/aci-go-synapse: aci.blbl.cr/aci-go-nerve: aci.blbl.cr/aci-logshipper:27 cat./prod-dc1/services/redis-meeting-point/service-manifest.yml --containers: - aci.blbl.cr/aci-redis: aci.blbl.cr/aci-redis-dictator:20 - aci.blbl.cr/aci-go-nerve: aci.blbl.cr/aci-prometheus-redis-exporter: nodes: - hostname: trip-meeting-point1 gelf: level: INFO fleet: - MachineMetadata=rack=110 - Conflicts=*trip-meeting-point* - hostname: trip-meeting-point2 fleet: - MachineMetadata=rack=210 - Conflicts=*trip-meeting-point* - hostname: trip-meeting-point3 fleet: - MachineMetadata=rack=310 - Conflicts=*trip-meeting-point* nodes: - hostname: redis-meeting-point1 fleet: - MachineMetadata=rack=110 - Conflicts=*redis-meeting-point* ggn prod-dc1 trip-meeting-point update -y - hostname: redis-meeting-point2 fleet: - MachineMetadata=rack=210 - Conflicts=*redis-meeting-point* - hostname: redis-meeting-point3 fleet: - MachineMetadata=rack=310 - Conflicts=*redis-meeting-point* ggn prod-dc1 redis-meeting-point update -y

13 Volatile by design trip-meeting-point dependencies cat./prod-dc1/services/trip-meeting-point/service-manifest.yml --containers: - aci.blbl.cr/aci-trip-meeting-point: v-979da34 - aci.blbl.cr/aci-go-synapse: aci.blbl.cr/aci-go-nerve: aci.blbl.cr/aci-logshipper:27 trip-meeting-point aci-trip-meeting-point aci-go-synapse aci-go-nerve aci-logshipper [...] cat./aci-trip-meeting-point/aci-manifest.yml --name: aci.blbl.cr/aci-trip-meeting-point:{{.version}} aci: dependencies: - aci.blbl.cr/aci-java: [...] cat./aci-java/aci-manifest.yml --name: aci.blbl.cr/aci-java: aci: dependencies: - aci.blbl.cr/aci-debian: aci.blbl.cr/aci-common:7 aci-java aci-hindsight aci-debian aci-common

14 Volatile - When should I redeploy? A change in my own app/container: immutable A change on a sidecar container or its dependencies Noisy neighbours: mutualization When you are ready for instability your are HA

15 How?

16 Infrastructure Ecosystem front1 php nginx create Service Codebase dgr etcd Hardware mysql-main1 monitoring mysqld nerve monitoring synapse nerve zookeeper Service Discovery synapse store Container Registry Distributed init system build nerve fleet cluster ggn CoreOS rkt PODs run host bare-metal servers 1 type of hardware 3 disk profiles

Infrastructure Ecosystem front1 php nerve

dgr etcd Hardware mysqld nerve monitoring

store Container Registry Distributed init

CoreOS rkt PODs run host fleet kubernetes

17 Infrastructure Ecosystem front1 php nerve mysql-main1 nginx create Service Codebase dgr etcd Hardware mysqld nerve monitoring synapse nerve Service Discovery synapse store Container Registry Distributed init system build monitoring zookeeper ggn helm CoreOS rkt PODs run host fleet kubernetes bare-metal servers 1 type of hardware 3 disk profiles

haproxy if changes are detected go-nerve does health checks and reports to

18 Service Discovery client pod Zookeeper backend pod go-synapse go-nerve /database/node1 /database go-synapse watches zookeeper service keys and reloads haproxy if changes are detected go-nerve does health checks and reports to zookeeper in service keys HAProxy Applications hit their local haproxy to access backends node1

19 Stateful Services into containers MariaDB as an example

20 Stateful and volatile by design?

21 The recipe/prereqs/pillars to succeed: Abolish Slavery For a given service, every node have the same role Be Quiet! A node should be able to restart without impacting the app Build Smart Services can be operate by any SRE

22 MariaDB as an example

23 Abolish Slavery For a given service, every node have the same role

24 Asynchronous vs. Synchronous MariaDB Cluster means Master Slave Slave Slave No Single Point of Failure No Replication Lag Auto States Transfers wsrep wsrep wsrep MariaDB Cluster wsrep wsrep As fast as the slowest

25 The Target Writes MariaDB Cluster Reads Containers wsrep wsrep wsrep MariaDB Cluster wsrep wsrep Writes go on one node Reads are balanced on the others

26 How to hit the target? Service Discovery

27 Nerve - Track and report service status # cat env/prod-dc1/services/mysql-main/attributes/nerve.yml --override: nerve: services: - name: "mysql-main" port: 3306 reporters: - {type: zookeeper, path: /services/mysql/main} checks: - type: sql driver: mysql datasource: "local_mon:local_mon@tcp( :3306)/" # zookeepercli -c lsr /services/mysql/main mysql-main1_ _ba0f1f8b mysql-main2_ _734d63da mysql-main3_ _dde45787 # zookeepercli -c get /services/mysql/main/mysql-main1_ _ba0f1f8b3 { "available":true, "host":" ", "port":3306, "name":"mysql-main1", "weight":255, "labels":{ "host":"r10-srv4" } }

28 Synapse - Service discovery router # cat env/prod-dc1/services/tripsearch/attributes/synapse.yml --override: synapse: services: - name: mysql-main_read path: /services/mysql/main port: name: mysql-main_write path: /services/mysql/main port: 3308 serveroptions: backup serversort: date # cat env/prod-dc1/services/tripsearch/attributes/tripsearch.yml override: tripsearch: database: read: host: localhaproxy database: tripsearch user: tripsearch_rd port: 3307 write: host: localhaproxy database: tripsearch user: tripsearch_wr port: 3308

29 Be Quiet! A node should be able to restart without impacting the app

30 Nerve - Readiness Probe # cat env/prod-dc1/services/mysql-main/attributes/nerve.yml --override: nerve: services: - name: "mysql-main" port: 3306 reporters: - {type: zookeeper, path: /services/mysql/main} checks: - type: sql driver: mysql request: "SELECT 1" datasource: "local_mon:local_mon@tcp( :3306)/" mysql -h ulocal_mon -plocal_mon -p3306 -e SELECT 1; Starting Pod mysql-main1 Nerve check is KO Starting MySQL Nerve check is KO MySQL is syncing (IST/SST) Nerve check is KO MySQL is ready Nerve check is OK

31 Nerve - Grace Period # cat env/prod-dc1/services/mysql-main/attributes/nerve.yml --override: nerve: services: - name: "mysql-main" port: 3306 reporters: - {type: zookeeper, path: /services/mysql/main} checks: - type: sql driver: mysql datasource: "local_mon:local_mon@tcp( :3306)/" disablecommand: "/report_remaining_processes.sh" disablemaxdurationinmilli: Stop Pod Call /disable on Nerve s API Set weight to 0 = no more new sessions will go into the services. Wait The remaining sessions are finishing their job SELECT COUNT(1) FROM processlist WHERE user LIKE 'app_%'; Pod Stopped The service can be shutdown without risk.

32 Build Smart Services can be operate by any SRE

33 Example: Use Service Discovery to find peers

34 Use Service Discovery to find peers Eg: the wsrep_cluster_address attribute in Galera Cluster Description: The addresses of cluster nodes to connect to when starting up. Good practice is to specify all possible cluster nodes, in the form gcomm://<node1 or ip:port>,<node2 or ip2:port>,<node3 or ip3:port>. Specifying an empty ip (gcomm://) will cause the node to start a new cluster. d ery to fin v o c s i D ervice Ask the S l-main peers? mysq mysql-main No peernode2 found! node1, node1 node1 node2 node3 node1 node2 node3 wsrep_cluster_address = gcomm://node1,node2 gcomm:// gcomm://node1

35 Next challenges Kubernetes, the Cloud

36 Kubernetes, the Cloud, why now?

37 Kubernetes, the Cloud, why now? Fleet is deprecated Fleet is no longer developed and maintained by CoreOS. Kubernetes From a simple Distributed init system to the standard for container orchestration. Docker rkt-based implementation of Kubernetes has a poor adoption.

38 3 years old servers Kubernetes, the Cloud, why now? We need to renew our hardware. Google Kubernetes Engine & Managed Services Allows us to focus on services. Service Oriented Architecture Delegated Ownership.

39 Kubernetes and stateful services?

40 Kubernetes Statefulsets StatefulSets control Pods that are based on an identical spec. Stable, unique network identifiers. Stable, persistent storage. Ordered, graceful deployment, scaling and rolling updates.

41 Google Kubernetes Engine...

42 Why are we excited about GKE? Native suport of Liveness and Readiness probes This + resources limitations make powerfull orchestration Release granularity, from Pod to Deployment/Statefulset Native Service Discovery (kube-proxy and Services) GCEPersistentDisk provisioner to manage Persistent Volumes

43 See you next year for 100% GKE Powered Carpooling!

Building an on premise Kubernetes cluster DANNY TURNER

Building an on premise Kubernetes cluster DANNY TURNER Outline What is K8s? Why (not) run k8s? Why run our own cluster? Building what the public cloud provides 2 Kubernetes Open-Source Container Management