APACHE COTTON. MySQL on Mesos. Yan Xu xujyan

Size: px

Start display at page:

Download "APACHE COTTON. MySQL on Mesos. Yan Xu xujyan"

Kory Mathews
6 years ago
Views:

1 APACHE COTTON MySQL on Mesos Yan Xu xujyan 1

2 SHORT HISTORY Mesos: cornerstone of Twitter s compute platform. MySQL: backbone of Twitter s data platform. Mysos: started as a hackweek Apache Cotton: fluffy, elastic & cloudy. Call for collaboration / contribution! 2

3 WHY MYSQL ON MESOS From manual MySQL administration to selfservice. Distributed systems should manage themselves. 3

4 WHY COTTON Reusability vs. flexibility. Generic schedulers: stateless tier one services. MySQL: stateful & complex. Interactive: Cotton coordinates MySQL instances during their lifecycles. Push common functionality down! 4

5 COTTON INTRO 5

6 MYSQL ON A MESOS CLUSTER Services S S S S S S S frameworks mesos-master S S mesos-slaves 6

7 MYSQL ON A MESOS CLUSTER Services S S S S S S S frameworks cotton-scheduler mesos-master 6 S S mesos-slaves

8 MYSQL ON A MESOS CLUSTER Services S S S S S S S frameworks cotton-scheduler mesos-master 6 S S mesos-slaves cotton-executor

9 MYSQL ON A MESOS CLUSTER Services S S S S S S S frameworks mesos-master 6 S S mesos-slaves cotton-executor

10 MYSQL ON A MESOS CLUSTER Services S S S S S S S frameworks mesos-master 6 S S mesos-slaves cotton-executor

11 COTTON CLUSTERS MySQL cluster Cotton cluster Cotton cluster Mesos cluster MySQL cluster MySQL cluster 7

12 SCHEDULER + EXECUTOR Scheduler: global view, central scheduling, react to events (from Mesos and executors), coordinate instances (e.g. Elector). Scheduler deploy, upgrade, HA, service discovery, : managed by Aurora, state stored in ZK. Executor: Bootstrap, launch, monitor, interact. 8

13 CREATE CLUSTER cluster0 : nodes=3, size=small, user, passwd, backup_id Cotton Scheduler Offers mesos-master mesos-slaves 9

14 Cotton Scheduler LAUNCH 3 TASKs LAUNCH TASK A LAUNCH TASK B LAUNCH TASK C mesos-master mesos-slaves 10

15 RUNNING A Cotton Scheduler 3 TASKs RUNNING RUNNING B RUNNING C mesos-master mesos-slaves 11

16 GTID_EXECUTED? A Cotton Scheduler B C mesos-master mesos-slaves 12

17 Master = Slave(MAX(GTIDs)) GTIDs A Cotton Scheduler B C mesos-master mesos-slaves 13

18 ELECT A A elected A Cotton Scheduler B C mesos-master mesos-slaves 14

19 Master: A Slaves: B, C zk:// A PROMOTE Cotton Scheduler B REPARENT C REPARENT mesos-master mesos-slaves 15

20 ELECT B Instance A unhealthy A Cotton Scheduler Host C down C B mesos-master mesos-slaves 16

21 A Cotton Scheduler LAUNCH TASKS D,E B C mesos-master mesos-slaves 17

22 SERVICE DISCOVERY MySQL connection: ip, port, user, password API input: cluster name, user(, password) API out: (immediate) ZooKeeper path(, password) Cotton publishes to ZK: ip:port master/slave roles 18

23 DISCOVERY MECHANISMS ServerSets w/ ZK libraries Proxying (haproxy-marathon-bridge) DNS (mesos-dns) Scheduler REST API Mesos commons library 19

24 MONITORING Watching ZK group: When is the MySQL cluster ready? (e.g. in an hour) When has old master died & a new master elected? Monitoring stats: Is the cluster healthy? Is Cotton service healthy? <scheduler>/vars.json Mesos exported stats (container stats, <slave>/monitor/statistics.json) Executor exported stats (MySQL specific) 20

25 USING COTTON 21

26 CUSTOMIZING COTTON Organization level (e.g. Twitter) Merge custom scripts and configs. Implement Installer, BackupStore interfaces. Cotton cluster level (e.g. Devel): scheduler flags. MySQL cluster level (e.g. Ads): API request arguments. 22

27 BACKUP STORE Discover, fetch, decrypt, extract, post-processing, etc. Customization: TwitterBackupStore Flags: --backup_store_args. (e.g. HDFS args) API args: backup_id. Organization specific: group, cluster, partition, timestamp, etc. Supporting defaults and conventions: latest, etc. 23

28 INSTALLER Discover, fetch, install. Customization: TwitterPackageInstaller Twitter uses its own MySQL releases (with its own custom config variables) Auxiliary utils and libs. Flags: --installer_args. (e.g. Package management args) API args: package. release, version, tags (latest, live), etc. Filesystem isolation (target 0.25): MySQL as a separate layer. 24

29 CONFIGS AND SCRIPTS Customization: Executor files e.g. my.cnf. Unpacked to disk upon launch. Flags: --executor_environ. JSON => ExecutorInfo::command::environment. e.g. MYSOS_DEFAULTS_FILE=<custom_my_cnf>. 25

30 DEPLOYING COTTON Cotton (PyPI) + Your customization => Your Cotton. PEX: Python EXecutable with all its dependencies bundled. cotton_executor.pex: fetchable by mesos-fetcher. cotton_scheduler.pex: managed by Aurora. 26

31 DEVELOPING COTTON 27

32 RETAINING MYSQL STATE Persistent volumes: available since Reserve hosts for Cotton through roles. Scheduler: create persistent volumes. Executor: sandbox bind mounted from a persistent volume (a directory managed by Mesos and not GCed). Scheduler: look for the old volumes from offers and reuse if possible. 28

33 ROAD AHEAD Client auth & authz; admin privilege. Replicate and failover using GTID. Isolate disk IO on shared hosts. Multiple MySQL versions per Cotton cluster. Scheduling contraints. Add instances to existing clusters. 29

34 RESOURCES #apache-cotton on freenode.org 30

35 QUESTIONS? 31

CONTINUOUS DELIVERY WITH MESOS, DC/OS AND JENKINS

APACHE MESOS NYC MEETUP SEPTEMBER 22, 2016 CONTINUOUS DELIVERY WITH MESOS, DC/OS AND JENKINS WHO WE ARE ROGER IGNAZIO SUNIL SHAH Tech Lead at Mesosphere @rogerignazio Product Manager at Mesosphere @ssk2