(using Apache Mesos, Jenkins and Marathon)

Transcription

1 Sunil Shh BUILDING A CONTAINERI{S Z}ED CONTINUOUS DEPLOYMENT PIPELINE (using Apche Mesos, Jenkins nd Mrthon) 1

2 Introduction WHY BOTHER? Why is continuous deployment interesting now? Google Trends for continuous delivery (blue) nd continuous deployment (red) 2

3 Introduction WHY BOTHER? 1. It s much esier to get compute resources nowdys! Doesn t cost much (sometimes it s even free - just sk grdute student) EVERY pltform nd their dog hs n API 3

4 Introduction WHY BOTHER? 1. It s much esier to get compute resources nowdys! 2. It turns out getting sleep is good for you! The Ntionl Sleep Foundtion recommends 7-9 hours of sleep per night Continer orchestrtors tke the mnul pin out of wking up nd rebooting n ppliction (to vrying degrees of success) Let your devs dev nd ops sleep! 4

5 Introduction WHY BOTHER? 1. It s much esier to get compute resources nowdys! 2. It turns out getting sleep is good for you! 3. Continers men you cn! No need for humns to ssh in nd `pt-get pckge instll python-mylibrry123`. Is this the beginning of the robot uprising? 5

6 APACHE MESOS: THE STORY OF 6

7 Apche Mesos: The Story Of THE BIRTH OF MESOS TWITTER TECH TALK APACHE INCUBATION The grd students working on Mesos give tech tlk t Twitter. Spring 2009 Mesos enters the Apche Incubtor. September 2010 Mrch 2010 December 2010 CS262B MESOS PUBLISHED Ben Hindmn, Andy Konwinski nd Mtei Zhri crete Nexus s their CS262B clss project. Mesos: A Pltform for Fine-Grined Resource Shring in the Dt Center is published s technicl report. 7

8 Apche Mesos: The Story Of GRAD STUDENTS LEARNED HOW TO SHARE Shring resources between btch processing frmeworks: Hdoop MPI Sprk Wht does n operting system provide? Resource mngement Progrmming bstrctions Security Monitoring, debugging, logging 8

9 Apche Mesos: The Story Of CLUSTERING YOUR RESOURCES FOR YOU Apche Mesos is cluster resource mnger. It hndles: Aggregting resources nd offering them to schedulers Lunching tsks (i.e. processes) on those resources Communicting the stte of those tsks bck to schedulers Tsks cn be: Long running services Ephemerl / btch jobs 9

10 Apche Mesos: The Story Of A SIMPLE MESOS CLUSTER ON MONDAY MORNING scheduler mster Sir, I hve some spre resources: 4 CPUs, 8 GB of memory nd 1 TB of disk. gent 10

11 Apche Mesos: The Story Of A SIMPLE MESOS CLUSTER ON MONDAY MORNING scheduler Hey, scheduler, would you like some compute resources? mster gent 11

12 Apche Mesos: The Story Of A SIMPLE MESOS CLUSTER ON MONDAY MORNING scheduler Not right now, but thnks! mster gent 12

13 Apche Mesos: The Story Of A SIMPLE MESOS CLUSTER ON MONDAY MORNING Hppy Mondy! Here s bunch of work. scheduler mster gent 13

14 Apche Mesos: The Story Of A SIMPLE MESOS CLUSTER ON MONDAY MORNING scheduler Chnged your mind? mster gent 14

15 Apche Mesos: The Story Of A SIMPLE MESOS CLUSTER ON MONDAY MORNING scheduler How d you know? Mind running this for me plese? mster gent 15

16 Apche Mesos: The Story Of A SIMPLE MESOS CLUSTER ON MONDAY MORNING scheduler mster Still got those spre resources? This tsk wnts to run on them. Let me know how it goes. gent 16

17 Apche Mesos: The Story Of A SIMPLE MESOS CLUSTER ON MONDAY MORNING scheduler mster Sure thing, it s running. gent 17

18 Apche Mesos: The Story Of A SIMPLE MESOS CLUSTER ON MONDAY MORNING scheduler Your tsk is running! I ll let you know if tht chnges. mster gent 18

19 Apche Mesos: The Story Of FAIRNESS FOR ALL SCHEDULERS scheduler A scheduler B Looks like you re not using your fir shre of resources - wnt some more? gent gent scheduler C scheduler D scheduler E gent gent mster gent 19

20 Apche Mesos: The Story Of HELPING YOUR OPERATOR SLEEP WELL scheduler A scheduler B scheduler C scheduler D mster 1 mster 2 mster 3 gent gent scheduler E ZooKeeper I just strted! Who s in chrge round here? gent gent gent 20

21 Apche Mesos: The Story Of HELPING YOUR OPERATOR SLEEP WELL scheduler A scheduler B scheduler C scheduler D mster 1 mster 2 mster 3 gent gent gent scheduler E ZooKeeper Check in with mster 1! gent gent 21

22 Apche Mesos: The Story Of MESOS CLUSTERS CAN BE REALLY, REALLY LARGE scheduler A scheduler A scheduler A scheduler A scheduler A scheduler B scheduler B scheduler B scheduler B scheduler B scheduler C scheduler C scheduler C scheduler C scheduler C scheduler D scheduler D scheduler D scheduler D scheduler D scheduler E scheduler E scheduler E scheduler E scheduler E ZooKeeper m1 m2 m3 m4 m5 m6 m7 m8 m9 22

23 Apche Mesos: The Story Of NOT USING MESOS IS EXPENSIVE! time 23

24 Apche Mesos: The Story Of PRODUCTION MESOS USERS Government Agencies

25 MARATHON (OR, HOW TO RUN MICROSERVICES ON MESOS) 25

26 Mrthon MARATHON TALKS TO MESOS Mesos cn t run pplictions on its own (!) Tht s where schedulers like Apche Auror nd Mrthon come in. Mrthon keeps your ppliction running! A bit like distributed init.d. 26

27 Mrthon A SELF SERVE INTERFACE TO YOUR CLUSTER ` 27

28 Mrthon MARATHON MANAGES THE APPLICATION LIFECYCLE Just lunched my product! Mrthon { "continer": { "docker": { "imge": "ssk2/pchecon-demo:1.0", }, "type": "DOCKER" }, "cpus": 0.1, "id": "pchecon-demo-pp", "instnces": 1, "mem": 128 mster } pchecon-demo:1.0 gent gent gent gent gent 28

29 Mrthon MARATHON MANAGES THE APPLICATION LIFECYCLE Fetured on Hcker News! SCALE! SCALE! Mrthon { "continer": { "docker": { "imge": "ssk2/pchecon-demo:1.0", }, "type": "DOCKER" }, "cpus": 0.1, "id": "pchecon-demo-pp", "instnces": 5, "mem": 128 mster } pchecon-demo:1.0 pchecon-demo:1.0 pchecon-demo:1.0 pchecon-demo:1.0 pchecon-demo:1.0 gent gent gent gent gent 29

30 Mrthon MARATHON MANAGES THE APPLICATION LIFECYCLE Time for v2! Mrthon { "continer": { "docker": { "imge": "ssk2/pchecon-demo:2.0", }, "type": "DOCKER" }, "cpus": 0.1, "id": "pchecon-demo-pp", "instnces": 5, "mem": 128 mster } pchecon-demo:2.0 pchecon-demo:2.0 pchecon-demo:2.0 pchecon-demo:2.0 pchecon-demo:2.0 pchecon-demo:1.0 pchecon-demo:1.0 pchecon-demo:1.0 pchecon-demo:1.0 pchecon-demo:1.0 gent gent gent gent gent 30

31 Mrthon MARATHON MANAGES THE APPLICATION LIFECYCLE Time for v2! Mrthon { "continer": { "docker": { "imge": "ssk2/pchecon-demo:2.0", }, "type": "DOCKER" }, "cpus": 0.1, "id": "pchecon-demo-pp", "instnces": 5, "mem": 128 mster } pchecon-demo:2.0 pchecon-demo:2.0 pchecon-demo:2.0 pchecon-demo:2.0 pchecon-demo:1.0 pchecon-demo:1.0 pchecon-demo:1.0 pchecon-demo:1.0 gent gent gent gent pchecon-demo:2.0 gent 31

32 Mrthon MARATHON MANAGES THE APPLICATION LIFECYCLE Time for v2! Mrthon { "continer": { "docker": { "imge": "ssk2/pchecon-demo:2.0", }, "type": "DOCKER" }, "cpus": 0.1, "id": "pchecon-demo-pp", "instnces": 5, "mem": 128 mster } pchecon-demo:2.0 gent pchecon-demo:2.0 gent pchecon-demo:2.0 pchecon-demo:2.0 pchecon-demo:1.0 pchecon-demo:1.0 gent gent pchecon-demo:2.0 gent 32

33 Mrthon MARATHON MANAGES THE APPLICATION LIFECYCLE Time for v2! Mrthon { "continer": { "docker": { "imge": "ssk2/pchecon-demo:2.0", }, "type": "DOCKER" }, "cpus": 0.1, "id": "pchecon-demo-pp", "instnces": 5, "mem": 128 mster } pchecon-demo:2.0 pchecon-demo:2.0 pchecon-demo:2.0 pchecon-demo:2.0 pchecon-demo:2.0 gent gent gent gent gent 33

34 JENKINS ON MESOS (AND WHY YOU SHOULD PROBABLY BE RUNNING IT LIKE THIS) 34

35 Jenkins on Mesos WHEN IT BEGAN Continuous Integrtion is soooo futuristic nd this interfce is beutiful. Jenkins mster 35

36 Jenkins on Mesos THE OLD WORLD Oh, is tht Jenkins cluster still up? I do ll my builds on Rspberry Pi now. My builds tke hours to run! Life is good. Jenkins mster Jenkins mster Jenkins mster Jenkins gent Jenkins gent Jenkins gent Jenkins gent Jenkins gent Jenkins gent Jenkins gent Jenkins gent Jenkins gent Jenkins gent Jenkins gent Jenkins gent Jenkins gent Jenkins gent Jenkins gent Jenkins gent Jenkins gent 36

37 Jenkins on Mesos JUST USE WHAT YOU NEED, WHEN YOU NEED IT AND SHARE THE LOVE RESOURCES Life is good. Life is good. Jenkins mster 1 Life is good. Jenkins mster 2 Jenkins mster 3 Mesos mster Jenkins gent (mster 1) Jenkins gent (mster 1) Jenkins gent (mster 3) Jenkins gent (mster 3) Jenkins gent (mster 3) Mesos gent Mesos gent Mesos gent Mesos gent Mesos gent 37

38 Jenkins on Mesos JUST USE WHAT YOU NEED, WHEN YOU NEED IT AND SHARE THE LOVE RESOURCES Life is good. Tem dinner! Let s bounce. Jenkins mster 1 Life is good. Jenkins mster 2 Jenkins mster 3 Mesos mster Jenkins gent (mster 3) Jenkins gent (mster 3) Jenkins gent (mster 3) Jenkins gent (mster 3) Jenkins gent (mster 3) Mesos gent Mesos gent Mesos gent Mesos gent Mesos gent 38

39 Jenkins on Mesos JUST USE WHAT YOU NEED, WHEN YOU NEED IT AND SHARE THE LOVE RESOURCES Life is good. Tem dinner! Let s bounce. Jenkins mster 1 Life is good. Jenkins mster 2 Jenkins mster 3 Mesos mster Jenkins gent (mster 3) Jenkins gent (mster 3) Jenkins gent (mster 3) Jenkins gent (mster 3) Jenkins gent (mster 3) Mesos gent Mesos gent Mesos gent Mesos gent Mesos gent 39

40 Jenkins on Mesos BUILDING DOCKER IN DOCKER: ONE WEIRD TRICK This brve new world of continers running in continers hs bit of whle nd whle egg* problem. We run everything inside continer to mke it esy to bundle dependencies nd to isolte it from other processes. But when the thing tht s running wnts to build continer, wht do you do? *Yes, I know. 40

41 Jenkins on Mesos BUILDING DOCKER IN DOCKER: ONE WEIRD TRICK One recommended pproch is to bind mount in the host system s Docker demon. `docker run mesosphere/jenkins:0.2.2` `docker run ssk2/pchecon-demo:f4ccd67` Docker demon Mesos gent 41

42 Jenkins on Mesos BUILDING DOCKER IN DOCKER: ONE WEIRD TRICK This doesn t work for Mesos though! It doesn t trck continers tht it doesn t lunch. The sibling continer becomes orphned nd runs forever. `docker run ssk2/pchecon-demo:f4ccd67` Docker demon Mesos gent 42

43 Jenkins on Mesos BUILDING DOCKER IN DOCKER: ONE WEIRD TRICK Our solution is to use customised Docker-in-Docker continer. This is little slower but Mesos tkes cre of the resources! `docker run ssk2/pchecon-demo:f4ccd67` `docker run mesosphere/jenkins:0.2.2` Docker demon Mesos gent 43

44 CONTINUOUS DEPLOYMENT 44

45 Continuous Deployment PIPELINE: A FIRST PASS 2015 Mesosphere, Inc. All Rights Reserved. 45

46 1. Configurtion 2. Build 3. Deploy 2015 Mesosphere, Inc. All Rights Reserved. 46

47 Continuous Deployment 1. CONFIGURATION Building CD pipeline requires configurtion in couple of plces: 1. Docker nd Mrthon files in your repo 2. Build configurtion in Jenkins* *in the future, you ll be ble to check in your build configurtion longside your repository too! 2015 Mesosphere, Inc. All Rights Reserved. 47

48 1. CONFIGURATION DEPENDENCY MANAGEMENT Docker is becoming the de-fcto continer formt for pckging pplictions: Encpsultes dependencies Runs on your lptop Runs on your cluster Mesos nd Mrthon hve ntive support for Docker. Just stick Dockerfile (or two) in the root of your repository! 2015 Mesosphere, Inc. All Rights Reserved. 48

49 1. CONFIGURATION DEPENDENCY MANAGEMENT FROM jekyll/jekyll ADD site /srv/jekyll 2015 Mesosphere, Inc. All Rights Reserved. 49

50 1. CONFIGURATION APPLICATION CONFIGURATION Mrthon ppliction definitions re JSON files tht describe: resources required how mny instnces to run wht commnd to run how to check your ppliction is helthy mrthon.json should live in the root of your project repository Mesosphere, Inc. All Rights Reserved. 50

51 { 1. CONFIGURATION "id": "pchecon-demo", APPLICATION CONFIGURATION "continer": { "type": "DOCKER", "docker": { "imge": "ssk2/pchecon-demo:ltest", "network": "BRIDGE", "portmppings": [{ "continerport": 80, "protocol": "tcp" }] } }, "lbels": { "HAPROXY_0_VHOST": "sunil-889-publicsl-781ifozhqg3z us-west-2.elb.mzonws. com", "HAPROXY_GROUP": "externl" }, "instnces": 1, "cpus": 0.1, "mem": 128 } 2015 Mesosphere, Inc. All Rights Reserved. 51

52 Continuous Deployment 2. BUILDING It s trivil to instll Jenkins on DCOS: 1. Crete JSON file: {"jenkins": {"frmework-nme": "my-jenkins" }} 2. Instll: $ dcos pckge instll --options=my-jenkins-config.json jenkins : 3.??? 4. Profit! 2015 Mesosphere, Inc. All Rights Reserved. 52

53 Continuous Deployment 2. BUILDING Now, set up Jenkins: 1. Instll the Mrthon plugin 2. Sve your Docker Hub credentils 3. Set up triggered build to build nd push Docker imge docker build. -t ssk2/whereisbot:$(git_branch) docker push ssk2/whereisbot:$(git_branch) 4. Set up triggered build to updte mrthon.json using jq nd PUT to Mrthon http PUT < mrthon.json 2015 Mesosphere, Inc. All Rights Reserved. 53

54 Continuous Deployment 2. BUILDING Next, let s crete build: 1. Set up build tht polls GitHub periodiclly to build nd push Docker imge docker login -u ${DOCKER_HUB_USERNAME} -p ${DOCKER_HUB_PASSWORD} -e sunil@mesosphere.com docker build -t ssk2/pchecon-demo:$(git_commit). docker push ssk2/pchecon-demo:$(git_commit) 2. Add Mrthon post deploy step pointing to the DC/OS Mrthon: Set ny vribles you d like to override Mesosphere, Inc. All Rights Reserved. 54

55 Continuous Deployment 3. DEPLOYING When you PUT to Mrthon s API, you trigger deployment. http PUT < mrthon.json Mrthon ttempts to scle ppliction to desired stte by: Lunching new instnces By defult try to lunch 100% of instnces requested t once Killing old instnces when new instnces re helthy 2015 Mesosphere, Inc. All Rights Reserved. 55

56 THANK YOU! Come nd tlk to us! Emil me t sunil@mesosphere.io Slides will be up t Check out Other Mesosphere tlks: 11:15m Elstic Hdoop Clusters on Mesos with Apche Myrid 2:30pm Stteful Services on Apche Mesos 56