Marathon & Metronome Mesosphere, Inc. All Rights Reserved. 1

Size: px

Start display at page:

Abel Mosley
6 years ago
Views:

Plugins Extension Interface is defined as a scala trait (java interface) All extension interfaces are published as separate maven artifact Plugins have to implement one or

6 Plugins Extension Interface is defined as a scala trait (java interface) All extension interfaces are published as separate maven artifact Plugins have to implement one or more extension interfaces Marathon can load plugins and use the custom defined functionality See all active plugins via /v2/plugins 2016 Mesosphere, Inc. All Rights Reserved. 6

7 Plugin Interface: Authentication + Authorization Easy to interface with existing authentication providers (OAuth, LDAP, OpenId..) Fine grained access control Allows for multi tenancy 2016 Mesosphere, Inc. All Rights Reserved. 7

Plugin Interface: Validation + TaskInfo customization Validate AppDefinitions Define or

9 Plugin Example Create a jar with your plugin Create a plugin configuration $> marathon \ --plugin_dir /plugins \ --plugin_conf plugin.conf plugin.conf: {"plugins": {"my-secrets-directive-processor": { "plugin": "m.m.p.task.runspectaskprocessor", "implementation": "my.comp.secretsprocessor", "tags": [ "config", "secrets" ], "info": { "name": "My Comp Secrets Plugin" } }}} 2016 Mesosphere, Inc. All Rights Reserved. 9

11 Persistent Storage Do you need it? A container that replaces a terminated container [ ] doesn't care about previously persisted data [ ] takes care of replication on its own [ ] can only properly work with data from the previous instance 2016 Mesosphere, Inc. All Rights Reserved. 11

12 Persistent Storage Workaround In the past, people used to run stateful service by pinning instances to certain nodes. Downsides of that are It s a manual process Prevents optimal resource utilization Cumbersome to scale 2016 Mesosphere, Inc. All Rights Reserved. 12

13 Persistent Volumes Marathon will reserve required resources create persistent volumes launch based on offers pin tasks to agents { } "id": "/stateful", "container": { "type": "Mesos", "volumes": [{ "containerpath": "mydata", "mode": "RW", "persistent": {"size": 100} }] }, "residency": { "tasklostbehavior": "WAIT_FOREVER" } 2016 Mesosphere, Inc. All Rights Reserved. 13

14 Persistent Volumes: Gotcha s Persistent volumes are not deleted unless you delete the service delete a task adding?wipe=true MultiDisk not yet supported Can't use pre-reserved resources yet hostname UNIQUE constrained services should be scaled 1 at a time 2016 Mesosphere, Inc. All Rights Reserved. 14

15 External Volumes (experimental since 1.1) Uses storage provider in a cluster Provides network volumes on any node More flexible scheduling with less node constraints { "id": "hello", "cmd": "/usr/bin/tail -f /dev/null", "container": { "type": "MESOS", "volumes": [{ "containerpath": "test-rexray-volume", "external": { "size": 100, "name": "my-test-vol", "provider": "dvdi", "options": {"dvdi/driver": "rexray"} }, "mode": "RW" }] }} 2016 Mesosphere, Inc. All Rights Reserved. 15

16 External Volumes: Gotcha s An application can currently have at most 1 instance Storage is not automatically cleaned up Storage solutions other than EBS are rather untested 2016 Mesosphere, Inc. All Rights Reserved. 16

Readiness Checks AppDefinition can define Readiness Checks Executed on every Task during start or update Defines a barrier, every task has to pass once Deployments

18 Readiness Checks AppDefinition can define Readiness Checks Executed on every Task during start or update Defines a barrier, every task has to pass once Deployments will wait for all tasks to become ready Readiness results are available via the API Allows for deployment customization 2016 Mesosphere, Inc. All Rights Reserved. 18

19 Readiness Check Example Definition similar to Health Checks Currently only HTTP based readiness checks are supported Currently only one readiness check is supported { } "id": "/with/readiness", "readinesschecks": [ { "name": "myreadycheck", "protocol": "HTTP", "path": "/v1/plan", "portname": "http", "intervalseconds": 10, "timeoutseconds": 3, "httpstatuscodesforready": [ 200 ], "preservelastresponse": true } ] 2016 Mesosphere, Inc. All Rights Reserved. 19

20 Readiness Check Results /v2/deployments will show the current readiness check result Last response is preserved on demand [{ "id": "97c136bf-5a d94-480d9fbb01c8", "currentactions": [{ "action": "ScaleApplication", "app": "/foo", "readinesscheckresults": { "/foo": [{ "taskid": "foo.c9de6033", "lastresponse": { "body": "{}", "contenttype": "application/json", "status": 500 }, "name": "myreadycheck", "ready": false }]} }] }] 2016 Mesosphere, Inc. All Rights Reserved. 20

Customize Deployments: Example Kafka Scheduler Marathon usually starts/updates the Kafka scheduler The Kafka scheduler is responsible for

21 Customize Deployments: Example Kafka Scheduler Marathon usually starts/updates the Kafka scheduler The Kafka scheduler is responsible for starting updating broker nodes Synchronize Kafka Deployments with Marathon Deployments via Readiness checks 2016 Mesosphere, Inc. All Rights Reserved. 21

23 Docker Container Exemplary docker config Note container type MESOS Does not support all properties of the docker containerizer If you don t really need those, go with UNICON { } "id": "/unified/docker", "container": { "type": "MESOS", "docker": { "image": "mesosphere:marathon/latest", "credential": { "principal": "username", "secret": "password" } } } 2016 Mesosphere, Inc. All Rights Reserved. 23

24 AppC Container Exemplary AppC config Note type MESOS Optional labels { } "id": "/unified/appc", "container": { "type": "MESOS", "appc": { "image": "mesosphere:marathon/latest", "labels": { "foo": "bar" }, "forcepullimage": true } } 2016 Mesosphere, Inc. All Rights Reserved. 24

25 Use GPU as a resource Gpus are supported as a resource Only full gpus can be used (no fractions) { } "id": "/with/gpus", "instances": 2, "cmd": "my_gpu_app", "cpus": 1.5, "disk": 1024, "mem": 128, "gpus": Mesosphere, Inc. All Rights Reserved. 25

28 ZK Layout ZK Limits Flat storage layout will limit the nr of tasks: ~60K tasks Object size can not be bigger than 1mb: ~2000 Apps Store the tasks in a nested layout: 1 million Tasks Only store object references: > 50K Apps 2016 Mesosphere, Inc. All Rights Reserved. 28

30 Use Marathon core as a library Extensive task scheduling functionality Under active development Lots of added features Various improvements Accelerate adding features in the future 2016 Mesosphere, Inc. All Rights Reserved. 30

31 Job Definition Simple JobSpec specifying a docker based job { } "id": "prod.example.app", "run": { "cmd": "nuke --dry --master local", "cpus": 1.5, "mem": 32, "disk": 128, "docker": {"image": "foo/bla:test"} } 2016 Mesosphere, Inc. All Rights Reserved. 31

Deploying WordPress and MySQL

Deploying WordPress and MySQL In this tutorial you will learn how to deploy a WordPress and MySQL installation on a DC/OS cluster and stop worrying about network management. Overview The goal of this tutorial