Mesosphere and the Enterprise: Run Your Applications on Apache Mesos. Steve Wong Open Source Engineer {code} by Dell

Transcription

1 Mesosphere and the Enterprise: Run Your Applications on Apache Mesos Steve Wong Open Source Engineer {code} by Dell

2 Open source at Dell EMC {code} by Dell EMC is a group of passionate open source engineers and advocates working to build a community around software-based infrastructure. Contribute to meaningful OSS projects Create new thought leading OSS applications Drive awareness of OSS opportunities with Dell EMC product teams Participate in relevant community engagement projects Act in the interest of building a community Platinum Sponsor 2

3 Why Containers? They make managing applications easier Speed up software innovation Deployments are simple Creates portable, shareable and reusable units of code and requirements Code Code and requirements Container 3

4 What is Mesosphere DC/OS? From a resource perspective: It s an infrastructure abstraction layer: Aggregates systems into a single resource pool. From an application perspective: It simplifies management of distributed technologies A scheduler (= dispatcher). Dispatches workloads which consume the pooled resources. 4

5 History Started at UC Berkeley in 2009, became an Apache project in 2013 after nearly two years in incubation 5

6 Project Objective Deliver Google s Borg to the rest or world See Wired: How Twitter rebuilt Google s Borg March

7 Two levels of Application Support Generic Run anything that can be launched in a standard Linux shell + Docker images Framework Allows you to implement a sub-scheduler Extends Mesos to provide custom workload placement and resource allocation For example, reserve 10% of the cluster s resources to Hadoop jobs, framework manages prioritized dispatch 7

8 With Marathon & Metronome, all types of jobs are managed 8

9 Marathon Marathon acts as a service dispatcher You provide instructions and Marathon does its best to keep the service up on the cluster, providing automated scaling and high availability restarts 9

10 Metronome a chron for the datacenter Specify: The amount of CPU your job will consume. The amount of memory your job will consume. The disk space your job will consume. The schedule for your job, in cron format. You can also set the time zone and starting deadline. 10

11 Overview of Enterprise DC/OS 11

12 Where can you run Mesos? 12

13 Any limitations on what I can use for cluster nodes? They do not have to be identical Processor model, speed or memory size variations are OK They can concurrently run non-mesos services don t have to give Mesos control over 100% of node s CPU, memory or disk. Elastic scaling is supported nodes can come and go 13

14 Production Users & Customers Government Agencies 14

15 Efficiency and Utilization DCOS Multiplexing 30-40% utilization, up to 96% at some customers Industry Average 12-15% utilization Typical Datacenter siloed, over-provisioned servers, low utilization 4X DCOS Datacenter automated schedulers, workload multiplexing onto the same machines 15

16 How big can this really go? Performance Characteristics 16

17 Performance metric at scale 50,000 containers deployed live on stage 1 minute 12 seconds 17

18 Architecture 18

19 How {code} by Dell EMC fits with Mesos 19

20 Back in 2012 container aps must be stateless? 20

21 The picture today 21

22 External Volume Mount Solution 22

23 External Volume Mount Addresses Availability Risk Migrate container to another host your storage is gone Host goes down your service goes down Scale Limitation Need more storage than the host has? Sorry 23

24 Components REX-Ray rexray.codedellemc.com 24

25 REX-Ray The leading container storage orchestration engine enabling persistence for cloud native workloads Cloud Native Interoperability Open Source REX-Ray rexray.codedellemc.com Enterprise Ready High Availability CLI Intuitiveness Effortless Deployment Architectural Choices Multi-Platform Storage Management Storage agnostic (block/file/object) 25

26 Mesos Container Framework Integration Marathon from Mesosphere Apache Aurora (originally used by Twitter) ElasticSeach 29

27 End-to-End Container Persistence Leadership for Mesos mesos-module-dvdi Docker Volume Driver Client CLI DVDCLI REX-Ray What does it do Enables external storage to be created / mounted / unmounted with each agent task Abstracts Docker Volume Drivers for Mesos Storage orchestration engine Cool factor First Mesos Agent external storage module Merged into Mesos 1.0 (2016) Allows the use of any Docker volume driver with Mesos! Vendor agnostic DC/OS 1.7+ framework (2016) 30

28 ScaleIO Framework for Mesos A Software-based Storage Framework using ScaleIO to turn commidy hardware and direct attached storage (DAS) into a globally accessible and scalable storage platform that can be deployed anywhere Framework installs and configures ScaleIO on all Mesos Agent (compute) nodes. As more Agents are added to the cluster, ScaleIO is installed and adds storage resources to the cluster. Deploy and Configure ScaleIO without knowing operational details Centralized Monitoring of the storage platform Persistent storage native to the container scheduling platform using REX-Ray 31

29 {code} + Mesos MARATHON ScaleIO-Framework installs ScaleIO software on all Mesos Agents Mesos Module DVDI is officially merged into Mesos 1.0 and allows any Docker Volume Driver to be used with Mesos Frameworks REX-Ray is installed on every host and performs storage orchestration SUPPORTED STORAGE PLATFORMS WITH REX-RAY 32

30 Demo! DC/OS / Marathon Mesos Database Deploy 33

31 Steve github.com/cantbewong HOL01 Use REX-Ray & ScaleIO w/ Docker, Mesos and Kubernetes rexray.codedellemc.com codedellemc.com {code} by Dell EMC is a group of passionate open source engineers and advocates working to build a community around software-based infrastructure. blog.codedellemc.com github.com/codedellemc/labs 34

32