Introduction to Mesos and the Datacenter Operating System

Transcription

1 Introduction to Mesos and the Datacenter Operating System Artem Harutyunyan 2016 Mesosphere, Inc. All Rights Reserved.

2 INTRO $ whoami ARTEM HARUTYUNYAN ALICE Offline ( ) AliEn (porting to Windows, authentication, scheduling, etc) Fellow at PH/SFT ( ) CernVM Co-Pilot (LHC@home 2.0) CernVM (File System) Infrastructure Architect/Engineering Director at Qualys ( ) Engineering Director at Mesosphere ( ) 2015 Mesosphere, Inc. All Rights Reserved. 2

3 MESOS WHAT IS APACHE MESOS? Apache Mesos is a general purpose cluster manager Mesosphere, Inc. All Rights Reserved. 3

4 MESOS IS IT A BATCH SYSTEM? Wait!? Like a batch system? 2015 Mesosphere, Inc. All Rights Reserved. 4

5 MESOS IS IT A BATCH SYSTEM? NO! 2015 Mesosphere, Inc. All Rights Reserved. 5

6 MESOS 2-LEVEL SCHEDULING Framework Framework Framework Master Agents 2015 Mesosphere, Inc. All Rights Reserved. 6

7 MESOS LEVEL OF INDIRECTION Framework Framework Framework responsible for allocation (and reallocation) of resources Mesos (master) Mesos (agents) 2015 Mesosphere, Inc. All Rights Reserved. 7

8 MESOS LEVEL OF INDIRECTION framework framework responsible for allocation (and reallocation) of resources Mesos (master) Mesos (agents) 2015 Mesosphere, Inc. All Rights Reserved. 8

9 GOAL AUTOMATED OPERATIONS OF DISTRIBUTED SYSTEMS Software should manage itself. Most distributed systems are difficult to manage but they don t need to be Mesosphere, Inc. All Rights Reserved.

10 MESOS BUILT ON MESOS Mesosphere, Inc. All Rights Reserved. 10

11 MESOS PORTED TO MESOS Mesosphere, Inc. All Rights Reserved. 11

12 RESULT BREAK OUT OF TRADITIONAL INFRASTRUCTURE SILOS TRADITIONAL APPROACH UNIFIED APPROACH Complete workload portability Container App 1 PaaS 1 Container App 2 PaaS 2 Big Data Analytics 1 Big Data Analytics 2 Stateful Service 1 Stateful Service 2 Container Apps (All) PaaS (All) Big Data Analytics (All) Apache Mesos and the DC/OS Stateful Service (All) Deploys on-prem or in cloud Many silos. Management nightmare. Lengthy cycles to deploy code. Low utilization. High performance and resource isolation. Easy scalability and multi-tenancy. Fault tolerant and highly available. Highly efficient with highest utilization Mesosphere, Inc. All Rights Reserved. 12

13 THE DATACENTER COMPUTER 2016 Mesosphere, Inc. All Rights Reserved. 13

14 PRINCIPLES 1. TREAT MACHINES AS CATTLE NOT PETS Keep the base operating system small and simple, run containerized applications. 2. AUTOMATE WITH SOFTWARE NOT HUMANS Let software schedule software, i.e., handle failures, improve utilization, and manage maintenance Mesosphere, Inc. All Rights Reserved. 14

15 MESOS DATACENTER KERNEL provides common functionality every new distributed system reimplements: don t reinvent the wheel! resource allocation resource deallocation resource reservations resource isolation resource monitoring failure detection package distribution task starting, killing, cleanup volume management 2015 Mesosphere, Inc. All Rights Reserved. 15

16 MESOS INTERESTING FEATURES Quota Oversubscription Maintenance Persistence Primitives External Storage Security CVMFS Integration (presented at Mesoscon NA 2016) Optimistic Offers (upcoming) 2015 Mesosphere, Inc. All Rights Reserved. 16

17 DATACENTER OPERATING SYSTEM (DC/OS) Distributed Systems Kernel (Mesos) Distributed systems kernel to abstract resources 18

18 DATACENTER OPERATING SYSTEM (DC/OS) User Interface (GUI & CLI) Datacenter Operating System (DC/OS) Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage) Distributed Systems Kernel (Mesos) Distributed systems kernel to abstract resources 19

19 DATACENTER OPERATING SYSTEM (DC/OS) User Interface (GUI & CLI) Datacenter Operating System (DC/OS) Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage) Distributed Systems Kernel (Mesos) Distributed systems kernel to abstract resources On Premise AWS Azure 20

20 WHAT S IN THE BOX? Mesos! API, CLI and GUI Service Discovery & Load Balancing Storage Volumes Package Manager Installer for on premise and cloud 2016 Mesosphere, Inc. All Rights Reserved.

21 2016 Mesosphere, Inc. All Rights Reserved.

22 2016 Mesosphere, Inc. All Rights Reserved.

23 DC/OS 100% open source (ASL2.0) + Kicked off with a big, diverse community An umbrella for ~30 OSS projects + Roadmap and designs + The build tool chain + Docs and tutorials Not limited in any way Familiar, with a few new features 24

24 THE UNIVERSE 25

25 26

26 THE COMMUNITY 27

27 THANK YOU Check out dcos.io! 2016 Mesosphere, Inc. All Rights Reserved.