Introduction to Mesos and the Datacenter Operating System Artem Harutyunyan (artem@mesosphere.io) 2016 Mesosphere, Inc. All Rights Reserved.
INTRO $ whoami ARTEM HARUTYUNYAN ALICE Offline (2004-2010) AliEn (porting to Windows, authentication, scheduling, etc) Fellow at PH/SFT (2010-2012) CernVM Co-Pilot (LHC@home 2.0) CernVM (File System) Infrastructure Architect/Engineering Director at Qualys (2012-2015) Engineering Director at Mesosphere (2015 - ) 2015 Mesosphere, Inc. All Rights Reserved. 2
MESOS WHAT IS APACHE MESOS? Apache Mesos is a general purpose cluster manager. 2015 Mesosphere, Inc. All Rights Reserved. 3
MESOS IS IT A BATCH SYSTEM? Wait!? Like a batch system? 2015 Mesosphere, Inc. All Rights Reserved. 4
MESOS IS IT A BATCH SYSTEM? NO! 2015 Mesosphere, Inc. All Rights Reserved. 5
MESOS 2-LEVEL SCHEDULING Framework Framework Framework Master Agents 2015 Mesosphere, Inc. All Rights Reserved. 6
MESOS LEVEL OF INDIRECTION Framework Framework Framework responsible for allocation (and reallocation) of resources Mesos (master) Mesos (agents) 2015 Mesosphere, Inc. All Rights Reserved. 7
MESOS LEVEL OF INDIRECTION framework framework responsible for allocation (and reallocation) of resources Mesos (master) Mesos (agents) 2015 Mesosphere, Inc. All Rights Reserved. 8
GOAL AUTOMATED OPERATIONS OF DISTRIBUTED SYSTEMS Software should manage itself. Most distributed systems are difficult to manage but they don t need to be. 2016 Mesosphere, Inc. All Rights Reserved.
MESOS BUILT ON MESOS 2009 2010 2013 2014 2015 2015 Mesosphere, Inc. All Rights Reserved. 10
MESOS PORTED TO MESOS 2011 2012 2013 2014 2015 2015 Mesosphere, Inc. All Rights Reserved. 11
RESULT BREAK OUT OF TRADITIONAL INFRASTRUCTURE SILOS TRADITIONAL APPROACH UNIFIED APPROACH Complete workload portability Container App 1 PaaS 1 Container App 2 PaaS 2 Big Data Analytics 1 Big Data Analytics 2 Stateful Service 1 Stateful Service 2 Container Apps (All) PaaS (All) Big Data Analytics (All) Apache Mesos and the DC/OS Stateful Service (All) Deploys on-prem or in cloud Many silos. Management nightmare. Lengthy cycles to deploy code. Low utilization. High performance and resource isolation. Easy scalability and multi-tenancy. Fault tolerant and highly available. Highly efficient with highest utilization. 2015 Mesosphere, Inc. All Rights Reserved. 12
THE DATACENTER COMPUTER 2016 Mesosphere, Inc. All Rights Reserved. 13
PRINCIPLES 1. TREAT MACHINES AS CATTLE NOT PETS Keep the base operating system small and simple, run containerized applications. 2. AUTOMATE WITH SOFTWARE NOT HUMANS Let software schedule software, i.e., handle failures, improve utilization, and manage maintenance. 2015 Mesosphere, Inc. All Rights Reserved. 14
MESOS DATACENTER KERNEL provides common functionality every new distributed system reimplements: don t reinvent the wheel! resource allocation resource deallocation resource reservations resource isolation resource monitoring failure detection package distribution task starting, killing, cleanup volume management 2015 Mesosphere, Inc. All Rights Reserved. 15
MESOS INTERESTING FEATURES Quota Oversubscription Maintenance Persistence Primitives External Storage Security CVMFS Integration (presented at Mesoscon NA 2016) Optimistic Offers (upcoming) 2015 Mesosphere, Inc. All Rights Reserved. 16
DATACENTER OPERATING SYSTEM (DC/OS) Distributed Systems Kernel (Mesos) Distributed systems kernel to abstract resources 18
DATACENTER OPERATING SYSTEM (DC/OS) User Interface (GUI & CLI) Datacenter Operating System (DC/OS) Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage) Distributed Systems Kernel (Mesos) Distributed systems kernel to abstract resources 19
DATACENTER OPERATING SYSTEM (DC/OS) User Interface (GUI & CLI) Datacenter Operating System (DC/OS) Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage) Distributed Systems Kernel (Mesos) Distributed systems kernel to abstract resources On Premise AWS Azure 20
WHAT S IN THE BOX? Mesos! API, CLI and GUI Service Discovery & Load Balancing Storage Volumes Package Manager Installer for on premise and cloud 2016 Mesosphere, Inc. All Rights Reserved.
2016 Mesosphere, Inc. All Rights Reserved.
2016 Mesosphere, Inc. All Rights Reserved.
DC/OS 100% open source (ASL2.0) + Kicked off with a big, diverse community An umbrella for ~30 OSS projects + Roadmap and designs + The build tool chain + Docs and tutorials Not limited in any way Familiar, with a few new features 24
THE UNIVERSE 25
26
THE COMMUNITY 27
THANK YOU Check out dcos.io! 2016 Mesosphere, Inc. All Rights Reserved.