Building a Data-Friendly Platform for a Data- Driven Future

Size: px

Start display at page:

Download "Building a Data-Friendly Platform for a Data- Driven Future"

Kevin McBride
6 years ago
Views:

REINFORCING TRENDS microservices containerization

REINFORCING TRENDS big data & analytics microservices containerization

10 TREND TOWARDS MICROSERVICES 1.Do one thing and do it well (UNIX). 2.Compose! 3.Test and debug in isolation. 4.Captures organizational structure (many teams working in parallel) Mesosphere, Inc. All Rights Reserved. 10

11 MICROSERVICES Traditional Application Architecture Today s Microservices Application Architecture Hard to scale, wasting resources REST APIs Scalable, efficient and fully dynamic Siloed teams Many functions in a single process Cross-functional teams organized around capabilities Each element of functionality defined as microservices 2015 Mesosphere, Inc. All Rights Reserved.

CONTAINERIZATION then more moving parts now less

15 ANATOMY OF MODERN APPLICATIONS Functions & Logic Big Data & Analytics X... Big Data Processing... Data Storage... Message Queue.. X X. Anything else X Microservices in containers Distributed Systems 2016 Mesosphere, Inc. All Rights Reserved.

16 NEED FOR NEW INFRASTRUCTURE Microservices, containers, and big data lead to more difficult operations w/out proper infrastructure. Challenges: 1)Deployment and scheduling for fault-tolerance. 2)Scheduling for elasticity and efficiency. 3)Service discovery ( naming ) and networking. 4)Operations: maintenance and upgrades. 5) 2015 Mesosphere, Inc. All Rights Reserved. 16

19 BREAK OUT OF TRADITIONAL INFRASTRUCTURE SILOS TRADITIONAL APPROACH UNIFIED APPROACH Container App 1 PaaS 1 Container App 2 PaaS 2 Big Data Analytics 1 Big Data Analytics 2 Stateful Service 1 Stateful Service 2 Container Apps (All) PaaS (All) Big Data Analytics (All) Apache Mesos and the DC/OS Stateful Service (All) Deploy on-pr em or in cloud Many silos. Management nightmare. Lengthy cycles to deploy code. Low utilization. High performance and resource isolation. Easy scalability and multi-tenancy. Fault tolerant and highly available. Highly efficient with highest utilization Mesosphere, Inc. All Rights Reserved. 19

UTILIZATION RUN EVERYTHING ON THE SAME SHARED

Automated schedulers, workload multiplexing, less

21 UTILIZATION RUN EVERYTHING ON THE SAME SHARED INFRASTRUCTURE 30-40% utilization, up to 96% for some users Industry Average 12-15% utilization Siloed, over-provisioned servers, low utilization Automated schedulers, workload multiplexing, less machines or more applications 2015 Mesosphere, Inc. All Rights Reserved. 21

22 EVOLUTION: FROM STATIC TO DYNAMIC INFRASTRUCTURE Existing Computing Infrastructure Is Inefficient And Not Suitable for Modern Workloads Computing Infrastructure Evolution t MAINFRAME Data / Transaction Processing PHYSICAL (x86) Client-Server Apps (ERP, CRM, Productivity) VIRTUAL Web-Based Applications (Enterprise & Consumer) UNIFIED Stateless Microservices Stateful Distributed Systems Analytics Siloed, Static, Monolithic & Manual Efficient, Dynamic, Agile & Automated 2016 Mesosphere, Inc. All Rights Reserved. 22

24 DATACENTER COMPUTER PRINCIPLES 1. TREAT MACHINES AS CATTLE NOT PETS Keep the base operating system small and simple, run containerized applications. 2. AUTOMATE WITH SOFTWARE NOT HUMANS Let software schedule software, i.e., handle failures, improve utilization, and manage maintenance Mesosphere, Inc. All Rights Reserved. 24

25 Datacenter Computer Needs an Operating System OS OS OS desktop computer server datacenter

26 DATACENTER OPERATING SYSTEM (DC/OS) Distributed Systems Kernel (Mesos) Distributed systems kernel to abstract resources 26

27 DATACENTER OPERATING SYSTEM (DC/OS) User Interface (GUI & CLI) Datacenter Operating System (DC/OS) Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage) Distributed Systems Kernel (Mesos) Distributed systems kernel to abstract resources 27

28 DATACENTER OPERATING SYSTEM (DC/OS) User Interface (GUI & CLI) Datacenter Operating System (DC/OS) Core system services (e.g., distributed init, cron, service discovery, package mgt & installer, storage) Distributed Systems Kernel (Mesos) Distributed systems kernel to abstract resources On Premise AWS Azure 28

MESOS WHAT IS APACHE MESOS? Apache Mesos is a general purpose cluster manager (i.e., not just focused on batch computation).

32 SOLUTIONS ACADEMIA VS INDUSTRY DIFFERENT SOFTWARE Academia Industry MPI (Message Passing Interface) Apache (mod_perl, mod_php) Web Service (Java, Ruby, ) 2016 Mesosphere, Inc. All Rights Reserved. 32

34 SOLUTIONS ACADEMIA VS INDUSTRY CLUSTER MANAGEMENT Academia PBS (Portable Batch System) TORQUE SGE (Sun Grid Engine) Industry SSH Puppet / Chef Capistrano / Ansible Cluster Managers 2016 Mesosphere, Inc. All Rights Reserved. 34

Grid Engine) Industry SSH Puppet / Chef Capistrano /

36 SOLUTIONS ACADEMIA VS INDUSTRY CLUSTER MANAGEMENT Academia PBS (Portable Batch System) TORQUE SGE (Sun Grid Engine) Industry SSH Puppet / Chef Capistrano / Ansible Batch Computation! 2016 Mesosphere, Inc. All Rights Reserved. 36

37 Mesos is a cluster manager with a master/agent architecture masters agents

MESOS 2-LEVEL SCHEDULING coordinator coordinator

39 Mesos challenged the status quo of cluster managers

40 cluster manager status quo application specificatio n the specification includes as much information as possible to assist the cluster manager in scheduling and execution cluster manager

41 cluster manager status quo application wait for task to be executed cluster manager

42 cluster manager status quo application resul t cluster manager

43 problems with specifications hard to specify certain desires or constraints hard to update specifications dynamically as tasks execute and finish/fail

44 the bigger picture cluster manager has inadequate knowledgeof distributed system s execution needs/semantics to make optimal decisions distributed system s execution needs/semantics can t easily or efficiently be expressed to cluster manager

45 MapReduce specification?

46 MapReduce specification? course-grained fine-grained

47 MapReduce specification? course-grained fine-grained best resource utilization, but hard if impossible to specify

48 MapReduce specification? course-grained fine-grained worst resource utilization, but easy to express (how most cluster managers run something like Hadoop)

49 distributed systems register with the Mesos master(s) in order to run computations frameworks masters agents

50 Mesos model coordinator request 3 CPUs 2 GB RAM a request is purposely simplified subset of a specification including just the required resources at that point in time masters

51 question: what should you do if you can t satisfy a request?

52 question: what should you do if you can t satisfy a request? wait until you can

53 question: what should you do if you can t satisfy a request? wait until you can offer best you can immediately

54 question: what should you do if you can t satisfy a request? wait until you can offer best you can immediately

55 Mesos model coordinator offer hostname 4 CPUs 4 GB RAM masters

56 Mesos model coordinator offer offer hostname offer hostname offer 4 CPUs hostname 4 4 CPUs hostname GB RAM 4 4 CPUs GB RAM 4 4 CPUs GB RAM 4 GB RAM masters

57 Mesos model coordinator offer offer hostname offer hostname offer 4 CPUs hostname 4 4 CPUs hostname GB RAM 4 4 CPUs GB RAM 4 4 CPUs GB RAM 4 GB RAM distributed system uses the offers to perform it s own scheduling masters

58 Mesos model coordinator task 3 CPUs 2 GB RAM distributed system uses the offers to perform it s own scheduling masters

59 Mesos model coordinator task 3 CPUs 2 GB RAM distributed system uses the offers to perform it s own scheduling masters multi-level scheduling

60 1 st level: master makes allocations via offers 2 nd level: distributed systems schedule tasks using offers

isolation resource monitoring failure detection package distribution task starting, killing,

62 MESOS DATACENTER KERNEL today tomorrow YOU provides common functionality every new distributed system re-implements: resource allocation resource deallocation resource reservations resource isolation resource monitoring failure detection package distribution task starting, killing, cleanup volume management don t reinvent the wheel! 2016 Mesosphere, Inc. All Rights Reserved. 62

63 distributed systems are hard to build

64 distributed systems are hard to operate: deploy, maintain, update

65 operating distributed systems

66 operating distributed systems download deploy (read book; HA? arrangement?) monitor (read book; logs, metrics, alerting) maintain debug (read book, ask internet, IRC) fix problem (update runbooks, write scripts, build ancillary system!) upgrade (and/or redeploy)

67 perspective distributed systems should be able to operatethemselves: deploy, monitor, update, upgrade, etc need an interface that enables distributed systems to communicatewith the underlying infrastructure, and vice versa

68 perspective distributed systems should be able to operate themselves: deploy, monitor, update, upgrade, etc Apache Mesos need an interface that enables distributed systems to communicate with the underlying infrastructure, and vice versa

69 why: the bigger picture humans have inadequate knowledge of distributed system needs/semantics to make optimal decisions (even after reading the book) distributed system execution needs/semantics can t easily or efficiently be expressed to humans and/or underlying infrastructure

70 Linux applications operate themselves on Linux, when they need more CPU, they ask the Linux kernel for more CPUs (processes, threads); when they need certain operations to be performed (e.g., write/read to file), they ask the Linux kernel;

71 Mesos distributed systems operate themselves on Mesos, when they need more CPU, they ask Mesos for more CPUs (tasks, containers); when they need certain operations to be performed (e.g., create a persistent volume), they ask Mesos;

AUTOMATED OPERATIONS OF DISTRIBUTED SYSTEMS Software will manage itself, using Mesos and the DC/OS API Most distributed systems are difficult to manage but they don t

72 AUTOMATED OPERATIONS OF DISTRIBUTED SYSTEMS Software will manage itself, using Mesos and the DC/OS API Most distributed systems are difficult to manage but they don t need to be. HDFS Distributed file system Kafka Messaging backbone Cassandra Distributed database Spark Data processing engine 2016 Mesosphere, Inc. All Rights Reserved.

73 THE UNIVERSE 73

CASE STUDY Forging Ahead with Mesos, Containers and DC/OS Having now run our event streaming and big data ingestion pipeline services in production on DC/OS, across 3 regions, over the last year,

75 CASE STUDY Forging Ahead with Mesos, Containers and DC/OS Having now run our event streaming and big data ingestion pipeline services in production on DC/OS, across 3 regions, over the last year, we've achieved the following results: A 66% reduction in AWS Instances Cost Improvements up to 57% An impressive 40 sec time to deploy a new build with zero downtime A 3 min time to stand up a new region 100% Uptime Total Resources needed: 1 DevOps Engineer 75

76 76

Introduction to Mesos and the Datacenter Operating System

Introduction to Mesos and the Datacenter Operating System Artem Harutyunyan (artem@mesosphere.io) 2016 Mesosphere, Inc. All Rights Reserved. INTRO $ whoami ARTEM HARUTYUNYAN ALICE Offline (2004-2010) AliEn