Container 2.0. Container: check! But what about persistent data, big data or fast data?!

@unterstein @joerg_schad @dcos @jaxdevops Container 2.0 Container: check! But what about persistent data, big data or fast data?! 1

OVERVIEW noun ˈmīkrō/ /ˈsərvəs/ : an approach to application development in which a large application is built as a suite of modular services. Each module supports a specific business goal and uses a simple, well-defined interface to communicate with other modules.* Microservices are designed to be flexible, resilient, efficient, robust, and individually scalable. *From whatis.com 2017 Mesosphere, Inc. All Rights Reserved.

MICROSERVICES - Polyglot - Single Responsibility - Smaller Teams - Utilization - Machine types/groups - Dependency hell App Service App Service Operating System App Service Service Operating System Service Service Service Service Operating System Service Machine Machine Machine Infrastructure 2017 Mesosphere, Inc. All Rights Reserved. 6

Run everything in containers! Gerard Julien/AFP

CONTAINERS - Rapid deployment - Dependency vendoring - Container image repositories - Spreadsheet scheduling App App App Service Service Service Service Service Service Service Service Service Container Runtime Container Runtime Container Runtime OS OS OS Machine Machine Machine Infrastructure 2017 Mesosphere, Inc. All Rights Reserved. 8

CONTAINER ORCHESTRATION CONTAINER SCHEDULING - Placement - Replication/Scaling - Resurrection - Rescheduling - Rolling Deployment - Upgrades - Downgrades - Collocation RESOURCE MANAGEMENT - Memory - CPU - GPU - Volumes - Ports - IPs - Images/Artifacts SERVICE MANAGEMENT - Labels - Groups/Namespaces - Dependencies - Load Balancing - Readiness Checking 2017 Mesosphere, Inc. All Rights Reserved. 15

CONTAINER ORCHESTRATION Orchestration Web Apps & Services Service Management Scheduling Resource Management Container Runtime Container Runtime Container Runtime Machine & OS Machine & OS Machine & OS Machine Infrastructure 2017 Mesosphere, Inc. All Rights Reserved. 16

DATA PROCESSING AT HYPERSCALE EVENTS INGEST STORE ANALYZE ACT Ubiquitous data streams from connected devices Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications Sensors Devices Apache Kafka Apache Cassandra Apache Spark Akka Clients DC/OS 2016 Mesosphere, Inc. All Rights Reserved. 18

Data Model Relational Key-Value Graph Document Time-Series Files Schema SQL Foreign Keys/Joins OLTP/OLAP Simple Scalable Cache Complex relations Social Graph Recommend ation Fraud detections Schema-Less Semi-structu red queries Product catalogue Session data 2017 Mesosphere, Inc. All Rights Reserved. 24

NAIVE APPROACH Spark/Hadoop Kafka mysql microservice Industry Average 12-15% utilization Cassandra Typical Datacenter siloed, over-provisioned servers, low utilization 2017 Mesosphere, Inc. All Rights Reserved. 26

MULTIPLEXING OF DATA, SERVICES, USERS, ENVIRONMENTS Spark/Hadoop Kafka mysql microservice Cassandra Typical Datacenter siloed, over-provisioned servers, low utilization Mesos, DC/OS automated schedulers, workload multiplexing onto the same machines 2017 Mesosphere, Inc. All Rights Reserved. 28

Apache Mesos A top-level Apache project A cluster resource negotiator Scalable to 10,000s of nodes Fault-tolerant, battle-tested An SDK for distributed apps Native Docker support 2017 Mesosphere, Inc. All Rights Reserved. 29

DC/OS ENABLES MODERN DISTRIBUTED APPS Modern App Components Microservices (in containers) Big Data + Analytics Engines Functions & Logic Streaming Batch Machine Learning Search Time Series SQL / NoSQL Analytics Databases Datacenter Operating System (DC/OS) Distributed Systems Kernel (Mesos) Any Infrastructure (Physical, Virtual, Cloud) 31

THE BASICS DC/OS is 100% open source (ASL2.0) + A big, diverse community An umbrella for ~30 OSS projects + Roadmap and designs + Docs and tutorials Not limited in any way Familiar, with more features + Networking, Security, CLI, UI, Service Discovery, Load Balancing, Packages,... 32

THE UNIVERSE 34

Container Storage Default Sandbox Simple to use, Task failures Local Persistent Volumes Task failures, (permanent) Node failures Distributed File System/External Storage Node failures, non-local writes 2017 Mesosphere, Inc. All Rights Reserved. 37

Simple things should be simple; complicated things should be complicated ~100 LOC Defaults Simple YAML interface Default, customizable behaviors Most services probably fall here ~1,000 LOC Custom Plans & Strategies Need to be a service expert Need to write a little Java code Don t need to understand DC/OS 10K+ LOC Build Your Own Scheduler with commons libraries Can t use the defaults Don't mind writing a lot of Java Willing to understand DC/OS 2016 Mesosphere, Inc. All Rights Reserved. 40

Hello-World: YAML name: "helloworld" principal: "helloworld-principal" zookeeper: master.mesos:2181 api-port: 8080 pods: helloworld: count: 5 tasks: server: goal: RUNNING cmd: "echo hello world >> hello-container-path/output && sleep 1000" cpus: 1 mem: 256 volumes: - path: "hello-container-path" type: ROOT size: 5000 2017 Mesosphere, Inc. All Rights Reserved. 42

Keep it running Monitoring - Collecting metrics - Routing events - Downstream processing - Alerting - Dashboards - Storage (long-term retention) Logging - Scopes - Local vs. Central - Security considerations 2016 Mesosphere, Inc. All Rights Reserved. 44

Keep it running Maintenance - Cluster Upgrades - Cluster Resizing - Capacity Planning - User & Package Management - Networking Policies - Auditing - Backups & Disaster Recovery Troubleshooting - Debugging - Services - System - Tracing - Chaos Engineering 2016 Mesosphere, Inc. All Rights Reserved. 45