Container 2.0. Container: check! But what about persistent data, big data or fast data?!

Size: px
Start display at page:

Download "Container 2.0. Container: check! But what about persistent data, big data or fast data?!"

Transcription

1 Container 2.0 Container: check! But what about persistent data, big data or fast data?! 1

2 Jörg Schad Distributed Systems Johannes Unterstein Distributed Applications 2016 Mesosphere, Inc. All Rights Reserved. 2

3 In the beginning there was a big Monolith 2017 Mesosphere, Inc. All Rights Reserved. 3

4 COMPUTERS Application Operating System Hardware 2017 Mesosphere, Inc. All Rights Reserved. 4

5 OVERVIEW noun ˈmīkrō/ /ˈsərvəs/ : an approach to application development in which a large application is built as a suite of modular services. Each module supports a specific business goal and uses a simple, well-defined interface to communicate with other modules.* Microservices are designed to be flexible, resilient, efficient, robust, and individually scalable. *From whatis.com 2017 Mesosphere, Inc. All Rights Reserved.

6 MICROSERVICES - Polyglot - Single Responsibility - Smaller Teams - Utilization - Machine types/groups - Dependency hell App Service App Service Operating System App Service Service Operating System Service Service Service Service Operating System Service Machine Machine Machine Infrastructure 2017 Mesosphere, Inc. All Rights Reserved. 6

7 Run everything in containers! Gerard Julien/AFP

8 CONTAINERS - Rapid deployment - Dependency vendoring - Container image repositories - Spreadsheet scheduling App App App Service Service Service Service Service Service Service Service Service Container Runtime Container Runtime Container Runtime OS OS OS Machine Machine Machine Infrastructure 2017 Mesosphere, Inc. All Rights Reserved. 8

9 Demo 1 Chuck Norris Jokes via HTTP vxuoy 2017 Mesosphere, Inc. All Rights Reserved. 9

10 CONTAINERS - Rapid deployment - Dependency vendoring - Container image repositories - Spreadsheet scheduling App App App Service Service Service Service Service Service Service Service Service Container Runtime Container Runtime Container Runtime OS OS OS Machine Machine Machine Infrastructure 2017 Mesosphere, Inc. All Rights Reserved. 10

11 CONTAINER SCHEDULING 2017 Mesosphere, Inc. All Rights Reserved. 11

12 RESOURCE MANAGEMENT 2017 Mesosphere, Inc. All Rights Reserved. 12

13 SERVICE MANAGEMENT 2017 Mesosphere, Inc. All Rights Reserved. 13

14 CONTAINER ORCHESTRATION CONTAINER SCHEDULING RESOURCE MANAGEMENT SERVICE MANAGEMENT - Load Balancing - Readiness Checking 2017 Mesosphere, Inc. All Rights Reserved. 14

15 CONTAINER ORCHESTRATION CONTAINER SCHEDULING - Placement - Replication/Scaling - Resurrection - Rescheduling - Rolling Deployment - Upgrades - Downgrades - Collocation RESOURCE MANAGEMENT - Memory - CPU - GPU - Volumes - Ports - IPs - Images/Artifacts SERVICE MANAGEMENT - Labels - Groups/Namespaces - Dependencies - Load Balancing - Readiness Checking 2017 Mesosphere, Inc. All Rights Reserved. 15

16 CONTAINER ORCHESTRATION Orchestration Web Apps & Services Service Management Scheduling Resource Management Container Runtime Container Runtime Container Runtime Machine & OS Machine & OS Machine & OS Machine Infrastructure 2017 Mesosphere, Inc. All Rights Reserved. 16

17 Meanwhile... MapReduce is crunching Data 2017 Mesosphere, Inc. All Rights Reserved. 17

18 DATA PROCESSING AT HYPERSCALE EVENTS INGEST STORE ANALYZE ACT Ubiquitous data streams from connected devices Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications Sensors Devices Apache Kafka Apache Cassandra Apache Spark Akka Clients DC/OS 2016 Mesosphere, Inc. All Rights Reserved. 18

19 DATA PROCESSING AT HYPERSCALE EVENTS INGEST STORE ANALYZE ACT Ubiquitous data streams from connected devices Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications Sensors Devices Apache Kafka Apache Cassandra Apache Spark Akka Clients DC/OS 2016 Mesosphere, Inc. All Rights Reserved. 19

20 STREAM PROCESSING Apache Storm Apache Spark Apache Samza Apache Flink Apache Apex Concord cloud-only: AWS Kinesis, Google Cloud Dataflow 2016 Mesosphere, Inc. All Rights Reserved. 20

21 EXECUTION MODEL Micro-Batching Native Streaming 2016 Mesosphere, Inc. All Rights Reserved. 21

22 DATA PROCESSING AT HYPERSCALE EVENTS INGEST STORE ANALYZE ACT Ubiquitous data streams from connected devices Ingest millions of events per second Distributed & highly scalable database Real-time and batch process data Visualize data and build data driven applications Sensors Devices Apache Kafka Apache Cassandra Apache Spark Akka Clients DC/OS 2017 Mesosphere, Inc. All Rights Reserved. 22

23 Datastores 2017 Mesosphere, Inc. All Rights Reserved. 23

24 Data Model Relational Key-Value Graph Document Time-Series Files Schema SQL Foreign Keys/Joins OLTP/OLAP Simple Scalable Cache Complex relations Social Graph Recommend ation Fraud detections Schema-Less Semi-structu red queries Product catalogue Session data 2017 Mesosphere, Inc. All Rights Reserved. 24

25 Datacenter 2017 Mesosphere, Inc. All Rights Reserved. 25

26 NAIVE APPROACH Spark/Hadoop Kafka mysql microservice Industry Average 12-15% utilization Cassandra Typical Datacenter siloed, over-provisioned servers, low utilization 2017 Mesosphere, Inc. All Rights Reserved. 26

27 2017 Mesosphere, Inc. All Rights Reserved. 27

28 MULTIPLEXING OF DATA, SERVICES, USERS, ENVIRONMENTS Spark/Hadoop Kafka mysql microservice Cassandra Typical Datacenter siloed, over-provisioned servers, low utilization Mesos, DC/OS automated schedulers, workload multiplexing onto the same machines 2017 Mesosphere, Inc. All Rights Reserved. 28

29 Apache Mesos A top-level Apache project A cluster resource negotiator Scalable to 10,000s of nodes Fault-tolerant, battle-tested An SDK for distributed apps Native Docker support 2017 Mesosphere, Inc. All Rights Reserved. 29

30 2017 Mesosphere, Inc. All Rights Reserved. 30

31 DC/OS ENABLES MODERN DISTRIBUTED APPS Modern App Components Microservices (in containers) Big Data + Analytics Engines Functions & Logic Streaming Batch Machine Learning Search Time Series SQL / NoSQL Analytics Databases Datacenter Operating System (DC/OS) Distributed Systems Kernel (Mesos) Any Infrastructure (Physical, Virtual, Cloud) 31

32 THE BASICS DC/OS is 100% open source (ASL2.0) + A big, diverse community An umbrella for ~30 OSS projects + Roadmap and designs + Docs and tutorials Not limited in any way Familiar, with more features + Networking, Security, CLI, UI, Service Discovery, Load Balancing, Packages,... 32

33 33

34 THE UNIVERSE 34

35 35

36 Demo 2 Scaling DC/OS with Chuck Norris vxuoy 2017 Mesosphere, Inc. All Rights Reserved. 36

37 Container Storage Default Sandbox Simple to use, Task failures Local Persistent Volumes Task failures, (permanent) Node failures Distributed File System/External Storage Node failures, non-local writes 2017 Mesosphere, Inc. All Rights Reserved. 37

38 DC/OS SDK For Stateful Services 2016 Mesosphere, Inc. All Rights Reserved. 38

39 State Machine on Mesos a State Machine for each task: state kept in zookeeper framework runs event loop and handles state changes 2017 Mesosphere, Inc. All Rights Reserved. 39

40 Simple things should be simple; complicated things should be complicated ~100 LOC Defaults Simple YAML interface Default, customizable behaviors Most services probably fall here ~1,000 LOC Custom Plans & Strategies Need to be a service expert Need to write a little Java code Don t need to understand DC/OS 10K+ LOC Build Your Own Scheduler with commons libraries Can t use the defaults Don't mind writing a lot of Java Willing to understand DC/OS 2016 Mesosphere, Inc. All Rights Reserved. 40

41 Simple things should be simple; complicated things should be complicated ~100 LOC Defaults Simple YAML interface Default, customizable behaviors Most services probably fall here ~1,000 LOC Custom Plans & Strategies Need to be a service expert Need to write a little Java code Don t need to understand DC/OS 10K+ LOC Build Your Own Scheduler with commons libraries Can t use the defaults Don't mind writing a lot of Java Willing to understand DC/OS 2016 Mesosphere, Inc. All Rights Reserved. 41

42 Hello-World: YAML name: "helloworld" principal: "helloworld-principal" zookeeper: master.mesos:2181 api-port: 8080 pods: helloworld: count: 5 tasks: server: goal: RUNNING cmd: "echo hello world >> hello-container-path/output && sleep 1000" cpus: 1 mem: 256 volumes: - path: "hello-container-path" type: ROOT size: Mesosphere, Inc. All Rights Reserved. 42

43 Keep it running! 2017 Mesosphere, Inc. All Rights Reserved. 43

44 Keep it running Monitoring - Collecting metrics - Routing events - Downstream processing - Alerting - Dashboards - Storage (long-term retention) Logging - Scopes - Local vs. Central - Security considerations 2016 Mesosphere, Inc. All Rights Reserved. 44

45 Keep it running Maintenance - Cluster Upgrades - Cluster Resizing - Capacity Planning - User & Package Management - Networking Policies - Auditing - Backups & Disaster Recovery Troubleshooting - Debugging - Services - System - Tracing - Chaos Engineering 2016 Mesosphere, Inc. All Rights Reserved. 45

46 Questions? Psssssssst Code: we are hiring! grnh.se/ ie76ru 2017 Mesosphere, Inc. All Rights Reserved. 46

@unterstein #bedcon. Operating microservices with Apache Mesos and DC/OS

@unterstein #bedcon. Operating microservices with Apache Mesos and DC/OS @unterstein @dcos @bedcon #bedcon Operating microservices with Apache Mesos and DC/OS 1 Johannes Unterstein Software Engineer @Mesosphere @unterstein @unterstein.mesosphere 2017 Mesosphere, Inc. All Rights

More information

The SMACK Stack: Spark*, Mesos*, Akka, Cassandra*, Kafka* Elizabeth K. Dublin Apache Kafka Meetup, 30 August 2017.

The SMACK Stack: Spark*, Mesos*, Akka, Cassandra*, Kafka* Elizabeth K. Dublin Apache Kafka Meetup, 30 August 2017. Dublin Apache Kafka Meetup, 30 August 2017 The SMACK Stack: Spark*, Mesos*, Akka, Cassandra*, Kafka* Elizabeth K. Joseph @pleia2 * ASF projects 1 Elizabeth K. Joseph, Developer Advocate Developer Advocate

More information

@joerg_schad Nightmares of a Container Orchestration System

@joerg_schad Nightmares of a Container Orchestration System @joerg_schad Nightmares of a Container Orchestration System 2017 Mesosphere, Inc. All Rights Reserved. 1 Jörg Schad Distributed Systems Engineer @joerg_schad Jan Repnak Support Engineer/ Solution Architect

More information

Advanced Continuous Delivery Strategies for Containerized Applications Using DC/OS

Advanced Continuous Delivery Strategies for Containerized Applications Using DC/OS Advanced Continuous Delivery Strategies for Containerized Applications Using DC/OS ContainerCon @ Open Source Summit North America 2017 Elizabeth K. Joseph @pleia2 1 Elizabeth K. Joseph, Developer Advocate

More information

Building a Data-Friendly Platform for a Data- Driven Future

Building a Data-Friendly Platform for a Data- Driven Future Building a Data-Friendly Platform for a Data- Driven Future Benjamin Hindman - @benh 2016 Mesosphere, Inc. All Rights Reserved. INTRO $ whoami BENJAMIN HINDMAN Co-founder and Chief Architect of Mesosphere,

More information

Distributed Data on Distributed Infrastructure. Claudius Weinberger & Kunal Kusoorkar, ArangoDB Jörg Schad, Mesosphere

Distributed Data on Distributed Infrastructure. Claudius Weinberger & Kunal Kusoorkar, ArangoDB Jörg Schad, Mesosphere Distributed Data on Distributed Infrastructure Claudius Weinberger & Kunal Kusoorkar, ArangoDB Jörg Schad, Mesosphere Kunal Kusoorkar Director Solutions Engineering, ArangoDB @neunhoef Jörg Schad Claudius

More information

Using DC/OS for Continuous Delivery

Using DC/OS for Continuous Delivery Using DC/OS for Continuous Delivery DevPulseCon 2017 Elizabeth K. Joseph, @pleia2 Mesosphere 1 Elizabeth K. Joseph, Developer Advocate, Mesosphere 15+ years working in open source communities 10+ years

More information

Introduction to Mesos and the Datacenter Operating System

Introduction to Mesos and the Datacenter Operating System Introduction to Mesos and the Datacenter Operating System Artem Harutyunyan (artem@mesosphere.io) 2016 Mesosphere, Inc. All Rights Reserved. INTRO $ whoami ARTEM HARUTYUNYAN ALICE Offline (2004-2010) AliEn

More information

Deploying Applications on DC/OS

Deploying Applications on DC/OS Mesosphere Datacenter Operating System Deploying Applications on DC/OS Keith McClellan - Technical Lead, Federal Programs keith.mcclellan@mesosphere.com V6 THE FUTURE IS ALREADY HERE IT S JUST NOT EVENLY

More information

Mesosphere and Percona Server for MongoDB. Jeff Sandstrom, Product Manager (Percona) Ravi Yadav, Tech. Partnerships Lead (Mesosphere)

Mesosphere and Percona Server for MongoDB. Jeff Sandstrom, Product Manager (Percona) Ravi Yadav, Tech. Partnerships Lead (Mesosphere) Mesosphere and Percona Server for MongoDB Jeff Sandstrom, Product Manager (Percona) Ravi Yadav, Tech. Partnerships Lead (Mesosphere) Mesosphere DC/OS MICROSERVICES, CONTAINERS, & DEV TOOLS DATA SERVICES,

More information

Mesosphere and Percona Server for MongoDB. Peter Schwaller, Senior Director Server Eng. (Percona) Taco Scargo, Senior Solution Engineer (Mesosphere)

Mesosphere and Percona Server for MongoDB. Peter Schwaller, Senior Director Server Eng. (Percona) Taco Scargo, Senior Solution Engineer (Mesosphere) Mesosphere and Percona Server for MongoDB Peter Schwaller, Senior Director Server Eng. (Percona) Taco Scargo, Senior Solution Engineer (Mesosphere) Mesosphere DC/OS MICROSERVICES, CONTAINERS, & DEV TOOLS

More information

Scale your Docker containers with Mesos

Scale your Docker containers with Mesos Scale your Docker containers with Mesos Timothy Chen tim@mesosphere.io About me: - Distributed Systems Architect @ Mesosphere - Lead Containerization engineering - Apache Mesos, Drill PMC / Committer

More information

利用 Mesos 打造高延展性 Container 環境. Frank, Microsoft MTC

利用 Mesos 打造高延展性 Container 環境. Frank, Microsoft MTC 利用 Mesos 打造高延展性 Container 環境 Frank, Microsoft MTC About Me Developer @ Yahoo! DevOps @ HTC Technical Architect @ MSFT Agenda About Docker Manage containers Apache Mesos Mesosphere DC/OS application = application

More information

Building/Running Distributed Systems with Apache Mesos

Building/Running Distributed Systems with Apache Mesos Building/Running Distributed Systems with Apache Mesos Philly ETE April 8, 2015 Benjamin Hindman @benh $ whoami 2007-2012 2009-2010 - 2014 my other computer is a datacenter my other computer is a datacenter

More information

Sunil Shah SECURE, FLEXIBLE CONTINUOUS DELIVERY PIPELINES WITH GITLAB AND DC/OS Mesosphere, Inc. All Rights Reserved.

Sunil Shah SECURE, FLEXIBLE CONTINUOUS DELIVERY PIPELINES WITH GITLAB AND DC/OS Mesosphere, Inc. All Rights Reserved. Sunil Shah SECURE, FLEXIBLE CONTINUOUS DELIVERY PIPELINES WITH GITLAB AND DC/OS 1 Introduction MOBILE, SOCIAL & CLOUD ARE RAISING CUSTOMER EXPECTATIONS We need a way to deliver software so fast that our

More information

Modern Stream Processing with Apache Flink

Modern Stream Processing with Apache Flink 1 Modern Stream Processing with Apache Flink Till Rohrmann GOTO Berlin 2017 2 Original creators of Apache Flink da Platform 2 Open Source Apache Flink + da Application Manager 3 What changes faster? Data

More information

MODERN BIG DATA DESIGN PATTERNS CASE DRIVEN DESINGS

MODERN BIG DATA DESIGN PATTERNS CASE DRIVEN DESINGS MODERN BIG DATA DESIGN PATTERNS CASE DRIVEN DESINGS SUJEE MANIYAM FOUNDER / PRINCIPAL @ ELEPHANT SCALE www.elephantscale.com sujee@elephantscale.com HI, I M SUJEE MANIYAM Founder / Principal @ ElephantScale

More information

Mesosphere and the Enterprise: Run Your Applications on Apache Mesos. Steve Wong Open Source Engineer {code} by Dell

Mesosphere and the Enterprise: Run Your Applications on Apache Mesos. Steve Wong Open Source Engineer {code} by Dell Mesosphere and the Enterprise: Run Your Applications on Apache Mesos Steve Wong Open Source Engineer {code} by Dell EMC @cantbewong Open source at Dell EMC {code} by Dell EMC is a group of passionate open

More information

SCALING LIKE TWITTER WITH APACHE MESOS

SCALING LIKE TWITTER WITH APACHE MESOS Philip Norman & Sunil Shah SCALING LIKE TWITTER WITH APACHE MESOS 1 MODERN INFRASTRUCTURE Dan the Datacenter Operator Alice the Application Developer Doesn t sleep very well Loves automation Wants to control

More information

Using the SDACK Architecture to Build a Big Data Product. Yu-hsin Yeh (Evans Ye) Apache Big Data NA 2016 Vancouver

Using the SDACK Architecture to Build a Big Data Product. Yu-hsin Yeh (Evans Ye) Apache Big Data NA 2016 Vancouver Using the SDACK Architecture to Build a Big Data Product Yu-hsin Yeh (Evans Ye) Apache Big Data NA 2016 Vancouver Outline A Threat Analytic Big Data product The SDACK Architecture Akka Streams and data

More information

Microsoft Azure Databricks for data engineering. Building production data pipelines with Apache Spark in the cloud

Microsoft Azure Databricks for data engineering. Building production data pipelines with Apache Spark in the cloud Microsoft Azure Databricks for data engineering Building production data pipelines with Apache Spark in the cloud Azure Databricks As companies continue to set their sights on making data-driven decisions

More information

Big Data Technology Ecosystem. Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara

Big Data Technology Ecosystem. Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara Big Data Technology Ecosystem Mark Burnette Pentaho Director Sales Engineering, Hitachi Vantara Agenda End-to-End Data Delivery Platform Ecosystem of Data Technologies Mapping an End-to-End Solution Case

More information

A Generic Microservice Architecture for Environmental Data Management

A Generic Microservice Architecture for Environmental Data Management A Generic Microservice Architecture for Environmental Data Management Clemens Düpmeier, Eric Braun, Thorsten Schlachter, Karl-Uwe Stucky, Wolfgang Suess KIT The Research University in the Helmholtz Association

More information

Big Data. Big Data Analyst. Big Data Engineer. Big Data Architect

Big Data. Big Data Analyst. Big Data Engineer. Big Data Architect Big Data Big Data Analyst INTRODUCTION TO BIG DATA ANALYTICS ANALYTICS PROCESSING TECHNIQUES DATA TRANSFORMATION & BATCH PROCESSING REAL TIME (STREAM) DATA PROCESSING Big Data Engineer BIG DATA FOUNDATION

More information

The Hadoop Ecosystem. EECS 4415 Big Data Systems. Tilemachos Pechlivanoglou

The Hadoop Ecosystem. EECS 4415 Big Data Systems. Tilemachos Pechlivanoglou The Hadoop Ecosystem EECS 4415 Big Data Systems Tilemachos Pechlivanoglou tipech@eecs.yorku.ca A lot of tools designed to work with Hadoop 2 HDFS, MapReduce Hadoop Distributed File System Core Hadoop component

More information

How to Keep UP Through Digital Transformation with Next-Generation App Development

How to Keep UP Through Digital Transformation with Next-Generation App Development How to Keep UP Through Digital Transformation with Next-Generation App Development Peter Sjoberg Jon Olby A Look Back, A Look Forward Dedicated, data structure dependent, inefficient, virtualized Infrastructure

More information

Deploying and Operating Cloud Native.NET apps

Deploying and Operating Cloud Native.NET apps Deploying and Operating Cloud Native.NET apps Jenny McLaughlin, Sr. Platform Architect Cornelius Mendoza, Sr. Platform Architect Pivotal Cloud Native Practices Continuous Delivery DevOps Microservices

More information

WHITEPAPER. MemSQL Enterprise Feature List

WHITEPAPER. MemSQL Enterprise Feature List WHITEPAPER MemSQL Enterprise Feature List 2017 MemSQL Enterprise Feature List DEPLOYMENT Provision and deploy MemSQL anywhere according to your desired cluster configuration. On-Premises: Maximize infrastructure

More information

Flash Storage Complementing a Data Lake for Real-Time Insight

Flash Storage Complementing a Data Lake for Real-Time Insight Flash Storage Complementing a Data Lake for Real-Time Insight Dr. Sanhita Sarkar Global Director, Analytics Software Development August 7, 2018 Agenda 1 2 3 4 5 Delivering insight along the entire spectrum

More information

Cisco Tetration Analytics

Cisco Tetration Analytics Cisco Tetration Analytics Enhanced security and operations with real time analytics John Joo Tetration Business Unit Cisco Systems Security Challenges in Modern Data Centers Securing applications has become

More information

Advantages of using DC/OS Azure infrastructure and the implementation architecture Bill of materials used to construct DC/OS and the ACS clusters

Advantages of using DC/OS Azure infrastructure and the implementation architecture Bill of materials used to construct DC/OS and the ACS clusters Reference implementation: The Azure Container Service DC/OS is a distributed operating system powered by Apache Mesos that treats collections of CPUs, RAM, networking and so on as a distributed kernel

More information

Docker and Oracle Everything You Wanted To Know

Docker and Oracle Everything You Wanted To Know Docker and Oracle Everything You Wanted To Know June, 2017 Umesh Tanna Principal Technology Sales Consultant Oracle Sales Consulting Centers(SCC) Bangalore Safe Harbor Statement The following is intended

More information

Activator Library. Focus on maximizing the value of your data, gain business insights, increase your team s productivity, and achieve success.

Activator Library. Focus on maximizing the value of your data, gain business insights, increase your team s productivity, and achieve success. Focus on maximizing the value of your data, gain business insights, increase your team s productivity, and achieve success. ACTIVATORS Designed to give your team assistance when you need it most without

More information

AGILE DEVELOPMENT AND PAAS USING THE MESOSPHERE DCOS

AGILE DEVELOPMENT AND PAAS USING THE MESOSPHERE DCOS Sunil Shah AGILE DEVELOPMENT AND PAAS USING THE MESOSPHERE DCOS 1 THE DATACENTER OPERATING SYSTEM (DCOS) 2 DCOS INTRODUCTION The Mesosphere Datacenter Operating System (DCOS) is a distributed operating

More information

AWS Lambda: Event-driven Code in the Cloud

AWS Lambda: Event-driven Code in the Cloud AWS Lambda: Event-driven Code in the Cloud Dean Bryen, Solutions Architect AWS Andrew Wheat, Senior Software Engineer - BBC April 15, 2015 London, UK 2015, Amazon Web Services, Inc. or its affiliates.

More information

Designing MQ deployments for the cloud generation

Designing MQ deployments for the cloud generation Designing MQ deployments for the cloud generation WebSphere User Group, London Arthur Barr, Senior Software Engineer, IBM MQ 30 th March 2017 Top business drivers for cloud 2 Source: OpenStack user survey,

More information

MapR Enterprise Hadoop

MapR Enterprise Hadoop 2014 MapR Technologies 2014 MapR Technologies 1 MapR Enterprise Hadoop Top Ranked Cloud Leaders 500+ Customers 2014 MapR Technologies 2 Key MapR Advantage Partners Business Services APPLICATIONS & OS ANALYTICS

More information

Oracle Application Container Cloud

Oracle Application Container Cloud Oracle Application Container Cloud Matthew Baldwin Principal Product Manager Safe Harbor Statement The following is intended to outline our general product direction. It is intended for information purposes

More information

Extend NonStop Applications with Cloud-based Services. Phil Ly, TIC Software John Russell, Canam Software

Extend NonStop Applications with Cloud-based Services. Phil Ly, TIC Software John Russell, Canam Software Extend NonStop Applications with Cloud-based Services Phil Ly, TIC Software John Russell, Canam Software Agenda Cloud Computing and Microservices Amazon Web Services (AWS) Integrate NonStop with AWS Managed

More information

Containerization Dockers / Mesospere. Arno Keller HPE

Containerization Dockers / Mesospere. Arno Keller HPE Containerization Dockers / Mesospere Arno Keller HPE What is the Container technology Hypervisor vs. Containers (Huis vs artement) A container doesn't "boot" an OS instead it loads the application and

More information

DATA SCIENCE USING SPARK: AN INTRODUCTION

DATA SCIENCE USING SPARK: AN INTRODUCTION DATA SCIENCE USING SPARK: AN INTRODUCTION TOPICS COVERED Introduction to Spark Getting Started with Spark Programming in Spark Data Science with Spark What next? 2 DATA SCIENCE PROCESS Exploratory Data

More information

The Emergence of the Datacenter Developer. Tobi Knaup, Co-Founder & CTO at

The Emergence of the Datacenter Developer. Tobi Knaup, Co-Founder & CTO at The Emergence of the Datacenter Developer Tobi Knaup, Co-Founder & CTO at Mesosphere @superguenter A Brief History of Operating Systems 2 1950 s Mainframes Punchcards No operating systems Time Sharing

More information

CS-580K/480K Advanced Topics in Cloud Computing. Container III

CS-580K/480K Advanced Topics in Cloud Computing. Container III CS-580/480 Advanced Topics in Cloud Computing Container III 1 Docker Container https://www.docker.com/ Docker is a platform for developers and sysadmins to develop, deploy, and run applications with containers.

More information

STATE OF MODERN APPLICATIONS IN THE CLOUD

STATE OF MODERN APPLICATIONS IN THE CLOUD STATE OF MODERN APPLICATIONS IN THE CLOUD 2017 Introduction The Rise of Modern Applications What is the Modern Application? Today s leading enterprises are striving to deliver high performance, highly

More information

Big Data Architect.

Big Data Architect. Big Data Architect www.austech.edu.au WHAT IS BIG DATA ARCHITECT? A big data architecture is designed to handle the ingestion, processing, and analysis of data that is too large or complex for traditional

More information

Bring Context To Your Machine Data With Hadoop, RDBMS & Splunk

Bring Context To Your Machine Data With Hadoop, RDBMS & Splunk Bring Context To Your Machine Data With Hadoop, RDBMS & Splunk Raanan Dagan and Rohit Pujari September 25, 2017 Washington, DC Forward-Looking Statements During the course of this presentation, we may

More information

Introduction to Big-Data

Introduction to Big-Data Introduction to Big-Data Ms.N.D.Sonwane 1, Mr.S.P.Taley 2 1 Assistant Professor, Computer Science & Engineering, DBACER, Maharashtra, India 2 Assistant Professor, Information Technology, DBACER, Maharashtra,

More information

API, DEVOPS & MICROSERVICES

API, DEVOPS & MICROSERVICES API, DEVOPS & MICROSERVICES RAPID. OPEN. SECURE. INNOVATION TOUR 2018 April 26 Singapore 1 2018 Software AG. All rights reserved. For internal use only THE NEW ARCHITECTURAL PARADIGM Microservices Containers

More information

Containers or Serverless? Mike Gillespie Solutions Architect, AWS Solutions Architecture

Containers or Serverless? Mike Gillespie Solutions Architect, AWS Solutions Architecture Containers or Serverless? Mike Gillespie Solutions Architect, AWS Solutions Architecture A Typical Application with Microservices Client Webapp Webapp Webapp Greeting Greeting Greeting Name Name Name Microservice

More information

Cloud Analytics and Business Intelligence on AWS

Cloud Analytics and Business Intelligence on AWS Cloud Analytics and Business Intelligence on AWS Enterprise Applications Virtual Desktops Sharing & Collaboration Platform Services Analytics Hadoop Real-time Streaming Data Machine Learning Data Warehouse

More information

Increase Value from Big Data with Real-Time Data Integration and Streaming Analytics

Increase Value from Big Data with Real-Time Data Integration and Streaming Analytics Increase Value from Big Data with Real-Time Data Integration and Streaming Analytics Cy Erbay Senior Director Striim Executive Summary Striim is Uniquely Qualified to Solve the Challenges of Real-Time

More information

Reactive Microservices Architecture on AWS

Reactive Microservices Architecture on AWS Reactive Microservices Architecture on AWS Sascha Möllering Solutions Architect, @sascha242, Amazon Web Services Germany GmbH Why are we here today? https://secure.flickr.com/photos/mgifford/4525333972

More information

Apache Flink Big Data Stream Processing

Apache Flink Big Data Stream Processing Apache Flink Big Data Stream Processing Tilmann Rabl Berlin Big Data Center www.dima.tu-berlin.de bbdc.berlin rabl@tu-berlin.de XLDB 11.10.2017 1 2013 Berlin Big Data Center All Rights Reserved DIMA 2017

More information

Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme STO1794BE Evolution of vsan Vijay Ramachandran Sr.Director, Product Management Christos Karamanolis VMware Fellow & CTO of Storage & Availability #VMworld #STO1794BE Disclaimer This presentation may contain

More information

How we built a highly scalable Machine Learning platform using Apache Mesos

How we built a highly scalable Machine Learning platform using Apache Mesos How we built a highly scalable Machine Learning platform using Apache Mesos Daniel Sârbe Development Manager, BigData and Cloud Machine Translation @ SDL Co-founder of BigData/DataScience Meetup Cluj,

More information

MESOS A State-Of-The-Art Container Orchestrator Mesosphere, Inc. All Rights Reserved. 1

MESOS A State-Of-The-Art Container Orchestrator Mesosphere, Inc. All Rights Reserved. 1 MESOS A State-Of-The-Art Container Orchestrator 2016 Mesosphere, Inc. All Rights Reserved. 1 About me Jie Yu (@jie_yu) Tech Lead at Mesosphere Mesos PMC member and committer Formerly worked at Twitter

More information

Hadoop 2.x Core: YARN, Tez, and Spark. Hortonworks Inc All Rights Reserved

Hadoop 2.x Core: YARN, Tez, and Spark. Hortonworks Inc All Rights Reserved Hadoop 2.x Core: YARN, Tez, and Spark YARN Hadoop Machine Types top-of-rack switches core switch client machines have client-side software used to access a cluster to process data master nodes run Hadoop

More information

WHITE PAPER. RedHat OpenShift Container Platform. Benefits: Abstract. 1.1 Introduction

WHITE PAPER. RedHat OpenShift Container Platform. Benefits: Abstract. 1.1 Introduction WHITE PAPER RedHat OpenShift Container Platform Abstract Benefits: Applications are designed around smaller independent components called microservices. Elastic resources: Scale up or down quickly and

More information

Qualys Cloud Platform

Qualys Cloud Platform 18 QUALYS SECURITY CONFERENCE 2018 Qualys Cloud Platform Looking Under the Hood: What Makes Our Cloud Platform so Scalable and Powerful Dilip Bachwani Vice President, Engineering, Qualys, Inc. Cloud Platform

More information

Creating a Recommender System. An Elasticsearch & Apache Spark approach

Creating a Recommender System. An Elasticsearch & Apache Spark approach Creating a Recommender System An Elasticsearch & Apache Spark approach My Profile SKILLS Álvaro Santos Andrés Big Data & Analytics Solution Architect in Ericsson with more than 12 years of experience focused

More information

Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme CNA1612BU Deploying real-world workloads on Kubernetes and Pivotal Cloud Foundry VMworld 2017 Fred Melo, Director of Technology, Pivotal Merlin Glynn, Sr. Technical Product Manager, VMware Content: Not

More information

70-532: Developing Microsoft Azure Solutions

70-532: Developing Microsoft Azure Solutions 70-532: Developing Microsoft Azure Solutions Exam Design Target Audience Candidates of this exam are experienced in designing, programming, implementing, automating, and monitoring Microsoft Azure solutions.

More information

The Idiot s Guide to Quashing MicroServices. Hani Suleiman

The Idiot s Guide to Quashing MicroServices. Hani Suleiman The Idiot s Guide to Quashing MicroServices Hani Suleiman The Promised Land Welcome to Reality Logging HA/DR Monitoring Provisioning Security Debugging Enterprise frameworks Don t Panic WHOAMI I wrote

More information

Deploying and Operating Cloud Native.NET apps

Deploying and Operating Cloud Native.NET apps Deploying and Operating Cloud Native.NET apps Jenny McLaughlin, Sr. Platform Architect Cornelius Mendoza, Sr. Platform Architect Pivotal Cloud Native Practices Continuous Delivery DevOps Microservices

More information

Hadoop, Yarn and Beyond

Hadoop, Yarn and Beyond Hadoop, Yarn and Beyond 1 B. R A M A M U R T H Y Overview We learned about Hadoop1.x or the core. Just like Java evolved, Java core, Java 1.X, Java 2.. So on, software and systems evolve, naturally.. Lets

More information

Security and Performance advances with Oracle Big Data SQL

Security and Performance advances with Oracle Big Data SQL Security and Performance advances with Oracle Big Data SQL Jean-Pierre Dijcks Oracle Redwood Shores, CA, USA Key Words SQL, Oracle, Database, Analytics, Object Store, Files, Big Data, Big Data SQL, Hadoop,

More information

Launching StarlingX. The Journey to Drive Compute to the Edge Pilot Project Supported by the OpenStack

Launching StarlingX. The Journey to Drive Compute to the Edge Pilot Project Supported by the OpenStack Launching StarlingX The Journey to Drive Compute to the Edge Pilot Project Supported by the OpenStack Foundation Ian Jolliffe, WIND RIVER SYSTEMS Director Engineering @ian_jolliffe Project Overview An

More information

Apache Ignite TM - In- Memory Data Fabric Fast Data Meets Open Source

Apache Ignite TM - In- Memory Data Fabric Fast Data Meets Open Source Apache Ignite TM - In- Memory Data Fabric Fast Data Meets Open Source DMITRIY SETRAKYAN Founder, PPMC https://ignite.apache.org @apacheignite @dsetrakyan Agenda About In- Memory Computing Apache Ignite

More information

Cloud Providers more AWS, Aneka

Cloud Providers more AWS, Aneka Basics of Cloud Computing Lecture 6 Cloud Providers more AWS, Aneka and GAE Satish Srirama Outline More AWS Some more PaaS Aneka Google App Engine Force.com 16.05.2012 Satish Srirama 2/51 Recap Last lecture

More information

Developing Microsoft Azure Solutions (70-532) Syllabus

Developing Microsoft Azure Solutions (70-532) Syllabus Developing Microsoft Azure Solutions (70-532) Syllabus Cloud Computing Introduction What is Cloud Computing Cloud Characteristics Cloud Computing Service Models Deployment Models in Cloud Computing Advantages

More information

Cloud + Big Data Putting it all Together

Cloud + Big Data Putting it all Together Cloud + Big Data Putting it all Together Even Solberg 2009 VMware Inc. All rights reserved 2 Big, Fast and Flexible Data Big Big Data Processing Fast OLTP workloads Flexible Document Object Big Data Analytics

More information

Ingest. David Pilato, Developer Evangelist Paris, 31 Janvier 2017

Ingest. David Pilato, Developer Evangelist Paris, 31 Janvier 2017 Ingest David Pilato, Developer Evangelist Paris, 31 Janvier 2017 Data Ingestion The process of collecting and importing data for immediate use in a datastore 2 ? Simple things should be simple. Shay Banon

More information

We are ready to serve Latest Testing Trends, Are you ready to learn?? New Batches Info

We are ready to serve Latest Testing Trends, Are you ready to learn?? New Batches Info We are ready to serve Latest Testing Trends, Are you ready to learn?? New Batches Info START DATE : TIMINGS : DURATION : TYPE OF BATCH : FEE : FACULTY NAME : LAB TIMINGS : PH NO: 9963799240, 040-40025423

More information

Fluentd + MongoDB + Spark = Awesome Sauce

Fluentd + MongoDB + Spark = Awesome Sauce Fluentd + MongoDB + Spark = Awesome Sauce Nishant Sahay, Sr. Architect, Wipro Limited Bhavani Ananth, Tech Manager, Wipro Limited Your company logo here Wipro Open Source Practice: Vision & Mission Vision

More information

Western Michigan University

Western Michigan University CS-6030 Cloud compu;ng Google App engine Sepideh Mohammadi Summer II 2017 Western Michigan University content Categories of cloud compu;ng Google cloud plaborm Google App Engine Storage technologies Datastore

More information

Hadoop An Overview. - Socrates CCDH

Hadoop An Overview. - Socrates CCDH Hadoop An Overview - Socrates CCDH What is Big Data? Volume Not Gigabyte. Terabyte, Petabyte, Exabyte, Zettabyte - Due to handheld gadgets,and HD format images and videos - In total data, 90% of them collected

More information

/ Cloud Computing. Recitation 5 February 14th, 2017

/ Cloud Computing. Recitation 5 February 14th, 2017 15-319 / 15-619 Cloud Computing Recitation 5 February 14th, 2017 1 Overview Administrative issues Office Hours, Piazza guidelines Last week s reflection Project 2.1, OLI Unit 2 modules 5 and 6 This week

More information

Ingest. Aaron Mildenstein, Consulting Architect Tokyo Dec 14, 2017

Ingest. Aaron Mildenstein, Consulting Architect Tokyo Dec 14, 2017 Ingest Aaron Mildenstein, Consulting Architect Tokyo Dec 14, 2017 Data Ingestion The process of collecting and importing data for immediate use 2 ? Simple things should be simple. Shay Banon Elastic{ON}

More information

Containers, Serverless and Functions in a nutshell. Eugene Fedorenko

Containers, Serverless and Functions in a nutshell. Eugene Fedorenko Containers, Serverless and Functions in a nutshell Eugene Fedorenko About me Eugene Fedorenko Senior Architect Flexagon adfpractice-fedor.blogspot.com @fisbudo Agenda Containers Microservices Docker Kubernetes

More information

Intro to Big Data on AWS Igor Roiter Big Data Cloud Solution Architect

Intro to Big Data on AWS Igor Roiter Big Data Cloud Solution Architect Intro to Big Data on AWS Igor Roiter Big Data Cloud Solution Architect Igor Roiter Big Data Cloud Solution Architect Working as a Data Specialist for the last 11 years 9 of them as a Consultant specializing

More information

Learn. Connect. Explore.

Learn. Connect. Explore. Learn. Connect. Explore. No More Storage Nightmares An Open Solution for Container Persistent Storage Learn. Connect. Explore. CONTAINERS vs VIRTUALIZATION Containers Abstracts OS Kernel Mostly Linux One

More information

Esper EQC. Horizontal Scale-Out for Complex Event Processing

Esper EQC. Horizontal Scale-Out for Complex Event Processing Esper EQC Horizontal Scale-Out for Complex Event Processing Esper EQC - Introduction Esper query container (EQC) is the horizontal scale-out architecture for Complex Event Processing with Esper and EsperHA

More information

Issues Fixed in DC/OS

Issues Fixed in DC/OS Release Notes for 1.10.4 These are the release notes for DC/OS 1.10.4. DOWNLOAD DC/OS OPEN SOURCE Issues Fixed in DC/OS 1.10.4 CORE-1375 - Docker executor does not hang due to lost messages. DOCS-2169

More information

Integrate MATLAB Analytics into Enterprise Applications

Integrate MATLAB Analytics into Enterprise Applications Integrate Analytics into Enterprise Applications Aurélie Urbain MathWorks Consulting Services 2015 The MathWorks, Inc. 1 Data Analytics Workflow Data Acquisition Data Analytics Analytics Integration Business

More information

CONTINUOUS DELIVERY WITH MESOS, DC/OS AND JENKINS

CONTINUOUS DELIVERY WITH MESOS, DC/OS AND JENKINS APACHE MESOS NYC MEETUP SEPTEMBER 22, 2016 CONTINUOUS DELIVERY WITH MESOS, DC/OS AND JENKINS WHO WE ARE ROGER IGNAZIO SUNIL SHAH Tech Lead at Mesosphere @rogerignazio Product Manager at Mesosphere @ssk2

More information

Microservices. Chaos Kontrolle mit Kubernetes. Robert Kubis - Developer Advocate,

Microservices. Chaos Kontrolle mit Kubernetes. Robert Kubis - Developer Advocate, Microservices Chaos Kontrolle mit Kubernetes Robert Kubis - Developer Advocate, Google @hostirosti About me Robert Kubis Developer Advocate Google Cloud Platform London, UK hostirosti github.com/hostirosti

More information

Architecting Microsoft Azure Solutions (proposed exam 535)

Architecting Microsoft Azure Solutions (proposed exam 535) Architecting Microsoft Azure Solutions (proposed exam 535) IMPORTANT: Significant changes are in progress for exam 534 and its content. As a result, we are retiring this exam on December 31, 2017, and

More information

Cloud-Native Applications. Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0

Cloud-Native Applications. Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0 Cloud-Native Applications Copyright 2017 Pivotal Software, Inc. All rights Reserved. Version 1.0 Cloud-Native Characteristics Lean Form a hypothesis, build just enough to validate or disprove it. Learn

More information

App Service Overview. Rand Pagels Azure Technical Specialist - Application Development US Great Lakes Region

App Service Overview. Rand Pagels Azure Technical Specialist - Application Development US Great Lakes Region App Service Overview Quickly create powerful cloud apps using a fully-managed platform Rand Pagels Azure Technical Specialist - Application Development US Great Lakes Region Security & Management Platform

More information

POWERING THE INTERNET WITH APACHE MESOS

POWERING THE INTERNET WITH APACHE MESOS Neil Conway, Niklas Nielsen, Greg Mann & Sunil Shah POWERING THE INTERNET WITH APACHE MESOS 1 MESOS: ORIGINS 2 THE BIRTH OF MESOS TWITTER TECH TALK APACHE INCUBATION The grad students working on Mesos

More information

Running MarkLogic in Containers (Both Docker and Kubernetes)

Running MarkLogic in Containers (Both Docker and Kubernetes) Running MarkLogic in Containers (Both Docker and Kubernetes) Emma Liu Product Manager, MarkLogic Vitaly Korolev Staff QA Engineer, MarkLogic @vitaly_korolev 4 June 2018 MARKLOGIC CORPORATION Source: http://turnoff.us/image/en/tech-adoption.png

More information

Things Every Oracle DBA Needs to Know about the Hadoop Ecosystem. Zohar Elkayam

Things Every Oracle DBA Needs to Know about the Hadoop Ecosystem. Zohar Elkayam Things Every Oracle DBA Needs to Know about the Hadoop Ecosystem Zohar Elkayam www.realdbamagic.com Twitter: @realmgic Who am I? Zohar Elkayam, CTO at Brillix Programmer, DBA, team leader, database trainer,

More information

CONTINUOUS DELIVERY WITH DC/OS AND JENKINS

CONTINUOUS DELIVERY WITH DC/OS AND JENKINS SOFTWARE ARCHITECTURE NOVEMBER 15, 2016 CONTINUOUS DELIVERY WITH DC/OS AND JENKINS AGENDA Presentation Introduction to Apache Mesos and DC/OS Components that make up modern infrastructure Running Jenkins

More information

Edge Foundational Training

Edge Foundational Training Edge Foundational Training Give your team the tools to get up and running with Edge Edge Foundational Training provides the tools and information needed to start using Edge whether in the cloud or on premises.

More information

The Future of Real-Time in Spark

The Future of Real-Time in Spark The Future of Real-Time in Spark Reynold Xin @rxin Spark Summit, New York, Feb 18, 2016 Why Real-Time? Making decisions faster is valuable. Preventing credit card fraud Monitoring industrial machinery

More information

TIBCO Complex Event Processing Evaluation Guide

TIBCO Complex Event Processing Evaluation Guide TIBCO Complex Event Processing Evaluation Guide This document provides a guide to evaluating CEP technologies. http://www.tibco.com Global Headquarters 3303 Hillview Avenue Palo Alto, CA 94304 Tel: +1

More information

Exam : Implementing Microsoft Azure Infrastructure Solutions

Exam : Implementing Microsoft Azure Infrastructure Solutions Exam 70-533: Implementing Microsoft Azure Infrastructure Solutions Objective Domain Note: This document shows tracked changes that are effective as of January 18, 2018. Design and Implement Azure App Service

More information

Onto Petaflops with Kubernetes

Onto Petaflops with Kubernetes Onto Petaflops with Kubernetes Vishnu Kannan Google Inc. vishh@google.com Key Takeaways Kubernetes can manage hardware accelerators at Scale Kubernetes provides a playground for ML ML journey with Kubernetes

More information

EASILY DEPLOY AND SCALE KUBERNETES WITH RANCHER

EASILY DEPLOY AND SCALE KUBERNETES WITH RANCHER EASILY DEPLOY AND SCALE KUBERNETES WITH RANCHER 2 WHY KUBERNETES? Kubernetes is an open-source container orchestrator for deploying and managing containerized applications. Building on 15 years of experience

More information

Towards a Real- time Processing Pipeline: Running Apache Flink on AWS

Towards a Real- time Processing Pipeline: Running Apache Flink on AWS Towards a Real- time Processing Pipeline: Running Apache Flink on AWS Dr. Steffen Hausmann, Solutions Architect Michael Hanisch, Manager Solutions Architecture November 18 th, 2016 Stream Processing Challenges

More information