Scheduling Computational and Storage Resources on the NRP

Size: px
Start display at page:

Download "Scheduling Computational and Storage Resources on the NRP"

Transcription

1 Scheduling Computational and Storage Resources on the NRP Rob Gardner Dima Mishin University of Chicago UCSD Second NRP Workshop Montana State University August 6-7, 2018 slides: 1

2 Kubernetes scheduling resources CPU, RAM: QoS: Request Limit Guaranteed Burstable Best Effort Preemption Eviction Pod priority Resources starvation 2

3 Kubernetes scheduling resources GPU: Request Extended resources: Request Limit Schedulers: Default Can write your own Can run multiple nrp.io/tpu: 16 3

4 Federation of clusters Managing policies across clusters Common labels for nodes, namespaces Respect local policies when scheduling resources 4

5 Federation of clusters Names conflicts Requires custom dataplane running in federation cluster to enforce common policies 5

6 Federation of clusters Mounting storage from other clusters Data locality? 6

7 Federation V1 Sync resources across clusters Deploy to multiple clusters Cross cluster discovery Allow workloads communicate Common DNS for all clusters 7

8 Federation V2 (documents, almost no code) 8

9 approaches to scaling federations slateci.io 9

10 Federation Topology - how to scale? 10

11 Interoperability for Platforms Create science platforms across institutions and facilities Deal with access, privilege & security in shared environment Adopt a virtual organization trust model Site autonomy and policies respected Cluster groupings, WAN-stretched clusters or regional federations 11

12 SLATE Concepts & Components Containerized services in managed clusters Widely used open source technologies for growth and sustainability SLATE additions Curated services Create a Loose federation of clusters & platforms 12

13 Globus Auth signup/login developers (& admins) cluster admins 13

14 Platform Client Access for App Developers 14

15 Platform Client Access for App Developers 15

16 Platform Client Access for App Developers We'll need both 16

17 Policy and Trust SLATE applications will be curated into a trusted application catalog Applications must define and request all needed network, disk, device, etc access. Think application permissions on your phone Site policies must be respected Access, privileges, capabilities are controlled and transparent 17

18 Deploying an "Application" -like 18

19 scheduling pods for science 19

20 containerized by Machine Learning for HL-LHC on PRP scripted kubectl to PRP

21 containerized by Teaching Platform on PRP/CHASE-CI Users request CPU and GPU Service kubectl to PRP CHASE-CI gpus 21

22 scheduling persistent services 22

23 containerized by Globus Connect Service Goal: Automate deployment of Globus endpoints on DTN nodes in SciDMZ Containerize Globus Connect Service GCSv5 - service deployed, testing Helm charts for easy installation Investigating how to ease integration to campus storage 23

24 containerized by OSG StashCache StashCache: XRootD-based data federation used by OSG to support data delivery for LIGO, CMS and individual researchers to computing sites c.f. next talk by F. Wuerthwein: deployments on PRP and Internet2 Bring experiment data from large "data lakes" to fast local caches on the edge, adjacent to the compute Provide O(10-100TB) storage, shared among virtual organizations and connected to HPC LAN Progress is good - Containerized and working on creating Helm chart Pattern for other LHC XRootD-based federations (XCache, AAA) 24

25 scheduling batch science workflows schedule needed service pods first overlay an application sheduler 25

26 Scheduling Science Applications (w/ familiar tools)

27 Scheduling Science Applications (w/ familiar tools) Minutes after pod deployment on FIONA8, MIT student doing quantum simulations for materials 27

28 Scheduling Science Applications (w/ familiar tools) OSG to SLATE 28

29 Scheduling Science Applications (w/ familiar tools) Similar efforts at CERN - Keynote at KubeCon EU 2018 (video) slides 29

30 Scheduling Storage OSiRIS (CC*DNI DIBBS 2015, NSF award # ) is building a distributed Ceph-based, multi-institutional storage infrastructure that lets researchers write, manage, and share data from their own computing facility locations ( This storage is VO-allocatable and potentially dynamically schedulable Collaboration between SLATE and OSiRIS focuses on software defined networking to orchestrate our infrastructures Next up: exploring OSiRIS block storage hosting SLATE containers usable across one or more SLATE platform deployments Future: OSiRIS storage dynamically allocated for SLATE VOs 30

31 Summary & Conclusions Multiple projects tackling scheduling and resource sharing on our emerging national research platform Much progress in deployments & federated edge cluster models challenges ahead applying policy-driven scheduling: cluster-level, federation-level, multi-federations Meanwhile we can keep the NRP filled with science apps Groups are forging best practices and design patterns for application DevOps, and other tools in a k8s-enabled ecosystem All good news for multi-institution science collaborations and the CI engineers providing the infrastructure! 31

32 Thank you! & Acknowledgements PRP: NSF OAC "CC*DNI DIBBs: The Pacific Research Platform" CHASE-CI: NSF CNS "CI-New: Cognitive Hardware and Software Ecosystem Community Infrastructure" SLATE: NSF OAC "CIF21 DIBBs: EI: SLATE and the Mobility of Capability" OSiRIS: CC*DNI DIBBS 2015, NSF award # OSG: NSF PHY The Open Science Grid, The Next Five Years: Distributed High Throughput Computing for the Nation's Scientists, Researchers, Educators, and Students VC3: Department of Energy ASCR/NGNS DDRM project "Virtual Clusters for Community Computation" US ATLAS Operations: NSF PHY "U.S. ATLAS Operations: Discovery and Measurement at the Energy Frontier" slides: 32

33 More "App" Examples 33

34 containerized by perfsonar Make as lightweight containers as possible for easy deployment across clusters Integration with a Slate Platform Provider Central perfsonar Node Automated as much as possible while maintaining full flexibility of the testpoint bundle Allows reduction of requirements per testpoint and allows for easy centralized access 34

35 containerized by Squid proxies Created containers running the OSG packaging of "Frontier-Squid" Important for software caching with CERN CVMFS Containers are designed to expose the full Squid configuration so that they can be used in an arbitrary environment Packaged the containers into a SLATE application The application exposes a minimum set of parameters which coordinates the configuration of Squid, kubernetes and SLATE For more information see: ons/frontiersquid/ 35

36 slate extra 36

37 Scheduling Storage OSiRIS (CC*DNI DIBBS 2015, NSF award # ) is building a distributed Ceph-based, multi-institutional storage infrastructure that lets researchers write, manage, and share data from their own computing facility locations ( This storage is VO-allocatable and potentially dynamically schedulable A Challenge: Multi-site Ceph clusters take a performance hit with latency OSiRIS user provisioning chain can be used to assign users and VO storage pools to separate clusters - data replication out-of-band. Ceph RGW (S3) has concept of geographic zones and built-in replication between them though this is not yet used in OSiRIS (we are one large shared cluster with RGW endpoints at each site) 37

38 OSIRIS-SLATE Collaboration Collaboration between SLATE and OSiRIS focuses on software defined networking to orchestrate our infrastructures Next up: exploring OSiRIS block storage hosting SLATE containers usable across one or more SLATE platform deployments Current Kubernetes Ceph support is in a state of flux with migration to Container Storage Interface (CSI) based provisioning CSI Ceph plugin requires admin level credentials to enable dynamic provisioning Pre-provisioned volumes can be attached without admin credentials. 38

39 SLATE Vision Building Block for Multi-Institution Research Platforms 39

40 Services as a Service! SLATE is designed from the ground up to be programmable and scalable Applications specified and deployed to sites in a declarative way $ slate-client app --vo mylab --cluster dmzk8s install myapp Applications are monitored through their entire lifecycle, re-deployed if failure occurs 40

41 A New Stack for Research Platforms Rethinking the stack from the ground up. Minimal OS running a container orchestration engine Advanced networking integrated Everything is ephemeral System state lives "in the cloud" Site administrators only have to worry about hardware 41

42 Monitoring: Apps & Infrastructure SLATE catalog apps have optional flag for granular logs shipped to its own index in central elasticsearch cluster. All SLATE sites forward metrics automatically to SLATE Elasticsearch cluster. All SLATE members has access to Kibana dashboard. 42

43 SLATE Core Services 43

44 Analytics: leverage CC* SAND-NMA 44

45 SLATE Deployment options 45

46 SLATE v1.0 Currently developing a "minimum viable product" for SLATE Virtual and hardware-based environments Tooling to register and "SLATEify" k8s clusters Launch perfsonar, Globus, and data caching services 46

47 SLATE Virtual Environment Single-node, CoreOS-based Kubernetes Virtual Machine "Zero to SLATE" Downloadable bundle KVM or Libvirt Register with SLATE, instantiate services, VOs, etc Currently being used at UChicago on FIONA hardware 47

48 SLATE VM And reachable by web browser SLATE cluster & public IPs Service available on public IP 48

49 SLATE Hardware Four components for v1.0 High performance switch (40/100 Gbps) Dedicated perfsonar and Management nodes Hypervisor-like "Edge Service" node for K8S Publicly posted hardware configurations for interested partners Self-service portal for registering and configuring devices for your SLATE Cluster 49

50 SLATE on existing K8S Run SLATE services on an existing cluster SLATE merely a user Only uses limited "admin" privileges for initial installation You control policy Allocate resources, make scheduling decisions as you see fit Easy to install, easy to remove 50

SLATE. Services Layer at the Edge. First Meeting of the National Research Platform Montana State University August 7-8, 2017

SLATE. Services Layer at the Edge. First Meeting of the National Research Platform Montana State University August 7-8, 2017 SLATE Services Layer at the Edge Rob Gardner University of Chicago Shawn McKee University of Michigan Joe Breen University of Utah First Meeting of the National Research Platform Montana State University

More information

PRP Distributed Kubernetes Cluster

PRP Distributed Kubernetes Cluster PRP Distributed Kubernetes Cluster John Graham (UCSD) Dmitry Mishin (SDSC) Shawfeng Dong (SLAC) Workshop on Applying Advanced AI Workflows in Astronomy and Microscopy UCSC Silicon Valley Campus, Santa

More information

Storage Virtualization. Eric Yen Academia Sinica Grid Computing Centre (ASGC) Taiwan

Storage Virtualization. Eric Yen Academia Sinica Grid Computing Centre (ASGC) Taiwan Storage Virtualization Eric Yen Academia Sinica Grid Computing Centre (ASGC) Taiwan Storage Virtualization In computer science, storage virtualization uses virtualization to enable better functionality

More information

Adding Cloud Based Interactive Compute Capabilities to Globus Endpoints

Adding Cloud Based Interactive Compute Capabilities to Globus Endpoints Adding Cloud Based Interactive Compute Capabilities to Globus Endpoints Ben Galewsky Research Programmer, National Center for Supercomputing Applications bengal1@illinois.edu http://www.nationaldataservice.org/

More information

Kubernetes 101. Doug Davis, STSM September, 2017

Kubernetes 101. Doug Davis, STSM September, 2017 Kubernetes 101 Doug Davis, STSM September, 2017 Today's Agenda What is Kubernetes? How was Kubernetes created? Where is the Kubernetes community? Technical overview What's the current status of Kubernetes?

More information

Kuberiter White Paper. Kubernetes. Cloud Provider Comparison Chart. Lawrence Manickam Kuberiter Inc

Kuberiter White Paper. Kubernetes. Cloud Provider Comparison Chart. Lawrence Manickam Kuberiter Inc Kuberiter White Paper Kubernetes Cloud Provider Comparison Chart Lawrence Manickam Kuberiter Inc Oct 2018 Executive Summary Kubernetes (K8S) has become the de facto standard for Cloud Application Deployments.

More information

Kubernetes Integration with Virtuozzo Storage

Kubernetes Integration with Virtuozzo Storage Kubernetes Integration with Virtuozzo Storage A Technical OCTOBER, 2017 2017 Virtuozzo. All rights reserved. 1 Application Container Storage Application containers appear to be the perfect tool for supporting

More information

Launching StarlingX. The Journey to Drive Compute to the Edge Pilot Project Supported by the OpenStack

Launching StarlingX. The Journey to Drive Compute to the Edge Pilot Project Supported by the OpenStack Launching StarlingX The Journey to Drive Compute to the Edge Pilot Project Supported by the OpenStack Foundation Ian Jolliffe, WIND RIVER SYSTEMS Director Engineering @ian_jolliffe Project Overview An

More information

Evolution of the ATLAS PanDA Workload Management System for Exascale Computational Science

Evolution of the ATLAS PanDA Workload Management System for Exascale Computational Science Evolution of the ATLAS PanDA Workload Management System for Exascale Computational Science T. Maeno, K. De, A. Klimentov, P. Nilsson, D. Oleynik, S. Panitkin, A. Petrosyan, J. Schovancova, A. Vaniachine,

More information

On-Premises Cloud Platform. Bringing the public cloud, on-premises

On-Premises Cloud Platform. Bringing the public cloud, on-premises On-Premises Cloud Platform Bringing the public cloud, on-premises How Cloudistics came to be 2 Cloudistics On-Premises Cloud Platform Complete Cloud Platform Simple Management Application Specific Flexibility

More information

Analytics Platform for ATLAS Computing Services

Analytics Platform for ATLAS Computing Services Analytics Platform for ATLAS Computing Services Ilija Vukotic for the ATLAS collaboration ICHEP 2016, Chicago, USA Getting the most from distributed resources What we want To understand the system To understand

More information

INTRODUCING CONTAINER-NATIVE VIRTUALIZATION

INTRODUCING CONTAINER-NATIVE VIRTUALIZATION INTRODUCING CONTAINER-NATIVE VIRTUALIZATION Cats and Dogs Living Together Stephen Gordon Principal Product Manager Red Hat Fabian Deutsch Manager, Software Engineering Red Hat sgordon@redhat.com / @xsgordon

More information

Introduction to the Open Service Broker API. Doug Davis

Introduction to the Open Service Broker API. Doug Davis Introduction to the Open Service Broker API Doug Davis IBM dug@us.ibm.com @duginabox A Brief History... u PaaS with a mission to make managing Cloud apps simple $ cf push myapp $ cf scale myapp -i 5 u

More information

Conference The Data Challenges of the LHC. Reda Tafirout, TRIUMF

Conference The Data Challenges of the LHC. Reda Tafirout, TRIUMF Conference 2017 The Data Challenges of the LHC Reda Tafirout, TRIUMF Outline LHC Science goals, tools and data Worldwide LHC Computing Grid Collaboration & Scale Key challenges Networking ATLAS experiment

More information

The Latest EMC s announcements

The Latest EMC s announcements The Latest EMC s announcements Copyright 2014 EMC Corporation. All rights reserved. 1 TODAY S BUSINESS CHALLENGES Cut Operational Costs & Legacy More Than Ever React Faster To Find New Growth Balance Risk

More information

Akraino & Starlingx: A Technical Overview

Akraino & Starlingx: A Technical Overview Akraino & Starlingx: A Technical Overview Agenda Why Edge Computing? What s Edge Computing? Akraino and Its Building Blocks StarlingX Technical Overview Collaboration in Akraino Community LC3 Beijing 2018

More information

Flying HTCondor at 100gbps Over the Golden State

Flying HTCondor at 100gbps Over the Golden State Flying HTCondor at 100gbps Over the Golden State Jeff Dost (UCSD) HTCondor Week 2016 1 What is PRP? Pacific Research Platform: - 100 gbit network extending from Southern California to Washington - Interconnects

More information

Cisco CloudCenter Solution with Cisco ACI: Common Use Cases

Cisco CloudCenter Solution with Cisco ACI: Common Use Cases Cisco CloudCenter Solution with Cisco ACI: Common Use Cases Cisco ACI increases network security, automates communication policies based on business-relevant application requirements, and decreases developer

More information

Evolution of Cloud Computing in ATLAS

Evolution of Cloud Computing in ATLAS The Evolution of Cloud Computing in ATLAS Ryan Taylor on behalf of the ATLAS collaboration 1 Outline Cloud Usage and IaaS Resource Management Software Services to facilitate cloud use Sim@P1 Performance

More information

HPE Synergy HPE SimpliVity 380

HPE Synergy HPE SimpliVity 380 HPE Synergy HPE SimpliVity 0 Pascal.Moens@hpe.com, Solutions Architect Technical Partner Lead February 0 HPE Synergy Composable infrastructure at HPE CPU Memory Local Storage LAN I/O SAN I/O Power Cooling

More information

Kubernetes 1.9 Features and Future

Kubernetes 1.9 Features and Future OpenShift Commons Briefing: Kubernetes 1.9 Features and Future Derek Carr - Lead Engineer, Kubernetes What s new this time around? RELEASE STATS Shorter release (end of year) 6000+ pull requests merged

More information

VC3. Virtual Clusters for Community Computation. DOE NGNS PI Meeting September 27-28, 2017

VC3. Virtual Clusters for Community Computation. DOE NGNS PI Meeting September 27-28, 2017 VC3 Virtual Clusters for Community Computation DOE NGNS PI Meeting September 27-28, 2017 Douglas Thain, University of Notre Dame Rob Gardner, University of Chicago John Hover, Brookhaven National Lab A

More information

Leveraging Globus Identity for the Grid. Suchandra Thapa GlobusWorld, April 22, 2016 Chicago

Leveraging Globus Identity for the Grid. Suchandra Thapa GlobusWorld, April 22, 2016 Chicago Leveraging Globus Identity for the Grid Suchandra Thapa GlobusWorld, April 22, 2016 Chicago Open Science Grid Helps researchers speed up their research using high throughput computing methods Helps campus

More information

Airship A New Open Infrastructure Project for OpenStack

Airship A New Open Infrastructure Project for OpenStack Airship A New Open Infrastructure Project for OpenStack Declaratively define your OpenStack & Kubernetes Infrastructure Airship A New Open Infrastructure Project for OpenStack Declaratively define your

More information

Cisco Enterprise Cloud Suite Overview Cisco and/or its affiliates. All rights reserved.

Cisco Enterprise Cloud Suite Overview Cisco and/or its affiliates. All rights reserved. Cisco Enterprise Cloud Suite Overview 2015 Cisco and/or its affiliates. All rights reserved. 1 CECS Components End User Service Catalog SERVICE PORTAL Orchestration and Management UCS Director Application

More information

Red Hat Atomic Details Dockah, Dockah, Dockah! Containerization as a shift of paradigm for the GNU/Linux OS

Red Hat Atomic Details Dockah, Dockah, Dockah! Containerization as a shift of paradigm for the GNU/Linux OS Red Hat Atomic Details Dockah, Dockah, Dockah! Containerization as a shift of paradigm for the GNU/Linux OS Daniel Riek Sr. Director Systems Design & Engineering In the beginning there was Stow... and

More information

Kubernetes: Twelve KeyFeatures

Kubernetes: Twelve KeyFeatures Kubernetes: Twelve KeyFeatures Kubernetes is a Greek word which means helmsman, or the pilot of a ship. It is an open source project that was started by Google and derived from Borg, which is used inside

More information

An Introduction to Kubernetes

An Introduction to Kubernetes 8.10.2016 An Introduction to Kubernetes Premys Kafka premysl.kafka@hpe.com kafkapre https://github.com/kafkapre { History }???? - Virtual Machines 2008 - Linux containers (LXC) 2013 - Docker 2013 - CoreOS

More information

Scientific data processing at global scale The LHC Computing Grid. fabio hernandez

Scientific data processing at global scale The LHC Computing Grid. fabio hernandez Scientific data processing at global scale The LHC Computing Grid Chengdu (China), July 5th 2011 Who I am 2 Computing science background Working in the field of computing for high-energy physics since

More information

Cloud Computing the VMware Perspective. Bogomil Balkansky Product Marketing

Cloud Computing the VMware Perspective. Bogomil Balkansky Product Marketing Cloud Computing the VMware Perspective Bogomil Balkansky Product Marketing Cloud Computing - the Key Questions What is it? Why do you need it? How do you build (or leverage) one (or many)? How do you operate

More information

Red Hat Roadmap for Containers and DevOps

Red Hat Roadmap for Containers and DevOps Red Hat Roadmap for Containers and DevOps Brian Gracely, Director of Strategy Diogenes Rettori, Principal Product Manager Red Hat September, 2016 Digital Transformation Requires an evolution in... 2 APPLICATIONS

More information

STATUS OF PLANS TO USE CONTAINERS IN THE WORLDWIDE LHC COMPUTING GRID

STATUS OF PLANS TO USE CONTAINERS IN THE WORLDWIDE LHC COMPUTING GRID The WLCG Motivation and benefits Container engines Experiments status and plans Security considerations Summary and outlook STATUS OF PLANS TO USE CONTAINERS IN THE WORLDWIDE LHC COMPUTING GRID SWISS EXPERIENCE

More information

Running MarkLogic in Containers (Both Docker and Kubernetes)

Running MarkLogic in Containers (Both Docker and Kubernetes) Running MarkLogic in Containers (Both Docker and Kubernetes) Emma Liu Product Manager, MarkLogic Vitaly Korolev Staff QA Engineer, MarkLogic @vitaly_korolev 4 June 2018 MARKLOGIC CORPORATION Source: http://turnoff.us/image/en/tech-adoption.png

More information

PSOACI Why ACI: An overview and a customer (BBVA) perspective. Technology Officer DC EMEAR Cisco

PSOACI Why ACI: An overview and a customer (BBVA) perspective. Technology Officer DC EMEAR Cisco PSOACI-4592 Why ACI: An overview and a customer (BBVA) perspective TJ Bijlsma César Martinez Joaquin Crespo Technology Officer DC EMEAR Cisco Lead Architect BBVA Lead Architect BBVA Cisco Spark How Questions?

More information

Architecting Microsoft Azure Solutions (proposed exam 535)

Architecting Microsoft Azure Solutions (proposed exam 535) Architecting Microsoft Azure Solutions (proposed exam 535) IMPORTANT: Significant changes are in progress for exam 534 and its content. As a result, we are retiring this exam on December 31, 2017, and

More information

How to build scalable, reliable and stable Kubernetes cluster atop OpenStack.

How to build scalable, reliable and stable Kubernetes cluster atop OpenStack. How to build scalable, reliable and stable Kubernetes cluster atop OpenStack Bo Wang HouMing Wang bo.wang@easystack.cn houming.wang@easystack.cn Cluster resources management Cluster data persistence Contents

More information

ENHANCE APPLICATION SCALABILITY AND AVAILABILITY WITH NGINX PLUS AND THE DIAMANTI BARE-METAL KUBERNETES PLATFORM

ENHANCE APPLICATION SCALABILITY AND AVAILABILITY WITH NGINX PLUS AND THE DIAMANTI BARE-METAL KUBERNETES PLATFORM JOINT SOLUTION BRIEF ENHANCE APPLICATION SCALABILITY AND AVAILABILITY WITH NGINX PLUS AND THE DIAMANTI BARE-METAL KUBERNETES PLATFORM DIAMANTI PLATFORM AT A GLANCE Modern load balancers which deploy as

More information

Architectural overview Turbonomic accesses Cisco Tetration Analytics data through Representational State Transfer (REST) APIs. It uses telemetry data

Architectural overview Turbonomic accesses Cisco Tetration Analytics data through Representational State Transfer (REST) APIs. It uses telemetry data Solution Overview Cisco Tetration Analytics and Turbonomic Solution Deploy intent-based networking for distributed applications. Highlights Provide performance assurance for distributed applications. Real-time

More information

UK Tier-2 site evolution for ATLAS. Alastair Dewhurst

UK Tier-2 site evolution for ATLAS. Alastair Dewhurst UK Tier-2 site evolution for ATLAS Alastair Dewhurst Introduction My understanding is that GridPP funding is only part of the story when it comes to paying for a Tier 2 site. Each site is unique. Aim to

More information

FIVE REASONS YOU SHOULD RUN CONTAINERS ON BARE METAL, NOT VMS

FIVE REASONS YOU SHOULD RUN CONTAINERS ON BARE METAL, NOT VMS WHITE PAPER FIVE REASONS YOU SHOULD RUN CONTAINERS ON BARE METAL, NOT VMS Over the past 15 years, server virtualization has become the preferred method of application deployment in the enterprise datacenter.

More information

ACCENTURE & RED HAT ACCENTURE CLOUD INNOVATION CENTER

ACCENTURE & RED HAT ACCENTURE CLOUD INNOVATION CENTER ACCENTURE & RED HAT ACCENTURE CLOUD INNOVATION CENTER HYBRID CLOUD MANAGEMENT & OPTIMIZATION DEVOPS FOR INFRASTRUCTURE SERVICES ACCENTURE CLOUD INNOVATION CENTER PUSHING CUSTOM CLOUD SOLUTIONS TO THE MAX.

More information

HPC learning using Cloud infrastructure

HPC learning using Cloud infrastructure HPC learning using Cloud infrastructure Florin MANAILA IT Architect florin.manaila@ro.ibm.com Cluj-Napoca 16 March, 2010 Agenda 1. Leveraging Cloud model 2. HPC on Cloud 3. Recent projects - FutureGRID

More information

Cornell Red Cloud: Campus-based Hybrid Cloud. Steven Lee Cornell University Center for Advanced Computing

Cornell Red Cloud: Campus-based Hybrid Cloud. Steven Lee Cornell University Center for Advanced Computing Cornell Red Cloud: Campus-based Hybrid Cloud Steven Lee Cornell University Center for Advanced Computing shl1@cornell.edu Cornell Center for Advanced Computing (CAC) Profile CAC mission, impact on research

More information

Smarter Storage with Containerized Applications. Always Aligned with your Changing World

Smarter Storage with Containerized Applications. Always Aligned with your Changing World Smarter Storage with Containerized Applications Always Aligned with your Changing World Gabriel Lopez Solutions Architect, Zadara Storage Accumulated years of experience in advanced software design and

More information

Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme

Disclaimer This presentation may contain product features that are currently under development. This overview of new technology represents no commitme CNA2080BU Deep Dive: How to Deploy and Operationalize Kubernetes Cornelia Davis, Pivotal Nathan Ness Technical Product Manager, CNABU @nvpnathan #VMworld #CNA2080BU Disclaimer This presentation may contain

More information

Designing MQ deployments for the cloud generation

Designing MQ deployments for the cloud generation Designing MQ deployments for the cloud generation WebSphere User Group, London Arthur Barr, Senior Software Engineer, IBM MQ 30 th March 2017 Top business drivers for cloud 2 Source: OpenStack user survey,

More information

Deploying Cloud Network Services Prime Network Services Controller (formerly VNMC)

Deploying Cloud Network Services Prime Network Services Controller (formerly VNMC) Deploying Cloud Network Services Prime Network Services Controller (formerly VNMC) Dedi Shindler - Sr. Manager Product Management Cloud System Management Technology Group Cisco Agenda Trends Influencing

More information

IRNC:RXP SDN / SDX Update

IRNC:RXP SDN / SDX Update 30 September 2016 IRNC:RXP SDN / SDX Update John Hess Darrell Newcomb GLIF16, Miami Pacific Wave: Overview Joint project of CENIC (California regional research and

More information

CoreOS and Red Hat. Reza Shafii Joe Fernandes Brandon Philips Clayton Coleman May 2018

CoreOS and Red Hat. Reza Shafii Joe Fernandes Brandon Philips Clayton Coleman May 2018 CoreOS and Red Hat Reza Shafii Joe Fernandes Brandon Philips Clayton Coleman May 2018 Combining Industry Leading Container Solutions RED HAT QUAY REGISTRY ETCD PROMETHEUS RED HAT COREOS METERING & CHARGEBACK

More information

Open Service Broker API: Creating a Cross-Platform Standard Doug Davis IBM Shannon Coen Pivotal

Open Service Broker API: Creating a Cross-Platform Standard Doug Davis IBM Shannon Coen Pivotal Open Service Broker API: Creating a Cross-Platform Standard Doug Davis IBM Shannon Coen Pivotal Motivations Application development teams require services From app dependencies to team enablement Managed

More information

ALICE Grid Activities in US

ALICE Grid Activities in US ALICE Grid Activities in US 1 ALICE-USA Computing Project ALICE-USA Collaboration formed to focus on the ALICE EMCal project Construction, installation, testing and integration participating institutions

More information

INTEGRATING HPFS IN A CLOUD COMPUTING ENVIRONMENT

INTEGRATING HPFS IN A CLOUD COMPUTING ENVIRONMENT INTEGRATING HPFS IN A CLOUD COMPUTING ENVIRONMENT Abhisek Pan 2, J.P. Walters 1, Vijay S. Pai 1,2, David Kang 1, Stephen P. Crago 1 1 University of Southern California/Information Sciences Institute 2

More information

Container in Production : Openshift 구축사례로 이해하는 PaaS. Jongjin Lim Specialist Solution Architect, AppDev

Container in Production : Openshift 구축사례로 이해하는 PaaS. Jongjin Lim Specialist Solution Architect, AppDev Container in Production : Openshift 구축사례로 이해하는 PaaS Jongjin Lim Specialist Solution Architect, AppDev jonlim@redhat.com Agenda Why Containers? Solution : Red Hat Openshift Container Platform Enterprise

More information

Choosing the Right Container Infrastructure for Your Organization

Choosing the Right Container Infrastructure for Your Organization WHITE PAPER Choosing the Right Container Infrastructure for Your Organization Container adoption is accelerating rapidly. Gartner predicts that by 2018 more than 50% of new workloads will be deployed into

More information

MSB to Support for Carrier Grade ONAP Microservice Architecture. Huabing Zhao, PTL of MSB Project, ZTE

MSB to Support for Carrier Grade ONAP Microservice Architecture. Huabing Zhao, PTL of MSB Project, ZTE MSB to Support for Carrier Grade ONAP Microservice Architecture Huabing Zhao, PTL of MSB Project, ZTE ONAP Architecture Principle: Microservices ONAP Architecture Principle: ONAP modules should be designed

More information

VMworld 2018 Content: Not for publication or distribution

VMworld 2018 Content: Not for publication or distribution Hybrid IT: HPE s composable, software-defined, dynamic datacenter vision Speaker name and title Month day, year Today s accidental architecture is hybrid by default Describe how IT infrastructure has become

More information

<Insert Picture Here> Enterprise Data Management using Grid Technology

<Insert Picture Here> Enterprise Data Management using Grid Technology Enterprise Data using Grid Technology Kriangsak Tiawsirisup Sales Consulting Manager Oracle Corporation (Thailand) 3 Related Data Centre Trends. Service Oriented Architecture Flexibility

More information

Paperspace. Architecture Overview. 20 Jay St. Suite 312 Brooklyn, NY Technical Whitepaper

Paperspace. Architecture Overview. 20 Jay St. Suite 312 Brooklyn, NY Technical Whitepaper Architecture Overview Copyright 2016 Paperspace, Co. All Rights Reserved June - 1-2017 Technical Whitepaper Paperspace Whitepaper: Architecture Overview Content 1. Overview 3 2. Virtualization 3 Xen Hypervisor

More information

Globus Platform Services for Data Publication. Greg Nawrocki University of Chicago & Argonne National Lab GeoDaRRS August 7, 2018

Globus Platform Services for Data Publication. Greg Nawrocki University of Chicago & Argonne National Lab GeoDaRRS August 7, 2018 Globus Platform Services for Data Publication Greg Nawrocki greg@globus.org University of Chicago & Argonne National Lab GeoDaRRS August 7, 2018 Outline Globus Overview Globus Data Publication v1 Lessons

More information

Private Cloud at IIT Delhi

Private Cloud at IIT Delhi Private Cloud at IIT Delhi Success Story Engagement: Long Term Industry: Education Offering: Private Cloud Deployment Business Challenge IIT Delhi, one of the India's leading educational Institute wanted

More information

Open Cloud Reference Architecture

Open Cloud Reference Architecture Open Cloud Reference Architecture Version 1.4, February 2018 Abstract The OpenNebula Cloud Reference Architecture is a blueprint to guide IT architects, consultants, administrators and field practitioners

More information

OSiRIS. Project and participants overview Structural overview and site details Orchestration, monitoring and visualization Networking, NMAL, SDN

OSiRIS. Project and participants overview Structural overview and site details Orchestration, monitoring and visualization Networking, NMAL, SDN OSiRIS Distributed Ceph and Software Defined Networking for Multi-Institutional Research Benjeman Meekhof University of Michigan Advanced Research Computing Technology Services October 6, 2016 Project

More information

Important DevOps Technologies (3+2+3days) for Deployment

Important DevOps Technologies (3+2+3days) for Deployment Important DevOps Technologies (3+2+3days) for Deployment DevOps is the blending of tasks performed by a company's application development and systems operations teams. The term DevOps is being used in

More information

OpenNebula on VMware: Cloud Reference Architecture

OpenNebula on VMware: Cloud Reference Architecture OpenNebula on VMware: Cloud Reference Architecture Version 1.2, October 2016 Abstract The OpenNebula Cloud Reference Architecture is a blueprint to guide IT architects, consultants, administrators and

More information

CONTAINERS AND MICROSERVICES WITH CONTRAIL

CONTAINERS AND MICROSERVICES WITH CONTRAIL CONTAINERS AND MICROSERVICES WITH CONTRAIL Scott Sneddon Sree Sarva DP Ayyadevara Sr. Director Sr. Director Director Cloud and SDN Contrail Solutions Product Line Management This statement of direction

More information

Scaling Across the NRP Ecosystem From Campus to Regional to National - What Support Is There? 2NRP Workshop Bozeman, Montana Tuesday, August 7, 2018

Scaling Across the NRP Ecosystem From Campus to Regional to National - What Support Is There? 2NRP Workshop Bozeman, Montana Tuesday, August 7, 2018 Scaling Across the NRP Ecosystem From Campus to Regional to National - What Support Is There? 2NRP Workshop Bozeman, Montana Tuesday, August 7, 2018 1 Panel Lars Fischer NORDUnet Kate Mace ESnet Marla

More information

Pasiruoškite ateičiai: modernus duomenų centras. Laurynas Dovydaitis Microsoft Azure MVP

Pasiruoškite ateičiai: modernus duomenų centras. Laurynas Dovydaitis Microsoft Azure MVP Pasiruoškite ateičiai: modernus duomenų centras Laurynas Dovydaitis Microsoft Azure MVP 2016-05-17 Tension drives change The datacenter today Traditional datacenter Tight coupling between infrastructure

More information

Containers Infrastructure for Advanced Management. Federico Simoncelli Associate Manager, Red Hat October 2016

Containers Infrastructure for Advanced Management. Federico Simoncelli Associate Manager, Red Hat October 2016 Containers Infrastructure for Advanced Management Federico Simoncelli Associate Manager, Red Hat October 2016 About Me Kubernetes Decoupling problems to hand out to different teams Layer of abstraction

More information

Energy Management with AWS

Energy Management with AWS Energy Management with AWS Kyle Hart and Nandakumar Sreenivasan Amazon Web Services August [XX], 2017 Tampa Convention Center Tampa, Florida What is Cloud? The NIST Definition Broad Network Access On-Demand

More information

Accelerate at DevOps Speed With Openshift v3. Alessandro Vozza & Samuel Terburg Red Hat

Accelerate at DevOps Speed With Openshift v3. Alessandro Vozza & Samuel Terburg Red Hat Accelerate at DevOps Speed With Openshift v3 Alessandro Vozza & Samuel Terburg Red Hat IT (R)Evolution Red Hat Brings It All Together What is Kubernetes Open source container cluster manager Inspired by

More information

OpenStack Magnum Pike and the CERN cloud. Spyros

OpenStack Magnum Pike and the CERN cloud. Spyros OpenStack Magnum Pike and the CERN cloud Spyros Trigazis @strigazi OpenStack Magnum OpenStack Magnum #openstack-containers Kubernetes, Docker Swarm, Apache Mesos, DC/OS (experimental) aas Deep integration

More information

Baremetal with Apache CloudStack

Baremetal with Apache CloudStack Baremetal with Apache CloudStack ApacheCon Europe 2016 Jaydeep Marfatia Cloud, IOT and Analytics Me Director of Product Management Cloud Products Accelerite Background Project lead for open source project

More information

CHARTING THE FUTURE OF SOFTWARE DEFINED NETWORKING

CHARTING THE FUTURE OF SOFTWARE DEFINED NETWORKING www.hcltech.com CHARTING THE FUTURE OF SOFTWARE DEFINED NETWORKING Why Next-Gen Networks? The rapid and large scale adoption of new age disruptive digital technologies has resulted in astronomical growth

More information

Design patterns for data-driven research acceleration

Design patterns for data-driven research acceleration Design patterns for data-driven research acceleration Rachana Ananthakrishnan, Kyle Chard, and Ian Foster The University of Chicago and Argonne National Laboratory Contact: rachana@globus.org Introduction

More information

Module Day Topic. 1 Definition of Cloud Computing and its Basics

Module Day Topic. 1 Definition of Cloud Computing and its Basics Module Day Topic 1 Definition of Cloud Computing and its Basics 1 2 3 1. How does cloud computing provides on-demand functionality? 2. What is the difference between scalability and elasticity? 3. What

More information

Fast and Easy Persistent Storage for Docker* Containers with Storidge and Intel

Fast and Easy Persistent Storage for Docker* Containers with Storidge and Intel Solution brief Intel Storage Builders Storidge ContainerIO TM Intel Xeon Processor Scalable Family Intel SSD DC Family for PCIe*/NVMe Fast and Easy Persistent Storage for Docker* Containers with Storidge

More information

Container Adoption for NFV Challenges & Opportunities. Sriram Natarajan, T-Labs Silicon Valley Innovation Center

Container Adoption for NFV Challenges & Opportunities. Sriram Natarajan, T-Labs Silicon Valley Innovation Center Container Adoption for NFV Challenges & Opportunities Sriram Natarajan, T-Labs Silicon Valley Innovation Center Virtual Machine vs. Container Stack KVM Container-stack Libraries Guest-OS Hypervisor Libraries

More information

Securing Microservice Interactions in Openstack and Kubernetes

Securing Microservice Interactions in Openstack and Kubernetes Securing Microservice Interactions in Openstack and Kubernetes Yoshio Turner & Jayanth Gummaraju Co- Founders @ Banyan https://www.banyanops.com Banyan Founded in the middle of 2015 In San Francisco, CA

More information

Table of Contents 1.1. Introduction. Overview of vsphere Integrated Containers 1.2

Table of Contents 1.1. Introduction. Overview of vsphere Integrated Containers 1.2 Table of Contents Introduction Overview of vsphere Integrated Containers 1.1 1.2 2 Overview of vsphere Integrated Containers This document provides an overview of VMware vsphere Integrated Containers.

More information

Kubernetes - Load Balancing For Virtual Machines (Pods)

Kubernetes - Load Balancing For Virtual Machines (Pods) Kubernetes - Load Balancing For Virtual Machines (Pods) 4 th of Feb 2018 Yanir Quinn Senior Software Engineer Red Hat This presentation is licensed under a Creative Commons Attribution 4.0 International

More information

Cisco Container Platform

Cisco Container Platform Cisco Container Platform Pradnesh Patil Suhail Syed Cisco Spark How Questions? Use Cisco Spark to communicate with the speaker after the session 1. Find this session in the Cisco Live Mobile App 2. Click

More information

A day in the life of a log message Kyle Liberti, Josef

A day in the life of a log message Kyle Liberti, Josef A day in the life of a log message Kyle Liberti, Josef Karasek @Pepe_CZ Order is vital for scale Abstractions make systems manageable Problems of Distributed Systems Reliability Data throughput Latency

More information

A Virtual Comet. HTCondor Week 2017 May Edgar Fajardo On behalf of OSG Software and Technology

A Virtual Comet. HTCondor Week 2017 May Edgar Fajardo On behalf of OSG Software and Technology A Virtual Comet HTCondor Week 2017 May 3 2017 Edgar Fajardo On behalf of OSG Software and Technology 1 Working in Comet What my friends think I do What Instagram thinks I do What my boss thinks I do 2

More information

SOLUTION OVERVIEW THE ARUBA MOBILE FIRST ARCHITECTURE

SOLUTION OVERVIEW THE ARUBA MOBILE FIRST ARCHITECTURE SOLUTION OVERVIEW THE ARUBA MOBILE FIRST ARCHITECTURE March 2018 Table of Contents Introduction...1 Design...2 Use Cases...2 Underlay...3 Overlay...3 Dynamic Segmentation...3 Non-Stop Networking...4 Summary...5

More information

Allowing Users to Run Services at the OLCF with Kubernetes

Allowing Users to Run Services at the OLCF with Kubernetes Allowing Users to Run Services at the OLCF with Kubernetes Jason Kincl Senior HPC Systems Engineer Ryan Adamson Senior HPC Security Engineer This work was supported by the Oak Ridge Leadership Computing

More information

Journey to the Cloud Next Generation Infrastructure for the future workforce.

Journey to the Cloud Next Generation Infrastructure for the future workforce. Journey to the Cloud Next Generation Infrastructure for the future workforce. Steven J. Davis Managing Director Infrastructure Consulting ASEAN Accenture What is the Internet & Cloud The Internet traditionally

More information

Containerization Dockers / Mesospere. Arno Keller HPE

Containerization Dockers / Mesospere. Arno Keller HPE Containerization Dockers / Mesospere Arno Keller HPE What is the Container technology Hypervisor vs. Containers (Huis vs artement) A container doesn't "boot" an OS instead it loads the application and

More information

Singularity in CMS. Over a million containers served

Singularity in CMS. Over a million containers served Singularity in CMS Over a million containers served Introduction The topic of containers is broad - and this is a 15 minute talk! I m filtering out a lot of relevant details, particularly why we are using

More information

Secure Kubernetes Container Workloads

Secure Kubernetes Container Workloads Secure Kubernetes Container Workloads with Production-Grade Networking Cynthia Thomas Irena Berezovsky Tim Hockin CIA IT operations have top secret apps for their agents, most of which require isolation

More information

SCA19 APRP. Update Andrew Howard - Co-Chair APAN APRP Working Group. nci.org.au

SCA19 APRP. Update Andrew Howard - Co-Chair APAN APRP Working Group. nci.org.au SCA19 APRP Update Andrew Howard - Co-Chair APAN APRP Working Group 1 What is a Research Platform Notable Research Platforms APRP History Participants Activities Overview We live in an age of rapidly expanding

More information

Cloud Systems 2018 Training Programs. Catalog of Course Descriptions

Cloud Systems 2018 Training Programs. Catalog of Course Descriptions Cloud Systems 2018 Training Programs Catalog of Course Descriptions Catalog of Course Descriptions INTRODUCTION...3 Open 2 2018 Introduction Ericsson has developed a comprehensive Training Programs service

More information

ENTERPRISE-GRADE MANAGEMENT FOR OPENSTACK WITH RED HAT CLOUDFORMS

ENTERPRISE-GRADE MANAGEMENT FOR OPENSTACK WITH RED HAT CLOUDFORMS TECHNOLOGY DETAIL ENTERPRISE-GRADE MANAGEMENT FOR OPENSTACK WITH RED HAT CLOUDFORMS ABSTRACT Enterprises engaged in deploying, managing, and scaling out Red Hat Enterprise Linux OpenStack Platform have

More information

DEPLOYING NFV: BEST PRACTICES

DEPLOYING NFV: BEST PRACTICES DEPLOYING NFV: BEST PRACTICES Rimma Iontel Senior Cloud Architect, Cloud Practice riontel@redhat.com Julio Villarreal Pelegrino Principal Architect, Cloud Practice julio@redhat.com INTRODUCTION TO NFV

More information

RED HAT OPENSHIFT A FOUNDATION FOR SUCCESSFUL DIGITAL TRANSFORMATION

RED HAT OPENSHIFT A FOUNDATION FOR SUCCESSFUL DIGITAL TRANSFORMATION RED HAT OPENSHIFT A FOUNDATION FOR SUCCESSFUL DIGITAL TRANSFORMATION Stephanos D Bacon Product Portfolio Strategy, Application Platforms Stockholm, 13 September 2017 1 THE PATH TO DIGITAL LEADERSHIP IT

More information

CLOUD-NATIVE APPLICATION DEVELOPMENT/ARCHITECTURE

CLOUD-NATIVE APPLICATION DEVELOPMENT/ARCHITECTURE JAN WILLIES Global Kubernetes Lead at Accenture Technology jan.willies@accenture.com CLOUD-NATIVE APPLICATION DEVELOPMENT/ARCHITECTURE SVEN MENTL Cloud-native at Accenture Technology ASG sven.mentl@accenture.com

More information

What s New in Red Hat OpenShift Container Platform 3.4. Torben Jäger Red Hat Solution Architect

What s New in Red Hat OpenShift Container Platform 3.4. Torben Jäger Red Hat Solution Architect What s New in Red Hat OpenShift Container Platform 3.4 Torben Jäger Red Hat Solution Architect OpenShift Roadmap OpenShift Container Platform 3.2 Kubernetes 1.2 & Docker 1.9 Red Hat

More information

São Paulo. August,

São Paulo. August, São Paulo August, 28 2018 Going Cloud Native with Cloud Foundry Luis Macedo Sr Platform Engineer, Pivotal @luis0macedo What is Cloud Native Platform? - A platform that delivers predictable deployment

More information

Joe Butler, Principal Engineer, Director Cloud Services Lab. Nov , OpenStack Summit Paris.

Joe Butler, Principal Engineer, Director Cloud Services Lab. Nov , OpenStack Summit Paris. Telemetry the foundation of intelligent cloud orchestration. Joe Butler, Principal Engineer, Director Cloud Services Lab. Nov 3 2014, OpenStack Summit Paris. http://sched.co/1xj2lm9 Datacenter Trends and

More information

Walkthrough OCCAM. Be on the lookout for this fellow: The callouts are ACTIONs for you to do!

Walkthrough OCCAM. Be on the lookout for this fellow: The callouts are ACTIONs for you to do! Walkthrough OCCAM Be on the lookout for this fellow: The callouts are ACTIONs for you to do! When you see the check mark, compare your work to the marked element Objectives In this presentation you ll

More information

Grid Middleware and Globus Toolkit Architecture

Grid Middleware and Globus Toolkit Architecture Grid Middleware and Globus Toolkit Architecture Lisa Childers Argonne National Laboratory University of Chicago 2 Overview Grid Middleware The problem: supporting Virtual Organizations equirements Capabilities

More information