OpenStack from the basics to production: Experiences on Jetstream

Transcription

1 OpenStack from the basics to production: Experiences on Jetstream David Hancock Acting Principal Investigator Jetstream Program Director Advanced Cyberinfrastructure Indiana University Pervasive Technology Institute J. Michael Lowe Lead Systems Programmer High Performance Systems UITS Research Technologies

2 Overview Brief IU background Jetstream at 10,000 meters Cows and buzzwords Jetstream at 10 meters Dragons Basics of OpenStack Jetstream at 1 meter Challenges & Opportunities Futures

3 IU Campuses and Medical School Centers IU Campuses IU School of Medicine campuses and clinics

4 IU Overview Fall 2016 Number Undergraduate 93,740 Graduate 12,397 Doctoral - Research 4,323 Doctoral - Practice 3,700 Total Students 114,160 Staff 11,498 Faculty 9,005 Grand Total 134,633 Overall Operating budget $3.5B USD Grant Awards of $614M in 2016 Centralized IT Org - UITS 700+ professional staff 500+ part time staff

5 National Science Foundation Funding in HPC Traditionally concentrated on enabling peta-scale capability via track I/II programs Blue Waters 13.3 petaflops, 2012 Stampede 9.6 petaflops, 2013 Comet ~2.0 petaflops, 2014 Have funded research into building clouds and computer science testbeds CloudLab Chameleon Now funding clouds to do research Bridges (Hybrid system) Jetstream

6 Jetstream - Expanding NSF XD s reach and impact Around 350,000 researchers, educators, & learners received NSF support in 2015 Less than 2% completed a computation, data analysis, or visualization task on XD/XSEDE program resources Less than 4% had an XSEDE Portal account 70% of researchers surveyed* claimed to be resource constrained Why are the people not using XD/XSEDE systems not using them? Perceived ease of access and use HPC resources the traditional view of what XSEDE offers - are often not wellmatched to their needs They just don t need that much capability *XSEDE Cloud Survey Report -

7 What is Jetstream and why does it exist? NSF s first production cloud facility Part of the NSF extreme Digital (XD) program Provides on-demand interactive computing and analysis Enables configurable environments and architectures User-friendly, widely accessible cloud environment User-selectable library of preconfigured virtual machines

8 What is Jetstream, continued Programmable cyberinfrastructure Cloudy: clouds, more than just virtual machines (VM) Old way: robust (expensive) infrastructure, weak (cheap) software Cloudy way: commodity infrastructure, robust software Cows, not pets: pets take great amount of care, feeding, and you name them; cows you intend to have high turnover and you give them numbers. Home for persistent science gateways Flickr user Nanak26 - Normandie Reproducibility: Share VMs and then store, publish via IU Scholarworks (DOI)

9 Who uses it? Researchers needing a handful of cores (1 to 44/vCPU) Software creators and researchers needing to create their own customized virtual machines and workflows Science gateway creators using Jetstream as either the frontend or processor for scientific jobs STEM Educators teaching on a variety of subjects

10 Not just the usual suspects Physics, chemistry, and other usual HPC suspects are represented, but Jetstream also is home to projects on: Financial analysis / Economics Political science Humanities / Text analysis Network analysis Computer Science / Machine learning Satellite data analysis

11 Jetstream usage highlights As of March 1, 2017: 290 active XSEDE projects covering 53 fields of science and active users representing 165 institutions Over 56 million CPU hours allocated to XSEDE projects since June science gateways 12 education/teaching allocations serving almost 400 undergraduate and graduate students Averaging ~600 concurrent VMs 99.8% availability, 98.7% job completion

12 About those gateways IRIS - Serving large scale earthquake and geographical data for analysis Unidata - Providing distribution and analysis of meteorological data OpenMRS - Providing medical records systems for the resource-constrained SEAGrid - Computational chemistry, molecular and fluid dynamics, and structural mechanics gateway NAMDRunner - Based on the GenApp gateway over 1 million computing hours used to date for MD Coming gateways: CIPRES Gateway, The Neuroscience Gateway, ChemCompute gateway, UltraScan III

13 Galaxy riding Jetstream Galaxy is a platform for biomedical research, focused on accessibility, transparency and reproducibility The main project instance (usegalaxy.org) has more than 100,000 registered users executing 300,000+ jobs each month Many users need more capacity than the public quota, or other customizations (e.g., new tools) Use Jetstream as a bursting platform From Galaxy Main, offload jobs onto a remote Slurm cluster running on Jetstream instances Run Galaxy Interactive Environments (i.e., Dockerized IPython/RStudio containers) in an isolated environment on a Swarm cluster running on Jetstream Use Jetstream as a self-service platform Pre-built Galaxy image configured with hundreds of tools and access to TBs of genomic reference data, available via the self-launch model within minutes Allows users to acquire (free) resources, and gives them complete control

14 Jetstream Galaxy bursting: 8,264 unique users,115,582 jobs

15 Jetstream for education Jetstream has been used in multiple IU informatics graduate courses INFO 535 Management, Access, and Use of Big and Complex Data INFO 590 Topics in Informatics BlueWaters Workflow Workshop Multiple Research Data Alliance Workshops Workshops/classes using Galaxy, data analysis in finance using R, security and intrusion detection, and principles in cloud computing

16 What is Jetstream, continued Software layers Atmosphere web interface library of images, generic, domain specific simplify VM administration OpenStack: software tools for building and managing cloud computing platforms for public and private clouds. KVM hypervisor: what the VMs run on Ceph: storage platform that stores data on a single distributed computer cluster, and provides interfaces for object-, block- and file-level storage. Operating systems: CentOS, Ubuntu, Windows? Applications: e.g. software developed by the domain specialist, gateways, etc.

17 Platform overview OpenStack API access Globus Auth Web App Atmo Services Atmosphere API XSEDE Accounting Agave API access (work in progress) OpenStack Ceph OpenStack Ceph S3/Swift access to Ceph Indiana University TACC

18 Jetstream system overview

19 Production cloud hardware (per site) Hardware Number Specifications Function (IU) Dell PowerEdge M630 blades Dell PowerEdge R630 1U server 320 2x Intel E5-2680v3 Haswell GHz 128 GB RAM 2 TB local disk 7 2x Intel E5-2680v3 Haswell GHz 128 GB RAM 2 TB local disk Compute hosts OpenStack services Cluster management High Availability Databases RabbitMQ Dell PowerEdge R730xd 2U servers Dell S6000-ON network switches 20 2x Intel E5-2680v3 Haswell GHz 64 GB RAM 48 TB storage for Ceph pool ~1 PB Ceph storage Gb/s ports Top of Rack Spine

20 Hardware and instance flavors Flavor vcpus RAM Storage Per Node tiny small medium large /60* 4 xlarge /60* 2 xxlarge /60* 1 Short-term storage comes as part of launched instance Long-term storage is XSEDE-allocated *Flavors updated March 2017, storage-rich flavors will not be imaged

21 Jetstream s Atmosphere interface (no login required to this point)

22 Jetstream s Atmosphere interface (Select identity provider)

23 Jetstream s Atmosphere interface (Authenticate)

24 Jetstream s Atmosphere interface (user s home space)

25 HPC vs Cloud Adapting to a different environment: No reservations, no queueing More interactive use and less/no batch queuing What? No parallel filesystem?!? Being your own admin hey, we have root! You really can have almost any (Linux) software you want** Constantly getting new features ** Here there be dragons Flickr user José Silenzi - dragon Flicker user Haz No Reservation

26 OpenStack organization

27 OpenStack projects

28 Openstack projects the core services Service Name Adoption Maturity Age Identity Keystone 96% 7/8 5 yrs Images Glance 95% 6/8 7 yrs Block device Cinder 88% 7/8 5 yrs Networking Neutron 93% 7/8 5 yrs Compute Nova 95% 8/8 7 yrs Object device Swift 52% 7/8 7 yrs

29 Openstack projects some other services Service Name Adoption Maturity Age Dashboard Horizon 87% 6/8 5 yrs Telemetry Ceilometer 55% 1/8 4 yrs Database Trove 13% 3/8 3 yrs Orchestration Heat 67% 6/8 4 yrs Provisioning Ironic 17% 2/8 3 yrs

30 Openstack projects some other services Service Name Adoption Maturity Age Map/Reduce Sahara 10% 3/8 3 yrs Shared Filesystems Manila 14% 5/8 3 yrs DNS Service Designate 16% 3/8 3 yrs Containers Magnum 11% 2/8 2 yrs Application Catalog Murano 11% 1/8 2 yrs

31 Cloud at-a-glance Internet SSL HA Proxy Controller 1 Controller 2 Controller 3

32 Load balancing Load Balancer 1 I P 1 I P 2 Load Balancer 2 I P 2 I P 1 Keep Alive DNS Round Robin IP1 IP2

33 High availability database layout Spine 1 Spine 2 Top of Rack Top of Rack Database 1 Database 3 Database 2 Load Balancer 1 Load Balancer 2 Management

34 Glance - Cinder - Ceph Monitor Client OSD 1 OSD 2 OSD N

35 Network block storage RADOS Block Device (RBD) emulates block devices with objects and offsets Qemu VM RBD Ceph Object 1 Object 2 Object 3 Object N

36 OpenStack Instance Creation Workflow Keystone Glance Client Ceph Compute Nova Cinder

37 VXLAN packet Ethernet IP/UDP Ethernet IP

38 Hypervisor VM Bridge VXLAN Hypervisor VM Neutron networking Network node VXLAN Bridge T a p NetNS T a p Bridge Bridge VXLAN Internet

39 Data collection OpenStack Services - Notifications Ceilometer - Collection Gnocchi Time Series Measurements Aodh - Alarms Panko Event Storage

40 New services on Jetstream Magnum Container Infrastructure Sahara Map Reduce Heat - Orchestration Nova -Compute Cinder - Block Storage Neutron - Networking

41 Upgrade process Internet Old Database Schema SSL HA Proxy Controller 1 Old Version Controller 2 Old Version Controller 3 Old Version

42 Upgrade (cont) Internet Transitional Database Schema SSL HA Proxy Controller 1 New Version Controller 2 Old Version Controller 3 Old Version

43 Upgrade (cont) Internet New Database Schema SSL HA Proxy Controller 1 New Version Controller 2 Old Version Controller 3 Old Version

44 Upgrade complete Internet New Database Schema SSL HA Proxy Controller 1 New Version Controller 2 New Version Controller 3 New Version

45 Jetstream timeline what comes next? Year 1 operations review ~June 2016 Increasing EOT personnel, refining message to new communities Potential summer REU (Research Experiences for Undergraduates) Continuously soliciting research allocation requests plus Startup and Education allocations including Science Gateways! Adding services as deemed useful/mature Atmosphere enhancements on a regular cycle ~monthly Hoping to bring above the cloud image build repo via XSEDE

46 Requesting access to Jetstream You can request startup allocations anytime. (Startups are simple!) You can request allocations for educational use anytime. Next submission period for large allocations is NOW - 15 March April We are happy to help you prepare a request and create a successful proposal. You do not have to have prior use of Jetstream to be successful. You do need a US-based collaborator

47 Jetstream partners

48 Help / References Wiki / Documentation: User guides: XSEDE KB: help@xsede.org Campus Champions: Paper describing Jetstream: Jetstream: A self-provisioned, scalable scince and engineering cloud environment Configuration management:

49 Questions? Project website: Project Direct or License Terms Hancock, David. April 12, Experiences on Jetstream HPC Advisory Council.Also available at: Jetstream is supported by NSF award (Craig Stewart, IU, PI) XSEDE is supported by NSF award (John Towns, UIUC, PI) This research was supported in part by the Indiana University Pervasive Technology Institute, which was established with the assistance of a major award from the Lilly Endowment, Inc. Opinions presented here are those of the author(s) and do not necessarily represent the views of the NSF, IUPTI, IU, or the Lilly Endowment, Inc. Items indicated with a are under copyright and used here with permission. Such items may not be reused without permission from the holder of copyright except where license terms noted on a slide permit reuse. Except where otherwise noted, contents of this presentation are copyright 2015 by the Trustees of Indiana University. This document is released under the Creative Commons Attribution 3.0 Unported license ( This license includes the following terms: You are free to share to copy, distribute and transmit the work and to remix to adapt the work under the following conditions: attribution you must attribute the work in the manner specified by the author or licensor (but not in any way that suggests that they endorse you or your use of the work). For any reuse or distribution, you must make clear to others the license terms of this work.

50 Things left behind for reference Flickr user Oiluj Samall Zeid - Lejos de Yulín

51 Just for fun: Happy Cluster Mad Cluster

52 Infrared image of Jetstream

53 Network Topology Chassis To Top of Rack Switches Chassis Sixteen blades per chasses Two switches per chassis 10 Gb/s Cooling 8 Blades 10 Gb/s per blade S w It c h S w It c h 10 GB/s per blade 8 Blades Cooling 40 Gb/s LAG 2 x 40 Gb/s Inter-switch links

54 Network Topology (cont.) Chassis to Top of Rack 40 Gb/s LAG Top of Rack Switch Four chassis per rack Two switches per chassis

55 Network Topology (cont.) 100 Gb/s uplink to Internet2 40 Gb/s LAG Two spines tie into two network blades in the datacenter switch Spine 1 Spine 2 Top of Rack Top of Rack Top of Rack Seven racks tie into the two spine switches 40 Gb/s