HPC over Cloud. July 16 th, SCENT HPC Summer GIST. SCENT (Super Computing CENTer) GIST (Gwangju Institute of Science & Technology)

Transcription

1 HPC over Cloud July 16 th, HPC Summer GIST (Super Computing CENTer) GIST (Gwangju Institute of Science & Technology) Dr. JongWon Kim jongwon@nm.gist.ac.kr

2 Interplay between Theory, Simulation, and Experiment

3 Software- Defined Infrastructure - 3 -

4 Open Innovation Platform T-Shaped vs I-Shaped Development Software-defined X (Infrastructure, Interfaces )

5 Agile & Economic Service Realization over Software-Defined Infrastructure Open Data + Open API + Open Resource (Compute + Storage + Networking)

6 Open API & Platforms (Big Data & Cloud) 6

7 Software-Defined Infrastructure Unified, Programmable & Virtualized Resources Templates

8 Configuration/Control/Visibility Challenges & Open APIs via Inter-connected Functions Compute Storage Networking DevOps X Zero-touch Configuration Instant Visibility Collective Analysis Flexible Control (forwarding, )

9 Let s Create Smart Things and Realize Smart Services Build Open APIs with Functions Converged (C/N/S) SmartX Box with Programmable & Virtualized Resources Architecture Your Smart Things with API Tools - 9 -

10 Industrial Internet, Smart Planet, Internet of Everything

11 Cloud: Rapid Expansion & Competition Cloud DC Traffic Amazon AWS Microsoft Azure Google Cloud VMware Hybrid Cisco Global Cloud Index OpenStack (RH, IBM, HP, Dell, ) Network Service Provider Cloud (AT&T, NTT, )

12 Futuristic Multilayer-integrated & Convergent Wireless + Mobile Networks (Cloud WAN Fabric + Service-aware Edge) Service-aware Edge Cloud Data Centers Cloud Data Centers Cloud WAN Fabric Wireless + Mobile (MiddleBox, ) (IP+Optical Integration) Cloud DC Cloud DC Cloud Data Centers Wireless + Mobile Last Modified 11/02/2013 IP??, More Switching + Simpler Routing?

13 HPC over Cloud

14 HPC over Cloud: Target & Benefits Interfacing With External components Sophisticated Migration policies Higher efficiency HPC over Cloud Benefits Workload bursting Accelerated Collaboration Pay-per-use Super Computing HPC over Cloud Cloud Scalability Simplified self-service Computing Economic (Easy & Improved Sharing), Green (Energy-efficient)

15 Meta-Operation National HPC Infrastructure F3 Racing Team = SC Center Target Problems General Computing #1 General Computing #2 Technical Computing Dedicated computing Software (Service) HPC over Cloud Unified HPC Boxes Platform (Tool) Infrastructure (Resource) Integrated Resource Pools Dedicated Resources

16 Vision for National HPC Services (Human Resources instead of Hardware Resources) Software Software Platform Platform Infrastructure (Hardware) Infrastructure (Hardware)

17 HPC over Cloud Institute HPC Centers National HPC Cloud Centers (Shared Infra/Platform) HPC SW Services (Licenses) CPU/GPUintensive Memory-intensive Data-intensive Virtualized HPC Clusters Dedicated HPC Centers

18 Building Shared National HPC Resources (every 6 years) Coordinated and Operated by National HPC Center HPC Cloud Center #1 HPC Cloud Center #2 국가 지방 HPC Cloud Center #3 HPC Cloud Center #4 #5 #6 기업

19 Virtualized HPC with HPC Boxes Scalein/out CPU/GPU-intensive Memory-intensive BW-intensive Pools of Unified HPC Boxes Dynamic On-demand Provisioning of Virtualized HPC

20 - 20 -

21 HPC over Cloud: HPC, HSC, High Performance Computing: Single program High Scalability Computing: Throughput focus; Can be distributed; e.g. Physics Simulation

22 HPC over Cloud Examples: MegaRun Cycle Computing MegaRun: Schrödinger s quantum chemistry software on 156,000 cores across 8 AWS regions (to sort through 205,000 compounds for solar panel). (Nov. 2013) $33,000 / 18 hours vs $68 million / 18 hours; $132,000 / 10 months (existing internal 300-core cluster)

23 HPC over Cloud Examples: CERN / Rackspace OpenLab OpenLab: Hybrid Cloud Model based on OpenStack (open-source cloud OS) COMPUTE NETWORKING STORAGE Rackspace Public Cloud Rackspace Private CERN Demonstration of federated identity and aggregated services between a Rackspace Private Cloud at CERN and at least one other cloud. CERN Private Cloud

24 DevOps Automation

25 DevOps for Software-Defined Infrastructure Development + Operation OpenStack (Puppet, Chef, )

26 Software Lifecycle and CI (Continuous Integration) / CD (Continuous Deployment) 26 Software Lifecycle: Development Testing (Staging) / QA Production / Deployment Master Software Coding (for Cloud OS Kernel + Service Frameworks and Tools) and Execute Continuous Integration for Agile and Economic Service Realization

27 System: a set of interacting or interdependent components forming an integrated whole A Computer System

28 CoreLink Cache Coherent Network based on AMBA (Advanced Microcontroller Bus Architecture) 5 CHI(Coherent Hub Interface) On a single die (i.e., chip)

29 Computers in Data Centers

30 Rack Scale Computing

31 Cloud Datacenter as a BIG Computer Luiz André Barroso, Urs Hölzle, The Datacenter as a Computer: An Introduction to the Design of Warehouse- Scale Machines Google Omega & Apache Mesos

32 Containers vs VMs Packaging workload tasks and Scaling

33 Docker: Open-Source Lightweight Container Open-Source Container Management Kubernetes LXC (Linux Container)

34 Docker & Open-Source Container Management Kubernetes Kubernetes (koo-ber-nay'-tace): Microsoft, IBM and Red Hat, CoreOS, SaltStack and Mesosphere, builds on top of Docker to construct a clustered container scheduling service.

35 Open-source Cluster Management Software Apache Mesos Apache Hadoop YARN

36 Cloud & PaaS (Platform as a Service) Model-driven PaaS: domain specific languages PaaS (Products; Frameworks): apaas (App Server aas) and language runtimes / DBaaS Mini-PaaS (Foundational PaaS, IaaS+): App. Containers + α 36 Dzone Cloud Report 2014

37 Big Data & Software- Defined Infrastructure

38 Big Data: Data Types and Frameworks Structured (relational and legacy) Unstructured data Hadoop/ MapReduce NoSQL

39 Cloud Datacenter as a BIG Computer Luiz André Barroso, Urs Hölzle, The Datacenter as a Computer: An Introduction to the Design of Warehouse- Scale Machines Google Omega & Apache Mesos

40 OpenStack Cloud OS for a Big Cloud Computer

41 Testbed with SmartX Boxes: Cloud Data Center in a Box Service layer (experiment layer) VM VM L2 VM L3 L2 VM A Virtual Playground For Experiment A VM VM L2 VM L3 L2 VM A Virtual Playground For Experiment B VM VM L2 VM L3 L2 VM A Virtual Playground For Experiment Z OF@TEIN Underlay Network VCPU VCPU VCPU VCPU VCPU VCPU VCPU VCPU Memory Memory Memory Memory Memory Memory Memory Memory Virtual Resource layer VM#1 VM#2 VM#3 VM#1 VM#2 VM#1 VM#2 VM#3 VCPU Vmemory Nova vstorage Cinder vswitch Neutron Opnestack VCPU Vmemory Nova vstorage Cinder vswitch Neutron Opnestack VCPU Vmemory Nova vstorage Cinder vswitch Neutron Opnestack KVM(Hypervisor) Kernel OS KVM(Hypervisor) Kernel OS KVM(Hypervisor) Kernel OS Physical resource layer Storage (SSD/HDD) Memory CPU NIC SmartX Box #1 Storage (SSD/HDD) Memory CPU NIC SmartX Box #2 Storage (SSD/HDD) Memory CPU NIC SmartX Box #K DevOps Templates for Virtual Playgrounds + OpenStack-based Functions APIs + SDN-Coordinated vnetworking

42 SmartX Box: Converged Resources - Workload Diversified Functions Auto Scaling Continuous Integration Instant Visibility Flexible Control Zero-touch Configuration 42

43 Thank you! Sent Inquiry to