MOC Dataset Repository and Big Data as a Service Platform

Transcription

1 MOC Dataset Repository and Big Data as a Service Platform 1

2 BDaaS MOC 2

3 BDaaS MOC Umbrella talk: 2

4 BDaaS MOC Umbrella talk: Showcase how MOC research projects fit together 2

5 BDaaS MOC Umbrella talk: Showcase how MOC research projects fit together Demonstrate opportunities provided by MOC combine innovation offered by industry partners and researchers demonstrate at-scale capability sharing a production grade HPC system enable innovation across hw, sw, platform layers 2

6 BDaaS MOC 3

7 BDaaS MOC MOC BDaaS: Host and serve data sets Stand up BigData environments on-demand Operate in-situ on these data sets 3

8 MOC Cloud Dataverse Public Dataset Repository 4

9 MOC Cloud Dataverse Public Dataset Repository BigData clusters spinned via OpenStack Sahara to compute over datasets 5

10 MOC if not enough resources Cloud Dataverse Public Dataset Repository BigData clusters spinned via OpenStack Sahara to compute over datasets 6

11 MOC Expand OpenStack cluster using HIL and BMI if not enough resources Cloud Dataverse Public Dataset Repository BigData clusters spinned via OpenStack Sahara to compute over datasets 7

12 MOC prefetch while cluster is being deployed BigData Caching: cache datasets for the BigData cluster Cloud Dataverse Public Dataset Repository BigData clusters spinned via OpenStack Sahara to compute over datasets 8

13 BDaaS HW Infrastructure Sharing compute resources of the Engage1 HPC MGHPCC ~300 Servers: 18 racks, each with 18 servers Sharing orchestrated via HIL & BMI x 18 9

14 BDaaS Infrastructure Extended the cluster by adding 10Gbs Intel NIC to each server Brocade VDX switch and a Cache Server to each rack Each Cache Server contains two 1.5TB Intel NVMe-SSDs x 18 10

15 BDaaS Network Brocade switches connected in a bifurcated ring topology Three distinct paths between any pairs Bisectional bwidth: 1.05 Tbps Average hops:

16 BDaaS Storage 10 Lenovo X3550 Servers Distributed as three racks in the ring 90 HDDs 326 TB storage Ceph with 90 OSDs 12

17 One OpenStack Rack Engage1 OpenStack Cluster Expand/shrink based on load Use HIL and BMI to orchestrate sharing 13

18 Summary 14

19 Summary effort to host/serve large-scale datasets & provide fast computation platform to analyze them 14

20 Summary effort to host/serve large-scale datasets & provide fast computation platform to analyze them Convergence point for many MOC projects Cloud Dataverse, HIL, BMI, BigData Caching, Monitoring, SDN 14

21 Summary effort to host/serve large-scale datasets & provide fast computation platform to analyze them Convergence point for many MOC projects Cloud Dataverse, HIL, BMI, BigData Caching, Monitoring, SDN Innovation across industry partners & academia combined Specific networking, storage and computation solutions Tested 14

22 Summary effort to host/serve large-scale datasets & provide fast computation platform to analyze them Convergence point for many MOC projects Cloud Dataverse, HIL, BMI, BigData Caching, Monitoring, SDN Innovation across industry partners & academia combined Specific networking, storage and computation solutions Tested Significant user-interest drive for MOC and services 14