A New NSF TeraGrid Resource for Data-Intensive Science

Transcription

1 A New NSF TeraGrid Resource for Data-Intensive Science Michael L. Norman Principal Investigator Director, SDSC Allan Snavely Co-Principal Investigator Project Scientist Slide 1

2 Coping with the data deluge Advances in computing technology have resulted in a Moore s Law for Data Amount of digital data from instruments doubles every 18 months (DNA sequencers, CCD cameras, telescopes, MRIs, etc.) Density of storage media keeping pace with Moore s law, but not I/O rates Time to process exponentially growing amounts of data is growing exponentially Latency for random access limited by disk read head speed Slide 2

3 What is Gordon? A data-intensive supercomputer based on SSD flash memory and virtual shared memory SW Emphasizes MEM and IOPS over FLOPS A system designed to accelerate access to massive data bases being generated in all fields of science, engineering, medicine, and social science Random IO to SSD x faster than HDD In production in 1Q2012 We have a working prototype called Dash available for testing/evaluation now (LSST, PTF) Slide 3

4 The Memory Hierarchy of a Typical HPC Cluster Shared memory programming Message passing programming Latency Gap Disk I/O Slide 5

5 The Memory Hierarchy of Gordon Shared memory programming Disk I/O Slide 6

6 Gordon s 3 Key Innovations Fill the latency gap with large amounts of flash SSD 256 TB >35 million IOPS Aggregate CPU, DRAM, and SSD resources into 32 shared memory supernodes for ease of use 8 TFLOPS 2 TB DRAM 8 TB SSD High performance parallel file system 4 PB >100 GB/s sustained Slide 7

7 Results from Dash*: a working prototype of Gordon *available as a TeraGrid resource Slide 8

8 Palomar Transient Factory (PTF) collab. with Peter Nugent Nightly wide-field surveys using Palomar Schmidt telescope Image data sent to LBL for archive/analysis 100 new transients every minute Large, random queries across multiple databases for IDs Slide 9

9 PTF-DB Transient Search DASH-IO- SSD Forward Q1 11s (145x) Backward Q1 100s (24x) Existing DB 1600s 2400s Random Queries requesting very small chunks of data about the candidate observations Slide 10

10 MOPS Application Runs Faster Under vsmp as on Hardware SMP and ccnuma Moving Object Pipeline System (MOPS) used in Asteroid Tracking. Part of the Large Synoptic Survey Telescope (LSST) Project Collab. with Jonathan Myers, LSST Corp. Algorithm is serial (no MPI) 135 GB required for the test case Dash Node is a dual socket, 8 core Nehalem node with 48 GB of memory. Triton PDAF is an 8 socket, 32 core, Shanghai node with 256GB memory Ember is a SGI Altix UV system with 384 Nehalem cores, 2 TB RAM in a SSI Slide 11

11 Gordon Training Events Getting Ready for Gordon: Using vsmp May 10-11, 2011 (next week) Will be recorded for web download Getting Ready for Gordon: Summer Institute August 6-17, 2011 Contact Susan Rathbun Slide 12

12 How to get time NOW: Request a start-up allocation on Dash at After Sept 2011: Request a start-up or large allocation on Gordon at For more information, see Or me at MLNORMAN@UCSD.EDU Slide 13

13 RESERVE SLIDES

14 Gordon Architecture: Supernode 32 Appro Extreme-X compute nodes Dual processor Intel Sandy Bridge 240 GFLOPS 64 GB 2 Appro Extreme-X IO nodes Intel SSD drives 4 TB ea. 560,000 IOPS ScaleMP vsmp virtual shared memory 2 TB RAM aggregate 8 TB SSD aggregate vsmp memory virtualization 240 GF Comp. Node 64 GB RAM 4 TB SSD I/O Node 240 GF Comp. Node 64 GB RAM

15 Gordon Architecture: Full Machine 32 supernodes = 1024 compute nodes Dual rail QDR Infiniband network 3D torus (4x4x4) 4 PB rotating disk parallel file system >100 GB/s D D D D D D

16 Gordon Aggregate Capabilities Speed Mem (RAM) Mem (SSD) Mem (RAM+SSD) Ratio (MEM/SPEED) IO rate to SSDs Network bandwidth Network latency Disk storage Disk IO Bandwidth >200 TFLOPS 64 TB 256 TB 320 TB 1.31 BYTES/FLOP 35 Million IOPS 16 GB/s bi-directional 1 msec. 4 PB >100 GB/sec