FROM FILE SYSTEMS TO SERVICES: CHANGING THE DATA MANAGEMENT MODEL IN HPC

Transcription

1 imulation, Observation, and Software: upporting exascale storage and I/O FROM FILE SYSTEMS TO SERVICES: CHANGING THE DATA MANAGEMENT MODEL IN HPC ROB ROSS, PHILIP CARNS, KEVIN HARMS, JOHN JENKINS, AND SHANE SNYDER GARTH GIBSON, GEORGE AMVROSIADIS, CHUCK CRANOR, AND QING ZHENG JEROME SOUMAGNE AND JOE LEE GALEN SHIPMAN AND BRAD SETTLEMYER Argonne National Laboratory Carnegie Mellon University The HDF Group Los Alamos National Laboratory

2 CHANGES IMPACTING HPC DATA AND STORAGE

3 MORE STORAGE/MEMORY LAYERS DRAM Lustre Parallel File System HPSS Parallel Tape HPC Before 2016 Memory Parallel File System Archive HPC After 2016 Memory Burst Buffer Parallel File System Campaign Storage Archive Why BB: Economics (disk bw/iops too expensive) PFS: Maturity and BB capacity too small Campaign: Economics (tape bw too expensive) Archive: Maturity and we really do need a forever 1-2 PB/sec Residence hours Overwritten continuous 4-6 TB/sec Residence hours Overwritten hours 1-2 TB/sec Residence days/weeks Flushed weeks GB/sec Residence months-year Flushed months-year 10s GB/sec (parallel tape Residence forever Slide from Gary Grider (LANL).

4 SIMULATION WORKFLOW APEX Workflows, LANL, NERSC, SNL, SAND O, LA-UR

5 SPECIALIZATION OF DATA SERVICES Application Executables and Libraries SPINDLE Checkpoints Application Data SCR FTI Intermediate Data Products DataSpaces Kelpie MDHIM 5

6 Rusty ADLB Data store and pub/sub. DataSpaces Manish Data store and pub/sub. DataWarp Burst Buffer mgmt. FTI Franck Checkpoint/restart mgmt. Kelpie Dist. in-mem. key/val store SPINDLE Exec. and library mgmt. Provisioning Comm. Local Storage Fault Mgmt. and Group Membership Security MPI ranks MPI RAM N/A N/A Indep. job Admin./ sched. Dart DVS/ lnet RAM (SSD) XFS, SSD Under devel. Ext. monitor N/A Kernel, lnet MPI ranks MPI RAM, SSD N/A N/A MPI ranks Launch MON Nessie RAM (Object) N/A TCP RAMdisk N/A Obfusc. IDs Shared secret

7 COMPOSING DATA SERVICES

8 OUR GOAL Enable composition of data services for DOE science and systems Application-driven Identify and match to science needs Traditional data roles (e.g., checkpoint, data migration) New roles (e.g., equation of state/opacity databases) Develop/adapt building blocks Communication Concurrency Local Storage Resilience Authentication/Authorization 8

9 COMMUNICATION: MERCURY Mercury is an RPC system for use in the development of high performance system services. Development is driven by the HDF Group with Argonne participation. Portable across systems and network technologies Efficient bulk data movement to complement control messages Builds on lessons learned from IOFSL, Nessie, lnet, and others Client RPC proc Metadata (unexpected +expectedmessaging) Bulk Data (RMA transfer) RPC proc Server Network Abstraction Layer 9

10 CONCURRENCY: ARGOBOTS Argobots is a lightweight threading/tasking framework. Features relevant to I/O services: Flexible mapping of work to hardware resources Ability to delegate service work with fine granularity across those resources Modular scheduling We developed asynchronous bindings to: Mercury LevelDB POSIX I/O Working with Argobots team to identify needed functionality (e.g., idling) 10 ES 1 U T T U Argobots Execution Model S T T Scheduler S U S T Pool Sched E E E E U ULT T Tasklet... E Event ES n

11 THREE EXAMPLE SERVICES

12 1. REMOTELY ACCESSIBLE OBJECTS API for remotely creating, reading, writing, destroying fixed-size objects/extents libpmem ( for management of data on device Client app Argobots Margo Mercury Object API Argobots Margo Mercury libpmem RAM, NVM, SSD CCI IB/verbs CCI Target P. Carns et al. Enabling NVM for Data-Intensive Scientific Services. INFLOW 2016, November

13 1. REMOTELY ACCESSIBLE OBJECTS: HOW MUCH LATENCY IN THE STACK? 1000 Write Read C1 C usec NOOP Latency (us) noop MiB 512 KiB 256 KiB 128 KiB 64 KiB 32 KiB 16 KiB 8 KiB 4 KiB 2 KiB 1 KiB Access size (bytes) FDR IB, RAM disk, 2.6 usec round-trip (MPI) latency measured separately

14 2. TRANSIENT FILE SYSTEM VIEWS: DELTAFS Supporting legacy POSIX I/O in a scalable way. App proc App proc Deltafs Deltafs lib Deltafs lib server proc Deltafs server proc ls -l tail -F Deltafs fuse 2 RPC deltafs servers for metadata /deltafs Deltafs comm world All procs are user-space, and run on compute nodes 4 monitor progress 3 directly access file data 1 load snapshot(s) 5 dump snapshot(s) 14

15 3. CONTINUUM MODEL COUPLED WITH VISCOPLASTICITY MODEL Shockwave Viscoplasticity model [1]: Lulesh continuum model: - FFT based PDE solver - Lagrangian hydro dynamics - Structured sub-mesh - Unstructured mesh Future applications are exploring the use of multi-scale modeling As an example: Loosely coupling continuum scale models with more realistic constitutive/response properties e.g., Lulesh from ExMatEx Fine scale model results can be cached and new values interpolated from similar prior model calculations R. Lebensohn et al, Modeling void growth in polycrystalline materials, Acta Materialia,

16 3. FINE SCALE MODEL DATABASE Goals Minimize fine scale model executions Minimize query/response time Load balance DB distribution Approach Start with a key/value store Distributed approx. nearest-neighbor query Data distributed to co-locate values for interpolation Import/export to persistent store Status Mercury-based, centralized in-memory DB service Investigating distributed, incremental nearest-neighbor indexing Application domain Distributed DB Import/export DB instances Query 6D space for nearest neighbors 16

17 FINAL THOUGHTS Stage is set for distributed services in HPC Richer resource management Increasing emphasis on workflows Convergence of data intensive and computational science If we re going to get rid of POSIX, we need alternative(s) Real opportunity to make life easier for applications And have fun doing it! 17

18 THIS WORK IS SUPPORTED BY THE DIRECTOR, OFFICE OF ADVANCED SCIENTIFIC COMPUTING RESEARCH, OFFICE OF SCIENCE, OF THE U.S. DEPARTMENT OF ENERGY UNDER CONTRACT NO. DE-AC02-06CH11357.