NNSA Advanced Simulation and Computing An Overview of Data Management Issues

Transcription

1 NNSA Advanced Simulation and Computing An Overview of Data Management Issues Steve Louis Lawrence Livermore National Lab LLNL ASC VIEWS Program Lead TEL: FAX: Presented at the March 16-18, 2004 DMW 2004 Workshop Stanford Linear Accelerator Center UCRL-PRES This work was performed under the auspices of the U.S. Department of Energy by University of California Lawrence Livermore National Laboratory under contract No. W-7405-Eng-48.

2 DISCLAIMER This document was prepared as an account of work sponsored by an agency of the United States Government. Neither the United States Government nor the University of California nor any of their employees, makes any warranty, express or implied, or assumes any legal liability or responsibility for the accuracy, completeness, or usefulness of any information, apparatus, product, or process disclosed, or represents that its use would not infringe privately owned rights. Reference herein to any specific commercial products, process, or service by trade name, trademark, manufacturer, or otherwise, does not necessarily constitute or imply its endorsement, recommendation, or favoring by the United States Government or the University of California. The views and opinions of authors expressed herein do not necessarily state or reflect those of the United States Government or the University of California, and shall not be used for advertising or product endorsement purposes. This work was performed under the auspices of the U.S. Department of Energy by University of California Lawrence Livermore National Laboratory under contract No. W-7405-Eng-48.

3 Typical ASC application characterization ASC codes at LLNL are complex multi-physics codes. The codes integrate initial value partial differential equations for the conservation of particles, momentum, and energy for important elements and constituents of the devices. Typical calculations use 10,000 to 1,000,000,000 mesh cells, depending on the problem and the desired resolution. The larger problems must be domain decomposed to fit on available memory sizes for distributed memory systems. Partial differential equations are solved with a combination of explicit, implicit, and Monte Carlo techniques. Linear and non-linear solvers play an important role. The problems are integrated in time from an initial state to a final state. DMW 2004 Workshop, SLAC, 16-Mar

4 Typical ASC application characterization Typical phases Interactive problem set-up for a large simulation Running one or more very large 2-D or 3-D calculations Visualization, comparison, validation, archive of results Typically the large calculations need terascale computing ASC uses terascale computing and 1000 s of CPUs now As these codes run over many thousands of processors, huge data dumps must be frequently made to allow for restarts (a.k.a. defensive I/O). Visualization and/or physics files are also saved regularly for subsequent analyses (a.k.a. productive I/O). DMW 2004 Workshop, SLAC, 16-Mar

5 From current LLNL Compute & I/O Model Derived from peak, platform and network bandwidth, historical usage patterns, user input, and projections A week long run can generate up to 30 TB of data, and moving data to archive at 1/10 rate at which it was generated now requires an I/O throughput rate of 495 MB/s However, only half the data might typically need to be stored, so the planned throughput rate to archive is less ~250 MB/s Some users tend to keep data on the platform for postprocessing and visualization purposes, but some users may move entire dataset to a separate visualization server Site-Wide-Global-File-System would significantly shift this model and would reduce the need to explicitly move data DMW 2004 Workshop, SLAC, 16-Mar

6 Some relevant ancient history on data issues Joint DOE/NSF 1998 Workshop Series on Data and Visualization Corridors (DVCs) Oxnard, CA January (Frameworks) Santa Fe, NM March 4-6 (User Requirements) Bodega Bay, CA April 6-8 (Technology Trends) Duck, NC May (Writing Retreat I) Wye River, MD July 5-8 (Writing Retreat II) Report published September 1998, Technical Report CACR-164, There is a pressing need for new methods of handling truly massive datasets, of exploring and visualizing them, and of communicating them over geographic distances (From Foreword of Report on the 1998 DVC Workshop Series ) DMW 2004 Workshop, SLAC, 16-Mar

7 Some 1998 workshop recommendations (most, if not all, are still relevant today ) Establish a vigorous, interdisciplinary program to improve the ability to see and understand output from large data sources Conduct new research and development focused on data management, graphics, and scientific visualization for large-scale data Increase the federal effort and annual investment in DVC R&D by $ M per year over current levels Develop and support a national strategy to incorporate results of DVC R&D into national laboratories, research centers, and national infrastructure programs DMW 2004 Workshop, SLAC, 16-Mar

8 DVC system model, without archiving (a la John van Rosendale, circa 1998) WAN SAN LAN Simulation Engine Data Manipulation Engine Rendering Engne Rendering Engne wow! DMW 2004 Workshop, SLAC, 16-Mar

9 DVC model is still relevant for cluster-based hardware and software now deployed at LLNL New levels of graphics performance based on COTS technologies (with Lintel, Quadrics, nvidia cards) Tight coupling to the compute platform via Lustre and gige links Distributed parallel software stack (open source Chromium, DMX) Parallel, scalable end-user applications (e.g., VisIt, Blockbuster) Multiple display capabilities (Power Walls, office high-resolution displays) Provides the blueprint for future Purple-related visualization and data deployment MCR: 1,116 P4 Compute nodes Common Lustre file system (90TB) 128 Port QsNet Elan3 1,152 Port QsNet Elan3 GbEnet Federated Switch PVC: 58 P4 Render/6 Display nodes Digital Desktop Analog Desktop Displays 3x2 PowerWall DMW 2004 Workshop, SLAC, 16-Mar

10 LLNL Open Computing Facility (OCF) Clusters, Networks, Storage 4 Login nodes 16 Gateway 350 MB/s with 6 Gb-Enet delivered Lustre I/O over 4x1GbE GW GW MDS MDS B451 Thunder 1,024 Port QsNet Elan4 1,004 Quad Itanium2 Compute Nodes 4 Login nodes 32 Gateway 190 MB/s with 4 Gb-Enet delivered Lustre I/O over 2x1GbE GW GW MDS MDS B439 MCR 1,152 Port QsNet Elan3 1,114 Dual P4 Compute Nodes 2 Login nodes 32 Gateway 190 MB/s with 4 Gb-Enet delivered Lustre I/O over 2x1GbE GW GW MDS MDS B439 ALC Port QsNet Elan3 924 Dual P4 Compute Nodes Federated Ethernet B113 LLNL External Backbone SW SW SW SW SW SW SW SW 24 PFTP 1024 I/O Nodes PPC440 TSF OST Heads B OST Heads HPSS Archive BG/L torus, global tree/barrier 65,536 Dual PowerPC 440 Compute Nodes BB/MKS Version 6 Dec 23, 2003 OCF SGS File System Cluster (OFC) OST OST OST OST OST OST OST MM Fiber 1 GigE SM Fiber 1 GigE Copper 1 GigE 2Gig FC GW OST OST OST Terabytes GW PVC Port Elan3 52 Dual P4 Render Nodes OST B451 OST OST 2 Login nodes 6 Dual P4 Display Dual P4 Head FC RAID 146, 73, 36 GB

11 ASC Data Storage and I/O Roadmap Area CY 02 CY 03 CY 04 CY 05 CY 06 CY 07 ASC Perf. Targets SGSFS 30 TF 1 PB Archive 7-20 GB/s parallel FS 1 GB/s to Arch. tape Lustre Lite on Linux SIO Libs Limited App use Archive HPSS 4.1 production DFS COTS DFS in production Lustre Lite limited production Use by key Apps TF 7 PB Archive 100 GB/s parallel FS 10 GB/s to Arch. tape Lustre w. OST striping Broad App use HPSS 4.5 HPSS 5.1 production Metadata fixes Pilot NFSv4 on Linux 180 GB/disk 30 MB/s single disk 300 GB tape capacity 70 MB/s max tape rate Deploy NFSv4 Lustre early prod. HPSS 6.1 replace DCE Integrate NFSv4 w. Lustre 600 GB/disk 80 MB/s single disk 600 GB tape capacity 120 MB/s tape rate 200 TF 25 PB Archive 200 GB/s parallel FS 20 GB/s to Arch.tape Lustre stable prod. Perf. tuned for Lustre TBD 1200 GB/disk 200 MB/s single disk 2 TB tape capacity 200 MB/s tape rate DMW 2004 Workshop, SLAC, 16-Mar

12 LLNL HPSS storage slide from three years ago Accomplishments A 20x performance increase in 15 months (faster nets and disks) PSE Milepost demonstrated 170 MB/s aggregate throughput White-to-HPSS Large single file transfer rates of up to 80MB/s White-to-HPSS Large singe file transfer rates of up to 150MB/s White-to-SGI Challenges Yearly doubling of throughput is needed for next machine MB/s Aggregate Throughput to Storage Moved to Faster Disk on Faster Nodes & multi-node Concurrency Moved to Jumbo GE & Parallel Striping Moved to SP Nodes 120 MB/s 170 MB/s 40 Moved to 20 HPSS 1 MB/s 4 MB/s 6 MB/s 9 MB/s 0 FY96 FY97 FY98 FY99 FY00 FY01 At 170 MB/s, 2TB of data moves to storage in less than 4 hours. A year and a half ago it took two and a half days to move the same amount of data DMW 2004 Workshop, SLAC, 16-Mar

13 Continued improvement in throughput needed to meet requirements of new ASC platforms Aggregate Throughput to Storage Moved to Faster Disk using multiple Htar sessions on multiple nodes Moved to Jumbo GE, Parallel Striping, Faster Disk & Nodes using multiple pftp sessions 12/03 Throughput 854 MB/s 1,037 MB/s MB/s MB/s Moved to HPSS 4 MB/s Moved to SP Nodes 6 MB/s 9 MB/s 170 MB/s 120 MB/s FY96 FY97 FY98 FY99 FY00 FY01 FY02 FY03 Note that this graph represents a 115x performance improvement in four years! DMW 2004 Workshop, SLAC, 16-Mar

14 A Tri-lab historical timeline for motivating improvement in scalable parallel file systems Proposed PathForward activity for SGSFS Propose initial architecture PathForward proposal with OBSD vendor, Panasas born RFQ, analysis, recommend funding open source OBSD development and NFSv4 efforts Begin partnering talks negotiations for OBSD and NFSv4 PathForwards Build initial requirements document SGSFS workshop You re Crazy Tri-Lab joint requirements document complete PathForward team formed to pursue an RFI/RFQ approach, RFI issued, recommend RFQ process Alliance contracts placed with universities on OBSD, overlapped I/O and NFSv4 Lustre PathForward effort is born U Minn Object Archive begins Another workshop: Re-invent Posix I/O? Are We Still Crazy? DMW 2004 Workshop, SLAC, 16-Mar

15 From the June 2003 HECRTF workshop report For info: NNSA Tri-labs (Lee Ward of SNL, Tyce McLarty of LLNL, Gary Grider of LANL) were the ASC I/O representatives at this workshop Overwhelming consensus was POSIX I/O is inadequate 5.5. Data Management and File Systems We believe legacy, POSIX I/O interfaces are incompatible with the full range of hardware architecture choices contemplated The interface does not fully support the needs for parallel support along the I/O path An alternative, appropriate operating system API should be developed for high-end computing systems DMW 2004 Workshop, SLAC, 16-Mar

16 LLNL ASC SDM project organization areas emphasize many data management issues Metadata Infrastructure and Applications Development effort creating metadata-based environment for managing and simplifying data access (Metadata Tools Project) Data Access and Preparation Research project helping scientists explore terabytes of scientific simulation data by permitting ad-hoc queries over the data (Ad Hoc Query Project) Data Discovery Research projects looking at various aspects of feature extraction, data mining, and pattern recognition (Sapphire Project) Data Models and Formats Development effort generating models and file formats to ensure that ASC's scientific data can be freely exchanged (Limit Point Systems contract) DMW 2004 Workshop, SLAC, 16-Mar

17 Scalable Visualization Tool Development for Interactive Exploitation of Large Data Sets VIEWS developed tools (e.g., VisIt, TeraScale Browser provide vehicle for advanced research capabilities Improved large surface handling Parallel distributed volume rendering Topological data representations View-dependent surface rendering Programmable HW graphics rendering time t0 0 isovalue Isosurface area colormap min max DMW 2004 Workshop, SLAC, 16-Mar

18 A take home message: This five-year-old slide on issues is still as relevant as ever Traditional systems for archives and data management not necessarily suitable for the organization of ASC simulation data Traditional systems for realistic rendering and visualization not necessarily suitable for exploration of ASC simulation data ASC needs scalable, flexible methods for: navigation / archive of massive data sets efficient data subset selection / retrieval time-step multivariate animation capability interactive computational monitoring / steering advanced application development and debugging distance and distributed access to massive data DMW 2004 Workshop, SLAC, 16-Mar

19 The Long Term Challenge for ASC: (yet another five-year-old, but relevant slide) Simple linear scaling of existing data management components won t necessarily work A new use paradigm will be required Introduce users to new, innovative tools Motivate and enable vigorous research efforts Explore high-risk, high-reward technologies Identify technology shortfalls and barriers DMW 2004 Workshop, SLAC, 16-Mar