Outline. In Situ Data Triage and Visualiza8on

Transcription

1 In Situ Data Triage and Visualiza8on Kwan- Liu Ma University of California at Davis Outline In situ data triage and visualiza8on: Issues and strategies Case study: An earthquake simula8on Case study: A turbulent combus8on simula8on Closing remarks and discussion 1

2 In Situ Visualiza8on Reducing/visualizing data in situ as the simula8on is running The most feasible solu8on for extreme- scale data analysis What does it mean to be running in situ? A defini8on for in situ processing: Process the data before it is wriuen to the disk. Two technical direc8ons meet this defini8on: Co- located on a node (techniques that exploit data locality) Concurrent processing (shipping data to dedicated vis/analysis nodes, possibly reducing first) 2

3 Mo8va8on for in situ Monitoring: Want to make sure the calcula8on is running well Data reduc8on: We won t be able to write all this data we must do it in situ And new opportuni8es: we have a chance to do something we haven t before we ll get to see all the data and that is a posi8ve we can capture transient events If we find something interes8ng, we can add extra auen8on to that event steering the analysis Debugging and performance op8miza8on Current state of the art Tightly coupled in situ analysis and visualiza8on that are as scalable as the simula8on Efforts for write once, use many in produc8on se`ng (Paraview and VisIt); problems remain in terms of minimizing memory footprint. Some experience understanding what problems can be solved in situ and what can t. Some efforts in iden8fying interes8ng data in situ. Feature detec8on (e.g. finding a hurricane in a climate code) Embedded as part of I/O pipeline (i.e. advanced concurrent processing) 3

4 Issues With Current Approaches Some analysis requires looking at large 8me windows how will we do this? You may not know what is interes8ng up front. New paradigms for skipping back in 8me. How to obtain interac8ve explora8on in this se`ng? Some vis & analysis rou8nes are fundamentally memory heavy (e.g. derived quan88es) and some are fundamentally compute heavy. How does these fit? Not just solving one problem; we will be doing mul8ple analyses concurrently. Further, aimed at single inves8gator approaches how do we allow large groups of people to analyze the data in situ? (e.g. climate, policy decisions) In Situ Visualization Requirements Integra8on of simula8on and visualiza8on codes Low memory overhead Low computa8onal cost Sharing the domain decomposi8on and data structures Scalable parallel visualiza8on algorithms Memory bandwidth is a key issue in some processing models (e.g. the concurrent model where a dedicated node is processing a large amount of data) Addi8onal requirements for interac8ve monitoring/ steering and different types of visualiza8on 4

5 Run simula8on pipelines end- to- end in parallel Eliminate scalability boulenecks Execute all components on the same processors Enable simula8on- 8me visualiza8on steering Increase sustained flops as the problem size increases Won SC06 HPC Analy8cs Challenge on 1024 processors of Cray XT3 5

6 0.661M 9.92M 31.3M 114M 462M 1.22B Elements 6

7 5/30/11 Case Study II: a Turbulent Combustion Simulation Direct numerical simula8on of turbulent combus8on (up to 2025x1600x400) Visualizing both par8cle data and volume data Using a new highly scalable parallel renderer Visualiza8on takes under 1% of overall 8me Using up to 6,480 processors of the Cray XT5 at NCCS/ORNL The largest, most scalable in situ visualiza8on ever achieved (2009) In Situ Vis is feasible! In Situ Visualiza8on Results 1215x960x240 CH2O/HO2 CH3/OH 7

8 Test Results VisIt Through the libsim library of VisIt, the simula8on can interface with VisIt with minimal modifica8on to the simula8on code. There are two interfaces in libsim: one to drive the VisIt server, and one to hand data to the VisIt server upon request. In the libsim library, a database plug- in uses data access callback func8ons to read data from memory in response to requests form the VisIt server 8

9 ParaView Two ways: Through ParaView s coprocessing library providing a programma8c interface to ParaView Using ADIOS I/O framework avoiding disk I/O That is, by using the ADIOS API for I/O, simula8on data can be wriuen to disk or to a stage area hosted by another parallel applica8on. Summary In situ processing offers ways to more intelligently figure out which data to get off the machine Simula8ons will need to accommodate hard analysis issues. Paradigm shiq to analysis- driven science/ discovery- driven science? There will be a transi8on for what scien8sts can and cannot do in terms of analysis; how we will help/ enable this transi8on? Difficult upcoming conversa8ons with communi8es that are leery of in situ. (e.g. climate) There are many remaining research challenges with in situ processing. 9

10 Final Remarks Visualiza8on is the interface for solving data intensive problems Visualiza8on may be used for valida8on, explora8on, and presenta8on. Visualiza8on should be an integral part of the overall engineering design or scien8fic discovery process. In situ processing is absolutely required as we move towards exascale. Discussion 10

11 US DOE ASCR US DOE SciDAC Program US NSF PetaApps Program US NSF OCI Program US NSF CCF Program US NSF FODAVA Program US NSF HECURA Program US AirForce HP Labs Nokia Research AT&T Research Acknowledgments 11