Parallel Visualization, Data Formatting, Software Overview

Transcription

1 Parallel Visualization, Data Formatting, Software Overview Sean Ahern Remote Data Analysis and Visualization Center Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the National Science Foundation

2 Who are you? Director of the University of Tennessee Center for Remote Data Analysis and Visualization Visualization lead at ORNL s Leadership Computing Facility ORNL co-i for the SciDAC Visualization and Analysis Center for Enabling Technologies (VACET)

3 How do we handle the hard cases? But what about large, distributed data? Or distributed rendering? Or distributed displays? Or all three?

4 Applications are pushing data set sizes toward Exascale Spatial resolution Astrophysics, Radiation transport, Combustion, Chemistry, Molecular dynamics, Fusion, Multivariate Climate, Radiation transport, V&V, Temporal resolution Molecular dynamics, Climate, Fusion, Ensembles Climate, V&V,

5 Data scale limits scientific understanding Spatial resolution increasing in many domains Too many zones to see on screen (with or without high-res Powerwall) Temporal complexity increasing Climate simulations will have 100,000s of time steps Manually finding temporal patterns is tedious and error-prone Multivariate overload Climate simulations have variables Issues of data models and domain-specific data E.g. Multigroup radiation fields

6 End-to-end schematic Result nput HPC Simulation System Data generation! Query

7 Hero Increasing Dataset Size Large Medium Small Dataset size growth

8 Data size categories Small: Data is small enough to easily move anywhere Analysis/vis is generally done on local workstations Medium: Hero Large Data won t fit on local workstations Generally have to process on fat SMP systems Large: Medium Rather painful to move Requires distributed parallelism Hero: Small Functionally impossible to move Only approachable on largest computational platforms

9 Large data remote visualization Largest datasets require use of institutional resources Reduces data movement issues Allows exploitation of multiple GPUs Provides visualization to remote users Exploited by VisIt, ParaView, EnSight Hero Large Mediu Smal

10 Hero data Purchasing separate analysis systems at the petascale can be prohibitively expensive: $5-20 million Working to move largest vis/analysis tools to HPC architecture VisIt on Cray XT4/5 See Cray User s Group 08 ParaView on Cray XT4/5 Hero Large Medium See Cray User s Group 09 Parallel R on Cray XT5 This year (hopefully) Small

11 A lot of success has been through data flow networks (pipelines)

12 A lot of success has been through data flow networks (pipelines) Different data formats: NetCDF, HDF, text, CSV, PDB, Different types of data operations: Slicing, resampling, mesh transforms, filtering, Different ways of plotting: Pseudocolor, isosurfaces, volume rendering These are independent of each other!

13 Make these independent modules Data reading Data operations Data plotting Read data Filter Filter Plot

14 Make these independent modules Data reading Data operations Read data Read data Data plotting Filter Filter Filter This is a data flow network Plot Plot

15 Many software systems use this method VTK AVS/Express SCIRun

16 Fine for small to medium sized data Hero Increasing Dataset Size Large Medium Small

17 How do we handle the hard cases? But what about large, distributed data? Or distributed rendering? Or distributed displays? Or all three?

18 A good solution: data parallelization for all components Parallel Simulation Code P1 P0 P2 P3 Identical data flow networks on each processor. Networks differentiated by portion of data they operate on. I/O P0 P1 P2 P3 Minimal synchronization Scattered/gather Parallel Analysis Code No distribution (i.e. scatter), because scatter is done through choice of what data to read. Gather: done when rendering Proc 0 Proc 1 Proc 2 Distribution of data is key Rendering/compositing

19 Decomposition of data is key issue Parallelization strategy: each processor works on piece of data Key: Must have good decomposition of data Two I/O situations: Reading from file, file format enforces decomposition Reading from file, file format allows for arbitrary decomposition Three processing situations: P1 P0 P2 P3 Pre-calculated decomposition Must do overloading of chunks when more chunks than processors. Dynamic decomposition at I/O Must understand how to read out arbitrary chunks of data On-the-fly redistribution Processing-dependent

20 This can take us a long way Weak scaling study (isocontouring, volume rendering): ~63M cells/core Machine Type Problem Size # cores Jaguar Cray XT5 2T 32k Franklin Cray XT4 1T, 2T 16k, 32k Dawn BG/P 4T 64k Juno Linux 1T 16k, 32k Ranger Sun Linux 1T 16k Purple AIX 0.5T 8k 2T cells, 32K procs

21 This can take us a long way Weak scaling study (isocontouring, volume rendering): ~63M cells/core Machine Type Problem Size # cores Jaguar Cray XT5 2T 32k Franklin Cray XT4 1T, 2T 16k, 32k Dawn BG/P 4T 64k Juno Linux 1T 16k, 32k Ranger Sun Linux 1T 16k Purple AIX 0.5T 8k Since this work, people have reached 4 trillion and 8 trillion cells. 2T cells, 32K procs

22 This is only part of the puzzle Parallel Simulation Code P1 P0 P2 P3 I/O P0 P1 P2 P3 Parallel Analysis Code Proc 0 Proc 1 Proc 2 Rendering/compositing

23 The job of turning into imagery is complex One or more generators Fed to one or more displays display display display display display display display display display

24 Rendering to a single tile (sort last) Depth sort Only good for opaque For each set of, generate image and depth When merging, compare depth and choose pixel that s closest to the viewer Alpha blending Parallel volume rendering / transparent Process all in back-to-front order (not always easy to do in parallel) Blend with transparency, creating a final image Compositing algorithms: Binary tree Direct send Binary swap / n-way swap 2-3 swap Radix-swap display

25 Rendering to a single tile (sort last) Depth sort Only good for opaque For each set of, generate image and depth When merging, compare depth and choose pixel that s closest to the viewer Alpha blending Parallel volume rendering / transparent Process all in back-to-front order (not always easy to do in parallel) Blend with transparency, creating a final image Compositing algorithms: Binary tree Direct send Binary swap / n-way swap 2-3 swap Radix-swap display

26 Rendering to multiple tiles (sort first) Break up based upon camera frustum Select only that corresponds to each tile Send subset of over network to be rendered display display display display

27 Rendering to multiple tiles (sort first) Break up based upon camera frustum Select only that corresponds to each tile Send subset of over network to be rendered display display display display Used for many Powerwalls

28 Both at the same time Complicated combinations of sort first and sort last Some libraries to make this easier: Chromium IceT Equalizer Paracomp SAGE display display display display

29 Performance is highly dependent upon I/O rates Reading data into memory takes the most time Much more than data processing Much more than rendering and compositing Parallel filesystems are essential for large data Data formats designed for parallel filesystems are essential for large data

30 Joule metric run Weak scaling study (isocontouring, volume rendering): 103M-321M cells on 4k-12k cores Superlinear scaling in processing But wallclock time is dominated by I/O Isocontour % time spent in I/O 103 M cells 321 M cells 2.48 s 87.9% s 94.9%

31 Trillion cell study confirms that I/O dominates Weak scaling study (isocontouring, volume rendering): ~63M cells/core Machine Type Problem Size # cores Jaguar Cray XT5 2T 32k Franklin Cray XT4 1T, 2T 16k, 32k Dawn BG/P 4T 64k Juno Linux 1T 16k, 32k Ranger Sun Linux 1T 16k Purple AIX 0.5T 8k -Approx I/O time: 2-5 minutes -Approx processing time: 10 seconds 2T cells, 32K procs

32 VisIt: an end user visualization and analysis tool for extremely large data 5 major use cases: data exploration, data analysis, visual debugging, comparison, and presentation Demonstrated scaling to >100k cores Strong emphasis on building a usable, robust tool for scientists. Plugin model allows for wide range of capabilities Open source project, with direct support from DOE (SciDAC, NEAMS, ASC, BER) and NSF (TeraGrid XD) visitusers.org wiki with 400+ pages of documentation and over 80,000 page views Over 100,000 downloads R&D100 Award in 2005

33 ParaView: open-source, multi-platform data analysis and visualization application Based on VTK: Deep pipeline modification Quantitative and qualitative use cases Developed to process large datasets using distributed memory computing resources: laptops to supercomputers Superb, best-in-class rendering infrastructure Customizable GUI Access grid integration for collaboration Commercial support from Kitware

34 EnSight: leading commercial parallel visualization tool Specialized for engineering analysis CFD, FEA, etc. Commercial support from CEI Distributed rendering infrastruture Direct Powerwall support Virtual reality Distributed parallel data processing and rendering

35 Rmpi: Parallel statistical analysis Lustre file system Node 0 Node 1 Node 2 Node 0 produces graphics files MPI (via modified Rmpi) Write data reader to bring data from parallel file system Develop data-parallel analysis with full capability of R on every node

36 Central file store Scalable parallel throughput Lustre, GPFS, Panassas,... Shared between source and destination Provides zero copy access to simulation results Allows smart deployment of remote visualization capabilities

37 Data formatting is important to data understanding The purpose of computing is insight, not numbers. Richard Hamming The most exciting phrase to hear in science, the one that heralds new discoveries, is not 'Eureka!' (I found it!) but 'That's funny...' Isaac Asimov

38 We want our data model to reflect how the scientist thinks of the data coordinate atoms collections multidimensional fields materials arrays mesh arrays bytes named scalars vectors refinement bits words arrays meshes interpolation bonds time Simple Complex Each step gets us closer to the ideal. But each step is more domain-specific.

39 Meshes All data lives on a mesh Discretizes space into points and cells 1D, 2D, 3D All of these over time (up to 4D) Can have lower-dimensional meshes in a higher-dimensional space (e.g. 2D surface in 3D space) Provides a place for data to be located Defines how data is interpolated

40 Mesh structure 1D Curves 2D/3D meshes Rectilinear Curvilinear Unstructured Curve Rectilinear Points AMR Curvilinear Unstructured Molecular Molecular Domain decomposed Ghost zones/nodes Points AMR

41 Variables Scalars, Vectors, Tensors Sits on points or cells of a mesh Points: linear interpolation Cells: piecewise constant Could have different dimensionality than the mesh (e.g. 3D vector data on a 2D mesh)

42 Materials Sometimes used in engineering codes Describes disjoint spatial regions at a sub-grid level Volume/area fractions

43 Parallel meshes Provides aggregation for meshes A mesh may be composed of hundreds of thousands of mesh blocks. Allows data parallelism

44 AMR meshes Mesh blocks can be associated with patches and levels. Allows for aggregation of meshes into AMR hierarchy levels.

45 Data model goals Rich Encompasses computational domain as much as possible Sharability Being able to easily move data between tools Being able to give data to other people Rapid I/O Minimize time dumping out data Minimize time reading data Parallel I/O Support access by multiple threads at once on parallel platforms. More Self description Changes to data shouldn t require a change in the format New data should be discoverable without human intervention

46 Data Storage OpenFOAM Fluent CGNS NetCDF XDMF Flash EnSight Gold ADIOS VTK S3D STL Silo ViSUS HDF5 Tecplot PDB Chombo

47 Underlying libraries NetCDF: Named array/scalar storage Generalized attributes Parallel access HDF5: Named array/scalar storage Hierarchical storage Parallel read and write ASCII Simple for humans to read and write Horrible for computers to read, especially in parallel

48 Higher-level libraries and formats ADIOS: Used for very fast I/O off HPC systems HDF5 and native storage Silo: Very rich (parallel) data model on top of HDF5 PDB (Protein Data Bank) Used for storage of molecular data Based on ASCII VTK: Native storage for the VTK pipeline visualization system Both ASCII and binary formats

49 Higher-level libraries and formats (cont) XDMF: Neat format that splits data and metadata ASCII XML file for metadata HDF5 file for heavyweight data.

50 Community standards NetCDF Climate and Forecast (CF) convention: Promotes sharing of data among climate researchers Standard naming Coordinate systems Dimensions, missing data, units, etc. Many communities build formats on top of others: S3D is over NetCDF GTC, Chombo, Enzo, PFLOTRAN, Flash, etc. are over HDF5

51 Summary Parallel visualization is an important capability for gaining insight from petascale (and soon to be exascale) data. It s as difficult of a job as doing the original HPC simulation itself. Rendering is not yet a solved problem. Data storage and data models are critical to high performance and scientific understanding.