Large Data Visualization

Transcription

1 Large Data Visualization Seven Lectures 1. Overview (this one) 2. Scalable parallel rendering algorithms 3. Particle data visualization 4. Vector field visualization 5. Visual analytics techniques for complex network data 6. Advanced concepts for large data visualization 7. In-situ visualization and concluding remarks 1

2 Lecture Goals and Format Introduce new approaches to large data analysis and visualization problems Focus on concepts rather than implementation details Consider real-world application problems Motivate design and discussion of next generation technologies rather than technologies of today Provide research prototypes for a proof of concept Hands-On Sessions Four software prototypes: 2

3 Hands-On Sessions Four software prototypes: 1. View dependent streamline visualization 2. Sketch based streamline and particle trajectory visualization 3. An animation framework for volume visualization 4. Network analysis and visualization Interactive demonstration by Cheng-Kai (today) Three tasks (today, 6/15, 6/16, 6/21) Playing with different streamline visualization methods using 1 and 2 Making a volume visualization animation using 3 Exploring network/multidimensional data using 4 Data sets Presentation and discussion of your results (6/23) Outline Overview of my research and activities Overview of large data visualization Sources of large data Supercomputing Image acquisition Internet Others Technical approaches Data reduction methods 3

4 Visualization and Interface Design Innovation Research Projects Flow visualization Volume data visualization Large data visualization Parallel visualization Biomedical data visualization Graph and network visualization Visual analytics Software visualization Performance visualization Intelligent visualization Remote, collaborative visualization Advanced interfaces and interaction techniques 4

5 Five-year project sponsored by U.S. DOE SciDAC involving investigators from 4 universities and 2 national labs Advance the visualization technology to enable knowledge discovery at extreme scale, foster awareness of and communication about new visualization technologies, and put these technologies into the hands of application scientists Science application driven projects Enable scientists to see the previously unseen and more effectively communicate with others their findings Outreach Activities Workshops, tutorials, panels, etc. Ultrascale Visualization Workshop at the annual Supercomputing Conference meeting to be approved by SC The first IEEE Large Data Analysis and Visualization Symposium, held in conjunction with VisWeek 2011 Technical papers Posters (7/15) Visualization contest (7/29) The first Workshop on In Situ Visualization (to be organized) 5

6 Major Science Applications Turbulent combustion (Sandia) Supernova (ORNL) Groundwater (PNNL) Climate (ANL) Cosmology (Los Alamos) Fusion energy (PPPL) Life sciences Material and chemistry Core-collapse Supernova 6

7 Core-collapse Supernova Combustion 7

8 Fusion Supercomputing 1. Tianhe-1A/China (NVIDIA) 2.57 petaflop 2. Jaquar/US (Cray XT5-HE) 1.75 petaflop 3. Nabulae/China (Intel/NVIDIA) 1.27 petaflop 4. TSUBAME 2.0/Japan (HP/Xeon/NVIDIA) 5. Hopper/US (Gray XE6) 8

9 Sources of Large Data: Simulations Fusion 5TB/run now and 25TB/run in 5 years Astrophysics TB/run now Combustion Over 300TB/run now Climate Over 300TB/run by 2015 (2km) Sources of Large Data: Image Acquisition Astronomy LSST 10PB/yr in 2015 (1GB/sec) SKA 100EB/yr in 2022 (1TB/sec) Earth science data (NASA) Fusion (ITER) Biological science 9

10 Sources of Large Data: Internet Network connectivity and traffic Web content Personal home pages Institution/company home pages Wiki pages Publications Blogs and chat rooms Other social networks applications E-commerce E-science E-education Information Explosion and Human Cognitive Capacity Moore s Law/ amount of data in the world human cognitive capacity 2011 How to organize and present information? 10

11 Large Data Analysis & Visualization Shared resources Supercomputer Storage Visualization Machine Large Data Analysis & Visualization Supercomputer Storage Visualization Machine 11

12 Large Data Analysis & Visualization Data reduction Subsampling, compression, or physically based, knowledge assisted feature extraction and data reduction Parallel data analysis and visualization Post-processing, co-processing, or in situ processing Remote visualization* Transfer data, extracts/geometry, or images Standard Compression Approaches Irreversible (Lossy): General: Transform -> Quantization -> Entropy Coding JPEG 2000: Wavelet Transform -> SQ -> Arithmetic Coding Reversible (Lossless): 1. Predictive Coding: prediction with entropy encoding of difference b/n prediction and actual values 2. Hierarchical Coding: multi-resolution transform (e.g. wavelet) with entropy coding of coefficients 12

13 Reversible Compression Methods Predictive methods Simple and easy to implement Provide very fast encoding and decoding Require only one pass over data to encode Do not support sophisticated data access Hierarchical methods Provide a multi-resolution representation supporting progressive transmission Require computationally expensive encoding and large memory footprint Require multiple passes to encode (convolution) Reversible Compression of Scientific Data floating-point data produced by simulation codes Existing approaches fall into two groups: 1. In-situ compression*: encoding during simulation Fast encoding with simple predictive coding schemes Encoding suboptimal (due to constraints on time/ complexity) 2. Post-processing compression: encoding after simulation Good results with wavelet schemes Encoding quality good (good rates, multi-resolution, lossy option) Encoding too expensive to perform in-situ 13

14 Two Approaches to Data Reduction I: Important-driven approach Time-varying data reduction and visualization II: Application-driven approach Feature-directed reduction and visualization of data from a turbulent combustion simulation Importance Importance Analysis of Time-Varying Volume Data Amount of change, or unusualness in the data Block-wise approach Analyze for a spatial block over time Importance evaluation Amount of information by itself New information over the time series Information theory Amount of information by itself entropy New information over the time series conditional entropy 14

15 Entropy and Mutual Information (marginal) entropy H(X) = conditional entropy H(X Y) = p(x, y)log p(y) p(x, y) x X x X y Y mutual information I(X;Y) = p(x, y)log x X y Y p(x)log p(x) joint entropy H(X,Y) = p(x, y)log p(x, y) x X y Y p(x, y) p(x) p(y) The amount of information X and Y share. Relations with Venn Diagram H(X) H(X) mutual information I(X;Y) H(Y) entropy H(Y) H(X Y) conditional entropy H(Y X) Conditional entropy H(X Y) = H(X) I(X;Y) Joint entroy H(X,Y) =H(X) +H(Y) I(X;Y) 15

16 Importance Value Calculation Y Y Y X Y Y Y Importance Curves I I regular T earthquake data set T 16

17 Importance Curves I climate data set I periodic T T Importance Curves I I turbulent T vortex data set T 17

18 Clustering Importance Curves Cluster Highlighting 18

19 Cluster Highlighting Cluster Highlighting 75% of data blocks 19

20 Abnormality Detection Conditional entropy A: El Niño B: La Niña Time Step Selection uniform- based importance- based 20

21 Importance- Driven Visualization Identify data importance using conditional entropy Cluster importance curves for underlying data classification and prioritization Lead to clearer visualization and better understanding Direct data reduction Wang, C., Yu, H., AND Ma, K.-L. Importance-Driven Time-Varying Data Visualization. IEEE Transactions on Visualization and Computer Graphics, 14(6): , Application-Driven Visualization Make use of scientists domain knowledge Target the intended visualization tasks Locate reference features and compute distance to the a feature of interest Direct data reduction and facilitate interactive visualization Process data in situ 21

22 Application Driven Data Reduction and Visualization One feature of interest is the mixture fraction surface = 0.2, other variables such as HO 2 can be compressed in a distance- based manner Visualization of Turbulent Eddies Small eddies are hidden in the multi-layer flow 22

23 Allowing scientists to see for the first time the interaction of small turbulent eddies with the preheat layer of a turbulent flame, a region that was previously obscured by the multi-scale nature of turbulence. Computed in situ and data reduction opportunity! Compression Combustion Data 23

24 Compression Combustion Data Compression Hurricane Data 24

25 Compression Hurricane Data Compression & Performance 25

26 Application Driven Data Compression Leverage domain knowledge for feature identification Preserve important features with distance-based compression May be used together with conventional compression techniques such as quantization, difference and runlength encoding, etc. Wang, C., Yu, H., AND Ma, K.-L. Application-Driven Compression for Visualizing Large-Scale Time-Varying Volume Data, IEEE Computer Graphics and Applications, 30(1):59-69, Summary When data reduction/loss is inevitable, importance and application driven approaches prove feasible. Parallel visualization is absolutely needed for making sense of data in extreme scale. (Next Lecture) The design of interactive and intelligent visual interfaces is key to exploratory data analysis and visualization. (Lectures 3-4) Domain knowledge + statistical analysis + interactive visualization = promising solutions to understanding large data. (Lecture 5) Visualization by proxy and intelligent visualization are promising concepts. (Lecture 6) At petascale and exascale, in situ visualization is the most plausible solution. (Lecture 7) Visualization should be an integrated part of the overall knowledge discovery process. 26

27 Acknowledgments US DOE ASCR US DOE SciDAC Program US NSF PetaApps Program US NSF OCI Program US NSF CCF Program US NSF FODAVA Program US NSF HECURA Program US AirForce HP Labs Nokia Research AT&T Research 27