Scientific Visualization at JSC

Transcription

1 Mitglied der Helmholtz-Gemeinschaft Scientific Visualization at JSC Jens Henrik Göbbert 1 1 Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH, Germany Cross-Sectional-Team Visualization j.goebbert@fz-juelich.de Folie 1

2 Visualization at JSC Cross-Sectional Team Visualization Domain-specific User Support and Research at JSC Scientific Visualization R&D + support for visualization of scientific data Virtual Reality VR systems for the analysis and presentation Multimedia multimedia productions for websites, presentations or on TV Folie 2

3 InfiniBand Visualization at JSC General Hardware Setup JURECA 12x Visualization Nodes 2 GPUs Nvidia Tesla K40 per node 12 GB RAM on each GPU 2x Vis. Login Nodes jurecavis.fz-juelich.de 10x Vis. Batch Nodes Keep in mind: 8 nodes with 512 GB RAM 2 nodes with 1024 GB RAM Visualization is NOT limited to Visualization nodes (incl. GPUs) only. (software rendering is possible on any node) 10x vis batch nodes 2x vis login nodes 12x login nodes 1872x compute nodes Data GPFS Folie 3

4 Outline Visualization at JSC Cross-Sectional Team Visualization Setup - Software & Hardware Simplify your life: Remote 3d Visualization In Situ Visualization Introduction Coupling Strategies Example Folie 4

5 Mitglied der Helmholtz-Gemeinschaft Simplify your life: Remote 3D Visualization on JURECA with VNC Folie 5

6 Motivation lowering the barriers to visualization Why remote visualization makes sense for the user. Do not spend time installing & configuring visualization software. Do not move your data. Do not be satisfied with your local hardware. makes sense for CS-Team Visualization. Install & configure visualization software once for all users. Any new feature is available to all users. Fix bugs once for all users. More time left for the interesting things. Folie 6

7 Motivation lowering the barriers to visualization desktop symbels for vis apps, LLview, nice blue JSC background MOTD window visualization application CPU, memory utilization clock counting up/down GPU utilization VNC utilization Folie 7

8 InfiniBand Remote 3D Visualization General Setup JURECA User s Workstation 10x vis batch nodes Firewall 2x vis login nodes vis login node: - direct user access - no accounting - shared with other users - no parallel jobs (no srun) vis batch node: - access via batch system - accounting - exclusive usage - parallel jobs possible 12x login nodes 1872x compute nodes Data GPFS Folie 8

9 Remote 3D Visualization with X forwarding + Indirect Rendering Traditional Approach (X forwarding + Indirect Rendering) ssh X <USERID>@<SERVER> uses GLX extension to X Window System X display runs on user workstation OpenGL command are encaplusated inside X11 protocol stream OpenGL commands are executed on user workstation disadvantages User s workstation requires a running X server. User s workstation requires a graphic card capable of the requied OpenGL. User s workstation defines the quality and speed of the visualization. User s workstation requires all data needed to visualize the 3d scene. Try to AVOID for 3D visualization. Folie 9

10 Remote 3D Visualization with VNC (Virtual Network Computing) + GLX forking State-of-the-Art Approach (VNC with VirtualGL) vncserver, vncviewer platform independent application independent session sharing possible advantages No X is required on user s workstation (X display on server, one per session). No OpenGL is required on user s workstation (only images are send). Quality of visualization does not depend on user s workstation. Data size send is independent from data of 3d scene. Try to USE for 3D visualization. Folie 10

11 Remote 3D Visualization with VNC (Virtual Network Computing) + GLX forking GLX forking vglrun (VirtualGL) OpenGL applications send both GLX and X11 commands to the same X display. Once VirtualGL is preloaded into an OpenGL application, it intercepts the GLX function calls from the application and rewrites them. The corresponding GLX commands are then sent to the X display of the 3d X server, which has a 3D hardware Disadvantages: accelerator attached. Multiple users on a login node share the same graphic cards. No resource management for memory/compute of graphic cards. CPU encodes the VNC stream. Currently no hardware accelerated image encoding. Folie 11

12 InfiniBand Remote 3D Visualization Simplify the access JURECA 10x vis batch nodes Firewall 2x vis login nodes manually startup VNC connection 1. ssh to HPC system 2. authenticate via SSH key pair 3. look for an existing VNC server (you may want to reconnect) 4. submit job via slurm 5. wait for job to start start VNC server on node 6. establish SSH tunnel 7. setting up SSH agent forwarding 8. start TurboVNC client and connect 12x login nodes 1872x compute nodes This is far too time consuming. Data GPFS Folie 12

13 InfiniBand Remote 3D Visualization Simplify the access JURECA 10x vis batch nodes Firewall 2x vis login nodes automatically startup VNC ScienTific Remote Desktop Launcher (Strudel) Windows, OS X, Linux written in Python developed by Monash University, Melbourne, Australia 12x login nodes 1872x compute nodes Data GPFS One-Click solution. Folie 13

14 Mitglied der Helmholtz-Gemeinschaft Invitation: Try JURECA Jens Visualization Henrik Göbbert 1 Nodes 1 Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH, Germany Cross-Sectional-Team Visualization j.goebbert@fz-juelich.de Folie 14

15 Invitation Try JURECA Visualization Nodes VNC with VirtualGL (using Strudel) download & install SSH (Putty for Windows) TurboVNC Strudel configure download SSH keys get passphrase for one of the SSH keys j.goebbert@fz-juelich.de load SSH key to your SSH agent Please follow the instructions from run & play start Strudel and connect Folie 15

16 Invitation Try JURECA Visualization Nodes VNC with VirtualGL (using Strudel) Keep in mind: SSH key agent must be running on your system. SSH key for JURECA must be loaded into the SSH key agent. Please send us your Feedback. j.goebbert@fz-juelich.de Folie 16

17 Try JURECA Visualization Nodes OSPRay ParaView CPU ray tracing framework for scientific vis. rendering efficient rendering on CPUs ray tracing / high fidelity rendering made for scientific visualization Built on top of Embree (Intel ray tracing kernels) Intel SIMD Program Compiler OSPRay Integrated into ParaView, VMD, VisIt, VL3, EasternGraphics,... FIU Ground Water Simulation Texas Advanced Computing Center (TACC) and Florida International University Folie 17

18 Try JURECA Visualization Nodes OSPRay with ParaView Ray tracing within ParaView ParaView (standard renderer) Build option in ParaView 5.2 by default Why ray tracing? gives more realistic results adds depth to your image can be faster on large data ParaView (OSPRay) Requirement: CPUs: Anything SSE4 and newer (in part, including Intel Xeon Phi Knights Landing) Cooperation with Electrochemical Process Engineering (IEK-3) Jülich Forschungszentrum GmbH, Germany Folie 18

19 Try JURECA Visualization Nodes Parallel ParaView Folie 19

20 Mitglied der Helmholtz-Gemeinschaft In Situ Visualization Jens Henrik Göbbert 1 1 Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH, Germany Cross-Sectional-Team Visualization j.goebbert@fz-juelich.de Folie 20

21 Why do we need In Situ Visualization? Visualization the old way FZ Jülich Jülich Storage Cluster FZ AIA, RWTH Aachen University wikipedia.org, Chevron FZ Jülich Data FZ Jülich Why changing a running System.? Folie 21

22 Why do we need In Situ Visualization? Bottlenecks for Visualization X Example: AIA, RWTH Aachen University Large Eddy Simulation Analysis of Chevron Geometries 123 TB (= 30 days to download with 50 AIA, RWTH Aachen University Moving data between computing centers becomes a major bottleneck. Folie 22

23 Why do we need In Situ Visualization? Bottlenecks for Visualization X X Moving data to the storage devices will become a major bottleneck. Folie 23

24 What is In Situ Visualization? Basics Visualizing results of a running simulations without the need to copy the data to a storage device. Data compression Reduce I/O traffic and volume. Access to more data. Post-processing Dump Times: In situ processing Enable real-time simulation exploration. Reduce latency to first result. Avoid large-scale post-processing..... Folie 24

25 What is In Situ Visualization? Basics Visualizing results of a running simulations without the need to copy the data to a storage device. But. Simulation- and Visualization code might have different internal data structures. might scale best with different parallelization strategies. Simulation- and Visualization developer Coupling simulation- and visualization codes is a challenge. might have different priorities. might not work in the same team. Folie 25

26 What is In Situ Visualization? Staged vs. On-Node Staged In Situ Visualization (or in transit visualization) On-Node In Situ Visualization Folie 26

27 What is In Situ Visualization? Staged vs. On-Node Staged In-Situ Visualization On-Node In-Situ Visualization Advantages: No need to share hardware resources (cpu / memory) between simulation and visualization. Disadvantages: Data needs to be copied between nodes or even clusters. No zero-copy possible. More difficult to implement. Advantages: No need to copy data between nodes or clusters. Zero-copy possible. Simpler to implement. Disadvantages: Hardware resources (cpu / memory) need to be shared between simulation and visualization. Folie 27

28 What is In Situ Visualization? Loose vs. Tight Loose Coupling Tight Coupling Advantages: Less dependencies at compile/link time Two separate applications. Disadvantages: Data transfer between Simulation and Visualization more difficult to implement. Zero-copy difficult. Advantages: Simpler to implement. Single executable. Zero-copy possible. Disadvantages: Bug in visualization code might affect simulation. On-Node in situ visualization by design. Folie 28

29 Lowering the barriers to in situ visualization What are the major barriers? Major barriers to in situ visualization are first, the individual implementation-, optimization- and coupling-costs to integrate the needed functionality to each simulation code and setup can often not be justified. second, the usage of in situ visualization requires much training for scientists who's research work in general does not focus on visualization in the first place. Folie 29

30 Direct Approch coupling simulation code to in-situ visualization simulation VisIt/ Libsim ParaView/ Catalyst sim. developer vis. developer ParaView/Catalyst VisIt/Libsim Folie 30

31 Adapter Approach coupling simulation code to in situ visualization simulation VisIt/ Libsim ParaView/ Catalyst sim. developer vis. developer Damaris - SENSEI - Strawman - GLEAN - JUSITU - Folie 31

32 Adapter Approach coupling simulation code to in situ visualization simulation VisIt/ Libsim ParaView/ Catalyst data adapter bridge analyze adapter Folie 32

33 Adapter Approach coupling simulation code to in situ visualization data adapter bridge analyze adapter Folie 33

34 Mitglied der Helmholtz-Gemeinschaft In Situ Visualization Jens Example Henrik Göbbert 1 1 Jülich Supercomputing Centre, Forschungszentrum Jülich GmbH, Germany Cross-Sectional-Team Visualization j.goebbert@fz-juelich.de Folie 34

35 JUSITU Applications lately coupled with VisIt/Libsim psopen flow solver (DNS) highly resolved turbulence pseudo-spectral approach Institute for Combustion Technology RWTH Aachen University, Germany Chair of Num. Thermo-Fluid Dyn. TU Freiberg, Germany Fortran90 + MPI + OpenMP JUQUEEN BigWeek participant CIAO flow solver (LES, DNS) multiphysics, multiscale structured finite difference method level-set for surface tracking level-set/volume-of-fluid interface Lagrange particle solver tabulated/finite rate chemistry overset mesh refinement moving meshes Institute for Combustion Technology RWTH Aachen University, Germany Fortran90 + MPI JUQUEEN BigWeek participant ZFS flow solver (LES, DNS) multiphysics, multiscale finite volume method lattice-boltzmann method discontinuous Galerkin method level-set for surface tracking Lagrangian particle solver Institute of Aerodynamics Aachen RWTH Aachen University, Germany C MPI + OpenMP + GPU Folie 35

36 Why In Situ Visualization An example. spray injection & droplet formation Folie 36

37 Ratio of soot trans. intensity Why In Situ Visualization An example. spray injection & droplet formation Geometry of injection systems increasingly complex Emission and pollutant formation strongly depend on injection systems Experimental investigation of geometry impact very challenging smaller k-factor larger k-factor High-fidelity spray simulations required! Baumgarten, C., Mixture formation in internal combustion engines, Springer, Aye, M.M. et al., Studying the influence of k-factor of different nozzles on spray and combustion under diesel engine-like conditions in a high pressure spray chamber (submitted). Folie 37

38 Why In Situ Visualization An example. spray injection & droplet formation Experimental investigation of injection systems very challenging due to small length (~100 μm) and high velocity scales (~600 m/s). Highly resolved simulations result in large amount of data which can be reduced with in-situ visualization. Predictive simulations using in-situ visualization and studying injection systems and resulting cavitation give more insight. Le Chenadec, V. and Pitsch, H., A 3D Unsplit Forward/Backward Volume-of-Fluid Approach and Coupling to the Level Set Method, Journal of Computational Physics 233:10-33, 2013 Le Chenadec, V. and Pitsch, H., A monotonicity preserving conservative sharp interface flow solver for high density ratio two-phase flows, Journal of Computational Physics 249: , 2013 Folie 38

39 Why In Situ Visualization An example. spray injection & droplet formation Dump data with high temporal and spatial resolution needed. Folie 39

40 Why In Situ Visualization An example. spray injection & droplet formation Application CIAO on JUQUEEN (specific setup) excellent scaling of solver file output is bottleneck at scale dump data with high temporal resolution not possible => In situ visualization helps! Folie 40

41 Summary & Conclusion Try the visualization nodes at JSC Send me your feedback. In Situ Visualization Size of simulation data is rising constantly and I/O becomes a major bottleneck. A huge number of simulation codes and scientists will benefit from in situ visualization (in situ processing). But, decisions have to be made how to coupling simulation and analysis tools in future. Folie 41

42 Questions? rendered with Blender from a DNS of a diesel injection spray of ITV, RWTH Aachen University Folie 42