An In-Situ Visualization Approach for the K computer using Mesa 3D and KVS

Transcription

1 An In-Situ Visualization Approach for the K computer using Mesa 3D and KVS Kengo Hayashi 1,2, Naohisa Sakamoto 1,2 Jorji Nonaka 2, Motohiko Mastuda 2, Fumiyoshi Shoji 1 Kobe Univesity, 2 RIKEN Center for Computational Science ISC WORKSHOP ON IN SITU VISUALIZATION 2018 ( WOIV 2018 )

2 Background K Computer A Japanese leading-edge supercomputer built in 2011 SPARC64fx CPU architecture The use of a two-staged parallel file system ( GFS and LFS ) # Racks 864 # Nodes 82,944 Node (CPU) SPARC64fx Login Node SPARC64fx CPU Network Peak Perf. TOFU PF Stage-In File System Storage FEFS 30 PB GFS Stage-Out LFS 2

3 In-situ Visualization Mesa 3D Graphics Library Graphics driver Swrast Softpipe LLVMpipe OpenSWR Legacy software rasterizer Gallium software driver Gallium LLVM driver highly optimized for the Intel x86 CPUs and Accelerators HPC using Mesa driver ANL Mira (IBM Blue Gene/Q) swrast driver ORNL Titan (Cray XK7) softpipe driver 3

4 Existing visualization framework ParaView and VisIt OSS for general purpose high performance visualization Based on the Visualization Tool Kit ( VTK ) Utilizes the OpenGL for the graphics For In-situ Visualization ParaView Catalyst VisIt-Libsim Mesa3D graphics driver 4

5 Visualization Framework for K Computer Fujitsu Visualization Library Official Visualization Tool Particle-based volume rendering APIs for C/C++ and Fortran codes [A.Ogasa et al., 2012] SURFACE [M.Fujita et al., 2014] Scalable and Ubiquitous Rendering Framework for Advanced Computing Environment Highly scalable ray tracer LSGL graphics library optimized with SIMD vectorization functionalities for K computer Visualization result of seismic wave Earthquake Research Institute, Univ. of Tokyo 5

6 Large-scale unstructured volume rendering Particle-based volume rendering (PBVR) Volume data is represented as particles Visibility ordering is not required Interactive rendering with GPUs Supported cell types: tet, tet2, hex, hex2, pyramid, prism [N.Sakamoto et al, 2007] log(1 α ) ρ = πr 2 Δt Particle density Opacity (Num. of particles within a unit volume) 26M tet2 cells 71M hex cells 18M prism cells x 20 time steps 282k tet cells on 40LCDs 6

7 Objective Mesa 3D (OSMesa) on K computer K computer is only capable of compiling the initial version of Mesa 3D library with legacy graphics driver (swrast) which implements only the fixed graphics pipeline OpenGL-based KVS library KVS is a multiple platform OpenGL-based general purpose visualization library developed at Kyoto University and Kobe University Parallel particle-based volume rendering for In-situ visualization of large-scale unstructured volume on K computer 7

8 Overview of our approach In-situ visualization with particle-based volume rendering on K computer Implementation of Mesa3D with llvmpipe for SPARC64fx Parallelization of PBVR based on KVS Simulation Run KVS rendering method Pointer to the data Render Render... Render Gather and Merge Image File Render SPARC64fx CPU Memory KVS based rendering code libosmesa swrast softpipe llvmpipe Mesa Compiler LLVM JIT Compiler OpenGL API Fixed-function graphics pipeline Programmable graphics pipeline GLSL Shader Codes Vertex Shader Code Geometry Shader Code fragment Shader Code Parallelized PBVR Mesa3D on K computer 8

9 In-Situ Visualization for the K computer Fujitsu Library We tried Visualization Application Visualization Library PBVR Non-OpenGL Visualization Application (SURFACE etc) Visualization Application Graphics Library (OpenGL) 9

10 Mesa 3D on SPARC64fx Swrast ( Legacy software rasterizer ) - Fixed graphics pipeline - Fujitsu compiler( official compiler and support GCC 4.4.7) Softpipe ( Gallium software driver ) - Programmable graphics pipeline - GCC 6 ( C++ 11 ) Llvmpipe ( Gallium LLVM driver ) - Programmable graphics pipeline - GCC 6 ( C++ 11 ) - LLVM JIT No support for SPARC64 Performance 10

11 Mesa-llvmpipe LLVM-JIT Relocation processing lib/executionengine/runtimedyld/runtimedyldelf.cpp No support for SPARC architecture switch (Arch) { case Triple::ARCH break; default: } x86; x86_64 arm; armeb; thumb; thumbeb ppc; ppc64; ppc64le systemz bpfe1; bpfeb llvm_unreachable("unsupported CPU type!"); case Triple::sparcv9: resolvesparcv9relocation(section, Offset, Value, Type, Addend, SymOffset); break; We use Mesa , LLVM and GCC

12 KVS Kyoto Visualization System [N.Sakamoto et al, 2015] Multi-platform OpenGL-based general purpose visualization library Implemented traditional rendering method ( isosurface, slice, etc.) Providing parallel offscreen rendering APIs for both C/C++ and Fortran based simulation codes Particle-based volume rendering rendering (PBVR ) 12

13 Particle-based Volume Rendering (PBVR ) 1. Particle generation - Estimate the density of particles 2. Particle projection - Rendering for each particle set 3. Ensemble averaging generation projection averaging 13

14 Parallel PBVR Overview of parallel processing Repetition Sub-Volume 1 Generation Sub-Particle 1 Projection Sub-Image 1 Image Sub-Volume 2 Sub-Volume n Generation Generation Sub-Particle 2 Sub-Particle n Projection Projection Sub-Image 2 Sub-Image n Image Composition Averaging Final Image 14

15 el PBVR using 234Compositor Image Composition Technique or is 234Compositor a flexible parallel - A flexible parallel image siting library for massively compositing - Extended the Binary-Swap zation environments. algorithm e Binary-Swap algorithm, - Enabled the handling of arbitrary number of nodes the handling of arbitrary des. et al 2017] [J.Nonaka et[j.nonaka al,2017] 15

16 Experiment Data Magnus force acting on a rotating sphere placed in a uniform flow Specifications Number of nodes: 15,321,546 Number of cells: 18,899,767 Cell shape: prism cell Evaluation Rendering performance of PBVR Particle generation Image composition Particle projection [M.Muto et al., 2012] 16

17 Result Processing times with different number of MPI processes - Decrease Particle Generation - Increase Image Composition - Changeless Particle Projection 17

18 Result Rendering times with different number of ensemble averaging 18

19 Conclusion In-situ visualization framework which utilizes the Mesa 3D Enable the OpenGL-based visualization application using KVS to run on K computer with SPARC64fx CPUs Future work Integrate with real simulation codes and evaluate the stability, scalability, and performance 19