Automated performance modeling of the UG4 simulation framework

Transcription

1 Automated performance modeling UG4 simulation framework Andreas Vogel 1, Alexru Calotoiu 2, Arne Nägel 1, Sebastian Reiter 1, Alexre Strube 3, Gabriel Wittum 1, Felix Wolf 2 1 Goe-Universität Frankfurt, Frankfurt am Main, Germany 2 Technische Universität Darmstadt, Darmstadt, Germany 3 Forschungszentrum Jülich, Jülich, Germany Munich, January 2016 Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework

2 Introduction Goal: Find scaling issues in large parallel codes. Target code: UG4 (Unstructured Grids 4), a simulation framework for solving partial differential equations on unstructured grids. Idea: Create performance models with fine granularity Usage: Measure scaling behavior different code kernels. Create scaling models for each such kernel. Do this for thouss kernels automation required! Automated performance analysis to identify current bottlenecks. Prediction possible bottlenecks for even larger parallel runs. Allows for prediction resource consumption for larger core counts. Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework

3 Outline UG4 Overview parallelization concepts Application Drug diffusion through human skin Performance Modeling Ideas Results Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework

4 UG4 overview UG4 1 is a simulation framework for solution PDE s. Fast Cross platform C++ code. Modular plugin based structure. Full scripting support through Lua Java. Flexible distributed unstructured grids. FE-/FV-discretizations. Various applications such as groundwater-flow, elasticity, neuroscience, biology, many more. 100k lines code. Developed with massively parallel applications in mind: Multigrid solvers for optimal complexity (important!). Efficient distribution grid hierarchies. Very good scalability shown for up to cores 2. 1) Vogel, A., Reiter, S., Rupp, M., Nägel, A., Wittum, G.: UG 4: A novel flexible stware system for simulating PDE based models on high performance computers. Comp. Vis. Sci. 16(4), (2013) 2) Reiter, S., Vogel, A., Heppner, I., Rupp, M., Wittum, G.: A massively parallel geometric solver on hierarchically distributed grids. Com. Vis. Sci 16(4), (2013) Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework

5 UG4 parallelization concepts Interfaces allow for data exchange beten distributed grid objects. I MH l,a,b I SH l,b,a B A I MH l,b,c I SH l,c,b master interface, I MH l,a,c I S H l,c,a slave interface. C Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework

6 UG4 parallelization concepts Interfaces allow for data exchange beten distributed grid objects. Hierarchical distribution guarantees good comp/comm ratio. l P 0 P 1 P 2 P 3 P 0 P 1 P 2 P 3 P 0 P 2 P 0 P 0 P 1 P 2 P 3 ghosts v-interfaces Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework

7 UG4 parallelization concepts Interfaces allow for data exchange beten distributed grid objects. Hierarchical distribution guarantees good comp/comm ratio. Horizontal/vertical interfaces are used for communication. &'(#)# &'(#%#!"#!$#!%# &'(#$# &'(#"# *+,-.+/012#-/0',314'# (',5412#-/0',314'# Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework

8 Application: Drug diffusion 1. Application: through human Human skin skin Stratum corneum Model human skin 1 : Skin - Geometry 3d Stratum granulosum Stratum spinosum Stratum basale Grid for Stratum corneum: Modeling drug diffusion through human skin Application medicine leads to diffusion substances into skin Questions: Pathway substances, temporal behavior,... Diffusion equation: Andreas Vogel, G-CSC, University Frankfurt, Germany t c s (t, x) = (D s c s (t, x)), green: Lipid - channels red: Skin cells s {cor, lip} FV-Simulation drug diffusion Andreas through Vogel, G-CSC, University Frankfurt, Stratum Germany corneum Hex-Grid with Corneocytes (red) Lipid channels (green). Difficulties: jumping coefficients anisotropic elements. 1) Nägel, A., Heisig, M., Wittum, G.: A comparison two- three-dimensional models for simulation permeability human stratum corneum. European Journal Pharmaceutics Biopharmaceutics 72(2), (2009) Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework

9 Skin-Problem: Solution process refined: good element quality coarse: bad element quality Generation MG-hierarchy through parallel anisotropic refinement. Andreas Vogel, G-CSC, University Frankfurt, Germany Element quality improves with each refinement less solver steps. Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework

10 enables researcher totests, deduct desired information. data can be parsed studies: In first two focus on modeling drugthis diffusion through each automatic pre-benchmark analyze post-processing scripts thatunder draw information store human skin. First, code behavior ak scaling, For run, data written out a certain format that Skin-Problem: Solution Using tools from Sec. 3 interpretation. Sec. 4, is process analyze in UG4 code in three n subit more for manual vary densely diffusivity skin cells over several ranges magnitude. In third enables researcher totests, deduct desired information. This data can be parsed studies: In firstcarried two focus on shown modeling drug diffusion through steps outdifferent JUBE are in Fig. 2. information Preparation, compilation, study, compare two types solvers, again under ak scaling: automatic pre post-processing scripts that draw human skin. First, analyze code behavior under ak scaling, nstore execution, analysis steps might exist multiple times JUBE will perform geometric solver unpreconditioned conjugate gradient (CG) it more densely for manual interpretation. vary diffusivity skin cells over several ranges magnitude. In third aforementioned steps in sequence. It is important to note that JUBE is method. steps carried out JUBE are shown in Fig. 2. Preparation, compilation, study,able compare different types runs solvers, again under ak scaling: to easily two create combinatorial multiple parameters. For example, execution, analysis steps might exist multiple times JUBE will perform geometric conjugate gradient (CG) in a scaling experiment, one can numbers processes, 8 solver Andreas Vogel et unpreconditioned al. simply specify multiple aforementioned steps in sequence. It is skin. important toelement notequality that isepidrug though /or human good outermost partjube will create refined: method. /ordiffusion different solver setups physical parameter, JUBE 30 able to create for combinatorial runs multiple parameters. For example, dermis (stratum corneum) consists flattened, dead cells (corneocytes), that oneeasily experiment each possible combination, submit tongmg resource p all L m DoF are surrounded inter-cellular lipid. stratumnumbers corneumisprocesses, natural in a scaling experiment, one can simply specify multiple 25 manager, collect allan results, display m toger , Drug diffusion though human skin. outermost part epibarrier to protect underlying tissue, but stillparameter, allows for certain /or different solver JUBE create throughput 2,271,049will 27 20setups /or physical Solve dermis (stratum corneum) consists flattened, dead cells (corneocytes), that ,961, coarse: bad element quality concentrations (e.g.,possible drugs, medicine). latter process can be modeled a one experiment for each combination, submit all m to resource Init ,869,025!!"" are surrounded inter-cellular lipid.m stratum corneum is 29natural Assemble diffusion process, in which diffusion coe cient within corneocytes differs manager, allan results, display toger ,139,670, collect Results 10 barrier to protect tissue, but still allows for throughput certain from oneunderlying in lipid. Different geometric representations stratum #$!!"" concentrations (e.g.,been drugs, medicine). process can beformodeled 5 used Kernel Model time corneum have to compute latter diffusional throughput [17].[s] a Using tools from%#sec. 3diffusion Sec.coe cient 4 analyzed code in log three studies: Solve UG p differs diffusion process, in which within 8.17 corneocytes!" " 0 In following studies, use a 2drug brick--mortar model 3). 5 Results Init log22 p (Fig In two tests2two focus on diffusion through human modeling 213 Green: corneocytes (dead, flattenedgeometric skin in cells) Assemble 1.78 s from Assuming onefirst indiffusion lipid. Different representations stratum driven transport two subdomains {cor, lip} (corprocesses skin. First, analyze code behavior in a ak scaling follod study Red: lipid channels Generation MG-hierarchy through parallel anisotropic refinement. corneum have lipid), been Fig. used to compute diffusional throughput [17]. computational grid isover given 4. governing Left: Measured wallclock times (marks) models (lines) in for3d assembly, Usingneocyte, tools from Sec. 3 Sec.equation 4 cells analyzed UG4 three varying diffusivity skin ranges code magnitude. Instudies: third Diffusion equation: two solver initialization, solution skin 3d problem. Right, top: 3). In following studies, use a brick--mortar model (Fig. In study first twoprovide tests focus on modeling drug diffusion through human a comparison for different types solver: geometric Number gridimproves refinements (L), two degrees each freedom (DoF) number solver steps. Element quality with refinement iterations less cs (t, x) = two (D cmodels Assuming driven transport in sfollod lip} (cors (t, x)). solver (n ).t Right, Performance for gmg is compared inbottom: a ak scaling study tokernels. {cor, unpreconditioned skin. First,diffusion solver analyze code behavior in assubdomains ak scaling study neocyte, lipid), governing equation is given Solution conjugate gradient varying diffusivity (CG) method. skin cells over ranges magnitude. In third diffusion coe cient D is assumed to be constant within each subdomain study provide a comparisons for two different types solver: geometric s {cor, lip}, but may (t, x) beten = (Dsubdomains. t csdiffer s cs (t, x)). For scalability analysis, solver is compared in aak scaling study to unpreconditioned steady though state 5.1compute Drug diffusion concentration human skindistribution. conjugateasgradient (CG) method. solver, employ a geometric method, accelerated an outer diffusion coe cient Ds is assumed to be constant within each subdomain We lip}, consider amay model forbeten permeability human skin. outermost conjugate gradient method. uses afor damped Jacobi smoor, two s {cor, but differ subdomains. scalability analysis, part epidermis (stratum corneum) consists flattened, dead (cor(resp. three) smoothing steps in 2d (resp. 3d), a V-cycle, an LU base solver. compute steady state concentration distribution. 5.1 Drug diffusionfig. though (left) human skin(right) solution for a 2d geometry. cells 5. Instationary stationary 10 Concentration once a an substance at residuum two different timesteps. neocytes) are surrounded inter-cellular lipid. stratum corneum iterations completed an absolute reduction 10 isis As solver, that employ a geometric accelerated outer 1) Nägel, A., Heisig, M., Wittum, G.: statemethod, art in computational modelling skinan permeation. natural barrier to protect underlying but still allows foroutermost achieved. main di culty this uses problem is bad aspect ratiothrough Advanced Drug Delivery (2012) We consider a model permeability tissue human skin. conjugate gradient method. a damped smoor, icallyfor interesting fluxes at Systems bottom domain, FJacobi := c ds, two bot put epidermis certain concentrations drugs, can be framework computational domain vs. 30µm for channels). This is resolved Sebastian Reiter Performance modeling process UG4 simulation iteration count for(e.g., Frankfurt solver are collected using JUBE (Tab. 2). part (stratum corneum) consists lipid flattened, dead cells (cor(resp. three) smoothing steps(0.1µm in G-CSC 2d (resp. 3d), amedicine). V-cycle, anlatter LU base solver. Time [s] Geometry: brick--mortar Andreas Vogel, G-CSC, University Frankfurt, Germany bot

11 Performance Modeling for UG4 - Setting Setting: UG4 s code contains PROFILE calls at many crucial points. Different prile-backends exist. Here ScoreP 1 is used. Weak scaling study for steady state drug diffusion problem. Solver: Geometric Multigrid, Jacobi-smoor, outer CG. 1) 2) A. Vogel, A. Calotoiu, A. Strube, S. Reiter, A. Nägel, F. Wolf, G. Wittum. 10,000 performance models per minute scalability UG4 simulation framework. In J. L. Träff, S. Hunold, F. Versaci, editors, Euro-Par 2015: Parallel Processing, vol oretical Computer Science General Issues, pages Springer International Publishing, 2015 Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework

12 Performance Modeling for UG4 Performance Modeling: Run simulations at different process numbers, record timings. Generate performance-model for each code-kernel ( ) each metric (5-10) finding best fit in PMNF : f (p) = n k=1 c k p ik log j k 2 (p) Sort analyze models asymptotic behavior. Complexity O(log p) is considered fine. (*): Performance Model Normal Form 1) Calotoiu, A., Hoefler, T., Poke, M., Wolf, F.: Using automated performance modeling to find scalability bugs in complex codes. In: Proc. ACM/IEEE Conference on Supercomputing (SC13), Denver, CO, USA. ACM (November 2013) 2) Calotoiu, A., Hoefler, T., Wolf, F.: Mass-producing insightful performance models. In: Workshop on Modeling & Simulation Systems Applications, University Washington. Seattle, Washington (Aug 2014) 3) Picard, R.R., Cook, R.D.: Cross-validation regression models. Journal American Statistical Association 79(387), (1984) Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework

13 sparse matrix assembling, (bottom). 1 R2, absolute difference Results bottlenecks beteni R Identifying Models: optimum scaledperformance Grid 10, which setup can be considered a normalized 2 3 error, confirms good quality all models Time Kernel Model time = f (p) [ms] Bytes sent 1 R2 Model LoadUGScript æ MPI Allreduce log p init levels æ MPI Allreduce log p2 init top surface æ MPI Allreduce p1/4 Issue: Kernel 1 R2 [10 3 ] tes = f (p) [10 3 ] O(MPI Allreduce) p O(MPI Allreduce) p O(MPI Allreduce) Time Invocations Broadcast membership info using MPI_Allreduce for array length p 2 2 Model 1 R Model Found issue: MPI_Allreduce used for array length 1p. R 3 3 Now: MPI_Allreduce replaced MPI_Comm_split time = f (p) [10 ] invocations = f (p) [10 ] Where: InitSolver: Processes have to signal wher y 76.9 want to take part in communication on certain levels. GMG æ PreSmooth æ jacobi log p log p GMG æ prolongate log p log p2 O(log p) 1) for MPI_Comm_split are known to scale Enhanced assemble linear algorithms with 1) Weak-scaling analysis 3d skin model. Using 3d skin model 1) Siebert, C., Wolf, F.: Parallel sorting with minimal data. In: Recent Advances in Message Passing Interface, 2 pp Springer, Now: above, Using MPI_Comm_split which p).1 described fix 2011 diffusion parameter to scales Dcor = with O(log Tab. 1 shows models for a scalability issue detected. In se kernels, create MPI communicator groups for each level hierarchy, excluding processes Andreas Vogel, G-CSC, University Frankfurt, Germany Fast do issue possible from location group that not own a grid thanks part on to fine level.grained In order to inform performance models. every process on se memberships, employ an MPI Allreduce for an array length p, resulting in a p O(MPI Allreduce) dependency, that will lead to scalability issues large counts. In se kernels, substituted Siebert, C., Wolf, F.: Parallelfor sorting withprocess minimal data. In: Recent Advances in Message Passing Interface, pp Springer,for 2011 MPI Comm split MPI Allreduce, also eliminating linearly growing input. First tests do not show a significant improvement in runtime, hover now Sebastian Reiter G-CSC Frankfurt Performance modeling have UG4 simulation framework dependency is O(MPI Comm split), whose scaling properties been analyzed

14 Models: Multigrid Results8 II Andreas Validating optimal complexity GMG Vogel et al. 30 p L Time [s] Solve Init Assemble Processes DoF ngmg 6 290, ,271, ,961, ,869, ,139,670,081 Kernel Solve Init Assemble Model for time [s] log2 p log22 p 1.78 Fig. 4. Left: Measured wallclock timescount (marks) models(lines) 1) assembly, properties: iteration independent mesh for size Known solver initialization, solution skin 3d problem. Right, top: ak scaling properties ifdegrees one iteration step(dof) scales Weak scaling study for 3d skin Good Number grid refinements (L), problem: freedom number iterations solver (n Right, bottom: Performance models kernels. gmg ). When number processes (1985) aforfactor 8, refine 1) Hackbusch, W.: Multi-grid methods applications,grows vol. 4. Springer once more. Number elements per process is same in all runs (ak scaling). Known: Number MGVogel, iterations independent mesh size. Andreas G-CSC, University Frankfurt, Germany crucial for good ak scalability! Observed: Slight increase in iteration numbers (anisotropy in lipid layers). No GMG code kernel scales worse than O(log p)! Very good overall behavior. Fig. 5. Instationary (left) scaling stationary (right) solution for a 2d geometry. Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework

15 Results III Models: Comparison GMG scalability CG scalability Conjugate 10,000 performance models per minute gradient UG Time [s] p L CG GMG Assemble Processes DoF 7 66, , ,050, ,198, ,785,409 Kernel CG GMG Assemble ncg ngmg Model (time [s]) p log22 p Fig. 6. Left: Measured times (marks) models (lines) for assembling solver increases for (CG) laplace problem roughly a factor Right, 2 1) top: execution for count conjugate gradient (GMG) methods. CG: iteration Number grid refinements (L), degrees freedom (DoF) solver iteraweak scaling study for 2d Laplace scaling: increase process count problem: factor 4 withnumber each grid refinement 2d ak tions (ncg, ngmg ). Right, bottom: Performance models for kernels. When number processes grows a factor 4, refine once more. 1) Hackbusch, W.: Iterative solution large sparse systems equations (1994) 5.1 Analysis algebraic solvers Observed: geometric solver justuniversity one Frankfurt, many Germany possible choices for solvers Andreas Vogel, is G-CSC, GMG iteration numbers stay constant. for a sparse matrix problem. refore, perform a ak scaling comparison CG iteration numbers increase a factor 2 (expected from ory) beten method used as a linear solver unpreconditioned conjugate gradient method applied 2 to same problem. GMG: O(log2 p) CG: O( p) Weak scaling comparison conjugate gradient. To allow a oretical analysis, choose a ll known test problem: For model Sebastian Reiter G-CSC Frankfurt2 Performance modeling UG4 simulation framework

16 Conclusions Outlook Automated performance modeling: Helps to find scalability issues in existing codes. Fine granularity helps to easily identify responsible code kernels. Suited for large code-bases massive parallelism. Plus: Automated unit tests, resource consumption, etc. Results for UG4 : UG4 features solvers with nearly optimal ak scalability. Efficient massively parallel runs on flexible unstructured grids. Next steps: Detailed analysis massively parallel adaptive runs. Special focus on domain partitioning, redistribution,... Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework

17 End Thank You Financial support from DFG Priority Program 1648 Stware for Exascale Computing (SPPEXA) is gratefully acknowledged. Gauss Centre for Supercomputing (GCS) is gratefully acknowledged for providing computing time on GCS share supercomputer JUQUEEN at Jülich Supercomputing Centre (JSC). Sebastian Reiter G-CSC Frankfurt Performance modeling UG4 simulation framework