Multicore-aware parallelization strategies for efficient temporal blocking (BMBF project: SKALB)

Transcription

1 Multicore-aware parallelization strategies for efficient temporal blocking (BMBF project: SKALB) G. Wellein, G. Hager, M. Wittmann, J. Habich, J. Treibig Department für Informatik H Services, Regionales Rechenzentrum Erlangen Friedrich-Alexander-Universität Erlangen-Nürnberg

2 SKALB project survey artners: TU Braunschweig TU Dortmund / IANUS HLRS Stuttgart Erlangen: LSS / RRZE (projectlead) Ass. partners: Intel, RAY, BASF, Sulzer, hhpberlin, H Goal: Efficient implementation of Lattice-Boltzmann FD solvers for complex multi-physics applications on petascale supercomputers Main tasks of RRZE: Multicore-specific implementation & optimization of kernels & full applications. Static domain decomposition Benchmarking and showcases (BMBF all H-Software für skalierbare arallelrechner ) Funding: 1,8 M 22. Juni 2010 SKALB Multicore-aware parallelization 2

3 Multicore-aware parallelization approach: Efficient, parallel, multicore-aware temporal blocking roof of concept: Simple iterative method w. regular stencil updates (Jacobi) Elaborate spatial blocking techniques have been implemented (not shown) Socket-/node level implementation + multi-halo data exchange for multi-node computations 1. M. Wittmann, G. Hager, J. Treibig and G. Wellein: Leveraging shared caches for parallel temporal blocking of stencil codes on multicore processors and clusters. Submitted to arallel rocessing Letters. arxiv: J. Treibig, G. Wellein and G. Hager: Efficient multicore-aware parallelization strategies for iterative stencil computations. Submitted to Journal of omputational Science. arxiv: M. Wittmann, G. Hager and G. Wellein: Multicore-aware parallel temporal blocking of stencil codes for shared and distributed memory. Accepted for roceedings of LS10, the Workshop on Large- Scale arallel rocessing at IDS 2010, April 23rd, 2010, Atlanta, GA. arxiv: G. Wellein, G. Hager, T. Zeiser, M. Wittmann, H. Fehske: Efficient temporal blocking for stencil computations by multicore-aware wavefront parallelization. roceedings of rd Annual IEEE International omputer Software and Applications onference (OMSA2009), IEEE omputer Society, p DOI /OMSA BEST AER AWARD 22. Juni 2010 SKALB Multicore-aware parallelization 3

4 The multicore (r)evolution A standard compute node 2-socket shared-memory node of cache-coherent Non-Uniform Memory Access (ccnuma) type Shared aches: MI L3 AHE Memory L3 AHE MI Memory Data exchange Thread synchronization Data oherency! L3 cache traffic? Scalable? MI parallel? AMD Opteron Istanbul 6 cores@2.8 GHz L1: 64 KB / L2: 512 KB / L3: 6 MB 2 X 2 X DDR GB/s HT GB/s/dir Intel Xeon Westmere 6 cores@2.93 GHz L1: 32 KB; L2: 256 KB; L3: 12MB 2 X 3 X DDR GB/s 2 X QI GB/s/dir 22. Juni 2010 SKALB Multicore-aware parallelization 4

5 Multicore-aware parallelization Jacobi solver a simple prototype do k = 1, N do j = 1, N do i = 1, N y(i,j,k) = b* (x(i-1,j,k)+x(i+1,j,k)+ x(i,j-1,k)+x(i,j+1,k)+ x(i,j,k-1)+x(i,j,k+1)) enddo enddo enddo A simple but representative test erformance Metric: Million Lattice Site Updates per second (MLUs) Equivalent MFLOs: 6 FLO/LU * MLUs Bandwidth requirements: 16 Byte / Lattice Site Update (LU) if write allocate on y is surpressed (24 Byte/LU otherwise) erformance estimate for best baseline: B M / (16 Byte/LU) (B M : attainable memory bandwidth as measured with stream) 22. Juni 2010 SKALB Multicore-aware parallelization 5

6 Multicore-aware parallelization Towards multicore-awareness: Naive / calssical approach core0 core1 ache Memory x j-direction k-direction do t=1,t Max!$OM ARALLEL DO private( ) do k=1,n do j=1,n do i=1,n y(i,j,k) = enddo enddo enddo y x enddo 22. Juni 2010 SKALB Multicore-aware parallelization 6

7 Multicore-aware parallelization Reuse of in-cache data between threads: wavefronts Reduce main memory accesses by a factor 2! y(:,:,:) is obsolete! core0 core1 Save main memory data transfers for y(:,:,:)! y-direction z-direction tmp(:,:,3) Memory x(:,:,:) Use ring buffer tmp(:,:, 0:3) which fits into the cache Sync threads/cores after each k-iteration core0: x(:,:,k-1:k+1) t tmp(:,:,mod(k,4)) core1: tmp(:,:,mod(k-3,4):mod(k-1,4)) x(:,:,k-2) t Juni 2010 SKALB Multicore-aware parallelization 7

8 Multicore-aware parallelization Wavefront parallelization: Multiple updates 1 x 4 distribution core0 core1 core2 core3 tmp1(0:3) tmp2(0:3) tmp3(0:3) x( :, :, : ) Running tb wavefronts requires tb-1 temporary arrays tmp to be held in cache! Max. performance gain: tb = 4 But extensive use of cache bandwidth! 22. Juni 2010 SKALB Multicore-aware parallelization 8

9 Multicore-aware parallelization Wavefront Jacobi on state-of-the art multicores B olc ~ 10 MI B olc ~ 2-3 MI B olc ~ 10 ompare against optimal baseline! erformance gain ~ B olc = L3 bandwidth / memory bandwidth MI 22. Juni 2010 SKALB Multicore-aware parallelization 9

10 Multicore-aware programming First lectures on the node level Fast shared on-chip resources: Room for new ideas! Fast thread synchronisation through shared caches Wavefront works fine with Intel compilers but fails with gcc: OM BARRIER: 4-core ore i7: 11,000 cycles (gcc 4.3.3) vs. 800 cycles (icc 11.0) Ability to exploit parallel structures at finer granularity (shorter loops, frequent synchronisation) Technique can easily be extended to many regular stencil based iterative methods, e.g. Gauß-Seidel ( done) Lattice-Boltzmann flow solvers ( work in progress) Multigrid ( BMBF proposal EMMMA) For multi-node applications an hybrid MI+OpenM approach needs to e implemented (next slides) 22. Juni 2010 SKALB Multicore-aware parallelization 10

11 Multicore-aware programming Multi-layer halo exchange Temporal blocking requires multi-layer halo Using diagonal communication elimination (DE) (Ding/He S 2001) Exchanging halo with neighbors done only along the coordinate directions 2 More complex stencils, e.g. occurring at lattice Boltzmann methods, need more attention for deciding which data to communicate Juni 2010 SKALB Multicore-aware parallelization 11

12 Multicore-aware programming Impact of Multi-layer Halo on erformance Assumptions for model: No overlap between communication and computation QDR-InfiniBand 3.2 GB/s 1.8 µs latency Node performance 2 GLU/s Reduced latency by message aggregation strong scaling Degrade due to halo work No impact for large domain sizes 22. Juni 2010 SKALB Multicore-aware parallelization 12

13 Multicore-aware programming Strong Scaling Nehalem E cluster QDR Infiniband Domain size: GB Benefit of temporal blocking eaten by communication overhead (~ 60-70%) ipelining 1N suffers from bulksynchronized communication 22. Juni 2010 SKALB Multicore-aware parallelization 13

14 Multicore-aware programming Weak Scaling Nehalem E luster QDR Infiniband Subdomain size per node: Benefit from temporal blocking can be maintained 22. Juni 2010 SKALB Multicore-aware parallelization 14

15 arallel GU computing with Lattice-Boltzmann solvers a focus of SKALB

16 reliminary comparison Blue-Gene/ GU-luster 96 GUs: NVIDIA Tesla 24 host nodes D3Q13 discretization #cores precision # grid nodes Blue-Gene/ double 10 9 ~20 GLUS Ludwig (96 GUs) single 3.2 x 10 9 ~45 GLUS absolute performance M. Krafczyk slide 16 ih Gauß-Allianz H Status onference, June 22-24,2010

17 WALBERLA: Block-/patch-structured heterogeneous U/GU D3Q19 Halo exchange between blocks Single precision computations GU cluster at HLRS Blocks GU: 1 GU: 22, U:1 Nodes rocesses 2 x GU 60 x GU 2 x GU + 60 x GU + 60 x GU + 60 x GU + 6 x U 180 x U 420 x U 660 x U MFLUS ~3 TFlop/s 22. Juni 2010 SKALB Multicore-aware parallelization 17

18 Besten Dank Financial support through KONWIHR-II projects OMI4papps, BMBF roject SKALB (01 IH08003A) Juni 2010 SKALB Multicore-aware parallelization 18