An Efficient Parallel Load-balancing Framework for Orthogonal Decomposition of Geometrical Data

Transcription

1 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS An Efficient Parallel Load-balancing Framework for Orthogonal Decomposition of Geometrical Data Bruno R. C. Magalhães Farhan Tauheed Thomas Heinis Anastasia Ailamaki Felix Schürmann Blue Brain Project, École Polytechnique Fédérale de Lausanne (EPFL), CH Data-Intensive Applications and Systems laboratory, EPFL, CH ISC 06, Frankfurt st June 06

2 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS AGENDA. We present our problem;. We detail an algorithm for the efficient orthogonal decomposition of spatial data;. We compare it to commonly used methods, showing better accuracy and a lower time to solution

3 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS INTRO Spatial data decomposition is an important problem for High Performance Computing, applied in several fields: I astrophysics eg N-body simulations, Gordon Bell Prize (GBP) winners of 009, 00, 0; I cardiac model simulations, GBP 05 finalist I fluid dynamics e.g. cloud cavitation, GBP 0 winner; I materials engineering e.g. materials crystallization, GBP 0 winner; I weather forecasting; I direct volume rendering;

4 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS DATA OVERVIEW (For simplicity, only 5% of neurons are presented) I Neurons spatially discretized as tree of compartments (cylinders); I Extremely dense data structures; I Approximately 0K cylinders per neuron; I Mouse brain: approx. 80M neurons; Human brain: 00B neurons;

5 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS STATE OF THE ART -APPROXIMATION METHODS Single-Axis Non-Uniform Grid Sort Tile Recursive Orthogonal Rec. Bisection Histogram-based Sampling-based Threshold-based # X # Y # Z Serial, SA / NUG Serial, STR Parallel (e.g. Zoltan), ORB 5

6 6 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS PROBLEM STATEMENT Min, max and mean compartments count per compute node, post-ghosting for alternative datasets of 0K neurons (00M compartments): Dense data structures are highly penalized by approxim. methods: - High time to solution; - Barrier on maximum input circuit size;

7 7 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS SORT BALANCE SPLIT I A framework underlying a balanced orthogonal spatial decomposition of spatial data. I Three steps per dimension:. Distributed sorting;. Distributed load balancing;. Network split;

8 8 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS SORT ( MPI GATHERV, MPI BCAST, MPI ALLTOALLV) rank 0 8 elements rank 7 elements rank 6 elements Step : Local data sorting Step : Collection of local samples, gathered by root node Step : broadcast of sample of samples, representing new data distribution intervals rank 0 rank rank Step : Data is redistributed based on distribution intervals. Final data is already sorted elements elements 8 elements

9 9 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS BALANCE ( MPI ALLTOALL, MPI ALLTOALLV) rank 0 rank rank Step : broadcast of elements count elements 6 elements elements Step : redistribution of data, based on mean count per node elements 6 elements 7 elements

10 0 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS SPLIT ( MPI COMM SPLIT) rank 0 rank rank rank rank rank 5 6 rank comm 0 Step : each node calculates new rank and sub-group id Step : based on previous, split network in K independent networks, e.g: if K= rank 0 rank rank 0 rank rank 0 rank 0 rank comm 0 comm comm comm I The comm split is the base of the new recursive step; I new sub-comms process new data set on next dimension;

11 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS EXAMPLE: X DECOMPOSITION OF 6 ELEMENTS Key: rank rank rank rank y x Initial data sorting and load balancing on X axis network split on X axis sorting and load balancing on Y axis network split on Y axis In Brief... I 6 collective communication calls per dimension, independently of network size; I One local sort operation, independently of data size or initial placement; I Accurate final spatial decomposition; I Recursive algorithm: we split main problem in smaller sub-problems, to be executed in parallel.

12 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS TIME TO SOLUTION -BLUEGENE/Q

13 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS WEAK AND STRONG SCALING

14 ISC06 OVERVIEW SORT-BALANCE-SPLIT RESULTS CLOSING REMARKS We presented the Sort-Balance-Split, a framework for the accurate orthogonal decomposition of spatial data. More accurate and lower time to solution on very dense datasets; I Compared with standard configurations of existing methods; Methods tested on a BlueGene/Q supercomputer We plan to open-source the SBS in the near future; Thank you for your attention. Acknowledgments: Research supported by funding from the ETH Domain for the Blue Brain Project (BBP); BlueBrain IV BGQ system financed by ETH Board Funding to BBP and hosted at the Swiss National Supercomputing Center (CSCS). We thank James King, Stuart Yates and Fabien Delalondre for technical discussions.