Optimising MPI Applications for Heterogeneous Coupled Clusters with MetaMPICH

Transcription

1 Optimising MPI Applications for Heterogeneous Coupled Clusters with MetaMPICH Carsten Clauss, Martin Pöppe, Thomas Bemmerl Lehrstuhl für Betriebssysteme RWTH Aachen

2 Table of Contents Part 1 Metacomputing / Coupled Clusters What is a Metacomputer? Our Solution: MetaMPICH Advantages / Disadvantages Part 2 Optimising MPI Applications Boundary Value Problems on Structured Grids Load Balancing on Metacomputers Our Solutions: SmartPart / MetaComm

3 Coupled Clusters Metahost A Metahost B SCI SCI SCI SCI ATM Metacomputer

4 Example of a Possible Use Nice idea, but how? Berlin Aachen Dresden

5 How to build a Metacomputer? MetaMPICH MP-MPICH Transparent MPI system on metacomputers - Windows 2000/XP, Linux, Solaris -SCI, TCP, SHMEM internal - TCP and AAL5 external

6 Architecture of MetaMPICH Metahost A Metahost B SCI SCI SCI Router Router

7 Architecture of MetaMPICH MPI Application MPI process MPI Router process MPI Router process MPI process MPI API MPI API MPI API Abstract Device Interface ADI device ADI device ADI device local network Cluster Network local network

8 Architecture of MetaMPICH Metahost A MPI Application Metahost B MPI process MPI Router process MPI Router process MPI process MPI API MPI API MPI API ADI device ADI device ADI device local network Cluster Network local network

9 Architecture of MetaMPICH Metahost A MPI Application Metahost B MPI process Router Router MPI process Pseudo MPI API device MPI API Pseudo device MPI API gateway tunnel tunnel gateway ADI device ADI device ADI device local network Cluster Network local network

10 Architecture of MetaMPICH Metahost A MPI Application Metahost B MPI process MPI Router process MPI Router process MPI process MPI API MPI API MPI API gateway tunnel tunnel gateway ADI device ADI device ADI device local network Cluster Network local network

11 Architecture of MetaMPICH Router Router SCI SCI SCI Router Router

12 Architecture of MetaMPICH Router Router Router SCI Router Router Router

13 Metacomputing Advantage Disadvantage Many more MPI processes by using the remote computational power Inter-cluster communication is the system s bottleneck Existing applications cannot benefit from the fast internal networks

14 Process Grouping Division of the algorithmic problem:

15 Process Grouping MPI_COMM_WORLD MPI_COMM_LOCAL MPI_COMM_LOCAL

16 Metacomputing Pay Attention to Load Balance!!! Clusters do NOT need to be identical!!! - Different CPU Power - Different Number of Nodes - Different Internal Networks

17 Metacomputing Challenges: Inter-Metahost Communication Bottleneck Heterogeneous System: Load Balance!!!

18 Table of Contents Part 1 Metacomputing / Coupled Clusters What is a Metacomputer? Our Solution: MetaMPICH Advantages / Disadvantages Part 2 Optimising MPI Applications Boundary Value Problems on Structured Grids Load Balancing on Metacomputers Our Solutions: SmartPart / MetaComm

19 Boundary Value Problems Distribution of Temperature in a Plate: T(x,y) ΔT=0

20 Boundary Value Problems Discretised Problem: T(xi,yj)

21 Boundary Value Problems Memory Cells:

22 Boundary Value Problems Discretising the Problem: Iterative Solver: big and sparsely populated linear equation systems simple example: Jacobior Gauss-Seidel method

23 Boundary Value Problems i-1, j Iteration Rule: i, j-1 i, j+1 i, j+1 T[ i ][ j ] = 0.25 (T[ i-1][ j ] + T[ i+1][ j ] + T[ i ][ j-1] + T[ i ][ j+1] ) Large number of iterations over the grid Approximation for the solution

24 Boundary Value Problems Discrete Solution:

25 Parallelisation Domain Decomposition: Proc0 Proc1 Exchange into Ghost Lines Proc1

26 Using a Metacomputer Grouping of Processes?

27 Using a Metacomputer Inner Boundaries:

28 Using a Metacomputer Reduced Communication: Cluster A Cluster B

29 Using a Metacomputer Load Balance?

30 Using a Metacomputer Domain Distribution onto Processes: ProcA0 ProcA1 ProcB1 ProcB2 ProcB3

31 Using a Metacomputer Load Balance VS Communication ProcA0 ProcA1 ProcB0 ProcB1 ProcB2

32 Metacomputing? N N N 50% reduction

33 Metacomputing? time Communication Calculation 50% reduction grid height N

34 Smart Partitioner A smart partition scheme provides: - Load Balance and - Reduced Communication (if possible) A smart partition scheme depends on: -the given boundary value problem -the structures of the metacomputer Smart Partitioner (SmartPart)

35 Smart Partitioner Determination of Cut Metrics:

36 Smart Partitioner Communication Metric:

37 Smart Partitioner Load Balance Metric:

38 Smart Partitioner Superposition: best cut

39 Smart Partitioner Decomposition Patterns: three metahosts / first cut: horizontal

40 Smart Partitioner Decomposition Patterns: three metahosts / first cut: vertical

41 Example: CFD Simulation

42 Example: CFD Simulation Cluster A

43 Example: CFD Simulation Cluster A Cluster B Cluster C

44 Adaptation Layer Smart Partitioner Optimal decomposition scheme for your problem on your metacomputer How to use it in your application? Adaptation Layer on top of MetaMPICH your applications can easily attach to Communication Library (MetaComm)

45 Adaptation Layer MetaComm: - additional communication functions - can replace all explicit MPI functions Optimised Communication: - based on smart partition scheme - performs every possible reduction - metacomputer is still transparent

46 Adaptation Layer normal case: Application (iterative simulation) Parallelising MPI

47 Adaptation Layer Coupled Clusters: Application (iterative simulation) Parallelising MetaMPICH MPI

48 Adaptation Layer transparent use NO good performance SmartPart Application (iterative simulation) MetaComm Parallelising MetaMPICH MPI MPI

49 Conclusion And Outlook Metacomputing And Coupled Clusters - Software Library MetaMPICH - Challenges of metacomputing Optimising of Applications - Algorithms on structured grids - SmartPart and MetaComm

50 Conclusion And Outlook VIOLA-Project: Jülich Düsseldorf Köln Aachen Bonn

51 Conclusion And Outlook Metacomputing can be a powerful way to increase performance. Metacomputer is a heterogeneous System!!! You should always search for ways to optimise your applications!!!

52 Thank you for your attention! Any Questions? Lehrstuhl für Betriebssysteme RWTH Aachen