Architecture-Aware Graph Repartitioning for Data-Intensive Scientific Computing

Transcription

1 Architecture-Aware Graph Repartitioning for Data-Intensive Scientific Computing Angen Zheng, Alexandros Labrinidis, Panos K. Chrysanthis Advanced Data Management Technologies Laboratory Department of Computer Science University of Pittsburgh 2014 BigGraphs Workshop

2 Graph Partitioning and Repartitioning in Scientific Simulations Computation and Communication Graph Vertices---Computational Units 1

3 Graph Partitioning and Repartitioning in Scientific Simulations Computation and Communication Graph Vertex weight: computational cost size: migration cost Vertices---Computational Units Edges---Communication Edge weight: communication cost 2

4 Graph Partitioning and Repartitioning in Scientific Simulations Graph Partitioning Balanced Partitioning Even Load Distribution Minimal Edge-Cuts Minimal Comm Cost 3

5 Graph Partitioning and Repartitioning in Scientific Simulations Time-Evolving Graph The optimal partitioning of the graph changes. 4

6 Graph Partitioning and Repartitioning in Scientific Simulations Graph Repartitioning Balanced Load Distribution Minimal Communication Cost Minimal Migration Cost Existing Graph (Re)Partitioners Assume uniform comm costs among partitions 5

7 Nonuniform Inter-Node Comm Costs Grouping the most communicating vertices to compute nodes as close as possible. 6

8 Nonuniform Intra-Node Comm Costs Grouping the most communicating vertices to cores sharing more cache levels. 7

9 AragonLB Overview AragonLB Architecture-Aware Graph RepartitiONing for Load Balancing 2-Level Repartitioner Inter-Node Repartitioning Regrouping Architecture-Agnostic Repartitioning Architecture-Aware Refinement (TopoFM) Intra-Node Repartitioning HierCacheLB FlatCacheLB 8

10 Roadmap AragonLB Internals Inter-Node Repartitioning Intra-Node Repartitioning Evaluation HierCacheLB FlatCacheLB Setup Results Conclusions Acknowledgements 9

11 Inter-Node Repartitioning cc=1 cc=1 cc=6 cc=6 14 Units Comm Cost (4 Edge-Cuts) 10

12 Inter-Node Repartitioning: Regrouping cc=1 cc=1 cc=6 cc=6 14 Units Comm Cost (4 Edge-Cuts) 11

13 Inter-Node Repartitioning: Repartitioning cc=1 cc=1 cc=6 8 Units Comm Cost (3 Edge-Cuts) 6 Units Migration Cost 12

14 Inter-Node Repartitioning: TopoFM 13

15 Inter-Node Repartitioning: TopoFM Compute initial gain 14

16 Inter-Node Repartitioning: TopoFM Compute initial gain a: P1->P2 gstd(a) = (1-2)*1 15

17 Inter-Node Repartitioning: TopoFM Compute initial gain a: P1->P2 gstd(a) = (1-2)*1 gtopo(a) = 1*(6-1) 16

18 Inter-Node Repartitioning: TopoFM Compute initial gain a: P1->P2 gstd(a) = (1-2)*1 gtopo(a) = 1*(6-1) gmig(a) = 1*(6-1) 17

19 Inter-Node Repartitioning: TopoFM 9 Compute initial gain a: P1->P2 gstd(a) = (1-2)*1 gtopo(a) = 1*(6-1) gmig(a) = 1*(6-1) g(a) =

20 Inter-Node Repartitioning: TopoFM Compute initial gain -3-2 gstd(a) = (1-2)*1 gtopo(a) = 1*(6-1) gmig(a) = 1*(6-1) g(a) =

21 Inter-Node Repartitioning: TopoFM Compute initial gain Select maximal gain vertex, a

22 Inter-Node Repartitioning: TopoFM Compute initial gain Select maximal gain vertex, a. Move a to P

23 Inter-Node Repartitioning: TopoFM Compute initial gain Select a maximal gain vertex, a. Move a to P2 Update the gain of a s nbors. 22

24 Inter-Node Repartitioning: TopoFM Compute initial gain Repeat Select a maximal gain vertex, v. Move v to P2 Update the gain of v s nbors. 23

25 Inter-Node Repartitioning: TopoFM 4 Units Comm Cost (4 Edge-Cuts) 1 Unit Migration Cost 24

26 Inter-Node Repartitioning: TopoFM 25

27 Roadmap AragonLB Internals Inter-Node Repartitioning Intra-Node Repartitioning Evaluation HierCacheLB FlatCacheLB Setup Results Conclusions Acknowledgements 26

28 Intra-Node Repartitioning: HierCacheLB 1. Tree Topology 2. Hierarchical Repartitioning 27

29 Intra-Node Repartitioning: FlatCacheLB Main Idea: Partition the subgraph directly to k parts. Explore all possible assignments. alpha: # of computation steps performed between 2 consecutive repartitioning steps w(pi, Pj): the amount of comm between Pi and Pj c(pi, Pj): inter-core comm cost between Pi and Pj 28

30 Intra-Node Repartitioning: FlatCacheLB Main Idea: Partition the subgraph directly to k parts. Explore all possible assignments. vs(pi, Pj): the amount of data migrated between Pi and Pj c(pi, Pj): inter-core comm cost between Pi and Pj 29

31 Roadmap AragonLB Internals Inter-Node Repartitioning Intra-Node Repartitioning Evaluation HierCacheLB FlatCacheLB Setup Results Conclusions Acknowledgements 30

32 Evaluation: Dataset Combustion Simulation Dataset Vertex Degree V E Min Max Avg. 115,351 2,865, Graph # of Partitions Degree of Imbalance Synthetic Datasets G8 G64 G128 G256 G

33 Evaluation: Platform Evaluation Platform 3-D Torus 5 * 5* 5 Compute Node Sockets L1 L2 L3 2-quad core Private Private Shared 32

34 Evaluation: Algorithms Baselines: ParmetisRepart[1] ZoltanRepart[2] AragonLB Inter-Node Repartitioning Intra-Node Repartitioning PTF Parmetis + TopoFM FlatCacheLB PTH Parmetis + TopoFM HierCacheLB ZTF Zoltan + TopoFM FlatCacheLB ZTH Zoltan + TopoFM HierCacheLB [1]. Parmetis [2]. Zoltan 33

35 Varying # of Partitions alpha=500 ZTH/F PTH/F Up to 60% improvement v.s. Zoltan Up to 46% improvement v.s. Parmetis 34

36 Varying # of Comp. Steps G512 ZTH/F PTH/F Up to 30% improvement 35

37 Varying Sized 3D-Torus G512, alpha=500 Up to 32% improvement 36

38 Breakdown Comm and Mig Volume G512, alpha=500 2-Level Repartitioning Intra-Node Repartitioning Lower inter-node volume (30%~35% Reduction) Bigger as # of hops increases 37

39 Conclusions Proposed a new architecture-aware graph repartitioner, AragonLB Considers the heterogeneity in inter-node communication intra-node communication Experimental study with a combustion simulation dataset Up to 60% improvement (v.s. Parmetis and Zoltan) More gains as heterogeneity increases 38

40 Acknowledgments Many thanks to our collaborators: Peyman Givi, Patrick Pisciuneri, Medhi Nik, Levent Yilmaz, and Esteban Meneses Work funded in part by NSF CBET NSF OIA