Piccolo. Fast, Distributed Programs with Partitioned Tables. Presenter: Wu, Weiyi Yale University. Saturday, October 15,

Transcription

1 Piccolo Fast, Distributed Programs with Partitioned Tables 1 Presenter: Wu, Weiyi Yale University

2 Outline Background Intuition Design Evaluation Future Work 2

3 Outline Background Intuition Design Evaluation Future Work 3

4 MapReduce Data Master 4

5 MapReduce Data Master Assign Map / Reduce 4

6 MapReduce Data Master 4

7 MapReduce Data Master Map 4

8 MapReduce Data Master Reduce 4

9 MapReduce Data Master 4

10 MPI / RPC Data 5

11 MPI / RPC Data Messages All Around 5

12 MPI / RPC Data 5

13 Distributed Shared Memory 6

14 Distributed Shared Memory Underlying Message write / read 6

15 Distributed Shared Memory Non-atomic Hard to optimize 6

16 Key-Value Table 7

17 Key-Value Table Underlying Message put / get 7

18 Key-Value Table Atomic Easy to optimize 7

19 Outline Background Intuition Design Evaluation Future Work 8

20 What do we need? MapReduce MPI / RPC DSM k/v Table 9

21 What do we need? In-memory MapReduce MPI / RPC DSM k/v Table 9

22 What do we need? In-memory Data-centric MapReduce MPI / RPC DSM k/v Table 9

23 What do we need? In-memory Data-centric MapReduce MPI / RPC Exposing globally shared state DSM k/v Table 9

24 What do we need? In-memory Data-centric MPI / RPC Exposing globally shared state DSM k/v Table 9

25 What do we need? In-memory Data-centric MPI / RPC Exposing globally shared state No low-level messages DSM k/v Table 9

26 What do we need? In-memory Data-centric Exposing globally shared state No low-level messages DSM k/v Table 9

27 What do we need? In-memory Data-centric Exposing globally shared state No low-level messages Easy to use / optimize DSM k/v Table 9

28 What do we need? In-memory Data-centric Exposing globally shared state No low-level messages Easy to use / optimize k/v Table 9

29 What do we need? In-memory Data-centric Exposing globally shared state No low-level messages Easy to use / optimize 9

30 Is k/vtable enough? Replace put-get pairs to atomic ops Improving locality Load Balancing Rapid and Reliable Checkpoint 10

31 Outline Background Intuition Design Evaluation Future Work 11

32 Overview Master 12

33 Overview Master Assign Partition & Task 12

34 Overview Master

35 Overview Master 3 7 Execute Kernel 12

36 Overview Master 12

37 Overview Master Kernel Finished 12

38 Overview Master 12

39 Expressing Locality Reduce remote read (get) Co-locate a kernel execution with some table partitions Co-locate partitions of different tables (with same partition id) 13

40 User-defined accumulators 14

41 User-defined accumulators a <- get(a) b <- get(b) res <- a + b put(b, res) 14

42 User-defined accumulators a <- get(a) a <- get(a) b <- get(b) res <- a + b put(b, res) 14

43 User-defined accumulators a <- get(a) a <- get(a) b <- get(b) res <- a + b update(b, a) put(b, res) 14

44 Load Balance Master 15

45 Load Balance Master Assign Partition & Task 15

46 Load Balance Master 15

47 Load Balance Master Execute Kernel 15

48 Load Balance Master Execute Kernel Migrate Partition 15

49 Load Balance Master Execute Kernel Migrate Partition 15

50 Load Balance Master Execute Kernel 15

51 Load Balance Master Execute Kernel Steal Work 15

52 Load Balance Master Execute Kernel Steal Work 15

53 Load Balance Master Execute Kernel 15

54 Load Balance Master 15

55 Checkpoint Master 16

56 Checkpoint Master Start Checkpoint 16

57 Checkpoint Master Snapshot 16

58 Checkpoint Master Snapshot 16

59 Checkpoint Master Log Ops 16

60 Checkpoint Master 16

61 Checkpoint Master Finish Checkpoint 16

62 Checkpoint Master 16

63 Outline Background Intuition Design Evaluation Future Work 17

64 Scaling - Speedup 8 Speedup Workers K-Means N-Body Matrix Multiply PageRank Ideal Figure 6: Scaling performance (fixed default input size) 18

65 Scaling - Input Relative Runtime Workers K-Means Matrix Multiply PageRank Ideal Figure 7: Scaling input size. 19

66 Scaling - Input (cont.) 1.2 Relative Runtime Workers K-Means Pagerank Ideal Figure 8: Scaling input size on EC2. 20

67 Comparison with MapReduce on Hadoop k-means (secs) PageRank (secs) Piccolo Hadoop Workers Figure 9: Per-iteration running time of PageRank and k-means in Hadoop and Piccolo (fixed default input size). 21

68 Comparison with MPI Relative Time Piccolo MPI Workers Figure 10: Runtime of matrix multiply, scaled relative to MPI. 22

69 Load Balance Runtime (secs) Iteration number Normal - No Stealing Normal - Stealing Slow Worker - No Stealing Slow Worker - Stealing Figure 11: Effect of Work Stealing and Slow Workers 23

70 Outline Background Intuition Design Evaluation Future Work 24

71 Future Work Log-based scalable failure handling More user-defined accumulator per table Distributed as Parallel 25

72 Thanks ~.~ 26