Asynchronous Graph Processing

Transcription

1 Asynchronous Graph Processing CompSci Instructor: Ashwin Machanavajjhala (slides adapted from Graphlab talks at UAI 10 & VLDB 12 and Gouzhang Wang s talk at CIDR 2013) Lecture 15 : Spring 13 1

2 Recap: Pregel Superstep Superstep Superstep Superstep 3 Figure 2: Maximum Value Example. Dotted lines are messages. Shaded vertices have voted to halt. Lecture 15 : Spring 13 2

3 Graph Processing Dependency Graph Local Updates IteraBve ComputaBon My Interests Friends Interests Lecture 15 : Spring 13 3

4 This Class Asynchronous Graph Processing Lecture 15 : Spring 13 4

5 Example: Belief PropagaBon p(x 1,x 2,,x n ) / Y Y u(x u ) u,v(x u,x v ) u2v (u,v)2e Want to compute marginal distribubon at each node. Lecture 15 : Spring 13 5

6 Belief PropagaBon Belief at a vertex depends on messages received from neighboring verbces v b x ϕ (x ) m (x ), m (x ) u Lecture 15 : Spring 13 6

7 Belief PropagaBon Belief at a vertex depends on messages received from neighboring verbces v b x ϕ (x ) m (x ) m (x ), b (x ) m (x ) φ, (x, x ) m (x ) (2) u Lecture 15 : Spring 13 7

8 Original BP Algorithm A B C D E F G H I Lecture 15 : Spring

9 Original BP Algorithm can be inefficient Spends Bme updabng nodes which have already converged Challenge = Boundaries Lecture 15 : Spring 13 9

10 Residual BP ImplementaBon A B C Scheduler D E F G H I Lecture 15 : Spring 13 10

11 Residual BP ImplementaBon A B C Scheduler D E F G H I Lecture 15 : Spring 13 11

12 Residual BP ImplementaBon A B C Scheduler D E F G H I Lecture 15 : Spring 13 12

13 Residual BP ImplementaBon Ordering based on residual (max change in message value) B A B C D D E F Scheduler G H I Lecture 15 : Spring 13 13

14 Residual BP ImplementaBon A B C D Scheduler D E F G H I Lecture 15 : Spring 13 14

15 Residual BP ImplementaBon B F A B C D B C G E D E F Scheduler G H I Lecture 15 : Spring 13 15

16 Residual BP converges faster 1 [Elidan et al UAI 2006] % of runs converged AGBP RGBP time in seconds Lecture 15 : Spring 13 16

17 Summary Asynchronous serial graph algorithms can converge faster than synchronous parallel graph algorithms Is there a way to correctly transform asynchronous serial algorithms to run in a parallel seyng? Lecture 15 : Spring 13 17

18 GRAPHLAB Lecture 15 : Spring 13 18

19 GraphLab Data Graph Shared Data Table Scheduling Update FuncBons and Scopes 19

20 Data Graph A Graph with data associated with every vertex and edge. X 1 X 2 X 3 X 4 x 3 : current belief X 5 X 6 X 7 X 8 X 9 X 10 X 11 Φ(X 6,X 9 ): Binary potenbal :Data 20

21 Update FuncBons Update Func=ons are operabons which are applied on a vertex and transform the data in the scope of the vertex BP Update: - Read messages on adjacent edges - Read edge potenbals - Compute a new belief for the current vertex - Write new messages on edges 21

22 Update FuncBon Schedule a CPU 1 a b c d h a e f g i b CPU 2 h i j k d 22

23 Update FuncBon Schedule CPU 1 a b c d a e f g i b CPU 2 h i j k d 23

24 StaBc Schedule Scheduler determines the order of Update FuncBon EvaluaBons Synchronous Schedule: Every vertex updated simultaneously Round Robin Schedule: Every vertex updated sequenbally 24

25 Need for Dynamic Scheduling Converged Slowly Converging Focus Effort 25

26 Dynamic Schedule a CPU 1 a b c d h a e f g b h i j k CPU 2 26

27 Dynamic Schedule Update FuncBons can insert new tasks into the schedule FIFO Queue Priority Queue Splash Schedule Wildfire BP [SelvaBci et al.] Residual BP [Elidan et al.] Splash BP [Gonzalez et al.] 27

28 Global InformaBon What if we need global informabon? Algorithm Parameters? Sufficient StaBsBcs? Sum of all the verbces? 28

29 Shared Data Table (SDT) Global constant parameters Constant: Temperature Constant: Total # Samples 29

30 Sync OperaBon Sync is a fold/reduce operabon over the graph " Accumulate performs an aggregabon over verbces " Apply makes a final modificabon to the accumulated data " Example: Compute the average of all the verbces Sync! Accumulate FuncBon: Add Apply FuncBon: Divide by V

31 Shared Data Table (SDT) Global constant parameters Global computabon (Sync Opera=on) Constant: Temperature Sync: Loglikelihood Constant: Total # Samples Sync: Sample Statistics 31

32 Safety and Consistency 32

33 Write- Write Race Write- Write Race If adjacent update funcbons write simultaneously Lek update writes: Final Value Right update writes: 33

34 Race CondiBons + Deadlocks Just one of the many possible races Race- free code is extremely difficult to write GraphLab design ensures race- free operabon 34

35 Scope Rules Guaranteed safety for all update funcbons 35

36 Full Consistency Only allow update funcbons two verbces apart to be run in parallel Reduced opportunibes for parallelism 36

37 Obtaining More Parallelism Not all update funcbons will modify the enbre scope! Belief Propaga=on: Only uses edge data Gibbs Sampling: Only needs to read adjacent verbces 37

38 Edge Consistency 38

39 Obtaining More Parallelism Map opera=ons. Feature extracbon on vertex data 39

40 Vertex Consistency 40

41 SequenBal Consistency GraphLab guarantees sequen=al consistency For every parallel execu=on, there exists a sequen=al execu=on of update funcbons which will produce the same result. Parallel CPU 1 CPU 2 Bme SequenBal CPU 1 41

42 GraphLab Data Graph Shared Data Table Scheduling Update FuncBons and Scopes 42

43 DISTRIBUTED GRAPHLAB Lecture 15 : Spring 13 43

44 DistribuBng GraphLab NOT SHARED- NOTHING (unlike MapReduce / Pregel) Need to have distributed shared memory No change to the update step Need to to distributed scheduling Need to ensure distributed consistency Need to ensure fault tolerance Lecture 15 : Spring 13 44

45 Distributed Graph ParBBon the graph across mulbple machines. 45

46 Distributed Graph Ghost verbces maintain adjacency structure and replicate remote data. ghost verbces 46

47 Distributed Graph Cut efficiently using HPC Graph parbboning tools (ParMeBs / Scotch / ) ghost verbces 47

48 Update FuncBons User- defined program: applied to a vertex and transforms data in scope of vertex Pagerank(scope){ // Update the current vertex data vertex.pagerank = α ForEach inpage: vertex.pagerank += (1 α) inpage.pagerank // Reschedule Neighbors if needed if vertex.pagerank changes then reschedule_all_neighbors; } 48

49 Distributed Scheduling Each machine maintains a schedule over the verbces it owns. a f h a b c d g e f g h i j k Distributed Consensus used to identify completion 49

50 Distributed Consistency SoluBon 1 SoluBon 2 Graph Coloring Distributed Locking

51 Edge Consistency via Graph Coloring VerBces of the same color are all at least one vertex apart. Therefore, All verbces of the same color can be run in parallel! 51

52 ChromaBc Distributed Engine Execute tasks on all vertices of color 0 Execute tasks on all vertices of color 0 Time Ghost Synchronization Completion + Barrier Execute tasks on all vertices of color 1 Execute tasks on all vertices of color 1 Ghost Synchronization Completion + Barrier 52

53 Problems Require a graph coloring to be available. Frequent Barriers make it extremely inefficient for highly dynamic systems where only a small number of verbces are acbve in each round. 53

54 Distributed Consistency SoluBon 1 SoluBon 2 Graph Coloring Distributed Locking

55 Distributed Locking Edge Consistency can be guaranteed through locking. : RW Lock 55

56 Consistency Through Locking Acquire write- lock on center vertex, read- lock on adjacent. 56

57 Consistency Through Locking Multicore Setting PThread RW- Locks Distributed Setting A B C D Distributed Locks " Challenges " Latency " Solution " Pipelining CPU Machine 1 A C B D Machine 2 57 A B C D

58 No Pipelining lock scope 1 Time scope 1 acquired update_funcbon 1 release scope 1 Process request 1 Process release 1 58

59 Pipelining / Latency Hiding Hide latency using pipelining Time lock scope 1 lock scope 2 lock scope 3 scope 1 acquired scope 2 acquired scope 3 acquired update_funcbon 1 release scope 1 update_funcbon 2 release scope 2 Process request 1 Process request 2 Process request 3 Process release 1 59

60 Checkpoints for Fault Tolerance 1: Stop the world 2: Write state to disk

61 vertices updated Snapshot Performance Because we have to stop the world, One 2.5 x slow machine slows everything down! no 108 snapshot No Snapshot Snapshot =me Slow machine async. snapshot sync. snapshot Snapshot One slow machine time elapsed(s) 61

62 Bexer CheckpoinBng Based on [Chandy, Lamport 85] Edge consistent update funcbon Algorithm 5: Snapshot Update on vertex v if v was already snapshotted then Quit Save D v // Save current vertex foreach u 2 N[v] do // Loop over neighbors if u was not snapshotted then Save data on edge D u$v Schedule u for a Snapshot Update Mark v as snapshotted Lecture 15 : Spring 13 62

63 Async. Snapshot Performance No penalty incurred by the slow machine! vertices updated 2.5 x no 108 snapshot No Snapshot async. snapshot One slow machine sync. snapshot Snapshot time elapsed(s) 63

64 Summary Asynchronous serial graph algorithms can converge faster than synchronous parallel graph algorithms GraphLab provides high level abstracbons for wribng asynchronous graph algorithms Takes care of consistency and scheduling Distributed GraphLab Graph processing using color- steps Consistency ensured via pipelined distributed locking Fault tolerance via fine grained checkpoinbng Lecture 15 : Spring 13 64