Case Study 4: Collaborative Filtering. GraphLab

Transcription

1 Case Study 4: Collaborative Filtering GraphLab Machine Learning/Statistics for Big Data CSE599C1/STAT592, University of Washington Carlos Guestrin March 14 th, 2013 Carlos Guestrin Social Media Science Adver0sing Web! Graphs encode the rela0onships between: People Facts Products Interests Ideas! Big: 100 billions of ver0ces and edges and rich metadata! Facebook (10/2012): 1B users, 144B friendships! Twi@er (2011): 15B follower edges Carlos Guestrin

2 Facebook Graph Carlos Guestrin 2013 Slide from Facebook Engineering presentation 3 Addressing Graph- Parallel ML Data-Parallel Map Reduce Feature ExtracOon Cross ValidaOon CompuOng Sufficient StaOsOcs Graph-Parallel Graph- Parallel AbstracOon Graphical Models Gibbs Sampling Belief Propagation Variational Opt. Collaborative Filtering Tensor Factorization Semi-Supervised Learning Label Propagation CoEM Data-Mining PageRank Triangle Counting Carlos Guestrin

3 Asynchronous Belief PropagaOon Challenge = Boundaries Many Updates Synthetic Noisy Image Cumulative Vertex Updates Few Updates Graphical Model Algorithm idenofies and focuses on hidden sequenoal structure Carlos Guestrin Synchronous v. Asynchronous n Bulk synchronous processing: Computation in phases n All vertices participate in a phase n Though OK to say no-op All messages are sent Simpler to build, like Map-Reduce n No worries about race conditions, barrier guarantees data consistency n Simpler to make fault-tolerant, save data on barrier Slower convergence for many ML problems In matrix-land, called Jacobi Iteration Implemented by Google Pregel 2010 n Asynchronous processing: Vertices see latest information from neighbors n Most closely related to sequential execution Harder to build: n Race conditions can happen all the time Must protect against this issue n More complex fault tolerance n When are you done? n Must implement scheduler over vertices Faster convergence for many ML problems In matrix-land, called Gauss-Seidel Iteration Implemented by GraphLab 2010, 2012 Carlos Guestrin

4 The GraphLab Goals Know how to solve ML problem on 1 machine Efficient parallel predicoons Carlos Guestrin Data Graph Data associated with veroces and edges Graph: Social Network Vertex Data: User profile text Current interests esomates Edge Data: Similarity weights Carlos Guestrin

5 How do we program graph computaoon? Think like a Vertex. - Malewicz et al. [SIGMOD 10] Update FuncOons User- defined program: applied to vertex transforms data in scope of vertex pagerank(i, scope){ } Carlos Guestrin

6 Connected Components Carlos Guestrin Update FuncOon Example: Connected Components Carlos Guestrin

7 The Scheduler The scheduler determines order vertices are updated CPU 1 a b c d Scheduler b a h i CPU 2 h e f g i j k Carlos Guestrin Example Schedulers! Round- robin! SelecOve scheduling (skipping):! round robin but jump over un- scheduled veroce! FIFO! PrioriOze scheduling! Hard to implement in a distributed fashion! ApproximaOons used (each machine has its own priority queue) Carlos Guestrin

8 Ensuring Race- Free Code How much can computaoon overlap? Carlos Guestrin Need for Consistency? Higher Throughput (#updates/sec) No Consistency PotenOally Slower Convergence of ML Carlos Guestrin

9 GraphLab Ensures Sequen0al Consistency For each parallel execution, there exists a sequential execution of update functions which produces the same result Parallel CPU 1 CPU 2 time Sequential Single CPU Carlos Guestrin Consistency in CollaboraOve Filtering Train RMSE Dynamic Inconsistent Inconsistent updates Dynamic Consistent updates Updates Millions Nemlix data, 8 cores Carlos Guestrin

10 The GraphLab Framework Graph Based Data Representa.on Update FuncOons User Computa.on Scheduler Consistency Model Carlos Guestrin Triangle CounOng in Graph 40M Users 1.2B Edges Total: 34.8 Billion Triangles Hadoop GraphLab 1536 Machines 423 Minutes 64 Machines, 1024 Cores 1.5 Minutes Carlos Guestrin 2013 Hadoop results from [Suri & Vassilvitskii '11] 20 10

11 CoEM (Jones et al., 2005) Named Entity Recognition Task Is Dog an animal? Is Catalina a place? dog <X> ran quickly Australia travelled to <X> Catalina Island <X> is pleasant Carlos Guestrin Never Ending Learner Project (CoEM) Vertices: 2 Million Edges: 200 Million Hadoop 95 Cores 7.5 hrs Distributed GraphLab 32 EC2 machines 80 secs Carlos Guestrin

12 Women on the Verge of a Nervous Breakdown The CelebraOon recommend City of God Wild Strawberries What do I recommend??? La Dolce Vita Carlos Guestrin Interpreting Low-Rank Matrix Completion (aka Matrix Factorization) X = L R Carlos Guestrin

13 Matrix Completion as a Graph X = X ij known for black cells X ij unknown for white cells Rows index users movies Columns index index movies users Carlos Guestrin Coordinate Descent for Matrix Factorization: Alternating Least-Squares min L,R X (L u R v r uv ) 2 (u,v,r uv)2x:r uv6=? n n n Fix movie factors, optimize for user factors X Independent least-squares over users min (L u R v r uv ) 2 L u v2v u Fix user factors, optimize for movie factors X Independent least-squares over movies min (L u R v r uv ) 2 R v u2u v System may be underdetermined: n Converges to Carlos Guestrin

14 Alternating Least Squares Update Function min L u X v2v u (L u R v r uv ) 2 min Rv X (L u R v r uv ) 2 u2uv recommend Carlos Guestrin t = L (t) u SGD for Matrix Factorization in Map-Reduce? R (t) v r uv " L (t+1) u R (t+1) v # " (1 t u )L (t) u (1 t v )R v (t) t t R v (t) t t L (t) u # recommend Carlos Guestrin

15 6. Before GraphChi: Going small with GraphLab 7. After 8. After Solve huge problems on small or embedded devices? Key: Exploit non- volaole memory (starong with SSDs and HDs) Carlos Guestrin GraphChi disk- based GraphLab Challenge: Random Accesses Novel GraphChi solu0on: Parallel sliding windows method è minimizes number of random accesses Carlos Guestrin

16 Naive Graph Disk Layouts 19 5! Symmetrized adjacency file with values, vertex in- neighbors out- neighbors 5 3:2.3, 19: 1.3, 49: 0.65, : 2.3, 881: : 1.4, 9: 12.1,... synchronize 5: 1.3, 28: 2.2,...! or with file index pointers Random write vertex in- neighbor- ptr out- neighbors 5 3: 881, 19: 10092, 49: 20763, : 2.3, 881: Random... read read/write 19 3: 882, 9: 2872,... 5: 1.3, 28: 2.2,... Carlos Guestrin Parallel Sliding Windows Layout Shard: in- edges for subset of veroces; sorted by source_id in- edges for veroces sorted by source_id Shard 1 Shard 2 Shard 3 Shard 4 Shards small enough to fit in memory; balance size of shards Carlos Guestrin

17 Parallel Sliding Windows ExecuOon Load subgraph for ver0ces in- edges for veroces sorted by source_id Shard 1 Shard 2 Shard 3 Shard 4 Load all in- edges in memory What about out- edges? Arranged in sequence in other shards! And sequenoal writes! Carlos Guestrin Parallel Sliding Windows ExecuOon Load subgraph for ver0ces in- edges for veroces sorted by source_id Shard 1 Shard 2 Shard 3 Shard 4 Load all in- edges in memory Only O(P 2 ) random reads per pass on enore graph Carlos Guestrin

18 Triangle CounOng on Graph Total: 34.8 Billion Triangles 40M Users 1.2B Edges Hadoop GraphChi GraphLab Machines 423 Minutes 59 Minutes 59 Minutes, 1 Mac Mini! 64 Machines, 1024 Cores 1.5 Minutes Carlos Guestrin 2013 Hadoop results from [Suri & Vassilvitskii '11] 35 Release 2.1 available now Documentation Code Tutorials (more on the way) GraphChi 0.1 available now 18

19 What you need to know n Data-parallel versus graph-parallel computation n Bulk synchronous processing versus asynchronous processing n GraphLab system for graph-parallel computation Data representation Update functions Scheduling Consistency model Carlos Guestrin