Scalable Network Analysis

Transcription

1 Inderjit S. Dhillon University of Texas at Austin COMAD, Ahmedabad, India Dec 20, 2013

2 Outline Unstructured Data - Scale & Diversity Evolving Networks Machine Learning Problems arising in Networks Recommender Systems Link Prediction Sign Prediction Formulation as Missing Value Estimation Scalable Algorithms NOMAD: Distributed matrix completion algorithm Results on Applications Conclusions 2

3 Structured Data Data organized into fields: Relational databases, spreadsheets, XML Highly optimized for storage & retrieval (e.g. using SQL) 3

4 Structured Data Focus is on data format, efficient storage & search Less or no uncertainty in semantics: e.g. businesses know the fields of the data 4

5 Unstructured Data Modern data is unstructured and diverse Networks, Graphs Text Images, Videos 5

6 Unstructured Data Much greater growth rate 6

7 Unstructured Data Dynamic aspects of unstructured data: Constantly evolving Uncertainties abound: What should I ask of the data? Seek insights Heterogeneity renders traditional database models inadequate 7

8 Unstructured Data Buzzwords - Big Data & Data Science Machine Learning: Predictive models for data Engineering perspective: Scale matters 8

9 Network Graphs Social networks (Friendship) APQ6 AQP1 AQP5 Bipartite networks (Membership, Ratings, etc.) MIP AQP2 GK Gene networks (Functional interaction) 9

10 Graph Evolution Social networks are highly dynamic Constantly grow, change quickly over time Users arrive/leave, relationships form/dissolve Understanding graph evolution is important 10

11 Graphs meet Machine Learning Network analysis: Understanding structure & evolution of networks Formulate predictive problems on the adjacency matrix of the graph Confluence of graph theory & machine learning 11

12 Users Recommender Systems Movies Link Prediction Netflix problem: 100M ratings, 0.5M users, 20K movies A Toy Problem In Comparison! Facebook growth 12

13 Users Recommender Systems Movies 13

14 Link Prediction in Social Networks?? Network at time T Network at time T + 1 Problem: Infer missing relationships from a given snapshot of the network 14

15 Predicting gene-disease links? 15

16 Signed Social Networks Like Dislike Like / Dislike? Sign Prediction Problem: Given a snapshot of the signed social network, predict the signs of missing edges 16

17 Formulation as Missing Value Estimation

18 Users Low-rank Matrix Completion Movies 18

19 Users Low-rank Matrix Completion Movies? 19

20 Low-rank Matrix Completion 20

21 Low-rank Matrix Completion min W 2R m k,h2r n k X (i,j)2 (A ij W i H T j ) 2 + (kw k 2 F + khk 2 F ) 21

22 Low-rank Matrix Completion? min W 2R m k,h2r n k X (i,j)2 (A ij W i H T j ) 2 + (kw k 2 F + khk 2 F ) 22

23 Low-rank Matrix Completion 3.44 min W 2R m k,h2r n k X (i,j)2 (A ij W i H T j ) 2 + (kw k 2 F + khk 2 F ) 23

24 Link Prediction Can be posed as matrix completion problem A (t) = Issue: Only positive relationships are observed Test Link Score (4,5) 1 (1,4) 1 (3,4) 1 (1,5) 1 24

25 Link Prediction Formulate Biased Matrix Completion Problem: min W 2R n k,h2r n k X (i,j)2 (A ij W i H T j ) 2 + X (i,j) /2 (A ij W i H T j ) 2 + (kw k 2 F + khk 2 F ) A (t) = Test Link Score (4,5) 0.59 (1,4) 0.59 (3,4) 0.67 (1,5)

26 Signed Social Networks Balanced Not balanced Friend of a friend is a friend Enemy of an enemy is a friend Social Balance [Harary,1953]: In real-world signed networks, triangles tend to be balanced 26

27 Signed Social Networks Theorem: All triangles in a network are balanced if and only if there exist two antagonistic groups. 27

28 Signed Social Networks Weakly Balanced Not balanced Relaxation: Weak balance Allow triangles with all negative edges 28

29 Signed Social Networks Theorem: All triangles in a network are weakly balanced if and only if there exist k antagonistic groups. 29

30 Sign Prediction Theorem: A k-weakly balanced signed network has rank at most k. Sign inference can be posed as low-rank matrix completion Theorem: If there are no small groups, the underlying network can be exactly recovered, under certain conditions. K. Chiang et al. Prediction and Clustering in Signed Networks: A Local to Global Perspective. To appear in JMLR. 30

31 Scalable Algorithms

32 Stochastic Gradient Sample random index (i,j) and update corresponding factors: w i w i ((A ij w T i h j )h j + w i ) h j h j ((A ij w T i h j )w i + h j ) Time per update O(k) Effective for very large-scale problems 32

33 Distributed Stochastic Gradient Descent (DSGD) [Gemulla et al. KDD 2011] Decoupled updates Easy to parallelize But communication & computation are interleaved 33

34 DSGD 34

35 DSGD Curse of the last reducer 35

36 DSGD 36

37 NOMAD Goal: Keep CPU & network simultaneously busy. Non-locking stochastic Multi-machine algorithm for Asynchronous & Decentralized matrix factorization Asynchronous distributed solution. 37

38 NOMAD Nomadic variables queue h 1 h 5 h 4 h 3 Native variables w 1 w 2 w 3 Workers w 4 h 2 38

39 NOMAD Nomadic variables queue h 1 h 5 h 3 Native variables Push h 4 w 1 w 2 w 3 Workers w 4 h 2 h 4 39

40 NOMAD 40

41 NOMAD 41

42 NOMAD 42

43 NOMAD 43

44 NOMAD 44

45 NOMAD 45

46 NOMAD 46

47 NOMAD 47

48 NOMAD 48

49 NOMAD 49

50 NOMAD 50

51 NOMAD Algorithm 1. Initialize: Randomly assign h j columns to worker queues 2. Parallel Foreach q in {1,2,...,p} 3. If queue[q] not empty then 4. (j, h j ) queue[q].pop() 5. for ( i, j) in (q) do j 6. Do SGD updates 7. end for 8. Sample q uniformly from {1,2,...,p} 9. queue[q ].push( (j, h j ) 10. end if 11. Parallel End 51

52 NOMAD Algorithm 1. Initialize: Randomly assign h j columns to worker queues 2. Parallel Foreach q in {1,2,...,p} 3. If queue[q] not empty then 4. (j, h j ) queue[q].pop() 5. for ( i, j) in (q) do j 6. Do SGD updates Distributed setting: 7. end for Write over the network 8. Sample q uniformly from {1,2,...,p} 9. queue[q ].push( (j, h j ) 10. end if 11. Parallel End Concurrent object 52

53 Algorithm Complexity Average space required per worker: O(mk/p + nk/p + /p) Average time for one sweep: O( k/p) 53

54 Results on Applications

55 Recommender Systems Netflix dataset: 2,649,429 users, 17,770 movies, ~100M ratings Multicore Distributed 55

56 Recommender Systems Synthetic dataset: ~85M users, 17,770 items, ~8.5B observations 56

57 Link Prediction Flickr dataset: 1.9M users & 42M links. Test set: sampled 5K users. 57

58 Predicting Gene-Disease Links CATAPULT(Blom et al.,2013) Katz Biased MF Test pair rank CATAPULT(Blom et al.,2013) Katz Biased MF Singleton gene rank Pr(Rank <= x) Pr(Rank <= x) Rank Rank 58

59 Sign Prediction Epinions dataset (+ve & -ve reviews): 131K nodes, 840K edges, 15% edges negative. MF-ALS is faster and achieves higher accuracy. MF-ALS takes 455 secs on network with 1.1M nodes & 120M edges 59

60 Conclusions Rapid growth of unstructured data demands scalable machine learning solutions for analysis Machine learning problems arising in network analysis can be cast in the matrix completion framework Our proposed asynchronous distributed algorithm NOMAD outperforms state-of-the-art matrix completion solvers Beyond Matrix Completion: Asynchronous distributed framework for solving machine learning problems 60