Apache Giraph. for applications in Machine Learning & Recommendation Systems. Maria Novartis

Transcription

1 Apache Giraph for applications in Machine Learning & Recommendation Systems Maria Novartis Züri Machine Learning Meetup #5 June 16, 2014

2 Apache Giraph for applications in Machine Learning & Recommendation Systems Maria Novartis Züri Machine Learning Meetup #5 June 16, 2014

3 Outline Machine Learning Cases Why Apache Giraph? Walk-through example for Recommendation Systems What s more? 3

4 Machine Learning cases Friend Recommendation Fake account detection 4

5 Machine Learning cases Product recommendation Online advertisements 5

6 Machine Learning cases Route planning Delivery scheduling 6

7 Machine Learning cases Graphs are everywhere Graphs need processing! 7

8 (Graphs ABC) Graph: a representation of a set of objects V = Vertices (nodes) E = Edges (links) Graphs capture the relationship between objects ) Graphs can be directed or undirected 8

9 Graphs need processing! So what? 9

10 Challenge #1 Scale of graphs indexes ~50B pages has ~1.1B users has ~570M users has ~530M users 10

11 Challenge #2 Complexity of graphs Compute shortest distance from google.com à Multiple passes to compute the result à Inherent dependencies make it hard to parallelize 11

12 MapReduce Well established Efficient for big data analytics Not efficient with iterative algorithms (stateless) Graph algorithms are iterative 12

13 Why Apache Giraph? Explicitly designed for graph processing on top of the Hadoop ecosystem 13

14 The story. Google Pregel (2010) Apache Top Level Project (2012) 1.1 release (2014) Donated to ASF by Yahoo! (2011) 1.0 release (2013) Supported by: Facebook Yahoo! LinkedIn 14

15 Giraph follows the Pregel model or Bulk Synchronous Parallel 15

16 I am a vertex! How would I coordinate with other vertices to solve the problem? Thinking like a vertex 16

17 Shortest Paths I only know my value and who my neighbors are 17

18 Receive messages à Update value à Send messages Vertices compute asynchronously 18

19 Global Synchronization Synchronization barrier 19

20 And again 20

21 And again 21

22 Giraph super powers Message-passing communication In-memory computation à stateful Global synchronization Iterations à Iterations à Iterations 22

23 Recommendation Systems 23

24 Collaborative Filtering Recommendation systems technique June 16, 2014 Apache Giraph for applications in Machine Learning Maria Stylianou 24

25 Giraph for Recommendation Systems Stochastic Gradient Descent algorithm (SGD) 25

26 Giraph for Recommendation Systems Stochastic Gradient Descent algorithm (SGD) 26

27 Giraph for Recommendation Systems Stochastic Gradient Descent algorithm (SGD) 27

28 Giraph for Recommendation Systems Stochastic Gradient Descent algorithm (SGD) 28

29 Giraph for Recommendation Systems Stochastic Gradient Descent algorithm (SGD) 29

30 Giraph for Recommendation Systems Stochastic Gradient Descent algorithm (SGD) 30

31 What s more Okapi ML The 1 st advanced ML toolkit for Giraph Available as open source Code available at: Documentation: 31

32 The Okapi library Collaborative filtering Alternating Least Squares Stochastic Gradient Descent Singular Value Decomposition Collaborative Less-is-More (CLiMF) Context-aware recom. (TFMAP) Bayesian Personalized Ranking Popularity Ranking Clustering Affinity propagation Kmeans Graph analytics Clustering coefficient Graph partitioning K-Core PageRank Semi-clustering Shortest distances SybilRank and adding Triangle counting 32

33 What s more Giraph in Action The 1 st book for Giraph First steps with Giraph Build applications Integrate with other tools More! More details: 33

34 Apache Giraph for applications in Machine Learning & Recommendation Systems Maria Novartis Züri Machine Learning Meetup #5 June 16, 2014