CS246: Mining Massive Datasets Jure Leskovec, Stanford University

Size: px

Start display at page:

Download "CS246: Mining Massive Datasets Jure Leskovec, Stanford University"

Berniece Thornton
5 years ago
Views:

1 CS6: Mining Massive Datasets Jure Leskovec, Stanford University

2 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets Training data 100 million ratings, 80,000 users, 17,770 movies 6 years of data: Test data Last few ratings of each user (.8 million) Evaluation criterion: Root Mean Square Error (RMSE) Netflix Cinematch RMSE: Competition 700+ teams $1 million prize for 10% improvement on Cinematch

3 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 17,700 movies 80,000 users

4 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 80,000 users 17,700 movies ??? 1?? 1 Test Data Set SSE = r uu r uu (i,u) R

/6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 5 Basically the winner of the Netflix Challenge Multi-scale modeling of the data: Combine top level, regional modeling of the

5 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 5 Basically the winner of the Netflix Challenge Multi-scale modeling of the data: Combine top level, regional modeling of the data, with a refined, local view: Global: Overall deviations of users/movies Factorization: Addressing regional effects CF (k-nn): Extract local patterns Global effects Factorization CF/NN

Baseline estimation: Joe will rate The Sixth Sense stars Local neighborhood (CF/NN):

6 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 6 Global: Mean movie rating:.7 stars The Sixth Sense is 0.5 stars above avg. Joe rates 0. stars below avg. Baseline estimation: Joe will rate The Sixth Sense stars Local neighborhood (CF/NN): Joe didn t like related movie Signs Final estimate: Joe will rate The Sixth Sense.8 stars

7 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 7 Earliest and most popular collaborative filtering method Derive unknown ratings from those of similar movies (item-item variant) Define similarity measure s ij of items i and j Select k-nearest neighbors, compute the rating N(i; u): items most similar to i that were rated by u rˆ ui = j N ( i; u) s j N ( i; u) ij s r ij uj s ij similarity of items i and j r uj rating of user u on item j N(i;u) set of similar items

8 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 8 In practice we get better estimates if we model deviations: ^ r ui ( ) s r b (; ) ij uj uj = b + j N iu ui s j N(; iu) ij baseline estimate for r ui μ = overall mean rating b u = rating deviation of user u = avg. rating of user u μ b i = avg. rating of movie i μ Problems: 1) Similarity measures are arbitrary ) Pairwise similarities neglect interdependencies among neighbors ) Taking a weighted average is restricting

9 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 9 Use a weighted sum rather than weighted avg.: ( ) ^ r = b + w j N(; iu) r b How to set w ij? ui ui ij uj uj Allow: Remember, error metric is SSE: r uu r uu (i,u) R Find w ij that minimize SSE on training data! min v r vv b vi + w ii r vj b vj w j N i;v Models relationships between item i and its neighbors j w ij can be learnt through gradient decent based on u and all other users v that rated i j N(; iu) w ij 1

10 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 10 Find w ij that minimize SSE on training data! min r vv b vi + w ii r vj b vj w v Gradient decent j N i;v Iterate until convergence: w w - η w Where: η learning rate w ii = r vv b vv + w ik r vk b vk v k N i;v for j N i; v (else w ii = 0) r vv b vv

11 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 11 So far: Weights w ij derived based on their role; no use of an arbitrary similarity measure (w ij s ij ) Explicitly account for interrelationships among the neighboring movies Next: Latent factor model Extract regional correlations ( ) r = b + w r b ^ ui ui j N(; iu) ij uj uj Global effects Factorization CF/NN

12 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 1 The Color Purple serious Amadeus Braveheart Geared towards females Sense and Sensibility Ocean s 11 Lethal Weapon Geared towards males The Lion King The Princess Diaries funny Independence Day Dumb and Dumber

13 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 1 SVD on Netflix data: R Q P T users items P T Q For now let s assume we can approximate the rating matrix R as a product of thin Q P T There are important differences between SVD and the real SVD. We will get to them later

14 How to estimate the missing rating? items users? r ii = q i p u T 5 1 items users P T Q /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 1

15 How to estimate the missing rating? items users? r ii = q i p u T 5 1 items users P T Q /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 15

16 How to estimate the missing rating? items users. 1 5 r ii = q i p u T 5 1 items users P T Q /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 16

17 The Color Purple serious Amadeus Braveheart Geared towards females Sense and Sensibility Ocean s 11 Lethal Weapon Factor 1 Geared towards males The Lion King The Princess Diaries Factor funny Independence Day Dumb and Dumber /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 17

18 The Color Purple serious Amadeus Braveheart Geared towards females Sense and Sensibility Ocean s 11 Lethal Weapon Factor 1 Geared towards males The Lion King The Princess Diaries Factor funny Independence Day Dumb and Dumber /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 18

19 Remember SVD: A: Input data matrix U: Left singular vecs V: Right singular vecs Σ: Singular values U SVD gives minimum reconstruction error (MSE!) min U,V,Σ ii m A ii UΣV T ii So in our case, SVD on Netflix data: R Q P T A = R, Q = U, P T = Σ V T r ii = q i pt u But, we are not done yet! R has missing entries! n A /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 19 m Σ The sum goes over all entries. Our A/R has missing entries! n V T

20 [Bellkor Team] /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets items SVD isn t defined when entries are missing Use specialized methods to find P, Q min r ii q i pt u P,Q i,u R r ii = q i pt u Don t require cols of P, Q to be orthogonal/unit length P, Q map users/movies to a latent space Q The most popular model among Netflix contestants users P T

21 Want to minimize SSE for test data Idea: Minimize SSE on Training data Want large f (# of factors) to capture all the signals But, test SSE begins to rise for f > Regularization is needed Allow rich model where there are sufficient data Shrink aggressively where data are scarce min P, Q training ( r ui q error λ regularization parameter i p T u ) + λ /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 1 u p u + length i q i ??? 1?? 1

22 The Color Purple serious Amadeus Braveheart Geared towards females Sense and Sensibility Ocean s 11 Lethal Weapon Factor 1 Geared towards males min P, Q training ( r ui q p The Princess Diaries i T u ) + λ min factors error + λ length u p u + i q i The Lion King Factor funny Independence Day Dumb and Dumber /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets

23 The Color Purple serious Amadeus Braveheart Geared towards females Sense and Sensibility Ocean s 11 Lethal Weapon Factor 1 Geared towards males min P, Q training ( r ui q p The Princess Diaries i T u ) + λ min factors error + λ length u p u + i q i The Lion King Factor funny Independence Day Dumb and Dumber /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets

24 The Color Purple serious Amadeus Braveheart Geared towards females Sense and Sensibility Ocean s 11 Lethal Weapon Factor 1 Geared towards males min P, Q training ( r ui q p The Princess Diaries i T u ) + λ min factors error + λ length u p u + i q i The Lion King Factor funny Independence Day Dumb and Dumber /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets

25 The Color Purple serious Amadeus Braveheart Geared towards females Sense and Sensibility Ocean s 11 Lethal Weapon Factor 1 Geared towards males min P, Q training ( r ui q p The Princess Diaries i T u ) + λ min factors error + λ length u p u + i q i The Lion King Factor funny Independence Day Dumb and Dumber /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 5

26 Want to find matrices P and Q: T + + ( rui qi pu ) λ pu qi, Q training u i Online stochastic gradient decent: Initialize P and Q (random, using SVD) Then iterate over the ratings and update factors: For each r ui : min P T ε uu = r uu q i p u q i q i + η ε uu p u λ q i p u p u + η ε uu q i λ p u (derivative of the error ) (update equation) (update equation) η learning rate /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 6

27 Koren, Bell, Volinksy, IEEE Computer, 009 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 7

/6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 8 user bias movie bias user-movie interaction Baseline predictor Separates users and movies Benefits from insights into user s behavior Among

28 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 8 user bias movie bias user-movie interaction Baseline predictor Separates users and movies Benefits from insights into user s behavior Among the main practical contributions of the competition User-Movie interaction Characterizes the matching between users and movies Attracts most research in the field Benefits from algorithmic and mathematical innovations μ = overall mean rating b u = bias of user u b i = bias of movie i

of other ratings user gave recently (day-specific mood, anchoring, multi-user accounts) (Recent)

29 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 9 We have expectations on the rating by user u of movie i, even without estimating u s attitude towards movies like i Rating scale of user u Values of other ratings user gave recently (day-specific mood, anchoring, multi-user accounts) (Recent) popularity of movie i Selection bias; related to number of ratings user gave on the same day ( frequency )

30 r uu = μ + b u + b i + q i p u T Overall mean rating Bias for user u Bias for movie i User-Movie interaction Example: Mean rating: µ =.7 You are a critical reviewer: your ratings are 1 star lower than the mean: b u = -1 Star Wars gets a mean rating of 0.5 higher than average movie: b i = Predicted rating for you on Star Wars: = =. /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 0

31 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 1 Solve: min Q, P ( u, i) R ( T r ( + b + b + q p )) µ ui + λ λ Is typically selected via grid-search on a validation set u i i goodness of fit ( ) q + p + b + b i regularization Stochastic gradient decent to find parameters Note: Both biases (b u, b i ) as well as interactions (q i, p u ) are treated as parameters (we estimate them) u u u i

32 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets Global average: User average: Movie average: 1.05 Netflix: Basic Collaborative filtering: 0.9 Collaborative filtering++: 0.91 Latent factors: 0.90 Latent factors+biases: 0.89 Final BellKor: Grand Prize: 0.856

33 CF (no time bias) Basic Latent Factors Latent Factors w/ Biases 0.91 RMSE Millions of parameters /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets

changed Movie age Users prefer new movies without any reasons Older movies are just

35 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 5 Sudden rise in the average movie rating (early 00) Improvements in Netflix GUI improvements Meaning of rating changed Movie age Users prefer new movies without any reasons Older movies are just inherently better than newer ones Y. Koren, Collaborative filtering with temporal dynamics, KDD 09

36 Original model: r ui = µ +b u + b i + q i p u T Add time dependence to biases: r ui = µ +b u (t)+ b i (t) +q i p u T Make parameters b u and b i to depend on time (1) Parameterize time-dependence by linear trends () Each bin corresponds to 10 consecutive weeks Add temporal dependence to factors p u (t) user preference vector on day t Y. Koren, Collaborative filtering with temporal dynamics, KDD 09 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 6

37 RMSE CF (no time bias) Basic Latent Factors CF (time bias) Latent Factors w/ Biases + Linear time factors + Per-day user biases + CF Millions of parameters /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 7

38 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 8 Many options for modeling Variants of the ideas we have seen so far Different numbers of factors Different ways to model time Different ways to handle implicit information Other models (not described here) Nearest-neighbor models Restricted Boltzmann machines Model averaging is useful. Linear model combining

39 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 9

40 June 6 th submission triggers 0-day last call /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 0

41 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 1 Ensemble team formed Group of other teams on leaderboard forms a new team Relies on combining their models Quickly also get a qualifying score over 10% BellKor Continue to get small improvements in their scores Realize that they are in direct competition with Ensemble Strategy Both teams carefully monitoring the leaderboard Only sure way to check for improvement is to submit a set of predictions This alerts the other team of your latest score

42 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets Submissions limited to 1 a day Only 1 final submission could be made in the last h hours before deadline BellKor team member in Austria notices (by chance) that Ensemble posts a score that is slightly better than BellKor s Frantic last hours for both teams Much computer time on final optimization run times carefully calibrated to end about an hour before deadline Final submissions BellKor submits a little early (on purpose), 0 mins before deadline Ensemble submits their final entry 0 mins later.and everyone waits.

43 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets

44 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets

45 /6/01 Jure Leskovec, Stanford C6: Mining Massive Datasets 5 Some slides and plots borrowed from Yehuda Koren, Robert Bell and Padhraic Smyth Further reading: Y. Koren, Collaborative filtering with temporal dynamics, KDD

CS246: Mining Massive Datasets Jure Leskovec, Stanford University

CS246: Mining Massive Datasets Jure Leskovec, Stanford University CS6: Mining Massive Datasets Jure Leskovec, Stanford University http://cs6.stanford.edu Training data 00 million ratings, 80,000 users, 7,770 movies 6 years of data: 000 00 Test data Last few ratings of