CS 7: Information Retrieval Recommender Systems : Implementation and Applications Acknowledgements Many slides in this lecture are adapted from Xavier Amatriain (Netflix), Yehuda Koren (Yahoo), and Dietmar Jannach (TU Dortmund), Jimmy Lin, Foster Provost
Recap of Recommendation Approaches Collaborative Content-based Knowledge-based Pros No knowledge-engineering effort, serendipity of results, learns market segments No community required, comparison between items possible Deterministic recommendations, assured quality, no cold-start, can resemble sales dialogue Cons Requires some form of rating feedback, cold start for new users and new items Content descriptions necessary, cold start for new users, no surprises Knowledge engineering effort to bootstrap, basically static, does not react to short-term trends /6/06 CS 7: Information Retrieval. Spring 06
Recap: What works (in real world ) /6/06 CS 7: Information Retrieval. Spring 06
Today: Applications Products: Netflix challenge Information: News recommendation Social: Who to follow Human issues (how recommendations are used) /6/06 CS 7: Information Retrieval. Spring 06
Netflix Challenge /6/06 CS 7: Information Retrieval. Spring 06
Movie rating data Training data Test data user movie score user movie score 6? 96? 7?? 768 7? 76?? 68 8? 9? 7? 6 76 6 69? 6 6 6 8? /6/06 CS 7: Information Retrieval. Spring 06 6
Netflix Prize Training data 00 million ratings 80,000 users 7,770 movies 6 years of data: 000-00 Test data Last few ratings of each user (.8 million) Evaluation criterion: root mean squared error (RMSE) Netflix Cinematch RMSE: 0.9 Competition 700+ teams $ million grand prize for 0% improvement on Cinematch result $0,000 007 progress prize for 8.% improvement /6/06 CS 7: Information Retrieval. Spring 06 7
Overall rating distribution Third of ratings are s Average rating is.68 From TimelyDevelopment.com /6/06 CS 7: Information Retrieval. Spring 06 8
#ratings per movie Avg #ratings/movie: 67 /6/06 CS 7: Information Retrieval. Spring 06 9
#ratings per user Avg #ratings/user: 08 /6/06 CS 7: Information Retrieval. Spring 06 0
Average movie rating by movie count More ratings to better movies From TimelyDevelopment.com /6/06 CS 7: Information Retrieval. Spring 06
Most loved movies Title The Shawshank Redemption Lord of the Rings: The Return of the King The Green Mile Lord of the Rings: The Two Towers Finding Nemo Raiders of the Lost Ark Forrest Gump Lord of the Rings: The Fellowship of the ring The Sixth Sense Indiana Jones and the Last Crusade Avg rating.9..06.60..0.99... Count 78 97 8088 0676 900 76 8076 79 999 07 CS 7: Information Retrieval. Spring 06 /6/06
Important RMSEs Global average:.96 erroneous User average:.06 Personalization Movie average:.0 Cinematch: 0.9; baseline BellKor: 0.869; 8.6% improvement Grand Prize: 0.86; 0% improvement Inherent noise:???? accurate /6/06 CS 7: Information Retrieval. Spring 06
Challenges Size of data Scalability Keeping data in memory Missing data 99 percent missing Very imbalanced Avoiding overfitting Test and training data differ significantly movie #6 /6/06 CS 7: Information Retrieval. Spring 06
The Rules Improve RMSE metric by 0% RMSE= square root of the averaged squared difference between each prediction and the actual rating amplifies the contributions of egregious errors, both false positives ("trust busters") and false negatives ("missed opportunities"). 00 million training ratings provided User, Rating, Movie name, date /6/06 CS 7: Information Retrieval. Spring 06
/6/06 CS 7: Information Retrieval. Spring 06 6
/6/06 CS 7: Information Retrieval. Spring 06 7
Lessons from the Netflix Prize Yehuda Koren The BellKor team (with Robert Bell & Chris Volinsky) movie #868 movie #76 movie #0 /6/06 CS 7: Information Retrieval. Spring 06 8
The BellKor recommender system Use an ensemble of complementing predictors Two, half tuned models worth more than a single, fully tuned model /6/06 CS 7: Information Retrieval. Spring 06 9
Effect of ensemble size 0.89 0.89 0.887 Error - RMSE 0.88 0.88 0.88 0.877 0.87 0.87 0.87 8 9 6 0 #Predictors /6/06 CS 7: Information Retrieval. Spring 06 0
The BellKor recommender system Use an ensemble of complementing predictors Two, half tuned models worth more than a single, fully tuned model But: Many seemingly different models expose similar characteristics of the data, and won t mix well Concentrate efforts along three axes... /6/06 CS 7: Information Retrieval. Spring 06
The three dimensions of the BellKor system The first axis: Multi-scale modeling of the data Combine top level, regional modeling of the data, with a refined, local view: k-nn: Extracting local patterns Factorization: Addressing regional effects Global effects Factorization k-nn /6/06 CS 7: Information Retrieval. Spring 06
Multi-scale modeling st tier Global effects: Mean rating:.7 stars The Sixth Sense is 0. stars above avg Joe rates 0. stars below avg Baseline estimation: Joe will rate The Sixth Sense stars /6/06 CS 7: Information Retrieval. Spring 06
Multi-scale modeling nd tier Factors model: Both The Sixth Sense and Joe are placed high on the Supernatural Thrillers scale Adjusted estimate: Joe will rate The Sixth Sense. stars /6/06 CS 7: Information Retrieval. Spring 06
Multi-scale modeling rd tier Neighborhood model: Joe didn t like related movie Signs Final estimate: Joe will rate The Sixth Sense. stars /6/06 CS 7: Information Retrieval. Spring 06
The three dimensions of the BellKor system The second axis: Quality of modeling Make the best out of a model Strive for: Fundamental derivation Simplicity Avoid overfitting Robustness against #iterations, parameter setting, etc. Optimizing is good, but don t overdo it! global local quality /6/06 CS 7: Information Retrieval. Spring 06 6
The three dimensions of the BellKor system The third dimension will be discussed later... Next: Moving the multi-scale view along the quality axis global local??? quality /6/06 CS 7: Information Retrieval. Spring 06 7
Local modeling through k-nn Earliest and most popular collaborative filtering method Derive unknown ratings from those of similar items (movie-movie variant) A parallel user-user flavor: rely on ratings of like-minded users (not in this talk) /6/06 CS 7: Information Retrieval. Spring 06 8
k-nn 0 9 8 7 6 6 users movies - unknown rating - rating between to /6/06 CS 7: Information Retrieval. Spring 06 9
k-nn 0 9 8 7 6? 6 users movies - estimate rating of movie by user /6/06 CS 7: Information Retrieval. Spring 06 0
k-nn 0 9 8 7 6? 6 users Neighbor selection: Identify movies similar to, rated by user movies /6/06 CS 7: Information Retrieval. Spring 06
k-nn 0 9 8 7 6? 6 users Compute similarity weights: s =0., s 6 =0. movies /6/06 CS 7: Information Retrieval. Spring 06
k-nn 0 9 8 7 6.6 6 users Predict by taking weighted average: (0.*+0.*)/(0.+0.)=.6 movies /6/06 CS 7: Information Retrieval. Spring 06
Properties of k-nn Intuitive No substantial preprocessing is required Easy to explain reasoning behind a recommendation Accurate? /6/06 CS 7: Information Retrieval. Spring 06
k-nn on the RMSE scale Global average:.96 erroneous User average:.06 Movie average:.0 k-nn 0.96 0.9 Cinematch: 0.9 BellKor: 0.869 Grand Prize: 0.86 Inherent noise:???? accurate /6/06 CS 7: Information Retrieval. Spring 06
k-nn - Common practice. Define a similarity measure between items: s ij. Select neighbors -- N(i;u): items most similar to i, that were rated by u. Estimate unknown rating, r ui, as the weighted average: s r b ij uj uj r ui b ui j N ( i; u) j N ( i; u) s ij baseline estimate for r ui /6/06 CS 7: Information Retrieval. Spring 06 6
k-nn - Common practice. Define a similarity measure between items: s ij. Select neighbors -- N(i;u): items similar to i, rated by u. Estimate unknown rating, r ui, as the weighted average: Problems: r ui b ui j N ( i; u). Similarity measures are arbitrary; no fundamental justification. Pairwise similarities isolate each neighbor; neglect interdependencies among neighbors. Taking a weighted average is restricting; e.g., when neighborhood information is limited s r b ij uj uj j N ( i; u) s ij /6/06 CS 7: Information Retrieval. Spring 06 7
Interpolation weights Use a weighted sum rather than a weighted average: r b w r b ui ui j N ( i; u) ij uj uj (We allow w ) j N ( i; u) ij Model relationships between item i and its neighbors Can be learnt through a least squares problem from all other users that rated i: r b w r b Min w v u vi vi j N ( i; u) ij vj vj /6/06 CS 7: Information Retrieval. Spring 06 8
Interpolation weights r b w r b Min w v u vi vi j N ( i; u) ij vj vj Interpolation weights derived based on their role; no use of an arbitrary similarity measure Explicitly account for interrelationships among the neighbors Mostly unknown Challenges: Deal with missing values Avoid overfitting Efficient implementation Estimate inner-products among movie ratings /6/06 CS 7: Information Retrieval. Spring 06 9
Latent factor models serious The Color Purple Amadeus Braveheart Geared towards females Sense and Sensibility Ocean s Lethal Weapon Geared towards males Dave The Princess Diaries The Lion King Independence Day Dumb and Dumber Gus escapist /6/06 CS 7: Information Retrieval. Spring 06 0
Latent factor models users items ~ users. -... -... - -..8 -.... -.9 ~ items -. -..6... -.8..7 -...6..7.. -.9. -..9. -.7.8..7 -. -.6..... -.7. - -.7. A rank- SVD approximation /6/06 CS 7: Information Retrieval. Spring 06
Estimate unknown ratings as inner-products of factors: users items? ~ users. -... -... - -..8 -.... -.9 ~ items -. -..6... -.8..7 -...6..7.. -.9. -..9. -.7.8..7 -. -.6..... -.7. - -.7. A rank- SVD approximation /6/06 CS 7: Information Retrieval. Spring 06
Estimate unknown ratings as inner-products of factors: users items? ~ users. -... -... - -..8 -.... -.9 ~ items -. -..6... -.8..7 -...6..7.. -.9. -..9. -.7.8..7 -. -.6..... -.7. - -.7. A rank- SVD approximation /6/06 CS 7: Information Retrieval. Spring 06
Estimate unknown ratings as inner-products of factors: users items. ~ users. -... -... - -..8 -.... -.9 ~ items -. -..6... -.8..7 -...6..7.. -.9. -..9. -.7.8..7 -. -.6..... -.7. - -.7. A rank- SVD approximation /6/06 CS 7: Information Retrieval. Spring 06
Latent factor models. -.. ~ -. -.. -.7.6...... -. -.8. -..7 -....6...7 -.. -. -.9.8. -. -..9.. -.7.8...7. -. -.6 -.9.. -.7. Properties: SVD isn t defined when entries are unknown use specialized methods Very powerful model can easily overfit, sensitive to regularization Probably most popular model among contestants //006: Simon Funk describes an SVD based method /9/006: Free implementation at timelydevelopment.com /6/06 CS 7: Information Retrieval. Spring 06
Factorization on the RMSE scale Global average:.96 User average:.06 Movie average:.0 factorization 0.9 0.89 Cinematch: 0.9 BellKor: 0.869 Grand Prize: 0.86 Inherent noise:???? /6/06 CS 7: Information Retrieval. Spring 06 6
BellKore Approach User factors: Model a user u as a vector p u ~ N k (, ) Movie factors: Model a movie i as a vector q i ~ N k (γ, Λ) Ratings: Measure agreement between u and i: r ui ~ N(p ut q i, ε ) Maximize model s likelihood: Alternate between recomputing user-factors, moviefactors and model parameters Special cases: Alternating Ridge regression Nonnegative matrix factorization /6/06 CS 7: Information Retrieval. Spring 06 7
Combining multi-scale views Residual fitting global effects factorization Weighted average regional effects k-nn local effects A unified model factorization k-nn /6/06 CS 7: Information Retrieval. Spring 06 8
Localized factorization model Standard factorization: User u is a linear function parameterized by p u r ui = p ut q i Allow user factors p u to depend on the item being predicted r ui = p u (i) T q i Vector p u (i) models behavior of u on items like i /6/06 CS 7: Information Retrieval. Spring 06 9
Results on Netflix Probe set RMSE vs. #Factors 0.9 0.9 RMSE 0.9 0.9 Factorization Localized factorization More accurate 0.9 0.9 0.90 0 0 0 60 80 #Factors /6/06 CS 7: Information Retrieval. Spring 06 0
The third dimension of the BellKor system A powerful source of information: Characterize users by which movies they rated, rather than how they rated A binary representation of the data users movies 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 users movies /6/06 CS 7: Information Retrieval. Spring 06
The third dimension of the BellKor system Great news to recommender systems: Works even better on real life datasets (?) Improve accuracy by exploiting implicit feedback Implicit behavior is abundant and easy to collect: Rental history Search patterns Browsing history... Implicit feedback allows predicting personalized ratings for users that never rated! movie #770 /6/06 CS 7: Information Retrieval. Spring 06
Recommended Stories Signed in, search history enab
Challenges Scale Number of unique visitors in last months: several millions Number of stories within last months: several millions Item Churn News stories change every 0- mins Recency of stories is important Cannot build models ever so often Noisy ratings Clicks treated as noisy positive vote
Approach Content-based vs. Collaborative filtering Collaborative filtering based on co-visitation Content agnostic: Can be applied to other application domains, languages, countries Google s key strength: Huge user base and their data. Could have used content based filtering Focus on algorithms that are scalable
Algorithm Overview Obtain a list of candidate stories For each story: Obtain scores (y, y, y ) Final score = w i *y i User clustering algorithms (Minhash (y ), PLSI (y )) Score ~ number of times this story was clicked by other users in your cluster Story-story co-visitation (y ) Score ~ number of times this story was co-clicked with other stories in your recent click history
Algorithm Overview User clustering done offline as batch process Can be run every day or - times a week Cluster-story counts maintained in real time New users May not be clustered Rely on co-visitation to generate recommendations
Architecture Bigtables Read Stats StoryTable (cluster +covisit counts) Personalization Servers Update Stats User clustering (Offline) (Mapreduce) Rank Request Rank Reply Click Notify Read user profile UserTable (user clusters, click hist) Update profile News Frontend Webserver
User clustering - Minhash Input: User and his clicked stories u u Su { s, s,..., s u m } User similarity = Su S / u S u S u Output: User clusters. Similar users belong to same cluster
Minhash Randomly permute the universe of clicked stories { s MH ( u), s,..., s } { s ', s ',..., s ' m m min( s j u min defined by permutation P{ MH ( u Pseudo-random permutation Compute hash for each story and treat hash-value as permutation index Treat MinHash value as ClusterId Probabilistic clustering ) ) MH ( u )} S S u u } S S u u
MinHash as Mapreduce Map phase: Input: key = userid, value = storyid Compute hash for each story (parallelizable across all data) Output: key = userid value = storyhash Reduce phase: Input: key = userid value = <list of storyhash values> Compute min over the story hash-values (parallelizable across users)
PLSI Framework u s u z s u z u Users s Latent Classes Stories P(s u) = z P(s z)*p(z u)
Clustering - PLSI Algorithm Learning (done offline) ML estimation Learn model parameters that maximize the likelihood of the sample data Ouput = P[z j u] s P[s z j ] s P[z j u] s lead to a soft clustering of users Runtime: we only use P[z j u] s and ignore P[s z j ] s u z s P[z u] P[s z]
PLSI (EM estimation) as Mapreduce E step: (Map phase) q*(z; u,s,ø) = p(s z)p(z u)/( z p(s z)p(z u)) M step: (Reduce phase) p(s z) = u q*(z; u,s,ø)/( s u q*(z; u,s,ø)) p(z u) = s q*(z; u,s,ø)/( z s q*(z; u,s,ø)) Cannot load the entire model from prev iteration in a single machine during map phase Trick: Partition users and stories. Each partition loads the stats pertinent to it
PLSI as Mapreduce
Covisitation For each story s i store the covisitation counts with other stories c(s i, s j ) Candidate story: s k User history: s,, s n score (s i, s j ) = c(s i, s j )/ m c(s i, s m ) total_score(s k ) = n score(s k, s n )
Experimental results
Live traffic clickthrough evaluation
Summary Most effective approaches are often hybrid between collaborative and content-based filtering c.f. Netflix, Google News recommendations Lots of data available, but algorithm scalability critical (use Map/Reduce, Hadoop) Suggested rules-of-thumb (take w/ grain of salt): If have moderate amounts of rating data for each user, use collaborative filtering with knn (reasonable baseline) When little data available, use content (model-) based recommendations /6/06 CS 7: Information Retrieval. Spring 06 7
/6/06 CS 7: Information Retrieval. Spring 06 7
/6/06 CS 7: Information Retrieval. Spring 06 7
/6/06 CS 7: Information Retrieval. Spring 06 7
/6/06 CS 7: Information Retrieval. Spring 06 76
/6/06 CS 7: Information Retrieval. Spring 06 77
/6/06 CS 7: Information Retrieval. Spring 06 78
EXPLAINING RECOMMENDATIONS /6/06 CS 7: Information Retrieval. Spring 06 79
Explanation is Critical /6/06 CS 7: Information Retrieval. Spring 06 80
Explanations in recommender systems Additional information to explain the system s output following some objectives
Objectives of explanations Transparency Validity Trustworthiness Persuasiveness Effectiveness Efficiency Satisfaction Relevance Comprehensibility Education
Explanations in general How? and Why? explanations in expert systems Form of abductive reasoning Given: KB RS i (item i is recommended by method RS) Find KB KB s.t. KB RS i Principle of succinctness Find smallest subset of KB KB s.t. KB RS i i.e. for all KB KB holds KB RS i But additional filtering Some parts relevant for deduction, might be obvious for humans [Friedrich & Zanker, AI Magazine, 0]
Taxonomy for generating explanations in RS Major design dimensions of current explanation components: Category of reasoning model for generating explanations White box Black box RS paradigm for generating explanations Determines the exploitable semantic relations Information categories
RS paradigms and their ontologies Classes of objects Users Items Properties N-ary relations between them Collaborative filtering Neighborhood based CF (a) Matrix factorization (b) Introduces additional factors as proxies for determining similarities
RS paradigms and their ontologies Content-based Properties characterizing items TF*IDF model Knowledge based Properties of items Properties of user model Additional mediating domain concepts
Similarity between items Similarity between users Tags Tag relevance (for item) Tag preference (of user)
Results from testing the explanation feature Explanation +* + ** sign. < %, * sign. < % Perceived Utility Trust +** +** +** +** Positive Usage exp. Recommend others Intention to repeated usage Knowledgeable explanations significantly increase the users perceived utility Perceived utility strongly correlates with usage intention etc.
Probabilistic View (Foster Provost) /6/06 CS 7: Information Retrieval. Spring 06 89
Example /6/06 CS 7: Information Retrieval. Spring 06 90
Consumers Increasingly Concerned About Inferences Made About Them /6/06 CS 7: Information Retrieval. Spring 06 9
Example Result /6/06 CS 7: Information Retrieval. Spring 06 9
Critical Eveidence ( Counter-factural ) Models can be viewed as evidencecombining systems For a specific decision, can ask: What is a minimal set of evidence such that if it were not present, the decision would not have been made? /6/06 CS 7: Information Retrieval. Spring 06 9
Removing Evidence: Cloacking /6/06 CS 7: Information Retrieval. Spring 06 9
How Many Likes Have to Remove? /6/06 CS 7: Information Retrieval. Spring 06 9
Explanations in recommender systems: Summary There are many types of explanations and various goals that an explanation can achieve Which type of explanation can be generated depends greatly on the recommender approach applied Explanations may be used to shape the wishes and desires of customers but are a double-edged sword On the one hand, explanations can help the customer to make wise buying decisions; On the other hand, explanations can be abused to push a customer in a direction which is advantageous solely for the seller As a result a deep understanding of explanations and their effects on customers is of great interest.
Resources Recommender Systems Survey: http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=97 Koren et al. on winning the NetFlix challenge: http://videolectures.net/kdd09_koren_cftd/ http://www.searchenginecaffe.com/009/09/winning- m-netflix-prize-methods.html http://www.recommenderbook.net/ /6/06 CS 7: Information Retrieval. Spring 06 97