Recommender Systems: User Experience and System ssues Joseph A. Konstan University of Minnesota konstan@cs.umn.edu http://www.grouplens.org Scope of Recommenders Purely Editorial Recommenders Content Filtering Recommenders Collaborative Filtering Recommenders Hybrid Recommenders Winter 2003 1 Winter 2003 4 About me Associate Professor of Computer Science & Engineering, Univ. of Minnesota Ph.D. (1993) from U.C. Berkeley GU toolkit architecture Teaching nterests: HC, GU Tools Research nterests: General HC, and... Collaborative nformation Filtering Multimedia Authoring and Systems Visualization and nformation Management nteresting Applications and their Delivery Wide Range of Algorithms Simple Keyword Vector Matches Pure Nearest-Neighbor Collaborative Filtering Machine Learning on Content or Winter 2003 2 Winter 2003 5 A Quick ntroduction What are recommender systems? Tools to help identify worthwhile stuff Filtering interfaces E-mail filters, clipping services Recommendation interfaces Suggestion lists, top-n, offers and promotions Prediction interfaces Evaluate candidates, predicted ratings Classic Collaborative Filtering MovieLens* K-nearest neighbor algorithm Model-free, memory-based implementation ntuitive application, supports typical interfaces *Note newest release uses updated architecture/algorithm Winter 2003 3 Winter 2003 6
CF Classic Compute pairwise corr. Winter 2003 7 Winter 2003 10 Submit Request Recommendations ratings request Winter 2003 8 Winter 2003 11 Store dentify Neighbors ratings find good Neighborhood Winter 2003 9 Winter 2003 12
Select tems; Predict predictions recommendations Neighborhood Winter 2003 13 Hoop Dreams Star Wars Understanding the Computation Pretty Woman Titanic Blimp Rocky XV Joe D A B D?? John A F D F Susan A A A A A A Pat D A C Jean A C A C A Ben F A F Nathan D A A Winter 2003 16 Hoop Dreams Star Wars Understanding the Computation Pretty Woman Titanic Blimp Rocky XV Joe D A B D?? John A F D F Susan A A A A A A Pat D A C Jean A C A C A Ben F A F Nathan D A A Hoop Dreams Star Wars Understanding the Computation Pretty Woman Titanic Blimp Rocky XV Joe D A B D?? John A F D F Susan A A A A A A Pat D A C Jean A C A C A Ben F A F Nathan D A A Winter 2003 14 Winter 2003 17 Hoop Dreams Star Wars Understanding the Computation Pretty Woman Titanic Blimp Rocky XV Joe D A B D?? John A F D F Susan A A A A A A Pat D A C Jean A C A C A Ben F A F Nathan D A A Hoop Dreams Star Wars Understanding the Computation Pretty Woman Titanic Blimp Rocky XV Joe D A B D?? John A F D F Susan A A A A A A Pat D A C Jean A C A C A Ben F A F Nathan D A A Winter 2003 15 Winter 2003 18
Hoop Dreams Star Wars Understanding the Computation Pretty Woman Titanic Blimp Rocky XV Joe D A B D?? John A F D F Susan A A A A A A Pat D A C Jean A C A C A Ben F A F Nathan D A A ML-comedy Winter 2003 19 Winter 2003 22 Hoop Dreams Star Wars Understanding the Computation Pretty Woman Titanic Blimp Rocky XV Joe D A B D?? John A F D F Susan A A A A A A Pat D A C Jean A C A C A Ben F A F Nathan D A A ML-clist Winter 2003 20 Winter 2003 23 ML-home ML-rate Winter 2003 21 Winter 2003 24
ML-search Talk Roadmap ntroduction Application Space Algorithms nfluencing Users Group Recommenders Recommending Research Papers Winter 2003 25 Winter 2003 28 ML-slist Recommender Application Space Dimensions of Analysis Domain Purpose Whose Opinion Personalization Level Privacy and Trustworthiness nterfaces Algorithms Used Winter 2003 26 Winter 2003 29 ML-buddies Domains of Recommendation Content to Commerce News, information, text Products, vendors, bundles Winter 2003 27 Winter 2003 30
Google: Content Example Buy.com customers also bought Winter 2003 31 Winter 2003 34 Half.com OWL Tips Winter 2003 32 Winter 2003 35 Purposes of Recommendation The recommendations themselves Sales nformation Epinions Sienna overview Education of user/customer Build a community of users/customers around products or content Winter 2003 33 Winter 2003 36
ReferralWeb PHOAKS Winter 2003 37 Winter 2003 40 Experts Ordinary phoaks People like you Whose Opinion? Generic Personalization Level Everyone receives same recommendations Demographic Matches a target group Ephemeral Matches current activity Persistent Matches long-term interests Winter 2003 38 Winter 2003 41 Wine.com Expert recommendations Lands End Winter 2003 39 Winter 2003 42
CDNowAlbumadvisorecomendations Brooks Brothers My CDnow Winter 2003 43 Winter 2003 46 Cdnow album advisor Privacy and Trustworthiness Who knows what about me? Personal information revealed dentity Deniability of preferences s the recommendation honest? Biases built-in by operator business rules Vulnerability to external manipulation Winter 2003 44 Winter 2003 47 nterfaces Types of Output Predictions Recommendations Filtering Organic vs. explicit presentation Types of nput Explicit mplicit Winter 2003 45 Winter 2003 48
Collaborative Filtering Algorithms Non-Personalized Summary Statistics K-Nearest Neighbor Dimensionality Reduction Content + Collaborative Filtering Graph Techniques Clustering Classifier Learning s Non-Personalized Good Enough? What happened to my favorite guide? They let you rate the restaurants! What should be done? Personalized guides, from the people who know good restaurants! Winter 2003 49 Winter 2003 52 Zagat Guide Newark Overview Collaborative Filtering Algorithms Non-Personalized Summary Statistics K-Nearest Neighbor user-user item-item Dimensionality Reduction Content + Collaborative Filtering Graph Techniques Clustering Classifier Learning Winter 2003 50 Winter 2003 53 Zagat Guide Detail tem-tem Collaborative Filtering Winter 2003 51 Winter 2003 54
tem-tem Collaborative Filtering tem-tem Matrix Formulation Target item 1 2 u R R R - R 1 2 3 i-1 i m-1 m u R R R - R s i,1 s i,3 s i,i-1 prediction s i,m Winter 2003 55 m 2nd 1st 4th 3rd 5th 5 closest neighbors weighted sum Raw scores for prediction generation regression-based Approximation based on linear regression Winter 2003 58 tem-tem Collaborative Filtering tem-tem Discussion Good quality, in sparse situations Promising for incremental model building Small quality degradation Big performance gain Winter 2003 56 Winter 2003 59 tem Similarities 1 2 3 i j n-1 n 1 2 R - R R u m-1 m R R s i,j =? R R R - Used for similarity computation Winter 2003 57 Collaborative Filtering Algorithms Non-Personalized Summary Statistics K-Nearest Neighbor Dimensionality Reduction Singular Value Decomposition Factor Analysis Content + Collaborative Filtering Graph Techniques Clustering Classifier Learning Winter 2003 60
Dimensionality Reduction Latent Semantic ndexing Used by the R community Worked well with the vector space model Used Singular Value Decomposition (SVD) Main dea Term-document matching in feature space Captures latent association Reduced space is less-noisy Singular Value Decomposition Reduce dimensionality of problem Results in small, fast model Richer Neighbor Network ncremental Update Folding in Model Update Winter 2003 61 Winter 2003 64 R m X n R k = SVD: Mathematical Background U U k m X rk Winter 2003 62 S S k r k X rk V V k k r X n The reconstructed matrix R k = U k.s k.v k is the closest rank-k matrix to the original matrix R. Collaborative Filtering Algorithms Non-Personalized Summary Statistics K-Nearest Neighbor Dimensionality Reduction Content + Collaborative Filtering Graph Techniques Horting: Navigate Similarity Graph Clustering Classifier Learning Rule-nduction Learning Bayesian Belief Networks Winter 2003 65 m x n SVD for Collaborative Filtering 1. Low dimensional representation O(m+n) storage requirement m x k k x n. 2. Direct Prediction Talk Roadmap ntroduction Application Space Algorithms nfluencing Users Cosley et al, CH 2002 Group Recommenders Recommending Research Papers Winter 2003 63 Winter 2003 66
Does Seeing Predictions Affect User? RERATE: Ask 212 users to rate 40 movies 10 with no shown prediction 30 with shown predictions (random order): 10 accurate, 10 up a star, 10 down a star Compare ratings to accurate predictions Prediction is user s original rating Hypothesis: users rate in the direction of the shown prediction Winter 2003 67 % Accuracy Matters 80% 60% 40% 20% 0% Down Accurate Up Prediction manipulation Below At Above Winter 2003 70 The Study Domino Effects? The power to manipulate? Winter 2003 68 Winter 2003 71 % Seeing Matters 80% 60% 40% 20% 0% Not show n Show n Prediction show n? Below At Above Rated, Unrated, Doesn t Matter Recap of RERATE effects: Showing prediction changed 8% of ratings Altering shown prediction changed 12% Similar experiment, UNRATED movies 137 experimental users, 1599 ratings Showing prediction changed 8% of ratings Altering shown prediction changed 14% Winter 2003 69 Winter 2003 72
But Users Notice! Goals Users are often insensitive UNRATED part 2: satisfaction survey Control group: only accurate predictions Experimental predictions accurate, useful? ML predictions overall accurate, useful? Manipulated preds less well liked Surprise: 24 bad = MovieLens worse! Explore group recommender design space See if users would want and use a group recommender, at least for movies Study behavior changes in group members group vs. other users new users via groups vs. other new users Learn lessons about group recommenders Winter 2003 73 Winter 2003 76 Talk Roadmap ntroduction Application Space Algorithms nfluencing Users Group Recommenders O Conner et al, ECSCW 2001 Recommending Research Papers Characteristics of groups public or private many or few Design ssues permanent or ephemeral Formation and evolution of groups joining policy administration and rights Winter 2003 74 Winter 2003 77 Recommending for Groups Problem: People watch movies together Solution: A recommender for groups ssues Group formation, rules, composition Recommender algorithm for groups User interface Design ssues What is a group recommendation? group user vs. combined individuals social good functions Privacy and interface issues control over joining groups withholding and recommendations balancing between info overload and support Winter 2003 75 Winter 2003 78
PolyLens ML-buddies Design choices private, small, administered, invited groups combine individual recs with minimum misery high-information interface with opt-out External invitations added by popular demand Winter 2003 79 Winter 2003 82 Field Test Results and Lessons Users like and use group recommenders groups have value for all members groups can help with outreach to new members Users trade privacy for utility Groups are both permanent and ephemeral Users must be able to find each other Talk Roadmap ntroduction Application Space Algorithms nfluencing Users Group Recommenders Recommending Research Papers McNee et al, CSCW 2002 Winter 2003 80 Winter 2003 83 Subsequent Redesign Citation Webs Move from groups to buddies Hybrid permanent/ephemeral Lower overhead for subgroups Support e-mail based invitation Viral marketing No need for ML logins Remove most privacy settings Buddy model supports person-to-person rather than person-to-group Wonderful network already exists between research papers Papers link to other papers AC in Researchndex (Lawrence, 1999) A citation versus a full paper Citations are pointers to papers Leverage in CF System? Winter 2003 81 Winter 2003 84
CF using Citation Webs Map the citation web directly onto the ratings matrix Full papers are users Citations are items Votes GroupLens Model Winter 2003 85 Winter 2003 88 GroupLens Model GroupLens Model Votes Winter 2003 86 Winter 2003 89 GroupLens Model GroupLens Model Votes Votes Request Winter 2003 87 Winter 2003 90
GroupLens Model Votes Request Winter 2003 91 Winter 2003 94 GroupLens Model Votes Recommendation Request Winter 2003 92 Winter 2003 95 CF using Citation Webs For a full paper, we can recommend citations Every paper has ratings in the system Other citation web mappings are possible, but many are have problems Winter 2003 93 Winter 2003 96
nformal Results Off-line studies showed promise On-line study showed different algorithms met different needs General positive attitude from users Typically one or two useful recommendations in a set of five That s enough to be useful A lot of work remains to be done! Winter 2003 97 Winter 2003 100 Talk Roadmap ntroduction Application Space Algorithms nfluencing Users Group Recommenders Recommending Research Papers Conclusions! Winter 2003 98 Winter 2003 101 Two Grand Challenges Diminishing Marginal Returns Why today s recommenders won t be enough ten years from now Temporal Recommendation Beyond reacting Winter 2003 99 Winter 2003 102
CF Under Diminishing Returns Original goal of CF was to help people sift through the junk to find the good stuff. Today, there may be so much good stuff that you need to sift even more. Wine for the Time Evolving taste can we help a wine newcomer build her palate? Could we identify wines that take her a step or two beyond her current ones? Can we do so by augmenting regular collaborative filtering with temporal models? Certain types of content yield diminishing returns, even with high quality Winter 2003 103 Winter 2003 106 Portfolios of Content What if my recommender knows which articles ve read, and can identify articles by topic? What if it sees that experience marginal returns from reading similar articles on a topic? Could we downgrade some articles based on lack of new content? Could we discover which articles using collaborative filtering? Acknowledgements This work is being supported by grants from the National Science Foundation, and by grants from Net Perceptions, nc. Many people have contributed ideas, time, and energy to this project. Winter 2003 104 Winter 2003 107 Temporal Collaborative Filtering Today s CF systems may expire or degrade ratings, but do little to detect or predict changes in preference. Ripe area with lots of commercial applications Recommender Systems: User Experience and System ssues Joseph A. Konstan University of Minnesota konstan@cs.umn.edu http://www.grouplens.org Winter 2003 105 Winter 2003 108