Understanding and Recommending Podcast Content

Size: px

Start display at page:

Download "Understanding and Recommending Podcast Content"

Adela McKenzie
5 years ago
Views:

1 Understanding and Recommending Podcast Content Longqi Yang Computer Science Ph.D. Candidate Twitter Funders: 1

2 Collaborators 2

3 Why Podcast 3

4 Emerging Interfaces for Podcast Content Consumption 2

5 5

6 What s special about podcasts (content) the architecture of the podcast is the precise antidote for the flaws of the present. It is deep where now is shallow. It is insulated from ads where now is completely vulnerable. It is a chance for thinking and reflection; it has an attention span an order of magnitude greater than the Tweet. It is an opportunity for serious (and playful) engagement. It is healthy eating for a brain-scape that now gorges on fast food. --- Lawrence Lessig (Professor of Law at Harvard Law School) 6

7 What s special about podcasts (content) It turns out, certain things humans can only do well if they do it slowly. Eating, cooking, reflecting, thinking, loving: These are the things we need to pace and pause We should all spread the idea that every healthy mind spends time every week in slow thinking --- Lawrence Lessig (Professor of Law at Harvard Law School) 7

8 What s special about podcasts (user) Past (What you listened before) Future (What you aspire to listen in the future, user intentions and aspirations) 8

9 What s special about podcasts (user) People listened to episodes from subscribed channels (subscription-based consumption) 9

10 Computational Support for Podcasts articles posts Aa music rec. Past search 10

11 Computational Support for Podcasts articles posts Aa Podcast music rec. Past search 11

12 Computational Support for Podcasts articles posts Aa Podcast music rec. Past Podcast search 12

13 Agenda More than Just Words (WSDM 19) Debias Offline Recommendation Evaluation (Recsys 18) Intention Informed Recommendations (Under Review) 13

14 Content == Words Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

15 Podcast Content == Words (itunes Podcast directory) Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

16 Podcast Content > Words Conversational Paralinguistic Musical Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

Podcast Content > Words Conversational Paralinguistic Musical https://podcastfasttrack.

17 Podcast Content > Words Conversational Paralinguistic Musical Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

Our Goal: Modeling Non-textual Characteristics of Podcasts feature representation Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah

18 Our Goal: Modeling Non-textual Characteristics of Podcasts feature representation Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

19 A Naïve Solution MFCC IS09 IS13 Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

20 A Naïve Solution MFCC IS09 IS13 Expected to be sub-optimal Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

Our approach: Unsupervised Representation Learning large unlabeled podcast corpus Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah

21 Our approach: Unsupervised Representation Learning large unlabeled podcast corpus Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

22 Our approach: Unsupervised Representation Learning Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

23 Our approach: Unsupervised Representation Learning Fine-grained variations Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

24 Adversarial Learning-based Podcast Representation (ALPR) vectors sampled from a uniform distribution Generator features (ALPR) Discriminator CE Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

25 Adversarial Learning-based Podcast Representation (ALPR) Train the generator vectors sampled from a uniform distribution features (ALPR) Label=1 (real) Generator Discriminator CE Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

26 Adversarial Learning-based Podcast Representation (ALPR) Train the discriminator and the classifier spectrograms of real podcast audio Label=1 (real) CE Discriminator Generator CE Label=0 (generated) Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

Adversarial Learning-based Podcast Representation (ALPR) The generator 1 8 512 16 256 32 128 64 64 128 x z 32 64

Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin.

27 Adversarial Learning-based Podcast Representation (ALPR) The generator x z fully connected deconv, 5x5 stride 2 deconv, 5x5 stride 2 deconv, 5x5 stride 2 deconv, 5x5 stride 2 Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

pooling D(x) Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin.

28 Adversarial Learning-based Podcast Representation (ALPR) The discriminator conv, 5x5 stride 2 conv, 5x5 stride 2 conv, 5x5 stride 2 conv, 5x5 stride x fully connected 8 32 global average pooling D(x) Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

29 Adversarial Learning-based Podcast Representation (ALPR) Corpus: 88,728 episodes (18,433 channels) Training: 42,370 episodes Evaluation: 46,358 episodes Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

30 Evaluations Attributes Classification (binary) Calm vs. Energetic Humorous vs. Serious Popularity prediction (binary) Top channels on itunes vs. Others Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

31 Attributes Classification How calm or energetic is the audio presentation? calm energetic How humorous or serious is the audio presentation? humorous serious Does the above audio presentation contain men s or women s voices? Men s Women s Both Other Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

32 Attributes Classification 500 negative positive negative positive Count humorous (0) serious (10) calm (0) energetic (10) Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

33 Baselines MFCC IS09 IS13 Autoencoder (AE) and Variational Autoencoder (VAE): AE Discriminator D D x z Generator G VAE µ σ sample Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

34 Attributes Classification Performance Energy Seriousness Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

35 Why does ALPR outperform AE and VAE? Real Adv. VAE AE Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

36 Popularity Prediction Performance Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

37 Popular vs. unpopular channels (Energy score) Longqi Yang, Yu Wang, Drew Dunne, Michael Sobolev, Mor Naaman, and Deborah Estrin. More than just words: Modeling non-textual characteristics of podcasts. In 12th ACM International Conference on Web Search and Data Mining (WSDM),

38 Agenda More than Just Words (WSDM 19) Debias Offline Recommendation Evaluation (Recsys 18) Intention Informed Recommendations (Under Review) 38

39 Offline Evaluation of Recommendation Algorithm user-item interactions (, ) (, ) (, ) recommendation algorithms R rewards Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

40 Offline Evaluation of Recommendation Algorithm user-item interactions (, ) (, ) (, ) recommendation algorithms Pros: Cost effective. Efficient. Iterate faster. R Experiment before deployment. rewards Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

41 Offline Evaluation of Recommendation Algorithm user-item interactions (, ) (, ) (, ) recommendation algorithms Pros: Cost effective. Efficient. Iterate faster. R Experiment before deployment. rewards Cons: The data is Missing-Not-At-Random (MNAR) Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

42 Offline Evaluation procedure item j user i user i interacted with item j Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

43 Offline Evaluation procedure train/test Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

44 Offline Evaluation procedure 1. Train and validate a recommendation model 2. Averaged performance over held-out (user, item) interaction pairs (Average-Over-All) Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

Offline Evaluation procedure Rating-based

Averaged performance over held-out (user, item)

Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge

Unbiased offline recommender evaluation for

45 Offline Evaluation procedure Rating-based recommendation systems Implicit feedback-based recommendation systems 1. Train and validate a recommendation model 2. Averaged performance over held-out (user, item) interaction pairs (Average-Over-All) Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

46 Previous work: Average-Over-All is biased for rating-based recommendation systems, because ratings are MNAR [Marlin et al. 09], [Schnabel et al. 16], [Steck 10], [Steck 11], and [Steck 13] Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

47 Previous work: Average-Over-All is biased for rating-based recommendation systems, because ratings are MNAR [Marlin et al. 09], [Schnabel et al. 16], [Steck 10], [Steck 11], and [Steck 13] Previous work: Average-Over-All is unbiased for implicit feedback-based recommendation systems, because implicit feedback is missing uniformly at random. [Lim 15] Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

48 This work: Average-Over-All is biased for implicit feedbackbased recommendation systems, because implicit feedback is NOT missing uniformly at random. Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

trending recommendation Popularity bias (Users are more likely to be exposed to popular items) Longqi Yang, Yin Cui,

49 This work: Average-Over-All is biased for implicit feedbackbased recommendation systems, because implicit feedback is NOT missing uniformly at random. trending recommendation Popularity bias (Users are more likely to be exposed to popular items) Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

50 A Hypothetical Example Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations) 1 : : 1 Algorithm 1 Performance Algorithm 2 Performance Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

51 A Hypothetical Example Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations) 1 : : 1 Algorithm 1 Performance Algorithm 2 Performance Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

52 A Hypothetical Example Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations) 1 : : 1 Algorithm 1 Performance Algorithm 2 Performance Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

53 A Hypothetical Example Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations) 1 : : 1 Algorithm 1 Performance Algorithm 2 Performance Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

54 A Hypothetical Example Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations) 1 : : 1 Algorithm 1 Performance Algorithm 2 Performance Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

55 A Hypothetical Example Popular Items Long-tail Items # of liked items (over all items) # of liked items (over observations) 1 : : 1 Algorithm 1 Performance Any sensible evaluation Algorithm 2 Performance Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

56 A Hypothetical Example Popular Items Long-tail Items # of liked items (over all items) 1 : 10 # of liked items (over observations) Average-Over- All 10 : 1 Algorithm 1 Performance Algorithm 2 Performance Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

57 <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> Formalize Reward! Item rankings predicted by an algorithm Ideal evaluation: R(Ẑ) = 1 U X u2u 1 S u X i2s u c(ẑu,i) Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

58 <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> Formalize Reward! Item rankings predicted by an algorithm Predicted ranking of item i for user u Ideal evaluation: R(Ẑ) = 1 U X u2u 1 S u X i2s u c(ẑu,i) Items liked by user u among the entire item set scoring metric <latexit sha1_base64="krmqkdady8zbxihq/ornrxywwbe=">aaab9xicdvdlsgmxfm3uv62vqks3wsk4gmamj6m7ghuxfewd2rfk0kwbmkmgjkouof/hxouibv0xd/6n6unq0qmxdufcy733hamjsjvoh5vbw9/y3mpvf3z29/ypiodhbsvsiuklcyzkn0skmmpjs1pnsderbmuhi51wcjn3o3dekir4jz4mjijrinoiyqsndkuwkjspyey0phhqldl2vvop+z507hlnlxtvqzzhr1/uogs7c5tacs1b8b0/fdincdeyiav6rppoiensu8zirnbpfukqnqar6rnkuuxukc2unsezowxhjkqprufc/t6roviparyazhjpsfrtzcw/vf6qo3qquz6kmnc8xbsldgob5xhaizueazy1bgfjza0qj5fewjugciaer0/h/6tt2a5ju9deqvfzxzehj+aunamx+kabrkattaagejyaj/bs3vup1ov1umznwauzy/ad1tsnkxgs5q==</latexit> <latexit sha1_base64="krmqkdady8zbxihq/ornrxywwbe=">aaab9xicdvdlsgmxfm3uv62vqks3wsk4gmamj6m7ghuxfewd2rfk0kwbmkmgjkouof/hxouibv0xd/6n6unq0qmxdufcy733hamjsjvoh5vbw9/y3mpvf3z29/ypiodhbsvsiuklcyzkn0skmmpjs1pnsderbmuhi51wcjn3o3dekir4jz4mjijrinoiyqsndkuwkjspyey0phhqldl2vvop+z507hlnlxtvqzzhr1/uogs7c5tacs1b8b0/fdincdeyiav6rppoiensu8zirnbpfukqnqar6rnkuuxukc2unsezowxhjkqprufc/t6roviparyazhjpsfrtzcw/vf6qo3qquz6kmnc8xbsldgob5xhaizueazy1bgfjza0qj5fewjugciaer0/h/6tt2a5ju9deqvfzxzehj+aunamx+kabrkattaagejyaj/bs3vup1ov1umznwauzy/ad1tsnkxgs5q==</latexit> <latexit sha1_base64="krmqkdady8zbxihq/ornrxywwbe=">aaab9xicdvdlsgmxfm3uv62vqks3wsk4gmamj6m7ghuxfewd2rfk0kwbmkmgjkouof/hxouibv0xd/6n6unq0qmxdufcy733hamjsjvoh5vbw9/y3mpvf3z29/ypiodhbsvsiuklcyzkn0skmmpjs1pnsderbmuhi51wcjn3o3dekir4jz4mjijrinoiyqsndkuwkjspyey0phhqldl2vvop+z507hlnlxtvqzzhr1/uogs7c5tacs1b8b0/fdincdeyiav6rppoiensu8zirnbpfukqnqar6rnkuuxukc2unsezowxhjkqprufc/t6roviparyazhjpsfrtzcw/vf6qo3qquz6kmnc8xbsldgob5xhaizueazy1bgfjza0qj5fewjugciaer0/h/6tt2a5ju9deqvfzxzehj+aunamx+kabrkattaagejyaj/bs3vup1ov1umznwauzy/ad1tsnkxgs5q==</latexit> <latexit sha1_base64="krmqkdady8zbxihq/ornrxywwbe=">aaab9xicdvdlsgmxfm3uv62vqks3wsk4gmamj6m7ghuxfewd2rfk0kwbmkmgjkouof/hxouibv0xd/6n6unq0qmxdufcy733hamjsjvoh5vbw9/y3mpvf3z29/ypiodhbsvsiuklcyzkn0skmmpjs1pnsderbmuhi51wcjn3o3dekir4jz4mjijrinoiyqsndkuwkjspyey0phhqldl2vvop+z507hlnlxtvqzzhr1/uogs7c5tacs1b8b0/fdincdeyiav6rppoiensu8zirnbpfukqnqar6rnkuuxukc2unsezowxhjkqprufc/t6roviparyazhjpsfrtzcw/vf6qo3qquz6kmnc8xbsldgob5xhaizueazy1bgfjza0qj5fewjugciaer0/h/6tt2a5ju9deqvfzxzehj+aunamx+kabrkattaagejyaj/bs3vup1ov1umznwauzy/ad1tsnkxgs5q==</latexit> Reward for (u, i) pair Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

59 <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> Formalize Reward! Item rankings predicted by an algorithm Predicted ranking of item i for user u Ideal evaluation: R(Ẑ) = 1 U X u2u 1 S u X i2s u c(ẑu,i) Items liked by user u among the entire item set scoring metric <latexit sha1_base64="krmqkdady8zbxihq/ornrxywwbe=">aaab9xicdvdlsgmxfm3uv62vqks3wsk4gmamj6m7ghuxfewd2rfk0kwbmkmgjkouof/hxouibv0xd/6n6unq0qmxdufcy733hamjsjvoh5vbw9/y3mpvf3z29/ypiodhbsvsiuklcyzkn0skmmpjs1pnsderbmuhi51wcjn3o3dekir4jz4mjijrinoiyqsndkuwkjspyey0phhqldl2vvop+z507hlnlxtvqzzhr1/uogs7c5tacs1b8b0/fdincdeyiav6rppoiensu8zirnbpfukqnqar6rnkuuxukc2unsezowxhjkqprufc/t6roviparyazhjpsfrtzcw/vf6qo3qquz6kmnc8xbsldgob5xhaizueazy1bgfjza0qj5fewjugciaer0/h/6tt2a5ju9deqvfzxzehj+aunamx+kabrkattaagejyaj/bs3vup1ov1umznwauzy/ad1tsnkxgs5q==</latexit> <latexit sha1_base64="krmqkdady8zbxihq/ornrxywwbe=">aaab9xicdvdlsgmxfm3uv62vqks3wsk4gmamj6m7ghuxfewd2rfk0kwbmkmgjkouof/hxouibv0xd/6n6unq0qmxdufcy733hamjsjvoh5vbw9/y3mpvf3z29/ypiodhbsvsiuklcyzkn0skmmpjs1pnsderbmuhi51wcjn3o3dekir4jz4mjijrinoiyqsndkuwkjspyey0phhqldl2vvop+z507hlnlxtvqzzhr1/uogs7c5tacs1b8b0/fdincdeyiav6rppoiensu8zirnbpfukqnqar6rnkuuxukc2unsezowxhjkqprufc/t6roviparyazhjpsfrtzcw/vf6qo3qquz6kmnc8xbsldgob5xhaizueazy1bgfjza0qj5fewjugciaer0/h/6tt2a5ju9deqvfzxzehj+aunamx+kabrkattaagejyaj/bs3vup1ov1umznwauzy/ad1tsnkxgs5q==</latexit> <latexit sha1_base64="krmqkdady8zbxihq/ornrxywwbe=">aaab9xicdvdlsgmxfm3uv62vqks3wsk4gmamj6m7ghuxfewd2rfk0kwbmkmgjkouof/hxouibv0xd/6n6unq0qmxdufcy733hamjsjvoh5vbw9/y3mpvf3z29/ypiodhbsvsiuklcyzkn0skmmpjs1pnsderbmuhi51wcjn3o3dekir4jz4mjijrinoiyqsndkuwkjspyey0phhqldl2vvop+z507hlnlxtvqzzhr1/uogs7c5tacs1b8b0/fdincdeyiav6rppoiensu8zirnbpfukqnqar6rnkuuxukc2unsezowxhjkqprufc/t6roviparyazhjpsfrtzcw/vf6qo3qquz6kmnc8xbsldgob5xhaizueazy1bgfjza0qj5fewjugciaer0/h/6tt2a5ju9deqvfzxzehj+aunamx+kabrkattaagejyaj/bs3vup1ov1umznwauzy/ad1tsnkxgs5q==</latexit> <latexit sha1_base64="krmqkdady8zbxihq/ornrxywwbe=">aaab9xicdvdlsgmxfm3uv62vqks3wsk4gmamj6m7ghuxfewd2rfk0kwbmkmgjkouof/hxouibv0xd/6n6unq0qmxdufcy733hamjsjvoh5vbw9/y3mpvf3z29/ypiodhbsvsiuklcyzkn0skmmpjs1pnsderbmuhi51wcjn3o3dekir4jz4mjijrinoiyqsndkuwkjspyey0phhqldl2vvop+z507hlnlxtvqzzhr1/uogs7c5tacs1b8b0/fdincdeyiav6rppoiensu8zirnbpfukqnqar6rnkuuxukc2unsezowxhjkqprufc/t6roviparyazhjpsfrtzcw/vf6qo3qquz6kmnc8xbsldgob5xhaizueazy1bgfjza0qj5fewjugciaer0/h/6tt2a5ju9deqvfzxzehj+aunamx+kabrkattaagejyaj/bs3vup1ov1umznwauzy/ad1tsnkxgs5q==</latexit> Reward for user u Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

60 <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> <latexit sha1_base64="zyvwipk9uixn125ap0aibwydfrc=">aaacehicdvdpaxnbgj1nq42xaqzhxoygsaewg3yxuqw81fuf5gckixw7+yuzmjo7zhxbccf/ghf/fs8ewopxj978b5ykewzrbwop977h8f5akokodh8flypdj0+pqs9qz49fvhxvf33sd3lpbfzerni7tmghkgz7jenhslaiolu4sjcftv7ggq2tubmivyetdxmjz1iaewlaf/eruhmlzinn3hgfzaqgzhi64ma4qhlujs30tn4im504iaoeh804jjmo8irqd8j2wqnmueod7xe5rf8cz7konroscpwbrwfbkzvykklhpjyuhrygljdhkacgnlrjeldow996jeoz3ppnio/uvxnr0m6tdoovnddcpfa24r+8uumz9mqttvesgnh/0axunhk+xydn0qigtfiehc8ubrclsoansa7mr/jtlp+f9fvnypnpruy33s9rzafsjj2zil1nxxbbllmpcfazfwu37db4enwl7olv96evyj95wx4g+peb612dha==</latexit> Formalize Reward! Item rankings predicted by an algorithm Predicted ranking of item i for user u Ideal evaluation: R(Ẑ) = 1 U X u2u 1 S u X i2s u c(ẑu,i) Items liked by user u among the entire item set scoring metric <latexit sha1_base64="krmqkdady8zbxihq/ornrxywwbe=">aaab9xicdvdlsgmxfm3uv62vqks3wsk4gmamj6m7ghuxfewd2rfk0kwbmkmgjkouof/hxouibv0xd/6n6unq0qmxdufcy733hamjsjvoh5vbw9/y3mpvf3z29/ypiodhbsvsiuklcyzkn0skmmpjs1pnsderbmuhi51wcjn3o3dekir4jz4mjijrinoiyqsndkuwkjspyey0phhqldl2vvop+z507hlnlxtvqzzhr1/uogs7c5tacs1b8b0/fdincdeyiav6rppoiensu8zirnbpfukqnqar6rnkuuxukc2unsezowxhjkqprufc/t6roviparyazhjpsfrtzcw/vf6qo3qquz6kmnc8xbsldgob5xhaizueazy1bgfjza0qj5fewjugciaer0/h/6tt2a5ju9deqvfzxzehj+aunamx+kabrkattaagejyaj/bs3vup1ov1umznwauzy/ad1tsnkxgs5q==</latexit> <latexit sha1_base64="krmqkdady8zbxihq/ornrxywwbe=">aaab9xicdvdlsgmxfm3uv62vqks3wsk4gmamj6m7ghuxfewd2rfk0kwbmkmgjkouof/hxouibv0xd/6n6unq0qmxdufcy733hamjsjvoh5vbw9/y3mpvf3z29/ypiodhbsvsiuklcyzkn0skmmpjs1pnsderbmuhi51wcjn3o3dekir4jz4mjijrinoiyqsndkuwkjspyey0phhqldl2vvop+z507hlnlxtvqzzhr1/uogs7c5tacs1b8b0/fdincdeyiav6rppoiensu8zirnbpfukqnqar6rnkuuxukc2unsezowxhjkqprufc/t6roviparyazhjpsfrtzcw/vf6qo3qquz6kmnc8xbsldgob5xhaizueazy1bgfjza0qj5fewjugciaer0/h/6tt2a5ju9deqvfzxzehj+aunamx+kabrkattaagejyaj/bs3vup1ov1umznwauzy/ad1tsnkxgs5q==</latexit> <latexit sha1_base64="krmqkdady8zbxihq/ornrxywwbe=">aaab9xicdvdlsgmxfm3uv62vqks3wsk4gmamj6m7ghuxfewd2rfk0kwbmkmgjkouof/hxouibv0xd/6n6unq0qmxdufcy733hamjsjvoh5vbw9/y3mpvf3z29/ypiodhbsvsiuklcyzkn0skmmpjs1pnsderbmuhi51wcjn3o3dekir4jz4mjijrinoiyqsndkuwkjspyey0phhqldl2vvop+z507hlnlxtvqzzhr1/uogs7c5tacs1b8b0/fdincdeyiav6rppoiensu8zirnbpfukqnqar6rnkuuxukc2unsezowxhjkqprufc/t6roviparyazhjpsfrtzcw/vf6qo3qquz6kmnc8xbsldgob5xhaizueazy1bgfjza0qj5fewjugciaer0/h/6tt2a5ju9deqvfzxzehj+aunamx+kabrkattaagejyaj/bs3vup1ov1umznwauzy/ad1tsnkxgs5q==</latexit> <latexit sha1_base64="krmqkdady8zbxihq/ornrxywwbe=">aaab9xicdvdlsgmxfm3uv62vqks3wsk4gmamj6m7ghuxfewd2rfk0kwbmkmgjkouof/hxouibv0xd/6n6unq0qmxdufcy733hamjsjvoh5vbw9/y3mpvf3z29/ypiodhbsvsiuklcyzkn0skmmpjs1pnsderbmuhi51wcjn3o3dekir4jz4mjijrinoiyqsndkuwkjspyey0phhqldl2vvop+z507hlnlxtvqzzhr1/uogs7c5tacs1b8b0/fdincdeyiav6rppoiensu8zirnbpfukqnqar6rnkuuxukc2unsezowxhjkqprufc/t6roviparyazhjpsfrtzcw/vf6qo3qquz6kmnc8xbsldgob5xhaizueazy1bgfjza0qj5fewjugciaer0/h/6tt2a5ju9deqvfzxzehj+aunamx+kabrkattaagejyaj/bs3vup1ov1umznwauzy/ad1tsnkxgs5q==</latexit> Reward for the algorithm <latexit sha1_base64="58zcswwb0/mrbmw4ak6tfutvoiw=">aaacaxicdvdlsgmxfm3uv62vqhvbtbairkpa+3inuhgpylwopwtso21ozjikd5rs6szfcencebf+htv/xvqhqohzhmm593jzjh8razgxdy81mzs3v5bezcwtr6yuzdc3lqxojic60eqbk59budkcokpucbub4kgv4nlvhy38yxswvurohpsxnepeiwqgbucntbjbz3dltzsg2ldsauwqo43ebtjk5lie1ur7rezzvliqfytvr9x7uga0kgdj5mguj63s+3vbiysecixi1jyklmbmgbuuqsewc51yilno8q40hi14cly5gccy0l2ntd4r6ajpwp2+mechtf3qd5mhx6797y3ev7xggkgtozbrncbeyniosbrftud10ly0ifd1hehcbzecii43xkarlenk+epk/ycxxxzb8dni7ra0rsnntsko2smfuiwh5jickdor5i48kcfy7n17j96l9zoztxntnu3ya97bj5yylve=</latexit> <latexit sha1_base64="58zcswwb0/mrbmw4ak6tfutvoiw=">aaacaxicdvdlsgmxfm3uv62vqhvbtbairkpa+3inuhgpylwopwtso21ozjikd5rs6szfcencebf+htv/xvqhqohzhmm593jzjh8razgxdy81mzs3v5bezcwtr6yuzdc3lqxojic60eqbk59budkcokpucbub4kgv4nlvhy38yxswvurohpsxnepeiwqgbucntbjbz3dltzsg2ldsauwqo43ebtjk5lie1ur7rezzvliqfytvr9x7uga0kgdj5mguj63s+3vbiysecixi1jyklmbmgbuuqsewc51yilno8q40hi14cly5gccy0l2ntd4r6ajpwp2+mechtf3qd5mhx6797y3ev7xggkgtozbrncbeyniosbrftud10ly0ifd1hehcbzecii43xkarlenk+epk/ycxxxzb8dni7ra0rsnntsko2smfuiwh5jickdor5i48kcfy7n17j96l9zoztxntnu3ya97bj5yylve=</latexit> <latexit sha1_base64="58zcswwb0/mrbmw4ak6tfutvoiw=">aaacaxicdvdlsgmxfm3uv62vqhvbtbairkpa+3inuhgpylwopwtso21ozjikd5rs6szfcencebf+htv/xvqhqohzhmm593jzjh8razgxdy81mzs3v5bezcwtr6yuzdc3lqxojic60eqbk59budkcokpucbub4kgv4nlvhy38yxswvurohpsxnepeiwqgbucntbjbz3dltzsg2ldsauwqo43ebtjk5lie1ur7rezzvliqfytvr9x7uga0kgdj5mguj63s+3vbiysecixi1jyklmbmgbuuqsewc51yilno8q40hi14cly5gccy0l2ntd4r6ajpwp2+mechtf3qd5mhx6797y3ev7xggkgtozbrncbeyniosbrftud10ly0ifd1hehcbzecii43xkarlenk+epk/ycxxxzb8dni7ra0rsnntsko2smfuiwh5jickdor5i48kcfy7n17j96l9zoztxntnu3ya97bj5yylve=</latexit> <latexit sha1_base64="58zcswwb0/mrbmw4ak6tfutvoiw=">aaacaxicdvdlsgmxfm3uv62vqhvbtbairkpa+3inuhgpylwopwtso21ozjikd5rs6szfcencebf+htv/xvqhqohzhmm593jzjh8razgxdy81mzs3v5bezcwtr6yuzdc3lqxojic60eqbk59budkcokpucbub4kgv4nlvhy38yxswvurohpsxnepeiwqgbucntbjbz3dltzsg2ldsauwqo43ebtjk5lie1ur7rezzvliqfytvr9x7uga0kgdj5mguj63s+3vbiysecixi1jyklmbmgbuuqsewc51yilno8q40hi14cly5gccy0l2ntd4r6ajpwp2+mechtf3qd5mhx6797y3ev7xggkgtozbrncbeyniosbrftud10ly0ifd1hehcbzecii43xkarlenk+epk/ycxxxzb8dni7ra0rsnntsko2smfuiwh5jickdor5i48kcfy7n17j96l9zoztxntnu3ya97bj5yylve=</latexit> Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

61 Formalize Reward! Average-Over-All: ˆRAOA (Ẑ) = 1 U X u2u 1 S u X i2s u c(ẑu,i) Items liked by user u (observed) Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

62 Formalize Bias E O h ˆRAOA (Ẑ) i 6= R(Ẑ) O u,i =1if(u, i) is observed, and O u,i = 0 otherwise O u,i B(1,P u,i ) Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

63 Inverse-Propensity-Scoring (IPS) ˆR AOA (Ẑ) = 1 U X u2u 1 S u X i2s u c(ẑu,i) 1 ˆR IPS (Ẑ P )= U X u2u 1 S u X i2s u c(ẑu,i) P u,i Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

64 Inverse-Propensity-Scoring (IPS) ˆR AOA (Ẑ) = 1 U X u2u 1 S u X i2s u c(ẑu,i) 1 ˆR IPS (Ẑ P )= U X u2u 1 S u X i2s u c(ẑu,i) P u,i E O h ˆRIPS (Ẑ P ) i = R(Ẑ) Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

65 Self-Normalized Inverse-Propensity-Scoring (SNIPS) [Swaminathan et al.15] 1 ˆR IPS (Ẑ P )= U X u2u 1 S u X i2s u c(ẑu,i) P u,i 1 ˆR SNIPS (Ẑ P )= U X u2u 1 P i2s u 1 P u,i X i2s u c(ẑu,i) P u,i Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

66 Estimating Propensity Scores Factor: Popularity bias (Users are more likely to be exposed to popular items) Assumptions: User-independence assumption P u,i = P (O u,i = 1) = P (O,i = 1) = P,i Two-steps assumption P,i = P select,i P interact select,i User preference is not affected by item presentation P interact select,i = P interact,i Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

67 Estimating Propensity Scores Popularity bias model [Steck 11]: ˆP select,i / (n i ) Observed item popularity Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

68 Estimating Propensity Scores Popularity bias model [Steck 11]: ˆP select,i / (n i ) ˆP,i / (n i ) ( +1 2 ) Estimated from known online content serving policy Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

69 Measuring bias in recommender evaluation (Yahoo! music rating dataset) Mean Absolute Error (MAE), Recall Model Average- Over-All! SNIPS (# = 1.5)! SNIPS (# = 2.0)! SNIPS (# = 2.5)! SNIPS (# = 3.0) U-CML A-CML BPR PMF R SNIPS produces significantly lower MAE Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

70 Measuring bias in recommender evaluation (Yahoo! music rating dataset) Mean Absolute Error (MAE), Recall Model Average- Over-All! SNIPS (# = 1.5)! SNIPS (# = 2.0)! SNIPS (# = 2.5)! SNIPS (# = 3.0) The accuracy of recommending popular items is a significant overestimation of the true recommendation performance U-CML A-CML BPR PMF R SNIPS produces significantly lower MAE Longqi Yang, Yin Cui, Yuan Xuan, Chenyang Wang, Serge Belongie, and Deborah Estrin. Unbiased offline recommender evaluation for missing-not-at-random implicit feedback. In 12th ACM Conference on Recommender Systems (Recsys),

71 Agenda More than Just Words (WSDM 19) Debias Offline Recommendation Evaluation (Recsys 18) Intention Informed Recommendations (Under Review) 71

72 2-by-2 Randomized Controlled Trial RS: How do intention-informed recommendations modulate users podcast content choices? Longqi Yang, Michael Sobolev, Yu Wang, Jenny Chen, Drew Dunne, Christina Tsangouri, Nicola Dell, Mor Naaman, and Deborah Estrin. How intention informed recommendations modulate choices: A field study of spoken word content. Under Review,

73 2-by-2 Randomized Controlled Trial RS: How do intention-informed recommendations modulate users podcast content choices? Intended topics-informed Rec. No Rec. (Trending chart) X Intended channels-informed Rec. No Rec. (Subscription-only) Longqi Yang, Michael Sobolev, Yu Wang, Jenny Chen, Drew Dunne, Christina Tsangouri, Nicola Dell, Mor Naaman, and Deborah Estrin. How intention informed recommendations modulate choices: A field study of spoken word content. Under Review,

Main Findings 72% related onboarding subs. Intended topics-informed Rec. 36% related field subs. 24% related field listening Intended channels-informed Rec.

74 Main Findings 72% related onboarding subs. Intended topics-informed Rec. 36% related field subs. 24% related field listening Intended channels-informed Rec. 127% listening exploration Longqi Yang, Michael Sobolev, Yu Wang, Jenny Chen, Drew Dunne, Christina Tsangouri, Nicola Dell, Mor Naaman, and Deborah Estrin. How intention informed recommendations modulate choices: A field study of spoken word content. Under Review,

Main Findings Intended topics-informed Rec. X Intended channels-informed Rec. User satisfaction No Rec. (Trending chart) X Intended channels-informed Rec.

75 Main Findings Intended topics-informed Rec. X Intended channels-informed Rec. User satisfaction No Rec. (Trending chart) X Intended channels-informed Rec. User satisfaction Longqi Yang, Michael Sobolev, Yu Wang, Jenny Chen, Drew Dunne, Christina Tsangouri, Nicola Dell, Mor Naaman, and Deborah Estrin. How intention informed recommendations modulate choices: A field study of spoken word content. Under Review,

76 Longqi Yang Ph.D. candidate Computer Science, Cornell Tech, Cornell University Web: bit.ly/longqi Twitter Github link, documents, and tutorials Connected Experiences Lab Small Data Lab Funders: 76

Collaborative Metric Learning

Collaborative Metric Learning Andy Hsieh, Longqi Yang, Yin Cui, Tsung-Yi Lin, Serge Belongie, Deborah Estrin Connected Experience Lab, Cornell Tech AOL CONNECTED EXPERIENCES LAB CORNELL TECH 1 Collaborative