DRN: A Deep Reinforcement Learning Framework for News Recommendation

Transcription

1 DRN: A Deep Reinforcement Learning Framework for News Recommendation Guanjie Zheng, Fuzheng Zhang, Zihan Zheng, Yang Xiang, Nicholas Jing Yuan, Xing Xie, Zhenhui (Jessie) Li 11/9/18 1

2 Introduction: Why reinforcement recommendation First round Second round Equal rewarding recommendation for current round Images from: /9/

3 Introduction: News recommendation is dynamic The life period for news is usually very short. User s interest may change during time. Ratio of click for di erent categories 100% 80% 60% 40% 20% 0% Auto Business Politics Education Entertainment Military Real estate Technology Week Society Sports Travel Others 11/9/18 3

4 Introduction: Is there more than click/noclick? User s clicks on news are usually very dense in a short period. Then, user usually leave the app! # of request t = 24 hours t = 48 hours User may return everyday! Time interval between two consecutive request (hour) 11/9/18 4

5 Introduction: Should we keep recommending similar items? Lebron James will be the MVP! Tony Parker has come back from injury! Paul Gasol promises to help the Spurs in the playoff. Will you get bored if all the recommended news are from NBA when you are browsing the sports news? Images from: 11/9/18 5

6 Method: Using reinforcement learning to do recommendation Environment State User News... Action 1 Action 2 Action m Action Agent Reward DQN Explore Click / no click User activiness Memory 11/9/18 6

7 t 1 t 2 t 3 t 4 t 5 Timeline Interaction log Candidates Candidates Candidates Candidates Candidates Training Policy Policy Policy Policy Policy Offline Part Push Explore Minor update Push Minor update Push Major update Push Minor update Push Feedback Feedback Feedback Replay Mini-batch Feedback Feedback Activeness analysis Memory Online Part 11/9/18 7

8 Method: Dueling network structure value and advantage function Q(s, a) V(s) A(s, a) User features Context features User-news features News features 11/9/18 8

9 Method: user activeness modeling -- survival analysis User activeness decay function User activeness User activeness t 1 t 2 t 3 t 4 t 5 t 6 t 7 t 8 t 9 t 10 Time 11/9/18 9

10 Method: Effective exploration Current Network Explore Network Step1: get recommendation from! and "! Step2: probabilistic interleave these two lists Step3: get feedback from user and compare the performance of two network Step4: if "! performs better, update! towards it A B C List A C D List Probabilistic Interleave Push to user Collect feedback Keep A C D C D B List Feedback Step towards Model choice 11/9/18 10

11 Dataset # of users # of news # of request # of times pushed to users 11/9/18 11

12 Results: Offline 0.20 LR FM W&D LinUCB HLinUCB DN DDQN DDQN+U DDQN+U+EG DDQN+U+DBGD 0.15 CTR Request sessions 11/9/18 Accuracy 12

13 Results: Online Accuracy Diversity 11/9/18 13

14 Summary of motivation and solution Motivation Long term effect in recommendation Dynamic nature of news recommendation Consider more measures for long term effect Recommendation diversity Solution Deep reinforcement learning (DRL) Online learning feature of DRL Reward functiondesign of DRL Explore in DRL 11/9/18 14

15 Conclusion We propose a reinforcement learning framework to do online personalized news recommendation, taking care of both immediate and future reward. Our framework can be generalized to many other recommendation problems. We consider user activeness to help improve recommendation accuracy, which can provide extra information than simply using user click labels. Our system has been deployed online in a commercial news recommendation application. Extensive offline and online experiments have shown the superior performance of our methods. 11/9/18 15