Deep Reinforcement Learning and the Atari MARC G. BELLEMARE Google Brain

Size: px

Start display at page:

Download "Deep Reinforcement Learning and the Atari MARC G. BELLEMARE Google Brain"

Clyde Walters
5 years ago
Views:

1 Deep Reinforcement Learning and the Atari 2600 MARC G. BELLEMARE Google Brain

2 DEEP REINFORCEMENT LEARNING Inputs Hidden layers Output Backgammon (Tesauro, 1994) Elevator control (Crites and Barto, 1996) Helicopter control (Ng et al., 2003) Transfer in tic-tac-toe (Rivest and Precup, 2003; Bellemare et al., unpublished) Atari game-playing (Mnih et al., 2015) Go (Silver et al., 2016)

3 Deep RL Inputs Hidden layers Output Challenge domains Algorithms

4 SOME CHALLENGES IN DEEP RL Training data is not i.i.d. Hard to get confidence intervals Potential divergence issues (e.g. van Hasselt et al., 2015) State space often unknown Model-free methods reign supreme (simulation a plus)

5 Diuk et al., 2008 Barbados RL Workshop, 2008 Naddaf, 2008 Bellemare et al., MGB, Naddaf, Veness, Bowling 2013 Mnih et al., 2013, 2015 Hausknecht et al Lipovetsky et al. 2015

7 pixels pixels 60 HZ 128 BYTES

8 Observation Action The Arcade Learning Environment (Bellemare et al., 2013)

9 Observation Action Reward The Arcade Learning Environment (Bellemare et al., 2013)

11 GENERAL COMPETENCY CC Wikipedia

12 NARROW COMPETENCY CC Wikipedia

13 Diverse Interesting Independent

14 Diverse We argue that reinforcement learning is particularly vulnerable to environment overfitting and propose as a remedy generalized methodologies [ ] Whiteson et al., 2011

15 Diverse

16 Diverse

17 Interesting (to people) CC Wikipedia

18 Independent (by and for people)

19 EARLY ATTEMPTS ( ) BASS/Basic Features RAM LSH on pixels DISCO: Object Detection

35 Average Baseline Scores 1.4 Median Baseline Scores Median Inter-Algorithm Scores 1 30 1.2 0.8 25 1 20 15 10 5 0 Basic BASS DISCO LSH RAM 0.8 0.6 0.4 0.

20 35 Average Baseline Scores 1.4 Median Baseline Scores Median Inter-Algorithm Scores Basic BASS DISCO LSH RAM Basic BASS DISCO LSH RAM Basic BASS DISCO LSH RAM Fraction of Games SCORE DISTRIBUTION RAM Basic BASS LSH DISCO Inter-Algorithm Score 0.2 0

21 DEEP Q-NETWORKS (DQN) Mnih et al., 2015 Image credit: Volodymyr Mnih

22 GRAYSCALE POLICY POOL + DQN DOWNSAMPLE STACK REPLAY MEMORY TARGET NETWORK LEARNING ALGORITHM T Q = r + P max Q

23 <latexit sha1_base64="o22ij337m584pl9kikl/mk0r0ni=">aaaichicfzvfbxtfemdppdql0jlciy8rrcojpjevqgivvwqouue4wumtne3wwht7e/yqu3fh7rry2e4x4npwhnjlw/dcz2huj5p7u3qsfxpzm9mzndk/qsq4nv3+p51b791+/0737gdrh3507/7h6w8+odxjxff2qhorqloaacz4ze4mn4kdpyorgqj2mrh8lvgxr5nspimpztjly0mmmy84jqzuk/ufpsnnbgbmkc30bgetkukt7+yuq6bnc5phsigym2cv+hlhkzgsiczjymljbsi8rpjuoxq4sdhwetjujoc2fxvbigxsofqiykhcxibqa+ux3cl6r7/tzx/ufvxs6hnlm5o8ulpayulnkswgcql1hd9pzdgsztgvzk3huwypozdkyi5ajilkemzz9b16cjoqrymcx2xqrq16wck1xsoalcuxm91kmfl/mjnjwnsbwskd6uzojvpmbhmczg2lazfsnbfijcjrdaq5ytsijqiekg6zqnrgob0g+lepkayjgdtdxqs5rqpfhozxjrkfr3mqyyitiirvnaiyfq3h7deaql/j0gl6wlmczy0syv8416bnfxrwoq8q9fwl7lxoxosevehriw4qdnci5xv63ql7fbrfoscvetyiwwodumax9gefhzad9wmfnracg4msb4e9apqmtmksa6rsqovna6laypsn3m7mhjyiv4vpjnngvs1qns1iz2g3+4/jlnhdhliiybziyb2i60s/zxidmnibiu2d6mdiljhefwflu2clwzcymrvyjba4nnip7lvbsthobjccnjbyhlznkisyav7zqqphlkb2kh9nhxut6xo2jtyr3rucamie39jd941hecmqhdzan4zuvxc41xlfpbum2jjkmavx06tmf8qkuzd1qghl2wejsgruzfln2ea2x8vhsxtxagvgwh9blfhkeh6dej7miu9n2uejldyo3m0/yuzmsugwn/e5t/mo6pnnbve4hbmnny6xwo9uuga7qmqa5xv7+wqbjse5rdj44+1mepchnky0bgm1ikbpuzgsknb0 r L( ) L( ) LEARNING ALGORITHM L( ) = 1 2 (r + max Q (x 0,a 0 ) Q (x, a)) 2 a 0 2A T Q = r + P max Q

24 Bellemare, Ostrovski, Guez, Thomas, Munos, 2015

25 GRAYSCALE POLICY POOL + DQN DOWNSAMPLE STACK REPLAY MEMORY TARGET NETWORK LEARNING ALGORITHM T Q = r + P max Q

26 LSTM layer (Hausknecht GRAYSCALE and Stone, 2015) Recurrent network (Mnih et al., 2016) POOL + DOWNSAMPLE STACK POLICY DQN REPLAY MEMORY TARGET NETWORK LEARNING ALGORITHM T Q = r + P max Q

27 GRAYSCALE POLICY POOL + DOWNSAMPLE STACK DQN REPLAY MEMORY TARGET NETWORK LEARNING T Q = r + ALGORITHM Dueling networks (Wang et al., 2016) Asynchronous actor-critic (Mnih et al., 2016) P max Q Bootstrap Q-functions (Osband et al., 2016) Noisy networks (Fortunato et al.; Plappert et al., 2017)

28 POOL DOWNSAMPLE GRAYSCALE Prioritized replay (Schaul et al., 2016) Importance sampling (Gruslys et al., 2018) Off-policy + corrections (Gelada et al., in prep.) STACK POLICY DQN REPLAY MEMORY TARGET NETWORK LEARNING ALGORITHM T Q = r + P max Q

29 Double Q-learning GRAYSCALE (van Hasselt et al., 2015) Advantage learning (Bellemare et al., 2015) Q(λ) (Harutyunyan + et al., 2016) POOL DOWNSAMPLE Retrace (Munos et al., 2016) STACK Distributional methods (Bellemare et al., 2017, Dabney et al., 2018, ) DQN POLICY REPLAY MEMORY TARGET NETWORK LEARNING ALGORITHM T Q = r + P max Q

30 1. Distributional reinforcement learning 2. Exploration with pseudo-counts

31 x E R(x) : $-2000 R(x) $ (200) 139

32 R(x 0 )+ R(x 1 )+ 2 R(x 2 )... BELLMAN EQUATION V (x) =E R(x)+ E V (x 0 ) x0 P

33 GROUND TRUTH V (x) =E R(x)+ E V (x 0 ) x0 P IMPLIED MODEL SCHOKNECHT 2002; PARR ET AL., 2008; SUTTON ET AL. 2008

34 <latexit sha1_base64="gehkvpxgpdusdoiecww/ybk0nco=">aaaiihicfzvbb9s2fmflbus875a2j30hanritycwggfdowtoxum3lmmcnhhthi5aubrnrjq0km7lmhrb6/b99k32ukobrutaabbi8/sfnqpdmxv5xonb4n/oru8+/ez2590vel9+9fu3327cuttw4ujsdkjdp5snllhm5we70vz77dssjajxz2/cy5cpf/ooscxd4fgvizyrzbbwkadeg8nz+g98gso+gt9c3+0izokiuy2wz6yanaojzdgzpei/idwjqpcsb6/7fzuzzsezuc/ucl4ukqi80c6ktxbxyddfbyiyces+m2s0qrj3phxspvpfbqiljfo6yoamixfq0jznbdrznvqd7uh2ohbdlhp9q3hgzp3bmfzcuhas0nqnsp3bg0hpdjgau58lpbxqlcl0kszyotqdipiamgykevqqlb6ahhj+guaztephifbqkvxqcqlnqsls44eynotadjoqpjqqrk56+tpe8cbaabbqpkxpwkc6ronkqb6xudx/cq1cjyevqnrojkea1k89pj8lqdamuzjzwges+ypfnurkveorkoou51wqobkweg8h7d0nyxydz2d6oje4/tzkfpm6srr0tejpw/rthb5t0ecv+rxfjyr0qewhftps0bmkpwvr/qrdb9hjcj1u0b0k3uuaxvkhfxzydfbjch23nn2daruuowj6rkykaisqzajlnz4bini6xlvnnl0c4ruyyokn7ocu1vkcjcz2jp00sim9xb6brjb7ktff0kshdpamzptg+gmsjtqu35tcmphcc4izk6xjfifji5dcr5kyhypq54kdbrllgst5eaujwmsxwlnyk2qusnfkrjpy1bausc5tv4leyjyt9dbfe3ixpjldnfx9lynrrucslewyn8g07dh4hkrml6n9iakil+uawpyzl0mz+lcrzz2lm9sc553mbaqhlibze2lycxzfo75dhrl4ulfkhkw/rndsiv423fx/wxkf9ioz48fknto3m7pfaw9onexclixfeseul8a61isr2m0rbjae4ldk4i230tbhhdaxhrba5yoyvuj5slky0avvxm0pwdwngh2ulmh6b3cyxstmw5fhnu6u5ep2z52lco4qu3kztrvjnw17sg0f/th/9qk4tbrwfeubtwnz1mprmfwrnbjolnq56pzv+bvzt7fxhxqfd5/k0ludwueevxu6l/4hb0imka==</latexit> IMPLIED MODEL V (x) =E R(x 0 )+ R(x 1 )+ 2 R(x 2 )+ x 0 = x, a t = E [R(x 0 )] + E [R(x 1 )] + 2 E [R(x 2 )] +.

35 VENESS, BELLEMARE, HUTTER, CHUA, DESJARDINS (2015)

36 <latexit sha1_base64="z3v7rkzg+vlq3h00sc3tusddgcs=">aaaid3icfzxrbts2fmfldqu7dje0wz/tczgjq7oggrumanghqlt66iylmzmmbhrtnsiksolql5j0j4cv9gx7mn0b9nwpsafye+xqkh1dugqwrz7f/4ih5/dijyir3e//07ly9ympr3wvf7rx4+nppv1s8+atkyoxkritgotynnpemcejdqk5fuw0kyyenmavvponlr94w6ticxsslwmbhgqw8ybtose03fxt9aonfcu9i75+hdble0y1wh6fote62kqn/bvohsizeoyk77ux/ve7uwu3t1a/1qpwnccmn9pfw55ymhnwdpev2ulmr7/tzx/ubrhlo+euz3b681qk/zguqhzpkohsy7ef6ikhunmqwlabf4olhj6tgrtdmyihuxotzypdd8dioycw8is0yq1vd0ncpzahb8qq6llqmmv8p6bnyw10y1vsbaorkw4etaypkovmes1cchyc6rjzaigfs8iawekdumlhvojoisruqxnry4pzdilmsgszp3ugmvd8opei/w1pvjmktmhszcmdr2/gip1ky6h55btmn2cg27lriszzlgvq0wo9bdgxffqyrz9u6jmwparqoxydvoigrc8q9kxf9yt0v0wpk/s4rfcqdc9rjksdvvbh01mnkntucvyosux55qdpmzazllhrayytb+orcyltt3i7gbnfivwopljilfvoojplo8yku+m+joxgd7dpgvuuxwf6gw10wgbvsryppl8gukjj+y0ppzkptccymblydba4nmip9cpbtxaucx5ysebg+ebyhl/af+a1nvizvctqlvlvrlbqylxdusr6zbsisyrm2aue+pce/rlaznajf97lvnnxcxzlfhrbocx7dkaq1ezs2o9ysor5hvdycuzplgmbgvnwmd3c5rjonksrxbaaxu7eymecogbfop4lc+kzue00usmj4p36qzxozcpnntmfj51wfjyy3gz6brnapplhxmnf5vl4kzuwg40ry03yjl17hccscdmsmnjjbdt6nxakuwqhtvorw2eocfagzvhsxbj5azjnrkodlwvg/w2o7e3inbwz5ugjtxzaxtezrxbxuc2bqd0y7e64/r338nve4+/lw+u686xzlbplum5957hzozn0thzq/nu50fmic7v7e/ep7p/dvwrplu7p87lte7p//wdfj+fc</latexit> V (x) =E R(x 0 )+ R(x 1 )+ 2 R(x 2 )+ = E Z (x) VENESS, BELLEMARE, HUTTER, CHUA, DESJARDINS (2015)

37 BELLMAN EQUATION (x) =E Z (x) = E R(x)+ E Z (X 0 ).

38 DISTRIBUTIONAL BELLMAN EQUATION Z (x) D = R(x)+ Z (X 0 )

39 Z (x) D = R(x)+ Z (X 0 ) VALUE DISTRIBUTION Z (x 0 1) Z (x) x p(x x) = 0.5 r = 0 r = 1 x 0 1 Z (x 0 2) p(x x) = 0.5 x 0 2 Bellman (1957): Bellman equation for mean Sobel (1982): for variance Engel (2003): for Bayesian uncertainty Azar et al. (2011), Lattimore & Hutter (2012): for higher moments Morimura et al. (2010, 2010b): for densities

40 $150 $300 + $200

41 $450 $300 + $200

42 $450 + $300 $200 +

43 $450 + $300 $200 +

44 Q(x, a) GRAYSCALE p(z x, a) Discrete distribution POLICY POOL + DOWNSAMPLE STACK C51 REPLAY MEMORY TARGET NETWORK LEARNING ALGORITHM T Q = r + P max Q

45 GRAYSCALE POLICY POOL + DOWNSAMPLE STACK ε-greedy w.r.t DQN expected value REPLAY MEMORY TARGET NETWORK LEARNING ALGORITHM T Q = r + P max Q

46 APPROXIMATION PROJECTION

47 1.From x, a, sample a transition: P Z r, X 0,A 0 R(x, a),p( x, a), ( X 0 ) (a) 2.Compute sample backup P Z ˆT Z(x, a) :=r + Z(X 0,A 0 ) (b) 3.Project onto approximation support: R+ P Z ˆT Z(x, a) (c) 4.Update towards projection, i.e. take a KL-minimizing step T Z (d)

50 Mean Median > H.B. > DQN DQN 228% 79% 24 0 DDQN 307% 118% DUEL. 373% 151% PRIOR. 434% 124% PR. DUEL. 592% 172% C51 701% 178% 40 50

54 DISTRIBUTIONAL PERSPECTIVE BELLEMARE ET AL., 2016 QUANTILE REGRESSION DABNEY ET AL., 2017 CRAMER DISTANCE BELLEMARE ET AL., 2017 IMPLICIT QUANTILE NETWORKS DABNEY ET AL., 2018 ANALYSIS OF C51 ROWLAND ET AL., 2017 DISTRIBUTIONAL ACTOR-CRITIC BARTH-MARON ET AL., 2018 AS EXPECTED LYLE ET AL., 2018 PGMRL WORKSHOP, ICML GENERATIVE QUANTILE NETWORKS OSTROVSKI ET AL., 2018 THE RAINBOW AGENT HESSEL ET AL., 2018

55 1. Distributional reinforcement learning 2. Exploration with pseudo-counts

56 Pong Seaquest Montezuma s Revenge September 2017

59 ?

60 <latexit sha1_base64="qbbexo8ugwarbyypna0tfspy9ya=">aaah2xicfzxdbhq3fmchkcyktar62rurkxsgq2inqtskcomirsgiczuqhjd1auxxehat2dnt2wu7mxpru6q3frcephfc0ffp8cxsmh+ukxbnzpn9j318/bwkgmvt73+4dpnkf1evda7fwpvy5ldf31q/fedij3nf2sfnrkkoa6kz4de7nnwidpwqrmqg2kvg9injr94wpxksh5hlysastgmecuomucbrz/bulxrkpnqevgf8j/cusekqzouuutoxiw2enzdomcesywjicth4jdddyhceb/qq2bi/nlnv9jf7+ypahl8axa98hppb1xy4tohcsthqqbqe+f3ujc1rhlpbsju81ywl9jrm2qjmmeimxzyfc4bugideualgfxuue6srlkitlziapsrmppvmof+pmzms9w6dsulin3iy0c9jy+n0blhmi5siuuamqa7ukoqkciiwybcqoiwk0rlrhbqykhqfypqaonntuzvtolbmah7wkfh4hqe6lnkiqfitkidw13dm3tie5jcolayvm4vd2cgr9mwwnehxhr636oskfd2iwxw61al7fbrfoomkhbtosywetohohe606egfhrtodovuz81i6b0k3msg66mkpwofb7sldgk724xnmzil1ltzysuabksbopco95o5hyxcn2case7ylw7v2sloloxk/2hsorulqxatnjw6t/rxl+iawd5ubadclybryhl1wjbszbzaeeyzwgvgcaiburr6na0shcecsw4esczvxm2sm2gbxholdvsyte1z3s4k6ybwnwxkbfa+mziskd250mp3rur4kpadxmdn29mq7mbgeswkfqymzzvbelkaxzn/ppnendybhs1nnyqqevcrzz25zw0pio/mbmqjlicrp7zysmgg/a51frgtn87cr6oupi4+r584ivk2wgixz0wmg3b+pnjcz7t+c7z4hmkjk4vlpfasm5sdjsp9jfokfxqfwjkqhfyzvhhpwz8twssuqrbwk2l4fbxjkmmht88nbryl7hkxahflgbh/ixn3zzedr4z9+ohcpmc/15ldcff5zzupbrz9s <latexit sha1_base64="clsxqiehyqgyrzu7iv0cq+47hlg=">aaaif3icfzxbbhs3eebxarql7s1phtsarixassmy2qjaixqbykrbwtrwzcd2hjucwkw4emfyd0nsqwrmn/mn/zq+fx3tyz+i/9dhxuy9pfla2tk5m5zhdc9birg2g8e/nwsfxp/wrvfmrxsff/lpz59v3vriwmdlrdkrjuwstgkimearozlcchaskezkinil4pyj4y9em6v5hb2adcimkswjhnjkdkimm2/376365d56hhaglszibtfizboupub4tqqkjworhfeztasthdwxkdmepynckqymlocarwjtnrtbsft9rlbuu0fy71aranhadekt1q+usam8dgywmd3sdbyh2ypagl8ipa94xtnbn1z4ftolzjghgmh95g8sm7fegu4fszfwuroe0hmyz2cgrkqypbfzxvj0fzqzfmykfpfbmbbqyynuei0dsjtelhstoex/mboqtejwwskd6kzojvxhynmula2laj5subtixmg1cs24gmklnqiekg6zqnrboiyg2lmpkeyxgdtdxqo5rqpfhoyxjrlnxvnef0va5vwqoquw+gao2g80hhurzsymz6gpbm6ucps8trv0pejpwvrlhb5s0z0k3wnrgwo9anfhhq5b9lrct1t0r0l3wvswqg9bdldcd9nmsfr+be83nfvxhr63ninrgypajpq+cvoywjoqo25504aomoi/wf1mzrmc4xcwzsr37eeh6mwvprb2u/8wcshu4hkluxe4omten7pehwz2pmj7opoveiumvt+u50hqc1swmdnvooyihbu55py6lsucxyjldhpilauufvefjacvqcwgryym9iibp8s6ixunmwkbuncbrxqyp1f28h3lmtsxya6v4c+7avl3s2cxytikbm3zztcftkzxtn/ajfhndubhy9knsyofvgrdz25z28p8o9mnkiyfcozplbysnai/qt0f5stnc/frkcwpwvfbt52jwtfc3ame+7gjz8edzxs25ze7xamzndizcbeufk6nrbczsqxjvrf3rxbyepldkom37jvpfybqmmiqphjfjj+hpfkl7fjzzemwi5tfcansm3t/fsfulmqgjgz78ngl+zc9qjoh4k7ymzdtwzj+dtsfbpv73/ued4tb66b3pfe1d8/zve+9x95p3tg78qj3b+d256vone7v3t+6f3b/yk2vdqqf217t6f79h9rs7nk=</latexit> EXPLORATION Bellman equation: Q(x, a) =r(x, a)+ E max x0 P a 0 2A Q(x0,a 0 ) Exploration bonus (Strehl and Littman, 2008): Q(x, a) =ˆr(x, a) + E x 0 ˆP max a 0 2A Q(x0,a 0 )+p N(x, a)

61 r = -1 safe path transition fn. S The Cliff G optimal path state r = -100 optimal policy reward fn.

62 Most observations experienced once: up to 90% singletons (Blundell et al., 2016)

63 GENERATIVE MODEL Train Generate

64 <latexit sha1_base64="dqatesxjbvsir0w2pjl4l/rzzue=">aaahl3icfzvtb9s2emfvrqu7rnva9dwwn8smat0qbnywyeohys3qpgxluidn0jzimcjqzbmhkzwkozuspkxfbh9s32zhwu700fward DENSITY MODEL Train Query n (x) Assumptions 1. density frequency 2. model quality = generalization

65 n (x) = ˆN n (x) ˆN n (x) = n (x) {ˆn 0 n(x) = ˆN n (x)+1 ˆn n(x) 0 n(x) n (x) Bellemare, Srinivasan, Ostrovski, Schaul, Saxton, Munos, 2016

66 CODING PROBABILITY (DENSITY) n (x) TRAIN 0 n(x) RECODING PROBABILITY

67 THE CTS MODEL n (x) := Q K i=1 i n(x i x <i ) x i,j

69 periods without Pseudo-counts salient events start position pseudo-counts salient event pseudo-counts Frames (1000s)

70 <latexit sha1_base64="qbbexo8ugwarbyypna0tfspy9ya=">aaah2xicfzxdbhq3fmchkcyktar62rurkxsgq2inqtskcomirsgiczuqhjd1auxxehat2dnt2wu7mxpru6q3frcephfc0ffp8cxsmh+ukxbnzpn9j318/bwkgmvt73+4dpnkf1evda7fwpvy5ldf31q/fedij3nf2sfnrkkoa6kz4de7nnwidpwqrmqg2kvg9injr94wpxksh5hlysastgmecuomucbrz/bulxrkpnqevgf8j/cusekqzouuutoxiw2enzdomcesywjicth4jdddyhceb/qq2bi/nlnv9jf7+ypahl8axa98hppb1xy4tohcsthqqbqe+f3ujc1rhlpbsju81ywl9jrm2qjmmeimxzyfc4bugideualgfxuue6srlkitlziapsrmppvmof+pmzms9w6dsulin3iy0c9jy+n0blhmi5siuuamqa7ukoqkciiwybcqoiwk0rlrhbqykhqfypqaonntuzvtolbmah7wkfh4hqe6lnkiqfitkidw13dm3tie5jcolayvm4vd2cgr9mwwnehxhr636oskfd2iwxw61al7fbrfoomkhbtosywetohohe606egfhrtodovuz81i6b0k3msg66mkpwofb7sldgk724xnmzil1ltzysuabksbopco95o5hyxcn2case7ylw7v2sloloxk/2hsorulqxatnjw6t/rxl+iawd5ubadclybryhl1wjbszbzaeeyzwgvgcaiburr6na0shcecsw4esczvxm2sm2gbxholdvsyte1z3s4k6ybwnwxkbfa+mziskd250mp3rur4kpadxmdn29mq7mbgeswkfqymzzvbelkaxzn/ppnendybhs1nnyqqevcrzz25zw0pio/mbmqjlicrp7zysmgg/a51frgtn87cr6oupi4+r584ivk2wgixz0wmg3b+pnjcz7t+c7z4hmkjk4vlpfasm5sdjsp9jfokfxqfwjkqhfyzvhhpwz8twssuqrbwk2l4fbxjkmmht88nbryl7hkxahflgbh/ixn3zzedr4z9+ohcpmc/15ldcff5zzupbrz9s <latexit sha1_base64="clsxqiehyqgyrzu7iv0cq+47hlg=">aaaif3icfzxbbhs3eebxarql7s1phtsarixassmy2qjaixqbykrbwtrwzcd2hjucwkw4emfyd0nsqwrmn/mn/zq+fx3tyz+i/9dhxuy9pfla2tk5m5zhdc9birg2g8e/nwsfxp/wrvfmrxsff/lpz59v3vriwmdlrdkrjuwstgkimearozlcchaskezkinil4pyj4y9em6v5hb2adcimkswjhnjkdkimm2/376365d56hhaglszibtfizboupub4tqqkjworhfeztasthdwxkdmepynckqymlocarwjtnrtbsft9rlbuu0fy71aranhadekt1q+usam8dgywmd3sdbyh2ypagl8ipa94xtnbn1z4ftolzjghgmh95g8sm7fegu4fszfwuroe0hmyz2cgrkqypbfzxvj0fzqzfmykfpfbmbbqyynuei0dsjtelhstoex/mboqtejwwskd6kzojvxhynmula2laj5subtixmg1cs24gmklnqiekg6zqnrboiyg2lmpkeyxgdtdxqo5rqpfhoyxjrlnxvnef0va5vwqoquw+gao2g80hhurzsymz6gpbm6ucps8trv0pejpwvrlhb5s0z0k3wnrgwo9anfhhq5b9lrct1t0r0l3wvswqg9bdldcd9nmsfr+be83nfvxhr63ninrgypajpq+cvoywjoqo25504aomoi/wf1mzrmc4xcwzsr37eeh6mwvprb2u/8wcshu4hkluxe4omten7pehwz2pmj7opoveiumvt+u50hqc1swmdnvooyihbu55py6lsucxyjldhpilauufvefjacvqcwgryym9iibp8s6ixunmwkbuncbrxqyp1f28h3lmtsxya6v4c+7avl3s2cxytikbm3zztcftkzxtn/ajfhndubhy9knsyofvgrdz25z28p8o9mnkiyfcozplbysnai/qt0f5stnc/frkcwpwvfbt52jwtfc3ame+7gjz8edzxs25ze7xamzndizcbeufk6nrbczsqxjvrf3rxbyepldkom37jvpfybqmmiqphjfjj+hpfkl7fjzzemwi5tfcansm3t/fsfulmqgjgz78ngl+zc9qjoh4k7ymzdtwzj+dtsfbpv73/ued4tb66b3pfe1d8/zve+9x95p3tg78qj3b+d256vone7v3t+6f3b/yk2vdqqf217t6f79h9rs7nk=</latexit> <latexit sha1_base64="6r4+sicrfo+xjaz8wmujrcalwqq=">aaaifxicfzxdbts2fmflbqu77kpjdrfeedokpjsrwmoafh0cnkilblisowmspgkng6kobckipjj0z4fv5z5ht7o7ybe73jpsjxb04uqfxqxyis/vf8jdcyjss0khzwdwt+fwbx9+dlt75+o1tz797po76xtfnop4rjic8dim1znhniqighmjtahniqimvrbeepdpm/7ydsgt4ujylbmyszanrca4m2iarp92ulxoswdkh9azm0qvvw8jntipgagwsicbiv1seqqflfsjlfdjfhpl0coiqvlusncgzt5hm7m3gyxjg7mk/ntsqdfuqqegyue/vsbmax1slr6kabo2we8ntgf5q9ont2z0npizttzul6gf87meypcqax3hdhiztkwzwuni1+hcq8l4jzvcbtyjjkgpbr5ssu6jxsdbrpaxgzjbqx6wsa2x0kolzgammywz/h8zm1mb3wyqpqpdimkej8zwrmncqmslkij5sexmsjirxyjme7jebunk4koinzhmosfi1ucmpzekmtnemwwvawwhflenfplvrklljc2klnwcpbx9jubwk49xw0s+pfwfvjjbg2ehfye1rnozcj1r0vcv+qpfdyt0t0wpkvsorycvomzr8wo9b9h9ct1v0emkpw7rvqrds5vj0ocvfnh01qcvetpy9g5k7hn2oombgjil1lzzucube0yhop+w9psx+ywi72aapmjydxmqs0wuwvz23cdrntl96kowoifidadpskchgn+mgn00/ykbmhorb2wptw2hrceu1moxim/s5hilvu5xlrrfozaclrhyprmaxvc4ar4mnj96nldx+tgrrjty17apsvnzrvb4i/b8ro/9gw//mir2ean/3ktxetczibvigwkwzq9wcwnnrjl7euimluua4ydnn8yqdjejyzrlpm57xhsa1yhi3aax7tjseajd6fvsc3fnyjrloo1uiih4v36sscyy4eafcyh8bs713gz7brnaivlxxmnf5vb4kz2uk11u5hrngxv3dsetiqxumnztftz6nxaluwqxtdoro+ekcfzjo3p+xbjzaskvnpvlmp1bgmd3jbg8muzjnwv5ef7x2u5xv7nnm6ndop1u2x1su4ff954my1vrjnpp+drzclznofpe+dezoscod/7tbhs+6tzr/t79o/tn969ceqtt+nzp1j7u3/8brvbt6a==</latexit> EXPLORATION Bellman equation: Q(x, a) =r(x, a)+ E max x0 P a 0 2A Q(x0,a 0 ) Exploration bonus (Strehl and Littman, 2008) Q(x, a) =ˆr(x, a) + E x 0 ˆP max a 0 2A Q(x0,a 0 )+p N(x, a) Pseudo-count bonus Q(x, a) =ˆr(x, a)+ E x 0 ˆP max a 0 2A Q(x0,a 0 )+q ˆN(x)

72 START END DQN + bonus DQN

73 Average Score Training Frames (millions)

74 <latexit sha1_base64="jxoybhwqgddcnoviwxohiaiweek=">aaaiunicfzvtb9s2eidldmu8rnvs7uo+edpajosxwmoadh0cnkihbljiowle3isuqfg0tysuhjlu7dd6c/sn+7k/sk87vdjws1cbtk733b2pxypptwxxptp5p3hv/iefpthofrb5+cmvvvxq69hjcx3nfgvnnbkrgvhem8fddma4ewwwvyxix7al//pvwi/em6v5fj6axzqnjrmhfmqpmadytv7chouaoeptuwfaxdm76ok+ukgglkwjlnjiocukplsbr012er4tkqncksw9s54hzeoe6ugcbrjm143bidzbqvjx8csgiciypd1f2y76ht3svmnjpzoe5fud+b0egplfhv8dx40mmxksmxmkgte9rvznr5m+qc64udby8qfvpxowx0fez5kfhgqi9zxbmzqhjcpwkli8iweatqm9jmn2bwjiinehtysdoyegcdaouvalduq1rq9lpnyl6yoljgaiqyxr/h8ze1ka3szwso90jscz+mloetidgrbslkxrtcatowsnucavo0ysqcbuczgvohoicdxqceuxxtgcg+qwkzwnzacy0py2uqlgpz/qvejzreoljx+ib+kq/ukjwnmwsji+is1o5kajsg/iueihbtqo0bcf+rzgdwr0oezpcvskrrsf2q3rywk9rngjaj2q0dmcpa3rwwi9jkvf0scfffx11ucfel5z9ns59n3bq/pomzi51ltzfs2b+ksbqfsot6s5bznch2casz6wnxnuzvmwtrcz3rdhmtgthlarwmw+hx5fq0r/srltmtibih2b6g9iljhifwdz09hmtmawzmrpoelgwmkls9fxusjhjljkoihe0shvjlqfcpdyldzsbjs1t2mcjdzvrevy5ngsdbdwkcj7ubah77vhcc2q7a7h74fxsu5mwilfpbumluwqwrtikl4l+hmmibouawpbzr6kvcsgiosysza3pc0+qqsrrtaav+7qzgcxvkmtd3my4rtkkefg/ri9l53dftdfzfjmk0w4pr5sqrmtt7papazgxemn81b4lfbs8pzyjzvmk/bhdogwkby6dn64nugfm4sfyq1bwnze/zxkklxs9l+vfm7sa/wegnrbarrjv8vrcseya0egfbg/mu+ymzjlenxvbvvmqgvnp+y5nt33+mfwy25+azwdb5xvnw3hdz47l51fnb5z5tdgdqpxuggmnv7e+lfzan7pto81cp+vndltfpgfpkv+ig==</latexit> CREDIT ASSIGNMENT ISSUES IN EXPLORATION Another important key factor: fast credit assignment apple Q(x t,a t )=q r + (x t,a t )+ " 1 X max Q(x t+1,a 0 ) a 0 2A # +(1 q) i r + (x t+i,a t+i ) i=0 Mixed Monte-Carlo (MMC) update

75 EFFECT OF MIXED MONTE CARLO UPDATE

76 REMOVING EXTRINSIC REWARDS

Breakout Super-human agents Scoring exploit Sub-human agents Space Invaders Asteroids Bowling 81-83: 27% sub-human, 8% exploit 36 games Freeway Ice Hockey Tennis Ms.

78 Breakout Super-human agents Scoring exploit Sub-human agents Space Invaders Asteroids Bowling 81-83: 27% sub-human, 8% exploit 36 games Freeway Ice Hockey Tennis Ms. Pac-Man Assault Asterix Beam Rider Elevator Action Enduro James Bond Seaquest Up N Down Bank Heist Krull Frostbite Private Eye Road Runner 77 Q*Bert Venture Pong Video Pinball 15 games Battlezone Berzerk Boxing Carnival Centipede Crazy Climber Fishing Derby Phoenix Skiing 80 & before: 93% super-human, 7% sub-human Amidar Atlantis Chopper Cmd. Demon Attack Gopher Name this Game Pooyan River Raid Star Gunner Time Pilot Yar s Revenge Zaxxon Tutankham Alien Gravitar Journey Escape Kangaroo Pitfall! Kung-Fu Master H.E.R.O. Montezuma s Revenge 6 games Solaris 84 and after: only 1/6 super-human Double Dunk

79 Deep Reinforcement Learning and the Atari 2600 MARC G. BELLEMARE Google Brain

Slides credited from Dr. David Silver & Hung-Yi Lee

Slides credited from Dr. David Silver & Hung-Yi Lee Review Reinforcement Learning 2 Reinforcement Learning RL is a general purpose framework for decision making RL is for an agent with the capacity to