Linguistic scaffolds for policy learning. Jacob Andreas Berkeley Microsoft Semantic Machines MIT

Size: px

Start display at page:

Download "Linguistic scaffolds for policy learning. Jacob Andreas Berkeley Microsoft Semantic Machines MIT"

Magdalene Caldwell
5 years ago
Views:

1 Linguistic scaffolds for policy learning Jacob Andreas Berkeley Microsoft Semantic Machines MIT

2 Linguistic scaffolds for policy learning Work on language! Jacob Andreas Berkeley Microsoft Semantic Machines MIT

What RL can do for language What language can do for RL Crafting environment replace make plank the last getletter wood of use the word tools drop makehead stick get wood use workb change make cloth

3 What RL can do for language What language can do for RL Crafting environment replace make plank the last getletter wood of use the word tools drop makehead stick get wood use workb change make cloth the final getletter grassto use t i facto add makea rope z if the get last grass character useis tools a make bridge get iron get wood every vowel make bed becomes y get wood use tools change make axe only the get first wood consonant use workb to first make shears & last 3 get letters wood use workb delete get gold every vowel get iron get wood replace get gemall n s get with wood c use workb

4 What RL can do for language Daniel Fried Ronghang Hu Volkan Cirik w/ Anja Rohrbach, L.P. Morency, Taylor Berg-Kirkpatrick, Trevor Darrell and Dan Klein

5 Generation & understanding Turn right and walk through the kitchen. Go right into the living room and stop by the rug. [Anderson et al. 18]

6 A reference game [Frank & Goodman 12]

7 glasses" [Frank & Goodman 12]

8 glasses" [Frank & Goodman 12]

9 glasses" [Frank & Goodman 12]

10 glasses" [Frank & Goodman 12]

11 The rational speech acts model L 0 (. glasses) L 0 (. hat) 1/2 1/2 0 1 [Frank & Goodman 12]

12 The rational speech acts model L 0 (. glasses) L 0 (. hat) 1/2 1/2 0 1 S 1 ( glasses. ) L 0 (. glasses) 1 1/3 S 1 ( hat. ) 0 2/3 [Frank & Goodman 12]

13 The rational speech acts model L 1 (. glasses ) S 1 ( glasses. ) L 1 (. hat ) 3/4 1/4 0 1 S 1 ( glasses. ) L 0 (. glasses) 1 1/3 S 1 ( hat. ) 0 2/3 [Frank & Goodman 12]

14 Pragmatics Q: Do you know what time it is?

15 Pragmatics Q: Do you know what time it is? A: Yes

16 Pragmatics Q: Do you know what time it is? A: Yes I find his cooking very interesting. [Grice 70]

17 RSA game tree speaker hat glasses

18 RSA game tree: as speaker speaker (listener) +1 hat hat glasses glasses -1

19 RSA game tree: as speaker speaker (listener) +1 hat hat glasses glasses -1

20 RSA game tree: as speaker speaker (listener) +1 hat hat glasses glasses -1

21 RSA game tree: as listener (speaker) listener?? glasses glasses?

22 RSA game tree: as listener (speaker) listener? Language use is gameplay!? glasses glasses?

23 A recipe for pragmatic text generation 1. Train a base listener model smiley glasses plain glasses hat & man glasses glasses

24 A recipe for pragmatic text generation 1. Train a base listener model 2. Train a reasoning speaker to win when playing with the listener +1 hat hat glasses glasses -1

25 Application: image captioning 1. Train an image retrieval / gen model a snake is slithering away from Jenny

26 Application: image captioning 2. Describe images using the listener model for search at inference time +1 a snake is slithering away -1 the sun is in the sky -1 [A & Klein 16, Vedantam et al. 17] -1

27 Application: image captioning 2. Describe images using the listener model as a training-time reward ( self-play ) -1 the sun is in the sky captioner model retrieval loss [Yu et al. 16, Mao et al. 16]

28 Descriptive captions [Vedantam et al. 17] seq2seq captioner: this bird has a yellow breast with a short pointy bill pragmatic captioner: a small yellow bird with black stripes on its body and black stripe on the wings.

29 Contrastive captions without contrastive data! (a) (b) (c) Mike is holding a baseball bat. The snake is slithering away from Mike & Jenny. [A & Klein 16]

30 Application: instruction generation 1. Train a base instruction following model 2. Train an instruction generation model to get the follower to goal states

31 Application: instruction generation seq2seq: Walk past the dining room table and chairs and wait there. speaker-listener: Walk past the dining room table and chairs and take a right into the living room. Stop once you are on the rug. human: Turn right and walk through the kitchen. Go right into the living room and stop by the rug. [Fried, Hu, Cirik et al. 18]

32 Listener mode human: Go through the door on the right and continue straight. Stop in the next room in front of the bed. seq-to-seq speaker-listener [Fried, Hu, Cirik et al. 18]

33 The rules of the game +1 glasses glasses

34 The rules of the game +1 hat hat

35 Killer robots [Lewis et al. 17] Bob: i can i i everything else Alice: balls have zero to me to me to me to me to me to me to me to me to Bob: you i everything else Alice: balls have a ball to me to me to me to me to me to me to me

36 Killer robots [Lewis et al. 17] Bob: i can i i everything else Alice: balls have zero to me to me to me to me to me to me to me to me to Bob: you i everything else Alice: balls have a ball to me to me to me to me to me to me to me

37 Problems to work on How do we use tools like self-play and tree search while remaining within the rules of natural language? How do we do efficient search in string- valued action spaces?

38 Problems to work on How do we use tools like self-play and tree search while remaining within the rules of natural language? How do we do efficient search in string- valued action spaces?

39 What language can do for RL w/ Dan Klein and Sergey Levine

40 A crafting game make planks make sticks

41 Learning with sketches get wood get wood use saw use axe

42 [Su$on et al. 99] The options framework

43 Unsupervised option learning +r [Bacon & Precup 16]

44 Learning with intermediate rewards +r +r [Kearns & Singh 02, Kulkarni et al. 16]

45 Segmenting demonstrations Ï [Stolle & Precup 02, Fox & Krishnan et al. 16]

46 Learning from sketches get wood use saw Ï [A, Klein & Levine 17]

47 Modular policies get wood use saw π 1 π 2 get wood use axe π 1 π 3

48 Modular policies get wood use saw π 1 π 2 get wood use axe π 1 π 3

49 Modular policies TURN LEFT π 1 get wood

50 Results: crafting game

51 Results: crafting game Sketches: modular Reward Sketches: joint Unsupervised x 10 6 episodes

52 Results: locomotion

53 Results: locomotion Sketches: modular Reward Sketches: joint Unsupervised x 10 8 Tmesteps

54 Generalization What if I don t get a sketch at test Tme????

55 Generalization What if I don t get a sketch at test Tme? 100 Unsupervised Sketches Training AdaptaTon

56 Moral A little bit of (structured) language goes a long way!

57 Beyond structured sketches Language learning Learning from demonstrations itch itctch emboldens emboldecs dogtrot dogtrot first & last 3 letters loneliness locelicess vein??? [A, Klein & Levine 17]

58 Beyond structured sketches Language learning Learning from demonstrations itch itctch emboldens emboldecs dogtrot dogtrot first & last 3 letters loneliness locelicess vein???

59 Pretraining via language learning wonderful f( ;, ) wonful first & last 3 letters [Branavan et al., 09]

60 Concept learning emboldens vein loneliness L(f( ;, ), ) emboldecs veic locelicess

61 Concept learning emboldens vein loneliness L(f( ;, ), ) emboldecs veic locelicess every vowel becomes i

62 Concept learning emboldens vein loneliness L(f( ;, ), ) emboldecs veic locelicess every vowel becomes i 128.6

63 Concept learning emboldens vein loneliness L(f( ;, ), ) emboldecs veic locelicess every vowel becomes i change consonants to c 52.3

64 Concept learning emboldens vein loneliness L(f( ;, ), ) emboldecs veic locelicess every vowel becomes i change consonants to c 52.3 replace n with c 8.3

65 Prediction (f( ;, ), ) replace n with c

66 Evaluation loonies L(f( ;, ), ) replace n with c

67 Evaluation loonies f( ;, ) loocies replace n with c

68 <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> As multitask learning Pretraining data Training data wonderful wonful emboldens emboldecs glabrous glaous vein veic itch itctch loneliness locelicess arg min L(f( itctch itch ;, )) arg min <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit>??? L(f( veic vein;, )) first & last 3 letters replace n with c [Caruana 97]

69 As inverse reinforcement learning vein replace n with c??? {<latexit sha1_base64="h0oi12vvfczflvoyzoq3lwhxltu=">aaacu3icbvfda9rafj3erxqtbvxrl6flyatlsupbqytib/igwmftczsh3exutmmnk3rmiiwxf6p/rf/fbyfzkhbrhrnonhppnbl30koky8pwyvnv3b5z997g/edbw81hj0dbt45nwwuom17kup+myfakhtmrrmttsimuqcst9pxdp5/8qg1eqb7zzyvxaqslcshboiozxe4w0itcqishbfppkk/yxuungwwfcfupx9no3hmnd9jddf54kkn/3zvwdrbnadp8ajogwagpm6kgtmnnfd2o+jmupgs144d1ztqas5y1ywgctsm+6e0qdwbmhjhktrxnlpw8llbzlsgyerrwnm5aw8eltggrdvbaz2gbcwcvfgjipr+zptuoywheareupt37r6obwphlkbrmrkezrnxk/7r5bfnxcsnuvvtufhvrxktqs9r9bs2erm7l0ghgwri3un4ggrh1fxaw9+h60fjz1f1soqzb6ufnmpk2b6xd156zasknmfof101wvd+nwmn09wb8+hyy5qz5rrbjhetkjtkkh8krmrfofnnb3gtvz3/jc/+7l1epvjd4npjr4de/ad1k1zy=</latexit> l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 83 3u sm 8d Lc ni lp iv fo pg 9/ DP 9R =0 >" AA CA XF ci Zb ND gs xm IF zu a9 O+ br rz 8N ie Ii ys 04 Xa ig c7 Bi sx 3D GS sk cn qt ZG EZ yg hq 6n ma 95 ZB G/ bn yj cb Mx 7u tc wf fi y5 3Y 59 ow U5 bw /z xt jk 7u /e Hs U1 3P Oq Wb 0j a8 Zt W6 am QL Zp LJ YN Ii sb E4 Ad z0 AH pa pa EJ PH Rr K7 o7 19 B9 Sa Fb 5i nm CE kz ti Yi WU sk B1 1k lw b7 wk vt 7g C+ lf Vr Oq Un 1g Sj AL hf CK ad 3D v0 WN OF Gl UG 8w NI gm p0 To ym ag FF RQ QL VY u7 cu nd ov xn CW 7p Mh lg v+ Fd ht tk 05 kl yb Eb PT mv xr /D w6 z0 7M J4 yc Sb Iz gc 4y G/ cg Gd s4 0I pj Ck va cj qa KG V2 x0 Rn FJ Lq RH Mu 9E CW K8 uh c3 xp UQ VM dj g5 RF 4k JS QR Bl WQ rv CL Z3 P/ Nz Lt ha 9d 2X bv /b K7 qe ko nv A6 5L B6 dr go Uf VQ E1 YU ez L0 zv X4 6w 8X cu 8K q4 C8 az +K 7f +F 1W go =c /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 09 a7 hl H1 gk Ks qe RM w7 at LR IV 5O =o >" AA CA Hs ci Vb NF 9b wq HE CX wv tk Lb yl Vs Vp k2 1K rs Ai Jh QC IA Dk Ci xk ab 1V EF c2 dy ta an 2S 7g yq R+ 8v /N L4 5B sx HQ LT LS ea /3 MP Je z7 gr jp /4 nh NE 66 ef uf av 73 Y0 tp Yn Pe rj bv 6J uz cn 5c 2r tw 3z wi EK FW yc su Lx GN 6I ch m4 +l b+ Xb 7T h+ qn uk tv Vp 6g Cm SZ KV cw 6F pk 8v GO gc 0l Up UG LM O9 0O Ln rk xq KY py wt 42 fb V0 84 i8 /d DO My u/ QR vr m9 8N nj Dg P3 /c +e xy zy F0 mj Kh ej R6 DZ Z3 67 pp ka zg TX Ai D3 IO F1 1x 2w Y2 n9 R8 0D kj Gh Es G/ MO 25 xg W1 L1 Vx FW uk Zw Fh je 2c ad tc B4 7K Li GU +G XC Ms FS xh ou KN bk ne 0b Dw FM WL tv 2V px Pw b6 U4 ZA 1s 5K 9v Pn Da 1a vn fy mt th T+ 1J mo ap xi fv yn bp WS N1 9+ g+ dh ID Vr 5x FA Lw 1/ Kb 0z DE /t P7 vi OY S/ aw /P 6m xn Uv GY 9v I1 W2 wd gn bp 2r Pn IV s3 kb 60 b6 O4 5T IL nl Zy Ye Jb ne IZ M9 US ej Dk kf za mk 8c JL 2r va Be cy ah D9 lh s6 Qr D6 f0 HP F5 Eq j8 Ke 9t PY /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 09 a7 hl H1 gk Ks qe RM w7 at LR IV 5O =o >" AA CA Hs ci Vb NF 9b wq HE CX wv tk Lb yl Vs Vp k2 1K rs Ai Jh QC IA Dk Ci xk ab 1V EF c2 dy ta an 2S 7g yq R+ 8v /N L4 5B sx HQ LT LS ea /3 MP Je z7 gr jp /4 nh NE 66 ef uf av 73 Y0 tp Yn Pe rj bv 6J uz cn 5c 2r tw 3z wi EK FW yc su Lx GN 6I ch m4 +l b+ Xb 7T h+ qn uk tv Vp 6g Cm SZ KV cw 6F pk 8v GO gc 0l Up UG LM O9 0O Ln rk xq KY py wt 42 fb V0 84 i8 /d DO My u/ QR vr m9 8N nj Dg P3 /c +e xy zy F0 mj Kh ej R6 DZ Z3 67 pp ka zg TX Ai D3 IO F1 1x 2w Y2 n9 R8 0D kj Gh Es G/ MO 25 xg W1 L1 Vx FW uk Zw Fh je 2c ad tc B4 7K Li GU +G XC Ms FS xh ou KN bk ne 0b Dw FM WL tv 2V px Pw b6 U4 ZA 1s 5K 9v Pn Da 1a vn fy mt th T+ 1J mo ap xi fv yn bp WS N1 9+ g+ dh ID Vr 5x FA Lw 1/ Kb 0z DE /t P7 vi OY S/ aw /P 6m xn Uv GY 9v I1 W2 wd gn bp 2r Pn IV s3 kb 60 b6 O4 5T IL nl Zy Ye Jb ne IZ M9 US ej Dk kf za mk 8c JL 2r va Be cy ah D9 lh s6 Qr D6 f0 HP F5 Eq j8 Ke 9t PY /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 8K Kq ta 3u er Gy +M dz 8Z xl Kl +8 bm =Y >" AA CA 3u ci Vb NF 9b wq HE CX wv Um ub IH ex qq h0 qa cv FI IJ UR Sl xj FA ke qt 02 qj JK 9M am k2 ql 0O ri HN j+ 4/ /3 gw ne xg dd pm tk /5 mf Tj j3 Jt CL D2 8D f6 3l t7 76 2/ jd fy od e8 Ta dr 2H x0 TN p1 jr Jj ye G1 pc JG is Jw Vk uv ZJ Rp UF vk 0E Xv fj a6 /c BU Rt tf s/ ss YJ aw QI eu gb ZH MW nr Zc I6 SU cr QL va 00 Sy sf r8 GK Dj li N3 jb 7t TQ 99 J3 fh 2b jg e8 uy fh yd cw +A 5x jm fy q2 hr mf Kr Fj UD ek 82 9r Rt 0V JN lu zy ld fj KT xm 1a Tm Mj hb ON Dy g3 Rb MA kz Oi kn 92 kt cw rl Yh lx oe Zy 2R lf w4 0a VF ix 7G ad AY 8X ha 4Y 7d AA Sh uz j+ bt uu Yu Oj la qd wu Gt /f Td gs Fj qm G1 2V ZP 1p Sr /P 8p r1 7m J+ 1G UF ct CW yr K7 0a tl bs fv Jo Qn Ky c1 go cb /C Wd sy B9 7A uf wz 2J Kh X4 Zj d9 S3 V8 ra lc tf ME 2I 79 Dw 0p z4 rm tl Aw /o xv Q3 ny 6r RZ IO +2 uh DO 2w UG +G 5Q S2 TE pe XH I5 8B MJ kd jr 5j e5 41 7L 91 5/ P3 e/ ++ KX 6X u3 5B mr E6 /X G8 C3 XT gm == /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t cost function arg min Ê [L(f( ;, ))] <latexit sha1_base64="ehqpi+osq3iuv/rus12ofbsdhee=">aaadhhicnvjni9raeo3erzw6oqthl43dwkzikiig4gxxazysuikzuzadqqvtmwk2x9vdeyy2j6/6f/w13ssr4c/xaictr3fwkwvpxr/qeql63xgvcav9/4fjxrh46fkvravetevbn24odm4dqrkwhke8zep5hipctbq41ujnefxjhdzo8cg+edbmj96hvkis3uplhweo80kkgoo2vdt4owig57koioya6p44hbpt0xosynjhlw4/ow3yvt3zze6u97ugmrqurostal2iy/oiiqztufomre5tppl1ui31vpekg5qht9zcgdz/pp5la8q6z4zepghgg6wucse53v+3hg2g/stvgp4hqq+gpi+damfzzknj6xwlztnqahb4lq4nsc14ho3haouv8boy48zcanjuoen6aojimglns2m/qtoo/bvcqk7umo/tyxzytzlryx/lzrvoh4dgffwtsecrh6v1rnvj2+umizdidba0algutlfkfycba/soppyc7swsx1nd1xvk0kw8z3qlmg6wfp1pzzwsntdytos8ohwwcfxj8obhco9pb+uwuupukjejycoyr16sazil3amdd85h55p72f3ifnw/ry66tl9zm5wj9/sv7qn7sg==</latexit> <latexit sha1_base64="ehqpi+osq3iuv/rus12ofbsdhee=">aaadhhicnvjni9raeo3erzw6oqthl43dwkzikiig4gxxazysuikzuzadqqvtmwk2x9vdeyy2j6/6f/w13ssr4c/xaictr3fwkwvpxr/qeql63xgvcav9/4fjxrh46fkvravetevbn24odm4dqrkwhke8zep5hipctbq41ujnefxjhdzo8cg+edbmj96hvkis3uplhweo80kkgoo2vdt4owig57koioya6p44hbpt0xosynjhlw4/ow3yvt3zze6u97ugmrqurostal2iy/oiiqztufomre5tppl1ui31vpekg5qht9zcgdz/pp5la8q6z4zepghgg6wucse53v+3hg2g/stvgp4hqq+gpi+damfzzknj6xwlztnqahb4lq4nsc14ho3haouv8boy48zcanjuoen6aojimglns2m/qtoo/bvcqk7umo/tyxzytzlryx/lzrvoh4dgffwtsecrh6v1rnvj2+umizdidba0algutlfkfycba/soppyc7swsx1nd1xvk0kw8z3qlmg6wfp1pzzwsntdytos8ohwwcfxj8obhco9pb+uwuupukjejycoyr16sazil3amdd85h55p72f3ifnw/ry66tl9zm5wj9/sv7qn7sg==</latexit> <latexit sha1_base64="ehqpi+osq3iuv/rus12ofbsdhee=">aaadhhicnvjni9raeo3erzw6oqthl43dwkzikiig4gxxazysuikzuzadqqvtmwk2x9vdeyy2j6/6f/w13ssr4c/xaictr3fwkwvpxr/qeql63xgvcav9/4fjxrh46fkvravetevbn24odm4dqrkwhke8zep5hipctbq41ujnefxjhdzo8cg+edbmj96hvkis3uplhweo80kkgoo2vdt4owig57koioya6p44hbpt0xosynjhlw4/ow3yvt3zze6u97ugmrqurostal2iy/oiiqztufomre5tppl1ui31vpekg5qht9zcgdz/pp5la8q6z4zepghgg6wucse53v+3hg2g/stvgp4hqq+gpi+damfzzknj6xwlztnqahb4lq4nsc14ho3haouv8boy48zcanjuoen6aojimglns2m/qtoo/bvcqk7umo/tyxzytzlryx/lzrvoh4dgffwtsecrh6v1rnvj2+umizdidba0algutlfkfycba/soppyc7swsx1nd1xvk0kw8z3qlmg6wfp1pzzwsntdytos8ohwwcfxj8obhco9pb+uwuupukjejycoyr16sazil3amdd85h55p72f3ifnw/ry66tl9zm5wj9/sv7qn7sg==</latexit> <latexit sha1_base64="ehqpi+osq3iuv/rus12ofbsdhee=">aaadhhicnvjni9raeo3erzw6oqthl43dwkzikiig4gxxazysuikzuzadqqvtmwk2x9vdeyy2j6/6f/w13ssr4c/xaictr3fwkwvpxr/qeql63xgvcav9/4fjxrh46fkvravetevbn24odm4dqrkwhke8zep5hipctbq41ujnefxjhdzo8cg+edbmj96hvkis3uplhweo80kkgoo2vdt4owig57koioya6p44hbpt0xosynjhlw4/ow3yvt3zze6u97ugmrqurostal2iy/oiiqztufomre5tppl1ui31vpekg5qht9zcgdz/pp5la8q6z4zepghgg6wucse53v+3hg2g/stvgp4hqq+gpi+damfzzknj6xwlztnqahb4lq4nsc14ho3haouv8boy48zcanjuoen6aojimglns2m/qtoo/bvcqk7umo/tyxzytzlryx/lzrvoh4dgffwtsecrh6v1rnvj2+umizdidba0algutlfkfycba/soppyc7swsx1nd1xvk0kw8z3qlmg6wfp1pzzwsntdytos8ohwwcfxj8obhco9pb+uwuupukjejycoyr16sazil3amdd85h55p72f3ifnw/ry66tl9zm5wj9/sv7qn7sg==</latexit>

70 <latexit sha1_base64="/8rfhipqr2j1mpofacatkwdehou=">aaacz3icbvfdaxqxfm2mx3w0utvhx4llwlzkmsmigg8ufoapfvtw28jmwo5k7mxdmx9nmsosr3z1r/ip/ddmzqdit15iodkn5+bem6ssqpsw/o35167fuhlr63zw5+72vfudnqdhuqwvxxkvzaloetaoryezi4zek0oh5ine4+tsbasff0glrvl8nqsk4xywhcgeb+ooxedxiifa5qjymdra98fzmj2f15bslgu3dfgvbcwn7uqou7vbhycy+tddcecac5ok9n2zsmxatzkwoxvkm+/ytts3pixdybm7pf03vmhambummsq4ujzubxdvdibhjoycxgvrd4akj4pfjrfn0plxorags9b6howvis0oi7jejmc1xgr4gsxx7maboerydmu0dosylgalcqswtgp/dvjitv7libvz9q03tzb8nzavtfyytqkoaomfxz+u1zkakry/rfohkbu5cgc4eq5wyk9batfuhwp2dl0vcj+6vj8qvgbk9ct2i2o6wfp0qzx1s26a0ea4rokjvukutqldz8ppm36uw+qreuzgjcivyjr8iadkrrg38j57r72pf+h/9b/7p9zxfa/3pcsxwv/5b8ay3sy=</latexit> <latexit sha1_base64="/8rfhipqr2j1mpofacatkwdehou=">aaacz3icbvfdaxqxfm2mx3w0utvhx4llwlzkmsmigg8ufoapfvtw28jmwo5k7mxdmx9nmsosr3z1r/ip/ddmzqdit15iodkn5+bem6ssqpsw/o35167fuhlr63zw5+72vfudnqdhuqwvxxkvzaloetaoryezi4zek0oh5ine4+tsbasff0glrvl8nqsk4xywhcgeb+ooxedxiifa5qjymdra98fzmj2f15bslgu3dfgvbcwn7uqou7vbhycy+tddcecac5ok9n2zsmxatzkwoxvkm+/ytts3pixdybm7pf03vmhambummsq4ujzubxdvdibhjoycxgvrd4akj4pfjrfn0plxorags9b6howvis0oi7jejmc1xgr4gsxx7maboerydmu0dosylgalcqswtgp/dvjitv7libvz9q03tzb8nzavtfyytqkoaomfxz+u1zkakry/rfohkbu5cgc4eq5wyk9batfuhwp2dl0vcj+6vj8qvgbk9ct2i2o6wfp0qzx1s26a0ea4rokjvukutqldz8ppm36uw+qreuzgjcivyjr8iadkrrg38j57r72pf+h/9b/7p9zxfa/3pcsxwv/5b8ay3sy=</latexit> <latexit sha1_base64="/8rfhipqr2j1mpofacatkwdehou=">aaacz3icbvfdaxqxfm2mx3w0utvhx4llwlzkmsmigg8ufoapfvtw28jmwo5k7mxdmx9nmsosr3z1r/ip/ddmzqdit15iodkn5+bem6ssqpsw/o35167fuhlr63zw5+72vfudnqdhuqwvxxkvzaloetaoryezi4zek0oh5ine4+tsbasff0glrvl8nqsk4xywhcgeb+ooxedxiifa5qjymdra98fzmj2f15bslgu3dfgvbcwn7uqou7vbhycy+tddcecac5ok9n2zsmxatzkwoxvkm+/ytts3pixdybm7pf03vmhambummsq4ujzubxdvdibhjoycxgvrd4akj4pfjrfn0plxorags9b6howvis0oi7jejmc1xgr4gsxx7maboerydmu0dosylgalcqswtgp/dvjitv7libvz9q03tzb8nzavtfyytqkoaomfxz+u1zkakry/rfohkbu5cgc4eq5wyk9batfuhwp2dl0vcj+6vj8qvgbk9ct2i2o6wfp0qzx1s26a0ea4rokjvukutqldz8ppm36uw+qreuzgjcivyjr8iadkrrg38j57r72pf+h/9b/7p9zxfa/3pcsxwv/5b8ay3sy=</latexit> <latexit sha1_base64="/8rfhipqr2j1mpofacatkwdehou=">aaacz3icbvfdaxqxfm2mx3w0utvhx4llwlzkmsmigg8ufoapfvtw28jmwo5k7mxdmx9nmsosr3z1r/ip/ddmzqdit15iodkn5+bem6ssqpsw/o35167fuhlr63zw5+72vfudnqdhuqwvxxkvzaloetaoryezi4zek0oh5ine4+tsbasff0glrvl8nqsk4xywhcgeb+ooxedxiifa5qjymdra98fzmj2f15bslgu3dfgvbcwn7uqou7vbhycy+tddcecac5ok9n2zsmxatzkwoxvkm+/ytts3pixdybm7pf03vmhambummsq4ujzubxdvdibhjoycxgvrd4akj4pfjrfn0plxorags9b6howvis0oi7jejmc1xgr4gsxx7maboerydmu0dosylgalcqswtgp/dvjitv7libvz9q03tzb8nzavtfyytqkoaomfxz+u1zkakry/rfohkbu5cgc4eq5wyk9batfuhwp2dl0vcj+6vj8qvgbk9ct2i2o6wfp0qzx1s26a0ea4rokjvukutqldz8ppm36uw+qreuzgjcivyjr8iadkrrg38j57r72pf+h/9b/7p9zxfa/3pcsxwv/5b8ay3sy=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="38u3msd8clinplviofgp/9pdr90=">aaacfxicbzdnsgmxfiuz9a+orbzrn8eiiisy40axgi7cibxsd3sgksnctqgzzegyqhn6am59bz/gnbjybcxmu7ctfwif5yy395wo5uwbz/txkju7e/sh1up3qobwj08atz6wmalqpzjlnyiibs4eda0zhaapapjehprr7k7o919basbfi5mneczkitiyuwks1bk1wl7bkwtvg7+cflrvqonug1jslafhkcdad30vnwfolgguw8inmg0potmygaffqrlqyv7uucdnvonxwcp7hmgl+vdfthkt50lkbybetpvmrxd/6w0zm74jcybszicgy4/ggcdg4si0jpkcavjcaqgk2v0xnrjfqlhrume9wc8khu3cpxqumvjd5gfrk4sjrqlbqwvrlc3z/pzntlahd9x2vbb/7keqokvn6al56brdogfuqv1euyze0lvz4xw6x8uck84q8czak+f7f+w1ogc=</latexit> <latexit sha1_base64="ygegflhxuqc9mg7/d4825e2/rmy=">aaacu3icbvfda9rafj3ej9by6tzxxwzlysu6jd5ywrdbht5uroc2hz0qbiy326gtstofyhlzy/xz/hsn2sh264wem+fmptp3tn5iywwc/wrco3fv3d/afha93nl99hiyt3nmaqc5znkta32rg0epfm6tsbivgo1q5rlp86v3vx7+dburtfpqvw2mfsyvkauh66ls8voagv5wqmumldctatll19cocsoq4x8dfkt78yvf+cxhyfsnhzl6t7srvgpszz63h7uszryczuzu1cvdyvdtqr+jjd3wtl3pme2rsehyg5wucae5pckm+/eshorebski9slyp9lesmukmrskleusjfkkcwptfrqvxgixmwewax4fs1x4qkbck7bd6r098exby1r7t1k6sp92tfazs6pyv7mf12xqpfk/befs+szthwqcrcxxb5voulvt/mfoitryk1ceanfc35xys9darx++ih1ap4vgt973c4mabk2ft2p63qbyj25czz2sdzdzjos2ohs1s+jz8ium2+qpeuamjcfh5b05jqdktniwfbwmxgdh4xgoqreoogzgzj+qgxv+/w2httir</latexit> <latexit sha1_base64="ygegflhxuqc9mg7/d4825e2/rmy=">aaacu3icbvfda9rafj3ej9by6tzxxwzlysu6jd5ywrdbht5uroc2hz0qbiy326gtstofyhlzy/xz/hsn2sh264wem+fmptp3tn5iywwc/wrco3fv3d/afha93nl99hiyt3nmaqc5znkta32rg0epfm6tsbivgo1q5rlp86v3vx7+dburtfpqvw2mfsyvkauh66ls8voagv5wqmumldctatll19cocsoq4x8dfkt78yvf+cxhyfsnhzl6t7srvgpszz63h7uszryczuzu1cvdyvdtqr+jjd3wtl3pme2rsehyg5wucae5pckm+/eshorebski9slyp9lesmukmrskleusjfkkcwptfrqvxgixmwewax4fs1x4qkbck7bd6r098exby1r7t1k6sp92tfazs6pyv7mf12xqpfk/befs+szthwqcrcxxb5voulvt/mfoitryk1ceanfc35xys9darx++ih1ap4vgt973c4mabk2ft2p63qbyj25czz2sdzdzjos2ohs1s+jz8ium2+qpeuamjcfh5b05jqdktniwfbwmxgdh4xgoqreoogzgzj+qgxv+/w2httir</latexit> <latexit sha1_base64="lmvrh5/kvgafyub7+qrreexaogu=">aaacxnicbvfda9rafj3ej9zo7by++jk4flais+kdfxwpfkafklzw28jocdetm+3qyssdmvswgpcv+lp8n06yuezwcwlnzplzzu6dtjlc2dd85fl37t67v7h5ihj4aovx9mhn99sutey446us9xkkbqvqolpcsjyvnekrsjxll993+tk1aink9duuk4wlwcircw7wucno5x4dvsieshhaometfmkurmrikcue+/x4le3ef27lfvv7wr8pzfsvuxloafyitzupbdiwczvlrhtuke28z+2o70mkxcumxwjftamxa1kt5jrxotk9tkbjcbr2rw+daabjmtrjsuntsazkdyhkcgngzkowsned2gousq1ybbacfgklnduooeatn/3pld1ztebzurtpwdqz/zoakixzfqnb2bvr1rwo/j82r23+jm6eqmqliq8oymtjbum7h6gz0mitxdoaxat3v8ovqao37vkc9gfdlxo/udzpfwqwpx7ednnve8a6dom6q5bcakp1cd0gp6+mutinvotjw3fdkdfju/kmtehedsghosinzea4t+g99f57b/6rr/za/7ba6nud5wm5uf6p3xy+2b8=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> As a language game arg min??? f L vein replace n with c veic speaker model listener loss

71 Results: string editing accuracy Identity Multitask Meta This Work

72 Results examples true description true output emboldens kisses emboldecs kisses replace all n s with c loocies loneliness locelicess loonies vein dogtrot veic dogtrot change any n to a c loocies pred. description pred. output

73 Problems How do we bootstrap from (unannotated) exploration of the environment alone? How good are inferred descriptions as explanations?

74 Problems How do we bootstrap from (unannotated) exploration of the environment alone How good are inferred descriptions as explanations?

75 Conclusions

76 Conclusions Use RL in NLP by formulating language generation / understanding as reward maximization rather than supervised learning.

77 Conclusions Use language in (I)RL as a scaffold for learning options, goal representations, cost functions. Languages encode 100k years of accumulated knowledge about which abstractions are useful take advantage of it!

78 Thanks! also Looking for NLP jobs? Ask me about Microsoft! Applying to PhD programs? Ask me about MIT!

CD _. _. 'p M CD, CD~~V. C ~'* Co ~~~~~~~~~~~~- CD / X. pd.0 & CD. On 0 CDC _ C _- CD C P O ttic 2 _. OCt CD CD (IQ. q"3. 3 > n)1t.

CD _. _. 'p ~~M CD, CD~~~~V. C ~'* Co ~~~~~~~~~~~~- CD / X. pd.0 & CD. On 0 CDC _ C _- CD C P O ttic 2 _. OCt CD CD (IQ. q3. 3 > n)1t. n 5 L n q"3 +, / X g ( E 4 11 " ') $ n 4 ) w Z$ > _ X ~'* ) i 1 _ /3 L 2 _ L 4 : 5 n W 9 U~~~~~~ 5 T f V ~~~~~~~~~~~~ (Q ' ~~M 3 > n)1 % ~~~~V v,~~ _ + d V)m X LA) z~~11 4 _ N cc ', f 'd 4 5 L L " V +,