Linguistic scaffolds for policy learning. Jacob Andreas Berkeley Microsoft Semantic Machines MIT

Linguistic scaffolds for policy learning Jacob Andreas Berkeley Microsoft Semantic Machines MIT

Linguistic scaffolds for policy learning Work on language! Jacob Andreas Berkeley Microsoft Semantic Machines MIT

What RL can do for language What language can do for RL Crafting environment replace make plank the last getletter wood of use the word tools drop makehead stick get wood use workb change make cloth the final getletter grassto use t i facto add makea rope z if the get last grass character useis tools a make bridge get iron get wood every vowel make bed becomes y get wood use tools change make axe only the get first wood consonant use workb to first make shears & last 3 get letters wood use workb delete get gold every vowel get iron get wood replace get gemall n s get with wood c use workb

What RL can do for language Daniel Fried Ronghang Hu Volkan Cirik w/ Anja Rohrbach, L.P. Morency, Taylor Berg-Kirkpatrick, Trevor Darrell and Dan Klein

Generation & understanding Turn right and walk through the kitchen. Go right into the living room and stop by the rug. [Anderson et al. 18]

A reference game [Frank & Goodman 12]

glasses" [Frank & Goodman 12]

glasses" [Frank & Goodman 12]

glasses" [Frank & Goodman 12]

glasses" [Frank & Goodman 12]

The rational speech acts model L 0 (. glasses) L 0 (. hat) 1/2 1/2 0 1 [Frank & Goodman 12]

The rational speech acts model L 0 (. glasses) L 0 (. hat) 1/2 1/2 0 1 S 1 ( glasses. ) L 0 (. glasses) 1 1/3 S 1 ( hat. ) 0 2/3 [Frank & Goodman 12]

The rational speech acts model L 1 (. glasses ) S 1 ( glasses. ) L 1 (. hat ) 3/4 1/4 0 1 S 1 ( glasses. ) L 0 (. glasses) 1 1/3 S 1 ( hat. ) 0 2/3 [Frank & Goodman 12]

Pragmatics Q: Do you know what time it is?

Pragmatics Q: Do you know what time it is? A: Yes

Pragmatics Q: Do you know what time it is? A: Yes I find his cooking very interesting. [Grice 70]

RSA game tree speaker hat glasses

RSA game tree: as speaker speaker (listener) +1 hat hat -1 +1 glasses glasses -1

RSA game tree: as speaker speaker (listener) +1 hat hat -1 +1 glasses glasses -1

RSA game tree: as speaker speaker (listener) +1 hat hat -1 +1 glasses glasses -1

RSA game tree: as listener (speaker) listener?? glasses glasses?

RSA game tree: as listener (speaker) listener? Language use is gameplay!? glasses glasses?

A recipe for pragmatic text generation 1. Train a base listener model smiley glasses plain glasses hat & man glasses glasses

A recipe for pragmatic text generation 1. Train a base listener model 2. Train a reasoning speaker to win when playing with the listener +1 hat hat -1 +1 glasses glasses -1

Application: image captioning 1. Train an image retrieval / gen model a snake is slithering away from Jenny

Application: image captioning 2. Describe images using the listener model for search at inference time +1 a snake is slithering away -1 the sun is in the sky -1 [A & Klein 16, Vedantam et al. 17] -1

Application: image captioning 2. Describe images using the listener model as a training-time reward ( self-play ) -1 the sun is in the sky captioner model retrieval loss [Yu et al. 16, Mao et al. 16]

Descriptive captions [Vedantam et al. 17] seq2seq captioner: this bird has a yellow breast with a short pointy bill pragmatic captioner: a small yellow bird with black stripes on its body and black stripe on the wings.

Contrastive captions without contrastive data! (a) (b) (c) Mike is holding a baseball bat. The snake is slithering away from Mike & Jenny. [A & Klein 16]

Application: instruction generation 1. Train a base instruction following model 2. Train an instruction generation model to get the follower to goal states

Application: instruction generation seq2seq: Walk past the dining room table and chairs and wait there. speaker-listener: Walk past the dining room table and chairs and take a right into the living room. Stop once you are on the rug. human: Turn right and walk through the kitchen. Go right into the living room and stop by the rug. [Fried, Hu, Cirik et al. 18]

Listener mode human: Go through the door on the right and continue straight. Stop in the next room in front of the bed. seq-to-seq speaker-listener [Fried, Hu, Cirik et al. 18]

The rules of the game +1 glasses glasses

The rules of the game +1 hat hat

Killer robots [Lewis et al. 17] Bob: i can i i everything else.............. Alice: balls have zero to me to me to me to me to me to me to me to me to Bob: you i everything else.............. Alice: balls have a ball to me to me to me to me to me to me to me

Killer robots [Lewis et al. 17] Bob: i can i i everything else.............. Alice: balls have zero to me to me to me to me to me to me to me to me to Bob: you i everything else.............. Alice: balls have a ball to me to me to me to me to me to me to me

Problems to work on How do we use tools like self-play and tree search while remaining within the rules of natural language? How do we do efficient search in string- valued action spaces?

Problems to work on How do we use tools like self-play and tree search while remaining within the rules of natural language? How do we do efficient search in string- valued action spaces?

What language can do for RL w/ Dan Klein and Sergey Levine

A crafting game make planks make sticks

Learning with sketches get wood get wood use saw use axe

[Su$on et al. 99] The options framework

Unsupervised option learning +r [Bacon & Precup 16]

Learning with intermediate rewards +r +r [Kearns & Singh 02, Kulkarni et al. 16]

Segmenting demonstrations Ï [Stolle & Precup 02, Fox & Krishnan et al. 16]

Learning from sketches get wood use saw Ï [A, Klein & Levine 17]

Modular policies get wood use saw π 1 π 2 get wood use axe π 1 π 3

Modular policies get wood use saw π 1 π 2 get wood use axe π 1 π 3

Modular policies TURN LEFT π 1 get wood

Results: crafting game

Results: crafting game Sketches: modular Reward Sketches: joint Unsupervised 0 1 2 3 x 10 6 episodes

Results: locomotion

Results: locomotion Sketches: modular Reward Sketches: joint Unsupervised 0 1 2 3 x 10 8 Tmesteps

Generalization What if I don t get a sketch at test Tme????

Generalization What if I don t get a sketch at test Tme? 100 Unsupervised 75 89 76 Sketches 50 25 47 42 0 Training AdaptaTon

Moral A little bit of (structured) language goes a long way!

Beyond structured sketches Language learning Learning from demonstrations itch itctch emboldens emboldecs dogtrot dogtrot first & last 3 letters loneliness locelicess vein??? [A, Klein & Levine 17]

Beyond structured sketches Language learning Learning from demonstrations itch itctch emboldens emboldecs dogtrot dogtrot first & last 3 letters loneliness locelicess vein???

Pretraining via language learning wonderful f( ;, ) wonful first & last 3 letters [Branavan et al., 09]

Concept learning emboldens vein loneliness L(f( ;, ), ) emboldecs veic locelicess

Concept learning emboldens vein loneliness L(f( ;, ), ) emboldecs veic locelicess every vowel becomes i

Concept learning emboldens vein loneliness L(f( ;, ), ) emboldecs veic locelicess every vowel becomes i 128.6

Concept learning emboldens vein loneliness L(f( ;, ), ) emboldecs veic locelicess every vowel becomes i 128.6 change consonants to c 52.3

Concept learning emboldens vein loneliness L(f( ;, ), ) emboldecs veic locelicess every vowel becomes i 128.6 change consonants to c 52.3 replace n with c 8.3

Prediction (f( ;, ), ) replace n with c

Evaluation loonies L(f( ;, ), ) replace n with c

Evaluation loonies f( ;, ) loocies replace n with c

<latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit> As multitask learning Pretraining data Training data wonderful wonful emboldens emboldecs glabrous glaous vein veic itch itctch loneliness locelicess arg min L(f( itctch itch ;, )) arg min <latexit sha1_base64="6dtskclha7pt7ua1kd9l70axyac=">aaacqxicbzbbsxwxfmcz1ra62nbbhr0ef2etzzmrqgu9sntddxuvxbv2luvn9s0atdjj8qawdovh8nn4k+136efwjh71ymacq6s+spjx/+clef8kv9jrgp4n5p7mp332fggxtbt84uwr9us3+y4rrmc+yfrmdxnwqktbpklsejhbbj0opeiov1b+wu+0tmzmj6y5djvmjeylaplsqb22yratlc0orgl+o5t245otasy81tjvnx/mlfmen56ur4/anbax1sufqtrahzw1m2pfx+nmfboncqxodaiwp2ejlqrqogvfhcmcxdfmcodrgey3lovjznznk2oeztyvq7xw/+0oqts31yk/qygo3h2veh/zbgwln4alnhlbamtdq2mhogw8iomppuvbauobhjx+r1wcgqvbpsxw/a39lba3/l3bovqgzl4rmyrnncqv+bci+9e8hp2nxht2ot0pnc0vtwwlbiwtsi6l2ee2yb6zhdzngp2xc/ab/ql+brfbzxb1d3quahresv8qulkffy6v0w==</latexit>??? L(f( veic vein;, )) first & last 3 letters replace n with c [Caruana 97]

As inverse reinforcement learning vein replace n with c??? {<latexit sha1_base64="h0oi12vvfczflvoyzoq3lwhxltu=">aaacu3icbvfda9rafj3erxqtbvxrl6flyatlsupbqytib/igwmftczsh3exutmmnk3rmiiwxf6p/rf/fbyfzkhbrhrnonhppnbl30koky8pwyvnv3b5z997g/edbw81hj0dbt45nwwuom17kup+myfakhtmrrmttsimuqcst9pxdp5/8qg1eqb7zzyvxaqslcshboiozxe4w0itcqishbfppkk/yxuungwwfcfupx9no3hmnd9jddf54kkn/3zvwdrbnadp8ajogwagpm6kgtmnnfd2o+jmupgs144d1ztqas5y1ywgctsm+6e0qdwbmhjhktrxnlpw8llbzlsgyerrwnm5aw8eltggrdvbaz2gbcwcvfgjipr+zptuoywheareupt37r6obwphlkbrmrkezrnxk/7r5bfnxcsnuvvtufhvrxktqs9r9bs2erm7l0ghgwri3un4ggrh1fxaw9+h60fjz1f1soqzb6ufnmpk2b6xd156zasknmfof101wvd+nwmn09wb8+hyy5qz5rrbjhetkjtkkh8krmrfofnnb3gtvz3/jc/+7l1epvjd4npjr4de/ad1k1zy=</latexit> l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 83 3u sm 8d Lc ni lp iv fo pg 9/ DP 9R =0 >" AA CA XF ci Zb ND gs xm IF zu a9 O+ br rz 8N ie Ii ys 04 Xa ig c7 Bi sx 3D GS sk cn qt ZG EZ yg hq 6n ma 95 ZB G/ bn yj cb Mx 7u tc wf fi y5 3Y 59 ow U5 bw /z xt jk 7u /e Hs U1 3P Oq Wb 0j a8 Zt W6 am QL Zp LJ YN Ii sb E4 Ad z0 AH pa pa EJ PH Rr K7 o7 19 B9 Sa Fb 5i nm CE kz ti Yi WU sk B1 1k lw b7 wk vt 7g C+ lf Vr Oq Un 1g Sj AL hf CK ad 3D v0 WN OF Gl UG 8w NI gm p0 To ym ag FF RQ QL VY u7 cu nd ov xn CW 7p Mh lg v+ Fd ht tk 05 kl yb Eb PT mv xr /D w6 z0 7M J4 yc Sb Iz gc 4y G/ cg Gd s4 0I pj Ck va cj qa KG V2 x0 Rn FJ Lq RH Mu 9E CW K8 uh c3 xp UQ VM dj g5 RF 4k JS QR Bl WQ rv CL Z3 P/ Nz Lt ha 9d 2X bv /b K7 qe ko nv A6 5L B6 dr go Uf VQ E1 YU ez L0 zv X4 6w 8X cu 8K q4 C8 az +K 7f +F 1W go =c /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 09 a7 hl H1 gk Ks qe RM w7 at LR IV 5O =o >" AA CA Hs ci Vb NF 9b wq HE CX wv tk Lb yl Vs Vp k2 1K rs Ai Jh QC IA Dk Ci xk ab 1V EF c2 dy ta an 2S 7g yq R+ 8v /N L4 5B sx HQ LT LS ea /3 MP Je z7 gr jp /4 nh NE 66 ef uf av 73 Y0 tp Yn Pe rj bv 6J uz cn 5c 2r tw 3z wi EK FW yc su Lx GN 6I ch m4 +l b+ Xb 7T h+ qn uk tv Vp 6g Cm SZ KV cw 6F pk 8v GO gc 0l Up UG LM O9 0O Ln rk xq KY py wt 42 fb V0 84 i8 /d DO My u/ QR vr m9 8N nj Dg P3 /c +e xy zy F0 mj Kh ej R6 DZ Z3 67 pp ka zg TX Ai D3 IO F1 1x 2w Y2 n9 R8 0D kj Gh Es G/ MO 25 xg W1 L1 Vx FW uk Zw Fh je 2c ad tc B4 7K Li GU +G XC Ms FS xh ou KN bk ne 0b Dw FM WL tv 2V px Pw b6 U4 ZA 1s 5K 9v Pn Da 1a vn fy mt th T+ 1J mo ap xi fv yn bp WS N1 9+ g+ dh ID Vr 5x FA Lw 1/ Kb 0z DE /t P7 vi OY S/ aw /P 6m xn Uv GY 9v I1 W2 wd gn bp 2r Pn IV s3 kb 60 b6 O4 5T IL nl Zy Ye Jb ne IZ M9 US ej Dk kf za mk 8c JL 2r va Be cy ah D9 lh s6 Qr D6 f0 HP F5 Eq j8 Ke 9t PY /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 09 a7 hl H1 gk Ks qe RM w7 at LR IV 5O =o >" AA CA Hs ci Vb NF 9b wq HE CX wv tk Lb yl Vs Vp k2 1K rs Ai Jh QC IA Dk Ci xk ab 1V EF c2 dy ta an 2S 7g yq R+ 8v /N L4 5B sx HQ LT LS ea /3 MP Je z7 gr jp /4 nh NE 66 ef uf av 73 Y0 tp Yn Pe rj bv 6J uz cn 5c 2r tw 3z wi EK FW yc su Lx GN 6I ch m4 +l b+ Xb 7T h+ qn uk tv Vp 6g Cm SZ KV cw 6F pk 8v GO gc 0l Up UG LM O9 0O Ln rk xq KY py wt 42 fb V0 84 i8 /d DO My u/ QR vr m9 8N nj Dg P3 /c +e xy zy F0 mj Kh ej R6 DZ Z3 67 pp ka zg TX Ai D3 IO F1 1x 2w Y2 n9 R8 0D kj Gh Es G/ MO 25 xg W1 L1 Vx FW uk Zw Fh je 2c ad tc B4 7K Li GU +G XC Ms FS xh ou KN bk ne 0b Dw FM WL tv 2V px Pw b6 U4 ZA 1s 5K 9v Pn Da 1a vn fy mt th T+ 1J mo ap xi fv yn bp WS N1 9+ g+ dh ID Vr 5x FA Lw 1/ Kb 0z DE /t P7 vi OY S/ aw /P 6m xn Uv GY 9v I1 W2 wd gn bp 2r Pn IV s3 kb 60 b6 O4 5T IL nl Zy Ye Jb ne IZ M9 US ej Dk kf za mk 8c JL 2r va Be cy ah D9 lh s6 Qr D6 f0 HP F5 Eq j8 Ke 9t PY /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 8K Kq ta 3u er Gy +M dz 8Z xl Kl +8 bm =Y >" AA CA 3u ci Vb NF 9b wq HE CX wv Um ub IH ex qq h0 qa cv FI IJ UR Sl xj FA ke qt 02 qj JK 9M am k2 ql 0O ri HN j+ 4/ /3 gw ne xg dd pm tk /5 mf Tj j3 Jt CL D2 8D f6 3l t7 76 2/ jd fy od e8 Ta dr 2H x0 TN p1 jr Jj ye G1 pc JG is Jw Vk uv ZJ Rp UF vk 0E Xv fj a6 /c BU Rt tf s/ ss YJ aw QI eu gb ZH MW nr Zc I6 SU cr QL va 00 Sy sf r8 GK Dj li N3 jb 7t TQ 99 J3 fh 2b jg e8 uy fh yd cw +A 5x jm fy q2 hr mf Kr Fj UD ek 82 9r Rt 0V JN lu zy ld fj KT xm 1a Tm Mj hb ON Dy g3 Rb MA kz Oi kn 92 kt cw rl Yh lx oe Zy 2R lf w4 0a VF ix 7G ad AY 8X ha 4Y 7d AA Sh uz j+ bt uu Yu Oj la qd wu Gt /f Td gs Fj qm G1 2V ZP 1p Sr /P 8p r1 7m J+ 1G UF ct CW yr K7 0a tl bs fv Jo Qn Ky c1 go cb /C Wd sy B9 7A uf wz 2J Kh X4 Zj d9 S3 V8 ra lc tf ME 2I 79 Dw 0p z4 rm tl Aw /o xv Q3 ny 6r RZ IO +2 uh DO 2w UG +G 5Q S2 TE pe XH I5 8B MJ kd jr 5j e5 41 7L 91 5/ P3 e/ ++ KX 6X u3 5B mr E6 /X G8 C3 XT gm == /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t l< ta xe ti s ah _1 ab es 46 "= 0H Io 21 VV cf FZ Vl Yo OZ 3Q Wl Xh tl =U >" AA CA 3u ci Vb df 9a AR JF E3 xr tq vb RX 6l LF ay lt US Bp YQ it /B gi MW tf zc hs E3 ux mt nm 3k mr ii xw 6f /P fr F/ yb Zf HK rb Rh On Hn PP bn 3l k0 Ko 8Y wp vy vn b3 z5 99 G7 e/ BD 8w H1 0j bd 4T N5 WW Ou 1M K7 pu m+ FY KA Th rm Mr TT is UM cq TS P9 dx 5p 8/ GQ E1 bq Z7 YZ xv QA ls sc BH io Zo ex w4 I0 Ct Iq hs fb pp kk Y/ Ux NU WG FW Fc Pu 9X ON H3 nm 9d dj fd 45 KK /N Z3 wv rd nb da 8p Ja GO aw pg 6M gk mt nn df O2 j+ Um Gp 1s 44 1D Tz aq 5s 1Y Wy cg st +M E6 Q0 WD MB jh Kh rt Nx pl 8W ll ZB sl YG Re WR mn A5 8W le gt rg VD Ab 2z BG wc Vc GF ij rp z+ Tp Ou Wy eh ra Ue Tp 73 6r BO pw lh bk Mr kr ze nr kx 7/ 5R fb XN Sc UN VV Ut Hf RV kx qt 9S 9r sb E2 mr l7 g0 gh rw 3i nu G4 rg 1h xf wa +9 6h f0 Zj f1 S1 Qo bz u6 Nf PM 2K 6B Dx 51 z6 sa Nk FM fo 01 w1 Dv N+ mw 0n W9 8B H+ YY Q5 5Z rr Jb Eh kt TJ kk 8H Rk Rm OF nf bn g3 vt 3z j/ /c 7+ 1L pe jv 4d pn rj d4 /e da k1 z1 =Y /< al et ix >t cost function arg min Ê [L(f( ;, ))] <latexit sha1_base64="ehqpi+osq3iuv/rus12ofbsdhee=">aaadhhicnvjni9raeo3erzw6oqthl43dwkzikiig4gxxazysuikzuzadqqvtmwk2x9vdeyy2j6/6f/w13ssr4c/xaictr3fwkwvpxr/qeql63xgvcav9/4fjxrh46fkvravetevbn24odm4dqrkwhke8zep5hipctbq41ujnefxjhdzo8cg+edbmj96hvkis3uplhweo80kkgoo2vdt4owig57koioya6p44hbpt0xosynjhlw4/ow3yvt3zze6u97ugmrqurostal2iy/oiiqztufomre5tppl1ui31vpekg5qht9zcgdz/pp5la8q6z4zepghgg6wucse53v+3hg2g/stvgp4hqq+gpi+damfzzknj6xwlztnqahb4lq4nsc14ho3haouv8boy48zcanjuoen6aojimglns2m/qtoo/bvcqk7umo/tyxzytzlryx/lzrvoh4dgffwtsecrh6v1rnvj2+umizdidba0algutlfkfycba/soppyc7swsx1nd1xvk0kw8z3qlmg6wfp1pzzwsntdytos8ohwwcfxj8obhco9pb+uwuupukjejycoyr16sazil3amdd85h55p72f3ifnw/ry66tl9zm5wj9/sv7qn7sg==</latexit> <latexit sha1_base64="ehqpi+osq3iuv/rus12ofbsdhee=">aaadhhicnvjni9raeo3erzw6oqthl43dwkzikiig4gxxazysuikzuzadqqvtmwk2x9vdeyy2j6/6f/w13ssr4c/xaictr3fwkwvpxr/qeql63xgvcav9/4fjxrh46fkvravetevbn24odm4dqrkwhke8zep5hipctbq41ujnefxjhdzo8cg+edbmj96hvkis3uplhweo80kkgoo2vdt4owig57koioya6p44hbpt0xosynjhlw4/ow3yvt3zze6u97ugmrqurostal2iy/oiiqztufomre5tppl1ui31vpekg5qht9zcgdz/pp5la8q6z4zepghgg6wucse53v+3hg2g/stvgp4hqq+gpi+damfzzknj6xwlztnqahb4lq4nsc14ho3haouv8boy48zcanjuoen6aojimglns2m/qtoo/bvcqk7umo/tyxzytzlryx/lzrvoh4dgffwtsecrh6v1rnvj2+umizdidba0algutlfkfycba/soppyc7swsx1nd1xvk0kw8z3qlmg6wfp1pzzwsntdytos8ohwwcfxj8obhco9pb+uwuupukjejycoyr16sazil3amdd85h55p72f3ifnw/ry66tl9zm5wj9/sv7qn7sg==</latexit> <latexit sha1_base64="ehqpi+osq3iuv/rus12ofbsdhee=">aaadhhicnvjni9raeo3erzw6oqthl43dwkzikiig4gxxazysuikzuzadqqvtmwk2x9vdeyy2j6/6f/w13ssr4c/xaictr3fwkwvpxr/qeql63xgvcav9/4fjxrh46fkvravetevbn24odm4dqrkwhke8zep5hipctbq41ujnefxjhdzo8cg+edbmj96hvkis3uplhweo80kkgoo2vdt4owig57koioya6p44hbpt0xosynjhlw4/ow3yvt3zze6u97ugmrqurostal2iy/oiiqztufomre5tppl1ui31vpekg5qht9zcgdz/pp5la8q6z4zepghgg6wucse53v+3hg2g/stvgp4hqq+gpi+damfzzknj6xwlztnqahb4lq4nsc14ho3haouv8boy48zcanjuoen6aojimglns2m/qtoo/bvcqk7umo/tyxzytzlryx/lzrvoh4dgffwtsecrh6v1rnvj2+umizdidba0algutlfkfycba/soppyc7swsx1nd1xvk0kw8z3qlmg6wfp1pzzwsntdytos8ohwwcfxj8obhco9pb+uwuupukjejycoyr16sazil3amdd85h55p72f3ifnw/ry66tl9zm5wj9/sv7qn7sg==</latexit> <latexit sha1_base64="ehqpi+osq3iuv/rus12ofbsdhee=">aaadhhicnvjni9raeo3erzw6oqthl43dwkzikiig4gxxazysuikzuzadqqvtmwk2x9vdeyy2j6/6f/w13ssr4c/xaictr3fwkwvpxr/qeql63xgvcav9/4fjxrh46fkvravetevbn24odm4dqrkwhke8zep5hipctbq41ujnefxjhdzo8cg+edbmj96hvkis3uplhweo80kkgoo2vdt4owig57koioya6p44hbpt0xosynjhlw4/ow3yvt3zze6u97ugmrqurostal2iy/oiiqztufomre5tppl1ui31vpekg5qht9zcgdz/pp5la8q6z4zepghgg6wucse53v+3hg2g/stvgp4hqq+gpi+damfzzknj6xwlztnqahb4lq4nsc14ho3haouv8boy48zcanjuoen6aojimglns2m/qtoo/bvcqk7umo/tyxzytzlryx/lzrvoh4dgffwtsecrh6v1rnvj2+umizdidba0algutlfkfycba/soppyc7swsx1nd1xvk0kw8z3qlmg6wfp1pzzwsntdytos8ohwwcfxj8obhco9pb+uwuupukjejycoyr16sazil3amdd85h55p72f3ifnw/ry66tl9zm5wj9/sv7qn7sg==</latexit>

<latexit sha1_base64="/8rfhipqr2j1mpofacatkwdehou=">aaacz3icbvfdaxqxfm2mx3w0utvhx4llwlzkmsmigg8ufoapfvtw28jmwo5k7mxdmx9nmsosr3z1r/ip/ddmzqdit15iodkn5+bem6ssqpsw/o35167fuhlr63zw5+72vfudnqdhuqwvxxkvzaloetaoryezi4zek0oh5ine4+tsbasff0glrvl8nqsk4xywhcgeb+ooxedxiifa5qjymdra98fzmj2f15bslgu3dfgvbcwn7uqou7vbhycy+tddcecac5ok9n2zsmxatzkwoxvkm+/ytts3pixdybm7pf03vmhambummsq4ujzubxdvdibhjoycxgvrd4akj4pfjrfn0plxorags9b6howvis0oi7jejmc1xgr4gsxx7maboerydmu0dosylgalcqswtgp/dvjitv7libvz9q03tzb8nzavtfyytqkoaomfxz+u1zkakry/rfohkbu5cgc4eq5wyk9batfuhwp2dl0vcj+6vj8qvgbk9ct2i2o6wfp0qzx1s26a0ea4rokjvukutqldz8ppm36uw+qreuzgjcivyjr8iadkrrg38j57r72pf+h/9b/7p9zxfa/3pcsxwv/5b8ay3sy=</latexit> <latexit sha1_base64="/8rfhipqr2j1mpofacatkwdehou=">aaacz3icbvfdaxqxfm2mx3w0utvhx4llwlzkmsmigg8ufoapfvtw28jmwo5k7mxdmx9nmsosr3z1r/ip/ddmzqdit15iodkn5+bem6ssqpsw/o35167fuhlr63zw5+72vfudnqdhuqwvxxkvzaloetaoryezi4zek0oh5ine4+tsbasff0glrvl8nqsk4xywhcgeb+ooxedxiifa5qjymdra98fzmj2f15bslgu3dfgvbcwn7uqou7vbhycy+tddcecac5ok9n2zsmxatzkwoxvkm+/ytts3pixdybm7pf03vmhambummsq4ujzubxdvdibhjoycxgvrd4akj4pfjrfn0plxorags9b6howvis0oi7jejmc1xgr4gsxx7maboerydmu0dosylgalcqswtgp/dvjitv7libvz9q03tzb8nzavtfyytqkoaomfxz+u1zkakry/rfohkbu5cgc4eq5wyk9batfuhwp2dl0vcj+6vj8qvgbk9ct2i2o6wfp0qzx1s26a0ea4rokjvukutqldz8ppm36uw+qreuzgjcivyjr8iadkrrg38j57r72pf+h/9b/7p9zxfa/3pcsxwv/5b8ay3sy=</latexit> <latexit sha1_base64="/8rfhipqr2j1mpofacatkwdehou=">aaacz3icbvfdaxqxfm2mx3w0utvhx4llwlzkmsmigg8ufoapfvtw28jmwo5k7mxdmx9nmsosr3z1r/ip/ddmzqdit15iodkn5+bem6ssqpsw/o35167fuhlr63zw5+72vfudnqdhuqwvxxkvzaloetaoryezi4zek0oh5ine4+tsbasff0glrvl8nqsk4xywhcgeb+ooxedxiifa5qjymdra98fzmj2f15bslgu3dfgvbcwn7uqou7vbhycy+tddcecac5ok9n2zsmxatzkwoxvkm+/ytts3pixdybm7pf03vmhambummsq4ujzubxdvdibhjoycxgvrd4akj4pfjrfn0plxorags9b6howvis0oi7jejmc1xgr4gsxx7maboerydmu0dosylgalcqswtgp/dvjitv7libvz9q03tzb8nzavtfyytqkoaomfxz+u1zkakry/rfohkbu5cgc4eq5wyk9batfuhwp2dl0vcj+6vj8qvgbk9ct2i2o6wfp0qzx1s26a0ea4rokjvukutqldz8ppm36uw+qreuzgjcivyjr8iadkrrg38j57r72pf+h/9b/7p9zxfa/3pcsxwv/5b8ay3sy=</latexit> <latexit sha1_base64="/8rfhipqr2j1mpofacatkwdehou=">aaacz3icbvfdaxqxfm2mx3w0utvhx4llwlzkmsmigg8ufoapfvtw28jmwo5k7mxdmx9nmsosr3z1r/ip/ddmzqdit15iodkn5+bem6ssqpsw/o35167fuhlr63zw5+72vfudnqdhuqwvxxkvzaloetaoryezi4zek0oh5ine4+tsbasff0glrvl8nqsk4xywhcgeb+ooxedxiifa5qjymdra98fzmj2f15bslgu3dfgvbcwn7uqou7vbhycy+tddcecac5ok9n2zsmxatzkwoxvkm+/ytts3pixdybm7pf03vmhambummsq4ujzubxdvdibhjoycxgvrd4akj4pfjrfn0plxorags9b6howvis0oi7jejmc1xgr4gsxx7maboerydmu0dosylgalcqswtgp/dvjitv7libvz9q03tzb8nzavtfyytqkoaomfxz+u1zkakry/rfohkbu5cgc4eq5wyk9batfuhwp2dl0vcj+6vj8qvgbk9ct2i2o6wfp0qzx1s26a0ea4rokjvukutqldz8ppm36uw+qreuzgjcivyjr8iadkrrg38j57r72pf+h/9b/7p9zxfa/3pcsxwv/5b8ay3sy=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="38u3msd8clinplviofgp/9pdr90=">aaacfxicbzdnsgmxfiuz9a+orbzrn8eiiisy40axgi7cibxsd3sgksnctqgzzegyqhn6am59bz/gnbjybcxmu7ctfwif5yy395wo5uwbz/txkju7e/sh1up3qobwj08atz6wmalqpzjlnyiibs4eda0zhaapapjehprr7k7o919basbfi5mneczkitiyuwks1bk1wl7bkwtvg7+cflrvqonug1jslafhkcdad30vnwfolgguw8inmg0potmygaffqrlqyv7uucdnvonxwcp7hmgl+vdfthkt50lkbybetpvmrxd/6w0zm74jcybszicgy4/ggcdg4si0jpkcavjcaqgk2v0xnrjfqlhrume9wc8khu3cpxqumvjd5gfrk4sjrqlbqwvrlc3z/pzntlahd9x2vbb/7keqokvn6al56brdogfuqv1euyze0lvz4xw6x8uck84q8czak+f7f+w1ogc=</latexit> <latexit sha1_base64="ygegflhxuqc9mg7/d4825e2/rmy=">aaacu3icbvfda9rafj3ej9by6tzxxwzlysu6jd5ywrdbht5uroc2hz0qbiy326gtstofyhlzy/xz/hsn2sh264wem+fmptp3tn5iywwc/wrco3fv3d/afha93nl99hiyt3nmaqc5znkta32rg0epfm6tsbivgo1q5rlp86v3vx7+dburtfpqvw2mfsyvkauh66ls8voagv5wqmumldctatll19cocsoq4x8dfkt78yvf+cxhyfsnhzl6t7srvgpszz63h7uszryczuzu1cvdyvdtqr+jjd3wtl3pme2rsehyg5wucae5pckm+/eshorebski9slyp9lesmukmrskleusjfkkcwptfrqvxgixmwewax4fs1x4qkbck7bd6r098exby1r7t1k6sp92tfazs6pyv7mf12xqpfk/befs+szthwqcrcxxb5voulvt/mfoitryk1ceanfc35xys9darx++ih1ap4vgt973c4mabk2ft2p63qbyj25czz2sdzdzjos2ohs1s+jz8ium2+qpeuamjcfh5b05jqdktniwfbwmxgdh4xgoqreoogzgzj+qgxv+/w2httir</latexit> <latexit sha1_base64="ygegflhxuqc9mg7/d4825e2/rmy=">aaacu3icbvfda9rafj3ej9by6tzxxwzlysu6jd5ywrdbht5uroc2hz0qbiy326gtstofyhlzy/xz/hsn2sh264wem+fmptp3tn5iywwc/wrco3fv3d/afha93nl99hiyt3nmaqc5znkta32rg0epfm6tsbivgo1q5rlp86v3vx7+dburtfpqvw2mfsyvkauh66ls8voagv5wqmumldctatll19cocsoq4x8dfkt78yvf+cxhyfsnhzl6t7srvgpszz63h7uszryczuzu1cvdyvdtqr+jjd3wtl3pme2rsehyg5wucae5pckm+/eshorebski9slyp9lesmukmrskleusjfkkcwptfrqvxgixmwewax4fs1x4qkbck7bd6r098exby1r7t1k6sp92tfazs6pyv7mf12xqpfk/befs+szthwqcrcxxb5voulvt/mfoitryk1ceanfc35xys9darx++ih1ap4vgt973c4mabk2ft2p63qbyj25czz2sdzdzjos2ohs1s+jz8ium2+qpeuamjcfh5b05jqdktniwfbwmxgdh4xgoqreoogzgzj+qgxv+/w2httir</latexit> <latexit sha1_base64="lmvrh5/kvgafyub7+qrreexaogu=">aaacxnicbvfda9rafj3ej9zo7by++jk4flais+kdfxwpfkafklzw28jocdetm+3qyssdmvswgpcv+lp8n06yuezwcwlnzplzzu6dtjlc2dd85fl37t67v7h5ihj4aovx9mhn99sutey446us9xkkbqvqolpcsjyvnekrsjxll993+tk1aink9duuk4wlwcircw7wucno5x4dvsieshhaometfmkurmrikcue+/x4le3ef27lfvv7wr8pzfsvuxloafyitzupbdiwczvlrhtuke28z+2o70mkxcumxwjftamxa1kt5jrxotk9tkbjcbr2rw+daabjmtrjsuntsazkdyhkcgngzkowsned2gousq1ybbacfgklnduooeatn/3pld1ztebzurtpwdqz/zoakixzfqnb2bvr1rwo/j82r23+jm6eqmqliq8oymtjbum7h6gz0mitxdoaxat3v8ovqao37vkc9gfdlxo/udzpfwqwpx7ednnve8a6dom6q5bcakp1cd0gp6+mutinvotjw3fdkdfju/kmtehedsghosinzea4t+g99f57b/6rr/za/7ba6nud5wm5uf6p3xy+2b8=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> <latexit sha1_base64="lwz4difhtsmxqy3zapcm+i1hvok=">aaacxnicbvfda9rafj3ej9zodaupvgwuha3okojyql+kh9chihxctratws3kzjs0matzoswx4f/xz/lvngsj2k0xes6cm+fm3dtpxqhtwvcx59+6fefuxua94p6dryeprtupt3vlfcczr4pknaegsrasz0ayas9rhvcmbz6ll+86/ewrki0q+cusa4xlweircw7gucno5w4dtsiftbgaometfmkurixkljxc/xp8qdvxhvu5xe5u8mddgf3rrovzglli0+zdmztmgkvmi5i6pz33ur31fyika5mxc+27arrmlwucndiufkfhywgctso+6e0qdwbmhjpjtr0tllxclignl0drertwjm5agcelbanmndbal2gbcwcllkjjpj+9ptuoywhekfdjq3v2x0cdpdblmnu7u3b1utar/9pm1ut7csnkbq1kvjootwu1fe0ehmzcitff0ghgsri7un4bcrhxzxew9+h6ufjr5x6quygp1pnmmh7ba9aha9dzteqggk2p6yy4ftwnwmn0+fx48o0wyk3yldwjexkrpxjijsgjmrhubxgvvtfenn/ks9/631zbfw/wpchxyv/xgxd+2cm=</latexit> As a language game arg min??? f L vein replace n with c veic speaker model listener loss

Results: string editing accuracy 76 50 62 18 Identity Multitask Meta This Work

Results examples true description true output emboldens kisses emboldecs kisses replace all n s with c loocies loneliness locelicess loonies vein dogtrot veic dogtrot change any n to a c loocies pred. description pred. output

Problems How do we bootstrap from (unannotated) exploration of the environment alone? How good are inferred descriptions as explanations?

Problems How do we bootstrap from (unannotated) exploration of the environment alone How good are inferred descriptions as explanations?

Conclusions

Conclusions Use RL in NLP by formulating language generation / understanding as reward maximization rather than supervised learning.

Conclusions Use language in (I)RL as a scaffold for learning options, goal representations, cost functions. Languages encode 100k years of accumulated knowledge about which abstractions are useful take advantage of it!

Thanks! also Looking for NLP jobs? Ask me about Microsoft! Applying to PhD programs? Ask me about MIT! jaandrea@microsoft.com