AlphaGo Overview. Overview. Digression: Zero sum, alternating move games. Original AlphaGo approach. Recent AlphaGo Zero approach 10/24/17

Transcription

1 AlphGo Overview Ron Prr CompSci Duke Univerity Overview Digreion: Zero um, lternting move gme Originl AlphGo pproch Recent AlphGo Zero pproch 1

2 2 Alternting move, zero-um, 2-plyer gme Ordinry bellmn eqution Two plyer å + = ' ', ', mx V P R V g å + = ' min mx ', ', mx V P R V g å + = ' mx min ', ', min V P R V g V min nd V mx Vmin i, by definition, the negtive of Vmx No need to tore two eprte vlue function Cn tore jut one nd flip the ign bed upon who i plying

3 Originl AlphGo Ingredient Policy network trined by upervied lerning Policy network trined by policy grdient Vlue function network Monte Crlo Tree Serch MCTS 3

4 Supervied Policy Network Similr to neuro-gmmon jut tried to mimic humn move CNN with oftmx output lyer Trined on 30 million expert move 57% ccurcy on held out tet et Seem low, but keep in mind tht thi not binry prediction problem, o thi i wy better thn flipping coin Alo trined fter rollout network tht got 24% ccurcy ued thi for ft rollout RL policy network Ued me network tructure upervied network Initilized to me weight upervied network Trined uing policy grdient Trined initilly gint upervied network, then gint previou verion of the RL policy network with ome rndomiztion I thi correct? NO! Still worked very well! 80% win rte gint upervied network 85% win rte gint Pchi, very trong MCTS progrm t the time 2 mteur dn Surpriing tht it plyed thi well with no lookhed 4

5 Vlue network Should predict probbility of win given bord poition Auming both plyer ply the me policy I thi reonble thing to ume? Tried trining on the humn dtbe, but thi didn t do well No ingle policy ued? Reltively mll mount of trining dt, o overfit Pper mention correltion, but not cler Trined on dt generted from the RL policy network Did reonbly well: MSE Did not overfit bdly MCTS Doe MCTS with explortion bonu The firt time new node i encountered: Evluted by the vlue network, AND A rollout uing the ft, upervied policy network for ech ction Initilize Q-vlue etimte for ech ction Weight explortion bonue by prior probbilitie = policy network ditribution Reult mixed uing voodoo contnt When time i up done erching Algorithm chooe the mot viited node from the root Ignore the vlue of the root, though mot viited node will tend to hve high vlue 5

6 Obervtion Supervied lerning network trined on humn move w better for rollout thn RL network, even though RL network produced tronger plyer Why? Author rgue tht upervied network covered the pce more, but tht not rigorou rgument Performnce On ingle CPU, AlphGo ignificntly outperformed ll vilble computer plyer ee figure 4 in pper Rollout, the vlue network, nd the policy network were ll individully pretty trong Combining them mde them tronger Ditributed verion bet the bet Europen go plyer 6

7 Comprion with Deep Blue Deep Blue bet computer che plyer Deep blue ued little/no mchine lerning Evluted more bord poition more brute force Supervied nd reinforcement lerning reduce erch pce Lerned vlue function nd rollout initilize new leve w/reonble vlue Le time pend on exhutive erch AlphGo Hitory Nture pper publihed in Jnury 2016 Plyed bet living humn go plyer, Lee Sedol, poibly one of the tronget go plyer ever in Mrch 2016 Ued between 1K nd 2K CPU, GPU, nd poibly Google new TPU not cler if thee were counted GPU Won 4/5 gme Subequently reveled tht Vlue network w tuned by elf ply Ued bigger NN nd 48 TPU Ply w decribed urpriing nd originl Mde move tht were unexpected t firt, but mde ene in hindight 7

8 AlphGo Hitory II Plyed the current bet go plyer Ke Jie in My 2017 Won 3/3 gme Livetrem w cenored in Chin: Surpriing thing bout AlphGo Rollout with ft, upervied policy better thn rollout with RL policy enble more erching? RL policy ued only indirectly to trin vlue function Serchle ply w urpriingly good for both RL policy nd V Contrt with che: Serch eem eentil for reonble che ply Rollout le helpful Deep blue w hrdwre triumph AlphGo i n AI/ML triumph 8

9 Originl AlphGo Zero Wht i AlphGo Zero Announced in mid October 2017 with much hype Lern to ply with zero humn knowledge No obviou reltionhip to Coke Zero 9

10 Difference from AlphGo Clic I Ue ingle convolutionl network to propoe ction oftmx nd produce vlue etimte Firt time node i viited: NN ign vlue nd probbilitie of ction me clic? No hndcrfted feture were thee previouly dicued? No rollout Ued Jut ingle mchine with 4 TPU Difference from AlphGo Clic II Exploit rottion nd reflection invrince did clic do thi? When gme end: Neurl network i trined to mximize imilrity between predicted nd ctul outcome for ech tte in the gme Network i lo trined to mke ction probbilitie conitent with MCTS ction election probbilitie 10

11 Trining Million of gme of elf-ply 64 GPU, 19 CPU Surped the verion tht defeted Lee Sedol fter jut 36 hour of trining After ~30 dy, urped AlphGo Mter verion tht hd been beting top humn mter 60-0 online See figure 6 Remrkble thing bout AlphGo Zero Compre w/td gmmon TD gmmon did pretty well without expert feture Needed expert feture to excel AlphGo Zero ued rw bord poition eentilly imge Advntge of convolutionl network? Independently lerned expert level knowledge of importnt bord/end gme configurtion Developed new pproche to known Go problem Simpler nd clener lgorithm thn clic Reduced computtionl reource t execution time 11

12 Why thi mtter Previouly viewed chllenge problem for AI Previou chllenge problem were olved uing: Simple lgorithm nd mive hrdwre Deep blue Specil purpoe hrdwre nd lgorithm utonomou driving AlphGo Zero ue lmot no domin pecific humn knowledge: Rule of the gme Symmetrie I thi wterhed event for AI/ML? Could be Some quetion tht need to be nwered: MCTS help for other domin, but eem like prticulrly big win for Go Cn MCTS + RL be big of win for other domin? Humn Go bility leverged humn pttern recognition prowe W humn Go dominnce n rtifct of vnihing edge humn hd in pttern recognition? Are humn the right benchmrk Cn thi be done with le reource thn Google-like firm? Wht doe thi y bout generl intelligence? 12