Reinforcement Learning by Policy Improvement. Making Use of Experiences of The Other Tasks. Hajime Kimura and Shigenobu Kobayashi

Transcription

1 Reinforcemen Learning by Policy Improvemen Making Use of Experiences of The Oher Tasks Hajime Kimura and Shigenobu Kobayashi Tokyo Insiue of Technology, JAPAN genfe.dis.iech.ac.jp, kobayasidis.iech.ac.jp Absrac. We propose a new reinforcemen learning algorihm based on a policy gradien mehodcombining wih imporance sampling. Sochasic gradien mehods can easily be applied o coninuous sae-acion conrol problems, and also i can improve policies under some class of POMDP environmens, however, he algorihms are limied o on-policy learning, i.e., he improvemen is execued by he experiences under he only arge policy. Imporance sampling echniques enable he gradien mehods o make use of all experiences for he learning. We develop he algorihm for coninuing (no episodic) asks under discouned reward crieria, and demonsrae hrough simulaions of a maze world and a wo-link arm robo problem. Inroducion Reinforcemen learning (RL) [] is a promising approach for robos o obain conrol rules hrough a process of rial and error. Learning conrol in real world applicaions is o deal wih boh coninuous sae-acion spaces, and parially observabiliy ha causes non-markovian effecs. Sochasic gradien mehods [7] are one approach o overcome such problems. In many policy gradien mehods, a parameerized sochasic policy is updaed according o he gradien ofhevalue funcion wih respec o he policy parameers. I is easy o implemenmulidimensional coninuous acion by using some coninuous disribuion (e.g. normal disribuion) for he policy funcion [][]. Sochasic gradien mehods can be classified o direc policy search algorihms [9] ha would no make use of Markov properies of he environmens. Therefore, he sochasic gradien mehods can improve policies under some class of POMDP environmens. A drawback of he convenional sochasic gradien mehods is ha he learning is oo slow. One reason is ha he algorihms are limied o on-policy learning, i.e., he improvemen can be execued by he experiences under he only arge policy. In many real-robo applicaions, he same robo mus experience rial and error o learn many policies for he respecive asks. Therefore, sharing he same experiences for he learning of all policies will save grea amoun of rials. Imporance sampling echniques enable he policy search algorihms o execue offpolicy policy improvemen[3], bu hese mehods are limied o applying episodic asks. In his paper, we propose a new reinforcemen learning algorihm based on a policy gradien mehod combining wih imporance sampling, and develop i for coninuing (no episodic) asks under discouned reward crieria. Background

2 . Environmen Model Le S denoe sae space, A be acion space, R be a se of real number. A eachdiscree ime, he agen observes sae s S, selecs acion a A, and hen receives an insananeous reward r Rresuling from sae ransiion in he environmen. In general, he reward and he nex sae may be random, bu heir probabiliy disribuions are assumed o depend only on s and a in Markov decision processes (MDPs), in which many reinforcemen learning algorihms are sudied. In MDPs, he nex sae s + is chosen according o he ransiion probabiliy T (s ;a;s + ), and he reward r is given randomly according o he expecaion r(s ;a). The agen does no know T (s ;a;s + )andr(s ;a) ahead of ime. The objecive of reinforcemen learning is o consruc a policy ha maximizes he agen's performance. A policy ß is defined as probabiliy disribuion over acion space under given sae. A naural performance measure for finie (or infinie) horizon asks is he cumulaive discouned reward: V = k= fl k r +k, () where he discoun facor,» fl<specifies he imporance of fuure rewards, and V is he value a ime. In MDPs, he value under policy ß is defined as a funcion of sae: V ß (s) =E " X fi # fl r fififi s = s; ß, () where Ef g denoes he expecaion. The objecive in MDPs is o find an opimal policy ha maximizes he value of each sae s defined by Equaion. In his sandard noaion of MDPs, he reward is defined as scalar, bu in his paper, we give reward signal as a vecor, and each componen in he reward vecor is given for each differen ask respecively, and each ask holds is policy.. Imporance Sampling Imporance sampling is a classical echnique for handling a mismach of disribuions, ha is, wewould like daa drawn from he disribuion of he arge policy ß o esimae values (or is gradien), bu he daa is drawn from he disribuion of he oher behavior policy ß. Le H = f< s ;a ;r ;:::s T ;a T ;r T ;s T + >g denoe he se of all possible experience sequence of lengh T +, and le a hisory h(s) H wih s = s. R(h) denoes he cumulaive discouned rewards: R(h) = P T fl r. Consider a sample hisory drawn from he disribuion induced by he behavior policy ß execuing muliple imes in he environmen. Using a sandard imporance sampling mehod [3] [5], he esimaed sae value is given by ^V ß (s) =R(h) Pr(hjß) Pr(hjß ) = R(h)ß ß ß T ßß ß = r + flr + fl r + fl T ß ß ß T r T T ßß ß, T (3) where ß denoes probabiliy (or densiy) of execuing acion a under given policy ß, r denoes he reward in he sequence h a ime. Inuiively, he likelihood raio ß ß ß T on reward r ß should no depend on he fuure afer ime. Based on his idea, ß ß T per-decision imporance sampling [] isshown by: ß ^V ß Y ß (s) = r ß = fl k r () flr ß ß ß ß flr T ß ß ß T ß ß ß T ß k= k

3 I is shown ha hese algorihms are unbiased, ha is, Ef ^V ß (s)g = V ß (s) where T. For he policy improvemen, we should calculae he gradien of he value funcion. Assume ha he parameerized policy funcion ß is given by he policy parameer, and he behavior policy ß is given by a parameer, hen ( ) ( Y V ß ß (s) =E fl k Q ) r ß =E fl k= ß k r Q (5) k k= ß k k= The calculaion of he gradien of he acion likelihood wih respec o he policy parameers Q k= ß k is given by: ky Y Y X ß k = ß k = ß k ln ß k. () k= ß k ln ky This calculaion can easily be done incremenallybyaccumulaing he eligibiliy ln ß. k= 3 Gradien Mehod using Imporance Sampling 3. The Algorihm We develop a policy improvemen algorihm based on he value gradien shown in he previous secion. Figure shows he ouline of he algorihm. When he behavior policy ß is equal o he arge policy ß, hen he algorihm is exacly he same as he sandard sochasic gradien mehods. Muliple policies can be applied o he arge policy, however, each policy needs memory space for he policy parameer vecor, is eligibiliy race μe() and discouned sum of he likelihood raio L(), ha is a scalar variable, respecively. Avery similar idea is already proposed by [8] excep for applying coninuing asks. The policy parameer vecor would be updaed appropriaely by hesumof ; In our experimens, is averaged according o μ +fl μ and updae policy parameers by + ff μ j μ j for each ime sep, where ff is a small posiive consan, and j jdenoes L norm. 3. Feaures Since he likelihood raio ß ß ß T is calculaed incremenally on each imesep ß ß ß T by L(), he learner does no need o hold sae-acion-reward sequences. I is incorporaed ino he calculaion of he eligibiliy race in Figure. Alhough he behavior policy is limied o one on each sep, muliple arge policies can be learned concurrenly. If he reward signal is given byavecor, and each componen of he vecor is assigned o he corresponding ask, hen he agen can learn muliple asks from he same rial-and-error experiences. Especially, when all policies are similar a he early sage of he learning, he policies can share he same experiences, and he learning efficiency will be proporional o he number of he asks. However, he learning proceeds and he differences of he acion selecion probabiliy beween he asks are geing large, hen he weigh of sharing he experiences ends o ge small because he likelihood raio ß =ß would approach o zero. k=

4 . Observe sae s,choose acion a wih probabiliy (or densiy) following a behavior policy ß (a ; ;s ), and perform i.. Calculae ß ha is a probabiliy of selecing acion a under he condiion of he arge policy ß in he sae s. 3. Calculae he eligibiliy of he arge policy parameer : e() = ln ß. Updae he eligibiliy race according o: L() = ( + fll( )) ß ß μe() = L()e()+fl ß ß μe( ), (7) where fl is a discoun facor, and L() is an accumulaed likelihood raio. 5. Observe immediae reward r and updae policy parameers: =(r b)μe(), where b is a reinforcemen baseline parameer.. Le +, and go o sep. Figure : The policy improvemen algorihm combined wih imporance sampling. 3.3 Analysis of he Algorihm As sown in Figure, he discouned sum of he likelihood raio L() isgiven by L() =(+fll( )) ß ß = ß ß + fl ß ß ßß + fl ß ß ß ßß ß + + fl ß ß ß ßß ß (8) The backups of he policy parameer are given by =(r b)μe() =(r b) =(r b) L()e()+fl ß ß μe( ) L()e()+fl ß ß L( )e( ) + + fl ß ß ß ßß ß L()e() (9) From Equaion 8 and 9, we ge " ß = (r b) e() ß + fl ß ß ßß + fl ß ß ß ßß ß +fle( ) ß ß ß +fle( ) ß ß ß ß ß ß + fl ß ß ß ß ß + + fl e() ß ß ß ßß ß = (r b) " ß ß + fl ß ß 3 ß ß 3 ß ß + + fl ß ß ß ßß ß + + fl ß ß ß ß ß ß # + + fl 3 ß ß 3 ß ß ß 3 ß e()+fl ß ß ßß (e()+e( )) + fl ß ß ß ßß ß (e()+e( ) + e( ))

5 # + + fl ß ß ß ßß ß (e()+e( ) + + e()) () By Equaion and, he sum of all backups is given by = (r b) ß ß e + + fl T (r T b) ß ß ß T ßß ß (e + + e T ) T = +(r b) ß ß e + + fl T (r T b) ß ß ß T ßß ß (e + + e T ) T + +(r T b) ß T e T = k= ß T fl k (r k b) ßß + ß k k= fl k (r k b) ß ß + ß k ßß + ß (e + e e k ) k (ß ß + ß k )= V () The las ransformaion of Equaion is obained by he derivaive of Equaion wih respec o. Equaion 5 and derive he following resuls: E ( lim T ) = E ( lim T V ) = V ß (s ) () I is shown ha he arge policy parameer can be improved owards he sum of he gradien of he value funcion on each ime sep. Noe ha his resul does no depend on he Markov propery, herefore i can be exended o non-markovian environmens. Experimens. Maze World: A MDP Environmen s s s s 3 s s 5 s s 7 s 8 s 9 s s s s 3 s s 5 s s 7 s 8 s 9 s s s s 3 - -? sae S : saes acion A :Tomove Norh, Eas, Souh or Wes ( acions) The agen canno move across he wall. When he agen eners any goal, he agen jumps all saes wih uniform disribuion. Reward R a (s; s ) = if he agen eners he goal, or oherwise. The symbol O denoes he agen. The curren sae is, s = s. Figure : A maze problem wih four asks. Task : Goal if Souh is seleced a s 9,Task : Goal if Eas is seleced a s,task : Goal if Eas is seleced a s,task 3: Goal if Norh is seleced a s. The algorihm is demonsraed in a maze world wih four asks as shown in Figure. In he imporance sampling by swiching policies, he agen changes is behavior policy for Task (policy ), Task (policy ), Task (policy ) or Task 3 (policy 3) in urn a inervals of 5 seps. In he imporance sampling by a random policy, he agen keeps is iniial behavior policy. In he imporance sampling mehods, he learning for all policies are concurrenly execued sharing he same experience under

6 he behavior policy. I is compared wih a simple policy gradien mehod, in which he learning is execued only in he arge policy ha equals o he behavior policy. Tha is, he policy in he simple gradien mehod is updaed by using he daa of only 5 seps whereas he agen performs seps. The simple policy gradien mehod can be easily arranged by modifying our algorihm o ß = ß for each ask. The agen observes he posiion in he maze ( saes), and selecs one acion from Norh, Eas, Souh, or Wes. When he agen eners one of he goal, he only corresponding policy receives posiive reward r =,orr = for oher policies, and hen he agen jumps nex sae wih uniform disribuion. The agen calculaes acion-probabiliy of policies in he curren sae s by: P j (a i js) = exp( j i (s)) exp( j (s)) + exp( j (s)) + exp( j (s)) + exp( j 3(s)), where P j (a i js) denoes he probabiliy of selecing acion a i in sae s following he policy ß j,and j i (s) is he policy parameer of ß j. If he acion a i is seleced, he eligibiliy e j (s) for he policy parameer k j i (s) isgiven by ( e j k(s) = P j k(s) ln P (a j (a j i js) = i js) if k = i P j (a i js) if k = i We adoped his scheme o he calculaion of ß, ß and e() in Figure. The simulaion resuls are shown in Figure 3. The discoun facor fl equals :9 and learning rae ff equals : for boh algorihms. The learning performance of he proposed mehod is abou wo or hree imes as good as ha of he sandard gradien mehod, excep for Task. The reason is ha since he goal of Task is locaed in he deph of he maze, he number of visiing o he goal of Task ends o small.. Locomoion Tasks on a Two-link Arm Robo: Applying Coninuous Acion Space We applied he algorihm o a robo shown in Figure. The objecive of learning is o find conrol rules o move forward (Task ) or backward (Task ). The joins are conrolled by servo moors ha reac o angular-posiion commands. A each ime sep, he agen observes angular-posiion of wo moors and a ouch sensor aached o he leg op according o (ffi ;ffi ;ffi 3 ), where each componen is normalized beween» ffi ;ffi», and selecs acion according o he behavior policy. The angularposiion ffi ;ffi is defined in wo dimensional coninuous space. The ouch sensorffi 3 akes only or. The acion is a desinaion of angular-posiion. Le a vecor (a () ;a () ) be he desinaion of angular posiions of wo-moors as an acion oupu. I is also defined in wo dimensional coninuous space. When he agen selec acion, he he moors move owards he commanded posiions. When he join-angles move o he commanded posiion, or changing he sensor variables, hen he reward is given as he resul of he ransiion, and he ime sep proceeds o he nex sep. When he sensor variable changes in he way of moving join-moors, he angular-posiion would no correspond o he desinaion posiion. For his reason, here exiss uncerainy of he sae ransiion. For Task, he immediae reward is he disance of he body movemen caused by he previous acion. When he robo moves backward, he policy for Task receives negaive reward. For Task, all rewards are simply given by invering he sign of he reward of Task. The funcion of ß and ß are given by Gaussian: The acion a (i) for he moor (i) is seleced following normal disribuion N(μ i ;ff i ), where

7 Averaged Reward ( seps) 8 Averaged Reward ( seps) Averaged Reward ( seps) 8 Averaged Reward ( seps) Figure 3: The performance of he algorihm compared wih a sandard gradien mehod in he maze world, averaged over rials. The errorbar denoes he sandard deviaion. The op lef shows Task, op righ istask, boom lef is Task, boom righ istask 3. μ i ==( + exp( P k x k i;k )andff i ==( + exp( i ). In his experimen, we ried wo ways of generaing he feaure vecor X: One is broad sae generalizaion ha akes X =(x ;:::x )=(ffi ;ffi ;ffi 3 ; ffi ; ffi ; ffi 3 ). The oher is a fixed-grid approximaion, which consiss of 8x8x boxels over he sae space. The deails of he calculaion for ß, ß and e() are mosly omied here for he reason of he limied space, bu i is he same as he implemenaion for a four-legged robo in our works []. In his experimen, we use he same parameers in all he algorihms, fl = :9 and ff = :. Figure 5 and show learning curves for he robo on he ask and. The imporance sampling mehods are oo bad and quie unsable wih using broad sae generalizaion. Using he grid approximaion, he proposed mehods ouperformed han he sandard gradien mehod, however, he acquired conrol rules are inferior o hose of he sandard mehod wih using broad sae generalizaion. 5 Discussion and Fuure Work We propose a policy gradien mehod combining wih imporance sampling. I enables off-policy policy improvemen under he environmens ha are coninuing (nonepisodic) asks, and/or coninuous-acion space. The effec of sharing he experiences among differen policies is clearly observed in he maze problem. However, i is weak when he he agen adops broad sae generalizaion as he sae represenaion. The reason is ha since he sae occupancy desribuion is differen beween off-policy and on-policy, he off-policy learning wih broad sae generalizaion ends o spoil good

8 Figure : The lef is a real wo-link robo. The righ-side shows a simulaor of he robo used in his experimen. The sae-acion space is coninuous Averaged Reward ( seps).5.5 Averaged Reward ( seps) Figure 5: Learning curves of he algorihms using broad sae generalizaion averaged over rials on he robo simulaor. The lef is Task, he righ is Task. The imporance sampling mehods are inferior o he sandard gradien mehods. acon-disribuion around on-policy saes. One way o avoid his undesirabiliy is o use localized sae represenaion such as grid approximaion mehods. However, he grid approximaion mehods cause problems arising from curse of dimensionaliy. Designing appropriae sae represenaion for imporance sampling policy improvemen mehods in he high-dimensional sae space is challenging fuure work. We are rying o exend our mehod o an acor-criic algorihm ha esimaes sae value for each arge policy and makes use of he value for he policy improvemen. We also inend o apply he algorihm o a real one leg o four leg robos as fuure work. References [] Kimura, H. & Kobayashi, S.: An Analysis of Acor/Criic Algorihms using Eligibiliy Traces: Reinforcemen Learning wih Imperfec Value Funcion, Proc. of 5h Inernaional Conference on Machine Learning, pp.78 8 (998). [] Kimura, H., Yamashia, T. & Kobayashi, S.: Reinforcemen Learning of Walking Behavior for a Four-Legged Robo, h IEEE Conference on Decision and Conrol (CDC), pp. (). [3] Peshkin, L. & Shelon, C.R.: Learning from Scarce Experience, Proc. of 9h Inernaional Conference on Machine Learning (ICML), pp [] Precup, D., Suon, R.S. & Singh, S.: EligibiliyTraces for Off-Policy Policy Evaluaion, Proc. of he 7h Inernaional Conference on Machine Learning, pp (). [5] Shelon, C.R.: Policy Improvemen for POMDPs using Normalized Imporance

9 Averaged Reward ( seps).5.5 Averaged Reward ( seps) Figure : Learning curves of he algorihms using he grid sae approximaion (8x8x) averaged over rials on he robo simulaor. The lef is Task, he righ is Task. The imporance sampling mehods are ouperformed han he sandard gradien mehods. Sampling, Proc. of he 7h Conference on Uncerainy in Arificial Inelligence, (UAI) pp.9 53 (). [] Suon, R.S. & Baro, A.: Reinforcemen Learning: An Inroducion, A Bradford Book, The MIT Press (998). [7] Suon, R.S., McAlleser, D., Singh, S. & Mansour, Y.: Policy Gradien Mehods for Reinforcemen Learning wih Funcion Approximaion, Advances in Neural Informaion Processing Sysems (NIPS), pp (). [8] Uchibe, E. & Doya, K.: Reinforcemen Learning using Imporance Sampling for Muliple Learning Modules, Technical repor of he Insiue of Elecronics, Informaion and Communicaion Engineerings (3, in Japanese). [9] Williams, R.J.: Simple Saisical Gradien Following Algorihms for Connecionis Reinforcemen Learning, Machine Learning 8, pp. 9 5 (99).