DQL: A New Updating Strategy for Reinforcement Learning Based on Q-Learning

Transcription

1 DQL: A New Updting Strtegy for Reinforcement Lerning Bsed on Q-Lerning Crlos E. Mrino 1 nd Edurdo F. Morles 2 1 Instituto Mexicno de Tecnologí del Agu, Pseo Cuhunáhuc 8532, Jiutepec, Morelos, 6255, MEXICO cmrino@tlloc.imt.mx 2 ITESM-Cmpus Cuernvc, Pseo de l Reform 182-A, Col. Loms de Cuernvc, Temixco, Morelos, 62589, MEXICO emorles@cmpus.mor.itesm.mx Abstrct. In reinforcement lerning n utonomous gent lerns n optiml policy while intercting with the environment. In prticulr, in one-step Q-lerning, with ech ction n gent updtes its Q vlues considering immedite rewrds. In this pper new strtegy for updting Q vlues is proposed. The strtegy, implemented in n lgorithm clled DQL, uses set of gents ll serching the sme gol in the sme spce to obtin the sme optiml policy. Ech gent leves trces over copy of the environment (copies of Q-vlues), while serching for gol. These copies re used by the gents to decide which ctions to tke. Once ll the gents rech gol, the originl Q-vlues of the best solution found by ll the gents re updted using Wtkins Q-lerning formul. DQL hs some similrities with Gmbrdell s Ant-Q lgorithm [4], however it does not require the definition of domin dependent heuristic nd consequently the tuning of dditionl prmeters. DQL lso does not updte the originl Q-vlues with zero rewrd while the gents re serching, s Ant-Q does. It is shown how DQL s guided explortion of severl gents with selected exploittion (updting only the best solution) produces fster convergence times thn Q-lerning nd Ant-Q on severl testbed problems under similr conditions. 1 Introduction Reinforcement lerning is n on-line technique tht pproximtes dynmic progrmming. The externl environment is modeled s discrete-time, finite stte, Mrkov decision process. Ech ction is ssocited with rewrd. The tsk of reinforcement lerning is to mximize the long-term discounted rewrd per ction. Reinforcement lerning hs been recently pplied to multi gent settings. The min purpose is to coordinte gents to complete tsk. In coordintion problems, ech gent is responsible for portion of the problem, nd most of the time, decisions of n gent ffect other gents performnce or solution. Exmples include the solution of network routing problems in [6] nd coordintion L. De Redt nd P. Flch (Eds.): ECML 21, LNAI 2167, pp , 21. c Springer-Verlg Berlin Heidelberg 21

2 DQL: A New Updting Strtegy for Reinforcement Lerning 325 gmes such s soccer [7]. Multi gent reinforcement lerning hve lso been used in pursuit gmes, where hunter tries to cpture prey. In these problems, gents shre senstions of the loction of the prey, communicte its loction to its prtners nd updte their reltive loction in order rech the prey [13]. Price nd Boutilier [1] proposed method clled implicit imittion. In this pproch pprentice gents lern from the experience of mentor gents bout its own cpbilities in unvisited prts of the spce. Imittion is performed extrcting model from the experienced gent behvior. This pproch ws proved in the solution of mzes using model bsed reinforcement lerning lgorithms, speeding lerning drmticlly. Other interesting problems, solved using multi gent reinforcement lerning, re those known s n-plyer coopertive repeted gmes. In these problems gents interct in limited resource environment selecting ctions tht mximize rewrd. The chosen ctions constitute joint ction. Ech joint ction is ssocited with rewrd function; the decision problem is coopertive since there is single rewrd function reflecting the utility ssessment of ll the gents. Agents must cooperte in order to select those ctions representing the mximl individul nd tem benefit. Some pproches to estblish coopertive behvior between gents for these kind of problems include [1,2,5]. In most of these pproches, single gent reinforcement lerning methods re pplied without much modifiction. In this pper, we propose n lterntive strtegy for updting vlue functions. The min motivtion behind this reserch is to improve the convergence times of Q-lerning with distributed reinforcement lerning setting, where set of gents hve the sme gol, nd together cooperte by leving trces to find n optiml policy for the sme problem. The hypothesis is tht using more explortion with set of gents nd controlled exploittion, by leving trces between gents nd reinforcing only the best solution proposed by the gents, produces fster convergence times. DQL performnce ws compred ginst Q-lerning [14] nd Gmbrdell nd Dorigo s Ant-Q lgorithm [4], which is distributed reinforcement lerning lgorithm used in the solution of the trveling slesmn problem. The three lgorithms were tested on severl problems over the whole rnge of the α nd γ prmeters used in the Q-lerning formul. It is shown tht DQL hs fster convergence times thn one-step Q-lerning nd Ant-Q under similr conditions. The pper is orgnized s follows. Section 2 gives brief overview of Q- lerning nd Ant-Q. Section 3 describes DQL. Section 4, presents the four test problems used to mesure the lgorithms performnce nd discusses the min results. Finlly, Section 5 concludes nd gives future reserch directions. 2 Q-Lerning In this study, ech reinforcement lerning gent uses the one-step Q-lerning lgorithm [14]. Its lerned decision policy is determined by the stte-ction pir vlue function, Q(s, ), which estimtes long-term discounted rewrds for ech stte-ction pir. Given current stte s Snd vilble ctions i A s, Q-lerning gent selects most of the time n ction with the highest estimted

3 326 C.E. Mrino nd E.F. Morles Q(s, ) nd with smll probbility ε, selects n lterntive ction. The gent then executes the ction, receives n immedite rewrd r, nd moves to the next stte s. In ech step, the gent updtes Q(s, ) by recursively discounting future utilities nd weighting them by positive lerning rte α: [ ] Q(s, ) Q(s, )+α r + γ mx Q(s, ) Q(s, ) A s where ( γ 1) is discount prmeter. As n gent explores the stte spce, its estimte Q improves grdully, nd, eventully, ech mx A Q(s, ) pproches: E { } s n=1 γn 1 r t+n. Here rt is the rewrd received t time t due the ction chosen t time t 1. Wtkins nd Dyn [15] hve shown tht this Q-lerning lgorithm converges to n optiml decision policy for finite Mrkov decision process. (1) 2.1 Ant-Q An interesting distributed reinforcement lgorithm, originlly proposed by Gmbrdell nd Dorigo [4], is Ant-Q. Ant-Q ws used to solve trveling slesmn problems nd cn be seen s n improvement over previous system clled nt systems [3]. The generl ide of Ant-Q is to use set of gents serching for the sme best policy. Following n nlogy with nt colonies, ech gent updtes its Q vlues, s in Q-lerning, but without considering ny rewrd (r = in Eq. 1), fter executing ech ction. This updting represent trces tht cn be followed by other gents. Once ll the gents rech gol (n episode), stte-ction pir evlution functions of the best solution re updted using delyed rewrd (r ) s expressed in Eq. 1. This mens tht some Q vlues will be updted severl times on ech episode, first without rewrds by ll the gents tht followed the sme stte-ction pir, nd once more with rewrds if the stte-ction pir is prt of the best pth. This repeted updting is not clerly justified nd is difficult to prove if the convergence properties of Q-lerning still hold. Ant-Q introduced severl dditionl mechnisms to the Q-lerning frmework. In prticulr, the selection policy is defined s combintion of domin dependent heuristic function (HE(s, )) nd the best Q-vlues. This combintion introduces two new prmeters (δ nd β) tht estimte the relevnce of HE(s, ) with respect to Q(s, ) vlues nd tht need to be tuned for ech prticulr ppliction domin. In generl n ɛ-greedy strtegy is used nd HE is combined with Q vlues s follows: rgmx {AQ(s, ) δ HE(s, ) β }. 3 DQL DQL follows similr ides of Ant-Q but without loosing the min properties of Q- lening nor introducing extr prmeters or heuristics. The generl ides, nd min differences with Ant-Q, re tht it does not use ny domin dependent

4 DQL: A New Updting Strtegy for Reinforcement Lerning 327 heuristic (nd consequently no dditionl prmeters) nd it updtes the Q- vlues only once (for the best solution found by ll the gents). DQL llows more explortion, s severl gents re serching t the sme time, nd promotes better exploittion, since the updtes on the Q-vlues re performed only over the best solutions 1. All the gents hve ccess to temporry copy of the stte-ction pir evlution functions (Q C (s, )). Ech time n gent hs to select n ction, it looks t this copy nd decides, bsed on its informtion, which ction to tke. Once the gent performs the selected ction, it updtes the copy of the stte-ction vlue pir using Eq. 2, where Q C (s, ) represents copy of the originl Q(s, ) pirs. [ ] Q C (s, ) Q C (s, )+α γ mx Q C (s, ) Q C (s, ) (2) A s This is similr to wht Ant-Q does, however in this cse the updtes re performed over copies of the originl Q vlues nd the originl Q-vlues re consequently not ffected t this stge. All the gents re moved one step t time, updting nd shring their common Q C vlues until reching stopping criterion. The gents use the copies of the Q vlues to decide which ctions to tke following n ɛ greedy policy. When ll the gents hve found solution the Q vlue copies re discrded nd the stte-ction pirs considered in the best solution receive rewrd which reinforce their vlues ccording to Eq. 1. This updtes the originl Q-vlues from which new copy is creted for the next cycle. The whole process is repeted until reching termintion criterion (see Tble 1). Tble 1. DQL lgorithm. Initilize Q(s, ) rbitrrily Repet (for n episodes) Initilize s, copy Q(s, ) toq C(s, ) Repet (for ech step of episode) Repet (for m gents) Tke ction, observe r, s Q C(s, ) Q C(s, )+α [γ mx Q C(s, ) Q C(s, )] s s ; Until s is terminl Evlute the m proposed solutions Assign rewrds to the best solution found nd updte the Q vlues: Q(s, ) Q(s, )+α [r + γ mx Q(s, ) Q(s, )] 1 In the tested problems, the best solution of one episode is the shortest pth found by one gent in tht episode.

5 328 C.E. Mrino nd E.F. Morles All the gents ct on the sme environment nd hve ccess to the sme Q nd Q C vlues. The copies of the Q vlues re used s guidnces to the gents of wht seems to be promising sttes. However, only the best solution found by ll the gents receives n ctul rewrd. There re two min differences with respect to Ant-Q: Prtil updtes re performed over copies of the Q-vlues voiding multiple updtes with nd without rewrds. There is no need to define domin dependent heuristic or to tune extr prmeters s in Ant-Q. The min motivtion behind DQL is tht it llows: More explortion s more gents re used during serch More exploittion s only relevnt (best) solutions re effectively rewrded The hypothesis is tht this lterntive strtegy for updting vlue functions chieves, in generl, fster convergence times thn one-step Q-lerning, regrdless the vlues of α nd γ. To test this hypothesis, we performed severl experiments over four problems with different complexity nd nture, compring Q-lerning, Ant-Q nd DQL performnce. Although, the tests were performed of deterministic stte trnsition domins, our frmework cn lso be pplied to stochstic stte trnsition functions. 4 Experimentl Results All the experiments were performed on the sme mchine nd the lgorithms were similrly coded by the sme uthor 2. Although DQL nd Ant-Q use multiple gents the lgorithms re implemented sequentilly. Two mze problems were first considered s they re problems where Q- lerning normlly shows good performnce. For these problems the lgorithms were tested over ll possible α nd γ vlues with.25 increments. ε =.1 ws considered for the three lgorithms in both mze problems. Ech experiment ws performed thirty times nd we report the men CPU time, men number of episodes, nd men number of steps per episode 3. Algorithm execution stops when the optiml policy (solid lines in Figure 1) is reched in five consecutive episodes. As mentioned erlier, the Ant-Q lgorithm ws designed for the solution of trveling slesmn problems (TSPs). Two TSP instnces previously solved with Ant-Q re lso included in the tests. The sme prmeter vlues nd stopping criteri used with Ant-Q were used for DQL nd Q-lerning. Tbles of results include best solution found, stndrd devition of solutions, the men of ll the solutions, nd the men CPU time to rech the stopping criterion. In this cse, every lgorithm ws executed 15 times over 2 episodes. 2 All lgorithms re coded in C++. 3 The men number refers to the men of ll the solutions found t prticulr episode.

6 DQL: A New Updting Strtegy for Reinforcement Lerning Grid World with Wind The first experiment ws run on the windy grid world shown in Figure 1 left. The objective is to find the optiml pth from S to G considering wind force, which shifts upwrds the resulting stte when moving horizontlly, the strength of which vries from column to column s shown t the bottom of Figure 1. For instnce, moving horizontlly (either left or right) from squre which hs wind force of 1 (indicted t the bottom of Figure 1), cuses the gent to move one squre bove its intended destintion. However, moving verticlly (either down or up) does not produce ny effect 4. Ant-Q nd DQL were both run with 3 gents. Ant-Q ws tested with nd without heuristic. The results re shown in Figures 2 nd 3 without heuristic for Ant-Q. Restrictions S optiml policy G ctions S G ctions optiml policy Fig. 1. Grid world in which horizontl movement is ltered by loction-dependent upwrd wind (left) nd windy world with restrictions (right). Results from the three lgorithms re plotted using three different line types, dshed for Ant-Q, dsh-dot for Q-lerning, nd continuous for DQL. There re four lines for ech lgorithm, one for ech vlue of γ. The + symbol correspond to γ =.25, to γ =.5, to γ =.75, nd to γ =1.. Figure 2 shows the men CPU time required for ech lgorithm to rech the stopping criterion. For Ant-Q nd DQL this time corresponds to the men time required for ll the solutions found t ech episode. As cn be seen from the results, both Ant-Q nd DQL clerly outperformed Q-lerning for ll the tested vlues of α nd γ, significntly reducing the convergence times. Figure 3 shows the men number of episodes nd the men number of steps per episode required for the three lgorithms to rech the stopping criterion. For these two metrics DQL performnce ws the best of the three lgorithms, nd Q-lening ws ble to outperform Ant-Q for α.5. The previously described results re for Ant-Q without using ny heuristic. When Ant-Q ws tested using s heuristic the inverse of the Mnhttn distnce, 4 An gent is not llowed to move outside the borders.

7 33 C.E. Mrino nd E.F. Morles.9 Grid world.8 Q-lerning.7.6 Ant-Q men CPU time DQL Fig. 2. Men CPU time in seconds to rech the optiml solution five consecutive episodes. 2 Grid world 18 Grid world Q-lerning DQL Ant-Q men number of episodes men number of steps episode Ant-Q Q-lerning 4 4 DQL Fig. 3. Men episodes (left) nd steps per episode (right) required to rech the optiml solution. it ws not ble to converge 5 with three different combintions of (δ, β): (1, 1) Q- vlues nd heuristic function eqully importnt, (2, 1) Q-vlues more importnt thn the heuristic function, nd (1, 2) heuristic function more importnt thn Q- vlues. Although, the heuristic used my be resonble for some mze problems, it is cler tht in generl, finding suitble heuristic my be very difficult tsk. We lso decided to test vrint of DQL, clled DQL-2, where ech gent performs complete episode before strting with the next gent. Tht is, performing m episodes in sequence without shring informtion while performing the tsk. Figure 4 compres the men CPU times of DQL (here s DQL-1) 5 Rech the optiml policy five consecutive episodes before reching 5, trnsitions.

8 DQL: A New Updting Strtegy for Reinforcement Lerning 331 ginst this episodic updting pproch (DQL-2). As it cn be pprecited in the figure, shring informtion while performing tsk reduces convergence times. Although not shown in the pper, due to restrictions in spce, similr results were observed in the other problems..7 Grid world.6 g=.25 DQL-1 DQL-2.5 g=.5 men CPU time.4.3 g=.75 g= g=.25 g=.5 g=.75 g= Fig. 4. Men CPU times in seconds between DQL nd n episodic vrint (DQL-2) We lso mesured the verge number of totl updtes of Q vlues (Q C + Q) in DQL ginst the number of Q updtes of Q-lerning (see Figure 5). As cn be seen in the figure, lthough DQL updtes lrger number of totl Q- vlues, it converges fster. We believe tht the extr informtion shred by the gents during the process helps to reduce convergence times. Similr behvior ws observed on the other problems. 15 Q-Lerning vs DQL-1 1 Q(s,) updtes DQL-1 Q-Lerning Episode Number Fig. 5. Averge number of totl Q-vlue updtes for DQL nd Q-lerning.

9 332 C.E. Mrino nd E.F. Morles 4.2 Grid World with Wind nd Trp This problem ws designed to generte more difficult mze. An obstcle blocking the optiml pth is included to the windy grid world, forcing gents to serch for n lterntive route. Figure 1 right shows the mze nd the optiml policy tht gents must find. The sme opertion conditions nd prmeters used in the previous mze were considered. Figures 6 nd 7 show the mesures for the three metrics obtined with the three lgorithms under study. Agin, the figures show only the performnce of Ant-Q without heuristic s it ws not ble to converge with the Mnhttn distnce heuristic. In Figure 6 it cn be observed tht Ant-Q is ble to outperformed DQL men CPU time for some combintions of α nd γ: (α =.25,γ.25), α =.5,γ =.5, nd (α =1,γ =.5, 1.). It shows, however, to be much more dependent on the vlues of these prmeters. On the other hnd, DQL shows more stble behvior in reltion to the number of episodes nd steps per episode required for the gents to stisfy the stopping criterion. 3.5 Restricted Grid world 3 Q-lerning 2.5 men CPU time Ant-Q DQL Fig. 6. Men CPU time in seconds to rech the optiml solution five consecutive episodes. 4.3 Trveling Slesmn Problems Ant-Q ws originlly developed to solve instnces of TSP. For them, the uthors of Ant-Q reported results where Ant-Q outperformed severl lterntive lgorithms. We took two instnces of TSP with the sme settings used in the originl Ant-Q pper [4]. The first problem is the 3 cities symmetric TSP known s Oliver3 proposed in [9], nd the second problem is the 48 cities symmetric TSP known s Ry48p proposed in [11]. Ant-Q prmeters for the pseudo rndom proportionl ction choice rule were the sme used by Gmbrdell, tht is, β =2., δ =1., α =.1, γ =.3, nd HE(i, j) =1/d i,j, being i, j cities nd d i,j the distnce between them. For

10 DQL: A New Updting Strtegy for Reinforcement Lerning Restricted Grid world 3.5 x 14 Restricted Grid world Q-lerning 1 Q-lerning 3 men number of episodes men number of steps episode Ant-Q Ant-Q 2 DQL.5 DQL Fig. 7. Men episodes (left) nd steps per episode (right) required to rech the optiml solution. DQL nd Q-lerning, the sme vlues for the α nd γ prmeters were used. Q(s, ) vlues were initilized to the inverse of the number of cities times the verge length of edges, nd n ε-greedy selection policy, with ε =.1 6. The lgorithms considered 2 steps or trnsitions for Oliver3 problem, nd 6 for ryp48. The number of gents were the sme in DQL nd Ant-Q, 3 for Oliver3 nd 48 for Ry48p. For Q-lerning single gent ws used. The performnce ws evluted repeting ech tril 15 times. We report the verge performnces. The CPU time correspond to the verge running times to rech the best result. Tble 2. Results for the two TSPs. Oliver3 Ry48p Best DQL Men Std. Dev CPU Q-lerning Men Std. Dev CPU Ant-Q Men HE Std. Dev CPU Ant-Q Men no HE Std. Dev CPU This policy is equivlent to consider q =.9 in the originl Ant-Q lgorithm

11 334 C.E. Mrino nd E.F. Morles In the TSP problems ll strtegies found the sme best solution. Although Ant-Q ws specilly tuned for this type of problems it did not show the best performnce. It is lso interesting to note tht Q-lerning show the lowest verge solution for Ry48p. It is however cler from the results, tht the verge convergence times of DQL re much smller thn the other strtegies. In these prticulr cses, the heuristic function dded to Ant-Q ws useful for reducing convergence times nd stndrd devitions (which ws not the cse for the grid world problems). 5 Conclusions nd Future Work This pper introduces new strtegy for updting Q vlues implemented in n lgorithm clled DQL. DQL uses set of gents serching the sme gol in the sme spce. Trces (copies of Q-vlues updted without rewrds) re used to guide the explortion of gents. The originl Q vlues of only the best solution found by ll the gents is updted using the one-step Q-lerning formul. It ws shown how DQL s guided explortion of severl gents with selected exploittion (updting only the best solution) produces fster convergence times thn Q- lerning nd Ant-Q on severl testbed problems under similr conditions. The heuristic nd extr prmeters needed by Ant-Q does not seem to be producing ny benefits. Additionlly, selecting good heuristic cn be difficult tsk. DQL, on the other hnd, does not require extr prmeters nd shows, in generl, better convergence times. DQL updting strtegy is performed only on the best pth mong m solutions. In order to preserve the convergence properties of Q-lerning, we need to show tht ll the Q-vlue stte-ction pirs hve non zero probbility of being updted. This is prt of our future work. Before trying prllel version, which seems nturl extension, we would like to perform more tests nd compre the results ginst different strtegies for updting Q vlues, such s Monte Crlo. It will lso be interesting to run DQL without updting the copies of the Q-vlues to ssess its influence in the results (which is like running Q-lerning severl times without updting nd then updte the best results found so fr). Acknowledgments. Thnks to the nonymous reviewers for their helpful comments on the initil drft of this pper. The first uthor ws supported by IMTA. This reserch ws supported by CONACyT under grnt 33-A. References 1. Boutilier, C.: Sequentil Optimlity nd Coordintion in Multi gent Systems, In Proc. of IJCAI-99, Stockholm, Sweden, Clus, C., Boutilier, C.: The Dynmics of Reinforcement Lerning in Coopertive Multigents Systems, In Proc. of AAAI-97 Multigent Lerning Workshop, pg , Providence, 1997.

12 DQL: A New Updting Strtegy for Reinforcement Lerning Dorigo, M.: Optimiztion, Lerning, nd Nturl Algorithms, PhD thesis, Politecnico d Milno, Itly, Gmbrdell, L., M., Dorigo, M.: Ant-Q: A reinforcement Lerning Approch to the Trveling Slesmn Problem, In Proceedings of the 12th Interntionl Conference on Mchine Lerning, pp , Morgn Kufmnn, Hu, J., Wellmn, M.: Multigent Reinforcement Lerning: Theoreticl Frmework nd n Algorithm, In Proc. 15th Int. Conf. on Mchine Lerning, pp , Morgn Kufmnn, Littmn, M., Boyn, J.: A Distributed Reinforcement Lerning Scheme for Network Routing, In Proc. Int. Workshop on Applictions of Neurl Networks to Telecommunictions, pp , J. Alspector, et l., (eds.), Lwrence Erlbum, Hillsdle, NJ, Littmn, M.: Mrkov Gmes s Frmework for Multigent Reinforcement Lerning, In Proc. 11th Int. Conf. on Mchine Lerning, pp , New Brunswick, NJ, 1994, Morgn Kufmnn. 8. Mrino, C., Morles, E.: A New Distributed Reinforcement Lerning Algorithm for the solution of Multiple Objective Optimiztion Problems, In O. Ciro et l., eds. Lecture Notes in Artificil Intelligence, 1793: , April Oliver, I., Smith, D., Hollnd, J.R.: A study of Permuttion Crossover Opertors on the Trveling Slesmn Problem, In Proc. 2nd Int. Conf. n Genetic Algorithms, pp , J.J. Grefenstette (ed.), Lwrence Erlbum, Hillsdle, NJ, Price, B., Boutilier, C.: Implicit Imittion in Multigent Reinforcement Lerning, In Proc. 16th Int. Conf. on Mchine Lerning, pp., Reinelt, G.: The Trveling Slesmn: Computtionl Solutions for TSP Applictions, Springer Verlg, Berlin, Sutton, R., Brto, A.: Reinforcement Lerning n Introduction, MIT Press, Cmbridge, MA, Tn, M.: Multigent Reinforcement Lerning: Independent vs. Coopertive Agents, In Proc. 1th Int. Conf. on Mchine Lerning, pp , Amherst, MA, Wtkins, C.: Lerning from Delyed Rewrds. PhD thesis, Cmbridge University, Cmbridge, MA, Wtkins, C., Dyn, P.: Q-Lerning, Mchine Lerning, 3: , 1992.