Policy-contingent state abstraction for

Transcription

1 Policy-contingent stte bstrction for hierrchicl MDPs oelle Pineu nd Geoffrey Gordon School of Computer Science Crnegie Mellon University Pittsburgh PA Abstrct Hierrchiclly structured plnning problems often provide gret opportunities for stte bstrction: high-level plnners cn ignore fine detils while low-level plnners cn focus only on specific tsks. Most previous hierrchicl MDP lgorithms rely on hnd-designed knowledgeintensive stte bstrctions. We propose insted n utomtic lzy lgorithm which plns from the bottom of the hierrchy up nd finds good bstrctions for high-level plnners contingent on their subtsks optiml policies. This policy-contingent pproch cn often bstrct wy more detils thn humn specificlly becuse it hs ccess to more informtion (nmely its subtsks policies) when it decides whether to cluster sttes. We demonstrte tht our lgorithm finds better bstrctions thn humn designers on well-known plnning tsks. 1 Introduction Numerous decision-mking problems hve been solved by csting them s Mrkov Decision Processes (MDPs) [12]. MDPs offer powerful frmework for discovering nytimenywhere ction selection strtegies for gents cting in stochstic domins [8]. Mny domins however require very lrge number of sttes nd/or ctions to describe the problem nd consequently re beyond the rech of trditionl MDP solution methods. In n ttempt to ddress such problems divide-nd-conquer -type pproches hve been developed tht exploit structure in the domin to fcilitte solving. Approches such s MAXQ [5] HAM [11] OPTIONS [10] nd ALisp [1] leverge structure in the control (ction & policy) spce to prtition complex tsk into mny hierrchiclly-relted subtsks. This typiclly leds to fster policy-lerning however much greter speed-up cn often be obtined by lso introducing stte bstrction s done in MAXQ nd ALisp. In this cse both the hierrchy nd stte bstrction re fixed from the beginning prior to ny problemsolving. As result the stte bstrction must be gnostic: ssuming no knowledge of locl/prtil policies nd preserving sufficient resolution to represent ny policy. This pper proposes hierrchicl MDP lgorithm which insted uses policy-contingent stte bstrction. The ide is to interleve the bstrction nd the policy determintion strting with lower-level subtsks nd moving up the hierrchy. Abstrction in higher-level subtsks tkes plce fter lower-level subtsks hve been solved thus the bstrction in

2 these high-level subtsks needs only be consistent with one policy per lower-level subtsk (s well s ll possible policies for the current subtsk). Consequently we obtin significntly more model reduction nd hence more sclble lgorithm. Other contributions of our lgorithm include the utomtiztion of the stte bstrction (in contrst with the hndcrfted clustering required in MAXQ nd ALisp) n extension to represent completion costs (s in MAXQ nd ALisp) nd finlly stright-forwrd extension to POMDPs. To see the dvntge of policy-contingent bstrction suppose tht robot wnts to nvigte round the CMU CS building strting t the first-floor entrnce nd rriving t the fourth-floor Robot Lb. Assume it hs subtsk GoToFloor(n) in which it cn use either elevtor1 (with hevy student trffic nd ending in the 4th floor lobby) or elevtor2 (with low student trffic nd ending in corridor ner the Lb). Without knowledge of GoToFloor(4) s policy the next subtsk must pln to rech the lb from two loctions. However hving first solved GoToFloor(4) which lerns to use elevtor2 then the higherlevel pln cn ignore the 4th floor lobby nd thus bstrct wy more informtion. 2 Review of MDPs A Mrkov Decision Process (MDP) is probbilistic frmework to perform optiml ction selection in stochstic domins. We ssume the stndrd formultion for MDPs [8] where n MDP is defined to be 4-tuple. A discrete set of sttes represents the domin where n gent must select between ctions in set. The gent ims to mximize its expected (discounted) sum of rewrds: "! $# &% (' *) # % where is the discount fctor nd +' )-. is the rewrd received t time / for executing in stte. Finlly the probbilistic distribution ' 10" 2 ) defines the stte-to-stte trnsition probbility conditioned on ction. The primry chllenge of MDPs is to optimize policy 3 mpping stte to ction selection such tht the expected sum of rewrds is mximized. For smll- to medium-size domins the optiml policy cn be found nlyticlly by solving the Bellmn eqution: 4 ' 5) 76 98;:< ' *)>=@? ACBEDF 3 ' ) %I 6 8 : < ' *)>=? A B DF ' 2 0 ) 4 ' 0 )HG ' 2 0 ) 4 ' 0 )HG For lrge domins however the Bellmn eqution is imprcticl nd pproximtion techniques hve been developed to reduce computtion. One importnt re of pproximtion is in using structurl ssumptions to decompose lrge problems. 2.1 Hierrchicl MDP pproches Hierrchicl MDP problem solving ddresses complex plnning problem by leverging domin knowledge to set intermedite gols. The intermedite gols define seprte subtsks nd constrin the solution serch spce thereby ccelerting solving. Existing hierrchicl MDP pproches include MAXQ [5] HAM [11] ALisp [1] nd OPTIONS [10]. Most pproches ssume tht the domin knowledge necessry to define subtsks is provided by the designer. Subtsks (we use nottion to represent subtsk) re formlly defined using combintion of elements including: llowed strt sttes expected gol sttes fixed/prtil policies reduced ction sets nd locl rewrd functions. In ddition to ccelerting problem solving dvntges of hierrchicl MDPs include the bility to: hierrchiclly relte subgols shre subtsks between multiple prents define K One exception is work on utomticlly finding intermedite gols for OPTIONS [9]. (1) (2)

3 ? subtsk-specific restricted ction spces nd stte bstrction nd finlly use either online or btch lgorithms to optimize globl policy nd lern the full vlue function. In terms of solution qulity the gurntees offered vry between hierrchicl pproches. To better discuss optimlity criterion we introduce two useful definitions (dpted from [5]): Definition 1 - Hierrchicl Optimlity: Given the clss of ll policies consistent with hierrchy then policy 3 is sid to be hierrchiclly optiml if it chieves the most rewrd mongst ll policies 3. Definition 2 - Recursive Optimlity: Given subtsk with ction set nd the clss of ll policies vilble in ssuming fixed policies for subtsks below in the hierrchy then policy 3 is sid to be recursively optiml if it chieve the most rewrd mongst ll policies 3. MAXQ chieves recursively optiml solutions wheres HAM OPTIONS nd ALisp chieve hierrchiclly optiml solutions. The min difference between the two cn be ttributed to context. A recursively optiml solution is obtined when subtsk policies re optimized without regrds to the context in which they re clled. In contrst hierrchicl optimlity is chieved by keeping trck of ll possible contexts for clling subtsks which is key when subtsks hve multiple gol sttes. There is trde-off between solution qulity nd representtion: hierrchicl optimlity is lwys t lest s good s recursive optimlity however recursively-optiml lgorithm cn llow further stte bstrction. 2.2 Automtic stte bstrction in MDPs The overll gol of stte bstrction is to infer function ' ) mpping sttes to clusters of sttes such tht we cn then perform plnning over clusters. In generl there re mny possible wys of clustering sttes. Therefore the rel chllenge lies in grouping sttes in such wy tht we cn ) pln over clusters with miniml loss of performnce compred to plnning over the entire stte spce nd b) significntly reduce plnning time. Den et l. [3] proposed simple three-step model minimiztion lgorithm for stndrd (nonhierrchicl) MDP stte bstrction. To infer ' ) function mpping sttes to the (expnding) set of clusters : Step I - Initilize stte clustering: Let ' ) ' ) if ' *) ' )( $- (3) Step II - Check stbility of ech cluster: A cluster is deemed stble iff ' ; 0 )? ' ; 0 )+ ' 5)! " (4) ACBED ACB D Step III - If cluster is unstble then split it: Let!#* $% (5) such tht Step 2 is stisfied (with corresponding re-ssignment of ' )+ "$ & ). In step I set of overly-generl clusters re proposed. Steps II nd III re then pplied itertively grdully splitting clusters ccording to slient differences in model prmeters until intr-cluster differences re sufficiently smll to ignore. This lgorithm exhibits the following desirble properties (which we stte without proof see [3 4] for detils): 1. Plnning over clusters converges to the optiml solution. 2. The lgorithm cn be relxed to llow pproximte (' -stble) stte bstrction. 3. Assuming fctorized MDP ll steps cn be implemented such tht we cn void fully enumerting the stte spce. This lst point is of prticulr interest: voiding full stte spce enumertion is prmount for lrge problems both during stte bstrction nd during plnning/solving.

4 + - Both Dietterich [5] nd Andre&Russell [1] hve shown tht there re tremendous opportunities for stte bstrction in hierrchicl MDP plnning where the prevlence of bstrction is direct result of imposing the hierrchy. In the next section we describe n lgorithm which uses utomtic stte bstrction in the context of hierrchies. 3 Hierrchicl MDPs with policy-contingent stte bstrction We propose new hierrchicl MDP pproch which interleves plnning nd stte bstrction. The motivtion behind interleved plnning nd bstrction is to dely stte bstrction for higher-level subtsks until the lower-level subtsks hve been solved. This is in contrst to the typicl pproch of first defining stte bstrction for ll subtsks nd then solving them. The min insight is tht if we dely finding the stte bstrction for subtsk until lower-level policy 3 is fixed then stte bstrction in cn be contingent only on 3 hence policy-contingent stte bstrction s opposed to llowing sufficient stte detil to ccommodte ny policy 3 (gnostic stte bstrction). Consequently we cn obtin fr more stte bstrction in higher-level subtsks. 3.1 Structurl ssumptions Our lgorithm strts with set of structurl ssumptions which re similr to erlier pproches nd more specificlly re identicl to MAXQ. Formlly we re given tsk grph (see figure 1 for n exmple) where ech lef node represents primitive ction from the originl MDP ction set. Ech internl node hs the dul role of representing both distinct subtsk (whose ction set is defined by its immedite children in the hierrchy) s well s n bstrct ction (we use br s in to denote bstrct ctions) in the context of the bove-level subtsk. A subtsk is formlly defined by: # the set of ctions which re llowed in subtsk. Bsed on the hierrchy there is one ction for ech immedite child of. the set of terminl sttes for subtsk where. Subtsk is terminted upon reching stte t which point control is returned to the higher-level subtsk which initited. ' ) the pseudo-rewrd function specifying the reltive desirbility of ech terminl stte. By definition ' 5) "$. In ddition we require prmeterized model of the MDP domin:. This is deprture from mny reinforcement lerning hierrchicl pproches however it is necessry to perform utomtic stte bstrction (see equtions 3-4.) 3.2 Algorithm Description Our lgorithm is the following: Given n MDP nd tsk hierrchy Step 1: Re-formulte structured stte spce: following bottom-up ordering: For ech subtsk Step 2: Set prmeters: ' *) nd ' *)+ "$ )+ Step 3: Minimize subtsk: Step 4: Solve subtsk: A ' 3 We use nottion for terminl sttes (both gol nd non-gol) insted of the more common in n ttempt to void confusion with the nottion for trnsition probbilities! #"$%"&('*).!.) In prctice terminl sttes. is often set to zero for gol sttes nd to lrge negtive vlue for non-gol

5 # MDP: Hierrchy: h s0 s 1 s 3 2 s h STRUCTURED STATE SPACE 3 4 h 1 s 0 h 1 s 1 h 1 s 2 h 1 s 3 h 0 s 0 h 0 s 1 h 0 s 2 h 0 s h 1 s 3 0 h 1 s 1 h 1 s 3 2 h 1 s h 1 c 3 0 h 1 c h 1 c 3 0 h 1 c 1 SET PARAMETERS for h 1 CLUSTER STATES for h 1 SOLVE POLICY for h 1 where c 0 ={s 0 s 2 c 1 ={s 1 s h 2 0 s 1 0 h 0 s 1 h 0 s 1 2 h 0 s h 2 0 c 1 0 h 0 c 1 h 0 c 2 h 0 c 1 0 h 0 c 1 SET PARAMETERS for h 0 CLUSTER STATES for h 0 SOLVE POLICY for h h 0 c 2 where c 0 ={s 0 c 1 ={s 1 c 2 ={s 2 s 3 Figure 1: Simple 4-stte problem with 2-subtsk hierrchy. Non-zero trnsition probbilities re illustrted in the MDP. The subtsk hierrchy is illustrted in the top right corner. Note tht the finl policy of K $ (bottom left) is used to model K (right column). Figure 1 illustrtes the entire process for simple 4-stte 2-subtsk problem. In step 1 Re-formulte structured stte spce we use n ide first introduced in the HAM frmework [11] tht re-formultes the MDP to reflect structurl ssumptions. The new stte spce is spnned by the cross of the originl stte spce nd hierrchy stte. Step 2 Set prmeters ppropritely trnsltes conventionl trnsition nd rewrd prmeters to the structured problem representtion. This includes copying originl trnsition nd rewrd prmeters from for ll relevnt primitive ctions s well s inferring prmeters for the5 newly-introduced bstrct ctions. Given subtsk with stte set # nd ction set where # invoke corresponding lower-level subtsks equtions 6-9 trnslte the prmeters from the originl MDP to the structured stte spce. Cse 1: Primitive ctions (see full-line trnsition rrows in figure 1) ' * ) ' * )+ * * (6) ' * ) ' )( If $ (7) ' H)+ If $ Cse 2: Abstrct ctions (see dotted-line trnsition rrows in figure 1) ' ) ' 3 ' )( )+ ' ) ' 3 ' ))( ' )( If (8) (9) One should note tht equtions 8-9 depend on 3 - the finl policy of subtsk. This enforces the policy-contingent spect of our lgorithm. Becuse prmeter setting is delyed until ll lower-level subtsks hve been solved then stte bstrction in must be sufficient to represent only 3 s opposed to ny policy 3. For the cse -! "!!)) -!!) we use true rewrd not pseudo-rewrd.

6 = Given tht ll sttes in cluster must shre the sme vlue steps 3 nd 4 cn be folded into one. Thus the model minimiztion lgorithm with vlue updtes is: Step I-III - Sme s equtions 3-5 Step IV - Updte vlue of ech cluster: 4 '!5) : < ' *)>= #? B D '!1 ; $ 0 ) 4 ' 0 )CG& "! For non-hierrchicl MDPs this is equivlent to Boutilier et l. s decision-tree representtion of MDP vlue functions [2]. ) The vlue function of the finl 4 policy solution cn be retrieved by looking t the vlue function of the top subtsk: '. Becuse it fixes low-level subtsk policies prior to solving higher-level subtsks the lgorithm is limited to recursive optimlity (rther thn hierrchicl). However for mny problems the solution qulity trde-off is smll compred to the mount of policy-contingent bstrction chieved. To execute the finl policy we dopt top-down polling pproch. At every time step we strt by querying the policy of the top subtsk nd if it returns n bstrct ction we query the policy of tht subtsk nd so on down the tree until primitive ction is returned. 3.3 Extensions to policy-contingent bstrction nd plnning A key ttribute of our lgorithm is the fct tht we model bstrct ctions ccording to their one-step effect (see equtions 8-9) s opposed to their cumultive effect: ' )? ' )+ if which most hierrchicl MDP pproches fvor (e.g. MAXQ ALisp Options). The implictions of using one-step effects re two-fold. On the positive side using one-step effects removes the need to detect subtsk terminl sttes nd therefore our lgorithm cn esily be extended to Prtilly Observble MDPs. POMDPs re generliztion of MDPs which llow for prtil stte observbility [7]. Plnning in POMDPs is typiclly exponentil in the size of the domin mking exct solutions intrctble for ll but trivil problems. To extend the policy-contingent hierrchicl lgorithm described bove to POMDPs we mke two modifictions. First we modify the model minimiztion stbility criteri to lso check for similr observtion probbilities ' 2 50E) ; nd second we set policy-contingent observtion prmeters (similrly to equtions 6&8 but for ' 5) nd ' $ 5) ). The negtive consequence of using one-step effects is tht we do not explicitly represent subtsk completion costs (s defined in MAXQ nd ALisp) which gretly reduces the opportunity for funnel bstrction. In the prgrph below we discuss n extension of our lgorithm which uses the decomposed vlue function nd chieves full funnel bstrction. *) Extension to decomposed vlue function. Given hierrchicl representtion the MDP vlue function (eqn. 1) cn be decomposed into three prts: ' ' ' )9= ' ) where is the expected rewrd of tking ction is the expected rewrd of completing subtsk fter hving tken nd is the expected rewrd of finishing the entire tsk fter hving completed. A 2-prt decomposition ws first introduced in MAXQ. The 3-prt decomposition ws lter defined in ALisp [1]. Both included hnd-crfted bstrction functions ' ) nd ' ) s well s ' ) for ALisp. (10) (11) *) = Funnel bstrction is chieved when subtsk hs few terminl sttes (for exmple moving robot to doorwy) by decoupling costs before nd fter the terminl sttes.! #"$ ) To obtin n even more concise representtion is stored only for primitive ctions nd in the cse of bstrct ctions is recursively clculted online when necessry (see [5] for detils).

7 ? ) We cn modify the policy-contingent lgorithm to utomticlly find the decomposed bstrction functions. Since policy-contingency restricts us to recursive optimlity we ignore externl costs ' ) nd compute two seprte functions: ' ) for primitive ctions (bsed strictly on immedite rewrd see eqn. 3) nd ' ) for both primitive nd bstrct ctions. In the cse of ' ) clustering requires identicl completion costs: cn be non-trivil. The min difficulty is in estimt- (see eqn. 11) nd '. 0 MAXQ discusses this in detil ) nd proposes recursive lgorithm to clculte. It is worth pointing out tht policy-contingency offers n dvntge on this issue nmely tht these must only be evluted once unlike in MAXQ which requires re-estimting for ech vlue updte. ' 0 ) ' ACB"D In prctice utomticlly finding ' ) ing ' 50) ' 0 )>= 50 ) ' ' 0 )) Extension to Q-bstrction: During model minimiztion it is lso possible to compute 4 (bstrcting over (which bstrcts over ' 5) ). This is often 4 used when hnd-crfting bstrction functions. The dvntge of bstrcting insted of is tht we cn llow different stte bstrctions under different ctions potentilly resulting in n exponentil reduction in the number of clusters. To bstrct we fix ny policy which visits every stte-ction pir nd mke new MDP whose sttes re the stte-ction pirs of nd whose trnsitions re given by our fixed policy. We then run the utomted model minimiztion lgorithm on this new MDP. n bstrction of ' ) 4 Results - Txi domin ' ) ) insted of just ' ) The txi domin is well-known hierrchicl MDP introduced by Dietterich [5]. The overll tsk (see Figure 2) is to control txi gent with the gol of picking up pssenger from n initil loction nd then dropping him/her off t desired destintion. The initil stte is selected rndomly however it is fully observble nd trnsitions re deterministic. Figure 2b represents the MAXQ control hierrchy used for this domin. The structured stte spce cn % be described by fetures: XYpssengerdestintionH where : : : :. Tble 1 compres stte clustering results for this tsk (Txi1) s well s second lrger domin (Txi2). In Txi2 the pssenger cn strt from ny loction on the grid compred to only YBRG in Txi1. This tsk is hrder for MAXQ nd ALisp becuse of the need to represent mny more completion costs in ' /. In both tsks the policy-contingent bstrction pproch (PolCA) is ble to utomticlly discover more bstrction thn either MAXQ or ALisp. #sttes #ctions QL HSMQ MAXQ ALisp PolCA-Q Txi Txi (12) Tble 1: Number of prmeters required to lern solution. QL=Q-lerning HSMQ=hierrchicl semi-mrkov Q-lerning[6] PolCA-Q=policy-contingent bstrction (clustering on Q-vlues but without the decomposed vlue extension). The bstrction results for MAXQ nd ALisp in Txi1 re published results; Txi2 results re hnd-crfted following creful reding of ech lgorithm. All lgorithms chieve optiml performnce for Txi1. PolCA-Q lerns the optiml policy for Txi2; run-time performnce of MAXQ nd ALisp is un-verified for this tsk. 5 Conclusion This pper introduces novel hierrchicl MDP lgorithm which interleves plnning nd bstrction to solve complex MDP problems. In ddition to utomting the stte bstrc-

8 Root Get Put Pickup Nv(t) Putdown North South Est West ' ) ' ) Figure 2: The txi domin is represented using four fetures: XYPssengerDestintion. The XY represent 5x5 grid world; the pssenger cn be t ny of: YBRGtxi ; the destintion is one of: YBRG. The txi gent cn select from six ctions: NSEWPickupPutdown. Actions hve uniform -1 rewrd. Rewrd for the Pickup ction is -1 when the gent is t the pssenger loction nd -10 otherwise. Rewrd for the Putdown ction is +20 when the gent is t the destintion with the pssenger in txi nd R=-10 otherwise. tion for hierrchies this lgorithm fetures previously un-exploited policy-contingent stte bstrction s well s stright-forwrd extensions to either POMDP or completion-cost representtions. We hve not yet implemented the decomposed vlue representtion for policy-contingent bstrction; this will be the subject of future work. Finlly we predict tht better understnding of the interction between ction hierrchies nd stte bstrction my led to new methods for utomticlly discovering ction hierrchies. Acknowledgements: We thnk T. Dietterich nd D. Andre for helpful exchnges concerning MAXQ nd ALisp s well s S. Thrun N.Roy nd C.Bererton for helpful comments. References [1] D. Andre nd S. Russell. Stte bstrction for progrmmble reinforcement lerning gents. In Proceedings of AAAI [2] C. Boutilier R. Derden nd M. Goldszmidt. Stochstic dynmic progrmming with fctored representtions. AI ournl [3] T. Den nd R. Givn. Model minimiztion in Mrkov decision processes. In Proceedings of the 1997 Interntionl oint Conference on Artificil Intelligence [4] T. Den R. Givn nd S. Lech. Model reduction techniques for computing pproximtely optiml solutions for Mrkov decision processes. In Proceedings of UAI [5] T. G. Dietterich. Hierrchicl reinforcement lerning with the MAXQ vlue function decomposition. ournl of Artificil Intelligence Reserch 13: [6] T. G. Dietterich. An overview of MAXQ hierrchicl reinforcement lerning. In Proceedings of the Symposium on Abstrction Reformultion nd Approximtion (SARA [7] L. P. Kelbling M. L. Littmn nd A. R. Cssndr. Plnning nd cting in prtilly observble stochstic domins. Artificil Intelligence 101: [8] L. P. Kelbling M. L. Littmn nd A. W. Moore. Reinforcement lerning: A survey. ournl of Artificil Intelligence Reserch 4: [9] A. McGovern nd A. G. Brto. Automtic discovery of subgols in reinforcement lerning using diverse density. In Proceedings of ICML [10] A. McGovern D. Precup B. Rvindrn S. Singh nd R. S. Sutton. Hierrchicl optiml control of MDPs. In Proc. of the Yle Workshop on Adptive nd Lerning Systems [11] R. Prr nd S. Russell. Reinforcement lerning with hierrchies of mchines. In Advnces in Neurl Informtion Processing Systems volume [12] R. S. Sutton nd A. G. Brto. Reinforcement Lerning: An Introduction. MIT Press 1998.