PPS: User Manual. Krishnendu Chatterjee, Martin Chmelik, Raghav Gupta, and Ayush Kanodia

Transcription

1 PPS: User Mnul Krishnendu Chtterjee, Mrtin Chmelik, Rghv Gupt, nd Ayush Knodi IST Austri (Institute of Science nd Technology Austri), Klosterneuurg, Austri In this section we descrie the tool fetures, user mnul including the input file formt, nd detiled description of the exmples used. 0.1 Fetures The tool lods the input file with the POMDP nd the prity ojective. Depending on the rguments, the PPS tool cn perform the following opertions (see for further detils). Visulize the textul input POMDP using the DOT lnguge. Reduce POMDP with prity ojective, to n equivlent POMDP with cobüchi ojective ( prity ojective with only 2 priorities), nd visulize the POMDP. Given POMDP, with cobüchi ojective, it constructs its elief-oservtion POMDP Ĝ, nd visulizes it. Given the elief-oservtion POMDP Ĝ, if there exists n lmost-sure winning strtegy, it outputs finite-stte mchine, where the sttes encode the memory elements, nd trnsitions leled y ctions nd oservtions stnd for memory updtes. As efore the tool cn visulize the finite-memory strtegy. Note, tht it is possile to visulize the POMDP, using the DOT lnguge, fter performing ny of the reductions, to get the intermedite results. 0.2 User Mnul Dependencies. The following softwre will e required to run the PPS tool: Jv Runtime Environment or Jv Development Kit (version 1.7 onwrds). The PPS tool hs een developed using Jv 1.7. Downlod from downlods/jre7-downlods html JGrphT. This is the grph hndling lirry tht hs een used to work with the POMDPs, treted s directed multigrphs. The JGrphT.jr lirry is required to run the softwre. Downlod from GrphViz. This softwre is required to convert the files tht re output y the PPS tool in the dot formt for grphs, into visul grphs. Downlod from http: // The commnd for converting dot file into its corresponding grph (in the png formt) dot -Tpng source > destintion. Detiled documenttion for GrphViz cn e found t http: //

2 Running the PPS tool. To run the softwre, downlod the jr file pps.jr from Instll ll dependencies. Now issue the commnd "jv -jr pps.jr" from the commnd line terminl. The user interfce will now open. A detiled exmple of how to use the tool is presented lter. File Formt. We now present the input file formt. nme-of-input-pomdp [<Nme>]# oservtions [<os 1>,<os 2>,..., <os j>] # sttes [<stte 1>, <stte 2>,..., <stte n>]# stte { <stte nme>; <priority vlue>; <stte oservtion>; T (if this is strt stte) or F (if this is not strt stte)} # ctions [<ction 1>,<ction 2>,..., <ction k>] # trnsitions [<source stte>,<ction>, { <trget stte 1>; <trget stte 2>;...; <trget stte m> }]# The input file is treted sequence of tokens, seprted y the # chrcter. As the exct proility of trnsition is not importnt in qulittive nlysis, the file formt is designed in wy tht it stores only the support of the trnsition function. Smple POMDP Input File. In this prt we present n exmple of POMDP in the input file formt nd provide detiled description. Figure 1 shows the corresponding POMDP generted using GrphViz. NAME: [Demo POMDP] # OBSERVATIONS: [o1,o2,o3] # STATES: [{x; 1; o1; F}, {y; 2; o2; F}, {z; 1; o2; F}, {w; 2; o3; T}] # ACTIONS: [,] # TRANSITIONS: [x,,{w}] # TRANSITIONS: [x,,{z;w}] # TRANSITIONS: [y,,{z}] # TRANSITIONS: [y,,{x}] # TRANSITIONS: [z,,{x;y}] # TRANSITIONS: [z,,{x;y;z}] # TRANSITIONS: [w,,{w}] # TRANSITIONS: [w,,{w;z}] # We now explin the ove POMDP: The nme of the input POMDP is Demo POMDP The oservtions re o1, o2, o3 The sttes re x, priority 1, oservtion o1. y, priority 2, oservtion o2. z, priority 1, oservtion o2. w, priority 2, oservtion o3, which is strt stte.

3 w, Strt: true, O: o3,pr: 2 z, Strt: flse, O: o2,pr: 1 y, Strt: flse, O: o2,pr: 2 x, Strt: flse, O: o1,pr: 1 Fig. 1. The Demo POMDP generted using GrphViz The ctions re nd. The following re the set of trnsitions on given ctions, for every stte of the POMDP In stte x, on ction, the next stte is w with proility 1. In stte x, on ction, the next stte is with positive pro. z, or w. In stte y, on ction, the next stte is z with proility 1. In stte y, on ction, the next stte is x with proility 1. In stte z, on ction, the next stte is with positive pro. x, or y. In stte z, on ction, the next stte is with positive pro. x, or w, z. In stte w, on ction, the next stte is w with proility 1. In stte w, on ction, the next stte is with positive pro. w, or z. Smple Tool Usge. In this prt we show smple tool wlkthrough. We will use AlterntePOMDP.txt exmple to illustrte the wlkthrough. This input file is lso ville on the wesite long with the PPS tool on Figure 2 shows the corresponding POMDP generted using the GrphViz tool. NAME: [POMDP Alternte] # OBSERVATIONS: [o1] # STATES: [ {x; 2; o1; F}, {y; 2; o1; F}, {z; 1; o1; T}] # ACTIONS: [,] # TRANSITIONS: [x,,{y}] # TRANSITIONS: [x,,{z}] # TRANSITIONS: [y,,{z}] # TRANSITIONS: [y,,{x}] #

4 TRANSITIONS: [z,,{x;y;z}] # TRANSITIONS: [z,,{x;y;z}] # There re three sttes in the POMDP, x, y nd the initil stte z. All the sttes hve the sme oservtion o1, therefore the Plyer does not receive ny useful informtion from the ply. There re two ctions, nd. In stte x, tking ction (resp. ) rings the POMDP deterministiclly to stte y (resp. z), nd in stte y, tking ction (resp. ) rings the POMDP deterministiclly to stte x (resp. z). Plying ny ction in stte z rings the POMPD to ll three sttes with positive proility. Sttes x nd y hve priority 2 nd stte z hs priority 1. Therefore, the gol in the POMDP is to ply strtegy tht visits stte z only finitely often with proility 1. Note, tht once the POMDP reches stte x, if the plyer plys the strtegy tht lterntes etween plying followed y, then the POMDP will lwys sty in sttes x nd y, oth of which hve priority priority 2. Similrly, once the POMDP reches stte y, if the strtegy lterntes etween plying followed y, then the POMDP will lwys sty in sttes y nd x. x, Strt: flse, O: o1,pr: 2 z, Strt: true, O: o1,pr: 1 y, Strt: flse, O: o1,pr: 2 Fig. 2. POMDP Alternte generted using GrphViz In the following prt we enumerte the required steps in order to solve whether there exists finite-memory strtegy ensuring the ojective with proility Run the commnd jv -jr pps.jr. A dilog ox opens, s shown in Figure Select the POMDP input file AlterntePOMDP.txt, y clicking the utton. This will open dilog ox s in Figure In the cse of AlterntePOMDP.txt the ojective is specified only y priorities 1 nd 2, therefore we cn select the rdio utton corresponding to cobüchi ojectives to speed up the computtion. In cse the ojective hs lso other priorities, select the rdio utton corresponding Prity ojective. 4. To generte the dot formt for the input POMDP, check the first option. Click the corresponding Sve As utton to specify the loction nd the nme of the file to

5 Fig. 3. Opening Screen Fig. 4. Select Input POMDP e sved to. Figure 5 illustrtes the sme. One cn then use GrphViz to generte n imge of the file from the dot formt. Figure 2 ws generted in this wy. 5. In cse the input POMDP is specified with prity ojective, select the Output cobüchi POMDP checkox to generte the dot formt of n equivlent POMDP with cobüchi ojective. Click the corresponding Sve As utton to specify the loction nd the nme of the file to e sved to. 6. To generte the dot formt for the elief-oservtion POMDP which is generted s prt of the lgorithm, select the check ox next to Output Belief Os. POMDP. Click the corresponding Sve As utton to specify the loction nd nme of the file to e sved to. This is illustrted in Figure 6 Fig. 5. Generte Input POMDP Fig. 6. Generte elief-oservtion POMDP 7. To generte the dot formt file for the finl strtegy which is generted, select the check ox next to Generte Strtegy. Click the corresponding Sve As utton to specify the loction nd nme of the file to e sved to. This is illustrted in Figure Now click the Run utton. After witing for progrm execution, you will see dilog ox. An exmple of the sme is illustrted in Figure 8. This provides informtion out whether there exists finite-memory lmost-sure winning strtegy, nd the execution time for the progrm. The dot formt files which were selected

6 for genertion will e found t the specified loctions. This completes progrm execution. Fig. 8. Result Screen Fig. 7. Generte Strtegy Grph Now, GrphViz (See Dependencies ) must e used in order to visulize the grphs, which hve een output in the dot formt. An Explntion for the Strtegy. After the execution the tool outputs tht there exists finite-memory lmost-sure winning strtegy nd outputs the strtegy grph. Here, we provide n intuitive explntion for the strtegy which is given y the PPS tool, for the input POMDP shown in Figure 2. The output strtegy is shown in Figure 9. The finite-memory strtegy σ = (σ u, σ n, M, m 0 ) corresponding to the Figure is defined s follows: The memory set M contins four memory elements {m1, m2, m3, m4}. The initil memory is the memory element tht is successor of the strt ction from the node leled s StrtStte, i.e., the initil memory element is m1. The ction selection function σ n for memory m selects uniformly ll ctions outgoing from node m in the grph, i.e., σ n (m1) selects ction with proility 0.5 nd ction with proility 0.5, σ n (m3) selects ction with proility 1, etc. The memory updte function σ u given memory element m, ction nd oservtion o chooses the new memory uniformly from the successor sttes of stte m y edges leled y ction nd trget sttes leled with o, i.e., σ u (m1,, o1) updtes with uniform proility to m2, m3, nd m4. In cse of the memory element m3 we hve σ u (m3,, o1) updtes with proility 1 to m4. One cn see tht y plying the strtegy σ the Plyer reches the memory elements m3 nd m4 with proility 1. From these memory elements the strtegy prescries to lternte ctions nd. As ws mentioned efore, this strtegy ensures tht the initil stte z is visited only finitely often lmost-surely. 0.3 Exmple - Spce Shuttle The spce shuttle exmple originlly comes from [1], nd long with the originl POMDP we lso consider slight vrints of the model presented in [2]. We will de-

7 StrtStte strt m1 Os: o1 m2 Os: o1 m3 Os: o1 m4 Os: o1 Fig. 9. Finite-memory lmost-sure winning strtegy scrie the esiest vrint of the prolem, the remining vrints will e descried in the end of the exmple. It models simple spce shuttle docking prolem, where the shuttle must dock y cking up into one of the two spce sttions. The gol is to visit oth sttions infinitely often. Figure 10 originlly comes from [2] nd shows schemtic representtion of the model. The left most nd right most sttes in Figure 10 re the docking sttions, the most recently visited docking sttion is leled with MRV, nd the lest recently visited docking sttion is leled with LRV. The property of the model is tht whenever LRV sttion is visited it utomticlly chnges its stte to MRV. Both of the ctions go forwrd nd turn round re deterministic. Sttes: There re 11 sttes in the POMDP, corresponding to the position of the shuttle: 0 - docked in LRV; 1 - just outside spce sttion MRV, front of ship fcing sttion; 2 - spce, fcing MRV; 3 - just outside spce sttion LRV, ck of ship fcing sttion; 4 - just outside spce sttion MRV, ck of ship fcing sttion; 5 - spce, fcing LRV; 6 - just outside spce sttion LRV, front of ship fcing sttion; 7 - docked in MRV, the initil stte; 8 - successful delivery; 9 - ump into LRV; 10 - ump into MRV. Oservtions: There re 7 oservtions corresponding to wht cn e seen from the shuttle: o0 see LRV forwrd, o1, see MRV forwrd, o2 docked in MRV, o3 see nothing, o4 docked in LRV, o5 umping into docking sttion, nd finlly o6 is oserved upon successful delivery. Actions: There re three ctions tht cn e chosen: go forwrd (f), turn round (), nd ckup (). Trnsition reltion: If the shuttle is fcing sttion (sttes 1 nd 6) the ckup ction succeeds only with proility 0.3, hs no effect with proility 0.4, nd with proility 0.3 cts like turn round ction. Whenever in spce (sttes 2 nd 5) the ckup ctions succeeds with proility 0.8, hs no effect with proility 0.1, nd with the

8 remining proility 0.1 hs the sme effect s n comintion of turning round nd ckup ction. Finlly, when the shuttle is djcent to sttion nd fcing wy (sttes 3 nd 4), it hs proility of 0.7 of ctully docking to sttion, nd with the remining proility 0.3 hs no effect. In the remining sttes 8, 9, nd 10 the ction effect is deterministic. Fig. 10. Spce shuttle docking prolem Ojective. The prity ojective of the model is defined y the priority ssignment function s follows: ll sttes from 1 to 7 hve priority 3; the stte 8 which represents successful delivery into lest recently visited sttion, hs priority 2. Whenever the shuttle is fcing sttion, it should try to ckup into the sttion s trying to move forwrd results into ump into sttion represented with sttes 9 nd 10 with priority 1. Therefore, the ojective cn e intuitively explined s trying to visit oth of the sttions infinitely often, while trying to ump only finitely often with proility 1. The input file for the POMDP cn e downloded here The more complex vrints of this model differ in the numer of sttes tht re required to trvel through the spce, nd therefore hve higher uncertinty out the position of the shuttle. 0.4 Exmple - Cheese Mze The mze is shown in Figure 11 is introduced in [2]. We will descrie only the smllest vrint in detil. The gol of the plyer is to rech gol stte while trying to void poison in d sttes. The plyer is only prtilly informed, its oservtion corresponds to wht would e seen in ll four directions immeditely djcent to the loction. After the gol stte is reched the plyer is respwned with positive proility in multiple sttes of the mze nd the gme is restrted.

9 Sttes: There re 11 sttes in the POMDP tht re illustrted on Figure 11. The gme strts in stte 6. The poison is plced in sttes 8 nd 9, nd 10 is the gol stte. Oservtions: There re 7 oservtions corresponding to wht would e seen in ll four directions immeditely djcent to the loction, i.e., sttes 5, 6, nd 7 do hve the sme oservtion. The oservtions re s follows: o0, the wlls re NW; o1, the wlls re NS; o2, the wll is N; o3, the wlls re NE; o4, the wlls re WE; o5, poisoned stte; nd o6, the gol stte. Actions: There re four ctions ville corresponding to the movement in the four compss directions (north n, est e, south s, west w). Trnsition reltion: Actions tht ttempt to move outside of the mze hve no effect on the position. The rest of the moves is deterministic in ll 4 ctions. Fig. 11. Cheese mze prolem Ojective: The ojective is to visit the gol stte 10 infinitely often, while getting poisoned in sttes 8 nd 9 only finitely often. This is encoded s prity ojective with 3 priorities. Stte 10 hs priority 2, sttes 8 nd 9 hve priority 1, nd every other stte hs priority 3. Whenever the gol stte is reched, the mze is restrted with proility 1/3 to stte 0, with proility 1/3 to stte 2, nd with proility 1/3 to stte 4 in the esiest vrint of the prolem. The other vrints re medium nd difficult, nd differ in the numer of sttes the mze cn restrt to, intuitively incresing the uncertinty in the POMDP nd lso result into longer running times. We lso consider more difficult setting of the prolem y constructing n intermedite nd lrge size mzes with more sttes ut sed on the sme principle. The input file for the POMDP cn e downloded here Exmple - Grid We will descrie only the grid 4 4, the other lrger vrints of the grid only differ in size. The prolem consists of 4 y 4 grid of loctions. There is single gol stte, nd

10 multiple trp sttes tht re plced eforehnd ut not known to the plyer. Whenever gol stte is reched the gme is restrted. The gol of the plyer is to lern the position of the trps nd visit the gol infinitely often, while visiting the trp stte only finitely often with proility 1. Sttes: There re 33 sttes in the POMDP. The uncertinty out the plcement of the trps is modeled y two grids of sttes 4 4 nd n dditionl initil stte strt tht hs trnsition to oth of the grids. The coding of the sttes is s follows, the stte ijk corresponds to stte in the i-th copy, j-th row, nd k-th column. The gol stte in ech grid is the lower right corner, i.e., stte 033 nd 133 (rows nd columns re numered 0-3). x x x x x x x x Fig. 12. Trp plcement Oservtions: There re 4 oservtions, the initil stte hs oservtion o0, the trp sttes hve oservtion o2, the gol stte hs oservtion 03, nd ll the remining sttes hve oservtion o1. Actions: As in the previous exmple, there re 4 ctions ville corresponding to the movement in the four compss directions (north n, est e, south s, west w). Trnsition reltion: The initil stte of the POMDP is the stte strt, no mtter wht ction is plyed the next stte is with proility 0.5 the upper left corner in one of the grids, nd with the remining proility the upper left corner of the second grid. In the grid ll the ctions re deterministic nd ttempts to move outside of the grid hve no effect on the position. Whenever gol stte is reched the gme is restrted the the upper left corner of the sme grid (the trp sttes sty in the sme position). Ojective: The ojective is to lern in which grids the plyer is nd s in the previous exmples try to rech the gol stte infinitely often while visiting the trp sttes only finitely often with proility 1. This is encoded s prity ojective with 3 priorities, the gol stte hs priority 2, the trp sttes hve priority 1, nd ll the remining sttes hve priority 3. The plcement of the trp sttes we hve considered for the 4 4 grid is depicted on Figure 12. The other vrints differ in the size of the grid nd plcements of the trp sttes. The input file for the POMDP cn e downloded here

11 References 1. Lonnie Chrismn. Reinforcement lerning with perceptul lising: The perceptul distinctions pproch. In AAAI, pges Citeseer, M. L. Littmn, A. R. Cssndr, nd L. P. Kelling. Lerning policies for prtilly oservle environments: Scling up. In ICML, volume 95, pges Citeseer, 1995.