POMDP: Introduction to Partially Observable Markov Decision Processes Hossein Kamalzadeh, Michael Hahsler

Transcription

1 POMDP: Intoduction to Patially Obsevable Makov Decision Pocesses Hossein Kamalzadeh, Michael Hahsle The R package pomdp povides an inteface to pomdp-solve, a solve (witten in C) fo Patially Obsevable Makov Decision Pocesses (POMDP). The package enables the use to simply define all components of a POMDP model and solve the poblem using seveal methods. The package also contains functions to analyze and visualize the POMDP solutions (e.g., the optimal policy). In this document we will give a vey bief intoduction to the concept of POMDP, descibe the featues of the R package, and illustate the usage with a toy example. Intoduction on POMDPs A patially obsevable Makov decision pocess (POMDP) is a combination of an MDP to model system dynamics with a hidden Makov model that connects unobsevant system states to obsevations. The agent can pefom actions which affect the system (i.e., may cause the system state to change) with the goal to maximize a ewad that depends on the sequence of system state and the agent s actions. Howeve, the agent cannot diectly obseve the system state, but at each discete point in time, the agent makes obsevations that depend on the state. The agent uses these obsevations to fom a belief of in what state the system cuently is. This belief is called a belief state and is expessed as a pobability distibution ove the states. The solution of the POMDP is a policy pescibing which action is optimal fo each belief state. The POMDP famewok is geneal enough to model a vaiety of eal-wold sequential decision-making poblems. Applications include obot navigation poblems, machine maintenance, and planning unde uncetainty in geneal. The geneal famewok of Makov decision pocesses with incomplete infomation was descibed by Kal Johan Åstöm in 1965 in the case of a discete state space, and it was futhe studied in the opeations eseach community whee the aconym POMDP was coined. It was late adapted fo poblems in atificial intelligence and automated planning by Leslie P. Kaelbling and Michael L. Littman (Littman 2009). A discete-time POMDP can fomally be descibed as a 7-tuple ( S, A, T, R, Ω, O, λ), whee S = {s 1, s 2,..., s n } is a set of states, A = {a 1, a 2,..., a m } is a set of actions, T is a set of conditional tansition pobabilities T (s s, a) fo the state tansition s s. R : S A R is the ewad function, Ω = {o 1, o 2,..., o k } is a set of obsevations, O is a set of conditional obsevation pobabilities O(o s, a), and λ [0, 1] is the discount facto. At each time peiod, the envionment is in some state s S. The agent chooses an action a A, which causes the envionment to tansition to state s S with pobability T (s s, a). At the same time, the agent eceives an obsevation o Ω which depends on the new state of the envionment with pobability O(o s, a). Finally, the agent eceives a ewad R(s, a). Then the pocess epeats. The goal is fo the agent to choose actions at each time step that maximizes its expected futue discounted ewad [ ] E γ t R(s t, a t ) t=0 1

2 . Package Functionality Solving a POMDP poblem with the pomdp package consists of two steps: 1. Define a POMDP poblem using the function POMDP, and 2. solve the poblem using solve_pomdp. Defining a POMDP Poblem The POMDP function has the following aguments, each coesponds to one of the elements of a POMDP. st(ags(pomdp)) function (discount = 0, states, actions, obsevations, stat = "unifom", tansition_pob, obsevation_pob, ewad, values = "ewad", name = NA) Next we descibe the aguments in detail and give examples: discount: Is the used discount facto λ, a eal numbe in the ange [0, 1]. discount = 0.9 states: Defines the set of states S using a vecto of stings. states = c("state1", "state2", "state3") actions: Defines the set of actions A using a vecto of stings. actions = c("action1", "action2") obsevations: Defines the set of obsevations Ω using a vecto of stings. obsevations = c("obs1", "obs2") stat: The initial pobability distibution ove the system states S. It can be specified in seveal ways. A vecto of n pobabilities in [0, 1], that add up to 1, whee n is the numbe of states. stat = c(0.5, 0.3, 0.2) The sting unifom fo a unifom distibution ove all states. stat = "unifom" A vecto of intege indices specifying a subset as stat states. The initial pobability is unifom ove these states. Fo example, only state 3 o state 1 and 3: stat = 3 stat = c(1, 3) A vecto of stings specifying a subset as stat states. stat <- "state3" stat <- c("state1", "state3") A vecto of stings stating with "-" specifying which states to exclude fom the unifom initial pobability distibution. stat = c("-", "state2") tansition_pob: Defines the conditional tansition pobabilities T (s s, a). The pobabilities depend on the end-state s, the stat-state s and the action a. The set of conditional tansition pobabilities can be specified in seveal ways: 2

3 A data fame with 4 columns, whee the columns specify action, stat-state, end-state and the pobability, espectively. The fist 3 columns ae eithe the names o the index of the action o states. tansition_pob = data.fame( "action" = c( "action1", "action1", "action1", "action1", "action1", "action1", "action1", "action1", "action1", "action2", "action2", "action2", "action2", "action2", "action2", "action2", "action2", "action2"), "stat-state" = c( "state1", "state1", "state1", "state2", "state2", "state2", "state3", "state3", "state3", "state1", "state1", "state1", "state2", "state2", "state2", "state3", "state3", "state3"), "end-state" = c( "state1", "state2", "state3", "state1", "state2", "state3", "state1", "state2", "state3", "state1", "state2", "state3", "state1", "state2", "state3", "state1", "state2", "state3"), "pobability" = c( , 0.5, 0, 0.7, 0.3, 0.4, 0.4, 0.2, 0, 0.6, 0.4, 0.1, 0.9, 0, 0.7, 0.3, 0) ) A named list of m matices whee each matix epesents one of the m actions. Each matix is squae of size n n whee n is the numbe of states. Matices can also be defined as "identity" o "unifom". tansition_pob = list( action1 = matix(c( 0.1, 0.4, 0.5, 0, 0.7, 0.3, 0.4, 0.4, 0.2), now = 3, byow = TRUE), action2 = matix(c( 0, 0.6, 0.4, 0.1, 0.9, 0, 0.7, 0.3, 0), now = 3, byow = TRUE)) tansition_pob = list( action1 = matix(c( 0.1, 0.4, 0.5, 0, 0.7, 0.3, 0.4, 0.4, 0.2), now = 3, byow = TRUE), action2 = unifom ) obsevation_pob: Specifies the conditional obsevation pobabilities O(o s, a). The pobabilities depend on the action, the end-state, and the obsevation. The set of conditional obsevation pobabilities can be specified in seveal ways: A data fame with 4 columns, whee the columns specify action, end-state, obsevation and the pobability, espectively. The fist 3 columns could be eithe the name o the index of the action, state, o obsevation. The special chaacte * can be used to indicate that the pobability applies fo all actions, states o obsevations. obsevation_pob = data.fame( "action" = c( "*", "*", "*", "*", "*", "*"), "end-state" = c("state1", "state1", "state2", "state2", "state3", "state3"), "obsevation" = c( "obs1", "obs2", "obs1", "obs2", "obs1", "obs2"), "pobability" = c( 0.1, 0.9, 0.3, 0.7, 0.4, 0.6)) A named list of m matices, one fo each actions. Each matix is of size n o whee n is the numbe of states and o is the numbe of obsevations. (each matix should have a name in the list and the name should be one of the actions). Matices can also be defined as "identity" o "unifom". obsevation_pob = list( action1 = matix(c(0.1, 0.9, 0.3, 0.7, 0.4, 0.6), now = 3, byow = TRUE), action2 = matix(c(0.1, 0.9, 0.3, 0.7, 0.4, 0.6), now = 3, byow = TRUE)) obsevation_pob = list( action1 = unifom, action2 = matix(c(0.1, 0.9, 0.3, 0.7, 0.4, 0.6), now = 3, byow = TRUE)) ewad: This agument coesponds to the ewad function R. The ewad function in its most geneal fom depends on action, stat-state, end-state and the obsevation. The ewad function can be specified seveal ways: 3

4 A data fame with 5 columns, whee the columns specify action, stat-state, end-state, obsevation and the ewad, espectively. The fist 4 columns could be eithe the name o the index of the action, state, o obsevation. The special chaacte * can be used to indicate that the same ewad applies fo all actions, states o obsevations. ewad = data.fame( "action" = c("action1", "action1", "action1", "action2", "action2", "action2"), "stat-state" = c("*", "*", "*", "*", "*", "*"), "end-state" = c("state1", "state2", "state3", "state1", "state2", "state3"), "obsevation" = c("*", "*", "*", "*", "*", "*"), "ewad" = c(10000, 2000, 50, 150, 2500, 100)) A named list of m lists, whee m is the numbe of actions (names should be the actions). Each list contains n named matices whee each matix is of size n o, in which n is the numbe of states and o is the numbe of obsevations. Names of these matices should be the name of states. ewad = list( "action1" = list( "state1" = matix(c(1, 2, 3, 4, 5, 6), now = 3, byow = TRUE), "state2" = matix(c(3, 4, 5, 2, 3, 7), now = 3, byow = TRUE), "state3" = matix(c(6, 4, 8, 2, 9, 4), now = 3, byow = TRUE)), "action2" = list( "state1" = matix(c(3, 2, 4, 7, 4, 8), now = 3, byow = TRUE), "state2" = matix(c(0, 9, 8, 2, 5, 4), now = 3, byow = TRUE), "state3" = matix(c(4, 3, 4, 4, 5, 6), now = 3, byow = TRUE))) values: This agument indicates whethe the poblem is minimization o a maximization. If the values ae costs then the poblem is a minimization and if they ae ewads then it is a maximization. The default is ewad. values = "cost" values = "ewad" name: This agument can be used to name the POMDP poblem defined by the use. This way the use can keep tack of the POMDP poblems he defines. name = "Test Poblem" Solving a POMDP POMDP poblems ae solved with the function solve_pomdp with the following aguments. st(ags(solve_pomdp)) function (model, hoizon = NULL, method = "gid", paamete = NULL, vebose = FALSE) The model agument is a POMDP poblem ceated using the POMDP function. The hoizon agument specifies the finite time hoizon (i.e, the numbe of time steps) consideed in solving the poblem. If the hoizon is unspecified (i.e., NULL), then the algoithm continues unning iteations till it conveges to the infinite hoizon solution (Anthony Rocco Cassanda 1998). The method agument specifies what algoithm the solve should use. Available methods including "gid", "enum", "twopass", "witness", and "incpune". Futhe solve paametes can be specified as a list as paametes. The list of available paametes can be obtained using the function solve_pomdp_paamete(). Finally, vebose is a logical that indicates whethe the solve output should be shown in the R console o not. The output of this function is an object of class POMDP. Helpe Functions The package offes seveal functions to help with managing POMDP poblems and solutions. The functions model, solution, and solve_output extact diffeent elements fom a POMDP object etuned by solve_pomdp(). The package povides a plot function to visualize the solution s policy gaph using the package igaph. The gaph itself can be extacted fom the solution using the function policy_gaph(). 4

5 The Tige Poblem Example We will demonstate how to use the package with the Tige Poblem (Anthony R. Cassanda, Kaelbling, and Littman 1994). A tige is put with equal pobability behind one of two doos, while teasue is put behind the othe one. You ae standing in font of the two closed doos and need to decide which one to open. If you open the doo with the tige, you will get hut by the tige (negative ewad), but if you open the doo with the teasue, you eceive a positive ewad. Instead of opening a doo ight away, you also have the option to wait and listen fo tige noises. But listening is neithe fee no entiely accuate. You might hea the tige behind the left doo while it is actually behind the ight doo and vice vesa. The states of the system ae the tige behind the left doo (tige-left) and the tige behind the ight doo (tige-ight). Available actions ae: open the left doo (open-left), open the ight doo (open-ight) o to listen (listen). Rewads associated with these actions depend on the esulting state: +10 fo opening the coect doo (the doo with teasue), -100 fo opening the doo with the tige. A ewad of -1 is the cost of listening. As a esult of listening, thee ae two obsevations: eithe you hea the tige on the ight (tige-ight), o you hea it on the left (tige-left). The tansition pobability matix fo the action listening is identity, i.e., the position of the tige does not change. Opening eithe doo means that the game estats by placing the tige unifomly behind one of the doos. Specifying the Tige Poblem The poblem can be specified using function POMDP() as follows. libay("pomdp") TigePoblem <- POMDP( name = "Tige Poblem", discount = 0.75, states = c("tige-left", "tige-ight"), actions = c("listen", "open-left", "open-ight"), obsevations = c("tige-left", "tige-ight"), stat = "unifom", tansition_pob = list( "listen" = "identity", "open-left" = "unifom", "open-ight" = "unifom"), obsevation_pob = list( "listen" = matix(c(0.85, 0.15, 0.15, 0.85), now = 2, byow = TRUE), "open-left" = "unifom", "open-ight" = "unifom"), ewad = data.fame( "action" = c("listen", "open-left", "open-left", "open-ight", "open-ight"), "stat-state" = c("*", "tige-left", "tige-ight", "tige-left", "tige-ight"), "end-state" = c("*", "*", "*", "*", "*"), 5

6 ) "obsevation" = c("*", "*", "*", "*", "*"), "ewad" = c(-1, -100, 10, 10, -100)) TigePoblem POMDP model: Tige Poblem $name [1] "Tige Poblem" $discount [1] 0.75 $states [1] "tige-left" "tige-ight" $actions [1] "listen" "open-left" "open-ight" $obsevations [1] "tige-left" "tige-ight" $stat [1] "unifom" $tansition_pob $tansition_pob$listen [1] "identity" $tansition_pob$`open-left` [1] "unifom" $tansition_pob$`open-ight` [1] "unifom" $obsevation_pob $obsevation_pob$listen [,1] [,2] [1,] [2,] $obsevation_pob$`open-left` [1] "unifom" $obsevation_pob$`open-ight` [1] "unifom" $ewad action stat.state end.state obsevation ewad 1 listen * * * -1 2 open-left tige-left * *

7 3 open-left tige-ight * * 10 4 open-ight tige-left * * 10 5 open-ight tige-ight * * -100 $values [1] "ewad" Solving the Tige Poblem Now, we can solve the poblem using the default algoithm (finite gid, a fom of point-based value iteation). tige_solved <- solve_pomdp(tigepoblem) tige_solved Solved POMDP model: Tige Poblem method: gid belief states: 5 total expected ewad: The output is an object of class POMDP which contains the solution. solution(tige_solved) POMDP solution $method [1] "gid" $paamete NULL $alpha coeffecient 1 coeffecient $pg belief_state action tige-left tige-ight 1 1 open-left listen listen listen open-ight 3 3 $belief tige-left tige-ight belief_state e e e e e e e e e e e e e e

8 e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e e+00 1 $belief_popotions tige-left tige-ight $total_expected_ewad [1] $initial_belief_state [1] 3 The solution contains the following elements: belief: A data fame of all the belief states (ows) used while solving the poblem. Thee is a column at the end that indicates which hypeplane (specified in alpha below) povides the best value fo the given belief state. belief_popotions: A data fame with the pobability distibution fo each belief state. alpha: A data fame with the coefficients of the optimal hypeplanes. pg: A data fame containing the optimal policy gaph. Rows ae belief states. Column two indicates the optimal action fo the belief stae. Columns thee and afte epesent the tansitions fom belief state to belief state depending on obsevations. total_expected_ewad: The total expected ewad of the optimal solution. initial_node: The index of the initial belief state in the policy gaph. Visualization In this section, we will visualize the policy gaph povided in the solution by the solve_pomdp function. plot(tige_solved) 8

9 2: listen Belief Popotions tige left tige ight tige ight tige ight 1: open left tige left tige ight tige left 3: initial listen tige ight tige left tige ight 5: open ight tige left tige left 4: listen The policy gaph can be easily intepeted. Without pio infomation, the agent stats at the belief state maked with initial. In this case the agent beliefs that thee is a chance that the tige is behind ethe doo. The optimal action is displayed inside the state and in this case is to listen. The obsevations ae labels on the acs. Let us assume that the obsevation is tige-left, then the agent follows the appopiate ac and ends in a belief state that has a vey high pobability of the tige being left. Howeve, the optimal action is still to listen. If the agent again heas the tige on the left then it ends up in a belief state that has open-ight as the optimal action. Thee ae acs back to the initial state which eset the poblem. Since we only have two states, we can visualize the piecewise linea convex value function as a simple plot. alpha <- solution(tige_solved)$alpha alpha coeffecient 1 coeffecient plot(na, xlim = c(0, 1), ylim = c(0, 10), xlab = "Belief space", ylab = "Value function") fo(i in 1:now(alpha)) abline(a = alpha[i,1], b= alpha[i,2], col = i) legend("topight", legend = 1:now(alpha), col = 1:now(alpha), lwd=1) 9

10 Value function Belief space Refeences Cassanda, Anthony R., Leslie Pack Kaelbling, and Michael L. Littman Acting Optimally in Patially Obsevable Stochastic Domains. In Poceedings of the Twelfth National Confeence on Atificial Intelligence. Seattle, WA. Cassanda, Anthony Rocco Exact and Appoximate Algoithms fo Patially Obsevable Makov Decision Pocesses. PhD thesis, Povidence, RI, USA: Bown Univesity. Littman, Michael L A Tutoial on Patially Obsevable Makov Decision Pocesses. Jounal of Mathematical Psychology 53 (3): doi: /j.jmp