Coarticulation: An Approach for Generating Concurrent Plans in Markov Decision Processes

Transcription

1 Coarticulation: An Approac for Generating Concurrent Plans in Markov Decision Processes Kasayar Roanimanes Sridar Maadevan Department of Computer Science, University of Massacusetts, Amerst, MA Abstract We study an approac for performing concurrent activities in Markov decision processes (MDPs) based on te coarticulation framework. We assume tat te agent as multiple degrees of freedom (DOF) in te action space wic enables it to perform activities simultaneously. We demonstrate tat one natural way for generating concurrency in te system is by coarticulating among te set of learned activities available to te agent. In general due to te multiple DOF in te system, often tere exists a redundant set of admissible sub-optimal policies associated wit eac learned activity. Suc flexibility enables te agent to concurrently commit to several subgoals according to teir priority levels, given a new task defined in terms of a set of prioritized subgoals. We present efficient approximate algoritms for computing suc policies and for generating concurrent plans. We also evaluate our approac in a simulated domain. 1. Introduction Every day in our life we constantly perform concurrent activities. By exploiting many degrees of freedom (DOF) in our body (e.g., arms, legs, eyes, etc), we are able to simultaneously commit to several tasks and as a result generate concurrent plans. As an example consider a driving task wic may involve subgoals suc as safely navigating te car, talking on te cell pone, and drinking coffee, wit te first subgoal taking precedence over te oters. Having te benefit of extra DOF in our body, we are able to simultane- Appearing in Proceedings of te 22 nd International Conference on Macine Learning, Bonn, Germany, Copyrigt 2005 by te autor(s)/owner(s). ously commit to multiple subgoals. For example we can control te weels by te left arm and use te rigt arm to answer te cell pone, or drink coffee. In general concurrent decision making is a callenging problem, since subgoals often ave conflicting objectives and compete for te limited DOF in te system. However, te main callenge wit tis problem is te combinatorial space of possible concurrent actions tat includes every possible combination of control signals (e.g., primitive actions in MDPs) for controlling te DOF in te system. In tis paper we study an approac for generating concurrent plans based on te coarticulation framework introduced in (Roanimanes et al., 2004). We demonstrate ow tis approac can cope wit te curse of dimensionality incurred in systems wit excess degrees of freedom, and tat it can be viewed as one natural way for generating concurrent plans. Te key idea is tat for many goal-oriented activities in addition to te optimal policy often tere exists a set of ascending (and possibly sub-optimal) policies tat guarantee acieving te goal wit a cost of a sligt deviation from optimality. Suc flexibility enables te agent to simultaneously commit to multiple subgoals. We argue tat coarticulation is a natural way for generating parallel execution plans for several reasons. First, many concurrent decision making problems can be actually viewed as concurrent optimization of a set of prioritized subgoals, in wic te agent manages its DOF to simultaneously commit to tem. Second, because of te multiple DOF in te system, learned activities offer more flexibility in terms of te range of ascending policies associated wit tem. For example in te driving task, wile te best policy for driving te car would be to control te weel using bot arms, by exploiting te extra DOF in our body we can perform te same task sub-optimally by engaging one arm for turning te weel and releasing te oter arm for committing to te oter subgoals of lower pri-

2 ority suc as drinking coffee, for example. However, te key advantage of coarticulation in concurrent decision making lies in its efficient searc in te exponential space of concurrent actions. Te action selection mecanism in tis approac is restricted to tose tat ascend te value functions associated wit eac activity. Tis interactive searc enables te agent to perform te searc in a muc smaller set of concurrent actions wit a controllable cost in optimality. In tis work, we also present approximate algoritms for scaling te coarticulation framework to large domains, suc as concurrent decision making in MDPs, were te concurrent action space is exponentially large. Unfortunately te algoritms presented in (Roanimanes et al., 2004) do not scale to suc large domains. Tus efficient algoritms for computing te set of ascending policies for activities, and also scalable algoritms for te action selection problem is required. Most of te related work in te context of Markov decision processes assume tat te subprocesses modeling te activities are additive utility independent (Boutilier et al., 1997; Sing & Con, 1998; Guestrin & Gordon, 2002) and do not address concurrent planning wit a set of learned activities modeled as temporally extended actions. In contrast we focus on problems were te overall utility function may be expressed as a non-linear function of sub-utility functions tat ave different priorities. Te rest of tis paper is organized as follows: In section 2 we briefly overview te coarticulation framework. In section 3 we present approximate algoritms for scaling te coarticulation framework to large problems suc as te concurrent decision making problem. We present our empirical results in a simulated domain in section 4. Finally, we summarize te paper in section 5 and describe some future directions. 2. Coarticulation in MDPs Coarticulation in MDPs (Roanimanes et al., 2004) can be viewed as te problem of simultaneously committing to multiple subgoals, placing more weigt on subgoals of iger priority. More formally, tis approac assumes tat te agent as access to a set of learned activities modeled by a set of minimum costto-goal ɛ-redundant controllers ζ = {C i } n i=1. Eac controller is designed to acieve a subgoal ω i from a set of subgoals Ω = {ω i } n i=1, and is modeled as a subgoal option (Precup, 2000) defined over an MDP M = S, A, R, P. A controller is ɛ-redundant if it admits multiple optimal or ɛ-ascending policies. A policy π is ɛ-ascending if it satisfies te following conditions: 1. Ascendancy: s S, 2. ɛ-optimality: E s {V (s )} V (s) > 0 (1) Ps π(s) s S, Q (s, π(s)) 1 ɛ V (s) (2) Te first condition (ascendancy) guarantees tat te policy arrives in a goal state in a bounded number of steps. Te second condition (ɛ-optimality) assures te policy deviates from te optimal policy by a factor inversely proportional to ɛ (note tat in minimum cost-to-goal problems all rewards are negative, except in a goal state). Given a minimum cost-to-goal controller C, we can compute te set of policies tat are ɛ-ascending on VC in every state s S by verifying te conditions in Equations 1 and 2 for every action a A(s). We define te ɛ-redundant set for every state s S as: A ɛ C(s) = {a A(s) a is eiter optimal, or ɛ-ascending} Now, given a subset of prioritized subgoals ω = {ω i } k i=1, te agent as to devise a policy tat simultaneously commits to te most possible number of subgoals according to teir degree of significance. We can tink of tis problem as a multi-criterion reinforcement learning problem (Gabor et al., 1998), in wic te reward signal is a vector wose elements are te rewards associated wit te controllers, and a lexicograpical ordering of suc reward vectors are defined according to te priority order of te controllers. For specifying te order of priority relation among te controllers we use te expression C j C i, were te relation expresses te subject-to relation (following (Huber, 2000)). Tis equation sould read: controller C j subject-to subtask C i. A priority ranking system is ten specified by a set of relations {C j C i }. Witout loss of generality we assume tat te controllers are prioritized based on te following ranking system: {C j C i i < j}. Te action selection mecanism in te coarticulation framework takes te ordered intersection of te ɛ-redundant sets computed for every controller in progress. Te ordered intersection operator works as follows: First, we set U 1 = A ɛ1 C 1, and ten for i = 2, 3,..., k we perform: U i = U i 1 A ɛi C i. If U i =, we cose U i = U i 1, and continue to te next iteration. After te algoritm terminates, U k returns a set of actions tat acieve te subgoals according to teir order of priority. Te computational cost of computing te redundant-sets in every state s A ɛ C (s) is linear in te number of states and actions:

3 O( S A ). Also te computational cost of performing te action selection mecanism in every state s is O((k 1) (max C A ɛ C (s) )2 ). w i H1 H 2 H k Coarticulation and Concurrent Decision Making In Section 1, we described several reasons for wy coarticulation could be one natural way for generating parallel execution plans in systems wit multiple DOF. For example in te driving task, we can design tree controllers associated wit te subgoals: C navigate, C cell, and C coffee, were eac controller is defined over an MDP M = S, A, R C, P. Te overall objective can be approximated in terms of te concurrent optimization of te subgoals using te following priority ranking system: C navigate C cell C coffee. For eac controller, we can compute te redundant sets, and ten perform te action selection algoritm tat we described in Section 2. As stated in Section 1, one immediate problem wit applying coarticulation to tis problem is te combinatorial space of concurrent actions. In tis example, te set of actions in te MDP M can be described via a set of action variables a = {a i } n i=1, were eac a i controls a DOF in te system. Eac assignment ā to te set of action variables denotes a concurrent action a A. In te driving task, for example, tere are action variables associated wit te arms, ands, eyes, ead, and so on. Te total number of concurrent actions is exponential in te number of action variables, and ence te algoritms for computing te redundant sets and te action selection mecanism tat we described in Section 2 are computationally intractable for suc problems. In te rest of tis section, we present an approximate metod for efficiently solving tis problem. Te general idea in our approac is rater tan verifying te conditions in Equations 1 and 2 for te exponential set of concurrent actions, wic is intractable, we only verify tem for te top concurrent actions tat ave te top state-action values in state s. We sow tat our approximate algoritm computes te top concurrent actions wit te computational complexity logaritmic in. Te parameter is an input parameter wic can be tuned to counterbalance te tradeoffs between te computational complexity and te flexibility of te controller. Let C be a redundant controller defined over an MDP M = S, A, R, P, were te set of concurrent actions A are described via a set of discrete action variables a = {a i } n i=1. Eac action variable a i takes on discrete H i T1( wi) T 2 ( w i) T k ( wi) k values Extract top values Ti ( wi) Figure 1. Visualization of step 5 of te Algoritm 1. Eac function H i returns a sorted table of size. From cross summation of table values across H i functions, a new table of te top summations is produced. values from some finite domain Dom(a i ). Witout loss of generality, we assume tat a i a, Dom(a i ) = d. We furter assume tat te optimal state-action value function Q associated wit te controller C is approximated using linear function approximation tecniques (Sutton & Barto, 1998) and admits te following linear additive form: Q (s, {a i } n i=1) Q (s, {a i } n i=1) = m Q i (s, u i ) i=1 (3) were eac Q i (s, u i ) is a local function defined over states and a subset of action variables u i a. We introduce an operator Γ a wic returns te top values of a function Q (s, a). By exploiting te linearity of te approximate state-action value function (Equation 3) we can use an algoritm in spirit similar to te variable elimination algoritm in Bayesian networks and efficiently compute Γ a. Q (s, a). Our approac is inspired by te action selection algoritm introduced in (Guestrin et al., 2002) tat actually solves te special case for = 1 (i.e., Γ 1 a), wic is te max a operator. It is also closely related to te problem of finding te most probable configurations in probabilistic expert systems (Nilsson, 1998). Te general idea is rater tan summing all local functions and ten performing te Γ a operator, we perform it over variables one at a time, using only summands tat involve te eliminated variable. For example, consider

4 te following state-action value function defined over a set of action variables a = {a 1, a 2, a 3 }: Q (s, a 1, a 2, a 3) = Q 1(s, a 1) + Q 2(s, a 1, a 2) + Q 3(s, a 2, a 3) by applying te Γ a operator and te variable elimination algoritm, we obtain: Γ {a 1,a 2,a 3 }. Q (s, a 1, a 2, a 3) = Γ {a 1,a 2,a 3 }.(Q 1(s, a 1) + Q 2(s, a 1, a 2) + Q 3(s, a 2, a 3)) = Γ {a 1 }. (Q 1(s, a 1) Γ {a 2 }. Q 2(s, a 1, a 2) Γ {a 3 }.Q 3(s, a 2, a 3) Note tat in te above equation, we used te special sum operator, because at eac elimination step te summation is performed over functions tat return te top maximum values of te past elimination steps, wic need to be combined in order to obtain te updated top maximum values as a result of te elimination of te next variable. Note tat in te special case = 1, tis operator turns into te plus operator. Te above procedure is summarized in Algoritm 1. Te key computational steps are te steps 5 and 6 of tis algoritm. Before describing te details of tese two steps, first we introduce some useful notation. Let H(w) denote a one-to-many function defined over a set of variables w. Figure 1 sows a tabular view of tis function, and demonstrates te details of te computations performed in te step 5 of te Algoritm 1. For every setting of variables w, it returns te sorted top values and assignments to te subset of eliminated variables from te previous steps (tables T ( w) in Figure 1). Before te elimination algoritm starts, we can represent eac summand Q i (s, u i ) as some function H i (u i ) (for simplifying equations, we drop te state s from te notations), were every assignment of te variables ū i is mapped to a single value Q i (s, ū i ). Assume tat te algoritm is at iteration i, were te variable a i is selected for elimination. Let {H j } k j=1 be te set of summands tat involve te variable a i. Also let y i denote te rest of te variables involved in {H j } k j=1 tat are connected to a i, and let w i = y i {a i }. As sown in Figure 1, for every setting of variables w i summand H j returns a sorted top values and also te assignments to a subset of past eliminated variables (represented as tables T j ( w i )). Tere are k suc tables and we need to compute te top maximum values from te set of all cross summations of k elements, one from eac table T j ( w i ). Tere are k suc values and a naive approac would first compute te wole k summations, and ten extract te top maximum values, wit te computational complexity of O( k (k 1) + log()) (te first term is te complexity of computing te summations, and te second term denotes te complexity of sorting tese values). However considering tat eac table is sorted, we can perform te above computation more efficiently. Rater tan summing all te values across all tables, we perform te summation over two tables at a time, and extract a new table wit te top maximum values of te pairwise table summation. We ten repeat it for te rest of te tables. Wen performing te pairwise cross summation over two tables, we only need to perform te summation only over te top elements from eac table, since te tables are sorted. Te computational complexity of tis metod is O((k 1)( + log())). Te final top elements are stored in a new function H i (w i ) for te setting w i. Algoritm 1 Function Γ a Inputs: s Current state m i=1 Q i(s, u i ) Q function {a i } Elimination order Number of top max elements Outputs: {ā i, v i } i=1 Top assignments and values 1: Let F = {Q i (s, u i )} m i=1 set of summands 2: wile not all variables eliminated do 3: Pick te next variable a i 4: Extract all summands {H j } from F tat involve a i 5: Perform: H i ({H j }) 6: Eliminate a i from H i to obtain H i 7: Add H i to F 8: end wile Te details of te computations of step 6 of te Algoritm 1 are demonstrated in Figure 2. Note tat step 5 returns a newly introduced function H(w i ) tat involves te variable a i. Te elimination takes place in step 6. First, a new function H (y i ) is introduced tat involves only te variables connected to a i (i.e., y i ). Every setting ȳ i is mapped to d tables (were Dom(a i ) = d), eac for one assignment of te variable a i. Eac table contains te top values and settings for a subset of eliminated variables in te previous steps. We need to extract te top values across tese tables. Tere are d sorted tables of size, and we can extract te top values across tem wit te computational complexity of O(.d). Te new set of values are ten stored in te function H i (y i) for te setting ȳ i (see Figure 2). Tis yields te overall computational complexity of

5 O(n d w (klog() +.d)) = O(n k d w log()) for te Algoritm 1. Tis complexity is logaritmic in, and exponential in te network widt (Decter, 1999) induced by te structure of te approximate stateaction value function. w y i = ia i H i. d Dom(a ) = d i d tables Table 1. Action Variables Left arm (a l ) Rigt arm (a r ) Eyes (a e ) pick pick fixate-on-waser waser-to-front waser-to-front fixate-on-front front-to-rack front-to-rack fixate-on-rack rack-to-front rack-to-front no-op front-to-waser front-to-waser stack stack no-op no-op Hi T i (y 1) i T i (y i2) T i (y id)... T(y ) i Given a set of redundant controllers {C i } k i=1, we can perform Algoritm 1 and compute te redundant sets for eac controller. Note tat te cardinality of te redundant set for te controller C i is at most i. Tus te computational complexity of te action selection mecanism tat we described in Section 2 in every state s becomes O((k 1) (max i i ) 2 ) wic is polynomial in. Extract top values Figure 2. Visualization of step 6 of te Algoritm 1 were te variable a i is eliminated. 4. Experiments In tis section we present a concurrent decision making task and apply te coarticulation approac using te approximate metods tat we described in Section 3. Figure 3 sows a robot wit tree degrees of freedom, By performing Algoritm 1 in state s, we obtain te top concurrent actions and teir values. We can ten verify te ascendancy and ɛ-optimality conditions tat we described in Section 2, for eac action. In eiter case, we need to compute V (s). From te Bellman optimality equation (Sutton & Barto, 1998) we ave V (s) = max a Q (s, a) Γ 1 Q a (s, a) wic can be computed using Algoritm 1. Tus verification of te ɛ-optimality condition can be efficiently done. For te ascendancy condition, we need to compute te expected optimal value of te next states given tat te concurrent action ā is executed in state s. Expanding te optimal state-action value function for te action ā yields: Q (s, ā) = R(s, ā) + γe s Pās {V (s )} by subtracting γv (s) from bot sides and rearranging te terms, we obtain: E s Pās {V (s )} V (s) = 1 γ (Q (s, ā) γr(s, ā) γv (s)) Note tat te rigt and side of te Equation 4 can be efficiently computed for a concurrent action ā and can be used to verify te ascendancy condition. Dis Rack Front Dis Waser Figure 3. Te robot s task is to empty te dis-waser and stack te plates in te dis-rack. namely, te eyes, te left arm, and te rigt arm. Te robot s task is to empty te dis-waser and stack te plates in te dis-rack. Eac arm of te robot at any time can be in tree predefined positions: waser, rack, and front as sown in Figure 3. In order to make a successful arm movement from a source position to a target position, te robot needs to first fixate at te target position. Te eyes of te robot can also fixate on any of tese positions. Te set of actions tat te robot can perform is described via a set of action variables a = {a l, a r, a e }, eac controlling a DOF in te system. Action variable a l controls te left arm, a r controls te rigt arm, and a l controls te eyes of te robot. Table 1 sows te control actions for eac action vari-

6 Table 2. State Variables s waser l pos, r pos, e pos l stat, r stat s rack 0, 1..., n waser as-plate stacked front empty not-stacked rack able. Tere are control actions tat move one arm from a source position to a destination position. However, te arms cannot move directly from waser to rack, and vice versa. In order to perform suc movements, te robot needs to first move te arm from te source position to te front position, and ten from tat position to te target position in two primitive steps. Te control action pick picks up a plate from te waser, if te arm is positioned at te dis-waser, and tere is a plate to pick up. Te control action stack stacks a plate into te dis-rack if te arm is olding a plate and is positioned at te dis-rack. Te robot can also transfer a plate from one arm to te oter, if bot arms are positioned in front of te robot, and te empty arm executes te pick control action. Te control actions for te eye movements cause te robot to fixate on te specified position. Tere is also a noop action for eac DOF, tat does not influence tat DOF. Te states of te robot are also described via a set of state variables summarized in Table 2. State variable s waser keeps track of te number of plates in te diswaser. State variable l pos sows te current position of te left arm (i.e, waser, front, rack). State variable l stat describes te current status of te left and, i.e., weter it is olding a plate or it is empty. Similarly state variable r pos and r stat describe te position and status of te rigt arm. State variable e pos, describes te current gaze of te robot s eyes. Finally, state variable s rack describes weter or not a plate as been stacked in te dis-rack. Any assignment to te set of action variables forms a concurrent action. However, not all concurrent actions are allowed for execution in every state. Actions are pruned to simplify learning and enforce safety constraints (Huber, 2000). For example te left arm can execute te action pick only wen it is located at te dis-waser and is empty, and tere is also a plate to pickup. Any concurrent action tat violates te safety constraints is referred to as an invalid action. If te robot executes an invalid action, it receives a large negative reward. Te actions tat control te gaze of te robot reflect te limitations of a real robot system. Te robot is required to look at a target position before being able to move any of its arms to tat position. Recall tat te overall objective is to empty te diswaser and stack te plates in te dis-rack in te smallest number of steps. Tis objective can be approximated in terms of concurrent optimization of two competing subgoals: ω stack, and ω pick, wit te priority ranking system: ω stack ω pick. ω stack is te subgoal of stacking a plate in te dis-rack, and ω pick is te subgoal of picking up a plate from te diswaser. Tese subgoals compete for te DOF in te robot (i.e., eyes and arms). We design two controllers C pick, and C stack tat acieve eac subgoal. Note tat suc controllers can be viewed as general purpose object manipulation controllers for picking up and stacking objects across different tasks. We can model te controller C pick as an option I pick, π pick, β pick, were I pick is te set of states at wic te robot can pick up a plate. Tis consists of te states were at least one of te robot s ands is empty and tere is a plate in te dis-waser. Te policy π pick, specifies a closed loop policy for picking up a plate. β pick defines te termination condition for tis option and it occurs wen te robot picks up a plate from te dis-waser. Similarly, we can model te controller C stack as an option I stack, π stack, β stack, were I stack is te set of states at wic te robot can stack a plate. Tis consists of te states were te robot is olding at least one plate. Te policy π stack specifies a closed loop policy for stacking a plate. β stack defines te termination condition for tis option and it terminates wen te robot stacks a plate in te dis-rack. Note tat due to multiple DOF in te system, bot controllers are ɛ-redundant for some ɛ. For example te robot can pick up a plate eiter by its left arm, or by its rigt arm. Or, it can stack a plate eiter by moving te arm tat is currently olding te plate to te dis-rack and stack te plate, or it can and it to te oter and and use te oter and to stack it. It can be verified tat te sequential solution (no coarticulation) tat involves executing C pick and ten C stack in sequence, does not provide te most efficient solution. For example wile te robot is stacking a plate eld by its rigt and, it can concurrently pick up a new plate wit its left and. By coarticulating between tese two controllers, te robot can perform an action tat acieves te objective of te superior controller (i.e., C stack ), wile committing to te objective of te subordinate controller (i.e., C pick ), if te intersection of te redundant sets of tese two controllers is non-empty in te current state. To furter

7 Table 3. CMACs tilings Tiling # of tiles Q 1 (s waser, l pos, l stat ) 12 Q 2 (s waser, r pos, r stat ) 12 Q 3 (s waser, s rack, r pos, r stat ) 24 Q 4 (s waser, s rack, a l, a r ) 196 Q 5 (s waser, l pos, l stat, e pos, a l ) 252 Q 6 (s waser, r pos, r stat, e pos, a r ) 252 Q 7 (s waser, s rack, l pos, l stat, e pos, a l ) 504 Q 8 (s waser, l pos, l stat, e pos, a l, a e ) 1008 Q 9 (s waser, r pos, r stat, e pos, a r, a e ) 1008 Q 10 (s waser, l pos, r pos, e pos, a l, a r ) 2646 In order to learn te optimal state-action value function associated wit eac controller, we used sparsecoarse-coded function approximator (CMACs) (Albus, 1981) combined wit Sarsa(λ) algoritm (Sutton, 1996). We used CMACs consisting of 10 tilings (for te total of 5914 tiles) as listed in Table 3. For eac controller, we learned te approximate value function wic can be expressed as Q (s, {a l, a r, a e }) = 10 i=1 Q i(s i, a i ), were s i and a i are te subset of state and action variables tat are involved in te tiling Q i. Next, we applied te approximation algoritm tat we described in Section 3 for computing te ɛ- redundant sets for eac controller, using te elimination order {a l, a r, a e }. Figure 4 sows te performance Steps seq coart:ep 0.85 coart:ep illustrate tis, consider te following scenario: assume tat te robot is currently olding a plate wit its rigt arm positioned at te front and te controller C stack is in progress. Also, assume tat its left arm is positioned at front and is empty. In tis state, te robot can execute at least two ɛ-ascending actions wit respect to te C stack controller: (1) move te rigt arm to te dis-rack and concurrently look at te front position; (2) move te rigt arm to te dis-rack and concurrently look at te dis-waser position. Note tat te second action is also ɛ-ascending wit respect to te C pick controller, since by looking at te diswaser position, te robot can ten move its empty left arm to te dis-waser in order to pick a plate. By coarticulating between tese two controllers, te robot executes te second action tat is ɛ-ascending wit respect to bot controllers. Note tat wile tis action moves te robot s rigt arm to te dis-rack to stack te plate, concurrently it moves te empty left arm to te dis-waser in order to pick a new plate. In our experiments, bot controllers are defined over an MDP M = S, A, R, P, were te states and actions are described via te set of variables given in Tables 2, and 1. All actions are stocastic; tey succeed wit probability p and fail wit probability (1 p). Wen actions succeed, tey cange te state of te robot to te next state as described above. Upon failure, or executing an invalid action, te robot does not cange its state. All actions are also rewarded 1 upon termination State Figure 4. Performance of te coarticulation approac using different values of ɛ and te sequential approac. of te coarticulation metod, and also te sequential approac in wic te C stack and C pick are executed in sequence (using teir optimal policy) wit no coarticulation. Te performance is measured in terms of te total number of steps for completion of te task. Tese results are averaged over 20 tasks, eac consisting of 27 episodes. Eac episode is associated wit a starting state, wit 20 plates in te dis-waser, and te robots arms are set to empty. Te orizontal axis depicts te starting states. An starting state is defined in terms of te various configurations of te state variables (e.g., initial positions of te arms and te eyes, and teir status, etc) tat are relevant to te task (i.e., at least one controller can be initiated). Te bottom two plots are te performance of te coarticulation framework using ɛ = 0.70, 0.85, and = 10 wen computing te redundant sets for eac controller. From tese results, we can observe tat te coarticulation approac outperforms te sequential case in every starting state. Figure 5 sows te total number of coarticulation in te above task wen te system is initialized in different starting states, for different values of ɛ. Note tat by coosing ɛ = 0.99, te controllers offer less flexibility and ence te total amount of coarticulation decreases. Next, in order to measure te accuracy of our approximate metod wen computing te redundant sets, we computed te exact value function witout te function approximation, and ten computed te redundant sets. Figures 6 and 7 sows te total number of its and misses of te ɛ-ascending actions in ev-

8 Average Coarticulation State coart:ep 0.99 coart:ep 0.85 coart:ep 0.70 Figure 5. Total number of coarticulation for different starting states, and different values of ɛ. Number of its coart:ep Pick controller State Figure 6. Number of correct ɛ-ascending actions computed by te approximate metod for controller C pick in every state tat te controller can be executed. ery state in wic te controller C pick can be executed. As it can be seen from tese figures, our approximation tecnique as missed only a few actions in a small number of states. We also measured te false positive rate in every state and interestingly tere was no false alarm in neiter of states. Tis suggests tat our approximation metod can compute te redundant sets wit a ig precision. 5. Concluding Remarks In tis paper we studied an approac for scaling te coarticulation framework to large domains, suc as concurrent decision making in systems wit excess DOF. We presented an efficient approximate algoritm for computing te set of ɛ-redundant policies for redundant controllers. Altoug we considered structured actions, we did not fully exploit te underlying structure tat many MDPs offer. We are currently investigating ow suc structure could be exploited for more efficient action selection mecanism in te coarticulation framework. Number of misses coart:ep Pick controller State Figure 7. Number of missed ɛ-ascending actions computed by te approximate metod for controller C pick in every state tat te controller can be executed. Acknowledgments We would like to tank Natan Srebro for is useful comments and discussion. Tis researc was supported in part by te National Science Foundation under grant ECS References Albus, J. (1981). Brain, beavior, and robotics. ByteBooks. Boutilier, C., Brafman, R., & Geib, C. (1997). Prioritized goal decomposition of Markov decision processes: Towards a syntesis of classical and decision teoretic planning. Proceedings of te Fifteent International Joint Conference on Artificial Intelligence (pp ). San Francisco: Morgan Kaufmann. Decter, R. (1999). Bucket elimination: A unifying framework for probabilistic inference. Artificial Intelligence. Gabor, Z., Kalmar, Z., & Szepesvari, C. (1998). Multi-criteria reinforcement learning. Guestrin, C., & Gordon, G. (2002). Distributed planning in ierarcical factored mdps. In te Proceedings of te Eigteent Conference on Uncertainty in Artificial Intelligence (pp ). Edmonton, Canada. Guestrin, C., Lagoudakis, M., & Parr, R. (2002). Coordinated reinforcement learning. In Proceedings of te ICML-02. Sydney Australia. Huber, M. (2000). A ybrid arcitecture for adaptive robot control. Doctoral dissertation, University of Massacusetts, Amerst. Nilsson, D. (1998). An efficient algoritm for finding te m most probable configurations in bayesian networks. Statistics and Computing, 8, Precup, D. (2000). Temporal abstraction in reinforcement learning. Doctoral dissertation, Department of Computer Science, University of Massacusetts, Amerst. Roanimanes, K., Platt, R., Maadevan, S., & Grupen, R. (2004). Coarticulation in markov decision processes. Proceedings of te Eigteent Annual Conference on Neural Information Processing Systems: Natural and Syntetic. Vancouver, Canada. Sing, S., & Con, D. (1998). How to dynamically merge markov decision processes. Proceedings of NIPS 11. Sutton, R., & Barto, A. (1998). An introduction to reinforcement learning. Cambridge, MA.: MIT Press. Sutton, R. S. (1996). Generalization in reinforcement learning: Successful examples using sparse coarse coding. Advances in Neural Information Processing Systems (pp ). Te MIT Press.