A Variable Elimination Approach for Optimal Scheduling with Linear Preferences

Transcription

1 A Varable Elmnaton Approach for Optmal Schedulng wth Lnear Preferences Ncolas Meuleau and Robert A. Morrs NASA Ames Research Center Moffet Feld, Calforna , USA Nel Yorke-Smth SRI Internatonal Menlo Park, Calforna 9425, USA Abstract In many practcal schedulng problems, feasble solutons can be partally ordered accordng to dfferences between the temporal objects n each soluton. Often, these orderngs can be computed from a compact value functon that combnes the local preference values of the temporal objects. However, n part because t s natural to vew temporal domans as contnuous, fndng complete, preferred solutons to these problems s a challengng optmzaton task. Prevous work acheves tractablty by makng restrctons on the model of temporal preferences, ncludng lmtng representatons to bnary and convex preferences. We propose an applcaton of Bucket Elmnaton (BE) to solve problems wth pecewse lnear constrants on temporal preferences wth contnuous domans. The key techncal hurdle s developng a tractable elmnaton functon for such constrants. Ths proof of concept takes a step toward an ablty to solve schedulng problems wth rcher models of preference than prevously entertaned. Further, t provdes a complementary approach to exstng technques for restrcted models, because the complexty of BE, whle exponental n the treewdth of the problem, s polynomal n ts sze. Introducton Practcal problems requrng the assgnment of tmes to planned events are pervasve n the world, from mundane applcatons lke calendar management (Mofftt, Pentner, & Yorke-Smth 26) to the more exotc such as scence observatons for planetary exploraton (Bresna, Khatb, & McGann 26). In cases such as these, feasble temporal assgnments can be ordered on the bass of the degree to whch certan preferences are adhered to. Such preferences can sometmes be expressed as a smple functon of tmes, (e.g., t s preferable that an event start as early as possble) but may also express complex relatonshps (e.g., t s preferable that a set of events have roughly the same duraton). There are numerous efforts n representng and solvng problems n decson makng wth preferences. Practcal problems are often challengng because they nvolve many preference crtera over contnuous temporal varables. Much of the prevous work n optmzaton of temporal preferences has sought to manage the potental complexty of Carnege Mellon Unversty such problems by lmtng the expressve power of the language for expressng preferences. A common model, the Smple Temporal Problem wth Preferences (STPP) (Morrs et al. 24) expands the Smple Temporal Network (Dechter, Mer, & Pearl 1991) by allowng numercal preferences on the admssble values of temporal varables. STPPs combne both hard constrants that must be satsfed for a soluton to be vald, and soft constrants that assgn a preference value to each admssble assgnment. Global preferences are obtaned by summng the local preferences of temporal objects. STPPs nhert from Smple Temporal Problems (STPs) the lmtaton to unary and bnary constrants. Moreover, all exstng effcent soluton technques for STPPs assume convex or sem-convex preferences. Notably, the most sgnfcant result n STPPs s the use of Lnear Programng (LP) to solve problems wth bnary pecewse lnear convex preferences as descrbed n (Morrs et al. 24). Despte such progress n solvng plannng and schedulng problems wth temporal preferences, restrctons mposed by bnary, convex preference functons result n lmtatons n the expressve power of the language of constrants. Consder a natural temporal preference such as Maxmze the temporal gap (duraton) between A and B. Graphcally, ths preference could be expressed as a V-shaped non-convex functon from tmes to preference values that reaches (the least preferred value) at tme as shown n Fg. 1. Removng the restrcton on constrants mposed by the STPP paradgm requres an alternatve approach to fndng solutons. A natural course of acton at ths pont s to consder general approaches to constrant optmzaton, based ether on condtonng (backtrackng, branch and bound) or on Bucket Elmnaton (BE), whch s a generalzaton of dynamc programmng (Dechter 1999; 23). The nterest n ths paper s n buldng a temporal solver based on BE for a model of tme n whch constrants are pecewse lnear over contnuous domans, but not necessarly bnary or convex. The result of ths work s the ablty to solve schedulng problems wth rcher models of preference than prevously entertaned. Further, we provde a complementary approach to exstng technques for restrcted models, because the complexty of BE, whle exponental n the treewdth of the problem, s polynomal n ts sze. The man contrbuton of ths paper s the proof of the soundness of an approach whose compettveness wll be explored n future work.

2 Fgure 1: A non-convex functon over 1 t 1 that expresses the preference: maxmze the temporal dfference between two events. The next secton surveys prevous work on BE. There follows the defnton of the class of contnuous temporal problem consdered. It s based on pecewse lnear value functons that encode both hard and soft constrants, and that are not necessarly bnary or convex. We then show that BE can be appled to ths doman, demonstratng by constructon that elmnaton preserves pecewse lnearty. To mprove the effcency of the elmnaton process, the noton of the wtness s appled to the elmnaton functon, leadng to our fnal algorthm. Fnally, prelmnary expermental results usng the full BE algorthm are summarzed. Bucket Elmnaton Bucket Elmnaton (BE) s a complete approach to solvng constrant optmzaton problems that generalzes dynamc programmng (Dechter 1999; 23). Intutvely, BE solves a problem by a seres of replacements of varables by constrants that summarze the effect of each varable on the best soluton to the problem. In ths approach, hard and soft constrants on an ordered set of varables X = {x 1, x 2,...x n } are represented as numercal functons, here called value functons V, defned over a subset of X called the scope of V. A hard constrant s represented by a value functon that takes the value n each state where t s satsfed, and elsewhere. A soft constrant s represented by a functon takng fnte values that reflects the preferences over the varables n ts scope. A value functon can take both fnte and nfnte values and represent both soft and hard constrants at the same tme. Gven a set V of such preference functons, bucket B() s defned as the set of constrants havng x n ther scope as ther hghest varable, accordng to the orderng of X. For a fxed, we call X = {x j1, x j2,... x jk, x } the smallest subset of varables such that the scope of each utlty n B() s ncluded n X. We then defne X = X \ {x } = {x j1, x j2,... x jk }. Techncally, X regroups all parents of x n the nduced constrant graph (Dechter 1999). BE elmnates varables n reverse order by creatng a constrant V for each x elmnated followng V = Elm ( V B() V ) whch s defned as (x j1, x j2,... x jk ) : V (x j1, x j2,... x jk ) = max V (x j1, x j2,... x jk, x ). x V B() V s added to the bucket B(j k ), where j k s the ndex of the hghest varable n X. BE termnates when all the varables are elmnated (the resultng functon V s then a constant functon returnng the hghest value of any soluton to the problem). The optmal state x that yelded the hghest value can be recovered easly based on ths computaton. The complexty of BE strongly contrasts wth that of standard condtonng algorthms such as backtrackng search and branch and bound (Dechter 1999). The worst-case complexty of these algorthms s O(m n ) where n s the number of varables and m s the sze of the dscrete doman of each varable. Therefore, t s exponental n the problem sze (number of varables). In contrast, the complexty of BE s exponental n a parameter called the treewdth (or nduced wdth) of the problem and, f the treewdth s fxed, polynomal n the problem sze. The nduced wdth (Dechter 1999) depends on the orderng of varables n X. Therefore, research as been drected toward fndng varable orderngs that mnmze the treewdth. Unfortunately, t turns out to be an NP-hard problem; (Dow & Korf 27) s an example of the heurstc approach. It s mportant to note that all prevous treewdth mnmzaton technques can be appled to the contnuous framework descrbed n ths paper as well as to any other nstance of BE. BE has been used successfully to solve Constrant Optmzaton Problems (Dechter 23). The key contrbuton of ths paper s to apply t to schedulng problems wth contnuous domans. Problem Formulaton We defne a framework where pecewse lnear value functons are used to represent both soft and hard constrants over a set of temporal varables X = {x 1, x 2,... x n } wth contnuous domans. Defnton 1 For any subset of contnuous varables X = {x 1, x 2,...x k }, let R[X] denote the k-dmensonal contnuous space R k where the axes are labeled wth varables n X. Defnton 2 Gven a subset of contnuous varables X = {x 1, x 2,...x k }, a (k + 1)-tuple (α, 1, 2,... k ) s used to represent several quanttes: A lnear constrant C that s true n all states x = (x 1, x 2,...x k ) such that α + k l x l. We (1) say that x satsfes C ff C s true n x; A lnear utlty functon u that assocates the reward u(x) = α + k l x l to each state x = (x 1, x 2,...x k ). To represent a hard constrant, the constant α can take the value ;

3 A lnear polcy µ mplementng the control law x α + k l x l for a gven ndex / ( 1, 2,... k ). The set X s called the scope of the constrant, utlty, or polcy. Defnton 3 A convex polygon s defned as a set of lnear constrants, C = {C, }. Each constrant represents a face of the polygon (the term face s sometmes used n a twodmensonal settng, and sngular pont n 1D). The ponts belongng to the polygon are those that satsfy all constrants n C. The scope of a polygon s the unon of the scopes of ts faces. Note that ths s a loose defnton of a polygon that encompasses unbounded peces. Defnton 4 A value functon pece s defned as a par P = (C, u) where C s a convex polygon and u s a lnear utlty functon. To store the optmal polcy, some peces also carry a lnear polcy µ. The scope of a pece s the set of varables appearng n ts edges, utlty functon, or polcy. Defnton 5 A pecewse lnear value functon wth scope X = {x 1, x 2,... x k } s defned as a collecton of value functon peces V = {P, } such that the polygons of the peces P consttute a partton of R[X], and the scope of each pece P s ncluded n X. Defnton 6 A lnear constrant optmzaton problem s a par P = (X, V) where X = {x 1, x 2,... x n } s a set of contnuous temporal varables and V s a set of pecewse lnear value functons V = {P, } wth scope ncluded n X. A soluton of P s a state x = (x 1..., x n ) R[X]. A soluton x s optmal f t maxmzes V V V (x). Solvng Lnear Constrant Optmzaton Problems usng Bucket Elmnaton Formally, BE s based on two operators over functons: summng and varable elmnaton (Elm ). The soundness of BE for pecewse lnear value functons s formally derved from Theorem 1, whch we establsh from two Lemmas. Lemma 1 The sum of a fnte number of pecewse lnear value functons s a pecewse lnear value functon. Proof: The sum of two value functons s computed by enumeratng all non-empty ntersectons of two of ther peces, and summng the utlty functons carred by these peces. Because the ntersecton of two convex polygons s a convex polygon, and the sum of two lnear utlty functons s a lnear utlty functon, t results n a well-defned pecewse lnear value functon. Lemma 2 If Q s a pecewse lnear value functon wth scope X = {x j1, x j2,...x jk, x }, then Elm (Q) s a pecewse lnear value functon wth scope X = {x j1, x j2,... x jk }. Proof: To prove ths lemma, we show how Elm (Q) can be computed and make clear that, by constructon, the resultng functon s a pecewse lnear value functon. The frst operaton n the calculaton of Elm (Q) s to project the partton of R[X] defned by Q on R[X ] (or, loosely speakng, to project Q on X ). It conssts of computng a partton of R[X ] n convex polygons, as shown n Fg. 2 for X = {X, X 1 }. Ths s acheved by frst projectng the peces of Q on R[X ], whch creates only convex polygonal peces, and then computng all possble ntersectons of projected peces. Therefore, the partton computed by projecton contans only convex polygonal peces. Consder now a partcular projected pece P and call PQ the set of peces of Q that contrbutes to P (ntersectng the horzontal strpe assocated wth P n Fg. 2). If as any translaton of the x -axs that projects nsde of P, then enters P through a well-defned face C P and exts through another face C + P (see Fg. 2). Because utlty s lnear over P, maxmum utlty s attaned on a face CP that s ether C P or C+ P. Ths allows dervng an analytcal expresson of the optmal decson rule for x. If the optmal of pece P s attaned on the face represented by C P : αc + k αc j l x jl + α C x (C P = C P or C+ P ), then the optmal decson rule (polcy) for pece P s µ P : x αc α C α C j l α C x jl. (2) Ths rule guarantees that α C + k αc j l x jl + α C x =, and thus that we are choosng a pont on the face represented by CP. As long as parent varables (x j 1, x j2,...x jk ) are n P, the optmal value of Q over pece P s attaned by µ P, whch s a lnear polcy wth scope X. Now that the optmal polcy for pece P s known, t s easy to derve an analytcal expresson of the optmal value of Q over P. If u P s the lnear utlty assocated wth pece P and µ P s the optmal polcy n P, then the optmal value s the return of u P when x s chosen followng µ P. In other words, f u P (x) = α u + k αu j l x jl +α u x and µ P : x α µ + k αµ j l x jl + α µ x, then an analytcal expresson of V p s obtaned by substtutng x for µ P n u P : V P (x) = α u + α u α µ + ( α u jl + α u α µ ) j xjl l. (3) The resultng utlty s lnear wth scope X. To compute Elm (Q), we need to maxmze Q wth respect to x. By defnton: (x j 1, x j 2,...x j k ) P : [ max Q(x x j1, x j 2,...x j k, x ) ] = [ max Vp (x P PQ j 1, x j 2,...x j k ) ]. However, the pece P that maxmzes Q may vary over P. Therefore, P may be splt n at most as many peces as n P Q. For each P P Q, we buld a convex polygon C P by frst copyng all faces of P and then addng to ths set the followng constrants: P P Q, P P : u P (x) u P (x), (4)

4 x P C P C + P P x 1 Algorthm 1 A straghtforward mplementaton of Th. 1 1: Compute an analytcal representaton of Q = P V B() V. 2: Project Q on X. 3: for each projected pece P do 4: for each pece P of Q that contrbutes to P (P P Q) do 5: Compute the optmal return V P of P followng Eqn. 3. 6: Compute the constrant set C P as n proof of Lemma 2. 7: f C P s consstent then 8: Add the pece (C P,V P) to V. 9: end f 1: end for 11: end for Fgure 2: Projectng of Q on X = {x }: The convex polygonal partton of the plane represents the peces of Q. Horzontal strpes represent the partton of R[X ] obtaned by projectng Q. where u P and u P are the lnear utlty functons carred by P and P. These constrants are well-defned lnear constrants wth scope X. They ensure that the optmal of Q over P s carred by pece P. If the resultng set C P s consstent, then C P s a well-defned polygon and the trple (C P, V P, µ P ) s a well-defned value functon pece of Elm (Q). Because Elm (Q) s made only of such peces, t s a well-defned pecewse lnear value functon. Theorem 1 If bucket B() contans only pecewse lnear value functons, then the functon V resultng from an applcaton of Eqn. 1 s also a pecewse lnear value functon. Proof: The clam results drectly from Lemmas 1 and 2. Bucket Elmnaton s fundamental equaton (Eqn. 1) starts wth a unversal quantfcaton over (x j1, x j2,... x jk ). In the lnear framework defned n the prevous secton, these varables have nfnte contnuous domans. Therefore, applyng Eqn. 1 requres a contnuous nfnty of maxmzatons. Fortunately, Theorem 1 shows that the computaton can be performed only once for each pece of V, and there s a fnte number of such peces. A drect applcaton of Th. 1 solves P V, x = (x j1, x j2,... x jk ) P : V (x) = max V (x j1, x j2,... x jk, x ), x V B() ths computaton beng performed analytcally. That s, an analytcal expresson of V (and the assocated optmal polcy) that apples to the whole pece P s computed nstead of a sngle numercal value. In ths way, Eqn. 1 s mplctly solved over the nfnte set of values. Algorthm 1 s a straghtforward procedure computng Eqn. 5. It follows closely the proof of Lemmas 1 and 2. It s mportant to note that, for a fxed treewdth, the complexty of ths procedure s polynomal n the sze of the problem, whch s a vector four varables: () The number of temporal varables n; () The number of value functons V B(); () the maxmum number of peces n a functon V B(); (5) (v) the maxmum number of constrants n a pece of a functon V B(). The algorthm s exponental n the treewdth because t has to buld a pecewse lnear value functon V wth dmenson as hgh as the treewdth. Whle beng conceptually smple, Alg. 1 presents two bottlenecks: () computng the multdmensonal sum Q, and () projectng Q on X. Although t s guaranteed to be polynomal n the problem sze as long as the treewdth s bounded, these operatons become expensve when the treewdth ncreases, because the sze of the largest functon V that the algorthm has to buld s equal to the nduced wdth of the problem. For nstance, calculatng Q requres computng all non-empty ntersectons of peces of functons n B(). Computng non-empty ntersectons s done by consderng all possble combnatons of peces, puttng all the faces of these peces n a sngle set, and testng the consstency of ths set. If the set s consstent, then the peces have a non-empty ntersecton. In hgh dmensons, ths consstency test s expensve, makng the summaton an overly expensve process. In the next secton, we present an algorthm that avods these two ptfalls. A Wtness Algorthm Algorthm 2 s a wtness algorthm that effcently mplements Th Ths algorthm assocates wth every pece P of V a wtness state-vector x P = (x P j 1, x P j 2,...x P j k ) that testfy to the exstence of P n V. It then performs the fnte calculaton P V : V (x P j 1, x P j 2,... x P j k ) = max V (x P x j 1, x P j 2,... x P j k, x ). V B() The result of ths local computaton s generalzed to produce analytcal expressons of the optmal utlty and polcy over P. The strength of ths algorthm s that once (x j1, x j2,...x jk ) has been substtuted by (x P j 1, x P j 2,... x P j k ), then all functons n B() are pecewse lnear functons of the sngle varable x, that s V (x ) = α + x. Therefore, summng them and maxmzng them s extremely easy. 1 We borrow the term wtness from the POMDP lterature. Lttman s wtness algorthm uses wtness ponts to testfy for the presence of α-vectors n the optmal soluton of a POMDP (Kaelblng, Lttman, & Cassandra 1998). (6)

5 Algorthm 2 A wtness algorthm 1: whle V s not totally defned do 2: Pck a state x = `x j 1, x j 2,... x j k R[X] where V s not defned yet. 3: Determne the characterstcs of the pece of V contanng x (utlty u and polcy µ). 4: Buld a set of constrants ensurng that these characterstcs reman. Ths constrant set represent the faces of the pece of V contanng x. 5: Smplfy the constrant set bult at prevous step. 6: Add the new pece to V. 7: end whle Ths contrasts strongly wth Alg. 1 that endures a very hgh cost n manpulatng multdmensonal functons. Selectng an unexplored state x R[X ] (lne 2 of Alg. 2): Ths step s a bottleneck where we face the multdmensonal space avoded otherwse. It can be performed systematcally by a costly and complex recursve procedure. Random and pseudo-random technques mght be more effcent despte havng no guarantee of completeness. Our prototype mplementaton employs a smple but neffcent technque that checks only the ponts on a dscrete grd coverng the set of possble values for the varables n X. In the followng we call PV the pece of V contanng x. x s the wtness of PV. The next steps am at buldng P V. Determnng the characterstcs (µ and u) of PV (lne 3 of Alg. 2)): Our algorthm performs these operatons n a partcular way that mnmzes computaton tme. Gven x = ( ) x j 1, x j 2,... x j k, the frst operaton performed s to substtute (x j1, x j2,...x jk ) for x n every functon V B(). The resultng functon called V [x ] s a pecewse lnear functon of the sngle varable x : V [x ](x ) = α + x. We later call ths operaton nstantatng V as x. Next all V [x ] are summed to produce the functon Q[x ] that s the nstantaton of Q = V B() V n x. In Fg. 2, Q[x ] represents the varaton of Q along the axs that passes through x. Agan, these operatons are straghtforward and the result s a lnear functon of x only. Fnally, we determne the maxmum of Q[x ] w.r.t. x. Ths s a straghtforward optmzaton of a 1-dmensonal pecewse lnear functon. We then have solved the BE fundamental equaton n the partcular pont x wth mnmum effort. We obtan an analytcal expresson of the optmal utlty u and the optmal polcy µ n PV by generalzng ths local computaton as explaned below. Because the V [x ] s and Q[x ] are undmensonal functons, ther faces are sngular ponts between two adjacent peces. One can show that each sngular pont of V [x ] s and Q[x ] s nherted from a face of some functon V B(). Frst, V [x ] s obtaned by traversng the pecewse lnear value functon V wth the axs defned by (x j1, x j2,... x jk ) = x, as n Fg. 2. The sngular ponts of V [x ] are the ponts where ntersects a face of V. Therefore, each sngular pont s nherted from a constrant C of V. Second, Q[x ] s obtaned by ntersectng and summng the undmensonal pecewse lnear functons V [x ]. Therefore, each of ts sngular ponts s nherted from a sngular pont of some functon V [x ], and so t s nherted from a constrant C of V. To determne the analytcal soluton, we must keep track of the relatonshps between sngular ponts of Q[x ] and constrants of V s; fortunately, mantanng ths nformaton s relatvely nexpensve. Snce Q[x ] s pecewse lnear, ts optmum s attaned n one of ts sngular ponts. The constrant at the orgn of ths sngular pont determnes the optmal polcy over PV. If the constrant s C : α C + k αc j l x jl + α C x then the optmal polcy of VP s gven by Eqn. 2. Ths s a well-defned lnear polcy wth scope n X. Now that we know the polcy µ attached to VP, we want to determne an analytcal expresson of the utlty u attached to ths pece of V. We cannot carry on the same computaton as n the proof of Lemma 2 because we do not have an explct representaton of Q. However, agan, keepng track of smple nformaton acheves the same result. Indeed we only need to memorze the pece of each V that contrbutes to each pece of Q[x ], whch s straghtforward and very cheap. Then, f we denote P as the pece of Q[x ] contanng the optmal value, we can trace back the pece P V of each V B() that contrbutes to P. By summng the utlty functons carred by these peces, we get an analytcal expresson of the utlty of the pece of Q contanng P. As n the proof of Lemma 2, we know the utlty and the optmal polcy assocated wth the optmal pece of Q. Therefore, we can get an analytcal expresson of the utlty attached to VP by substtutng x by µ n u followng Eqn. 3. The precedng dscusson shows that there s no need to employ a complex multdmensonal representaton of Q to determne the optmal polcy and utlty assocated wth VP. In ths calculaton, the most costly step s the nstantaton of each functon V n x. We show below how ths operaton can be performed effcently. Instantatng V B() n x = ( ) x j 1, x j 2,... x j k (used at lne 3 of Alg. 2): The procedure used for nstantatng a functon V n x loops through all peces of V and, for each pece P = (C, u), checks whether or not the -axs passng through x traverses P. Ths check s performed by substtutng (x j1, x j2,... x jk ) by x n every constrant C C. It transforms a constrant C : α + k l x l nto undmensonal constrant C[x ] that s ether or x α x α α jl x j l f >, α jl x j l f <.

6 C 5 C 1 C 2 = C + K(C 5 ) P K(C 4 ) K(C 2 ) C 4 = C K(C 3 ) K(C 1 ) Fgure 3: Instantatng V : (the axs defned by (x j1, x j2,... x jk ) = x ) enters P through face C = C 4 and leaves through face C + = C 2. For each constrant C, K(C) s the coordnate (on ) of the pont where traverses face C. We call C + the set of constrants of the frst type and C the set of constrants of the second type. Then we defne K(C) = α α jl x j l as the coordnate (on ) of the pont where traverses face C (see Fg. 3), and C + = argmn C C +[K(C)] and C = argmax C C [K(C)]. Then traverses P ff K(C ) < K(C + ). If traverses P then the segment of contaned nto P defnes a pece [K(C ), K(C + )] of V [x ]. The constrant assocated wth sngular pont K(C ) s C and the constrant assocated wth K(C + ) n C +. The detal of ths computaton s presented n Alg. 3. Computng the faces of the new pece (lne 4 of Alg. 2): In the proof of Lemma 2, the faces of a new pece are computed by addng to the faces of P (see Fg. 2) the constrants defned by Eqn. 4. We follow the same general prncple to derve the set of faces of P V. Frst, because we do not have an explct representaton of Q, we do not have an easy access to the faces of P. Fortunately, an equvalent constrant set can be recovered usng a smple set of ponters. The constrants of a pece P of the projecton of Q on X express the fact that the axs passng through x = (x j1, x j2,... x jk ) traverses the same sequence of peces of Q for all x n P (cf. Fg. 2). A necessary condton for ths holdng s that traverses the same sequence of peces of each V B(). Gven a functon V B(), we can buld a set of constrants mplyng that traverses a fxed sequence of peces of V n two steps: For each pece P of V that s traversed by, add a constrant mplyng that P s stll traversed by ; For each pece P of V that s not traversed by, add a constrant mplyng that P s stll not traversed by. Ths procedure s lnear n the number of peces n V. Snce t s performed once for each pece P, the algorthm s quadratc n the maxmum number of peces n a value functon. Therefore, we adopt the cheaper alternatve that bulds a sngle set of constrants: C 3 Algorthm 3 Instantatng V n x = ( x j 1, x j 2,...x j k ). 1: for each pece P = (C, u) of V do 2: C, C +. 3: for each face C = (α, α j1, α j2,... α jk, ) of C do 4: f x s n the scope of C ( ) then 5: f > then 6: C + C + {C}. 7: else 8: C C {C}. 9: end f 1: Compute K(C) = α P k α jk x j k. 11: else 12: // x s not n the scope of C ( = ) 13: Substtute (x j1, x j2,... x jk ) for x n C and check for consstency: α + P k αj l x j l? 14: f Consstent then 15: Skp to next face of C (go to lne 3). 16: else 17: Skp to next pece of V (go to lne 1). 18: end f 19: end f 2: end for 21: Compute C = arg max C C [K(C)] and C + = arg mn C C +[K(C)]. 22: f K(C + ) <= K(C ) then 23: // Inconsstency detected 24: Skp to next pece of V (go to lne 1). 25: else 26: Substtute (x j1, x j2,... x jk ) for x n u to get a functon u[x ](x ). 27: Add the pece `[K(C ), K(C + )], u[x ] to V [x ]. Attach to sngular pont K(C ) the constrant C and attach C + to K(C + ). 28: end f 29: // Contnue to next pece of V 3: end for 31: Return V [x ] For each pece P of V that s traversed by, add a constrant mplyng that stll enters and exts P through the same faces. Ths creates an equvalent set of constrants, but the complexty s lnear n the number of peces traversed by whch s generally much smaller than the total number of peces. Now, gven a pece P of V, we ensure enters and exts through the same face of P usng two types of constrant: C C, C C : K(C) K(C ), and C C +, C C + : K(C + ) K(C). We add a followng constrant to ensure traverses P : K(C ) K(C + ). Formally, f C : α + k α l x l, and C + : α + + k α+ l x l, and C = α + k l x l, then these constrants are computed as (respectvely): ( ) α α α α j + l α α j l x jl, (7)

7 x 1 1 proof of Lemma 2, ths pece s splt n as many sub-peces as there are peces P of Q that satsfy Eqn. 4 for some x. Here, we are not nterested n all these peces but only the one contanng the current wtness x. Thus, we complete the set of faces of PV by addng the constrants defned by Eqn. 4 for the partcular pece P that carres the optmum of Q[x ] as computed at step 1. If u P (x) = α + k j=1 α j l x jl and u P (x) = α + k j=1 α j l x jl, Eqn. 4 becomes α α + ( ) α jl α jl xjl. (11) x Equatons 7 to 11 generate the set of all faces of P V. Fgure 4: The partton of R 2 defned by the peces of Q. In ths example, there are two value functons n B : reward functon R 1(x 1) at the orgn of the horzontal faces n the fgure, and a functon R 1(x 1 x ) that represents pecewse lnear preferences on the dfference between x and x 1 and that nduces the dagonal constrants. Ths s a standard stuaton encountered n many STPPs. The axs and 1 traverse the same sequence of peces of both R 1 and R 1. However, they traverse dfferent peces of Q. ( ) α α+ α jl α + + α+ j l α + x jl, (8) ( ) α + α + α α + j α + l α + α j l α x jl. (9) Equatons 7 to 9 allow for the buldng of a constrant set that guarantees that traverses a fxed sequence of peces of each V B(). Ths condton s necessary to ensure that we stay nsde of a sngle pece P, but t s not suffcent. A counter-example s provded n Fg. 4. Fortunately, ths ssue can be addressed by addng a small number of smple constrants. In the example of Fg. 4, and 1 traverse the same sequence of peces of each V B(), but they traverse dfferent sequences of peces for all functons taken together. Both axes start n the frst pece of R 1 and the frst pece of R 1 ; enters the second pece of R 1 before enterng the second pece of R 1, whle 1 does the opposte. Therefore, we get a complete set of constrants to characterze the faces of P n the followng way: For any two adjacent peces [a, b] and [b, c] of Q[x ], such that C a s the constrant assocated to sngular pont a durng the constructon of Q[x ] and C b s the constrant assocated wth b, we add the constrant α b K(C a ) < K(C b ). If C a : α a + k αa l x l and C b : α b + k αb l x l, then t translates as ( ) α b αa α b jl α a + αa j l α a x jl. (1) α b The constrants defned by Eqn. 7 to 1 are the faces defnng a unque pece P of the projecton of Q on X. In the Smplfyng Constrant Sets (lne 5 of Alg. 2): The constrant set bult by the procedure descrbed above may contan redundant constrants, or constrants that are useless because they are domnated by other constrants. For nstance, f C : x s n the constrant set, then any constrant C : x K for some K > can be omtted n the representaton of the pece. Indeed, our experments show that the wtness algorthm produces many such constrants. The algorthm complexty s senstve to the number of constrants n the descrpton of each pece. Although the maxmum number of constrants n a pece s guaranteed to be polynomal n the problem sze, t s preferable to keep ths number as low as possble. Therefore, n lne 5 of Alg. 2 we try to purge the set from useless constrants. Smplfyng a set C of lnear constrants wth scope X s a dffcult problem. At ths pont we are consderng several approaches to address t: A smlar problem arses n the optmzaton of POMDPs where we must determne a mnmal set of lnear faces to represent a pecewse lnear convex functon. Here the problem s addressed by solvng multple LPs. Ths approach seems relatvely easly transferable to our sub-problem. Solvng the dual program of an LP M allows determnng the constrants that are bndng n the optmal soluton of M (Schrjver 1986). Defnng M = (X, V, ) where s the objectve functon that returns n all states, then any pont nsde of the polygon defned by C s an optmal soluton of M. Solvng the dual problem of M can thus compute a mnmum set of faces. Our current mplementaton of the wtness algorthm just prunes the set C from unary constrants that are domnated by other unary constrants. Useless k-ary constrants, k > 1, are kept n the descrpton of the pece, whch can hurt the algorthm. In general, there s a trade-off between the tme spent prunng the set of constrants, and the mpact of useless constrants on complexty. Note that the problem of smplfyng C s exponental n the number of varables n ts scope X. However, n our case, the scope of the functons s bounded by the treewdth of the problem. Therefore, wth a bounded treewdth, complexty s polynomal n the sze of the problem.

8 R [, 2] [, 2] [, 2] [1, 8] t 2 [2, 7] t t 1 Fgure 5: A smple STPP wth non-convex preferences. Interval [l, u ] attached to vertex t represents the hard constrant l t u. Interval [l j, u j] on edge t t j represents the hard constrant l j t j t u j. There s a total of four value functons n B(2): one for the hard constrant on t 2, two for the hard constrants on t 2 t and t 2 t 1, and the reward R2(t 2) represented at the top. t 2 V t 1 Fgure 6: The 3D value-functon V 2 obtaned by elmnatng t 2 n the problem of Fg. 5. Ponts not assgned a value n ths graph are those that break a hard constrant (hence they have value). t Summary and Future Work Ths paper provdes proof of concept that bucket elmnaton can be used to optmze a rch model of temporal preferences. The potental of ths concept for schedulng wth preferences s twofold. Frst, a rcher preference model opens up generaton of schedules that better reflect the desres of the modeler; second, the alternate poston of BE n the space tme trade-off opens up a complementary approach compared to pror art. After showng the relevance and soundness of the approach, we provded a grounded algorthm. A prototype mplementaton s capable of solvng a rcher class of problems than prevous approaches. It has been run successfully on problems nvolvng non convex and k-ary constrants. Fg. 6 shows the result of one teraton of the wtness algorthm on the STPP of Fg. 5. At ths pont, our mplementaton of the wtness algorthm suffers from two weaknesses: () smplstc grd-base procedure used to fnd a new wtness x where V s not yet defned, () procedure used to smplfy constrant sets that cannot handle more than unary constrants. Our ongong work s to mprove the effcency of the prototype mplementaton, and to compare the BE approach both theoretcally and emprcally to alternate approaches, such as the use of repeated Lnear Programmng wth a branch-and-bound search. In partcular, because bnary and convex STPPs s a subclass of problems that can be represented as lnear constrant optmzaton problems, t would be nterestng to compare our algorthm wth the results n (Morrs et al. 24). BE and LP approaches can both solve ths subclass of problems, but ther complexty bears on dfferent parameters. Our future work wll am at showng expermentally that BE can be a compettve approach for large problems wth small treewdth. Acknowledgments. The work of the author from SRI Internatonal was supported by the Defense Advanced Research Projects Agency (DARPA) under Contract No. FA875-7-D-185/4. Any opnons, fndngs and conclusons or recommendatons expressed n ths materal are those of the author(s) and do not necessarly reflect the vews of DARPA, or the Ar Force Research Laboratory (AFRL). References Bresna, J. L.; Khatb, L.; and McGann, C. 26. Msson operatons plannng wth preferences: An emprcal study. Internatonal Workshop on Plannng and Schedulng for Space. Dechter, R.; Mer, I.; and Pearl, J Temporal constrant networks. Artfcal Intellgence 49: Dechter, R Bucket elmnaton: A unfyng framework for reasonng. Artfcal Intellgence 113(1-2): Dechter, R. 23. Constrant Processng. San Francsco, CA: Morgan Kaufmann. Dow, P., and Korf, R. 27. Best-frst search for treewdth. In Proceedngs of the Twenty Second Natonal Conference on Artfcal Intellgence, Menlo Park, CA: AAAI Press. Kaelblng, L.; Lttman, M.; and Cassandra, A Plannng and actng n partally observable stochastc domans. Artfcal Intellgence 11: Mofftt, M. D.; Pentner, B.; and Yorke-Smth, N. 26. Mult-crtera optmzaton of temporal preferences. In Proc. of CP 6 Workshop on Preferences and Soft Constrants (Soft 6), Morrs, P.; Morrs, R.; Khatb, L.; Ramakrshnan, S.; and Bachmann, A. 24. Strateges for global optmzaton of temporal preferences. In Proceedngs of the Tenth Internatonal Conference on Prncples and Practce of Constrant Programmng, Berln, Germany: Sprnger-Verlag. Schrjver, A Theory of Lnear and Integer Programmng. New York, NY: John Wley and Sons.