Infinite Time Optimal Control of Hybrid Systems with a Linear Performance Index

Transcription

1 Infinite Time Optimal Control of Hybrid Systems with a Linear Performance Index Mato Baotić, Frank J. Christophersen, and Manfred Morari Automatic Control Laboratory, ETH Zentrum, ETL K 1, CH 9 Zürich, Switzerland baotic fjc Abstract We consider the constrained infinite time optimal control problem for the class of discrete time linear hybrid systems. When a linear performance index is used the infinite time optimal solution is a piecewise affine (PWA) state feedback control law. In this paper we present an algorithm that computes the optimal solution in a computationally efficient manner. It combines a dynamic programming exploration strategy with a multi-parametric linear programming solver and basic polyhedral manipulation. Index Terms constrained systems, infinite time, optimal control, discrete time, linear hybrid systems, dynamic programming, multi-parametric linear program. I. INTRODUCTION In the last few years several different techniques have been developed for the analysis and controller synthesis for hybrid systems [1], [], [], [], [5]. A significant amount of the research in this field has focused on solving constrained optimal control problems, both for continuous time and discrete time hybrid systems. We consider the class of discrete time linear hybrid systems. In particular the class of constrained piecewise affine (PWA) systems that are obtained by partitioning the state space into polyhedral regions and associating with each region a different affine state update equation, cf. [], []. For such a class of systems the constrained finite time optimal control (CFTOC) problem can be solved by means of multiparametric programming [5]. The solution is a piecewise affine state feedback control law and can be computed by using multi-parametric mixed-integer quadratic programming (mp-miqp) for a quadratic performance index and multiparametric mixed-integer linear programming (mp-milp) for a linear performance index, cf. [5], [7], []. However, as recently shown by Borrelli et al. [9] for a quadratic performance index and by Baotić et al. [1] for a linear performance index, it is possible to obtain the optimal solution to the CFTOC problem without the use of integer programming. In [9], [1] the authors propose algorithms based on a dynamic programming strategy combined with multi-parametric quadratic or linear program (mp-qp or mp- LP) solvers. The main advantage of the infinite time solution is the guaranteed stability of the closed-loop system. This is not the case for the finite time solution where for linear systems the stability of receding horizon control has to be (artificially) enforced by terminal end constraints, terminal weights, or invariant terminal sets, cf. [11], and for the class of PWA systems very few and restrictive stability criteria are known [1]. In this paper we present a novel, computationally efficient algorithm to solve the constrained infinite time optimal control (CITOC) problem with linear performance index for PWA systems. The algorithm combines a dynamic programming exploration strategy with a multi-parametric linear programming solver and basic polyhedral manipulation. The developed algorithm converges to the solution of the Hamilton-Jacobi-Bellman equation and thus avoids potential pitfalls of other conservative approaches. However, the algorithm cannot obtain optimal solutions that have an unbounded cost. This is hardly a limitation to the applicability of the method, since in most practical applications we want to steer the state to some equilibrium point by spending a finite amount of energy. This paper is organized as follows: in Section II we give a brief overview of known results to the CFTOC problem for PWA systems. In the following Section III we discuss the solution to the CITOC problem for the class of PWA systems. A novel algorithm to obtain the solution to the CITOC problem with a linear performance index is given in Section IV. In Section V an example for the computation of the infinite time solution is presented. II. CONSTRAINED FINITE TIME OPTIMAL CONTROL Consider the class of linear discrete time hybrid systems that can be described as constrained piecewise affine systems of the following form x(t + 1) = f PWA (x(t), u(t)) = A i x(t) + B i u(t) + f i, if [ ] x(t) u(t) D i (1) where D i := {[ u x ] H i x + J i u K i }, t, x R n is the state, u R m is the control input, and {D i } d i=1 is the polyhedral partition of the sets of the extended state+input space R n+m. Note that linear state and input constraints of the general form Kx(t) + Lu(t) M are incorporated in the description of D i. Throughout this paper we will consider costs that penalize the deviation of the state and control action from the origin in the extended state+input space. However, all presented results also hold for any non-zero equilibrium point since such problems are easily translated to the steer to the origin problem by a simple substitution of the variables. Hence we define the following cost function T 1 J(U T, x()) := P x(t ) p + Qx(k) p + Ru(k) p k= ()

2 and consider the constrained finite time optimal control (CFTOC) problem J (x()) := min J(U T, x()), () U T { x(t + 1) = fpwa (x(t), u(t)), subj. to x(t ) X f () where the column vector U T := [u(),..., u(t 1) ] R mt is the optimization vector, T is the time horizon, X f is the terminal target region and Qx p with p {1, } in () denotes the corresponding standard vector 1-norm or -norm. Additionally, we assume that R, Q, and P are of full column rank. We summarize the main result of the solution to the CFTOC problem (1) () which is proved in [1], [5]. Theorem II.1 (Solution to the CFTOC). The solution to the optimal control problem (1) () with p {1, } is a piecewise affine state feedback control law of the form u (k) = F k i x(k) + G k i, if x(k) P k i (5) where P k i, i = 1,..., N k, is a polyhedral partition of the set X k of feasible states x(k) at time k with k =,..., T 1. In the case that the receding horizon policy is used the optimal control is given as a constant state feedback law of the form u RH(t) = F i x(t) + G i, if x(t) P i () for t and thus only N (possibly different) control laws have to be stored. As shown by the authors in [1] the CFTOC problem (1) () can be solved in an efficient way by solving an equivalent dynamic program (DP) backwards in time. The DP has the following form J k (x(k)) := min u(k) Qx(k) p + Ru(k) p + J k+1(f PWA (x(k), u(k)), (7) subj. to f PWA (x(k), u(k)) X k+1 () for k = T 1,...,, with boundary conditions where X T = X f, and J T (x(t )) = P x(t ) p (9) X k := { x R n u, f PWA (x, u) X k+1} (1) = N k i=1p k i is the set of all initial states for which the problem (7) () is feasible. With p {1, } the dynamic programming problem (7) () can be reformulated as a multi-parametric linear program (mp-lp), cf. [5], where the state, x(k), is treated as a parameter and the control input, u(k), as an optimization variable. By solving such a program at each iteration step k we obtain the PWA optimal control law (5) and the PWA function J k (x(k)) = Γ k i x(k) + k i, if x(k) P k i (11) that represents the so called cost-to-go. III. CONSTRAINED INFINITE TIME OPTIMAL CONTROL By letting T the cost function () takes the following form J (U, x()) := Qx(t) p + Ru(t) p (1) t= and () () becomes the constrained infinite time optimal control (CITOC) problem J (x()) := min U J (U, x()), (1) subj. to x(t + 1) = f PWA (x(t), u(t)) (1) where by the column vector U := [u(), u(1),...] we denote the optimization vector and by U the optimizer of (1). As in the CFTOC problem, Qx p with p {1, } in (1) denotes the corresponding standard vector 1-norm or -norm, and we assume that R and Q are of full column rank. Assumption III.1. The CITOC problem (1) (1) is well defined, i.e. the minimum is achieved and J (x()) < for any feasible state x(). It seems natural to try to solve the CITOC problem (1) (1) in a similar manner as the CFTOC problem is solved in Section II, i.e. by using a dynamic program J k (x(t)) := min u(t) Qx(t) p + Ru(t) p + J k 1(f PWA (x(t), u(t)), (15) subj. to f PWA (x(t), u(t)) X k 1 (1) for k = 1,...,, with initial conditions X = { x R n u, (x, u) d i=1d i }, and (17) J (x(t)) = (1) where d i=1 D i is the domain of f PWA (x, u) and X k := { x R n u, f PWA (x, u) X k 1} (19) = N k i=1p k i is the set of all initial states for which the problem (15) (1) is feasible. Beside Assumption III.1 some additional considerations have to be taken into account in order to establish the equivalence between the solution to the CITOC problem and the solution to the DP problem. For further details see [1]. We assume that the conditions in [1] are met, i.e. that the following assumption holds. Assumption III.. If the DP iterations (15) (1) converge, then the obtained solution is also solving the CITOC problem (1) (1). Now we are ready to state the theorem that characterizes the solution to the CITOC problem (1) (1). Theorem III. (Solution to the CITOC). Under Assumption III.1 and Assumption III. the solution to the optimal

3 1 u 1 Input u State x 1 rag replacements State x FIG. 1: FEASIBLE STATE+INPUT SPACE AND OPTIMAL INFINITE TIME CONTROL LAW FOR EXAMPLE () Time t FIG. : CLOSED-LOOP SIMULATION FOR EXAMPLE () FOR DIFFER- ENT INITIAL VALUES. control problem (1), (1) (1) with p {1, } is a piecewise affine state feedback control law of the form u (t) = F i x(t) + G i, if x(t) P i () where Pi, i = 1,..., N, is a polyhedral partition of the set X of feasible states x(t) at time t with t. Proof. Theorem III. follows by the construction of the DP iterations (15) (1). For more details see Algorithm IV.1 and Algorithm IV. presented in the following section. Note that the optimal solution () has the same form as the receding horizon solution (). This means that with the control law u (t) the closed-loop response and open-loop response of the system (1) are identical. The value function J (x(t)) has the PWA form J (x(t)) = Γ i x(t) + i, if x(t) P i (1) and can be considered as the cost-to-go to the origin. As observed by the authors in [1], it may happen that the dynamic program (15) (1) converges after a finite number of steps k = k + 1 <. Here we mean by convergence that in two successive iterations of the dynamic program the value functions as well as their domains do not change, i.e. Jk ( ) J k 1 ( ). Thus we stop the dynamic program when the following condition is met j 1 {1,..., N k } P j1 j {1,..., N k 1 } such that P j, Γ k j 1 Γ k 1 j < ε, k j 1 k 1 j < ε () where ε > is some small tolerance. This would seem to imply that the optimal control law steers any feasible state x() after at most k time steps to the origin. However, several observations should be made. First we should point out that the 1-/ -norm CITOC problem may lead to two types of solutions: (a) an optimal control sequence that in a finite number of time steps steers the state to the origin, and (b) an optimal control sequence that takes an infinite number of time steps to steer the state to the origin. This type of behavior may be observed even for linear systems as shown in Example III. below. Note that due to the numerical accuracy when checking the Condition () we may stop the DP iterations after some k = k < k. Although, strictly speaking, we are making an error, in practice this error is unavoidable and is of minor importance and can therefore be neglected. On the other hand we gain an extra benefit from such an error since earlier convergence of the DP reduces the time needed for the computation of the solution and in addition we may get a solution even when some optimal trajectories actually take an infinite number of time steps to converge to the origin. Example III. (CITOC of a linear system). Consider the one-dimensional linear system x(t + 1) =.x(t) + u(t) if.x(t) u(t), x(t) [ 5, 5], u(t) [ 1, 1]. () The constrained infinite time optimal control problem (1) (1) is solved with Q = 1, R = 1, and X = [ 5, 5] for p =. If one considers x(t) [, 5] it is obvious that the infinite time optimal control policy is in fact u (t) =.x(t) and therefore the optimal value function or cost-to-go can be computed with x(t) =.5 t x() as J (x()) = min x(k) + u(k) u(t) k= = 1..5 k x() =.x(). k= From this we see that for any state starting in [, 5] it will take an infinite amount of steps until the state reaches the origin; whereas if the state x(t) [ 1, ] the infinite time optimal control law is given by u (t) =.x(t) and thus drives the state in one single step to the origin. With similar considerations for the rest of the feasible state

4 space one finds for the infinite time optimal control law.x(t) if x(t) [, 5], u (t) =.x(t) if x(t) [ 1, ], 1 if x(t) [ ] 5, 1 and for the cost-to-go.x() if x() [, 5], J (x()) = 1.x() if x() [ 1, ], 1.9x(). if x() [ ] 5, 1. The feasible state space and the optimal infinite time control law is depicted in the extended state+input space in Figure 1. The optimal infinite time closed-loop trajectory for three different initial values is depicted in Figure. IV. ALGORITHM FOR THE INFINITE TIME OPTIMAL SOLUTION In this section we first describe how to improve a rather general DP problem (15) (1) for the computation of the infinite time solution and then we present the implementation of the algorithm. In the rest of the paper for some variable or function τ we denote as τ its restriction to the neighborhood of the origin (x, u) = (, ). For instance, D describes the domain of those PWA dynamics that are valid for the origin { D i } d i=1 := {D j [ ] D j} d j=1, while f PWA represents the restriction of the function f PWA to that domain. Furthermore, when we say SOLVE iteration k of a DP, we mean formulate a multiparametric linear program (mp-lp) for it and obtain a triplet of expressions for the value function, the optimizer, and the polyhedral partition of the feasible space ( J k (x), u k(x), P k i, i = 1,..., N k). () By inspection of the DP problem (15) (1) we see that at each iteration step we are solving d N k 1 mp-lps. After that, by using polyhedral manipulation we have to compare all generated regions, check if they intersect and remove the redundant ones, before storing a new partition that has N k regions. For a system with only one equilibrium point, i.e. the origin (, ) itself, all trajectories that converge to that point in an infinite number of time steps have to go through some of the PWA dynamics associated with the region D i, i = 1,..., d, and regions P j, j = 1,..., N, that are touching the origin, cf. Example III.. Thus at the beginning, instead of focusing on the whole feasible state space and all dynamics, we can limit our algorithm to the neighborhood of the origin and only after attaining convergence, we proceed with the exploration of the rest of the state space. In this way at each iteration of the DP we would on average have to solve a much smaller number of mp-lps. We will call the solution to such a restricted problem the core C. Note that in general the core C is a non-convex polyhedral partition. Any positive invariant set is a valid candidate for the core C, as long as an associated control strategy is feasible and steers the state to the origin. The only prerequisite is that for any given initial feasible state (i.e. the state for which the original problem has a solution) we can reach at least one element of the core in a finite number of time steps. However, as its name says, the core is used as a seed for the future construction and exploration of the feasible state space. Thus, obtaining a good suboptimal solution is desirable. The task of solving the CITOC problem (1) (1) is split into two subproblems and respective algorithms. In the first algorithm we explore the portion of the state space around the origin and construct the core of the infinite time solution. In the second algorithm, starting from the core C, we build a sequence of additions called rings R k to the core C k 1 until the algorithm converges for the whole feasible space. At the end we have the infinite time solution S. Here with C, R, and S we denote the triplets of the form given in Equation (). In an ideal scenario the core C would already be a part of an infinite time optimal solution, and every ring R k would also be a part of an infinite time solution. Then in all intermediate steps we would have to explore only the one step ahead optimal transitions from all PWA dynamics to the latest ring (instead of going from all dynamics to the initial core and all previous rings). In practice we are likely to observe suboptimal scenarios: the newly generated ring, R k, may contain polyhedra with associated cost functions that are worse than the infinite time solution and thus such polyhedra will be altered in the future steps of the algorithm. Algorithm IV.1 (Generating the CORE of the Infinite Time Solution). INPUT: k max, ε, f PWA(x, u), { D i, i = 1,..., d} OUTPUT: The core C LET S ) ( J (x) =, ů (x) =, { P 1 = R n } LET k =, finished = FALSE WHILE k < k max AND NOT finished LET k k + 1 FOR i = 1 TO d FOR EACH S k 1 P k 1 j s i,j SOLVE min Qx p + Ru p u + J k 1( f PWA(x, u)), (5) subj. to (x, u) D i, () k 1 f PWA(x, u) P j (7) LET S k INTERSECT & COMPARE {s i,j} LET S k RESTRICTION of S k to the origin IF S k S k 1 (in the sense of Condition ()) THEN finished = TRUE, C = S k In the step where INTERSECT & COMPARE is done, we are removing redundant polyhedra, i.e. we remove such polyhedra that are completely covered with other polyhedra which have a better (meaning smaller) value function expression. If some polyhedron is only partially covered with better regions the part with the smaller cost can be partitioned into

5 a set of convex polyhedra. Thus we preserve the polyhedral nature of the feasible space partition in each iteration of the Algorithm IV.1. Note that in the SOLVE step we are solving a smaller number of problems than in the general DP. Since we are restricting ourselves to the neighborhood of the origin, the number of regions at each step is likely to remain rather small and should stay constant after a certain number of iterations. However, the choice of the initial J = may lead to a big number of iterations depending on the desired precision. If a better initial guess is known, it can be used to speed up Algorithm IV.1. Note that if Algorithm IV.1 ends successfully then the cost associated to C represents the best current upper bound of J (x()), x() C if for all time steps the trajectory remains inside the core. If it were not so, then by Assumption III. the Algorithm IV.1 could improve the cost in some of the regions P i C by going through additional iterations. After we have constructed the initial core, C, we can proceed with the exploration of the rest of the state space as described in the following Algorithm IV.. Algorithm IV. (Generating the Infinite Time Solution). INPUT: k max, ε, f PWA(x, u), {D i, i = 1,..., d}, the core C OUTPUT: The infinite time solution S LET Solution S C LET Ring R C LET k =, finished = FALSE WHILE k < k max AND NOT finished LET k k + 1 FOR i = 1 TO d FOR EACH P k 1 j R k 1 s i,j SOLVE min Qx p + Ru p u + Jk 1(f PWA(x, u)), () subj. to (x, u) D i, (9) f PWA(x, u) P k 1 j () LET S k INTERSECT & COMPARE S k 1, {s i,j}, LET C k S k S k 1 (in the sense of Condition ()) LET R k S k \C k IF R k = THEN finished = TRUE, S = S k V. EXAMPLE Consider the piecewise affine system [1] [ ] x(t + 1) =. cos α(t) sin α(t) x(t) + [ sin α(t) cos α(t) 1 ] u(t), { π α(t) = if [1 ]x(t), π if [1 ]x(t) <, x(t) [ 1, 1] [ 1, 1], u(t) [ 1, 1]. (1) J (x) x FIG. : STATE SPACE PARTITION OF THE VALUE FUNCTION J (x) OF THE INFINITE TIME SOLUTION S. SAME COLORING IMPLIES THE SAME COST VALUE. The constrained infinite time optimal control problem (1) (1) is solved with Q = [ 1 1 ], R = 1, and X = [ 1, 1] [ 1 1] for p =. As described in Section IV the algorithm is divided into two parts: first the so called inner core C is constructed via a dynamic programming approach. After the convergence of this inner core it serves as a seed (or optimal upper bound of the value function in this particular step) for the second part of the algorithm where from the seed the rest of the feasible state space is explored until the piecewise affine value function for the whole feasible state space does not change for two successive steps in the exploration procedure. For Example (1) the inner core, C, is computed in 11. seconds in 5 iteration steps. Figure (a) shows the state space partitioning comprising 1 polyhedral regions of the value function of the inner core. Figure (b) shows the state space partition of the value function with 1 polyhedral regions at the intermediate step k = of the second part of the algorithm. Figure (c) shows the state space partition C of the current optimal upper bound of the infinite horizon value function which does not change from the intermediate step k = to k = and it consists of 1 regions. This can be viewed as the new core in step k = from which the ring R in Figure (d) was computed in step k =. After k = 7 steps of the second part of the algorithm the whole feasible state space is explored and the value function Jk (x) does not change from step k = 7 to k = (Figure (e) and Figure ) and thus the infinite horizon solution J (x) is obtained. The constructed state space partition consists of 5 polyhedral regions. The PWA control law of the infinite horizon solution consists of different piecewise affine control laws and is depicted in Figure (f). The same coloring implies the same affine control law. The infinite horizon solution to (1) (1) for this example was solved in 1 seconds on a Pentium,. GHz machine running MATLAB R.1. This shows the efficiency of the proposed algorithm compared to the approach given in [1] where the computation of the infinite horizon solution took 5 5 1

6 g replacements x x x u = u = u = 1 (a) Value function of the inner core C (b) Value function at the intermediate construction step k = (c) Core of the value function at the intermediate construction step k = g replacements x x x u = u = u = (d) Ring of the value function at the intermediate construction step k = (e) Value function of the infinite horizon solution (f) PWA control law of the infinite horizon solution. FIG. : STATE SPACE PARTITION OF THE VALUE FUNCTION AND OF THE PIECEWISE AFFINE CONTROL LAW OF EXAMPLE (1). SAME COLORING IMPLIES THE SAME COST VALUE OR TO THE SAME AFFINE CONTROL LAW, RESPECTIVELY seconds. ACKNOWLEDGMENT We wish to thank Francesco Borrelli and Pascal Grieder for their helpful discussions. VI. REFERENCES [1] M.S. Branicky and G. Zhang. Solving hybrid control problems: Level sets and behavioral programming. In Proc. American Contr. Conf., Chicago, Illinois USA, June. [] E.D. Sontag. Nonlinear regulation: The piecewise linear approach. IEEE Trans. Automatic Control, (): 5, April 191. [] A. Bemporad, F. Borrelli, and M. Morari. Optimal controllers for hybrid systems: Stability and piecewise linear explicit form. In Proc. 9th IEEE Conf. on Decision and Control, Sydney, Australia, December. [] J. Lygeros, C. Tomlin, and S. Sastry. Controllers for reachability specifications for hybrid systems. Automatica, 5():9 7, [5] F. Borrelli. Discrete Time Constrained Optimal Control. Dr. sc. techn. thesis, Swiss Federal Institute of Technology (ETH), Zürich, Switzerland, May. [] W.P.M.H. Heemels, B. De Schutter, and A. Bemporad. Equivalence of hybrid dynamical models. Automatica, 7(7):15 191, 1. [7] V. Dua and E.N. Pistikopoulos. An algorithm for the solution of multiparametric mixed integer linear programming problems. Annals of Operations Research, 99:1 19,. [] E.C. Kerrigan and D.Q. Mayne. Optimal control of constrained, piecewise affine systems with bounded disturbances. In Proc. of the Conf. on Decision & Control, pages , Las Vegas, Nevada, USA, December. CDC. [9] F. Borrelli, M. Baotić, A. Bemporad, and M. Morari. An Efficient Algorithm for Computing the State Feedback Solution to Optimal Control of Discrete Time Hybrid Systems. In American Control Conference,. In press. [1] M. Baotić, F.J. Christophersen, and M. Morari. A new Algorithm for Constrained Finite Time Optimal Control of Hybrid Systems with a Linear Performance Index. Technical report, Automatic Control Laboratory, ETH Zürich, November. [11] D.Q. Mayne, J.B. Rawlings, C.V. Rao, and P.O.M. Scokaert. Constrained model predictive control: Stability and optimality. Automatica, ():79 1, June. [1] A. Bemporad and M. Morari. Control of systems integrating logic, dynamics, and contraints. Automatica, 5():7 7, March [1] D.Q. Mayne. Constrained optimal control. European Control Conference, Plenary Lecture, September 1. [1] N.L. Stokey and R.E. Lucas. Recursive Methods in Economic Dynamics. Harvard University Press, Cambridge, Massachusetts, 1 edition, 199.