New Architectures for Hierarchical Predictive Control

Transcription

1 Preprint, 11th IFAC Symposim on Dynamics and Control of Process Systems, inclding Biosystems Jne 6-8, 216. NTNU, Trondheim, Norway New Architectres for Hierarchical Predictive Control Victor M. Zavala Department of Chemical and Biological Engineering University of Wisconsin-Madison Madison, WI 5376 USA ( Abstract: We analyze the strctre of the Eler-Lagrange (EL) conditions of a long-horizon optimal control problem. The analysis reveals that the conditions can be solved by sing bloc Gass-Seidel (GS) schemes. We prove that sch schemes can be implemented in the primal space by solving seqences of short-horizon optimal control problems. This analysis also reveals that a traditional receding-horizon (RH) scheme is eqivalent to perforg a single GS sweep. We have also fond that we can se adjoint information from a coarse long-horizon problem to constrct teral penalties that correct the policies of the RH scheme. We observe that this scheme can be interpreted as a hierarchical controller in which a coarse high-level controller transfers long-horizon information to a low-level, short-horizon controller of fine resoltion. The reslts open the door to a new family of hierarchical control architectres that can handle mltiple time scales systematically. Keywords: RH, hierarchical control, time scales, Eler-Lagrange 1. BASIC NOTATION AND SETTING We start by providing basic notation and defining the technical problem. Relevant references are provided as we proceed with the discssion. We consider the following long-horizon optimal control problem: z( ),( ) T ϕ(z(τ), (τ), w(τ))dτ (1a) s.t. ż(τ) = f(z(τ), (τ), w(τ)), τ [, T ] (1b) z() = z. (1c) Here, z( ), ( ), and w( ) are state, control, and distrbance trajectories, respectively. The cost and system mappings ϕ( ) and f( ) are assmed to be smooth. We lift the long-horizon problem by partitioning the horizon T into n stages. This lifting approach was proposed by Boc and Plitt (1984) in the context of mltiple-shooting. We define the sets N := {..n 1} and N := N \{n 1}; and we assme the stages to be of eqal length h := T/n. The partitioning gives rise to the lifted problem, z ( ), ( ) N h ϕ(z (τ), (τ), w (τ))dτ (2a) s.t. ż (τ) = f(z (τ), (τ), w (τ)), N, τ [, h] (2b) z +1 () = z (h), N (2c) z () = z. (2d) We will analyze the stage strctre of the lifted optimal control problem. In doing so we will redce the notation to a imm, in sch a way that it retains the essential featres of the strctre we are interested in highlighting. We first note that we do not consider ineqality and path constraints and we eliate dependencies of the mappings on the distrbances. These changes will not alter the stage strctre of the lifted problem. We transcribe the lifted problem into a finite-dimensional nonlinear programg problem by applying an implicit Eler scheme with m inner stages of eqal length δ := h/m (other discretization schemes can also be applied). We define the sets of inner discretization points M := {..m 1}. The discretized problem is, z,j,,j N ϕ(z,j+1,,j+1 ) (3a) s.t. (ν,j+1 ) z,j+1 = z,j + δf(z,j+1,,j+1 ), N, j M (3b) (λ ) z, = z 1,m, N. (3c) Here, ν,j are the dal variables of the inner dynamic eqations (3b), and λ are the dal variables of the stagetransition eqations (4b). The dal variables are scaled by the constant 1/δ. We se the dmmy parameter z 1,m := z to simplify notation. We denote the discretized longhorizon problem (4) as P. We simplify notation frther by eliating the dynamic eqations from the notation (3b). This, again, does not alter the stage strctre. We obtain the compact problem, ϕ(z,j+1,,j+1 ) (4a) z,j,,j N s.t. (λ ) z, = z 1,m, N. (4b) A controller based on recrsive soltions of P mst captre distrbance signals that evolve over mltiple time scales (e.g., noise, weather, prices) and mst handle slow and fast components of the dynamical system (e.g., fast and slow chemical reactions, recycle systems). Despite advances in comptational methods for optimal control, this might not be possible to do. This is is becase the soltion of P might reqire very fine discretization meshes and/or Copyright 216 IFAC 43

2 IFAC DYCOPS-CAB, 216 Jne 6-8, 216. NTNU, Trondheim, Norway expensive nmerical integration procedres to captre dynamic effects at all time scales. Reviews on the topic are presented by Diehl et al. (29) and Zavala and Biegler (29). We also note that the presence of mltiple time scales plays a role in the resoltion and pdate freqency of the control. For instance, as noted by Findeisen et al. (27), if distrbances are fast it is necessary to se a compatible control resoltion. The complexity of P is traditionally addressed by sing a RH scheme which sees to approximate the optimal long-horizon policy by solving seqences of fine-resoltion short-horizon problems. In particlar, one can solve the following short-horizon problems seqentially for =,..., N 1: z,j,,j ϕ(z,j+1,,j+1 ) (5a) s.t. (λ ) z, = z 1,m. (5b) Here, the initial state z 1,m is fixed and is obtained from the soltion of the problem at 1. We will show that this RH scheme is a bloc GS iteration applied to the soltion of the Eler-Lagrange (EL) conditions of (4). This observation will help s derive hierarchical schemes to address the intractability of P. 2. STRUCTURE OF EULER-LAGRANGE CONDITIONS We grop variables by stages by defining the vectors z := (z,,..., z,m ), := (,1,...,,m ), and ν := (ν,1,..., ν,m ). We ths obtain the bloc form of P, φ(z, ) N (6a) s.t. (λ ) Π z = Π z 1, N. (6b) The strctre of the mapping φ( ) is given by: φ(z, ) := ϕ(z,j+1,,j+1 ). (7) The coefficient matrices Π and Π satisfy Π z = z, and Π z 1 = z 1,m. We also define the fixed dmmy vector z 1 satisfying Π z 1 = z 1,m = z. The Lagrange fnction of P is given by L(z,, λ ) := N φ(z, ) λ T (Π z Π z 1 ), and its first-order optimality conditions are (8) = z φ Π T λ + Π T +1λ +1, N (9a) = z φ n 1 Π T n 1λ n 1 (9b) = φ, N (9c) = Π z Π z 1, N. (9d) Here, z φ := z φ( ) and φ := φ( ). System (9) is the discrete-time version of the EL conditions of the lifted problem (2). Moreover, the dal variables λ can be tied together to form discrete-time profiles of the adjoint variables of the lifted problem. These properties are discssed in the boo of Biegler (21). We note that the bloc component of the EL conditions corresponding to each stage N is given by, = z φ Π T λ + Π T +1 λ +1 = (1a) = φ (1b) = Π z Π z 1 (1c) For fixed Π z 1 = z 1,m and λ +1, (1) are the firstorder conditions of the primal stage problem: φ(z, ) + (λ +1 ) T Π +1 z (11a) z, s.t. (λ ) Π z = Π z 1. (11b) For the last stage = n 1 we have the bloc component of the EL conditions: = z φ n 1 Π T n 1λ n 1 = (12a) = φ n 1 (12b) = Π n 1 z n 1 Π n 1 z n 2. (12c) For fixed Π n 1 z n 2 = z n 2,m these are the first-order conditions of the primal stage problem, φ(z n 1, n 1 ) (13a) z n 1, n 1 s.t. (λ n 1 ) Π n 1 z n 1 = Π n 1 z n 2. (13b) From the strctre of (1) and (11) we can see that copling between neighboring stages 1,, and + 1 is introdced throgh the states z 1,m and adjoints λ BLOCK GS SCHEMES Or ey observation is that we mae is that we can solve the EL conditions (9) of the long-horizon problem by sing bloc GS schemes. Assme that the adjoints λ are fixed to λ l = for all N. At stage = and with fixed z 1 l = z we solve the short-horizon problem: z,j,,j ϕ(z,j+1,,j+1 ) + δ(λ l +1) T z,m (14a) s.t. (λ ) z, = z l 1,m. (14b) We refer to this problem as P and introdce the notation,m, λl+1 ) P (z 1,m, l λ l +1) (15) to indicate the inpts and otpts of problem P. The primal-dal soltion of P solves bloc of the EL conditions (1) for fixed initial state Π z 1 = z 1,m l and adjoint λ +1 = λ l +1. Note also that P is eqivalent to the stage problem (11). From the soltion of P we obtain the teral state z l+1,m and we se this as initial state for P +1 to compte +1,n, λl+1 +1 ) P,m, λl +2 ). We contine the recrsion ntil reaching the last stage, = n 1. At this stage we solve problem P n 1 : ϕ(z n 1,j+1, n 1,j+1 ) (16a) z n 1,j, n 1,j s.t. (λ n 1 ) z n 1, = z l n 2,m. (16b) With this we compte (zn 1,m l+1, λl+1 n 1 ) P n 1(zn 2,m, l ). The primal-dal soltion of P n 1 solves the optimality system (12) for fixed initial state Π n 1 z n 2 = zn 2,m l obtained from the soltion of P n 2. Moreover, P n 1 is eqivalent to (13). After solving P n 1 we have pdated all the state (primal) z l+1 and adjoint λ l+1 variables. In Figre 1 we can see 44

3 IFAC DYCOPS-CAB, 216 Jne 6-8, 216. NTNU, Trondheim, Norway z` 1 `+1 =)=) (z, )+( `+1) T +1 z s.t. z = z` 1 ( ) z`+1 =) =) ` s.t. (z +1, +1 )+( `+2) T +2 z z +1 = +1 z`+1 ( +1 ) =)=) z`+1 +1 `+1 +2 Fig. 1. Setch of primal and dal pdates in GS scheme. that the pdates of the states march forward in time while the pdates march bacward in time. After this sweep, we retrn to the first stage = and repeat the recrsion to obtain z l+2, λ l+2. We repeat this procedre n GS times. We smmarize the GS scheme below: GS Scheme I) GIVEN z, set conter l, set z l 1,m z, and set λ l for =,..., n 1. FOR l =,..., n GS DO: II) FOR =,..., n 2 SOLVE,m, λl+1 ) P 1,m, λl +1 ). III) FOR = n 1 SOLVE n 1,m, λl+1 n 1 ) P n 1(z l n 2,m, ). IV) SET l l + 1 and RETURN TO Step II). We note that the GS scheme can be implemented by sing off-the-shelf optimization tools. All that is needed to solve the EL conditions (9) is the ability to solve the NLPs P. The strctre of the bloc GS scheme also reveals that a RH scheme is eqivalent to perforg a single GS iteration with adjoints λ l =, N. The adjoints λ l +1 encode important global information of the ftre horizon beyond the short-horizon h of P. In particlar, we note that the adjoints can be interpreted as teral costs. The teral term (λ l +1 )T z,m in P is a first-order approximation of the so-called cost-to-go. To see this, consider a problem with two stages and + 1: φ(z, ) + φ(z +1, +1 ) (17a) z,,z +1, +1 s.t. Π z = Π z l 1 (λ ) (17b) Π +1 z +1 = Π +1 z (λ +1 ) (17c) This problem can be written as: z, φ(z, ) + Q(Π +1 z ) (18a) s.t. Π z = Π z l 1 (λ ) (18b) where, Q(Π +1 z ) := φ(z +1, +1 ) (19a) z +1, +1 s.t. Π +1 z +1 = Π +1 z (λ +1 ). (19b) Here Q( ) is the cost-to-go fnction. If we linearize the cost-to-go at the crrent state gess z l we obtain, φ(z, ) z, + Q(Π +1 z l ) + Q(Π +1 z l ) T (Π +1 z Π +1 z l ) (2a) s.t. Π z = Π z l 1 (λ ) (2b) where the term Q(Π +1 z l )T Π +1 z l is fixed and therefore irrelevant. From dality we now that the gradient of the cost-to-go Q(Π +1 z l ) is precisely the adjoint λl +1 of the state transition constraint (19b). In other words, the adjoint is the sensitivity of the cost-to-go with respect to the initial state Π +1 z. Conseqently, problem (2) is eqivalent to, φ(z, ) + (λ l z, +1) T Π +1 z (21a) s.t. Π z = Π z l 1 (λ ). (21b) This is precisely the stage sbproblem P solved in the GS scheme. We also note that the adjoints are exact approximations of the cost-to-go at an optimal soltion of P. The cost-to-go information is neglected by the RH scheme and this can lead to a poor approximation of the long-horizon soltion. A ey insight that we gain from or analysis and from the cost-to-go interpretation is that we can correct the RH scheme to better approximate the longhorizon soltion if we are capable of obtaining estimates of the adjoints λ l. Moreover, in the ideal case where sch adjoint estimates are optimal for P, the corrected RH scheme will deliver optimal long-horizon profiles for P. 4. COARSENING-BASED CORRECTION We can obtain estimates of the adjoints λ l by perforg mltiple GS iterations. This can be seen as a self-correcting RH scheme. GS schemes provide the comptational advantage that they only need to solve only short-horizon problems. GS schemes, however, are well nown for exhibiting slow convergence or no convergence at all. We address this isse by sing adjoint estimates λ l obtained from the soltion of a coarsened long-horizon problem. To perform coarsening, we consider a coarse grid with m c elements and m c m (sch that the reslting coarse problem is tractable). We define the coarse set as M c := {..m c 1} and the corresponding coarsened long-horizon problem P c. We se the notation (λ l,..., λ l n 1) P c ( z) to indicate the inpts and otpts of P c. We consider the scheme: Corrected GS Scheme I) GIVEN z, set conter l, set z l 1,m z. II) SOLVE (λ l,..., λ l n 1) P c ( z) and FOR l =,..., n GS DO: III) FOR =,..., n 2 SOLVE,m, λl+1 ) P 1,m, λl +1 ). IV) FOR = n 1 SOLVE n 1,m, λl+1 n 1 ) P n 1(z l n 2,m, ). V) SET l l + 1 and RETURN TO Step III). In this scheme, P c transfers global (long-horizon) information of problem P to the local (short-horizon) problems P. This is depicted in Figre 2. This approach can be seen as a hierarchical control scheme in which a coarsegrained high-level controller spervises a fine-grained lowlevel controller. In other words, the coarse long-horizon problem provides teral costs (i.e., a cost-to-go) to the RH controller so that this better approximates the soltion of the long-horizon problem. Dal information is 45

4 IFAC DYCOPS-CAB, 216 Jne 6-8, 216. NTNU, Trondheim, Norway P c m c +1 P ` +1 m ` +1 ` n 1 n 1 Fig. 2. Transfer of adjoint information from coarse longhorizon problem to fine short-horizon problems. transferred in this hierarchical controller; as opposed to traditional hierarchical schemes that transfer state information. This is ey becase dal information provided by the coarse controller can be refined by the low-level controller (throgh GS pdates) and ths it is possible to achieve primal-dal optimality. In the comprehensive review on hierarchical predictive control of Scattolini (29), the athor notices that systematic design methods for hierarchical control are still lacing. More specifically, no hierarchical schemes have been proposed that systematically aggregate and refine trajectories at mltiple time scales. In addition, existing schemes have been tailored to achieve feasibility bt do not have optimality garantees. Scattolini also arges that it is important to consider mltirate methods (compting controls at different time resoltions). As can be seen, the hierarchical control strctre created in this wor addresses both of these isses. It is also worth remaring that several hierarchical schemes for MPC have been proposed that decople time scales sing singlar pertrbations. For more information on sch approaches, the reader is referred to the wor of Ellis et al. (213); Chen et al. (212); Scattolini (29). These approaches enable handling of slow and fast modes bt do not address mltiple time scales. Or analysis borrows concepts of mltigrid control. We now describe a general mltigrid framewor to highlight the many connections that exist with hierarchical control. Mltigrid is a scientific compting paradigm widely sed for the simlation and control of systems described by PDEs Borzı (23). The idea behind mltigrid is to discretize the PDE space-time domain in a seqence of grids of increasing resoltion. At the highest level is the coarsest grid that is comptationally tractable. The ey observation is that the coarse level can captre slow moving fronts occrring at low freqencies bt will miss the fast moving fronts occrring at higher freqencies. The coarse state, control, and adjoint (dal or co-state) fields are passed to the second level of the hierarchy. At this level, the coarse fields are refined sing a smoothing scheme (typically a GS scheme). The GS scheme exchanges information among its nearest neighbors. By doing this, the GS scheme can eliate fast local moving fronts qicly (bt not necessarily the slow global ones). Conseqently, by combining the coarse and fine levels one sees to target the entire freqency spectrm. As can be seen, many of the problems arising in the simlation and control of PDEs also appear in hierarchical control. We have addressed several technical isses that enabled s to transfer ideas between these two domains. These inclde: We established connections between the strctre of the EL conditions and the soltion of stage-by-stage optimal control problems in the primal space. This enables s to se GS schemes in a general setting and establishes a connection between RH control and GS. We se a primal representation of stage problems P corresponding to the stage bloc of the EL conditions. With this we demonstrate that coarsening schemes can be sed to obtain adjoint information that is sed as teral penalties of short-horizon stage problems. We demonstrate that GS and coarsening schemes can be implemented sing off-the-shelf optimization tools, as opposed to intrsive linear algebra maniplations sed in PDE stdies. We demonstrate that only dal information needs to be commnicated between hierarchical levels, as opposed to only state and control information commnicated in mltigrid PDE schemes. The lifting approach avoids the need to perform interpolation of state and adjoint profiles in order to move from a coarse grid to a fine grid (as is typically done in PDEs). In the proposed approach, all that is needed from the coarse problem are the adjoints at the stage transition points. 5. NUMERICAL STUDIES We discss the nmerical concepts sing a CSTR example and we then demonstrate how to se hierarchical control in a more complicated microgrid control example. 5.1 CSTR We consider the nonlinear Hics-Ray CSTR reactor problem reported in Zavala and Anitesc (21). The system states are the concentration of reactant c( ) and the temperatre of reacting mixtre t( ). The control is the cooling water flow ( ). After lifting, we denote the adjoint associated with the concentration as λ c and the adjoint associated with temperatre as λ t. We partition the fll time horizon in n = 1 stages and discretize each stage sing an implicit Eler scheme with m = 1 grid points. In Figre 3 we present the control profiles obtained with the GS Scheme. As can be seen, the profiles obtained with a standard RH scheme (first iteration of GS scheme) severely deviate from the optimal ones. GS approximates the longhorizon soltion fairly well after three iterations. In Figre 4 we present the optimal and approximate adjoint profiles obtained from coarsening. The coarse profiles are obtained by discretizing the stage sing m c = 2 points (compared to the m = 1 points sed for the optimal profile). This is a coarsening factor of five. As can be seen, the adjoint profiles exhibit errors bt the overall strctre is preserved. We se the coarse adjoint profiles to correct the GS scheme (provide teral penalties for the short-horizon stage problems). In Figre 5 we present the convergence of GS with and withot correction. As can be seen, coarse correction delivers close-to-optimal performance optimal performance after a single GS iteration. 46

5 IFAC DYCOPS-CAB, 216 Jne 6-8, 216. NTNU, Trondheim, Norway Gass Seidel Time Uncorrected Corrected Time Fig. 3. Scheme convergence for control. First iteration (top) and third iteration (bottom). λ t λ c (m=1) Coarse (m c =2) Time Fig. 4. and coarsened adjoint profiles for temperatre (top) and concentration (bottom). 6. MICROGRID We now demonstrate how the hierarchical controller can manage distrbances with mltiple time scales. We consider the microgrid system shown in Figre 6. The system has two generators and a battery storage system. The objective to find the optimal operating policy for the generators and storage that imizes cost over a long horizon spanning 4 days. We assme that the electrical load (distrbance) of this system has three freqency components (load has periods of 24 hr, 12 hr, and 1 hr). These are shown in the right panel of Figre 7. In the left panels of Figre 7 we show the coarsened load profile and the error indced by applying sch coarsening. As can be seen, coarsening misses the high freqency. In the left colmn of Figre 8 we present, from top to bottom, the storage policies for short-horizon RH control (GS withot adjoint information), GS scheme after two iterations, coarse control, and hierarchical control. We split the horizon in 4 stages of one day and coarsen the horizon by a factor of for. In the right colmn we present the sboptimality errors for the corresponding storage policies with respect to the long-horizon problem P. From the first Fig. 5. Control policies for GS scheme with and withot coarsening correction. First iteration (top) and second iteration (bottom). Solar Radiation Solar Radiation Distribtion Bsbar Solar Radiation Grid = DC-DC Converters Distribtion Distribtion Electrolyzer Generator I Generator Bsbar Bsbar Fel Cell II Storage Battery System Ban = DC-DC = DC-DC = = Converters = Converters = Hydrogen Storage Fig. 6. Microgrid Electrolyzer control example. Electrolyzer Fel Cell Fel Cell Load [MW] Load Coarsening Error [MW] To Sbstation Load Grid= DC-ACGrid Inverters = Load Hydrogen StorageHydrogen Storage Load [MW] DC-AC Inverters DC-AC Inverters Battery Ban Load Battery Ban Fig. 7. Real and coarsened loads (top left), load coarsening error (bottom left), and freqency components of load (right). row we see that the RH policy is far from optimal. This is becase the controller marches in stages of one day and ths ignores long horizon load trends. As a reslt, the RH controller depletes the crrent inventory in trying to imize short-term generation cost. In the second row we see that a GS scheme can better approximate the optimal policy after the second iteration becase it can estimate the adjoints that are sed to captre long horizon information. The error in the storage policy is large in 47

6 Generation [MW] Adjoint [$/MW] Storage Flow [MW] IFAC DYCOPS-CAB, 216 Jne 6-8, 216. NTNU, Trondheim, Norway Receding-Horizon) Gass-Seidel (It 2) Coarse Hierarchical Fig. 8. From top to bottom: Storage policies and errors with short-horizon RH scheme, GS scheme (after 2 iterations), coarse control, and hierarchical control. magnitde bt it is flattened ot, indicating that the GS scheme eliates the high freqencies of the load bt not the low freqencies. In the third row we see that a pre coarse controller has close-to-optimal performance bt the storage error shows that it misses the high freqencies of the load. In the forth row we see that hierarchical control obtains optimal performance becase it can deal with high and low freqencies. In Figre 9 we present storage, generation, and adjoint policies. We again note that RH policies are far from optimal and that the GS scheme can recover them after a few iterations RH GS (It 5) Fig. 9. From top to bottom: storage, adjoint, and generation profiles. 7. CONCLUSIONS AND FUTURE WORK We presented an analysis of the Eler-Lagrange conditions for a lifted optimal control problem. This enabled s to derive bloc GS schemes capable of solving intractable long-horizon problems by solving seqences of tractable short-horizon problems. Or analysis revealed that a RH scheme is eqivalent to perforg a GS sweep of the Eler-Lagrange conditions. We have also sed or analysis to derive strategies to correct adjoint profiles by sing coarsening. This approach can interpreted as a hierarchical control strctre in which a coarse high-level controller transfers long-horizon information to a low-level, shorthorizon controller of fine resoltion. Or reslts open the door to the design of new hierarchical control applications. As part of ftre wor, we wold lie to gain additional insight on conditions garanteeing convergence of GS schemes and we will design mlti-level hierarchies. ACKNOWLEDGEMENTS This material is based pon wor spported by the U.S. Department of Energy, Office of Science, nder an Early Career Award. The athor thans Mihai Anitesc for pointing ot existing wor on mltigrid control. REFERENCES Biegler, L.T. (21). Nonlinear programg: Concepts, algorithms, and applications to chemical processes. SIAM. Boc, H.G. and Plitt, K.J. (1984). A mltiple shooting algorithm for direct soltion of optimal control problems. 9th IFAC World Congress, Bdapest. Borzı, A. (23). Mltigrid methods for parabolic distribted optimal control problems. Jornal of Comptational and Applied Mathematics, 157(2), Chen, X., Heidarinejad, M., Li, J., and Christofides, P.D. (212). Composite fast-slow mpc design for nonlinear singlarly pertrbed systems. AIChE Jornal, 58(6), Diehl, M., Ferrea, H.J., and Haverbee, N. (29). Efficient nmerical methods for nonlinear MPC and moving horizon estimation. In Nonlinear Model Predictive Control, Ellis, M., Heidarinejad, M., and Christofides, P.D. (213). Economic model predictive control of nonlinear singlarly pertrbed systems. Jornal of Process Control, 23(5), Findeisen, R., Raff, T., and Allgöwer, F. (27). Sampleddata nonlinear model predictive control for constrained continos time systems. In Advanced Strategies in Control Systems with Inpt and Otpt Constraints, Springer. Scattolini, R. (29). Architectres for distribted and hierarchical model predictive control A review. Jornal of Process Control, 19(5), Zavala, V.M. and Biegler, L.T. (29). Nonlinear programg strategies for state estimation and model predictive control. In Nonlinear Model Predictive Control, Zavala, V.M. and Anitesc, M. (21). Real-time nonlinear optimization as a generalized eqation. SIAM Jornal on Control and Optimization, 48(8),