A Linear Programming Approach to Concave Approximation and Value Function Iteration

Transcription

1 A Linear Programming Approach to Concave Approximation and Value Function Iteration Ronaldo Carpio Takashi Kamihigashi May 18, 2015 Abstract A basic task in numerical computation is to approximate a continuous function, sometimes with shape constraints. One approach that guarantees convexity or concavity is polyhedral approximation, which may take an outer linearization or inner lineraization representation; evaulation of the approximation at a given point involves solving a linear program. This method is reasonably efficient and readily scales to higher dimensions. Furthermore, this representation allows us to quickly perform value function iteration, as well as compute the concave conjugate (Fenchel transform). We illustrate how this technique can be used to implement the Fast Bellman Iteration algorithm proposed in Carpio & Kamihigashi (2015). School of Business and Finance, University of International Business and Economics, Beijing , China. rncarpio@yahoo.com RIEB, Kobe University, 2-1 Rokkodai, Nada, Kobe Japan. tkamihig@rieb.kobe-u.ac.jp.

2 1 Introduction A basic task in computational economics is to approximate an arbitrary concave or convex function while preserving its shape. In one dimension, this is easily done with linear interpolation or shape-preserving splines. In higher dimensions, this becomes a nontrivial task; approximation methods that effectively handle higher dimensions (e.g. Delaunay triangulation, Chebyshev polynomials) do not guarantee that the function is concave. Shape-preserving splines have been extended to two, but not a higher number of dimensions. In this paper, we explore the usage of polyhedral approximations, which are evaluated by solving a linear program. This approach has been used to solve dynamic programming problems (Fukushima & Waki 2013); our approach is similar to theirs in its overall goals, but we focus on illustrating how to use LP approximations for general purpose tasks in computation. In addition, we show how the Fast Bellman Iteration algorithm proposed in Carpio & Kamihigashi (2015) can be implemented using LP approximations. This requires efficiently computing the concave conjugate (Fenchel transform) of the utility and value functions; as it turns out, the LP representation allows us to find the concave conjugate at essentially zero cost. 2 Outer & Inner Linearizations Let f : R n R be a concave function, and let u 1,...u m dom f be a set of gridpoints with function values f(u 1 ),...f(u m ). Suppose for each gridpoint, we also have a supergradient g(u 1 ),...g(u m ). To evaluate the outer linearization approximation ˆf of f at a point α = (α 1,...α n ), we solve the primal and dual linear programs using the parameters min c T x s.t. Gx h, Ax = b (1) max h T z b T y s.t. G T z + A T y + c = 0, z 0 (2) (3) c = ( g(u 1 ) 1 ) (4) G =... 1 (5) g(u m ) 1

3 f(u 1 ) + g(u 1 x 1 h =. (6) f(u m ) + g(u m x m A = ( I n 0 ) (7) b = ( α 1... α n (8) This problem has n + 1 variables and m + n constraints. Suppose the primal and dual solutions are x, z, and y. Then ˆf(α) = c T x, and y is a supergradient of ˆf at α. Each tuple (ui, f(u i ), g(u i )) defines a supporting hyperplane to f tangent at the point (u i, f(u i )); the hypograph of ˆf is the intersection of the m lower half-spaces defined by the m gridpoints. This approximation can have an infinite domain; the LP solver should report that it is infeasible or unbounded if evaluated at an α outside dom f. We can also evaluate the inner linearization approximation ˆf(α) by using the parameters c = ( f(u 1 )... f(u m ) G = I m (9) (10) h = 0 (11) A = u T 1... u T (12) m b = ( 0 α 1... α n (13) This problem has m variables and m + n + 1 constraints. ˆf(α) = ct x and a supergradient of at α is given by the last n components of y ˆf. This essentially finds a point in the convex hull of the points {(u i, f(u i ))} i by solving for the weights allocated to each extreme point. Note that the outer linearization can have an infinite domain, while the inner linearization must have a finite domain. In our experience it is faster to evaluate the outer linearization, at the cost of specifying more information. There are two main types of algorithms for solving linear programs: the simplex method and interior-point methods. Simplex methods have the advantage that they can be warm-started, that is, they can start at the solution to a previously solved problem. We will compare the performance of the Python packages CVXOPT (an interior-point solver) and CyLP, which is an interface to the Coin-OR CLP library (a simplex method solver). 2

4 When implementing these methods, we commonly evaluate a given smooth function f at gridpoints u 1,...u m and obtain its gradients f(u 1 ),... f(u m ). Note that this contains more information than is strictly needed to specify the supporting hyperplanes of ˆf; in addition to each hyperplane, we also know that the constraint defined by hyperplane i will be active at u i. This will be useful later on when we want to compute the conjugate (Fenchel transform) of our approximations. 3 Operations 3.1 Linear Combination Suppose we have two outer linear approximations ˆf 1, ˆf 2 specified by a set of gridpoints with associated function value and supergradient information {u 1i, f 1 (u 1i ), g 1 (u 1i )} i, {u 2j, f 2 (u 1j ), g 2 (u 1j )} j respectively, and with LP parameters (c 1, G 1, h 1, A 1, b 1 ), (c 2, G 2, h 2, A 2, b 2 ) respectively. For β 1, β 2 0, we can compute the linear combination β 1 ˆf1 +β 2 ˆf2 evaluated at α by solving a LP with the parameters c = β 1 c T 1 β 2 c T T 2 (14) G1 G = (15) G 2 h = h T 1 h T T 2 (16) A1 A = (17) A 2 b = ( b T 1 b T 2 (18) This is also an outer linearization. This formulation gives an exact solution, at the cost of doubling the number of variables and the dimensions of the constraint matrices. A faster, but less accurate, method would be to simply evaluate ˆf 1 and ˆf 2 at the union of the gridpoints {u 1i } i {u 2j } j, add the function values and supergradients, and create a new outer linearization with the resulting values. This can be easily extended to any number of terms. 3

5 3.2 Composition Suppose we have an an outer function f : R n R and an inner function h : R p R n, and we are able to compute supergradients for f, and an n p Jacobian matrix of supergradients for h on a set of gridpoints u 1,...u m R m. We can construct an outer linearization approximation of (f h) by using the gridpoints u 1,..., u m, function values f(h(u 1 )),...f(h(u m )), and supergradients calculated via the chain rule: g(u i ) = J h(ui )(f)j u h, where J x (f) denotes the Jacobian matrix of supergradients of the function f, evaluated at the point x. 3.3 Supremal Convolution Suppose we have two outer linear approximations ˆf 1, ˆf 2 specified by {u 1i, f 1 (u 1i ), g 1 (u 1i )} i, {u 2j, f 2 (u 1j ), g 2 (u 1j )} j respectively. Given beta 1, β 2 0, we wish to compute an approximation to the supremal convolution of β 1 ˆf1 and β 2 ˆf2 : ( ˆf 1 # ˆf 2 )(α) = inf x [ ] β 1 ˆf1 (x) + β 2 ˆf2 (α x) (19) We can construct an outer linearization approximation evaluated at α by solving a LP with the parameters c = β 1 c T 1 β 2 c T T 2 (20) G1 G = (21) G 2 h = ( h T 1 h T 2 (22) A = ( A 1 A 2 ) (23) b = ( α 1... α n (24) 3.4 Concave Conjugate (Fenchel Transform) Given any function f : R n R, the concave conjugate (also known as the Fenchel transform), f : R n R, is defined as f (p) = inf x R n{pt x f(x)}, p R n (25) 4

6 If f is an outer linearization specified by {u i, f ( u i ), g ( u i )} i, then its concave conjugate is an inner linearization specified by the convex hull of the points {(g ( u i ), f(u i ) + g(u i x i )} i (Rockafellar & Wets (2009hm (d)). Likewise, if f is an inner linearization specified by the points {u i, f ( u i )} i, f is an outer linearization specified by the set of (gridpoint, function value, supergradient) tuples {g ( u i ), f(u i ) + g(u i u i, u i )} i, where g(u i ) is a supergradient of f at u i. This representation allows us to find the concave conjugate of concave piecewise linear functions at essentially no cost. Note that the vertices of the outer linearization are implicitly determined, as are the gradients of the faces of the inner linearization. If we assume that dom f is R n + and f is bounded from above, increasing and concave, then dom f is also R N +. For practical purposes, given an outer linearization specified by {u i, f ( u i ), g ( u i )} i, we will approximate its concave conjugate with the outer linearization specified by {(g ( u i ), f(u i ) + g(u i x i ), u i } i. This approximation becomes exact as the number of gridpoints increases to infinity. 4 Dynamic Programming Consider the following infinite horizon dynamic programming problem specified by the Bellman equation v(x, z) = sup {u(y) + β π(z z)v(f(x) y)} (26) y R n z Z z is a random shock that follows a time-homogeneous, finite Markov chain with state space Z. pi(z z) is the probability of transitioning to state z, given that the initial state is z. u and f z are assumed to be concave and increasing; we assume that if v is concave, then so is v f z. A useful fact is that given two concave functions f 1, f 2, the concave conjugate of their supremal convolution is the sum of the conjugates of the individual functions (Rockafellar & Wets (2009), Thm (a)): (f 1 #f 2 ) (p) = f 1 (p) + f 2 (p) (27) This implies that the Bellman operator, which normally involves expensive optimization operations, can be transformed into an operation that only requires algebraic operations. As in Carpio & Kamihigashi (2015), we can take the concave conjugate of the Bellman operator Bv = sup {u(y) + β π(z z)v(f z (x) y)} (28) y R n z Z 5

7 to obtain the dual Bellman operator (B v)(p) = u (p) + (β z Z π(z z)v f z ) (p)} (29) The supremum operator in the Bellman equation is transformed into an additive operation on the conjugates of u and βev. In Carpio & Kamihigashi (2015), we proposed the Fast Bellman Iteration algorithm which numerically implements the dual Bellman operator. Here, we propose an implementation using outer and inner linearizations that extends our original implementation to higher dimensions and to a general family of transformation functions f z (so long as concavity of the value function is preserved): Given outer linearization approximations to u, f z, and a set of initial guesses for the value function w z, the algorithm is as follows: 1. For each z, z Z, compute the composition (w z f z ). 2. For each z Z, compute the expected value function as the linear combination Ev z = z Z π(z z)(w z f z ). 3. Compute the concave conjugate of each expected value function multiplied by the discount factor, (βev z ). 4. Add u, the concave conjugate of the utility function u, to each concave conjugate in part (3). This gives (Bw), the concave conjugate of the Bellman operator applied to w. 5. Compute the concave conjugate of each (Bw), to obtain Bw. 6. Check for numerical convergence. If convergence is reached, terminate; otherwise, go to step (1). References Carpio, R., Kamihigashi, T., 2015, Fast Bellman Iteration: An Application of Legendre-Fenchel Duality to Infinite-Horizon Dynamic Programming in Discrete Time, RIEB Discussion Paper DP Fukushima, K., Waki, Y., 2013, A polyhedral approximation approach to concave numerical dynamic programming, Journal of Economic Dynamics & Control 37, Rockafellar, R.T., Wets, R.J.B., 2009, Variational Analysis, Springer, Dordrecht. 6