15. Cutting plane and ellipsoid methods

Transcription

1 EE 546, Univ of Washington, Spring Cutting plane and ellipsoid methods localization methods cutting-plane oracle examples of cutting plane methods ellipsoid method convergence proof inequality constraints Cutting plane methods 15 1

2 Localization methods localize desired point in some set, which becomes smaller at each step require one subgradient of objective or constraint functions at each step handle nondifferentiable convex (and quasiconvex) problems typically require more memory and computation per step than subgradient method but can be much more efficient (in theory and practice) Cutting plane methods 15 2

3 Cutting-plane oracle goal: find a point in convex set C R n, or determine that C is empty cutting-plane oracle: when queried at x R n, oracle either asserts that x C, or returns a separating hyperplane between x and C: a 0, a T z b for z C, a T x b oracle provides black-box description of C neutral cut: a T x = b (query point is on boundary of halfspace that is cut) deep cut: a T x > b (query point is in interior of halfspace that is cut) Cutting plane methods 15 3

4 Unconstrained minimization take set of minimizers of f as C cutting-plane oracle (for convex f): if 0 g f(x), then g T (z x) 0 defines a (neutral) cut (a,b) = (g,g T x) at x proof: g T (z x) > 0 implies z / C because f(z) f(x)+g T (z x) > f(x) Cutting plane methods 15 4

5 interpretation: level curves of f x g g T (z x) 0 by evaluating g f(x) we rule out halfspace in search for optimum get one bit of info (on location of x) by evaluating g Cutting plane methods 15 5

6 Deep cut for unconstrained minimization suppose we know a number f with f(x) > f f (e.g., the smallest value of f found so far in an algorithm) deep cut: if g f(x), then a deep cut is given by g T (z x)+f(x) f 0 proof: g T (z x)+f(x) > f implies f(z) > f, so z / C. Cutting plane methods 15 6

7 Feasibility problem C is solution set of convex inequalities f i (x) 0, i = 1,...,m cutting-plane oracle: if x not feasible, find j with f j (x) > 0 and evaluate g j f j (x); f j (x)+gj T (z x) 0 is a deep cut proof: f j (x)+g T j (z x) > 0 implies z / C because f j (z) > f j (x)+g T j (z x) > 0 Cutting plane methods 15 7

8 Inequality constrained problem C is the set of optimal points of convex problem cutting-plane oracle: minimize f 0 (x) subject to f i (x) 0, i = 1,...,m if x is not feasible, say f j (x) > 0, we have (deep) feasibility cut f j (x)+g T j (z x) 0 where g j f j (x) if x is feasible, we have (neutral) objective cut g T 0(z x) 0 where g 0 f 0 (x) (or, deep cut g T 0(z x)+f 0 (x) f 0 if f [p,f 0 (x)) is known) Cutting plane methods 15 8

9 (Conceptual) cutting-plane algorithm given initial polyhedron P 0 = {z Az b} known to contain C repeat for k = 1,2,... choose a point x (k) in P k 1 and query the cutting-plane oracle at x (k) if x (k) C, return x (k) else, add new cutting-plane a T k z b k if P k =, quit P k := P k 1 {z a T kz b k } choice of query point: should be near center of P k 1 ; want to pick x (k) so that P k is as small as possible, for any cut Cutting plane methods 15 9

10 geometry P k 1 g g x (k) x (k) P k P k gives uncertainty of C after iteration k Cutting plane methods 15 10

11 Lower bound on complexity problem class: find x C R n, where the following is known about C C is convex C is contained in {x x R} C contains {x x x c r} a cutting-plane oracle for C bound on complexity: no localization algorithm can guarantee a complexity lower than nlog 2 ( R 2r ) iterations Cutting plane methods 15 11

12 proof: suppose we run a localization algorithm for k < nlog 2 ( R 2r ) iterations. we will construct a resisting oracle for a hyperrectangle C = {x c d x c+d} that does not contain any of the k query points and satisfies max( c i +d i ) R, mind i R i i 2 r k/n therefore, the algorithm failed to find a point in C in k steps even though {x x c r} C {x x R} Cutting plane methods 15 12

13 the oracle and c, d are constructed as follows: initially, c = 0, d = R at iteration j, define i = j n j 1 n (cycle through the n coordinates) if x is the query point at iteration j, then if x i c i, update c, d as and return the cut e T i (z x) 0 if x i < c i, update c, d as c i := c i d i /2, d i := d i /2 c i := c i +d i /2, d i := d i /2 and return the cut e T i (z x) 0 Cutting plane methods 15 13

14 Example: bisection on R for minimizing convex f : R R given interval P 0 = [l,u] containing x repeat: x := (l+u)/2; if f 0 (x) < 0, l := x; else u := x x (k+1) P k P k+1 Cutting plane methods 15 14

15 iteration complexity length(p k ) = u (k) l (k) = u(k 1) l (k 1) 2 = 1 2 length(p k 1) so length(p k ) = 2 k length(p 0 ) number of steps required for uncertainty (in x ) r: log 2 length(p 0 ) r = log 2 initial uncertainty final uncertainty length(p k ) measures our uncertainty in x uncertainty is halved at each iteration; get exactly one bit of info Cutting plane methods 15 15

16 Specific cutting-plane algorithms methods vary in choice of query point center of gravity (CG) algorithm maximum volume ellipsoid (MVE) cutting-plane method Chebyshev center cutting-plane method analytic center cutting-plane method (ACCPM) Cutting plane methods 15 16

17 Center of gravity algorithm take for x (k) the center of gravity of P k 1 (denoted CG(P k 1 ) x (k) = CG(P k 1 ) = P k 1 xdx P k 1 dx theorem if S R n convex, x cg = CG(S), g 0, vol(s {x g T (x x cg ) 0}) (1 1/e)vol(S) 0.63vol(S) (independent of dimension n) Cutting plane methods 15 17

18 advantages of CG-method guaranteed convergence affine-invariance iteration complexity is essentially optimal (follows from theorem on last page) disadvantage finding x (k) = CG(P k 1 ) is much harder than original problem (but, can modify CG-method to work with approximate CG computation) Cutting plane methods 15 18

19 Extensions multiple cuts oracle returns set of linear inequalities instead of just one, e.g., all violated inequalities all inequalities (including shallow cuts) multiple deep cuts at each iteration, append (set of) new inequalities to those defining P k nonlinear cuts use nonlinear convex inequalities instead of linear ones localization set no longer a polyhedron Cutting plane methods 15 19

20 Dropping constraints the problem number of linear inequalities defining P k increases at each iteration hence, computational effort to compute x (k+1) increases the solution: drop or prune constraints drop redundant constraints keep only a fixed number N of (the most relevant) constraints (can cause localization polyhedron to increase!) Cutting plane methods 15 20

21 Outline localization methods cutting-plane oracle examples of cutting plane methods ellipsoid method convergence proof inequality constraints Cutting plane methods 15 21

22 Ellipsoid method history developed by Shor, Nemirovski, Yudin in 1970s used in 1979 by Khachian to show polynomial solvability of LPs properties each step requires cutting-plane or subgradient evaluation modest storage (O(n 2 )) modest computation per step (O(n 2 )), via analytical formula extremely simple to implement efficient in theory slow but steady in practice; rarely used Cutting plane methods 15 22

23 Motivation in cutting plane methods, serious computation needed to find next query point localization polyhedron grows in complexity as algorithm progresses (with pruning, can keep m proportional to n, e.g., m = 4n) ellipsoid method addresses both issues, but retains theoretical efficiency Cutting plane methods 15 23

24 Ellipsoid algorithm for minimizing convex function idea: localize x in an ellipsoid instead of a polyhedron given an initial ellipsoid E 0 known to contain x repeat for k = 1,2, query oracle to get a neutral cut a T z b at x (k), the center of E k 1 2. set E k := minimum volume ellipsoid covering E k 1 {z a T z b} a E k 1 E k x (k) Cutting plane methods 15 24

25 differences with cutting-plane methods localization set doesn t grow more complicated generating query point is trivial but, we add unnecessary points in step 2 interpretation reduces to bisection for n = 1 can be viewed as an implementable version of the center-of-gravity cutting plane method Cutting plane methods 15 25

26 Updating the ellipsoid g E E = {z (z x) T P 1 (z x) 1} E + x E + is min. volume ellipsoid covering E {z g T (z x) 0} x + update formula (for n > 1): E + = {z (z x + ) T (P + ) 1 (z x + ) 1}, x + = x 1 n+1 P g, P+ = n2 n 2 1 (P 2 n+1 P g gt P) where g = ( 1 g T Pg )g Cutting plane methods 15 26

27 Simple stopping criterion for unconstrained problem lower bound on optimal value minimize f(x) f(x ) f(x (k) )+g (k)t (x x (k) ) f(x (k) )+ inf z E k 1 g (k)t (z x (k) ) = f(x (k) ) g (k)t P (k 1) g (k) second inequality holds since x E k 1 simple stopping criterion to guarantee f(x (k) ) f(x ) ǫ: g (k)t P (k 1) g (k) ǫ Cutting plane methods 15 27

28 Basic ellipsoid algorithm ellipsoid described as E(x,P) = {z (z x) T P 1 (z x) 1} given ellipsoid E(x,P) containing x, accuracy ǫ > 0 repeat 1. evaluate g f(x) 2. if g T Pg ǫ, return x; else update ellipsoid x := x 1 n2 P g, P := n+1 n 2 1 where g = ( 1 g T Pg )g ( P 2 ) n 1 P g gt P Cutting plane methods 15 28

29 Interpretation change coordinates z = P 1/2 z so uncertainty is isotropic (same in all directions), i.e., E is unit ball take subgradient step with fixed length 1 n+1 Shor refers to ellipsoid method gradient method with space dilation in direction of gradient Cutting plane methods 15 29

30 Proof of convergence assumptions: we consider the unconstrained problem minimize f(x) f is Lipschitz: f(y) f(x) G y x 2 {x f(x) < f +ǫ} E 0 E 0 is ball with radius R reduction of volume: can show that vol(e k+1 ) < e 1 2n vol(ek ) (reduction factor degrades rapidly with n, compared to CG cutting-plane method) Cutting plane methods 15 30

31 proof: suppose f(x (i) ) > f +ǫ, i = 1,...,k at iteration i we only discard points with f(z) f(x (i) ); therefore {z f(z) < f +ǫ} E k from Lipschitz condition, z x 2 ǫ/g implies f(z) < f +ǫ; hence B = {z z x 2 ǫ/g} E k therefore vol(b) vol(e k ), so α n (ǫ/g) n e k 2n vol(e0 ) = e k 2n αn R n (α n is volume of unit ball in R n ); this gives k 2n 2 log(rg/ǫ) Cutting plane methods 15 31

32 geometric intuition: E 0 x x (k) E k = { x x 2 ǫ/g} f(x) f +ǫ conclusion: for k > 2n 2 log(rg/ǫ), f (k) best f +ǫ Cutting plane methods 15 32

33 Interpretation of complexity since x E 0 = {x x x (1) 2 R}, our prior knowledge of f is f(x (1) ) GR f f(x (1) ) prior uncertainty in f is GR after k iterations our knowledge of f is posterior uncertainty in f is ǫ f k best ǫ f f k best iterations required: ( ) prior uncertainty 2n 2 log(rg/ǫ) = 1.39n 2 log 2 posterior uncertainty efficiency: 0.72/n 2 bits per gradient evaluation Cutting plane methods 15 33

34 Inequality constrained problems if x (k) feasible, update ellipsoid with objective cut g T 0(z x (k) )+f 0 (x (k) ) f (k) best 0, g 0 f 0 (x (k) ) f (k) best is best objective value of feasible iterates so far if x (k) infeasible, update ellipsoid with feasibility cut g T j (z x (k) )+f j (x (k) ) 0, g j f j (x (k) ) assuming f j (x (k) ) > 0 Cutting plane methods 15 34

35 Stopping criterion if x (k) is feasible, we have lower bound on p as before: p f 0 (x (k) ) g (k)t 0 P (k 1) g (k) 0 if x (k) is infeasible, we have for all x E k 1 f j (x) f j (x (k) )+g (k)t j (x x (k) ) f j (x (k) )+ inf g (k)t (z x (k) ) z E k 1 = f j (x (k) ) g (k)t j P (k 1) g (k) j hence, problem is infeasible if for some j, f j (x (k) ) g (k)t j P (k 1) g (k) j > 0 Cutting plane methods 15 35

36 stopping criteria: terminate algorithm when x (k) known to be ǫ suboptimal: x (k) is feasible and g (k)t 0 P (k 1) g (k) 0 ǫ or problem is shown to be infeasible: f j (x (k) ) g (k)t j P (k 1) g (k) j > 0 for some j Cutting plane methods 15 36

37 References and sources Yu. Nesterov, Introductory Lectures on Convex Optimization: A Basic Course (2004) (sections 3.2.5, 3.2.6) S. Boyd, course notes for EE364b, Convex Optimization II L. Vandenberghe, Lecture notes for EE236C - Optimization Methods for Large-Scale Systems (Spring 2011), UCLA Cutting plane methods 15 37