Convex Functions & Optimization

Transcription

1 672 Conve Functions & Optimization Aashray Yadav Abstract - My research paper is based on the recent work in interior-point methods, specifically those methods that keep track of both the primal and dual optimization variables (hence primal-dual methods). These methods are special because they are numerically stable under a wide range of conditions, so they should work well for many different types of constrained optimization problems.however, you can always find a constrained optimization problem that is difficult enough to break these methods. Keywords - Introduction, Types of Optimization, Graphical Minima, Conve function, Conve vs. Non-conve, Functions, Conve Hull, Test for conveity and Concavity, Conve Region, Solving Techniques, Some common conve OP s, LP Visualization, Quadratic Programming, QP Visualization, Interior Point Method,CVX:Conve Optimization, Building Conve Functions, Verifying Conveity Remarks, References Introduction 3.Graphical Minima Optimization is the mathematical discipline which is concerned with finding the maima and minima of functions, possibly subject to constraints. It helps in various field such as Architecture, Nutrition, Electrical circuits, Economics, Transportation,etc. 2.Types of Optimization a)to find the minimum of the function a)a real function of n variables f ( 1, 2,, n ) with or without constrains min f (, y) 2 2 y 2 b)unconstrained optimization c) Optimization with constraints min f (, y ) 2 2 y 2 0 or min f (, y ) 2 2 y 2 2 5, y 1 or min f (, y ) 2 2 y 2 y 2 What is special about a local ma or a local min of a function f ()? at local ma or local min f ()=0 f () > 0 if local min f () < 0 if local ma Aashray Yadav is pursuing Bachelor degree in Software engineering at Delhi Technological University, India

2 673 d)eamples 4.Conve Function a)definition The weighted mean of function evaluated at any two points is greater than or equal to the function evaluated at the weighted mean of the two points Conve b)procedure a)pick any two points, y and evaluate along the function, f(), f(y) b)draw the line passing through the two points f() and f(y) c)conve if function evaluated on any point along the line between and y is below the line between f() and f(y) c)graph Not Conve

3 674 5.Local Optima is Global (simple proof) Not Conve 7. Functions Conve 6. Conve vs. Non-conve Conve A function is called conve (strictly conve) if is replaced by (<). f() f ( ) 0 a b Concave A function is called concave over a given region R if: f ( a (1 )b ) f (a ) (1 ) f (b ) where: a, b R, and 0 1. The function is strictly concave if is replaced by >. f() f ( ) 0 a b

4 675 8.Conve Hull A set C is conve if every point on the line segment connecting and y is in C. The conve hull for a set of points X is the minimal conve set containing X. Eample: f ( ) f () 2 f () 2 f () f () 2 f () H ( ), 1 4, f 0 then f ( ) is concave. 2 2 f If f ( ) 2 0 then f ( ) is conve. If f ( ) For a multivariate function f() the conditions are:- 10.Conve Region f() Strictly conve conve concave strictly concave , 2 7. Hence, f () is strictly conve. eigenvalues: I 2 H h() Hessian Matri +ve def +ve semi def -ve semi def -ve def b a a conve region non conve region b 9.Test for Conveity and Concavity H is -ve def (-ve semi def) iff A conve set of points eist if for any two points, a and b, in a region, all points: T H 0 ( 0), 0. a (1 )b, 0 1 T H 0 ( 0), 0. Convenient tests: H() is strictly conve (+ve def) (conve) (+ve semi def)) if: 1. all eigenvalues of H() are 0 ( 0) or 2. all principal determinants of H() are 0 ( 0) on the straight line joining a and b are in the set. If a region is completely bounded by concave functions then the functions form a conve region.

5 11.Solving Techniques 14.Quadratic Program Can use definition (prove holds) to prove If function restricted to any line is conve, function is conve If 2X differentiable, show hessian >= 0 Often easier to: Convert to a known conve OP E.g. QP, LP, SOCP, SDP, often of a more general form Combine known conve functions (building blocks) using operations that preserve conveity Similar idea to building kernels QP: Quadratic objective, affine constraints 676 LP is special case Many SVM problems result in QP, regression If constraint functions quadratic, then Quadratically Constrained Quadratic Program (QCQP) 12. Some common conve OPs Of particular interest for this book and chapter: linear programming (LP) and quadratic programming (QP) LP: affine objective function, affine constraints 15.QP Visualization -e.g. LP SVM, portfolio management 13.LP Visualization 16.Interior Point Method Solve a series of equality constrained problems with Newton s method Approimate constraints with log-barrier (appro. of indicator) Note: constraints form feasible set -for LP, polyhedra As t gets larger, approimation becomes better

6 CVX: Conve Optimization 18.Building Conve Functions a)introduction From simple conve functions to comple: some operations that preserve compleity Nonnegative weighted sum Composition with affine function Pointwise maimum and supremum Composition Minimization Perspective ( g(,t) = tf(/t) ) CVX is a Matlab toolbo Allows you to fleibly epress conve optimization problems Translates these to a general form and uses efficient solver (SOCP, SDP, or a series of these) 19.Verifying Conveity Remarks All you have to do is design the conve optimization problem Plug into CVX, a first version of algorithm implemented More specialized solver may be necessary for some applications b)cvx - Eamples I) Quadratic program: given H, f, A, and b cv_begin variable (n) minimize ( *H* + f *) subject to A* >= b cv_end II) For more detail and epansion, consult the referenced tet, Conve Optimization Geometric Programs also conve, can be handled with a series of SDPs (skipped details here) CVX converts the problem either to SOCP or SDM (or a series of) and uses efficient solver 20.References a) Conve Optimization Boyd and Vandenberghe b) Tokhomirov, V. M. "The Evolution of Methods of Conve Optimization." Amer. Math. Monthly 103, 65-71, c) Conve Optimization Theory-Mathworld,Wolfram d)matlab- SVM-type formulation with L1 norm cv_begin variable w(p) variable b(1) variable e(n) epression by(n) by = train_label.*b; minimize( w'*(l + I)*w + C*sum(e) + l1_lambda*norm(w,1) ) subject to X*w + by >= a - e; e >= ec; cv_end