Convex Optimization. Chapter 1 - chapter 2.2

Transcription

1 Convex Optimization Chapter 1 - chapter 2.2

2 Introduction In optimization literatures, one will frequently encounter terms like linear programming, convex set convex cone, convex hull, semidefinite cone and simplex..what do they mean? The first two chapters will give definition on basic terms in Convex Optimizations.

3 Chapter Mathematical Optimization A mathematical optimization problem, or just optimization problem, has the form minimize f 0 x subject to f i x b i, i = 1,, m. (1.1) x is called optimal or a solution to 1.1 if it has smallest objective value amount all vector that satisfied the constraint. (1.1) is Linear Program if it satisfied f i αx + βy = αf i x + βf i (y) (1.2) (1.1) is Convex Optimization Problem if it satisfied f i αx + βy αf i x + βf i (y) (1.3) For any x, y R n and all α, β R For f 0,, f i

4 1.1.1 Applications : In portfolio optimization, for example, we seek the best way to invest some capital in a set of n assets Another example is device sizing in electronic design, which is the task of choosing the width and length of each device in an electronic circuit. In data fitting, the task is to find a model, from a family of potential models, that best fits some observed data and prior information And many more..

5 1.2 Least Square Problem A least-squares problem is an optimization problem with no constraints minimize f 0 x = Ax b 2 k 2 = i=1 a T 2 i x b i Analytical solution: (A T A)x = A T b Application: in a overdetermined Ax = b system, how find appoint that minimized all violation? Readily solvable with O n 2 k complexity for dense matrix A (1.4)

6 1.2.2 linear programming Another important class of optimization problems is linear programming, in which the objective and all constraint functions are linear: minimize c T x subject to a i T x b i, i = 1,, m (1.5) vectors c, a 1,, a m R n and scalars b 1, b m R

7 1.2.2 linear programming There is no simple analytical formula for the solution of a linear program (as there is for a least-squares problem), but there are a variety of very effective methods for solving them, including Dantzig s simplex method, and the more recent interior point methods Complexity in practice O(n 2 m), not factoring in precision O(log 1 ε )

8 1.2.2 Using linear programming example Chebyshev approximation problem minimize max i=1,,k a i T x b i (1.6) Can be solve by linear program: minimize t subject to a i T x t b i, i = 1, k (1.7) a i T x t b i i = 1, k

9 1.2.2 Using linear programming example One of the aim is to required the skill of recognized problem such as Chebyshev approximation problem and transform them into Linear Programming

10 1.3.2 The challenge is to formulate a problem in to Convex Optimization, consider the maturity of technology like interior Point Method, if a problem is formulated into Convex Optimization, then it can be readily solved In the GO method we went over before, leverage on SDP, which is a special case of formulation in Convex Optimization

11 1.3 Convex Optimization minimize f 0 x subject to f i x b i, i = 1,, m. (1.8) f 0,, f m R n R is Convex Optimization Problem, For f 0,, f i f i αx + βy αf i x + βf i (y) (1.3) For any x, y R n and all α, β R Least Square and Linear Program are special cases of Convex Optimization

12 1.4 Nonlinear optimization Nonlinear optimization (or nonlinear programming) is the term used to describe an optimization problem when the objective or constraint functions are not linear, but not known to be convex. Sadly, there are no effective methods for solving the general nonlinear programming problem (1.1). Even simple looking problems with as few as ten variables can be extremely challenging, while problems with a few hundreds of variables can be intractable. Methods for the general nonlinear programming problem therefore take several different approaches, each of which involves some compromise. Compromise between quality of solution vs time used.

13 1.4 Nonlinear optimization Roles of Convex Optimization in Nonlinear Optimization Formulate a Nonlinear Optimization into Convex Optimization, with a different solution space. Solution might not be feasible in the original Nonlinear Optimization. Initialization for local optimization. Solve the convex formulated problem once to obtain a initialized solution. Provides a lower bound to the original Nonlinear Optimization.

14 Chapter Affine and convex sets Lines and line segments Suppose x 1 x 2 are two points in R n. Points of the form y = θx θ x 2 Forms the line passes x 1, x 2. θ = 1, y = x 1. θ = 0, y = x 2.,θ R

15 2.1.2 Affine Sets A set C R n is affine if line through any two distinct point in C is in C. I.e for any x 1, x 2 C, θ R and, we have θx θ x 2 in C, This can be generalized into more than two points: θ 1 x θ k x k, where θ i + + θ k = 1 Is a affine combination. The set of all affine combinations of points in some setc R n is called Affine hull of C aff C = {θ 1 x θ k x k x 1,, x k C, θ i + + θ k = 1} Affine hull is the smallest affine set that contains C

16 2.1.3 Affine dimension and relative interior We define the affine dimension of a set C as the dimension of its affine hull As an example consider the unit circle in R 2, ie {x R 2 x x 1 2 = 1}, its affine hull is R 2 so its affine dimension is 2.

17 2.1.3 Affine dimension and relative interior

18 2.1.3 Affine dimension and relative interior

19 2.1.4 Convex sets A set C is a convex set if the line segment between any two points in C lies in C, i.e.,

20 2.1.4 Conex Combination Note: the difference between Convex combination and affine combination is, in convex combination, θ i 1

21 2.1.4 Conex hull

22 2.1.4 Conex hull The convex hull of a set C, denoted conv C, is the set of all convex combinations of points in C: Conv C = {θ 1 x θ k x k x 1,, x k C, θ i + + θ k = 1, θ i 0} Note: aff C = {θ 1 x θ k x k x 1,, x k C, θ i + + θ k = 1} As the name suggests, the convex hull conv C is always convex. It is the smallest convex set that contains C

23 2.1.5 Cones A set C is called a cone, or nonnegative homogeneous, if for every x C and θ 0,we have θx C. A set C is a convex cone if it is convex and a cone, which means that for any x 1, x 2 C and θ 1, θ 2 0, we have θ 1 x 1 + θ 2 x 2 C Note:, a ray can be a special case of cone

24 2.1.5 Conic combination

25 2.1.5 Conic Hull

26 2.2 Some important examples Hyperplanes and halfspaces Note:z = x y

27 2.2 Some important examples Hyperplanes

28 2.2.1 Hyperplanes

29 2.2.1 half space

30 2.2.1 half space

31 2.2.3 norm cones (second order cone)

32 2.2.3 norm cones

33 2.2.4 polyhedra

34

35 2.2.4 simplexes

36 2.2.5 The positive semidefinite cone

37 2.2.5 The positive semidefinite cone

38 2.2.5 The positive semidefinite cone (example)