Mathematical Programming and Research Methods (Part II)

Transcription

1 Mathematical Programming and Research Methods (Part II) 4. Convexity and Optimization Massimiliano Pontil (based on previous lecture by Andreas Argyriou) 1

2 Today s Plan Convex sets and functions Types of convex programs Algorithms Convex learning problems 2

3 Convexity Simple intuition, originates from simple geometric shapes (e.g. polygons) Convex Convex NOT Convexity plays a very important role in optimization 3

4 Convex Sets Definition 1. A set C IR d is called convex if λx + (1 λ)y C for all x, y C,λ [0,1] I.e. if x and y are in the set C, then the whole line segment {λx + (1 λ)y : λ [0,1]} lies also in C 4

5 Convex Sets (contd.) We call λx + (1 λ)y a convex combination of x and y whenever λ [0, 1] Generally, for any k IN, the sum k λ i x i i=1 is called a convex combination of the points x 1,...,x k IR d whenever λ 1,...,λ k 0 and k λ i = 1 i=1 5

6 Convex Sets (contd.) Clearly, if a set C is convex, all convex combinations of points in C (for all k IN) belong to C This set of all convex combinations is called the convex hull of C In general, given a set S IR d (S need not be convex), the convex hull of S is the set { k } k conv(s) := λ i x i : x i S, λ 1,...,λ k 0, λ i = 1, k IN i=1 i=1 The convex hull is the smallest convex set containing S 6

7 Convex Sets (contd.) S = conv(s) if and only if S is convex 7

8 Examples of Convex Sets Affine sets, i.e. sets of solutions of linear equations {x : Ax = b} Convex cones, i.e. sets containing any nonnegative combination k θ i x i, θ 1,...,θ k 0, of their points i=

9 Examples of Convex Sets (contd.) Hyperplanes, i.e. sets of the form {x : a x = b}, where a IR d, a 0, b IR (since they are special cases of affine sets) Halfspaces, i.e. sets of the form {x : a x b}, where a IR d, a 0, b IR 9

10 Polyhedra A polyhedron is a set defined by a finite number of affine equalities and inequalities P = {x : a i x b i, i = 1,...,m, c j x = d j, j = 1,...,p} where a 1,...,a m, c 1,...,c p IR d, b 1,...,b m,d 1,...,d p IR Polyhedra are convex sets 10

11 Polyhedra (contd.) A bounded polyhedron is called a polytope A set is a polytope if and only if it is the convex hull of a finite set of points 11

12 The Positive Semidefinite Cone We use the notations X 0 X 0 to denote that a d d matrix X is positive semidefinite and positive definite, respectively The sets and S d + := {X IR d d : X 0} S d ++ := {X IR d d : X 0} are called the positive semidefinite cone and positive definite cone, respectively 12

13 The Positive Semidefinite Cone (contd.) S d + and S d ++ are convex cones Proof. For any A, B S d +, θ 1, θ 2 0, the matrix θ 1 A + θ 2 B is psd. E.g. in IR 2 2, the positive semidefinite cone consists of the matrices of the form ( ) x y such that x, z 0, xz y y z

14 Norms A norm, denoted by, is a function from IR d to IR + such that 1. w 0, for all w IR d 2. w = 0 if and only if w = 0 3. aw = a w, for all a IR, w IR d (homogeneity) 4. w + z w + z, for all w, z IR d (triangle inequality) Important example: the L p norm where p [1,+ ) w p := ( d i=1 w i p ) 1 p 14

15 Norms (contd.) We have already seen the L 2 norm it is the regularizer in ridge regression, SVM etc. ( d )1 2 w 2 = = (w w) 2 1 i=1 w 2 i The L 1 norm w 1 = d w i i=1 Letting p +, we get the L norm w = d max i=1 w i 15

16 Norm Balls The unit ball for a norm is the set {w : w 1} L 1 unit ball L 2 unit ball L unit ball 16

17 Norm Balls (contd.) In general, any norm ball of the form {w : w c r} where c IR d and r 0 are the center and radius of the ball, respectively, is a convex set 17

18 Convex Functions A function f : IR d IR is called convex if for all x, y IR d, λ [0, 1] f(λx + (1 λ)y) λf(x) + (1 λ)f(y) Intuition: the line segment connecting any two points on the graph of f lies above the graph Convex Non-convex 18

19 Convex Functions (contd.) Similarly, a function f is called concave if f(λx + (1 λ)y) λf(x) + (1 λ)f(y) for all x, y IR d, λ [0, 1] or, equivalently, if f is convex 19

20 Strict Convexity A function f : IR d IR is called strictly convex if f(λx + (1 λ)y) < λf(x) + (1 λ)f(y) for all x, y IR d, x y, λ (0,1) I.e. the line segment connecting any two points on the graph of f lies strictly above the graph Equivalently, the graph of a strictly convex function does not contain any line segments 20

21 Strict Convexity (contd.) Not strictly convex Strictly convex 21

22 Jensen s Inequality If f is convex, it follows by induction that ( k ) k f λ i x i λ i f(x i ) i=1 i=1 for all k IN, x 1,...,x k IR d, λ 1,...,λ k 0, such that k i=1 λ i = 1 It can be generalised to integrals and expected values (it is used e.g. to derive the EM algorithm) 22

23 Continuity / Differentiability Theorem 1. If a function f is convex on IR d then it is also continuous on IR d. Proof. Not easy. There are convex functions which are not differentiable everywhere (and others which are) 23

24 Second Order Condition Theorem 2. Assume that a function f is twice differentiable on IR d. Then f is convex if and only if its Hessian is psd. 2 f(w) 0 for all w IR d Recall that the Hessian is the matrix formed by the second partial derivatives ( ) f d 2 f(w) := (w) w i w j i,j=1 Note: the condition 2 f 0 implies strict convexity, but the converse is not true see [Boyd & Vanderberghe] 24

25 Examples of Convex Functions Affine functions w a w + b are both convex and concave Exponentials, powers, log-sum-exp w IR e aw w IR w p w IR d log for any a IR for any p 1 ( d ) i=1 ew i 25

26 Examples of Convex Functions (contd.) Psd. quadratic functions f(w) = w Aw + a w + b for all w IR d where A S d +, a IR d, b IR 26

27 Examples of Convex Functions (contd.) Max function w max{w 1,...,w d } 27

28 Norms Are Convex Functions Every norm is a convex function Proof. Let w, z IR d and λ (0, 1). Then λw +(1 λ)z triangle inequality λw + (1 λ)z = homogeneity λ w +(1 λ) z No norm is strictly convex (to see this, select z = aw with a > 0,a 1) The square of every norm, w w 2, is a convex function Proof. Do it as an exercise. 28

29 Operations that Preserve Convexity Question: If f 1,...,f q : IR d IR are convex functions, which operations F can we apply so that f := F(f 1,...,f q ) is also convex? Nonnegative weighted sums where θ 1,...,θ q 0 f = q θ i f i i=1 Proof. Easy from the definition. 29

30 Operations that Preserve Convexity (contd.) Composition of a convex function with an affine map f(x) = g(ax + b) for all x IR d where g : IR n IR is convex, A IR n d is a matrix and b IR n Proof. Let x, y IR d,λ (0, 1). Then f(λx + (1 λ)y) = g(λax + (1 λ)ay + b) = g (λ(ax + b) + (1 λ)(ay + b)) λg(ax + b) + (1 λ)g(ay + b) = λf(x) + (1 λ)f(y) 30

31 Operations that Preserve Convexity (contd.) Maximum of convex functions f = max{f 1,...,f q } Proof (for q = 2, can be easily generalised to any q). Let x, y IR d, λ (0, 1). Then f(λx + (1 λ)y) = max{f 1 (λx + (1 λ)y), f 2 (λx + (1 λ)y)} max{λf 1 (x) + (1 λ)f 1 (y), λf 2 (x) + (1 λ)f 2 (y)} max{λf 1 (x), λf 2 (x)} + max{(1 λ)f 1 (y), (1 λ)f 2 (y)} Extends also to infinite sets of convex functions = λf(x) + (1 λ)f(y) 31

32 Proving Convexity Thus, to prove convexity of a function f, there are several approaches From the definition of convexity Compute the Hessian of f and show that it is psd. Decompose f as a nonnegative weighted sum of convex functions Decompose f as the composition of a convex and an affine function Decompose f as a maximum of convex functions 32

33 Examples Show that the function f(w) = w w = w 2 is strictly convex Proof. The Hessian of f equals 2 f(w) = 2I d which is positive definite. 33

34 Examples (contd.) Show that the quadratic function f(w) = w Aw + a w + b where A S d +, a IR d, b IR, is convex Proof. Write A = R R for some matrix R IR d d. Then w Aw = (Rw) (Rw), which is a composition of the convex function w w w and a linear map. The term w a w + b is affine and hence convex. Thus f is convex, as the sum of convex functions. Alternatively, we may compute the Hessian, which equals 2A, which is psd. 34

35 Proving Strict Convexity To prove strict convexity of a function f Use the definition (with strict inequality, x y and λ (0, 1)) Compute the Hessian of f and show that it is positive definite Decompose f as a sum of a convex and a strictly convex function (easy to prove this property) Note: When does the convex-affine composition operation apply? Example: the quadratic function f(w) = w Aw + a w + b is strictly convex if and only if A 0 (since the Hessian equals 2A) 35

36 Convex Optimization The problem min w IR d f(w) subject to f i (w) 0 i = 1,...,M (1) a j w = b j j = 1,...,P where f, f 1,...,f M are convex functions, is called a convex program or convex optimization problem The function f whose value we wish to minimise is called the objective function 36

37 Remarks The set of points w satisfying the constraints f i (w) 0, a j w = b j is called the feasible set The feasible set is convex: for any feasible points x, y, f i (λx + (1 λ)y) λf i (x) + (1 λ)f i (y) 0 and a j (λx + (1 λ)y)) = λb j + (1 λ)b j = b j In general, if we minimize a convex objective function over a convex set, the problem can be rewritten in form (1) (in principle at least; sometimes not practically possible) Many problems of interest can be rewritten in the form (1); they do not necessarily appear in that form however 37

38 Remarks (contd.) Minimum (1) does not always exist! (could be an infimum or could be ) The set of solutions (minimisers) of problem (1) is convex (easy to show) In particular, if the function f is strictly convex, then there is a unique minimiser (if any exists) 38

39 Remarks (contd.) There are no local minima outside the set of minimisers This is important because it implies that algorithms will not get stuck away from the solution(s) Thus, the great appeal of convex programs is that they can be solved! (many of them in polynomial time) 39

40 Examples where A, B 0 min w IR d w Aw + a w subject to w Bw + b w + c 0 d w = e min w IR d a w subject to b 1w c 1 b 2w c 2 d 1w = e 1 40

41 Regularization min w IR d m E ( ) w x i, y i + γ w 2 i=1 (R) Assume that E(,y) is a convex function for every y IR Then, problem (R) is a convex program; this program is unconstrained Indeed, the objective function is convex, as a sum of convex functions: E(w x i, y i ) is convex as a convex-affine composition and w 2 is also convex, as we have already seen It can be shown that the minimum exists (under mild assumptions on E) 41

42 Regularization (contd.) Example 1: ridge regression min w IR d m (y i w x i ) 2 + γ w 2 i=1 Convex program, since the function z (z y) 2 is convex for every y IR 42

43 Regularization (contd.) Example 2: we have seen (in Lecture 1) that SVM is equivalent to the regularization problem with γ = 1 2C min w IR d m max{1 y i (w x i ), 0} + γ w 2 i=1 This is a convex program, since the function z max{1 yz, 0} (the hinge loss) is convex for every y { 1, 1}; indeed, it is a maximum of convex (in particular, affine) functions 43

44 Regularization (contd.) How about the SVM primal 1 min w IR d 2 w 2 + C m i=1 subject to y i (w x i ) 1 ξ i, ξ i 0, for i = 1,...,m ξ i It is also easy to see that this is a convex program, but now the variables include w i and ξ i as well The objective function is convex (quadratic in w, linear in ξ i ); the functions 1 ξ i y i (w x i ) and ξ i in the inequality constraints are also convex 44

45 Regularization (contd.) Similarly, the SVM and ridge regression dual problems can be seen to be convex problems In general, the dual of regularization min c IR m m E ( ) c g i, y i + γ c Gc i=1 (C) is a convex problem (assuming as before that the loss function is convex) Indeed, the quadratic form c Gc is a convex function of c since the Gram matrix G is positive semidefinite 45

46 Regularization (contd.) min w IR d min c IR m m E ( ) w x i, y i + γ w 2 i=1 m E ( ) c g i, y i + γ c Gc i=1 (R) (C) Problem (R) has a unique solution; indeed, the term w 2 is strictly convex; hence the objective function is also strictly convex However, problem (C) has a unique solution only if G 0, i.e. only if the feature vectors φ(x i ) are linearly independent; otherwise, there are infinite optimal ĉ, but the corresponding ŵ = m i=1 ĉiφ(x i ) is unique 46

47 Convex Programs with Linear Equality Constraints The following special type of convex program can be solved using Lagrange multipliers min w IR d f(w) subject to a j w = b j j = 1,...,P (2) where f is a convex and differentiable function Set the gradient to zero: f(w) = P j=1 c ja j, for some c j IR; the set of solutions of this equation is the same as the set of minimisers of (2) (by a theorem); a dual problem can also be obtained 47

48 Important Convex Optimization Problems Linear Programming Quadratic Programming Semidefinite Programming Dedicated off-the-shelf algorithms exist for each of the above categories In machine learning, algorithms have been developed for special subtypes of such problems 48

49 Linear Programming (LP) min w IR d c w subject to d i w e i i = 1,...,M (3) a j w = b j j = 1,...,P The feasible set is a polyhedron (bounded or not) Problem (3) may have one, none, or infinite solutions Interesting fact: the dual problem is also a linear program 49

50 Linear Programming (contd.) It can be shown that the solution (if unique) will be one of the vertices The simplex algorithm is one of the oldest optimization algorithms (Dantzig in the 40s) 50

51 Linear Programming (contd.) Intuition of simplex: find a vertex to start from; from each vertex, move to a neighbour so that the objective function decreases; terminate if there is no such neighbour Time complexity: very good in almost all cases, but very bad (exponential) in the worst case In practice, very fast for typical problems and can be applied to large data sets Methods developed in the 80s (interior-point methods) have been applied to linear programming and are of polynomial-time complexity 51

52 Quadratic Programming (QP) min w IR d w Aw + c w subject to d i w e i i = 1,...,M (4) a j w = b j j = 1,...,P where A 0 If A 0, the solution (if any) is unique (due to strict convexity) The dual is also a quadratic program 52

53 Quadratic Programming (contd.) The idea behind simplex does not apply here; in fact, the minimiser could be anywhere in the feasible set (on the boundary or in the interior) The difficulty in solving QP is due to the fact that the solution may lie on the boundary of the feasible set 53

54 Interior-Point Methods Idea: change the objective function by adding to it a barrier function The barrier depends on the constraints and is parameterised by a parameter t Unconstrained minimisation of the barrier function gives a solution in the interior of the feasible set Changing t appropriately, the algorithm converges to the solution of (4) in polynomial time These methods can handle problems of reasonably large size 54

55 Ridge Regression as QP min w IR d m (y i w x i ) 2 + γ w 2 i=1 Ridge regression is an unconstrained QP Just need to solve a linear system using standard methods 55

56 SVM as QP 1 min w IR d 2 w 2 + C m i=1 ξ i max α IR m 1 2 α Aα + s. t. y i (w x i ) 1 ξ i s. t. 0 α i C ξ i 0 for i = 1,...,m m i=1 α i for i = 1,...,m SVM is a QP with inequality constraints The SVM dual is a QP with box constraints 56

57 Algorithms for SVM One approach to solve SVMs is with interior-point methods For large datasets (say m > 10 3 ) it is practically impossible to solve the dual problem with such methods (matrix A is dense!) A typical approach is to iteratively optimize wrt. an active set A of dual variables, fixing the rest. Set α = 0, choose q m and an active set A of q variables. We repeat until convergence the steps Solve the problem wrt. the variables in A Remove one variable from A which satisfies the KKT conditions and add one variable, if any, which violates the KKT conditions. If no such variable exists, stop 57

58 QCQP min w IR d w Aw + c w subject to w Bw + d i w e i i = 1,...,M where both A,B are psd. a j w = b j j = 1,...,P It is called a quadratically constrained quadratic program (QCQP) Larger family; contains the family of QP The dual problem is not a QCQP, in general 58

59 QCQP (contd.) The feasible set is the intersection of ellipsoids and/or a polyhedron It is faster to solve a QP with a dedicated method than to use a QCQP solver 59

60 SDP min w IR d c w subject to w 1 F w n F n + G 0 a j w = b j j = 1,...,P There is a linear matrix inequality (LMI) constraint Multiple LMIs reduce to an equivalent problem with just one LMI The dual problem of an SDP is also an SDP LP QP QCQP SDP (LPs, QPs, QCQPs can be rewritten as SDPs) 60

61 Bibliography Lectures available at: See also Boyd and Vandenberghe, Convex Optimization, 2004, Secs , 3.1.1, 3.1.5, 3.1.8, , 4.1.1, 4.2.1, 4.2.2, 4.3, 4.4,