Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization. Author: Martin Jaggi Presenter: Zhongxing Peng
|
|
- Benjamin May
- 6 years ago
- Views:
Transcription
1 Revisiting Frank-Wolfe: Projection-Free Sparse Convex Optimization Author: Martin Jaggi Presenter: Zhongxing Peng
2 Outline 1. Theoretical Results 2. Applications
3 Outline 1. Theoretical Results 2. Applications
4 Problem Formulation Constrained convex optimization problem where 1 f is a convex function, 2 D is a compact convex set. A set is compact if it is closed and bounded. min f(x) (1) x D
5 Poor Man s Approach Since the function f is convex, its linear approximation must lie below the graph of the function. Basic properties and examples 69 f(y) f(x) + f(x) T (y x) (x, f(x)) Figure 3.2 If f is convex and differentiable, then f(x)+ f(x) T (y x) f(y) for all x, y dom f. is given by Define dual function w(x) as the minimum of the linear approximation to f at point x over the domain D. Thus, we have weak duality Ĩ C(x) = for each pair x, y D. { 0 x C x C. The convex function ĨC is called the indicator function of the set C. w(x) f(y) (2)
6 Subgradient Subgradient of a function g is a dsubgradient x f(x) is called of f (not a subgradient necessarily toconvex) f at x, and at xbelongs if to { } f(x) = d x f(y) X f(y) f(x) + f(x) g T (y + y x) x, d for x, all fory y D (3) ( (g, 1) supports epi f at (x, f(x))) PSfrag replacements f(x 1 ) + g T 1 (x x 1) f(x) f(x 2 ) + g T 2 (x x 2) f(x 2 ) + g T 3 (x x 2) x 1 x 2 g 2, g 3 are subgradients at x 2 ; g 1 is a subgradient at x 1
7 Dual Function For a given x D, and any choice of a subgradient d x f(x), we define a dual function as The property of weak-duality is Lemma 1 (Weak duality) w(x, d x ) = min y D f(x) + y x, d x (4) For all pairs x, y D, it holds that w(x, d x ) f(y). (5) If f is differentiable, we have w(x) = w(x, f(x)) = min f(x) + y x, f(x). (6) D
8 Duality Gap g(x, d x ) is called the duality gap at x, for the chosen d x, i.e. g(x, d x ) = f(x) w(x, d x ) = max y D x y, d x (7) which is a simple measure of approximation quality. According to Lemma 1, we have w(x, d x ) f(x ) (8) f(x) w(x, d x ) f(x) f(x ) (9) f(x) w(x, d x ) f(x) f(x ) (10) g(x, d x ) f(x) f(x ) 0. (11) where f(x) f(x ) is primal error. If f is differentiable, we have g(x) = g(x, f(x)) = max x y, f(x). (12) y D
9 Relation to Duality of Norms Observation 1 { } For optimization over any domain D = x X x 1 being the unit ball of some norm, the duality gap for the optimization problem min x D x is given by where is the dual norm of. g(x, d x ) = d x + x, d x (13) Proof. Since the dual norm is defined as x = sup y 1 y, x, consider duality gap satisfies g(x, d x ) = max y D x y, d x = max y D y, d x + x, d x. (14)
10 Frank-Wolfe Algorithm l l. Algorithm 1 Frank-Wolfe (1956) Let x (0) D for k = 0... K do Compute s := arg min s D s, f(x (k) ) Update x (k+1) := (1 γ)x (k) + γs, for γ := 2 k+2 end for In terms of conver- A step of this algorithm is illustrated in the inset figure: weat have a current position x, the algorithm considers where the linearization of the objective function, and moves x (k+1) =(1 γ)x (k) + γs towards a minimizer of =x (k) + this γ(s linear x (k) ) function (taken(15) f over the same domain). f(x)
11 Geometry Interpretation min f(x) x2d f(x) x D R d
12 Compare to Classical Gradient Descent Define the descent direction as follows y, f(x) < 0 (16) Frank Wolfe algorithm: always choose the best descent direction over the entire domain D Classical gradient descent: x (k+1) = x (k) + α f(x (k) ) (17) where α 0 is the step size. It only uses local information to determine the step-directions. It faces the risk of walking out of the domain D. It requires projection steps after each iteration.
13 Greedy on a Convex Set 22 Convex Optimization without Projection Steps Algorithm 1 Greedy on a Convex Set Input: Convex function f, convex set D, target accuracy ε Output: ε-approximate solution for problem (2.1) Pick an arbitrary starting point x (0) D for k = 0... do Let α := 2 k+2 Compute s := ExactLinear ( f(x (k) ), D ) {Solve the linearized primitive problem exactly} or Compute s := ApproxLinear ( ) f(x (k) ), D, αc f {Approximate the linearized primitive problem} Update x (k+1) := x (k) + α(s x (k) ) end for We call a point x X an ɛ-approximation if g(x, d x ) ɛ for some choice of subgradient d x f(x).
14 Linearized Optimization Primitive In the above algorithm, ExactLinear(c, D) minimizes the linear function x, c over D. It returns s by s = arg min y, c (18) y D We search for a point s that realizes the current duality gap g(x), that is the distance to the linear approximation, as follows g(x, d x ) = f(x) w(x, d x ) = max y D x y, d x. (19)
15 Linearized Optimization Primitive In the above algorithm, ApproxLinear(c, D, ɛ ) approximates the minimum of the linear function x, c over D. It returns s such that s, c = arg min y, c + ɛ. (20) y D For several applications, this can be done significantly more efficiently than the exact variant.
16 The Curvature The Curvature constant C f of a convex and differentiable function f, with respect to a compact domain D is defined as C f = sup x,s D α [0,1] y=x+α(s x) 1 (f(y) f(x) y x, f(x) ) (21) α2 It bounds the gap between f(y) and its linearization. f(y) f(x) y x, f(x) is known as the Bregman divergence. For linear functions f, it holds that C f = 0.
17 Convergence in Primal Error Theorem 1 (Primal Convergence) For each k 1, the iterative x (k) of the exact variant of Algorithm 1 satisfies f(x (k) ) f(x ) 4C f K + 2, (22) where x ( ) D is an optimal solution to problem (1). For the approximate variant of Algorithm 1, it holds that f(x (k) ) f(x ) 8C f K + 2. (23) In other words, both algorithm variants deliver a solution of primal error at most ɛ after O( 1 ɛ ) many iterations.
18 Duality Gap Theorem 2 (Primal-Dual Convergence) Let K = 4Cf ɛ. We run the exact variant of Algorithm 1 for K iterations (recall that the step-sizes are given by α (k) = 2 k+2, 0 k K), and then continue for another K + 1 iterations, now with the fixed step-size α (k) = 2 K+2 for K k 2K + 1. Then the algorithm has an iterate x (ˆk), K ˆk 2K + 1, with duality gap bounded by g(x (ˆk) ) ɛ (24) The same statement holds for the approximate variant of Algorithm 1, when setting K = instead. 8Cf ɛ
19 Choose Step-Size by Line-Search Instead of the fixed step-size α = 2 α [0, 1] by line-search. If we define ( f α = f The optimal α is α = arg min f α [0,1] x (k+1) (α) ) = f k+2 ( α f α = s x (k), f, we can find the optimal ( ) x (k) + α(s x (k) ). (25) ( ) x (k) + α(s x (k) ). (26) x (k+1) (α) ) = 0. (27)
20 Greedy on a Convex Set using Line-Search A Projection-Free First-Order Method for Convex Optimization 29 Algorithm 2 Greedy on a Convex Set, using Line-Search Input: Convex function f, convex set D, target accuracy ε Output: ε-approximate solution for problem (3.1) Pick an arbitrary starting point x (0) D for k = 0... do Compute s := ExactLinear ( f(x (k) ), D ) or ( Compute s := ApproxLinear Find the optimal step-size α := arg min α [0,1] Update x (k+1) := x (k) + α(s x (k) ) end for ) f(x (k) ), D, 2Cf f k+2 ( x (k) + α(s x (k) ) ) If this equation can be solved for α, then the optimal such α can directly be used as the step-size in Algorithm 1, and the convergence guarantee of Theorem 2.3 still holds. This is because the improvement in each step will
21 Relating the Curvation to the Hessian Matrix Hessian matrix of f: is the second derivative of f, i.e. 2 f. Second order Taylor-expansion of function f at point x is f(x + α(s x)) =f(x) + α(s x) T f(x) where z [x, y] D and Thus, we have + α2 2 (s x)t 2 f(z)(s x) (28) y = x + α(s x) (29) α(s x) = y x, (30) f(y) = f(x) + (y x) T f(x) (y x)t 2 f(z)(y x) (31)
22 Relating the Curvation to the Hessian Matrix f(y) f(x) y x, f(x) = 1 2 (y x)t 2 f(z)(y x) (32) According to the definition of C f C f = sup x,s D α [0,1] y=x+α(s x) 1 (f(y) f(x) y x, f(x) ), (33) α2 we have C f 1 sup x,y D 2 (y x)t 2 f(z)(y x) (34) z [x,y] D
23 Relating the Curvation to the Hessian Matrix Lemma 2 For any twice differentiable convex function f over a compact convex domain D, it holds that C f 1 2 diam(d)2 sup λ max ( 2 f(z)) (35) z D where diam( ) is the Euclidean diameter of the domain.
24 Relatinig the Curvation to the Hessian Matrix Proof. According to Cauchy-Schwarz inequality we have a, b a b (36) (y x) T 2 f(z)(y x) y x 2 2 f(z)(y x) 2 (37) y x 2 2 f(z)(y x) 2 2 (38) y x 2 y x f(z) spec (39) diam(d) 2 sup λ max ( 2 f(z)) (40) z D Aa where A spec = sup 2 a 0 a 2 is the spectral norm, which is the largest eigenvalue for a positive-semidefinite A.
25 VS. Lipschitz-Continuous Gradient Lemma 3 Let f be a convex and twice differential function, and assume that the gradient f is Lipschitz-continuous over the domain D with Lipschitz-constant L > 0. Then C f 1 2 diam(d)2 L (41) where Lipschitz-continuous means there exits L > 0 satisfies f(y) f(x) 2 L y x 2 (42)
26 Figure 2.2 Some simple convex and nonconvex sets. Left. The hexagon, Optimizing whichover includes its boundary Convex (shown darker), Hulls is convex. Middle. The kidney shaped set is not convex, since the line segment between the two points in the set shown as dots is not contained in the set. Right. The square contains some boundary points but not others, and is not convex. Figure 2.3 The convex hulls of two sets in R 2. Left. The convex hull of a set of fifteen points (shown as dots) is the pentagon (shown shaded). Right. The convex hull of the kidney shaped set in figure 2.2 is the shaded set. The convex hull of a set V, denoted conv(v ), is the set of all convex combination of points in V : { conv(v ) = θ 1 x θ k x k x i V, θ i 0, i = 1,, k, Roughly speaking, a set is convex if every point in the set can be seen by every other point, along an unobstructed straight path between them, where unobstructed means lying in the set. Every affine set is also convex, since it contains the entire line between any two distinct points in it, and therefore also the line segment between the points. Figure 2.2 illustrates some simple convex and nonconvex sets in R 2. } We call a point of the form θ 1x 1 + θ kx k, where θ θ k = 1 and θ i 0, i = 1,..., k, aθconvex 1 + combination + θ k of the = points 1 x 1,..., x k. As with affine sets, it can be shown that a set is convex if and only if it contains every convex combination of its points. A convex combination of points can be thought of as a mixture or weighted average of the points, with θ i the fraction of x i in the mixture. The convex hull conv(v ) is always convex.. (43) The convex hull of a set C, denoted conv C, is the set of all convex combinations of points in C: It is the smallest convex set that contains V. conv C = {θ 1x θ kx k x i C, θ i 0, i = 1,..., k, θ θ k = 1}.
27 Optimizing over Convex Hulls Consider the case where domain D is the convex hull of a set V, i.e. D = conv(v ). Lemma 4 (Linear Optimization over Convex Hulls) Let D = conv(v ) for any subset V X, and D compact. Then any linear function y y, c will attain its minimum and maximum over D at some vertex v V. In many applications, the set V is often much easier to describe than the full compact domain D, the result in the above lemma will be usefull to solve the linearized subproblem ExactLinear() more efficiently.
28 Outline 1. Theoretical Results 2. Applications
29 many gradient evaluations. We will crucially make use of the fact t every linear function attains its minimum at a vertex of the simplex Sparse Approximation (SA) over the Simplex Formally, for any vector c R n, it holds that min s T c = min c i. T s n i The unit property Simplex is defined easy to verify as in the special case here, but is also a di consequence of the small Lemma 2.8 which we have proven for gen convex hulls, n = if {x we accept R n xthat 0, the x unit 1 = simplex 1} is the convex (44) hull of unit basis vectors. We have obtained that the internal linearized primi Then, optimization can be solved onexactly Simplex byischoosing ExactLinear min (c, f(x) n ) := e i with i = arg min (45) c i. x n i Algorithm 3 Sparse Greedy on the Simplex Input: Convex function f, target accuracy ε Output: ε-approximate solution for problem (3.1) Set x (0) := e 1 for k = 0... do Compute i := arg min i ( f(x (k) ) ) i Let α := 2 k+2 Update x (k+1) := x (k) + α(e i x (k) ) end for
30 SA over the Simplex Theorem 3 (Convergence of Sparse Greedy on the Simplex) For each k 1, the iterate x (k) of the above algorithm satisfies f(x k ) f(x ) 4C f k + 2 (46) where x n is an optimal solution to problem in (45). Furthermore, for any ɛ > 0, after at most 2 4Cf ɛ + 1 = O( 1 ɛ ) many steps, it has an iterate x (k) of sparsity O( 1 ɛ ), satisfying g(x (k) ) ɛ.
31 SA over the Simplex Its duality gap is g(x, d x ) = f(x) w(x, d x ) (47) ( ) = f(x) min f(x) + y x, d x (48) y D = max y D x y, d x (49) = x T d x min y D yt d x (50)
32 Figure 2.2 Some simple convex and nonconvex sets. Left. The hexagon, SA over the which includes Simplex its boundary (shown darker), is convex. Middle. The kidney shaped set is not convex, since the line segment between the two points in the set shown as dots is not contained in the set. Right. The square contains some boundary points but not others, and is not convex. Figure 2.3 The convex hulls of two sets in R 2. Left. The convex hull of a set of fifteen points (shown as dots) is the pentagon (shown shaded). Right. The convex hull of the kidney shaped set in figure 2.2 is the shaded set. The convex hull of a set V, denoted conv(v ), is the set of all convex combination of points in V : { conv(v ) = θ 1 x θ k x k x i V, θ i 0, i = 1,, k, Roughly speaking, a set is convex if every point in the set can be seen by every other point, along an unobstructed straight path between them, where unobstructed means lying in the set. Every affine set is also convex, since it contains the entire line between any two distinct points in it, and therefore also the line segment between the points. Figure 2.2 illustrates some simple convex and nonconvex sets in R 2. } We call a point of the form θ 1x 1 + θ kx k, where θ θ k = 1 and θ i 0, i = 1,..., k, aθconvex 1 + combination + θ k of the = points 1 x 1,..., x k. As with affine sets, it can be shown that a set is convex if and only if it contains every convex combination of its points. A convex combination of points can be thought of as a mixture or weighted average of the points, with θ i the fraction of x i in the mixture. The convex hull conv(v ) is always convex.. (51) The convex hull of a set C, denoted conv C, is the set of all convex combinations of points in C: It is the smallest convex set that contains V. conv C = {θ 1x θ kx k x i C, θ i 0, i = 1,..., k, θ θ k = 1}.
33 SA over the Simplex Lemma 5 (Linear Optimization over Convex Hulls) Let D = conv(v ) for any subset V X, and D compact. Then any linear function y y, c will attain its minimum and maximum over D at some vertex v V. Proof. 1 Assume s D satisfies s, c = max y D y, c. 2 Represent s = i α iv i, where i α i = 1. 3 We have s, c = α i v i, c = α i v i, c (52) i i
34 SA over the Simplex Its duality gap is g(x, d x ) = f(x) w(x, d x ) (53) ( ) = f(x) min f(x) + y x, d x (54) y D Because, here we have then = max y D x y, d x (55) = x T d x min y D yt d x (56) D = n = {x R n x 0, x 1 = 1} (57) g(x) = g(x, f(x)) = x T f(x) min i ( f(x)) i (58)
35 SA over the Simplex: Example Consider the following problem min x n f(x) = x 2 2 = x T x, (59) whose gradient is f(x) = 2x. Then, we have f(y) f(x) y x, f(x) = y T y x T x 2(y x) T x (60) According to the definition of C f C f = sup x,s D α [0,1] y=x+α(s x) = y x 2 2 (61) = x + α(s x) x 2 2 (62) = α 2 s x 2 2 (63) 1 (f(y) f(x) y x, f(x) ) (64) α2 = sup x,s n x s 2 2 = diam( n ) 2 = 2 (65)
36 SA with Bounded l 1 -Norm The l 1 -ball is defined as n = {x R n x 1 1}. (66) Then, optimization with bounded l 1 -norm is Observation 2 For any vector c R n, it holds that where i arg max j c j. min x n f(x) (67) e i sgn(c i ) arg max y T c (68) y n
37 Observe that in each iteration, this algorithm only introd one new non-zero coordinate, so that the sparsity of x (k) is SAUsing withthis Bounded observation l 1 -Norm for c = f(x) in our general Alg therefore directly obtain the following simple method for l convex optimization, as depicted in the Algorithm 4. Algorithm 4 Sparse Greedy on the l 1 -Ball Input: Convex function f, target accuracy ε Output: ε-approximate solution for problem (3.3) Set x (0) := 0 for k = 0... do ( Compute i := arg max i f(x (k) ), i and let s := e i sign (( f(x (k) ) ) ) i Let α := 2 k+2 Update x (k+1) := x (k) + α(s x (k) ) end for
38 SA with Bounded l 1 -Norm Theorem 4 (Convergence of Sparse Greedy on the l 1 -Ball) For each k 1, the iterate x (k) of the above algorithm satisfies f(x k ) f(x ) 4C f k + 2 (69) where x n is an optimal solution to problem in (67). Furthermore, for any ɛ > 0, after at most 2 4Cf ɛ + 1 = O( 1 ɛ ) many steps, it has an iterate x (k) of sparsity O( 1 ɛ ), satisfying g(x (k) ) ɛ.
39 Optimization with Bounded l -Norm The l -ball is defined as n = {x R n x 1}. (70) Then, optimization with bounded l 1 -norm is Observation 3 For any vector c R n, it holds that where (s c ) i = sgn(c i ) { 1, 1}. min f(x) (71) x n s c arg max y T c (72) y n
40 Optimization with Bounded l -Norm Optimization with Bounded l -Norm Algorithm 5 Sparse Greedy on the Cube Input: Convex function f, target accuracy ε Output: ε-approximate solution for problem (3.4) Set x (0) := 0 for k = 0... do Compute the sign-vector s of f(x (k) ), such that s i = sign (( f(x (k) ) ) i), i = 1..n Let α := 2 k+2 Update x (k+1) := x (k) + α(s x (k) ) end for Theorem 3.7. For each k 1, the iterate x (k) of Algorithm 5
41 Optimization with Bounded l -Norm Theorem 5 (Convergence of Sparse Greedy on the l 1 -Ball) For each k 1, the iterate x (k) of the above algorithm satisfies f(x k ) f(x ) 4C f k + 2 (73) where x n is an optimal solution to problem in (71). Furthermore, for any ɛ > 0, after at most 2 4Cf ɛ many steps, it has an iterate x (k) with g(x (k) ) ɛ. + 1 = O( 1 ɛ )
42 Semidefinite Optimization (SDO) with Bounded Trace The set of positive semidefinite (PSD) matrices of unit trace is defined as S = {X R n n X 0, Tr(X) = 1}. (74) Then, optimization of PSD with bounded trace is min f(x) (75) x S
43 52 Applications to Sparse and Low Rank Approximation SDO with Bounded Trace trix M S n n. We will see that in practice for example Lanczos or the power method can be used as the internal optimizer ApproxLinear(). Algorithm 6 Hazan s Algorithm / Sparse Greedy for Bounded Trace Input: Convex function f with curvature C f, target accuracy ε Output: ε-approximate solution for problem (3.5) Set X (0) := vv T for an arbitrary unit length vector v R n. for k = 0... do Let α := 2 k+2 Compute v := v (k) = ApproxEV ( f(x (k) ), αc f ) Update X (k+1) := X (k) + α(vv T X (k) ) end for Here ApproxEV(A, ε ) is a subroutine that delivers an approximate ApproxEV(A, ɛ ) returns v, which satisfies smallest eigenvector (the eigenvector corresponding to the smallest eigenvalue) to a matrix A with the desired accuracy ε > 0. More precisely, it v T Av λ min (A) + ɛ must return a unit length vector v such that v T (76) Av λ min (A) + ε. Note that as our convex function f takes a symmetric matrix X as an argument, its gradients f(x) are given as symmetric matrices as well.
44 SDO with Bounded Trace Theorem 6 For each k 1, the iterate X (k) of the above algorithm satisfies f(x k ) f(x ) 8C f k + 2 (77) where X S is an optimal solution to problem in (75). Furthermore, for any ɛ > 0, after at most 2 8Cf ɛ + 1 = O( 1 ɛ ) many steps, it has an iterate X (k) of rank O ( 1 ɛ ), satisfying g(x (k) ) ɛ.
45 Nuclear Norm Regularization We consider the following problem which is equivalent to min f(z) + µ Z (78) Z R m n min f(z) (79) Z R m n, Z t 2 where f(z) is any differentiable convex function. is the nuclear norm (trace norm, Schatten 1-norm, the Ky Fan r-norm) of a matrix Z = r σ i (Z) (80) i=1 where σ i is the i-th largest singular value of Z, r is the rank of Z
46 Nuclear Norm Low Rank Frobenius norm: Z F = Z, Z = Tr(Z T Z) 1 2 ( m n r = = i=1 j=1 X 2 ij i=1 σ 2 i ) 1 2 (81) Operator norm (induced 2-norm): Z = λ max (Z T Z) = σ 1 (Z) (82) Nuclear norm: Z = r σ i (Z) (83) i=1
47 Nuclear Norm Low Rank According to the definitions of Z F = ( r i=1 σ 2 i we have following inequalities ) 1 2, Z = σ 1 (Z), Z = r σ i (Z), i=1 Z Z F Z r Z F r Z (84)
48 Nuclear Norm Low Rank C is a given convex set. The convex envelop of a (possible nonconvex) function f : C R is the largest convex function g such that g(z) f(z) for all z C. g is the best pointwise approximation to f. According to the above inequalities, if Z 1, we have rank(z) Z Z = rank(z) Z (85) Nuclear norm is the tightest convex lower bound of the rank function. Theorem 7 The convex envelop of rank(z) on the set {Z R m n : Z 1} is the nuclear norm Z.
49 Corollary 1 Any nuclear norm regularized problem min f(z) (86) Z R m n, Z t 2 is equivalent to a bounded trace convex problem min ˆf(X) X S (m+n) (m+n) s. t. Tr(X) = t X 0 (87) where ˆf is defined by ˆf(X) = f(z) and ( V Z X = Z T W where V S m m, W S n n. ) (88)
50 function argument X needs to be rescaled by 1 t in order to have unit trace, which however is a very simple operation in practical applications. Therefore, we can directly apply Hazan s Algorithm 6 for any max-norm regularized problem as follows: Nuclear Norm Regularization Algorithm 8 Nuclear Norm Regularized Solver Input: A convex nuclear norm regularized problem (4.2), target accuracy ε Output: ε-approximate solution for problem (4.2) 1. Consider the transformed symmetric problem for ˆf, as given by Corollary Adjust the function ˆf so that it first rescales its argument by t 3. Run Hazan s Algorithm 6 for ˆf(X) over the domain X S. Using our analysis of Algorithm 6 from Section 3.4.1, we see that Algorithm 8 runs in time near linear in the number N f of non-zero entries of the min f(x) gradient f. This makes it very attractive in min particular for ˆf(X) X S (n) (n) X S (m+n) (m+n) recommender systems applications and matrix completion, where f is a sparse matrix (same sparsity s. t. Tr(X) pattern= as 1, the observed entries), which s. t. we Tr(X) will = discuss t, in more detail in XSection (89) X 0 (90)
51 Nuclear Norm Regularization Corollary 2 After at most O ( ) 1 ɛ many iterations, Algorithm 8 obtains a solution that is ɛ close to the optimum of (86). The algorithm requires a total of Õ probability). ( Nf ɛ 1.5 ) arithmetic operations (with high
52 Thank You!
Shiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 2. Convex Optimization
Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 2 Convex Optimization Shiqian Ma, MAT-258A: Numerical Optimization 2 2.1. Convex Optimization General optimization problem: min f 0 (x) s.t., f i
More informationConvexity Theory and Gradient Methods
Convexity Theory and Gradient Methods Angelia Nedić angelia@illinois.edu ISE Department and Coordinated Science Laboratory University of Illinois at Urbana-Champaign Outline Convex Functions Optimality
More informationConvex sets and convex functions
Convex sets and convex functions Convex optimization problems Convex sets and their examples Separating and supporting hyperplanes Projections on convex sets Convex functions, conjugate functions ECE 602,
More informationConvex sets and convex functions
Convex sets and convex functions Convex optimization problems Convex sets and their examples Separating and supporting hyperplanes Projections on convex sets Convex functions, conjugate functions ECE 602,
More informationConvexity: an introduction
Convexity: an introduction Geir Dahl CMA, Dept. of Mathematics and Dept. of Informatics University of Oslo 1 / 74 1. Introduction 1. Introduction what is convexity where does it arise main concepts and
More informationLecture 2: August 31
10-725/36-725: Convex Optimization Fall 2016 Lecture 2: August 31 Lecturer: Lecturer: Ryan Tibshirani Scribes: Scribes: Lidan Mu, Simon Du, Binxuan Huang 2.1 Review A convex optimization problem is of
More informationConvex Optimization MLSS 2015
Convex Optimization MLSS 2015 Constantine Caramanis The University of Texas at Austin The Optimization Problem minimize : f (x) subject to : x X. The Optimization Problem minimize : f (x) subject to :
More informationLecture 2: August 29, 2018
10-725/36-725: Convex Optimization Fall 2018 Lecturer: Ryan Tibshirani Lecture 2: August 29, 2018 Scribes: Adam Harley Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have
More informationConvex Optimization. 2. Convex Sets. Prof. Ying Cui. Department of Electrical Engineering Shanghai Jiao Tong University. SJTU Ying Cui 1 / 33
Convex Optimization 2. Convex Sets Prof. Ying Cui Department of Electrical Engineering Shanghai Jiao Tong University 2018 SJTU Ying Cui 1 / 33 Outline Affine and convex sets Some important examples Operations
More informationConvex Optimization / Homework 2, due Oct 3
Convex Optimization 0-725/36-725 Homework 2, due Oct 3 Instructions: You must complete Problems 3 and either Problem 4 or Problem 5 (your choice between the two) When you submit the homework, upload a
More informationCOM Optimization for Communications Summary: Convex Sets and Convex Functions
1 Convex Sets Affine Sets COM524500 Optimization for Communications Summary: Convex Sets and Convex Functions A set C R n is said to be affine if A point x 1, x 2 C = θx 1 + (1 θ)x 2 C, θ R (1) y = k θ
More informationLecture 4: Convexity
10-725: Convex Optimization Fall 2013 Lecture 4: Convexity Lecturer: Barnabás Póczos Scribes: Jessica Chemali, David Fouhey, Yuxiong Wang Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer:
More informationLecture 19: Convex Non-Smooth Optimization. April 2, 2007
: Convex Non-Smooth Optimization April 2, 2007 Outline Lecture 19 Convex non-smooth problems Examples Subgradients and subdifferentials Subgradient properties Operations with subgradients and subdifferentials
More informationLecture 2 September 3
EE 381V: Large Scale Optimization Fall 2012 Lecture 2 September 3 Lecturer: Caramanis & Sanghavi Scribe: Hongbo Si, Qiaoyang Ye 2.1 Overview of the last Lecture The focus of the last lecture was to give
More informationConvexity I: Sets and Functions
Convexity I: Sets and Functions Lecturer: Aarti Singh Co-instructor: Pradeep Ravikumar Convex Optimization 10-725/36-725 See supplements for reviews of basic real analysis basic multivariate calculus basic
More information11 Linear Programming
11 Linear Programming 11.1 Definition and Importance The final topic in this course is Linear Programming. We say that a problem is an instance of linear programming when it can be effectively expressed
More informationAspects of Convex, Nonconvex, and Geometric Optimization (Lecture 1) Suvrit Sra Massachusetts Institute of Technology
Aspects of Convex, Nonconvex, and Geometric Optimization (Lecture 1) Suvrit Sra Massachusetts Institute of Technology Hausdorff Institute for Mathematics (HIM) Trimester: Mathematics of Signal Processing
More informationLecture 2: August 29, 2018
10-725/36-725: Convex Optimization Fall 2018 Lecturer: Ryan Tibshirani Lecture 2: August 29, 2018 Scribes: Yingjing Lu, Adam Harley, Ruosong Wang Note: LaTeX template courtesy of UC Berkeley EECS dept.
More informationContents. I Basics 1. Copyright by SIAM. Unauthorized reproduction of this article is prohibited.
page v Preface xiii I Basics 1 1 Optimization Models 3 1.1 Introduction... 3 1.2 Optimization: An Informal Introduction... 4 1.3 Linear Equations... 7 1.4 Linear Optimization... 10 Exercises... 12 1.5
More informationTutorial on Convex Optimization for Engineers
Tutorial on Convex Optimization for Engineers M.Sc. Jens Steinwandt Communications Research Laboratory Ilmenau University of Technology PO Box 100565 D-98684 Ilmenau, Germany jens.steinwandt@tu-ilmenau.de
More informationCMU-Q Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization. Teacher: Gianni A. Di Caro
CMU-Q 15-381 Lecture 9: Optimization II: Constrained,Unconstrained Optimization Convex optimization Teacher: Gianni A. Di Caro GLOBAL FUNCTION OPTIMIZATION Find the global maximum of the function f x (and
More informationConvex Optimization - Chapter 1-2. Xiangru Lian August 28, 2015
Convex Optimization - Chapter 1-2 Xiangru Lian August 28, 2015 1 Mathematical optimization minimize f 0 (x) s.t. f j (x) 0, j=1,,m, (1) x S x. (x 1,,x n ). optimization variable. f 0. R n R. objective
More informationCS 435, 2018 Lecture 2, Date: 1 March 2018 Instructor: Nisheeth Vishnoi. Convex Programming and Efficiency
CS 435, 2018 Lecture 2, Date: 1 March 2018 Instructor: Nisheeth Vishnoi Convex Programming and Efficiency In this lecture, we formalize convex programming problem, discuss what it means to solve it efficiently
More informationMathematical and Algorithmic Foundations Linear Programming and Matchings
Adavnced Algorithms Lectures Mathematical and Algorithmic Foundations Linear Programming and Matchings Paul G. Spirakis Department of Computer Science University of Patras and Liverpool Paul G. Spirakis
More informationSparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual
Sparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know learn the proximal
More informationAffine function. suppose f : R n R m is affine (f(x) =Ax + b with A R m n, b R m ) the image of a convex set under f is convex
Affine function suppose f : R n R m is affine (f(x) =Ax + b with A R m n, b R m ) the image of a convex set under f is convex S R n convex = f(s) ={f(x) x S} convex the inverse image f 1 (C) of a convex
More informationMathematical Programming and Research Methods (Part II)
Mathematical Programming and Research Methods (Part II) 4. Convexity and Optimization Massimiliano Pontil (based on previous lecture by Andreas Argyriou) 1 Today s Plan Convex sets and functions Types
More informationPOLYHEDRAL GEOMETRY. Convex functions and sets. Mathematical Programming Niels Lauritzen Recall that a subset C R n is convex if
POLYHEDRAL GEOMETRY Mathematical Programming Niels Lauritzen 7.9.2007 Convex functions and sets Recall that a subset C R n is convex if {λx + (1 λ)y 0 λ 1} C for every x, y C and 0 λ 1. A function f :
More information2. Convex sets. x 1. x 2. affine set: contains the line through any two distinct points in the set
2. Convex sets Convex Optimization Boyd & Vandenberghe affine and convex sets some important examples operations that preserve convexity generalized inequalities separating and supporting hyperplanes dual
More informationCharacterizing Improving Directions Unconstrained Optimization
Final Review IE417 In the Beginning... In the beginning, Weierstrass's theorem said that a continuous function achieves a minimum on a compact set. Using this, we showed that for a convex set S and y not
More informationConvex optimization algorithms for sparse and low-rank representations
Convex optimization algorithms for sparse and low-rank representations Lieven Vandenberghe, Hsiao-Han Chao (UCLA) ECC 2013 Tutorial Session Sparse and low-rank representation methods in control, estimation,
More informationIntroduction to Modern Control Systems
Introduction to Modern Control Systems Convex Optimization, Duality and Linear Matrix Inequalities Kostas Margellos University of Oxford AIMS CDT 2016-17 Introduction to Modern Control Systems November
More informationLecture 19 Subgradient Methods. November 5, 2008
Subgradient Methods November 5, 2008 Outline Lecture 19 Subgradients and Level Sets Subgradient Method Convergence and Convergence Rate Convex Optimization 1 Subgradients and Level Sets A vector s is a
More informationLecture 2 - Introduction to Polytopes
Lecture 2 - Introduction to Polytopes Optimization and Approximation - ENS M1 Nicolas Bousquet 1 Reminder of Linear Algebra definitions Let x 1,..., x m be points in R n and λ 1,..., λ m be real numbers.
More informationConic Duality. yyye
Conic Linear Optimization and Appl. MS&E314 Lecture Note #02 1 Conic Duality Yinyu Ye Department of Management Science and Engineering Stanford University Stanford, CA 94305, U.S.A. http://www.stanford.edu/
More information2. Convex sets. affine and convex sets. some important examples. operations that preserve convexity. generalized inequalities
2. Convex sets Convex Optimization Boyd & Vandenberghe affine and convex sets some important examples operations that preserve convexity generalized inequalities separating and supporting hyperplanes dual
More informationLecture 5: Properties of convex sets
Lecture 5: Properties of convex sets Rajat Mittal IIT Kanpur This week we will see properties of convex sets. These properties make convex sets special and are the reason why convex optimization problems
More informationSupport Vector Machines.
Support Vector Machines srihari@buffalo.edu SVM Discussion Overview 1. Overview of SVMs 2. Margin Geometry 3. SVM Optimization 4. Overlapping Distributions 5. Relationship to Logistic Regression 6. Dealing
More informationCS675: Convex and Combinatorial Optimization Spring 2018 Consequences of the Ellipsoid Algorithm. Instructor: Shaddin Dughmi
CS675: Convex and Combinatorial Optimization Spring 2018 Consequences of the Ellipsoid Algorithm Instructor: Shaddin Dughmi Outline 1 Recapping the Ellipsoid Method 2 Complexity of Convex Optimization
More informationNumerical Optimization
Convex Sets Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Let x 1, x 2 R n, x 1 x 2. Line and line segment Line passing through x 1 and x 2 : {y
More informationSection 5 Convex Optimisation 1. W. Dai (IC) EE4.66 Data Proc. Convex Optimisation page 5-1
Section 5 Convex Optimisation 1 W. Dai (IC) EE4.66 Data Proc. Convex Optimisation 1 2018 page 5-1 Convex Combination Denition 5.1 A convex combination is a linear combination of points where all coecients
More informationConvex Optimization. Convex Sets. ENSAE: Optimisation 1/24
Convex Optimization Convex Sets ENSAE: Optimisation 1/24 Today affine and convex sets some important examples operations that preserve convexity generalized inequalities separating and supporting hyperplanes
More informationWeek 5. Convex Optimization
Week 5. Convex Optimization Lecturer: Prof. Santosh Vempala Scribe: Xin Wang, Zihao Li Feb. 9 and, 206 Week 5. Convex Optimization. The convex optimization formulation A general optimization problem is
More informationDM545 Linear and Integer Programming. Lecture 2. The Simplex Method. Marco Chiarandini
DM545 Linear and Integer Programming Lecture 2 The Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Outline 1. 2. 3. 4. Standard Form Basic Feasible Solutions
More informationConvex Sets. CSCI5254: Convex Optimization & Its Applications. subspaces, affine sets, and convex sets. operations that preserve convexity
CSCI5254: Convex Optimization & Its Applications Convex Sets subspaces, affine sets, and convex sets operations that preserve convexity generalized inequalities separating and supporting hyperplanes dual
More informationMachine Learning for Signal Processing Lecture 4: Optimization
Machine Learning for Signal Processing Lecture 4: Optimization 13 Sep 2015 Instructor: Bhiksha Raj (slides largely by Najim Dehak, JHU) 11-755/18-797 1 Index 1. The problem of optimization 2. Direct optimization
More informationOptimality certificates for convex minimization and Helly numbers
Optimality certificates for convex minimization and Helly numbers Amitabh Basu Michele Conforti Gérard Cornuéjols Robert Weismantel Stefan Weltge May 10, 2017 Abstract We consider the problem of minimizing
More informationConvex Sets (cont.) Convex Functions
Convex Sets (cont.) Convex Functions Optimization - 10725 Carlos Guestrin Carnegie Mellon University February 27 th, 2008 1 Definitions of convex sets Convex v. Non-convex sets Line segment definition:
More informationConvexization in Markov Chain Monte Carlo
in Markov Chain Monte Carlo 1 IBM T. J. Watson Yorktown Heights, NY 2 Department of Aerospace Engineering Technion, Israel August 23, 2011 Problem Statement MCMC processes in general are governed by non
More informationFinding Euclidean Distance to a Convex Cone Generated by a Large Number of Discrete Points
Submitted to Operations Research manuscript (Please, provide the manuscript number!) Finding Euclidean Distance to a Convex Cone Generated by a Large Number of Discrete Points Ali Fattahi Anderson School
More informationCS675: Convex and Combinatorial Optimization Spring 2018 Convex Sets. Instructor: Shaddin Dughmi
CS675: Convex and Combinatorial Optimization Spring 2018 Convex Sets Instructor: Shaddin Dughmi Outline 1 Convex sets, Affine sets, and Cones 2 Examples of Convex Sets 3 Convexity-Preserving Operations
More informationEE/ACM Applications of Convex Optimization in Signal Processing and Communications Lecture 6
EE/ACM 150 - Applications of Convex Optimization in Signal Processing and Communications Lecture 6 Andre Tkacenko Signal Processing Research Group Jet Propulsion Laboratory April 19, 2012 Andre Tkacenko
More informationA General Greedy Approximation Algorithm with Applications
A General Greedy Approximation Algorithm with Applications Tong Zhang IBM T.J. Watson Research Center Yorktown Heights, NY 10598 tzhang@watson.ibm.com Abstract Greedy approximation algorithms have been
More informationLec13p1, ORF363/COS323
Lec13 Page 1 Lec13p1, ORF363/COS323 This lecture: Semidefinite programming (SDP) Definition and basic properties Review of positive semidefinite matrices SDP duality SDP relaxations for nonconvex optimization
More informationResearch Interests Optimization:
Mitchell: Research interests 1 Research Interests Optimization: looking for the best solution from among a number of candidates. Prototypical optimization problem: min f(x) subject to g(x) 0 x X IR n Here,
More informationCS675: Convex and Combinatorial Optimization Fall 2014 Convex Functions. Instructor: Shaddin Dughmi
CS675: Convex and Combinatorial Optimization Fall 2014 Convex Functions Instructor: Shaddin Dughmi Outline 1 Convex Functions 2 Examples of Convex and Concave Functions 3 Convexity-Preserving Operations
More informationAn accelerated proximal gradient algorithm for nuclear norm regularized least squares problems
An accelerated proximal gradient algorithm for nuclear norm regularized least squares problems Kim-Chuan Toh Sangwoon Yun Abstract The affine rank minimization problem, which consists of finding a matrix
More informationConvex Optimization. Erick Delage, and Ashutosh Saxena. October 20, (a) (b) (c)
Convex Optimization (for CS229) Erick Delage, and Ashutosh Saxena October 20, 2006 1 Convex Sets Definition: A set G R n is convex if every pair of point (x, y) G, the segment beteen x and y is in A. More
More informationIntroduction to optimization
Introduction to optimization G. Ferrari Trecate Dipartimento di Ingegneria Industriale e dell Informazione Università degli Studi di Pavia Industrial Automation Ferrari Trecate (DIS) Optimization Industrial
More informationApplied Lagrange Duality for Constrained Optimization
Applied Lagrange Duality for Constrained Optimization Robert M. Freund February 10, 2004 c 2004 Massachusetts Institute of Technology. 1 1 Overview The Practical Importance of Duality Review of Convexity
More informationLecture 12: Feasible direction methods
Lecture 12 Lecture 12: Feasible direction methods Kin Cheong Sou December 2, 2013 TMA947 Lecture 12 Lecture 12: Feasible direction methods 1 / 1 Feasible-direction methods, I Intro Consider the problem
More informationSimplex Algorithm in 1 Slide
Administrivia 1 Canonical form: Simplex Algorithm in 1 Slide If we do pivot in A r,s >0, where c s
More informationLinear Programming. Larry Blume. Cornell University & The Santa Fe Institute & IHS
Linear Programming Larry Blume Cornell University & The Santa Fe Institute & IHS Linear Programs The general linear program is a constrained optimization problem where objectives and constraints are all
More informationLecture 5: Duality Theory
Lecture 5: Duality Theory Rajat Mittal IIT Kanpur The objective of this lecture note will be to learn duality theory of linear programming. We are planning to answer following questions. What are hyperplane
More informationOptimality certificates for convex minimization and Helly numbers
Optimality certificates for convex minimization and Helly numbers Amitabh Basu Michele Conforti Gérard Cornuéjols Robert Weismantel Stefan Weltge October 20, 2016 Abstract We consider the problem of minimizing
More informationMath 5593 Linear Programming Lecture Notes
Math 5593 Linear Programming Lecture Notes Unit II: Theory & Foundations (Convex Analysis) University of Colorado Denver, Fall 2013 Topics 1 Convex Sets 1 1.1 Basic Properties (Luenberger-Ye Appendix B.1).........................
More informationConstrained optimization
Constrained optimization A general constrained optimization problem has the form where The Lagrangian function is given by Primal and dual optimization problems Primal: Dual: Weak duality: Strong duality:
More information6 Randomized rounding of semidefinite programs
6 Randomized rounding of semidefinite programs We now turn to a new tool which gives substantially improved performance guarantees for some problems We now show how nonlinear programming relaxations can
More informationDistance-to-Solution Estimates for Optimization Problems with Constraints in Standard Form
Distance-to-Solution Estimates for Optimization Problems with Constraints in Standard Form Philip E. Gill Vyacheslav Kungurtsev Daniel P. Robinson UCSD Center for Computational Mathematics Technical Report
More information60 2 Convex sets. {x a T x b} {x ã T x b}
60 2 Convex sets Exercises Definition of convexity 21 Let C R n be a convex set, with x 1,, x k C, and let θ 1,, θ k R satisfy θ i 0, θ 1 + + θ k = 1 Show that θ 1x 1 + + θ k x k C (The definition of convexity
More informationSupport Vector Machines.
Support Vector Machines srihari@buffalo.edu SVM Discussion Overview. Importance of SVMs. Overview of Mathematical Techniques Employed 3. Margin Geometry 4. SVM Training Methodology 5. Overlapping Distributions
More informationORIE 6300 Mathematical Programming I November 13, Lecture 23. max b T y. x 0 s 0. s.t. A T y + s = c
ORIE 63 Mathematical Programming I November 13, 214 Lecturer: David P. Williamson Lecture 23 Scribe: Mukadder Sevi Baltaoglu 1 Interior Point Methods Consider the standard primal and dual linear programs:
More informationEfficient Iterative Semi-supervised Classification on Manifold
. Efficient Iterative Semi-supervised Classification on Manifold... M. Farajtabar, H. R. Rabiee, A. Shaban, A. Soltani-Farani Sharif University of Technology, Tehran, Iran. Presented by Pooria Joulani
More informationCS 473: Algorithms. Ruta Mehta. Spring University of Illinois, Urbana-Champaign. Ruta (UIUC) CS473 1 Spring / 36
CS 473: Algorithms Ruta Mehta University of Illinois, Urbana-Champaign Spring 2018 Ruta (UIUC) CS473 1 Spring 2018 1 / 36 CS 473: Algorithms, Spring 2018 LP Duality Lecture 20 April 3, 2018 Some of the
More information2. Optimization problems 6
6 2.1 Examples... 7... 8 2.3 Convex sets and functions... 9 2.4 Convex optimization problems... 10 2.1 Examples 7-1 An (NP-) optimization problem P 0 is defined as follows Each instance I P 0 has a feasibility
More information16.410/413 Principles of Autonomy and Decision Making
16.410/413 Principles of Autonomy and Decision Making Lecture 17: The Simplex Method Emilio Frazzoli Aeronautics and Astronautics Massachusetts Institute of Technology November 10, 2010 Frazzoli (MIT)
More informationBilinear Programming
Bilinear Programming Artyom G. Nahapetyan Center for Applied Optimization Industrial and Systems Engineering Department University of Florida Gainesville, Florida 32611-6595 Email address: artyom@ufl.edu
More informationDivide and Conquer Kernel Ridge Regression
Divide and Conquer Kernel Ridge Regression Yuchen Zhang John Duchi Martin Wainwright University of California, Berkeley COLT 2013 Yuchen Zhang (UC Berkeley) Divide and Conquer KRR COLT 2013 1 / 15 Problem
More informationGate Sizing by Lagrangian Relaxation Revisited
Gate Sizing by Lagrangian Relaxation Revisited Jia Wang, Debasish Das, and Hai Zhou Electrical Engineering and Computer Science Northwestern University Evanston, Illinois, United States October 17, 2007
More information1. Introduction. performance of numerical methods. complexity bounds. structural convex optimization. course goals and topics
1. Introduction EE 546, Univ of Washington, Spring 2016 performance of numerical methods complexity bounds structural convex optimization course goals and topics 1 1 Some course info Welcome to EE 546!
More informationApproximation Algorithms
Chapter 8 Approximation Algorithms Algorithm Theory WS 2016/17 Fabian Kuhn Approximation Algorithms Optimization appears everywhere in computer science We have seen many examples, e.g.: scheduling jobs
More informationFAQs on Convex Optimization
FAQs on Convex Optimization. What is a convex programming problem? A convex programming problem is the minimization of a convex function on a convex set, i.e. min f(x) X C where f: R n R and C R n. f is
More informationSimplicial Hyperbolic Surfaces
Simplicial Hyperbolic Surfaces Talk by Ken Bromberg August 21, 2007 1-Lipschitz Surfaces- In this lecture we will discuss geometrically meaningful ways of mapping a surface S into a hyperbolic manifold
More informationMVE165/MMG630, Applied Optimization Lecture 8 Integer linear programming algorithms. Ann-Brith Strömberg
MVE165/MMG630, Integer linear programming algorithms Ann-Brith Strömberg 2009 04 15 Methods for ILP: Overview (Ch. 14.1) Enumeration Implicit enumeration: Branch and bound Relaxations Decomposition methods:
More informationIntroduction to Convex Optimization. Prof. Daniel P. Palomar
Introduction to Convex Optimization Prof. Daniel P. Palomar The Hong Kong University of Science and Technology (HKUST) MAFS6010R- Portfolio Optimization with R MSc in Financial Mathematics Fall 2018-19,
More informationEllipsoid Algorithm :Algorithms in the Real World. Ellipsoid Algorithm. Reduction from general case
Ellipsoid Algorithm 15-853:Algorithms in the Real World Linear and Integer Programming II Ellipsoid algorithm Interior point methods First polynomial-time algorithm for linear programming (Khachian 79)
More informationPoint-Set Topology 1. TOPOLOGICAL SPACES AND CONTINUOUS FUNCTIONS
Point-Set Topology 1. TOPOLOGICAL SPACES AND CONTINUOUS FUNCTIONS Definition 1.1. Let X be a set and T a subset of the power set P(X) of X. Then T is a topology on X if and only if all of the following
More informationAMS : Combinatorial Optimization Homework Problems - Week V
AMS 553.766: Combinatorial Optimization Homework Problems - Week V For the following problems, A R m n will be m n matrices, and b R m. An affine subspace is the set of solutions to a a system of linear
More information15.082J and 6.855J. Lagrangian Relaxation 2 Algorithms Application to LPs
15.082J and 6.855J Lagrangian Relaxation 2 Algorithms Application to LPs 1 The Constrained Shortest Path Problem (1,10) 2 (1,1) 4 (2,3) (1,7) 1 (10,3) (1,2) (10,1) (5,7) 3 (12,3) 5 (2,2) 6 Find the shortest
More informationConvex Optimization Lecture 2
Convex Optimization Lecture 2 Today: Convex Analysis Center-of-mass Algorithm 1 Convex Analysis Convex Sets Definition: A set C R n is convex if for all x, y C and all 0 λ 1, λx + (1 λ)y C Operations that
More informationKernel Methods & Support Vector Machines
& Support Vector Machines & Support Vector Machines Arvind Visvanathan CSCE 970 Pattern Recognition 1 & Support Vector Machines Question? Draw a single line to separate two classes? 2 & Support Vector
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms - C) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationLecture 11: Clustering and the Spectral Partitioning Algorithm A note on randomized algorithm, Unbiased estimates
CSE 51: Design and Analysis of Algorithms I Spring 016 Lecture 11: Clustering and the Spectral Partitioning Algorithm Lecturer: Shayan Oveis Gharan May nd Scribe: Yueqi Sheng Disclaimer: These notes have
More informationInteger Programming Theory
Integer Programming Theory Laura Galli October 24, 2016 In the following we assume all functions are linear, hence we often drop the term linear. In discrete optimization, we seek to find a solution x
More information2. Convex sets. affine and convex sets. some important examples. operations that preserve convexity. generalized inequalities
2. Convex sets Convex Optimization Boyd & Vandenberghe affine and convex sets some important examples operations that preserve convexity generalized inequalities separating and supporting hyperplanes dual
More informationConvex Optimization M2
Convex Optimization M2 Lecture 1 A. d Aspremont. Convex Optimization M2. 1/49 Today Convex optimization: introduction Course organization and other gory details... Convex sets, basic definitions. A. d
More informationLagrangian Relaxation: An overview
Discrete Math for Bioinformatics WS 11/12:, by A. Bockmayr/K. Reinert, 22. Januar 2013, 13:27 4001 Lagrangian Relaxation: An overview Sources for this lecture: D. Bertsimas and J. Tsitsiklis: Introduction
More informationSupport Vector Machines
Support Vector Machines SVM Discussion Overview. Importance of SVMs. Overview of Mathematical Techniques Employed 3. Margin Geometry 4. SVM Training Methodology 5. Overlapping Distributions 6. Dealing
More informationPRIMAL-DUAL INTERIOR POINT METHOD FOR LINEAR PROGRAMMING. 1. Introduction
PRIMAL-DUAL INTERIOR POINT METHOD FOR LINEAR PROGRAMMING KELLER VANDEBOGERT AND CHARLES LANNING 1. Introduction Interior point methods are, put simply, a technique of optimization where, given a problem
More information11.1 Facility Location
CS787: Advanced Algorithms Scribe: Amanda Burton, Leah Kluegel Lecturer: Shuchi Chawla Topic: Facility Location ctd., Linear Programming Date: October 8, 2007 Today we conclude the discussion of local
More information