Convex Optimization. Lijun Zhang Modification of

Transcription

1 Convex Optimization Lijun Zhang Modification of

2 Outline Introduction Convex Sets & Functions Convex Optimization Problems Duality Convex Optimization Methods Summary

3 Mathematical Optimization Optimization Problem

4 Applications Dimensionality Reduction (PCA) max Clustering (NMF) s. t. 1 min, s. t. 0, 0 Classification (SVM) min,

5 Least-squares The Problem Given, predict by Properties

6 Linear Programming The Problem Properties

7 Convex Optimization Problem The Problem Conditions

8 Convex Optimization Problem The Problem Properties

9 Nonlinear Optimization Definition The objective or constraint functions are not linear Could be convex or nonconvex

10 Outline Introduction Convex Sets & Functions Convex Optimization Problems Duality Convex Optimization Methods Summary

11 Affine Set

12 Convex Set

13 Convex Cone

14 Some Examples (1)

15 Some Examples (2)

16 Some Examples (3)

17 Operations that Preserve Convexity

18 Convex Functions

19 Examples on

20 Examples on and

21 Restriction of a Convex Function to a Line

22 First-order Conditions

23 Second-order Conditions

24 Examples

25 Operations that Preserve Convexity

26 Positive Weighted Sum & Composition with Affine Function

27 Pointwise Maximum Hinge loss: l max 0,1

28 The Conjugate Function

29 Examples

30 Outline Introduction Convex Sets & Functions Convex Optimization Problems Duality Convex Optimization Methods Summary

31 Optimization Problem in Standard Form

32 Optimal and Locally Optimal Points

33 Implicit Constraints

34 Convex Optimization Problem

35 Example

36 Local and Global Optima

37 Optimality Criterion for Differentiable

38 Examples

39 Popular Convex Problems Linear Program (LP) Linear-fractional Program Quadratic Program (QP) Quadratically Constrained Quadratic program (QCQP) Second-order Cone Programming (SOCP) Geometric Programming (GP) Semidefinite Program (SDP)

40 Outline Introduction Convex Sets & Functions Convex Optimization Problems Duality Convex Optimization Methods Summary

41 Lagrangian

42 Lagrangian

43 Lagrange Dual Function

44 Lagrange Dual Function

45 Least-norm Solution of Linear Equations

46 Lagrange Dual and Conjugate Function

47 The Dual Problem

48 Weak and Strong Duality

49 Weak and Strong Duality

50 Slater s Constraint Qualification

51 Complementary Slackness

52 Karush-Kuhn-Tucker (KKT) Conditions

53 KKT Conditions for Convex Problem

54 An Example SVM (1) The Optimization Problem Define the hinge loss as Its Conjugate Function is

55 An Example SVM (2) The Optimization Problem becomes It is Equivalent to The Lagrangian is

56 An Example SVM (3) The Lagrange Dual Function is Minimize one by one

57 An Example SVM (4) Finally, We Obtain The Dual Problem is

58 An Example SVM (5) Karush-Kuhn-Tucker (KKT) Conditions

59 An Example SVM (5) Karush-Kuhn-Tucker (KKT) Conditions Can be used to recover from

60 An Example SVM (5) Karush-Kuhn-Tucker (KKT) Conditions Can be used to recover from

61 Outline Introduction Convex Sets & Functions Convex Optimization Problems Duality Convex Optimization Methods Summary

62 More Assumptions Lipschitz continuous Strong Convexity, Smooth,

63 Performance Measure The Problem Convergence Rate Iteration Complexity

64 Gradient-based Methods The Convergence Rate GD Gradient Descent AGD Nesterov s Accelerated Gradient Descent [Nesterov, 2005, Nesterov, 2007, Tseng, 2008] EGD Epoch Gradient Descent [Hazan and Kale, 2011] SGD SGD with -suffix Averaging [Rakhlin et al., 2012]

65 Gradient Descent (1) Move along the opposite direction of gradients

66 Gradient Descent (2) Gradient Descent with Projection Projection Operator

67 Analysis (1)

68 Analysis (2)

69 Analysis (3)

70 A Key Step (1) Evaluate the Gradient or Subgradient Logit loss

71 A Key Step (1) Evaluate the Gradient or Subgradient Logit loss Hinge loss

72 A Key Step (2) Evaluate the Gradient or Subgradient Logit loss Hinge loss

73 A Key Step (3) Evaluate the Gradient or Subgradient Logit loss Hinge loss

74 Outline Introduction Convex Sets & Functions Convex Optimization Problems Duality Convex Optimization Methods Summary

75 Summary Convex Sets & Functions Definitions, Operations that Preserve Convexity Convex Optimization Problems Definitions, Optimality Criterion Duality Lagrange, Dual Problem, KKT Conditions Convex Optimization Methods Gradient-based Methods

76 Reference (1) Hazan, E. and Kale, S. (2011) Beyond the regret minimization barrier: an optimal algorithm for stochastic strongly-convex optimization. In Proceedings of the 24th Annual Conference on Learning Theory, pages Nesterov, Y. (2005) Smooth minimization of non-smooth functions. Mathematical Programming, 103(1): Nesterov, Y. (2007). Gradient methods for minimizing composite objective function. Core discussion papers.

77 Reference (2) Tseng, P. (2008). On acclerated proximal gradient methods for convexconcave optimization. Technical report, University of Washington. Boyd, S. and Vandenberghe, L. (2004). Convex Optimization. Cambridge University Press. Rakhlin, A., Shamir, O., and Sridharan, K. (2012) Making gradient descent optimal for strongly convex stochastic optimization. In Proceedings of the 29th International Conference on Machine Learning, pages