Projection onto the probability simplex: An efficient algorithm with a simple proof, and an application
|
|
- Frederick Randolph Stevens
- 5 years ago
- Views:
Transcription
1 Proection onto the probability simplex: An efficient algorithm with a simple proof, and an application Weiran Wang Miguel Á. Carreira-Perpiñán Electrical Engineering and Computer Science, University of California, Merced {wwang5,mcarreira-perpinan}@ucmerced.edu September 3, 203 Abstract We provide an elementary proof of a simple, efficient algorithm for computing the Euclidean proection of a point onto the probability simplex. We also show an application in Laplacian K-modes clustering. Proection onto the probability simplex Consider the problem of computing the Euclidean proection of a point y = [y,...,y D ] R D onto the probability simplex, which is defined by the following optimization problem: x R D 2 x y 2 a s.t. x = x 0. This is a quadratic program and the obective function is strictly convex, so there is a unique solution which we denote by x = [x,...,x D ] with a slight abuse of notation. 2 Algorithm The following ODlogD algorithm finds the solution x to the problem: Algorithm Euclidean proection of a vector onto the probability simplex. Input: y R D Sort y into u: u u 2 u D Find ρ = max{ D: u + u i > 0} Define λ = ρ ρ u i Output: x s.t. x i = max{y i +λ,0}, i =,...,D. The complexity of the algorithm is doated by the cost of sorting the components of y. The algorithm is not iterative and identifies the active set exactly after at most D steps. It can be easily implemented see section 4. The algorithm has the following geometric interpretation. The solution can be written as x i = max{y i + λ,0}, i =,...,D, where λ is chosen such that D x i =. Place the values y,...,y D as points on the X axis. Then the solution is given by a rigid shift of the points such that the points to the right of the Y axis sum to. The pseudocode above appears in Duchi et al. 2008, although earlier papers Brucker, 984; Pardalos and Kovoor, 990 solved the problem in greater generality. b c
2 Other algorithms The problem can be solved in many other ways, for example by particularizing QP algorithms such as active-set methods, gradient-proection methods or interior-point methods. It can also be solved by alternating proection onto the two constraints in a finite number of steps Michelot, 986. Another way Boyd and Vandenberghe, 2004, Exercise 4., solution available at materials/lsocoee364b/hw4sol.pdf is to construct a Lagrangian formulation by dualizing the equality constraint and then solve a D nonsmooth optimization over the Lagrange multiplier. Algorithm has the advantage of being very simple, not iterative, and identifying exactly the active set at the solution after at most D steps each of cost O after sorting. 3 A simple proof A proof of correctness of Algorithm can be found in Shalev-Shwartz and Singer 2006 and Chen and Ye 20, but we offer a simpler proof which involves only the KKT theorem. We apply the standard KKT conditions for the problem Nocedal and Wright, The Lagrangian of the problem is Lx,λ,β = 2 x y 2 λx β x where λ and β = [β,...,β D ] are the Lagrange multipliers for the equality and inequality constraints, respectively. At the optimal solution x the following KKT conditions hold: x i y i λ β i = 0, i =,...,D 2a x i 0, i =,...,D 2b β i 0, i =,...,D 2c x i β i = 0, i =,...,D 2d D x i =. 2e From the complementarity condition 2d, it is clear that if x i > 0, we must have β i = 0 and x i = y i +λ > 0; if x i = 0, we must have β i 0 and x i = y i + λ + β i = 0, whence y i + λ = β i 0. Obviously, the components of the optimal solution x that are zeros correspond to the smaller components of y. Without loss of generality, we assume the components of y are sorted and x uses the same ordering, i.e., y y ρ y ρ+ y D, x x ρ > x ρ+ = = x D, and that x x ρ > 0, x ρ+ = = x D = 0. In other words, ρ is the number of positive components in the solution x. Now we apply the last condition and have = D x i = x i = y i +λ whichgivesλ = ρ ρ y i. Henceρisthekeytothesolution. OnceweknowρthereareonlyD possible values of it, we can compute λ, and the optimal solution is obtained by ust adding λ to each component of y and thresholding as in the end of Algorithm. It is easy to check that this solution indeed satisfies all KKT conditions. In the algorithm, we carry out the tests for =,...,D if t = y + y i > 0. We now prove that the number of times this test turns out positive is exactly ρ. The following theorem is essentially Lemma 3 of Shalev-Shwartz and Singer Theorem. Let ρ be the number of positive components in the solution x, then ρ = max{ D: y + y i > 0}. 2
3 Proof. Recall fromthekktconditions 2 thatλρ = ρ y i, y i +λ > 0for i =,...,ρandy i +λ 0for i = ρ+,...,d. In the sequel, we show that for =,...,D, the test will continue to be positive until = ρ and then stay non-positive afterwards, i.e., y + y i > 0 for ρ, and y + y i 0 for > ρ. i For = ρ, we have y ρ + ρ y i = y ρ +λ = x ρ > 0. ii For < ρ, we have y + y i = y + y i = y + i=+ y i + Since y i +λ > 0 for i =,...,ρ, we have y + y i > 0. iii For > ρ, we have y + y i = y + y i = y + y i = y i = y + i=ρ+ = y +λ+ i=+ y i +ρλ y i +λ. i=+ y i = y +ρλ ρy +λ+ i=ρ+ y y i. i=ρ+ Notice y +λ 0 for > ρ, and y y i for i since y is sorted, therefore y + y i < 0. y i Remarks. We denote λ = y i. At the -th test, λ can be considered as a guess of the true λ indeed, λ ρ = λ. If we use this guess to compute a tentative solution x where x i = max{y i +λ,0}, then it is easy to see that x i > 0 for i =,...,, and x i =. In other words, the first components of x are positive and sum to. If we find x + = 0 or y + +λ 0, then we know we have found the optimal solution and = ρ because x satisfies all KKT conditions. 2. To extend the algorithm to a simplex with a different scale, i.e., x = a for a > 0, replace the u i terms with a u i in Algorithm. 4 Matlab code The following vectorized Matlab code implements algorithm. It proects each row vector in the N D matrix Y onto the probability simplex in D dimensions. function X = SimplexProY [N,D] = sizey; X = sorty,2, descend ; Xtmp = cumsumx,2-*diagsparse./:d; X = maxbsxfun@us,y,xtmpsub2ind[n,d],:n,sumx>xtmp,2,0; 3
4 5 An application: Laplacian K-modes clustering Consider the Laplacian K-modes clustering algorithm of Wang and Carreira-Perpiñán 203 which is an extension of the K-modes algorithm of Carreira-Perpiñán and Wang, 203a. Given a dataset x,...,x N R D and suitably defined affinity values w nm 0 between each pair of points x n and x m for example, Gaussian, we define the obective function Z,C s.t. λ 2 N N N K w mn z m z n 2 x n c k 2 z nk G σ m=n= K z nk =, for n =,...,N k= n=k= z nk 0, for n =,...,N, k =,...,K. Z is a matrix of N K, with Z = z,...,z N, where z n = z n,...,z nk are the soft assignments of point x n to clusters,...,k, and c,...,c K R D are modes of the kernel density estimates defined for each cluster where G 2 gives a Gaussian. The problem of proection on the simplex appears in the training problem, i.e., in finding a local imum Z, C of 3, and in the out-of-sample problem, i.e., in assigning a new point to the clusters. Training The optimization of 3 is done by alternating imization over Z and C. For fixed C, the problem over Z is a quadratic program of NK variables: 3a 3b 3c Z λtr Z LZ tr B Z 4a s.t. Z K = N 4b Z 0, 4c where b nk = G x n c k /σ 2, n =,...,N, k =,...,K, and L is the graph Laplacian computed from the pairwise affinities w mn. The problem is convex since L is positive semidefinite. One simple and quite efficient way to solve it is to use an accelerated gradient proection algorithm. The basic gradient proection algorithm iteratively takes a step in the negative gradient from the current point and proects the new point onto the constraint set. In our case, this proection is simple: it separates over each row of Z i.e., the soft assignment of each data point, n =,...,N, and corresponds to a proection on the probability simplex of a K-dimensional row vector, i.e., the problem. Out-of-sample mapping Given a new, test point x R D, we wish to assign it to the clusters found during training. We are given the affinity vectors w = w n and g = g k, where w n is the affinity between x and x n, n =,...,N, and g k = G x c k /σ 2, k =,...,K, respectively. A natural and efficient way to define an out-of-sample mapping zx is to solve a problem of the form 3 with a dataset consisting of the original training set augmented with x, but keeping Z and C fixed to the values obtained during training this avoids having to solve for all points again. Hence, the only free parameter is the assignment vector z for the new point x. After dropping constant terms, the optimization problem 3 reduces to the following quadratic program over K variables: z 2 z z+γg 2 5a s.t. z K = 5b z 0 5c where γ = /2λ N n= w n and N z = Z w w = w n N n = w z n n n= 4
5 is a weighted average of the training points assignments, and so z+γg is itself an average between this and the point-to-centroid affinities. Thus, the solution is the proection of the K-dimensional vector z + γg onto the probability simplex, i.e., the problem again. LASS In general, the optimization problem of eq. 4 is called Laplacian assignment model LASS by Carreira-Perpiñán and Wang 203b. Here, we consider N items and K categories and want to learn a soft assignment z nk of each item to each category, given an item-item graph Laplacian matrix L of N N and an item-category similarity matrix B of N K. The training is as in eq. 4 and the out-of-sample mapping as in eq. 5, and the problem of proection on the simplex appears in both cases. References S. Boyd and L. Vandenberghe. Convex Optimization. Cambridge University Press, Cambridge, U.K., P. Brucker. An On algorithm for quadratic knapsack problems. Operations Research Letters, 33:63 66, Aug M. Á. Carreira-Perpiñán and W. Wang. The K-modes algorithm for clustering. Unpublished manuscript, arxiv: , Apr a. M. Á. Carreira-Perpiñán and W. Wang. A simple assignment model with Laplacian smoothing. Unpublished manuscript, 203b. Y. Chen and X. Ye. Proection onto a simplex. Unpublished manuscript, arxiv:0.608, Feb J. Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra. Efficient proections onto the l -ball for learning in high dimensions. In A. McCallum and S. Roweis, editors, Proc. of the 25th Int. Conf. Machine Learning ICML 08, pages , Helsinki, Finland, July C. Michelot. A finite algorithm for finding the proection of a point onto the canonical simplex of R n. J. Optimization Theory and Applications, 50:95 200, July 986. J. Nocedal and S. J. Wright. Numerical Optimization. Springer Series in Operations Research and Financial Engineering. Springer-Verlag, New York, second edition, P. M. Pardalos and N. Kovoor. An algorithm for a singly constrained class of quadratic programs subect to upper and lower bounds. Math. Prog., 46 3:32 328, Jan S. Shalev-Shwartz and Y. Singer. Efficient learning of label ranking by soft proections onto polyhedra. J. Machine Learning Research, 7: , W. Wang and M. Á. Carreira-Perpiñán. Laplacian K-modes clustering. Unpublished manuscript,
The K-modes and Laplacian K-modes algorithms for clustering
The K-modes and Laplacian K-modes algorithms for clustering Miguel Á. Carreira-Perpiñán Electrical Engineering and Computer Science University of California, Merced http://faculty.ucmerced.edu/mcarreira-perpinan
More informationMathematical Programming and Research Methods (Part II)
Mathematical Programming and Research Methods (Part II) 4. Convexity and Optimization Massimiliano Pontil (based on previous lecture by Andreas Argyriou) 1 Today s Plan Convex sets and functions Types
More informationLecture 2 September 3
EE 381V: Large Scale Optimization Fall 2012 Lecture 2 September 3 Lecturer: Caramanis & Sanghavi Scribe: Hongbo Si, Qiaoyang Ye 2.1 Overview of the last Lecture The focus of the last lecture was to give
More informationIntroduction to Constrained Optimization
Introduction to Constrained Optimization Duality and KKT Conditions Pratik Shah {pratik.shah [at] lnmiit.ac.in} The LNM Institute of Information Technology www.lnmiit.ac.in February 13, 2013 LNMIIT MLPR
More informationThis lecture: Convex optimization Convex sets Convex functions Convex optimization problems Why convex optimization? Why so early in the course?
Lec4 Page 1 Lec4p1, ORF363/COS323 This lecture: Convex optimization Convex sets Convex functions Convex optimization problems Why convex optimization? Why so early in the course? Instructor: Amir Ali Ahmadi
More informationIntroduction to Operations Research
- Introduction to Operations Research Peng Zhang April, 5 School of Computer Science and Technology, Shandong University, Ji nan 5, China. Email: algzhang@sdu.edu.cn. Introduction Overview of the Operations
More informationProgramming, numerics and optimization
Programming, numerics and optimization Lecture C-4: Constrained optimization Łukasz Jankowski ljank@ippt.pan.pl Institute of Fundamental Technological Research Room 4.32, Phone +22.8261281 ext. 428 June
More informationLECTURE 13: SOLUTION METHODS FOR CONSTRAINED OPTIMIZATION. 1. Primal approach 2. Penalty and barrier methods 3. Dual approach 4. Primal-dual approach
LECTURE 13: SOLUTION METHODS FOR CONSTRAINED OPTIMIZATION 1. Primal approach 2. Penalty and barrier methods 3. Dual approach 4. Primal-dual approach Basic approaches I. Primal Approach - Feasible Direction
More informationData Analysis 3. Support Vector Machines. Jan Platoš October 30, 2017
Data Analysis 3 Support Vector Machines Jan Platoš October 30, 2017 Department of Computer Science Faculty of Electrical Engineering and Computer Science VŠB - Technical University of Ostrava Table of
More informationLecture 15: Log Barrier Method
10-725/36-725: Convex Optimization Spring 2015 Lecturer: Ryan Tibshirani Lecture 15: Log Barrier Method Scribes: Pradeep Dasigi, Mohammad Gowayyed Note: LaTeX template courtesy of UC Berkeley EECS dept.
More informationUnconstrained Optimization Principles of Unconstrained Optimization Search Methods
1 Nonlinear Programming Types of Nonlinear Programs (NLP) Convexity and Convex Programs NLP Solutions Unconstrained Optimization Principles of Unconstrained Optimization Search Methods Constrained Optimization
More informationLecture 2 - Introduction to Polytopes
Lecture 2 - Introduction to Polytopes Optimization and Approximation - ENS M1 Nicolas Bousquet 1 Reminder of Linear Algebra definitions Let x 1,..., x m be points in R n and λ 1,..., λ m be real numbers.
More informationLinear methods for supervised learning
Linear methods for supervised learning LDA Logistic regression Naïve Bayes PLA Maximum margin hyperplanes Soft-margin hyperplanes Least squares resgression Ridge regression Nonlinear feature maps Sometimes
More informationCOMS 4771 Support Vector Machines. Nakul Verma
COMS 4771 Support Vector Machines Nakul Verma Last time Decision boundaries for classification Linear decision boundary (linear classification) The Perceptron algorithm Mistake bound for the perceptron
More informationSupport Vector Machines.
Support Vector Machines srihari@buffalo.edu SVM Discussion Overview 1. Overview of SVMs 2. Margin Geometry 3. SVM Optimization 4. Overlapping Distributions 5. Relationship to Logistic Regression 6. Dealing
More informationIntroduction to Machine Learning
Introduction to Machine Learning Maximum Margin Methods Varun Chandola Computer Science & Engineering State University of New York at Buffalo Buffalo, NY, USA chandola@buffalo.edu Chandola@UB CSE 474/574
More informationConvex Optimization. Erick Delage, and Ashutosh Saxena. October 20, (a) (b) (c)
Convex Optimization (for CS229) Erick Delage, and Ashutosh Saxena October 20, 2006 1 Convex Sets Definition: A set G R n is convex if every pair of point (x, y) G, the segment beteen x and y is in A. More
More informationChapter 15 Introduction to Linear Programming
Chapter 15 Introduction to Linear Programming An Introduction to Optimization Spring, 2015 Wei-Ta Chu 1 Brief History of Linear Programming The goal of linear programming is to determine the values of
More informationIntroduction to Modern Control Systems
Introduction to Modern Control Systems Convex Optimization, Duality and Linear Matrix Inequalities Kostas Margellos University of Oxford AIMS CDT 2016-17 Introduction to Modern Control Systems November
More informationDavid G. Luenberger Yinyu Ye. Linear and Nonlinear. Programming. Fourth Edition. ö Springer
David G. Luenberger Yinyu Ye Linear and Nonlinear Programming Fourth Edition ö Springer Contents 1 Introduction 1 1.1 Optimization 1 1.2 Types of Problems 2 1.3 Size of Problems 5 1.4 Iterative Algorithms
More informationSimplex Algorithm in 1 Slide
Administrivia 1 Canonical form: Simplex Algorithm in 1 Slide If we do pivot in A r,s >0, where c s
More informationShiqian Ma, MAT-258A: Numerical Optimization 1. Chapter 2. Convex Optimization
Shiqian Ma, MAT-258A: Numerical Optimization 1 Chapter 2 Convex Optimization Shiqian Ma, MAT-258A: Numerical Optimization 2 2.1. Convex Optimization General optimization problem: min f 0 (x) s.t., f i
More informationApplied Lagrange Duality for Constrained Optimization
Applied Lagrange Duality for Constrained Optimization Robert M. Freund February 10, 2004 c 2004 Massachusetts Institute of Technology. 1 1 Overview The Practical Importance of Duality Review of Convexity
More informationCalifornia Institute of Technology Crash-Course on Convex Optimization Fall Ec 133 Guilherme Freitas
California Institute of Technology HSS Division Crash-Course on Convex Optimization Fall 2011-12 Ec 133 Guilherme Freitas In this text, we will study the following basic problem: maximize x C f(x) subject
More informationThe Affine Scaling Method
MA33 Linear Programming W. J. Martin October 9, 8 The Affine Scaling Method Overview Given a linear programming problem in equality form with full rank constraint matrix and a strictly positive feasible
More informationOptimization for Machine Learning
Optimization for Machine Learning (Problems; Algorithms - C) SUVRIT SRA Massachusetts Institute of Technology PKU Summer School on Data Science (July 2017) Course materials http://suvrit.de/teaching.html
More informationMinimum-Polytope-Based Linear Programming Decoder for LDPC Codes via ADMM Approach
Minimum-Polytope-Based Linear Programg Decoder for LDPC Codes via ADMM Approach Jing Bai, Yongchao Wang, Member, IEEE, Francis C. M. Lau, Senior Member, IEEE arxiv:90.07806v [cs.it] 23 Jan 209 Abstract
More information11 Linear Programming
11 Linear Programming 11.1 Definition and Importance The final topic in this course is Linear Programming. We say that a problem is an instance of linear programming when it can be effectively expressed
More informationA Short SVM (Support Vector Machine) Tutorial
A Short SVM (Support Vector Machine) Tutorial j.p.lewis CGIT Lab / IMSC U. Southern California version 0.zz dec 004 This tutorial assumes you are familiar with linear algebra and equality-constrained optimization/lagrange
More informationMATH3016: OPTIMIZATION
MATH3016: OPTIMIZATION Lecturer: Dr Huifu Xu School of Mathematics University of Southampton Highfield SO17 1BJ Southampton Email: h.xu@soton.ac.uk 1 Introduction What is optimization? Optimization is
More informationPOLYHEDRAL GEOMETRY. Convex functions and sets. Mathematical Programming Niels Lauritzen Recall that a subset C R n is convex if
POLYHEDRAL GEOMETRY Mathematical Programming Niels Lauritzen 7.9.2007 Convex functions and sets Recall that a subset C R n is convex if {λx + (1 λ)y 0 λ 1} C for every x, y C and 0 λ 1. A function f :
More informationLec13p1, ORF363/COS323
Lec13 Page 1 Lec13p1, ORF363/COS323 This lecture: Semidefinite programming (SDP) Definition and basic properties Review of positive semidefinite matrices SDP duality SDP relaxations for nonconvex optimization
More informationConvexity Theory and Gradient Methods
Convexity Theory and Gradient Methods Angelia Nedić angelia@illinois.edu ISE Department and Coordinated Science Laboratory University of Illinois at Urbana-Champaign Outline Convex Functions Optimality
More informationEfficient Euclidean Projections in Linear Time
Jun Liu J.LIU@ASU.EDU Jieping Ye JIEPING.YE@ASU.EDU Department of Computer Science and Engineering, Arizona State University, Tempe, AZ 85287, USA Abstract We consider the problem of computing the Euclidean
More informationConvex Optimization. Lijun Zhang Modification of
Convex Optimization Lijun Zhang zlj@nju.edu.cn http://cs.nju.edu.cn/zlj Modification of http://stanford.edu/~boyd/cvxbook/bv_cvxslides.pdf Outline Introduction Convex Sets & Functions Convex Optimization
More informationFinding Euclidean Distance to a Convex Cone Generated by a Large Number of Discrete Points
Submitted to Operations Research manuscript (Please, provide the manuscript number!) Finding Euclidean Distance to a Convex Cone Generated by a Large Number of Discrete Points Ali Fattahi Anderson School
More informationPrograms. Introduction
16 Interior Point I: Linear Programs Lab Objective: For decades after its invention, the Simplex algorithm was the only competitive method for linear programming. The past 30 years, however, have seen
More informationConstrained optimization
Constrained optimization A general constrained optimization problem has the form where The Lagrangian function is given by Primal and dual optimization problems Primal: Dual: Weak duality: Strong duality:
More informationThe Constrained Laplacian Rank Algorithm for Graph-Based Clustering
The Constrained Laplacian Rank Algorithm for Graph-Based Clustering Feiping Nie, Xiaoqian Wang, Michael I. Jordan, Heng Huang Department of Computer Science and Engineering, University of Texas, Arlington
More information60 2 Convex sets. {x a T x b} {x ã T x b}
60 2 Convex sets Exercises Definition of convexity 21 Let C R n be a convex set, with x 1,, x k C, and let θ 1,, θ k R satisfy θ i 0, θ 1 + + θ k = 1 Show that θ 1x 1 + + θ k x k C (The definition of convexity
More informationLocal and Global Minimum
Local and Global Minimum Stationary Point. From elementary calculus, a single variable function has a stationary point at if the derivative vanishes at, i.e., 0. Graphically, the slope of the function
More informationCS446: Machine Learning Fall Problem Set 4. Handed Out: October 17, 2013 Due: October 31 th, w T x i w
CS446: Machine Learning Fall 2013 Problem Set 4 Handed Out: October 17, 2013 Due: October 31 th, 2013 Feel free to talk to other members of the class in doing the homework. I am more concerned that you
More informationAdvanced Operations Research Techniques IE316. Quiz 1 Review. Dr. Ted Ralphs
Advanced Operations Research Techniques IE316 Quiz 1 Review Dr. Ted Ralphs IE316 Quiz 1 Review 1 Reading for The Quiz Material covered in detail in lecture. 1.1, 1.4, 2.1-2.6, 3.1-3.3, 3.5 Background material
More information21-256: Lagrange multipliers
21-256: Lagrange multipliers Clive Newstead, Thursday 12th June 2014 Lagrange multipliers give us a means of optimizing multivariate functions subject to a number of constraints on their variables. Problems
More informationLecture 2. Topology of Sets in R n. August 27, 2008
Lecture 2 Topology of Sets in R n August 27, 2008 Outline Vectors, Matrices, Norms, Convergence Open and Closed Sets Special Sets: Subspace, Affine Set, Cone, Convex Set Special Convex Sets: Hyperplane,
More informationLecture notes on the simplex method September We will present an algorithm to solve linear programs of the form. maximize.
Cornell University, Fall 2017 CS 6820: Algorithms Lecture notes on the simplex method September 2017 1 The Simplex Method We will present an algorithm to solve linear programs of the form maximize subject
More information4. Simplicial Complexes and Simplicial Homology
MATH41071/MATH61071 Algebraic topology Autumn Semester 2017 2018 4. Simplicial Complexes and Simplicial Homology Geometric simplicial complexes 4.1 Definition. A finite subset { v 0, v 1,..., v r } R n
More informationLecture 7: Support Vector Machine
Lecture 7: Support Vector Machine Hien Van Nguyen University of Houston 9/28/2017 Separating hyperplane Red and green dots can be separated by a separating hyperplane Two classes are separable, i.e., each
More informationFast-Lipschitz Optimization
Fast-Lipschitz Optimization DREAM Seminar Series University of California at Berkeley September 11, 2012 Carlo Fischione ACCESS Linnaeus Center, Electrical Engineering KTH Royal Institute of Technology
More informationof Convex Analysis Fundamentals Jean-Baptiste Hiriart-Urruty Claude Lemarechal Springer With 66 Figures
2008 AGI-Information Management Consultants May be used for personal purporses only or by libraries associated to dandelon.com network. Jean-Baptiste Hiriart-Urruty Claude Lemarechal Fundamentals of Convex
More informationConvex Optimization MLSS 2015
Convex Optimization MLSS 2015 Constantine Caramanis The University of Texas at Austin The Optimization Problem minimize : f (x) subject to : x X. The Optimization Problem minimize : f (x) subject to :
More informationInverse and Implicit functions
CHAPTER 3 Inverse and Implicit functions. Inverse Functions and Coordinate Changes Let U R d be a domain. Theorem. (Inverse function theorem). If ϕ : U R d is differentiable at a and Dϕ a is invertible,
More informationAVERAGING RANDOM PROJECTION: A FAST ONLINE SOLUTION FOR LARGE-SCALE CONSTRAINED STOCHASTIC OPTIMIZATION. Jialin Liu, Yuantao Gu, and Mengdi Wang
AVERAGING RANDOM PROJECTION: A FAST ONLINE SOLUTION FOR LARGE-SCALE CONSTRAINED STOCHASTIC OPTIMIZATION Jialin Liu, Yuantao Gu, and Mengdi Wang Tsinghua National Laboratory for Information Science and
More informationModified Augmented Lagrangian Coordination and Alternating Direction Method of Multipliers with
Modified Augmented Lagrangian Coordination and Alternating Direction Method of Multipliers with Parallelization in Non-hierarchical Analytical Target Cascading Yongsu Jung Department of Mechanical Engineering,
More informationThe Simplex Algorithm
The Simplex Algorithm Uri Feige November 2011 1 The simplex algorithm The simplex algorithm was designed by Danzig in 1947. This write-up presents the main ideas involved. It is a slight update (mostly
More informationMathematical and Algorithmic Foundations Linear Programming and Matchings
Adavnced Algorithms Lectures Mathematical and Algorithmic Foundations Linear Programming and Matchings Paul G. Spirakis Department of Computer Science University of Patras and Liverpool Paul G. Spirakis
More informationContents. I Basics 1. Copyright by SIAM. Unauthorized reproduction of this article is prohibited.
page v Preface xiii I Basics 1 1 Optimization Models 3 1.1 Introduction... 3 1.2 Optimization: An Informal Introduction... 4 1.3 Linear Equations... 7 1.4 Linear Optimization... 10 Exercises... 12 1.5
More informationSequential Coordinate-wise Algorithm for Non-negative Least Squares Problem
CENTER FOR MACHINE PERCEPTION CZECH TECHNICAL UNIVERSITY Sequential Coordinate-wise Algorithm for Non-negative Least Squares Problem Woring document of the EU project COSPAL IST-004176 Vojtěch Franc, Miro
More informationarxiv: v1 [math.at] 8 Jan 2015
HOMOLOGY GROUPS OF SIMPLICIAL COMPLEMENTS: A NEW PROOF OF HOCHSTER THEOREM arxiv:1501.01787v1 [math.at] 8 Jan 2015 JUN MA, FEIFEI FAN AND XIANGJUN WANG Abstract. In this paper, we consider homology groups
More informationMA4254: Discrete Optimization. Defeng Sun. Department of Mathematics National University of Singapore Office: S Telephone:
MA4254: Discrete Optimization Defeng Sun Department of Mathematics National University of Singapore Office: S14-04-25 Telephone: 6516 3343 Aims/Objectives: Discrete optimization deals with problems of
More informationConvexity: an introduction
Convexity: an introduction Geir Dahl CMA, Dept. of Mathematics and Dept. of Informatics University of Oslo 1 / 74 1. Introduction 1. Introduction what is convexity where does it arise main concepts and
More informationMath 5593 Linear Programming Lecture Notes
Math 5593 Linear Programming Lecture Notes Unit II: Theory & Foundations (Convex Analysis) University of Colorado Denver, Fall 2013 Topics 1 Convex Sets 1 1.1 Basic Properties (Luenberger-Ye Appendix B.1).........................
More informationREGULAR GRAPHS OF GIVEN GIRTH. Contents
REGULAR GRAPHS OF GIVEN GIRTH BROOKE ULLERY Contents 1. Introduction This paper gives an introduction to the area of graph theory dealing with properties of regular graphs of given girth. A large portion
More information1 Linear Programming. 1.1 Optimizion problems and convex polytopes 1 LINEAR PROGRAMMING
1 LINEAR PROGRAMMING 1 Linear Programming Now, we will talk a little bit about Linear Programming. We say that a problem is an instance of linear programming when it can be effectively expressed in the
More informationGeometry. Every Simplicial Polytope with at Most d + 4 Vertices Is a Quotient of a Neighborly Polytope. U. H. Kortenkamp. 1.
Discrete Comput Geom 18:455 462 (1997) Discrete & Computational Geometry 1997 Springer-Verlag New York Inc. Every Simplicial Polytope with at Most d + 4 Vertices Is a Quotient of a Neighborly Polytope
More informationHyperbolic structures and triangulations
CHAPTER Hyperbolic structures and triangulations In chapter 3, we learned that hyperbolic structures lead to developing maps and holonomy, and that the developing map is a covering map if and only if the
More informationSparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual
Sparse Optimization Lecture: Proximal Operator/Algorithm and Lagrange Dual Instructor: Wotao Yin July 2013 online discussions on piazza.com Those who complete this lecture will know learn the proximal
More informationAnalysis of high dimensional data via Topology. Louis Xiang. Oak Ridge National Laboratory. Oak Ridge, Tennessee
Analysis of high dimensional data via Topology Louis Xiang Oak Ridge National Laboratory Oak Ridge, Tennessee Contents Abstract iii 1 Overview 1 2 Data Set 1 3 Simplicial Complex 5 4 Computation of homology
More informationBilinear Programming
Bilinear Programming Artyom G. Nahapetyan Center for Applied Optimization Industrial and Systems Engineering Department University of Florida Gainesville, Florida 32611-6595 Email address: artyom@ufl.edu
More informationNumerical Optimization
Convex Sets Computer Science and Automation Indian Institute of Science Bangalore 560 012, India. NPTEL Course on Let x 1, x 2 R n, x 1 x 2. Line and line segment Line passing through x 1 and x 2 : {y
More informationConvex Optimization - Chapter 1-2. Xiangru Lian August 28, 2015
Convex Optimization - Chapter 1-2 Xiangru Lian August 28, 2015 1 Mathematical optimization minimize f 0 (x) s.t. f j (x) 0, j=1,,m, (1) x S x. (x 1,,x n ). optimization variable. f 0. R n R. objective
More informationLinear Programming. Readings: Read text section 11.6, and sections 1 and 2 of Tom Ferguson s notes (see course homepage).
Linear Programming Learning Goals. Introduce Linear Programming Problems. Widget Example, Graphical Solution. Basic Theory: Feasible Set, Vertices, Existence of Solutions. Equivalent formulations. Outline
More informationLinear Programming. Widget Factory Example. Linear Programming: Standard Form. Widget Factory Example: Continued.
Linear Programming Widget Factory Example Learning Goals. Introduce Linear Programming Problems. Widget Example, Graphical Solution. Basic Theory:, Vertices, Existence of Solutions. Equivalent formulations.
More informationLinear programming and duality theory
Linear programming and duality theory Complements of Operations Research Giovanni Righini Linear Programming (LP) A linear program is defined by linear constraints, a linear objective function. Its variables
More informationConvex Optimization M2
Convex Optimization M2 Lecture 1 A. d Aspremont. Convex Optimization M2. 1/49 Today Convex optimization: introduction Course organization and other gory details... Convex sets, basic definitions. A. d
More informationKernel Methods & Support Vector Machines
& Support Vector Machines & Support Vector Machines Arvind Visvanathan CSCE 970 Pattern Recognition 1 & Support Vector Machines Question? Draw a single line to separate two classes? 2 & Support Vector
More informationNew Methods for Solving Large Scale Linear Programming Problems in the Windows and Linux computer operating systems
arxiv:1209.4308v1 [math.oc] 19 Sep 2012 New Methods for Solving Large Scale Linear Programming Problems in the Windows and Linux computer operating systems Saeed Ketabchi, Hossein Moosaei, Hossein Sahleh
More informationLecture Notes 2: The Simplex Algorithm
Algorithmic Methods 25/10/2010 Lecture Notes 2: The Simplex Algorithm Professor: Yossi Azar Scribe:Kiril Solovey 1 Introduction In this lecture we will present the Simplex algorithm, finish some unresolved
More informationHowever, this is not always true! For example, this fails if both A and B are closed and unbounded (find an example).
98 CHAPTER 3. PROPERTIES OF CONVEX SETS: A GLIMPSE 3.2 Separation Theorems It seems intuitively rather obvious that if A and B are two nonempty disjoint convex sets in A 2, then there is a line, H, separating
More informationCombinatorial Geometry & Topology arising in Game Theory and Optimization
Combinatorial Geometry & Topology arising in Game Theory and Optimization Jesús A. De Loera University of California, Davis LAST EPISODE... We discuss the content of the course... Convex Sets A set is
More informationLecture Linear Support Vector Machines
Lecture 8 In this lecture we return to the task of classification. As seen earlier, examples include spam filters, letter recognition, or text classification. In this lecture we introduce a popular method
More informationLecture 4: Rational IPs, Polyhedron, Decomposition Theorem
IE 5: Integer Programming, Spring 29 24 Jan, 29 Lecture 4: Rational IPs, Polyhedron, Decomposition Theorem Lecturer: Karthik Chandrasekaran Scribe: Setareh Taki Disclaimer: These notes have not been subjected
More information(1) Given the following system of linear equations, which depends on a parameter a R, 3x y + 5z = 2 4x + y + (a 2 14)z = a + 2
(1 Given the following system of linear equations, which depends on a parameter a R, x + 2y 3z = 4 3x y + 5z = 2 4x + y + (a 2 14z = a + 2 (a Classify the system of equations depending on the values of
More informationA PARAMETRIC SIMPLEX METHOD FOR OPTIMIZING A LINEAR FUNCTION OVER THE EFFICIENT SET OF A BICRITERIA LINEAR PROBLEM. 1.
ACTA MATHEMATICA VIETNAMICA Volume 21, Number 1, 1996, pp. 59 67 59 A PARAMETRIC SIMPLEX METHOD FOR OPTIMIZING A LINEAR FUNCTION OVER THE EFFICIENT SET OF A BICRITERIA LINEAR PROBLEM NGUYEN DINH DAN AND
More informationSection 5 Convex Optimisation 1. W. Dai (IC) EE4.66 Data Proc. Convex Optimisation page 5-1
Section 5 Convex Optimisation 1 W. Dai (IC) EE4.66 Data Proc. Convex Optimisation 1 2018 page 5-1 Convex Combination Denition 5.1 A convex combination is a linear combination of points where all coecients
More informationConvex Optimization / Homework 2, due Oct 3
Convex Optimization 0-725/36-725 Homework 2, due Oct 3 Instructions: You must complete Problems 3 and either Problem 4 or Problem 5 (your choice between the two) When you submit the homework, upload a
More informationConvex Optimization CMU-10725
Convex Optimization CMU-10725 Ellipsoid Methods Barnabás Póczos & Ryan Tibshirani Outline Linear programs Simplex algorithm Running time: Polynomial or Exponential? Cutting planes & Ellipsoid methods for
More informationAlternating Projections
Alternating Projections Stephen Boyd and Jon Dattorro EE392o, Stanford University Autumn, 2003 1 Alternating projection algorithm Alternating projections is a very simple algorithm for computing a point
More information1. Introduction. performance of numerical methods. complexity bounds. structural convex optimization. course goals and topics
1. Introduction EE 546, Univ of Washington, Spring 2016 performance of numerical methods complexity bounds structural convex optimization course goals and topics 1 1 Some course info Welcome to EE 546!
More information6 Randomized rounding of semidefinite programs
6 Randomized rounding of semidefinite programs We now turn to a new tool which gives substantially improved performance guarantees for some problems We now show how nonlinear programming relaxations can
More informationDistance-to-Solution Estimates for Optimization Problems with Constraints in Standard Form
Distance-to-Solution Estimates for Optimization Problems with Constraints in Standard Form Philip E. Gill Vyacheslav Kungurtsev Daniel P. Robinson UCSD Center for Computational Mathematics Technical Report
More informationClustering. SC4/SM4 Data Mining and Machine Learning, Hilary Term 2017 Dino Sejdinovic
Clustering SC4/SM4 Data Mining and Machine Learning, Hilary Term 2017 Dino Sejdinovic Clustering is one of the fundamental and ubiquitous tasks in exploratory data analysis a first intuition about the
More informationDM545 Linear and Integer Programming. Lecture 2. The Simplex Method. Marco Chiarandini
DM545 Linear and Integer Programming Lecture 2 The Marco Chiarandini Department of Mathematics & Computer Science University of Southern Denmark Outline 1. 2. 3. 4. Standard Form Basic Feasible Solutions
More informationLinear Programming. Larry Blume. Cornell University & The Santa Fe Institute & IHS
Linear Programming Larry Blume Cornell University & The Santa Fe Institute & IHS Linear Programs The general linear program is a constrained optimization problem where objectives and constraints are all
More informationConvexity. 1 X i is convex. = b is a hyperplane in R n, and is denoted H(p, b) i.e.,
Convexity We ll assume throughout, without always saying so, that we re in the finite-dimensional Euclidean vector space R n, although sometimes, for statements that hold in any vector space, we ll say
More informationLinear Programming Problems
Linear Programming Problems Two common formulations of linear programming (LP) problems are: min Subject to: 1,,, 1,2,,;, max Subject to: 1,,, 1,2,,;, Linear Programming Problems The standard LP problem
More informationKernel Methods. Chapter 9 of A Course in Machine Learning by Hal Daumé III. Conversion to beamer by Fabrizio Riguzzi
Kernel Methods Chapter 9 of A Course in Machine Learning by Hal Daumé III http://ciml.info Conversion to beamer by Fabrizio Riguzzi Kernel Methods 1 / 66 Kernel Methods Linear models are great because
More informationLecture 18: March 23
0-725/36-725: Convex Optimization Spring 205 Lecturer: Ryan Tibshirani Lecture 8: March 23 Scribes: James Duyck Note: LaTeX template courtesy of UC Berkeley EECS dept. Disclaimer: These notes have not
More informationCS675: Convex and Combinatorial Optimization Spring 2018 The Simplex Algorithm. Instructor: Shaddin Dughmi
CS675: Convex and Combinatorial Optimization Spring 2018 The Simplex Algorithm Instructor: Shaddin Dughmi Algorithms for Convex Optimization We will look at 2 algorithms in detail: Simplex and Ellipsoid.
More informationMachine Learning. Topic 5: Linear Discriminants. Bryan Pardo, EECS 349 Machine Learning, 2013
Machine Learning Topic 5: Linear Discriminants Bryan Pardo, EECS 349 Machine Learning, 2013 Thanks to Mark Cartwright for his extensive contributions to these slides Thanks to Alpaydin, Bishop, and Duda/Hart/Stork
More information