Support Vector. Machines. Algorithms, and Extensions. Optimization Based Theory, Naiyang Deng YingjieTian. Chunhua Zhang.

Transcription

1 Support Vector Machines Optimization Based Theory, Algorithms, and Extensions Naiyang Deng YingjieTian Chunhua Zhang CRC Press Taylor & Francis Group Boca Raton London New York CRC Press is an imprint of the Taylor fs Francis Croup, an Informa business A CHAPMAN & HALL BOOK

2 List of Figures xvii List of Tables xxi Preface xxiii List of Symbols xxvii 1 Optimization 1 11 Optimization Problems in Euclidian Space An example of optimization problems Optimization problems and their solutions Geometric interpretation of optimization problems 4 12 Convex Programming in Euclidean Space Convex sets and convex functions Convex sets Convex functions Convex programming and their properties Convex programming problems Basic properties Duality theory Derivation of the dual problem Duality theory Optimality conditions Linear programming Convex Programming in Hilbert Space Convex sets and Frechet derivative Convex programming problems Duality theory Optimality conditions 20 *14 Convex Programming with Generalized Inequality Constraints in Euclidian Space Convex programming with generalized inequality con straints Cones Generalized inequalities 21 ix

3 X Convex programming with generalized in equality constraints Duality theory Dual cones Derivation of the dual problem Duality theory Optimality conditions Second-order cone programming Second-order cone programming and its dual problem Software for second-order cone programming Semidefinite programming Semidefinite programming and its dual prob lem Software for semidefinite programming *15 Convex Programming with Generalized Inequality Constraints in Hilbert Space A"-convex function and Frechet derivative Convex programming Duality theory Optimality conditions Linear Classification Presentation of Classification Problems A sample (diagnosis of heart disease) Classification problems and classification machines Support Vector Classification (SVC) for Linearly Separable Problems Maximal margin method Derivation of the maximal margin method 2212 Properties of the maximal margin method 222 Linearly separable support vector classification 2221 Relationship between the primal and dual problems Linearly separable support vector classifica tion Support vector Linear C-Support Vector Classification Maximal margin method Derivation of the maximal margin method 2312 Properties of the maximal margin method 232 Linear C-support vector classification Relationship between the primal and dual problems Linear C-support vector classification

4 xi 3 Linear Regression Regression Problems and Linear Regression Problems Hard e-band Hyperplane Linear regression problem and hard -band hyperplane Hard -band hyperplane and linear classification Optimization problem of constructing a hard -band hy perplane Linear Hard c-band Support Vector Regression Primal problem Dual problem and relationship between the primal and dual problems Linear hard -band support vector regression Linear ^-Support Vector Regression Primal problem Dual problem and relationship between the primal and dual problems Linear -support vector regression 79 4 Kernels and Support Vector Machines From Linear Classification to Nonlinear Classification An example of nonlinear classification Classification machine based on nonlinear separation Regression machine based on nonlinear separation Kernels Properties Construction of kernels Basic kernels Operations keeping kernels Commonly used kernels Graph kernel Support Vector Machines and Their Properties Support vector classification Algorithm Support vector Properties Soft margin loss function Probabilistic outputs Support vector regression Algorithm Support vector Properties ^-Insensitive loss function Flatness of support vector machines Runge phenomenon Flatness of e-support vector regression 115

5 xii Flatness of C-support vector classification 44 Meaning of Kernels Basic Statistical Learning Theory of C-Support Vector Clas sification Classification Problems on Statistical Learning Theory Probability distribution Description of classification problems Empirical Risk Minimization Vapnik Chervonenkis (VC) Dimension Structural Risk Minimization An Implementation of Structural Risk Minimization Primal problem Quasi-dual problem and relationship between quasidual problem and primal problem Structural risk minimization classification Theoretical Foundation of C-Support Vector Classification on Statistical Learning Theory Linear C-support vector classification Relationship between dual problem and quasi-dual problem Interpretation of C-support vector classification 6 Model Construction Data Generation Orthogonal encoding Spectrum profile encoding Positional weighted matrix encoding Data Preprocessing Representation of nominal features Feature selection F-score method Recursive feature elimination method Methods based on p-norm support vector clas sification (0 < p < 1) Feature extraction Linear dimensionality reduction Nonlinear dimensionality reduction Data compression Data rebalancing Model Selection Algorithm evaluation Some evaluation measures for a decision func tion 172

6 xiii 6312 Some evaluation measures for a concrete algo rithm Selection of kernels and parameters Rule Extraction A toy example Rule extraction Implementation Stopping Criterion The first stopping criterion The second stopping criterion The third stopping criterion Chunking Decomposing Sequential Minimal Optimization Main steps Selecting the working set Analytical solution of the two-variables problem Software Variants and Extensions of Support Vector Machines Variants of Binary Support Vector Classification Support vector classification with homogeneous decision function Bounded support vector classification Least squares support vector classification Proximal support vector classification i/-support vector classification z/-support vector classification Relationship between v-svc and C-SVC Significance of the parameter v Linear programming support vector classifications (LPSVC) LPSVC corresponding to C-SVC LPSVC corresponding to J/-SVC Twin support vector classification Variants of Support Vector Regression Least squares support vector regression ^-Support vector regression ^-Support vector regression Relationship between u-svr and e-svr The significance of the parameter v Linear programming support vector regression (LPSVR) Multiclass Classification 232

7 xiv 831 Approaches based on binary classifiers One versus one One versus the rest Error-correcting output coding Approach based on ordinal regression machines Ordinal regression machine Approach based on ordinal regression ma chines Crammer-Singer multiclass support vector classification Basic idea Primal problem Crammer-Singer support vector classification Semisupervised Classification PU classification problem Biased support vector classification'101' Optimization problem The selection of the parameters C+ and C_ Classification problem with labeled and unlabeled in puts Support vector classification by semidefinite program ming Optimization problem Approximate solution via semidefinite pro gramming Support vector classification by semidefinite programming Universum Classification Universum classification problem Primal problem and dual problem Algorithm and its relationship with three-class classification Construction of Universum Privileged Classification Linear privileged support vector classification Nonlinear privileged support vector classification A variation Knowledge-based Classification Knowledge-based linear support vector classification 872 Knowledge-based nonlinear support vector classification Robust Classification Robust classification problem The solution when the input sets are polyhedrons Linear robust support vector classification 8822 Robust support vector classification The solution when the input sets are superspheres

8 xv Linear robust support vector classification 8832 Robust support vector classification Multi-instance Classification Multi-instance classification problem Multi-instance linear support vector classification Optimization problem Linear support vector classification Multi-instance support vector classification Multi-label Classification Problem transformation methods Algorithm adaptation methods A ranking system Label set size prediction Algorithm 297 Bibliography 299 Index 315