Distance Weighted Discrimination and Second Order Cone Programming

Transcription

1 Distance Weighted Discrimination and Second Order Cone Programming Hanwen Huang, Xiaosun Lu, Yufeng Liu, J. S. Marron, Perry Haaand Apri 3, Introduction This vignette demonstrates the utiity and fexibiity of the R-package dwd in conducting cassification and optimization probems. Distance Weighted Discrimination (DWD) is a recenty deveoped powerfu cassification method which was originay motivated for soving the High Dimensiona Low Sampe Size (HDLSS) exampes (Marron et a. (2007)), but can be appied to many other exampes as we. One of the big advantages of DWD over Support Vector Machine (SVM) is that it can overcome data piing probem in high dimensiona situations. The origina DWD paper (Marron et a. (2007)) ony describes the impementation of the binary cassification method. The muticass version of DWD has aso been deveoped in Huang et a. (2011). It is suggested that a users refer to these pubications in order to understand the DWD terminoogy and principes in greater detai. The impementation of DWD is more chaenging than the impementation of SVM because it requires soving an optimization probem caed Second Order Cone Programming (SOCP). The existing DWD software was written in Matab and empoyed a very efficient SOCP sover from SDPT3 (semidefinite-quadratic-inear programming) package deveoped by Toh et a.. SDPT3 impemented an infeasibe path-foowing agorithm for soving conic optimization probems invoving semidefinite, second-order and inear cone constraints. The R-package dwd was deveoped on the basis of the existing Matab version but incudes some additiona features. The key step was to buid a R SOCP sover which was impemented using exacty the same agorithm as used by the corresponding Matab version. For the convenience of the users who are famiiar with using SVM in R, the main cassification functions and arguments in dwd are formatted in a simiar way to the one used by the SVM functions in the kernab package. To hep the users in using this software, some exampes to iustrate the coding of probem data are provided. In addition, an efficient R Quadratic Programming (QP) sover based on the SOCP is aso incuded in this package which provides a usefu too for those users who want to deveop their own SVM package. 2 DWD impementation and output Both SVM and DWD are margin-based cassification methods in the sense that they buid the cassifier through finding a decision boundary to separate the casses. DWD uses a different criterion from SVM. It seeks to achieve the goa by maximizing the average distance rather than the minimum distance among the casses. Simiar to the ksvm function from 1

2 kernab, which has been widey used in SVM anaysis, we deveoped a function caed kdwd in this package for doing DWD anaysis. To sove muticass cassification probem, two different methods are used. The first one is to use the one-versus-one approach, in which cassifiers are trained on each pair of casses and the cass abe is predicted by a voting scheme. The second one is to buid a singe cassifier incuding a casses simutaneousy and sove a big optimization probem. Every DWD anaysis requires two eements from a dataset: (1) a matrix of predictor, which shoud be in the form of an n d matrix, where n represents the sampe size and d represents the dimension. (2) a response vector of ength n with each eement corresponding to one sampe. The appropriate scaing steps can be taken by using scaed option. It shoud be noted that in the current version of dwd, missing vaues are not aowed, and must be imputed prior to anaysis. In this vignette, we wi use the iris dataset as iustration. > ibrary(dwd) > data(iris) iris is a data frame of 4 measurements across 150 sampes with response variabe being the species of each sampe. The goa of interest is to predict the species based on the given measurement. This is a 3-cass cassification probem since there are tota 3 different species in the sampes which are provided in iris$species. > tabe(iris$species) setosa versicoor virginica An exampe for the kdwd function is shown beow. > resuts <- kdwd(species~., data=iris) > predict(resuts,iris) The basic output from kdwd is an object of cass kdwd. Showing objects of cass kdwd wi print detais on the resuts for a cassifier incuded in the mode. For each cassifier, the optima soution of the parameters are dispayed aong with the fina prima and dua objective vaues. The cross vaidation error rate can aso be returned with the argument cross = k, where k is the number of the fods used in the cross-vaidation. DWD is different from SVM in that it aows a data points rather than ony support vectors to have direct infuence on the determination of the decision boundary. So unike ksvm, the outputs of kdwd do not incude any information about the support vectors. The current version of dwd can ony sove inear DWD probems. We are sti working on incorporating into the kerne trick such that it can be used to sove the genera noninear probems as we. More exampes of the impementation and the changes to functiona arguments of kdwd are detaied in hep documents. 3 The Second Order Cone Programming sover SOCP is a noninear convex probem that incudes inear and (convex) quadratic programs as specia cases. It can be formuated in the foowing equations: n q min c q i, xq i + c, x + c u, x u i=1 2

3 s.t. n q A q i xq i + A x + A u x u = b i=1 x q i Kq i q i, x K n, x u R nu Here c q i, xq i are vectors in R q i and K q i q is the quadratic or second-order cone defined by K q i q := {x = [x 0 ; x] R q i : x 0 x T x}. Simiary, c, x are vectors of dimension n, K n is the nonnegative orthant R n +, and c u, x u are vectors of dimension n u. The dua probem associated with the probem above is: s.t. min b T y (A q i )T y + z q i = cq i, zq i Kq i q, i = 1,, n q (A ) T y + z = c, z K n, (A u ) T y = c, y R m. DWD is one of the important appications of SOCP, but SOCP can be used to sove many other probems as we. The package kdwd provides a stand-aone SOCP sover caed sqp. The agorith impemented in sqp is an infeasibe prima-dua path-foowing agorithm, described in detai in Toh et a.. The basic idea is that, at each iteration, we first compute a predictor search direction aimed at decreasing the gap between the prima and dua objective vaues. After that, the agorithm generates a corrector step with the intention of keeping the iterates cose to the centra path. The most crucia part is to sove a inear system which is especiay chaenging in situations when a big sparse matrix is incuded. To increase the speed and effeciency, the sparse matrix package Matrix is incorporated to dea with high dimensiona arge datasets. The caing syntax of is sqp as foows: Input arguments: output = sqp(bk, At, C, b, OPTIONS, X0, y0, Z0) bk: a ist describing the bock structure of the SOCP probem, which wi be described in detai in the foowing. At, C, b: SOCP data. OPTIONS: a ist of parameters (optiona). X0,y0,Z0: an initia iterate. If the input argument OPTIONS is omitted, defaut vaues specified in the function sqparameters are used. Output arguments: x: the optima soution to the SOCP. y,z: the dua soutions. info: summary information. runhist: run history. 3

4 The names chosen for the output arguments expain their contents. The argument inf o is a ist containing performance information such as inf o$termcode, inf o$obj, inf o$gap, inf o$pinf eas, inf o$dinf eas, inf o$cputime whose meanings are expained in sqp. The argument runhist is a ist which records the history of various performance measures during the run; for exampe, runhist$gap records the compementarity gap at each interior-point iteration. Whie (X, y, Z) normay gives approximatey optima soutions, if inf o$termcode is 1 the probem is suspected to be prima infeasibe and (y, Z) is an approximate certificate of infeasibiity, with b T y = 1, Z in the appropriate cone, and A T y + Z sma, whie if info$termcode is 2 the probem is suspected to be dua infeasibe and X is an approximate certificate of infeasibiity, with C, X = 1, X in the appropriate cone, and AX sma. The impementation requires the user to specify the bock structure of the given SOCP probems. Let L be the tota number of bocks in the SOCP probem. The bock structure of the probem data is described by a ist of ength L caed bk. The content of each of the eements of the ist is given as foows. If the ith bock is a quadratic bock consisting of p sub-bocks, of dimensions q i1, q i2,, q ip such that p k=1 = q i, then bk$type[i] = q bk$size[[i]] = c(q i1, q i2,, q ip ) At[[i]] = [q i m] C[i], X[i], Z[i] = [q 1 1] If the kth bocks is the inear bock, then bk$type[k] = bk$size[[k]] = n At[[i]] = [n m] C[i], X[i], Z[i] = [n 1] Simiary, if the kth bocks is the unrestricted bock, then bk$type[k] = u bk$size[[k]] = n u At[[i]] = [n u m] C[i], X[i], Z[i] = [n u 1] The foowing exampe shows how sqp ca a data fie that is stored in the package > data(sqpdata) > son <- sqp(bk=sqpdata$bk,at=sqpdata$at,c=sqpdata$c,b=sqpdata$b,x0=sqpdata$x0, + y0=sqpdata$y0,z0=sqpdata$z0) 4 The Quadratic Programming sover The optimization program invoved in SVM is caed Quadratic Programming (QP) which is a specia case of SOCP. Simiar to SOCP, the appication of QP is not imited to SVM, it 4

5 can be used to many other areas as we. The package dwd aso provides a QP sover based on SOCP for soving the foowing probem min d T b bt Db s.t.a T b b 0. SOCP based QP sover is formatted in a simiar way but has been shown to be more efficient than existing QP sover: sove.qp in quadprob package. The main routine is sove_qp_socp, whose caing syntax is as foows: Input arguments. output = sove QP SOCP(Dmat, dvec, Amat, bvec) Dmat: matrix appearing in the quadratic function to be minimized. dvec: vector appearing in the quadratic function to be minimized. Amat: matrix defining the constraints under which we want to minimize the quadratic function. bvec: vector hoding the vaues of b 0. Output arguments. soution: vector containing the soution of the QP probem. The foowing exampe shows how sove_qp_socp ca a simuated data fie > Dmat <- matrix(0,3,3) > diag(dmat) <- 1 > dvec <- c(0,5,0) > Amat <- matrix(c(-4,-3,0,2,1,0,0,-2,1),3,3) > bvec <- c(-8,2,0) > sove_qp_socp(dmat,dvec,amat,bvec) 5 References Marron, J. S., Todd, M. J. and Ahn, J. (2007) Distance-Weighted Discrimination, Journa of the American Statistica Association, Vo. 102, No. 480, pp H. Huang, Y. Liu, Y. Du, C.M. Perou, D.N. Hayes, M.J. Todd, and J.S. Marron, Muticass distance weighted discrimination with appications to batch adjustment, submitted. K.C. Toh, M.J. Todd, and R.H. Tutuncu, SDPT3 a Matab software package for semidefinite programming, Optimization Methods and Software, 11 (1999), pp Aexandros Karatzogou, Aex Smoa, Kurt Hornik, Kerne-based Machine Learning Lab, CRAN - Package kernab. Berwin A. Turach, quadprog: Functions to sove Quadratic Programming Probems, CRAN - Package quadprog 5