Applications of Support Vector Machines for Pattern Recognition: A Survey

Transcription

1 Applcatons of Support Vector Machnes for Pattern Recognton: A Survey Hyeran Byun and Seong-Whan Lee 2 Department of Computer Scence, Yonse Unversty Shnchon-dong, Seodaemun-gu, Seoul , Korea hrbyun@cs.yonse.ac.kr 2 Department of Computer Scence and Engneerng, Korea Unversty Anam-dong, Seongbuk-gu, Seoul 36-70, Korea swlee@mage.korea.ac.kr Abstract. In ths paper, we present a comprehensve survey on applcatons of Support Vector Machnes (SVMs) for pattern recognton. Snce SVMs show good generalzaton performance on many real-lfe data and the approach s properly motvated theoretcally, t has been appled to wde range of applcatons. Ths paper descrbes a bref ntroducton of SVMs and summarzes ts numerous applcatons. Introducton SVMs are a new type of pattern classfer based on a novel statstcal learnng technque that has been recently proposed by Vapnk and hs co-workers [-3]. Unlke tradtonal methods (e.g. Neural Networks), whch mnmze the emprcal tranng error, SVMs am at mnmzng an upper bound of the generalzaton error through maxmzng the margn between the separatng hyperplane and the data [4]. Snce SVMs are known to generalze well even n hgh dmensonal spaces under small tranng sample condtons [5] and have shown to be superor to tradtonal emprcal rsk mnmzaton prncple employed by most of neural networks [6], SVMs have been successfully appled to a number of applcatons rangng from face detecton, verfcaton, and recognton [5-,26,50-56,76-80,83], object detecton and recognton [2-5,24,47,57], handwrtten character and dgt recognton [6-8,45], text detecton and categorzaton [9,58-6], speech and speaker verfcaton, recognton [20-23], nformaton and mage retreval [33-36,87], predcton [37-4] and etc.[22,27-32,4,42,53,62,64,65,74]. In ths paper, we am to gve a comprehensve survey on applcatons of SVMs for pattern recognton. Ths paper s organzed as follows. We gve a bref explanaton on SVMs n Secton 2 and a detaled revew of SVMs-related technques n Secton 3. Secton 4 descrbes the lmtatons of SVMs. We conclude ths paper n Secton 5. S.-W. Lee and A. Verr (Eds.): SVM 2002, LNCS 2388, pp , Sprnger-Verlag Berln Hedelberg 2002

2 24 Hyeran Byun and Seong-Whan Lee 2 Support Vector Machnes Classcal learnng approaches are desgned to mnmze error on the tranng dataset and t s called the Emprcal Rsk Mnmzaton (ERM). Those learnng methods follow the ERM prncple and neural networks are the most common example of ERM. On the other hand, the SVMs are based on the Structural Rsk Mnmzaton (SRM) prncple rooted n the statstcal learnng theory. It gves better generalzaton abltes (.e. performances on unseen test data) and SRM s acheved through a mnmzaton of the upper bound (.e. sum of the tranng error rate and a term that depends on VC dmenson) of the generalzaton error [-3,43-45]. 2. Lnear Support Vector Machnes for Lnearly Separable Case The basc dea of the SVMs s to construct a hyperplane as the decson plane, whch separates the postve (+) and negatve (-) classes wth the largest margn, whch s related to mnmzng the VC dmenson of SVM. In a bnary classfcaton problem d where feature extracton s ntally performed, let us label the tranng data x R wth a label y {, + }, for all the tranng data =,..., l, where l s the number of data, and d s the dmenson of the problem. When the two classes are lnearly separable n R, we wsh to fnd a separatng hyperplane whch gves the smallest general- d zaton error among the nfnte number of possble hyperplanes. Such an optmal hyperplane s the one wth the maxmum margn of separaton between the two classes, where the margn s the sum of the dstances from the hyperplane to the closest data ponts of each of the two classes. These closest data ponts are called Support Vectors (SVs). The sold lne on Fg. represents the optmal separatng hyperplane. Fg.. Lnear separatng hyperplanes for the separable case. The support vectors are crcled (taken from [44])

3 Applcatons of Support Vector Machnes for Pattern Recognton 25 Let s suppose they are completely separated by a d-dmensonal hyperplane descrbed by w x + b = 0 The separaton problem s to determne the hyperplane such that w x + b + for postve examples and w x + b for negatve examples. Snce the SVM fnds the hyperplane, whch has the largest margn, t can be found by mnmzng 2 w 2 mn w, b Φ ( w ) = w 2 2 The optmal separatng hyperplane can thus be found by mnmzng equaton (2) under the constrant (3) to correctly separate the tranng data. y ( x w + b) 0, Ths s a Quadratc Programmng (QP) problem for whch standard technques (Lagrange Multplers, Wolfe dual) can be used [43,5,69,70]. The detaled explanaton on QP problems and alternatve researches are descrbed n Sub-secton 2.4. () (2) (3) 2.2 Lnear Support Vector Machnes for Non-separable Case In practcal applcatons for real-lfe data, the two classes are not completely separable, but a hyperplane that maxmzes the margn whle mnmzng a quantty proportonal to the msclassfcaton errors can stll be determned. Ths can be done by ntroducng postve slack varables ξ n constrant (3), whch then becomes y ( x w + b) ξ, (4) If an error occurs, the correspondng ξ must exceed unty, so ξ s an upper bound for the number of msclassfcaton errors. Hence the objectve functon (2) to be mnmzed can be changed nto mn{ w 2 / 2 + C l = ξ } where C s a parameter chosen by the user that controls the tradeoff between the margn and the msclassfcaton errors. A larger C means that a hgher penalty to msclassfcaton errors s assgned. Mnmzng equaton (5) under constrant (4) gves the Generalzed Separatng Hyperplane. Ths stll remans a QP problem. The nonseparable case s llustrated n Fg. 2. (5)

4 26 Hyeran Byun and Seong-Whan Lee Fg. 2. Lnear separatng hyperplane for the non-separable case (taken from [44]) 2.2. Nonlnear Support Vector Machnes and Kernels Nonlnear Support Vector Machnes An extenson to nonlnear decson surfaces s necessary snce real-lfe classfcaton problems are hard to be solved by a lnear classfer [4]. When the decson functon s not a lnear functon of the data, the data wll be mapped from the nput space nto a hgh dmensonal feature space by a nonlnear transformaton. In ths hgh dmensonal feature space, the generalzed optmal separatng hyperplane shown n Fg 3 s constructed [43]. Cover s theorem states that f the transformaton s nonlnear and the dmensonalty of the feature space s hgh enough, then nput space may be transformed nto a new feature space where the patterns are lnearly separable wth hgh probablty [68]. Ths nonlnear transformaton s performed n mplct way through so-called kernel functons. (a) nput space (b) feature space Fg. 3. Feature space s related to nput space va a nonlnear map Φ, causng the decson surface to be nonlnear n the nput space (taken from [33]) Inner-Product Kernels In order to accomplsh nonlnear decson functon, an ntal mappng Φ of the data nto a (usually sgnfcantly hgher dmensonal) Eucldean space H s performed as

5 Applcatons of Support Vector Machnes for Pattern Recognton 27 ( j Φ : R n H, and the lnear classfcaton problem s formulated n the new space wth dmenson d. The tranng algorthm then only depends on the data through dot product n H of the form Φ x ) Φ( x ). Snce the computaton of the dot products s prohbtve f the number of tranng vectors Φ ( x ) s very large, and snce Φ s not known a pror, the Mercer s theorem [44] for postve defnte functons allows to replace Φ x ) Φ( x ) by a postve defnte symmetrc kernel functon K x, x ), that s, ( j ( j K x, x ) = Φ( x ) Φ( x ). In tranng phase, we need only kenel functon K x, x ) and ( j j ( j ( j j Φ( x ) does not need to be known snce t s mplctly defned by the choce of kernel K x, x ) = Φ( x ) Φ( x ). The data can become lnearly separable n feature space although orgnal nput s not lnearly separable n the nput space. Hence kernel substtuton provdes a route for obtanng nonlnear algorthms from algorthms prevously restrcted to handlng lnear separable datasets [75]. The use of mplct kernels allows reducng the dmenson of the problem and overcomng the so-called dmenson curse [3]. Varant learnng machnes are constructed accordng to the dfferent kernel functon K( x, x j ) and thus construct dfferent hyperplane n feature space. Table shows three typcal kernel functons. Table. Summary of nner-product kernels [68] Kernel functon Polynomal kernel Gaussan (Radal-bass) kernel Mult-layer perceptron (sgmod) Inner Product Kernel K( x, x ), =,2,..., N T d K ( x, x ) = ( x x + ) 2 2 K( x, x ) = exp( x x / 2σ ) T K( x, x) = tanh( β 0x x + β), β and 0 β are decded by the user 2.3 Quadratc Programmng Problem of SVMs 2.3. Dual Problem In equaton (2) and (3), the optmzaton goal Φ( x ) s quadratc and the constrants are lnear, t s a typcal QP. Gven such a constraned optmzaton problem, t s possble to construct another problem called the dual problem. We may now state the dual problem: gven the tranng sample {( x } N, d ), fnd the = Lagrange multplers { } N that maxmze the objectve functon a = N N N T Q( α) = α α α d d x x = 2 = = subject to the constrants j j j (6)

6 28 Hyeran Byun and Seong-Whan Lee N () α = d = 0 (2) α 0 for =,2,..., N We also may formulate the dual problem for non-separable pattern usng the method of Lagrange multplers. Gven the tranng sample {( x } N, d ), fnd the Lagrange multplers { a } N that maxmze the objectve = = functon N N N T Q( α) = α α α d d x x = 2 = = subject to the constrants N () α d = 0 = (2) 0 α C for =,2,..., N j j j where C s a user-chosen postve parameter. The objectve functon Q (α ) to be maxmzed for the case of non-separable problems n the dual problem s the same as the case for the separable problems except for a mnor but mportant dfference. The dfference s that the constrants α 0 for the separable case s replaced wth the more strngent constrant 0 α C for the non-separable case [68] How to Solve the Quadratc Problem A number of algorthms have been suggested for solvng the dual problems. Tradtonal QP algorthms [7,72] are not sutable for large sze problems because of the followng reasons [70]: They requre that the kernel matrx be computed and stored n memory and t requres extremely large memory. These methods nvolve expensve matrx operatons such as the Cholesky decomposton of a large submatrx of the kernel matrx. For practtoners who would lke to develop ther own mplementaton of an SVM classfer, codng these algorthms s very dffcult. A few attempts have been made to develop methods that overcome some or all of these problems. Osuna et al. proved a theorem, whch suggests a whole new set of QP problems for SVM. The theorem proves that the large QP problem can be broken down nto a seres of smaller QP sub-problems. As long as at least one example that volate the Karush-Kuhn-Tucker (KKT) condtons s added to the examples for the prevous sub-problem, each step wll reduce the cost of overall objectve functon and mantan a feasble pont that obeys all of the constrants. Therefore, a sequence of QP sub-problems that always add at least one volator wll be guaranteed to converge [5]. Platt proposed a Sequental Mnmal Optmzaton (SMO) to quckly solve the SVM QP problem wthout any extra matrx storage and wthout usng numercal QP (7)

7 Applcatons of Support Vector Machnes for Pattern Recognton 29 optmzaton steps at all. Usng Osuna s theorem to ensure convergence, SMO decomposes the overall QP problem nto QP sub-problems. The dfference of the Osuna s method s that SMO chooses to solve the smallest possble optmzaton problem at every step. At each step, ()SMO chooses two Lagrange multplers to jontly optmze, (2)fnds the optmal values for these multplers, and (3)updates the SVMs to reflect the new optmal values. The advantage of SMO s that numercal QP optmzaton s avoded entrely snce solvng for two Lagrange multplers can be done analytcally. In addton, SMO requres no extra matrx storage at all. Thus, very large SVM tranng problems can ft nsde the memory of a personal computer or workstaton [69]. Keert et al. [73] ponted out an mportant source of confuson and neffcency n Platt s SMO algorthm that s caused by the use of sngle threshold value. Usng clues from the KKT condtons for the dual problem, two threshold parameters are employed to derve modfcatons of SMO. 2.4 SVMs Appled to Mult-Class Classfcaton The basc SVMs are for two-class problem. However t should be extended to multclass to classfy nto more than two classes [45,46]. There are two basc strateges for solvng q-class problems wth SVMs Mult-class SVMs: One to Others [45] Take the tranng samples wth the same label as one class and the others as the other class, then t becomes a two-class problem. For the q-class problem (q >2), q SVM classfers are formed and denoted by SVM, =,2,, q. As for the testng sample x, * * d ( x) = w x + b can be obtaned by usng SVM. The testng sample x belongs to jth class where d ( x) = max d ( x) j = ~ q Mult-class SVMs: Parwse SVMs 2 In the parwse approach, q machnes are traned for q-class problem [47]. The parwse classfers are arranged n trees, where each tree node represents an SVM. A bottom-up tree, whch s smlar to the elmnaton tree used n tenns tournaments was orgnally proposed n [47] for recognton of 3D objects and was appled to face recognton n [9,48]. A top-down tree structure has been recently publshed n [49]. There s no theoretcal analyss of the two strateges wth respect to classfcaton performance [0]. Regardng the tranng effort, the one-to-others approach s preferable snce only q SVMs have to be traned compared to q 2 SVMs n the parwse approach. However, at runtme both strateges requre the evaluaton of q- SVMs [0]. Recent experments on people recognton show smlar classfcaton performances for the two strateges [24]. (8)

8 220 Hyeran Byun and Seong-Whan Lee vs 4 not not vs vs not 2 not 4 not not 3 3 vs vs 3 2 vs (a) example of top-down tree structure (b) example of bottom-up tree structure Fg. 4. Tree structure for mult-class SVMs. (a) The decson Drected Acyclc Graph (DAG) for fndng the best class out of four classes. The equvalent lst state for each node s shown next to that node (taken from [49]), (b) The bnary tree structure for 8 classes. For a comng test data, t s compared wth each two pars, and the wnner wll be tested n an upper level untl the top of the tree s reached. The numbers -8 encode the classes (taken from [48,9]) 3 Applcatons of SVMs for Pattern Recognton In ths Secton, we survey applcatons of pattern recognton usng SVMs. We classfy exstng applcatons nto seven categores accordng to ther ams. Some methods, whch are not ncluded n major categores, are classfed nto other methods and there can be more applcaton areas whch are not ncluded n ths secton. Table 2 shows the summary of major SVMs-related applcatons Table 2. Summary of major SVMs-related applcatons Categores Face Detecton Major dfferences Frontal face detecton To speed up face detecton on skn segmented regon Summary of applcatons - appled SVM to face detecton frst - suggested novel decomposton algorthm [5] - face detecton/eye detecton [52] - ICA features as an nput [83] - orthogonal Fourer-Melln Moments as an nput [] - overcomplete wavelet decomposton as an nput [76]

9 Applcatons of Support Vector Machnes for Pattern Recognton 22 Face Detecton Face Vaerfcaton Object Recognton Object Recognton To speed up face detecton Mult-vew face detecton Combnaton of multple methods M2VTS database (EER=3.7%) M2VTS database (EER=.0) ORL database (Recognton Rate 97%) ORL database (Recognton Rate 98%) Own database (Recognton Rate 90%) Own database Own database (People Rec. Rate: 99.5% ; Pose Rec. Rate : 84.5%) - eyes-n-whole and face templates as preprocessng [78] - calculated reduced support vectors [79] - constructed separate SVMs for face detecton on dfferent vews [26, 54, 80] - egenface for a coarse face detecton followed by an SVM for fne detecton [55] - Majorty votng on outputs of 5 dfferent kernels of SVMs [77] - reformulated Fsher s lnear dscrmnant rato to quadratc problem to apply SVM [8] - showed that the performance of SVMs was relatvely nsenstve to the representaton space(pca, LDA) and preprocessng steps [5] - bottom-up tree mult-class method - nput feature for SVM was extracted by PCA [9,48] - suggested the modfed kernel to explore spatal relatonshps of the facal features [56] - top-down tree mult-class method - 3D range data for 3D shape features and 2D textures are projected onto PCA subspace and PC s are nput to SVMs [50] - compared component-based features wth global feature as an nput of SVM - SVM gave better performance when component-based features were used [0] - people recognton(4 people) - pose recognton(4 poses) - compared bottom-up and top-down mult-class SVM and the results showed smlar performance of two methods [24]

10 222 Hyeran Byun and Seong-Whan Lee Object Recognton Handwrtten Character/ Dgt Recognton COIL database (7200 mages 72 vews per each objects 00 objects) Own database Own database Own database Own database, Character recognton NIST database, Handwrtten dgt recognton (Rec. Rate:98.06%) - showed that SVMs gave a good performance for 3D object recognton from sngle vew - tested on many syntheszed mages wth nose, occluson, and pxel shftng [47] - llustrated the potental of SVMs n terms of the number of tranng vews per object(from 36 vews to 2 vews) for 3D object recognton - showed that the performance was decreased much when the number of tranng vews were less than 8 vews [5] - people detecton - recognzed trajectory of movng people [57] - detected movng vehcle - constructed the problem as 2-class problem by classfyng movng vehcle from shadows [3] - radar target recognton [4] - pedestran recognton [84] - used local vew and global vew for character recognton - local vew model for nput normalzaton - SVM, global vew model for recognton [6] - combned structural and statstcal features are nput to sngle SVM classfer - constructed dfferent SVM classfer for each feature and then combned 2 dfferent SVMs by rule-based reasonng - sngle SVM gave better performance[7]

11 Applcatons of Support Vector Machnes for Pattern Recognton 223 Handwrtten Dgt Recognton Speaker/ Speech Recognton Image Retreval Predcton NIST database, Handwrtten dgt recognton Utterance verfcaton for Speech recognton Speaker verfcaton/ recognton Brodatz texture database Correl mage database Fnancal tme seres predcton Bankruptcy predcton - compared the performance accordng to: effect of nput dmenson, effect of the kernel functon(lnear, Polynomal, Gaussan), comparson of dfferent classfer (ML, MLP, SOM+LVQ, RBF, SVM), comparson of 3 types of mult-class SVM(one-to-others, parwse, decson tree)[45] - extracted bologcally plausble features - showed that ther extracted features were lnearly separable features by usng lnear SVM classfer [8] - SVMs are used to accept keyword or reject non-keyword for speech recognton [22] - PolyVar telephone database s used [2] - new method for normalzng polynomal kernel to use wth SVMs, YOHO database, text ndependent, best EER=0.34% [23] - combned Gaussan Mxture Model n SVM outputs - text ndependent speaker verfcaton - best EER =.56% [20] - boundares between classes were obtaned by SVM [33] - SVMs were used to separate two classes of relevant and rrelevant mages [34, 36, 87] - C-ascendng SVMs were suggested based on the assumpton that t was better to gve more weghts on recent data than dstant data [4] - suggested to select sutable nput varables that tends to dscrmnate wthn the SVM kernel used [40]

12 224 Hyeran Byun and Seong-Whan Lee Gender classfcaton - FERET database : 3.4% error rate - compared SVM-based method to: lnear, quadratc, FLD, RBF, ensemble-rbf [27] Goal detecton - ghost goal detecton [64] Other Classfcatons Fngerprnt classfcaton Data condensaton Face pose classfcaton Other Classfcatons - Types of fngerprnts were classfed nto 5 classes [62] - extracted data ponts from huge databases and the accuracy of a classfer traned on ths reduced sets were comparable to results from tranng wth the entre data sets [42] - on FERET database [3,8] - bullet-hole classfcaton for autoscorng [32] - whte blood cell classfcaton [88] - spam categorzaton [89] - cloud classfcaton [74] - hyperspectral data classfcaton [28] - storm cell classfcaton [29] - mage classfcaton [30] 3. Face Detecton and Recognton Face detecton, verfcaton and recognton are one of the popular ssues n bometrcs, dentty authentcaton, access control, vdeo survellance and human-computer nterfaces. There are many actve researches n ths area for all these applcatons use dfferent methodologes. However, t s very dffcult to acheve a relable performance. The reasons are due to the dffculty of dstngushng dfferent persons who have approxmately the same facal confguraton and wde varatons n the appearance of a partcular face. These varatons are because of changes n pose, llumnaton, facal makeup and facal expresson [50]. Also glasses or a moustache makes dffcult to detect and recognze faces. Recently many researchers appled SVMs to face detecton, facal feature detecton, face verfcaton, recognton and face expresson recognton and compared ther results wth other methods. Each method used dfferent nput features, dfferent databases, and dfferent kernels to SVMs classfer. Face Detecton: The applcaton of SVM n frontal face detecton n mage was frst proposed by Osuna et al. [5]. The proposed algorthm scanned nput mages wth a 9x9 wndow and a SVM wth a 2nd-degree polynomal as kernel functon s traned wth a novel decomposton algorthm, whch guarantees global optmalty. To avod exhaustve scannng for face detecton, SVMs are used on dfferent features of seg-

13 Applcatons of Support Vector Machnes for Pattern Recognton 225 mented skn regons. Kumar and Poggo [52] recently ncorporated Osuna et al. s SVM algorthm n a system for real-tme trackng and analyss of faces on skn regon and also to detect eyes. In [83], SVMs classfed the ICA features after applyng skn color flter for face detecton and they showed that the used ICA features gave better generalzaton capacty than by tranng SVM drectly on the mage data. In Terrllon et al. [], they appled SVM to nvarant Orthogonal Fourer-Melln Moments as features for bnary face/non-face classfcaton on skn color-based segmented mage and compared the performance of SVM face detector to mult-layer perceptron n terms of Correct Face Detector (CD) and Correct Face Rejecton (CR). Also to speed up the face detecton, n [78], two templates : eyes-n-whole and face are used for flterng out face canddates for SVMs to classfy face and non-face classes. Another method to mprove the speed of the SVM algorthm, [79] found a set of reduced support vectors (RVs) whch are calculated from support vectors. RVs are used to speed up the calculaton sequentally. SVMs have also been used for mult-vew face detecton by constructng separate SVMs specfc to dfferent vews based on the pose estmaton. For face recognton, frontal vew SVM-based face recognzer s used f the detected face s n frontal vew after head pose estmaton [26,54,80]. Also combned methods are tred to mprove the performance for face detecton. In [55], they tested the performance of three face detecton algorthms, egenface method, SVM method and combned method n terms of both speed and accuracy for mult-vew face detecton. The combned method conssted of a coarse detecton phase by egenface method followed by a fne SVM phase and could acheve an mproved performance by speedng up the computaton and keepng the accuracy. Bucu et al. [77] attempted to mprove the performance of face detecton by majorty votng on the outputs of 5 dfferent kernels of SVM. Papageorgo et al. [76] appled SVM to overcomplete wavelet representaton as nput data to detect faces and people and Rchman et al. [82] appled SVM to fnd nose crosssecton for face detecton. Face Recognton and Authentcaton: The recognton of face s a well-establshed feld of research and a large number of algorthms have been proposed n the lterature. Machne recognton of faces yelds problems that belong to the followng categores whose objectves are brefly outlned [8]: Face Recognton: Gven a test face and a set of reference faces n a database, fnd the N most smlar reference faces to the test face. Face authentcaton: Gven a test face and a reference one, decde f the test face s dentcal to the reference face. Guo et al. [9,48] proposed mult-class SVM wth a bnary tree recognton strategy for face recognton. Normalzed feature extracted by PCA was the nput of the SVM classfer. For face recognton, the papers used dfferent nputs to an SVM classfer. Hesele et al. [0] developed a component-based method and global method for face recognton. In the component-based system they extracted facal components and combned them nto a sngle feature vector, whch s classfed by SVM. The global system used SVM to recognze faces by classfyng a sngle feature vector consstng of the gray values of the whole face mage. Ther results showed that component-

14 226 Hyeran Byun and Seong-Whan Lee based method outperformed the global method. Km et al. [56] modfed SVM kernel to explore spatal relatonshps among potental eye, nose, and mouth object and compared ther kernel wth exstng kernels. Wang et al. [50] proposed a face recognton algorthm based on both of 3D range and 2D gray-level facal mages. 2D texture and 3D shape features are projected onto PCA subspace and then ntegrated 2D and 3D features are an nput to SVM to recognze faces. For face authentcaton and recognton, Jonsson et al. [5] presented that SVMs extracted the relevant dscrmnatve nformaton from the tranng data and the performance of SVMs was relatvely nsenstve to the representaton space and preprocessng steps. Tefas et al. [8] reformulated Fsher s dscrmnant rato to a quadratc optmzaton problem subject to a set of nequalty constrants to enhance the performance of morphologcal elastc graph matchng for frontal face authentcaton. SVMs, whch fnd the optmal separatng hyperplane are constructed to solve the reformulated quadratc optmzaton problem for face authentcaton. 3.2 Object Detecton and Recognton Object detecton or recognton ams to fnd and track movng people or traffc stuaton for survellance or traffc control. Nakajma et al. [24] developed people recognton and pose estmaton as a mult-class classfcaton problem. Ths paper used bottom-up and top-down mult-class SVMs and the two types of SVM classfers showed very smlar performance. 3D object recognton was developed n [5] and [47]. Both of them used COIL object database, whch contaned 7200 mages of 00 objects wth 72 dfferent vews per each object. Roobaert et al. [5] proposed 3D object recognton wth SVMs to llustrate the potental of SVMs n terms of the number of tranng vews per object. Ther result showed that the performance was decreased much when the number of tranng vews was less than 8 vews. M. Pontl and A.Verr [47] used lnear SVMs for aspect-based 3D object recognton from a sngle vew wthout feature extracton, data reducton and estmatng pose. They tested SVM method on the syntheszed mages of COIL database wth nose, occluson, and pxel shfts and got very good performance. Pttore et al. [57] proposed a system that was able to detect the presence of movng people, represented the event by usng an SVM for regresson, and recognzed trajectory of vsual dynamc events from an mage sequence by SVM classfer. Gao et al. [3] proposed a shadow and headlghts elmnaton algorthm by consderng ths problem as a 2-class problem. That s, the SVM classfer was used to detect real movng vehcles from shadows. Some other object recogntons were on radar target recognton[4] and pedestran recognton [84]. 3.3 Handwrtten Character/Dgt Recognton Among the SVM-based applcatons, on the handwrtten dgt recognton problem, SVMs have shown to largely outperform all other learnng algorthms, f one excludes the nfluence of doman-knowledge [5]. A major problem n handwrtng recognton s the huge varablty and dstortons of patterns. Elastc models based on local observatons and dynamc programmng such as HMM are effcent to absorb ths varabl-

15 Applcatons of Support Vector Machnes for Pattern Recognton 227 ty. But ther vson s local [6]. To combne the power of local and global characterstcs, Chosy et al. [6] used NSPH-HMM for local vew and normalzaton. SVM for global vew s used for character recognton after normalzaton of NSPH-HMM. For handwrtten dgt recognton, SVMs are used n [7], [8] and [45]. Gorgevk et al. [7] used two dfferent feature famles (structural features and statstcal features) for handwrtten dgt recognton usng SVM classfer. They tested sngle SVM classfer appled on the both feature famles as one set. Also two feature sets are forwarded to 2 dfferent SVM classfers and obtaned results are combned by rule-based reasonng. The paper showed that sngle SVM classfer was better than rule-based reasonng appled to 2 ndvdual classfers. Teow et al. [8] had developed a vson-based handwrtten dgt recognton system, whch extracts features that are bologcally plausble, lnearly separable and semantcally clear. In ther system, they showed that ther extracted features were lnearly separable features over a large set of tranng data n a hghly non-lnear doman by usng lnear SVM classfer. In [45], they showed the performance of handwrtten dgt recognton accordng to () the effect of nput dmenson, (2) effect of kernel functons, (3) comparson of dfferent classfers(ml, MLP, SOM+LVQ, RBF, SVM) and (4) comparson of three types of multclass SVMs(one-to-others, par-wse, decson tree). 3.4 Speaker/Speech Recognton In speaker or speech recognton problem, the two most popular technques are dscrmnatve classfers and generatve model classfers. The methods usng dscrmnatve classfers consst of decson tree, neural network, SVMs, and etc. The wellknown generatve model classfcaton approaches nclude Hdden Markov models (HMM) and Gaussan Mxture models (GMM) [20]. For tranng and testng data, there are text dependent and text ndependent data. Bengo et al.[2] and Wan et al.[23] used SVMs for speaker verfcaton on dfferent data sets. In [2], they expermented on text dependent and text ndependent data and replaced the classcal thresholdng rule wth SVMs to decde accept or reject for speaker verfcaton. Text ndependent tasks gave sgnfcant performance mprovements. [23] proposed a new technque for normalzng the polynomal kernel to use wth SVMs and tested on YOHO database. Dong et al. [20] reported on the development of a natural way of achevng combnaton of dscrmnatve classfer and generatve model classfers by embeddng GMM n SVM outputs, thus created a contnuous densty support vector machne (CDSVM) for text ndependent speaker verfcaton. For utterance verfcaton whch s essental to accept keywords and reject non-keywords on spontaneous speech recognton, Ma et al. [22] have traned and tested SVMs classfer to the confdence measurement problem n speech recognton. 3.5 Informaton and Image Retreval Content-based mage retreval s emergng as an mportant research area wth applcatons to dgtal lbrares and multmeda databases[33]. Guo et al. [33] proposed a new metrc, dstance-from-boundary to retreve the texture mage. The boundares between classes are obtaned by SVM. To retreve more mages relevant to the query mage,

16 228 Hyeran Byun and Seong-Whan Lee SVM classfer was used to separate two classes of relevant mages and rrelevant mages n [36,34,87]. Drucker et al. [36], Tan et al. [34] and Zhang et al. [87] proposed that SVMs automatcally generated preference weghts for relevant mages. The weghts were determned by the dstance of the hyperplane, whch was traned by SVMs usng postve examples (+) and negatve examples (-). 3.6 Predcton The am of many nonlnear forecastng methods[37,39,40,4] s to predct next ponts of tme seres. Tay and Cao [4] proposed C-ascendng SVMs by ncreasng the value of C, the relatve mportance of the emprcal rsk wth respect to the growth of regularzaton term. Ths dea was based on the assumpton that t was better to gve more weghts on recent data than dstant data. Ther results showed that C-ascendng SVMs gave better performance than standard SVM n fnancal tme seres forecastng. Fan et al. [40] had adopted SVM approach to the problem of predctng corporate dstress from fnancal statements. For ths problem, the choce of nput varables (fnancal ndcators) affects the performance of the system. Ths paper had suggested selectng sutable nput varables that maxmze the dstance of vectors between dfferent classes, and mnmze the dstance wthn the same class. Eucldean dstance based nput selecton provded a choce of varables that tends to dscrmnate wthn the SVM kernel used. 3.7 Other Applcatons There are many more applcatons of SVMs for pattern recognton problems. Yang et al. [27] have nvestgated SVMs for vsual gender classfcaton wth low-resoluton thumbnal faces (2-by-2 pxels) processed from,755 mages from the FERET face database. Then they traned and tested each classfer wth the face mages usng fve fold cross valdaton. The performance of SVM (3.4% error) was shown to be superor to tradtonal pattern classfers (lnear, quadratc, FLD, RBF, ensemble- RBF). Gutta et al. [3] have appled SVMs to face pose classfcaton on FERET database and ther results yelded 00% accuracy. Also Huang et al. [8] appled SVMs to classfy nto 3 dfferent knds of face poses. Yao et al. [62] proposed to classfy fngerprnt types nto 5 dfferent fngerprnt classes. SVMs were traned on combnng flat and structured representaton and showed good performance and promsng approach for fngerprnt classfcaton. In addton, SVMs had been appled to many other applcatons such as data condensaton [42], goal detecton [64], and bullet-hole mage classfcaton [32]. Data condensaton [42] was to select a small subset from huge databases and the accuracy of a classfer traned on such reduced data set were comparable to results from tranng wth the entre data sets. The paper extracted data ponts lyng close to the class boundares, SVs, whch form a much reduced but crtcal set for classfcaton usng SVMs. But the problem of large memory requrements for tranng SVMs n batch mode was solved so that the tranng would preserve only the SVs at each ncremental step, and add them to the tranng set for the next step, called ncremental learnng. Goal detecton for a partcular event, ghost goals, usng SVMs was proposed by An-

17 Applcatons of Support Vector Machnes for Pattern Recognton 229 cona et al. [64]. Xe et al. [32] focused on the applcaton of SVM for classfcaton of bullet-hole mages n an auto-scorng system. The mage was classfed nto one, two or more bullet-hole mages by mult-class SVMs. Whte blood cells classfcaton[88], spam categorzaton[89], text detecton and categorzaton [85,86] and more others [63, 65] are appled SVMs.. 4 Lmtatons of SVM The performance of SVMs largely depends on the choce of kernels. SVMs have only one user-specfed parameter C, whch controls the error penalty when the kernel s fxed, but the choce of kernel functons, whch are well suted to the specfc problem s very dffcult [44]. Smola et al. [66] explaned the relaton between the SVM kernel method and the standard regularzaton theory. However, there are no theores concernng how to choose good kernel functons n a data-dependent way [4]. Amar and Wu [4] proposed a modfed kernel to mprove the performance of SVMs classfer. It s based on nformaton-geometrc consderaton of the structure of the Remannan geometry nduced by the kernel. The dea s to enlarge the spatal resoluton around the boundary by a conformal transformaton so that the separablty of classes s ncreased. Speed and sze s another problem of SVMs both n tranng and testng. In terms of runnng tme, SVMs are slower than other neural networks for a smlar generalzaton performance [68]. Tranng for very large datasets wth mllons of SVs s an unsolved problem [44]. Recently, even though Platt [69] and Keerth et al. [70] proposed SMO (Sequental Mnmzaton Optmzaton) and modfed SMO to solve the tranng problem, t s stll an open problem to mprove. The ssue of how to control the selecton of SVs s another dffcult problem, partcularly when the patterns to be classfed are nonseparable and the tranng data are nosy. In general, attempts to remove known errors from the data before tranng or to remove them from the expanson after tranng wll not gve the same optmal hyperplane because the errors are needed for penalzng nonseparablty [68]. Lastly, although some researches have been done on tranng a mult-class SVM, the work for mult-class SVM classfers s an area for further research [44]. 5 Concluson We have presented a bref ntroducton on SVMs and several applcatons of SVMs n pattern recognton problems. SVMs have been successfully appled to a number of applcatons rangng from face detecton and recognton, object detecton and recognton, handwrtten character and dgt recognton, speaker and speech recognton, nformaton and mage retreval, predcton and etc. because they have yelded excellent generalzaton performance on many statstcal problems wthout any pror knowledge and when the dmenson of nput space s very hgh. In ths paper, we dd not compare the performance results for same applcaton.

18 230 Hyeran Byun and Seong-Whan Lee Some researches compared the performance of dfferent knds of SVM kernels to solve ther problems and most results showed that RBF kernel was usually better than lnear or polynomal kernels. RBF kernel performs usually better than others for several reasons such as () t has better boundary response as t allows extrapolaton and (2) most hgh dmensonal data sets can be approxmated by Gaussan-lke dstrbutons smlar to that used by RBFs[8]. Among the applcaton areas, the most popular research felds to apply SVMs are for face detecton, verfcaton and recognton. SVMs are bnary class classfers and t was frst appled for verfcaton or 2 class classfcaton problems. But SVMs had been used for mult-class classfcaton problems snce one to others and parwse bottom-up, top-down mult-class classfcaton methods were developed. Most of applcatons usng SVMs showed SVMs-based problem solvng outperformed to other methods. Although SVMs do not have long hstores, t has been appled to a wde range of machne learnng tasks and used to generate many possble learnng archtectures through an approprate choce of kernels. If some lmtatons related wth the choce of kernels, tranng speed and sze are solved, t can be appled to more real-lfe classfcaton problems. Acknowledgements The authors would lke to thank Mr. Byungchul Ko for many useful suggestons that helped to mprove the presentaton of the paper. Ths research was supported by the Bran Neuronfomatcs Research Program and the Creatve Research Intatve Program of the Mnstry of Scence and Technology, Korea. References. B. Boser, I. Guyon, and V. Vapnk, A tranng algorthm for optmal margn classfers, In Proceedngs of Ffth Annual Workshop on Computatonal Learnng Theory, New York, (992). 2. C. Cortes and V. Vapnk, Support vector networks, In Proceedngs of Machne Learnng, vol. 20, pp , (995). 3. V. Vapnk, The nature of statstcal learnng theory, Sprnger, (995). 4. S. Amar and S. Wu, Improvng support vector machne classfers by modfyng kernel functons, In Proceedngs of Internatonal Conference on Neural Networks, 2, pp , (999). 5. K. Jonsson, J. Kttler, and Y.P. Matas, Support vector machnes for face authentcaton, Journal of Image and Vson Computng, vol. 20. pp , (2002). 6. Juwe Lu, K.N. Platanots, and A.N. Ventesanopoulos, Face recognton usng feature optmzaton and v-support vector machne, IEEE Neural Networks for Sgnal Processng XI, pp , (200). 7. F. Serald and J. Bgun, Retnal vson appled to facal features detecton and face authentcaton, Pattern Recognton Letters, vol. 23, pp , (2002).

19 Applcatons of Support Vector Machnes for Pattern Recognton A. Tefas, C. Kotropoulos, and I. Ptas, Usng support vector machnes to enhance the performance of elastc graph matchng for frontal face authentcaton, IEEE Transacton on Pattern Analyss and Machne Intellgence, vol. 23. No. 7, pp , (200). 9. G. Guo, S. Z. L, and K. L.Chan, Support vector machnes for face recognton, Journal of Image and Vson Computng, vol. 9, pp , (200). 0. B. Hesele, P. Ho, and T. Poggo, Face Recognton wth support vector machnes: global versus component-based approach, In Proceedngs of Eghth IEEE Int. Conference on Computer Vson, vol. 2, pp , (200).. T. J. Terrllon, M.N. Shraz, M. Sadek, H. Fukamach, and S. Akamatsu, Invarant face detecton wth support vector machnes, In Proceedngs of 5 th Int. Conference on Pattern Recognton, vol. 4, 20, pp , (2000). 2. E. M. Santos and H.M. Gomes, Appearance-based object recognton usng support vector machnes, In Proceedngs of XIV Brazlan Symposum on Computer Graphcs and Image Processng, pp. 399, (200). 3. D. Gao, J. Zhou, and Lepng Xn, SVM-based detecton of movng vehcles for automatc traffc montorng, IEEE Intellgent Transportaton System, pp , (200). 4. Z. L, Z. Weda, and J. Lcheng, Radar target recognton based on support vector machne, In Proceedngs of 5 th Int. Conference on Sgnal processng, vol. 3, pp , (2000). 5. D. Roobaert and M.M. Van Hulle, Vew-based 3D object recognton wth support vector machnes, In Proceedngs of IX IEEE Workshop on Neural Networks for Sgnal Processng, pp , (999). 6. C. Chosy and A. Belad, Handwrtng recognton usng local methods for normalzaton and global methods for recognton, In Proceedngs of Sxth Int. Conference On Document Analyss and Recognton, pp , (200). 7. D. Gorgevk, D. Cakmakov, and V. Radevsk, Handwrtten dgt recognton by combnng support vector machnes usng rule-based reasonng, In Proceedngs of 23 rd Int. Conference on Informaton Technology Interfaces, pp , (200). 8. L.N. Teow and K.F. Loe, Robust vson-based features and classfcaton schemes for off-lne handwrtten dgt recognton, Pattern Recognton, January, (2002). 9. C.S. Shn, K.I. Km, M.H. Park, and H.J. Km, Support vector machne-based text detecton n dgtal vdeo, In Proceedngs of IEEE Workshop on Neural Networks for Sgnal Processng, vol. 2, pp ,(2000). K. I. Km, K. Jung, S. H. Park, and H. J. Km, Support vector machne-based text detecton n dgtal vdeo, Pattern Recognton, vol 34, pp , (200). 20. X. Dong and W. Zhaohu, Speaker recognton usng contnuous densty support vector machnes, Electroncs Letters, vol. 37, pp , (200). 2. S. Bengo and J. Marethoz, Learnng the decson functon for speaker verfcaton, In Proceedngs of IEEE Int. Conference on Acoustcs, Speech, and Sgnal Processng, vol., pp , (200). 22. C. Ma, M.A. Randolph, and J. Drsh, A support vector machnes-based rejecton technque for speech recognton, In Proceedngs of IEEE Int. Conference on Acoustcs, Speech, and Sgnal Processng vol., pp , (200).

20 232 Hyeran Byun and Seong-Whan Lee 23. V. Wan and W.M. Campbell, Support vector machnes for speaker verfcaton and dentfcaton, In Proceedngs of IEEE Workshop on Neural Networks for Sgnal Processng X, vol. 2, (2000). 24. C. Nakajma, M. Pontl, and T. Poggo, People recognton and pose estmaton n mage sequences,, In Proceedngs of IEEE Int. Jont Conference on Neural Networks, vol. 4, pp , (2000). 25. E. Ardzzone, A. Chella, and R. Prrone, Pose classfcaton usng support vector machnes,, In Proceedngs of IEEE Int. Jont Conference on Neural Networks, vol. 6, pp , (2000). 26. J. Ng and S. Gong, Composte support vector machnes for detecton of faces across vews and pose estmaton, Image and Vson Computng, vol. 20, Issue 5-6, pp , (2002). 27. M. H. Yang and B. Moghaddam, Gender classfcaton usng support vector machnes,, In Proceedngs of IEEE Int. Conference on Image Processng, vol. 2, pp , (2000). 28. J. Zhang, Y Zhang, and T. Zhou, Classfcaton of hyperspectral data usng support vector machne, In Proceedngs of Int. Conference on Image Processng, vol., pp , (200). 29. L. Ramrez, W. Pedrycz, and N. Pzz, Severe storm cell classfcaton usng support vector machnes and radal bass approaches, In Proceedngs of Canadan Conference on Electrcal and Computer Engneerng, vol., pp. 87-9, (200). 30. Y. Zhang, R. Zhao, and Y. Leung, Image Classfcaton by support vector machnes, In Proceedngs of Int. Conference on Intellgent Multmeda, Vdeo and Speech Processng, pp , (200). 3. S. Gutta, J.R.J. Huang, P. Jonathon, and H. Wechsler, Mxture of experts for classfcaton of gender, ethnc orgn, and pose of human, IEEE Trans. on Neural Networks, vol., Issue.4, pp , (2000). 32. W.F. Xe, D.J. Hou, and Q. Song, Bullet-hole mage classfcaton wth support vector machnes, In Proceedngs of IEEE Sgnal Processng Workshop on Neural Networks for Sgnal Processng, vol., pp , (2000). 33. G. Guo, H.J. Zhang, and S.Z. L, Dstance-from-boundary as a metrc for texture mage retreval, In Proceedngs of IEEE Int. Conference on Acoustcs, Speech, and Sgnal Processng, vol. 3, pp , (200). 34. Q. Tan, P. Hong, and T.S. Huang, Update relevant mage weghts for contentbased mage retreval usng support vector machnes, In Proceedngs of IEEE Int. Conference on Multmeda and Expo, vol.2, pp , (2000). 35. P. Hong, Q. Tan, and T,S. Huang, Incorporate support vector machnes to content-based mage retreval wth relevance feedback, In Proceedngs of Int. Conference on Image Processng, vol. 3, pp , (2000). 36. H. Druker, B. Shahrary, and D.C. Gbbon, Support vector machnes: relevance feedback and nformaton retreval, Informaton Processng & Management, vol. 38, Issue 3, pp , (2002). 37. T. Van Gestel, J.A.K. Suykens, D.E. Baestaens, A. Lambrechts, G. Lanckret, B. Vandaele, B. De Moor, and J. Vandewalle, Fnancal tme seres predcton usng least squares support vector machnes wthn the evdence framework, IEEE Trans. On Neural Networks, vol. 2. Issue 4, pp , (200).

21 Applcatons of Support Vector Machnes for Pattern Recognton T. Frontzek, T. Navn Lal, and R. Eckmller, Predctng the nonlnear dynamcs of bologcal neurons usng support vector machnes wth dfferent kernels, In Proceedngs of Int. Jont Conference on Neural Networks, vol. 2, pp , (200). 39. D. Mckay and C. Fyfe, Probablty predcton usng support vector machnes, In Proceedngs of Int. Conference on Knowledge-Based Intellgent Engneerng Systems and Alled Technologes, vol., pp , (2000). 40. A. Fan and M. Palanswam, Selectng bankruptcy predctors usng a support vector machne approach, vol. 6, pp , (2000). 4. F. Tay and L.J. Cao, Modfed support vector machnes n fnancal tme seres forecastng, Neurocomputng, Sept., (200). 42. P. Mtra, C.A. Murthy, and S.K. Pal, Data condensaton n large database by ncremental learnng wth support vector machnes, In Proceedngs of 5 th Int. Conference on Pattern Recognton, vol. 2, pp , (2000). 43. B. Gutschoven and P. Verlnde, Mult-modal dentty verfcaton usng support vector machnes (SVM), In Proceedngs of The thrd Int. Conference on Informaton Fuson, pp. 3-8, (2000). 44. C. C. Burges, A tutoral on support vector machnes for pattern recognton, In Proceedngs of Int. Conference on Data Mnng and Knowledge Dscovery, 2(2), pp. 2-67, (998). 45. B. Zhao, Y. Lu, and S.W. Xa, Support vector machnes and ts applcaton n handwrtten numercal recognton, In Proceedngs of 5 th Int. Conference on Pattern Recognton, vol. 2, pp , (2000). 46. S. Bernhard, C.C. Burges, and A.J. Smola, Parwse classfcaton and support vector machnes, The MIT Press, C. Massachusetts, London England, pp , Jan. (999). 47. M. Pontl and A. Verr. Support vector machnes for 3-D object recognton, IEEE Trans. on Pattern Analyss and Machne Intellgence, pp , (998). 48. G. Guodong, S. L, and C. Kapluk, Face recognton by support vector machnes. In Proceedngs of IEEE Int. Conference on Automatc Face and Gesture Recognton, pp , (2000). 49. J. Platt, N. Chrstann, and J. Shawe-Taylor, Large margn DAGs for multclass classfcaton, Advances n Neural Informaton Processng Systems, (2000). 50. Y. Wang, C.S. Chua, and Y.K, Ho. Facal feature detecton and face recognton from 2D and 3D mages, Pattern Recognton Letters, Feb., (2002). 5. E. Osuna, R. Freund, and F. Gros, Tranng support machnes: An applcaton to face detecton. In Proceedngs of IEEE Conference on Computer Vson and Pattern Recognton, pp , (997). 52. V. Kumar and T. Poggo, Learnng-based approach to real tme trackng and analyss of faces, In Proceedngs of IEEE Int. Conference on Automatc Face and Gesture Recognton, (2000). 53. E. Hjelams and B. K. Low, Face Detecton: A Survey, Computer Vson and Image Understandng, 83, pp , (200).

22 234 Hyeran Byun and Seong-Whan Lee 54. J. Ng and S. Gong, Performng mult-vew face detecton and pose estmaton usng a composte support vector machne across the vew sphere, In Proceedngs of IEEE Int. Workshop on Recognton, Analyss, and Trackng of Faces and Gestures n Real-Tme Systems, (999). 55. Y. L, S. Gong, J. Sherrah, and H. Lddell, Mult-vew Face Detecton Usng Support Vector Machnes and Egenspace Modellng, In Proceedngs of Fourth Int. Conference on Knowledge-Based Intellgent Engneerng Systems & Alled Technologes, pp , (2000). 56. K.I. Km, J. Km, and K Jung, Recognton of facal mages usng support vector machnes, In Proceedngs of th IEEE Workshop on Statstcal Sgnal Processng, pp , (200). 57. M. Pttore, C. Basso, and A. Verr, Representng and recognzng vsual dynamc events wth support vector machnes, In Proceedngs of Int. Conference on Image Analyss and Processng, pp. 8-23, (999). 58. A. K. Jan and B. Yu, Automatc text locaton n mages and vdeo frames, Pattern Recognton, vol. 3, No. 2, pp , (998). 59. S. Antan, U. Garg, D. Crandall, T. Gandh, and R. Kastur, Extracton of text n vdeo, Dept. of Computer Scence and Eng. Pennsylvana Stat Unv., Techncal Report, CSE-99-06, (999). 60. I. Jang, B.C. Ko, and H. Byun, Automatc text extracton n news mages usng morphology, In Proceedngs of SPIE Vsual Communcaton and Image Processng, San Jose, Jan., (2002). 6. T. Joachms, Text categorzaton wth support vector machnes: learnng wth many relevant features, In Proceedngs of 0 th European Conference on Machne learnng, (999). 62. Y. Yao, G. L. Marcals, M. Pontl, P. Frascon, and F. Rol, Combnng flat and structured representatons for fngerprnt classfcaton wth recursve neural networks and support vector machnes, Pattern Recognton, pp. -0, (2002). 63. A. Gretton, M. Davy, A. Doucet, and P. J.W. Rayner, Nonstatonary sgnal classfcaton usng support vector machnes, In Proceedngs of th IEEE Workshop on Statstcal Sgnal Processng, pp , (200). 64. N. Ancona, G. Ccrell, A. Branca, and A. Dstante, Goal detecton n football by usng support vector machnes for classfcaton, In Proceedngs of Int. Jont Conference on Neural Networks, vol. pp. 6-66, (200). 65. S. I. Hll, P. J. Wolfe, and P. J. W. Rayner, Nonlnear perceptual audo flterng usng support vector machnes, In Proceedngs of th IEEE Int. Workshop on Statstcal Sgnal Processng, pp , (200). 66. A. J. Smola, B. Scholkopf, and K. R. Müller, The connecton between regularzaton operators and support vector kernels, Neural Networks,, pp , (998). 67. C. J. C. Burges, Smplfed support vector decson rules, In Proceedngs of 3 th Int, Conference on Machne Learnng, pp. 7-77, (996). 68. S. Haykn, Neural Networks, Prentce Hall Inc. (999). 69. J. Platt, Sequental Mnmal Optmzaton: A Fast Algorthm for Tranng Support Vector Machnes, Mcrosoft Research Techncal Report MSR-TR-98-4, (998).