Incremental Learning with Support Vector Machines and Fuzzy Set Theory

Transcription

1 The 25th Workshop on Combnatoral Mathematcs and Computaton Theory Incremental Learnng wth Support Vector Machnes and Fuzzy Set Theory Yu-Mng Chuang 1 and Cha-Hwa Ln 2* 1 Department of Computer Scence and Engneerng 2 Department of Computer Scence and Engneerng and College of General Educaton Natonal Sun Yat-Sen Unversty, Kaohsung 80424, Tawan * chln@cse.nsysu.edu.tw Abstract Over the past few years, a consderable number of studes have been made on Support Vector Machnes (SVMs) n many domans to mprove classfcaton or predcton. However, SVMs request hgh computatonal tme and memory when the datasets are large. Although ncremental learnng technques are vewed as one possble soluton developed to reduce the computaton complexty of the scalablty problem, few studes have consdered that some examples close to the decson hyperplane other than support vectors (SVs) mght contrbute to the tranng process. Consequently, we propose a novel algorthm by mprovng Syed s ncremental learnng method based on fuzzy set theory. At each learnng step, SVs and potental nformatve examples, called canddate examples (CEs), are added to the next ncremental learnng step. We expect to acheve better accuracy and less executon tme than other methods. In ths ongong study, the proposed algorthm would be nvestgated on fve standard machne learnng benchmark datasets to demonstrate the effectveness of the method. 1 Introducton In the last few years, many artcles have been devoted to the study on support vector machnes (SVMs) whch have good performance as the classfers n pattern recognton and text classfcaton. They also have been appled to categorze Spam [3][6], and sgnfcantly outperform the conventonal methods (e.g., Boostng, Roccho, Bayes, and k-nearest Neghbors). At the same tme, there are complcated classfcaton tasks to be solved n many other felds, such as bonformatcs, mage processng, socal studes, and stock market. Vapnk [11] ntroduced SVMs n SVMs have strong theoretcal groundwork and statstcal foundaton, whch are based on the structural rsk mnmzaton prncple [11] from computatonal learnng theory. By most nformatory data examples, called support vectors (SVs), the decson hyperplane s found wth the lowest true error. The true error of the hyperplane s the probablty that the hyperplane wll make an error on an unknown and randomly pcked test data. Some expermental results show that SVMs do not requre any parameter tunng, snce they can determne good parameter settng automatcally [6]. Compared to the customary learnng algorthms, SVMs handle large feature spaces wthout feature selecton. As real-word datasets expand n sze, there s a need to desgn some novel learnng algorthms to tran ncreasng nstances. Incremental (or actve) learnng procedures are one possble paradgm for reducng the sample complexty of large scale learnng task. There are many dfferent approaches of ncremental learnng to shorten the tranng tme [1][7][10]. Processng data n several steps nstead of learnng from randomly selected data, these methods would use less memory space and savng more computatonal tme. The am s to choose a part of samples for tranng and keep the performance as usng total tranng nstances. In the next secton, we ntroduce the related work about the ncremental learnng presented by the researchers and the fuzzy set theory. Secton 3 presents our proposed mechansm for ncremental learnng. Secton 4 descrbes the desgn of experment. Fnally, we conclude n Secton 5 wth some dscusson of our method, and remark on future research drectons. 2 Related Research 2.1 Support Vector Machne Support vector machnes are usually used to solve two-class classfcaton problems by tranng the datasets and obtanng the optmal separatng hyperplane. SVMs are also appled to mult-class classfcaton and regresson problems. Suppose we are gven k patterns x, y 1 1,, x y. The label y (-1, 1) and the assocated, k k * The correspondng author

2 The 25th Workshop on Combnatoral Mathematcs and Computaton Theory d vector x R where d s the dmenson of the vector. Let X be the space of patterns, Y be the space of labels. The decson functon s n form of sgn( w x b ), where w x means the nner product of w and x. If all k patterns are lnearly separable, we can fnd a decson functon lke y ( w x b ) 1, = 1, 2,, k (1) Tranng samples that satsfy the equalty are called support vectors. These two-class support vectors belong to dfferent hyperplanes. SVMs maxmze the margn when the norm of the weght vector w s mnmum. A margn s defned as the dstance between the two hyperplanes. Snce the optmzaton problem s dffcult to handle numercally, Lagrange multples are used to translate the problem nto an equvalent quadratc optmzaton problem [11]. Thus, we should maxmze (2) subject to (3) k k k 1 Q ( ) y y ( x x ) (2) j j j j 1 k y 0 (3) 1 When the tranng set s not lnearly separable, slack varableξ can be added to allow msclassfcaton of nosy or dffcult samples y ( w x b ) 1, 0 for all (4) To learn the datasets that are nonlnearly separable, SVMs make use of kernel functon K x 1, x 2. The kernel functon satsfes Mercer s Theorem [11]. Ths means that we can calculate the nner product of the vectors x 1 and x 2 after they have been mapped nto another feature space by a non-lnear mappng Φ: K x, x ( x ) ( x ) (5) 2.2 Incremental Learnng Incremental learnng algorthms for support vector machnes have receved consderable attenton over past few years. The dscusson of some exstng ncremental learnng s presented as follows. Syed et al. [10] presented an ncremental learnng procedure by parttonng the whole database nto subsets that fts nto the man memory. Then tranng SVM classfer ncrementally wth the parttons. The tranng would preserve only the support vectors at each ncremental step, and add them to the tranng set for the next step. Moreover, most current strateges of actve learnng perform the measurement of proxmty to the separatng hyperplanes. Mtra et al. [7] developed an actve support vector learnng algorthm, whch s a probablstc generalzaton of purely margn-based method. The lkelhood of an sample beng an support vector s estmated usng a combnaton of two factors: The margn of the partcular sample wth respect to the current hyperplane and the degree of confdence that the current set of support vectors provdes the actual support vector. Cheng [1] presented an mproved ncremental tranng method for SVMs usng actve query. In actve query, they assgn a weght to each example accordng to ts dstance to the separatng hyperlane and ts confdence factor. The confdence factor s calculated from the error upper bound of the SVMs to ndcate the closeness of the current hyperplane to the optmal hyperplane. The ntal set of tranng examples s collected by dvdng the whole tranng examples nto groups and applyng the k-means clusterng algorthm. 2.3 Fuzzy Set Theory There are many ambguous words n our conversaton, such as Tom s tal and It s very cold. We often descrbe crcumstances n such mprecse statements. Fuzzy set [12], whch s a mathematcal tool used to model an uncertan, ncomplete concept. The development of fuzzy set theory provdes a lot of applcatons n the daly lfe, such as ar-condtons, washng machnes, and automobles. Tradtonally, the crsp set s a set wth a boundary. For example, a set A of real numbers less than or equal to 12 can be presented as A { x x 12}. If x s less than or equal to 12, then x s belong to the set A whch has a certan boundary 12. The ordnary subset A of set X can be thought of a functon from X to the 2-element set {0, 1} [9], where the set X s called unverse of dscourse. 1 f x A A( x) 0 f x A (6) where A(x) s called characterstc functon of the set A. Crsp sets are also referred to as ordnary sets, classcal sets, non-fuzzy sets, or just sets

3 The 25th Workshop on Combnatoral Mathematcs and Computaton Theory Partton 1 Partton 2 Partton M Intal Dataset Fnal Dataset Fgure 1. The ncremental learnng procedure wth canddate examples The defnton of fuzzy sets s usually by the concept of membershp functon. If X s a collecton of objects denoted genercally by x, then a fuzzy set A n X s defned as a set of ordered pars [5]: A ( x, ( x)) x X (7) A Where A( x ) s called the membershp functon of the fuzzy set A. The value of A( x ) also means the degree of membershp of x n A. In contrast, elements of fuzzy sets are n varyng degrees of membershp functon. The membershp functon maps each element of X to a value between 0 and 1. Obvously, a fuzzy set s a smple extenson of a crsp set n whch the membershp functon s permtted to have any values between 0 and 1. If the value of the membershp functon A( x ) s restrcted to ether 0 or 1, then A s reduced to a crsp set and A( x ) s the characterstc functon of A. 3 The Proposed Method Some studes showed that the nstances whch near the decson hyperplane are nformatve. Thus, t mght be possble to fnd some data nstead of the closest ones. Fuzzy set theory s helpful for us to take these nstances n consderaton, and we can determne the optmal separatng hyperplane more effectvely. Hence, we modfy Syed s ncremental learnng algorthm based on fuzzy set theory. Syed s method parttoned a huge database that each partton would ft nto the man memory. The tranng would preserve only the support vectors at each ncremental step, and add them to the tranng set for the next step. The model obtaned by ths method should be the same or smlar to what would have been obtaned usng all the data to tran. Ther technque looked at the examples only once to determne f they wll become support vectors. Once dscarded, the vectors are not consdered agan. 3.1 Canddate Examples The tranng samples whch satsfy (1) are called support vectors. By fuzzy set theory, some examples outsde the two parallel hyperplanes formng the maxmum margn of the SVM nterest us. We thnk these non-support vectors, called canddate examples (CEs), are useful n tranng steps. Accordng to some factor, CEs can be farly selected. The factor s a type of membershp functon (Fgure 2), whch s exploted by fuzzy set theory. Usng membershp functon not only helps us fnd the relatve samples but also be aware of the data dstrbuton. Let A be the set of CEs. The membershp functon on set A s defned as 1 f x > d a ( ) x c x 1 e 0 f x d A (8) where a determnes the slope at the crossover pont, x = c, c shfts the functon rght or left, and d s half of the maxmum margn of the SVM. Because the examples whch are far away from the separatng hyperplane mght hardly be chosen as CEs, we set the parameter a < 0 to obtan the decreasng functon n the regon x > d. The value of d s usually scaled to 1 accordng to the dstance between the separatng hyperplane and each hyperplane whch forms the maxmum margn of the SVM. Moreover, the -442-

4 The 25th Workshop on Combnatoral Mathematcs and Computaton Theory Membershp degree μ A(x) Dstance x Fgure 2. The membershp functon of CEs wth m = 8 parameter c must be greater than d to present the dstrbuton of the CEs. If c d, CEs mght be non-nformatve for tranng steps or regarded as SVs. Both the value of a and c are set by the expermental processes. 3.2 The Improved Learnng Algorthm Table 1. The datasets used n experments Dataset No. of attrbutes Total nstances Cancer Dabetes German Heart Ionosphere Datasets There are avalable publc doman datasets provded by some organzaton, such as Ohsumed corpus (ftp://medr.ohsu.edu/pub/ohsumed/) and StatLb ( We would carry out the emprcal study by the datasets obtaned from the UCI machne learnng repostory [8]. The characterstcs of the fve datasets selected are descrbed n Table 1. Hence, we follow the above dea and consder more data than support vectors (Fgure 1). The basc concept of the proposed learnng algorthm s descrbed as follows. Step 1: Partton the ntal unlabeled tranng samples x 1, x 2,, x k nto m subsets of equal sze. Step 2: Randomly select a subset, and tran SVM classfer wth the subset of data. Select l examples from the remanng data whch are called CEs by fuzzy set theory. Step 3: Add the SVs and the CEs to the tranng set for the next step untl all parttons are used. At step 2, we randomly dvde the ntal datasets nto m groups. The reason s when there s a large number of tranng examples, t s tme-consumng to fnd the CEs. 4 Expermental Desgn In the experments, our method would use three kernels: lnear SVM, Gaussan RBF SVM and polynomal SVM n comparson wth fve exstng methods. 4.2 Evaluaton Measure We use the same set of parameters for all methods for comparson purposes. We set regularzaton constant C 2 = 1 and the varance of the Gaussan RBF 1. Our method would be compared wth fve exstng algorthms: BatchSVM, IncrSVM, SubsetSVM, ActveQuery, and LbSVM. BatchSVM [2] uses all the tranng examples to tran the SVM, t needs numerous data. At each ncremental step, IncrSVM randomly chooses the tranng examples. SubsetSVM [10] s an ncremental learnng method by parttonng the entre tranng dataset nto several subsets and only addng the SVs to the next step. ActveQuery [1] selects the ntal tranng examples by k-means clusterng nstead of random selecton, and actvely queres the nformatve examples based on confdence factor. LbSVM [4] s a lbrary for support vector machnes wth hgh accuracy. 5 Concluson Ether use only SVs n the tranng process mght lose some valuable data or use all tranng examples mght waste too much tme. In ths study, the proposed ncremental learnng algorthm ncorporates fuzzy set theory nto SVM classfer to solve the above problems. We consder not only the SVs are mportant, also some of the remanng examples are useful to tran the SVMs

5 The 25th Workshop on Combnatoral Mathematcs and Computaton Theory To fnd canddate examples, a membershp functon would be defned to express the degree of mportance of the potental examples and could be used to flter the tranng examples n the ncremental learnng steps. It s possble to adjust the parameters of the membershp functon to ncrease the canddate nstances. The proposed method could handle classfcaton problems and present a novel vew to recognze the useful examples n addtonal to SVs of SVMs by fuzzy set theory. At present, usng fuzzy set theory at each ncremental step mght spend more tme, t s possble to fnd an alteratve procedure to reduce the tme complexty n the future. References [1] C. Cheng, Frank Y. Shh, An mproved ncremental tranng algorthm for support vector machnes usng actve query, Pattern Recognton, Volume 40, Issue 3, March 2007, Pages [2] C. Cortes, V. Vapnk, Support-vector network, March. Learn. 20 (1995) [3] H. Drucker, D. Wu, and V. N. Vapnk, Support vector machnes for spam categorzaton, IEEE Trans. Neural Networks, vol. 10, no. 5, pp , [4] R. E. Fan, P. H. Chen, and C. J. Ln, Workng set selecton usng the second order nformaton for tranng SVM. Journal of Machne Learnng Research 6, , 2005 [5] J. S. R. Jang, C. T. Sun, E. Mzutan, Neuro-Fuzzy and Soft Computng. Prentce-Hall, Inc. Upper Saddle Rver, NJ, USA, [6] T. Joachms, Text Categorzaton wth Support Vector Machnes: Learnng wth Many Relevant Features. Proceedngs of the European Conference on Machne Learnng (ECML), Sprnger, [7] P. Mtra, C. A. Murthy, S. K. Pal, A probablstc actve support vector learnng algorthm, IEEE Trans, Pattern Anal. Mach. Intel. 26(3) (2004) [8] D. J. Newman, S. Hettch, C. L. Blake, C. J. Merz. UCI Repostory of machne learnng databases [ ml]. Irvne, CA: Unversty of Calforna, Department of Informaton and Computer Scence, [9] H. T. Nguyen, N. R. Prasad, C. L. Walker, E. A. Walker, A Frst Course n Fuzzy and Neural control. Chapman & Hall/CRC, Boca Raton, FL, [10] N.A. Syed, H. Lu, K. K. Sung, Incremental learnng wth support vector machnes, Proceedngs of the Internatonal Jont Conference on Artfcal Intellgence, Stockholm, Sweden, July [11] V. Vapnk, The Nature of Statstcal Learnng Theory. New York: Sprnger-Verlag, [12] L. A. Zadeh, Fuzzy Sets, Informaton and Control 8, (1965) -444-