Semi-supervised Classification Using Local and Global Regularization

Transcription

1 Proceedngs of the Twenty-Thrd AAAI Conference on Artfcal Intellgence (2008) Sem-supervsed Classfcaton Usng Local and Global Regularzaton Fe Wang 1, Tao L 2, Gang Wang 3, Changshu Zhang 1 1 Department of Automaton, Tsnghua Unversty, Bejng, Chna 2 School of Computng and Informaton Scences, Florda Internatonal Unversty, Mam, FL, USA 3 Mcrosoft Chna Research, Bejng, Chna Abstract In ths paper, we propose a sem-supervsed learnng (SSL) algorthm based on local and global regularzaton. In the local regularzaton part, our algorthm constructs a regularzed classfer for each data pont usng ts neghborhood, whle the global regularzaton part adopts a Laplacan regularzer to smooth the data labels predcted by those local classfers. We show that some exstng SSL algorthms can be derved from our framework. Fnally we present some expermental results to show the effectveness of our method. Introducton Sem-supervsed learnng (SSL), whch ams at learnng from partally labeled data sets, has receved consderable nterests from the machne learnng and data mnng communtes n recent years (Chapelle et al., 2006b). One reason for the popularty of SSL s because n many real world applcatons, the acquston of suffcent labeled data s qute expensve and tme consumng, but the large amount of unlabeled data are far easer to obtan. Many SSL methods have been proposed n the recent decades (Chapelle et al., 2006b), among whch the graph based approaches, such as Gaussan Random Felds (Zhu et al., 2003), Learnng wth Local and Global Regularzaton (Zhou et al., 2004) and Tkhonov Regularzaton (Belkn et al., 2004), have been becomng one of the hottest research area n SSL feld. The common denomnator of those algorthms s to model the whole data set as an undrected weghted graph, whose vertces correspond to the data set, and edges reflect the relatonshps between parwse data ponts. In SSL settng, some of the vertces on the graph are labeled, whle the remaned are unlabeled, and the goal of graph based SSL s to predct the labels of those unlabeled data ponts (and even the new testng data whch are not n the graph) such that the predcted labels are suffcently smooth wth respect to the data graph. One common strategy for realzng graph based SSL s to mnmze a crteron whch s composed of two parts, the frst part s a loss measures the dfference between the predctons and the ntal data labels, and the second part s a smoothness penalty measurng the smoothness of the predcted labels over the whole data graph. Most of the past Copyrght c 2008, Assocaton for the Advancement of Artfcal Intellgence ( All rghts reserved. works concentrate on the dervaton of dfferent forms of smoothness regularzers, such as the ones usng combnatoral graph Laplacan (Zhu et al., 2003)(Belkn et al., 2006), normalzed graph Laplacan (Zhou et al., 2004), exponental/teratve graph Laplacan (Belkn et al., 2004), local lnear regularzaton (Wang & Zhang, 2006) and local learnng regularzaton (Wu & Schölkopf, 2007), but rarely touch the problem of how to derve a more effcent loss functon. In ths paper, we argue that rather than applyng a global loss functon whch s based on the constructon of a global predctor usng the whole data set, t would be more desrable to measure such loss locally by buldng some local predctors for dfferent regons of the nput data space. Snce accordng to (Vapnk, 1995), usually t mght be dffcult to fnd a predctor whch holds a good predctablty n the entre nput data space, but t s much easer to fnd a good predctor whch s restrcted to a local regon of the nput space. Such dvde and conquer scheme has been shown to be much more effectve n some real world applcatons (Bottou & Vapnk, 1992). One problem of ths local strategy s that the number of data ponts n each regon s usually too small to tran a good predctor, therefore we propose to also apply a global smoother to make the predcted data labels more comply wth the ntrnsc data dstrbutons. A Bref Revew of Manfold Regularzaton Before we go nto the detals of our algorthm, let s frst revew the basc dea of manfold regularzaton (Belkn et al., 2006) n ths secton, snce t s closely related to ths paper. As we know, n sem-supervsed learnng, we are gven a set of data ponts X = {x 1,,x l,x l+1,,x n }, where X l = {x } l =1 are labeled, and X u = {x j } n j=l+1 are unlabeled. Each x X s drawn from a fxed but usually unknown dstrbuton p(x). Belkn et al. (Belkn et al., 2006) proposed a general geometrc framework for semsupervsed learnng called manfold regularzaton, whch seeks an optmal classfcaton functon f by mnmzng the followng objectve J g = l =1 L(y, f(x,w)) + γ A f 2 F + γ I f 2 I, (1) where y represents the label of x, f(x,w) denotes the classfcaton functon f wth ts parameter w, f F penalzes the complexty of f n the functonal space F, f I reflects 726

2 the ntrnsc geometrc nformaton of the margnal dstrbuton p(x), γ A, γ I are the regularzaton parameters. The reason why we should punsh the geometrcal nformaton of f s that n sem-supervsed learnng, we only have a small porton of labeled data (.e. l s small), whch are not enough to tran a good learner by purely mnmzng the structural loss of f. Therefore, we need some pror knowledge to gude us to learn a good f. What p(x) reflects s just such type of pror nformaton. Moreover, t s usually assumed (Belkn et al., 2006) that there s a drect relatonshp between p(x) and p(y x),.e. f two ponts x 1 and x 2 are close n the ntrnsc geometry of p(x), then the condtonal dstrbutons p(y x 1 ) and p(y x 2 ) should be smlar. In other words, p(y x) should vary smoothly along the geodescs n the ntrnsc geometry of p(x). Specfcally, (Belkn et al., 2006) also showed that f 2 I can be approxmated by Ŝ =,j (f(x ) f(x j )) 2 W j = f T Lf (2) where n s the total number of data ponts, and W j are the edge weghts n the data adjacency graph, f = (f(x 1 ),, f(x n )) T. L = D W R n n s the graph Laplacan where W s the graph weght matrx wth ts (, j)-th entry W(, j) = W j, and D s a dagonal degree matrx wth D(, ) = j W j. There has been extensve dscusson on that under certan condtons choosng Gaussan weghts for the adjacency graph leads to convergence of the graph Laplacan to the Laplace-Beltram operator M (or ts weghted verson) on the manfold M(Belkn & Nyog, 2005)(Hen et al., 2005). The Algorthm In ths secton we wll ntroduce our learnng wth local and global regularzaton approach n detal. Frst let s see the motvaton of ths work. Why Local Learnng Although (Belkn et al., 2006) provdes us an excellent framework for learnng from labeled and unlabeled data, the loss J g s defned n a global way,.e. for the whole data set, we only need to pursue one classfcaton functon f that can mnmze J g. Accordng to (Vapnk, 1995), selectng a good f n such a global way mght not be a good strategy because the functon set f(x, w), w W may not contan a good predctor for the entre nput space. However, t s much easer for the set to contan some functons that are capable of producng good predctons on some specfed regons of the nput space. Therefore, f we splt the whole nput space nto C local regons, then t s usually more effectve to mnmze the followng local cost functon for each regon. Nevertheless, there are stll some problems wth pure local learnng algorthms snce that there mght not be enough data ponts n each local regon for tranng the local classfers. Therefore, we propose to apply a global smoother to smooth the predcted data labels wth respect to the ntrnsc data manfold, such that the predcted data labels can be more reasonable and accurate. The Constructon of Local Classfers In ths subsecton, we wll ntroduce how to construct the local classfers. Specfcally, n our method, we splt the whole nput data space nto n overlappng regons {R } n =1, such that R s just the k-nearest neghborhood of x. We further construct a classfcaton functon f for regon R, whch, for smplcty, s assumed to be lnear. Then g predcts the label of x by g (x) = w T (x x ) + b, (3) where w and b are the weght vector and bas term of g 1. A general approach for gettng the optmal parameter set {(w, b )} n =1 s to mnmze the followng structural loss n Jˆ l = (w T (x j x ) + b y j ) 2 + γ A w 2. =1 x j R However, n sem-supervsed learnng scenaro, we only have a few labeled ponts,.e., we do not know the correspondng y for most of the ponts. To allevate ths problem, we assocate each y wth a hdden label f, such that y s drectly determned by f. Then we can mnmze the followng loss functon nstead to get the optmal parameters. J l = (y f ) 2 + λj ˆ l (4) = =1 n =1 x j R (w T (x j x ) + b f j ) 2 + γ A w 2 Let Jl = x j R (w T(x j x ) + b f ) 2 + γ A w 2, whch can be rewrtten n ts matrx form as [ ] Jl = G w b f 2 where G = x T 1 x T 1 x T 2 x T 1 1.., f = x T n x T 1 γa I d 0 f 1 f 2. f n 0. where x j represents the j-th neghbor of x, n s the cardnalty of R, and 0 s a d 1 zero vector, d s the dmensonalty of the data vectors. By takng (w,b ) Jl = 0, we can get that [ ] w = (G b T G ) 1 G T f (5) Then the total loss we want to mnmze becomes ˆ J l = J l = f GT G f, (6) 1 Snce there s only a few data ponts n each neghborhood, then the structural penalty term w wll pull the weght vector w toward some arbtrary orgn. For sotropy reasons, we translate the orgn of the nput space to the neghborhood medod x, by subtractng x from the tranng ponts x j R 727

3 where G = I G (G T G ) 1 G T. If we partton G nto four block as [ A n n G = B n d ] C d n Let f = [f 1, f 2,, f n ] T, then Thus D d d [ ] [ f T G f = [f T A B 0] f C D 0 Ĵ l = ] = f T A f f T A f. (7) Furthermore, we have the followng theorem. Theorem 1. ( A = I n X T H 1 X + XT H 1 X 11 T X T H 1 n c XT H 1 X 11 T n c 11T X T H 1 X + 11T n c n c where H = X X T + γ A I d, c = 1 T X T H 1 X 1, 1 R n 1 s an all-one vector, and A 1 = 0. Proof. See the supplemental materal. Then we can defne the label vector f = [f 1, f 2,, f n ] T R n 1, the concatenated label vector ˆf = [f T 1,f2 T,,fT n ]T and the concatenated block-dagonal matrx A A 2 0 Ĝ = , 0 0 A n whch s of sze n n. Then from Eq.(7) we can derve that J l = ˆf T Ĝˆf. Defne the selecton matrx S {0, 1} n P n, whch s a 0-1 matrx and there s only one 1 n each row of S, such that ˆf = Sf. Then J ˆ l = f T S T ĜSf. Let M = S T ĜS R n n, (8) whch s a square matrx, then we can rewrte ˆ J l as X ˆ J l = f T Mf. (9) SSL wth Local & Global Regularzatons As stated n secton 3.1, we also need to apply a global smoother to smooth the predcted hdden labels {f }. Here we apply the same smoothness regularzer as n Eq.(2), then the predcted labels can be acheved by mnmzng J = =1 (y f ) 2 + λf T Mf + γ I n 2fT Lf. (10) By settng J / f = 0 we can get that ( f = J + λm + γ ) 1 n 2L Jy, (11) ), where J R n n s a dagonal matrx wth ts (, )-th entry { 1, f x s labeled; J(, ) = (12) 0, otherwse, y s an n 1 column vector wth ts -th entry { y, f x y() = s labeled; 0, otherwse Inducton To predct the label of an unseen testng data pont, whch has not appeared n X, we propose a three-step approach: Step 1. Solve the optmal label vector f usng. Step 2. Solve the parameters {w, b } of the optmal local classfcaton functons usng Eq.(5). Step 3. For a new testng pont x, frst dentfy the local regons that x falls n (e.g. by computng the Eucldean dstance between x to the regon medods and select the nearest one), then apply the local predcton functons of the correspondng regons to predct ts label. Dscussons In ths secton, we dscuss the relatonshps between the proposed framework wth some exstng related approaches, and present another mxed regularzaton framework for the algorthm presented n secton. Relatonshp wth Related Approaches There has already been some sem-supervsed learnng algorthms based on dfferent regularzatons. In ths subsecton, we wll dscuss the relatonshps between our algorthm wth those exstng approaches. Relatonshp wth Gaussan-Laplacan Regularzed Approaches Most of tradtonal graph based SSL algorthms (e.g. (Belkn et al., 2004; Zhou et al., 2004; Zhu et al., 2003)) are based on the followng framework f = argmn f (f y ) 2 + ζf T Lf, (13) =1 where f = [f 1, f 2,, f l,, f n ] T, L s the graph Laplacan constructed by Gaussan functons. Clearly, the above framework s just a specal case of our algorthm f we set λ = 0, γ I = n 2 ζ n Eq.(10). Relatonshp wth Local Learnng Regularzed Approaches Recently, Wu & Schölkopf (Wu & Schölkopf, 2007) proposed a novel transducton method based on local learnng, whch ams to solve the followng optmzaton problem f = arg mn f (f y ) 2 + ζ =1 n f o 2, (14) =1 where o s the label of x predcted by the local classfer constructed on the neghborhood of x, and the parameters of the local classfer can be represented by f va mnmzng some local structural loss functons as n Eq.(5). 728

4 Ths approach can be understood as a two-step approach for optmzng Eq.(10) wth γ I = 0: n the fst step, t optmzes the classfer parameters by mnmzng local structural loss (Eq.(4)); n the second step, t mnmzes the predcton loss of each data ponts by the local classfer constructed just on ts neghborhood. A Mxed-Regularzaton Vewpont In secton 3.3 we have stated that our algorthm ams to mnmze J = =1 (y f ) 2 + λf T Mf + γ I n 2fT Lf (15) where M s defned n Eq.(8) and L s the conventonal graph Laplacan constructed by Gaussan functons. It s easy to prove that M has the followng property. Theorem 2. M1 = 0, where 1 R n 1 s a column vector wth all ts elements equalng to 1. Proof. From the defnton of M (Eq.(8)), we have M1 = S T ĜS1 = S T Ĝ1 = 0, Therefore, M can also be vewed as a Laplacan matrx. That s, the last two terms of R l can all be vewed as regularzaton terms wth dfferent Laplacans, one s derved from local learnng, the other s derved from the heat kernel. Hence our algorthm can also be understood from a mxed regularzaton vewpont (Chapelle et al., 2006a)(Zhu & Goldberg, 2007). Just lke the multvew learnng algorthm, whch trans the same type of classfer usng dfferent data features, our method trans dfferent classfers usng the same data features. Dfferent types of Laplacans may better reveal dfferent (maybe complementary) nformaton and thus provde a more powerful classfer. Experments In ths secton, we present a set of experments to show the effectveness of our method. Frst let s descrbe the basc nformaton of the data sets. The Data Sets We adopt 12 data sets n our experments, ncludng 2 artfcal data sets g241c and g241n, three mage data sets USPS, COIL, dgt1, one BCI data set 2, four text data sets cornell, texas, wsconsn and washngton from the WebKB data set 3, and two UCI data sets dabetes and onosphere 4. Table 1 summarzes the characterstcs of the datasets. 2 All these former 6 data sets can be downloaded from benchmarks.html. 3 WebKB/. 4 MLRepostory.html. Table 1: Descrptons of the datasets Datasets Szes Classes Dmensons g241c g241n USPS COIL dgt cornell texas wsconsn washngton BCI dabetes onosphere Methods & Parameter Settngs Besdes our method, we also mplement some other competng methods for expermental comparson. For all the methods, ther hyperparameters were set by 5-fold cross valdaton from some grds ntroduced n the followng. Local and Global Regularzaton (). In the mplementaton the neghborhood sze s searched from {5, 10, 50}, γ A and λ are all searched from {4 3, 4 2, 4 1, 1, 4 1, 4 2, 4 3 } and we set λ + γi n = 1, the wdth of the Gaussan smlarty when constructng 2 the graph s set by the method n (Zhu et al., 2003). Local Learnng Regularzaton (). The mplementaton of ths algorthm s the same as n (Wu & Schölkopf, 2007), n whch we also adopt the mutual neghborhood wth ts sze search from {5, 10, 50}. The regularzaton parameter of the local classfer and the tradeoff parameter between the loss and local regularzaton term are searched from {4 3, 4 2, 4 1, 1, 4 1, 4 2, 4 3 }. Laplacan Regularzed Least Squares (). The mplementaton code s downloaded from manfold_regularzaton/software.html,, n whch the wdth of the Gaussan smlarty s also set by the method n (Zhu et al., 2003), and the extrnsc and ntrnsc regularzaton parameters are searched from {4 3, 4 2, 4 1, 1, 4 1, 4 2, 4 3 }. We adopt the lnear kernel snce our algorthm s locally lnear. Learnng wth Local and Global Consstency (). The mplementaton of the algorthm s the same as n (Zhou et al., 2004), n whch the wdth of the Gaussan smlarty s also by the method n (Zhu et al., 2003), and the regularzaton parameter s searched from {4 3, 4 2, 4 1, 1, 4 1, 4 2, 4 3 }. Gaussan Random Felds (). The mplementaton of the algorthm s the same as n (Zhu et al., 2003). Support Vector Machne (). We use lb (Fan et al., 2005) to mplement the algorthm wth a lnear kernel, and the cost parameter s searched from {10 4, 10 3, 10 2, 10 1, 1, 10 1, 10 2, 10 3, 10 4 }. 729

5 (a) g241c (b) g241n (c) USPS (d) COIL (e) dgt1 5 5 (f) cornell (g) texas 4 (h) wsconsn () washngton (j) BCI number of randomly labeled ponts (k) dabetes (l) onosphere Fgure 1: Expermental results of dfferent algorthms. 730

6 Table 2: Expermental results wth 10% of the data ponts randomly labeled g241c ± ± ± ± ± ± g241n ± ± ± ± ± ± USPS ± ± ± ± ± ± COIL ± ± ± ± ± ± dgt ± ± ± ± ± ± cornell ± ± ± ± ± ± 968 texas ± ± ± ± ± ± wsconsn ± ± ± ± ± ± washngton ± ± ± ± ± ± BCI ± ± ± ± ± ± dabetes ± ± ± ± ± ± onosphere ± ± ± ± ± ± Expermental Results The expermental results are shown n fgure 1. In all the fgures, the x-axs represents the percentage of randomly labeled ponts, the y-axs s the over 50 ndependent runs. From the fgures we can observe The algorthm works very well on the toy and text data sets, but not very well on the mage and UCI data sets. The and algorthm work well on the mage data sets, but not very well on other data sets. The algorthm works well on the mage and text data sets, but not very well on the BCI and toy data sets. works well when the data sets are not well structured, e.g. the toy, UCI and BCI data sets. works very well on almost all the data sets, except for the toy data sets. To better llustrate the expermental results, we also provde the numercal results of those algorthms on all the data sets wth 10% of the ponts randomly labeled, and the values n table 2 are the mean classfcaton accuraces and standard devatons of 50 ndependent runs, from whch we can also see the superorty of the algorthm. Conclusons In ths paper we proposed a general learnng framework based on local and global regularzaton. We showed that many exstng learnng algorthms can be derved from our framework. Fnally experments are conducted to demonstrate the effectveness of our method. References Belkn, M., Matveeva, I., and Nyog, P. (2004). Regularzaton and Sem-supervsed Learnng on Large Graphs. In COLT 17. Belkn, M., and Nyog, P. Towards a Theoretcal Foundaton for Laplacan-Based Manfold Methods. In COLT 18. Belkn, M., Nyog, P., and Sndhwan, V. (2006). Manfold Regularzaton: A Geometrc Framework for Learnng from Labeled and Unlabeled Examples. Journal of Machne Learnng Research 7(Nov): Bottou, L. and Vapnk, V. (1992). Local learnng algorthms. Neural Computaton, 4: Chapelle, O., Ch, M. and Zen, A. (2006). A Contnuaton Method for Sem-Supervsed s. ICML 23, Chapelle, O., B. Schölkopf and A. Zen. (2006). Sem- Supervsed Learnng. 508, MIT Press, Cambrdge, USA. Fan, R. -E., Chen, P. -H., and Ln, C.-J. (2005). Workng Set Selecton Usng Second Order Informaton for Tranng. Journal of Machne Learnng Research 6. Lal, T. N., Schröder, M., Hnterberger, T., Weston, J., Bogdan, M., Brbaumer, N., and Schölkopf, B. (2004). Support Vector Channel Selecton n BCI. IEEE TBE, 51(6). Gloub, G. H., Vanloan, C. F. (1983). Matrx Computatons. Johns Hopkng UP, Baltmore. Hen, M., Audbert, J. Y., and von Luxburg, U. (2005). From Graphs to Manfolds-Weak and Strong Pontwse Consstency of Graph Laplacans. In COLT 18, Schölkopf, B. and Smola, A. (2002). Learnng wth Kernels: Support Vector Machnes, Regularzaton, Optmzaton, and Beyond. The MIT Press, Cambrdge, MA. Smola, A. J., Bartlett, P. L., Scholkopf, B., and Schuurmans, D. (2000). Advances n Large Margn Classfers, The MIT Press. Vapnk, V. N. (1995). The Nature of Statstcal Learnng Theory. Berln: Sprnger-Verlag, Wang, F. and Zhang, C. (2006). Label Propagaton Through Lnear Neghborhoods. ICML 23. Wu, M. and Schölkopf, B. (2007). Transductve Classfcaton va Local Learnng Regularzaton. AISTATS 11. Zhou, D., Bousquet, O., Lal, T. N. Weston, J., & Schölkopf, B. (2004). Learnng wth Local and Global Consstency. In NIPS 16. Zhu, X., Ghahraman, Z., and Lafferty, Z. (2003). Sem- Supervsed Learnng Usng Gaussan Felds and Harmonc Functons. In ICML 20. Zhu, X. and Goldberg, A. (2007). Kernel Regresson wth Order Preferences. In AAAI. 731