Semi-Supervised Discriminant Analysis Based On Data Structure

Transcription

1 IOSR Journal of Computer Engneerng (IOSR-JCE) e-issn: ,p-ISSN: , Volume 17, Issue 3, Ver. VII (May Jun. 2015), PP Sem-Supervsed Dscrmnant Analyss Based On Data Structure Xuesong Yn *, Rongrong Jang, Lfe Jang Department of Computer Scence & echnology, Zheang Rado & V Unversty, Hangzhou, , Chna Abstract: Dmensonalty reducton s a key data-analytc technque for mnng hgh-dmensonal data. In ths paper, we consder a general problem of learnng from parwse constrants n the form of must-lnk and cannotlnk. As one knd of sde nformaton, the must-lnk constrants mply that a par of nstances belongs to the same class, whle the cannot-lnk constrants compel them to be dfferent classes. Gven must-lnk and cannot-lnk nformaton, the goal of ths paper s learn a smooth and dscrmnatve subspace. Specfcally, n order to generate such a subspace, we use parwse constrants to present an optmzaton problem, n whch a least squares formulaton that ntegrates both global and local structures s consdered as a regularzaton term for dmensonalty reducton. Expermental results on benchmark data sets show the effectveness of the proposed algorthm. Keywords: Dmensonalty reducton; Parwse constrants; Hgh-dmensonal data; Dscrmnant analyss. I. Introducton Applcatons n varous domans such as face recognton, web mnng and mage retreval often lead to very hgh-dmensonal data. Mnng and understandng such hgh-dmensonal data s a contemporary challenge due to the curse of dmensonalty. However, the current researches [3,4,5,6,13] show that reducng the dmensonalty of data has been a feasble method for more precsely characterng such data. herefore, dmensonalty reducton (DR for short) makes sense n many practcal problems [2,3,5] snce t allows one to represent the data n a lower dmensonal space. Based on the avalablty of the sde nformaton, dmensonalty reducton algorthms fall nto three categores: supervsed DR [7,12,28,35], sem-supervsed DR [8,16,18,21,22] and unsupervsed DR [10,13,14,15,34]. In ths paper, we focus on the case of sem-supervsed DR. Wth few constrants or class label nformaton, exstng sem-supervsed DR algorthms appeal to proectng the observed data onto a low-dmensonal manfold, where the margn between data form dfferent classes s maxmzed. Most algorthms n ths category, such as Localty Senstve Dscrmnant Analyss [20], Localty Senstve Sem-supervsed Feature Selecton [36], and Sem-supervsed Dscrmnant Analyss [26], greatly take the ntrnsc geometrc structure nto account,.e. local structure of the data, ndcatng that nearby nstances are lkely to have the same label. However, global structure of the data, mplyng that nstances on the same structure (typcally referred to as a cluster or a manfold) are lke to have the same label [1], s gnored n those algorthms. In fact, when mnng the data, global structure s the same sgnfcance as local structure [1,10,11,29]. herefore, t s necessary to ntegrate both global and local structures nto the process of DR. he key challenge s how we can ncorporate relatvely few parwse constrants nto DR such that the represented data can stll capture the avalable class nformaton. o ths end, we propose a sem-supervsed dscrmnant analyss algorthm wth ntegratng both global and local structures (SGLS), whch naturally address DR under sem-supervsed settngs. Gven parwse constrants, the key dea n SGLS s to ntegrate both global and local structures n sem-supervsed dscrmnant framework so that both dscrmnant and geometrc structures of the data can be accurately captured. he SGLS algorthm has the same flavor as not only supervsed DR algorthms, whch try to adust the dstance among nstances to mprove the separablty of the data, but unsupervsed DR algorthms as well, whch make nearby nstances n the orgnal space close to each other n the embeddng space. he remander of the paper s organzed as follows. In Secton 2, related work on the exstng algorthms of sem-supervsed DR s dscussed. In Secton 3, we ntroduce the general framework of our proposed SGLS algorthm. Secton 4 dscusses the expermental results on a number of real-world data sets. Fnally, we draw conclusons n Secton 5. II. Related Work Unsupervsed DR has attracted consderable nterest n the last decade [9,19,23,25,29]. Recently, a few researchers [5,9,13,17] have consdered the case that dscovers the local geometrcal structure of the data manfold when the data le n a low-dmensonal space of the hgh-dmensonal ambent space. Algorthms exploted along ths lne are used to estmate geometrcal and topologcal propertes of the embeddng space, *Correspondng author: Xuesong Yn, ynxs@nuaa.edu.cn DOI: / Page

2 such as Localty Preservng Proectons (LPP) [34], Local Dscrmnant Embeddng (LDE) [35], Margnal Fsher Analyss (MFA) [37] and ransductve Component Analyss (CA) [27]. Dfferent wth typcal unsupervsed DR algorthms, more lately, K-means was appled n the dmenson-reduced space for avodng the curse of dmensonalty n several algorthms [25,31] whch perform clusterng and dmensonalty reducton smultaneously. Lnear Dscrmnant Analyss (LDA) [7,12], capturng the global geometrc structure of the data by smultaneously maxmzng the between-class dstance and mnmzng the wthn-class dstance, s a wellknown approach for supervsed DR. It has been used wdely n varous applcatons, such as face recognton [5]. However, a maor shortcomng of LDA s that t fals to dscover the local geometrcal structure of the data manfold. Sem-supervsed DR can be consdered as a new ssue n DR area, whch learns from a combnaton of both the sde nformaton and unlabeled data. In general, the sde nformaton can be expressed n dverse forms, such as class labels and parwse constrants. Parwse constrants nclude both must-lnk form n whch the nstances belong to the same class and cannot-lnk form n whch the nstances belong to dfferent classes. Some sem-supervsed DR algorthms [20,26,27,30,36,37] ust appled partly labeled nstances to maxmze the margn between nstances form dfferent classes for mprovng performance. However, n many practcal data mnng applcatons, t s farly expensve to label nstances. In contrast, t could be easer for an expert or a user to specfy whether some pars of nstances belong to the same class or not. Moreover, the parwse constrants can be derved from labeled data but not vce versa. herefore, makng use of parwse constrants to reduce the dmensonalty of the data has been an mportant topc [8,16,33]. III. he SGLS Algorthm 3.1 he Integratng Global and Local structures Framework N Gven a set of nstances x 1, x n R, we can use a k-nearest neghbor graph to model the relatonshp, n between nearby data ponts. Specfcally, we put an edge between nodes and f x and x are close,.e. x and x are among k nearest neghbors of each other. Let the correspondng weght matrx be S, defned by S 2 x x t 0 otherwse exp( ) x ( ) ( ) 2 Nk x or x Nk x (1) Where Nk( x) denotes the set of k nearest neghbors of x. In general, f a weghted value S s large, x and x are close to each other n the orgnal space. hus, n order to preserve the localty structure, the embeddng space obtaned should be as smooth as possble. o learn an approprate representaton y ( y [ y1,, y ] n ) wth the localty structure, t s common to mnmze the followng obectve functon [13,34]: 2 y 2 y S y Ly (2) where D s a dagonal matrx whose dagonal entres are column (or row, snce S s symmetrc) sum of S, that s D n S. L D S s the Laplacan matrx. 1 On the other hand, we expect that the proectons{ ax } generated by a proecton vector, a R d, whch span the fnal embeddng space, approach the suffcently smooth embeddngs{ y } as much as possble. hs expectaton can be satsfed by the least squares formulaton whch can ncorporate the local structure nformaton va a regularzaton term defned as n Eq. (2). Mathematcally, a proecton vector a can be acqured by solvng the followng optmzaton problem [27]: 2 mn J( a, y) X a y y Ly (3) where 1 0 s the trade-off parameter. Here we assume that the data matrx X has been centered. akng the dervatve of J(a,y) wth respect to y and settng t equals to zero, we get 1 DOI: / Page

3 It follows that Sem-supervsed Dscrmnant Analyss based on Data Structure J ( a, y) 2( y X a 1Ly ) 0 y 1 y* ( I 1L) X a (4) (5) where I s the dentty matrx of sze n. Applyng Eq. (5) to Eq. (3), we can elmnate y and obtan the followng optmzaton problem wth regard to a: mn J ( a) J( a, y*) y* ( L)( I L) y * a X ( I L) ( L) X a a XGX a where G s a postve sem-defnte matrx of sze n by n and has the followng expresson: (6) G ( I L) ( L) (7) We can observe from Eq. (6) that mnmzng J wll brng the smooth subspace where both global and local structures of data are calculated wth regard to a. 3.2 Dscrmnat Analyss Wth Parwse Constrants Suppose we are gven two sets of parwse constrants ncludng must-lnk (M) and cannot-lnk (C). In the prevous work [32,38,40,41], such constrants were used for learnng an adaptve metrc between the prototypes of nstances. However, recent research has shown that the dstance metrc learned on hghdmensonal space s relatvely unrelable [33,39]. Instead of usng constrant-guded metrc learnng, n ths paper our goal s to use parwse constrants to fnd a proecton vector such that n the embeddng space the dstance between any par of nstances nvolved n the cannot-lnk constrants s maxmzed whle that between any par of nstances nvolved must-lnk constrants s mnmzed. If the proecton vector a s acqured, we can ntroduce such a transformaton: y a x (8) Under ths transformaton, we calculate the followng sum of the squared dstances of the pont pars n M: dw ( a x a x ) ( a x a x ) ( x, x ) M (9) a S a where S s the covarance matrx of the pont pars n M: w Correspondngly, for the pont pars n C, we have where S has the followng expresson form: b w S ( x x )( x x ) (10) w ( x, x ) M d b (11) a Sba S ( x x )( x x ) (12) b ( x, x ) C Smlar to LDA that seeks drectons on whch the data ponts of dfferent classes are far from each other whle requrng data ponts of the same class to be close to each other, we try to maxmze d b and mnmze d w. hus, we can optmze the followng problem to obtan the proecton drecton: a Sba max (13) a a S a w DOI: / Page

4 3.3 he Optmzaton Framework So far, we pursue the proecton drecton a from two formulatons: (1) we ntroduce a least squares formulaton for fndng a, whch facltes the ntegraton of global and local structures; (2) we obtan the proecton vector by maxmzng the cannot-lnk covarance and smultaneously mnmzng the must-lnk covarance. When the frst formulaton s consdered as a regularzaton term, we arrve at the followng optmzaton problem: a Sba max (14) a (1 ) a S a J ( a ) 2 w 2 where 0 2 1controls the smoothness of estmator. he optmal proecton vector a that maxmzes the obectve functon (14) s acqured by the maxmum egenvalue soluton to the followng generalzed egenvalue problem. S a ((1 ) S XGX ) a (15) b 2 w 2 In real-world applcatons, we usually need more proecton vectors to span a subspace rather than ust one. Let the matrx characterng such a target subspace be A [ a1,, a d ] whch s formed by the egenvectors assocated wth the d largest egenvalues of Eq. (15). Also, t s worth notng that snce our method s lnear, t Ax. has a drect out-of-sample extenson to novel sample x,.e. Based on the analyss above, we propose to develop a sem-supervsed dmensonalty reducton algorthm whch ntegrates both global and local structures defned n Eq. (14). he correspondng algorthm, called SGLS s presented n Algorthm 1. Algorthm 1 Input: X, M, C. N d Output: A proecton matrx A R. 1 Construct an unsupervsed graph on all n nstances to calculate S as n Eq. (2) and L = D S. 2 Present a least squares formulaton together wth the local structure nformaton to compute matrx G as n Eq. (7). 3 Compute Sw and Sb as n Eqs. (10) and (12). 4 Compute a generalzed egenvalue problem Eq. (15) and let the solved egenvectors be a,, 1 ad order of decreasng egenvalues. A [ a,, a d ]. 5 Output 1 o get a stable soluton of the egen-problem n Eq. (15), the matrx s requred to be non-sngular whch s not true when the number of features s larger than the number of nstances,.e. N > n. In such case, we can apply the khonov regularzaton technque [24,26] to mprove the estmaton. hus, the generalzed egenproblem n ths paper can be rewrtten as: n an S a ((1 ) S XGX I ) a (16) b 2 w 2 N where I N s the dentty matrx of sze N and 0 s a regularzaton parameter. IV. Experments In ths secton, we study the performance of our SGLS algorthm n terms of clusterng accuracy on several real-world data sets from dfferent applcaton domans. We set k = 5 and t = 1 for the constructon of the smlarty matrx S defned n Eq. (1). 4.1 Expermental Setup In order to verfy the effcacy of the proposed algorthm, we compare t wth the state-of-the-art, semsupervsed dmensonalty reducton algorthms ncludng SSDR [22], SCREEN [33], LADKM [31] and LMDM [16]. We mplemented SGLS n the MALAB envronment. All experments were conducted on a PENIUM DUAL 3G PC wth 2.0 GB RAM. We compared all algorthms on seven benchmark data sets, ncludng Segment and Dgts from UCI machne Learnng Repostory, three document data sets DOC1, DOC2 and DOC3 from the 20-Newgroups data, and two mage data sets : USPS handwrtten data and Yale Face B (YALEB). he statstcs of all data sets are summarzed n able 1. DOI: / Page

5 able 1: Summary of the test datasets used n our experment Data sets Instance Dmenson Class Segment Dgts DOC DOC DOC USPS YALEB As the unfed clusterng platform, we use K-means algorthm as the clusterng method to compare several algorthms after dmensonalty reducton. For each data set, we ran dfferent algorthms for 20 tmes and the comparson was based on the average performance. he parameter n SGLS are always set to 1 10 and f wthout extra explanaton. Moreover, for the far comparson, the dmensonalty of all the algorthms s set to K-1 (K s the number of clusterng). o evaluate the clusterng performance, n ths paper, we employ normalzed mutual nformaton (NMI) as the clusterng valdaton measure, whch s wdely used to verfy the performance of clusterng algorthms [8,25,33]. hen, NMI s defned as I( X, Y) NMI (17) H( X ) H( Y) where X and Y denote the random varables of cluster membershps from the ground truth and the output of clusterng algorthm, respectvely. I( X, Y) s the mutual nformaton between X and Y. H( X ) and HY ( ) are the entropes of X and Y respectvely. 4.2 Expermental Results DOI: / Page

6 Fgure 1 Clusterng performance on 6 data sets wth dfferent number of constrants able 2 presents the NMI results of varous algorthms on all seven data sets. Also, Fgure 1 shows the clusterng performance of standard K-means appled to the proected data by dfferent dmensonalty reducton algorthms wth dfferent numbers of parwse constrants. As can be seen, our SGLS algorthm acheves the best performance on Dgts, DOC1, DOC3, USPS, and YALEB data sets. In fact, the performance of our algorthm s always comparable to that of the other algorthm on the rest of all seven data sets used n ths paper. Specfcally, we can obtan the followng man observatons from able 2 and Fgure 1: (1) he SGLS algorthm sgnfcantly outperforms not only LMDM and LDAKM on all of the seven data sets used n the experment, but SCREEN and SSDR on the sx data sets as well. hs can be contrbuted to such a fact that ntegratng both global and local structures n SGLS may be benefcal. (2) It s nterestng to note that SGLS slghtly underperform both SSDR on Segment and SCREEN on DOC2. hs mples that n certan case such as Segment, global and local structures may capture smlar nformaton so that ntegratng such both structures seems to hardly help n the dmensonalty reducton process. (3) It s clear from the presented results that LDAKM performs relatvely worse that the other algorthms. hs can be explaned that LDAKM does not employ any sde nformaton to pursue the proecton drecton, whch only makes use of the abundant unlabeled nstances. hus, LDAKM has a greatly nferor performance. (4) Due to the fact that SCREEN and LMDM ust parwse constrants to learn an optmal subspace, ther performances are worse that those of SSDR and SGLS respectvely. In fact, a large number of the unlabeled data play an mportant role for sem-supervsed dmensonalty reducton [8,26,27]. (5) Snce SSDR can apply both the parwse constrants and the unlabeled data to learn a subspace, t can provde a relatvely satsfyng clusterng performance n comparson wth LDAKM, SCREEN and LMDM. Compared wth our SGLS, however, SSDR has stll provded the farly nferor performance. Actually, SSDR can be consdered as the constraned Prncpal Component Analyss (PCA). hus, t fals to preserve local neghborhood structures of the nstances n the reduced low-dmensonal space. able 2: NMI comparsons on seven data sets Data sets LDAKM SCREEN SSDR LMDM SGLS Segment Dgts DOC DOC DOC USPS YALEB V. Conclusons In ths paper, we propose a novel sem-supervsed dmensonalty reducton algorthm, called SGLS. For dmensonalty reducton and clusterng, SGLS ntegrates both global and local geometrc structures so that t can be naturally extended to deal wth the abundant labeled data, as graph Laplacan s defned on all the nstances. Compared wth the state-of-the-art dmensonalty reducton algorthms on a collecton of benchmark data sets n terms of clusterng accuracy, SGLS can always acheve a clear performance gan. Extensve experments exhbt the advantages of the novel sem-supervsed clusterng method of SGLS+K-means. DOI: / Page

7 Acknowledgments he research reported n ths paper has been partally supported by Natural Scence Foundaton of Zheang Provnce under Grant No. Y , Publc Proect of Zheang Provnce under Grant No. 2013C33087, Academc Clmbng Proect of Young Academc Leaders of Zheang Provnce No. pd References [1]. D. Zhou, O. Bousquet,.N. Lal, J. Weston, and B. Scholkopf. Learnng wth local and global consstency. In Advances n Neural Informaton Processng Systems 16, 2003, pp [2]. P. Garca-Laencna, G. Rodrguez-Bermudez, J. Roca-Dorda. Explorng dmensonalty reducton of EEG features n motor magery task classfcaton, Expert Systems wth Applcatons, 41 (11) (2014) [3]. Y. Ln,. Lu, C. Fuh Dmensonalty Reducton for Data n Multple Feature Representatons. In Advances n Neural Informaton Processng Systems 21, 2008, pp [4]. X. Zhu, Z. Ghahraman, and J. Lafferty. Sem-Supervsed Learnng usng Gaussan Felds and Harmonc Functons, Proc. wenteth Int l Conf. Machne Learnng, 2003, pp [5]. X. He, S. Yan, Y. Hu, P. Nyog, H. Zhang, Face recognton usng laplacanfaces, IEEE rans. Pattern Anal. Mach. Intell. 27 (3) (2005) [6]. S. Rowes, L. Saul. Nonlnear dmensonalty reducton by locally lnear embeddng, Scence, 290 (5500) (2000) [7]. R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Classfcaton. Wley-Interscence, Hoboken, NJ, 2nd edton, [8]. X. Yn,. Shu, Q. Huang. Sem-supervsed fuzzy clusterng wth metrc learnng and entropy regularzaton. Knowledge-Based Systems, 2012, 35(11): [9]. M. Belkn and P. Nyog. Laplacan Egenmaps for Dmensonalty Reducton and Data Representaton. Neural Computaton, 15(6) (2003) [10]. J. enenbaum, V. de Slva, and J. Langford. A global geometrc framework for nonlnear dmensonalty reducton. Scence, 290(5500) (2000) [11]. F. sa. Dmensonalty reducton technques for blog vsualzaton, Expert Systems wth Applcatons, 38(3) (2011) [12]. J. Yang, A.F. Frang, J.Y. Yang, D. Zhang, Z. Jn, KPCA plus LDA: A complete kernel fsher dscrmnant framework for feature extracton and recognton, IEEE rans. Pattern Anal. Mach. Intell. 27 (2) (2005) [13]. M. Belkn, P. Nyog, Laplacan egenmaps and spectral technques for embeddng and clusterng, n: Advances n Neural Informaton Processng Systems 14, 2001, pp [14]. D. Ca, X. He, J. Han, Document clusterng usng localty preservng ndexng, IEEE rans. Knowl. Data Eng. 17 (12) (2005) [15]. J. He, L. Dng, L. Jang, Z. L, Q. Hu. Intrnsc dmensonalty estmaton based on manfold assumpton, Journal of Vsual Communcaton and Image Representaton, 25( 5)(2014) [16]. S. Xang, F. Ne, and C. Zhang. Learnng a Mahalanobs dstance metrc for data clusterng and classfcaton. Pattern Recognton, 41(12) (2008) [17]. D. Ca, X. He, J. Han, H.-J. Zhang, Orthogonal laplacanfaces for face recognton, IEEE rans. Image Process. 15 (11) (2006) [18]. Z. Zhang, M. Zhao,. Chow. Margnal sem-supervsed sub-manfold proectons wth nformatve constrants for dmensonalty reducton and recognton, Neural Networks, 36(2012) [19]. J. Chen, J. Ye, and Q. L. Integratng Global and Local Structures: A Least Squares Framework for Dmensonalty Reducton, n: IEEE Computer Socety Conference on Computer Vson and Pattern Recognton, 2007, pp [20]. D. Ca, X. He, K. Zhou, J. Han, H. Bao, Localty senstve dscrmnant analyss, n: Internatonal Jont Conference on Artfcal Intellgence, 2007, pp [21]. W. Du, K. Inoue, K. Urahama, Dmensonalty reducton for sem-supervsed face recognton, n: Proceedngs of the Fuzzy Systems and Knowledge Dscovery, 2005, pp [22]. D. Zhang, Z.-H. Zhou, S. Chen, Sem-supervsed dmensonalty reducton, n: SIAM Conference on Data Mnng (SDM), 2007, pp [23]. X. Yang, H. Fu, H. Zha, J. Barlow, Sem-supervsed nonlnear dmensonalty reducton, n: Proceedngs of the Internatonal Conference on Machne Learnng, 2006, pp [24]. M. Belkn, P. Nyog, V. Sndhwan, Manfold regularzaton: a geometrc framework for learnng from labeled and unlabeled examples, J. Mach. Learnng Res. 1 (1) (2006) [25]. J. Ye, Z. Zhao, H. Lu. Adaptve dstance metrc learnng for clusterng. In: Proc. of the Conference on Computer Vson and Pattern Recognton, 2007, pp [26]. D. Ca, X. He, J. Han, Sem-supervsed dscrmnant analyss, n: Proceedngs of the Internatonal Conference on Computer Vson, 2007, pp [27]. W. Lu, D. ao, and J. Lu. ransductve Component Analyss, n: Proceedngs of the Internatonal Conference on Data Mnng, 2008, pp [28]. P. Han, A. Jn, A. Kar. Kernel Dscrmnant Embeddng n face recognton, Journal of Vsual Communcaton and Image Representaton, 22(7) (2011) [29]. J. Ye. Least Squares Lnear Dscrmnant Analyss, n: Proceedngs of Internatonal Conference on Machne Learnng, 2007, pp [30]. X. Chen, S. Chen, H. Xue, X. Zhou. A unfed dmensonalty reducton framework for sem-pared and sem-supervsed mult-vew data, Pattern Recognton, 45(5) (2012) [31]. C. Dng,. L. Adaptve dmenson reducton usng dscrmnant analyss and K-means clusterng. In: Proceedngs of the Internatonal Conference on Machne Learnng, 2007, pp [32]. E. Xng, A. Ng, M. Jordan, S. Russell. Dstance metrc learnng wth applcaton to clusterng wth sde-nformaton. In: Advances n Neural Informaton Processng Systems 15, 2003, pp [33]. W. ang, H. Xong, S. Zhong, J. Wu. Enhancng sem-supervsed clusterng: a feature proecton perspectve. In: Proceedngs of the Internatonal Conference on Knowledge Dscovery and Data Mnng, 2007, pp [34]. X. He, P. Nyog. Localty preservng proectons, In: Advances n Neural Informaton Processng Systems 16, [35]. H. Chen, H. Chang, and. Lu. Local dscrmnant embeddng and ts varants. In: Proceedngs of the Internal Conference on Computer Vson and Pattern Recognton, 2005, pp [36]. J. Zhao, K. Lu, X. He, Localty senstve sem-supervsed feature selecton, Neurocomputng, 71 (10-12), (2008) DOI: / Page

8 [37]. S. Yan, D. Xu, B. Zhang, H.J. Zhang, Q. Yang, and S. Ln. Graph embeddng and extenson: A general framework for dmensonalty reducton. IEEE ransactons on PAMI, 29(1) (2007) [38]. B. Kuls, S. Basu, I. Dhllon, and R. Mooney. Sem-supervsed graph clusterng: a kernel approach. In: Proceedngs of the Internatonal Conference on Machne Learnng, 2005, pp [39]. D. Meng, Y. Leung, Z. Xu. Passage method for nonlnear dmensonalty reducton of data on mult-cluster manfolds, Pattern Recognton, 46 (8) (2013) [40]. M. Blenko, S. Basu, R.J. Mooney, Integratng constrants and metrc learnng n sem-supervsed clusterng, n: Proceedngs of Internatonal Conference on Machne Learnng, 2004, pp [41]. D.Y. Yeung, H. Chang, Extendng the relevant component analyss algorthm for metrc learnng usng both postve and negatve equvalence constrants, Pattern Recognton 39 (5) (2006) DOI: / Page