Enclosing Machine Learning

Transcription

1 Enclosng Machne Learnng Xunka We* Department of Arcraft and Power Engneerng Unversty of Ar Force Engneerng Xan, Shaanx Chna voce: fax: emal: Ynghong L Department of Arcraft and Power Engneerng Unversty of Ar Force Engneerng Xan, Shaanx Chna voce: fax: emal: ynghong_l@126.com Yufe L Department of Arcraft and Power Engneerng Unversty of Ar Force Engneerng Xan, Shaanx Chna fax: emal: horzon_lyf@hotmal.com (* Correspondng author)

2 Enclosng Machne Learnng Xunka We, Unversty of Ar Force Engneerng, Chna Ynghong L, Unversty of Ar Force Engneerng, Chna Yufe L, Unversty of Ar Force Engneerng, Chna INTRODUCTION As s known to us, Cognton process s the nstnct learnng ablty of human beng, and ths process s perhaps the most complex but hghly effcent and ntellgzed nformaton processng process. For the cognton process of the natural world, human always transfers the feature nformaton to the bran through percepton, and then the bran wll process the feature nformaton and remember the feature nformaton for the gven objects. Snce the nventon of computer, scentsts are always workng toward mprovng ts artfcal ntellgence, and hope one day the computer could have ther own genune ntellgent bran lke the human bran. However, accordng to the cognton scence theory, the human bran can be mtated but cannot be completely reproduced. Thus, to let the computer truly thnk by themselves seems easy yet there s stll a long way to accomplsh ths objectve. Currently, artfcal ntellgence s stll an mportant and actve drecton of functon mtaton of the human bran. Yet tradtonally, the Neural-computng and neural networks famles are the majorty part of the drecton (Haykn, 1994). By mtatng the workng mechansm of human neuron of the bran, scentst found the neural networks computng theory accordng to expermental progresses such as Percepton neurons and Spkng neurons (Gerstner

3 & Kstler, 2002) n understandng the workng mechansm of neurons. For a long tme, the related research works manly emphasze on neuron model, neural network topology, learnng algorthm, and thus there are qute floursh large famles (Bshop, 1995) such as, Back Propagaton neural networks (BPNN), Radcal Bass Functon neural networks (RBFNN), Self Organzaton Map (SOM), and varous other varants. Neural-computng and neural networks (NN) famles have made great achevements n varous aspects. Recently, statstcal learnng and support vector machnes (SVM) (Vapnk, 1995; Scholkopf, & Smola, 2001) draw extensve attenton, show attractve and excellent performances n varous areas (L, We & Lu, 2004) compared wth NN, whch mply that artfcal ntellgence can also be made va advanced statstcal computng theory. Nowadays, these two methods tend to merge under statstcal learnng theory framework. BACKGROUND It should be noted as for NN and SVM, the functon mtaton s from the mcrocosmc vew utlzng the mathematc model of neuron workng mechansm. However the whole cognton process can also be summarzed as two basc prncples from the macroscopcal vew,.e. the frst s that human always cognzes thngs of the same knd, the second s that human recognzes thngs of a new knd easly wthout affectng the exstng knowledge. These two common prncples are easly concluded. In order to make the dea more clearly, we frstly analyze the functon mtaton explanaton of NN and SVM. The functon mtaton of human cogntve process for pattern classfcaton va NN and SVM can be explaned as follows (L & We, 2005). Gven the tranng pars (sample features, class ndcator), we can tran a NN or a SVM learnng machne. The tranng process of these learnng machnes actually mtates the learnng ablty of human beng.

4 For clarty, we call ths process cognzng. Then, the traned NN or SVM can be used for testng an unknown sample and determne the class ndcator t belongs to. The testng process of an unknown sample actually mtates the recognzng process of human beng. We call ths process recognzng. From the mathematc pont of vew, both these two learnng machnes are based on the hyperplane adjustment, and obtan the optmum or sub-optmum hyperplane combnatons after the tranng process. As for NN, each neuron acts as a hyperplane n the feature space. The feature space s dvded nto many parttons accordng to the selected tranng prncple. Each feature space partton s then lnked wth a correspondng class, whch accomplshes the cognzng process. Gven an unknown sample, t only detects the partton where the sample locates n and then assgns the ndcator of ths sample, whch accomplshes the recognzng process. Lke NN, SVM s based on the optmum hyperplane. Unlke NN, standard SVM determnes the hyperplane va solvng a QP convex optmzaton problem. They have the same cognzng and recognzng process except dfferent solvng strateges. Now, suppose we have a complete sample database, and f a totally unknown and novel sample comes, both SVM and NN wll not naturally recognze t correctly and conversely prefer to assgn a most close ndcator n the learned classes (L & We, 2005). However, ths phenomenon s qute easy for human to handle wth. If we have learned some thngs of the same knd before, gven smlar thngs we can easly recognze them. If we have never encountered wth them, we can also easly tell that they are fresh thngs. Then under supervsed learnng of them, we can then remember ther features n the bran wthout changng other learned thngs.

5 The root cause of ths phenomenon s the learnng prncple of the NN or SVM cognzng algorthm, whch s based on feature space partton. Ths knd of learnng prncple may amplfy each class s dstrbuton regon especally when the samples of dfferent knds are small due to ncompleteness. Ths makes t mpossble to automatcally detect the novel samples. Here comes the concern: how to make t automatcally dentfy the novel samples lke human. MAIN FOCUS Human beng generally cognzes thngs of one knd and recognzes complete unknown thngs of a novel knd easly. So the answer s why not let the learnng machne cognze or recognze lke human beng (L, We & Lu, 2004). In other words, the learnng machne should cognze the tranng samples of the same class regardless of the other classes, so that our ntenton s focused only on each sngle class. Ths pont s mportant to assure that all the exstng classes are precsely learned wthout amplfcaton. To learn each class, we can just let each class be cognzed or descrbed by a cogntve learner. It uses some knd of model to descrbe each class nstead of usng feature space partton so as to mtate the cognzng process. Therefore, now there s no amplfcaton occur dfferent from NN or SVM. The boundng boundary of each cogntve learner scatters n the feature space. All the learners boundares consst of the whole knowledge of the learned classes. For an unknown sample, the cogntve class recognzer then detects whether the unknown sample s located nsde a cogntve learner s boundary to mtate the recognzng process. If the sample s completely new (.e., none of the traned cogntve learner contans the sample), t can be agan descrbed by a new cogntve learner and the new obtaned learner can be added to the feature space wthout affectng others. Ths concludes the basc process of our proposed enclosng machne learnng paradgm (We, L & L, 2007A).

6 Mathematc Modelng In order to realze above deas for practcal usage, we have to lnk the deas wth concrete mathematc models (We, L & L, 2007A). Actually the frst prncple can be modeled as a mnmum volume enclosng problem. The second prncple can be deally modeled as a pont detecton problem. In fact, the mnmum volume enclosng problem s qute hard to solve for samples from arbtrary dstrbuton,and the actual dstrbuton shape mght be rather complex for calculatng drectly. Therefore, a natural alternatve s to use regular shapes such as sphere (Fscher, Gartner, & Kutz, 2003), ellpsod and so on to enclose all samples of the same class wth mnmum volume to approxmate the true mnmum volume enclosng boundary (We, Löfberg, Feng, L & L, 2007). Moreover, the approxmaton method can also be easly formulated as a convex optmzaton problem. Thus t can be effcently solved n polynomal tme usng state-of-the-art avalable open source solvers such as SDPT3 (Toh, Todd & Tutuncu, 1999), SEDUMI (Sturm, 1999), YALMIP (Löfberg, 2004) etc. Consequently, the pont detecton algorthm can be easly concluded va detectng ts locaton nsde t or not. Enclosng Machne Learnng Concepts Usng prevous modelng methods, we can now ntroduce some mportant concepts. Note that the new learnng methodology now actually has three aspects. The frst aspect s to learn each class respectvely, we call t cogntve learnng. The second aspect s to detect unknown samples locaton and determne ts ndcator, we call t cogntve classfcaton. Whle the thrd aspect s to conduct a new cogntve learnng process, we call t feedback self-learnng, and the thrd aspect s for mtaton of the character of learnng samples of new knd wthout affectng the exstng knowledge. The whole process can be depcted n Fg 1. We now can gve followng two defntons.

7 Class Learner. A cogntve class learner s defned as the boundng boundary of a mnmum volume set whch encloses all the gven samples. The cogntve learner can be ether a sphere or an ellpsod or ther combnatons. Fg 2, Fg 3 and Fg 4 depct the examples of sphere learner, ellpsod learner, and combnatonal ellpsod learner n 2D. Cognze Output Learner Recognze Output Results Class Learner Recognzer Yes Is Novel? No Self-Learnng Fg.1 Enclosng Machne Learnng Process. The real lne denotes the cognzng process. The dotted lne denotes the recognzng process. The dash-dotted lne denotes the feedback self-learnng process. Fg. 2 Sphere Learner

8 Fg.3 Ellpsod Learner Fg.4 Combnatonal Ellpsod Learner Remarks: As for the above llustrated three type learner, we can see that the sphere learner generally has the bggest volume, and next s sngle Ellpsod learner, and the combnatonal Ellpsod learner has the smallest volume. Recognzer. A cogntve recognzer s defned as the pont detecton and assgnment algorthm. The cogntve learner should own at least followng features to get commendable performance: A. regular and convenent to calculate B. boundng wth the mnmum volume C. convex bodes to guarantee optmalty D. fault tolerant to assure generalzaton performance.

9 The basc geometrc shapes are the best choces. Because they are all convex bodes and the operatons lke ntersecton, unon or complement of the basc geometrc shapes can be mplemented usng convex optmzaton methods easly. So we propose to use basc geometrc shapes such as sphere, box or ellpsod to serve as base learner. The cogntve learner s then to use these geometrc shapes to enclose all the gven samples wth the mnmum volume objectve n the feature space. Ths s the most mportant reason why we call ths learnng paradgm enclosng machne learnng. Cogntve Learnng & Classfcaton algorthms We frst nvestgate the dfference between enclosng machne learnng and other feature space partton based methods. Fg.5 gves a geometrc llustraton of the dfferences. For the cognzng (or learnng) process, each class s descrbed by a cogntve class descrpton learner. For the recognzng (or classfcaton) process, we only need to check whch boundng learner the testng sample locates nsde (a) (b) Fg.5. A geometrc llustraton of learnng a three class samples va enclosng machne learnng vs. feature space partton learnng paradgm. (a) For the depcted example, the cogntve learner s the boundng mnmum volume ellpsod, whle the cogntve recognzer s actually the pont locaton detecton algorthm of the testng sample. (b) All the three classes are separated by three hyperplanes.

10 But for the partton based learnng paradgm, among the learnng process, each two classes are separated va a hyperplane (or other boundary forms, such as hypersphere etc.). Whle among the classfcaton process, we need to check whether t s located on the left sde or the rght sde of the hyperplane and then assgn the correspondng class ndcator. We can see that the feature space partton learnng paradgm n fact amplfy the real dstrbuton regons of each class. But the enclosng machne learnng paradgm obtans more reasonable dstrbuton regon of each class. In enclosng machne learnng, the most mportant step s to obtan a proper descrpton of each sngle class of samples. From mathematc pont of vew, our cogntve class descrpton methods actually are the so-called one class classfcaton method (OCC) (Scholkopf, Platt, Shawe-Taylor, Smola, & Wllamson, 2001). OCC can recognze the new samples that resemble the tranng set and detect uncharacterstc samples, or outlers, to avod the ungrounded classfcaton. By far, the well-known examples of OCC are studed n the context of SVM. For ths problem, One Class Support Vector Machnes (OCSVM) (Tax & Dun, 1999) s frstly proposed. The OCSVM frst maps the data from the orgnal nput space to a feature space va some map, and then construct a hyperplane n whch separate the mapped patterns from the orgn wth maxmum margn. The one-class SVM proposed by Tax (Tax, 2001) s named support vector doman descrpton (SVDD), whch seeks the mnmum hypersphere that encloses all the data of the target class n a feature space. In ths way, t fnds the descrptve area that covers the data and excludes the superfluous space that results n false alarms. However, both OCSVM and SVDD depend on the Eucldean dstance, whch s often sub-optmal. An mportant problem n Eucldean dstance based learnng algorthm s the scale of

11 the nput varables. And thus Tax et al (Tax & Juszczak, 2003) proposes a KPCA based technques to rescale the data n a kernel feature space to unt varance n order to reduce the nput varable scale nfluences to mnma. And People proposed to maxmze the Mahalanobs dstance of the hyperplane to the orgn nstead, whch s the core dea of the One Class Mnmax Probablty Machne (OCMPM) (Lanckret, Ghaou & Jodan, 2002) and the Mahalanobs One Class Support Vector Machnes (MOCSVM) (Tsang, Kwok, & L, S., 2006). Because the Mahalanobs dstance s normalzed by the covarance matrx, t s lnear translaton nvarant. Therefore, we need not worry about the scales of nput varables. What s more, to allevate the undesrable effects of estmaton error n the covarance matrx, we can easly ncorporate a pror knowledge wth an uncertanty model and then address t as a robust optmzaton problem. Because Ellpsod and the accompanyng Mahalanobs dstance own many commendable vrtues mentoned above, we proposed to ncorporate Ellpsod and Mahalanobs nto class learnng. And then currently our man progress towards class learnng or cognzng s that we proposed a new mnmum volume enclosng ellpsod learner and several Mahalanobs dstance based OCC methods. In our prevous works, we proposed a QP based Mahalanobs Ellpsodal Learnng Machne (QP-MELM) (We, Huang & L, 2007A) and QP based Mahalanobs Hyperplane Learnng Machne (QP-MHLM) (We, Huang & L, 2007B) va solvng the dual form, and applcatons to real world datasets show promsng performances. However, as s suggested (Boyd, & Vandenberghe, 2004), f both the prmal form and dual form of an optmzaton problem s feasble, then the prmal form s more preferable. Therefore, we proposed a Second Order Cone Programmng representable Mahalanobs Ellpsodal Learnng

12 Machne (SOCP-MELM) (We, L, Feng & Huang, 2007A). And accordng to ths new learner, we developed several useful learnng algorthms. Mnmum Volume Enclosng Ellpsod Learner In ths new algorthm, we summarze several solutons (see Kumar, Mtchell, & Yldrm, 2003; Kumar, P. & Yldrm, 2005; Sun & Freund, 2004). As for the SDP soluton, we can drectly solve ts prmal form usng Schur complement theorem. As for the lndet soluton, we can solve ts dual effcently n polynomal tme. As for the SOCP soluton (We, L, Feng, & Huang, 2007B), we can also effcently solve ts prmal form n polynomal tme. We suppose all the samples are centered frstly. So we only gve results for mnmum volume enclosng ellpsod center at the orgn for ths case. But t s straghtforward for lft the ellpsod wth center n a d dmenson space to a generalzed ellpsod wth center at the orgn n a d 1dmenson space, for more detal, the reader may check the paper (We, L, Feng & Huang, 2007A) for more detal. Gven samples X m n R T 1, suppose c x x c x c ellpsod, then the mnmum volume problem can be formulated as (, ) : { : 1} s the demanded mn ln det Ab, 1 T 1 x c x c 1 st (1) However, ths s not a convex optmzaton problem. Fortunately, t can be transformed nto followng convex optmzaton problem mn ln det A Ab, T Ax b Ax b 1 st.. A 0, 1, 2,, n (2)

13 1 2 A usng matrx transform. 1 2 b c In order to allow errors, usng Schur Complete theorem, (2) can be represented n followng SDP form mn ln det A Ab,, 1 I Ax b st.. T 0 Ax b 1 n (3) Solvng (3), we can then obtan the mnmum volume enclosng ellpsod. Yet, SDP s qute demanded especally for large scale or hgh dmensonal data learnng problem. T As for 1 2 ( c, ) : { x : x c x c R }, we can reformulate the prmal form of mnmum volume enclosng problem as followng SOCP form: mn R, 0, c R N 1 st.. R 0, 0, 1,2,, N. T 1 ( x c) Σ ( x c) R, (4) Accordngly, t can be kernelzed as mn w, R, 0 R n 1 1 n Ω Q k Kw R 2 ( ), st.. R 0, 0, 1,2,, n. (5) Where c the center of the ellpsod s, R s the generalzed radus, n s number of samples, T and K Q ΩQ. C

14 So as to obtan more effcent solvng method, except above prmal form based methods, we can also reformulate the mnmum volume enclosng ellpsod centered at orgn as followng optmzaton problem: n 1 mn ln det U, 1 T 1 x x 1 st.. 0, 1,2,, n (4) Where balances the volume and the errors, 0 s slack varable. Actually va optmzaton condtons and KKT condtons, ths problem can be effcently solved usng followng dualzed representaton form: max ln det 0 st.. 1,, n n n T xx 1 1 (5) Where s the dual varable. We see that (5) cannot be kernelzed drectly, therefore we need to use some trcks [] to kernelzed ts equvalent counterpart max ln det 0 st.. 1,, n n 1 (6) 1 Where s the dual varable, : k( x1, x1 ) k( xn, x1 ), : n k ( xn, x1 ) k( xn, xn)

15 Multple Class Classfcaton algorthms As s ponted out prevously, cogntve learnng s actually to use mnmum volume geometrc shapes to enclose each class samples for mtatng the learnng process of human bran. Thus for multple class classfcaton problem, a naturally soluton s frstly to use mnmum volume geometrc shapes to approxmate each class samples dstrbuton, and then for gvng unknown samples, we only need to check whether they are nsde a learner or not. But these are for deal cases, where no overlaps occur n each sngle class dstrbutons. When overlaps occur, we proposed two algorthms to handle ths case (We, Huang & L, 2007C). For the frst soluton, we use a dstance based metrc, we would lke to assgn t to the closest class. Ths algorthm can be summarzed as f ( x) arg mn x c R k{1,2,, m} (7) Where denotes Mahalanobs norm. Another way s to use optmum Bayesan decson theory, and assgn ts ndcator to the class wth maxmum posteror probablty: P k M f( x) arg max exp( x c ) k{1,2,, m} 2 Rk (2 R ) k d 2 2 k 2 (8) where d s the dmenson of the feature space and 1 P s the pror dstrbuton of N k N 1y k 1 class k. Accordng to (8) the decson boundary between class 1 and 2 s gven by d 2 x c M 1 R1 2 R1 d 2 x c M 2 R2 2 R2 P(2 ) exp( ) P(2 ) exp( ) 1 (9)

16 And ths s equvalent to R 2 2 x c x c 1 M 2 M T 2 1 T R2 (10) Therefore we can gve a new decson rule x c M f ( x) arg max( Tk ) R k 2 k 2 (11) where T d log R log P can be estmated from the tranng samples. k k k Remarks. Actually, we also proposed a sngle MVEE learner based two class classfcaton algorthm (We, L, Feng & Huang, 2007A), whch owns both features of MVEE descrpton and SVM dscrmnaton. Then usng One Vs One or One Aganst One, we can also get a multple class classfcaton algorthm. Except ths, we are now workng on a multple class classfcaton algorthm at complexty of a sngle MVEE based two class classfcaton algorthm, whch s expected to obtan promsng performances. Gap tolerant SVM desgn Here we brefly revew the new gap tolerant SVM desgn algorthm. Ths new algorthm s based on the mnmum volume enclosng ellpsod learner for assurng a compact descrpton of all the samples. We frstly fnd the MVEE around all the samples and thus obtan a Mahalanobs transform. We then use the Mahalanobs transform to whten all the samples and thus map them to a sphere dstrbuton. Then we construct standard SVMs n ths whten space. The MVEE gap tolerant classfer desgn algorthm can be summarzed as Step1, Solve MVEE and obtan and center c Step2, Whten data usng Mahalanobs transform t 2 ( x c) and get new sample n parst, y 1 T Step3, Solve standard SVM and get the decson functon y( x) sgn( w t b). 1

17 Separaton Hyperplane Separaton Hyperplane M Mnmum Volume Enclosng Ellpsod M Mnmum Volume Enclosng Sphere Fg 6 MVEE gap tolerant classfer llustraton Remarks. Ths algorthm s very concse and has several commendable features worth notng. The classfer desgned usng ths algorthm has less VC dmenson compared wth tradtonal ones. Also ths algorthm s scale nvarant. For more detals, the reader should refer to (We, L & Dong, 2007). FUTURE TRENDS In the future, more learner algorthms wll be developed. Another mportant drecton s to develop set based combnatonal learner algorthm (We, & L, 2007; We, L, & L, 2007B). Also more reasonable classfcaton algorthms wll be focused. Except theoretcal developments, we wll also focus on applcatons such as novelty detecton (Dola, Page, Whte & Harrs, 2004), face detecton, ndustral process condton montorng, and many other possble applcatons. CONCLUSION In ths artcle, we ntroduced enclosng machne learnng paradgm. We focused on ts concept defnton, and progresses n modelng the cognzng process va mnmum volume enclosng ellpsod. We then ntroduced several learnng and classfcaton algorthms based on

18 MVEE. And we also report a new gap tolerant SVM desgn method based MVEE. Fnally, we gve future development drectons. REFERENCES Bshop, C. M. (1995) Neural Networks for Pattern Recognton, 1st edn, Oxford: Oxford Unversty Press Boyd, S. & Vandenberghe, L. (2004) Convex Optmzaton, 2nd edn, Cambrdge: Cambrdge Unversty Press Dola, A. N., Page, S. F., Whte, N. M. & Harrs, C.J. (2004) D-optmalty for Mnmum Volume Ellpsod wth Outlers. In: Proceedngs of the Seventh Internatonal Conference on Sgnal/Image Processng and Pattern Recognton (UkrOBRAZ'2004), Fscher, K., Gartner, B., & Kutz, M. (2003) Fast Smallest-Enclosng-Ball Computaton n Hgh Dmensons, In: Algorthms - ESA 2003, LNCS 2832, Gerstner, W. & Kstler, W. M. (2002) Spkng Neuron Models: Sngle Neurons, Populatons, Plastcty, 1st edn, Cambrdge: Cambrdge Unversty Press Haykn, S. (1994) Neural Networks: A Comprehensve Foundaton, 1st edn, NJ: Prentce Hall Press Löfberg, J. (2004) YALMIP: A Toolbox for Modelng and Optmzaton n MATLAB. ~joloef Kumar, P., Mtchell, J. S. B. & Yldrm, E. A. (2003) Approxmate mnmum enclosng balls n hgh dmensons usng core-sets, The ACM Journal of Expermental Algorthmcs, 8(1), 1-29 Kumar, P. & Yldrm, E. A. (2005) Mnmum volume enclosng ellpsods and core sets, Journal of Optmzaton Theory and Applcatons, 126(1), 1 21

19 Lanckret, G. Ghaou, L. E. & Jodan, M. (2002) Robust Novelty Detecton wth Sngle-Class MPM. In: Becker, S., Thrun, S. & Obermayor, K. (Eds.): NIPS 15 L, Y. H., We, X. K & Lu, J. X. (2004) Engneerng Applcatons of Support Vector Machnes. 1st edn. Bejng: Weapon Industry Press L, Y. H. & We, X. K. (2005) Fuson Development of Support Vector Machnes and Neural Networks, Journal of Ar Force Engneerng Unversty, 4, Scholkopf, B., Platt, J., Shawe-Taylor, J., Smola, A., & Wllamson R. (2001) Estmatng the Support of a Hgh Dmensonal Dstrbuton, Neural Computaton, 13(7), Scholkopf, B. & Smola, A. (2001) Learnng wth Kernels, 1st edn, Cambrdge, MA: MIT Press Shawe-Taylor, J., Wllams, C., Crstann, N. & Kandola, J. S. (2002) On the Egenspectrum of the Gram Matrx and Its Relatonshp to the Operator Egenspectrum. In: N. CesaBanch et al. (Eds.): Proceedngs of the 13th Internatonal Conference on Algorthmc Learnng Theory (ALT2002), LNAI 2533, Sturm, J. F. (1999) Usng SeDuM, a Matlab Toolbox for Optmzaton over Symmetrc Cones, Optmzaton Methods and Software, 11-12, Sun, P. & Freund, R. M. (2004) Computaton of Mnmum-Volume Coverng Ellpsods, Operatons Research, 5, Tax, D. M. J. (2001) One-class Classfcaton: Concept-learnng n the Absence of Counter- Examples, PhD Thess, Delft Unversty of Technology Tax, D. M. J. & Dun, R. P. W. (1999) Support Vector Doman Descrpton, Pattern Recognton Letters, 20, Tax, D. M. J. & Juszczak, P. (2003) Kernel Whtenng for One-Class Classfcaton, Internatonal Journal of Pattern Recognton and Artfcal Intellgence, 17(3),

20 Toh, K.C., Todd, M.J. & Tutuncu, R.H. (1999) SDPT3-a Matlab Software Package for Semdefnte Programmng, Optmzaton Methods and Software, 11, Tsang, Ivor W. Kwok, James T. & L, S. (2006) Learnng the Kernel n Mahalanobs One-class Support Vector Machnes, n Proceedng of IJCNN 2006 Conference, Vapnk, V. N. (1995) The Nature of Statstcal learnng theory, 1st edn, New York: Sprnger- Verlag We, X. K., Huang, G. B. & L, Y. H. (2007A) Mahalanobs Ellpsodal Learnng Machne for One Class Classfcaton. In Proceedng of ICMLC 2007, Accepted We, X. K., Huang, G. B. & L, Y. H. (2007B) A New One Class Mahalanobs Hyperplane Learnng Machne based on QP and SVD. In Proceedng of LSMS2007, DCDIS journal, Accepted We, X. K., Huang, G. B. & L, Y. H. (2007C) Bayes Cogntve Ellpsodal Learnng Machne for Recognzng Process Imtaton. In Proceedng of LSMS2007, DCDIS journal, Accepted We, X. K., L, Y. H & Feng, Y. (2006) Comparatve Study of Extreme Learnng Machne and Support Vector Machnes. In: Wang, J. et al. (Eds.): Advances n Neural Networks - ISNN 2006, LNCS 3971, We, X. K., L, Y. H., Feng, Y. & Huang, G.B. (2007A) Mnmum Mahalanobs Enclosng Ellpsod Machne for Pattern Classfcaton. In: Huang, D.-S., Heutte, L. & Loog, M. (Eds.): ICIC 2007, CCIS 2, We, X. K., L, Y. H, Feng, Y. & Huang, G.B. (2007B) Solvng Mahalanobs Ellpsodal Learnng Machne va Second Order Cone Programmng. In: Huang, D.-S., Heutte, L. & Loog, M. (Eds.): ICIC 2007, CCIS 2,

21 We, X. K. & L, Y. H. (2007) Lnear Programmng Mnmum Sphere Set Coverng for Extreme Learnng Machnes, Neurocomputng, DOI: /j.neucom We, X.K., L, Y.H., and Dong, Y. (2007) A New Gap Tolerant SVM Classfer Desgn based on Mnmum Volume Enclosng Ellpsod, In Chna Conference of Pattern Recognton 2007, Accepted. We, X. K., L, Y. H, & L, Y. F. (2007A) Enclosng machne learnng: Concepts and Algorthms, Neural computng and Applcatons, DOI /s y We, X. K., L, Y. H. & L, Y. F. (2007B) Optmum Neural Network Constructon Va Lnear Programmng Mnmum Sphere Set Coverng. In: Alhajj, R. et al. (Eds.): ADMA 2007, LNAI 4632, We, X. K., Löfberg, J., Feng, Y., L, Y. H., & L, Y.F. (2007) Enclosng Machne Learnng for Class Descrpton. In: Lu, D. et al. (Eds.): Advances n Neural Networks - ISNN 2007, LNCS 4491,

22 KEY TERMS AND THEIR DEFINITIONS Enclosng Machne Learnng: It s a new machne learnng paradgm whch s based on functon mtaton of human beng s cognzng and recognzng process. Cogntve Learner: A cogntve learner s defned as the boundng boundary of a mnmum volume set whch encloses all the gven samples to mtate the learnng process. Cogntve Recognzer: A cogntve recognzer s defned as the pont detecton and assgnment algorthm to mtate the recognzng process. MVEE Gap Tolerant Classfer: A MVEE Gap Tolerant Classfer s specfed by the shape matrx and locaton of an ellpsod, and by two hyperplanes, wth parallel normals. The set of ponts lyng n between (but not on) the hyperplanes s called the margn set. Ponts that le nsde the ellpsod but not n the margn set are assgned a class, 1, dependng on whch sde of the margn set they le on. All other ponts are defned to be correct: they are not assgned a class. A MVEE gap tolerant classfer s n fact a specal knd of Support Vector Machne whch does not count data fallng outsde the ellpsod contanng the tranng data or nsde the margn as an error.