The Discriminate Analysis and Dimension Reduction Methods of High Dimension

Transcription

1 Open Journal of Socal Scences, 015, 3, 7-13 Publshed Onlne March 015 n ScRes. The Dscrmnate Analyss and Dmenson Reducton Methods of Hgh Dmenson Lan Fu 1, 1 Statstcs School, Jangx Unversty of Fnance of Economcs, Nanchang, Chna Research Center of Appled Statstcs, Jangx Unversty of Fnance of Economcs, Nanchang, Chna Emal: fulan @sna.com Receved September 014 Abstract Statstcal methods have been gettng constant development snce 1970s. However, the statstcal methods of the bg data are no longer restrcted wth these methods whch are lsted n the textbook. Ths paper manly demonstrates the Dscrmnaton Analyss of Multvarate Statstcal Analyss, Lnear Dmensonalty Reducton and Nonlnear Dmensonalty Reducton Method under the crcumstances of the wde range of applcatons of hgh-dmensonal data. Ths paper ncludes three parts. To begn wth, the paper llustrates a developng trend from the data to the hgh-dmensonal. Meanwhle, t analyzes the mpacts of the hgh-dmensonal data on dscrmnate analyss methods. The second part represents the necessty of the dmensonalty reducton studes n the era of the hgh-dmensonal data overflowng. Then, the paper focuses on ntroducng the man methods of the lnear dmensonalty reducton. In addton, ths paper covers the basc dea of the nonlnear dmensonalty reducton. Moreover, t systematcally analyzes the breakthrough of the tradtonal methods. Furthermore, t chronologcally demonstrates the developng trend of the dmensonalty reducton method. The fnal part shows a comprehensve and systematc concluson to the whole essay and descrbes a developng prospect of the dmenson reducton methods n the future. The purpose of ths essay s to desgn a framework of a performance system whch s subject to the characterstcs of Chna Hgh-tech enterprses. It based on the analysng the prncples and sgnfcance of the performance system of Hgh-tech enterprses. The framework wll promote the standardze management of Hgh-tech enterprses of Chna. Keywords Hgh-Dmensonal Data, Method of Dmensonalty Reducton, Dscrmnant Analyss, Lnear Dmensonalty Reducton, Manfold Structure 1. Introducton The comng century s surely the century of data [1]. People dd have access to a large amount of fnancal data, mage data and space data recently. These data have a common feature whch s they are all hgh-dmensonal data. Actually, most data people obtaned are hgh-dmensonal n the world. The ncrease of data dmenson brngs the Gospel of the dmenson meanwhle leads to the data dsaster. Ths nformaton whch s hdden How to cte ths paper: Fu, L. (015) The Dscrmnate Analyss and Dmenson Reducton Methods of Hgh Dmenson. Open Journal of Socal Scences, 3,

2 by the hgh-dmensonal data provdes a possblty for solvng the scentfc problems. However, t s dffcult to magne the challenges of the data processng technology because of the ncrease of the data dmenson. The method of dmensonalty reducton s defned as a low-dmensonal expresson way whch can fathfully reflect the nherent feature of the raw data n hgh-dmensonal data. It caused the wdespread attenton of people because the dmenson reducton method s consdered as an effectve measure to prevent the curses of dmensonalty [] happenng. So far people had found a lot of dfferent dmenson reducton methods whch nvolve the Projecton Pursut Method, Prncple Component Analyss (PCA) and Manfold Learnng. However, the Isomer, Locally Lnear Embeddng (LLE) and Palladan Egen map are all based on the Manfold Learnng [3]. There are several dfferent methods of dvson for the dmenson reducton methods. Accordng to the nature of data, the method of dmenson reducton can be dvded nto Lnear Dmensonalty Reducton and Nonlnear Dmensonalty Reducton. In terms of the geometrcal structure of the preservaton degree of the dmenson reducton, t ncludes local approach and global approach. These dmenson reducton methods obtan systematcal analyss and contrast.. Hgh-Dmensonal Tendency of the Data and Its Impact.1. The Dervatve of Hgh-Dmensonal Data Wth the development of the scence and technology, people can obtan a great deal of nformaton to reveal the objectve law whch s covered by superfcal phenomenon. Ths leads to a tendency whch s the data s gradually development from the low-dmenson to the hgh-dmenson. In other words, the mprovement of data dmenson brngs the Gospel of the dmenson meanwhle result n the data dsaster [4]. The curses of dmensonalty was frst put forward by Bellman n At that tme, the curses of dmensonalty refers to the ncensement of the dmenson would lead to an exponental growth of samples n certan statstcal ndex model. Meanwhle, the ndcatng length and complexty of the processng nformaton would have a exponental growth. Nowadays the curses of dmensonalty manly refer to the ntrnsc sparsely n the hgh dmensonal data space. In other words, t s the spatal representaton n the hgh dmensonal data space. The curses of dmensonalty s caused by the ntrnsc dmenson rather than the representaton dmenson. It affects some felds whch nvolve Statstcal Estmaton, Numercal Integraton, Optmzaton Problems, and Probablty Densty Estmaton so on. Thus, the dmenson reducton method as a low-dmensonal expresson method of the hgh-dmensonal data gettng the essental characterstcs of the orgnal date can effectvely avod the curses of dmensonalty. It causes the wdespread attenton of people... Impact of Hgh Dmensonalty for Classfcaton There s a common problem n the process of classfcaton. The sze of the dmenson p of feature vectors s much greater than the sze of the tranng sample n. In addton, only a part of p feature vectors plays a mportant role n the classfcaton. Although the tradtonal analyss method s extremely mportant, t s not sutable for hgh-dmensonal case. The followng statement llustrates the nfluences of the hgh-dmenson for the Fsher dscrmnate analyss and ndependent classfcaton crtera respectvely. Besdes, the statement assumes the dscrmnaton occurs n the two hgh-dmensonal populatons. And the probablty of the two populatons s 50%. Meanwhle, there s comparablty between Sample n 1 and smple n...1. Fsher Dscrmnaton Based on the Hgh Dmensonalty 1 The Fsher dscrmnate s δ ( x ) ( x u T F ) ( u u 1 = Σ 1 ). It can assure both Σ and Σ are non ll-condtoned matrx n the parameter space. It also can guarantee one of the Mahalanobs dstances of two categores s postve C. Therefore, t can the utlzes ˆF δ of the sample data estmaton to obtan the asymptotc optmalty n low-dmenson stuaton. However, the stuaton s dfferent from hgh-dmenson. One reason s that the true covarance matrx s not ll-condtoned matrx. Another reason s that the abnormal value of the sample covarance s not sutable for the Fsher crteron because the stuaton of the sze of dmensonalty s greater than the sze of the sample.... Impact of the Dmensonalty for the Independent Crtera The dffcultes of the hgh-dmensonal dscrmnated le n a lot of nose characterstcs. These nose characte- 8

3 rstcs have no any functon for the reducton of the classfcaton errors. There s another reason for the poor results of the Fsher dscrmnate under the hgh-dmensonalty crcumstances. That s people can ndvdually estmates for each parameter. However, the error wll be great as people estmate multple feature estmaton at the same tme. Therefore, ths wll lead to the sgnfcant ncrease of the error probablty [5]. 3. The Man Method of the Dmenson Reducton 3.1. The Classfcaton of the Dmenson Reducton Method In the lght of the nature of the data, the method of dmenson reducton can be dvded nto Lnear Dmensonalty Reducton and Nonlnear Dmensonalty Reducton. These two methods can be further dvded nto some manstreamng classfcaton methods and the detals [6]. As shown Fgure The Man Methods of the Lnear Dmenson Reducton The lnear dmenson reducton method s most wdely applcaton n the dmensonalty reducton. There are some excellent propertes n the lnear dmenson reducton method. These propertes nvolve easly explcable, an analytcal solutons, smple calculaton and the effectveness of the lnear structure of the data collecton. As matter of fact, the lnear dmensonalty reducton method s based on the dfferent optmzaton crteron. It seeks the best lnear model. Ths s totally dfferent wth nonlnear dmensonalty reducton method. However, t s the common property of all lnear dmensonalty reducton methods. The lnear dmensonalty reducton methods manly nclude the Prncpal Component Analyss and the Lnear Dscrmnate Analyss. The nonlnear dmensonalty reducton methods [7]-[9] chefly nvolve the Locally Lnear Embeddng, Palladan Egen maps, Isomer, Multdmensonal Scale Method and the Local Tangent Space Method so on Prncpal Component Analyss (PCA) The Prncpal Component Analyss [10] was frst put forward by Pearson (K.Pearson) n However, Hostellng (H. Hostellng) establshed a mathematcal foundaton for PCA n The theoretcal bass of PCA s based on the karhunen-levy expanson. The basc dea of PCA s that the orgnal varables can be transformed nto several comprehensve varables whch are able to reflect the man propertes of the thngs. Meanwhle, the hgh-dmensonal data can be shadowed on the low-dmensonal space. Ths ensures a most effectve low-dmensonal data n the least square sense and truly represents the orgnal hgh-dmensonal data. Ths method focuses on a fdelty optmzaton of the orgnal hgh-dmensonal data. It tres to solve a problem of the projecton drecton and obtan the representaton of the orgnal data n the least square sense. From the mathematcal pont of vew, the startng pont of the prncpal component analyss s the sample covarance matrx S. Fgure 1. The classfcaton of the dmenson reducton method. 9

4 s1 s1... s 1p s1 s... s p S = s1p sp sp s s the varance of the varable x. S j s the covarance between varable x and varable x j. There s no lnear correlaton between the varables x and the varables x j f the covarance s zero. The strength of the lnear correlaton between the varables x and the varables x j can be expressed by the correlaton coeffcent R y whch s equal sj ( ss j ). The change of the man spndle would transform the number of p correlatve varables ( x1, x, xp ) nto the number of p uncreatve varables ( z1, z, zp ). The varable of the new axes descrbed by an egenvector u. T z = u ( x X) That s the th prncpal components. (The mean of Z s zero, the varance s l whch s the th egenvalue.) There are some man steps of the dmenson reducton. The frst step s to obtan the egenvalues and the egenvectors of the covarance matrx, then rank the sze of the egenvalues. The second step s to choose the egenvector of the bg egenvalues correspondng and put ths egenvector as a projecton vector. The fnal step s the hgh-dmensonal data would be projected onto the subspace of the projecton vector spanned. Undoubtfully, there are some problems and lmtatons on the PCA. Frst of all, the PCA have to assume the approxmate normalty of the nput space. Secondly, the PCA have to assume that the nput data s real and contnuous. Thrdly, the PCA wll fal f the dstrbuton of the data had the complex manfold structure Lnear Dscrmnaton Analyss (LDA) The Lnear Dscrmnaton Analyss was frst put forward by Fsher (R.Fsher) n The LDA s smlar to the PCA. However, there are some dfferences between the LDA and the PCA. There are some common ponts between LDA and LDA. Frstly, LDA and PCA both are obtaned by a set of projecton vectors, and utlze the hgh-dmenson data to project the low-dmensonal space. Therefore, LDA and PCA have the smlar procedures. The second pont s that the same purpose of LDA and PCA both are dmensonalty reducton. Obvously, there are some dfferences between LDA and PCA. Frst of all, the projecton vectors of LDA and PCA are totally dfferent. The reason s that the key pont of LDA and PCA are dfferent n the process of dmensonalty reducton. PCA make the optmzaton of the orgnal hgh-dmenson data fdelty of the dmensonalty reducton data as the key. It tres to get a drecton of an optmal projecton. Then the projecton data can represent the orgnal data as far as possble n the constrant condtons. However, LDA focus on the optmzaton of the dfferent data dscrmnaton of the dmensonalty reducton data. The purpose of the LDA s Tyr as far as possble to separate the two types of data. 3.3.The Major Methods of the Nonlnear Dmenson Reducton Local Lnear Embeddng (LLE) The man steps of the Local Lnear Embeddng [11] [1] (LLE) nclude the followng ponts. The frst step s to fnd a set of the K-nearest neghbors of each pont. It means to compute a set of weghts for each pont that best descrbe the pont as a lnear combnaton of ts neghbors. Then t uses an egenvector-based optmzaton technque to fnd the low-dmensonal embeddng of ponts, such that each pont s stll descrbed wth the same lnear combnaton of ts neghbors. The Eucldean dstance and Djkstra dstance of keepng the surface characterstcs of sample ponts can be used n the dstance computaton. The second step s to get a local reconstructon weght matrx of the sample through computng the neghbor pont of each sample. Then t constructs a cost functon to measure the reconstructon error. The cost functon s expressed as ε ( W) = X- XW j j j 10

5 The W of the reconstructon weght s mnmzed under two constrants. One of the constrants s the W wll be zero f pont x s not a neghbor of the pont x j. Another constrant s W j=1. In other words, t s to get W. It means to solve the problem of the least square wth constrant. j The thrd step s to compute the output value of the sample by the local reconstructon weght matrx and the neghbor pont of the sample. Each sample x wll be mapped onto the low-dmensonal vector Y. The man deas of LLE nclude the followng aspects. The frst aspect s to assume each sample pont and ts neghbor pont n a lnear regon of manfold. Another one s to transform the global nonlnear nto the local lnear and the local overlap felds of each sample and neghborng pont can supply the global structure nformaton Local Tangent Space Algnment (LTSA) The emergence of the LTSA (014) s later than other dmensonalty reducton methods. The bass of the LTSA algorthm whch s proposed by Zhang Zhenyue and other researchers nclude the followng parts. One bass s that the LTSA can obtan a new global nonlnear structure by ntegratng the local lnear nformaton and local lnear analyss. Ths structure has the propertes of a nonlnear manfold. The LTSA ncludes two steps whch are projecton and ntegraton. The LTSA can properly show the good propertes of the geometry feature of the manfold. But t also has lmtatons. These lmtatons ndcate that the LTSA cannot do very well to the manfold learnng for hgh curvature Dmensonalty Reducton From the Begnnng to Nowadays The study of the dmensonalty reducton can be traced to The man methods of the dmensonalty reducton nclude Lnear Dscrmnate Analyss, Prncpal Component Analyss and Projecton Pursut so on. The developed dmensonalty reducton methods n recent years manly nvolve the Nonlnear Dmensonalty reducton (It s based on kernel.), the dmensonalty reducton of the two-dmensonalty and tensor, the Manfold Learnng and localzed dmensonalty reducton as well as Sem-supervsed Dmensonalty Reducton Nonlnear Dmensonalty Reducton Based on the Kernel Snce Schölkopf (B. Schölkopf) and other researchers ntroduced the kernel methods to the feld of the dmensonalty reducton, they proposed the classc Kernel Prncpal Component Analyss. In addton, researchers combned wth the kernel methods and other tradtonal dmensonalty reducton methods. Then they proposed some dscrmnate analyss methods. These methods nclude Fsher Dscrmnate Analyss, Kernel Canoncal Correlaton Analyss and Kernel Independent Component Analyss. So far, almost all of the lnear dmensonalty reducton methods have correspondng kemelzed verson. The kernel methods have become standard practce methods for a lnear dmensonalty reducton transformng nto a no lnearzaton dmensonalty reducton. The key pont s the selecton of the kernel and the kernel parameter n the dmenson reducton algorthm whch s based on the kernel. Researchers also focus on mprovng the operatonal effcency of the kernel dmensonalty reducton algorthm n the bg data Two-Dmensonalzaton and Tensor Dmensonalty Reducton The tradtonal model s defned as a sample pont whch s represented as a vector model of a pont n n-dmensonal space. It s obvously that the tradtonal model s more easly to be handled by varous statstcal methods. However, the tradtonal model s no longer sutable for applcaton n practcal under ths stuaton. Ths stuaton s the sample s usually non-vector model and wll transform two dmensonal models nto correspondng vector models. Ths stuaton would lead to the damage of the structure nformaton of the orgnal matrx and the adverse consequences of ncreased tme cost and storage cost. Therefore, the concept of the tensor dmensonalty reducton was proposed. The dea of two-dmensonal matrx mage feature extracton was frst put forward by Lu (K. Lu). Ths dea avods the above adverse consequences. That s people can drectly wthdraw the egenvalues and not necessary to pull a mage nto a vector. Snce then, the method had access to the development and research from dfferent angles. The Two-Dmensonal Prncpal Component Analyss (DPCA), Two-Dmensonal Lnear Dscrmnate Analyss (DLDA) and Tow-dmensonal Canoncal Correlaton Analyss (DCCA) were proposed. The methods of two dmensonal- 11

6 zaton and tensor dmensonalty reducton have the advantages of hghly computng effcency. Meanwhle, there are many expermental results whch ndcate the two-dmenson method s obvously better than one-dmensonal method n some datasets. Recently, the representaton method of the two dmensonlzaton and tensor model has been wdely appled n the felds of the machne learnng, pattern recognton and computer recognton. Tensor Lnear Dscrmnate Analyss,The tensor lnear dscrmnate algorthm tres to expand the dstance between classes. Meanwhle, t tres to shrnk the dstance wthn a class. There are some smlar between the thought of the tensor lnear dscrmnate analyss and the tradtonal lnear dscrmnate analyss. The dfferences s that the tradtonal lnear dscrmnate analyss method s only to seek a projecton vector x as n-dmensonal sample; N s the total sample; X belongs to the selecton of classes s = ( = 1,, N s ) ; N S s the sample of the sth categores; c s the number of categores for all samples. However, the tensor lnear dscrmnate analyss method s to seek and obtan the process of a seres of projecton matrx. Then t transforms the tensor sample nto the tensor data of the low dmenson. Meanwhle t would maxmze the class dstance and mnmze the wthn class dstance. The tensor lnear dscrmnate analyss method s to deal wth samples and extract the features of samples n the tensor feld. Ths can be consdered as an extenson of the tradtonal lnear dscrmnate analyss method n the tensor subspace. Compared wth vector, the tensor subspace effectvely retaned the sample nformaton and made full use of the collectng nformaton. At the same tme, the tensor lnear dscrmnate analyss method overcomes the ssue of the small sample sze n the tradtonal lnear dscrmnate method and mproves the learnng performance Manfold Learnng and Localzed Dmensonalty Reducton The manfold learnng has become a new hot research n machne learnng feld. Ths s happened n 000 whch s the year of the magazne scence publshed two classc artcles about the sometrc mappng (ISOMAP) and locally lnear embeddng (LLE). The manfoldng learnng s another way to obtan nonlnear dmensonalty reducton except for the kernel method. One of the basc assumptons of the manfold learnng s to dstrbute samples on a potental manfold. Thus, the nternal dmensonalty of the manfold s generally not hgh even though the nput space s the hgh-dmensonal space. After the emergence of Isomer and LLE method, there have been a number of other manfold learnng algorthms. These algorthms nvolved maxmum varance extenson, Laplace feature mappng and local tangent space algnment so on. Compared wth the tradtonal lnear dmensonalty reducton algorthm, the manfold learnng can effectvely solve the problem of the nonlnear. However, most manfold algorthms stll have some ssues about the low-effcency of computng and dffcult extendng of the test samples. The followng algorthm has been mproved for above problems respectvely. The locally preservng projecton s proposed by He (X. He) and other researchers s one of the most representatves. As a matter of fact, the locally preservng projecton s the lberalzed verson of the Laplace feature mappng. There are many dmensonalty reducton algorthms of the localzaton besdes the localty preservng projectons. These methods all utlzed the thoughts of the local preservaton. Wth the ntroducton of the class nterval thoughts of the support vector machne, the dmensonalty reducton algorthm of the nearest neghbor dscrmnate analyss was proposed. The comparson of the nearest neghbor dscrmnate analyss and localty preservng projecton shows that the nearest neghbor dscrmnate analyss not only can ntegrate category nformaton but also can obtan the dmensonalty reducton of the optmal dscrmnate ablty. Recently, a relevant typcal example was proposed n the scentfc feld. Yan (S. Yan) and other researchers proposed a general framework of the map embeddng. Most of the above dmensonalty reducton methods was nvolved n ths framework. The constructon and the selecton of parameters of a map s the key step n the dmenson redacton methods of the map embeddng. 4. The Summary and Prospect of the Dmenson Reducton Methods Undoubtfully, data dmensonalty reducton s an effectve tool for the work of the data dggng. That nformaton whch s hdden by the hgh-dmensonal data provdes a possblty for solvng the scentfc problems. Ths possblty s dependng on the mprovement and development of the dmenson reducton methods. The dmenson reducton method can obtan an effectvely low-dmensonal representaton and a way of the essental cha- 1

7 racterstcs of reflectng the orgnal data. Then the method of processng low-dmensonal data can be used. Ths can avod data dsaster. In terms wth the nature of the data, the method of dmenson reducton can be dvded nto two categores. One category s Lnear Dmensonalty Reducton. The other s Nonlnear Dmensonalty Reducton. In fact, the Lnear Dmensonalty Reducton s based on the dfferent crterson of optmzaton and to seek the optmal lnear model. Ths s totally dfferent from the Nonlnear Dmensonalty Reducton. However, the data of the practcal applcaton s usually not the lnear combnaton wthn features. Consequently, t trggered the study of the manfold learnng n the feld of the nonlnear dmensonalty reducton. The manfold learnng method commonly can be dvded nto three categores. These three categores are the global comparson of the lnear model, the nonlnear method of preservng local property and the nonlnear method of preservng global property. The method of the global comparson of the lnear model majorly refers to LLC. Whle the nonlnear method of preservng local property ncludes LLE, Hessan LLE, Palladan Egen map and LTSA. The nonlnear method of preservng global property nvolves the nonlnear dmenson reducton of the multdmensonal scalng of based on kernel (Kernel PCA, Fsher Analyss), Isomap and Dffuson map so on. The man dfferences between the global method and the local method le n the local structure and the way of embeddng. The dmenson reducton method was wdely welcomed n many felds snce t was proposed. These felds are the face recognton feld and cluster analyss feld so on. Over the past years, the dmenson reducton method has been mproved. However, there are some lmtatons n realty. These lmtatons especally presented n the manfold method. The frst lmtaton s that t s dffcult to obtan the mappng relatonshp from the hgh-dmensonal space to the low-dmensonal space durng the process of dmensonalty reducton. The second lmtaton s the dmenson reducton problem of processng dmenson jump. Thrdly, t s dffcult to get an effectve study method of the manfold as the data collecton s relatvely sparse. Fnally, t s lack of the studes n mprovng the effcency of handlng large amounts of data after the combnaton of the manfold algorthm and parallel computng. References [1] Donoho, L. (000) Hgh Dmensonal Data Analyss: the Curses and Blessngs of Dmensonalty. Present Data Amercan Mathematcs Socety Conference. [] Lu, T. (005) Dmenson Reducton Theory and Applcaton of the Hgh Dmensonal Data. 5-31, [3] XaoSheng, Y. and Nng, Z. (007) The Method Research of the Dmenson Reducton. Informaton Scence 8th. [4] SongCan, C. and DaoQang, Z. (009) The dmenson Reducton of the Hgh Dmensonal Data. Chna Computer Federaton Informaton 8th. [5] Fan, J.Q. and Fan, Y.Y. (008) Hgh Dmensonal Classfcaton Usng Features Annealed Independent Rules. Annals of Statstcs, January 3. [6] Van der Maaten (009) Dmensonalty Reducton: A Comparatve Revew. Journal of Machne, 1-. [7] Demers, D. and Cottrell, G. Non-Lnear Dmensonalty Reducton. Dept. of Computer Scence & Engr, September, 67. [8] ZhongBao, L., Guangzhou, P. and WengJuan, Z. (013) The Manfold Dscrmnate Analyss. Journal of Electroncs & Informaton Technology 9th. [9] DeWeng, H., JanZhou, X., JunSong, Y. and Tan, Z. (007) The Analyss and Applcaton of the Nonlnear Manfold Learnng. Progress n Natural Scence 8th. [10] Wood, F. (009) Prncpal Component Analyss. December 68. [11] Ma, R., Song, Y.X. and Wang, J.X. (008) Mult-Manfold Learnng Usng Locally Lnear Embeddng Nonlnear Dmensonalty Reducton. Journal of Tsnghua Unversty, [1] Rowesst, S. (000) Nonlnear Dmensonalty Reducton by Locally Lnear Embeddng. Scence, December. 13