Performance Analysis of Hybrid (supervised and unsupervised) method for multiclass data set

Transcription

1 IOSR Journal of Computer Engneerng (IOSR-JCE) e-issn: ,p-ISSN: , Volume 16, Issue 4, Ver. III (Jul Aug. 2014), PP Performane Analyss of Hybrd (supervsed and unsupervsed) method for multlass data set Rahul R. Chakre 1, Dr. Radhakrshna Nak 2 1 (PG Student, CSE, MI, Aurangabad, Maharashtra, Inda) 2 (Prof. and Head, CSE, MI, Aurangabad, Maharashtra, Inda) Abstrat: Due to the nreasng demand for multvarate data analyss from the varous applaton the dmensonalty reduton beomes an mportant task to represent the data n low dmensonal spae for the robust data representaton. In ths paper, multvarate data analyzed by usng a new approah SVM and ICA to enhane the lassfaton auray n a way that data an be present n more ondensed form. radtonal methods are lassfed nto two types namely standalone and hybrd method. Standalone method uses ether supervsed or unsupervsed approah, whereas hybrd method uses both approahes. hs paper onssts of SVM (support vetor mahne) as supervsed and ICA (Independent omponent analyss) as a unsupervsed approah for the mprovement of the lassfaton on the bass of dmensonalty reduton. SVM uses SRM (strutural rsk mnmzaton) prnple whh s very effetve over ERM (empral rsk mnmzaton) whh mnmzes an upper bound on the expeted rsk, as opposed to ERM that mnmzes the error on the tranng data, whereas ICA uses maxmum ndependene maxmzaton to mprove performane. he perpendular or rght angel projeton s used to avod the redundany and to mprove the dmensonalty reduton. At last step a lassfaton algorthm s used to lassfy the data samples and lassfaton auray s measured. Experments are performed for varous two lasses as well as multlass dataset and performane of hybrd, standalone approahes are ompared. Keywords: Dmensonalty Reduton, Hybrd Methods, Supervsed Learnng, Unsupervsed Learnng, Support Vetor Mahne (SVM), Independent Component Analyss (ICA). I. Introduton In the reent years data s nreased not only n terms of number of samples but also n terms of number of dmensons n many applatons suh as gene expresson data analyss, where the number of dmensons n the raw data ranges from hundreds to tens of thousands. From the theoretal pont vew more dmensons gves more understandng. hs paper assumes the ndependeny among the dmensons or features but n pratal nluson of more features or dmensons leads to undesrable performane.e. known as urse of dmensonalty. Multvarate data analyss brngs great dffulty n pattern reognton, mahne learnng and data mnng. herefore the dmensonalty reduton beomes very mportant task. Dmensonalty reduton s a well-known data mnng problem, whh s usually onsdered as an mprovement for pre-proessng tehnque for subsequent mahnng. On ompleton of ths task t wll gves very desrable advantages suh as redung the measurement, storage and transmsson, redung tranng and utlzaton tmes, defyng the urse of dmensonalty to mprove predton performane n terms of speed, auray and smplty, faltatng data vsualzaton and data understandng. A lot of dmensonalty reduton methods were proposed to deal wth these hallengng tasks. Due to the omputatonal omplexty of data and lassfaton n real-world applatons, t seems not an easy task to buld a general data reduton tehnque, so researhes on dmensonalty reduton have been onduted for last several deades and are stll extratng muh attenton from pattern reognton and data mnng soety. Dmensonalty reduton s nothng but the extratng the essental features n a way that hgh dmensonal data an be presented n low dmensonal spae. Supervsed and unsupervsed are the two popular methods for the dmensonalty reduton n frst type lass labels s known and n seond type lass labels are unknown. Hybrd dmensonalty reduton method [1] s the ombnaton of these two approahes whh uses both the rtera. In supervsed learnng LDA [2] and Support Vetor Mahne [3] are the tow famous dmensonalty reduton tehnques, whereas n unsupervsed learnng PCA [4] and Independent Component Analyss [5] and last there s ombnaton supervsed and unsupervsed approah whh s known as Support Vetor Mahne and Independent Component Analyss. II. Lterature Survey 2.1 Lnear Dsrmnant Analyss Basally ths s a supervsed method n whh lass labels are known or n other words some pror nformaton about the dataset s avalable. It s also alled as fsher s dsrmnant analyss. here are two approahes for the LDA [2] namely lass dependent and lass ndependent transformaton. hs paper fouses 93 Page

2 Performane Analyss of Hybrd (supervsed and unsupervsed) method for multlass data set on the multlass data set for the dmensonalty reduton, therefore frst approah.e. lass dependent transformaton s used. hs type of approah nvolves maxmzng the rato of between lass varane to wthn lass varane. he man objetve s to maxmze ths rato so that adequate lass separablty s obtaned. he lass-defnte type approah nvolves usng two optmzng rtera for transformng the data sets ndependently. he objetve of LDA [2] s to perform dmensonalty reduton whle keepng as muh of the lass dsrmnatory nformaton as possble. Mathematal Formulaton of LDA B (1) 1 S N ( )( ) Where, S represents the between lass satter matrx. X B X Inlude many th lass data. s the mean of data n X X s the mean of the entre data X. Denotes the number of lass n X. S (x )(x ) w 1 Where SW s the wthn satter matrx. Based on the S and B S W W W SBW (3) W SWW, the LDA a rteron s shown n above formula. W s the seleted matrx n whh maxmzes rato of between lass satter matrx and wthn lass satter matrx of the projets samples. For the above fsher agan prove that for the maxmzaton of ths effetveness of lass seperablty s maxmum. S W (2) should be non-sngular so the Lmtatons of LDA he major drawbak of ths method s t only fouses on the lass dsrmnatory funton t does not work for the feature dsrmnant funtons. Agan LDA have small sample sze problem t does not work properly when there s small samples or nstanes are gven, whle workng wth ths approah prmary step s to alulate the lass mean, agan LDA have some problems wth ommon mean problem, due to ths lassfaton auray gets redued. LDA s a parametr n the nature so t assumes the unmodel of Gaussan lkelhood therefore t does not work properly wth the non Gaussan dstrbuton. he robustness problem s also there n LDA.For avodng some these problems there are methods namely null spae LDA for small sze small problem [6], dsrmnatve ommon vetor for the lass mean problem, Orthogonal entrod method for robustness problem [7].he LDA [2] also extended for the klda for dealng wth non-lnearty, but all these method are nherted from the LDA so no one method avod the all the above mentoned problems. 2.2 Prnpal Component Analyss hs s very popular unsupervsed dmensonalty reduton method n whh lass labels are unknown to us or In other words no pror nformaton about the dataset s avalable to. It s also known for the the Karhunen-Loeve transform or Fator Analyss. he man objetve of PCA [3] s to redue the number of dmensons or features and transfer set of orrelated varables nto set of unorrelated varables n a way that the orgnal dataset s mapped nto lower dmensonal spae. PCA projets the data n least square sense n whh t aptures the bg varablty n the data and gnores the small varablty. Due to ths features whh are present n the varous lass wll be separated n a way that dmensonalty reduton takes plae effetvely, so t an plae that feature from hgh dmensonal spae to low dmensonal spae. he lassfaton auray s better n ths type of method Page

3 Performane Analyss of Hybrd (supervsed and unsupervsed) method for multlass data set Mathematal Formulaton for the PCA W arg max W S W Where x s the th S (x )(x ) 1 n dmensonal data among N multdmensonal dataset. When M s the dmenson m n, of the dmenson vetor s fulflls the m s W xr where W shows mappng of a good or optmal ovarane of the dmensons desrbed by the above total satter matrx. Based on the predetermned the W. he projeton s hosen n suh way that the determnant of total satter matrx wll maxmum Lmtatons for the PCA It works very undesrable for the nonlnear dataset. Due to ths lassfaton auray s drastally redued. But for that nonlnear omponent analyss s used.e. KPCA [8], t gves a sutable kernel for transformaton of low dmensonal spae nto hgh dmensonal but agan t nreases the omputatonal omplexty. PCA also suffers from the small nstane problem. Due to whh lassfaton auray gets drastally redued. PCA only fous on the feature or dmensons of the dataset t does not are about the lass labels. hs s also one of the major drawbaks whle dong the dmensonalty reduton. Regresson model [9] s also another approah for the avodng ths problem but t fals at some pont. 2.3 Support Vetor Mahne SVM s presely depend upon the theoretal model of learnng wth the surety of effent performane. Basally SVM use the struture rsk mnmzaton prnpal by the use of lnear funton whh s omng for the avalable datasets for the lassfaton purpose. he man am s to buld a good lassfer well for the unknown samples. he SVM s hghest level of development of tehnque at present tme lassfaton method whh s wdely used n the statal learnng envronment. he objetve of support vetor mahne s to obtan the most favorable hyper plane for lnearly separable datasets and extends for the nonlnear datasets by the transformaton of the orgnal dataset nto hgh dmensonal spae by the use of approprate kernel funton. SVM maxmzes the margn around the separatng hyper plane. he deson funton s fully spefed by subset of tranng samples, the support vetors.e. the data ponts whh are losest to the deson surfae whh are mostly dffult to lassfy. Due to the use of strutural rsk mnmzaton prnpal the SVM does not suffers from the small sample sze problem and ommon mean problem. It very effetvely redues the dmensons and gves better lassfaton performane. 2.4 Independent Component Analyss Independent omponent analyss s nothng but fndng out the underlyng fators or the dmensons from the multvarate statstal data. Basally t s famous blnd soure separaton but t an be use for the dmensonalty reduton. Independent Component Analyss [5] s dfferent from other method beause t looks for the dmensons or fators whh are statstal ndependent and non-gaussans. ICA s an unsupervsed method n whh pror knowledge about the lass labels s unknown. It s a more advantageous than the PCA [4] beause PCA seeks the projeton whh has maxmum varane whereas ICA [5] seeks the projeton whh has maxmum ndependene. hat s the reason n ths paper ICA s used for the maxmzaton of ndependene due to whh lassfaton auray gets nreased. For the mplementaton of the ICA there are two algorthms namely maxmum lkelhood soure separatons and Informx [10] but FastICA [11] algorthm due to ts omputatonal and oneptual smplty for the multvarate data analyss. In ICA, the ntal step s enterng and whtenng proess then there s FastICA [11] algorthm. 2.5 Conluson from Lterature Survey Dmensonalty reduton s well known problem n real world applaton. In above there are only standalone approahes are used and every ndvdual approah s sufferng from some mportant ssues due to that lassfaton auray gets redued. he exstng hybrd methods are also suffers from the above problems beause all the methods are nherted ether from the supervsed or unsupervsed approah some of them are asymmetr prnpal and dsrmnant analyss(apcda) [12], ICA augmented LDA [13], dsrmnant non negatve matrx fatorzaton (DNNF) [14].Due to the above problems a new approah whh s hybrd n the nature named SVM+ICA s used, t frstly utlzes supervsed rtera that strutural rsk mnmzaton and then t utlzes seond unsupervsed rtera ndependeny among the feature maxmzaton. So the steps for new proposed approah whh s as follows (4) (5) 95 Page

4 Performane Analyss of Hybrd (supervsed and unsupervsed) method for multlass data set III. Proposed Work 3.1 SVM and ICA As the name ndates t s the ombnaton of two dfferent approahes.e. SVM as supervsed and ICA as unsupervsed. It s a hybrd method whh s used for the dmensonalty reduton. It uses both rtera s for the multvarate data analyss.as explaned n the thrd pont SVM uses SRM prnpal so t gves the best lassfaton auray than other methods. For the mplementaton of SVM, LIBSVM software pakage [15] s used. In more detal, SVM redues the strutural rsk due to ths the projeton whh are omng from the SVM gves the superor generalzaton ablty to enhane the lassfaton auray among the multvarate data set. he man objetve of the SVM s to maxmze the margn whh gves a better data representaton whh results n the desrable lassfaton performane and another advantage s that t gves projetons whh are well known for the onstruton of ompetent subspae for the dmensonalty reduton. In ths hybrd method SVM s treated as supervsed part for dmensonalty reduton. SVM provdes optmum deson surfae wth the mnmum strutural rsk by the use quadrat onstraned optmzaton problem. he dual problem of multlass lassfaton s gven by arg mn {1/ 2 Q 1} Subjet to a 0, 0 b. (6) Where s nothng but the langrage s multpler for solvng the optmzaton problem. y are the data samples and a wll be the multlass ndex. b s the some relaxaton parameter to avod the empral rsk, but X as a weght vetor s used for the mappng n the lnear way. so the t s gven by n (7) X a y R he most desrable set of mappng vetors derves from the SRM prnple the startng proess X1,l.there should be the par wse orthogonally among the SVM and ICA, whh s also denoted by the X1, l X1 l,.he X1,l s alulated as a onstraned optmzaton ssue whh s as follows 2 z arg mn y z Subjet to X, 0 l,1 z (8) Where Z represents the projeted data onto the subspae orthogonal to X 1,l, and parallel to the deson hyper plane(s). Due to the orthogonalty between X 1,l and any omponents n the deson hyperplane, the strutural rsk mnmzaton and ndependene maxmzaton are solated and performed one by one holdng ndependene between any par of x s and x s where {1... l} and j {l 1...} he seond part of our hybrd method s ICA whh plays the mportant role n ths paper. ICA s manly deal wth the data whh are statally ndependent and non Gaussan. ICA provdes the projeton whh gves us the maxmum ndependeny among the features or dmensons results n a better data representaton whh have a key role n the mprovement of the lassfaton auray. In ths system ICA s used as an unsupervsed omponent. he FastICA algorthm nvolves two sequental proesses, the one unt estmaton and deorrealaton among the weght vetors. he one unt estmates the weght vetors as follows, x E{zg(x,z) E{g'(x,z)} (9) Where x s the temporal approxmaton of the ndependent omponent wth j {l 1...m}. g s the dervatve of the non-quadrat funton ntrodued n and g(u) tanh(au),g s the dervatve of g, g '(u) 2 seh (u). he purpose of the deorrelaton proess s to keep dfferent weght vetors from onvergng to the same maxmum. he deflaton sheme based on symmetr deor relaton helps remove dependeny among x s as follows j 1/2 1 l, m 1 l, m (X 1 l, m,x 1 l, m) (10) X X 96 Page

5 Performane Analyss of Hybrd (supervsed and unsupervsed) method for multlass data set Where X l 1, represents deorrelated mappngs based on x 1, [x l m 1 l, x m] from ndependene maxmzaton. he thrd part of proposed hybrd system s nothng but the lassfaton. For the lassfaton purpose nearest neghbor algorthm s used. he workng prnpal of ths algorthm s t stores the all avalable ases and reates new ases based on the dstane funton. hs algorthm s also used n many areas of pattern lassfaton. In the proposed system t plays very mportant role to avod the dmensonalty dsaster. Fgure. 1. Blok dagram for the proposed hybrd method Fg. 1 gves the omplete nformaton about the proposed hybrd dmensonalty reduton method. LIBSVM software pakage s used for the mplementaton of SVM whh s our frst supervsed omponent, then the seond one s the projeton both lnear and nonlnear projetons are used aused of lnear and nonlnear SVM, thrd one s unsupervsed omponent whh s mplemented by usng FastICA algorthm, and the last one s one of the lassfaton algorthm s used for the lassfaton purpose whh s a performane metr. Fg. 1 shows that multvarate data set named Y s gven as nput to the LIBSVM whh has the dmenson of n, after workng on that t s redued to the m dmenson n a suh way that m n. hen by the use of projeton matrx there s omputaton of a olumn vetor from the hybrd proess SVM+ICA.he supervsed omponent wll generate the X1,l vetor after that X1,l vetor wll be generated by the unsupervsed omponent then the fnal weght vetor wll be gven to the lassfaton algorthm for the alulaton of the lassfaton auray. IV. Expermental Results And Analyss Natural dataset named arda arrhythma whh a multlass n the nature s used for the dmensonalty reduton purpose. Basally ths hybrd method s analyzed over the auray of lassfaton of arrhythma dataset whh ontan lst of 452 patents or samples whh are to be lassfy among the 16 types of aner due to some ambguty n the dataset suh as nonssteny n the samples, mssng some samples or elements only 13 type of aner are lassfed, for the mssng elements some random varables are used. hs dataset provdes some dfferent haratersts for samples per lass and dmensonalty. he number of feature or dmensons s redued up to 95% due to the use of proposed hybrd method. he nearest neghbor algorthm s used for the lassfaton auray whh s our performane metr. Some other metrs are also used for ths whh are senstvty and spefty. Above all the parameters are alulated by the use of onfuson matrx. he expermental result shows that the proposed hybrd method performs extremely well wth the lassfaton auray up to whh outperforms the other methods suh as LDA PCA ICA Page

6 Performane Analyss of Hybrd (supervsed and unsupervsed) method for multlass data set able 1: omparson wth the standalone approahes Approah Classfaton auray No of projetons Senstvty (%) Spefty (%) (%) LDA PCA ICA SVM+ICA Fgure 2: omparson wth respet to redued dmensonalty he Fg. 2 represents the omparson of other methods wth respet to SVM and ICA. As shown n the hart the SVM and ICA work very well for dmensonalty reduton. V. Future Sope As talkng about the future sope, there are some mportant ssues whh are as mentoned, the frst one s the onstraned optmzaton tehnques for SVM the seond on s the onstruton of the subspae and the last one s the ntegraton of ICA nto the sngle formula for enhanement of the workng speed. In another way sngle formulaton does not gves the lear understandng about the dmensonalty reduton but t s apable of avodng the possble error ame durng the stepwse evoluton of the proess. Another major ssue wll be dealng wth the non-lnearty. here s a lear soluton for ths s use kernel beause t s very user-frendly method but agan there s problem that whh type of kernel s to use for dealng wth nonlnearty for transformaton of low dmensonal data n hgh dmensonal spae t s very hard to determne whh kernel has to be trusted for the free dmensons. VI. Conluson New hybrd method SVM and ICA whh performs effetvely for hgh dmensonal data analyss, as t uses both supervsed and unsupervsed learnng method and gves better lassfaton results ompare to the tradtonal method whh are LDA, PCA and ICA. he tradtonal methods uses only one rteron ether supervsed or unsupervsed. he proposed algorthm provdes projetons, suh that SVM whh s used as supervsed omponent for mnmzaton of the strutural rsk and ICA whh s used as unsupervsed omponent for maxmzaton of ndependeny among the features.he ombnaton of both approahes gves the advantage of both methods, so t performs wth the hgh degree of auray. hs approah s used for the lassfaton of 98 Page

7 Performane Analyss of Hybrd (supervsed and unsupervsed) method for multlass data set the multvarate data analyss.he expermental results and analyss shows dmensonalty s redued, for whh one aganst all strategy s used (One aganst one for two lass data set). Referenes [1] Sangwoo Moon and Harong Q, Hybrd Dmensonalty Reduton Method Based on Support Vetor Mahnes and Independent Component Analyss, IEEE ransatons on Neural networks and Learnng Systems,vol. 23, no. 5, may [2] A.M.Martnez and A.C.Kak, PCA versus LDA, IEEE rans.pattern Anal.Mah.Intell. vol.23, no. 2, pp , Feb [3] C. J. C. Burges. A tutoral on support vetor mahnes for pattern reognton. Data Mnng and Knowledge Dsovery, 2(2): , [4] L. Cao, K. Chua, W. Chong, H. Lee, and Q. Gu. A omparson of PCA, KPCA and ICA for dmensonalty reduton n support vetor mahne. Neroomp, 55: , [5] Hyvarnen, A., Karhunen, J., Oja, E.: Independent Component Analyss and Its Applaton John Wley & Sons,In.,2001 [6] L.-F. Chen, H.-Y. M. Lao, M.-. Ko, J.-C. Ln, and G.J. Yu, Anew LDA-based fae reognton system whh an solve the small sample sze problem, Pattern Reognt., vol. 33, no. 10, pp , [7] H. Park, M. Jeon, and J. B. Rosen, Lower dmensonal representaton of text data based on entrods and least squares, BI Numeral Math vol. 43, no. 2, pp , [8] B. Sholkopf, A. Smola, and K. Muller, Nonlnear omponent analyss as a Kernel Egenvalue problem, Neural Comput., vol. 10, no. 5, pp, , [9] H. Wold, Estmaton of prnpal omponents and related models by teratve least squares, n Multvarate Analyss. New York: Aadem, pp , [10] Jean-Fran os Cardoso, Infomax and maxmum lkelhood for soure separaton, IEEE,Letters On Sgnal Proessng Vol. 4, No. 4, pp , [11] Hurr, Gavert, Sarela, and Hyvarnen, A.,\he FastICA Pakage for MALAB," [12] X.Jang, Asymmetr Prnpal omponent and dsrmnant analyses for pattern lassfaton, IEEE rans.pattern Anal. Mah. Intell., vol. 31, no. 5, pp , May [13] K. Kwak and W. Pedryz, Fae reognton usng an enhaned ndependent omponent analyss approah, IEEE rans. Neural Network, vol. 18, no. 2, pp , Mar [14] S. Zaferou, A. efas, I. Buu, and I. Ptas, Explotng dsrmnant nformaton n nonnegatve matrx fatorzaton wth applaton to frontal fae verfaton, IEEE rans. Neural Netw. vol. 17, no. 3, May [15] C.-C. Chang and C.-J. Ln. LIBSVM: A lbrary for support vetor mahnes. Software avalable at jln/lbsvm, Page