Greedy algorithms of feature selection for multiclass image classification

Transcription

1 Greedy agorthms of feature seecton for mutcass mage cassfcaton E F Goncharova and A V Gade,2 Samara Natona Research Unversty, Moskovskoe Shosse 34, Samara, Russa, Image Processng Systems Insttute - Branch of the Federa Scentfc Research Centre Crystaography and Photoncs of Russan Academy of Scences, Moodogvardeyskaya str. 5, Samara, Russa, Abstract. o mprove the performance of remote sensng mages mutcass cassfcaton we propose two greedy agorthms of feature seecton. he dscrmnant anayss crteron and regresson coeffcents are used as the measure of feature subset effectveness n the frst and second methods respectvey. he man beneft of the but agorthms s that they estmate not the ndvdua crteron for each feature, but the genera effectveness of the feature subset. As there s a bg mtaton on the number of rea remote sensng mages, avaabe for the anayss, we appy the Markov random mode to enarge the mage dataset. As the pattern for mage modeng, a random mage beongng to one of the 7 casses from the UC Merced Land-Use dataset has been used. Features have been extracted wth hep of MaZda software. As the resut, the argest fracton of correcty cassfed mages accounts for 95%. Dmenson of the nta feature space consstng of 28 features has been reduced to 5 features, usng the greedy strategy of removng a feature, based on the near regresson mode.. Introducton Mutcass or mutnoma cassfcaton s a sgnfcant and compcated step, whch can be apped n sovng varous computer vson tasks. Large number of technques has been deveopedto perform the task of mutnoma mage cassfcaton. Some of them appy neura networks, whe the others tend to adapt the cassca methods of machne earnng to mprove the quaty of the cassfcaton resuts. In ths paper we present two greedy agorthms of feature seecton to mprove the performance of mutcass mage cassfcaton. An mage tsef can be descrbed by varous numerca characterstcs. For exampe, the MaZda software for texture anayss [] estmates amost 300 hstogram and texture features, moreover, t ncudes procedures for ther reducton and cassfcaton. It shoud be notced that not a the extracted features have smar nfuence on mage dstngushng. Redundancy features can affect the performance of cassfcaton bady and requre addtona computatona cost. he feature seecton methods have been wdey deveoped n recent years. Some researchers propose feature seecton methods based on custerng process. In paper [2] the agorthm s but n the foowng way: frsty, obects are custered, than the features whch provde the bggest dstance between custers centrods areappended to the subset of the most nformatve features. In [3] authors present the nove approach of dmensonaty reducton for hyperspectra mage cassfcaton. o reduce the numbers of varabes they use nter band bock correaton coeffcent technque and R IV Internatona Conference on "Informaton echnoogy and Nanotechnoogy" (IN-208)

2 decomposton. he support vector machnes agorthm has beenapped to fuf the cassfcaton task. Cassfcaton accuracy for mages from dfferent databases s between 83 and 99%. In ths work we use MaZda software to extract more than 200 texture features per mage. he most nformatve features are seected wth the hep of two greedy strateges, based on the dscrmnant anayss and near regresson mode, respectvey. he proposed agorthms enabe us to seect descrptors whch have the strongest effect on mutnoma mage cassfcaton. As there s a huge mtaton on the number of mages, avaabe for the anayss, we aso consder the agorthm of mage modeng based on the appyng of Markov random feds [4]. he experments are carred out on mages beongng to 7 and use casses, 00 for each cass, from the UC Merced Land-Use dataset, whch provdes aera optca mages. o measure the sgnfcance of feature subset we estmate the cassfcaton error, usng k-nearest-neghbor scheme. o estmate the effectveness of mage synthess, we compare the descrpton of the generated and source mage n the best feature subset, usng the Eucdean dstance between two feature vectors. 2. Feature extracton An mage s characterzed by ts ntensty matrx ( M I N), where M N s an mage sze. Rmn (, ) + Gmn (, ) + Bmn (, ) I( mn, ) =, m=, M, n=, N, () 3 RGBs,, an ntensty of red, green, and bue component of the mage resouton ce havng coordnates ( mn, ) respectvey. I( mn, ) ranges n vaue from 0 to I, where I s a maxmum grey eve. o extract the features we compute numerca descrptors of an mage, whch, eventuay, are gong to be used to perform the feature seecton procedure and further cassfcaton. he MaZda software s apped to form the set of features, descrbng nput mages []. he hstogram s cacuated va the ntensty of each mage pxe, cacuated by (), regardess to the spata correaton among the pxes. he foowng descrptors are computed: mean ntensty, varance, skewness, kurtoss, and percentes. he next type of features ncudes the textura characterstcs, cacuated wth the gray-eve spata dependence matrx. It s but accordng to the foowng rue: Pdd (, ) = {( mn, ) {,2,..., M} {,2,..., N} I( mn, ) = I, ( m+ d ) } 2, n+ d2 =,, = 0, L. hus, the foowng features are cacuated for fve dfferent dstances n four drectons: anguar second moment, contrast, entropy, and correaton. Features from the other group are cacuated based on the autocorreaton functon, whch descrbes the dependency between mage pxes. he cacuated features, as we as the prevous ones, are estmated for fve dfferent dstances n four drectons. 3. Methods of feature seecton 3.. Formuatn of feature seecton task he man dea of feature seecton process s to mprove the cassfcaton performance. hus, et Ω be a set of obects for recognton. he set Ω s dvded nto L non-overappng casses. Φ x, whch dentfes the o fuf the cassfcaton task we shoud create the mappng functon ( ) feature vector x, xx ΦΦ ( number of features), wth ts cass. Φ ( x) shoud be as smar to the dea mappng functon Φ ( x) as possbe. Φ( x) s the dea mappng functon, whch s aware of the nformaton about the rea obect s cass.cassfcaton s consdered an nstance of supervsed earnng, that s why Φ ( x) s created on the bass of a tranng set of data U Ω, contanng obect wth the known cass abes. IV Internatona Conference on "Informaton echnoogy and Nanotechnoogy" (IN-208) 39

3 he am of feature seecton step s to extract the subset of the most nformatve features, whch provdes the east cassfcaton error. o cassfy the feature vectors we appyk-nearest-neghbor scheme. Accordng to ths method, an obect s cassfed by a maorty vote of ts neghbors. he cassfer assgns the cass of the x vector to the cass of ts k-nearest neghbors. he dstance between two feature vectors s cacuated as the Eucdean dstance (2): (, ) ( ) 2 ρ xy = x y, x R?, y R, (2) where s a number of features. he nearest neghbor error rate s assessed by the formua (3). { x U Φ( x) Φ ( x) } ε =, U = We shoud notce that U, whch s a test set, shoud be ndependent of the tranng set,.e. U U=. In order to avod overfttng of cassfcaton mode and get more accurate resuts, the eave-one-out cross-vadaton technque s apped. Normazaton of data s a cruca step of cassfcaton process. As dfferent features can be measured n vared scaes they affect the cassfcaton performance dfferenty. o avod ths probem, a the features n dataset shoud be standardzed. herefore, the feature vectors get zero mean and unt varance. o acheve ths goa we shoud estmate the expected vaue x( ) and varance σ x ( ) for each feature. x( ) = x( ), x( ), Ω x Ω 2 σ ( ) = ( ( ) ( )), x x x σx ( ). Ω x Ω hus, each feature can be standardzed by appyng formua (4). x ( ) x ( ) x Ω x ( ) =, =,. (4) σ 3.2. Greedy addng agorthm based on the dscrmnant anayss When we have severa casses, feature seecton ams on choosng the features whch provde the strongest cass separabty. In dscrmnant anayss theory the crteron of separabty s evauated usng wthn-cass, between-cass, and mxture scatter matrces. Let x be a random vector, beongng to the feature space. herefore, to measure the mportance of the current feature space we shoud evauate the degree of soaton of vectors, beongng to dfferent casses [5]. he feature seecton method based on the dscrmnant anayss crteron was proposed n paper [6]. here the separabty of two casses was assessed wth hep of the dscrmnant crteron[7]. In ths work we generaze that technque to the case of severa casses. A wthn-cass scatter matrx (5) shows the scatter of ponts around ther respectve cass mean vectors (6), and s cacuated as foows: x = x, x. (5) U Ω x U Ω x ( ) R = ( x x)( x x), R. (6) U Ω x U Ω (3) IV Internatona Conference on "Informaton echnoogy and Nanotechnoogy" (IN-208) 40

4 Pror probabty of cass Ω s expressed by P( Ω ) = U Ω U covarance matrx of a sampes among a the casses, t s defned by: Rmx = ( x xmx )( x xmx ), Rmx, U x U L where = ( ) x xp Ω, x s a mean vector of mxture dstrbuton. mx mx = 0 hus, the dscrmnant crteron s formuated as J ( ) = L = 0 P tr R ( Ω ) tr R.. he mxture scatter matrx s the Crteron J ( ) tres to assess the nfuence of feature set on the wthn-cass compactness and nter-cass separabty. o seect the most nformatve features we propose greedy addng strategy. On the frst step of the agorthm current set of features s empty ( 0 ) =. On the step, we observe a the sets, formed as foows ( ) = ( ) { }, and cacuate the crteron J J( ), whch provde the maxmum vaue of crteron J : =. We choose the feature subset ( ) = ( ) arg max J, ( ) arg max J( ( ) { = }). [ ; ] Z\ ( ) [ ; ] Z\ ( ) hen the above steps are repeated unt we get the requred number of features Greedy agorthm of feature removng based on the regresson mode he second agorthm deveops the method, examnng n paper [6]. he regresson anayss studes the reatonshp between the output (dependent) varabe and one, or more, ndependent descrptors. For the bnary cassfcaton the number of cass can be consdered as the dependent varabe, whch s nfuenced by feature vector. In the case of mutnoma cassfcaton we cannot use the number of cass as an output, thus we present the functon Ψ ( x) : Ξ [ 0;] Z, whch determnes whether the feature beongs to the cass or not. he functon s defned by:, ( ), y x = Ψ ( x) = 0, y( x). hereby, Ψ ( x) s a dependent varabe, whch s affected by the feature vector x Ξ(. ) o assess the degree of feature vector nfuence we shoud bud L near regresson equatons: Ψ = Xθ + ε, = 0, L, where Ψ = ( Ψ Ψ2 Ψ n ) the output vector; X obect-feature matrx; ( 0 ) = regresson weghts; ε ( ε ε2 ε n ) θ θ θ θ = error vector. he unknown parameters are estmated by appyng the method of east squares: Ψ Xθ Ψ Xθ ( ) ( ) mn. herefore, L vectors θ whch characterze the coeffcents n near regresson are found for each ˆ θ ˆ θ ˆ θ ˆ θ s expressed by: of L casses. Vector = ( 2 ) θ IV Internatona Conference on "Informaton echnoogy and Nanotechnoogy" (IN-208) 4

5 L ˆ 2 ( θ ), = 0 θ = =,. (7) he measure of the nfuence of each feature s evauated accordng to the vector ˆ θ eement. o seect the most nformatve features we propose greedy removng strategy. he nta feature subset ncudes a the features ( 0 ) =. han we sequentay remove the worst feature from the current subset and rebud the near regresson as foows: on the step of the agorthm we create L near regresson modes Ψ ( ) = X ( ) θ( ), then vector ˆ θ ( s cacuated for the current feature subset ) ( ). ˆ θ k s removed from the current subset: he feature wth the mnma vaue of ( ) ( ) ˆ ( ) = ( ) \ arg mn θ ( ) ( k + ). k [ ; ] Z ( ) he steps of the agorthm are terated unt a requred number of features s obtaned. 4. Image modeng 4.. Markov random feds ImagemodengsperformedwthhepofMarkovrandomfeds. Let F = { F S} be a mutvarate random varabe, whch s defned on the dscrete set of ndex S = {, 2,..., N}. F s a random varabe that takes vaues { F = f, F2 = f2,..., Fn = fn}. he probabty of random varabe F takng the vaue of f s denoted as P( f ). hus, F s a random fed. Confguraton f = ( f, f2,..., f n ) s a specfc reazaton of random varabe F. Let Ν= { Ν S} be a neghborhood system, where Ν set of eements negh bourng. hus, the nodes that nfuence the oca characterstcs of the -node are ncuded n ts neghbourhood system. Markov random feds satsfy the foowng formua (8): fpf ( = f F = f, ) = PF ( = f F = f, Ν ). (8) Hence, Markov random feds mpy condtona ndependence [8]. Accordng to (8) F ony depends on the nodes ncuded n ts neghbourhood Ν. hus, f nodes n the neghbourhood are known than the vaues of F for and Ν do not affect F [9] Image modeng Suppose that the set of ndces S defnes the set of ponts on the 2D pane. he dscrete mage s a reazaton of 2D random varabe F, defned n the ponts S. Foowng by the condtona ndependence of F, wecanassumethatthentenstyvaueofeachmagepxe can be predcted on the bass of severa nodes, ncuded n ts neghbourhood. hus, we can present the foowng strategy of mage modeng usng the Markov random feds. On the each step k of the agorthm the neghbourhood system Ν s created for pxe of the mage G () ( k ). han ths neghbourhood system s compared wth the neghbourhood of the correspond pxe, beongng to the nput mage Gn (). Gn () s a sampe rea mage for synthess.he pxe s vaues are set as foows: G( k + ) () = Gn arg mn ( ρ ( Ν ( G ), ( )) * n Ν G( k ) ) * S. he dstance ρ ( xy, ) s defned by formua (2). he nta mage G(0) s approxmated by the whte nose. IV Internatona Conference on "Informaton echnoogy and Nanotechnoogy" (IN-208) 42

6 In ths paper we propose to use causa 5-neghbourhood system. hs negh bourhood pattern s shown n fgure. he pecuarty of ths type of negh bourhood s that t contans ony those nodes Ν G ncudes aready assgned pxes. that precede the current output pxe. hat means that ( ( k) ) Fgure. he nstance of causa 5-neghbourhood system. he qurrenty processng pxes marked bybacksquare. 5. Expermenta resuts 5.. Experments of feature seecton he experments were carred out on the mages from the UC Merced Land-Use dataset, whch conssts of the aera optca mages, beongng to dfferent casses (agrcutura fed, forest, beach, etc.), 00 for each cass. Each mage measures pxes. In ths work we anayzed mages, beongng to 7 casses (agrcutura fed, forest, budngs, beach, gof course, chaparra, and freeway). fgure 2 ustrates sampe mages beongng to the mentoned above casses. o get the correct cassfcaton resuts the Leave-one-out cross-vadaton technque was apped. he tota number of features, extracted wth the MaZda accounts for 28. a) b) c) d) e) f) g) Fgure 2. Sampe mages from UC Merced Land Use dataset: fed (a); forest (b); budngs (c); beach (d); gof course (e), chaparra (f), freeway (g). he resuts obtaned wth the dscrmnant and regresson anayss methods are shown n tabe. he most nformatve groups of features, seected wth the two proposed strateges, aong wth the cassfcaton error (3), obtaned on these groups, are presented n tabes 2 and 3. Havng anayzed the resuts, we can concude that the greedy removng agorthm, based on the near regresson mode, performed best on ths mutnoma cassfcaton task. he owest cassfcaton error rate of 0.05 was acheved n feature space, consstng of the 5 features from the 28 nta. IV Internatona Conference on "Informaton echnoogy and Nanotechnoogy" (IN-208) 43

7 abe. he features seected wth the greedy agorthms, based on dscrmnant and regresson anayss respectvey, n descendng order of prorty. Dscrmnant anayss Regresson anayss Feature number Feature name Feature number Feature name 37 SSumVarnc 96 S202DfEntrp 30 S0DfEntrp 74 S02DfEntrp 24 S0InvDfMom 85 S22DfEntrp 40 SDfVarnc 07 S30DfEntrp 79 S22InvDfMom 7 S44Entropy 34 SSumOfSqs 25 S55Entropy 32 SContrast 27 S55DfEntrp abe 2. Groups of the most nformatve features, seected wth the dscrmnant anayss. Features ε 3 37, 30, , 30, 24, , 30, 24, 40, , 30, 24, 40, 38, , 30, 24, 40, 38, 2, 55, 3, 42, 209, 44,, 9, 69, 80, 6, 8, , 30, 24, 40, 38, 2, 55, 3, 42, 209, 44,, 9, 69, 80, 6, 8, 85, abe 3. Groups of the most nformatve features, seected wth the regresson anayss. Features ε 3 96, 74, , 74, 85, , 74, 85, 07, , 74, 85, 07, 40, , 74, 85, 07, 40, 63, 5, 29, 73, 8, 0, 54, 66, 99, , 74, 85, 07, 40, 63, 5, 29, 73, 8, 0, 54, 66, 99, 43, he best group ncudes varous textura features, extracted for 4 dmensons: 2, 3, 4 and 5. he greedy addng agorthm maxmzed the dscrmnant anayss crteron provded worse resuts. he owest cassfcaton error rate of 0.32 was acheved on the set, consstng of 47 features. We shoud notce that the fracture of the mages that were cassfed correcty n the whoe space of 28 features accounts for 63%. hat means that both anayzed technques have succeeded n dmenson reducton and mprovng cassfcaton performance Experments of mage modeng o carry out the experments of mage synthess we have presented the nta mages the UC Merced Land-Use dataset n the greyscae. he resuts of modeng are shown n fgure 3.o check the quaty of syntheszed mages we performed the comparson of the feature vectors for the nput sampe and the obtaned mage. he vectors n cude 5 best features, seected by the greedy removng agorthm. he,, ξ xy x R, y R sexpressedby measure of equaty ( ) ξ 2 2 k k. k = ( xy, ) = ( x y) IV Internatona Conference on "Informaton echnoogy and Nanotechnoogy" (IN-208) 44

8 a) b) c) d) e) f) g) abe 4 presents the vaue of ξ ( xy, ) Fgure 3. Sampes of syntheszed and rea mages (the frst s syntheszed, and the second s rea): fed (a); forest (b); budngs (c); beach (d); gof course (e), chaparra (f), freeway (g). for the mages syntheszed for 7 casses. Contnuaton of tabe 4 abe 4. Measure of equaty for the syntheszed mages. ξ xy, Cass ( ) Beach 0.5 Chaparra 0.08 Fed 0.4 Forest 0.09 Freeway 0.09 Gof course 0.09 Budngs 0.3 he resuts shown n tabe 4 prove that the proposed method performs successfuy for the mages wth sma scae structure. For exampe, syntheszed mages beongng to the casses: chaparra and fed, turned to be qute smar to the rea mages. However the quaty of syntheszed mages contanng arge scae structure s ower. Its modeng demands arge neghborhoods whch eads to the ncreasng computatona cost. o sove ths probem, method, based on the mutresouton mage IV Internatona Conference on "Informaton echnoogy and Nanotechnoogy" (IN-208) 45

9 pyramds, s proposed n paper [4]. In that methodcomputaton s saved because the arge scae structures are presented more compacty by a few pxes n a certan ower resouton pyramd eve. 6. Concuson hus, for the task of the remote sensng mages cassfcaton the subset of nformatve features was extracted. We proposed two greedy strateges for nformatve feature seecton. he feature vector, seected wth the greedy removng agorthm, based on budng the regresson mode, produced the best cassfcaton performance (usng the nearest-neghbor cassfcaton method) on the mages from the UC Merced Land Use dataset. he mnma cassfcaton error rate made up In comparson to that, the greedy addng agorthm maxmzed the dscrmnant anayss crteron provded worse resuts. he owest cassfcaton error rate of 0.32 was acheved on the set, consstng of 47 features. We shoud notce that the fracture of the mages that were cassfed correcty n the whoe space of 28 features accounted for 63%. Overa, appyng the feature seecton methods eads to mprovng the mutnoma mage cassfcaton performance and dmenson reducton. Usng ony 5 (of 28 nta) descrptors aows to cassfy 95% of mages correcty o ncrease the number of mages, avaabe for anayss, we apped the agorthm of mage modeng on the bass of Markov random feds. he expermenta resuts showed that ths technque can be apped for synthess mages wth the ow scae structure. o generate sampes, contanng arge scae structure, the proposed agorthm shoud be adopted. One of the possbe varants s to appy mutresouton mage pyramds. 7. References [] Strzeeck M A, Szczypnsk P, Materka A and epaczko A 203 A software too for automatc cassfcaton and segmentaton of 2D/3D medca mages Nucear Instruments and Methods n Physcs Research [2] Lu C, Wanga W, Zhao, Shen X and onan M 207 A new feature seecton method based on a vadty ndex of feature subset Pattern Recognton. Leters [3] Reshma R, Sowmya V and Soman P 206 Dmensonaty Reducton Usng Band Seecton echnque for erne Based Hyperspectra Image Cassfcaton Proc. Computer Scence [4] We L Y and Levoy M 2000 Fast texture synthess usng tree-structured vector quantzaton Proc. of the 27th annua conf. on Computer graphcs and nteractve technques [5] Gade A V 205 A method for adustng drected texture features n bomedca mage anayssprobems Computer Optcs 39(2) DOI: / [6] Goncharova E and Gade A 207 Feature Seecton Methods for Remote Sensng Images Cassfcaton 3rd Int. conf. Informaton echnoogy and Nanotechnoogy [7] utkova V V and Gade A V 205 Study of nformatve feature seecton approaches for the texture mage recognton probem usng Laws masks Computer Optcs 39(5) DOI: / [8] Wnker G 202 Image Anayss, Random Feds and Dynamc Monte Caro Methods (Sprnger- Verag) p 387 [9] L S Z 2009 Markov random fed modeng n mage anayss(sprnger-verag) p 356 Acknowedgments hs work was supported by the Federa Agency for Scentfc Organzatons under agreement No. 007-GZ/Ch3363/26. IV Internatona Conference on "Informaton echnoogy and Nanotechnoogy" (IN-208) 46