Support Vector classifiers for Land Cover Classification

Transcription

1 Map Inda 2003 Image Processng & Interpretaton Support Vector classfers for Land Cover Classfcaton Mahesh Pal Paul M. Mather Lecturer, department of Cvl engneerng Prof., School of geography Natonal Insttute of Technology Unversty of Nottngham Kurukshetra Unversty Park Haryana (Inda) Nottngham, NG7 2RD, UK 1. Introducton Much research effort n the past ten years has been devoted to analyss of the performance of artfcal neural networks n mage classfcaton (Benedktsson et al., 1990; Heermann and Khazene, 1992). The preferred algorthm s feedforward mult-layer perceptron usng back-propagaton, due to ts ablty to handle any knd of numercal data, and to ts freedom from dstrbutonal assumptons. Although neural networks may generally be used to classfy data at least as accurately as statstcal classfcaton approaches a number of studes have reported that users of neural classfers have problems n settng the choce of varous parameters durng tranng (Wlknson, 1997). The choce of archtecture of the network, the sample sze for tranng, learnng algorthms, and number of teratons requred for tranng are some of these problems. A new classfcaton system based on statstcal learnng theory (Vapnk, 1995), called the support vector machne has recently been appled to the problem of remote sensng data classfcaton (Huang et al., 2002; Zhu and Blumberg, 2002; Gualter and Cromp, 1998). Ths technque s sad to be ndependent of the dmensonalty of feature space as the man dea behnd ths classfcaton technque s to separate the classes wth a surface that maxmse the margn between them, usng boundary pxels to create the decson surface. The data ponts that are closest to the hyperplane are termed "support vectors". The number of support vectors s thus small as they are ponts close to the class boundares (Vapnk, 1995). One maor advantage of support vector classfers s the use of quadratc programmng, whch provdes global mnma only. The absence of local mnma s a sgnfcant dfference from the neural network classfers. Map Inda Conference 2003 GISdevelopment.net, All rghts reserved.

2 Map Inda 2003 Image Processng & Interpretaton 2. Classfcaton Methods Three classfcaton algorthms used for ths study are the maxmum-lkelhood, mult-layer backpropagaton neural network and support vector classfer. A bref summary of these classfers s gven below. 2.1 Maxmum-Lkelhood Classfer The Maxmum lkelhood Classfer (MLC) s based on the assumpton that the members of each class are normally dstrbuted n feature space. MLC s a pxelbased method and can be defned as follows: a pxel wth an assocated observed feature vector X s assgned to class X c f c f g ( X ) > ( X ) g k for all k,, k = 1,, N For multvarate Gaussan dstrbutons g k ( X) s gven by: where and g k 1 2 t 1 g k ( X ) =ln(p( c ))- ln ( X M ) ( X M ) k 1 2 k k Mk and k are the sample mean vector and covarance matrx of class k, s the dscrmnatng functon. k Implementaton of the MLC nvolves the estmaton of class mean vectors and covarance matrces usng tranng patterns chosen from known examples of each partcular class. 2.2 Artfcal Neural Network Classfer A feed-forward artfcal neural network (ANN) s used n ths study. Ths s the most wdely used neural network model, and ts desgn conssts of one nput layer, at least one hdden layer, and one output layer. Each layer s made up of non-lnear processng unts called neurons, and the connectons between neurons n successve layers carry assocated weghts. Connectons are drected and allowed only n the forward drecton, e.g. from nput to hdden, or from hdden layer to a subsequent hdden or output layer. Non-lnear processng s performed by applyng an actvaton functon to the summed nputs to a unt. Backpropagaton s a gradent-descent algorthm that mnmses the error between the output of the tranng nput/output pars and the actual network outputs (Bshop, 1995). Therefore, a set of nput/output pars s repeatedly presented to the network Map Inda Conference 2003 GISdevelopment.net, All rghts reserved.

3 Map Inda 2003 Image Processng & Interpretaton and the error s propagated from the output back to the nput layer. The weghts on the backwards path through the network are updated accordng to an update rule and a learnng rate. ANNs are not solely specfed by the characterstcs of ther processng unts and the selected tranng or learnng rule. The network topology,.e. the number of hdden layers, the number of unts, and ther nterconnectons, also has an nfluence on classfer performance. In ths study we use the network archtecture and number of patterns used for tranng suggested by Kavzoglu (2001). 2.3 Support vector classfer In the two-class case, a support vector classfer attempts to locate a hyperplane that maxmses the dstance from the members of each class to the optmal hyperplane. The prncple of a support vector classfer s brefly descrbed next. Assume that the tranng data wth k number of samples s represented by {, y } n =1,,k,where x R s an n-dmensonal vector and y { 1, + 1} x, s the class label. These tranng patterns are sad to be lnearly separable f a vector w (whch determnng the orentaton of a dscrmnatng plane) and a scalar b (determne offset of the dscrmnatng plane from orgn) can be defned so that nequaltes (1) and (2) are satsfed. w + b +1 for all y = +1 (1) x w x + b 1 for all y = -1 (2) The am s to fnd a hyperplane whch dvdes the data so that that all the ponts wth the same label le on the same sde of the hyperplane. Ths amounts to fndng w and b so that y ( x + b ) 0 w (3) > If a hyperplane exsts that satsfes (3), the two classes s sad to be lnearly separable. In ths case, t s always possble to rescale w and b so that mn y 1 k ( w x + b ) 1 That s, the dstance from the closest pont to the hyperplane s can be wrtten as 1 w. Then (3) Map Inda Conference 2003 GISdevelopment.net, All rghts reserved.

4 Map Inda 2003 Image Processng & Interpretaton y ( x + b ) 1 w (4) The hyperplane for whch the dstance to the closest pont s maxmal s called the optmal separatng hyperplane (OSH) (Vapnk, 1995). As the dstance to the closest pont s 1 w, the OSH can be found by mnmsng w 2 under constrant (4). The mnmsaton procedure uses Lagrange multplers and Quadratc Programmng (QP) optmsaton methods. If λ, = 1,.,k are the non-negatve Lagrange multplers assocated wth constrant (4), the optmsaton problem becomes one of maxmsng (Osuna et.al. 1997): 1 L ( λ ) = λ λ λ y y ( x x ) (5) 2, under constrants If λa = ( λ,..., λ ) a 1 a k λ 0, = 1,..,k. s an optmal soluton of the maxmsaton problem (5) then the optmal separatng hyperplane can be expressed as: a w a = y λ x (6) The support vectors are the ponts for whch a λ > 0 when the equalty n (4) holds. If the data are not lnearly separable, a slack varable ξ, =1,,k can be ntroduced wth y ξ 0 (Cortes and Vapnk 1995) such that (4) can be wrtten as ( x + b ) 1 + ξ 0 w (7) and the soluton to fnd a generalsed OSH, also called a soft margn hyperplane, can be obtaned usng the condtons mn w, b, ξ 1,... ξ k y 1 2 w k 2 + C = 1 ξ ( x + b ) 1 + ξ 0 w (9) ξ 0 =1, k. (10) The frst term n (8) s same as n as n the lnearly separable case, and controls the learnng capacty, whle the second term controls the number of msclassfed ponts. The parameter C s chosen by the user. Larger values of C mply the assgnment of a hgher penalty to errors. (8) Map Inda Conference 2003 GISdevelopment.net, All rghts reserved.

5 Map Inda 2003 Image Processng & Interpretaton Where t s not possble to have a hyperplane defned by lnear equatons on the tranng data, the technques descrbed above for lnearly separable data can be extended to allow for non-lnear decson surfaces. A technque ntroduced by Boser et al. (1992) maps nput data nto a hgh dmensonal feature space through some nonlnear mappng. The transformaton to a hgher dmensonal space spreads the data out n a way that facltates the fndng of lnear hyperplanes. After replacng x by ts mappng n the feature space Φ ( x), equaton (5) can be wrtten as: L 1 λ (11) 2, ( ) = λ λ λ y y ( Φ ( x ) Φ ( x )) To reduce computatonal demands n feature space, t s convenent to ntroduce the concept of the kernel functon K (Crstann and Shawe-Taylor, 2000; Cortes and Vapnk 1995) such that: K ( x x ) Φ ( x ) Φ ( x ), = (12) Then, to solve equaton (11) only the kernel functon s computed n place of computng Φ ( x), whch could be computatonally expensve. A number of kernel functons are used for support vector classfer. Detals of some kernel functons and ther parameters used wth SVM classfers are dscussed by Vapnk (1995). SVM was ntally desgned for bnary (two-class) problems. When dealng wth several classes, an approprate mult-class method s needed. A number of methods are suggested n lterature to create mult-class classfers usng two-class methods (Hsu and Ln, 2002). In ths study, a "one aganst one" approach (Knerr et al., 1990) (In ths method, all possble two-class classfers are evaluated from the tranng set of n classes, each classfer beng traned on only two out of n classes. There would be a total of n (n-1)/2 classfers. Applyng each classfer to the vectors of the test data gves one vote to the wnnng class. The pxel s gven the label of the class wth most votes. To generate mult-class SVMs and a radal bass kernel functon (defned as e γ x y 2 ) was used. 3. Data The frst of the two study areas used n the work reported here are located near the town of Lttleport n eastern England. The second s a wetland area of the La Map Inda Conference 2003 GISdevelopment.net, All rghts reserved.

6 Map Inda 2003 Image Processng & Interpretaton Mancha regon of Span. For the Lttleport area, ETM+ data acqured on 19 th June 2000 s used. The classfcaton problem nvolves the dentfcaton of seven land cover types (wheat, potato, sugar beet, onon, peas, lettuce and beans) for the ETM+ data set. For the La Mancha study area, hyperspectral data acqured on 29 th June 2000 by the DAIS 7915 arborne magng spectrometer were avalable. Eght dfferent land cover types (wheat, water body, dry salt lake, hydrophytc vegetaton, vneyards, bare sol, pasture lands and bult up area) were specfed. The DAIS data show moderate to severe strpng problems n the optcal nfrared regon between bands 41 and 72. Intally, the frst 72 bands n the wavelength range 0.4 µm to2.5µmwere selected. All of these bands were examned vsually to determne the severty of strpng. Seven bands dsplayng very severe strpng problems (bands and 68-72) were removed from the data set. The strpng n the remanng bands was removed by automatcally enhancng the Fourer transform of each mage (Cannon et al., 1983; Srnvasan et al., 1988). The nput mage s frst dvded nto overlappng 128-by-128-pxel blocks. The Fourer transform of each block s calculated and the log-magntudes of each FFT block are then averaged. The averagng process removes all frequency doman quanttes except those whch are present n each block;.e., some sort of perodc nterference. The average power spectrum s then used as a flter to adust the FFT of the entre mage. When an nverse Fourer transform s performed, the result s an mage wth perodc nose elmnated or sgnfcantly reduced. 4. Result and dscussons Random samplng was used to collect the tranng and test pxels for both ETM+ and DAIS data set. Total selected pxels were dvded nto two parts, one for tranng and one for testng the classfers, so as to remove any possble bas resultng from the use of same set of pxels for both the testng and tranng phases. A total of 2700 tranng pxels and 2037 test pxels for ETM+ data and a total of 1600 tranng (200 pxels/class) and 3800 test pxels were used for DAIS data. Kappa values and overall classfcaton accuraces are calculated for each of the classfers used n ths study wth ETM+ data, whle overall accuracy s calculated for the DAIS hyperspectral data. The Z statstc s also used to test the sgnfcance of apparent dfferences between the three classfcaton algorthms, Map Inda Conference 2003 GISdevelopment.net, All rghts reserved.

7 Map Inda 2003 Image Processng & Interpretaton usng ETM+ data. For ths study, a standard back-propagaton neural classfer was used. All user-defned parameters are set as recommended by Kavzoglu (2001). Lke neural network classfers the performance of support vector classfer depends on a number of user-defned parameters whch may nfluence the fnal classfcaton accuracy. For ths study a radal bass kernel wth γ (kernel specfc parameter) value as two and C = 5000 s used for both data sets. The values of these parameters were chosen after a number of trals and the same parameters are used wth DAIS data set. Ths study also suggests that, n comparson wth the NN classfer, t s easer to fx the values of the user defned parameters for SVM. As mentoned earler, SVM nvolves n solvng a quadratc programmng problem wth lnear equalty and nequalty constrants whch has only a global optmum. In comparson the presence of local mnma s a sgnfcant problem n tranng the neural network classfers. Results obtaned usng ETM+ data suggests that support vector classfer perform well n comparson wth neural and statstcal classfer (Tables 1 and 2). Table 1. Classfcaton accuraces acheved wth dfferent classfers. Classfer used Accuracy (%) Kappa value Maxmum lkelhood Neural network Support vector Table 2. Calculated Z values for comparson between dfferent classfcaton systems wth ETM+ data. Shaded values ndcate sgnfcant mprovements n the performance of frst classfer at the 95% confdence level (Z crtcal value = 1.96). Classfers Z value SVM vs. Neural network 2.46 SVM vs. maxmum lkelhood 5.45 Map Inda Conference 2003 GISdevelopment.net, All rghts reserved.

8 Map Inda 2003 Image Processng & Interpretaton Further, the tranng tme taken by support vector classfer s 0.30 mnutes n comparson of 58 mnutes by the NN classfer on a dual processor sun machne. Results suggest that support vector classfer performance s statstcally sgnfcant n comparson wth NN and ML classfers. To study the behavour of support vector classfer wth DAIS Hyperspectral data a total of sxty fve features (bands) was used, a total of 65 features (spectral bands) were avalable, as seven features wth severe strpng were dscarded, as explaned above. The ntal number of features used was fve, and the experment was repeated wth 10, 15,, 65 features, gvng a total of 13 experments. Fgure 1 suggests that, n comparson to the other classfers, the performance of the support vector classfer s qute good wth small tranng data set rrespectve of the number of features used Accuracy (%) Maxmum lkelhood neural network Support vector Maxmum lkelhood neural network Support vector Number of bands Fgure 1. Classfcaton accuraces obtaned wth DAIS hyperspectral data usng dfferent classfcaton algorthms. The tranng data set sze s 200 pxels/class. Results obtaned from analyss of the hyperspectral data suggest that classfcaton accuracy usng SVM ncreases almost contnuously as a functon of the number of features, wth the sze of the tranng data set held constant, whereas the overall classfcaton accuraces produced by the ML, DT and NN classfers declne slghtly once the number of bands exceeds 50 or so. Thus, suggestng that the Map Inda Conference 2003 GISdevelopment.net, All rghts reserved.

9 Map Inda 2003 Image Processng & Interpretaton Hughes phenomenon (Hughes, 1968) of decreasng classfer performance as the dmensonalty of the feature space ncreases beyond a threshold s not supported by the experments usng the support vector classfers. 5. Conclusons The obectve of ths study was to assess the utlty of support vector classfers for land cover classfcaton usng mult- and hyper-spectral data sets n comparson wth most frequently used ML and NN classfers. The results presented above suggest several conclusons. Frst the support vector classfer outperforms ML and NN classfers n term of classfcaton accuracy wth both data sets. Several user-defned parameters affects the performance of the support vector classfer, but ths study suggests that t s easer to fnd approprate values for these parameters than t s for parameters defnng the NN classfer. The level of classfcaton accuracy acheved wth the support vector classfer s better than both ML and NN classfers when used wth small number of tranng data. Acknowledgement The DAIS data were collected and processed by DLR and were kndly made avalable by Prof. J. Gumuzzo of the Autonomous Unversty of Madrd. Computng facltes were provded by the School of Geography, Unversty of Nottngham. References Benedktsson, J. A., Swan, P. H., and Erase, O. K., 1990, Neural network approaches versus statstcal methods n classfcaton of multsource remote sensng data. IEEE Transactons on Geoscence and Remote Sensng, 28, Bshop, C. M. (1995) Neural Networks for Pattern Recognton. Oxford: Clarendon Press Boser, B., Guyon, I., and Vapnk, V. N., 1992, A tranng algorthm for optmal margn classfers. Proceedngs of 5 th Annual Workshop on Computer Learnng Theory, Pttsburgh, PA: ACM, Map Inda Conference 2003 GISdevelopment.net, All rghts reserved.

10 Map Inda 2003 Image Processng & Interpretaton Cannon, M., Lehar, A., and Preston, F., 1983, Background pattern removal by power spectral flterng. Appled Optcs, 22(6), Chang, C., and Ln, C., 2001, LIBSVM: A Lbrary for Support Vector Machnes. Computer Scence and Informaton Engneerng, Natonal Tawan Unversty, Tawan, Cortes, C., and Vapnk, V. N., 1995, Support vector networks. Machne Learnng, 20, Crstann, N., and Shawe-Taylor, J., 2000, An Introducton to Support Vector Machnes. London, Cambrdge Unversty Press. Gualter, J. A. and Cromp, R. F., 1998, Support vector machnes for hyperspectral remote sensng classfcaton. Proceedngs of the 27 th AIPR Workshop: Advances n Computer Asssted Recognton, Washngton, DC, October 27, Heerman, P. D., and Khazene, N., 1992, Classfcaton of multspectral remote sensng data usng a back propagaton neural network. IEEE Transactons on Geoscence and Remote Sensng, 30, Hsu, C. W., and Ln, C. -J., 2002, A comparson of methods for mult-class support vector machnes, IEEE Transactons on Neural Networks, 13, Huang, C., Davs, L. S. and Townshend, J. R. G., 2002, An assessment of support vector machnes for land cover classfcaton. Internatonal Journal of Remote Sensng, 23, Hughes, G. F., 1968, On the mean accuracy of statstcal pattern recognzers. IEEE Transactons on Informaton Theory, IT-14, Kavzoglu, T., 2001, An Investgaton of the Desgn and Use of Feed-forward Artfcal Neural Networks n the Classfcaton of Remotely Sensed Images. PhD thess. School of Geography, The Unversty of Nottngham, Nottngham, UK. Knerr, S., Personnaz, L., and Dreyfus, G., 1990, Sngle-layer learnng revsted: A stepwse procedure for buldng and tranng neural network. Neurocomputng: Algorthms, Archtectures and Applcatons, NATO ASI, Berln: Sprnger-Verlag. Osuna, E. E., Freund, R., and Gros, F., 1997, Support vector machnes: tranng and applcatons. A. I. Memo No. 1602, CBCL paper No. 144, Artfcal Intellgence laboratory, Massachusetts Insttute of Technology, ftp://publcatons.a.mt.edu/a-publcatons/pdf/aim-1602.pdf Map Inda Conference 2003 GISdevelopment.net, All rghts reserved.

11 Map Inda 2003 Image Processng & Interpretaton Srnvasan, R., Cannon, M., and Whte, J., 1988, Landsat data destrpng usng power spectral flterng. Optcal Engneerng, 27, Vapnk, V. N., 1995, The Nature of Statstcal Learnng Theory. NewYork: Sprnger-Verlag. Wlknson, G. G. (1997) Open questons n neurocomputng for earth observaton. In Neuro-Computaton n Remote Sensng Data Analyss, edted by I. Kanellopoulos, G. G. Wlknson, F. Rol and J. Austn. London: Sprnger, Zhu, G. and Blumberg, D. G., 2002, Classfcaton usng ASTER data and SVM algorthms; The case study of Beer Sheva, Israel. Remote Sensng of Envronment, 80, Map Inda Conference 2003 GISdevelopment.net, All rghts reserved.