The Classification of Imbalanced Spatial Data

Transcription

1 The Classfcaton of Imbalanced Spatal Data Alna Lazar Department of Computer Scence and Informaton Systems Youngstown State Unversty Youngstown, OH Bradley A. Shellto Department of Geography Youngstown State Unversty Youngstown, OH Abstract Ths paper descrbes a method of mprovng the predcton of urbanzaton. The four datasets used n ths study were extracted usng Geographcal Informaton Systems (GIS). Each dataset contans seven ndependent varables related to urban development and a class label whch denotes the urban areas versus the rural areas. Two classfcaton methods Support Vector Machnes (SVM) and Neural Networks (NN) were used n prevous studes to perform the two-class classfcaton task. Prevous results acheved hgh accuraces but low senstvty, because of the mbalanced feature of the datasets. There are several ways to deal wth mbalanced data, but two samplng methods are compared n ths study. Introducton The am of ths paper s to show that class mbalance has a powerful mpact on the performance of bnary classfcaton algorthms. Most machne learnng algorthms provde models wth better performances when traned usng balanced tranng datasets. However, most of the real-world datasets from varous domans lke medcal dagnoss, document classfcaton, fraud and ntruson detecton are hghly mbalanced towards the postve or the mnorty class. In general, classfcaton algorthms are desgned to optmze the overall accuracy performance. However, for mbalanced data, good accuracy does not mean that most examples from the mnorty class were correctly classfed. Therefore, addtonal performance measures lke recall, f- measure, g-means, AUC should be ncluded when we study mbalanced problems. One common approach to solve the mbalance problem s to sample the data to buld an equally dstrbuted tranng dataset. Several samplng technques were proposed and analyzed n the lterature (Van Hulse, Khoshgoftaar, and Napoltano 007) ncludng random under-samplng, random over-samplng and more ntellgent samplng technques. A second class of methods uses meta-costs and assgns dfferent penaltes for the msclassfed nstances, dependng on ther true class. The problem wth ths type of methods s that t s hard to come up wth a good penalty cost. The last type of methods s the algorthmc-based approach. They tweak the classfer to accommodate mbalanced datasets. The algorthm-based methods use meta-learnng (Lu, An, and Huang 006, Zhu 007) or onlne actve learnng (Ertekn et al. 007) to buld better classfers. Dfferent combnatons of these methods were also reported. Real-world mbalanced datasets come from dverse applcaton areas lke medcal dagnoss, fraud detecton, ntruson detecton, gene proflng, and obect detecton from satellte mages (Kubat, Holte, and Matwn 1998). Our study nvestgates the effect of two samplng technques when appled on four large GIS datasets wth an mbalance rato between.4 and 1.5. The four datasets contan over a mllon nstances each, therefore there s no need to use over-samplng. Besdes that, over-samplng s known to ntroduce excessve nose and ambguty. Instead, the samplng methods consdered were random samplng, under-samplng and the Wlson s edtng algorthm n combnaton. SVM and NN were used before n varous studes to predct urbanzaton and land cover wth almost smlar results, but dfferent predcton patterns (Lazar and Shellto 005, Shellto and Lazar 005). Even f SVM tself does not provde a mechansm to deal wth mbalanced data, t can be easly modfed. SVM bulds the decson boundary on a lmted number of nstances that are close to the boundary, beng unaffected by nstances far away from the boundary. Ths observaton can be used as an actve learnng selecton strategy that provdes a balanced tranng set for the early tranng stages of the SVM algorthm (Ertekn et al. 007). In the Background secton we summarze related studes that deal wth the problem of mbalanced datasets. The secton Support Vector Machnes and Mult-Layer Perceptrons presents the methods used, whle the secton descrbng our experments presets a comparson between

2 random samplng, under-samplng and Wlson s edtng. The last secton presents the conclusons. Background Prevous research (Lazar and Shellto 005, Panowsk et al. 005, Panowsk et al. 00, Panowsk et al. 001, Shellto and Lazar 005, Shellto and Panowsk 003) has shown that classfcaton methods such as Support Vector Machnes (SVM) and Neural Networks (NN) can be successfully used to predct patterns of urbanzaton n large datasets. SVM and NN can then be used as predctve tools to determne f grd cells can be accurately predcted as urban or non-urban cells. The effectveness of the predctve capablty of the SVM and NN can be measured through standard accuracy and other measures. The dataset generated for Mahonng County had over 1,000,000 nstances and the mbalanced rato was appromately 5:1. Even f the accuracy for both SVM and NN were over 90%, the recall was qute low 55%. Lately, several studes dealt wth mbalanced datasets and ther effect on classfcaton performance; however none of the studes ncluded datasets wth over a mllon nstances. Extensve expermental results usng several samplng technques combned wth several classfcaton methods appled on several datasets were reported by (Van Hulse, Khoshgoftaar, and Napoltano 007). The samplng technques consdered were: random mnorty oversamplng, random maorty oversamplng, one-sde selecton, Wlson s edtng, SMOTE (Akban, Kwek, and Japkowcz 004), borderlne SMOTE and cluster-based oversamplng. They concluded that some of the more complcated samplng technques especally one-sde selecton and cluster-based oversamplng exhbt nferor performance n comparson wth some of the smple samplng technques. Support Vector Machnes The machne learnng algorthms named support vector machnes proposed by (Vapnk 1999) consst of two mportant steps. Frstly, the dot product of the data ponts n the feature space, called the kernel, s computed. Secondly, a hyperplane learnng algorthm s appled to the kernel. Let (x, y ), = 1,,l, be the tranng set of examples. The decson y {-1, 1} s assocated wth each nput nstance x R N for a bnary classfcaton task. In order to fnd a lnear separatng hyperplane wth good generalzaton abltes, for the nput data ponts, the set of hyperplanes w,x +b=0 s consdered. The optmal hyperplane can be determned by mamzng the dstance between the hyperplane and the closest nput data ponts. The hyperplane s the soluton of the followng problem: mn 1 τ ( w) w (1) = l l w R R, b R where y ( w, + b) 1 for all = 1,, l. One challenge s that n practce an deal separatng hyperplane may not est due to a large overlap between nput data ponts from the two classes. In order to make the algorthm fleble a nose varable ε 0 for all = 1,,l, s ntroduced n the obectve functon as follows: 1 τ ε () l mn ( w, ε ) = w + C l l w R R, b R = 1 when y ( w, x + b) 1 ε for all = 1,, l. By usng Lagrange multplers the prevous problem can be formulated as the followng convex mamzaton problem (Lu, An, and Huang 006): l 1 l W ( α ) = α = y = y α α K(, x 1, 1 ) (3) when the followng condtons are true, 0 α C for l all = 1,, l, and = α y = 1 0. Here the postve constant C controls the trade-off between the mamzaton of (3) and the tranng error mnmzaton, ε. From the optmal hyperplane equaton the decson functon for classfcaton can be generated. For any unknown nstance x the decson wll be made based on: l f ( x) = sgn( = yα K(, x) + b) (4) 1 whch geometrcally corresponds to the dstance of the unknown nstance to the hyperplane. The method descrbed untl now works well on lnear problems. Functon K, the kernel from (4) enables good results for nonlnear decson problems. The dot product of the ntal nput space s called the new hgher-dmensonal feature space. l l K : R R R, K x, x ) = φ( x ), φ( x ) (5) ( A polynomal kernel, the radal bass and the sgmod functon are sutable kernels wth smlar behavor n terms of the resultng accuracy and they can be tuned by changng the values of the parameters. There s no good method to choose the best kernel functon. The results reported n ths paper were obtaned by usng the followng radal bass functon (Schölkopf and Smola 00) as kernel.

3 x K(, x ) = exp( ) (6) γ Mult-layer Perceptron Neural Networks The mult-layer perceptron (MLP) (Wtten and Frank 000) s a popular technque because of ts well-known ablty to perform arbtrary mappngs, not only classfcatons. Usually bult out of three or four layers of neurons, the nput layer, the hdden layers and the output layer, ths network of neurons can be traned to dentfy almost any nput-output functon. The back-propagaton algorthm used for the tranng process adusts the synaptc weghs of the neurons accordng wth the error at the output. Durng the frst step of the algorthm the predcted outputs are calculated usng the nput values and the network weghts. Afterwards, n the backward pass the partal dervatves of the cost functon are propagated back through the network and the weghts are adusted accordngly. The problem wth the MLP methods s that they are susceptble to converge towards local mnmums. MLP methods are consdered as black box, because t s mpossble to obtan snap-shots of the process. Samplng Methods Snce the datasets consdered have over a mllon nstances we decded to nvestgate under-samplng (US). Ths samplng technque dscards random nstances from the maorty class untl the two classes are equally represented. The other samplng method used n ths study s called Wlson s edtng (Barandela el al. 004) (WE). A k-means nearest neghbor classfcaton procedure s used wth k=3 to classfy each nstance n the tranng set usng all the remanng nstances. Afterwards, all the nstances from the maorty class that were msclassfed are removed. Performance Metrcs Especally n the case of mbalanced datasets, classfcaton accuracy alone s not the best metrc to evaluate a classfer. Several other performance metrcs can be used n order to get a more comprehensve pcture of the classfer s capabltes. Recall or senstvty s the metrc that measures the accuracy on the postve nstances, It can be defned as TruePostve / (TruePostve + FalseNegatve). Specfcty measures the accuracy on the negatve nstances and can be defned as TrueNegatve / (TrueNegatve + FalsePostve). Both senstvty and specfcty are ncorporated n the g-means measure (Ertekn et al. 007), whch s defned as square root from senstvty * specfcty. Datasets Seven broad predctor varables, whch ad n descrbng the dstrbuton of urbanzaton wthn the countes, were constructed usng ESRI s ArcGIS 9. software package. ArcGIS allows for modelng of a vast array of geospatal technques, ncludng the cell-by-cell raster models. These varables were chosen as they reflect large-scale factors that nfluence the patterns of urbanzaton and general urban trends for the regon, as well as beng smlar to GIS varables for urban modelng wthn the Mdwest (Panowsk et al. 005, Panowsk et al. 00, Panowsk 001, Shellto and Panowsk 003). The varables constructed were: a. Dstance to Cty Centers b. Dstance to Hghways c. Dstance to Interstates d. Dstance to Ralroads e. Dstance to Lakes f. Dstance to Rvers g. Densty of Agrculture For the county, a seres of base layers was compled to buld the varables. The NLCD (Natonal Land Cover Database) 001 data was used for locaton of urban areas and as a source of agrcultural data. Base layers for hghways, nterstates, and ralways were drawn from US Census 000 TIGER fles. Lakes and rvers data was derved from Oho Department of Transportaton (ODOT) data. All base layers were proected nto the UTM (Unversal Transverse Mercator) proecton and used to develop the predctor varables n raster format at 30m resoluton. Dstance varables were created by calculatng the Eucldan dstance of each cell from the closest feature n the base layers. The densty varable was constructed by usng a 3x3 movng wndow neghborhood operaton and summng up the number of base layer grd cells n the neghborhood. Urban land was dentfed by selectng all grd cells wth the developed classfcaton n the NLCD dataset. Predctor varables for each county were constructed by ncorporatng data from ther borderng Oho countes, to smulate the nfluence of nearby spatal factors outsde the county borders (for nstance, the promty of a nearby cty center n a borderng county could potentally effect the urban development wthn the target county). The resultant predctor varables created at ths mult-county level were then clpped down to the boundares of the chosen county and used n the analyss. Ths type of data was extracted for four countes from the state of Oho: Delaware, Holmes, Mahonng and Medna. All four resultng datasets contan more than a mllon nstances each. Table 1 shows for each county dataset how many nstances belong to the postve class, how many nstance belong to the negatve class and the rato between the postve and the negatve nstances. All datasets are

4 mldly mbalanced from a.4:1 rato for Mahonng County to a 1.5:1 rato for Holmes County. Delaware Holmes Mahonng Medna Table 1. Number of Tranng Instances and Ratos # Postve 09,765 90, ,411 8,819 # Negatve 1,106,749 1,19, ,43 987,405 Rato 5.761: : : :11 For the frst set of experments we used two classfers, the SVM and the Mult-Layer Perceptron (MLP). We used the lbsvm (Chang and Ln 001) software to run the parameter search, the tranng and the testng for SVM and Weka for the MLP. The experments were smlar wth the experments reported n (Lazar and Shellto 005, Shellto and Lazar 005) for Mahonng County. Random stratfed samplng, whch mantans the rato of postve versus negatve nstances n the datasets, was used to generate datasets of 10,000 nstances for the parameter search and datasets of 50,000 for tranng sets. A grd parameter search was performed for the SVM classfer and the values for the two parameters C and gamma are lsted below n table. County dataset, whch also has the lowest mbalanced rato. Lookng at the low recall values for the other three datasets, we need to nvestgate ways to better classfy the nstances from thee postve class. Experments usng dfferent samplng technques are reported n the next secton. Experments We run experments usng RapdMner (Merswa et al. 006) on the four datasets Delaware, Holmes, Mahonng and Medna as follows. For each dataset we performed 5 runs of a fve-fold cross valdaton wth the lbsvm software. The rbf kernel was used. The two parameters C and gamma where changed to values prevously found by runnng a grd parameter search. Table. Parameters C and gamma for the LbSVM Delaware Holmes Mahonng Medna C gamma Fgure 1. Recall for Holmes County Dataset Next, both classfers SVM and MLP weree traned on the 50,000 nstances datasets and the models obtaned were tested usng the entre datasets. The results obtaned are reported n Table 3. For each dataset and for each classfer (SVM and MLP) three performance metrcs are lsted: accuracy, recalll and g-means. Table 3. Classfcaton Performances for NN and SVM Accuracy Recall G-means Del Hol Mah Med SVM MLP SVM MLP SVM MLP The results show that even f SVM has hgher accuracy for three of the datasets MLP has hgher recall, so a better classfcaton of the postve nstances for three of the datasets. Recall has the largest values for the Mahonng Fgure. G-means for Holmes County Dataset Three samplng technques were used: random stratfed samplng (RS), equal under-samplng (US) and Wlson s

5 edtng samplng (WE). Each experment was terated through subsample datasets wth szes between 100 and 5000, wth a step of 100. The results are shown on two countes Holmes and Medna, due to space lmtaton. The Holmes County has the hghest mbalanced rato of appromately 1.5 and Medna has a 4.3 mbalanced rato. All four fgures show that both under-samplng and Wlson s edtng samplng have a great nfluence on the classfcaton performance of the SVM learner. As accuracy s not relevant n the case of mbalanced datasets we looked at recall and g-means. The Wlson e edtng worked only slghtly better than the equal under-samplng g, but requred extensve preprocessng. The bggest dfference n performance can be seen n Fgure 1 wth the recall for the Holmes County. Conclusons We have presented an expermental analyss performed on large mbalanced GIS extracted datasets. The goal was to fnd what samplng technques mprove the classfcaton performance especally for the mnorty class. It s mportant n the casee of mbalanced datasets thatt addtonal performance measures lke recall and g-means are compared n addton to the usual accuracy. We concluded that both equal under-samplng and Wlson s edtng work better that ust smple random stratfed samplng, but there s no sgnfcant dfference between the two. Further research may nvestgate how other learners lke Neural Networks orr Decson Trees perform wth under- cost-senstve learnng, and meta-learners are other samplng and Wlson s edtng samplng. Over-samplng, alternatves that can be used to mprove the performance for ourr datasets. References Akban, R.; Kwek, S.; and Japkowcz N Applyng support vector machnes to mbalanced datasets. Proceedngs of European Conference on Machne Learnng Psa, Italy, Sprnger- Verlag, Germany. Barandela, R.; Valdovnos, R. M.; Sanchez J. S.; and Ferr, F. J The Imbalancedd Tranng Sample Problem: Under or Over Samplng? In Jont IAPR Internatonal Workshops on Structural, Syntactc, and Statstcal Pattern Recognton (SSPR/SPR 04), Lecturee Notes n Computer Scence 3138: Fgure 3. Recall for Medna County Dataset Chang,, C.; and Ln, C-J. 001 LIBSVM : a lbrary for support vector machnes, 001. Software at /lbsvm. Last accessed 01/15/011. Crstann, N; and Shawe-Taylor, J An Introducton to Support Vector Machnes and Other Kernel-based Learnng Methods, Cambrdge Unversty Press, Cambrdge, England. Ertekn, S.; Huang, J.; Bottou, L.; and Lee Gles, C Learnng on the border: actve learnng n mbalanced data classfcaton. In Proceedngs of the Sxteenth ACM Conference on Conference on Informaton and Knowledge Management (Lsbon, Portugal, November 06-10, 007). CIKM ' ACM, New York, NY.. Koggalage, R.; and Halgamuge, S Reducng the Number of Tranng Samples for Fast Support Vector Machne Classfcaton. Neural Informaton Processng Letters and Revews (3): Fgure 4. G-means for Medna County Dataset Kubat, M.; Holte, R. C.; and Matwn. S Machne Learnng for thee detecton of ol splls n satellte radar mages. Machne Learnng, 30(-3): Lazar, A.; and Shellto, B. A Comparng Machne Learnng Classfcaton Schemes a GIS Approach. In Proceedngs of

6 ICMLA 005: The 005 Internatonal Conference on Machne Learnng and Applcatons, IEEE. Lu, Y.; An A.; and Huang, X Boostng Predcton Accuracy on Imbalanced Datasets wth SVM Ensembles. Lecture Notes n Artfcal Intellgence, vol. 3918: Merswa, I.; Wurst, M.; Klnkenberg, R.; Scholz, M.; and Euler, T YALE: Rapd Prototypng for Complex Data Mnng Task. In Proceedngs of the 1th ACM SIGKDD Internatonal Conference on Knowledge Dscovery and Data Mnng (KDD- 06). Panowsk, B.; Pthada, S.; Shellto, B. A.; and Alexandrds, K Calbratng a neural network based urban change model for two metropoltan areas of the upper Mdwest of the Unted States. Internatonal Journal of Geographcal Informaton Scence 19 (): Panowsk, B.; Brown, D.; Shellto, B. A.; and Mank, G. 00. Use of Neural Networks and GIS to Predct Land Use Change. Computers, Envronment, and Urban Systems 6(6): Panowsk, B.; Shellto, B. A.; Bauer, M. and Sawaya, K Usng GIS, Artfcal Neural Networks and Remote Sensng to Model Urban Change n the Mnneapols-St. Paul and Detrot Metropoltan Areas. In Proceedngs of the ASPRS Annual Conference, St. Lous, MO. Schölkopf, B.; and Smola, A. 00. Learnng wth Kernels. MIT Press, Cambrdge Massachusetts. Shellto, B. A.; and Lazar, A Applyng Support Vector Machnes and GIS to Urban Pattern Recognton. In Papers of the Appled Geography Conferences, volume 8. Shellto, B. A.; and Panowsk, B Usng Neural Nets to Model the Spatal Dstrbuton of Seasonal Homes. Cartography and Geographc Informaton Scence 30 (3): Van Hulse, J.; Khoshgoftaar, T. M.; and Napoltano. A Expermental perspectves on learnng from mbalanced data. In Proceedngs of the 4th Internatonal Conference on Machne Learnng (Corvals, Oregon, June 0-4, 007). Z. Ghahraman, Ed. ICML '07, vol. 7. ACM, New York, NY, Vapnk, V. N The Nature of Statstcal Learnng Theory, nd edton, Sprnger-Verlag, New York, NY. Wtten, I. H.; and Frank, E Data Mnng: Practcal Machne Learnng Tools and Technques wth Java Implementaton, Morgan Kaufmann Publshers, Burlngton, MA. Zhu, X Lazy Baggng for Classfyng Imbalanced Data. In Seventh IEEE Internatonal Conference on Data Mnng Omaha, NE.