Title: A Novel Protocol for Accuracy Assessment in Classification of Very High Resolution Images

Transcription

1 2009 IEEE. Personal use of ths materal s permtted. Permsson from IEEE must be obtaned for all other uses, n any current or future meda, ncludng reprntng/republshng ths materal for advertsng or promotonal purposes, creatng new collectve works, for resale or redstrbuton to servers or lsts, or reuse of any copyrghted component of ths work n other works. Ttle: A Novel Protocol for Accuracy Assessment n Classfcaton of Very Hgh Resoluton Images Ths paper appears n: IEEE Transactons on Geoscence and Remote Sensng Date of Publcaton: 2010 Author(s): Claudo Persello and Lorenzo Bruzzone Volume: 48, Issue: 3 Page(s): DOI: /TGRS

2 A Novel Protocol for Accuracy Assessment n Classfcaton of Very Hgh Resoluton Images Claudo PERSELLO, Student Member IEEE, and Lorenzo BRUZZONE, Senor Member IEEE Dept. of Informaton Engneerng and Computer Scence, Unversty of Trento, Va Sommarve, 14, I Trento, Italy; e-mal: claudo.persello@ds.untn.t, lorenzo.bruzzone@ng.untn.t. Abstract - Ths paper presents a novel protocol for the accuracy assessment of thematc maps obtaned by the classfcaton of very hgh resoluton (VHR) mages. As the thematc accuracy alone s not suffcent to adequately characterze the geometrcal propertes of hgh resoluton classfcaton maps, we propose a protocol that s based on the analyss of two famles of ndces: ) the tradtonal thematc accuracy ndces, and ) a set of novel geometrc ndces that model dfferent geometrc propertes of the objects recognzed n the map. In ths context, we present a set of ndces that characterze fve dfferent types of geometrc errors n the classfcaton map: 1) over-segmentaton, 2) under-segmentaton, 3) edge locaton, 4) shape dstorton, and 5) fragmentaton. Moreover, we propose a new approach for tunng the free parameters of supervsed classfers on the bass of a multobjectve crteron functon that ams at selectng the parameter values that result n the classfcaton map that jontly optmze thematc and geometrc error ndces. Expermental results obtaned on Quckbrd mages show the effectveness of the proposed protocol n selectng classfcaton maps characterzed by a better tradeoff between thematc and 2

3 geometrc accuracy than standard procedures based only on thematc accuracy measures. In addton, results obtaned wth Support Vector Machnes (SVM) classfers confrm the effectveness of the proposed mult-objectve technque for the selecton of free parameter values for the classfcaton algorthm. Index Terms accuracy assessment, classfcaton maps, thematc accuracy, geometrc accuracy, mage classfcaton, very hgh resoluton mages, remote sensng. I. INTRODUCTION Wth the avalablty of very hgh resoluton mages acqured by satellte multspectral (MS) scanners (e.g., GeoEye-1, Quckbrd, Ikonos, SPOT 5), t s possble to acqure detaled nformaton on the shape and the geometry of the objects present on the ground. Ths detaled nformaton can be exploted by automatc classfcaton systems to generate land-cover maps that exhbt a hgh degree of geometrcal detals. The precson that the classfcaton system can afford n the characterzaton of the geometrcal propertes of the objects present on the ground s partcularly relevant n many practcal applcatons, e.g., n urban area mappng, buldng characterzaton, target detecton, crop felds classfcaton n precson farmng, etc. In ths context, t s necessary to further develop both algorthms for characterzng the textural and geometrc nformaton present n VHR mages, and effectve classfcaton technques capable to explot these propertes for ncreasng the classfcaton accuracy. In the lterature, several technques have been proposed for the classfcaton of VHR mages. Among the others, we recall the use of texture, geometrc features, and morphologcal transformatons for characterzng the context of each sngle pxel, and the use of classfcaton algorthms that can operate n large dmensonal feature spaces (e.g., SVM) [1]-[5]. Nonetheless, a major open ssue n 3

4 classfcaton of VHR mages s the lack of adequate strateges for a precse evaluaton of the qualty of the produced thematc maps. The most common accuracy assessment methodology n classfcaton of VHR mages s based on the computaton of thematc accuracy measures accordng to collected reference data. However, the thematc accuracy alone does not result suffcent for effectvely characterzng the geometrcal propertes of the objects recognzed n a map, because t assesses the correctness of the land-cover labels of sparse test pxels (or regons of nterests) that do not model the actual shape of the objects n the scene. Thus, often maps derved by dfferent classfers (or wth dfferent parameter values for the same classfer) that have smlar thematc accuracy exhbt sgnfcantly dfferent geometrc propertes (and thus global qualty). For ths reason, n many real classfcaton problems the qualty of the maps obtaned by the classfcaton of VHR data s assessed also through a vsual nspecton. However, ths procedure can provde just a subjectve evaluaton of the map qualty that can not be quantfed. Thus, t s mportant to develop accuracy assessment protocols for a precse, objectve, and quanttatve characterzaton of the qualty of thematc maps n terms of both thematc and geometrc propertes [6]. These protocols could be used not only for assessng the qualty of thematc maps generated by dfferent classfcaton systems, but also for better drvng the model selecton of a sngle classfer,.e., the selecton of the optmum values for the free parameter of a supervsed categorzaton algorthm. An mportant area n whch some studes related to the aforementoned problem have been done n the past s that of landscape ecology. Some approaches have been proposed n the landscape ecology lterature to compare dfferent maps by consderng the spatal structure of the landscape [7] (and thus not only the thematc accuracy). As an example, n [8] dfferent comparson methods that consder both the spatal structure and the pxel-based overlap (.e., the thematc accuracy) smultaneously are presented. However, these methods are developed n a dfferent framework and do not consder the partcular propertes of classfcaton maps derved 4

5 form VHR remote sensng mages and the ssues related to the tunng of the free parameters of a classfer. In ths paper we address the abovementoned problem by proposng a novel protocol for a precse, automatc, and objectve characterzaton of the accuracy of thematc maps derved from VHR mages. The proposed protocol s based on the evaluaton of two famles of ndces: ) thematc accuracy ndces, and ) a set of novel geometrc ndces that assess dfferent propertes of the objects recognzed n the thematc map. The proposed protocol can be used to: ) objectvely characterze the thematc and geometrc propertes of classfcaton maps; ) to select the map that better ft specfc user requrements; or ) to dentfy the map that exhbts n average best global propertes f no specfc requrements are defned. Moreover, we propose a novel approach for tunng the free parameters of supervsed classfcaton algorthms (e.g., SVM), whch s based on the optmzaton of a mult-objectve problem. The am of ths approach s to select the parameter values that result n a classfcaton map that exhbts hgh geometrc and thematc accuraces. The paper s organzed nto sx sectons. The next secton presents the background on the assessment of thematc accuracy of land-cover maps. Secton III descrbes the proposed accuracy assessment protocol, and dscusses the two famles of presented geometrc and thematc ndces. Secton IV llustrates the proposed mult-objectve crteron for the tunng of the free parameters (model selecton) of a classfer. Secton V presents the obtaned expermental results, whle secton VI draws the concluson of the paper. II. BACKGROUND ON THEMATIC ACCURACY ASSESSMENT OF CLASSIFICATION MAPS In ths secton we brefly recall the man concepts on the procedures used to assess the thematc accuracy of a classfcaton map obtaned by a supervsed classfer [9], [10]. In general, two man ssues should be addressed: ) the collecton of the labeled samples for both tranng and testng a supervsed algorthm (whch may requre the subdvson of the reference sample set n 5

6 two or more dsjont sets); and ) the choce of the statstcal measure to evaluate the error (or accuracy) n pattern classfcaton. Wth respect to the frst ssue, several resamplng methods have been proposed n the pattern recognton and statstcal lterature, e.g., resubsttuton, holdout, leave-one-out, cross-valdaton, bootstrap [11]-[14]. Holdout s one of the most wdely adopted resamplng strateges n remote sensng applcatons. It conssts n parttonng the avalable labeled samples n two ndependent sets or n drectly collectng two ndependent sets of samples n separate areas of the scene. One set s used for tranng the classfer, the other one for assessng the classfcaton accuracy. In some cases t s preferable to splt the avalable samples n three sets: ) one for tranng the algorthm (tranng set); ) one for tunng the free parameters of the classfer (valdaton set); ) one for assessng the fnal accuracy (test set). Holdout s less computatonally demandng wth respect to other methods (e.g., leave-one-out and k-fold cross valdaton) and t s partcularly relable when the avalable labeled samples are acqured n two spatally dsjont portons of the scene. Indeed, n ths case t s possble to asses the generalzaton capablty of the classfer for test pxels that are spatally dsjont from the ones used for the tranng (whch may present a dfferent spectral behavor). Wth all the mentoned resamplng methods, t s mportant to adopt a stratfed approach,.e., the tranng and test sets (or each of the k folds) should contan approxmately the same proportons of the class labels as the orgnal dataset. Otherwse mbalanced and skewed results can be obtaned. Wth respect to statstcal measures for accuracy evaluaton, the complete descrpton of the nformaton that comes out from the comparson of the classfcaton of test samples wth the reference labeled data s gven by the confuson (or error) matrx N. N s a square matrx of sze C C (where C s the number of nformaton classes n the consdered problem) defned as: 6

7 N n11 n12 n13... n1 C n n = n nc n CC (1) The generc element nj of the matrx denotes the number of samples classfed nto category ( = 1,..., C) by the supervsed classfer that are assocated wth label j ( j = 1,..., C) n the reference data set. Ths representaton s complete as the ndvdual accuracy of each category s descrbed along wth both the errors of ncluson (commsson errors) and errors of excluson (omsson errors) [9]. From the confuson matrx, dfferent ndces can be derved to summarze the nformaton wth a scalar value. Let us consder the sum of the elements of the row, n + C = n j= 1 j (whch s the number of samples classfed nto the category n the classfcaton map), and the sum of the elements of column j, n + j C = n j= 1 j (whch s the number of samples belongng to category j n the reference data set). Two commonly adopted ndces are the overall accuracy (OA) and the kappa coeffcent of accuracy (kappa), defned as: OA = C C = 1 n n n n n n kappa = C j + + = 1 = 1 C 2 n n+ n+ = 1 (2) (3) where n s the total number of test samples. The OA represents the rato between the number of samples that are correctly recognzed by the classfcaton algorthm wth respect to the total number of test samples. The kappa coeffcent of accuracy s a measure based on the dfference between the actual agreement n the confuson matrx (as ndcated by the man dagonal) and the 7

8 chance agreement, whch s ndcated by the row and column totals (.e., the margnals). The kappa coeffcent s wdely adopted as t uses also off-dagonal elements of the error matrx, and as t compensates for chance agreement. However, as ponted out n [15], kappa statstcs has also unfavorable features. The man objecton to the kappa coeffcent s that t was ntroduced as a measure of agreement for two observers (see [16]). Thus, the kappa coeffcent evaluates the departure from the assumpton that two observers ratngs are statstcally ndependent, rather than a measure of classfcaton accuracy. For ths reason, n [15] t s suggested to use other measures nstead of kappa statstc, e.g., the class-averaged accuracy defned as: C n 1 j= 1 CA = C n + j jj, (4) or an alternatve coeffcent based on Kullback-Lebler nformaton. We refer the reader to [9]-[11] for further detals on accuracy assessment procedures n remote sensng mage classfcaton. It s mportant to pont out that all the abovementoned thematc accuracy measures do not consder the geometrcal qualty of the map under assessment and the shape of the objects present n the scene, thus resultng n the mpossblty to assess the correctness of the geometry of the objects recognzed by the classfcaton algorthm. Ths s reasonable to evaluate the qualty of classfcaton maps obtaned by medum or low resoluton mages, where the geometry of the objects s dffcult to characterze. On the contrary, for adequately assessng the qualty of classfcaton maps obtaned by VHR mages, t s mportant to defne ndces capable to evaluate the geometrcal propertes of the maps, and to use them together wth more tradtonal thematc ndces. III. PROPOSED PROTOCOL FOR ACCURACY ASSESSMENT IN VHR IMAGES In ths secton we present the proposed protocol for accuracy assessment that s based on the computaton of both thematc and geometrc ndces. The proposed procedure for thematc accuracy assessment s a smple refnement of the more tradtonal procedures descrbed n the 8

9 prevous secton, whch takes nto account partcular propertes of the classfcaton of VHR mages. On the contrary, the ntroducton of geometrc ndces to characterze the propertes of the objects present n VHR mages s one of the man contrbutons of the paper. Thematc and geometrc ndces are descrbed n the followng two subsectons, respectvely. A. Thematc error ndces When VHR mages are consdered, we can clearly dentfy two dfferent contrbutons to the overall thematc accuracy: ) the accuracy obtaned on homogeneous areas, where pxels are characterzed by the spectral sgnature of only one class; ) the accuracy obtaned on borders of the objects and detals, where pxels are assocated wth a mxture of the spectral sgnatures of dfferent classes. These two contrbutons model the atttude of a classfer to correctly classfyng homogeneous regons and hgh frequency areas, allowng a more precse assessment of the qualty of the classfcaton map. The classfcaton of mxed pxels s a dffcult task wth crsp classfers, whch should decde for the predomnant class n the area assocated wth the pxel (fuzzy classfers may be adopted n ther place for consderng the contrbutons of the dfferent landcover types to the spectral sgnature assocated wth each sngle pxel [17]). The proposed thematc accuracy assessment conssts of the calculaton of two separate ndces: ) thematc accuracy on homogeneous areas, ) thematc accuracy on edge areas. Ths s accomplshed extendng the holdout strategy by defnng two ndependent test sets: one on homogeneous areas (pxel nsde objects), the other one on edge areas (pxels on the boundares of objects). Ths results n the calculaton of two ndependent confuson matrces. Any ndex derved from the confuson matrces (e.g., overall accuracy, kappa coeffcent, etc.) may be adopted to calculate the accuracy on the two separate test sets. It s worth notng that dfferent ndces provde dfferent nformaton and can be used together (see the next secton for a detaled dscusson on the combned use of multple ndces for the tunng of the free parameters of a supervsed classfer). 9

10 B. Geometrc error ndces The geometrc accuracy of a classfcaton map s related to ts precson n reproducng the correct geometry, the shapes, and the boundares of the objects (e.g., buldngs, streets, felds, etc.) present n the scene under nvestgaton. In ths paper, n order to quantfy the geometrc accuracy of maps characterzed by very hgh spatal resoluton, we defne a set of object-based ndces (error measures) that evaluate dfferent geometrc propertes of the objects represented n a thematc map wth respect to a reference map. Some of these ndces are partally nspred to the measures used n the accuracy assessment of segmentaton maps, whle others are mported from dfferent domans of mage processng. These ndces are computed by usng a reference map that defnes the exact shape, structure and poston of a set = { O O O } Ο 1, 2,..., of d objects (e.g., buldngs) d adequately dstrbuted n the scene under nvestgaton and wth dfferent propertes (see the example n Fgure 2). Generally, gven the hgh resoluton of VHR mages, the map of reference objects can be easly defned by photonterpretaton (few objects are suffcent for a good characterzaton of the propertes of the map). Please note that the labels of the classes of the reference objects are not requred for the computaton of the geometrc accuracy ndces. In ths way the evaluaton of the geometrc propertes of the objects recognzed n the map can be separated from the assessment of the thematc accuracy. Moreover, we do not requre havng reference objects for all the classes consdered n the classfcaton problem, but only for the classes for whch the geometrc propertes are mportant and the precse shape can be easly defned (e.g., buldngs, felds, lakes, brdges, etc). 10

11 Fgure 1 Example of a map of reference objects. Let us consder that the thematc map under assessment (e.g., obtaned by an automatc algorthm or by photonterpretaton) s made up of a set M = { M M M },,..., r 1 2 of r dfferent regons of connected pxels (wth 4 or 8-connectvty), such that each pxel n M, j = 1,2,..., r, s j assocated wth the same label L j, where L j s one of the C nformaton classes n { ω ω ω } Ω= 1, 2,...,. In order to calculate the geometrc error measures, t s necessary to dentfy C for each object O n the reference map the correspondng regon n the thematc map M. Ths can be done by consderng the degree of overlappng between the pxels n the reference object O and n the regons area wth the object M j, j = 1,2,..., k. The regon M n the map wth the hghest overlappng O (.e., wth the hghest number of common pxels) s selected accordng to: M = arg max O M j M M (5) where s the cardnalty of a set, and here s used to extract the number of pxels (area) from a regon (see the example n Fgure 2). Gven a par ( O, M ), t s possble to calculate a set of local geometrc error measures err, = 1,2,..., d, h= 1, 2,..., m, that evaluate the degree of ( h ) msmatchng (n terms of m dfferent specfc geometrc propertes) between the reference object and the correspondng regon on the map. Global error measures ( h) err, h 1, 2,..., = m, can then be defned on the bass of the local measures. 11

12 Fgure 2 Example of a reference object O and the regons on the map that overlap wth t. The regon M 1 has the hghest overlappng area wth O and s selected accordng to (5). The adopted measures are: 1) over-segmentaton error, 2) under-segmentaton error, 3) edge locaton error, 4) fragmentaton error, and 5) shape error. 1) Over-segmentaton Smlarly to the segmentaton process, ths error refers to the subdvson of a sngle object nto several dstnct regons n the classfcaton map [see the example n Fgure 3 (a)]. The proposed local error measure can be wrtten as: O M OS( O, M ) = 1 (6) O Ths measure evaluates the rato between the overlappng area of the two regons ( O, M ) wth respect to the area of the reference object. The ndex OS s defned n order to scale the output values n the range [0,1). The hgher s the value of the error, the hgher s the level of oversegmentaton of the object O n the consdered classfcaton map. The value of ths error s 0 n the optmal case where the two regons are n full agreement, whle t tends to 1 n the worst case of just one common pxel among the two regons. 2) Under-segmentaton - The under-segmentaton refers to the classfcaton errors that result n group of pxels belongng to dfferent objects fused nto a sngle regon. The proposed local error measure s defned as: O M US( O, M ) = 1 (7) M 12

13 Unlke the over-segmentaton, the under-segmentaton error s computed by consderng the rato between the area of overlappng among M and O, and the area of the regon on the map M. Also the US error vares n the range [0,1). Value 0 of ths ndex corresponds to perfect agreement between M and O, whle values close to 1 reflect a hgh amount of undersegmentaton (.e., the regon M and O ). M s much bgger than the area of overlappng between the regons (a) (b) Fgure 3 (a) Example of over-segmentaton: the regon M recognzed on the map s smaller than the reference object O. (b) Example of under-segmentaton: the regon M recognzed on the map s bgger than to the reference object O. 3) Edge locaton - Ths ndex measures the precson of the object edges recognzed n the classfcaton map wth respect to those of the actual object. Let eo ( ) denote the operator that extracts the set of edge pxels from a generc regon O. In ths framework, we consder the possblty to ntroduce a tolerance n the recognton of the object borders. Ths can be mplemented by adoptng an operator e() that extracts the border lne of the objects wth a wdth greater than 1 pxel (e.g., 2 or 3 pxels). The defnton of the border error s gven by: eo ( ) em ( ) ED( O, M ) = 1 (8) eo ( ) 13

14 Ths error measure vares n the range [0,1) lke the prevous ones. A perfect matchng n the borders of the two regons M and O leads to an error value equal to 0, whereas a large msmatchng among the regon edges results n error values close to 1. 4) Fragmentaton error - The fragmentaton of a classfcaton map refers to the problem of sub-parttonng sngle objects nto dfferent small regons. In order to quanttatvely measure ths type of error, we defne a measure based on the number r of regons M j, j = 1, 2,..., r, that have at least one pxel n common wth the reference object all the regons overlappng wth the reference object O. For ths reason, we defne the set O as: { M, j 1,2,..., r : O M } j j R of R = = (9) The proposed fragmentaton error s then defned by the followng equaton: FG ( O, M ) = r 1 O 1 (10) Ths error value s scaled n a range[0,1]. The value s 0 n the optmal case when only one regon M j s overlappng wth the reference object pxels of the object O belong to dfferent regons wth respect to the sze (area) of the reference object O, whereas t s 1 n the worst case where all the M j on the map. The measure s normalzed O. It s worth notng that the fragmentaton error s correlated wth the over-segmentaton error, but dffers from the latter because t takes nto account all the r regons regon M obtaned by (5). M j that overlap wth the real object O, nstead of the area of the sngle 5) Shape error - Ths error s used to evaluate the shape dfference between an object O and the correspondng regon M j on the map. In order to characterze the shape of an object, several shape factors have been proposed n the lterature and can be adopted (e.g., compactness, 14

15 sphercty, eccentrcty [18]). Thus the shape error can be defned as the absolute value of the dfference n the selected shape factor sf () of the two regons M and O : SH = sf ( O ) sf ( M ) (11) It s worth notng that by adoptng shape factors normalzed n the range [0,1], the defned shape error measure wll vary n the same range. On the bass of the above defned measures of local errors (.e., errors assocated wth sngle objects n the map), t s then possble to estmate global behavors of the geometrc propertes of the classfcaton map. Global error measurements can be obtaned by averagng the local errors over the d measurements assocated wth the reference objects n Ο,.e., a generc global error measure characterzng property ( h) err of the map can be expressed as: err ( h) 1 d ( h) err d = 1 =, (12) where err s a local error h on the object. In ths way we gve the same weght to the errors ( h ) over the d objects, ndependently from ther sze. Other possble defntons of the global measures may take nto account the sze of the dfferent objects,.e., err ( h) 1 d ( h) O err d = 1 = (13) or can weght dfferently the objects on the bass of specfc user-defned requrements,.e., err ( h) 1 d ( h) λ err d = 1 =, (14) where λ, = 1,2,..., n are defned by the user. For example, the user may specfy that geometrc errors on buldngs are more mportant than geometrc errors on other objects, lke streets, crop felds or lakes. Global measures are then used to estmate dfferent geometrc propertes of the 15

16 map. Combnng the dfferent global ndces n a sngle measure that averages geometrc ndces s also possble. Nevertheless, ths procedure would result n a measure that s dffcult to understand. (a) (b) (c) Fgure 4 Example of a regon M recognzed on the map (wth correspondng reference object O ) that exhbts (a) large edge error, (b) hgh level of fragmentaton, and (c) relatvely hgh shape error. IV. PROPOSED MULTI-OBJECTIVE STRATEGY FOR CLASSIFIER PARAMETER OPTIMIZATION Besdes qualty assessment of classfcaton maps obtaned accordng to dfferent procedures (e.g., dfferent automatc classfers, photonterpretaton, etc.), an accuracy ndex s also an mportant measure for tunng the free parameters of supervsed classfers (ths process s also ndcated as model selecton). Let us consder a generc supervsed algorthm for whch a vector θ of free parameters should be selected n order to optmze the qualty of the output map. Standard approaches are based on the adopton of a scalar ndex to assess the thematc accuracy of the map (e.g., the overall accuracy or the kappa coeffcent), and on the selecton of the vector θ that maxmzes such a scalar value on the test samples. If a vector I of qualty ndces that characterze dfferent thematc and geometrc propertes of the classfcaton map s consdered, the selecton of θ should be based on a dfferent optmzaton strategy. The smplest (yet emprcal and only partally relable) strategy s to defne a sngle error functon E() combnng the m proposed error measures accordng to a weghted average: E m ( j) ( ΟM, ) = cj err (15) j= 1 16

17 where the terms c, j = 1, 2,..., m are defned by the user. The set of parameter values of θ that j produces the classfcaton map that mnmzes E( ΟM, ) represents the soluton to the consdered problem. Nevertheless, ths formulaton has an mportant drawback: the defnton of the c j (whch sgnfcantly affects the fnal result) s very crtcal because of the dfferent ntrnsc scales of the consdered errors. In addton, the physcal nformaton conveyed by the resultng global ndex s dffcult to understand. To overcome ths drawback, we propose to model our problem as a mult-objectve mnmzaton problem, where the mult-objectve functon g( θ ) s made up of m dfferent objectves g1(), θ g2(),..., θ g m () θ that represent the set of adopted error measures computed for dfferent values of the classfer parameters (e.g., dfferent thematc and geometrc ndces). All the dfferent objectves of g( θ ) have to be jontly mnmzed and are consdered equally mportant. In general all the proposed thematc ndces (evaluated on homogeneous and border areas wth dfferent statstcal parameters) and geometrc ndces could be used for the defnton of g( θ ). However, dependng on the applcaton, t could be more approprate to use dfferent subsets of the presented ndces as objectves of the optmzaton problem (e.g., for meetng some partcular qualty propertes of the classfcaton map requred by the end users). Thus, the mult-objectve problem can be formulated as follows: { g θ } g θ = [ g θ g θ g θ ] mn ( ), ( ) ( ), ( ),..., ( ) θ S h subject to θ = ( θ, θ,..., θ ) S, h m (16) where S denotes the search space for the classfer parameters. Ths problem s characterzed by a vector-valued objectve functon g( θ ) and cannot be solved n order to derve a sngle soluton lke n optmzaton problems characterzed by a sngle objectve functon. Instead, a set of optmal solutons * P can be obtaned by followng the concept of Pareto domnance. In greater detal, a soluton * θ s sad to be Pareto optmal f t s not domnated by any other soluton n the search 17

18 * space,.e., there s no other θ such that g ( θ) g ( θ ) ( = 1, 2,..., m) and g * j() < gj( ) θ θ for at least one j ( j = 1, 2,..., m). Ths means that * θ s Pareto optmal f there exsts no other subset of classfer parameters θ whch would decrease an objectve wthout smultaneously ncreasng another one (Fgure 5 clarfes ths concept wth a graphcal example). The set * P of all optmal solutons s called Pareto optmal set. The plot of the objectve functon of all solutons n the Pareto set s called Pareto front PF = {() g θ θ P }. The man advantage of the mult-objectve * * approach s that t avods to aggregate metrcs capturng multple objectves nto a sngle measure. On the contrary, t allows one to effectvely dentfy dfferent possble tradeoffs between maps exhbtng dfferent thematc and geometrc propertes. Fgure 5 Example of Pareto optmal solutons and domnated solutons n a two-objectve search space. soluton Because of the complexty of the search space, an exhaustve search of the set of optmal * P s unfeasble. Thus, nstead of dentfyng the true set of optmal solutons, we am to estmate a set of non-domnated solutons * ˆP wth objectve values as close as possble to the Pareto front. Ths estmaton can be done wth dfferent mult-objectve optmzaton algorthms [e.g., mult-objectve evolutonary algorthms (MOEA) [19], [20]]. The fnal selecton of the optmal soluton among all estmated non-domnated solutons s demanded to the user, who can select the best tradeoff among the consdered objectves on the bass of the specfc applcaton 18

19 (e.g., one could tolerate to have under-segmented maps rather than over-segmented ones, or prefer to have less fragmented objects rather than hgh precson n the shape, etc.). V. EXPERIMENTAL RESULTS Ths secton presents an expermental analyss amed at studyng the relablty of the proposed protocol for accuracy assessment of classfcaton maps obtaned by VHR mages. We frst appled the proposed ndces to the qualty assessment of dfferent thematc maps obtaned by the classfcaton (carred out wth dfferent automatc technques) of a Quckbrd mage acqured on the cty of Pava, Italy. Then, n a second set of experments, we appled the proposed multobjectve strategy to the model selecton of an SVM classfer n the analyss of a dfferent Quckbrd mage acqured on the cty of Trento, Italy. In our mplementaton of the geometrc ndces we consdered a tolerance of 3 pxels for the edge locaton error, and we selected the eccentrcty [18] as shape factor for the evaluaton of the shaper error. The global geometrc errors were computed on the bass of (12). A. Qualty assessment of classfcaton maps The frst consdered dataset s made up of a Quckbrd multspectral mage acqured on the cty of Pava (northern Italy) on June 23, In partcular, we used a panchromatc mage and a pan-sharpened multspectral mage (see Fgure 6a) obtaned by applyng a Gram Schmdt fuson technque [21] to the panchromatc channel and to the four bands of the multspectral mage. The mage sze s pxels wth a spatal resoluton of 0.7m. Greater detals about ths dataset can be found n [1]. Table 1 presents the number of labeled reference samples for each set and class. Test pxels used for the assessment of thematc accuracy were collected on both edge and homogeneous areas. Test set pxels were taken from areas of the scene spatally dsjont from those related to the tranng samples. Fgure 6b shows the map of reference objects used for the evaluaton of the geometrc error ndces. In partcular, sx dfferent buldngs were manually 19

20 selected and consdered as reference objects. It s worth notng that gven the very hgh resoluton of the mages the procedure for dgtzng few reference objects s smple and very fast. Table 1 - Number of samples n the tranng and test sets (Pava data set) Class Tranng set Number of patterns Test set on Test set on edge areas homogeneous areas Water 180 Tree areas Grass areas Roads Shadow Red buldngs Gray buldngs Whte buldng TOTAL (a) (b) Fgure 6 (a) Real color composton of the mage acqured by the Quckbrd satellte on the cty of Pava (northern Italy). (b) Map of reference objects. In our experments, we obtaned dfferent thematc maps of the scene by usng dfferent automatc classfcaton systems. The dfferent systems were defned by varyng the feature vector (.e., consderng only spectral features or also multscale/multlevel contextual features), the supervsed classfcaton algorthms (.e., parallelepped, maxmum lkelhood, and SVM 20

21 classfers), and n some cases addng a post-processng phase for regularzng the fnal classfcaton map. These systems were chosen wth the goal to obtan classfcaton maps wth dfferent propertes. Fgure 7 shows the thematc maps obtaned by the dfferent consdered classfcaton systems. In partcular, the maps (a)-(b)-(c)-(d) are obtaned by consderng a feature vector that s made up of only the orgnal spectral features. Map (a) s obtaned by usng a very smple parallelepped classfer (wth σ = 2 ) [22]; map (b) s derved adoptng a Gaussan Maxmum Lkelhood (ML) classfer; map (c) s obtaned by applyng a majorty flter (wth a sldng wndow of sze 3 3) as post-processng to the map (b) [22]; map (d) s the result of the classfcaton wth SVM (usng Gaussan kernels). The maps (e)-(f)-(g)-(h) are yelded usng both spectral and contextual features, and adoptng SVM as classfcaton algorthm. Map (e) s obtaned consderng features extracted on the bass of the generalzed Gaussan pyramd decomposton. In detal, the mages were teratvely analyzed by a Gaussan kernel low-pass flter (wth 5 5 square analyss wndow) and were under-sampled by factor two. We exploted fve levels of pyramdal decomposton to characterze the spatal context of pxels and to label each pxel of the scene under nvestgaton. Maps (f)-(g)-(h) are obtaned usng the multlevel contextbased feature-extracton approach proposed n [1]; dfferent statstcal parameters are extracted from the pxels n each regon defned at sx dfferent levels by a herarchcal segmentaton process. In partcular, for map (f) we consdered the mean value for the frst fve levels and the standard devaton for the levels three, four, and fve; for map (g) we consdered only the mean for all frst fve levels. Map (h) s obtaned consderng the mean value extracted from all sx segmentaton levels. 21

22 (a) Parallelepped (b) ML (c) ML wth post-processng (d) SVM 22

23 (e) SVM Gaussan Pyramd (f) SVM multlevel features 5 levels (1) (g) SVM multlevel features 5 levels (2) (h) SVM multlevel features 6 levels Water Tree areas Grass areas Roads Shadow Red buldngs Gray buldngs Whte buldng Fgure 7 Thematc maps obtaned by dfferent classfcaton systems appled to the Pava Quckbrd mage. Table 2 and 3 report the thematc accuraces and the geometrc error ndces assocated wth the obtaned maps, respectvely. Consderng the eght dfferent maps, we can easly observe that, as expected, thematc maps obtaned by pxel-based classfcaton approaches [maps (a)-(b)-(d)] are less accurate than those obtaned by context-based approaches. Ths general behavor s clearly 23

24 ponted out also by thematc accuracy ndces. The geometrc error measurements gve us mportant addtonal nformaton about the dfferent propertes of the maps. In partcular, we note that maps obtaned by pxel-based approaches are generally more over-segmented and fragmented than the maps obtaned by context-based classfcaton systems, but they have also the mportant property to be less under-segmented. In the consdered scene, we can observe that the buldngs are very close each others. Thus, most of the consdered classfers merge regons assocated to dstnct objects (.e., buldngs) nto a sngle regon. The aforementoned problem s captured by the proposed geometrc ndces, whch ndcate that most of the maps have an under-segmentaton error that s hgher than the over-segmentaton error [except for map (a)]. Ths problem strongly affects also the recognton of the correct shape of the objects. For ths reason, we can observe that on ths data set the shape error s hghly correlated wth the under-segmentaton error. We can further observe that the edge locaton error s n general qute hgh for all obtaned maps (even f a tolerance of 3 pxels s consdered). Ths ndcates that the consdered classfcaton technques can scarcely model the correct borders of the objects. Table 2 Thematc accuraces computed on the test set on homogeneous areas, edge areas, and on both of them (complete test set) evaluated n terms of Overall Accuracy (OA) and kappa coeffcent (kappa) (Pava data set). Map Complete Test set (homog. + edge areas) Test set on homogeneous Areas Test set on edge Areas OA% kappa OA% kappa OA% kappa (a) Parallelepped 63.6% % % (b) ML 83.0% % % (c) ML post-processng 84.4% % % (d) SVM 84.2% % % (e) SVM Gaussan Pyramd 86.3% % % (f) SVM Multlevel 5 levels (1) 90.0% % % (g) SVM Multlevel 5 levels (2) 88.9% % % (h) SVM Multlevel 6 levels 89.3% % %

25 Table 3 Geometrc error ndces (Pava data set). Undersegmentaton Oversegmentaton Map Edge locaton Fragmentaton Shape (a) Parallelepped 26.9 % 44.6 % 77.4 % 27.6 % 13.8 % (b) ML 26.2 % 9.7 % 66.9 % 9.4 % 14.5 % (c) ML post-processng 30.2 % 8.5 % 68.0 % 8.2 % 16.2 % (d) SVM 16.9 % 12.7 % 58.4 % 7.4 % 12.9 % (e) SVM Gaussan Pyramd 29.3 % 6.3 % 64.2 % 4.7 % 16.8 % (f) SVM Multlevel 5 levels (1) 47.6 % 4.1 % 74.1 % 3.1 % 24.8 % (g) SVM Multlevel 5 levels (2) 26.8 % 6.2 % 62.4 % 5.9 % 19.2 % (h) SVM Multlevel 6 levels 27.1 % 4.8 % 58.5 % 4.3 % 17.0 % Analyzng the sngle maps, we can observe that map (a) has very low qualty n terms of thematc accuracy and n terms of most of the geometrc ndces. In partcular, ths map s sharply over-segmented and fragmented as ndcated by the geometrc errors; ths s confrmed by a vsual nspecton. Map (b) has better qualty than map (a): t exhbts hgher thematc accuracy (both on homogeneous and border areas) and better geometrc propertes n terms of under-segmentaton and border error. Map (c) [obtaned by a post-processng appled to map (b)] results n slghtly hgher thematc accuracy, and n smaller over-segmentaton, fragmentaton and border errors than map (b). Nevertheless, the majorty post-processng leads to slghtly ncrease the undersegmentaton error. Map (d) s the most accurate among those obtaned wth a pxel-based approach: ths s ponted out by both thematc and geometrc ndces. In partcular, ths map exhbts the smallest under-segmentaton and edge locaton errors among all consdered maps. Map (e) exhbts mportant advantages wth respect to the aforementoned maps, showng smaller over-segmentaton and fragmentaton errors as well as hgher thematc accuraces. Nevertheless, the thematc accuraces (especally on border areas) are smaller than those of maps (f)-(g)-(h). The geometrc ndces result partcularly mportant for the characterzaton of the dfferent maps obtaned by the multlevel feature-extracton technque [maps (f)-(g)-(h)], whch have hgh and very smlar thematc accuraces. Map (f) s the most accurate from a thematc pont of vew, but 25

26 maps (g) and (h) exhbt better geometrc characterstcs (e.g., under-segmentaton and edge locaton error) than map (f). As t s possble to observe n Fgure 8, map (f) s affected by undersegmentaton problems, as t merges dfferent objects n the same regon. On the contrary, map (h) correctly models the dfferent buldngs. Ths dfference s clearly ponted out by the values of the under-segmentaton error. Thus, consderng both thematc and geometrc ndces, we can select map (h) as more relable than map (f) (whch would be preferred consderng only thematc accuraces) because t presents a better tradeoff among dfferent propertes of the maps. It s worth notng that the property of correctly recognzng and dstngushng sngle objects n the scene can be very mportant for urban area analyss, especally n applcatons lke buldng detecton. In general, the selecton of the hghest qualty map depends on the knd of applcaton and/or on end-user requrements. In ths context, the proposed ndces are a valuable tool that can drve the selecton of the best thematc map n accordance to the applcaton constrants. (a) Detal of map (f) (b) Detal of map (h) 26

27 (c) Detal of map (f) (d) Detal of map (h) Fgure 8 Detals of the thematc maps: (a)-(c) under-segmentaton problems n map (f); (b)-(d) correct recognton of dstnct buldngs n map (h) (Pava data set). B. Mult-objectve strategy for the model selecton of supervsed algorthms In the second set of experments, we used the proposed mult-objectve technque for the model selecton of a support vector machne (SVM) classfer wth radal bass functon (RBF) Gaussan kernels [23], [24]. The free parameters of the classfer are the regularzaton term C and 2 the spread σ of the Gaussan kernel. The experments were carred out on a VHR mage acqured by the Quckbrd multspectral scanner on an urban area n the south of the cty of Trento (Italy), on July 2006 (see Fgure 6a). We used a panchromatc mage and a pan-sharpened multspectral mage obtaned by applyng a Gram Schmdt fuson technque to the panchromatc channel and to the four bands of the multspectral mage. The mage sze s pxels wth a spatal resoluton of 0.7 m. From the panchromatc and pan-sharpened multspectral bands we extracted textural features by applyng an occurrence flter wth 5 5 wndow sze and computng mean, data range, and varance. Thus, the fnal feature vector s made up of 20 features (5 spectral features and 15 textural features). The avalable set of reference samples ncluded a tranng set, a test set on homogeneous areas, and test set on border areas. Sx classes were consdered: 1) roads, 2) red buldngs, 3) dark buldngs, 4) brght buldngs, 5) shadow, and 6) vegetaton. Table 4 presents the number of labeled reference samples for each set and class. Fgure 6b shows the map of the 11 reference objects used for the evaluaton of the geometrc error ndces. 27

28 (a) (b) Fgure 9 (a) Real color composton of the multspectral mage acqured by the Quckbrd satellte on the cty of Trento (northern Italy). (b) Map of reference objects. Table 4 - Number of samples n the tranng and test sets (Trento data set) Number of patterns Class Test set on Test set on Tranng set edge areas homogeneous areas Roads Red roof buldngs Dark roof buldngs Brght roof buldngs Shadow Vegetaton TOTAL The strategy for the model selecton proposed n secton IV can be appled consderng dfferent sets of thematc and geometrc ndces as objectves of the optmzaton problem, dependng on the specfc applcaton. In our analyss, we performed two sets of experments consderng: ) seven objectves (two thematc and fve geometrc error ndces), ) two objectves (one thematc and one geometrc error ndces). These two sets of experments represent examples of the practcal use of the proposed multobjectve approach n real problems, but any other 28

29 combnaton of thematc and geometrc ndces may be used n the optmzaton problem for the parameter tunng. ) Experments wth seven error ndces n the optmzaton problem In ths set of experments we defned the model selecton as a mult-objectve optmzaton problem made up of seven objectves: the fve geometrc measures presented n secton III (.e., under-segmentaton, over-segmentaton, edge locaton, fragmentaton, and shape errors) and the two thematc errors based on kappa coeffcent (calculated as 1-kappa) on the homogeneous and border test sets. Please note that n our expermental analyss we used a thematc error ndex based on the popular kappa coeffcent, but any other ndex may be used n ts place (e.g., the overall error). For the estmaton of the Pareto-optmal solutons, we adopted a genetc mult-objectve algorthm (a varaton of NSGA-II) [25]. The populaton sze was set to 30 and the maxmum number of generaton to 20. Among all Pareto-optmal solutons obtaned by the genetc algorthm we selected seven solutons (used as an example n ths dscusson), characterzed by dfferent tradeoffs among the dfferent ndces (see Table 5). The selected solutons are characterzed by the lowest error among all solutons for each ndex [e.g., map (2a) presents the hghest thematc accuracy on homogeneous areas, map (2b) exhbts the hghest thematc accuracy on edge areas, map (2c) exhbts the mnmum under-segmentaton error, etc.]. 29

30 Table 5 Thematc and geometrc accuracy/error ndces of seven solutons selected among all Pareto-optmal ponts estmated by the genetc algorthm. Each selected soluton exhbt an accuracy ndex that has the hghest value among all solutons (experments wth seven error ndces n the optmzaton problem) Thematc SVM parameters Geometrc errors accuraces Map 2 kappa kappa Undersegmentatosegmentaton locaton Over- Edge C 2σ Fragm. Shape homog. edge (2a) % 21.9% 51.3% 13.3% 14.7% (2b) % 25.5% 59.4% 15.4% 14.0% (2c) % 29.9% 56.6% 13.1% 11.2% (2d) % 19.6% 63.6% 12.4% 16.8% (2e) % 25.2% 50.0% 15. 5% 13.9% (2f) % 21.4% 59.4% 10.5% 18.4% (2g) % 30.2% 52.4% 12.7% 11.0% All these Pareto optmal solutons are assocated wth maps havng dfferent thematc and geometrc propertes. For example, Fgure 10 shows some detals of the maps (2a)-(2g) and (2c)- (2d). Map (2a) (Fgure 10a) exhbts the hghest kappa coeffcent of accuracy on homogeneous areas, but the shape of red-roof buldngs s not well recognzed. On the contrary, map (2g) (Fgure 10b) has a smaller thematc accuracy, but better models the shape of the buldngs. Ths behavor can also be observed by a vsual nspecton of the maps. Map (2c) (Fgure 10c) has the lowest under-segmentaton error, whereas map (2d) (Fgure 10d) has good over-segmentaton propertes, n spte of sgnfcant under-segmentaton errors (whch also affect the recognton of the shape of the objects). 30

31 (a) Detal of map (2a) (b) Detal of map (2g) (c) Detal of map (2c) (d) Detal of map (2d) Fgure 10 Detals of maps assocated wth dfferent Pareto-optmal solutons (experments wth seven error ndces n the optmzaton problem, Trento data set). ) Experments wth two error ndces n the optmzaton problem In ths second set of experments, two objectves were consdered n the optmzaton problem: 1) the kappa coeffcent of accuracy on homogeneous areas, and 2) the undersegmentaton error. Ths represents an example n whch we would lke to select the SVM model that results n classfcaton maps wth the best tradeoff among thematc accuracy and precson n detectng separate buldngs (under-segmentaton error). The genetc algorthm adopted for the estmaton of the Pareto front resulted n the estmaton of the ten optmal solutons reported n Table 6. 31

32 Table 6 Pareto-optmal solutons estmated by the genetc algorthm for the experment wth two error ndces (Trento data set) Map SVM parameters Error ndces 2 C 2σ 1-kappa (homog. areas) Under-segmentaton error (3a) % 9.95% (3b) % 9.43% (3c) % 10.36% (3d) % 13.23% (3e) % 6.86% (3f) % 7.89% (3g) % 13.62% (3h) % 13.49% (3) % 7.19% (3l) % 7.77% Fgure 11 shows the estmated Pareto front. The selecton of one model for the SVM classfer (.e., the values of C and 2 2σ ) depends on the requrements of the specfc applcaton. For example, we selected three possble models from the Pareto-optmal solutons that leads to: ) the map wth the hghest kappa coeffcent of accuracy on homogeneous areas [map (3g)], ) the map wth the lowest under-segmentaton error [map (3e)], ) a good tradeoff between the two competng objectves [map (3c)]. A qualtatve vsual analyss of the obtaned maps confrms that map (3g) (Fgure 11a) has some under-segmentaton problems (but t has the smallest possble under-segmentaton error for the obtaned kappa value), map (3e) (Fgure 11b) s less undersegmented (and exhbts the hghest possble kappa accuracy for the value of the obtaned undersegmentaton error), and map (3c) (Fgure 11c) can be consdered a good tradeoff between the two consdered objectves. 32

33 under-segmentaton error kappa Fgure 11 Estmated Pareto-optmal solutons for the experment wth two error ndces n the optmzaton problem. (a) Detal of map (3g) (b) Detal of map (3c) (c) Detal of map (3e) Fgure 12 - Detals of the maps assocated wth the three selected solutons (experment wth two error ndces n the optmzaton problem, Trento data set). It s worth notng that dfferent error ndces can be ncluded n the mult-objectve model selecton. The choce of the error ndces should reflect the propertes that the end-users desre to optmze n the classfcaton map. Other experments, carred out usng dfferent error ndces, confrmed the relablty of the proposed mult-objectve model-selecton technque based on the proposed accuracy assessment protocol. 33

34 VI. DISCUSSION AND CONCLUSION In ths paper a novel protocol for the accuracy assessment of thematc maps obtaned by the classfcaton of VHR mages has been presented. The proposed protocol s based on the evaluaton of a set of error measures that can model the thematc and geometrc propertes of the obtaned map. In partcular, we presented a set of ndces that characterze fve dfferent types of geometrc errors n the classfcaton map: 1) over-segmentaton, 2) under-segmentaton, 3) edge locaton, 4) shape dstorton, and 5) fragmentaton. The proposed geometrc measures can be jontly used wth the tradtonal thematc accuracy measures for a precse characterzaton of the propertes of a thematc map derved by VHR mages. The presented protocol can be used n three dfferent frameworks: ) assessng the qualty of a classfcaton map n an automatc, objectve, and quanttatve way; ) selectng the classfcaton map, among a set of dfferent maps, that s more approprate for the specfc applcaton on the bass of user-defned requrements; or ) selectng the values of the free parameters of a supervsed classfcaton algorthm that result n the most approprate classfcaton map. Regardng ths latter pont, we have ntroduced a new technque for tunng the free parameters of supervsed classfers that s based on the optmzaton of a mult-objectve problem, whch results n parameter values that jontly optmze thematc and geometrc error ndces on the classfcaton map. Expermental results, obtaned on two VHR mages, confrms that the proposed geometrc ndces can accurately characterze the propertes of classfcaton maps, provdng objectve and quanttatve error measures, whch are n agreement wth the observatons derved by a vsual nspecton of the consdered maps. Moreover, the proposed approach for tunng the free parameters of supervsed classfers resulted effectve n the selecton of the free parameters of SVM classfers. Ths approach allows one to better characterze the tradeoff among the dfferent thematc and geometrc ndces and to select the model n accordance wth user requrements and applcaton constrants. 34