A DECISION LEVEL FUSION METHOD FOR OBJECT RECOGNITION USING MULTI- ANGULAR IMAGERY

Transcription

1 A DECISION LEVEL FUSION METHOD FOR OBJECT RECOGNITION USING MULTI- ANGULAR IMAGERY F. Tabb Mahmoud a, *, F. Samadzadegan a, P. Renartz b a Dept. of Surveyng and Geomatcs, College of Engneerng, Unversty of Tehran, Tehran, Iran- (fmahmoud,samadz@ut.ac.r b Dept. of Photogrammetry and Image Analyss, Remote Sensng Technology Insttute, German Aerospace Centre (DLR, Oberpfaffenhofen Weßlng, Germany- peter.renartz@dlr.de KEY WORDS: Object Recognton, Decson Level Fuson, Vsblty Map, Shadow Recovery, Texture, Weghtng Strategy ABSTRACT: Spectral smlarty and spatal adjacency between varous nds of objects, shadow and occluded areas behnd hgh rse objects as well as complex relatonshps lead to object recognton dffcultes and ambgutes n complex urban areas. Usng new multangular satellte magery, hgher levels of analyss and developng a context aware system may mprove object recognton results n these stuatons. In ths paper, the capablty of mult-angular satellte magery s used n order to solve object recognton dffcultes n complex urban areas based on decson level fuson of Object Based Image Analyss (OBIA. The proposed methodology has two man stages. In the frst stage, object based mage analyss s performed ndependently on each of the mult-angular mages. Then, n the second stage, the ntal classfed regons of each ndvdual mult-angular mage are fused through a decson level fuson based on the defnton of scene context. Evaluaton of the capabltes of the proposed methodology s performed on mult-angular WorldVew-2 satellte magery over Ro de Janero (Brazl.The obtaned results represent several advantages of mult-angular magery wth respect to a sngle shot dataset. Together wth the capabltes of the proposed decson level fuson method, most of the object recognton dffcultes and ambgutes are decreased and the overall accuracy and the appa values are mproved. 1. INTRODUCTION The complex nature and dverse composton of land cover types found wthn urban areas exhbt several dffcultes n producng large scale topographc maps from VHR satellte magery (Shacelford and Davs, Ths stuaton also potentally leads to lower accuracy n pxel based mage classfcaton approaches whch only use spectral characterstcs of remotely sensed data (Mynt et al., 2011; Shacelford and Davs, 2003; Zhou and Troy, 2008; Blasche, 2010; Duro et al., As many land cover classes n an urban envronment have smlar spectral sgnatures, spatal nformaton such as heght and topologcal relatonshps must be exploted to produce accurate classfcaton maps. Already many researchers have nvestgated the potental of the object based mage analyss (OBIA approaches for dealng wth VHR mages and complextes n urban areas (Mynt et al., 2011; Zhou and Troy, 2008; Blasche, 2010; Peets and Etzon, 2010; Jacqun et al., 2008; Lalberte et al., As t s depcted n prevous researches n the feld of OBIA, the accuracy of object recognton results n complex urban areas drectly depends on the segmentaton and rule based classfcaton processes (Ivts et al., 2005; Platt and Rapoza, 2008; Zhou and Troy, 2008; Mynt et al., 2011; Lalberte et al., Moreover, dependng on the vewng angle of the sensor, some parts of urban objects may be occluded by hgh rse objects such as buldngs or trees (Habb et al., 2007; Bang et al., Therefore, usng only sngle shot magery, t s very dffcult to obtan valuable object recognton results. On the other hand, stereo magery (from just two vewng drectons may partally mprove object recognton results, but stll there mght be mssng nformaton for fllng up occluded areas. The unque aglty and collecton capacty of some modern satellte sensors such as WorldVew-2 provde mage sequences of a sngle target from many observaton angles wthn one satellte orbt. Fuson of the nformaton comng from multangular magery s valuable for fllng up occluded areas and obtanng relable object recognton results. Therefore, usng mult-angular magery together wth contextual nformaton and hgher levels of modfcaton should be utlzed n order to mprove object recognton results n complex urban areas based on VHR satellte mages. In ths paper a context aware system s proposed for decson level fuson of the object recognton results and solvng problems arsng from occlusons and shadow areas, based on the mult-angular VHR remotely sensed data. 2. PROPOSED METHOD As depcted n fgure 1, the proposed object recognton method s composed of two man stages: sngle vew and mult vews processes. In the frst stage, regon's nternal contextual nformaton s utlzed for object classfcaton on each of the ndvdual mages. Per segment spectral and textural pxels' nteractons together wth structural features based on sze, shape and heght of a segmented regon, generate regon's nternal context. Ths nd of nformaton s utlzed for object based mage analyss composed of mage segmentaton and object classfcaton on each of the ndvdual mages. Second stage of the proposed * Correspondng author. Ths contrbuton has been peer-revewed. The peer-revew was conducted on the bass of the abstract. 409

2 methodology performs decson level fuson on the mult vew classfed regons based on scene context n order to reduce ambgutes and uncertantes n the generated classfcaton map Textural Features: the grey value relatonshps between each pxel and ts neghbours n the pre-dentfed segmented regons. N 1 N 1 2 Energy I(, j (3 0 j 0 N 1 N 1 Entropy I(, jlog(i(, j (4 0 j 0 where Energy & Entropy = Samples of textural features I (, j =value of mage pxel (,j N=neghbour sze n the segmented regon Fgure 1. General structure of the proposed object recognton method 2.1 Processng Based on Sngle Vew Object based mage analyss requres generatng segmented regons as classfcaton unts. In ths research, mult-resoluton segmentaton technque s appled on the content of each of the ndvdual mages n order to segment t nto regons. The multresoluton segmentaton procedure starts wth sngle mage objects of one pxel and repeatedly merges a par of mage objects nto larger ones. The mergng decson s based on the local homogenety crteron, descrbng the smlarty between adjacent mage objects (Baatz and Schape, After performng segmentaton, a nowledge based classfcaton process should be performed on each of the segmented regons. Therefore, t s necessary to gather proper nowledge composed of per-segment nteractons between pxels and structural characterstcs of each segmented regon n order to provde regon's nternal context. Followng tems represent some sample of these features wth ther basc mathematcs: Spectral Features: the ratos between reflectance values of varous spectral bands generate normalzed dfference ndces (NDI and smple ratos (SR. NDI,l,l (Band Band (Band Band (1 l l SR Band Band (2 where NDI, l = normalzed dfference ndex between bands & l SR,l =smple rato between bands & l Band=mage channel l Structural Features: Calculatng sutable structural features based on spatal characterstcs and heghts of segmented regons provde another part of nternal context for usng n the object classfcaton process. In ths paper, 2D structural features such as area, elongaton and soldty are used together wth relef and surface roughness as 3D structural features. Elongaton MajorLengt h MnorLengt h (5 Soldty Area ConvexArea (6 where Major & Mnor Lengths = Major & mnor axes of the boundng polygon Convex area= Area of the smallest convex polygon that can contan the regon After generaton of the above mentoned spectral, textural and structural features based on the mage data and heght products, an optmum selecton of features for recognzng each ndvdual object classes has the most effectve role n generatng a rch nowledge base of regon's nternal context. The object classfcaton can be performed by encapsulatng the nowledge base contanng regon's nternal context, nto a rule set and defnton of a strategy for object recognton. The proposed strategy s a mult-process classfcaton model that s a progressve process composed of multple steps. In the frst step, the entre segmented mage dataset s classfed based on spectral reasonng rules. Then, n the next steps, classfed objects are modfed based on textural and structural reasonng rules, respectvely (Tabb Mahmoud et al., Despte the hgh potental of the strateges n OBIA, wrong classfed regons due to occluson and shadow through hgh rse 3D objects, decrease the relablty of the object recognton results. Therefore, usng the proposed decson level fuson of the mult vews object based mage analyss can handle these dffcultes n an effcent way. 2.2 Processng Based on Mult Vews Hgh rse objects together wth vewng angle of the sensor mae some uncertantes n the object classfcaton results whch cannot be solved usng only sngle shot magery. Therefore, n the proposed object recognton methodology another processng stage s defned based on the context aware decson level fuson of the object based mage analyss on mult vews. Ths stage s composed of two man operatons; pre-analyss on object classfcaton results for all mages and the decson level fuson of the mult vews. Ths contrbuton has been peer-revewed. The peer-revew was conducted on the bass of the abstract. 410

3 2.2.1 Pre-Analyss: Total vsblty map generated from all of the ndvdual vsblty maps provdes a man tool for performng vsblty analyss. Vsblty analyss determnes the number of vews for all, pxel (,j of the ground space are vsble. Performng vsblty analyss as a pre-analyss operaton, one can categorze all of the ground space pxels nto three groups: vsble n all mages, vsble n some mages and vsble n none of the (occluded n all mages. Ths categorzaton s useful for the defnton of scene context n the proposed decson level fuson algorthm Decson level fuson of mult vews: The frst step of the decson level fuson s the bac projecton from each ground space pxel to ts pre-dentfed vsble mages (all mages for the group of vsble n none based on the results of the pre-analyss operaton. These bac projectons fnd the objects and sub-objects n mult vews those pxels belong to. Therefore, there are three groups of areas contanng varous object types and ther sub objects n mult vews. As scene context of an object can be defned n terms of ts cooccurrences wth other objects and ts occurrences n the whole scene, nformaton regardng sensor's loo angle, dstance from occluded areas, heghts and areas of sub-objects those pxels belong to, are the fundamentals for scene context defnton. In ths research, scene contextual nformaton s utlzed for weghtng all class objects n order to assgn them to the ground space pxels. So, ths proposed decson fuson strategy taes place n the level of the pre-defned object classes based on the object classfcaton results. Therefore, f there are n varous recognzable object classes n mult vews, also n dfferent weghts should be calculated for assgnng each of the object classes to the ground space pxels. Weghtng class objects hghly depends on the scene context and ground space categorzaton as a result of vsblty analyss. Followng tems represent the proposed weghtng strategy based on scene context: 1. Off-nadr weght depends on the sensor nformaton such as off-nadr vewng angle for each of the mult vew mages. Classfed regons n the nearest to nadr vews have largest off-nadr weghts. 2. Weght of occluson, depends on the longest dstances between each object regon and ts neghbourng occluded areas n the mult vew mages. More dstance from occluded areas leads to a larger weght of occluson. 3. Weght of structural features, depends on the spatal nformaton such as the area of the object regon. For nstance, f pxel (x,y belongs to a large sub-object, the weght of structural features becomes larger. 4. Weght of topologcal relatonshps, depends on the neghbourng relatonshps between occluded regons and ther nearest vsble object regons. Determnng the weght of topologcal relatonshp for an occluded sub-object n all vews depends on ts neghbourng vsble objects wth smallest heght dfferences. In other word, f there s small amount of heght dfference between two neghbourng object regons, class label of vsble regon assgns to the occluded one. For ground space pxels those are categorzed n the groups of vsble n all or vsble n some mages, calculatng the weght of scene contextual nformaton for each object class s based on the summaton of off-nadr, occluson and structural weghts n ther vsble vews. For ground space pxels those are vsble n none of the mages, the summaton of off-nadr weght and weght of topologcal relatonshps n all vews s calculated as the weght of scene context for each object class. In addton, classfcaton accuraces of varous object types n each vew also affect on the decson fuson results. Weghts of object classes are determned based on the user and producer accuraces n each of the mult vews object based mage analyss. 2A U(Class A P(Class W (Class (7 AU(Class AP(Class where W (Class = weght of classfcaton for object class n vew A =User accuracy of object class n vew U(Class A =Producer accuracy of object class n P(Class vew Therefore, total weght of each object class composed of weghts of scene context and classfcaton n varous vews. The object class wth the largest weght should be selected as the wnner class label for each of the ground space pxels. If the wnner class s shadow, structural and heght based relatons are used n order to determne true object types nstead of shadow area. 3. EXPERIMENTS AND RESULTS The potental of the proposed object recognton methodology s evaluated for automatc object recognton n mult-angular WorldVew-2 satellte magery over Ro de Janero (Brazl whch were collected n January 2010 wthn a three mnutes tme frame wth satellte elevaton angles of 44.7 and 56.0 n the forward drecton, and 59.8 and 44.6 n the bacward drecton, as shown n Fgure 2. The mult-angular sequence contans the downtown area of the cty, ncludng a number of large and hgh buldngs, commercal and ndustral structures, and a mxture of communty pars and prvate housng. Moreover, usng mult-angular WorldVew-2 magery the DSM s generated from multple pars of panchromatc stereo mages n eppolar geometry usng the Sem-Global Matchng (SGM algorthm (Hrschmüller 2008; d Angelo et al., 2010; Srmace et al., In order to obtan OBIA results based on mult-angular mages, mult-resoluton segmentaton algorthm s appled on each of the ndependent satellte mages usng ecognton software wth the values 90, 0.2 and 0.1 for the scale parameter, compactness and shape parameters, respectvely. Then, varous spectral, textural and structural features are measured on mage regons for the generaton of the nowledge base contanng nternal context and performng object level classfcaton on segmented regons. Buldng, road, tree, grass land and shadow area are the predentfed object classes based on vsual nspectons. Despte shadow s not a real object class, detectng real objects under shadow areas based on spectral responses s a dffcult tas dealng wth VHR magery. Therefore, n ths paper shadow s frst recognzed as separate object class and n a later step, we are gong to recover shadow areas based on decson level fuson of topologcal relatonshps n mult-angular vews. Before decson level fuson of mult-angular mages, by performng vsblty analyss on total vsblty map, ground space categorzed nto three groups: pxels are vsble n Ths contrbuton has been peer-revewed. The peer-revew was conducted on the bass of the abstract. 411

4 a b a b c d c d e Fgure 2. Dataset, a WV-2 magery wth 56º satellte elevaton angle, b WV-2 magery wth 44.7º satellte elevaton angle, c WV-2 magery wth 59.8ºsatellte elevaton angle, d WV-2 magery wth 44.6º satellte elevaton angle, e Dgtal Surface Model (Ground space all mages, pxels are vsble n some mages and just pxels are vsble n none of the (occluded n all mages. After performng decson level fuson on the OBIAs based on the proposed context aware strategy, analyss represents that decson level fuson removes shadow from object recognton results and detects road regons occluded by hgh rse buldngs especally n pxels whch are vsble n all or some mages. Fgure 3 compares the object based mage analyss from each of the mult-angular mages wth decson level fuson of them. Accordng to the obtaned results, despte the hgh potental of the utlzed regons' nternal context, hgh rse buldngs together wth small elevaton angles of the sensor ncrease dffcultes n object recognton n urban areas. However, usng scene context and classfcaton accuraces n the proposed decson level fuson algorthm can mprove the classfcaton results. Moreover, for the quanttatve evaluaton of the results from decson fuson system, some areas of the pre-defned object classes are manually selected by an expert operator on 3D model generated from mult-angular scenes (Fgure 4. Sample areas are then compared wth ther correspondng results of dfferent steps of the proposed object recognton algorthm. As depcted n table 1, the comparson s based on the number of correctly detected pxels (true postve, wrongly detected pxels (false postve, and the not correctly recognzed pxels (false negatve, determned after performng the object recognton algorthm. After that, usng quanttatve values for each object class, completeness, correctness and qualty crtera are determned for the obtaned results (Tabb Mahmoud et al e Fgure 3. Results of the object based mage analyss and decson level fuson of them, a OBIA of WV-2 magery wth56º satellte elevaton angle, b OBIA of WV-2 magery wth 44.7º satellte elevaton angle, c OBIA of WV-2 magery wth 59.8ºsatellte elevaton angle, d OBIA of WV-2 magery wth 44.6º satellte elevaton angle, e decson level fuson result Fgure 4. Manually selected sample regons Table 1. Accuracy assessment of the obtaned results from the proposed decson fuson algorthm Decson Level Object Classes Fuson Results Buldng Road Tree Grass True Postve False Postve False Negatve Completeness Correctness Qualty Overall Accuracy Kappa Ths contrbuton has been peer-revewed. The peer-revew was conducted on the bass of the abstract. 412

5 In order to perform more quanttatve analyss on the results, object based mage analyss of each of the mult-angular mages are compared wthn sample regons and then, ther overall accuraces and appa are compared wth ther decson level fuson results. As t s depcted n table 2, the largest values of overall accuracy and appa belong to the vews wth largest elevaton angles and by decreasng the elevaton angles; overall accuraces and appa are decreasng. Comparng tables 1 & 2 wth each other shows that usng proposed algorthm for decson level fuson of mult-angular mages ncrease the amount of overall accuracy and appa values n classfcaton results. Table 2. Accuraces of mult-angular object based mage analyss Elevaton Angles Overall Accuracy Vew 1 Vew 2 Vew 3 Vew 4 56º 44.7º 59.8º 44.6º Kappa Fgure 5 llustrates consderable mprovement n qualty of context aware decson level fuson results wth respect to the object based mage analyss of each of the mult-angular mages. Fgure 5. Comparson between the qualty of Mult-angular OBIAs and ther Decson level fuson results 4. CONCLUSION A context aware strategy s proposed for decson level fuson of the object based mage analyss results based on mult-angular WorldVew-2 satellte magery. Accordng to the varous elevaton angles of the sensors, hgh rse 3D objects such as buldngs may cause occluson and shadow areas n the VHR remotely sensed magery. In such a stuaton, true class labels of some parts of object regons cannot be detected. So, large number of false postve and false negatve pxels decreases the classfcaton accuraces. Usng the proposed decson level fuson method of mult-angular magery based on the developed context aware system mproves the classfcaton accuracy. Results are shown that vsblty analyss on class labels before and after fuson and performng decson level fuson on the OBIA results of mult-angular mages reduce the amount of wrongly classfed pxels for 3D object classes and reveals mproved class labellng of the occluded areas. Ths method stll needs further modfcatons n the feld of defnton of contextual nformaton. Moreover, ncorporatng a Ldar DSM and mult-angular satellte or aeral mages can be tested for mprovement of object recognton results n complex urban areas. 5. REFERENCES Baatz, M.,Schape, A., Mult-resoluton segmentaton: an optmzaton approach for hgh qualty mult-scale mage segmentaton.in Angewandte Geographsche Informatonsverabetung. XII. Betragezum AGIT-Symp. Salzburg, T. Strobl, T. Blasche and G. Gresebner (Eds, Bang, K.I., Habb, A.F., Km, C., Shn, S., 2007.Comprehensve Analyss of Alternatve Methodologes for True Orthophoto Generaton From Hgh Resoluton Satellte and Aeral Imagery.Amercan Socety for Photogrammetry and Remote Sensng, Annual Conference, Tampa, Florda, USA. Blasche, T., Object based mage analyss for remote sensng. ISPRS Internatonal Journal of Photogrammetry and Remote Sensng 65(1, d Angelo, P., Image matchng and outler removal for large scale DSM generaton. Convergence n Geomatcs. CGC & ISPRS. ISPRS Symposum Commsson I, June 2010, Calgary, Canada, on CD-ROM. Duro, D.C., Franln, S.E., Dubé, M.G., A comparson of pxel-based and object-based mage analyss wth selected machne learnng algorthms for the classfcaton of agrcultural landscapes usng SPOT-5 HRG magery. Remote Sensng of Envronment 118 ( Habb, A.F., Km, E., Km, C., New methodologes for true ortho-photo generaton. Photogrammetrc Engneerng and Remote Sensng 73(1, Hrschmüller, H., Stereo Processng by Sem global Matchng and Mutual Informaton, IEEE Transactons on Pattern Analyss and Machne Intellgence, 30 (2, Ivts, E., Koch, B., Blasche, T., Jochum, M., Adler, P., Landscape structure assessment wth mage grey-values and object-based classfcaton at three spatal resolutons. Internatonal Journal of Remote Sensng 26(14, Jacqun, A., Msaova, L., Gay, M A hybrd objectbased classfcaton approach for mappng urban sprawl n perurban envronment. Landscape and Urban Plannng 84 (2008, Lalberte, A.S., Brownng, D.M., Rango, A., A comparson of three feature selecton methods for object-based classfcaton of sub-decmeter resoluton UltraCam-L magery. Internatonal Journal of Appled Earth Observaton and Geonformaton 15 ( Mynt, S.W., Gober, P., Brazel, A., Grossman-Clare, S., Weng, Q., Per-pxel vs. object-based classfcaton of urban land cover extracton usng hgh spatal resoluton magery. Remote Sensng of Envronment 115(5, Ths contrbuton has been peer-revewed. The peer-revew was conducted on the bass of the abstract. 413

6 Peets, A., Etzon, Y., Automated recognton of urban objects and ther morphologcal attrbutes usng GIS.ISPRS Archve XXXVIII (4-8-2-W9, "Core Spatal Databases- Updatng, Mantenance and servces-from theory to practce" Hafa, Israel. Platt, R.V., Rapoza, L., An Evaluaton of an Object- Orented Paradgm for Land Use/Land Cover Classfcaton. The Professonal Geographer60(1, Shacelford, A.K., Davs, C.H., A Herarchcal Fuzzy Classfcaton Approach for Hgh-Resoluton Multspectral Data over Urban Areas. IEEE Transacton on Geoscences and Remote Sensng 41(9, Srmace, B., Taubenböc, H., Renartz, P., Ehlers, M Evaluaton of automatcally generated 3-D cty models based on sx dfferent DSMs from arborne and space-borne sensors. IEEE Journal of Selected Topcs n Appled Earth Observatons and Remote Sensng, 5(1, TabbMahmoud, F., Samadzadegan, F., Renartz, P., Object orented mage analyss based on mult-agent recognton system.can.j. Computers & Geoscences 54 ( Zhou, W., Troy, A., An Object-Orented Approach for Analyzng and Characterzng Urban Landscape at the Parcel Level. Internatonal Journal of Remote Sensng 29, Ths contrbuton has been peer-revewed. The peer-revew was conducted on the bass of the abstract. 414