Hybrid Neuro-Bayesian Spatial Contextual Reasoning for Scene Content Understanding

Transcription

1 12th Internatonal Conference on Informaton Fuson Seattle, WA, USA, July 6-9, 2009 Hybrd Neuro-Bayesan Spatal Contextual Reasonng for Scene Content Understandng Dens Garagc *, Majd Zandpour *, Frank Stolle *, Matthew Antone, and Bradley J. Rhodes * * Adaptve Reasonng Technologes Drectorate, Fuson Technology and Systems Dvson Sgnal Understandng and Networkng Dvson BAE Systems Advanced Informaton Technologes 6 New England Executve Park Burlngton, MA 01803, USA {dens.garagc, majd.zandpour, frank.stolle, matthew.antone, brad.rhodes}@baesystems.com Abstract Geospatal scene content understandng facltates a large number of ncreasngly mportant applcatons. These range from tools to help ntellgence analysts perform rapd, hgh-precson dentfcaton of urban scene content to other cvlan and mltary securty applcatons such as geospatal queres, functonal object level change detecton, and msson plannng. In ths paper, we present ntal research results from a mult-faceted approach for determnng and understandng scene content. Our approach performs context-dependent probablstc reasonng on a set of object hypotheses obtaned from a sute of ndvdual object detecton algorthms. Ths neurally-nspred reasonng approach mproves the qualty of object detectons wthn a gven scene and enhances scene content understandng by fusng low-level features, dentfed objects, hgh-level context, and spatal constrants to more accurately determne the nature of specfc scene level targets. We present results from applcaton of our hybrd spatal contextual reasonng approach to a set of objects automatcally obtaned from an urban scene by a sute of state-of-the art detecton algorthms. We demonstrate that reasonng on the ndvdual detector outputs produces mproved precson-recall performance over usng the detector outputs alone. Keywords: Scene understandng, complex object recognton, assocatve learnng, Bayesan networks, probablstc reasonng. 1 Introducton Mltary operatons n urban areas requre detaled understandng about the locaton and classfcaton of common objects and spatal features. Learnng spatal context for scene content understandng has been an endurng goal of computer vson researchers snce the late 1960 s [7], and at present there s no sngle approach that can acheve ths objectve. In recent years, the role of contextual nfluences n object recognton has become an mportant topc, due both to the psychologcal bass of context n the human vsual system [6] and to the object recognton algorthmc mprovements that contextual nfluences have provded [12]. One of the most straghtforward forms of representng the context of an object s n terms of ts cooccurrence wth respect to other objects. The work n [10] and [9] demonstrates the use of ths context, where the presence of a certan object class n an mage probablstcally nfluences the presence of a second class. Whle these methods acheve good results when many dfferent object classes are labeled per mage, they are unable to leverage unsupervsed data for contextual object recognton. In addton to co-occurrence context, many approaches take nto account the spatal relatonshps between objects. At the descrptor level, Wolf et al. [13] detect objects usng a descrptor wth a large capture range, allowng the detecton of the object to be nfluenced by surroundng mage features. We have prevously reported a successful approach to learnng contextual nfluences on object recognton and scene understandng [14], [10]. In [14] we proposed a bologcally-nspred algorthm for automatc scene understandng and complex object recognton whch does not requre any handcrafted a pror knowledge. It ncrementally learns, wth or wthout a pror knowledge, the assocatons and nterdependences between compound objects and ther prmtve components. In addton, the spatal relatonshps among the smple consttuents and ther probabltes of par-wse occurrences are learned ncrementally. Based on some of the conclusons drawn from our pror results, we have pursued several enhancements, ncludng the complementary use of assocatve learnng and Bayesan network paradgms to encode the preferences for certan spatal and co-occurrence relatonshps as well as provdng greater capacty for scene nterpretaton and detal examnaton. Ths hybrd contextual reasonng approach mproves the qualty of object detectons wthn a gven scene and enhances scene content understandng, whle offerng all standard benefts of a graphcal model formulaton (e.g., well-known learnng and nference technques) [1], [3]. In ths paper we present a hybrd spatal contextual reasonng approach, and demonstrate ts performance on a set of ndvdual object detectons obtaned from an urban scene by a sute of ISIF 984

2 state-of-the art object detectors. We also report mprovements n precson-recall obtaned when our approach reasons over the raw detector outputs, as compared wth the performance of the ndvdual detectors alone. 2 Methodology In ths secton, we present experments and ntal results from a mult-faceted approach for determnng and understandng scene content. Our reasonng soluton operates on results from state-of-the-art scene segmentaton and object recognton algorthms, combnng ther detectons through complementary use of assocatve learnng and Bayesan networks. Ths neurally-nspred hybrd learnng technque fuses low-level features, dentfed objects, hgh-level context, and spatal constrants to more accurately determne the presence and dentty of specfc scene level objects. The frst component of the proposed contextual reasonng algorthm, Object Probablstc Assocatve Learnng (OPAL) [14], s based on Probablstc Neural Assocatve Incremental Learnng (pnail) [10] and automatcally dscovers the condtonal probabltes and herarchcal structure of elements comprsng a scene. OPAL s output s a belef map of possble scene types based on the set of smple objects present n a gven scene. The second component of our hybrd contextual reasonng algorthm, a Bayesan network, uses ths belef map to ntalze learnng of spatal relatonshps between consttuent objects over the space of possble relatonshp networks usng standard structure learnng algorthms [3]. Snce a Bayesan network consttutes a complete probablstc model of the varables n a doman, the network contans all nformaton needed to answer any probablstc query about these varables. One of the benefts of ths top-down reasonng strategy, whch utlzes scene and spatal context nformaton, s to provde a regularzaton mechansm to reduce ambguty and false postves (msclassfcatons) that arse when classfyng objects or scenes ndvdually. The system terates as nformaton from below and above s propagated through the representaton herarchy and constrant networks as llustrated n Fgure 1. Each pass of the algorthm updates the hypotheses that best explan the complete scene gven the avalable nformaton; thus, queres at any pont durng algorthm teraton wll retreve the best avalable nformaton at that tme. Detals of the learnng and nference modes of our hybrd spatal contextual reasonng approach are presented n sectons 2.1 and Object Probablstc Assocatve Learnng A fundamental role of ths component s to represent objects as herarchcal collectons of features, other objects, and scene context nformaton [14]. To embody the dversty of object classes we need to support, for example, objects as specfc as a stop sgn and as general as a processng plant. In our approach objects are represented as graphs whose nodes are features or other objects, and whose edges represent spatal or other relatonshps between nodes. Node assocatons n the form of lkelhoods are ncrementally learned, wth or wthout a pror knowledge for each labeled object, from a small number of truthed data examples. Fgure 2 llustrates a set of learned node assocatons representng the Gas Staton object as composed of spatal relatonshps wth the Gas Pump, Roof, Sgn, and Small Buldng objects n the context of Urban terran. The output of the assocatve learnng process s a belef map for a set of combned objects ndcatng the lkelhood of possble Fgure 1. Hybrd probablstc reasonng s performed on products from ndvdual detectors that extract locatons and attrbutes of object canddates from raw data. The performance characterstcs of each detector provde addtonal bottom up nputs to the hybrd reasonng engne. As descrbed n the text, OPAL produces assocatve mappngs between objects whch are then used by the Bayesan reasonng to enhance the belef map of the scene. Ths map can also be reasoned upon to provde top-down feedback to adjust the operatng ponts of ndvdual detectors. 985

3 scenes based on the set of smple objects present. For each observaton of a set of smple and scene-level objects, OPAL can make predctons about possble scenes wth weghts ndcatng probabltes of smple objects havng prevously been observed n each scene. However, there may be many non-zero weghts emanatng from the smple object nodes, resultng n a redundant network for whch a closed-form statstcal nference soluton does not exst. In order to overcome ths lmtaton of our OPAL algorthm, we use a graphcal model formulaton (e.g., Bayesan networks) to revse a scene s belef map produced by OPAL. The a posteror map generated by ths network s then used as the fnal scene belef map. 2.2 Bayesan Networks for Spatal Context Belef Refnement To learn context-senstve models of spatal relatonshps among smple objects observed n a scene, we tran a Bayesan network to learn spatal relatonshps between nstances n an object-centrc frame of reference. A Bayesan network [8] encodes the jont probablty dstrbuton of a set of v varables (e.g., objects n a scene) { x1,..., x v }, as a drected acyclc graph and a set of condtonal probablty tables (CPTs). In ths paper we assume all varables are dscrete, or have been predscretzed. Each node corresponds to a varable (.e., an object), and the CPT assocated wth t contans the probablty of each state of the varable gven every possble combnaton of states of ts parents. The set of parents of x, denoted π, s the set of nodes wth an arc to x, n the Features of an Abstract Gas Staton Roof Sgn Car Offce Gas Pumps Scene-level Representaton of a Gas Staton Sgn Roof Vehcle Gas Pump Offce Gas Pump Urban terran Encode Feature/Object Type Dstance Drecton Count/Densty Fgure 2. Hghly varable objects are represented as constellatons of features and sub-objects wth flexble geospatal constrants. Context ncludes nearby objects, features and underlyng terran types; attrbutes nclude feature dstance, drecton and densty. graph. The structure of the network encodes the asserton that each node s condtonally ndependent of ts nondescendants gven ts parents. Thus the probablty of an arbtrary event X = { x1,..., x v } can be computed as P v v ( X ) = P ( x ) (, ) 1 π P O f = = 1. Low-level detectors report as detectons all locatons for whch a condtonal probablty PO (, f ) (.e., a condtonal probablty that th object s detected gven the feature vector, f ) s above a threshold chosen to gve a desred trade-off between false postves and mssed detectons. Because OPAL unrolls nto (e.g., ntalzes the structure and parameters of) a Bayesan network for each scene, we can use standard learnng and nference methods. In partcular, we learn the parameters of our Bayesan network model usng the Expectaton-Maxmzaton (EM) [2] algorthm and perform nference usng a standard varant of Markov Chan Monte Carlo samplng [1]. Furthermore, we learn the set of actve relatonshps from a large canddate relatonshp pool usng a structure search nterleaved wth the EM [2]. At test tme, our system generates the a posteror belef map as the fnal scene belef map. Learnng Bayesan Networks: Gven a tranng set T = { X1,..., Xk,... Xn} where X k = { xk,1,..., xk, v}, the goal of learnng s to fnd the Bayesan network that best represents the jont probablty dstrbuton Px ( k,1,..., x k, v). One approach s to fnd the network W (.e., the set of scene contextual relatonshps) that maxmzes the lkelhood of the data or ts logarthm [3]: n LL ( W T ) log P ( x π ) v =. In ths paper we k= 1 = 1 W k, k, assume no known structure of contextual relatonshps n a scene, and focus on the problem of learnng network structure and parameters. To acheve the best combnaton of accuracy and effcency, we employ hll-clmbng search ntalzed wth a network constructed by OPAL, rather than an empty or random ntal network. At each search step, hll-clmbng creates all feasble varatons of the current network obtaned by addng, deletng, or reversng any sngle arc (.e., probablstc dependences between objects n a scene). The best varaton becomes the new current network, and the process repeats untl no varaton mproves the score. We extend the loglkelhood scorng functon by addng a complexty penalty. For example, we mnmze MDL( W T ) = 0.5m log n LL( W T ) [3], where m s the number of parameters n the network. Our learnng process outputs an actve set of relatonshps and the parameters of our model,.e., learned condtonal probabltes that encode the strengths of the probablstc dependences between objects. 986

4 3 Expermental Results In ths secton we report the performance of our hybrd spatal contextual reasonng algorthm on a set of objects automatcally detected n an urban scene by a sute of state-of-the art recognton algorthms. Our system frst ngests 3D LIDAR pont data and extracts smple features such as color and elevaton statstcs on a multscale 2.5D grd. Layer-based processng on ths grd allows characterzaton and clusterng of large and coarsely-defned regons such as vegetaton, buldngs, pavement, and water, whle raw 3D processng parttons the scene nto domnant surfaces (e.g., walls, rooftops, and ground) and canddate object segments (e.g., compact objects such as vehcles, fre hydrants, and lamp posts). The segments are further analyzed to determne plausble class lkelhoods usng a set of matchng and recognton algorthms specalzed accordng to overall object sze and shape. Contextual nformaton gudes our overall object recognton strategy, conservng and focusng computatonal resources by restrctng both search regons and plausble category lsts. Context feeds down from above (e.g., we search for wndows and doors only on walls and search for alleys among buldngs) and feeds up from below (for example, the presence of pumps, vehcles, and an awnng may ndcate a gas staton). The man objectve of our current expermental work s to assess the performance of our context-dependent probablstc reasonng to reduce ambguty and msclassfcatons caused by the ndvdual object detecton algorthms above. If an object was detected but not assgned a class label, we wsh to determne whether contextual scene knowledge and observed evdence of other objects detected nearby can be used to classfy ths object. Formally, ths can be defned as estmatng a condtonal probablty PO ( = 1 f), where O = 1 ndcates the presence of the th object and f s a set of features extracted from the mage. For tranng and testng we used an urban scene contanng several thousand objects that were manually annotated wth one of 42 category labels. The outputs of local object detectors, n conjuncton wth context-related features, served as nputs to the hybrd reasonng component. Here we compare the performance of two methods for the task of reducng the number of false postves: ) usng low-level ndvdual object detectors; and ) usng our hybrd contextual scene reasonng Fgure 3. Precson-Recall (PR) generated usng Hybrd algorthm (dashed lne) and Indvdual Object Detectors (sold lne); (Top-left) PR curves for Wndow object class; (Top-rght) PR curves for Trees object class; (Bottom-left) PR curves for Street Lght object class; (Bottom-rght) POR curves for Sdewalk object class. 987

5 Fgure 4. Precson-Recall (PR) generated usng Hybrd algorthm (dashed lne) and Indvdual Object Detectors (sold lne); (Top-left) PR curves for Door object class; (Top-rght) PR curves for Curb object class; (Bottom-left) PR curves for Post object class; (Bottom-rght) POR curves for Sgn object class. algorthm. As an alternatve to ROC curves, we summarze these results usng Precson-Recall (PR) curves, whch can expose dfferences between algorthms that are not apparent n ROC space. We plot the PR curves produced by our hybrd algorthm aganst the curves produced by ndvdual object detectors (see Fgures 3 5), and observe that our hybrd contextual scene reasonng algorthm provded an mprovement n accuracy for all but the wndow category. 4 Conclusons An mportant component of hgher level fuson and decson makng s knowledge dscovery. One form of knowledge s a set of relatonshps between enttes and ther lkelhoods of co-occurrence. We developed a hybrd scene contextual reasonng approach that enables scene understandng and complex object recognton. The OPAL component of our contextual reasonng approach s a probablstc assocatve learnng algorthm that automatcally dscovers the condtonal probabltes and herarchcal structure of prmtve objects comprsng a scene. The Bayesan network component learns spatal relatonshps between objects n an object-centrc reference frame. The dscovery of co-occurrence and spatal relatonshps between objects from the truth data wthout a pror knowledge s an mportant characterstc of our approach. Ths knowledge dscovery from data depends on the appearance of scene objects captured n the truth data. Results of our hybrd contextual reasonng approach show that the presence of certan neghborng objects wthn a learned vcnty of a canddate object n a scene narrows the possble class set and enhances correct predctons. In addton, we demonstrated mprovements n object detecton accuracy obtaned when our approach reasons over the ndvdual detector outputs compared to the accuracy obtaned when only outputs from the ndvdual detectors are used. One mmedate future task s to apply ths reasonng algorthm to the detecton and classfcaton of complex objects wth many common features. For nstance, a street sgn, a traffc sgn, and a lamppost all have a thn cylndrcal core wth structures at the top and can lead low-level detecton algorthms to produce ambguous labels. 988

6 Fgure 5. Precson-Recall (PR) generated usng Hybrd algorthm (dashed lne) and Indvdual Object Detectors (sold lne); (left) PR curves for Trafc Lght object class; (Rght) PR curves for Stars object class. Acknowledgements The materal presented n ths paper s based upon work supported by the AFOSR under Contract No. FA C-018 and by DARPA under Contract No. HM C References [1] J. Baxter. A Bayesan/nformaton theoretc model of learnng to learn va multple task samplng, Machne Learnng, I, 7 39, [2] A. P. Dempster, N. M. Lard, and D. B. Rubn, Maxmum lkelhood from ncomplete data va the EM algorthm, Journal of the Royal Statstcal Socety: Seres B, 39, 1 38, [3] D. Heckerman, A tutoral on learnng wth Bayesan networks, In M. Jordan (Ed.), Learnng n Graphcal Models, Cambrdge, MA: MIT Press, [4] D. Hoem, A.A. Efros, and M. Hebert, Puttng objects n perspectve, Proc. IEEE Comp. Vs. Pattern Recog., 2, , [5] A. Hollngworth, and J.M. Henderson, Does consstent scene context facltate object detecton, J. Exp. Psychol. Gen., 127, , [6] A. Olva, and A. Torralba, The role of context n object recognton, Trends n Cogntve Scences, 11, , [7] S.E. Palmer, The effects of contextual scenes on the dentfcaton of objects, Cognton & Memory, 3, , [8] J. Pearl, Probablstc Reasonng n Intellgent Systems, San Mateo, CA: Morgan Kaufmann., [9] A. Rabnovch, A. Vedald, C. Gallegullos, E. Wewora, and S. Belonge, Objects n Context, Proceedngs IEEE 11th Internatonal Conference on Computer Vson, Vol. 1, 1 8, [10] B.J. Rhodes, Taxonomc knowledge structure dscovery from magery-based data usng the Neural Assocatve Incremental Learnng (NAIL) algorthm, Informaton Fuson, 8, , [11] A. Torralba, Modelng global scene factors n attenton, J. Opt. Soc. Am., A, 20, , [12] A. Torralba, K. Murphy, W. Freeman, and M. Rubn, Context-based vson system for place and object recognton, Proceedngs IEEE 9 th Internatonal Conference on Computer Vson, Vol. 1, , [13] L. Wolf, and S. Blesch, A crtcal vew of context, Internatonal Journal of Computer Vson, 69, , [14] M. Zandpour, B. J. Rhodes, and N. A. Bomberger, COALESCE: A probablstc ontology-based scene understandng approach, Proceedngs of the 11th Internatonal Conference on Informaton Fuson, Cologne, Germany, June 30 July 3,