Using Articulated Scene Models for Dynamic 3D Scene Analysis in Vista Spaces

Transcription

1 Topical Edito: Pof. Chistian Wöhle, 3D RESEARCH 03, 04 (2010) /3DRes.03(2010)04 3DR EXPRESS w Using Aticulated Scene Models fo Dynamic 3D Scene Analysis in Vista Spaces Niklas Beute Agnes Swadzba Fanz Kummet Sven Wachsmuth Received: 29 August 2010 / Revised: 29 Septembe 2010 / Accepted: 10 Octobe D Reseach Cente and Spinge 2010 Abstact 1 In this pape we descibe an efficient but detailed new appoach to analyze complex dynamic scenes diectly in 3D. The aising infomation is impotant fo mobile obots to solve tasks in the aea of household obotics. In ou wok a mobile obot builds an aticulated scene model by obseving the envionment in the visual field o athe in the so-called vista space. The aticulated scene model consists of essential knowledge about the static backgound, about autonomously moving entities like humans o obots and finally, in contast to existing appoaches, infomation about aticulated pats. These pats descibe movable objects like chais, doos o othe tangible entities, which could be moved by an agent. The combination of the static scene, the self-moving entities and the movable objects in one aticulated scene model enhances the calculation of each single pat. The econstuction pocess fo pats of the static scene benefits fom emoval of the dynamic pats and in tun, the moving pats can be extacted moe easily though the knowledge about the backgound. In ou expeiments we show, that the system delives simultaneously an accuate static backgound model, moving pesons and movable objects. This infomation of the aticulated scene model enables a mobile obot to detect and keep tack of inteaction patnes, to navigate safely though the envionment and finally, to stengthen the inteaction with the use though the knowledge about the 3D aticulated objects and 3D scene analysis. Keywods Vista space, Aticulated scene model, Mobile obot, Peson tacking, 3D backgound modeling 1. Intoduction Niklas Beute ( ) Agnes Swadzba Fanz Kummet Sven Wachsmuth Bielefeld Univesity Bielefeld, NRW GERMANY nbeute@uni-bielefeld.de Embodied agents, both humans and mobile obots, have to peceive, to analyze and to segment obseved sceney into meaningful pats to deal with and communicate about the unknown and dynamic envionment. In this pape we pesent a 3D scene analysis appoach, which enables mobile obots to solve such poblems by gatheing boad knowledge about thei envionment only by obsevation of the sceney. Because of the boad aea of equiements the obot needs infomation about diffeent pats of the wold. Fist, the obot has to detect and tack humans as possible inteaction patnes o moving obstacles to avoid possible collisions. Second, the static scene pats like walls, cupboads o tables have to be segmented to give a boad knowledge about the oom stuctue fo e. g. navigation puposes 45 o oom classification 41. In contast to othe typical backgound modeling appoaches 36,19,26, ou suggestion is to distinguish as well between static objects and objects like chais, teddy beas o othe smalle objects that can be moved by an agent 42. Instead of building a complex ontology of human envionments to descibe which pats may be moving o could belong to the static backgound and equipping the obot with stong detectos fo each possibility, we popose to lean an aticulated scene model on the basis of scene obsevation. This bottom-up leaning of spatial awaeness enables a mobile obot to extact essential knowledge about the envionment, which is achieved only by obsevation. The aticulated scene model is composed of the following thee scene pats. Definition 1. (Aticulated scene model): Static scene (Neve changing pats) Moving entities (e. g. humans o obots) Movable objects (e. g. chais, doos) This model is updated in one single and simultaneous computation. Fig. 1 is meant to give an example. On the left the accodant fame of the scene is pesented and on the ight an example of a 3D aticulated scene model is shown. Coloed in black is the static backgound, in oange

2 2 3D Res. 03, 04 (2010) and bown ae two aticulated objects and in geen the actual tacked human is displayed. Usually, the obsevation of an envionment efes to the lage-scale-space 22,27, whee a main popety is the necessity of locomotion to peceive the space, which could be, e.g., a complete flat o apatment. In the poposed system we apply the obsevation on the so-called vista space, which descibes the visual field only by slightly moving the gaze. (a) 2D amplitude image (b) 3D point cloud Fig. 1 In the left image the fame of an example sequence is shown. In (b) two detected aticulated scene pats ae shown (cupboad doo, wate can) in ed and oange, which emege afte a few seconds of obsevation, if an agent moves the specific object. The gay 3D points belong to the backgound and the geen points to the cuent tacked peson. Definition 2. (Vista space): The vista space is a pat of the wold, which can be viewed at the same moment, only be slightly moving the gaze. This means that the system elies on the peception of a single oom o pats of a oom and that the shot obsevation time limits the numbe of available fames. By the use of the vista space we can deive the assumption that the fathest measuement in the scene descibes the backgound. If an object appeas in font of peviously seen static pats, we can assume a moved object, while upcoming obsevations of moe distant points indicate a emoved object. Assumption 1. (Vista space assumption): The fathest measuement in the scene descibes the backgound. As vista space models delive complementay infomation to lage-scale space models the combination of both model types into a common epesentation, e.g., using the Hybid Spatial Semantic Hieachy (HSSH) of Beeson et. al 1 will fom the foundation fo modeling spatial knowledge of the entie envionment an agent inteacts with. Howeve, in this pape we focus on the analysis of the vista space as we ae going to integate the system in the home-tou scenaio 7, in which a use guides the obot aound in his flat and paticula vista space situations aise. The obot needs a meaningful sensoy input to peceive the envionment, which in ou case is achieved by using a time-of-flight 3D camea. The 3D data is extended with additional 3D velocities using optical flow. The use of a 3D senso tanslates the poblem to an inheent 3D intepetation task. Ou poposed system builds the aticulated scene model only by obseving the 3D sceney fo a few seconds, theeby segmenting the envionment into diffeent pats and incopoating the aleady leaned knowledge. The humans in the obseved scene ae detected by consideation of velocity infomation and a weak object model suitable fo many diffeent kinds of objects. The human is tacked by a hybid paticle filte with mean shift, which enables the obot to keep tack of the movements of the human. The calculated tajectoies supply a boad knowledge about the typical movement aeas in the scene and additionally, the obot gets the equied positions of possible inteaction patnes. In contast to othe backgound modeling stategies, the aticulated pats of the scene ae sepaated fom the static scene, which ae nomally incopoated again into the backgound model afte the objects become static again. Usually, even with a stong detecto the aticulated objects ae had to detect as they could have any shape o size. Hee, the aticulated pats ae detected though the vista space assumption. The static scene is composed of the emaining pats afte excluding the pesons and the movable objects. Though the exclusion of dynamic pats the static scene is vey eliable fo navigation o scene classification, as many potentially changing pats have been aleady emoved. Howeve, the main advantages of the poposed system ae based on the paallelism and the geneality of the detection of the diffeent pats of the aticulated scene model. Though the detection and exclusion of moving pesons and movable objects the building of the static scene is much moe obust. On the othe hand the knowledge about the static scene enhances the detection of humans as the static backgound could be subtacted and the detection can be limited to dynamic pats of the cuent obsevation. The static scene again is used in the assumption of the vista space to detect the aticulated scene pats. Diffeent to existing appoaches, movable objects can be detected without the explicit detection of a movement of the paticula object, but though the knowledge about the static scene and the infomation fom obsevation. The contibution of the poposed model is a solid basement of infomation, which could be used by many othe applications as input. In the following, we want to pesent some ideas o possible applications fo the aticulated scene model. The possibilities ae compehensive as the model is a good stating point fo seveal leaning o inteaction scenaios. As mentioned befoe, the tacked pesons o moving objects could be diectly associated as inteaction patnes. On the othe hand, the infomation about thei movements can be used as data fo typical movement aeas o pathways, which could be used fo navigation puposes of the obot. The aticulated pats appaently enable the obot to ecognize objects, which ae handled by the human o moe simply, which objects ae movable. This knowledge could be utilized in a tabletop-leaning scenaio, whee each object put onto the table could be easily ecognized as a new object independent fom its topology o appeaance. Again, using the whole infomation about the ecognized objects and the appeaance and disappeaance aeas we get an idea about the action spaces of these objects. In the case of a doo, we could see the aticulation o the opening ange of the doo as an action aea. Seveal othe scenaios ae supposable, but as the main contibution of this pape is a solid basement of knowledge fo a mobile obot we skip

3 3D Res. 03, 04 (2010) 3 futhe suggestions how to use the aticulated scene model in a specific application. The pape is stuctued as follows. Fist, elated wok is pesented in section 2 to give an oveview of othe wok done in the diffeent fields coveed by this pape. The poposed system in geneal is descibed in the subsequent section 3. The pepocessing of the senso data is explained in section 4 and the computation of the extended optical flow in the subsequent section 5. The detection and tacking of moving entities is descibed in section 6, followed by the desciption how to build the static pats and how to detect the movable pats of the aticulated scene model in section 7. In the end, we explain ou expeiments and we show the esults of ou algoithm on seveal selfceated data sets in section Related Wok Reseach on dealing with dynamic scenes has become moe and moe impotant since the manual analysis of the huge amount of video data povided by video suveillance is not suitable any moe. Divese methods have been developed to model the backgound that can be subtacted fom the cuent image to extact the moving foegound. The appoaches ange fom classical Gaussian Mixtue Models (GMMs) 36 to the use of codebooks 19 modeling the pixels eithe sepaately fom each othe o incopoating neaby pixels using subspaces 26. Fo a lot of appoaches a static backgound is mandatoy howeve Sheikh and Shah intoduced an appoach that is able to cope with unifomly moving backgound like a ive 35. The wok fom Bostow and Essa 4 descibes the othe way aound. They popose to obseve the foegound and to analyze the motion of dynamic objects to decompose a scene fom a single view as multiple layes. By extending the single view to multiple views Guan et al. 12 ecove static 3D shapes of static occlusions by obseving the human motion shapes. Multiple views including depth-sensos ae able to econstuct eliable 3D scenes, if the camea setup is well calibated 13. The poblem of a moving camea has to be consideed, if we tansfe appoaches fo detecting moving egions developed in a static suveillance scenaio to a obotic scenaio. This can be done diectly by an egomotion compensation 34, by visual navigation 10 o by detecting moving objects though inconsistencies in a scene motion field aising fom a optical flow computation 20. Anothe poblem in obotics scenaios is the shot obsevation time and the unknown envionment so that a pevious taining of the backgound is not possible. Theefoe, Hayman and Eklundh 14 developed a Bayesian model fo incopoating the possibility that the backgound has not yet been uncoveed. Besides fom moving pesons also movable objects ae inteesting fo a obot. Movable objects ae chaacteized by occasional elocation and longe static peiods. In classical backgound subtaction appoaches such objects will be integated into the backgound model afte elocation thus cannot be detected anymoe 29. Sandes et al. 31 ty to solve this poblem by integating pixel infomation ove time. The pixel histoy is clusteed to tempoal coheent clustes, the so-called tempoal signatues, which allow detecting quasi-static objects unde the condition of these objects having aived and depated fom the scene. As movable objects belonging to the class of scene stuctuing elements like a chai ae of special inteest fo a obot some appoaches ty to find such scene elements though analyzing the human activity instead of detecting them diectly. Fo example, tajectoies can be segmented to actions using Hidden Makov Models (HMMs) 29 concluding that the location of an action points to an object associated with an action like, e. g., sitting down is coupled with a chai. Altenatively, clusteing of motion histogams computed pe scene cell allows an image segmentation poviding inteesting indoo scene egions like a sofa 9. The analysis of tajectoies of moving objects can eveal besides image egions that coespond to scene elements geneal semantic egions like junctions o paths that do not match a specific movable object. Analyzing peson tajectoies in indoo ooms could eveal semantic egions like a gouping of table and chais 21. Analyzing peson activities and ca tajectoies in outdoo envionments could povide models of oads and paths 45, walkable gound sufaces 3, o outes, paths, and junctions 24. A detailed eview of futhe methods fo undestanding scene activity is given in 6. In the case of detecting movable objects, e. g., a doo, which motion is caused by a human manipulation 31 tajectoies of such objects eveal thei possible aticulation. Inspied by aticulated body models, Stum and colleagues 38 developed techniques fo leaning kinematic models of scene elements like table o dawe. As tacking of such objects is a challenging poblem they bypass it in thei pape though attaching makes to test objects. In thei last pape 37 they have pesented an automatic tacking of a plana suface fom a cupboad doo o a dawe font fo obsevation situations esticted to a close-up view of the suface. Ou aticulated scene model aims to combine backgound modeling with detection of semantic scene elements. As we focus on the modeling of dynamic 3D scenes the assumption that static measuements which ae futhest away detemine the scene backgound allows an elegant way to model the backgound especially in obotic scenaios whee obsevation times ae shot. Subtacting the backgound in 3D eveals diectly quasistatic/aticulated objects without special equiements like an object has to aive and depat 31 and independent fom thei shape o size o the human activity connected to them. Detecting abitay aticulated scene elements using human activity equies ecognition abilities of a lot of diffeent daily-life activities, which means that a huge database of all possible actions is needed fo taining. Ou appoach povides fo 3D data a bypass to this exhaustive leaning poblem. 3. System Oveview The obot s pupose is to inteact with the human and to wok with him in the same envionment, but the envionment is natually not static and the human moving in font of the obot is inhibiting the backgound modeling pocess. Theefoe, the obot should acquie knowledge

4 4 3D Res. 03, 04 (2010) about its suounding by detecting and tacking moving objects, modeling the static backgound without these pesons and peceiving scene changes in the vista space. In the pocess the obot obseves its envionment passively, which means the obot camea stays static fo a few seconds to gathe infomation befoe the obot changes its view and obseves the next vista space. The algoithm is designed to calculate an aticulated scene model M fo each of the vista spaces (see fig. 2). The model consists of the dynamic pats D, the static backgound S and the obseved aticulated scene pats O. The model fo one vista space is updated as long as the obot does not change its view. The model M t 1 is updated by popagating it to the next fame at time t. In each fame the following pocesses ae accomplished to update the model: 1. Model popagation: The model M t 1 fom the pevious fame is popagated to the cuent fame 2. Peception & Pepocessing: The actual senso input is pepocessed and annotated with velocities 3. Entity Tacking: Moving objects ae detected and tacked to exclude them fom the static scene 4. Scene Modeling: The backgound and the movable objects ae adapted The pepocessing caes fo the 3D data smoothing and velocity computation Vt based on optical flow esulting in 6D data as senso input fo fame t. The next step is to detect and tack the moving pats, named as Entity Tacking. Theeby, the detection and tacking of moving pesons is suppoted by the knowledge of the actual static scene St 1 geneated out of all pevious fames and vice vesa. Fig. 2: The aticulated scene model is calculated fo each vista space. The model is updated fom fame to fame by obseving the sceney. Utilizing the model fom the pevious fame and the senso data fom the cuent fame the updated model can be calculated by two steps. Fist, the entity tacking detects and tacks moving objects by shifting a cylindical model though the potential dynamic points Dtpot. The potential points ae all points, which ae not confom to the known static backgound. Second, the static scene and the aticulated objects ae adapted. Theefoe, all found moving objects ae subtacted and the poduced potential static points S tpot ae analyzed with the vista space assumption to sepaate movable objects fom the updated static scene. In a fist step, the known static scene points n fom the pevious fame (1) S t 1 = s ti i =1Kn { } ae subtacted fom the cuent scene Ft = f ti i =1Kn { } (2) The emaining potential dynamic points Dtpot = Ft S t 1 (3) V ae annotated with the velocity data t. Based on the potential dynamic points Dtpot new objects ae detected.

5 3D Res. 03, 04 (2010) 5 Using a clusteing algoithm and a simple cylindical object model, the moving objects ae found and subsequently tacked with a hybid paticle filte with mean shift. The potential points pot ε t D t (4) which belong to a dynamic object ae passed to the cuent aticulated scene model M t. In the scene-modeling step these points ε t ae subtacted fom the actual fame F t to identify the potential static points pot St = Ft ε t (5) in the cuent fame. By applying the vista space assumption and utilizing the knowledge S t 1 fom the last fame the movable objects O t that fom the aticulated scene pats can be detected and the static backgound S t can be updated, simultaneously. Both ae passed to the cuent aticulated scene model M t, which is popagated again to the next fame. The updating of the vista space ends if the obot changes its view and duing the motion of the camea fom one vista space to anothe the model computation is stopped. At this moment the outcome fom the aticulated scene pats O t ae all the aeas whee a movable object is newly detected by the vista space assumption. Fom the moment the obot obseves a new vista space the building of the next aticulated scene model begins. By incopoating the motion of the obot the vista spaces can be meged to build a global knowledge base. Hee, we utilize the motion infomation fom a lase-based SLAM appoach 45. The typical obsevation time fo one vista space is about 15 to 20 seconds. Huhle 18 fo outlie detection and smoothing using Non-local Means filte 5. Smoothing techniques elying only on the ToF data ae amplitude thesholding with a fix value 25, emoving of flying pixels at edges via detecting iteatively geometic outlies taking into account the 2D neighbohood 17, and coecting the amplitude values using distance values and vice vesa 28. In this pape we applied pepocessing techniques poposed in 40. The distance image is smoothed with a distance-adaptive median filte, which uses fo each pixel a diffeent mask size (e.g. 3 3, 5 5, o 7 7) depending on the distance value of the pixel. Geneally, pixels with lage distance value ae filteed with smalle filte masks, and vice vesa, so that significant stuctues at lage distances ae not blued, and at the same time, noisy sufaces at small distances can be smoothed. As the amplitude value efes to the quality of the distance measuement, points with a small amplitude value ae emoved fom the final 3D point cloud. The theshold needed adapts automatically to diffeent eflectance popeties in diffeent scenes, as it is a faction of the mean amplitude value pe fame. Futhe, edge points (so-called flying pixels ) aising in the case whee light fom the foe- and the backgound hits the same pixel simultaneously ae ejected if the amount of nea neighbos in the 2D neighbohood is insufficient. Last, 3D coodinates ae geneated out of the distances with egad to a 3D camea coodinate system. 4 Pepocessing (a) (b) Ou system uses the Swissange SR3000 povided by Swiss Cente fo Electonics and Micotechnology (CSEM) 45 deliveing a matix of distance measuements independent fom textue and lighting conditions. It consists of CMOS pixel sensos, which ae able to detemine actively the distance between the optical cente of the camea and the eal 3D wold point via measuing the time-of-flight of a nea-infaed signal. Besides the distance value matix (Fig. 3(b)), the camea povides pe fame a matix containing amplitude values (Fig. 3(a)). The amplitude value indicates the amplitude of the eflected nea-infaed signal eceived by the senso and implies theefoe the amount of light eflected by a wold point. A small amplitude coesponds to a small amount of light eflected and theefoe indicates a weak signal. Seveal eseaches have aleady developed pepocessing and calibation techniques dealing with noise aising fom the diffeent eflectance popeties and chaacteistics of the ToF cameas, like additional infa-ed light in the scene, and measuement eos at edges (socalled flying pixels ). Schille 32 poposed an automatic calibation of the entie 3D ToF signals using a bunch of diffeent cameas. Colo infomation is also used by (c) (d) Fig. 3 Raw data acquied fom the Time-of-Flight senso: (a) amplitude image, (b) distance image, (c) unpepocessed 3D point cloud, and (d) pepocessed 3D point cloud. With the assumption of ideal pespective pojection, the known position of the pincipal point, pixel sizes, and focal length, the 3D coodinates can be computed fom the distances via ay popotions in tiangles. As a esult the computed 3D points ae oganized egulaly in a 2D matix. Fig. 3(a) to 3(c) show a fame of the 3D ToF camea consisting of an amplitude image, a distance image, and the aw 3D point cloud. Fig. 3(d) pesents the esulting 3D point cloud afte applying the descibed pepocessing techniques. In ode to distinguish between static pats of the scene and moving pesons o objects the motion in the 3D point cloud has to be detemined. In the following an imagebased method fo motion computation is pesented which

6 6 3D Res. 03, 04 (2010) can be applied hee easily by teating the point cloud as plana depth maps o images Motion Computing using Optical Flow A widely used technique is to compute the dense Optical Flow using each 2D image pixel. The optical flow is the distibution of appaent velocity of moving bightness pattens in an image and aises both fom the elative objects and the viewe s motion [11]. The flow of a constant bightness pofile I ( x, y, t) = I ( x + dx, y + dy, t + dt) = I ( x + vx dt, y + vy dt, t + dt) = I ( x + v dt, y + v dt, t dt) (6) x y + I I I v x + v y = (7) x y t T is descibed by the constant velocity vecto v 2 D = ( vx, vy). Usually, the estimation of optical flow is founded on diffeential methods. They can be classified into global stategies, which attempt to minimize a global enegy functional 15 and local methods that optimize some local enegy-like expession. A pominent algoithm developed by Lucas and Kanade 23 uses the spatial intensity gadient of the images to find a good match using a type of Newton- Raphson iteation. They assume the optical flow to be constant within a cetain neighbohood N which allows solving the Optical Flow Constaint Eq. 7 via least squae minimization. Hee, we have used a hieachical implementation of Lucas s and Kanade s algoithm witten by Sohaib Khan ( As the Swissange camea povides nomal 2D intensity images based on the amplitude values it is possible to educe the 3D coespondence poblem to a 2D coespondence poblem and to compute the optical flow fo each fame of a sequence of ToF images ( F 1, F 2,...) based on data of two consecutive fames ( F i, F i 1). Each pixel of fame F i is annotated with a 2D velocity vecto T v 2 D = ( vx, v y ) as shown in Fig. 4(a), which esults into pixel coespondences between fame F i and fame F i 1. As 3D infomation is available fo each pixel these pixel coespondences can be diectly tansfomed into 3D point i i 1 coespondences ( p k, pl ), which can be used to compute T i i 1 3D velocities v3 D = ( vx, vy, vz ) = pk pl. Fig. 4(b) pesents a 3D point cloud of one fame of a test sequence annotated with 3D velocity vectos. The pocessing fom 2D optical flow on 2D images to eal 3D velocities is suppoted by the used hadwae. As the Swissange camea povides good distance measuements velocities with eliable values especially in the z component can be detemined. This is usually not suitable fo many othe camea setups like steeo igs o multi-camea systems. The velocity annotated 3D point cloud esults in 6D data. Due to the low esolution of the camea and inaccuacies of the optical flow eoneous velocity vectos at changing depth steps ae computed. To get id of those outlies a 5 5 median filte is applied sepaately to the thee components v x, v y, v z of the flow vecto v 3D. In Fig. 4(c) the smoothed esult of the 3D velocity field of Fig. 4(b) can be seen. (a) (b) (c) Fig. 4 Velocity pocessing with the optical flow method: (a) 2D velocity vectos (b) 3D velocity vectos fom combining 2D velocities and point coespondences in consecutive images, (c) the latte smoothed component wise by a median filte. Each 3d velocity vecto is displayed and colo coded with espect to its length: ed denoting a big motion vecto and blue a small one. 6. Detection and Tacking of Dynamic Objects (a) Clusteing of pot D t (b) Pobability distibution (c) Found object (d) Tajectoy Fig. 5 The images explain the tacking algoithm. The blue points belong to the static scene S t 1. The dynamic pixels P t ae plotted in oange. (a) At a fist stage the dynamic points ae clusteed, geneating small motion attibuted egions. (b) The objects ae detected and tacked using the obsevation function (see Eq. Eo! Refeence souce not found.). The pobability of the paticle distibution is plotted in geen. (c) The maximum of the obsevation function denotes the found object (shown as geen box). (d) The esulting object tajectoy is plotted in cyan. The blue cicle contains the object at the actual position. The dynamic scene analysis involves the detection and tacking of moving objects, which on the one hand enhances the segmentation of the diffeent scene pats and which is on the othe hand useful fo the undestanding of the scene as the tajectoies of the objects give a boad pictue of the movements in the actual vista space. Using the 3D point cloud and the annotated 3D velocities, we can simplify the scene by applying a 6D hieachical clusteing technique. The segmentation is enhanced though the incopoation of the velocity infomation in the ealy clusteing stage, because it enables the segmentation of neighboing objects, like a peson walking in font of a wall. The fist step is to span small contiguous egions in the cloud of the 6D points, based on featues fo spatial

7 3D Res. 03, 04 (2010) 7 poximity and homogeneity of the velocities. We apply a hieachical clusteing using the complete linkage algoithm 2, which, choosing a small banching facto in the hieachical tee, delibeately ove-segments the scene, geneating many small motion-attibuted clustes 33 (see Fig. 5(a)). Each calculated cluste is annotated with the 2D position of its centoid pojected on the gound plane, a weight facto accodant to the numbe of included points and the mean velocity of all these points. Fom hee on, pesons and objects ae epesented by an upight cylinde of vaiable adius, which is a suitable model fo the moving entities in ou scenaios (hee, humans). The object hypothesis h(a) is chaacteized by a five dimensional paamete vecto a = [ x, y,, vθ, v ] (8) whee x and y ae the centoid on the gound plane, the adius and v θ the diection and v the magnitude of the velocity of the cylinde. The next step is the detection of the moving objects. Hee, the cylindical model is advantageous fo the detection as the velocity computation is noisy and many points between the found clustes have diffeent velocities. This means, that the hypothesis coves mostly the full moving object even if the velocity data is noisy. Theeby, the cylinde does not need to touch the gound o to satisfy any specific height, because all clustes ae pojected on the gound plane. This is especially useful if the moving object is only patially seen. The detection only needs a few clustes denoting a moving object and aftewads, using the cylindical model, all clustes that ae lying in the cylinde ae added to the moving object. To geneate a hypothesis, the cylinde is shifted though the small clustes seaching fo meaningful collections of clustes with simila velocities. Gouping close clustes togethe, a hypothesis is found if the weight of all clustes togethe is highe than a cetain theshold. Hee, 20 close points moving in a simila diection ae sufficient. Theeby, the cylinde has the expansion fom the lowest to the highest detected moving cluste. The adius is weighted using a Gaussian with the mean at typical peson dimension. Aftewads, each cluste integated into a hypothesis is maked as an aleady found object to ensue each cluste is used only fo one hypothesis. All found hypothesis ae additionally annotated with an id to identify them ove the obsevation time. The detection by moving needs the object to move at least one time and aftewads, it is capable to tack the object even if it is not moving anymoe. If a potentially moving object has neve been seen moving so fa, a stonge detection algoithm is needed. As we have diffeent moving objects like pesons, obots, cleaning machines o othe self-moving objects we would need seveal detectos and classifies fo each goup. Hee, we only utilize a human classifie, because most of the attending moving objects in the Home-Tou-Scenaio ae humans and the othe objects ae mostly detected with the peviously mentioned clusteing algoithm. As a human classifie we utilize the Histogams of Oiented Gadients detecto 8, which uses the gadient featues of local bins consolidated in one histogam. A tained suppot vecto machine diffeentiates between humans and non-humans, deliveing ectangula egions including pesons. In the calculated 2D aea we cluste the included points with a k- means clusteing with k=2 to sepaate foe- and backgound. The cylinde is fitted into the 3D foegound and the hypothesis is added to initial detection set. All extacted hypotheses fom the cuent fame ae meged with the ones tacked fom the pevious fame esulting in one hypotheses matix fo each fame. The tacking of the hypotheses is calculated like follows. The K hypotheses in the cuent fame t ae tacked based on the position, velocity and size of each hypothesis in the pevious fame h t 1 ( a ), utilized in a hybid kenel paticle k filte with mean shift 33. The paticle filte ceates a set of new hypotheses s t k (h) fo each tacked object, called paticles, and distibutes them with a fist ode motion model mixed with a andom Gaussian noise (see Fig. 6(a)). This distibution of paticles coves the potential movement of most moving objects as it follows linea and andom movement. (a) Paticle distibution (b) Mean shift Fig. 6 (a) The paticle distibution follows a motion model and a andom distibution to cove all possible motions of a human. The figue shows the distibution of the paticles in the XY plane. The movement of the object is in positive X diection. The paticles ae distibuted in the accodant diection and fo andom movements as well. (b) The distibuted paticles ae weighted and then shifted with mean shift to ecove the best possible object position. (Hee, shown fo a andom distibution) In ode to identify the new position of each hypothesis the paticles ae ated with an undelying obsevation. The obsevation is based on the elative position, elative velocity and weight of all clustes within the cylinde of each hypothesis weighted with Gaussian kenels. ρ ( s ) = K ( s ) K ( l, s ) K ( l, s ) (9) k k l s k d k v 2 2 ( s = k ) ( s 2 k ) K ( sk ) exp exp 2 (10) 2H,min 2H, max 2 d( l) d( sk ) K (, ) = exp d l sk (11) 2 2H d 2 ( ) v( l) v( sk ) K, = exp v I sk (12) 2 2Hv With K s ) keeping the adius in a ealistic ange, ( k Kd ( l, sk ) educing the impotance of clustes futhe away K I, masking out clustes fom the cylinde cente, and ( ) k v s k having diffeing velocities. The functions ( ), d( ), and v( ) extact the adius, the 2D position on the gound plane and the velocity of a cluste l o a hypothesis s k. The kenel widths H ae detemined empiically. Eq. Eo! Refeence souce not found. is also called the obsevation function

8 8 3D Res. 03, 04 (2010) ρ ( s k ) of the paticle filte. The outcome is a density appoximation based on the object hypothesis and the attibutes of the appending clustes, with the maxima coesponding to the actual objects (Fig. 5(b)). Seveal mean shift iteations efine the paticles to concentate at the local maxima of the distibution, which deceases the needed amount of paticles (see Fig. 6(b)). The combination of mean shift and paticle filte is ideal to add the stengths of both pats in one algoithm. Mean shift sometimes sticks in local minima, which could be esolved due to the sampling of the paticles. The paticle filte needs many paticles to estimate the undelying density function, which can be avoided though the combination with mean shift. Individual paticles selected fom these best modes of the distibution epesent the objects found in the cuent fame (Fig. 5(c)). Fo each tacked object hypothesis, all 3D points associated with this object ae back pojected in the 2D amplitude image and used fo computing a 2D convex hull of the tacked object. All points within this 2D polygon ae maked as non-static points and ae finally excluded fom the econstuction step. 7. Adaptive Backgound Modeling The algoithm pesented in Fig. 7 is applied to each time step of the obsevation to calculate the updated static scene S { i t = s t } and the movable objects O t. Theefoe, the algoithm uses as input the cuent fame F { } i t = f t and the i last known static scene St 1 = { s t 1} and the dynamic clustes ε t fom the pevious fame. The dynamic clustes contain the 3D points fom the moving objects of the tacking module. These points ae emoved, befoe the update pocess takes place. The static scene is updated in line 11, if the diffeence of a known static point s i t 1 to the actual fame point f t i is below the senso noise level θ d. Then, the static point and the cuent point ae accumulated to a new static point s t i with impoved eliability. Othewise, it has to be detemined if a new static scene point is detected in line 16 o the point belongs to a movable object in line 19. The vista space assumption is used to identify the matching case. All points belonging to movable objects ae saved in a sepaate list, whee the time of detection and the numbe of times the points has been seen ae consideed. Clusteing these points in space and time the diffeent objects can be sepaated. Consequently, objects can only be sepaated if they appea at a diffeent point in time o at least at diffeent places. 8. Results Fig. 7 Algoithm pe time step t fo backgound adaptation and movable object detection. So fa we poposed methods to distinguish between static and moving pats in a scene. In the following, the calculated moving objects ae extacted and the static pats of the obsevation ae analyzed. By applying the vista space assumption and utilizing the knowledge fom the last fame the movable objects that fom the aticulated scene pats can be detected and the static backgound can be updated, simultaneously. The basis of the vista space assumption that the most distant measuement in the cuent view descibes the backgound has to be expanded due to noise of the 3D senso. Theefoe, we intoduce a theshold θ d above that a change in the distance is significant and do not aise fom noise (hee, θ d = 10cm given by the noise level of the camea). The evaluation of an aticulated scene model does not follow typical standad epots, as it is not feasible to build a complete gound tuth model. Hence, we split up the diffeent pats of the model and we compaed the static scene to a gound tuth model and to some simple backgound modeling techniques to give quantitative esults. In the following, the poposed system M ADAPT is evaluated by compaing the esults to the naive appoach of only summing up the images and building the mean fo each pixel ( M MEAN ). It is also compaed to the neglecting of moving pixels M MPIX and last, to MTRACK 39 whee only dynamic objects ae detemined though tacking without backgound model feedback and no distinction is made between static backgound and static movable objects. All methods ae checked against a gound tuth static scene model M GT, which has been taken without any movable o moving object fo each sequence. The aticulated pats and the tajectoies of the moving objects ae pesented qualitatively in illustations. The undelying data sets S ae self-ceated and they show diffeent challenging dynamic scenes. The human shows diffeent moving behavios o stops moving, which makes it difficult to detect him as not static. Futhemoe, the human inteacts with the envionment as he cleans up S S3, moves chais, seaches a teddy bea S s2, opens and closes doos S S 4 and eaanges teddy beas S S1, wate cans S S5 and plants S S 6. Each un i of a sequence belonging to one scenaio j is labeled with S,. sj i

9 3D Res. 03, 04 (2010) 9 S s1, S s1, S s1, 3 S s1, 4 S s1, 5 S s1, 6 M MEAN 103± ± ± ± ± ±262 M MPIX 64±121 74±184 79± ±216 99±230 95±193 M TRACK 71± ±209 75±189 97±212 79±308 98±219 M 18±59 19±47 21±61 24±78 24±68 21±55 ADAPT S s2, S s2, S s3, S s3, S s4, S s4, M MEAN 95± ±147 89±105 85± ± ±639 M MPIX 71±155 80±118 63±145 61± ± ±635 M TRACK 84±182 85±140 71± ±712 51±165 74±218 M ADAPT 20±96 16±37 20±58 22±52 14±26 75±319 S s2, S s2, S s3, S s3, S s4, S s4, M MEAN 95± ±147 89±105 85± ± ±639 M MPIX 71±155 80±118 63±145 61± ± ±635 M TRACK 84±182 85±140 71± ±712 51±165 74±218 M ADAPT 20±96 16±37 20±58 22±52 14±26 75±319 Table 1 Evaluation of fou econstuction methods on 17 sequences (mean eo ± mean vaiance). The eo shown in the table is computed as mean Euclidean distance ove all model points to the coesponding gound tuth points. The mean eo is given in mm as well as the mean vaiance. The high eo in S s5, 2 esults fom a wide ange view, whee the senso poduced a high amount of noise. peson, which povides a sound backgound model. Table 1 shows an analysis of the aising eos fom the backgound modeling stategies. (a) M GT (b) M MEAN (c) M MPIX M TRACK (e) M ADAPT (d) Fig. 8 Results of scene S fo the evaluated algoithms. In the font the s 2, econstucted 3D static scenes and in the back the accodant 2D images can be seen. (a) Shows the gound tuth. In (b) the econstuction by simple aveaging, in (c) the econstuction by excluding moving pixels, and in (d) the econstuction by tacking objects is shown. In the 2D image the wong econstuction can be seen as a ghost of the peson moving in the scene. (e) Shows the esult using the poposed method. The colos encode the eo of the model if compaed to the gound tuth blue means small and ed means big eo. In Fig. 8 the esulting static scenes fo one example vista space ae pesented. The figue shows the esulting 3D static scene fom the diffeent backgound modeling techniques and the 2D image ceated fom this model. The colos encode the eo of the models compaed to the gound tuth, whee blue denotes a small eo and ed a big eo. The naive backgound modeling stategies failed in emoving the peson coectly in all fames, which esults in a big eo at those positions of the 3D point cloud, whee the peson is still visible. This gets appaent as a ghost appeas at the same positions in the 2D image. The appoach pesented in this pape eliably emoves the Fig. 9 The images show divese objects detected by ou method. All pesented objects have been moved aound by the human in the scene. Diffeent colos encode diffeent objects. The pictues show nicely the huge vaiability in detecting movable objects due to ou model independent appoach. The fist value is the mean Euclidean distance in mm ove all pixels compaed to the gound tuth and the second value denotes the coesponding standad deviation. The pesented eos affim the viewable impession fom Fig. 8 as M ADAPT esults in the lowest eo ates. The ates ae pomising with an eo mostly at 2cm and neve above 10cm. Even in scene S s4, 4, whee spase static points in

10 10 the doo can be detected, the esult of the poposed method is much moe obust than the naive appoaches, whee the mean eo is always above 20cm. A mostly low standad deviation stands fo good esults in each point as well. Fig. 10 Two examples showing the segmented movable objects in a 2D image. The fist and the thid image ae the oiginal images and the second and fouth show the maked object. The coloed aeas belong to diffeent ecognized objects, which have been moved at least one time. Highe eo ates ( Ss 5, 2 ) could occu due to noise aising fom the 3D senso, if the obseved scene has some disadvantageous chaacteistics. The senso has inceasing noise pe distance and it is sensitive to eflecting and black suface. You can see this in the mentioned figue in the open doo in fame 1 and 26. 3D Res. 03, 04 (2010) to the diffeent vista spaces. The eliability is dependent on the amount of the movement of the camea. Fo small movements the eo aveages at millimete40. (a) Tajectoy (b) Tajectoy Fig. 12 In (a)-(b) tacking esults of the poposed system ae shown (in cyan). In both views the ed pixel denote the dynamic and the blue ones the static pats of the scene. The ight scene is taken fom S s 2, (see Fig. 13 (g)) In geneal using an ICP algoithm with additional loopclosing esults in bette econstuctions30,16. In the second image of Fig. 11 all tajectoies fom one obsevation of a vista space ae plotted. One human walked thee times back and foth. His movements ae consistently tacked. Two othe example vista spaces and thei esulting tajectoies ae shown in Fig. 12(a)- 12(b) using two diffeent views. The tacking woks eliably in most cases. In the emaining cases the inaccuacy can be taced back to the single use of 3D infomation. The eos esult fom apid changes in the movement o specific actions in the sceney. These actions ae e. g. found in scene s2, whee the peson walks to a closet and opens the doo. The opening shows a simila point cloud and movement like the peson, which stops moving at the same moment. Hence, the hypothesis of the closet is moe simila to the peson as the peson itself and the tacking fails duing the opening and closing of the closet. (a) Combined vista spaces (b) Tacks fom human movements Fig. 11 (a) Subsequent vista spaces can be combined by a tansfomation of the paticula spaces in one wold coodinate system. The tansfomation is extacted out of the motion of the obot. (b) All tacks of a human walking behind a table (in cyan). The human walked thee times back and foth. Fig. 9 gives some examples of the detected aticulated scene pats. The found objects ae colo-coded in the image and they ae sepaated fom the backgound to show the vaiability in detecting divese objects due to ou model independent appoach. The objects can be maked diectly in the 2D image (see Fig. 10), which could be used by futhe pocesses to calculate moe pecise infomation like the shape o textue of the objects. Fig. 11 pesents an example of a combination of two subsequent vista spaces. Hee, we tansfom the vista spaces into the same wold coodinate system by incopoating the movement of the obot. The two images in the back belong (a) S s1, (b) S s1, 2 (c) S s1, 3 (d) S s1, 4 (e) S s1, 5 (f) S s1, 6 (g) S s 2, (h) S s 2, 2 Fig. 13 (a)-(h): Fo all ecoded sequences the leant backgound model (blue points) and the detected movable objects (oange points) ae shown. In the bottom left thee selected images of the sequence chaacteize the tide of events fom bottom to top finishing with the last fame in the backgound.

11 3D Res. 03, 04 (2010) 11 aeas o whee to pay attention. An example video showing the aticulated scene model can be found on the web. ( nbeute/aticulatedsc enemodel.html ) (a) S s3, (b) S s3, (c) S s4, (d) S s4, Acknowledgement This wok is patly funded by the Coopeative Reseach Cente CRC673 Alignment in Communication. (e) S s4, 3 (f) S s4, 4 (g) S s5, (h) S s5, Fig. 14 (a)-(h): Fo all ecoded sequences the leant backgound model (blue points) and the detected movable objects (oange points) ae shown. In the bottom left thee selected images of the sequence chaacteize the tide of events fom bottom to top finishing with the last fame in the backgound. (a) S s6, (b) S s7, Fig. 15 (a)-(b): Fo all ecoded sequences the leant backgound model (blue points) and the detected movable objects (oange points) ae shown. In the bottom left thee selected images of the sequence chaacteize the tide of events fom bottom to top finishing with the last fame in the backgound. Finally, the aticulated scene model fo each of the sequences is plotted in Fig In the bottom left a filmstip gives an idea of the pesented sequence, stating at the bottom and ending with the big pictue in the backgound. The coesponding fame numbes ae shown in the bottom ight in each image. The static backgound model elates to the blue 3D points and the found aticulated pats coespond to the coloed aeas, wheeas diffeent colos encode diffeent objects. 9. Conclusions We pesented in this pape an efficient appoach to analyze dynamic scenes diectly in 3D. The vista space assumption enables a mobile obot to segment knowledge about the static backgound, the moving entities and which objects ae movable combined in one aticulated scene model out of its obsevations. The gatheed knowledge builds a good basement fo many following eseach aeas like object leaning, navigation o just as an attention on human action spaces. We ae going to integate the static 3D backgound model in ou SLAM appoach to ealize a bette and safe navigation. We ae also planning to investigate moe wok in the detection of the aticulations of seveal objects, like the opening ange of a doo o the typical movement aeas of humans to develop an undestanding of safe movement Refeences 1. P. Beeson, M. MacMahon, J. Modayil, A. Muaka, B. Kuipes, B. Stankiewicz (2007) Integating multiple epesentations of spatial knowledge fo mapping, navigation, and communication. In Poceedings of the Symposium on Inteaction Challenges fo Intelligent Assistants, AAAI Sping Symposium Seies. Stanfod, CA. AAAI Technical Repot SS M. Bethold, D. J. Hand (2003) Intelligent Data Analysis, 2nd edn. Spinge 3. M.D. Beitenstein, E. Sommelade, B. Leibe, L. van Gooland I. Reid (2008) Pobabilistic paamete selection fo leaning scene stuctue fom video. In Poceedings of the Bitish Machine Vision Confeence 4. G. Bostow, I. Essa (1999) Motion based decompositing of video. IEEE. 1: A. Buades, B. Coll, J. M. Moel (2005) A non-local algoithm fo image denoising, In Intl. Confeence on Compute Vision and Patten Recognition (CVPR) 6. H. Buxton (2003) Leaning and undestanding dynamic scene activity, A eview. Image and Vision Computing. 21: COGNIRON (2004) The cognitive obot companion, (FP6-IST ) 8. N. Dalal, B. Tiggs (2005) Histogams of Oiented Gadients fo Human Detection. In CVPR H. M, Dee, R. Faile, D. C. Hogg, A. G. Cohn (2008) Modelling scenes using the activity within them. In Poceedings of the Intenational Confeence on Spatial Cognition VI Spinge-Velag, Belin, Heidelbeg 10. A. Ess, B. Leibe, K. Schindle, L. van Gool (2009) Robust multipeson tacking fom a mobile platfom, IEEE Tans. Patten Anal. Mach. Intell. 31(10): J. Gibson (1950) The peception of the visual wold. Riveside Pess. 12. L. Guan, J. Fanco, M. Pollefeys (2007) 3d occlusion infeence fom silhouette cues, In, Compute Vision and Patten Recognition, CVPR. Ieee, Minneapolis, MN. 13. L. Guan, M. Pollefeys (2008) A unified appoach to calibate a netwok of camcodes and tof cameas, E. Hayman, J. O. Eklundh (2003) Statistical backgound subtaction fo a mobile obseve, In, Poceedings of the Intenational Confeence on Compute Vision

12 12 3D Res. 03, 04 (2010) 15. B. K. Hon, B. G. Schunck (1981) Detemining optical flow, In, Atificial Intelligence. 17: B. K. P. Hon, H. Hilden, S. Negahdaipou (1988) Closed-fom solution of absolute oientation using othonomal matices, Jounal of the optical society Ameica. 5(7): B. Huhle, P. Jenke, W. Stae (2007) On-the-fly scene acquisition with a handy multisenso-system, In, Wokshop on Dynamic 3D Imaging (Dyn3D). 18. B. Huhle, T. Schaie, P. Jenke, W. Stae (2008) Robust non-local denoising of coloed depth data, In, Intl. Confeence on Compute Vision and Patten Recognition (CVPR), Wokshop on Time of Flight Camea based Compute Vision (TOF-CV). 19. K. Kim, T. H Chalidabhongse, D. Hawood, L. Davis (2005) Real-time foegound backgound segmentation using codebook model, Real-Time Imaging. 11: J. Klappstein, T. Vaudey, C. Rabe, A. Wedel, R. Klette (2009) Moving object segmentation using optical flow and depth infomation, In, Poceedings of the Symposium on Advances in Image and Video Technology K. Koile, K. Tollma, D. Demidjian, H. Shobe, T. Daell (2003) Activity zones fo context-awae computing, In, Poceedings of the Intenational Confeence on Ubiquitos Computing, Lectue Notes in Compute Science. 2864: B. Kuipes (1999) The spatial semantic hieachy. Atificial Intelligence, 119: B. D. Lucas, T. Kanade (1981) An iteative image egistation technique with an application to steeo vision, In, Poceedings of the Intenational Joint Confeence on Atificial Intelligence D. Makis, T. Ellis (2003) Automatic leaning of an activity-based semantic scene model, In, Poceedings of the Confeence on Advanced Video and Signal Based Suveillance. 25. S. May, B. Wene, H. Sumann, K. Pevolz (2006) 3D Time-of-Flight cameas fo mobile obotics, In, Intl. Confeence on Intelligent Robots and Systems (IROS) A. Mittal, A. Monnet, N. Paagios (2009) Scene modeling and change detection in dynamic scenes: A subspace appoach, Compute Vision and Image Undestanding. 113(1): D.R. Montello (1993) Scale and multiple psychologies of space, In, Lectue Notes in Compute Science: Spatial Infomation Theoy A Theoetical Basis fo GIS. 716: S. Opisescu, D. Falie, M. Ciuc, V. Buzuloiu (2007) Measuements with tof cameas and thei necessay coections, In, Intl. Symposium on Signals, Cicuits & Systems (ISSCS). 29. P. Peusum, S. Venkatesh, G. West, H. H. Bui (2004) Using inteaction signatues to find and label chais and floos. Pevasive Computing 3(4): S. Rusinkiewicz, M. Levoy (2001) Efficient vaiants of the icp algoithm, In, INTERNATIONAL CONFERENCE ON 3-D DIGITAL IMAGING AND MODELING. 31. B. C. S. Sandes, T. C. elson, R. Sukthanka (2002) A theoy of the quasi-static wold, In, Poceedings of the Intenational Confeence on Patten Recognition. 3: I. Schille, C. Bede, R. Koch (2008) Calibation of a pmd camea using a plana calibation object togethe with a multi-camea setup, In, The Intenational Achives of the Photogammety, Remote Sensing and Spatial Infomation Sciences. 37Pat B3a: J. Schmidt, C. Wöhle, L. Küge, T. Gövet, C. Hemes (2007) 3D scene segmentation and object tacking in multiocula image sequences, In, Poceedings of the Intenational Confeence on Compute Vision Systems. 34. J. Schmüddeich, V. Willet, J. Egget, S. Rebhan, C. Goeick, G. Sagee, E. Köne (2008) Estimating object pope motion using optical flow, kinematics, and depth infomation, IEEE Tans Syst Man Cyben B Cyben. 38: Y. Sheikh, M. Shah (2005) Bayesian modeling of dynamic scenes fo object detection, Tansactions on Patten Analysis and Machine Intelligence. 27(11): C. Stauffe, W. E. L. Gimson, (1999) Adaptive backgound mixtue models fo eal-time tacking, In, Poceedings of the IEEE Compute Society Confeence on Compute Vision and Patten Recognition J. Stum, K. Konelige, C. Stachniss, W. Bugad (2010) Vision-based detection fo leaning aticulation models of cabinet doos and dawes in household envionments, In, Poceedings of the Intenational Confeence on Robotics and Automation. 38. J. Stum, V. Pedeep, C. Stachniss, C. Plagemann, K. Konolige, W. Bugad (2009) Leaning kinematic models fo aticulated objects, In, Poceedings of the Intenational Joint Confeence on Atificial Intelligence Swadzba, A., Beute, N., Schmidt, J., Sagee, G.: Tacking objects in 6d fo econstucting static scenes. In: Poceedings of the IEEE Compute Society Confeence on Compute Vision and Patten Recognition Wokshops (2008) 40. A. Swadzba, B. Liu, J. Penne, O. Jesosky, R. Kompe (2007) A compehensive system fo 3D modeling fom ange images acquied fom a 3d tof senso, In, Poceedings of the Intenational Confeence on Compute Vision Systems. 41. A. Swadzba, S. Wachsmuth (2008) Categoizing peceptions of indoo ooms using 3D featues, In, Lectue Notes in Compute Science: Stuctual, Syntactic, and Statistical Patten Recognition. 5342: (2008) 42. A. Swadzba, N. Beute, S. Wachsmuth, F. Kummet (2010) Dynamic 3D Scene Analysis fo Acquiing Aticulated Scene Models, In, Poceedings of the Intenational Confeence on Robotics and Automation. 43. X. Wang, K. Tieu, E. Gimson (2006) Leaning semantic scene models by tajectoy analysis, In, Poceedings of the Euopean Confeence on Compute Vision, lncs. 3953: