OBSTACLE LOCALIZATION IN 3D SCENES FROM STEREOSCOPIC SEQUENCES

Transcription

1 OBSTACLE LOCALIZATION IN 3D SCENES FROM STEREOSCOPIC SEQUENCES Piotr Skulimowski and Paweł Strumiłło Institute of Electronics, Technical Universit of Łódź 211/215 Wólcańska, , Łódź, Poland phone: + (48) , fa: + (48) , piotr.skulimowski@p.lod.pl, pawel.strumillo@p.lod.pl web: ABSTRACT A method for 3D scene segmentation from stereoscopic image sequences registered b a moving camera sstem is proposed. The method identifies 6DoF ego-motion parameters first. Then, a scene model is built b projecting a triangular mesh onto the dense disparit maps. The triangles are clustered so forming surfaces of separate scene objects that are assigned to obstacle categor or contet scene objects such as walls depending on their location versus the observer. The method was tested on real stereoscopic scenes and validated on OpenGL generated 3D virtual scenes. Its advantages are: low computing demand and capabilit of identifing location and shape parameters of the selected obstacles. The developed sstem is aimed to serve as a vision module part to the travel aid prototpe sstem for the blind. 1. INTRODUCTION Vision based analsis of 3D scenes is an important research topic in robotics, automatic navigation and surveillance sstems. It belongs to the class of inverse problems, aimed at inferring 3D scene geometr from 2D multiple-view projections (e.g. stereoscopic images). Recentl, a number of novel approaches that were devoted to solution of simpler 3D scene reconstruction tasks, like obstacle detection, tracking of moving objects, or route planning have been proposed. Combination of binocular stereoscop and scanning along parallel planes (to the ground surface) was adopted in [1] for real-time tracking of human silhouettes in 3D scenes. In [2] a similar objective was achieved b combining two vision modules: a monocular one, that retrieves feature elements and a binocular one for depth estimation. An invehicle sstem for detection of urban-traffic objects within a short range was proposed in [3]. It assumes planar road surface and proposes an efficient edge-indeed stereo matching algorithm. In [4] an original image segmentation method is proposed that combines depth information and object surface features for region-growing tpe segmentation. A segmentation technique for detection of scene objects was also proposed in [5]. An assumption was adopted, however, that all objects are located in common base plane and are of simple shapes. In [6] a scene segmentation algorithm in the disparit image (disparit value is inversel proportional to depth) is proposed that uses a trinocular stereovision sstem mounted on a vehicle. This kind of sstems features multiple baselines which allow achieving more precise results if compared to the two camera sstem. In an attempt of aiding blind people in navigation (the NAVI sstem) was constructed [7]. Object segmentation is carried out in image domain. Distances of the detected obstacles that are verball communicated to the blind are estimated b using disparit techniques. In our work we assume the stereovision sstem (further termed camera sstem) can move freel in a 3D environment. Segmentation of selected scene objects is performed in the disparit map domain. The proposed method allows for scene partitioning into obstacles and scene contet objects like walls and ground surface. An advantage of this object based approach is that stationar scene elements can be continuousl tracked even if the are occluded or move out of the field of view of the binocular camera sstem. 2. OVERVIEW OF THE METHOD The developed algorithm is aimed at scene segmentation for the auditor scene displa purposes. The camera sstem is assumed to move freel through a static environment. Thus, the 6DoF (6 Degrees of Freedom) camera motion parameters, known as ego-motion parameters, can be defined b T = U V W two vectors: the translational motion vector [ ] T ω = α β γ. and the rotational motion vector [ ] T Figure 1 Camera model in a right-handed coordinate sstem Our previous studies have shown, that it is possible to estimate the ego-motion parameters b means of the so called visual odometr, i.e. b using visual information onl [8][9]. These parameters are used in the novel algorithm proposed here for 3D scene segmentation. Its ke steps are illustrated in Figure 2.

2 Figure 2 Overview of the proposed 3D scene analsis method The concept of the algorithm is as follows. Firstl, the disparit map is computed b means of block matching methods. The Sum of Squared Differences (SAD) between the corresponding piels in the compared blocks is used as the matching criterion. The identified disparit value is verified b using a two step computation procedure in which the matchings are first computed b comparing blocks from the left image to blocks in the right image and vice versa. Another approach that has been found effective is analsis of the SAD function shape in the vicinit of the detected minimum [10]. Net, camera sstem ego-motion estimation is carried out [9]. Finall, stationar scene objects can be tracked in subsequent frames of the moving camera sstem. The ground and walls are mapped as non-obstacle scene objects if the fall within a predefined 3D orientation angle versus the observer. Scene objects that do not obe plane equations are considered as obstacles. The obstacles are labelled, their location and shape features are identified. 3. PLANE DETECTION In the first step of the algorithm the depth image is divided up into regular mesh of isosceles right triangles (see Fig. 3). Within the area of each triangle the criterion for correct depth estimation is verified as indicated in the previous section. This is an important step, because it eliminates triangles containing non-tetured surfaces and the ones that belong to different scene objects. Then, 3D coordinates of scene elementar points, indicated b triangle vertices are calculated. These points serve for writing surface equations defined b each of the isosceles triangles, which in turn allow for associating a normal vector with each of the triangular shaped surface as shown in Fig. 4. Figure 4 Normal vectors associated with the constructed tringles Initiall the triangles form regular mesh as shown in Fig. 3. Further, ever triangle is compared with each other, and if the triangles fulfil a given similarit measure, the are merged into larger geometric figures so forming seeds of the local planes regions for scene objects detection procedure. An inner product of normal vectors associated with the triangles is used as a similarit measure: s a b = a b + a b + a b = (1) If a given triangle does not match the plane equation (i.e., T T is a predetermined threshold) the s < s, where s triangle is removed from the plane triangle candidate list, but stored for further processing. After several iteration steps (3-5 iterations for the tested scene images) the triangles are labelled to the ones representing the detected planes. In each such step the number of labelled triangles is updated and concurrentl so is updated the plane equation so that the best fit (according to the least means squares criterion) to the preselected set of triangles is achieved. Finall, the most numerous sets of labeled triangles are considered to form fragments of scene planes. An algorithm for determining equations of the wall planes and for counting their number was implemented. Due to poor depth resolution of the disparit image (8 cm stereovision base, note the envisioned application as the travel aid for the blind) the sie of triangle grid should be chosen carefull. Too small a sie results in man identified triangles that are falsel positioned perpendicularl to the Z (depth) ais. On the other hand, too large a sie complicates plane detection due to large number of triangle vertices covering different objects and plane regions. Eperiments show that good estimate for grid sie is 16 piels, for depth map resolution of piels. 4. TRACKING OF PLANES IN 3D SCENE Let a plane eisting in a 3D scene is given b equation (2) with an assumption, that the origin of the coordinate sstem associated with the camera sstem does not belong to this plane. A (2) + B + C +1 = 0 Figure 3 Depth map generated for the scene shown in Fig. 8 and divided up into regular mesh of triangles

3 Our objective is to track the detected plane, i.e. update its i.e. update its coefficients. The plane is uniquel defined if its normal vector and the coordinate of a single point that belongs to the plane are given. An normal vector of the plane given in (2) is defined b three coefficients [ A,B,C] and points satisfing the plane equation, can be simpl found b using an straight line parallel to the normal vector hitting the plane. Having calculated camera motion parameters [8] it is possible to transform scene coordinates and vector coordinates in the old (for frame t) camera location correspond- =, to the camera coordinate ingl (,, ), u [ u ] T u, u in the new (for frame t+1) camera location ( ', ', ' ) [ u', u', u' ] T u '= : βu γu αu + γu + αu + βu ' = + Xdt & = U βz + γy ' = + Ydt & = V γx + αz ' = + Zdt & = W αy + βx, where V = [ X & Y& Z& ] T is the velocit of scene points in camera coordinate sstem [11]. Camera ego-motion parameters in each subsequent image frames, allow for skipping plane detection procedure, hence reducing the computing demand required for plane tracking. Additional advantage of the proposed plane detection procedure is that the detected planes can still be robustl tracked while their large parts become occluded b other voluminous obstacles or objects in movement. Note, however, that long term plane tracking the algorithm for plane detection needs to be reinitialied. 5. OBSTACLES DETECTION AND LABELLING Once scene planes are detected the disparit (depth) image is segmented into two sub-regions, i.e. planes and objects (obstacles). The segmentation relies on testing the following condition for ever piel ( ) map: (3) (4) d, in the dense disparit d, ) d plane (, ) Th ( (5) where d plane (, ) is the disparit value for points identified as belonging to an of the detected scene planes. The threshold value Th is a constant set eperimentall. If this threshold is too low then plane regions are falsel segmented as objects, conversel, for to high a threshold fragments of larger scene objects (obstacles) are detected onl. Images in Figs. 5 and 6 show the detected objects for different threshold values Th for the scene shown in Figure 7. Figure 5 Segmented scene image for Th = 0.7 Figure 6 Segmented scene image for Th = 1.2 Once image regions corresponding to scene objects (obstacles) are detected their location in 3D scene and their approimate sie is identified. Object labelling is carried out b appling a region growing procedure for each object region with a predicate defining the threshold T ob for disparit variations. If the disparit variations for the analsed region are lower than the threshold, region piels are assigned the same label, otherwise, new object label is created for the region. The regions are built from image blocks of sie piels. Results of this segmentation procedure for different threshold values T ob are shown in Figures 7 and 8. The lower the threshold the larger the number of false objects are detected. For higher threshold values, some objects are glued to the pre-detected planes. To solve this problem we propose to find objects seed points first b using higher T ob values and then lowering this threshold once the object is labelled. This method proved to generate stable labellings of scene objects. Please observe superior object labelling results shown in Figure 9 to the results shown in Figs. 7 and 8. Due to decreasing computational accurac of distant feature points detection of far awa objects is of poor qualit. This eplains wh object no 5 in Fig. 7 after parameter update is no longer detected. For the purpose of the considered application the assumed maimum object distance is intentionall limited to 4-5 m.

4 Figure 7 Object detection for threshold T ob =0.7 Figure 10 OpenGL 3D rendering of a scene from Fig. 8. Figures 11 and 12 show analsis results for the outdoor scenes. Figure 8 Object detection for threshold T ob =1.4 Figure 11 Outdoor scene segmentation result Figure 9 Objects labelling result obtained b using T ob =1.4 for initial object detection and T ob =0.7 for region growing procedure 6. RESULTS The proposed algorithm was tested on real scene stereovision sequences recorded inside and outside the universit building. The sequences were recorded using stereovision camera sstem (two PointGre colour digital cameras). Plane and obstacle detection results are visualised using OpenGL graphics libraries. Detected planes and obstacles are rendered using triangles identified b their corresponding normal vectors. OpenGL rendering for the image scene illustrated in Fig. 9 is shown in Figure 10. Figure 12 OpenGL 3D rendering of an outdoor scene The described scene segmentation procedure was verified using OpenGL simulated sequences. The OpenGL package allows not onl for generating stereovision sequences for a moving camera sstem, but also for reproducing disparit maps with subpiel accurac. We have used this tool for testing our scene obstacle localiation method.

5 8. ACKNOWLEDGEMENTS This work was supported in part b the Ministr of Science and Higher Education of Poland research grant no. 3T11B in ears and in part b the Mechanism for Support of Innovative Ph.D. Student Research (WIDDOK) grant no. Z/2.10/II/2.6/04/05/U/2/06 financed b the European Social Fund and the Polish Ministr of Econom. Figure 13 Test scene rendered using OpenGL package We have compared area of objects visible surfaces computed precisel from the OpenGL package and objects surface areas estimated from stereoscopic images and b emploing the proposed algorithm in which objects area is calculated as a sum of triangular areas forming the detected objects (see Figure 13). Results of the true and estimated object areas and distances are listed in Table 1. Table 1 Estimation results of objects true surface area and their distance from the camera sstem Object Distance (arbitrar units) Area (arbitrar units) True Estimated True Estimated Sphere on the left Sphere on the right Cone CONCLUSIONS A procedure for object detection from depth map sequences is implemented under the assumption of a freel moving stereovision sstem within a stationar scene (no objects in motion). The main result is objects (obstacles) localiation and their area estimation. Also, if there are planar elements within the scene the are appropriatel detected and their orientation versus the observer identified. This has an important practical consequence for navigating the sstem in the environment since the detected ground plane region indicates scene terrain free of obstacles. The aim of this research is to develop a passive navigation sstem, capable of obstacle detection that will serve as a model for the electronic travel aid for the blind. The concept is to convert the detected obstacles into spatialied auditor icons reflecting sie and location of the obstacle within the scene. Currentl, the work is under wa in which the ke elements of the presented scene segmentation procedures are being implemented on a DSP platform for real time sstem operation. REFERENCES [1] G. Garibotto, and C. Cibei, "Title 3D scene analsis b real-time stereovision,'' in Image Processing, ICIP IEEE International Conference, Ponań, Poland, September , pp. II [2] D. Lefee, S. Mousset, M. Bertoi, and A. Bensrhair, "Cooperation of passive vision sstems in detection and tracking of pedestrians,'' in Intelligent Vehicles Smposium, 2004 IEEE, Parma, Ital, June , pp [3] Y. Huang, "Obstacle detection in urban traffic using stereovision,'' in Proc. Intelligent Transportation Sstems, 2005, Vienna, Austria, September , pp [4] J. Fernande, and J. Aranda, "Image segmentation combining region depth and object features,'' in Proc. 15th International Conference on Pattern Recognition, 2000, September , pp vol. 1. [5] N. F. Kim, and Jai Song Park, "Segmentation of object regions using depth information,'' in Proc International Conference on Image Processing, ICIP '04, October , pp vol. 1. [6] B.K. Quek, J. Ibane-Guman, and K. W. Lim, "Featurebased perception for autonomous unmanned navigation,'' in Proc. Industrial Electronics Societ, IECON nd Annual Conference of IEEE, November , pages: 6 pp. [7] F. Wong, R. Nagarajan, and S. Yaacob, "Application of stereovision in a navigation aid for blind people,'' in Proc Joint Conference of the Fourth International Conference on Information, Communications and Signal Processing, 2003 and the Fourth Pacific Rim Conference on Multimedia, December , pp vol. 2. [8] P. Skulimowski, and P. Strumiłło, "Refinement of disparit map sequences from stereo camera ego-motion parameters", International Conference on Signals and Electronic Sstems, Łódź, 2006, vol. 1, pp , [9] P. Skulimowski, and P. Strumiłło, "Detekcja płascn w sekwencji obraów stereowijnch", V Smpojum Naukowe Techniki Pretwarania Obrau, Serock, 2006, pp (in Polish). [10] H. Muhlmann, D. Maier, R. Hesser, and R. Manner, "Calculating dense disparit maps from colour stereo images, an efficient implementation,'' in Proc. Workshop o Stereo and Multi-Baseline Vision, (SMBV 2001), Kauai, HI, USA, September 12 October , pp [11] A. R. Bruss, and B. K. P. Horn, "Passive Navigation,'' Computer Vision, Graphics, and Image Processing, vol. 21, No. 1, pp. 3-20, Jan