Image-Based Floor Segmentation in Visual Inertial Navigation

Transcription

1 -Based Floor Segmentation in Visal Inertial Navigation Gillem Casas Barceló, Ghazaleh Panahandeh, and Magns Jansson KTH Royal Institte of Technology ACCESS Linnaes Center Abstract This paper presents a floor segmentation algorithm for indoor seqences that orks ith single grey-scale images. The portion of the floor closest to the camera is segmented by jdiciosly joining a set of horizontal and vertical lines, previosly detected. Since the proposed method is not based on compting the vanishing point, the system can deal ith any kind of indoor scenes and adapts qickly to camera movements. A second contribtion is the detection of moving featres for points ithin the segmented floor area. Based on the estimated camera ego-motion, the grond plane homography is derived. Then, the expected optical flo for the grond points is calclated and sed for rejecting featres that belong to moving obstacles. A key point of the designed method is that no restrictions on the camera motion are imposed for the homography derivation. I. INTRODUCTION Segmenting the floor is a challenging problems in image processing and has a ide range of applications in the engineering field [], [2]. Speclar reflections and textred floors are the main difficlties faced by floor segmentation algorithms [3]. Besides, accrate approaches mst deal ith changes in the illmination and strctre of the scene [4]. In mobile robot navigation systems, visal-based techniqes are replacing historical schemes de to their simplicity and lo cost. Sch systems need to recognize the strctre of the scene and avoid both static and moving obstacles. Therefore, floor segmentation, moving objects detection, and obstacle removal become significant tasks for giding the robot ithin an environment [5], [6]. Floor segmentation is also becoming interesting to be sed in grond plane-based camera ego-motion estimation in vision-aided inertial navigation systems (INS), sch as [7] and [8]. Althogh the motion estimation in these methods is based on the grond plane featres, they do not specifically address the problem of floor detection. The designed method has been conceived to be integrated into the INS presented in [9]. A. Related ork Dring the past years, a large amont of ork related to obstacle avoidance and grond plane detection has been done (e.g., [0], []). Regarding single camera techniqes, Wang et al. [2] presented a region-based method that provides a local obstacle map at high resoltion in real-time. In [3], a homography-based techniqe is introdced to estimate the grond plane normal. Then, the floor is detected by compting plane normals from motion fields in image seqences. Hoever, these methods are restricted to static environments. On the other hand, there are also many methods in the literatre focsed on otdoor environments and detection of moving vehicles [4], [5]. In [6], the backgrond motion is estimated and compensated sing an affine transform, hile [7] takes advantage of knoing the camera ego-motion and exploits spatial constraints to detect the motion in a second step. Odometric information is sed by Braillon et al. [8] to model the expected motion of the points on the grond plane. The location of the moving obstacles is determined by the points that do not follo this model. Hoever, to the best of or knoledge, only fe methods specifically face the problem of floor segmentation (e.g., [2], [4]). The most similar approach to the one presented in this paper as implemented by Li and Birchfield [3]. They designed a techniqe, applied to single color images, that combines three visal ces for evalating the likelihood of horizontal intensity edges to be part of the all-floor bondary. Since their algorithm comptes the vanishing point, it is restricted to typical corridor scenes and adapts sloly to camera movements. B. Approach overvie Unlike other single image methods, sch as [3], or proposed system orks ith grey-scale images and does not reqire to compte the vanishing point. Conseqently, it adapts faster to changes in camera motion and is able to deal ith all types of indoor scenes. A floor polyline is defined, containing the all-floor and floor-obstacles bondaries. In order to dra this polyline, an acte ay of joining the most important lines from an edge detector is applied. Finally, a mask describing the floor area is generated. A relevant attribte of or ne approach is that only the bondaries belo the half image are considered, assming that a sfficient part of the floor ith enogh featre points is ithin this region. After segmenting the floor, featre extraction and matching is performed beteen consective frames. Based on the estimated ego-motion of the camera, the homography matrix for the grond plane is calclated. In contrast to [8], hich is restricted to forard motion, the homography in or method is derived for general motion and rotation of the camera. Then, the expected optical flo is compted for featres ithin the floor mask at the crrent frame. Moving featres are detected

2 Edge and Line Detec5on Single Edge detector Hogh Transform (a) (b) Horizontal lines Ver+cal lines Floor Polyline Sketching Mask and polyline from previos frame (c) (d) Floor Mask Genera5on Fig.. General scheme for the proposed floor segmentation method. by comparing their expected and real motion, given by the estimated optical flo and the correspondences, respectively. II. FLOOR SEGMENTATION ALGORITHM As shon in Fig., the proposed method is divided into three main blocks: ) edge and line detection, 2) floor polyline sketching and 3) floor mask generation. In this section, the three parts are discssed. A. Edge and Line Detection The first block of or algorithm consists of detecting, identifying and describing the main lines of the image, hich define the strctre of the scene. The detected lines are divided into to grops: horizontal and vertical. First of all, the Canny edge detector [9] is applied to detect the edges from the given pictre (see Fig. 2b). Then, a list of all the points belonging to every edge is generated from the Canny mask. De to real orld noisy conditions, short sprios edges that are not important regarding the scene strctre description might appear. Ths, edges shorter than 60 pixels are removed at this point. Since the algorithm to dra the floor polyline reqires straight vertical and horizontal lines, each edge is fitted into a set of straight segments. In order to prne and classify the line segments, the Hogh transform is applied [20]. In the transformed domain, line segments are divided into to sets: vertical (listv) and horizontal (listh). Based on the tested seqences, a slope range to classify the line segments is determined. A line is classified as vertical if its slope is ithin ±0 of the vertical direction. Horizontal segments are given a ider slope range: ±65 of the horizontal direction. The rest of the lines are rejected. In grond plane-based ego-motion estimation approaches, the grond featres closest to the camera have the main contribtion to the motion estimation. Since or method is designed for sch applications, it is not spposed to segment the hole floor, bt only a sfficient part that is the closest to the camera. Hence, lines above the half of the image ill not be taken into accont, redcing the comptational cost of the algorithm. Applying the half image constraint entails: (e) Fig. 2. Partial images from Floor Segmentation Algorithm. (a) Original image. (b) Canny edge detector mask. (c) Detected edges, each edge is painted in a random different color. (d) Line segments from detected edges. (e) Horizontal and vertical lines after classifying them in Hogh domain, red line is at of the image. (f) Remaining lines after applying the half image 2 constraint. Vertical lines hose bottom points lie above it are removed. Vertical lines ith both top and bottom points belo it are removed. Horizontal lines hose beginning and ending points lie above it are removed. Horizontal lines ith jst one beginning/ending point belo it are ct. The ne beginning/ending point is set at the point of the line that corresponds to the y coordinate vale eqal to one half of the height of the image. B. Floor Polyline Sketching A polyline representing the all-floor and floor-obstacles bondaries at the bottom half image is dran by jdiciosly joining lines from the to lists defined in Section II-A. Knoledge of the height and orientation of the camera, as ell as typical strctre of indoor scenes and geometric constraints are taken into accont to select hich lines to se. The main idea is to dra a polyline, from left to right, connecting the endings of the horizontal and vertical lines one progressively enconters. For every iteration in the main algorithm (see Fig. 3), the first step is to find hich point, ithin listh and listv, is most to the left. While there is still some element in any list, the line that has its beginning point (in the case of horizontal lines) or its bottom point (in the case of vertical lines) most to the left is selected. Dring all the procedre, the coordinates of the last point of the line segment that is being dran are stored (last, lasty ). When a vertical line is chosen, its bottom point is directly sed for draing the floor polyline. Hoever, in order to (f)

3 listh & listv empty? Find point most to the le>. First line? Hori- zontal line? *Condi- 7ons (a) (b) Remove line from listh. Hori- zontal line? *Condi- 7ons *Act, Act3, Act4, Act5, Act6. *Act, Act3, Act6. *Act2, Act3, Act6. listh & listv empty? GO TO TOP Remove line from listh. *Act2, Act3, Act4, Act5, Act6. Dra a horizontal line at ½ of the image. Join the last point ith the right corner by straight line. line dran yet? Act: Join point to {last,lasty} by a straight line. Condi4ons: (at least mst be accomplished) Act2: Join point to le> corner by a straight line. Larger than 50 pixels? Act3: Update last and lasty Orienta7on <±60º AND beginning/ending Act4: Follo the hole horizontal line. point closer than 60 pixels from a corner? Act5: Reset beginning points of horizontal lines at the le> Beginning/ending point closer than 45 pixels of last to (last+,f(last+)), f(x) refers to the eqa7on from a bodom point of a ver7cal line? of the modified line. Remove all lines at the le> of last. Act6: Remove line from list. Fig. 3. Flochart of the Floor Polyline Sketching algorithm. avoid the effect of sprios edges that might appear becase of textred floors or speclar reflections, three conditions are checked hen a horizontal line is selected. At least one of these conditions mst be satisfied to consider the horizontal line as a segment of the floor polyline. The to first ones are related to ho the all-floor bondary shold look like, hile the third one is to ensre obstacle detection. The first condition arises becase horizontal lines that are part of the all-floor bondary are expected to be long. De to the height and the orientation of the camera, they are also expected to describe a certain angle ith respect to the vertical direction, hich flfills the reqirements of the second condition. For possible obstacles, both the horizontal line of its base and the vertical lines of its edges are detected, so the third condition holds. When all the lines have been analyzed, the last considered point is joined ith the right corner of the image by a horizontal straight line, giving the final floor polyline. If no line has been sed so far, the hole bottom part of the image is assmed to be part of the grond. Conseqently, the floor polyline becomes a straight horizontal line at one half of the height of the image. C. Floor Mask Generation A first version of the mask is generated by setting the pixels belo the polyline to hite. The rest of the pixels in the image remain in black. The area of the floor region is compted by smming-p the nmber of hite pixels ithin the mask Npix t. In order to avoid sdden changes that might redce dramatically the floor area, Npix t is compared ith STOP (c) Fig. 4. Floor polyline and mask generation. (a) Original image. (b) Lines sed to define the floor polyline. (c) Final floor polyline. (d) Final floor mask. the area defined by the mask at the previos frame Npix t. If the ne area is more than 30% smaller than the area of the floor region at the previos frame Npix t < 0.7Npix t, the method keeps the same floor polyline and the same mask as the previos frame. Real obstacles do not appear fast enogh in order to redce more than 30% of the region of the floor from one frame to the next one. On the contrary, changes in the illmination of the scene or textred floors can case this effect and the system mst ignore them. (d) III. MOVING FEATURES DETECTION Althogh moving objects ith ell-defined edges are rejected by the floor mask, irreglar moving obstacles, sch as feet, might partially lie ithin the mask. To avoid selecting featres belonging to moving obstacles, a moving featres detection method for featres inside the mask has been designed. After establishing featre correspondences beteen consective frames, the expected optical flo of the grond points is compared ith the real motion of each featre correspondence in order to detect moving featres. Using the provided camera ego-motion estimation from [9], the homography matrix H for the grond plane is derived folloing a similar procedre as the one presented in [8]. Then, the expected optical flo f(, v) is compted from H, only for the featre correspondences inside the floor mask at the crrent frame. The proposed soltion in [8] is derived for the specific case here the orientation of the camera relative to the grond is fixed and the motion is straight forard. In contrast to their method, here the derivation of H is generalized. The details of the this derivation are given in the Appendix. A. Featre Prning In the first step, featres are removed if their optical flo vector and their correspondence vector angles differ more than a certain threshold. For the tested pairs of frames, this threshold is heristically set to 20. Mismatches and featres ith a significant different motion direction from the motion of the camera are removed right at this point. In order to prne remaining moving featres, a similarity vale is calclated.

4 err pdf frame26 frame (a) density 50 (b) Fig. 5. Prning of moving featres. (a) All featres correspondences beteen frames ith their corresponding estimated optical flo. (b) Correspondences after applying the floor mask and prning moving featres To begin ith, the intensity difference beteen every featre in the crrent frame and its expected position in the previos image is compted. This difference vale is defined as the sm of sqares differences (SSD) ithin a neighborhood region arond the to positions: D= [It ( + i, v + j) It t ( + i, v + j)]2 () (i,j) W here It (, v) is the intensity vale of a featre at (, v)t in the crrent frame and It t (, v ) is the intensity vale of the featre in the previos image (, v )T = (, v)t tf~(, v)t. For the tested pairs of frames, the neighborhood indo size is set to 9 9, hich reaches a good compromise beteen performance and comptational time. Assming constant illmination, D tends to zero for featres on the grond, hile it has greater vales for featres belonging to moving obstacles. Then, a similarity vale for every match is defined as D, here D is the normalized difference. This implies that the similarity is beteen 0 and, giving an idea of the likelihood of each featre correspondence to be part of the grond. Finally, the similarity vale is thresholded and all featres hose similarity vale is belo a threshold are removed. This parameter reglates the amont of featres that are rejected and can be optimized at every frame. For the tested seqences, it is set to 0.7, hich shos a good performance (see Fig. 5). IV. E PERIMENTAL R ESULTS The implemented algorithms have been tested sing several forard-looking seqences reflecting typical indoor scenes challenges, sch as speclar reflections or textred floors. An AVT Gppy monochrome camera as sed, hich generated images ith a resoltion of 752x480 pixels, 8 bits and 0Hz. It as rigidly monted at the top of a trolley, at 85cm height, and shifted 25 toards the floor to maximize the floor area belo the half of the image. Examples of the recorded seqences as ell as the floor segmentation otpt videos for all the tested seqences are available at the YoTbe channel [2]. In order to qantitatively evalate the performance of the floor segmentation algorithm, the folloing test as carried ot. Up to 500 random frames from the set of recorded seqences ere manally labeled yielding a grond trth all-floor and floor-obstacles bondary at the bottom half part of the image. The error of the algorithm is defined as err vales Fig. 6. Error measrement (err) pdf for the 500 labeled images. The red vertical line is sitated at 0.8. the percentage of the area of the difference, beteen the grond trth and the estimated polyline, ith respect to the total area of the grond trth floor. This difference can be compted as the sm, over all the colmns in the image, of the difference beteen the y coordinates of the grond trth and the estimated polyline as: P y (x) ygt (x) (2) err = x N pixgt Fig. 6 depicts the probability density fnction of err over all the labeled frames. tice that, for most of the tested frames, the err has a very lo vale. Considering the floor polyline estimation as a sccess if err < 0.8, or approach correctly detects the floor region in the 89.8% of cases. Fig. 7 presents a set of sample otpt frames of the algorithm. These reslts prove that or approach can be applied in different environments, even ith the presence of speclar reflections, changes in the illmination and textred floors. Besides, it is able to deal ith big moving obstacles ith ell-defined edges. Hoever, tests have proved that the floor segmentation method itself is not able to reject small irreglar obstacles, sch as people s feet, from the floor mask (see Fig. 8). Hence, a moving featres detection method has been implemented as ell. On the contrary, no qantitative performance evalation has been carried ot for the moving featres detection. Tests have proved that the optical flo estimation fails in arond 40% of the analyzed frames. Nevertheless, hen a correct estimation is achieved, the moving featres prning method, together ith the floor mask, is able to remove more than 90% of the featres belonging to moving obstacles (see Fig. 5).

5 V. CONCLUSIONS AND FUTURE WORK This paper has presented a method to segment the floor and detect moving featres. The floor segmentation algorithm can deal ith difficlt illmination conditions, speclar reflections and static obstacles. Moreover, it has been proved to be able to adapt qickly to changes in the scene. The estimated motion parameters of the camera are assmed to be knon for detecting moving featres. A noteorthy contribtion of the designed method is that no restrictions on the camera motion are introdced for the grond plane homography estimation. Hoever, reslts have proved that the optical flo estimation only holds in arond 60% of the tested examples. So far, no pattern is detected in the erroneos estimations. A deep analysis mst be carried ot to find the cases of this lo percentage of effectiveness. We believe that exploiting temporal information in the homography estimation old increase the performance. APPENDI HOMOGRAPHY AND OPTICAL FLOW DERIVATION The projection eqation for general motion of the camera of 3D points into the image pixel coordinates is: v = KRb c [ R b Rn p ] b Y n Z (A.3) W orld here K is the matrix of the intrinsic camera parameters; Rb c is the direction-cosine matrix that rotates a vector from camera to body-pixels and it is constant all along the seqence; vector p contains the estimated position of the camera at the crrent frame; Rn b is the matrix that rotates from body to navigationframes. It is defined by the angles of the rotated coordinate system (eqation (2.3) in [22]) as: Rn b = R R 2 R 3 R 2 R 22 R 23 (A.4) R 3 R 32 R 33 R = cos(ψ) cos(θ) R 2 = sin(ψ) cos(θ) R 3 = sin(θ) R 2 = sin(ψ) cos(φ) + cos(ψ) sin(θ) sin(φ) R 22 = cos(ψ) cos(φ) + sin(ψ) sin(θ) sin(φ) R 23 = cos(θ) sin(φ) R 3 = sin(ψ) sin(φ) + cos(ψ) sin(θ) cos(φ) R 32 = cos(ψ) sin(φ) + sin(ψ) sin(θ) cos(φ) R 33 = cos(θ) cos(φ) (A.5a) (A.5b) (A.5c) (A.5d) (A.5e) (A.5f) (A.5g) (A.5h) (A.5i) here H is defined by: and J = H = KR c bj R R 2 R p x R 2 p y R 3 p z R 2 R 22 R 2 p x R 22 p y R 23 p z R 3 R 32 R 3 p x R 32 p y R 33 p z (A.7) (A.8) the temporal derivative of eqation (A.6) is compted: v ẇ = Ḣ Y (A.9) Grond The final relation, combining eqations (A.6) and (A.9), becomes: here v ẇ = ḢH Ḣ = KR c b J v (A.0) (A.) By definition, the optical flo can be expressed as: ( ) f(, v, ) = ( ), ( v ) = ( ) ẇ v vẇ, 2 2 The folloing transformations are sed: ( ) v = ( ) and v v = ẇ v = ḢH v ẇ v ẇ (A.2) (A.3) (A.4) The final expression for the optical flo is then defined as: ( f(, v ẇ ) ) = v v ẇ (A.5) Assming that the grond is flat and located at Z = 0: v = H Y (A.6) Grond

6 Fig. 7. Floor Segmentation Algorithm performance. Random frames from three different seqences proving that the method can deal ith speclar reflections, textred floors and both static and moving obstacles ith ell defined edges. Fll seqences can be fond in [2]. Fig. 8. Floor Segmentation Algorithm performance. tice that the feet are partially inside the defined floor area, in most of the cases. REFERENCES [] Z. Chen and S. Birchfield, Visal detection of lintel-occlded doors from a single image, in IEEE Compter Vision and Pattern Recognition Workshops (CVPR), 8, pp. 8. [2] D. Lee, M. Hebert, and T. Kanade, Geometric reasoning for single image strctre recovery, in Compter Vision and Pattern Recognition (CVPR), 9, pp [3] Y. Li and S. Birchfield, -based segmentation of indoor corridor floors for a mobile robot, in Intelligent Robots and Systems (IROS), 200, pp [4] N. Pears and B. Liang, Grond plane segmentation for mobile robot visal navigation, in IEEE Intelligent Robots and Systems,, vol. 3, pp [5] D. Conrad and G. DeSoza, Homography-based grond plane detection for mobile robot navigation sing a modified EM algorithm, in Proc. of ICAR, May. 200, pp [6] G. Panahandeh, N. Mohammadiha, and M. Jansson, Grond plane featre detection in mobile vision-aided inertial navigation, in Intelligent Robots and Systems (IROS), 202. [7]. Song, L. Seneviratne, and K. Althoefer, A kalman filter-integrated optical flo method for velocity sensing of mobile robots, Mechatronics, IEEE/ASME Transactions on, vol. 6, no. 3, pp , 20. [8] C. Hide, T. Botterill, and M. Andreotti, Lo cost vision-aided im for pedestrian navigation, in Ubiqitos Positioning Indoor Navigation and Location Based Service (UPINLBS), 200, pp. 7. [9] G. Panahandeh, D. Zachariah, and M. Jansson, Exploiting grond plane constraints for visal-inertial navigation, in Position Location and Navigation Symposim (PLANS), 202, pp [0] T. Wekel, O. Kroll-Peters, and S. Albayrak, Vision based obstacle detection for heeled robots, in IEEE Control, Atomation and Systems (ICCAS), 8, pp [] P. Lombardi, M. Zanin, and S. Messelodi, Unified stereovision for grond, road, and obstacle detection, in IEEE Intelligent Vehicles Symposim, 5, pp [2] H. Wang, K. Yan, W. Zo, and Y. Peng, Real-time obstacle detection ith a single camera, in Indstrial Technology, 5, pp [3] Y. Kim and H. Kim, Layered grond floor detection for vision-based mobile robot navigation, in IEEE Robotics and Atomation, 4, vol., pp [4] B. Jng and G. S. Skhatme, Detecting moving objects sing a single camera on a mobile robot in an otdoor environment, in in International Conference on Intelligent Atonomos Systems, 4, pp [5] J. Odobez and P. Bothemy, Detection of mltiple moving objects sing mltiscale mrf ith camera motion compensation, in Processing (IEEE), 994, vol. 2, pp [6] A. Behrad, A. Shahrokni, S. A. Motamedi, and K. Madani, A Robst Vision-based Moving Target Detection and Tracking System, in and Vision Compting conference,. [7] J. Klappstein, F. Stein, and U. Franke, Monoclar motion detection sing spatial constraints in a nified manner, in IEEE Intelligent Vehicles Symposim, 6, pp [8] C. Braillon, C. Pradalier, J. Croley, and C. Lagier, Real-time moving obstacle detection sing optical flo models, in IEEE Intelligent Vehicles Symposim, 6, pp [9] J. Canny, A comptational approach to edge detection, Pattern Analysis and Machine Intelligence,986, vol. PAMI-8, no. 6, pp [20] P. Hogh, Method and means for recognizing complex patterns, U.S. Patent , 962. [2] Yotbe channel ith tested seqences. Available online from: videos?flo=grid&vie=. [22] M. B. Jay Farrell, The Global Positioning System and Inertial Navigation, st ed. McGra-Hill Professional, 998.