2 Sensor Feature Extraction 2. Geometrical Constraints The three{dimensional information can be retrieved from a pair of images if the correspondence

Transcription

1 In: Proc. of the 3rd Asian Conference on Computer Vision, Hong Kong, Vol. I, January 998. Identication of 3D Reference Structures for Video-Based Localization? Darius Burschka and Stefan A. Blum Laboratory for Process Control and Real-Time Systems Technische Universitat Munchen D-8333 Munchen, Germany Abstract The bootstrap{problem of self{localization in indoor environments is a demanding task for initial localization and topological navigation. The ability to determine the position in an a priori known or already explored environment allows unsupervised use of mobile robots in environments such as private households. This paper presents our approach to identify a given set of known reference structures in a three{dimensional map of the local environment. This map is constructed from the data extracted by a line-based stereo camera system mounted on a mobile vehicle. We present the method used to identify objects and to compute the vehicle's position in a world frame. Motivation Navigation in indoor environments requires dependable knowledge about the robot's position and a precise model of the environment. Dynamic changes in the environment caused by human inuence or the operation of other autonomous mobile vehicles increase the dierences between the sensor data and the model prediction that is based on an a priori model of the environment. The a priori model becomes gradually useless [4, 5] as the environment changes. The explored information is often incomplete and uncertain. The signicance of the correct interpretation decreases compared to other possible matches. Therefore, continuous exploration of the changes in the environment is necessary. In [2], a three{dimensional model capable of storing the changing sensor data during an exploration of the environment was presented. The three{dimensional lines are stored at their geometrical position and are be used for prediction of future sensor readings or for obstacle avoidance. All objects are referenced to a local structure visible in this region. The size of this local region is restricted by the range of the applied sensor or the structure of the environment. For example, the walls of a room can reduce the possible size of the local region dened by the sensor range.? The work presented in this paper was supported by the Deutsche Forschungsgemeinschaft as a part of an interdisciplinary research project on \Information Processing in Autonomous Mobile Robots"(SFB33). 28

2 2 Sensor Feature Extraction 2. Geometrical Constraints The three{dimensional information can be retrieved from a pair of images if the correspondence between extracted elements (e.g., lines) is established and the exact position of the cameras is known. The number of possible matching candidates can be reduced by applying constraints such as the epipolar constraint, uniqueness and continuity [3]. Our sensor system computes the 3D information for the endpoints of the detected lines []. It is almost impossible to align the two optical systems precisely and stably to a parallel arrangement. This orientation error must be taken into account to get accurate results. Most algorithms compute the geometry of the epipolar lines for a given situation. We propose an alternative approach for transforming the original data into the parallel case with minimum computation. For example, a rotation ' around the y{axis results in the following transformation of the image coordinates: x cos '+sin ' p = x z = x z y p = y z =? x z xpcos '+sin ' sin '+cos' =?xpsin '+cos ' y z? x sin '+cos' = yp z?xpsin '+cos' : () The rotation around the z{axis results in a simple rotation of the image coordinates. The advantage of these equations is that the world coordinates of the projected obstacles may remain unknown. We use these equations () to transform the camera system to the parallel cameras case. The initial orientation errors are estimated in an o-line calibration of extrinsic and intrinsic camera parameters [7]. The orientation of the cameras may change during a mission due to vibrations and camera repositioning errors. It is corrected in an on-line re-calibration method based on the explored information [2]. 2.2 Virtual 3D Sensor In our system we use a model of a \virtual 3D sensor" situated between the two cameras. The properties of this sensor can be achieved by dierent real sensors. It is possible to replace the stereo system by any other sensor capable of reconstruction of three{dimensional lines. 3 Dynamic Local Feature Map 3. Possible Input Sources The Dynamic Local Map (DLM) stores a local region at the abstraction level of the applied sensor system to support topological modeling. The data stored in the DLM come from dierent sources [2]. The most important source is the sensor system, which registers the recent changes in the environment. Another source is a global model of the environment that stores reliable information generated from CAD{models or previously-veried lines. The map also stores hypothetical features of the environment generated by the Predictive Spatial Completion (PSC) module. These hypothetical features are based on statistics and are used to control the path planning and to support the sensor system. 29

3 3.2 Data Formats Currently, we store only line shaped features described by their three-dimensional endpoints and orientation. This information is rened in consecutive steps. In addition, each feature in the map is described by its condence and accuracy. This information is used in navigation tasks to decide, which features should be employed for localization to reduce the errors caused by 3D lines resulting from false correspondences in the stereo system. 3.3 Internal Structure Flaws in the sensor feature extraction can be reduced by comparing new explored data with the former sensor readings stored in the DLM. The DLM stores a local region of the environment described by the three{dimensional lines. Multiple and partially contradictory requirements forced us to develop a multi{level indexing structure. The upper layers consist of a two{dimensional grid containing octrees to store the explored features eciently and to minimize time-consuming memory transfers for feature updates or localization changes [, 2]. This structure adapts to the features' distribution in the local environment. 3.4 Update of the Stored Information Our fast reconstruction of the 3D information is based on fast interaction with the DLM. The possible match candidates for a given feature exceeding a specied quality value are stored in the DLM. Therefore, the DLM also stores false information. In consecutive sensor readings this information is veried from dierent positions. False features are deleted if they cannot be veried. Therefore, it is important that our sensor data processing not be free-running, but be triggered from the planning instance. In this way, sensor readings from dierent positions are guaranteed. Multiple readings from the same position would also stabilize the false features. This procedure permits the handling of moving objects in the environment. Each endpoint is described by its precision. The precision p of a detected endpoint depends on its distance from the camera. The precision of an endpoint describes its maximum position error and is adjusted each time the feature can be veried. The feature is specied by a condence value c for its existence. The condence value c is modied with an exponential function c(f) =? e?g(f old+f ) =? (? c old )e?gf ; f each time it can be matched. The value f describes the condence in the current step. It may vary from (only one matching candidate) to (useless information). The value g describes the speed at which this value is changed. It must be adapted to the applied matching algorithm. 4 Recognition Our approach is based on a hypothesize&test strategy underlying an interpretation tree. Because of the high computational cost of these algorithms, several tests for truncations are added, including rigidity and visibility constraints. Therefore, a relevant-rst search is implied. In our application, a set O = fo ; O 2 ; : : : ; O r ; : : : ; O R g of relevant objects that are described with 3D line 3

4 segments is presumed. Therefore, an object O r consists of a set of segment lines S r = fs ; S 2 ; : : : ; S j ; : : : ; S m g 2. Our approach to identify those a priori known reference structures from a list of line features S = fs ; S 2 ; : : : ; S i ; : : : ; S ng delivered from the DLM is composed of two parts: an o-line object preprocessing to gain a minimal set of relevant line segment triplets and on-line identication, triggered by these triplets. 4. Preprocessing Our approach to preprocess the reference objects is to nd some of their properties that are not explicitly derivable from their geometric description. We propose to use only the most relevant features for generation of an object's hypotheses. Grouping of three linearly independent line features at a time seems to be suitable for this application (section 4.2). The visibility of the groups is decided from a set of synthetic projections P = fp ; P 2 ; : : : ; P p ; : : : ; P Z g of the object's CAD-model generated for dierent aspects. A virtual camera is positioned at dierent locations on a spheric surface enclosing the model. A facility to exclude regions on the sphere's surface is provided. The visibility of each single line segment is determined for each projection with a Z-buering algorithm. Because only discrete points on the surrounding sphere are selected, a binary criterion v for the visibility of a line feature is sucient: v pjr = ; if 7% of the object's r segment j are visible in projection p ; otherwise. (2) We created a simple heuristic calculation of a rating R j for each line segment S j consisting of a rating r l of the segment length l j, an evaluation of the information content that is derived from a term A jj 3, and the number of aspects in which a segment is visible in all projections. R j = r l (l j ) (ld mx j = A jj + )? Z ZX p= v pj : (3) The obtained visibility information is used for the generation of triple line groups for each object. At least one visible segment triplet must exist in each projection P p. The number T of chosen triplets is considered to be as small as possible 4. A generated triplet is supposed to describe the sight of an object uniquely. The distances and angles between line segments should be as large as possible combined with a high rating R j. If possible, a segment line should only be used once for triplet generation because even high rated segment lines may not be detected, due to poor illumination for example. This leads to another heuristic rating for each triplet combination T ^ u : U u = (R u + R u2 + R u3 ) jdet(m u )j (d u;u2 + d u2;u3 + d u3;u ); (4) For occlusion check (described below), a plane-based description is also necessary. 2 In case of uniqueness, the index r will be left out in the following. 4 The cost of the online identication is proportional to T. 3

5 where M u is a matrix of the unit direction vectors, d uk;uk is the distance between the two line segments ^Suk and ^Suk 5. An iterative strategy is applied to extract the best triplet set reducing stepwise the number of projections and resulting in a number of 3 to 9 relevant triplets. 4.2 Identication Clustering of Image Features The lines delivered from the DLM are clustered rst. The implied clustering algorithm groups neighboring, connected segments, tolerating small errors. Clusters consisting of only a few line segments are deleted. The remaining position clusters are processed in another clustering step to build groups of approximately parallel segments called direction clusters. At least three direction clusters must emerge from each position cluster, otherwise the corresponding position cluster is deleted. This condition is necessary for unique determination of the object's pose. Initial Guess for 's Pose All emerged segment triplets are searched, primarily to build a correspondence tree for each position cluster without pose restrictions. Neighborhood relationships such as the angle and distance between two segment lines [6] are criteria for establishing correspondences between image and reference segments. Matching candidates for the second and third feature of a group must be searched only in certain direction clusters, reducing the number of required comparisons. A prehypothesis is established if all three reference segments of a triplet could be associated. Based on the prehypothesis we are able to guess the pose and orientation of the reference object. We dene two 3 3-matrices K and K, that store the unit direction vectors of the image and the reference triplets. A transformation between the reference frame and the image frame can be expressed as a matrix R = K K?, which is, in the best case, an orthogonal rotation matrix, in which small errors are tolerated. R is normalized in order to compute the three rotational degrees of freedom (DOF). The remaining three translational DOF are calculated by simply using a reference point for each reference triplet and associated image triplet. The translation can be determined by T = a + S? b, where the oset between the segment triplet's center and the object's center in the object frame is denoted as a. S is the oset between the position of center C T of model segment triplet in the object frame and the center CT of the associated triplet in the image space. Because b = R a, the translation can be denoted as T = S + (I? R) a, where I is identity matrix. Hypothesis Generation The probability of the existence of a certain object depends on the existence and, especially, on the accuracy of the associations of the predicted features. A segment wise linear accuracy function a(e) is dened by orientation accuracy a ('), longitudinal accuracy a l (u) and parallel accuracy a p (d). 5 Note that the number of possible combinations is about :25 m 3, assuming that the probability that a single line feature is visible in a single projection is about :5. 32

6 A total accuracy A ji of a pairing of predicted line segment ~ Sj and image line segment S i is obtained by A ji = a (') a l (u) a p (d): (5) Any non-zero accuracy value eects an association between S j and Si and is kept. The obtained accuracy of a model line segment S j is denoted as A j. A probability P of the object's prehypothesis is calculated by P m P j= = R j A j v p? j P m R. (6) j= j v p? j A probability of the hypothesis P is created with v p? j := : 8 j by using (6). Only if P and P exceed certain thresholds will a hypothesis be generated. Hypothesis Pruning Hypotheses belonging to the same object are fused if they are consistent with respect to their poses. Then their probability is recalculated. Often, two or more hypotheses are generated for dierent, but similar objects at similar pose. Even if an object has moderate symmetries, dierent hypotheses for its orientation may be created. In our approach, the object's hypothesis with the highest probability survives. The threshold for recognition P reg has to be chosen high in order to keep the probability for misdecisions low. Correction of Recognized 's Pose The method of gradient descent is applied to minimize a squared error sum S(t x ; t y ; t z ; ; ; ), which is dened by mx S = m i= mx = m i= jm i B i j2 + jm i2 B i2 j2 B 2 d 2 i + d 2 i2 ; (7) S i B M d E S j B2 M 2 d 2 B 2 where m is the number of predicted model segment lines to which any image line can be associated Results of Recognition The object recognition algorithm was tested on several objects by measuring the hypothesis' probability with dierent degrees of distortion. Here, a parallel shift error distorting image features within a range of [; p max ] and an orientation error of their direction within a range of [; max ] were simulated. The maximal errors were varied between cm and 5cm in 5cm increments and and 2 in 2 increments, respectively. In order to normalize and keep the results independent from a certain aspect, all features of a object's model were registrated into the DLM. The hypothesis' probability was averaged on 25 measurements at a time, the bounds for parallel accuracy were selected to cm d min 5cm, and orientation accuracy to 5 ' min. The threshold for 6 m is used only as a scale factor and has no inuence on the result of the gradient descent method. E 33

7 the prehypothesis' probability P was chosen to be zero to enable the desired behavior as stated above. Fig. shows the testing results of the objects "quader" (a) and the object "cubicle" (c). The inuence of a threshold was also examined, whereby the value.75 has to be exceeded for a hypothesis' probability to count as a recognized object. probability probability (a) 5 (c) 5 δmax 5 δmax p max 2 3 p max.5.5 recognition rate recognition rate 5 (b) 5 (d) 5 δmax 5 δmax p max 2 3 p max 5 Localization The pose estimation of the mobile robot is performed by comparing object's pose referred to the exploration frame to its reference pose in the landmark description. A recognized object ^O b consistent with its stored position and orientation (jz r? Z l j z max ; j r? l j max and j r? l j max7 generates a local hypothesis L b. Fig. 2 shows the relationships between the frame of a local map and the exploration frame for a single unique recognized object. The transformation parameters (X,Y,') stored in the local hypothesis are calculated as: cos '? sin ' X Y = Xl Y l? sin ' Fig.. Probability of object's hypotheses cos ' X l X Xm x local map frame Y ϕ xr ye x e x Y m y r mobile robot ϕ m x ϕ e y Y l y exploration frame αr α l ϕ y recognized object Fig. 2. Ane transformations between frames xr y r ; ' = l? r : (8) Hence, if possible local hypotheses can be generated for similar transformation parameters, an algorithm to fuse those candidates will be applied. If both of the following conditions are true, two local hypotheses L i and L j will be combined into a new local hypothesis denoted as L ij : D ij = j(x i ; Y i ) T? (X j ; Y j ) T j D max and ' ij = j' i? ' j j ' max : (9) The thresholds D max and ' max can be chosen depending on the required accuracy of the localization. The new generated combined local hypothesis L ij is provided with weighted mean values. Its quality is dened by Q ij = (Q i + Q j ) (? D ij 4 D max? ' ij 4 ' max ): () 7 r-indexed parameters are recognized parameters and l-indexed are stored landmarks parameters 34

8 This fusing procedure has to be repeated until all local hypotheses are combined. As a result, a set of combined and not combined local hypotheses W = fl ; L 2 ; : : : ; L b ; : : : ; L B g emerges. "table6x" (4) The highest qualied local hypothesis is D5 D denoted as L b? = (X? ; Y? ; '? ). The resulting "table6x" (3) "cubicle5x95" (2) C C5 B A B2 C3 D4 C4 "table6x" (5) D3 A "table+box" Fig. 3. The local hypotheses in an exemplary scenario transformation parameters (X?,Y?,'? ) are used to transform the stored DLM content. To determine the 3 DOFs (X m ; Y m ; ' m ) of the mobile robot referring to the local map frame from (X?,Y?,'? ), a similar ane transformation as in (8) is applied. This strategy was tested in a project room consisting of various table groups, walls, cubicles and other items (g. 3). The four landmarks A?D were a priori stored in the representation of the robot. The object recognition algorithm delivered ve identied objects ()? (5), whereby object (5) was a misinterpretation 8. As a result, eight local hypotheses emerged. Landmark A and B produced correct local hypotheses at each case. Because landmark C and D are related to the same object and three interpretations were created, six further local hypotheses were generated. 6 Future Work We plan to enhance particularly the precision of the explored information by fusing camera data with data retrieved from the laser range nder. A mission expert for topological navigation will be developed to use the presented tools. References. D. Burschka, C. Eberst, and C. Robl. Vision Based Model Generation for Indoor Environments. In ICRA97, pages 94{945, D. Burschka and G. Farber. Active Controlled Exploration of 3D Environmental Models Based on abinocular Stereo System. In ICAR97, pages 97{977, Monterey, California, USA, July Oliver Faugeras. Three-Dimensional Computer Vision. Massachusetts Institute of Technology, The MIT Press, Cambridge, Massachusetts London, England, A. Hauck and N. O. Stoer. A hierarchic world model supporting video-based localisation, exploration and object identication. In 2. Asian Conference on Computer Vision, Singapore, 5. { 8. Dec., pages (III) 76{8, Gunter Magin, Achim Ru, Darius Burschka, and Georg Farber. A dynamic 3D environmental model with real-time access functions for use in autonomous mobile robots. Robotics and Autonomous Systems, 4:9 { 3, G. Stockman. recognition and localization via pose clustering. Computer Vision, 4:36{387, Roger Y. Tsai. A Versatile Camera Calibration Technique for High Accuracy 3D Machine Vision Metrology Using O-the-Shelf TV Cameras and Lenses. IEEE Transactions of Robotics and Automation, RA-3(4):323{344, August 987. () 8 The recognized object was a smaller table that did not belong to the a priori knowledge of the scenario. 35