Bitangent 3. Bitangent 1. dist = max Region A. Region B. Bitangent 2. Bitangent 4

Transcription

1 Ecient pictograph detection Dietrich Buesching TU Muenchen, Fakultaet fuer Informatik FG Bildverstehen Munich, Germany 1 Introduction Pictographs are ubiquitous in many technical environments. Usually they are of high contrast and positioned so that they are visible from many directions. This facilitates their detection and makes it interesting for several tasks (e.g. automatic navigation of mobile robots, trac sign recognition). Using pictographs for automatic navigation could make the installation of special purpose landmarks unnecessary. In our work we allow for 6 degrees of freedom in the pose of the target pictograph. This makes simple comparisons (e.g. correlation) of model and image very time consuming. Instead we employ recognition by alignment (e.g. [5]). This method proceeds in several steps: 1) Feature points are detected in model and image. Possible features are for example bitangents (lines which are tangent to two dierent sections of a curve - usually to two sides of a concavity) and corners. 2) Pose hypothesis generation by matching of model and image features. 3) Hypothesis verication by comparison of model and image edges. We use ane transformations ximg y img = A xmod y mod + b as approximations of the transformation from model to image. This is reasonable as long as dierences in depth of object parts are small in comparison to the distance to the object. We extend the alignment approach to planar object recogniton by two contributions. We employ bitangents on pairs of image regions (see g. 1, right) for feature point calculation. Constraints limit the number of possible pairs that have to be considered during matching. Our second contribution is the selection of a small set of stable model features in a training phase. Similar to [1] we call these features "focus features". During recognition all selected model focus features are matched with all appropriate image features. The selection of a small set of focus features is therefore critical for ecient recognition. 2 The recognition algorithm These ideas were integrated in a system for pictograph detection. Recognition starts with a segmentation of the image in regions of homogeneous brightness. In the next step feature points are detected. Since an ane transformation is used, a minimum of three feature points are neccesary for hypothesis generation. The features used are simple bitangents on one region (a), pairs of such bitangents (b), tripels of consecutive corners on a region boundary (c) and bitangents on the convex hulls of pairs of image regions (d).

2 Some comments on these feature types: a) Bitangents are detected with the algorithm described in [3]. The point with maximal distance to the bitangent on the part of the curve between the two bitangency points is the third ane invariant point (see gure 1, left). b) Some pictographs consist of regions with more than one concavity. Often a least-squares solution for the transform based on matching pairs of bitangents is more robust. c) Corners are not ane invariant features, but some salient corners can be detected in a wide range of transformations. By using three concecutive corners the number of possible pose hypotheses remains linear in the number of corners. d) With an algorithm described in [2] bitangents on pairs of image regions can be computed in an expected time of O(log 2 (n)), where n is the number of vertices on the convex hull of the regions. This improves on a previous algorithm [4] that has a complexity of O(n). A focus feature consists of two bitangents with four invariant points (again a least-squares solution is used for pose computation). Figure 5 illustrates feature types a,b and d. A corner feature is used in the lower part of gure 3. P1 P2 Bitangent 1 Bitangent 3 dist = max Region A P3 Bitangent 2 Region B Bitangent 4 Figure 1: Left: Bitangent on a concavity. The points of tangency (P1 and P2) and the point of the curve between these two points with maximum distance to the bitangent (P3) are ane invariant. Right: Two disjoint convex regions and the four tangents touching both regions 2.1 Verication of pose hypotheses The verication of the generated pose hypotheses is very similar to the method used in [5]. Predicted model edgels are veried by nding an image edgel with similar orientation and the same polarity close to the predicted position. Currently used tolerances are 5 pixel in position and 30 o in orientation. The used model and image edges are the boundaries of regions found during segmentation. As in [5] verication is accelerated by the use of a distance transformation of the binary edge image. The fraction of transformed model edges found in the image is used as a measurement of the quality of a hypothesis. A hypothesis will be accepted if this fraction is above a threshold computed during the selection of focus features (section 3). A further acceleration is possible by using only a fraction (currently a fth) of the model edgels in a preliminary verication step. If the fraction of veried edgels is less than half of the threshold for acceptance, the hypothesis is immediately rejected without checking the remaining model edgels. For many pictographs the reliability of verication can be increased by the additional matching of intensities. This second stage of verication is only invoked if the fraction of edgels found in the rst stage exceeds the threshold for recognition. Intensities are matched for points perpendicular to the matched edge segments. In the present implementation points are sparsely sampled (every third edge segment, 2 pixel spacing in the direction normal to the edges) with a maximum distance of 8 pixels to the model edges. The position of matched image points is calculated relative to the matched image edges rather than to the transformed model edges. This ensures that the tolerance to small positional deviations of chamfer matching also applies to the matching of intensities. Information about the position of the matched image edges can be propagated during the calculation of the distance transform. In order to evaluate the match of intensities the histogramms of pixels matched to black and white model pixels are analyzed. Pairs of peaks in these two histogramms with a plausible dierence in intensity are considered. A pair of peaks is selected so that as many matched pixels as possible are similar (dierences can be attributed to noise) to the intensity of the peaks. The ratio of correctly matched intensity values is considered in the overall match quality measure. It would also be possible to use normalized cross correlation for the comparison of the intensities of matched model and image pixels. However for partially occluded objects there are often spurious

3 matched with bright model pixel matched with dark model pixel 300 No. pixel intensity Figure 2: Left : A correct object hypothesis; for white model edges a corresponding image edge was found. Right: Histogram of the intensities of image pixels corresponding to model pixels that are close to the matched edge pixels (see text). matches of model edges with the occluding clutter. If the intensity of nearby pixels is used for correlation the results are dicult to predict. The analysis of histogramms seems to be more robust because outliers are simply ignored. For pictographs with thin lines and other ne details intensity information was not very reliable (presumably due to smoothing). In the experimental evaluation of section 4 matching of intensities was only used for the recognition of partially occluded objects (which did not contain many ne structures). 2.2 Constraints for the matching of pairs of regions The main problem with the use of pairs of regions for the generation of pose hypotheses is the quadratic increase of the number of pairs with the number of regions. Constraints on image regions that can be matched with a given pair of model regions are used to prune this search space. One of the used constraints for the matching of regions is based on the ratio of the areas of pairs of model and image regions. This ratio is invariant against ane transformations. In the current implementation some dierences between model and image regions caused by instabilities in the localization of edges (up to 0.7 pixel) were accepted. Often some prior knowledge of the pose of an object is available. This knowledge implies constraints on possible ane transformations from model to image. A simple constraint concerns the area of image regions that can matched to model regions. s min and s max are bounds on the length of the vectors s 2 min A mod A img s 2 max A mod a11 a 21 and a12 a 22 derived from the transformation matrix A. The lengths of these vectors indicate the scaling of the x and y components of the model. They are inuenced by the distance of the object and its rotation out of the image plane. A mod and A img are areas of corresponding model and image regions.

4 Another constraint on possible correspondences of model and image regions is based on the amount of stretching or compresssion that is possible for a given vector between points in the model. Bounds on this value are caused by limitations on the rotation of the pictograph out of the image plane. They can be used to derive constraints on the ratio of areas of model and image regions and the distance of the pair of regions. The transformation matrix A can be written as a composition of a rotation described by a matrix R w1, a scaling in x- and y-direction and another rotation with matrix R w2 : A = R w1 sx 0 0 s y R w2 It is assumed that the ratio between the two values s x and s y is less than a value skew max. Without loss of generality we suppose that s x s y. It follows that 1 sy s x skew max. For the distances between the centers of image and model regions d img und d mod we dene: d rel = dimg d mod. All distances within the object are deformed by a factor that is bounded by s x and s y : In addition: Now it follows that: s x d img d mod s y =) s 2 x d 2 rel s 2 y (1) A img A mod = s x s y (according to the denition of s x and s y ) A img = s 2 s y x A mod s x s 2 x skew max d 2 rel skew max and also: A img = s 2 s x s y A mod s y 2 y skew max d2 rel skew max If s y s x and therefore 1 sx s y skew max, the inequality 1 is reversed and the resulting estimations remain the same. Another constraint on image regions implies that the mean intensity of an image region is within loose bounds (currently 150) of the intensity of the corresponding model region. For the recognition of the pictograph in the left part of gure 6 the range of possible scales was limited between 0.2 and 4.0 and the rotation out of the image plane was assumed to be less than For this example the mentioned constraints reduced the number of pose hypotheses generated by pairs of image regions from to Feature selection The major goal of the feature selection phase is to nd stable focus features which are necessary to compute good pose hypotheses. It is possible to compare features by analysing their properties. For example the used features tend to give good pose hypotheses when the distances between feature points are large. However this determination of feature stability by analysis or heuristics has some drawbacks. Feature detection is a complex process with several steps. It is dicult to describe the result of this process by some simple rules. Therefore in this work we evaluate the stabilities of features empirically using a set of training images of the given object (pictograph). Object poses in the training images should be representative for the range of object poses expected in object detection tasks. In the performed experiments a high resolution image of a given pictograph was articially transformed with dierent transformations in order to obtain the training images. One of the training images is used for the generation of candidate focus features. These are all bitangents, tripels of consecutive corners along a region boundary and pairs of bitangents on two image regions found in this training image. In order to limit the number of interregion bitangent features, only regions with an area above a given threshold are used. For every suitable pair of image regions two features are generated: the outer bitangents 1 and 2 and the inner bitangents 3 and 4. The stability of recognition with the dierent possible focus features is measured by using the remaining training images. Each focus feature is used to nd the object in these images. For this task the algorithms used in the production phase of recognition and described in the previous sections are employed. The stability of recognition is measured as the fraction of transformed model edgels found close to corresponding image edgels for the correct matching of the tested focus feature. This fraction is shown for two candidate focus features

5 in gure 3. The transformation from model to image pose is known for the training images. Therefore it is possible to decide if a generated pose hypothesis is correct. For every possible focus feature and all training images the quality of the correct pose hypothesis is determined. Based on this information one or more focus features are selected so that the minimum match quality of the best focus feature over all images M mingood = min im2im max f2f matchquality(f; im) (Im: test images, F: selected focus features) is exceeding a given threshold. A greedy search strategy is used for this problem. Figure 3: The evaluation of two focus features. Upper row: Models with focus features indicated. Lower row: Test of focus features with real images (veried edgels: white): fraction in left pictograph (bitangents on the leftmost and rightmost part of the suitcase): 0.93; in right pictograph (three consecutive corners on the right side of the glass): 0.73 In order to determine the detection threshold for the production phase the maximum match quality M maxbad using the selected focus features in a set of training images not containing the model object is computed. The threshold is then calculated as min(m mingood ; M mingood+m maxbad + (current implementation: = 1:0; = 2:5). 4 Results and discussion The proposed recognition method was used in experiments with 24 pictographs found at airports. They are shown in g. 4. For every pictograph the selected focus features were used to detect the unoccluded pictograph in 10 test images. 17 pictographs were detected in all 10 test images. 3 were missed once and the remaining 4 were not detected in 3-5 images. Two false positive detections occured for a single pictograph. Most of the diculties with recognition were caused by an instable segmentation of thin lines or small regions. The left part of g. 6 is a typical example for the recognition of an unoccluded pictograph. Fig. 5 also illustrates the selected focus features for some pictographs. For all pictographs 21 bitangents on pairs of regions, 3 simple bitangents, 5 pairs of simple bitangents and 2 triples of corners were selected. This demonstrates the high stability of hypotheses generated with bitangents on pairs of regions. Recognition time was measured on a HP K-260 Workstation (180 MHz). The segmentation of the test images of size pixel took 3-4 seconds. For the following matching phase between 1 and 3 s were used. 31 focus

6 Figure 4: The used 24 pictographs Figure 5: Examples for pictographs and the selected focus features: bitangents on pairs of image regions, except for "glass" (pair of bitangents) and "aircraft" (single bitangent)

7 features were selected from 996 candidate features for all 24 images. Therefore without feature selection matching would have taken about 32 times as long. A major fraction of candidate features were bitangents on region pairs. If these bitangents are used for complex models a selection of focus features is almost mandatory. In order to evaluate the recognition of partially occluded objects views of 6 pictographs were used (altogether 72 images). The degree of occlusion was approximately 50% in these images. Occlusions were anticipated during feature selection and dierent features were chosen than in the unoccluded case. The target pictograph was found in 59 cases and 22 false positives occured. With the additional use of intensity information the results could be improved to 69 correct recognitions and 7 false positive detections. For the comparison of intensities model and image points close to the matched edgels were used. The right part of gure 6 is a typical example for a successful recognition. Figure 6: Examples for successful detections in test images, left: unoccluded pictograph, middle: partially occluded object (used features were bitangents on the regions "frontlight of car" and "body of key"), right: magnied versions of the found pictographs The experimental results show that our approach allows for reliable and fast recognition of most of the used objects. For pictographs with ne details more reliable segmentation methods (e.g. as used in OCR) are likely to improve recognition accuracy. The entire approach could be made more powerful by using additional feature types (e.g. pairs of straight lines). References [1] R.C. Bolles and R.A. Cain. Recognizing and locating partially visible objects: the local-feature-focus method. International Journal of Robotics Research, 1(3):57{82, [2] D. Buesching. Ecient pictograph detection using bitangents. Technical report available at Technische Universitaet Muenchen, Forschungsgruppe Bildverstehen, [3] D. Buesching. Eciently nding bitangents. In International Conference on Pattern Recognition, Track A, pages 428{432, Vienna, August [4] F.P. Preparata and S.J. Hong. Convex hulls of nite sets of points in two and three dimensions. Communications of the ACM, 20(2):87{93, February [5] C.A. Rothwell, A. Zisserman, D.A. Forsyth, and J.L. Mundy. Planar object recognition using projective shape representation. International Journal of Computer Vision, 16:57{99, 1995.