2D Matching of 3D Moving Objects in Color Outdoor Scenes *

Transcription

1 2D Matching of 3D Moving Objects in Color Outdoor Scenes * Marie-Pierre Dubuisson and Ani1 K. Jain Department of Computer Science Michigan State University East Lansing, MI dubuisso@cps.msu.edu, jain@cps.msu.edu Abstract This paper describes an object matching system which is able to extract objects of interest from outdoor scenes and match them. Our application (in the domain of IVHS involves measuring the average travel time in a roa d network. The extraction of the object of interest is performed by fusing multiple cues including motion, color, edges, and model information. Two objects extracted from images captured by two independent cameras at different times are then matched to evaluate their similarity. Color indexing based on histogram matching is used to avoid matching all possible pairs of objects. To resolve ambiguities, further matching is done by measuring the Hausdorff distance between two sets of edge points. The object matching system was given 2 sets of 40 vehicles. It was able to identify 23 of the 30 correct matches and all the false matches were rejected. Color indexing reduced the number of candidates for a match from 40 to 2. This matching accuracy is adequate to obtain a reliable estimate of the average travel time. 1 Introduction Matching is a generic operation in image processing and computer vision which is used to determine the similarity of two entities (points, curves, subimages, objects, etc. of the same type. A number of different operations f all under the broad term of matching. Template matching is performed by evaluating the cross-correlation of a template and the corresponding window in the image. Image matching consists of determining the correspondence between image features such as points, lines, corners, or homogeneous regions, in two different images. We define object matching as a procedure for deciding whether two different 3D objects are identical based on their 2D images. The two objects are sensed possibly in different backgrounds by two independent cameras at different,times, but in approximately the same pose. An object matching system is simpler to construct than an object recognition system [8]. In an object matching system, precise models of the 3D objects are usually not available and not required. Instead, we have two (or more) sensed images of the objects Research supported by a grant from the U.S. Department of Transportation through the Great Lakes Center for Truck Transportation Research available to us. The objects are observed in approximately the same aspect, are entirely contained in the image, are not occluded, but are sensed in different environments. The output of an object matching system consists of a measure indicating the similarity of two objects independent of the backgrounds in the two images. It is important to realize that a robust extraction of the object of interest needs to be performed before any matching can occur due to different and complex backgrounds in the two images. Our application for object matchiri is in the domain of the IVHS (Intelligent Vehicle$Highway Systems) program [ll]. One of the two primary objectives of IVHS is to reduce travel time by assisting the traveler to avoid congested traffic situations and find the minimum travel time path through a. road network. One solution to the problem of measuring the average time for vehicles to go from one point to another in a road network is to match license plates. However, the camera needs to be focused on the license plate and the license plate has to be large enough in the sensed image so that individual characters can be recognized. These constraints are difficult to overcome in dense traffic situations, especially when the vehicles are moving at high speed. We are studying the feasibility of the following approach: cameras are placed on the sides of the road to capture the images, and then vehicles are matched with previously observed vehicles using color and edge features. In this application, it is not necessary to find all the correct matches; it is enough to find a sufficient number of reliable matches in order to compute some statistical information about the travel time of a vehicle from one point to another in a road network. A block diagram of our object matching system is shown in Figure 1. Two cameras are placed about a mile apart from each other in a road network. Image analysis is performed on both the road image sequences to extract the objects of interest (the front vehicle in the lane closest to the camera). The object extraction procedure is described in Figure 2. An integration of several cues, including a motion segmentation mask of the moving areas in the image sequence, homogeneous color regions, and edge information is used to identify the moving objects in the image. Given a generic model of a vehicle, incremental grouping and constrained search are then used to locate the front vehicle. The object extraction proce /94 $3.00 Q 1994 IEEE 887

2 Figure 1: A block diagram of the object matching system. Figure 2: A block diagram of the object extraction system. dure gives a set of edge points both inside and on the contour of the vehicle. A small window is extracted in the vicinity of the vehicle doors to characterize the color of the vehicle. The vehicles observed by the first camera are stored ill a database as a set of edge points and a color patch. Color histogram matching is used as an indexing step to reduce the number of potential matches. Edge matching is then performed on the reduced database to further refine the match and compute the similarity between the vehicles. 2 Segmentation of Moving Objects Our method to extract moving objects from an image sequence was first published in [4]; it was then enhanced and reported in [5]. The method is based on the fusion of a motion segmentation mask obtained by image subtraction and a set of homogeneous color regions obtained by the integration of a split-and-merge algorithm and edges resulting from the Canny edge detector. The flowchart of the algorithm is given in the top half of Figure 2. Motion segmentation based on image subtraction is used to obtain an approximate location of the moving objects in the input image sequence. Three color frames fl, f 2, and f3 containing the moving object are extracted at times tl, t2, and t3, respectively from the image sequence. The difference image defined as d(i,j) = max {Ifi(i,j) - fi(i,j)i x Ifk(i,j) - fk3(i,j)t} k=r,g,r (?I is thresholded to keep only those pixels where a significant change has taken place. A motion segmentation mask M, corresponding to the moving areas is generated by filling in the gaps in the thresholded image. For color segmentation, the regions obtained by the split-and-merge algorithm [6] adapted to color images are further merged using ed e information obtained &Jm the Canny edge detector f2] based on three criteria (boundary strength, color similarity, and connec- tivity measure). This procedure results in a set of regions R = {RI, R2,.., RP} which are fused with the motion segmentation mask M, to produce an accurate mask of the moving object MO. The motion segmentation mask M, is first smoothed using active contours (or snakes) as described by Kass et al. [9]. We have used the dynamic programming approach proposed by Amini et al. [l] to minimize the total energy. The contour generated by the snakes algorithm is then used to decide whether a region R, belongs to a moving object(s) and construct the new mask MO. Figure 3 compares the accuracy of the motion segmentation mask M, and the moving object mask MO. Figure 4 shows a few examples of the contours of moving vehicles in different backgrounds which were extracted by our algorithm. These results appear reasonable. We have also shown that the same algorithm produces accurate contours of moving objects in different domains [5]. 3 Model-based segmentation In the presence of multiple moving vehicles in the scene (Figures 4(e)-(f)), our segmentation algorithm groups them together to form a single moving object. 888

3 (4 (b) Figure 6: Fitting a generic model of a vehicle to the edge points: (a) single vehicle in the scene; (b) two vehicles in the scene. objects can be defined by the following algorithm. Figure 4: Contour extraction of moving vehicles. L 10.2L e 1 < 0.4L Figure 5: A generic model of a vehicle. For our application, the front vehicle (closest to the camera) needs to be isolated. In the situation shown in Figure 4(e), if the velocities of the vehicles were known, they could probably be used to separate the two vehicles because they are moving in opposite directions. In the case of Figure 4(f), however, the two vehicles are moving in the same direction and the velocity information alone would not be precise enough to separate them. Our algorithm for the extraction of the front vehicle is presented in the bottom half of Figure 2. We use a top-down approach to find an instance of the vehicle model in the image. Of course, it is not feasible to create a model for all possible vehicles that can be found on roads and then perform object recognition; the resulting model database would be too large. Therefore, we have defined a generic and rather coarse model of a vehicle consisting of five linear features pi, i = 1,...,5 as shown in Figure 5. The edge points generated by color segmentation which are inside the mask MO and their line segment approximations are used to find the model features in the image. Using the constraints on the lengths of the line segments and the angles between them in the model, we can use incremental grouping to fit the model to the data. Note that since we are looking for only the front vehicle, it is unlikely that any of its features will be occluded in the scene. Incremental grouping using constrained search for non-occluded 0 find (PI, the first feature in the model; 0 for i = 2,..., N, where N is the number of features find (pi constrained by all (oj, j = 1,..., i- 1. The five features which are extracted from the image are then joined to build a model of the vehicle. Figures 6 shows examples of the extracted model (superimposed on the edges) in an image containing only one vehicle and an image containing two moving vehicles. This top-down approach produces a very crude mask M, of the front vehicle. A final mask Mj of the front vehicle is obtained by combining MO, M,, and the set of homogeneous color regions R = {RI,..., Rp}. Let nf denote the number of pixels in region Rd that are inside the mask.mu n MO. Decide that region Ri is part of the final mask Mj if nf/ni > A 7 x 7 averaging filter is also applied before the final mask Mj is obtained. Figure 7 shows examples of the front vehicle extraction process applied to 6 different image sequences containing one or more moving vehicles where the contour of the final mask of the front vehicle Mj is overlaid on the input image. These contours show the robustness of our model-based segmentation scheme. 4 Color Indexing Most object recognition techniques first generate a number of hypotheses about the identity and pose of the object, and then select the best hypothesis. Since the verification stage for selecting the best hypothesis is often very time consuming, particularly when the model database is large, indexing methods have been frequently used in object recognition to reduce the number of hypotheses generated. Indexing is also needed in our system to avoid expensive edge matching between all possible pairs of objects. In our application, the object database contains the vehicles observed by camera 1 (see Figure 1). When a vehicle is detected by the second camera, it is first compared with all the vehicles in the database using color information. More precise matching is done only if the colors of the two vehicles are sufficiently similar. 889

4 Figure 8: Location of the color patch. Figure 7: E xtraction of the front vehicle. Swain and IaUard [lo] proposed a method for Omparing color images using histograms. Multi-dimensional color histograms are compared using a minimum operator. We have slightly modified their measure to handle object matching instead of matching an is* lated model and a scene containing the mode], We have used RGB color features. The color histogram matching score, between two histograms Hl(r, g, b) and H2(r, g, b) is defined as follows: are inherently very similar, so a comparison based on vehicle contour alone will not have enough discriminatory power. Some authors have presented matching methods based on feature points using relaxation techniques. These techniques have the disadvantage of being computationally demanding when the number of eature points is large. A diffgrent approach is to match relational structures (graphs or stars) using subgraph isomorphism. Recently, Huttenlocher et al. [7] proposed a method for Omparing edge images using he Hausdorff distance. This technique is very appropriate for our application since the two objects are already described point by edge sets points A a;d The is Hausdorff defined as: distance between two (2) where 1. I denotes the number of pixels used to compute the histogram. Note that S HI, H2) takes values between 0 and 1, and Sc(H1, i 2) = 1 if and only if one and is some norm (e.g., Manhattan or Euclidean). The Hausdorff distance measures the similarity between two point sets at fixed positions. Huttenlocher et al. proposed an efficient algorithm to compute the Hausdorff distance between all possible relative positions of the two sets and then find the minimum distance. Let 7 be the set of all possible transformations t. The matching score between the two point sets is defined as: of the histograms is totally included in the other one. Initial experiments on color matching indicated that the entire vehicle should not be used to compute the 3D color histogram. Since all the vehicles have typically black wheels and grayish hubcabs and windows, the 3D color histogram always shows a peak for these colors, no matter which vehicle is being observed. Therefore, we have decided to use only a part of the vehicle to compute the 3D color histogram. A small color patch (20 x 50 pixels) is extracted near the location of the vehicle doors as shown in Figure 8 using model information and is used to compute the color histogram (lh1l = lh21 = 1000). 5 Edge Matching Given two objects that have been extracted from their respective scenes, the object matching system computes the similarity between them. A number of techniques for object (2D shape) matching using moments or Fourier descriptors have been presented in the literature. The advantage of these methods is that they are invariant to size, rotation, and translation, but the main disadvantage is that they are not information-preserving. The difficulty in using these techniques for our application is that different vehicles D,(A,B) = minh(d,tb) t 7 (5) For our application, the translation between the two point sets can be approximated by the translation between their centers of masses (Cl and Cz). The rotation between the two images in our sensing setup is negligible. The vehicles in the sensed images captured by camera 2 were a little bit smaller than the vehicles in the images taken by camera 1. So, we define the set of possible transformations as : 7, {(tzlty,s) s.t < s < 1, (CI= - C2, - 5) < ts < (CI= - Cz, + 5), (6) (Cl, - c2, - 5) < t, < (Cl, - c2, + 5))

5 6 Experimental Results R.oad image sequences were recorded using two Sharp camcorders in Troy, Michigan on May 24, 1993 around 2:45 pm. The speeds of the vehicles were between 30 and 35 mph, and the traffic was busy enough to result in multiple moving vehicles in the same image frame. The tapes were played on a VCR in our laboratory and a 3-frame sequence for each vehicle was digitized. The first set of 40 vehicles {Vi,..., Vi,} was obtained from the first tape (camera l), and a second set of 40 vehicles {Yt,..., Vio}was obtained from the second tape (camera 2). Ten of the vehicles in the first set did not have any matching vehicle in the second set, resulting in only 30 true matches out of 40 possible matches. For the IVHS application considered here, missing some of the true matches will not affect the statistical measurement of travel time in a road network, but false matches will produce wrong results. So, we have implemented a reject option that would avoid false matches. The moving object extraction process was applied to all the 40 3-frame sequences from set 1 and 40 color patches and edge images were stored in the database of vehicles seen by the first camera. The same processing steps were applied to the second set. Color indexing was performed first. The color patch of vehicle V; was matched with the color patches of all the vehicles Vi, j = 1,..., 40 in the first set using color histogram matching. The color histogram matching measure S, is a similarity measure; the higher its value, the more similar the two vehicles. For an indexing technique to be efficient, it must reduce the number of candidate matches. In other words, the true match should be highly ranked compared to the other matches. The rank of the true match was the highest in 18 out of 30 cases, and second highest in 9 cases. But, in 13 cases, either the match did not exist or it had a lower rank. We have decided to keep only the 2 vehicles having the highest color histogram matching scores as candidates for edge matching. This retains 27 of the 30 true matches. As a result, the number of candidate matches was reduced from 40 to 2, resulting in a 95% reduction in the edge matching time. Edge matching based on the Hausdorff distance was then used to match vehicle Vf and {V; s.t. rank(s,(v:,vj)) 5 2) to compute an edge matching score De. The total matching score is defined as S, = %. Since S, is a similarity measure and De is a dissimilarity measure, St is a similarity measure. The highest value of St should correspond to the true match. Two total matching scores Stl and St, were computed for each vehicle V?. A match should be accepted or rejected based on these two numbers. A match was accepted if both the highest total matching score and the difference between the two total matching scores exceeded some thresholds. "(St,, St,) > 0.2 IS,, - S,I > 0.1 (7) This resulted in all the false matches to be rejected and 23 of the 30 true matches to be axepted. 7 Summary The proposed object matching system has been shown to meet the requirements of our application in the IVHS domain. We have shown that it is possible to accurately extract the moving objects in a color image using a 3-frame sequence. A generic model of a vehicle and an incremental grouping approach using constrained search enable us to extract the front vehicle. We have also shown that the indexing process based on color histogram matching is powerful in reducing the size of the object database to be matched. The matching results are very encouraging. We were able to find 23 of the 30 true matches and eliminate all 10 false matches. References A. Amini, T. Weymouth, and R. Jain. Using dynamic programming for solving variational problems in vision. IEEE Trans. PAMI, 12(9): , J. Canny. A computational approitch to edge detection. IEEE Trans. PAMI, 8(6): , V. Coutance. La Couleur en Vision par Ordinateur: Application ci la Robotique. PhD thesis, Laboratoire d'automatique et d'analyse des Systkmes du CNRS, Universitk Paul Sabatier, Toulouse, France, M.-P. Dubuisson and A. K. Jain. Object contour extraction using color and motion. In Proc. IEEE Conf. Computer Vision and Pattern Recognition, pages , New York, NY, M.-P. Dubuisson and A. K. Jain. Contour extraction of moving objects in complex outdoor scenes. to appear in International Journal of Computer Vision, S. Horowitz and T. Pavlidis. Picture segmentation by a tree traversal algorithm. Journal of the ACM, 23(2): , D. P. Huttenlocher, G. A. Klanderman, and W. J. Rucklidge. Comparing images using the Hausdorff distance. IEEE Trans. PAMI, 15(9): , A. K. Jain and P. J. Flynn (eds.). 3D Object Recognition Systems. Elsevier, Amsterdam, M. Kass, A. Witkin, and D. Terzopoulos. Snakes: Active contour models. Int. Journal of Computer Vision, pages , M. J. Swain and D. H. Ballard. Color indexing. Int. Journal of Computer Vision, 7(1):11-32, Transportation Research Board. Advanced Vehicle and Highway Technologies. Technical Report No. 232, National Research Council,