Multimodal Stereo Image Registration for Pedestrian Detection

Transcription

1 Multimoal Stereo Image Registration for Peestrian Detection Stephen Krotosky an Mohan Trivei Abstract This paper presents an approach for the registration of multimoal imagery for peestrian etection when the signicant epth ifferences of objects in the scene preclues a global alignment assumption. Using maximization-of-mutualinformation matching techniques an sliing corresponence winows over calibrate image pairs, we emonstrate successful registration of color an thermal ata. We evelop a robust metho using isparity voting for etermining the registration of each object in the scene an provie a statistically base measure for evaluating the match conence. Testing shows successful registration in complex scenes with multiple people at ifferent epths an levels of occlusion. I. INTRODUCTION A large number of automotive accients involve peestrians. In orer to alleviate this problem, ongoing research is being performe to etect an track peestrians from both moving vehicles an the static infrastructure. The use of camera systems for etection is common, incluing both color [1] [2] an thermal imagery [3] [4] [5]. While [6] shows a comparison of color an thermal methos, we wish to be able to provie a metho for combining the two moalities. Visual an thermal infrare imagery provie isparate, but complementary information about a scene. Visual cameras capture the reective light properties of objects in the scene, while thermal infrare cameras are sensitive to the thermal emissivity properties of the same objects. The pairing of these two moalities is interesting, as their combination provies information about the scene that is not reaily obtaine from the human visual system. Namely, the combination of visual an thermal infrare imagery can provie the color, epth, motion, an thermal properties that can be use to etect, track an analyze people in a scene. In orer to be able use the ata from multiple cameras in a meaningful way, corresponing ata from each image must be matche, or registere, so that the information from each moality can be properly attribute for the higher level tasks of etection, tracking an analysis. Because the ata from visual an thermal imagery appears very ifferent in each image, ning the appropriate registration for objects in the corresponing images is a challenging task. Typically, successful multimoal image registration techniques require that the cameras an scene be oriente in such a way that registration can be escribe with a global image transformation [7] [8] [9] [10] [11]. Such techniques fail, however, This work was sponsore in part by grants from the U.S. DoD Technical Support Working Group, U.C. Discovery an Volkswagen Research Laboratory. S. Krotosky an M. Trivei are with the Computer Vision an Robotics Research Laboratory, University of California, San Diego, 9500 Gilman Dr. 0434, La Jolla, CA 92093, USA krotosky@ucs.eu, mtrivei@ucs.eu when objects in the scene are locate at ifferent observation epths, as the registration will vary with each object ue to the parallax effects. This work presents a metho for multimoal image registration that can successfully n local corresponences for people in the scene, even in situations where peestrians are locate at largely ifferent relative epths in the scene, where a global transformation moel woul surely fail. Using calibrate an rectie image pairs, we can analyze sliing corresponence winows in the color reference image, an n its appropriate match in the thermal image using maximization of mutual information. Combining votes from multiple corresponence winows into a isparity accumulation matrix, we are able to robustly etermine the appropriate isparity for registering each person in the scene an associate this isparity with a statistically relevant conence measure. This gives a resultant isparity image that can accurately register the people in the scene, as well as provie aitional valuable information that can be use for further scene segmentation in cases of occlusion. By registering images in this way, not only are we able to have success when a global transformation is an invali assumption, but the resulting isparity image is also a valuable feature that can be use for peestrian etection, tracking an activity analysis. II. MULTIMODAL IMAGE REGISTRATION USING MUTUAL INFORMATION AND DISPARITY VOTING Our propose registration algorithm buils upon the work of Chen, et al. [12] an aresses several of the limitations of their work. First, by calibrating the color an thermal cameras we can obtain rectie images that will reuce the search space for corresponing objects. Although this metho will also rely on an initial foregroun silhouette extraction, it oes not require highly accurate segmentation, an unlike [12] an [13], it also oes not require that iniviual people be segmente from each other, only that they be reasonably segmente from the backgroun. We will also utilize the maximization of mutual information technique for corresponence matching for multimoal imagery, but will exten it by using a isparity voting algorithm from the results of sliing corresponence winows. This will provie for robust isparity estimation as well as a statistical conence value for the isparity measure. Figure 1 shows a owchart outlining our algorithmic framework. A. Multimoal image calibration It is esirable to calibrate the color an thermal infrare cameras. Knowing the intrinsic an extrinsic calibration

2 algorithm coul potentially be use with success. For these specic experiments, foregroun segmentation in the visual imagery was one using the coebook moel propose by Kim, et al. [15]. In the thermal imagery, the foregroun is obtaine using a simple intensity threshol uner the assumption that the people in the foregroun are hotter than the backgroun. The corresponing foregroun images are F L an F R, respectively. Aitionally, the color image is converte to grayscale for mutual information base matching. Example input images an foregroun maps are shown in Fig. 2. (a) Fig. 1. Flowchart of multimoal image registration algorithm. parameters transforms the epipolar lines in the isparity corresponence search to lie along the image scanlines, enabling matching to be a one-imensional search. Calibration can be performe using stanar techniques, such as those available in the Camera Calibration Toolbox for Matlab [14]. The toolbox assumes input images from each moality where a calibration boar is visible in the scene. In typical visual setups, this is simply a matter of placing a checkerboar pattern in front of the camera. However, ue to the large ifferences in visual an thermal imagery, some extra care nees to be taken to ensure the calibration boar looks similar in each moality. A solution is to use a stanar calibration boar an illuminate the scene with high intensity halogen bulbs place behin the cameras. This effectively warms the checkerboar pattern, making the visually ark checks appear brighter in the thermal imagery. Placing the boar uner constant illumination reuces the blurring associate with thermal iffusion an keeps the checkerboar eges sharp. B. Image acquisition an foregroun extraction The acquire an rectie image pairs are enote as I L, the left color image, an I R, the right thermal image. Due to the high ifferences in imaging characteristics, it is very ifcult n corresponences for the entire scene. Instea, registration is focuse on the pixels that correspon to foregroun objects of interest (people). Naturally then, it is esirable to etermine which pixels in the frame belong to the foregroun. In this step, only a rough estimate of the foregroun pixels is necessary an a fair amount of false positives an negatives is acceptable. Any goo segmentation (c) Fig. 2. Image acquisition an Foregroun extraction: (a) input color image, color foregroun map, (c) input thermal image, () thermal foregroun map. C. Corresponence matching using maximization of mutual information Once the foregroun regions are obtaine, the corresponence matching can begin. Matching occurs by xing a corresponence winow along one reference image in the pair an sliing the winow along the secon image that is the best match. Let h an w be the height an with of the image, respectively. For each column i 0... w, let W L,i be a corresponence winow in the left image of height h an with M centere on column i. Dene a corresponence winow W R,i, in the right image having height h an centere at a column i +, where is a isparity offset. For each column i, a corresponence value is foun for all min... max. Given the two corresponence winows W L,i an W R,i,, we rst linearly quantize the image to N levels such that () N 8Mh (1) where Mh is the area of the corresponence winow. The result in (1) comes from Thevenaz an Unser's [16] suggestion that this equation is reasonable to etermine the number of levels neee to give goo results for maximizing the mutual

3 information between image regions an was also aopte by Chen, et al. [12]. Now we can compute the quality of the match between the two corresponence winows by measuring the mutual information between them. To o this, we also utilize the stanar mutual information computation techniques for image patches also aopte by [12]. The metho is outline again here for convenience. The mutual information between two image patches is ene as (a) 0.4 I(L, R) = l,r P L,R (l, r) log P L,R(l, r) P L (l)p R (r) (2) where P L,R (l, r) is the joint probability mass function (pmf) an P L (l) an P R (r) are the marginal pmf's of the left an right image patches, respectively. The two-imensional histogram, g, of the corresponence winow is utilize to evaluate the pmf's neee to etermine the mutual information. The histogram g is an N by N matrix so that for each point, the quantize intensity levels l an r from the left an right corresponence winows increment g(l, r) by one. Normalizing by the total sum of the histogram gives the probability mass function P L,R (l, r) = g(l, r) l,r g(l, r) (3) The marginal probabilities can be easily etermine by summing P L,R (l, r) over the appropriate imension. P L (l) = l P R (r) = r P L,R (l, r) (4) P L,R (l, r) (5) Now that we are able to etermine the mutual information for two generic image patches, let's ene the mutual information between two specic image patches as I i, where again i is the center of the reference corresponence winow an i+ is the center of the secon corresponence winow. For each column i, we have a mutual information value I i, for min... max. The isparity i that best matches the two winows is the one that maximizes the mutual information i = arg max I i, (6) The process of computing the mutual information for a specic corresponence winow is illustrate in Figure 3. An example plot of the mutual information values over the range of isparities is also shown. D. Disparity voting with sliing corresponence winows We wish to assign a vote for i, the isparity that maximizes the mutual information, to all foregroun pixels in the reference corresponence winow. Dene a isparity voting matrix D L of size (h, w, max min + 1), the range of isparities. Then given a column i, for each image pixel that is in the corresponence winow an foregroun map, I i, (c) Fig. 3. Mutual Information of Corresponence Winows: (a) left image with the re rectangle enoting its corresponence winow, right image with the green rectangles enoting the range of corresponence winows searche, (c) mutual information I i, for a range of isparities min... max. (u, v) (W L,i & F L ), we a to the isparity voting matrix at D L (u, v, i ). (u, v) (W L,i & F L ), i D L (u, v, i ) = F L,i (u, v) (7) Since the corresponence winows are M pixels wie, pixels in each column in the image will have M votes for a corresponence matching isparity value. For each pixel (u, v) in the image, D L can be thought of as a istribution of matching isparities from the sliing corresponence winows. Since it is assume that all the pixels attribute to a single person are at the same istance from the camera, a goo match shoul have a large number of votes for a single isparity value. A poor match woul be wiely istribute across a number of ifferent isparity values. Fig. 4 shows the isparity voting matrix for a sample row in the color image. The x-axis of the image is the columns i of the input image. The y-axis of the image is the range of isparities = min... max. Entries in the matrix correspon to the number of votes given to a specic isparity at a specic column in the image. Brighter areas correspon to a higher vote tally. The complementary process of corresponence winow matching is also performe by keeping the right thermal infrare image xe. The algorithm is ientical to the one escribe above, switching the left an right enotations. The corresponing isparity accumulation matrix is given as D R. Once the isparity voting matrices have been evaluate for the entire image, the nal isparity registration values can be etermine. For both the left an right images, we etermine the best isparity value an its corresponing conence

4 Fig. 4. Disparity Voting Matrix for a sample row in the color image. On the x-axis is the image column i an the y-axis is the range of isparities. Brighter areas correspon to higher votes for a isparity from the corresponence winow matching. (a) measure as DL(u, v) = arg max D L (u, v, ) (8) CL(u, v) = max D L (u, v, ) (9) For a pixel (u, v) the values of CL (u, v) represent the number of times the best isparity value DL (u, v) was vote for. A higher conence value inicates that the isparity maximize the mutual information for a large number of corresponence winows an in turn, the isparity value is more likely to be accurate than at a pixel with lower conence. Values for DR an C R are similarly etermine. The values of DR an C R are also shifte by their isparities so that they align to the left image: (c) () Fig. 5. Disparity an conence images: (a) left isparity image DL an left conence image CL obtaine by searching over the thermal image for a xe visual image corresponence winow. Similarly, (c) an () show shifte isparities DS an conences C S when the thermal corresponence winow is xe. D S(u, v + D R(u, v)) = D R(u, v) (10) C S(u, v + D R(u, v)) = C R(u, v) (11) Fig. 5 shows examples of the isparity an conence images obtaine from (8) an (9), respectively. The isparities from the right thermal image have been use to shift the image pixels so that the corresponing pixels align. Notice how the isparity values in Fig. 5a an Fig. 5c are the same for corresponing people in the two images. Once the two isparity images are aligne, they can be combine. We have chosen to combine them using an OR operation. This tens to give the most complete results an can help to ll holes an errors in the foregroun extraction of the two moalities. { D D (u, v) = L (u, v), CL (u, v) C S (u, v) DS (u, v), C L (u, v) < C S (u, v) (12) The resulting image D (u, v), shown in Fig. 6, is the isparity image for all the foregroun object pixels in the image. It can be use to register multiple objects in the image, even at very ifferent epths from the camera. III. RESULTS Our evelope algorithm for multimoal image registration was teste using color an thermal ata collecte where the cameras were oriente in the same irection with a baseline of 10 cm. The color camera's zoom was ajuste so that objects in the color image appear at the same scale as Fig. 6. The resulting isparity image D from combining the left an right isparity images DL an D S as ene in (12). the images from the xe-lens thermal camera. The cameras were oriente so that the optical axis is approximately parallel to the groun. This position was use to satisfy the assumption that there woul be approximately constant isparity across all pixels associate with a specic person in the frame. Placing the cameras in this sort of position is a reasonable thing to o, an such a position is similar to the view from a vehicular mounte camera system. Vieo was capture as peestrians move naturally throughout an outoor environment. The goal was to obtain registration results for various congurations of people incluing ifferent positions, istances from camera, an levels of occlusion. Fig. 7 shows the result of registration for the example frame carrie throughout the algorithmic erivation. Fig. 7a shows the initial alignment of the color an thermal images, while Fig. 7b shows the alignment after shifting the foregroun pixels by the resulting isparity image D shown in Fig. 6. The thermal foregroun pixels are overlai (in green) on the color foregroun pixels (in pink). The resulting registration in Fig. 7 is successful in aligning the foregroun areas associate with each of the three people in the scene. Upon visual inspection, it is clear that the thermal ata accurately overlays the color ata. This inicates that isparity voting with sliing corresponence winows is

5 (a) Fig. 7. (a) Initial alignment of color foregroun pixels (pink) with thermal foregroun pixel (green) before registration. Final alignment after multimoal stereo image registration. ifferent people. Distinguishing merge peestrian regions is a challenging task an the isparity information from this technique can be use to segment them into istinct regions corresponing to separate people without assuming any prior moel of person shape or conguration. Aitionally, if the isparity information can be properly calibrate to provie accurate epth information, the technique woul also give stereo ata. The stereo information coul be use to valiate the segmentation of people in the scene as well as be a valuable aitional input for etection, tracking an analysis. a vali feature for establishing the local isparities between corresponing objects in the color an thermal imagery. Each person in the scene lies at a ifferent istance from the camera an yiels a ifferent isparity value that will align its corresponing image components. Registration algorithms that rely on global alignment cannot successfully hanle this type of situation, yet the propose algorithm can provie for successful multimoal registration on a per object level in cases when a global alignment is not obtainable. (a) (c) () Fig. 8. Example registration results: (a) input color image, input thermal image, (c) unaligne overlay of color an infrare foregrouns before registration, () aligne overlay of color an infrare foregrouns after registration. Examples of successful registration for aitional frames are shown in Fig. 8. Columns (a) an show the input color an thermal images. Column (c) illustrates the initial alignment of the people in the scene. Column () shows the resulting alignment an overlay after the multimoal image registration has been performe. These aitional examples show the success of the propose registration technique uring relatively ense scenes, where people are being signicantly occlue an are at wiely isparate epths from the camera. In each case, though, the isparity image gives goo alignment of the color an thermal ata for each person in the scene. In aition to aligning the color an thermal ata for people in the scene, the isparity information can be useful in segmenting the people in the scene. In examples where the initial foregroun segmentation silhouettes combine multiple people ue to occlusions, the resulting isparity image still inicates separate isparity measures for the regions corresponing to the A. Disparity Comparison to Stereo from Two Visual Images When evaluating the alignment quality of the multimoal stereo image registration, it is interesting to compare the multimoal isparity image to a isparity image generate from two visual images. Stereo matching images from the same moality is a well-stuie area an many algorithms exist to n these corresponences. For our tests, we use the SVS algorithm from SRI International [17] to evaluate the isparities from two cameras that were capturing simultaneous to the multimoal cameras an in the same position an orientation. Some example results are shown in Fig. 9. Column (a) shows the color input image, column shows the thermal input image, column (c) shows the isparity image compute from the two visual images an column () shows the results using our metho for multimoal stereo image registration. Since the two stereo corresponence algorithms work on ifferent images at ifferent scales, ifferences in the intensity values of the compare corresponing images are not of real note. What is of interest is that the stereo isparities from the multimoal imagery compare favorably in shape, completeness, an t to the peestrians to the isparity image from the single visual moality. Distinct isparity regions for iniviual peestrians are evient in both isparity images an the use of initial segmentation in the multimoal case means that the isparity regions conform better to the actual shape of peestrians. While the visual stereo computations work without regar for foregroun regions, further processing for peestrian etection usually requires this, so it is justiable to use foregroun segmentation in the multimoal stereo case. Aitionally, the use of multimoal stereo image registration benets from the fact that ata from the thermal image ata can easily be incorporate into further processing algorithms for etecting, tracking an analysis for peestrian etection. While these initial results are promising, further analysis is necessary to test the algorithm's robustness to aitional challenging congurations an poses of people an in situations where other non-people objects populating the scene coul lea to registration errors. The current registration results appear qualitatively goo, but extensive groun truth testing woul be valuable in etermining some quantitative measures of registration success.

6 (a) Color Image Thermal Image (c) Visual Disparity () Multimoal Disparity Fig. 9. Disparity Comparison: (a) input color image, input thermal image, (c) isparity image compute from two visual images, () isparity image from multimoal stereo image registration. IV. CONCLUSIONS In this paper we have presente a novel algorithm for registering multimoal images in scenes where people appear at ifferent istances from the camera so that traitional global alignment assumptions o not hol. Using maximization of mutual information matching techniques on sliing corresponence winows to populate a isparity accumulation matrix, we have emonstrate a robust metho for etermining registration isparities with conence values in multimoal imagery. The metho has proven successful for situations that inclue signicant occlusion of people in the scene as the resulting isparity image can be use to for segmentation renement in occlue regions. This type of registration algorithm can provie the registration an isparity ata that can be use to combine multimoal imagery for peestrian etection, tracking an activity analysis applications. REFERENCES [1] N. Nir, O. Masou, N. Papanikolopoulos, an A. Issacs, Detection of loitering iniviuals in public transportation areas, in IEEE Intelligent Vehicles Symposium, [2] L. Zhao an C. Thorpe, Stereo an neural network-base peestrian etection, IEEE Trans. Intell. Transport. Syst., vol. 1, no. 3, pp , Sept [3] M. Bertozzi, E. Binelli, A. Broggi, an M. Del Rose, Stereo visionbase approaches for peestrian etection, in IEEE CVPR Workshop on Object Tracking an Classication beyon the Visible Spectrum, [4] F. Xu, X. Liu, an K. Fujimura, Peestrian etection an tracking with night vision, IEEE Trans. Intell. Transport. Syst., vol. 6, no. 1, pp , Mar [5] M. Yasuno, N. Yasua, an M. Aoki, Peestrian etection an tracking in far infrare images, in Computer Vision an Pattern Recognition Workshop, 2004 Conference on, [6] Y. Fang et al., Comparison between infrare-image-base an visibleimage-base approaches for peestrian etection, in IEEE Intelligent Vehicles Symposium, [7] E. Coiras, J. Santamaria, an C. Miravet, Segment-base registration technique for visual-infrare images, Optical Engineering, vol. 39, no. 1, pp , Jan [8] J. Davis an V. Sharma, Fusion-base backgroun-subtraction using contour saliency, in IEEE CVPR Workshop on Object Tracking an Classication beyon the Visible Spectrum, [9] M. Irani an P. Ananan, Robust multi-sensor image alignment, in Computer Vision, Sixth International Conference on, [10] M. Itoh, M. Ozeki, Y. Nakamura, an Y. Ohta, Simple an robust tracking of hans an objects for vieo-base multimeia prouction, in IEEE Conference on Multisensor Fusion an Integration for Intelligent Systems, [11] G. Ye. (2005) Image registration an super-resolution mosaicing. thesis/at-adfa/uploas/ approve/at-adfa /public/01front.pf. [12] H. Chen, P. Varshney, an M. Slamani, On registration of regions of interest (ROI) in vieo sequences, in IEEE Conference on Avance Vieo an Signal Base Surveillance (AVSS'03), [13] H. Chen, S. Lee, R. Rao, M. Slamani, an P. Varshney, Imaging for conceale weapon etection, IEEE Signal Processing Mag., pp , Mar [14] J.-Y. Bouguet. Camera calibration toolbox for matlab. [15] K. Kim, T. Chaliabhongse, D. Harwoo, an L. Davis, Real-time foregroun-backgroun segmentation using coebook moel, Real- Time Imaging, vol. 11, no. 3, pp , June [16] P. Thevenaz an M. Unser, Optimization of mutual information for multiresolution image registration, IEEE Trans. Image Processing, vol. 9, no. 12, pp , Dec [17] K. Konolige, Small vision systems: harware an implementation, in Eighth International Symposium on Robotics Research, 1997.