Color Image Fusion for Concealed Weapon Detection

Transcription

1 In: E.M. Carapezza (Ed.), Sensors, and ommand, ontrol, ommuniations, and intelligene (C3I) tehnologies for homeland defense and law enforement II, SPIE-571 (pp ). Bellingham, WA., USA: The International Soiety for Optial Engineering. Color Image Fusion for Conealed Weapon Detetion Alexander Toet TNO Human Fators, Kampweg 5, 3769 DE Soesterberg, The Netherlands ABSTRACT Reent advanes in passive and ative imaging sensor tehnology offer the potential to detet weapons that are onealed underneath a person's lothing or arried along in bags. Although the onealed weapons an sometimes easily be deteted, it an be diffiult to pereive their ontext, due to the non-literal nature of these images. Espeially for dynami rowd surveillane purposes it may be impossible to rapidly asses with ertainty whih individual in the rowd is the one arrying the observed weapon. Sensor fusion is an enabling tehnology that may be used to solve this problem. Through fusion the signal of the sensor that depits the weapon an be displayed in the ontext provided by a sensor of a different modality. We propose an image fusion sheme in whih non-literal imagery an be fused with standard olor images suh that the result learly displays the observed weapons in the ontext of the original olor image. The proedure is suh that the relevant ontrast details from the non-literal image are transferred to the olor image without altering the original olor distribution of this image. The result is a natural looking olor image that fluently ombines all details from both input soures. When an observer who performs a dynami rowd surveillane task, detets a weapon in the sene, he will also be able to quikly determine whih person in the rowd is atually arrying the observed weapon (e.g. the man with the red T-shirt and blue jeans ). The method is illustrated by the fusion of thermal 8-12 µm imagery with standard RB olor images. Keywords: Image fusion, infrared, olor, onealed weapons 1. INTRODUCTION Conealed weapon detetion is needed anywhere large onentrations of the publi an be targeted for terrorist ations inluding airports, publi offie buildings, subways, shools, banks and malls. Unobtrusive detetion of onealed weapons on persons or in (abandoned) bags would provide law enforement a powerful tool to fous resoures and inrease traffi throughput in high- risk situations. A wide range of sensor systems is urrently available, eah with its own strength and weaknesses. No single sensor will ompletely satisfy the onealed weapon detetion mission. Sensor fusion is an enabling tehnology that may inrease the sensitivity, and redue the number of false alarms and lutter by ombining the signals of two or more sensors of different and omplementary modalities 3-5,8,11,18. Here we propose a new image fusion sheme that produes fused olor images that appear similar to standard olor images. The fused images are therefore easy to interpret for human observers, and are therefore highly suitable for surveillane purposes. The fusion proess proposed here involves the fusion of pereptually relevant ontrast details from one or more nonliteral images to the ahromati hannel of a standard olor image of the same sene. Sine we don t want undesirable ross-hannel artifats in the resulting fused image (i.e. as a result we want a fused image that has the same olors as the original olor image) we perform our omputations in a pereptually deorrelated olor spae. This olor spae was reently presented as a useful tool for manipulating olor images 9,15. Sine it is logarithmi, uniform hanges are to a first approximation equally detetable. In the next setions we will explain the method and show some results of the appliation of the method to the fusion of thermal 8-12 µm imagery with standard olor imagery. toet@tm.tno.nl 372

2 2. METHODS The aim of the present study is to fuse thermal and olor images suh that the strutural information from the greylevel thermal image is transferred to the olor image, while the appearane of the resulting fused olor image is idential to that of the original olor image. We ahieve this goal by applying a pereptually deorrelated olor spae that was reently introdued to enhane the olor representation of syntheti imagery 9,15. The pairs of input images used in this study onsist of respetively a normal daylight RB olor photograph and a 8-12 µm thermal greylevel image of the same sene. In the following we assume that both images are registered. In pratie, registration is easily ahieved by first performing an affine transformation that warps one image onto the other, using fiduial registration points that are visible in both images 16. The olor fusion proedure is as follows. First, the soure and target image are both transformed to a LMS one response spae. The different bands of daytime olor images are usually orrelated. Sine we want to be able to transfer details from the thermal image to the olor image, without hanging the overall olor distribution of the olor image, we first need to transform the input imagery to a spae whih minimizes the orrelation between hannels. Therefore, through prinipal omponent analysis, we rotate the axes in the LMS one spae to ahieve maximal deorrelation between the data points. The first prinipal omponent represents an ahromati hannel, while the other two omponents orrespond to olor opponent hannels 9,15. Then, the ahromati omponent of respetively the daylight olor image and the thermal image are merged though a pyramidal image fusion sheme. The mean and standard deviation of the resulting fused luminane image are set equal to those of the original olor image, to preserve the look and feel of the original olor image 9. Finally, the fused image is transformed bak to RB spae for display. The result is similar to the original daylight image but also inludes pereptually relevant details from the thermal image. In the following setions we first disuss the RB to LMS transform. Seond, we desribe the prinipal omponent transform in LMS one spae that is applied to deorrelate the different hannels. Third, we briefly disuss the pyramidal luminane image merging sheme. Finally, we will show that the inverse transform of the fused olor image bak to RB spae results in a olor image that fluently ombines all relevant details of both input images while it looks similar to the original olor image RB to LMS transform First the RB tristimulus values are onverted to devie independent XYZ tristimulus values. This onversion depends on the harateristis of the display on whih the image was originally intended to be displayed. Beause that information is rarely available, it is ommon pratie to use a devie-independent onversion that maps white in the hromatiity diagram to white in RB spae and vie versa 6. X R Y Z B (1) The devie independent XYZ values are then onverted to LMS spae by L X M Y S.. 1. Z (2) 373

3 Combination of (1) and (2) results in L R M S B (3) The data in this olor spae shows a great deal of skew, whih is largely eliminated by taking a logarithmi transform: / 6 log L log M log S (4) The inverse transform from LMS one spae bak to RB spae is as follows. First, the LMS pixel values are raised to the power ten to go bak to linear LMS spae. Then, the data an be onverted from LMS to RB using the inverse transform of Equation (3): R L M B S (5) 2.2. Prinipal omponent transform The prinipal omponent transform 7,1,12 effetively rotates the LMS oordinate axes suh that the pixel omponents are maximally deorrelated. The set of normalized eigenvetors of the ovariane matrix of the set of pixel values, arranged in order of inreasing eigenvalues, onstitute the olumn vetors of the orresponding rotation matrix. Let R be the rotation matrix that deorrelates the olor image pixels. The pixel values of the olor image and the thermal image in this new oordinate system are then respetively given by / / R 6 6 (6) and / t / t t R t 6 t 6 t (7) where the indies and t refer to the olor and thermal images respetively Fusion of the ahromati image omponents When ombining information from the thermal image with the olor image into a single display it is essential that the relevant ontrast details of the individual images are preserved in the final image, and that no spurious pattern elements (that may interfere with subsequent analysis) are introdued by the merging proess. We therefore applied a pyramidal 374

4 image fusion sheme to merge the ahromati omponent of both input images 2,13,14,17. A 7-level Laplaian pyramid 2 was used, in ombination with a maximum absolute ontrast node (i.e. pattern element) seletion rule. This proedure ensures that pereptually relevant image details from all individual bands are represented in the final graysale fused image. An image pyramid is a olletion of images at different spatial sales that together represent the original soure image. Suh a multi-resolution image representation an be obtained through a reursive redution of the input image, i.e. a ombination of low-pass or band-pass filtering and deimation. The well-known Laplaian pyramid, introdued by Burt and Adelson 2, is a sequene of images in whih eah image is Laplaian filtered and subsampled opy of its predeessor. The onstrution of this pyramid is as follows. First, a aussian or low-pass pyramid is onstruted. The original image is adopted as the bottom or zero-level of the aussian pyramid. Eah node of pyramid level i (1 i N, where N is the index of the top level of the pyramid, i.e. the lowest resolution level) is obtained as a (aussian) weighted average of the nodes at level i 1 that are positioned within a 5 5 window entered on that node. Beause of the redution in spatial frequeny ontent eah image in the sequene an be represented by an array that has only half the dimensions of its predeessor. The proess whih generates eah image in the sequene from its predeessor is alled a REDUCE operation, sine both the sample density and the resolution are dereased. Thus for 1 l N REDUCE, meaning l 1 we have [ ] l 2 l 1 (8) m, n 2 ( i, j) w( m, n) (2 i + m, 2 j + n) l where w represents a standard binomial aussian filter of 5 5 pixels extent 1. A set of band-pass filtered images L, L1,..., LN 1, that orrespond to Laplaian of differene of low-pass filtered images, an be obtained by taking the differene of suessive levels of the aussian pyramid. Sine these levels differ in sample density it is neessary to interpolate new values between the given values of the lower frequeny image before it an be subtrated from the higher frequeny image. Interpolation is ahieved simply by defining the EXPAND operation as the inverse of the REDUCE operation. Let l, k be the image obtained by applying EXPAND to l k -times. Then (9) and meaning l, l, k EXPAND l, k 1 l (1) 2 i + m j + n l, k ( i, j) 4 w( m, n) l, k 1, m, n where only integer oordinates ontribute to the sum. The sequene and (11) [ ] i i i+ 1 L i is then defined by L EXPAND for i N 1 (12) L N (13) Thus, every level is a differene of two levels in the aussian pyramid, making it equivalent to a onvolution with a Laplaian-like band-pass filter. The Laplaian pyramid is a omplete representation of the input image. an be reovered exatly by reversing the steps used in the onstrution of the pyramid: N and N L (14) N 375

5 [ ] L + EXPAND for i N 1 (15) i i i+ 1 The Laplaian image fusion sheme is a three step proedure 2,13,14. First, a Laplaian pyramid is onstruted for eah of the soure images. Next, a Laplaian pyramid is onstruted for the omposite image by seleting or ombining values from orresponding nodes in the omponent pyramids. Finally, the omposite or fused image is reovered from its pyramid representation through the EXPAND and add reonstrution proedure. Here we apply the Laplaian image fusion sheme to the ahromati omponents of the olor image and the thermal image: / Fusion [ /, / ] (16) f t where Fusion represents the Laplaian image fusion proedure Color image fusion First we determine the first order statistis of the ahromati omponent of the original olor input image. Let / and σ represent respetively the mean and standard deviation of the ahromati band of this image. A fused olor image is obtained by replaing the ahromati omponent of the original olor input image by the result of the Laplaian image fusion proedure. The omponents of this fused image are given by / f,, 6. To assure that the final olor fused image will have the same appearane as the original olor image 9,15, the first order statistis of / are set equal to those of / : Finally, the resulting fused image / f,, rotation R -1, loglms, LMS, and XYZ olor spae using Equation (5). σ / f / f / f + / (17) σ f 6 is transformed bak to RB spae for display, via the inverse f 376

6 3. EXAMPLES Some examples of the fusion of thermal 8-12!"#%$'&(%)*&+&(-,/.12435(. and 2. Note that the onealed weapon is ompletely invisible in the visual images, whereas it is displayed at high ontrast in the thermal imagery. The outline of the grip and the end of the barrel are learly visible in Figures 1b and 2b,e,h. Features orresponding to the trigger assembly and the urve between the rear of the grip and the bak of the barrel are also visible. Also, the ontours of the pair of sissors an easily be pereived in Figure 1e. The ombined or fused display of both image modalities yields images (Figs. 1 and f, and Figs. 2,f and I) in whih the onealed weapon is learly depited in the ontext of its loal bakground. (a) (b) () (d) (e) (f) Figure 1 Left olumn (a,d): original olor photographs. Middle olumn (b,e): thermal 8-12 µm images. Right olumn (,f): olor fused results. Upper row: a lok 17 9mm gun hidden in the bakpoket of a pair of jeans, also partly overed with a polo shirt. Lower row: a hidden pair of sissors. 377

7 (a) (b) () (d) (e) (f) (g) (h) (i) Figure 2 Left olumn (a,d,g): original olor photographs. Middle olumn (b,e,h): thermal 8-12 µm images. Right olumn (,f,i): olor fused results. First row: Smith & Wesson Magnum. Seond row: Walther pp Third row: Sigsauer p 226 9mm. 4. CONCLUSIONS We presented a method to fuse non-literal (e.g. thermal) and olor images suh that the strutural information from the non-literal greylevel image is transferred to the olor image, while the appearane of the resulting fused olor image is idential to that of the original olor image. We ahieve this goal by applying the transform in a pereptually deorrelated olor spae 9,15. The method an be applied in standoff surveillane systems, sine it is apable to learly depit onealed weapons in the ontext of a full olor image of the surveillane sene. The proedure is suh that the relevant ontrast details from the non-literal image are transferred to the olor image without altering the original olor distribution of this image. The result is a natural looking olor image that fluently ombines all details from both input soures. When an observer who performs a dynami rowd surveillane task, detets a weapon in the sene, he will also be able to quikly tag the individual in the rowd that is atually arrying the observed weapon. 378

8 REFERENCES 1. Burt, P.J., The pyramid as a struture for effiient omputation, In: A. Rosenfeld (Ed.), Multiresolution image proessing and analysis, pp. 6-35, Springer, Berlin, E, Burt, P.J. and Adelson, E.H., Merging images through pattern deomposition, In: A.. Tesher (Ed.), Appliations of Digital Image Proessing VIII, SPIE-575, pp , The International Soiety for Optial Engineering, Bellingham, WA, Chen, H.-M., Varshney, P.K., Pramod, K., Uner, M.K. & Rama, L.C. Sensor fusion algorithms and performane limits, (Report A539193), Eletrial Engineering and Computer Siene Department, Universtity of Syrause, New York, (21). 4. Currie, N.C., Demma, F.J., Ferris, D.D., MMillan, R.W. and Wiks, M.C., ARPA/NIJ/Rome Laboratory onealed weapon detetion program: an overview, In: I. Kadar & V. Libby (Ed.), Signal proessing, sensor fusion, and target reognition, SPIE-2755, pp , The International Soiety for Optial Engineering, Bellingham, WA., USA, Currie, N.C., Demma, F.J., Ferris, D.D., MMillan, R.W., Wiks, M.C. and Zyga, K., Imaging sensor fusion for onealed weapon detetion, In: I.K. Rudin & S.K. Bramble (Ed.), Investigative image proessing, SPIE-2942, pp , The International Soiety for Optial Engineering, Bellingham, WA., USA, Fairhild, M.D., Color appearane models, Addison Wesley Longman, In., Reading, MA, Hall, E.L., Computer Image Proessing, Aademi Press, New York, USA, Rama, L.C., Uner, M.K., Varshney, P.K., Alford, M.. and Ferris, D.D., Morphologial filters and waveletbased image fusion for onealed weapons detetion, In: B.V. Dasarathy (Ed.), Sensor Fusion: Arhitetures, Algorithms, and Appliations II, SPIE-3376, pp , The International Soiety for Optial Engineering, Bellingham, WA, Reinhard, E., Ashikhmin, M., ooh, B. and Shirley, P., Color transfer between images, IEEE Computer raphis and Appliations, 21(5), pp , Rihards, J.A., Remote sensing digital image analysis, Springer Verlag, Berlin, Slamani, M.A., Rma, L., Uner, M.K., Varshney, P.K., Weiner, D.D., Alford, M., Ferris, D.D. and Vanniola, V., Enhanement and fusion of data for onealed weapon detetion, In: I. Kadar (Ed.), Signal proessing, sensor fusion, and target reognition VI, SPIE-368, pp. 8-19, The International Soiety for Optial Engineering, Bellingham,WA, Taylor,P. (1999). Statistial methods. In: M.Berthold & D.J.Hand (Eds.), Intelligent data analysis. (pp ). Berlin, E: Springer Verlag. 13. Toet, A., Image fusion by a ratio of low-pass pyramid, Pattern Reognition Letters, 9, pp , Toet, A., Hierarhial image fusion, Mahine Vision and Appliations, 3, pp. 1-11, Toet, A. Paint the night: applying daylight olours to nighttime imagery, (Report TM-2-B6), TNO Human Fators, Soesterberg, The Netherlands, (22). 16. Toet, A. and IJspeert, J.K., Pereptual evaluation of different image fusion shemes, In: I. Kadar (Ed.), Signal Proessing, Sensor Fusion, and Target Reognition X, SPIE-438, pp , The International Soiety for Optial Engineering, Bellingham, WA, Toet, A., van Ruyven, J.J. and Valeton, J.M., Merging thermal and visual images by a ontrast pyramid, Optial Engineering, 28(7), pp , Uner, M.K., Rama, L.C., Varshney, P.K. and Alford, M.., Conealed weapon detetion: an image fusion approah, In: I.K. Rudin & S.K. Bramble (Ed.), Investigative Image Proessing, SPIE-2942, pp , The International Soiety for Optial Engineering, Bellingham, WA., USA,