Control and Data Fusion e-journal: CADFEJL Vol. 1, No. 2, pp , Mar-Apr 2017.

Transcription

1 Control and Data Fusion e-journal: CADFEJL Vol., No., pp. 7-39, Mar-Apr 07. mplementation and Validation o Visual and nrared mage Fusion Techniques in C#.NET Environment B. Hela Saraswathi and VPS Naidu Avionics Department, JNTUK, Kakinada, ndia. Multi Sensor Data Fusion Lab, CSR-National Aerospace Laboratories, Bangalore, ndia. D: helasaraswathi@gmail.com &vpsnaidu@gmail.com Abstract: This paper presents the implementation o image usion techniques by means o an image usion application C#mFuse, developed in C#.NET. C# programming language is a simple, type-sae, object-oriented language that allows programmers to build a variety o applications. C#mFuse application implements our usion methods viz., Alpha Blending (AB), Principle Component Analysis (PCA), Laplacian Pyramid (LP), and Discrete Wavelet Transorm (DWT) or a visual and a thermal image (still images) and or real-time images o the Enhanced Vision System (EVS). The perormance o these usion techniques is evaluated using usion perormance metrics. LP based image usion technique proved to provide better usion when compared to the other techniques. Source code is provided so that the reader can understand the techniques and use or his research work (pl. contact by s). Keywords: mage usion in C#; image processing; Laplacian Pyramid; wavelets. ntroduction mage Fusion is a process o combining the eatures o two or more images o a scene into a single image that is more inormative and is suitable or visual perception or computer processing. Fusion o images can be achieved using various usion methods viz., pixel by pixel averaging [], principal component analysis [], wavelets [], discrete cosine transorm [], Laplacian pyramid [3] etc. When the images to be used are in the same scene but have dierent Field o View (FOV), image registration is required. n this paper, a visual image and a thermal image are considered or usion. While using EO image (colour image o RGB ormat) and R image (a gray image), the usion has to be taken place at intensity level, as this preserves the colour inormation o EO image. Thereore, the EO image is to be converted rom RGB to HS (Hue Saturation ntensit beore perorming the usion. Ater the usion o ntensity component () o EO image and R image, an H and S component o EO image have to be added to the used image and is to be converted to RGB to get back the colour inormation. Figure describes the steps involved in usion o still images. Real-Time image usion is perormed on Electro-optical (EO) and nrared (R) images obtained rom Enhanced Vision System (EVS). The application development or implementing the usion techniques is done in C#, a.net programming language. t is a high-level language and is very easy to work with, compared to the low-level C++. C# is best-suited language or sotware and web applications development in windows platorm[4].thereore, C# is considered in this study to develop C#mFuse..RGB and HS Colour Models RGB and HS are colour models o an image. n RGB model, colours are represented by the amount o red light, green light, and blue light relected rom a scene and they are represented numerically with a set o three numbers, each o which ranges rom 0 to 55. White has the highest RGB value o (55, 55, 55) and black has the lowest value o (0, 0, 0). HS colour model represents every colour with their Hue, Saturation and ntensity. Hue o a colour describes the colour itsel in the orm o an angle between [0 0,360 0 ]. 0 0 means red, 0 0 means green, 40 0 means blue, 60 0 is yellow and is magenta. Saturation component gives a measure o dilution o a colour with white colour. The range o S component is [0, ]. ntensity is the overall lightness or brightness o the colour, deined numerically as the average o the equivalent RGB values [5].. RGB to HS Conversion Conversion o an image rom RGB to HS can be achieved in this application by using EmguCV 7

2 command Cvnvoke.CvtColor(), and parameter ColorConversion.RgbHsvFull [6]..3HS to RGB Conversion HS to RGB conversion can be perormed using the same command, Cvnvoke.CvtColor() but parameter changes to ColorConversion.HsvRgbFull [6].. mage Fusion Techniques Four-usion techniques viz., Alpha Blending, Laplacian pyramid, Principal Component Analysis and Discrete Wavelet Transorm are implemented in the Fusion Application. The ollowing sections give a brie description o these techniques.. Alpha Blending Alpha blending is a process o combining an image with its background. The level o transparency o background and oreground images is controlled by and respectively. Eq. () governs usion o images in Alpha Blending. * ( ( )* ( ) () ( y Where 0, - used image, and - input images to be used ( - Pixel index Figure shows the inormation low diagram o image usion using Alpha Blending. Either o the EO and R images can be taken as the background image. n this study, EO image is taken as background image as it contains colour inormation as well as higher FOV. The transparency o background image is decided by given by the user and the transparency o oreground image is equal to. Alpha blending is done by using Cvnvoke.AddWeighted(), anemgucv unction [6], which takes and as parameters. varies rom 0 to and when it is 0.5, it results in pixel by pixel averaging o EO and R images.. Principal Component Analysis (PCA) PCA technique is similar to Alpha Blending; the only dierence is in determining the weightage to be given to each input image. The weightage is determined dynamically in PCA, whereas the user has to give the weightage (i.e. transparenc in Alpha Blending. PCA technique determines the weightage by calculating the Eigen values rom the Eigen vectors rom the image matrices. The inormation low diagram o PCA-based image usion algorithm is shown in Figure3. The images to be used, ( and ( are arranged in two column vectors and their empirical means are subtracted. The resulting vector has a dimension N, where N is length o each image vector. Eigen vector and Eigen values or the resulting vector are computed and the eigenvectors corresponding to the larger eigenvalue are obtained. The normalized components P and P (i.e., P P ) are computed rom the obtained eigenvector []. The Eigen values are computed using the unction Cvnvoke.Eigen() [6].The used image is: P ( P ( ) () ( y.3 Laplacian Pyramid mage pyramid is a representation o an image in multiple scales. t is constructed by perorming repeated smoothing and subsampling on the image. t is used to extract the eatures o interest, attenuate noise, reduce redundancy, enhance the image and or eicient coding. There are two kinds o image pyramids. Gaussian pyramid, which is constructed by smoothing the image with an appropriate ilter and then subsampling it by a scaling actor o repeatedly. Gaussian pyramid is an example or low pass image pyramids. dk g k (3) Here, g is the Gaussian convolution kernel, is input k image o k th level o Gaussian pyramid and is the dk down sampled image o k th level. The other kind is the Laplacian pyramid, which is a band pass pyramid, constructed by taking the dierence between the adjacent levels o image pyramid. Laplacian pyramids computed as the dierence between the original image and the low pass iltered image. (4) Lk Lk is the Laplacian image o k th level []. K dk y y,3,5,... Where uk y (5) 0 otherwise Laplacian based image usion involves pyramid construction and image usion. Laplacian pyramid is constructed or both EO and R images rom their respective Gaussian pyramids. This is done using the unctions Cvnvoke.PyrDown() and Cvnvoke.PyrUp() [6]. Figure 4 shows the Laplacian pyramid based image usion architecture. Coarse level (inal level images obtained ater decomposition) images o Gaussian pyramids o both EO and R images are averaged (pixel by pixel). The resultant image is up-sampled and added with the maximum image (between EO and R) o Laplacian pyramid in the immediate intermediate level. This is repeated until 8 uk

3 it reaches the ine level (original image) image o Gaussian pyramid..4 Discrete Wavelet Transorm (DWT) Wavelet transorm is a superset o Fourier transorm. n Fourier theory, signal is decomposed into sine and cosines and in wavelet transorm, the signal is projected on a set o wavelet unctions. While Fourier transorm provides good resolution in requency domain, wavelet transorm provides good resolution in both requency and time domains. DWT uses a discrete set o wavelet scales and translation and it decomposes the signal into mutually orthogonal set o wavelets. m n DWT, the dilation actor is a and translation m actor isb n where m and n are integers. Wavelet transorm o a -D signal (x) is deined as: a, b a, b x W x x xdx (6) Scaling unction describes the scaling properties and the wavelets are constructed using it. The translation and dilation o mother wavelet is deined as: x b a b y (7), a a Wavelet separately ilters and down-samples the -D data (image) in horizontal and vertical directions. The input image yis iltered by low pass ilter and high pass ilter in horizontal direction and then downsampled by a actor o two to create coeicient matrices L yand H y. Then the coeicient matrices are both low pass iltered and high pass iltered in vertical direction and down sampled by actor o two, to create sub bands LL y, HL y, LH y and HH y. The LL yrepresents approximation o input image y. Then HL y, LH y and HH y represent the horizontal, vertical and diagonal inormation o the input image y respectively. The inverse -D wavelet transorm is used to reconstruct the original input image yrom the sub bands LL y, HL y, LH y and HH y. This involves column upsampling and iltering using low pass and high pass ilters or each sub images. Row up sampling and iltering with low pass and high pass ilters and summation o all matrices would construct the original image y []. The wavelet coeicient matrices or EO and R images are computed by applying DWT. This matrix contains approximation (LL), horizontal (LH), vertical (HL) and diagonal (HH) components o each image. The used wavelet coeicient matrix is obtained by applying usion rules (Average rule or LL components and maximum rule or the remaining). Finally, the nverse Discrete Wavelet Transorm (DWT) is applied on the used wavelet coeicient matrix to get used image. The inormation low diagram o DWT usion is shown in Figure5. 3. Fusion Quality Metrics Perormance o usion techniques can be evaluated using two types o analyses: Subjective Analysis and Objective Analysis. Subjective Analysis is based on personal opinion, interpretation and judgement. Objective Analysis is based on theory and numerical calculations. Objective Analysis on used image is done using Fusion Quality Metrics. These are o two types, Reerence metrics and No Reerence metrics. Reerence metrics involves comparison o used image with a reerence image whereas No Reerence metrics do not require a reerence image. No Reerence metrics are computed in this application as it involves usion o two dierent images and no single image can be taken as a reerence. These metrics are used to compare the eiciency o each usion technique and to ind out the best perorming technique. A set o perormance metrics are implemented in this usion application, whose description is ollowed in the ollowing section. 3. Standard Deviation Standard deviation is a measure that is used to quantiy the amount o variation o a set o data values. Low standard deviation indicates that the data values tend to be close to the mean o the set and high standard deviation value indicates that the data points spread out over a wide range o values. n image processing, standard deviation is variation o pixel values with respect to the mean o all pixel values o an image [7]. M N y (8) MN x0 y0 Where, M and N represents the number o rows and columns, is the mean o all pixel values in the used image. 3. Entropy Entropy is a measure o inormation content o an image. t is sensitive to noise and unwanted rapid luctuations. t is high or the image with high inormation content [7]. 9

4 L He h ( i)log h ( i) i0 (9) Where h is the normalized histogram o the used image. The unit o entropy is bits/pixel. 3.3 Cross Entropy Cross entropy is used to veriy the similarity in inormation content between input and used image. Low values o cross entropy indicate that the input and used images are almost similar [7]. Overall cross entropy o the input images, and the used image is CE( ; ) CE( ; ) CE(, : ) (0) Where L hi ( i) i CE( ; ) hi ( i)log( ) i i0 hi ( i) () L hi ( i) CE( ; ) hi ( i)log( ) i0 hi ( i) () 3.4 Spatial Frequency Spatial requency reers to the level o detail present in a stimulus per degree o visual angle. A scene with small details and sharp edges contains more high spatial requency inormation than one composed o large coarse stimuli. This metric indicates the overall activity level in the used image [7]. Spatial requency criterion SF is: SF Where row requency o the image RF M N [ MN x y RF CF (3) ( ( y )] And column requency o the image CF M N [ MN x y ( ( x, ] (4) (5) 3.5 Fusion Mutual normation Fusion Mutual normation indicates the degree o dependence o the used image on the source images. The larger value o usion mutual inormation implies better quality [7]. The joint histogram o source image yand x y i j and or source image, is deined as h, yand yis deined as h i j mutual inormation is deined as ollows,,. The FM M M Where M M h i, j i, jh i, j (6) M N h i, j log (7) i j h h i, j i, jh i, j M N h i, j log (8) i j h 3.6 Fusion Quality ndex The range o this metric is 0 to. indicates the used image contains all the inormation rom the source images. FQ c w)( ( w) Q(, w) ( ( w)) Q(, w)) ww ( Where w andcw max (9) computed over a window, over a window and w normalized version o c w& Q, w c is is the quality index over a window or given source image and used image [7]. 3.7Average Contrast Contrast is a visual characteristic that makes an object or its representation in an image distinguishable rom other objects and the background. n visual perception, contrast is determined by the dierence in the colour and brightness o the object and other objects within the same ield o view and higher contrast value is preerable [7]. M N Cavg C( (0) ( M )( N ) x y Where, M and N represents the number o rows and columns o an image. For an R image, the contrast is the gradient calculated or the image as a single component: C( ( () ( ( x y iˆ (, ) ˆj x y Where, = gradient operator y=mage pixel value at y Average gradient relects the clarity o an image. t measures the spatial resolution in an image i.e. larger average gradient indicates a higher resolution. Higher 30

5 value o Average Contrast is an indication o better image quality. For a colour image, the colour contrast is given by the average o gradients o Red, Green and Blue considered individually as ollows: R( G( B( C( 3 () 3.8 Average Luminance Luminance describes the amount o light that passes through, or is emitted rom a particular area, and alls within a given solid angle. t indicates the amount o luminous power perceived by an eye looking at the surace rom a particular angle o view. Luminance is thus an indicator o how bright the surace will appear [7]. L avg MN M N x y ( For colour image, M N R( G( B( Lavg MN x y 3 Higher luminance value represents the higher brightness value o an image. (3) (4) 3.9Energy Energy returns the sum o squared elements in the Gray Level Co-occurrence Matrix (GLCM). t is also known as uniormity, uniormity o energy or angular second moment. The energy o an image lies between zero and one [7]. E 8 8 g( i, j) i j (5) 3.0Homogeneity Homogeneity is a condition in which all the constituents are o the same nature. Homogeneity returns a value that measures the closeness o the distribution o elements in the Gray Level Cooccurrence Matrix (GLCM) to the GLCM diagonal i.e. i all the pixels in a block are within a speciic dynamic range. The range homogeneity is rom zero to one. Homogeneity is or a diagonal GLCM [7]. 8 8 g( i, j) hom i j i j (6) 3. SNR Signal to Noise Ratio will be high when the reerence and used images are alike. Higher value o SNR implies better usion. SNR (7) Where, is the mean and is the standard deviation o the used image. 4. System Requirements mplementation o usion in C#mFuse, image usion application, need some pre-requisites that include hardware equipment (only or real-time image usion) and sotware setup. The details o both are discussed in the sections ollowed. 4. Hardware Setup Hardware equipment includes EVS, Frame Grabber, RS-70 connectors and a computer. EVS has two imaging sensors (EO and R) which operate using +Vbattery. The outputs o the cameras are connected to the rame grabber through two RS-70 cables. The hardware setup used or implementing real-time image usion is shown in Figure EO and R maging Sensor Speciications The EO imaging sensor is a CMOS sensor that senses the relected energy with the requency o the energy relected giving an image. t requires a light source to provide the image and is more sensitive than the eye. The R imaging sensor senses based on the radiant energy emitted by objects. This cannot be detected by human eye as the wavelength o emitted radiations alls in inrared region o electro-magnetic spectrum. R imaging sensor incorporates an uncooled 34x56 pixels micro bolometer. t has an internal heater to derost its protective window. The technical speciications o EO and R cameras are given in the Table. 4..Sensoray Frame Grabber A our-channel Sensory Frame Grabber (55) is used or capturing image rames rom both the cameras at a desired rame rate. Total capture rate is 60 rames/sec rom each channels or NTSC colour video, but when channels are used simultaneously, capture rate is 30 rames/sec each, or 4 channels at 5 rames/sec each. Here, two channels are used and so the maximum rame rate achieved is 30 rames/sec. The digitized output rom the rame grabber is given to the computer by using USB cable [8]. 4. Sotware Setup A computer installed with Visual Studio (v03 or higher) is required to develop this application. Fusion methods are implemented using EmguCV, a C# 3

6 wrapper or Open Source Computer Vision (OpenCV) image processing library in Visual Studio platorm. EmguCV opens OpenCV library o programming unctions, mainly aimed at real-time computer vision to C# developers. C#mFuse was developed using Windows Forms(in C# programming language) [9]. 4.. nstalling EmguCV Working with images in Visual Studio C# requires external packages to be installed in the project. Respective.dll iles o the package can be downloaded and included in the project. Using Package Manager Console (PMC) to install external packages is an easier way. Locate PMC in Visual Studio at Tools >Library Package Manger > Package Manager Console. Open it and type nstall-package EmguCV to install the corresponding package in the current project [0]. Using PMC, addition o packages into the project can be automated instead o manually going to the NuGet U to add packages. The packages will automatically be downloaded rom the internet, using the web-link speciied in Tools > Library Packet Manager > Package Manager Settings > Package Manager > Packager Sources. EmguCV package or this application is downloaded rom Re. []. 5.Real-Time mage Fusion Real-Time usion o images involves capturing the images in real-time, image registration and applying usion techniques that results in a used image. The Fusion process involves the steps shown in Figure mage Registration mage registration is a process o aligning two or more images o the same scene. One image is taken as reerence image (ixed image) and geometric transormations will be applied on the other image (moving image) to align with the reerence image. Here, EO image is the reerence image as it has higher FOV and transormations are applied to R image. mage registration is done o-line or EO and R images using Control Point image registration toolbox o MATLAB []. n Control Point image registration, the user has to select same eature points on both images manually as shown in Figure 8.Control Points (ixed points and moving points) are obtained rom MATLAB and used to get the Transormation matrix (Aine Transorm) using Cvnvoke.GetAine() command [6]. Aine Transormation is a x3 matri used to express linear transormations (rotations and scale operations) and translation (vector addition). When shapes in the moving image exhibit shearing, this transormation is applied. Straight lines remain straight and parallel lines remain parallel, but rectangles become parallelograms. A minimum o three non-collinear points are needed to iner aine transorm. Aine transorm obtained or images in Figure 8 is shown below: The command Cvnvoke.WarpAine() transorms the image [6]. This results in a registered image (registered R image in this case) as shown in Figure Results and Discussions This application is capable o using two user-given images using our dierent usion algorithms viz., Alpha blending, Laplacian Pyramid, PCA and DWT. 6.Still mages Once the application is run, the ront end o the C#mFuse appears as shown in Figure 0. The user has to select the images to be used. These images must be o the size and size must be in powers o. the images are o dierent sizes, the application pops up a window showing a message that the images are o dierent sizes. Later, the user has to select the usion method rom the drop down menu. C#mFuse applies the selected usion technique and gives used image output along with the usion quality metrics displayed in the Fusion Quality Metrics pane. t also displays the time taken by the application in perorming the usion. The images taken or usion are in same FOV. Thereore, image registration is not required. Figure shows the EO and R images obtained rom Re. [3].n Figure(a), the lower part o the scissors is hidden behind the black wrapper. On the other hand, the R image shows ull image o the scissors because o the penetrating nature o R camera. Thereore, in Figure (b), the hidden part o scissors and some holes on the wrapper, which can t be ound in EO image, are visible, but colour inormation is not there. So, the used image obtained by applying various usion methods is expected to contain all the eatures o both the images. 6..Alpha Blending n Alpha Blending, used images obtained by varying the transparency (using various values) are shown in Figure. n Figure (a), is 0., so it gives more weightage to R image, showing all the eatures o R 3

7 image like the holes in the wrapper, lower part o the scissors that is hidden in EO image etc. t retains the colour inormation o EO image. Similarly, in Figure (c), is 0.8, so used image elevates the eatures o EO image giving it high weightage while retaining eatures o R image. n Figure (b), is 0.5, the used image shows that equal weightage is given to both the input images and this is nothing but a simple pixel by pixel averaging o input images. Alpha Blending results in a very smooth usion, which is because o its high Fusion Quality ndex compared to the other usion methods, as shown in Table. t is a very simple usion method and so the elapsed time or usion is very less. Table 3 shows variations in Fusion Quality Metrics as is increased in its range [0, ]. t is observed that the usion perormance in terms o these metrics is better or lower values o where R image is given high weightage. 6.. Principle Component Analysis PCA usion results in an image, which shows dark background, similar to the used images obtained with high alpha values in alpha blending. This means that dynamically calculated weightages highlight EO image than R or this set o images. The used image retains the eatures o both the input images. The overall perormance o usion is better than alpha blending and DWT, as per the Fusion Quality Metrics shown in Table 4. As this method involves calculation o weightages, it takes relatively more time to use the images. The usion o EO and R images using PCA technique is shown in Figure Laplacian Pyramid Laplacian pyramid technique is observed to be edgesensitive and highlights the edges o all the objects in input images. The used image contains the edge inormation o the EO image and temperature highlighted edge inormation rom R image. This technique exhibits high Average Luminance (brightness), compared to other usion techniques. Thereore, the used images are relatively brighter. Variations in Fusion Quality Metrics with the number o decomposition levels o Laplacian Pyramid, is shown in Table 4. The number o levels o image pyramid is limited to n-. This technique exhibits better usion at lower levels o decomposition. The brightness o the used image increases with increased levels o decomposition, which can be observed in Figure 4. From the metrics shown in Table, it is inerred that Laplacian pyramid technique perorms better than other usion techniques. t has better usion quality and the time elapsed or usion is also relatively less Discrete Wavelet Transorm The used image obtained by applying wavelet transorm or one level o decomposition is shown in Figure5. The used image contains temperaturesensitive inormation rom R image and coloursensitive inormation rom EO image. DWT exhibits high contrast in the used image. t is a very time consuming technique as it involves pixel level operations. The used images obtained by applying DWT and Alpha Blending exhibit similar properties, as per the Fusion Quality Metrics shown in Table. A variation in Fusion Quality Metrics with decomposition levels is shown in Table 5. DWT shows relatively good usion at higher levels o decomposition. 6. Real-Time mages C#mFuse captures real-time images rom EO and R cameras and applies the selected usion technique. Figure 6(a) shows the EO image and 6(b) shows R image captured in real-time. 6.. Alpha Blending Alpha Blending is applied on EO and R images or values 0., 0.5 and 0.8. The results are shown in Figure 7. As is increased, the contents o EO image are highlighted. Fused image has high Homogeneity compared to other methods and so it appears close to the input images. The Fusion Quality Metrics or Alpha Blending in comparison with other methods is shown in Table 6.For small values o, R image is highlighted. t is a simple usion method and consumes very less time than other usion methods. 6.. Principle Component Analysis PCA used image exhibits very good Fusion Quality ndex compared to other techniques as per the Fusion Quality Metrics shown in Table 6. Thereore, the used image shown in Figure 8 shows a very smooth usion. Fused image highlights EO image as the Eigen value corresponding to the EO image is high in value. t is observed that PCA results in better usion next to Laplacian Pyramid method Laplacian Pyramid Laplacian Pyramid technique image exhibits very good usion compared to other methods, as per the Fusion 33

8 Metrics in Table 6. The used image appears very bright because o high Luminance. Fusion is perormed up to our levels o decomposition and used images are shown in Figure 9.t is observed that used images gives high weightage to the edges o all the objects rom images to be used Discrete Wavelet Transorm DWT used image exhibits high contrast compared to other methods. ts perormance is observed to be very close to alpha blending, except that it is highly time consuming, involving complex operations. Figure 0 shows the DWT used image or one level o decomposition o real-time EO and R images. The time elapsed increases with increased levels o decomposition. CONCLUSON Four image usion techniques viz., alpha blending, Laplacian Pyramid, PCA and DWT are implemented and tested on still images as well as real-time EO and R images. The perormance o these methods is evaluated using a set o Fusion QualityMetrics. Based on used images and Fusion Quality Metrics, it is concluded that the Laplacian pyramid based usion method provides better results compared to the other usion methods. Source code is provided to understand the usion techniques and or easy implementation. REFERENCES VPS Naidu and J.R Raol, Pixel-level mage Fusion using Wavelet and Principal Component Analysis, Deence Science Journal, Vol.58, No.3, May 008 VPS Naidu, Discrete Cosine Transorm-based mage Fusion, Special ssue on Mobile ntelligent Autonomous System, Deence Science Journal, Vol. 60, No., pp.48-54, Jan VPS Naidu, Block DCT-based mage Fusion Techniques, e- Journal o Science & Technology,(), 9, accessed on 8 th July, 07 5 Raael C. Gonzalez and Richard E. Woods-Digital mage Processing-Second Edition, tml/bb b7-5a-594c-43b6da.htm, as accessed on 8 th July, 07 7 VPS Naidu and L. Garlin Delphina, Assessment o Colour and nrared images using No-reerence mage Quality Metrics, Proceedings o NCATC as accessed on 8 th July, as accessed on 6 th July, as accessed on 6 th July, 07 as accessed on 6 th July, 07 ns.html#inputarg_transormationtype, as accessed on 8 th July, 07 3 D. Looney and D.P. Mandic, Fusion o Visual and Thermal images using complex extension o EMD, EEE Xplore 009 Table Technical Speciications o EO and R imaging sensors Technical Speciications EO imaging sensor R imaging sensor Detector Type Focal Plane Array (FPA) uncooled micro bolometer CMOS ¼ Field o View 36 (H) x 7 (V) with 9 38 (H) x 5 (V) with 6.8 mm lens mm lens Output Formats Analog, CCR/PAL composite video, 75Ω NTSC/PAL Analog, Raw RGB,.0Vpp /75ΩComposite Video Signal nput Power Supply 6-6 V DC Digital Core 5V DC ~ 4VDC 34

9 Table Fusion Quality Metrics (FQM) or still images (Figure ) Alpha Principle Laplacian Blending Component Pyramid ( = 0.5) Analysis (levels =) Fusion Quality Metric DWT (levels =) Standard Deviation Entropy Cross Entropy Spatial Frequency Fusion Mutual normation Fusion Quality ndex Average Contrast Average Luminance Energy Homogeneity SNR Time Elapsed (ms) Table 3Variation in Fusion Quality Metrics with in Alpha Blending or still images (Figure ) Fusion Quality value [0,] Metric Standard Deviation Entropy Cross Entropy Spatial Frequency Fusion Mutual normation Fusion Quality ndex Average Contrast Average Luminance Energy Homogeneity SNR Table 4 Variation in FQMs with no. o levels o decomposition in Laplacian Pyramid or still images Fusion Quality Number o levels o decomposition (L) Metric Standard Deviation Entropy Cross Entropy Spatial Frequency Fusion Mutual normation Fusion Quality ndex Average Contrast Average Luminance Energy Homogeneity SNR

10 Table 5 Variation in Fusion Quality Metrics with levels o decomposition in DWT or still images Fusion Quality Number o levels o decomposition (L) Metric Standard Deviation Entropy Cross Entropy Spatial Frequency Fusion Mutual normation Fusion Quality ndex Average Contrast Average Luminance Energy Homogeneity SNR Table 6 Fusion Quality Metrics or Real-Time images (Figure 5) Fusion Quality Metric Alpha Blending ( = 0.5) Principle Component Analysis Laplacian Pyramid (levels =) DWT (levels =) Standard Deviation Entropy Cross Entropy Spatial Frequency Fusion Mutual normation Fusion Quality ndex Average Contrast Average Luminance Energy Homogeneity SNR Time Elapsed (ms) EO mage R mage RGB to HS mage usion Figure Steps involved in image usion o still images H & S HS to RGB Fused RGB mage Registered Source mages Fused mage 36

11 Figure Alpha Blending based image usion Figure 6Hardware Setup or Real-Time image usion Registered Source mages PCA P P Fused mage Registered Source mages P Figure 3PCA based image usion Laplacian Pyramid Construction P Coarse level image Fused mage Capture EO Frame Capture R Frame RGB to HS Aine matrix mage Registration mage usion H & S HS to RGB Fusion Rules Fused RGB mage Figure 7Steps involved in real-time image usion Coarse level image Pyramid Reconstruction Figure 4 Laplacian Pyramid based image usion Registered Source mages Wavelet Coeicients Fused Wavelet Coeicients Fused mage DWT Fusion Rules DWT DWT Figure 5 Wavelet based image usion Figure 8 Point selection in Control Point toolbox or image registration in MATLAB EVS Frame Grabber +V EO RS 70 USB R EO R 37

12 Figure 9 Registered R image (c) Figure Fused images or alpha (a) 0., (b) 0.5 and (c) 0.8 Figure 3PCA Fused image Figure 0Front-end view o C#mFuse application (a) (b) (a) Figure (a) EO image, (b) R image (b) (c) (d) Figure 4Fused images with dierent Laplacian Pyramid levels(a), (b), (c) 3 and (d) 4 (a) (b) 38

13 Figure 5DWT Fused image or one level o decomposition Figure 8PCA Fused image (a) (b) Figure 6(a) EO image and (b) R image (a) (b) (c) Figure 9Fused images with dierent Laplacian Pyramid levels (a), (b), (c) 3 and (d) 4 (d) (c) Figure 7Fused images or alpha (a) 0., (b) 0.5 and (c) 0.7 Figure 0DWT Fused image or one level o decomposition 39