Automatic Optical and IR Image Fusion for Plant Water Stress Analysis

Transcription

1 12th International Conference on Information Fusion Seattle, WA, USA, July 6-9, 2009 Automatic Optical and IR Image Fusion for Plant Water Stress Analysis Weiping Yang, Xuezhi Wang*, Ashley Wheaton, Nicola Cooley, and Bill Moran* * Melbourne Systems Laboratory Faculty of Engineering, University of Melbourne, Australia b.moran@unimelb.edu.au, xwang@unimelb.edu.au ATRLAB, School of Electronic Science and Engineering, National University of Defense Technology, Changsha, Hunan, , P.R.China wpyang@126.com Department of Land and Environment, University of Melbourne, Australia awheaton@unimelb.edu.au, n.cooley@unimelb.edu.au Abstract Automatic registration of an optical image and an associated IR image is a key step toward to automation of canopy temperature measurement in the process of plant water stress analysis. In this context, the scene of the IR image is completely included in the optical image and the transform between the two images may involve translation and rotation by a small angle, though a small scale difference may also be present. This automatic registration of data from two quite different imaging regimes presents several challenges, and is not susceptible to several common image processing techniques. In this paper, an automatic image registration algorithm, based on the fundamental cross correlation method is designed, which can avoid human intervention in the alignment process and is suitable for the application to plant water stress analysis where significant numbers of images need to be processed. The effectiveness of the software design is demonstrated via our experiments and the registration error performance is compared to the cases where the similarity criterion is replaced by that of mutual information and correlation ratio respectively. program to estimate spatial and temporal variation in water status of grapevines using data collected from remotely sensed digital images [5]. The project aims to build up an automatic controlled irrigation program with nondestructive sensing and measurement systems [6]. In the process of plant water stress analysis, the temperature profile of a specific area can be acquired from the software output of the IR camera. With a reference optical image, which is taken at the same spot using a normal camera to provide a true view of the IR image scene, the area of interest (e.g., plant leaves other than ground or sky) may be identified in the IR image. Subsequently temperature data associated with the area of interest is extracted and statistical analysis can be performed. In order to obtain statis- Keywords: Automatic image registration, Optical and IR image fusion, Water stress analysis using temperature profile, Thermal imagery, Cross correlation. 1 Introduction Advanced infrared (IR) sensing technology enables acquisition of high quality thermography imagery [1] to provide capability for analysis of plant water stress [2] from canopy temperature data obtained from IR cameras [3, 4]. The Melbourne School of Land and Environment at The University of Melbourne is currently conducting a research Figure 1: Example of image pair 1: Cabernet Sauvignon grapevines. f IR = IR image (right) and f o = optical image of the scene (left). The scene of the IR image is in that of the optical image. tically robust data, a large number of images over a time course should be collected, processed and analyzed. It is de ISIF 1053

2 sirable for image collection to be integrated into real time automated control irrigation programs and therefore essential that the image analysis process be automated. A key step towards this process is to automatically determine the overlap area between the IR image and associated optical image. An example of optical and IR image pair taken on grapevines is given in Figure 1, in which we need to identify the area overlapped with the IR image from the left optical image. Clearly, whether we can correctly register the overlapped area from the optical image can significantly influence of canopy temperature estimation outcomes. The major difficulties which arise in this type of image registration are listed below. Taken by different type of sensors and possibly at different view angle and different time, the pair of optical and IR images are in general not matched exactly. At the best matching overlap area, the intensities of both images can be quite different. Therefore, approaches involving image intensity are unlikely to obtain a satisfied registration performance. Apart from some similarity of overall structures, it is difficult to identify consistent feature points from both images in some popular feature spaces via an automatic registration method, such as the scale invariant feature transformation (SIFT) method [7]. A variety of image registration techniques and algorithms which may be used in this automatic alignment application are available in the literature [8] (and references therein). The approaches fundamentally fall into two main categories, i.e., area-based methods and feature-based methods. The SIFT method perhaps is the most representative approach in the feature-based automatic methods. The SIFT implementation is able to find distinctive points that are invariant to location, scale and rotation, and robust to affine transformations (changes in scale, rotation, shear, and position) and changes in illumination for images of the same source or of the same type of sensors [7]. When this is the case, the algorithm is particularly effective. However, for our application the success rate of SIFT is less than 10%. In most cases, there is simply no SIFT point at all in the IR image. Solutions based on area correlation technique [9] seems to be more robust to our problem, except for those which use intensity (or color) dependent functions as similarity measures, such as the Fourier transformation type [10] and mutual information type [11] approaches. The maximum correlation coefficient detection method was initially proposed in [12] and later extended in [9] by considering the correlation in a feature space. Instead of using cross correlation coefficient, Huttenlocher et. al. used the Hausdorff distance as a similarity measure to register binary images that are the output of an edge detector [13]. To deal with the problem of multimodality medical image registration, Maes et. al. proposed 1054 a method in [14] which applies mutual information (MI) to measure the statistical dependence or information redundancy between the image intensities of corresponding pixels in both images. Using correlation ratio (CR) as the similarity measure, Roche et. al. proposed a CR method in [15] which assumes the pixel-pair intensities between two registered images are functional dependent. These area-based methods were summarized in [16]. In this paper, we consider the registration problem between an optical image and an IR image 1. The images are taken using a hand held camera, and a hand held thermal imager, over a time span of less than a minute. In the situation studied, the scene of the IR image is completely included in the optical image and the transform between the two images may involve translation and rotation by a small angle, though a small scale difference may also be present. An automatic image registration algorithm based on the fundamental cross correlation method is designed, which avoids human intervention in the alignment process and is suitable for the application to plant water stress analysis where significant amounts of data are required. The effectiveness of the software design is demonstrated via our experiments and the registration error performance is compared to the cases where the similarity criterion is replaced by that of mutual information and correlation ratio respectively. The rest of the paper is organized as follows. The problem description is given in the next section. We present the automatic cross correlation (ACC) image registration approach in detail in the Section 3. Experimental results and discussions are given in Section 4, followed by the conclusion. 2 Problem Description Let F o and F IR denote the optical and IR images of an image pair respectively. From the application at hand, we can reasonably assume that: 1. The scene of F IR is completely within that of F o. In other words, there is an area in the optical image F o where the scene of the IR image F IR is approximately matched. 2. The matching area in F o may differ by a translation b, a small rotation θ [say θ ( 10 o,10 o )] and a slight scaling s 1 as in (1). Therefore, a point (x,y) on the reference image F IR and the related point (x o, y o ) on the base image F o are connected by a linear transformation. [ xo y o ] [ cos θ, sinθ = s sinθ, cos θ ] [ x y ] [ bx + b y where s is the scaling factor, θ is the rotation angle and [b x, b y ] is a translation vector. The registration problem in the context of (1) is to find the parameters s, θ and b and obtain the optical image part ] (1) 1 To be precise, the intensity of the IR image is a function of the canopy temperature.

3 F OIR F o associated with the IR image F IR using these parameters. In addition, we need the registration process to be completed automatically. Note that, in general there is no complete matching between the two images and we can only find the overlapping area of the best matching. σ 2 u,v = σ 2 IR = 1 S(f u,v ) 1 S(f u,v) ( fu,v () f u,v ) 2 ( fir () f IR ) 2 3 Automatic Cross Correlation Alignment Algorithm Implementation The ACC algorithm designed and implemented in this work is based on the cross correlation (CC) method described in [12]. The algorithm is described in Table 1 and the process flowchart is given in Figure 2. In the algorithm, auxiliary filters are used to remove intensity information so that the correlation of the pair of images is purely based on the geometric structure over the area of interest. This is justified by the following observations: As the intensities of the best overlapped pixels of the image pair are not consistent, direct application of the CC, MI or CR method simply does not work at all. Consequently, we used a binary edge detector filter to remove intensity information while retaining structure information. It was found in our experiment, by using the black and white (BW) edge images, the CC method has a very high success alignment rate over MI and CR methods even use resolution reduced images, though almost no common edge can be found in the images as shown in Figure 3. Two methods using freely available SIFT software were tried for the registration problem. However, as discussed in Section 4, neither of them were able to produce consistent results. Most pairs of optical and IR images collected in this work can be manually registered via several control points identified by eye. Apart from potentially a larger registration error, this is time demanding and not suitable for processing large numbers of images. Denote by S(f), the size (or resolution) of the image f and f u,v, the (u, v)th image window of a given search grid over the image f o. The cross correlation coefficient matrix R in Table 1 is computed by R(u, v) = where S(fIR ) ( fu,v () f ) ( u,v fir () f ) IR σ u,v σ IR (2) f u,v = f IR = S(f 1 u,v) f u,v () S(f u,v ) 1 f IR () (2) will be calculated for every point (u, v) over the search grid of size S(R). 1. Input: F o,f IR (possibly in reduced resolution) and predefined parameters: search grid, θ range and accuracy. 2. Binary filtering using the Canny type edge detector: F o f o, F IR f IR. 3. CC1 loop: (a) for each given search grid point (u, v), compute the cross correlation coefficient using (2). (b) weight the coefficient matrix R using the predefined probability distribution (for the location of the overlapped area). (c) find the control point CP1 which is the location corresponding to the maximum value of R,i.e., cp1 = arg max(r) (3) R 4. CC2 loop: estimate the rotation angle ˆθ. This is done by turning the candidate image window about the CP1 within a given θ range and accuracy (grid) and calculating the cross correlation coefficient R θ using (2). The estimated rotation angle is then given by ˆθ = arg max(r θ ) (4) θ i 5. based on estimated CP1 and ˆθ estimate the overlapped area from F o and output the image F OIR. Table 1: The steps of automatic image registration. 4 Experiment Results and Discussions The ACC algorithm was tested using many sample pair of images which were collected by the researchers from different plant fields. All of them can be successfully registered with tolerable errors 2. For the purpose of performance comparison, we also implemented the algorithms which use the normalized MI (NMI) and CR as a similarity measure described in [16], respectively. Both of them use a gray scaled F o and F IR. 2 In fact the ground truth is unknown. So, what we mean tolerable error is that the registration result has passed visual examination conducted by an expert. 1055

4 F O F IR ED O f CC1 O f IR ED IR Control Point Estimator (a) (b) Figure 4: Color images of the aligned pair in Figure 3. F O cp1 Initial overlapped f O part CC2 f O cp1 Rotation Estimator Transform Parameters & Matching Output θ^ θ range F OIR Figure 2: Image alignment algorithm flowchart. (a) (b) Output Figure 3: Comparison of the aligned image pair filtered by a canny edge detector. (a) filtered optical image f o. (b) filtered IR image f IR. In the NMI method the correlation matrix R MI (u, v) is calculated by i R MI (u, v) = (P u,v(i) log P u,v (i) + P IR (i) log P IR (i)) P OIR,u,v() log P OIR,u,v () (5) where P OIR,u,v () is the joint probability that the intensities of F u,v () and F IR () are at levels i and j respectively. P u,v (i) and P IR (i) are the marginal probability of the images F u,v () and F IR (). These probabilities can be computed from the normalized joint and marginal intensity histograms. In the CR method the correlation matrix R CR (u, v) is calculated by where and Var IR = i Var u,v (i) = ( i 2 P IR (i) ip IR (i) i ) 2 1 j 2 P OIR,u,v () P u,v (i) j 1 j P OIR,u,v () P u,v (i) All summations in (5) and (6) are taken over image intensity space. As mentioned earlier, when taking the maximum value of the correlation matrix R, a probability weighting matrix is used to eliminate possible false maxima on the edge of the optical image. Implicitly, we assume that the correct alignment area is centered at the center of the optical image has a Gaussian distribution with the standard deviation σ. In the experiment, σ takes a value of approximately 1/3 of the image size. Figure 5 illustrates the weighted result for the example image pair 1 (see Figure 1), where the correlation matrix is obtained using ACC. To demonstrate the image registration performance, we present two examples. Image pair 1, as shown in Figure 1, was taken from the scene of Cabernet Sauvignon grapevines. Image pair 2, as shown in Figure 6, was taken from a similar scene to that of image pair 1 but with a tag box. The tag box provided a reference temperature for analysis purpose. The registration results are presented in Figures Since the ground truth is unknown, as given in Table 2, we compared the registration error for NMI and CR methods against the ACC results which were deemed to be the best alignments by visual examination. The registration error is measured using the distance between the estimated location of the registered image and the truth on the optical image. Image ACC NMI CR Pair Deviation Error Deviation Error Deviation Error Pair 1 [0, 0] 0 [ 12, 26] [ 36, 59] Pair 2 [0, 0] 0 [1, 22] [ 40, 83] Table 2: Registration error comparison j 2 R CR (u, v) = 1 1 Var u,v (i)p u,v (i) (6) Var IR Discussions: i 1056

5 (a) Figure 6: Example image pair 2: Cabernet Sauvignon grapevines with a tag box. (b) Figure 7: ACC result: matched image pair 1, where θ = 0.3 o and the shift b = [214, 134 ]. (c) Figure 5: Weighted correlation coefficient matrix of size S(F o ). (a) correlation coefficients; (b) weighting function; (c) weighted correlation coefficients. 1. Figures 7 12 and Table 2 demonstrated that the proposed ACC algorithm provides the best and consistent alignment result compared to that of NMI and CR in the context of automatic registration of optical and IR image pairs. Using the same performance comparison technique, we considered 10 more pairs of images taken from various plants. As indicated in Table 3, the result is consistent with our claim. 2. From our experiments, we found that SIFT method produces mixed results with less than 10% success rate and hence is not suitable for our application. water stress analysis. The proposed algorithm has used the information of coherent image structure but isolated image intensity impact in the alignment process and hence produces satisfactory and robust registration result. The ACC performance is compared to other area-based methods which can be used for the underlying application in the automatic registration context. We found that some image pairs taken from low altitude balloons were only partially matched with each other with large rotation angles as a result of limited sensor controllability. The automatic alignment of these images is an issue and will be addressed in our future work. References [1] N.A.L.Archer, and H.G. Jones. Integrating hyperspectral imagery at different scales to estimate component surface temperatures, International Journal of Remote Sensing, vol. 27, no. 11, pp , June [2] I. Leinonen, and H. G. Jones. Combining thermal 5 Conclusion and visible imagery for estimating canopy temperature In this paper, the ACC algorithm is proposed for registration of optical and IR image pairs automatically for plant Botany, vol. 55, no. 401, pp , June and identifying plant stress, Journal of Experimental 1057

6 Figure 8: ACC result: matched image pair 2, where θ = 0.4 o and the shift b = [230, 145 ]. Figure 10: NMI result: matched image pair 2, where θ = 0.2 o and the shift b = [231, 167 ]. Figure 9: NMI result: matched image pair 1, where θ = 0.4 o and the shift b = [202, 160 ]. [3] C. Campillo, M. H. Prieto, C. Daza, M.J. Monino, and M.I. Garcia. Using digital images to characterize canopy coverage and light interception in a processing tomato crop, Journal of HortScience, Vol. 43, no. 6, pp , [4] M. Saudreau, A. Marquier, B. Adam, P. Monney, and H. Sinoquet. Experimental study of fruit temperature dynamics within apple tree crowns, Elsevier: Journal of Agricultural and Forest Meteorology Vol. 149, pp , [5] A. D. Wheaton, N. Cooley, G. Dunn, I. Goodwin, and S. Needs. Evaluation of infrared thermography to determine the crop water status of Cabernet Sauvignon grapevines, Poster paper of 13th Australian Wine Industry Technical Conference, Adelaide, 28 July 2 August, Figure 11: CR result: matched image pair 1, where θ = 0.5 o and the shift b = [175, 75 ]. conductance over leaf surfaces, Journal of Plant Cell And Environment,vol. 22, no.9, pp , [7] D. G. Lowe. Object recognition from local scaleinvariant features, Proc. of the Seventh IEEE International Conference on Computer Vision, vol. 2, pp , [8] B. Zitová, and J. Flusser. Image registration methods: a survey, Journal of Image and Vision Computing (Elsevier), vol. 21, pp , [9] W. K. Pratt. Correlation techniques of image registration, IEEE Trans. on AES, vol. AES-10, no. 3, pp , May [10] H. Liu, B. Guo, and Z. Feng. Pseudo-Log-Polar Fourier Transform for Image Registration, IEEE Signal Processing Letters, vol. 13, no. 1, pp , Jan [6] H. G. Jones. Use of thermography for quantitative [11] P. Viola, and W. M. Wells III. Alignment by Maximization of Mutual Information, International Jour- studies of spatial and temporal variation of stomatal 1058

7 Figure 12: CR result: matched image pair 2, where θ = 0.3 o and the shift b = [190, 228 ]. ACC NMI CR Deviation Error Deviation Error Deviation Error [2.5, 2.3] 3.40 [6.9, 11.6] [32.4, 92.7] Table 3: Registration error averaged over 10 pairs nal of Computer Vision, vol. 24, no. 2, pp , [12] P.F. Anuta. Digital registration of multispectral video inagery, Journal of Soc. Photo-Optical Instrum. Engs., vol. 7, pp , Sep [13] D. P. Huttenlocher, G. A. Klanderman, and W. J. Rucklidge. Comparing images using the Hausdorff distance, IEEE trans. on Pattern Analysis and Machine Intelligence, vol. 15, no. 9, pp , Sep [14] F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, and P. Suetens. Multimodality image registration by maximisation of mutual information, IEEE trans. on Medical Imaging, vol. 16, no. 2, pp , April [15] A. Roche, G. Malandain, X. Pennec, and N. Ayache. The correlation ratio as a new similarity measure for multimodal image registration, Proc. First Int. Conf. on Medical Image Computing and Computer-Assisted Intervention (MICCAI 98), vol. 1496, pp , [16] Y.H. Lau, M. Braun and B. F. Hutton. Non-rigid image registration using a median-filtered coarse-to-fine displacement field and a symmetric correlation ratio, Journal of Physics in Medicine and Biology, vol. 46, pp ,