MULTI-TEMPORAL AND MULTI-SENSOR IMAGE MATCHING BASED ON LOCAL FREQUENCY INFORMATION

Transcription

1 Intenational Achives of the Photogammety Remote Sensing and Spatial Infomation Sciences Volume XXXIX-B XXII ISPRS Congess 25 August 01 Septembe 2012 Melboune Austalia MULTI-TEMPORAL AND MULTI-SENSOR IMAGE MATCHING BASED ON LOCAL FREQUENCY INFORMATION Xiaochun Liu a * Qifeng Yu a Xiaohu Zhang a Yang Shang a Xianwei Zhu a Zhihui Lei a a Aeonautical and Astonautical Science and TechnologyNational Univesity of Defense TechnologyChangsha HunanChina liuxiaochun @gmail.com; yuqifeng@vip.sina.com; zhangxiaohu@vip.163.com; jmgc108@vip163.com; jmgc108@vip.163.com; jmgc108@vip.163.com Commission III KEY WORDS: Image Matching; Local Aveage Phase; Local Weighted Amplitude; Local Best-Matching Point; Similaity Measuement; Local Fequency Infomation; ABSTRACT: Image Matching is often one of the fist tasks in many Photogammety and Remote Sensing applications. This pape pesents an efficient appoach to automated multi-tempoal and multi-senso image matching based on local fequency infomation. Two new independent image epesentations Local Aveage Phase (LAP and Local Weighted Amplitude (LWA ae pesented to emphasize the common scene infomation while suppessing the non-common illumination and senso-dependent infomation. In ode to get the two epesentations local fequency infomation is fistly obtained fom Log-Gabo wavelet tansfomation which is simila to that of the human visual system; then the outputs of odd and even symmetic filtes ae used to constuct the LAP and LWA. The LAP and LWA emphasize on the phase and amplitude infomation espectively. As these two epesentations ae both deivative-fee and theshold-fee they ae obust to noise and can keep as much of the image details as possible. A new Compositional Similaity Measue (CSM is also pesented to combine the LAP and LWA with the same weight fo measuing the similaity of multi-tempoal and multi-senso images. The CSM can make the LAP and LWA compensate fo each othe and can make full use of the amplitude and phase of local fequency infomation. In many image matching applications the template is usually selected without consideation of its matching obustness and accuacy. In ode to ovecome this poblem a local best matching point detection is pesented to detect the best matching template. In the detection method we employ self-similaity analysis to identify the template with the highest matching obustness and accuacy. Expeimental esults using some eal images and simulation images demonstate that the pesented appoach is effective fo matching image pais with significant scene and illumination changes and that it has advantages ove othe state-of-the-at appoaches which include: the Local Fequency Response Vectos (LFRV Phase Conguence (PC and Fou Diectional-Deivative-Enegy Image (FDDEI especially when thee is a low signal-to-noise atio (SNR. As few assumptions ae made ou poposed method can foeseeably be used in a wide vaiety of image-matching applications. 1. INTRODUCTION Multi-tempoal and multi-senso image matching is an inevitable poblem aising in a vaiety of applications such as multisouce data fusion change analysis image mosaic vision navigation and object ecognition. Because the efeence image and the seaching image diffe in elation to time o the type of senso the elationship between the intensity values of the coesponding pixels is usually complex and unknown. Fo instance the contasts of the images may diffe o the scenes may change damatically ove time. In othe wods the two images ae not globally coelated. Theefoe multi-tempoal and multi-senso image matching pesents a challenging poblem. Note that we assume that the matching image pais have aleady been egisteed hence geometic distotion is not discussed in this pape. The cuent automatic matching techniques geneally fall into two categoies: featue-based methods and aea-based methods. Featue-based methods which ae by fa the most popula utilize extacted featues with the most widely used featues including egions lines o cuves and points [1-3]. If featues can be extacted obustly and the featue coespondences ae eliably established then the featue-based methods can be successfully applied [4 5]. Howeve fo multi-tempoal and multi-senso images it is vey difficult to extact common featues that exist in both images because of hash contast changes diffeent sensos and scene changes. In addition because the templates suounding each featue point ae not big enough the coect ate of featue coespondences is quite low. As Figue 1 shows the efeence image is captued by an infaed camea wheeas the seaching image is captued by a visible light camea. We use the most commonly used featuebased method SIFT to detect and then match the featue points. Fom Figue 1 we can easily see that few common featues ae detected and only fou pais of points ae coectly matched which is fa fom meeting the equiements of the application. In contast with the featue-based methods aea-based methods usually take advantage of much lage template which means they ae able to toleate moe noise and scene changes. The aea-based methods commonly involve image epesentation and similaity measuement [6 7]. Some common similaity measuements used in the existing matching algoithms ae: (i * Coesponding autho. This is useful to know fo communication with the appopiate peson in cases with moe than one autho. ` 485

2 Intenational Achives of the Photogammety Remote Sensing and Spatial Infomation Sciences Volume XXXIX-B XXII ISPRS Congess 25 August 01 Septembe 2012 Melboune Austalia nomalized-coelation (ii sum of squaed bightness diffeences and (iii mutual infomation [8]. The image epesentations commonly used fo aea-based methods ae some gadient opeatos which include: Canny Sobel Pewitt Kisch Laplacian of Gaussian and Susan etc.. Using these spatial deivative opeatos one can emphasize the edges cones and blobs that epesent some illumination invaiant component of images. Howeve these gadient epesentations ae usually sensitive to noise. Theefoe the gadient epesentations ae not able to handle some noisy multi-senso and multi-tempoal images efficiently. Repesentation based on local fequency infomation was intoduced in [9]. Woking in the fequency domain local phase and amplitude infomation ove many diffeent scales and oientations wee used to constuct a dimensionless measue of similaity that has high localization. Howeve due to the unweighted local fequency the algoithm was unable to clealy emphasize common infomation such as edges and cones. Theefoe the pefomance of ou evaluation expeiment is unsatisfactoy. As phase conguence is condition-independent and invaiant to illumination changes it was employed to epesent images in [10]. Howeve when calculating phase conguence with the denominato epesenting the sum of the amplitude in the Log-Gabo expansion spaces a division opeato is inevitably involved. As the value of the denominato is usually small in the textue-less egions the method pesented in [10] is quite sensitive to noise in the textue-less egions as shown in Figue 2. This pape addesses these difficulties by using local fequency infomation obtained fom Log-Gabo wavelets ove many scales and oientations. A compositional similaity measuement and a local best matching point detection ae also pesented to make the pesented image matching appoach moe obust and accuacy. Figue 1. The esults of the SIFT featue detection and matching. Fom top to bottom: the efeence image and seaching image; the esults of the featue detection; the esults of the featue matching afte outlie emoval. DC component in the even-symmetic filte. (A zeo DC value cannot be maintained in Gabo functions fo bandwidths ove one octave [11]. On the linea fequency scale the Log-Gabo function has a tansfe function of the fom: g( ω = e 2 (log( ω/ ω0 2 2(log( κ/ ω0 whee ω is the filte s cente fequency. To obtain constantshape atio filtes the tem κ / ω must also be held constant 0 fo e o vaying ω 0. Let I denote the signal M and s M denote the s even-symmetic (cosine and odd-symmetic (sine wavelets espectively and e o denote the even-symmetic and s s odd-symmetic filte outputs at location x.we can think of the esponses of each quadatue pai of filtes as foming a esponse vecto The amplitude given by e o s s s s 0 (1 [ e ( x o (] x = [( Ix M Ix ( M ] (2 As and phaseφ at a given wavelet scale is s 2 2 As = e s + os φs = atan2 e s os ( At each point x in a signal we will have an aay of these esponse vectos with one vecto fo each scale and oientation of the filte. These esponse vectos fom the basis of ou localized epesentation of the signal. An estimate of F(x can be fomed by summing the even filte convolutions. Similaly H(x can be estimated fom the odd filte convolutions. F = es (4 s H = os s The aveage phase is given by (3 2. LOCAL AVERAGE PHASE AND LOCAL WEIGHTED AMPLITUDE ( atan2 F/ H 180 if : atan2 ( F/ H 0 π φ = atan2 ( F/ H + π 180 if : atan2 ( F/ H < 0 π (5 2.1 Local Aveage Phase In this woking the wavelet tansfomation is used to obtain the fequency infomation which is local to a point in the signals. To peseve phase infomation the nonothogonal wavelets in the symmetic/anti-symmetic quadatue pais ae adopted. Rathe than using Gabo filtes we pefe to use Log-Gabo functions because Log-Gabo filtes allow abitaily lage bandwidth filtes to be constucted while maintaining a zeo whee φ anging fom 0 to 180 can be seen as the phase of the sum of the esponse vectos ove many scales and oientations. The local aveage phase which emphasizes the phase infomation of local fequency is used as one of the image epesentations fo multi-tempoal and multi-senso images. Anothe epesentation Local Weighted Amplitude is designed fo extacting the amplitude infomation of local fequency. Appaently it is independent of LAP. The calculation of the LWA is simila to that of phase conguence except the division 486

3 Intenational Achives of the Photogammety Remote Sensing and Spatial Infomation Sciences Volume XXXIX-B XXII ISPRS Congess 25 August 01 Septembe 2012 Melboune Austalia opeato is not intoduced which makes ou new image epesentation moe obust to noise even in the textue-less image egion. 2.2 Local Weighted Amplitude In this wok we extend the phase conguence to LWA which is moe suitable fo multi-tempoal and multi-senso image epesentation. The equation of LWA ( A w is expessed as the summation of oientations and scales s: A = W A φ T w s s (6 whee c is the cut-off value of the filte esponse spead below which the fequency amplitude values become penalized andγ is a gain facto that contols the shapness of the cut-off. Eq.6 Eq.10 give us the quantities needed to calculate this vesion of the LWA without any division opeato. The LAP and LWA ae both used to extact the common components of multi-tempoal and multi-senso images such as edges contous and blobs. Note that the two image epesentations do not involve any thesholding and theefoe peseve all the image details. This is in contast to commonly used epesentations (e.g. edge maps contous point featues which eliminate most of the detailed vaiations within the local image egions. whee denotes that the enclosed quantity is not pemitted to be negative; A epesents the amplitude at scale s and s oientation ; andt compensates fo the influence of noise and is estimated empiically. φ is a sensitive phase deviation of the thoientation and is defined as: ( ( φ = cos φ φ sin φ φ s s (7 The calculation of this new LWA A ( x can be pefomed using dot and coss poducts between the filte output esponse vectos to calculate the cosine and sine of ( φ s φ. The unit vecto epesenting the diection of the weighted mean phase angle φ is given by ( φe φo = s w es os s 2 2 es + os s s Using dot and coss poducts one can obtain: ( ( ( A φ = A cos φ φ sin φ φ s s s s = e φ + o φ e φ o φ s e s o s o s e Clealy a point of fequency amplitude is only significant if it occus ove a wide ange of fequencies. Thus as a measue of featue significance fequency amplitude should be weighted by some measue of the spead of the fequencies pesent. A phase significance weighting function can then be constucted by applying a sigmoid function to the filte esponse spead value: (8 (9 1 W = (10 ( c s 1 + e γ Figue 2. The esults of PC and LWA. Fom left to ight: the aw image; the PC map; and the LWA map. Note that the LWA map is much moe obust and stable than the PC map especially in textue-less image egions such as the sky sea and gound. 3. THE COMPOSITIONAL SIMILARITY MEASUREMENT As discussed above The LAP and LWA ae independent of each othe. In ode to combine the infomation of the LAP and LWA we pesent a new similaity measuement: CSM which is able to take advantage of moe infomation than those commonly used similaity measues [9 10] and is theefoe able to impove the obustness and applicability of image matching. The LWA is designed to emphasize the common amplitude components fo multi-tempoal and multi-senso images and has a stonge anti-noise capability than the commonly used Phase Conguence. Howeve fom the theoetic analysis and expeimental esults we know that the LWA is an image contast-dependent vaiation. To ovecome this poblem we employ the zeo-mean nomalized coss-coelation (ZNCC as the similaity measue function which is a contast invaiant vaiation. If we define f and A g as the coesponding LWA A map pai f and g A A as the mean value within the template window W aound pixel ( x y in f and A ( uv in g A espectively and S as the seaching window whee ( i j W( uv S the ZNCC can be expessed as follows: 487

4 Intenational Achives of the Photogammety Remote Sensing and Spatial Infomation Sciences Volume XXXIX-B XXII ISPRS Congess 25 August 01 Septembe 2012 Melboune Austalia ZNCC( uv = ( fa( x+ iy + j fa ( ga( u+ iv + j ga i j ( fa( x+ iy + j fa ( ga( u+ iv + j ga i j i j 2 2 (11 S = 1 Max( ZNCC N (14 Because of the nomalization ZNCC is invaiant to image contast linea changes which can be used to compensate the disadvantage of the LWA. As the LAP epesents the phase of local fequency vectos it is an image contast invaiant vaiation. Theefoe fo efficiency we pesent an Extended Mean Absolute Diffeence (EMAD as the similaity measue athe than ZNCC. If we define f p and g p as the LAP pai the definition of EMAD is given by: (12 EMAD( uv = 255 f ( x+ iy + j g ( u+ iv + j If we define MaxEMAD ( ( P P MaxZNCC i j as the maximum value of ZNCC matix and as the maximum value of EMAD matix espectively then the newly pesented CSM can be expessed as follows: ZNCC( uv + 1 EMAD( uv (13 CSM( uv = + MaxZNCC + 1+ ε MaxEMAD + ε A vey small positive constant ε is added to the denominato in case of a small Max and/o ZNCC Max. Fom Equ.13 we can EMAD see the value ange of EMAD and ZNCC ae both nomalized theefoe they have the same value ange. The maximum value of the ZNCC component ZNCC( uv + 1 is 1 and the same MaxZNCC + 1+ ε applies to the EMAD component EMAD( uv. Theefoe MaxEMAD + ε the CSM is able to combine the LAP and LWA infomation with equal weight and make full use of them. 4. LOCAL BEST MATCHING POINT DETECTION In this wok the goal of local best matching point detection is to detemine the template which has the highest matching accuacy within a cetain image egion. The cente of the template is named as Local Best Matching Point. In ode to find this template we must claify what the featue of the template is. If the template centeed on a point is shifted the textue within the template obviously changes and then we can know this template is unique and is also suitable fo image matching. Theefoe the local best matching point can be detected using the self-similaity measuement. We fist need to evaluate the suitability measuement of each point suounding the taget point and then choose the point with the highest suitability measuement as the local best featue point. The detailed algoithm poceeds as follows: (1 Pick a point fom the egion centeed on the taget point and then calculate the suitability of the selected point. The definition of suitability can be expessed as follows: Fist as shown in Figue 3 pick anothe eight points which ae centeed on the selected point and equally spaced on a cicle of adius ; Second if we define the template centeed on the selected point as the cente template and the template centeed on the othe eight points as the neighboing templates we can choose the cente template as the efeence template and calculate its self-similaity measuement with neighboing templates. In this wok we use ZNCC as the self-similaity measuement. If we define ZNCC N as the self-similaity measuement of the neighboing template then the suitability measuement of the template S can be defined as: whee Max( ZNCCN of ZNCC N. is the maximum self-similaity value (2 Successively pick a point fom the egion which ae centeed on the taget point with a cicle of adius R as shown in Figue 3. Simila to step 1 get the suitability measuement fo all these selected points. (3 Find the point with the highest suitability measuement and identify this point as the local best matching point. (4 Conduct the image matching using the template centeed on the local best matching point. Afte matching based on the geometic tansfomation between the efeence image and the seaching image calculate the coesponding point of the taget point. Taget g g Neighboing Figue 3. Local Best Matching Point Detection 5. EXPERIMENTS 5.1 Expeiments Using Real Images R We evaluate the pefomance of the poposed method using some eal images which include: a pai of infaed and visible images and a pai of SAR and visible images. We compae the matching esults obtained fom the poposed algoithm with those fom thee existing state-of-the-at methods based on Local Fequency Response Vectos (LFRV [9] Phase Conguence (PC [10] and Fou Diectional-Deivative-Enegy Image (FDDEI [12]. As shown in Figue 4 many taget points ae fist selected fom the efeence image (left and the inteval of the taget points is 20 pixels. The fou diffeent image matching appoaches then conduct on the seaching images (ight to seach the coesponding points. The size of the template is 101(pixel 101(pixel and the size of the seaching egion is 201(pixel 201(pixel. If the distance fom a matching esult to its coesponding tuth-value is less than 1.5 pixels we identify this matching esult as coect. The Coect Rate obtained fom fou diffeent methods ae shown in Figue 5. The expeiments using eal images show that ou new method is effective fo matching multi-senso and multi-tempoal images which cannot be effectively handled by the taditional methods. Fom Figue 5 we can see the aveage accuacy ate of ou new method is much highe than othe methods. Moeove when matching the SAR and Visible images pai the pefomances of the thee taditional methods educe damatically. Howeve ou new method is still able to obustly handle the image pai. g 488

5 Intenational Achives of the Photogammety Remote Sensing and Spatial Infomation Sciences Volume XXXIX-B XXII ISPRS Congess 25 August 01 Septembe 2012 Melboune Austalia Whee M N is defined as the height and width of image espectively v (i j is the gay value of pixel without noise and u (i j is the gay value of pixel with noise. If we define the value of noise is n (i j then we can get the equation: u (i j = v (i j + n (i j Figue 4. The aw image pai of SAR & Visible and Infaed& Visible light (16 Figue 6. The images with diffeent SNRs. The values of SNRs ae and fom left to ight. The images with diffeent SNRs ae shown in Figue 6. The expeiments using simulation images ae simila to expeiments using eal images. We choose the image without Gaussian noise as the efeence image and the image with Gaussian noise is used as the seaching images. Afte Image Matching expeiments we can get many diffeent coect ates coesponding to diffeent SNRs as shown in Figue7. Figue 5. Coect Rate obtained fom fou diffeent methods 5.2 Expeiments Using Simulation Images In ode to evaluate the pefomance of ou pesented method compehensively we also do many expeiments using simulation images. The simulation images ae made as follows: add Gaussian white noise to the aw image. The Gaussian noise is geneated by imnoise function of Matlab. The mean of Gaussian is given a same value of 0 fo all simulation images and the values of the vaiance ae ascending fom 0.1 to 3.5. Without loss of geneality SNR is employed to descibe the degee of noise. The definition of SNR goes as follows: M N 2 ( v (i j i =1 j =1 SNR = 10 log10 M N 2 u ( i j v ( i j ( i =1 j =1 Figue 7. Coect Rate obtained fom fou diffeent methods with diffeent SNRs Fom the expeiments using the simulation images as shown in Figue 7 we can easily know that ou epesented method can handle the Gaussian noise obustly. As SNR deceases the coect ates of the thee taditional methods decease damatically. Howeve the coect ate of ou method deceases vey slowly. The poposed method has a significant advantage in the case of low SNR. (15 489

6 Intenational Achives of the Photogammety Remote Sensing and Spatial Infomation Sciences Volume XXXIX-B XXII ISPRS Congess 25 August 01 Septembe 2012 Melboune Austalia 6. CONCLUSION This pape pesents a new matching appoach fo multitempoal and multi-senso images. We popose the LAP and LWA to epesent the image. The new epesentations which ae based on Log-Gabo wavelet tansfomation ae deivativefee and theshold-fee. Theefoe the new epesentations ae obust to noise and can extact the common components of image pais without loss of image detail. The CSM is then pesented to combine the infomation fom the LAP and LWA (independent of each othe with the same weight. Using this compositional scheme we ae able to make full use of the LAP and LWA to impove the obustness and accuacy of the image matching. A local best matching point detection method based on self-similaity analysis is pesented to choose the template with the most distinct featue in the egion centeed on the taget point. Compaed with the taditional matching methods based on local fequency esponse vectos phase conguence and fou diectional-deivative-enegy images ou method has significant advantage. Fom the expeiments using eal images and simulation images we have demonstated that the pesented method can obtain moe obust and accuacy matching esults even in the case of vey low SNR and fo the vey noisy SAR and visible images. Because few assumptions ae made ou poposed method can foeseeably be used in a wide vaiety of image-matching applications. [10]Liu Z. and Laganiee R Phase conguence measuement fo image similaity assessment In: Patten Recognition Lettes 28 (1 pp [11] Kovesi P.D Phase conguency: A low-level image invaiant. In: Psychological Reseach 64 pp [12] Iani M. and Anadan P Robust multi-senso image alignment In: Poceedings of the 6th Intenational Confeence on Compute Vision Bombay India pp Refeences: [1] Lindebeg T Featue detection with automatic scale selection. In: Intenational Jounal of Compute Vision 30(2 pp [2] D. G. Lowe Distinctive image featues fom scaleinvaiant key points. In: Intenational Jounal of Compute Vision. 60 pp [3] Hais C. and Stephens M A combined cone and edge detecto. In: Poceedings of the Alvey Vision Confeence pp [4]Ke Y. and Sukthanka R PCA-SIFT: A moe distinctive epesentation fo local image desciptos. In: CVPR (2 pp [5] Lowe D.G Object ecognition fom local scaleinvaiant featues. In: Poceedings of Intenational Confeence on Compute Vision pp [6] Fenàndez X Template matching of binay tagets in gay scale images: a nonpaametic appoach. In: Patten Recognition 30(7 pp [7] Khosavi M and Schafe R.W Template matching based on a gayscale hit-o-miss tansfom. In: IEEE Tansactions on Image Pocessing 5(6 pp [8] Viola P. and Wells W. III Alignment by maximization of mutual infomation. In: Poceedings of the Intenational Confeence on Compute Vision pp [9]Kovesi P Image coelation fom local fequency infomation. In: Poceedings of the Austalian Patten Recognition Society Confeence Bisbane pp