n others; multple brghtness values n one mage may map to a sngle brghtness value n the other mage, and vce versa. In other words, the two mages are us

Transcription

1 Robust Mult-Sensor Image Algnment Mchal Iran Dept. of Appled Math and CS The Wezmann Insttute of Scence Rehovot, Israel P. Anandan Mcrosoft Corporaton One Mcrosoft Way Redmond, WA 98052, USA Abstract Ths paper presents a method for algnment of mages acqured by sensors of derent modaltes (e.g., EO and IR). The paper has two man contrbutons: () It dentes an approprate mage representaton for mult-sensor algnment,.e., a representaton whch emphaszes the common nformaton between the two mult-sensor mages, suppresses the non-common nformaton, and s adequate for coarse-to-ne processng. () It presents a new algnment technque, whch apples global estmaton to any choce of a local smlarty measure. In partcular, t s shown that when ths regstraton technque s appled to the chosen mage representaton wth a local-normalzed-correlaton smlarty measure, t provdes a new mult-sensor algnment algorthm whch s robust to outlers, and apples to a wde varety of globally complex brghtness transformatons between the two mages. Our proposed mage representaton does not rely on sparse mage features (e.g., edge, contour, or pont features). It s contnuous and does not elmnate the detaled varatons wthn local mage regons. Our method naturally extends to coarse-to-ne processng, and apples even n stuatons when the mult-sensor sgnals are globally characterzed by low statstcal correlaton. 1 Introducton In mages acqured by sensors of derent modaltes (EO, IR, radar, etc), the relatonshp between the brghtness values of correspondng pxels s usually complex and unknown: Vsual features present n one sensor mage may not appear n the other mage, and vce versa; contrast reversal may occur between the two mages n some mage regons, whle not Part of ths work was done whle the authors were at Sarno Corporaton under NASA Ames Research Center grant NAS It was contnued whle Mchal Iran was at the Wezmann Insttute under DARPA grant N through US Oce of Naval Research. 1

2 n others; multple brghtness values n one mage may map to a sngle brghtness value n the other mage, and vce versa. In other words, the two mages are usually not correlated n ther entrety,.e., they are not globally correlated (often, not even statstcally correlated). There are two fundamental questons that a mult-sensor algnment algorthm should address: () What s a good mage representaton to work wth (.e., what representaton wll brng out the common nformaton between the two mult-sensor mages, whle suppressng the non-common nformaton)? () What s an approprate smlarty measure for matchng the two mages wthn the selected representaton? Prevous work on mult-sensor mage algnment (e.g., [4, 6, 10, 7, 8, 11]) can broadly be classed nto two major classes of algorthms. These classes der n the way they address the two abovementoned questons: 1. Methods that use an nvarant mage representaton. By nvarant mage representaton we refer to a representaton that s nvarant to changes n brghtness and contrast, as well as to contrast reversal. Some examples of nvarant mage representatons are edge maps [4], orented edge vector elds [6], contour features [7], and feature ponts [8]. Such representatons am at ncreasng the vsual smlarty between of the two mages. Once ths s acheved, regstraton technques that assume smlar appearance (e.g., that are based on the brghtness constancy assumpton) can be appled. For example, the regstraton methods employed n [4, 6] are extensons of the drect gradent-based regstraton methods [2, 5]). However, n the process of creatng an nvarant mage representaton, mportant mage nformaton s usually lost. For example, n [4, 6, 7] there s a thresholdng step. Ths step usually elmnates most of the detaled varatons wthn local regons of the mages, leavng only a sparse set of hghly sgncant mage features. Moreover, the choce of threshold s very data and sensor dependent. 2

3 2. Methods that use an nvarant smlarty measure to regster the mult-sensor mages, and therefore do not requre an nvarant mage representaton. An example of such a smlarty measure s Mutual Informaton [11], whch s a measure of the statstcal correlaton between two mages. The method suggested by [11] s appled drectly to the raw mult-sensor ntensty mages, and does not requre an nvarant mage representaton. Ths method assumes, however, that the statstcal correlaton between the two mages s global, an assumpton whch s often volated (e.g., Fgure 4). Moreover, the statstcal correlaton between raw mult-sensor mages tends to decrease wth the reducton n spatal resoluton (Secton 2). Therefore, [11] n ts current form does not naturally extend to coarse-to-ne estmaton, whch s often used to handle large msalgnments. These ssues wll be referred to n Secton 2. In order to address the ssues mentoned above, we have developed an approach whch uses a locally nvarant smlarty measure whle globally constranng the local matches. In partcular, our approach to mult-sensor mage algnment does not assume global correlaton (regular or statstcal) of the mages, but only a local one. The underlyng chosen mage representaton s contnuous, and avods thresholdng and hence loss of mage detal. The representaton s nvarant to contrast reversal, provdes orentatonal senstvty, and s sutable for coarse-to-ne processng. The estmaton process has a bult-n outler rejecton mechansm, whch s crtcal to mult-sensor algnment due to the pluralty of non-common mage features across the two mages (as a matter of fact, n many stuatons there are more \outlers" than \nlers" n a mult-sensor mage par). The moton models used n ths work were 2D parametrc transformatons. The algorthm, however, can be extended to 3D moton models as well. The rest of the paper s organzed as follows: Secton 2 descrbes the chosen mage 3

4 representaton. Secton 3 descrbes the global algnment method wth a local smlarty measure. Secton 4 presents results of applyng our algorthm to IR/EO mage pars. 2 The Image Representaton The underlyng assumpton of mult-resoluton algnment s that the correspondng sgnals at all resoluton levels contan enough correlated structure to allow stable matchng. Ths assumpton s generally true when an mage par s obtaned by the same sensor, or by two derent cameras of same modalty. However, n mult-sensor mage pars (.e., mage pars taken by sensors of derent modaltes), the sgnals are correlated prmarly n hgh resoluton levels, whle correlaton between the sgnals tends to degrade substantally wth the reducton n spatal resoluton. Ths s because hgh resoluton mages capture hgh spatal frequency nformaton, whch corresponds to physcal structure of the scene that s common to the two mages. Low resoluton mages, on the other hand, depend heavly on llumnaton and on the photometrc and physcal magng propertes of the sensors (whch are characterzed by low frequency nformaton), and these are substantally derent n two mult-modalty mages. To capture the common scene detal nformaton whle suppressng the non-common llumnaton and sensor-dependent propertes, the mages are transformed nto hgh-pass energy mages (e.g., see [3]). An example of such an energy mage s a Laplacan-energy mage, whch s formed by rst hgh-pass lterng the mage wth a Laplacan lter, then squarng t. Ths facltates coarse-to-ne search based on sgnal detals. In [3] the Laplacan-energy mage s used for eectvely detectng small (hgh-resoluton) temporal changes already at low resoluton levels. Hgh-pass energy mage representatons are useful for mult-sensor algnment, because: 4

5 () The creaton of such energy mages does not nvolve any thresholdng, and therefore preserves all mage detal. Ths s n contrast to \nvarant" representatons (e.g., edge maps [4], edge vectors [6], contours [7], pont features [8]), whch elmnate most of the detaled varatons wthn local mage regons. () The mage nformaton whch s elmnated n the creaton of the hgh-pass energy mages s exactly that whch s not common to the two mult-sensor mages. In partcular: (a) the sensor-dependent low-resoluton nformaton s elmnated, and (b) contrast-reversal whch may occur between the sensors (e.g., Fg. 3) s removed by the squarng operaton. In other words, the energy mage representaton s nvarant to contrast reversal. () As mentoned n [3], a pyramd data structure of the hgh-pass energy mage projects hgh resoluton sgnal nformaton nto low resoluton levels. In our case, ths facltates coarse-to-ne algnment based on correlated scene detals, as opposed to usng pyramds of the raw mult-sensor mages (whch contan uncorrelated sensor nformaton at low spatal resolutons). However, the Laplacan, beng a rotatonally nvarant operator, does not preserve drectonal nformaton. Ths leads to potental false correspondences of patterns that are orented along derent drectons n the Laplacan energy mages. The energy-mage representaton that we use s based on drectonal-dervatve lters rather than a Laplacan lter. On top of the abovementoned advantages of hgh-pass energy mages, the drectonal-dervatve-energy also preserve drectonal nformaton, and thereby avod ths problem. Ths further enhances the robustness of the regstraton algorthm aganst the numerous outlers so common n a mult-sensor mage par. The drectonal dervatve lter s appled to the raw mage n four drectons (horzontal, vertcal, and the two dagonals). Then, each of the four generated dervatve mages s squared. (Snce the squarng operaton doubles the frequency band, the raw mage s ltered 5

6 wth a Gaussan pror to the dervatve lterng, to avod alasng eects). The algnment algorthm (Secton 3) s appled smultaneously to all 4 correspondng mult-sensor pars of drectonal-dervatve-energy mages, seekng a sngle parametrc transformaton ~p, whch smultaneously brngs all drectonal pars nto algnment (see Secton 3). Fg. 1 shows an example of the four drectonal-dervatve-energy pars constructed from a mult-sensor mage par. Fg. 2 shows the Gaussan pyramd constructed for one of the four mult-sensor pars of drectonal-dervatve-energy mages. 3 The Algnment Algorthm To algn the mult-band energy mage representaton (Secton 2), our algnment algorthm uses a local correlaton-based smlarty measure, wthout assumng global correlaton (regular or statstcal) between the mages. We have appled the algorthm wth a normalzedcorrelaton-based local smlarty measure for reasons explaned below. However, t can be smlarly appled wth a local statstcal-correlaton-based smlarty measure (e.g., based on Mutual Informaton), or any other approprate local measure. The global parametrc estmaton s appled drectly to the collecton of all local correlaton surfaces, whle avodng an ndependent local search for peaks n the ndvdual surfaces. Global algnment has the advantage of drectly estmatng the global parametrc transformaton, wthout rst commttng to any partcular matches locally. In other words, local matchng s constraned by global algnment. Such a scheme s useful n any algnment algorthm, but s partcularly crtcal n mult-sensor algnment, due to the pluralty of outlers across sensors and hence the unrelablty of local matches. Although global algnment has been used for mage regstraton, t has been based on mnmzng the ntensty derences 6

7 (a) (b) (c) (d) (e) Fgure 1: The four drectonal-dervatve-energy mage pars. Left column: EO. Rght column: IR. (a) The raw mult-sensor mage par. (b) horzontal dervatve energy, (c) vertcal dervatve energy, (d,e) energes of dagonal dervatves. 7

8 a) b) Fgure 2: The Gaussan pyramd constructed for one of the four pars of drectonaldervatve-energy mages (Fg. 1.d): (a) EO. (b) IR. between the correspondng pxels n the two mages,.e., usng the sum of squared derences (SSD) as the match measure. That s, the smlarty measure s based on the \brghtness constancy" assumpton, whch s severely volated n a mult-sensor mage par (even n the energy mages). In ths work, we have generalzed global algnment technques to use any local smlarty measure (e.g., normalzed correlaton, SSD, or any other measure) whch s sutable for the partcular algnment problem. Ths s done va global regresson appled drectly to the local smlarty-measure surfaces (e.g., correlaton surfaces), as descrbed n Secton 3.1. In partcular, we found normalzed-correlaton to be a sutable smlarty measure for mult-sensor energy-mage algnment. 8

9 Global algnment s partcularly crtcal when usng the drectonal-dervatve mages: no pror local estmaton process can produce meanngful local matches on a drectonaldervatve mage par, as these mages lack nformaton n the drecton perpendcular to the drectonal dervatve (the \aperture problem"). The smultaneous and global regstraton of all (four) drectonal pars, however, provdes full drectonal nformaton. Moton Models: When the scene can be approxmated by a planar surface, or when the baselne between the two sensors s small relatve to ther dstance from the scene, then the dsplacement eld between the two mages can be modeled n terms of a sngle 2D parametrc transformaton (see [2] for a taxonomy of moton models). We have focused our attenton on algnment usng a 2D parametrc transformaton, although our approach generalzes to 3D models as well. Speccally, we focus on parametrc transformatons whch are lnear n ther unknown parameters fp g. For such transformatons, the moton vector ~u(x; y) = (u(x; y); v(x; y)) T can be expressed as: ~u(x; y; ~p) = X(x; y) ~p; (1) where X(x; y) s a matrx whch depends only on the pxel coordnates (x; y), and ~p = (p 1 ; :::; p n ) T s the unknown parameter vector. For example, for an ane transformaton: " u(x; y; ~p) v(x; y; ~p) # = " p1 + p 2 x + p 3 y p 4 + p 5 x + p 6 y # ; (2) therefore, n ths case: ~p = (p 1 ; p 2 ; p 3 ; p 4 ; p 5 ; p 6 ) T and X = " 1 x y x y # ; and for a quadratc transformaton: " u(x; y; ~p) v(x; y; ~p) # = " p1 + p 2 x + p 3 y + p 7 x 2 + p 8 xy p 4 + p 5 x + p 6 y + p 7 xy + p 8 x 2 9 # ;

10 therefore: ~p = (p 1 ; p 2 ; p 3 ; p 4 ; p 5 ; p 6 ; p 7 ; p 8 ) T and X = " 1 x y x 2 xy x y xy y 2 # : The Normalzed-Correlaton as a Local Smlarty Measure: Normalzed-correlaton of two sgnals s nvarant to local changes n mean and contrast. In other words, when the two sgnals are lnearly related, ther normalzed-correlaton s 1. When the lnear relatonshp does not hold, but the two sgnals contan smlar spatal varatons (as measured n the form of local uctuatons), the normalzed-correlaton wll stll gve a value close to unty. In general, however, the global relatonshp between two mult-sensor mages s complex, and therefore the two sgnals are not globally correlated (even after computng the energy mages). Statstcal correlaton s a better global measure than regular or normalzed correlaton, but may stll not be a strong enough global smlarty measure, because multple brghtness values n one mage may map to a sngle brghtness value n the other mage, and vce versa. Locally, however, wthn small mage patches whch contan correspondng mage features, statstcal correlaton s hgh. Normalzed-correlaton s a lnear approxmaton of the statstcal correlaton of two sgnals n a small wndow, and s cheaper to compute. The energy mages that we compute tend to hghlght the local varatons that correspond to local structure n the scene. These mages are nvarant to contrast reversal, but vary n mean and contrast. When the relatonshp between correspondng patches devates from lnear, the normalzed-correlaton (appled over local wndows) s less than 1, but s stll hgh for the correct dsplacement. For other dsplacements the normalzed-correlaton wll be low, especally for hghly textured mage patches. Therefore, the local normalzed-correlaton surface of such patches wll be concave wth a promnent peak at the correct dsplacement. 10

11 For correspondng mage patches that contan mutually exclusve mage features (.e., mage features whch appear n only one of the 2 mult-sensor mages { a thng whch occurs frequently), the local correlaton surface wll not have a concave shape wth a promnent peak. Therefore, the structure of the local-normalzed-correlaton surfaces provdes useful nformaton for algnment. The nformaton from all of these local structures, however, should be smultaneously used to determne the global algnment parameters. Ths s essental to avod the numerous potental false matches n lmted local analyss. Ths s acheved va global regresson appled drectly to the collecton of local normalzed-correlaton surfaces, as descrbed n Secton Global Algnment wth Local Correlaton Gven two mages, f and g, and ther drectonal-dervatve energy mages, ff g 4 =1 and fg g 4 =1, nd the parametrc transformaton ~p whch maxmzes the sum of all local normalzedcorrelaton values. Let S (x;y) (u; v) denote a correlaton surface correspondng to a pxel (x; y) n f. For any shft (u; v) of g relatve to f, S (x;y) s dened as: S (x;y) (u; v) def = f (x; y) N g (x + u; y + v) where N denotes normalzed correlaton computed over a small wndow. Let ~u = (u(x; y; ~p); v(x; y; ~p)) denote the moton eld descrbed by the parametrc transformaton ~p. Then the parametrc regstraton problem can be stated as follows: Fnd the parametrc transformaton ~p that maxmzes the global smlarty-measure M(~p): M(~p) = x;y S (x;y) (u(x; y; ~p); v(x; y; ~p)) = x;y S (x;y) (~u(x; y; ~p)): (3) 11

12 To solve for ~p that maxmzes M(~p), we use Newton's method [9], whch teratvely ts quadratc approxmatons to the objectve functon, and renes the peak locaton that maxmzes these quadratc surfaces. In order to provde the context for our use of Newton's method for the maxmzaton problem at hand, we rst brey outlne the steps of ths method. Gven the current estmate of the moton parameters ~p 0, let M(~p) = M( ~p 0 ) + (r ~p M( ~p 0 )) T ~ p + ~ p T HM ( ~p 0 ) ~ p (4) denote the quadratc approxmaton of M(~p) around ~p 0, where, ~ p = ~p? ~p 0 s the unknown renement step of ~p 0 that we want to solve for, r ~p M denotes the gradent of M, and H M denotes the Hessan of M (.e., the matrx of second dervatves), both computed around ~p 0. Accordng to Newton's method [9], the renement ~ p computed based on ths approxmaton s: ~ p =?(HM ( ~p 0 ))?1 r ~p M( ~p 0 ) (5) To apply the Newton's renement step to our problem, we derved the expressons for r ~p M and H M n terms of the measurable mage quanttes,.e., the collecton of correlaton surfaces fs (x;y) g: Usng the chan-rule of derentaton, we obtan r ~p M(~p) = x;y; r ~p S (~u) = x;y; (X T r ~u S (~u)) H M (~p) = x;y; (X T H S (~u) X) (6) where X s the matrx dened n Eq. (1), r ~u S s the gradent of S (x;y) (~u), and H S s the Hessan of S (x;y) (~u). In other words, the quadratc approxmaton of M around ~p 0 s obtaned by combnng the quadratc approxmatons of each of the local correlaton surfaces fs (x;y) g x;y; around the 12

13 local dsplacement vector ~u 0 = ~u(x; y; ~p 0 ), whch s nduced at pxel (x; y) by the parametrc transformaton ~p 0 (estmated at the prevous teraton). Substtutng Eqs. (6) nto Eq. (5) provdes an expresson for the renement step ~ p n terms of the correlaton surfaces fs (x;y) g: ~ p =?(x;y; X T H S ( ~u 0 )X)?1 ( x;y; X T r ~u S ( ~u 0 )) (7) Note that these steps do not make any assumptons about the local correlaton surface, except that t s twce derentable. Thus, any local smlarty-measure can be substtuted for correlaton, and our method wll stll apply. The steps of the algorthm: To account for large msalgnments between pars of mages, we perform mult-resoluton coarse-to-ne estmaton, e.g., as n [2]. A Laplacan (or a Gaussan) pyramd s constructed for each of the energy mages. Let f l and g l ( = 1; 2; 3; 4) denote the drectonal-dervatve energy mages at resoluton level l n the pyramds of f and g, respectvely. Startng at the coarsest resoluton level wth ~p 0 ntally set to 0, the followng steps are performed at each resoluton level: 1. For each pxel (x; y) at f l ( = 1; 2; 3; 4), compute a local normalzed-correlaton surface around the dsplacement ~u 0 (.e., around the dsplacement estmated at the prevous teraton). In practce, the correlaton surface s estmated only for a small number of dsplacements ~u of g l wthn a radus d around ~u 0,.e.: S l (x;y) (~u) = f l(x; y) N g l(x + u; y + v) ; 8~u = (u; v) s:t: jj~u? ~u 0 jj d where the radus d s determned by the sze of the masks used for dscretely estmatng the 13

14 (x;y) rst and second order dervatves of S l (~u) at ~u 0. In our current mplementaton we used Beaudet's masks [1] to estmate the rst and second order dervatves of the surfaces. We have expermented both wth 3 3 masks (.e., d = 1) and wth 5 5 masks (.e., d = 2). 2. Perform the regresson step of Eq. (7) to compute the parametrc renement ~ p. 3. Update ~p 0 : ~p 0 := ~p 0 + ~ p, and go back to step 1. After repeatng the above process for a few teratons (typcally 4), the parameters ~p are propagated to the next resoluton level, and the process s repeated at that resoluton level. The process s stopped when the teratve process at the hghest resoluton level s completed. In practce, to mprove performance, we add an mage warpng step before each teraton (as n [2]). The nspecton mages fg g are warped towards the reference mages ff g accordng to the current estmated parametrc transformaton ~p 0. After warpng the mages, ~p 0 s set to 0, and ~ p s estmated between the pars of references and warped nspecton mages. Warpng compensates for the spatal dstortons between the pars of mages (e.g., scale derence, rotatons, etc), and hence mproves the qualty of the correlaton. Outler rejecton: To further condton and robustfy the regresson step of Eq. (7), only pxels (x; y) for whch the quadratc approxmaton of S (x;y) (~u) around ~u 0 s concave are used n the regresson process. Other pxels are gnored. Snce correspondng multsensor mage patches whch have mutually exclusve mage features wll not tend to have a concaved-shaped local correlaton surfaces, they wll be elmnated from the regresson at ths pont. Moreover, the contrbuton of each pxel to the regresson step s weghted 14

15 by the determnant of t Hessan. Ths bult-n outler rejecton mechansm provdes the algorthm wth a strong lockng property onto a domnant parametrc moton, even n the presence of ndependent motons, nose, and exclusve features that appear n only one of the sensor-mages (but not n the other). 4 Examples The algnment algorthm descrbed n Secton 3 was mplemented and appled wth an ane parametrc model (Eq. 2) to pars of mult-sensor mages. Fg. 3 shows result of algnment of two mult-sensor mages (vsble and IR) obtaned by sensors mounted on an arcraft approachng landng. Note the sgncant derence n scale between the two mages (due to sgncantly derent nternal sensor parameters). Also note that contrast reversal occurs n some parts of the mages (e.g., runway markngs), whle not n others (e.g., runway boundares). The algorthms has been appled successfully even n very challengng stuatons, such as the one shown n Fg 4. Note the sgncant derence n mage content between the two sensor-mages. Apart from havng sgncantly derent appearance, there are many noncommon features (.e., outlers) n the mult-sensor mage par, whch can theoretcally lead to false matches. These are overcome by the bult-n outler mechansm of our algorthm (see Secton 3). 15

16 a) b) c) d) Fgure 3: Mult-sensor Algnment. (a) EO mage. (b) IR mage. (c) Composte dsplay of the two mult-sensor mages before algnment. Horzontal strps from the two mages are splced together. Note the sgncant msalgnments between the mages (e.g., the runway markngs and the borders of the runway). (d) Composte (splced) dsplay of the two mult-sensor mages after algnment. Note that all structures n the scene are algned. 16

17 a) b) c) d) Fgure 4: Mult-sensor Algnment. (a) EO mage. (b) IR mage. (c) Composte (splced) dsplay before algnment. (d) Composte (splced) dsplay after algnment. Note n partcular the perfect algnment of the water-tank at the bottom left of the mages, the buldng wth the arched-doorway at the rght, and the roads at the top left of the mages. 17

18 Acknowledgement The authors would lke to thank Peter Burt for hs nspraton and gudance durng ths work. References [1] Paul R. Beaudet. Rotatonally nvarant mage operators. In Internatonal Conference on Pattern Recognton, pages 579{583, [2] J.R. Bergen, P. Anandan, K.J. Hanna, and R. Hngoran. Herarchcal model-based moton estmaton. In European Conference on Computer Vson, pages 237{252, Santa Margarta Lgure, May [3] P.J. Burt. Smart sensng wth a pyramd vson machne. Proceedngs of the IEEE, 76:1006{1015, [4] K. J. Dana and P. Anandan. Regstraton of vsble and nfrared mages. In Proc. SPIE Conf. on Arch., Hardware and FLIR n Auto. Targ. Rec., pages 1{12, [5] M. Iran, B. Rousso, and S. Peleg. Computng occludng and transparent motons. Internatonal Journal of Computer Vson, 12(1):5{16, January [6] R. Kumar, K. Dana, and P. Anandan. Frameless regsteraton of mr and ct 3d volumetrc data sets. In Proc. of the Workshop on Applcatons of Computer Vson II, Sarasota, Fl., [7] H. L, B.S. Manjunath, and S.K. Mtra. A contour-based approach to multsensor mage regstraton. IEEE Trans. on Image Processng, pages 320{334,

19 [8] H. L and Y.T. Zhou. Automatc eo/r sensor mage regstraton. In IEEE Int. Conf. on Image Proc., volume B, pages 161{164, [9] Davd G. Luenberger. Lnear and Nonlnear Programmng. Addson Wesley, [10] P. A. van den Elsen and M.A. Vergever. Marker-guded multmodalty matchng of the bran. European Radology, 4(1):45{51, [11] P. Vola and W. Wells III. Algnment by maxmzaton of mutual nformaton. In Internatonal Conference on Computer Vson, pages 16{23, Cambrdge, MA, June