Aligning Non-Overlapping Sequences,

Transcription

1 Internatonal Journal of Computer Vson 48(1), 39 51, 2002 c 2002 Kluwer Academc Publshers. Manufactured n The Netherlands. Algnng Non-Overlappng Sequences, YARON CASPI AND MICHAL IRANI Department of Computer Scence and Appled Math, The Wezmann Insttute of Scence, Rehovot, Israel casp@wsdom.wezmann.ac.l ran@wsdom.wezmann.ac.l Abstract. Ths paper shows how two mage sequences that have no spatal overlap between ther felds of vew can be algned both n tme and n space. Such algnment s possble when the two cameras are attached closely together and are moved jontly n space. The common moton nduces smlar changes over tme wthn the two sequences. Ths correlated temporal behavor, s used to recover the spatal and temporal transformatons between the two sequences. The requrement of consstent appearance n standard mage algnment technques s therefore replaced by consstent temporal behavor, whch s often easer to satsfy. Ths approach to algnment can be used not only for algnng non-overlappng sequences, but also for handlng other cases that are nherently dffcult for standard mage algnment technques. We demonstrate applcatons of ths approach to three real-world problems: () algnment of non-overlappng sequences for generatng wde-screen moves, () algnment of mages (sequences) obtaned at sgnfcantly dfferent zooms, for survellance applcatons, and, () mult-sensor mage algnment for mult-sensor fuson. Keywords: spato-temporal algnment, temporal synchronzaton, mult-sensor algnment, algnment for wdescreen moves, algnment across dfferent zooms 1. Introducton The problem of mage algnment (or regstraton) has been extensvely researched, and successful approaches have been developed for solvng ths problem. Some of these approaches are based on matchng extracted local mage features, other approaches are based on drectly matchng mage ntenstes. A revew of some of these methods can be found n Torr and Zsserman (1999) and Iran and Anandan(1999). However, all these approaches share one basc assumpton: that there s suffcent overlap between the two mages to allow extracton of common mage propertes, namely, that there s suffcent smlarty between the two mages ( Smlarty of mages s used here n the broadest sense. It could range from A shorter verson of ths paper appeared n ICCV 2001 (Casp and Iran, 2001). Ths work was supported by the Moross Laboratory for Vson and Motor Control. gray-level smlarty, to feature smlarty, to smlarty of frequences, and all the way to statstcal smlarty such as mutual nformaton (Vola and Wells III, 1995). In ths paper the followng queston s addressed: Can two mages be algned when there s very lttle smlarty between them, or even more extremely, when there s no spatal overlap at all between the two mages? When dealng wth ndvdual mages, the answer tends to be No. However, ths s not the case when dealng wth mage sequences. An mage sequence contans much more nformaton than any ndvdual frame does. In partcular, temporal changes (such as dynamc changes n the scene, or the nduced mage moton) are encoded between vdeo frames, but do not appear n any ndvdual frame. Such nformaton can form a powerful cue for algnment of two (or more) sequences. Casp and Iran (2000) and Sten (1998) have llustrated an applcablty of such an approach for algnng two sequences based on common dynamc scene nformaton. However, they assumed that the same temporal changes n the scene (e.g., movng

2 40 Casp and Iran objects) are vsble to both vdeo cameras, leadng to the requrement that there must be sgnfcant overlap n the FOVs (felds-of-vew) of the two cameras. In ths paper we show that when two cameras are attached closely to each other (so that ther centers of projectons are very close), and move jontly n space, then the nduced frame-to-frame transformatons wthn each sequence have correlated behavor across the two sequences. Ths s true even when the sequences have no spatal overlap. Ths correlated temporal behavor s used to recover both the spatal and temporal transformatons between the two sequences. Unlke carefully calbrated stereo-rgs (Slama, 1980), our approach does not requre any pror nternal or external camera calbraton, nor any sophstcated hardware. Our approach bears resemblance to the approaches suggested by Demrdjan et al. (2000) Horaud and Csurka (1998) and Zsserman et al. (1995) for auto-calbraton of stereo-rgs. But unlke these methods, we do not requre that the two cameras observe and match the same scene features, nor that ther FOVs wll overlap. The need for consstent appearance, whch s a fundamental assumpton n mage algnment or calbraton methods, s replaced here wth the requrement of consstent temporal behavor. Consstent temporal behavor s often easer to satsfy (e.g., by movng the two cameras jontly n space). A smlar dea was used for hand-eye calbraton n robotcs research (e.g., Tsa and Lenz (1989) and Horaud and Dornaka (1995)). Our approach s useful not only n the case of nonoverlappng sequences, but also n other cases where there s very lttle common appearance nformaton between mages, and are therefore nherently dffcult for standard mage algnment technques. Ths gves rse to a varety of real-world applcatons, ncludng: () Mult-sensor algnment for mage fuson. Ths requres accurate algnment of mages (sequences) obtaned by sensors of dfferent sensng modaltes (such as Infra-Red and vsble lght). Such mages dffer sgnfcantly n ther appearance due to dfferent sensor propertes (Vola and Wells III, 1995). () Algnment of mages (sequences) obtaned at dfferent zooms. The problem here s that dfferent mage features are promnent at dfferent mage resolutons (Dufournaud et al., 2000). Algnment of a wde-fov sequence wth a narrow-fov sequence s useful for detectng small zoomed-n objects n (or outsde) a zoomed-out vew of the scene. Ths can be useful n survellance applcatons. () Generaton of wde-screen moves from multple non-overlappng narrow FOV moves (such as n IMAX moves). Our approach can handle such cases. Results are demonstrated n the paper on complex real-world sequences, as well as on manpulated sequences wth ground truth. 2. Problem Formulaton We examne the case when two vdeo cameras havng (approxmately) the same center of projecton but dfferent 3D orentaton, move jontly n space (see Fg. 1). The felds of vew of the two cameras do not necessarly overlap. The nternal parameters of the two cameras are dfferent and unknown, but fxed along the sequences. The external parameters relatng the two cameras (.e., the relatve 3D orentaton) are also unknown but fxed. Let S = I 1,...,I n+1 and S = I 1,...,I m+1 be the two sequences of mages recorded by the two cameras. 1 When temporal synchronzaton (e.g., tme stamps) s not avalable, then I and I may not be correspondng frames n tme. Our goal s to recover the transformaton that algns the two sequences both n tme and n space. Note the term algnment here has a broader meanng than the usual one, as the sequences may not overlap n space, and may not be synchronzed n tme. Here we refer to algnment as dsplayng one sequence n the spatal coordnate system of the other sequence, and at the correct tme shft, as f obtaned by the other camera. When the two cameras have the same center of projecton (and dffer only n ther 3D orentaton and ther nternal calbraton parameters), then a smple fxed homography H (a 2D projectve transformaton) descrbes the spatal transformaton between temporally Fgure 1. Two vdeo cameras are attached to each other, so that they have the same center of projecton, but non-overlappng feldsof-vew. The two cameras are moved jontly n space, producng two separate vdeo sequences I 1,...,I n+1 and I 1,...,I n+1.

3 Algnng Non-Overlappng Sequences 41 Fgure 2. Problem formulaton. The two sequences are spatally related by a fxed but unknown nter-camera homography H, and temporally related by a fxed and unknown tme shft t. Gven the frame-to-frame transformatons T 1,...,T n and T m,we want to recover H and t. correspondng pars of frames across the two sequences (Hartley and Zsserman, 2000). If there were enough common features (e.g., p and p ) between temporally correspondng frames (e.g., I and I ), then t would be easy to recover the ntercamera homography H, as each such par of correspondng mage ponts would provde lnear constrans on H: p = Hp. Ths, n fact, s how most mage algnment technques work (Hartley and Zsserman, 2000). However, ths s not the case here. The two sequence do not share common features, because there s no spatal overlap between the two sequences. Instead, the homography H s recovered from the nduced frameto-frame transformatons wthn each sequence. Let T 1,...,T n and T m be the sequences of frame-to-frame transformatons wthn the vdeo sequences S and S, respectvely. T s the transformaton relatng frame I to I +1. These transformatons can be ether 2D parametrc transformatons (e.g., homographes or affne transformatons) or 3D transformatons/relatons (e.g., fundamental matrces). We next show how we can recover the spatal transformaton H and the temporal shft t between the two vdeo sequences drectly from the two sequences of transformatons T 1,...,T n and T m. The problem formulated above s llustrated n Fg Recoverng Spatal Algnment Between Sequences Let us frst assume that the temporal synchronzaton s known. Such nformaton s often avalable (e.g., from tme stamps encoded n each of the two sequences). Secton 4 shows how we can recover the temporal shft between the two sequences when that nformaton s not avalable. Therefore, wthout loss of generalty, t s assumed that I and I are correspondng frames n tme n sequences S and S, respectvely. Two cases are examned: () The case when the scene s planar or dstant from the cameras. We refer to these scenes as 2D scenes. In ths case the frame-to-frame transformatons T can be modeled by homographes (Secton 3.1). () The case of a non-planar scene. We refer to these scenes as 3D scenes. In ths case the frame-to-frame relatons can be modeled by fundamental matrces (Secton 3.2) Planar or Dstant (2D) Scenes When the scene s planar or dstant from the cameras, or when the jont 3D translatons of the two cameras are neglgble relatve to the dstance of the scene, then the nduced mage motons wthn each sequence (.e., T 1,...,T n and T n ) can be descrbed by 2D parametrc transformatons (Hartley and Zsserman, 2000). T thus denotes the homography between frame I and I +1, represented by 3 3 non-sngular matrx. We next show that temporally correspondng transformatons T and T are related by the same fxed ntercamera homography H (whch relates frames I and I ). Let P be a 3D pont n the planar (or the remote) scene. Denote by p and p ts mage coordnates n frames I and I, respectvely (the pont P need not be vsble n the two frames,.e., P need not be wthn the FOV of the cameras). Let p +1 and p +1 be ts mage coordnates n frames I +1 and I +1, respectvely. Then, p +1 = T p and p +1 = T p. Because the coordnates of the vdeo sequences S and S are related by a fxed homography H, then: p = Hp and p +1 = Hp+1. Therefore: HT p = Hp+1 = p +1 = T p = T Hp (1) Each p could theoretcally have a dfferent scalar assocated wth the equalty n Eq. (1). However, t s easy to show that because the relaton n Eq. (1) holds for all ponts p, therefore all these scalars are equal, and hence: HT = T H. (2) Because H s non-sngular we may wrte T = HT H 1,or T = s HT H 1 (3)

4 42 Casp and Iran where s s a (frame-dependent) scale factor. Equaton (3) s true for all frames,.e., for any par of correspondng transformatons T and T ( = 1..n) there exsts a scalar s such that T = s HT H 1.It shows that there s a smlarty relaton 2 (or a conjugacy relaton ) between the two matrces T and T (up to a scale factor). A smlar observaton was made for case of hand-eye calbraton (e.g., Tsa and Lenz (1989) and Horaud and Dornaka (1995)), and for autocalbraton of a stereo-rg (e.g. Zsserman et al. (1995)). Denote by eg(a) = [λ 1,λ 2,λ 3 ] t a3 1 vector contanng the egenvalues of a 3 3 matrx A (n decreasng order). Then t s known ( Pearson (1983) p. 898.) that: () If A and B are smlar (conjugate) matrces, then they have the same egenvalues: eg(a) = eg(b), and, () The egenvalues of a scaled matrx are scaled: eg(sa) = s(eg(a)). Usng these two facts and Eq. (3) we obtan: eg(t ) = s eg(t ) (4) where s s the scale factor defned by Eq. (3). Equaton (4) mples that the two vectors eg(t ) and eg(t ) are parallel. Ths gves rse to a measure of smlarty between two matrces T and T sm(t, T ) = eg(t ) t eg(t ) eg(t ) eg(t (5) ), where s the vector norm. For real valued egenvalues, Eq. (5) provdes the cosne of the angle between the two vectors eg(t ) and eg(t ). Ths property wll be used later for obtanng the temporal synchronzaton between the two sequences (Secton 4). Ths measure s also used for outler rejecton of bad frame-toframe transformaton pars, T and T (Secton 6.3). The remander of ths secton explans how the fxed nter-camera homography H s recovered from the lst of frame-to-frame transformatons T 1,...,T n and T n. For each par of temporally correspondng transformatons T and T n sequences S and S,wefrst compute ther egenvalues eg(t ) and eg(t ). The scale factor s whch relates them s then estmated from Eq. (4) usng least squares mnmzaton (three equatons, one unknown). 3 Once s s estmated, Eq. (3) (or Eq. (2)) can be rewrtten as: s HT T H = 0 (6) Eq. (6) s lnear n the unknown components of H. Rearrangng the components of H n a 9 1 column vector : h = [H 11 H 12 H 13 H 21 H 22 H 23 H 31 H 32 H 33 ] t, Eq. (6) can be rewrtten as a set of lnear equatons n h: M h = 0 (7) where M sa9 9 matrx defned by T, T and s : M = s T t T 11 I T 12 I T T 21 I s T t T 22 I T 13 I 23 I T 31 I T 32 I s T t T 33 I 9 9 where I s the 3 3 dentty matrx. Equaton (7) mples that each par of correspondng transformatons T and T contrbutes 9 lnear constrans n the unknown homography H (.e., h), out of whch at most 6 constrants are lnearly ndependent (see Secton 6). Therefore, n theory, at least two such pars of ndependent transformatons are needed to unquely determne the homography H (up to a scale factor). In practce, we use all avalable constrants from all pars of transformatons to compute H. The constrants from all the transformatons T 1,...,T n and T n can be combned nto a sngle set of lnear equatons n h: A h = 0 (8) where A s a 9n 9 matrx: A = [ M 1. ]. Equaton (8) s M n a homogeneous set of lnear equatons n h, that can be solved n a varety of ways (Bjorck, 1996). In partcular, h may be recovered up to scale by computng the egenvector whch corresponds to the smallest egenvalue of the matrx A t A D Scenes When the scene s nether planar nor dstant, the relaton between two consecutve frames of an uncalbrated camera s descrbed by the fundamental matrx (Hartley and Zsserman, 2000). In ths case the nput to our algorthm s two sequences of fundamental matrces between successve frames, denoted by F 1,...,F n and F 1,...,F n. Namely, f p I and p +1 I +1 are correspondng mage ponts, then: p+1 t F p = 0. Although the relatons wthn each sequence are characterzed by fundamental matrces, the nter-camera transformaton remans a homography H. Ths s because the two cameras stll share the same center of projecton (Secton 2). Each fundamental matrx F can be decomposed nto a homography + eppole (Hartley and Zsserman,

5 Algnng Non-Overlappng Sequences ) as follows: F = [e ] T where e s the eppole relatng frames I and I +1, the matrx T s the nduced homography from I to I +1 va any plane (real or vrtual). [ ] s the cross product matrx ([v] w = v w). The homographes, T 1,...,T n and T n, and the eppoles e 1,...,e n and e 1,...,e n, mpose separate constrants on the nter-camera homography H. These constrants can be used separately or jontly to recover H Homography-Based Constrants. The homographes T 1,...,T n and T n (extracted from the fundamental matrces F 1,...,F n and F 1,...,F n, respectvely), may correspond to dfferent 3D planes. In order to apply the algorthm of Secton 3.1 usng these homographes, we need to mpose plane-consstency across the two sequences (to guarantee that temporally correspondng homographes correspond to the same plane n the 3D world). One possble way for mposng plane-consstency across (and wthn) the two sequences s by usng the Plane + Parallax approach (Kumar et al., 1994; Iran et al., 1998; Shashua and Navab, 1994; Sawhney, 1994). However, ths approach requres that a real physcal planar surface be vsble n all vdeo frames. Alternatvely, the threadng method of Avdan and Shashua (1998) or other methods for computng consstent set of camera matrces (e.g., Beardsley et al., 1998), can mpose plane-consstency wthn each sequence, even f no real physcal plane s vsble n any of the frames. Plane consstency across the two sequences can be obtaned, e.g., f Avdan and Shashua (1998) s ntated at frames whch are known to smultaneously vew the same real plane n both sequences. Ths can be done even f the two cameras see dfferent portons of the plane (allowng for nonoverlappng FOVs), and do not see that plane at any of the other frames. Ths approach s therefore less restrctve than the Plane + Parallax approach Eppole-Based Constrants. The fundamental matrces F 1..F n and F 1..F n also provde a lst of eppoles e 1,...,e n and e 1,...,e n. These eppoles are unquely defned (there s no ssue of plane consstency here). Snce the two cameras have the same center of projecton, then for any frame : e = He, or more specfcally: (e ) x = [h 1h 2 h 3 ] e (e [h 7 h 8 h 9 ] e ) y = [h 4h 5 h 6 ] e (9) [h 7 h 8 h 9 ] e Multplyng by the domnator and rearrangng terms yelds two new lnear constrans on H for every par of correspondng eppoles e and e : [ e t 0 t (e ) ] x et 0 t e t (e ) h = 0 (10) y et 2 9 where 0 t = [0, 0, 0]. Every par of temporally correspondng eppoles, e and e, thus mposes two lnear constrants on H. These 2n constrants ( = 1,...,n) can be added to the set of lnear equatons n Eq. (8) whch are mposed by the homographes. Alternatvely, the eppole-related constrants can be used alone to solve for H, thus avodng the need to enforce planeconsstency on the homographes. Theoretcally, four pars of correspondng eppoles e and e n general poston (no 3 on the same lne) are suffcent. 4. Recoverng Temporal Synchronzaton Between Sequences So far we have assumed that the temporal synchronzaton between the two sequences s known and gven. Namely, that frame I n sequence S corresponds to frame I n sequence S, and therefore the transformaton T corresponds to transformaton T. Such nformaton s often avalable from tme stamps. However, when such synchronzaton s not avalable, we can recover t. Gven two unsynchronzed sequences of transformatons T 1,...,T n and T m, we wsh to recover the unknown temporal shft t between them. Let T and T + t be temporally correspondng transformatons (namely, they occurred at the same tme nstance). Then from Eq. (4) we know that they should satsfy eg(t ) eg(t + t ) (.e., the 3 1 vectors of egenvalues should be parallel). In other words, the smlarty measure sm(t t, T t + t ) of Eq. (5) should equal 1 (correspondng to cos(0),.e., an angle of 0 between the two vectors). All pars of correspondng transformatons T and T + t must smultaneously satsfy ths constrant for the correct tme shft t. Therefore, we recover the unknown temporal tme shft t by maxmzng the followng objectve functon: SIM( t) = sm(t, T + t )2 (11)

6 44 Casp and Iran The maxmzaton s currently performed by an exhaustve search over a fnte range of vald tme shfts t. To address larger temporal shfts, we apply a herarchcal search. Coarser temporal levels are constructed by composng transformatons to obtan fewer transformaton between more dstant frames. The objectve functon of Eq. (11) can be generalzed to handle sequences of dfferent frame rates, such as sequences obtaned by NTSC cameras (30 frame/sec) vs. PAL cameras (25 frames/sec). The rato between frames correspondng to equal tme steps n the two sequences s 25:30 = 5:6. Therefore, the objectve functon that should be maxmzed for an NTSC-PAL par of sequences s: SIM( t) = ( sm T 5(+1) 5 ) 2, T 6(+1)+ t (12) 6+ t Where T j s the transformaton from frame I to frame I j. In our experments, all sequences were obtaned by PAL vdeo cameras. Therefore only the case of equal frame-rate (Eq. (11)) was expermentally verfed. We found ths method to be very robust. It successfully recovered the temporal shft up to feld (half-frame) accuracy. Sub-feld accuracy may be further recovered by nterpolatng the values of SIM( t) obtaned at dscrete tme shfts. 5. Applcatons Ths secton llustrates the applcablty of our method to solvng some real-world problems, whch are partcularly dffcult for standard mage algnment technques. These nclude: () Algnment of nonoverlappng sequences for generaton of wde-screen moves from multple narrow-screen moves (such as n IMAX flms), () Algnment of sequences obtaned at sgnfcantly dfferent zooms (e.g., for survellance applcatons), and () Algnment of mult-sensor sequences for mult-sensor fuson. We show results of applyng the method to complex real-world sequences. All sequences whch we expermented wth, were captured by off-the-shelf consumer CCD cameras. The cameras were attached to each other to mnmze the dstance between ther centers of projectons. The jont camera moton was performed manually (.e., a person would manually hold and rotate the two attached cameras). No temporal synchronzaton tool was used. The frame-to-frame nput transformatons wthn each sequence (homographes T 1,...,T n and 1,...,T n ) were extracted usng the method descrbed n Iran et al. (1994). Inaccurate frame-to-frame transformatons T are pruned out by usng two outler detecton mechansms (see Secton 6.3). The nput sequences were usually several seconds long to guaranty sgnfcant enough moton. The temporal tme shft was recovered usng the algorthm descrbed n Secton 4 up to feld accuracy. Fnally, the best thrty or so transformatons were used n the estmaton of the nter-camera homography H (usng the algorthm descrbed n Secton 3.1). T 5.1. Algnment of Non-Overlappng Sequences Fgure 3 shows an example of algnment of nonoverlappng sequences. The left camera s zoomed-n and rotated relatve to the rght camera. The correct spato-temporal algnment can be seen n Fg. 3(c). Note the accurate algnment of the runnng person both n tme and n space. Our approach to sequence algnment can be used to generate wde-screen moves from two (or more) narrow feld-of-vew moves (such as n IMAX moves). Such an example s shown n Fg. 4. To verfy the accuracy of algnment (both n tme and n space), we allowed for a very small overlap between the two sequences. However, ths mage regon was not used n the estmaton process, to mtate the case of truly non-overlappng sequences. The overlappng regon was used only for dsplay and verfcaton purposes. Fgure 4(c) shows the result of combnng the two sequences (by averagng correspondng frames) after spato-temporal algnment. Note the accurate spatal as well as temporal algnment of the soccer player n the averaged overlappng regon Algnment of Sequences Obtaned at Dfferent Zooms Often n survellance applcatons two cameras are used, one wth a wde FOV (feld-of-vew) for observng large scene regons, and the other camera wth a narrow FOV (zoomed-n) for detectng small objects. Matchng two such mages obtaned at sgnfcantly dfferent zooms s a dffcult problem for standard mage algnment methods, snce the two mages dsplay dfferent features whch are promnent at the dfferent resolutons. Our sequence algnment approach may be used for such scenaros. Fgure 5 shows three such

7 Algnng Non-Overlappng Sequences 45 Fgure 3. Algnment of non-overlappng sequences. (a) and (b) are temporally correspondng frames from sequences S and S. The correct tme shft was automatcally detected. (c) shows one frame n the combned sequence after spato-temporal algnment. Note the accuracy of the spatal and temporal algnment of the runnng person. For color sequences see examples. The results of the spato-temporal algnment (rght column of Fg. 5) are dsplayed n the form of averagng temporally correspondng frames after algnment accordng to the computed homography and the computed tme shft. In the frst example (top row of Fg. 5) the zoom dfference between the two cameras was approxmately 1:3. In the second example (second row) t was 1:4, and n the thrd example (bottom row) t was 1:8. Note the small red flowers n the zoomed vew (Fg. 5.2.(b)), that can barely be seen n the correspondng low resoluton wde-vew frame (Fg. 5.2 (a)). The same holds for the Pagoda n Fg. 5.3 (b) Mult-Sensor Algnment Images obtaned by sensors of dfferent modaltes, e.g., IR (Infra-Red) and vsble lght, can vary sgnfcantly n ther appearance. Features appearng n one mage may not appear n the other, and vce versa. Ths poses a problem for mage algnment methods. Our sequence algnment approach, however, does not requre consstent appearance between the two sequences, and can therefore be appled to solve the problem. Fgure 6 shows an example of two such sequences, one captured by a near IR camera, whle the other by a regular vdeo (vsble-lght) camera. The scene was shot n twlght. In the sequence obtaned by the regular vdeo camera (Fg. 6(a)), the outdoor scene s barely vsble, whle the nsde of the buldng s clearly vsble. The IR camera, on the other hand, captures the outdoor scene n great detal, whle the ndoor part (llumnated by cold neon lght) was nvsble to the IR camera (Fg. 6(b)). The result of the spato-temporal algnment s llustrated by fusng temporally correspondng frames. The IR camera provdes only ntensty nformaton, and was therefore fused only wth the ntensty (Y) component of the vsble-lght camera (usng the mage-fuson method of Burt and Kolczynsk (1993)). The chrome components (I and Q) of the vsble-lght camera supply the color nformaton. The reader s encouraged to vew color sequences at

8 46 Casp and Iran Fgure 4. Wde-screen moves generaton. (a) and (b) are temporally correspondng frames from sequences S and S. The correct tme shft was automatcally detected. (c) shows one frame n the combned sequence. Correspondng vdeo frames were averaged after spato-temporal algnment. The small overlappng area was not used n the estmaton process, but only for verfcaton (see text). Note the accuracy of the spatal and temporal algnment of the soccer player n the overlappng regon. For color sequences see 6. Analyss In ths secton we evaluated the effectveness and stablty, of the presented approach emprcally (Secton 6.1) and theoretcally (Secton 6.2) and numercally (Secton 6.3) Emprcal Evaluaton In order to emprcally verfy the accuracy of our method, we took a real vdeo sequence (see Fg. 7) and generated from t pars of sequences wth known (ground truth) spatal transformaton H and temporal shft t. We then appled our algorthm and compared the recovered H and t wth the ground truth. For the case of non overlappng sequences, the real sequence of Fg. 7 was splt n the mddle, producng two non-overlappng sub-sequences of half-a-frame wdth each. The true (ground truth) homography H therefore corresponds to a horzontal shft by the wdth of a halved frame (352 pxels), and t n ths case s 0. The nter-camera homography H was recovered up to a msalgnment error of less than 0.7 pxel over the entre mage. The temporal shft ( t = 0) was recovered accurately from the frame-to-frame transformatons. To emprcally verfy the accuracy of our method n the presence of large zooms and large rotatons, we ran the algorthm on followng three manpulated sequences wth known (ground truth) manpulatons: We warped the sequence of Fg. 7 (once by a zoom factor of 2, once by a zoom factor of 4, and once rotated t by 180 ) to generate the second sequence. The results are summarzed n Table 1. The reported resdual msalgnment was measured as follows: The recovered homography was composed wth the nverse of the ground-truth homography: Htrue 1 H recovored. Ideally, the composed homography should be the dentty

9 Algnng Non-Overlappng Sequences 47 Fgure 5. Fndng zoomed regon. Ths fgure dsplays three dfferent examples (one at each row), each one wth dfferent zoom factor. The left column (1.a, 2.a, 3.a) dsplay one frame from each of the three wde-fov sequences. The temporally correspondng frames from the correspondng narrow-fov sequences are dsplayed n the center column (1.b, 2.b, 3.b). The correct tme shft was automatcally detected for each par of narrow/wde FOV sequences. Each mage on the rght column shows super-poston of correspondng frames of the two sequences after spatotemporal algnment, dsplayed by color averagng (1.c, 2.c, 3.c). For color sequences see Fgure 6. Mult-sensor algnment. (a) and (b) are temporally correspondng frames from the vsble-lght and near-ir sequences, respectvely (the temporal algnment was automatcally detected). The nsde of the buldng s vsble only n the vsble-lght sequence, whle the IR sequence captures the detals outdoors (e.g., the dark trees, the sgn, the bush). (c) shows the results of fusng the two sequences after spato-temporal algnment. The fused sequence preserves the detals from both sequences. Note the hgh accuracy of algnment (both n tme and n space) of the walkng lady. For more detals see text. For color sequences see

10 48 Casp and Iran Fgure 7. The sequence used for emprcal evaluaton. (a, b, c) are three frames (0, 150, 300) out of the orgnal 300 frames. Ths sequence was used as the base sequence for the quanttatve experments summarzed n Table 1. matrx. The errors reported n Table 1 are the maxmal resdual msalgnment nduced by the composed homography over the entre mage. In all the cases the correct t was recovered (not shown n the table). Table 1. Quanttatve results. Appled Recovered Max resdual transformaton transformaton msalgnment Horzontal shft Horzontal shft 0.7 pxels of 352 pxels of pxels Zoom factor = 2 Zoom factor = pxels Zoom factor = 4 Zoom factor = pxels Rotaton by 180 Rotaton by pxels Ths table summarzes the quanttatve results wth respect to ground truth. Each row corresponds to one experment. In each experment a real vdeo sequence (Fg. 7) was warped ( manpulated ) by a known homography, to generate a second sequence. The left column descrbes the type of spatal transformaton appled to the sequence, the center column descrbes the recovered transformaton, and the rght column descrbes the resdual error between the ground-truth homography and the recovered homography (measured n maxmal resdual msalgnment n the mage space). In all 4 cases the correct temporal shft was recovered accurately. See text for further detals. The followng notatons are used: Denote by λ 1,λ 2,λ 3 the egenvalues of the matrx B n decreasng order ( λ 1 λ 2 λ 3 ). Denote by u b1, u b2, u b3 the correspondng egenvectors wth unt norm ( u b1 = u b2 = u b3 =1). Denote by r j the algebrac multplcty 5 of the egenvalue λ j, and denote by V j ={ v R n : B v = λ j v} the correspondng egen subspace Basc Constrants. Smlar (conjugate) matrces (e.g., B and G) have the same egenvalues but dfferent egenvectors. Ther egenvectors are related by H. Ifu b s an egenvector of B wth correspondng egenvalue λ, then Hu b s an egenvector of G wth the same egenvalue λ: G(Hu b ) = λ(hu b ). The same holds for egen subspaces. If V s an egen subspace of B correspondng to an egenvalue λ, then H(V ) s an egen subspace of G wth the same egenvalue λ. We nvestgate the number of constrants mposed on H by B and G accordng to the dmensonalty of ther egen subspaces. Let V be the egen subspace correspondng to an egenvector u b of B. We nvestgate three possble cases, one for each possble dmensonalty of V,.e., dm(v ) = 1, 2, Unqueness of Soluton Ths secton studes how many pars of correspondng transformatons T and T are requred n order to unquely resolve the nter-camera homography H. To do so we examne the number of constrants mposed on H by a sngle par of transformatons va the smlarty equaton Eq. (3). Snce we can extract the scale factor s drectly from T and T (see Secton 3.1) we can omt the scale factor s and study the followng queston: How many constrants does an equaton of the form G = HBH 1 mpose on H? (e.g., B = T and G = T ).4 Case I. dm(v ) = 1. Ths case mostly occurs when all three egenvalues are dstnct, but can also occur f some egenvalues have algebrac multplcty two or even three. In all these cases, V s spanned by the sngle egenvector u b. Smlarly H(V ) s spanned by the egenvector u g of G. Therefore: Hu b = αu g (13) wth an unknown scale factor α. Equaton (13) provdes 3 lnear equatons n H and one new unknown α, thus n total t provdes two new lnearly ndependent constrants on H.

11 Algnng Non-Overlappng Sequences 49 Case II. dm(v ) = 2. Ths occurs n one of the followng two cases: (a) when there exsts an egenvalue wth algebrac multplcty two, or (b) when there s only one egenvalue wth algebrac multplcty three, but the egen subspace spanned by all egenvectors has dmensonalty of two. 6 When dm(v ) = 2 then two egenvectors span V (w.l.o.g., u b1 and u b2 ). Then every lnear combnaton of u b1 and u b2 s also an egenvector of B wth the same egenvalue. Smlarly, every lnear combnaton of u g1 and u g2 s an egenvector of G wth the same egenvalue. Therefore: Hu b j = α j u g1 + β j u g2 (14) where α j and β j are unknown scalars ( j = 1, 2). Hence, each of the two egenvectors u b1 and u b2 provdes 3 lnear equatons and 2 new unknowns. Therefore, n total, together they provde 2 new lnear constrants on H. Case III. dm(v ) = 3. In ths case any vector s an egenvector (all wth the same egenvalue λ). Ths s the case when B = G = λi are the dentty transformaton up to scale,.e., no camera moton. In ths case (as expected) B and G provde no addtonal constrants on H Countng Constrans. So far we counted the number of constrants mposed on H by a sngle egen subspace. In order to count the total number of lnear constrants that B and G mpose on H, we analyze every possble combnaton of egen subspaces accordng to the algebrac multplcty of ther egenvalues: 1. λ =λ j =λ k. Ths mples V =V j =V k and dm(v ) = dm(v j ) = dm(v k ) = λ = λ j =λ k (V = V j =V k ). There are two such cases: (a) dm(v = V j ) = 2, and dm(v k ) = 1. (b) dm(v = V j ) = 1, and dm(v k ) = λ = λ j = λ k. In ths case there s only a sngle egen subspace V = V = V j = V k. Its dmensonalty may be 1, 2, or 3. The followng table summarzes the number of lnearly ndependent constrants for each of the above cases: Egenvalue Egen No. of lnearly algebrac subspace ndependent Case multplcty dmensonalty constrants (1) λ =λ j =λ k V = V j = V k =1 6 (2.a) λ = λ j =λ k V = V j =2, V k =1 4 (2.b) λ = λ j =λ k V = V j =1, V k =1 4 (3.a) λ = λ j = λ k V = V j = V k =1 2 (3.b) λ = λ j = λ k V = V j = V k =2 2 (3.c) λ = λ j = λ k V = V j = V k =3 0 To summarze: When B (and G) have ether two or three dstnct egenvalues (whch s typcal of general frame-to-frame transformatons), then two ndependent pars of transformatons suffce to unquely determne H. Ths s because each par of transformatons mposes 4 to 6 lnearly ndependent constrants, and n theory 8 ndependent lnear constrants suffce to unquely resolve H (up to arbtrary scale factor) Numercal Stablty The fnal step n our algorthm s to solve a homogeneous set of lnear equatons (Eq. (8)). Care has to be taken when solvng ths system. For example, naccuraces n the estmated frame-to-frame transformatons decrease the accuracy of the fnal output. The prevous secton showed that two ndependent pars of transformatons may suffce to unquely determne H. In practce, however, to ncrease numercal stablty, we use all avalable constrants from all pars of relable transformatons after subsamplng of the sequences, outler rejecton and normalzaton. These are explaned next: Temporal Subsamplng. When the frame-to-frame transformatons are too small, we often temporally subsample the sequences to obtan more sgnfcant transformatons between successve frames. In our experments where vdeo clps were a couple of hundred frames long, we usually used 30 relable transformatons between dstant (non-successve) frames. Such temporal subsamplng should be done after recoverng the temporal synchronzaton, to assure that t s done n a temporally synchronzed manner across the two sequences. Outler Rejecton. Inaccurate frame-to-frame transformatons T are pruned out by usng two outler detecton mechansms:

12 50 Casp and Iran () The transformaton between successve frames wthn each sequence are computed n both drectons. Let T be the transformaton from I to I +1, and T Reverse the transformaton from I +1 to I. Then we measure the devaton of the composed matrx T T Reverse from the dentty matrx n terms of the maxmal nduced resdual msalgnment of pxels,.e., Relablty(T ) = max T T Reverse p p (15) p I () The smlarty crteron of Eq. (5) can also be used to verfy the degree of smlarty between a par of transformatons T and T. After t has been estmated and before H s estmated, an unrelable par of transformatons can be detected and pruned out by measurng the devaton of Sm(T, T ) from 1. However, the frst outler crteron (that of Eq. (15)) proved to be more powerful. Matrx Normalzaton. Usng the heurstc provded n Golub and Van Loan (1989) (orgnly derved for Gaussan elmnaton) we normalze (scale) components of the nput matrces T and T n a way that the rows of the matrx A of Eq. (8) wll have approxmately the same norm. Ths s an equvalent step to the scalng proposed by Hartley (1997) for recoverng fundamental matrces. Ths step ndeed mprove the results. Preferred Camera Motons. When acqurng the sequences of nput transformatons, we usually have control over the camera moton. In general, any type of camera moton provdes a frame-to-frame transformaton whch nduces constrants on the nter-camera homography H. However, some transformatons provde more stable sets of equatons than others. In partcular, we would lke to generate sequences of transformatons whch provde more relable components n each column of the matrx A n Eq. (8). For example, mageplane rotatons (.e., rotatons about the optcal axs of one of the cameras) usually provde relable entres n all columns of M (a block of A), thus mpose stable constrants on H. To conclude, the camera rg can (and should) be moved freely, however, t s recommended that a few of the camera movements nclude non-neglgble mage-plane rotatons. 7. Concluson Ths paper presents an approach for algnng two sequences (both n tme and n space), even when there s no common spatal nformaton between the sequences. Ths was made possble by replacng the need for consstent appearance (whch s a fundamental requrement n standard mages algnment technques), wth the requrement of consstent temporal behavor, whch s often easer to satsfy. We demonstrated applcatons of ths approach to real-world problems, whch are nherently dffcult for regular mage algnment technques. Acknowledgment The authors would lke to thank R. Basr and L. Zelnk- Manor for ther helpful comments. Notes 1. The subscrpt s used to represent the frame tme ndex, and the superscrpt prme s used to dstngush between the two sequences S and S. 2. A matrx A s sad to be smlar to a matrx B f there exsts an nvertble matrx M such that A = MBM 1 (see Pearson (1983)). The term conjugate matrces s also often used. 3. Alternatvely, the nput homographes can be normalzed to have determnant equal to 1, to avod the need to compute s. 4. A general analyss of matrx equatons of the form GH = HB may be found n Gantmakher (1959). 5. If λ 1 =λ 2 =λ 3 then the algebrac multplcty of all egenvalues s 1 (r j = 1).Ifλ 1 = λ 2 =λ 3 then the algebrac multplcty of λ 1 and λ 2 s 2, and the algebrac multplcty of λ 3 s 1 (r 1 = r 2 = 2 and r 3 = 1). Ifλ 1 = λ 2 = λ 3 then the algebrac multplcty of λ 1, λ 2, and λ 2 s 3 (r 1 = r 2 = r 3 = 3). 6. Egenvalues wth algebrac multplcty 2 and 3 are not rare. For example a homography defned by pure shft ( x, y) has the form: H = [ 10 x 01 y]. Ths matrx has a sngle egenvalue 00 1 λ 1 = λ 2 = λ 3 = 1 wth algebrac multplcty three. The correspondng egen subspace has dmensonalty 2. It s spanned by two lnearly ndependent egenvectors [1, 0, 0] t and [0, 1, 0] t. References Avdan, S. and Shashua, A Threadng fundamental matrces. In European Conference on Computer Vson. Beardsley, P.A., Torr, P.H.S., and Zsserman, A D model aquston from extended mage sequences. In Proc. 4th European Conference on Computer Vson, LNCS 1065, Cambrdge, pp Bjorck, A Numercal Methodes for Least Squares Problems. SIAM: Phladelpha.

13 Algnng Non-Overlappng Sequences 51 Burt, P.R. and Kolczynsk, R.J Enhanced mage capture through fuson. In Internatonal Conference on Computer Vson, pp Casp, Y. and Iran, M A step towards sequence-to-sequence algnment. In IEEE Conference on Computer Vson and Pattern Recognton, Hlton Head Island, South Carolna, June 2000, pp Casp, Y. and Iran, M Algnment of non-overlapng sequences. In Internatonal Conference on Computer Vson, vol. II, Vancouver, Canada, pp Demrdjan, D., Zsserman, A., and Horaud, R Stereo autocalbraton from one plane. In European Conference on Computer Vson, pp Dufournaud, Y., Schmd, C., and Horaud, R Matchng mages wth dfferent resolutons. In IEEE Conference on Computer Vson and Pattern Recognton, Hlton Head Island, South Carolna, June 2000, pp Gantmakher, F.R The Theory of Matrces. Chelsea Pub.: New York. Golub, Gene and Van Loan, Charles Matrx Computatons. The Johns Hopkns Unversty Press: Baltmore and London. Hartley, R.I In defence of the 8-pont algorthm. IEEE Trans. on Pattern Analyss and Machne Intellgence, 19(6): Hartley, R. and Zsserman, A Multple Vew Geometry n Computer Vson. Cambrdge Unversty Press: Cambrdge. Horaud, R. and Csurka, G Reconstructon usng motons of a stereo rg. In Internatonal Conference on Computer Vson, pp Horaud, R. and Dornaka, F Hand-eye calbraton. Internatonal Journal of Robotcs Research, 14(3): Iran, M. and Anandan, P About drect methods. In Vson Algorthms Workshop, Corfu, pp Iran, M., Anandan, P., and Wenshall, D From reference frames to reference planes: Mult-vew parallax geometry and applcatons. In European Conference on Computer Vson, Freburg, June 1998, pp Iran, M., Rousso, B., and Peleg, S Computng occludng and transparent motons. Internatonal Journal of Computer Vson, 12(1):5 16. Kumar, R., Anandan, P., and Hanna, K Drect recovery of shape from multple vews: Parallax based approach. In Internatonal Conference on Pattern Recognton, pp Pearson, C.E. (Ed.), Handbook of Appled Mathematcs, 2nd edn. Van Nostrand Renhold Company: New York. Sawhney, H D geometry from planar parallax. In IEEE Conference on Computer Vson and Pattern Recognton, June 1994, pp Shashua, A. and Navab, N Relatve affne structure: Theory and applcaton to 3D reconstructon from perspectve vews. In IEEE Conference on Computer Vson and Pattern Recognton, Seattle, WA, June 1994, pp Slama, C.C Manual of Photogrammetry. Amercan Socety of Photogrammetry and Remote Sensng. Sten, G.P Trackng from multple vew ponts: Selfcalbraton of space and tme. In DARPA IU Workshop, Montery CA, pp Torr, P.H.S. and Zsserman, A Feature based methods for structure and moton estmaton. In Vson Algorthms Workshop, Corfu, pp Tsa, R.Y. and Lenz, R.K A new technque for full autonomous and effcent 3D robotcs hand/eye calbraton. IEEE Journal of Robotcs and Automaton, 5(3): Vola, P. and Wells III, W Algnment by maxmzaton of mutual nformaton. In Internatonal Conference on Computer Vson, pp Zsserman, A., Beardsley, P.A., and Red, I.D Metrc calbraton of a stereo rg. In Workshop on Representatons of Vsual Scenes, pp