Novel View Synthesis by Cascading Trilinear Tensors

Transcription

1 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 4, NO. 4, OCTOBER-DECEMBER Novel Vew Synthess by Cascadng Trlnear Tensors Sha Avdan and Amnon Shashua, Member, IEEE Abstract We present a new method for syntheszng novel vews of a 3D scene from two or three reference mages n full correspondence. The core of ths wor s the use and manpulaton of an algebrac entty termed the trlnear tensor that lns pont correspondences across three mages. For a gven vrtual camera poston and orentaton, a new trlnear tensor can be computed based on the orgnal tensor of the reference mages. The desred vew can then be created usng ths new trlnear tensor and pont correspondences across two of the reference mages. Index Terms Image-based renderng, trlnear tensor, vrtual realty, mage manpulaton. F 1 INTRODUCTION T HIS paper addresses the problem of syntheszng a novel mage from an arbtrary vewng poston gven two or three reference mages (regstered by means of an optc-flow engne) of the 3D scene. The most sgnfcant aspect of our approach s the ablty to synthesze mages that are far away from the vewng postons of the sample reference mages wthout ever explctly computng any 3D nformaton about the scene. Ths property provdes a mult-mage representaton of the 3D obect usng a mnmal number of mages. In our experments, for example, two closely spaced frontal mages of a face are suffcent for generatng photorealstc mages from vewponts wthn a 60 degree cone of vsual angle further extrapolaton s possble, but the mage qualty degrades. We propose a new vew-synthess method that maes use of the recent development of multlnear matchng constrants, nown as trlneartes, that were frst ntroduced n [4]. The trlneartes provde a general (not subect to sngular camera confguratons) warpng functon from reference mages to novel syntheszed mages governed drectly by the camera parameters of the vrtual camera. Therefore, we provde a true mult-mage system for vew synthess that does not requre a companon depth map nor the full reconstructon of camera parameters among the reference cameras, yet s general and robust. The core of ths wor s the dervaton of a tensor operator that descrbes the transformaton from a gven tensor of three vews to a novel tensor of a new confguraton of three vews. Thus, by repeated applcaton of the operator on the seed tensor of the reference mages wth a sequence of desred vrtual camera postons (translaton and orentaton), we obtan a chan of warpng functons (tensors) from the set of reference mages (from whch the seed tensor was computed) to create the desred vrtual vews. We also show that the process can start wth two reference ²²²²²²²²²²²²²²²² The authors are wth the Insttute of Computer Scence, The Hebrew Unversty, Jerusalem 91904, Israel. E-mal: {avdan, shashua}@cs.hu.ac.l. For nformaton on obtanng reprnts of ths artcle, please send e-mal to: tvcg@computer.org, and reference IEEECS Log Number vews by havng the seed tensor be comprsed of the elements of the fundamental matrx of the reference vews. A shorter verson of ths paper appeared n [4]. 1.1 Novelty Over Prevous Wor The noton of mage-based renderng systems s ganng momentum n both the computer graphcs and computer vson communtes. The general dea s to avod the computatonally ntensve process of acqurng a 3D model followed by renderng and, nstead, to use a number of reference mages of the obect (or scene) as a representaton from whch novel vews can be syntheszed drectly by means of mage warpng. The wor n ths area can be roughly dvded nto three classes: 1) mage nterpolaton, ) off-lne (Mosac-based) synthess, and 3) on-lne synthess. The frst class, mage nterpolaton, s desgned to create n-between mages among two or more reference mages. Ths ncludes mage morphng [8], drect nterpolaton from mage-flows ( multdmensonal morphng ) [10], [37], mage nterpolaton usng 3D models nstead of mage-flow [1], and physcally correct mage nterpolaton [40], [41], [55]. All but the last three references do not guarantee to produce physcally correct mages and all cannot extrapolate from the set of nput mages that s, create novel vewng postons that are outsde of the vewng cone of the reference mages. For example, Setz and Dyer [41] have shown that one can nterpolate along the base-lne of an mage par and obtan physcally correct mages (unle flow-based nterpolaton [10], [37]). Ther approach proceeds by frst rectfyng the mages, nterpolatng along the eppolar lnes (whch are parallel after the rectfcaton), and, then, nvertng the rectfcaton for the fnal renderng. Unfortunately, only mages along the lne connectng the two model mages can be generated n ths way and the user s not allowed to move freely n 3D space. Instead of flow-feld nterpolaton among the reference mages, t s possble to nterpolate drectly over the plenoptc /98/$ IEEE

2 94 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 4, NO. 4, OCTOBER-DECEMBER 1998 functon [1] a functon whch represents the amount of lght emtted at each pont n space as a functon of drecton. Levoy and Hanrahan [31] and Gortler et al. [19] nterpolate between a dense set of several thousand example mages to reconstruct a reduced plenoptc functon (under an occluson-free world assumpton). Hence, they consderably ncrease the number of example mages to avod computng optcal flow between the reference mages. In the second class, the off-lne (mosac-based) synthess, the synthess s not created at run tme nstead, many overlappng mages of the scene are taen and then sttched together. The smplest sttchng occurs when the camera moton ncludes only rotaton n whch case the transformaton between the vews s parametrc and does not nclude any 3D shape (the transformaton beng a D proectve transformaton, a homography). Ths was cleverly done by [13] n what s nown as QucTme VR. Szels and Kang [51] create hgh-resoluton mosacs from low-resoluton vdeo streams, and Peleg and Herman [35] relax the fxed camera constrant by ntroducng the proecton manfold. A drawbac of ths class s that one cannot correctly smulate translatonal camera moton from the set of reference mages. The maor lmtaton of the aforementoned technques s that a relatvely large number of reference mages s requred to represent an obect. The thrd class, on-lne synthess, along the lnes of ths paper, reduces the number of acqured (reference) mages by explotng the 3D-from-D geometry for dervng an on-lne warpng functon from a set of reference mages to create novel vews on-the-fly based on user specfcaton of the vrtual camera poston. Laveau and Faugeras [30] were the frst to use the eppolar constrant for vew synthess, allowng them to extrapolate, as well as nterpolate, between the example mages. Eppolar constrants, however, are subect to sngulartes that arse under certan camera motons (such as when the vrtual camera center s collnear wth the centers of the reference cameras), and the relaton between translatonal and rotatonal parameters of the vrtual camera and the eppolar constrant s somewhat ndrect and, hence, requres the specfcaton of matchng ponts. The sngular camera motons can be relaxed by usng the depth map of the envronment. McMllan and Bshop [33] use a full depth map (3D reconstructon of the camera moton and the envronment) together wth the eppolar constrant to provde a drect connecton between the vrtual camera moton and the reproecton engne. Depth maps are easly provded for synthetc envronments, whereas, for real scenes, the process s fragle, especally under small base-lne stuatons that arse due to the requrement of dense correspondence between the reference mages/mosacs [0]. The challenges facng an optmal on-lne synthess approach are, therefore: Implct Scene Modelng: To reduce, as much as possble, the computatonal steps from the nput correspondence feld among the reference mages to useful algebrac structures that would suffce for generatng new vews. For example, t s lely that the base-lne between reference vews would be very small n order to facltate the correspondence process. Thus, computng the full set of camera parameters (or, equvalently, the depth map of the scene) s not desrable as t may produce unstable estmates, especally for the translatonal component of camera moton (the eppoles). It s thus desrable to have the camera parameters reman as much as possble mplct n the process. Nonsngular Confguratons: To rely on warpng functons that are free from sngulartes under camera moton. For example, the use of the fundamental matrx, or concatenaton of fundamental matrces, for dervng a warpng functon based on eppolar lne ntersecton (cf. [18]) s undesrable on ths account due to sngulartes that arse when the camera centers are collnear. Drvng Mode: The specfcaton of the vrtual camera poston should be ntutvely smple for the user. For example, rotaton and translaton of the camera from ts current poston s prevalent among most 3D vewers. None of the exstng approaches for on-lne synthess satsfes all three requrements. For example, [30] satsfes the frst requrement at the cost of complcatng the drvng mode by specfyng control ponts; usng depth maps provdes an ntutve drvng mode and lac of sngulartes but does not satsfy the mplct scene modelng requrement. We propose an approach relyng on concatenatng trlnear warpng functons that leave the scene and the camera parameters mplct, does not suffer from sngulartes, and s governed by the prevalent drvng mode used by most 3D vewers. CASCADING TENSORS The vew synthess approach s based on the followng paradgm: Three vews satsfy certan matchng constrants of a trlnear form, represented by a tensor. Thus, gven two vews n correspondence and a tensor, the correspondng thrd vew can be generated unquely by means of a warpng functon, as descrbed below n more detal. We then derve a drver functon that governs the change n tensor coeffcents as a result of movng the vrtual camera. We begn wth basc termnology; more advanced detals can be found n the Appendx..1 Notatons A pont x n the 3D proectve space 3 3 s proected onto the pont p n the D proectve space 3 by a 3 4 camera proecton matrx A = [A, v ] that satsfes p Ax, where represents equalty up to scale. The left 3 3 mnor of A, denoted by A, stands for a D proectve transformaton of some arbtrary plane (the reference plane), and the fourth column of A, denoted by v, stands for the eppole (the proecton of the center of camera 1 on the mage plane of camera ). In a calbrated settng, the D proectve transformaton s the rotatonal component of camera moton (the reference plane s at nfnty), and the eppole s the translatonal component of camera moton. Snce only relatve camera postonng can be recovered from mage measurements, the camera matrx of the frst camera poston n a sequence of postons can be represented by [I; 0].

3 AVIDAN AND SHASHUA: NOVEL VIEW SYNTHESIS BY CASCADING TRILINEAR TENSORS 95 In the case of three vews, we adopt the followng conventon: The relatonshp between the 3D and the D spaces s represented by the 3 4 matrces, [I, 0], [A, v ], and [B, v ],.e., p = [I, 0]x p [A, v ]x p [B, v ]x, where p = (x, y, 1), p = (x, y, 1), p = (x, y, 1) are matchng ponts wth mage coordnates (x, y), (x y ), (x, y ). We wll occasonally use tensor notatons as descrbed next. We use the covarant-contravarant summaton conventon: A pont s an obect whose coordnates are specfed wth superscrpts,.e., p = (p 1, p,...). These are called contravarant vectors. An element n the dual space (representng hyperplanes lnes n 3 ), s called a covarant vector and s represented by subscrpts,.e., s = (s 1, s,...). Indces repeated n covarant and contravarant forms are summed over,.e., p s = p 1 s 1 + p s p n s n. Ths s nown as a contracton. An outer-product of two 1-valence tensors (vectors), a b, s a -valence tensor (matrx) c whose, entres are a b note that, n matrx form, C = ba. A vector can be represented by ts symbol, say p, or by ts tensor form p (the range of the ndex s assumed nown by context). An element of a vector can be represented by ts desgnated symbol (f t exsts), say p = (x, y, 1), or by ts tensor form p = (p 1, p, p 3 ). Lewse, a matrx can be represented by ts symbol, say B, or by ts tensor form b, and ts elements by desgnatng values to the ndecs: b 3 s a scalar and b s the th row of B.. The Trlnear Tensor The trlnear tensor s a array of 7 entres descrbed by a blnear functon of the camera matrces A, B: 7 = v b v a, (1) where a, b are the elements of the homographes A, B, respectvely, and v, v are the eppoles of the frst mage n the second and thrd mages, respectvely (see the Appendx for dervaton). The Fundamental matrx F = [v ] x A, where [] x s the sew-symmetrc matrx defnng the cross-product operaton, can also be embedded n a trvalent tensor ) = v a v a = F l, () where F l are the elements of F and l s the cross-product tensor l u v = u v. Further detals can be found n the Appendx...1 The Trlnear Tensor for Reproecton Let s be any lne concdent wth p,.e., s p = 0, for example, the horzontal ( 1, 0, x ) and vertcal lnes (0, 1, y ) span all other lnes concdent wth p. Let r be any lne concdent wth p. Then, the tensor acts on the trplet of matchng ponts n the followng way: l Fg. 1. Each of the four trlnear equatons descrbes a matchng between a pont p n the frst vew, some lne s µ passng through the matchng pont p n the second vew and some lne r ρ passng through the matchng pont p n the thrd vew. In space, ths constrant s a meetng between a ray and two planes. µ ps r ρ 7 = 0, (3) where s µ are two arbtrary lnes (s 1 and s ) ntersectng at p, and r ρ are two arbtrary lnes ntersectng p. Snce the free ndces are µ, ρ each n the range 1,, we have four trlnear equatons (unque up to lnear combnatons), as can be seen n Fg. 1. The tensor conssts of 7 coeffcents and, snce each matchng pont contrbutes four lnearly ndependent equatons, one needs at least seven matchng ponts across three mages to lnearly recover the trlnear tensor. Once recovered, the tensor can be used for reproecton because, gven two reference mages and a tensor, the thrd mage s unquely determned and can be syntheszed by means of a warpng functon appled to the two reference mages, as follows. Let p,p be gven, then, snce p s µ 7 s a pont that concdes wth all lnes passng through p, then ps µ 7 p, (4) whch provdes a set of four equatons for p (.e., a redundant set). Ths process s referred to as reproecton n the lterature. There are alternatve ways of performng reproecton wthout recoverng a 3D model, such as by ntersectng eppolar lnes usng the Fundamental matrx [18]; however, those are senstve to degenerate stuatons (le when the three camera centers are collnear). The tensor reproecton approach s free from sngulartes of camera postons and s, therefore, a preferred choce. Comparatve studes of varous approaches for reproecton usng algebrac nvarants can be found n [7], [4], where [7] concludes that all approaches do well under favorable vewng geometry (low amount of nose and camera centers are far from beng collnear) and, n challengng stuatons, the tensor reproecton approach performs the best.

4 96 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 4, NO. 4, OCTOBER-DECEMBER 1998 Fg.. We generate tensor <1,, 4>, that relates mages 1, wth some novel mage 4, from the seed tensor <1,, 3> and the vrtual camera moton parameters [D, t ] from mage 3 to mage 4. Tensor <1,, 3> relates mages 1,, and 3 and s computed only once at the preprocessng stage. Tensor <1,, 4> s computed every tme the user specfes a new [D, t ]. We use tensor <1,, 4> to render the novel mage (mage 4) from reference mages 1,. In mage-based renderng, we would le to obtan the tensor (the warpng functon) va user specfcaton of locaton of vrtual camera, rather than by the specfcaton of (at least) seven matchng ponts. If one nows (or recovers) the full relatve orentaton (rotaton and translaton) between the frst two vews 1,, then nserton of relatve orentaton between vews 1 and the desred vew 3 n the tensor equaton (1) would provde the desred tensor for warpng vews 1, onto 3 (see, for example, [14]). One can, however, create the desred tensor wthout nowng the full moton between vews 1, by ntroducng the tensor operators descrbed next..3 The Basc Tensor Operator The basc tensor operator descrbes how to modfy (transform) a tensor so as to represent a new confguraton of three cameras. We are partcularly nterested n the case where only one camera has changed ts poston and orentaton. Thus, by repeated applcaton of the operator on a seed tensor wth a sequence of desred vrtual camera postons (translaton and orentaton), we obtan a chan of warpng functons (tensors) from the set of acqured mages (from whch the seed tensor was computed) to create the desred vrtual vews (see Fg. ). Consder four vews generated from camera matrces [I; 0], [A; v ], [B; v ], and [C; v ],.e., the 3D proectve representaton of the obect space remans unchanged or, n other words, the homography matrces A, B, C correspond to the same reference plane π (f at nfnty, then A, B, C are the rotatonal component of camera moton). The tensor of vews 1,, 3 s 7 and we denote the tensor of vews 1,, 4 as *, * = v c v a. (5) We wsh to descrbe * as a functon of 7 and the ncremental camera moton from camera 3 to 4. Let the moton parameters from camera 3 to 4 be descrbed by a 3 3 homography matrx D (from mage plane 3 to mage plane 4) and translaton t. Due to the group property of homography matrces (snce the reference plane π s fxed), C = DB and, hence, we have: * l 4 l 9 l 4 9 = v d b v a = d 7 + dl v v a and, snce t = Dv v, we have the followng result: l l * = d 7 + t a. (6) Gven a seed tensor 7 and a user specfed camera moton D, t from the last vew to the desred vew whch s compatble wth the proectve representaton of the obect space (.e., the matrx D and A are due to the same reference plane), then the formula above wll generate a new tensor * that can be used to reproect the frst two model vews (vews 1, ) onto the desred novel vew. The seed tensor can be a trlnear tensor of three vews, or the tensor embeddng ) of the Fundamental matrx. In other words, the process can start wth two model vews or wth three model vews once t has started, the subsequent tensors are of three vews (the frst two vews and the novel vews). What s left to consder s how to ensure that the homographes A and D are due to the same plane. There could be two approaches. One approach s to have the user specfy where some physcal plane seen n the two model vews should be n the novel vew. A possble algorthm can be as follows: 1) Compute the seed tensor 7. ) Accept from the user four coplanar ponts defnng some (vrtual or physcal) plane π and use them to compute the homography matrx A from the system of lnear equatons Ap p for each par of matchng ponts. 3) Accept from the user the translaton vector t and the proectons of four coplanar ponts from the plane π on the novel vew. The four ponts, are enough to construct the homography D and, as a result, recover the new tensor * from (6). 4) Use the new tensor * together wth the dense correspondence between the two model mages to synthesze (reproect) the novel vew. Ths algorthm has the advantage of avodng the need to calbrate the cameras at the expense of assumng the exstence of a plane n the 3D scene and a somewhat ndrect user nterface. Smlar methods for specfyng the novel camera poston by means of specfyng few mage control ponts were suggested by [30], [41]. The second alternatve, whch s the one we preferred, s to try and estmate the plane at nfnty,.e., the rotaton, between the two model mages (to be descrbed later). As a result, the homography matrx D becomes the rotatonal component of camera moton and the process of specfyng

5 AVIDAN AND SHASHUA: NOVEL VIEW SYNTHESIS BY CASCADING TRILINEAR TENSORS 97 the poston of the novel mage s smplfed and more ntutve for the user. Our assumptons of the nternal camera parameters are mld (the prncpal pont s assumed to be at the center of the mage and the focal length s assumed to be the mage wdth), yet the algorthm s robust enough to produce Quas-Eucldean [34] settngs whch result n plausble novel vews. To summarze, we start wth the seed tensor of the reference mages and modfy t accordng to the user specfed poston D, t of the vrtual camera poston. The modfed tensors, together wth the dense correspondence between two of the model mages, are used for renderng the novel mages, as can be graphcally seen n Fg IMPLEMENTATION To mplement the method presented n ths paper, one needs several buldng blocs: dense correspondence between a par of mages, robust recovery of the seed tensor (ether the trlnear tensor for three reference mages or the fundamental matrx, n ts tensor form, for two reference mages), a correct reproecton mechansm, handlng Occlusons. A large body of wor s devoted to the problem of fndng dense correspondence between a par of mages [3], [9], recovery of the trlnear tensor [4], [47], [17], [5], [5], [15], and recovery of the fundamental matrx [16], [3]. Any of the methods descrbed there can be used wth our algorthm. Here, we gve our mplementaton. 3.1 Dense Correspondence We obtan dense correspondence usng a Lucas-Kanade style optcal-flow [3] embedded n a herarchcal framewor [9]. For every pxel, we estmate ts moton (u, v) usng the well nown equaton: (a) (b) Fg. 3. Vew synthess for two or three reference mages. In both cases, the process s dvded nto two parts. The preprocessng stage s done only once and the actual renderng s done for every mage. (a) Three model mages. (b) Two model mages. Ix II x y IxIy I y u v = II II x t y t where I x, I y, I t are the sum of dervatves n the x, y- drectons and tme, respectvely, n a 5 5 wndow centered around the pxel. We construct a Laplacan pyramd [11] and recover the moton parameters at each level, usng the estmate of the prevous level as our ntal guess. In each level, we terate several tmes to mprove the estmaton. Ths s done by warpng the frst mage toward the second mage, usng the recovered moton parameters, and then repeatng the moton estmaton process. Typcally, we have fve levels n the pyramd and we perform two to four teratons per level. The ntal moton estmaton at the coarsest level s zero. 3. Robust Recovery of the Seed Tensor The seed tensor s recovered from the reference mages and, snce the number of reference mages may be ether two or three, slghtly dfferent algorthms are needed. We descrbe the procedure for recoverng the Fundamental matrx and nform the reader when t devates from the algorthm for recoverng the trlnear tensor. The steps for computng the Fundamental matrx/tensor are: Fnd Matchng Ponts: The method we use s a varant of Harrs corner detector [1]. For every pxel n the frst mage, we compute the optc-flow matrx Ix II x y IxIy, I y, where I x, I y are the sum of dervatves n the x and y drectons, respectvely, n a 5 5 wndow centered around the pxel, and extract ts egenvalues. We select ponts wth ther smaller egenvalue above some predefned threshold. (Usually, we set the threshold to

6 98 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 4, NO. 4, OCTOBER-DECEMBER 1998 be about seven gray-level values. Snce we sum the square values over a 5 5 wndow, we requre the smallest egenvalue to be larger than 1,5 = 7 * 5 * 5.) We trac them from the frst mage to the second mage and bac to the frst mage. (In case of three model mages, we trac the ponts from the frst mage to the thrd, va the second mage and vce-versa.) Ponts that return, at the end of the loop, to ther orgnal poston (up to dstance of one pxel) are accepted. Robust Estmaton of the Trlnear Tensor/Fundamental Matrx: The prevous stage usually produces several hundreds ponts, referred to as good ponts, whch are then normalzed to ensure good numercal soluton [3]. Ths normalzaton conssts of applyng a smlarty transformaton so that the transformed ponts are centered at the orgn and the mean dstance from the orgn s. The normalzed good ponts are used for computng the tensor n a robust statstcs manner [36] to avod the effects of outlers. The robust estmaton s done by a repettve lottery of a subset of seven good ponts (n case of three model mages) or eght good ponts (n case of two model mages), recoverng the fundamental matrx (trlnear tensor, for three model mages), and verfyng the qualty of the fundamental matrx (trlnear tensor) on the remanng set of good ponts. The qualty of the fundamental matrx s measured by the dstance of the ponts n the second mage from ther eppolar lne (for three model mages, the qualty of a tensor s measured by the dstance of the proected thrd pont and the actual thrd pont). Only ponts wth error of less than a specfed threshold (typcally, one pxel) are consdered. The best fundamental matrx (trlnear tensor) s the one wth the largest number of supportng ponts, and we compute t agan n a usual least-squared manner, usng all the ponts that supported t. (a) (b) Fg. 4. Forward mappng s decomposed nto smaller bacward mappng problems. The corners of a rectangle n the source mage are mapped to a quadrlateral n the destnaton mage and a bacward map s computed for all the pxels n the quadrlateral. (a) Source mages. (b) Destnaton mage. Fnally, n case of only two model mages, we need to convert the recovered fundamental matrx nto a tensor form, as descrbed n the Appendx. 3.3 Recoverng the Rotaton We recover the small-angles rotaton matrx between two model mages drectly from the tensor, under the assumpton that the prncpal pont s n the center of the mage and that the focal length s equal to the wdth of the mage. Ths assumptons proved to be suffcent for our method to recover a reasonable approxmaton to the correct rotaton matrx (one can use ether 7 or ) dependng on whether two or three model vews are used). where 7 Ω Ω Ω X Y Z = det = det = det K = det , (7) stands for 47, 7, 7 9, etc., and the vector Ω = (Ω X, Ω Y, Ω Z ) s the rotaton axs, and the magntude of the vector s the magntude of the rotaton around ths axs. Large angles can be recovered by teratvely computng ( 7 n + 1) ( l n ) ( l A 7 n ) ( n) ( n) l l =, where A, 7 are the recovered rotaton matrx n the prevous teraton and the prevous tensor, respectvely. Ths method was frst presented n [39] for the purpose of vdeo stablzaton. 3.4 Reproecton Process The reproecton process s performed every tme we wsh to generate a novel vew. Frst, we compute the new tensor, of the frst two model mages and the novel vew and, then, use (4), ps µ 7 p to obtan the coordnates of the pont n the novel vew, whch s a set of four equatons wth three unnowns (snce p = [ux, uy, u] T s D homogeneous coordnate) whch we solve n a least-square manner. To overcome the forward mappng problem, we splt the problem nto smaller bacward mappng tass as follows: We map rectangles of sze n n pxels n the source mages to quadrlaterals n the target mage and then compute the bacward mappng from the destnaton to the source mage [56] as can be seen f Fg. 4. Ths method gves a nce trade-off between speed and qualty by changng the sze of the rectangle. Fg. 5 demonstrates the qualty of the results for n = 1,, 5, 10. K K K

7 AVIDAN AND SHASHUA: NOVEL VIEW SYNTHESIS BY CASCADING TRILINEAR TENSORS 99 vsblty problems, as a more rgorous treatment s needed for the general case [9], nevertheless, the procedure s farly robust n practce. (a) (b) 4 EXPERIMENTS 4.1 Capturng the Images In the examples below, we followed the followng gudelnes: The two mages were taen wth a sngle camera that was movng a few centmeters between shots. The moton of the camera s desgned to avod creatng occluded areas between the mages. The obect s placed at about 50 cm from the camera and flls most of the vewng area. We found that short base-lne between the mages greatly enhances the qualty of the correspondence and that our method s robust enough to generate syntheszed mages that are far away from the orgnal vewng cone. Lghtng condtons are normal and nclude ether daylght or even fluorescent lght. The camera types used vared from the standard ndy-cam to scanned mages from a 35mm camera. (c) Fg. 5. Performng forward mappng wth rectangles of sze n n pxels. The larger n s, the faster the reproecton performs. See text for more detals. (a) n = 1, (b) n =, (c) n = 5, (d) n = Handlng Occlusons The mage-based renderng algorthm descrbed above does not handle vsblty ssues,.e., the surface s assumed transparent. It s possble to use a smple proectve Z- bufferng procedure for enforcng the constrant that the surface s opaque (cf. [30], [38], [46]). If two pxels are reproected onto the same cell, we smply choose the one close to the eppole n mage 1 correspondng to the proecton of the new center of proecton onto mage 1. The eppolar ponts can be recovered lnearly from the tensor 7 [47]. Note that we recover the eppolar ponts only for resolvng vsblty problems, not for reproecton, thus, naccuracy n the eppoles would not affect the accuracy of reproecton. It s also mportant to note that the proectve Z-bufferng s not always guaranteed to correctly resolve (d) 4. Expermental Results Wth Real Images The tensor-based renderng method was mplemented on real mages of both artfcal and natural obects. In each example, a move was created by specfyng a set of ey frames, each by a rotaton and translaton of the vrtual camera from one of the model mages. The parameters of rotaton and translaton were then lnearly nterpolated (not the mages, only the user-specfed parameters) to the desred number of frames. Also, we handled vsblty problems, to obtan better results. In all the cases presented, we assumed the prncpal pont of the camera to be at the center of the mage and the focal length to be the wdth of the mage. In the frst example, two mages of an Afrcan statue ( pxels each) were captured usng a standard ndy-cam wthout calbraton procedure or controlled lght. The obect s about 10 cm n heght and was placed some 30 cm n front of the camera. We ran the preprocessng stage, whch taes about 30 seconds on an SGI Indy machne, and then defned a set of ey frames by movng the vrtual camera n 3D space. Next, we nterpolated between the parameters of the ey frames to obtan the parameters for the entre move. We computed the novel trlnear tensor for each novel mage and then reproected the vrtual mage. The reproecton process taes about 5 seconds wth rectangle sze of 1 1 pxels for the warpng stage. Some of the generated mages can be seen n Fg. 6. We repeated the process wth two human subects. For the frst subect, the camera was at about 80 cm from the subect and the sze of the captured mages was We repeated the same procedure descrbed for the Afrcan statue and present the result n Fg. 7. For the second subect, the camera was at about 50 cm from the person and the captured mages were of sze The results can be seen n Fg 8. In another test, we have downloaded a par of mages from the CMU/VASC mage database (the mages courtesy of Hoff and Ahua [8]). No nformaton about the camera nternal or external parameters s nown or used. The mages are pxels each and an example for a syntheszed mage can be seen n Fg. 9.

8 300 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 4, NO. 4, OCTOBER-DECEMBER 1998 (a) (b) (c) (d) (e) (f) (g) (h) Fg. 6. An example of mage synthess usng optc-flow and a tensor. The orgnal mages are (a) and (e), the rest of the mages are taen from the generated move. They should be seen left to rght, top to bottom. (a) (b) (c) (d) (e) (f) Fg. 7. The orgnal two mages (a) and (d). Novel vews of the face (b), (c), (e), (f).

9 AVIDAN AND SHASHUA: NOVEL VIEW SYNTHESIS BY CASCADING TRILINEAR TENSORS (a) (b) 301 (c) Fg. 8. The orgnal mages (a), (b) are used to synthesze the novel vew (c). (a) (b) (c) Fg. 9. The orgnal mages (a), (b) are used to synthesze the novel vew (c). We have also measured the senstvty of the reproecton process to errors n correspondence. Ths was done by addng ncreasng levels of nose to the correspondence values. The nose was added ndependently to the x and ycomponents of the correspondence and was unformly dstrbuted n the range [ n, n], where n goes up to fve pxels. Fg. 11 presents the results from whch we notce a graceful degradaton of the renderng process despte the fact the dstance between the model vews s very small. In other words, the robustness of the synthess method allows us to extrapolate consderably from the orgnal vewng cone. We extended our approach to handle dynamc scene as well by treatng t as a seres of statc scenes n tme. A par of synchronzed, statonary cameras captured a flexble obect (n ths case, facal expressons). Snce the cameras are statonary, we compute the seed tensor only once. For every par of mages, we compute dense correspondence, generate a novel tensor from the seed tensor and the user specfed parameters, and reproect the novel vew. The result s a fly-through around a talng head, as can be seen n Fg. 10. Notce that all the examples contan a consderable degree of extrapolaton (.e., vews that are outsde the vewng cone of the orgnal two model vews). 5 SUMMARY We have shown the use of the trlnear tensor as a warpng functon for the purpose of novel vew synthess. The core of ths wor s the dervaton of a tensor operator that descrbes the transformaton from a gven tensor of three vews to a novel tensor of a new confguraton of three vews. Durng the entre process, no 3D model of the scene s necessary nor s t necessary to recover camera geometry or eppolar geometry of the model mages. In addton, the synthess process can start wth only two model vews and ther fundamental matrx, but the later steps follow the trlnear tensor machnery whch ensures lac of sngular confguratons and provde a natural drvng mode. Experments have demonstrated the ablty to synthesze new mages from two closely-spaced model mages, where the vewng range of the syntheszed mages far exceed the vewng cone of the model mages (extrapolaton of vewng poston, rather than nterpolaton). The lmtatons of the technque are manly concerned wth the correspondence problem. The fact that our method accepts closely spaced mages (typcally, the cameras are few centmeters apart) greatly enhances the qualty of the optcal-flow procedure. In addton, we are currently nvestgatng methods for automatcally extractng the focal length drectly from the trlnear tensor, thus removng the need to assume some predefned value for the focal length. Indeed, the vew synthess problem can be solved by means of 3D reconstructon usng the eppolar geometry of a par of mages. Bascally, there are two ssues at hand. Frst, why use the tensor formulaton rather than Fundamental matrces? Second, why not use depth maps as an ntermedate step?

10 30 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 4, NO. 4, OCTOBER-DECEMBER 1998 (a-1) (b-1) (c-1) (a-) (b-) (c-) (a-3) (b-3) (c-3) (a-4) (b-4) (c-4) Fg. 10. The left two columns show some of the facal expresson captured wth the statonary par of cameras. The rghtmost column shows some vrtual vews. The preprocess stage of computng the seed tensor was done only once for the upper par of mages. For each addtonal par, we start from the same seed tensor. Regardng the use of Tensors, the tensor formulaton on one hand does not suffer from sngular motons and generalzes the formulaton of the Fundamental matrx (see Secton 1. on the tensor embeddng of the Fundamental matrx). On the other hand, the collecton of Fundamental matrces does not always contan the full representaton of the 3D-from- D geometry (le when the camera centers are collnear [7]), and prelmnary wor on degeneracy ndcates that the famly of crtcal (ambguous) surfaces s at most a curve or a fnte set of ponts under the tensor formulaton, compared to ambguous surfaces under Fundamental matrx formulaton [45]. Furthermore, as was mentoned n the Introducton, the tensor formulaton satsfes the three requrements necessary for an on-lne synthess system, such as ntutvely smple drvng mode and sngular-free motons. Therefore, even though the complete story of the 3Dfrom-D geometry s stll unfoldng, the tensor formulaton appears as a vable alternatve for representng 3D manpulatons from collecton of vews.

11 AVIDAN AND SHASHUA: NOVEL VIEW SYNTHESIS BY CASCADING TRILINEAR TENSORS 303 (a) (b) (c) Fg. 11. We measure the senstvty of the reproecton process to errors n the correspondence by addng ncreasng levels of nose to the correspondence values. (a), (b), and (c) show an extrapolated vew of the statue wth nose levels of 0, 1.5, and 5 pxels, respectvely. Fg. 1. Reprnt from []. On the left are the four nput mages. For every expresson, we have a par of mages from dfferent vew ponts. We combne classcal technques of mage morphng to handle the nonrgd transformaton of the model and our tensor-based approach to generate novel vew ponts. Regardng the use of depth maps, we smply show that one can do wthout them and, moreover, one can do wthout explctly recoverng the camera translaton between the model vews (whch s the most error senstve component of moton estmaton). Thus, havng most of the unnecessary elements (camera translaton, depth maps) reman mplct, we stand a better chance of focusng only on the relevant elements. We have ncluded a senstvty test to demonstrate the robustness of the renderng process n the presence of added nose to the mage correspondences. The results show that, even wth the small baselne we have wored wth, the process degrades very gracefully. In addton, we beleve that, by worng n the mage doman and avodng depth maps, one can extend current D mage morphng technques wth the ad of the trlnear tensor. An example for such a combnaton appeared n [], where we combne nonrgd facal transformatons usng classc vew morphng technques wth our trlnear-tensor approach to handle rgd transformatons. For example, we show n Fg. 1 an example of combnng rgd and nonrgd transformatons n a sngle framewor. Fnally, we are currently worng [6] on an extenson of the method to handle an arbtrary number of mages by constructng a consstent camera traectory between all the model mages and usng the recovered tensor parameters for vew synthess.

12 304 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 4, NO. 4, OCTOBER-DECEMBER 1998 APPENDIX A.1 THE TRILINEAR TENSOR OF THREE IMAGES Consder a sngle pont x n space proected onto three vews wth camera matrces [I; 0], A, B wth mage ponts p, p, p, respectvely. Note that x = (x, y, 1, λ) for some scalar λ. Consder A = [A; v ], where A s the 3 3 prncple mnor of A and v s the fourth column of A. Consder p Ax and elmnate the scale factor: T a1 x a1p + λv1 x = = T T a3 x a3x + λv3 (8) T T a x a1p + λv y = =, T T a3 x a3x + λv3 (9) where a s the th row of A. These two equatons can be wrtten more compactly as follows: T T λ s v + s T Ap = 0 (10) T λ s v + s T Ap = 0, (11) where s = ( 1, 0, x) and s = (0, 1, y). Yet, n a more compact form, consder s, s as row vectors of the matrx! " $# x s µ = y, where = 1,, 3 and µ = 1,. Therefore, the compact form we obtan s descrbed below: λ µ sv psa µ + = 0, (1) where µ s a free ndex (.e., we obtan one equaton per range of µ). Smlarly, let B = [B; v ] for the thrd vew p Bx and let r ρ be the matrx,! " $# x r ρ = y. And, lewse, λ ρ r v p r ρ + b = 0, (13) where ρ = 1, s a free ndex. We can elmnate λ from (1) and (13) and obtan a new equaton: µ ρ ρ µ 4s v 94p r b 9 4r v 94p s a9 = 0, and, after groupng the common terms: µ ρ 4 9 0, ps r v b v a = and the term n parentheses s a trvalent tensor we call the trlnear tensor: 7 = v b v a.,, = 1,, 3. (14) And the tensor equatons (the 4 trlneartes) are: µ ps r ρ 7 = 0. (15) Hence, we have four trlnear equatons (note that µ, ρ = 1, ). In more explct form, these trlneartes loo le: x 7 p x x 7 p + x 7 p 7 p = 0, y 7 p y x 7 p + x 7 p 7 p = 0, x 7 p x y 7 p + y 7 p 7 p = 0, y 7 p y y 7 p + y 7 p 7 p = 0. Snce every correspondng trplet p, p, p contrbutes four lnearly ndependent equatons, then seven correspondng ponts across the three vews unquely determne (up to scale) the tensor 7. Equaton (14) was frst ntroduced n [4] and the tensor dervaton leadng to (15) was frst ntroduced n [43]. The trlnear tensor has been well-nown n dsguse n the context of Eucldean lne correspondences and was not dentfed at the tme as a tensor but as a collecton of three matrces [48], [49], [54] (a partcular contracton of the tensor nown as correlaton contracton). The ln between the two and the generalzaton to proectve space was dentfed later by Hartley [], [4]. Addtonal wor n ths area can be found n [47], [17], [53], [6], [44], [3], [4], [50]. A. Tensor Embeddng of the Fundamental Matrx We return to (14) and consder the case where the thrd mage concde wth the second. The camera matrces for both mages are A = [A; v ] and ths specal tensor can be wrtten as: = v a v a, (16) whch s composed of the elements of the fundamental matrx, as the followng lemma shows. LEMMA 1. The two-vew-tensor ) s composed of the elements of the fundamental matrx: l ) = F l where F l s the fundamental matrx and l s the crossproduct tensor. l PROOF. We consder (14) wth ) = F l to derve the followng equaltes: psr ) = l psr 4 Fl9 = l p4sr 9Fl = 0. (17) ) l p o The two-vew-tensor s an admssble tensor that embodes the fundamental matrx n a three-mageframewor. The algorthm that wors wth the trlnear tensor of three vews can wor wth ths tensor as well. Further detals can be found n [5]. ACKNOWLEDGMENTS An on-lne demo can be found on the World Wde Web at Amnon Shashua would le to acnowledge US-IS BSF contract and the European ACTS proect AC074.

13 AVIDAN AND SHASHUA: NOVEL VIEW SYNTHESIS BY CASCADING TRILINEAR TENSORS 305 REFERENCES [1] [] [3] [4] [5] [6] [7] [8] [9] E.H. Adelson and J.R. Bergen, The Plenoptc Functon and the Elements of Early Vson, Computatonal Models of Vsual Processng M. Landy and J.A. Movshon, eds., Chapter 1. Cambrdge, Mass.: MIT Press, S. Avdan, T. Evgenou, A. Shashua, and T. Poggo, Image-Based Vew Synthess By Combnng Trlnear Tensors and Learnng Technques, VRST, S. Avdan and A. Shashua, Tensoral Transfer: On the Representaton of n > 3 Vews of a 3D Scene, Proc. ARPA Image Understandng Worshop, S. Avdan and A. Shashua, Novel Vew Synthess n Tensor Space, Proc. IEEE Conf. Computer Vson and Pattern Recognton, S. Avdan and A. Shashua, Unfyng Two-Vew and Three-Vew Geometry, Proc. ARPA Image Understandng Worshop, S. Avdan and A. Shashua, Threadng Fundamental Matrces, Proc. European Conf. Computer Vson, June E.B. Barrett, P.M. Payton, and G. Gheen, Robust Algebrac Invarant Methods wth Applcatons n Geometry and Imagng, Proc. SPIE Remote Sensng, July T. Beer and S. Neely, Feature-Based Image Metamorphoss, Proc. SIGGRAPH, 199. J.R. Bergen, P. Anandan, K.J. Hanna, and R. Hngoran, Herarchcal Model-Based Moton Estmaton, Proc. European Conf. Computer Vson, 199. [10] D. Beymer, A. Shashua, and T. Poggo, Example Based Image Analyss and Synthess, Techncal Report A.I. Memo No. 1431, Artfcal Intellgence Laboratory, Massachusetts Inst. of Technology, [11] P.J. Burt and E.H. Adelson, The Laplacan Pyramd as a Compact Image Code, IEEE Trans. Comm., vol. 31, pp , [1] S.E. Chen and L. Wllams, Vew Interpolaton for Image Synthess, Proc. SIGGRAPH, [13] S.E. Chen, QucTmeVR An Image-Based Approach to Vrtual Envronment Navgaton, Proc. SIGGRAPH, [14] T. Evgenou, Image-Based Renderng Usng Algebrac Technques, master s thess, Massachusetts Inst. of Technology, June [15] O. Faugeras and T. Papadopoulo, A Nonlnear Method for Estmaton the Proectve Geometry of 3 Vews, Proc. Int l Conf. Computer Vson, [16] O.D. Faugeras, What Can Be Seen n Three Dmensons wth an Uncalbrated Stereo Rg? Proc. European Conf. Computer Vson, 199. [17] O.D. Faugeras and B. Mourran, On the Geometry and Algebra of the Pont and Lne Correspondences Between N Images, Proc. Int l Conf. Computer Vson, [18] O.D. Faugeras and L. Robert, What Can Two Images Tell Us About a Thrd One? Proc. European Conf. Computer Vson, [19] S.J. Gortler, R. Grzeszczu, R. Szels, and M. Cohen, The Lumgraph, Proc. SIGGRAPH, pp , [0] W.E.L. Grmson, Why Stereo Vson Is Not Always About 3D Reconstructon, Techncal Report A.I. Memo No. 1435, Artfcal Intellgence Laboratory, Massachusetts Inst. of Technology, [1] C. Harrs and M. Stephens, A Combned Corner and Edge Detector, Proc. Fourth Alvey Vson Conf. Intellgence, pp , [] R. Hartley, Lnes and Ponts n Three Vews A Unfed Approach, Proc. ARPA Image Understandng Worshop, [3] R. Hartley, In Defence of the 8-Pont Algorthm, Proc. Int l Conf. Computer Vson, [4] R. Hartley, A Lnear Method for Reconstructon from Lnes and Ponts, Proc. Int l Conf. Computer Vson, [5] R. Hartley, Mnmzng Algebrac Error n Geometrc Estmaton Problems, Proc. Int l Conf. Computer Vson, [6] A. Heyden, Reconstructon from Image Sequences By Means of Relatve Depths, Proc. Int l Conf. Computer Vson, [7] A. Heyden and K. Astrom, Algebrac Varetes n Multple Vew Geometry, Proc. European Conf. Computer Vson, Apr [8] W. Hoff and N. Ahua, Surfaces from Stereo: Integratng Feature Matchng, Dsparty Estmaton, and Contour Detecton, IEEE Trans. Pattern Analyss and Machne Intellgence, vol. 11, no., pp , Feb [9] S. Laveau and O. Faugeras, Orented Proectve Geometry for Computer Vson, Proc. European Conf. Computer Vson, [30] S. Laveau and O.D. Faugeras, 3-D Scene Representaton as a Collecton of Images, Proc. Int l Conf. Pattern Recognton, [31] M. Levoy and P. Hanrahan, Lght Feld Renderng, Proc. SIG- GRAPH, pp. 31-4, [3] B.D. Lucas and T. Kanade, An Iteratve Image Regstraton Technque wth an Applcaton to Stereo Vson, Proc. IJCAI, pp , Vancouver, Canada, [33] L. McMllan and G. Bshop, Plenoptc Modelng: An Image- Based Renderng System, Proc. SIGGRAPH, [34] A. Zsserman P.A. Beardsley, and D.W. Murray, Sequental Updatng of Proectve and Affne Structure from Moton, Int l J. Computer Vson, vol. 3, no. 3, pp , [35] S. Peleg and J. Herman, Panoramc Mosac by Manfold Proecton, Proc. IEEE Conf. Computer Vson and Pattern Recognton, [36] P.H.S. Torr A. Zsserman, and D. Murray, Moton Clusterng Usng the Trlnear Constrant Over Three Vews, Proc. Worshop Geometrcal Modelng and Invarants for Computer Vson, [37] T. Poggo and R. Brunell, A Novel Approach to Graphcs, Techncal Report A.I. Memo No. 1354, Artfcal Intellgence Laboratory, Massachusetts Inst. of Technology, 199. [38] M. Iran, R. Kumar, P. Anandan, J. Bergen, and K. Hanna, Representaton of Scenes from Collectons of Images, Proc. IEEE Worshop Representaton of Vsual Scenes, [39] B. Rousso, S. Avdan, A. Shashua, and S. Peleg, Robust Recovery of Camera Rotaton from Three Frames, Proc. IEEE Conf. Computer Vson and Pattern Recognton, [40] S.M. Setz and C.R. Dyer, Physcally-Vald Vew Synthess By Image Interpolaton, Proc. IEEE Worshop Representaton of Vsual Scenes, [41] S.M. Setz and C.R. Dyer, Vew Morphng, Proc. SIGGRAPH, pp. 1-30, [4] A. Shashua, Algebrac Functons for Recognton, IEEE Trans. Pattern Analyss and Machne Intellgence, vol. 17, no. 8, pp , Aug [43] A. Shashua and P. Anandan, The Generalzed Trlnear Constrants and the Uncertanty Tensor, Proc. ARPA Image Understandng Worshop, [44] A. Shashua and S. Avdan, The Ran4 Constrant n Multple Vew Geometry, Proc. European Conf. Computer Vson, Also n CIS report 950, Technon, Nov [45] A. Shashua and S.J. Mayban, Degenerate n Pont Confguratons of Three Vews: Do Crtcal Surfaces Exst? Techncal Report TR 96-19, Hebrew Unv. of Jerusalem, Nov [46] A. Shashua and S. Toelg, The Quadrc Reference Surface: Applcatons n Regsterng Vews of Complex 3D Obects, Proc. European Conf. Computer Vson, [47] A. Shashua and M. Werman, On the Trlnear Tensor of Three Perspectve Vews and Its Underlyng Geometry, Proc. Int l Conf. Computer Vson, [48] M.E. Spetsas and J. Alomonos, Structure from Moton Usng Lne Correspondences, Int l J. Computer Vson, vol. 4, no. 3, pp , [49] M.E. Spetsas and J. Alomonos, A Unfed Theory of Structure from Moton, Proc. ARPA Image Understandng Worshop, [50] G. Sten and A. Shashua, Model Based Brghtness Constrants: On Drect Estmaton of Structure, Proc. IEEE Conf. Computer Vson and Pattern Recognton, [51] R. Szels and S.B. Kang, Drect Methods for Vsual Scene Reconstructon, Proc. IEEE Worshop Representaton of Vsual Scenes, [5] P.H.S. Torr and A. Zsserman, Robust Parametrzaton and Computaton of the Trfocal Tensor, Image and Vson Computng, vol. 15, pp , [53] B. Trggs, Matchng Constrants and the Jont Image, Proc. Int l Conf. Computer Vson, [54] J. Weng, T.S. Huang, and N. Ahua, Moton and Structure from Lne Correspondences: Closed Form Soluton, Unqueness and Optmzaton, IEEE Trans. Pattern Analyss and Machne Intellgence, vol. 14, no. 3, Mar [55] T. Werner, R.D. Hersch, and V. Hlavac, Renderng Real-World Obects Usng Vew Interpolaton, Proc. Int l Conf. Computer Vson, [56] G. Wolberg, Dgtal Image Warpng. Los Alamtos, Calf.: IEEE CS Press, 199.

14 306 IEEE TRANSACTIONS ON VISUALIZATION AND COMPUTER GRAPHICS, VOL. 4, NO. 4, OCTOBER-DECEMBER 1998 Sha Avdan receved the BSc degree n mathematcs and computer scence from Bar- Ilan Unversty, Ramat-Gan, Israel, n Currently, he s a PhD canddate at the Hebrew Unversty, where hs research nterests are n computer vson and computer graphcs. In addton, he has been worng for the past 10 years n the ndustry n the felds of CAD, GIS, and photogrammetry. Amnon Shashua receved the BSc degree n mathematcs and computer scence from Tel-Avv Unversty, Tel-Avv, Israel, n 1986; the MSc degree n mathematcs and computer scence from the Wezmann Insttute of Scence, Rehovot, Israel, n 1989; and the PhD degree n computatonal neuroscence, worng at the Artfcal Intellgence Laboratory, from the Massachusetts Insttute of Technology, n 1993.Dr. Shashua s a senor lecturer at the Insttute of Computer Scence, The Hebrew Unversty of Jerusalem, Hs research nterests are n computer vson and computatonal modelng of human vson. Hs prevous wor ncludes early vsual processng of salency and groupng mechansms, vsual recognton, mage synthess for anmaton and graphcs, and theory of computer vson n the areas of three-dmensonal processng from a collecton of two-dmensonal vews.