of Multiple Cameras Abstract This paper addresses the problem of calibrating camera parameters using variational methods.

Transcription

1 A Varatonal Approach to Problems n Calbraton 1 of Multple Cameras Gozde Unal 1, Anthony Yezz 2, Stefano Soatto 3, and Greg Slabaugh 1 Abstract Ths paper addresses the problem of calbratng camera parameters usng varatonal methods. One problem addressed s the severe lens dstorton n low cost cameras. For many computer vson algorthms amng at reconstructng relable representatons of 3D scenes, the camera dstorton effects wll lead to naccurate 3D reconstructons and geometrcal measurements f not accounted for. A second problem s the color calbraton problem caused by varatons n camera responses that result n dfferent color measurements and affects the algorthms that depend on these measurements. We also address the extrnsc camera calbraton that estmates relatve poses and orentatons of multple cameras n the system, and the ntrnsc camera calbraton that estmates focal lengths and the skew parameters of the cameras. To address these calbraton problems, we present mult-vew stereo technques based on varatonal methods that utlze partal and ordnary dfferental equatons. Our approach can also be consdered as a coordnated refnement of camera calbraton parameters. To reduce computatonal complexty of such algorthms, we utlze pror knowledge on the calbraton object, makng a pecewse smooth surface assumpton, and evolve the pose, orentaton, and scale parameters of such a 3D model object wthout requrng a 2D feature extracton from camera vews. We derve the evoluton equatons for the dstorton coeffcents, the color calbraton parameters, the extrnsc and ntrnsc parameters of the cameras, and present expermental results. Index Terms calbraton, varatonal methods, color calbraton, lens dstorton calbraton, camera parameters refnement. 1 Semens Corporate Research, Prnceton NJ 854. Correspondng author: gozde.unal@semens.com 2 School of Electrcal Engneerng, Georga Insttute of Technology, Atlanta GA Computer Scence Department, Unversty of Calforna, Los Angeles, CA 995.

2 2 I. INTRODUCTION The problem of recoverng a 3D representaton of a scene from multple 2D mages has been one of the man research nterests n computer vson. Many of the exstng stereo technques nvolve preprocessng the camera mages to extract 2D features such as corners, lnes, and contours of objects n the scene. These features are then used to fnd correspondences between camera vews. In practce, searchng for features and establshng correspondences s not an easy task due to nose and local extrema. Early varatonal approaches to the 3D reconstructon problem were poneered by Faugeras et.al. [1] who also reled on local feature matchng. A more recent varatonal approach by Yezz and Soatto [2, 3] proposed a jont regon-based mage segmentaton and smultaneous 3D stereo reconstructon technque. Ths paper addresses camera calbraton technques bult on ths later stereo reconstructon framework that avods searches for local correspondences and s versatle enough to accommodate the new applcatons to be shown. A tradeoff s acheved by makng a pecewse smooth object assumpton and a constant background assumpton, however, extracton of 2D features from gven camera vews are not requred. Camera calbraton refers to the problem of fndng the mappng between the 3D world and the camera or mage plane. For most computer vson algorthms amed at reconstructng relable dgtal representatons of 3D scenes, accurate camera calbratons are essental. There has been a great deal of research on camera calbraton problem as early as n 7 s [4]. In most of the prevous technques, some set of features are extracted from mages of a known calbraton pattern, and ntrnsc camera parameters as well as camera pose and orentaton (extrnsc camera parameters) are estmated by a mnmzaton of an overall cost functonal [5 13]. Many calbraton technques use both nonlnear mnmzaton and closed form solutons as n [14]. In ths paper, we develop a coordnated refnement technque for the extrnsc camera parameters, ntrnsc camera parameters: lens dstorton, focal lengths, skew, and also estmaton of camera color

3 3 calbraton parameters n a coupled way wthn a multple camera system 1. For geometrcal measurements, an ntrnsc camera parameter, the camera lens dstorton, s an mportant ssue, and wll result n naccurate 3D reconstructons f not taken nto account. Another common problem n mult-vew stereo technques s caused by color mscalbratons between cameras due to dfferent sensor characterstcs. Extrnsc parameters of the cameras on the other hand, determne the relatve poses and orentatons of cameras, and ther correct estmaton s one of the frst phases of a camera calbraton system. A. Relaton to Prevous Work and Contrbutons 1) Lens Dstorton: The deal pnhole camera model leads to magng of world lnes as lnes on the mage plane, and smplfes many computatons and consderatons [6]. However, for most real cameras wth wde-angle or nexpensve lenses ths assumpton does not hold, and nonlneartes ntroduced by a well-known phenomenon referred to as a lens dstorton should be taken nto account. The correspondng dstorton parameters should be estmated for each camera. In many exstng calbraton technques, good estmates for extrnsc and ntrnsc camera parameters are frst obtaned by a pnhole camera model neglectng lens dstorton. Then dstorton calbraton s performed whle holdng the other parameters fxed [17 19]. Ths s possble because the mappng from 3D world coordnates to the 2D mage plane can be decomposed nto a perspectve projecton and a mappng that models the devatons from the deal pnhole camera. A popular group of lens dstorton calbraton methods n the lterature, manly under the category known as plumb lne methods, rely on a frst step of extractng edges from the mages. Ether a user manually selects the mage curves or there must be a way to relably estmate mage edges whch correspond to lnear 3D segments n the world. An optmzaton problem s set up by defnng a measure of how much each detected segment s dstorted. The curved lnes n the mage whch do not really correspond to 3D lne segments wll consttute outlers n ths optmzaton procedure [17, 2 23]. Other technques such 1 An ntal verson of ths work that addresses lens dstorton and color calbraton can be found n [15] and [16], and an ntal work addressng extrnsc camera calbraton appears n [3].

4 4 as [24] rely on pont correspondences. Gven a set of 3D ponts, the assocated eppolar and trlnear constrants are arranged nto a tensor, whch s computed wth estmated dstorton parameters at each step to mnmze a reprojecton error n an teratve manner. In another group of methods as n [25 27], a drect soluton strategy s employed to fnd camera calbraton parameters by ncorporatng lens dstorton as well. Our contrbuton s a new dstorton calbraton technque that does not rely on extracton of edges and search for pont correspondences. The former may not be an easy task due to nose and local extrema. Instead, we devse an ntegrated calbraton technque n whch the dstorton parameters of cameras are computed n a tghtly coupled framework. The desred couplng of multple camera vews comes from estmatng a common 3D object (n ths case the calbraton object). In other words, we mnmze the cost between the reprojecton of the 3D calbraton object and the mage measurements by evolvng the dstorton parameters of the cameras. In our dstorton calbraton algorthm, we use a whte bar object, made from a foam core as shown n Fg.1 on the left. We capture ts vews before a dark background wth the mult-vew stereo rg system, a desktop mult-camera system desgned for remote multmeda collaboraton, developed by HP Labs [28]. The mages of the calbraton object captured from three of the fve cameras n the rg are gven n Fg.1. Many desktop mult-camera systems use wde angle and nexpensve cameras whch produce severe dstorton effects as can be observed n the gven mages. Fg. 1. Three out of fve camera vews of the real calbraton object shown on the left. As we wll show, wth ths technque we can also ncorporate other parameters of calbraton nto the same varatonal framework and get ther locally optmal estmates as well. 2) Color Calbraton: Another common problem n mult-vew stereo technques s caused by color mscalbratons between cameras resultng from varatons n camera responses due to dfferent sensor

5 5 characterstcs, ambent condtons lke temperature, manufacturng dfferences, and so on. These yeld dfferent color measurements between cameras, and affect the algorthms that depend on these measurements. Camera color calbraton refers to the problem of estmatng the color calbraton parameters of cameras to overcome these unwanted effects. A common approach taken toward ths problem s to calbrate each camera ndependently through comparsons wth known colors on a color calbraton object/envronment [28, 29]. The color calbraton object we use, shown n Fg.2 s a color cube wth patches of known colors whose Fg. 2. Photograph of the color calbraton object. mages are captured from each camera. Demosacng coeffcents are calculated ndependently for each camera based upon the absolute colors of the calbraton object and the measured color responses of each camera. Slght errors and dfferences that arse from ths ndependent calbraton procedure sometmes lead to notceable seams or dscontnutes n the texture mappng process durng the transton of the texture map between neghborng cameras. Our goal s to help even out these dscrepances by devsng a relatve nter-camera color calbraton technque n whch the demosacng parameters of cameras are calculated jontly n a tghtly coupled framework rather than just one camera at a tme. Smlar to our approach to lens dstorton calbraton, the desred couplng of the multple camera vews comes from estmatng a common 3D shape, and n addton a common radance functon for the calbraton object (n ths case, the color cube). We take advantage of the fact that the object shape s known up to locaton and scale to smplfy the problem. Hence, we estmate the pose parameters of the cube, the radance functon on the cube, and the color calbraton coeffcents for each camera. 3) Extrnsc and Intrnsc Calbraton: Followng the same phlosophy as mentoned n the other two calbraton problems above, extrnsc and ntrnsc calbraton parameters can be estmated n a varatonal framework usng the general stereoscopc framework of Yezz-Soatto.

6 6 It should be noted that due to dfferental nature of the estmaton equatons derved, the extrnsc and ntrnsc update equatons requre rough ntal values. Ths s a well-known feature of almost all of the recent state-of-the-art energy functonals used n segmentaton (e.g., Mumford-Shah energy, geodesc energy,...),.e., the solutons are locally optmal, hence startng far away from the real soluton may lead to solutons that get stuck at local extrema far from the desred soluton. Nevertheless, the usefulness of a refnement stage n extrnsc and ntrnsc camera parameters wll be demonstrated va the mprovement n the fnal 3D reconstructons. A nce feature of the methodology presented n ths paper s that t can ntegrate several dfferent problems n geometrc and color calbraton nto an overall unfed system based on the jont segmentaton framework to evolve pose, color, dstorton, extrnscs, and ntrnscs. The organzaton of ths paper s as follows. We frst present a varant of the Yezz-Soatto algorthm n whch a 3D object s allowed to move wth a sem-affne moton model n Secton II. We developed ths scheme for our applcatons n calbraton, where the 3D object shape s roughly known (up to 3 scales and rgdty) to obtan more effcent and faster algorthms. We then present a novel technque for lens dstorton calbraton n Secton III and a novel technque for relatve nter-camera color calbraton n Secton IV. We apply the same calbraton deas for ntrnsc camera calbraton n Secton V, and for extrnsc camera calbraton problem n Secton VI. Conclusons and dscussons are gven n Secton VII. II. EVOLUTION EQUATIONS OF 3D OBJECT MOTION PARAMETERS The Yezz-Soatto 3D stereo reconstructon model bulds a cost on the dscrepancy between the reprojecton of a model surface wth a radance f : R 3 R, the background (nfntely far away) wth radance b : R 3 R, and the actual measurements from multple camera vews. Let g denote the transformaton from world coordnates to camera coordnates: g : X X = (X, Y, Z ) T, and π denote the perspectve transformaton from camera frame to the mage plane: π : X x = (x = X Z, y = Y Z ) T. On the mage plane, the cost functonal for the Yezz-Soatto model can be wrtten as a jont segmentaton problem over regons of n camera mages I wth doman Ω = R R c (R denotes the foreground regon),

7 7 and wth 3 color channels k (R, G, B): E = n [f k ((π g ) 1 (x )) I k (x )] 2 dω + R =1 n [b k I k ]2 dω (1) =1 R c Ths energy can be lfted back onto surface S : E(S) = =1 n [(f k (X) I k (π g (X))) 2 (b k I k )2 ] X (X)σ(X )da, (2) S where σ s the Jacoban of the change of coordnates from the mage plane to the surface, X s the vsblty functon of a voxel on the surface, and da s the area measure of surface S. The deformaton of the surface S w.r.t. ths energy or data fdelty measure s then obtaned by fndng the partal dfferental equaton (PDE) that s the gradent descent flow of the energy E. A popular class of numercal technques known as Level Sets Methods [3], s utlzed to evolve the surface S va the evoluton of a 3D functon Ψ : R 3 R. Nevertheless, an update of the level set functon s requred after each teraton of the assocated PDE, and even wth more effcent narrowband schemes [31], there s a consderable amount of computaton nvolved. For our ntended applcatons, n whch there s a calbraton object whose shape can be roughly known a pror, rather than deformng the surface of the 3D object, we wll evolve ts pose and scale parameters nstead. Next, usng the energy E n Eq. 2 we wll derve the ordnary dfferental equatons (ODEs) to update the parameters of the surface moton modeled by a sem-affne transformaton, whch s more general than a smlarty but less general than a fully affne transformaton. Let the orgnal rgd surface be denoted by S o, then S = g s (S o ), or X = g s (X o ) = R s X o +T s, and let λ denote parameters of the rgd moton g s of the surface S o wth rotaton R s and translaton T s. Then the gradent of the energy E w.r.t. λ s gven by: E(λ) λ = = S (X) < X k F,N > da + (X) λ λ F k F k S o da (gs (X o )) < (gs X o ),R s N o > da o λ + 2(f k (gx o ) I k ) < (gs X o ), fs k λ > da o (3)

8 8 where F k = [(f k (X) I k (π g (X))) 2 (b k I k )2 ] X (X)σ(X ) s the Mumford-Shah term from Eq. 2 (also n [2]). The dervaton follows from shape optmzaton tools [32] that provde the shape dervatves n curve and surface evoluton framework. N denotes the surface normal vector. Note that the vsblty functon X (g s (X o )), ncluded n the data term F k ( ) s computed usng the orgnal vsblty functon but compensated by R T s (C T s ) ), where C s a camera center. The second term n Eq.(3) s the regon term correspondng to the foreground object whereas the frst one s the boundary term. In our applcatons, the background s modeled by a pecewse constant radance, therefore we omt the background regon term n the equaton. For translaton parameters: < (gs X o ),R s N o >= R s N o. λ For rotaton parameters: Z o Y o < (gs X o ),R s N o >=< R s λ Z o X o,r s N o >=< R s ˆXo,R s N o >, (4) Y o Xo } {{ } ˆX o where we utlze exponental coordnates (see [33] for detals on ths representaton) for the global rotaton parameters of the surface. We note that a matrx n an nner product expresson, when operated on a vector, wll ncorporate each of ts row vectors n the nner product to result n a vector: < x 1,x 2,...,x n,y >= (< x 1,y >, < x 2,y >,..., < x n,y >). For further flexblty n ntalzng a model surface, we add three scalng parameters along the X, Y, and Z axes. Then the sem-affne transformaton for a pont X o on the surface becomes: X = g s (X o ) = s x RSX o + T, where S = s y. The gradent of the energy w.r.t. the scalng parameters λ = s j s s z derved smlarly to the above: F k S o (gs (X o )) < (gs X o ),R s N o > da o λ

9 9 where < (gs X o ) S,R s N o > = < R s λ λ X o,r s N o > wth e.g. 1 X o.x (g s X o ) S = R s X o = R s s 1 s 1 X o = R s < (gs X o ),R s N o > = < R s λ X o Y o Z o } {{ } R X s, R s N o >. (5) The evolutons for the rgd moton parameters λ are then gven by the gradent descent equatons: λ t = E λ = λ t = E λ = λ t = E λ = S o F k (g s (X o )) } {{ } F k R s N o da o, (for translaton). (6) S o F k < R s ˆXo,R s N o > da o, (for rotaton). (7) S o F k < R X s,r s N o > da o, (for scalng). (8) Here note that the vsblty functon X (g s (X o )) s computed usng the orgnal vsblty functon but 1/s x compensated by the S 1 R T s (C T s ) ), where S 1 = 1/s y. Note that we can generalze 1/s z ths dea n a straghtforward fashon by consderng S to be more general than a smple dagonal matrx n order to accommodate a fully affne moton of the surface. We wll use equatons (6-7-8) n updatng the pose of the surface S to estmate ts correct placement n the 3D space for the calbraton applcatons presented n Sectons III, and IV.

10 1 III. LENS DISTORTION CALIBRATION The lens dstorton s usually modeled by a functon defned from the deal mage plane to the dstorted mage plane. One approach s to decompose t nto two terms: radal and tangental dstorton [17]. The radal dstorton s a deformaton along the radal drecton from a center of dstorton pont to an mage pont, and the tangental dstorton s a deformaton n a drecton perpendcular to the radal drecton, and s neglgble for many cameras. To model the radal dstorton effects, a commonly used dstorton functon D(r) s gven by (1 + k 1 r 2 + k 2 r ) where r s the radus from the center of dstorton to a pont on the deal mage plane. The prncpal pont (u, v ) s often used as the center for radal dstorton [6], whch we wll also adopt. Below ˆx s the dstorted mage coordnates, and D s the dstort functon: ˆx = Dx = (1 + k 1r 2 + k 2r )x, (9) r 2 = (x 2 + y 2 ), and k j s the j th dstorton coeffcent for camera. In Eq. 9, we assume that k = 1, whch can be changed to an arbtrary k value. A. Calbraton of the Lens Dstorton Parameters Notaton: World X ßÞÐ g X ßÞÐ π ¼ x = X Z = x Y Z = y 1 ½ ¼ ßÞÐ L u u (u, v) (mage coordnates), where D s the ½ L v v 1 dstort functon n Eq. 9, and L u and L v are the focal lengths. The gradent of the energy (1), assumng a sngle mage channel over the dstorted mage plane, w.r.t. dstorton parameters kj s gven by kj t = E = F kj ((D π g ) 1ˆx ) < ˆx, ˆn ĉ kj > dŝ (1) where F = (f I ) 2 (b I ) 2, subscrpt corresponds to each camera vew, and ˆn denotes the normal vector to the occludng boundary ĉ of regon R on the dstorted mage plane. We only consder the boundary term (ŝ s the arclength of the contour ĉ on the mage plane: the dstorted or actual mage coordnates) as we assume the foreground and background have constant radance. We desgn the lens dstorton calbraton object to satsfy ths assumpton.

11 11 We want to lft ths ntegral back onto occludng boundary C of the surface. Note that ˆx k j are gven by ˆx k 1 = r 2 x, ˆx k 2 = r 4 x,... ˆx k j = r 2j x hence < ˆx, ˆn kj > dŝ =< r 2j π(x ), J s (D π)x > ds =< r 2j π(x ), JD π s X > ds (11) where J denotes the 2 2 nnety degree rotaton matrx, D = (1 + k 1 r 2 + k 2 r ), and π = 1 Z 2 Z X Z Y s the Jacoban of the perspectve projecton π. We can contnue to smplfy: < ˆx kj, ˆn > dŝ = r 2j D < [π(x )] 2 1, = r2j D Z 2 = r2j D Z 3 X 1 Z 2 < 1 Z Y, Z Y Z X Z X < Z Y Y X Z Y Notng that Z X = X < ˆx, ˆn kj > dŝ = r2j D Z 3 X Y (12) 1 Z X 1 Z Y [ ] X > ds s 3 1, X s > ds = r2j D Z 3 < Z Y Z X, we have < ˆx k j, ˆn > dŝ = r2j D < Z 3 < X X s, X Y [ ] X > ds s 3 1, X s > ds X Y > ds = r2j D < X Z 3 N, X, X s X Y > ds, and > ds. (13) Substtutng Eq.(13) nto Eq.(1), we get the calbraton equaton kj t = r 2j D X F C Z 3 < N, X Y > ds (14)

12 12 for the lens dstorton parameters kj. Note that the dstorton calbraton method we propose can handle dfferent models of dstorton by changng the D functon, and related dervatves n Eq.(14). B. Usng Several Poses of the Object When camera vews from multple poses of the object are avalable, we can take advantage of the exstence of varously dstorted vews n calbratng the lens dstorton. In the frst phase, we estmate both pose and dstorton coeffcents from separate experments. To smplfy the explanaton, let us assume that we want to solve for only one dstorton coeffcent k 1 for each camera. Once we obtan rough estmates for the object pose and dstorton coeffcents k 1, we can fuse a common dstorton k 1 from these separate experments for each camera and then jontly evolve k 1 s as follows: k 1 Mposes t = m=1 C,m F,m r 2j D X,m Z 3,m < N,m, X,m Y,m > ds. (15) At the same tme, we evolve the pose parameters of separate poses of the object as descrbed n Secton II, the only dfference beng the ncorporaton of the new common dstorton n the equatons. For nstance, we evolve any of them for a gven pose as follows: λ t = F (g s (X o )) < (gs X o ) S o λ,r s N o > da o (16) where F ncludes computaton of I (D π g (g s (X o ))) wth the new common dstorton coeffcents k 1 n the multplyng dstorton factor D. C. Expermental Results For our calbraton algorthm, we ntalze a surface model of the real calbraton object whch s shown from several vantage ponts n Fg.3. After ntalzng the surface, the frst phase of our algorthm s to evolve ts pose parameters to poston the 3D object model roughly n the correct locaton n 3D space. For the experments presented here captured va HP Labs stereo rg system, we used three dfferent poses of the calbraton object, but we can ncrease the number of poses used n the process. Example evolutons

13 13 of the pose parameters are shown n Fg.4, for three dfferent pose captures of the calbraton object n each column (showng only one camera vew for each pose). The dstorton coeffcents are also evolved at a slower pace. That s, the tme step used n the assocated ODE s small n the frst phase. In the experments, the dstorton functon D n (9) wth one dstorton coeffcent k 1 for each camera s used. After the separate evolutons for each of the poses have converged, common ntal dstorton coeffcents are computed as the average of the results from phase 1. In the second phase of the algorthm, we evolve the dstorton coeffcents for each camera agan separately but summed over dfferent poses. We show sample vews of pose 1, 2, and 3 n row 1 of Fg.s As the dstorton coeffcents converge to true values, the reprojecton of the surfaces onto the dstorted vews results n a better match to the mage data and contnues to mnmze the overall energy. Such mages wth reprojectons are shown on the second row of Fgures The undstorted vews shown as well on the thrd row. The straghtenng effect of ths operaton on the curved lnes can be clearly observed n these mages. IV. COLOR CALIBRATION For color calbraton, the dfferences n absolute colors measured n the response of each camera are modeled by a smple multplcatve factor n each of color RGB channel measurements and an addtve offset parameter. The frst varaton of our energy functonal E usng ths model leads to gradent descent flows: E = [f k (α,k I k α +β,k)]i k dω [b k (α,k I k +β,k)]i k dω, (17),k R R c E = [f k (α,k I k +β,k )]dω [b k (α,k I k +β,k )]dω. (18) β,k R for the color calbraton parameters α,k and β,k for each camera, and k {R, G, B}, where I k, fk, and b k are from one of the three color channels {R, G, B}. Note that one can extend ths framework to RGGB mages n a straghtforward fashon. In our test calbraton experments, we utlzed whte nose addtve offsets and multplcatve scalng coeffcents to perturb the measured mages, thereby exaggeratng the effect of color mscalbratons. On R c

14 14 a synthetcally created example n Fg.8, where the correct geometry and radance functon are known, we show such mscalbraton effects on the orgnal vews, and vews durng the evoluton of α s and β s n Eq.s (17-18), and vews after these parameters have converged. In addton, n Fg.9, the curves depct the true α and β values for all nne camera vews, and the convergence of the estmated parameters towards the real values. Smlarly n Fgure 1, the color cube wth orgnal colors are shown from some camera vews frst, then shown after ther color calbraton parameters are perturbed. Fnally, the convergence of the color parameters results n a corrected set of colors as shown n the vews. Also shown n Fgure 11 are the evolutons of the color calbraton parameters for the shown vews. We have to note here agan that due to relatve calbraton framework among cameras, the updated parameters may not always result n absolute values but stll provde useful outputs for the mult-camera systems. V. INTRINSIC PARAMETER CALIBRATION We show the evoluton of three of the man ntrnsc camera parameters: focal lengths, denoted by L u and L v for each of the coordnates on the mage plane, and the skew parameter a. Incluson of the skew parameter between the two plane coordnates leads to an ntrnscs matrx of the form L u a u π = L v v, 1 then the Jacoban of the perspectve transformaton becomes (compare to Eq. 12): π = L u /Z a/z L u X /Z 2 ay /Z 2 L v /Z L v Y /Z 2 = 1 Z 2 L u Z az L u X ay L v Z L v Y The dervatves of the mage coordnates w.r.t. each of the ntrnsc parameters are computed from the overall energy functonal as before (smlar to our dervatons of the lens dstorton calbraton parameters. n Secton III): E L j = [(f k I k )2 (b k I k x )2 ],n C }{{} ds. L j F k

15 15 For the focal length parameter L u, we have < x L u,n > ds = = 1 Z 3 Notng that Z X = X Y X πc, L u s Jπ C ds L u X /Z + ay /Z = L u L u Z X L v Z az L v Y L u X + ay < x L u,n > ds = 1 Z 3 X = L v Z 3, 1 Z 2, X s > ds = 1 Z 3 L v Z L v Y L u Z az L u X + ay < L v Z X Y X, X s, then for the focal length parameter L u we obtan: < L v X < X X s, X, X s > ds X > ds = L v X Z 3 < N, X [ ] X ds (19) s ds. > ds. (2) Due to the skew parameter, the equatons for the second focal length parameter L v wll be slghtly dfferent. Ths tme ncorporatng the dervatve w.r.t. L v n Eq. 19 : L u Z < x,n > ds = 1 L v Z 3 < L v Z az, X s > ds = 1 Z 3 < Y L v Y L u X + ay Agan notng that L u Y Z az Y L u X Y + ay 2 L u Z Y az Y L u X Y + ay 2, X s > ds ay = X L u Y, then for the focal length parameter L v, we have: ay < N, L u Y > ds. (21) < x L v,n > ds = X Z 3

16 16 Note that when the skew parameter a s, whch s a wdely used conventon, the above equaton reduces to a symmetrc form of the Eq. 2 derved for L u. Fnally, we derve smlarly the update equatons for the skew parameter a: L u Z < x a,n > ds = 1 Y Z 3 < L v Z az, X s >= 1 Z 3 < L v Z Y L v Y L u X + ay Ths tme notng that Z Y = X Y 2 Y < x a, n > ds = L v X Z 3 Y 2, then for the skew parameter a, we have: < N, Y, X s > ds. > ds. (22) The fnal evoluton equatons for the three ntrnsc parameters for each camera are then gven by: L u, t L v, t a t = = = C F k L v Z 3 C F k 1 Z 3 C F k L v Z 3 < X N, X > ds (23) ay < X N, L u Y > ds (24) < X N, Y > ds (25) In Fgure 12 a synthetc color cube example s shown. The ntrnsc parameters, focal lengths L u, and L v, are ntalzed to perturbed values and when the ntrnsc calbraton update equatons have converged, both the projectons of the cube surface onto the mages and the evoluton of the focal lengths are shown.

17 17 VI. EXTRINSIC CAMERA CALIBRATION We now consder the same energy functonal as a functon of the extrnsc calbraton parameters Λ = (λ 1,...,λ 6 ) for each camera mage I. Notce that the only term n our energy functonal E whch depends upon Λ s the correspondng fdelty term n E data (due to the dependence of π 1 ), assumng a constant background radance n the scene : E data, (S, f, b, Λ ) = R ( f k (π 1 (ˆx)) I k (ˆx))2 dω + R c ( b k I k (ˆx))2 dω, (26) where ˆx denotes mage coordnates as before (for smplcty of dscusson, dstorton D = 1). A. Intal expresson of gradent If we let ĉ = R denote the boundary of R then we may express the partal dervatve of E wth respect to one of the calbraton parameters λ j as follows. E λ j = boundary term + foreground term ( (f = k (π 1 (ˆx)) I k (ˆx))2 ( ) b k I k (ˆx)) 2 ĉ, ˆn dŝ ĉ λ j ( + 2 f k (π 1 (ˆx)) I k (ˆx)) S f k( π 1 (ˆx) ), π 1 (ˆx) dω (27) R λ j In the boundary term, dŝ denotes the arclength measure of ĉ, and ˆn denotes ts outward unt normal. In the foreground term, S denotes the natural gradent operator on the surface S. B. Rewrtng the boundary term Ultmately, we wll compute all quanttes by ntegratng along the current estmate of the surface snce that s the actual object represented by our data structures. Thus, t s more convenent to express the contour ntegral around ĉ (ŝ) n the mage plane as a contour ntegral around C (s) on the surface S

18 18 nstead, (where π (C )=ĉ and where s s the arclength parameter of C ). They may be related as follows. ĉ, ˆn dŝ = π (C ), λ j λ j s Jπ 1 (C ) ds, where J = 1 = = 1 Z 3 1 Z 2 X s, = 1 X Z 3 = X Z 3 Z X Z Y x, λ j 1 Z 2 Z Y Z X Y X s, X X ds = 1 λ j Z 3 X λ j,n ds X λ j Z Y Z X ds X,X X ds λ j s X s (snce X and X s are perpendcular tangent vectors to S) Thus, the boundary term wrtten as an ntegral on the surface S (along the occludng contour C ) has ds the followng form: λ j t = C ( (f ) k I k 2 ( b k I k ) ) 2 X g,n Z 3 ds, (28) λ j whch s also the update equatons for the extrnsc parameter j for camera wth a pecewse constant assumpton on the foreground and the background radance. C. Rewrtng the foreground term The frst step n rewrtng the foreground/background ntegrals s to re-express the dervatve of the back-projected 3D pont X = π 1 (ˆx, Λ ) wth respect to the calbraton parameter λ j n terms of the dervatve of the forward projecton π (x, Λ ) = π(g (X, Λ )), snce π has an analytc form whle π 1 does not. We begn by fxng a 2D mage pont ˆx and note that ( ) ( ( ) ) ˆx = π X(ˆx, Λ ), Λ where X(ˆx, Λ ) = π 1 (ˆx, Λ ) = g 1 π 1 (ˆx), Λ

19 19 and thus dfferentaton wth respect to λ j yelds: = ( ) π X π X, Λ = + π λ j X λ j λ j = 1 Z 2 Z Z X Z Y g X X + 1 λ j Z 2 Z g X = X λ j Z Y Z Z X Z Y X Z Y g λ j g (29) λ j Notce, though, that (29) does not unquely specfy X/ λ j but merely gves a necessary condton. We must supplement (29) wth the addtonal constrant that X/ λ j must be orthogonal to the unt normal N of S at the pont X n order to obtan a unque soluton. X λ j N = ( or equvalently g ) X N = X λ j (3) Now, combnng equatons (29) and (3), we have Z X Z X g X g Z Y = X λ j Z Y λ j N x N y N z 1 ( ) Z X Z X 1 X g g = λ j X Z Y Z Y λ j N x N y N z X = ( g ) 1 Z N z + Y N y X N y X z Z X X g λ j Z (X N ) Y N x Z N z + X N x Y z Z Y λ j Z N x Z N y Z z X = ( g ) 1 X N X N x X N y X N z X g λ j X N Y N x X N Y N y Y N z λ j Z N x Z N y X N Z N z ( ) 1 ( X g = I X ) N g, (31) λ j X X N λ j

20 2 where s the Kronecker product, and I s the 3 3 dentty matrx. The second step proceeds n the same manner as outlned earler n rewrtng the data fdelty terms n E data by notng that the measure n the mage doman dω and the area measure on the surface da are related by dω = σ(x,n ) da where σ(x,n ) = (X N )/Z 3. Then the foreground term n Eq.(27) s gven by = ( ) 2 f k I k S f k( π 1 (ˆx) ), R 2 π 1 (R ) ( f k I k ) S f k (X), X λ j π 1 (ˆx) dω λ j X N da (32) Z 3 Therefore, the followng foreground term wll be added to the update equaton of the extrnsc parameter n Eq.(28) : λ j t = 2 π 1 (R ) ( ) f k I k S f k (x), Z 3 ( ) 1 ( g (X N ) g ( ) ) g N X da.(33) X λ j λ j In Fgure 13, several photos from a set of 32 mages of a toy skater doll are shown. When the ntal extrnsc parameters are off as observed n the projectons of the foreground object onto the mages (shown by an orange mask), a vsual hull created usng the uncorrected extrnsc camera parameters s sgnfcantly away from the real doll surface. After the extrnsc calbraton equatons (28) plus (33) are evolved to convergence, vsual hull created usng the updated extrnsc parameters demonstrates the correcton and true refnement provded by the derved equatons. In Fgure 14, we depct the extrnsc refnement stablty by showng the uncertanty ellpsods drawn around each camera center. Parameters were perturbed n x, y, z drectons randomly several tmes, and converged properly for varatons up to 8%. VII. RESULTS AND CONCLUSIONS The toy skater example shown n Fgure 15 demonstrates the smultaneous evoluton of the extrnsc and ntrnsc parameters for the 32 cameras, along wth the projectons of the foreground surface. The vsual hulls created wth agan the ntal set of camera parameters and the evolved set of camera parameters dsplay a correct refnement of the camera parameters.

21 21 For most of the experments we utlzed a volume, and a volume for the Bust dataset. Wth a volumetrc sgned dstance representaton n our C++ mplementaton wthout any code optmzaton on a Pentum 2.4 GHz processor, each sngle teraton to compute all calbraton gradents takes on the order of 1 seconds dependng on the number of camera vews as well, and convergence takes about 5-4 teratons dependng on the ntalzaton, hence a computaton tme of about 8-6 mnutes. However, a mesh representaton on the object may be easer to work wth snce the parameter update equatons we derved are ordnary dfferental equatons. A common ssue for any calbraton procedure s that when there are shape symmetres or constant radance on the object, camera pose parameter estmaton s not stable, however, these do not affect the 3D reconstructon (e.g. multple vews on a sphere do not allow estmatng camera pose, but they stll allow estmatng the shape of the sphere). Regardng the radance assumptons, because our algorthm ntegrates nformaton globally on the entre collecton of mages, t s far less senstve to ths accdent than algorthms based on local statstcs, such as pont feature correspondences. Therefore, symmetres are not an obstacle snce our goal s not to obtan the absolute calbraton parameters (ground truth) but to help refne 3D reconstructon. From ths perspectve, the only crteron of concern s the re-projecton error. We expermented wth a full turn head sequence usng Intel s Vang Gogh Bust data for testng the ssue of shape complexty. We utlzed only 16 camera vews from the avalable 33 camera mages for ease and speed of computatons. We computed re-projecton errors: a Type II error (error of omsson) and Type I error (error of commsson) by counts of voxels for several camera vews used durng our experments both after perturbaton of the camera parameters and after evoluton of the parameters as shown n Table 1. After refnement stage, the Type I error dropped by 95%, and Type II error dropped by 4%. As remarked above, our goal s not to obtan absolute camera parameters but to help 3D reconstructon algorthm to obtan objects correctly, whch s acheved. The Bust data comprses of numerous vews, and ths facltated the followng experment to show the practcalty of our calbraton correcton. For the three camera vews, out of the 16 vews, we delberately

22 22 used wrong camera calbraton parameters, whch belong to that of the neghbor vews n the sequence n Fgure 16. Ths represents a possble perturbaton n a real lfe scenaro,.e. the cameras are accdentally moved a lttle bt after the calbraton and the vews that are captured afterwards are a lttle bt off. The 3D reconstructon of the Bust object on the top rght shows the erroneous surfaces obtaned n ths case. Wth our coordnated refnement of the extrnsc parameters usng Eq.(28) and (33), the mprovements n the reprojecton errors and the 3D reconstructon are observed n Fgure 16. A real color calbraton experment s carred out usng HP Labs stereo rg system. We captured mages, shown n Fg.17, of the color calbraton object from fve cameras. Notce that the frst pcture s somewhat darker than the others, second and thrd pctures appear lghter, and there s a color msmatch. A cube surface s rgdly regstered wth the scene, also the radance functon on the cube s estmated as shown n bottom row of Fg.17. The second row shows vews after the evoluton of color calbraton coeffcents are completed. The thrd row shows the projectons of the model surface onto the vews. It can be vsually assessed that color responses of the cameras have acheved a balancng effect, and helped to obtan a better texture mappng as well. Next we demonstrate a calbraton experment usng pctures from a handheld camera wth no camera calbraton nformaton avalable. In ths scenaro, the varatonal calbraton technques we presented requre some rough ntal values that we obtaned through a self calbraton software currently under development. We have a 13 set of pctures taken around the Statue of Lberty, coverng about 22/36 degrees of a crcle around the statue, a few of the vews shown n Fgure We obtaned ntal camera parameters: extrnscs and ntrnscs ncludng the skew parameter. A rough calbraton results n the projectons shown n Fgure 18. After evoluton of the camera parameters: extrnscs, ntrnscs ncludng the skew parameter, and color parameters, the comparson s done wth the vsual hulls of before and after evoluton camera parameters n Fgure 19. One can observe the correcton n the Statue of Lberty surface wth a better set of camera parameters obtaned wth the derved update equatons throughout the 2 We thank our colleague Irwn Sobel at HP Labs for provdng these pctures.

23 23 paper. We also show blow-up regons n Fgure 2 from some of the camera vews before and after the evoluton of the color camera parameters, and the colors are modfed towards achevng some relatve agreement among the cameras whch can however only be subjectvely judged. A. Dscussons One may argue that the requrement of some rough ntal extrnsc and ntrnsc camera parameters lmts the usablty of ths technque. However, the refnement or correcton of camera parameters from a perturbed state of a prevous calbraton s a real world problem that constantly presents obstacles to the usage of multple camera systems. After a very good ntal calbraton, the cameras over tme may see small changes n ther parameters. For nstance, extrnsc parameters wll often be changed partcularly due to unwanted accdental moton. Smlarly, the ntrnscs and color parameters of the cameras may go through small varatons due to ambent condtons and wear-off. Therefore, the presented camera calbraton framework proves to be a useful tool for mult-camera systems. B. Conclusons In ths paper, we employed the 3D stereo technques based on varatonal deas to varous camera calbraton refnement problems. We have presented new mult-vew stereo technques to: evolve pose parameters of a 3D model object to take advantage of the known shape of calbraton object, and to reduce computatonal complexty, evolve dstorton parameters of cameras gven a 3D model shape, evolve color calbraton parameters of cameras gven a 3D model shape, evolve ntrnsc parameters of cameras, evolve extrnsc parameters of cameras. Pros and cons of ths technque are dscussed as follows: A nce feature of the methodology presented n ths paper s that t can ntegrate several small and dfferent problems such as dstorton calbraton, color calbraton nto an overall unfed system based on the jont segmentaton framework, and smultaneously evolve pose, color, dstorton, extrnsc, and other parameters as well.

24 24 We make pecewse smooth object assumpton and a constant background assumpton, whch may be a lmtaton f the background s to be modeled as well. However, a background model may be added to ths framework f needed. The presented methods elmnate the need for search of mage edges, pont correspondences from mages, whch can be very senstve to pxel-level nose whereas our approach beng based on mage regons for comparsons, s not as senstve to nose. Another advantage of our framework s that t easly accommodates addtonal data. In the more classcal approaches to stereo, brngng n more data, or addng more mages to the algorthm mght not help all the tme, that s f somethng goes wrong n the ndependent segmentaton phase of even one mage, t destroys the whole process of reconstructons and geometry. On the other hand, addng more data to ths jont segmentaton framework wll only mprove robustness, provdng more tolerance towards errors. For the dstorton calbraton method, more mprovements may be obtaned wth utlzng more poses, hence many more camera mages of the calbraton object, and more than one dstorton coeffcent n the model selected. One can also utlze more general/complcated dstorton models than the smple polynomal D functon. Currently, we have an mplct representaton of the calbraton objects,.e. the cube or the rectangular bar. Computng surface normals, vsblty functons for the surface occludng boundary from ths mplct representaton s not perfectly exact, and the quanttes are slghtly smeared. A future drecton towards more effcent algorthms, s to use an explct representaton of the calbraton object to more accurately descrbe the occludng boundares. Wth ths approach, 3D grds are not needed for the data structures, resultng n ncreased accuracy, speed and decreased memory requrements. Camera calbraton s partcularly suted to our framework, snce t does not have to be done n real-tme, and also the envronmental condtons may be allowed to vary to a degree (e.g. our choce of a constant colored foreground object before a dark background).

25 25 ACKNOWLEDGEMENTS We acknowledge HP Labs, Palo Alto, CA, for ther support to G. Unal and A. Yezz through grants to Georga Tech. for fundng of ths work. We thank our colleagues at HP Labs: Bruce Culbertson, Harlyn Baker, Irwn Sobel, Tom Malzbender, and Donald Tanguay for frutful dscussons and ther support. We also thank Haln Jn for provdng us the Intel s Bust dataset. Fg. 3. Intalzed surface model shown from three dfferent vantage ponts. Fg. 4. Column 1: Pose1. Row 1: one camera mage shown, Row2: wth projecton of ntalzed surface (orange mask), Rows 3-5: durng evoluton of the pose parameters of the surface, Row 6: wth converged pose parameters. Columns 2-3: same as column 1 for poses 2 and 3, respectvely.

26 26 Fg. 5. Pose 1. Row 1: Three out of fve captured vews. Row 2: Projected surface after dstorton parameters have converged. Row 3: Undstorted wth the obtaned dstorton coeffcents. Fg. 6. Pose 2. Row 1: Three out of fve captured vews. Row 2: Projected surface after dstorton parameters have converged. Row 3: Undstorted wth the obtaned dstorton coeffcents.

27 27 Fg. 7. Pose 3. Row 1: Three out of fve captured vews. Row 2: Projected surface after dstorton parameters have converged. Row 3: Undstorted wth the obtaned dstorton coeffcents. Fg. 8. Row 1: Three orgnal vews (cameras 1-7-9). Row 2: The same three dfferent after delberate smulated mscalbraton of the greyscales. The same three vews whle evolvng the calbraton parameters: Rows 3-4 ntermedate stages, Row 5: The vews after evoluton of the calbraton parameters has completed.

28 Camera number: Evoluton tme Camera number: Evoluton tme Evoluton tme Camera teratons Camera number: Camera number: Evoluton tme Camera number: Evoluton tme Camera number: Evoluton tme Camera 1 teratons Camera number: Evoluton tme Camera number: Evoluton tme Camera number: Evoluton tme Camera 7 teratons 28 Alpha Alpha Alpha Alpha Alpha Alpha Alpha Alpha Alpha Fg. 9. Evoluton of the parameter α for dfferent camera vews. True α value s shown as a dotted lne. Fg. 1. Some camera vews shown durng the evoluton of the color calbraton. Top: orgnal vews, Mddle: Perturbed vews, Bottom: Fnal vews after convergence. Note the color smlarty n top and bottom rows. alpha alpha alpha Fg. 11. Evoluton of the parameter α for dfferent vews for R,G,B channels of the synthetc color cube. True α value s shown as a dotted lne.

29 Camera 2 teratons Camera 5 teratons Camera 7 teratons 29 focal length: x ( ), y(.) focal length: x ( ), y(.) focal length: x ( ), y(.) Fg. 12. Top: Three camera vews shown durng the evoluton of the ntrnsc parameters of an ntal cube wth projectons from the ntal surface, Mddle: Fnal vews after convergence of the ntrnsc parameters of the surface. Also shown at the bottom are the evoluton of the two focal length parameters for each shown camera vew (red and green curves) along wth the true (blue curve) focal lengths. Fg. 13. Four camera vews shown (top) durng the evoluton of the extrnsc parameters of an ntal surface of a toy skater, Row 2: Vews shown wth projectons from the ntal surface, Row 3: Fnal vews after convergence of the extrnsc camera parameters. Vsual hull generated usng the mscalbrated ntal extrnsc parameters (row 2 rght); vsual hull generated usng the converged extrnsc parameters (row 3 rght).

30 3 Fg. 14. Uncertanty ellpsods drawn around each camera center for the toy skater data show the extrnsc refnement stablty (rght: zoomed nto one camera s perturbatons). Fg. 15. Row 1: Four camera vews durng the evoluton of the extrnsc plus ntrnsc parameters of a toy skater wth projectons of the ntal surface, Row 2: Fnal vews after convergence of the camera parameters. Vsual hull generated usng the mscalbrated ntal parameters (row 1 rght); vsual hull generated usng the converged parameters (row 2 rght).

31 31 Type II error Camera Type I error Intal Fnal Intal Fnal Cam Cam Cam Cam Cam Cam Cam Cam Cam Cam Cam Table 1. Type I and Type II errors n counts of voxels for several camera vews for Bust data (Fg. 16) after perturbaton of camera parameters (Intal), and after evoluton of parameters (Fnal). Fg. 16. Camera vews 78,167, and 24 n top row are used delberately wth camera calbraton parameters of camera vews 77, 166, and 239 of the Van Gogh Bust dataset. Top: Three camera vews shown wth projectons from the ntal surface n row 2, here note the resultng ntal msmatch n projected slhouettes. Row 3: Fnal vews after convergence of the camera parameters. Vsual hull surfaces obtaned by usng wrong calbraton parameters for vews 78, 167, 24 on the rght (top row) and surfaces wth corrected calbraton parameters n bottom row.

32 32 Fg. 17. Some camera vews shown durng the evoluton of the color calbraton parameters of the HP color calbraton object surface. Top: Fve camera vews; Row 2: Fnal vews after convergence of the extrnsc camera parameters; Row 3: Same shown wth projectons of the converged cube; Bottom: Color calbraton cube wth reconstructed radance on the surface from two dfferent vantage ponts.

33 33 Fg. 18. Some camera vews shown durng the evoluton of the camera calbraton parameters of the Statue of Lberty surface. Top: Fve camera vews shown wth projectons from the ntal surface n Row 2; Row 3: Fnal vews after convergence of the camera parameters. Fg. 19. Vsual hull surfaces wth ntal rough calbraton parameters(top), and wth refned calbraton parameters (bottom), also wth radance texture mapped onto the surfaces.

34 34 Fg. 2. Some camera vews before and after the color calbraton for the statue of lberty. REFERENCES [1] O. Faugeras and R. Kerven, Varatonal prncples, surface evoluton pdes, level set methods and the stereo problem, Tech. Rep., INRIA, [2] A. Yezz and S. Soatto, Stereoscopc segmentaton, Int. J. Computer Vson, vol. 53, pp , 23. [3] A. Yezz and S. Soatto, Structure from moton for scenes wthout features, n Proc. IEEE Conf. on Computer Vson and Pattern Recognton, 23, pp [4] I. Sobel, On calbratng computer controlled cameras for percevng 3d scenes, AI, vol. 5, pp , [5] R. Y. Tsa, A versatle camera calbraton technque for hgh-accuracy 3d machne vson metrology usng off-the-shelf tv cameras and lenses, J. Robotcs and Automaton, vol. 3, pp , [6] R. Hartley and A. Zsserman, Multple Vew Geometry n Computer Vson, Cambrdge Unversty Press, 2. [7] B. Trggs, Camera pose and calbraton from 4 or 5 known 3d ponts, n Proc. IEEE Conf. on Computer Vson and Pattern Recognton, 1999, pp [8] P. Gurdjos and P. Sturm, Methods and geometry for plane-based self-calbraton, n Proc. IEEE Conf. on Computer Vson and Pattern Recognton, 23, pp [9] O.D. Faugeras, Q.T. Luong, and S.J. Maybank, Camera self-calbraton: Theory and experments, n Proc. European Conf. Computer Vson, 1992, pp [1] Y. Seo, A. Heyden, and R. Cpolla, A lnear teratve method for auto-calbraton usng the DAC equaton, n Proc. IEEE Conf. on Computer Vson and Pattern Recognton, 21, pp [11] J. Olenss, Fast and accurate self calbraton, n Proc. Int. Conf. on Computer Vson, 1999, pp [12] M. Pollefeys, R. Koch, and L. Van Gool, Self-calbraton and metrc reconstructon n spte of varyng and unknown ntrnsc camera parameters, Int. J. Computer Vson, vol. 32, no. 1, pp. 7 25, [13] T. Svoboda, D. Martnec, and T. Pajdla, A convenent mult-camera self-calbraton for vrtual envronments, PRESENCE: Teleoperators and Vrtual Envronments, vol. 14, no. 4, 25. [14] J. Hekkla and O.Slven, A four-step camera calbraton procedure wth mplct mage correcton, n Proc. IEEE Conf. on Computer Vson and Pattern Recognton, 1997, pp

35 35 [15] G. Unal and A. Yezz, A varatonal approach to problems n calbraton of multple cameras, n Proc. IEEE Conf. on Computer Vson and Pattern Recognton, 24, pp [16] G. Unal, A. Yezz, H. Baker, and B. Culbertson, A varatonal approach to calbraton of multple cameras, Tech. Rep., HP Labs, HPL , Palo Alto, CA, 24. [17] F. Devernay and O. Faugeras, Straght lnes have to be straght, Machne Vson and Applcatons, vol. 13, pp , 21. [18] J. Weng, P. Cohen, and M. Hernou, Camera calbraton wth dstorton models and accuracy evaluaton, IEEE PAMI, vol. 14, pp , [19] Z. Zhang, A flexble new technque for camera calbraton, Tech. Rep., Mcrosoft Research, MSR-TR-98-71, [2] D.C. Brown, Decenterng dstorton of lenses, Photogrammetrc Engneerng, vol. 32, [21] M. T. El-Melegy and Aly A. Farag, Nonmetrc lens dstorton calbraton: Closed-form solutons, robust estmaton and model selecton, n Proc. Int. Conf. on Computer Vson, 23, pp [22] S.B. Kang, Semautomatc methods for recoverng radal dstorton parameters from a sngle mage, Tech. Rep., CRL Tech. Report CRL 97/3, [23] R. Swamnathan and S.K. Nayar, Nonmetrc calbraton of wde-angle lenses and polycameras, n Proc. IEEE Conf. on Computer Vson and Pattern Recognton, 1999, pp [24] G.P. Sten, Lens dstorton calbraton usng pont correspondences, n Proc. IEEE Conf. on Computer Vson and Pattern Recognton, 1997, pp [25] A.W. Ftzgbbon, Smultaneous lnear estmaton of multple vew geometry and lens dstorton, n CVPR, 21. [26] F. Du and M. Brady, Self-calbraton of the ntrnsc parameters of cameras for actve vson systems, n Proc. IEEE Conf. on Computer Vson and Pattern Recognton, 1993, pp [27] H.S. Sawhney and R. Kumar, True mult-mage algnment and ts applcaton to mosacng and lens dstorton correcton, n Proc. IEEE Conf. on Computer Vson and Pattern Recognton, 1997, pp [28] H.H. Baker, D. Tanguay, I. Sobel, D. Gelb, M.E. Goss, B.W. Culbertson, and T. Malzbender, The colseum mmersve teleconferencng system, Tech. Rep., HP Labs, HPL , 22. [29] E. Marszalec, On-lne color camera calbraton, n Proc. Int. Conf. on Pattern Recognton, 1994, pp [3] S. Osher and J.A. Sethan, Fronts propagatng wth curvature dependent speed: Algorthms based on the Hamlton-Jacob formulaton, J. Computatonal Physcs, vol. 49, pp , [31] D. Adalstensson and J.A. Sethan, A fast level set method for propagatng nterfaces, J. Computatonal Physcs, vol. 118, pp , [32] J. Sokolowsk and J.P. Zoleso, Introducton to Shape Optmzaton, Shape Senstvty Analyss, Sprnger Verlag, Berln, Helderberg, [33] R.M. Murray, Z. L, and S. Sastry, A Mathematcal Introducton to Robotc Manpulaton, CRC Press, 1994.