Finding Intrinsic and Extrinsic Viewing Parameters from a Single Realist Painting

Transcription

1 Fndng Intrnsc and Extrnsc Vewng Parameters from a Sngle Realst Pantng Tadeusz Jordan 1, Davd G. Stork,3, Wa L. Khoo 1, and Zhgang Zhu 1 1 CUNY Cty College, Department of Computer Scence, Convent Avenue and 138th Street, New York NY tedjj13@gmal.com, wlkhoo@gmal.com, zhu@cs.ccny.cuny.edu Rcoh Innovatons, 88 Sand Hll Road, Sute 115, Menlo Park CA Stanford Unversty, Department of Statstcs, Stanford CA artanalyst@gmal.com Abstract. In ths paper we studed the geometry of a three-dmensonal tableau from a sngle realst pantng Scott Fraser s Three way vantas (006). The tableau contans a carefully chosen complex arrangement of objects ncludng a moth, egg, cup, and strand of strng, glass of water, bone, and hand mrror. Each of the three plane mrrors presents a dfferent vew of the tableau from a vrtual camera behnd each mrror and symmetrc to the artst s vewng pont. Our new contrbuton was to ncorporate sngle-vew geometrc nformaton extracted from the drect mage of the wooden mrror frames n order to obtan the camera models of both the real camera and the three vrtual cameras. Both the ntrnsc and extrnsc parameters are estmated for the drect mage and the mages n three plane mrrors depcted wthn the pantng. Keywords: camera calbraton, perspectve geometry, art analyss. 1 Introducton The problem of reconstructng a three-dmensonal scene from multple vews s well explored, and a number of general methods, such as those based on correlaton, relaxaton, dynamc programmng, have been developed and fully characterzed [5, 6]. Three-dmensonal reconstructon and metrology can be based on sngle vews as well [3, 1]. Crmns and hs colleagues [] have recently appled such technques to the analyss of pantngs, for nstance reconstructng the vrtual spaces n Masaco s Holy Trnty (c. 145), Pero della Francesca s Flagellaton of Chrst (c. 1453), Hendrck V. Steenwck s St. Jerome n hs study (1630), Jan Vermeer s A lady at the Vrgnals wth a gentleman ( ), and others. These methods reveal both the hgh geometrc accuraces n some passages, and the geometrc nconsstences n others, propertes that are nearly mpossble to determne by eye. Such analyses shed new lght on these works and the artsts workng methods, for nstance revealng whether an artst lkely used geometrcal ads durng the executon of ther work. Recently Smth, Stork and Zhang reconstructed the three-dmensonal space depcted n a hghly realstc modern pantng, Scott Fraser s Three way vantas (Fg. 1) usng tradtonal multple-vew reconstructon methods appled to the drect vew and X. Jang and N. Petkov (Eds.): CAIP 009, LNCS 570, pp , 009. Sprnger-Verlag Berln Hedelberg 009

2 94 T. Jordan et al. a vew vsble n a depcted mrror [11]. Even though usng reflected mages by mrrors s a very popular approach for stereo vson n computer vson [7, 9, 10, 1], t was the frst tme to analyze a pantng wth such a setup. However, there were some lmtatons n that prevous work as well as unexplored opportuntes. For nstance, the mages of the frames of the mrrors provde geometrc constrants about the centers of projecton of the mages depcted wthn each mrror, and the earler scholarshp dd not ncorporate that nformaton when reconstructng the three-dmensonal space. Secton descrbes the pantng, prevous scholarshp and an overvew of our new approach to camera parameter estmaton. Secton 3 ntroduces some notatons and constrants used n the paper. Sectons 4 to 7 descrbe the detals of estmatng both ntrnsc and extrnsc parameters of the man and vrtual cameras: fndng mage center, estmatng focal length, representng mrror planes and locatng vrtual cameras. Secton 8 gves a bref summary and dscusson. y M 0 p 3 p 4 M M 1 M 3 Z m 1 v v 3 m x n n1 o v 1 f p 1 p Y X O Fg. 1. The work, the notatons and the constrants The Work and Problem Addressed Fg. 1 shows the work we consder, Scott Fraser s Three way vantas (006). Ths pantng was commssoned as part of The Object Project, n whch ffteen artsts were commssoned to create works, each contanng fve specfed objects: hand mrror, bone, moth, ball of strng and drnkng glass [4]. Our method for estmatng the camera models for the vrtual cameras s based on the sngle-mage nformaton of the mrror frames n the prmary mage of the pantng; the mage seen from the artst s

3 Fndng Intrnsc and Extrnsc Vewng Parameters from a Sngle Realst Pantng 95 vewng pont (the man camera). We construct geometrc nvarants such as horzontal/vertcal lnes and vanshng ponts from the locatons of the vertces of ts frame, to estmate the focal length, the center of projecton of the man camera, and the locaton and orentaton of each mrror. Then, we use the pose nformaton of each mrror to compute the locaton and orentaton of the vrtual camera by fndng ts coordnate system that s symmetrc to the artst s vewng pont n the plane of the mrror. Usng the knowledge of the mrror frames, both the ntrnsc and extrnsc parameters between a par of stereo cameras can be found. Ths s dffcult f basc fundamental matrx method s used, as n [11], whch used only lmted number of ponts on the table n both the real mage and the mrror mages. 3 Notatons and Constrants We label the mrrors, readng rght to left, M, ther assocated reflected mages I, and correspondng centers of projecton C, for = 1,, 3. The mddle frame s labeled as M 0. Let s use the mrror M 1 as an example. Each of the four vertces of the frame rectangle, P has a correspondng mage pont p = (a, b, f), = 1,, 3, 4. These mage ponts can also be vewed as vectors from the optcal center O to those ponts. Snce P 1 P P 3 P 4, the drecton of the parallel lnes can be estmated as ( p p ) ( p ) v =. (1) p Snce P 1 P 3 P P 4, the drecton of the parallel lnes can be estmated as v = p p ) ( p ). () ( 4 3 p1 The normal of the rectangular mrror surface s Here are a few notes: v 3 v1 v =. (3) 1. Pont (a, b ) should be measured n the xoy coordnate system that s algned wth the man camera coordnate system O-XYZ. Therefore the mage center o should be estmated frst for usng the above equatons.. The focal length f s unknown and should be estmated frst to use Eqs. (1) (3). 3. If the projectons of a par of parallel lnes are not parallel n the mage, such as p 1 p and p 3 p 4, the drecton of the par can be calculated through ther vanshng pont n the mage plane, (v 1x, v 1y, f), therefore v 1 ( 1x 1y f v, v, ). (4) Otherwse, the vanshng pont s n nfnte, and the thrd dmenson of the drecton vector such as v should be zero. Ths s the advantage to use Eqs. (1) to (3) to fnd drectons of parallel lnes nstead of usng vanshng pont estmaton. In the followng, we wll descrbe methods n estmatng the followng parameters of the real camera and the three vrtual cameras created by the three mrrors: 1). the mage center o(c x,c y ); ). the focal length f; 3). the plane equaton of each mrror; and

4 96 T. Jordan et al. 4). the pose of the three vrtual cameras related to the real camera (the artst s eye). The four cameras share the same focal length and the same mage center, but the mages of the three mrrored cameras are reversed n the x drecton. The ntrnsc geometrc constrants (assumptons) we use are the followng: 1. All three mrrors are rectangular.. The two flankng plane mrrors have the same nherent shape and are arranged vertcally, each rotated by an unknown angle, and that the central plane mrror s vewed frontally, tpped forward by an unknown angle. 3. The back edges of the two flankng mrrors are at the same dstance. 4. The aspect rato of the mage s 1:1. In addton to the above constrants, by analyzng the mages of the frames and mrrors, we have also observed that the left and rght flankng mrrors (M 3 and M 1 ) and the central frame (M 0 ) are vertcal, and the mddle nset mrror (M 1 ) s only tlted n the y drecton. 4 Fndng Image Center To fnd the mage center, we wll need to have both a set of 3D horzontal lnes and a set of vertcal lnes whose projectons are not parallel n the mage. By fndng the vanshng pont of the two pars of horzontal edges of the left and rght flankng mrrors, (vl x, vl y ) and (vr x, vr y ), the y coordnate of the mage center can be determned as c y =( vl y + vr y )/. By fndng the vanshng pont of the two vertcal edges of the nset mrror, (vv x, vv y ), the y coordnate of the mage center can be determned as c x = vv x. In Fg., usng the dgtal mage coordnate system (x, y ) as shown n Fg., we obtan the results n Table 1: x (vr x, vr y ) (vl x, vl y ) y (vv x, vv y ) Fg.. Estmatng mage center usng vanshng ponts Table 1. Vanshng ponts portrayed n Fg. Usng left mrror Usng rght mrror Usng mddle mrror (319.8, 57.1) ( , 519.7) (868.6, 654.3)

5 Fndng Intrnsc and Extrnsc Vewng Parameters from a Sngle Realst Pantng 97 5 Estmatng Focal Length The focal length estmaton s the most crtcal step. Once we fnd f, the rest of steps wll be rather straghtforward. We can fnd the focal length f by usng the fact that the left and rght flankng mrrors have exactly the same wdth,.e. W 1 = W n Fg. 3. n w n 1 Z m 1 w 1 m β θ θ 1 β 1 W W 1 m' n' f X O Fg. 3. Focal length estmaton usng the equal-wdth constrant We scaled the mrrors nto pxel representaton so the dstance of the two back edges (m 1 and n 1 ) are f (refer to Fg. 1). The drectons of n 1 n and m 1 m can be represented by the vanshng ponts, v m = (v mx, v my, f) = (8.6, 7., f), generated from the two sets of horzontal edges. The dot product of normalzed v m and the X- axs (1, 0, 0) should be v m / v m (1, 0, 0) = cos! 1, assumng that v m s n the horzontal plane as the X-axs (from assumpton No ). We can derve a smlar relaton for v n. Therefore, the angles! 1 and! can be represented as functons of f: θ1 = tan ( f / vmx ), θ = tan ( f / vnx ). (5) From the mage, locaton m 1, m and n 1, n can be measured (n the x drecton, refer to Fg. 1). Therefore, the two dstances w 1 and w can be calculated as w 1 = m -m 1, w = n -n 1. The two angles β 1 and β can be also represented as functons of f: β 1 = tan ( f / m ), β = tan ( f / n ). (6) Usng the sne law wth trangles n 1 n n and m 1 m m, and usng the fact that W 1 = W, we can derve that: where a m an vnx amvmx f = am an w1 w =, an =. m + v n + v mx nx. (7)

6 98 T. Jordan et al. Usng the measurements from the mage, we obtan f = (pxels) Note that ths method does not work f the two mrrors are symmetrc. We calculated the two o o angles usng Eq. (5): θ 1 = tan ( f / vmx ) = 4.6, θ = tan ( f / v nx ) = The two mrrors have dfferent flankng angles, so they are not symmetrc, therefore we can obtan the focal length estmaton. 6 Representng Mrror Planes Once we obtan the value of the focal length f, we can use the general equatons (1) to (3) to calculate the drectons of the two edges of each mrror M, and ts normal n. These equatons work for all the cases no matter f we can fnd vanshng ponts or not. The results of the rotaton matrces for M (=0, 1,, 3) are lsted n Table. Table. The rotaton matrces ( nd row) and angles (3 rd row, n degrees) of M ( = 0, 1,, 3) M 0 M 1 M M (0.66, -0.76, 0.31) (3.59, 43.16, 1.6) (19.38, -0.3, 0.16) (0.9, , 0.35) From the rotaton matrces t can be seen that the three angles of the mddle frame M 0 are almost zero degrees, the mddle nset mrror M s manly tpped forward, the rght mrror M 1 and the left mrror M manly have flankng angles consstent wth the results obtaned by usng Eq. (5). The results hghly agree wth our assumptons of the ntrnsc geometry constrants n Secton. The plane equaton of the mrror can be represented as n P + d = 0. (8) c where P c = ( X c, Y c, Z c ) t s represented n the camera coordnate system O-XYZ. In order to fnd d, and further fnd the vrtual camera parameters mrrored by each mrror, we wll buld a world coordnate system on each mrror. For example, for mrror M 1, after we fnd the three vectors and then normalzed them nto column unt vectors, stll represented n v 1, v and v 3, we can defne a world coordnate system usng the mddle of the mrror plane as ts orgn, and the three vectors as ts three coordnate axes. The transformaton between the camera coordnate system and the world coordnate system of the th mrror M ( = 0, 1,, 3) can be represented as Pc R 1 Pw + T1 =. (9) where P c =( X c, Y c, Z c ) t, P w =( X w, Y w, Z w ) t, R 1 = (r pq )3x3, whch s (v 1, v, v 3 ) for M 1, and T 1 to be determned. The projecton of P c nto the mage of the man camera s (x, y) = (f X c /Z c, f Y c /Z c ).

7 Fndng Intrnsc and Extrnsc Vewng Parameters from a Sngle Realst Pantng 99 To fnd the translatonal vectors for all the three mrrors, we use the dmenson of the mddle frame M 0 as a reference. To calculate the translatons for mrrors M 1 and M 3, we use ther heghts h= pxels, whch can be measured n mage wth respect to the heght H of the mddle frame, snce they are at about the same dstance. Smlarly, to calculate the translaton for the mddle nset mrror, we use ts wdth w = pxels, whch can be measured aganst the wdth W. Table 3. The translatons and the dstances of M 0 to M 3 Mrror T 1 ( = 0, 1,, 3) d ( = 0, 1,, 3) M 0 (-51.3, , 048.5) M 1 (-430.1, -15.7, 199.7) M (-50.9, -18.6,051.1) M 3 (340.5, -14.6, ) Locatng Vrtual Cameras After we obtan the plane equaton for each mrror (Eq. (8), the mrrored coordnate system,.e., the vrtual camera C, can be easly obtaned, as n [9]. Here we use a coordnate transformaton method to fnd the relaton between each vrtual camera C and the real camera C, by fndng the rotaton matrx R and translatonal vector T ( = 1,, 3). In our mplementaton, we use Eq. (9) to represent the orgn and the three axes of the camera coordnate system n the world coordnate system X w Y w Z w of each mrror M. Snce X w OY w s the mrror plane, the mrrored orgn and axes can be smply obtaned by changng the sgns of ther Z w components. Then we do a smlar procedure as n Eq. (9) to fnd the transformaton (characterzed by R and T, = 1,, 3) between the world coordnate system M and the vrtual camera C : Pw R P + T =. (10) Combnng Eqs (9) and (10), we can fnd the transformaton between the real camera and the vrtual camera: P = R P + T. (11) c R = R1 R, T = R1 T + T. 1 (1) Table 4 shows the estmated results. Usng the full 6 degree-of-freedom (DOF) relaton between each vrtual camera and the man camera, stereo reconstructon can be performed, and the accuracy of the pantng can be analyzed. Table 4. The transformatons between vrtual camera C ( = 1,, 3) and the man camera M M 1 M M 3 R T (-99.7, 153.4,498.7) (0.4, 14.4, ) (00., 13.3, 69.3)

8 300 T. Jordan et al. 8 Conclusons and Dscussons Our method for estmatng the camera models for the vrtual cameras was based on sngle-vew analyss. The relatve 3D structures of the rectangles mrrors and frames are estmated by usng ther perspectve analyss wth a few assumptons. The camera calbraton for the cameras s then successfully performed. Its results can then be used for 3D reconstructon and pantng analyss, whch s descrbed n an accompanyng paper [8]. The 3D estmates of both the frames and regular objects are consstent among the sngle-vew analyss, and results from multple stereo trangulatons. However, there are some stereo and perspectve nconsstences across four vews. These could ether be the accuraces of the perspectve dstortons, orentatons and szes of the mrrors, or of the locatons of objects nsde the mrrors, whch cannot be easly observed by eye. As such, our work extends the new dscplne of computer vson appled to the study of fne art. Acknowledgements We thank Scott Fraser for nformaton about hs workng methods and permsson to reproduce hs pantng. The work s also partally supported by NSF under Grant No. CNS References [1] Crmns, A.: Accurate vsual metrology from sngle and multple uncalbrated mages. ACM Dstngushed Dssertaton Seres. Sprnger, London (001) [] Crmns, A., Kemp, M., Zsserman, A.: Brngng pctoral space to lfe: computer technques for the analyss of pantngs. In: Bentkowska-Kafel, A., Cashen, T., Gardner, H. (eds.) Dgtal Art Hstory: A Subject n Transton, pp (005) [3] Crmns, A., Red, I., Zsserman, A.: Sngle vew metrology. IJCV 40(), (1999) [4] Doherty, M.S.: Object Project: Fve objects, ffteen artsts. Evansvlle Museum of Arts, Hstory and Scence. Evansvlle, IN (007) [5] Faugeras, O.: Three-Dmensonal Computer Vson: A Geometrc Vewpont. MIT Press, Cambrdge (1993) [6] Hartley, R., Zsserman, A.: Multple Vew Geometry n Computer Vson, nd edn. Cambrdge Unversty Press, Cambrdge (003) [7] Huang, P.-H., La, S.-H.: Contour-based structure from reflecton. In: CVPR 006, vol. 1, pp (006) [8] Khoo, W., Jordan, T., Stork, D., Zhu, Z.: Reconstructon of a three-dmensonal tableau from a sngle realst pantng. In: 15th Int. Conf. Vrtual Systems & Multmeda, September 9-1 (009) [9] Kumar, R.K., Ile, A., Frahm, J.-M., Pollefeys, M.: Smple calbraton of non-overlappng cameras wth a mrror. In: CVPR 008 (008) [10] Nene, S.A., Nayar, S.K.: Stereo wth mrrors. In: ICCV 1998, pp (1998) [11] Smth, B., Stork, D.G., Zhang, L.: Three-dmensonal reconstructon from multple reflected vews wthn a realst pantng: an applcaton to Scott Fraser s Three way vantas. SPIE/IS&T, Bellngham (009) [1] Zhu, Z., Ln, X., Sh, D., Xu, G.: A Sngle camera stereo system for obstacle detecton. In: 4th Int. Conf. Informaton Systems Analyss and Synthess, July 1-16, vol. 3, pp (1998)