3D Reconstruction from 360 x 360 Mosaics 1

Transcription

1 CENTER FOR MACHINE PERCEPTION 3D Reconstuction fom 36 x 36 Mosaics CZECH TECHNICAL UNIVERSITY {bakstein, pajdla}@cmp.felk.cvut.cz REPRINT Hynek Bakstein and Tomáš Pajdla, 3D Reconstuction fom 36 x 36 Mosaics, in Poceedings of the CVPR, to be published Copyight: IEEE Available at ftp://cmp.felk.cvut.cz/pub/cmp/aticles/bakstein/bakstein-cvpr.pdf Cente fo Machine Peception, Depatment of Cybenetics Faculty of Electical Engineeing, Czech Technical Univesity Technická, 66 7 Pague 6, Czech Republic fax , phone , www:

2 3D Reconstuction fom 36 x 36 Mosaics Hynek Bakstein Tomáš Pajdla Cente fo Machine Peception, Dept. of Cybenetics Faculty of Electical Eng., Czech Technical Univesity 35 Pague, Czech Republic {bakstein,pajdla}@cmp.felk.cvut.cz Abstact We ae studying the geomety of a 36 x 36 mosaic image fomation. A 36 x 36 mosaic camea model and a calibation pocedue ae poposed. It is shown that only one point coespondence is needed in ode to acquie epipolaily ectified images. The 36 x 36 mosaic camea model is theefoe detemined by only one intinsic paamete. It is shown that the elation between coodinates estimated with diffeent values of intinsic 36 x 36 mosaic camea paametes is a scaling of all scene point coodinates with additional nonlinea changes in the z coodinates of the scene points. Expeimental esults veifying the econstuction of eal scene points ae pesented.. Intoduction Recently, thee has been an inceased inteest in the development of panoamic sensos. Seveal appoaches have used a mio to enlage the field of view [,,, 5, 9, 8]. Othe methods employed moving pats to captue the whole panoamic scene [, 9, 3]. Finally, thee ae systems combining both mios and camea movement [6]. Some of these systems can be modeled by the cental pojection [8]. Howeve, many of the poposed methods lead to a noncental camea. A noncental camea is a camea whee the light ays foming the image do not intesect in one common point. Instead, fo example, they ae tangent to a cicle, o intesect a common line. Recently, attention was paid to the geometical popeties of noncental cameas. It has been shown that steeo geometies of noncental cameas exist and a genealized epipola geomety This wok was suppoted by the following gants: GAČR //97, OMNIVIEWS IST , MŠMT KONTAKT /9, and MŠMT 33. The authos would also like to thank Vladimí Smutný and Radim Šáa fo suggestions. with double uled quadic epipola sufaces has been descibed [7, ]. The 36 x 36 mosaic camea model, its calibation pocedue, and the elation between scene econstuctions fom uncalibated 36 x 36 mosaic images ae pesented in this pape. The 36 x 36 [6] mosaic is an example of a noncental camea whee the light ays ae tangent to a cicle, see Figue. Let us explain the geomety of the light ays that fom the mosaic. A plane π is otated on a cicula path C with a adius. The plane π is pependicula to the plane δ, in which the cicle C lies, and it is tangent to the cicle C. Let us take at each otation position all the light ays that lie in the plane π and intesect the point whee the plane π touches the cicle C. Because each point outside the cicle C can be obseved by two light ays, the 36 x 36 mosaic camea povides a complete spheical mosaic fo both left and ight eye afte otation of the plane π about 36. See [6] fo moe details about the 36 x 36 mosaic. π δ Figue. 36 x 36 mosaic camea geomety. We can imagine that the plane π epesents a D omnidiectional camea [8] which is otated on a cicula path. This D camea and the motion togethe compose the 36 x 36 mosaic camea. Let us mention two possible ealizations of the D omnidiectional camea. The fist consists of a D CCD camea with a telecentic lens obseving a conical mio with a 9 angle at the apex, depicted in Figue (a). The second is a D CCD camea equipped with a fish eye lens with a field of view of at least 8, e.g. Nikon fish

3 eye convete shown in Figue (b). Light ays fom both camea ealizations will lie in the plane π. The poposed 36 x 36 mosaic camea model assumes that fo the ealization with the mio, a cicle on the mio is pojected onto a cicle in the image acquied by the CCD camea. This can be achieved when the mio base is paallel to the image plane and the camea has squaed pixels o when the camea is calibated. When a fish eye lens is used, the cental camea has to be calibated fist. Fo the development of the following theoy we assume that the D CCD camea and the coesponding optics wee calibated and thus the D omnidiectional camea was calibated too. in Figue 3 that the z coodinate of the point depends only on the angle and x and y coodinates depend only on the angles α, α. Theefoe, a plana view of the 36 x 36 mosaic camea geomety in the xy plane can be used to illustate the computation of the x and y coodinates of the scene points, as it is depicted in Figue. C z C C y p (x,y,z)=p (x,y) x Figue x 36 mosaic camea geomety. (a) (b) Figue. Expeimental setup. (a) telecentic camea and a conical mio, (b) cental camea with Nikon fish eye convete. C α α l δ x d l P y. 3D Reconstuction fom 36 x 36 Mosaic γ.. 36 x 36 Mosaic Camea Model A cental camea s image coodinates (u, v) diectly poduce a vecto p = (u, v, ) epesenting the coesponding light ay in some camea centeed coodinate system with a cente C = (,, ). This constuction cannot be utilized in case of the 36 x 36 mosaic camea, whee the light ays do not intesect at one common point. Some epesentation of geneal lines in 3D space has to be chosen. We descibe the light ay by a point C = ( cos α, sin α, ) on the viewing cicle C and a vecto p = ( sin(α) cos(), cos(α) cos(), sin()). Angles α and ae shown in Figues 3 and. Fo each point in the scene, outside the viewing cicle, thee ae two light ays which ae tangent to C and intesect in a scene point P, see Figue 3. Thee ae thee angles which define each pai of light ays intesecting in one scene point P : α, α and, see Figues 3 and. Povided with the angles identifying the light ays, 3D coodinates of the scene point P can be computed. It can be obseved Figue. 36 x 36 mosaic camea geomety - a view in the xy plane. Using basic tigonometic ules and identities, the following expessions fo the x, y and z can be deived: x = cos δ cos γ () y = sin δ cos γ () z = tan γ tan, (3) whee is the adius of the viewing cicle, γ = α α, and δ = α+α. An impotant issue is the elation between the angles α, α, and and the pixel coodinates (u, v) in the mosaic images. To find it, two questions have to be answeed: fist, what is the elation between pixels and degees and second, which pixels coespond to zeo angles, i.e. fom which image coodinates ae the angles measued, see Figue 5.

4 36 8 α,α vpix v u α upix Figue 5. Mosaic image coodinate system (u, v) and its elation to the 36 x 36 mosaic camea paametes - angles α, α, and and thei zeo values α and. To answe the fist question is easy because ou D omnidiectional camea as well as its motion ae calibated. The angles α and α coespond to hoizontal image axis, while the angle coesponds to the vetical axis. The images contain a full panoamic view of the scene in the hoizontal diection and each contains half of the full panoamic view in the vetical diection. Theefoe, unde the assumption of a constant speed of otation, the elation between degees and pixels fo the angles α and α can be computed as upix 36 and fo the angle as vpix 8, whee upix and vpix denote the numbe of pixels in hoizontal and vetical diections espectively. The second question is moe complex. The elation between pixel coodinates in the images (u, v) and (u, v) and the angles can be descibed as: α = 36u upix α () α = 36u upix α (5) = 8v vpix. (6) Note that α = α which follows fom the fact that unpimed as well as pimed ays lie in the same plane π. The change of the zeo α angle coesponds to a otation of the 36 x 36 mosaic camea aound the z axis, see Figues 3 and. Theefoe it can be incopoated into the extinsic camea paametes. As peviously stated, each point outside the viewing cicle C can be seen by two light ays. Theefoe, two mosaic images can be acquied. In each otation step, two sets of the light ays, belonging to the two mosaic images, ae ceated by splitting the plane π into two half planes by some line l in π. If the line l is pependicula to the plane δ, the images of the coesponding points ae on the same ows in the two mosaic images. Such a pai of images is called a ectified pai. In a pactical ealization with a telecentic camea and a conical mio, the light ays ae imaged onto a cicle which is split by the line l, paallel to the vetical axis of the image and passing though the image of the tip of the cone, into two half cicles, see Figue 7. Due to the 36 x 36 mosaic camea setup, (the image plane of the CCD camea may be otated inside the plane π) the line l does not have to be pependicula to the plane δ. This ceates an angle between the line l and a line l, see Figue 6. The images ae not ectified, moeove the coesponding points may lie in the same image, as depicted in Figue 6(b). The same effect is obtained if the elevation angle is measued fom a wong zeo value. Label I in Figue 6(a) denotes the coect division of ays by the line l, label II coesponds to a wong division by l. When dividing the ays into fou pats, maked in the figue, then, in the coect case, pats, and 3, compose the ight eye and the left eye mosaic espectively. In the incoect case, pat is moved to the top of the left eye mosaic and pat 3 to the bottom of the ight eye mosaic. Theefoe, the point A with the coesponding point A ae both contained in the same mosaic image. It is inteesting that only one known point coespondence is equied to estimate the value of the elevation angle that coesponds to the division by the line l. Denoting the elevation angles of the coesponding points, and some incoect zeo elevation angle, the coect zeo elevation angle can be computed as: = + ( ). (7) Because the coesponding points should lie on the same image ows, the images have to be shifted in opposite diections until they ae ectified. Rows which wee moved fom the top of one image will appea on top of the othe image and symmetically the same occus with the bottom ows of the images. This is due to the fact that the 36 x 36 images have the topology of a tous []. Afte the ectification, it can be assumed that coesponds to the middle ow of the image mosaics. A known angle is assumed in the following section x 36 Mosaic Camea Calibation In the following text, the tem 36 x 36 mosaic camea calibation efes to the estimation of unknown paametes of the poposed 36 x 36 mosaic camea model, the adius and a coodinate system change. Povided with known coodinates of the scene points and thei images, the 36 x 36 mosaic camea paametes can be estimated. It is convenient to expess the elation between the scene point coodinates, the coesponding image coodinates, and the (unknown) 36 x 36 mosaic camea

5 I Left eye mosaic I Right eye mosaic A A II A A B B 3 II B B A A B B l 3 l (a) (b) 3 This is just an expanded fom of (8). A coect matix A has ank, because the last column is a linea combination of the fist, the fifth, and the ninth column. Howeve, due to the pesence of noise in eal data, the matix A may have ank 3. Unknown paametes foming the vecto h can be found using standad methods, such as SVD..3. Classes of Reconstuctions Figue 6. Effect of a change of the zeo elevation angle, see text fo a desciption. paametes in the following fom: Ah =, (8) whee matix A contains known point coodinates and vecto h is composed fom the unknown 36 x 36 mosaic camea paametes. Thee is only one intinsic paamete, the adius, needed to be estimated in ode to expess light ays as vectos in a 36 x 36 mosaic camea centeed coodinate system. A igid motion tansfomation is then used to identify these vectos with vectos in a scene coodinate system. Denoting coodinates of points in the 36 x 36 mosaic camea centeed coodinate system as x i = (x i, y i, z i, ) T, the elation between coodinates in the scene coodinate system x i = ( x i, ỹ i, z i, ) T can be geneally expessed as: ω i x i = H x i. (9) The matix H epesents both intinsic and extinsic 36 x 36 mosaic camea paametes and can be decomposed into matices K and M, H = KM, whee K = () contains 36 x 36 mosaic intinsic camea paametes, the adius, and the matix ( ) R t M = T () contains the extinsic paametes, whee the 3 3 matix R stands fo the otation and the 3 vecto t fo the tanslation. Notice that in ou case the fouth ow of the matix H equals (,,, ), theefoe we can wite (8) afte some eaanging in the fom: x T i T T x i T x T i T y i T T x T i z i. h h. h 3 =. () Anothe question, which should be answeed, is what kinds o classes of econstuctions ae possible when no calibation points ae povided and whethe something can be econstucted at all. It can be obseved fom ()-(3) that if the adius of the 36 x 36 mosaic camea is changed fom to, then the coodinates of the econstucted scene point x = (x, y, z ) can be computed fom the oiginal coodinates x = (x, y, z) as: x y z = x y z. (3) When no point coespondence is povided and the images ae not ectified, the eo in estimation of the zeo elevation angle leads to a nonlinea change in the z coodinate of the econstucted point. If the diffeence between and estimated zeo elevation angle is =, the following elation holds: z z d tan = d tan z + d, () whee d = x + y as depicted in Figue. Howeve, we can always ectify images when we can econstuct a single point because a one point coespondence is sufficient fo the ectification. Also, the zeo otation angle α can be one of the paametes of the 36 x 36 mosaic camea. It detemines the oientation of the 36 x 36 mosaic camea with espect to the 36 x 36 mosaic camea coodinate system. In this case, it s incoect estimation esults in a otation aound the z axis by an angle α. Theefoe, it can be coveed by a igid motion tansfomation of the coodinates. Putting it all togethe, the tansfomation descibing the elation between econstucted points and points measued in the 36 x 36 mosaic camea centeed coodinate system is just a scaling of all scene point coodinates (3). If the images ae not ectified then a nonlinea tansfomation () of the z coodinates of the points is added, howeve, only one point coespondence is needed in ode to ectify the images. In geneal, the D omnidiectional camea does not have to be calibated and the motion does not have to be cicula. Instead, they can be changed accodingly by an unknown

6 pojective tansfomation. Then the 36 x 36 mosaic camea will povide a pojective econstuction of the scene instead of a similaity econstuction. 3. Expeimental Results A slice camea [6] was used in the expeiments. The 36 x 36 mosaic camea was composed of a conical mio with the apex angle 9 and a telecentic lens mounted on a standad D CCD camea. Both the D CCD camea and the mio wee mounted on a motoized tuntable, see Figue (a), and known calibation points wee deployed in the scene. The D CCD camea mio ig was otated on a cicula path with the mio symmety axis pependicula to the axis of otation. An image was captued at each otation step and one cicle of pixels was extacted fom the image. The cicle was split into two half cicles which wee staightened into two half slices composing the two mosaics, the left half fo the left eye pojection and the ight half fo the ight eye pojection, see Figue 7. The tuntable was otated with Left eye mosaic Image acquied by a CCD camea Right eye mosaic Figue 7. Mosaic composition fom slices.. angula step which esulted in 8 snapshots fom which one cicle, which was one pixel wide, was extacted. The esolution of the two panoamic images was 8 pixels in the hoizontal and 878 pixels in the vetical diection. The esulting mosaic images fo the ight eye and the left eye pojection ae shown in Figue 9 and Figue espectively. The images ae aligned so that the most distant points (the doo on the back wall) lie in the same columns. Such an alignment allows the dispaity in the image pais to be clealy seen and was done only fo visualization puposes. A total numbe of 3 image pais wee selected fom all expeiments and images of the calibation points wee detected manually. Each of these image pais was acquied with a diffeent adius of viewing cicle. Then the scene coodinates of the calibation points wee computed. Figue 8 shows the econstucted points togethe with thei coodinates measued manually in the scene coodinate system. Reconstuction eo (the Euclidean distance between Measued points =mm =98mm =76mm Figue 8. Reconstucted points fom thee expeiments with diffeent adii compaed with measued points. the points measued manually in the scene and coodinates of the scene points computed fom manually detected image coodinates) fo each calibation point is shown in Figue. Thei espective point numbe is witten below each point. The distance affects the esulting eo. Geneally, the longe the distance fom the 36 x 36 mosaic camea cente, the lage is the influence of the coespondence detection eos on the pecision of the econstuction. Fo example, one pixel eo in the detection of points in the images coesponds to appoximately 5 mm eo fo the neaest scene points and 35 mm eo fo the most distant points. In this expeiment we did not explicitly calibated the D omnidiectional camea because it involves the calibation of the whole setup. Instead we used a squae pixel camea and caefully aligned all components of the setup. An explicitly calibated D camea would also impove the pecision of the econstuction. Figue illustates the econstuction eo befoe and afte the 36 x 36 mosaic calibation. Radius = mm was measued by hand and used as the initial 36 x 36 mosaic camea paamete. Calibation matix H was computed fom the odd calibation points and esults wee veified on the even points. It can be obseved that the oveall size of the eo deceased although some points have a bigge eo than befoe the calibation. This may be due to the fact that only five points wee used fo the calibation and thei eo (caused by noise, 36 x 36 mosaic camea assembly etc.) influenced the esult. Moe calibation points would impove the calibation pecision.. Conclusion A 36 x 36 mosaic camea model was pesented in this pape. A calibation pocedue was deived and a elation between econstuctions fom uncalibated views was de- 7

7 Figue 9. Right eye mosaic. Distance fom measued points [mm] 7 =mm =98mm =76mm Distance fom the camea [mm] Figue. Reconstuction eo, see text fo a desciption. temined. It tuns out that this 36 x 36 mosaic camea has a simple model than a pespective camea because thee is only one intinsic paamete fo a ectified pai of images and two othewise. It is also shown that only one point coespondence is equied fo the ectification. The Relation between scene points econstucted fom an uncalibated 36 x 36 mosaic camea and measued points in the scene is a similaity instead of a geneal pojective tansfomation as it is the case of a pespective camea. The expeimental esults coolate the pesented theoy. Refeences [] S. Bake and S. K. Naya. A theoy of single-viewpoint catadioptic image fomation. Intenational Jounal of Compute Vision, 35():75 96, 999. [] A. M. Buckstein and T. J. Richadson. Omniview cameas with cuved suface mios. In IEEE Wokshop on Omnidiectional Vision (OMNIVIS ), pages 79 8, June. [3] J. Chai, S. B. Kang, and H.-Y. Shum. Rendeing with nonunifom appoximate concentic mosaics. In Second Wokshop on Stuctue fom Multiple Images of Lage Scale Envionments, SMILE, July. [] C. Geye and K. Daniilidis. A unifying theoy fo cental panoamic systems and pactical implications. In D. Venon, edito, Euopean Confeence on Compute Vision ECCV, volume, pages 5 6, June-July. [5] R. A. Hicks and R. Bajcsy. Catadioptic sensos that appoximate wide-angle pespective pojections. In IEEE Wokshop on Omnidiectional Vision (OMNIVIS ), pages 97 3, June. [6] S. K. Naya and A. Kamaka. 36 x 36 mosaics. In IEEE Confeence on Compute Vision and Patten Recognition (CVPR ), volume, pages , June. 7 Distance fom measued points [mm] Figue. Left eye mosaic. Estimated paametes, mean=39.3 Calibated camea, mean= Clibation point numbe Figue. Reconstuction befoe (adius = mm was measued by hand) and afte the 36 x 36 mosaic calibation. [7] T. Pajdla. Epipola geomety of some non-classical cameas. In B. Lika, edito, Poceedings of Compute Vision Winte Wokshop, pages 3 33, Febuay. [8] T. Pajdla, T. Svoboda, and V. Hlava c. Epipola geomety of cental panoamic cameas. In R. Benosman and S. B. Kang, editos, Panoamic Vision : Sensos, Theoy, and Applications. Spinge Velag, Belin, Gemany, st edition,. [9] S. Peleg, M. Ben-Eza, and Y. Pitch. Omnisteeo: Panoamic steeo imaging. PAMI, 3(3):79 9, Mach. [] S. Seitz. The space of all steeo images. In J. Little and D. Lowe, editos, ICCV : Poc. Eighth IEEE Intenational Confeence on Compute Vision, volume, pages 6 35, July. [] H.-Y. Shum, A. Kalai, and S. M. Seitz. Omnivegent steeo. In Poc. of the Intenational Confeence on Compute Vision (ICCV 99), volume, pages 9, Septembe 999.