Stereo Imaging Using a Camera with Stereoscopic Adapter

Transcription

1 Stereo Iaging Using a Caera with Stereoscopic Adapter Woontack Woo, agyu Ki* and Yuichi Iwadate** ATR MIC Labs, Kyoto , Japan Eail:wwoo { ngki,yiwadate ic.atr.co.jp ABSTRACT In this paper, we analyze the characteristics of the stereoscopic adapter, which is a cost-effective way to generate stereo video sequences with a caera. We also propose an efficient way to copensate for the inherent distortions. In general, stereo sequences can be captured using a pair of caeras, but the resulting sequences tend to yield various well-known probles due to different characteristics of the pair of stereo caeras. Meanwhile, a caera with the stereoscopic adapter provides a natural way to capture and display stereoscopic video. It allows users to access all the functions built into the caera, e.g. zoo, auto-focus, auto-eposure, special effects, etc. The cost however is the reduced quality of the videos since the adapter allows capturing stereo video sequences in the field sequential forat, i.e. left and right iages in different scan lines, respectively. In addition, it generates size and color distortions due to the physical configuration of the irror in the adapter. We analyze and copensate for such distortions to reduce possible errors in vision applications eploiting the stereo iages. According to our preliinary study, the adapter with the proposed copensation schee will pave the way for various low-cost iage-based virtual reality applications at hand. Inde ters - stereo caera, calibration, rectification, disparity estiation, stereoscopic adapter. I. ITRODUCTIO Eploiting stereo iages provides various advantages over using an iage. For eaple, stereo vision provides 3D inforation, such as orientations and distances, of the objects in the scene. In addition, well-designed stereoscopic displays convey a very copelling sense of 3D depth. Although 3D perception can be achieved through various other cues (such as geoetric perspective, relative size, shade, teture gradient, occlusion, otion, disparity, etc.), binocular depth perception is considered to be uch ore powerful. Even in 2D display environents, eploiting the 3D depth inforation can lighten the burden of iagevideo processing, analysis and counication in various levels, e.g. help segent objects fro the background [1,11], which has been one of the hardest coputational vision probles. *. Ki is a Ph.D. candidate at Pohang University of Science and Technology (POSTECH), Pohang, Korea. ** Y. lwadate is now with HK Science and Technology Research Labs, Tokyo, Japan. Eail: yiwadate@strl.nhk.or.jp The difficulties in stereo iaging ainly ste fro capturing well-controlled stereo iages, which is a key step toward accurate depth estiation. In general, stereo iages can be captured using a pair of stereo caeras, where each caera captures a scene fro a slightly different perspective. However, several well-known probles arise fro capturing stereo iagesvideo sequences, since two caeras will generally have slightly different physical characteristics. Without accurate caera calibration, we ay fail to estiate accurate 3D inforation and to provide realistic 3D effects on the screen. Meanwhile, a caera with a stereoscopic adapter, e.g. uview syste, can be considered as a new way to capture stereo iagesvideo sequences [1]. The optical adapter, placed in front of the lens of a caera, allows for the caera to capture stereo video sequences. As a result, it can alleviate the probles, which arise fro the different characteristics of a pair of stereo caeras. It also allows users to access all the functions built into the cacorder, e.g. zoo, auto-focus, auto-eposure, special effects, etc. ote however that the cost of this single lens-based approach is the reduction in quality of the resulting stereo video sequences. The adapter captures each iage of the stereo pair in the different line, i.e. field sequential forat. In addition, inherent distortions due to the irror in the stereoscopic adapter ay deter the usage of the resulting stereoscopic iages in various applications eploiting 3D depth inforation. Therefore, a siple yet successful way to copensate for these distortions is essential for the adapter to be used in real vision applications. In this paper, we focus on copensation for these 3D distortions in vision applications, rather than the 3D display itself. However, it is worthy noting that copensation is also essential for cofortable stereoscopic 3D display. First we analyze the characteristics of the adapter, then we eplain how to copensate for the resulting distortions based on analyzed results. To analyze the characteristics of the adapter, we perfor a three-step procedure as follows. First, we separate the test sequence in the field sequential forat into the abovebelow forat and then transfor to side-byside forat by bilinear interpolation. Second, we rectify the iage, i.e. copensate for the size distortion, by using the paraeters obtained fro the Tsai algorith [8]. Finally, we copensate for the color degradation, which is inevitable due to the projection through the irror in the adaptor, based on the analyzed statistics. According to our preliinary study, the stereoscopic adapter with the proposed $1. 2 IEEE 1512

2 copensation schee will pave the way for various low-cost iage-based virtual reality applications at hand. This paper is organized as follows. In Section 2, we introduce a way to capture stereo video sequences using a caera with the stereo adapter. In Section 3, we analyze the characteristics of the stereoscopic adapter and eplain how to copensate for the resulting ebedded distortions. Soe eperiental results and discussion are given in Section 4 and 5. II. STEREOSCOPIC VIDEO WITH A CAMERA A. Field Sequential 3D Video As shown in Figure 1, the stereoscopic adapter (e.g. u- View syste [1]) consists of a sturdy black plastic housing, a reflecting irror and liquid crystal shutters (LCS). The prisatic bea splitter and the orthogonally positioned polarizing surfaces (1.45"l.25") in the LCS open and close the light valves, to record either the direct iage or the irror reflected iage on alternate fields of the video. As a result, the left iage is recorded during the "odd" field and the right iage during the "even" field, or vice versa. As shown in Figure 1, the synchronization of the light valves with the alternating fields of the cacorder is achieved through the cable connecting the video-out of the cacorder and the connector in the adapter. ~;;t~}; I':,,._J u View Adapter llaterla od Forat I VV LEFT RIGHT AboveBelow F ~ t VV Sld~ b~- gt,~e ydrn~t Figure 1. Capturing stereo video sequences using a caera with stereoscopic adapter. B. FieM Separation and Interpolation It is necessary to convert the video sequence to abovebelow forat. As eplained earlier, the adapter produces a field sequential stereoscopic 3-D video by siultaneously recording the second eye view to the cacorder. The resulting field sequential video can be display on a 2D (TV) onitor or 2D screen with special stereo glasses. The field sequential forat, however, is an inconvenient forat to be used in various other vision applications. For eaple, applying processing, such as filtering or transforation, to the field sequential video can cause a loss in quality of the stereo iages because such processing propagates the effects into the interlaced lines and thus produces 3D artifacts. For the sae reason, the available video copression schee cannot be eploited to save the hard disk space or liited channel bandwidth. Therefore, we first separate the field sequential forat to abovebelow forat, where the left iage is placed to the top part of the iage and the right iage to the botto part, or vice versa. After the field separation, we transfor the iage to side-by-side forat. We now need to perfor teporal or spatial interpolation to each iage to provide a high quality of 2D3D iagesvideo sequences. We eperience the flickering effects when we display the stereoscopic 3D videos, which are captured using the adapter in 6Hz. The stereoscopic 3D video in 6Hz is not as sooth as copared to the 2D video in 6Hz, because the 2D onitor allocates 3Hz to the right iage and the other 3Hz to the left iage. In addition, displays (such as head-ounted display, polarized screen or auto-stereoscopic display) require projecting an iage in the original size to provide cofortable 3D display. The spatial interpolation is also required in 2D applications eploiting only 3D depth inforation (e.g. z-keying). The spatial interpolation is achieved by line copy, doubling of the size, or linear interpolation between lines as follows. F? : rzi+, = (G L + G~+, )2 (1) where FL and GL denote the left iages in side-by-side and abovebelow forats, respectively. The superscript i represents the inde of the row in the iage. The right iage can be interpolated in a siilar way. III. DISTORTIO AD COMPESATIO A. Geoetry of a Pinhole Caera Figure 2 shows the basic geoetry of the pinhole caera odel and its projection fro the 3D world coordinate of a feature point (in ra) [Xw Yw Zw] to 2D iage coordinate (in piels) [u v]. The 3D caera coordinate is centered at the optical center,,., with the cz-ais being the sae as the optical ais. The iage coordinate is centered at the intersection point with the optical ais in the iage plane. The distance between the optical center and the iage plane is the focal lengthf We first transfor the augented 3D world coordinate of the object [Xwyw zw l]r to the 3D caera coordinate [c y,. zc] r. The transforation can be perfored by a 33 rotation atri R and a 3l translation vector T=[t ty tz] r, as shown in (2). 1513

3 where the rotation atri R is represented as a product of three 33 rotation atries, i.e. RgReR ~, corresponding to the rotation along roll (), pitch () and yaw (~), respectively. R= ii c o -s o llc:oo,,, s, s, c!1 s cojl-=o ceil where s and c denote sine and cosine, respectively. The 3D caera coordinates are projected onto the augented ideal iage coordinate [i Ye 1] T by the perspective projection under pinhole caera odel. =± j [i], zc.i: o,,, (3) L o o llzcj where f denotes the effective focal length of the caera. As usual, the origin of the iage coordinate is in the upper left corner of the iage array and the unit of the iage coordinate is not "eter" but the nuber of "piels". Therefore, the augented actual (or coputer) iage coordinates in piels [ut v, 1] T are obtained fro [i Yi 1]r by applying the following transforation. I;]I u o.olr,1 = Sy vy, (4) o o ljl1 j where the (u~ vo) denotes the offset (pieis) in the iage and s denotes the scaling (piels). The scaling is defined as, s=a k d Jp, where d is the center-to-center distance between adjacent sensor eleent, and s and p, respectively, denote the nuber of sensor eleents in horizontal (vertical) direction in the CCD and the nuber of piels in an iage scan (vertical) line. ote that, in general, both the uncertainty paraeter a and the lens distortion paraeter k are set to one, but in a real situation both should be introduced [2,8]. By cobining (2)-(4), the 3D world coordinate is related to the 2D iage coordinate and vice versa. ii q j--- ~., ~ zw) a caera is perfored by observing a calibration object, e.g. one or two plane odel, whose geoetry is known with a very good precision with respect to a 3D coordinate syste attached to this apparatus. The resulting caera paraeters allow us to learn the distortions that occurred during projection through the stereoscopic adapter. In general, standard stereo caera calibration techniques follow 3-step procedures. First, they establish a list of 3D world coordinates and corresponding 2D iage coordinates. Given the list, caera paraeters are estiated for each caera using a set of equations. Finally, the epipolar geoetry is constructed fro the projection atrices. We apply Tsai algorith to deterine the distortion odel and odel paraeters [Tsai98]. The algorith estiates 11 odel paraeters: five intrinsic (also called internal or interior) and si etrinsic (also called eternal or eterior) paraeters. The intrinsic caera paraeters include the effective focal length f, the first order radial lens distortion coefficient 1 l, the principal point (the center of radial lens) [c, Cy], the scale factor to account for any uncertainty due to frae grabber horizontal scanline resapling s. The etrinsic paraeters include the rotation atri R (rotation angles for the transfor between the world and caera coordinate fraes) and transforation atri T (translational coponents for the transforation between the world and caera coordinate fraes). C. Stereo Rectification Figure 3 shows the epipolar geoetry between a pair of iages. In 3D video shooting, the tradeoffs between 3D effects and 3D distortions are unavoidable. The objects at the convergence point, corresponding to zero-disparity, appear on the 2D screen and others will appear with relative depth. As the convergence point oves toward the caera, the 3D distortions such as "keystone effects" increase [9]. Even if we ove the convergence point backward, it would be very difficult to eliinate such distortion at all object points. If the convergence point is not in the infinite point, the stereoscopic adaptor causes the rotation, as well as translation, of the virtual caera. As a result, epipolar lines are not aligned with coordinate ais and are not parallel, wh!c!~ a!~es.c!.!sp~.ity, est!at!on c!!ff!cu!t~...,y,z) i Oc i Optical ais Figure 2. Caera geoetry and iage projection. B. Stereo Calibration ow we are ready to investigate the virtual caera configuration of the caera with stereoscopic adapter. We first perfor caera calibration to capture the relationship between the 3D world coordinate and its 2D perspective projection onto the virtual stereo caeras. The calibration of... CR Figure 3. Epipolar geoetry between a pair of iages. Once the caera odel paraeters have been identified, we copensate for the size distortion by an operation known as stereo rectification [4-5]. The rectification brings the two iage planes to be coplanar to a coon plane in space [7]. Let C L and C R be a pair of pinhole caeras in 3D space. Each 1514

4 caera is represented by a 34 hoogeneous transfor atri, H=[PI -PC], where the vector C denotes the position of the caera's optical center. The 33 projection atries, P~. and PR, between the augented 3D world coordinates O and the iage coordinates, XL and XR, can be calculated using Tsai algorith. The rectification can be accoplished by the transforation atri obtained fro the relationship between the two atrices, P~. and PR [7]. Given two projection atri, HL=IPLI -PLCL] and HR=[PRI -PRCR], the corresponding points in the iages, L and XR, are related by XR=PRPL'IXL. The projective atri, PRPL -j, projects the iage plane of IL to that of IR. More generally, the atri, PL.PL -j, represents a planar projective transforation onto a new iage 1.'. It is convenient to choose the world coordinate syste so that both CL and CR lie on the world X-ais, i.e. Q=[XL ] r and CR=[XR ] r. The reaining aes are chosen in a way that reduces the distortion incurred by iage reprojection. After a proper rectification process, the rectified iages have the following properties; all epipolar lines are parallel to the horizontal scan line and thus corresponding points have identical vertical coordinates. As a result, the atching process of two iages can be siplified and efficient. D. Color Histogra Modification Before we eploit stereo iages, we need color odification. The color distortion occurs due to another inherent weakness of capturing stereo video with the stereoscopic adapter. The orthogonally positioned polarizing surfaces in the adapter yield stereo video sequences with different levels of color. ote that color equalization not only allows cofortable stereoscopic 3D display, but also helps to estiate accurate 3D depth inforation, especially when the depth is estiated based on the intensity level. To equalize the color levels of both iages in the stereo pairs, we use 3 test pairs, where each pair contains only one color, i.e. red, green or blue. We first select the region of interest fro the given pairs of stereo iages and then estiate statistics regarding the color distortion. Given the statistics, we noralize and odify the color histogras. The new intensity (color) of the left iage, the projected iage through the irror, FLU, is noralized using the forula defined as follows. err (FL--L)+R (5) FL- = f~l where F, o and represent the iage, standard deviation and average, respectively. The subscript L and R denote left and right iages, respectively. E. Dispari~. Estiation Disparity estiation is recognized as the ost difficult step in stereo iaging. The task of disparity estiation in a 3D reconstruction is to find correspondence in a pair of stereo iages and estiate 3D position by triangulation. Many estiation algoriths have been proposed but the resulting disparity is not accurate enough to be used in real applications. To overcoe the weakness of available estiation schees, we adopt a hierarchical block atching schee with hybrid cues, which eploit edge, as well as intensity siilarity, based on MRF fraework [1-12]. We start with a block to aintain robustness of the estiation and then segent the block. Since the estiation error within the block is non-unifor, we segent the block according to the estiation error and variance level. The edge inforation is eploited to estiate accurate disparity along the object boundaries. IV. EXPERIMETAL RESULTS After ounting the u-view adapter on the SOY TRV9, we capture test patterns and sequences in field sequential forat. We then digitize the sequences using the video card, DV Rapter, with the capturing software, Adobe Preier. To aintain a stereo sync signal, the captured iages are set to full size (7248 at 3. fps) [1]. The even lines contains the picture for left-eye and the odd lines for right eye. As eplained in Section II and III, we first transfor the iages in the field sequential forat to the side-by-side forat and then interpolate using (1) to recover the original size. In our eperients, a non-coplanar pattern is used to estiate caera calibration paraeters. As shown in Figure 4, the 12 rectangles in the calibration wedge are projected on to the 2D iage plane. The corner points of the rectangles on the wedge are projected onto iage planes and the iage coordinates corresponding to the calibrated points on the wedge are deterined using siple iage processing techniques. The calibration is perfored using Tsai algorith [8]. The resulting caera paraeters are shown in Table 1. Figure 5 shows the resulting virtual stereo caera configuration of the caera with the stereoscopic adapter. As shown in Figure 5, the virtual caera is not positioned on the sae plane as the reference (right) caera. The set back of the virtual (left) caera results in the size distortions in a pair of stereo iages and the rotation of the caera yields additional "keystone" error. According to our eperients, the virtual caera is set back about 5 fro the reference caera due to the irror in the adapter. The closer convergence point causes ore size distortion. The set back and rotation of the virtual caera is copensated for using the caera paraeters obtained by Tsai algorith, which are shown in Table 1. After the caera calibration and iage rectification, we perfor color odification using (5). To analyze the characteristics of color distortion, we capture three test iages, each containing only one color coponent, i.e. red, green and blue, respectively. Figure 6, (c) and (e) show the histogras of uncopensated iages for each color coponent. As shown in Figure 6, (d) and (f), the left video sequences can be copensated by using (5) for red, green and blue coponents, respectively. ote that the statistics (ean and variance of each sequence) can be estiated during caera calibration process and be applied on the fly. 1515

5 Figure 7 and show a pair of stereo iages in field sequential forat and abovebelow forat, respectively. We first rectify the iages using the calibration paraeters to copensate for size distortion. We then odify each of the color coponents since, as epected, the pair of stereo iages has slightly different color levels. Figure 8 copares cuulative histogras of the original and the copensated pairs of stereo iages, in ters of each color coponent, i.e. red, green and blue, respectively. According to our eperiental results on the outdoor scenes, the polarization of LCD in the adapter causes color shift in the red coponent on the right iage. The coparison between the original and the calibrated right iage is shown in Figure 9. The calibration and rectification will iprove disparity estiation draatically. V. DISCUSSIO We first analyzed the characteristics of the stereoscopic adapter, and then proposed an efficient schee to recover the original quality of video. As shown in our eperiental results, in addition to the classical lens distortions, the stereoscopic adapter generates different kinds of distortions in ters of size and color. Those distortions can be copensated for by the stereo rectification and color odification. According to our preliinary study on the characteristics of the stereoscopic adapter, we believe the adapter with the proposed copensation schee can provide a practical solution to capture 3D video at hand with reasonable quality. As a result, with the portability and effectiveness, the adapter with the proposed copensation schee will play a key role in generating iage-based photorealistic virtual environent with depth inforation. The reaining work for the adapter to be used in real applications is to analyze the effects according to the changing focused point and zooing environents [8]. [6] J. Park and C. Lee, "Robust Estiation of Caera Paraeters fro Iage Sequence for Video Coposition," Signal Processing: Iage Counication, vol. 9, pp.43-53, [7] S. Seitz and C. Dyer, "View Morphing," SIGGRAPH 96, pp. 21-3, [8] R. Y. Tsai, A versatile Caera Calibration Technique for High- Accuracy 3D Machine Vision Metrology Using Off-the-Shelf TV Caeras and Lenses", IEEE Journal of Robotics and Autoation, Vol. RA-3, o. 4, August 1987, pages [9] W. Woo, "Rate Distortion Based on Dependent Coding for Stereo Iages and Video: Disparity Estiation and Dependent Bit Allocation," Ph.D. Dissertation, USA LA, CA, USA, [1] W. Woo and Y. iwadate, "Object-oriented Hybrid Segentation Using Stereo Iages," Proc. IVCP', Jan. 2. [11] W. Woo and A. Ortega, " Overlapped Block Disparity Copensation with Adaptive Windows for Stereo Iage Coding," 1EEE Tr. on CSVT, vol. 9, no.6, pp , Mar. 2. [12] W. Woo,. Ki and Y. lwadate, "Object Segentation for Z- keying Using Stereo Iages," in Proc. WCC-ICSP', Aug. 2. Figure 4. Test pattern for stereo calibration and rectification. (right iage, 7224) field sequential forat abovebelow forat. \ Acknowledgeent The authors would like to thank Mr. David Chersky and 3-D Video, Inc., CA, USA, for providing the u-view stereoscopic adaptor used in this paper. References \ [1] u-view Stereoscopic Adapter, 3-D Video, Inc., dvideo.co [2] C. Cheung and W. Brown, "3D Shape Measureent using Three Caera Stereopsis in Optics," Proc. SPIE Illuination and Iage Sensing for Machine Vision II, vol. 85, pp , [3] U. Dhond and J. Aggarwal, "Binocular Versus Trinocular Stereo," in IEEE Proc. Int. Conf. on Robotics and Autoation, pp , 199 [4] O. Faugeras, " Three-diensional Coputer Vision: A Geoetric Viewpoint," MIT Press, Cabridge, MA, [5] C. Loop and Z. Zhang, "Coputing Rectifying Hoographies for Stereo Vision," in IEEE Proc. Int. Conf. on Coputer Vision and Pattern Recognition, vol. 1, pp , Jun Figure 5. Virtual caera configuration. before calibration after calibration. 1516

6 i Left Right Left Right f [en] 68.7( R [deg] 151.~ kl [l^2] 2.5E-5! 4.5E-5 Ry [deg] -42.5S C lpis] 359.6! Rz [deg] Cy [pils] ; T [nun] s Ty [nun] Tz [] Table 1. The resulting caera paraeters. right caera virtual left caera. Red Coponent lo8o... i... i... o... i... ~... I... i... i... p i 4... ~- --~...," 2... ~... ~... I Red Coponenl ~oooo... ~... u 8... ~...?... ~... P, 4... i... 7 $ 2...!... i i " -; I... Cqlo F Le~t... ~lp~l~l {.a~ GenCoponent ~ u P i 4... ~... '... ~... T o ~ooo... i... K.5"-i'"-i... i o i...! $ ColorLevel ~encornponb~n& :...,... u 8... ~...? i... ~ 4... i... Color L~l... (c)... (d)... Blue Copone~ 1... :... :... Blue Coponent Red Coponent,ooo... i... i... 8o i i, i i i 6... :4oo i 7i i i!~ooo o... i._.i... i... i... _ ? 25 Color L.ev~l C~een Coponent I...!... :... ;... ~...,- ooo... i.... : ~J~... l... ~... --;,-...~...!... P : : :,, :,< 4 i ff T s : :-v: ~ u 8 rn P GO i 4 e I s 2C d (c) Blue Coponent Color Leve4 "i Red Coponent 1oo... i... T ~ 8ooo ~ ~ 6 if> 4 " s 2... i,... i_ i i L (e) (o Figure 8. Coparison of Histogra. red-original red-odified (c) green- original (d) greenodified (e) blue- original (f) blue- odified. Color Lev61 Green Coponent I OOO... : ii i:: ii: iiiiiiiii,~8ooo... L...i... i... i... ~,. IP :'('~ " (d) ue Coponent 1... i : '... i... i... i u 8... : i... p i i 4... ~... ;... 8ooo... i --:... :...!..., Color Le~,sH (e) ( Figure 6. Histogra of test color iages. red-left red-right (c) green-left (d) green-right (e) blue-left (f) blue-right.!... (a!... Figure 9. Color copensation. (left iage, 7248) the original the copensated iage. Figure 7. Test iages (7248). field sequential forat abovebelow forat. 1517