Disparity Fusion Using Depth and Stereo Cameras for Accurate Stereo Correspondence

Transcription

1 Disparit Fusion Using Depth and Stereo Cameras for Accurate Stereo Correspondence Woo-Seok Jang and Yo-Sung Ho Gwangju Institute of Science and Technolog GIST 123 Cheomdan-gwagiro Buk-gu Gwangju Republic of Korea ABSTRACT Three-dimensional content 3D creation has received a lot of attention due to numerous successes of 3D entertainment. Accurate stereo correspondence is necessar for efficient 3D content creation. In this paper we propose a disparit map estimation method based on stereo correspondence. The proposed sstem utilizes depth and stereo camera sets. While the stereo set carries out disparit estimation depth camera information is projected to left and right camera positions using 3D transformation and upsampling is processed in accordance with the image size. The upsampled depth is used for obtaining disparit data of left and right positions. Finall disparit data from each depth sensor are combined. In order to evaluate the proposed method we applied view snthesis from the acquired disparit map. The eperimental results demonstrate that our method produces more accurate disparit maps compared to the conventional approaches which use the single depth sensors. Kewords: 3D warping depth fusion disparit estimation stereo correspondence 1. INTRODUCTION With the huge success of three-dimensional 3D movies interest in 3D entertainment sstems is increasing recentl. 3D multimedia applications give audiences the opportunit to eperience 3D. The 3D eperience comes from the left and right ees seeing different views. Thus audiences can perceive 3D using two separate views for the left and right ees. For this several data formats have been proposed. The teture plus depth approach is one of these formats. This format uses an ordinar 2D image accompanied b a depth map. The non-eisting view can be snthesized b depth image based rendering DIBR [1]. Benefits of this format are the fleibilit to render views with variable baseline and the increased compressibilit of depth data due to its characteristics [2]. Thus this format is practical for man 3D multimedia applications. Depth information can mainl be acquired b two approaches: active and passive sensor based depth estimation methods. The former emplos phsical sensors such as infrared ra IR laser and light pattern to measure depth data directl. Depth cameras structured light sensors and 3D scanners are emploed in this approach [3]. Usuall active sensors are more effective than passive sensors in terms of the qualit of produced depth images. However the produce onl low resolution images and generall require epensive devices. The latter on the other hand indirectl estimates depth information from 2D images captured b at least two cameras [4]. Stereo matching is the most widel used passive sensor based method [5]; its advantages are low cost and fleible resolution. However passive sensors also obtain miscalculated depth information in several tpes of regions. Disparit data can be converted into depth information b using a stereo image pair in combination with triangulation [6]. Disparit data can be acquired b finding the corresponding points in other images for piels in one image. The correspondence problem is to compute the disparit map which is a set of the displacement vectors between the corresponding piels. For this problem two images of the same scene taken from different viewpoints are given and it is assumed that these images are rectified for simplicit and accurac of the problem. From this assumption corresponding points are found in same horizontal line of two images. A disparit map acquired b stereo matching can be represented b a gra scale image. Depth of each piel is perceived from the disparit map. The object is close to viewpoint as intensit value of a piel in the disparit map is high. The objective of this paper is to obtain accurate depth information using depth and stereo images. Thus we present an accurate disparit map acquisition method through stereo correspondence. We design a disparit estimation sstem to

2 strengthen the merits and make up for the weaknesses for active and passive depth sensors. The proposed method deals with full unsolved problems b fusing and refining the depth data. 2. PROBLEM STATEMENT Over the past several decades a variet of stereo-image-based depth estimation methods have been developed to obtain accurate depth information. However accurate measurement of stereo correspondence from natural scene still remains problematic due to difficult correspondence matching in several regions: tetureless periodic teture discontinuous depth and occluded areas [7]. First since color data of the tetureless and periodic teture region in left and right images are so similar each other in a wide range correspondence matching often fails because of its ambiguit. Second in case of the depth discontinuous region such as the edge region smeared color values eist which leads to ineffective correspondence matching. Lastl in the occluded region some piels ma appear in one image but not in the other image; So there is no corresponding piel. These problems can be solved for accurate depth information. Figure 1 illustrates correspondence matching problem in stereo matching. a Left image Figure 1. Correspondence problem in stereo matching b Right image Usuall depth cameras are more effective in producing high qualit depth information than the stereo-based estimation methods. However depth camera sensors also suffer from inherent problems. Especiall the produce low resolution depth images due to challenging real-time distance measuring sstems. Such a problem makes depth cameras not practical for various applications. Figure 2 represents resolution difference between the regular color camera and depth camera. Recent approaches of fusing active and passive sensors have shown improvements on depth qualit b making up for weakness of each sensor method [8 9]. In our work we propose a disparit fusion method to make up for weakness of stereo-based estimation and carr out better correspondence matching b adding a depth camera to the stereo sstem. 3. DEPTH FUSION SYSTEM FOR STEREO CORRESPONDENCE In general stereo matching can be categorized into two approaches: local and global methods. Local methods are generall efficient for compleit [10]. However these make blurred object borders and the removal of small detail at the depth discontinuit depending on the size of correlation window. In order to solve this problem global methods have

3 been proposed [11]. Global methods define an energ function b Markov Random Field MRF and optimize this function using several optimization algorithms such as belief propagation [12] and graph cut [13]. Figure 2. Resolution problem in depth camera The proposed method is initiall motivated b global stereo matching [14]. The information of the depth camera is included as a component of the global energ function to acquire more accurate and precise depth information. Depth camera processing is implemented as follows: 1 Depth data is warped to its corresponding position of the stereo views b 3D transformation. 3D transformation is composed of two processes. First the depth data is backprojected to the 3D space based on the camera parameters. Then the backprojected data in the 3D space is projected to the target stereo views. 3D transformation is performed as follows. T -1 T z = RsrcAsrc u v1 du v + tsrc 1 T -1 T l m n = AdstRdst z - tdst 2 u ' v' = 1/ n m / n 3 where A src R src and t src are intrinsic rotation and translation parameters in the depth camera data respectivel. Similarl A dst R dst and t dst are those in the target color view. d uv is depth value at uv coordinate in the depth camera. The depth data is sent to the 3D space b Eq. 1 and projected to the target view b Eq. 2. u v in 3 represents the projected coordinate to the target view. Figure 3 shows the 3D transformation of depth data. 2 depth-disparit mapping is processed [15]. Due to the different representation of the actual range of the scene correction of the depth information is required. 3 We perform joint bilateral upsampling JBU to interpolate the low resolution depth data. This method used high resolution color and low resolution depth image to increase the resolution of the depth image. Figure 4 illustrates JBU process. The processed information of the depth camera is applied as the additional evidence for data term of disparit fusion energ function. The disparit fusion energ function for accurate disparit estimation is defined as E = å Edata + Esmooth 4

4 Figure 3. 3D transformation of depth camera data Figure 4. Joint bilateral upsampling process where E data is a data term which measure the piel similarit and E smooth is the smoothness term which penalizes depth variations. The following function is applied to data term for energ function of depth estimation. 1 ] [ d I I k d d d I I k d E R L up R L data = b a 5 I L is the piel value in the left image given coordinate. I R d is the matched piel value in the right image given the disparit value at in the left image denoted b d. d up is the upsampled disparit data obtained from the

5 previous step. The upsampled disparit information enhances the precision and accurac of the final depth values b allowing large depth variation. The smoothness term is based on the degree of difference among the depth values of neighboring piels. å E = d - 6 smooth d t tîn N represents the neighboring piels of the current piel. The algorithm is hierarchicall processed to acquire more accurate depth values in the tetureless region. Furthermore we appl post-processing to the acquired disparit map to improve the qualit [16]. Figure 5 illustrates the overall framework of the proposed method. Stereo images Depth camera data Camera Calibration Camera Calibration 3D transformatio n Depthdisparit mapping Disparit upsampling Fusion depth estimation Depth refinement Final disparit map Figure 5. Overall framework of depth fusion sstem

6 4. EXPERIMENTAL RESULTS In order to evaluate the performance of the proposed method we compare our proposed method with the stereo-imagebased and depth upsampling methods. The are captured at the resolution of piels. The depth camera emplos the resolution piels. Camera calibration was applied to these data sets for the accurac of the processes. Figure 6 shows the disparit results of JBU stereo matching and proposed fusion method. a JBU b Stereo matching c Proposed fusion Figure 6. Overall framework of depth fusion sstem The JBU results represent that boundar is not good but we can obtain the precision disparit data especiall in regards to homogeneous region. On the other hand the stereo matching method cannot produce disparit detail in homogeneous region. We compare view snthesis results for calculating disparit accurac. Figure 7 shows the snthesized results for each method. The results indicate that the proposed method outperforms other comparative methods. The visual comparison of the eperimental results demonstrates that the proposed method can represent the disparit detail and improve the qualit in the vulnerable areas of stereo matching.

7 a JBU b Stereo matching c Proposed fusion Figure 7. View snthesized results 5. CONCLUSION This paper presents a novel stereo disparit estimation method eploiting a depth camera. The depth camera is used to supplement crude disparit results of stereo matching. The camera arra is determined to reduce the inherent problems of each depth sensor. Furthermore the disparit acquisition algorithm deals with full unsolved problems b fusing and refining the depth data. This increases the precision and accurac of the final disparit values b allowing large disparit variation. ACKNOWLEDGEMENT This research was supported b the Cross-Ministr Giga KOREA Project of the Ministr of Science ICT and Future Planning Republic of KoreaROK. [GK13C0100 Development of Interactive and Realistic Massive Giga-Content Technolog]

8 REFERENCES [1] L. Zhang and W. J. Tam Stereoscopic image generation based on depth images for 3DTV IEEE Transactions on Broadcasting vol. 51 no. 2 pp June [2] G. Tech K. Muller and T. Wiegand Evaluation of view snthesis algorithms for mobile 3DTV 3DTV Conference pp [3] E. K. Lee and Y. S. Ho Generation of multi-view video using a fusion camera sstem for 3D displas IEEE Transactions on Consumer Electronics vol. 56 no. 4 pp [4] W. S. Jang and Y. S. Ho Efficient disparit map estimation using occlusion handling for various 3D multimedia applications IEEE Transactions on Consumer Electronics vol. 57 no. 4 pp [5] D. Scharstein R. Szeliski and R. Zabih A taonom and evaluation of dense two-frame stereo correspondence algorithms International Journal of Computer Vision vol. 47 pp [6] R. Hartle and A. Zisserman Multiple View Geometr in Computer Vision 2nd ed. Cambridge Universit Press 2003 pp [7] W. S. Jang and Y. S. Ho Discontinuit preserving disparit estimation with occlusion handling Journal of Visual Communication and Image Representation vol. 25 no. 7 pp Oct [8] E. K. Lee and Y. S. Ho "Generation of High-qualit Depth Maps using Hbrid Camera Sstem for 3-D Video" Journal of Visual Communication and Image Representation vol. 22 paper Issue 1 pp [9] Y. S. Kang Y. S. Ho Generation of Multi-view Images Using Stereo and Time-of-Flight Depth Cameras International Conference on Embedded Sstems and Intelligent Technolog pp [10] Hirschmuller P. R. Innocent and J. M. Garibaldi Real-time correlation-based stereo vision with reduced border errors International Journal of Computer Vision vol. 47 no.1/2/3 pp Apr [11] O. Veksler Fast variable window for stereo correspondence using integral images IEEE Computer Societ Conference on Computer Vision and Pattern Recognition vol 1 pp [12] J. Sun N. N. Zheng and H. Y Shum "Stereo matching using belief propagation" IEEE Transactions on Pattern Analsis and Machine Intelligence vol. 25 no. 7 pp Jul [13] Y. Bokov O. Veksler and R. Zabih Fast approimate energ minimization via graph cuts" IEEE Transactions on Pattern Analsis and Machine Intelligence vol. 23 no. 11 pp Nov [14] Q. Yang L. Wang and N. Ahuja A Constant-Space Belief Propagation Algorithm for Stereo Matching Computer Vision and Pattern Recognition pp June [15] Y. S. Kang Y. S. Ho High-qualit Multi-view Depth Generation Using Multiple Color and Depth Cameras International Workshop on Hot Topics in 3D pp [16] Q. Yang C. Engels and A. Akbarzadeh Near real-time stereo for weakl-tetured scenes British Machine Vision Conference pp