Towards Real-time Stereo using Non-uniform Image Sampling and Sparse Dynamic Programming

Transcription

1 Towards Real-time Stereo using Non-uniform Image Sampling and Sparse Dynamic Programming Michel Sarkis and Klaus Diepold Institute for Data Processing, Technische Universität München Arcisstr. 21, Munich, Germany [michel, Abstract Constructing the 3D mesh of a scene from stereo images is a major task in computer vision. It usually involves several steps including stereo matching and meshing. Unfortunately, the time required to generate the 3D mesh is time demanding due to the large amount of pixels to be processed. In this work, we propose a framework to accelerate the overall process. The key issue is to first reduce the number of pixels by approximating an image with a content adaptive mesh. The nodes of the mesh are sparse and they represent the non-uniform samples of the image. To benefit from the reduced set of pixels, we formulate a dynamic programming based stereo matching algorithm which computes the depth only at the sparse samples. We then show by setting up some tests using some real images that the non-uniform samples are sufficient to recover the original dense depth map of the scene by interpolating them using the mesh. The results obtained also show that the employment of the proposed strategy reduces the overall processing time of stereo matching to more than 50% of the original time. We are now able to construct scenes in real-time using less computational resources. 1. Introduction Stereo-based scene reconstruction is a well established research topic in computer vision. It plays an essential role in many applications like telepresence and virtual reality. It usually consists of several steps. The stereo images are rectified. A stereo matching algorithm follows to determine the depth map of the scene upon which the 3D points are computed. Then, a mesh is constructed to create a 3D model of the scene which can be rendered using a graphical processing unit (GPU) on a virtual display. This research is sponsored by the German Research Foundation (DFG) as a part of the SFB 453 project, High-Fidelity Telepresence and Teleaction. Constructing the 3D mesh model of a scene is computationally very demanding mainly due to the application of stereo matching and surface reconstruction algorithms. The time required to obtain the 3D mesh depends on various factors: the complexity of the algorithms used, the size of the stereo images, i.e. the number of points to be matched and meshed, the number of disparity levels. To enhance the speed of stereo matching, most of the recent works formulate the problem using GPUs to parallelize the computations and attain real-time speed [4, 5, 24, 26, 27], while others increase the hardware capacity by using a large PC cluster [11]. In order to boil down the size the 3D mesh and accelerate its rendering on a GPU, the developed algorithms reduce the number of points and build a simplified mesh using the reduced set [2, 13, 17, 21]. Stereo matching and surface simplification are two different problems which are usually treated independently. Their common overhead is the large amount of pixels that have to be processed. In applications like telepresence and robot navigation, however, these two components occur together. Treating them independently leads to a decrease in the overall performance of the application. The justification of the last claim is simple. A simplified mesh is usually built using a reduced set of points which are chosen depending on certain criteria, see [13,17,21] for example. It is thus necessary to compute the depth values only at these points to obtain the complete mesh model of the scene. Doing so will ameliorate the speed of stereo matching and minimize the required computational resources since the number of pixels to be processed is reduced. Three points are mainly discussed in this work. We will first derive an algorithm that estimates the significant features in an image by detecting its non-uniform samples. The obtained samples are sufficient to construct the mesh approximation of the scene while preserving its content. Then, we will propose a sparse stereo matching strategy based on dynamic programming to compute the disparity values of the non-uniform samples. We will then argue that the estimated depth values of those samples are enough to recover

2 the dense depth map of the scene by interpolating them using the mesh. To justify this point, we will setup a test using the Middlebury data set [19, 20] which evaluates the quality of the recovered dense disparity maps. In addition, the results we obtained show that the overall time needed for scene reconstruction reduces significantly. The rest of this work is divided as follows. Section 2 describes some related work. In Section 3, the proposed strategy is derived. Section 4 evaluates the performance of the scheme. Finally, conclusions are drawn in Section Related Work 2.1. Stereo Matching Three different groups of techniques exist to compute a disparity map. First, there are local methods which are based on the block matching technique. Second, there are scanline based methods which minimize a cost function over each scanline. Finally, there are global methods which operate on the entire image to minimize a cost function. The local approaches are quite fast and can achieve a real-time speed. Their outcome, however, suffer from inconsistencies and they usually result in low-quality depth maps even when using the latest developments to remedy their deficiencies as the one described in [8]. The scanline based methods are also fast and produce depth maps with better quality when compared to the local methods [19]. Moreover, there has been a lot of research lately to accelerate these techniques while preserving the quality of the depth maps. In [1], low complexity cost functions were defined and used in addition to the fact that DP was modified to prune the bad search regions. In [10], quadtree subregioning was employed to partition the image into several sub-images while the GPU is employed to parallelize the computations in [4, 5, 24, 27]. The global methods as the ones described in [3, 7] are usually too slow to use in realtime applications unless they are formulated using graphical hardware [26]. These methods, however, result in depth maps which exhibit the best quality [19] Meshing The Stereo Images Approximating 3D points with a triangular mesh has been dealt with thoroughly in the literature. The main difficulty in this area is the long time required to construct the mesh due to the large number of points that have to be processed, e.g. for a image, points have to be meshed. To overcome this burden, the number of points is reduced before building the mesh. Some algorithms are designed to operate on the 3D data and can be applied after stereo reconstruction [6, 13, 14, 17, 21]. While others reduce the number of points in the image and then construct the mesh using the reduced point set. The reduced set of pixels is usually referred to as the non-uniform samples of the image since the obtained sparse image will be irregularly sampled depending on the variation of the pixels intensities. In [25], quadtree subregioning was used in order to represent the image with a mesh, where the nodes of the mesh are the desired non-uniform samples. In [28], the gradient image is computed to determine a feature image. Then, using the gradient image, the non-uniform samples are determined by optimally placing the points using the Floyd-Steinberg error diffusion algorithm. In [16], nonuniform samples are chosen by computing the skewness of the pixels and then choosing the pixels that are above a predefined threshold. The advantage of these methods over the straightforward 3D meshing techniques is that the nonuniform samples can be selected very efficiently. 3. The Proposed Approach The aim of our work is to accelerate the overall speed of scene acquisition starting from the fact that an image can be faithfully represented by a subset of its pixels, the nonuniform samples, to capture the scene it represents. Nonuniform image sampling is the process of sampling the image at an irregular sampling grid. The grid has a high sampling density, i.e. the image is sampled with high rate where there intensity variation is high, otherwise it is low. An example is illustrated in Figures 1a and 1c. In Section 3.1, we will derive the method to build the 2D mesh representation of the left image where the vertices of the triangles form the desired non-uniform samples. Then, we will discuss how to match the samples to the right image of the stereo pair in Section Meshing The Stereo Images With Tritree Tritree (triangle tree) subdivisions consists of partitioning an area into a triangular grid [23]. A triangle is first generated which includes all the area to be triangulated. Then, it is subdivided in a balanced manner into smaller triangles until all the area of interest is meshed. In this work, a different strategy is used that allows the representation of an image with an adaptive mesh. An image is first divided into two triangles along one of the longest diagonals. After that, each of the two triangles is split recursively until a predefined criterion is satisfied. The difference between the methods is illustrated in Figure 2. Tritree subdivisions is similar in spirit to the quadtree subdivisioning method used in [25] to represent an image with a mesh. There, an image is partitioned into rectangles which can be subsequently subdivided into two or more triangles depending on an error criterion. Therefore, it appears more natural to start directly by building a tree of triangles instead of building a tree of rectangles and transforming it into triangles at a later stage. Let V i (x i, y i, I i ) with i = 1, 2, 3 be the three vertices of a random triangle T under consideration in an image. If the

3 From (4), the equation of the plane is directly obtained as ax + by + ci + k = 0. (5) (a) (c) Figure 1. Sample result using the proposed algorithm. a: The sample Lab image. b: The 2D mesh obtained by the proposed tritree algorithm. c: The sparse depth map obtained using the proposed sparse DP approach. d: Top view of the 2.5D mesh obtained by combining the sparse depth map with the 2D mesh of the image. division criterion is met, see (7), T should be divided from its longest edge which has to maximize the expression (b) (d) max V i (x i, y i ) V j (x j, y j ), (1) with i, j = 1, 2, 3 and i j. The new vertex, or the nonuniform sample, is the middle point of the longest edge and is obtained by V = 1 2 (V i(x i, y i ) + V j (x j, y j )). (2) The points of an image describe a 3D space represented by the 2D-coordinates of the pixel in the image and the corresponding intensity 1. When building the mesh, a triangle T of the mesh should represent the pixels that it covers as accurately as possible. Each triangle is formed by three points which are parts of the nodes or the vertices of the mesh. The plane Π described by the vertices of T, i.e. V i, is defined using the normal equation as n p(x, y, I) + k = 0, (3) where p(x, y, I) denotes a pixel lying on the plane, k is a real constant and n is the normal vector to the plane. The normal vector can be computed with the three vertices V i (x i, y i, I i ) as the cross product of any two edges of the triangle as n = [V2 V 1 ] [V 3 V 1 ] = a b. (4) c 1 Note that in the case of color images, the same derivation applies but have to be conducted for all the color planes. Thus, it is now possible to recover the intensity value Î of a pixel lying inside the triangle T by rewriting (5) Î = ax + by + k. (6) c Consequently, one can check now how each triangle represents the pixels which lie within by comparing the reconstructed intensities of the pixels with the original ones. A well known quality metric is the Peak Signal to Noise Ratio (PSNR) [15] which is defined as ( ) PSNR = 10 log, (7) MSE where MSE stands for the Mean Squared Error defined across the area A of the triangle T by MSE = 1 A A 1 n=0 ( I (p n ) Î (p n)) 2. (8) The symbol denotes the floor operator. The intensities of the pixels are assumed to vary between 0 and 255. If the PSNR of the reconstructed intensities of T using (6) is lower than a predefined threshold ɛ, then T is decomposed into two smaller triangles. This step is repeated for each triangle until ɛ is satisfied across the whole image. The area of a triangle cannot be less then unity since it will not contain any inlying pixel. Thus, if the division of T leads to a child triangle with area less then unity, the division should be deleted from the triangle tree and T and should not be divided anymore. Note that the area of a triangle cannot be very big in order to avoid fitting a single plane in very large image regions. In this work, the maximum area of a triangle was set to 100 pixels. (a) (b) Figure 2. Sketch illustrating the tritree subdivisions applied to an image. a: The original algorithm of [23]. b: The proposed tritree partitioning scheme Sparse Stereo Matching Sparsity of the data is a property that has been researched extensively in various fields including computer vision. By

4 exploring the sparsity of the data, it is possible to design fast algorithms because only a small system of equations has to be solved. Since the obtained left image from Section 3.1 is sparse, it makes sense taking advantage of this fact. Sparse stereo is usually performed using a local approach. The main disadvantage of such methods is that the smoothness constraint cannot be enforced. In addition, their output have a lot of tendencies for errors at the object boundaries [19]. Another possibility is to apply semi-dense stereo matching techniques as in [9, 18] which determine the depth at some pixels of the image considered to be reliable. Applying the latter type of techniques does not meet the goal of this work since the found pixels do not have to coincide with the locations of the non-uniform samples. And for each non computed sample, the triangles of the mesh where it belongs will be missing from the reconstructed scene and thus deteriorating its quality. Apart from that, we choose DP to accomplish this task since it is one of the fastest optimization methods and leads to a good quality. DP was first introduced in stereo vision in the context of edge based methods [12]. Then, the interest in it grew since it is one of the fastest optimization techniques and leads to good quality depth maps. DP is derived in this section to find the depth of the sparse non-uniform samples and, hence, the naming Sparse DP. The inputs are a sparse left image and a full right image. Let p (x, y) be a pixel in the left image lying on the y th scanline and q (x, y ) be the corresponding pixel in the right image. In this work, the stereo images are assumed to be rectified such that the conjugate epipolar lines become horizontal and share the same y coordinate. Consequently, the disparity value d can be expressed as a function of the pixel difference in the scanline such that d = x x Computation of Cost Function Various pixel-based cost functions exist in the literature. The ones, which are used the most are the absolute difference and the squared difference due to their simplicity. The cost function C used in the present work is the absolute difference which is defined at a disparity value d to be C (x, y, d) = p (x, y) q (x + d, y). (9) In this case, the disparity value of p corresponds to the location where C (x, y, d) reaches a minimum. In general, any pixel-based cost function could be used. The important issue here is to evaluate only the costs over the support region of each non-uniform sample. The support region of a pixel p (x, y) is defined to be the region along which the cost is aggregated [19]. Suppose that the support region S of p is a box filter of size w h. If p (x, y) is a non-uniform sample, C is evaluated at all the pixels of S. The reason is that the aggregated cost of p must be computed over all of S, see (10) Aggregation of Cost The sought disparity surface is assumed to be smooth since the neighboring pixels of p are likely to have the same disparity value. Therefore, the initial costs are usually aggregated. The aggregated cost C a of p defined over its support region S is C a (x, y, d) = x,ỹ w S ( x, ỹ, d) C ( x, ỹ, d), (10) where w S is a weighting function defined over S. The main advantage of sparse DP is that the aggregated costs have only to be evaluated at the non-uniform samples and hence the process is sped up. By looking at Figure 1c for example, the number of non-uniform samples needed to obtain the 3D mesh of the scene amounts to 42% of the total number of pixels. Hence, the effort of the evaluation of the aggregated costs reduces to 42% of the total effort as well Disparity Optimization Using Sparse DP In DP, the stereo correspondence problem is formulated as finding the optimal path through the disparities that minimizes an energy function defined over a scanline. The energy function is usually minimized by conducting a 2D search in a plane defined by C a and it is written as E (d) = E data (d) + λ E smooth (d), (11) where E data (d) is the data term defined using the aggregated costs reflected by (10), E smooth (d) is the smoothness term and λ is a constant to penalize depth discontinuities. In sparse DP, Equation (11) is minimized along a sparse scanline. Therefore, the smoothness term should take the sparsity of the data into account. It can be defined as Ẽ smooth (d) = x τ x d (x) d (x prev ), (12) where x prev represents the neighboring non-uniform sample and d ( ) is the corresponding disparity value. The scalar τ x is related to the distance between the samples. It should be high if the neighboring samples are close and it should be monotonically decreasing as the distance between the samples increases. This is related to the fact that the closer the samples are, the more probable there is a discontinuity and vice versa. In this work, this constant is chosen to be τ x = (x x prev ) 2 since it insures that (12) is continuous and it allows for interpolation between the neighboring samples. It is important to mention that when filling the cost volume, the scalar τ x should reflect the distance between the samples only when penalizing the costs related with the previous pixel, e.g. C (x prev, d 1). To penalize the cost of a pixel going to a different disparity level, e.g. C (x, d + 1), the distance between the samples should be

5 ignored since this cost is not related to the previous sample. In such a case, the scalar τ x must be set to unity. Using the derived equations, the DP optimization is performed at each scanline y to extract the best disparity path. The obtained result is a sparse depth map as the one shown in Figure 1c. By combining the depth map obtained from sparse DP and the mesh of the previous section, the 2.5D mesh of the scene can be constructed as shown in Figure 1d. In the proposed scheme, we did not deal with the problems of occlusion detection or inter-scanline consistency that usually result from the application of DP. However, our approach can be integrated in any DP based technique that takes these points into account. In the presented results, we will be using the DP algorithms proposed in [19, 24]. 4. Experimental Results We have performed tests, which consist of evaluating the performance of our proposed method using the Middlebury data set [19, 20] and some images as the ones shown in Figure 1 and in the first row of Figure 7. These images will be referred to as the Lab image and the Hall image respectively. All the tests were performed on an AMD Athlon XP 64 bit processor (2.2 Ghz, 2 GB RAM) using a Linux operating system and C++ programming language. The performance of the proposed tritree method is assessed by measuring both the length of the mesh and the time required to build the mesh at various PSNR levels, see Equation 7. Figure 3 shows the mesh length of the Tsukuba, Teddy, Lab and Hall images in comparison to the meshing method of Yang [28] and that of Ramponi [16]. It can be directly observed that tritree clearly outperforms the other methods. Looking at Table 1, tritree requires more time to obtain the result since the whole image is used to determine the samples whereas in [16, 28] gradient images are used. Although tritree is slower than the methods of Yang and Ramponi, it has the advantage of possessing a parallel structure. Each new triangle can be independently subdivided from the other ones since they are not related, see Figure 2b. Consequently, it will be possible to subdivide the computations among a cluster of CPUs working in parallel. Figure 4 shows the average time required by tritree applied to a sequence of images at 40 db PSNR. The measurements were performed using a cluster of four AMD processors similar to the ones used in the previous experiments. Each CPU was assigned an equal amount of the image to be processed and the complete mesh of the frame is then gathered in one data set by padding each part. To force such parallelism in the C++ programming language, it is necessary to use the multi-threading library, i.e. pthread, and initiate each part of the image as a single thread. Using a single CPU, the average frame rate is low as was remarked in Table 1 for the image. But as the number increases to 4 CPUs, i.e. (4 parallel threads), the rate increases to 4 VGA frames per second (fps). Tritree can be easily divided among a cluster of CPUs (or a multi-core processor) while using other methods, the job is more complicated. The main reason lies in the fact that if the image is split to be processed with several CPUs, there will be a difficulty in connecting the partial meshes into a single one. (a) (c) (d) Figure 3. Comparison of the variation of the mesh length versus the PSNR threshold of the proposed tritree with the methods of [16, 28]. a: the Tsukuba image. b: the Teddy image. c: the Hall image. d: the Lab image. (b) Algorithm Tsukuba Teddy Lab Tritree Yang [28] Ramponi [16] Table 1. Time Evaluation in seconds of tritree with several other meshing approaches at 40 db PSNR. Figure 4. Average time needed by tritree to generate a mesh versus the number of processors used in parallel. The PSNR threshold was set to 40 db. The images used were of size The proposed sparse DP method is also tested in terms of speed and quality by comparing it to the DP implementations of [19,24]. The quality is tested using the Middlebury data set. The visual output is shown in Figure 5, while the metrics are tabulated in Table 2. The metrics were obtained by taking the ground truth depth maps found in [19, 20], extracting the disparity values of the non-uniform samples and then computing the percentage of the error between the sparse disparity maps obtained using each version of sparse

6 DP and the ground truth values. The results show that the error rate has increased less than 4% on average with respect to the total number of pixels when compared to the original DP methods. To get a feeling on what does the little increase in the error rate means in the quality of the depth maps produced by DP, we show in Figure 6 the depth maps produced by the original algorithm of [24] along with the reconstructed dense depth maps of its proposed sparse version. The latter was obtained by interpolating the sparse depth maps shown in the second row of Figure 5 using Equations 4 to 6. As we can see, the visual quality is hardly effected. This is also confirmed in Table 3 obtained from the Middlebury evaluation web-tool for 0.5 pixels error threshold. On the contrary, the table shows us that the sparse version of [24] is better than other existing dense DP based algorithms. Nevertheless, the performance of sparse DP deteriorates near depth discontinuities. This can be justified for the interpolation of the sparse disparity map extends the edges and leads to the higher values in Table 3. To further elaborate on our results, we show in Figure 7 the combined outcome of tritree and sparse DP to construct a 2.5D mesh approximation of two real stereo images. DP of [19] DP of [24] Image Sparse Original Sparse Original Tsukuba Venus Teddy Cones Average Table 2. Percentage error of the DP methods of [19, 24] with their sparse versions using the Middlebury data set. The non-uniform samples were computed at 40 db PSNR. The time comparison between the proposed sparse DP versions and the corresponding original versions is illustrated in Table 4. Sparse DP reduces the time of the disparity computation tremendously for it exploits the sparsity of the non-uniformly sampled images. As the size of the image increases, the improvement in time also increases since non-uniform sampling eliminates the non-significant pixels in the image and hence unnecessary computations. At an image resolution of pixels and at 150 disparity levels, the time needed to compute the depth map reduces on average to 50% of the original time. The speed of stereo matching can be also improved by distributing the computations over a cluster of CPUs as was done with tritree. The strategy we will follow is to assign an equal amount of the scanlines of the image which must be processed to a CPU; then, all the partial results are combined by simple padding. Figure 8 shows the average rate (fps) required by the DP technique of [24] to generate a disparity map along with its proposed sparse version. The measurements were performed on the same CPU cluster described before. We applied each of these algorithms to the same VGA video sequence used in tritree by setting the maximal disparity range 100. Sparse DP is able to achieve an average rate of 1.5 fps using 4 CPUs. Compared to the original approach, the computation of the dense depth maps would have required at least twice the number of CPUs to reach this frame rate. The proposed approach was integrated into a stereo camera placed on a pan-tilt unit to construct the 3D model of a room. The motion of the camera is known along with the intrinsic parameters. The images have a VGA resolution. The maximum disparity value was set to 150. The threshold of tritree was set to 40 db. The experiment is shown in the accompanying video. 5. Conclusion A new scheme was presented in this work to accelerate stereo-based scene reconstruction. The left image of a stereo pair is first meshed using the proposed tritree subregioning method, which approximates the left image with a sparse image containing the non-uniform samples. Then, the a sparse disparity map is computed by estimating the depth values of the sample using a proposed sparse DP approach. Tests show that tritree is able to construct the mesh approximation of the image while significantly reducing its size. Moreover, using sparse DP, depth maps are now obtained up to 50% faster than with using dense DP without a significant loss in their quality. References [1] S. Birchfield and C. Tomasi. Depth discontinuities by pixelto-pixel stereo. Int. J. Computer Vision, 35(3): , Dec [2] T. Burkert, J. Leupold, and G. Passig. A photorealistic predictive display. Presence: Teleoperators and Virtual Environments, 13(1):22 43, Feb [3] P. F. Felzenszwalb and D. P. Huttenlocher. Efficient belief propagation for early vision. Int. J. Computer Vision, 70(1):41 54, Oct [4] M. Gong and R. Yang. Image-gradient-guided real-time stereo on graphics hardware. In Int. Conf. 3-D Digital Imaging and Modeling, pages , Jun [5] M. Gong and Y.-H. Yang. Near real-time reliable stereo matching using programmable graphics hardware. In IEEE Conf. Computer Vision and Pattern Recognition, pages , Jun [6] M. H. Gross, R. Gatti, and O. G. Staadt. Fast multiresolution surface meshing. In IEEE Conf. Visualization, pages , Oct [7] H. Hirschmüller. Stereo processing by semi-global matching and mutual information. IEEE Trans. Pattern Analysis and Machine Intelligence, [8] H. Hirschmüller, P. R. Innocent, and J. Garibaldi. Real-time correlation-based stereo vision with reduced border errors. Int. J. Computer Vision, 47(1): , Apr

7 Figure 5. Visual outcome of the proposed sparse DP algorithm applied to several DP based stereo matching methods. The non-uniform samples were obtained at 40 db PSNR threshold. The first row is the sparse version of [19] while the second row is that of [24]. Avg. Tsukuba Venus Teddy Cones Algorithm Rank nonocc all disc nonocc all disc nonocc all disc nonocc all disc DP of [24] DP of [5] Sparse DP DP of [22] DP of [19] Table 3. Evaluation of the proposed sparse DP of [24] with its original version and several DP algorithms using the Middlebury web-tool. The results shown are measured at 0.5 pixels threshold. Figure 6. The resulting dense depth maps from the DP of [24] in the first row along with the interpolated depth maps of its proposed sparse version shown in Figure 5 in the second row. [9] J. Kostkova and R. Sara. Stratified dense matching for stereopsis in complex scenes. In British Machine Vision Conf., pages , Sep [10] C. Leung, B. Appleton, and C. Sun. Fast stereo matching by iterated dynamic programming and quadtree subregioning. In Britich Machine Vision Conf., pages , Sep [11] J. Mulligan, X.. Zabulis, N. Kelshikar, and K. Daniilidis. Stereo-based environment scanning for immersive telepresence. IEEE Trans. Circuits and Systems for Video Technology, 14(3): , Mar [12] Y. Ohta and T. Kanade. Stereo by intra- and inter-scanline search using dynamic programming. IEEE Trans. Pattern Analysis and Machine Intelligence, 7(2): , [13] Y. Ohtake, A. Belyaev, and H. P. Seidel. An integrating approach to meshing scattered point data. In ACM Symp. Solid and Physical Modeling, pages 61 69, Jun [14] R. Pajarola. Overview of quadtree-based terrain triangulation and visualization. Technical Report UCI-ICS-02-01, Information and Computer Science, University of California Irvine, 2002.

8 Image Image Disparity DP of [19] DP of [24] Name Size Range Sparse Original Sparse Original Tsukuba Venus Teddy Cones Lab Hall Table 4. Time Evaluation in seconds of the DP methods of [19, 24] with their proposed sparse versions using various images. [15] M. Rabbani and P. W. Jones. Digital Image Compression Techniques. SPIE Press, 1st edition, [16] G. Ramponi and S. Carrato. An adaptive sampling algorithm and its application to image coding. Image and Vision Computing, 19(7): , May [17] A. D. Sappa and M. A. Garcia. Coarse-to-fine approximation of range images with bounded error adaptive triangular meshes. SPIE J. Electronic Imaging, 16(2), Apr [18] R. Sara. Finding the largest unambiguous component of stereo matching. In European Conf. Computer Vision, pages , May [19] D. Scharstein and R. Szeliski. A taxonomy and evaluation of dense two-frame stereo correspondence algorithms. Int. J. Computer Vision, 47(1):7 42, Apr [20] D. Scharstein and R. Szeliski. High-accuracy stereo depth maps using structured light. In IEEE Conf. Computer Vision and Pattern Recognition, pages , Jun [21] C. Shen, J. F. O Brien, and J. R. Shewchuk. Interpolating and approximating implicit surfaces from polygon soup. In ACM SIGGRAPH (ACM Trans. Graphics), pages , Aug [22] O. Veksler. Stereo correspondence by dynamic programming on a tree. In IEEE Conf. Computer Vision and Pattern Recognition, pages , Jun [23] S. Ø. Ville. A structured tri-tree search method for generation of optimal unstructured finite element grids in two and three dimenstions. Int. J. Numerical Methods in Fluids, 14(7): , [24] L. Wang, M. Liao, M. Gong, R. Yang, and Nistér. High quality real-time stereo using adaptive cost aggregation and dynamic programming. In Int. Symp. 3D Data Processing, Visualization and Transmission, pages , Jun [25] Y. Wang and O. Lee. Use of two-dimensional deformable mesh structures for video coding, part II the analysis problem and a region-based coder employing an active mesh representation. IEEE Trans. Circuits and Systems for Video Technology, 6(6): , Dec [26] Q. Yang, L. Wang, R. Yang, S. Wang, M. Liao, and D. Nistér. Real-time global stereo matching using hierarchical belief propagation. In British Machine Vision Conf., pages , Sep [27] R. Yang, M. Pollefeys, and S. Li. Improved real-time stereo on commodity graphics hardware. In IEEE Conf. Computer Vision and Pattern Recognition Workshops, Jun [28] Y. Yang, M. N. Wernick, and J. G. Brankov. A fast approach for accurate content-adaptive mesh generation. IEEE Trans. Image Processing, 12(8): , Aug Figure 7. Outcome of the proposed reconstruction method applied to the Tsukuba image of [19] and the sample hall image. The first row is the original image. The second row shows the constructed 2D mesh with tritree at 40 db PSNR. The third row reflects the reconstructed 2.5D mesh after incorporating the depth values of sparse DP. Figure 8. Performance of the original DP matching algorithm of [24] along with its proposed sparse version depending on the number of processors used. In the test, a VGA sequence with maximum disparity level of 100 was used. The PSNR threshold of tritree was set to 40 db.