hsgm: Hierarchical Pyramid Based Stereo Matching Algorithm

Transcription

1 hsgm: Hierarchical Pyramid Based Stereo Matching Algorithm Kwang Hee Won and Soon Ki Jung School of Computer Science and Engineering, College of IT Engineering, Kyungpook National University, 1370 Sankyuk-dong, Buk-gu, Daegu , South Korea Abstract. In this paper, we propose a variant of Semi-Global Matching, hsgm which is a hierarchical pyramid based dense stereo matching algorithm. Our method aggregates the matching costs from the coarse to fine scale in multiple directions to determine the optimal disparity for each pixel. It has several advantages over the original SGM: a low space complexity and efficient implementation on GPU. We show several experimental results to demonstrate our method is efficient and obtains a good quality of disparity maps. Keywords: stereo matching, disparity map, hierarchical SGM, GPU. 1 Introduction According to a recent tendency of supporting Single-Instruction in Multiple- Cores with Multiple-Data concept in processing units, many of existing stereo matching algorithms are revisited in terms of parallelism which can be accomplished through the multi-core CPUs and the general purpose utilization of GPUs. Besides, like full HD resolution videos for 3D TV or satellite imageries for city modeling, the rapid increase of image data can also accelerate the development of high-performance parallel stereo matching algorithms. Moreover, there are many computer vision applications that require the stereo matching algorithm of real-time performance. The up-to-date stereo algorithms perform a global optimization by defining an objective function and optimize it. But the time complexities of those optimizations are, however, considerably high even though the computing powers are constantly increased. One more issue of the global optimization can be its enormous memory consumption, which can potentially cause relevant page fault that degrades the overall performance of an algorithm, especially for the highresolution input data. Hirschmüller suggests the Semi-Global Matching (SGM) to reduce the computation cost while maintaining the quality of matching result by substituting multidirectional 1D optimization for 2D global optimization [1]. The matching costs of a certain pixel for candidate disparities are computed locally and aggregated along all specified directions, which are 8 or 16 directions suggested by the author, up to the current pixel. In consequence, the required memory is J. Blanc-Talon et al. (Eds.): ACIVS 2011, LNCS 6915, pp , c Springer-Verlag Berlin Heidelberg 2011

2 694 K.H. Won and S.K. Jung proportional to the number of pixels, the disparity range, and the number of directions. Furthermore, the optimal disparities of the pixels are not determined until the aggregated costs from all directions are assigned. This computational dependency with the relatively large memory usage is a hindrance to the maximum use of parallelism, because the conventional GPUs are designed to utilize on-chip memory which is fast but has limited size of 16k to 48k per independent logical block. In this paper, we propose a different version of multi-directional optimization performed on the pyramid images. Our method aggregates the matching costs from the coarse to fine scales in multiple directions to determine the optimal disparity for each pixel. The original SGM aggregates the matching costs along 1-pixel width path, but the suggested algorithm aggregates matching costs along down the cone-shaped volume on the scale-space as shown in Fig. 1. We named our algorithm as hierarchical Semi-Global Matching, hsgm. P P Fig. 1. The matching cost aggregation in SGM(left) and hsgm(right) hsgm has several advantages over the original SGM. First, hsgm has a lower space complexity, because it requires the resolution of the higher level of pyramid for the matching cost aggregation. Second, hsgm can be easily implemented on the current GPU architecture, since the hierarchical approach resolves the computation dependency on the optimization process of 1D path so that it maximizes the use of parallelism. The remainder of this paper consists of the following sections. In section 2, the stereo matching algorithms focused on parallelism are briefly introduced. In section 3, we present the semi-global optimization process on a hierarchical pyramid and introduce its GPU implementation in detail. In section 4, we performed a couple of experiments to evaluate our algorithm comparing to the original SGM and its GPU versions. In section 5, we conclude the paper with some future works. 2 Related Work In the famous Middlebury ranking for stereo algorithms, many of top ranked algorithms contain the global optimization process with the objective function as their important part. Nevertheless, as we mentioned in the previous section, they

3 hsgm: Hierarchical Pyramid Based Stereo Matching Algorithm 695 still suffer from the expensive computation cost and the inefficient memory utilization. The computation time and memory space complexities of many stereo matching algorithms are analyzed well in the literature of Guerra-Filho [2]. The original SGM was trying to reduce the computational expenses of 2D global optimizations, but the aggregated matching costs along each path are dependent on the costs of the previous pixel on the path. As we addressed, this dependency can effect negatively on the implementation of parallelism. Many variants of SGM have been proposed up to now, most of which try to reduce their computation time and memory consumptions. Ernst et al. [3] implemented the SGM on GPU by aggregating matching costs on multiple paths simultaneously. Gibson and Marques [4] also implemented it on GPU with a similar strategy but optimized their implementation using parallel sorting, caching on the shared memory, and so on. Both implementations are much faster than the original SGM, but they still require the same amount of memory space with the original one. Humenberger et al. [5] suggested a modified SGM by dividing the input image into many horizontal strips to reduce the required memory size. They assumed that each strip is a whole image. This schemecanbeappliedongpuimplementation by decreasing the strip (or block) size to fit the on-chip memory. However, their assumption fails on the boundaries of each strip by producing discontinuity. Gehrig et al. [6] proposed to utilize the low-resolution disparity map in order to enhance the computation speed of the current resolution disparity map. hsgm makes use of the aggregated matching cost from the costs on the higher level instead of the low-resolution disparity. Moreover, the matching costs are aggregated down to the current level with adaptively determined penalty according to the similarity between pixels of current and higher level. Humenberger et al. [7] compared the performance of stereo matching algorithms on CPU, Digital Signal Processor (DSP), and two GPUs. They employed local aggregation of matching costs within a specified size of local windows by which the parallelism can be accomplished more easily. However, local optimization often produces outliers for the weakly textured area that is larger than the window size. Our algorithm also aggregates matching costs locally on the current level of the pyramid, but utilizes the aggregated matching costs from the higher level. Therefore, hsgm is easily parallelized on current-generation GPUs while maintaining the quality of disparity map resulted from a global optimization. 3 Semi-Global Optimization on the Hierarchical Pyramid 3.1 Propagation of Aggregated Matching Costs After the matching cost C(p, d) between a pixel p of one image and a pixel p + d of the other is computed, the multi-path aggregation is performed to determine the optimal disparity in the original SGM. The aggregated cost at pixel p on the directional path r is denoted by

4 696 K.H. Won and S.K. Jung L r (p, d) = C(p, d) ( ) +min L r (p r,d),l r (p r,d± 1) + P 1, min L r (p r,i)+p 2 i min L r (p r,k), (1) k where d is the disparity, p r represents the precedence pixel of p on the path r, P 1 and P 2 ( P 1 ) are the penalties for small disparity changes and large ( 1) disparity changes along the path, respectively. Similarly, the aggregated matching cost on the hierarchical pyramid is represented with the current level l n and the upper level (low-resolution) l n 1 is computed using Equation 2. L r (l n,p,d) = C(l n,p,d) ( +min L r (l n 1,p r,d),l r (l n 1,p r,d± 1) + P 1, ) min L r (l n 1,p r,i)+p 2 min L r (l n 1,p r,k). (2) i k In Fig. 2, the precedence constraint which is caused by the Equation 1 and 2, are represented. The L r (p 0,d i ) is required to evaluate the L r (p 1,d i )insgm. However, L r (l n,p 0,d i )andl r (l n,p 1,d i ) can be obtained independently along the path from higher hierarchy (dashed arrow). The aggregation process from the higher level to the current level often fails to support the minor pixels which are smashed in the lower resolution. To prevent this drawback, we adaptively determine the penalty P 2 as follows P 2 = P (P 2 P 1 ) 1 + I(l n 1,p r ) I(l n,p), (3) where I(l n,p) is the value of pixel p in the current level l n,andi(l n 1,p)is the value of pixel in the upper level. The optimal disparity at the pixel p is determined by computing the total cost for each disparity d from all paths r that pass through the pixel p. argmin d L r (l n,p,d). (4) r p 0 p 1 p 0 p 1 l n l n-1 Matching Cost Aggregation: SGM l n-2 Matching Cost Aggregation: hsgm Fig. 2. The aggregation paths and the precedence constraint in SGM and hsgm

5 hsgm: Hierarchical Pyramid Based Stereo Matching Algorithm 697 Finally, the outliers are removed by the post-processing such as median-filtering and left-right consistency check. 3.2 The GPU Implementation There are several reasons why hsgm is well suited for a parallel implementation on GPU. First, the scalable kernel can be designed and evaluated on each scalespace without modification. Second, our algorithm divides the input image into many precedence-independent local regions on each scale. A local region can be allocated into blocks with multiple threads in CUDA architecture. The overall performance is increased by assigning relevantly accessed variables to shared (on-chip) memory in each block. After the hierarchical pyramid is constructed, the aggregated costs of each level are computed using the scalable invocation of the same kernel and saved in the global memory. For each kernel invoke, the input image of the level l n is divided into many local regions and assigned to each block. Theblocks are assigned to multiple cores for block-level parallelism and each block provides thread-level parallelism with thread synchronization method in a block. Fig. 3 shows the processing steps in a single block. Theh, w, andn d represents the height and width of a local window, and the number of disparity, respectively. At first, the image patches for each block are copied in shared memory and then the matching costs are computed at h n d threads. For the next step, the matching costs are aggregated. This process starts from the L r (l n 1,p r,d i ) which reside in the global memory using Equation 2. After that, the matching costs of the current level are aggregated using the Equation 1 along all specified paths. Those aggregated costs for each pixel are accumulated in the shared memory and finally the optimal disparity is selected using Equation 4 for every pixels in the window. The required size of shared memory S for local aggregation of matching costs in each block is computed as S = w 2 + w 2 n d +2w n d + w 2 n d, (5) for each Block: h n d threads L r (l n-1, p -r, d i ) w h processed concurrently processed concurrently r L r (l n ) Left patch h w + n d - 1 Right patch h w Matching Cost Computation path w Cost Aggregation for each Path h w Disparity Selection Fig. 3. The GPU implementation of hsgm: the processing steps in each block

6 698 K.H. Won and S.K. Jung where w w is the resolution of the local window, n r is the number of aggregation directions, and n d is the disparity range. In Equation 5, the second term represents the size of matching costs for each pixel and for all disparities. The third term represents the accumulation buffer of the current pixel for all disparities n d on w paths plus the same size for the previous pixels of each path. The block dimension will be w n d. The last term is for the total amount of the aggregated cost in Equation 4. The size of local window is determined inversely by putting the shared memory as the maximum (16k or 48k). The size of local window of lower-resolution is larger than that of current scale because the disparity range is reduced by the scale (< 1.0). 4 Experimental Result We compared hsgm with several different implementations of SGM, the SGM on CPU, SGM on GPU with global aggregation of matching costs, and SGM on GPU with local aggregation in overlapped window. We first implemented SGM and hsgm on CPU to compare the quality of the resulted disparity maps, because the qualities of results from each GPU versions are the same with the results of their CPU implementations. Both CPU implementations consist of the hierarchical computation step of Mutual Information (HMI) as matching cost [12], the multi-path aggregation step, and the post-processing step including median filtering and left-right consistency check. Without supporting the sub-pixel disparity, other parameters like penalties, P 1 and P 2 are selected experimentally. For hsgm, we use for the local window size, and three levels of hierarchical pyramid with the scale factor, 0.5 while the bigger local window requires the larger local memory, the much smaller window size than this degrades the quality of computed disparity map. The Middlebury stereo set is used as input images and the results are evaluated through the webpage [8]. The values on the Table 1 represents the percent of bad pixels compare to the ground truth. The threshold for the disparity error is 1 and the bad pixels are not evaluated for the partially occluded and boundary regions. The values of SGM are different from that of the original paper [1,8]. It is partially because our implementation do not support the sub-pixel accuracy and may have different penalty values and hold-filling strategy from that one. However, those elements can be applied to both methods to improve the quality of results. Table 1. Evaluation of SGM and hsgm (percent of bad pixels) Data Set Methods Tsukuba Venus Teddy Cones SGM hsgm

7 hsgm: Hierarchical Pyramid Based Stereo Matching Algorithm 699 Fig. 4. The resulted disparity maps of SGM (left column) and hsgm (right column) The quality of resulted disparity map of hsgm is similar to that of SGM as shown in Table 1 and Fig. 4, while the size of required memory is about 30 percent of the original SGM. For the next experiment, we implemented and compared a GPU version of hsgm and two GPU versions of SGM using CUDA. The first GPU version of the SGM is similar to the implementation of Ernst et al. [3], and Gibson and Marques [4]. The global memory is allocated to accumulate the aggregated matching costs of each path for the SGM. We also utilized the on-chip memory as the cache for the input image and intermediate aggregated costs. The other GPU implementation of SGM divides the input image into many overlapped local windows and finds the optimal disparity in the local window. This implementation is much faster than previous one because the precedence between any two local areas are not considered. But the tile effect, which is drawback of local optimization as we addressed in section 2, can occur as shown in Fig. 5. The local window size is determined for eachscaleusingtheequation5withfixed on-chip memory size. In our experiment, we use Nvidia GTX580, and image of with 32-levels of disparity. Table 2 shows the computation times of various implementations of SGM and hsgm in milliseconds. Contrary to the local window version of SGM, the Table 2. The performance of GPU implementations SGM hsgm Methods CPU GPU GPU global memory global memory local window local window Computation time (ms)

8 700 K.H. Won and S.K. Jung Fig. 5. The tile effect of local optimization hsgm removes the tile defects between two adjacent windows by aggregating the matching cost of higher level hierarchy with additional 17% of computation time. hsgm used the 1/8 times of the memory space compared to SGM in this experiment. It is because hsgm aggregates the matching costs of the current level from the aggregation costs of the upper level. For your reference, the computation time of CPU-hSGM is almost similar to that of CPU-SGM. 5 Conclusion and Future Work We proposed a hierarchical pyramid based stereo matching algorithm. Our algorithm is different from other hierarchical algorithms in terms of utilizing the aggregated costs itself instead of the determined disparity values from the higher hierarchy. We accomplished the effect of global optimization by using independent local optimizations combined with the aggregated matching cost of higher hierarchy in the image pryramid. As a result, the intuitive parallelization using the current generation of GPUs is possible. hsgm requires less memory space and is faster than other GPU implementations while achieving a good quality of disparity map. Our future work will be the application of our algorithm to some computer vision problem such as object detection or tracking problems that require realtime execution and accurate range images, simultaneously. Acknowledgement. This research was supported by Basic Science Research Program through the National Research Foundation of Korea(NRF) funded by the Ministry of Education, Science and Technology( ).

9 References hsgm: Hierarchical Pyramid Based Stereo Matching Algorithm Hirschmüller, H.: Accurate and Efficient Stereo Processing by Semi-Global Matching and Mutual Information. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp (2005) 2. Guerra-Filho, G.: An Optimal Time-Space Algorithm for Dense Stereo Matching. Journal of Real-Time Image Processing, 1 18 (2010) 3. Ernst, I., Hirschmüller, H.: Mutual Information Based Semi-Global Stereo Matching on the GPU. In: Bebis, G., Boyle, R., Parvin, B., Koracin, D., Remagnino, P., Porikli, F., Peters, J., Klosowski, J., Arns, L., Chun, Y.K., Rhyne, T.-M., Monroe, L. (eds.) ISVC 2008, Part I. LNCS, vol. 5358, pp Springer, Heidelberg (2008) 4. Gibson, J., Marques, O.: Stereo Depth with a Unified Architecture GPU. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops (2008) 5. Humenberger, M., Engelke, T., Kubinger, W.: A Census-Based Stereo Vision Algorithm Using Modified Semi-Global Matching and Plane Fitting to Improve Matching Quality. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, 6th Workshop on Embedded Computer Vision (2010) 6. Gehrig, S.K., Rabe, C.: Real-Time Semi-Global Matching on the CPU. In: IEEE Conference on Computer Vision and Pattern Recognition Workshops, 6th Workshop on Embedded Computer Vision (2010) 7. Humenberger, M., Zinner, C., Kubinger, W.: Performance Evaluation of a Census- Based Stereo Matching Algorithm on Embedded and Multi-Core Hardware. In: Proceedings of the 6th Int. Symposium on Image and Signal Processing and Analysis (2009) Kolmogorov, V., Zabih, R.: Computing Visual Correspondence with Occlusions using Graph Cuts. In: Int. Conference on Computer Vision, vol. 2, pp (2001) 10. Sum, J., Li, Y., Kang, S., Shum, H.-Y.: Symmetric Stereo Matching for Occlusion Handling. In: IEEE Conference on Computer Vision and Pattern Recognition, vol. 2, pp (2005) 11. Hirschmüller, H.: Stereo Vision in Structured Environments by Consistent Semi- Global Matching. In: IEEE Conference on Computer Vision and Pattern Recognition (2006) 12. Kim, J., Kolmogorov, V., Zabih, R.: Visual Correspondence Using Energy Minimization and Mutual Information. In: IEEE International Conference on Computer Vision (2003)