th European Signal Processing Conference (EUSIPCO ) Bucharest, omania, August 7-3, EFFICIENT STEEO MATCHING BASED ON A NEW CONFIDENCE METIC Won-Hee Lee, Yumi Kim, an Jong Beom a Department of Electrical Engineering, KAIST, Daejeon, Korea ABSTACT In this paper, we ose a new confience metric for efficient stereo matching. To measure the confience of a stereo match, we refer to the curvatures aroun the two minimum costs of a cost curve, the size of aggregation kernel, an the occlusion information. Using the ose confience metric, we then esign a weighte meian filter, in orer to refine the initially estimate isparities with a small In the esign of weighte meian filter, we overcome the performance egraation ue to a small kernel size by utilizing the filter information of previously processe pixels. It is foun that the performance of the ose stereo matching algorithm is competitive to the other existing local algorithms even with a small size of Inex Terms stereo matching, confience metric, aggregation kernel size. INTODUCTION With the recent popularity of 3D isplay, various 3D isplays without glasses have been evelope []. Among those isplays, a isplay that uses arrays of vertically oriente cylinrical lenslets shows a remarkably goo actual appearance compare to the others. This lenslet-base isplay nees a series of multiple views to generate the 3D effect. Those multiple view images can be generate by using pixel isparities, which are obtaine from two stereo images transmitte to the isplay. To obtain the isparities, stereo matching between the stereo images is necessary. For the TV application, however, the stereo matching shoul be performe in real-time. Therefore, the algorithm may nee to be implemente in VLSI with consieration of low computational cost. Several stereo matching algorithms ha been implemente in VLSI [, 3]. However, those algorithms may be improve for more efficient VLSI implementation. To esign a stereo matching algorithm with low computational complexity, the aggregation kernel size is to be small because the aggregation process takes a large portion of computation loa. However, a small size of aggregation kernel may cause a problem especially in a textureless area where little information for matching exists insie the To examine the stereo matching performance accoring to the aggregation kernel size, a stereo matching algorithm [4] is performe by using two ifferent sizes of aggregation kernel, 35 x 35 an 5 x 35, respectively, an the corresponing isparity maps are given in Fig.. As expecte, the isparity map obtaine with a smaller aggregation kernel inclues much larger erroneous areas especially in the textureless area (see Fig. ()). However, the increase is usually negligible in the texture area. Therefore, the isparities in the texture area are more reliable for refining incorrect isparities in the neighboring textureless area. Hence, for efficient refinement of the isparity map obtaine with a small size kernel, we ose a new confience metric. Our stereo matching algorithm base on the ose metric is fully local an progressive so as to satisfy a VLSI framework. The paper is organize as follows. In section, we briefly escribe a stereo matching algorithm that we aopt for initial isparity estimation. We also review various confience metrics. The ose confience metric an the corresponing stereo matching algorithm are escribe in section 3. Section 4 shows some experimental results. Finally, we conclue the paper in section 5.. ELATED WOK.. Cross-base stereo matching algorithm To estimate an initial isparity map, we aopt a local stereo matching algorithm, which is calle the cross-base stereo matching algorithm [4]. In the algorithm, the raw matching cost is first compute for every pair of corresponing pixels. Namely, L e x, min I x I x, T, () c c c, G, B where I L an I represent the left an right images, x an enote the spatial coorinate an the isparity, respectively, an T enotes a truncation limit of matching cost. By using the obtaine raw matching cost, the aggregate cost is compute as c x, e s,, () U x su x EUASIP, - ISSN 76-465 39
(a) (b) (c) I I L Apply initial stereo matching. Color histogrambase segmentation CS Calculate confience measure. C efinement () (e) Figure. Disparity maps obtaine by using aggregation kernels of ifferent sizes. (a) Left image of the Venus set, (b), (c) isparity map an the corresponing ba pixel map (error > ) obtaine by using a kernel of 35 x 35, an (), (e) isparity map an its ba pixel map (error > ) by using a kernel of 5 x 35. where U (x) is a local support region [4], an U (x) enotes the number of support pixels in U (x). Finally, the isparity is selecte accoring to the Winner-Takes-All strategy, as follows. x arg min c x,,,, (3) where (x) enotes the initial isparity, an max enotes the maximum isparity... Confience metrics Due to the inherent ambiguity of stereo matches, several metrics were ose to measure the confience level of match [5]. Those confience metrics utilize an aggregate cost, curvature of the cost curve, an left-right consistency. The followings are some examples of existing metrics.... Matching score metric (MSM) This metric uses the aggregate cost efine in Eq. () as a confience measure, namely, C MSM ( x ) c ( x, ). (4) Here, i enotes the isparity that reveals the i th minimum cost. Intuitively, a lower cost implies a more correct match.... Curvature of cost curve metric (CU) The curvature of cost curve aroun the minimum cost inicates the confience of a match, because the cost increases rapily near the isparity having the minimum cost. Accoringly, CU is efine as max Figure. Overall iagram of the ose algorithm. C CU x c x, c x, c x,. (5)..3. Naive peak ratio metric (PKN) A istinctive minimum cost in the cost curve can be consiere an evience of confient match. Accoringly, PKN is efine as a ratio of the two minimum costs, i.e., C PKN x c x c x,. (6),..4. Naive winner margin metric (WMNN) Similar to PKN, WMNN utilizes the istinctiveness of the minimum cost. Instea of using a ratio, WMNN computes a margin between two minimum costs an normalize it with the sum of total costs. c x, c x, C x. (7) WMNN c x,..5. Left right ifference metric (LD) Consistency between left an right isparity maps can be use to check if a match is confient. LD is efine by using a margin between two minimum costs an the consistency of the minimum cost across the left an right isparity maps, i.e., C LD x c x, c x,. (8) c x, min c x, Here, min{c (x -, )} implies the minimum value of a cost curve at the corresponing pixel in the right image. 3. THE POPOSED ALGOITHM 4
6 5 4 3 6 5 4 3 6 5 4 3 (a) 3 5 7 9 35793579333353739443454749553555759 3 5 7 9 35793579333353739443454749553555759 (b) 6 5 4 3 6 5 4 3 6 5 4 3 () 3 5 7 9 35793579333353739443454749553555759 3 5 7 9 35793579333353739443454749553555759 (e) In aition, a new metric nees to provie a low confience value for a perioic pattern. However, since the perioic pattern can have a large curvature aroun the minimum cost as shown in Fig. 3(), its confience value may become large. To alleviate this problem, a new metric may nee to be esigne using two minimum cost values as in PKN an WMNN metrics. It is observe that a perioic pattern has high curvatures aroun at least two minimum costs. This observation may be utilize for esigning a new confience metric so that it can successfully hanle perioic patterns. Accoringly, a new metric is ose as 3 5 7 9 35793579333353739443454749553555759 (c) Figure 3. (a), (b), (c) Cost graphs of correctly estimate ot pixels; (), (e), (f) cost graphs of incorrectly estimate cross pixels. The overall iagram of the ose algorithm is escribe in Fig.. In the algorithm, we first estimate an initial isparity map by using the cross-base stereo matching algorithm with a limite size of We then examine the confience of initial isparity at each pixel by using the ose confience metric. Here, the metric is calculate by using the cost graph obtaine uring the initial isparity estimation process. To refine the isparities of low confience, we etermine neighboring pixels that belong to the same object, because they ten to have correct isparities. To etermine those neighboring pixels, we perform the color segmentation by aopting a simple an efficient histogram-base algorithm [6], an assign the corresponing color segment inex, CS, to each pixel. We then perform the refinement by using the initial isparity, confience measure, an color segment inex. 3.. Propose confience metric Since the limitation on the aggregation kernel size prouces a consierable amount of ba pixels, it is important to iscriminate them from the correctly estimate ones. To esign a new confience metric uner various conitions, we examine many ifferent types of cost graphs. Figs. 3(a), (b), an (c) show the cost graphs for three correctly estimate pixels, while Figs. 3(), (e), an (f) show the graphs for the other three incorrectly estimate pixels. Since the correctly estimate pixels in Figs. 3(a) an (b) commonly provie a large curvature aroun the minimum cost, CU can provie a large confience value. However, CU cannot provie a large confience value for a small curvature in Fig. 3(c), since it is calculate by using only three cost values. Therefore, a new confience metric is neee so that a cost graph with a small curvature in Fig. 3(c) can also provie a high confience value. The metric shoul also iscriminate from flat cost graphs shown in Figs. 3(e) an (f). 3 5 7 9 35793579333353739443454749553555759 (f) x U x * c x, LoG * c x,, (9) C LoG where LoG represents a Laplacian of Gaussian filter of n- taps. Note that C inclues an occlusion check as in LD so that it becomes zero if a pixel is foun to be occlue through a cross-check between left an right isparity maps. The LoG filter in Eq. (9) extracts the curvature information across a range larger than that incluing three cost values in the CU metric. This filter characteristic improves the performance of the confience metric especially for a cost graph with a small curvature. Introucing a multiplication factor relying on the number of support pixels, we also improve the metric performance for a cost graph with a small curvature. This is because the number of support pixels corresponing to a small curvature graph is usually large. In aition, subtracting the curvature of the secon minimum cost from that of the first minimum cost, we can successfully istinguish a perioic pattern from a low texture area. 3.. efinement Since the computational cost is our main consieration in esigning a stereo matching algorithm, local approaches such as a weighte meian filter [7], a joint bilateral filter [8], etc., may be esirable rather than a global one. Among them, we aopt a weighte meian filter that is performe only for the isparities of neighboring pixels that belong to the same color segment. x, y MED w, w,..., w N, () N where i represents the initial isparity of neighboring pixels whose color segment inex is the same as that of the center pixel. Meanwhile, enotes a uplication operator, an weight, w i, is efine as a sigmoi function of confience metric, i.e., exp C, w () 4
Table. Error percentages accoring to aopte confience metrics in the ose algorithm. Error threshol of. is use. (a) (b) (c) () (e) (f) Figure 4. (a) Initial isparity map, (b) ba pixel map, an (c) ose confience map for Tey; () initial isparity map, (e) ba pixel map, an (f) ose confience map for Venus..4...8.6.4..35.3.5..5..5 CU, AUC(.9) MSM, AUC(4.3) PKN, AUC(.45) WMNN,AUC(.5) LD, AUC(.76) POP, AUC(.8) 3 4 5 6 7 8 9 (a) CU, AUC(6.9) MSM, AUC(6.) PKN, AUC(5.5) WMNN,AUC(5.9) LD, AUC(5.5) POP, AUC(4.88) 3 4 5 6 7 8 9 (c) Figure 5. Performance of confience metrics for (a) Venus, (b) Tsukuba, (c) Cones, an () Tey. where enotes an offset an etermines a slope of the sigmoi function. Due to the small size of filtering kernel, the filtering is applie only to the limite number of pixels. Hence, to effectively enlarge a filtering range, we vertically agate the filtere result of a current pixel. The agate ata consist of a filtere isparity, weight, an color segment inex, CS. If the agate isparity has a larger weight than the current filtere isparity, the current weighte meian filtere isparity is replace with the agate isparity. Otherwise, all the agate ata are replace with the current ones. 4. EXPEIMENTAL ESULTS To emonstrate the performance of the ose algorithm, experiments were performe using the benchmark.9.8.7.6.5.4.3...5..5..5 CU, AUC(.4) MSM, AUC(.98) PKN, AUC(.49) WMNN,AUC(.65) LD, AUC(.79) POP, AUC(.4) 3 4 5 6 7 8 9 (b) CU, AUC(6.78) MSM, AUC(5.87) PKN, AUC(5.5) WMNN,AUC(5.85) LD, AUC(4.93) POP, AUC(4.) 3 4 5 6 7 8 9 () Confience metric Venus Tsukuba Cones Tey Avg. CU.8.9.6 3.4 7.43 MSM 8.6 5.5 4.9.5. PKN.5.85.7 5.4 8.36 WMNN.9.67 3. 4.4 8.7 LD.56.86. 4.4 7.96 Propose.83.77.4.6 7.6 Table. Error percentages of stereo matching algorithms for several Milebury stereo sets. Error threshol of. is use. Algorithm Venus Tsukuba Cones Tey Avg. [4].96.65.7 5. 7.85 [].9.85 9.79 3.3 6.53 Propose.83.77.4.6 7.5 Milebury stereo atabase [9]. In the experiments, the aggregation kernel is limite to 5 x 35 an the filtering kernel is set to 5 x 63. Parameters T, n,, an are set to 6, 7,, an, respectively. To examine the performance of the ose confience metric subjectively, we visualize the confience metric with a confience map in Fig. 4. A confience map may be compare to the corresponing ba pixel map. Since the correctly estimate isparity is to be assigne to a high value of confience measure, a confience map shoul closely correspon to the ba pixel map as shown in Fig. 4. To examine the performance of the ose confience metric objectively, we aopt an objective measure, the area uner the curve (AUC) [5]. To etermine the AUC, we orer pixels accoring to their confience measure an prouce a curve of error rate as a function of the participation of pixels. The area uner the curve, or AUC, then represents the ability of a confience metric to preict correct matches. A smaller value of AUC implies that the confience metric assigns a larger confience value to a correctly estimate isparity. Fig. 5 shows error rate graphs of four test sets, Venus, Tsukuba, Cones, an Tey, for ifferent confience metrics. It is note in the graphs that the ose metric provies the smallest AUC for all the test sets. The effectiveness of a confience metric can also be examine by applying it to the refinement step of the ose stereo matching algorithm. Table shows that the ose confience metric prouces relatively lower error rates for all the test sets than the other metrics. To examine the performance of the ose local stereo matching algorithm, we compare its matching result with those of the other local algorithms reporte in the Milebury benchmark [9]. We can note in Table that the ose algorithm provies average error percentages similar to those of the others, even with a smaller size of 4
(a) (b) (c) () Figure 7. Disparity maps obtaine by using the algorithms in [4] (st row) an [9] (n row), an by using the ose algorithm, for (a) Tsukuba, (b) Venus, (c) Tey, an () Cones. 5. CONCLUSIONS In this paper, we esign an efficient stereo matching algorithm with the size limitation on both aggregation an filtering kernels. Applying a weighte meian filter that is base on the ose confience metric, we can successfully refine initial isparities. To overcome the size limitation on aggregation kernel, we effectively utilize previously obtaine filtering results by agating them. The experimental result shows the performance of the ose algorithm with a small size of aggregation kernel is competitive to the existing algorithms with a large size of 6. ACKNOWLEDGEMENT This work was supporte by a grant from LG Display Co. Lt., Korea. 7. EFEENCES [] P. Benzie, J. Watson, P. Surman, I. akkolainen, K. Hopf, H. Urey, V. Sainov, an C. V. Kopylow, A survey of 3DTV isplays: Techniques an technologies, IEEE TCSVT, 7. [] M. Hariyama, N. Yokoyama, an M. Kameyama, frame/sec stereo matching VLSI processor with aaptive winow-size control, IEEE Asian Soli-State Circuits Conference, 6. [3] S. Park, C. Chen, an H. Jeong, VLSI Architecture for MF base stereo matching, in SAMOS, LNCS 4599, 7. [4] K. Zhang, J. Lu, an G. Lafruit, Cross-base local stereo matching using orthogonal integral images, IEEE TCSVT, 9. [5] X. Hu an P. Morohai, Evaluation of stereo confience inoors an outoors, in CVP,. [6] J. Delon, A. Desolneux, J. L. Lisani, an A. B. Petro, A nonparametric approach for histogram segmentation, IEEE TIP, 7. [7] L. Yin,. Yang, M. Gabbouj, an Y. Neuvo, Weighte meian filters: A tutorial, IEEE Trans. Circuits Syst. II, 996. [8] S. Mattoccia, S. Giarino, an A. Gambini, Accurate an efficient cost aggregation strategy for stereo corresponence base on approximate joint bilateral filtering, in ACCV, 9. [9] D. Scharstein an. Szeliski, A taxonomy an evaluation of ense two-frame stereo corresponence algorithms, IJCV. [] K.-J. Yoon an I.-S. Kweon, Aaptive support-weight approach for corresponence search, IEEE TPAMI, 6. 43