SSD Matching Using Shift-Invariant Wavelet Transform

Transcription

1 SSD Matchig Usig Shift-Ivariat Wavelet Trasform Fagmi Shi, Neil Rothwell Hughes ad Geoff Roberts Mechatroics Research Cetre Uiversity of Wales College, Newport Allt-Yr-Y Campus PO Box 18 Newport NP2 5XR, UK Abstract The covetioal area-based stereo matchig algorithm suffers from two problems, the widowig problem ad computatioal cost. Multiple scale aalysis has log bee adopted i visio research. Ivestigatio of the wavelet trasform suggests that -- dilated wavelet basis fuctios provide chageable widow areas associated with the sigal frequecy compoets ad hierarchically represet sigals with multiresolutio structure. This paper discusses the advatages of applyig wavelet trasforms to stereo matchig ad the weakess of Mallat s multiresolutio aalysis. The shift-ivariat dyadic wavelet trasform is exploited to compute a image disparity map. Experimetal results with sythesised ad real images are preseted. 1 Itroductio Fidig correspodece is a ill-posed problem i stereo visio. Area-based stereo matchig is oe of the covetioal solutios. It compares the itesity similarity betwee widowed areas of two stereo images. The sum of squared differece (SSD) [1] is commoly used as the similarity measure: x = + σ 2 ssd( x) = L( τ ) R( τ ) (1) τ x σ where x is the pixel idex over the stereo images L ad R, τ idexes over the local area aroud x withi ±σ. It is well kow that this method suffers from the widowig problem ad computatioal cost [1]. I order to alleviate these problems, multistage strategies were developed by visio 113 BMVC 21 doi:1.5244/c.15.13

2 researchers such as multistage matchig by dividig images ito small blocks of equal size [2], multiscale matchig based o Gaussia filtered images [3], [4], [5], ad the pyramid structure that geerates sets of low-pass ad bad-pass filtered images [6]. A more geeral hierarchical architecture ad fast implemetatio was created by Mallat i 1989 followig a study of wavelet cocepts. This is kow as wavelet multiresolutio aalysis (MRA) [7]. The advatage of the wavelet trasform is that it uses wide widows for lowfrequecy compoets ad arrow widows for high-frequecy compoets [8]. These widows are formed by dilatios ad traslatios of a prototype (or mother) wavelet. Thus, if SSD matchig is performed o the wavelet trasforms of sigals, the widowig problem with the covetioal SSD approach is aturally solved. However, Mallat s multiresolutio aalysis lacks shift-ivariace, which will be discussed i the followig sectio. Stereo matchig requires a shift-ivariat trasform because stereo image pairs ca be cosidered as the shifted versios of each other (with distortios). This makes MRA usuitable for matchig. This paper discusses the stregths of wavelet trasforms whe applied to stereo matchig ad some alterative wavelet methods to Mallat s MRA. The Dyadic wavelet trasform is exploited to compute a dese disparity map. Experimetal results usig sythesised ad real images are preseted. 2 Properties of the Wavelet Trasform for Stereo Matchig Moder wavelet theory was motivated iitially for the sake of a better time-frequecy sigal represetatio tha the short time Fourier trasform (STFT) ad to overcome its drawbacks. I cotrast with the STFT that uses a costat widow for the whole sigal, the wavelet trasform uses wide widows for low-frequecy compoets ad arrow widows for high-frequecy compoets [8]. It achieves this by decomposig a sigal ito the dilatios ad traslatios of a mother wavelet. Let ϕ(t) deote a mother wavelet, which is a small oscillatory fuctio with fiite support. A family of wavelets ϕ a,b (t) is the represeted by 1/ 2 ϕ ( t) = a ϕ (( t b) / ) (2) a, b a where a is the scale parameter, b is the traslatio parameter, ad a, b R. The wavelet trasform represets a sigal x(t) by a ifiite set of such basis fuctios: 1/ 2 WT ( b, a) = x( t) a ϕ (( t b) / a) dt (3) 2.1 Automatic Widowig Aalysis I order to illustrate the time-frequecy resolutio of a wavelet trasform, Figure 1 shows the coverage of a wavelet i the time-frequecy plae. It is evidet that whe the frequecy iterval goes up by a scale factor, the time iterval goes dow by the same factor. Let t ad f deote the widow width of the mother wavelet i time ad i the spectral domai, ad t ab ad f ab are the correspodig deotatios of the scaled ad shifted wavelet. That is: 114

3 t f = t τ s f τ s (4) This reveals that the product of the widow width of time ad frequecy is costat at all scales [8]. This property is oe of the most importat advatages that wavelet trasforms provide. I cotrast with the STFT applyig either arrow or wide widow (but ot both) to the whole sigal, wavelet trasforms are able to aalyse highfrequecy compoets usig small widows ad low-frequecy compoets usig big widows. This property is ideal whe dealig with o-statioary sigals that cotai both short high-frequecy compoets ad log low-frequecy compoets. frequecy time Figure 1 Time-frequecy plae of wavelet trasform: widow area is costat at all scales A image is a typical o-statioary sigal, which cosists of a slowly chagig backgroud ad rapidly chagig details. After decomposig a image, a set of images at differet resolutios is obtaied. At coarser resolutios, the matchig is performed by comparig wider areas leadig to larger ucertaity i disparity localisatio. At fier scales, the compared areas ted to be more localised ad smaller ucertaity i disparity localisatio is expected. The widowig problem that occurs with SSD matchig is the aturally solved by choosig the right wavelet. 2.2 Why MRA Is ot Suitable for Matchig I equatio (3), if parameter a ad b are cotiuous real values, the the trasform is called a cotiuous wavelet trasform. The wavelet basis fuctios costitute a overcomplete represetatio i which iformatio is highly redudat. This redudacy ca be reduced by discretisig a ad b. Daubechies [9] foud that whe a=2, b=k2,, k Z, the basis fuctios { ϕ ( x) = 2 ϕ(2 x k ) } are orthogoal for certai k choices of wavelet. Stimulated by the pyramidal approach i visio, Mallat, as a former visio researcher, proposed a fast implemetatio for the wavelet orthogoal decompositio [7]. This is the well kow multiresolutio aalysis (MRA). MRA decomposes a sigal ito the same size subimages at dyadic scales. At each scale a approximatio part ad a detail part are formed by passig the sigal through a half-bad low-pass filter ad a half-bad high-pass filter, ad subsequetly dowsamplig them by two. The approximatio part is the hierarchically decomposed. Dowsamplig a sigal simply discards every other samplig poit. This 115

4 operatio reduces the umber of sigal samples by a half whe the scale is doubled. Shift-ivariace (or time-ivariace) meas that if a sigal is delayed i time, its trasform result is delayed as well. Dowsamplig is ot shift-ivariat. Neither, therefore, is MRA. This issue was discussed by Strag [1] ad the shift-ivariace problem was cosidered to be the mai drawback of MRA. For stereo matchig, ituitively, oe image ca be assumed to be the shifted versio of the other. The shifted value with respect to each pixel is the disparity, which is depedet o the pixel positio. Oly shift-ivariat wavelet trasforms ca be used for matchig. 2.3 Shift-Ivariat Wavelet Trasforms The cotiuous wavelet trasform possesses the property of shift-ivariace. However, its high redudacy gives rise to high computatioal cost. Approximate shift-ivariace with effective computatio eeds to be achieved. Mallat used the dyadic wavelet trasform ad the zero-crossigs of the dyadic wavelet trasform to reduce the represetatio size [11]. Simocell [12] built steerable filters to achieve a shiftable trasform that is joitly ivariat i positio, scale ad orietatio. More recetly, a shift-ivariat complex wavelet trasform [13] with perfect recostructio has bee costructed ad applied to image processig ad computer visio. The wavelets used i these papers could be applied to the matchig problem. This paper will discuss the applicatio of the dyadic wavelet trasform to disparity computatio. The motivatio for this is discussed i the ext sectio. 3 Correspodece Matchig Usig the Dyadic Wavelet Trasform To simplify the umerical computatios ad maitai shift-ivariace, the scale parameter a of equatio (3) is discretised alog a dyadic sequece {2, Z} while leavig the shift parameter b cotiuous. The dyadic wavelet trasform (DWT(b, j))has the followig form: / 2 DWT ( b, ) = x( t)2 ϕ (( t b) / 2 ) dt (5) Mallat [14] proved that uder certai coditio dyadic wavelet trasform defies a complete ad stable represetatio. The algorithmic efficiecy if gretly improved compared with the cotiuous wavelet trasform. The iformatio is still redudat due to the cotiuous traslatio parameter. However, it is good for matchig task aimig at dese disparity map output. Figure 2 gives two sigals (epipolar lies from two sythesised stereo images). The decompositio results at three scales, 2 1, 2 2 ad 2 3, are show i Figure 3. The sum of squared differece (SSD) [15] is applied to the trasformed sigals to measure the similarity of the correspodig poits. The smaller the SSD value, the more likely it is that the poits correspod to each other. Ulike the covetioal SSD, directly applied to the image itesity value, the SSD here is applied to the wavelet coefficiets. 116

5 15 1 Shifted sca lies left right Figure 2 Sca lies from stereo images 1 8 scale Dyadic wavelet trasform left right 4 scale scale Figure 3 1D dyadic wavelet trasform at three scales Let x 1 (t) ad x 2 (t) be two sca lies of stereo images, their dyadic wavelet trasforms are deoted by DWT 1(2, t) ad DWT 2(2, t), respectively, where N. The SSD measure (ssd(,x)) is defied as: x+ 2 σ 2 ssd(, x) = DWT1(2, τ ) DWT 2(2, τ ) (6) τ = x 2 σ where σ is the size of a iterval where the eergy of the mother wavelet is mostly cocetrated [11]. From equatio (6), it ca be see that at each scale the searchig area is ( 2 σ, 2 σ ). Matchig should be take at as much scales as possible. The maximum scale max (max) should be determied by: 2 σ sigal legth. Besides the epipolar costrait [16], other costraits e.g. similarity, uiqueess, orderig ad cotiuity [17] are also applied alog with equatio (6). Figure 4 gives the computed disparity result at three scales. The correspodig SSD values are also recorded as a measure of the matchig cofidece. The smaller the SSD value is, the higher is the cofidece of the matchig. For compariso, the parameter at three scales is ploted i oe figure, see Figure

6 After the computatio from the above steps, each pixel correspods to two parameters, its disparity value, d(,x), ad its SSD value, SSD (,x). The most ituitive way is to choose the right scale N for each pixel disparity so that at that scale its SSD value is a miimum of all the scales. That is, if N = mi{ ssd(, x)}, the d(x)=d(n,x). 15 disparity, scale disparity, scale disparity, scale Figure 4 Disparity at three scales ssd for three scales scale 2 1 scale 2 2 scale Figure 5 SSD at three scales Oe of the big problems of SSD matchig is its oise characteristics. Figure 6(a) shows the computatioal result usig above method. It ca be see that the disparity value at the right ed, i.e. aroud pixel 12 is ot very good. This is the geeral case whe the same program is tested with some other more complicated image pairs, which show worse results at some poits. I some cases, for example, the poits at the border of the image may ot have matches, or images corrupted with oise give rise to ustable fluctuatig results. Thus oise reductio is eeded. The oise problem ca be dealt with usig oe of two possible methods, both of which use a additioal matchig costrait to remove the poits with higher SSD value tha a threshold. Hard thresholdig ad soft thresholdig [18] are employed i the two methods, respectively. The stadard deviatio is adopted as a soft threshold i this paper. Figure 6(b) gives the computatioal results usig soft thresholdig, which shows sharp edges ad stable disparity values. 118

7 8 Shift at row 6 8 Shift at row (a) (b) Figure 6 Computed disparity, (a) without threshold, (b) soft threshold 4 Experimetal Results with Images For the iitial test, the commoly used radom dot stereograms [19] are costructed. Figure 7 shows the sythesised Squares images of size 128*128. The cetral two squares are right shifted by 4 ad 8 pixels, respectively, betwee the two images. The image matchig is performed alog the theoretical epipolar lies of the images. The disparity map ad the depth map for Squares are give i Figure 8. Dot1 Dot Figure 7 Stereo pair: Squares Figure 8 Disparity map ad depth map: Squares 119

8 To icrease the complexity of the image features, aother pair of radom dot stereograms, Radom Square showed i Figure 9, is tested. I cotrast with the Squares images, the pixels i Radom Squares are radom values betwee ad 1. I Figure 1, the left figure gives the groud truth disparity map ad right shows the computatioal results usig the above method. Figure 9 Stereo pair 2: Radom Squares Figure 1 A compariso of the groud truth data ad computed data (I) Left: groud truth disparity map, Right: disparity map usig wavelets Computatio with real image pairs is also tested. Oe imagery popularly used is show i Figure 11, which ca be dowloaded from the web site, The groud truth ad estimated disparity maps usig dyadic wavelet trasform are displayed i Figure 12. Figure 11 Stereo pair 3: Tsukuba images 12

9 Figure 12 A compariso of the groud truth data ad computed data (II) Left: groud truth disparity, right: estimated disparity result usig wavelets Aother pair of real images was take i the authors laboratory ad used i [2] is show i Figure 13, the right figure of which gives the computed disparity map. Figure 13 Real stereo pairs 4 ad the disparity map 5 Coclusio ad Future Work This paper has preseted a wavelet approach to computig disparity maps. It is of vital importace to apply shift-ivariat wavelet trasforms whe usig wavelet techiques for stereo matchig. As a iitial approach to the applicatio of wavelet trasforms to stereo matchig, the dyadic wavelet trasform was used to develop a matchig algorithm. The sum of squared differece is defied based o the values of dyadic wavelet trasform coefficiets. The experimetal results give rise to promisig disparity maps. This demostrates the viability of applyig the wavelet trasform approach to stereo matchig. However, a better compromise betwee the algorithmic efficiecy ad iformatio redudacy could be made because the traslatio parameter of the dyadic wavelet trasform remais cotiuous. Further ivestigatio of other wavelet trasform techiques such as wavelet zero-crossigs ad dual-tree complex wavelet trasforms applied to the matchig problem is therefore beig carried out. Ad the compariso of wavelet-based algorithms with stadard matchig algorithms will be made i the future work. 121

10 6 Refereces [1] Trucco, E. ad Verri, A., Itroductory Techiques for 3-D Computer Visio, Pretice Hall, [2] Rosefeld, A. ad Thursto, M., Coarse-fie Template Matchig, IEEE Tras. System, Ma, ad Cyberetics, vol. 7, pp , [3] Marr, D. ad Poggio, T., A Computatioal Theory of Huma Stereo Visio, Proc. R. Soc. Lod., vol. 24, pp , [4] Grimso, W., A Computer Implemetatio of a Theory of Huma Stereo Visio, Phil. Tras. Royal Soc. Lodo, vol. V292, pp , [5] Grimso, W. E. L., Computatioal Experimets with a Feature-Based Stereo Algorithm, IEEE Tras. Patter Aalysis ad Machie Itelligece, vol. 7, pp , [6] Burt, P. ad Adelso, E. H., The Laplacia Pyramid as a Compact Image Code, IEEE Tras. Commuicatios, vol. 31, pp , [7] Mallat, S., A Theory for Multiresolutio Sigal Decompositio: the Wavelet Represetatio, IEEE Trasactios o Patter Aalysis ad Machie Itelligece, vol. 11, pp , [8] Chui, C. K., Wavelets: A Tutorial i Theory ad Applicatios, Academic Press, [9] Daubechies, I., Orthoormal Bases of Compactly Supported Wavelets, Commuicatios o Pure Applied Mathematics, vol. 41, pp , [1] Strag, G. ad Nguye, T., Wavelets ad Filter Baks, Wellesley-Cambridge Press, Secod Ed., [11] Mallat, S., Zero-crossigs of a Wavelet Trasform, IEEE Trasactios o Iformatio Theory, vol. 37, pp , [12] Simocell, E. P., Freema, W. T., Adelso, E. H., ad Heeger, D. J., Shiftable Multiscale Trasforms, IEEE Tras. Iformatio Theory, vol. 38, pp. P587-67, [13] Kigsbury, N. G., The Dual-tree Complex Wavelet Trasform: A New Techique for Shift Ivariace ad Directioal Filters, IEEE Digital Sigal Processig Workshop, DSP 98, Bryce Cayo, pp. Paper o 86, [14] Mallat, S., A Wavelet Tour of Sigal Processig, Academic Press, 1998 [15] Aada, P., Computig Dese Displacemet Fields with Cofidece Measures i Scees Cotaiig Occlusio, Proceedigs DARPA Image Uderstadig Workshop, pp , [16] Faugeras, O., Three Dimesioal Computer Visio: a Geometric Viewpoit, The MIT Press, [17] Marr, D., Visio, W. H. Freema ad Compay, [18] Chambolle, A., Devore, R. A., Lee, N. Y., ad Lucier, B. J., Noliear Wavelet Image Processig: Variatioal Problems, Compressio, ad Noise Removal Through Wavelet Shrikage, IEEE Trasactios o Image Procesig, vol. 7, pp , [19] Julesz, B., Foudatios of Cyclopea Perceptio, Uiversity of Chicago Press, [2] Rothwell Hughes, N., Fuzzy Filters for Depth Map Smoothig, PhD Thesis, Uiversity of Wales,