Overcomplete Steerable Pyramid Filters and Rotation Invariance

Transcription

1 vercomplete Steerable Pyramid Filters and Rotation Invariance H. Greenspan, S. Belongie R. Goodman and P. Perona S. Raksit and C. H. Anderson Department of Electrical Engineering Department of Anatomy and Neurobiology California Institute of Tecnology Wasington University Scool of Medicine Pasadena CA St. Louis M 6311 (ayit@micro.caltec.edu) Abstract A given (overcomplete) discrete oriented pyramid may be converted into a steerable pyramid by interpolation. We present a tecnique for deriving te optimal interpolation functions (oterwise called steering coefficients). Te proposed sceme is demonstrated on a computationally efficient oriented pyramid, wic is a variation on te Burt and Adelson pyramid. We apply te generated steerable pyramid to orientation-invarianttexture analysis to demonstrate its excellent rotational isotropy. Hig classification rates and precise rotation identification are demonstrated. 1 Introduction riented filters play a key role in early visual processes and in image processing. Earlier work by Freeman and Adelson [1] and Perona [2] as sown ow te use of steerable filters allows one to use a small set of suc filters and still treat all orientations in a uniform way. Freeman and Adelson address te problem of syntesizing exactly steerable filters. Perona addresses te problem of calculating te best steerable approximation to a given impulse response. We approac ere a tird related problem. It is sometimes desirable to use a particular set of oriented filters, due to certain desired filter caracteristics, computational complexity, existing ardware implementations or oter constraints. Te question arises: wat is te best way to interpolate a given set of filters? In tis paper we present a tecnique for deriving te optimal set of interpolation functions (steering coefficients) for a given overcomplete discrete representation, tus generating a steerable representation. We illustrate our tecnique using as a sample case an oriented Laplacian pyramid, wic allows for a computationally efficient Gabor-like filtering sceme. Te oriented Laplacian pyramid filtering sceme is a variation on te Burt and Adelson pyramid [3,4]. It was presented in te context of a texture-recognition system in [5]. Some of its interesting caracteristics are its computational efficiency and compactness, wic lead to minimal ardware requirements. We describe it in section 2. We next sow ow te oriented pyramid, wic as 8=3 redundancy (tis is a more compact representation tan as previously been used in te literature [1,3]), can be transformed into a steerable one. In sections 3 and 4 we present te procedure to calculate te interpolation functions wic give us a steerable representation. We conclude tis paper by addressing te issue of rotation invariant recognition via te extracted steerable representation. We present a sceme for generating rotationallyinvariant feature-vectors for an input image, togeter wit extracting te actual rotation information. Hig classification rates and precise rotation estimation results are presented for bot syntetic and natural textured images, wic demonstrate te usefulness and precision of te steerable pyramid in te difficult real-world task of rotation invariant texture recognition. 2 Te riented Laplacian Pyramid Tere is bot biological and computational evidence supporting te use of a bank of orientation-selective bandpass filters, suc as te Gabor filters, for te initial feature extraction pase of many image-processing tasks. Tese tasks include edge-detection, motion-flow analysis and texture recognition [5,6,7,8]. rientation and frequency responses are extracted from local areas of te input image and te statistics of te coefficients caracterizing te local area form a representative feature vector. In te application domains listed above, te extracted feature vectors are used as an intermediate step towards orientation analysis, or oter iger-level analysis. Te constraints are terefore computational efficiency and memory requirements (especially important for real-world applications), as opposed to acieving a complete self-inverting representation wic is important for coding and reconstruction purposes. It is tis distinction wic motivates us into using te oriented

2 Laplacian pyramid described below, wic is bot computationally efficient and compact. Te pyramid filtering sceme In a pyramid representation te original image is decomposed into sets of lowpass and bandpass components via Gaussian and Laplacian pyramids, respectively [3]. Te Gaussian pyramid consists of lowpass filtered (LPF) versions of te input image, wit eac stage of te pyramid computed by lowpass filtering and subsampling of te previous stage. Te Laplacian pyramid consists of bandpass filtered (BPF) versions of te input image, wit eac stage of te pyramid constructed by te subtraction of two corresponding adjacent levels of te Gaussian pyramid. We use te Filter-Subtract-Decimate (FSD) Laplacian pyramid [4], wic is a variation on te Burt Laplacian pyramid [3]. In te following we refer to te input image as G, te LPF versions are labeled G 1 tru G N wit decreasing resolutions and te corresponding BPF versions are labeled L tru L N respectively G2 G1 RIGINAL IMAGE G GAUSSIAN PYRAMID L2 L1 L LAPLACIAN PYRAMID m 1 m 2 m 3 m m 1 m 4 m m 3 m 2 1 m 4 m 3 m RIENTED PYRAMID PYRAMID IMAGE REPRESENTATIN P 21 P 11 P 24 PWER MAPS Figure 1: Block diagram of te oriented pyramid generation. Te power maps represent te local statistics of te oriented pyramid s coefficients (section 5). Freq. r. P 14 P 1 32 P 4 G n+1 = W G n ; L n = G n ; G n+1 G n+1 = Subsampled G n+1 (1) Te LPF, W, is Gaussian in sape, normalized to ave its coefficients sum to 1. Te values used in tis work for W, wic is a 5-sample separable filter, are (1/16, 1/4, 3/8, 1/4, 1/16). In order to extract te orientationally tuned band-pass filtering responses, te oriented pyramid is formed next. Te oriented pyramid is te result of modulating eac level of te Laplacian pyramid wit a set of oriented sine waves, followed by anoter LPF operation using a separable filter, and corresponding subsampling, as defined in (2): n = LP F [e (i ~ k ~r) Ln [x y]] (2) were n is te oriented image at scale n and orientation, ~r = x~{ + y~ (x and y are te spatial coordinates of te Laplacian image), ~ k =(=2)[cos ~{ + sin ~ ] and =(=N )( ; 1) ( = 1::N ): In tis work we use 4 oriented components (N = 4). From (2), eac level (n) of te pyramid is tus modulated by te following complex sinusoids: m 1 (x y) =e i(=2)x ; m 2 (x y) =e i(p 2=4)(x+y) m 3 (x y) =e i(=2)y ; m 4 (x y) =e i(p 2=4)(y;x) Tese four modulators differ only in teir orientations, wic are 45 9 or 135 for m 1 troug m 4, respectively. Te origin of x and y is taken to be te center of te image being modulated. Note tat te modulating (3) Figure 2: Left: A set of oriented pyramid filters, n. Real and imaginary components are presented, top and bottom, respectively, for n = and = 1::4. Rigt: Power spectra caracteristics for te cosen filter set (+ conjugate counterparts). frequency remains constant for eac level of te pyramid. After modulation, te Laplacian images are lowpass filtered and subsampled. At tis point, te Laplacian images ave effectively been filtered by a set of log-gabor filters: k(x 1 2 y) = 2 e;(x +y 2 )=2 mk (x y) ; k = 1::4 (4) Fig. 1 sows a block-diagram of te orientation-pyramid generation. A set of oriented-pyramid filters are displayed in Fig. 2. Pyramid caracteristics We briefly present some caracteristics of te pyramid. Te interested reader is referred to [1,3,4,1] for more elaborate details. - In te FSD pyramid (1), subtraction occurs before te decimation step. Tis ensures tat aliasing does not get incorporated into te mid-band regions of te bandpass images, L n, and gives better bandpass caracteristics overall. - Te FSD pyramid allows for a simple pipeline arcitecture.

3 - Te filtering operation in (2) is not te standard one found in te literature. Usually, te original image is filtered wit a set of oriented sinewave modulated Gaussian filters. In (2) te oriented filters are applied to te bandpass image, L n. Tis ensures good low-frequency (dc) rejection. In addition, a reversal in te ordering of te filtering operations is performed (te image is first modulated by a sinewave and ten LPFed, rater tan modulating te LPF prior to convolving wit te image). Tis reversal gives us separable filters, and terefore it allows for a computationally efficient filtering sceme. We next investigate te redundancy of te generated pyramid. Te redundancy in te nonoriented Laplacian pyramid representation is 4/3. In te pyramid sceme defined above, we use four complex oriented filters to create eigt oriented bandpass components from eac nonoriented Laplacian level. Te eigt include te real and imaginary response maps from eac complex oriented filter modulation. Since tis involves lowpass filtering after te modulation, it is possible to subsample tese oriented bands by a factor of 2 in eac dimension. From eac band of size M M, we tus old 8 bands of size M 2 =4. Te total number of pixels at eac level terefore increases from M 2 to (M 2 8)=4 = M 2 2, leading to an increase of redundancy by a factor of two. verall, te redundancy of our oriented pyramid is 4=3 2 = 8=3. Tis pyramid is more compact tan oter oriented pyramids described in te literature wic usually exibit 16=3 redundancy [1-3]. In te following sections we investigate into te extracted pyramid kernels. We sow tat we span te orientation space, we extract te interpolation coefficients tat allow steerability in orientation, and finally, we utilize tese properties for rotation invariant recognition. 3 Spanning te rientation Space In tis section we sow tat te oriented pyramid as defined in te preceding section, wit te selected oriented kernels of 45 bandwidt, spans te orientation space. Following te work of Perona [2], we make use of te singular-value decomposition (SVD) to investigate te independence of te set of oriented pyramid kernels (as in equation 2). Tis procedure consists of te following steps: Generate 36 oriented pyramid kernels (at a single scale) via equation 2 wit N = 36. Concatenate eac of tese 36 kernel matrices into column-vectors, and combine tese column vectors to form a large matrix, A Perform te SVD by finding te matrices U, V, and Percent of total sum Cumulative sum of singular values Index of Singular Value Figure 3: SVD decomposition for te oriented pyramid kernels. Te first seven singular values contain approx. 99:5% of te sum of all te singular values. diagonal matrix suc tat A = UV T (5) Te diagonal matrix contains te square roots of te positive eigenvalues of A T A. Te number of nonzero eigenvalues in is equal to te number of linearly independent column vectors in A. Upon inspecting te results of te SVD, te first seven singular values, 1.. 7,in contain approximately P 99:5% of te sum of all te singular values ( i ). Tis is sown in Fig. 3. Te above result indicates tat a set of eigt filters, i.e., an orientation bandwidt of 45, is sufficient to span te 36 of orientation space wit more tan 99% accuracy. Te four filters, n1 troug n4, and teir conjugate counterparts, wic we ereon term n5 troug n8, satisfy tis requirement. Te cosen set of 4 filters are sown in Fig. 2 left. Te filters combined power spectra (above 4 togeter wit teir conjugate counterparts) covers uniformly te 36 orientation space, as sown in Fig. 2 rigt. 4 Interpolating in rientation Space Given te set of oriented pyramid filters, n1 troug n8,wenextdefine te interpolation functions (or steering coefficients) wic allow us to use te finite set of eigt filters (per scale) to syntesize oriented filters across te entire orientation space. Note tat in tis section we assume te input image to te pyramid to be a delta function, G (x y) =(x y). We wis to use a finite set of oriented filters to calculate te output of filters at any orientation in a continuum. Let n k () k = 1::8, represent te interpolation coefficients in orientation space. We wis to calculate te filter output for any given angle, wic

4 we define as ˆF (x), via a linear interpolation sceme as follows: 8 ˆF n (x) = n k () n k (x) (6) k=1 For clarity purposes we ereon avoid using te scale notation, wit te understanding tat te following derivation is performed at eac scale, n, independently. ur goal is to minimize te error between te filter output, F (x), and te interpolated output, ˆF (x), (in space for a particular orientation ): Comparison of beta (solid) and sinc (dased) We ave min n k kf (x) ; ˆF (x)k 2 < 2 (7) kf (x); ˆF (x)k 2 = kf (x)k 2 +k ˆF (x)k 2 ;2F (x) ˆF (x)i were and k ˆF (x)k 2 = = k k F (x) ˆF (x)i = k () k (x) () (x)i (8) k k i < 2 (9) = = (x) F (x)i F i < 2 Γ () (1) wit Γ () =F i < 2. Using (8-1), we need to minimize te following expression wit respect to : k k i < 2 ; 2 k {z } a Γ () {z } b (11) Te derivative wit respect to z of te left term (a) gives: 2 P k k z k i. Te derivative of te rigt term (b) gives: ;2Γ z (). Equating te sum of te above two terms to zero leads to te following equation for te s: k z k i < 2 k () =Γ z () In matrix form we ave: z = 1::N k = 1::N (12) = ; (13) Angle in degrees Figure 4: ne caracteristic interpolation function (solid), as compared wit te Sinc function (dased). wit = k i < 2, = a column of te k s, k = 1::N, and ; = a column of te Γ z s, z = 1::N. In te ortonormal case, is a diagonal (identity) matrix since k i = k : In tat case from (12) we get: () =Γ () =F i < 2 (14) For te nonortonormal case, te solution requires more computation one metod would be Gauss elimination metod, te oter would be to decompose by SVD to UV T and use tis to calculate ;1. Here, UU T = I and VV T = I. Te inverse matrix, ;1, can be found as: ;1 = V[diag(1=( j )]U T (15) Te solution for can now be extracted as: = V[diag(1=( j )]U T ; (16) were in te case of a zero eigenvalue, j =, te corresponding 1= j in ;1 gets replaced by a zero. Te above scenario takes care of all possible matrices, even if te matrix is not full rank. verall, if ; is in te range of ten te extracted functions are exact. If ; is not in te range ten te functions are te closest we can find in least-squares sense; i.e., minimizing j ; ;j. Using te eigt 45 bandwidt oriented filters, 1 troug 8, we extract te eigt steering coefficients ( k for k = 1::8), as outlined above. A plot of 5 () over te range ; 36 is sown in Fig. 4. It is very similar to a sinc function. Te curves for eac k ()k = 1::8 are cyclic sifts of one anoter at 45 increments. Wit te interpolation functions in and, we can now go back and calculate te oriented filters, ˆF, from te finite

5 .5.4 Percent error in oriented pyramid reconstruction Feature vectors are extracted from te oriented pyramid via te following procedure: Power maps are first extracted from te oriented pyramid. Te power of eac filtered map can be defined as: Percent error Angle in degrees Figure 5: Percent error in te reconstruction of oriented filters across te continuous orientation space, = ; 36 degrees, from te finite filter set, k k = 1::8. set of oriented filters, k, as: ˆF = k k () k (17) across te continuous orientation space, = ; 36. Fig. 5 sows te percent error, E(), in te reconstruction of te oriented filters, for = ; 36 wit steps of 5. Here te interpolation error E is defined as: E() = F ; ˆF kf k 1: (18) Note tat te peak error is less tan 1%. Tis is in agreement wit te SVD bound found in section 2. We ave tus completed te proof of te pyramid steerability. An alternative sceme to te interpolation functions derivation of above wic does not involve matrix inversions, makes use of te Gram-Scmidt ortogonalization process. Tis sceme is outlined in [1]. 5 Application of te Steerable Pyramid Kernels for Rotation-Invariant Texture Recognition We conclude tis paper by demonstrating te application of te steerable pyramid kernels to rotation-invariant texture recognition. Here we are interested in learning a set of textured inputs, following wic we would like to recognize new test inputs as belonging to one of te prelearned classes, even if te new input is rotated relative to te original input. Furtermore, we wis to state te orientation of te test input relative to te original one. An example of 1 textures, 8 taken from te Brodatz texture database [9], and 2 (cardboard, denim) acquired by us, is presented in Fig. 6. P n = j n j n= 1 2 = : (19) Te power maps form a pyramid of te local statistics of te oriented pyramid s coefficients, wic caracterize te image local-area response to te different orientations and frequencies. Levels and 1 of te power-pyramid are lowpassed and subsampled to be te size of te smallest level of te pyramid (see Fig. 1). Eac pixel in te resultant power maps tus represents an 8 8 window in te original image. 15 dimensional feature-vectors are formed from te extracted power maps. Tese vectors consist of te 4 oriented components per scale togeter wit a non-oriented component extracted from te Laplacian pyramid. For additional details on te feature extraction stage see [5]. For a given input texture we define a caracteristic curve (per scale) across orientation space as te texture s response curve to any oriented filter in te 36 space. Te four oriented components (per scale) and teir conjugate counterparts, sample te texture s caracteristic curve at eigt points. Tese samples cycle along te continuous curve as te texture is rotated. We ave so far sown te steerability of te oriented filters (17). Te exact interpolation equation for te filter powers is complex and will not be derived ere. Given te fact tat te energy is lowpass in orientation we make te approximation tat te filter output powers can be interpolated wit te (sinc-like) functions (tis is confirmed by empirical observations): ˆP k k ()P k : (2) Here, P k k = 1::4, are te 4 oriented power components, P n = 1::4, and P k k = 5::8 is a duplicate set representing te power components of te conjugate counterparts. ˆP represents te estimated power map for te texture rotated at. We test te estimation accuracy on a few texture examples. Fig. 7 presents te estimation error, E(), across orientation space (steps of 5 ), for a set of 5 textures. Here: E() = P ; ˆP kp k 1: (21) wit P representing te actual power map extracted from te input texture rotated to, and ˆP representing te estimated response based on te original, nonrotated power maps. Te error is less tan 3%. Tese results demonstrate tat te finite set of oriented filters wic we cose

6 Figure 6: 1 texture database. Top row (left to rigt): bark, calf, clot, cardboard, denim. Bottom row (left to rigt): grass, pig, raffia, water, wood. Percent Error Percent error in feature vector calculation clot denim raffia wood news Angle in degrees Figure 7: Percent error in te calculation of caracteristic curves, 5 texture case. for our representation gives us a steerable pyramid and indicates te validity of te interpolation functions in a real application. Furtermore, te results confirm te validity of (2). Sifting to a DFT representation We note tat four samples allow us to reconstruct te caracteristic curve for eac texture in orientation space. We can terefore sift to any oter four-dimensional representation. For recognition purposes it is beneficial to use a sift-invariant representation, suc as te Discrete Fourier Transform (DFT). Companion feature-vectors are formed, f, ˆ wic contain te 4-point DFT of te oriented components for eac level, as in (22), in addition to te tree unaltered non-oriented components from te original featurevector. In our case, f q represents te four components P nq. ˆ f k+1 = 3 q= f q+1e ;iqk=2 for k = (22) Te DFT can be used to create companion feature-vectors tat are invariant in magnitude and reveal troug teir pase te rotation of te input texture. To begin investigating te prospect of an invariant feature waveform, te feature-vectors for an ideal sinusoidal grating texture at orientations from to 45 wit steps of 5 were set aside, for a total of 1 feature-vectors. Ten a companion set of 1 feature-vectors was formed. Fig. 8 top sows magnitudes of te 1 DFT s for te ideal sinusoidal grating and denim test textures. Fig. 8 bottom sows te pase of te DFT s for te sinusoidal grating. Te magnitudes overlap onto a single caracteristic curve. Wit regard to te pase, eiter f2 ˆ or f4 ˆ can be inspected to determine te amount of rotation on te input. Rotation invariant texture recognition results Finally, we present results of applying te above analysis to a 1-texture recognition task (see Fig. 6). Te test consists of presenting different images from te input set, wit eac image arbitrarily rotated at one of 5 angles: (, 1, 2, 3, 4) degrees. In te recognition process featurevectors are extracted and eac component is averaged over te entire image, to produce one representative featurevector per input. Te extracted feature-vector, f, isnext used to generate te companion feature-vector, f, ˆ via te DFT transformation of te previous section. For eac of te 1 textures we investigate te magnitude deviation of te representative feature-vector, f, ˆ as te input texture is rotated. We compare te standard deviation witin eac class, c i (in te 15 dimensional space), to te average (and minimum) distance between te mean of class c i and te means of all oter texture classes c j, j 6= i; i.e. te average (/min) interclass distance. Tis is sown in te following table: texture innerclass interclass interclass std. avg. min. bark calf clot cardboard denim grass pig raffia water wood In te above results we observe more tan a factor of 1 difference between te innerclass and average interclass distances, for most textures. Looking at te minimum interclass distance for eac texture, we again see tat it is muc larger tan te innerclass standard deviation, except for te clot and pig texture pair, for wic te representative feature-vector means are very close. Tis difficulty is inerent in te similarity of te textures, as can be seen in Fig. 6. Te small innerclass standard-deviation strongly

7 18 DFT magnitude for rotated denim and sinusoidal grating (:5:45) 6 Summary and Conclusions Magnitude sine denim Component of DFT pase of DFT DFT pase for rotated sinusoidal grating (5:5:85) nt term of DFT Figure 8: Top: DFT magnitudes for 1 rotated sinusoidal and denim test textures. Bottom: DFT pase for 1 rotated ideal sinusoidal-grating textures. indicates te consistency of te DFT magnitude representation; i.e., te invariance of te response wit te rotation of te input textures. To make tis claim stronger, we use te well known K-nearest-neigbor classification sceme on te set of 1 textures above. In te training stage, we use examples per texture class, wit no rotation. Te test set consists of a new set of textures, wic are rotated arbitrarily in one of te 5 angles: ( ) degrees. In tis 1 texture recognition case we get 1% correct classification. nce te identity is found we utilize te pase information from te DFT representation, to estimate te orientation of te test input, relative to te original texture from te training set. Here we are interested in te error, in degrees, between te true rotation angle and te estimated rotation angle. Te average rotation-angle estimation error for te 1 textures is :84. Bot te perfect class identification and te igresolution orientation estimation, as presented above, are very encouraging results in te difficult domain of rotation invariant natural texture identification. In tis paper we ave presented an optimal tecnique for deriving te set of interpolation functions (or steering coefficients) wic enable us to convert a given overcomplete oriented filter set into a steerable representation. As a sample case we ave cosen to work on a computationally efficient oriented Laplacian pyramid. We described te caracteristics of te 8=3 redundant pyramid and ave sown tat te pyramid is steerable by defining a set of eigt 45 bandwidt oriented kernels and deriving te corresponding steering coefficients. Properties of te kernels and interpolation functions ave been investigated. Finally, we demonstrated igly encouraging results in applying te pyramid for rotation-invariant recognition. A similar framework to te one presented ere can be applied to scale invariance. Future work involves extending te work to scale and rotation invariant texture recognition on large databases [11]. Acknowledgements Tis work was supported in part by Pacific Bell, and in part by ARPA and NR under grant no. N14-92-J-186. H. Greenspan was supported in part by an Intel fellowsip. P. Perona is supported by NSF Researc Initiation grant IRI S. Raksit and C. H. Anderson are supported in part by NR Grant N14-89-J References [1] W. T. Freeman and E. H. Adelson, Te Design and Use of Steerable Filters, IEEE Trans. on Pattern Analysis and Macine Intelligence, Vol. 13, No. 9, , Sept [2] P. Perona, Deformable Kernels for Early Vision, IEEE Conference on Computer Vision and Pattern Recognition, pages , June [3] P. J. Burt and E. A. Adelson, Te Laplacian Pyramid as a Compact Image Code, IEEE Transactions on Communications, Vol. 31, , [4] C. H. Anderson, A Filter-Subtract-Decimate Hierarcical Pyramid Signal Analyzing and Syntesizing Tecnique, United States Patent 4,718,14, [5] H. Greenspan, R. Goodman, R. Cellappa and C. Anderson, Learning Texture Discrimination Rules in a Multiresolution System, to appear in te special issue on Learning in Computer Vision of te IEEE Transactions on Pattern Analysis and Macine Intelligence, July [6] H. E. Knutsson and G. H. Granlund, Texture analysis using twodimensional quadrature filters, In IEEE WorksopComp. Arc. Patt. Anal. Im. Dat. Base Man., Pasadena, CA, [7] M. R. Turner, texture discrimination by gabor functions, Biol. Cybern, 55:71-82, [8] A. C. Bovik, M. Clark and W. S. Geisler, MulticannelTextureAnalysis Using Localized Spatial Filters, IEEE Transactions on Pattern Analysis and Macine Intelligence, Vol. 12, 55-73, 199. [9] P. Brodatz, Textures, New York:Dover, [1] H. Greenspan, P.D. Tesis in preparation. [11] H. Greenspan, S. Belongie, P. Perona and R. Goodman, Rotation Invariant Texture Recognition Using a Steerable Pyramid, submitted to te 12t International Conference on Pattern Recognition, ct