Texture and Shape for Image Retrieval Multimedia Analysis and Indexing

Transcription

1 Texture and Shape for Image Retrieval Multimedia Analysis and Indexing Winston H. Hsu National Taiwan University, Taipei Office: R512, CSIE Building Communication and Multimedia Lab () October 23, 2007 Outline Texture Statistical features Spectral features Edge Shape -2-1

2 Reminder Homework #2 Due: (noon, Tuesday, November 13) Rule deliver quality work on time with integrity!! Midterm A small recap of what we mentioned (major literatures) High-level concepts mentioned in the course Open book (no computer) but requiring no print-out Mailing list Syllabus (tentative) 1 9/25/07 holiday 2 10/02/07 introduction 3 10/09/07 mpeg; shot detection 4 10/16/07 cbr overview; color 5 10/23/07 texture+shape; relevance feedback 6 10/30/07 multidimensional indexing; feature reduction 7 11/06/07 midterm 8 11/13/07 gmm+cbir; svm+cbir (graphical/discriminative models) 9 11/20/07 structure discovery (sports; story) 10 11/27/07 TRECVID; concept detection; image annotation 11 12/04/07 concept detection; image annotation 12 12/11/07 un-/supervised clustering (clustering) 13 12/18/07 video retrieval 14 12/25/07 intro audio/music 15 01/01/08 holiday 16 01/08/08 project presentation #1, # /15/08 final (no course) 18 01/22/08 project report due

3 Scenario of Content-Based Image Retrieval query image retrieved images Image Database feature extraction distance metric feature (vector) space -5- Fusion of Multimodal Features How to weigh the feature significance? Cross-validation approach User-selected Automatically weighting by relevance feedback 1 Score -> Ranking -> Retrieval Results by Different Features 1 0 N Normalised Results Fusion approaches such as: Sum (Borda fuse) WtSum (weigthed Borda Fuse) Max (Round-Robin) * From Kieran Mc Donald -6-3

4 What is texture Texture Has structures or repetitious pattern, i.e., checkboard Has statistical patterns, i.e., grass, sand, rock Why texture? Applications to satellite images, medical images Describe contents of real world images, i.e., clouds, fabrics, surfaces, wood, stone Data set e.g., Brodatz: famous texture photographs for imagetexture analysis Man-made textures & natural objects

5 Mosaic of Brodatz Texture Types of Computational Texture Features Structural describing arrangement of texture elements Statistical characterizing texture in terms of statistical features Co-occurrence matrix Tamura (coarseness, directionality, contrast) Multiresolution simultaneous autoregressive model (MRSAR) Edge histogram Spectral based on analysis in spatial-frequency domain Fourier domain energy distribution Gabor Pyramid-structure wavelet transform (PWT) Tree-structure wavelet transform (TWT) Laws Filter

6 Co-occurrence Matrix Co-occurrence matrix C d Specified with a displacement vector d = {(row, column)} Entry C d (i, j) indicates how many times a pixel with gray level i is separated from a pixel of gray level j by the displacement vector d Usually use normalized version of C d physical meaning? d = (1, 1) Sometimes use symmetric version of C d Examples Co-occurrence Matrix (cont.) * From Prof. Leow Wee Kheng, NUS

7 Co-occurrence Matrix (cont.) Consider the following example (black = 1, white = 0) For d=(1,1), the only non-zero entries are at (0,0) and (1,1) captures diagonal structure For d=(0,1), the only non-zero entries are at (0,1) and (1,0) captures horizontal structure Co-occurrence Matrix (cont.) Measures on the following features What does it mean when entropy has the largest value as the N d (i,j) are equal? A almost-obsolete feature Not effective for classification and retrieval Expensive to compute

8 Tamura Selected Textual Properties fine / coarse high contrast / low contrast roughness / smooth directional / non-directional line-like / blob-like regular / irregular Usefulness in Describing Texture Psychophysical experiments high correlation between some groups of properties Coarseness Contrast Roughness Similar correlations Orientation Line-like Regularity Computational measures Coarseness Contrast Orientation

9 Tamura Coarseness Goal Pick a large size as best when coarse texture is present, or a small size when only fine texture Step 1: Compute averages at different scales at every points Tamura Coarseness (cont.) Step 2: compute neighborhood difference at each scale on opposite sides of different directions

10 Tamura Coarseness (cont.) Step 3: select the scale with the largest variation Step 4: compute the coarseness crs Tamura Contrast Gaussian-like histogram distribution low contrast Histogram polarization. Is it Gaussian? How many peaks it has? Where they are? Polarization can be estimated by the kurtosis ()

11 Tamura Contrast (cont.) distribution with two separate peaks unimodal distribution Contrast estimate is given by: Tamura Orientation Building the histogram of local edges at different orientations By deriving the edge magnitude at X and Y directions

12 Tamura Orientation (cont.) Compute the estimate from the sharpness of the peaks By summing the second moments around each peak e.g., flat histogram large 2nd moment (variance) small orientation -23- (MR)SAR [Mao 92] Each pixel is a random variable whose value is estimated from its neighboring pixels + noise A kid of Markov Random Field model SAR Model (Simultaneous Autoregressive) Describes each pixel in terms of its neighboring pixels. MRSAR Model (MultiResolution SAR) Describing granularities by representing textures at variety of resolutions SAR applied at various image levels SAR Metric parameter differences SAR model parameters SAR input image image pyramid

13 Edge histogram (EHD) Edge Histogram Captures the spatial distribution of the edge in six statues: 0º, 45º, 90º, 135º, non direction and no edge. Utilizing the filters Global EHD of an image: Concatenating 16 sub EHDs into a 96 bins Local EHD of a segment 90 edge 0 edge 45 edge 135 edge non-directional edge Grouping the edge histogram of the image-blocks fallen into the segment Macro-block Image-block Vector Space Concept Orthonormal Bases (d-dim. vectors) Any vector in a vector space can be expanded by the set of orthonormal signals Response for basis k, Transform to the new bases (1D/2D) Fourier bases are sets of orthornomal signals

14 F( g( x, y) )( u, v) = g x, y The Fourier Transform Represent function on a new basis Think of functions as vectors, with many components We now apply a linear transformation to transform the basis dot product with each basis element ## R 2 ( )e ( ) dxdy!i2" ux +vy In the expression, u and v select the basis element, so a function of x and y becomes a function of u and v basis elements have the!i2" ux +vy form e ( ) Visual Sinus Pattern* *The following 5 slides are from Jaap van de Loosdrecht, Noordelijke Hogeschool Leeuwarden

15 Visual Sinus Pattern w/ Low Frequency Sinus Pattern Rotated 45 Deg

16 2D Sinus Pattern 2D Rectangle Difference in spatial vs. frequency domain 1D sync function of different scales

17 Interpreting the Power Spectrum Explain structures in power spectrum 3 dark 2 3 bright high frequency low frequency DC 1 Phase and Magnitude Fourier transform of a real function is complex difficult to plot, visualize instead, we can think of the phase and magnitude of the transform Phase is the phase of the complex transform Magnitude is the magnitude of the complex transform Curious fact all natural images have about similar magnitude transform hence, phase seems to matter, but magnitude largely doesn t Same for audio? Demonstration Take two pictures, swap the phase transforms, compute the inverse - what does the result look like?

18 This is the magnitude transform of the zebra pic

19 This is the phase transform of the zebra pic

20 This is the magnitude transform of the cheetah pic This is the phase transform of the cheetah pic

21 Reconstruction with zebra phase, cheetah magnitude Reconstruction with cheetah phase, zebra magnitude

22 F T Natural Images and Their FT What happened to the FT patterns when the texture scale and orientation are changed? Frequency Domain Features Fourier domain energy distribution Angular features (directionality) where, Radial features (coarseness) where, Uniform division may not be the best!!

23 Gabor Texture Fourier coefficients depend on the entire image (Global) we lose spatial information Objective: local spatial frequency analysis Gabor kernels: looks like Fourier basis multiplied by a Gaussian The product of a symmetric (even) Gaussian with an oriented sinusoid Gabor filters come in pairs: symmetric and anti-symmetric (odd) Each pair recover symmetric and anti-symmetric components in a particular direction (k x, k y ): the spatial frequency to which the filter responds strongly σ : the scale of the filter. When σ = infinity, similar to FT We need to apply a number of Gabor filters are different scales, orientations, and spatial frequencies Example Gabor Kernel Zebra stripes at different scales and orientations and convolved with the Gabor kernel The response falls off when the stripes are larger or smaller The response is large when the spatial frequency of the bars roughly matches the windowed by the Gaussian in the Gabor kernel Local spatial frequency analysis zebra image Gabor kernel magnitude of the filtered image

24 Gabor Texture (cont.) Image I(x,y) convoluted with Gabor filters h mn (totally M x N) Using first and 2nd moments for each scale and orientations odd Features: e.g., 4 scales, 6 orientations 48 dimensions even Gabor kernels Gabor Texture (cont.) Arranging the mean energy in a 2D form structured: localized pattern oriented (or directional): column pattern granular: row pattern random: random pattern scale orientation frequency domain

25 Laws Texture Energy Features Non-Fourier type bases Match better to intuitive texture features The filter algorithm Filter the input image using texture filters Computer texture energy by summing the absolute value of filtered results in local neighborhoods around each pixel Combine features to achieve rotational invariance Law s Texture Masks (1) Basic 1D masks can be extended to create 2D masks L5 (Level) = [ ] (Gaussian) gives a center-weighted local average E5 (Edge) = [ ] (gradient) responds to row or column step edges S5 (Spot) = [ ] (LoG) detects spots R5 (Ripple) = [ ] (Gabor) detects ripples

26 Law s Texture Masks (2) Create 2D mask E5 L5 E5L5-51- Laws Filters (2D)

27 Laws Process -53- Wavelet Features (PWT, TWT) Wavelet PWT: pyramid-structured wavelet transform Decomposition of signal with a family of basis functions with recursive filtering and sub-sampling Each level, decomposes 2D signal into 4 subbands, LL, LH, HL, HH (L=low, H=high) Recursively decomposes the LL band Feature dimension (3x3x1+1)x2 = 20 TWT: pyramid-structured wavelet transform Some information in the middle frequency channels Feature dimension 40x2 = 80 original image PWT TWT

28 Texture Comparisons [Ma 98] Retrieval performance of different texture features according to the number of relevant images retrieved at various scopes using Corel Photo galleries # of relevant images MRSAR (M) Gabor TWT PWT MRSAR Tamura (improved) Coarseness histogram directionality edge histogram Tamura # of top matches considered Texture Comparisons (cont.) [Ma 98] Retrieval performance of texture features in terms of the number of top matches considered using Brodatz album recall Running Gabor MRSAR (M) TWT PWT MRSAR Tamura (improved) Tamura Coarseness histogram directionality edge histogram Running # of top matches considered

29 Texture Comparisons (cont.) Images of rock samples in applications related to oil exploitation [Li 00] Texture Comparisons (cont.) Images of rock samples in applications related to oil exploitation Gabor descriptors outperform the others [Li 00]

30 Distance metrics DO matter All based on Gabor features Euclidean vs. learned (supervised) distance metric The later was maintained with texture thesaurus Learned Similarity [Ma 96] Euclidean distance learned (supervised) distance -59- Shape Region-base descriptor Contour-based Shape Descriptor 2D/3D Shape Descriptor Some relevant ones are included in MPEG-7 Not easy to derive automatically [Bober 01]

31 Region-based vs. Contour-based Descriptor Columns indicate contour similarity Outline of contours Rows indicate region similarity Distribution of pixels Region-based Descriptor Express pixel distribution within a 2D object region Employs a complex 2D Angular Radial Transformation (ART) 35 fields each of 4 bits Rotational and scale invariance Robust to some non-rigid transformation L 1 metric on transformed coefficients Advantages Describing complex shapes with disconnected regions Robust to segmentation noise Small size Fast extraction and matching

32 Contour-based Descriptor It s based on Curvature () Scale-Space (CSS) representation Found to be superior to Zernike moments ART Fourier-based Turning angles Wavelets Rotational and scale invariance Robust to some non-rigid transformations For example Applicable to (a) Discriminating differences in (b) Finding similarities in (c)-(e) (a) (b) (c) (d) (e) Problems in Shape-based Indexing Many existing approaches assume Segmentation is given Human operator circle object of interest Lack of clutter and shadows Objects are rigid Planar (2-D) shape models Models are known in advance

33 Summary Texture features Statistical Spectral Texture computation are time-consuming compressed domain features? Shape features Multimodal fusion are quite helpful Next week Efficient indexing on high-dimensional data Feature reduction