LECTURE 4: FEATURE EXTRACTION DR. OUIEM BCHIR

Transcription

1 LECTURE 4: FEATURE EXTRACTION DR. OUIEM BCHIR

2 RGB COLOR HISTOGRAM

3

4 HSV COLOR MOMENTS hsv_image = rgb2hsv(rgb_image) converts the RGB image to the equivalent HSV image. RGB is an m-by-n-by-3 image array whose three planes contain the red, green, and blue components for the image. HSV is returned as an m-by-n-by-3 image array whose three planes contain the hue, saturation, and value components for the image.

5 HSV MODEL,CONT D Hue (0-360 ); the color, cp. the dominant wavelength (128) Saturation (0-1); the amount of white (130) Value (0-1); the amount of black (23)

6 HSV COLOR MOMENTS m = mean(x) s = std(x) y = skewness(x) [ m1 m2 m3 s1 s2 s3 y1 y2 y3]

7 EDGE HISTOGRAM Edge histogram descriptor (EHD) represents the frequency and directionality of edges within each image region. Simple edge detector operator are used to detect edges and group into five categories: vertical, horizontal, diagonal, anti-diagonal and nonedge. the EHD includes five bins corresponding to frequencies of the five categories.

8 EDGE HISTOGRAM (0,0) (0,1) (0,2) (0,3) sub-image (1,0) (1,1) (1,2) (1,3) image-block (2,0) (2,1) (2,2) (2,3) (3,0) (3,1) (3,2) (3,3)

9 EDGE MASKS a) vertical b) horizontal c) 45 degree d) 135 degree e)non-directional edge edge edge edge edge block_size image-block block_size

10 LOCAL EDGE HISTOGRAM Histogram bins Semantics Local_Edge [0] Vertical edge of sub-image at (0,0) Local_Edge [1] Horizontal edge of sub-image at (0,0) Local_Edge [2] 45degree edge of sub-image at (0,0) Local_Edge [3] 135 degree edge of sub-image at (0,0) Local_Edge [4] Non-directional edge of sub-image at (0,0) Local_Edge [5] Vertical edge of sub-image at (0,1) : : : : : : Local_Edge [74] Non-directional edge of sub-image at (3,2) Local_Edge [75] Vertical edge of sub-image at (3,3) Local_Edge [76] Horizontal edge of sub-image at (3,3) Local_Edge [77] 45degree edge of sub-image at (3,3) Local_Edge [78] 135 degree edge of sub-image at (3,3) Local_Edge [79] Non-directional edge of sub-image at (3,3)

11 GLOBAL HISTOGRAM local bins global bins semi-global bins

12 SHAPE FEATURE a: semi-major axis b: semi-minor axis F1 and F2 foci of the ellipse e: eccentricity of an ellipse is the ratio of the distance between the two foci, to the length of the major axis or e = 2f/2a = f/a. For an ellipse the eccentricity is between 0 and 1 (0<e<1). When the eccentricity is 0 the foci coincide with the center point and the figure is a circle. As the eccentricity tends toward 1, the ellipse gets a more elongated shape. It tends towards a line segment

13 SHAPE FEATURE For two-dimensional data, the second central moment is the covariance matrix. If X is an n-by-2 matrix of the points in the region, the covariance matrix Sigma in MATLAB can be computed as mu=mean(x,1); X_minus_mu=X - repmat(mu, size(x,1), 1); Sigma=(X_minus_mu'*X_minus_mu)/size(X,1);

14 SHAPE FEATURE The eccentricity, orientation, area, solidity, and extent are calculated. [eccentricity, orientation, area, solidity, extent] Eccentricity is calculated by first finding an ellipse with the same second moments as the region and then computing the ratio of the distance between the foci of the ellipse and its major axis length.

15 SHAPE FEATURE Orientation is defined as the angle in degrees between x-axis and the major axis of the ellipse containing the same second-moments as the region. Area is defined as the actual number of pixels within the region.

16 SHAPE FEATURE Solidity is computed as where ConvexArea is the number of pixels in the smallest convex polygon that can fully contain the region, known as the convex hull.

17 SHAPE FEATURE Extent is defined as the proportion of the pixels in the bounding box of the regions that are also in the region. It is computed as the Area divided by the area of the bounding box. Check the regionprops matlab function

18 OBJECTIVE OF MPEG-7 Standardize content-based description for various types of audiovisual information Enable fast and efficient content searching, filtering and identification Describe several aspects of the content (low-level features, structure, semantic, models, collections, creation, etc.) Address a large range of applications ( user preferences) Types of audiovisual information: Audio, speech Moving video, still pictures, graphics, 3D models Information on how objects are combined in scenes Descriptions independent of the data support Existing solutions for textual content or description

19 Salembier EXAMPLE OF QUERIES Text: Find AV material with the concepts described by the text Semantic: Find AV material corresponding to the specified semantic Image: Find an image with a similar characteristic (global or local) Music: Play a few notes and search for corresponding musical pieces Motion: Find video with specific object motion trajectories

20 Visual Descriptors Color Texture Shape Motion 1. Histogram Scalable Color Color Structure GOF/GOP 2. Dominant Color 3. Color Layout Texture Browsing Homogeneous texture Edge Histogram Contour Shape Region Shape Camera motion Motion Trajectory Parametric motion Motion Activity

21 WEB IMAGE SEARCH

22 WEB IMAGE SEARCH

23 WEB IMAGE SEARCH

24 RETRIEVAL PERFORMANCE EVALUATION Let the number of ground truth images for a query q be NG(q) Compute K(q)=min(4*NG(q), 2* max{ng(q)} ) Compute NR(q), number of found items in first K(q) retrievals, Compute MR(q)=NG(q)-NR(q), number of missed items Compute the ranks Rank(k) of the found items by counting the rank of the first retrieved item as one. A Rank of (1.25K(q)) is assigned to each of the ground truth images which are not in the first K(q) retrievals. Compute the normalized modified retrieval rank NMRR(q) as follows (next slide). Note that the NMRR(q) will always be in the range of [0.0,1.0].

25 AVERAGE RETRIEVAL RATE (AVR) AND ANMRR Compute AVR(q) for query q as follows: AVR( q) NG( q) Rank( k) NG( q) k 1 Compute the modified retrieval rank as follows: MRR( q) = AVR( q) - 0.5(1 + NG( q)) Normalized MRR, NMRR = MRR(q)/(1.25*K *NG(q)) ANMRR 1 Q Q q 1 NMRR( q)