Content-Based Image Retrieval Some Basics

Transcription

1 Content-Based Image Retrieval Some Basics Gerald Schaefer Department of Computer Science Loughborough University Loughborough, U.K. Abstract. Image collections are growing at a rapid rate, motivating the need for efficient and effective tools to query these databases. Contentbased image retrieval (CBIR) techniques extract features directly from image data and use these, coupled with a similarity measure, to search through image collections. In this paper, we introduce some of the basic image features that are used for CBIR. 1 Introduction While image libraries are growing at a rapid rate (personal image collections may contain thousands, commercial image repositories millions of images [1]), most images remain un-annotated [2], preventing the application of a typical text-based search. Content-based image retrieval (CBIR) [3, 4] does not require any extra data, as it extracts image features directly from the image data and uses these, coupled with a similarity measure, to query image collections. Image features typically describe the colour, texture, and shape content of the images, and in this paper we review several well-known descriptors that are employed in CBIR. Our emphasis is on rather simple image features which nevertheless have been shown to be effective for CBIR. In Section 2, we discuss some basic colour image features, while Section 3 focusses on incorporating spatial information into colour-based retrieval. Section 4 reviews texture image features, whereas in Section 5, we present some shape-based retrieval techniques. Section 6 concludes the paper. 2 Colour features Colour features are the most widely used feature type for CBIR and are at the heart of various image retrieval search engines such as QBIC [5] and Virage [6]. 2.1 Colour moments The simplest colour descriptor for CBIR are colour moments [7]. The n-th central (normalised) moment of a colour distribution is defined as M n (I) = n 1 N (M1 (I) c(x,y)) n, (1)

2 with M 1 (I) = 1 N c(x,y), (2) where N is the number of pixels of image I and c(x,y) describes the colour of the pixel at location (x,y). The distance between two images is defined as the sum of absolute distances between their moments (L 1 norm) 2.2 Colour histograms d MNT (I 1,I 2 ) = n M i (I 1 ) M i (I 2 ). (3) i=1 Swain and Ballard [8] introduced the use of colour histograms, which record the frequencies of colours in the image, to describe images in order to perform image retrieval. Indeed, it was Swain and Ballard s work that laid the foundations for the field of CBIR as we know it today. As distance measure they introduced (the complement of) histogram intersection defined as d HIS (I 1,I 2 ) = 1 N min(h 1 (k),h 2 (k)), (4) k=1 where H 1 and H 2 are the colour histograms of images I 1 and I 2, and N is the number of bins used for representing the histogram. It can be shown [8] that histogram intersection is equivalent to the L 1 norm and hence a metric. An alternative to the L 1 norm is to use the Euclidean distance (L 2 norm) between two histograms. This approach was taken in the QBIC system [9] and also addresses the problem of possible false negatives due to slight colour shifts by taking into account the similarity between separate histogram bins. This can be expressed in a quadratic form distance measure as d QBIC (I 1,I 2 ) = (H 1 H 2 )A(H 1 H 2 ) T, (5) where H 1 and H 2 are again the two (vectorised) colour histograms, and A is an N N matrix containing inter-bin colour differences. 2.3 Colour signatures Rather than using colour histograms, a more compact descriptor for encoding the colour distribution of images is a colour signature. Colour signatures are a set {(c 1,ω 1 ),(c 2,ω 2 ),...,(c m,ω m )} where c i define colour co-ordinates and ω i their associated weights (i.e. their relative frequencies in the image). A common way of deriving colour signatures for images is through a clustering process. Once colour signatures for images are determined, these signatures can be compared by a metric known as the earth mover s distance [10] which is a flow-based measure defined as m n i=1 j=1 d EMD (I 1,I 2 ) = f ijd ij m n i=1 j=1 f, (6) ij

3 which is based on the linear programming problem subject to i=1 j=1 Work(S 1,S 2,F) = m i=1 j=1 n f ij d ij (7) f ij 0 1 i m,1 j n (8) n f ij ω pi 1 i m j=1 m f ij ω qj i=1 1 j n m n m n f ij d ij = min( ω pi, ω qj ), where S 1 and S 2 are the colour signatures of images I 1 and I 2, F = [f ij ] is the work flow to be minimised in order to transform one colour signature to the other one, and d ij denote the colour differences between colour clusters. i=1 j=1 3 Spatial colour features Simple colour features such as colour histograms are fast to compute, and are invariant to rotation and translation as well as robust to scaling and occlusions. On the other hand, they do not carry any information about the spatial distribution of the colours. Consequently, several methods try to address this weakness. 3.1 Colour coherence vectors Colour coherence vectors [11] were introduced as such a method of introducing spatial information into the retrieval process. Colour coherence vectors consist of two histograms: one histogram of coherent and one of non-coherent pixels. Pixels are considered to be coherent if they are part of a continuous uniformly coloured area and the size of this area exceeds some threshold τ where τ is usually defined as 1% of the overall area of an image. The L 1 norm is used as the distance metric between two colour coherence vectors d CCV (I 1,I 2 ) = N [ H1(k) c H2(k) c + H1(k) s H2(k) ] s, (9) k=1 where H c i and H s i and are the histograms of coherent and non-coherent (scattered) pixels respectively.

4 3.2 Colour correlograms Another approach to incorporate information on the spatial correlation between the colours present in an image are colour correlograms [12], defined as with γ (k) c i,c j (I) = PR p1 I ci,p 2 I[p 2 I cj, p 1 p 2 = k], (10) p 1 p 2 = max x 1 x 2, y 1 y 2, (11) where c i and c j denote two colours and (x k,y k ) denote pixel locations. In other words, given any colour c i in the image, γ gives the probability that a pixel at distance k away is of colour c j. As full colour correlograms are expensive both in terms of computation and storage requirements, usually a simpler form called auto-correlogram (ACR) defined as α c (k) (I) = γ c,c (k) (I) (12) is often being used, i.e. only the spatial correlation of each colour to itself is recorded. Two CCRs are compared using d CCR (I 1,I 2 ) = 3.3 Spatial-chromatic histograms i,j [m],k [d] γ(k) c i,c j (I 1 ) γ c (k) i,c j (I 2 ) i,j [m],k [d] (1 + γ(k) c i,c j (I 1 ) + γ c (k) i,c j (I 2 )). (13) Spatial-chromatic histograms (SCHs) [13] are another alternative for representing both colour and spatial information. They consist of a colour histogram h(k) = A k n m, (14) where A k is a set having the same colour k, and n and m are the dimensions of the image; and location information on each colour characterised through its baricentre b(k) = 1 1 x, 1 1 y, (15) n A k m A k (x,y) A k (x,y) A k and the standard deviation of distances of a given colour from its baricentre 1 σ(k) = d(p,b(k)) A k 2. (16) p A k The SCH is then given as and similarity between two SCHs calculated as N k=1 min(h I 1 (k),h I2 (k)) H SCH (k) = [h(k),b(k),σ(k)], (17) d SCH (I 1,I 2 ) = 2 (18) ) ( 2 d(bi1 (k),d(b I2 (k)) 2 + min(σi 1 (k),σi 2 (k)) max(σ I1 (k),σ I2 (k))

5 4 Texture features Texture features do not exist at a single pixel but are rather a description of a neighbourhood of pixels. Texture features often complement colour features to improve retrieval accuracy in CBIR, and are also attractive since texture is typically difficult to describe in terms of words. 4.1 Local binary patterns (LBP) Local binary patterns (LBP) are a simple yet effective texture analysis technique [14]. It assigns, on a pixel basis, descriptors that describe the neighbourhood of that pixel and then forms a histogram of those descriptors. In detail, let g ( 1, 1) g ( 1,0) g ( 1,1) B = g (0, 1) g (0,0) g (0,1) (19) g (1, 1) g (1,0) g (1,1) describe the 3 3 grayscale block of a pixel at location (0,0) and its 8-neighbourhood. The first step is to subtract the value of the central pixel and consider only the resulting values at the neighbouring locations g ( 1, 1) g (0,0) g ( 1,0) g (0,0) g ( 1,1) g (0,0) LBP 1 = g (0, 1) g (0,0) g (0,1) g (0,0) (20) g (1, 1) g (0,0) g (1,0) g (0,0) g (1,1) g (0,0) Next an operator s(x) = { 1 for x 0 0 for x < 0 (21) is assigned at each location resulting in LBP 2 = s(g ( 1, 1) g (0,0) ) s(g ( 1,0) g (0,0) ) s(g ( 1,1) g (0,0) ) s(g (0, 1) g (0,0) ) s(g (0,1) g (0,0) ) (22) s(g (1, 1) g (0,0) ) s(g (1,0) g (0,0) ) s(g (1,1) g (0,0) ) Each pixel of the 8-neighbourhood is encoded as either 0 or 1 and an LBP histogram with 256 bins can be built as an image descriptor. 4.2 Co-occurrence matrix Co-occurrence matrices of an image I are defined by [15] n m { 1 if I(x,y) = i and I(x + p,y + q) = j C(i,j) = 0 otherwise x=1 y=1 (23) where i and j correspond to image (grey-level) values, and p and q are offset values. Typically several (p, q) pairs are employed and from the corresponding co-occurrence matrices several statistical features such as the entropy C(i,j)log C(i,j) calculated to form a feature vector. ij

6 5 Shape features Since true shape features would require segmentation, often global shape feature or feature distributions are employed in CBIR. Shape features are often combined with colour and/or texture features. 5.1 Edge histograms A simple yet effective shape feature can be derived by describing edge direction information [16]. Following an edge detection step using the Canny edge detector [17], a histogram of edge directions (typically in 5 degree steps) is generated, and then smoothed. Since it is a histogram feature, it can be compared using e.g. histogram intersection as in Eq. (4). 5.2 Image moments Image moments of an Image I are defined by m pq = M 1 y=0 N 1 Rather than m pq often central moments with µ pq = M 1 y=0 x=0 x p y q I(x,y) (24) N 1 (x x) p (y ȳ) q I(x,y) (25) x=0 x = m 10 m 00 ȳ = m 01 m 00 are used, i.e. moments where the centre of gravity has been moved to the origin (i.e. µ 10 = µ 01 = 0). Central moments have the advantage of being invariant to translation. Normalised central moments defined by η pq = µ pq µ γ 00 (26) with γ = p + q p + q = 2,3,... are also invariant to changes in scale. It is well known that a small number of moments can characterise an image fairly well. In order to achieve invariance to rotation, rather than using the moments themselves algebraic combinations thereof known as moment invariants

7 are used that are independent of these transformations. One such set of moment invariants are Hu s original moment invariants given by [18] M 1 = µ 20 + µ 02 (27) M 2 = (µ 20 µ 02 ) 2 + 4µ 2 11 M 3 = (µ 30 3µ 12 ) 2 + 3(µ 21 + µ 03 ) 2 M 4 = (µ 30 + µ 12 ) 2 + (µ 21 + µ 03 ) 2 M 5 = (µ 30 3µ 12 )(µ 30 + µ 12 )[(µ 30 + µ 12 ) 2 3(µ 21 + µ 03 ) 2 ] + (3µ 21 µ 03 )(µ 21 + µ 03 )[3(µ 30 + µ 12 ) 2 (µ 21 + µ 03 ) 2 ] M 6 = (µ 20 µ 02 )[(µ 30 + µ 12 ) 2 (µ 21 + µ 03 ) 2 ] + 4µ 11 (µ 30 + µ 12 )(µ 21 + µ 03 ) M 7 = (3µ 21 µ 03 )(µ 30 + µ 12 )[(µ 30 + µ 12 ) 2 3(µ 21 + µ 03 ) 2 ] + (µ 30 3µ 12 )(µ 21 + µ 03 )[3(µ 30 + µ 12 ) 2 (µ 21 + µ 03 ) 2 ] which can be employed as a shape descriptor for CBIR. 6 Conclusions In this paper, we have reviewed several basic image features employed for contentbased image retrieval. In particular, we have looked at colour, spatial colour, texture and shape features in this context. Further details and other image features are discussed in survey papers such as [3, 4], while more advanced CBIR topics are discussed in [19]. References 1. Osman, T., Thakker, D., Schaefer, G., Lakin, P.: An integrative semantic framework for image annotation and retrieval. In: IEEE/WIC/ACM International Conference on Web Intelligence. (2007) Rodden, K.: Evaluating Similarity-Based Visualisations as Interfaces for Image Browsing. PhD thesis, University of Cambridge Computer Laboratory (2001) 3. Smeulders, A., Worring, M., Santini, S., Gupta, A., Jain, R.: Content-based image retrieval at the end of the early years. IEEE Trans. Pattern Analysis and Machine Intelligence 22 (2000) Datta, R., Joshi, D., Li, J., Wang, J.Z.: Image retrieval: Ideas, influences, and trends of the new age. ACM Computing Surveys 40 (2008) Niblack, W., Barber, R., Equitz, W., Flickner, M., Glasman, D., Petkovic, D., Yanker, P.: The QBIC project: Querying images by content using color, texture and shape. In: Conf. on Storage and Retrieval for Image and Video Databases. Volume 1908 of Proceedings of SPIE. (1993) Bach, J., Fuller, C., Gupta, A., Hampapur, A., Horowitz, B., Humphrey, R., Jain, R.: The Virage image search engine: An open framework for image management. In: Storage and Retrieval for Image and Video Databases. Volume 2670 of Proceedings of SPIE. (1996) Stricker, M., Orengo, M.: Similarity of color images. In: Conf. on Storage and Retrieval for Image and Video Databases III. Volume 2420 of Proceedings of SPIE. (1995)

8 8. Swain, M., Ballard, D.: Color indexing. Int. Journal of Computer Vision 7 (1991) Faloutsos, C., Equitz, W., Flickner, M., Niblack, W., Petkovic, D., Barber, R.: Efficient and effective querying by image content. journal of Intelligent Information Retrieval 3 (1994) Rubner, Y., Tomasi, C., Guibas, L.: The earth mover s distance as a metric for image retrieval. Int. Journal of Computer Vision 40 (2000) Pass, G., Zabih, R.: Histogram refinement for content-based image retrieval. In: 3rd IEEE Workshop on Applications of Computer Vision. (1996) Huang, J., Kumar, S., Mitra, M., Zhu, W.J., Zabih, R.: Image indexing using color correlograms. In: IEEE Int. Conference Computer Vision and Pattern Recognition. (1997) Cinque, L., Levialdi, S., Pellicano, A.: Color-based image retrieval using spatialchromatic histograms. In: IEEE Int. Conf. Multimedia Computing and Systems. (1999) Ojala, T., Pietikäinen, M., Harwood, D.: A comparative study for texture measures with classification based on feature distributions. Pattern Recognition 29 (1996) Haralick, R.: Statistical and structural approaches to texture. Proceedings of the IEEE 67 (1979) Jain, A., Vailaya, A.: Image retrieval using color and shape. Pattern Recognition 29 (1996) Canny, J.: A computational approach to edge detection. PAMI 8 (1986) Hu, M.: Visual pattern recognition by moment invariants. IRE Transactions on Information Theory 8 (1962) Schaefer, G.: Content-based image retrieval - advanced topics. In: Int. Conference on Man-Machine Interactions. (2011) (in this volume).