866 FAN Yun, WANG Runsheng Vol.17 The paper is organized as follows. In the next section an image retrieval system is overviewed; in Section 3 the met

Transcription

1 Vol.17 No.6 J. Comput. Sci. & Technol. Nov An Image Retrieval Method Using DCT Features FAN Yun ( Π) and WANG Runsheng (Ξ Λ) ATR National Lab, National University of Defense Technology, Changsha , P.R. China fanyun cn@yahoo.com Received April 6, 2001; revised November 13, Abstract In this paper a new image representation for compressed domain image retrieval and an image retrieval system are presented. To represent images compactly and hierarchically, multiple features such as color and texture features directly extracted from DCT coefcients are structurally organized using vector quantization. To train the codebook, a new Minimum Description Length vector quantization algorithm is used and it automatically decides the number of code words. To compare two images using the proposed representation, a new efcient similarity measure is designed. The new method is applied to an image database with 1,005 pictures. The results demonstrate that the method is better than two typical histogram methods and two DCT-based image retrieval methods. Keywords image retrieval, vector quantization, color histogram, DCT 1 Introduction Content-based image retrieval now plays a central role in the application areas such as multimedia database systems and digital libraries, and has got more and more attentions [1;2]. Direct retrieval from compressed images is a very active and important research area, because most image and video data are stored and transmitted via a compressed format to save storage and transmission cost. The latest image retrieval methods in the compressed domain are reviewed in [3]. Several techniques are proposed for extracting features and retrieving directly from DCT coefcients [4 10]. These methods are successful with a high retrieval efciency. The fact that the structure information for representing image and multiple kinds of features now are seldom used simultaneously proves that the current compressed domain retrieval techniques can still be improved signicantly. In this paper, we present a new image representation for image retrieval, and a corresponding retrieval system. Images are represented compactly using the vector quantization technique. Using the new representation, multiple kinds of features, such as color and texture, extracted directly from DCT coefcients, are structurally organized. Unlike traditional histogram-based methods, our image representation has no feature dimension restriction, and is tailored for every image, and can be extended to a multi-hierarchy representation such as the whole image, a sub-image and a region naturally. To train the codebook, a new minimum description length (MDL) based algorithm is used to decide the number of code words automatically. To compare two images, a new efcient similarity measure is proposed, which is some kind of encoding-based distance and can be viewed as the extension of the well-known Kullback-Leibler measure of cross entropy. It is natural to use the new measure to compare two multi-dimensional feature distributions. The new method is applied to an image database with 1,005 pictures. Satisfying results are got. It demonstrates that the precision ratio and the recall ratio of the new method is higher than the two typical histogram and the two DCT-based retrieval methods. Furthermore the new method needs no IDCT, so it is more efcient and suitable for retrieving images on the Internet. Sub-image retrieval is also implemented and compared with the histogram intersection method. Experimental results demonstrate that the new method is more effective. According to the compression algorithm, image is divided into many non-overlap 8 8 blocks, feature vectors are extracted from every block, making up the feature vector set. Cluster centers named code vectors are obtained by clustering analysis, the set of code vectors is named codebook, and the code vector is named code word.

2 866 FAN Yun, WANG Runsheng Vol.17 The paper is organized as follows. In the next section an image retrieval system is overviewed; in Section 3 the method for extracting features is presented; in Section 4 the codebook training algorithm and the method of representing images are presented; the similarity measure is presented in Section 5. Finally experimental results are presented. 2 Overview of the DCT-Based CBIR System An image content-based retrieval system is implemented for JPEG images. This system mainly includes three parts: off-line feature extraction, on-line retrieval and on-line training. The structure of the system is shown by Fig.1. At the off-line feature extraction phase, the color and texture features of images in the database are directly extracted from DCT coefcients. To represent images hierarchically and compactly, these features are structurally organized using the codebook, relative histogram and label-map trained by our vector quantization algorithm. These pre-computed representations are stored in the feature database. At the on-line retrieval phase, an example image is provided by the user, and the relative representation is constructed and compared with that of the images in the database. The retrieval result is returned according to similarity scores. At the on-line training phase a simple mechanism for relevance feedback is used to improve retrieval precision. Fig.1. The structure of the proposed DCT-based image retrieval system. 3 Feature Extraction from DCT Coefcients Now DCT is adopted by many compression standards such as JPEG, MPEG 1/2, H.261/H.263. For simplicity, feature extraction is only discussed for JPEG images in this paper. The standard JPEG [11] dened by the Joint Picture Expert Group is a commonly used method for the lossy image compression for photographic images. The JPEG compression is a transform-based coding algorithm. It reduces the visual information that is not important for the human eye and it de-correlates pixels and therefore eliminates redundancy. The JPEG compression standard uses the block-based discrete cosine transformation (DCT). The image is sampled using non-overlapping blocks of the size of 8 8 pixels, which are transformed by the 2D DCT. The coefcients of the transformed block are quantized and coded by a Huffman entropy encoder. Fig.2 shows the block diagram of the JPEG encoder. The JPEG decoder is composed of the inverse processing steps in a reverse order. Fig.2. Flow chart of JPEG.

3 No.4 An Research of Image Retrieval Method Using DCT Features 867 The image is scanned with a sampling window (block) from top to bottom and left to right. The pixels in this sampling window of the size of 8 8 are transformed using DCT according to the equation: DCT(u; v) = 1 4 K(u)K(v) 7 7 x=0 y=0 (2x +1)uß (2y +1)vß cos cos ρ p 2=2; for fl = 0 K(fl) = 1; for fl 6= 0 M xy u = 0;:::;7; v = 0;:::;7 It is well known that most of the high-frequency coefcients of block DCT transform is close to zero, so we can only use the low- and middle- frequency coefcients for retrieval. The chromatic signal (Y-Cr-Cb) of the image is used in JPEG. DC coefcients are extracted from three channels respectively and converted to the R-G-B signal. Then R, G, B are chosen to be color features, and the conversion equation is as follows: 8 >< >: R = 1:0000Y + 1:4020C b G = 1:0003Y 0:3341C b 0:7137C r (2) B = 0:9984Y +1:7720C b 0:0022C r The AC coefcient of block DCT transform contains rich texture information, but it is sensitive to the shift, and has block effect evidently. According to [12], the power spectrum of the coefcients has the property invariant to the shift, and has less block effect. Horizontal and vertical spectrums are dened as follows: 8 >< >: P v (f c ) = P h (f c ) = P t = n 1 i=0 n 1 i=0 DCT(f c ;i) 2 ; f c = 0;:::;(n 1) DCT(i; f c ) 2 ; f c = 0;:::;(n 1) n 1 n 1 DCT(i; j) 2 i=0 j=0 In our method, P v (f c ), P h (f c ), f c = 1;:::;4 of the luminance channel are chosen to be the texture features for retrieval. Then feature vector = fr; G; B; P v (1);:::;P v (4), P h (1);:::;P h (4)g can be constructed for every 8 8 DCT block. And they are input to the vector quantization training algorithm to obtain the codebook, histogram and label map, which are then stored in the feature database. 4 Representing an Image Using Multiple Features In this section, the codebook training algorithm is rstly presented, and then the representation of an image using multiple features is introduced. 4.1 Training Algorithm A new vector quantization algorithm based on Minimum Description Length (MDL) is adopted to train the codebook, which can decide the number of code words automatically. The algorithm can be described briefly as follows. First, the following assumptions are made: (1) All quantities are specied with a nite precision, which means that codebook A and the feature vector of the training set S can be described by K bits, K = 16Λ dim of the feature vector; (2) The samples in S are independently and identically distributed. (1) (3)

4 868 FAN Yun, WANG Runsheng Vol.17 Based on codebook A, the length of encoding S can be divided into two items. One is the length of encoding codebook A, the other is the length of encoding S using A, which can be subdivided into the following two costs. One is L(S(A)), the length for encoding the index of A to which the vectors in S have been assigned. The variable length coding according to their P probability of occurrence is m adopted, i.e., p i = n i =js i j, where n i = js i j. Therefore L(S(A)) = n i=1 i log 2 p i. The other is the length for encoding the residual error, denoted by L("(S A )). If a particular distribution p("(x)) is assumed for the error, the error can be encoded using entropy, i.e., L("(x)) = log 2 (p("(x))). It is assumed that the error obeys an independent and normal distribution with zero mean and a xed variance of ff 2 in each dimension. For convenience, ff 2 = 100, i.e., p("(x)) = Q d i=1 length of encoding the error term is then given by L("(x)) = P d j=1 where is a constant. Therefore, the cost of encoding S using A is given by: LS = mk + L(S(A)) + m i=1 p 1 2ßff e (x "2 i) 2ff 2, the " 2 (x i) ff 2 2ln2 + d log 2( p 2ßff) d log 2, x2s i L(x c i ) (4) Our goal is to minimize LS, i.e., to determine the number of code words m and c i, 1» i» m, so as to minimize LS. The complete training algorithm can be formulated as 4 steps: Step 1. Initialize the number of code words m by randomly drawing samples from S, and we choose m = 20. Step 2. Use the unsupervised k-mean algorithm to adjust the codebook vector. Step 3. Remove the most superfluous code word (in MDL sense). Step 4. If no code word is removed, and the change in adjustment is small, the algorithm converges, else go to Step 2. The algorithm is applied to every image I i in the database and the corresponding codebooks C i = fc j, 1» j» m i g are acquired. Summing up the frequenies of occurrence of all codebook vectors, the histograms H i = fh j, 1» j» m i g are obtained. Using the minimum distance principle, every 8 8 DCT block is labeled, and then the label map whose size is 1/64 of the original image is created. It is shown by experiments that if ff 2 varies in a large-scale way, the distribution of codebook vectors also varies somewhat. 4.2 Hierarchical Representation of the Image Codebook and corresponding histogram reflect the probability distribution of feature vectors, so they can be viewed as the representation of the whole image. Using the label map, we can restore the spatial distribution of feature vectors. If applying the connected component analysis to the label map, we can get a set of regions with different codebook vectors, denoted as region = freg 1 ; reg 2 ;:::;reg n g. Every region has many attributes such as area, shape feature, location, and the relation with other regions. Then the query for region, region set and sub-image can be realized in this fashion. And the feature extracted one time can be used by queries of different types and different levels simultaneously. In this paper, only the representation of a whole image and querying the whole image by examples and sub-image retrieval are considered. 5 Similarity Measure Based on Codebook To compare two images using the new representations is equivalent to measuring the similarity between their corresponding histograms based on different codebooks, which is not solved efciently now. To solve this problem, a new similarity measure is proposed. Given codebooks C i = fc ik, 1» k» m i g, C j = fc jk, 1» k» m j g and corresponding histograms H i = fh ik, 1» k» m i g, H j = fh jk, 1» k» m j g of two images I i, I j. C i and H i can be viewed as the compact representation of the image I i. This representation is encoded using the C j and H j of image I j. By the optimal

5 No.4 An Research of Image Retrieval Method Using DCT Features 869 variable length coding method, the encoding length with a similar form described in Section 4 is obtained, which is denoted as follows: L(C i ;H i jc j ;H j ) = K Λ m j m i h ik log 2 (h jk Λ)+ m i h ik L("(c ik c jk Λ)) (5) k Λ = arg min 1»l»m j kc ik c jl k (6) Error term "(c ik c jk Λ) is assumed to obey an independent and normal distribution with zero mean and a xed variance of ff 2 in each dimension, for convenience, ff 2 = 100. Ignoring constants, the length of encoding the error term is then given by: L("(c ik c jk Λ)) = kc ik c jk Λk 2 ff 2 (7) C j and H j are also encoded using themselves, the coding length is got as follows (now there is no error term): L(C j ;H j jc j ;H j ) = m j Λ K m j Then distance d(i; j) of C j, H j to C i, H i is dened as follows: h jk log 2 (h jk ) (8) d(i; j) = jl(i i ;C j ;H j ) L(I j ;C j ;H j )j (9) d(i; j) = mj h jk log 2 (h jk ) m i h ik log 2 (h jk Λ) + m i h ik L("(c ik c jk Λ)) (10) It is obvious that the rst term of (10) measures the difference between H j and H i, and the second term measures the difference between C j and C i. Similarly we can dene distance d(j; i) of C i, H i to C j, H j. d(j; i) = jl(i j ;C i ;H i ) L(I i ;C i ;H i )j (11) According to d(j; i) and d(i; j), we dene distance D ij between H i and H j as: D ij = max(d ij ;d ji ) (12) D ij is symmetric and does not obey the triangle inequality. If C j = C i and m i = m j, and D ij is rewritten in the following form then we get: D ij = mj mj h jk log 2 (h jk ) h jk log 2 hjk h ik m j m j + D ij = d ij + d ji (13) h ik log 2 (h jk ) + mj h jk log 2 (h ik ) m j h ik log 2 (h ik ) h ik log 2 hik h jk = kl(i i ;I j ) (14) From (14), it is inferred that the widely used Kullback-Leibler measure of cross entropy kl(i i ;I j ) is the lower bound of D ij, if D ij is dened as (13) and C j = C i and m i = m j. D ij can be viewed as the extension of the Kullback-Leibler measure. While the K-L measure is used to measure two probability distributions on the same base without too high feature dimensions, our similarity measure is efcient to measure the distributions on different bases with different base numbers and has no dimension restriction.

6 870 FAN Yun, WANG Runsheng Vol.17 6 Experimental Results In our system, querying by examples is implemented as follows. First, color and texture features are extracted from DCT coefcients of the example image. Codebook and histogram are acquired and compared with those of the images in the database based on the proposed similarity measure. The retrieval results are ranked according to similarity scores. To improve precision, a simple relevance feedback technique is adopted. The user labels the retrieval image as relevant or irrelevant, then codebooks and histograms of relevant images are stored, i.e., relevance = fi 1 ;I 2 ;:::;I r g. When retrieval goes on, the similarity measure is revised as follows: D(I;relevance) = Fig.3. Left-up corner is an example image, the second image of left-up is rotated by 90 degree, the width of the third is enlarged 1.5 times, the fourth is rotated by 15 degree, the third image of the second row is added with a Gaussian noise ff = 30. min D(I;I i ) (15) I i2relevance To verify the effectiveness, the new method is applied to a database with 1,005 images. These images come from the fei-cui" database. These images can be classied into 6 categories: (1) 70 car images; (2) 530 scene images; (3) 100 instrument images; (4) 100 human images; (5) 50 cartoon images; (6) 155 color texture images. The average size of these images is It costs about 15 minutes to calculate and store the features of the images of the whole database with the Pentium 466MHz and 64M memory machine. First, we test the resistance of our algorithm to noise and the invariance to rotation, shift and varying in size. Although DCT is block-based, and has no invariance, but the features extracted and the organization structure and similarity measure used remedy this drawback to some degree. Experimental results verify the invariance and resistance of the algorithm, and a result is illustrated in Fig.3. The typical histogram-based method with color histogram and color correlogram, which are introduced in [13] in detail, is implemented to compare with our method. Here we briefly review the two methods. I is an n n image, and is quantized to m colors, denoted as C 1 :::C m. Pixel p = (x; y) 2 I, and I(p) is the pixel color value. The L 1 norm is adopted to measure the distance between two different pixels. If p 1 = (x 1 ;y 1 ), then p 2 = (x 2 ;y 2 ), jp 1 p 2 j = max(jx 1 x 2 j; jy 1 y 2 j). The color histogram is dened as h ci (I) = n 2 Pr [p 2 2 I ci ] (16) p2i If d 2 [n] is predened, then correlogram is dened as fl (k) C ic j (I) = Pr p 1 2Ic i p 2 2I [p 2 2 I cj jjp 1 p 2 j = k]; i; j 2 [m]; k 2 [d] (17) In our experiment, d = 4, D = f1; 3; 5; 7g, the following distance is used to measure the similarity between two images: ji I 0 jh ci (I) h ci (I 0 )j j h = 1 + h ci (I)+h ci (I 0 (18) ) ji I 0 j fl = i2[m] i;j2[m] k2[d] jfl (k) c ic j (I) fl (k) c ic j (I 0 )j 1+fl (k) c ic j (I)+fl (k) c ic j (I 0 ) Two DCT-based retrieval methods [6;7] are also implemented to compare with the new method, which are briefly reviewed. In the rst method [6], DC coefcients are extracted from three channels (19)

7 No.4 An Research of Image Retrieval Method Using DCT Features 871 respectively and converted to the R-G-B signal, each channel is quantized to 256 bins, then three normalized histograms h i I (m) m=1;:::;256, i = 1; 2; 3, are created for each channel. The following distance is used to measure the similarity between two images I 1 and I 2 : D(I 1 ;I 2 ) = i=1 m=1 min(h i I 1 (m);h i I 2 (m)) (20) Shneier's method [7] is based on the mutual relationship between the DCT coefcients of unconnected regions in both the query image and the target image. Here, a set of 2K windows are selected, and are randomly paired, producing K pairs of windows. For each window the average of each DCT coefcient is computed, resulting in a 64-dimensional feature vector. The feature vectors corresponding to a pair of windows are compared and each pair of components is assigned with a bit (0 or 1) depending on their similarity. Thus, each pair of windows will be assigned with 64 bits. The similarity of the query image and target image is determined by the overall similarity of all the bits in all window pairs. In our experiments, K is assigned with 32. Querying by examples is considered without using the relevance feedback mechanism. 24 images are returned as the retrieval result. 30 images are chosen randomly from every image content category as the query image, and the images in the database are labeled with relevant beforehand manually, and compared with the retrieval result to acquire two indices: recall and precision dened as follows: 8 >< >: recall = precision = the number of retrieved relevant images the number of relevant images in whole database the number of retrieved relevant images the number of all retrieved images (21) Fig.4. The image on the left-top corner is the query image. (a) The result of color histogram. (b) The result of correlogram. (c) The result of our method, the rst 12 images of retrieval are shown. Table 1. The Statistic Results of Experiments Indices Methods Car (%) Human (%) Scene (%) Instrument (%) Cartoon (%) Texture (%) Average (%) Color histogram Correlogram Recall Shneier method DC histogram Our method Color histogram Correlogram Precision Shneier method DC histogram Our method Table 1 shows that the precision and recall of our method are higher than those of other four methods on average, especially in the retrieval of color texture. Fig.4, Fig.5 and Fig.6 show some experimental results. The precision and recall of our method for the image retrieval of some category is lower than the correlogram method and the precision of the car image retrieval is somewhat lower

8 872 FAN Yun, WANG Runsheng Vol.17 than that of Shneier's method [7]. It is because the information of pixel spatial distribution is not considered especially in our method. In the future, a measure of spatial distribution will be considered, and it may be helpful to improve retrieval efciency. Fig.5. The image on the left-top corner is the query image. (a) The result of color histogram. (b) The result of correlogram. (c) The result of our method, the rst 12 images of retrieval are shown. Fig.6. The image on the left-top corner is the query image. (a) The result of color histogram. (b) The result of correlogram. (c) The result of our method, the rst 12 images of retrieval are shown. The sub-image retrieval using our image content representation schema is also introduced in this paper. A rectangle is used to sketch the sub-image si of image i, and then the codebook and corresponding histogram of the sub-image are trained, denoted as C i = fc si k, 1» k» m si g and H si = fh si k, 1» k» m si g. A measure d(si;j) is dened to determine the possibility that the database image j contains the sub-image si as follows: m si d(si;j) = ff min(h si m si k ;H jk Λ) (1 ff) h si k kc si k c jk Λk 2 ff 2 (22) k Λ = arg min 1»l»m j kc si k c jl k;ff is a constant;ff is the weight;ff 2 [0; 1] in experiments; ff = 0:5;ff = 100: By (22), it is clear that if the image j contains the sub-image si, then d(si;j) is large, otherwise d(si;j) is small. And if the codebook of the image j and that of si are the same, then d(si;j) is the same as the measure of the histogram intersection. To test the efciency of the new sub-image retrieval method, similar to the experiments of querying by examples, 30 images are chosen randomly from every image content type as the query image, then a rectangle is used to sketch the sub-image, and the images in the database are labeled with relevant beforehand, and compared with the retrieval result to acquire recall and precision. The color histogram intersection method is implemented and compared with the new method, the results are illustrated in Table 2. It is clear that our new sub-image retrieval method is better than the histogram intersection method.

9 No.4 An Research of Image Retrieval Method Using DCT Features 873 Table 2. The Statistic Results of Sub-Image Retrieval Experiments Indices Methods Car(%) Human(%) Scene(%) Instrument(%) Cartoon(%) Texture (%) Average (%) Recall Histogram intersection Our method Histogram Precision intersection Our method In the future, image retrieval by regions will also be implemented using our image content representation schema, and the current method will also be improved. References [1] Gudivada V N, Raghavan V W. Content based image retrieval systems. IEEE Computer, 1995, 28(9): [2] Bach J R, Fuller C, Gupta A et al. The virage image search engine: An open framework for image management. In Proc. SPIE 2670: Storage and Retrival for Still Image and Video Databases IV, San Jose, CA, USA, Feb., 1996, pp [3] Mandal M K, Idris F, Panchanathan S. A critical evaluation of image and video indexing techniques in the compressed domain. Image and Vision Computing, 1999, 17: [4] Smith J R, Chang S F. Transform features for texture classication and discrimination in large image databases. In Proc. IEEE Int. Conf. Image Processing, Austin, Texas, 1994, (3): [5] Reeves R, Kubik K, Osberger W. Texture characterization of compressed serial images using DCT coefcients. In Proc. SPIE 3022: Storage and Retrieval for Image and Video Databases V, San Jose, California, 1997, pp [6] Furht Borko, Saksobhavivat P. A fast content-based multimedia retrieval technique using compressed data. In SPIE 3237: Conference on Multimedia Storage and Archiving Systems III, Boston, 1998, pp [7] Shneier M, Mottaleb M A. Exploiting the JPEG compression scheme for image retrieval. IEEE Transactions on Pattern Analysis and Machine Intelligence, 1996, 18(8): [8] Abdel-Malek A A, Hershey J E. Feature cueing in the discrete cosine domain. Journal of Electronic Imaging, 1994, (3): [9] Shen B, Sethi I K. Direct feature extraction from compressed images. In Proc. SPIE, 2670, 1996, pp [10] Chang R F, Kuo W J, Tsai H C. Image retrieval on uncompressed and compressed domains. In ICIP 2000, Toronto, Canada, [11] Wallace G K. The JPEG still picture compression standard. Communication ACM, 1991, 34(4): [12] Richard E, Robert S Ledley. Texture discrimination using discrete cosine transformation shift-insensitive descriptors. Pattern Recognition, 2000, 33: [13] Huang J, Kumar S R. Spatial color indexing and applications. International Journal of Computer Vision, 1999, 35(3): FAN Yun was born in He received his Ph.D. degree from the School of Electronics Engineering, National University of Defense Technology in His research interests include content-based retrieval, image understanding, and pattern recognition. WANG Runsheng was born in He is now a professor and a Ph.D. supervisor in the School of Electronics Engineering, National University of Defense Technology. His research interests include image understanding, pattern recognition, and information fusion.