Dominant colour extraction in DCT domain

Transcription

1 Image and Vision Computing 24 (2006) Dominant colour extraction in DCT domain Jianmin Jiang a,b, *, Ying Weng b, PengJie Li c a Southwest University, China b University of Bradford, UK c Institute of Acoustics, Chinese Academy of Sciences Received 23 March 2004; received in revised form 11 January 2006; accepted 5 April 2006 Abstract As most of MPEG and JPEG compression standards are DCT-based, we propose a simple, low-cost and fast algorithm to extract dominant colour features directly in DCT domain without involving full decompression to access the pixel data. As dominant colour is one of the colour features used in constructing MPEG-7 dominant colour descriptors, the proposed algorithm would provide useful technique for those already compressed videos and images, where MPEG-7 dominant colour descriptors are needed. While the proposed algorithm presents advantages in terms of computing efficiency, i.e. eliminating the need of IDCT for compressed videos and images, extensive experiments also support that the proposed algorithm achieves competitive performances in extracting dominant colour features when compared with the pixel domain extraction described in MPEG-7. Crown Copyright q 2006 Published by Elsevier B.V. All rights reserved. Keywords: Dominant colour features; MPEG-7; Feature extraction in compressed domain 1. Introduction Along with the digital revolution and the success of international standardization of image compression techniques, rapid increase in volume of images and videos are seen almost everywhere including media publications, WEBbased publications, e-something, etc. The rich source of visual information made available to our daily life brought the need to develop algorithms and techniques to browse, search, retrieve and manage the content of those compressed digital images and videos. To this end, research on contentbased image indexing and retrieval is receiving tremendous interests within both the academic community and the industrial sectors [1,2]. Since most of such research is carried out in pixel domain, in which input images are assumed to be in their original raw format, full decompression would have to be performed prior to the content description or content-based management developed in pixel domain. In practice, it maybe arguable that this sort of overhead computing does not really present any problem since full decompression is fast, and its algorithm * Corresponding author. address: j.jiang1@bradford.ac.uk (J. Jiang). implementation is efficient. Thus, it could be accepted that the cost of IDCT, or indeed the full decompression, will not be a bottleneck for such compressed image management. When millions of such compressed images need to be accessed, however, the computing cost for full decompression could become an issue, which may deserve our attention for better solutions. As a result, a new wave of research within the academics is emerging for direct content access and feature extraction in compressed domain rather than in pixel domain [10 13]. Following the successful development of MPEG-1, MPEG-2 and MPEG-4 standards, MPEG has now developed a new standard, MPEG-7 [4 8], which is formally known as multimedia content description interface to describe multimedia content. While the previous standards focus on coding and representation of audio visual content, MPEG-7 focuses on content description with various modalities including image, video, audio, speech, graphics, and their combinations. To follow the research efforts in contentbased image indexing and retrieval, MPEG-7 visual standard for content description has developed a range of descriptors corresponding to common features including colour, texture, shape, and motion [5,7,8]. Among all the features, colour descriptors are mainly composed of scalable colour /$ - see front matter Crown Copyright q 2006 Published by Elsevier B.V. All rights reserved. doi: /j.imavis

2 1270 J. Jiang et al. / Image and Vision Computing 24 (2006) descriptor (SCD), dominant colour descriptor (DCD) [5,9], colour layout descriptor, and group-of-pictures colour descriptor. The DCD is mainly developed for fast and efficient image browsing and retrieval via the description of local spatial colour distribution at global level. For a given region of images, the colours are clustered into a small number of representative colours to produce a more compact representation of the colour distribution inside the image. In comparison with those histogram-based approaches, the DCD only contains elements of representative colours, their percentages in a region, spatial coherency of the colour and the colour variance. To provide efficient description for those already compressed images and videos, we propose in this paper a simple dominant colour extraction algorithm directly in DCT domain to formulate the MPEG-7 DCD. In comparison with the existing stateof-the-art, i.e. MPEG-7 and other colour-based description of image and video content published in the literature, our contribution and its novelty can be highlighted as: while all existing work extracts dominant colour features in pixel domain, our proposed algorithm can extract dominant colour features directly in DCT domain. When millions of compressed images and video frames need to be accessed with their dominant colour descriptions, the proposed technique will directly work with their compressed format without full decompression. Yet the existing techniques including MPEG-7 would need full decompression to get their pixel data before their dominant colour feature can be extracted. The remaining part of the paper is further divided into two further sections. While Section 2 provides the detailed description of the proposed algorithm design, Section 3 covers the assessment and evaluation of the proposed algorithm via extensive experiments. Finally, some concluding remarks are also given in this section. 2. Dominant colour extraction in DCT domain MPEG-7 dominant colour extraction mainly comes from the work by Deng and Manjunath [9]. This descriptor highlights the salient colour in images or regions in pixel domain via colour clustering process. Hence, the descriptor is defined to include the representative colours, their percentages in the region, spatial coherency of dominant colours, and colour variances for each dominant colour. Its full definition is given below F Z ffc i ;p i ;v i g;s;g ði Z 1;2.;NÞ (1) where c i is the ith dominant colour; p i represents its percentage value; v i is its colour variance; and s represents the spatial coherency. It is reported that this descriptor achieves competitive performances in comparison with those high-dimensional histogram methods [7], while being used in image indexing within the 3D colour space. Although, significant advantages have been illustrated with this descriptor in terms of colour representation and low dimension, it cannot be used to process compressed images without decompression since it is limited to pixel domain, which requires sequence of operations including entropy decoding and IDCT. On the other hand, detailed analysis of DCT coefficients reveals that there exist many statistic features in DCT coefficients, reflecting the same content and the same nature of the image as that of pixels. While the DC coefficient represents the average intensity value among a block of pixels, the AC coefficients reflect the variance of colour changes in pixels within the same block. When arranged in a zig-zag order, the AC coefficients are also capable of indicating frequency components in an ascending order. Following this relationship analysis between the DCT coefficients and the image content in pixels, it becomes clear that each block of 8!8 pixels as designed in JPEGs [3] and MPEGs can be classified into a dominant colour block or a non-dominant colour block. Since the number of non-zero AC coefficients directly relates to the variance of colour components among the pixels, it can be used as a parameter to control the block classification of either a dominant colour block or a nondominant colour block. To improve the accuracy, a further parameter can be added to indicate the position of the last non-zero AC coefficient along the zig-zag order, since it represents the highest frequency component available inside the block. Therefore, all the blocks can be classified into dominant colour blocks, non-dominant colour blocks or uncertain blocks via the following mechanism in DCT domain ( block i Z dominant_colour; if N nz!t 1 &P!T 2 (2) uncertain; otherwise where block i stands for the ith block inside the image along the raft-scanning order, N nz is the number of non-zero AC coefficients inside block i, P represents the position of the last non-zero AC coefficient along the zig-zag order inside the block i, and finally T 1 and T 2 stand for the first two thresholds we introduced to identify the dominant-colour blocks. For those uncertain blocks, there exist two possibilities. One is that the block concerned is a non-dominant colour block, and the other is that the block contains dominant colour in smaller regions. To identify those definitely nondominant colour blocks, controlling the number of AC coefficients remain to be sufficient. One extreme case is that all 63 AC coefficients are non-zero. Therefore, the following condition can be set up to determine those definitely nondominant colour blocks: ( block i Z non_dominant_colour if N nzot 3 (3) uncertain; otherwise For the latter that dominant colour may exist in smaller blocks, we can use our previous work [13] to decompose the 8!8 block into four sub-blocks of 4!4 coefficients in DCT domain. Specifically, given a block of 8!8 DCT coefficients represented by [C], the matrix of four sub-blocks of 4!4

3 J. Jiang et al. / Image and Vision Computing 24 (2006) Table 1 Parameter matrix [A] K K K K K K K K K K K K K K DCT coefficients can be derived from the following equation [13] " # C11 C12 Z 2! ½AŠ! ½CŠ! ½AŠ T (4) C21 C22 where Cij is a matrix of 4!4 DCT coefficients representing the decomposed sub-block, and [A] is an 8!8 parameter matrix, whose values are given in Table 1. As the number of AC coefficients is now reduced to 16 in those sub-blocks, we propose to use the square sum of AC coefficients to determine those dominant colour sub-blocks instead of (2). This is detailed below ( sub_block i Z dominant_colour; if S!T 4 (5) non_dominant; otherwise where SZ P15 ACkis 2 the square sum of all 4!4 AC coefficients kz1 inside the sub_block, and T 4 is the fourth threshold introduced for the block classification. As the block decomposition in DCT domain can be repeated until one single coefficient is reached [13], the block classification can be further tuned to identify the dominant colour at the levels of 2!2 blocks, and each individual coefficient, which would be the same as that of MPEG-7 in pixel domain. In practice, however, it is found that one level decomposition gives us sufficient accuracy in dominant colour extraction. After all the blocks are classified, those non-dominant colour blocks or sub_blocks are discarded, and the mean value of the pixels inside each dominant colour block is calculated in DCT domain to construct the MPEG-7 descriptor as well as the histogram for dominant colour analysis. According to the DCT equations [8], the mean value of pixels inside each block can be obtained in DCT domain via the following m Z 1 Cð0;0Þ (6) N where NZ8 for dominant colour blocks and NZ4 for dominant sub-blocks. In this way, each dominant colour block or sub-block is essentially represented by one single value, which can also be used to construct a so called dominant colour image. A few such images are illustrated in Fig. 3. The above design requires four thresholds, which can be determined empirically in principle in accordance with the relationship between DCT coefficients and the image content in pixels. For example, it can be inferred that blocks with 30 nonzero AC coefficients should support that there exist no dominant colour in an 8!8 block. This is because when more than 30 different variance values are present in this pixel block, it certainly indicates that no colour component dominates among all pixels inside this block. As this threshold increases, the number of blocks identified as non-dominant colour will be reduced, yet more accuracy is achieved. If the threshold decreases, however, the accuracy of extraction may suffer, although more non-dominant colour blocks will be identified. As a result, T 3 is set up to be 30 in the proposed algorithm. With thresholds T 1 and T 2, the principle is similar. As T 1 is used to monitor the number of non-zero AC coefficients and this number indicates the variety of intensity values inside the block, we set T 1 to be 12, which is around one fifth of the total number of non-zero AC coefficients. As T 2 OT 1, and the position of the last non-zero AC coefficient indicates the maximum frequency component inside the block, we set T 2 to be at 20. Such settings for T 1 wt 3 are permanent for the proposed algorithm, which are applied to all images tested. Extensive experimental results suggest that the settings are correct and no adjustment is needed. T 4 is an important parameter in this algorithm, which provides more delicate information about the dominant colour in smaller regions. Since those smaller regions proved to be more difficult to extract dominant colour information, an adaptive approach needs to be developed to determine the specific value for T 4. Given M blocks of AC coefficients for a compressed image or a video frame, an AC absolute sum, A s i, can be calculated for the ith block by: A s i Z X63 jz1 jac j j; ci2½0;mk1š (7) Via arranging all the AC absolute sums into an ascending order with respect to their frequencies, a curve can be produced. One example for Lena.jpg is shown in Fig. 1. From this curve, the maximum curvature can be taken as the value of T 4, which remains adaptive to the content feature of each individual image. As the curve is produced out of discrete samples, a mean filtering is applied to smooth the curve and help us to locate the maximum curvature. To reduce the

4 1272 J. Jiang et al. / Image and Vision Computing 24 (2006) Fig. 1. Illustration of square sum curves. complexity in the algorithm design, an alternative way is to set T 4 to be 10% of the maximum AC absolute sum, which is roughly around the maximum curvature point. Our experiments do not reveal any significant difference between these two approaches. As a result, the proposed dominant colour extraction algorithm can be summarised in Fig Experimental results and conclusions To evaluate the proposed algorithm, we carried out extensive experiments to extract dominant colour features in comparison with the existing MPEG-7 techniques in pixel domain. As given in (1), the MPEG-7 colour descriptor contains four elements. Since s is optional according to Fig. 2. Summary of the proposed algorithm. MPEG-7 [5 8], however, our experiments are focused on the comparison of three other elements, i.e. c i p i and v i. Considering the fact that each image may have a large number of dominant colour elements, we arrange the detailed comparison in three phases. In the first phase, we compare the first four major dominant colours between the proposed algorithm in DCT domain and the original MPEG-7 descriptor in pixel domain, which are given in Tables 2 4. Table 2 lists all the elements of the MPEG-7 DCD (dominant colour descriptor) for the first four dominant colour for Lena.jpg, and Table 3 lists the same details for Pepper.jpg. In Table 4, we choose the colour image Baboon.jpg as the sample to illustrate the detailed comparison of all the three components between the DCDs extracted by the proposed algorithm and by the original MPEG-7. Fig. 3 illustrates the experimental results in the second phase of comparison, where the first 16 dominant colours for each image are illustrated in bar-charts. To cover all the dominant colour elements extracted by the proposed algorithm and overcome the drawback that tables and barcharts have limited space to include all dominant colour elements, we further arrange all those classified blocks displayed as an image for visual inspections, in which each pixel data are represented by m as specified in (6). These are illustrated in Fig. 4, where white blocks are those nondominant colour blocks and thus discarded. From all the phases of comparisons, the following observations can be made about the proposed algorithm: The dominant colour extracted in DCT domain effectively represents the salient colour in the original image. The dominant colour descriptor extracted by the proposed algorithm in DCT domain remains very close to that of MPEG-7 in pixel domain. The reconstructed images via dominant colour (m) reveals that the content of original images is preserved very well. The proposed algorithm essentially extracts dominant colour in terms of blocks or sub-blocks rather than

5 J. Jiang et al. / Image and Vision Computing 24 (2006) Table 2 MPEG-7 dominant colour descriptor for Lena.jpg Lena pixel domain DCT domain Percentage in pixel Percentage in DCT Variance in pixel 1st nd rd th Variance in DCT Table 3 MPEG-7 dominant colour descriptor for Peppers.jpg Peppers pixel domain DCT domain Percentage in pixel Percentage in DCT Variance in pixel 1st nd rd th Variance in DCT pixels due to the fact that dominant colour must have a reasonable critical mass to be regarded as dominant. Under this context, the proposed algorithm can be implemented in two modes. The first mode is to extract dominant colour from blocks of 8!8 pixels only, and the second is to extract dominant colour with blocks of 8!8 pixels as well as sub-blocks of 4!4 pixels. In the first mode, the computing cost of the proposed algorithm is only three comparisons given by (2) and (3). In the second mode, those uncertain blocks will incur: (i) computing cost for deriving four sub-blocks (4!4 DCT coefficients), which is specified by (4). From Eq. (4), it can be seen that standard implementation will require 64!8!2Z1024 multiplication and 56!8!2Z896 additions; (ii) computing cost for examining the four sub-blocks specified by (5), where deriving S requires 15 multiplication and 14 additions. Therefore, the total computing cost is 1039 multiplication and 910 additions. In contrast with the existing pixel-domain dominant colour extraction, the IDCT alone would require 4096 multiplication and 4032 additions [14], yet such computing cost is needed for every block inside the image. For the proposed algorithm, only a small number of blocks need subblock examination. As a matter of fact, for all the test images reported in the paper, dominant colour is extracted only at the block level and no sub-block division is incurred. n addition, we applied the proposed algorithm to a database of around 1000 JPEG compressed images to perform a small scale of content-based image retrieval via the dominant colour descriptors in both pixel domain as proposed by MPEG-7 and DCT domain as proposed in this paper. The experiments reveal that almost exactly the same retrieval results were observed in terms of dominant colour Table 4 MPEG-7 dominant colour descriptor pixel domain DCT domain Percentage in pixel Percentage in DCT Variance in pixel Y component of colour Baboon.jpg 1st nd rd th Cr component of colour Baboon.jpg 1st nd rd th Cb component of colour Baboon.jpg 1st nd rd th Variance in DCT

6 1274 J. Jiang et al. / Image and Vision Computing 24 (2006) Fig. 3. Comparison of dominant colour histograms with 16 bins between the proposed and MPEG-7 in pixel domain.

7 J. Jiang et al. / Image and Vision Computing 24 (2006) Fig. 4. Illustration of reconstructed image samples via extracted dominant colour. retrieval. Fig. 5 illustrates such retrieved samples, where the query image is presented at the top left corner. In addition, the mean value of dominant colour blocks extracted in the proposed algorithm can also be exploited to construct a so-called dominant colour histogram. According to the dominant colour extraction principle, this histogram should reflect the major structure of those general histograms constructed from the original pixel data. In

8 1276 J. Jiang et al. / Image and Vision Computing 24 (2006) Fig. 5. Illustration of dominant colour-based image retrieval. this way, the proposed technique can be used to predict the image histogram directly in DCT domain, which could lead to histogram extraction rather than just dominant colour extraction. Fig. 6 illustrates such detailed comparisons between the extracted dominant colour histogram in DCT domain and the original image histogram based on colour intensity values in pixel domain. It is shown that the dominant colour histograms constructed

9 J. Jiang et al. / Image and Vision Computing 24 (2006) videos to formulate content descriptors in the process of their compression, the proposed algorithm will be very useful in extracting MPEG-7 dominant colour descriptors directly from those already compressed videos and images. As a result of the successful JPEG and MPEG standardization activities, almost all digital videos are now stored in compressed format. Therefore, feature extraction or image processing in DCT domain will provide certain advantages in reducing computing cost and direct accessing the content of compressed image and videos. Finally, the authors wish to acknowledge the financial support from the Resource: The Council for Museums, Archives and Libraries in the UK under the research grant LIC/RE/082, and the Natural Science Foundation in China. References Fig. 6. Visual comparison between the dominant colour histogram extracted by the proposed algorithm and the general colour histogram extracted in pixel domain. from the proposed algorithm do reflect the major structure of the image histogram in pixel domain. This will provide good potential for further research in extracting image histograms directly from DCT domain, which would be useful for other applications in processing compressed images. In this paper, an algorithm for dominant colour extraction in DCT domain is proposed, which provides effective and fast processing technique for content description of compressed videos and images. While MPEG-7 works on raw images and [1] R. Brunelli, O. Mich, M. Modena, A survey on the automatic indexing of video data, J. Visual Commun. Image Represent. (1999) [2] B. Ko, J. Peng, H. Byun, Region-based image retrieval using probabilistic feature relevance learning, Pattern Anal. Appl. 4 (2 3) (2001) [3] G.K. Wallace, The JPEG still picture compression standard, Commun. ACM 34 (4) (1991) [4] Overview of MPEG-7 standard (version 5.0), ISO/IEC JTC1/SC29/WG11 N4031, March [5] T. Sikora, The MPEG-7 visual standard for content description an overview, IEEE Trans. Circuits Syst. Video Technol. 11 (6) (2001) [6] B.S. Manjunath, J. Ohm, V.V. Vasudevan, A. Yamada, Color and texture descriptors, IEEE Trans. Circuits Syst. Video Technol. 11 (6) (2001) [7] M. Bober, MPEG-7 visual shape descriptors, IEEE Trans. Circuits Syst. Video Technol. 11 (6) (2001) [8] S. Jeannin, A. Divakaran, MPEG-7 visual motion descriptors, IEEE Trans. Circuits Syst. Video Technol. 11 (6) (2001) [9] Y. Deng, B.S. Manjunath, C. Kenney, An efficient color representation for image retrieval, IEEE Trans. Image Process. 10 (1) (2001) [10] G. Feng, J. Jiang, JPEG compressed image retrieval via statistical features, Pattern Recogn, in press. [11] R. Reeves, K. Kubik, W. Osberger, Texture characterization of compressed aerial images using DCT coefficients, Proceedings of SPIE: Storage and Retrieval for Image and Video Databases V, vol. 3022, February 1997, pp [12] M. Liu, J. Jiang, C. Hou, Combination of image indexing and compression, Proceedings of ICASSP 02, [13] J. Jiang, G. Feng, The spatial relationship of DCT coefficients between a block and its sub-blocks, IEEE Trans. Signal Proc. 50 (5) (2002) [14] J. Jiang, Y. Weng, Video extraction for fast content access to mpeg compressed videos, IEEE Trans. Circuits, Syst. Video Technol. 14 (5) (2004)