Feature Selection to Relate Words and Images

Transcription

1 The Open Inforation Systes Journal, 2009, 3, Feature Selection to Relate Words and Iages Wei-Chao Lin 1 and Chih-Fong Tsai*,2 Open Access 1 Departent of Coputing, Engineering and Technology, University of Sunderland, Sunderland, SR6 0DD, UK 2 Departent of Inforation Manageent, National Central University, 300 Jhongda Rd., Jhongli, Taiwan Abstract: Iage annotation, i.e. apping words into iages, is currently a ajor research proble in iage retrieval. In particular, iages are usually segented into a nuber of regions, and then low-level iage features are extracted fro the segented regions for annotation. As the extracted iage features ay contain soe noisy features, which could degrade the recognition perforance when the nuber of keywords assigned to iages is very large, iage feature selection needs to be considered. In this paper, a Pixel Density filter (PDfilter) and Inforation Gain (IG) are used as the feature selection techniques. By using Corel as the dataset, 10, 50, 100, 150 and 190 keywords annotation are setup for coparisons. The experiental result shows that PDfilter and IG can increase the precision of iage annotation by colour or texture features. However, they do not enhance the annotation perforance by the cobined colour and texture features. 1. INTRODUCTION Multiedia databases have becoe very popular for various applications, such as digital libraries, edical iages, and news photos. Content-based Iage Retrieval (CBIR) systes provide an effective and efficient retrieval technique to access ultiedia databases for users to query relevant iages through soe perceptual (or low-level) features, such as colour, texture, shape and spatial object layouts. This technique requires coplete indexing facilities and adequate data structures, to filter out iages unrelated to a query, then visualise the final retrieval results in an accurate way [1]. The ost successful applications are based on supervised achine-learning classifiers to atch iages with relevant keywords for general keyword-based queries. Therefore, noisy features are the priary proble for iage indexing that cause current systes still not to be very accurate with large nubers of vocabularies. This paper focuses on noisy feature filtering, to investigate one possible reason why current systes are unlikely to scale to larger vocabularies. A Pixel Density filter (PDfilter) and Inforation Gain (IG) [2, 3] are suggested to solve this proble. PDfilter is used for iage feature representation, to ake the iage feature signature allow siilar discriinating power aong iages to a detailed pixel by pixel coparison. IG focuses on iage feature selection that holds ost inforation about the analysis category, in order to identify the ost useful data in the syste training stage. Finally, the siilarity easure tool is setup by a k-nearest Neighbour (k-nn) classifier that provides a supervised achine-learning technique to allow the syste to assign relevant keywords to the training and testing iages. A short review of CBIR and its challenges are given in Section 2. In Section 3, PDfilter and IG are described to solve the iage feature selection proble. Section 4 presents *Address correspondence to this author at the Departent of Inforation Manageent, National Central University, 300 Jhongda Rd., Jhongli, Taiwan; Tel: ; Fax: ; E-ail: cftsai@gt.ncu.edu.tw the experients and copares the perforance of (1) k-nn, (2) IG (3) Colour histogra (4) PDfilter, and (5) PDfilter+IG for the iage classification task over 190 vocabularies. Conclusion and future work are provided in Section RELATED WORK Content-based Iage Retrieval (CBIR) was proposed in the early 1990s and has been an active research area for over ten years. It is a technique based on the extracting iage contents to retrieve relevant iages, such as colour, texture, shape, etc. In particular, CBIR systes extract and index iages low-level features autoatically, aiing to provide the capability to support visual queries, an intuitive query approach, and autoatic description content features of iages [4]. To extract iage contents, iage segentation is one ajor task. The result of iage segentation provides ore detailed inforation that describes an analysed iage in ters of different areas or regions before low-level feature extraction [5]. In literature, coplex segentation schees have been applied in current iage retrieval systes. To segent an iage, the optial shape and size of subsegents can be deterined by different resolutions, which is an active area of research. Then low-level features can be extracted autoatically fro each of the sub-segents, including colour, texture, shape content, spatial object layout recognition, and so on. Based on a cobination of different low-level features to identify high-level concepts, iage seantics can be found autoatically using suitable rules and previous knowledge [6]. Additionally, iages ay be given descriptive etadata off-line anually, typically catalogue inforation, which is about the object s creator, event activity, tie of creation, and so on. The retrieval stage is followed by the iage indexing stage. Various retrieval environents have been developed that enable users to query iage databases through different kinds of retrieval strategy including Query by Keyword (QBK), Query by Exaple (QBE), and Query by Feature / Bentha Open

2 10 The Open Inforation Systes Journal, 2009, Volue 3 Lin and Tsai extract Iage segent P1 x Predoinant bucket P2 Pixels quantify Fig. (1). Exaple of PDfilter operation 1. (QBF) [1]. Visualisation tools inspect the initial set of iages retrieved in response to the query, and iprove the effectiveness of the syste by filtering out the iages which are not suitable for the query [7]. Relevance feedback is the process of analysing responses to retrieved inforation that perits query refineent to adjust retrieval repeatedly in order to atch the user's original needs. User judgeents of search results are fed back into the syste to refine the query so that new results are ore fitting. The seantic gap proble is the ain challenge for CBIR systes. It occurs in the translation of low-level features into high-level concepts. This proble causes soe keywords never to be assigned to their correct iages, and thus users are very likely un-satisfied by the retrieval results of current systes [8]. A nuber of investigations have worked towards deciphering this proble. Tsai et al. [6] have shown the possibility of using achine learning technology to autoatically assign 150 controlled keywords to unknown and unseen iages. Although the process is far fro perfectly accurate it appears sufficiently effective for practical use, if cobined with state-of-the-art browsing technology. However, ideally powerful iage searching environents would like to operate with uch larger vocabularies than at present, in order to enable current CBIR systes to be incorporated into coercial search engines, like Google and Yahoo, for general users to approach. 3. METHODOLOGIES 3.1. Pixel Density Filter (PDfilter) Iage feature values are derived fro individual pixels initially. However, iage s representative features cannot siply use every single characteristic, since every iage is coposed with very large nubers of individual pixels that will introduce ore noise than considering an overall value for each region. In ost related studies, such as Barnard et 1 In order to show clearly the distribution of the pixel values, this exaple uses the tiling segentation. However, the focus of this paper is to segent an iage into regions with colour siilarity. al. [8] and Tsai [6], the representative feature coes fro the average of all pixel values within a region or tiling area. This courses iage s representative features can be too siilar to each other. For exaple, regions coposed of a ixture of red and green will be both represented in the sae way, as brown colour. Particularly, the colour histogra [9] is the ost extracted feature to solve this proble. It provides ore discriination power when ore colour buckets are coprised [10]. Nevertheless, the nuber of buckets is usually liited by colour look-up on huan vision. Therefore, there are only up to 256 colour buckets in current approach, and the ajority is less than 128 buckets. PDfilter is used for pixel value selection which can ake representative features ore siilar to their values in the original iage. In general, it ay increase colour buckets. It supplies a better clustering odel to perfor with the individual feature space in each region. Every single pixel can be addressed into a bucket area with forula (1), where D eans the diensions out of N diensions; x D is the value of each diension and x D is the axiu value of each diension in the whole iage collection. Each pixel s feature value is quantified into a bucket A(p) of a coordinate figure in S divisions (we set S = 10 to have 1,000 buckets for analyses). Finally, the representative feature values coe fro the average of the pixel values in the predoinant bucket. An exaple with hue (H), saturation (S) and value (V) colour space [11] is shown in Fig. (1). The representative feature to the analysed region after PDfilter selection will be the sybol at point P2 (0.736, 0.25, 0.249), with black, in bucket 227, in contrast to the average of all pixel values at location P1 (0.389, 0.441, 0.418), with green. A D S N x D 1 ( p) = ( S ) + 1 (1) x D= 1 D 3.2. Inforation Gain (IG) Inforation Gain (IG) [3] is applied for training data selection, which allows each category to be represented by

3 Feature Selection to Relate Words and Iages The Open Inforation Systes Journal, 2009, Volue 3 11 its ost iportant feature vectors with noise and uncertainty reduction. In addition, it is one of the ost used feature selection ethod in text categorisation. Based on forula (2), IG has weighted su of gain value to predict the presence or absence of a ter in the analysed docuent. G( t) = + t) i = i = 1 i = 1 1 Ci) log Ci) Ci t)log Ci t) + t) Ci t) log Ci t) (2) In our approach, k-eans clustering is used before perforing IG, in order to sort original training feature vectors of the analysis category into a nuber of k clusters. We set k = 10 in the experients since the feature vectors are too siilar between each other. That is, without using PDfilter soe docuents cannot be grouped into ore than 10 clusters. In forula (2), let be the total nuber of regions in an iage and { Ci} i= 1 is the set of iages within the analysis category, then the gain value G(t) to every individual cluster can be obtained. In addition, IG is intuitively appealing for uncertainty data reduction. There are no algoriths for the optial splitting value identified. In practice, the threshold is based on heuristics to find a near-optial value [12]. Only the docuents with an above threshold gain value are retained as new training data for future classification. Finally, the new training data to our experients are taken fro the single cluster with the highest gain value. 4. EXPERIMENTS 4.1. Task We focus on feature value selection by PDfilter and IG, in order to recognise useful training data for the syste. Our research questions are (1) can the PDfilter ake a colour and/or texture feature signature to allow siilar discriinating power aong iages as a detailed pixel by pixel coparison? (2) does IG provide a suitable odel for colour and/or texture feature vector selection? 4.2. Experiental Set Up For experients, five different approaches are evaluated, which are (1) feature extractor, (2) PDfilter, (3) IG, (4) colour histogra and (5) siilarity easure, as shown in Fig. (2). As a result, five independent systes (shown in Table 1) were ipleented, with different cobinations of feature value calculation and training region selection. Other experiental systes are divided into three experients by different iage feature representations: (1) colour (2) texture and (3) colour+texture. For the colour feature, particularly, the colour histogra syste is used for the coparisons, which is based on 125 colour buckets [13]. Experiental Training stage Training data selection Tra. I Baseline syste Color hist. Tra. II Colour histogra IG Tra. III IG syste Training iages PDfilter Tra. IV PDfilter syste Feature extractor IG Tra. V PDfilter+IG syste Siilarity easure Iage collection Testing iages Tes. I For Baseline & IG syste Color hist. Tra. II For Colour histogra syste Keyword assign Fig. (2). Experiental fraework. PDfilter Tes. III For PDfilter & PDfilter+IG syste Training data selection Experiental Testing stage

4 12 The Open Inforation Systes Journal, 2009, Volue 3 Lin and Tsai Table 1. Experiental Systes Feature Val. Calc. Training Reg. Sel. I II III A B Baseline IG Colour histogra PDfilter PDfilter+IG I: average of all pixels value within the region; II: average ost colour in the region; III: average ost predoinant values in the region. A: the region holding the central pixel (64, 64); B: the region selection by IG. The iage collection coes fro the Corel Stock Photo Libraries 2, which is the ost coonly used dataset for CBIR experients. Corel use 100 iages to describe a topic (category) and soe of the descriptions are spread over ore then one category. In this paper, we only consider one category to such descriptions, as we use one keyword to represent a single category. At the sae tie, the WordNet 3 lexical reference syste is used to deterine whether a keyword is concrete or abstract. The experients only use concrete concepts and up to 190 keywords. Each individual keyword category provides 20 and 40 iages for the training and test data respectively. Additionally, experients were perfored over 10, 50, 100, 150 and 190 keywords. The 10 to 150 keyword experients consist of ten sub-experients using rando subsets of keywords selected fro the 190 keyword set. Feature extractor is used in autoatic low-level feature extraction. There are three steps in this coponent. Firstly, iages are resized into 128x128 pixels. Then each iage is segented by the Noralized Cuts (Ncut) algorith [14] into five regions based on their colour siilarly. Finally, colour and texture features are extracted fro these five subiages (regions) respectively. The experients described here ap iage features into 3 diensions in HSV colour space and 16 diensions in four levels of Daubechies wavelet texture feature decoposition [15]. Different coponents were ipleented in every independent syste to have different cobinations of the training and testing data. However, the k-nn classifier supplies a siilarity easure between the training and testing data. According to Jain et al. [16], we set k = 1 (1NN), which can provide reasonable classification results for ost pattern recognition probles. It assigns relevant keywords into new instances, in order to provide an effective query ethod through keyword-based retrieval Evaluation and Discussion F-easure, as weighted haronic ean of recall and precision, is used for syste centred evaluation in this paper. Fig. (3) shows the experiental results using various feature selection techniques for the colour feature. The results are iproved after IG selection. After PDfilter, both 2 Software review at: ht 3 Available at: systes are better then the Baseline and Colour histogra. In particular, for PDfilter+IG, the F-easure increases to 6.33% fro 4.13% provided by the baseline over 190 categories. That is, the PDfilter and IG increase the precision of assigning keywords to their related iages for colour features. Fig. (3). Experiental results for the colour feature approach, over 10, 50, 100, 150 and 190 categories. Table 2 shows that only IG iproves the annotation perforance with texture features (very slightly). Additionally, the results show that PDfilter and IG do not enhance the annotation perforance by the cobined colour and texture features. Table 2. Experiental Results of Texture and Colour+Texture Approach, in 190 Categories Only F-easure Baseline IG PDfilter PDfilter+IG texture 4.70% 4.83% 3.76% 3.65% colour+texture 7.10% 6.88% 5.65% 5.33% 5. CONCLUSION AND FUTURE WORK PDfilter and IG are two proising approaches to iage annotation using colour features. For the texture features, every pixel s feature value is coputed by the relationship with its neighbouring pixels, a relationship obscured by PDfilter and IG. This is the ain reason why the systes cannot iprove the perforance by texture and colour+texture features. In future we will expand the investigation by directly extracting texture features, attepting to apply IG between each pixel and its neighbours, to iprove precision. We will also work with larger vocabularies, including abstract keywords. In addition, other iage data sets, such as the IAPR TC-12 Benchark [17], the TRECVid [18], and the University of Washington iage collection 4, will be used for syste evaluation. 4 Available at:

5 Feature Selection to Relate Words and Iages The Open Inforation Systes Journal, 2009, Volue 3 13 REFERENCES [1] A. del Bibo, Visual Inforation Retrieval. San Francisco: Morgan Kaufann, [2] L. Breian, J. H. Friedan, R. A. Olshen, and C. J. Stone, Classification and Regression Trees. California: Wadsworth & Brooks, [3] Y. Yang and J. O. Pedersen, "A Coparative Study on Feature Selection in Text Categorization," in 14th International Conference on Machine Learning (ICML '97), 1997, pp [4] A. W. M. Seulders, M. Worring, S. Santini, A. Gupta, and R. Jain, "Content-based iage retrieval at the end of the early years," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, pp , Dec [5] F. Idris and S. Panchanathan, "Review of Iage and Video Indexing Techniques," Visual Counication and Iage Representation, vol. 8, pp , Jun [6] C.-F. Tsai, K. McGarry, and J. Tait, "CLAIRE: A odular support vector iage indexing and classification syste," ACM Transactions on Inforation Systes (TOIS), vol. 24, pp , Jul [7] J. K. Wu, M. S. Kankanhalli, J. Li, and D. Hong, Perspectives on Content-based Multiedia Systes. London, Boston, Dordrecht: Kluwer Acadeic Publishers, [8] K. Barnard, P. Duygulu, D. Forsyth, N. de Freitas, D. M. Blei, and M. I. Jordan, "Matching words and pictures," Journal of Machine Learning Research, vol. 3, pp , Feb [9] M. J. Swain and D. H. Ballard, "Color Indexing," International Journal of Coputer Vision, vol. 7, pp , Nov [10] F. Long, H. Zhang, and D. D. Feng, "Fundaentals of Content- Based Iage Retrieval," in In Multiedia Inforation Retrieval and Manageent: Technological Fundaentals and Applications, D. D. Feng, W. C. Siu, and H. Zhang, Eds. Berlin, Heidelberg, New York: Springer-Verlag, [11] F. van der Heijden, Iage Based Measureent Systes: Object Recognition and Paraeter Estiation. Chichester: John Wiley & Sons, [12] C. D. Manning and H. Schütze, Foundations of Statistical Natural Language Processing. London, Cabridge: MIT, [13] T. Lai, CHROMA: A Photographic Iage Retrieval Syste, UK: University of Sunderland, [14] J. Shi and J. Malik, "Noralized cuts and iage segentation," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 22, pp , [15] I. Daubechies, Ten Lectures on Wavelets. Philadelphia: Society for Industrial and Applied Matheatics, [16] A. K. Jain, R. P. W. Duin, and J. Mao, "Statistical Pattern Recognition: A Review," IEEE Transitions on Pattern Analysis and Machine Intelligence, vol. 22, pp. 4-37, Jan [17] M. Grubinger, P. Clough, H. Müller, and T. Deselears, "The IAPR TC-12 Benchark: A New Evaluation Resource for Visual Inforation Syste," in International Workshop OntoIage '06 Language Resources for Content-Based Iage Retrieval, Genoa, Italy, 2006, pp [18] A. F. Seaton, W. Kraaij, and P. Over, "TREC Video Retrieval Evaluation: A Case Study and Status Report," in RIAO 2004: Coupling Approaches, Coupling Media and Coupling Languages for Inforation Retrieval, Avignon, France, 2004, pp Received: Noveber 01, 2008 Revised: March 01, 2009 Accepted: March 13, 2009 Lin and Tsai; Licensee Bentha Open. This is an open access article licensed under the ters of the Creative Coons Attribution Non-Coercial License ( which perits unrestricted, non-coercial use, distribution and reproduction in any ediu, provided the work is properly cited.