UJM at ImageCLEFwiki 2008

Transcription

1 UJM at ImageCLEFwiki 2008 Christophe Moulin, Cecile Barat, Mathias Géry, Christophe Ducottet, Christine Largeron To cite this version: Christophe Moulin, Cecile Barat, Mathias Géry, Christophe Ducottet, Christine Largeron. UJM at ImageCLEFwiki Cross Language Evaluation Forum 2008, Sep 2008, Aarhus, Denmark. Springer Berlin / Heidelberg, Volume 5706/2009, pp , Lecture Notes in Computer Science. < / >. <ujm > HAL Id: ujm Submitted on 25 May 2009 HAL is a multi-disciplinary open access archive for the deposit and dissemination of scientific research documents, whether they are published or not. The documents may come from teaching and research institutions in France or abroad, or from public or private research centers. L archive ouverte pluridisciplinaire HAL, est destinée au dépôt et à la diffusion de documents scientifiques de niveau recherche, publiés ou non, émanant des établissements d enseignement et de recherche français ou étrangers, des laboratoires publics ou privés.

2 UJM at ImageCLEFwiki 2008 Christophe Moulin, Cécile Barat, Mathias Géry, Christophe Ducottet, Christine Largeron Université de Lyon, F-69003, Lyon, France Université de Saint-Étienne, F-42000, Saint-Étienne, France CNRS UMR5516, Laboratoire Hubert Curien {christophe.moulin, cecile.barat, mathias.gery, ducottet, Abstract. This paper reports our multimedia information retrieval experiments carried out for the ImageCLEF track (ImageCLEFwiki[10]). We propose a new multimedia model combining textual and/or visual information which enables to perform textual, visual, or multimedia queries. We experiment the model on ImageCLEF data and we compare the results obtained using the different modalities. Our multimedia document model is based on a vector of textual and visual terms. Textual terms correspond to textual words while the visual ones are computed using local colour features. We obtain good results using only the textual part and we show that the visual information is useful in some particular cases. 1 Introduction The capacity of data storage increases constantly, making possible the collection of large amount of information of all kinds, as texts, images, videos or combinations of them. In order to retrieve documents in such amount of data, information retrieval techniques tailored for the data types are required. First early methods in multimedia retrieval only considered the textual part of documents [2]. In order to take into account as much information as possible, other methods began to exploit the image names. The content of image is more and more used to improve results. Different features can be used such as colour [9], texture[5] or shape [1] information. The ImageCLEF collection consists of multimedia documents made up of text and images. In this paper, we present our participation to the Image- CLEFwiki[10] task. Our research goals are twofold: First, we aim to propose a multimedia document model combining text and image modalities adapted for multimedia retrieval. Second, we study the performance of our model compared to a text retrieval approach. In order to benefit from our experience with textual model, we develop a vector-based model composed of textual and visual terms. The textual terms correspond to words of the text. The visual terms are obtained through a bag of words approach. Local colour descriptors are extracted from images and quantized by k-means leading to an image vocabulary.

3 After presenting our model, we describe the submitted runs. Then, we comment on the results we obtained and conclude. 2 Visual and Textual Document Model ImageCLEFwiki is a multimedia collection where each document is composed of text and one image. User needs are represented by queries ( topics ), which are also multimedia (text, image and concept). Hence a multimedia document model is necessary to handle such a collection. We focus our work on combining textual and visual information without using the concept field of the topics. Figure 1 shows our system architecture detailed in the following paragraphs. Fig. 1. System architecture based on a vector of textual and visual terms. 2.1 Textual Representation Model One of the most known document model in textual information retrieval is the vector space model introduced by Salton and al. [8]. This model is based on a textual vocabulary T = {t 1,...,t j,...t T } where denotes the cardinal of the set T. The document i, denoted d i, is represented as a vector of weights w i,j,j 1... T where w i,j is the weight of the term t j in the document d i : d i = (w i,1,...,w i,j,...,w i, T ). In order to calculate the weight of a term t j in a document d i, a tf.idf formula is usually applied. The term frequency tf i,j measures the relative frequency of a term t j in a document d i. We use the one defined in the Okapi formula from Robertson and Jones [7]: tf i,j = (k 1 + 1) n i,j n i,j + k 1 (1 b + b di d avg ) where k 1 = 1.2 and b = 0.75 are two constants empirically defined in the Okapi formula, n i,j is the occurrence of the term t j in the document d i, d i is the size of the document d i and d avg is the average size of all documents in the corpus. The size of document corresponds to the number of terms in this document.

4 The inverse document frequency idf j measures the discriminatory power of a term t j and is defined as [7]: idf j = log D df j df j where D is the number of documents in the collection and df j is the number of documents in which the term t j occurs at least one time. The weight w i,j is then obtained by multiplying tf i,j and idf j. This weight is high when the term t j is frequent in the document d i but rare in the others. In our case, the number of terms in the vocabulary T is after applying a Porter stemming[6]. The indexing has been performed with the Lemur software Visual Representation Model In order to combine the visual information with the textual one, we also represent images with a vector of visual words. Using these visual words, it is possible to use the tf.idf formula in the same way as in textual model. It is therefore necessary to create a visual vocabulary V = {v 1,...,v j,...,v V } as in [4]. Our method consists in partitioning all images into 16x16 grids, a minimum of 8x8 pixels being required for each cell. It leads to about 256 cells per image, or about 38 million over all images. For each cell, we compute a feature vector containing the colour properties of the region. The vector has 6 dimensions which correspond to the mean and R R+G+B, G R+G+B R+G+B and the standard deviation for where R, G and B are the red, green and blue components of the cell. We apply a k-means algorithm [3] over 4 millions of cells randomly selected within the 38 millions of cells to obtain visual terms, which correspond to our visual vocabulary V for k has been chosen arbitrarily while 4 millions correspond to the maximum number of cells we could compute due to the ressources required by the k-means computation. Each visual term represents a cluster of feature vectors. Then, each new image can be represented using a vector of visual terms. It is decomposed into a 16x16 grid and the local features are computed. Each cell is then assigned to the closest visual term from our visual vocabulary V, using the euclidean distance. In the same way as for textual words, the weight of each visual term is computed using a tf.idf approach. 3 Experiments Using the model described in the previous section, we present our approach for multimedia document retrieval from multimedia queries. Then we describe the submitted runs to ImageCLEFwiki in order to evaluate our model. 1 Lemur project :

5 3.1 Queries and Matching As mentioned before, the ImageCLEFwiki topics are composed of text, image and concept modalities. However, our model is designed to only take into account text and image modalities. Our retrieval approach consists in computing a similarity score between each document d i and a query k, denoted q k, using the Okapi method. Documents are then ranked according to their scores. The following expression is used to compute the score: score(q k,d i ) = u j q k tf i,j idf j qtw k,j where qtw k,j is defined as: qtw k,j = (k 3 + 1) n k,j k 3 + n k,j where k 3 = 7 is a constant defined in the okapi formula, u j represents a textual term t j or a visual term v j and where n k,j represents the occurrence of the term u j in the query q k. Let us insist on the fact that the term u j can be either a textual term t j or a visual term v j and that queries can be composed of textual terms only, visual terms only, or both (textual and visual terms) which allows to perform text only queries, image only queries or multimedia queries. The textual terms used for queries are those provided with topics. When visual terms are used, they are extracted either from the topic images or from the collection images as detailed in the following paragraph. 3.2 Submitted Runs We have submitted 6 runs to ImageCLEFwiki 2008, labelled from LaHC run01 to LaHC run06. In order to simplify the notation, we use run 01 instead of LaHC run01. All these runs are presented and summarised with their characteristics in Figure 2. Part of them are automatic (auto), others required manual selection of some relevant documents (manual). Thanks to these 6 runs, we aim to study several aspects of our model, as the choice of the visual words and local features, the combination of textual and visual words for a query and the performance improvements obtained when adding visual information to a pure textual model. We define a baseline, run 01, that corresponds to a pure text model. It uses only textual terms for the query and scoring of documents. Its results are noted R 1. We do not use neither feedback nor query expansion for this automatic run. All other runs exploit both textual and visual information of documents. They consist of two successive queries: first one, Q 1, is textual and corresponds to the baseline (R 1 ), while the second one is a visual or textual and visual query (Q 2 ). As the user did not always provide an image for the query, we decided to build the visual information from the baseline results (R 1 ) either automatically

6 run name first query (Q 1) run type Q 1use second query (Q 2) results LaHC run01 t auto - - R 1 LaHC run02 t auto v 10 v 10 R 2 LaHC run03 t manual v 100 v 100 R 3 LaHC run04 t auto manual - v 100 t+ { vq if i q exists v 100 else LaHC run05 t manual v 100 t+v 100 R 5 LaHC run06 - auto - - R 6= R 1 R 2 R 4 with t: text only query: u j q T, R i : results of run i, i q: query image, v 10: automatic selection of the first 10 results from R 1, v 100: manual selection of the relevant documents in the first 100 results from R 1, v q: visual words extracted from the query image Fig. 2. Presentation of the runs: run 01 is the text run (baseline). run 02 to run 06 consist of two successive queries: the first one Q 1 correspond to a textual query while the second one Q 2 is a visual or textual and visual query. Visual words are selected from query images, by an automatic or manual selection. or manually. Runs are automatic when we select all the visual words of the top 10 retrieved documents issued from the baseline (v 10 ), assuming these results as relevant. Runs are manual when the user is asked to select relevant documents among the first 100 results of the baseline. (v 100 ). There is no limit in the number of selected documents as the user has to choose all relevant documents over the first 100 results. In both cases, automatic and manual runs, the visual words of the selected documents are chosen to build the second query (Q 2 ) except for the run 04 where the topic image is used when it is available. Run 02 and run 06 are automatic. Run 02 only uses visual information v 10 for the second query and its results are noted R 2. For run 06, an intersection is performed between the results of the baseline and those of run 02 (R 1 R 2 ). Results of this run are noted R 6. This intersection is interesting as it emphasises the gain of the visual information use. When a document is retrieved with run 02, and not with run 06, it means that only the visual information lets us find this document. The higher is the number of relevant retrieved documents with run 02 and not with run 06, the more efficient is the visual information. Run 03, run 04 and run 05 are manual. The second query for Run 03 is only visual, while run 04 and run 05 use multimedia queries. For run 03, all the visual words of the selected images are used for the second query. For run 04 and run 05, we perform a query expansion in order to analyse the combination

7 of textual and visual information (t+ v 100 ). We keep the textual words of the initial query and add some visual words. For run 05, these words come from the manual selected images. Run 04 proceeds as run 05, unless a query image is provided for the considered topic. In that case, the second query is composed of the visual words extracted from the query image (v q ). Thanks to these two runs, we can study the influence of the number of relevant images used for queries. 4 Results All our results are summed up in Table 1. The comparison of our textual results with other participants is presented in Table 2. Concerning visual approaches, we did not compare our runs to those of other participants as they are too different from each other. We comment the results below according to the information used for the query. Table 1. Summary of our results. Number of Number of Rank Run MAP P@10 retrieved relevant retrieved documents documents 22 LaHC run LaHC run LaHC run LaHC run LaHC run LaHC run Table 2. Best textual baseline runs of each participant. Rank Participant Run MAP P@10 11 sztaki bp acad textonly qe cwi cwi lm txt curien LaHC run ualicante IRn chemnitz cut-txt-a imperial SimpleText irit SigRunText upeking zhou ugeneva unige text baseline upmc-lip6 TFUSION TFIDF LM utoulon LSIS TXT method Text-based retrieval. The text only run (run 01) corresponds to our best run. This run is ranked 22 out of 77 runs. If we consider every baseline runs (text only using neither feedback nor query expansion) from other participants, this run is ranked 3 out of 11 (Table 2). Our textual model is quite good and let us retrieve relevant documents out of provided by the ground truth. Image-based retrieval. Concerning the visual runs (run 02 and run 03), the automatic run (run 02) is the worst of our results while the manual run (run 03) is our second one. As the precision at ten documents retrieved (P@10) for run 01 is low ( (see table 1)), it is not surprising that run 02 is our worst run. Indeed, only 36,80% of the visual information used for the query

8 is meaningful. For run 03, using only the visual words, we are able to retrieve more relevant documents than the number of relevant documents selected by the user and used for the query. For example, we retrieved 42 relevant documents for the query blue flowers whereas we had just selected 9 images manually. Unfortunately, there are some topics for which the results are bad and only a third of the topics leads to improvements. This is due to the visualness of the query. It is obvious that for a query like blue flower, the visual information is more useful than for a query like peace anti-war protest. If we consider all the relevant results from run 02 and run 03, we are able to find relevant documents. This means that using only the visual information we are able to find a fifth of the relevant documents which is quite encouraging. Improvements with visual information. Even if using the visual only model leads to worse results than the text only model, the visual information brings complementary relevant documents that are not found with the text query. The comparison between run 01, 02 and 06 informs us that over all topics, 214 new relevant documents are retrieved with only the visual query. Indeed, as we can see on Table 1, 643 relevant documents are found with run 02 and 429 with run 06. As run 06 corresponds to the intersection between the results of run 01 and the results of run 02, this means that = 214 relevant documents are found using only the visual information. To take an example with the blue flower topic, 32 relevant documents are found with run 01 and 30 with run 02. The intersection of both results coming from run 06 gives 13 shared documents. Thus, 17 relevant documents are retrieved with only the visual information. Performing the same experiment between run 01 and run 03, we find that 351 relevant documents are found with the visual information and not with the textual one. Text and image combination. The conclusion about the combination of textual and visual words did not improve the results as expected. From Table 1, we can see that run 01 leads to relevant documents. Using only visual information, we find 351 relevant documents with run 03. If we had perfectly combined the textual and the visual information, we should have found relevant documents for run 05. However, we obtain less results with only 987 relevant retrieved documents out of the expected. The comparison between run 03 and run 05 tells us that 92% of documents are shared between these two runs. Thus adding directly 2 or 3 textual words to the visual words in the query is not efficient as it gives too many importance to visuals words. Image topic use. The difference between run 04 and run 05 shows us that one image is not enough efficient to represent the visual information. Indeed, for topics with one image, the results (run 04) are always worse than results obtained when several images are selected (run 05). This can be explained by the fact that topic images were not representative enough. Furthermore, it is obvious that one image can not be enough expressive compared to several relevant images.

9 5 Conclusion We proposed a vector based model for multimedia documents. Thanks to the ImageCLEFwiki[10] collection, we were able to test our model as this collection provides visual and textual information. We obtained encouraging results with visual words only based on coloured features. For future work, concerning the textual part, we could exploit additional information such as image names. For example, TheWhiteHouse.jpg could be replace by The White House. For the visual part, as our local image features are basic, other features such as texture and edge information would surely lead to performance improvements. We also plan to automatically select the number of visual words using machine learning approaches. Finally, we aim to combine more efficiently textual and visual information. 6 Acknowledgements This work was partly supported by the LIMA project 2 and the Web Intelligence project 3 which are 2 Rhône-Alpes region projects of ISLE cluster 4. References 1. Vittorio Ferrari, Loic Fevrier, Frédéric Jurie, and Cordelia Schmid. Groups of adjacent contour segments for object detection. IEEE Transactions on Pattern Analysis and Machine Intelligence, 30(1):36 51, Abby A. Goodrum. Image information retrieval: An overview of current research. Informing Science, 3:2000, A.D. Gordon. Classification. Chapman & Hall, Frédéric Jurie and Bill Triggs. Creating efficient codebooks for visual recognition. In International Conference on Computer Vision, David G. Lowe. Object recognition from local scale-invariant features. In Proceedings of the International Conference on Computer Vision ICCV, Corfu, pages , Martin F. Porter. An algorithm for suffix stripping. Program, 14(3): , Stephen E. Robertson, Steve Walker, Micheline Hancock-Beaulieu, Aarron Gull, and Marianna Lau. Okapi at trec-3. In Text REtrieval Conference, pages 21 30, Gerard Salton, A. Wong, and C. S. Yang. A vector space model for automatic indexing. Communations of the ACM, 18(11): , Sabrina Tollari and Hervé Glotin. Wisti: a simple efficient textuo-visual web image retrieval model - specifications and benchmarks. ImagEval, Theodora Tsikrika and Jana Kludas. Overview of the wikipediamm task at Image- CLEF In Carol Peters, Danilo Giampiccol, Nicola Ferro, Vivien Petras, Julio Gonzalo, Anselmo Peñas, Thomas Deselaers, Thomas Mandl, Gareth Jones, and Mikko Kurimo, editors, Evaluating Systems for Multilingual and Multimodal Information Access 9th Workshop of the Cross-Language Evaluation Forum, Lecture Notes in Computer Science, Aarhus, Denmark, sep 2008 (printed in 2009). 2 LIMA project: 3 WI project: 4 ISLE cluster: