Exploring Image, Text and Geographic Evidences in ImageCLEF 2007

Transcription

1 Explorng Image, Text and Geographc Evdences n ImageCLEF 2007 João Magalhães 1, Smon Overell 1, Stefan Rüger 1,2 1 Department of Computng Imperal College London South Kensngton Campus London SW7 2AZ, UK 2 Knowledge Meda Insttute The Open Unversty Walton Hall Mlton Keynes MK7 6AA, UK (j.magalhaes@mperal.ac.uk, smon.overell01@mperal.ac.uk, s.rueger@open.ac.uk) Abstract Ths year, ImageCLEF2007 data provded multple evdences that can be explored n many dfferent ways. In ths paper we descrbe an nformaton retreval framework that combnes mage, text and geographc data. Text analyss mplements the vector space model based on non-geographc terms. Geographc analyss mplements a placename dsambguaton method and placenames are ndexed by ther Getty TGN Unque Id. Image analyss mplements a query by semantc example model. The paper concludes wth an analyss of our results. Fnally we dentfy the weaknesses n our approach and ways n whch the system could be optmsed and mproved. Categores and Subject Descrptors H.3 [Informaton Storage and Retreval]: H.3.1 Content Analyss and Indexng; H.3.3 Informaton Search and Retreval; H.3.4 Systems and Software; H.3.7 Dgtal Lbrares; H.2.3 [Database Managment]: Languages Query Languages General Terms Measurement, Performance, Expermentaton Keywords Semantc Image Retreval, Geographc Image Retreval 1 Introducton Ths paper presents a system that ntegrates vsual, text and geographc data. Such systems are n great demand as the rchness of metadata nformaton ncrease together wth the sze of multmeda collectons. We evaluated the system on the ImageCLEF Photo IAPR TC-12 photographc collecton to assess some of the evdence combnaton strateges and other desgn aspects of our system. As the system mplemented a set of prelmnary algorthms we were able to clearly see the dfferent mpacts of each component of our system. Ths paper s organzed as follows: secton 2 dscusses the dataset characterstcs; sectons 3 to 6 present our system: the text data processng algorthms, the geographc data processng algorthms, the mage data processng algorthms, and the combnaton strateges. Results are presented n secton 7 and fnally secton 8 presents the conclusons. 2 ImageCLEF Photo Data ImageCLEF Photo dataset s deal to test our system: t ncludes metadata nformaton (both textual descrptons and geographc nformaton), and vsual nformaton. The dataset has 20,000 mages wth the correspondng metadata. The photos vary n qualty, levels of nose, and llustrate several concepts, actons or events. Metadata enrches the mages by addng nformaton such as the fact that a street s n some locaton or the professon of one of the persons n the photos. A more thorough descrpton of the dataset can be found n [4]. The goal of ths dataset s to smulate a scenaro where collectons have heterogeneous sources of data and users submt textual queres together wth vsual examples. Ths s smlar to TRECVID search task wth a slght

2 dfference concernng geographc data and actons that are only possble to detect on vdeos (e.g. walkng/runnng). 3 System DOCNO TITLE NOTES LOCATION annotatons/00/60.eng Palma The man shoppng street n Paraguay Asuncón, Paraguay DATE March 2002 IMAGE THUMBNAIL mages/00/60.jpg thumbnals/00/60.jpg Table 1 Example of metadata nformaton avalable on the collecton. The mplemented system has two separate ndexes for mages related data and another one for metadata related data. Next we wll descrbe how nformaton s analysed and stored on both ndexes. 3.1 Metadata Indexes The ndexng stage of Forostar begns by extractng named enttes from text usng ANNIE, the Informaton Extracton engne bundled wth GATE. GATE s Sheffeld Unversty's General Archtecture for Text Engneerng. Of the seres of tasks ANNIE s able to perform, the only one we use s named entty recognton. We consder ANNIE a black box where text goes n, and categorsed named enttes are returned; because of ths, we wll not dscuss the workngs of ANNIE further here but rather refer you to the GATE manual [2] Named Entty Felds We ndex all the named enttes categorsed by GATE n a Named Entty feld n Lucene (e.g. Polce, Cty Councl, or Presdent Clnton ). The named enttes tagged as Locatons by ANNIE we ndex as Named Entty Locaton (e.g. Los Angeles, Scotland or Calforna ) and as a Geographc Locaton (descrbed n Secton 3.1.3). The body of the GeoCLEF artcles and the artcle ttles are ndexed as text felds. Ths process s descrbed n the next secton Text Felds Text felds are pre-processed by a customsed analyser smlar to Lucene s default analyser [1]. Text s splt at whte space nto tokens, the tokens are then converted to lower case, stop words dscarded and stemmed wth the Snowball Stemmer. The processed tokens are held n Lucene s nverted ndex Geographc Felds The locatons tagged by the named entty recognser are passed to the dsambguaton system. We have mplemented a smple dsambguaton method based on heurstc rules. For each placename beng classfed we buld a lst of canddate locatons, f the placename beng classfed s followed by a referent locaton ths can often cut down the canddate locatons enough to make the placename unambguous. If the placename s not followed by a referent locaton or s stll ambguous we dsambguate t as the most commonly occurrng locaton wth that name. Topologcal relatonshps between locatons are looked up n the Getty Thesaurus of Geographcal Names (TGN) [5]. Statstcs on how commonly dfferent placenames refer to dfferent locatons and a set of synonyms for each locaton are harvested from our Geographc Co-occurrence model, whch n turn s bult by crawlng Wkpeda [13]. Once placenames have been mapped to unque locatons n the TGN, they need to be converted nto Geographc felds to be stored n Lucene. We store locatons n two felds: Coordnates. The coordnate feld s smply the lattude and longtude as read from the TGN. Unque strngs. The unque strng s the unque d of ths locaton, preceded wth the unque d of all the parent locatons, separated wth slashes. Thus the unque strng for the locaton London, UK s the unque d for London ( ), preceded by ts parent, Greater London ( ), preceded by

3 ts parent, Brtan ( )... untl the root locaton, the World ( ) s reached. Gvng the unque strng for London as \ \ \ \ \ Note the text, named entty and geographc felds are not orthogonal. Ths has the effect of multplyng the mpact of terms occurrng n multple felds. For example f the term London appears n text, the token london wll be ndexed n the text feld. London wll be recognsed by ANNIE as a Named Entty and tagged as a locaton (and ndexed as Locaton Entty, London ). The Locaton Entty wll then be dsambguated as locaton and correspondng geographc felds wll be added. Prevous experments conducted on the GeoCLEF data set n [11] showed mproved results from havng overlappng felds. We concluded from these experments that the ncreased weghtng gven to locatons caused these mprovements. 3.2 Images Indexes The mage ndexng part of our system creates hgh-level semantc ndexng unts that allow the user to access the vsual content wth query-by-keyword or query-by-semantc-example. In our ImageCLEF experments we only used query by semantc example. Followng the approach proposed n [9], each keyword corresponds to a statstcal model that represents that keyword n terms of the vsual features of the mages. These keyword models are then used to ndex mages wth the probablty of observng the keyword on each partcular mage. Next we wll descrbe the dfferent steps of the vsual analyss algorthm, see [9] for detals Vsual Features Three dfferent low-level features are used n our mplementaton: margnal HSV dstrbuton moments, a 12 dmensonal colour feature that captures the hstogram of 4 central moments of each colour component dstrbuton; Gabor texture, a 16 dmensonal texture feature that captures the frequency response (mean and varance) of a bank of flters at dfferent scales and orentatons; and Tamura texture, a 3 dmensonal texture feature composed by measures of mage s coarseness, contrast and drectonalty. We tled the mages n 3 by 3 parts before extractng the low-level features. Ths has two advantages: t adds some localty nformaton and t greatly ncreases the amount of data used Feature Data Representaton We create a vsual vocabulary where each term corresponds to a set of homogenous vsual characterstcs (colour and texture features). Snce we are gong to use a feature space to represent all mages, we need a set of vsual terms that s able to represent them. Thus, we need to check whch vsual characterstcs are more common n the dataset. For example, f there are a lot of mages wth a wde range of blue tones we requre a larger number of vsual terms representng the dfferent blue tones. Ths draws on the dea that to learn a good hgh-dmensonal vsual vocabulary we would beneft from examnng the entre dataset to look for the most common set of colour and texture features. We buld the hgh-dmensonal vsual vocabulary by clusterng the entre dataset and representng each term as a cluster. We follow the approach presented n [8], where the entre dataset s clustered wth a herarchcal EM algorthm usng a Gaussan mxture model. Ths approach generates a herarchy of cluster models that corresponds to a herarchy of vocabulares wth a dfferent number of terms. The deal number of clusters s selected va the MDL crteron Maxmum Entropy Model Maxmum entropy (or logstc regresson) s a statstcal tool that has been appled to a great varety of felds, e.g. natural language processng, text classfcaton, mage annotaton. Thus, each keyword w s represented by a maxmum entropy model, ( ) MaxEnt ( w = β F ( )), p w V V F V s the feature data representaton defned on the prevous secton of vsual feature vector V, and where ( ) w β s the vector of the regresson coeffcents for keyword w. We mplemented the bnomal model, where one class s always modelled relatvely to all other classes, and not a multnomal dstrbuton, whch would mpose a model that does not reflect the realty of the problem: the

4 multnomal model mples that events are exclusve, whereas n our problem keywords are not exclusve. For ths reason, the bnomal model s a better choce as documents can have more than one keyword assgned Images Indexng by Keyword ImageCLEF data s not annotated wth keywords, thus we used a dfferent dataset. Ths dataset was compled by Duygulu et al. [3] from a set of COREL Stock Photo CDs. The dataset has some vsually smlar keywords (jet, plane, Boeng), and some keywords have a lmted number of examples (10 or less). Each mage s annotated wth 1-5 keywords from a vocabulary of 371 keywords of whch we modelled 179 keywords to annotate ImageCLEF mages. 4 Query Processng The prevous sectons descrbed how the dataset nformaton s processed and stored. Ths secton wll descrbe how the user query s processed and matched to the ndexed documents. Smlarly to the documents processng the user s query s dvded nto ts text and mage elements. Fgure 1 llustrates the query processng and how multple evdences are combned. Fgure 1 Query processng and evdence combnaton. The textual part s processed n the same way as at ndexng tme and a combnaton strategy fuses the results from the dfferent textual part processng (text terms, named enttes and nameplaces). Each mage s also processed n the same way as prevously descrbed and a combnaton strategy fuses the results from the mages. Thus, only the documents-query smlarty and the evdence combnaton s new n the query processng part. 4.1 Text Query Processng The queryng stage s a two step process: (1) manually constructed queres are expanded and converted nto Lucene s bespoke queryng language; (2) then we query the Lucene ndex wth these expanded queres Manually Constructed Query The queres are manually constructed n a smlar structure to the Lucene ndex. Queres have the followng parts: a text feld, a Named Entty feld and a locaton feld. The text feld contans the query wth no alteraton. The named entty feld contans a lst of named enttes referred to n the query (manually extracted). The locaton feld contans a lst of locaton relatonshp pars. These are the locatons contaned n the query and ther to the locaton beng searched for. A locaton can be specfed ether wth a placename (optonally dsambguated wth a referent placename), a boundng box, a boundng crcle (centre and radus), or a geographc feature type (such as lake or cty ). A relatonshp can ether be exact match, contaned n (vertcal topology), contaned n (geographc area), or same parent (vertcal topology). The negaton of relatonshps can also be expressed.e. excludng, outsde, etc. We beleve such a manually constructed query could be automated wth relatve ease n a smlar fashon to the processng that documents go through when ndexed. Ths was not mplemented due to tme constrants.

5 4.1.2 Expandng the Geographc Query The geographc queres are expanded n a ppe-lne. The locaton relaton pars are expanded n turn. The relaton governs at whch stage the locaton enters the ppelne. At each stage n the ppelne the geographc query s added to. At the frst stage an exact match for ths locaton s unque strng s added: for London ths would be \ \ \ \ \ Then places wthn the locaton are added, ths s done usng Lucene s wld-card character notaton: for locatons n London ths becomes \ \ \ \ \ \*. Then places sharng the same parent locaton are added, agan usng Lucene s wld-card character notaton. For London ths becomes all places wthn Greater London, \ \ \ \ \*. Fnally the coordnates of all the locatons fallng close to ths locaton are added. A closeness value can manually be set n the locaton feld, however default values are based on feature type (default values were chosen by the authors). The feature of London s Admnstratve Captal, the default value of closeness for ths feature s 100km. See [12] for further dscusson on the handlng of geographc queres Combnng usng the VSM A Lucene query s bult usng the text felds, named entty felds and expanded geographc felds. The text feld s processed by the same analyzer as at query tme and compared to both the notes and ttle felds n the Lucene ndex. We defne a separate boost factor for each feld. We defne a separate boost factor for each feld. These boost values were set by the authors durng ntal teratve tests (they are comparable to smlar weghtng n past GeoCLEF papers [10] and [14]). The ttle had a boost of 10, the notes a boost of 7, named enttes a boost of 5, geographc unque strng a boost of 5 and geographc co-ordnates a boost of 3. The geographc, text and named entty relevance are then combned usng Lucene s Vector Space Model. 4.2 Image Query Processng In automatc retreval systems, processng tme s a pressng feature that drectly mpacts the usablty of the system. We envsage a responsve system that processes a query and retreves results wthn 1 second per user, meanng that to support multple users t must be much less than 1 second. Fgure 2 presents the archtecture of the system. We mplemented a query by semantc example algorthm [7] that s dvded nto three parts: Semantc Multmeda Analyser: The semantc multmeda analyser nfers the keywords probabltes and s desgned to work n less than 100ms. Another mportant ssue s that t should also support a large number of keywords so that the semantc space can accommodate the semantc understandng that the user gves to the query. Secton 3.2 presented the semantc multmeda analyser used n ths paper, see [9] for detals. Fgure 2 Query by semantc example system. Indexer: Indexer uses a smple storage mechansm capable of storng and provdng easy access to each keyword of a gven multmeda document. It s not optmsed for tme complexty. The same ndexng mechansms used for content based mage retreval can be used to ndex content by semantcs.

6 Semantc Multmeda Retreval: The fnal part of the system s n charge of retrevng the documents that are semantcally close to the gven query. Frst t must run the semantc multmeda analyser on the example to obtan the keyword vector of the query. Then t searches the database for the relevant documents accordng to a semantc smlarty metrc on the semantc space of keywords. In ths part of the system we are only concerned wth studyng functons that mrror human understandng of semantc smlarty. See [7] for detals Semantc Space In the semantc space multmeda documents are represented as a feature vector of the probabltes of the T keywords (179 n our case), = d dw dw [ ] 1,..., T where each dmenson s the probablty of keyword w beng present on that document. Note that the vector of keywords s normalsed f the smlarty metrc needs so (normalsaton s dependent on the metrc). These keywords are extracted by the semantc-multmeda-analyser algorthm descrbed n Secton 3.2. It s mportant that the semantc space accommodates as many keywords as possble to be sure that the user dea s represented n that space wthout losng any concepts. Thus, systems that extract a lmted number of keywords are less approprate. Ths desgn requrement pushes us to the research area of metrcs on hghdmensonal spaces. We use the tf-df vector space model. Each document s represented as a vector d, where each dmenson corresponds to the frequency of a gven term (keyword) w from a vocabulary of T terms (keywords). The only dfference between our formulaton and the tradtonal vector space model s that we use P ( w d ) nstead of the classc term frequency TF ( w d ). Ths s equvalent because all documents are represented by a hgh-dmensonal vocabulary of length T and P ( w d ) T TF ( w d ). Thus, to mplement a vector space model we set each dmenson of a document vector as d = P ( w d ) IDF ( w ). The nverse document frequency s defned as the logarthm of the nverse of the probablty of a keyword over the entre collecton D, Semantc Smlarty Metrc = D. IDF ( w ) log ( P ( w )) Documents d and queres q are represented by vectors of keyword probabltes that are computed as was explaned n the prevous secton. Several dstance metrcs exst n the tf-df representaton that compute the smlarty between a document dj vector and a query vector q. We rank documents by ther smlarty to the query mage accordng to the cosne-dstance metrc. The cosne smlarty metrc expresson s: ( q d ) sm, = Multple Images Query Combnaton T, q d = 1. T 2 T 2 ( q ) ( d ) 1 1 = = The semantc smlarty metrc gves us the dstance between a sngle mage query and the documents n the database. There are two major strateges of combnng multple examples of a query: (1) mergng the examples nto a sngle query nput and produce a sngle rank; (2) submt several queres and combne the ranks. Obvously each of these two types of combnatons uses dfferent algorthms. For ImageCLEF2007 we mplemented a smple and straght forward combnaton strategy: we submt one query for each example and combne the smlarty values from all ndvdual queres: ( q1 q2 q3 D) = { ( q1 D) ( q2 D) ( q3 D) } sm,,, sm,, sm,, sm,.

7 Ths s an OR operaton whle an AND operaton would be acheved wth a query vector that s the product of all ndvdual query mage vectors. We return the top 1000 results. 4.3 Rank Combnaton The text query and the mage query are processed ndependently and are later combned by a smple lnear combnaton. Prevous work on ths area has found a set of good weghts to combne text and mage ranks. Applyng these weghts we reach the expresson that combnes ranks: CombnedRank = ( 1000 ImageRankPos ) ( 1000 TextGeoRankPos ). The dfferent metrc spaces hold dfferent smlarty functons, thus producng ncompatble numercal measures. Thus, we used the document rank poston (e.g. ImageRankPos ) to compute the fnal rank. The produced metrc gves the mportance of each document for the gven query. Ths lnear combnaton only consders the top 1000 documents of each rank. Documents beyond ths poston are not consdered. 5 Results and Dscusson We ran 5 experments: Text: The Text part of the query only; TextNoGeo: The Text part of the query wth geographc elements removed; ImageOnly: The Image part of the query only; Geo: The geographc part of the mage only; and Combnaton: A combnaton of the Text, ImageOnly and Geo runs. Note the TextNoGeo, ImageOnly and Geo runs are orthogonal. Results are presented on Fgure 3. We can see that TextNoGeo results acheved the best results. Next was the Text results, however, there s not a sgnfcant dfference (usng the Wlcoxon Sgned Rank Test [6]). Ths was a bt surprsng as one would expect to mprove results when you add locaton nformaton to the query and to the documents. We beleve that the decrease n performance was due to the fact that some queres use the geographc part as nclusve or exclusve. Text TextNoGeo ImageOnly Geo Combnaton Fgure 3 Retreval results for the dfferent types of analyss. The Geo results were statstcally sgnfcantly worse than all the other results wth a confdence of 99.98%. Ths s due to mnmal nformaton beng used (generally a lst of placenames). Only 26 of the 60 queres contaned geographc references. Across these geographc queres the Geo method acheved an MAP of compared to for TextNoGeo and for ImageOnly. In fact across these 26 queres there s no sgnfcant dfference between Text, TextNoGeo, Combnaton and Geo methods. Ths shows that (for the geographc queres) the geographc component of the query s extremely mportant. Image results also acheved very low results, whch gven the scope of the evaluaton mght seem a bt surprsng. Ths s related to the uses of the mages to llustrate keywords that are not obvous. For example from Fgure 4 we can see that n some cases t s very dffcult to guess the query or how mages should be combned.

8 Fgure 4 Image examples for query people n San Francsco. Summng up all these problems that we faced on sngle data-type evdences, together wth unbalanced combnatons of the dfferent types of evdences, we found that the fnal rank s almost an average of ndvdual ranks. 6 Conclusons and Future Work Most of our work was done on the documents analyss and ndexng part of the evaluaton. However, t became evdent that our sngle combnaton strategy does not cover all possble types of queres. In some cases mages should be combned wth AND, others wth OR operatons. The same happened wth text and geographc, e.g. locatons can be nclusve ( n San Francsco ) or exclusve ( outsde Australa ). Moreover, some mages only llustrate part of the query ( people n San Francsco, Fgure 4) and t s obvously dffcult to dentfy correct results wth only the vsual part of the query. All these lessons show that t s essental to make good use of the dfferent algorthms by combnng them properly accordng to the query text. Moreover, the query analyss must produce an accurate logcal combnaton of the dfferent enttes of the query to acheve a good retreval performance. In our future work we would lke to repeat the experments descrbed n ths paper usng a combnaton strategy based on the logcal structure of the query. 7 References [1] "Apache Lucene Project," [2] H. Cunnngham, "GATE, a General Archtecture for Text Engneerng," Computers and the Humantes, vol. 36, pp , May [3] P. Duygulu, K. Barnard, N. de Fretas, and D. Forsyth, "Object recognton as machne translaton: Learnng a lexcon for a fxed mage vocabulary," n European Conf. on Computer Vson, Copenhagen, Denmark, 2002, pp [4] M. Grubnger, P. Clough, HanburyAllan, and H. Müller, "Overvew of the ImageCLEF 2007 Photographc Retreval Task," n Workng Notes of the 2007 CLEF Workshop, Budapest, Hungary, [5] P. Harpng, User s Gude to the TGN Data Releases. The Getty Vocabulary Program 2.0 edton, 200. [6] D. Hull, "Usng statstcal testng n the evaluaton of retreval experments," n ACM SIGIR Conference, Pttsburgh, PA, USA, 1993, pp [7] J. Magalhães, S. Overell, and S. Rüger, "A semantc vector space for query by mage example," n ACM SIGIR Conf. on research and development n nformaton retreval, Multmeda Informaton Retreval Workshop, Amsterdam, The Netherlands, [8] J. Magalhães and S. Rüger, "Logstc regresson of generc codebooks for semantc mage retreval," n Int'l Conf. on Image and Vdeo Retreval, Phoenx, AZ, USA, [9] J. Magalhães and S. Rüger, "Informaton-theoretc semantc multmeda ndexng," n ACM Conference on Image and Vdeo Retreval Amsterdam, The Netherlands, [10] B. Martns, N. Cardoso, M. Chaves, L. Andrade, and M. Slva, "The Unversty of Lsbon at GeoCLEF 2006," n Workng Notes for the CLEF Workshop, Alcante, Span, [11] S. Overell, J. Magalhães, and S. Rüger, "Forostar: A system for GIR," n Lecture Notes from the Cross Language Evaluaton Forum 2006, 2006.

9 [12] S. Overell, J. Magalhães, and S. Rüger, "GIR experements wth Forostar at GeoCLEF 2007," n ImageCLEF 2007, Budapest, Hungary, [13] S. Overell and S. Rüger, "Geographc co-occurrence as a tool for GIR," n CIKM Workshop on Geographc Informaton Retreval, Lsbon, Portugal, [14] M. E. Ruz, S. Shapro, J. Abbas, S. B. Southwck, and D. Mark, "UB at GeoCLEF 2006," n In Workng Notes for the CLEF Workshop, 2006.