Semantic Metric for Image Library Exploration

Transcription

1 MM0472 Semantc Metrc for Image Lbrary Exploraton A. Moslovć, Member, IEEE, B. Rogowtz, Member, IEEE, Abstract We propose a method for semantc categorzaton and retreval of photographc mages based on low-level mage descrptors derved from perceptual experments. The method apples multdmensonal scalng and herarchcal clusterng analyss to dentfy canddate semantc categores nto whch human observers organze mages. Through a seres of subectve experments we refne our defnton of these categores and select a set of low-level mage features that unquely descrbe them. We then devse a new mage smlarty metrc and develop a prototype system, whch dentfes the semantc category of the mage and retreves smlar mages from the database. We tested the metrc on a set of new mages and compared the categorzaton results wth that of human observers. Our results provde a good match to human performance, thus valdatng the use of human udgments to develop semantc descrptors. Our method can be used for the enhancement of current retreval methods, better organzaton of mage/vdeo databases, and the development of more ntutve navgaton schemes, browsng methods and user nterfaces. H Index Terms Image retreval, semantc modelng. I. INTRODUCTION gh-level semantc concepts play an mportant role n the way we perceve mages and measure ther smlarty. Unfortunately, these concepts are not related to mage attrbutes drectly. Although there are many sophstcated algorthms to descrbe color, shape and texture features []-[4] these algorthms do not capture mage semantcs and have many lmtatons when dealng wth broad-content mage databases. Yet, due to ther computatonal effcency, low-level vsual attrbutes are wdely used for content-based retreval (CBR) systems, leavng the user wth a task of brdgng the gap between the low-level nature of these prmtves and hghlevel semantcs people use to udge mage smlarty. In ths work we propose an approach to overcomng ths gap and pose the followng queston: Is t possble to fnd correlatons between hgh-level semantcs and low-level descrptors and use them to capture the semantc meanng of an mage? In the CBR communty the problem of mage semantc modelng was manly dentfed as a scene recognton/obect detecton task. One system of ths type s IRIS [5], whch uses color, texture, regonal and spatal nformaton to derve the most lkely nterpretaton of the scene and generate text descrptors, whch can be nput to any text retreval system. Another approach n capturng the semantc meanng of a query mage s represented by technques that allow system to learn assocatons between semantc concepts and prmtve features from the user feedback. The earlest such system was FourEyes MIT [6]. Smlar approach was also taken by Chang et al., who ntroduced the concept of the semantc vsual template [7]. In vdeo ndexng and retreval, recent attempts to ntroduce semantc concepts nclude [8], [9]. By ncorporatng some form of the relevance feedback, these systems provde the user wth a tool for dynamcally constructng semantc flters. However, the ablty of these matched flters to capture the semantc content depend entrely on the qualty of the mages, the wllngness of the user to cooperate, and the degree to whch the process converges to a satsfactory semantc descrptor. Due to great dffcultes n classfyng mages on general level most of the recently developed systems were confned to usng low-level pctoral propertes n a doman specfc applcaton. Example applcatons nclude pattern matchng [0], fndng obectonable mages on the Internet [], face and people detecton [2], [3], classfcaton nto cty versus landscape [4], ndoor-outdoor classfcaton [5], and shape retreval [6]. As a result of applyng fxed combnatons of features for mage classfcaton, the capablty of these systems to search broad-content databases s consderably lmted. One general approach based on the use of semantc classfcaton methods was presented n [7]. However, although ths work dscusses several potentally useful mage classes ( photograph, landscape, textured, wth people, etc) the correspondng semantc models are yet to be developed, and the classfcaton s lmted only to texture vs. nontexture and graph vs. photograph classes. To overcome these lmtatons our research focuses on understandng mportant semantc categores that drve our vsual percepton, studyng human smlarty udgments to extract meanngful, dscrmnatng features of these semantc categores, and mplementng perceptually-based feature extracton algorthms and smlarty metrcs nto a realworkng system. The work we descrbe has three parts: experments, modelng, and mplementaton/testng. In the frst part we conducted several subectve experments amed at: a) developng and refnng a set of canddate perceptual categores n the doman of photographc mages, b) dervng a semantc name for each category and c) dentfyng a combnaton of low-level features that best descrbe each category. In the modelng part, we desgned a new smlarty metrc for mage annotaton and retreval usng the semantc concepts. Fnally, we have mplemented a prototype annotaton/retreval system, tested t aganst the udgments of human observers, and appled t on a large number of mages collected on the Internet. The paper s organzed as follows: Secton II descrbes the

2 MM experments and analyses conducted to develop and refne our set of semantc categores. Experments conducted to determne low-level mage descrptors for each category are descrbed n Secton III. Feature extracton algorthms are descrbed n Secton IV. Secton V ntroduces new metrcs for semantc categorzaton of mages. Secton VI descrbes an experment n whch the metrc s performance s compared wth human udgments, usng a new set of mages. Secton VI also ncludes mage categorzaton and retreval examples from a large set of mages collected on the Internet. Dscusson and fnal remarks are found n Secton VII. II. EXPERIMENTS We conducted three experments: ) mage smlarty experment amed at developng and refnng a set of canddate semantc categores n the doman of photographc mages, 2) a category namng and descrpton experment amed at refnng these categores, dervng a semantc name for each category and a set of low-level features to descrbe t, and 3) an mage categorzaton experment to test the results of the metrc, derved from the prevous experments, aganst the udgments of human observers on a new set of photographc mages. Images used n the experments were selected from standard photo CD collectons to nclude a wde range of topcs: people, nature, buldngs, texture, obects, ndoor scenes, anmals, etc. We bult three expermental sets. The frst 97 mages (Set ) were dentcal to those used n our earler mage smlarty study [8]. Sets 2 and 3, wth 99 and 78 mages respectvely, were chosen to refne ntal fndngs and test the results. Seventeen subects partcpated n the experments. All subects had normal color vson and were not famlar wth the nput mages. A. Selecton of ntal categores In our prevous work [8], we had used two methods for measurng smlarty between the 97 mages n our data set, and had appled multdmensonal scalng (MDS) to analyze the resultng smlarty matrces. Both methods revealed two maor axes, one we labeled human vs. non-human, and the other we labeled natural vs. manmade. In both results, we observed that the mages clustered nto what appeared to be semantc groupngs, but we dd not analyze ths result further. As a startng pont n determnng the canddate categores that drve human smlarty udgments, we used the smlarty data from [8] and performed herarchcal cluster analyss (HCA). We found that perceptual dstances between the 97 mages were ndeed organzed nto clusters. To confrm the stablty of the most mportant clusters n the HCA soluton we splt the orgnal data n several ways and performed separate HCAs for each part. As suggested n [9], we elmnated some of the stmul from the data matrx and appled the HCA for the remanng stmul. The clusters that remaned stable for varous solutons determned the ntal categores (IC). B. Experment : Dervng the semantc categores The purpose of Experment was to collect a new set of smlarty udgments whch would allow us to: ) examne the perceptual valdty and relablty of the ntal categores dentfed by the HCA, 2) develop a fnal set of semantc categores, and 3) establsh the connectons between the categores. For ths experment, we prnted 97 thumbnals of mages n Set, organzed by cluster, and glued them on a tabletop, accordng to ther ntal categores. The mages were organzed smlarly to ther orgnal MDS confguraton wth a spatal gap between the dfferent categores. We also prnted thumbnals of mages from Set 2 (mages that have not been used n dervng clusters). Twelve subects partcpated n the experment. They were asked to assgn each mage from Set 2 nto one of the ntal categores, placng them onto the tabletop so that the most smlar mages were near each other. We provded no nstructons concernng the characterstcs on whch the smlarty udgments were to be made, snce ths was the very nformaton we were tryng to dscover. To counterbalance any effect the orderng of the stmul mght have on the subectve udgments the order of the stmul was random and dfferent for each subect. The subects were not allowed to change the ntal categores. However, the subects were allowed to do whatever they lked wth the new mages - change ther assgnments durng the experment, move mages from category to category, keep them on the sde and decde later or start ther own categores. Fnally, at the end of experment, the subects were asked to explan some of ther decsons. As we wll descrbe later, these explanatons as well as relatve placements of mages wthn the categores were valuable n data analyss. (Some addtonal aspects regardng the desgn of the experments, as well as some of ther lmtatons are addressed n Secton VII) C. Data analyss and results for Experment The frst step n the data analyss was to compute the smlarty matrx S2 for the mages from Set 2. The matrx entry S2 (, ) represents a number of tmes mages and occur n the same category. We then used multdmensonal scalng to analyze ths smlarty matrx. Note, that n our case matrx elements represent smlartes. Snce MDS methods are based on the dea that the scores are proportonal to dstances, t was necessary to preprocess the collected data accordng to the followng relaton: dssmlarty = NS - smlarty. () where NS s number of subects n the experments. We appled 2D MDS to the smlarty matrx S2. As already mentoned, [8] reported two domnant dmensons n human percepton of mage smlarty: ) natural vs. man-made, and 2) human vs. non-human. To examne f the same dmensons apply to the new data, we rotated the confguraton to resemble the soluton from [8] and supermposed the dmensons on t. We observed excellent match between the confguratons, wth the same nterpretaton for the domnant axes. The second step n the data analyss was to test the stablty of the ntal categores and further refne them. To do so, we

3 MM computed the smlarty matrx S 2, IC for the mages from Set 2 and the ntal categores IC. The matrx entry (, ) s computed n the followng way: S 2, IC = number of tmes mages and occured n the same category,, Set 2 = number of tmes mage Set 2, S2, IC (, ) = occured n the category, IC = d(, ), IC where d(, ) s the Eucldean dstance between the centrods of the ntal clusters normalzed to occupy the same range of values as smlarty measures and. Once the smlarty matrx was computed we appled the HCA to determne the fnal set of semantc categores (FC), whch now ncluded 96 mages. The HCA result s shown n Fg.. The frst supercluster that emerged from our experments represents mages of people, followed by the clusters wth mages of man-made obects and man-made envronments. The remanng mages were further subdvded nto natural scenes and natural obects (pctures of anmals, plants, etc.) thereby confrmng the multdmensonal scalng results on the frst set of mages. D. Experment 2: Namng and descrbng the categores The confguraton n Fg. represents one possble set of classes people used to organze photographc mages. Snce testng the human percepton s an extremely dffcult task, we were aware that these classes may reflect our decsons n desgnng the frst experment. Therefore, to refne the algorthmcally derved categores and determne whether these were semantcally dstnct we conducted a new task, n whch we asked observers to assgn names to the fnal categores dentfed n Experment. To further delneate the categores and dentfy hgh-level mage features that dscrmnate them, we also asked the observers to provde descrptors for each category. To do so we prnted a category notebook, whch contaned one sheet for each category. All mages from the category were prnted on the top half of the sheet. The bottom half provded a lne on whch the subect was nstructed to wrte the name he/she would assgn to the category. Eleven subects partcpated n ths experment. Sx of them had taken part n Experment and were famlar wth the mages and clusterng procedure, whle the other fve subects had not seen the mages before and were not famlar wth the obectves of the experment. Each subect was gven the notebook, asked to name each category and wthn the blank space wrte ts bref descrpton and man propertes. Ths task was helpful n many ways. Frst, t was used to assess the robustness of the categores and test whether people see them n consstent manner. For example, f the HCA revealed two separate categores, yet the names and descrptors were ndstngushable, we merged the categores. The experment also helped us establsh f the determned categores are semantcally relevant. And fnally, the wrtten explanatons were used n determnng pctoral features that best capture the semantcs of each category. E. Results and analyss for Experment 2 Based on the semantc names the observers assgned to each category, we constructed the set of 20. Although ndvdual subects used dfferent descrptons to characterze dfferent categores, there were many consstent trends. We were surprsed to dscover that certan obects had domnatng nfluence to the nterpretaton of mage content. For example, n the nature categores for all subects water, sky/clouds, snow and mountans emerged as very mportant cues. These were often strongly related to each other, determnng the organzaton and lnks between the groups. The same held true for mages wth people. Our observers were very senstve to the presence of people n the mage, even when t depcted a natural scene, obect, or manmade structure. We also observed that color composton and color features played an mportant role n comparng natural scenes and were rarely used when descrbng mages wth people, obects and lfe-scenes. Wthn these categores, spatal organzaton, spatal frequency and shape features nfluenced smlarty udgments. Wth an excepton of flowers, fruts and exotc anmals, strong hues (such as brght red, yellow, lme green, pnk, etc.) were not n the descrptons of natural scenes. Therefore, we found these colors useful n dentfyng manmade obects n the pcture. Addtonal characterstcs of man-made mages were regular textures, straght lnes, straght boundares, sharp edges, and geometry (as opposed to rgd boundares, hgh curvature and random dstrbuton of edges, whch were found mportant n descrbng scenes from nature). Appendx provdes a complete lst of categores and detaled descrpton of ther key features.. To develop a measure of the dstance between categores we have also computed the confuson matrx CM, where each entry CM(, ) represents the average number of mages from category c placed wth mages from category c. Together wth the comments from the subects, we used these values to nvestgate the relatonshps and establsh transtons between the categores. As a result we constructed a graph ndcatng possble connectons and transtons between the categores. Each category was represented as a node n the graph, and two nodes were connected f the correspondng categores had the confuson rato above a predefned threshold. III. LOW LEVEL DESCRIPTORS AND IMAGE SEMANTICS Havng dentfed a set of canddate semantc categores, the next step was to model them for the use n mage retreval and browsng applcatons. Here we focused on the hgher-level descrptors provded by the observers wth the followng queston n mnd: Is t possble to fnd a set of low-level features and ther organzaton capable of capturng semantcs of a partcular category? As a startng pont we

4 MM used the wrtten descrptons of the categores gathered n the second experment and devsed a lst of verbal descrptors people found crucal n dstngushng them. We have then translated these descrptors nto calculable mage-processng features. For example, mage consstng prmarly of a human face, wth lttle or no background scene, used to descrbe category Portrats, n mage-processng language corresponds to domnant, large skn colored regon. Or, busy scene, used to descrbe category Crowded scenes wth people, n mage-processng language corresponds to hgh spatal frequences. We then further expanded the lst by addng some features we consdered useful, producng a lst of over 40 mage-processng features, called the complete feature set (CFS). As an llustraton, here we lst some of the them: number of regons after segmentaton (large, medum, small, one regon), mage energy (hgh, medum, low), regularty (regular, rregular), exstence of the central obect (yes, no), edge dstrbuton (regular, drectonal, nondrectonal, rregular), color composton (brght, dark, saturated, pale, gray overtones), blobs of brght color (yes, no), presence of geometrc structures (yes, no), number of edges (large, medum, small, no edges), corners (yes, no), straght lnes (occasonal, defnng an obect, no straght lnes), etc. Note that feature values n our representaton are dscrete and the results of the correspondng mage-processng operatons should be quantzed to reflect these verbal descrptons. To fnd features that correlate wth the semantcs of each category we used an IBM data mnng and vsualzaton software and compared expermental data wth mageprocessng descrptors for a set of 00 mages. Specfcally, for each category we were nterested n fndng a feature combnaton that dscrmnates that category aganst all other mages. For example, we found the followng feature combnaton unque for mages from category Ctyscapes : SKIN = no, FACE = no, NATURE = no, ENERGY = hgh, # EDGES = large, # REGIONS = large, REGION SIZE = small, CENTRAL OBJECT = no, DETAILS = yes Smlar analyss was performed for all categores. We have dscovered that wthn a certan category not all the features were equally mportant. For example, all mages n the Ctyscapes category had hgh spatal frequences, many detals, domnant brown/gray overtones, and a large number of small regons. These features are thus consdered as requred features (RF) for the Ctyscapes category. On the other hand, most of the mages from ths category (but not all of them) had straght lnes or regons wth regular geometry, due to manmade obects n the scene. Or, although the colors were predomnantly brow/gray/dark, many mages had blobs of saturated colors, agan due to man-made obects n the scene. Therefore, straght lnes, geometry and blobs of saturated color were consdered as frequently occurrng (FO) features for the Ctycsapes category. IV. OVERVIEW OF THE FEATURE EXTRACTION PROCESS When descrbng mages humans typcally use attrbutes lke there s a human n the pcture, background s black, there s one round obect n the center, etc. We made an attempt to desgn feature extracton methods that capture semantc attrbutes used n these descrptons. Here we brefly descrbe the mportant steps n the feature extracton process. Snce human observaton of a scene s very dfferent from the recorded mage and nvarant to changes n lghtng condtons, we frst use a varant of the Von Kres adaptaton to estmate the scene llumnant and apply smple chromatc adaptaton transform [20]. The resultng mage s then subected to three types of segmentaton: texture segmentaton, color segmentaton and foreground/background segmentaton. Frst, each relevant regon from the color segmentaton s assgned a structure contanng nformaton about ts sze, boundary, texture (from the texture segmentaton), mean color and colorname (e.g. red, lght pnk, dark gray ). The texture and color maps are then combned to acheve the foreground/background segmentaton and determne f there s a domnant obect (or obects) n the mage. For each relevant obect we compute smple shape features (boundary, eccentrcty, moments, symmetry features), color name, color and texture propertes. In the current mplementaton texture map s generated by computng a set of drectonal edge maps followed by a regon growng procedure. For the color segmentaton we use the mean-shft algorthm [2]. For each extracted regon a novel color namng procedure s carred by comparng the average regon color wth the set of standard colors descrbed n the ICCS NBS Color Dctonary [22] usng the modfed L 2 -norm n the Lab color space as the color namng metrc [20]. One of the mportant semantc queues s the presence of humans n the mage - the skn feature. We desgned a novel algorthm for detecton of skn regons. Ths s a geometrc method where mage pxels are nterpreted as ponts n the 4D-Eucldan space. The coordnates of a pxel n ths space are the three Lab color coordnates and the measure of the overall color varaton. The later s totally encoded n the spatal Jacoban of (L, a, b) but we used only ts Eucldean norm n. To buld a skn color model n ths space, we used skn pxels collected from the tranng set of 260 mages. The manfold of the skn color s then reconstructed va the 4D ansotropc dffuson, by solvng a desgnated Partal Dfferental Equaton. The skn regons are dentfed by computng the dstance between each relevant regon and the skn manfold [23]. All regonal features are then combned to provde global descrptors. These nclude number of regons, number of blobs, number of regons wth specfc color, measures of local and global contrast. Furthermore, the color names from all relevant regons are combned nto a color name hstogram to determne a color appearance of the mage. The hstogram generates descrptons such as graysh, vvd colors, dark mage, green regons, etc. In many cases colornames only, or n combnaton wth other features (such as spatal attrbutes, boundary and sze features), are capable of capturng mage semantcs. For example, upper regons labeled lght blue and bottom regons labeled strong green may

5 MM represent sky and grass, regons wth regular boundares, geometry and vvd colors are very lkely to be man-made obects, whle modfers such as brownsh and dark convey the mpresson of the atmosphere n the scene. V. CATEGORIZING AND RETRIEVING IMAGES A. Semantc categorzaton/annotaton Havng determned one possble set of smlarty categores, ther relatonshps and dstnct features, our obectve s to devse an mage smlarty metrc that embodes our perceptual fndngs and models the behavor of subects n categorzng mages. The metrc we propose s based on the followng observatons from our experments: Each semantc category, c, s unquely descrbed by the set of features deally, these features can be used to separate the category from other categores n the set. Therefore, to descrbe the category c, we wll use the followng feature vector: f c ) = [ RF ( c )... RF ( c ) FO ( c )... FO ( c )] (3) ( M N where: { RF ( c ) =,..., M } s a set of M requred features, and { FO ( c ) =,..., N} s a set of N frequently occurrng features for the category c. To assgn a semantc category to the nput mage we need a complete feature set for that mage, CFS(x). However, when comparng x to the semantc category c, we wll use only a subset of features, f ( x c ), consstng of the features that capture the semantcs of that category: f ( x c ) = [ RF ( x c )... RFM ( x c ) FO ( x c )... FO ( x c )] The smlarty between the mage x and category c s computed va the followng metrc: sm( c ) = sm( f ( x c ), f ( c )) = M N τ ( RF ( x c ), RF ( c )) τ ( FO N = where: = N ( x c ), FO ( c )) (4) (5), ( ) a = b τ( a, B) =, and B = { b } =,..., I. (6) 0, ( ) a b The metrc represents a formal descrpton of what we found so far. To assgn the semantc category c to the mage all requred features have to be present and at least one of the frequently occurrng features has to be present. As the feature could have more than one value (I possble values), t s compared to each possble value va (6). B. Image retreval Besdes semantc categorzaton, the proposed metrc can be used to measure smlarty between two mages, x and y as: sm( y c ) = N N τ ( = M τ ( = FO ( x c ), FO ( y c )) RF ( x c ), RF ( y c )) (7) sm y) = max( sm( y c ) ) (8) ( However, note that the smlarty score s greater than zero only f both mages belong to the same category. To allow comparson across all categores we propose a less strct metrc. We frst ntroduce the smlarty between mages x and y, assumng that both of them belong to the category c as: sm( y c ) = 2 N (+ τ ( FO = Assumng that M 2 M ( x c ), FO ( + τ ( RF N = ( y c ))). x c and ( x c ), RF ( y c ))) y c the overall smlarty s: sm y) = [ sm( y c ) + sm( y c )]/2 (0) ( VI. TESTING THE ALGORITHM To test the metrc we developed a SMILE (Semantc Metrc for Image Lbrary Exploraton) applcaton. In the categorzaton task, SMILE loads a precomputed complete feature set for the selected mage and apples the rules (Appendx) and metrc (5) to compute the smlarty measure between the mage and each semantc category. The mage s then assgned to the semantc category wth the hghest smlarty score. In the retreval task, the user s asked to specfy the query mage and the database s searched to fnd all mages smlar to the query accordng to the strct (8) or lessstrct metrc (0). We tested the categorzaton algorthm usng 274 mages (97 mages from Set, 99 mages from Set 2 and 78 new mages). Instead of the automatc feature extracton algorthms we decded to use a manually flled feature table. Ths perfect feature table helped us gan better understandng of the metrc by elmnatng possble msclassfcatons due to feature extracton naccuraces. Categorzaton results for Sets and 2 are gven n Table I. Snce mages from both Set and Set 2 were used to dscover the semantc categores and ther descrptors, we were nterested n analyzng the behavor of the model n classfyng new mages. To do so, we have performed another experment. For ths experment we selected 78 new mages (mages that were not used n the experments). The mages were chosen accordng to the same crtera used n buldng Sets and 2. To test the metrc obectvely, we avoded selectng perfect replcas of the expermental mages and ncluded mages, whch, n our opnon, could be categorzed n several ways. We asked sx subects to classfy each mage nto one of the 25 semantc categores, or leave t unassgned f unable to fnd approprate category. There was a reasonable degree of concordance between the subects - out of 78 mages, 24 of them were (9)

6 MM assgned to the same category by all sx subects (Group ), 22 were assgned to one of two dfferent categores (Group 2), and 6 were assgned to one of three categores (Group 3). The remanng 6 mages were not relably assgned nto any category. We have then tested the algorthm aganst these human udgments. For Group, the algorthm categorzed 79% of the mages correctly, for Group 2, the algorthm dentfed one of the two categores selected by the subects 86% of the tme, and for Group 3 the algorthm dentfed one of the three categores 00% of the tme. There were 6 mages that were not relably categorzed by the human observers. For of these 6 mages, the algorthm performed lke at least one of the observers. Altogether, there were ten msclassfed mages, and we conclude that the algorthm s as roughly good as the humans are n assgnng the categores. Fgs. 2-6 show examples of mage categorzaton and retreval usng our metrc. Fg. 2 shows several categorzaton results for the new set, Fg. 3 shows several msclassfed mages from Sets and 2, whle Fg. 4 shows mages assgned nto one of the related categores. By analyzng the mage categorzaton results we observed that many mages were classfed nto more than one category. In addton to mages from Set 3, whch had been selected to fall somewhere n between exstng categores, there were several mages from Sets and 2 that also confused the subects n our experments. Fg. 5 shows several examples of multple categorzaton. Hence, nstead of assgnng an mage nto only one category, combnng the categorzaton results provded addtonal nformaton about the mage and brought a new dmenson nto the model. Although our study was lmted to 96 tranng mages and 78 test mages, we beleve the presented results are promsng. The canddate semantc categores we have dentfed, although not perfect, are stable and relable, and seem robust when tested wth mages especally selected to challenge our model. To further test the method, we have appled the semantc categorzaton and retreval scheme wth the automatcally extracted features (descrbed n Secton IV) to a large number of mages collected from the Internet (approxmately 3,500 mages ndexed by a web robot). Fgs. 7 and 8 are examples of mage categorzaton for categores Waterscapes and Outdoor archtecture. Fg. 9 shows an mage retreval example. As t can be seen, the results are very promsng. Although some mages are classfed nto a wrong category, these msclassfcatons are manly due to errors n the feature extracton procedure. For example, by nspectng the features computed for the Mountan mage, (the last mage n Fg. 7), we realzed that several bottom regons were labeled blue and nterpreted as water. Smlarly, the skn detecton procedure faled to dentfy the faces n the last example n Fg. 8, and the mage s labeled ncorrectly as Outdoor archtecture. VII. DISCUSSION AND CONCLUSIONS We have dentfed canddate semantc categores human observers use to organze photographc mages. Snce mage smlarty s such a hgh-dmensonal udgment, t s not realstc to beleve that we have uncovered the fnal mmutable set of categores. We consder these to be a useful startng pont and as the expermental decsons represent only partal and subectve vew of the mage world, we are not makng an attempt n clamng the generalty of the soluton. Testng human percepton s an extremely dffcult task and we are aware that the expermental setup poses many lmtatons to our conclusons. The sze of the data set, ncremental nature of the procedure, and the technques used may have nfluenced the outcome of the experments. Our ntal classes were obtaned from the set of 97 mages, whch may not be a suffcent enough sample to generalze human behavor. Unfortunately, the sze of the ntal database was an unavodable choce, as subects attenton decreased sgnfcantly wth the length of the experment. Hence, the number of mages n the database represented the tradeoff between the dversty of the semantc content and relablty of the subectve data. To compensate for the sze of the database we have carefully selected the expermental mages to provde a broad semantc content. We kept the same obectve n the subsequent experments, whle ncludng mages that challenged the ntal model (by fallng between the categores or beng sgnfcantly dfferent from them). To mnmze the effect of the selecton of the ntal categores, n Experment we allowed the subects to ntroduce new classes and used ther subectve remarks n checkng the confdence of the fnal soluton. Furthermore, although a common method for dscoverng the hdden structure n the data, the HCA s far from beng robust and the fnal result s hghly related to the subectve decson on when to stop the procedure. Ths problem was partally elmnated by repeatng the procedure and keepng only clusters that appear robust n varous confguratons. In addton to dfferent HCA s, we reled on the results from Experment 2 (n whch we asked observers to provde category names and descrptons) to further refne the categores. For example, f the herarchcal cluster analyss merged two sub-categores, yet the observers used dstnctly dfferent names and descrptors to dentfy them, we kept them as two separate categores (and vce versa). Ths may account for several small categores wth 2 or 3 mages only. Although these categores are not as stable as larger clusters representng generc classes, we tred to accept the fact that human categorzaton goes beyond the generc prototypes as we tend to attach qute detaled lngustc labels. Ths s also supported by the resultng MDS confguratons. Although generc classes were the most dstngushable, our udgments of mage smlarty were not perfectly two-dmensonal and mages clustered nto addtonal semantc categores. Our system dffers from tradtonal retreval technques n several ways. Frst, we use the noton of adaptve features tuned to the semantc categores nto whch observers organze photographc mages. We do not, for example, use a color hstogram for each mage nstead, we use the features that dscrmnate between semantc categores. Furthermore, the

7 MM delneaton between "requred" and "frequently occurrng" features captures the fact that some descrptors are more mportant for some categores than for others. For example, long lnes, straght boundares are crtcal features for dentfyng "Outdoor archtecture" but rrelevant n dentfyng people. In current mage retreval paradgm the crteron for success s whether the system dentfes all the exstng dentcal or near dentcal mages n the database. Although ths can be of nterest n some lmted applcatons, such as cleansng a database of duplcate mages or selectng the "best shot" of somethng n a roll of flm, n most real applcatons, we expect that the user really would lke to fnd semantcally smlar mages as opposed to vsually dentcal ones. In ths case, the nput mage would be used as a seed for specfyng the semantcs of the query, not as a request for a near copy. Our categores, although not perfect, follow the dea of semantc concepts as gudelnes n our percepton of mage smlarty. These gudelnes are not determnstc and depend on a task, user or envronment. Therefore, the system we propose does not requre that the categores be perfect. It only requres that the categores capture enough of the nformaton about smlarty udgments to help organze the mages semantcally. We beleve that the most useful applcatons for mage lbrares wll not be those that retreve exact mage matches, but those that provde a meanngful context for browsng and navgaton. The current approach certanly provdes a dfferentaton n such tasks. VIII. APPENDIX C Portrats: (RF) Domnant skn colored, oval regon. C2a People outdoors: (RF) Medum szed skn colored regon. (FO) An ndcator of nature (sky, grass, water). Lghter colors, mage appears brght. Lght dstrbuton typcal for outdoor llumnaton. C2b People ndoors: (RF) Medum szed skn colored regon. (FO) No ndcators of nature. Darker, less saturated colors. Image appears dark. Lght dstrbuton typcal for ndoor llumnaton. C3 Outdoors scenes wth people: (RF) Small blobs wth skn color, solated pxels wth skn color or human slhouettes. (FO) A domnant ndcator of nature (sky as an upper blue regon, grass as a bottom green regon.). Lghter colors, mage appears brght. C4 Crowds of people: (RF) Small blobs wth skn color, solated pxels wth skn color or human slhouette. Hgh spatal frequences. Large number of small regons, many detals. (FO) Color composton domnated by darker colors, gray or brown overtones. Straght lnes, regons wth regular geometry, brght blobs. C5 Ctyscapes: (RF) No skn-colored regons, pxels, or slhouette. Hgh spatal frequences. Large number of small or medum szed regons. Many detals. (FO) Color composton domnated by darker colors, gray or brown overtones. Straght lnes, regons wth regular geometry or brght/saturated blobs. C6 Outdoor archtecture: (RF) No skn-colored regons, pxels, or slhouette. Low or medum spatal frequences. Obects wth straght boundares. (FO) Small nature regons (sky or grass). C7 Techno-scenes : (RF) No skncolored regons, pxels, or slhouette. Medum spatal frequences. Lght dstrbuton typcal for outdoor llumnaton. No ndcators of nature. Medum or long edges, mostly straght or curved. (FO) Color composton domnated by darker colors, gray or brown overtones. Straght lnes, regons wth regular geometry, brght regons or blobs of brght/saturated color. C8a Obects ndoors: (RF) No skn-colored regons, pxels, or slhouette. Low or medum spatal frequences. Few large regons n foreground/background settng. Central obect wth straght lnes, curved contours, or geometry. (FO) No ndcators of nature. Lght dstrbuton typcal for ndoor llumnaton. Color composton domnated by darker colors, gray or brown overtones. C8b Indoor scenes wth obects: (RF) No skn-colored regons, pxels, or slhouette. Image energy concentrated n md-frequences. No ndcators of nature. Lght dstrbuton typcal for ndoor llumnaton. No central obect. Many smaller obects and detals n a scene. (FO) Domnant colors are darker, mostly brown and gray. Straght lnes, regons wth regular geometry, brght/saturated regons or blobs of brght/saturated color. C9 Obects outdoors: (RF) No skn-colored regons, pxels, or slhouette. Image energy concentrated n lower frequences. Few large regons n typcal foreground/background settng. Central obect wth straght lnes, curved contours, or geometry. (FO) Nature regons. Lghter colors, mage appears brght. Lght dstrbuton typcal for outdoor llumnaton. C0 Waterscenes wth human nfluence: (RF) No skn-colored regons, pxels, or slhouette. Unform blue regon (stll water) or textured whte/gray/blue regon (waves). Nature domnates the scene. No central obect wth straght/curved boundares or geometry. Man-made obects are dentfed by ther regular placement, brght colors (blobs of red, whte, etc.), geometry or straght lnes. Man-made regons are much smaller than nature. (FO) Other ndcators of nature (sky, mountans, coastlne, grass/tree texture). C Landscapes wth human nfluence: (RF) No skn-colored regons, pxels, or slhouette. No water. Large nature regons. No central obect wth straght/curved boundares or geometry. Number of regons s small or medum. Domnant green or earthy colors. Man-made obects dentfed by ther regular placement, brght colors, geometry or straght lnes. Man-made regons are much smaller than nature. C2 Landscapes wth water: (RF) No skn-colored regons, pxels, or slhouette. Unform blue regon or textured whte/gray/blue regon. Nature domnates the scene. No central obect wth straght/curved boundares or geometry. Number of regon s small or medum. Regon boundares are rgd. No straght lnes (except horzon or coastlne). No geometry and blobs of saturated color. C3 Mountans: (RF) No skn regons. An outlne of a mountan n the center of the pcture. No water. Nature domnates the scene. No central obect wth straght/curved boundares or geometry. Small number of regons. Regon boundares are rgd. No straght lnes (except horzon or coastlne). No geometry and blobs of saturated color. C4 Sky/clouds: (RF) No skn regons. Sky/clouds regon domnates the pcture. No central obect wth straght/curved boundares or geometry. Regon boundares are rgd. No straght lnes (except horzon or coastlne). C5 Wnter & snow: (RF) No skn regons. Large snow regon (or regons). No outlne of a mountan. No water. Regon boundares are rgd or curved. No straght lnes. No geometry, regons and blobs of saturated color. (FO) Only few domnant colors. Hgh contrast. C6 Green landscapes: (RF) No skn regons. Impresson of green as the only color. No strong hues. Domnant nature regons. Regon boundares are rgd or curved. No straght lnes. No geometry, regons and blobs of saturated color. C7 Felds and folage: (RF) No skn regons. One or more nature regons (sky, grass, etc.). No water or mountan. Regon boundares are rgd. Domnant fall colors, earthy colors and greens. No straght lnes, geometry and blobs of saturated color. (FO) Large textured regons wth a pattern typcal for trees, leaves, grass. C8a Close-ups of plants: (RF) No skn regons. No textured regons. Central obect wth rgd/curved boundares and brght/saturated color. Green background. Only few domnant colors. Regon boundares are curved or rgd.

8 MM Image energy s n the low or md-low frequences. No straght lnes or geometry. C8b Plants on a small scale: (RF) No skn regons. One or more ndcators of nature (sky, grass, etc.). No central obect. Regon boundares are rgd. Image energy s n the hgh or md-hgh frequences. At least one saturated, brght color. Green s among domnant colors, but not as a background. Domnant colors are sparsely dstrbuted. No straght lnes or geometry. C8c Plants touched by humans: (RF) No skn regons. No textured regons. No straght boundares or geometry. Regon boundares are curved or rgd. At least one regon wth saturated or brght color. (FO) Darker colors or dark background. Shadows and black regons. Regularly placed obects. No ndcators of nature. Lght dstrbuton typcal for ndoor llumnaton. C9a Waterlfe: (RF) No skn regons. Unform blue regon or textured whte/gray/blue regon. Water domnates the scene (blue background or blue as the most domnant color). No central obect wth straght lnes, sharp edges, curved contours, or geometry. No gray overtones, domnant dark or brown colors. (FO) No central obect and segmentaton yelds mostly medum and small regons. Central obect wth curved boundares. C9b Close-ups of anmals: (RF) No skn regons. Image energy s n the lower or md-lower frequences. At least one ndcator of nature. Central obect wth rgd or curved boundares. Few large regons n typcal foreground/background settng. No straght lnes, straght boundares, or geometry. Lght dstrbuton typcal for outdoor llumnaton. No gray overtones, domnant dark or brown colors. (FO) Green background. No strong hues. Nature n the background. C9c Anmals n nature: (FO) No skn regons. At least one nature regon, large or medum szed. No water. No central obect. Number of regons s small or medum. Regon boundares are rgd. No blobs of saturated brght colors. At least one textured regon. No straght lnes, straght boundares, or geometry. C20 Textures and patterns: (RF) Texture pattern occupes the whole mage. REFERENCES [] W. Nblack, R. Berber, W. Equtz, M. Flckner, E. Glasman, D. Petkovc, and P.Yanker, The QBIC proect: Querng mages by content usng color, texture and shape, n Proc. SPIE Storage and Retreval for Image and Vdeo Data Bases, pp , 994. [2] H. Tamura, S. Mor, and T. Yamawak, Textural features correspondng to vsual percepton, IEEE Transactons Systems, Man and Cybernetcs, vol. 8, pp , 982. [3] A. Moslovc, J. Kovacevc, J. Hu, R. J. Safranek, K. Ganapathy, "Matchng and retreval based on the vocabulary and grammar of color patterns", IEEE Trans.Image Proc., vol. 9, no., pp , Jan [4] M. Swan, and D. Ballard, Color ndexng, Int. Journal of Computer Vson, vol. 7, no., pp. -32, 99. [5] T. Hermes, et al. Image retreval for nformaton systems, Storage & Retreval for Image and Vdeo Databases, Proc. SPIE 2420, , 995. [6] T. Mnka, An mage database browser that learns from user nteracton, MIT Meda Laboratory Techncal Report #365, 996. [7] S. F. Chang, W. Chen, and H. Sundaram, Semantc vsual templates: lnkng vsual features to semantcs, n Proc. IEEE Internatonal Conference on Image Processng, Chcago, Illnos, pp , 995 [8] M. Naphade, and T. Huang, Probablstc framework for semantc vdeo ndexng, flterng and retreval, IEEE Transactons on Multmeda, vol. 3, no., pp. 4-5, March 200. [9] A. M. Ferman, and M. Tekalp, Probablstc analyss and extracton of vdeo content, n Proc. IEEE Int. Conf. Image Processng, Kobe, Japan, Oct [0] A. Moslovc, J. Kovacevc, D. Kall, R. J. Safranek, and K. Ganapathy, "Vocabulary and grammar of color patterns", IEEE Trans. on Image Processng, vol. 9, no. 3, pp , March [] M. Fleck, D. A. Forsyth, and C. Bregler, Fndng naked people, Proc. European Conf. Computer Vson, vol. 2, pp , 996. [2] J. R. Bach, S. Paul, and R. Jan, A vsual nformaton management system for the nteractve retreval of faces, KnowData, vol. 5, no. 4, pp , August 993. [3] Y. L, B. Tao, S. Ke, W. Wolf, Semantc mage retreval through human subect segmentaton and characterzaton, Proc. SPIE Storage and Retreval for Image and Vdeo Databases, vol. 3022, pp , 997. [4] A. Valaya, A. Jan, and H. J. Zhang, On mage classfcaton: cty versus landscape, Proc. IEEE Workshop Content-based Access of Image and Vdeo Lbrares, pp. 3-8, June 988. [5] M. Szummer and R. Pcard, Indoor-outdoor classfcaton, Proc. Int. Workshop on Content-Based Access of Image and Vdeo Databases, pp. 42-5, Bombay, Inda, Jan [6] W. Zhu, and T. Syeda-Mahmood, Image organzaton and retreval usng a flexble shape model, Proc. Int. Workshop on Content Based Access of Image and Vdeo Databases, pp..3-39, Bombay, Inda, Jan [7] J. Z. Wang, J. L, and G. Wederhold, SIMPLIcty: Semantcs-senstve ntegrated matchng for pcture lbrares, IEEE Trans. Pattern Analyss and Machne Intellgence, vol. 23, no. 9, Sept. 200 [8] B. Rogowtz, T. Frese, J. Smth, C. A. Bouman, and E. Kaln, Perceptual mage smlarty experments, n Proc. of SPIE, 997. [9] R. Duda, and P. Hart, Pattern classfcaton and scene analyss, John Wley & Sons, New York, NY, 973. [20] A. Moslovc, A method for color namng and descrbng the color compston of an mage, to appear n IEEE Trans. Image Processng. [2] D.Comancu, P.Meer, Mean shft analyss and applcatons, IEEE Int. Conf. Comp. Vs., pp , Greece, 999. [22] K. Kelly, and D. Judd, The ISCC-NBS color names dctonary and the unversal color language, NBS Crcular 553, Nov., 955. [23] J. Gomes, and A. Moslovc, "A varatonal approach to recoverng a manfold from sample ponts", Proc. European Conf. Computer Vson, ECCV 2002, Copenhagen, May Aleksandra (Saška) Moslovć (M 92) was born n Belgrade, Yugoslava, n 968. She receved her Ph.D. n Electrcal Engneerng, from the Unversty of Belgrade, Yugoslava, n 997, and snce then has worked at IBM Research (2000-present), Bell Laboratores ( ) and as a Faculty Member Unversty of Belgrade ( ). Her research nterests nclude multdmensonal sgnal processng, pattern recognton, modelng and human percepton. In 200 Dr. Moslovć receved the Young Author Best Paper Award from the IEEE Sgnal Processng Socety. Dr. Moslovć s a member of the IEEE Multdmensonal Sgnal Processng Techncal Commttee and an Assocate Edtor for the IEEE Transactons on Image Processng. (a) (b) (c) Fg. 2: Images from Set 3 categorzed as: a) outdoor archtecture, b) ndoor scene wth man-made obects, and c) landscape wth human nfluence. (a) (b) (c) Fg. 3: Images beng ncorrectly categorzed as: a) ndoor scene wth man-made obects, b) man-made obect outdoors, and c) man-made obects outdoors. (a) (b) (c) Fg. 4: Images whch were assgned nto one of the related categores; a) green landscape, b) landscape wth water, c) man-made obects outdoors.

9 MM (a) (b) (c) Fg. 5: Examples of multple categorzaton. Images are categorzed as: a) ctyscape + water scene wth human nfluence + outdoor archtecture, b) landscape wth mountans + landscape wth felds and folage, and c) landscapes wth human nfluence + man-made obects outdoors. Fg. 8: An example of mage categorzaton wth the automatcally generated features. The mages belong to the category Outdoor archtecture. The sze of the database s 3,506 mages. Fg. 6: An mage retreval example wth the less-strct metrc. The mage n the top left corner s the query, followed by the most smlar mages. Fg. 9: An mage retreval example usng the strct metrc and automatcally computed features. The frst mage on the left s the query (from category Obects ndoors ) followed by the best matches. Set Set 2 Total TABLE I: THE RESULTS OF SEMANTIC CATEGORIZATION Correctly Mslabeled Classfed Not labeled nto related labeled Images category Fg. 7: An example of mage categorzaton wth the automatcally generated features. The mages belong to the category Waterscapes. c2a c2b c c6 c7 c7 c4 c3 c3 c5 c9 c9 c5 c4 c8 c8 c20 c c2 c6 c0 Fg. : Result of the HCA and the fnal set of clusters.