What s wrong with existing

Transcription

1 Feature Article Integrate Browsing an Querying for Image Databases The image atabase system El Niño uses a new interaction moel that aims to overcome the problem of the semantic gap where the meaning that the user has in min for an image is at a higher semantic level than the features on which the atabase operates. To solve this problem, we replace the usual query paraigm with a more active exploration process an evelope an interface base on these premises. Simone Santini an Ramesh Jain University of California, San Diego What s wrong with existing image atabases? Most people woul agree that something is inee amiss, but they usually have a har time pinpointing the problems. When we starte working on image atabases at the Visual Computing Laboratory at the University of California, San Diego, we aske ourselves the same question. In the course of the last three years, we believe we ve starte to glimpse some of the answers, an we ve trie to incorporate the lessons we learne in our El Niño system. El Niño is the collective name of a group of search engines, interface tools, an communication an integration moules for managing image repositories. This article escribes the architecture of El Niño an its most characteristic aspects. We starte our work by analyzing the meaning of images an the role of semantics in the search process. In traitional atabases, the semantics of a particular recor is a function of its syntax an of the semantics of its atomic constituents. If the bits in the thir column of a certain relational table can be interprete as the number 90,000, then the employee to whom that recor correspons earns $90,000 per year. This assumption has been, more or less explicitly, extene to image atabases. The common outcome of using many of the existing image atabases is a mix of excitement an frustration. To unerstan the point, consier Figure 1, which is a typical example of a query an answer from a current image atabase. The query is the image in the top-left corner, an the similarity criterion inclue a mixture of global color, local color, eges, an texture that we selecte using four knobs in the interface. Some of these images are acceptable answers to the query, while others appear out of place. We can show, however, that most of them aren t out of place at all. The face in the top right-han image, for instance, is there because the structure an position of the woman s forehea resembles the arch above the oor in the query. The mile left-han image is there because the woman s vest is similar in color an position to the oor in the query, an so on. The atabase obviously mae some sense out of the similarity criterion that we selecte. Yet the results appear meaningless from the user s point of view. We can analyze the situation as follows. The user has some semantic specification in min ( I want to see images of ol oors ). He foun an example of an ol oor an concocte a similarity criterion that, accoring to him, capture a notion of similarity that coul inuce that semantic (the oor is blue, an it has a fairly well-efine structure). The atabase use this similarity to sort the images an returne the best results. The similarity inuce a semantics in the atabase images, but not the right one. We call this problem the semantic gap. Users have a fairly rich semantics in min when they start selecting a similarity criterion, but the atabase oesn t have the right tools to express it. Connecte to this, we have the problem of query refinement. Receiving an answer like that of Figure 1 is not ba at all, proviing that we know how to change the similarity criterion to improve it. In this sense, the atabase fails. It s not clear how to manipulate our four knobs to receive a better answer we on t know whether we shoul give more weight to color, less weight to texture, or whatever. In fact, our four knobs probably on t give us enough expressive power to let us efine the similarity criterion we woul nee. Interface an query specification In El Niño, the interface an the query specification moalities are strictly connecte an represent the most istinctive characteristics of the system X/00/$ IEEE

2 Figure 2. A Moigliani portrait place in a context that suggests painting. Figure 1. An example of a query from a firstgeneration image atabase. We starte our work by consiering the meaning of an image. The same image can mean a number of ifferent things epening on the particular circumstances of the query. Moreover, as a simple experiment can reveal, the meaning attribute by people to an image epens on the context in which the image is presente. For instance, showing people the images of Figure 2 an asking them to ientify the image in the mile (a man with a bear), the wor portrait appears more frequently than the wor face. The opposite happens if we ask people to name the central image in Figure 3. Context is essential for etermining the meaning of an image an for juging image similarity. Unfortunately, context isn t something that we can coify easily. Context is a set of social an cultural conventions that epen critically on our role as participants in a network of interactions. Since cognition, in its full sense, is inextricably tie to context, we can t expect any significant cognitive ability from a context-free atabase. Our solution is to let the user be constantly aware of the overall context in which the atabase is placing the images by isplaying a configuration of images. A configuration is a set of images isplaye in such a way that their mutual istance reflects their similarity as currently interprete by the atabase. Figures 2 an 3 can be taken as examples of configurations. Note that although the configurations contain in part the same Figure 3. A Moigliani portrait place in a context that suggests face. images, the meanings they convey are quite ifferent. In El Niño the minimal isplayable unit isn t a single image, but a configuration of images which, much like the configurations of Figures 2 an 3, gives the user information on what kin of similarity measure the atabase is using (since, as we ve seen, an image taken by itself oesn t convey any meaning). If we re looking for a portrait an the atabase shows us the configuration of Figure 2, we re on the right track. If the atabase shows us the configuration in Figure 3, there s something wrong with our similarity criterion. El Niño gives the user tools for exploring the cognitive map of the atabase. The user can pan, zoom the map, an apply other isplay operators such as fish-eye lenses. Browsing, as a supplement for querying, has alreay stimulate interest in the image atabase arena. 1-3 The isplay portion of our interface can be seen essentially as browsing the atabase cognitive map. Another important aspect of our interaction moel is the use of context feeback as the com- July September

3 Figure 4. A schematic escription of an interaction using a irect manipulation interface. Figure 5. Interaction involving the creation of visual concepts. IEEE MultiMeia (a) (c) (a) (c) (b) () (b) () munication channel between the user an the atabase. The user oesn t intervene irectly on the parameters of the istance measure, instea (as in relevance feeback 4 ) selecting a number of positive examples an a number of negative examples. In aition to the stanar relevance feeback, the user can place the examples in the relative positions that they shoul have. For instance, assume that a user is looking for pictures of people. Although all the pictures of people woul be marke as positive examples, the close-ups coul be clustere together to inicate that they shoul be consiere very similar, while the full-figure shots woul be groupe together in another part of the isplay. In other wors, rather than the user trying to unerstan the properties of the similarity measures use by the atabase, the atabase shoul use the user s categorization to evelop new similarity measures. Figure 4 shows a schematic of a user interaction with configuration feeback. In Figure 4a the atabase proposes a certain istribution of images (shown as colore rectangles), representing the current similarity criterion. For instance, the system consiers the green image similar to the orange one, an the brown image similar to the purple one. In Figure 4b the user move some images aroun to reflect his interpretation of the relevant similarities. Figure 4c shows the result. Accoring to the user, the re an green images resemble each other, while the brown image is quite ifferent from them. The images that the user selecte form anchors for etermining the new similarity criterion. Given the new position of the anchors, the atabase will reefine its similarity measure an change the configuration to that of Figure 4. In this case, the atabase consiers the re an the green "Concept" box images as quite similar, an the brown quite ifferent. Note that the result isn t a simple rearrangement of the images in the interface. For practical reasons, an interface can t present more than a small fraction of the images in the atabase. Typically, our system isplays the 100 to 300 most relevant images. The reorganization involves the whole atabase. Figure 4 shows how some images will isappear from the isplay (the purple image in Figure 4a) an some will appear (the yellow, gray, an cyan images in Figure 4). This feeback mechanism constitutes the funamental interaction moe between the user an the atabase, but other collateral tools can be attache to the interface. We introuce two visual concepts an visual ictionaries. A visual concept is a set of images that the user regars as equivalent (or almost equivalent). It serves as the vehicle of a specific semantic value in the context of the current query. These images can be collecte in a single unit that behaves in the interface like any other image (although it represents a more abstract unit than a single image). An interaction involving visual concepts looks like the schematic shown in Figure 5. Looking at the isplay of Figure 5a, the user still ecie to consier the re an green images as close to each other, but in aition to this, the user regars them as having enough semantic rel- 28

4 evance in the current context to eserve a special status as a concept. The user opens a concept box an rags the images insie the box. The box then serves as an icon to replace the images in the isplay space. A visual concept works much like a cluster of images kept very close in the isplay space, but in aition to this, ancillary information can be attache to the concept box as metaata. So, if the user of a museum creates a concept calle mannerist maonna, the wors mannerist an maonna can replace the actual images in a query. The secon tool the visual ictionary erives from the same esire to integrate textual an visual information. Attaching labels to a atabase suffers from two well-known rawbacks: 1. it s an expensive operation (which limits its usefulness to applications with high ae value) an Query Car Labele subset Dog Car Man House Tree Man Boat Dog Car Sun Tree House Car Tree Man Boat Sun Tree Partial answer Visual query Visual engine Complete atabase Final answer 2. it oesn t capture all the meanings of an image. In a visual ictionary, we label a subset of a atabase an use the results of the textual search as a starting point for the visual search. Figure 6 shows a visual ictionary s structure. Let s assume that a user is looking for some romantic images of ol cars on quiet country roas. We have a large atabase D of images an a subset A D that has been labele (or for which every image has some text attache). Note that A might not contain the images we re looking for an that its labeling might be too coarse for the semantics we re consiering (for example, images are rarely labele accoring to emotional connotations like romanticity ). On the other han, A will probably contain some examples of cars, an we ll be able to retrieve them as a (partial) match to the query ol cars on country roas. Although these cars aren t what we re looking for, we can use them as visual examples to start a visual query in the whole atabase D. The visual ictionary solves two major problems of text an visual atabases: 1. It overcomes the problems of labeling schemes. It s not necessary to label the whole atabase or to try to capture in text all the minutiae of an image. We on t expect a goo answer from the visual ictionary, but just enough examples to start a visual query. 2. The visual ictionary provies a convenient way to start a visual search. Apart from the iea of rawing a sketch of what you re looking for, there s no commonly accepte way of posing a query to a atabase. A visual ictionary is a tool for starting visual queries. In aition, we can easily integrate the visual ictionary with visual concepts. You can think of a visual concept as a ictionary entry that the user generates in the context of a query. This observation highlights the problem of a visual concept s scope an life span. They coul be limite to a session or to a user, or they coul be share among all users, thereby becoming a permanent entry in the visual ictionary. This is essentially a policy problem that s best resolve on an application-byapplication basis. Architecture El Niño is a collection of search engines connecte to a meiator that communicates with the user via a user interface (Figure 7, next page). In our current implementation, we wrote the interface in Java an mae it available through the Web, so the user runs it from within a Web browser. Each engine efines one or more feature spaces an a representation of images in each space. In aition, each engine measures similarity between images using a similarity criterion Figure 6. Schematic representation of a search involving a visual ictionary. July September

5 Local interface Operator Meiator The meiator in El Niño hanles three ifferent tasks: ispatching the queries to the right engines, managing the configuration feeback process, an managing the atabase name space. Meiator stub Remote connection Local connection Meiator Engine Engine Engine Database Database Name space Database Operator Operator Figure 7. The overall architecture of El Niño. Search engines are connecte to a meiator that communicates with users via a user interface. Text Color AND T Text engine Database s 1 (I, J ) Meiator C s 1 (I, J ) s 2 (I, J ) f (s 1, s 2 ) s 2 (I, J ) Color engine Database AND Figure 8. An example of the query ispatch process. Here the similarity criterion inclues text an color similarity. The AND operator joins the results of the two criteria. Query ispatch. The meiator primarily establishes a similarity criterion to etermine the similarity between two images. Figure 8 illustrates this process. The similarity criterion is specifie as a graph. The query graph on the left in Figure 8, for instance, establishes that the similarity criterion shoul inclue text (T) an color (C) similarity, an the results of the two criteria shoul be joine together using the AND operator. (We ll escribe the operator algebra of El Niño in the section Similarity algebra. ) The queries T an C can contain parameters for the configuration of the query engines. For instance, the query C coul inclue the information that color saturation shoul be isregare for this particular criterion. We ll iscuss a more sophisticate moel of engine configuration in the following section. We allocate two engines (a text an a color engine) to serve the query an use the query parameters to configure them. Whenever users ask to compute the istance between two images I an J in the atabase, the text engine returns a value s 1 (I, J), an the color engine returns a value s 2 (I, J). As specifie by the query graph, these two scores must be joine together using the AND function. For each image, given the values s 1 (I, J) an s 2 (I, J) the AND operator returns the value s(i, J) = f (s 1 (I, J), s 2 (I, J)) as the similarity between image I an image J with respect to the current similarity criterion. Once we set a similarity measure, El Niño allows two interrogation moalities a user can ask to return the first k images from a given reference point in the feature space (an the corresponing istances) or to compute the istance between two images in the atabase. The first moality woul suffice for an interface base on a simple browser that just isplays the images closest to the query. Displaying configurations requires knowlege of the istances between all pairs of images so that the images can be place in proper positions in the interface. Displaying a configuration entails (from the meiator point of view) the following steps: IEEE that can be aapte by changing the values of a number of parameters. 1. Establish a criterion, either by parameters set in the criterion graph or as a result of configuration feeback (see below). 2. Issue a query to retrieve the first k images clos- 30

6 est to a referent. (In our geometric approach, the feature space is transforme so that the referent is always the origin, therefore there s no nee to explicitly efine it as part of the query process (see the section Search engines ). These images form the isplay set D. 3. Request the istances between all pairs of images in the set D via the istance moality. The interface uses these istances for etermining the images positions in the configuration. Configuration feeback. The main moality of interaction between the interface an the engines is configuration feeback. The user selects a number of images on the screen an places them in a position that for the purpose of the current query reflects their mutual similarity. The similarity engines use this information to aapt their similarity criterion. For each engine, the istance function epens on a parameter vector w, which must be change to aapt the similarity criterion in a way that reflects the user requirements. This requires an optimization process, an El Niño recognizes two types of engines: self-aapting an non-self-aapting. Self-aapting engines contain an internal optimizer an run their own optimization process (Figure 9). The meiator sens the engines the ID of the images that the user has selecte an a table with the require istances between these images. The engine runs its own optimization process an etermines the optimal value of the vector w. This operational moe minimizes the loa on the meiator an network communication, but requires the presence of an optimization algorithm in the engine. If an engine oesn t have an internal optimization proceure, the meiator hanles the optimization (Figure 10). In this case, the engine must provie proceures for reaing an upating the parameter vector w an for etermining the istance between two images given a parameter vector. The meiator will run its own optimizer an communicate to the engine the new values of the vector w. The optimization process resulting from a configuration feeback is in general largely unconstraine. A user typically selects at most a ozen images for feeback, resulting in at most some 60 istance constraints. On the other han, the similarity measure can easily epen on more than 100 parameters. We nee to constrain the optimization problem to avoi overfitting, which interpolates the ata but fails to capture the tren. w 1 w 2 w 3 w 4 w 5 w 6 h 1 h 2 h 3 h 4 h 5 h 6 h 7 h 8 h 9 Distance ata from the meiator We re looking for a significant global change in the istance measure, not pathological aaptation of single parameters. We tackle this problem using the two concepts of space curvature an natural istance. For each feature space in every engine we efine a natural istance. Formally, the efinition of the natural istance is arbitrary, but intuitively it correspons to a somewhat neutral istance isotropic an uniform over the space. This istance etermines what we conventionally call a zero curvature space. (This is just a conventional esignation. The actual Gaussian curvature of the space might not be zero.) When the istance parameters change, the istance Engine Similarity measure Parameters vector Optimizer Figure 9. Aaptation of the similarity measure in a self-optimizing engine. h 1 h 2 h 3 h 4 h 5 h 6 h 7 h 8 h 9 Optimizer Meiator Cost Dist Vector write Vector rea Engine Similarity measure w 1 w 2 w 3 w 4 w 5 w 6 Parameters vector Figure 10. Meiatorassiste aaptation of the similarity measure in a non-self-optimizing engine. July September

7 Local thumbnail Web server Thumbnail archive Insert (URL) Engine Feature extractor Figure 11. Insertion of a new image in El Niño. h Meiator Insertion request (URL) (h, f ) Database (h, URL) Insertion request (URL, h) Engine f Feature extractor Hanle URL Database Web site with the original image Location table (h, f ) function changes too, an this results in a curvature in the feature space. We associate the curvature etermine by the parameters w with a cost F c (w, w 0 ), where w 0 is the parameter vector corresponing to the natural istance. If the configuration entere by the user is characterize by the istances = [ 12, 23,, h 1, h ] (where ij is the requeste istance between the ith an the jth images in the configuration), an if the parameters w inuce a istance vector (w) between the same images, then we measure the mismatch by a cost F ( (w) ). The optimizer minimizes the criterion: the engines initiates the insertion process. That engine s aministrator communicates the URL of the esire image to the engine. In turn, the engine issues an insertion request to the meiator that contains the image s URL. If the meiator grants the insertion, it generates a new hanle an communicates it to the engine, thus authorizing the insertion. Currently, the meiator oesn t perform any check to guarantee that the images being inserte aren t alreay present in the system beyon checking for uplicate URLs. When the meiator authorizes the request, the engine ownloas the image, extracts the proper features, an stores it into its internal inex. At the same time, the meiator issues an insertion request to all other engines in the system, specifying the URL of the image an the hanle assigne to it. Each engine will ownloa the image, encoe it, an store it in its own inex. Finally, the meiator inserts the image s hanle an URL in a location table so that the location can be mae available to the user. Optionally, the meiator can ownloa the image an create a thumbnail version in a local archive. A Web server running on the same machine makes the thumbnails available to the interface through the Web. Search engines El Niño s architecture buils on the iea of efining ifferent search engines for the ifferent nees of the system an integrating them using the meiator s operator algebra (which we ll introuce in the section Similarity algebra ). In this section we present the search engines we ve implemente so far as part of our system. El Niño currently has three engines: ( ( ) ) + c( 1 0) F ' w F w, w In other wors, the optimizer tries to fin a solution that places the images in the configuration at the right istance an at the same time eviates as little as possible from the natural istance function. a specialize features engine that uses a combination of stanar image analysis techniques, an image ecomposition engine that uses a wavelet ecomposition generate by a transformation group to represent images, an IEEE MultiMeia Database name space. When a user inserts an image into the atabase, the meiator assigns each image a system-wie image hanle. Typically, all the engines contain the same images. (This is not a strict requirement, but images not represente in some engines may fail to respon properly to queries involving those engines.) Figure 11 shows the insertion process for a simple system containing only two engines. One of a textual engine that implements the visual ictionary. Specialize features engine We evelope the specialize features engine using pieces of stanar image analysis software as a reference for measuring the performance of other engines that we evelope or will evelop in the future. 32

8 The specialize features engine uses three types of features representing color, structure, an texture, which are the stanar choices of many existing atabases. (Part of our interest in eveloping a specialize features engine was to see how our interface coul improve the performance of these stanar techniques.) Feature histograms represent color an structure, while the mean an variance of local features represent texture. Each feature is partially localize by iviing the images in a fixe number R of rectangles an computing the features separately for each rectangle. Figure 12 shows how the specialize features engine s feature vector is organize. The feature vector contains three composite feature vectors one each for color, structure, an texture. Each composite feature vector contains R local feature vectors one for each of the rectangles in which the image is ivie. Each local feature contains a feature extractor. The three types of features (color, structure, an texture) are extracte by each of the local feature extractors on each of the R rectangles as follows: Color local features Color composite feature Feature vector Shape composite feature Shape local features Texture composite feature Texture local features Color. Pixels are represente in the HSV color space, an each imension in the color space is represente as a separate single imensional histogram. Rather than iviing HSV into bins an computing the histogram, we compute the first three moments of the statistical istribution of the histogram irectly from the pixel ata, following the technique evelope by Stricker an Orengo. 5 Every local color feature consists of nine numbers three statistical moments for each of the three color channels. Structure. A histogram of ege irections represents structure. Eight filters compute the strength of eges along eight ifferent irections. For each irection, a histogram of the strength of the eges along that irection is compute. As in the color case, we represent each histogram with its first three statistical moments. Each local structure vector is therefore represente as 24 numbers (three moments for each of the eight irections). Texture. We represente texture using Manjunath an Ma s Gabor features. 6 For every point in the rectangle, we apply 30 filters (six irections an five scales) an take the norm of the complex number resulting from the application of the filter. A texture is represente by the averages an variances of each of the components of the feature vector when varying across the rectangle, for a total of 60 numbers. For each local feature, we efine a istance function to compute the istance between two instances of that feature. In the specialize feature engine, the istance between two local feature vectors x an y is always a weighte Minkowski istance of the type ( x, y) = w x y i L i i i with p > 0. The weights w are set inepenently for each rectangle as a consequence of the configuration feeback process. The composite feature computes the istance between two vectors as a weighte sum of the istances between corresponing local features. Finally, the istances relative to the three composite features are weighte an put together in the complete feature vector using one of the AND operators efine in the section Similarity algebra. p 1 p Figure 12. The feature vector of the specialize feature engine containing three composite feature vectors one each for color, structure, an texture. July September

9 (a) (b) (c) Figure 13. Geoesics in the query space for (a) Eucliean space, (b) a space with one concentration of features, an (c) a space with two concentrations of features. IEEE MultiMeia Image ecomposition engine The image ecomposition engine buils on the iea of eriving a feature vector general enough so that, in principle, every conceivable similarity criterion can be expresse using it. Selecting a particular feature vector in general constrains the types of similarity we can represent. It s well known, for instance, that when using global histograms it s impossible to enforce any similarity criterion epening on the spatial istribution of features. To guarantee the generality of the features, we impose a reconstruction constraint an image s feature set must be general enough to allow (at least in principle) the reconstruction of the complete image. The feature space erives from a multiresolution ecomposition of the image generate by a iscrete subgroup of a suitable transformation group. The choice of transformation group is etermine by invariance consierations. 7 As an example, consier the use of the affine group to generate the transform. An element of the group is etermine by three parameters: the two spatial coorinates x an y, an the scale parameter s. In a color image, the value of a pixel is an element of C, the color manifol imbee in R 3. A pixel in an image is an element of the siximensional space G C, where G is the affine transformation group. We call this the image space. An image is a set of elements in this space. Both the transformation group an the color space can be enowe with a natural metric, which makes it possible to efine the istance between two coefficients. With the istance between two coefficients, it s possible to efine the istance between two sets (that is, between two images) in a stanar way. Defining the istance between two images as the istance between two sets helps us keep the imensionality of the space low an avoi many of the problems of the imensionality curse. Most multiresolution ecompositions will generate intolerably large numbers of coefficients. For instance, the ecomposition of a image generates about 21,000 coefficients, each represente by six numbers. Fortunately, this representation is highly reunant an, for a atabase application, we on t nee the complete reconstruction of the image. Since the image is represente as a set of coefficients in a metric space, it s natural to use vector quantization to reuce the number of coefficients necessary. We can represent images with as little as 50 coefficients an still obtain acceptable results, although usually we use 100 or 200 coefficients. Imposing a metric on the image space etermines the similarity criterion. Thanks to the generality of our feature representation, virtually all similarity criteria can be represente as a metric in the image space. A mixture of a Gaussian moel creates a probability istribution that contains most of the features of the positive examples an as little as possible of the features of the negative examples. We can then erive the metric of the image space from this istribution. As an illustration, Figure 13a shows a number of geoesics (lines of minimal istance) of a hypothetical 2D image space with no samples selecte. The image space is Eucliean, an the geoesics are straight lines. If the user selects a number of images with a concentration of features aroun the point (0.4, 0.4), the image space is istorte an its geoesics appear as that shown in Figure 13b. If the user selects a set of images with two concentrations of features aroun (0.2, 0.2) an (0.8, 0.8) the geoesics look like Figure 13c. More general spa- 34

10 tial istributions of the positive an One of toay's most famous irectors, negative examples generate more Stanley Kubrick complex istance measures. ie yesteray The geometry of the image space <IMG SRC="kubrick.jpg"> in his house just implicitly efines categories in El outsie Lonon Niño. The samples collecte by the user form the context from which conceptualization emerges. These categories on t rely on a preefine ontology, as is the case for simple schemes base on weighting istances compute on preefine features. Text engine The text engine uses some elementary techniques from information retrieval to associate labels to images an to retrieve images base on the labels. Label association occurs automatically whenever possible. In the case of Web images, we parse the page an collect the text surrouning the IMG tag from which the URL of the image was erive using a technique similar to that use by Smith an Chang in their WebSeek system. 8 The text goes through two ifferent processing stages (Figure 14). The first stage (common wors removal) removes all the wors so common in the language that they on t provie any information about the content of the image. We inclue articles, prepositions, an the like in this category. We also inclue time specifications (toay, yesteray, an so on), since they might not be vali when the images are retrieve from the atabase. The secon stage (stemming) attempts to extract the root of a wor. In this phase the system removes suffixes like the s of the plural form or the e of the past tense. Stemming is prone to error. Some wors have erivations with raically ifferent roots, like go an went, while wors with ifferent meanings can originate from the same root an iffer only in the suffix, like terrible an terrific. Information retrieval systems solve these problems with a-hoc rules. The current text engine in El Niño simply ignores them. We re planning to inclue a more sophisticate text search engine in future versions. Once the system has assigne keywors to the images in the visual ictionary, we use the technique of vector space similarity to etermine the match between the query an an image. Let I be an image with associate a set A of keywors. Every wor in A receives a weight 1/ A. Similarly, the query contains a set B of terms, each one with a weight 1/ B. The similarity between the query an image I is given by A B AB Common wors removal This measure has a simple geometric interpretation. Vectors in a (very high-imensional) space represent the lists of keywors with one axis for each wor in the ictionary. The vector corresponing to a list has component 1 along an axis if the corresponing keywor is part of the list, 0 otherwise. The previous similarity function equals the cosine of the angle between the image vector an the query vector. Similarity algebra The meiator contains a number of operators to combine the results of the ifferent engines. The operators take two similarity measures relative to two engines an transform them into a new similarity measure resulting from their combination. The operators efine an algebra in the space of istance functions. 9 These operators act on the istance functions the single engines efine. All the istance functions are efine in F F (where F is an appropriate feature space) an take values in [0, 1]. Consier a query Q, an let S c2 (Q,I) [0,1) be the similarity between the query Q an image I accoring to the criterion c 1 implemente by engine e 1. We can interpret this value as the truth value of the preicate Image I is like the query Q accoring to criterion e 1. Similarly, we have the value S c2 (Q,I) representing the similarity between Q an I accoring to criterion c 2. We make the following hypothesis. The similarity of Q an I with respect to criterion c 1 an c 2 epens only on the values s 1 (Q, I), s 2 (Q, I). In this case, we can write ( ) (an similarly, with a function for the OR cons Q, I s Q, I, s Q, I ( ) = ( ) ( ) c1 c2 c1 c2 famous irectors Stanley Kubrick ie house Lonon Stemming famous irector Stanley Kubrick eath house Lonon Figure 14. Text processing to create labels for the visual ictionary engine. July September

11 Figure 15. The interface of El Niño at the beginning of a search process. IEEE MultiMeia Table 1. Comparing similarity between the query an two images. s 1 s 2 I I Table 2. Average ( W q ) an variance (σ 2 )of the weighte isplacement for the 2 similarity algebra consiere. Min W q σ nective). 10 The function implements the AND operator while the function implements the OR operator. Similarly, we efine the NOT operator as s :2 (Q,I)=:(s c (Q,I)). Formally, in El Niño a istance in a function in L 2 [F F, R + ], which is a Hilbert space, an the operators form an algebra on this space. On this Hilbert space we efine the two operators: AND ( ), an OR ( ). The AND operator, for instance, has a signature: :L 2 [F F, R + ] L 2 [F F, R + ] L 2 [F F, R + ] an similarly for the OR operator. The NOT operator ( ) is efine on L 2 [F F, R + ]. ψ To allow query optimization, it s important that the two operations be istributive. If 1, 2, 3 are istance measures, then we shoul have an 1 ( 2 3 ) = ( 1 2 ) ( 1 3 ) 1 ( 2 3 ) = ( 1 2 ) ( 1 3 ) In stanar logic, the negation operator ( ) has three important properties: the e Morgan s theorems an the involutive property of the negation. These can be written as ( 1 2 ) = ( 1 ) ( 2 ) ( 1 2 ) = ( 1 ) ( 2 ) ( x) = x In the class of operators that we use, we can make one an only one of the three relations vali. The only operator efinitions that make it possible to satisfy all three relations are 1 2 = max ( 1, 2 ) 1 2 = min ( 1, 2 ) = 1 The min an max operators have the isavantage that for any value of 1, 2, ( 1, 2 ) an ( 1, 2 ) epen on only one of them. (The use of max an min is inverte with respect to the norm establishe, for instance, in fuzzy logic. The reason is that we are ealing with istances rather than similarities.) For instance, assume that we have two criteria of similarity, s 1 an s 2, an two images I 1, an I 2. The similarity between I 1, I 2, an the query accoring to the two criteria is given by Table 1. Intuitively, the similarity of the query with respect to s 1 an s 2 shoul be higher for image I 2, but if we use the min function, the value is the same for the two images. 36

12 Fagin 10 propose using two classes of ual functions that he calle norms an co-norms. Each norm has a ual co-norm (for example, min is a norm, an max is the ual conorm). Using a norm for the function, the ual co-norm for the function, an the NOT function above, the e Morgan theorems an involution are satisfie. Unfortunately, none of Fagin s functions (except for min an max) make an istribute. We obtain a class of operators that generates a istributive algebra of operators as follows. Consier a monotonically increasing function ψ : [0, 1) R such that ψ(0) = 0 an ψ(x) for x 1. The inverse ψ 1 is efine for all x 0 an takes values in [0, 1). The logic operators can then be efine as 1 2 = ψ 1 (ψ( 1 ) ψ( 2 )) 1 2 = ψ 1 (ψ( 1 ) + ψ( 2 )) Figure 16. One of our target images. Figure 17. Results of the search after selecting two images that resemble Figure 16. Note that the image marke with an x has been exclue from the search. This efinition enforces istributivity, but forces us to compromise about the negation operator. In particular, it s possible to choose the negation operator to satisfy exactly one of the properties above (that is, one of the two e Morgan s theorems or the involution property), but it s impossible to satisfy more than one. Whether these operators are sufficiently better than min an max to justify renouncing such important properties is essentially an empirical matter. We conucte a series of experiments to measure the agreement between the AND operator an the user intuitive notion of AND. Table 2 reports the weighte isplacement inex as efine in Narasimhalu, Kankanhalli, an Wu 11 (that is, the lower the number, the better the result). Analysis of variance reveals that the ifference is significant within a 5 percent possibility of error. A query example Here we present a brief query example of El Niño an its interface. Figure 15 shows the interface of El Niño with a configuration of ranom images isplaye. The user was aske to look for a certain group of car images that look like Figure 16 (not necessarily that particular one). In Figure 15 there are no cars like the one we re looking for, but there are a couple of cars. The user selecte one (most similar to the target) an marke another one with an x. The secon car image will etermine the similarity measure of the atabase, but it won t be consiere a query example (the atabase won t return images similar to that car); its value is in the establishment of a reference istance between the two car examples. Figure 17 shows the results after a few interactions (for the sake of clarity, from now on we only show the image space rather than the whole interface). Two images of the type that we re looking for have appeare. The user selecte them an place them close to one another. The user also place a thir car image at a certain istance but exclue it from the search process. Note that the selection process gets refine as we procee. During some of the previous interactions, the car July September

13 the istance measure irectly. Rather, interaction procees entirely through an augmente form of relevance feeback that we call configuration feeback. Figure 18. Further refinement of the search isplays more examples similar to Figure 16. Note that images marke with an x are farther away from the positive examples. IEEE MultiMeia image that s now exclue was selecte as a positive example because, relative to what was presente at the time, it was similar to our target. Now that we re zeroing in on the images we re actually intereste in, the re car is no longer similar to what we nee. Figure 18 shows the results of the next iteration. Now we have a number of examples of the images we re looking for. Further iterations (such as the selection represente in Figure 18) can be use to obtain more examples of that class of images. Note that the negative examples (marke with an x ) are place much farther away from the positive examples than in the previous case. This leas to a more iscriminating istance measure, which, in effect, helps the system focus on the class of images we re looking for. Conclusion El Niño attempts to overcome the limitation of traitional search engines an, more specifically, fins an alternative to the well-known query-byexample paraigm. The interaction moel of El Niño is base on the following two ieas: Context is essential for etermining the meaning of an image. Contextuality is enforce by showing the user a configuration of the feature space rather than a set of images. In a configuration, the images aren t just presente, but place in such a way that their mutual istance reflects their similarity as currently interprete by the atabase. The user never manipulates the parameters of The functional specifications of El Niño are base on results of user interaction with traitional image repositories. For instance, Markkula an Sormounen 12 stuie the search patterns of journalists looking for images to illustrate stories. El Niño aresses the following issues raise in Markula an Sormounen s research: Images are searche following several ifferent moels that are har to integrate. Markkula an Sormounen foun that about half of the images were searche by some very specific label ( I nee an image of Frank Zappa ). Others are searche by a more generic situation ( I want an image of a rock concert ). Other searches involve characteristics har to characterize with labels ( A quiet countrysie ). Users on t like clear-cut answers a certain amount of browsing is appreciate. 12 On the other han, browsing shoul not take too much time. When browsing is one manually (as in the case of most existing archives), users on t want to browse more than a few ozen images. These issues conitione the esign of El Niño at all levels. The most evient influence is in the esign of the user interface. Starting from an analysis of the meaning of images, we esigne an interface in which browsing an searching are inistinguishable an integrate. The meaning of images emerges from the interaction of the user with the atabase an is iscovere rather than encoe into the atabase. The same issues also conition our architectural esign for El Niño. Since we nee to eal with several query moels at the same time (ranging from visual browsing to keywor queries), we nee to place special emphasis on the integration of ifferent similarity engines. We base our architecture esign on a meiator that integrates ifferent engines an logic-like operators that put together their responses with this necessity in min. Future research will focus on a better integration between images an text. That is, we aim to esign the interface so that both images an wors can be represente as a configuration an be manipulate. MM 38

14 References 1. R.R. Korfhage an K.A. Olsen, Image Organization Using VIBE, a Visual Information Browsing Environment, Proc. Int l Society for Optical Engineering (SPIE), Vol (Digital Image Storage an Archiving Systems), SPIE Press, Bellingham, Wash., V. Chalana et al., Meplus: A Meical Image Analysis an Browsing Environment, Proc. SPIE, Vol (Meical Imaging), SPIE Press, Bellingham, Wash., R.L. Plante et al., Java, Image Browsing, an the NCSA Astronomy Digital Image Library, Proc. Astronomical Society of the Pacific Conference Series, Astronomical Society of the Pacific, San Francisco, Vol. 125, Y. Rui et al., Relevance Feeback: A Power Tool for Interactive Content-Base Image Retrieval, IEEE Trans. Circuits an Systems for Vieo Technology, Vol. 8, No. 5, 1998, pp M. Stricker an M. Orengo, Similarity of Color Images, Proc. SPIE, Vol (Storage an Retrieval of Image an Vieo Databases III), SPIE Press, Bellingham, Wash., Feb. 1995, pp B.S. Manjunath an W.Y. Ma, Texture Features for Browsing an Retrieval of Image Data, IEEE Trans. Pattern Analysis an Machine Intelligence, Vol. 18, No. 8, 1996, pp S. Santini, Explorations in Image Databases, PhD thesis, Dept. of Computer Science an Engineering, Univ. of California, San Diego, Jan J.R. Smith an S-F. Chang, Visually Searching the Web for Content, IEEE MultiMeia, Vol. 4, No. 3, July-Sept. 1997, pp S. Aali et al., A Multi-Similarity Algebra, Proc ACM SIGMOD Conf. on Management of Data, ACM Press, New York, 1998, pp R. Fagin, Combining Fuzzy Information from Multiple Systems, Proc. 15th ACM Symp. Principles of Database Systems, ACM Press, New York, 1996, pp A.D. Narasimhalu, M.S. Kankanhalli, an J. Wu, Benchmarking Multimeia Databases, Multimeia Tools an Applications, Vol. 4, No. 3, May 1997, pp M. Markkula an E. Sormounen, Searching for Photos Journalists Practices in Pictorial IR, The Challenge of Image Retrieval: A Workshop an Symposium on Image Retrieval, J.P. Eakins, D.J. Harper, an J. Jose, es., University of Northumbria at Newcastle, Feb Simone Santini is a project scientist at the Visual Computing Laboratory at the University of California, San Diego. His current research interests are moels of similarity perception, similarity-base search in image an vieo atabases, interactive visual information systems, an multiple perspective interactive vieo. He receive a Laurea egree in electrical engineering from the University of Florence, Italy in 1990, an an MS (1996) an a PhD (1998) in computer science from the University of California, San Diego. He is on the eitorial boar of the journal Multimeia Tools an Applications. Ramesh Jain is Presient an CEO of Praja in San Diego, an Professor Emeritus of electrical an computer engineering, an computer science an engineering at the University of California, San Diego. His current research interests are in multimeia information systems, image atabases, machine vision, an intelligent systems. He receive a BE from Nagpur University, Inia, in 1969 an a PhD from IIT, Kharagpur, Inia, in He is a fellow of AAAI an the Society of Photo-Optical Instrumentation Engineers, an a member of ACM, Pattern Recognition Society, an Manufacturing Engineers. He was founing eitor in chief of IEEE MultiMeia, an is currently on the eitorial boars of that publication, Machine Vision an Applications, Pattern Recognition, an Image an Vision Computing. Reaers may contact Santini at the University of California, San Diego, 9500 Gilman Dr., La Jolla, CA , ssantini@ece.ucs.eu. July September