Content-based Image Retrieval

Transcription

1 Content-based Image Retrieval Allan Hanbury Pattern Recognition and Image Processing Group Institute of Computer-Aided Automation TU Vienna, Favoritenstr. 9/1832 A-1040 Vienna, Austria Contents Understanding of Content First-generation image retrieval systems New-generation image retrieval systems Current research on general image understanding 2 1

2 Image Content 3 levels: Perceptual: Mostly Mauve, with some green and yellow Green and yellow frame Semantic: a pub sign Impressions, Emotions and Meaning: One can get a beer there! 3 Another Example: Perceptual: Concentric circles Yellow, red, black Yellow surrounded by red Semantic: A Scene from the film 2001 A Space Odyssey A HAL 9000 Computer Impressions, Emotions and Meaning: Advanced technology Fear of advanced technology 4 2

3 Understanding of content The word content often has different meanings in the computer science and humanities disciplines: Computer scientists use content to mean perceptual properties. In the humanities, content normally refers to meaning. Result some disappointment with content-based image retrieval systems! 5 First-generation image retrieval systems Images or videos in a database are annotated manually with keywords or descriptive text. Text-searching is done on these annotations. 7 3

4 Relevance feedback Find image with landscape Query Query by by text text Visualisation Search Search Engine Engine Indexing Annotation (manual) Off line On line 8 From del Bimbo Text Annotation Is more difficult than it appears. Can be done as: Free-text annotation E.g. Fishing vessels in Northern Ireland Keyword annotation Free choice of keywords Controlled Vocabulary Ontology In its simplest form, a set of keywords with a hierarchical structure, but there exist more complicated relations. Example from the Iconclass ontology ( definitions in total) 9 4

5 Keyword Annotation The GettyImages Archive ( has a list of keywords used for annotating their image archive. Examples: Life Events: Childbirth, Recruitment, Working Overtime, Office Romance, Wedding, Retirement, Lifestyles, Social Issues, Age Contrast, Concepts: Friendship, Funky, Futility, Gambling, Gratitude, Greed, Growth, Individual People: One Person, One Man Only, One Woman Only, One Baby Boy Only, One Baby Girl Only, One Boy Only, One Teenage Boy Only, See this document for many more keywords: 10 Leaving image annotation to unknown people without a restricted keyword list can be a bad idea. The GIMP-Savvy Photo archive ( allows this. Here is an example of one of the annotations: 14 5

6 15 Annotating with a game The ESP-game is an interesting attempt to make manual annotation fun. Author prediction: 5000 people continuously playing the game could assign a label to all images indexed by Google in 31 days. 16 6

7 Information associated with images or videos Content-independent metadata: e.g. format, author s name, date, location, cost of filming, etc. Content-dependent metadata: colour, texture, shape, spatial relationship, motion, etc. Content-descriptive metadata: relation of the image to the real world, e.g. Objects in the image: car, person, tiger, Emotions: happy, sad, Meaning: 17 Current Internet image search technology The image search techniques used by Google, Yahoo, etc. are basically text searches. The algorithms look for the search text in Names of the image files. The text around images in web pages. This can work well, but only when the image is well described. 18 7

8 sunset Simple search term 19 otto wagner A more difficult search 20 8

9 two green bottles Semantic search 21 Why this image? From

10 How to get your images listed high in Google image search Use a descriptive and keyword rich file name Use seven to eight keywords in your title and alt text Place a short one line description of the image directly below the image Wrap the content around the image Try to put the image closer to the top of the page By Amit Agarwal 23 How good is Google? Test done by Fergus et al. (Jan. 2003) 10 search words tested. All images returned were classified as Good, Intermediate or Junk

11 25 New-generation image retrieval systems Use keywords as well as information extracted directly from the image. Make use of techniques from: image processing pattern recognition Tend to work on the perceptual level of image content

12 Find image with landscape Relevance feedback Query Query by by text text and/or and/or Query Query by by visual visual example Visualisation Feature Feature extraction (automatic) Annotation (manual) Search Search Engine Engine Indexing Off line On line 27 Adapted from del Bimbo Retrieval of images by perceptual properties Also called Query by visual example. Examples can be authored by the user or extracted from image samples. The system checks the similarity between the visual content of the user s query and database images. Relevance feedback is often employed to allow the user to improve the images returned

13 The following perceptual properties are often used in query by example: by colour by shape Search Search Engine Engine by texture by spatial relationships 29 Different types of query by visual example Query by image Query by painting Query by sketch 30 13

14 Query by image The user gives an example image and the system tries to choose the images that are closest to this image in perceptual properties. Different systems use different combinations of perceptual properties. Sometimes the user has the possibility to set the importance of colour, texture, etc. IKONA System, INRIA 31 Query by painting The user draws the colours to be retrieved and their spatial arrangement. Picasso System, University of Florence Link to QBIC 32 14

15 Query by sketch The user sketches the shape of the object to be found. Picasso System, University of Florence 33 Extracting the perceptual properties (features) of an image Perceptual properties are usually represented as lists of numerical values (feature vectors). These values are calculated by feature extraction algorithms. Colour feature extraction algorithm (e.g. colour histogram) Colour feature vector: [1.3, -51.9, 308.0, 19.7, ] Texture feature extraction algorithm (e.g. co-occurrence matrix) Texture feature vector: [-6.3, 0.2, 99.3, -53.8, ] 35 15

16 Comparing the perceptual properties of images After the feature extraction stage, all images in a database have feature vectors associated with them. The images are compared by comparing the feature vectors. For example, distances or similarities between the feature vectors can be calculated feature vectors: [1, 2] [2, 2] [5, 6] Colour histograms An 8-bit RGB colour image contains = colours. This would make a very long histogram! Instead, full RGB colour histograms are usually represented using the RGB cube 37 16

17 As there are three colour components, they form a threedimensional RGB colour space. RGB Cube software by P. Colantoni (

18 40 Histograms produced by: Colorspace software, P. Colantoni,

19 Colour histograms in CBIR Colour histograms are a good representation of the colours present in an image. However, comparing histograms of colours is computationally expensive. The image colours are therefore usually quantised, meaning that the number of colours is reduced, often to 64 or 256 colours. 42 Original 64 colours Colour quantisation 11 colours 64 colours 64 colours 64 colours From: R. Schettini, G. Ciocca, S. Zuffi, in Color Imaging Science: Exploiting Digital Media, John Wiley,

20 The colour histograms are then similar to the greyscale histograms. Except that each bin represents a colour and not a greylevel. From A. Del Bimbo, Visual Information Retrieval 44 Further example: 45 20

21 Histogram comparison 46 From Schiele, Mikolajczyk, Leibe, Computer Vision Lecture 2005 The histograms can be compared using histogram intersection. The similarity between two (normalised) histograms H und H' with the same number of bins is: (, ) min ( H, H ) S H H = i where i is the bin index. Similarity of 1 means that the histograms are identical. i i 47 21

22 Example: Histogram Histogram Min =0.2 Min =0 Min =0.2 Min =0.1 Min =0.1 S =

23 For comparison: Histogram S = Histogram Disadvantages of colour histograms Different images could have the same histograms. A small colour difference can have a big influence on the colour similarity measure. The influence of the lighting is not taken into account 51 Images from Schettini et al. 23

24 Neon light Daylight Tungsten light 52 From Improving these systems to search on the semantic level Research is now underway to allow better searching on the semantic level of image content. This makes use of techniques from computer vision (image understanding). A hot topic of research is the automatic annotation of images by keywords

25 Two gaps to cross There are two gaps which cause difficulties in CBIR [Smeulders et al., 2000]: Semantic gap: The user seeks semantic similarity, but the database can only provide similarity by data processing. Sensory gap: The gap between the properties in an image and the properties of the object plays a limiting role in retrieving the content of the image. 2D image of a 3D scene. Lighting conditions often unknown. Occlusion. From 54 Relation to narrow and broad image domains A narrow domain has a limited and predictable variability in all relevant aspects of its appearance. [Smeulders et al., 2000] There is very little change in the content of the images. Usually the recording circumstances are similar for all images. Examples: Images of lithographs recorded under white light with frontal view. Images of frontal views of faces recorded on a clear background

26 A broad domain has an unlimited and unpredictable variability in its appearance even for the same semantic meaning. [Smeulders et al., 2000] The contents of the images are unconstrained. Examples: Archives of stock photos All the images in the internet. Often, image domains lie between these extremes. Examples: Pictures of faces in a crowd. Family photographs for one family. In general, image understanding and CBIR techniques tend to work better for narrower image domains. 56 The long-term aim of current research is to: Close the semantic gap by as much as possible. Hence to develop a general purpose image understanding algorithm. Given any image, this algorithm should describe the contents and possibly the relations between them. This is needed in order to be able to intelligently search large libraries of images and videos

27 There are two main approaches to image understanding being investigated at the moment: Segmentation-based approaches Interest point-based approaches 58 Segmentation-based approaches [0.8, 50.8, ] Based on the following steps: Automatic segmentation of the image into meaningful regions. Calculation of the perceptual feature vectors for each region. Classification of the regions as different objects based on the feature vectors (pattern recognition). [0.9, 51.1, ] [0.4, 10.0, ] [0.9, 50.2, ] 59 27

28 60 from K. Barnard et al. (JMLR 2003) Results using the segmentation-based approach from P. Carbonetto et al. (ECCV 2004) 61 28

29 Interest point-based approaches A set of interest points is extracted from the image by an algorithm. Many algorithms are available to do this. From Bishop ICPR talk 62 To be useful, these features should be invariant to: Image scaling Image rotation Change in illumination Change in 3D camera viewpoint. The same features will then be found in another image of the same object. 63 From D. Lowe, CVPR03 tutorial 29

30 Example of interest point-based object recognition The aim is to discriminate between cows and sheep. Training phase: Each of a set of training images is labelled as cow or sheep. Interest points are extracted. Colour and texture features at each interest point are calculated. A pattern recognition algorithm is trained to recognise the features corresponding to cow, sheep or other. Usage: Interest points are extracted from an image. Colour and texture features at each interest point are calculated. Each interest point is classified as a cow- or sheep-like feature. Based on these local classifications, the whole image is classified as cow or sheep. Work by C. Bishop (2004) 64 From Bishop ICPR talk 65 30

31 From Bishop ICPR talk 66 From Bishop ICPR talk 67 31

32 From Bishop ICPR talk 68 Example of a system for automatic image annotation The a-lip system (Li and Wang, 2003): Training phase: Uses a training set of images labelled with keywords. divides the images into 4 4 pixel blocks, and extracts 3 colour and 3 texture features from each block. A pattern recognition algorithm learns which keywords are associated with which feature values. Usage: The same features are extracted from each image to be annotated. The trained pattern recognition algorithm associates keywords with these features

33 a-lip system 70 Using a priori information Use common knowledge about the world to simplify the image labelling. Example: Kutics et al. image annotation system (2003) Segments an image. Extracts colour, texture and shape features from the segmented image. Associates words with these features Blue, black, green (colour) Dotted, smooth (texture) Circular, elongated (shape) Expands on the annotation by using an ontology

34 Kutics et al. system 72 Using Google to get extra information Idea of Grefenstette et al. (2006) x x banana x banana banana is x OR bananas are x blue green red orange yellow

35 Summary Current CBIR systems tend to retrieve based on perceptual features. Systems which can automatically extract semantic information from images would be very useful. Research on general image understanding systems for achieving this is underway. The EU MUSCLE Network of Excellence is working on this topic

36 Content-based Image and Video Retrieval Links Image Links Image-Seeker LTU Technologies, France: QBIC colour and layout searches at the Hermitage Museum in St. Petersburg, Russia - IKONA INRIA Imedia, France: PicToSeek - ISIS Group University of Amsterdam: Pic2Seek - ISIS Group University of Amsterdam: Marvel IBM: Blobworld University of California, Berkeley: (the last time I tried the demo, it didn t work, but there are some pages showing results of sample queries) SIMPLIcity James Z. Wang Group - Pennsylvania State University: a-lip (Automatic Linguistic Indexing of Pictures) James Z. Wang Group - Pennsylvania State University: Video Links Video Google Visual Geometry Group, Oxford University: MPEG-7 Annotation Tool IBM: (tool for annotating the visual content of videos available for download) Multimodal Annotation Tool IBM: (tool for annotating both the visual and audio content of videos available for download)