CBIR. content-based image retrieval

Transcription

1 CBIR content-based image retrieval

2 Problems with Image Retrieval A picture is worth a thousand words The meaning of an image is highly individual and subjective

3 What is the topic of this image? What are right keywords to index this image? What words would you use to retrieve this image?

4 CBIR Art Collections e.g. Fine Arts Museum SF, Hermitage Medical Image Databases CT, MRI, Ultrasound, The Visible Human Scientific Databases e.g. Earth Sciences General Image Collections Corbis, Getty Images The World Wide Web

5 Two Classes of CBIR Narrow vs. Broad Domain Narrow - špecifický Medical Imagery Retrieval Finger Print Retrieval Satellite Imagery Retrieval Broad - všeobecný Photo Collections Internet

6 Challenges Semantic gap The semantic gap is the lack of coincidence between the information that one can extract from the visual data and the interpretation that the same data have for a user in a given situation. User seeks semantic similarity, but the database can only provide similarity by data processing. Huge amount of objects to search among. Incomplete query specification. Incomplete image description.

7 What is a query? an image you already have a rough sketch you draw a symbolic description of what you want e.g. an image of a woman with a child

8 Aims of an user in CBIR browsing through a large set of images from unspecified sources category search retrieve arbitrary image representative of some class target search - search for a precise copy of an image (copyright protection) search for a picture to go with a broad story or to illustrate a document

9 On the right a block of text from Moby Dick. This text is processed to obtain nouns, verbs, adjectives and adverbs and the terms are disambiguated by a voting process. The resulting text is used as a query to Barnard et al. s joint probability model, where the search returns images that have high joint probability with the collection of words. On the left, the images returned by this query. The query appears to be very successful (among other things, there s a picture of a whaleboat with sailors in it harpooning a whale).

10 What is CBIR? Query Image Retrieved Images Image Database Feature Space Image Features Similarity Assessment

11

12 History of Image Retrieval Traditional text-based image search engines Manual annotation of images Use text-based retrieval methods E.g. Flowers in a pond Water lilies Nymphaea Caerulea

13 Limitations of text-based approach Problem of image annotation Large volumes of databases Valid only for one language Problem of human perception Subjectivity of human perception Too much responsibility on the end-user Problem of deeper (abstract) needs Queries that cannot be described at all

14 Text annotation Free-text annotation E.g. Fishing vessels in Northern Ireland Keyword annotation Free choice of keywords Controlled Vocabulary Ontology set of keywords with a hierarchical structure Example from the Iconclass ontology ( definitions in total)

15 Image Retrieval as Text Retrieval Three Ways to go Manually Assign Keywords to each image Use text associated with the images (captions, web pages) Analyse the image content to automatically assign keywords

16 Manual Keywording Expensive Can only really be justified for high value collections advertising Unreliable Do the indexers and searchers see the images in the same way? Feasible

17 Cheap Powerful Associated Text

18 Possible Sources of Associated text Filenames Anchor Text Web Page Text around the anchor/where the image is embedded <meta name="description" content="jun 17, Two green bottles, sitting on a wall..."/> <img src=" alt="two green bottles, sitting on a wall..." width="600" height="800"> <p>dsc01822.jpg</p> <p>two green bottles, sitting on a wall...</p>

19 Automatic Keyword Assignment A form of Content Based Image Retrieval Learn a mapping from the low level image features to the words or concepts Find that category of images to which a keyword applies

20 Similarity The term similarity has different meaning for different people. Even the same person uses different similarity measures in different situations.

21 How similar are these two images

22 image similarity depends on distance measure & features 1. color similarity 2. texture similarity 3. shape similarity 4. object and relationship similarity

23 Variety of Similarity Degree of difficulty Similar color distribution Similar texture pattern Similar shape/pattern Histogram matching Texture analysis Image Segmentation, Pattern recognition Similar real content Eternal goal :-)

24 Images containing similar colors

25 Images containing similar shape

26 Images containing similar content

27 Similarity Similarity measure S Q,D between images S Q,D = s(f Q,F D ) = g( d(f Q,F D ) ) d metric/distance

28 1. Euclidean Distance (L2) 2. City Block Distance 3. Canberra Metric 4. Histogram Intersection 5. Jeffrey Divergence 6. Bhattacharyya Distance 7. Chi-Square 8. Bray Curtis Distance 9. Angular Separation Distance 10. Chord Distance 11. Non-Correlation 12. Matusita Distance 13. Soergel 14. Wave Hedges 15. WED Distance 16. Kolmogorov-Smirnov Statistic 17. Kuiper 18. Mean Distance

29 Color similarity color percentages the user chooses colors from a color table and indicates the desired percentage of each color particular placement of the colors within the image is not a factor in the search

30 Results of a QBIC search based on color percentages; the query specified 40% red, 30% yellow, and 10% black

31 The intersection of normalized histograms Euclidean distance

32 Scaled L1-norm distance Scaled L2-norm distance Scaled Histogram Intersection Histogram Quadratic Distance where a ij is a similarity matrix between colors i and j and dij is the Euclidean distance between two colors i,j

33

34 Color layout - grid choose colors grid squares (from a table) grid square color distance measure d color compares each grid square of the query to the corresponding grid square of a potential matching image d I, Q) d gridded_co lor( color g ( C I ( g), C Q ( g)) The representation of the color in a grid square: 1. the mean color in the grid square 2. the mean and standard deviation of the color 3. a multi-bin histogram of the color

35 Results of an image database search in which the query is a painted grid. (Images from the MIT Media Lab VisTex database:

36 Texture similarity same spatial arrangements of colors, but not necessarily the same colors texture description vector (e.g. Haralick, Laws, Gabor, Fourier ) single texture images - texture description vector in an entire image more general images - texture description vectors are calculated at each pixel for a small (e.g. 15x15) neighborhood about that pixel the pixels are grouped by a clustering algorithm that assigns a unique label to each different texture category

37 Results of an image database search based on texture similarity (Images from the MIT Media Lab VisTex database)

38 Shape similarity Fourier descriptors {a 0, a M } Q - query shape; a Q - sequence of FDs I - image shape; a I - sequence of FDs M Q Fourier i i i M I d ( I, Q) a a 2

39 elastic matching (Del Bimbo 97) boundary matching technique the query shape is deformed to become as similar as possible to the image shape the distance between the query shape and the image shape depends on two components: 1) the energy required to deform the query shape into a shape that best matches the image shape 2) a measure of how well the deformed query actually matches the image.

40 Object and relationship similarity Spatial Relationships (Carson 99) once objects can be recognized, their spatial relationships can also be determined use color and texture to segment images into regions - objects or scene backgrounds objects that stand out well and have a particular color or texture pattern can be found in this way (as tigers and zebras)

41 Blobworld

42 Typically color is used Notes Texture has proved difficult for people to understand Shape possibly the same, and also user interface - most people can t draw! Alternatives include time and recently location (GPS Cameras) It appears people can take in and will inspect many more images than texts when searching

43 Query Space Display Besides just showing the images that match the query Images are placed in such a way that distances between images in the display reflect S Q,D

44 Navigational Approaches to Image Retrieval Layout images in a virtual space in an arrangement which will make some sense to the user Allow them to navigate around this projected space (scrolling, zooming in and out)

45 Demands to the navigation system Representation of the collection has a form of 2D vectors (as icons, points on the monitor) The set of images having higher level of similarity is displayed when bringing near the region The set of images having lower level of similarity is displayed when moving away from the current region

46 Clustering cluster image collection into the set of clusters construct 2-D projection of each cluster Examples of clusters

47 Clustering methods Hierarchical clustering Nonhierarchical clustering Single link Complete link Average link K-means Fuzzy clustering Kohonen neural networks (SOM) single link minimal distance between objects involved in clusters complete link maximal distance between objects involved in clusters

48 Projection methods Linear Nonlinear Principal component analysis (PCA) Classical MDS Sammon projection

49 Multi-dimensional scaling (MDS)

50 Metric MDS: Distances between data items are given, a configuration of points which gives rise to those distances is sought Objective function which is minimized: E M [ ( k, l) d( k, l)] k l 2

51 cold colors warm colors light colors MDS dark colors

52 Sammon s Mapping Closely related to metric MDS Tries to preserve pairwise distances Errors in distance preservation are normalized with the original distance Objective function: E S [ ( k, l) d( k, l)] k l ( k, l) 2 putting more importance on the small distances.

53 Nonmetric MDS: Only the rank order of the distances is important A monotonically increasing function that acts on the original distances: the rank order can be better preserved Normalized objective function: E N k l 1 [ ( (, )) (, )] 2 f k l d k l k l [ d( k, l)] 2

54

55 Relevance Feedback the user should be allowed to interact with the system to refine the results of a query until he/she is satisfied More Like this

56 User Interaction Query Space: Q = {I Q,F Q,S Q,Z Q } I Q - selection of images from the large image archive I F Q - selection of features from feature set F S Q - similarity function Z Q - set of labels to capture goal dependent semantics

57 Interacting with Query Space The process of query specification and display is iterated, where, in each step, the user revises the query. User feedback leads to an update of query space: Both positive and negative examples are used. Each iteration, the probability of being the target for an image in I Q is increased or decreased

58 GUI relevance feedback slider or checkbox

59 Precision P n r 2 Performance Measurement number of relevant retrieved images number of retrieved images Recall R n r 1 number of relevant retrieved images totalnumber of relevant images in DB The word relevant is the difficulty here to make these measurements, we need to know what items are relevant to a query. This is a question on which competent human informants can differ.

60 Precision Recall Graph Plot recall on horizontal axis; precision on vertical axis; and vary the threshold for making positive predictions

61 precision and recall The F1 Measure P = precision R = recall This is twice the harmonic mean of P and R. We can plot F1 as a function of the classification threshold θ

62 Good System vs. Bad System A good system should have high level descriptions of image contents. A good system should cover a wide range of users requirements.

63 ADL AltaVista Photofinder Amore ASSERT BDLP Blobworld CANDID C-bird Chabot CBVQ DrawSearch Excalibur Visual RetrievalWare FIDS FIR FOCUS ImageFinder ImageMiner Related Works ImageRETRO ImageRover ImageScape Jacob LCPD MARS MetaSEEk MIR NETRA Photobook Picasso PicHunter PicToSeek QBIC Quicklook2 SIMBA SQUID Surfimage SYNAPSE TODAI VIR Image Engine VisualSEEk VP Image Retrieval System WebSEEk WebSeer WISE