Bayesian Approaches to Content-based Image Retrieval

Transcription

1 Bayesian Approaches to Content-based Image Retrieval Simon Wilson Georgios Stefanou Department of Statistics Trinity College Dublin

2 Background Content-based Image Retrieval Problem: searching for images with certain content in a large digital database; Canon (2002): digital images taken per year, and total multi-media data generation bytes/year. Search may be: 1 Specific: I want an image of Van Gogh s Sunflowers ; 2 Categorical: I want an image of a beach at sunset with palm trees on the left hand side, an image of the Taj Mahal, etc.; 3 Browsing: I m looking for a painting to hang in my living room. Images are capable of much more complex semantics than text; It is unrealistic to expect good search results from a simple text-based query.

3 Background Applications Entertainment (images to create an emotion etc.); Document preparation (incl. newspapers); Teaching; Graphical design and advertising; Medicine (looking for similar NMR images to a patient); Security (face recognition); More generally, multi-media data retrieval.

4 Background Text Annotation A simple way to search a database is if each image is annotated by text that describes it; This is common in many commercial databases (media organisations, art libraries, etc.); However this is far from a perfect solution: It s expensive (humans must annotate); It s subjective; It can t possibly describe all of an image s interpretations. even very good annotations will not be able to match many queries.

5 CBIR Systems Content-based Image Retrieval A better way to search is Content-based Image Retrieval. CBIR consists of two elements: 1 A feature extraction algorithm that describes the content of each image; 2 A retrieval algorithm that uses the features to retrieve images according to a query. Successful retrieval algorithms always work interactively with the user by a process called relevance feedback.

6 CBIR Systems Feature Extraction 1 A computer extracts features of an image, to do with colour, texture, location and shape of objects; These features (hopefully) describe well the content (or semantics) of the image; This can be done off-line and needs to be done only once; Searching the database is based on these features and a similarity measure between them; This is a decreasing function of a distance between their features.

7 CBIR Systems Feature Extraction 2 An image X is a matrix {X ij i = 1,..., n 1 ; j = 1,..., n 2 }; X ij is colour of pixel (i, j); colour is a 3-vector, for example in RGB-space X ij = (R ij, G ij, B ij ) {0,..., 255} 3 Feature vector of length d is f (X ) R d ; Distance between images X 1 and X 2 is d(x 1, X 2 ) = f (X 1 ) f (X 2 ) ; Similarity measure s(x 1, X 2 ) = exp( d(x 1, X 2 )) or d(x 1, X 2 ) 1, etc.

8 CBIR Systems Feature Extraction 3 Typical features: histograms of colours, autocorrelograms at different pixel distances, colour coherence vectors, locations directions lengths of edges, location shape colour of objects; However, good automatic object detection and image segmentation is difficult to achieve.

9 CBIR Systems The Semantic Gap Humans search for images using high-level meaning, such as what is happening in the scene, emotional reaction, presence of faces and objects; However computers understand images in terms of colour, autocorrelations between pixels, and crude measures of where objects are and their shape; Because of the vast scope of possible queries, and inadequacy of object recognition systems, we are restricted to using such low-level features; The difference between the two is called The Semantic Gap;

10 CBIR Systems Relevance Feedback An interactive process between user and system; Aids the retrieval process and attempts to bridge the semantic gap; CBIR system displays images and the user then judges them by marking one or more that he/she perceives to be really relevant; The system then updates its retrieval in light of this information; More sophisticated feedback involves: marking objects in the image, marking images as not relevant, etc.

11 CBIR Systems The CBIR System 1 Initial set of N D images from the database is displayed: These may be randomly chosen; More sophisticated systems allow the user to draw a simple representation of what is wanted, then N D most similar images in database are displayed. 2 Repeat until image found: User chooses one or more image from display set that are most relevant to query; System updates and displays a new set of N D images in light of this information.

12 Bayesian Approach Bayesian CBIR PicHunter Developed by Cox et. al. (2000); Uses the simple case of target search and one image per display chosen; Assume database X = {X 1,..., X N } of images, one of which is target T ; At t-th iteration, system displays a set D t X and user selects A t D t as most relevant; Let H t = {D 1, A 1,..., D t, A t }; Objective: compute P(T H t ), T X.

13 Bayesian Approach Bayesian CBIR 2 We have P(T H t ) P(H t T ) P(T ) { t } = P(A i, D i T, H i 1 ) P(T ) = i=1 { t } P(A i D i, T ) P(T ); i=1 P(A i D i, T ) is the likelihood of the user picking A i from D i, were the target T to be known.

14 Bayesian Approach Bayesian CBIR 3 P(A i D i, T ) is a function of the relative similarity between A i and T in D i, e.g. P(A i D i, T, σ) = exp ( d(a i, T )/σ) X D i exp ( d(x, T )/σ), A i D i, for a precision parameter σ; σ measures how well the distance measure describes the user s choices as most relevant image.

15 Bayesian Approach A Better Distance Measure We divide our features into 3 classes: colour (CL); texture (TX); segmentation/objects (SG). Let F be the class used, then: P(A i D i, T, σ, F ) = exp ( d F (A i, T )/σ) X D i exp ( d F (X, T )/σ), A i D i, where d F is distance measure on features in class F.

16 Bayesian Approach Posterior Computation 1 Assume uniform priors: P(T = X i ) = N 1, X i X ; P(σ) = 1/σ max, 0 σ σ max ; P(F ) = 1/3, F {CL,TX,SG}. After the t-th iteration compute: P(T, σ, F H t ) { t } exp ( d F (A i, T )/σ) X D i exp ( d F (X, T )/σ) i=1 P(T )P(σ)P(F ).

17 Bayesian Approach Posterior Computation 2 This computation is done on-line so must be quick (absolutely no MCMC!); We discretise σ and compute over all 3NN σ combinations; In MATLAB code, this allows N σ = 20 and N = 5000 to be computed in 5 seconds; Computation is ideally suited to parallelisation.

18 Bayesian Approach Choosing D t+1 Once we have computed P(T, σ, F H t ) we must display D t+1 ; Usual choice is N D most probable images from P(T H t ); But this might not be sensible; Problem can be framed as a decision problem; Let U(D, T ) be the utility of picking a set D when T is the target, then D t+1 = arg max D X D =N D E(U(D, T )) = U(D, T )P(T H t ). T X

19 Bayesian Approach Indicator Utility Let U I (D, T ) = { 1, if T D, 0, otherwise. Then E(U(D, T )) = T D P(T H t); So D t+1 is indeed the set of N D most probable images.

20 Bayesian Approach Entropy Utility We might want to choose D to maximise information content; Try a utility based on negative expected entropy from choosing an image from D: U E (D, T ) = A D E(A, D) P(A D, T ), where E(A, D) = T X P(T A, D) log(p(t A, D)). To speed up computation we say P(A D, T ) = where ˆσ = E(σ H t ). X D exp ( d(a, T )/ˆσ) exp ( d(x, T )/ˆσ),

21 Bayesian Approach Evaluating D t+1 with U E (D, T ) E(U E (D, T )) not separable in T ; Cannot evaluate all ( ) X N D combinations; It s an online computation so must be fast; Tried two approaches: 1 Simulated Annealing; 2 Monte Carlo generation of sets D.

22 Bayesian Approach Possible Display Strategy A good strategy might be to: In early iterations we want to maximise information content (U E ); Later we want to display images that we think are close to what the user wants (U I ); So might at iteration t use U(D, T ) = α t U E (D, T ) + (1 α t ) U I (D, T ), where α 1 = 1 and α t 0.

23 Evaluation Example: 15 image database with 2 features FEATURE FEATURE 1

24 Evaluation D 2 under Indicator Utility 0.8 INDICATOR UTILITY FEATURE FEATURE 1

25 Evaluation D 2 under Entropy Utility ENTROPY UTILITY FEATURE FEATURE 1

26 Evaluation Expected Utilities for D 2 comparison of computational methods (average over 100 trials) Utility Exact Random Generation Sim. Annealing of 100 subsets 100 iterations Indicator Entropy

27 Evaluation Example: Bridgeman Art Library A database of 1066 paintings; Some 500 features computed on each image; PCA done on each feature class - reduces to 140 principal components.

28 Evaluation Initial Display Set (centre top image selected)

29 Evaluation Next Display Set under Indicator Utility

30 Evaluation Next Display Set under Entropy Utility

31 Evaluation Expected Utilites for D 2 in Paintings Example (average over 100 trials) Utility Exact Random Generation Simulated Annealing of 100 subsets 100 iterations Indicator Entropy

32 Evaluation Evaluation through Target Testing In target testing, an image is selected from the database; The user must find that image using the CBIR system; The number of iterations until the image is found is recorded; Only really appropriate for indicator utility (or α t U E (D, T ) + (1 α t )U I (D, T )); We used N D = 9 images displayed per iteration, and U I (D, T ). Eight professors and graduate students in the Statistics Department volunteered to look for 15 images. Next slide shows some of the images used in our target testing to compare our Bayesian approach with simple PicHunter (inference on T only).

33 Evaluation Some Target Testing Images

34 Evaluation Results of Target Testing One Way ANOVA on Number of Iterations to Find Target Factor SS df p-value for F-test Intercept < System Person Image < Error No significant difference between systems; However image effect is large.

35 Extensions Extensions (Work in Progress) Object Identification. Assume image X i is partitioned into n i objects O i = {O i1,..., O ini }; User can pick one of these as most relevant part of image; Likelihood now something like (after some independence assumptions): P(A i, O ij D i, T, O T, σ, F ) = P(A i D i, T, σ, F ) P(O ij A i, O T, σ, F ), where P(O ij A i, O T, σ, F ) = exp ( min k d F (O ij, O Tk )/σ) l exp ( min k d F (O il, O Tk )/σ). a function of the closest object in T.

36 Extensions Extensions (Work in Progress 2) More than one image may be selected as relevant. In simplest case, selecting m i images A i1,..., A imi from D i has likelihood: P(A i1,..., A imi D i, T, σ, F ) m i = P(A ij A i1,..., A i,j 1 D i, T, σ, F ) = j=1 m i j=1 exp( d F (A ij, T )/σ) X D i {A i1,...,a i,j 1 } exp( d F (X, T )/σ).

37 Extensions Extensions (Work in Progress 3) Selecting a counter-example. User chooses an image as not relevant, with likelihood: P(A i D i, T, σ, F ) = exp(d F (A i, T )/σ) X D i exp(d F (X, T )/σ).

38 Extensions Extensions (Work in Progress 4) Category Search. Searching for a category of images C X. Let { 1, if X i C, T i = 0, otherwise, and p i = P(T i = 1); Assuming independence of T i given p = (p 1,..., p N ) we have P(C = p) = P(T i = 0, i p) = i (1 p i ). Aim is to compute P(p H t );

39 Extensions Extensions (Work in Progress 5) Possible likelihood is P(A i D i, p) = p Ai j D i p j but not a function of distance. Better is a function of the T = (T 1,..., T N ): P(A i D i, T, σ, F ) ( ) exp P 1 j = T j j T j =1 d F (A i, T j )/σ X D T C ( exp ), P 1 j T j j T j =1 d F (X, T j )/σ if T Ai = 1 and 0 otherwise; Computational problems here.

40 Conclusion Issues Model assessment? Other types of evaluation (especially for other utilities); Prior for T based on an image sketch S of query: P(T ) exp( d(s, T )/σ).