Multimedia Information Retrieval

Transcription

1 Multimedia Information Retrieval Lecture 8 Lecturer: Theo Gevers Lab: MMIS gevers@science.uva.nl http: http:

2 Broad shape, multi-local invariant descriptions: introduction invariant descriptions: introduction Text, colour, shape and texture Translation: rotation: scale: ˆ Ë Ê ˆ - Ë Ê = ˆ Ë Ê 1 cos cos sin cos y x y x a a a a ˆ Ë Ê = ˆ Ë Ê 1 y x s y x ˆ Ë Ê + ˆ Ë Ê = ˆ Ë Ê 1 1 y x t t y x

3 Text, colour, shape and texture Broad shape, multi-local (finite point sets) Voting schemes: finite point sets Histogram matching Geometric hashing Generalized Hough transform Alignment methods Finite point sets

4 Mosaicing Feike Winkelman Case 4 Mosaicing

5 Mosaicing input movie: mosaic creation:

6 How does it work Mosaicing Estimate motion between successive frames I t and I t+1. The motion model used is the affine transformation. Use this estimation to warp I t+1 into the coordinate system of I t.

7 The affine model Mosaicing Linear transformation (handles translation, zoom, shear, rotation and combinations of these): u i = a 1 + a x i + a 3 y i v i = a 4 + a 5 x i + a 6 y i or, in matrix form: (horizontal velocity) (vertical velocity) a a 5 0 u i v i 1 = x i y i 1 a 3 a 6 0 a 1 1 a 4

8 How to estimate Mosaicing I t0 : I t1 :

9 Mosaicing Transform coordinates image I t1 warped into I t0 coordinates (using A inverse ):

10 Mosaicing Accuracy displaced frame difference:

11 More than images Mosaicing If: A t1 : affine motion from image I t1 to I t0, A t : affine motion from image I t to I t1, A t (I t ) : transforms coordinates I t to I t-1 with A Then: A t1 (A t (I t )) puts image I t into the coordinate system of image I t0. (In matrix form affine transformations can be multiplied to concatenate their effect.)

12 More than images Mosaicing t 1 t t 3 t 4 t 5 t 6 Only pixels of last frame are used when updating the mosaic

13 Results Mosaicing

14 Results Mosaicing

15 Results: mean Mosaicing

16 Results: median Mosaicing

17 Results: median Mosaicing

18 Results: median Mosaicing

19 More results Mosaicing

20 Mosaicing More results Techniques: Mosaics. Shot and key-frame detection. Analysis of camera-motion.

21 0. Preview 1. Vision retrieval demands general domain. Text, colour, shape and texture 3. Searching and finding 4. Modelling 5. Relevance feedback 6. Compression 7. Indexing 8. Object localisation/visualisation

22 4 Modelling Searching individual images Boolean Probabilistic Vector Space Model Binary VSM Weighted VSM

23 Searching individual images Boolean model 4 Modelling Definition : For the Boolean model, the index term weight variables are all binary i.e., w Œ{0,1}. A query q is a conventional Boolean i, j r expression. Let q be the disjunctive normal form for the query q. dnf r r Further, let q be any of the conjunctive components of q. cc dnf The similarity of a document d to the query q is defined as : j Ï r r r r r Ô1if $ q ( q Œq ) Ÿ (" k, g ( d) = g ( q )) sim( d, q) = Ì cc cc dnf i i i cc j ÔÓ 0 otherwise

24 4 Modelling Searching individual images Boolean queries Boolean algebra Logical connectives: AND, OR, and NOT Typical query is: restaurants AND (mideastern OR vegetarian) AND inexpensive With stemming: restaurant AND (mideast OR veget) AND inexpens

25 4 Modelling Searching individual images Problems with Boolean retrieval systems 1. No feature weighting. The feature is present or absent. Music by Beethoven, preferably a sonata 1.a Beethoven AND sonata 1.b Beethoven OR sonata 1.c (Beethoven AND sonata) OR Beethoven 1.d Beethoven. Misstated queries by people: incorrect interpretation of the Boolean connectives AND and OR A person seeking Saturday night entertainment:.a dinner AND sports AND symphony.b dinner OR sports OR symphony

26 4 Modelling Searching individual images Problems with Boolean retrieval systems 3. Order of precedence for the logical connectives. System 1: NOT first within the parentheses, followed by AND, followed by OR, with a left-to-right precedence System : Left-to-right order of precedence without regard to the operators A OR B AND C ----> System 1: A OR (B AND C) A OR B AND C ----> System : (A OR B) AND C

27 4 Modelling Searching individual images Problems with Boolean retrieval systems 4. Highly complex queries given by the user on input yields many partial responses to be gathered into the final response. Solution is to recast queries into either: disjunctive normal form (DNF) or conjunctive normal form (CNF) Three levels of expression in a disjunctive normal form (DNF): Terms, which are individual words or features that occur naturally or negated Conjuncts, which are joined by AND Disjuncts, which are joined by OR For example: (concert AND dinner AND NOT play) OR (swimming AND tennis) OR (baseball AND NOT football)

28 4 Modelling Searching individual images Disjunctive Normal Form DNF: example Truth Table Full DNF Query: (A OR B) AND (C OR NOT D) AND (D OR B) DNF: (A ANDC AND D) OR (B AND C) OR (B AND (NOT D)) Row A B C D (A OR B) (C ORNOT D) ( D OR B) Expression 1 T T T T T T T T A AND B AND C AND D OR T T T F T T T T A AND B AND C AND (NOT D) OR 3 T T F T T F T F 4 T T F F T T T T A AND B AND (NOT C) AND (NOT D) OR 5 T F T T T T T T A AND (NOT B) AND C AND D OR 6 T F T F T T F F 7 T F F T T F T F 8 T F F F T T F F 9 F T T T T T T T (NOT A) AND B AND C AND D OR 10 F T T F T T T T (NOT A) AND B AND C AND (NOT D) OR 11 F T F T T F T F 1 F T F F T T T T (NOT A) AND B AND (NOT C) AND (NOT D) 13 F F T T F T T F 14 F F T F F T F F 15 F F F T F F T F 16 F F F F F T F F

29 4 Modelling Documents Inverted files Example: A sample text and an inverted index build on it. The words are converted to lower-case and some are not indexed. The occurrences point to character positions in the text block1 block block3 block This is a text. A text has many words. Words are made from letters. Vocabulary Occurrences Block occurrences letters made many text words , , ,

30 Searching individual images element u of Fuzzy set theory Definition : A fuzzy subset A of by a membership function m 4 Modelling a : U universe of U a number m ( u) in the interval[0,1] A A discourseu is characterized Æ [0,1] which associates with each Definition : Let U be the universe of discourse, A and B be two fuzzy subsets of U, and A be the complement of A relative to U. Also, let u be an element of U. Then, m m m A ( u) = 1- m ( u) A» B A«B ( u) = ( u) = A max( m min( m A A ( u), m ( u), m B B ( u)) ( u))

31 4 Modelling Searching individual images Extended Boolean model Definition An IR system based on extended Boolean models is a quadruple [D,Q,F,R(qi,dj)] where (1) T is a set of index terms to represent queries and documents () D is a set of documents. Each document d is represented by {(t1,w1),,(tn,wn)}, where wi designates the weights (3) Q is a set of queries (binary weights). (4) R(qi,dj) is a ranking function which associates a real number with a query qi in Q and a document representation dj in D. Such ranking defines an ordering among the documents with regard to the query qi.

32 Searching individual images Extended Boolean model 4 Modelling (0,1) (1,1) (0,1) (1,1) x OR y x AND y y y d d (0,0) x (1,0) (0,0) x (1,0) sim( q OR, d) = x + y sim( q AND, d) = 1- (1 - x ) + (1 - y )

33 4 Modelling Searching individual images fuzzy set : r - norm : r - norm : Boolean model: AND and OR operators Extended Boolean : AND OR AND OR AND OR MIN( w, w 1 MAX( w 1, w Ê (1 - w1 ) (1 - w 1- Ë n Ê w w Ë n ) n r Ê (1 - w1 ) (1 - w 1- Ë n r Ê w w Ë n ) r n ˆ 1/ ˆ 1/ r n ) n ) ˆ r 1 r 1/ ˆ 1/ r

34 4 Modelling Searching individual images Boolean Model: conclusion Queries of the form: Binary weights and no ranking. q = k Ÿ ( k k ) c Most users find it awkward to express queries in Boolean form. Suited when query contain only a few terms. a Extensions: fuzzy Boolean and extended Boolean models For example: Q = ( f i, w i ) AND ( f j, w j b ) OR ( f k, w k )