GEMINI GEneric Multimedia INdexIng

Transcription

1 GEMINI GEneric Multimedia INdexIng GEneric Multimedia INdexIng distance measure Sub-pattern Match quick and dirty test Lower bounding lemma 1-D Time Sequences Color histograms Color auto-correlogram Shapes 1

2 GEneric Multimedia INdexIng Given a database of multimedia objects Design fast search algorithms that locate objects that match a query object, exactly or approximately Objects: 1-d time sequences Digitized voice or music 2-d color images 2-d or 3-d gray scale medical images Video clips E.g.: Find companies whose stock prices move similarly Applications time series: financial, marketing (click-streams!), ECGs, sound; images: medicine, digital libraries, education, art higher-d signals: scientific db (eg., astrophysics), medicine (MRI scans), entertainment (video) 2

3 Sample queries Find medical cases similar to Smith's Find pairs of stocks that move in sync Find pairs of documents that are similar (plagiarism?) Find faces similar to Tiger Woods $price $price $price day day distance function: by expert day (eg, Euclidean distance) 3

4 Generic Multimedia Indexing 1 st step: provide a measure for the distance between two objects Distance function d(): Given two objects O 1, O 2 the distance (=dissimilarity) of the two objects is denoted by d(o 1, O 2 ) E.g., Euclidean distance (sum of squared differences) of two equal-length time series ε-similarity query Given a query object Q, find all objects O i from the database are ε-similar (identical for ε = 0) to Q {O i DB d(q, O i ) < ε}. 4

5 Types of Similarity Queries Whole match queries: Given a collection of S objects O 1,, O s and a query object Q find data objects that are within distance ε from Q Types of Similarity Queries Sub-pattern Match: Given a collection of S objects O 1,, O S and a query (sub-) object Q and a tolerance ε identify the parts of the data objects that match the query Q 5

6 Idea method requirements Fast: sequential scanning and distance calculation with each and every object too slow for large databases Dynamic: easy to insert, delete, and update objects Basic idea Focus on whole match queries Given a collection of S objects O 1,, O s, a distance/dis-similarity function d(o i, O j ), and a query object Q find data objects that are within distance ε from Q Sequential scanning? May be too slow.. for the following reasons: Distance computation is expensive (e.g., editing distance in DNA strings) The Database size S may be huge Faster alternative? 6

7 GEneric Multimedia INdexIng Christos Faloutsos QBIC 1994 A feature extraction function maps the high dimensional objects into a low dimensional space Objects that are very dissimilar in the feature space, are also very dissimilar in the original space Basic idea Faster alternative: Step 1: a quick and dirty test to discard quickly the vast majority of non-qualifying objects Step 2: use of SAMs (R-trees, Hilbert-Curve,..) to achieve faster than sequential searching Example: Database of yearly stock price movements Euclidean distance function Characterize with a single number ( feature ) Or use two or more features 7

8 Basic idea - illustration Feature2 S1 F(S1) day F(Sn) Sn Feature day A query with tolerance ε becomes a sphere with radius ε Basic idea caution! The mapping F() from objects to k-dim. points should not distort the distances d(): distance of two objects d feature (): distance of their corresponding feature vectors Ideally, perfect preservation of distances In practice, a guarantee of no false dismissals How? 8

9 Objects represented by vectors that are very dissimilar in the feature space are expected to be very dissimilar in the original space If the distances in the feature space are always smaller or equal than the distances in the original space, a bound which is valid in both spaces can be determined The distance of similar objects is smaller or equal to ε in the original space and, consequently, it is smaller or equal to ε in the feature space as well... 9

10 Lower bounding lemma if distance of similar objects is smaller or equal to ε in original space then it is as well smaller or equal ε in the feature space d feature (F(O 1 ),(O 2 )) " d(o 1,O 2 ) " # o.k. d(o 1,O 2 ) " # $ $ % d feature (F(O 1 ),F(O 2 )) WRONG! d feature (F(O 1 ),F(O 2 )) " # $ $ $ % d(o 1,O 2 ) " # d feature (F(O 1 ),F(O 2 )) " # " d(o 1,O 2 ) "? No object in the feature space will be missed (false dismissals) in the feature space There will be some objects that are not similar in the original space (false hints/alarms) 10

11 That means that we are guaranteed to have selected all the objects we wanted plus some additional false hits in the feature space In the second step, false hits have to be filtered from the set of the selected objects through comparison in the original space 11

12 Time sequences white noise brown noise Fourier spectrum... in log-log Time sequences Conclusion: colored noises are well approximated by their first few Fourier coefficients Colored noises appear in nature 12

13 Time sequences Eg.: GEMINI Important: Q: how to extract features? A: if I have only one number to describe my object, what should this be? 13

14 1-D Time Sequences Distance function: Euclidean distance Find features that: Preserve/lower-bound the distance Carry as much information as possible(reduce false alarms) If we are allowed to use only one feature what would this be? The average extending it 1-D Time Sequences... If we are allowed to use only one feature what would this be? The average extending it The average of 1st half, of the 2nd half, of the 1st quarter, etc. Coefficients of the Fourier transform (DFT), wavelet transform, etc. 14

15 Feature extracting function 1. Define a distance function 2. Find a feature extraction function F() that satisfies the bounding lemma Example: Discrete Fourier Transform (DFT) preserve Euclidian distances between signals (Parseval's theorem) F() = DTF which keeps the first coefficients of the transform 1-D Time Sequences Show that the distance in feature space lower-bounds the actual distance DFT? Parseval s Theorem: DFT preserves the energy of the signal as well as the distances between two signals d(x,y) = d(x,y) where X and Y are the Fourier transforms of x and y If we keep the first k n coefficients of DFT we lower-bound the actual distance d feature (F(x),F(y)) = k"1 # f = 0 X f "Y f 2 n"1 # $ X f "Y f f = 0 2 n"1 # = x i " y i i= 0 2 % d(x,y) 15

16 Time sequences - results keep the first 2-3 Fourier coefficients faster than seq. scan no false dismissals total time cleanup-time r-tree time # coeff. kept Time sequences - improvements: could use Wavelets, or DCT could use segment averages 16

17 Images - color what is an image? A: 2-d array 2-D color images Color histograms Each color image a 2-d array of pixels Each pixel 3 color components (R,G,B) h colors each color denoting a point in 3-d color space (as high as 2 24 colors) For each image compute the h-element color histogram each component is the percentage of pixels that are most similar to that color The histogram of image I is defined as: For a color C i, H ci (I) represents the number of pixels of color C i in image I OR: For any pixel in image I, H ci (I) represents the possibility of that pixel having color C i. 17

18 2-D color images Color histograms Usually cluster similar colors together and choose one representative color for each color bin Most commercial CBIR systems include color histogram as one of the features (e.g., QBIC of IBM) No space information Color histograms - distance One method to measure the distance between two histograms x and y is:!! aij ( xi " yi )( x j " t d 2 ( x, y) = ( x " y) # A# ( x " y) = y ) h h i h j j where the color-to-color similarity matrix A has entries a ij that describe the similarity between color i and color j 18

19 Images - color Mathematically, the distance function is: Color histograms lower bounding 1 st step: define the distance function between two color images d()=d h () 2 nd step: find numerical features (one or more) whose Euclidean distance lower-bounds d h () If we allowed to use one numerical feature to describe the color image what should it be? Avg. amount for each color component (R,G,B) t x = ( R, G, B ) avg R avg = (1/ P) avg P! Where, and similarly for G and B avg p= 1 R( p) Where P is the number of pixels in the image, R(p) is the red component (intensity) of the p-th pixel 19

20 Color histograms lower bounding Given the average color vectors and of two images we define d avg () as the Euclidean distance between the 3-d average color vectors 3 2 t 2 davg ( x, y) = ( x " y) #( x " y) =!( xi " yi ) i= 1 3 rd step: to prove that the feature distance d avg () lower-bounds the actual distance d h ()......by the ``Quadratic Distance Bounding'' theorem it is guaranteed that the distance between vectors representing histograms is bigger or equal as the distance between histograms of average color images. The proof of the ``Quadratic Distance Bounding'' theorem is based upon the unconstrained minimization problem using Langrange multipliers Main idea of approach: First a filtering using the average (R,G,B) color, then a more accurate matching using the full h-element histogram x y Images - color time seq scan performance: w/ avg RGB selectivity 20

21 Color auto-correlogram pick any pixel p1 of color C i in the image I at distance k away from p1 pick another pixel p2 what is the probability that p2 is also of color C i? Red? k P2 P1 Image: I Color auto-correlogram The auto-correlogram of image I for color C i, distance k: $ ( k ) C i ( I ) # Pr[ p1 " p2 = k, p2! IC p1! I i C i ] Integrate both color information and space information 21

22 Color auto-correlogram Implementations Pixel Distance Measures Use D8 distance (also called chessboard distance): d max ( p,q) = max( p x " q x, p y " q y ) Choose distance k=1,3,5,7 Computation complexity: Histogram: Correlogram:!( n 2 )!(134* n 2 ) 22

23 Implementations Features Distance Measures: D( f(i 1 ) - f(i 2 ) ) is small I 1 and I 2 are similar m= R,G,B k=distance or histogram: I # I' $ For correlogram: h! h ( I) # h ( I') Ci Ci i" [ m] 1+ hc ( I) + hc ( I' ) i i I # I' % $ % ( I) #% ( k ) ( k ) Ci Ci! ( k ) ( k ) i" [ m], k" [ d ] 1+ % C ( I) + % C i ( I') i ( I') Color Histogram vs Correlogram Correlogram method Query Image (512 colors) 1st 2nd 3rd 4th 5th Histogram method 1st 2nd 3rd 4th 5th 23

24 Color Histogram vs Correlogram Query Correlogram method: 1 st Histogram method: 48 th Target Color Histogram vs Correlogram Query Correlogram method: 1 st Histogram method: 31 th Target 24

25 Color Histogram vs Correlogram Query 1 Query 2 Query 3 Query 4 Target C: 178 th H: 230 th C: 1 st H: 1 st C: 1st H: 3 rd C: 5th H: 18 th The correlogram method is more stable to contrast & brightness change than the histogram method. Color Histogram vs Correlogram The color correlogram describes the global distribution of local spatial correlations of colors. It s easy to compute It s more stable than the color histogram method 25

26 Images - shapes Distance function: Euclidean, on the area Q: how to do dim. reduction? A: Karhunen-Loeve (PCA) Images - shapes Performance: ~10x faster log(# of I/Os) all kept # of features kept 26

27 Mutlimedia Indexing Conclusions GEMINI is a popular method Whole matching problem Should pay attention to: Distance functions Feature Extraction functions Lower Bounding Particular application Conclusions GEMINI works for any setting (time sequences, images, etc) uses a quick and dirty filter faster than seq. scan 27

28 GEneric Multimedia INdexIng distance measure Sub-pattern Match quick and dirty test Lower bounding lemma 1-D Time Sequences Color histograms Color auto-correlogram Shapes 28