Contend Based Multimedia Retrieval

Transcription

1 Contend Based Multimedia Retrieval CBIR Query Types Semantic Gap Features Segmentation High dimension IBMS QBIC GIFT, MRML Blobworld CLUE SIMPLIcity CBMR Multimedia Automatic Video Analysis 1

2 CBIR Contend Based Image Retrieval Most known! CBMR Contend Based Music Retrieval? Contend Based Video Retrieval CBIR Traditional text-based image search engines Manual annotation of images Use text-based retrieval methods E.g. Water lilies Flowers in a pond <Its biological name> 2

3 Query Types exact query e.g. image predicate amount sky > 20% approximate query sketch image example group example positive example negative example Retrieval Engine - Problems Semantic Gap The image is used to understand its meaning The only available independent information is the image itself (low-level pixel data) Solution: image understanding system (IU) Artificial Intelligence: Works only in an very narrow domain The technical problem is that of automatically deriving a sensible description from an image 3

4 Impact depends on the image domain Narrow Domain Broad Domain Image Retrieval Retrieval by image (not domain specific!) Query: Image or an sketch as an example How to judge image similarity? Result collects of images that have similar spatial characteristics 4

5 CBIR Architecture Learning 5

6 Feature Extraction Enhance the image information Extracted Features: Image Signature Representation of important information Significant compression! Feature Extraction - Feature Types Color Texture Shape Layout 6

7 Color CIE 1931 XYZ color space Created 1931 Film versus RGB color Images Color RGB Histogram 7

8 Texture Shape 8

9 10/16/09 Layout Atmospheric similar images Images with the most similar color characteristic 9

10 10/16/09 Images ordered according to the distance to the image y Most similar images... y= Color Layout Need for Color Layout How it works: Global color features give too many false positives Divide whole image into sub-blocks Extract features from each sub-block Can we go one step further? Divide into regions based on color feature concentration This process is called segmentation... See intermedia, Oracle 10i 10

11 Example: Color layout ** Image adapted from Smith and Chang : Single Color Extraction and Image Query k-means Example (K=2) x x x x Pick seeds Reassign clusters Compute centroids Reasssign clusters Compute centroids Reassign clusters Converged! 11

12 Algorithm /*For RGB images*/ Random initialization of k cluster centers /* x i = R i, G i, B i ; c j = R j, G j, B j of pixel at i or j*/ do { -assign to each x i in the dataset the nearest cluster center (centroid) c j according to d 2 -compute all new cluster centers } until ( E new - E old < ε or number of iterations max_iterations) Segmentation Strong Segmentation Segmented objects Weak Segmentation Regions, not nessacary objetcs Color segmentation 12

13 Backend image collection (big) + metainformation (e.g. image signatures) High dimensionality (Problem) Many data sets (images) dimension reduction? multi-dimensional indexing structures Distinction to Pattern recognition Quantity of the data! Problem of high dimensions (Example) Mean Color = RGB = 3 dimensional vector Color Histogram = 256*3 dimensions Effective storage and speedy retrieval needed Data Amount!!! Traditional data-structures not sufficient!!! Solution, indexing: R-trees, SR-Trees etc 13

14 2-dimensional space Point A D2 D1 3-dimensional space 14

15 Now, imagine An N-dimensional box!! We want to conduct a nearest neighbor query. Indexing: R-trees are designed for speedy retrieval of results for such purposes Designed by Guttmann in 1984 Sample CBIR architecture 15

16 IBM s QBIC QBIC Query by Image Content First commercial CBIR system Model system influenced many others Uses color, texture, shape features Text-based search can also be combined Uses Gemini and R*-trees for indexing QBIC Search by color ** Images courtesy : Yong Rao 16

17 QBIC Search by shape ** Images courtesy : Yong Rao QBIC Query by sketch ** Images courtesy : Yong Rao 17

18 GIFT Framework for Content-based Multimedia Retrieval Communication Protocol: MRML (Multimedia Retrieval Markup Language) Image Retrieval Plugin: VIPER GIFT Is the result of a research of the University of Geneva The GIFT (the GNU Image-Finding Tool) is a Content Based Image Retrieval System Permits Query by example on Images Improve query results by relevance feedback Has a distributed architecture (Client - Server) The protocol for the C-S communication is MRML 18

19 GIFT How it works? Indexing of the database prior to query Then features are extracted from all images in the DB Each image is translated in a variable length sequence of features which describe the image Defines a set of about 80,000 possible image features based on colour and textures at different scales and in a hierarchical decomposition of the image Each feature is assigned a weight determined dependent on the frequency of the feature within the image and within the collection Local and global color information (Grabor filter) Local and global texture information 19

20 Signature = List of Discrete Features 80,000 Stored Features Each Image: O(10 3 ) Features Combination of CBIR with Annotation Short-Term Learning: Relevance Feedback Long-Term Learning: Log-File Analysis The feature information is stored in an inverted file, together with weights such as document and collection frequencies An inverted file index contains for each word a list of references to all the documents in which it occurs A full inverted index additionally contains information about where in the documents the words appear Document IDs and local positions 20

21 Example Rows... Columns... Red... Yellow... Blue... When a query is made the inverted file is then searched for the best match in the database (not a good solution?) The result of the query is therefore the list of all images with their respective similarity found with the example set Supports positive and negative feedback 21

22 MRML (Multimedia Retrieval Markup Language) GIFT: Uses distributed architecture For that reason is necessary some kind of standard mark-up language and communication protocol Are traditionally described in form of a dataflow diagram (DFD) Uses text based query languages and communication methods between client and server Parsers encode and decode the query language XML-based languages such as MRML that makes this task a little but easier (ready make parsers exist) 22

23 MRML MRML=Multimedia Retrieval Markup Language Created by the University of Geneva The aim is to standardise access to Multimedia Retrieval software components Is a XML based protocol with a formal specification Some features: Extensibility: Provide a framework which permit independent growth of the products No preferred implementation language Independence of third-party libraries MRML MRML-based communications have the structure of a remote procedure calls (RPC) The client connect to the server Sends a request Stays connected with the server until the server breaks the connection The server shuts down the connection after sending the MRML message which answer the request 23

24 MRML: Connection Connection request: <mrml > <get-server-properties /> </mrml> Connection: <mrml > <get-algorithms collection-id = "c1" /> </mrml> <mrml > <server-properties /> </mrml> <collection-list > <collection collection-id = "c-tsr500-id" collection-name = "TSR500" /> </collection-list> <algorithm-list > <algorithm> algorithm-id = "a-idf-id" algorithm-name = "Classical TF/IDF" algorithm-type = "adefault" collection-id = " c-tsr500-id " > <property/> </algorithm> </algorithm-list> MRML: Querying MRML currently includes only QBE (Query by Example), but it has been designed to be extensible Consists of a list of images and the corresponding relevance levels assigned to them by the user The query step is dependent on the query paradigms offered by the interface and the search engine <mrml > session-id = "1" transaction-id = "44" > <query-step> session-id = "1" resultsize = "30" algorithm-id = "algorithm-default" > <user-relevance-list > <user-relevance-element image-location = " user-relevance = "1" /> <user-relevance-element image-location = " user-relevance = "-1" /> </user-relevance-list> </query-step> </mrml> 24

25 Blobworld Select an appropriate scale for each pixel, and extract color, texture, and position features for that pixel at the selected scale Group pixels into regions by modeling the distribution of pixel features (Blob) Describe the color distribution and texture of each region for use in a query Segmentation into blobs 25

26 query Blobworld query for tiger images using two blobs The overall weights are 1.0 for the tiger blob and 0.5 for the grass blob For both blobs, the color weight is 1.0 and the texture weight is 0.5. CLUE Cluster-based Image Retrieval Scheme Similarity of Result Images to each other Browsing with a Two-Level Display Scheme First Level: one Representative Image for each Cluster Second Level: all Images within the Selected Cluster Local Semantic Structure of Result Images 26

27 A similarity-driven approach that can be built upon virtually any symmetric real-valued image similarity measure It uses a graph-theoretic algorithm to generate clusters 27

28 SIMPLIcity Image Classification Semantic Categories Textured vs. Non-textured Graph vs. Photograph Broad Domain vs. Narrow Domain Integrated Region Matching 28

29 Partition an image into 4 4 blocks Extract wavelet-based features from each block Use k-means algorithm to cluster feature vectors into regions SIMPLIcity 29

30 Multimedia Video segments Color Camera motion Motion activity Mosaic Still regions Color Shape Position Texture Moving regions Color Motion trajectory Parametric motion Spatio-temporal shape Audio segments Spoken content Spectral characterization Music: timbre, melody, pitch What is CBMR? CBMR : Content-based Music Retrieval Traditional database query : Text-based or SQL-based Our goal : Music retrieval by singing/humming 30

31 Compare by DTW Wave File DTW Mid File Music Retrieval By Singing/ humming Happy Birthday Note starts Note ends Note starts Note ends A note has two important attributes Pitch: It tells people which tone to play Duration: It tells people how long a note needs to be played Notes are represented by symbols Staff Note name Note pitch Do Re Mi Fa So La Si Do 31

32 Humming La, Recorder Wave to Symbols Approximate String Match Retrieval Result Wave files MP3 files MIDI files Feature Extraction Various Music Formats to Symbols Music Database Music Database Indexing Multimedia Video segments Color Camera motion Motion activity Mosaic Still regions Color Shape Position Texture Moving regions Color Motion trajectory Parametric motion Spatio-temporal shape Audio segments Spoken content Spectral characterization Music: timbre, melody, pitch 32

33 Key Frame Extraction Shot Detection Key Frame Extraction 1. Decompose video segment into shots 2. Compute key/representative frame for each shot 3. Query by CBIR 4. Use frame from highest scoring shot Various Clues in Video Retrieval 33

34 10/16/09 Generates transcript to enable text-based retrieval from spoken language documents Improves text synchronization to audio/video in presence of scripts SILENCE MUSIC electric cars are Text Extraction they are the jury every toy owner hopes to please Raw Audio Raw Video 34

35 Automatic Video Analysis and Index Scene Cuts Yellowstone Camera Static Static Zoom Objects Adult Female Animal Two adults Action Head Motion Left Motion None Captions [None] Yellowstone [None] Scenery Indoor Outdoor Indoor Video Search: Features Shape Outer Boundary based vs. region based Fourier descriptors Moment invariants Finite Element Method (Stiffness matrix- how each point is connected to others; Eigen vectors of matrix) Turing function based (similar to Fourier descriptor) convex/ concave polygons[arkin et al] Wavelet transforms leverages multiresolution [Chuang & Kao] Chamfer matching for comparing 2 shapes (linear dimension rather than area) 3-D object representations using similar invariant features Well-known edge detection algorithms. Face Face detection is highly reliable - Neural Networks [Rwoley] - Wavelet based histograms of facial features [Schneiderman] Face recognition for video is still a challenging problem. - EigenFaces: Extract eigenvectors and use as feature space OCR OCR is fairly successful technology. Accurate, especially with good matching vocabularies. Script recognition still an open problem. ASR Automatic speech recognition fairly accurate for medium to large vocabulary broadcast type data Large number of available speech vendors. Still open for free conversational speech in noisy conditions. 35

36 Conclusion Common methods for CBIR,CBMR,etc...: Feature extrection Indexing CBIR Query Types Semantic Gap Features Segmentation High dimension IBMS QBIC GIFT, MRML Blobworld CLUE SIMPLIcity CBMR Multimedia Automatic Video Analysis 36