Media Retrieval (2) Prepared by. Ling Guan Jose Lay Paisarn Muneesawang Ning Zhang Rui Zhang. Outlines (revisited)

Transcription

1 Media Retrieval (2) Prepared by Ling Guan Jose Lay Paisarn Muneesawang Ning Zhang Rui Zhang 1 Outlines (revisited) Introduction: Intellectual Foundation of Multimedia Information Retrieval Retrieval Models Text Retrieval: Database, Bibliographic, and Keyword Searches Content based Retrieval: Object matching and beyond Indexing: Inverted file Searching: Multimodality and Query adaptation 2 1

2 Content based Retrieval (the conventional approaches) Images, Audio, and Video Caption Text Multimodalities Search methods and search engines Result visualization/presentation Let us get start with a tour! 3 Examples of The MMR systems Text based (and Content based) Querying Images Google s Image Search Yahoo Image Search (AltaVista s Image Search) TinEye Image Search (Toronto Local) Audio Yahoo Audio Search Compaq s SpeechBot (no longer exist) Video and Multiple modalities Facebook.com Lycos Multimedia Search (evolving from a single search engine into a focused network of community and social sites) FAST AllTheWeb Search emediasearch.com (covering the Middle East region) 4 2

3 Google s image search: new feature 5 Google s image search: new feature 6 3

4 Google s image search: new feature 7 Yahoo (Alta Vista) Image Search Yahoo Image Search 8 4

5 TinEye, developed by Idée, Inc 9 Compaq s SpeechBot (closed by HP) 10 5

6 FAST s AllTheWeb Search (bought out by Overture) 11 Qwiki (acquired by Yahoo!) Traditional search result presentation grid based It turns static information into dynamic experience 12 6

7 MediaMill Semantic video search engine Cross browser Galaxy browser Sphere browser 13 3D Visualization/Presentation MIT Tangible HCI Lab 14 7

8 Non general searching sites Image/Video Hosting sites Primarily function as sharing sites for on line community. Indexed large database provide searching features for users. Image hosting Flickr, Panoramio, Picassa Video hosting YouTube Vertical Search Engine Focus on a specific segment of online contents Academic: IEEE Explore, Google Scholar Map: with yellow page info Google Maps and StreetView, Bing Maps and StreetSide. Food and Shopping Yummly, Bing shopping, Yelp Information Wikipedia 15 The Extended Text based Systems Processing textual information to infer on audiovisual content. Often rely on the filename. koala.jpg Using meta information embedded in the web page. <IMG SRC= ryerson1234.gif" WIDTH="300" HEIGHT="60" ALT= koala watching TV"> Further processing of the container page. The webpage is called: Trailer for Starwars episode I: Phantom the Menace, and in there you find a link to the phantom.mpg file. 16 8

9 Speech Pre processing In a speech retrieval engine, e.g., SpeechBot, a speech processing mechanism is applied to convert audio data into textual data, then conventional text based indexing and searching could be applied. 17 Touring the First Generation CBR Systems Visual based Querying University of California at Irvine s MARS (Initially developed at Univ. of Illinois) IBM s QBIC 18 9

10 MARS 19 QBIC at Hermitage Museum 20 10

11 What did you observe? What has been extended? Which applications may be supported? What is the constraint? 21 CBR: a brief history In 1970s, image data are commonly archived independently and indexed using text based databases. Indexing carries subjectivity of annotation. Manual and expensive. Into 1990s, efforts are taken to remove person indexing and to automate the mechanism. Image data themselves are used as indexes. Full automation is idealized. Content Based Retrieval was born

12 CBR: a brief history 2 The CBR was founded on the computer vision and pattern recognition idealism and their techniques. CBR employs object matching mechanisms. Earlier systems are characterized by Query by Example approach. Features commonly used are: Color, Texture, and Shape. Motion features in case of videos. 23 CBR: a brief history 3 As Computer vision ideal is yet to achieve, CBR found itself to be constrained in a green house. Many set off for sophisticated feature exploration and computational reasoning adventures. Deep NNs have provided a viable solution. A strong call for abandoning the full automation idealism for returning to human interaction. HCI techniques such as relevant feedbacks started to rollout

13 Between Yesterday & Tomorrow An interesting question to ask today may well be: is the object matching idea all we want? Typical current CBR query would mostly read: retrieve all documents that presents a blonde wearing blue dress driving a red porsche. Now let us suggest some offsets: How about retrieve all paintings created using chiaroscuro technique? What differences are there between Christmas Cards and Birthday Cards? Where do you want to go today? chiaroscuro pictorial representation in terms of light and shade without regard to color 25 The Passage Go Deeper. Enhancing what we have got. Human centered computing. Distributed Computing. Multi modalities querying and processing. Go Sidewise. Call for better multimedia understanding. Efforts to find out which information and semantics are useful and how they could be derived and managed Concept based (or semantic) retrieval. Transcending the Object Matching Idealism. Multimedia information mining. Big data Deep learning 26 13

14 Content based Retrieval? Traditional Bibliographic Objectives Object Matching Keyword Searches 27 Roadmap Test 1: Tuesday, October 16, 1 hour Coverage of material: everything studied up to Lecture 5. The 1 st half of the class is normal teaching, and the 2 nd is test time. Open notes (paper copies only). You may consider bringing in a sheet of paper summarizing the contents of the course up to and including Lecture 5. Material taught on October 16 will not be included in Test 1. Office hours: Monday, October 15, 2:30 4:30pm, ENG 315. Submit your project proposal electronically by Friday, October 12, 11:59pm