Welcome to the class of Web Information Retrieval. Min ZHANG

Transcription

1 Welcome to the class of Web Information Retrieval Min ZHANG

2 Visual Information Retrieval Min ZHANG

3 What Is Visual IR & The Importance Visual IR 3

4 What is Visual Information Retrieval? Retrieve, from a database, images or image sequences that are relevant to a query. Del Bimbo, Alberto. Visual Information Retrieval. Image retrieval and video retrieval An extension of traditional IR designed to include visual media Visual IR 4

5 Text IR and Visual IR A lot of commonality between text IR and visual IR Similar procedure Similar algorithms Interactivity with visual content is essential to visual IR More complicated More interesting Visual IR 5

6 Large Amounts of Visual Information Oct. 10, 2009, 4 billion images Sep. 18, 2010, 5 billion images May 5, 2013, 8+billion images 3 million images uploaded every day Visual IR 6

7 Large Amounts of Visual Information 15 hours of new videos uploaded every minute Visual IR 7

8 Large Amounts of Visual Information Billions of Web Images Visual IR 8

9 The Impact to Industry Many new applications & opportunities Image/video/music search engines New business model for advertising Visual recognition (Google Goggles) Augmented reality Visual IR 9

10 e.g.1 Deliver Online Ads Inside Images [ACM Multimedia 08] Large-scale Image Content Analysis Delivering Online Advertisement Inside Images Content + Context Analysis Targeted Advertising BMW DRIVE YOU TYOU THE TO WAY THE RIGHT WAY Visual IR 10

11 e.g.1 Deliver Online Ads Inside Images (Cont.) [ACM Multimedia 08] Large-scale Image Content Analysis Delivering Online Advertisement Inside Images Content + Context Analysis Targeted Advertising Visual IR 11

12 e.g.2 Google Goggles Visual IR 12

13 e.g.3 Augmented Reality A TED talk and demo: (highlights: 2 50, 4 30, 7 00) (also at E.g. New York Nearest Subway Visual IR 13

14 What Is Visual IR & The Importance Different Approaches Visual IR 14

15 Three Types of Information Associated with Image or Video Content-independent metadata author, format, date, location, etc. Content-dependent data color, texture, shape, motion, etc. Content-descriptive metadata meaning associated with visual signs and scenes Visual IR 15

16 Three Types of Information: Example Content-independent metadata Mohammed Al-Naser 2008:10:22 16:09:31 Paris Nikon D300 Content-dependent data Green, blue, yellow Texture Content-descriptive metadata Castle, Clouds, Sky, Tree, Grass Disneyland Visual IR 16

17 Three Types of Visual IR Text based Retrieval Content based Retrieval Semantic based Retrieval Visual IR 17

18 Three Types of Visual IR Text based Retrieval Surrounding text Caption Close caption, visible to those who choose to decode or activate them Open caption, visible to all viewers Speech transcription via Automatic Speech Recognition (ASR) Traditional Text Retrieval Visual IR 18

19 Text based Retrieval Systems (previous) Google Image Search (images.google.com) title, surrounding text, etc. Visual IR 19

20 Text based Retrieval Systems Google Video (video.google.com) title, tag, etc. Visual IR 20

21 Text based Retrieval Systems Blinkx ( Speech transcription Visual IR 21

22 Three Types of Visual IR Text based Retrieval But it s not good enough! Limitations Content based Retrieval Semantic based Retrieval Visual IR 22

23 Text based Retrieval Limitation A picture is worth thousand words Not everything can be described in text easily Textual information is not synchronized with visual information (e.g. Video v.s. Web page) Not everything described in text appears in the picture Not everything in the picture is described in text Visual IR 23

24 Three Types of Visual IR Text based Retrieval Content based Retrieval Semantic based Retrieval Visual IR 24

25 Content based Retrieval Kato, T. (1992). Database architecture for content-based image retrieval Low level features Color Shape Texture Query by examples Sample images or videos Indexing problem Visual IR 25

26 A Simple CBIR System Image DB Metadata Feature Extraction Color Texture Shape Offline Query & Feedback Result User Interface Memory Similarity Ranking Online Visual IR 26

27 Features: color RGB color space: Red, green, blue [0~255] Inconsistent with human perception Conversion from RGB to gray scale Y = * R * G * B Y = (R + G + B) / 3 Other color spaces HSV (Hue, saturation and value) S = 1 min(r, G, B) / max(r, G, B), V = max(r, G, B) Lab, Luv, Visual IR 27

28 Feature: color histogram An approximation to the color distribution Color Histogram Divide the space into limited number of bins E.g. R, G, B 4 levels, then 4 x 4 x 4 = 64-D Count the number of pixels in each color bin Produces vector representations Visual IR 28

29 Feature: color features Invariant with translation, rotation and scale Information Lost Quantization Spatial Other color features Color moment u N Color coherence vector N N 1 N i P 2 ij ) 2 s ( ( ) 3 i Pij ui ) i ( ( Pij ui ) j1 N j1 N j1 Split histogram into #coherent_pixels and #incoherent_pixels with each color Coherent vs. incoherent: Based on whether it is part of a large similarly-colored region Visual IR 29

30 CBIR based on Color Histogram (example) Visual IR 30

31 Features: textual Important but hard to describe No standard definition From a psychological point of view, texture features that strike a human observer are granularity, directionality and repetitiveness Statistical Texture Measures Typically measure properties: e.g. contrast, correlation, entropy. Structural Texture Modeling Model the texture as a repeating function of atomic textures like bricks on a wall. Automatically finding the atomic textures is difficult. Wavelet texture Visual IR 31

32 Features: shape Salient Points-based Retrieval Are there certain important pixels which contain the most information? Visual IR 32

33 Content based Retrieval Kato, T. (1992). Database architecture for content-based image retrieval Low level features Color Shape Texture Query by examples Sample images or videos Indexing problem Visual IR 33

34 Index Challenges Input Image database: Millions or billions Image representation: Bag-of-visual- word Similarity measure: Cosine similarity Problem Similarity Search How to efficiently find top K nearest neighbors with low memory and time cost? Just like Search for Long Query If a query contains 1000 keywords: Need to access 1000 inverted lists The intersection of 1000 inverted lists may be empty The union of 1000 inverted list may be the whole corpus Visual IR 34

35 Key Idea: Dimension Reduction + Residual Error Preservation Dimension Reduction + Residual Error Preservation Visual IR 35

36 Content based Retrieval Systems (1) QBIC (Query By Image Content) 1995, 1 st commercial CBIR systems Query by color Query by layout Visual IR 36

37 Content based Retrieval Systems (2) MARS: Relevance feedback Yong Rui, Thomas S. Huang, Sharad Mehrotra. Contentbased image retrieval with relevance feedback in MARS. ICIP references Visual IR 37

38 Three Types of Visual IR Text based Retrieval Content based Retrieval A little bit better But still far from what we want! Limitations Semantic based Retrieval Visual IR 38

39 Content based Retrieval Limitations Sensory Gap Gap between the object in the real world and the information in a (computational) description derived from a recording of that scene Semantic Gap Lack of coincidence between the information that one can extract from the visual data and the interpretation that the same data have for a user in a given situation Visual IR 39

40 Gap Between Concept And Feature Visual IR 40

41 Gap Between Concept And Feature Visual IR 41

42 Subjectivity of human visual perception Visual IR 42

43 Gap at Different Level Low level blue, orange, horizontal line Mid level sky, tree, boat High level lonesome boat Visual IR 43

44 Three Types of Visual IR Text based Retrieval Content based Retrieval Semantic based Retrieval Visual IR 44

45 Semantic based Retrieval Concept description Visual IR 45

46 Semantic based Retrieval Systems (1) IMARS (IBM Multimedia Analysis and Retrieval System) Wall Street Journal 2004 Innovation Award in the multimedia category Visual IR 46

47 Semantic based Retrieval Systems (2) SMARTV (SeMantic Analysis and ReTrieval for Video) Intelligent Multimedia Group, Tsinghua University Visual IR 47

48 Semantic based Retrieval How to get concept description Manual annotation (late in 1970s) Social tagging Automatic annotation Visual IR 48

49 Search-based Image Annotation [CVPR 06] 2.4 million images Visual IR 49

50 Annotation Examples (2.4 Million Images) house, castle, church, summer ; garden ; trees, water, sky ; ruins sunset, water ; beach ; zoo ; lake sky ; lake, water, river ; clouds ; trees, mountains, snow ; building summer, mountains snow ; city, sky model ; girl ; studio mountain, lake, water, tree ; hills, valley ; sky house, town, window, village butterfly ; flower ; fly ; frog, water ; tree, ground Visual IR 50

51 Advance The Research on Image Annotation Year: Million Year: Million Year: Billion MSRA (CVPR 06, TPAMI 08) 2.4M Photos from Photo Forums MIT (CVPR 08, TPAMI 08) 80M 32x32 Tiny Images from Search Engines MSRA 2B Image Thumbnails from the Web Visual IR 51

52 Arista Image Search To Annotation [CVPR 10] Visual IR 52

53 Search-based Image Annotation discussion Based On Duplicate Search from 2B Images Perfect for popular images Celebrity, Product, Landmark, Cartoon, Paintings However, not well for personal images When there is no duplicate, the system will fail. Tag quality needs to be improved Based on Similarity Search from 2.4M Images Work well for general concepts, but far from perfect. Fundamental research is needed to bridge the semantic gap. Visual IR 53

54 Annotation -- GWAP The ESP game The Peekaboom Game Visual IR 54

55 What Is Visual IR & The Importance Different Approaches New Techniques/applications Visual IR 55

56 I. Draw a Sketch, Then Search! (Link) Visual IR 56

57 II. Text to Image Translation A lichen Ctenophore Ctenophore in which have have the long long fungus tentacles tentacles and and component flattened flattened is an body body ascomycete A lichen in which the fungus component is an ascomycete Word Image? A domed rock formation where a core of rock has moved upward and pierced through the more brittle overlying strata A cadenced A cadenced trot trot executed executed by by the the horse horse in one in one spot spot A domed rock formation where a core of rock has moved upward and pierced through the more brittle overlying strata Visual IR 57

58 III. Great Success in CV and Multimedia: 3D vision Build Rome in One Day Photo Tourism, SIGGRAPH 2006 Building Rome in a day (Link) Video (Link) (Dubrovnik, 4,619 images, 3,485,717 points) Visual IR 58

59 Build Rome in A Day Visual IR 59

60 Build Rome in A Day Visual IR 60

61 Build Rome in A Day Visual IR 61

62 Build Rome in A Day # Images # Cores Match Time Reconstruction Time Largest Component 150, Hours 8 Hours 2,106 Visual IR 62

63 IV. Great Success in CV and Multimedia: Tracking with 3D Kinect Using Kinect technology How Kinect Works (link) Ultra-Realistic: virtual dressing (link) Visual IR 66

64 Summary Visual information retrieval is a multi-disciplinary research direction involving CV, MM and IR. Internet + data-driven Many traditionally impossible problems become possible How to collect data? How to scale up? How to measure similarity? Semantic gap? Inter play between text data and visual features Many interesting applications As well as challenges Visual IR 67

65 The End! Visual IR 68