Content-based Image and Video Retrieval

Transcription

1 Content-based Image and Video Retrieval Vorlesung, SS 2011 Hazım Kemal Ekenel, Rainer Stiefelhagen, CV-HCI Research Group: Course website:

2 What is Multimedia? Definition from the ACM Special Interest Group on Multimedia Retreat in 2003 More than one media (text, images, audio, video) that are correlated Examples: Time correlated: Video with text transcript of the audio Spatially correlated: Images on a page with associated text A less strict definition: Not Just Text Images Audio Video

3 Multimedia Search Outline Text Search Keywords Image Search Search based on tags (FlickR, FaceBook) Search based on surrounding text (Google) Content based search Audio Search Search based on metadata (itunes) Content based search (MuscleFish, Foote) Video Search Search based on text (Google/UTube) Search based on associated media (Lectures with slides) Search based on content (TrecVid News Search)

4 What is image retrieval? Description Based Image Retrieval (DBIR) Use text for retrieval (from URL, webpage, captions..) Content-Based Image Retrieval (CBIR) Query by Example Query by Image/Video Content

5 Why image retrieval? Huge amounts of images are everywhere: how to manage this data? A picture is worth thousand words Not everything can be described in text Not everything is described in text

6 Why content based image retrieval? Automatic generation of textual annotations for a wide spectrum of images is not feasible. Annotating images manually is a cumbersome and expensive task for large image databases. Manual annotations are often subjective, contextsensitive and incomplete. Google and others use text-based search. Results are not perfect.

7 DBIR vs. CBIR DBIR Fulltext search algorithms are applicable Search results corresponds to image semantics CBIR Automatic index construction Index is objective Manual annotating is hardly feasible Manual annotations are subjective Semantic gap Querying by example is not convenient for a user

8 Image Search - Tags Search over tags associated with images Users manually add Tags to images FlickR FaceBook Find images with tags that match match the query keyword Limitations Tags require human effort to create Tags may be wrong Alia

9 Image Search - Text Use text associated with images for search Search web for images Use surrounding text Text in URL for image filename Text in HTML on page Same as text search Sunset at Rocky Point Example: Google Image Search for Sunset gives Sunset at Rocky Point in Australia Sunset Beach, Oahu Frank Smiles at Sunset Frank Smiles at Sunset Sunset Beach

10 Image Similarity Search Query is an image Search finds similar images Similarity is defined by features of the image Color Content Color Histogram Color Corellogram Image descriptors Gradients at image keypoints Quantize for Visual words Faces Detection Recognition Query Image Search Results

11 Image Search Faces Face Detection Search for all images with faces Find faces in images Ex: Google advance search for images with faces Photo Collection Faces in Photo Collection Face Detection

12 Face Recognition Image Search Faces Search for all images of a particular person

13 Video Search News Programs Find segments of news on a topic of interest Find news story Find shots within story TRECVID Sponsored by NIST (National Institute of Standards) Data base of 60 hours of news video (ABC, NBC) in 2004 similar content other years Task user has 15 minutes to find shots relevant to a topic Example Topics Find shots of a hockey rink with at least one of the nets fully visible from some point of view Find shots zooming in on the US Capitol dome Find shots of Saddam Hussein

14 Video Search Whole Video Search for an entire video Search using surrounding text and/or content Example: Google/YouTube Search for sunset

15 Levels of image/video retrieval Level 1: Based on color, texture, shape features Images are compared based on low-level features, no semantics involved A lot of research done, is a feasible task Level 2: Bring semantic meanings into the search E. g. identifying human beings, horses, trees, beaches Requires retrieval techniques of level 1 Level 3: Retrieval with abstract and subjective attributes Find pictures of a particular birthday celebration Find a picture of a happy beautiful woman Requires retrieval techniques of level 2 and very complex logic

16 Common components of CBIR system indexation image feature extraction database retrieval query feature extraction comparison result Relevance feedback: query refinement

17 Sample Application I: like.com Similarity-based Shopping Content-based image and video retrieval 17

18 Sample Application I: like.com Content-based image and video retrieval 18

19 Sample Application II: Viewdle.com Face Search in Videos, used by Reuters Content-based image and video retrieval 19

20 Multimedia Grand Challenge Content-based image and video retrieval 20

21 Multimedia Grand Challenge Topics Accenture Challenge: Analysis of Video Footage Captured in Uncontrolled Environments CeWe Challenge: The Next Generation of Tangible Multimedia Products Current TV Challenge: Media Production in the Age of Community Google Challenge: Robust, As-Accurate-As-Human Genre Classification for Video HP Challenge: Robust Identification of Informative Multimedia Content in Web Pages Content-based image and video retrieval 21

22 Multimedia Grand Challenge Topics Nokia Challenge: Where was this Photo Taken, and How? Radvision Challenge: Video Conferencing To Surpass In-Person Meeting Experience Radvision Challenge: Real-time Data Collaboration Adaptation for Multi-Device Video Conferencing Yahoo! Challenge: Robust Automatic Segmentation of Video According to Narrative Themes Yahoo! Challenge: Robust Clustering Guided by User Intent in Image Search Content-based image and video retrieval 22

23 TRECVID Evaluations A video retrieval evaluation campaign from the National Institute of Standards and Technology (NIST), US. Promote progress in content-based analysis, detection, retrieval in large amount of digital video Combine multiple errorful sources of evidence Achieve greater effectiveness, speed, and usability Confront systems with unfiltered data and realistic tasks Measure systems against human abilities Content-based image and video retrieval 23

24 VideOlympics Content-based image and video retrieval 24

25 Overview of the Lecture (tentative) Datum Thema Introduction / overview Visual Descriptors 25.4 Keine Vorlesung (Ostern) Local Descriptors & Segmentation Machine Learning for CBIR/Classifiers: SVM, knn, Shot boundary dectection and TV genre classification Content-based image and video analysis (TrecVid) Content-based image and video analysis (TrecVid) II Person / face detection and recognition 13.6 Keine Vorlesung (Pfingsten) Copy/near duplicate detection Event recognition Image and Video semantics Search: Automatic and Interactive search / relevance feedback / Tools and software libraries for image and video analysis Content-based image and video retrieval 25

26 Image Segmentation Identification of homogenous region in the image Partition an image into meaningful regions with respect to a particular application Content-based image and video retrieval 26

27 Descriptors -Color Color spaces RGB HSV YCbCr Color-based features Color Histogram Color Moments Color Correlogram... Content-based image and video retrieval 27

28 Descriptors -Texture One or more basic local patterns that are repeated in a periodic manner Texture with repeated local patterns Local pattern Content-based image and video retrieval 28

29 Texture-based Features Tamura Features Coarseness Contrast Directionality Co-occurrence Matrix Edge Histogram Homogeneous Texture Descriptor Content-based image and video retrieval 29

30 Descriptors -Shape Content-based image and video retrieval 30

31 Machine Learning for CBIR Overview of classification methods Bayesian, nearest neighbor, neural networks, support vector machines Clustering & relevant machine learning techniques NN SVM Content-based image and video retrieval 31

32 Shot boundary detection Cut Fade Out/In Dissolve Content-based image and video retrieval 32

33 TV Genre Classification Content-based image and video retrieval 33

34 TV Genre Classification Content-based image and video retrieval 34

35 Content-based Image and Video Analysis IBM System Content-based image and video retrieval 35

36 Content-based Image and Video Analysis UKA System Content-based image and video retrieval 36

37 Automatic person tagging in photo albums & social networks Content-based image and video retrieval 37

38 Person/face detection and recognition in videos Content-based image and video retrieval 38

39 Event recognition Content-based image and video retrieval 39

40 Surveillance Event Detection Content-based image and video retrieval 40

41 Copy/near duplicate detection Near duplicate image: An image is called a near-duplicate of a reference image if it is close, according to some defined measure, to the reference image Near duplicate image cases: Being perceptually identical Being images of the same 3D scene Content-based image and video retrieval 41

42 Copy/near duplicate detection - Applications Compression of video files Detection of doubles in large image databases Reduction of required disk space Detection of copyright violations Grouping images in search results Content-based image and video retrieval 42

43 Grouping images in search results Results for the query flight of a bee using Google Image Search Content-based image and video retrieval 43

44 Grouping images in search results Grouped Results Greater diversity of images on the first page! Content-based image and video retrieval 44

45 Image and Video semantics Low-level visual features such as color, texture, and shapes can be extracted from images to represent and index image content. However, they are not completely descriptive for meaningful retrieval! Level 1: Based on color, texture, shape features Images are compared based on low-level features, no semantics involved A lot of research done, is a feasible task Level 2: Bring semantic meanings into the search E. g. identifying human beings, horses, trees, beaches Requires retrieval techniques of level 1 Content-based image and video retrieval 45

46 Search: Automatic and Interactive search / relevance feedback Content-based image and video retrieval 46

47 Sample Search Topics Content-based image and video retrieval 47

48 Databases, tools and software libraries for image and video analysis Content-based image and video retrieval 48

49 Organizational VL-Website: Termine Montags, 11:30 13:00 Uhr, Raum -101 Vertiefungsgebiete 13 Anthropomatik 14 Kognitive Systeme Dozenten Hazım Kemal Ekenel, Geb , Zi. 231 Rainer Stiefelhagen, Geb , Zi. 230 Content-based image and video retrieval 49

50 Vorbereitung Organizational (2) Vorlesungsfolien auf Website ergänzend Papers lesen (werden online bereitgestellt) L: coursemember PW: 321meins Fragen, falls irgendwas unklar ist! In der Vorlesung und/oder Termin Ergänzend: Seminar Inhaltsbasierte Bild- und Videoanalyse Content-based image and video retrieval 50

51 Seminar Inhaltsbasierte Bild- und Videoanalyse Mittwochs, 15:45 Uhr, Raum 201 Themen wie in der Vorlesung Vortrag und Folien sollten auf Englisch gehalten werden Kontakt: Hazım Kemal Ekenel, Geb , Zi. 231 Rainer Stiefelhagen, Geb , Zi. 230 Content-based image and video retrieval 51

52 Forschungsgruppe MS-MMI Forschungsgruppen MS-MMI am KIT und Fraunhofer IOSB, Ziel: Neue Verfahren zur visuellen Wahrnehmung von Menschen Nutzung für wahrnehmende Computer, Roboter, Räume Forschungsthemen: Personenlokalisierung Wo? Personenidentifikation / Gesichtserkennung Wer? Erkennung von Gesten, Handlungen, Ereignissen Was? Erkennung von Blickrichtung und Aufmerksamkeit Wohin? Anwendungen Mensch-Roboter-Interaktion Intelligente aufmerksame Umgebungen Videoanalyse und Retrieval Weitere Bereiche: Sicherheit, Kundenanalyse 52

53 CBIR MS-MMI Quaero: A research & development program to build multimedia search engine Started in 2008, funded by French State, 5-year research and development program Our part: Person retrieval TV genre classification High-level feature detection Content-based image and video retrieval 53

54 Person Retrieval Content-based image and video retrieval 54