Lecture 12: Video Representation, Summarisation, and Query

Transcription

1 Lecture 12: Video Representation, Summarisation, and Query Dr Jing Chen NICTA & CSE UNSW CS9519 Multimedia Systems S Last week Structure of video Frame Shot Scene Story Why video structure analysis Transition effect between shots Shot segmentation Scene segmentation COMP9519 Multimedia Systems Lecture 12 Slide 2 J Chen Last Week-- A diagram of video structure Last Week -- Structure of Video Sequence video data story scene1 scene2 Scenes (stories) scene #1 scene #2... scene #8 shots... shot #1 shot #2 shot #3 shot #4 shot #19 shot #20 shot #21 keyframe keyframe keyframe keyframe keyframe keyframe keyframe... shot1 shot2 shot3 shot4 shot5 frames1,2, N * H.B.Kang, Video Abstraction Techniques For A Digital Library COMP9519 Multimedia Systems Lecture 12 Slide 3 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 4 J Chen

2 Last Week- Why video structure analysis? Typical video retrieval system diagram Last Week -- Types of Shot Transition Video data Segmentation Key frame computation Video browsing Color Motion Shape Cut Fade Dissolve Wipe Feature extraction Some applications: Indexing and browsing Non-linear editing Event detection Video query, retrieval and production * Yan Liu & Fei Li COMP9519 Multimedia Systems Lecture 12 Slide 5 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 6 J Chen Last Week -- video shot segmentation methods Spatial domain approaches Pixel domain -- Frame differencing Motion compensated frame differencing Histograms (global, joint and local) Model driven Compressed domain approaches DCT coefficients DC terms MB types of B frame Last Week -- Thresholding vs Clustering based shot segmentation Thresholding Local decision (based on the info of very few frames) Thresholds are typically highly sensitive to the type of input video Clustering View shot segmentation as a k-class unsupervised clustering problem Assign frames to one of the k classes via k-means Global decision Not only eliminates the need for threshold setting but also allows multiple features to be used simultaneously to improve the performance COMP9519 Multimedia Systems Lecture 12 Slide 7 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 8 J Chen

3 Last Week Scene Segmentation Reference frame (R-Frame) Last Week A diagram of scene segmentation with scene transition graph Distance between two shots R-frames for shot i R Ai R Bi R Ci... R-frames for shot j R Aj R Bj R Cj R Cj... Clustering of Shots Visual Dissimilarity COMP9519 Multimedia Systems Lecture 12 Slide 9 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 10 J Chen Outline We have covered Feature extraction Image/video retrieval system Video structure analysis Q: how do we visualise the results How to browse large video files? How to present the retrieval results? Requirements Under the constraint of the available screen space Efficiently and user friendly Browse a large number of videos (1000s) Outline Storyboards Hierarchical browser Video directory browser Video summary Video skimming Thumbnail images Browsing many clips Concept of video query COMP9519 Multimedia Systems Lecture 12 Slide 11 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 12 J Chen

4 Video browser/retrieval interface Traditional interfaces: storyboards How to show the temporal variation in the video? child storyboard animated gifs skims hierarchical browser Visual Browsing Example 1 Lateral browser surrounding temporal browser, courtesy of Imperial College London * H. Sundaram COMP9519 Multimedia Systems Lecture 12 Slide 13 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 14 J Chen Carnegie Mellon Visual Browsing Example 2 Best people shots, Carnegie Mellon Informedia system: Storyboards Say this is the result of a search for video data.. What are the problems here? * H. Sundaram COMP9519 Multimedia Systems Lecture 12 Slide 15 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 16 J Chen

5 Child storyboards Once we click on each thumbnail, this action pops up a child storyboard. the storyboard shows the temporal behavior of the video 15 min of video. still no audio! clicking plays the shot how to select key-frames? clustering Key-frame Selection Considerations Flexibility (number and level) Fidelity (content comprehension) Approaches Fixed number, fixed spacing First/last frame, clean frame Difference, motion Clustering Cluster all frames using complete link algorithm (FXPAL) Use the maximum of the pair-wise distances between frames to determine the inter-cluster similarity, and produces small, tightly bound clusters * H. Sundaram * H. Sundaram COMP9519 Multimedia Systems Lecture 12 Slide 17 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 18 J Chen More Advance Browsing and Summarization Schemes Key-frame Based Hierarchical Video Browser Hierarchical browser Video directory browser Video summary Video skimming Thumbnail images Browsing many clips COMP9519 Multimedia Systems Lecture 12 Slide 19 J Chen * H.J.Zhang et al, Video Parsing, Retrieval and Browsing: An Integrated and Content-Based Solution, ACM MM 2005 COMP9519 Multimedia Systems Lecture 12 Slide 20 J Chen

6 Visual Summaries for Video News Cluster at the Scene Level Data Structure and Browser Layout for Key- Frame Based Hierarchical Browser * ClassView paper COMP9519 Multimedia Systems Lecture 12 Slide 21 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 22 J Chen Scene Transition Graph Web-based Video Directory Browser (FX- PAL) COMP9519 Multimedia Systems Lecture 12 Slide 23 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 24 J Chen

7 Key Frames Attached to Time Scale The positions of the key-frames are marked by blue triangles along a mouse-sensitive time scale adjacent to the key-frame As the mouse moves over the time scale, the key-frame for the corresponding time is shown and the triangle for that keyframe turns red. Mapping Confidence Scores to Gray Levels Metadata: annotation of audio/video Translate metadata values into confidence score Present the confidence score by levels of gray High confidence areas are marked in black Areas of lower confidence fade progressively to white COMP9519 Multimedia Systems Lecture 12 Slide 25 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 26 J Chen Confidence Score Display Features can be selected from a pull-down menu Metadata media player COMP9519 Multimedia Systems Lecture 12 Slide 27 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 28 J Chen

8 Demo of video directory browser Video Manga (FXPAL) Summary of video Present key-frames (images) in a compact, visually pleasing display Given : 2D space constraints, Key-frame set importance measures What is the best display layout? Comic book concept Issues: Time Order vs. Layout Order Preserve high-level structures Importance Measures? COMP9519 Multimedia Systems Lecture 12 Slide 29 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 30 J Chen Discard Key-frames (in Video Manga) Key frame extraction Starting point mentioned earlier Too many key-frames! Calculate an importance score for each segment based on its rarity and duration. Longer shots are preferred because they are likely to be important in the video. Repeated shots receive lower scores They do not add much to the summary even if they are long. Segments with an importance score higher than a threshold are selected to generate a pictorial summary. For each segment chosen, the frame nearest the center of the segment is extracted as a representative key-frame. Frames are sized according to the importance measure of their originating segments Higher importance segments are represented with larger key-frames. Key-frame Packing Arrange the key frames in a logical order Fit the available space efficiently COMP9519 Multimedia Systems Lecture 12 Slide 31 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 32 J Chen

9 Video Manga (FX-PAL) Web-based Interactive Video Summary Browse the video based on either key-frames or the timeline Pop up captions as the mouse moves over an image Space saving COMP9519 Multimedia Systems Lecture 12 Slide 33 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 34 J Chen Playing video Clicking on a key-frame starts video playback from the beginning of that segment Demo of Video Manga COMP9519 Multimedia Systems Lecture 12 Slide 35 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 36 J Chen

10 Video skims A temporal, multimedia abstraction that incorporates both video and audio information from a longer source. Goal: to communicate the essential content of a video in an order of magnitude less time. Generalized Video Skim Creation Process (CMU-Informedia) COMP9519 Multimedia Systems Lecture 12 Slide 37 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 38 J Chen Audio/video alignment in skims Default skim (DEF) Dropping video at regular intervals Image centric skim (IMG) Emphasizes visual content, decomposing the source into component shots, detecting important objects, such as faces and text, and identifying structural motion within a shot Audio-centric skim (AUD) Derives solely from audio information. Automatic speech recognition and alignment techniques register the audio track to the video s text transcript. integrated best skim (BOTH) merges the image centric and audiocentric approaches while maintaining moderate audio/video synchrony. Top-rated audio regions are selected as in the AUD skim The audio is then augmented with imagery selected using IMG heuristics from a temporal window extending five seconds before and after the audio region. Skim Video : Extracting Significant Content Original Video (1100 frames) Skim Video (78 frames) COMP9519 Multimedia Systems Lecture 12 Slide 39 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 40 J Chen

11 The informedia skims Temporal abstraction motivates viewers Time compression preserve essential data Segments with matching words are combined Each segment is extended based on the goodness scores of the ending point, until the time budget is reached Issues: Choppy presentation Temporal syntax (e.g., dialog) Early cutout of sentence, scene, audio Thumbnail images *Sundaram COMP9519 Multimedia Systems Lecture 12 Slide 41 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 42 J Chen Empirical Study Into Thumbnail Images Text-based Result List COMP9519 Multimedia Systems Lecture 12 Slide 43 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 44 J Chen

12 Naïve Thumbnail List (Uses First Shot Image) Query-based Thumbnail Result List COMP9519 Multimedia Systems Lecture 12 Slide 45 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 46 J Chen Query-based Thumbnail Selection Process 1000 Thumbnail Study Results Decompose video segment into shots. 2. Compute representative frame for each shot. 500 Time (secs.) 25 Titles Browsed 0 Text First Query 0 Text First Query Locate query scoring words (shown by arrows). 4. Use frame from highest scoring shot Score (max = 400) 7 5 1(terrible)- 9(wonderful) COMP9519 Multimedia Systems Lecture 12 Slide 47 J Chen 0 1 COMP9519 Multimedia Systems Lecture 12 Slide 48 J Chen Text First Query Text First Query Copyright 2003 Michael G. Christel 48 CarnegieMellon

13 Empirical Study Summary* Thumbnail View with Query Relevance Bar Significant performance improvements for query-based thumbnail treatment over other two treatments Subjective satisfaction significantly greater for query-based thumbnail treatment Subjects could not identify differences between thumbnail treatments, but their performance definitely showed differences! *Christel, M., Winkler, D., and Taylor, C.R. Improving Access to a Digital Video Library. In Human-Computer Interaction: INTERACT97, Chapman & Hall, London, 1997, COMP9519 Multimedia Systems Lecture 12 Slide 49 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 50 J Chen Copyright 2003 Michael G. Christel 50 CarnegieMellon Close-up of Thumbnail with Relevance Bar Match bars Relevance score of [0, 100] This document has score of 30 Color-coded scoring words: Asylum contributes some, rights a bit, refugee contributes 50% * Christel Query-based thumbnail Shortcut to storyboard COMP9519 Multimedia Systems Lecture 12 Slide 51 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 52 J Chen

14 Using Match Info to Reduce Storyboard Size Browsing many files (FXPAL) COMP9519 Multimedia Systems Lecture 12 Slide 53 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 54 J Chen Video editing user interface (FXPAL) The top display lets users select clips from the raw video. The bottom display lets the users organize the clips along the timeline and change the lengths of the clips. Flipping through images in a pile Cluster all clips by the similarity of their color histograms Place similar clips into the same pile Each clip is represented by one key-frame in a pile The clips are stacked in temporal order. COMP9519 Multimedia Systems Lecture 12 Slide 55 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 56 J Chen

15 Expanding a pile of video clips To see the additional images in a pile, the user can expand the pile by clicking on it. The current display is faded out and the images of the pile are shown in an area in the middle of the faded out display. The timeline displays the coverage of the expanded view in light gray and the coverage of the pile in a darker color as before. Outline Storyboards Hierarchical browser Video directory browser Video summary Video skimming Thumbnail images Browsing many clips Concept of video query COMP9519 Multimedia Systems Lecture 12 Slide 57 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 58 J Chen Video Query Feature-based Similarity Search Query types Point query specifies a point in the data space and retrieves all point objects in the database with identical coordinates: Range query Given a query point Q, a distance r, and a distance function M, retrieve all points P from the database, which have a distance smaller or equal to r from Q according to M: Nearest neighbor query Given a query point Q, retrieve the nearest neighbor point P from the database, ie, find object K-nearest neighbor query Given a query point, return the k nearest neighbor points COMP9519 Multimedia Systems Lecture 12 Slide 59 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 60 J Chen

16 Distance functions Euclidean (L2) Manhattan (L1) Maximum (L ) Weighted Euclidean Weighted maximum Ellipsoid where W is a positive definite similarity matrix Query without index Sequential scan Sequentially scan through all records in the database Size of database Storage cost is O(dn), where d is the dimensionality of a record, n is the size of the DB, assuming floating point data The time to process a query is O(dn) Infeasible for a large database with millions of records! Q: a better solution to search? Index COMP9519 Multimedia Systems Lecture 12 Slide 61 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 62 J Chen Conclusion Storyboards Hierarchical browser Video directory browser Video summary Video skimming Thumbnail images Browsing many clips Concept of video query Some references Chapter 4 of Book Multimedia Information Retrieval and Management An Interactive Comic Book Presentation for Exploring Video. John Boreczky, Andreas Girgensohn, Gene Golovchinsky, and Shingo Uchihashi in CHI 2000 Conference Proceedings, ACM Press, pp , 2000., April 1, 2000 Christel, M., Smith, M., Taylor, C.R., and Winkler, D. Evolving Video Skims into Useful Multimedia Abstractions. In Proc. ACM CHI 98 (Los Angeles, CA, April 1998), ACM Press, Christel, M., Winkler, D., and Taylor, C.R. Improving Access to a Digital Video Library. In Human-Computer Interaction: INTERACT97, Chapman & Hall, London, 1997, COMP9519 Multimedia Systems Lecture 12 Slide 63 J Chen COMP9519 Multimedia Systems Lecture 12 Slide 64 J Chen