Audio-Visual Content Indexing, Filtering, and Adaptation

Transcription

1 Audio-Visual Content Indexing, Filtering, and Adaptation Shih-Fu Chang Digital Video and Multimedia Group ADVENT University-Industry Consortium Columbia University 10/12/ /2001 S.-F. Chang Columbia U. 1

2 Challenge: tools/systems for searching and filtering document image/video Content Archives WWW push Filtering & adaptation (preference) search author pull organization presentation Important Issues: Information Overload, Limited User Attention, Time-Sensitive Heterogeneous Platforms Example Products: AltaVista, Google, DoCoMo/IBM, NewsTake, TiVo/Philips, PC DVR 10/2001 S.-F. Chang Columbia U. 2

3 MPEG-7 and issues ISO/IEC Multimedia Content Description Interface Feature Extraction MPEG-7 Description Search/Filtering Application Issues: Active research in analysis and indexing System issues: delivery, compression What applications? 10/2001 S.-F. Chang Columbia U. 3

4 Re-examine Content Production/Usage Chain text metadata feature extraction pattern recognition indexing speech/text recognition textual search similarity matching structure browsing event/object search summary Production & post-production A-V Data Analysis Interaction Viewer 10/2001 S.-F. Chang Columbia U. 4

5 Types of Content Indexing Reverse engineering Production structure parsing (shot, camera) Metadata preservation Format, Production, Credits, etc. Feature measurement audio-visual, spatio-temporal objects and features Content recognition and organization Semantic meaning description Scene grouping and skimming 10/2001 S.-F. Chang Columbia U. 5

6 When do we need automated tools Conditions for high impact Metadata that s not available from production Work that humans are not good at (e.g., low-level computation) Large volume, low individual value Time sensitive Acceptable performance Less Promising Areas Content from digital production tools with metadata Prime content that can afford manual solutions 10/2001 S.-F. Chang Columbia U. 6

7 Case 1: Live Sports Video Filtering and Navigation Linear/Passive Video (Images excluded) Interactive Video Highlights Pitches Runs By Player By Time Set your Own Save Delete Edit (images excluded) Time sensitive interest Massive production and audience Time compressibility Temporal structure and production rules

8 Beyond Ringer and Clip Download Services: Messaging Localized information Media/game download Multi-person games TV phone On-site purchase Time-sensitive short video messages suiting personal needs Image showing sports video on mobile handset excluded

9 Regular Structure and Views in Sports Video Entire Tennis Video Set Serve Game Closeup, Crowd Commercials Elementary Shots 10/2001 S.-F. Chang Columbia U. 9

10 Regular set of views Pitching Close-up pitcher Catcher (Images excluded) (Images excluded) (Images excluded) Base hit First base Full field (Images excluded) (Images excluded) (Images excluded) Important issues: canonical view beginning of recurrent semantic unit view transition pattern types of events 10/2001 S.-F. Chang Columbia U. 10

11 Real-Time Sports Video Parsing Detect and classify recurrent semantic units Real-time processing simple features compressed-domain processing Combine global-level and object-level classification (Chang, Zhong, Kumar 01) 10/2001 S.-F. Chang Columbia U. 11

12 Detecting recurrent semantic units: Serve Compressed video Images showing different views of tennis serve excluded Every I/P frames Shot Detection Key frames/shot Adaptive Color Filter Down-sampled I/P frames Object level verification Color, lines, player Canonical View Detection (Chang, Zhong, Kumar 00) 10/2001 S.-F. Chang Columbia U. 12

13 Learning spatio-temporal rules Level 1: Object Pitching Level 2: Object-part Ground Pitcher Batter Level 3: Perceptual Area Grass Sand Level 4: Region Regions Regions Regions Regions Image showing baseball pitch excluded 10/2001 S.-F. Chang Columbia U. 13

14 Systematic learning of object rules Initial input tagging training images scene model Automated segmentation Visual Apprentice (with A. Jaimes) Output: Classifiers and rules classifiers features rules (Jaimes and Chang 99) 10/2001 S.-F. Chang Columbia U. 14

15 Application: Time-Sensitive Mobile Video Content Adaptive Streaming Server or Gateway Live Video Event Detection Content Adaptation Encode/ Mux Buffer Structure Analysis Client Streaming User Demux/ Decoder Buffer 10/2001 S.-F. Chang Columbia U. 15

16 Content Adaptive Streaming important segments: video mode non-important segments: still frame + audio + captions adaptive video rate channel rate time input channel output accumulated rate time t 0 t 1 10/2001 S.-F. Chang Columbia U. 16

17 Recurrent semantic units in soccer video? (sec.) play 0.0 break Game a sequence of play and break segments Play/break No canonical views/events Sporadic events (start and end) Shot boundary play boundary 10/2001 S.-F. Chang Columbia U. 17

18 Heuristic Model: Features Views Structure (Image Excluded) global zoom-in close-up Learning Phase Grass Color Classifier (unsupervised) View classification (unsupervised) Threshold Adaptation Segmentation Rules Sampled frames Decision Phase View Classification Play-Break Temporal Segmentation Structure info

19 Modeling Transitions with HMM Segment Features (color, motion, objects) Train HMM families Break Play Test Data ML Detection + Linear Regression Play-Break Transition Model (Dynamic Programming) Play-Break Classification Rate: over different games 87%, 86%, 85%, 73% 10/2001 S.-F. Chang Columbia U. 19

20 Case 2: Long Programs with Loose Structures 2:20 transient 4:04 5:22 Images of shots excluded Goal: Scene/Topic Segmentation, Browsing, Preview Challenge: Diverse production styles and views 10/2001 S.-F. Chang Columbia U. 20

21 Domain Consideration Film: Does not satisfy impact conditions Not much value except archived collection in libraries less popular selection in PVR/STB Challenging, rich styles and media Fun 10/2001 S.-F. Chang Columbia U. 21

22 Problems and Approaches Explore production models Explore multimedia integration Incorporate structural and special cues Explore viewer s perceptual models Goal: Computable features/theories + models Scene structures and summaries 10/2001 S.-F. Chang Columbia U. 22

23 Production Constraints Video Scenes Camera placement [180 o rule] overlapping background causes chromatic consistency Lighting/chromaticity continuity 10/2001 S.-F. Chang Columbia U. 23

24 Audition Psychology Audio Scenes A. Bregman 90: Auditory Scene Analysis Unrelated sounds seldom begin/end at the same time. Sounds from the same source change properties smoothly and gradually over time. Changes in acoustic states affect all components of the sound (e.g., walking away from a ringing bell) Human perception uses long term grouping (e.g., a series of footsteps). An audio scene change is said to occur when majority of the sound sources change. 10/2001 S.-F. Chang Columbia U. 24

25 Explore Audio-visual association Complementary audio-visual structures first hour of Blade Runner video: 28 scenes audio only: 33 scenes audio/video agreement: 24 video change only: silent or transient scenes audio change only: mood/state change singleton scene boundaries video scene boundary audio scene boundary Sundaram & Chang ICASSP 00 10/2001 S.-F. Chang Columbia U. 25

26 Mixing audio-visual scenes 65% 21% Video scene boundary Audio scene boundary 10% 4% sense & sensibility 10/2001 S.-F. Chang Columbia U. 26

27 Film: Common Scene Structures 4 common scene types in film Progressive or Coherent: same location, similar camera takes, consistent audio Images of shots excluded time (Demo) 10/2001 S.-F. Chang Columbia U. 27

28 Common Scene Types (2) Transient : different locations, dissimilar camera takes, evolving audio Images of shots excluded MTV-type: fast, dissimilar shots, consistent audio (Demo) 10/2001 S.-F. Chang Columbia U. 28

29 Common Scene Types (3) Dialog: repeating structures, consistent audio Images of shots excluded 10/2001 S.-F. Chang Columbia U. 29

30 Computable Scene Detection Architecture data audio features shot detection Audio-scenes silence video-scenes fusion structure computable scenes 10/2001 S.-F. Chang Columbia U. 30

31 The Viewer/Listener Model memory t o to : decision instant attention span time Memory (M): Net amount of data remaining in viwer s memory, e.g., 32 sec. Attention span (AS): The most recent data occupying user s attention, e.g., 16 sec. Measure the group, long-term coherence between M and AS (Kender and Yeo 98, Sundaram and Chang 00) 10/2001 S.-F. Chang Columbia U. 31

32 Video Scene Segmentation T m shot key-frames T as f a t f b Group coherence based on viewer memory model Rab (, ) = ( 1 dab (, )) f f ( 1 tt ) a b m C( to) = R( a, b) C max( to) a Tas b { Tm \ Tas }

33 Audio Scene Segmentation An audio scene is a collection of sources compute multi-scale features Component decomposition Detect model correlation decay slope change? trend majority vote sinusoidal (11 speech/music features) noise (sundaram and chang 00)

34 Detecting Dialog Structure cyclic autocorrelation N 1 1 ( n ) d ( o i, o m o d ( i + n, N )) N i = 0 (images excluded) frequency = Accuracy: 80%/94% - 91%/100% precision recall 10/2001 S.-F. Chang Columbia U. 34

35 Aligning Scene Boundaries Audio: source correlation Video: group coherence ambiguity window aligned : video boundary : audio boundary : windows

36 Fusion Rules Synchronized a-scene and v-scene scene Exceptions: Ignore scenes within structures Ignore (weak v- scene, weak a-scene) Add (strong v-scene, silence) Require tighter synchronization if (weak v, normal a) silence structure A-scene boundary V-scene boundary Weak V-scene boundary 10/2001 S.-F. Chang Columbia U. 36

37 Open Issues Video Use higher-level cognitive models Currently focus on group coherence between groups Syntax structure within scenes not considered Potential use of information theory and temporal modeling Audio Source separation (music, speech, noise, etc) Scene-level organization Visualization, summarization, matching 10/2001 S.-F. Chang Columbia U. 37

38 Scene Skimming Reduce to a short-time preview syntax preserved Find the skim with the highest utility, satisfying the production syntax constraints 10/2001 S.-F. Chang Columbia U. 38

39 Utility of a shot Explore Perceptual Model and Scene Syntax Image excluded Image excluded (a) (b) Is comprehension time related to the visual spatio-temporal complexity of the shot? The presence of detail robs a shot of its screen time. 10/2001 S.-F. Chang Columbia U. 39

40 Shot complexity and comprehension time Represent shot by its key-frame A shot is selected at random The subject was asked to correctly answer four questions in minimum time: Who What When Where Why was not asked

41 Time-complexity relationship Plot of average time vs. complexity shows two bounds U ( c) = 2.40c b L ( c) = 0.61c b time U b L b comprehension time increases with complexity complexity 10/2001 S.-F. Chang Columbia U. 41

42 Shot condensation 4.5 original shot U b time reduce to upper bound L b 0 complexity /2001 S.-F. Chang Columbia U. 42

43 Film syntax The specific arrangement of shots so as to bring out their mutual relationship. [ sharff 82 ]. Minimum number of shots in a scene The particular ordering of the shots The specific duration of the shots, to direct viewer attention Changing the scale of the shots Film makers think in terms of phrases of shots and not individual shots. 10/2001 S.-F. Chang Columbia U. 43

44 The progressive phrase Two well chosen shots will create expectations of the development of narrative; the third well-chosen shot will resolve those expectations. [ sharff 82 ]. Hence, a phrase (a group of shots) must at least have three shots. Maximal compression: eliminate all the dark shots. 10/2001 S.-F. Chang Columbia U. 44

45 The dialog Depicting a conversation between m people requires 3m shots. [ sharff 82 ]. Hence, a dialog must at least have six shots Maximal compression: eliminate all the dark shots. 10/2001 S.-F. Chang Columbia U. 45

46 Utility Optimization Framework Objective function for each skim Utility of Skim Rhythm Penalty Dropped shot penalty Find the skim with the minimal cost, satisfying the syntax constraints syntax preserved 10/2001 S.-F. Chang Columbia U. 46

47 Scene Skimming Original-1 Original-2 10:3 time reduction 5:1 time reduction Skim-1 Skim-2 10/2001 S.-F. Chang Columbia U. 47

48 Echo Video Digital Library & Remote Medicine Echo Video Domain Knowledge View Segmentation KeyFrame, Heart Beat summary Diagnosis Reports Transition Model View Recognizer (Ebadollahi and Chang 00) Object Annotation Adaptive network delivery Content search

49 Conclusions Analysis and interaction Production model Media Integration Viewer model Applications Time-sensitive filtering Adaptive delivery Skimming Navigation 10/2001 S.-F. Chang Columbia U. 49

50 Acknowledgements Live Sports Filtering: Di Zhong and Raj Kumar Soccer Video Parsing: Lexing Xie, Peng Xu, Ajay Divakaran, Anthony Vetro, Huifang Sun Scene Segmentation and Skimming: Hari Sundaram Medical Video: Shahram Ebadollahi 10/2001 S.-F. Chang Columbia U. 50

51 More Information Columbia Digital Video/Multimedia Group ADVENT Industry/University Consortium /2001 S.-F. Chang Columbia U. 51