Multimedia Databases. Wolf-Tilo Balke Younès Ghammad Institut für Informationssysteme Technische Universität Braunschweig

Size: px

Start display at page:

Download "Multimedia Databases. Wolf-Tilo Balke Younès Ghammad Institut für Informationssysteme Technische Universität Braunschweig"

Rudolph Stone
5 years ago
Views:

1 Multimedia Databases Wolf-Tilo Balke Younès Ghammad Institut für Informationssysteme Technische Universität Braunschweig

Previous Lecture Audio Retrieval - Query

2 Previous Lecture Audio Retrieval - Query by Humming - Melody: Representation and Matching Parsons-Codes Dynamic Time Warping - Hidden Markov Models (Introduction) Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 2

3 9 Video Retrieval 9 Video Retrieval 9.1 Hidden Markov Models (continued from last lecture) 9.2 Introduction into Video Retrieval Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 3

4 9.1 Hidden Markov Model A HMM has at any time additional timeinvariant observation probabilities A HMM consists of A homogeneous Markov process with state set Transition probabilities Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 4

5 9.1 Hidden Markov Model Start distribution Stochastic process of observations with basic sets And observation probabilities of observation o k in state q j Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 5

6 9.1 HMM Example Observations: Type 1 Type 2 Type 3 Type 4 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 6

7 9.1 Evaluation Observation probability Given the observation sequence and a fixed HMM λ How high is the probability that λ has generated the observation sequence? =? Important for selecting between different models Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 7

8 9.1 Evaluation Let Then: be a state sequence Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 8

9 9.1 Evaluation Furthermore is also valid And is valid for Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 9

10 9.1 Evaluation Thus the total probability for observation O is: Substituting in our previous results we obtain: Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 10

9.1 Evaluation Most probable state sequence Given the observation sequence and a fixed HMM λ What is the state sequence which generates the observation sequence

11 9.1 Evaluation Most probable state sequence Given the observation sequence and a fixed HMM λ What is the state sequence which generates the observation sequence o, with the highest probability? Maximum likelihood estimator: maximize Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 11

12 9.1 Evaluation Because we know that and that is constant for fixed sequences of observations, instead of maximizing we can also maximize Definition: Maximal for the most probable path leading to the state q i (at time t) Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 12

13 9.1 Evaluation is valid Therefore corresponds to a state sequence assuming that the occurrence of the observation sequence O, is the most likely Such a path can be constructed in steps by means of dynamic programming, via Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 13

14 9.1 Evaluation The corresponding algorithm is the Viterbi algorithm (Viterbi, 1967) Initial step: for set For inductively set Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 14

15 9.1 Evaluation Termination: Recursive path identification for t [1 : T - 1] set for probability p: Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 15

16 9.1 Evaluation Given a fixed HMM, for each sequence of observations, the Viterbi algorithm provides a sequence of states which has most probably caused the observations (Maximum likelihood estimator) Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 16

9.1 Training of HMMs Problem: Transition-, observationand start probabilities are often unknown Idea: training the parameters of the HMM λ Given an observation sequence

17 9.1 Training of HMMs Problem: Transition-, observationand start probabilities are often unknown Idea: training the parameters of the HMM λ Given an observation sequence training sequence Task: determine the model parameters λ = (A, B, π) to maximize Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 17

18 9.1 Training of HMMs The training sequence should not be too short The maximization of the probability leads to a high-dimensional optimization problem Solved e.g., through the Baum-Welch algorithm, which calculates a local optimum (Baum and others, 1970) Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 18

19 9.1 Training of HMMs Baum-Welch algorithm: Begin with an initial estimate of parameters: either arbitrary or based on additional knowledge Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 19

20 9.1 Training of HMMs These statistics can be used for an iterative reestimation of the parameters Define forward variables: Then: and is valid, for Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 20

21 9.1 Training of HMMs And backward variables: for and for Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 21

22 9.1 Training of HMMs Then, the probability to be in q i at time t if o has been observed, is: (conditional probability) Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 22

23 9.1 Training of HMMs And the probability to be at time t in state q i and at time t+1 in state q j is: (conditional probability) Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 23

24 9.1 Training of HMMs Then, the expected value of the number of times, state q i was left from, is: And the expected value of the number of transitions from q i to q j is given by: Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 24

25 9.1 Training of HMMs The Baum-Welch algorithm then sets in the r-th iteration (r 0): with defined by,, Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 25

26 9.1 Training of HMMs,, are the initial values The new estimated values are defined by: = (#expected transitions from q i to q j )/ (#expected transitions from q i ) Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 26

9.1 Training of HMMs Where has the value 1 if state o k was observed at time t in the training sequence, and otherwise the value 0 If there are several

27 9.1 Training of HMMs Where has the value 1 if state o k was observed at time t in the training sequence, and otherwise the value 0 If there are several training sequences, indicates the corresponding relative frequencies Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 27

28 9.1 Training of HMMs Now if we build for each parameter re-estimation HMM then we can shown that: Thus, models with newer estimates get better until a (at least local) maximum is reached Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 28

9.1 Applications of HMMs Back to music recognition Feature extraction tries to convert a signal into a string Encoding acoustic events Music signals are sequences of acoustic events Segment the audio

29 9.1 Applications of HMMs Back to music recognition Feature extraction tries to convert a signal into a string Encoding acoustic events Music signals are sequences of acoustic events Segment the audio file, and determine the acoustic events for each segment The implementation of any acoustic event can be described by a state sequence of a HMM Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 29

9.1 Back to Music Recognition If there are several models for acoustic events, we can use a maximum likelihood estimator, to identify the model which

30 9.1 Back to Music Recognition If there are several models for acoustic events, we can use a maximum likelihood estimator, to identify the model which generates the observation sequence with the highest overall probability Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 30

31 9.1 Idea Train H HMMs Training by manual (small H) or automatic (large H) mapping between acoustic events and segments (observation sequences) of a signal Each HMM represents a specific acoustic event Then determine the most probable producer and attribute to each segment, the corresponding event Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 31

32 9.1 Training example Assignment of two segments to events A and B Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 32

33 9.1 Training example Extracted segments are used as training examples for the HMMs of the corresponding events If the following feature sequence belongs to A, after appropriate quantization then this observation sequence will be used: Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 33

34 9.1 Often used HMMs Complete HMM (ergodic model); e.g., with 3 states: Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 34

35 9.1 Often used HMMs Left-right model (Bakis model); e.g., with 3 states Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 35

36 9.1 Feature Extraction with HMMs Given: a sequence of observations from feature sequence of length T: Goal: find sequence so that feature subsequence is associated to HMM Example: given o 1, o 2, o 3, o 4, o 5 and 3 HMM models λ 1, λ 2, λ 3 (2, 1), (1, 3), (3, 4) which associates: λ 2 with (o 1, o 2 ) λ 1 with (o 3 ) and λ 3 with (o 4, o 5 ) Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 36

37 9.1 Realization Combine all H HMM graphs completely (with equal probability) Embedded in a macro-hmm We can start with any model and migrate to any other model Any sequence of acoustic events can be represented in the macro-hmm Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 37

38 9.1 Realization The macro-hmm behaves like a normal HMM All possible events are equally represented in the macro-hmm With the Viterbi algorithm we can establish the most probable state sequence Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 38

39 9.1 Illustration Macro-HMM with as the graph corresponding to HMM H = 4 Example for H = 4 Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 39

40 9.1 Illustration For each single HMM graph we need of course only to completely connect the states in the macro graphs, which may occur as a start and end states Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 40

41 9.1 Problems in Real Applications The data stream can be an infinite sequence feature, and it is not clear exactly when an event begins Build a sequence of sub-sequences and apply the Viterbi algorithm to each subsequence Unfortunately, with complexity of Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 41

42 9.1 Problems in Real Applications How do we choose the best window size w for the sub-sequences? Choose w sufficiently large so that a higher number of HMM graphs can be traversed If while passing through the various paths of the macro-hmm with the Viterbi algorithm, a sufficiently large probability value occurs, then break the computation and return the HMMs we have traversed until then and the corresponding time points Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 42

43 9.1 HMM by example Classify pig coughs into morbid or healthy by using HMM Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 43

44 9.1 HMM by example Extract features e.g., spectrogram of a pig cough Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 44

45 9.1 HMM by example Find a HMM which represents a pig cough as good as possible Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 45

46 9.1 HMM by example Train (Baum-Welch algorithm) the HMM for different cough types (vary diseases), e.g., Pasteurella disease Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 46

47 9.1 HMM by example When a pig coughs use the Viterbi algorithm, to establish if the pig is ill or not! Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 47

) Presentations, lectures Video conferencing.

48 9.2 Video Retrieval Video data Increasingly important for exchanging information Illustrative clips (simulations, animations, etc.) Presentations, lectures Video conferencing... Particularly more frequently on the internet e.g., YouTube, video on demand,... Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 48

9.2 Video Data YouTube Statistics Over 6 billion hours of video are watched each month on YouTube That's almost an hour for every person on Earth, and 50% more than in 2012 100 hours of video

49 9.2 Video Data YouTube Statistics Over 6 billion hours of video are watched each month on YouTube That's almost an hour for every person on Earth, and 50% more than in hours of video are uploaded to YouTube every minute Management of video data is among others a problem of scalability Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 49

50 9.2 Video Data Regarding video data, it is necessary to efficiently: Store it Make it accessible And be able to recover it Today's databases Blobs, smart blobs Retrieval on metadata Splitting into key-frames Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 50

prototype into a commercial database Multimedia Databases

51 9.2 Database support Example: IBM AIV Extenders for IBM DB2 UDB (Development now discontinued) Incorporating the QBIC prototype into a commercial database Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 51

9.2 Database support Description on the IBM Web site: DB2 Video Extender adds the power of video retrieval to SQL queries. You can integrate video data and traditional business data in a single query.

52 9.2 Database support Description on the IBM Web site: DB2 Video Extender adds the power of video retrieval to SQL queries. You can integrate video data and traditional business data in a single query. For example, you can query a news database for video news clips about a specific subject, and list the playing time of each video clip. Then use the Video Extender to play the video clips. IBM ( Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 52

53 9.2 Database support Example usage: For example, an advertising agency stores information about its campaigns in a DB2 database and uses DB2 AIV Extenders to store its print ads, broadcast and video ads. One SQL query can retrieve multimedia data, such as print or broadcast ads for a particular year or client, as well as related business data in the database. IBM ( Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 53

54 9.2 Database support Using DB2 Video Extender, you can define new data types and functions for video data using DB2 Universal Database s built-in support for user-defined types and user-defined functions. Secure and recover video data. Video clips and their attributes that you store in a DB2 database are afforded the same security and recovery protection as traditional business data. IBM ( Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 54

55 9.2 Database support Goal: allows you to store and query video data as easily as you can traditional data Import and export video clips and their attributes into and out of a database. When you import a video clip, the DB2 Video Extender stores and maintains video attributes such as frame rate, compression format, and number of video tracks. IBM ( Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 55

56 9.2 Database support Query video clips based on related business data or by video attributes. You can search for video clips based on data that you maintain, such as a name, number, or description; or by data that the DB2 Video Extender maintains, such as the format of the video or the date and time that it was last updated. IBM ( Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 56

57 9.2 Database support Play video clips. You can use the DB2 Video Extender to retrieve a video clip. You can then use the DB2 Video Extender to invoke your favorite video browser to play the video clip. The DB2 Video Extender supports a variety of video file formats, and can work with different file-based video servers. IBM ( Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 57

9.2 Video Retrieval Topics Main problems: continuous medium Composed of several streams Image stream with visual information (often different views / camera angle of a scene) Audio

58 9.2 Video Retrieval Topics Main problems: continuous medium Composed of several streams Image stream with visual information (often different views / camera angle of a scene) Audio stream (usually more than 1, e.g., synchronous tracks on DVDs) Stream of text (subtitles, news flash...) Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 58

59 9.2 Video Retrieval Topics Main problems: organization Video is a structured medium in time and space Videos can not be seen as a set of individual frames, but as a document Video abstraction decomposes video clips into structured parts (visual table of contents) Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 59

9.2 Video Retrieval Topics How do queries work in video retrieval? Specification of certain features in SQL (as in the IBM extender) Specification of semantic content? E.g.

60 9.2 Video Retrieval Topics How do queries work in video retrieval? Specification of certain features in SQL (as in the IBM extender) Specification of semantic content? E.g., via keywords: news about politics Query by example? One possibility are sample images: Find all the movies with this actor Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 60

61 9.2 Video Retrieval Topics Retrieval technology: comprises all the other problem groups Image Retrieval for the description of independent images, key frames, etc. Audio Retrieval for the evaluation of the sound track, voice recognition, etc. Text retrieval for the search in any subtitles, summaries, transcriptions of the audio track, etc. Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 61

62 9.2 Retrieval Technology However, these techniques can also be combined in the case of video data Person recognition using segmentation and detection of subtitles Assignment based on the actor's voice Classify objects by shape and speech information Detect exciting sports scenes by the audience's applause, etc. Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 62

9.2 Other Features Other features which don t occur in image, audio and text retrieval, affect the detection of the temporal behavior of objects in space Movement of objects (direction, speed),

63 9.2 Other Features Other features which don t occur in image, audio and text retrieval, affect the detection of the temporal behavior of objects in space Movement of objects (direction, speed), instead of simple recognition as in the image retrieval E.g., Car moves slowly from left to right or two people walk together Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 63

64 9.2 Other Features Recognition of movement is normally done through the comparison of shapes in a sequence of images In one frame e.g., by edge detection Transforming shapes in successive images using translation, rotation and scaling If successful, the type of transformation provides information about the parameters of motion Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 64

9.2 Other Features The extraction of moving objects is supported in MPEG-4 encoded streams Separate compression of background and the (moving) foreground objects Since fore-/background

65 9.2 Other Features The extraction of moving objects is supported in MPEG-4 encoded streams Separate compression of background and the (moving) foreground objects Since fore-/background elements change only little, we only need to detect shifts and possibly changes in the camera angle Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 65

66 9.2 Other Features Detection of camera movement Changes in camera angle (zooming, fade in/out,...) Movement of the camera itself (e.g., through background analysis Recognition through various models for the individual effects Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 66

67 9.2 Other Features Time/place relationships Object motion results in trajectories in time and space Intersection of trajectories (e.g., car accidents on the observed crossings) Comparisons between different media E.g., all the videos with a movement from the upper right corner to lower left corner Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 67

of the content, usually with annotated key frames Highlights are scene cuts of certain video

68 9.2 Result Presentation Retrieval of the best videos is very expensive, the quality is difficult to evaluate Retrieval result as a set of video abstractions Summary sequences provide an overview of the content, usually with annotated key frames Highlights are scene cuts of certain video passages (trailer) Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 68

69 9.2 Application of Video Retrieval Personal news (all clips on interesting topics) Entertainment Automatic recognition of film genres (love story, action movie, comedy,...) Detection of advertising in TV Automatic recording of material from the television Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 69

70 This Lecture Hidden Markov Models (continued from last lecture) Introduction into Video Retrieval Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 70

71 Next lecture Video Abstraction Shot Detection Multimedia Databases Wolf-Tilo Balke Institut für Informationssysteme TU Braunschweig 71

Multimedia Databases. 9 Video Retrieval. 9.1 Hidden Markov Model. 9.1 Hidden Markov Model. 9.1 Evaluation. 9.1 HMM Example 12/18/2009

Multimedia Databases. 9 Video Retrieval. 9.1 Hidden Markov Model. 9.1 Hidden Markov Model. 9.1 Evaluation. 9.1 HMM Example 12/18/2009 9 Video Retrieval Multimedia Databases 9 Video Retrieval 9.1 Hidden Markov Models (continued from last lecture) 9.2 Introduction into Video Retrieval Wolf-Tilo Balke Silviu Homoceanu Institut für Informationssysteme