A Framework for Dialogue Detection in Movies

Transcription

1 A Framework for Dialogue Detection in Movies Margarita Kotti, Constantine Kotropoulos, Bartosz Ziólko, Ioannis Pitas, an Vassiliki Moschou Department of Informatics, Aristotle University of Thessaloniki Box 452, Thessaloniki 5424, Greece Abstract. In this paper, we investigate a novel framework for ialogue etection that is base on inicator functions. An inicator function efines that a particular actor is present at each time instant. Two ialogue etection rules are evelope an assesse. The first rule relies on the value of the cross-correlation function at zero time lag that is compare to a threshol. The secon rule is base on the cross-power in a particular frequency ban that is also compare to a threshol. Experiments are carrie out in orer to valiate the feasibility of the aforementione ialogue etection rules by using groun-truth inicator functions etermine by human observers from six ifferent movies. A total of 25 ialogue scenes an another 8 non-ialogue scenes are employe. The probabilities of false alarm an etection are estimate by cross-valiation, where 7% of the available scenes are use to learn the threshols employe in the ialogue etection rules an the remaining 3% of the scenes are use for testing. An almost perfect ialogue etection is reporte for every istinct threshol. Introuction Digital movie archives have become a commonplace nowaays. Research on movie content analysis has been very active. A ialogue scene can be efine as a set of consecutive shots which contain conversations of people []. However, there is a possibility of having shots in a ialogue scene that o not contain any conversation or even any person. The elements of a ialogue scene are: the people, the conversation an the location is taking place in [2]. The basic shots in a ialogue scene are: (i) Type A shot: Shot of actor A s face; (ii) Type B shot: Shot of actor B s face; (iii) Type C shot: Shot with both faces visible. A set of recognizable ialogue acts accoring to semantic content is propose in [3]: (i) Statements; (ii) Questions; (iii) Backchannels; (iv) Incomplete utterance; (v) Agreements; Appreciations. Dialogue etection in movies follows specific rules since movie making is a kin of art [5]. Lehane states that in a 2-person ialogue there is usually a A-B-A-B structure of camera angles, thus making ialogue etection feasible [4]. However, the person who speaks at any given time is not always the one isplaye. Shots of other participants reactions are frequently inserte. In aition, the shot of the speaker may not inclue his face, i.e. the rear view of his hea might be This work has been supporte by the FP6 European Union Network of Excellence MUSCLE Multimeia Unerstaning through Semantics, Computation an LEarning (FP ).

2 epicte. Furthermore, shots of other persons or objects might be inserte in the ialogue scene. Eviently, these shots a to the complexity of the ialogue etection problem, ue to their noneterministic nature. Numerous methos for ialogue etection have been propose, because such a preprocessing step is useful for vieo analysis, inexing, browsing, searching, an summarization. Both vieo an auio information channels coul be exploite for efficient ialogue etection. For example, automatically extracte low-level an mi-level visual features are use to etect ifferent types of scenes, focusing on ialogue sequences [4]. Emotional stages as a means for segmenting vieo are propose in [6]. The etection of monologues base on auio-visual information is iscusse in [7] where a noticeably high average ecision performance is reporte. Relate topics to ialogue etection are face etection an tracking [8], speaker turn etection [9], an speaker tracking []. The aforementione research is compliant with the MPEG-7 stanar. In this paper, we propose a novel framework for ialogue etection that is base on inicator functions. In practice, inicator functions can be obtaine by speaker turn etection followe by speaker clustering or by face etection followe by a similar clustering proceure. However, in this paper we are intereste in setting up the etection framework in the ieal situation where the inicator functions are error free. Towars this goal groun truth inicator functions are employe. Two ialogue etection rules are evelope. The first rule employs the value of the cross-correlation function at zero time-lag an the secon one is base on the cross-power in specific frequency ban. Both quantities are compare to corresponing threshols. Experiments are carrie out using the auio streams extracte from six ifferent movies while the groun-truth inicator functions are efine by human observers. To valiate the feasibility of the ialogue etection rules, the cross-valiation approach is utilize, where 7% of the auio streams is use to efine the two threshols, an the remaining 3% is use for testing. Experimental results inicate that an almost perfect ialogue etection is achievable. The outline of the paper is as follows. The propose ialogue etection rules are iscusse in Section 2. In Section 3, the ialogue scenes use for the experimental evaluation of the propose metho an the training proceure are escribe. In Section 4, performance evaluation is presente an finally conclusions are rawn in Section 5. 2 Dialogue etection 2. Inicator Functions Inicator functions are frequently use in statistical signal processing. They are closely relate to zero-one ranom variables use in the erivation of the probabilities of events through expecte values []. In maximum entropy probability estimation, inicator functions are use to insert constrains quantifying facts stemming from the training ata that constitute our knowlege about the ranom experiment. An example is language moeling [2]. Inicator functions have also been use in the analysis of the DNA sequences [3]. Let us suppose that we have an auio recoring of N samples, where N is the prouct of uration of the auio recoring multiplie by the sampling frequency an

3 we know exactly when a particular actor (i.e. speaker) appears. Such information can be quantifie by the inicator function of say actor A, I A (n), efine as: { when actor A is present at sample n I A (n) = () otherwise. For a ialogue, at least two actors shoul be present. Let us call them A an B with corresponing inicator functions I A (n) an I B (n), respectively. Besies their presence, the actors shoul be active, that is their inicator functions shoul not be zero uring the entire scene uration. To avoi such irregularities, we can measure a proper norm of the inicator function, e.g. the L norm or the L 2 norm, etc. Since the inicator functions amit non-negative values, their L norm is simply the sum of the inicator function values: N S A = I A (n). (2) n= Two characteristic inicator functions for a ialogue scene are plotte in Figure (a). There are several possibilities for a ialogue scene. For example, there might be auio.8.8 IA(n).6.4 IA(n) n, time (msec) n, time (msec).8.8 IB(n).6.4 IB(n) n, time (msec) (a) n, time (msec) (b) Fig.. (a) Inicator functions of two actors in a ialogue scene. (b) Inicator functions of two actors in a non-ialogue scene (i.e. monologue). frames where both actors speak. Auio frames corresponing to short silences shoul be tolerate. In aition, the auio backgroun in ialogue scenes might contain music or environment noise that shoul not prevent ialogue etection. For the time-being, since optimal (i.e. groun-truth) inicator functions are employe, such cases are not ealt with explicitly. An example of a scene where there is no ialogue is shown in Figure (b). It is seen that I B (n) is zero for all n. This is the case of an inactive actor for whom S B =. 2.2 Cross-Correlation The cross-correlation is a measure of similarity between two signals. It is efine as: N c AB () ={ N n= I A(n + )I B (n) if N c BA ( ) if (N ) (3)

4 where N is the total number of samples in the auio stream an is the time-lag. For =, the cross-correlation is equal to the prouct of the two inicator functions I A (n) an I B (n). Practically, this means that the greater the value of c AB () is, the longer time the two actors speak simultaneously. The cross-correlation for the ialogue shown in Figure (a) is epicte in Figure 2(a). It can be seen that c AB () > cab().5. φab(f) ,lag 4 (a) f (Hz) (b) Fig. 2. (a) Cross-correlation of the inicator functions for two actors participating in a ialogue. (b) Cross-power spectrum for two actors participating in a ialogue. For the scene corresponing to the two inicator functions plotte in Figure (b), the cross-correlation is zero throughout its omain. From the aforementione observations, a plausible ialogue etection rule is: where ϑ is an appropriately chosen threshol. c AB () ϑ (4) 2.3 Cross-Power Spectrum Another useful notion to be exploite for ialogue etection is the cross-power spectrum, i.e., the iscrete-time Fourier transform of the cross-correlation: φ AB (f) = N = (N ) c AB () exp( j2π f) (5) where f [.5,.5] is the frequency in cycles per sampling interval. In orer to robustify the ialogue etection, we propose to examine the cross-power p in the frequency ban [.65, 5] that has been etermine by analyzing the measure cross-power spectra p = 5.65 φ AB (f) 2 f. (6) When there is a ialogue, p amits a value that epens on the area uner the crosspower spectrum φ AB (f). Figure 2(b) shows the cross-power spectrum ensity over

5 the frequencies [,.5]. For negative frequencies, φ AB ( f) =φ AB (f). On the other han, in the non-ialogue scene corresponing to the two inicator functions plotte in Figure (b), the cross-power spectrum is ientically zero. Accoringly, the secon ialogue etection rule propose is: where ϑ 2 is clearly an appropriately chosen threshol. p ϑ 2 (7) 3 Data Set an Training Proceure In total, 33 recorings were extracte from the following six movies: Analyze That, Col Mountain, Jackie Brown, Lor of the Rings I, Platoon, an Secret Winow. The total uration of the 33 recorings is 3 min an 7 sec. The auio track was igitize in PCM at a sampling rate of 48 khz an the quantize sample length was 6 bit two-channel. 25 out of the 33 recorings correspon to ialogue scenes, while the remaining 8 o not contain any ialogue. For each recoring, the groun-truth inicator function of the actors appearing in the scene is etermine an for each pair of inicator functions their cross-correlation sequence is calculate. In orer to check the efficiency of the propose etection rules (4) an (7), we nee to estimate for each rule the probability of etection an the probability of false alarm. The aforementione probabilities stem from the binary hypothesis etection problem where the null hypothesis is H : the scene is not a ialogue an the alternative hypothesis H : the scene is a ialogue. Then, the probability of etection is for rule (4): P () = Prob(rule (4) ecies the scene is ialogue H ) (8) an the probability of false alarm is given by: P () f = Prob(rule (4) ecies the scene is ialogue H ). (9) P (2) an P (2) f are efine similarly for rule (7). To estimate P (i) an P (i) f, i =, 2, cross-valiation is employe. The available cross-correlation sequences an their cross-power spectrum ensities are ivie into two isjoint subsets. The first subset is use for training an the secon subset is use for testing. 7% of the available ata are use for training an the remaining 3% for testing. This means that the 23 ranomly selecte cross-correlation sequences an their corresponing cross-power spectrum ensities are use for training an the remaining 9 are use for testing. When selecting the 23 training sequences we simultaneously preserve the ratio between ialogue an non ialogue scenes, i.e. 8 cross-correlation sequences corresponing to ialogue scenes an another 6 corresponing to non ialogue scenes. Similarly, the testing cross-correlation sequences were forme by 7 auio streams corresponing to ialogue scenes an another 2 corresponing to non ialogue scenes. Because of the relatively small amount of the training sequences we applie the leave-one-out metho to estimate the probability of etection. That is 22 out of the 23

6 training sequences are use to estimate the probability of etection an the estimation is repeate by leaving a ifferent training sequence out for all training sequences (i.e. 23 times). Let P (i;r) (ϑ r i ) be the probability of etection for the ith rule that employs the threshol ϑ r i when the rth training sequence is left out. Figure 3(a) shows the average P () (ϑ ) versus ϑ. The curve is estimate by averaging the probabilities measure in the 23 repetitions. The corresponing plot of the average P (2) (ϑ 2) versus ϑ 2 is epicte in Figure 3(b) P () (ϑ).5 P (2) (ϑ2) ϑ ϑ2 (a) (b) Fig. 3. (a) The average P () (2) (ϑ) versus ϑ for the first rule. (b) The average P (ϑ2) versus ϑ2 for the secon rule. Let ϑ i be chosen as the minimum threshol value such that P (i;r) (ϑ r i )=. Table a summarizes the threshols etermine for each training sequence left out. By applying the minimum threshol value an using the entries of Table a, we fin that ϑ = an ϑ 2 =.4, respectively. 4 Performance Evaluation During Testing For the 9 auio streams left out for testing, their corresponing cross-correlations an cross-power spectrum ensities are compute an the values of c AB () an p are collecte in Table b. The first seven rows in Table b correspon to ialogue scenes an the last two correspon to non-ialogues. From the inspection of Table b, it is seen that only the 6th cross-correlation sequence is not etecte as corresponing to a ialogue scene by applying the etection rule (4), although it is. It is also seen that there are no false alarms. The secon etection rule (7) can rectify the just escribe miss-etection. A simple OR rule, i.e. c AB () ϑ OR p ϑ 2. () can yiel a perfect ialogue etection. To compensate the lack of real inicator functions, a number of synthetic inicator functions amitting real values within [, ] have been create an incluing in the test phase. The nature of syntectic inicator functions create an the performance of rule () is summarize in Table 2.

7 Table. (a) The 23 pairs of ϑ an ϑ 2 uring the training proceure. (b) The 9 pairs of crosscorrelation value at zero lag an cross-power in the frequency ban f [.65, 5] for the test recorings. sequence left out, r ϑ r ϑ r test auio cab() p stream inex Conclusions In this paper, we have propose a novel framework for ialogue etection in movies base on inicator functions. Experiments are carrie out using inicator function groun truth extracte from real movies. Cross-valiation was use to estimate the probabilities of etection an false alarm. The experimental results emonstrate the feasibility of the propose etection rules in 33 movie segments. In the future, we plan to exten our movie atabase. Moreover, the groun truth inicator functions will be replace by actual ones erive by either speaker turn etection followe by speaker tracking or face etection followe by face tracking by their combination. References. A. A. Alatan an A. N. Akansu, Multi-moal ialog scene etection using hien-markov moels for content-base multimeia inexing, J. Multimeia Tools an Applications, vol. 4, pp. 37-5, L. Chen an M. T. Özsu, Rule-base extraction from vieo, in Proc. 22 IEEE Int. Conf. Image Processing, vol. II, pp , P. Král, C. Cerisara, an J. Kleckova, Combination of classifiers for automatic recognition of ialogue acts, in Proc. 9th European Conf. Speech Communication an Technology, pp , B. Lehane, N. O Connor, an N. Murphy, Dialogue scene etection in movies using low an mi-level visual features, in Proc. Int. Conf. Image an Vieo Retrieval, pp , 25.

8 Table 2. Synthetic inicator functions, their corresponing c AB() an p values, an final ecision. Nature of the inicator function c AB() p Dialogue etection Aing Gaussian noise N(,.5) inepenently to both inicator functions an har limiting to [, ] correct Aing a consierable amount of silence between correct speaker turn points (here 33.3% of the average ialogue uration is silence). Aing a consierable amount of overlap between correct speaker activities (the overlap amounts to 33.3% of the average ialogue uration). Moeling between-speaker silence as a Gaussian ranom correct variable N(.5,.5) Moeling between-speaker silence as a uniform ranom variable correct Moeling between-speaker silence/music/noise as constant correct value of. 5. D. Arijon, Grammar of the Film Language. Silman-James Press, A. Vassiliou, A. Salway, an D.Pitt, Formalising stories: sequences of events an state changes, in Proc. 24 IEEE Int. Conf. Multimeia an Expo, vol. I, pp , Hong- Kong, Taiwan G. Iyengal, H. J. Nock, an C. Neti, Auio-visual synchrony for etection of monologues in vieo archives, in Proc. 23 IEEE lnt. Conf. Acoustics, Speech, an Signal Processing, vol. I, pp , April 23, Hong Kong. 8. K. Sobottka an I.Pitas, A novel metho for automatic face segmentation, facial feature extraction an tracking, Image Communication an Signal Processing, vol. 2, no. 3, pp , June M. Kotti, E. Benetos, an C. Kotropoulos, Automatic speaker change etection with the bayesian information criterion using MPEG-7 features an a fusion scheme, in Proc. 26 IEEE Int. Symp. Circuits an Systems, May 26, Kos, Greece.. L. Lu an H. Zhang, Speaker change etection an tracking in real-time news broacast analysis, in Proc. 24 IEEE Int. Conf. Acoustics, Speech, an Signal Processing, vol. I, pp , June 24.. A. Papoulis an S. V. Pillai, Probabilities, Ranom Variables, an Stochastic Processes, 4/e. N.Y.: McGraw-Hill, F. Jelinek, Statistical Methos for Speech Recognition. Cambrige, Massachusetts: The MIT Press, R. J. Boys an D. A. Henerson, A Bayesian approach to DNA sequence segmetation, in Proc. 24 Biometrics, vol. 6, no 3, pp. 573, September 24.