Novel TV Commercial Detection in Cookery Program Videos

Transcription

1 roceedings of the World Congress on Engineering and Computer Science 009 Vol Novel TV Commercial Detection in Cooker rogram Videos Venkatesh.N, Rajeev.B, and. Girish Chandra Abstract n this paper, we present a simple but ver effective method for detecting commercial boundaries in a cooker program. Our approach to this problem is to first identif the best audio feature to detect commercial boundar for cooker s with and without background music. Further, we boost the confidence level of detecting boundar using novel method of logo (program name) frames detection indicating the start and end of the commercial at the boundaries b audio. Eperiments show that combining audio features and clues ields good recall and precision on a test set s from different ndian television broadcast data. nde Terms Commercial Detection, combining Audio and visual features, cooker show.. NTRODUCTON The recent technologies such as set-top boes coupled with the reduction of costs for digital media storage have led to the introduction of recording option for all kinds of television (TV) programs. n this contet, audiovisual analsis tools that help the user to manage the huge amount of data are ver important in a ver competitive market to introduce the novel recording devices. Among other analsis tools, detection of TV advertisements is a topic with man practical applications. For instance, from the point of view of a TV end-user, it could be useful to avoid commercials in personal recordings. Eisting commercial detection approaches can be generall divided into two categories: feature-based and recognition-based approaches. While the feature-based approaches use some inherent characteristics of TV commercials to distinguish commercials and other tpes of s, the recognition-based methods attempt to identif commercials b searching a database that contains known commercials. The challenges faced b both approaches are the same: how to accuratel detect commercial breaks, each of which consists of a series of commercials; and how to automaticall perform fast commercial recognition in real time. anuscript received Jul 6, 009. Venkatesh. N is with Tata Consultanc Services nnovation Lab, lot #96, E ndustrial area, White field main road, Bangalore , ndia (phone: ; venk.n@tcs.com). Rajeev.B is with Tata Consultanc nnovation Services Lab, lot #96, E ndustrial area, White field main road, Bangalore , ndia ( rajeev.b@tcs.com).. Girish Chandra is with Tata Consultanc Services nnovation Lab, lot #96, E ndustrial area, White field main road, Bangalore , ndia ( m.gchandra@tcs.com). Commercial break detection [1] [] have based their strategies in studing the relation between audio silences and black frames as an indicator of commercials boundaries. The analsis is performed in either compressed [1] or uncompressed [] domains. n [3] also specific countr regulations about commercials broadcast is used as a further clue. Another interesting approach is presented in [4], where the overlaid tet trajectories are used to detect commercial breaks. The idea here is that overlaid tet (if an usuall remains more stable during the program time than in the case of commercials. n [5] Hidden arkov odel (H) based commercial detection is presented with two different observations taken for each shot: logo presence and shot duration. Generall black frame [6] is used as the crucial information for detecting the commercial in processing. But, in ndian TV stations black frames are absent and our methodolog works without using black frames as clues. Our simple and accurate approach for commercial detection uses audiovisual features. nitiall, we find the commercial boundaries using a combination of different audio features and the confidence level of the boundar will be further enhanced or validated using logo (program name) frames detection onl at commercial boundaries b audio based feature and later we use features combination of rewitt edges[7] and Harris corner points[8] for image matching of logo frame appearing at the start of the commercial and end of the commercial logo frame for measuring the similarit in terms of simple piel intensit based matching. The Block diagram of the sstem proposed in this paper is shown in figure 1. Cooker Video clue based detection Event Audio data Fig.1 Block diagram of commercial detection re-processing and Feature Etraction Audio based detection

2 roceedings of the World Congress on Engineering and Computer Science 009 Vol The rest of this paper is organized as follows. n Section reprocessing and Feature etraction is presented. Section 3 describes the proposed scheme used to detect commercial boundaries. Results and discussion are presented in Section 4. Finall conclusion and future work are presented in Section 6.. RE-ROCESSNG AND FEATURE EXTRACTON A. re-processing The audio signal of cooker is etracted and is segmented into audio frames with 50% overlap and 3 msec duration. B. Audio Feature Etraction 1) Spectral Rolloff (R): Spectrum rolloff [1] point is the boundar frequenc f r below which a 85% of the spectral energ for a given audio frame is concentrated. This feature is used to differentiate voiced from unvoiced speech. t is defied as the frequenc below which an eperimentall chosen percentage of the accumulated magnitudes of the spectrum is concentrated. Unvoiced speech has a high proportion of energ contained in the high-frequenc range of the spectrum. f r k = 0 R( k) = 0.85 K 1 k = 0 R( k)... (1) f f r is the largest value of k for which equation 1 is satisfied then this frequenc f r is the rolloff. ) Zero-Crossing Rate (ZCR): t is defined as the number of times the audio waveform changes its sign in the duration of the frame or in other words it is the number of time-domain zero crossings within a speech frame. t is called a measure of the dominant frequenc in a signal. The zero crossing rate of the frame ending at time instant m is defined b, m 1 ZCR = sgn[ ( n)] sgn[ ( n 1)]... () n= m N + 1 1, ( n) 0 sgn[ ( n)] 1, ( n) < 0 where 3) Short-Time Energ (E): The short-time energ of a frame is defined as the sum of squares of the signal samples normalized b the frame length. N 1 = E N n 1 0 C. Video Feature Etraction ( n)... (3) 1) rewitt Edge detection: This process detects outlines of an object and boundaries between objects and the background in the image [7]. The basic edge-detection operator is a matri area gradient operation that determines the level of variance between different piels. The edge-detection operator is calculated b forming a matri centered on a piel chosen as the center of the matri area. f the value of this matri area is above a given threshold, then the middle piel is classified as an edge. The rewitt operator measures two components. The vertical edge component is calculated with kernel K X and the horizontal edge component is calculated with kernel K Y. The intensit of the gradient in the current piel is given b K X + K Y K X = K Y = Fig. rewitt vertical and horizontal operators ) Harris corner points: The Harris corner detector is a popular interest point detector due to its strong invariance to [8] rotation, scale, illumination variation and image noise. The Harris corner detector is based on the local autocorrelation function of a signal where the local auto-correlation function measures the local changes of the signal with patches shifted b a small amount in different directions. Basic idea here is to find the points where edges meet that in other words strong brightness changes in orthogonal directions. E( u, v) = w( [ ( + u, + v) ( ]... (4) where w( is a window function at point (, (+u,+v) is shifted intensit and ( is intensit at point (. For small shifts [u,v] we have a bilinear approimation: u E ( u, v) [ u, v] (5) v where is a matri computed from image derivatives: = w( (6) easure of corner response is given b: det = λ1λ..... (7) trace = λ1 + λ R = det k ( trace )..... (8) where λ1, λ are eigenvalues of. Choosing the points with large corner response function R (R > threshold) and considering the points of local maima of R gives us the Harris corner points.. COERCAL DETECTON SCHEE A. Audio Feature Based Detection Having etracted audio features, we use heuristic rules found b eperimentation for commercial detection. 1) Zero Crossing Rate & Spectral Roll off: With Zero crossing rates (Z) we set a threshold of 75% of maimum

3 roceedings of the World Congress on Engineering and Computer Science 009 Vol number of zero crossings value across all the frames denoted b T Z. We consider the frames which has the zero crossing rates value greater than eperimentall found threshold T Z as boundaries in commercial which is given in equation 9. Let N be the number of frames. if Z(n) > TZ Z (n)... (9) Similarl we identif the frames corresponding to boundar detection of commercials using Spectral roll off (R) b having an eperimentall found threshold denoted b T SF and it given in equation 10. if R(n) > TSF R (n)... (10) Having found frames corresponding to boundar b ZCR (Z') and SR(R') we perform a logical AND operation between Z' and R' given b equation 11 which results in reducing the false detection. ZR(n) = Z ( n) & R ( n) where, n = 1KN... (11) The above mentioned feature combination technique works onl for cooker containing the background music or it fails miserabl and hence we initiall check for number of zero crossings frames N Z and if it eceeds the threshold T F, then we go for the other set of features as eplained in section.a.. n this case threshold T F is given b equation 1 where, N is the total number of frames. N T F =.... (1) ) Spectral Energ We have observed in man cooker s without background noise that during commercial interval, the spectral energ (E) is high we make use of this information for detecting commercial boundaries. Frames eceeding the eperimentall found threshold T E are considered as the ke frames for commercial boundar detection as given in equation 13. if E(n) > TE E (n)... (13) B. Video Feature Based Detection Once we have the commercial boundaries using audio features we validate those boundaries using logo (program name) frames matching at start and end of commercials. We identif frame with logo at the boundaries b audio based features which reduces computational time for processing. mage matching algorithm using combination of edge and corner points as features are as given below: nitiall we convert colour image to gra scale image and etract edges and corner points of an image as a feature for image matching. We match the combined features which are in binar forms b simple piel intensit matching as eplained below: Step 1: =0 where is the number of piels matched. Step : if ( R( = 1 & T ( = 1) then = +1 where R is the reference template image at point ( and T( is test image at point (. Step 3: = 100 WR where W R is the number of white piels in reference template image R. Step 4: f is greater than 80% then we can sa that two images are almost similar.. RESULTS AND DSCUSSON n order to eamine the proposed methodolog and to carr out the relevant eperimentation we have recorded four s from three different ndian TV stations. Videos are captured at 5 frames per second with resolution and the audio was etracted from these s and stored in uncompressed wav format at khz with 16 bits per sample for further processing. Commercial details for the recorded data are provided in table 1. We have performed our eperiments using ATLAB programming in windows operating sstem. Background usic Video Length Commercial Length in seconds TABLE 1 DETALS OF DATA Video 1 Video Video3 Video4 resent resent Absent resent 3min 49sec (i)197-en d of the 1 min 58 sec (ii) min 49 sec (i) min 37 sec (i) (ii) (iii) 1103 till end of the The first two s ( 1 and ) were found to have background music and hence the are processed b non-energ features like ZCR and Spectral Roll off.

4 roceedings of the World Congress on Engineering and Computer Science 009 Vol Non-Energ based features fail miserabl for without background music and hence we need to involve energ features. n the present case we have used short time energ feature. The audio based commercial boundar detection are shown in figure (a), 4(a) and further processing b using features are as shown in fig (b), 4(b) for 1 and respectivel. Logo (program name) at the commercial boundaries is shown in figure 3(a). Correspondingl rewitt edge detection and Harris point detection for logo frame are shown in figure 3(b) and 3(c) respectivel. 3(c) (a) (b) Fig. a) Audio feature based boundar for 1, b) Video feature based boundar for Fig.3 a) Logo (program name) frame. b) rewitt edge detection for logo frame. c) Harris corner points for logo frame. 4(a) 4(b) Fig.4 a) Audio feature based boundar for, b) Video feature based boundar for. Table gives a detailed split up of all the commercials b audio. TABLE RESULTS USNG ONLY AUDO FEATURES 3(a) Total No of false missed event detecte d true Video Video 0 0 Video Video We can observe from figure 1(a) and figure 3(a) that using audio features lot of frames are highlighted during the commercial. From observation and eperiment we consider onl the first and last frame as a boundar for the commercial if and onl if the time interval between first highlighted frame and last highlighted frame eceeds at least 1 minute because commercials usuall lasts for more than one minute. 3(b) We measure the sstem performance using Recall and recision given b equation 14 and 15 and tabulated the results accordingl in table 3. Recall = The number of true The actual number of... (14)

5 roceedings of the World Congress on Engineering and Computer Science 009 Vol recision= The number of true The total number of... (15) From table we can see that using onl audio based features there are some false in 3 and 4 because it corresponds to prize distribution for the quiz winner of last episode. Here both audio and visual changes are ver similar to commercial. To overcome these problems we have made use of clues. TABLE 3 RESULTS USNG AUDOVSUAL FEATURES Game Recall recision Video Video 1 1 Video Video TABLE 4 RESULTS USNG BOTH AUDO AND VDEO FEATURES Total No of false missed event detecte d Video Video 0 0 Video Video true From table 4 we observe that using our simple clues of matching logo frames at commercial boundaries for refining the boundaries b audio we are able to get rid of the false b audio. One more observation is that the number of missed remain same in 4 because we tr to locate the logo frames onl at the boundar instances b audio and if the are missed b the audio then it cannot be recovered b processing but instead it is used onl to validate and increase the confidence level of commercial boundaries b audio. speech and non voiced speech segments for s with background music are considered, hence covering all kind of possible variations in data. Confidence level of commercial boundar has been improved and we were able to get rid of the false detections b using logo (program name) frame matching at commercial boundaries. The above presented sstem can be ver much used for real-time embedded sstem as it is time efficient, simple and amenable for eas implementation. Our future work will be focused on using more robust audio features for detecting commercial boundaries without missing an and further we are tring to eperiment on some more cooker s and also on different TV shows like news, serials and realit shows. REFERENCES [1] D. A. Sadlier et al., Automatic TV advertisement detection from mpeg bitstream, Journal of the att. Rec. Societ, vol. 35, no. 1, pp. 15, Dec. 00. [] Y. Li and C.-C. Ja Kuo, Detecting commercial breaks in real TV program based on audiovisual information, in SE roc. on S, vol 41, Nov [3] R. Lienhart, C. Kuhmnch, and W. Effelsberg, On the detection and recognition of television commercials, in roc. EEE Conf. on CS, Otawa, Canada, [4] N. Dimitrova, ultimedia content analsis: the net wave, in roc. of the nd CVR, llinois, USA, Aug.003. [5] Alberto Albiol, ar ıa Jos e Ch. Full`a, Antonio Albiol, Luis Torres, Commercials Detection using Hs,004. [6] Alberto Albiol ar, aría José, Ch. Fullà, Antonio Albiol, Detection of TV Commercials n roceedings of the nternational Conference on Acoustics, Speech and Signal rocessing,004. C. J. Kaufman, Rock ountain Research Lab., Boulder, CO, private communication, a [7] Hong Shan Neoh, Asher Hazanchuk, Adaptive Edge Detection for Real-Time Video rocessing using FGAs, nternational Signal rocessing Conference (SC), Santa Clara, California, September 7-30, 004. [8] C. Harris,. Stephens. "A combined corner and edge detector" roceedings of the 4th Alve Vision Conference: pp ,1988. TABLE 5 RESULTS USNG BOTH AUDO AND VSUAL FEATURES Game Recall recision Video Video 1 1 Video Video V. CONCLUSON AND FUTURE WORK n this paper we have presented a commercial detection scheme based on both audio and visual features. n the case of audio detection we have two sets of features based on the tpe of data. Energ based features (spectral energ entrop for s without background music and features like (ZCR and Spectral Roll Off) for distinguishing between