Video summarization based on shot boundary detection with penalized contrasts

Transcription

1 Video suarization based on shot boundary detection with penalized contrasts Paschalina Medentzidou and Constantine Kotropoulos Departent of Inforatics Aristotle University of Thessaloniki Thessaloniki 54124, GREECE Eail: Abstract In this paper, we propose a novel technique for shot boundary detection and video suarization that is based on change point detection with penalized contrasts. The proposed ethod extracts a proper tie series fro the video, calculating the ean level of the hue coponent of consecutive video fraes in the Hue Saturation Value color space. Change point detection is applied to the tie series, estiating the nuber of shot transitions and their location. The posterior distribution of the change point sequence is defined, that splits the video into hoogenous teporal segents. A representative frae is selected in each segent and siilar or eaningless keyfraes are deleted fro the suary. The resulting suary is of coparable quality to that of the state of the art techniques. Keywords Change Point Detection; Shot Boundary Detection; Bayesian Inforation Criterion; Keyfrae Extraction. I. INTRODUCTION The aount of video data being recorded and distributed every day has been increased due to the low storage cost, the high speed of internet connections, and the ability of individuals to capture videos using sart devices and caeras. As a consequence, the need for a short representation of video data has eerged. Video suarization ais to tackle this need. It is defined as the process of finding a sequence of still or oving pictures (with or without audio), representing the content of a video in a concise anner. This way the essential essage of the original video is preserved [1]. As set in [2], the video suary ust contain high priority entities and events fro the video, exhibit reasonable degrees of continuity, and be free of repetition. There are two types of video suarization, naely static keyfrae extraction and dynaic video skiing. A keyfrae is a frae that represents the content of a logical unit, like a shot or scene, for exaple. This content ust be as representative as possible [3]. A dynaic video skiing consists of a collection of associated audio-video sub-clips selected fro the original sequence, but with a uch shorter length [4]. One advantage of a video ski over a keyfrae set is its ability to include audio and otion eleents that enhance both the expressiveness and inforation of the suary. On the other hand, keyfrae sets are not restricted by any tiing or synchronization issues [5]. This paper addresses the video suarization proble as a static keyfrae extraction challenge. A novel ethod for change point detection in the ean is applied to a video tie series extracted by calculating the ean of the hue coponent in Hue Saturation Value (HSV) color space in consecutive video fraes. Change point detection in the ean of a signal is the process of finding changes in the average level of the signal paraeters. Soe of its applications include fault detection in cheical processes, detection and identification of seisic waves, bioedical signal processing, usic restoration, changes in DNA sequences, speech and speaker segentation [6]. The novelty of the proposed approach for shot boundary detection and video suarization, is in extracting a proper video tie series capturing the video structure and its odeling as a linear odel with additive noise. Noise accounts for sall changes in the ean hue of consecutive fraes that are attributed to caera otion, entering of objects, or exiting a shot, etc. Akaike (AIC) and Bayesian (BIC) inforation criteria are frequently eployed in order to find a parsionious solution to this odel. In a Bayesian fraework, the conditional distribution of the change point sequence is constructed. A Markov Chain Monte Carlo (MCMC) is eployed to saple the posterior distribution. A Stochastic Approxiation of Expectation Maxiization (SAEM) estiates the distribution paraeters, as detailed in Section III-A-III-C. The detected change points correspond to the candidate shot cuts that split the video into hoogenous teporal segents. Representative keyfraes are chosen fro each segent, while eaningless keyfraes and keyfraes that are siilar to others are rejected, as being redundant. The keyfraes that are derived, create the video suary describing the ain video content. The reainder of the paper is organized as follows. Section II reviews existing video suarization ethods. Section III describes the proposed technique for shot cut detection and video suarization. Section IV reports and discusses experiental results on a benchark dataset. Finally, soe concluding rearks are given and future research objectives are identified in Section V. II. RELATED WORK Many static suarization ethods were proposed. Mundur et al. [7] developed a ethod based on Delaunay Triangulation, which was applied to cluster video fraes. The ethod saples the fraes of the original video and represented the by the color histogra in the HSV color space. The Delaunay diagra was built and clusters were

2 obtained by separating the diagra edges. For each cluster, the frae nearest to its center was selected as keyfrae. Furini et al. introduced the STIll and MOving Video Storyboard, (STIMO) [8]. The STIMO was designed to produce video storyboards on the fly. It was coposed of three ain phases. First, the video was analyzed and the color description was extracted in the HSV color space. Second, a clustering algorith was applied to color description. Third, eaningless video fraes were reoved fro the produced suary. de Avila et al. proposed a video suarization technique, called VSUMM based on color feature extraction fro the color histogras in the HSV color space [9]. The video fraes were sapled at 1 frae per second (fps) and then clustered by k-eans algorith, while redundant fraes were reoved. Ngo et al. proposed a unified approach for video suarization based on the analysis of video structures and video highlights. The ain coponents of this approach were scene odeling and highlight detection. A video was represented as an undirected graph and the noralized cut algorith partitioned the graph into video clusters. Video suaries were generated fro a teporal graph [10] A teporal segentation into seantically consistent segents, deliited by shot boundaries and general change points was perfored in [11]. Iportance scores were assigned to each segent, using a Support Vector Machine classifier. The resulting video suary assebles the sequence of segents with the highest scores. Detection of attention-invoking audiovisual segents in ovies was proposed on the basis of saliency odels for the audio, visual, and textual inforation conveyed in a video strea [12]. Video suarization using shot boundary detection based on the utual inforation and the joint entropy between fraes was developed in [13]. III. PROPOSED VIDEO SUMMARIZATION METHOD The proposed video suarization ethod executes the following steps: (a) A tie series is extracted, which describes the video structure by calculating the average hue per frae in consecutive video fraes. (b) Change point detection is applied to the aforeentioned tie series in non overlapping sliding windows of 500 fraes. When the nuber of fraes is not integer ultiple of the window size then an overlap is allowed between the last two window instances. That is, if a video consists of 1350 fraes, then the ethod is applied to fraes 1-500, , and 850 to Duplicate detected changes are deleted. Change points are siultaneously detected by iniizing a penalized contrast defined in Section III-B and III-C. (c) Each video segent defined by pairs of consecutive change points is represented by the teporal iddle frae. For exaple, the iddle frae for the segent defined by the 10th and 81th frae is the 46th frae (d) Meaningless fraes (i.e., onochroatic ones) characterized by a low ratio of edge pixels are deleted. (e) Highly correlated fraes, fraes with siilar RGB color histogra as well as siilar texture are rejected. Fig. 1. cuts. Mean of the hue channel in video v57.pg and detected shot A. Tie series extraction A tie series fro each video is extracted that odels the video structure. It is defined as the ean of the hue coponent of consecutive video fraes. The HSV color space is used, because it provides an intuitive representation of color that atches the color perception by huans. Frae averages are eployed instead of frae differences that are coonly used in the literature, because the proposed ethod for change point detection identifies changes in the ean level of a signal. The tie series in Fig. 1 depicts the average hue value per frae for the video v57.pg in the dataset eployed in [9]. It exhibits a change every tie a shot cut occurs. Let the tie series be odeled as d i = µ i + σ e i, i = 1, 2,...N where {e i } is a sequence of zero-ean rando variables with variance equal to one and N is the nuber of video fraes. The ethod proposed by M. Lavielle [14] 1 is applied for change point detection. This ethod is proved efficient for both abrupt and gradual transition detection. The ethod estiates the nuber of change points (i.e., shot cuts) and their location using penalized contrasts. Forally, the posterior distribution of the change point sequence is defined as a function of the penalized contrast in a Bayesian fraework, as is detailed next. B. Shot cut detection In ost cases, the abrupt changes in a signal {d i } are attributed to a paraeter θ = {µ}, where µ is the ean level of the tie series. This assuption is used to define the contrast function J(, d). Let K be an integer indicating the nuber of video segents (shots) and = ( 1, 2,..., K 1 ) be a sequence of integers defining the shot boundaries. The sequence is satisfying the following ordering 0 < 1 < 2 <... < K 1 < N. For any k, such that 1 k K, let U(d k 1 +1,..., d k µ) be a contrast function useful for the estiation of the unknown value of paraeter µ in shot k. A Gaussian log-likelihood can be used as a contrast function even if {e i } is not a sequence of Gaussian rando variables, i.e.: U(d k 1 +1,..., d k µ) = i= k Matlab code is available at lavielle/prograes lavielle.htl (d i µ) 2. (1)

3 Let G be defined as G(d k 1 +1,..., d k ) = (2) U(d k 1 +1,..., d k ˆµ (d k 1 +1,..., d k )). The iniu contrast estiate ˆµ (d k 1 +1,..., d k ) on segent k satisfies Then, G(d k 1 +1,..., d k ) U(d k 1 +1,..., d k µ). (3) G(d k 1 +1,..., d k ) = i= k 1 +1 (d i d k 1 +1: k ) 2 (4) where d k 1 +1: k is the saple ean in the shot, having boundaries at k and k, i.e., d k 1 +1: k = k d i. 1 ( k k 1 ) i= k 1 +1 A proper contrast function is defined as [14]: J(, d) = 1 N K G(d k 1 +1,..., d k ) (5) k=1 where d = (d 1,..., d N ), 0 = 0 and K = N. The objective is to find soe instants 1 < 2 <... < K 1, such that, µ K 1 +1 = µ K 1 +2 = = µ K. When the nuber of shot cuts is unknown, it can be estiated by iniizing a penalized version of J(, d). For any sequence of shot cuts, let pen() be a function of that increases with the nuber of shots K() and resebles the penalty ter in BIC added to the goodness of fit ter. The sequence of shot cuts ˆ iniizes H() = J(, d) + 2σ2 pen(). (6) N If the second ter in (6) is a function of N that goes to 0 at an appropriate rate as N goes to infinity, the estiated nuber of segents K( ˆ) converges in probability to K. Following the ost popular inforation criteria, such as the AIC and the BIC, the siplest penalty function is pen() = K(). To defend this choice, let d i = µ i + σ e i, 1 i N, where µ i is constant in each particular segent defined by the boundaries = ( 1, 2,..., K 1 ), i.e., µ (1<i<1) = µ 1,..., µ (K 1 <i<n) = µ K. If the sequence {e i } is a sequence of Gaussian white noise with variance 1, a penalized leastsquares estiate obtained by iniizing: H(, d) = 1 N K() k=1 i= k 1 +1 (d i d k ) 2 ) + 2σ2 pen() (7) N where the penalty function takes the for ( ) pen() = K() 1 + c log N K() iniizes E{ ˆµ µ 2 }, where ˆµ is the estiated sequence of the tie series eans and µ is the real sequence of the tie series eans. In [14], c was set to 2.5. The nuber of hoogenous tie series segents (shots) K can be estiated as follows [14]: 1) For 1 K K MAX, let (8) J K = JK MAX JK J KMAX J 1 (K MAX 1) + 1. (9) The new sequence J K is noralized such that J1 = K MAX and J KMAX = 1. The aforeentioned sequence decreases with an average slope equal to -1. 2) For 2 K K MAX 1, let D K = J K 1 2 J K + J K+1 and D 1 =. Then, the iniu penalized contrast (MPC) estiate of K is given by: K MP C = ax{1 K K MAX 1 D K > S} (10) where S is a threshold. KMP C is defined as the greatest nuber of shots K, such that the second derivative of J is greater than S. If no second derivative is greater than S, then there are no shot changes and K MP C = 1. Many different nuerical experients were concluded in order to find that the best perforing threshold value is S = C. Bayesian approach Let us express H() in (6) as H() = J(, d) + β pen() with β > 0. For a fixed value of β, the procedure described in Section III-B yields only one solution ˆ ˆK(β). In general, assue the following posterior distribution: p( d; α, β) = D (d ; α, β) exp ( α(j(, d) + βpen())) (11) where D (d ; α, β) is a noralizing constant, and α > 0. The ode of this posterior distribution is the iniu contrast estiate of shot cuts. Obviously, the ode depends on α, β that have to be estiated. First, the paraeters of the posterior distribution are estiated by the SAEM algorith, which yields the axiu likelihood estiates of α and β [15]. Second, an MCMC procedure is used to estiate the posterior distribution p( d; α, β). Let T be a teperature paraeter, which controls how the posterior distribution is concentrated around its ode. For different values of T (0, 1]: 1) Use the MCMC algorith to saple the conditional distribution p( d; ˆα/T, ˆβ) a) estiate the arginal conditional probability P { a changepoint at d i d; ˆα/T, ˆβ} for 1 i N 1, where d i is the i-th observation. b) estiate the conditional probability P {K() = K d; ˆα/T ˆβ}. 2) Copute the axiu a posteriori estiate of by iniizing J(, d) + ˆβpen(). The paraeter T should be chosen sall enough to neglect the odels aditting a low posterior probability and to increase the probability of the ost likely odels. Here, the MCMC algorith creates a hoogeneous Markov chain, since the teperature paraeter T reains constant. Accordingly, the axiization of the conditional distribution is achieved by using a siulated annealing procedure. D. Eliination of siilar and eaningless candidate keyfraes After the extraction of videos tie series and the shot boundary detection, we draw the teporal iddle frae of each segent as representative keyfrae. Any detected onochroatic keyfraes are discarded. Such keyfraes occur priarily due to fade-in or fade-out effects. As suggested in [9], eaningless keyfraes can be identified by the standard deviation of the hue color histogra. Zero or sufficiently sall values of standard deviation indicate such eaningless keyfraes. Here, the eaningless keyfraes are deleted by thresholding the ratio of pixels in edges at each frae. That is, if

4 TABLE I. PERFORMANCE EVALUATION OF VIDEO SUMMARIZATION METHODS. Precision Recall F-score CUS error Average K OV DT STIMO VSUMM VSUMM PENCONT Fig. 2. Saple video suarization obtained by the proposed ethod PENCONT and ground truth provided by 5 users. the ratio of the pixels in edges is less that 1% then the keyfrae is deleted. In onochroatic keyfraes, edges are not detected, so such keyfraes can easily be identified. To extract the edges, the Prewitt edge detector is used. Redundant keyfraes are eliinated based on the RGB color histogra. A siilarity atrix is calculated for the keyfraes that are candidate for the suary. The 16 bin color histogras of the red, green, and blue channel are extracted. The three histogras are erged in a single color vector of size 48. Color vectors are copared using the Manhattan distance. If the distance is less than τ HIST = 0.3, then aong the two copared keyfraes the first is deleted. During the coputation of the siilarity atrix, if the distance between two keyfraes is less than τ HIST, then calculations stop. There is no need to calculate the distance between this frae and the others since the rejection condition has been reached. This way, coputational coplexity decreases. To easure if two candidate keyfraes are siilar with respect to their texture, a siilarity atrix is built, coputing the Manhattan distance between their GIST descriptors [16]. If the distance is less than the threshold τ GIST = 0.2, then one of the is deleted. SURF features that detect and describe points of interest in a keyfrae are extracted as well. If two candidate keyfraes have ore than τ SURF = 4 atching points, they are considered siilar and they are also deleted [17]. Finally, the correlation between candidate keyfraes is calculated. If it exceeds τ CORR = 0.9, then the candidate keyfraes are considered siilar and one of the is deleted fro the suary. Note that the ain body of the duplicate keyfraes is discarded by the RGB color histogra, which is the coputationally fastest ethod to reject redundant fraes. Therefore, the siilarity atrices that are calculated for the SURF and GIST features, and the correlation between the keyfraes are created using a uch saller nuber of fraes than the nuber of fraes used to build the siilarity atrix of RGB color histogras. In Fig.2, the video suarization results and the ground truth of 5 users are shown for a video of the database. IV. EXPERIMENTAL EVALUATION The publicly available database 2, which consists of 50 videos chosen fro Open Video Project 3 has been used. Each video is about 100s long and has 3036 fraes on average. This database was also used for video suarization in [9] and [18]. To evaluate the quality of video suaries, the ethod proposed in [9], called Coparison of User Suaries (CUS), was eployed. In the CUS, the video suary is built anually by a nuber of users fro videos sapled at 1 fps. The user suaries (US) are taken as reference ground truth and are copared with the autoatic suaries (AS) derived by the ethod. A frae fro the US is considered as atched with a frae fro AS, if the Manhattan distance of their hue histogras is less than τ H = 0.5 [9]. Note that only one frae fro the AS can be atched with a specific frae fro the US. This ensures that there will not be nuber of atches exceeding the nuber of fraes in the US. The AS generated by different algoriths are copared with all the US to obtain quantitative assessent. Evaluation etrics used in [9], [18] to easure suarization quality of each algorith include the precision (P ), the recall (R), the F score, and the CUS error defined next: P = n atched n AS (12) R = n atched n US (13) F score = 2 P R P + R (14) CUS error = nas n atched n US (15) where n atched is the nuber of atched keyfraes between the AS and US, n AS is the nuber of keyfraes in the AS, n US is the nuber of keyfraes in a US. The proposed ethod, coined as video suarization with penalized contrasts (PENCONT), is copared with the DT [7], the STIMO [8], the VSUMM1 and the VSUMM2 [9], and the OV [19], whose suaries can be found at the website hosting the database. The results are suarized in the Table I. PENCONT has achieved results coparable to those of the state of the art techniques. Specifically, it has achieved the sae precision as the VSUMM1, the sae Recall as VSUMM2 and OV, and a copetitive CUS error of The average nuber of keyfraes per suary extracted by the PENCONT is 8.42, which is closer to the average nuber of keyfraes extracted by users (i.e., 8.64). PENCONT is preferred aong the other video suarization techniques, when the nuber of shots in a video is unknown and when real-tie operation is needed. The feature extracted fro each video is siple and can be coputed fast. Moreover, the coplexity of PENCONT can be adjusted by perforing change point detection on sub-sapled tie series. V. CONCLUSIONS In this paper, we have addressed shot boundary detection in video as a change point detection proble, using penalized contrasts on an appropriate tie series easuring the average hue value per frae over the video duration. Shot boundaries have been detected effectively even in cases of gradual transitions. The shot boundaries have been used to define hoogenous teporal video segents. The iddle frae fro each segent was extracted and considered as a keyfrae. Meaningless fraes have been rejected by observing the ratio of frae pixels in iage edges. Siilar fraes were detected and reoved based on their color captured by the RGB histogra, their texture easured by the GIST and SURF features, and their correlation. The resulting suary

5 is satisfactory and coparable to that of state of the art techniques with respect to coonly used objective figures of erit. Future work could eploy autoregressive data odels in the description of the video structure and study the ipact of different noise distributions on the efficiency of the ethod. [18] S. Mei, G. Guan, Z. Wang, S. Wan, M. He, and D. Feng, Video suarization via iniu sparse reconstruction, Pattern Recognition, vol. 48, no. 2, pp , [19] D. DeMenthon, V. Kobla, and D. Doerann, Video suarization by curve siplification, in Proc. 6th ACM Int. Conf. Multiedia, pp VI. ACKNOWLEDGEMENTS This research has been co-financed by the European Union (European Regional Developent Fund - ERDF) and Greek national funds through the Operation Progra Copetitiveness-Cooperation Research Funding Progra: 11SYN ATLAS. REFERENCES [1] S. Pfeiffer, R. Lienhart, S. Fischer, and W. Effelsberg, Abstracting digital ovies autoatically, Journal Visual Counication and Iage Representation, vol. 7, no. 4, pp , [2] M. Albanese, C. Cesarano, M. Fayzullin, A. Picariello, and V.S. Subrahanian, Encyclopedia of Multiedia, Springer US, [3] A. Hanjalic and H. Zhang, An integrated schee for autoated video abstraction based on unsupervised cluster-validity analysis, IEEE Trans. Circuits and Systes for Video Technology, vol. 9, no. 8, pp , [4] J. Na and A. Tewfik, Video abstract of video, in Proc. 3rd IEEE Workshop Multiedia Signal Processing, 1999, pp [5] B. Truong and S. Venkatesh, Video abstraction: A systeatic review and classification, ACM Trans. Multiedia Coput. Coun. Appl., vol. 3, no. 1, Feb [6] M. Basseville and I. Nikiforov, Detection of Abrupt Changes: Theory and Application, vol. 104, Prentice Hall Englewood Cliffs, [7] P. Mundur, Y. Rao, and Y. Yesha, Keyfrae-based video suarization using delaunay clustering, Journal Digital Libraries, vol. 6, no. 2, pp , [8] M. Furini, F. Geraci, M. Montangero, and M. Pellegrini, STIMO: Still and oving video storyboard for the web scenario, Multiedia Tools and Applications, vol. 46, no. 1, pp , [9] S. de Avila, A. da Luz, and A.A. De Araujo, VSUMM: A siple and efficient approach for autoatic video suarization, in Proc. 15th Int. Conf. Systes, Signals and Iage Processing, June 2008, pp [10] C. Ngo, Y. Ma, and H. Zhang, Video suarization and scene detection by graph odeling, IEEE Trans. Circuits and Systes for Video Technology, vol. 15, no. 2, pp , [11] D. Potapov, M. Douze, Z. Harchaoui, and C. Schid, Category-specific video suarization, in Proc. European Conf. Coputer Vision, pp Springer, [12] G. Evangelopoulos, A. Zlatintsi, A. Potaianos, P. Maragos, K. Rapantzikos, G. Skouas, and Y. Avrithis, Multiodal saliency and fusion for ovie suarization based on aural, visual, and textual attention, IEEE Trans. Multiedia, vol. 15, no. 7, pp , Nov [13] Z. Cernekova, I. Pitas, and C. Nikou, Inforation theory-based shot cut/fade detection and video suarization, IEEE Trans. Circuits and Systes for Video Technology, vol. 16, no. 1, pp , Jan [14] M. Lavielle, Using penalized contrasts for the change-point proble, Signal Processing, vol. 85, no. 8, pp , [15] B. Delyon, M. Lavielle, and E. Moulines, Convergence of a stochastic approxiation version of the e algorith, The Annals of Statistics, vol. 27, no. 1, pp , March [16] A. Oliva and A. Torralba, Modeling the shape of the scene: A holistic representation of the spatial envelope, Journal Coputer Vision, vol. 42, no. 3, pp , [17] H. Bay, T. Tuytelaars, and L. Van Gool, SURF: Speeded up robust features, in Proc. European Conf. Coputer Vision, pp Springer, 2006.