Performance Comparisons of Gradual Transition Detection Methods

Transcription

1 Performance Comparisons of Gradual Transition Detection Methods Shraddha C. Nistane M.E. student at Department of Electronics & Telecommunication,MITCOE Pune, India Dr. Krishna K. Warhade Professor at Department of Electronics & Telecommunication,MITCOE, Pune, India. ABSTRACT Video shot boundary detection is fundamental steps of video analysis, summarization and retrieval. For classifying shot and shot transition type and detecting video shot boundaries many algorithm have been proposed. Major challenge in the presence of object and camera motion is finding gradual transition. Therefore to develop a method which not only detect the transition but also differentiate it from camera and object motion is a challenging task. In this paper, we have compared B-spline interpolation method with traditional metrics like likelihood ratio, twin comparison, cross correlation and wavelet decomposition. Experimental study is performed on a number of videos that include dissolve transition and fast motion of camera and objects. The performance comparison of B-spline algorithm with other existing techniques validates its effectiveness in terms of better Recall & Precision. the Internet are greatly increased [1]. Videos are widely used in many kinds of areas such as entertainment, education and sports. However, video is a high dimensional data, does not have clear structure, and contains a mass of semantic information, so video retrieval and browsing is difficult and time-consuming for users. So searching for the video sections in which we are interested is a very difficult task. Therefore, it has been challenging task to develop new tools and technologies for efficient and effective indexing, browsing and retrieval of video data. Shot boundary detection is the first step in video retrieval. A sequence of frames with no major changes in the visual content taken by a single camera is termed as Shot. According to whether the transition between shots is abrupt or gradual, the shot boundaries can be categorized into two types: cut transition (CT) and gradual transition (GT). Keywords Shot boundary detection, gradual transition, dissolve, fades, wipe. I. INTRODUCTION With the development of the Internet and video processing technology, acquiring, storing and transmitting video are more and more convenient, and the amount of videos on 1 Fig.1. Consecutive frames with abrupt transitions

2 Fig.2. Consecutive frames of dissolve transitions Transition from one shot to another occurs suddenly in abrupt transition (AT) [2] Fig.1 shows consecutive frames with abrupt transition.gt (Gradual Transition) is a gradual transition. All non-cut transitions are gradual transitions GT, including dissolves, fade in/out, and wipes [2]. A dissolve in a video sequence is a transition where intensity of first frame gradually decreases and intensity of next frame gradually increases. During the fading transition, some monochrome frames separate two shots spatially and temporally. A wipe occurs when pixels from the second shot replace those of the first shot in a regular pattern. Fig.2 shows consecutive frames of dissolve transition. In this paper, we focused on the detection of gradual transition in the presence of motion. Specifically, we use a B- spline interpolation curve fitting technique [3] for estimating the associated linear-like production features and make use of goodness of fitting to detect the presence of the dissolve transition effects. The rest of the paper is organized as follows. A brief survey of previous works is presented in Section II. Section III presents the metric used for evaluation of shot boundary detection. The test video sequence, evaluation criterion and results have been describe in Section IV. Finally, we conclude this paper in Section V. 2 II. PREVIOUS WORK Several approaches have been proposed for the shot boundary detection. Ali Amiri [1] have proposed video shot boundary detection using eigen value decomposition and Gaussian transition detection. Y- N Li et al. [2] have used thresholding and bisection based composition on large number of non-boundary frames. Using motion intensity and motion suppression value Yang Xu et al. [4] have proposed 3-DWT based motion support algorithm. Jun Li et al. [5] have used color and the edge in different direction from wavelet transition coefficient. Vaselein chasanis et al [6] have proposed algorithm for detecting abrupt cut and dissolve by using color histogram and ᵡ 2 value. The algorithm proposed in [7], based on k-step slipped window using low feature and edit feature of the video shot. Using global feature such as color histogram, Mohanta et al. [8] have proposed model based shot boundary detection. Na. Lv et al. [9] have proposed video shot boundary detection algorithm by using gray variance based method and block color histogram. Using C frames, T.Lu and P.N Suganthan [10] have proposed an accumulation based algorithm. Classification algorithm for shot boundary detection has been proposed by K. I Koumousis [11] using kappa coefficient. Liu and Jian Xun Li[12] have proposed video segmentation algorithm by calculating difference between DC images of all I frames images. After reviewing the literature in details, we found that most of the algorithms are unable to differentiate between gradual transition and motion. While most of the algorithms which gives good results with hard cuts, fails to provide output with soft cuts. Sudden and extensive changes in visual content occur usually in hard cuts, while soft cuts feature shows slow and gradual changes. We have described major metrics and method

3 in section III and are to detect gradual transitions in the test video sequences. III. MAJOR METRIC USED FOR EVALUTION Likelihood Ratio: Jain et al. [13] computed a likelihood ratio test based on the assumption of uniform of second order statistics. It is a standard hypothesis test in which a ratio of probabilities is used as the test statistic. This is a statistical method which expands on the idea of pixel difference by breaking the images into regions and comparing the obtained statistical measures of the pixel in those regions. It is defined as algorithm, it has been assumes that two frames which have a common background and unchanging objects will show little difference in their histograms. The basic formulation for histogram comparison is as follows: the histogram (either color or grayscale) is computed for each frame and the difference is calculated as D(i, i+1) = (2) Where Hi(j) is the j th element of the histogram of the i th frame, and B is the number of bins in the histogram. In this paper we used a difference function defined by the histogram intersection D (i, i+1) = 1- Intersection ( ) = (1) Where, LHR is a likelihood ratio between two consecutive regions, where is the mean of current frame and is the standard deviation of current frame. Limitation of the likelihood ratio is that no change will be detected if the two images, having the same variance and mean but totally different probability density functions, are compared. = 1-. (3) Cross Correlation metric: The cross-correlation [15] coefficient has been widely used as a metric for shot boundary detection. The correlation between consecutive frames is computed as CM ( i ) = Twin Comparison: The twin-comparison shot detection scheme [14] is a well known method to detect both abrupt and gradual transitions. It utilizes two thresholds, one is for finding abrupt transitions and the other is for locating gradual transitions. The twin-comparison method in addition to interframe differences uses cumulative differences between frames of a gradual transition. In twin comparison 3... (4) Where, CM is cross-correlation metric coefficient between consecutive frames. Then CM is found out for 1 I K 1( i.e. for all consecutive frames in the video clip) using Eq. 4 and then threshold is applied to detect shot boundaries. A high correlation signifies similar frames, probably belonging to the

4 same shot, whereas a low value is an indication of a shot break. Wavelet based metric: We have decomposed each frame by using third level decomposition using daubechies second order filter as shown in Fig. (3). Fig. (3). Three level wavelet decomposition The wavelet transform [16] is used as a shot detection metrics (5) values. The algorithm is described in details in [3]. Step: 1 1) Read a video sequence and de-frame into group of 20 frames 2) Eliminate RGB to gray frame 3) Decompose the input video sequence up to scale of J=3 Step: 2 Compute mean and variance of LL3 part of each frame. Step: 3 Calculate the interframe standard deviation. Step: 4 Obtain B-spline interpolation curve by putting the values of standard deviation as input. Repeat the above steps for next 20 frames till the last frames of video clip are reached. IV. TEST VIDEO AND EVALUTION CRITERION USED FOR METRIC EVALUTION: Test Video Sequence: Where, N is a image size, l are no.of levels, k are frame number. B-spline interpolation Based Method B-spline interpolation method uses goodness of fitting to detect the presence of gradual transitions. In this algorithm, first step to extract low-scale and low-resolution image sequence from a video sequence by third level wavelet decomposition. In the second step, Mean and variance of LL3 part of each frame is found out. In third step, difference values of each sub-band helps in computing standard deviation. In the fourth and last step the B-spline interpolated curve on each pixel-stream of the windowed frames is estimated by putting standard deviation 4 All algorithms have been tested on clips of movies Titanic (TI), Alvin and chipmunks part 2 (AC2), Alpha wind and clock (AWC), Landscape (LS), color harmony for your home (CHFYH). These video sequences are manually observed frame by frame to find actual transition using virtual dub software. We mostly considered the video clips from these movies in the presence of dissolve transition with fast camera and object motion. Number of frames considered for test video sequence in each movie is shown in Table 1. Evalution criterion: For evaluation performance of shot boundary detection algorithm, the two metrics recall and precision are used.

5 Recall is defined as (6) Whereas precision is defined as (7) Where D is the total number of actual frames with dissolves boundaries, C is the number of dissolve frames correctly detected by the algorithm; M is the number of number of dissolve frames missed by the algorithm and FP is false positives detected by the algorithm. F1 measure is used to rank the performance of the different algorithms [4]. F1 combines recall and precision with equal weight. F1 measure is a harmonic average of recall and precision and is given as below. (8) Evalution Results of the Traditional Shot Boundary Detetction Metric : The performance of traditional metrics such as B-spine Interpolation, likelihood ratio, WDBF, correlation and twin comparison has been compared on the same video data sequence. The performance comparison between all these algorithm are shown in Table 2. From the Table 2, overall it has been observed that due to the fast camera and object motion the likelihood ratio, WDBF, correlation and twin comparison method gives poor results in terms of false positive, whereas B-spline interpolation gives more accuracy than other metrics. We have selected the test video sequence which contains significant camera motion with dissolve transition. Fig.4 shows consecutive frames from the movie Landscape. The portion of the video considered for analysis consist of 409 frames with camera motion (frames 3-84) and 69 frames with dissolve transition (frames 79-96) and (frame ) Table 1. Number of frames considered for analysis from test video sequence Movie Numbe r of frames No. of dissolve s AC LS TI CHFY H AW C have applied B-spline interpolation, likelihood ratio, WDBF, correlation, twin comparison to the above test video sequence. Fig. 5(a), Fig. 5(b), Fig. 5(c) shows results using B-spline interpolation, WDBF and correlation metrics over 409 consecutive frames. From the Fig. 5(a), it can be observed that algorithm has detected all the frames of dissolves between frame and frame It can be clearly observed from Fig. 5(b) that WDBF metric is unable to find initial frames of dissolves. This metrics also missed some dissolves frame between frames

6 Table 2. Performance comparison results between B-spline interpolation, likelihood ratio, WDBF, correlation and Twin comparison Metric Video AC2 LS TI CHFYH AWC B-spline R P F LHR R P F WDBF R P F Correlation R P F Twin comparison R P F

7 WDBF B-spline ineterpolation metric International Journal of Computer & Mathematical Sciences frame 4 frame 40 frame 80 frame 84 frame 90 frame 99 frame 196 frame 200 frame 205 frame 207 frame 124 frame 211 frame 217 frame 222 frame 228 frame Fig 4. Consecutive frames from movie clip Landscape showing dissolve transition object motion whereas; B-spline interpolation interpoated algorithm is able to detect dissolve transitions in such conditions. The performance of B- spline is better than the other metric in terms of Recall, Precision and F1 measure Frame Index Fig. 5(a) Result using B-spline interpolation Fig 5(c). shows that cross correlation metric detects more false positives due to fast camera and object motion. Hence, WDBF and Correlation metrics gives poor performance to detect dissolve transition in the presence of camera and 4.5 x Frame Index Fig. 5(b) Result using WDBF metric 7

8 cross correlation metric International Journal of Computer & Mathematical Sciences Frame Index Fig.5(c). Result using cross correlation metric V.CONCLUSION In this paper, we have compared B-spline interpolation method with traditional metrics like likelihood ratio, twin comparison, cross correlation metric and wavelet based metric. Specifically, we use a B-spline interpolation curve fitting technique for estimating the associated linear-like production features and make use of goodness of fitting to determine the presence of the target transition effects. It has been observed that B-spline interpolation gives better results in identifying gradual effects such as dissolve, fade and wipe transitions. Also the performance of this algorithm is better than the other metric in terms of Recall, Precision and F1 measure. We have extensively tested major metrics and presented experimental results that were obtained by applying these approaches on video sequence from five different movies. Hence, Bspline interpolation algorithm plays an important role in finding the gradual transitions and classifies their types and pattern but its gives false positives in presence fast camera and object motion. Reference: [1] Ali Amiri, Mahmood Fathy, Video shot boundary detection using generelized eigen value decomposition and Gaussian transition detection, Computing and Informatics, vol. 30, pp , [2] Y.-N, Li, Z.-M, Lu, X.-M, Niu, Fast video shot boundary detection framework employing pre-processing techniques, IET, Image Process, vol. 3, iss. 3, pp , 2009 [3] Jeho Nam and Ahmed H. Tewfik, Detection of gradual transitions in video sequences using B-spline interpolation IEEE Transaction on Multimedia, vol. 7, iss. 4, pp , Aug [4] Yang Xu, Xu De, Gaun Tengfei, Wu Aimin, Lang Congyan, 3 DWT based motion suppression for video shot boundary detection, Knowledge - based Intellingent Information and Engineering System, Lecture notes in Computer Science, vol. 3682, pp , 2005 [5] Jun Li, Youdong Ding, Yunyu Shi, Qingyue Zeng, DWT-based shot boundary detection using support vector machine, Information Assurance and Security, vol. 1, pp , Aug [6] Vasileios Chasanis, Aristidis Likas, Nikolaos Galatsanos, Simultaneous detection of abrupt cuts and dissolves in videos using support vector machines, Pattern Recognition Letters 30, pp , 2009 [7] Tuanfa Qin, Jiayu Gu, Huiting Chen, Zhenhua Tang, A fast shot boundary detection based on K-step slipped window, 2 nd IEEE international Conference on Network Infrastructure and Digita Content, pp , Sept

9 [8] Partha Pratim Mohanta, Sanjoy Kumar Saha, and Bhabatosh Chanda, A model-based shot boundary detection techniqueusing frame transition parameters, IEEE transaction on Multimedia, vol. 14, iss. 1, pp , Feb [9] Na Lv, Zhiquan Feng and Jingliang Peng, mutual information based video shot boundary detection Image Analysis and Signal Processing, pp. 1-5, 9-11 Nov [10] T. LU tong, P.N. Suganthan, An accumulation algorithm for video shot boundary detection, Multimedia Tools and Applications,vol. 22, iss. 1, pp , Jan [11] K. Koumousis, V. Fotopoulos, A. N. Skodras, A new approach to gradual video transition detection, Informatics (PCI), pp , 5-7 Oct 2012 [12] Liu Liu, Jian-Xun Li, A novel shot segmentation agorithm based on motion edge feature, 2010 Synopsis on Photonics and Optoelecctronic, pp. 1-5, 19-20, June [13] Jain, R., Kasturi, R., Schunck, B.: Machine vision, pp , McGraw-Hill, New York, [14] Fa-Xin Yu1, Zhe-Ming Lu1 and Yue-Nan Li2 Dissolve detection based on twin-comparison with curve fitting International Journal of Innovative, ISSN ,vol. 7, No. 5(A), pp , May [15] Sethi, I.K, Patel, N.: A statistical approach to scene chane detection, SPIE proc. Storage Retr. Image video database III 2420, pp , 1995 [16] Khin Thandar Tint, Dr. Kyi Soe, Key frame extraction for video summarization using DWT wavelet statistics International journal of research in computer engineering & technology, vol. 2, No. 5, May