Transition Detection Using Hilbert Transform and Texture Features

Transcription

1 Amerian Journal of Signal Proessing 1, (): 35-4 DOI: 1.593/.asp.1.6 Transition Detetion Using Hilbert Transform and Texture Features G. G. Lashmi Priya *, S. Domni Department of Computer Appliations, National Institute of Tehnology, Tiruhirappall Tamilnadu, India Abstrat In this paper, we propose a new method for deteting shot boundaries in video sequenes by performing Hilbert transform and extrating feature vetors from Gray Level Co-ourrene Matrix (GLCM). The proposed method is apable of deteting both abrupt and gradual transitions suh as dissolves, fades and wipes in the video sequenes. The derived features on proessing through Kernel -means lustering proedure results in effiient detetion of abrupt and gradual transitions, as tested on TRECVid video test set ontaining various types of shot transition with illumination effets, obet and amera movement in the shots. The results show that the proposed method yields better result ompared to that of the existing transition detetion methods. Keywords Transition detetion, Hilbert Transform, GLCM, Feature Vetors, Kernel -means lustering, Performane Evaluation 1. Introdution Video shot boundary detetion plays maor role in digital video analysis, having appliations in many important video analysis domains lie video indexing, video ompression, video ontent browsing, retrieval and others. The video shot represents the elementary unit of the video and refers to a ontinuous sequene of one frame aptured uninterruptedly by one amera. Shots are assembled during the editing phases using varieties of tehniques lie fade, dissolve and wipe are referred as gradual shot transitions. A rapid transition from one shot to another is simply referred to as hard uts or uts (abrupt shot transition). A soft ut represents a gradual transition between two shots, whih means a sequene of video frames that belong to both the first and the seond video shot. The proess of identifying the different shot transition within a video sequenes is nown as Video Shot Boundary Detetion or Shot Based Video Segmentation or video partitioning[1]. Some diffiulties are faed in finding the shot boundaries in the presene of amera and obet motion, illumination variations and speial effets. Two approahes that are broadly used for video shot boundary detetion are pixel domain proessing and ompressed domain proessing. Comparison of various shot detetion methods has shown that the pixel domain methods are more aurate when ompared to that of ompressed * Corresponding author: gg_lashmipriya@yahoo.o.in (G.G.Lashmi Priya) Published online at Copyright 1 Sientifi & Aademi Publishing. All Rights Reserved domain methods[-4]. On the other hand, ompressed domain methods wors faster. In order to improve the auray of the detetion proess in ompressed domain, more important and onfined features are to be extrated from the video data. The effiieny of the detetion proess lies on extrated features, metris used to find relation between the onseutive frames and the transition identifiation proess whih ategories shot boundary and not shot boundary. Initial way to he for visual ontent dissimilarity is to ompare the pixel intensities between the two onseutive frames[3]. They are very sensitive to amera and sene obet motion in the video sequenes. Histograms[3] are the most ommon method used to detet shot boundary. It omputes gray level or olor histogram of two onseutive frames. This method has one maor benefit that the spatial distribution of the information is disregarded. The drawba is that if two frames have quite same histogram while their ontents are dissimilar extremely, they may result in miss detetion. In papers[5] the authors have ompared the ontent of two onseutive frames using the edge hange ratio (ECR). The main disadvantage of this method is they produe relatively high number of false hits, espeially in the video senes in whih they ontain high speed motion. Yoo et.al[6] have used the inter-frame orrelation between two onseutive frames and +1to detet abrupt transition and used loalized edge blo information for deteting gradual transitions. The maor drawba of this method is that they are sensitive to illumination effet, amera and obet movement. Yoshihio awai et. al. [7] have proposed a method that utilizes multiple features for shot boundary detetion. In this paper eah extrated feature differene value is ompared

2 36 G. G. Lashmi Priya et al.: Transition Detetion Using Hilbert Transform and Texture Features with the predefined threshold at different stages. The main drawba of this method is the use of predefined threshold value for eah stage of the detetion proess. As an improvement to the method[7], Lian in his wor[8] has used multiple features (pixel wise differenes, olor histogram, motion) whih detet transitions in serial manner using different threshold values. The number of thresholds used is omparatively less than the previous method[7]. Zhang et al. [9] have used the HSV histogram differenes of the two onseutive frames as the feature for evaluating the olor information between the frames. The main disadvantage of this method is that if two onseutive dissimilar frames have same histogram may result in missed hit. In paper[1] Color Layout Desriptor (CLD) is onsidered as feature and the differene between the onseutive frames are omputed. This method wors well for hard uts but las its performane in the presene of long dissolve and fades regions in the video stream. Abdul Salam[11] in his paper has suggested that 1-D Hilbert transform an be utilized to handle image signals and video transmission. Author has also onluded that Hilbert transform is onsidered to be useful in manipulating images. In our previous paper[1] we have deteted abrupt transitions using Hilbert transform and GLCM based features. As an extension, we have made hanges in the features to be onsidered for detetion proess whih wors well for both abrupt and gradual transitions. Also our extended wor is ompared with the reent methods and the performane of the proposed wor is disussed in setion 3. In the proposed method, we have seleted the texture information as the feature vetor. To redue the detetion time, the features are extrated from the Hilbert transformed frames[11]. Then the proposed method uses ernel -means lustering[13] to lassify the video frames as shot or no shot. The rest of this paper is organized as follows. The proposed method is presented in setion along with the desription and method for identifying transitions. Experimental results are disussed in setion 3 and finally onlusion is derived in setion Hilbert Transform The Hilbert transform[14] of a funtion f(t) an also be represented as: 1 1 g() t = H{ f()} t = f()* t = f() t dτ π t π (1) t τ g(t) an be evaluated as the produt of the transform of f(t) with i*sgn( ω), gt ( ) = F( ω)( i*sgn( ω)) () where F( ω) is the Fourier transform of the funtion f() t, sgn( ω ) is the odd signum funtion. sgn( ω ) is +1 for ω >, for ω = and -1 for ω <. The Hilbert transform is onsidered as a filter, whih simply shifts phases of all frequeny omponents of its input, by ± π/ radians. In general, video is a sequential olletion of frames whose intensity values are represented as a set of real numbers R. More features lie olor, edge, texture et., information an be extrated from eah frame. Instead of using the whole frames intensity values as features, transformed information an also be onsidered, is the ondensed representation of the original. In our wor the transformed information is obtained by applying above disussed Hilbert transformation over the frames. To perform Hilbert transformation, the input video data is to be onverted into 1-D signal f (t). The RGB frame shown in figure a is onverted into gray sale image as shown in figure b and the Hilbert transformed image is shown in figure. Based on this transformed information, GLCM features are extrated.. Proposed Method In the proposed method, the video information is onverted into Hilbert transformed signal and the Gray Level Co-ourrene Matrix (GLCM) of the transformed data have been used as the soure to extrat features for deteting shot boundaries in video sequenes. Texture features are extrated from GLCM and the distane measure between the adaent frames is alulated. Based on the similarity values between the onseutive frames, lustering of shot transition frames and non-transition frames are performed using Kernel -means lustering. The blo diagram of the proposed method is shown in figure 1. In the next subsetions detailed explanation about Hilbert transforms, Gray level o-ourrene matrix, texture features extration and ernel -means lustering are disussed. Figure 1. Blo diagram of the proposed method

3 Amerian Journal of Signal Proessing 1, (): Figure. (a) Original frame (b) Gray sale frame () Hilbert transformed frame.. Gray Level o-ourrene Matrix (GLCM) The texture filter funtions provide information about the texture of a frame but fail to provide information about shape. A statistial method that onsiders the spatial relationship of pixels is the Gray-Level Co-ourrene Matrix (GLCM) and also nown as the gray-level spatial dependene matrix[15]. GLCM is alulated by finding the frequeny of the gray level pixel intensity value i ours in a speifi spatial relationship to a pixel with the value. Eah element () in the resultant GLCM is the sum of the number of times that the pixel with value i ourred in the speified spatial relationship to a pixel with value in the input frame. Here, the o-ourrene matrix is omputed based on two parameters, whih are the relative distane d between the pixel pair () and their relative orientation θ. d is measured in pixel number. Normally θ is quantized in four angles (, 45, 9, 135 ). Let Pi (,, dθ, ) represents the GLCM for an image I ( m, n) for distane d and diretion θ an be defined as m n 1, if I( p, q) = i and I( p + dθ, q + dθ1) = Pi (, d,, θ ) = (3) p= 1q= 1, otherwise For a hosen distane d, four angular GLCM are onsidered ie. Pi (, d,, ), Pi (, d,,45 ), Pi (, d,,9 ), Pi (, d,,135 ). For eah θ value, it s dθ and dθ1 values are (, 1) for, (-1,-1) for 45, (1, ) for 9, (1,-1) for Texture Features Extrated from GL CM Various texture features[11] an be extrated from GLCM and from that four features Contrast (f 1 ), Correlation (f ), Energy (f 3 ) and Homogeneity (f 4 ) are onsidered. The ontrast feature f 1 measures the intensity ontrast between a pixel and its neighbour over the whole image and is alulated using (4). The range of ontrast depends on the size of the matrix. i.e. Range =[, (size(glcm,1)-1)^]. The orrelation feature f is a measure of gray tone linear dependeny in the image. It measures how a pixel is orrelated to its neighbour over the whole image and it is alulated using (5). Correlation is 1 or -1 for a perfetly positively or negatively orrelated image. The energy feature f 3 provides the sum of squared elements in the GLCM. It is also nown as uniformity, uniformity of energy. Energy is alulated using (6). The homogeneity feature f 4 measures the loseness of the distribution of elements in the GLCM to the GLCM diagonal. Homogeneity is 1 for a diagonal GLCM and alulated using (7). f1 = i Pi (, ) (4) f ( i µ i)( µ Pi ) (, ) = (5) σσ f 3 i = Pi (, ) (6) Pi (, ) f4 = (7) 1+ i Where µ is mean, σ is the standard deviation of GLCM ( P ). However, for a onstant image the texture features are f1 =, f = NAN, f3 = f4 = 1. The four features f1, f, f3, f4 are the funtions of distane and angle. For a hosen distane d, four angular GLCM are measured and hene a set of four values for eah of the four features are obtained. On the whole 16 feature measures are generated. To redue the number of features, the average of four angular GLCM is taen using (8) and then the four features are extrated as the Average GLCM (AGLCM). 1 AGLCM = P (, i, d, θ ) (8) 4 θ =,45,9,135 A four dimensional vetor F = [ f1 f f3 f4 ] from AGLCM is onstruted whih is feature representing eah frame. In order to find the relationship between the onseutive frames, dissimilarity / similarity between these frames are to be arried out. The simplest method for shot detetion is to ompute absolute differene D i(, + 1) between the F of and +1 th frame. Thus the relation between the onseutive frames is alulated using: D = F F (9) i (, + 1) i ( ) i ( + 1) The resultant D is again a four dimensional differene vetor whih represents the disontinuity between the onseutive frames and +1. After omputing a set of features omputed from eah frame and similarity between the onseutive frames, a shot hange detetion algorithm is required to detet where the atual transition taes plae by using the disontinuity values..3. Shot Change Detetion In order to detet the video shots, the set of differene

4 38 G. G. Lashmi Priya et al.: Transition Detetion Using Hilbert Transform and Texture Features vetors D are proessed using lustering method. The frames are grouped into two lusters: Shot transition and no transition lusters. In general, -means lustering[13] is one of the most popular lustering algorithms. The standard -means lustering tehniques separate the luster by a hyper-plane; where squared Eulidean distane is used as the distortion measure. Given a set of data points x1, x,..., x n, the -means algorithm finds lusters C 1, C,..., C that minimizes the obetive funtion DC ( ) = 1 = x x i i C xi m where m = (1) = 1 xi C C The th luster is denoted by C and the entroid of the luster is denoted by m. A maor drawba is this standard lustering tehnique is it annot separate lusters that are non-linearly separable in input spae. To tale this problem, ernel -means[13] is emerged where data points are mapped to a higher dimensional feature spae using a non linear funtion φ [16] before lustering points. Then ernel -means partitions the points by linear separators in the new spae. The main obetive of the ernel -means an be written as a minimization of DC ( ) = 1 = ϕ( x ) x i i C ϕ( xi) m where m = (11) = 1 xi C C As a result, given a ernel matrix, distane between the points and entroid is alulated. A ernel funtion is ommonly used to map the data points to inner produts. Table 1. Desription of the video sequenes in the test set Using the ernel -means lustering algorithm the feature vetors obtained from (9) are lustered into shot transition and no transition lusters. 3. Experimental Results and Disussion The proposed method for deteting video shot boundary is tested on various TRECVid video datasets[16] and on few other publially available data[17]. The videos ontaining obet/ amera motion and illumination effets inside single shots are seleted. Video sequenes, varies in duration, frame rate and resolution. To speed up the alulations the videos are downsampled with resolution 18 x 16 were used in our experiments. Desription of the video sequenes in the sample test set is depited in table 1. A orretly deteted shot transition is alled a hit (H), a not deteted shot is alled a missed hit (M) and a falsely deteted shot transition is alled a false hit (F). For the evaluation of the proposed wor, the preision, reall and ombined measures are alulated using Pr eision ( P) = H ( H + F) Re all ( V ) = H ( H + M ) Combined Measures ( F1) =. P. V ( P + V ) The higher these ratios are, the better the performane. Our proposed algorithm is implemented in Matlab 7.6, Table. shows the experimental results of the proposed method for detetion of abrupt and gradual transition. Video Sequenes Number of Frames Duration(s) Abrupt Transition Gradual Transition Total Number of transitions TRECVid Data set[16] BG_ BG_ BG_ BG_ Open data set[17] Anni Anni Anni Bor CNN News Movie Total Table. Performane evaluation of the proposed method Video Sequenes Abrupt Transition Gradual Transition Overall Transition P V F1 P V F1 P V F1 BG_ BG_ BG_ BG_ Anni Anni Anni Bor CNN News Movie Average

5 Amerian Journal of Signal Proessing 1, (): Table 3. Quality measures for various transition detetion methods Proposed method HSV Histogram SBD using Multiple Color Layout desriptor Video Correlation Based differenes features Sequenes P V F1 P V F1 P V F1 P V F1 P V F1 BG_ BG_ BG_ BG_ Anni Anni Anni Bor CNN News Movie Average However, based on the frame numbers in the first luster the start and end of the shots are identified. The experimental results of the existing methods lie orrelation based[6], Color layout desriptor[1], multiple features[8] and HSV histogram differenes[9] are ompared with our proposed method. The results for overall transitions in the video sequenes are onsidered. On ompared to other methods listed in the table 3, our proposed method outperforms with ombined measure of 9 % where other methods yields 8%, 85%, 83% and 87%. 4. Conlusions a. Sum of omponents of Differene Vetor In this paper, a new shot boundary detetion algorithm is proposed whih extrats feature vetors from GLCM of the Hilbert transformed video. The derived feature vetors on proessing through Kernel based -means lustering proedure results in effiient detetion of transitions. Experiments have been performed on TRECvid data sets and publially available data sets. As a onlusion, our proposed method s performane is better when ompared with that of the other existing methods. REFERENCES [1] Tudor Barbu, A novel automati video ut detetion tehniques using Gabor filtering, Computer and Eletrial Engineering, vol.35, pp , Sep. 9. b. Clustering using ernel -means algorithm Figure 3. Results obtained for Anni1 video sequene The sum of the omponents of differene vetor D i alulated using (9) of onseutive frames is shown in figure 3a. The patterns obtained in the figure represent the abrupt and gradual transitions in the video sequenes. In figure 3b, the feature vetors are lustering using Kernel -means lustering as shot transition and no transition lusters. The start frame of shot are lustered together in shot transition luster and other frames are in no transition luster. [] U.Garg Kastur and S.H.Strayer, Performane haraterization of video shot hange detetion methods, IEEE Trans. on Ciruits and Systems for Video Tehnology, CSVT-1(1), pp.1-13,. [3] Rainer Lienhart, Comparison of automati shot boundary detetion algorithm, Image and video proessing VII, in Pro. SPIE, , [4] J. S. Borezy and L. Rowe, Comparison of video shot boundary detetion tehniques, in Pro. IS&T/SPIE Storage and Retrieval for Still Image and Video Databases IV, vol. 67, pp ,1996.

6 4 G. G. Lashmi Priya et al.: Transition Detetion Using Hilbert Transform and Texture Features [5] R. Zabih, J. Miller, and K. Ma A feature-based algorithm for deteting uts and lassifying sene breas, in Pro. ACM Multimedia 95, San Franiso, CA, pp. 189, [6] Hun-Woo Yoo & Han-Jin Ryoo & Dong-Si Jang, Gradual shot boundary detetion using loalized edge blos, Multimedia tools appls. 8: 83-3, 6. [7] Y. Kawa H. Sumiyosh and N. Yagi. Shot Boundary Detetion at TRECVID 7, In TRECVID 7 Worshop, Gaithersburg, 7. [8] Shiguo Lian. Automati video temporal segmentation based on multiple features. Soft Computing, Vol. 15, , 11. [9] Weigang Zhang, et. al., Video Shot Detetion Using Hidden Marov Models with Complementary Features, In Proeedings of the First International Conferene on Innovative Computing, Information and Control.Vol.3. mputersoiety.org/1.119/icicic.6.549, 6. [1] Damian Borth, Adrian Ulges, Christian Shulze, Thomas M. Breuel: Keyframe Extration for Video Tagging & Summarization. In: Informatitage pp , 3. [11] Ahmed O. Abdul Salam, Hilbert transform in image proessing in Pro. ISIE, pp , [1] Lashmi Priya G.G., Domni S., Video ut detetion using Hilbert transform and GLCM, in pro. of IEEE-International Conferene on Reent Trends in Information Tehnology, , 11. [13] Inderit S. Dhillon, Yuqiang Guan, and Brian Kulis, Weighted Graph Cuts without Eigenvetors: A Multilevel Approah, IEEE Transations On Pattern Analysis And Mahine Intelligene, VOL. 9, NO. 11, , 7. [14] Panhamumar D Shula, Complex wavelet transforms and their appliations, Thesis of Master of Philosophy(M.Phil.), University of Strathlyde, United Kingdom, 3. [15] Harali, R.M., Shanmugam, K., Dinstein, I.: Textural Features for Image Classifiation, IEEE Trans. Systems Man Cybernet. SMC-3, 61 61, [16] B. Sholopf, A. Smola, and K.-R. Muller, Nonlinear Component Analysis as a Kernel Eigenvalue Problem, Neural Computation, vol. 1, pp , [17] TRECVID Dataset website: [18] Video Dataset: