Video Content Representation using Optimal Extraction of Frames and Scenes

Transcription

1 Vdeo Content Representaton usng Optmal Etracton of rames and Scenes Nkolaos D. Doulam Anastasos D. Doulam Yanns S. Avrths and Stefanos D. ollas Natonal Techncal Unversty of Athens Department of Electrcal and Computer Engneerng 9, Heroon Polytechnou str., Zografou, Athen Greece e-mal: Abstract In ths paper, an effcent vdeo content representaton s proposed usng optmal etracton of characterstc frames and scenes. Ths representaton, apart from provdng browsng capabltes to dgtal vdeo database also allows more effcent content-based queres and ndeng. or performng the frame/scene etracton, a feature vector formulaton of the mages s proposed based on color and moton segmentaton. Then, the scene selecton s accomplshed by clusterng smlar scenes based on a dstorton crteron. rame selecton s performed usng an optmzaton method for locatng a set of mnmally correlated feature vectors.. Introducton The rapd development of vdeo and multmeda applcatons has enabled users to handle large amounts of vsual nformaton. However, tools and algorthms for effectve organzaton and management of vdeo databases and for content-based search and retreval are stll lmted. or ths reason, a new standardzaton phase s currently n progress by the MPEG group n order to develop algorthms for audovsual codng (MPEG-4 [7] and content-based vdeo storage, retreval and ndeng n multmeda applcaton based on object etracton from scenes (MPEG-7 [8] []. In the contet of ths paper, we present an effcent vdeo content representaton usng optmal etracton of characterstc frames and scenes of vdeo sequences. Ths representaton, apart from provdng browsng capabltes to dgtal vdeo database also allows content-based queres and ndeng to be performed more effcently. Several approaches for ndeng and retreval from vdeo sequences have been proposed n the recent lterature. In [5] a framework whch enables contentbased retreval of vdeo sequences usng moton and teture cues has been proposed. Another approach dealng wth ndeng and retreval usng relevance feedback of mars was presented n [9]. In [] t s developed a method for buldng an mage representaton usng lbrary bass elements that are facltated by a jont adaptve space and frequency graph. Automatc ndeng of TV news recordngs has been analyzed n [6] where shots contanng persons are dentfed and news tems are recovered. An approach for automatc vdeo segmentaton and content-based retreval based on a temporally wndowed prncpal component analyss of a subsampled verson of the vdeo sequence s presented n [4]. The above technque ether eplot color, moton or teture nformaton n order to provde content-based query capabltes or use egenvalue decomposton to reduce the mage dmenson. Our approach s orented to etractng a small amount of nformaton whch s suffcent to provde a meanngful representaton of a vdeo sequence. Ths approach not only provdes a more effcent way for vdeo ndeng, but also results n reducng the storage requrements and thus permts easy management of multmeda databases. In one of our earler works [], an ntegrated framework for automatc etracton of characterstc frames has been proposed. The etracton mechansm was based on tme varatons of the frame feature vector generated usng color and moton segmentaton. However, snce smlar frames may be characterzed by dfferent segment the overall procedure was rather senstve and heavly dependent on the adopted segmentaton algorthm. In ths paper, the frame selecton mechansm s enhanced by ntroducng an optmzaton method for locatng a set of mnmally correlated feature vectors. urthermore, a scene selecton mechansm s proposed, based on mnmzaton of a dstorton crteron for clusterng the scene feature vectors.. eature etracton The feature vector etracton procedure s performed n a way smlar to [] and s brefly dscussed n the sequel. The scene and frame selecton mechansms are then descrbed, and epermental results are presented... Scene cut detecton The frst stage of the feature etracton procedure ncludes a scene cut detecton technque, n order to locate the man shots of a vdeo stream. Snce vsual

2 content s typcally stored n MPEG compressed format, t s preferable to perform the feature etracton drectly n the compressed doman. As a result n our approach scene cut detecton s acheved by computng the sum of the block moton estmaton error over each frame and detect frames for whch ths sum eceeds a certan threshold []... Color and moton segmentaton Color and moton segmentaton provde a powerful representaton of each vdeo frame, more orented to the human percepton. In general, the number, sze and locaton of objects as well as ther color, moton, or teture characterstcs gve more meanngful nformaton for an mage than raw pels. Thu a color and moton segmentaton technque s appled to each vdeo frame. Block resoluton has been adopted both for reducng the requred computatonal tme and eplotng nformaton whch already ests n the MPEG codng standard. To avod oversegmentaton problem we have proposed a herarchcal block-based segmentaton algorthm descrbed n []. Apart from nformaton provded by color or moton segmentaton other features are also ncluded n the feature vector, such as nformaton of color and moton hstograms or approprate ac coeffcents of the DCT transform..3. eature vector formulaton A multdmensonal feature vector s generated for each frame by transformng the mage doman to another doman (the feature one, more effcent for vdeo content descrpton. Color/moton segment propertes cannot be drectly used as feature vector elements snce ther sze s dfferent for each frame. To overcome ths problem and to acheve better feature representaton fuzzy classfcaton of the etracted propertes s performed as descrbed n []. nally, based on the feature vectors of all frames wthn a scene, a multdmensonal scene feature vector s constructed, descrbng the average frame propertes of the scene. 3. Scene selecton mechansm Based on scene feature vector an optmal etracton of the most characterstc scenes s performed. Ths s accomplshed by clusterng smlar scene feature vectors and selectng a lmted number of cluster representatves. Let s M R, =,,..., N S be the scene feature vector for the -th scene, where N S s the total number of scenes. Then S = { s, =,,, N S } s the set of all scene feature vectors. Let also S be the number of scenes to be selected and c, =,,..., S the feature vectors whch best represent those scenes. or each c, an nfluence set s formed whch contans all scene feature vectors s S whch are closer to c : Z = { s S : d( c < d( c j } ( j where d ( denotes the dstance between two vectors. A common choce for d ( s the Eucldean norm. In effect, the set of all Z defnes a partton of S nto clusters of smlar scenes whch are represented by the feature vectors c. Then the average dstorton, defned as = s = s Z D( c, c,..., c S d( c ( s a performance measure of the representaton of scene feature vectors by the cluster centers c. The optmal vectors $c are thus calculated by mnmzng D:,,..., = arg mn D( c, c,..., c ( S c, c,..., c M s R (3 Drect mnmzaton of the prevous equaton s a tedous task snce the unknown parameters are nvolved both n dstances d ( and nfluence zones. or ths reason, mnmzaton s performed n an teratve way usng the generalzed Lloyd or -means algorthm []. Startng from arbtrary ntal values c (, =,,..., S, the new centers are calculated through the followng equatons for n : S Z ( n = { s S : d( c ( n < d( c ( n j } (4 c ( n + = cent( Z ( n (5 where c (n denotes the -th center at the n-th teraton, and Z (n ts nfluence set. The center of Z (n s estmated by the functon cent( Z ( n = s (6 Z ( n s Z ( n where Z ( n denote the cardnalty of Z (n. The algorthm converges to the soluton ( c ˆ,,..., S after a small number of teratons. nally, the S most representatve scenes are etracted as the ones whose feature vectors are closest to c ˆ,,..., ˆ : ( c S sˆ = arg mn d (, =,,, s S 4. rame selecton mechansm After etractng the most representatve scene the net step s to select the most characterstc frames wthn each one of the selected scenes. Ths s acheved by mnmzng a correlaton crteron, so that the selected j S (7

3 frames are not smlar to each other. In partcular, the most characterstc frames are selected as the ones wth the mnmum correlaton among them. The selecton could also be performed usng the prevous optmzaton technque. However, that approach does not eplot the temporal relaton of feature vector whch s sgnfcant for the frame selecton procedure, as t s descrbed n the sequel. M Let us denote by f R, V = {,, N } the feature vector of the -th frame, where N L = s the total number of frames of a scene, and suppose that the most characterstc ones should be selected. The correlaton coeffcent of the feature vectors f, f j s defned as ρ j = Cj ( σ σ j (8 where C = ( f m T j ( f j m s the covarance of the N two vector m = = f / N s the average feature vector of the scene and σ = C s the varance of f. In order to defne a measure of correlaton between feature vector we frst defne the nde vector = (,, W V where W = {(,, V : < L < } (9 s the subset of V whch contans all sorted nde vectors. Thu each nde vector = (,, corresponds to a set of frame numbers. The correlaton measure of the feature vectors f, =,, s then defned as / R( = R(,, = (, ρ j ( = j= + Based on the above defnton t s clear that searchng for a set of mnmally correlated feature vectors s equvalent to searchng for an nde vector that mnmzes R (. Searchng s lmted n the subset W, snce nde vectors are used to construct sets of feature vectors. Therefore any permutatons of the elements of wll result n the same sets. The set of the least correlated feature vector correspondng to the most characterstc frame s thus represented by ˆ = ( ˆ,, ˆ = arg mn R( ( N W Unfortunately, the complety of an ehaustve search for the mnmum value of R ( s such that a drect mplementaton would be practcally unfeasble, snce the multdmensonal space W ncludes all possble sets (combnatons of frames. A dramatc reducton n complety s acheved, however, through logarthmc search, whch s performed n a way smlar to the search for block moton estmaton n vdeo sequences [3]. The algorthm s descrbed as follows: By lettng µ = L, we defne the ntal nde W as the element of W whch s closest to the mddle pont ~ = ( µ,, µ. It can be shown that µ /,, µ, µ +,, + / = ( µ ( f s even, and = ( µ /,, µ, µ, µ +,, µ + / s odd, where (3 f denotes nteger part. The neghborhood of s defned as N, S = { W : = + S p, p G } (4 ( where S = L = N / 4 s the ntal step sze and G = {,,}. Based on these defnton we calculate the net nde vector = arg mn R(. By lettng N (, S S = S /, we repeat the same steps: n = arg mn R(, S S / (5 N ( n, S n n = n for n =,, L (untl S n = and get the fnal result. ˆ = L W 9 N( N( 3 3 N( gure. Illustraton of the logarthmc search procedure for the smple -dmensonal case of =. The algorthm s based on the assumpton that frames whch are close to each other (n tme should have smlar properte and therefore ndces whch are close to each other (n W should have smlar correlaton measures. However, the technque performs equally well even n the case of random feature vector as shown by

4 eperments. The overall procedure for the very smple case of = s llustrated n gure. 3 shown n gure 3, where the representatve scene of each cluster s shown wth black border. It s clear that the four selected scenes gve a meanngful representaton of the content of the whole vdeo sequence. urthermore, t can be seen that each cluster contans scenes wth smlar properte such as number and complety of objects Cluster gure. The scenes of the test vdeo sequence. 5. Epermental results. Cluster The proposed algorthms were ntegrated nto a system that was tested usng several vdeo sequences from vdeo databases. The results obtaned from a TV news reportng sequence of total duraton.5 mnutes (375 frames are presented n the followng fgures. The sequence was frst parttoned n scenes and then the frame and scene feature vectors were etracted usng the aforementoned methodology. gure llustrates for each scene the frame whose feature vector s closest to the respectve scene feature vector. We have chosen to keep four representatve scenes ( S = 4, and thus four scene clusters are generated. Each cluster contans the scenes whose feature vectors were closest to respectve cluster center. The results of the scene selecton mechansm are 7 8 Cluster Cluster 4 gure 3. The four scene clusters generated by the scene selecton mechansm. The respectve selected (representatve scenes are shown wth black border.

5 The frame selecton mechansm was tested wth the last scene of cluster (scene. our frames were etracted out of a total of 55 frame usng logarthmc search wth = 4, and the representatve frames are shown n gure 4. Although a very small percentage of frames s retaned, one can perceve the content of scene by just eamnng the four selected frames. The correlaton measure R ( was also tested usng a large number of random nde vectors and ts probablty densty functon (hstogram s depcted n gure 5. Although our search algorthm requres about % of the computatonal tme of the random search, the located mnmum value of R( was ndeed very close to the actual mnmum, as shown by the vertcal dashed lne of gure Conclusons In ths paper, a mechansm for automatc etracton of the most representatve scenes and frames n vdeo databases was proposed. The scheme ncludes on the one hand, a mnmzaton of a dstorton crteron and one the other, an optmzaton technque for ndcatng the ndces of the most characterstc frames wthn each selected scene. To accomplsh the optmal etracton, we frst appled a color or moton segmentaton technque to vdeo frames n order to obtan an mage representaton more sutable for classfcaton. urthermore, to make the proposed archtecture more robust, a fuzzy representaton of the feature vectors was ntroduced. Epermental results ndcatng the good performance of the proposed scheme were provded by eamnng real TV programs. 7. References Probablty Densty gure 4. The four selected frames of scene Correlaton Measure R( gure 5. The probablty densty functon of the correlaton measure R(. The vertcal dashed lne shows the mnmum value located by the logarthmc search algorthm [] A. Doulam Y. Avrth N. Doulams and S. olla Indeng and Retreval of the Most Characterstc rames/scene Workshop on Image Analyss for Multmeda Interactve System pp. 5-, Louvan-la- Neuve, Belgum, 997. [] A. Gersho and R. M. Gray, Vector Quantzaton and Sgnal Compresson, Κluwer Academc Publsher 993. [3] H. Gharavand and M. Mll Block-Matchng Moton Estmaton Algorthms: New Result IEEE Trans. on Crc. & Systs. vol. 37, pp , 99. [4]. J. Han and A. H. Tewfk, Egen-Image Vdeo Segmentaton and Indeng, Proc. of IEEE ICIP, pp , Santa Barbara, USA, Oct [5] G. Iyerngar and A.B. Lppman, Vdeobook: An Eperment n Characterzaton of Vdeo, Proc. of IEEE ICIP, pp , Lausanne Swtzerland Sept [6] B. Meraldo, Automatc Indeng of TV New Workshop on Image Analyss for Multmeda Interactve System pp. 99-4, Louvan-la-Neuve, Βelgum, 997. [7] MPEG Vdeo Group, MPEG-4 Requrement ISO/IEC GTC/SC9/WG N679, Brstol MPEG Meetng, Aprl 997. [8] MPEG Vdeo Group, MPEG-7: Contet and Objectves (v.3, ISO/IEC GTC/SC9/WG N678, Brstol MPEG Meetng, Aprl 997. [9] Y. Ru, T.S. Huang and S. Mehrotra, Content-based Image Retreval wth Relevance eedback n Mar Proc. of IEEE Inter. Conf. on Image Processng (ICIP, vol., pp , Santa Barbara USA, October 997. [] Y. Ru, T. S. Huang and S.-. Chang, Dgtal Image/Vdeo Lbrary and MPEG-7: Standardzaton and Research Issue Proc. of IEEE Inter. Conf. on Acoustc Speech and Sgnal Processng (ICASSP, vol.6, pp , Seattle USA, May 998. [] J. R. Smth and S.-. Chang, Jont Adaptve Space and requent Bass Selecton, Proc. of IEEE Inter. Conf. on Image Processng (ICIP, vol. 3, pp. 7-75, Santa Barbara USA, October 997.