Content Based Retrieval

Transcription

1 2010 2nd Internatonal Conference on Sgnal rocessng Systems (ICSS) A Novel Commercal Break Detecton and Automatc Annotaton of TV rograms for Abstract In ths paper, we present a novel approach for automatc annotaton and content based vdeo retreval by makng use of the features extracted durng the process of detectng commercal boundares n a recorded Televson (TV) program. In our approach, commercal boundares are prmarly usng audo and the boundares are enhanced and valdated usng logo (program name) n the vdeo doman. Detected logo (program name) at the commercal boundares s used for automatc annotaton of recorded vdeo whch helps n fast content based vdeo retreval. The performance and valdty of our approach s demonstrated usng the vdeos recorded from dfferent Indan Televson broadcasts. Index Terms Audo vsual features, Automatc annotaton, Commercal detecton, Content based vdeo retreval, Correlaton classfer. I. INTRODUCTION The Televson (TV) remans and always has been one of the most powerful medum of communcaton snce ts ncepton. Today we are n the next generaton of televson where, the addtonal hardware and software have completely changed the way the televson s watched. Set-top boxes equpped wth personal vdeo recorders (VR) are now farly establshed and fast growng part of the TV landscape. In ths context, audovsual analyss tools that help the user to manage the huge amount of data are very mportant to ntroduce the novel recordng devces n the hghly compettve market. Among other analyss tools, detecton of TV advertsements s a topc wth many practcal applcatons. or nstance, from the pont of vew of a TV end-user, t could be useful to avod commercals n personal recordngs. Also of great utlty s to approprately ndex and retreve the large number of recorded vdeos automatcally. Exstng commercal detecton approaches can be generally dvded nto two categores: feature-based and matchng-based approaches. Whle the feature-based approaches use some nherent characterstcs of TV commercals to dstngush commercals and other types of vdeos, the matchng-based methods attempt to dentfy commercals by searchng a database that contans known commercals. The challenges faced by both approaches are the same: how to accurately Content Based Retreval Venkatesh N 1, Grsh Chandra 2 and Rajeev B 3 Embedded Systems Innovaton Lab, Tata Consultancy Servces, Bangalore, Inda E-mal: 1 venk.n@tcs.com, 2 m.gchandra@tcs.com, 3 rajeev.b@tcs.com /$26.00 C 2010 IEEE V2-577 detect commercal breaks, each of whch conssts of a seres of commercals; and how to automatcally perform fast commercal recognton n real tme. Smlarly, the vdeo retreval approaches can also be categorzed nto two dfferent approaches [7]. rst category focuses manly on vsual features, such as color hstograms, shapes, textures, or moton, whch characterze the low-level vsual content. Although these approaches use automatcally extracted features representng the vdeo content, they do not provde semantcs that descrbe hgh-level vdeo nformaton (concepts), whch s much more approprate for users when retrevng vdeo segments. The second one focuses on annotaton based approaches [8], whch makes use of keyword annotaton or text attrbute to represent hgh level concepts of the vdeo content. erformance of ths approach s affected by several factors, ncludng the fact that the search process depends solely on the predefned attrbuted nformaton. urthermore, manual annotaton process s very tedous and tme consumng. Commercal break detecton ([1][2]) have based ther strateges n studyng the relaton between audo slences and black frames as an ndcator of commercals boundares. The analyss s performed n ether compressed [1] or uncompressed [2] doman. In [3], specfc country regulatons about commercals broadcast s used as a further clue. Another nterestng approach s presented n [4], where the overlad text trajectores are used to detect commercal breaks. The dea here s that overlad text (f any) usually remans more stable durng the program tme than n the case of commercals. In [5], Hdden arkov odel (H) based commercal detecton s presented wth two dfferent observatons taken for each vdeo shot: logo presence and shot duraton. In most of the system, black frame [6] s used as the crucal nformaton for detectng the commercal n vdeo processng. But, n Indan TV statons black frames are absent and our methodology works wthout usng black frames as vdeo clues. Our smple and accurate approach for commercal detecton uses audovsual features. Intally, we fnd the commercal boundares usng a combnaton of dfferent audo features and the confdence level of the boundary wll be further enhanced and valdated usng logo (program name) vdeo frames. Ths detecton s carred out around the commercal boundares by audo. Image matchng of logo frame appearng at the start and end of the commercal logo s performed usng smple pxel ntensty based matchng.

2 2010 2nd Internatonal Conference on Sgnal rocessng Systems (ICSS) The proposed automatc annotaton and retreval system tres to fll the gap between the low-level meda features and hghlevel concepts for yeldng better performance usng computatonally smple procedures. The over all scheme s depcted n the block dagram shown n gure 1. In the drecton of provdng the requste detals of the proposed method and the performance analyss results, the pa per s organzed as follows. After presentng the reprocessng and feature extracton n Secton 2, the commercal detecton scheme s elaborated n Secton 3. Detals about automatc annotaton and retreval are provded n Secton 4 followed by results and dscusson n secton 5 and fnally Secton 6 captures the conclusons of the work, also spellng out the future drectons. Recorded TV rogram Vdeo data Audo data re-processng and eature Extracton The short term energy s computed per frame as the sum of squares of the sgnal samples normalzed by the frame length as gven n Equaton 1. 1 N = 1 2 E x ( (1) N n= 0 where, = 1,2,, In Equaton 1, N s the frame length (.e., the number of samples n the frame). The requste threshold s heurstcally fxed as T = 0.5 max (2) ( ) E The ER s computed usng the followng expresson 1 ER = x = 1 where, ( ) ( ) E (3) s the bnary-valued functon, defned as Vdeo clue based detecton Automatc annotaton Query mage Vdeo Database Retreval process Audo based detecton Retreved vdeos g. 1 System overvew II. RE-ROCESSING AND EATURE EXTRACTION A. re-rocessng The audo sgnal s extracted from the recorded TV program and s segmented nto audo frames wth 50% overlap and 23 msec duraton. B. Audo eature Extracton Energy peak rate (ER): Energy peak rate s defned as rato of the total number of short-term energy values exceedng the threshold T to the total number of frames () n Step 1: the evaluaton tme nterval. Based on the elaborate expermentaton, the evaluaton nterval s chosen as 5 seconds. V f x > T ( x) = (4) 0 else III. COERCIAL DETECTION SCHEE A. Audo feature based detecton Havng extracted audo features, we use heurstc rules found by elaborate expermentaton for commercal detecton. If ER s greater than the threshold θ then we declare that partcular frame les n the commercal break segment and hence we wll be able to detect the approxmate commercal boundares. urther, the exact commercal boundares wll be by the proposed vdeo processng algorthm. B. Vdeo feature based detecton After detectng the approxmate commercal boundares usng audo features, we valdate and further look for exact commercal boundares usng the logo (program name) vdeo frames matchng at the start and the end of commercals. The vdeo processng s carred out to detect logo (program name) around the boundares by audo. gure 2 shows an example of logo (program name) appearng n one of the TV program vdeo. The ratonale for ths two-step approach s to reduce the search space for logo detecton durng the vdeo processng step, hence sgnfcantly reducng the computatonal tme. Intally we convert color mage to bnary mage and match the mages by smple pxel ntensty matchng as explaned below: =0 where s the number of pxels matched.

3 2010 2nd Internatonal Conference on Sgnal rocessng Systems (ICSS) Step 2: ( ( x, y) = x y f R 1 & T ( x, y) = 1) then = +1 where R s the reference template mage at pont (x,y) and x y x, y T (, ) s test mage at pont ( ) Step 3: = 100 WR where, W R s the number of whte pxels n reference template mage R. Step 4: If s greater than 80% then the two mages can be consdered as smlar (agan based on expermental studes). g. 2 Logo (program name) frame The chosen audo and vdeo features descrbed n ths paper are more robust n dfferent scenaros and computatonally smple, thus are sutable for embedded applcatons, compared to the features we explored n our earler works [9]. IV. ECHANIS OR AUTOATIC ANNOTATION AND RETRIEVAL A. Automatc annotaton Once the commercal boundares have been the next step s to remove the commercal part by retanng only the recorded TV program and durng ths process logo (program name) frame n vdeo based boundary detecton process s concatenated at the begnnng of the Recorded TV program vdeo (wthout commercal) whch s not gong to affect the vsual dsplay because we are nsertng only one key frame at the begnnng and normally vdeo qualty wll be 25 frames per second. Key frame wll also conssts of some nformaton lke recordng date, tme and length of the vdeo whch provdes addtonal nformaton durng the retreval process for short lstng the closest matchng vdeos to the query. B. Content based Vdeo retreval There wll be a set of query mages whch s nothng but a logo (program name) representng the recorded TV programs among whch the user would choose hs nterested program s correspondng logo (program name) whch serves as a query/test mage. It s to be noted that these set of query mages can be ncorporated nto the modern day set-top box and the assocated remote can be provded wth a dedcated key to facltate the dsplay of query mages. Later, query mage wll be compared wth all vdeos frst frame and the best match s obtaned usng correlaton classfer as explaned below. Correlaton classfer s smple and computatonally fast and hence t has been used to fnd the best match between the query mage and the set of mages n the database. Query mage (A) and the extracted key frame (B) from the vdeo are converted to bnary mages and then correlaton coeffcent s computed between these two mages usng Equaton 5. r = m n ( A ( m, A )( B( m, B) 2 2 ( A ( m, A ) ( B( m, B) m n m n where =1 N, N s number of stored key frames n the database. A s the key frame belongng to th class of a database A s the mean of A mage B s the query/test mage B s the mean of an mage Largest r value ndcates us the closest match of key frame to the query mage. V. RESULTS AND DISCUSSION In order to examne the proposed methodology and to carry out the relevant expermentaton we have recorded eleven vdeos from seven dfferent Indan TV statons. Vdeos are captured at 25 frames per second wth resoluton and the audo was extracted from these vdeos and stored n uncompressed wav format at khz wth 16 bts per sample for further processng. Commercal detals for the recorded data are provded n table 1. We have performed our experments usng ATLAB programmng n wndows operatng system. Vdeos wth same class label ndcate that they are dfferent epsodes of same TV program whch s helpful durng retreval process. rom table 2 we observe that usng our smple vdeo clues of matchng logo vdeo frames at commercal boundares for refnng the boundares by audo we are able to (5) V2-579

4 2010 2nd Internatonal Conference on Sgnal rocessng Systems (ICSS) ncrease the confdence level of the commercal boundary detecton and able to fnd the exact commercal boundares. One mportant observaton from table 2 and table 3 s that false by audo processng s elmnated by the nformaton obtaned through vdeo processng step as the logo wll not be present at the start and the end of falsely commercal boundares usng audo. TABLE 1 DETAILS O DATA Vdeo Vdeo Length Commercal break Class (mn:sec) duraton (mn:sec) label 1 17:30 ) 12:25-17: :00 ) 08:10-11:45 1 ) 20:10-21: :09 ) 00:40-4:12, :19 ) 04:52-:52 2 ) 17:20-22: :17 ) 03:0-06:45 2 ) 16:15-20: :54 ) 06:42-13: :26 ) 02:08-05:25 4 ) 11:30-15: :41 ) 03:00-06:35 5 ) 14:20-17: :55 ) 07:40-12: :14 ) 03:4609:10 7 ) 14:45-19:25 ) 27:53-31: :24 ) 07:3011:35 ) 17:28-21:25 ) 30:10-33:24 7 Vdeo TABLE 2 RESULTS USING ONLY AUDIO EATURE Total No of false mssed event true The audovsual based commercal boundary detecton results are as shown n the table 3. We can have a desred trade off between number of falsely and the number of mssed by varyng a threshold n audo based feature gven by equaton 2. Vdeo TABLE 3 RESULTS USING BOTH AUDIO AND VIDEO EATURES Total No of false mssed event true Once the commercal detecton process s completed automatcally the vdeo wll be annotated usng key frame wth some nformaton appended at the begnnng of the vdeo. Currently expermentaton has been carred out for 7 classes (seven types of recorded TV programs). The content based vdeo retreval s expermented for all seven classes (query mages) usng correlaton classfer to match between query and reference mages to retreve the closest matchng automatcally annotated vdeos. Retreval performance results are comparatvely fast and accurate, wth no error n lmted data set consdered. VI. CONCLUSION AND UTURE WORK Ths paper descrbes the Commercal detecton for recorded TV program usng audovsual features. urther, automatc ndexng and content based retreval algorthm whch makes use of the nformaton extracted durng commercal detecton s proposed. Expermental result shows that the proposed system can be used for real tme embedded system (TV set-top box), as t s tme effcent, smple and amenable for easy mplementaton. The system provdes the user wth complete soluton of recordng TV program wthout advertsement and more mportantly, database management of the recorded TV programs s acheved through automatc annotaton and fast content based vdeo retreval. Our future work wll be focused on carryng out the experment on large vdeo archves. Expermentaton wll also be carred out to generalse the proposed scheme for dfferent knds of TV programs, lke news, serals, realty shows etc, encompassng both Indan and Internatonal TV broadcasts. V2-580

5 2010 2nd Internatonal Conference on Sgnal rocessng Systems (ICSS) REERENCES [1] D. A. Sadler et al., Automatc TV advertsement detecton from mpeg btstream, Journal of the att. Rec. Socety, vol. 35, no. 12, pp. 2 15, Dec [2] Y. L and C.-C. Jay Kuo, Detectng commercal breaks n real TV program based on audovsual nformaton, n SIE roc. on IS, vol 421, Nov [3] R. Lenhart, C. Kuhmnch, and W. Effelsberg, On the detecton and recognton of televson commercals, n roc. IEEE Conf. on CS, Otawa, Canada, [4] N. Dmtrova, ultmeda content analyss: the next wave, n roc. of the 2nd CIVR, Illnos, USA, Aug [5] Alberto Albol, ar ıa Jos e Ch. ull`a, Antono Albol, Lus Torres, Commercals Detecton usng Hs,2004. [6] Alberto Albol ar, aría José, Ch. ullà, Antono Albol, Detecton of TV Commercals In roceedngs of the Internatonal Conference on Acoustcs, Speech and Sgnal rocessng,2004. C. J. Kaufman, Rocky ountan Research Lab., Boulder, CO, prvate communcaton, ay [7] C A. Yoshtaka, T. Ichkawa, A Survey on Content-Based Retreval for ultmeda Databases, IEEE Transactons on Knowledge and Data Engneerng, 11(1), 1999, pp [8].S. Kankanhall, et al, Vdeo odelng Usng Strata-Based Annotaton, IEEE ultmeda, 7(1), 2000, pp [9] Venkatesh N, Rajeev B, Grsh Chandra, Novel TV Commercal Detecton n Cookery rogram Vdeos, roceedngs of the World Congress on Engneerng and Computer Scence 2009 Vol II, WCECS 2009, October 20-22, 2009, San rancsco, USA. V2-581