ABSTRACT 1. INTRODUCTION

Transcription

1 Robust SIFT-Based Descrptor for Vdeo Classfcaton Razyeh Salarfard, Mahshd lsadat Hossen, Mahmood Karman and Shohreh Kasae Department of Computer Engneerng, Sharf Unversty of Technology Tehran, Iran BSTRCT Volumnous amount of vdeos n today s world has made the subject of objectve (or sem-objectve) classfcaton of vdeos to be very popular. mong the varous descrptors used for vdeo classfcaton, SIFT and LIFT can lead to hghly accurate classfers. But, SIFT descrptor does not consder vdeo moton and LIFT s tme-consumng. In ths paper, a robust descrptor for sem-supervsed classfcaton based on vdeo content s proposed. It holds the benefts of LIFT and SIFT descrptors and overcomes ther shortcomngs to some extent. For extractng ths descrptor, the SIFT descrptor s frst used and the moton of the extracted eyponts are then employed to mprove the accuracy of the subsequent classfcaton stage. s SIFT descrptor s scale nvarant, the proposed method s also robust toward zoomng. lso, usng the global moton of eyponts n vdeos helps to neglect the local motons caused durng vdeo capturng by the cameraman. In comparson to other wors that consder the moton and moblty of vdeos, the proposed descrptor requres less computatons. Obtaned results on the TRECVIT 2006 dataset show that the proposed method acheves more accurate results n comparson wth SIFT n content-based vdeo classfcatons by about 15 percent. Keywords: Robust Vdeo Descrptor, SIFT, Vdeo Classfcaton, LIFT. 1. INTRODUCTION Wth recent technologcal advances and the huge boost n vdeo capturng devces, vdeo data has grown exponentally. Ths calls for a fast and accurate soluton n classfcaton for ndexng and retreval of the data. Humanbased vdeo classfcaton ncreases the tme and costs. Ths s where automatc vdeo data classfcaton comes nto vew for many researchers. For vdeo data classfcaton, approprate descrptors are extracted ntally and the vdeo class s determned based on these descrptors subsequently. The more the extracted descrptors show the dfferences among varous types of vdeos, the more accurate the classfcaton wll be. Snce a vdeo usually contans a sequence of frames, all extractable descrptors from ts frames can also be extracted. In most vdeo classfcaton methods only extractable descrptors from ts frames are utlzed ndependently and therefore the moton trajectory s gnored. Therefore, employng the moton trajectory can lead to a more accurate classfcaton. Vsual features of vdeo can be generally classfed to statc descrptors extracted from the man frame, extracted descrptors from the vdeo objects, and moton descrptors [1]. Statc descrptors from the man frame nvolve colorbased, texture-based, and shape-based descrptors [2, 3]. These statc descrptors only descrbe the vsual aspect of vdeo and are wea n descrbng other aspects such as objects and moton. Researchers use varous methods to extract objects from vdeo, for example, Vsser [4] uses the Kalman flter and Zhang [5] benefts spato-temporal ndependent component analyss (stic) and mult-scale analyss. The moton descrptor of vdeo s used n [6, 7, 8, 9]. Dfferent researches use varous nformaton of vdeo. For example, [6] uses moton vectors embedded n MPEG btstream as a vdeo descrptor, n [7] moton vector feld s used n order to extract a moton descrptor. lso, [8] extracts a vdeo moton descrptor based on local and global moton nformaton, and [9] benefts the spato-temporal dstrbuton wthn a shot for vdeo ndexng and retreval. long wth usng moton descrptor, [10] uses statc descrptors and SIFT descrptor to generate a new descrptor. However, usng varous features for vdeo retreval s tme consumng and can be non-applcable n tme-senstve tools. One of the most mportant descrptors n content-based vdeo classfcaton s the SIFT descrptor whch s a scale nvarant feature [11]. Image and vdeo classfcaton usng SIFT descrptor has a hgh accuracy. However, SIFT descrptor s appled on ndependent frames and thus gnores the moton n vdeos. In [12], a descrptor called local nvarant feature tracs (LIFT) s presented whch tracs the SIFT descrptor n consecutve frames of each shot. It consders the dynamsm of vdeo and consequently leads to better results. In order to equalze ts descrptor vector, the LIFT descrptor uses many complcated and tme-consumng calculatons whch are not approprate for onlne vdeo classfcaton. Usng the SIFT descrptor, n ths wor a descrptor smlar to LIFT s extracted whch tracs the SIFT

2 eyponts n consecutve frames and extracts the fnal descrptors, by mang these tracs equal to each other. The proposed descrptor s as accurate as LIFT whle t uses a very smple method to equalze the length of the descrptor vector whch results n reducng the tme complexty. 2. NOTTIONS ND FORMULTIONS Before explanng the proposed descrptor extracton algorthm, the used notatons and formulatons are descrbed n ths secton. bref descrpton of notatons s lsted n Table 1. Table 1: Notatons used n mplementaton. F th sampled frame Xj, j Coordnates of the j th eypont n the th frame S j SIFT descrptor of the j th pont n the th frame Half of square spatal wndow sde x The transmsson matrx to X curve n the K th trac y Transmsson matrx to curve n the K th trac Matrx contanng coeffcent of the X curve X Z Matrx contanng coeffcent of the curve Matrx contans ndex of ponts Each vdeo contans a number of shots and every shot conssts of some frames that are presented wthn a short tme nterval. In ths paper, F s the th frame selected out of every 25 successve frames. Each F has many eyponts that ther coordnates are denoted by Xj, j. The SIFT descrptor of each pont s denoted by S j. There are many tracs extracted for each shot. In order to extract a trac, a set of consecutve ponts are found where s half of a square spatal wndow sde for fndng the ponts. s shown n (1), the x matrx s calculated by usng the X coordnate of the K trac ponts. The th y matrx s calculated n the same way as well. The x and y matrces transform the K th trac to a twenteth degree polynomal curve, where X and contan coeffcents of the curve. x 1,1,...,1,1,1 x1, x2..., xn x1, x2..., x n 3. PROPOSED ROBUST VIDEO DESCRIPTOR In ths secton, the proposed robust descrptor for vdeo classfcaton s descrbed. ctually, the proposed descrptor s extracted to classfy the shots. s shown n Fgure 1, some frames are sampled from each shot. Then, the eyponts of these sampled frames are extracted. SIFT descrptor s then extracted for each eypont. mong the ponts n the neghborhood of these eyponts n the next frame, a pont whch has a more smlar SIFT descrptor s selected. By contnuaton of ths procedure a sequence of ponts wth smlar locaton and descrptor are generated. These tracs of ponts have dfferent lengths, thus by transmttng each trac to a constant degree polynomal curve and savng the curve coeffcent, a vector whch has a constant length s formed. Ths vector along wth the average of the SIFT descrptor form the sem-fnal descrptor. mong the extracted descrptors, by usng the bag-of-words method only some of them are selected to represent the others. These descrptors are the fnal descrptors whch are used n vdeo classfcaton stage. In the followng, the descrptor extracton and shot classfcaton are comprehensvely descrbed.

3 Fgure 1. Proposed robust vdeo descrptor. 3.1 Frame Samplng In 25 frames per second vdeos, assumng that the probablty of changng the vdeo content n less than a second s very low, here just one frame per 25 frames s selected to represent t. Thus, as shown n (2), a set of consecutve sampled frames represent a shot by Sh {F,F,...,F,...,F }. (2) SIFT Descrptor In order to extract the SIFT descrptors, eyponts are frst extracted from each frame. To fnd eyponts, among all exstng pxels n the frame, those whch are mmutable toward scale and rotaton changes n all scales are consdered. s shown n Fgure 2, for each eypont locatng n the mddle of small blocs the surroundng pxels are dvded nto 4 parts. Then, by usng a Gaussan functon, shown by a crcle n the fgure, a weght s assgned to each vector of these 4 parts. Fnally, a hstogram wth 8 dfferent drectons s formed for exstng vectors n each part [13]. fter SIFT extracton for each sampled frame F, a set of eyponts s obtaned where each has the locaton Xj, j and SIFT descrptor S j. j n Fgure 2. SIFT descrptor extracton [13]. 3.3 Robust Vdeo Descrptor The detals of each stage of the proposed descrptor are gven next Moton Estmaton In order to extract a trac, for each eypont n a frame, the smlar one n the next frame s found. smlar eypont has the followng condtons X X (3) j (j1)

4 (4) j (j1) S mn( S S ). (5) j S(j 1) j ( j 1) Equatons (3) and (4) denote that a smlar eypont should be located n a square spatal wndow of sze 2 and equaton (5) denotes that ths eypont has the most smlar SIFT descrptor among other eyponts n the mentoned neghborhood. If such a pont s found, t wll be added to the trac. Ths search wll contnue untl the last pont of the trac cannot fnd a smlar pont n the next frame. For each trac, the average of SIFT descrptors of the ponts wll also be saved. Snce there s a number of eyponts n each frame, and the last pont of each trac can be located anywhere n the next frames, many tracs wth dfferent lengths wll be generated Curve Estmaton s shown n the prevous subsecton, the lengths of the tracs are dfferent. Thus, n order to transmt these tracs to feature vectors, a vector wth a constant length should be extracted from each trac. For each trac, the sequence of X and elements n the tme dmenson are mapped to a curve. Thus, n order to extract the mentoned descrptors, each of these curves s transmtted to a twenteth degree polynomal curve. The X and matrces that denote the coeffcent of curves are calculated as X Z (6) y x Z. (7) s descrbed n Table 1, Z s a matrx that contans the ndex of traced eyponts and and are matrces that transmt X and to a polynomal curve. The coeffcent of these two polynomal curves along wth the average of the SIFT descrptor of all ponts n a trac, formng a 168 element vector, construct the sem-fnal descrptors. Therefore, wth a very smple method and a few calculatons a descrptor s extracted from the tracs of a shot where t represents the moton feature of vdeo well Bag-of-Words There are a number of extracted 168-element vectors for each shot. In order to classfy a shot, a constant number of vectors should be selected to represent t. The bag-of-words method s used to choose a specfc number of vectors among all nput vectors. In ths method, all these 168-element vectors are transmtted to a 168-element space. In ths space, a clusterng method s appled. It can be performed by clusterng method. The K-means clusterng s appled n ths paper. It groups the vectors nto K clusters and selects a vector from each cluster to represent that cluster. Thus, K number of 168-element vectors are selected to represent the shot. Now, for each shot, a vector wth an equal number of elements s extracted as ts descrptor. 3.4 Shot Classfcaton To classfy shots, a supervsed classfcaton has been used. To do so, we have used the 10 fold cross-valdaton method whch uses Support Vector Machne wth RBF ernel as ts classfer. 4. EXPERIMENTL RESULTS In ths secton, the proposed descrptor s compared wth SIFT n precson and complexty aspects. In order to evaluate the proposed method, the TRECVIT 2006 dataset s used. The proposed method s mplemented usng C programmng language. To run and test the method, the program has been run on a personal laptop wth INTEL CORE 2 DUO processor wth the process speed of GHz. One of the most prevalent crtera of vdeo classfcaton assessment s precson. Thus, the crtera of evaluatng the proposed descrptor s the average of the vdeo classfcaton precson n each label. Fgure 3 descrbes the effect of σ on the descrptor extracton precson. s ncreases, the number of adjacent eyponts wll be rased and consequently the probablty of fndng them wll be ncreased as well. Thus, as shown n Fgure 3, growth wll result n ncrement of average precson of classfcaton. But, by ncreasng for more than x y

5 10 pxels, the number of rrelevant eyponts added to the trac wll be ncreased and thus the average precson wll be decreased. Therefore, equal to 10 s chosen as an expermental setup. Fgure 4 shows the average precson of classfcaton for the proposed and SIFT descrptors n varous contents. ccordng to ths fgure, for vdeos havng md and hgh moton the precson of the proposed method s about 15 percent hgher than that of SIFT and n vdeos wth low moton (le arplane and exploson) the precsons are the same. In vdeos wth no moton (such as buldng exteror, waterscape, and smoe) the precson of the proposed s about 10 percent less than SIFT. The analyss on the descrptor extracton executon tme done on 2000 shots shows that the SIFT executon tme s 200 mllseconds and that for our proposed descrptor s 215 mllseconds and ths tme ncrease n executon tme s neglgble. Fgure 3. Effect of on average precson. Fgure 4. verage precson of proposed and SIFT descrptors. 5. CONCLUSION In ths paper a robust moton-based descrptor s proposed by usng SIFT. It s smple and fast. It uses the moton trajectory n vdeos to mprove the accuracy of the subsequent classfcaton stage. The expermental results show that the proposed method s effcent for content-based vdeo classfcaton wth neglgble tme overload. In order to have an effectve classfcaton n all vdeo contents, SIFT can be selected as descrptor for non-moton vdeo contents, whch s our future wor. REFERENCES [1] Hu, Wemng, et al. " survey on vsual content-based vdeo ndexng and retreval." Systems, Man, and Cybernetcs, Part C: pplcatons and Revews, IEEE Transactons on 41.6 (2011): [2] an, Rong, and lexander G. Hauptmann. " revew of text and mage retreval approaches for broadcast news vdeo." Informaton Retreval (2007): [3] mr, rnon, et al. "IBM research TRECVID-2003 vdeo retreval system." NIST TRECVID-2003 (2003). [4] R. Vsser, N. Sebe, and E. M. Baer, Object recognton for vdeo retreval, nproc. Int. Conf. Image Vdeo Retreval, London, U.K., Jul. 2002, pp [5] Zhang X P, Chen Z. n automated vdeo object extracton system based on spatotemporal ndependent component analyss and multscale segmentaton. EURSIP Journal on ppled Sgnal Processng, 2006, 2006: 184 [6] Dao, Mnh-Son, F. G. B. DeNatale, and ndrea Massa. "Vdeo retreval usng vdeo object-trajectory and edge potental functon." Intellgent Multmeda, Vdeo and Speech Processng, Proceedngs of 2004 Internatonal Symposum on. IEEE, [7]. Su, Chh-Wen, et al. Moton Flow-Based Vdeo Retreval. Multmeda, IEEE Transactons on 9.6 (2007): [8] Ma, u-fe, and Hong-Jang Zhang. "Moton texture: a new moton based vdeo representaton." Pattern Recognton, Proceedngs. 16th Internatonal Conference on. Vol. 2. IEEE, [9] Fablet, Ronan, Patrc Bouthemy, and Patrc Pérez. "Nonparametrc moton characterzaton usng causal probablstc models for vdeo ndexng and retreval." Image Processng, IEEE Transactons on 11.4 (2002): [10] Basharat, rslan, un Zha, and Mubara Shah. "Content based vdeo matchng usng spatotemporal volumes." Computer Vson and Image Understandng (2008): [11] Lowe, Davd G. "Object recognton from local scale-nvarant features."computer vson, The proceedngs of the seventh IEEE nternatonal conference on. Vol. 2. Ieee, [12] Mezars, Vasleos, nastasos Dmou, and Ioanns Kompatsars. "Local nvarant feature tracs for hgh-level vdeo feature extracton." nalyss, Retreval and Delvery of Multmeda Content. Sprnger New or, [13] Davd G. Lowe, Dstnctve Image Features from Scale-Invarant Keyponts, INTERNTIONL JOURNL OF COMPUTER VISION, vol. 60, n. 2, pp , 2004.