Video Content Description Using Fuzzy Spatio-Temporal Relations

Transcription

1 Proceedings of he 4s Hawaii Inernaional Conference on Sysem Sciences Video Conen Descripion Using Fuzzy Spaio-Temporal Relaions rchana M. Rajurkar *, R.C. Joshi and Sananu Chaudhary 3 Dep of Compuer Science, MGM s College of Engg. & Tech., Nanded (India) Dep. of Elecronics & Compuer Engineering Indian Insiue of Technology Roorkee Roorkee (India) archana_rajurkar@yahoo.com bsrac One of he imporan aspecs in digial video applicaions is spaial and emporal characerisic. very lile progress has been achieved on spaioemporal modeling of video daa. In his paper, we presen a fuzzy spaio-emporal model for video conen descripion ha suppors spaio-emporal queries. Fuzzy definiions and membership funcions of emporal relaions are given and second order fuzzy emporal relaions are proposed. model represenaion for he specificaion of fuzzy spaio-emporal relaionships among objecs in a video sequence is presened. Such represenaions are used for conen-based rerieval processing. We also repor he experimenal resuls performed on video sequences from MPEG-7 video conen archive. KEY WORDS Spaio-emporal modeling, spaio-emporal relaions, conen-based rerieval, video daabase, mulimedia daabase.. Inroducion The pas few years have seen explosive growh in mulimedia daa such as images, video and audio. The new possibiliies offered by he informaion highways have made a large amoun of video daa publicly available. Therefore, he need for a sysem ha provides he abiliy o sore and rerieve video in a way ha allows flexible and efficien search based on semanic conen is significan. Research on video rerieval based on semanic conens has no been fully explored ye. Mos of he exising video rerieval echniques search video based on visual feaures (e.g. color, exure and shape) [] [] [3]. video sequence is firs segmened ino shos and each sho is hen represened in erms of number of key frames [4] [5] [6]. Then visual feaures are used for rerieval. Key frame based mehods are resriced o global image feaures and do no ake ino consideraion objecs and evens in he video and choosing he keyframes is sill a challenging problem. Furhermore, in hese approaches he emporal naure of video is 3 Dep. of Elecrical Engineering Indian Insiue of Technology Delhi New Delhi- (India) sananuc@cse.iid.erne.in negleced. Very few sysems have addressed he issue of objec-based video rerieval [7] [8] and spaial modeling of video daa ha involve emporal informaion [9][0][][]. Therefore, here is grea need of spaioemporal model for video conen characerizaion ha enables users o query video conen using high-level conceps such as objecs and evens and heir spaioemporal relaions. Mos of he previous spaio-emporal models do no deal wih exracion of spaio-emporal relaions raher hey use precise definiions of spaial relaions [3] using eiher angle measuremens or minimum bounding recangles (MR) and emporal relaions [4]. The human abiliy o quickly deermine he spaial and emporal relaions beween any wo objecs is well known bu i has urned ou o be quie difficul o define precisely. Spaial relaions such as LEFT, OVE and ohers defy precise definiions, and seem o be bes modeled by fuzzy ses [5]. Furhermore, errors may occur in even-deecion, segmenaion and objec deecion due o use of precise definiions of spaial and emporal relaions and noise in he video daa. This necessiaes use of fuzzy spaial and emporal relaions for video conen descripion. In his paper we presen a new approach o video conen descripion. fuzzy spaio-emporal model is proposed ha is based on fuzzy direcional and opological relaions and fuzzy emporal relaions beween video objecs. We use he linguisic definiions of spaial relaions using hisogram of forces presened in [6]. Fuzzy definiions and membership funcions of emporal relaions are presened and he second order fuzzy emporal relaions ha are more informaive are proposed. video represenaion scheme is presened using he proposed fuzzy spaio-emporal model. The res of he paper is organized as follows. Secion describes moion segmenaion process used. The proposed fuzzy spaio-emporal model is presened in secion 3. Experimenal resuls are given in secion 4 and conclusions are presened in secion 5.. Moion Segmenaion /08 $ IEEE

2 Proceedings of he 4s Hawaii Inernaional Conference on Sysem Sciences Moving objecs are deeced in he video sequence using moion segmenaion mehods. In our model, moving objecs in every frame are deeced in he video sequence using moion segmenaion mehod described in [8]. This moion segmenaion echnique is bes suied for video sequences conaining objec moion wihin an oherwise saic scene, such as in surveillance and scene monioring applicaions. 3. Fuzzy Spaio-Temporal Model In video daa he spaial characerisics may change coninuously and herefore, any modeling echnique should be capable of capuring spaial feaures in a dynamic fashion. This combined characerizaion is referred as spaio-emporal modeling of video []. In he proposed model he descripion of a scene is expressed by he muual spaial relaionships beween every wo objecs and he emporal change called emporal specificaion of spaial relaionship of objecs wihin a scene measured by he number of frames in which a paricular objecs spaial relaionship holds. For each objec O i of a frame, is fuzzy spaial and emporal relaionships,, wih every oher objec, OST O j, in he same frame are represened using he relaionships beween he objecs and recorded in a vecor of size n(n-) if here are n objecs in a frame. The fuzzy spaio-emporal relaionship beween he wo objecs is defined as he funcion: OST (O i, O j ) = (S, T ) () where, S and T are he spaial and emporal relaionships, respecively, beween he objecs O i and O j in he frame. 3. Spaial relaionships The relaive posiions beween wo objecs O i and O j can be capured as a fuzzy spaial relaion using hisogram of forces (F-Hisograms) [5]. Spaial relaions such as lef of, above and ohers defy precise definiions, and seem o be bes modeled by fuzzy ses. Masakis and Wending [5] have inroduced he noaion of he F-hisogram (hisogram of forces), which generalizes and supersedes ha of he hisogram of angles. In [6] definiions of fuzzy direcional relaions and opological relaions using F-hisograms are presened. We make use of hese definiions of fuzzy spaial relaions as perceived by humans for capuring relaive posiion of a D objec O i wih regards o anoher objec O j. Each pair of objecs in every frame in he video sequence is represened by relaive posiion hisograms and hen he degree of ruh of a proposiion is in direcion θ of is compued. The degree of ruh is a real number generaed greaer han or equal o 0 (proposiion compleely false) and less han or equal o (proposiion compleely rue). The spaial relaionships, S, beween wo objecs are defined as follows: S = (R, O i, O j ) () The R represens he degree of ruh of a proposiion is in direcion θ of 3. Temporal relaionships llen [4] inroduced emporal inerval algebra for represening and reasoning abou emporal relaions beween evens represened as inervals. In many siuaions, precise descripion of emporal relaions may no be suied for he spaio-emporal models and in pracice many imes video daa conains noise. This may lead o errors in segmenaion, even-deecion and objec deecion. We presen Fuzzy definiions of emporal relaions in his secion ha can ake care of errors in even-deecion, segmenaion and objec deecion and allows flexible descripion of video scene and mach. In order o minimize he errors occurred in segmenaion, even deecion and objec deecion we presen fuzzy definiions and membership funcions of emporal relaions. Few definiions are shown in able. The emporal relaion T beween he spaial relaionships, S, of objecs O i and O j can be described in wo levels. In he firs level, he emporal inerval f for which a fuzzy spaial relaionship beween he wo objecs is valid is deermined. In he second level he second order fuzzy emporal relaionships beween he wo spaial relaions are described. The advanage of he second order fuzzy emporal relaions is ha hey are more informaive and provide global descripion of a sequence. Few of he proposed second order fuzzy emporal relaions using fuzzy spaial relaions are shown in he Table. Graphical illusraion of wo second order fuzzy emporal relaions are presened in figure. The emporal relaionship, T, beween he emporal inervals of wo spaial relaionships, S and S is defined as follows: T = ( S f ) and T = ( S FTOP S f f ) (3) where, <FTOP> is emporal operaor represening fuzzy emporal relaionship beween wo inervals and f is he emporal inerval for which he fuzzy spaial relaionship S is valid. 3.3 Represenaion of video sequence using fuzzy spaio-emporal model In his secion, we consider he represenaion of a video sequence using he proposed fuzzy spaio-emporal model. The video daabase V db conains video sequences S, S.S n as follows:

3 Proceedings of he 4s Hawaii Inernaional Conference on Sysem Sciences V = db S S S 3... S video sequence S is an ordered se of n frames, denoed S = { 0,,, n ), where n is he frame number n in he sequence. For each frame in he video sequence S he moving objecs are deeced, labeled using moion segmenaion mehod described in Secion. Then he aribues of labeled objecs are derived. For each objec O i of a frame, is fuzzy spaial and emporal relaionship OST, wih every oher objec, O j, is represened using he proposed fuzzy spaioemporal model as discussed in Secion 3. OST ( O i j) = ( S,T) Where S and T are defined by Eq. () and (3) respecively. For each frame fuzzy spaio-emporal relaionship beween all objec pairs (e.g. suppose here are four objecs in he frame ) is represened as follows: O ST = (O ST ST3 ST4 ST3 ST4 ST34 To capure he dynamic change in he fuzzy spaial relaionship of wo objecs O i and O j over he video scene lengh l he emporal inerval represening he number of frames in which he corresponding spaial relaion is valid is deermined. The emporal inerval of a spaial relaionship is found from he frame of he iniial appearing of a paricular spaial relaionship, which represen he beginning of he emporal inerval in which ha spaial relaionship is valid. Then he frame of he iniial appearance of he firs differen relaionship is deermined. Thus he duraion of he emporal inerval in which a paricular spaial relaion is valid is -. Like his he emporal inervals for all spaial relaions beween wo objecs over a sequence are compued o decide he maximum number of frames for which a paricular spaial relaionship beween wo objecs is valid. Depending on he query, he relevance membership funcion, which is he raio of oal number of frames in he sequence o he maximum number of frames for which he spaial relaionship in he query is valid for every sequence in he daabase, is compued. The deail algorihm for represenaion of video sequences in he video daabase using he proposed fuzzy spaio-emporal model is presened in figure. We used MPEG-7 video conen eri_od_a.mpg o illusrae he proposed video sequence represenaion scheme. Figure 3 shows few frames in he video sequence eri_od_a.mpg. Moving objecs in he sequence are deeced and labeled as person, person, Person3 and Person4 using he moion segmenaion mehod described in secion and heir aribues are derived. Iniial appearing of wo objecs Person and n ) person is found in frame 73 and heir spaial relaion is Person in whie shir is lef of Person in black shir shown in figure 3. The emporal inerval in which his spaial relaionship is valid sars a frame 73 and ends a frame 90. The same spaial relaionship holds for anoher emporal inervals from frame 36 o frame 343 and from he frame 79 o frame 900. The fuzzy spaio emporal relaionship is given by OST ( O73 73) = ( S73,T73) S 73 = (lef of, Person, Person) T = 354 Like his all he video sequences in he daabase are represened by fuzzy spaio-emporal relaionships among he objecs in he sequence. 4. Experimenal Resuls We used MPEG-7 video conen o evaluae he effeciveness of our approach. Video sequences in he daabase are represened using he proposed fuzzy spaio-emporal model as described in secion 3.3. Now, consider a query Find video sequences in which Person in whie shir is lef of Person in black shir. The query resuls are shown in Table 3. The emporal inervals represening he number of frames for which a spaial relaionship lef of is valid for every sequence and he relevance membership funcion, which is he raio of number of frames in a sequence and number of frames for which he spaial relaion in he query is valid for a sequence are given in he Table 3. Low value of RMF funcion corresponds o he more number of frames while high value of RMF corresponds o he less number of frames ha saisfy he spaial relaionship lef in he sequence. The sequences in he daabase are ranked hen depending he value of RMF. The video sequences having low RMF values are rerieved as similar o ha of he spaio-emporal relaion described in he query. For he query in quesion he mos similar sequence in he video daabase is eri_od_a.mpg, which has maximum number of frames 73 for which spaial relaionship Person in whie shir is lef of Person in black shir is valid. The daabase video sequences are ranked as,3, depending on he RMF value for he query in quesion. 5. Conclusions We have presened a fuzzy spaio-emporal model for video conen descripion ha suppors spaio-emporal queries. The proposed model is based on fuzzy direcional and opological relaions and fuzzy emporal relaion. The problems wih he use of precise spaioemporal relaions were highlighed. In order o minimize hese errors fuzzy definiions of emporal relaions are proposed. In addiion, he second order 3

4 Proceedings of he 4s Hawaii Inernaional Conference on Sysem Sciences emporal relaions ha are more informaive and provide global informaion abou he sequence are presened. The proposed model provides a mechanism ha represens he fuzzy spaio-emporal relaionships among he objecs in video sequences in he daabase and ranks he daabase sequences based on he query for effecive conen-based rerieval. We repored he resuls of our experimen on a sample video from MPEG-7 daa se. References []. H.J. Zhang, J. Wu, D. Zhong, S.W. Smoliar, n inegraed sysem for conen-based video rerieval and browsing, Paern Recogniion, 30(4) 997, []. Shih-Fu Chang, William Chen, Horace Meng, Hari Sundaram, D. Zhong, fully auomaed conenbased video search engine supporing spaioemporal queries, IEEE Transacions on Circuis and Sysems for Video Technology, 8(5), 998, [3]. Hampapur,. Gupa,. Horowiz, C.F. Shu, C. Fuller, J. ach, M.Gorkani, R. Jain, Virage Video Engine, Proc. SPIE: Sorage and Rerieval for Image and Video Daabases V, San Jose, 997, [4]. H.J. Zhang,. Kankanhalli, S.W. Smoliar, uomaic pariioning of full-moion video, Mulimedia Sysems, (), 993, 0-8. [5]. H.J. Zhang, C.Y. Low, S.W. Smoliar, J.H. Wu, Video parsing, rerieval and browsing: an inegraed and conen-based soluion, Proc. of he CM Mulimedia conference, San Francisco, C, November 5-9, 995, 5-4. [6]. M.. Moaleb, Nevenka Dimirova, Ranji Desai, J. Marino, CONIVS: Conen-based image and video access sysem, Proc. CM Mulimedia, oson, M US, 996, [7]. Yining Deng,.S. Manjunah, NeTra-V: Toward an objec-ased video represenaion, IEEE Trans. Circuis and Sysems for Video echnology, 8(5), 998, [8]. J.D. Courney, uomaic video indexing via objec moion analysis, Paern Recogniion, 30(4), 997, [9]..D. imbo, E. Vicario, D. Zingoni, Symbolic descripion and visual querying of image sequences using spaio-emporal logic, IEEE Trans. Knowledge and Daa Engineering, 7(4), 995, [0]. T.D.C. Lile,. Ghafoor, Inerval-based concepual models for ime dependen mulimedia daa, IEEE Trans. Knowledge and Daa Engineering, 5(4), 993, []. Serhan Dagas, rif Ghafoor, Indexing and rerieval of video based on spaial relaion sequences, Proc. CM Inl. Mulimedia Conference. (Par ), Oriando, FL, US, 999, 9-. []. Michael Vazirgiannis, Yannis Theodoridis, Timos Sellis, Spaio-emporal composiion and indexing for large mulimedia applicaions, Mulimedia Sysems, 6, 998, [3]. M.J. Egenhofer, R. Fanzosa, Poin-Se opological spaial relaions, Inernaional Journal on Geographic Informaion Sysems, 5(), 99, [4]. J.F. llen, Mainaining knowledge abou emporal inervals, Comm. of CM, 6(), 983, [5]. P. Masakis, L. Wendling, new way o represen he relaive posiion beween areal objecs, IEEE Transacions on Paern nalysis and Machine Inelligence, (7), 999, [6]. rchana M. Rajurkar, R.C. Joshi, Conen-ased Image Rerieval: Fuzzy Spaial Similariy pproach" Proc. of Inernaional Symposium on rificial Inelligence, ISI 00, For Panhala (Kolhapur), INDI, Dec 8-0, 00,

5 Proceedings of he 4s Hawaii Inernaional Conference on Sysem Sciences Table. Fuzzy Temporal Relaionships and he Corresponding Logical Descripion Fuzzy Temporal Relaionship Membership Funcion Logical Descripion ( x) = x (x) = e (x ) x < Objec disappear he appearance of x he objec, bu here may be some overlap mees 3 ( x) = x = mee mee (x) = e (x ) x <, x > Objec mee objec, bu here may be some overlap or hey may be disjoin Table. Second order Fuzzy Temporal Relaions Spaial Relaion Temporal Relaion Spaial Relaion Definiion of Second Order Fuzzy Temporal Relaions lef of efore righ of C L R C above efore below C C lef of Mees lef of C L mees L C above Mees below C mees C Noe: Se of inverse relaions exiss for all above relaions excep equals. The symbols used for spaial relaions are L lef of, R righ of, above, below. 5

6 Proceedings of he 4s Hawaii Inernaional Conference on Sysem Sciences Table 3. Temporal inervals and RMF values for he query Person in whie shir is lef of Person in black shir for he video sequences in he MPEG-7 video conen se Video Sequence Spaial Relaion Lef Relevance Membership Funcion (RMF) (eri_od_a.mpg) /73 =.3040 (eri_od_b.mpg) /0 = (eri_od_c.mpg) 9 890/9 = L R C (a) Fig. Graphical Illusraion of nd Order Fuzzy Temporal Relaion L R C Inpu: V db (video daabase) Oupu: Represenaion of video daabase sequences. Procedure: For every video sequences in he video daabase do he following. Moion segmenaion. Compue fuzzy spaio-emporal relaions. For every frame in he video sequence S, for each objec pair (O i, O j ) Compue fuzzy spaial relaions S = (R, O i, O j ) Compue he emporal inerval f for which he spaial relaionship S is valid T = S ) ( f 3. Compue he second order fuzzy emporal relaions beween every wo differen spaial relaions. T = (S < FTOP > S ) f f Fig. : lgorihm Represen_video fsm 6

7 Proceedings of he 4s Hawaii Inernaional Conference on Sysem Sciences Frame 73 Frame 90 Frame 36 Frame 343 Frame 79 Frame 900 Frame 636 Figure 3: Few sample frames in video eria_od_a.mpg 7