MPEG-7 Pictorially Enriched Ontologies for Video Annotation

Transcription

1 MPEG-7 Pctorally Enrched Ontologes for Vdeo Annotaton C. Grana, R.Vezzan, D. Bulgarell, R. Cucchara Dpartmento d Ingegnera dell Informazone Unverstà degl Stud d Modena e Reggo Emla Abstract. A system for the automatc creaton of Pctorally Enrched Ontologes s presented, that s ontologes for context-based vdeo dgtal lbrares, enrched by pctoral concepts for vdeo annotaton, summarzaton and smlarty-based retreval. Extracton of pctoral concepts wth vdeo clps clusterng, ontology storng wth MPEG-7, and the use of the ontology for stored vdeo annotaton are descrbed. Results on sport vdeos and TRECVID2005 vdeo materal are reported. 1. Introducton The emergng nterest n Vdeo Dgtal Lbrares (DLs) requres effcent solutons for vdeo annotaton and access to ts content. Vdeo annotaton assocates to each vdeo element a sutable descrpton, stored n the Vdeo DL together wth the vsual content. Often, MPEG- 7 s adopted as a unfed standard that ncludes vsual parts and XML-based descrptons. Due to the nflexblty of the standard, non-complant MPEG-7 extensons often are proposed [1]; as an alternatve, RDF-based descrptons are adopted usng OWL for representng the semantcs assocated wth the vdeo content [2]. However, n Vdeo DLs, most of the content cannot be easly descrbed by lngustc terms only, but the ntrnsc vsual content can be better expressed wth perceptual cues. Snce humans make use of ther vsual knowledge makng mental mages of the world, the vdeo annotaton should nclude a pctorally enrched knowledge of the vdeo content. Ths work s amed at the exploraton of ths new paradgm. Annotaton of Vdeo DLs needs an ntal vdeo parttonng nto basc structural elements. Ths work s focused on clps, whch are tme segments that are manually or automatcally extracted usng certan perceptual cues or certan metadata embedded n the vdeo [3,4]. Clps can correspond ether to shots or to sub-shots n edted vdeos. A vdeo s dvded n hundreds of clps of dfferent length, whch must be annotated. In context-based DLs, annotaton s generally based on a defned ontology. Ontologes are used by people, databases, and applcatons that need to share doman nformaton. Ontologes nclude computer-usable defntons of basc concepts n the doman and the relatonshps among them. The tradtonal approach to vdeo annotaton conssts of two steps: 1) manual creaton of the (textual) ontology where concepts and relatonshps are formalzed. In the smplest cases, the ontology s a taxonomy defnng the semantc categores only; 2) annotaton, by means of a manual or automatc matchng of unknown clps aganst the concepts of the ontology. In case of automatc annotaton, fnte state automata or probablstc reasonng are employed for a supervsed classfcaton of clps [5].

2 We propose to employ pctorally-enrched (pct-en) ontologes and perceptual smlarty to provde automatc vdeo annotaton. The new framework conssts n the followng new steps: 1) manual creaton of the textual ontology as descrbed before, but enrched wth pctoral concepts: some prototypal clps (called pctoral concepts or prototypes) are extrapolated from the vdeo and assocated wth each category as a vsual specalzaton of the textual concepts of the ontology. Ths can be done manually or automatcally. Manual defnton s not trval snce the prototypes should be selected after an analyss of a large manfold of vdeo clps. Thus, we propose an automatc selecton wth an unsupervsed herarchcal clusterng, whch extracts a varable number of pctoral concepts from large tranng sets of clps of each category; 2) annotaton by means of automatc matchng of unknown clps aganst the pctoral concepts of the ontology. Ths s based on perceptual-level smlarty, such as moton, color or shape whch are present n the clps. Unlke other proposals [9], no semantc-level rules requrng a pror knowledge of the context are employed, nor are predefned reasonng automata. Instead, perceptual smlarty alone gudes the assocaton of each clp to a pctoral concept, and consequently to the concept that corresponds to ts generalzaton. Results are very promsng, especally consderng that the approach s very general and can be appled to any vdeo DL whatsoever. The only context-based procedure s the ntal textual ontology creaton. The sole requrement s the avalablty of a suffcently large tranng set of clps to select pctoral concepts. Therefore, snce annotaton s purely based on vsual smlarty, ths approach s partcularly sutable n context-based Vdeo DLs, such as DL of a gven sport where many repettve and smlar actons are collected. In ths paper, the framework s defned and results are provded both for specfc sport vdeo clps of Formula 1 races and for generc vdeo clps taken form TRECVID Smlarty of vdeo clps The problem of clp smlarty s a generalzaton of the problem of mage smlarty: as for mages, each clp s descrbed by a set of vsual features. For every frame, some features about moton, color and shape can be exploted n order to create a representatve vector: fr fr,1 fr,2 fr, fe V = V, V,.., V (1) where fr s the ndex of frames and fe s ndex of features. However, n clp smlarty, feature vectors cannot be extracted for each frame snce clps can have dfferent length, nor can a sngle feature vector for the whole clp be representatve enough, because t does not take nto account the temporal varablty of the features. Here, a fxed number C of feature vectors are defned for each clp, computed on frames sampled at unform ntervals wthn the clp. For each fr frame, we explot four types of features: 1. The color hstogram, n 256 bns of HSV color space. 2. The 64 spatal color dstrbutons: to account for the spatal dstrbuton of the colors, an 8x8 grd s supermposed to the frame and the mean YCbCr color s computed for each area. 3. The 64 DCT coeffcents for the frame texture: the coeffcents of DCT transform are appled followng the MPEG-7 specfcaton for the Color Layout Descrptor. 4. The four man moton vectors: they are computed as the average of the MPEG moton vectors [7], extracted n each quarter of frame. The medan value has been adopted snce MPEG moton vector are not always relable and are often affected by nose. A dssmlarty ndex between two clps S and S s defned as

3 Fg. 1. Example of complete lnk clusterng on 10 sub-shots. The Dunn s Index selected level s shown wth a dotted lne. ( ) C N S,,,, fr fe S fr fe, = fe 1 fr= 1 fe= 1 d S S k V V (2) where, V s the feature vector, fe s the feature-type ndex, fr s the frame ndex of clp S. The L 1 -norm between the feature vectors s multpled by an approprate constant k fe. These emprcal constants are tuned to optmze the classfcaton results on a second tranng vdeo, dfferent from the vdeos employed durng the ontology creaton. 3. Pct-En ontology creaton After the defnton of the textual doman ontology, a pct-en ontology requres the selecton of the prototypal clps that can consttute pctoral concepts as specalzaton of each ontology category. Large tranng sets of clps for each category must be defned and an automatc process extracts some vsual prototypes for every category. Usng the prevously defned features and dssmlarty functon, we employ a herarchcal clusterng method, based on Complete Lnk [6]. Ths technque guarantees that each clp must be smlar to every other n the cluster and any other clp outsde the cluster has dssmlarty greater than the maxmum dstance between cluster elements. For ths clusterng method we defned the dssmlarty between two clusters C and C as dc (, C) = max d xy, (3) x C, y C ( ) The algorthm proceeds as follows: 1. Intally we have M clusters { S1},{ S2},,{ SM }. Let us call E the set of clusters. Each cluster contans a sngle clp. 2. The least dssmlar par of clusters, R and S, s found accordng to Eq. 3,.e. R and S are found such that d( SR, ) d( AB, ) AB, E. (4) 3. R and S are merged nto a new cluster. 4. If everythng s merged n a sngle cluster then the algorthm stops, else t resumes from to step 2. Ths algorthm produces a herarchy of clps parttons wth M levels and clusters at level (the ntal level s M). To mplement the algorthm, a proxmty matrx D was used: ntally, t contans the dstances between each par of clps. At each step, the matrx s up-

4 <?xml verson="1.0" encodng="so "?> <Mpeg7 xmlns="urn:mpeg:mpeg7:schema:2001" xmlns:xs=" xmlns:mpeg7="urn:mpeg:mpeg7:schema:2001" xs:schemalocaton="urn:mpeg:mpeg7:schema:2001 Mpeg xsd"> <Descrpton xs:type="classfcatonschemedescrptontype"> <ClassfcatonScheme ur="urn:mpeg:mpeg7:cs:formula1"> <Term termid="cameracar"/> <!-- Ths s the ontology --> <Term termid="spectators"/> <Term termid="externalcarvew"/> <Term termid="people"/> </ClassfcatonScheme> </Descrpton> <Descrpton xs:type="modeldescrptontype"> <Model xs:type="collectonmodeltype"> <Label href="urn:mpeg:mpeg7:cs:formula1:cameracar"/> <Collecton xs:type="contentcollectontype"> <VsualFeature xs:type="scalablecolortype" numofcoeff="16" numofbtplanesdscarded="0" > <Coeff> </Coeff> </VsualFeature> <!-- These are the pctoral concepts --> <Content xs:type="vdeotype"> <Vdeo> <MedaLocator xs:type="temporalsegmentlocatortype"> <MedaUr>fle://race1.mpeg</MedaUr> <MedaTme> <MedaTmePont>T00:00:02:20080F30000 </MedaTmePont> <MedaDuraton>PT2S15075N30000F</MedaDuraton> </MedaTme> </MedaLocator> </Vdeo></Content></Collecton></Model> </Descrpton> </Mpeg7> Fg. 2. Example of an MPEG-7 descrpton of a Pctorally Enrched Ontology. dated by deletng rows and columns correspondng to clusters R and S and addng a new row and column correspondng to the newly formed cluster. The values n the new row/column are the maxmum of the values n the prevous ones. The generaton of the 1 proxmty matrx requres 2 M ( M 1) computatons of d (, ). To provde the user wth a frst selecton of the number of clusters, n order to supply an automatc soluton, we chose to avod usng fxed thresholds, but employed Dunn s Separaton Index [8]. Let us defne Δ ( C) = d( C, C) (5) δ ( C, C ) = mn d x, y (6) x C, y C ( ) The Separaton Index at level n s: mn δ ( C, C) 1, n, SIn = (7) max Δ( Ck ) 1 k n The proposed level s the one whch maxmzes SI n. Ths approach was tested on dfferent tranng sets of vdeo clps: the automatc soluton has been vsually evaluated and was udged satsfyng enough. An asssted approach may be employed to allow the user to enlarge or restrct the number of clusters by browsng among the cluster herarchy. To characterze every vsual prototype, the average feature set, at each of the C selected frame s used, and the clp whch mnmzes the dstance from the average s employed.

5 Classfcaton Camera car External car vew Spectators People C 40 45% 20 22% 0 0% 30 33% E 10 1% % 30 2% 20 2% S 0 0% 0 0% % 0 0% P 10 8% 10 8% 0 0% % Table 1. Confuson matrx for the test on Formula 1 vdeos. 4. Ontologes n MPEG-7 Ontologes may be effectvely defned wth OWL, but ths language does not contan any construct for ncludng a pctoral representaton. On the other hand, such feature s present n the MPEG-7 standard. MPEG-7 has much less sophstcated tools for knowledge representaton, snce ts purpose of standardzaton lmts the defnton of new data types, concepts and complex structures. Nevertheless, the MPEG-7 standard can naturally nclude pctoral elements such as obects, key-frames, clps and vsual descrptors n the ontology descrpton. Therefore, our system stores the pct-en ontology followng the drectons of the MPEG- 7 standard, and n partcular uses a double descrpton provded by the Classfcaton- SchemeDescrptonType DS combned wth a ModelDescrptonType DS whch ncludes a Collecton ModelType DS. The classfcaton scheme allows the defnton of a taxonomy or thesaurus of terms whch can be organzed by means of smple term relatons (see Fg. 2 for the lst of terms of the Formula 1 taxonomy). The collecton model s nstead an AnalytcModel, and therefore t descrbes the assocaton of labels or semantcs wth collectons of multmeda content. The collecton model contans a Content CollectonType DS wth a set of vsual elements whch refer to the model beng descrbed. In partcular, we lnked the selected clps and a representaton of ther features by means of the Scalable Color D, Color Layout D and the Mxture Camera Moton Segment Type D. In Fg. 2, an example of an MPEG-7 descrpton of a pct-en ontology s provded, wheren the two dfferent parts,.e. the ontology and the pctoral concepts, can be seen. 5. Automatc annotaton The pct-en ontology, could be drectly employed to gve a vsual ndex of the elements of the vdeos. Indeed, ts sgnfcance s related wth ts usefulness n the annotaton of new clps of the same doman. Annotaton s provded by the nearest neghbor rule: from the new clp, C feature vectors are extracted and nstead of usng all the elements of the tranng set, the dstance s computed only aganst the prototypes that are stored wth ther features n the pct-en ontology. In ths manner we decouple the phase of ontology creaton, whch can be very tme consumng, and the phase of the annotaton that should be done n a fast way. Usng MPEG-7 for feature descrpton, allows the pct-en ontology to be used wthout problems by dfferent systems, wthout the need of usng the same software lbrary. Moreover, the match may be performed on a subset of the features ncluded nto the ontology, for example for tme constrants. Moreover the procedure s very general, and after the ntal textual taxonomy creaton, all the other steps are automatc and can be appled n dfferent vdeo contexts. In our experments, a good tradeoff between effcacy and computatonal load suggest the use of C = 5.

6 Classfcaton Group of people Person n foreground Computer graphcs G 132 (77%) 28 (16%) 12 (7%) P 135 (52%) 119 (46%) 4 (2%) C 2 (3%) 29 (40%) 41 (57%) Table 2. Classfcaton performance on a subset of TRECVID2005 vdeos. Fg. 3. Example nterface output for a query of <external car vew>. The frst extensve test of the system has been done on avalable vdeo DLs of Formula 1 races of the last three years. The classes consttutng the ontology are very smple and are: <camera car>, <spectators>, <external car vew>, and <people> (such as mechancs, ournalsts, or plots). 340 clps were employed to create the ontology, wth a total duraton of 18 mnutes (27800 frames). The tests were conducted on a 90-mnutes vdeo, wth 1340 clps ( frames). The acheved results are shown n Table 1, whle Fg. 3 shows an example of the software nterface n a query operaton. Each new clp has been classfed aganst the prototype and the correspondent textual concept has been annotated. The total percentage of correctly classfed clps s 90.3%. Apart from the <spectators> clps, whch are not really sgnfcant, n <external car vew> a really good recall rate can be observed. On the other hand, the classfcaton of camera car clps was not as effectve. A possble reason for ths s probably related to the low frequency of camera car clps on the ontology tranng set. In fact, clps belongng to <external car vew> were 58% of the total, whle <camera car> were only 10%. It s mportant to consder that a larger Dgtal Lbrary would lkely show a hgher number of sub-classes, whch would ncrease the computaton tme. In the current tests, a 10 mnutes vdeo requres 6 mnutes for the automatc classfcaton and annotaton. A second test was based on a subset of TRECVID2005 dataset: we used a general broadcastng materal ontology, made of 280 clps. In ths test the classes were <group of people>, <person n foreground>, <computer graphcs> (e.g. at the end of a news program) (Fg. 4). As expected, the results are worse compared to a doman-specfc vdeo DL. Due to the hgh varablty of perceptual aspects of the clps, an average 58% of correct recall has been reached (Table 2). It s worth notng that no threshold has been used n the annotaton, and no category of unknown clps, whch could have mproved the precson, has been added. Ths result s satsfactory, snce t means that more than one clp out of two can be automatcally annotated, wthout semantc rules and only usng perceptual smlarty. Nevertheless, we thnk that the creaton of a complete pct-en ontology wth source materal of such a broad and general doman s not feasble wthout obtanng an exploson of ontology sze and computaton requrements.

7 Fg. 4. Graphcal representaton of the TRECVID test ontology. The clps are grouped nto three classes, each of whch s descrbed by a set of vsual prototypes. Only very few prototypes are shown for space constrants. 6. Conclusons We presented a system for the creaton of a specfc doman ontology, enrched wth vsual features and references to multmeda obects. The ontology s stored n MPEG-7 complant format, and can be used to annotate new vdeos. The annotaton has shown satsfactory results when the doman s narrow and well defned, and becomes less effectve on contents from too broad a context. Nevertheless, ths approach allows a system to behave dfferently by smply provdng a dfferent ontology, thus expandng ts applcablty to mxed sources Dgtal Lbrares. Acknowledgments Ths work s supported by the DELOS NoE on Dgtal Lbrares, as part of the IST Program of the European Commsson (Contract G ). We would lke to thank Flppo Barber for hs work on clusterng and code fxng. References [1] J. Rcard, D. Coeyrolly, and A. Baskurt, Art extenson for descrpton, ndexng and retreval of 3D Obects, n Proceedngs of the 17th Internatonal Conference on Pattern Recognton, Cambrdge, UK, (2004) [2] A. Chebotko, Y. Deng, S. Lu, F. Fotouh, A. Arstar, H. Brugman, A. Klassmann, H. Sloetes, A. Russel, and P. Wttenburg, OntoELAN: an Ontology-based Lngustc Multmeda Annotator, Proceedngs of the 6th Internatonal Symposum on Multmeda Software Engneerng (ISMSE 2004), Mam (FL), USA, (2004) [3] C. Grana, G. Tardn, and R. Cucchara, MPEG-7 Complant Shot Detecton n Sport Vdeos, n Proceedngs of the IEEE Internatonal Symposum on Multmeda (ISM 2005), Irvne (CA), USA, (2005) [4] N. Lan, and Y. Tan, Probablstc approach to k-nearest neghbor vdeo retreval, Proceedngs of the IEEE Internatonal Symposum on Crcuts and Systems (ISCAS 2004), Vancouver, Canada, (2004) [5] S. Chang, W. Chen, H.J. Meng, H. Sundaram, and D. Zhong, A Fully Automated Content-Based Vdeo Search Engne Supportng Spatotemporal Queres, IEEE Transactons on Crcuts and System for Vdeo Technology, vol. 8 num. 5 (1998) [6] A.K. Jan, and R.C. Dubes, Algorthms for clusterng data, Prentce-Hall, Englewood Clffs, NJ (1988). [7] G. Tardn, C. Grana, R. March, and R. Cucchara, Shot Detecton and Moton Analyss for Automatc MPEG-7 Annotaton of Sports Vdeos, n Proceedngs of the 13th Internatonal Conference on Image Analyss and Processng, Caglar, Italy, (2005) [8] J.C. Dunn, A fuzzy relatve of the ISODATA process and ts use n detectng compact well-separated clusters, Journal of Cybernetcs, vol. 3 num. 3 (1973) [9] M. Bertn, R. Cucchara, A. Del Bmbo, and C. Torna, Vdeo Annotaton wth Pctorally Enrched Ontologes, Proceedngs of IEEE Internatonal Conference on Multmeda and Expo, Amsterdam, The Netherlands, (2005)