Video Classification and Retrieval with the Informedia Digital Video Library System

Transcription

1 Vdeo Classfcaton and Retreval wth the Informeda Dgtal Vdeo Lbrary System A. Hauptmann, R. Yan, Y. Q, R. Jn, M. Chrstel, M. Derthck, M.-Y. Chen, R. Baron, W.-H. Ln, and T. D. Ng. Carnege Mellon Unversty, School of Computer Scence, Pttsburgh PA, USA Ths paper s organzed n three parts. The frst part detals some of the lower level shot classfcaton work, the second part descrbes the manual retreval systems whle the last secton detals the nteractve retreval system for the Carnege Mellon Unversty TREC Vdeo Retreval Track runs. The descrpton of the data can be found elsewhere n the proceedngs of the 2002 TREC conference vdeo track overvew. Classfcaton In the TREC02 vdeo track, one of the man tasks s to detect varous semantc features concepts such as Indoor/Outdoor, People etc. Ths part contans the descrpton of the classfcaton tasks. We submtted runs for the followng classfcaton concepts n the TREC 2002 Vdeo Track. To obtan tranng data, we manually annotated each I-frame of the hours feature development collecton for each category. a. Outdoors: Segment contans a recognzably outdoor locaton,.e., one outsde of buldngs. Should exclude all scenes that are ndoors or are close-ups of objects (even f the objects are outdoor). b. Indoors: Segment contans a recognzably ndoor locaton,.e., nsde a buldng. Should exclude all scenes that are outdoors or are close-ups of objects (even f the objects are ndoor c. Ctyscape: Segment contans a recognzably cty/urban/suburban settng. d. Monologue: an event n whch a sngle person s at least partally vsble and speaks for a long tme wthout nterrupton by another speaker e. Face: at least one human face wth nose, mouth, and both eyes f. People: a group of two more humans g. Text Detecton: supermposed text large enough to be read h. Speech: human voce utterng recognzable words. Instrumental Sound: sound produced by one or more muscal nstruments, ncludng percusson nstruments Feature Extracton It s a crtcal challenge to fnd a good feature set for mage classfcaton. A number of mage features based on color and texture attrbutes have been reported n the lterature for mage retreval. We tred several of them and explored some new features at the same tme. Color Hstograms. We used the hstogram of 3*3 mage regons n HSV color space for each MPEG I-frame. The color features were derved from a hstogram n the quantzed HSV color space. Textures. We use the mean and varance of a texture orentaton hstogram for each of the 3*3 regons as texture feature. Edge features. We used a feature called the Edge Drecton Hstogram. A Canny edge detector was used to extract the edges from an mage. A total of 73 bns were used to represent the edge drecton hstogram of an mage; the frst 72 bns are used to represent the edge drectons quantzed at 5 o ntervals and the last bn represents a count of the number of pxels that ddn t contrbute to any edge. Edge drecton coherence vector. Ths feature stores the number of coherent versus non-coherent edge pxels wth the same edge drectons (consderng only horzontal and vertcal axs wthn a range of +/- 5 o,). We thresholded on the sze of every 8 connected components of edges n a gven drecton to decde whether the regon could be consdered coherent or not. Ths feature was used to dstngushed structured edges (lke edges of buldngs) from arbtrary edge dstrbutons.

2 Camera moton. We used statstcal dstrbuton patterns to detect the pan/tlt/zoom camera operatons based on the moton vectors of MPEG encodng. The resultng features encoded the presence/absence of these sx knds of camera operatons (pan left, pan rght, tlt up, tlt down, zoom n, zoom out) as a new type of feature for mage classfcaton. MPEG moton vectors. We transformed the moton vectors drectly encoded n the MPEG-compressed vdeo nto a dfferent knd of feature, namely a hstogram of the moton vector angle and velocty, as well as the wavelet coeffcents of moton vectors.. Although we expermented extensvely wth the features derved from camera moton analyss and the raw MPEG moton vectors, these addtonal features dd not contrbute to overall classfcaton accuracy. Classfcaton Algorthms We expermented wth several classfcaton tools for these tasks, ncludng SVM, KNN, Adaboostng and Decson Trees. Comparng ther performance usng cross valdaton on a comparatve large data set, we reached the concluson that support vector machne learnng was best, wth the power=2 polynomal as the kernel functon. Nonlnear functons usually performed better than lnear SVM kernel functons. The trade-off s that for nonlnear functons, the parameter space can be huge and therefore t may cause overfttng for small tranng datasets. Among the tasks, the ctyscape classfcaton suffered from the problem of nsuffcent postve tranng examples, whch s also the reason why we dd not submt a landscape classfcaton for evaluaton. For the ctyscape classfcaton tranng data, the postve examples (that s, the ctyscape mages vs. the non-ctyscape mages) comprsed only 12% of the whole data set. Such small ratos of postve examples n the tranng set cannot be well represented by the classfcaton methods we attempted. In addton, we nvestgated usng the ch-square functon as dstance functon based on publshed lterature. Contrary to publshed clams, the ch-square functon was not superor to any other functons. Cross_valdaton Due to the temporal correlaton between adjacent mages n a vdeo, an ntal cross valdaton based on random samplng of shots gave much better performance than approprate for the true predcton capablty of the models. Ths was due to the fact that smlar shots appeared throughout a sngle vdeo or move. So we performed a vdeo based cross valdaton based, usng 30 complete vdeos as tranng and then testng on the remanng 11 vdeos. Feature Selecton It s a challenge to select a good feature set for mage classfcaton. Qualfyng ther dscrmnaton ablty of each feature n the gven classfcaton problem s dffcult. We performed vdeo-based cross valdaton on tranng sets and compared the dfferent features performance based on the resultng classfcaton error and precson / recall of each task. For the camera moton related features and the MPEG moton vector related features, we explored numerous experments to test ther usefulness to the mage classfcaton task. However, they dd not gve conclusve results clearly. Fnally, we ended up not usng the camera or moton features n the fnal submsson. To get the best feature combnaton for each task, we performed a 6 folder move based cross valdaton on the three tranng sets on dfferent feature combnatons. The best feature combnatons were always ncluded texture, edge and color features. Snce the results were submtted as shot based features and not classfcaton from ndvdual mages, we ntegrated all I frame classfcaton results n a shot nto ths shot s feature detecton result. The confdence of a partcular feature detecton s the rato between number of feature presentng I frames vs. number of feature absent I frames n ths shot. Our results showed huge dfference of the performance of dfferent classfers. The reason of ths dscrepancy s possbly caused by the varablty of the tranng sets, the nconsstency between tranng set and the test sets, or the varyng dffculty of the dfferent classfcatons. Non-standard classfcaton for text, faces, people, monologues, speech and musc Varatons of the classfcaton approach were used for the face, people, monologue and audo categores. For the face category, we used the Schnederman face detector [19], exclusvely. For people. we extracted the followng features At the level of shots: Number of frames n a shot

3 Number of faces detected by the face detector Number of faces wth hgh confdence Number of faces wth low confdence Average confdence score of the faces n a shot, The standard devaton of the face scores, A smoothed mnmum face score, A smoothed maxmum score, Average pxel area for each detected face. For each I-frame wthn a shot we also extracted these frame-based features: Average number of faces per frame, Average number of faces per frame wth hgh confdence Average number of faces per frame wth low confdence Snce the total number of features was farly low, we traned a decson-tree based classfer (C4.5), whch outperformed SVM on ths task n cross-valdaton experments. Our contrastng people classfcaton submsson merely counted the number of faces vsble n each I-frame, and averaged ths over the whole shot. Ths baselne approach performed sgnfcantly worse wth a classfcaton error of vs The task of text-overlay classfer s to fnd scenes wth supermposed texts. Smply predctng a scene to be a text overlay based on whether or not the OCR engne s able to fnd text s not good enough because that OCR engne s qute error-prone. The features extracted were: 1. tme: related to the whole move, when s the OCR detected texts are found 2. #terms_wthn_a_shot 3. #dctonary_words_wthn_a_shot 4. average_popularty_vald_trgram_n_a_shot 5. average_popularty_vald_4gram_n_a_shot 6. average_no_alphabets_found_n_a_term 7. rato_dctonary_words_to_detected_terms 8. rato_length_of_all_dctonary_words_to_length_of_detected_terms For classfcaton, smlar to the people classfer, a decson tree (C4.5) was used nstead of a SVM. For monologues, we used as features: 1. The porton of tme where a least one (face) was detected. 2. The confdence of the face n every I-frame. 3. The number of speaker voce changes n one shot 4. The confdence n any sgnfcant audo change durng ths shot. 5. The number of faces present n one mage. These features were also fed nto an decson tree classfer. Speech and musc were classfed by the same speaker dentfcaton code as n the 2001 TREC vdeo track. Manual vdeo retreval wth classfcaton pseudo-relevance feedback Example-based mage retreval task has been studed for many years. The task requres the mage search engne to fnd the set of mages from a gven mage collecton that s smlar to the gven query mage. Tradtonal methods for content-based mage retreval are based on a vector model. These methods represent an mage as a set of features and the dfference between two mages s measured through a (usually Eucldean) dstance between ther feature vectors. Whle there have been no large-scale, standardzed evaluatons of mage retreval systems, most mage retreval systems are based on features such as color, texture, and shape that are extracted from the mage pxels. In our system two knds of low-level features are used for fndng smlar mages: color features and texture features. The color features are the cumulatve color hstograms for each separate color channel, where the three channels are derved from the HSV color space. We use 16 bns for hue and 6 bns for both saturaton and value. We generate a texture feature for each subblock of a 3*3 mage tessellaton. The texture features are obtaned through the

4 convoluton of the subblock wth varous Gabor Flters. In our mplementaton, 6 angles are used and each flter output s quantzed nto 16 bns. We compute a hstogram for each flter and generate ther central and second-order moments as the texture features. We concatenate all the features nto a longer feature vector for every mage;.e. one vector for all color features and one vector for all texture features. We use a smple nearest neghbor (NN) mage matchng algorthm on both color and texture to produce the ntal smlarty results. In a preprocessng step, each element of the feature vectors s scaled by the covarance of ts dmenson. We adopted the Eucldean dstance as the smlarty measure between two mages. Although nearest neghbor search s the most straghtforward approach to fndng the matchng mages, t suffers from two major drawbacks. Frst, rrelevant features n the vector are gven equal weght to mportant features, and thus retreval accuracy wll hurt decrease dramatcally. Feature selecton s therefore a necessary step pror to computng the nearest neghbor mages. In theory, relevance feedback, through re-weghtng and query refnement, s a powerful tool to refne the feature weghtng so as to provde more accurate results. However, t s mpossble to obtan the user judgment nformaton n most automatc retreval tasks. A second negatve aspect s the unjustfed dstance functon. Snce an approprate dstance measure s a functon of both the characterstcs of the dataset and of the queres, a smple Eucldean dstance functon s unlkely to work for all the queres and mages. Another concern s the normalzaton of the dfferent dmenson of a feature vector. To mtgate all these ssues, we propose a classfcaton-based pseudo-relevance feedback approach to refne the ntal retreval result. Support Vector Machnes (SVMs) are used as our basc classfer mechansm, snce SVMs are known to yeld good generalzaton performance compared to other classfcaton algorthms. The basc dea for ths approach s to augment the retreval results by ncorporatng the classfcaton output value through Pseudo-Relevance Feedback (PRF). The nput data for the classfer s based on the nformaton provded by our ntal retreval results. Standard PRF methods, whch orgnated n the text nformaton retreval communty, utlze the top-ranked documents as postve examples to mprove the accuracy. The dea s to re-weght the words n the document feature vector based on the words n the top ranked documents, whch are assumed to be postve examples. However, due to the poor ntal performance of current vdeo retreval system, even the very top-ranked results are not always the correct ones that meet the users nformaton need. Unlke n text retreval methods, t s more approprate to make use of the lowest ranked documents n the collecton after the ntal search, whch are more lkely to be the negatve examples. Therefore, we construct a classfer where the postve data are the query mage examples and the negatve data are sampled from the least confdent mage examples n the ntal retreval results. Snce the number of postve examples n our retreval task s always much smaller than the number of the negatve examples, we cast the problem nto the mbalanced dataset classfcaton framework. To sample more negatve examples but acheve an overall balanced dstrbuton of negatve and postve examples n the classfer tranng set, we apply an ensemble of SVMs to tackle the rare class problem. The overall procedure can be summarzed as follows, 1. Generate the ntal classfcaton results by nearest neghbor retreval for all the mages n the collecton. 2. Choose all the query mages as postve data. Denote the number of query mages as m. 3. Construct a negatve sub-collecton based on the ntal retreval results, whch are defned by the lowest 10% of the retreved data from the collecton. We sample k groups of negatve data from the negatve sub-collecton, where each group contans m query mages. Combne each group of negatve data and all the postve data as a tranng set. 4. Buld a classfer from each tranng set to produce new relevant score for any mages x ( x)(1 k), where s the ndex of tranng set 5. Combne the results n form of logstc regresson, whch s P( exp( β + 0 = 1 + x) = k 1+ exp( β + 0 k β f ( x)) = 1 In our system, we smply set 0 β f ( x)) β as 0, β ( 1 k) as equal values. f

5 Our approach presented here utlzes the collecton dstrbuton knowledge to refne the fnal result. Due to the good generalzaton ablty of the SVM algorthm, the most relevant features are selected automatcally. Also the approach yelds a better dstance functon based on the probablty estmaton compared wth the smple Eucldean dstance. Combnaton of multple agents As the frst step to ntegrate dfferent types of agents, all the relevance scores of the agents are converted nto posteror probablty. For each agent other than the classfcaton-based PRF agent, the posteror probablty s generated by a lnear transformaton of ther rank and scaled to the range of [0, 1]. All these posteror probabltes are smply lnear combnatons as follows: Score = a ( b P I c color ( + x) + b P t texture ( + x) + bprf PPRF ( + x)) + at Ptext ( + x) + am Pmove ( + x) where a I, at, a m s the weght for mage agent, text agent, move nformaton agent respectvely, whch are set to be 1, 1, 0.2. b c, bt, bprf are the weghts for the three search agents for mage retreval: NN on color, NN on texture and classfcaton PRF, whch are ether set to be 0 or 1 n our contrastve experments reported below. Speech Recognton The audo processng component of our vdeo retreval system splts the audo track from the MPEG-1 encoded vdeo fle, and decodes the audo and downsamples t to 16kHz, 16bt samples. These samples are then passed to a speech recognzer. The speech recognton system we used for these experments s a state-of-the-art large vocabulary, speaker ndependent speech recognzer. For the purposes of ths evaluaton, a word language model derved from a large corpus of broadcast news transcrpts was used. Prevous experments had shown the word error rate on ths type of mxed documentary-style data wth frequent overlap of musc and speech to be 35 40%. Text Retreval All retreval of textual materal was done usng the OKAPI formula. The exact formula for the Okap method s shown n Equaton (1) where tf(qw,d) s the term frequency of word qw n N df ( qw) document D, df(qw) s the document frequency for the tf ( qw, D)log( ) df ( qw) word qw and avg_dl s the average document length for Sm( Q, D) = qw Q D (1) all the documents n the collecton tf ( qw, D) Results avg _ dl We report our results n terms of mean average precson n ths secton, as shown n Table 1. Four dfferent combnaton of the retreval agents are compared n ths table, ncludng the combnaton of text agents (Text), move agents (Move), nearest neghbor on color (Color), nearest neghbor on texture (Texture) and classfcatonbased PRF (Classfcaton). The results show a sgnfcant ncrease n retreval qualty usng classfcaton-base PRF technque. Whle the text nformaton from the speech transcrpt accounts for the largest proporton of the mean average precson (0.0658), only a mnmal gan was observed n the mean average precson when the move ttle and abstract were also searched (0.0724) n addton to the speech transcrpts. The mage retreval component provded further mprovements n the scores to a mean average precson of Fnally, the PRF technque managed to boost the mean average precson to the fnal mean average precson score of Approach Precson Recall Mean Average Precson Text only (ASR) Text + Move nformaton (Abstract and Ttle) Text + Move + Image retreval (Color + Texture) Text + Move + Color + Texture + PRF Classfcaton Table 1 Vdeo Retreval Results on the 25 queres of the 2003 TREC vdeo track evaluaton.

6 Interactve Vdeo Retreval For the 2002 TREC vdeo track nteractve condton, we used the basc Informeda Dgtal Vdeo Lbrary system, as n the 2001 TREC Vdeo TREC. A few refnements to the nterface are dscussed and llustrated below. Fgure 1. Mult-document storyboards combne all shots from hghly relevant segments nto one dsplay. Snce IDVLS was desgned to return stores, whch can encompass multple shots as retreval results, we modfed the nterface to allow a shot-based presentaton of the results whch we called Multple document storyboards. The text was retreved n roughly 3-mnute story chunks, and all shots for that story were presented to the user. A storyboard dsplay, whch concatenated the top N relevant stores and ther shots, was used [Fgure]. Thus a user could vsually scan for relevant mages from a farly large storyboard dsplay of the top relevant stores and ther shots. Selectng a shot as relevant placed ths shot onto an answer set dsplay, whch could agan be edted before fnal submsson [Fgure]. Because of the large number of shots on the result storyboard, we placed the resoluton of the keyframe sze and the layout under user control. Thus a user can shrnk or enlarge the sze of the keyframes dsplayed on the storyboard, dependng on the desre to vsually nspect the keyframes more closely, or to vew the complete set. The sze of the wndow, and the total number of results dsplayed could also be modfed. We found that the query context plays a key role n flterng mage sets to manageable szes. The TREC 2002 mage feature set offered flterng capabltes for the classfed categores of ndoor, outdoor, faces, people, etc. The user nterface provded for a dsplay of the classfed feature values for every shot [Fgure]. The user was also able to control the threshold values for each of the feature categores. Ths enabled the dsplay to be more manageable by flterng out shots that were more lkely to

7 Fgure 2. Resoluton and layout of the storyboard can be modfed by the user. be feature X, and unlkely to be feature Y, dependng on the query context. Snce the dsplay showed the number of actve results, and provded drect feedback on the dstrbuton of the data, the large number of rrelevant shot could easly be fltered down to a manageable number, that was then vsually scanned by the user. The mult-document storyboard facltated quck nspecton of many mages. A frst-order flterng by query text provded an ntal set of mages that consttuted potental results. The mult-document storyboard based on 3-mnute segments and shots enabled the user to fnd relevant shots, whch were temporally near shots where query-words had been matched. The keyframe orderng by vdeo segment and tme useful. The classfed shot features were useful for flterng, but needed to be manually adjusted dependng on the partcular queres. Users were able to drlldown to detals, gong from keyframe mages to observng vdeo, whch was often necessary to elmnate uncertanty that could not be resolved by lookng at a stll mage frame. Fgure 3. Users can flter shots based on thresholds n any feature classfcaton category.

8 Fgure 4. Feature classfcaton statstcs are accessble for any shot. References 1. Hafner, J. Sawhney, H.S. Equtz, W. Flckner, M. and Nblack, W. Effcent Color Hstogram Indexng for Quadratc Form Dstance, IEEE Trans. Pattern Analyss and Machne Intellgence, 17(7), pp , July, Robertson S.E., et al.. Okap at TREC-4. In The Fourth Text Retreval Conference (TREC-4) Sato, T., Kanade, T., Hughes, E., and Smth, M. Vdeo OCR for Dgtal News Archve. In Proc. Workshop on Content-Based Access of Image and Vdeo Databases. (Los Alamtos, CA, Jan 1998), Sngh, R., Seltzer, M.L., Raj, B., and Stern, R.M. "Speech n Nosy Envronments: Robust Automatc Segmentaton, Feature Extracton, and Hypothess Combnaton," IEEE Conference on Acoustcs, Speech and Sgnal Processng, Salt Lake Cty, UT, May, A.W.M. Smeulders, M. Worrng, S. Santn, A. Gupta and R. Jan, Content-Based Image Retreval at the End of the Early Years, IEEE Trans. Pattern Analyss and Machne Intellgence, 22(12), pp , December, Swan M.J. and Ballard, B.H. Color Indexng, Int l J. Computer Vson, vol. 7, no. 1, pp , Tague-Sutclffe, J.M., The Pragmatcs of Informaton Retreval Expermentaton, revsed, Informaton Processng and Management, 28, , TREC 2002 Natonal Insttute of Standards and Technology, Text REtreval Conference web page, The TREC Vdeo Retreval Track Home Page,

9 10. Wactlar, H.D., Chrstel, M.G., Gong, Y., and Hauptmann, A.G. Lessons Learned from the Creaton and Deployment of a Terabyte Dgtal Vdeo Lbrary, IEEE Computer 32(2): Informeda Dgtal Vdeo Lbrary Project Web Ste. Carnege Mellon Unversty, Pttsburgh, PA, USA. URL A. Del Bmbo " Vsual Informaton Retreval", Morgan Kaufmann Ed., San Francsco, USA, Mojslovc, J. Kovacevc, J. Hu, R.J. Safranek, and S.K. Ganapathy, Matchng and Retreval Based on the Vocabulary and Grammar of Color Patterns, IEEE Trans. Image Processng, 9(1), pp , Gong, Y. Intellgent Image Databases: Toward Advanced Image Retreval. Kluwer Academc Publshers: Hngham, MA. 15. A. Valaya, A. Jan, and H.J. Zhang, "On mage classfcaton: cty mages vs. landscapes", Pattern Recognton, 31(12)(1998) Y. L and L. G. Shapro. "Consstent Lne Clusters for Buldng Recognton n CBIR," Internatonal Conference on Pattern Recognton, August M. Szummer, R. W. Pcard, Indoor-Outdoor Image Classfcaton, IEEE Internatonal Workshop on Contentbased Access of Image and Vdeo Databases, n conjuncton wth ICCV' Q. Iqbal and J. K. Aggarwal, "Applyng perceptual groupng to content-based mage retreval: Buldng mages," n IEEE Internatonal Conference on Computer Vson and Pattern Recognton, Fort Collns, Colorado, vol. 1, pp , June H. Schnederman, T. Kanade. "Probablstc Modelng of Local Appearance and Spatal Relatonshps for Object Recognton." IEEE Conference on Computer Vson and Pattern Recognton (CVPR), pp Santa Barbara, CA. 20. A. G. Hauptmann, R. Jn, N. Papernck, D. Ng, Y. Q, R. Houghton, and S. Thornton: Vdeo Retreval wth the Informeda Dgtal Vdeo Lbrary System. The 10th Text REtreval Conference (TREC 2001), Natonal Insttute of Standards and Technology (NIST) Gathersburg, Maryland, Fgure 5. The fnal result set can be revewed and edted before submsson.