Real-Time View Recognition and Event Detection for Sports Video

Transcription

1 Real-Tme Vew Recognton and Event Detecton for Sports Vdeo Authors: D Zhong and Shh-Fu Chang {dzhong, sfchang@ee.columba.edu} Department of Electrcal Engneerng, Columba Unversty For specal ssue on Multmeda Database Management Systems at Journal of Vsual Communcaton and Image Representaton, Correspondence Contact: Prof. Shh-Fu Chang Department of Electrcal Engneerng 500 W. 120 th St., Rm Columba Unversty New Yor, NY sfchang@ee.columba.edu tel: fax: url: 1

2 Real-Tme Vew Recognton and Event Detecton for Sports Vdeo Keywords: Vdeo Indexng, Content-Based Vdeo Retreval, Sports Vdeo Analyss, MPEG-7 Abstract In ths paper, we present a general framewor and new effectve algorthms to detect the syntactc structures that are at a level hgher than shots. In sports vdeo, such hgh-level structures are often characterzed by the specfc vews (e.g., ptchng or serve) and the subsequent temporal transton patterns wthn each temporal structural segment. We have developed robust statstcal models for detectng the doman-specfc vews wth real-tme performance and hgh accuracy. The models combne doman-ndependent global color flterng method and doman-specfc constrants on the spato-temporal propertes of the segmented regons (e.g., locatons, shapes, and moton of the objects). The real-tme performance was accomplshed by usng effcent compressed-doman processng at the front end and computatonal expensve object-level processng on fltered canddates only. Hgh-level events (e.g., stroes, net plays, baselne plays) are also detected after the vew recognton. Results of such structure and event detecton allow for effcent browsng and summarzaton of long sports vdeo programs. 2

3 1. Introducton In a typcal content-based vdeo ndexng system, vdeos are decomposed nto shots whch are then processed to obtan consttuent objects and features [1]. As shown n Fgure 1, these extracted enttes form a comprehensve feature lbrary and a useful framewor for descrbng vdeo content. Whle these features are useful n vdeo retreval such as smlarty searchng, they lac nformaton at the semantc level. To address ths problem, research ncorporatng doman specfc nowledge provdes a promsng drecton. In ths paper, we present a hgh-level scene and structure analyss system for sports vdeos. Ths system s bult on top of object segmentaton, feature extracton and matchng technques we have developed n our pror wors [10], [11], [12]. Specfcally, we present model-based and rule-based methods for detectng mportant, recurrent scenes n sports vdeos (e.g., ptchng scenes n baseball and servng scenes n tenns). Our methods use off-lne supervsed learnng and on-lne adaptve updatng to obtan effectve scene models and the assocated spato-temporal rules. To acheve hgh accuracy, we also adopt a mult-stage process n whch complex object-level features are analyzed n the later stage to verfy canddate vdeos detected by less complex models n the ntal stage. Real-tme performance s emphaszed n ths wor n order to satsfy the requrement of lve vdeo flterng and dynamc content-adaptve encodng. In [16], we combne the real-tme structure detecton tool presented here wth adaptve rate encodng to mprove the overall qualty of the vdeo vdeo segments of mportant events (e.g., ptchng) are allocated more bandwdth than other segments. We acheve the real-tme performance by extractng and analyzng most of the features n the compressed doman (.e., MPEG-1 or -2 compressed format). Gven the detected scenes and structures n the vdeo, we further develop an effectve summarzaton and browsng nterface allowng users to easly access content structure and event ndex of vdeos. Combnaton of doman nowledge and low-level features has been explored n some wors. In [9], an edge model s used to match anchorperson vews n specfc news programs. The story structure s then reconstructed by fndng anchorperson vews as well as commercals. In [3], logcal story unts (LSU s) are extracted from moves accordng to temporal consstency of frame-level features (e.g., color hstogram). The consstency s based on the assumpton that an event related to a specfc locaton and certan characters wll result n consstent features n a perod of tme. In [4], a mult-modalty memory model was used to detect audo-vsual scene boundares n flms by measurng the consstency of vsual features, audo features, and ther synchronzaton. Compared wth other types of vdeos, sports vdeos have dfferent characterstcs. A sports game usually occurs n one specfc locaton, contans rch moton nformaton and has well-defned syntactc and semantc structures. Event detecton n basetball and tenns vdeos has been studed n [7], [8], [13] respectvely, but wthout parsng hgh-level temporal structures. Vew and event classfcaton has been proposed n [14] for soccer vdeo ndexng, but wthout usng object-based verfcaton framewor and compressed-doman approaches for real-tme performance. Detaled comparson between the proposed methods and the pror wors wll be presented n Secton 2. In ths paper, we present a real-tme structure parsng and event detecton system for sports vdeos. Compared to exstng wor, our system has the followng unque features. 3

4 A general framewor for vdeo structure parsng and event detecton. Combnaton of doman-ndependent global model-based flterng methods wth domanspecfc object-level spato-temporal constrants. Mult-stage semantc scene detecton and verfcaton algorthms usng our unque movng object segmentaton and feature analyss methods. Real tme processng performance by processng most features n the compressed doman. Hgher accuracy demonstrated n specfc sports domans such as tenns and baseball. In the rest of the paper, we wll frst dscuss content structures n sports vdeos. The scene detecton and structural analyss system s descrbed n Secton 3, usng tenns as an example. Experment results n tenns and baseball vdeos are shown. In Secton 4, we present methods for detectng hgh-level events (e.g., net plays and stroes n tenns) occurrng wthn each scene. Fnally, n Secton 5, we descrbe an effectve summarzaton and browsng applcaton, whch allows users to easly access content structure and event ndex n the vdeo. Conclusons and future wors are summarzed n Secton Content Structures n Sports Vdeo Sports vdeo represents a popular type of broadcast TV programs. Compared to other vdeos such as news and moves, sports vdeos have well-defned content structures and doman-specfc rules. A long sports game s often dvded nto a few segments. Each segment may agan contan several sub-segments. For example, n football programs, a game contans two halves, and each half has two quarters. Wthn each quarter, there are many plays, and each play usually starts wth the formaton n whch players lne up on two sdes of the ball. In tenns, a game s dvded frst nto sets, then games and serves (as shown n Fgure 2). In addton, n a sports vdeo, there are a fxed number of cameras n the feld that result n recurrent dstnctve scenes throughout the vdeo. In tenns, when a serve starts, the scene s usually shown wth the full-court vews (Fgure 3). In baseball, typcally each ptch starts wth a ptchng vew taen from behnd the ptcher. These vews usually have consstent vsual-audo attrbutes that do not vary greatly from game to game. For example, n baseball, the ptchng scenes typcally show consstent features and spato-temporal relatonshps of consttuent objects of the scene (e.g., grass, ground, ptcher, catcher, and batter n the ptchng scene). Ths property has been explored n our pror wor n learnng effectve detectors for baseball ptchng based on a structured scene model [15]. In tenns, the full-court scenes typcally show consstent ground colors, court lnes, and the players (e.g., as shown n Fgure 3). Although there wll certanly be varatons n colors (dfferent court surfaces), lghtng, and vew angles, we argue the consstence s sgnfcant and effectve pre-flters can be developed to detect canddates whle more complex rules (such as rules assocated wth the lnes and players) can be used to mprove the detecton performance. These characterstcs allow us to develop effectve vdeo ndexng methods that combne doman-specfc syntax and rules, generc machne-learnng tools, and automatc tools for vdeo object analyss. Gven a sports vdeo, our prmary goal s to dentfy mportant structures and events that are nterestng to users, and automatcally or sem-automatcally buld a content ndex allowng for 4

5 effcent search and retreval of such events and structures. A few pror wors have been conducted to analyze sports vdeos. In [7], a basetball vdeo annotaton system s presented. Usng specfc nowledge of basetball games, the system detects wde-angle shots versus close-up shots. Camera motons are further analyzed wthn wde-angle shots to detect possble fast brea, steal and probable shots. When users want to retreve one type of event (e.g., fast brea), only sequences satsfyng correspondng moton crtera are shown. Tenns vdeos are studed n [8]. The approach s based on the extracton of a model for the tenns court-lnes from vdeos. A player tracng algorthms s developed to trac players, and then a reasonng module s used to map low-level postonal nformaton to hgh-level tenns play events. In [13], a sequence of ad hoc audo features (e.g., ball httng sound followed by crowd shoutng sound) are used to detect hghlght segments n baseball vdeos. In [14], frame-level features and some doman-specfc features are used to detect varous vews and events n soccer vdeos. In [15], our pror wor uses nteractve learnng tools to develop the optmal features and classfers n a herarchcal scene model for detectng vews n baseball. Constrants assocated wth the objects (e.g., color and shape of ground or grass regons) were automatcally dscovered by analyzng user annotated vdeo samples. Although a dfferent detecton framewor s used n ths paper, such dscovered object-level constrants are very useful and have been ncorporated n the object-level verfcaton stage of the system descrbed n ths paper. Our wor n ths paper shares smlar goals as the above wors but emphaszes more on parsng the temporal structure by detectng the recurrent content unts (e.g., play n baseball and serve n tenns). We adopt a dstnctve framewor, whch ncorporates a systematc learnng approach and an adaptve model selecton method. It also combnes doman-ndependent flterng methods wth specal refnement methods usng doman-specfc rules. Another challengng problem that has not been addressed n pror wors s the real tme performance ssue. Lve sports vdeo broadcastng has mportant applcatons. Interest of audence goes down greatly after the game results are nown. The real-tme capablty n generatng event ndces s crtcal n practcal applcatons for lve sports vdeos. Even for archved sports vdeos, such real-tme ndexng system stll provdes sgnfcant value n cuttng the potentally prohbtve cost nvolved n manual annotaton. In ths paper, we present a general framewor and new effectve algorthms to analyze structures, vews, and events of broadcasted sports vdeos n the real tme. Our approach conssts of two phases tranng and operaton, as shown n Fgure 4. Frst, n the tranng phase, feature models and object rules are learned automatcally or sem-automatcally. In the operaton phase, optmal models are selected to adapt to new vdeo programs from lve sources and used to detect target scenes and events n new vdeos. A scene verfcaton model s also appled to reduce false postves. Fnally, hgh-level events wthn detected scenes are detected based on constrant models on spato-temporal propertes of the segmented objects. 3. Mult-Stage Vew Recognton To automatcally analyze the temporal structure of a sport game, we need to detect vsual cues that ndcate the begnnng and/or endng of each structural element. Some common features occurrng between top-level sectons are commercals, embedded texts and specal logos. Many methods have been proposed for commercal and text detecton [5], [6]. Here we wll study the detecton of basc unts wthn a game, such as serves n tenns and ptches n baseball. These unts usually start 5

6 wth a specal vew. Smple color based approaches have been suggested n [8]; but based on our experments to be descrbed later, such approaches do not acheve adequate performance. Furthermore, as color nformaton vares from game to game, adaptve methods need to be exploted to handle such varatons. In our real-tme framewor, we frst use a fast adaptve color flterng method to dentfy possble canddates, then apply complex verfcaton methods usng spato-temporal propertes of segmented regons. In the followng we present the components of the system, and evaluaton results usng tenns vdeo as an example. 3.1 Color Based Adaptve Flterng Color based flterng s appled to ey frames of vdeo shots. Frst, the global color models are bult through tranng process based on clusterng. The manually selected tranng data contan sample segments from dfferent games wth dverse condtons. Assume h, =1,N are color hstograms of serve scenes n the tranng set. A -means clusterng s used to generate K models (.e., clusters), M,, M, such that, 1 K h, f D( h, M ) = mn( D( h, M )) (1) M j j K = 1 where D ( h, M ) s the dstance between h and the mean vector of M,.e. H = 1 M h h M M s the number of tranng scenes belongng to model M. Ths means that for each model M, H s used as the representatve feature vector. In order to accommodate dverse condtons (e.g., lghtng, court surface, and vewng angles), we delberately nclude a large number of color models n the tranng process, and then narrow down to a small set of models by adaptng the model pool to the ntal porton of the vdeo from test data. Ths rases a typcal chcen-and-egg problem, as we need to now serve scenes n the new data n order to tran a correct model. To solve the problem, we detect the frst L (e.g., a small number le 8) serve scenes usng all models (.e. clusters), M 1,, M K. That s to say, all models are used n the ntal flterng process. If one shot s close enough to any model, the shot wll be consdered as a lely canddate and passed to subsequent verfcaton processes (see 3.2 and 3.3). K h ' M, f ' ' )) j D( h, M j ) = mn( D( h, M and D( h ', M j ) < TH (2) = 1 ' where h s the color hstogram of the th shot n the new vdeo, and TH s a gven flterng threshold to accept shots wth suffcent color smlarty. In our experments (Secton 3.4), the TH value s chosen based on the emprcal dstrbuton modes obtaned from the tranng data. Shot s detected as a serve scene f the subsequent object-level verfcaton process (descrbed n 3.2) s also successful. In ths case, we mar ths serve scene as beng detected by model M (.e., classfy the scene nto the model M j ). If the verfcaton fals, ' h s removed from the set of j M j. and 6

7 After L serve scenes are detected, we fnd a matched model wth the most serve scenes. K M o, by searchng for the model M o = max( M ) (3) = 1 where M s the number of ncomng scenes beng classfed nto the model M. We consder model M o and a few (e.g., 2) of ts closest neghborng models as representatve models for the partcular vdeo programs under consderaton. The adaptve model selecton process descrbed above allows us to avod the burden of developng customzed models for dfferent vdeo sources (e.g., channels, courts). The same pool of models obtaned from a sngle tranng process are appled n the ntal stage when gven a new vdeo source. A matched model plus several of ts mnor varatons are dentfed n the ntal process and are then used for detectng target scenes n any future data from the same source. 3.2 Object Segmentaton Based Verfcaton Color hstograms are global features that can be computed and compared faster than real tme. However, wth the color feature only, our experments showed a detecton accuracy less than 80%. Many close-up scenes of the play feld and replay scenes are lely to be detected as false postves. To mprove detecton accuracy, we extend our prevous wor [10], [11] on salent regon and object segmentaton to compute localzed spatal-temporal features. Compared wth global features, such as color hstograms, spatal-temporal features are more relable and effectve n detectng gven scene models. Especally n sports vdeos, specal scenes often consst of multple objects subject to spato-temporal constrants (e.g., players n the server scene n tenns). In [11], we have developed a foreground object segmentaton system based on regon mergng, tracng, and bacground layer modelng. Here we extend such pror technque to detect and trac objects n sports. In regular unconstraned vdeo, as depth nformaton s not well preserved, many approaches have been proposed to use multple 2D parametrc models to estmate multple moton layers. Many exstng methods rely only on moton nformaton n groupng mage pxels or blocs nto moton layers, and thus usually result n naccurate segmentaton on moton boundares. As there s a strong nter-dependence between moton estmaton and layer segmentaton, poor segmentaton n turn wll degrade the accuracy of moton estmaton. To overcome these problems, applyng our pror results n regon segmentaton and tracng, we have developed a two-stage movng objects detecton method for sports vdeo. Ths method uses regons wth accurate boundares to effectvely mprove moton estmaton results. Furthermore, we explore the temporal constrant n a vdeo shot to acheve more relable object detecton results. The two general stages of our algorthm are shown n Fgure 5. In the frst stage, we apply an teratve moton layer detecton process based on the estmaton and mergng of affne moton models. Each teraton generates one moton layer. The dfference from exstng methods s that moton models are estmated from spatally segmented color regons nstead of ndvdual pxels or blocs. In the second stage, temporal constrants are appled to detect movng objects n spatal and temporal space. Layers n ndvdual frames are lned together based 7

8 on consstency and moton projecton of ther underlyng regons. One or more layers are declared as moton objects accordng to spato-temporal consstency rules. More detaled descrptons are avalable n [11]. To acheve real-tme performance, here segmentaton s performed on the down-sampled mages of the representatve frames of each shot (e.g., frst I frame n each shot). The down-samplng rate used n our experment s 4, both horzontally and vertcally, whch results n mages wth sze 88x60. Moton felds are estmated usng the herarchcal approach as proposed n [2]. An example of segmentaton and detecton results s shown n Fgure 6. Fgure 6 (b) shows the regon segmentaton result. The court s segmented as one large regon, whle the player closer to the camera s also extracted. The court lnes are not preserved due to down-samplng. Blac areas shown are tny regons that are dropped at the end of segmentaton process. Fgure 6 (c) shows the fnal movng object detecton result. In ths example, the result s qute satsfactory, and only the desred player s detected. Sometmes a few bacground regons may also be detected as foreground movng object. We wll address ths ssue by usng the tracng algorthm dscussed n Secton 4. Here for verfcaton purpose, as we wll descrbe below, the mportant requrement s not to mss the player. Gven the segmented regons and moton foreground objects, we apply the followng verfcaton rules derved manually based on doman nowledge. Frst, there must be a large regon (e.g. larger than two-thrds of the frame sze) wth consstent color (or ntensty for smplcty). Ths large regon corresponds to the tenns court. The unformty of a regon s measured by the ntensty varance of all pxels wthn the regon (Eq 4). N 1 2 = [ I( p ) I ( )] N (4) = 1 Var( p) p where N s the number of pxels wthn a regon p. I ( p ) s the ntensty of pxel and I ( p) s the average ntensty of regon p. If Var(p) s less than a gven threshold, the sze of regon p s examned to decde f t corresponds to the tenns court. Secondly, the sze and poston of the foreground movng player are examned. The condton s satsfed f a movng object wth certan sze condtons s detected wthn the lower half part of the prevously detected large regon correspondng to court. In a downszed 88x60 mage, the sze of a player s usually between 50 to 200 pxels. As our detecton method s appled at the begnnng of a serve, players usually stay near the bottom court lne. Thus the poston of a detected player has to be wthn the lower half part of the court. 3.3 Edge Based Verfcaton One unque characterstc of servng scenes n the tenns game s that there are horzontal and vertcal court lnes. Ideally f a camera s postoned at the rear-top pont of the court and all court lnes are captured (Fgure 2), rules correspondng to a complete court can be used to verfy serve scenes wth hgh precson. However, n practce, due to camera pannng, zoomng, or object occluson, usually not all court lnes are vewable. Use of full-court condtons wll result n a low recall rate of serve scene detecton. Snce we already use color flterng and regon-level verfcaton processes, matchng condtons of court lnes are made relatvely loose. An example of edge detecton usng the 5x5 Sobel operator 8

9 s gven n Fgure 7. Note that the edge detecton s performed on a down-sampled mage and nsde the detected court regon only. Hough transforms are conducted n four local wndows (shown n Fgure 8) to detecton straght lnes. Wndows 1 and 2 are used to detect vertcal court lnes, whle wndows 3 and 4 are used to detect horzontal lnes. It greatly ncreases the accuracy n detectng straght lnes to use local wndows nstead of a whole frame. As shown n the fgure, each par of wndows roughly cover a lttle bt more than half of a frame, and are postoned somewhat close to the bottom border. Ths s based on the observaton about the typcal court lne locatons wthn the court vew. The verfyng condton s that there are at least two vertcal court lnes and two horzontal court lnes beng detected. Note these lnes have to be apart from each other by a certan dstance, as nosy edge detecton and Hough transform may produce duplcate lnes. Ths s based on the assumpton that despte of camera pannng, there s at least one sde of the court, whch has two vertcal lnes, beng captured n the vdeo. On the other hand, camera zoomng wll always eep two of three horzontal lnes,.e., the bottom lne, mddle court lne and net lne, n the vew. 3.4 Experments and Dscusson We appled the above flterng and verfcaton scheme to tenns and baseball vdeos. Two dfferent color model pools and sets of verfcaton rules are traned and constructed respectvely. The color models used n the frst flterng stage nclude clusters extracted from short clps (a few mnutes) from dfferent sources (e.g., dfferent channels or games). We eep the tranng samples dverse n order to buld a comprehensve color model pool. When gven a test vdeo source, the best color model s dentfed based on the ntal detecton results. Whle vew varaton s a crtcal ssue for color only approaches, our approach has ncorporated regon-based and edge based verfcaton rules and thus s flexble n handlng such stuatons. Consderng there are only lmted types of play felds, we can lower our flterng threshold to allow more false alarms n the frst stage, and rely on the regon/edge based verfcaton process to mprove overall precson. We conducted two separate experments. The frst one use 1-hour test vdeo, part of whch s also present n the tranng set. Table 1 shows the results of ths test. The overall recall and precson rates are very good about 95%. In the second experment, we use test data that does not have overlap wth the tranng set. Table 2 ncludes the results, whch are comparable to the tranng performance dscussed above. Ths shows that our scheme s not senstve to vew varatons caused by changes n colors, slght camera angles, lghtng, and felds. Slghtly more false alarms are detected n the baseball vdeo, whle no more msses are observed. Ths means our verfcaton rules for baseball vdeo are rather relaxed. It s possble to add more restrctons on object sze, shape, poston and other features. On the other hand, for the tenns vdeo, there s no ncrease n the number of false postves. Ths s because the verfcaton model matches serve scenes very well. In a separate test, we test serve vew detecton usng the object-feature verfcaton model only, wthout the frst stage of color flterng. Interestngly we obtaned accuraces smlar to those descrbed above. Ths ndcates that our object-feature verfcaton model matches the tenns serve vews very well, at least for the test sequences we have used. However, the global color flterng stage s needed for real-tme processng purpose t s capable of removng unlely canddates qucly wthout needng to apply the fne verfcaton procedures on every ey frame. 9

10 These above results are very good compared to exstng approaches usng color matchng only. Based on our experments, prevously proposed approaches usng color hstogram flterng can only acheve about 80% precson n order to obtan near 100% recall. Furthermore, despte of usng advanced segmentaton and feature extracton, our overall detecton process s performed n real tme. 4. Event Detecton Technques descrbed above detect canoncal vews n sports vdeo. Such canoncal vews typcally correspond to start of recurrent semantc unts (e.g., serve n tenns) n sports vdeo. In ths secton, we focus on detectng and summarzng what happened n a scene. Especally, we have adapted our movng object detecton algorthm to trac a tenns player n real-tme, analyze the player s trajectory, and nfer the actons and events assocated wth the player. 4.1 Player Tracng In [11], we presented an automatc movng object detecton method that conssts of two stages: (1) an teratve moton layer detecton step that s performed n ndvdual frames, and (2) a temporal tracng process lnng layers across frames and classfy the lned layers nto foreground movng objects vs. bacground. Here we adapt ths approach to trac tenns players wthn court vew n real tme. In our present system, we focus on the player who s close to the camera. The player at the far sde s smaller and may not be nsde the vew. In order to acheve real-tme performance, we apply the object segmentaton methods on the down-szed mages, thus mae detecton of small objects dffcult. For local moton layer detecton, regons n down-sampled I- and P-frames are segmented and matched to extract moton layers. The reason to sp B-frames s because b-drecton predcted frames requre more computaton to decode. To ensure real-tme performance, only one par of anchor frames are processed every half second. For a MPEG stream wth a GOP sze of 15 frames, the I-frame and ts mmedate followng P-frame are used. Moton layer detecton s not performed n later P frames n the GOP. Ths change requres a dfferent temporal detecton process for detectng movng objects. We descrbe the enhancement below. As half second s a rather large gap for the estmaton of moton felds moton-based regon projecton and tracng from I frame to another I frame are not relable, especally when there are fast motons n a scene. Thus, a dfferent process s requred to match movng layers detected n ndvdual I-frames. We use the followng temporal flterng process to select and match objects that are detected n I frames. Assume O s the th object (=1,,K) at the th I-frame n a vdeo shot, p r, c and s are the center poston, mean color and sze of the object respectvely. We defne the dstance between O and another object at j th l I-frame, O j, as weghted sum of spatal, color and sze dfferences. l l l l D( O, O ) = w p p + w c c + w s s (5) j p j c j s j 10

11 l where w p, w c and w s are weghts on spatal, color and sze dfferences respectvely. If D ( O, O j ) l s smaller than a gven threshold, O_TH, objects O and O j match wth each other. We then defne the match between an object wth ts neghborng I-frame + δ as follows, l l 1 O+ δ, D( O, O+ δ ) < O _ TH F( O, + δ ) = (6) 0 otherwse where δ = ±1,..., n. Let M = δ =± 1,..., n F( O, + ) δ be the total number of frames that consst matches of object O (=1,,K) wthn the perod δ to + δ, we select the object wth maxmum M. r Ths means that f M = max ( M ), the r th object s ept at the th I-frame. The other objects are = 1,..., K dropped. The above process can be consdered as a general temporal medan flterng operaton. Ideally, the preserved object corresponds to the most salent movng object, namely the tenns player n the near sde. After the above selecton, we obtan the trajectory of the lower player by measurng the center coordnates of the selected movng objects n each I frame. Here some clarfcatons are due. Frst, f no object s found n a frame, lnear nterpolaton s used to fll the mssng pont. When there are more than one objects beng selected n a frame (n the stuaton when more than one objects have the same maxmum number as defned above), the one that s spatally close to the object n the prevous I frame s used. In addton, for speed reason, nstead of usng the affne moton model to compensate camera moton, here we use the detected net lnes to roughly algn dfferent frames. Experment results of player tracng n one serve scene are shown n Fgure 9. The frst row shows down sampled frames. The second row contans player tracng results. The body of the player s well traced and detected. Successful tracng of tenns players provdes a foundaton for hgh-level semantc event analyss. Compared wth the tracng algorthm n [8], whch computes resdual errors to fnd movng objects and then searches players n pre-defned wndows, our approach provdes hgher accuracy as well as real tme performance. 4.2 Trajectory Analyss The extracted trajectory s analyzed to obtan more detaled nformaton about each play. Presently, we focus on two aspects. The frst one s the poston of the player. As players usually play at bottom lnes, we want to fnd cases when a player moves to the net zone. The second one s to estmate the number of stres by the player wthn a serve. Users who want to learn stroe slls or play strateges may be nterested such nformaton. Gven a trajectory contanng K coordnates, p (=1,,K), at K successve I-frames, we frst detect stll ponts and turnng ponts. p s called a stll pont f, mn( p p 1, p p + 1 ) < TH (7) where TH s a pre-defned threshold. Furthermore, two consecutve stll ponts are merged nto one. If pont p s not a stll pont, the angle at the pont s computed. p s a turnng pont f o ( p p 1, p p + 1) < 90 (8) 11

12 An example of object trajectory s shown n Fgure 10. After detectng stll and turnng ponts, we use them to judge the player s postons. If there s a poston close to the net lne (vertcally), the serve s classfed as net-zone play. The estmated number of stroes s the sum of the numbers of turnng and stll ponts. Experment results of the one-hour test vdeo are gven n Table 3. In the vdeo, the ground truth ncludes 12 serves wth net play wthn about 90 serve scenes (see Table 1), and totally 221 stroes n all serves. Most net plays are correctly detected. False detecton of net plays s manly caused by ncorrect extracton of player trajectores or court lnes. Stroe detecton has a precson rate about 72%. Besde the reason of ncorrect player tracng, some errors are caused by lmtatons of our estmaton model. Frst, at the end of a serve, a player may or may not stre the ball n hs or her last move. Many serve scenes also show players walng n the feld after the play. In addton, a sever scene sometmes contans two serves f the frst serve faled. These may cause problems snce currently we detect stroes based on the movement nformaton of the player. To solve these ssues, more detaled analyss of moton such as speed, drecton, repeatng patterns n combnaton wth audo analyss (e.g., httng sound) or ball tracng wll be useful. 5. Summarzaton and Browsng As we have observed, vdeo data contan large amount of vsual and semantc nformaton. Even after content ndces are generated, how to show these ndces n a lmted-sze dsplay s a challengng ssue. In ths secton, we present a system for vdeo browsng and summarzaton usng the structure parsng and event detecton results presented earler. The system has two unque access methods for users to fnd desred vdeo shots. Frst, the system provdes a summarzaton nterface (Fgure 11a). It shows the statstcs of vdeo shots, dvded nto categores of long, ntermedate and short shots. It also shows the number of canoncal scenes n a specfc doman. For nstance, n tenns, these are serve, net-zone play, or commercal. Seeng these summares, users may follow up wth more specfc request by choosng a category (e.g., serve). As shown n Fgure 11b, users can choose to go to any nterestng scene categores drectly. The second nterface combnes the sequental temporal order and the herarchcal structure among vdeo shots wthn a vdeo. As shown n Fgure 12, the structure tree s lsted n the left wndow. In ths example, games are lsted at the top level. There are commercal breas between games. Under each game, there are many serves. Each serve contans the servng shots and a few follow-up shots. In the vdeo shown n Fgure 12, the frst game ncludes 16 serves. Each serve segment s labeled wth the length of the segment, type of play n ths serve and the approxmate number of stroes n a serve. For example, a label (L) S B 4 means a long segment, server, base-lne play and approxmately 4 stroes. All these elements are organzed as nodes n a tree. Ths allows users to easly navgate from top summary levels to detaled levels. When users clc on any nodes, the correspondng ey frame wll be shown n the rght wndow, and users can start to play the vdeo at the correspondng moment. 12

13 6. Concluson and Open Issues In ths paper, we present a general framewor for structure parsng and hgh-level event detecton for sports vdeos, usng tenns and baseball as examples. Our system combnes generc technques ( for feature extracton, object detecton/tracng, clusterng) and doman-specfc methods (for vew recognton and player event detecton). Real tme processng performance s acheved by explorng feature extracton and matchng n the compressed doman. Our experments have demonstrated hgh accuracy n sports domans such as tenns and baseball. Detecton of the structures and events n sports vdeo facltates development and deployment of new vdeo applcatons, such as hghlght generaton for long programs, lve flterng of mportant events etc. Under the framewor, there are many ssues that can be further explored to produce a accurate comprehensve summary of sports vdeos. For tenns vdeos, we can nclude audo nformaton to detect the number of stroes wthn each serve more accurately. Capton text showng the scores s common n broadcast sports programs. By detectng these text boxes and recognzng scores, we can recognze the status of a game. Readers are referred to [16] for an effectve system for sports event summarzaton by vdeo text recognton. References [1] S.-F. Chang, W. Chen, H. Meng, H. Sundaram, and D. Zhong, "VdeoQ: An Automated Content-Based Vdeo Search System Usng Vsual Cues", ACM 5th Multmeda Conference, Seattle, WA, Nov [2] M. Berlng, "Dsplacement Estmaton by Herarchcal Bloc Matchng", SPIE Vol 1001, Vsual Communcaton & Image Processng, [3] A. Hanjalc, R.L. Lagendj, J. Bemond, Automated Hgh-Level Move Segmentaton for Advanced Vdeo Retreval Systems, IEEE Transactons on Crcuts and Systems for Vdeo Technology, Vol.9, No.4, June [4] H. Sundaram and S.-F. Chang, Determnng Computable Scenes n Flms and ther Structures usng Audo Vsual Memory Models, ACM Multmeda 2000, Los Angeles, CA, Oct 30- Nov 3, [5] R. Lenhart, C. Kuhmunch and W. Effelsberg, On the Detecton and Recognton of Televson Commercals, Proc. of IEEE Internatonal Conference on Multmeda Computng and Systems, June, 1997, Ottawa, Canada. [6] T. Sato, T Kanade, E. K. Hughes, M. A.Smth, Vdeo OCR for Dgtal News Archves, Proc. Of the 1998 Internatonal Worshop on Content-based Access of Image and Vdeo Database, January 3, 1998 Bombay, Inda. [7] D.D. Saur, Y.-P. Tan, S.R. Kularn, and P.J. Ramadge, Automated Analyss and Annotaton of Basetball Vdeo, Proceedngs of SPIE's Electronc Imagng conference on Storage and Retreval for Image and Vdeo Databases V, Feb 1997, San Jose, CA. 13

14 [8] G. Sudhr, J.C.M. Lee and A. K. Jan, Automatc Classfcaton of Tenns Vdeo for Hghlevel Content-based Retreval, Proc. Of the 1998 Internatonal Worshop on Content-based Access of Image and Vdeo Database, January 3, 1998 Bombay, Inda. [9] H. Zhang, SY Tan, SW Smolar, and G. Yhong, Automatc Parsng and Indexng of News Vdeo, ACM/Sprnger-Verlag Journal of Multmeda Systems, 2 (6), pp , [10] D. Zhong and S.-F.Chang, "Vdeo Object Model and Segmentaton for Content-Based Vdeo Indexng", IEEE Internatonal Symposum on Crcuts and Systems, Hong Kong, June 9-12, [11] D. Zhong and S.-F. Chang, "Long-Term Movng Object Segmentaton and Tracng Usng Spato-Temporal Consstency, IEEE Internatonal Conference on Image Processng, Thessalon, Greece, October 7-10, [12] D. Zhong and S.-F. Chang, Structure Analyss of Sports Vdeo Usng Doman Models, IEEE Conference on Multmeda and Exhbton, Toyo, Japan, Aug , [13] Y. Ru, A. Gupta, and A. Acero, "Automatcally Extractng Hghlghts for TV Baseball Programs," the 8th ACM Internatonal Conference on Multmeda, Oct. 2000, Marna Del Rey, CA. [14] Y. Gong, L.T. Sn, C. Chuan, H. Zhang, and M. Saauch, Automatc parsng of TV soccer programs, Proc. ICMCS 95, Washngton D.C, May, 1995 [15] A. James and S.-F. Chang, Integratng Multple Classfers n Vsual Object Detectors Learned from User Input, 4th Asan Conference on Computer Vson (ACCV 2000), Tape, Tawan, January 8-11, [16] S.-F. Chang, D. Zhong, and R. Kumar, Real-Tme Content-Based Adaptve Streamng of Sports Vdeo, IEEE CVPR Worshop on Content-Based Access to Vdeo/Image Lbrary, Hawa, Dec [17] D. Zhang and S.-F. Chang, Event Detecton n Baseball Vdeo Usng Supermposed Capton Recognton, ACM Multmeda 2002, Juan Les Pns, France, December 1-6, (ACM MM 2002). 14

15 Vdeos Doman Knowledge Users Shot Search Feature Vdeo Objects Moton Features Stll Features Scene Detecton & Structure Parsng Vdeo Shots Audo/Text Feature Lbrary Indces and Structures Fgure 1. A typcal content based vdeo ndexng and retreval scenaro 15

16 Entre Tenns Game Set Game Serve Closeup Crowd Commercals Elementary Shots Fgure 2. The content structure of sports vdeo programs (example: tenns) 16

17 Fgure 3. Servng scenes from four dfferent tenns games 17

18 Tranng vdeo Supervsed learnng over generc features Domanspecfc object constrants Multple feature models New test vdeo Adaptve model selecton Object-level verfcaton Fne-level event analyss Fgure 4. system archtecture for sports vdeo structure analyss and event detecton 18

19 Vdeo regons (1) Iteratve moton layer detecton n ndvdual frames (2) Object refnement usng spto-temporal constrants at shot level Movng objects Fgure 5. Two-stage movng object detecton based on regon segmentaton and tracng results 19

20 (a) orgnal ey frame (b) regon segmentaton (c) movng object Fgure 6. An example of automatc regon segmentaton and movng object detecton (note the camera may be movng) 20

21 Fgure 7. Edge detecton wthn the detected court regon 21

22 Fgure 8. Applyng Hough-transform based lne detecton wthn specfc areas 22

23 Table 1 Vew detecton results (test vdeo overlaps wth tranng set) Ground truth # of Mss # of False Tenns (serve) Baseball (ptch)

24 Table 2 Vew detecton results (no overlap between tranng and test data) Ground truth # of Mss # of False Tenns (serve) Baseball (ptch)

25 Fgure 9. Tenns player tracng wthn a serve scene (results are shown on 8 I-frames wth 14 frames between every two I farmes) 25

26 turnng pont stll pont Fgure 10. Detecton of stll and turnng ponts n object trajectory 26

27 Table 3 Trajectory analyss results for one hour tenns vdeo # of Net Plays # of Stroes Ground Truth Correct Detecton False Detecton

28 (a) (b) Fgure 11. Summarzaton nterface provdng scene ndex to vdeo 28

29 Game 1 Serve 1, base-lne play, 2 stroes Key Frame Serve 2 Commercal Game 2 Fgure 12. Herarchcal Browsng Interface for Parsed Structures n Tenns Vdeo 29