A scheme for racquet sports video analysis with the combination of audio-visual information

Transcription

1 A sheme for raquet sports video analysis with the ombination of audio-visual information Liyuan Xing a*, Qixiang Ye b, Weigang Zhang, Qingming Huang a and Hua Yu a a Graduate Shool of the Chinese Aadamy of Siene, Beijing, China b Institute of Computing Tehnology, Chinese Aademy of Sienes, Beijing, China Shool of Computer Siene and Tehnology, Harbin Institute of Tehnology, Harbin, China ABSTRACT As a very important ategory in sports video, raquet sports video, e.g. table tennis, tennis and badminton, has been paid little attention in the past years. Considering the harateristis of this kind of sports video, we propose a new sheme for struture indexing and highlight generating based on the ombination of audio and visual information. Firstly, a supervised lassifiation method is employed to detet important audio symbols inluding impat (ball hit), audiene heers, ommentator speeh, et. Meanwhile an unsupervised algorithm is proposed to group video shots into various lusters. Then, by taking advantage of temporal relationship between audio and visual signals, we an speify the sene lusters with semanti labels inluding rally senes and break senes. Thirdly, a refinement proedure is developed to redue false rally senes by further audio analysis. Finally, an exiting model is proposed to rank the deteted rally senes from whih many exiting video lips suh as game (math) points an be orretly retrieved. Experiments on two types of representative raquet sports video, table tennis video and tennis video, demonstrate enouraging results. Keywords: Sports video analysis, raquet sports, audio-visual ombination. 1. INTRODUCTION Sports video is one of the most popular video programs all around the world. People reate an inreasing amount of digitized sports video day by day. It is neessary to find effetive ways for aessing these programs by their ontent. On the other hand, more and more people aess sports video using TV set-top box, PC, PDA, or even mobile phone, so it is important to extrat the valuable ontent in a sports video for saving both the user s time and downloading osts. Reently, sports video ontent analysis for automati semanti indexing and ontent summarization has been a hot researh topi. In the past years, lots of work has been done on sports video analysis and understanding. In 1, Adami et. al gives an overview of the present situation in sports video analysis. We an see that muh attention has been paid to soer and baseball, and both single signal (audio, visual, text) and multi-model have been used for their struture analysis and highlights extration. As far as raquet sports is onsidered, only tennis has been mentioned in the review. However, only visual information has been used for reognition of strokes based on HMM (Hidden Markov model) in 2 and deteting fundamental views using supervised learning and domain-speifi rules in 3. To the best of our knowledge, several approahes 4, 5, 6 have been proposed for event detetion and struture analysis in tennis video using multi-model information. In 4, visual features whih haraterize the type of shot view and audio features whih desribe the audio event within a video shot are merged by an HMM. In 5, the authors use visual features to apture the essene of sene geometry and audio features to identify the sound of the ball hitting. Visual and audio reognition results are ombined by a likelihood approah for high level ontent analysis. In 6, Xu et. al lassify video shots into several major lasses and among whih the lassifiation of the different audio signal segments is performed. Then rules are made on the audiovisual lassifiation result to detet exiting events. As we an see, although there are some works done on raquet sports video, the researh is mainly following the work on field sports video (mainly soer video) and there is still no general sheme for different kinds of raquet sports video * Further author information: send orrespondene to Liyuan Xing, Qixiang Ye, Weigang Zhang, Qingming Huang, {lyxing, qxye, wgzhang, qmhuang}@jdl.a.n; Hua Yu, yuh@gsas.a.n Visual Communiations and Image Proessing 2005, edited by Shipeng Li, Fernando Pereira, Heung-Yeung Shum, Andrew G. Tesher, Pro. of SPIE Vol (SPIE, Bellingham, WA, 2005) X/05/$15 doi: /

2 analysis. In this paper, we try to provide a general solution to analyze the raquet sports video by fully explore its harateristis on audio, visual features and their temporal relationship. The rest of the paper is organized as fallows. In setion 2, we outline the framework on raquet sports video analysis based on its audio and video harateristis. The details of supervised audio lassifiation and unsupervised video lassifiation are presented in setion 3. Rally detetion and ordering are desribed in setion 4. Experimental results are illustrated in setion 5 and onlusion of the paper with a disussion of future work is provided in setion SYSTEM OVERVIEW Different from field sports, raquet sports has no distint exiting event e.g. shot on goal event in soer video. A raquet sports video onsists of the best of any odd number of games, and eah game is made up of many story units whih are alled rallies in this paper. Eah rally is the period during whih the ball is in play, and a rally of whih the result is not sored is a let. Unless the rally is a let, a player shall sore a point. Play will be ontinuous throughout a math exept that any player is entitled to stop. Rally by rally ontent makes the raquet sports video have well defined temporal struture both in audio and visual information. Typially the audio trak of the raquet sports ontains impat, ommentator speeh, audiene heers, whih are mixed with eah other or muh noise from the situation. The audio information is more related to the state of the raquet sports, for example, the impat ourrene usually indiates the ball is playing, and the heers or exited ommentator speeh ourrene often indiates the end of the rally and the happen of some exiting events. Audio is a valuable and robust feature for raquet sports ontent analysis if we an identify some distint sound in math. Meanwhile, the broadast of a raquet sports is produed by a fixed number of ameras at fixed plae in the ourt for its given task. Then, there is muh similarity in the sene that made by the same amera. If we an group the video shots of similar ontent into same luster, the temporal video struture of a math will be lear. But pure visual feature is not suffiient for parsing the raquet sports struture sine it is quite diffiult to learn supervised lassifiation model for different kinds of video ontent. The goals of our work are raquet sports video struture parsing and highlight ordering. The experiments are arried out both in table tennis and tennis. Figure 1 is the proposed system framework. First, a supervised lassifiation method is used to detet important audio symbols suh as impat, audiene heers or ommentator speeh. In the mean time, an unsupervised sene ategorization algorithm is employed to group video shots into various lusters whih have not been speified with the semanti lusters labels. Then, by taking advantage of domain knowledge that rally senes have a strong relationship with obvious audio symbols, for example, impat sound always happens in the ourse of rallies, we identify semanti information of the video sene lusters by audio lassifiation results. After a refinement proedure for removing false rally senes through further audio analysis, rally and break segmentation ould be ahieved. Finally, the deteted rally senes are ranked by pre-defined exiting model, and exiting moments suh as game (math) points an be orretly retrieved from those ranked rally senes. Video program Demultiplex Supervised lassifiation Impat Cheer Speeh... Unsupervised Shot luster-1 lassifiation Shot luster-2 Shot luster-3... Sene reognize Rally senes Break senes Refinment Rally-break segmentation Ranking Ordered rally Figure 1: System framework 260 Pro. of SPIE Vol. 5960

3 3. AUDIO AND VIDEO CLASSIFICATION In this setion, we introdue the details on how to obtain the audio symbols and video lusters by supervised and unsupervised lassifiation, respetively Supervised audio lassifiation Generally speaking, there are several kinds of representative sound in raquet sports video inluding heer, applause, speeh, impat, and so on. In real onditions, it is diffiult to reognize these sound beause of the variane of different sound effet in different mathes. And some sound may be mixed with others in a math, whih makes them diffiult to be well lassified with a probabilisti method. Furthermore, sometimes the duration of sound is quite different in different sports video. We take some measures in the follows for the audio lassifiation task Feature extration and seletion Aording to the type of frame-level features on whih they are based, four groups of lip-level features are extrated, inluding both time-domain and frequeny-domain features. Short-time energy(ste) based features ontain the mean value of STE(1), the standard deviation of STE(2), the low STE ratio(3), the high STE ratio(4); Zero-rossing rate(zcr) based features are onsists of the mean value of ZCR(5), the standard deviation of ZCR(6), the low ZCR ratio(7), the mean value of different ZCR(8), the standard deviation of different ZCR(9); The only pith based feature is the mean value of pith(10); And the others are frequeny based features, whih are the mean value of brightness(11), the standard deviation of brightness(12), the mean value of bandwidth(13), the standard deviation of bandwidth(14), Spetrum Flux(SF,15), the mean value of sub-band power(16-19), the standard deviation of sub-band power(20-23), the mean value of LPCC (24-31), the standard deviation of LPCC (32-39), the mean value of MFCC(40-47), the standard deviation of MFCC(48-55). Espeially, the four sub-bands over the frequeny interval of 0HZ-fs/8HZ, fs/8hz-fs/4hz, fs/4hz-3*fs/8hz, 3*fs/8Hz-fs/2Hz. All these features are ommonly used for audio lassifiation and the detailed desriptions are in 7 and 8. We extrat a total of 55 features from a lip. Although all these features an be used to distinguish audio, some features may ontain more information than others. Using only a small set of the most powerful features will redue the time for feature extration and lassifiation. Furthermore, the existing researh has shown that when the number of training samples is limited, using a large feature set may derease the generality of a lassifier 9. Therefore, feature seletion is performed before they are fed into the lassifier. To selet the effetive features, SVM lassifier, whih will be introdued in the next setion, is employed to do the lassifiation evaluation. Kernel funtion for SVM is Radial Basis Funtion (RBF) with parameter gamma=1/dim. A forward searh algorithm 9 is used to perform the feature seletion task. Figure 2 and Table 1 are the orresponding experimental results. And the SVM ross validation is performed with three sets of randomly seleted training data and testing data. Figure 2 shows the performane urve in the feature seletion proess. We find that with the inrease of dimension of feature, the auray inreases sharply at first but slightly dereases when the number passes ertain value (17 dimensions) whih is seleted as the feature dimension. The seleted features are listed in the seond olumn in Table 1. It an be seen that some of the time-domain and frequeny-domain features are seleted, whih shows that both time domain and frequeny domain an reflet the signal. None of the Sub-bands features in the frequeny based features is seleted. It means these features have little disriminating power. In the following, training and lassifiation are performed on the 17 seleted features. Table1. Features used for audio lassifiation Feature set Features seleted Number of features Number of features seleted Energy based features 1,2,4 4 3 ZCR based features 6,8,9 5 3 Pith based features 1 0 Frequeny based features 11,15,25,26,34,36,45,46, ,54,55 Total The numbers in seond olumn represent the speifi features. Pro. of SPIE Vol

4 Figure 2: The feature seletion proess of 1s lip length Training and lassifiation We find that the performane of SVM-based lassifier is muh better than that of the KNN and GMM in audio lassifiation and segmentation in this task. Compared with other lassifiers, SVM is easier to train, needs fewer training samples and has better generalization ability. Besides, sine we annot define the probabilisti distribution funtions of the samples, we use SVM lassifier in our work. The ability of SVM working on small training samples will redue the hard labor on training sample marking. In the training and lassifiation proess, to determine the best sound lip length for samples we design a proedure based on SVM lassifiation evaluation. In the proedure, sound lassifiation results of different lip length are ompared on the urve in Figure 4. The point at whih the best lassifiation performane is obtained is seleted as sound lip length. For example, in the experiment, one seond is the best sound lips length for sound lassifiation in table tennis Unsupervised video shot lassifiation Sports video programs are normally aptured by a limited number of ameras in ertain loations. Most raquet sports videos are omposed of a restrited number of typial senes and video shots in eah type of senes have similar video ontent. Then, we propose a merging method to luster video shots of similar video ontent into same video luster 10. This method is generi and does not require any domain knowledge about the type of the proessed sports video. It performs in an unsupervised manner and relies on the sene similarity analysis of the shots in the video Merging In the merging proess, eah shot is represented by 5 key frames. Color histogram (256 dimensions) in HSV olor spae is employed as the low level features of these frames. We use the Eulidian distane between two shots (or senes) to represent their sene similarity. The smaller the distane is, the more similar the two shots (or senes) are and the higher their sene similarity is. Every time the two most similar shots are merged into a sene. This proedure is repeated until the merging stop riterion is satisfied before the merging proess stops to obtain the sene lustering results Stop riteria To determine the number of lusters, the stop riterion is defined based on a J value whih is defined aording to the Fisher Disriminant Funtion. J = K l = 0 J J t w = K N l = 0 i= 0 N i= 0 r s r s i i r s r s mean mean (1) 262 Pro. of SPIE Vol. 5960

5 where J t is the total inter-luster satter of the initial sene sequene, J w is the intra-luster satter of sene luster. N is the total sene number in the initial sene sequene, N is the shot number of sene luster. represents the Eulidean distane. s r i ( s r i ) denotes shot i in sene luster (the initial sene sequene), and s r mean ( s r mean ) denotes the mean feature value of shots in sene (the initial sene sequene). The value of J represents the total sene luster satter, whih desribes the ratio of intra-luster satter to inter-luster satter of the senes in the merging proessing. Atually, it is expeted that J value is small and the sene number is small. While in real situation, the smaller the sene number is, the large the J value will be. As a tradeoff between J value and the sene number, we hoose a point where the sum of the J value and the ratio of the sene number to the total number of senes in the initial sene sequene in the merging proess is the smallest, and take this one as the best merging stop point. Experiments show that the rally sene whih we are most is merged well. Figure 3 is a shot lustering demonstration in table tennis video. (a) (b) Figure 3: Video shot lassifiation. (a) Keyframes of shots in real video sequene.(b) Keyframes of shots lusters 4. RALLY DETECTION AND RANKING In sports video ontent analysis, the high-level semantis are always defined as the exiting events or highlights. By onsidering what people are most in the raquet sports, the first thing we think is that they hope to browse the video just like they read a book using the atalog, so it is neessary to struture the raquet sports into rallies. Another thing is that they expet to wath the most exiting parts of the math, so ranking the segmented rallies should be done Audio/visual ombination for rally-break segmentation When we onsider parsing the raquet sports struture into rallies, better results an be obtained by ombining the audio and visual information. In the previous steps, video shots have been merged into lusters. But we do not know the exat semanti of these video sene lusters. It is probably a seletive method that we use the domain olor to reognize the rally lasses. But this method is not general for all types of raquet sports video. On the other hand, the sound lasses deteted by audio detetion may not be aurate. But the sound lasses have strong semanti meaning and the sene lusters have good temporal boundary. In order to solve both the problems utilizing the good aspets, we seek help to the inevitable relationship between audio and visual information Sene reognition by audio symbol By observing eah of the video sene lusters, we find that there must be a sound lass oupling with eah video sene luster very well. For example in table tennis video, the impat sound mathes best with red field sene luster. Thus we an obtain the meaning of eah video sene luster by the audio symbol oupling best with it. And then we an oarsely segment a sports video into rally-break senes. The proedure is desribed as follows. Pro. of SPIE Vol

6 Supposing K sound lasses and N video sene lusters are obtained. Let m be the number of time slots and belong kn to the k-th sound lass that is distributed in the n-th sene luster. Suppose that k-th sound lass has M time slots and t ki is within the time domain of the i-th time slot. Then m kn = M i= 0 δ [ f ( t ) n] (2) n ' = arg n max( mkn ) (3) where f (t) stands for the result of video lustering, and f ( t) [1,2,... N], So n' is the exat math of k. For example, when the k-th sound lass is the Ping-Pong impat, the orresponding n' -th sene luster is the rally sene. After we know whih the rally luster is, the oarse rally event an be deteted Refinement by audio symbol Beause some replay or some asually senes may be similar with the rally sene, it is inevitable that the reall of rally event is high, however the preision may be a little lower. Then audio rules are employed to remove those rally senes suh as replay or some asually senes. We should make a tradeoff between the preision and the reall of the rally event to make some improvement. Some results of the refined rally event detetion are shown in Table 2 and Table 3, setion 5.3. The heuristi deision rules for rally event detetion in the refinement proessing are fully desribed in the following. RULE1: IF there is an impat in the time sope of oarse rally THEN judge whether at the end of the oarse rally there is exited speeh or heer IF it is THEN it is a rally ELSE it is not RULE2: IF there is no impat in the time sope of oarse rally THEN judge whether at the end of the oarse rally there is exited speeh or heer and whether at the beginning of the oarse rally there is plain speeh or heer IF both are satisfied THEN it is a rally ELSE it is not 4.2. Rally ranking by exitement The deteted rally event an be output as video ontent summarization. Furthermore, these rallies an be ranked by their exiting degrees. When we onsider highlight ordering, the last time of the heer and the pith of the ommentator speeh have very strong relationship with exitement. Long time heer and high pith are the natural response of the audiene and ommentator immediately after highlight. Also the duration of a rally is an important fator when we enjoy the math. It seems people prefer to wathing the long time onfront aording to our observation. By this analysis, we an define the exiting degree G of a rally by a linear model rally Grally = wduration Tduration + wheer Eheer + wspeeh P (4) speeh where w is the importane weight, T represents the duration of a rally, duration E represents the relative energy of the heer heer lips after the rally, P is the pith of speeh after a rally. The higher the speeh G value is, the more exiting the rally orresponding rally will be. 5. EXPERIMENT RESULTS The database used in our experiment is omposed of seonds table tennis videos whih are from totally 11 different living broadast programs of Athens Olympi 2004, and 7761 seonds tennis videos whih are from totally 4 different living broadast programs of Wimbledon These videos are ompressed by MPEG-2, digitized at 25 frames/s (PAL) and have a resolution of The audio signal is sampled at 48000Hz and 16 bits/sample. The audio stream is segmented into lips that are experimented in different time length with no overlapping with the previous ones. Eah lip is then divided into frames that are 512 samples long and are shifted by 256 samples from the previous frames. The lip is used as the lassifiation unit. In table tennis, lips of 957 seonds that omes from 4 different videos ki 264 Pro. of SPIE Vol. 5960

7 are used for audio training, and lips of 9579 seonds that omes from other 7 different videos are used for testing. In tennis, lips of 567 seonds that omes from 1 video is used for audio training, and lips of 7194 seonds that omes from other 3 different videos are used for testing Performane of different lip length for SVM lassifiation In order to determine the length of the lip for our feature analysis, the testing lip length is from 0.01s to 1.5s with different interval. It is beause 0.01s is already orresponding to the samples of a frame, so it is the minimum, and as there is no muh audio suh as impat lasting for a long time, we find 1.5s as the maximum is proper. In the experiment, 0.2s, 0.5s, 1s and 1.5s are seleted for training lip length to see the trend of performane with inreasing training lip length. The results of different aspets are illustrated in Figure 4. It an be seen that in () there is no muh differene, but in (a) and (b) they are in the ontrary sequene. It is reasonable that when the preision is higher, the reall may be lower. In the following proess of ombining, the preision of the impat is more important. So in (a), the highest one that is 1s should be seleted as the training lip length. The 1s line reahes the maximal value when the testing lip length is 1s, so both the training and testing lip length with 1s is the best. (a) (b) () Figure 4: (a) the preision of the impat (b) the reall of the impat () the total preision of the total five lasses 5.2. Rally-break segmentation Seven different table tennis videos with the total time of 9579 seonds are used for rally testing. The results are listed in Table 2. It an be seen that the rally event deteted only by audio lassifiation results is not satisfatory. This is beause the impat sometimes is not easy to detet when it is interfered by the ommentator speeh or other sound. However, there is muh improvement after audio/visual ombination is applied, with the preision and reall reahing 88.93% and 98.01%, respetively. After refinement, the reall dereases 3.48%, but the preision inreases 4.03%. The rally detetion results of 3 different tennis videos with the total time of 7194 seonds are listed in Table 3. It an be seen that the preision of rally event by oarse ombination is a little low, beause in tennis some senes with field for player medium view are very similar with rally sene. So the refinement after ombination is more important, and when the reall dereases 3.9%, the preision inreases 13.1%. Pro. of SPIE Vol

8 Table 2: Rally event detetion results in table tennis by audio only, audio/visual ombination and refinement Rally event By audio only By audio/visual ombination Refinement after ombination Video lip preision reall preision reall preision reall A(7:12) 15/22=68.2% 15/20=75% 20/27=74.1% 20/20=100% 20/22=90.9% 20/20=100% B(8:18) 14/16=87.5% 14/15=93.3% 15/16=93.8% 15/15=100% 15/15=100% 15/15=100% C(27:40) 55/77=71.4% 55/84=65.5% 84/89=94.4% 84/84=100% 81/85=95.3% 81/84=96.4% D(33:57) 51/87=58.6% 51/73=69.9% 70/79=88.6% 70/73=95.9% 68/74=91.9% 68/73=93.2% E(11:08) 11/15=73.3% 11/28=39.3% 27/30=90.0% 27/28=96.4% 26/29=89.7% 26/28=92.9% F(36:38) 75/109=68.8% 75/98=76.5% 93/106=87.7% 93/98=94.9% 84/94=89.4% 84/98=85.7% G(34:46) 40/49=81.6% 40/93=43.0% 92/98=93.9% 92/93=98.9% 87/93=93.5% 87/93=93.5% Average 72.77% 66.07% 88.93% 98.01% 92.96% 94.53% Table 3: Rally event detetion results in tennis by audio only, audio/visual ombination and refinement Rally event By audio only By audio/visual ombination Refinement after ombination Video lip preision reall preision reall preision reall A(40:13) 76/98=77.6% 76/111=68.5% 101/145=69.7% 101/111=91% 94/117=80.3% 94/111=84.7% B(48:59) 97/142=68.3% 97/123=78.9% 104/150=69.3% 104/123=84.5% 101/120=84.2% 101/123=82.1% C(30:42) 73/105=69.5% 73/97=75.3% 87/121=71.9% 87/97=89.7% 84/98=85.7% 84/97=86.6% Average 71.8% 74.2% 70.3% 88.4% 83.4% 84.5% 5.3. Rally ranking Sine there is no absolute ground truth for rallies exiting ranking, subjetive method is employed to evaluate the performane of the proposed ranking model. We think it is appropriate that the game (math) points are ranked by the linear model in the top 20% of the whole math, and the other rallies are in the order whih is also aeptable. 6. CONCLUSION AND FUTURE WORK In this paper, we propose a general sheme for raquet sports video ontent analysis inluding rallies detetion and ranking. The general approahes on supervised audio lassifiation and unsupervised video lassifiation make the method work well on different raquet sports video. The proposed rally segmentation method and ranking model is more reasonable than traditional sports video analysis method by fully exploring the harateristis of raquet sports video. In future work, more sophistiated audio/visual information ombination model will be developed and more reasonable rally ranking method an be explored. ACKNOWLEDGEMENT This work is partly supported by NEC Researh China on Context-based Multimedia Analysis and Retrieval Program and Siene 100 Plan of Chinese Aademy of Sienes. This work is a part of SPISES: SPorts Video Summarization and Enrihment System at REFERENCES 1. N. Adami, R. Leonardi, P. Migliorati, An overview of multi-modal tehniques for the haraterization of sport programmes, in Pro. SPIE - VCIP 03, pp , 8-11 July, Lugano, Switzerland, M. Petkovi, Z. Zivkovi, W. Jonker, Reognizing strokes in tennis videos using hidden markov models, in Pro. IASTED Int. Conf. Visualization, Imaging and Image Proessing, Spain, D. Zhong and S.-F. Chang, Struture analysis of sports video using domain models, IEEE Conferene on Multimedia and Exhibition, Tokyo, Japan, Aug , E. Kijak, G. Gravier, P. Gros, L. Oisel, F. Bimbot, HMM based struturing of tennis videos using visual and audio ues, in Pro. Intl. Conf. on Multimedia and Exhibition, R. Dayhot, A. Kokaram, and N. Rea, Joint audio-visual retrieval for tennis broadasts, in Pro. ICASSP, Pro. of SPIE Vol. 5960

9 6. M. Xu, L-Y. Duan, C.-S. Xu, Q. Tian, A fusion sheme of visual and auditory modalities for event detetion in sports video, in Pro. ICASSP, Lie Lu, Hong-Jiang Zhang, Stan Z. Li, Content-based audio lassifiation and segmentation by using support vetor mahines, Multimedia Systems 8: , Y. Wang, Z. Liu, and J. Huang, Multimedia ontent analysis using both audio and visual lues, IEEE Signal Proessing Magazine, 17(6):12--36, A.K. Jain, Statistial pattern reognition: a review, IEEE Trans. PAMI, 2, 4-37, Weigang Zhang, Qixiang Ye, Liyuan Xing. Qingming Huang and Wen Gao, Unsupervised sports video sene lustering and its appliations to story units detetion, in Pro. SPIE - VCIP 05, July, Beijing, China, Pro. of SPIE Vol