Player Number Recognition in Soccer Video using Internal Contours and Temporal Redundancy

Transcription

1 Player Number Recognition in Soccer Video using Internal Contours and Temporal Redundancy Matko Šarić, Hroje Dujmić, Vladan Papić, Nikola Rožić and Joško Radić Faculty of electrical engineering, Uniersity of Split R. Boskoica bb, Split, Croatia Abstract: - Player identification is important task in sport ideo content analysis. In case of soccer ideo player numbers can be exploited to perform recognition. Jersey numbers can be considered as scene text and main problems in localization and recognition are low resolution, low contrast, color aliasing, unconstrained background etc. Tis paper proposed new metod for player number localization and recognition. Firstly jersey regions are extracted using region growing algoritm. Tese regions are segmented using te component of HSV color space tat best separates jersey and number area. Ten, noel metod for player number localization based on internal contours is proposed. False number candidates are discarded based on area and aspect ratio. Extracted numbers are enanced using image smooting and rotation normalization. We also propose algoritm tat improes OCR recognition performance using temporal redundancy. Key-Words: - player number recognition, soccer ideo, HSV color space 1 Introduction Automated content-based ideo analysis of sports eents is researc area wit ig commercial potential. Annotation of ideo stream wit respect to teir semantic content allows efficient searcing and retrieal of ideo material. Many efforts in sports ideo annotation ae been made on important eents (igligts) detection like goals or sots on goal for soccer ideo. Te compreensie surey of sportsrelated indexing and retrieal can be found in [1]. Text in ideo images is classified into caption text, wic is artificially oerlaid on te image, or scene text, wic exists naturally in te image. Player numbers can be classified as scene text and recognition of tis kind of text is generally more difficult compared wit te general text detection. Problems are caused by ariation of orientation, size, position, illumination, focus etc. A surey of image and ideo text information extraction can be found in [2]. Regarding player number recognition we can cite Ye et al. [3]. Tey proposed jersey number detection metod were image is segmented using generalized learning ector quantization algoritm. Pixels are clustered into limited color-omogeneous regions in order to separate te jersey number from its background. Size and pipelike attributes of digital caracters are used to filter te candidates. Number recognition is preformed using K- NN classifier and Zernike moment features. In [4] algoritm proposed by Viola and Jones ([5]) is used for player numbers recognition. Tis algoritm is originally intended for face detection and it is based on cascade of classifiers. In [6] two-stage approac is proposed for player number detection: first step is extraction and selection of image corners using te Harris detector and second step is extraction of Maximally Stable Extremal Regions. Tese steps segment image in order to process it wit OCR software. Region adjacency grap and picture trees are exploited by Andrade et al. ([7]) to perform object searc using prior knowledge. Region analysis is furter applied to te candidate regions to isolate player number wic are ten processed wit OCR. In [11] same autors increase recognition count using te fact tat multiple instances of te same player number exist across a number of frames. We propose new metod for player number recognition. First step is searcing for regions of interest, i.e. jersey regions using region growing algoritm. Tese areas are segmented using component of HSV color space tat best separates number and jersey. Regarding teir spatial relation player numbers can be extracted from te segmented jerseys as internal contours. False candidates are ten filtered using area and aspect ratio. Image smooting and rotation normalization are employed before OCR processing. In order to enance recognition performance we propose algoritm tat uses temporal redundancy, i.e. te fact tat same number must exist across a certain number of frames. Rest of te paper is organized as follows. Section 2 discusses jersey region extraction and segmentation.. Section 3. describes number candidates extraction using internal contours. Algoritm tat improes recognition performance based on temporal redundancy is described ISSN: ISBN:

2 in section 4. Results are presented in section 5. Conclusions are made in section 6. 2 Jersey Region Extraction And Segmentation Segmentation of te wole ideo frame in order to find player number often results wit ery complex outcome. Prior knowledge about object (jersey and number color) can be used to reduce problem complexity. First step in our approac is jersey region extraction using region growing algoritm [12]. Criterion for seed point selection is Euclidean distance between frame pixel and jersey color in RGB color space. To be added to te jersey region, a pixel s color sould be close enoug to te color of te seed point: All extracted regions are potential player jerseys (figure 1b) and tey ae to be segmented in order to extract numbers. Rest of ideo frame is discarded from furter processing. We propose segmentation metod tat exploits prior knowledge about team colors. Number and jersey usually significantly differ in one component of HSV color space to make number clearly isible in different conditions. Because of tat segmentation is performed by tresolding tat component. Tis procedure clearly separates jersey and number (figure 1c) as wite object (jersey) wit ole (number). Spatial relation between tese two regions allows number extraction wit internal contours detection. Segmentation is described as follows: 1. Using sample image define mean colors for jersey and j = j j j and number (denoted by te HSV ectors [ s ] = [ n ns n ] pixel = [ pixel pixels pixel ] n ). Pixel color is denoted by 2. Create binary image: a) Number as acromatic color and jersey as cromatic color-segmentation by saturation tresolding if n < λ or n < λ and j > λ and j > λ (( s s ) ( )) ( ( s s ) ( )) ( < λ ) ( < λ ) if pixel and pixel s s ( _ (, ) < _ ) if diff j pixel ue tres ( ) _ diff j, pixel ( j pixel ) if ( j pixel ),, < 180 = o 360 j pixel oterwise b) Jersey as acromatic color and number as cromatic color- segmentation by saturation tresolding (( s < λs ) ( < λ )) ( s > λs ) ( > λ ) if j or j and n and n ( > _ ) if ( pixel < λ ) and ( pixel < λ ) if j V tres s s if ( pixel < V _ tres) ( < λ ) ( < λ ) if pixel and pixel s s if ( pixel > V _ tres) c) Number and jersey as cromatic color- segmentation by ue tresolding ( ( s > λs ) ( > λ )) ( ( s > λs ) ( > λ )) if n and n and j and j ( _ (, ) < _ ) if diff n pixel ue tres d) Number and jersey as acromatic colorsegmentation by alue tresolding ( ( s < λs ) ( < λ )) ( ( s < λs ) ( < λ )) if n or n and j or j ( > _ ) if ( pixel < V _ tres) if j V tres o ISSN: ISBN:

3 ( < _ ) if pixel V tres First step is system initialization tat can be done after obsering few seconds of ideo. HSV color space is applied to a wide range of applications since it clearly separates cromatic and intensity information, but use of tis color space as tree drawbacks: 1) ue is meaningless wen te intensity is ery low 2) ue is unstable wen te saturation is ery low 3) saturation is meaningless wen te intensity is ery low. Tus, our segmentation algoritm depends on mean saturations j and n of jersey and ( n and j ) and intensities ( ) s s number region. In cases a) and b) segmentation is performed by saturation tresolding. In case a) segmentation is additionally refined by separating jersey from background using ue difference. In case b) intensity difference is used for same purpose. According to [2] saturation tresold is set to 0.2 because tis alue proed to be good tradeoff between color loss and matcing of similar acromatic colors. Difference tresold for ue and intensity are empirically set to 20 o and 0.5. In case c) most important cue to separate jersey and number is ue and image is segmented using ue difference. Case d) coers situation were jersey and number are distinguised by intensity difference. Typical example is black number on wite jersey. 3 Number Region Extraction Image segmentation algoritm described in preious section produces few segmented frame regions tat are candidates for player jerseys. Spatial relation between number and jersey, tat is fact tat number is region surrounded by jersey region is exploited for number extraction. Jersey is represented as wite object wit black "ole" (number) and because of tat number candidates can be extracted by internal contours detection (figure 1d) in preiously segmented regions. Internal contours are found using algoritm implemented in OpenCV library [9]. In order to filter out non-number candidates area and aspect ratio of number bounding rectangle are used. Area is useful in eliminating small oles produced by segmentation process. Since it is assumed tat player number often occurs in close-up sots we set area tresold to 0.06% of image area. Number bounding rectangle usually as eigt is larger tan widt. According to tis fact eigt/widt ratio tresold is set to 1.1. Finally remaining bounding a) b) c) d) e) f) Figure 1. a) Original image b) Jersey region candidates c) Segmented jersey d) Internal contours e) Number candidates after area and aspect ratio filtering f) Number region cut from segmented image rectangles (figure 1d) are used to extract number regions from segmented image (figure 1e). To increase recognition performance enancement of extracted number regions is required before OCR processing. OCR software is sensitie to caracter rotation. To sole tis problem we used direction property of number region. Tis kind of region ISSN: ISBN:

4 descriptor makes sense in elongated regions only. Elongation is defined as ratio between te lengt and widt of te region minimum bounding rectangle and most often number region can be considered as elongated region. If te sape moments are known, direction (teta) can be computed as defined in [10]: * tan µ 11 θ = 2 µ 20 µ 02 were µ 11, µ 20 and 02 µ are central contour moments. Ten number region is rotated θ angle. Except rotation normalization number regions are also enanced using median filtering. Two digit numbers are represented as two numbers wit spatially adjacent bounding rectangles. In order to correctly extract number distance between rectangles is cecked. If tis distance is low enoug, two rectangles are framed togeter as new rectangle representing two digit number. 4 Temporal Redundancy In order to enance OCR recognition performance we exploited temporal redundancy, tat is fact tat same number is present across te certain number of frames. We propose algoritm tat process OCR results and detect certain player number only if it is repeated certain number of times. OCR results are saed in text file in wic one line contains numbers found in one ideo frame. Basic procedure is described as follows: count array containing number counts last last frame in wic number appears t1-tresold for number accepting t2-distance tresold for eac line number=find number in text line count [ number ] ++ last [ number ] ==frame_counter if ( count [ number ] >t1 ) && ( frame_counter-last [ number] ) for i=1 to100 frame_counter++ output number count number =0 last number =0 ( ) last [ i ] =0 count [ i ] =0 <t2 if frame_counter-last i >20 Our algoritm confirms OCR recognition of certain number wen it appears more tan t1 times. Distance in frames between two consecutie detections must be lower tan t2 oterwise number counter is set to 0. Tis approac reduces number of false positie recognition (number is recognized wen tere is none or number is recognized wrongly). Tresolds t1 and t2 are empirically set to 5 and Results We ae tested proposed metod on dataset composed of 2 soccer ideo clips in 640x480 and 640x352 resolution at 25fps. First clips contains frames (18:51 minutes) and second one frames (20:06 minutes). In tis dataset player numbers in close-up sots appear 31 times troug 1116 ideo frames in wic number was detected by uman obserer. Two measures are used for metod ealuation: recall and precision. Tey are defined as Number of correctly recognized numbers Recall = Number of player numbers Number of correctly recognized numbers Pr ecision = Number of recognized player numbers Results in table 1 are referred to ideo frames containing player number witout using temporal redundancy. Test set contains 1116 frames sowing player numbers. Results in table 2 are referred to sots containing jersey numbers, i.e. ere we exploited fact tat same number appear across te certain number of frames. Test set contains frames wit 31 sots sowing player number. TABLE I RECOGNITION PERFORMANCE IN VIDEO FRAMES (NOT USING TEMPORAL REDUNDANCY) Team Correct False Missed Recall Precisi on A % 97% B % 96% C % 78% D % 67% All % 87% ISSN: ISBN:

5 TABLE II RECOGNITION PERFORMANCE IN VIDEO SHOTS (USING TEMPORAL REDUNDANCY) Team Correct False Missed Recall Precisi on A % 100% B % 100% C % 60% D % 50% All % 71% Table 1. sows tat tere are lot of missed numbers. Tis type of error is caused by non-rigid deformation of caracters, blurred caracters (moe of players or camera), low contrast, ariation in illumination etc. All tese circumstances are ery often found in our frame set because we select all frames containing number regardless of image quality. Significantly better results are obtained for precision because of lower number of false positie detections. Results in table 2. sow tat use of temporal redundancy significantly improes recall rate (from 30% to 65%). Precision decrease is caused by fact tat test set for performance in sots contain not only frames wit numbers but all frames from ideo clips. We found 4 papers dealing wit player number detection and recognition ([3],[4],[6],[7]). In [4] and [6] results referred to sots are presented. Our approac aciees better recall and precision tan metod described in [4] (Recall=55%, Precision=55%). In comparison wit [6] (Recall=67%, Precision=84%). our metod as similar recall and weaker precision, but it must be empasized tat in [4] and [6] number recognition is performed only wen face is detected wile our test set includes all frames. Tis means tat our system is more rigorously tested on false positie detections were number is present wen tere is none Tests in literature are usually performed on datasets made of selected frames. In order to get objectie metod ealuation we include all frames from sots wit player number (recognition in frames), tat is all frames from entire ideo material (recognition in sots). Because of tis difference in datasets it is ard to directly compare our results wit results presented by oter autors. 5 Conclusion A metod for recognition of player numbers in soccer ideo based on temporal redundancy is proposed in tis paper. Number region localization is performed in noel way: jersey region are extracted using region growing algoritm and segmented by tresolding component of HSV color space tat best separates number and jersey area. Tis allows us to extract number regions by internal contours detection. Non-number regions are filtered out using area and aspect ratio of teir bounding rectangles. Before OCR processing numbers are enanced using rotation normalization and median filtering. We also propose algoritm tat exploits temporal redundancy and significantly increases recognition recall. Tis approac reduces negatie effects of nonrigid deformation, blur, low contrast etc. and. ACKNOWLEDGMENTS Tis work was supported by te Ministry of Science and Tecnology of te Republic Croatia under projects: ICT systems and serices based on integration of information ( ) and Computer Vision in Identification of Sport Actiities Kinematics ( ) REFERENCES [1] Kokaram, A. Rea, N. Dayot, R. Tekalp, M. Boutemy, P. Gros, P. Sezan, I., Browsing sports ideo, Signal Processing Magazine, IEEE, Vol 23, issue 2, Marc 2006 page(s): 47-58, [2] K. Jung, K.I. Kim and A.K. Jain, Text information extraction in images and ideo: a surey, Pattern Recognition,Volume 37, number 5, May 2004, pp [3] Q. Ye, Q. Huang and S. Jang. Jersey Number Detection in Sports Video for Atlete Identification. In Proc. of VCIP, July 2005 [4] Bertini, Marco, Bimbo, Alberto Del and Nunziati, Walter (2005), "Player identification in soccer ideos", MIR '05: Proceedings of te 7t ACM SIGMM international worksop on Multimedia information retrieal: [5] P. Viola and M. Jones. Rapid object detection using a boosted cascade of simple features. In Proc. of CVPR, pages , [6] Bertini, M.; Bimbo, A.D.; Nunziati, W., Matcing Faces wit Textual Cues in Soccer Videos, Multimedia and Expo, 2006 IEEE International Conference on Volume, Issue, 9-12, July 2006 Page(s): [7] E. L. Andrade Neto, E. Kan, J. C. Woods and M. Ganbari, Player Classification in Interactie Sport Scenes Using Prior Information Region Space Analysis ISSN: ISBN:

6 and Number Recognition, IEEE International Conference on Image Processing (ICIP) 2003, September 2003, Barcelona, Spain, ol. III, pp [8] Bertini, M., Bimbo, A. Del, Torniai C., Grana, C., Vezzani, R., Cucciara, R., Sports ideo annotation using enanced s istograms in multimedia ontologies, Image Analysis and Processing Worksops, ICIAPW t International Conference on, page(s): , Sept [9] Open Source Computer Vision Library, [10] Sonka, Hlaac, Boyle: Image Processing, Analysis, and Macine Vision, Tomson-Engineering; 3 edition (2007) [11] E. L. Andrade Neto, J. C. Woods and M. Ganbari, Outlier Remoal for Player Identification in Interactie Sport Scenes Using region Analysis, WIAMIS [12] R. M. Haralick and L. G. Sapiro, Image Segmentation Tecniques, Computer Vision, Grapics, and Image Processing, Vol. 29, No. 1, pp , Jan ISSN: ISBN: