Tracking Table Tennis Balls in Real Match Scenes for Umpiring Applications

Transcription

1 British Journal of Mathematis & Computer Siene 1(4): , 2011 SCIENCEDOMAIN international Traking Tale Tennis Balls in Real Math Senes for Umpiring Appliations K. C. P. Wong 1* and L. S. Dooley 1 1 Next Generation Multimedia Tehnology Group, Department of Communiation and Systems, The Open University, Milton Keynes, MK7 6AA, United Kingdom. Researh Artile Reeived 22 nd July 2011 Aepted 1 st Septemer 2011 Online Ready 15 th Septemer 2011 Astrat Judging the legitimay of tale tennis servies presents many hallenges where tehnology an e judiiously applied to enhane deision-making. This paper presents a purpose-uilt system to automatially detet and trak the all during tale-tennis servies to enale preise judgment over their legitimay in real-time. The system omprises a suite of algorithms whih adaptively exploit spatial and temporal information from real math video sequenes, whih are generally haraterised y high ojet motion, allied with ojet lurring and olusion. Experimental results on a diverse set of tale-tennis test sequenes orroorate the system performane in failitating onsistently aurate and effiient deision-making over the validity of a servie. Keywords: Video proessing; image segmentation; ojet detetion; ojet traking; tale tennis; umpiring; 1 Introdution Tale tennis math umpiring is a very demanding task, with the servie stage eing partiularly diffiult to judge eause so many oservations need to e made within a very short period of time. Aording to Delano (n.d.), up to 31 separate oservations are required to e heked within a one seond time window, with the umpire needing to deide upon the legitimay of a servie shortly after it is onduted. To ompound this prolem, some oservations relating to the height and deviation of the all rise are very hard for a human eing to judge. Setion of the laws of International Tale Tennis Federation (2008) state: "The server shall projet the all near vertially upwards, without imparting spin, so that it rises at least 16m after leaving the palm of the free hand and then falls without touhing anything efore eing struk." It is an espeially diffiult undertaking for a human to orretly and repeatedly assess the height and deviation of the all rise y mere visual inspetion. A more pragmati approah is to employ * Corresponding author: k..p.wong@open.a.uk;

2 omputerised tools apale of making aurate and fast measurements of the all rise, to aid the umpire in making orret deisions. An intuitive, non-disruptive way of evaluating the all rise is to apture every servie using a video amera and to then detet and trak in real time, the ojet of interest (OOI) i.e., the all, on a frame-wise asis. However, for a standard omputer system to aurately segment and trak in real time an OOI in math situations is extremely hallenging for a myriad of reasons, inluding: High ojet motion: The all travels fast. If the shutter speed of the amera is not suffiiently high, the ojet an eome lurred, olour faded and distorted in shape. Multiple moving ojets: Apart from the all, the players, tale-tennis ats and the rowd all exhiit different motion, so all detetion ased purely on motion is ineffetual other than for the simplest senes. Uneven lighting: Light soures are usually loated in the eiling, whih tends to make the upper portion of all appear righter than the lower part. Olusion: The all an e loked y the player, the at, lothing, or for other reasons, suh as its disappearane from view when it is thrown too high during a servie. Merging: When the ontrast etween the all and akground is low, the all may eome indistinguishale from the akground. Ojet Confusion: Bakground and foreground ojets whih have a similar olour, size and shape may e onfused with the all, suh as for example, with the irular haraters on the poster and the white all in the SIF (Soure Input Format) resolution Tale Tennis sequene in Figure 1. Small size: The size of the all is often only a few perent of the size of the frame, whih renders onventional histogram-ased detetion methods (An et al., (2010)) unsuitale. Time onstraint: As this is a real time appliation, the lateny inurred for deteting and traking the all must e minimised whih preludes omputationally intensive algorithms from eing adopted. While onsiderale literature exists in relation to general ojet detetion and traking, the orresponding literature relating to tale tennis appliations is limited. Desai et al. (2005) for instane, proposed a motion-ased multiple filter anks method for ojet traking whih was effetive in traking tale tennis alls, though the test sequenes used were relatively simple in nature omprising a plain akground and with the all eing the only moving ojet. Furthermore, no experimental results from an atual tale tennis math sene were presented. In ontrast, Chen and Zhang, (2006) used Kalman filtering and inremental Bayesian algorithms to trak the all in real math senes, and while the all was suessfully traked, the ojetive was the automati extration of game highlights rather than fast preise ojet detetion and traking and no orresponding auray evaluation was provided. Maggio and Cavallaro (2009) presented an alternative ojet traking algorithm ased upon a omination of Mean Shift searh and CONDENSATION in a partile filtering framework and a multiple semi-overlapping olour histograms ased target representation. Results revealed this tehnique was ale to detet the all in ertain frames of the SIF resolution Tale Tennis sequene used y the image proessing ommunity (Maggio and Cavallaro, 2009; Comaniiu et al., 2003), though auray was not suffiiently adequate for umpiring appliations. While all three approahes, to some extent, suessfully traked the all under presried onditions, reliale aurate and effiient ojet detetion and traking allied with low omputational overheads to enale real time proessing, remained a key ojetive. This provided 229

3 the motivation to investigate the development of a new system for fast and preise ojet detetion to assist umpires in making orret rulings on the all rise during the short servie phase of a tale tennis game. An innovative framework is presented to ahieve this aim whih exploits key spatial and temporal features of the OOI, with experimental results onfirming the effiay of the system in different tale-tennis math sequenes. The rest of the paper is organized as follows: Setion 2 disusses the underlying priniples of the all detetion and traking algorithms used in the new system. An experimental results analysis is then presented in Setion 3, with some onlusions eing provided in Setion 4. 2 Ball Detetion Method To address the all detetion hallenges desried in Setion 1, the proposed system employs various ojet detetion and analysis algorithms, whih are seletively applied from an initial ojet segmentation of a sequene to identify so-alled andidate alls. These algorithms exploit oth spatial and temporal information inluding ojet size, shape, olour, motion and predited trajetory to assist in the deision-making proess. 2.1 Ojet Segmentation The first stage in the detetion proess is to segment the andidate alls from a frame in the sequene. A andidate all is defined as an ojet that exhiits similar properties to the OOI, whih is the tale tennis all. Motion and olour-ased segmentation (Dooley and Karmakar, 2003) have een onsidered, though oth have fundamental drawaks. For example, as there are multiple moving ojets in a typial math sene, the motion of the all an often e diffiult to aurately estalish due to overlapping and motion-ased segmentation is also generally omputationally intensive. In ontrast, for olour-ased segmentation, lustering and thresholding tehniques were analysed (Image Proessing Toolox DEMOS, n.d.). The auray of the former heavily relies on otaining a good estimate of the initial numer of olour lusters. For math senes, seuring an aurate and reliale estimate of the numer of lusters is an intratale prolem as it an often vary etween frames. This provided the impetus to investigate for this partiular appliation, the use of a threshold-ased segmentation approah, espeially as it is straightforward to implement and omputationally effiient. Firstly, a inary differential image is reated where all pixels with a olour differene with the all less than a defined threshold are turned white, with those whose olour differene is greater than the threshold are turned lak. Neighouring white pixels are then merged together to form ojets (Shankar, 2008)). The segmentation auray however, is entirely dependent on the initially seleted threshold value, whih is very sensitive to noise whih makes automatially hoosing the most appropriate threshold diffiult. To overome this sensitivity prolem, a two-pass thresholding (TPT) method has een proposed y Wong (2009). 2.2 The Two-Pass Thresholding Method This tehnique signifiantly improved the preision of the threshold toleranes so enaling aurate OOI segmentation in high-resolution still images (Wong, 2009), while TPT has susequently een extended y Wong and Dooley, (2010) to enale OOI to e suessfully segmented in lower resolution sequenes. TPT proesses eah frame in two passes using two different thresholds. A oarse threshold is employed during the first pass so all pixels with a 230

4 dissimilar olour to the all are filtered. The remaining pixels form andidate all ojets in an analogous manner to the standard thresholding method. Due to the oarseness of the threshold used, the shape of a andidate all an e distorted, espeially around its ase due to light shading. The purpose of the first pass however, is to solely approximate andidate all loations. Any distorted andidate alls will e restored in the seond pass, where a more relaxed threshold is applied, ut only to those regions where in the original frame, andidate alls and their neighourhoods have een identified. The relaxed preision for oth thresholds is an attrative feature of the TPT tehnique. A poorly hosen threshold in the first pass for instane, only impats on the ojet size while ojet loations remain approximately onstant. In the seond pass, the regions of interest (ROI) are restrited to either one or more small areas where there is normally only one andidate all with a high ontrast akground. A less preise threshold therefore does not ompromise the overall segmentation performane. 2.3 Automati Tuning Of The Two-Pass Thresholds While the TPT method tolerates lower preision thresholds, it is still desirale for these to e automatially determined. This an e ahieved using a known referene loation for the all, whih is provided for example, y the user during the aliration stage. Wong and Dooley (2010) developed an iterative algorithm for tuning the thresholds, with the asi priniples eing summarised elow, where m, g, u and v are empirially defined onstants. In the first iteration, set the ROI for the urrent frame to m times the all diameter, where m is seleted to provide a larger ROI in order to tune the Pass 1 threshold. Apply TPT algorithm to segment the OOI. If multiple andidate alls (ojets) are produed in Pass 1, inrease the level of the threshold y u% for the next iteration and ontinue Pass 1 threshold tuning. If no andidate all remains after Pass 1, redue the threshold y u% and ontinue Pass 1 threshold tuning. If only one ojet remains after Pass 1, this is the desired threshold value and Pass 2 tuning ommenes. If the maximum numer of Pass 1 iterations is reahed and no suitale threshold is found, then selet the threshold that produed the minimum numer of andidate alls and start tuning the Pass 2 threshold. Calulate area differene etween the all (A ) and the ojet found (A o ) losest to the given all loation. If (A A o )/A lies within ±v% then set urrent threshold as the Pass 2 threshold and terminate tuning. Otherwise, if A o > A then inrease the Pass 2 threshold y g% in the next iteration, while if A o < A, then redue the Pass 2 threshold y g% and ontinue Pass 2 threshold tuning. If the maximum numer of Pass 2 tuning iterations is reahed with no suitale threshold eing found, selet the threshold with lowest (A A o ) value. 231

5 2.4 Adaptive Control Of The Region Of Interest As the OOI is very small ompared with the size of typial a frame ( 0.06%), searhing the entire frame for the OOI is omputationally very expensive. A more effiient strategy is to define a ROI where the proaility of finding the OOI is high. Furthermore, if the loations of the OOI in urrent and previous frames are known, its approximate loation in the next frame an e predited using extrapolation, and this predited loation then set as the entre of the ROI. As for the size of ROI, it is desirale to e made adaptive ased on the searh suesses. For example, when the OOI is not found in the urrent ROI, the size of the ROI needs to e enlarged. Likewise, when the OOI is loated within the urrent ROI, the size of the ROI an e minimised. Wong and Dooley (2010) proposed an adaptive algorithm to dynamially adjust the dimensions of the ROI and this has een emedded in the new system. For ompleteness, it is reprodued elow, where oth j and k are positive onstants whih are empirially determined during initialisation: In frame #1, set the size of ROI equal to the size of the frame. If the OOI is found, redue ROI for next frame to a small square of whih the length of the side is j times the diameter of the OOI If no OOI is found, then sale the length of the ROI in the next frame y k If the width (height) of the ROI is greater than the frame width (height), redue to the frame width (height). 2.5 Ojet Evalution Following segmentation, eah identified andidate all is analysed using a two stage evaluation. The first stage heks: i) if it has a rounded upper ontour (RUC), ii) its loation is onsistent with the predited loation (T), and iii) if it exhiits motion at oth its entre (M ) and the predited loation (M p ). The respetive parameters RUC, T, M and M p are formally defined in Tale 1. A andidate all must satisfy at least two of the aove four onditions in order to fulfil the first test, whose purpose is to oth eliminate andidate alls whih are unlikely to e the OOI, while onomitantly inluding andidate alls with imperfetions. For instane if the andidate all shape is distorted i.e., no RUC due to insuffiient lighting, ut the ojet is in the viinity of the predited loation and also exhiits motion, this andidate all passes the first test. The seond stage of the evaluation uses a set of spatial geometri measurements, whih are: area (A), maximum width (W), maximum height (H), perimeter (P), roundness (R) and the error funtion E RUC. These parameters are defined in Tale 2. An error funtion E an now e defined as: E = E RUC + w A A A A + w W W W H H + w H W H ( n + 1) n p + w p P P P + w R R R R (1) where w are the weightings applied to the spatial parameters, n p is the numer of spatial parameters, n is the numer of onditions a andidate all satisfied in the Stage-1 test, and A, W, H, P, R and A, W, H, P, R are the spatial parameters of the andidate all and atual all (OOI) respetively. 232

6 Tale 1. Summary of the spatial and temporal parameters used in the andidate all stage-1 test Rounded Upper Contour (RUC) Position (T) Motion (M ) Motion (M p ) RUC = 1 if E RUC < t RUC = 0 if E RUC t RUC where t RUC is a preset threshold and E RUC is an ojetive (error) funtion defined as: n d i r i = 1 r E RUC = N and d i = 2 ( x i x ) + ( y i y ) 2 where (x i,y i ) is a pixel in the upper ontour, i the pixel index, N the numer of pixels, r the radius and (x,y ) the entre otained y solving the equation of a irle for (x i,y i ). d i is the distane etween pixel i and the entre. T = 1 if L diff < t T = 0 if L diff t T where L diff is the Eulidean distane etween the OOI atual and predited loations and t T is a threshold. The predited loation is the linear extrapolation of OOI loations in previous frames. M = 1 if OC diff t M = 0 if OC diff < t M where OC diff is the Eulidean distane etween pixels at the entre of the andidate all and the OOI in previous frames and t M is a preset threshold. M p = 1 if OP diff t M = 0 if OP diff < t M where OP diff is the Eulidean distane etween pixels at the entre of the predited all loation and the OOI in previous frames. Tale 2. Summary of spatial parameters for the Stage-2 test Parameters Area (A) Maximum width (W) Maximum height (H) Perimeter (P) Roundness (R) Desription Ojet size (numer of pixels) Horizontal distane etween the left and rightmost pixels of the ojet. Vertial distane etween the top and ottom most pixels of the ojet Length of the ojet oundary Given y: 4 π A R = 2 P where 0<R 1 and R=1 is a irle. The andidate all with the smallest error E in (1) is then lassified as the OOI. The rationale for employing this 2-stage test is to diretly address the following three ojet detetion hallenges: 233

7 i. There are other ojets in the akground sene with similar geometri features and dimensions as the all. These may e mislassified as the OOI if only the Stage 2 test is applied. The heking mehanism for the predited loation and motion in Stage 1 prevents suh mislassifiation. ii. The shape of the andidate all is distorted due to insuffiient lighting, low frame rates or merging with other ojets. The andidate all may not then e deteted y the Stage 2 test alone. iii. The andidate all is either partially or fully oluded. The andidate all may not e orretly identified. Cheking the predited loation and motion in Stage 1 assists in deteting the OOI. 2.6 Blok Mathing Detetion While the OOI detetion method desried performs well under many varied lighting onditions, in ertain senarios, partiularly when proessing low ontrast sequenes, the thresholding-ased methods an fail. As a onsequene, a lok mathing (BM) detetion method has een inluded in the system, though this signifiantly inreases the omputational osts inurred. A sliding window (SW) is used to reursively proess the ROI on a pixel-y-pixel asis, and ompares the pixel-wise differene etween the urrent and referene loks so forming a BM error E BM defined as: E BM = W ROI H ROI i = 1 j = 1 ( pr ij prr respetively, and i and j are indies. ij ) 2 + ( pg W ROI ij H ROI prg ij 2 + ( p The referene lok is generated y extrating a lok of pixels from the largest inner square enlosed y the irumferene of the OOI, whih in 2D will e a irle. The size of the SW is exatly the same as the referene lok, so the OOI is onsidered to e the lok with the lowest E BM, provided it is less than an aeptale error level. The entre of the OOI is the entre of the lok. BM tehniques do have some drawaks. They are omputational expensive eause of the inherent searh overheads and are also unale to detet OOI whih are partially oluded. As a onsequene, BM is only applied in the system under speifi onditions, namely when an OOI is not identified using the TPT method and the ROI is minimised y the ROI adaptive ontrol algorithm desried in Setion Results Analysis To evaluate the performane of the new tale tennis all detetion system, a set of five test sequenes inluding a mixture of resolutions, and varying ojet motions were hosen to evaluate all the assorted detetion hallenges identified in Setion 1. Sequene 1 was extrated from the widely adopted SIF Tale Tennis sequene. The servie segment omprises 79 frames and ontains a numer of detetion hallenges inluding amera (gloal) and ojet (loal) motion, a lurred OOI due to the lower spatial and temporal resolutions and some ojet olusion. Sequenes 2 to 5 were aptured from atual math environments, though only the servie elements were analysed. The full harateristis and detetion hallenges of eah sequene are summarised in Tale 3. ) ij pr ij ) 2 234

8 Tale 3. Full harateristis and detetion hallenges of the five test sequenes Sequene 1 Sequene 2 Sequene 3 Sequene 4 Sequene 5 Tale Tennis ITTF Timo Boll Loal league High Throw No of frames Frame (pixels) 352 x x x x x 384 Capture rate 30 fps 30 fps 30 fps 100 fps 30 fps Ball size area (pixels) Ball Colour White Orange White White Orange Frame size Small Small Large Large Medium Variation in Ball size Large Medium Medium Large Medium Ojet motion High High High Low High Gloal (amera) High None Low None None motion Blurring of OOI 41 frames 7 frames 28 frames None 17 frames Merging No No No 2 frames 9 frames Olusion 12 frames No No 1 frame 17 frames Lighting Good Aeptale Good Dark Dark To ensure an equitale omparison, key system onstants were empirially determined during initialisation and maintained throughout all the experiments, with the only user inputs required eing the diameter, olour and loation of the all in frame #1 of a sequene. Numerial results showing the detetion rates and omputational times are presented in Tale 4, while the qualitative results for samples frames from eah test sequene are displayed in Figures 1 to 5 respetively. Tale 4. Numerial test results with the system parameters eing: j=3, m=2j, k=1.3, u=v=10, g=(100 A A o )/A and with the maximum numer of TPT tuning iterations eing set to 20 Sequene 1 Sequene 2 Sequene 3 Sequene 4 Sequene 5 No. of orret detetions No. of inorret detetions No. of undeteted 14 (2) (0) 25 (8) (frames with OOI olusion exluded) Detetion rate 81% (96%) 93% 93% 99% (100%) 36% (64%) (frames with OOI olusion exluded) Time for all rise (ses) 0.56 (7 frames) 0.55 (5 frames) 0.82 (8 frames) 1.08 (9 frames) 3.52 (12 frames) Additional test results inluding atual video outputs, the five test sequenes and further disussion an e found at The servie illustrated in Sequene 1 is in fat illegal, sine the all is not stationary on the player s palm efore the servie starts, and it is fully oluded y the player s hand for 12 frames. The reason for analysing this partiular sequene is eause it is widely used in ojet detetion researh and omparative results are availale. The sequene has a omplex textured akground 235

9 omprising ojets of similar shape and size to the OOI and exhiits gloal motion (amera zooming) together a numer of other detetion hallenges. The system suessfully deteted the OOI in 64 of the 79 frames and inorretly deteted an ojet as the OOI in only a single frame. Interestingly, neither the amera zooming nor the onfusing OOI-like ojets on the wall poster aused false OOI detetions, while suh features provided degraded detetion performane in the previous system (Wong and Dooley, 2010). Despite Sequene 1 not eing designed for tale tennis umpiring appliations, the system still gave an overall detetion rate of 81%, and if the frames where the OOI was oluded y the player during the servie are exluded from the analysis, the detetion rate was 96%. Some example frames are shown in Figure 1. The ROI is denoted y the yellow square; the red irles and rosses are the deteted OOI ontour and its entre respetively; and the green and lue irles are the loations of OOI derived using the mean shift and CONDENSATION algorithms (Maggio and Cavallaro, 2009) respetively. The detetion method proposed y this paper outperforms oth the mean shift and CONDENSATION algorithms, with only the omputationally expensive mean shift searh in a Partile Filtering framework omined with a multiple semi-overlapping olour histograms ased target representation (Maggio and Cavallaro, 2009), produing omparale results, whih annot e seen in Figure 1 eause they are overlapped y the red irles. Frame #78 shows where the OOI an e very diffiult to detet eause of fast amera zooming. This was not tested y Maggio, E., Cavallaro, A. (2009), yet the proposed method orretly deteted the all as evidened in Figure 1(e). All the loations of the deteted alls and the trajetory of the omplete servie are shown in Figure 1(f). The trajetory was formed y fitting a urve over the deteted all loations using ui-spline interpolation and was used to reover undeteted OOI loations. This trajetory was also used to estimate the height and deviation of the all rise. (a) Frame# 7 () Frame# 11 () Frame# 29 (d) Frame# 54 (e) Frame# 78 (f) Trajetory of the entire servie Figure 1: Sample proessed frames for Sequene 1 (a), () and () show the OOI ouning on the at; (d) the OOI at its zenith during the servie, (e) the penultimate frame of the servie, and (f) the full servie trajetory Sequene 2 was speifially designed for training purposes, so the amera is positioned to provide an umpire s view of a servie. Although the OOI, whih is orange in olour, appears small and sometimes lurred due to the low frame rate used, the system still ahieved a 93% detetion rate 236

10 with no inorret detetions. It is important to highlight that in a numer of frames light variations on the red-oloured tale tennis at produed a strong orrelation etween the olour intensities of the all and at (the merging prolem disussed in Setion 1). This was the reason for the unsuessful OOI detetion in just three frames, though when the all strikes the at in the final frame #45 of the servie and eomes lurred, the system still orretly deteted the OOI. Example frames from Sequene 2, overing various stages of the servie are shown in Figure 2. (a) Frame# 5 () Frame# 30 () Frame# 39 (d) Frame# 42 (e) Frame# 45 (f) Trajetory of the entire servie Figure 2: Sample proessed frames for Sequene 2 (a)-(e) respetively show the OOI on the palm, rising, at the highest point, falling and eing hit; and (f) the full servie trajetory Sequene 3 (ttcountenane, 2009) shows a servie onduted y a tale tennis player at an international math. (a) Frame# 1 () Frame# 7 () Frame# 14 (d) Frame# 26 (e) Frame# 30 (f) Trajetory of the entire servie Figure 3: Sample proessed frames for Sequene 3 (a)-(e) respetively show the OOI on the palm, rising, at the highest point, falling and eing hit; and (f) the full servie trajetory. 237

11 The sequene features a lose-up of the player, so the all appears to e large and in fast motion. As the apture rate is low, the OOI predominantly appears lurred in this sequene. There is also gloal motion as the viewing angle hanges and amera shake ours. Despite these detetion hallenges, the system ahieved a detetion rate of 93% with no inorret detetions. Figure 3 (a) (e) illustrates some example frames from Sequene 3, overing respetively where the all is on the palm of the hand, rising, at its zenith, falling and eing struk. Sequene 4 also shows an umpire s angle and view. This sequene was aptured at a sustantially higher frame rate and spatial resolution so the all appears to e travelling slower and is muh learer. However, the wide angle makes the OOI appear to e very small ompared to the frame size and high apturing rate makes the video appear darker due to the muh shorter exposure time used. When the OOI approahes the net, it eomes partially and then totally oluded, yet the system suessfully deteted the OOI in all frames, exept for one frame when the all was ompletely invisile. Example proessed frames for Sequene 4, overing where the all is on the palm, at the peak, eing hit, and examples of partial and full all olusion are provided in Figure 4. (a) Frame# 1 () Frame# 19 () Frame# 34 (d) Frame# 64 (e) Frame# 65 (f) Trajetory of the entire servie Figure 4: Sample proessed frames for Sequene 4 (a)-(e) respetively show the OOI on the palm, at the highest point, eing hit, ompletely oluded and half oluded; and (f) the full servie trajetory Sequene 5 represents a very hallenging senario, where the all has atually een thrown too high so it disappears for a signifiant numer of frames (17 out 39 frames) in the servie sequene. Additionally, the orange oloured all temporarily merges with either the palm or fae of the player in 9 of the frames. The height of the all rise of this servie far exeeds the rules and in this regard would e oserved y the umpire. However, the purpose for analysing this partiular sequene was to asertain the proessing limitations of the system in suh a hallenging detetion senario. The sequene was aptured using a high shutter speed so the OOI appears lear though the side effet is the video eomes darker. The OOI nevertheless still appears lurred in 17 frames as it falls at high speed and is struk. If the 17 frames where the OOI was not inside the frame (and hene annot e deteted) are exluded from the analysis, an overall detetion rate of 238

12 64% was attained. Figure 5 illustrates some sample proessed frames for Sequene 5, overing where the all is on the palm, rising, disappeared from the view, falling and eing hit. Despite the OOI not eing deteted for 25 frames, the missing all loations were still reasonaly estimated y the trajetory plot (using Cui-Spline interpolation) displayed in Figure 5(f). (a) Frame# 1 () Frame# 5 () Frame# 19 (d) Frame# 30 (e) Frame# 36 (f) Trajetory of the entire servie Figure 5. Sample proessed frames for Sequene 5 (a)-(e) respetively show the OOI on the palm, rising, disappeared from the view, falling and eing stuk; and (f) the full servie trajetory Finally, from a omputational effiieny perspetive, whenever the system was ale to trak the all, the adaptive ROI tehnique onstrained the OOI searh area to a square whose lengths were three times (j=3) the all diameter. When traking was lost however, the sides of the ROI were saled y 30% (i.e. k=1.3) and the time taken to reloate the OOI eame ommensurately longer. Depending on the numer of ojets within the ROI, the average time required for proessing a single frame in all experiments was approximately 100ms. In terms of OOI detetion and traking, to measure the all rise height of a servie Sequenes 1 to 4 inurred less than one seond to determine the all rise time, and as highlighted in Setion 1, this lateny is more than adequate to assist the umpire judge the legality of a partiular servie. In ontrast Sequene 5 inurred a proessing overhead of 3.52ses due to the high searh ost for frames where the OOI was not visile. While the omputational time for this sequene is onsideraly longer, pragmatially it would e aeptale as the high throw servie takes longer and the umpire would not require tehnologial aids to judge the legality of the all rise as it learly exeeded the height requirement. These results learly orroorate the system performane in aurately and effiiently deteting a tale tennis all from omplex real math sequenes involving a variety of amera angles, apture rates and math onditions, where oth ojet lurring and olusion are key hallenges to e effetively resolved. 239

13 8 Conlusion Many sports are inreasingly exploiting tehnology for verifiation purposes in key umpiring deisions. Tale-tennis has a myriad of diverse rules governing the legality of a servie and this paper has presented an aurate and effiient system for deteting and traking tale-tennis alls during the omplex high motion servie stage of a game. The system segments potential ojets into andidate alls prior to adaptively exploiting oth spatial and temporal information from real math videos to detet and trak the atual all. Experimental results on different test sequenes onfirm the system s onsistent performane in enaling fast and preise deision-making over the validity of a tale-tennis servie. It is important to stress that deteting ojets in a highly omplex environment and in real time, a purpose uilt detetion system, as demonstrated here, often outperforms more generi systems whih detet universal ojets. Furthermore, the algorithms developed are modular and flexile so they an e easily adapted to detet other ojets of interest suh as other all types, tyres on vehiles and earing or rollers on mahines. Referenes An, Y., Riaz, M, Park, J. (2010). CBIR ased on adaptive segmentation of HSV Colour spae. 12th International Conferene on Computer Modelling and Simulation, Marh Camridge, UK. Chen, W., Zhang, Y.J. (2006). Traking Ball and Players with Appliations to Highlight Ranking of Broadasting Tale Tennis Video. IMACS Multi-onferene on Computational Engineering in Systems Appliations. Volume 2, 4-6 Ot Page(s), , China. Comaniiu, D., Ramesh, V., Meer, P. Kernel-ased ojet traking, IEEE Trans. Pattern Analysis Mahine Intell. 25(5), Delano, L.F. (n.d.). Level 1 Seminar for Umpires. ITTF Umpires and Referees Committee. Aessed 24/06/2009. Desai, U.B., Merhant, S.N., Mukesh, Z., Ajishna, G., Purohit M., Phanish H.S. (2005). Small ojet detetion and traking: Algorithm, analysis and appliation. Leture Notes in Computer Siene. Springer. Berlin ISSN Dooley, L.S., Karmakar, G.C. (2003). A Fuzzy Rule-ased Colour Image Segmentation Algorithm. IEEE International Conferene on Image Proessing (ICIP 03), pp I Barelona. Sept Image Proessing Toolox DEMOS (n.d): Colour-Based Segmentation Using the L*a** Color Spae. Image Proessing Toolox 6.4. The MathWorks In, exfari.html. Aessed on 24/06/

14 International Tale Tennis Federation (2008). International Tale Tennis Handook 2008/ Aessed on 24/06/2009. ITTF Umpires and Referees Committee (n.d.), Umpires training videos. Aessed on 10/01/2007. Maggio, E., Cavallaro, A. (2009). Aurate appearane-ased Bayesian traking for maneuvering targets. Computer Vision and Image Understanding, 113, Shankar, J. (2008). Color-Based Segmentation with Live Image Aquisition. Matla Image Proessing Toolox weast, The MathWorks In. Aessed on 24/06/2009. ttcountenane (2009). Europe Top 12: Timo Boll-Vladimir Samsonov. [video online] Availale at:< Aessed 21 Deemer Wong, K.C.P. (2009). Identifying tale tennis alls from real math senes using image proessing and artifiial intelligene tehniques. International Journal of Simulation Systems, Siene & Tehnology, 10(7), Wong, K.C.P., Dooley, L.S. (2010). High-motion tale tennis all traking for umpiring appliations. In: IEEE 10th International Conferene on Signal Proessing, Ot Beijing, China Wong & Dooley; This is an Open Aess artile distriuted under the terms of the Creative Commons Attriution Liense ( whih permits unrestrited use, distriution, and reprodution in any medium, provided the original work is properly ited. 241