Intelligent Video Display to Raise Quality of Experience on Mobile Devices

Transcription

1 Intellgent Vdeo Dsplay to Rase Qualty of Experence on Moble Devces Changck Km* a, Jaeseung Ko a, Ilkoo Ahn a, Muhammad Usman a, Jae Hoon Kwon b, Jeongrok Park b, Young Hun Joo b, and Yun Je Oh b a Vsual Informaton Processng Laboratory, Informaton and Communcatons Unversty (ICU), Daejeon, Korea b Multmeda Laboratory, Telecommuncaton R&D Center, Telecommuncaton and Network Sector Samsung Electroncs Co., Ltd. ABSTRACT Moble devces have been transformed from voce communcaton tools to advanced tools for consumng multmeda contents. The extensve use of such moble devces entals watchng multmeda contents on small LCD panels. However, the most of vdeo sequences are captured for normal vewng on standard TV or HDTV, but for cost reasons, merely reszed and delvered wthout addtonal edtng. Ths may gve the small-dsplay-vewers uncomfortable experences n understandng what s happenng n a scene. For nstance, n a soccer vdeo sequence taken by a long-shot camera technque, the tny objects (e.g., soccer ball and players) may not be clearly vewed on the small LCD panel. Thus, an ntellgent dsplay technque needs to be developed to provde small-dsplay-vewers wth better experence. To ths end, one of the key technologes s to determne regon of nterest (ROI) and dsplay the magnfed ROI on the screen, where ROI s a part of the scene that vewers pay more attenton to than other regons. 1 In ths paper, whch s an extenson from our pror work, we focus on soccer vdeo dsplay for moble devces, and a fully automatc and computatonally effcent method s proposed. Instead of takng generc approaches utlzng vsually salent features, we take doman-specfc approach to explot the attrbutes of the soccer vdeo. The proposed scheme conssts of two stages: shot classfcaton, and ROI determnaton. The expermental results show that the proposed scheme offers useful tools for ntellgent vdeo dsplay for multmeda moble devces. Key words: Intellgent dsplay, Moble devces, Regon of nterest, Soccer vdeo analyss 1. INTRODUCTION The prolfc spread of moble technologes made t possble for TV vewers to watch vdeo any-tme, any-where. The emergng moble vdeo servces nclude DVB-H (Dgtal Vdeo Broadcastng-Handheld), T-DMB (Terrestral-Dgtal Multmeda Broadcastng), S-DMB (Satellte-DMB) and so on [1]. For nstance, DMB s currently beng put nto use n a number of countres. South Korea, n partcular, started S-DMB and T-DMB servce n May 1 and December 1, 2005 respectvely. However, mostly for cost reasons, most of the servces offer materals drectly reszed for moble consumpton. The experments reported n [2], amed to assess the mage resoluton and bt-rate requrements for dsplayng ths reszed materal on moble devces, ndcate that the prmary effect of reducng mage resoluton s a loss of vsual detal. Accordng to the expermental results from [2], the effect becomes more severe when the vdeo content type s lke feld sports, especally soccer vdeo, whereas becomng less severe n musc, news, and anmaton order. *ckm@cu.ac.kr; phone ; vega.cu.ac.kr/~ckm Applcatons of Dgtal Image Processng XXIX, edted by Andrew G. Tescher, Proc. of SPIE Vol. 6312, 63120N, (2006) X/06/$15 do: / Proc. of SPIE Vol N-1

2 Thus, for some vdeo content types, t s needed to provde an ntellgent dsplay scheme so the vewer of the small moble dsplay has a magnfed vew of a regon of nterest (ROI). The ROI can be defned as a porton of multmeda contents that users show more nterest or a part of the scene that vewers pay more attenton to than other regons. Determnng ROI benefts n the applcatons of context-aware content adaptaton, automatc browsng of large mages [3], transcodng, ntellgent nformaton management, and so on [4]. In addton, t can be a frst step to semantc level nterpretaton of a vdeo scene. There have been several schemes for determnng ROI. In [5], Itt et al. proposed a vsual attenton model, n whch several spatal vsual features are combned nto a sngle salency map to represent local conspcuty, but applcatons to movng pctures s not well studed. Cheng et al. [4] proposed a ROI determnaton framework for vdeo sequences based on ntensty, color, and moton. Ther goal s to locate the salent poston n the mage frames, assumng that the key subjects are exactly the most salent objects wth hgh color or ntensty contrast. Another approach for analyzng vsual attentons s proposed n [6], where user attenton models are presented for vdeo skmmng and summarzaton. They exploted audo-vsual features of semantcs, such as moton, speech, camera operaton, and lexcal nformaton. Fan et al. mplemented auto-zoomng functonalty for small dsplays usng those features of general vdeo [7]. Whle the abovementoned salency-orented methods are shown to be effectve n detectng ROIs, we note that the approaches are not sutable for the ntellgent dsplay of some feld sports vdeos, especally the soccer vdeo. That s because there exst mage frames that do not requre a magnfed vew. Also, there may exst evenly salent multple tny objects n a scene and thus multple ROIs mght be detected when the above salency-orented methods are used. Soccer vdeo s one of the most popular contents. However tny objects n the soccer vdeo prevent vewers from comfortable watchng on moble devces. Loss of object detals and shot type are some of major reasons. There must be some lower lmt to the resoluton not to lose detals. In ths paper, whch s an extenson from our pror work [8], we present an ntellgent dsplay technque to deal wth soccer vdeos only wth mproved performance. Instead of takng salency-orented approach nto account, we take doman-specfc approach, where the unque attrbutes of the soccer vdeos can be utlzed. In ths scenaro, we observe that there are two types of frames n the soccer vdeos: long-shot frames and others. The long-shot frames are mages captured n a long dstance, thus the szes of the ball and players are tny when they are vewed on a small dsplay (see Fg. 1-(a)). In ths case, t would be much better to understand the scene f a ROI s extracted and provded on an enlarged scale. For the other type of frames, whch may be ether md-shot frame or close-up frame, t s noted that vewng the orgnal whole mage s suffcent to understand the contents (see Fg. 1-(b) and (c)). Thus, for comfortable dsplay on a small LCD panels, the frames need to be classfed nto two categores, and only a part of whole mage frame needs to be extracted for the long-shot frames. Ths proposed soccer vdeo dsplay scheme conssts of two folds; shot classfcaton, and ROI determnaton. Compared to our pror system [8], t excludes ground color learnng stage, whch requres addtonal tme n the begnnng of a sequence. In the soccer vdeos, t s noted that the ground color plays an mportant role n separatng the ground domnant shots from others. Snce the ground colors are slghtly changng vdeo to vdeo, the ground color learnng stage s requred n the begnnng frames of the soccer vdeo. Once the ground domnant color range s decded, decson of the shot class becomes the second module, followed by ROI determnaton. (a) (b) (c) Fg. 1. Three knds of frame n soccer vdeo. (a) Long-shot frame, whch requres a magnfed vew of ROI. (b) Md-shot frame. (c) Close-up shot frame. The rest of ths paper s organzed as follows. Secton 2 presents the proposed algorthm for ntellgent dsplay of soccer vdeos on a small LCD panels. Secton 3 shows the expermental results and the concludng remarks follow n secton 4. Proc. of SPIE Vol N-2

3 2. PROPOSED ALGORITHM Before descrbng the proposed algorthm, some defntons need to be defned. Frst of all, a vdeo s defned as a sequence of shots, n whch a shot s a sequence of mages captured by a sngle camera recordng. Vdeo = Shot 0, K, Shot K 1 (1) where K s the number of shots n the vdeo. Also, the k-th shot can be denoted by a sequence of frames, as shown n (2). where 0 N 1 Shotk = fk,k, fk (2) f s the -th frame of Shot k and N the number of frames n the shot. We wll denote the -th frame n the whole k vdeo as f. ROI of the -th frame n the k-th shot s also denoted as ROI(f k ). Accordng to the shot class, ROI may correspond to entre mage frame or a part of t, as expressed n (3). ROI( fk) fk, f Class( Shotk) s a long-shot ROI( fk) = fk, f Class( Shotk) s not a long-shot (3) Ths means that ROI takes a small porton n a frame for a long-shot case, whereas ROI s determned dentcal to the orgnal mage frame otherwse. The block dagram of the proposed algorthm s shown n Fg. 2 Stage 1: shot classfcaton Stage 2: ROI determnaton Soccer Vdeo Constructng ground block map Pxel-based segmentaton Shot boundary detecton Ball detecton Long shot? N Dsplay whole frame Y ROI determnaton Dsplay ROI Fg. 2. Block dagram of the proposed algorthm 2.1 Detectng Ground Pxels Our goal n ths ground pxel detecton s to partton f nto ground area, whch wll be denoted by G, and remanng regons, expressed by G c. Obvously, a pxel (x,y) must belong to ether G or G c. When we defne a bnary array Ground(x,y), t s obtaned by takng two dfferent cases nto account n terms of ntensty value I, where I s defne as the mean value of r, g, and b; Proc. of SPIE Vol N-3

4 ((85g> 95 b) AND (95g> 85 r)) AND 1f ( g r) + b> 30 AND, Case 1: Ixy (, ) > 50, Groundxy (, ) = ( I < 150) 0 otherwse, (4) 1f ((85g > 95) b AND (95g > 85)) r, Case 2: Otherwse, Ground( x, y) = (5) 0 otherwse, where ndex (x,y) for r, g, b, and I s omtted for smplcty. As expressed n (4) and (5), we dvde the ground pxel detecton algorthm nto two cases: one s to deal wth normal ground color and the other wth shadowed ground color. Eq. (4), to deal wth normal ground color, s utlzed only f I(x,y) s greater than a predefned threshold, 50 n ths paper. Otherwse, (5) s used to detect ground pxels on the shadowed areas. The frst condton n (4) reflects the observaton on the ground pxels that g>r and g>b. After conductng enough observatons, the relatonshp has been slghtly modfed. The second condton n (4) s set to exclude unform color that s smlar to the ground color, and lastly the thrd condton s to prevent a ball and lnes from beng detected as ground pxels. The latter two condtons are not used n the second case, or shadowed ground pxels. Although the ground colors are slghtly dfferent vdeo by vdeo, the condtons derved by observng a lot of soccer vdeos works well as shown n Fg. 3. Unlke the algorthm proposed n our pror paper [8], where the ground color learnng stage s requred n the begnnng stage, ths new scheme shows more robust and faster performance wthout nvolvng the learnng stage. Also the proposed scheme s faster than those proposed by Ekn [9] or Wan [10], and performs well even n the case where a part of ground s shadowed as stated. (a) (b) (c) (d) Fg. 3. (a), (c) frames wth shadowed areas. (b), (d) Detected ground area (black) by usng the proposed scheme. 2.2 Constructng Ground Block Map (GBM) In the prevous sub-secton, we gave a decson whether or not each pxel belongs to ground. In order to have a prompt shot boundary detecton and shot class decson, the decson s performed block-wse, rather than pxel-wse. The whole frame s parttoned nto 16x16 blocks to generate a ground block map GB(,j). To determne f a block GB(,j) s a ground block or not, we perform a test expressed n (6). 1(ground block) f Ground( x, ) = 1 (, ) = c y GB j c (6) 0 (non - ground block) Otherwse. where ( x c, yc ) denotes a pxel poston located n the center of each block. By usng the center pxel only n determnng the attrbute of a block, the processng tme for constructng the ground block map s much shortened at the expense of a neglgble degradaton of the performance. Fg.4-(c) denotes the result of GBM constructon. Even f t s hghly lkely to contan more holes n the ground area than that from [8], the hole are easly flled by usng followng process Proc. of SPIE Vol N-4

5 GB(, j) = 1 f [ GB( 1, j) = 1 and { GB( + 1, j) = 1 or GB( + 2, j) = 1}] or [{ GB( 2, j) = 1 or GB( 1, j) = 1} and GB( + 1, j) = 1] (7) Ths s to fll the holes n the ground. If there exst one- or two-block gaps vertcally, they are flled to be a ground block. (a) (b) I S U (c) (d) Fg. 4. Comparson of Intal ground block maps. (a) Intal GBM from [8],(c) Intal GBM from the proposed algorthm, (b) and (d) hole-flled GBM from (a) and (c), respectvely. 2.3 Shot Boundary Detecton The shot boundary detecton s a fundamental process n the vdeo analyss. By gvng a decson of shot classes at shot boundares only, we can reduce the computaton complexty and the number of false decsons, and thus mprove the accuracy rate n shot class decson. Many approaches have been proposed to detect the shot boundares [11]. We explot the temporal block dfference (TBD) between ground block maps to make prompt and effcent shot boundary detecton. TBD = { GB 3( x, y) G B( x, y)} (8) x y where denotes Exclusve OR operaton. The current frame s determned as a boundary between shots when both TBD -1 < θshotchange and TBD θshotchange are satsfed. ( θ ShotChange = 30 n ths paper.) For the fast changng scene, the condton TBD θshotchange tends to be often satsfed, thus the second condton TBD -1 < θshotchange s combned to reduce the false detectons. Note that the current ground block map s compared wth that of three frames before to deal wth gradual changes due to fade n/out or other specal vsual effects. 2.4 Shot Class Decson Whenever shot boundary s detected, the shot class decson should be done. To ths end, we use ground block map to dstngush the long-shot frame from other types of frames. Once the holes are flled, the longest green segment for the -th block column, or LGS s searched. LGS s shown n Fg. 5. Now, we conduct shot classfcaton by measurng the length of LGSs nsde the golden secton defned n 9. As shown n Fg. 5, f there s at least one LGS smaller than predefned value, θ L, then the frame s determned as a non-long-shot frame. Otherwse, the frame s declared as a long-shot frame. Non-long-shot, f LGS < θl for GSLeft GSRght Class( fk ) = Long-shot, otherwse where θ L =BlocksInColumn/3, GSLeft=BlocksInRow 3/11, and GSRght=BlocksInRow 8/11. (9) Proc. of SPIE Vol N-5

6 GoId con (a) (b) Fg. 5. Parttonng a frame nto LGS s for shot class decson. (a) Long-shot. (b) Non-long-shot. 2.5 Segmentaton n Pxel Accuracy Each frame s classfed nto ether long-shot class or non-long-shot class after shot class decson accordng to (9). For the non-long-shot class, t s not necessary to determne ROI smaller than the whole mage frame. It s thought that the whole mage frame can provde a suffcent vew to vewers. However, ROI needs to be determned and magnfed to provde more comfortable vew for long-shot frames. We beleve that soccer vewers are more nterested n the area around the ball, thus wll take the locaton of the ball nto consderaton wth more mportance. We mantan a bnarzed map, whch s just an nverse of the Ground(x,y) expressed n (4) and (5), to detect the locaton of the ball. 0 f Ground( x, y) = 1 BnaryMap( x, y)= (10) 1 otherwse Fg. 6-(a) shows the bnarzed map. Note that non-ground blocks are not used n ths process. Then 8-connectvty Connected Component Labelng s conducted to get the lst of the object n the frame (see Fg. 6-(b)). In Fg. 6-(b), the Mnmum Boundng Rectangles (MBRs) are shown n yellow. Each object has attrbutes, such as MBR s aspect rato, average Value, number of pxels. Accordng to these attrbute, each object s categorzed nto ball-canddates, players, and others. For example, an object s categorzed nto a ball f the number of pxels n the MBR s 4 to 20, aspect rato 1.0 to 1.5, and the MBR possesses the ntensty peak of the frame. p;-- --=.- tt fl!! (a) Fg. 6. (a) Bnarzed mage. (b) Objects bounded by ther MBR. 2.6 Ball Detecton Although we have a lst of objects and dscrmnaton rules, t s not easy to determne whch object s the actual ball by usng one frame only. Because there are many ball-lke objects n the ground such as socks of players and fragments of lne segments. Moreover the ball may have shape dstorton or be occluded by players. Even some vdeo frames do not contan the ball. There have been many schemes to track the ball and players [12-14]. A trackng method proposed n [12] assumes that ntal ball poston s already known. In [13], they use soccer vdeo taken by a fxed camera whch s not the general case. The proposed algorthm n [14] requres future frames to obtan the ball trajectory and thus s not sutable for real-tme applcatons. We suggest a smple and causal method whch can be used for real-tme processng. Frst, we assume that the longest tracked ball-canddate has the hghest probablty to be a ball. Ths strategy can mnmze the nfluence of sudden nose. We mantan ball-canddates lst and keep addng the newly found ball-canddates (b) Proc. of SPIE Vol N-6

7 to t. Each ball-canddate has ts own age. In the next frame, ball-canddates are succeeded by the closest object n terms of both spatal dstance and attrbutes. If a successor has an attrbute of a ball, t s kept n the lst and ts age ncreases. On the contrary, f a successor does not have a smlar attrbute of a ball, ts age n the lst s decreased. We choose the oldest canddate as the most probable ball canddate. An object whose age s less than zero s removed from the lst. Wth ths scheme, we can detect the ball wth hgh accuracy. If there exsts no ball n the frame or t s faled to fnd t, ts locaton n the prevous frame s just coped and used. If fndng the ball s faled over three consecutve frames, the center pont of the frame s set to be the locaton of the ball. Snce t s easly noted that the poston of the ball does not have bg changes frame by frame, we employ a two-step search, where the ball canddate s searched n the frst search area, centered at the ball s locaton n the prevous frame (see Fg. 7). Only f there s no ball canddate n the area, the remanng area s searched. Ths two-step search yelds 19.31% mprovement compared to the case of entre frame search. Note that the area for the frst search s set to be same as the ROI wndow sze n ths paper, but may be reszed to reduce the processng tme. Thewdthofframe The ball posfon of prevous frame Fg. 7. Two step search to reduce computaton n ball detecton 2.7 Decson of ROI Wndow 1st BalI Search 2nd BaIl Search The smple and reasonable way to place the ROI wndow s to locate a ball and make the locaton as the center pont of ROI. However, placng the ball at the center of ROI through the consecutve frames would provde shaken and too-fast-movng ROI wndow to the vewers. In other words, the ROI wndow needs to be moved smoothly to provde comfortable vews to the small-dsplay-vewers. In addton, an acceleraton scheme s requred to deal wth a case where a ball s movng very fast throughout the frames. The solutons are provded n ths sub-secton. Once a frame s classfed as a long-shot frame by usng the shot class decson scheme, the center pont of the ROI wndow s ntalzed wth the locaton of the ball (LOB). wndow = lob (11) dsp = 0 (12) where wndow denotes the center pont of the ROI wndow n the -th frame, and lob the locaton of the ball detected. dsp denotes the dsplacement of the ROI wndow and s exploted to calculate the locaton of the new ROI wndow. It s ntalzed as zero for the frst frame n a long-shot. For the followng long-shot frames, the wndow s locaton s determned referrng to the dstance between lob and wndow -1 as expressed n (13). dff = lob -wndow -1 (13) For smplcty, let us consder the stuatons wth moton n the rght drecton and dff 0 wthout losng generalty. In ths stuaton, we may have three cases as follows. 1) 0 dff dsp wndow = wndow 1 (14) dsp 1 α + dsp (0 α 1) (15) Proc. of SPIE Vol N-7

8 2) dsp < dff frame. wdth /2 3) frame. wdth /2< dff wndow = wndow + dsp (16) 1 dsp = 1 dsp + a + 1 sgn( dff ) (17) wndow = wndow + dsp (18) 1 dsp = 1 dsp + a + 2 sgn( dff ) (19) Case 1) The dstance between the locaton of the ball and the center of the ROI wndow s smaller than the prevous movement. In ths case, the locaton of the ROI wndow does not change. Instead, the value for dsp +1 gets slghtly reduced by α. Case 2) The wndow s moved by dsp, and dsp +1 s updated by the rule shown n (17). Snce dsp > 0, the wndow hardly moves n the opposte way. Case 3) Ths case s dentcal to the prevous one except that the larger acceleraton parameter s adopted to deal wth the faster movement of the ball. That s, 0< a1 < a2 < 1. Note that the wndow should be always bounded nsde the frame as llustrated n Fg. 8. However, the value wndow obtaned above s not modfed. (a) (b) Fg. 8. (a) If the calculated locaton of the ROI wndow s out of boundary, (b) the wndow s bounded nsde the frame. (a) non-long shot frame (b) long shot frame Fg. 9. Screen shot of our system. All processes are fully automatc and run n real-tme. (Left of each shot) Orgnal frame wth ROI wndow ndcated by yellow lnes, (Rght of each shot) ROI s dsplayed on an enlarged scale f needed. 3. EXPERIMENTAL RESULTS The proposed system has been developed by usng Vsual Studo 2003 (C++) under Wn32 envronment and FFMpeg lbrary has been utlzed for MPEG decodng. We used four soccer vdeo sequences, whch are encoded n MPEG-1 wth the mage sze of 352x240 and the frame rate 29.97fps. The experments were conducted on a low-end PC (Pentum4 3.0GHz). The parameters for ROI determnaton are set to be 0.85, 0.05 and 0.15 for α, a1 and a 2, respectvely (refer to (14) to (19)). The sze of the ROI wndow was set to be 240x164, whch has a rato of 1: 0.68 n ether horzontal or vertcal drecton when compared to the orgnal frame. Proc. of SPIE Vol N-8

9 3.1 Evaluaton on ground detecton To evaluate the performance of the ground pxel detecton and ground map constructon proposed n Secton 3, we frst selected 10 dfferent soccer vdeos, where ground colors had enough varatons vdeo by vdeo. For each vdeo, 2 frames were selected arbtrarly, and two patches were extracted from each frame. One patch was from ground area, the other from non-ground area. The sze of the patches were , thus there exst 50 16x16 blocks n each patch. Fnally, the number of ground blocks was 1000 (= 50blocks/frame 2 frames/vdeo 10 vdeos), and so was that of non-ground blocks. We got 970 blocks detected as ground blocks out of 1000 ground blocks, and 42 blocks detected as ground blocks out of 1000 non-ground blocks. In other words, false negatve rate was and the false postve rate was 0.042, where both were defned as follows; 3.2. Evaluaton on Shot Classfcaton number of false negatves false negatve rate = number of postves number of false postves false postve rate =. number of negatves The expermental results are shown n Table 1. As shown n the table, accuraces are all over 90%. Ths s also compared wth our pror algorthm proposed n [8]. Both show good performance on the shot classfcaton. Note that the enhanced ground pxel detecton scheme (see (4) and (5)) and hole-fllng technque addressed n Sub-secton 2.2 compensate possble degradatons resultng from usng a center pxel only nstead of whole pxels n 16x16 blocks. Comparng processng tmes on the shot classfcaton, whch nclude ground map constructon, shot boundary detecton, and shot classfcaton, the proposed scheme requred 58μsec, whereas the pror one [8] 8,396μsec. Ths denotes the proposed scheme s about 145 tmes faster than the pror one. (20) Table 1. Accuracy of the proposed shot classfcaton technque. Pror Algorthm [8] Proposed Algorthm Long shot others Long shot others Long shot others Japan vs. Italy 99.76% % 99.92% 91.59% 97.44% Korea vs. Germany 99.77% 90.89% 98.47% 99.45% 99.24% 99.42% Brazl vs. Croata 99.54% 80.33% 93.06% 98.78% 77.61% 91.64% Korea vs. Costa Rca 100% 96.5% 97.67% 99.83% 99.71% 99.75% 3.3 Evaluaton on ROI Determnaton The objectve evaluaton on the determned ROI locaton s not easy. However, t s consdered that the ball should exst nsde ROI most of tme n case of long-shot frame. If the ball s out of ROI wndow for a long tme, vewers may not want to watch ths ROI-based dsplay any more. Hence, t can be a sgnfcant factor to evaluate the performance of the ROI determnaton. We counted the number of ROI determned frames and calculated the percentage to see how many ROI wndows contan the ball. ROI. The sze of ROI wndow adopted n the proposed system s 240x164 when the orgnal mage sze s 352x240. As shown n Table 2, t s found that the ball s hghly lkely to exst n the ROI wndow. Note that the frames were excluded n the evaluaton f the ball s occluded by players or out of the frame. Fg. 9 shows the screen shots captured from our demo system. As shown n the fgure, the ROI wndow s same as the orgnal mage n case of close-up and md shot, whereas t s a part of the shot otherwse. Proc. of SPIE Vol N-9

10 Table 2. Evaluaton of ROI determnaton Ht-rate (the ball nsde ROI) Pror Algorthm [8] Proposed Algorthm Japan vs. Italy 70.47% 85.52% Korea vs. Germany 80.13% 89.38% Brazl vs. Croata 64.80% 69.64% Korea vs. Costa Rca 77.83% 73.40% In addton, we provde a graph n Fg. 10 showng the relatonshp of locatons between the ball and ROI wndow along the tme axs. Each graph ndcates two mnute vdeo clp. The whte segments denote the ball exsts n the ROI wndow, whle the black segments the ball out of the wndow. Also, the dark gray segments denote the ball s out of orgnal frame, whle the lght gray segments ndcate non-long-shot frames, where the ball detecton s not performed. As seen n the fgure, the determned ROI wndows contan the ball most of tme (sec) (a) (b) (c) (d) Fg. 10. The relatonshp of locatons between the ball and ROI wndow along the tme axs. (whte): the ball exsts nsde ROI wndow, (black): the ball out of ROI wndow, (dark gray): the ball out of orgnal frame, (lght gray): ball detecton not conducted due to non-long-shot frame. (a) Japan Vs. Italy. (b) Korea Vs. Germany. (c) Brazl vs. Croata. (d) Korea vs. Costa Rca. 3.4 Performance evaluaton on PDA The proposed algorthm has also been mplemented on Mcrosoft Embedded Vsual Studo 4.0 for a PDA applcaton. The specfcatons of the PDA used here are summarzed n Table 3. For a performance test, we used one soccer vdeo clp whch s 120 second long and encoded n MPEG1 format. We took the experments three tmes and calculated average values. Table 4 shows elapsed tme for each sub-module. It s natural that processng speed for long-shot s slower than for non-long-shot, because the former contans addtonal module for ROI determnaton. Table 3. Test envronment for PDA portng (Hardware) Model No. CPU RAM screen HP PAQ hx4700 Intel PXA MHz 64 MB VGA (640x480), 4.0 nch Proc. of SPIE Vol N-10

11 Table 4. Elapsed tme for ntellgent dsplayng on PDA. (new) # of frames, N Elapsed tme( μs ), T T/N ( μs ) Total ,168,943(=T LS ) 19,095 Long shot 1 st Ball detecton ,167,657 17,152 2 nd ball detecton 2026 Shot classfcaton ,001,287 1,942 Non-long Total ,412,240(=T NL ) 1,368 shot Shot classfcaton ,412,240 1,368 Entre processng tme (ncludng MPEG decodng and dsplay) ,102,870 45,496 The followng equaton was used to assess processng tme for vdeo analyss, and resulted n 71.31fps, whch s more than two tmes faster than that from [8]. Average processng rate for analyss TLS + TNL = 1,000,000 (frames/sec) (21) N where T LS s an average elapsed tme for long-shot and T NL s for non-long-shot. N s the number of entre frames. To nclude decodng tme and screen dsplay, we used (22) and the computed total processng speed was 21.98fps, whereas t was fps n [8]. T Average playng rate = 1,000,000 (frames/sec), (22) N where T s total elapsed tme. Ths mprovement s represented n the form of bar graphs as shown n Fg. 11. The entre processng tme becomes 33% shortened compared to that from our pror algorthm [8]. The Elapsed Tme Comparson % 100% 100% Percentage(%) % 16.29% 67.37% Prevous Algorthm Proposed Algorthm 0 Long shot Non-long shot Total elapsed tme Fg. 11. Comparson of processng tmes Proc. of SPIE Vol N-11

12 4. CONCLUSION We have presented a system that provdes more comfortable vew to the small-dsplay-vewers by dstngushng ROI-necessary frames from other frames n the soccer vdeo, where the determned ROI wndow s ddplayed on an enlarged scale. To check robustness of our algorthm, we tested wth several vdeo clps. As explaned n the prevous secton, the proposed system performs well, provdng vewers wth comfortable vews. The proposed system runs faster than real-tme on a low end PC, and shows 70.58fps for ROI determnaton and 22.46fps for entre system runnng that ncludes MPEG decodng, ROI analyss, and screen dsplay. We beleve that users wll be able to enjoy ths ntellgent dsplay scheme wth great comfort. REFERENCE H. Knoche, J. D. McCarthy, and M. A. Sasse, Can small be beautful?: assessng mage resoluton requrements for moble TV, n MULTIMEDIA 05: Proceedngs of the 13th annual ACM nternatonal conference on Multmeda, pp , ACM Press, (New York, NY, USA), Hao Lu, Xng Xe, We-Yng Ma, and Hong-Jang Zhang Automatc browsng of large pctures on moble devces, n MULTIMEDIA '03: Proceedngs of the eleventh ACM nternatonal conference on Multmeda, pp , ACM Press, (New York, NY, USA), W. H. Cheng, W. T. Chu, and J. L. Wu, A vsual attenton based regon-of-nterest determnaton framework for vdeo sequences, IEICE Transactons on Informaton and Systems, E-88D, pp , L. Itt, C. Koch, and E. Nebur, A model of salency-based vsual attenton for rapd scene analyss, IEEE Transactons on Pattern Analyss and Machne Intellgence, vol. 20, no.11, pp , Nov Y. F. Ma and H. J. Zhang, A model of moton attenton for vdeo skmmng, n Proc. ICIP, pp , X. Fan, X. Xe, H. Q. Zhou, and WY Ma, Lookng nto Vdeo Frames on Small Dsplays, n MULTIMEDIA 03: Proceedngs of the eleventh ACM nternatonal conference on Multmeda, pp , ACM Press, (New York, NY, USA), Keewon Seo and Changck Km, "A Context-Aware Vdeo Dsplay Scheme for Moble Devces," n Proc. SPIE Vol.6074, p , Multmeda on Moble Devces II, San Jose, Jan A. Ekn, A. M. Tekalp, and R. Mehrotra, Automatc soccer vdeo analyss and summarzaton, IEEE Transactons on Image Processng, vol. 12, pp , K. Wan, X. Yan, X. Yu, and C. Xu, Real-tme goal-mouth detecton n mpeg soccer vdeo, n MULTIMEDIA 03: Proceedngs of the eleventh ACM nternatonal conference on Multmeda, pp , ACM Press, (New York, NY, USA), R. W. Lenhart, Comparson of automatc shot boundary detecton algorthms, Storage and Retreval for Image and Vdeo Databases VII 3656(1), pp , SPIE, Y. Seo, S. Cho, H. Km, and K. Hong, Where Are the Ball and Players? Soccer Game Analyss wth Color Based Trackng and Image Mosack, n ICIAP '97: Proceedngs of the 9th Internatonal Conference on Image Analyss and Processng-Volume II, pp , Sprnger-Verlag, London, UK, Y. Ohno, J. Mura, and Y. Shra, Trackng Players and a Ball n Soccer Games, n Int. Conf. On Multsensor Fuson and Integraton for Intellgent Sys., Tape, Tawan, X. Yu, C. Xu, H. W. Leong, Q. Tan, Q Tang, and K Wan, Trajectory-Based Ball Detecton and Trackng wth Applcatons to Semantc Analyss of Broadcast Soccer Vdeo, n MULTIMEDIA '03: Proceedngs of the eleventh ACM nternatonal conference on Multmeda, pp , ACM Press, (New York, NY, USA), Proc. of SPIE Vol N-12