Depth Estimation for 2D to 3D Football Video Conversion

Size: px

Start display at page:

Download "Depth Estimation for 2D to 3D Football Video Conversion"

Gwendolyn Casey
5 years ago
Views:

International Research Journal of Alied and Basic Sciences 2013 Available online at www.irjabs.

1 International Research Journal of Alied and Basic Sciences 2013 Available online at ISSN X / Vol, 6 (4): Science Exlorer ublications Deth Estimation for 2D to 3D Football Video Conversion Seyed-mohsen Seyed-mousavi Tasieh *1, Mehdi Rezaei 2 1. Faculty of Electrical and Comuter Engineering, 2. University of Sistan and Baluchestan, Zahedan, Iran *Corrosonding Auther idinmousavi@gmail.com ABSTRACT: Deth estimation is the main challenge of 2D to 3D video conversion. In this aer we roose a new algorithm for the deth estimation in 2D football videos. In this aroach, we use a football analyzer that it classifies football video shots into four tyes namely: Long-shot, Medium-shot, Close-u and Out-of-field shot. Then, the deth in each video shot is estimated according to the shot class. Simulation results indicate an accurate deth estimation erformance and a low comutational comlexity for the roosed method. Key words: 2D to 3D Video Conversion; Digital Video; Deth Estimation. INTRODUCTION Nowadays, the interest of watching 3D videos is increasing. Given that large amount of 2D videos are available, the new research is known as 2D to 3D video conversion is aeared. The accurate deth estimation of objects in 2D videos is the main challenge in this field. To overcome this, many aroaches have been suggested. As a well-known aroach, motion information is used for the deth estimation. In this aroach the assumtion is that, objects with lower deth have more relative motion and vice versa. Also it is considered the relationshi between cameras motions and the distance of camera and objects, then using the otical flow and mean-shift segmentation deth ma of image is estimated (Chao, 2012). In another method, the object deth is estimated based on vanishing oint in a static scene (Yu, 2011). Suosing a relation between color comonents and object deth in video, a deth estimation algorithm is roosed in (Tam, 2009). As another aroach, the objects are extracted based on edge and shae information while the object deth is assumed to be known (Feng, 2005, Chao-Chung, 2010). A combination of different techniques is also resented in (Zhang, 2011) for various video contents and therefore, it is not otimized for any secial video content. Sort videos have also a considerable art in 2D video contents and attracted many sectators. Thus, roducing 3D sort videos is the next ste in 3D technology. Caturing, roducing and broadcasting sort videos in form of 3D is costly and time consuming. Moreover, there are many catured 2D video contents that can be reused. Therefore, recently, it is focused on conversion of sort videos from 2D to 3D. Here we chose football videos as one of the most advocator sort videos. In a study by Schnyder et.al the authors investigated the conversion of 2D sort video to 3D by using a anorama method (Schnyder, 2011). They use a database from layers, field and other things, to segment frame into background anorama is constructed from the whole shot. Next, a deth ma is created for the whole anorama using assumtions about the lanner structure of the field. But, we know that in football videos the background is not static. Moreover, having a comlete data base from layers is imossible for all 2D football videos. However, there are many shot classification algorithms such as cited algorithms in (Sowjanya and Swati, 2012). In this aer we assume that shot classification is reviously done according to shot boundary detection on football video (Keewon, 2007). Therefore, in the resent study we investigated the conversion of 2D to 3D sort videos using football videos as one of the most attractive sort scenes for ublic. We used shot classification algorithm and estimated the deth of videos according to the shot class. The rest of this aer is organized as follows: Details of roosed deth estimation algorithm are resented in Section II. Then, exerimental results are summarized in Section III. Finally, a brief conclusion is given in Section IV.

MATERIAL and METHODS In the roosed method first the football ground region is extracted based on resented algorithm in (Keewon, 2007) by using the ground information.

Shot tyes are described as follows Long-shot was considered as a general diagram of game field that was recorded by a camera from far distance; such examle is shown in Figure.

1(b), the size of objects in Mid-shots was larger than in Long-shots. Close-u shots focused on the uer art of the layer which is catured by a near camera with maximum zoom, as shown in Figure. 1 (c).

This tye of shot included the viewers (fans) or other ersons out of the game field at a far distance as shown in Figure. 1(d).

2 MATERIAL and METHODS In the roosed method first the football ground region is extracted based on resented algorithm in (Keewon, 2007) by using the ground information. Video shots are classified into four generic tyes: Long-shot, Medium-shot, Close-u and Out-of-field. Shot tyes are described as follows Long-shot was considered as a general diagram of game field that was recorded by a camera from far distance; such examle is shown in Figure. 1(a), that objects are small and most art of the shot includes the field. 1 Mid-shot was concentrated on the secial art of the field. As shown in Figure. 1(b), the size of objects in Mid-shots was larger than in Long-shots. Close-u shots focused on the uer art of the layer which is catured by a near camera with maximum zoom, as shown in Figure. 1 (c). (a) (b) (c) (d) Figure 1. Examle of video shot tyes. (a) Long-shot. (b) Mid-shot. (c) close-u shot. (d) Out-of-field shot. Out-of-field shots were the same as zoom out. This tye of shot included the viewers (fans) or other ersons out of the game field at a far distance as shown in Figure. 1(d). To estimate the deth in each video shot we assume that lower area of a video frame has a deth less than uer area. A similar assumtion is used in (Schnyder, 2011). The roosed deth estimation algorithms for different video shots are exlained in the sequel. Deth Estimation for Long-Shot The used hyothesis for deth estimation is that lower areas in the video shot have a lower deth than uer areas. For this tye of shot, firstly, the ground area is extracted from the frame based on algorithms introduced in (Keewon, 2007). This algorithm includes three key stes: ground color learning, shot classification, and ROI determination. Samle results are shown in Figure. 2 (a). Then, a Dilation and Erosion method, as resented algorithm in (Halev, 2010), is used for more accurate searation of ground and the viewer latform. Next, by using the Sobel edge detector, the recise border between the ground and the latform is determined. Now, using a simle 3D model for the ground and latform, a deth is comuted for each frame ixel as exlained in the sequel. The model of a side view for the Long-shot frame is illustrated in Figure. 3. According to this model, the deth of ground and latform ixels increases by two rates namely Ground Deth er ixel and latform Deth er ixel. Also, D max is the uer oint row of the ground line and D min is the lower oint row of the ground line and the row of viewer latform that is comuted as: Dmax Dmin DG, (1) WG( x) WG( x)cos And 1.htt://en.wikiedia.org/wiki/Long_shot 476

D D G cos( ) (2) Where, D G and D are deth er ixel values for the ground and the latform area, resectively. W G (x) and W (x) are width of the ground and viewer latform in ixel, resectively.

Using the deth er ixel values, the deth of frame ixels is comuted as follow: Dmin DG x, if G( x, y) 1 & x xb Dmin DG xmin D( x, y) (3) if G( x, y) 0 for xmin x xmax & x xb Dmin DG xb D x xb if G( x,

3 D D G cos( ) (2) Where, D G and D are deth er ixel values for the ground and the latform area, resectively. W G (x) and W (x) are width of the ground and viewer latform in ixel, resectively. It is the angle between the viewer latform and the ground. Finally, D max and D min are the maximum and minimum deth assigned to the frames by another algorithm in a higher level. Using the deth er ixel values, the deth of frame ixels is comuted as follow: Dmin DG x, if G( x, y) 1 & x xb Dmin DG xmin D( x, y) (3) if G( x, y) 0 for xmin x xmax & x xb Dmin DG xb D x xb if G( x, y) 0 & xb x where D(x,y) is the deth value for each ixel in coordinates of (x,y). D G and D are deth er ixel values of ground and latform resectively. x B (y) is the vertical coordinate of boundary between ground and latform at horizontal coordinate y. G(x,y) is shown in Figure. 2 (b), which is consisting of ground, layer and latform. A fixed deth is assigned to ixels between x and x max that are belong to a layer on the ground. It should be noted that min x and x max may changed according to as y coordinate. min (a) (b) Figure 2. (a) ground binary of Long-shot frame. (b) ground detection. Figure 3. A side view of Long-shot frame. Deth Estimation for Medium Shot Utilizing the ground detection algorithm used in long-shot, the shot is divided to background and foreground regions. Then, the deths of foreground and background ixels are comuted as follows: 477

4 Dmin DF x, if G( x, y) 1. D( x, y) D D x, min F min min if G( x, y) 0 for x x xmax. Dmax Dmin DF (5) W F where D F denotes a deth er ixel value for the whole frame and W F is the whole frame width. A constant deth is assigned to ixels between and that are belong to a layer on the foreground. Deth Estimation for Close-u Shot Deth estimation algorithm for the close-u shots includes the following stes: First, using a simle thresholding on two comonents of the HSI color model, the skin regions are detected as: 1 if th1 H( x, y) th2 and th3 I( x, y) th4 S( x, y) (6) 0 other wise where, S(x,y) is a binary image in which ones are corresonding to ixels with a color close to human skin color. H(x,y) and I(x,y) are the hue and intensity color comonents of the shot. th1, th2, th3, and th4 denote four threshold values. The obtained S image corresonding to the samle close-u shot is shown in Figure. 4, (a). Next, by alying a thresholding on the hue color comonent of the shot the background regions are searated from detection the foreground (FG) regions as: 1 if th5 H( x, y) th6 B( x, y) (7) 0 other wise where, B(x,y) is a binary image in which ones are corresonding to the background region. th5 and th6 are two threshold. Afterward, using S(x,y) and B(x,y), the foreground regions can be extracted recisely as follows: 0, if s( x, y) 1 & B( x, y) 0 F( x, y) (8) 1, other wise where, F(x,y) is a binary image in which ones are corresonding to the foreground region. The foreground image corresonding to the samle close-u shot is shown in Figure. 4 (b). Then, the deths of foreground and background ixels are comuted as follows: Dmin DF x, if F( x, y) 1 D( x, y). (9) Dmin, if F( x, y) 0 Deth Estimation for Out-of-field Shots Searating foreground and background regions is a difficult task in the out-of-field shots. Therefore, for the deth estimation, it is assumed that the ixel deth is gradually increased from bottom to u regions of the shot i.e. D( x, y) Dmin D F x, (10) Estimated deth ma for the samle out-of-field shot is shown in Figure. 5(d). SIMULATION RESULTS All analysis was erformed using the software MATLAB 2010(b) under Win64 environment and running on a lato using entium 4 with 2 GHz CU. We used one football video sequences, which were encoded in MEG- 1 with the image size o and 30 fs (Iranian remier League, 2012).The Long-shot with θ=45 0, D min and D max was variable between and also, threshold values for Close-u shot were: th1=0.07, th2=0.23, th3=0, th4=0.58, th5=0.2 and th6=0.25. The exerimental results confirmed that our roosed method was quite straight forward to estimate deths of mentioned various shots as shown in Figure. 5. To evaluate the roosed deth estimation method from the comutational comlexity oint of view, the algorithm was imlemented on 100 frames of each shot tye and the comutation time was measured. To decrease the measurement error resulting of used (4) 478

multitasking oerating system, the deth estimation and time measurement was reeated five times. rovide results are resented in Table 1.

(a) The result of skin detection. (b) Background extraction.

comaring these results indicates that our roosed deth estimation algorithm has a much lower comutational comlexity.

In this method we used classification football video method according to the videos generic tyes.

5 multitasking oerating system, the deth estimation and time measurement was reeated five times. rovide results are resented in Table 1. According to the obtained results the average comutation time on different shots is 1.05 seconds. A comutation time of 6 to 8 (a) Figure 4. (a) The result of skin detection. (b) Background extraction. (b) seconds, rovided by a more owerful hardware, has been reorted for the algorithm resented in (Schnyder, 2011) by Schnider et. Al. comaring these results indicates that our roosed deth estimation algorithm has a much lower comutational comlexity. CONCLUSION In the resent study we used a unique model to estimate deths for football videos. In this method we used classification football video method according to the videos generic tyes. Next, we introduced methods to calculate corresonding deth for each class. According to some simulation results, we concluded that our suggested method is a romising tool for the deth estimation of 2D football video while it has a low comutational comlexity. However further studies are required to finalize this research task and to exand our algorithm for more different football video shots. (a) (b) (c) (d) Figure 5. Deth ma estimation for each shot tye. (a) deth of Long-shot tye. (b) deth of Mid-shot tye. (c) deth of close u shot. (d) deth of Out-of-field shot. 479

6 Table1. Comutation Time Evaluation Results Shot Tye Comutation Time for 100 frame (second) Ste 1 Ste 2 Ste 3 Ste 4 Ste 5 Average Average Time for 100 Time for one frame frame Long Medium Close-u Out-of-field REFRENCES Chao L, Christoher L Deth ma estimation from motion for 2D to 3D conversion, Electro/Information Technology (EIT), IEEE International Conference on. Chao-Chung C, Chung-Te L A novel 2D-to-3D conversion system using edge information, Consumer Electronics (ICCE), Digest of Technical aers International Conference on. Feng Y, Jiang J, Ison SS A shae-match based algorithm for seudo-3d conversion of 2Dvideos, roc. IEEE Conf. on Image rocessing (ICI), Vol. 3, , Genoa, Italy, Halevi RB Soccer analyzer final roject introduction to comutational and biological vision, Ben-Gurion University of the Negev, 29 July Keewon S, Jaeseung K An Intelligent Dislay Scheme of Soccer Video on Mobile Devices. Circuits and Systems for Video Technology, IEEE Transactions on 17(10): Schnyder L, Wang O D to 3D conversion of sorts content using anoramas, Image rocessing (ICI), 18th IEEE International Conference on. Sowjanya S, RM. Video shot boundary detection comarision of color histogram and GIST method, IJREAS 2(2). Tam WJ, Vazquez C, Seranza F Three-dimensional TV: a novel method for generat-ing surrogate deth mas using colour information, roc. SIE Electronic Imaging Stereoscoic Dislays and Alications XX, Vol. 7237, A A-9, San Jose, California, Yu F, Liu J, Ren Y, Sun J, Gao Y, Liu W Deth generation method for 2D to 3D conversion, IEEE International Conference on. Zhang L, Vázquez C, Knorr S D-TV content creation: automatic 2D-to-3D video conversion, IEEE Trans, VOL. 57, NO. 2, , 480

An Efficient Coding Method for Coding Region-of-Interest Locations in AVS2

An Efficient Coding Method for Coding Region-of-Interest Locations in AVS2 Mingliang Chen 1, Weiyao Lin 1*, Xiaozhen Zheng 2 1 Deartment of Electronic Engineering, Shanghai Jiao Tong University, China