Specialized Depth Extraction for Live Soccer Video

Transcription

1 Specialized Depth Extraction for Live Soccer Video Axon Digital Design Eindhoven, University of Technology November 18, 2010

2 Introduction Related Work Proposed Approach Results Conclusion Questions

3 Outline Introduction Related Work Proposed Approach Results Conclusion Questions 2D-To-3D Conversion 3D I 3D Cinema. I 3D broadcast. I 3D TV sets. I 3D Live Events.

4 2D-To-3D Conversion 3D Productions 3D cameras. 2D-to-3D conversion. Offline (semi-) automatic. Real-time.

5 2D-To-3D Conversion 3D Productions 3D cameras. 2D-to-3D conversion. Offline (semi-) automatic. Real-time.

6 2D-To-3D Conversion Why 2D-To-3D conversion? 2D-to-3D Conversion 3D Recording 10,000$ Widely available. 80,000$ Investment. No camera rig. No stereographer.

7 Stereoscopic 3D Binocular Disparity

8 Stereoscopic 3D From Depth to Stereo 3D Depth disparity. 2D-To-3D: 1. Extract depth. 2. Calculate disparity. 3. Render Left/Right image. Occlusion handling. Left Eye Right Eye Screen Left View Right View

9 Stereoscopic 3D From Depth to Stereo 3D Depth disparity. 2D-To-3D: 1. Extract depth. 2. Calculate disparity. 3. Render Left/Right image. Occlusion handling. Left Eye Right Eye Screen Left View Right View

10 2D-To-3D Conversion of Live Soccer Video Long Shot Images in Live Soccer.

11 Jung et al. Jung et al. [1]

12 Jung et al. Jung et al. cont. Advantages No occlusions. Few computations. Disadvantages Pan, tilt, zoom not modeled. Audience depth constant. No depth offset.

13 Proposed Approach Field and audience depth model. Exploit Camera + Scene information. Focal length Tilt/Pan angle Image sensor size Average player length

14 Image Model Input Camera Feed f, β t, β p, m x, m y 2D Input Feed Field + Player Detection Proposed Method User Input: Field Boundary Extraction Depth Offset Calculation L Depth Calculation AR Stereoscopic View Rendering Output 3D output

15 Field and Player Detection Field Detector (Seo et al. [2]) Train Hue, Sat, Val histograms. Extract PeakValueIndex, SaturationMean. G > 0.95 R R > 0.95 B Field, if V < 1.25 PeakValueIndex S > 0.8 SaturationMean (1)

16 Field and Player Detection Field Boundary Extraction Piecewise linear curve fit. (Cantoni [3]). Specify start, end and line intersection.

17 Field and Player Detection Player Detection Input image Connected Components Player 0.4 < AspectRatio < 3 if #Pixels > 0.05MN BBwidth BBheight #Pixels > 0.25

18 Field and Player Detection Player Detection Cont. Field Boundary Extraction aides Player Segmentation. Input Jung et al. [1] Proposed

19 Camera Depth Model Camera depth model Long shot camera Small lens aperture. High depth of field. Pin-hole camera. f

20 Camera Depth Model Field Depth Depth in YZ-plane: y β t Depth in XZ-plane: x β p f L p V α V β t -α V f L p H α H β p -α H D z D o D z D o D = D o sin β t sin( β t α V ) D = D o cos β p cos( β p α H )

21 Camera Depth Model Field Depth cont. Depth in XYZ-Plane. Image Sensor Line parallel to field boundary line D field (α V, α H ) = sin β D t cos β p o sin( β t α V ) cos( β p α H )

22 Camera Depth Model Field Depth Calculation A α V f L p V α V = arctan ( α H = arccos B α H ( L P V f L p D ) L p H C 2f 2 +L P 2 V +L P 2 D L P 2 H 2 (f 2 +L P 2 )(f V 2 +L P 2 ) D ) Field y ( x l, y l (x l ) ) L H p L D p L V p Vertical center line ( x o, y l (x o ) ) ( x o, y o ) Field boundary Parallel Field Lines Line l Depth offset unknown. x

23 Camera Depth Model Depth Offset y β t Player length 1.80m L P Tilt, f, Depth, L. f * α V L p D D o,i depth offset player i. * β t L z D o = median{d o,i i} D = L sin β t[l P m y +f tan α V]+f cos β t cos α V [L P m y +f tan α V ] f sin α V Player L

24 Camera Depth Model Audience Depth D aud (y) = D FB AR 0.1N (y y FB) + D FB, Depth (Y FB - 0.1N, D FB AR) ( Y FB, D FB ) Y

25 Camera Depth Model Player Depth y β t f * α V L p D * β t L z Player approximately constant. D player = D field (x i, y i ). Player Leng

26 Camera Depth Model Complete Camera Depth Model Depth Map Quantization. ( ) DEPTH(x, y) = 255 D(x,y) Dmin D max D min Clip outside [D min, D max ].

27 Depth Offset Calculation Depth Offset Calculation Zooming Keep focal length constant in depth offset calculation. Depth Offset Frame Number

28 Qualitative Comparison Qualitative Comparison Input Jung et al. [1] Proposed

29 Conclusion Jung et al. [1] proposed specialized 2D-3D conversion of long shot images. We propose more advanced model: 1. zooming, panning and tilting modeled. 2. Calculates depth offset from player length. 3. Improved player segmentation by field boundary extraction. Few computations. Hardware attractive. Future work: perceptual tests.

30 Questions Thanks for your attention.

31 Y. J. Jung, C. Kim, D. Park, Y. Kim, and J. Ko, Method, medium, and system for generating depth map of a video image, U.S. Patent US , August 6, K. Seo, J. Ko, I. Ahn, and C. Kim, An intelligent display scheme of soccer video on mobile devices, IEEE Transactions on Circuits and Systems for Video Technology, vol. 17, no. 10, pp , October A. Cantoni, Optimal curve fitting with piecewise linear functions, IEEE Transactions on Computers, vol. C-20, no. 1, pp , Januari 1971.