Recent Trend for Visual Media Synthesis and Analysis

Transcription

1 1 AR Display for Observing Sports Events based on Camera Tracking Using Pattern of Ground Akihito Enomoto, Hideo Saito HVRL: Hyper Vision i Research Lab. Keio University July 23, 2009 Recent Trend for Visual Media Synthesis and Analysis 2 Multiple Camera Virtualized Reality (CMU) Matrix NFL EyeVision 3D Display Technology Free-Viewpoint Video Auto-Stereoscopic Display Volumetric 3D Display Virtual Reality Mixed Reality Display

2 Virtualized Reality (CMU, ) 99) Example: 3 3-Men Basketball (These are movies.) Input sequence Synthetic court 4D Model For soccer scenes 4 cam 1 cam 4 cam 2 cam 3 cam1 cam2 cam3 cam4 [Inamoto and Saito ICPR2002] [Inamoto and Saito, ICPR2002] [Inamoto and Saito, IEEE Trans. MM, 07]

3 Calculation of Viewpoint Position Arbitrary View Synthesis of Soccer Scene Rendering on The Stadium 5 Multiple View Images Captured at A Stadium Virtual Views of The Stadium Ref.Cam1 Ref.Cam2 GUI Virtual View of The Dynamic Regions Overlay on The Stadium Image 6 Example of Free Viewpoint Images Cam 1 Cam 2 Cam 2 Cam 3 Cam 4

4 Observe Soccer Match on the Desktop [Inamoto, Saito ISMAR03] User sees a desktop stadium model in the real world with video see-through HMD and observes dynamic objects of soccer scene overlaid onto the display. 7 Video See-Through HMD Desktop Stadium Model Captured Soccer Match at Stadium Virtual View Generation Determination of Viewpoint Position Arbitrary View Synthesis of Soccer Scene Rendering on The Stadium 8 Multiple l View Images Captured at A Stadium Virtual Views of The Stadium GUI Ref.Cam1 Ref.Cam2 Overlay on The Stadium Image HMD Camera Image Virtual View of The Dynamic Regions Overlay on Desktop Stadium Model

5 Experimental Results 9 We have implemented immersive observation system for actual soccer matches. Captured soccer images : pixel, 24-bit-RGB color Camera 1 Camera 2 Camera 3 Camera 4 Camera Configuration at A Stadium Canon Video See-Through HMD Example 10 Frame 335 Stadium Camera 1 Stadium Camera 2 On Real Stadium Image ( Camera 1-2 w = 0.5 ) On Tabletop Stadium Model ( Camera 1-2 w = 0.47 z = 1.07 )

6 11 Unstable Camera Tracking Limited Camera Movement AR Baseball Presentation System [Uematsu, Saito, ICAT06, IVC09] 12 Virtual baseball game is overlaid onto a real field model. A user watches the game through a handheld monitor. The baseball game is replayed from a scorebook data file. Multiple planar markers are automatically integrated. web-camera baseball field model user 2D markers handheld monitor

7 Demo Video 13 In this presentation.. 14 Real soccer player captured with multiple cameras in stadium Observering camera Wide-area movement Real-time, stable tracking

8 System Configuration 15 Observer s View Field Pattern and AR Marker Observering Camera Camera Tracking for Registration 16 AR Toolkit + Natural Features [Motokawa ISMAR06]

9 17 Data Processing Flow 18 Off-Line Pose/Position of Observing Camera On- Line Observing Camera Stadium Camera (Fixed) Player Extraction Extracted Textures Overlay Ball Penalty Area

10 On Line Phase Calibration Stadium Camera Selection AR Display 19 Observing Camera Stadium Camera Projection Matrices Selection Position of Projection Overlay Calibration 20 Observing Camera Stadium Camera Projection Matrices Selection Position of Projection Overlay

11 Homography Based Calibration y Gilles Simon, et al.: Markerless tracking using planar structures in the scene, ISAR Z Y X Image x World On-Line Corner Detection 22 Observing Camera Intial Estimate Template Matching (a) Initial Estimate Refine 4 Corner Positions (b) Template (c) Refine

12 Selection of Stadium Camera 23 Observing Camera Stadium Camera Projection Matrices Selection Position of Projection Overlay 24 Stadium Camera with Closest Pan Angle (around Z axis) to Observing Camera is selected Observing Camera Z Y Z Y X X Stadium Camera

13 AR Display 25 Observing Camera Stadium Camera Projection Matrices Selection Position of Projection Overlay Position in Image Player Position x X Homography y ~ Y H Position on Ground Gou 26 Z Y y x Stadium Camera Image (x,y) (X,Y,0) X

14 Scaling Magnitude of Player Ratio of Translational Component of Both Cameras T Ratio = a a:coefficient T T:Stadium Camera T :Observing Camera T T 27 Stereo Matching Ball Position 28 (X,Y,Z) m (u,v) m (u,v ) v v u u P P

15 Overlaying Virtual Objects: P 29 Stadium Capturing Ajinomoto Stadium Three Cameras Video Size: Results 10m 20m 30 AR System CPU: Core2Duo 3.00GHz Memory: 2GB Video Size:

16 31 Effect of using Corners and Marker 32 Ground truth th is manually measured Average (pixel) Maximum (pixel) Both Marker only

17 Additional Functionalities 33 -Off-Side Detection -Ball Trajectory Display Conclusion 34 AR Display for Observing Sports Events Camera Tracking Using Pattern of Ground Marker + Corner points tracking

18 35 Google : HVRL