Augmented and Mixed Reality

Transcription

1 Augmented and Mixed Reality Uma Mudenagudi Dept. of Computer Science and Engineering, Indian Institute of Technology Delhi

2 Outline Introduction to Augmented Reality(AR) and Mixed Reality(MR) A Typical AR System Issues in AR Different methods of Augmenting object Algorithms in AR Abstract Model of AR/MR Initial Results Obtained Summary

3 Introduction to AR and MR AR combines real and virtual objects in real environment Virtual objects(3d model/images/video) are merged with real environment Milgram et al(1994) described the relation between AR, MR and VR REAL ENV Mixed Reality VR AR AV Figure 1: Continuum of real and virtual environment MR spectrum lies between the extremes of real life and Virtual reality(vr). Views of the real world are combined in some proportion with views of a virtual environment

4 Typical AR system AR SYS = Computer vision + Computer graphics + User interfaces SCENE CO ORD CAMERA POSITION REAL SCENE PZT REAL IMAGE CO ORD WORLD CO ORD VIR OBJ CO ORD VIR OBJ ALIGN GRAPHICS CAMERA TO REAL GRAPHICS RENDERING GRAPHICS CO ORD VIDEO IMAGE GRAPHICS IMAGE GRAPHICS IMAGE CO ORD Figure 2: Typical AR system

5 Issues in AR:Registration Process of estimating an optimal transformation between two images(also known as spatial Normalization) To align the virtual object to real objects in 3D

6 Issues in AR:Registration contd.. Most critical requirement of AR system :Since human visual system is very good at detecting even small mis-registration Methods: No generalized method of registration for all the type of augmentations Static errors:optical distortion, mechanical misalignment and incorrect viewing parameters Dynamic errors:system delays

7 Issues in AR: Tracking Tracking : View point tracking as the view point moves Tracked viewing pose defines the AR alignment and registration Issues: Foreshortening, Scaling, Occlusions

8 Issues in AR: Modeling Modeling : Modeling of the destination with Texture and extraction of the 3D model from the source environment Single view : Single view reconstruction Multiple view reconstruction: two view

9 Possible ways of augmenting Source 3D Model: Model is given Single Image: Extract 3D/2D Model of the object and Texture Multiple Images: Extract 3D Model of the object and Texture Video : Extract 3D Model of the object and Texture Destination 3D Model:Model of destination Single Image: Single view Reconstruction with Texture Multiple Images: Multiple view reconstruction with Texture Video: Tracking and Registration of destination

10 Possible ways of augmenting Source 3D Model: Model is given Single Image: Extract 3D/2D Model of the object and Texture Multiple Images: Extract 3D Model of the object and Texture Video : Extract 3D Model of the object and Texture Destination 3D Model:Model of destination Single Image: Single view Reconstruction with Texture Multiple Images: Multiple view reconstruction with Texture Video: Tracking and Registration of destination

11 Possible ways of augmenting Source 3D Model: Model is given Single Image: Extract 3D/2D Model of the object and Texture Multiple Images: Extract 3D Model of the object and Texture Video : Extract 3D Model of the object and Texture Destination 3D Model:Model of destination Single Image: Single view Reconstruction with Texture Multiple Images: Multiple view reconstruction with Texture Video: Tracking and Registration of destination

12 Source:3D Model, Destination:Single Image Destination Image Augmented image

13 Possible ways of augmenting Source 3D Model: Model is given Single Image: Extract 3D/2D Model of the object and Texture Multiple Images: Extract 3D Model of the object and Texture Video : Extract 3D Model of the object and Texture Destination 3D Model:Model of destination Single Image: Single view Reconstruction with Texture Multiple Images: Multiple view reconstruction with Texture Video: Tracking and Registration of destination

14 Source:Multiple Images, Destination:Single Image Source Images(2/7) Augmented image

15 Possible ways of augmenting Source 3D Model: Model is given Single Image: Extract 3D/2D Model of the object and Texture Multiple Images: Extract 3D Model of the object and Texture Video : Extract 3D Model of the object and Texture Destination 3D Model:Model of destination Single Image: Single view Reconstruction with Texture Multiple Images: Multiple view reconstruction with Texture Video: Tracking and Registration of destination

16 Source:Multiple Images, Destination:Multiple Images Source Images(2/20) Augmented image

17 Possible ways of augmenting Source 3D Model: Model is given Single Image: Extract 3D/2D Model of the object and Texture Multiple Images: Extract 3D Model of the object and Texture Video : Extract 3D Model of the object and Texture Destination 3D Model:Model of destination Single Image: Single view Reconstruction with Texture Multiple Images: Multiple view reconstruction with Texture Video: Tracking and Registration of destination

18 Source:3D model, Destination:Video Source1, Augmented1 Source2, Augmented2, Augmented2, Augmented2 and Augmented2 Augmented3

19 Possible ways of augmenting Source 3D Model: Model is given Single Image: Extract 3D/2D Model of the object and Texture Multiple Images: Extract 3D Model of the object and Texture Video : Extract 3D Model of the object and Texture Destination 3D Model:Model of destination Single Image: Single view Reconstruction with Texture Multiple Images: Multiple view reconstruction with Texture Video: Tracking and Registration of destination

20 Mixed Reality Mixed Reality Example Microsoft research lab: SIGGRAPH 2004 Video view interpolation using layered approach Color Segmentation based stereo algorithm Mattes near discontinuities Two layer compressed representation to handle matting Major disadvantages:synchronizing many cameras, and acquiring and storing of images

21 Move matching method Source:3D Model, Destination:Video Given by Zisserman et.al Initialization: Manually indicate the planar region in image-0. (corner detection and matching are restricted to this region) Detect interest points in image 0 Initialize camera calibration K Steady state: computing H from frame i to i 1 Detect interest points in two images say X 1 k N 1 k 1 Match interest points which maximizes the cross correlation in 7x7 mask: x j x 1 k X j N j 1 and

22 Move matching method contd.. Randomly sample subset of four matched pairs and compute homography. Each candidate H is tested against all the correspondence by computing distance between x 1 and Hx. Choose H for which most pairs are within the threshold Compute pose from H i Result set 1: Source, Tracking and Augmented Result set 2: Source Tracking and Augmented 1 i

23 Method-2,G. Simon et.al Results Augmented Initialization stage Camera parameters 3D/2D corr of 4 pts. computation of initial pose + set of visible model features in the first image + extraction of key points in the first frame Model of the obj to be added in the scene. Trajectory of the object in the scene image k >k+1 Tracking of set of visible model features in the current image Updating the set of tracked pts STEP 1 Key points are extracted in frame k+1. They are matched with the points extracted in k STEP 2 The view point is computed using mixing method STEP 3 The computer generated object is added in scene STEP 4

24 Augmented views:z-keying method z-keying method: Given by T. Kanade et.al Uses dense depth map as a switch For each pixel, the z-key switch compares the pixel depth values of two images, and routes the color value of the foreground image that is nearer to the camera for the merged output image Real and virtual objects will occlude correctly Uses real time stereo-machine:specifications are Number of cameras: 2 to 6 Frame rate: max 30frames/sec Depth image size: up to Disparity search range:60 pixels Results: Augmented

25 Abstract Model of AR/MR Static Part 3D model of space Dynamic Parts Temporal Info (real+virtual) Query F(V,t) 4D data: Space and Time Render Image

26 Problem of 4D Fly-through Static 3D Model: Reconstructed from set of images Temporal Information: Extract Temporal information for each view 4D data:3d space and time Query:F V t? Generate view from the static and temporal information and answer the query

27 Results Source:3D model and Destination: Still image Method: Single view reconstruction method Destination Image 3D model augmented into still image Augmented movie-1, Augmented movie-2 and Augmented movie-3

28 Results contd.. Source:Object extracted from video and Destination:Still image Method: Source: Object is extracted as 2D plane Destination: Sparse 3D modeling of possible occluders Registration: Object plane to ground(virtual) plane Homography Rendering: Modified ray tracing algorithm source movie Augmented movie

29 summary Typical AR system and Issues in AR Possible augmentation methods and review results Abstract model of AR and MR 4D fly through problem Preliminary results to wards the 4D fly through problem

30 Thank you THANK YOU