Overview. Augmented reality and applications Marker-based augmented reality. Camera model. Binary markers Textured planar markers

Transcription

1 Augmented reality

2 Overview Augmented reality and applications Marker-based augmented reality Binary markers Textured planar markers Camera model Homography Direct Linear Transformation

3 What is augmented reality? Reality is subjective Sight Hearing Smell Haptics Balance Augmenting sensory information

4 Augmenting visual information Superpositioning digital information on top of real imagery Who Framed Roger Rabbit (1988) Tom Caudell - Boeing (1990) Xerox & University of Toronto (1990) Virtual Fixtures (1992)

5 Classical approach Visual information acquisition: camera Camera localization Image: camera Depth: depth camera Other: GPS, WiFi, IMU Displaying augmented information: monitor, mobile phone, projector, smart glasses

6 Application examples (TV) Olympic games 2004 Monitor/TV Robotic camera USA elections 2008 CNN Hologram conference 35 cameras 20 computers

7 Application examples (Mobile) Mobile devices Pokemon Go Vuforia, ARKit Wearables BMW Hololens

8 AR using depth information Depth cameras Active (IR light) Passive (Stereo systems) Automatic scene reconstruction Easier interaction with objects Izadi et al. "KinectFusion: Real-Time Dynamic 3D Surface Reconstruction and Interaction", SIGGRAPH, 2011

9 AR with visual anchor Localization with visual information Detect key object in image Determine relative position of the object to camera Draw information with this relation

10 From point to pixel Transform 3D point to camera coordinate system (pixels) Required data K... camera calibration matrix (intrinsic parameters) R,t... rotation and translation matrices (extrinsic parameters)

11 AR with binary marker Detect markers that are easy to detect and identify Detect marker from edges Identify marker with correlation Known marker size Compute relation to camera Use corners of marker to compute relative position

12 AR with arbitrary planar marker Match an arbitrary surface Describe local texture Robust matching Less constrained can use existing textures from real world Posters Building facades

13 Applications of marker AR Catalogs Books Tourism Gaming

14 Detecting binary marker Machine vision approach Finding possible candidates Adaptive threshold Trace contours Estimate contours as polygons Contours with 4 corners

15 Recognizing binary marker Threshold on similarity Project region to reference position Normalized cross correlation Orientation test all four options

16 Generalizing transformation Determine camera extrinsics using a detected marker Transformation between planes (homography) Marker plane (reference) Camera plane (observed)

17 Required linear algebra Homogeneous coordinates Can describe point in infinity Same point if multiplied with scalar Transformation matrix contains translation Matrix form of vector (cross) product Express product as matrix multiplication of skew-symmetric matrix and vector

18 Systems of linear equations Get a set of variables from a set of equality relations Known pairs of vectors correspondences Linear transformation contains unknowns Formulate problem in matrix form: Sufficient data - unique solution: Not enough data underdetermined system Too much data overdetermined system

19 Direct Linear Transform Similarity relations Unknown multiplicative factor Rewritten as homogeneous equations Get rid of scaling factor

20 DLT for Homography Homography is 3x3 matrix 8 unknowns Assume Need 4 correspondences

21 The top two equations are independent, the last one is not. We can reorganize the system

22 Solving homogeneous system N correspondences give us 2*N equations We can get solution for the system using SVD The solution is the last column of V Minimal mean square error Reorganize vector into matrix

23 Optimizing extrinsics Homography projects points between two planes Determine camera position Inaccurate R and t Objective function is MSR, not reprojection error

24 Minimizing reprojection error Projection of 3D point to image coordinates is written as When R and t are optimal the difference between image point and projection of 3D point should be zero Cost function is sum of all distances, it is non-linear

25 Iterative optimization of R and t Initial R ant t from homography Fix t and minimize R Use quaternions Optimize with non-linear methods (e.g. Levenberg-Marquardt, Simplex) Fix R and minimize t Using least-squares optimization Repeat steps until convergence

26 Binary marker example Simple (no normalization) With normalization

27 Augmented reality with planar marker Natural surfaces Unable to detect corners robustly Partial occlusions Can detect feature-points Over-sample reference points Not all points will match correctly Robustly estimate homography

28 Matching keypoints Detector of keypoints Descriptor of regions SIFT, SURF BRIEF, ORB Matching descriptors Distance function Symmetric matches

29 Robust estimation of homography Many correspondences Over-determined system Not all correspondences are correct Robust matching Exclude outliers from calculation Find sub-set of correspondences that agree on a model

30 RANSAC Random Sample Consensus Meta algorithm (used for many tasks) Probabilistic interpretation Repeat k times Select random set of 4 correspondences Estimate model homography (DLT) Look which other pairs agree with the model (projection from one plane to the other is small enough) Take the model with largest support (inliers) RANSAC for line fitting (source: F. Moreno)

31 Reference plane example Disney AR coloring book Urban AR

32 Beyond planar markers Reference objects Deformable surfaces No reference

33 Realistic rendering Acquired images are degraded by various factors Augmented reality will be more immersive if these factors are replicated on virtual objects

34 Motion blur Exposure time + rapid motion Simulated by smoothing image with directional filter

35 Chromatic aberration Different wavelenghts bend under different angles Simulated by distorting individual color channels Image center Image border

36 Spherical aberration Spherical lenses do not focus light perfectly Rays on the edge are focused closer Compensated by using multiple lenses No abberation With abberation

37 Vignetting Optical vignetting multiple lenses Pixel vignetting Digital sensors Angle dependence Software compensation

38 Radial distortion Imperfect lenses Deviations are apparent near the edge of image

39 Realistic rendering pipeline Gerhard Reitmayr, Axel Pinz, Visual Coherence in Augmented Reality

40

41 Diminished reality Use AR to determine which part of the image to erase using inpainting Herling, J. and Broll, W., Advanced Self-contained Object Removal for Realizing Real-time Diminished Reality in Unconstrained Environments. ISMAR 2010