Augmented Reality, Advanced SLAM, Applications

Transcription

1 Augmented Reality, Advanced SLAM, Applications Prof. Didier Stricker & Dr. Alain Pagani Lecture 3D Computer Vision AR, SLAM, Applications 1

2 Introduction Previous lectures: Basics (camera, projective geometry) Structure From Motion Structured Light Dense 3D Reconstruction Depth Cameras Today: Insights into SLAM techniques Augmented Reality Applications of 3D Computer Vision Lecture 3D Computer Vision AR, SLAM, Applications 2

3 Introduction Previous lectures: Basics (camera, projective geometry) Structure From Motion Structured Light Dense 3D Reconstruction Depth Cameras Today: Insights into Advanced SLAM techniques Augmented Reality Applications of 3D Computer Vision Lecture 3D Computer Vision AR, SLAM, Applications 3

4 Recall: structure and motion (SAM) Unknown camera viewpoints Reconstruct Sparse scene geometry Camera motion Lecture 3D Computer Vision AR, SLAM, Applications 4

5 Offline vs. online structure and motion Offline: Online: E.g. as basis for dense 3D model reconstruction No real-time requirements, all images are available at once E.g. for mobile Augmented Reality in unknown environments Real-time requirements, images become available one by one, output required at each time-step Lecture 3D Computer Vision AR, SLAM, Applications 5

6 Online structure and motion (calibrated case) Reminder - Lecture 7 Iterative SFM Alternating estimation of camera poses and 3D feature locations (triangulation) from a (continuous) image sequence. Compute pose of first 2 cameras Relative Pose Problem 8 Point Algorithm 2D feature location (from image processing) t = 1 Camera pose t = 2 2D Matches Lecture 3D Computer Vision AR, SLAM, Applications 6

7 Online structure and motion (calibrated case) Reminder - Lecture 7 Iterative SFM Alternating estimation of camera poses and 3D feature locations (triangulation) from a (continuous) image sequence. Triangulate 3D points 3D feature location 2D feature location (from image processing) Camera pose t = 1 t = Lecture 3D Computer Vision AR, SLAM, Applications 7

8 Online structure and motion (calibrated case) Reminder - Lecture 7 Iterative SFM Alternating estimation of camera poses and 3D feature locations (triangulation) from a (continuous) image sequence. 3D feature location 2D feature location (from image processing) t = 1 t = 2 t = 3 Camera pose 2D Matches Lecture 3D Computer Vision AR, SLAM, Applications 8

9 Online structure and motion (calibrated case) Reminder - Lecture 7 Iterative SFM Alternating estimation of camera poses and 3D feature locations (triangulation) from a (continuous) image sequence. Estimate next camera pose (now from 2D/3D correspondences) 3D feature location Pose Problem PnP 2D feature location (from image processing) t = 1 t = 2 t = 3 Camera pose Lecture 3D Computer Vision AR, SLAM, Applications 9

10 Online structure and motion (calibrated case) Reminder - Lecture 7 Iterative SFM Alternating estimation of camera poses and 3D feature locations (triangulation) from a (continuous) image sequence. 3D feature location Triangulate additional 3D points 2D feature location (from image processing) t = 1 t = 2 t = 3 Camera pose Lecture 3D Computer Vision AR, SLAM, Applications 10

11 Online structure and motion (calibrated case) Reminder - Lecture 7 Iterative SFM Alternating estimation of camera poses and 3D feature locations (triangulation) from a (continuous) image sequence. 3D feature location Refine known 3D points with new camera poses 2D feature location (from image processing) t = 1 t = 2 t = 3 Camera pose Lecture 3D Computer Vision AR, SLAM, Applications 11

12 Online structure and motion (calibrated case) Reminder - Lecture 7 Iterative SFM Alternating estimation of camera poses and 3D feature locations (triangulation) from a (continuous) image sequence. 3D feature location 2D feature location (from image processing) t = 1 t = 2 t = 3 Camera pose Refine known cameras with new 3D points Lecture 3D Computer Vision AR, SLAM, Applications 12

13 Global Bundle Adjustment Global bundle adjustment: jointly optimize over all camera poses and 3D points (previous lecture) x = arg min k t=1 n l=1 r (i) Estimate Minimize over parameter vector containing all camera poses and 3D points 6 parameters for each camera + 3 for each 3D point 6k + 3l parameters must be estimated matrices are sparse! Residual/reprojection error Nonlinear estimation problem: use e.g. Levenberg-Marquard, start at the linear solution Open source libraries available, e.g. Sparse Bundle Adjustment (SBA) Lecture 3D Computer Vision AR, SLAM, Applications 13

14 Drift reduction using uncertainties Incorporate uncertainties, e.g. simple stochastic model and WLS estimation All entities modelled as Gaussian random variables Θ Lecture 3D Computer Vision AR, SLAM, Applications 14

15 3D point refinement Incorporate new camera view, each time the feature is observed in an image Methods: Repeated triangulation Recursive filtering (e.g. extended Kalman filter) Treated in lecture Computer Vision: Object and People Tracking Filter-based SLAM Lecture 3D Computer Vision AR, SLAM, Applications 15

16 SLAM with a Bayesian filter Continous image stream (image sequence) Triangulation difficult (short baseline) Matching of keypoint can drift over time SfM-based SLAM is not adapted Introduction of filtering techniques SLAM first used in robotics, with simpler sensors (ex: LIDAR) Lecture 3D Computer Vision AR, SLAM, Applications 16

17 Bayesian Tracking: the components State x t : camera position Measurement z t : image-based measurements Control input u t - in visual tracking: no control input Treated in lecture Computer Vision: Object and People Tracking State is hidden, only measurement is observed Markovian assumptions MA1: State x t depends only on previous state x t 1 MA2: Measurement z t depends only on state x t Lecture 3D Computer Vision AR, SLAM, Applications 17

18 Bayesian Tracking: derivation How to express p x t z 1:t when knowing p x t 1 z 1:t 1? p x t z 1:t = p x t z t, z 1:t 1 Treated in lecture Computer Vision: Object and People Tracking = p z t x t,z 1:t 1 p x t z 1:t 1 p z t z 1:t 1 = p z t x t p x t z 1:t 1 p(z t ) Bayes theorem Markovian assumption 1 = p z t x t p x t x t 1 p x t 1 z 1:t 1 dx t 1 p(z t ) Marginalisation (Chapman-Kolmogorov) p x t z 1:t = η p z t x t Measurement model p x t x t 1 p x t 1 z 1:t 1 dx t 1 Motion model Lecture 3D Computer Vision AR, SLAM, Applications 18

19 Bayesian Tracking: generic equation, components p x t z 1:t = η p z t x t Measurement model p x t x t 1 p x t 1 z 1:t 1 dx t 1 Motion model correct measure predict Solutions in the general case: Kalman Filter if the model is linear Gaussian Extended Kalman Filter if the model is non linear (linearization by Taylor expansion) Particle Filter in the general case In vision-based tracking, the models are not linear! Lecture 3D Computer Vision AR, SLAM, Applications 19

20 Filter-based SLAM The map (environment) has to be added in the equations: Probability of interest Motion model Measurement model Lecture 3D Computer Vision AR, SLAM, Applications 20

21 Filter-based SLAM Lecture 3D Computer Vision AR, SLAM, Applications 21

22 MonoSLAM (EKF-SLAM) MonoSLAM 1) EKF-based Filter initialization Map management ( Generate & delete features ) Prediction Prediction Measurements acquisition Measurements Acquisition Data association Update Update 1) A. J. Davison, I. D. Reid, N. D. Molton, O. Stasse, MonoSLAM: Real-Time Single Camera SLAM, IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 29, no. 6, June Lecture 3D Computer Vision AR, SLAM, Applications 22 22

23 MonoSLAM (EKF-SLAM) MonoSLAM Prediction Measurements Acquisition Update State x( t 1) x ( t 1) y 1 y2 v Prediction ˆ () q xv t W W r ( t 1) v ( t 1) t R ( t 1) q( ω ( t 1) t) W v ( t 1) R ω ( t 1) WR Dynamic System Model (Constant Velocity Model) W r () t : 3D position vector WR () () q t : orientation quaternion xv t W v () t : linear velocity vector R ω () t : angular velocity vector : landmark position vector y i Lecture 3D Computer Vision AR, SLAM, Applications 23 23

24 MonoSLAM (EKF-SLAM) MonoSLAM Prediction Measurements Acquisition Update Active search 1),2) Prediction of measurements u h ( ) ( ˆ i t hi xv( t)) v r K ˆ W () t qˆ WR () t ku y () i kv P t 1 k Find measurements T 1 For u ( uv, ) u Si u Matching the patch by NCC at h () t u i Max NCC value at Measurement h () t u i > threshold z ( t) h ( t) u i i S i : a covariance matrix for the 2D position of i th landmark 1) A. J. Davison, Active Search for Real-Time Vision, International Conference Computer Vision, ) M. Chli, A. J. Davison, Active Matching for Visual Tracking, Robotics and Autonomous Systems, 57(12): , Lecture 3D Computer Vision AR, SLAM, Applications 24 24

25 MonoSLAM (EKF-SLAM) MonoSLAM Prediction Measurements Acquisition Update Update z1( t) h1( t) x( t) xˆ ( t) K( t) zn( t) hn( t) K() t : a Kalman gain at time t 1) A. J. Davison, Active Search for Real-Time Vision, International Conference Computer Vision, ) M. Chli, A. J. Davison, Active Matching for Visual Tracking, Robotics and Autonomous Systems, 57(12): , Lecture 3D Computer Vision AR, SLAM, Applications 25 25

26 MonoSLAM (EKF-SLAM) MonoSLAM Initialization of features Delayed : SfM Undelayed : Inverse depth parameterization 1) Experiment 1.6GHz Pentium M processor 1) J. Civera, A. J. Davison, J. M. M. Montieal, Inverse Depth Parametrization for Monocular SLAM, IEEE Transactions on Robotics 24(5): , Lecture 3D Computer Vision AR, SLAM, Applications 26 26

27 Comparison SfM-based Filter-based Initialization Measurement 8-point algorithm NCC matching (from extracted feature points) KLT tracker Delayed : SfM Undelayed : Inverse depth parameterization Active search (prediction & template matching) KLT tracker Estimation technique SBA (after p3p algorithm) Kalman filtering (prediction & update) Tracking 3~400 points in a frame Working in real time within 100 landmarks Lecture 3D Computer Vision AR, SLAM, Applications 27 27

28 Demonstration Lecture 3D Computer Vision AR, SLAM, Applications 28

29 PTAM: Klein and Murray, ISMAR 2007 Title: Parallel tracking and mapping for small AR workspaces Known as PTAM system MANY features, (simple) correlation based tracking Parallel pose tracking and 3D reconstruction threads Local bundle adjustment (based on keyframes) Code, videos, papers, slides available here Lecture 3D Computer Vision AR, SLAM, Applications 29

30 Why is SLAM fundamentally harder? Frame by Frame SLAM Time One frame Find features Update camera pose and entire map Many DOF Draw graphics Lecture 3D Computer Vision AR, SLAM, Applications 30

31 Frame by frame SLAM Standard SLAM Updating entire map every frame is expensive Needs sparse map of high-quality features (A. Davison) Proposed approach Use dense map (of low quality features) Don t update the map every frame : Keyframes Split the tracking and mapping into two threads Lecture 3D Computer Vision AR, SLAM, Applications 31

32 Parallel Tracking And Mapping Proposed method - Split the tracking and mapping into two threads Time Thread #2 Mapping Update map One frame Thread #1 Tracking Find features Update camera pose only Draw graphics Lecture 3D Computer Vision AR, SLAM, Applications 32

33 Parallel Tracking and Mapping Tracking thread: Responsible estimation of camera pose and rendering augmented graphics Must run at 30 Hz Make as robust and accurate as possible Mapping thread: Responsible for providing the map Can take long time per key frame Make as rich and accurate as possible Lecture 3D Computer Vision AR, SLAM, Applications 33

34 Tracking thread Overall flow Pre-process frame Map Project points Project points Measure points Measure points Update Camera Pose Coarse stage Update Camera Pose Fine stage Draw Graphics Lecture 3D Computer Vision AR, SLAM, Applications 34

35 Pre-process frame Mono and RGB version of image 4 pyramid levels Detect FAST corners (E. Rosten et al ECC 2006) 640x x x120 80x Lecture 3D Computer Vision AR, SLAM, Applications 35

36 Pre-process frame Make for pyramid levels Detect Fast corners E. Rosten et al (ECCV 2006) 640x x x120 80x Lecture 3D Computer Vision AR, SLAM, Applications 36

37 Project Points Use motion model to update camera pose Constant velocity model Estimated current Pt+1 Previous pos Pt Previous pos Pt-1 t t Vt =(Pt Pt-1)/ t Pt+1=Pt+ t (Vt) Lecture 3D Computer Vision AR, SLAM, Applications 37

38 Project Points Choose subset to measure ~ 50 features for coarse stage 1000 randomly selected for fine stage 1000 ~50 640x x x120 80x Lecture 3D Computer Vision AR, SLAM, Applications 38

39 Measure Points Generate 8x8 matching template (warped from source keyframe:map) Search a fixed radius around projected position Use Zero-mean SSD Only search at Fast corner points Lecture 3D Computer Vision AR, SLAM, Applications 39

40 Update camera pose 6-DOF problem Obtain by SFM (Three-point algorithm) Lecture 3D Computer Vision AR, SLAM, Applications 40

41 Mapping thread Overall flow Stereo Initialization Wait for new key frame Add new map points Tracker Optimize map Map maintenance Lecture 3D Computer Vision AR, SLAM, Applications 41

42 Stereo Initialization Use five-point-pose algorithm D. Nister et. al Requires a pair of frames and feature correspondences Provides initial map User input required: Two clicks for two key-frames Smooth motion for feature correspondence Lecture 3D Computer Vision AR, SLAM, Applications 42

43 Wait for new key frame Key frames are only added if : There is a sufficient baseline to the other key frame Tracking quality is good When a key frame is added : The mapping thread stops whatever it is doing All points in the map are measured in the keyframe New map points are found and added to the map Lecture 3D Computer Vision AR, SLAM, Applications 43

44 Add new map points Aim: as many map points as possible Check all maximal FAST corners in the key frame : Check score Check if already in map Epipolar search in a neighboring key frame Triangulate matches and add to map Repeat in four image pyramid levels Lecture 3D Computer Vision AR, SLAM, Applications 44

45 Optimize map Use batch SFM method: Bundle Adjustment Adjusts map point positions and key frame poses Minimize reprojection error of all points in all keyframes (or use only last N key frames) Lecture 3D Computer Vision AR, SLAM, Applications 45

46 System and Results Environment Desktop PC (Intel Core 2 Duo 2.66 GHz) OS : Linux Language : C++ Tracking speed Total Key frame preparation Feature Projection Patch search Iterative pose update 19.2 ms 2.2 ms 3.5 ms 9.8 ms 3.7 ms Lecture 3D Computer Vision AR, SLAM, Applications 46

47 System and Results Mapping scalability and speed Practical limit 150 key frames 6000 points Bundle adjustment timing Key frames Local Bundle Adjustment 170 ms 270 ms 440 ms Global Bundle Adjustment 380 ms 1.7 s 6.9 s Lecture 3D Computer Vision AR, SLAM, Applications 47

48 Draw graphics Distorted rendering Plane estimation Lecture 3D Computer Vision AR, SLAM, Applications 48

49 Draw graphics What can we draw in an unknown scene? Assume single plane visible at start Run VR simulation on the plane Lecture 3D Computer Vision AR, SLAM, Applications 49

50 Draw graphics What can we draw in an unknown scene? Assume single plane visible at start Run VR simulation on the plane Lecture 3D Computer Vision AR, SLAM, Applications 50

51 Draw graphics What can we draw in an unknown scene? Assume single plane visible at start Run VR simulation on the plane Lecture 3D Computer Vision AR, SLAM, Applications 51

52 Draw graphics What can we draw in an unknown scene? Assume single plane visible at start Run VR simulation on the plane Lecture 3D Computer Vision AR, SLAM, Applications 52

53 Demonstration Lecture 3D Computer Vision AR, SLAM, Applications 53

54 Loop closing in SLAM Recognize previously visited location Update the beliefs accordingly Different solutions exists Bag of SIFT features Keyframe recognition Suppl. sensors Lecture 3D Computer Vision AR, SLAM, Applications 54

55 Introduction Previous lectures: Basics (camera, projective geometry) Structure From Motion Structured Light Dense 3D Reconstruction Depth Cameras Today: Insights into Advanced SLAM techniques Augmented Reality Applications of 3D Computer Vision Lecture 3D Computer Vision AR, SLAM, Applications 55

56 Augmented Reality AR is mostly based on 3D Computer Vision Camera calibration required First attempts with visual markers (based on homographies) Visual features (keypoints) (PnP problem) SLAM approaches Lecture 3D Computer Vision AR, SLAM, Applications 56

57 Calibration Matrix K 57

58 Augmented Reality Interface with rendering K Matrix Projection Matrix (e.g. opengl) R, t (pose) Modelview Matrix (e.g. opengl) Distortion parameters have to be estimated Undistort image, or Distorted rendering Visual coherence Lecture 3D Computer Vision AR, SLAM, Applications 58

59 Visual coherence Realistic integration between virtual and real Lecture 3D Computer Vision AR, SLAM, Applications 59

60 Visual coherence Requires to estimate the lighting conditions Light probe Direct estimation of camera artifacts (blur, colors) advanced 3D CV Lecture 3D Computer Vision AR, SLAM, Applications 60

61 Introduction Previous lectures: Basics (camera, projective geometry) Structure From Motion Structured Light Dense 3D Reconstruction Depth Cameras Today: Insights into Advanced SLAM techniques Augmented Reality Applications of 3D Computer Vision Lecture 3D Computer Vision AR, SLAM, Applications 61

62 3DCV: 3D reconstruction and printing Lecture 3D Computer Vision AR, SLAM, Applications 62

63 Lecture 3D Computer Vision AR, SLAM, Applications 63 Diffused Texture Appearance modeling Reference picture

64 Lecture 3D Computer Vision AR, SLAM, Applications 64

65 Measurements, planning Lecture 3D Computer Vision AR, SLAM, Applications Copyright 2013 Augmented Vision - DFKI 65 65

66 Person reconstruction for clothes industry Lecture 3D Computer Vision AR, SLAM, Applications 66

67 Gesture and HCI Lecture 3D Computer Vision AR, SLAM, Applications 67

68 We are hiring! Projects / Seminars Bachelor and Master theses Hiwi positions 3D computer vision Reconstruction 2D computer vision Lecture 3D Computer Vision AR, SLAM, Applications 68

69 : Questions + Exercises Next appointment Thanks! Lecture 3D Computer Vision AR, SLAM, Applications 69