Computer Vision I. Dense Stereo Correspondences. Anita Sellent 1/15/16

Transcription

1 Computer Vision I Dense Stereo Correspondences Anita Sellent

2 Stereo Two Cameras Overlapping field of view Known transformation between cameras From disparity compute depth [ Bradski, Kaehler: Learning OpenCV, O'Reilly 2008] 2

3 Triangulation: Top View fb fb Z= = xl x r d d =x l x r disparity B B x l +x r = Z Z f Z xl xr f f C1 B C2 3

4 Triangulation: Top View C1 B C2 4

5 Epipolar Lines

6 Epipolar Lines Epipolar Lines ( see CV 1)

7 Epipolar Lines Epipolar Lines ( see CV 1)

8 Epipolar Lines Epipolar Lines ( see CV 1)

9 Practically Cameras are never perfectly aligned. The lenses of the cameras are never perfect.

10 Practically Cameras are never perfectly aligned. The lenses of the cameras are never perfect. e om s ide v o Pr atic m the a m p! l e al h

11 Practically Image undistortion Remove radial and tangential lens distortion Stereo calibration Compute the geometrical relationships between cameras in space Stereo rectification Transform individual images so that they appear as if the had seem taken by two row aligned image planes 16

12 Practically Image undistortion Remove radial and tangential lens distortion x=g 1( ^x, ^y )= ^x (1+K 1 r +K 2 r + )+(P 2 (r +2 x^ )+2 P1 x^ ^y )(1+K 1 r +K 2 r + ) y =g 2 ( x^, ^y ) 17

13 Practically Image undistortion Remove radial and tangential lens distortion s=g 1 (x, y)=x (1+K 1 r 2 +K 2 r 4 + )+(P2 (r 2+2 x 2 )+2 P1 x y )(1+K 1 r 2 +K 2 r 4 + ) t=g 2( x, y ) Undistorted position Distorted position 18

14 Geometric Image Transformations T Coordinates: T (x, y ) (x, y) I (s, t) Image: 19

15 Image Interpolation Schemes Nearest Neighbor Approach ( x, y ) Bilinear Interpolation (x, y) 20

16 Practically Image undistortion Stereo calibration Stereo rectification Planar Strong features 21

17 A Stereo Vision System 22

18 Calibration (offline) Rectification Stereo pair Stereo Correspondences Rectified Stereo pair Disparity map Triangulation Depth map Intrinsic and extrinsic parameter A Stereo Vision System PC, FPGA 23

19 Disparity Estimation: Assumption I The color of a pixel does not change with the viewpoint 24

20 Quiz: Which Points Satisfy Color Constancy? : None of the above 25

21 Disparity Estimation: Assumption I The color of a pixel does not change with the viewpoint Noise Reflective Material Transparent Material Scattering Effects Pixels that are visible only in one camera 26

22 Disparity Estimation: Assumption I The color of a pixel does not change with the viewpoint Noise Reflective Material Transparent Material Scattering Effects Pixels that are visible only in one camera 27

23 Disparity Estimation: Assumption I The color of a pixel does not change with the viewpoint I l (x, y) I r ( x d, y ) 28

24 Disparity Estimation: Assumption I There are 256 different gray values in an image. This image has 450 pixels per line. 29

25 Disparity Estimation: Assumption II Neighboring pixels belong to the same surface d (x, y ) d (x +ϵx, y +ϵ y ) 30

26 Disparity Estimation: Assumption II Neighboring pixels belong to the same surface Neighboring pixels belong to a fronto-paralell plane d (x, y )=d (x +Δ x, y +Δ y ) 31

27 A Basic Stereo Algorithm: Blockmatching For each pixel For each disparity value Compare patch similarity Assign disparity with highest similarity 32

28 A Basic Stereo Algorithm: Blockmatching For each pixel For each disparity value Compare patch similarity Assign disparity with highest similarity 33

29 Sparse vs. Dense Depth Estimation II Sparse Feature Matches (e.g. Harris corners ) Highly distinguishable texture Region descriptor robust to scale, rotation,... Dense Correspondences For every pixel Search along epipolar line 34

30 A Basic Stereo Algorithm: Blockmatching For each pixel For each disparity value Compare patch similarity Assign disparity with highest similarity 35

31 Discretization of the Solution Space Pros: Can use fast optimization algorithms Restrict need for image interpolation More flexibility in similarity measures Cons: Only certain distances become measurable 36

32 Discretization of the Solution Space Assume limited range of discrete disparities 37

33 Disparity fb fb Z= = xl x r d d =x l x r disparity Z C1 C2 disparity 38

34 Discretization of the Solution Space Assume limited range of discrete disparities Discrete estimates Ground truth 39

35 A Basic Stereo Algorithm: Blockmatching For each pixel For each disparity value Compare patch similarity Assign disparity with highest similarity 40

36 What are good Similarity Measures? Sum of Squared differences ( on a window ) Sum of Absolute differences ( on a window ) Any robust function ( on a window ) Any robust function on a brightness dependent window Cross Correlation (Normalized ) Census Transform with Hamming distance... [ ] 41

37 Image Noise Photon/ Shot noise Dark noise Photons arrive at the sensor at random times Thermal activity leads to random signals Read noise Charge conversion introduces random error ^I (x, y)=i (x, y)+ n(x, y) n N (0, σ) 42

38 Gaussian Image Noise Noise for each pixel in each camera is independent I^l (x, y)=i l (x, y)+nl (x, y ) I^r (x, y )=I r (x, y)+nr ( x, y ) Difference of images with disparity I^l (x, y) I^r ( x d, y )=I l (x, y ) I r (x d, y )+n (x, y) Probability Pr (I l, I r d, θ)= ( x, y) N 1 exp ( σ 2 π 2 (I l (x ) I r (x d )) 2σ 2 ) Negative logarithm E= log( Pr (I l, I r d, θ))= x N 1 2 ( I (x) I (x d )) +C l r 2 2σ 43

39 Example: Line Fitting Given: a collection of data-points Task: fit a line to the data-points {(x i, y i)}i=1... N [ Forsyth, Ponce(2003): Computer Vision, Ch. 15] 44

40 Example: Line Fitting Given: a collection of data-points Task: fit a line to the data-points {(x i, y i)}i=1... N x = u +λ n x y v ny ()() ( ) λ N (0, σ) [ Forsyth, Ponce(2003): Computer Vision, Ch. 15] 45

41 Example: Line Fitting {(x i, y i)}i=1... N Given: a collection of data-points Task: fit a line to the data-points Approach: Gaussian noise in normal direction x = u +λ n x y v ny ()() ( ) λ N (0, σ) [ Forsyth, Ponce(2003): Computer Vision, Ch. 15] 46

42 Example: Line Fitting Reminder: Express lines as 2 L={(x, y) ℝ x n x + y n y =d } nx ny () d [ Forsyth, Ponce(2003): Computer Vision, Ch. 15] 47

43 Example: Line Fitting Reminder: Express lines as Likelihood function 2 L={(x, y) ℝ x n x + y n y =d } Pr (x nx, n y, d )= Pr (x i, y i n x, n y,d ) 2 i (nx x i +n y y i d ) 1 = exp( ) 2 i σ 2 π 2σ nx ny () d [ Forsyth, Ponce(2003): Computer Vision, Ch. 15] 48

44 Example: Line Fitting Reminder: Express lines as Likelihood function 2 L={(x, y) ℝ x n x + y n y =d } Pr (x nx, n y, d )= Pr (x i, y i n x, n y,d ) i Max likelihood nx ny () E= (n x x i +n y y i d) 2 i 2 2 n x +n y =1 d [ Forsyth, Ponce(2003): Computer Vision, Ch. 15] 49

45 Example: Line Fitting [ Forsyth, Ponce(2003): Computer Vision, Ch. 15] 50

46 Similarity Measures Do brightness differences have a normal distribution? (Is noise our only problem? ) 51

47 Similarity Measures Do brightness differences have a normal distribution? Gaussian image noise ^I (x, y)=i (x, y)+ n(x, y) Lecture : Optical Flow 52

48 Similarity Measures Do brightness differences have a normal distribution? Gaussian image noise Color constancy assumption Lecture : Optical Flow 53

49 Similarity Measures Do brightness differences have a normal distribution? Gaussian image noise Color constancy assumption Inadequate window sizes & occlusion Lecture : Optical Flow 54

50 Example: Line Fitting [ Forsyth, Ponce(2003): Computer Vision, Ch. 15] Lecture : Optical Flow 55

51 Example: Line Fitting Zoom in from left image [ Forsyth, Ponce(2003): Computer Vision, Ch. 15] Lecture : Optical Flow 56

52 Example: Line Fitting How can we de-emphasize the influence of distant outliers? Assume non-gaussian distributions of error use non-quadratic error functions for the log-likelihood Pr (x nx, n y, d )= Pr (x i, y i n x, n y,d ) i 1 = exp( i σ 2 π 2 (nx x i +n y y i d ) 2σ 2 ) E= log( Pr (x n x, n y, d ))=C 1 (n x xi + n y y i d) +C 2 2 i Lecture : Optical Flow 57

53 Example: Line Fitting How can we de-emphasize the influence of distant outliers? Assume non-gaussian distributions of error use non-quadratic error functions for the log-likelihood Pr (x nx, n y, d )= Pr (x i, y i n x, n y,d ) i 1 = exp( i σ 2 π 2 (nx x i +n y y i d ) 2σ 2 ) E= log( Pr (x n x, n y, d ))=C 1 (n x xi + n y y i d) +C 2 2 i =C1 ρ(n x xi +n y y i d)+ C2 i Lecture : Optical Flow 58

54 Robust Error Function Desirable properties Inliers should be punished approximately quadratically they follow a Gaussian distribution Outliers should have small/ no influence they are more probable than under Gaussian distribution Lecture : Optical Flow 59

55 Robust Error Function Influence function Assume here differentiable error function How much differs the energy with the location of datapoint? ^ x)=ρ(nx x +n y y d ) E( ρ ϕ= x Lecture : Optical Flow 60

56 Error Functions Quadratic Error function 2 x ρ(x σ)= 2 2σ Influence function x ϕ( x σ )= 2 σ Probability distribution 2 1 x Pr (x σ)= exp ( ) 2 Z 2σ Lecture : Optical Flow 61

57 Error Functions Linear Error function ρ(x σ)= x Influence function Probability distribution ϕ( x σ )= sign( x) 1 Pr (x σ)= exp ( ρ) Z Lecture : Optical Flow 62

58 Error Functions Huber Error function Influence function x ϵ 2 ρ(x σ)= x ϵ + x ϵ 2ϵ 2 x else ϕ( x σ )= Probability distribution x ϵ sign( x) else Lecture : Optical Flow 1 Pr (x σ)= exp ( ρ) Z 63

59 Error Functions Lorentian Student's t distribution Error function 1 2 ρ(x σ)=log(1+ 2 x ) 2σ Influence function 2x ϕ( x σ )= 2 2 2σ + x Lecture : Optical Flow Probability distribution x 2 α Pr (x σ)=(1+ 2 ) 2σ 64

60 Error Functions: Comparisons quadratic Huber linear Lorentian 2 x ρ(x σ)= 2 2 σ +x Geman McClure Lecture : Optical Flow 65

61 Error Distributions: Comparisons quadratic linear Huber Lorentian Lecture : Optical Flow 66

62 Example: Line Fitting Correct choice of scale parameter σ Lecture : Optical Flow 67

63 Example: Line Fitting σ too small all data points are weighted down Lecture : Optical Flow 68

64 Example: Line Fitting σ too large all data points are considered Lecture : Optical Flow 69

65 Convex Functions Any local minimum of a convex function is a global minimum Lecture : Optical Flow 70

66 Quiz: Which Error Function is Convex? 2 1 quadratic Huber linear 3 ρ(x σ)=log(1+ Lorentian x) 2 2σ x2 ρ(x σ)= 2 2 σ +x Geman McClure 5 : None 6 : all Lecture : Optical Flow 71

67 Disparity Space Image C (x, y, d ) d y x 72

68 Disparity Space Image d y x 73

69 A Basic Stereo Algorithm: Blockmatching For each pixel For each disparity value Compare patch similarity Assign disparity with highest similarity Result: Noisy, no smooth surfaces 74