Depth from Stereo. Sanja Fidler CSC420: Intro to Image Understanding 1/ 12

Transcription

1 Depth from Stereo Sanja Fidler CSC420: Intro to Image Understanding 1/ 12

2 Depth from Two Views: Stereo All points on projective line to P map to p Figure: One camera Sanja Fidler CSC420: Intro to Image Understanding 2/ 12

3 Depth from Two Views: Stereo All points on projective line to P in left camera map to a line in the image plane of the right camera Figure: Add another camera Sanja Fidler CSC420: Intro to Image Understanding 2/ 12

4 Depth from Two Views: Stereo If I search this line to find correspondences... Figure: If I am able to find corresponding points in two images... Sanja Fidler CSC420: Intro to Image Understanding 2/ 12

5 Depth from Two Views: Stereo I can get 3D! Figure: I can get a point in 3D by triangulation! Sanja Fidler CSC420: Intro to Image Understanding 2/ 12

6 Stereo Epipolar geometry Case with two cameras with parallel optical axes General case Sanja Fidler CSC420: Intro to Image Understanding 3/ 12

7 Stereo Epipolar geometry Case with two cameras with parallel optical axes First this General case Sanja Fidler CSC420: Intro to Image Understanding 3/ 12

8 We assume that the two calibrated cameras (we know intrinsics and extrinsics) are parallel, i.e. the right camera is just some distance to the right of left camera. We assume we know this distance. We call it the baseline. Sanja Fidler CSC420: Intro to Image Understanding 4/ 12

9 Pick a point P in the world Sanja Fidler CSC420: Intro to Image Understanding 4/ 12

10 Points O l, O r and P (and p l and p r ) lie on a plane. Since two image planes lie on the same plane (distance f from each camera), the lines O l O r and p l p r are parallel. Sanja Fidler CSC420: Intro to Image Understanding 4/ 12

11 Since lines O l O r and p l p r are parallel, and O l and O r have the same y, then also p l and p r have the same y: y r = y l! Sanja Fidler CSC420: Intro to Image Understanding 4/ 12

12 So all points on the projective line O l p l project to a horizontal line with y = y l on the right image. This is nice, let s remember this. Sanja Fidler CSC420: Intro to Image Understanding 4/ 12

13 Another observation: No point from O l p l can project to the right of x l in the right image. Why? Sanja Fidler CSC420: Intro to Image Understanding 4/ 12

14 Because that would mean our image can see behind the camera... Sanja Fidler CSC420: Intro to Image Understanding 4/ 12

15 Since our points p l and p r lie on a horizontal line, we can forget about y l for a moment (it doesn t seem important). Let s look at the camera situation from the birdseye perspective instead. Let s see if we can find a connection between x l, x r and Z (because Z is what we want). [Adopted from: J. Hays] Sanja Fidler CSC420: Intro to Image Understanding 5/ 12

16 We can then use similar triangles to compute the depth of the point P [Adopted from: J. Hays] Sanja Fidler CSC420: Intro to Image Understanding 5/ 12

17 We can then use similar triangles to compute the depth of the point P Sanja Fidler CSC420: Intro to Image Understanding 5/ 12

18 We can then use similar triangles to compute the depth of the point P Sanja Fidler CSC420: Intro to Image Understanding 5/ 12

19 For each point p l =(x l, y l ), how do I get p r =(x r, y r )? Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

20 For each point p l =(x l, y l ), how do I get p r =(x r, y r )? By matching on line y r = y l. Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

21 For each point p l =(x l, y l ), how do I get p r =(x r, y r )? By matching on line y r = y l. Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

22 For each point p l =(x l, y l ), how do I get p r =(x r, y r )? By matching. Patch around (x r, y r )) should look similar to the patch around (x l, y l ). Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

23 For each point p l =(x l, y l ), how do I get p r =(x r, y r )? By matching. Patch around (x r, y r )) should look similar to the patch around (x l, y l ). Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

24 For each point p l =(x l, y l ), how do I get p r =(x r, y r )? By matching. Patch around (x r, y r )) should look similar to the patch around (x l, y l ). Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

25 For each point p l =(x l, y l ), how do I get p r =(x r, y r )? By matching. Patch around (x r, y r )) should look similar to the patch around (x l, y l ). Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

26 For each point p l =(x l, y l ), how do I get p r =(x r, y r )? By matching. Patch around (x r, y r )) should look similar to the patch around (x l, y l ). Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

27 For each point p l =(x l, y l ), how do I get p r =(x r, y r )? By matching. Patch around (x r, y r )) should look similar to the patch around (x l, y l ). Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

28 For each point p l =(x l, y l ), how do I get p r =(x r, y r )? By matching. Patch around (x r, y r )) should look similar to the patch around (x l, y l ). Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

29 Version 2015: Can I do this task even better? Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

30 Version 2015: Train a classifier! How can I get ground-truth? Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

31 Version 2015: Train a Neural Network classifier! [J. Zbontar and Y. LeCun: Computing the Stereo Matching Cost with a Convolutional Neural Network. CVPR 15] Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

32 Version 2015: Train a Neural Network classifier! To get the most amazing performance Figure: Performance on KITTI (metrics is error, so lower is better) Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

33 For each point p l =(x l, y l ), how do I get p r =(x r, y r )? By matching. Patch around (x r, y r )) should look similar to the patch around (x l, y l ). Sanja Fidler CSC420: Intro to Image Understanding 6/ 12

34 We get a disparity map as a result Sanja Fidler CSC420: Intro to Image Understanding 7/ 12

35 We get a disparity map as a result Sanja Fidler CSC420: Intro to Image Understanding 7/ 12

36 Depth and disparity are inversely proportional Sanja Fidler CSC420: Intro to Image Understanding 7/ 12

37 Smaller patches: more detail, but noisy. Bigger: less detail, but smooth Sanja Fidler CSC420: Intro to Image Understanding 7/ 12

38 You Can Do It Much Better... With Energy Minimization on top, e.g., a Markov Random Field (MRF) K. Yamaguchi, D. McAllester, R. Urtasun, E cient Joint Segmentation, Occlusion Labeling, Stereo and Flow Estimation, ECCV2014 Paper: Code: Sanja Fidler CSC420: Intro to Image Understanding 8/ 12

39 You Can Do It Much Better... [K. Yamaguchi, D. McAllester and R. Urtasun, ECCV 2014] Occlusion Hinge Coplanar Disparity)image Flow)image Sanja Fidler CSC420: Intro to Image Understanding 9/ 12

40 Look at State-of-the-art on KITTI Where Ours means: [K. Yamaguchi, D. McAllester and R. Urtasun, ECCV 2014] How can we evaluate the performance of a stereo algorithm? Stereo Flow wsgm [Spangenberg,,et,al,,2013] AARBM [Einecke,,et,al,,2014] PR4Sceneflow [Vogel,,et,al,,2013] PCBP [Yamaguchi,,et,al,,2012] PR4Sf+E [Vogel,,et,al,,2013] StereoSLIC [Yamaguchi,,et,al,,2013] PCBP4SS [Yamaguchi,,et,al,,2013] Ours,(Stereo) VC4SF [Vogel,,et,al,,2014] Ours,(Joint) 2.83% 4.97% 4.86% 4.36% 4.04% 4.02% 3.92% 3.40% 3.39% 3.05% BTF4ILLUM [Demetz,,et,al,,2014] TGV2ADCSIFT [Braux4Zin,,et,al,,2013] NLTGV4SC [Ran_l,,et,al,,2014] MoMonSLIC [Yamaguchi,,et,al,,2013] PR4Sceneflow [Vogel,,et,al,,2013] PCBP4Flow [Yamaguchi,,et,al,,2013] PR4Sf+E [Vogel,,et,al,,2013] Ours,(Flow) Ours,(Joint) VC4SF [Vogel,,et,al,,2014] 6.52% 6.20% 5.93% 3.91% 3.76% 3.64% 3.57% 3.38% 2.82% 2.72% Error,>,3,pixels,(Non4Occluded) Error,>,3,pixels,(Non4Occluded) Autonomous driving dataset KITTI: Sanja Fidler CSC420: Intro to Image Understanding 10 / 12

41 From Disparity We Get... Depth: Once you have disparity, you have 3D Figure: K. Yamaguchi, D. McAllester and R. Urtasun, ECCV 2014 Sanja Fidler CSC420: Intro to Image Understanding 11 / 12

42 From Disparity We Get... Money ;) Sanja Fidler CSC420: Intro to Image Understanding 11 / 12

43 Stereo Epipolar geometry Case with two cameras with parallel optical axes General case Next time Sanja Fidler CSC420: Intro to Image Understanding 12 / 12