Introduction to. Stereo Vision. Roland Brockers

Transcription

1 ME/CS 132: Introduction to Vision-based Robot Navigation Stereo Vision Roland Brockers

2 Stereo vision 3D reconstruction using cameras as passive sensors tr 2D projection of 3D world At least 2 views for reconstruction Reconstruction relative to translation t Stereo: shape from known motion between two views rb- 2

3 Motivation rb- 3

4 Stereograms Invented b Sir Charles Wheatstone t 1838 rb- 4

5 Anaglphs Public Librar Stereoscopic Looking Room Chicago b Phillips 1923 rb- 5

6 3D vision left hemifield right hemifield right nasal retina optic chiasm right temporal retina optic nerve optic tract Primar visual corte [Schmidt & Lang 2005] rb- 6

7 Random dot stereogram Bela Julesz 1971 rb- 7

8 Autostereograms Eploit disparit as depth cue using single image. (Single image random dot stereogram Single image stereogram) Image from magicee.com rb- 8

9 3D vision Left view Depth image MER rover Opportunit at Victoria Crater rb- 9

10 Two view geometr Perspective projection in homogeneous coordinates 3D points Image points 4 ] [ T W Z Y X X 1) ( ] [ 3 w w T g p Perspective projection ) ( ] [ w w X Z Y Z X Z Rigid bod motion Rigid bod motion + perspective projection 4 3 ] [ t R X t R X ] [ t RX X 1-Feb-2011 rb- 10 ME/CS 132 t R

11 Two view geometr Epipolar constraint: 1 2 and t are coplanar rb- 11

12 Insert: Vector products Scalar product cos b a b a Matri form of cross product cos b a b a b a b a a a a a a b a a b a b b a b a b a b a a a b a b a a a a a a b a b a ] [ 0 ) ( b a a 0 a 2 a 1 1-Feb-2011 rb- 12 ME/CS ) ( ) ( b a b

13 Epipolar constraint: 1 2 and t are coplanar t t R t 2 plane normal t R t R 1 t] Ε [ t ] R 1 1 [ R t t t t R1 [ T 2 Ε1 0 1 Two view geometr 0 Essential matri [Longuet-Higgings 1981] rb- 13

14 Epipolar constraint: 1 2 and t are coplanar t t R t 2 plane normal t R t R 1 t] Ε [ t ] R 1 1 [ R t t t t R1 [ 1 Two view geometr 0 Essential matri [Longuet-Higgings 1981] T 2 Ε1 0 Maps a point 1 to an epipolar line E 1 5 independent parameters (up to scale) Single moving camera: Stereo vision: t R unknown prior to 3D reconstruction do motion estimation toda's lecture t R fied and known from calibration epipolar geometr can be pre-calculated use rectification + epipolar constraint for correspondence search rb- 14

15 Stereo vision What are some possible algorithms? Feature based: match features and interpolate Edge based: match edges and interpolate Dense methods: match all piels with windows use optimization: iterative updating dnamic programming energ minimization (regularization stochastic) graph algorithms rb- 15

16 Stereo vision Image processing pipeline Image acquisition & Pre-processing Correspondence search Passive triangulation rb- 16

17 Stereo vision Image processing pipeline Image acquisition & Pre-processing Image correction (e.g. removing of lens distortion) Rectification z q w p l b p s r q l. f l fn q rekt rb- 17

18 Rectification Project each image onto same plane which is parallel to the baseline Resample lines (and shear/stretch) to minimize distortion rb- 18

19 Rectification BAD! C. Loop Z. Zhang: Computing rectifing homographies for stereo vision. CVPR 1999 rb- 19

20 Rectification GOOD! C. Loop Z. Zhang: Computing rectifing homographies for stereo vision. CVPR 1999 rb- 20

21 Stereo vision Image processing pipeline Image acquisition & Pre-processing Correspondence search Difficulties due to ambiguities or occlusions Occlusions left view right view rb- 21

22 Stereo vision Image processing pipeline Image acquisition & Pre-processing Correspondence search Passive triangulation rb- 22

23 Passive triangulation o l l b s Projektionszentrum of Center projection z w o w w optische Principal ais Achse p w Disparit: Range: d z w w l b s ( l r ) 2d f d b s r o r r f Image Bildebenen planes focal length rb- 23

24 Passive triangulation Beware of triangulation error due to match uncertaint (alias effects) Because of missalignment reconstructed (blue) ras usuall don t intersect! rb- 24

25 Stereo vision Image processing pipeline Image acquisition & Pre-processing Correspondence search Passive triangulation rb- 25

26 Correspondence search Feature based: Etract feature points Match features using outlier rejection (epipolar constraint) Triangulate rb- 26

27 Correspondence search Edge based: Etract edge contours and match edge segments (edgels) R. Szeliski R. Weiss: Robust shape recover from occluding contours using a linear smoother. IJCV 1998 rb- 27

28 Correspondence search Image based: Find correspondes for ever piel in reference image? Looks difficult? Use stereostopic constraints! rb- 28

29 Correspondence search Image based: Find correspondes for ever piel in reference image? Epipolar constraint q w z p l b s p r q rekt q l f l. f n f p rb- 29

30 Correspondence search Image based: Find correspondes for ever piel in reference image? Epipolar constraint Limited disparit range No transparent objects: Uniqueness constraint [Marr & Poggio 1976] Solid objects: Continuit constraint [Marr & Poggio 1976] Disparit gradient limit (humans: < 1.0) [Burt & Julesz 1980] Order constraint rb- 30

31 Basic algorithm For each epipolar line For each piel in the reference image compare with each piel on same epipolar line within search range pick piel with minimum match cost Improvement: Use match windows! rb- 31

32 Eample algorithm: SAD real time stereo Uses a Sum of Absolute Differences (SAD) to calculate matching scores Runs on man mobile robot platforms including the MER eploration rovers on Mars L R rb- 32

33 SAD real time stereo SAD disparit range ma. disparit min. disparit rb- 33

34 SAD real time stereo SAD Calculate scores for all piels in the reference view and all displacements [d min d ma ] c SAD Correlation score rb- 34

35 SAD real time stereo SAD Find the corresponding piel with minimum SAD score And the winner takes all! c SAD Correlation score rb- 35

36 SAD real time stereo SAD But what do we do here? c SAD Correlation score rb- 36

37 SAD real time stereo SAD 77 correlation windows How can we get better results? Image pre-filtering to enhance contrast and reduce noise Post processing to clean-up 1-Feb-2011 ME/CS 132 near far rb- 37

38 Pre filter SAD 77 correlation windows DOB Differences of boes (DOB) bandpass filter Average I over small and large window and subtract DOB( ) mean( I( )) mean( I( )) w small w large - Crude approimation of a Laplace filter (LoG) But ver fast! Think of it as a background subtraction + noise reduction rb- 38

39 Pre filter SAD 77 correlation windows DOB pre-filter near 1-Feb-2011 ME/CS 132 far rb- 39

40 Post filter Left to right consistenc check Make sure the matching result is smmetrical: Check if a piel in the reference view p L for which we found a winning correspondence partner in the other view p R is also the winning piel for the correspondence search of p R. c L c R p R p L SAD C l ti SAD Correlation score for p L right image SAD Correlation score for p R left image p L p R rb- 40

41 Post filter Curvature check: Look at the curvature at the peak of the correlation function If the curvature is too flat we don t have a high confidence in the winning disparit reject this piel Simple measure for SAD curvature: curv ( ) c ( 1 ) 2c ( ) c ( 1 ) k c L c L p 1 Failed p 2 Passed rb- 41

42 Sub-piel precision Use bilinear interpolation to estimate the minimum position d c( 1) (1) 2( c(1) c( 1) 2c(0)) * c rb- 42

43 Post filter SAD 77 correlation windows DOB pre-filter LR & curv check near 1-Feb-2011 ME/CS 132 far rb- 43

44 Post filter Blob filter Eliminate small blobs under a specific size Left view Left-right consistenc check Blob filter How can we further improve our stereo result? Let s get complicated! rb- 44

45 Optimization methods Optimize the correlation values before picking the winning disparit P P d How do we optimize? d ma define an energ function E E f ( s ( d )) d min pick the solution for which the energ is minimal Disparit space image s(d) rb- 45

46 Optimization methods Local Optimization: Optimize i the disparit it of ever piel independentl d Semi-global optimization: Optimize i the disparit it for groups of piels Global optimization: Optimize i the disparit it for the whole image simultaneousl l rb- 46

47 Optimization methods Local Optimization: Fied matching window SAD scores: E d) I ( u d v) I ( u v) ( L R ( u v) w( ) How big should the neighborhood be? rb- 47

48 Optimization methods Smaller neighborhood: more details Larger neighborhood: fewer isolated mistakes w = 3 w = 20 rb- 48

49 Optimization methods Local Optimization: Shifted windows Pick the minimum matching score of all surrounding windows Color based adaptive windows adaptive weights based on color similarit segmentation based rb- 49

50 Optimization methods Semi-global optimization: Dnamic Programming 1D simultaneous optimization of disparities for each scan line Right scan line Left scan line M: match L: left image occlusion R: right image occlusion 18-Jan-2011 ME/CS 132 rb- 50

51 Dnamic Programming Search for the minimum cumulative cost path in each scan line Matching cost c ij. E.g. SSD: c ij I ( i) I ( j) 2 L R 2D energ function for potential ti correspondence pair E( i j) min( E( i 1 j 1) cij E( i 1 j) q0 E( i j 1) q0) Start condition: E( 11) e 11 No disparit change disparit disparit changes changes b +1 b -1 Initial matching scores Penalt for change 18-Jan-2011 ME/CS 132 rb- 51

52 Dnamic Programming 18-Jan-2011 ME/CS 132

53 Dnamic Programming 18-Jan-2011 ME/CS 132

54 Dnamic Programming Sample results (Note horizontal artifacts) A. F. Bobick S. S. Intille: Large occlusion stereo. IJCV Jan-2011 ME/CS 132 rb- 54

55 Optimization methods Global optimization: Graph Cut Graph Cut 2D simultaneous optimization for whole image 3D global energ function ) ( ) ( ) ( d E d E d E smoothness data ) ( ) ( )) ( ( ) ( w d v u f d E d c f d E d data Initial matching scores ) ( ) ( w d v u f d E d w u v smoothness Connection with neighbors 18-Jan-2011 ME/CS 132 rb- 55 Co ect o t e g bo s

56 Graph Cut Constructs a 3D graph with connections between neighboring cells Connection weights defined b global energ function E Cut finds the minimum energ surface 6 connections cut background foreground 18-Jan-2011 ME/CS 132 rb- 56

57 Graph Cut Iterative optimization update of energ function (-( swap epansion p modif smoothness penalt based on edges) Compute best piel precise disparit Input image Sum Abs Diff Graph cuts 18-Jan-2011 ME/CS 132 rb- 57

58 Cooperative Stereo Global 3D energ function ) ( ) ( ) ( d E d E d E ) ( ) ( ) ( d E d E d E smoothness data ) ( ) ( d d d data c e f d E ) ( ) ( d U w u v w v d u d smoothness d e e f d E Local support area Iterativel update the energ for each piel Local support area ) ( 0 1 U i U i i i e c e f e e 1-Feb-2011 rb- 58 ME/CS 132 When converged pick the disparit for the minimum score for each piel

59 Cooperative Stereo Local support area U inside a constant disparit level (2D-window function) inside disparit space (3D-window function) d d d 2D-support area 3D-support area Interpretation of the coupled variable sstem as a neural network [Marr & Poggio 1976] [Reimann & Haken 1994] rb- 59

60 Cooperative Stereo rb- 60

61 Stereo algorithms Ver active research field for more than 30 ears Lots of other methods: Baesian inference Markov Random Fields Relaation methods Belief Propagation Laered stereo rb- 61

62 Stereo evaluation rb- 62

63 Stereo best algorithms rb- 63

64 Other methods Illumination based stereo matching Li Zhang Brian Curless and Steven M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dnamic Programming. 3DPVT 2002 Lubomir Zagorchev and A. Ardeshir Goshtasb: A paint-brush laser range scanner. rb- 64

65 Bibliograph Chapter 11 of Szeliski s book R. Hartle A. Zisserman: Multiple view geometr in computer vision. Cambridge Universit Press Cambridge (good book about image based reconstruction) B. Julesz: Foundations of cclopean perception. Universit of Chicago Press Chicago (Random dot stereograms) H. Hirschmüller P. R. Innocent J. Garabaldi: Real-time correlation-based stereo vision with reduced border error. In: International Journal of Computer Vision vol. 47 (1-3) 2002 pp (Real-time SAD stereo algorithm) D. Scharstein and R. Szeliski. A taonom and evaluation of dense two-frame stereo correspondence algorithms. International Journal of Computer Vision i 47(1): (Algorithm evaluation) A.F. Bobick and S.S. Intille. Large occlusion stereo. Int Journal of Computer Vision 33(3) pp (Dnamic programming) Ro & Co: A Maimum-Flow Formulation of the N-camera Stereo Correspondence Problem. In: Proc. ICCV 1998 pp (Graph Cut) V. Kolmogorov R. Zabih: Computing visual correspondence with occlusions using graph cuts. In: Proceedings of the IEEE International Conference on Computer Vision (ICCV 01) Vancouver Kanada 2001 pp (Graph Cut) D. Marr T. Poggio: Cooperative computation of stereo disparit. In: Science vol pp pp (Biologicall inspired cooperative stereo algorithm) D. Reimann H. Haken: Stereovision b self-organisation. In: Biological Cbernetics vol pp (Biologicall inspired cooperative stereo algorithm) R. Brockers: Cooperative Stereo Matching with Color-Based Adaptive Local Support. CAIP (Cooperative stereo algorithm) Li Zhang Brian Curless and Steven M. Seitz. Rapid Shape Acquisition Using Color Structured Light and Multi-pass Dnamic Programming. In Proceedings of the 1st International Smposium on 3D Data Processing Visualization and Transmission (3DPVT) Padova Ital June pp (Structured light stereo) D. Scharstein R. Szeliski: High-accurac stereo depth maps using structured light. In: Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR 03) Madison WI USA 2003 pp (Structured light stereo) L. Zagorchev A. Goshtasb: A paintbrush laser range scanner. Computer Vision and Image Understanding vol. 101 pp (Laser based stereo) 18-Jan-2011 ME/CS 132 rb- 65