Stereo Matching! Christian Unger 1,2, Nassir Navab 1!! Computer Aided Medical Procedures (CAMP), Technische Universität München, Germany!!

Transcription

1 Stereo Matching Christian Unger 12 Nassir Navab 1 1 Computer Aided Medical Procedures CAMP) Technische Universität München German 2 BMW Group München German

2 Hardware Architectures. Microprocessors Pros: floating-point SIMD C/C Cons: power-consumption cost size Low Power/Low Cost Processors Pros: low power low cost C/C Cons: mostl no floating-point mostl no SIMD GPUs Pros: processing power Cons: difficult programming power-consumption cost size FPGA Pros: efficient low power low cost Cons: code is ver specifc VHDL/Verilog) Stereo Matching - Christian Unger 2

3 Challenges: Photometric Variations. Stereo Matching - Christian Unger 3

4 Challenges: Image Sensor Noise. Stereo Matching - Christian Unger 4

5 Challenges: Specularities. Stereo Matching - Christian Unger 5

6 Challenges: Foreshortening and the Uniqueness Constraint.? Stereo Matching - Christian Unger 6

7 Challenges: Perspective Distortions. Stereo Matching - Christian Unger 7

8 Challenges: Tetureless Regions.? Stereo Matching - Christian Unger 8

9 Challenges: Repetitive Structures and Tetures. Stereo Matching - Christian Unger 9

10 Challenges: Reflections. Stereo Matching - Christian Unger 10

11 Challenges: Reflections. Stereo Matching - Christian Unger 11

12 Challenges: Transparenc. Stereo Matching - Christian Unger 12

13 Challenges: Occlusions. Stereo Matching - Christian Unger 13

14 Challenges: Occlusions. Stereo Matching - Christian Unger 14

15 Taonom of Stereo Matching. Most stereo algorithms are composed of the following steps: 1. Preprocessing 2. Matching cost computation 3. Cost aggregation 4. Disparit computation 5. Disparit refinemen There are mainl two classes of algorithms: 1. Local methods simple search ): perform Global methods e.g. minimizing a global energ functional): perform 2 3 ) 4 Stereo Matching - Christian Unger 15

16 Preprocessing. To alleviate sensor noise and photometric distortions tpicall the following methods are emploed: Laplacian of Gaussian LoG) filtering Kanade et al. Development of a Video- Rate Stereo Machine. IROS 1995.) Histogram Equalization/Matching Subtraction of mean values computed in the neighbours of each piel Faugeras et al. Real-Time correlation-based stereo: Algorithm Implementation and Applications INRIA TR ) Bilateral filtering Ansar et al. Enhanced real-time stereo using bilateral filtering. CVPR 2004.) Stereo Matching - Christian Unger 16

17 Matching Cost. A matching cost measures the similarit of piels. Simple eamples: Absolute intensit difference AD) Squared intensit difference SD) I ) I ) L I ) I )) L R R 2 Stereo Matching - Christian Unger 17

18 Disparit Computation. The corresponding piel is chosen in a wa such that the similarit between the piels is high dissimilarit = low cost). Simple Winner Takes All -Algorithm: For ever piel select the disparit with lowest cost I ) I d ) L R Matching Cost d d ma Stereo Matching - Christian Unger 18

19 Eample Algorithm. Using this simple algorithm the disparit map looks like this: Left Camera Image Optimal Result Actual Result Bad) The disparit map is ver nois due to a low signal-to-noise ratio SNR). Remed: Cost Aggregation: Do not compare single piels but small patches. Stereo Matching - Christian Unger 19

20 Stereo Matching - Christian Unger 20 Cost Aggregation. Use a matching window around the piel of interest. Sum of absolute intensit differences SAD): W ) ) ) L R d I I

21 Stereo Matching - Christian Unger 21 Cost Aggregation. Eamples for such area-based matching costs: Sum of absolute differences SAD) Sum of squared differences SSD) Normalized Cross Correlation NCC) Mutual Information W ) ) ) L R d I I W ) 2 )) ) L R d I I

22 Cross Correlation WTA with SAD). Again we use the Winner-Takes-All -algorithm as described before but now with an areabased matching cost SAD). Left Camera Image Optimal Result Actual Result Still man errors) Better than before but still not optimal. Wh? Stereo Matching - Christian Unger 22

23 Problems with Fied Windows. The area-based approach has other problems: Assumes constant depth within the window this implicit assumption is violated at Depth discontinuities Slanted/non-planar surfaces Repetitive tetures Uniform areas Thin structures window larger than the structure) Stereo Matching - Christian Unger 23

24 Problems with Slanted Surfaces in Detail. z actual z ideal Implicitl flat fronto-parallel surfaces are assumed in most area-based approaches. Ideall the matching window should take the perspective distortion into account too. But this is a chicken and egg problem : We would need the depth for that but depth is actuall what we want to determine Stereo Matching - Christian Unger 24

25 Problems with Discontinuities in Detail. Left image Right image Background is misaligned The problem is that the matching windows cover piels which lie at different depths. => This leads to wrong vales in the matching costs. => Occluded piels should not be compared to an other piel. Possible remedies: Split the matching window into multiple parts Multiple Windows : Hirschmüller et al. Real-Time Correlation-Based Stereo Vision with Reduced Border Errors. IJCV 2002.). Shiftable windows change the center piel of the windows dnamicall) Stereo Matching - Christian Unger 25

26 Problems with Uniform and Repetitive Areas in Detail.? In such cases there are usuall man weak minima of the matching cost and image sensor noise easil leads to wrong matches. Possible remedies: Use larger matching windows adaptivel Veksler. Fast variable windows using integral images. CVPR 2003.). Measure the significance of the minimum are there other minima with similar costs) and detect weak minima. Stereo Matching - Christian Unger 26

27 Summar: Cross Correlation. However despite of the drawbacks of area-based approaches cross correlation WTA with SAD) is often adopted in practice. It is simple fast Real-Time on standard hardware) and has low memor requirements. Memor requirement is low because we need no additional information eecpt the disparit for ever piel. How about eecution time? Currentl: 1400 ms for piels Stereo Matching - Christian Unger 27

28 Improvement: Integral Images. TODO: include or eclude Integral Imges? 1 Slide: Integral Images Stereo Matching - Christian Unger 28

29 Improvement: Bo Filtering. Based on a mathematical observation that man matching costs can be separated such that the can be computed incrementall. If we know the SAD-value of a window at a certain position a nearb window can be computed incrementall. ) 1) 1 ) Stereo Matching - Christian Unger 29

30 Stereo Matching - Christian Unger 30 Improvement: Bo Filtering. The mathematics in detail: ) 1) ) ) 1 ) ) ) 1 ) 1 ) 1 ) 1 ) ) ) 1 1 ) : W d U W d U d SAD d SAD j d i IR j i IL d SAD j d i IR j i IL d SAD j d i IR j i IL d SAD W W j j d U W W i W W j W W i W W j W W i = = = = = = = = = = =

31 Improvement: SIMD. Single Instruction Multiple Data. More or less a purel implementational optimization utilizing advanced assembler routines. Allows to process data in parallel for eample work on up to 16 piels at a time) Of special interest is the PSADBW-instruction MMX SSE2): Computes the sum of absolute differences of eight/siteen unsigned bte integers Stereo Matching - Christian Unger 31

32 Eecution Time Revisited. Using these optimizations disparities can be processed in TODO ms. Can we go further? Yes to about 50%). B not processing all disparities: instead of a brute-force search use an iterative search technique that automaticall stops at a certain point Unger et al. Efficient Disparit Computation without Maimum Disparit for Real-Time Stereo Vision. BMVC 2009.). Stereo Matching - Christian Unger 32

33 Eample. Eample: Video-Sequence from the vehicle. Stereo Matching - Christian Unger 33

34 Global Methods: Overview. The area-based approach presented falls into the categor of local methods since the disparit-computation is done for ever single piel alone. However there is another big class of methods the global and semi-global methods. In these approaches the task of computing disparities is cast as an energ minimization problem. Tpicall an energ functional is formulated such as: where E D measures the piel similarit and E S penalizes disparit variations. Well known methods to solve such problems are Dnamic Programming Scanline Optimization Graph Cuts Belief Propagation E D) = E D) λe D) D S Stereo Matching - Christian Unger 34

35 Global Methods: Eamples. Adapting Belief Propagation Graph Cuts However this qualit is at the cost of eecution time e.g. Graph Cuts: 5 minutes). Stereo Matching - Christian Unger 35

36 Disparit Refinements: Sub-Piel Interpolation. We discussed onl the computation of integer valued disparities but the world is continuous. Real valued disparities ma be obtained b approimating the cost function locall using a parabola: I ) I d ) L R Matching Cost d d ma Stereo Matching - Christian Unger 36

37 Disparit Refinements: Left-Right Consistenc Check. Outlier detection routine. Perform the stereo matching two times: 1. B computing a disparit for ever piel of the left image left to right) 2. And again b computing a disparit for ever piel of the right image right to left) Then if these two disparities differ an outlier has been found. Stereo Matching - Christian Unger 37