Data Term. Michael Bleyer LVA Stereo Vision

Transcription

1 Data Term Michael Bleyer LVA Stereo Vision

2 What happened last time? We have looked at our energy function: E ( D) = m( p, dp) + p I < p, q > N s( p, q) We have learned about an optimization algorithm that can minimize our energy Belief Propagation We have investigated the smoothness function s(): First order smoothness functions Linear Potts model Truncated linear Second order smoothness Edge/segment-sensitive smoothness functions 2

3 What is Going to Happen Today? We will look at the data term 3

4 Data Term Michael Bleyer LVA Stereo Vision

5 The Data Term Measures the goodness of correspondences. The color of corresponding pixels are compared. Formally defined as p I E data ( D) = m( p, q) where m() computes the intensity/color dissimilarity between two pixels p is a pixel of the left image q is p s correspondence in the right image according to disparity map D. We will now look at different methods for defining the pixel dissimilarity function m(). For the first match measures, I will assume that the photoconsistency assumption holds: I will relax this assumption later on. For now, we will just consider grey-scale images: I will speak about color matching later on. 5

6 Absolute Intensity Difference Pixel dissimilarity is computed as the absolute difference in intensity values: m( p, q) = Ip Iq where Ip denotes the intensity of pixel p. Quite common choice in stereo matching 6

7 Squared Intensity Difference Pixel dissimilarity is computed as the squared difference in intensity values: m( p, q) = ( Ip Iq ) 2 Probably not a good choice for stereo matching: Sensitive to outliers Typically performs slightly worse than absolute difference. 7

8 Truncated Absolute Difference Pixel dissimilarity is computed as the truncated absolute difference in intensity values: m( p, q) = min( Ip Iq, k) where k is a user-defined value. Advantage: Robustness against outliers. Better performance in occluded regions: Since an occluded pixel does not have a matching point, it will have high pixel dissimilarity. Truncation lowers the data costs for the correct disparity. There is still hope that the correct disparity can be propagated from surrounding non-occluded pixels. Disadvantage: You have an additional parameter. An optimal value for k is difficult to find or might not even exist. 8

9 Sampling Insensitive Measure [Birchfield,PAMI98] In the real world, intensity is a continuous function Intensity X-Coordinates (Left image)

10 Sampling Insensitive Measure [Birchfield,PAMI98] When we take a photo, we sample this continuous curve to derive discrete pixels. Intensity X-Coordinates (Left image)

11 Sampling Insensitive Measure [Birchfield,PAMI98] Problem: Samples are typically taken at different curve positions in the left and right images. Intensity Intensity X-Coordinates (Left image) X-Coordinates (Right image)

12 Sampling Insensitive Measure [Birchfield,PAMI98] Due to this different sampling, corresponding pixels have different intensity values (and this will oftentimes lead to wrong matches). Intensity p Intensity p X-Coordinates X-Coordinates p and p are corresponding pixels, (Left image) (Right image) but have different intensity values

13 Sampling Insensitive Measure [Birchfield,PAMI98] Idea of [Birchfield,PAMI98]: We also look p s horizontal neighbors p p Intensity q r Intensity X-Coordinates (Left image) X-Coordinates (Right image)

14 Sampling Insensitive Measure [Birchfield,PAMI98] We interpolate the intensity of the pixel p- that lies in between p p q and q by I + I I p = 2 p Intensity p- q r Intensity p X-Coordinates (Left image) X-Coordinates (Right image)

15 Sampling Insensitive Measure [Birchfield,PAMI98] We also interpolate the intensity of the pixel p+ that lies in between p r p and r by I + I I p + = 2 p Intensity p- p+ q r Intensity p X-Coordinates (Left image) X-Coordinates (Right image)

16 Sampling Insensitive Measure [Birchfield,PAMI98] We compute the sampling insensitive matching as p m( p, p') = min( p p', p p', p + p' ) Intensity p- p+ q Intensity p X-Coordinates (Left image) X-Coordinates (Right image)

17 Sampling Insensitive Measure [Birchfield,PAMI98] We compute the sampling insensitive matching as m( p, p') = min( p p', p p', p + p' ) p Intensity p- p+ q Intensity p Advantage: We have also included the correctly sampled pixel p X-Coordinates X-Coordinates => low intensity dissimilarity and high chances for (Left image) correct match (Right image)

18 Sampling Insensitive Measure [Birchfield,PAMI98] We should do this in a symmetric way: m( p, p') = min( p p', p p', p + p', p p', p p' + ) p Intensity p- p+ q Intensity p - p p + X-Coordinates X-Coordinates (Left image) (Right image)

19 Violations of Photo-Consistency Assumption In real-world stereo images, the photo-consistency assumption is almost never perfectly fulfilled. (Left image) We call a pixel radiometric distorted if its intensity is different in left and right images. There are various reasons for radiometric distortions, e.g.: Different illumination conditions in the images Different exposure times Different sensor characteristics (Right image) 19

20 Radiometric Insensitive Match Measures Treatment of radiometric distortions has great impact on the quality of results. Unfortunately, none of the above match measures is able to cope with radiometric distortions. We will now learn about 3 radiometric insensitive measures: Mutual Information Zero mean Normalized Cross-Correlation (ZNCC) Census 20

21 Mutual Information [Hirschmueller,PAMI08] Advantage: Mutual Information is a pixel-based measure. In contrast to window-based measures, artifacts at disparity borders are avoided To compute Mutual Information matching scores, we need the disparity map: Chicken and Egg problem If we knew the disparity map, then we were already done. This dilemma is typically solved in an iterative fashion: 1. We compute an initial disparity map (e.g., using absolute differences as a dissimilarity function) 2. We compute Mutual Information scores using our current disparity map. 3. We compute a new disparity map using the Mutual Information scores. 4. Goto 2. 21

22 Mutual Information [Hirschmueller,PAMI08] Advantage: Mutual How Information can is we a pixel-based compute measure. the Mutual Information matching scores? In contrast to window-based measures, artifacts at disparity borders are avoided To compute Mutual Information matching scores, we need the disparity map: Disclaimer: Chicken and Egg problem There is quite a lot of theory If we knew the disparity map, then we were already done. This dilemma behind is typically that. solved I in will an iterative focus fashion: on the 1. We compute an initial disparity map (e.g., using absolute differences practical implementation as as a dissimilarity function) 2. We compute Mutual Information described scores in using our current disparity map. [Hirschmueller,PAMI08]. 3. We compute a new disparity map using the Mutual Information scores. 4. Goto 2. 22

23 Computing Mutual Information Scores For each pixel p, we look-up its matching point q in the right image using our current disparity map. We look up the intensity values Ip and Iq. We make an entry at <Ip,Iq> in the diagram bellow. Intensity Right Image 100 Ip=150 Iq= Intensity Left Image For each possible pair of intensity values <Ip,Iq>, our diagram stores how often this pair occurred in the disparity map. 23

24 Computing Mutual Information Scores Let us assume that all corresponding pixels have identical intensity values, i.e. there is no radiometric distortion. The diagram looks like this: Intensity Right Image 45 Intensity Left Image 24

25 Computing Mutual Information Scores Let us assume that the right image is darker than the left one. The diagram looks like this: Intensity Right Image <45 Intensity Left Image 25

26 Computing Mutual Information Scores Let us assume that the left image is darker than the right one. The diagram looks like this: Intensity Right Image >45 Intensity Left Image 26

27 Computing Mutual Information Scores This is what the diagram looks like for the Teddy test set: Black means that the intensity pair occurred very frequently in the disparity map Images taken from [Hirschmueller, PAMI08] 27

28 Computing Mutual Information Scores This is what the diagram looks like for the Teddy test set: Those intensity pairs that occurred frequently should be given low matching costs. Black means that the intensity pair occurred very frequently in the disparity map 28

29 Computing Mutual Information Scores We compute log(p) where P is our diagram. P -log(p) These are our Mutual Information scores Where white pixels mean low matching costs. Side node: For simplicity, I left out two steps where you apply Gaussian smoothing on P and log(p). 29

30 Disadvantage of Mutual Information Global model: Mutual Information can only model radiometric changes that are valid for the whole image. For example, the whole left image is darker than the right one. It cannot model radiometric changes that occur only locally. For example, the bottom left part of the image is darker in the left view. Unfortunately, radiometric distortions are oftentimes local. 30

31 Zero mean Normalized Cross-Correlation (ZNCC) Is defined on windows => Will lead to artefacts at disparity discontinuities Pixel dissimilarity m() computed as m( p, p d ) = q Wp q Wp ( I q ( I I p q ) 2 I p )( I q q Wp d ( I q I d p d I ) p d ) 2 where Wp is the set of all pixels in the window centered at p. Ip is the mean intensity computed over all pixels insider Wp. Subtraction of the mean value serves to normalize intensity values (robustness against radiometric changes) 31

32 Census We center a window on pixel p in the left image. 32

33 Census We center a window on pixel p in the left image. Apply the following transformation: If a pixel has smaller intensity than the window center pixel, write 0. Else write 1. 33

34 Census We center a window on pixel p in the left image. Apply the following transformation: If a pixel has smaller intensity than the window center pixel, write 0. Else write 1. Write the binary values as a bitstring. 34

35 Census We center a window on pixel p in the left image. Apply the following transformation: If a pixel has smaller intensity than the window center pixel, write 0. Else write 1. Write the binary values as a bitstring. Apply the same operations for the window centered on pixel q in the right image. 35

36 Census We center a window on pixel p in the left image. Apply the following transformation: If a pixel has smaller intensity than the window center pixel, write 0. Else write 1. Write the binary values as a bitstring. Apply the same operations for the window centered on pixel q in the right image. Census matching costs are computed as the Hamming distance between the bit strings: Number of positions at which binary values are different 36

37 Census - Discussion We do not directly match the intensities, but the local texture represented as a bit-string. If one image is darker than the other, the bit strings should still agree: You can, for example, add a value of 10 to the intensity values in p s window on the previous slide and will still get the same bit string. Increased robustness if window overlaps a disparity discontinuity. Problems in untextured regions If all pixel have very similar intensities, the values of the bit string largely depend on image noise. Leads to noisy results in untextured regions. 37

38 How Can We Incorporate Color Information? Typically this is done in the most simple way: Compute the match measure individually for each color channel Sum-up the values over all color channels Let us now investigate the role of color 38

39 Why Should Color Help? We have a blue pixel in the left image. We have 2 candidate matches in the right image: A blue pixel A yellow pixel It is quite clear that the blue pixel is the correct match. (Left Image) (Right Image) 39

40 Why Should Color Help? Let us now convert the color images into grey-scale images. In our example blue and yellow colors map to the same grey-value. It is no longer clear which of our 2 candidate pixels is the correct match => Color information reduces ambiguity! (Left Image) (Right Image) 40

41 Why Should Color Help? Let us now convert the color images into grey-scale images. In our example blue and yellow colors map to the same grey-value. It is no longer clear which of our 2 candidate pixels is the correct matchhowever, a lot of stereo algorithms => Color information reduces ambiguity! do not use color information. Is this for a reason? (Left Image) (Right Image) 41

42 Evaluation of Color Matching [Bleyer,3DPVT10] We evaluate the performance of 8 different color systems and grey scale matching: The different color systems affect the data term of our energy. Truncated linear is used as a smoothness term. Energy optimization is accomplished using the Simple Tree dynamic programming method (see session on dynamic programming) We use 30 ground truth pairs from Middlebury as test data. Error is computed as the percentage of pixels having an absolute disparity error > 1 pixel in non-occluded regions. 42

43 Test Set 43

44 Absolute Intensity Difference We compute the pixel dissimilarity as the absolute difference in intensity values. Results (see plot bellow): Grey-scale matching nearly always performs worst. The color system LUV seems to perform better than RGB. 44

45 Absolute Intensity Difference We compute the pixel dissimilarity as the absolute difference in intensity values. Results (see plot bellow): So using color is a good thing? Grey-scale matching nearly always performs worst. The color system LUV seems to perform better than RGB. Well, this is not the whole story. 45

46 Example Results for Dolls Test Set (Left Image) (Grey - Disparity) (LUV - Disparity) Let us have a closer look at the disparity maps. I show two disparity maps where One has been computed by using the absolute difference of grey values as a match function. One has been computed by using the summed-up absolute differences in LUV values.

47 Example Results for Dolls Test Set (Left Image) (Grey - Errors) (LUV - Errors) We should rather look at the error maps Black pixels are pixels whose disparity error is larger than 1 pixel in comparison against the ground truth. Errors are clearly smaller when using LUV.

48 Example Results for Dolls Test Set (Left Image) (Grey - Errors) (LUV - Errors) We should rather look at the error maps Black pixels are pixels whose disparity error is larger than 1 pixel in comparison against the ground truth. Errors are clearly smaller when using LUV.

49 Radiometric Problems in the Dolls Set We have the ground truth disparity map for the Dolls Set. => For each pixel p of the left image, we know its correct correspondence q in the right image. If there are no radiometric distortions, Ip-Iq should be equal to 0. In practice, we obtain the image shown on the right where: Bright pixels have a large value for Ip-Iq. These bright pixels are the result of radiometric distortions. (Data Costs of Ground Truth Solution)

50 Radiometric Problems in the Dolls Set We can apply thresholding on the ground truth data cost image. (Data Costs of Ground Truth Solution) (Smoothed Thresholding)

51 Radiometric Problems in the Dolls Set We can apply thresholding on the ground truth data cost image. There seems to be large overlap between errors in the grey-scale matching result and radiometric distorted regions. (Data Costs of Ground Truth Solution) (Smoothed Thresholding) (Disparity Error when using Grey-Scale Matching)

52 Radiometric Problems in the Dolls Set We can apply thresholding on the ground truth data cost image. There seems to be large overlap between errors in the grey-scale matching result and radiometric distorted regions. Errors in radiometric distorted regions seem to be effectively reduced when using color matching. (Data Costs of Ground Truth Solution) (Smoothed Thresholding) (Disparity Error when using LUV Matching)

53 Radiometric Problems in the Dolls Set We can apply thresholding on the ground truth data cost image. There seems to be large overlap between errors in the grey-scale matching result and radiometric distorted regions. Errors in radiometric Colordistorted might regions be of seem specific to be effectively reduced when using color matching. importance in radiometric distorted image regions (Data Costs of Ground Truth Solution) (Smoothed Thresholding) (Disparity Error when using LUV Matching)

54 Color Helps in Radiometric Distorted Regions We have extracted the radiometric distorted regions for all 30 test images. We now analyze the disparity error separately In regions affected by radiometric distortions In regions unaffected by radiometric distortions Average error percentage in regions affected by radiometric distortions A large improvement can be observed when using color (e.g., LUV) instead of grey scale matching (Grey) 54

55 Color Helps in Radiometric Distorted Regions We have extracted the radiometric distorted regions for all 30 test images. We now analyze the disparity error separately In regions affected by radiometric distortions In regions unaffected by radiometric distortions Regions unaffected by radiometric distortions Almost no improvement 55

56 Color Helps in Radiometric Distorted Regions We have extracted the radiometric distorted regions for all 30 test images. We now analyze the disparity error separately In regions affected by radiometric distortions In regions unaffected by radiometric distortions Average error percentage in all regions. The overall improvement is largely due to considerable improvement in radiometric distorted regions. 56

57 Why Not Directly Using a Radiometric Insensitive Measure? If color only helps in radiometric distorted regions, we can directly use radiometric insensitive measures instead of color. 3 radiometric insensitive measures are tested: Mutual information (MI) Zero mean Normalized Cross-Correlation (NCC) Census (CENSUS) All 3 match measures are applied on grey-scale images 57

58 Why Not Directly Using a Radiometric Insensitive Measure? NCC and CENUS seem to perform best This result is consistent with [Hirschmueller,PAMI09] NCC and CENSUS have the same effect as using color (AbsDif (LUV)): They improve performance in radiometric distorted regions (blue line) NCC and CENSUS are considerably better than color in this respect. 58

59 Using Color with Radiometric Insensitive Measures NCC used with grey scale and 8 color spaces CENSUS used with grey scale and 8 color spaces It seems to be a bad idea to use color in conjunction with radiometric insensitive measures: Grey-scale matching performs better than all 8 color spaces. How can this happen? Increased robustness of color in radiometric regions is not important anymore (NCC and CENSUS do a better job) You practically do not lose texture when deleting color Intensity is probably more robustly capture by nowadays cameras (less 59 noise in the intensity channel)

60 Using Color with Radiometric Insensitive Measures NCC used with grey scale and 8 color spaces My advice: You should not use color. You should definitely use a radiometric insensitive match measure. It seems to be a bad idea to use color in conjunction with radiometric insensitive measures: Grey-scale matching performs better than all 8 color spaces. CENSUS used with grey scale and 8 color spaces How can this happen? Increased robustness of color in radiometric regions is not important anymore (NCC and CENSUS do a better job) You practically do not lose texture when deleting color Intensity is probably more robustly capture by nowadays cameras (less 60 noise in the intensity channel)

61 Support Aggregation in Global Stereo Matching In the session on local stereo methods, we have learned about support aggregation: We do not match single pixels, but windows Until recently, using windows in global stereo was considered a bad idea: You get artefacts at disparity discontinuities!

62 Support Aggregation in Global Stereo Matching We have also learned about new segmentation-based aggregation schemes: They deliver excellent performance In general, they will even improve the performance near depth discontinuities Apart from increased computational costs, there is relatively little that speaks against using these aggregation methods for implementing your pixel dissimilarity function m(). Standard Support Weights Geodesic Support Weights

63 Summary Data Term: Standard dissimilarity measures: Absolute / Squared intensity differences Sampling insensitive measures Radiometric insensitive measures: Mutual information ZNCC Census The role of color Segmentation-based aggregation schemes

64 References [Birchfiled,PAMI98] S. Birchfield, C. Tomasi, A Pixel Dissimilarity Measure That Is Insensitive to Image Sampling, PAMI, vol. 20, [Bleyer,3DPVT10] M. Bleyer, S. Chambon, Does Color Really Help in Dense Stereo Matching?, 3DPVT, [Hirschmueller,PAMI08] H. Hirschmueller, Stereo Processing by Semi- Global Matching and Mutual Information, PAMI, vol. 30, no. 2, [Hirschmueller,PAMI09] H. Hirschmueller, D. Scharstein, Evaluation of Stereo Matching Costs on Images with Radiometric Differences, PAMI, vol. 31, no. 9, 2009.