CS 558: Computer Vision 3 rd Set of Notes

Transcription

1 1 CS 558: Computer Vision 3 rd Set of Notes Instructor: Philippos Mordohai Webpage: Philippos.Mordohai@stevens.edu Office: Lieb 215

2 Overview Denoising Based on slides b S. Lazebnik Edge detection Based on slides b S. Lazebnik and D. Hoiem Feature etraction: Corners Based on slides b S. Lazebnik Sampling images Based on slides b D. Hoiem 2

3 Image denoising How can we reduce noise in a photograph?

4 Moving average Let s replace each piel with a weighted average of its neighborhood The weights are called the filter kernel What are the weights for the average of a 33 neighborhood? bo filter Source: D. Lowe

5 Noise Original Impulse noise Salt and pepper noise Gaussian noise Salt and pepper noise: contains random occurrences of black and white piels Impulse noise: contains random occurrences of white piels Gaussian noise: variations in intensit drawn from a Gaussian normal distribution Source: S. Seitz

6 Gaussian noise Mathematical model: sum of man independent factors Good for small standard deviations Assumption: independent, zero-mean noise Source: M. Hebert

7 Reducing Gaussian noise Smoothing with larger standard deviations suppresses noise, but also blurs the image

8 Reducing salt-and-pepper noise What s wrong with the results?

9 Alternative idea: Median filtering A median filter operates over a window b selecting the median intensit in the window Is median filtering linear? Source: K. Grauman

10 Median filter What advantage does median filtering have over Gaussian filtering? Robustness to outliers Source: K. Grauman

11 Median filter Salt-and-pepper noise Median filtered MATLAB: medfilt2(image, [h w]) Source: M. Hebert

12 Gaussian vs. median filtering Gaussian Median

13 Sharpening revisited Source: D. Lowe

14 Sharpening filter Original Sharpening filter - Accentuates differences with local average Source: D. Lowe

15 Sharpening revisited What does blurring take awa? = original smoothed (55) detail Let s add it back: + α = original detail sharpened

16 Edge detection Winter in Kraków photographed b Marcin Rczek

17 Edge detection Goal: Identif sudden changes (discontinuities) in an image Intuitivel, most semantic and shape information from the image can be encoded in the edges More compact than piels Ideal: artist s line drawing (but artist is also using object-level knowledge) Source: D. Lowe

18 Origin of edges Edges are caused b a variet of factors: surface normal discontinuit depth discontinuit surface color discontinuit illumination discontinuit Source: Steve Seitz

19 Wh finding edges is important Group piels into objects or parts Cues for 3D shape Guiding interactive image editing

20 Closeup of edges

21 Closeup of edges

22 Closeup of edges

23 Closeup of edges

24 Edge detection An edge is a place of rapid change in the image intensit function image intensit function (along horizontal scanline) first derivative edges correspond to etrema of derivative

25 Derivatives with convolution For 2D function f(,), the partial derivative is: For discrete data, we can approimate using finite differences: To implement the above as convolution, what would be the associated filter? ), ( ), ( lim ), ( 0 f f f 1 ), ( ) 1, ( ), ( f f f Source: K. Grauman

26 Partial derivatives of an image f (, ) f (, ) or 1-1 Which shows changes with respect to?

27 Finite difference filters Other approimations of derivative filters eist: Source: K. Grauman

28 Image gradient The gradient of an image: The gradient points in the direction of most rapid increase in intensit How does this direction relate to the direction of the edge? The gradient direction is given b The edge strength is given b the gradient magnitude Source: Steve Seitz

29 Intensit Intensit profile Gradient

30 With a little Gaussian noise Gradient

31 Effects of noise Consider a single row or column of the image Where is the edge? Source: S. Seitz

32 Effects of noise Difference filters respond strongl to noise Image noise results in piels that look ver different from their neighbors Generall, the larger the noise the stronger the response What can we do about it? Source: D. Forsth

33 Solution: smooth first f g f * g d d ( f g) To find edges, look for peaks in ( f g) d d Source: S. Seitz

34 Derivative theorem of convolution Differentiation is convolution, and convolution is associative: d d ( f g) f g d d This saves us one operation f d d g f d d g Source: S. Seitz

35 Derivative of Gaussian filters Which one finds horizontal/vertical edges?

36 Derivative of Gaussian filters Are these filters separable?

37 Recall: Separabilit of the Gaussian filter Source: D. Lowe

38 Scale of Gaussian derivative filter 1 piel 3 piels 7 piels Smoothed derivative removes noise, but blurs edge. Also finds edges at different scales Source: D. Forsth

39 Review: Smoothing vs. derivative filters Smoothing filters Gaussian: remove high-frequenc components; low-pass filter Can the values of a smoothing filter be negative? What should the values sum to? One: constant regions are not affected b the filter Derivative filters Derivatives of Gaussian Can the values of a derivative filter be negative? What should the values sum to? Zero: no response in constant regions High absolute value at points of high contrast

40 The Cann edge detector 1. Filter image with derivative of Gaussian 2. Find magnitude and orientation of gradient 3. Non-maimum suppression: Thin wide ridges down to single piel width 4. Linking and thresholding (hsteresis): Define two thresholds: low and high Use the high threshold to start edge curves and the low threshold to continue them

41 The Cann edge detector original image Slide credit: Steve Seitz

42 The Cann edge detector norm of the gradient

43 The Cann edge detector thresholding

44 The Cann edge detector How to turn these thick regions of the gradient into curves? thresholding

45 Non-maimum suppression Check if piel is local maimum along gradient direction, select single ma across width of the edge requires checking interpolated piels p and r Source: D. Forsth

46 Bilinear Interpolation

47 The Cann edge detector Thinning (non-maimum suppression) Problem: piels along this edge didn t survive the thresholding

48 Hsteresis thresholding Use a high threshold to start edge curves, and a low threshold to continue them Source: Steve Seitz

49 Hsteresis thresholding Threshold at low/high levels to get weak/strong edge piels Trace connected components, starting from strong edge piels

50 Recap: Cann edge detector 1. Compute and gradient images 2. Find magnitude and orientation of gradient 3. Non-maimum suppression: Thin wide ridges down to single piel width 4. Linking and thresholding (hsteresis): Define two thresholds: low and high Use the high threshold to start edge curves and the low threshold to continue them MATLAB: edge(image, cann ); J. Cann, A Computational Approach To Edge Detection, IEEE Trans. Pattern Analsis and Machine Intelligence, 8: , 1986.

51 Effect of (Gaussian kernel spread/size) original Cann with Cann with The choice of depends on desired behavior large detects large scale edges small detects fine features Source: S. Seitz

52 Learning to detect boundaries image human segmentation gradient magnitude Berkele segmentation database:

53 pb boundar detector Martin, Fowlkes, Malik 2004: Learning to Detect Natural Image Boundaries on/grouping/papers/mfm-pami-boundar.pdf Figure from Fowlkes

54 pb Boundar Detector Figure from Fowlkes

55 Brightness Color Teture Combined Human

56 Results Pb (0.88) Human (0.95)

57 Results Pb Global Pb (0.88) Pb Human Human (0.96)

58 Human (0.95) Pb (0.63)

59 Pb (0.35) Human (0.90) For more: ision/bsds/bench/html/ color.html

60 Feature etraction: Corners

61 Wh etract features? Motivation: panorama stitching We have two images how do we combine them?

62 Wh etract features? Motivation: panorama stitching We have two images how do we combine them? Step 1: etract features Step 2: match features

63 Wh etract features? Motivation: panorama stitching We have two images how do we combine them? Step 1: etract features Step 2: match features Step 3: align images

64 Characteristics of good features Repeatabilit The same feature can be found in several images despite geometric and photometric transformations Salienc Each feature is distinctive Compactness and efficienc Man fewer features than image piels Localit A feature occupies a relativel small area of the image; robust to clutter and occlusion

65 Applications Feature points are used for: Image alignment 3D reconstruction Motion tracking Robot navigation Indeing and database retrieval Object recognition

66 A hard feature matching problem NASA Mars Rover images

67 Answer below (look for tin colored squares ) NASA Mars Rover images with SIFT feature matches Figure b Noah Snavel

68 Corner Detection: Basic Idea We should easil recognize the point b looking through a small window Shifting a window in an direction should give a large change in intensit flat region: no change in all directions edge : no change along the edge direction corner : significant change in all directions

69 Corner Detection: Mathematics Change in appearance of window W for the shift [u,v]: I(, ) E(u, v) E(3,2) W I v u I v u E ), ( 2 )], ( ), ( [ ), (

70 Corner Detection: Mathematics I(, ) E(u, v) E(0,0) W I v u I v u E ), ( 2 )], ( ), ( [ ), ( Change in appearance of window W for the shift [u,v]:

71 Corner Detection: Mathematics We want to find out how this function behaves for small shifts E(u, v) W I v u I v u E ), ( 2 )], ( ), ( [ ), ( Change in appearance of window W for the shift [u,v]:

72 Corner Detection: Mathematics First-order Talor approimation for small motions [u, v]: Let s plug this into E(u,v): v I u I I v u I ), ( ), ( W W W W v I uv I I u I v I u I I v I u I I I v u I v u E ), ( ), ( 2 ), ( 2 ), ( 2 2 ] [ )], ( ), ( [ )], ( ), ( [ ), (

73 Corner Detection: Mathematics The quadratic approimation can be written as where M is a second moment matri computed from image derivatives: v u M v u v u E ), ( I I I I I I M, 2,,, 2 (the sums are over all the piels in the window W)

74 The surface E(u,v) is locall approimated b a quadratic form. Let s tr to understand its shape. Specificall, in which directions does it have the smallest/greatest change? Interpreting the second moment matri v u M v u v u E ] [ ), ( E(u, v) I I I I I I M, 2,,, 2

75 First, consider the ais-aligned case (gradients are either horizontal or vertical) If either a or b is close to 0, then this is not a corner, so look for locations where both are large. b a 0 0 I I I I I I M, 2,,, 2 Interpreting the second moment matri

76 Interpreting the second moment matri Consider a horizontal slice of E(u, v): [ u v] M u const v This is the equation of an ellipse.

77 Interpreting the second moment matri Consider a horizontal slice of E(u, v): This is the equation of an ellipse. [ u v] M u v Diagonalization of M: M R R 0 2 The ais lengths of the ellipse are determined b the eigenvalues and the orientation is determined b R direction of the fastest change direction of the slowest change const ( ma ) -1/2 ( min ) -1/2

78 Quick Eigenvalue/Eigenvector Review The eigenvectors of a matri A are the vectors that satisf: The scalar is the eigenvalue corresponding to The eigenvalues are found b solving: In our case, A = H is a 22 matri, so we have The solution: Once ou know, ou find b solving

79 Visualization of second moment matrices

80 Visualization of second moment matrices

81 Interpreting the eigenvalues Classification of image points using eigenvalues of M: 2 Edge 2 >> 1 Corner 1 and 2 are large, 1 2 ; E increases in all directions 1 and 2 are small; E is almost constant in all directions Flat region Edge 1 >> 2 1

82 Corner response function R det( M ) trace( M ) ( ) α: constant (0.04 to 0.06) Edge R < 0 Corner R > 0 Flat region R small Edge R < 0

83 The Harris corner detector 1. Compute partial derivatives at each piel 2. Compute second moment matri M in a Gaussian window around each piel: C.Harris and M.Stephens. A Combined Corner and Edge Detector. Proceedings of the 4th Alve Vision Conference: pages , I w I I w I I w I w M, 2,,, 2 ), ( ), ( ), ( ), (

84 The Harris corner detector 1. Compute partial derivatives at each piel 2. Compute second moment matri M in a Gaussian window around each piel 3. Compute corner response function R

85 Harris Detector: Steps

86 Harris Detector: Steps Compute corner response R

87 The Harris corner detector 1. Compute partial derivatives at each piel 2. Compute second moment matri M in a Gaussian window around each piel 3. Compute corner response function R 4. Threshold R 5. Find local maima of response function (non-maimum suppression)

88 Harris Detector: Steps Find points with large corner response: R > threshold

89 Harris Detector: Steps Take onl the points of local maima of R

90 Harris Detector: Steps

91 Invariance and covariance We want corner locations to be invarian to photometric transformations and covarian to geometric transformations Invariance: image is transformed and corner locations do not change Covariance: if we have two transformed versions of the same image, features should be detected in corresponding locations

92 Intensit changes I a I + b Onl derivatives are used => invariance to intensit shift I I + b Intensit scaling: I a I R threshold R (image coordinate) (image coordinate) Partiall invariant to affine intensit change

93 Image translation Derivatives and window function are shift-invariant Corner location is covariant w.r.t. translation

94 Image rotation Second moment ellipse rotates but its shape (i.e. eigenvalues) remains the same Corner location is covariant w.r.t. rotation

95 Scaling Corner All points will be classified as edges Corner location is not covariant to scaling!

96 Sampling Wh does a lower resolution image still make sense to us? What do we lose? Image:

97 Subsampling b a factor of 2 Throw awa ever other row and column to create a 1/2 size image

98 Aliasing problem 1D eample (sinewave): Source: S. Marschner

99 Aliasing problem 1D eample (sinewave): Source: S. Marschner

100 Aliasing problem Sub-sampling ma be dangerous. Characteristic errors ma appear: Wagon wheels rolling the wrong wa in movies Checkerboards disintegrate in ra tracing Striped shirts look funn on color television Source: D. Forsth

101 Aliasing in video Slide b Steve Seitz

102 Aliasing in graphics Source: A. Efros

103 Sampling and aliasing

104 Nquist-Shannon Sampling Theorem When sampling a signal at discrete intervals, the sampling frequenc must be 2 f ma f ma = ma frequenc of the input signal This will allows to reconstruct the original perfectl from the sampled version v v v good bad

105 Anti-aliasing Solutions: Sample more often Get rid of all frequencies that are greater than half the new sampling frequenc Will lose information But it s better than aliasing Appl a smoothing filter

106 Algorithm for downsampling b factor of 2 1. Start with image(h, w) 2. Appl low pass filter im_blur = imfilter(image, fspecial( gaussian, 7, 1)) 3. Sample ever other piel im_small = im_blur(1:2:end, 1:2:end);

107 Anti-aliasing Forsth and Ponce 2002

108 Subsampling without pre-filtering 1/2 1/4 (2 zoom) 1/8 (4 zoom) Slide b Steve Seitz

109 Subsampling with Gaussian pre-filtering Gaussian 1/2 G 1/4 G 1/8 Slide b Steve Seitz

110 Slide Credits This set of sides contains contributions kindl made available b the following authors Svetlana Lazebnik Aleei Efros Li Fei-Fei Kristen Grauman Martial Hebert Derek Hoiem David Lowe Steve Marschner David Forsth Steve Seitz Richard Szeliski