CPSC 425: Computer Vision

Transcription

1 1 / 92 CPSC 425: Computer Vision Instructor: Jim Little little@cs.ubc.ca Department of Computer Science University of British Columbia Lecture Notes 2016/2017 Term 2

2 2 / 92 Menu February 14, 2017 Topics: Review for midterm Reading: (after midterm) Paper: Distinctive Image Features from Scale-Invariant Keypoints Forsyth & Ponce (2nd ed.) 5.4 Reminders: Midterm exam, in class, Thursday, February 16 See Web page regarding office hours Reading Week Feb www: piazza:

3 3 / 92 Today s Fun Example Demo of chromatic adaptation

4 4 / 92 Lecture 10: Re-cap Approaches to texture exploit pyramid (i.e. scaled) and oriented representations Human colour perception colour matching experiments additive and subtractive matching principle of trichromacy RGB and CIE XYZ are linear colour spaces Uniform colour space: differences in coordinates are a good guide to differences in perceived colour HSV colour space: more intuitive description of colour for human interpretation (Human) colour constancy: perception of intrinsic surface colour under different colours of lighting

5 5 / 92 Paths to Understanding Five distinct paths to a deeper understanding of CPSC 425 course material: 1 mathematics (i.e., theory) 2 visualize computation(s) 3 experiment on simple (test) cases on real images 4 read code 5 write code

6 6 / 92 Midterm Review: Readings Lecture 1 10 slides Assigned readings from Forsyth & Ponce (2nd ed.) Paper Texture Synthesis by Non-parametric Sampling Assignments 2 3 iclicker questions Lecture exercises Practice problems (with solutions)

7 7 / 92 Midterm Details 60 minutes Closed book, no calculators Format similar to posted practice problems Part A: Multiple-part true/false Part B: Short answer No coding questions

8 8 / 92 Linear Filters (cont d) I (X, Y ) = k k F(i, j) I (X + i, Y + j) j= k i= k For each X and Y, superimpose the filter, F(X, Y ), on the image centered at (X, Y ) Compute the new pixel value, I (X, Y ), as the sum of m m values, where each value is the product of the original pixel value in I(X, Y ) and the corresponding value in the filter

9 9 / 92

10 10 / 92 Linear Filters: Boundary Effects 3 standard ways to deal with boundaries 1 Ignore these locations Make the computation undefined for the top and bottom k rows and the leftmost and rightmost k columns 2 Pad the image with zeros Return zero whenever a value of I is required at some position outside the defined limits of X and Y 3 Assume periodicity The top row wraps around to the bottom row; the leftmost column wraps around to the rightmost column

11 11 / 92 Linear Filters The correlation of F(X, Y ) and I(X, Y ) is I (X, Y ) = k j= k k i= k F(i, j) I (X + i, Y + j) Visual interpretation: Superimpose the filter F on the image I at (X,Y), perform an element-wise multiply, and sum up the values Convolution is like correlation except filter flipped When F( i, j) = F(i, j) the two are equivalent

12 12 / 92 Linear Systems: Characterization Theorem Any linear, shift invariant operation can be expressed as a convolution

13 13 / 92

14 14 / 92 Example 1 (cont d): Smoothing with a Gaussian Idea: Weigh contributions of neighbouring pixels by nearness 2D Gaussian (continuous case): x 2 + y 2 G σ (x, y) = 1 2πσ 2 exp 2σ 2 Forsyth & Ponce (2nd ed.) Figure 4.2

15 15 / 92 Gaussian: Area Under the Curve σ σ σ σ σ σ σ σ 68% 95% 99.7% 99.99%

16 16 / 92 Efficient Implementation: Separability A 2D function of x and y is separable if it can be written as the product of two functions, one a function only of x and the other a function only of y Both the 2D box filter and the 2D Gaussian filter are separable Both can be implemented as two 1D convolutions: First, convolve each row with a 1D filter Then, convolve each column with a 1D filter Aside: or vice versa The 2D Gaussian is the only (non trivial) 2D function that is both separable and rotationally invariant.

17 17 / 92 Linear Filters: Additional Properties Let denote convolution. Let I(X, Y ) be a digital image. Let F and G be digital filters Convolution is associative. That is, G (F I(X, Y )) = (G F) I(X, Y ) Convolution is symmetric. That is, (F G) I(X, Y ) = (G F) I(X, Y ) Convolving I(X, Y ) with filter F and then convolving the result with filter G can be achieved in a single step, namely convolving I(X, Y ) with filter G F = G F

18 18 / 92 Bilateral Filter An edge-preserving non-linear filter Like a Gaussian filter, the filter weights depend on spatial distance from the center pixel Idea: Pixels nearby (in space) should have greater influence than pixels far away Unlike a Gaussian filter, the filter weights also depend on range distance from the center pixel Idea: Pixels with similar brightness value should have greater influence than pixels with dissimilar brightness value

19 19 / 92 Bilateral Filter Application: Denoising Figure credit: Alexander Wong

20 Sample Question: Filters What does the following 3 3 linear, shift invariant filter compute when applied to an image? / 92

21 21 / 92 Continuous Case y i(x,y) Denote the image as a function, i(x, y), where x and y are spatial variables x

22 22 / 92 Discrete Case Idea: Superimpose (regular) grid on continuous image y i(x,y) x Sample the underlying continuous image according to the tessellation imposed by the grid

23 23 / 92 Discrete Case (cont d) y i(x,y) pixel x Each grid cell is called a picture element (or pixel) Denote the discrete image as I(X, Y ) We can store the pixels in a matrix or array

24 24 / 92 Sampling It s clear that some information may be lost when we work on a discrete pixel grid. Figure credit: Forsyth & Ponce (2nd ed.) Figure 4.7

25 25 / 92 Sampling Theory Exact reconstruction requires constraint on the rate at which i(x, y) can change between samples rate of change means derivative the formal concept is bandlimited signal bandlimit and constraint on derivative are linked An image is bandlimited if it has some maximum spatial frequency A fundamental result (Sampling Theorem) is: For bandlimited signals, if you sample regularly at or above twice the maximum frequency (called the Nyquist rate), then you can reconstruct the original signal exactly

26 26 / 92 Sampling Theory (cont d) Sometimes undersampling is unavoidable, and there is a trade-off between things missing and artifacts. Medical imaging: usually try to maximize information content, tolerate some artifacts Computer graphics: usually try to minimize artifacts, tolerate some information missing

27 27 / 92 Template Matching Figure credit: Kristen Grauman

28 28 / 92 Template Matching We can think of convolution/correlation as comparing a template (the filter) with each local image patch. Consider the filter and image patch as vectors. Applying a filter at an image location can be interpreted as computing the dot product (recall: element-wise multiply and sum) between the filter and the local image patch. But... The dot product may be large simply because the image region is bright. We need to normalize the result in some way.

29 29 / 92 Template Matching Let a and b be vectors. Let θ be the angle between them We know cos θ = a b a b = a b (a a) (b b) where is dot product and is vector magnitude Correlation is a dot product Correlation measures similarity between the filter and each local image region Normalized correlation varies between 1 and 1 Normalized correlation attains the value 1 when the filter and image region are identical (up to a scale factor)

30 30 / 92 Template Matching Figure credit: Kristen Grauman

31 31 / 92 Example 1 (cont d): Normalized Correlation Template (left), image (middle), normalized correlation (right) Note peak value at the true position of the hand Credit: W. Freeman et al., Computer Vision for Interactive Computer Graphics, IEEE Computer Graphics and Applications, 1998

32 32 / 92 Sample Question: Template Matching True or false: Normalized correlation is robust to a constant scaling in the image brightness.

33 33 / 92 Scaled Representations Goals: to find template matches at all scales template size constant, image scale varies e.g., finding hands or faces when we don t know what size they will be in the image efficient search for image to image correspondences look first at coarse scales, refine at finer scales much less cost (but may miss best match) to examine all levels of detail find edges with different amounts of blur find textures with different spatial frequencies (i.e., different levels of detail)

34 34 / 92 Shrinking an Image no smoothing Gaussian σ = 1 Forsyth & Ponce (2nd ed.) Figures (top rows) Gaussian σ = 2

35 35 / 92 Image Pyramid An image pyramid is a collection of representations of an image Typically, each layer of the pyramid is half the width and half the height of the previous layer In a Gaussian pyramid, each layer is smoothed by a Gaussian filter and resampled to get the next layer

36 36 / 92 Example 1: A Gaussian Pyramid Forsyth & Ponce (2nd ed.) Figure 4.17

37 37 / 92 From Template Matching to Local Feature Detection Slide credit: Li Fei-Fei, Rob Fergus, and Antonio Torralba

38 38 / 92 Estimating Derivatives Recall, for a 2D (continuous) function, f (x, y) f x = lim ε 0 f (x + ε, y) f (x, y) ε Differentiation is linear and shift invariant, and therefore can be implemented as a convolution A (discrete) approximation is f x F (X + 1, Y ) F(X, Y ) x

39 39 / 92 Estimating Derivatives A similar definition (and approximation) holds for f y. Image noise tends to result in pixels not looking exactly like their neighbours, so simple finite differences are sensitive to noise. The usual way to deal with this problem is to smooth the image prior to derivative estimation.

40 What Causes an Edge? What causes an edge? Depth discontinuity Surface orientation discontinuity Reflectance discontinuity (i.e., change in surface material properties) Illumination discontinuity (e.g., shadow) Slide credit: Christopher Rasmussen 40 / 92

41 41 / 92 Smoothing and Differentiation Edge: a location with high gradient (derivative) Need smoothing to reduce noise prior to taking derivative Need two derivatives, in x and y direction We can use derivative of Gaussian filters because differentiation is convolution, and convolution is associative Let denote convolution D (G I(X, Y )) (D G) I(X, Y )

42 42 / 92 Gradient Magnitude Let I(X, Y ) be a (digital) image Let I x (X, Y ) and I y (X, Y ) be estimates of the partial derivatives in the x and y directions, respectively. Call these estimates I x and I y (for short) The vector [I x, I y ] is the gradient The scalar I x 2 + I y 2 is the gradient magnitude

43 43 / 92 Two Generic Approaches to Edge Detection i(r) d i(r) dr r y r d 2 i(r) dr 2 r x r Two generic approaches to edge point detection: (significant) local extrema of a first derivative operator zero crossings of a second derivative operator

44 44 / 92 Marr/Hildreth Laplacian of Gaussian (cont d) Steps: 1 Gaussian for smoothing 2 Laplacian ( 2 ) for differentiation where 2 f (x, y) 2 f (x, y) x f (x, y) y 2 3 locate zero-crossings in the Laplacian of the Gaussian ( 2 G) where 2 G(x, y) = 1 2πσ 4 [2 x 2 + y 2 ] σ 2 exp x 2 + y 2 2σ 2

45 45 / 92 Marr/Hildreth Laplacian of Gaussian (cont d) Here s a 1D plot of the Laplacian of the Gaussian ( 2 G)... g(t) t... with its characteristic Mexican hat shape

46 46 / 92 Canny Edge Detection (cont d) Steps: 1 Apply directional derivatives of Gaussian 2 Non-maximum suppression thin multi-pixel wide ridges down to single pixel width 3 Linking and thresholding Low, high edge-strength thresholds Accept all edges over low threshold that are connected to edge over high threshold

47 47 / 92 Edge Hysteresis One way to deal with broken edge chains is to use hysteresis Hysteresis: A lag or momentum factor Idea: Maintain two thresholds k high and k low Use k high to find strong edges to start edge chain Use k low to find weak edges which continue edge chain Typical ratio of thresholds is (roughly) k high k low = 2

48 48 / 92

49 49 / 92 How do humans perceive boundaries? Figure credit: Szeliski Fig Original: Martin et al Each image shows multiple (4-8) human-marked boundaries. Pixels are darker where more humans marked a boundary.

50 50 / 92 Sample Question: Edges Why is non-maximum suppression applied in the Canny edge detector?

51 We can think of a corner as any locally distinct 2D image feature that (hopefully) corresponds to a distinct position on an 3D object of interest in the scene. 51 / 92 What is a corner? Credit: John Shakespeare, Sydney Morning Herald

52 52 / 92 Why are corners distinct"? A corner can be localized reliably. Thought experiment: Place a small window over a patch of constant image value. If you slide the window in any direction, the image in the window will not change. Place a small window over an edge. If you slide the window in the direction of the edge, the image in the window will not change Cannot estimate location along an edge Place a small window over a corner. If you slide the window in any direction, the image in the window changes.

53 53 / 92 Autocorrelation Figure credit: Szeliski Fig. 4.5

54 54 / 92 Corner Detection Edge detectors perform poorly at corners Observations: The gradient is ill defined exactly at a corner Near a corner, the gradient has two (or more) distinct values

55 55 / 92 Harris Corner Detector The Harris corner detector Form the second-moment matrix: Sum over a small region around the hypothetical corner C = I I x 2 x I y Gradient with respect to x, times gradient with respect to y I I x y 2 I y Matrix is symmetric Slide credit: David Jacobs

56 56 / 92 Harris Corner Detector Filter image with Gaussian Compute magnitude of the x and y gradients at each pixel Construct C in a window around each pixel Harris uses a Gaussian window Solve for product of the λs If λs both are big (product reaches local maximum above threshold) then we have a corner Harris also checks that ratio of λs is not too high

57 57 / 92

58 58 / 92

59 59 / 92 Example 1: Harris Corners Originally developed as features for motion tracking Greatly reduces amount of computation (compared to tracking every pixel) Translation and rotation invariant (but not scale invariant)

60 60 / 92 Harris Corner Detector Not scale invariant: Slide credit: Trevor Darrell

61 61 / 92 Sample Question: Corners The Harris corner detector is stable under some image transformations (features are considered stable if the same locations on an object are typically selected in the transformed image). True or false: The Harris corner detector is stable under image blur.

62 62 / 92 Texture We will look at two main questions: 1 How do we represent texture? Texture analysis 2 How do we generate new examples of a texture? Texture synthesis

63 63 / 92 Texture Synthesis Why might we want to synthesize texture? 1 To fill holes in images (inpainting) Art directors might want to remove telephone wires. Restorers might want to remove scratches or marks. We need to find something to put in place of the pixels that were removed We synthesize regions of texture that fit in and look convincing 2 To produce large quantities of texture for computer graphics Good textures make object models look more realistic

64 64 / 92 Texture Synthesis Figure credit: Szeliski Fig

65 65 / 92 Texture Synthesis Photo Credit: Associated Press

66 Efros and Leung: Synthesizing One Pixel 66 / 92 Infinite sample image SAMPLE Generated image Assuming Markov property, what is conditional probability distribution What conditional of p, given probability the neighbourhood distribution of p, given window? the neighbourhood window? Instead of constructing a model, let s directly search the Directly search the input image for all such neighbourhoods to input produce image a histogram for all such for p neighbourhoods to produce a histogram To synthesize for p, p pick one match at random To synthesize p, just pick one match at random Credit: p

67 EfrosReally and Leung: Synthesizing Really Synthesizing One One Pixel Pixel 67 / 92 finite sample image SAMPLE p Generated image However, since our sample image is finite, an exact neighbourhood match might not be present Since the sample image is finite, an exact neighbourhood match might not be present So Find we the find best the match best using match SSD using error, SSD weighted error by (weighted a Gaussian by toa Gaussian emphasize to local emphasize structure, local and structure), take all samples and within take all some samples distance from that match within some distance from that match Credit:

68 68 / 92 Efros and Leung: Synthesizing Many Pixels For multiple pixels, "grow" the texture in layers In the case of hole-filling, start from the edges of the hole For an interactive demo, see (written by Julieta Martinez, a previous CPSC 425 TA)

69 Randomness Parameter Efros and Leung: Randomness Parameter 69 / 92 Credit:

70 70 / 92 Big Data" Meets Inpainting Figure credit: Hays and Efros 2007

71 71 / 92 Big Data" Meets Inpainting Algorithm sketch (Hays and Efros 2007): 1 Create a short list of a few hundred best matching" images based on global image statistics 2 Find patches in the short list that match the context surrounding the image region we want to fill 3 Blend the match into the original image Purely data-driven, requires no manual labelling of images

72 72 / 92 The Goal of Texture Analysis The Goal of Texture Analysis input image ANALYSIS Same or different True (infinite) texture generated image Compare textures and decide if they re made of the Compare same stuff. textures and decide if they re made of the same stuff Credit: Bill Freeman 25

73 73 / 92 Texture Segmentation Question: Is texture a property of a point or a property of a region? Answer: We need a region to have a texture. There is a chicken and egg problem. Texture segmentation can be done by detecting boundaries between regions of the same (or similar) texture. Texture boundaries can be detected using standard edge detection techniques applied to the texture measures determined at each point We compromise! Typically one uses a local window to estimate texture properties and assigns those texture properties as point properties of the window s center row and column

74 74 / 92 Texture Representation Observation: Textures are made up of generic sub-elements, repeated over a region with similar statistical properties Idea: Find the sub-elements with filters, then represent each point in the image with a summary of the pattern of sub-elements in the local region

75 75 / 92 Texture Representation Observation: Textures are made up of generic sub-elements, repeated over a region with similar statistical properties Idea: Find the sub-elements with filters, then represent each point in the image with a summary of the pattern of sub-elements in the local region Question: What filters should we use? Answer: Human vision suggests spots and oriented edge filters at a variety of different orientations and scales

76 76 / 92 Texture Representation Observation: Textures are made up of generic sub-elements, repeated over a region with similar statistical properties Idea: Find the sub-elements with filters, then represent each point in the image with a summary of the pattern of sub-elements in the local region Question: What filters should we use? Answer: Human vision suggests spots and oriented edge filters at a variety of different orientations and scales Question: How do we summarize? Answer: Compute the mean or maximum of each filter response over the region Other statistics can also be useful

77 77 / 92 Texture Representation Figure credit: Leung and Malik 2001

78 78 / 92 Spots and Bars (Fine Scale) Forsyth & Ponce (1st ed.) Figures

79 79 / 92 Spots and Bars (Coarse Scale) Forsyth & Ponce (1st ed.) Figures 9.3 & 9.5

80 80 / 92 Laplacian Pyramid Building a Laplacian pyramid: Create a Gaussian pyramid Take the difference between one Gaussian pyramid level and the next (before subsampling) Properties Also known as the difference-of-gaussian (DOG) function, a close approximation to the Laplacian It is a band pass filter each level represents a different band of spatial frequencies Reconstructing the original image: Reconstruct the Gaussian pyramid starting at top

81 81 / 92 Oriented Pyramids (cont d) Forsyth & Ponce (1st ed.) Figure 9.13 Reprinted from Shiftable MultiScale Transforms, by Simoncelli et al., IEEE Transactions on Information Theory, 1992, c 1992, IEEE

82 82 / 92 Final Texture Representation Steps: 1 Form a Laplacian and oriented pyramid (or equivalent set of responses to filters at different scales and orientations) 2 Square the output (makes values positive) 3 Average responses over a neighborhood by blurring with a Gaussian 4 Take statistics of responses Mean of each filter output Possibly standard deviation of each filter

83 83 / 92 Sample Question: Texture How does the top-most image in a Laplacian pyramid differ from the others?

84 84 / 92 Colour Matching Experiments: I Forsyth & Ponce (2nd ed.) Figure 3.2 Show a split field to subjects. One side shows the light whose colour one wants to match. The other a weighted mixture of three primaries (fixed lights)

85 85 / 92 Colour Matching Experiments: II Many colours can be represented as a positive weighted sum of A, B, C Write M = a A + b B + c C where the = sign should be read as matches This is additive matching Defines a colour description system two people who agree on A, B, C need only supply (a, b, c)

86 86 / 92 Colour Matching Experiments: II (cont d) Some colours can t be matched this way Instead, we must write M + a A = b B + c C where, again, the = sign should be read as matches This is subtractive matching Interpret this as ( a, b, c) Problem for designing displays: Choose phosphors R, G, B so that positive linear combinations match a large set of colours

87 87 / 92 Linear Colour Spaces A choice of primaries yields a linear colour space the coordinates of a colour are given by the weights of the primaries used to match it Choice of primaries is equivalent to choice of colour space RGB: Primaries are monochromatic energies, say nm, nm, nm CIE XYZ: Primaries are imaginary, but have other convenient properties. Colour coordinates are (X, Y, Z ), where X is the amount of the X primary, etc.

88 88 / 92 Uniform Colour Spaces McAdam ellipses demonstrate that differences in x, y are a poor guide to differences in perceived colour A uniform colour space is one in which differences in coordinates are a good guide to differences in perceived colour example: CIE LAB

89 89 / 92 HSV Colour Space More natural description of colour for human interpretation Hue: attribute that describes a pure colour e.g. red, blue Saturation: measure of the degree to which a pure colour is diluted by white light pure spectrum colours are fully saturated Value: intensity or brightness Hue + saturation also referred to as chromaticity.

90 90 / 92 Colour Constancy Image colour depends on both light colour and surface colour Colour constancy: determine hue and saturation under different colours of lighting It is surprisingly difficult to predict what colours a human will perceive in a complex scene depends on context, other scene information Humans can usually perceive the colour a surface would have under white light the colour of the reflected light (separate surface colour from measured colour)

91 91 / 92 Summary Table Summary of what we ve seen so far: Representation Result is... Approach Technique intensity dense (2D) template matching (normalized) correlation, SSD edge relatively sparse (1D) derivatives 2 G, Canny corner sparse (0D) locally distinct features Harris

92 Questions? 92 / 92