Designing Applications that See Lecture 3: Image Processing

Transcription

1 stanford hci group / cs377s Designing Applications that See Lecture 3: Image Processing Dan Maynes-Aminzade 5 January 008 Designing Applications that See

2 Reminders Register on Axess Assignment # out today, due next Tuesday in lecture Newsgroup: su.class.cs377s Next lecture is an interactive workshop, so bring your webcam if you have one Remember to check the course calendar for the latest readings

3 Today s Goals Get an overview of various low-level image processing techniques Understand what they are good for and when to use them Next lecture: hands-on demo of these techniques in action 3

4 Outline Image basics Color Image filters Features Shapes 4

5 Image Processing Which of these two images contains a red circle? 5

6 Image Processing Which of these two images contains a red circle? 6

7 Image Processing Which of these two images contains a red circle? 7

8 Preattentive Processing Certain basic visual properties are detected immediately by low-level visual system Pop-out vs. serial search Tasks that t can be performed in less than 00 to 50 milliseconds on a complex display Eye movements take at least 00 ms to initiate 8

9 Preattentive Visual Search Tasks 9

10 Cockpit Dials Detection of a slanted line in a sea of vertical lines is preattentive 0

11 Perspective (courtesy of Maria Petrou)

12 What is an Image? Digital representation of a real-world scene Composed of discrete picture elements (pixels) Pixels parameterized by Position Intensity Time

13 What is an Image? A D domain With samples at regular points (almost always a rectilinear grid) Whose values represent gray levels, colors, or opacities Common image types: sample per point (B&W or Grayscale) 3 samples per point (Red, Green, and Blue) 4 samples per point (Red, Green, Blue, and Alpha, a.k.a. Opacity) 3

14 Channels In an images with multiple samples per pixel, we refer to each set of one type of sample as a "plane" or "channel." 3 samples per pixel Blue Channel of Image, sample per pixel Red Channel of Image, sample per pixel (courtesy Andries van Dam) 4

15 Spatial Sampling (courtesy of Gonzalez and Woods) 5

16 Quantization (courtesy of Gonzalez and Woods) 6

17 What is Image Processing? Generally, an attempt to do one of the following: Restore an image (take a corrupted image and recreate a clean original) Enhance an image (alter an image to make its meaning clearer to human observers) Understand an image (mimic the human visual system in extracting meaning from an image) 7

18 Image Restoration Removing sensor noise Restoring old, archived film and images that has been damaged (courtesy of Gonzalez and Woods) 8

19 Image Restoration 9

20 Image Enhancement Often used to increase the contrast in images that are overly dark or light Enhancement algorithms often play to humans sensitivity to contrast (courtesy of Tobey Thorn) 0

21 Image Understanding Image understanding includes many different tasks Segmentation (identifying objects in an image) Classification (assigning labels to individual objects or pixels) Interpretation (extracting meaning from the image as a whole)

22 Image Processing: Hierarchy of Tasks (courtesy of B. Jähne)

23 Separating Regions with Filters 3

24 What is Color? A box of pencils? 4

25 What is Color? A quantity related to the wavelength of light in the visible spectrum? (courtesy of J.M. Rehg) 5

26 What is Color? A perceptual attribute of objects and scenes constructed by the visual system? (courtesy R. Beau Lotto) 6

27 Color Perception (courtesy R. Beau Lotto) 7

28 How Do We Use Color? In Biological Vision Distinguish food from nonfood Help identify predators and prey Check health of others In Computer Vision Find a person s skin Segment (group together) pixel regions belonging to the same object (courtesy of J.M. Rehg) 8

29 Colorblindness Ishihara Test for Color Blindness 9

30 Human Photoreceptors Rods Cones 30

31 Human Cone Sensitivities 3

32 Reflectance and Transmittance Reflectance Transmittance (courtesy of Brian Wandell) 3

33 Reflectance 33

34 Transmittance 34

35 Illumination Spectra 35

36 Reflectance Spectra 36

37 Assigning Names to Spectra 37

38 Additive Color Mixing 38

39 Subtractive Color Mixing 39

40 The Color Matching Experiment 40

41 Experiment Step (courtesy of Bill Freeman) 4

42 Experiment Step (courtesy of Bill Freeman) 4

43 Experiment Step (courtesy of Bill Freeman) 43

44 Experiment Step (courtesy of Bill Freeman) 44

45 Experiment Step (courtesy of Bill Freeman) 45

46 Experiment Step (courtesy of Bill Freeman) 46

47 Experiment Step (courtesy of Bill Freeman) 47

48 Conclusion from Experiment Three primaries are enough for most people to reproduce arbitrary colors Why is this the case? 48

49 Univariance Principle L = ρk ( λ) E( λ) dλ k L, M, S For illuminant E( λ) k = 49

50 Tristimulus Color Space Three axes represent responses of the three types of receptors Horseshoe-shaped cone is the set of all humanperceptible colors 50

51 Munsell Color System Hue (Wavelength) Value (Luminosity) Chroma (Saturation) 5

52 Color Space 5

53 How to Match a Color Choose primaries A, B, C Given an energy distribution, which amounts of primaries will match it? For each wavelength thin the distribution, ib ti determine how much of A, B, and C is needed to match light of that wavelength alone Add these together to produce a match 53

54 Matching on a Color Circle 54

55 RGB Color Matching 55

56 Geometry of Color (CIE) Perceptual color spaces are non-convex Three primaries can span the space, but weights may be negative. Curved outer edge consists of single wavelength primaries 56

57 CIE Chromaticity Diagram 57

58 RGB Color Space 58

59 Constructing Colors Color can be constructed in many ways 59

60 Color Spaces Use color matching functions to define a coordinate system for color Each color can be assigned a triple of coordinates with respect to some color space (e.g. RGB) Devices (monitors, printers, projectors) and computers can communicate colors precisely 60

61 McAdam Ellipses (0 times actual size) (Actual size) (courtesy of D. Forsyth) 6

62 Human Color Constancy Distinguish between Color constancy, which refers to hue and saturation Lightness constancy, which refers to gray-level. Humans can perceive Color a surface would have under white light (surface color) Color of the reflected light (limited ability to separate surface color from measured color) Color of illuminant (even more limited) (courtesy of J.M. Rehg) 6

63 Spatial Arrangement and Color Perception 63

64 Spatial Arrangement and Color Perception 64

65 Spatial Arrangement and Color Perception 65

66 Spatial Arrangement and Color Perception 66

67 Spatial Arrangement and Color Perception 67

68 (courtesy of D. Forsyth) 68

69 Land s Mondrian Experiments Squares of color with the same color radiance yield very different color perceptions Photometer:.0, 0.3, 0.3 Photometer:.0, 0.3, 0.3 White light Audience: Red Colored light Audience: Blue (courtesy of J.M. Rehg) 69

70 Lightness Constancy Algorithm The goal is to determine what the surfaces in the image would look like under white light. Compares the brightness of patches across their common boundaries and computes relative brightness Establish an absolute reference for lightness (e.g. brightest point is white ) 70

71 Lightness Constancy Example (courtesy of John McCann) 7

72 Finite Dimensional Linear Models ()= ε i ψ i () λ E λ m i= L k = σ k ( λ) ε ψ ( λ) r ϕ ( λ) m i= i i n j= j j dλ = m, n ε r i i=, j= j σ k ( λ) ψ ( λ) ϕ ( λ) i j dλ ()= r j ϕ j () λ ρλ n j= = m, n ε r i i=, j= j g ijk 7

73 From Images to Objects and Regions Attributes of regions Bounding edges Texture The need to compute and reason about spatial aggregations of pixels leads us to filtering Key problem is segmentation 73

74 Features and Filters: Questions What is a feature? What is an image filter? What is it good for? How can we find corners? How can we find edges? 74

75 What is a Feature? In computer vision, a feature is a local, meaningful, detectable part of an image 75

76 Why Use Features? High information content Invariant to changing viewpoint or changing illumination Reduces computational ti burden (courtesy of Sebastian Thrun) 76

77 One Type of Computer Vision Image Feature Feature : Feature N Computer Vision Algorithm Image Feature Feature : Feature N (courtesy of Sebastian Thrun) 77

78 Where Features Are Used Calibration Image Segmentation Correspondence in multiple images (stereo, structure t from motion) Object detection, classification (courtesy of Sebastian Thrun) 78

79 What Makes a Good Feature? Invariance View point (scale, orientation, translation) Lighting condition Object deformations Partial occlusion Other Characteristics Uniqueness Sufficiently many Tuned to the task (courtesy of Sebastian Thrun) 79

80 What is Image Filtering? Modify the pixels in an image based on some function of a local neighborhood of the pixels

81 Linear Filtering Linear case is simplest and most useful Replace each pixel with a linear combination of its neighbors. The specification of the linear combination is called the convolution kernel * = 7 kernel 8

82 I(.) I(.) I(.) I(.) I(.) I(.) I(.) I(.) I(.) g g g 3 g g g 3 g 3 g 3 g 33 f (i,j) = g I(i-,j-) + g I(i-,j) + g 3 I(i-,j+) + g I(i,j-) + g I(i,j) + g 3 I(i,j+) + g 3 I(i+,j-) + g 3 I(i+,j) + g 33 I(i+,j+) 8

83 Step (courtesy of Christopher Rasmussen)

84 Step (courtesy of Christopher Rasmussen)

85 Step (courtesy of Christopher Rasmussen)

86 Step (courtesy of Christopher Rasmussen)

87 Step (courtesy of Christopher Rasmussen)

88 Step

89 - Final Result I I Why is I large in some places and small in others? (courtesy of Christopher Rasmussen)

90 Filtering Example 90

91 Filtering Example 9

92 Filtering Example 9

93 Problem: Image Noise (courtesy of Forsyth & Ponce) 93

94 Solution: Smoothing Filter If object reflectance changes slowly and noise at each pixel is independent, then we want to replace each pixel with something like the average of neighbors Disadvantage: Sharp (high-frequency) features lost Original image 7 x 7 averaging neighborhood 94

95 Smoothing Filter: Details Filter types Mean filter (box) Median (nonlinear) Gaussian Can specify linear operation by shifting kernel over image and taking product 3 x 3 box filter kernel 95

96 Gaussian Kernel Idea: Weight contributions of neighboring pixels by nearness x 5, σ = Smooth roll-off reduces ringing seen in box filter 96

97 Gaussian Smoothing Example Original image Box filter 7 x 7 kernel 5 January 008 σ = Lecture 3: Image Processing σ = 3 97

98 Gaussian Smoothing Averaging Gaussian (courtesy of Marc Pollefeys) 98

99 Box vs. Cone Filters (courtesy Andries van Dam) 99

100 Smoothing Reduces Noise (courtesy of Marc Pollefeys) 00

101 What causes an edge? Depth discontinuity Change in surface orientation Reflectance discontinuity (change in surface properties) Illumination discontinuity (light/shadow) (courtesy of Christopher Rasmussen) 0

102 How Can We Find Edges? 0

103 Spatial Structure of Edges Edges exist at different scales from fine (whiskers) to coarse (stripes) Solution: Gaussian smoothing 03

104 Blurring Example 04

105 Scale Increasing smoothing: Eliminates noise edges Makes edges smoother and thicker Removes fine detail 05

106 Effect of Smoothing Radius pixel 3 pixels 7 pixels 06

107 The Edge Normal S = dx + α = arctan dy dy dx 07

108 Sobel Operator S = S = Edge Magnitude = S + S Edge Direction = tan - S S 08

109 The Sobel Kernel, Explained = /4 * [- 0 -] = /4 * [ ] 0 - Sobel kernel is separable! - - Averaging done parallel to edge 09

110 Combining Kernels ( I g) h = I ( g h) [ ] [ ] 0

111 Robinson Compass Masks

112 Robert s Cross Operator S = [ I(x, y) - I(x+, y+) ] + [ I(x, y+) - I(x+, y) ] or S = I(x, y) - I(x+, y+) + I(x, y+) - I(x+, y)

113 Claim Your Own Kernel! Prewitt Kirsch Frei & Chen Prewitt Sobel (courtesy of Sebastian Thrun) 3

114 What s Not a Convolution? Nonlinear systems For example, radial distortion of a fish-eye lens f (courtesy of M. Fiala) 4

115 John Canny s Edge Detector John Canny, Finding Edges and Lines in Images, Master s Thesis, MIT, June 983. Developed a practical edge detection algorithm with three properties Gaussian smoothing Non-maximum suppression (remove edges orthogonal to a maxima) Hysteresis thresholding Improved recovery of long image contours 5

116 Sobel Example (courtesy of Sebastian Thrun) 6

117 Canny Example (courtesy of Sebastian Thrun) 7

118 Comparison Sobel Canny (courtesy of Sebastian Thrun) 8

119 Canny Edge Detection Steps:. Apply derivative of Gaussian. Non-maximum suppression Thin multi-pixel wide ridges down to single pixel width 3. Linking and thresholding Low, high edge-strength thresholds Accept all edges over low threshold that are connected to edge over high threshold 9

120 Non-Maximum Suppression Select the single maximum point across the width of an edge. 0

121 Linking to the Next Edge Point Assume the marked point q is an edge point. Take the normal to the gradient at that point and use this to predict continuation points (either r or p).

122 Edge Hysteresis Hysteresis: A lag or momentum factor Idea: Maintain two thresholds: k HIGH and k LOW Use k HIGH to find strong edges to start t edge chain Use k LOW to find weak edges which continue edge chain Typical ratio of thresholds is roughly k HIGH / k LOW =

123 Canny Edge Detection (Example) gap is gone Original image Strong + connected weak edges Strong edges only Weak edges (courtesy of G. Loy) 3 3

124 Canny Edge Detection (Example) Using default thresholds in Matlab 4

125 Choosing Parameters 5

126 Choosing Parameters Fine scale, High threshold 6

127 Choosing Parameters Coarse scale, High threshold 7

128 Choosing Parameters Coarse scale, Low threshold 8

129 Image Arithmetic Just like matrices, we can do pixelwise arithmetic on images Some useful operations Differencing: Measure of similarity Averaging: Blend separate images or smooth a single one over time Thresholding: Apply function to each pixel, test value 9

130 Image Type Conversion Many processing functions work on just one channel, so the options are: Run on each channel independently Convert from color grayscale weighting each channel by perceptual importance 30

131 Thresholding Grayscale Binary: Choose threshold based on histogram of image intensities 3

132 Image Comparison: Histograms 3

133 Color Similarity One measure is Chrominance Distance : distance from color to a reference color 33

134 Image Comparison One approach to template matching is a sum of squared differences: 34

135 Correlation for Template Matching Note that SSD formula can be written: When the last term is big, the mismatch is small the dot product measures correlation: By normalizing by the vectors lengths, we are measuring the angle between them 35

136 Normalized Cross-Correlation Shift template image over search image, measuring normalized correlation at each point Local maxima indicate template matches 36

137 Template Matching Example (courtesy of Sebastian Thrun) 37

138 Statistical Image Comparison: Color Histograms Steps Histogram RGB/HSI triplets over two images to be compared Normalize each histogram by respective total number of pixels to get frequencies Similarity is Euclidean distance between color frequency vectors Insensitive to geometric changes, including different-sized images 38

139 Connected Components After thresholding, method to identify groups or clusters of like pixels Uniquely label each n-connected region in binary image 4- and 8-connectedness 39

140 Connected Components Example 40

141 Binary Operations Dilation, erosion Dilation: All 0 s next to a (Enlarge foreground) Erosion: All s next to a 0 0 (Enlarge background) 4

142 Image Moments: Region Statistics Zeroth-order: Size/area First-order: Position (centroid) Second-order: Orientation 4

143 Other Effects Art Effects Posterizing Faked Aging Impressionist pixel remapping Technical Effects Color remapping Grayscale conversion Contrast balancing (courtesy Andries van Dam) 43

144 Finding Corners Edge detectors perform poorly at corners. Corners provide repeatable points for matching, so are worth detecting. Problem: Exactly at a corner, gradient is ill-defined. However, in the region around a corner, gradient has two or more different values. 44

145 Harris Corner Detector C = I x I x I y I x I I y y 45

146 Simple Case First, consider case where: C = I I x x I y I x y y I I = This means dominant gradient directions align with x or y axis If either λ is close to 0, then this is not a corner, so look for locations where both are large. 46 λ 0 0 λ

147 General Case It can be shown that since C is rotationally symmetric: C = R λ 0 0 λ R So every case is like a rotated version of the one on last slide. 47

148 So, To Detect Corners Filter image with Gaussian to reduce noise Compute magnitude of the x and y gradients at each pixel Construct C in a window around each pixel (Harris uses a Gaussian window just blur) Solve for product of l S (determinant of C) If l S are both big (product reaches local maximum and is above threshold), we have a corner (Harris also checks that ratio of l S is not too high) 48

149 Gradient Orientation Closeup 49

150 Corner Detection Corners are detected where the product of the ellipse axis lengths reaches a local maximum. 50

151 Harris Corners (courtesy of Sebastian Thrun) 5

152 Example (σ=0.) (courtesy of Sebastian Thrun) 5

153 Example (σ=0.0) (courtesy of Sebastian Thrun) 53

154 Example (σ=0.0) (courtesy of Sebastian Thrun) 54

155 Features? Global, not Local 55

156 Vanishing Points 56

157 Vanishing Points A. Canaletto [740], Arrival of the French Ambassador in Venice 57

158 From Edges to Lines 58

159 Hough Transform b = mx + y b ' = m' x + y b '' = m'' ' x + y 59

160 Hough Transform: Quantization b m (courtesy of Sebastian Thrun) 60

161 Hough Transform: Algorithm For each image point, determine most likely line parameters b,m (direction of gradient) strength (magnitude of gradient) Increment parameter counter by strength value Cluster in parameter space, pick local maxima 6

162 Hough Transform: Results 6

163 Hough Transform? 63

164 Summary Many kinds of mathematical operations can be performed on images to extract basic information about shapes and features Image processing does not itself lead to image understanding But it s often a good start 64

165 Reading for Next Lecture Introduction to the MATLAB Image Processing Toolbox Assignment # Question #4: MATLAB introduction 65